Decryption: The Dungeon of Despair

[A_n_g_e_l_a]

An Escape Plan...

OK! You've captured your own Content Decryption Module (CDM); tried a few sites and found you still cannot get all the keys that you expected. And you are a bit fed-up!

I scratched at that brick wall too trying to find a way out the dungeon. At the time it felt like the wall was made of headers!

It actually isn't.

The real problem is being able to see and understand what is going on! Headers are relatively easy to sort out, in the main.

Headers go unseen in our normal web interactions and finding a way to see what is happening is essential.

But before we do - a note from our sponsor:-

Don't talk about Pay Sites

In what that follows, examples will come from sites that are free to access and which mainly broadcast media, in the clear, to the Universe; including towards aliens orbiting Alpha-Centuri. If they can see it for free in a few light years time - why can't I see it now?

There is no discussion here of pay sites like Netflix or HBO or Amazon, but the principles are the same. The forum rules forbid chat about pay-sites so please don't ask. I don't go after pay sites myself anyway - so have no knowledge to share.

We are going to look for 'patterns'. Patterns are useful to developers and books get published wih patterns of code that does stuff which developers incorporate in their code to do stuff for them too. It seems highly probable to me that Developer X will re-use code produced and shared by Developer Y.
Google also published, and then restricted access to, the Widevine API - all WV developers must conform to the needs of the API so there can only be a few patterns of interaction.

Let's start escaping the Dungeon of Despair.. but with some new tools!

Information is key. We need to see what is going on during the whole process of playing a video in a browser. After-all, we want to mimic some of that process with a python script.

Developer tools, in your browser under F12 allows some insight;
if you select Network to see the traffic;
click on the license url; and
select headers - you can see the contents quite readily.


[Attachment 67054 - Click to enlarge]Basic Headers

So you can sort headers out yourself by looking and thinking.

Think: What you are trying to do is mimic the browser by sending requests to a license server to get it to send you data, so your CDM can use the data to produce keys
OK, just think where you can see an example of what the browser sends? (Clue image above!)

Look: Inside the dev tools of your browser and inspect the request headers for the license url.
Your request headers need to look like that, in the main. With experience you will be able to miss out the junk parts of the the header's contents that make no difference to whether keys are obtained or not.

Headers are found by pasting the license url copied as a cURL from the browsers dev-tools into curlconverter.com. You want only the part

headers = {
                   ..
                   }

But also look inside params {..} And check there is nothing there that is inside the headers data in your dev-tools request headers. If there is, copy it to a new line in your headers section. Key-words to look out for are bearer, x-authorization-data, key. But, by inspecting the browser's license request you will be able to see exactly what is needed,

There will be more on headers below - headers can be grouped from not-needed to complex.

But lets move away from browser interactions for a moment, what about out interactions using using l3.py and our CDM?
The command window may show problems with the python code where the routine came to a halt but that is all. A typical beginners response from l3.py with bad headers is 'Check protobufs'.
How can we see the goings on there?

Sorenb kindly pointed me to HTTPToolkit!! Get it and run it. It is the bees knees! Scroll down to the second "Download Free for Windoze" button and clicking reveals versions for Mac and Linux as well as Windoze.

Open it and select "Existing Terminal"


[Attachment 67073 - Click to enlarge]Select Existing Terminal

Copy the code HttpToolKit provides - repeated here:-

eval "$(curl -sS localhost:8001/setup)"

into the terminal (command) window. Now any Terminal (command) interactions here will show up in the HttpToolKit window.

The code below shows an example terminal command; it will fail and give no screen output. Quite useless for understanding what is happening.

curl https://manifest.prod.boltdns.net/license/v1/cenc/widevine/1242911124001/99bf6c70-37ba-434c-9474-55dc2fde25a6/7ca86aec-55cf-4f9f-ba33-c2fccecb52fe?fastly_token=NjM1YWY0MWJfZDRkOWExYzI4NTg3OGFlZmFlMjE1NDA1MTZiZjQ3MjBlMDM0ZDg1Yzc5ODU4ZmQ3OTAwMWU3YzY4ODI2NDZjYw%3D%3D

But HttpToolKit captures the interaction between browser and web.


[Attachment 67096 - Click to enlarge]Method Not Allowed

By looking at the error it says "Method not allowed". The license server was expecting a POST but a GET Request was sent. So instead of no help from the Terminal. HttpToolKit shows our error.

Helpful or what?

There are alternatives to HttpToolkit - some recommend Charles-proxy or Fiddler; choose what you like best. For me, Charles Proxy took no account of my screen resolution - I dumped it.

The beauty of HttpToolKit is that you are not limited to one process (Terminal) to capture from. By choosing another capture process, for example a browser, we can see both ends of the problem. I chose Brave.. the Firefox version at the time wouldn't or couldn't download the Widevine browser add-on. Extensions can be added in the usual way and can persist after closure of HttpToolkit. Edit: sometimes if a browser has been updated, extensions via httptoolkit get lost and need re-installing.

Running our Brave browser now shows all the requests needed to load a page;-


[Attachment 67075 - Click to enlarge]
Filtered on 'manifest'

The images shows interaction from Brave and the last one '?' being from my terminal.

So now we have a more rounded tool than developer options in a browser: so we can see the interactions from many sources at the same time. Adb connected devices, like a phone, may also be intercepted (a rooted phone helps).

Android Studio started life for Developers creating Phone Apps, but it comes with an ability to emulate almost any version of Android phone. The emulation may be rooted.

HttpToolKit is privy to almost all traffic from an emulated phone. Except a few apps; Banking Apps of course; but some media Apps too will not work easily with HttpToolKit.


The general process of getting decryption keys

There are a range of types of interactions to permit the getting of keys depending on the Content Delivery Network.

The diagram below shows the general Widevine interactions with a cloud server - I found this on the web; ignore pollycon - it is just a trade name.

This is THE pattern - remember.


[Attachment 67056 - Click to enlarge]
General Widevine Process outlined. The pattern the developer follows:-

Looking and starting top right the lines down indicate time passage and shows the processes happening on the time-line and interactions between Client, Content Delivery Network (CDN) and the License Server. So, for the moment, ignoring the box labelled opt - which is a Widevine option - we see the browser, or our sripted code that mimics the browser, needs to:-

• Send a request with a token to the license server
• Receive the issued license and pass it to the Content Decryption Module to provide keys

One question now is:- where does the token go?

Let's look at that process in more detail with respect to tokens.

Many basic sites add the token to a url:
for example:-

https://manifest.prod.boltdns.net/license/v1/cenc/widevine/1555966122001/01fc4603-6263...EyNWMzNA%3D%3D

But a token may be passed in a number of ways - the list below shows some of the different requirements or code-patterns to obtain keys using some version of Pywidevine. It is a start and not definitive:-

Some CDN site requirements - as an overview - are:

1. pssh + license url + token in the url; - headers are basic and some, apparently, ignored e.g. uktvplay.co.uk, tg4.ie, some of stv.tv
   
2. pssh + license url + token + headers with specific requrements e.g. channel5.com
   If the referrer isn't channel5 you won't get your keys.
   
3. pssh + license + headers with a token as x-custom-data e.g. npostart.nl (I am told the x-custom-data expires in 45 seconds! So a script is needed to do the interactions).
   
4. pssh + license + headers + a constructed json packet (with an extra interchange before the final license delivered to be decoded in CDM to provide keys) e.g. channel4.com, rte.ie
   Note the interchange for this method is the 'opt' variation in the above diagram.

In more detail...

In The Temple of Doom we use curlconverter.com to explore the licence header. Copying the licence as a cURL. allows us to look more closely at header types.

As an aside: I am told copying as cURL is different on Windoze
copy cURL > copy as curl (Windows) for FireFox and
copy cURL > copy as curl (Bash) for Chrome.

I inhabit a Linux world and don't see such complication.

We can also see the Request Headers in HttpToolkit for any request made to a server.

Looking more closely at the header patterns above and referring to the patterns by number:

1. pssh license and basic headers: I use uktvplay as an example,
   
   l3py works with a basic set of headers. Curlconverter.com's output is below:-
   
   

   import requests
   
   
   headers = {
       'User-Agent': 'Mozilla/5.0 (Windows NT 6.3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36 Edg/101.0.1210.39',
       'Accept': '*/*',
       'Accept-Language': 'en-GB,en;q=0.5',
       # 'Accept-Encoding': 'gzip, deflate, br',
       'Content-type': 'application/octet-stream',
       'Origin': 'https://uktvplay.co.uk',
       'DNT': '1',
       'Connection': 'keep-alive',
       'Referer': 'https://uktvplay.co.uk/',
       'Sec-Fetch-Dest': 'empty',
       'Sec-Fetch-Mode': 'cors',
       'Sec-Fetch-Site': 'cross-site',
       # Requests doesn't support trailers
       # 'TE': 'trailers',
   }
   
   params = {
       'fastly_token': 'NjM1ZmE2MTlfMjM1ZGUwNTNkMzc1ZWEyZjA4ODU4YWMwOWFlODlkNTRlZWQwNDAwODVlODc2ZDRhN2U2N2ZlYTJlZGI5NGNhMQ==',
   }
   
   data = 'Cut'
   
   response = requests.post('https://manifest.prod.boltdns.net/license/v1/cenc/widevine/1242911124001/251dcec2-c1f2-4ade-abc2-a2258f2326d0/56686d88-a8b6-4fee-a08a-925902914abc', params=params, headers=headers, data=data)

   Looking at the cURL converter output there seems to be a token in the copy that curlconverter gave but labelled params. It looks important with a token in there. Why don't we need these headers then? After all we only user the bit 'headers ={...}' in our headers.py file.
   
   Let's look at the licence url:-
   
   

   https://manifest.prod.boltdns.net/license/v1/cenc/widevine/1242911124001/251dcec2-c1f2-4ade-abc2-a2258f2326d0/56686d88-a8b6-4fee-a08a-925902914abc?fastly_token=NjM1ZmE2MTlfMjM1ZGUwNTNkMzc1ZWEyZjA4ODU4YWMwOWFlODlkNTRlZWQwNDAwODVlODc2ZDRhN2U2N2ZlYTJlZGI5NGNhMQ%3D%3D

   And as a parameter at the end of the url is:-
   fastly_token=NjM1ZmE2MTlfMjM1ZGUwNTNkMzc1ZWEyZjA4O DU4YWMwOWFlODlkNTRlZWQwNDAwODVlODc2ZDRhN2U2N2ZlYTJ lZGI5NGNhMQ.
   
   So to answer our question;
   
   .... if the url carries the tokens then the headers.py can simply be:
   
   

   headers = {
       'User-Agent': 'Mozilla/5.0 (Windows NT 6.3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36 Edg/101.0.1210.39',
       'Accept': '*/*',
       'Accept-Language': 'en-GB,en;q=0.5',
       # 'Accept-Encoding': 'gzip, deflate, br',
       'Content-type': 'application/octet-stream',
       'Origin': 'https://uktvplay.co.uk',
       'DNT': '1',
       'Connection': 'keep-alive',
       'Referer': 'https://uktvplay.co.uk/',
       'Sec-Fetch-Dest': 'empty',
       'Sec-Fetch-Mode': 'cors',
       'Sec-Fetch-Site': 'cross-site',
       # Requests doesn't support trailers
       # 'TE': 'trailers',
   }

   Unless, of course, you see the browser sending a token in the headers too! So use your eyes spot what the browser sends and ensure you send similar.
   
   
2. pssh license and headers: I use channel5.com as an example.
   
   

   https://cassie.channel5.com/api/v2/licences/widevine/208/C5273420001?expiry=1664877498&tag=36323362323161363462613635613037363534376230346534663433653537376636303738363438

   import requests
   
   headers = {
       'User-Agent': 'Mozilla/5.0 (Windows NT 6.3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36 Edg/101.0.1210.39',
       'Accept': '*/*',
       'Accept-Language': 'en-GB,en;q=0.5',
       # 'Accept-Encoding': 'gzip, deflate, br',
       'Origin': 'https://www.channel5.com',
       'DNT': '1',
       'Connection': 'keep-alive',
       'Referer': 'https://www.channel5.com/',
       'Sec-Fetch-Dest': 'empty',
       'Sec-Fetch-Mode': 'cors',
       'Sec-Fetch-Site': 'same-site',
       'Content-Type': 'application/x-www-form-urlencoded',
   }
   
   params = {
       'expiry': '1664877498',
       'tag': '36323362323161363462613635613037363534376230346534663433653537376636303738363438',
   }
   
   data = '<snipped as it was an encrypted message from the browser>}'
   
   response = requests.post('https://cassie.channel5.com/api/v2/licences/widevine/208/C5273420001', params=params, headers=headers, data=data)

   
   [Attachment 67055 - Click to enlarge] Channel 5 Headers
   
   Looking at the curlconverter output for the parameters carried on the url, it lists the 'tag' (token) and also an expiry time. Expiry times are another trap to fall into and I will deal with them in a little more detail later.
   
   
3. X-custom -data passed in header: I use npostart.nl as an example
   
   
   [Attachment 67057 - Click to enlarge]npostart.nl showing x-custom-data
   
   And that was simply enough to achieve. But site developers seem to want to prevent us taking a copy - the x-custom-data is limited to 45 second lifetime. There is a working script by Pkp on Notaghost's site if your Dutch is fluent and you want access to npostart.nl's output.
   
   
4. Option:License Token Integration: The last of our list is the most complex and requires a special script. It is the 'opt box' from the WV process diagram above.
   For example rte.ie/player/
   
   I have started my browser from HttpToolKit and navigated to the RTE site. I found a video to play, then cleared all the earlier interactions. Then played the video. The process is below:-
   
   
   [Attachment 67092 - Click to enlarge]rte.ie/player/ with the mpd link selected and pssh highlighted
   
   But we are more concerned with the license interactions:
   
   
   [Attachment 67093 - Click to enlarge]The license request contains json code - highlighted.
   
   Let us look at that in bit closer
   
   
   [Attachment 67094 - Click to enlarge]a json packet sent with the request to the license server.
   
   How to do that?
   In my early days I wondered why I couldn't just send all the stuff I see the browser sending. until I realised all the browser sends is encrypted for the browser's CDM not ours. In our scripts we mimic the browser but make out own calls to the license server using data created by our CDM.
   
   Partial code for copying:
   

   def WV_Function(pssh, lic_url, cert_b64=None):
   	"""
   	Func, emulates license request and then decrypts obtained license.
   	Fields that changes every new request is signature, 
   	expirationTimestamp, 
   	watchSessionId, 
   	puid, and 
   	rawLicenseRequestBase64 
   
   
       Every DRM provider provides its own
   
       1.mechanism to create a license request (using the KeyID, device identifier, signing the request, etc.)
       2.mechanism to understand the license response received from the DRM License Server \
   		(the response is encrypted too) and extract the decryption key.
       3.rules around storing the license locally on the client, license renewal, expiry, etc.
   	"""
   	'''First create a session passing pssh to our CDM'''
   	wvdecrypt = WvDecrypt(init_data_b64=pssh, cert_data_b64=cert_b64, device=deviceconfig.device_android_generic) 
   	'''ask for an initial encrypted WidevineChallenge string to use in our call to the license server
   	   produced using pssh and data about our CDM - see point 1 above'''                  
   	raw_request = wvdecrypt.get_challenge()
   	request = b64encode(raw_request)
   	
   
   # rte.ie support
       # read headers.py produced from cURL of up-to-date license
   	# to extract releasePid from RTE licence url
   	myjson = json.loads(headers.data)
   	pid = myjson['getWidevineLicense']['releasePid']
   	responses.append(requests.post(url=lic_url, headers=headers.headers,
   		json={
   			"getWidevineLicense": 
   				{
   				'releasePid': pid,
   				'widevineChallenge': str(request, "utf-8" )
   				}, 
   				}))

   same code for reading:
   
   
   [Attachment 67103 - Click to enlarge]Rte.ie code
   
   I took this snippet from https://github.com/medvm/widevine_keys/blob/main/l3.py and added comments to try to make sense of the process.
   A big thank you to Medvm.
   
   The code above mimics the license request - see below.
   Other patterns of license interaction are included in his l3.py version. Select the one you want that suits the site you are after. Comment out or remove all the other methods otherwise you will get a python error.
   
   Edit: Medvm noticed that 'releasePid' is the same given to the browser as would be to us. He chooses to take it from the full curl copy of the license making a liar of me saying that headers.data is irrelevant. But note the Widevine Challenge string is generated by our CDM. /edit
   
   Channel 4 uses an initial interaction to get a license from the license server to get a key-decode-license from the license server! Again a json packet is exchanged. Find an example on the web - or here in the forum and adapt for your own use.

There will be riffs on the above all set-up to prevent access but the patterns will be similar. With the right tools you can follow along and detect what is happening.

It is a thoughtful process. Do not expect it to be easy.

The final wrinkle is time.

Some sites severely limit for how long you may access an mpd or call the license server after being served the link. There will be a time-string in the url and if you find your request failing and l3 complaining. You might have run out of time.

Here is a short rough and ready timestamp reader called mrwolf.py

'''
timestrings from license/mpd urls can be decoded
For example: Channel5.com url 'expiry=1664793250' gave a 24 hour expiry time.
https://cassie.channel5.com/api/v2/licences/widevine/208/C5433700001?expiry=1664793250&tag=3565343430643230353865363735623135633561633761366232626433316433366238653239383

Channel4.com (mpd) 
https://cf.jos.c4assets.com/CH4_42_6_900_72841002001001_001/CH4_42_6_900_72841002001001_001_J01.ism/
stream.mpd?c3.ri=13631889843531360042&mpd_segment_template=time&filter=(type=="video"&&
((DisplayHeight>=288)&&(systemBitrate<4800000)))||type!="video"&
ts=1664707499&e=600&st=RTQ-kSJkavQpKB2y8vxNIQP2lyvzB3Zs2QEVFyS5kWk

gives a 'now' timestamp (ts) with 600s (10 minutes) extension (e). So yt-dlp fails if not used within time.
'''
import time

t = time.localtime()
current_time = time.strftime("%Y-%m-%d %H:%M:%S", t)



timestamp = input("Enter the timestamp \n>")
my_time = time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(int(timestamp)))


print(f"The time now is:  {current_time}")
if "1970" in my_time:
    my_time = time.strftime('%H:%M:%S', time.localtime(int(timestamp)))
    print(f"h:m:s remaining:  {my_time}")
else:
    print(f"The timestamp is: {my_time}")

It gives a 'Now' output time as a Date string and the Timestamp time or a short remaining time - depending on input. So if you see a timestamp, Mr Wolf will tell the time. Only don't expect it to say 'dinner time' and chase you!

Gotcha
Increasingly there are traps for the unwary. Pywidwvine is a Python Module now at version 1.5. it insists on installing a python-module 'protobuf 4.x.x', which breaks a lot of things.
EDIT There is now a way to deal with protobuf 3.x.x being incompatible with protobut 4.x.x see https://forum.videohelp.com/threads/409040-Correcting-Protobuf-Downgrade-to-3-19-0-error


My thread "Decryption And The Temple of Doom" showed the use of WKS-KEYS - which had its own included scripts for pywidevine.
Some code released in the forum uses pywidevine installed by Python pip as al module. The module is then available globally within your system.
Having local and global modules with the same name creates python problems; the documented python method of defining a local module seemed not to work for me. Choose one system and stick to it.
EDITIf the protobufs used in WKS-KEYS is compiled to version 4 see https://forum.videohelp.com/threads/409040-Correcting-Protobuf-Downgrade-to-3-19-0-error then pywidevine module and WKS-KEYS can exist together. Just ensure L3 is in the WKS-KEYS path - it stops confusion between pywidevine/L3/cdm and global module pywidevine/cdm path.


I tend to get lazy and on curlConverter.com select to copy the whole curl rather than just headers{}. So in my headers.py file there may be headers{}, params{} and data{}. In that case in my python code I address headers=headers.headers (and data=headers.data for the rte.ie example - otherwise not used). Simple once you know.

That is all I know!

As usual please obey netiquette and ask questions here in this thread, or forum for general queries, and not by PM

[A_n_g_e_l_a]

A generic boltdns.net downloader
An example of a generic single downloader for any? website that uses boltdns.net. There are three styles of media presentation; m3u8, mpd and vmap; this program copes with them all. No manual inputs are needed - everything comes from the clipboard paste of Stream Detector values.
The program is written in python for Linux and needs copying to a file with suggested name boltdnsnet.py . Windows users will need to add the odd .exe when the program complains -as it will.

This script will download media from boltdns.net: including STV; Acorn TV; UKTV Play; TG4; TPTV Encore, and any site that uses boltdns.net (brightcove) as its provider. I suspect that includes TVNZ but am unable to test.

#!/usr/bin/env python3
#
# v 4.4
#
# A_n_g_e_l_a 26:07:2023
# 
'''
This program is a generic boltdns.net downloader. 
It takes a single or batch of The Stream Detector's Table Entry format for mpd url,  master.m3u8 or vmap -  input as a 'media_url' 
The program downloads, decodes and merges.

Written for Linux systems using WKS-KEYS with a working CDM.
For encrypted media, this program needs to be in WKS-KEYS folder to use your local CDM
or use the online remote CDM -@@@@@@@@@@@ as configured - see inside script  @@@@@@@@@@@@@@@@@@@@@@@
the program will save to ./output/ from the WKS_KEYS folder and you need create the folder 'output' if it doesn't exist.

N_m3u8DL-RE, mkvmerge and ffmpeg need to be in your systems Path or inside the working folder.
(Windows users may need to tack .exe on the end of the called programs in the code below if N_m3u8DL-RE, mkvmerge 
and ffmpeg is  not in Windows Path but in the working folder)

It is necessary to use 'The Stream Detector' browser plugin (Chrome + Firefox)
Find here: Firefox version - https://addons.mozilla.org/en-US/firefox/addon/hls-stream-detector/
Chrome version: https://github.com/rowrawer/stream-detector/releases  choose  hls_stream_detector-2.1X.XX.crx 
and in chrome://extensions/  set Developer mode (top-right window) and drag and drop the crx file into the window to install
If that does not work on Windows for you, try this:-

 https://forum.videohelp.com/threads/403491-Is-there-a-The-Stream-Detector-like-extension-for-Chrome#post2687785


working for :-
STV.tv both encrypted and encryption free
uktvplay.co.uk all encrypted.
Acorn TV
TPTV Encore: tptvencore.co.uk mainly unencrypted m3u8 but some 'vmap' cannot be played without adverts - 
        vmap extension needs vmapper.py that cuts adverts away from the video. 
TG4.ie  all encrypted

NOTE:
Acorn TV:
This is a very badly curated site and making sense of episodes and series and video-names is difficult.
Please accept this is the best I can do with such poor input from Acorn TV
If you load each episode (in order from first to last) an attempt will be made to add sensible numbering to the videonames.
Where series and/or episode numbers exist the program will try to deal with them.
Has pretty rubbish reloution too also sometimes offer HLS Streams at 720 most content is DASH at 576p or lower.
Not worth subscribing!

Suspected working generically  at TVNZ
and possibly other providers using boltdns.net for streaming
 
'''

import requests 
from pywidevine.L3.cdm import deviceconfig
from base64 import b64encode
from pywidevine.L3.decrypt.wvdecryptcustom import WvDecrypt
import re
import subprocess
import pyfiglet as PF
from termcolor import colored
#import pyperclip as PC
import os
import math
import httpx

## globals
global ACORN 
ACORN = False 
global UKTVPLAY
UKTVPLAY = False
global STV 
STV = False
global TG4
TG4 = False
global TPTVENCORE
TPTVENCORE = False
global GENERIC
GENERIC = False

global REMOTE

#@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
# There is a choice of CDM to use
# local or remote;
# yours or someone else's.
# To use your local CDM in the WKS-KEYS
# folder, set REMOTE=False.
#
REMOTE = True
#

headers = {
    'User-Agent': 'Dalvik/2.9.8 (Linux; U; Android 9.9.2; ALE-L94 Build/NJHGGF)',
    'Accept': '*/*',
    'Accept-Language': 'en-GB,en;q=0.9',
    'Connection': 'keep-alive',
}


# regex search for pssh and license url within mpd.
# all boltdns.net conform to this spec.
def get_pssh_lic(mpd_url):

    mpd = requests.get(mpd_url, headers).text
    lines = mpd.split("\n")
    for line in lines:
        m = re.search('<cenc:pssh>(AAAA.+?)</cenc:pssh>', line)
        n = re.search('bc:licenseAcquisitionUrl=\"(http.+?)\" xmlns:bc=\"urn:brightcove:2015\"', line) 
        if m:
            pssh = m.group(1)
        if n:
            lic_url = n.group(1)

    return pssh, lic_url

## pretty print screen divisions
def divides(text):  
    #text = text
    l = len(text)
    try:
        count, lines = os.get_terminal_size()
        count = int(count)
    except:
        count = 78
    if count <= 78:
        count = int(count)
    else:
        count = int(math.ceil(count/2))
    count = count - l
    if text != 'null':   
        line = (chr(9601) * int(math.ceil(count/2)))
        #print("\n")
        print(colored(line, 'green'), " ",  text, " ",   colored(line, 'green'))  
    else:
        line = (chr(9601) * int(count + 10))
        print(colored(line, 'green'))

# REMOTE CDM or Local CDM
def WV_Function(pssh: str, lic_url: str , cert_b64=None) -> str:
    if REMOTE:      
        print("Using a remote CDM")
        headers = {
        'accept': 'application/json, text/plain, */*',
        'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36',
        }
        json = {
            'password': 'password',
            'license': lic_url,
            'headers': 'Connection: keep-alive',
            'pssh': pssh,
            'buildInfo': '',
            'cache': True,
        }
        url = 'https://wvclone.fly.dev/wv'
        with httpx.Client(headers=headers) as client:
            r = client.post(url=url, json=json).text   
            m = re.search(r">(.{32}:.{32})<", r)
            if m:
                key = m.group(1)
            print(f"Keys found {key}\n")
            return key.lstrip()  
    else:
        print("Using CDM on this machine")
        wvdecrypt = WvDecrypt(init_data_b64=pssh, cert_data_b64=cert_b64, device=deviceconfig.device_android_generic)                   
        widevine_license = httpx.post(url=lic_url, data=wvdecrypt.get_challenge(), headers=None)
        license_b64 = b64encode(widevine_license.content)
        wvdecrypt.update_license(license_b64)
        Correct, keyswvdecrypt = wvdecrypt.start_process()
        if Correct:
            mykeys = ''
            for key in keyswvdecrypt:
                mykeys += key +' '
        print(f"Keys found {mykeys}\n")
        return mykeys 
    
    

# find mpd/m3u8 in vmap
# vmap seems rare these days
def parse_vmap(content):
    lines = content.split("\n")
    for line in lines:
        m = re.search(r'contenturi="(https.+?)" contentlength=', line)
        if m:
            media_url = m.group(1)
            return(media_url)
    
    return None

# add leading zero to series or episode
def pad_number(match):
    number = int(match.group(1))
    return format(number, "02d")


if __name__ == '__main__':
    #### main #### 
    SINGLE = False
    #words or symbols to replace to make videoname shorter
    replacements = {
            'One': '1',
            'Two': '2',
            'Three': '3',
            'Four': '4',
            'Five': '5',
            'Six': '6',
            'Seven': '7',
            'Eight': '8',
            'Nine': '9',
            'Ten': '10',
            'Eleven': '11',
            'Twelve': '12',
            '\n': '',
            ':': ' ',
            '.': '',
            ',': '',
            '*': '',
            '%': '',
            '!': '',
            '(': '',
            ')': '',
            '$': '',
            '–':'', # odd u+2013 char causing errors
            '-':'', # u+20dd
            'Limited Release TV': '',
            'Extra Content': 'Extra',
            'Episode': 'E',
            'Series': 'S',
            '&': 'and'
            }
    divides('null')
    print("\nA Generic Downloader for websites using BoltDNS.net\n\n")
    print("[info] URLs may be of the form:- mpd, master.m3u8 or vmap\nfrom any site serving media from boltdns.net.\n\nIncluding:-\nSTV; Acorn TV; UKTV Play; TG4; TPTV Encore; TVNZ\n\n")
    divides('null')
    print('[info] Use Stream Detector "copy as Table Entry"')
    lines = input("Enter TSD Table Entry: ")
    #lines = input("Ready to read clipboard TSD Table Entry: ")
    #lines = PC.paste()
    lines = lines.split('\n')
    global count
    count = len(lines) # used for sequence # if required
    if count==1:
        SINGLE=True
    for line in lines:
        divides("null")
        print(line)
        linetuple = line.split('|')
        media_url = linetuple[0].replace('\n','').replace(' ', '')
        media_url = media_url.encode('ascii', 'ignore').decode('utf-8') # Windows trap for clipboard phantoms
        #print(f"[info] downloading {media_url}")

        if '6204867266001' in line  or '1486976045' in line: # stv has two idents
            STV = True
            title = PF.figlet_format(' STV ', font='smslant')
            print(colored(title, 'green'))
            divides('An STV batch downloader')
        elif '3047407010001' in line:
            ACORN = True
            title = PF.figlet_format(' ACORN TV ', font='smslant')
            print(colored(title, 'green'))
            divides('An Acorn TV batch downloader')
        elif '1242911124001' in line:
            UKTVPLAY = True
            title = PF.figlet_format(' UKTV Play', font='smslant')
            print(colored(title, 'green'))
            divides('UKTV Play a batch downloader')
        elif '1555966122001' in line:
            TG4 = True
            title = PF.figlet_format(' TG4 ', font='smslant')
            print(colored(title, 'green'))
            divides('A TG4.ie batch downloader')
        elif 'Talking Pictures TV' in line: # stv and TPTV
            TPTVENCORE = True
            title = PF.figlet_format(' TPTV ', font='smslant')
            print(colored(title, 'green'))
            divides('A Talking-Pictures-TV batch downloader')
           
        else:
            GENERIC = True
            title = PF.figlet_format(' BoltDNS dot net ', font='smslant')
            print(colored(title, 'green'))
            divides('A Generic Boltdns.net batch downloader')
        if STV: 
            videoname = (linetuple[1]+ linetuple[2])
            for rep in replacements:
                videoname = videoname.replace(rep, replacements[rep])
            # decide if episode numbers needed
            if not SINGLE: 
                if re.search(r'\d', videoname):
                    next
                else:
                    videoname = f"{count}_{videoname}"
            # format videoname 
            videoname = re.sub(r"[\s]", "_", videoname)
            videoname = videoname.lstrip('_').rstrip('_') \
                .replace('STV_Player__','').replace('__', '_')
            videoname = re.sub(r".?(Mon|Tue|Wed|Thu|Fri|Sat|Sun)", '', videoname)
            videoname = re.sub(r".?(\d.\d\d_pm)", '', videoname) 
            videoname = re.sub(r'_E_(\d+)', r'_E\1', videoname) 
        if ACORN: 
            videoname1 = linetuple[1]
            videoname2 = linetuple[2]
            myregex = re.escape(videoname1)
            if re.search(myregex, videoname2):  #duplication
                videoname = videoname2
            else:
                videoname = f"{videoname1}_{videoname2}"
            for rep in replacements:
                videoname = videoname.replace(rep, replacements[rep])
            # remove repeats
            words = videoname.split()
            videoname = (' '.join(sorted(set(words), key=words.index)))
            
            videoname = re.sub('^ S 1  ', '', videoname)
            if not SINGLE:
                videoname = re.sub(r'S (\d)', r'S\1E'+format(count, "02d"), videoname )
            if re.search(r'\d', videoname):
                next
            else:
                videoname = f"{count}_{videoname}"
                
            videoname = videoname.replace(' ','_').rstrip('_')
            print(f"The videoname captured is {videoname}")
            
           
        if UKTVPLAY:
            videoname = linetuple[1].replace('\n','').replace(' ','_')\
                .replace('_Watch_','').replace('_Online_','')
            for rep in replacements:
                videoname = videoname.replace(rep, replacements[rep])
            if videoname.endswith('_'):
                videoname = videoname.rstrip(videoname[-1])
            if videoname.startswith('_'):
                videoname = videoname[1:]
                
        if TG4:
            videoname = linetuple[1]
            for rep in replacements:
                videoname = videoname.replace(rep, replacements[rep]).lstrip().rstrip().replace(' ','_').replace('__','-')

        if TPTVENCORE:
            tp_replacements = {
                '_Limited_Release_Films': '',
                '_New_to_Encore': '',
                'Talking_Pictures_TV_with_': '',
                '_Encore_Exclusive': '',
                'Limited Release TV':'',
                'Critically Acclaimed': '',
            }
            videoname = (linetuple[1]+ linetuple[2])
            for rep in replacements:
                videoname = videoname.replace(rep, replacements[rep])
            for rep in tp_replacements:
                videoname = videoname.replace(rep, tp_replacements[rep])
            # format videoname 
            videoname = re.sub(r"[\s]", "_", videoname)
            videoname = videoname.lstrip('_').rstrip('_')
            for tp_rep in tp_replacements:
                videoname = videoname.replace(tp_rep, tp_replacements[tp_rep])
            videoname = re.sub(r".?(\d.\d\d_pm)", '', videoname) 
            videoname = re.sub(r'_E_(\d+)', r'_E\1', videoname)
            videoname = videoname.rstrip('_') 

        if GENERIC:
            videoname = linetuple[1].lstrip().rstrip().replace(' ', '_')
            for rep in replacements:
                videoname = videoname.replace(rep, replacements[rep])

        videoname = re.sub(r"(\d+)", pad_number, videoname)
        videoname = re.sub(r"S_", 'S', videoname)
        if not STV:
            videoname = re.sub("_E_", 'E', videoname)
        # move mid string S01E01 to end
        videoname = re.sub(r'^(\b\w+)(S\d{1,2}E\d{1,2})(_\w*)', r'\1\3\2', videoname)  # for clean text

        #### decide type dash (mpd),  m3u8 or vmap
        dash = "dash" #encrypted
        m3u8 = 'm3u8'
        vmap = 'vmap' #encrypted or not
        
        if vmap in media_url:
            content = requests.get(media_url, headers).text
            media_url = parse_vmap(content)  # either m3u8 or dash
        if dash in media_url:
            pssh, lic = get_pssh_lic(media_url)
            # need keys
            mykeys = '' 
            mykeys = WV_Function(pssh, lic, None)
            divides("getting encrypted streams")
            command = [
                "N_m3u8DL-RE",
                media_url,
                "--auto-select",
                '--check-segments-count=False',
                "-sv",
                "best",
                "-sa",
                "id='audio-0'",
                "-ss",
                "id='subtitles-0':for=all", 
                "--save-name",
                videoname,
                "--save-dir",
                "./output",
                "--tmp-dir",
                "./",
                "-mt",
                "--key",
                mykeys,
                "-M",
                "format=mkv:muxer=mkvmerge",
                '--no-log',
                ]
            subprocess.run(command)
        
            
        elif m3u8 in media_url: 
            #divides('Getting encryption free stream')
            command = [
            "N_m3u8DL-RE",
            media_url,
            "--binary-merge",
            "--auto-select",
            "-sv",
            "best",
            "-sa",
            "id='audio-0'",
            "-ss",
            "id='subtitles-0':for=all",
            "--save-name",
            videoname,
            "--save-dir",
            "./output/",
            "--tmp-dir",
            "./",
            "-mt",
            "-M",
            "format=mkv:muxer=mkvmerge",
            '--no-log',
            ]
            subprocess.run(command)
        count -= 1  # used for sequence preface number in videoname for some providers (Acorn TV)
        os.system("rm -f ./output/*.m4a  ./output/*.srt ./output/*.mp4 ./output/*.ts") 
    os.system("reset") # N_m3u8DL-RE sometimes leaves terminal unable to print text
    exit(0)

EDIT Feb 2024
The above has the option of using a remote online CDM to get keys. Only it is now offline. And the script didn't play nicely on Windows.
An update is here https://files.videohelp.com/u/301890/boltgeneric.py

An ITVX Batch or Single Downloader
A batch downloader for ITVX using N_m3u8DL-RE - it is faster than yt-dlp; is uses table entries from The Stream Detector. No manual inputs are required.

## This program uses The Stream Detector to capture page URLs
## Stream Detector select 'options'
## in the box adjacent to'user-defined-commands enter -->  %origin%
## In TSD window select copy stream url as User-Defined-Command 1

## In addition to not have to faff about with opening and saving text files
## this program downloads, converts and merges subtitles.

## added option for a sequence number to add to the videoname (This will be the order they
## are selected in TSD). It removes the chance of an over-write of the video name when ITV
## uses a generic title like 'Inspector Morse' and without a series or episode number. 

## Subtitle conversion now uses ffmpeg


import re
import requests
import subprocess
from base64 import b64encode
from pathlib import Path
import httpx
from httpx import URL, Client
from selectolax.lexbor import LexborHTMLParser
import os
import  pyperclip as PC
from pywidevine.L3.cdm import deviceconfig
from pywidevine.L3.decrypt.wvdecryptcustom import WvDecrypt
import pyfiglet as PF
from termcolor import colored



# GLOBALS
OUT_PATH = Path('output')
OUT_PATH.mkdir(exist_ok=True, parents=True)
global count
global SEQ
################
# 
# NOTE
# seting for index number to preface videoname either True or False
# If you want each video in your clipboard to be numbered by a preface to
# the videoname set the value of SEQ = TRUE
#
################
SEQ = False

# Class configured as a singleton 
# only one instance is created
# if consructor called again original
# instance is returned
class ITV:
    _instance = None

    def __init__(self):
        raise RuntimeError('Call instance() instead')

    @classmethod
    def instance(cls):
        if cls._instance is None:
            print('Creating new instance of ITV Class')
            cls._instance = cls.__new__(cls)
            cls.host = 'itvpnpdotcom.blue.content.itv.com'
            cls.wv = 'https://cdrm-project.com/wv'
            timeout = httpx.Timeout(10.0, connect=60.0)

            cls.client = Client(
            headers={
                'authority': 'www.itv.com',
                'user-agent': 'Dalvik/2.9.8 (Linux; U; Android 9.9.2; ALE-L94 Build/NJHGGF)',
            },
            timeout=timeout,
        )
        return cls._instance


    def download(self, url: str) -> None:
        global count
        title, data = self.get_data(url)
    
        video = data['Playlist']['Video']
        media = video['MediaFiles']
        illegals = "*'%$!(),:;-"
        # replace  extraneous title data and 'illegal' characters
        videoname = ''.join(c for c in title if c.isprintable() and c not in illegals)    
        videoname = videoname.replace('\n','').replace('ITVX','').replace('otherepisodes','_extra').replace('  Series ','_S')\
            .replace('  Episode ','E').replace(' ','_')\
                .replace('&','and').lstrip('_').rstrip('_')
        videoname = re.sub(r"(\d+)", pad_number, videoname)
        subs_url = video['Subtitles'][0]['Href']
        print (subs_url)
        subs = self.client.get(subs_url)
        f = open(f"./{videoname}.subs.vtt", "w")
        subtitles = subs.text
        f.write(subtitles)
        f.close()

        # convert subtitles 
        #os.system(f"tt convert  -i {videoname}.subs.vtt -o {videoname}.subs.srt > /dev/null 2>&1")
        os.system(f"ffmpeg -loglevel quiet -hide_banner -i ./{videoname}.subs.vtt  ./{videoname}.subs.srt > /dev/null 2>&1")

        global SEQ  # prepend a sequence number to anonymous videos
        if SEQ:
            myvideoname = format(count, "02d") +'_'+ videoname
        else:
            myvideoname = videoname
        mpd_url = [f'{video["Base"]}{y}' for x in media if (y := URL(x['Href'])).path.endswith('.mpd')][0]
        lic_url = [x['KeyServiceUrl'] for x in media][0]

        pssh = self._get_pssh(mpd_url)
        key = self._get_key(pssh, lic_url)
        temp = URL(mpd_url).params['hdnea']
        temp = temp.replace('nohubplus', 'hdntl,nohubplus')
        cookie = f"cookie: {re.sub(r'^.*(?<=exp=)', 'hdntl=exp=', temp)}"

        m3u8dl = 'N_m3u8DL-RE'   # windows rename with .exe added
        subprocess.run([
            m3u8dl,
            mpd_url,
            '--append-url-params',
            #'--header',
            #cookie,
            #'--header',
            #f'host: {self.host}',
            #'--header',
            #f'user-agent: {self.client.headers["user-agent"]}',
            '--auto-select',
            '--save-name',
            myvideoname,
            '--save-dir',
            './',
            '--tmp-dir',
            './',
            '-mt',
            '--key',
            key,
            '-M',
            'format=mp4',
            '--no-log'
        ])
        
        command = [
        "mkvmerge",   # windows remame with .exe added
        "-q",
        f"{myvideoname}.mp4",
        f"{videoname}.subs.srt",  
        "-o",
        f"{myvideoname}.mkv"
        ]
        subprocess.run(command)
        subprocess.run(["mv", f"{myvideoname}.mkv",  f"{OUT_PATH}"])
        #os.system(f"rm {myvideoname}.mp4 {videoname}.subs.vtt {videoname}.subs.srt")
        count = count-1

    def get_data(self, url: str) -> tuple:
        r = self.client.get(url)
        html = LexborHTMLParser(r.text)
        try:
            title = html.css_first('title').text()
        except:
            print("Error. No html returned.\n\nCHECK you have the PAGE urls copied to clipboard")
            exit(0)
        magni_url = html.css_first('[data-video-id]').attributes.get('data-video-id')
        features = ['mpeg-dash', 'widevine', 'outband-webvtt', 'hd', 'single-track']
        payload = {
            'client': {'id': 'browser'},
            'variantAvailability': {
                'featureset': {'min': features, 'max': features},
                'platformTag': 'dotcom',
            }
        }
        r = self.client.post(magni_url, json=payload)
        return title, r.json()
    
    
    # Choose which def _get-key() to use 
    # YOU DO NOT NEED YOUR OWN CDM for this option
    # this one uses a remote CDM at cdrm-project.com
    # your IP address will be logged by them as having used their service
    # NOT A GOOD IDEA FOR LONG TERM USE
    
    '''def _get_key(self, pssh: str, lic_url: str) -> str:
        print("Using CDM at cdrm-project.com")
        payload = {
            'license': lic_url,
            'headers': 'connection: keep-alive',
            'pssh': pssh,
            'buildInfo': '',
            'proxy': '',
            'cache': True,
        }
        r = self.client.post(self.wv, json=payload)
        return LexborHTMLParser(r.text).css_first('li').text()
    '''


    # This version uses WKS-KEYS's and your own CDM to key fetch keys.
    # YOU NEED YOUR OWN CDM to use this method. 
    # The video site sees the key request as if from a normal web user.
    
    def _get_key(self, pssh, lic_url, cert_b64=None):
        print("Using CDM on this machine")
        wvdecrypt = WvDecrypt(init_data_b64=pssh, cert_data_b64=cert_b64, device=deviceconfig.device_android_generic)                   
        widevine_license = httpx.post(url=lic_url, data=wvdecrypt.get_challenge(), headers=None)
        license_b64 = b64encode(widevine_license.content)
        wvdecrypt.update_license(license_b64)
        Correct, keyswvdecrypt = wvdecrypt.start_process()
        if Correct:
            mykeys = ''
            for key in keyswvdecrypt:
                mykeys += key +' '
        print(f"Keys found {mykeys}\n")
        return mykeys 
    
    def _get_pssh(self, mpd_url: str) -> str:
        r = self.client.get(mpd_url)
        kid = (
            LexborHTMLParser(r.text)
            .css_first('ContentProtection')
            .attributes.get('cenc:default_kid')
            .replace('-', '')
        )
        s = f'000000387073736800000000edef8ba979d64acea3c827dcd51d21ed000000181210{kid}48e3dc959b06'
        return b64encode(bytes.fromhex(s)).decode()

# add leading zero to series or episode
def pad_number(match):
    number = int(match.group(1))
    return format(number, "02d")

def main() -> int:
    input("Press enter with PAGE urls in clipboard ")
    urls = PC.paste().split('\n')  
    global count 
    count = len(urls) 
    print(f"The URL list has {count} video(s)")
    myITV = ITV.instance()
    for url in urls:
        myITV.download(url)
    return 0

if __name__ == "__main__":
    title = PF.figlet_format(' I T V X ', font='smslant')
    print(colored(title, 'green'))
    main()

An RTE.ie Downloader - an example of case 4 above.

# -*- coding: utf-8 -*-
# for WKS-KEYS
# Created on: 10.12.2021
# Modified to clipboard cURL 04:01:2023
# Modified to prevent N_m3u8DL-RE bug counting incorrect fragments 22:03:2023
#

'''
reads from clipboard to parse cURL of license
search F12 -> Network tab for 'widevine' or modular to get license

Uses The Stream Detector plugin to detect the mpd. Use- 'Save as Table Entry'
Copy licence as cURL in developer tools  - filter on 'wide'

copy the following into a file called requirements.txt

pyfiglet==1.0.2
pyperclip==1.8.2
pywidevine==1.8.0
Requests==2.31.0
termcolor==2.4.0
uncurl==0.0.11
xmltodict==0.13.0

the from the same folder as RET.py and requiremnts.txt run

pip install pipreqs

Afer install of pipreqs run

pip insrall -r requirements.txt

'''

import requests
import json
from base64 import b64encode
#from getPSSH import get_pssh
from pywidevine.L3.cdm import deviceconfig
from pywidevine.L3.decrypt.wvdecryptcustom import WvDecrypt
import pyperclip as PC
import uncurl
import shutil
from collections import OrderedDict
import subprocess
from termcolor import colored
import pyfiglet as PF
import os
import math
import xmltodict

headers = {
    'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:108.0) Gecko/20100101 Firefox/108.0',
    'Accept': '*/*',
    'Accept-Language': 'en-GB,en;q=0.5',
    'content-type': 'text/plain;charset=UTF-8',
    'Origin': 'https://www.rte.ie',
    'DNT': '1',
    'Connection': 'keep-alive',
    'Sec-Fetch-Dest': 'empty',
    'Sec-Fetch-Mode': 'cors',
    'Sec-Fetch-Site': 'cross-site',
}

def WV_Function(pssh, lic_url, cert_b64=None):
	"""main func, emulates license request and then decrypt obtained license
	fields that change every new request is signature, expirationTimestamp, watchSessionId, puid, and rawLicenseRequestBase64 """
	wvdecrypt = WvDecrypt(init_data_b64=pssh, cert_data_b64=cert_b64, device=deviceconfig.device_android_generic)                   
	raw_request = wvdecrypt.get_challenge()
	request = b64encode(raw_request)
	# rte.ie support
	# context from uncurl
	myjson = json.loads(context.data)
	pid = myjson['getWidevineLicense']['releasePid']

	responses.append(requests.post(url=lic_url, headers=headers,
		json={
			"getWidevineLicense": 
				{
				'releasePid': pid,
				'widevineChallenge': str(request, "utf-8" )
				}, 
				}))

	for i, response in enumerate(responses):
		try:
			str(response.content, "utf-8")
		except UnicodeDecodeError:
			widevine_license = response
			print(f'{chr(10)}license response status: {widevine_license}{chr(10)}')
			break	
		else:
			if len(str(response.content, "utf-8")) > 500:
				widevine_license = response
				print(f'{chr(10)}license response status: {widevine_license}{chr(10)}')
				break
		if i == len(responses) - 1:
			print(f'{chr(10)}license response status: {response}')
			print(f'server reports: {str(response.content, "utf-8")}')
			print(f'server did not issue license, make sure you have correctly pasted a decoded cURL of the license url (see curlconverter.com) to headers.py.{chr(10)}')
		
	lic_field_names = ['license', 'payload', 'getWidevineLicenseResponse']
	lic_field_names2 = ['license']
	
	#open('license_content.bin', 'wb').write(widevine_license.content)

	try:
		if str(widevine_license.content, 'utf-8').find(':'):
			for key in lic_field_names:
				try: 
					license_b64 = json.loads(widevine_license.content.decode())[key]
				except:
					pass			
				else:
					for key2 in lic_field_names2:
						try: 
							license_b64 = json.loads(widevine_license.content.decode())[key][key2]
						except:
							pass
		else:
			license_b64 = widevine_license.content								
	except:
		license_b64 = b64encode(widevine_license.content)

	wvdecrypt.update_license(license_b64)
	Correct, keyswvdecrypt = wvdecrypt.start_process()
	if Correct:
		return Correct, keyswvdecrypt 

def get_pssh(mpd_url):
    pssh = ''
    try:
        r = requests.get(url=mpd_url)
        r.raise_for_status()
        xml = xmltodict.parse(r.text)
        mpd = json.loads(json.dumps(xml))
        periods = mpd['MPD']['Period']
    except Exception as e:
        pssh = input(f'\nUnable to find PSSH in MPD: {e}. \nEdit getPSSH.py or enter PSSH manually: ')
        return pssh
    try: 
        if isinstance(periods, list):
            for idx, period in enumerate(periods):
                if isinstance(period['AdaptationSet'], list):
                    for ad_set in period['AdaptationSet']:
                        if ad_set['@mimeType'] == 'video/mp4':
                            try:
                                for t in ad_set['ContentProtection']:
                                    if t['@schemeIdUri'].lower() == "urn:uuid:edef8ba9-79d6-4ace-a3c8-27dcd51d21ed":
                                        pssh = t["cenc:pssh"]
                            except Exception:
                                pass   
                else:
                    if period['AdaptationSet']['@mimeType'] == 'video/mp4':
                            try:
                                for t in period['AdaptationSet']['ContentProtection']:
                                    if t['@schemeIdUri'].lower() == "urn:uuid:edef8ba9-79d6-4ace-a3c8-27dcd51d21ed":
                                        pssh = t["cenc:pssh"]
                            except Exception:
                                pass   
        else:
            for ad_set in periods['AdaptationSet']:
                    if ad_set['@mimeType'] == 'video/mp4':
                        try:
                            for t in ad_set['ContentProtection']:
                                if t['@schemeIdUri'].lower() == "urn:uuid:edef8ba9-79d6-4ace-a3c8-27dcd51d21ed":
                                    pssh = t["cenc:pssh"]
                        except Exception:
                            pass   
    except Exception:
        pass                      
    if pssh == '':
        pssh = input('Unable to find PSSH in mpd. Edit getPSSH.py or enter PSSH manually: ')
    return pssh
		
## pretty print screen divisions
def divides(text):  
    #text = text
    l = len(text)
    count, lines = os.get_terminal_size()
    count = int(count)
    if count <= 78:
        count = int(count)
    else:
        count = int(math.ceil(count/2))
    count = count - l
    if text != 'null':   
        line = (chr(9601) * int(math.ceil(count/2)))
        #print("\n")
        print(colored(line, 'green'), " ",  text, " ",   colored(line, 'green'))  
    else:
        line = (chr(9601) * int(count + 10))
        print(colored(line, 'green'))



#### main and globals ####
currentFile = __file__
cachePath = "./output/cache" # may need to create or modify paths
outputPath = "./output"
divides("null")
title = PF.figlet_format(' R T E  ', font='smslant')
print(colored(title, 'green'))
divides("RTE Downloader: using mpd and license cURL")
input('Ready to read mpd as TABLE ENTRY from clipboard \nEnter when ready.')
mpd_str = PC.paste().split('|')
mpd_url = mpd_str[0]   
filename = mpd_str[1].replace(' - RTÉ Player ','').replace(' ','_').lstrip('_')

divides('Ready to read clipboard')
input('Curl of license: Enter when ready')
mycode = PC.paste()
context = uncurl.parse_context(mycode)
lic_url = context.url
myjson = json.loads(context.data)
releasePID = (myjson['getWidevineLicense']['releasePid'])
od = OrderedDict(context.headers) ## cast was needed.
headers = od # that's all
responses = []
license_b64 = ''
pssh = get_pssh(mpd_url)

divides('PSSH')
print(f'{chr(10)}PSSH obtained.\n{pssh}')
correct, keys = WV_Function(pssh, lic_url)
mykeys=""
f = open("keys.txt", "w")
for key in keys:
	mykeys += '--key ' + key + '\n'
	f.write(key + '\n')
divides('KEYS')	
print(mykeys)
f.close()

''' subs, if available, now found by N_m..RE

## prepare subtitles url
domain = ( mpd_url.split('.ism')[0])
part2 =  (domain.split('/')[-1])
subs_url = domain + '.ism/dash/' + part2 + '-textstream_eng=1000.vtt'
print('getting subs from: ' + subs_url)
os.system(f' curl -s -o {outputPath}/{filename}.subs.vtt {subs_url}')
'''
command = [
    "N_m3u8DL-RE",
    mpd_url,
    "--check-segments-count=False",
	    "-sv",
        "best",
        "-sa",
        "best",
        "-ss",
        "id='textstream_eng_1=1000':for=all",
    "--save-name",
    filename,
    "--save-dir",
    "./output",
    "--tmp-dir",
    "./",
    "-mt",
    "--use-shaka-packager",
    "--key-text-file",
    "keys.txt",
    "-M",
    "format=mkv:muxer=mkvmerge",
]
subprocess.run(command)

exit(0)

[codehound]

As usual, very comprehensive and excellently written.

[jack_666]

@A_n_g_e_l_a
Thanks for sharing.

[cedric8528]

The Bible!

Thank you so much A_n_g_e_l_a for your work

[basedcel]

bruh

[elfoeda]

Thanks!

Greetings from Brazil!

[[ss]vegeta]

Valuable read. Thanks Angela and good job.

[Quint]

Very nice, especially the HTTPToolkit hint!
Do I get this right: The whole json-thing is an additional request before getting the license itself only to get the token that is used later?

Also, what I didn't understand up to now (still a bit new to Python...):
For the license request you have to copy the request parameters all in one, convert it all to Python, put it all into headers.py (WKS-KEYS), ok.
I did this several times and up to now everything worked fine. Also if there had been additonal data:

headers=
{
blah
}
data='extra blah'

And sometimes there is a lot of data to pass! But in l3.py "headers.data" is not used in any way!
Does that mean the data is unnecessary? As far as I remember when I used getwvkeys.cc (while it was still normal there) it did NOT work without the data.

[ampersand]

nice, more valuable info to digest

one question about headers.py - if l3.py is only taking headers and data from it, can the "response = requests.post(...)" line be removed?

[A_n_g_e_l_a]

nice, more valuable info to digest

one question about headers.py - if l3.py is only taking headers and data from it, can the "response = requests.post(...)" line be removed?

You misunderstand; headers.py is only supportive of the request and passed as an extra field in the request method

widevine_license = requests.post(url=lic_url, data=wvdecrypt.get_challenge(), headers=headers)

That line sends the licence url; data created from sending the pssh to our CDM (the challenge); and the headers. If you delete the request there will be be no response for the CDM to decrypt and produce keys.

What you can do for sites that do not appear to read headers is to set 'headers=None' in that request line.

I just tried it with uktvplay.co.uk and it works without headers being sent.

[A_n_g_e_l_a]

Do I get this right: The whole json-thing is an additional request before getting the license itself only to get the token that is used later?

Yes.

sometimes there is a lot of data to pass! But in l3.py "headers.data" is not used in any way!
Does that mean the data is unnecessary? As far as I remember when I used getwvkeys.cc (while it was still normal there) it did NOT work without the data.

If the data is on the url we have no need. I cannot be definitive that data=headers.data is an unnecessary field. It may depend on the site - but use your eyes and see what the browser headers in the request have. We mimic remember.

I have no idea what wvgetkeys.cc were doing.

[Quint]

I cannot be definitive that data=headers.data is an unnecessary field. It may depend on the site...

The thing is, that "headers.data" is not used at all in l3.py. That confuses me a bit.
In l3.py in the license request "data" is passed, but it's taken always from the CDM.

[A_n_g_e_l_a]

The thing is, that "headers.data" is not used at all in l3.py. That confuses me a bit.
In l3.py in the license request "data" is passed, but it's taken always from the CDM.

Forget the headers 'data' field. Who know what the author of l3 was thinking. It may be used somewhere, but for the small number of sites I frequent, I haven't seen it used. And the way I have seen addressed it wouldn't get found anyway ''data=data', params=params' is meaningless.

Python allows multiple uses of the same label so long as it can understand the context. So 'data' in licence challenge is different from 'data' in headers.

My l3.py has this line:

widevine_license = requests.post(url=lic_url, data=wvdecrypt.get_challenge(), headers=headers.headers)

(Remember my headers.py is a total copy from curlconverter with data{..} and params{...} I need headers.headers to clarify the import.)

I am an experimenter by nature. The best way to find out is to try. So if you think there are too many data fields, drop one and see what happens.

Everyone: I really don't want this to become a tutorial or an individual coaching session. Read, analyze and experiment. It is such a better feeling when you get there yourself.

[Quint]

The thing is, that "headers.data" is not used at all in l3.py. That confuses me a bit.
In l3.py in the license request "data" is passed, but it's taken always from the CDM.

Forget the headers 'data' field. Who know what the author of l3 was thinking. It may be used somewhere, but for the small number of sites I frequent, I haven't seen it used. And the way I have seen addressed it wouldn't get found anyway ''data=data', params=params' is meaningless.

I fear we misunderstood each other, may be my English again. "headers.data" isn't addressed AT ALL anywhere. So if you put it in headers.py it only can be obsolete - at least in WKS-KEYS.

My l3.py has this line:

widevine_license = requests.post(url=lic_url, data=wvdecrypt.get_challenge(), headers=headers.headers)

(Remember my headers.py is a total copy from curlconverter with data{..} and params{...} I need headers.headers to clarify the import.)

Yes, I got that. I developed software for more than 30 years, just long before python, but it doesn't seem that complicated.

I am an experimenter by nature. The best way to find out is to try. So if you think there are too many data fields, drop one and see what happens.

Yes, I did, because I am still of the same kind as you are - maybe a bit less strong than in my younger days...
And learned that the license request in l3.py
widevine_license = requests.post(url=lic_url, data=wvdecrypt.get_challenge(), headers=headers.headers)

refers to wvdecrypt.py:
def get_challenge(self):
return self.cdm.get_license_request(self.session)

that refers to cdm.py:
def get_license_request(self, session_id):
Blah... (complicated)

So the "data" comes from the CDM each time new. The question that confuses me is still: Why is it not simply taken from the "data" in headers.py IF specified there. I am sure there is a good reason (the reason that explains everything this is all about, I fear), but I still don't get the point.
But you are very right: Time to learn and experiment!

[A_n_g_e_l_a]

So the "data" comes from the CDM each time new. The question that confuses me is still: Why is it not simply taken from the "data" in headers.py IF specified there. I am sure there is a good reason (the reason that explains everything this is all about, I fear), but I still don't get the point.
But you are very right: Time to learn and experiment!

The Wvdecrypt object is initilized with pssh and data about the CDM. So when a request to the license server is made this information is included. Headers.data is irrelevant - forget it, it doesn't get used. Any anyway it is the browser's request headers and nothing to do with our license request from python.

l3.py, as supplied, imports cdm erroneously; it never gets called directly in my l3.py and I have deleted that part of the import. I haven't had need to look in any detail beyond l3.py

[ampersand]

That line sends the licence url; data created from sending the pssh to our CDM (the challenge); and the headers. If you delete the request there will be be no response for the CDM to decrypt and produce keys.

I was talking about the the "response = requests.post(...)" line from headers.py:

response = requests.post('https://cassie.channel5.com/api/v2/licences/widevine/208/C5273420001', params=params, headers=headers, data=data)

I don't see this response used anywhere

[A_n_g_e_l_a]

That line sends the licence url; data created from sending the pssh to our CDM (the challenge); and the headers. If you delete the request there will be be no response for the CDM to decrypt and produce keys.

I was talking about the the "response = requests.post(...)" line from headers.py:

response = requests.post('https://cassie.channel5.com/api/v2/licences/widevine/208/C5273420001', params=params, headers=headers, data=data)

I don't see this response used anywhere

OK, sorry for my misunderstanding I think I'm with you now and I believe Quint may be making a similar point.

Curlconverter.com converts the licence request from the browser And the request carries data (encrypted) about the browser's CDM in 'data'. That is no use to us. For we need to make a request relevant to our CDM but we do need the bit headers={...} in headers.py.
Because I'm lazy and it is easier to copy curlconverter.com's output in its entirety, that is what I choose to do. But the request in there is never called.

The request.post line in the headers is similar to to the one we use. Only we are asked to input license in l3 py and it is then no longer "https://cassie.channel5.com/api/v2/licences/widevi.." but lic_url as it is used here:-

widevine_license = requests.post(url=lic_url, data=wvdecrypt.get_challenge(), headers=headers.headers)

Try and separate out in your mind what the browser does, and what we mimic with l3.py All we take from the browser is the headers field, the license url and the mpd url and nothing else. I too stumbled here; separating browser process and l3.py process; I mentioned it in my post.

[Quint]

Headers.data is irrelevant - forget it, it doesn't get used.

Ehm... - yes. That's what I pointed out. I only asked, why.

Any anyway it is the browser's request headers and nothing to do with our license request from python.

Yes. And it seems interesting to me, WHY it is not possible to simply use it.

l3.py, as supplied, imports cdm erroneously; it never gets called directly in my l3.py and I have deleted that part of the import. I haven't had need to look in any detail beyond l3.py

I didn't say that l3 imports cdm, I said l3 uses wvdecrypt which uses cdm.
But it seems you don't like to discuss, so will find out myself and don't disturb any longer.
Thanks a lot by the way for your other great posting, that let me learn how to dump the above!

[A_n_g_e_l_a]

Any anyway it is the browser's request headers and nothing to do with our license request from python.

Yes. And it seems interesting to me, WHY it is not possible to simply use it.

I make the point in my response to ampersand

Try and separate out in your mind what the browser does, and what we mimic with l3.py All we take from the browser is the headers field, the license url and the mpd url and nothing else. I too stumbled here; separating browser process and l3.py process; I mentioned it in my post.

Using what the browser sends for headers saves the need to programmatically wade through obfuscated javascripts from the CDN to assemble what is needed.

If it is unchanging data, every time we try some episode download in the browser, then we know it copyable for our python use.

If it has a messsage = <some code in base64> that will need to change ,from what the browser sends to what our cdm sends. Try decoding some of the base64 code from browser and CDM - use Httptoolkit to watch the terminal - watch the browser and see the differences in message contents.

To say headers.data is 'never used' is not quite true; I have made an edit to point 4 dealing with the Option and the RTE.ie example. If its on the url no need to be in headers.data is probably a safer way of saying it.

[Quint]

If it is unchanging data, every time we try some episode download in the browser, then we know it copyable for our python use.

Yes, there are constant and changing things f. e. a token that worked yesterday and has to be "renewed" today, and constant parts.
Seems that the "final" call of "get_license_request" must be unique each time, because the response has to be decrypted in the context that started it. Still don't get the point of this exactly. But thanks for your answer.

[A_n_g_e_l_a]

If it is unchanging data, every time we try some episode download in the browser, then we know it copyable for our python use.

Yes, there are constant and changing things f. e. a token that worked yesterday and has to be "renewed" today, and constant parts.
Seems that the "final" call of "get_license_request" must be unique each time, because the response has to be decrypted in the context that started it. Still don't get the point of this exactly. But thanks for your answer.

At the root of it all is SSL key-sharing, public/private keys and pretty good privacy . CDM keys and Browser keys are different. So the messages can only be read by one decryptor with the correct private key.

As I see the process: our CDM sends data to the server which includes pssh and our CDM's public key and data about our CDM - serial nos etc. It is not encrypted in this direction - only encoded base64, The server checks the pssh finds the decryption-keys for the particular media and encrypts those using our public key and sends the encrypted data back. The CDM decrypts the data with its private key to reveal the media decryption keys.

[Quint]

And thus the browser data is useless because in the end you would need the browser's private key to decrypt the answer. And the Decrypter/-guesser used chrome's private key which is now totally obfuscated. I begin to understand, thanks for the explanation.

[yawoo]

Thanks for your comprehensive guide! Finally I can obtain the key from the case 4 with the constructed json packet.

[celsoricardoweb]

Amazing!!

[k2000]

Hello Angela

tries to understand your famous step 4 JSON. At step of httptoolkit your code obtained you use with which terminal httptoolkit or windows CMD?.

if use the terminal of httptolkit and enter the code after I cannot enter any command.

[A_n_g_e_l_a]

Hello Angela

tries to understand your famous step 4 JSON. At step of httptoolkit your code obtained you use with which terminal httptoolkit or windows CMD?.

if use the terminal of httptolkit and enter the code after I cannot enter any command.

For step 4 you really need to construct some python code that makes the request that sends the json packet. I pointed to work by medvm; try adapting his code to your needs.
And on windows you would be runnng that via a windows CMD window.

[k2000]

Hello Angela

tries to understand your famous step 4 JSON. At step of httptoolkit your code obtained you use with which terminal httptoolkit or windows CMD?.

if use the terminal of httptolkit and enter the code after I cannot enter any command.

For step 4 you really need to construct some python code that makes the request that sends the json packet. I pointed to work by medvm; try adapting his code to your needs.
And on windows you would be runnng that via a windows CMD window.

Thank you I don't really know how to do this part in python, do you have more info for beginners in python please.
on the site you mentioned step 4 manages to have the keys my on another site with JSON how to know all this info to change?.

[deccavox]

An awful lot for me to take it since I'm still learning how all these processes work. But, exceptionally informative and it has given me far more insight into the inner workings of these processes than anything else I've found on line. Superb information. A huge thank you.

[deccavox]

As I said above, superb explanation as to how each of those webplayers respond between client and server. I've learnt a lot from that. Now, how to tranlate that information to get a key is way beyond me at the moment, but at least we have the help of the various packages that have been discussed for over a year now like WKS-KEYS and their derivatives.

Now, I have studied various tutorials (including YT) describing how to scrape actual web pages (eg. by using requests and beautifulsoup). But that's not what I want to do. What I would dearly love to be guided with is information on how to get information that's extracted from the various requests and responses as displayed in Developer Tools in Chrome browser. In particular things like mpd URL and license URL.

This would save me having to input those values manually. I want to write a script where I input the main page URL and that information like mpd url & licence url are extracted using python.

I just cannot find anything along those lines on the web.

Is it possible that anybody could guide me? I'm sure that others would find that useful as well. I'm not suggesting you hand me out a freebie, just some guidance or useful sites to get the information from would be superb.

Many thanks.