Best ITVX Downloader with Subtitles

[PhilipG]

I am trying to download this using the script in the first post but I had to comment out the following lines as i was getting the error "ModuleNotFoundError: No module named 'pywidevine.L3"

#from pywidevine.L3.cdm import deviceconfig
#from pywidevine.L3.decrypt.wvdecryptcustom import WvDecrypt

When I use the url as captured by the Streamdetector

https://www.itv.com/watch/king-of-the-roads/2a5089/2a5089a0062

I get the following shown in the CMD window


[Attachment 72810 - Click to enlarge]

Can anyone help me out as I have no clue.

I have the same problem when switching between this and the C4 downloader.

to use the ITVX etc scripts I run

pip uninstall pywidewine
pip install 'protobuf==3.20.3'

then, before using the C4 downloader again I run

pip install pywidevine

The solutions posted by others have not worked for me (or, more likely, I haven't properly understood the instructions). But the above is simple enough for me to do every time it's needed.

[A_n_g_e_l_a]

The solutions posted by others have not worked for me (or, more likely, I haven't properly understood the instructions). But the above is simple enough for me to do every time it's needed.

Thank you but none of that is relevant to the this issue if you read the error message.

[A_n_g_e_l_a]

Thank you A_n_g_e_l_a unfortunately my schooling was finished by 1987!

And mine ended 20 year earlier. But I had 'hello World!' scrolling too on a model B. I learnt BBC Basic and programmed for the number of rolls of wallpaper to decorate a room! I never looked back. Eventually used a Beeb to collect NOAA satellite data to display a '4 colour weather map' of the UK - (black, white,red and yellow) - all from a kit. Magic times.

[PhilipG]

The solutions posted by others have not worked for me (or, more likely, I haven't properly understood the instructions). But the above is simple enough for me to do every time it's needed.

Thank you but none of that is relevant to the this issue if you read the error message.

The issue is that installing pywidevine (needed for the c4 script) somehow mucks up the WKS-KEYS install. That much is clear to me.

It can't find pywidevine/L3? Well, installing pywidevine did not change the folder structure one jot. It's still there as a path relative to the script. __init__.py is still inside L3. I put a copy of __init__.py in the pywidevine folder and that made no difference. Does installing pywidevine stop it looking through relative paths for imports? No idea, my Python is extremely basic.

The best solution would be for someone to update either the c4 script to make it use WKS_KEYS, or the other scripts to use the .wvd generated from the L3 keys. But that someone is not going to be me while the quick workaround is available!

[A_n_g_e_l_a]

The solutions posted by others have not worked for me (or, more likely, I haven't properly understood the instructions). But the above is simple enough for me to do every time it's needed.

Thank you but none of that is relevant to the this issue if you read the error message.

The issue is that installing pywidevine (needed for the c4 script) somehow mucks up the WKS-KEYS install. That much is clear to me.

It can't find pywidevine/L3? Well, installing pywidevine did not change the folder structure one jot. It's still there as a path relative to the script. __init__.py is still inside L3. I put a copy of __init__.py in the pywidevine folder and that made no difference. Does installing pywidevine stop it looking through relative paths for imports? No idea, my Python is extremely basic.

The best solution would be for someone to update either the c4 script to make it use WKS_KEYS, or the other scripts to use the .wvd generated from the L3 keys. But that someone is not going to be me while the quick workaround is available!

Philip, you misunderstand. Bernard's issue has nothing to do with WKS-KEYS vs pywidewine believe me.

And for your information for the future Pythons's pywidevine plays very nicely with WKS-KEYS if you just recompile wv_proto2_pb2 to suit version 4 protobuf. There are instructions elsewhere in the forum. I understand you have currently chosen not to follow them.

[PhilipG]

The solutions posted by others have not worked for me (or, more likely, I haven't properly understood the instructions). But the above is simple enough for me to do every time it's needed.

Thank you but none of that is relevant to the this issue if you read the error message.

The issue is that installing pywidevine (needed for the c4 script) somehow mucks up the WKS-KEYS install. That much is clear to me.

It can't find pywidevine/L3? Well, installing pywidevine did not change the folder structure one jot. It's still there as a path relative to the script. __init__.py is still inside L3. I put a copy of __init__.py in the pywidevine folder and that made no difference. Does installing pywidevine stop it looking through relative paths for imports? No idea, my Python is extremely basic.

The best solution would be for someone to update either the c4 script to make it use WKS_KEYS, or the other scripts to use the .wvd generated from the L3 keys. But that someone is not going to be me while the quick workaround is available!

Philip, you misunderstand. Bernard's issue has nothing to do with WKS-KEYS vs pywidewine believe me.

And for your information for the future Pythons's pywidevine plays very nicely with WKS-KEYS if you just recompile wv_proto2_pb2 to suit version 4 protobuf. There are instructions elsewhere in the forum. I understand you have currently chosen not to follow them.

Okay, looks like I missed that detail about wv_proto2_pb2...

edit: found your post at https://forum.videohelp.com/threads/409040-Correcting-Protobuf-Downgrade-to-3-19-0-err...=wv_proto2_pb2

On my mac I needed to uninstall protobuf with

pip uninstall protobuf

then reinstall via HomeBrew

brew install protobuf

for protoc to install. Tested, now all fixed for me. Thanks!

[Fallito]

I am having this error. Any ideas/help will be welcome. Thanks. It gives me the same error with both my cdm or the remote one.

[A_n_g_e_l_a]

Are you using the address bar URLs?

[stabbedbybrick]

ITV have changed their JSON structure, so you get KeyError: "title" because it doesn't exist. The script needs to be edited to match the new structure.

[Fallito]

Are you using the address bar URLs?

Yes. This is from "A Spy Among Friends", 6 episode serie taken from the copy url button in Stream Detector.

https://www.itv.com/watch/a-spy-among-friends/2a7931/2a7931a0006
https://www.itv.com/watch/a-spy-among-friends/2a7931/2a7931a0005
https://www.itv.com/watch/a-spy-among-friends/2a7931/2a7931a0004
https://www.itv.com/watch/a-spy-among-friends/2a7931/2a7931a0003
https://www.itv.com/watch/a-spy-among-friends/2a7931/2a7931a0002
https://www.itv.com/watch/a-spy-among-friends/2a7931/2a7931a0001

[A_n_g_e_l_a]

Agree ITVX have made a change to some json code format.


[edit]
Corrected code at post 1
[/edit]

[A_n_g_e_l_a]

Full script edited to take account of ITVX site change https://forum.videohelp.com/editpost.php?do=updatepost&postid=2694766&posthash=ce702f2...65#post2694766

[A_n_g_e_l_a]

Are you using the address bar URLs?

Yes. This is from "A Spy Among Friends", 6 episode serie taken from the copy url button in Stream Detector.

https://www.itv.com/watch/a-spy-among-friends/2a7931/2a7931a0006
https://www.itv.com/watch/a-spy-among-friends/2a7931/2a7931a0005
https://www.itv.com/watch/a-spy-among-friends/2a7931/2a7931a0004
https://www.itv.com/watch/a-spy-among-friends/2a7931/2a7931a0003
https://www.itv.com/watch/a-spy-among-friends/2a7931/2a7931a0002
https://www.itv.com/watch/a-spy-among-friends/2a7931/2a7931a0001

You know you can get it in 1080p from stv?

[Fallito]

Agree ITVX have made a change to some json code format.

Edit two lines
line 193 to look like this

title = myjson["props"]["pageProps"]["programme"]["title"]

line 194 to look like this

magni_url = myjson["props"]["pageProps"]["seriesList"][0]["titles"][0]["playlistUrl"]

Take care to keep code indentatiion lined up.

Good. Some progress. Now is resolving to this:


[Attachment 72860 - Click to enlarge]

[Killer3737]

Full script edited to take account of ITVX site change https://forum.videohelp.com/editpost.php?do=updatepost&postid=2694766&posthash=ce702f2...65#post2694766

This works in that it downloads again but it doesn't seem to get the correct episode(s) now. I tested with this

https://www.itv.com/watch/heartbeat/Ya0757/Ya0757a0275

But it just downloads S01E01 and titled merely Heartbeat.mp4

[A_n_g_e_l_a]

Full script edited to take account of ITVX site change https://forum.videohelp.com/editpost.php?do=updatepost&postid=2694766&posthash=ce702f2...65#post2694766

This works in that it downloads again but it doesn't seem to get the correct episode(s) now. I tested with this

https://www.itv.com/watch/heartbeat/Ya0757/Ya0757a0275

But it just downloads S01E01 and titled merely Heartbeat.mp4

Mmm they've put the needed info at the end of the end of 1800 lines of json, so the script is picking up the very first episode . Look like ITVX are game playing. I'll look at it tomorrow or use this https://forum.videohelp.com/threads/407216-Decryption-The-Dungeon-of-Despair#post2669285 second script - it uses the mpd and stream detector table entry so won't have the json problems this script has. Downside - no subtitles.

[A_n_g_e_l_a]

But it just downloads S01E01 and titled merely Heartbeat.mp4

Now corrected in post #1 of this thread

[A_n_g_e_l_a]

Now is resolving to this:


[Attachment 72860 - Click to enlarge]

Have you got N_m3u8DL-RE installed as a binary file? Is it in your PATH? Is it in the same folder as as the script?

[Magicians]

Now is resolving to this:


[Attachment 72860 - Click to enlarge]

That's your problem - Windows user!!

Have you got N_m3u8DL-RE installed as a binary file? Is it in your PATH? Is it in the same folder as as the script? if it is rename N_m3u8DL-RE in the script to have .exe at the end.

pop-os is a linux distro

[A_n_g_e_l_a]

pop-os is a linux distro

Didn't see it.

Can't have got N_m3u8DL-RE installed then? All the same I don't want to sort out problems with people's systems. that has nothing to do with me.

[A_n_g_e_l_a]

Made final update to code at post 1. Hopefully working for series as well as one-off programmes and films

[Killer3737]

Made final update to code at post 1. Hopefully working for series as well as one-off programmes and films

Working on series again, thanks.

[day69walker]

Following on from A_n_g_e_l_a's advice in post 59, having installed the tt conv as A_n_g_e_l_a tried on the Windows virtual machine and using the modified script in post 1 I am now getting the following output shown

[Attachment 72879 - Click to enlarge]

Here is my code as it stands

# Angela 13:07:2023
# reworked to match recent changes at ITVX 
# 2:08:2023 revision 2
# @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
# With grateful thanks to sk8ord13 for code
# dealing with the remote CDM
# @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

## This program uses The Stream Detector to capture page URLs
## Stream Detector select 'options'
## in the box adjacent to'user-defined-commands enter -->  %origin%
## In TSD window select copy stream url as User-Defined-Command 1

## In addition to not having to faff about with opening and saving text files
## this program downloads, converts and merges subtitles.

## added option for a sequence number to add to the videoname (This will be the order they
## are selected in TSD). It removes the chance of an over-write of the video name when ITV
## uses a generic tile like 'Inspector Morse' and without a series or episode number. 

# @@@@@@@@@@@ IMPORTANT @@@@@@@@@@@@@@@@@@@@@
## subtitles need the pip install as below.
##
##  pip install --pre ttconv

# should you ever wish to run a convert subtitles routine from the command line:-
##  tt convert -i <input .vtt file> -o <output .srt file>
# @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

import re
import requests
import subprocess
from base64 import b64encode
from pathlib import Path
import httpx
from httpx import URL, Client
from selectolax.lexbor import LexborHTMLParser
import os
import pyperclip as PC
#from pywidevine.L3.cdm import deviceconfig
#from pywidevine.L3.decrypt.wvdecryptcustom import WvDecrypt
import pyfiglet as PF
from termcolor import colored
import json
import shutil

# GLOBALS
OUT_PATH = Path('output')
OUT_PATH.mkdir(exist_ok=True, parents=True)
global count
global SEQ
global REMOTE

#@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
# There is a choice of CDM to use
# local or remote;
# yours or someone else's.
# To use your local CDM in the WKS-KEYS
# folder, set REMOTE=False.
#
REMOTE = True
#
# @@@@@@@@@@@@@@@@@@@@@@@@
# NOTE
# seting for index number to preface videoname either True or False
# If you want each video in your clipboard to be numbered by a preface to
# the videoname set the value of SEQ = TRUE
# This is useful for series without numbers but relies on the correct
# order in the clipbord so select video1, video2, etc in sequence
# @@@@@@@@@@@@@@@@@@@@@@@

SEQ = False

# Class configured as a singleton 
# only one instance is created
# if consructor called again original
# instance is returned
class ITV:
    _instance = None

    def __init__(self):
        raise RuntimeError('Call instance() instead')

    @classmethod
    def instance(cls):
        if cls._instance is None:
            #print('Creating new instance of ITV Class')
            cls._instance = cls.__new__(cls)
            cls.host = 'itvpnpdotcom.blue.content.itv.com'
            timeout = httpx.Timeout(10.0, connect=60.0)

            cls.client = Client(
            headers={
                'authority': 'www.itv.com',
                'user-agent': 'Dalvik/2.9.8 (Linux; U; Android 9.9.2; ALE-L94 Build/NJHGGF)',
            },
            timeout=timeout,
        )
        return cls._instance


    def download(self, url: str) -> None:
        global count
        title, data = self.get_data(url)
    
        video = data['Playlist']['Video']
        media = video['MediaFiles']
        illegals = "*'%$!(),.:;"
        replacements = {
            '  Episode ': 'E',
            '  Series ': '_S',
            'otherepisodes': '_extra',
            'ITVX': '',
            ' ': '_',
            '&': 'and',
            '?': '',
        }
        # replace  extraneous title data and 'illegal' characters
        videoname = ''.join(c for c in title if c.isprintable() and c not in illegals)
        # standardize and compact videoname    
        for rep in replacements:  
            videoname = videoname.replace(rep, replacements[rep])
        videoname = re.sub(r"(\d+)", pad_number, videoname).lstrip('_').rstrip('_')
        result = re.search(r"(^.*S\d+)_(E\d*.*)", videoname)
        try:
            pre = result.group(1)
            post = result.group(2)
            videoname = pre+post
        except:
            pass

        print(videoname)
        subs_url = video['Subtitles'][0]['Href']
        subs = requests.get(subs_url)
        f = open(f"{videoname}.subs.vtt", "w")
        subtitles = subs.text
        f.write(subtitles)
        f.close()

        # convert subtitles 
        #os.system(f"tt convert  -i {videoname}.subs.vtt -o {videoname}.subs.srt > /dev/null 2>&1")
        #os.system(f"tt convert  -i {videoname}.subs.vtt -o {videoname}.subs.srt ")
        global SEQ  # prepend a sequence number to anonymous videos
        if SEQ:
            myvideoname = format(count, "02d") +'_'+ videoname
        else:
            myvideoname = videoname
        mpd_url = [f'{video["Base"]}{y}' for x in media if (y := URL(x['Href'])).path.endswith('.mpd')][0]
        lic_url = [x['KeyServiceUrl'] for x in media][0]

        pssh = self._get_pssh(mpd_url)
        key = self._get_key(pssh, lic_url)
        temp = URL(mpd_url).params['hdnea']
        temp = temp.replace('nohubplus', 'hdntl,nohubplus')
        cookie = f"cookie: {re.sub(r'^.*(?<=exp=)', 'hdntl=exp=', temp)}"

        m3u8dl = 'N_m3u8DL-RE'   # windows rename with .exe added
        subprocess.run([
            m3u8dl,
            mpd_url,
            '--append-url-params',
            '--header',
            cookie,
            '--header',
            f'host: {self.host}',
            '--header',
            f'user-agent: {self.client.headers["user-agent"]}',
            '--auto-select',
            '--save-name',
            myvideoname,
            '--save-dir',
            './',
            '--tmp-dir',
            './',
            '-mt',
            '--key',
            key,
            '-M',
            'format=mp4',
            '--no-log'
        ])
        
        command = [
        "mkvmerge", 
        "-q",
        f"{myvideoname}.mp4",
        #f"{videoname}.subs.srt",
        f"{videoname}.subs.vtt",
        "-o",
        f"{myvideoname}.mkv"
        ]
        subprocess.run(command)
        shutil.move(f"{myvideoname}.mkv",  f"{OUT_PATH}")
        #os.system(f"rm {myvideoname}.mp4 {videoname}.subs.vtt {videoname}.subs.srt")
        os.system(f"rm {myvideoname}.mp4 {videoname}.subs.vtt ")
        count = count-1


    def get_data(self, url: str) -> tuple:
        r = self.client.get(url)
        tree = LexborHTMLParser(r.text)
        jsondata = tree.root.css_first('#__NEXT_DATA__').text()
        myjson = json.loads(jsondata)
        title = myjson["props"]["pageProps"]["programme"]["title"]
        try:
            extendtitle = myjson["props"]["pageProps"]["episode"]["contentInfo"]
            title = f"{title}_{extendtitle}"
        except:
            pass
        try:
            magni_url = myjson["props"]["pageProps"]["episode"]["playlistUrl"]
        except:
            magni_url = myjson["props"]["pageProps"]["seriesList"][0]["titles"][0]["playlistUrl"]      

        features = ['mpeg-dash', 'widevine', 'outband-webvtt', 'hd', 'single-track']
        payload = {
            'client': {'id': 'browser'},
            'variantAvailability': {
                'featureset': {'min': features, 'max': features},
                'platformTag': 'dotcom',
            }
        }
        r = self.client.post(magni_url, json=payload)
        return title, r.json()

    # REMOTE CDM or Local CDM
    def _get_key(self, pssh: str, lic_url: str , cert_b64=None) -> str:
        if REMOTE:      
            print("Using a remote CDM")
            headers = {
            'accept': 'application/json, text/plain, */*',
            'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36',
            }
            json_data = {
                'password': 'password',
                'license': lic_url,
                'headers': 'Connection: keep-alive',
                'pssh': pssh,
                'buildInfo': '',
                'cache': True,
            }

            r = self.client.post('https://wvclone.fly.dev/wv', headers=headers, json=json_data).text   
            m = re.search(r">(.{32}:.{32})<", r)
            if m:
                key = m.group(1)
            print(f"Keys found {key}\n")
            return key.lstrip()  
        else:
            print("Using CDM on this machine")
            wvdecrypt = WvDecrypt(init_data_b64=pssh, cert_data_b64=cert_b64, device=deviceconfig.device_android_generic)                   
            widevine_license = httpx.post(url=lic_url, data=wvdecrypt.get_challenge(), headers=None)
            license_b64 = b64encode(widevine_license.content)
            wvdecrypt.update_license(license_b64)
            Correct, keyswvdecrypt = wvdecrypt.start_process()
            if Correct:
                mykeys = ''
                for key in keyswvdecrypt:
                    mykeys += key +' '
            print(f"Keys found {mykeys}\n")
            return mykeys 
    
    def _get_pssh(self, mpd_url: str) -> str:
        r = self.client.get(mpd_url)
        kid = (
            LexborHTMLParser(r.text)
            .css_first('ContentProtection')
            .attributes.get('cenc:default_kid')
            .replace('-', '')
        )
        s = f'000000387073736800000000edef8ba979d64acea3c827dcd51d21ed000000181210{kid}48e3dc959b06'
        return b64encode(bytes.fromhex(s)).decode()

# add leading zero to series or episode
def pad_number(match):
    number = int(match.group(1))
    return format(number, "02d")

def main() -> int:
    input("Press enter with PAGE urls in clipboard ")
    urls = PC.paste().split('\n')  
    global count 
    count = len(urls) 
    print(f"The URL list has {count} video(s)")
    myITV = ITV.instance()
    for url in urls:
        url = url.encode('ascii', 'ignore').decode()
        myITV.download(url)
    return 0

if __name__ == "__main__":
    title = PF.figlet_format(' I T V X ', font='smslant')
    print(colored(title, 'green'))
    main()
    exit(0)

I am getting King_of_the_Roads_S07E03.subs.vtt file that does have the correct subs in it. I don't actually require subtitles just the video with audio. As far as I can tell the subs part is working now?

Should my next port of call be the Linux machine.

Thank you all again for any help.

[A_n_g_e_l_a]

I am getting King_of_the_Roads_S07E03.subs.vtt file that does have the correct subs in it. I don't actually require subtitles just the video with audio. As far as I can tell the subs part is working now?

Should my next port of call be the Linux machine.

Is that code an different from what is at page 1?

It is generally not a good idea to post copies of other's code in your post as the original poster no longer has control over versions. It will become out of sync with the original and people will get confused. If you have made changes say what you did, and give before and after lines of code. I don't have the patience to wade through it line by line looking for differences. And I cannot imagine anyone else reading it to find them.

It is not unreasonable in a technical forum to expect users to have some technical knowledge and an ability to read errors reports and sort problems out themselves.

[stabbedbybrick]

A small piece of advice regarding subtitles is to write it to file and then use "--mux-import" in N_m3u8DL-RE to mux it with the video. No extra steps or libraries needed and works regardless of platform.

This is what I've been doing in my own scripts and it works great.

[A_n_g_e_l_a]

A small piece of advice regarding subtitles is to write it to file and then use "--mux-import" in N_m3u8DL-RE to mux it with the videot.

Useful thanks. N_m3u8DL-RE doesn't say it converts imports it muxes though and that would be an issue for me. I need the conversion.

[A_n_g_e_l_a]

. N_m3u8DL-RE doesn't say it converts imports it muxes though and that would be an issue for me. I need the conversion.

Just tested and it fails with vtt subtitles so still need subtitle conversion to srt.


[Attachment 72881 - Click to enlarge]

[stabbedbybrick]

I use it specifically with VTT files from ITV and it works great:


[Attachment 72893 - Click to enlarge]

Code snippet I use:

if subtitle != "":
        r = requests.get(url=f"{subtitle}", headers=HEADERS)
        with open(f"sub.vtt", "wb") as f:
            f.write(r.content)

And then

"path=sub.vtt:lang=eng:name='English'"

[A_n_g_e_l_a]

I use it specifically with VTT files from ITV and it works great:

And then

"path=sub.vtt:lang=eng:name='English'"

This line works for srt subs but as soon as I change srt to vtt (both files exist) it fails

 f"--mux-import:path=./{videoname}.subs.srt:lang=eng",

Even tried breaking that line with --mux-import and the rest on the next line.

But for me, with vtt embedded, the subs do not work on my video-player - it doesn't see them. (Dreambox Satellite box)

Which version of N_m3u8DL-RE do you have? Mine is 0.2.0

edit:
Thinking it through, it is most likely something in the subs.vtt from ITVX that breaks N_m

[stabbedbybrick]

I'm using 0.2.0 on Linux.

Are you sure you're writing the file as bytes, as in "wb"? It needs to be in its original state or it will fail when muxing.

The documentation for N_m3u8DL-RE isn't the best, but looking through the settings and processors, it does appear that it "fixes" and converts to SRT as default. And I'm assuming it treats imports the same way. But if it doesn't work on your media player then it's not much to do about it other than sticking to what works for you.