ITVX - Revisited

[A_n_g_e_l_a]

I've reworked the down-loader for ITVX to have a more graphical and selective feel with a totally new front end
and a slimed-down down-loader.
Plus I've added a site keyword search to more easily find the videos you want to see.


[Attachment 73947 - Click to enlarge]
Entry screen



[Attachment 73948 - Click to enlarge]
keyword search for 'rugby'



[Attachment 73949 - Click to enlarge]
all results shown - with some selected for download



[Attachment 73950 - Click to enlarge]
keys found and downloading with N_mu3u8DL-RE
English Subs muxed with download



[Attachment 73951 - Click to enlarge]
Saved to a <programme-title> based file-path of ./output/ITV/<programme-title/<programme-name>

Video names do not follow 'scene convention'; they have programme title and episode title - according to availability - and are readable. Video resolution is the highest available and usually 1280x720.
In some cases STV may provide a better resolution.

A zip file is below it contains requirements.txt for

pip install -r requirements.txt

to ensure all python modules are available.
Two python scripts itv_loader.py and slimitv.py. As with my STV loader these work together, but slimitvx.py works as a single downloader on its own taking the page URL.
The itv_loader will function either as a single episode down-loader or a multiple series down-loader.

The search function is filtered to show ITVX's free output only; the down-loader will not deal with subscription videos from ITVX.

As the scripts do more there will be many more paths through the script some of which may not have been exercised by me when error checking. The most likely failure points are with the video file-names but I haven't had to make a correction for some while so hope the issue is solved.

slimitvx.py does the downloading and key finding. Either WKS-KEYS or Pywidevine. As provided the selection is for WKS-KEYS but edit the slimitvx.py script to provide the the Pywidevine WVD location and set PYWIDEVINE variable to True.

Reports of script failure points are helpful. Reports of 'operator error' are not.

https://files.videohelp.com/u/301890/itv-update.zip

[sipho]

Nice one Angela!

[A_n_g_e_l_a]

pysubs2 got left off the list in requirements.txt. Sorry!

If you get an error mentoning pysubs2 just

 pip install pysubs2

Oh. and Windows users; the script looks better when run in 'Power Shell'

[dodgydodgy]

Hi,

I think the script gets thrown by program titles with colons in them (at least in Windows) - was trying it on the snooker:

Search word(s)?    snooker
[info] getting data for https://www.itv.com/watch/Snooker:-British-Open-Championship/10a1758

Use up/down keys + spacebar to de-select or re-select videos to download

Using WKS-KEYS CDM on this machine
Keys found 63661b6dbe3d4901b388972607580639:4e34af7d221f48bc3118664a03c37273

Traceback (most recent call last):
  File "C:\Bin\ITVX\WKS-KEYS\itv_loader.py", line 246, in <module>
    myITV.download(url)
  File "C:\Bin\ITVX\WKS-KEYS\slimitvx.py", line 178, in download
    OUT_PATH.mkdir(exist_ok=True, parents=True)
  File "D:\Python311\Lib\pathlib.py", line 1116, in mkdir
    os.mkdir(self, mode)
NotADirectoryError: [WinError 267] The directory name is invalid: 'output\\ITV\\Snooker:_British_Open_Championship'

I'm guessing the problem is it can't create "Snooker:_British_Open_Championship" because of the colon being disallowed in filenames in Windows.

Apologies if I'm on the wrong track and this is user error ...

[A_n_g_e_l_a]

Hi,

I think the script gets thrown by program titles with colons in them (at least in Windows) - was trying it on the snooker:

Search word(s)?    snooker
[info] getting data for https://www.itv.com/watch/Snooker:-British-Open-Championship/10a1758

Use up/down keys + spacebar to de-select or re-select videos to download

Using WKS-KEYS CDM on this machine
Keys found 63661b6dbe3d4901b388972607580639:4e34af7d221f48bc3118664a03c37273

Traceback (most recent call last):
  File "C:\Bin\ITVX\WKS-KEYS\itv_loader.py", line 246, in <module>
    myITV.download(url)
  File "C:\Bin\ITVX\WKS-KEYS\slimitvx.py", line 178, in download
    OUT_PATH.mkdir(exist_ok=True, parents=True)
  File "D:\Python311\Lib\pathlib.py", line 1116, in mkdir
    os.mkdir(self, mode)
NotADirectoryError: [WinError 267] The directory name is invalid: 'output\\ITV\\Snooker:_British_Open_Championship'

I'm guessing the problem is it can't create "Snooker:_British_Open_Championship" because of the colon being disallowed in filenames in Windows.

Apologies if I'm on the wrong track and this is user error ...

It errors out for me too. Thanks for reporting it. On the list to fix ....
[edit] error appears wider than just snooker. Keys are found OK but download refused with a 403 'Forbidden'

Its the weekend and things don't get changed in the backend on Sundays so I'll wait before chasing this.

[A_n_g_e_l_a]

Posted a code correction to follow the recent ITVX change at #1
I've inadvertently posted with PYWIDEVINE =True If you use WKS-KEYS just find it in ITVX and set it to False.

[dodgydodgy]

Thanks for the fix, but the link doesn't seem to work to download?

[A_n_g_e_l_a]

Thanks for the fix, but the link doesn't seem to work to download?

Silly me!

[dodgydodgy]

Can I check the file you linked to is the right one (i.e. the fixed one), because I get the same error as from the previous one:

Search word(s)?    snooker
[info] getting data for https://www.itv.com/watch/Snooker:-British-Open-Championship/10a1758

Use up/down keys + spacebar to de-select or re-select videos to download

Using WKS-KEYS CDM on this machine
Keys found 6fc6ca398644478187d7d68de647e2ed:8212c77653e970559a8b95b01024887d

Traceback (most recent call last):
  File "C:\Bin\ITVX\WKS-KEYS\itv_loader.py", line 246, in <module>
    myITV.download(url)
  File "C:\Bin\ITVX\WKS-KEYS\ITVX.py", line 178, in download
    OUT_PATH.mkdir(exist_ok=True, parents=True)
  File "D:\Python311\Lib\pathlib.py", line 1116, in mkdir
    os.mkdir(self, mode)
NotADirectoryError: [WinError 267] The directory name is invalid: 'output\\ITV\\Snooker:_British_Open_Championship'

[A_n_g_e_l_a]

Can I check the file you linked to is the right one (i.e. the fixed one), because I get the same error as from the previous one:

Working for me now with files already on VH. Keep me posted. Try deleting the folder __pycache__ it will re-built on next run.

[Attachment 74146 - Click to enlarge]

[dodgydodgy]

Are you running it on Windows, or Linux?

I was wondering if my original assumption was the case, that it's trying and failing to create a directory with a colon in it:

NotADirectoryError: [WinError 267] The directory name is invalid: 'output\\ITV\\Snooker:_British_Open_Championship'

[peterb999]

Hi Dodgydodgy.

I'm running it on windows and get the same as you for the snooker.

If you add the following after line 60 in ITVX.py it fixes it :

':': '',

It should then look like this :

def rinse(self,string):
illegals = "*'%$!(),.;" # safe for urls
string = ''.join(c for c in string if c.isprintable() and c not in illegals)
replacements = {
' ': '_',
'_-_': '_',
'&': 'and',
':': '',
}

Peter

[dodgydodgy]

Hi Dodgydodgy.

I'm running it on windows and get the same as you for the snooker.

If you add the following after line 60 in ITVX.py it fixes it :

':': '',

It should then look like this :

def rinse(self,string):
illegals = "*'%$!(),.;" # safe for urls
string = ''.join(c for c in string if c.isprintable() and c not in illegals)
replacements = {
' ': '_',
'_-_': '_',
'&': 'and',
':': '',
}

Peter

That did the trick, thanks very much!

[peterb999]

Glad I could help.

[A_n_g_e_l_a]

Glad I could help.

Thanks for the correction for Windows. I'm surprised Linux didn't cough too.
[Attachment 74147 - Click to enlarge] I've altered the code in the zip.

[Antiques5114]

Hi. Thank you for the script!

I'm getting an issue downloading any content from ITV using the latest version:

Traceback (most recent call last):
  File "/Users/blah/dev/itv/revisit/itv_loader.py", line 246, in <module>
    myITV.download(url)
  File "/Users/blah/dev/itv/revisit/ITVX.py", line 170, in download
    pssh = self.get_pssh(mpd_url)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/blah/dev/itv/revisit/ITVX.py", line 75, in get_pssh
    .attributes.get('cenc:default_kid')

Adding in some debug print()s, I can see that the url is, eg

https://itvpnpdotcom.blue.content.itv.com/10-1937-0001-002/34/1/VAR028-HD-s/10-1937-0001-002_34_1_VAR028-HD-s.ism/.mpd?filter=%2528%2528type%253D%253D%2522video%2522%2526%2526DisplayHeight%253C%253D720%2529%257C%257C%2528type%2521%253D%2522video%2522%2529%2529&hdnea=st%253D1696283526~exp%253D1696305126~acl%253D%2F10-1937-0001-002%2F%252A~hmac%253Dc54688d85ff7649899aa833f180cb0d96c9e322b5da0276b1739fee349aa2f6f

and that r.status_code is 403.

Any pointers would be much appreciated

[Killer3737]

Thanks for your continued efforts on this, much appreciated. I've switched over to this from the other version as you requested but after placing the downloaded files in the same WKS_KEYS folder I was using before and installing the requirements on Windows i'm getting this error:

Enter video url for download.
https://www.itv.com/watch/heartbeat/Ya0757/Ya0757a0014
Traceback (most recent call last):
File "D:\AV_Edit_Tools\itvx_downloader\NEW-OCT\WKS-KEYS\ITVX.py", line 268, in <module>
main()
File "D:\AV_Edit_Tools\itvx_downloader\NEW-OCT\WKS-KEYS\ITVX.py", line 260, in main
myITV.download(url)
File "D:\AV_Edit_Tools\itvx_downloader\NEW-OCT\WKS-KEYS\ITVX.py", line 167, in download
pssh = self.get_pssh(mpd_url)
^^^^^^^^^^^^^^^^^^^^^^
File "D:\AV_Edit_Tools\itvx_downloader\NEW-OCT\WKS-KEYS\ITVX.py", line 72, in get_pssh
.attributes.get('cenc:default_kid')
^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'attributes'

EDIT: Tried it in a wsl2 ubuntu instance as well, same error.

[dodgydodgy]

That URL works for me, suggesting perhaps a setup/config issue rather than a script issue?

You do need to change this in ITVX.py to use WKS-KEYS:

PYWIDEVINE = False

C:\Bin\ITVX\WKS-KEYS>python itv_loader.py
     ____  ______  _   __  _  __
    /  _/ /_  __/ | | / / | |/_/
   _/ /    / /    | |/ / _>  <
  /___/   /_/     |___/ /_/|_|


An ITVX Video Search, Selector and Downloader.


Enter any ITVX url for the series-title to download
https://www.itv.com/watch/heartbeat/Ya0757/Ya0757a0014
[info] Series found are:-
S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 S11 S12 S13 S14 S15 S16 S17 S18
[info]There are over 16 videos to display.
Enter the series number(s) to see a partial list,
        or enter '0' to show all episodes available

        Separate series numbers with a space

? S18
Use only numbers and Spaces!
[info] Series found are:-
S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 S11 S12 S13 S14 S15 S16 S17 S18
[info]There are over 16 videos to display.
Enter the series number(s) to see a partial list,
        or enter '0' to show all episodes available

        Separate series numbers with a space

? 18

Use up/down keys + spacebar to de-select or re-select videos to download

Using WKS-KEYS CDM on this machine
Keys found 0c5a07172f874401ab80c273964cac03:13cd1a439b12c86e5e9d0bc4e88fb11a

08:17:05.189 INFO : N_m3u8DL-RE (Beta version) 20230628
08:17:05.196 INFO : Loading URL: https://itvpnpdotcom.blue.content.itv.com/Y-0757-0349-001/34/1/VAR028-HD-s/Y-0757-0349-001_34_1_VAR028-HD-s.ism/.mpd?filter=%28%28type%3D%3D%22video%22%26%26DisplayHeight%3C%3D720%29%7C%7C%28type%21%3D%22video%22%29%29&hdnea=st%3D1696317412~exp%3D1696339012~acl%3D/Y-0757-0349-001/%2A~hmac%3Dc0dc89c3fb53930effc13d1a716595d850266e50469af1578516bc2488c22f87
08:17:05.224 INFO : Content Matched: Dynamic Adaptive Streaming over HTTP
08:17:05.224 INFO : Parsing streams...
08:17:05.236 WARN : Writing meta json
08:17:05.242 INFO : Extracted, there are 5 streams, with 4 basic streams, 1 audio streams, 0 subtitle streams
08:17:05.242 INFO : Vid *CENC 1024x576 | 2297 Kbps | video=2297335 | avc1.64001F | 464 Segments | ~46m21s
08:17:05.243 INFO : Vid *CENC 1024x576 | 1492 Kbps | video=1492929 | avc1.64001F | 464 Segments | ~46m21s
08:17:05.243 INFO : Vid *CENC 704x396 | 837 Kbps | video=837799 | avc1.64001F | 464 Segments | ~46m21s
08:17:05.243 INFO : Vid *CENC 704x396 | 500 Kbps | video=500653 | avc1.64001F | 464 Segments | ~46m21s
08:17:05.244 INFO : Aud *CENC audio=96000 | 96 Kbps | mp4a.40.2 | 2CH | 464 Segments | ~46m21s
08:17:05.244 INFO : Parsing streams...
08:17:05.259 INFO : Selected streams:
08:17:05.260 INFO : Vid *CENC 1024x576 | 2297 Kbps | video=2297335 | avc1.64001F | 464 Segments | ~46m21s
08:17:05.260 INFO : Aud *CENC audio=96000 | 96 Kbps | mp4a.40.2 | 2CH | 464 Segments | ~46m21s
08:17:05.260 WARN : Writing meta json
08:17:05.264 INFO : Save Name: Heartbeat_S18E01_Family_Matters
08:17:05.264 WARN : MuxAfterDone is detected, binary merging is automatically enabled
08:17:05.265 INFO : Start downloading...Vid 1024x576 | 2297 Kbps | video=2297335 | avc1.64001F
08:17:05.265 INFO : Start downloading...Aud audio=96000 | 96 Kbps | mp4a.40.2 | 2CH
08:17:05.286 WARN : Type: cenc
08:17:05.286 WARN : PSSH(WV): CAESEAxaBxcvh0QBq4DCc5ZMrAMiElktMDc1Ny0wMzQ5LTAwMV8zNDgB
08:17:05.286 WARN : KID: 0c5a07172f874401ab80c273964cac03
08:17:05.287 WARN : Reading media info...
08:17:05.300 WARN : Type: cenc
08:17:05.301 WARN : PSSH(WV): CAESEAxaBxcvh0QBq4DCc5ZMrAMiElktMDc1Ny0wMzQ5LTAwMV8zNDgB
08:17:05.301 WARN : KID: 0c5a07172f874401ab80c273964cac03
08:17:05.301 WARN : Reading media info...
08:17:05.507 INFO : [0x1]: Video, h264 (avc1), 1024x576
08:17:05.508 INFO : [0x1]: Audio, aac (mp4a), 96 kb/s
08:17:13.359 INFO : Binary merging...
08:17:13.737 INFO : Decrypting...
08:17:14.252 INFO : Binary merging...
08:17:14.331 INFO : Decrypting...
08:17:18.835 WARN : Heartbeat_S18E01_Family_Matters.mp4
08:17:18.835 WARN : Heartbeat_S18E01_Family_Matters.m4a
08:17:18.836 WARN : Heartbeat_S18E01_Family_Matters.subs.srt
08:17:18.836 WARN : Muxing to Heartbeat_S18E01_Family_Matters.MUX.mkv
08:17:20.944 WARN : Cleaning files...
08:17:20.974 WARN : Rename to Heartbeat_S18E01_Family_Matters.mkv
08:17:20.974 INFO : Done
[info] Heartbeat_S18E01_Family_Matters.mkv is in output\ITV\Heartbeat

I am running the script from within the WKS-KEYS folder and have the following:

[__pycache__]
[pywidevine]
[output]
[Logs]
aria2c.exe
downey.exe
ffmpeg.exe
ffplay.exe
ffprobe.exe
headers.py
ITVX.py
itv_loader.py
l1.py
l3.py
mkvextract.exe
mkvinfo.exe
mkvmerge.exe
mkvpropedit.exe
mkvtoolnix-gui.exe
mp4decrypt.exe
mp4dump.exe
mp4edit.exe
mp4encrypt.exe
N_m3u8DL-RE.exe
original_lic.bin
parsed_lic.bin
shaka-packager.exe
slimitvx.py
yt-dlp.exe
__init__.py

[A_n_g_e_l_a]

Thanks for your continued efforts on this, much appreciated. I've switched over to this from the other version as you requested but after placing the downloaded files in the same WKS_KEYS folder I was using before and installing the requirements on Windows i'm getting this error:.

Your script seems to have all it needs to run and the error appear to be getting no response to the mpd request. I have just tried the heartbeat program with both versions and each work for me with the same code as I've posted. So something is up.

Now this is worrying. On Sunday night when looking at where the fault may lie and stepping through the code I seem to remember getting an mpd request fail. But by Monday when taking a proper look it all sailed through and the error lay with sending cookies and whatnot with the video download request - as they were no longer required. (that is the mpd used again.)

I think change may be afoot at ITVX and we are in the middle of it. If this is the case perhaps it might be best to wait a few days for it to settle down. STV is a good alternative, meanwhile, but not I think for Heartbeat https://forum.videohelp.com/threads/411591-STV-Revisiting-the-Downloader

If you wish to try bestITVX.py find lines 185 to 190 inclusive and delete them. that is all lines between, but NOT including '--append-url-params', and '--auto-select', but I suspect you will have the same issue.

[A_n_g_e_l_a]

That URL works for me, suggesting perhaps a setup/config issue rather than a script issue?

Helpful, thanks

You can delete slimitvx.py if you want as it no longer gets called.

[peterb999]

I do get a 403 permission denied error like Antiques5114 on one PC (my server) everytime I try and download from ITV. It is odd as it works fine on another PC - very strange as the WKS folder etc. actually runs from the server on the PC that works. They are both running Windows 10. I have never found out why (on the server I can view ITVX shows actually on the ITVX website). It also has a 403 error if I run the direct script using stream detector in YT-DLP (using the YT-DLP script entry)

I just don't use the server to download, haha.

[Killer3737]

Thanks for your continued efforts on this, much appreciated. I've switched over to this from the other version as you requested but after placing the downloaded files in the same WKS_KEYS folder I was using before and installing the requirements on Windows i'm getting this error:.

Your script seems to have all it needs to run and the error appear to be getting no response to the mpd request. I have just tried the heartbeat program with both versions and each work for me with the same code as I've posted. So something is up.

Now this is worrying. On Sunday night when looking at where the fault may lie and stepping through the code I seem to remember getting an mpd request fail. But by Monday when taking a proper look it all sailed through and the error lay with sending cookies and whatnot with the video download request - as they were no longer required. (that is the mpd used again.)

I think change may be afoot at ITVX and we are in the middle of it. If this is the case perhaps it might be best to wait a few days for it to settle down. STV is a good alternative, meanwhile, but not I think for Heartbeat https://forum.videohelp.com/threads/411591-STV-Revisiting-the-Downloader.

Ok, understood. I tried a few random shows, all gave the same result. As I say I also fired up an ubuntu WSL instance and tried it there with the same result, so not a Windows speficic issue, my connection/IP perhaps. WSL_KEYS shouldn't be a problem, it's the same instance where I was successfully using your old script as of a couple of weeks ago.

Don't know if it helps any in debugging but the srt and vtt files DO get downloaded.

I'm not in any hurry, so hopefully things become clear and can be sorted soon. Always the way with these methods to download from services like this, a constant battle with the provider changing things.

If you wish to try bestITVX.py find lines 185 to 190 inclusive and delete them. that is all lines between, but NOT including '--append-url-params', and '--auto-select', but I suspect you will have the same issue.

Those lines are already commented out by yourself in the latest version of the zip I downloaded.

[A_n_g_e_l_a]

now irrelevant.

[A_n_g_e_l_a]

OK! Good News! Now I have the error!

[A_n_g_e_l_a]

I think i may have cracked it. I got the error this mornng only AFTER upgrading httpx.

Try this - only if your system doesn't work:

pip uninstall httpx

then

pip install httpx==0.23.3

[dodgydodgy]

Weirdly I have httpx==0.24.0 and it works okay for me

[A_n_g_e_l_a]

Weirdly I have httpx==0.24.0 and it works okay for me

My update took me to 0.25.0. So can we assume 0.25.0 is the culprit?
a way to see all versions is to put x in the pip install http==0.2x.x and it will fail but in doing so tells what versions there are.


[Attachment 74155 - Click to enlarge]

I see there is a beta version 1.0.0b0. I've just tried that and it works too, so 0.25.0 is the one to be avoided. Perhaps not because of itself but because of the other modules it installs. Who knows?

[sipho]

I think i may have cracked it. I got the error this mornng only AFTER upgrading httpx.

Try this - only if your system doesn't work:

pip uninstall httpx

then

pip install httpx==0.23.3

This works for me now. Wasn't before with version 0.25

[A_n_g_e_l_a]

Great! And thanks for the confirmation.

That was a hard one to spot!

[Antiques5114]

I can also confirm that the httpx downgrade fixes the mpd retrieval. Thank you!

I'm now hitting a new issue from WKS-KEYS:

unable to parse license - check protobufs

The value of widevine_license variable at line 107 of ITVX.py is

<Response [403 Forbidden]>

Again, any guidance would be much appreciated