UK Free to Air Downloader

[sipho]

I uploaded a mod to the yt-dlp extractor that gives 1080p.

Good to know.
Is that the one that uses a certificate?

No but it's similar to the 1080p from ctv in that you need to add a section to the ytp-dlp extractor plugin.

Post is here.

[A_n_g_e_l_a]

Updated to version 6. URL link at post #1

Corrections to My5. Subtitles should select the non-audio description channel in most cases now. The non-download of video without subtitles has been fixed
All checked working.

[peterb999]

Thanks Angela. I've downloaded the CH5 program and the subtitles work great.

[sipho]

I missed the part about automatic CH5 HMAC and AES update so did that manually but the search function worked beautifully. I needed a show from My5 and one of the other scripts available on this site was not able to grab it. This one worked like a boss!

Thank you.

[A_n_g_e_l_a]

... I needed a show from My5 and one of the other scripts available on this site was not able to grab it.

Thanks.

I take it you mean Stabbies stuff by 'other scripts' ? He is highly energetic and prolific; I take my hat off to him. We are not in competition. Command-line orientated software is a personal preference. My preference is being able to do everything I need to download video programmes without opening my browser, or doing a pre-search. You make your individual choices while you can; a surfeit of software cannot last forever.

I've re-written my5 to be faster using N_m3u8DL-RE and removed nearly 200 lines of code. And corrected no-directory selection being offered for download.

I've also altered the folder structure to

[Attachment 74567 - Click to enlarge]
It will be released after error checking

[sipho]

The one thing I like about the 'other' script (yes I'm referring to Stabby's) is the ability to leave the subtitle untouched in .vtt format. This allows me to process it according to my needs. For UK shows I use normal subs, subs for hearing impaired and subs for hearing impaired but with colours to aid in speaker identification. If the respective downloader software fixes or cleans the subs or even converts to .srt, the original formatting is often lost.

Looking forward to your latest version.

[A_n_g_e_l_a]

The one thing I like about the 'other' script (yes I'm referring to Stabby's) is the ability to leave the subtitle untouched in .vtt format. This allows me to process it according to my needs. For UK shows I use normal subs, subs for hearing impaired and subs for hearing impaired but with colours to aid in speaker identification. If the respective downloader software fixes or cleans the subs or even converts to .srt, the original formatting is often lost.

Looking forward to your latest version.

Unfortunately vtt will not play nicely on some playback methods and until vtt is more universal it will be srt. As an 'oldie' who uses subs for real, blue and yellow text is nice but the brain follows who's talking well enough without.

[sipho]

Maybe I wasn't clear enough. I convert the .vtt subs to .srt.

[A_n_g_e_l_a]

Maybe I wasn't clear enough. I convert the .vtt subs to .srt.

OK, I didn't take all the info in; but using vtt is the same as N_m3u8~ does isn't it? Or are you saying your converter in some way does a better job? Which converter do you use? I've played around with loads but leaving too much style in stops it working on playback for me.

However, for your needs you can always add an extra line in the command where N_m3u8DL-RE is being set up

--auto-subtitle-fix=False

will embed vtt . Or if you need vtt as a file you could switch-off muxing and get the bits for doing what you like with.

There is absolutely no reason to regard someone's script as a finished product. It is after-all an editable-thing and becomes a starting point to tailor for your use. So do your thing on the code!

[A_n_g_e_l_a]

V7 released with new file structure and major overhaul of My5. File at post #1

[peterb999]

Thanks Angela.

I've downloaded it and it works great, thanks again for your hard work.

[A_n_g_e_l_a]

Sorry to the 3 who have downloaded .. I introduced an error in UKTVPlay corrected at #1 now version 8

[sipho]

Version 8 now complains about this.

ERROR: shaka-packager not found!

It isn't in the requirements.txt nor in version 6.

Before I stumble forth to install it, perhaps confirm if it's really needed? I don't think I've ever used this package.

One more thing. Could there be an option to deselect all episodes upfront? For some shows there will be hundreds of episodes available. Useful to be able to just select the one you need.

[A_n_g_e_l_a]

Version 8 now complains about this.

ERROR: shaka-packager not found!

It isn't in the requirements.txt nor in version 6.

Before I stumble forth to install it, perhaps confirm if it's really needed? I don't think I've ever used this package.

One more thing. Could there be an option to deselect all episodes upfront? For some shows there will be hundreds of episodes available. Useful to be able to just select the one you need.

Shaka-packager has always been a necessary tool to install. it is listed on the top level README. It is used because it is faster than ffmpeg.

The maximum shows in any list will only be the number of shows in a series, unless you choose to list all shows. My5 wasn't collating into series. That error has been fixed.

If you only need a single show you would choose the single download from the menu.

[sipho]

I mean when you are using the search facility.

Thanks for the info on the shaq.

[A_n_g_e_l_a]

I mean when you are using the search facility.

Thanks for the info on the shaq.

Yes that's what I mean too! If you know you only need one video you could copy the url from the screen list; ctrl+c out of the script; re-run menu.py; select single.

But now I've more time I can answer you more fully.

The keyword search initially provides a list of programmes either with the keyword or an approximation of it. Nothing is checked as it is a single select type list. Once you've made your choice then the next list will be a series of episode numbers waiting for your input to select one or more series. Once you've made a selection the series videos are listed each with a checbox pre-ticked. It is the only pre-ticking done so you must be referring to this. But you want a single program only here. In practice I don't find unchecking a few tick-boxes onerous. Arrow button to advance and space-bar to uncheck, two-hands in a high-tempo rhythm. But you want more..

I thought about the use of pre-ticking carefully. Whilst I recognise either option will have it proponents I considered that new users would probably like a whole series to catch-up. And then as each week delivers a new video they could use the single download. That was the behaviour I observed of myself.

Now you ask if the checkboxes could be left unticked. The answer is yes. You ask can it be done by some setting. The answer is yes, if the setting is a configuration setting and sets no ticked check-boxes when the script runs. I suspect you'd soon be wanting the opposite when say MY5 have bought the rights to a new title and you want to download everything..

Ideally the User Interface would have another tickbox or a button that would tick or untick the other boxes in the list of displayed videos. It is a small thing you ask isn't it?

But here we run into the difference between a program and a script. A script choreographs a series of actions that we might normally do by hand. A program more than likely opens Graphic User Interface with a series of buttons and text-boxes etc. It sits there impassively waiting for your input. Just like your browser.

To program your single checkbox to wait for input and then change all the other boxes is programmatically a complex task. Behind the scenes it means setting up event listeners that can observe the state of the checkbox continuously waiting for change. It means there must be event drivers propagating change and event listeners reading the change and altering other checkbox settings. It is not something I would attempt with Python.

However, Python scripts are meant to be adaptable by end-users.


[Attachment 74585 - Click to enlarge]
remove the highlighted phrase to leave checkboxes unchecked - that is My5 in the image. Other scripts have the same setting at different positions.

[A_n_g_e_l_a]

Small edit to My5 - some videos without a season associated were failing. New update to version 9 at #1

[Spectrumhsm]

I was trying to download couple of episodes from stv and ran into errors I could not understand:

Use up/down arrows to select.
[info] getting data for https://player.stv.tv/summary/new-scotland

Use up/down keys + spacebar to de-select or re-select videos to download

[?] Series 1 Episode 1 https://player.stv.tv/episode/36n6/new-scotland
[?] Series 1 Episode 2 https://player.stv.tv/episode/36ns/new-scotland

(Mark with space, confirm with enter)
[info] Preparing to download ...
Traceback (most recent call last):
  File "C:\UK-FTA\stv_dl\stv_loader.py", line 380, in <module>
    stv.entrypoint(str(result))
  File "C:\UK-FTA\stv_dl\stvgetter.py", line 273, in entrypoint
    get_stage_one_data(client,videoid, seriesguid, guid, DRM)
  File "C:\UK-FTA\stv_dl\stvgetter.py", line 260, in get_stage_one_data
    subprocess.run(command)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1776.0_x64__qbz5n2kfra8p0\Lib\subprocess.py", line 548, in run
    with Popen(*popenargs, **kwargs) as process:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1776.0_x64__qbz5n2kfra8p0\Lib\subprocess.py", line 1026, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1776.0_x64__qbz5n2kfra8p0\Lib\subprocess.py", line 1538, in _execute_child
    hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [WinError 2] The system cannot find the file specified
PS C:\UK-FTA\stv_dl>

Thanks for any help.

[A_n_g_e_l_a]

[Attachment 74589 - Click to enlarge]

Works for me.

In the top level readme is a list of helper programs to install. The section that caused your error uses N_m3u8DL-RE, ffmpeg and MKVmerge (Mkvmerge from MKVToolNix)
Haven't we been here before with My5 or Channel4?

[Spectrumhsm]

=A_n_g_e_l_a;2710473
Works for me.

In the top level readme is a list of helper programs to install. The section that caused your error uses N_m3u8DL-RE, ffmpeg and MKVmerge (Mkvmerge from MKVToolNix)
Haven't we been here before with My5 or Channel4?

oops.. got it working.
downloaded both episodes.

Thanks for putting up with us pestering newbies.

[LastResort]

Hi Chaps...

I found the creation of the wvd file nice and easy.... (this made a nice change). Due to the creator of the aio script I created a folder called angela,... copied the blob and key to the outside of this folder. and ran the following in an elevated cmd prompt:
pywidevine create-device -k device_private_key -c device_client_id_blob -t "ANDROID" -l 3 -o C:\Users\xxxxxxxxxx\Desktop\UKTVPlay\Angela
This gave me a file called android_sdk.wvd within the angela folder.

Now I know this is going to sound really stupid, but From what I have read I cannot use the aio anywhere near the wks-keys folder. So I have copied the aio folder to C, so it looks like this:
C:\UK-FTA Now do I simply copy the android_sdk.wvd into UK-FTA folder??
replies and advice would be much appreciated.

Edit: I had a quick look through the i t v .py script and I saw "PYWIDEVINE = True" and I can see that further down it leads to
"if PYWIDEVINE:
print("Using pywidevine WVD on this machine")"

So would the android_sdk.wvd need to be copied into each of the seperate folders... i.e. one in C:\UK-FTA\ukfta\itv_dl another in C:\UK-FTA\ukfta\c4_dl ect, ect.

Or does the created wvd remember where it was created and thus always run from that original place i.e. C:\Users\xxxxxxxxxx\Desktop\UKTVPlay\Angela

-----------------------------------------------------------------------
I needed to test a theory.... I dropped my wvd into UK-FTA folder... and ran menu.py from an elevated cmd prompt...
I needed something to test on so went with series/search name/stv
searched for "enemy" without the quotes... and it found what i was after... a five episode series called "the last enemy" I selected the first in the list, which was the last enemy..
all 5 episodes were pre selected: so it was just a case of hitting enter: --- it's all going well so far,... until this:



An STV Video Search, Selector and Downloader.


Search word(s)? enemy


Use up/down arrows to select.
[info] getting data for https://player.stv.tv/summary/dcd-thelastenemy

Use up/down keys + spacebar to de-select or re-select videos to download

[info] Preparing to download ...
Traceback (most recent call last):
File "C:\UK-FTA\ukfta\stv_dl\stv_loader.py", line 380, in <module>
stv.entrypoint(str(result))
File "C:\UK-FTA\ukfta\stv_dl\stvgetter.py", line 273, in entrypoint
get_stage_one_data(client,videoid, seriesguid, guid, DRM)
File "C:\UK-FTA\ukfta\stv_dl\stvgetter.py", line 260, in get_stage_one_data
subprocess.run(command)
File "C:\Users\xxxxxxxxxx\AppData\Local\Programs\Python \Python311\Lib\subprocess.py", line 548, in run
with Popen(*popenargs, **kwargs) as process:
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\xxxxxxxxxx\AppData\Local\Programs\Python \Python311\Lib\subprocess.py", line 1026, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "C:\Users\xxxxxxxxxx\AppData\Local\Programs\Python \Python311\Lib\subprocess.py", line 1538, in _execute_child
hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [WinError 2] The system cannot find the file specified

C:\UK-FTA>

(I'm now lost...)

This was after I edited config.py within the configs folder and changed it to what I thought was correct:

# Edit the paths below
## WVDPATH is the full location of your wvd file created by 'pywidevine create device...'

WVDPATH = r"C:\UK-FTA/android_sdk.wvd"

# A base-directory for storage of downloaded videos. The storage path will be extended by channel name and series
# DO NOT USE A TRAILING SLASH ON THIS PATH

SAVEPATH = r"D:\YT_Rips\WKS-KEYS\output"

# These paths apply to all UK-FTA scripts

# All4 There is a choice of downloader. N_m3u8DL-RE is the default but vry rarely it fails to download with Mux Error reported.
# Yt-dlp will download these videos

C4_USES_N_m3u8DLRE = True # set to False to use yt-dlp

I tried running menu.py again and, expecting it to work, I had exactly the same error occur.

[A_n_g_e_l_a]

Now I know this is going to sound really stupid, but From what I have read I cannot use the aio anywhere near the wks-keys folder. So I have copied the aio folder to C, so it looks like this:
C:\UK-FTA Now do I simply copy the android_sdk.wvd into UK-FTA folder??

can do. Yes. Think about it: the wvd file is a locator that points to your blob and key. You can point from anywhere can't you?

So would the android_sdk.wvd need to be copied into each of the seperate folders...

No you put the location of the wvd file in configs/config.py and each script looks this up

# Edit the paths below
## WVDPATH is the full location of your wvd file created by 'pywidevine create device...'

WVDPATH = r"C:\UK-FTA/android_sdk.wvd"

# A base-directory for storage of downloaded videos. The storage path will be extended by channel name and series
# DO NOT USE A TRAILING SLASH ON THIS PATH

SAVEPATH = r"D:\YT_Rips\WKS-KEYS\output"

You need to use the python way of wrting a path to a file; use forward slashes
SAVEPATH = r"D:/YT_Rips/WKS-KEYS/output" for example - do the same for all the paths you've mentioned.

[A_n_g_e_l_a]

Pssst.

[Attachment 74694 - Click to enlarge]

[LastResort]

thanks Angela.... I did some more reading before coming back here..... and I almost have it working.... (this bit i cannot work out, as I have all of the relevant items already installed)

Search word(s)? enemy


Use up/down arrows to select.
[info] getting data for https://player.stv.tv/summary/dcd-thelastenemy

Use up/down keys + spacebar to de-select or re-select videos to download

[info] Preparing to download ...
13:26:01.583 INFO : N_m3u8DL-RE (Beta version) 20230628
13:26:01.607 ERROR: ffmpeg not found, please download at: https://ffmpeg.org/download.html
[info] The_Last_Enemy_S01E01.mkv is in C:\UK-FTA\output/STV/Dcd-Thelastenemy
[info] Preparing to download ...
13:26:05.168 INFO : N_m3u8DL-RE (Beta version) 20230628
13:26:05.191 ERROR: ffmpeg not found, please download at: https://ffmpeg.org/download.html
[info] The_Last_Enemy_S01E02.mkv is in C:\UK-FTA\output/STV/Dcd-Thelastenemy
[info] Preparing to download ...
13:26:08.745 INFO : N_m3u8DL-RE (Beta version) 20230628
13:26:08.770 ERROR: ffmpeg not found, please download at: https://ffmpeg.org/download.html
[info] The_Last_Enemy_S01E03.mkv is in C:\UK-FTA\output/STV/Dcd-Thelastenemy
[info] Preparing to download ...
13:26:12.269 INFO : N_m3u8DL-RE (Beta version) 20230628
13:26:12.294 ERROR: ffmpeg not found, please download at: https://ffmpeg.org/download.html
[info] The_Last_Enemy_S01E04.mkv is in C:\UK-FTA\output/STV/Dcd-Thelastenemy
[info] Preparing to download ...
13:26:15.850 INFO : N_m3u8DL-RE (Beta version) 20230628
13:26:15.875 ERROR: ffmpeg not found, please download at: https://ffmpeg.org/download.html
[info] The_Last_Enemy_S01E05.mkv is in C:\UK-FTA\output/STV/Dcd-Thelastenemy

C:\UK-FTA>

I have in the following folder: C:\Windows\System32

mkvmerge.exe
mp4decrypt.exe
N_m3u8DL-RE.exe

(I was'nt to sure of which part of shaka i needed so i downloaded both mpd_generator-win-x64.exe and packager-win-x64.exe) pure guess work told me the most likely to change the name of was packager-win-x64.exe to shaka-packager.exe.

p.s. i changed the output folder to something easier to find.

a quick edit of the config.py file so it now looks like this:
WVDPATH = r"C:\UK-FTA/android_sdk.wvd"

# A base-directory for storage of downloaded videos. The storage path will be extended by channel name and series
# DO NOT USE A TRAILING SLASH ON THIS PATH

SAVEPATH = r"C:/UK-FTA\output"
---------------------------------------------------------------------

really sorry Angela,...... No sleep for 14 hours does'nt help,..... and neither does a stinking cold....

i forgot to put in ffmpeg.exe, and, ffprobe.exe

That's really bad on my part,.... considering how often I use these.

All files are now downloading. seems I did not need to place these last two files into C:\Windows\System32, i just dropped them into C:\UK-FTA.

Thank you for your help.... it's very much appreciated.
I learn a little more each time,... and boy, do i do a lot of reading... lol

[LastResort]

Oh wow, Angela,.. is that v10??

[A_n_g_e_l_a]

seems I did not need to place these last two files into C:\Windows\System32, i just dropped them into C:\UK-FTA.

Glad your confidence is up and it is now working for you.
PATH is always the current folder and whatever is in the PATH environment value

And its V11. I've jumped one.

[LastResort]

Nice.... Fantastic that TalkingPics is in there.... I tried to find some episodes of something called "Footage Detective",... a small piece was played on a fbook group I had joined for an area I knew quite well. I asked the person, who had filmed it from her tv on a mobile what it was, and she just said i think its called footage detective, all i could make out is that it was the dj mike smith narrating,... I believe the term is "needle in a haystack" lol.

needless to say,... looking forward to v11.

[A_n_g_e_l_a]

Talking Pictures TV encore is added to the line-up.


[Attachment 74710 - Click to enlarge]

If film from the 30s to the 70s floats your boat then sail away for the evening with TP TV encore.
TP TV is an eclectic mix of nostalgia from public service broadcasts to early Popeye and Superman cartoons. It has **exciting** film titles like "The Night of the Living Dead" or "Radar Men From The Moon". I kid you not!!
If you were a child of the 50s or 60s, memory lane is here with 'Bulldog Drummond', 'Larry the Lamb', 'Sexton Blake', 'Paul Temple' And Maigret's featuring Bruno Cremer (In French with subtitles) or Rupert Davies.

Videos are added and removed weekly. It's the sort of stuff you might download but find there is always something better to watch!! I joke, of course! (I think?)

Don't expect series in the sense of one episode following another - TP TV is haphazardly curated. Some video urls are hidden by the javascript video-player and require a login. But use of search finds urls for most but not quite all videos on the site. Don't have high expectation for search; one word succeeds better than two, but experiment. The site response is pretty basic.

And don't expect high resolutions; a lot of this stuff was on film before resolution was invented, the maximum seems to be 576.

As a departure from a pre-ticked list - because search seems to return little more random list bearing small resemblance to the search term, the list of videos often bear little relation to each other. It makes little sense to pre-tick these videos.


WINDOWS
More generally .. after trialing this suite of scripts on Windows I see what a sad looking mess it is compared to Linux.
However, Windows Terminal from the MS (cr)App Store - https://apps.microsoft.com/detail/9N0DX20HK701 - works more like it does in Linux.
It is recommended you use Terminal for these scripts. If py <script name> fails be sure to try python <script name> or even python3 - one should work

My5
Remember my5 needs the HMAC and KEY updated from time to time. If you have Firefox installed on your system the updater in the menu should work otherwise see the readme in the my5-dl folder to get keys by hand.

Url for V11 at post #1

[LastResort]

Hi Angela,

I have attempted to update my5 keys.... but I believe I have made a mistake in the hmac-aes-update.py within my5_dl folder. Fairly sure I have made a mistake in the areas that require updating to my requirements.

(This is being typed on the very latest version of firefox).... and selenium has been installed.

Error:
C:\UK-FTA>menu.py
←[32m __ ____ __ _____________
/ / / / //_/___/ __/_ __/ _ |
/ /_/ / ,< /___/ _/ / / / __ |
\____/_/|_| /_/ /_/ /_/ |_|

←[0m
←[31mChoose Your Search, Selector and Downloader.
←[0m
←[32mWhich Channel?←[0m
Traceback (most recent call last):
File "C:\UK-FTA\ukfta\my5_dl\hmac-aes-update.py", line 61, in <module>
replace_line(file, 1 , text1)
File "C:\UK-FTA\ukfta\my5_dl\hmac-aes-update.py", line 26, in replace_line
lines = open(file_name, 'r').readlines()
^^^^^^^^^^^^^^^^^^^^
OSError: [Errno 22] Invalid argument: 'file://c:/UK-FTA/ukfta/my5_dl/config.py'

C:\UK-FTA>

---------------------------------------------------------------

hmac-aes-update.py

from selenium import webdriver
from selenium.webdriver.firefox.options import Options
import time
import json

## edit this
# r"file://C:/<path to file>
LOCATION_RETRIEVE_KEYS_HTML = r"file://C:/UK-FTA/ukfta/my5_dl/keys/retrieve-keys.html" -------------------------- here -----------------
## end edit - ONE MORE BELOW

def replace_line(file_name, line_num, text):
lines = open(file_name, 'r').readlines()
lines[line_num] = text
out = open(file_name, '+w')
out.writelines(lines)
out.close()

options = Options()
options.add_argument('-headless')
driver = webdriver.Firefox(options=options)

driver.get(f"{LOCATION_RETRIEVE_KEYS_HTML}")
time.sleep(3) # do not remove this! Edit seconds to wait up if you get 'None' reported.
source = driver.page_source
result = source.replace('<html><head></head><body>','').replace('</body></html>','')
mydict = json.loads(result)
driver.close()

hmac = f'''HMAC_SECRET = "{mydict['HMAC_SECRET']}"'''
aes = f'''AES_KEY = "{mydict['AES_KEY']}"'''



## edit this @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @@@@@@@@
# full path allows you to run this program from anywhere on your system
# file to be altered is:-
# r"file://C:/<path to file>
file = r'file://c:/UK-FTA/ukfta/my5_dl/config.py' ---------------------------- and here -------------------------

# replacements to make

text1 = f'{hmac}\n'
text2 = f'{aes}\n'

# replace_line(<path to your file> , <line number to change> < text to insert>)
# line numbers in text count from 0
replace_line(file, 1 , text1)
replace_line(file, 2 , text2)

---------------------------------------------

I'm not sure if the word "file" is required, I tried to look through some other scripts but couldn't find anything similar.

I have manually ran the retrieve-keys.html and that works fine, and I can see that both HMAC and AES have changed.

[peterb999]

Try it with three /// instead of two. Below is mine that works fine :

LOCATION_RETRIEVE_KEYS_HTML = r"file:///C:/rClone/FTA/ukfta/my5-dl/keys/retrieve-keys.html"