UK Free to Air Downloader

[dmn]

I've just tried this out and it seems great, comprehensive and simple to use - thanks so much for the effort of putting this together, but I did run into a few issues while using it.

I'm running this on Arch Linux 6.9.7-arch1-1 kernel.

I'll note the problems I've had below, @A_n_g_e_l_a if I can provide you with any more info for debugging please let me know.

First thing was just that I didn't have nano installed, it might be nice to use the environment variable $EDITOR rather than hardcoding, but it's not really a problem.

The second was that the binary name for N_m3u8DL-RE is normalised to n-m3u8dl-re on the version installed from the AUR, but again that's specific to my environment really and not a problem as such. If anyone does have the same problem I did a quick and dirty find and replace with:

$ readarray -t files < <(grep -rl N_m3u8DL-RE)
$ for f in "${files[@]}"; do sed -i 's/N_m3u8DL-RE/n-m3u8dl-re/g' "$f"; done

Onto the actual problems, the first thing I tried was fetching Thomas and Friends All Engines Go from My5. The first couple of episodes were fine, but it fails on the third one (https://www.channel5.com/show/thomas-friends-all-engines-go-e128df48-42b5-46a3-8ad4-ce...cse-to-deliver) with:

https://www.channel5.com/show/thomas-friends-all-engines-go-e128df48-42b5-46a3-8ad4-ce42e1f38113/season-25/licencse-to-deliver
Generating the episode URL...
Getting the encrypted content info...
Generating the content URL...
Getting content response...
Decrypting the content response...
[LICENSE URL] https://cassie.channel5.com/api/v2/licences/widevine/491/C5357200003?expiry=1722958319&tag=66643537333061393932373539653839663830313332646333616534633832356531633565376633
https://akadash0.akamaized.net/hydra/C5357200003/C5357200003B/20230818055347/C5357200003_SD-fixed-label.mpd
An error has occured. Is the AES key and the HMAC secret up-to-date?
                   The error message was:-
cannot access local variable 'subtitles_mpd' where it is not associated with a value
[LICENSE URL] https://cassie.channel5.com/api/v2/licences/widevine/491/C5357200003?expiry=1722958319&tag=66643537333061393932373539653839663830313332646333616534633832356531633565376633
https://akadash0.akamaized.net/hydra/C5357200003/C5357200003B/20230818055347/C5357200003_SD-fixed-label.mpd                                                                              
An error has occured. Is the AES key and the HMAC secret up-to-date?
                   The error message was:-
cannot access local variable 'subtitles_mpd' where it is not associated with a value
Traceback (most recent call last):
  File "/home/dmn/cdm/ukftad/UK-FTA/ukfta/my5_dl/my5getter.py", line 382, in main
    lic, mpd, subs = get_rendition(decrypted_message)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dmn/cdm/ukftad/UK-FTA/ukfta/my5_dl/my5getter.py", line 235, in get_rendition
    default_mpd,
    ^^^^^^^^^^^
UnboundLocalError: cannot access local variable 'default_mpd' where it is not associated with a value

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/dmn/cdm/ukftad/UK-FTA/ukfta/my5_dl/my5_loader.py", line 387, in <module>
    my5.main(url)
  File "/home/dmn/cdm/ukftad/UK-FTA/ukfta/my5_dl/my5getter.py", line 384, in main
    lic, mpd = get_rendition(decrypted_message)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dmn/cdm/ukftad/UK-FTA/ukfta/my5_dl/my5getter.py", line 235, in get_rendition
    default_mpd,
    ^^^^^^^^^^^
UnboundLocalError: cannot access local variable 'default_mpd' where it is not associated with a value

I did use the Update My5 option but it didn't resolve the issue. I had a brief look at my5_dl/my5getter.py and noticed that original_mpd has a value which isn't handled for that episode, it ends in _SD-fixed-label.mpd. I added a clause handling that case and it worked fine.

I then tried searching on BBC, after selecting a TV show I got:

Traceback (most recent call last):
  File "/home/dmn/cdm/ukftad/UK-FTA/ukfta/bbc_dl/getBBCLinks.py", line 450, in <module>
    main(pid)
  File "/home/dmn/cdm/ukftad/UK-FTA/ukfta/bbc_dl/getBBCLinks.py", line 192, in main
    allseries = populatetable(pid)
                ^^^^^^^^^^^^^^^^^^
  File "/home/dmn/cdm/ukftad/UK-FTA/ukfta/bbc_dl/getBBCLinks.py", line 141, in populatetable
    sql = f'DELETE FROM iplayer where rowid BETWEEN {total-1} AND {total};'
                                                     ^^^^^
UnboundLocalError: cannot access local variable 'total' where it is not associated with a value

I did see that total is only set in an except block, I tried setting it to None prior to the loop and then only running the SQL query if it had a value assigned, that prevented the error but then the only output is:

[info] No series of that number found. Exiting. Check and try again.

I tried with several shows and got the same thing, I haven't been able to get BBC working at all.


Edit: Never mind about any of that, my get_iplayer was broken because of some Perl issues, it's working fine.

Other than that though it's working great, really useful tool, thanks again for providing it.

[A_n_g_e_l_a]

... I did run into a few issues while using it.

Onto the actual problems, the first thing I tried was fetching Thomas and Friends All Engines Go from My5. The first couple of episodes were fine, but it fails on the third one (https://www.channel5.com/show/thomas-friends-all-engines-go-e128df48-42b5-46a3-8ad4-ce...cse-to-deliver) with:

Other than that though it's working great, really useful tool, thanks again for providing it.

Its good you've solved your own problems. Channel 5 sometimes summons up a new way of naming their mpds and its my guess some intern made a mistake - but it breaks the parser. I've sorted a number of these and am surprised they can still make more conjugations! I'll look at a more generalized solution as clauses seem to need too much upkeep.

Most of us are fetching N_m3u8DL-RE from source - there was a new build a short while ago on github; look under actions and sign in so files are downloadable Use that and you won't have naming errors.

[A_n_g_e_l_a]

Update to my5gettter.py https://files.videohelp.com/u/301890/my5getter.py

Removed lots of 'else if' statement dealing with My5 variation in mpd labelling and now process with regular expressions. So if an intern invents a new riff on an old mpd the script should no longer fail. I'll make a release eventually..

[jake2k8]

If you use ITVX and try to download anything except for the latest episode it appears to download the latest episode.
If you download the latest 2 episodes it appears to work.
EG. Love Island
Download S11E43 and it downloads S11E44.
However if you download both E43 and E44 it works correctly.
.

Corrected my silly error of logic trying to reconcile two lists https://files.videohelp.com/u/301890/uk-fta_15_2.zip

Thank you for another update Angela, I can't tell you how much I love this piece of software

[billybanana]

Update to my5gettter.py

Thanks Angela for the update. I was able to pick apart yours and Diazole's logic to get my My5 script working ok in a consistent format like my others.


[Attachment 81319 - Click to enlarge]

OMG their naming convention would drive you insane ! Too many permutations.

[A_n_g_e_l_a]

OMG their naming convention would drive you insane ! Too many permutations.

Absolutely it DID!! Why do you think I'm so fractious?

[A_n_g_e_l_a]

Thank you for another update Angela, I can't tell you how much I love this piece of software

Jake thank you so much for taking the time to give very positive feedback. I don't often respond to such messages but I appreciate them all very much.
UK-FTA has come a long way since its first iteration and only now with the gui addition is it starting to look useful. For me it would be good if the design model got picked up by others.

[A_n_g_e_l_a]

Thanks Angela, you're a star!

While writing, are you aware of a way to correctly download subtitles for e.g. Dixon Of Dock Green (on TPTV)? Other threads seem to suggest that they use multi-line subtitles, and N_m3u8dl-RE seems to only grab the first line, ignoring the rest. I confirmed the subtitles play correctly on the web-player, and also checked the VTT subtitles are also incorrect (in case it was a conversion thing).

Have you any suggestions? (other than to get my old ears syringed )

I find wireless headphones useful in some situations. My TV also runs VLC with which to locate and play a local video. It also allows web substitles to be remotely found and played with the video - I find this a killer-app for lots of my videos saved long ago.

However, pain that TPTV.py was, I've arranged for the subtitle file to be downloaded outside of N_m3u8DL-RE and then converted and muxed.
It seems to fix the subtitle chopped-off bits issue.

Let me know how this goes https://files.videohelp.com/u/301890/TPTV.py Replace TPTV.py with this download in UK-FTA/ukfta/tptvencore/

I'll roll-up a release later.

[billybanana]

and only now with the gui addition is it starting to look useful. For me it would be good if the design model got picked up by others.

I've often thought about this and hence why I started to make a few GUIs here and there with others back end code. Eventually I moved on to wanting to dev my own scripts to go with them rather than using the underlying freevine/devine etc as examples.

Not to hijack your thread, just my ramblings, I've been building my own Ozivine front end for months and it's now pretty much at a finished point for my own use case, but it's built in a different language to the python scripts and all it's API queries for the info it picks up is entirely separate to the scripts. The thought of me releasing that and having to maintain 11 different services in 2 different programming languages truly scares me.


[Attachment 81325 - Click to enlarge]

The fact that you have developed your GUI in the same language as your scripts is something that I ultimately want to try and do and is the path of least resistance I think from a maintenance perspective. I do think people don't actually realize the amount of work that just goes into support side of things after a release. Far from a set and forget exercise.

Anyway keep up the good work. Having looked through much of your code, I do truly appreciate the effort involved to dev it and keep the wheels turning.

[A_n_g_e_l_a]

....

The fact that you have developed your GUI in the same language as your scripts is something that I ultimately want to try and do and is the path of least resistance I think from a maintenance perspective. ....

Thanks Billy. I chose to use python for my GUI as it is universal - it plays on Windows, Linux and Macs.

It's the folks who are beginners who pose most problems as in their heads they are all just perfect IT-heros and it becomes your fault scripts don't work. That is their starting assumption anyway.

Then I get the older guys - used to patronizing women - who comment 'Can confirm it works' when no-one has asked. I seem to get that quite a lot but the guys who post scripts don't - peculiar place this! 2Ist Century passed it by? Do they imagine I'd post a non-working script - a woman couldn't possibly be up to the job for heaven's sake, could she?

There will be loads of takers who give nothing back, having no idea of the time effort and frustration that goes to produce anything worth-while.
A few valued users will say thanks, without any baggage - like asking for a new feature. I value those highly.

[A_n_g_e_l_a]

Release of version 16.1

1. Rewrote parts of My5 to generate mpd by regular expression for consistent output.
2. Rewrote parts of TPTV to deal with subs only displaying one line. Moved from N_m3u8DL-RE downloadind parsing subs to direct download with ffmpeg processing.
3. Replaced allhell3 with allhell3 GUI and added batch download facility to this downloader. Now pasting is a normal ctrl+V, or click pop-up menu item.
4. Updated gui.py to use PyQt6 libraries rather than PyQt5 - now consistent look an feel across both GUIs
   16.0 ->16.1 cosmetic change only!

New release https://files.videohelp.com/u/301890/uk-fta_16_1.zip


[Attachment 81368 - Click to enlarge]

[tenxob]

@Angela
Just downloaded your latest. Thank you!

[tvdownloaderfan]

Will there be / Can there be a windows version of this ?

[A_n_g_e_l_a]

Will there be / Can there be a windows version of this ?

Where have you been? Python works on Windows too. Search Google how to run python scripts on Windows.

[tenxob]

Then I get the older guys - used to patronizing women - who comment 'Can confirm it works' when no-one has asked. I seem to get that quite a lot but the guys who post scripts don't - peculiar place this! 2Ist Century passed it by? Do they imagine I'd post a non-working script - a woman couldn't possibly be up to the job for heaven's sake, could she?

A bit harsh!!! How is that chip?

[justdave]

Thank you for this app, and its updates.

[A_n_g_e_l_a]

I added a release 16.1 with only cosmetic changes this morning. There have been 6 downloads. If you were one of those 6 allhell3gui.py in UK-FTA/ukfta/ needs a small edit if you use light-mode. Dark mode is unaffected

Change line 169 to look like this:

[Attachment 81369 - Click to enlarge]
change 'color: white;' to color: black; as in the image.

I have corrected the file, keeping the same release number https://files.videohelp.com/u/301890/uk-fta_16_1.zip

If you have version 16.0 there is no reason to change - this is just a small cosmetic change to visually clarify the use of the GUI. (And sometimes I cannot stop fiddling!!)


[Attachment 81370 - Click to enlarge]

[tvdownloaderfan]

Can a windows version be done I wonder ?

[A_n_g_e_l_a]

Someone please explain. I've lost the will to live!

[billybanana]

Can a windows version be done I wonder ?

It's built in python the universal language. No need for a separate Windows version when this works perfectly well on Windows. Please do at least some research before asking this type of question. I really don't want Angela to do herself in.

[A_n_g_e_l_a]

Love you Billy!

https://forum.videohelp.com/threads/411884-UK-Free-to-Air-Downloader/page16#post2746044

[tvdownloaderfan]

Someone please explain. I've lost the will to live!

Leave out the smarmy sarcasm Miss ok ? Learn to respond with kindness only

[A_n_g_e_l_a]

RIMMER: Well, if you ask me, the Eskimos had the right idea. They KNEW
how to handle the elderly and the permanently baffled. Middle of the
night, they'd take them out into the blizzard, remove their pyjamas,
and just leave them to it.
KRYTEN: And that's how the Eskimos cared for their old people?
RIMMER: Absolutely! That's why there's no Eskimo word for 'Eastbourne'.

[BenDWire]

<Rant>

Someone please explain. I've lost the will to live!

Leave out the smarmy sarcasm Miss ok ? Learn to respond with kindness only

For goodness sake. It's YOU who needs to do the "learning" here. If you can't be bothered to read even the first post of this thread which answers your basic question, then why should anyone treat you with any respect whatsover? Are you such a child that you need spoonfeeding? You need to learn to recognise your elders (Sorry Angela) and betters and thank them for all their hard work, not stamp your little feet and wail.

Now go and stand in the corner until you've grown up. Then make the EFFORT (for it is required) to read all of the tutorials and posts before you ask for help. Everyone here will help if you're genuinely confused, but you have to help yourself first.
</Rant>

Meanwhile, back with the elders (of which I am one of the eldest) , thanks Angela for tweaking the TPTV download. I'll give it a try later tonight and report back with any issues I encounter. I really appreciate everything you do around here.

[BenDWire]

However, pain that TPTV.py was, I've arranged for the subtitle file to be downloaded outside of N_m3u8DL-RE and then converted and muxed.
It seems to fix the subtitle chopped-off bits issue.

Let me know how this goes ... I'll roll-up a release later.

Great job Angela, thanks so much for this. I've checked a couple of episodes of "Dixon Of Dock Green", and one of "Out Of Town" and the subtitles are perfect. The only slight issue I've noticed is that the programs don't download into their own folders anymore, but that's within my skillset to fix.

I've also used your new AllHell3 GUI and I'm seriously impressed! Thank you for all your hard work, I really appreciate it.

[A_n_g_e_l_a]

.... The only slight issue I've noticed is that the programs don't download into their own folders anymore, but that's within my skillset to fix.
.... .

Thanks Ben! The error is an error of logic flow and could never have worked. It once did however, so how it got in I cannot imagine. Anyway the fix is below if you have trouble finding it yourself.


[Attachment 81434 - Click to enlarge]

    name = myjson['name']
    if '-' in name:
        dir = name.split('-')[0].strip().replace(' ', '_')
    else:
        dir = ''
    name = rinse(name).replace(' - ', '_').replace(' ','_').replace('-','_') 

Make changes to match this. This fix will appear whenever I roll-up any more errors into a release - a long time away I hope!

[BenDWire]

You're right! I didn't find that by myself Thanks for the update.

[A_n_g_e_l_a]

An update for AllHellGui it now has the same parsing goodness as allhell3.py adapted for UK-FTA to inlude a batch download facility.
And the Directory missing error from Talking Pictures is rolled in too.

Little point updating unless you use TPTV or lots of esoteric sites with AllHell3Gui...

https://files.videohelp.com/u/301890/uk-fta_16_2.zip

[tenxob]

@Angela
Many thanks, once again, for your unending hard work.

I have always had a problem with the script deleting the batch file at the end and wondered if it was within my limited skillset to find a solution.
I suppose it is unique to my Windows setup. Anyway, I thought as I am starting to learn a bit about Python, I would have a crack at it.

Error was:


[Attachment 81471 - Click to enlarge]

The solution I found was to modify the getbatch.py to include the following line:


[Attachment 81472 - Click to enlarge]

Not sure if this is the correct way or not - but I had a go! It seems to work!

[A_n_g_e_l_a]

@Angela
Many thanks, once again, for your unending hard work.

I have always had a problem with the script deleting the batch file at the end and wondered if it was within my limited skillset to find a solution.
I suppose it is unique to my Windows setup. Anyway, I thought as I am starting to learn a bit about Python, I would have a crack at it.

Error was:


[Attachment 81471 - Click to enlarge]

The solution I found was to modify the getbatch.py to include the following line:


[Attachment 81472 - Click to enlarge]

Not sure if this is the correct way or not - but I had a go! It seems to work!

Makes sense - but never a problem for me.