Need to script extraction of m3u8 URL on a live video for use with ffmpeg

[BosseB]

I have an Ubuntu Server 20.04, which I use in order to download US news shows for viewing at times when I am awake.
Time difference is 6 hours.
Anyway I have scripted the downloads and they use ffmpeg commands to download live streams into 852x480 px mp4 video files.
But in order to do this my ffmpeg command needs the m3u8 URL of the actual video stream.
I can get this by opening the webpage holding the video player with the show in Firefox and hit F12. Then in the Network tab I can find the m3u8 item, right click it and select Copy URL.
Then I can paste this into my script and it will be able to download the stream in 1 hour parts.
But I have found that some sites change the m3u8 URL from time to time, so I need to somehow script the extraction of this URL so it can be used with the ffmpeg command.

I have asked in several places but as of now I do not have a solution.
The problem site is this:
http://www.freeintertv.com/view/id-2308

For other sites that change the m3u8 URL too I have been able to script the extraction, only not for this site.

Is there anyone here who is able to help with a command I can put into a script to extract the m3u8 URL and save it to a text file to be read by the download script?

[lub0112]

is it one specific channel url
for example the one you posted is nbc usa ?

the site you posted is just a mish mash of embeded players

[BosseB]

Yes this is a mish mash for sure.
They have a "search engine" for finding news shows all around the world and I use another of them for another channel and that is possible to script.
But this specific one has defeated my attempts so far.
Using Firefox with the F12 command works every time, though, so I figured scripting it should also work if a clever enough command was found.
I am using 10 different sources and most do not change m3u8 regularly, stay the same for weeks or more and then I can just repair it via Firefox when it does change.
On a few they do change but a fairly simple curl command can dig out the currently working URL so I execute that 5 min before recording starts.
But this site seems to use different methods for all its content. Another target here can be scripted but this does not work for the problem target.

[jack_666]

https://forum.videohelp.com/threads/404578-How-to-extract-and-use-an-m3u8-URL-to-downl...am#post2646315

[BosseB]

https://forum.videohelp.com/threads/404578-How-to-extract-and-use-an-m3u8-URL-to-downl...am#post2646315

That thread ends in a solution for the same multi-source site, but another video source.
It is for this main URL:
http://www.freeintertv.com/view/id-2565
And this has been working fine ever since I entered it into my script.

But for this source on the same main site it does not work:
http://www.freeintertv.com/view/id-2308

Notice how they differ only by the id number at the end, yet it seems impossible to get extraction working...
My only way out now is to manually extract m3u8 using Firefox some time close to the start of the show and hope for the best.
Not ideal for shows running in the middle of the night over here.

I have also tried to use xidel, but to no avail...

EDIT:
I have extracted the chname entity from the 2308 video using xidel and got this:
CHNAME: nbc_us
CHNAME64: bmJjX3Vz

So it correctly identifies NBC but....
When I used that chname in the following command it returned an m3u8 URL, but it is NOT the correct channel as one can view on the 2308 webpage!

curl -qSs -d 'chname=bmJjX3Vz%3D%3D&ch=http%3A%2F%2Fwww.freeintertv.com%2Fexternals%2Ftv-russia%2Fsmotret-tv3-online&html5=11' 'http://www.freeintertv.com/myAjax/get_item_m3u8/' | sed -e "s/^.*http\(.*\)m3u8.*$/http\1m3u8/g"
http://livedoc.cgtn.com/500d/prog_index.m3u8

Using this m3u8 to record gives me video from this documentary channel with Id=1188 instead...

So there is probably something else that also needs modification...

[BosseB]

Follow-up:
At 21:14 tonight I manually modified the m3u8 URL used to download the 2308 video in preparation for the recording, which was set to start at 23:00
But at that time the recording did not start anyway, because ffmpeg failed to find the stream...

When I checked it at about 23:20 I saw that I needed to update the URL, did so and rescheduled the recording to start at 23:26.
Now recording, for how long I don't know yet. Set to last until midnight.
I guess I have to manually try to figure out how often it changes so I know if this is viable or not. But that entails checking the url every hour or so to see how long it lives.

It just goes to show how this m3u8 extraction needs to be scripted....

[BosseB]

Follow-up #2
Now a day later and the URL I retrieved yesterday is still working!
It was extracted using Firefox with F12 when the video was playing, like I have done earlier.
But this time it seems to work permanently.
I have installed an hourly cron job to test downloading with this URL for a short time (20 s) and if ffmpeg fails then send me an alert email.
So far it has not happened (except when I tested it by intentionally changing the URL) for 22 hours.

So somehow I have found the golden URL without expiry date!

[BosseB]

Follow-up #2
Now a day later and the URL I retrieved yesterday is still working!
It was extracted using Firefox with F12 when the video was playing, like I have done earlier.
But this time it seems to work permanently.
I have installed an hourly cron job to test downloading with this URL for a short time (20 s) and if ffmpeg fails then send me an alert email.
So far it has not happened (except when I tested it by intentionally changing the URL) for 22 hours.

So somehow I have found the golden URL without expiry date!

Talked too soon!
At 00:32 (6 hours+ ago) I received the alert email that indicated a failure to download.
So the URL lifetime was longer but just over 24 hours still...
And an extraction script is still needed....

PS:
Why does this forum add links to some words like f f m p e g?

[BosseB]

Now fully examined the m3u8 URL and found the timeout.
Here is an URL taken at 07:55

https://nbcnews3.akamaized.net/hls/live/723429/NBCNewsNowa418c506/clear/hdntl=exp=1650261300~acl=%2f*~data=hdntl~hmac=a97bd5a535c6d05c49f3d81f52eb2ecf788273fa8f8c9e05cfdc5f4abf990552/VIDEO_1_4596000.m3u8

Notice this part of the URL:

exp=1650261300

At about 2 minutes later I did this to get current time in seconds:

date +%s
1650175021

Now subtract this from the exp part of the URL:

1650261300 - 1650175021 = 86279

This is 121 s from a day's worth of seconds (86400)...
Or about the time I spent while doing the URL extraction in Firefox.

So the duration before expiration is exactly 24 hours after extraction...

[BosseB]

Does anyone here know what FireFox does on its F12 debug page to find (amongst other) the m3u8 URL?

[LZAA]

https://mega.nz/folder/segBSQYI#r_QVxk8lqVMSKEOqE2WQ3A

[BosseB]

https://mega.nz/folder/segBSQYI#r_QVxk8lqVMSKEOqE2WQ3A

Thanks!!
I am trying to use it with my existing scripts. Standalone it works if I add a -t argument to ffmpeg.
For reference the link posted contains this script example:

#!/bin/bash

CURL=$(command -v curl)
FFMPEG=$(command -v ffmpeg)
URL=('https://tokens.playmakerservices.com/' -d $'{\n "pid": "2007524",\n "application": "NBCSports",\n "platform": "desktop",\n "cdn": "akamai",\n "url": "https://nbcnews3.akamaized.net/hls/live/723429/NBCNewsNowa418c506/clear/master.m3u8"\n}')
INPUT=$("$CURL" -qsS "${URL[@]}"  | jq -Mr '.akamai[] | .tokenizedUrl')

"$FFMPEG" -i "${INPUT[@]}" -c copy video.mp4

So it is an actual full example, which does start the download - very promising.
I need video formatting and time limits too as provided by my own script.

However my existing script (which does a lot more than just run ffmpeg and is used towards several different source sites) needs an m3u8 URL string and this is not what I get when I run this and stop at the FFMPEG line.

The INPUT variable contains this, when I stop just before the FFMPEG call:

https://nbcnews3.akamaized.net/hls/live/723429/NBCNewsNowa418c506/clear/master.m3u8?hdnts=st=1650467245~exp=1650467365~acl=/hls/live/723429/NBCNewsNowa418c506/clear/*~hmac=3dae63fd0a2915d348b8e309751e948d98eafc2b92a395bddd60a5d450fd392e

Unfortunately this does not work with my use of ffmpeg in the existing script which looks as below.
Note that the variables VIDEOURL and M3U8URL are read from a file prepared manually via Firefox for the 2308 site.
CAPTURETIME and TARGETFILE are call arguments 1 and 2 for the script.

VIDEOURL="http://www.freeintertv.com/view/id-2308" #Read from a file
M3U8URL=<the manually extracted m3u8 URL using FireFox from page VIDEOURL> #Read from a file
MODE="-referer \"${VIDEOURL}\"" #referer is needed to make the stream source to send data
CAPTURETIME=<number of seconds to record, call argument 1>
TARGETFILE=<destination file name, call argument 2>
CMD="ffmpeg -hide_banner ${MODE} -i \"${M3U8URL}\" -vf scale=w=-4:h=480 -c:v libx264 -preset fast -crf 26 -c:a copy -t ${CAPTURETIME} ${TARGETFILE}"

eval "$CMD"

Is it possible to adjust the curl call arguments so it digs out a valid m3u8 URL?
Otherwise I probably need to have a separate script created for this particular video source site...

[LZAA]

curl 'https://tokens.playmakerservices.com/' -d $'{\n "pid": "2007524",\n "application": "NBCSports",\n "platform": "desktop",\n "cdn": "akamai",\n "url": "https://nbcnews3.akamaized.net/hls/live/723429/NBCNewsNowa418c506/clear/master.m3u8"\n}'

[BosseB]

I decided to write an NBC specific download script where I do not read stuff from anywhere but use your suggested command in NBC.sh instead.
I copied the relevant stuff from the other script and customized it for this source.

So for all downloads from NBC I will use this and it seems to grab the input stream correctly every time it is executed, so there should not be any timeout problems.

Thanks @LZAA for this, much appreciated!