How to automatically extract m3u8 URL from this streaming site?

[BosseB]

I have asked here before regarding how to automatically extract m3u8 URL from a streaming site providing MSNBC news.
Now that site is no longer operational but I found another one that had a fixed m3u8 URL which I could extract by playing the video in FireFox and hitting the F12 button to inspect the activity.
Then in the scrolling data there is an item ending in m3u8 which I can right click and select "Copy URL" and the m3u8 URL is copied and I could paste it into my download command for ffmpeg.
That has worked for months but in recent days it has started changing at short intervals (merely hours) so the download script which uses it no longer works at all or downloads a different video...

So I need some command to extract the m3u8 URL just before the actual download starts...
The download uses ffmpeg and is done in a script on Ubuntu server that is started using "at" scheduling to run at regular hourly times nightly in my time zone.

But I am not clever enough to know how to actually write a bash command that can do this, so I am asking for help here.

The webpage hosting the streaming video is this:

https://livenewschat.eu/politics/

The current m3u8 URL is:

https://burgas.cdn.lncnetworks.host/hls/msnbc_live/index.m3u8

Previously working such URL strings from the same site the last 24 hours have been:

    M3U8="https://ligma.cdn.livenewschat.eu/hls/msnbc_live/index.m3u8"
    M3U8="https://ligma.cdn.livenewschat.eu/hls/ctvnews_live/index.m3u8"
    M3U8="https://deutsch.cdn.lncnetworks.host/hls/msnbc_live/index.m3u8"

but no longer work...

So it changes and can be extracted manually using FireFox.

Is there a way to automatically read this URL in s bash script on Ubuntu Server (no GUI available)?
The downloads are scheduled nightly in my time zone so I cannot manually modify the URL string...

Grateful for any help on this.


EDIT:
I have found another site that plays the MSNBC news video as a stream:
http://www.freeintertv.com/view/id-2565

But on this site the trick to use FireFox F12 to find the m3u8 URL does not work at all, no such item appears in the screen...
So is there some other way to get this site's m3u8 URL for use with ffmpeg?

[LZAA]

https://h3.ustvgo.la/MSNBC/myStream/playlist.m3u8?wmsAuthSign=c2VydmVyX3RpbWU9MTAvOS8y...ludXRlcz0yNDA=

[BosseB]

https://h3.ustvgo.la/MSNBC/myStream/playlist.m3u8?wmsAuthSign=c2VydmVyX3RpbWU9MTAvOS8y...ludXRlcz0yNDA=

Very terse reply, which of the target sites I asked about did it refer to?

And it did not work for the main (top) one:

URL used: https://livenewschat.eu/politics/ https://h3.ustvgo.la/MSNBC/myStream/playlist.m3u8?wmsAuthSign=c2VydmVyX3RpbWU9MTAvOS8yMDIyIDEwOjA5OjUxIEFNJmhhc2hfdmFsdWU9b2d6TEJsUjB0cnRER2JNYXEwcGZMQT09JnZhbGlkbWludXRlcz0yNDA=
[https @ 0x55765bf1dd00] HTTP error 403 Forbidden
https://h3.ustvgo.la/MSNBC/myStream/playlist.m3u8?wmsAuthSign=c2VydmVyX3RpbWU9MTAvOS8yMDIyIDEwOjA5OjUxIEFNJmhhc2hfdmFsdWU9b2d6TEJsUjB0cnRER2JNYXEwcGZMQT09JnZhbGlkbWludXRlcz0yNDA=: Server returned 403 Forbidden (access denied)
Video recording failed to start, sending email!

In the script I am using this command:

CMD="ffmpeg -hide_banner ${MODE} -i \"${M3U8URL}\" -vf scale=w=-4:h=${VIDEORES} -c:v libx264 -preset fast -crf 26 -c:a aac -t ${CAPTURETIME} ${TARGETFILE}"

Where:
VIDEOURL="https://livenewschat.eu/politics/" (actually read from a config file)
MODE="-referer \"${VIDEOURL}\""
M3U8URL="the long URL you posted above"
VIDEORES="480"
CAPTURETIME=the number of seconds to record
TARGETFILE= the output file

I think that ffmpeg will not accept an URL unless it ends with m3u8 in this case...
Or else there is something other in error.

BTW: Now the second site in my post has started to work again....

[VideoController.info]

Try this it scans for m3u8/mpd links and offers them for download.

[LZAA]

ffmpeg -hide_banner -i "https://h1.ustvgo.la/MSNBC/myStream/playlist.m3u8?wmsAuthSign=c2VydmVyX3RpbWU9MTAvOS8y MDIyIDExOjU2OjM0IEFNJmhhc2hfdmFsdWU9Vk1pQmdDbTNtUX FpUjJ4bDFMcjdkZz09JnZhbGlkbWludXRlcz0yNDA=" -c copy video.mp4
[hls @ 0000019e92222840] Skip ('#EXT-X-VERSION:3')
[hls @ 0000019e92222840] Opening 'https://h1.ustvgo.la/MSNBC/myStream/chunks.m3u8?nimblesessionid=315401968&wmsAuthSign= c2VydmVyX3RpbWU9MTAvOS8yMDIyIDExOjU2OjM0IEFNJmhhc2 hfdmFsdWU9Vk1pQmdDbTNtUXFpUjJ4bDFMcjdkZz09JnZhbGlk bWludXRlcz0yNDA=' for reading
[hls @ 0000019e92222840] Skip ('#EXT-X-VERSION:3')
[hls @ 0000019e92222840] Opening 'https://h1.ustvgo.la/MSNBC/myStream/l_690196_34363612_6878.ts?nimblesessionid=31540196 8&wmsAuthSign=c2VydmVyX3RpbWU9MTAvOS8yMDIyIDExOjU2 OjM0IEFNJmhhc2hfdmFsdWU9Vk1pQmdDbTNtUXFpUjJ4bDFMcj dkZz09JnZhbGlkbWludXRlcz0yNDA=' for reading
[hls @ 0000019e92222840] Opening 'https://h1.ustvgo.la/MSNBC/myStream/l_690196_34371954_6879.ts?nimblesessionid=31540196 8&wmsAuthSign=c2VydmVyX3RpbWU9MTAvOS8yMDIyIDExOjU2 OjM0IEFNJmhhc2hfdmFsdWU9Vk1pQmdDbTNtUXFpUjJ4bDFMcj dkZz09JnZhbGlkbWludXRlcz0yNDA=' for reading
Input #0, hls, from 'https://h1.ustvgo.la/MSNBC/myStream/playlist.m3u8?wmsAuthSign=c2VydmVyX3RpbWU9MTAvOS8y MDIyIDExOjU2OjM0IEFNJmhhc2hfdmFsdWU9Vk1pQmdDbTNtUX FpUjJ4bDFMcjdkZz09JnZhbGlkbWludXRlcz0yNDA=':
Duration: N/A, start: 34363.569178, bitrate: N/A
Program 0
Metadata:
variant_bitrate : 1371721
Stream #0:0: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p(tv, bt709), 640x360 [SAR 1:1 DAR 16:9], 59.94 fps, 59.94 tbr, 90k tbn
Metadata:
variant_bitrate : 1371721
Stream #0:1: Audio: aac (LC) ([15][0][0][0] / 0x000F), 44100 Hz, stereo, fltp
Metadata:
variant_bitrate : 1371721
Output #0, mp4, to 'a.mp4':
Metadata:
encoder : Lavf59.24.100
Stream #0:0: Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709), 640x360 [SAR 1:1 DAR 16:9], q=2-31, 59.94 fps, 59.94 tbr, 90k tbn
Metadata:
variant_bitrate : 1371721
Stream #0:1: Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp
Metadata:
variant_bitrate : 1371721
Stream mapping:
Stream #0:0 -> #0:0 (copy)
Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
frame= 1 fps=0.0 q=-1.0 size= 0kB time=00:00:00.00 bitrate= 39.1kb[https @ 0000019e9222b540] Opening 'https://h1.ustvgo.la/MSNBC/myStream/l_690196_34376125_6880.ts?nimblesessionid=31540196 8&wmsAuthSign=c2VydmVyX3RpbWU9MTAvOS8yMDIyIDExOjU2 OjM0IEFNJmhhc2hfdmFsdWU9Vk1pQmdDbTNtUXFpUjJ4bDFMcj dkZz09JnZhbGlkbWludXRlcz0yNDA=' for reading
frame= 599 fps=0.0 q=-1.0 size= 1280kB time=00:00:09.98 bitrate=1050.0kbframe= 935 fps=821 q=-1.0 size= 2048kB time=00:00:15.60 bitrate=1075.2kb[https @ 0000019e92226380] Opening 'https://h1.ustvgo.la/MSNBC/myStream/chunks.m3u8?nimblesessionid=315401968&wmsAuthSign= c2VydmVyX3RpbWU9MTAvOS8yMDIyIDExOjU2OjM0IEFNJmhhc2 hfdmFsdWU9Vk1pQmdDbTNtUXFpUjJ4bDFMcjdkZz09JnZhbGlk bWludXRlcz0yNDA=' for reading
[hls @ 0000019e92222840] Skip ('#EXT-X-VERSION:3')
[https @ 0000019e92226380] Opening 'https://h1.ustvgo.la/MSNBC/myStream/chunks.m3u8?nimblesessionid=315401968&wmsAuthSign= c2VydmVyX3RpbWU9MTAvOS8yMDIyIDExOjU2OjM0IEFNJmhhc2 hfdmFsdWU9Vk1pQmdDbTNtUXFpUjJ4bDFMcjdkZz09JnZhbGlk bWludXRlcz0yNDA=' for reading
...

[BosseB]

Well I got this:

$ ffmpeg -hide_banner -i "https://h1.ustvgo.la/MSNBC/myStream/playlist.m3u8?wmsAuthSign=c2VydmVyX3RpbWU9MTAvOS8y MDIyIDExOjU2OjM0IEFNJmhhc2hfdmFsdWU9Vk1pQmdDbTNtUX FpUjJ4bDFMcjdkZz09JnZhbGlkbWludXRlcz0yNDA=" -c copy -t 60 videotest.mp4

[https @ 0x561311c8d1c0] HTTP error 500 Internal Server Error
https://h1.ustvgo.la/MSNBC/myStream/playlist.m3u8?wmsAuthSign=c2VydmVyX3RpbWU9MTAvOS8y MDIyIDExOjU2OjM0IEFNJmhhc2hfdmFsdWU9Vk1pQmdDbTNtUX FpUjJ4bDFMcjdkZz09JnZhbGlkbWludXRlcz0yNDA=: Server returned 5XX Server Error reply

[LZAA]

https://h3.ustvgo.la/MSNBC/myStream/playlist.m3u8?wmsAuthSign=c2VydmVyX3RpbWU9MTAvOS8y...ludXRlcz0yNDA=

vs

https://h1.ustvgo.la/MSNBC/myStream/playlist.m3u8?wmsAuthSign=c2VydmVyX3RpbWU9MTAvOS8y...ludXRlcz0yNDA=

Do you see the difference?

[BosseB]

Try this it scans for m3u8/mpd links and offers them for download.

Thank you for the link to the add-on! Just tried it.

Well, it seems like it is a tool to extract the URL and save it to a playlist file.
When I do this it results in this kind of content in the saved file:

https://h1.ustvgo.la/MSNBC/myStream/chunks.m3u8?nimblesessionid=315547546&wmsAuthSign=c2VydmVyX3RpbWU9MTAvOS8yMDIyIDQ6NTI6MjYgUE0maGFzaF92YWx1ZT1zUU1NdVpiakZDMDBnNkVCVFRPL2ZBPT0mdmFsaWRtaW51dGVzPTI0MA==

Which very much resembles what Firefox F12 gives me when I play the video source on screen and hit F12:

https://h1.ustvgo.la/MSNBC/myStream/chunks.m3u8?nimblesessionid=315547546&wmsAuthSign=c2VydmVyX3RpbWU9MTAvOS8yMDIyIDQ6NTI6MjYgUE0maGFzaF92YWx1ZT1zUU1NdVpiakZDMDBnNkVCVFRPL2ZBPT0mdmFsaWRtaW51dGVzPTI0MA==

In fact they are identical!
And if I try it on a different site it returns the same as FierFox F12 does also for an actual m3u8 URL!

So the playlist.m3u8 file can be dropped onto VLC and it will *play* the current video stream.
Fine so far, now how can I use this in an ffmpeg command like the one I showed earlier to make ffmpeg record the stream onto an mp4 file?
The command I showed above works when I extract an url ending in m3u8 and use that url in my script command...
But the extracted URL for the problem site does not end in m3u8 and I believe this is why ffmpeg errors out.

Please advice on the way I can use this extracted "URL" as input in ffmpeg.

NOTICE:
When I tried to reach the videocontroller.info site from the page I reached with your link FireFox cannot find it...
Apparently the domain does not exist. Is this an abandoned add-on?

[BosseB]

https://h3.ustvgo.la/MSNBC/myStream/playlist.m3u8?wmsAuthSign=c2VydmVyX3RpbWU9MTAvOS8y...ludXRlcz0yNDA=

vs

https://h1.ustvgo.la/MSNBC/myStream/playlist.m3u8?wmsAuthSign=c2VydmVyX3RpbWU9MTAvOS8y...ludXRlcz0yNDA=

Do you see the difference?

Yes, one starts with h3 and the other with h1...
Apart from that they are exactly the same.

EDIT:
And they return the exact same response even when I change between h1 and h3

[Quint]

Yes, one starts with h3 and the other with h1...
Apart from that they are exactly the same.

No. 30 chars in the middle are different.

[BosseB]

Finally working with the manually extracted long m3u8 URL....

I must have mixed up sites or such (I am dealing with 14 sites and most are pretty straightforward) when trying to debug this...

But now when I use an ffmpeg command looking basically like this it does work:

ffmpeg -hide_banner -referer http://www.freeintertv.com/view/id-2565 \ 
-i https://h4.ustvgo.la/MSNBC/myStream/playlist.m3u8?wmsAuthSign=c2VydmVyX3RpbWU9MTAvOS8yMDIyIDc6NDk6NDggUE0maGFzaF92YWx1ZT1SZ3c3K05YUUNjVjdFL3JKR3R0NktRPT0mdmFsaWRtaW51dGVzPTI0MA== \
-vf scale=w=-4:h=480 -c:v libx264 -preset fast -crf 26 -c:a aac -t 3600 test7.mp4

Note: I used \ to split the long line for readability here on the forum but it turns out that when I use that on the command line it will not work, everything needs to be on a single line.

Now just remains the question if there is some hidden timeout in the very long -i argument, so will have to be retrieved in code?
Some sites have a static m3u8 URL, but I suspect that with this strange looking one it might be a short-lived URL.

If it will not last for say a week or so then I need to find a way to extract it using a bash command and use that at the time the recording is starting.
I will have to check back again in some hours and again tomorrow using the same command and see what happens...

So back to the original question:
How can I script the extraction of this very long input URL such that I can call an extractor function just before automatically starting the recording?
It will need to be a Linux script or command of some kind which will output the long URL.

[BosseB]

Time limit of m3u8 URL?
So now I have verified that the long m3u8 URL, which worked fine 10 hours ago no longer works, it generates errors.
But a newly extracted URL does work and it has for a couple of hours now.

Another observation:
The new URL created on my Windows 10 PC with FireFox does work on two different Ubuntu machines at home, but at the same time this URL does not work on an Ubuntu computer at my summer home 100 km away.
My home LAN and the summer home LAN (both on 250/250 fiber) are connected together by VPN so I can access the Ubuntu machines on both LAN sections using PuTTY the same way.
But these LAN sections are only connected for internal IP addresses, all other addresses are routed via their respective local gateways.
So they show different external IP addresses on communication with the Internet.

This URL seems to be only valid for my home network:

https://h4.ustvgo.la/MSNBC/myStream/playlist.m3u8?wmsAuthSign=c2VydmVyX3RpbWU9MTAvMTAvMjAyMiA2OjA3OjIyIEFNJmhhc2hfdmFsdWU9cnlEUzcrVThFdkU4N1d5bGdva3BHUT09JnZhbGlkbWludXRlcz0yNDA=

I wonder if anyone can advice how to decode the final section of this URL following ?wmsAuthSign=

UPDATE:
I found a site where I could decode the Base64 part.
It reads:

server_time=10/10/2022 6:07:22 AM&hash_value=ryDS7+U8EvE87WylgokpGQ==&validminutes=240

So the time to live seems to be 4 hours only...

This brings back the question in the thread title:

How can I automatically extract this m3u8 URL given the URL to the webpage hosting the streaming video player?

The solution has to be scripted such that it can be set to run at certain times, for example 5 min before a new recording starts.

[BosseB]

Now a bit more time has passed and the expiration happened.

So it is somewhere between 5 and 6 hours even though the URL itself states 240 minutes.
Even so a script that can extract the URL automatically is needed in order to record overnight etc.