Automate these steps for m3u8 URL extraction from streaming site, how?

[BosseB]

I need to automate the extraction of a working m3u8 URL from a streaming site where the URL is only valid for 4 hours.

For a long time the URL lifetime was more than a week so I could get the m3u8 manually and then let it be used by my ffmpeg based download script during scheduled automatic video downloads.
But then they shorted the lifetime to only 240 minutes, which forces an extraction just before use and this is not possible at the requested times.

The m3u8 URL is needed for downloading the video on a Linux Ubuntu server, but if I extract the URL manually it only works for 240 minutes.
So I really need to automate the extraction in a Linux bash script such that it can be done just before the URL is needed.

Manual extraction is possible using the FireFox add-on "Video Controller", but then I have to start the video and then click the add-on icon and select the m3u8 playlist item and save it as a file. Not really workable.

The following manual sequence can be used in FireFox to get to the m3u8 URL:

Steps to reproduce:
-------------------
1) Open video player website "http://www.freeintertv.com/view/id-2565" in Firefox
2) Right click player shown on screen
3) Select: This Frame -> Open Frame in New tab 
    (This page opens: https://ustvgo.tv/player.php?stream=MSNBC)
4) Right-click background -> View Page Source (the source of the player is shown)
5) Search for 'playlist.m3u8' and find this line:
var hls_src='https://h3.ustvgo.la/MSNBC/myStream/playlist.m3u8?wmsAuthSign=c2VydmVyX3RpbWU9MS8xNC8yMDIzIDI6Mzk6NDAgUE0maGFzaF92YWx1ZT1OYzZGdGJkaUJBWlYyZWpPcEJET2J3PT0mdmFsaWRtaW51dGVzPTI0MA==';

The data following 'hls_src=' is the wanted m3u8 URL.

Can someone here help to create a bash script, which can repeat the above steps and save the final m3u8 URL to a file or display on the terminal screen? I have searched in vain for such a method...

Notice:
The needed script must execute as a bash script on an Ubuntu 20.04.5 LTS server (no GUI).

EDIT:
I found that there is a simpler way than the above steps (removes one step):

Steps to reproduce:
-------------------
1) Open video player website "http://www.freeintertv.com/view/id-2565" in Firefox
2) Right click player shown on screen
3) Select: This Frame -> View frame source
4) Search for 'playlist.m3u8' and find this line:
var hls_src='https://h3.ustvgo.la/MSNBC/myStream/playlist.m3u8?wmsAuthSign=c2VydmVyX3RpbWU9MS8xNC8yMDIzIDI6Mzk6NDAgUE0maGFzaF92YWx1ZT1OYzZGdGJkaUJBWlYyZWpPcEJET2J3PT0mdmFsaWRtaW51dGVzPTI0MA==';

The Base64 encoded string wmsAuthSign decodes into this:

server_time=1/14/2023 2:39:40 PM&hash_value=Nc6FtbdiBAZV2ejOpBDObw==&validminutes=240

[rabbitrcok]

I managed to do that with extenstion written on my own.
If you know how to write javascript you can try it yourself.

[BosseB]

I managed to do that with extenstion written on my own.
If you know how to write javascript you can try it yourself.

I do not know where to start, but did you do a browser extension?
If so is there any way I can get a look at the source?
Just the part that extracts the m3u8 URL...

If the code uses the browser API to do stuff then I'm afraid it will not fly since there is no desktop on the Ubuntu server computer and therefore no browser...
And it needs to be a script that can run in the Linux console (as an at scheduled task or in crontab) to extract the m3u8 URL and stuff it into a file.
Of course if it is in python it could also work, but I have never programmed python myself and have only very little exposure to java. None for javascript...

[BosseB]

I found that there is a simpler way than the above steps (removes one step):

Steps to reproduce:
-------------------
1) Open video player website "http://www.freeintertv.com/view/id-2565" in Firefox
2) Right click player shown on screen
3) Select: This Frame -> View frame source
4) Search for 'playlist.m3u8' and find this line:
var hls_src='https://h3.ustvgo.la/MSNBC/myStream/playlist.m3u8?wmsAuthSign=c2VydmVyX3RpbWU9MS8xNC8yMDIzIDI6Mzk6NDAgUE0maGFzaF92YWx1ZT1OYzZGdGJkaUJBWlYyZWpPcEJET2J3PT0mdmFsaWRtaW51dGVzPTI0MA==';

Today I have found that this does not quite cut it as a manual way of getting the m3u8 URL...

Seems like Firefox is caching the page sources such that even though they change it shows an older version of the frame source when one tries to get at the new token after several hours have passed...
Chrome seems not to do this, instead it shows a new token every time one opens the frame source.

But anyway in Firefox the "Video Controller" add-on picks up the new token when used to save the m3u8 URL but it has to be the "playlist" version and not the "chunks" version in order to work with ffmpeg.
So what could be done in order to read the frame source and extract the m3u8 URL from it using a combination of for example curl or wget and sed or the like so it can be scripted?

[jack_666]

The task comes down to the source of this token


wmsAuthSign=c2VydmVyX3RpbWU..............PTI0MA==


I have searched but was unable to determine how or where this token is generated.
(It's origin is supposedly https://ustvgo.tv as per the curl call)


Find the source and the challenge is over.


Decoded token :
[server_time=1/15/2023 8:19:43 PM&hash_value=g01sjhsL5d8jViKDEsHV1Q==&validminute s=240]
[This is a variable hash value (typical g01sjhsL5d8jViKDEsHV1Q== (b64) 834d6c8e1b0be5df2356228312c1d5d5 (hex) ) ]


It appears that there is an algorithm that takes the server current time plus 240 minutes. The hash of the computed result comprises part of the token. Therefore the number of valid minutes cannot be changed unless the hash value is also changed.


Anyone?

[BosseB]

I have used wget to download the sources for analysis but I apparently did no know how to run wget properly so I got way too many of them. In fact I got 2752 directories, one for each of the sources they serve out...
Now I have to delete all except the id-2565, which is the one I am interested in...

[ElCap]

this is the request url that returns the tokenised playlist m3u8 - it needs the referer header to work.

curl "https://ustvgo.tv/player.php?stream=MSNBC" -H "Referer: https://ustvgo.tv/msnbc/"

this is in the response data

var hls_src='https://k2.ustvgo.la/MSNBC/myStream/playlist.m3u8?wmsAuthSign=c2VydmVyX3RpbWU9MS8xNS8yMDIzIDExOjU2OjIyIFBNJmhhc2hfdmFsdWU9VTZPZXM4dWVPK1hKK2E0QnlQWldtUT09JnZhbGlkbWludXRlcz0yNDA=';

parse that url out of the response and use it

[jack_666]

curl "https://ustvgo.tv/player.php?stream=MSNBC" -H "Referer: https://ustvgo.tv/msnbc/"


A big thank you to @ElCap. I really appreciate your kindly assistance. You are amazing.

[jack_666]

@BosseB

use this code to automate the capture of the playlist.m3u8


This command is for windows but you can easily adapt it to work on linux

curl -ks  "https://ustvgo.tv/player.php?stream=MSNBC" -H "Referer: https://ustvgo.tv/msnbc/" | awk "BEGIN{FS=OFS=\"\047\"} /hls_src=/{print \"\042\"$2\"\042\"}"

and the sample response is ...

"https://h4.ustvgo.la/MSNBC/myStream/playlist.m3u8?wmsAuthSign=c2VydmVyX3RpbWU9MS8xNi8y MDIzIDE6NDk6NDYgQU0maGFzaF92YWx1ZT1PZlk2ZmxnZWdtM2 ZZOU1TQWJySmtnPT0mdmFsaWRtaW51dGVzPTI0MA=="

This would not be possible, by me, without the kind help of ElCap

[lomero]

wow, one post better than the other !

great job guys

[BosseB]

this is the request url that returns the tokenised playlist m3u8 - it needs the referer header to work.

curl "https://ustvgo.tv/player.php?stream=MSNBC" -H "Referer: https://ustvgo.tv/msnbc/"

this is in the response data

var hls_src='https://k2.ustvgo.la/MSNBC/myStream/playlist.m3u8?wmsAuthSign=c2VydmVyX3RpbWU9MS8xNS8yMDIzIDExOjU2OjIyIFBNJmhhc2hfdmFsdWU9VTZPZXM4dWVPK1hKK2E0QnlQWldtUT09JnZhbGlkbWludXRlcz0yNDA=';

parse that url out of the response and use it

THANKS A MILLION!

So I created this test script:

#!/bin/bash
curl "https://ustvgo.tv/player.php?stream=MSNBC" -H "Referer: https://ustvgo.tv/msnbc/" --output 2565.txt
echo
CMD="grep \"hls_src=\" 2565.txt | sed -e 's/.*src=\(.*\);.*/\1/'"
eval "$CMD"
echo
echo "Done"

It produces this output:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  5202    0  5202    0     0   6098      0 --:--:-- --:--:-- --:--:--  6091

'https://h5.ustvgo.la/MSNBC/myStream/playlist.m3u8?wmsAuthSign=c2VydmVyX3RpbWU9MS8xNi8yMDIzIDEyOjIwOjUyIFBNJmhhc2hfdmFsdWU9bjYwcjA3WXROMWVNcGxDRi9lYTdIQT09JnZhbGlkbWludXRlcz0yNDA='

Done

But I cannot for the life of me figure out how to get rid of the single quotes at the start and end of the URL...

[BosseB]

But I cannot for the life of me figure out how to get rid of the single quotes at the start and end of the URL...

Well I solved it this way in my script:

#!/bin/bash
# This solution to finding the m3u8 of a site that puts a lifetime on the URL is based on a discussion
# on the web here:
# https://forum.videohelp.com/threads/408224-Automate-these-steps-for-m3u8-URL-extraction-from-streaming-site-how

OUTPUTFILE="$HOME/www/MSNBC/url_m3u8-7.txt"
TMPFILE="2565.tmp"

CMD="curl \"https://ustvgo.tv/player.php?stream=MSNBC\" -H \"Referer: https://ustvgo.tv/msnbc/\" --output $TMPFILE"
eval "$CMD" #Get m3u8 into temp file
echo #Since curl leaves output without a final linefeed
#Get m3u8 url line from source temp file:
CMD="grep \"hls_src=\" $TMPFILE | sed -e 's/.*src=\(.*\);.*/\1/'"
M3U8=$(eval "$CMD")
eval "rm $TMPFILE"  #No need for this anymore
#Remove first and last char from URL:
M3U8=${M3U8#"'"}
M3U8=${M3U8%"'"}
#Save to m3u8 url-file used by download script
eval "echo $M3U8 > $OUTPUTFILE"
echo "Done"

Now I just want to also get away from the temp file...
But I guess piping will not work since curl spews out a lot of user feedback when running...

[ElCap]

command: curl -s "https://ustvgo.tv/player.php?stream=MSNBC" -H "Referer: https://ustvgo.tv/msnbc/" | grep "hls_src=" $TMPFILE | awk -F\' '{ print $2 }
output: https://h4.ustvgo.la/MSNBC/myStream/playlist.m3u8?wmsAuthSign=c2VydmVyX3RpbWU9MS8xNy8yMDIzIDQ6NTU6NDAgQU0maGFzaF92YWx1ZT1lWE95eVQ5V0NIeUNjWVVkWkpsWW5RPT0mdmFsaWRtaW51dGVzPTI0MA==

-s is silent mode for curl and awk extracts the url without single quotes

[BosseB]

command: curl -s "https://ustvgo.tv/player.php?stream=MSNBC" -H "Referer: https://ustvgo.tv/msnbc/" | grep "hls_src=" $TMPFILE | awk -F\' '{ print $2 }
output: https://h4.ustvgo.la/MSNBC/myStream/playlist.m3u8?wmsAuthSign=c2VydmVyX3RpbWU9MS8xNy8yMDIzIDQ6NTU6NDAgQU0maGFzaF92YWx1ZT1lWE95eVQ5V0NIeUNjWVVkWkpsWW5RPT0mdmFsaWRtaW51dGVzPTI0MA==

-s is silent mode for curl and awk extracts the url without single quotes

I must have missed something...
When I run the command as copied from your post and pasted into a command window on Linux:

$ curl -s "https://ustvgo.tv/player.php?stream=MSNBC" -H "Referer: https://ustvgo.tv/msnbc/" | grep "hls_src=" $TMPFILE | awk -F\' '{ print $2 }
> ^C

And if I remove $TEMPFILE, which is not defined in the command, it ends up just showing a single > on the next line...

So as a last resort I created a new script file and put this into it:

#!/bin/bash
TEMPFILE="tempfile.txt"
curl -s "https://ustvgo.tv/player.php?stream=MSNBC" -H "Referer: https://ustvgo.tv/msnbc/" | grep "hls_src=" $TMPFILE | awk -F\' '{ print $2 }

But when I run this the following is displayed:

$ ./test
./test: line 3: unexpected EOF while looking for matching `''
./test: line 5: syntax error: unexpected end of file

If I remove the final | awk... then the command produces a result that is the expected full line:

$ curl -s "https://ustvgo.tv/player.php?stream=MSNBC" -H "Referer: https://ustvgo.tv/msnbc/" | grep "hls_src="
var hls_src='https://h4.ustvgo.la/MSNBC/myStream/playlist.m3u8?wmsAuthSign=c2VydmVyX3RpbWU9MS8xNy8yMDIzIDc6MjU6MzEgQU0maGFzaF92YWx1ZT1YaTZwWktYRXJ6NnYwRUhVTXNzbUF3PT0mdmFsaWRtaW51dGVzPTI0MA==';

So it seems to be the invocation of awk that breaks the command...

[ElCap]

oops, there shouldnt be "$TMPFILE" and i missed the last single quote

try this

curl -s "https://ustvgo.tv/player.php?stream=MSNBC" -H "Referer: https://ustvgo.tv/msnbc/" | grep "hls_src=" | awk -F\' '{ print $2 }'

works on ubuntu20

[BosseB]

oops, there shouldnt be "$TMPFILE" and i missed the last single quote

try this

curl -s "https://ustvgo.tv/player.php?stream=MSNBC" -H "Referer: https://ustvgo.tv/msnbc/" | grep "hls_src=" | awk -F\' '{ print $2 }'

works on ubuntu20

OK, thanks.
After googling a bit I tested the following awk statement and it also works:

curl -s "https://ustvgo.tv/player.php?stream=MSNBC" -H "Referer: https://ustvgo.tv/msnbc/" | grep "hls_src=" | awk -F[\'] '{ print $2 }'

What does the brackets [ ] do in this case?

[ElCap]

What does the brackets [ ] do in this case?

its just a different way of specifying the field separators - generally used if you need more than one

[BosseB]

What does the brackets [ ] do in this case?

its just a different way of specifying the field separators - generally used if you need more than one

Thanks, now simplified the retrieval script, which is run from cron once per hour:

#!/bin/bash
# This solution is based on a discussion on the web here:
# https://forum.videohelp.com/threads/408224-Automate-these-steps-for-m3u8-URL-extraction-from-streaming-site-how

M3U8SRC7="$HOME/www/MSNBC/url_m3u8-7.txt"
M3U8=$(curl -s "https://ustvgo.tv/player.php?stream=MSNBC" -H "Referer: https://ustvgo.tv/msnbc/" | grep "hls_src=" | awk -F\' '{ print $2 }')

#Save to m3u8 url-files used by download script
eval "echo $M3U8 > $M3U8SRC7"
echo "Done"

Thanks again for your help!

[hencha]

What does the brackets [ ] do in this case?

its just a different way of specifying the field separators - generally used if you need more than one

Thanks, now simplified the retrieval script, which is run from cron once per hour:

#!/bin/bash
# This solution is based on a discussion on the web here:
# https://forum.videohelp.com/threads/408224-Automate-these-steps-for-m3u8-URL-extraction-from-streaming-site-how

M3U8SRC7="$HOME/www/MSNBC/url_m3u8-7.txt"
M3U8=$(curl -s "https://ustvgo.tv/player.php?stream=MSNBC" -H "Referer: https://ustvgo.tv/msnbc/" | grep "hls_src=" | awk -F\' '{ print $2 }')

#Save to m3u8 url-files used by download script
eval "echo $M3U8 > $M3U8SRC7"
echo "Done"

Thanks again for your help!

M3U8SRC7="$HOME/www/MSNBC/url_m3u8-7.txt"

this is mean create folder before on /home/www/MSNBC/ before running that from cron ? or can make to other destination folder ?

[BosseB]

#!/bin/bash
# This solution is based on a discussion on the web here:
# https://forum.videohelp.com/threads/408224-Automate-these-steps-for-m3u8-URL-extraction-from-streaming-site-how

M3U8SRC7="$HOME/www/MSNBC/url_m3u8-7.txt"
M3U8=$(curl -s "https://ustvgo.tv/player.php?stream=MSNBC" -H "Referer: https://ustvgo.tv/msnbc/" | grep "hls_src=" | awk -F\' '{ print $2 }')

#Save to m3u8 url-files used by download script
eval "echo $M3U8 > $M3U8SRC7"
echo "Done"

M3U8SRC7="$HOME/www/MSNBC/url_m3u8-7.txt"

this is mean create folder before on /home/www/MSNBC/ before running that from cron ? or can make to other destination folder ?

That folder exists in my Ubuntu and is part of the automatic download system where other scripts look for this file when downloading from source #7.
And notice that the path specifier is $HOME, not /home!
In my case $HOME == /home/bosse but on some other machines I may use different logon names so I use $HOME to get to the right place always.

[BosseB]

--- ALERT! ---
A few days ago they changed the website such that the command shown above no longer works to extract the m3u8 URL!
curl returns a "404 Not Found" error!

My FireFox add-on manages to extract a m3u8 URL from this but it is not useful since it leads to a completely different newscast (CBS instead of MSNBC).

I guess nothing can be done about this except give up....

[codehound]

I guess nothing can be done about this except give up....

Or stop posting all this in plain view on public forums ? It doesn't take much working out...

[Sandy sai]

hello friends,

i trying to automatically extract m3u8 url from this site

This is free to use with a user and pass

https://www.sunnxt.com/live/64067/sun-news

https://suntvlive.akamaized.net/hls/live/2093435/SunNews/master.m3u8?hdnea=st=1674575040~exp=1674575340~acl=!*/2093435/SunNews/*~hmac=72e401491370d92e6f38dae4d190b4a0ae99eb062c74f3c6501c2d48ca57b890&PlayBackId=2456682745&cid=38926&country=IN&userid=2886272&nid=0&q=&bw=&dw=0&op=SUNNXT&did=0

it got expired after 5 mins

this part not necessary

&PlayBackId=2456682745&cid=38926&country=IN&userid=2886272&nid=0&q=&bw=&dw=0&op=SUNNXT&did=0

i read this forum but i am not well known about curl commands ,aslo the mega link posted by LZAA aslo not working

https://forum.videohelp.com/threads/405402-Need-to-script-extraction-of-m3u8-URL-on-a-live-video-for-use-with-ffmpeg

i aslo gone through the all json response but it not had enough info, to generate the master.m3u8 playlist

can u guys help me to get m3u8 with a small code

[jack_666]

The server USTVGO is dead

[Sandy sai]

Hello friend jack_666 help me pls posted above
Site is www.sunnxt.com

[jack_666]

Site is www.sunnxt.com ... unable to connect.

If this site requires log in credentials then I'm afraid that you are out of luck.

[Sandy sai]

Indian site.yes it requires login but free to use

[Sandy sai]

unable to connect.

by using chrome and type the sunnxt then it loads

[Attachment 68837 - Click to enlarge]

by clicking the url(www.sunnxt.com) in videoforum not working

[Attachment 68838 - Click to enlarge]

[jack_666]

The site now says


403 Forbidden

[Sandy sai]

works with Indian vpn

previously forum members helped me how to download using hlsdl

(https://forum.videohelp.com/threads/406832-how-to-download-this-stream#post2666166)

now i need to automate this because it expires in 5mins

kindly help me friends