How to automatically extract the m3u8 stream URL from this site?

28th Jul 2022 05:45 #1

Member

I have asked a similar question before on this forum and now I am facing the same problem for another streaming URL...
The previous thread is here for reference.

The previous question dealt with a few MSNBC streams and now I want to use NBC NOW instead...

Note:
I am doing all of my video stuff on a Linux Ubuntu 20.04 server (no GUI on this computer) using bash scripts and ffmpeg.

The source page where there is an embedded video player is here:
Code:
http://www.freeintertv.com/view/id-2308
When I visit the page and use Firefox F12 function to show the activity I am able to manually copy the m3u8 URL but it changes over time so I need some automatic extraction instead...
Currently the URL is:
Code:
https://nbcnews3.akamaized.net/hls/live/723429/NBCNewsNowa418c506/clear/hdntl=exp=1659090950~acl=%2f*~data=hdntl~hmac=e4642ea72493dc7025914c12d3d7ab0b08bbfe19116c51a6d19e94963947b477/VIDEO_1_4596000.m3u8
But the exp=1659090950 part changes regularly and then the m3u8 becomes unusable.

So I am wondering if there is a way to script the extraction of the m3u8 URL for use with ffmpeg (usage shown in previous thread).
I also want to use it for direct viewing in my KODI media center. The m3u8 URL works fine for that purpose too.

TIA!

Quote

28th Jul 2022 06:26 #2

ElCap

Member

You need to find the request that generates that tokenised url.
In this case it's the request to "https://tokens.playmakerservices.com/"
Have a look at the response tab for this request.
You will need to script the above request and parse the response to get the fresh m3u8 url.

Quote

28th Jul 2022 07:35 #3

BosseB

Member

I am probably not really sure I get this.
If I open the url "https://tokens.playmakerservices.com/" in my browser I get to a message saying:
"Missing Authentication Token"

So I am probably totally misunderstanding what you suggest...
How do I "look at the response tab"???

Quote

28th Jul 2022 22:01 #4

jack_666

Member

Ok, we got this json response from https://tokens.playmakerservices.com/
Code:
{
  "akamai": [
    {
      "token": "hdnts=st=1659060522~exp=1659060642~acl=/hls/live/723429/NBCNewsNowa418c506/clear/*~hmac=de0e1d94b5b0a6074aa98af32b26d8ab40dca4f893b0c19c66e2c03776563c9c",
      "tokenizedUrl": "https://nbcnews3.akamaized.net/hls/live/723429/NBCNewsNowa418c506/clear/master.m3u8?hdnts=st=1659060522~exp=1659060642~acl=/hls/live/723429/NBCNewsNowa418c506/clear/*~hmac=de0e1d94b5b0a6074aa98af32b26d8ab40dca4f893b0c19c66e2c03776563c9c",
      "sourceUrl": "https://nbcnews3.akamaized.net/hls/live/723429/NBCNewsNowa418c506/clear/master.m3u8"
    }
  ]
}
tokenizedUrl :
https://nbcnews3.akamaized.net/hls/live/723429/NBCNewsNowa418c506/clear/master.m3u8?hd...e2c03776563c9c

Access Denied when trying to download it.

You don't have permission to access "http://nbcnews3.akamaized.net/hls/live/723429/NBCNewsNowa418c506/clear/master.m3u8?" on this server.

Any ideas as to how to download this master.m3u8 manifest?

Quote

28th Jul 2022 23:35 #5

10362227

Member

looks like the header is incompletesame here
https://www.liveone.com/video/7b86d1ab-562f-446d-8617-2e57ce4c946b

[Attachment 66153 - Click to enlarge]

Code:

curl 'https://lxl-vod-1.akamaized.net/v0000182280fe7ad/7b86d1ab-562f-446d-8617-2e57ce4c946b/master.m3u8?hdnts=st=1659069115~exp=1659069118~acl=/v0000182280fe7ad/7b86d1ab-562f-446d-8617-2e57ce4c946b/master.m3u8*~id=462250a9-9c6d-4253-9e0d-0358b4df4641~hmac=0921eee2dffd92395ed2f32470aced71f5de9cb9f050161c009368fd86307011' \
  -H 'Accept: */*' \
  -H 'Accept-Language: zh-CN,zh;q=0.9,en;q=0.8,ja;q=0.7,fr;q=0.6,ru;q=0.5' \
  -H 'Connection: keep-alive' \
  -H 'Origin: https://www.liveone.com' \
  -H 'Referer: https://www.liveone.com/' \
  -H 'Sec-Fetch-Dest: empty' \
  -H 'Sec-Fetch-Mode: cors' \
  -H 'Sec-Fetch-Site: cross-site' \
  -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36' \
  -H 'sec-ch-ua: ".Not/A)Brand";v="99", "Google Chrome";v="103", "Chromium";v="103"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-platform: "Windows"' \
  --compressed

but It says "You don't have permission to access https://***". Looks like the header is incomplete

Last edited by 10362227; 29th Jul 2022 at 00:35.

Quote

29th Jul 2022 02:23 #6

ElCap

Member

Originally Posted by jack_666

Ok, we got this json response from https://tokens.playmakerservices.com/

Code:

{
  "akamai": [
    {
      "token": "hdnts=st=1659060522~exp=1659060642~acl=/hls/live/723429/NBCNewsNowa418c506/clear/*~hmac=de0e1d94b5b0a6074aa98af32b26d8ab40dca4f893b0c19c66e2c03776563c9c",
      "tokenizedUrl": "https://nbcnews3.akamaized.net/hls/live/723429/NBCNewsNowa418c506/clear/master.m3u8?hdnts=st=1659060522~exp=1659060642~acl=/hls/live/723429/NBCNewsNowa418c506/clear/*~hmac=de0e1d94b5b0a6074aa98af32b26d8ab40dca4f893b0c19c66e2c03776563c9c",
      "sourceUrl": "https://nbcnews3.akamaized.net/hls/live/723429/NBCNewsNowa418c506/clear/master.m3u8"
    }
  ]
}

tokenizedUrl :
https://nbcnews3.akamaized.net/hls/live/723429/NBCNewsNowa418c506/clear/master.m3u8?hd...e2c03776563c9c

Access Denied when trying to download it.

You don't have permission to access "http://nbcnews3.akamaized.net/hls/live/723429/NBCNewsNowa418c506/clear/master.m3u8?" on this server.

http?? shouldnt it be https? ie.

Code:

https://nbcnews3.akamaized.net/hls/live/723429/NBCNewsNowa418c506/clear/master.m3u8?hdnts=st=1659060522~exp=1659060642~acl=/hls/live/723429/NBCNewsNowa418c506/clear/*~hmac=de0e1d94b5b0a6074aa98af32b26d8ab40dca4f893b0c19c66e2c03776563c9c

works fine for me.
i copied the full request from the browser into a cmd window.
the response is this:

Code:

{
   "akamai":[
      {
         "token":"hdnts=st=1659078990~exp=1659079110~acl=/hls/live/723429/NBCNewsNowa418c506/clear/*~hmac=86b526ad336d1255ba7bdfac4bc4c43e4d39d3dc2a42732e367f37b825567fe4",
         "tokenizedUrl":"https://nbcnews3.akamaized.net/hls/live/723429/NBCNewsNowa418c506/clear/master.m3u8?hdnts=st=1659078990~exp=1659079110~acl=/hls/live/723429/NBCNewsNowa418c506/clear/*~hmac=86b526ad336d1255ba7bdfac4bc4c43e4d39d3dc2a42732e367f37b825567fe4",
         "sourceUrl":"https://nbcnews3.akamaized.net/hls/live/723429/NBCNewsNowa418c506/clear/master.m3u8"
      }
   ]
}

Code:

>curl "https://nbcnews3.akamaized.net/hls/live/723429/NBCNewsNowa418c506/clear/master.m3u8?hdnts=st=1659078990~exp=1659079110~acl=/hls/live/723429/NBCNewsNowa418c506/clear/*~hmac=86b526ad336d1255ba7bdfac4bc4c43e4d39d3dc2a42732e367f37b825567fe4"
#EXTM3U
#EXT-X-PUBLISHED-TIME:2022-07-29T07:15:44.984Z
#EXT-X-INDEPENDENT-SEGMENTS
#EXT-X-MEDIA:TYPE=CLOSED-CAPTIONS,GROUP-ID="cc",LANGUAGE="en",NAME="Closed Captions",DEFAULT=YES,AUTOSELECT=YES,INSTREAM-ID="CC1"
#EXT-X-STREAM-INF:BANDWIDTH=8238624,AVERAGE-BANDWIDTH=6596000,CODECS="mp4a.40.2,avc1.640028",RESOLUTION=1920x1080,FRAME-RATE=29.97,CLOSED-CAPTIONS="cc"
hdntl=exp=1659165406~acl=%2f*~data=hdntl~hmac=d1acca20b28a8c0dfcffe1926015f9e1b1d3cf3e86bcf5ac6e5786bae50a5b3e/VIDEO_0_6596000.m3u8
.......

Quote

29th Jul 2022 02:54 #7

ElCap

Member

Originally Posted by 10362227

looks like the header is incompletesame here
https://www.liveone.com/video/7b86d1ab-562f-446d-8617-2e57ce4c946b

Code:

curl 'https://lxl-vod-1.akamaized.net/v0000182280fe7ad/7b86d1ab-562f-446d-8617-2e57ce4c946b/master.m3u8?hdnts=st=1659069115~exp=1659069118~acl=/v0000182280fe7ad/7b86d1ab-562f-446d-8617-2e57ce4c946b/master.m3u8*~id=462250a9-9c6d-4253-9e0d-0358b4df4641~hmac=0921eee2dffd92395ed2f32470aced71f5de9cb9f050161c009368fd86307011' \
  -H 'Accept: */*' \
  -H 'Accept-Language: zh-CN,zh;q=0.9,en;q=0.8,ja;q=0.7,fr;q=0.6,ru;q=0.5' \
  -H 'Connection: keep-alive' \
  -H 'Origin: https://www.liveone.com' \
  -H 'Referer: https://www.liveone.com/' \
  -H 'Sec-Fetch-Dest: empty' \
  -H 'Sec-Fetch-Mode: cors' \
  -H 'Sec-Fetch-Site: cross-site' \
  -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36' \
  -H 'sec-ch-ua: ".Not/A)Brand";v="99", "Google Chrome";v="103", "Chromium";v="103"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-platform: "Windows"' \
  --compressed

but It says "You don't have permission to access https://***". Looks like the header is incomplete

different issue.
the tokenised url is only valid for 3 seconds when you generate it - compare this expiry epoch time value "1659069118" with the time of your request.
https://www.epochconverter.com/

so you can either script it so that you parse the response into a downloading program within the 3 seconds or just use the submanifest link.

the tokenised urls for the submanifests should be valid for a day, so choose the submanifest url with the quality you want.
response from browser for master.m3u8

Code:

#EXTM3U
#EXT-X-VERSION:6
#EXT-X-STREAM-INF:BANDWIDTH=4681600,RESOLUTION=1280x720,CODECS="avc1.64001f,mp4a.40.2"
hdntl=exp=1659166510~acl=%2f*~id=6aed7bcc-caa9-4f67-814c-bece62d07ff0~data=hdntl~hmac=03ba37c053daebe1812e8ab2c7ea5af8cd111b80b54ec4f9413c6945ca69c384/index_720.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=2961200,RESOLUTION=854x480,CODECS="avc1.64001f,mp4a.40.2"
hdntl=exp=1659166510~acl=%2f*~id=6aed7bcc-caa9-4f67-814c-bece62d07ff0~data=hdntl~hmac=03ba37c053daebe1812e8ab2c7ea5af8cd111b80b54ec4f9413c6945ca69c384/index_480.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=1496000,RESOLUTION=640x360,CODECS="avc1.64001e,mp4a.40.2"
hdntl=exp=1659166510~acl=%2f*~id=6aed7bcc-caa9-4f67-814c-bece62d07ff0~data=hdntl~hmac=03ba37c053daebe1812e8ab2c7ea5af8cd111b80b54ec4f9413c6945ca69c384/index_360.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=6881600,RESOLUTION=1920x1080,CODECS="avc1.640028,mp4a.40.2"
hdntl=exp=1659166510~acl=%2f*~id=6aed7bcc-caa9-4f67-814c-bece62d07ff0~data=hdntl~hmac=03ba37c053daebe1812e8ab2c7ea5af8cd111b80b54ec4f9413c6945ca69c384/index_1080.m3u8

request for 1080p submanifest

Code:

curl "https://lxl-vod-1.akamaized.net/v0000182280fe7ad/7b86d1ab-562f-446d-8617-2e57ce4c946b/hdntl=exp=1659166510~acl=%2f*~id=6aed7bcc-caa9-4f67-814c-bece62d07ff0~data=hdntl~hmac=03ba37c053daebe1812e8ab2c7ea5af8cd111b80b54ec4f9413c6945ca69c384/index_1080.m3u8" -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.134 Safari/537.36"

Quote

29th Jul 2022 03:28 #8

BosseB

Member

Originally Posted by 10362227

looks like the header is incomplete same here
https://www.liveone.com/video/7b86d1ab-562f-446d-8617-2e57ce4c946b

[Attachment 66153 - Click to enlarge]

Could I ask which browser your screenshot is from and on which platform?
I do not have the same pop-up menu selection when I run the site and use F12 to get to the analysis page...

I am using FireFox on Windows10 when interactively working on the task, but the final target for my download script is Ubuntu Server 20.04.
So I need to get a command running in bash on Ubuntu, which will produce a working m3u8 URL I can use in my ffmpeg download command (and also in KODI for the direct view of the stream).

Quote

29th Jul 2022 03:33 #9

BosseB

Member

[QUOTE=ElCap;2663688]

Originally Posted by 10362227

looks like the header is incompletesame here
https://www.liveone.com/video/7b86d1ab-562f-446d-8617-2e57ce4c946b

request for 1080p submanifest

Code:

curl "https://lxl-vod-1.akamaized.net/v0000182280fe7ad/7b86d1ab-562f-446d-8617-2e57ce4c946b/hdntl=exp=1659166510~acl=%2f*~id=6aed7bcc-caa9-4f67-814c-bece62d07ff0~data=hdntl~hmac=03ba37c053daebe1812e8ab2c7ea5af8cd111b80b54ec4f9413c6945ca69c384/index_1080.m3u8" -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.134 Safari/537.36"

I need a request for 854x480 resolution...

When I run your curl command line in Ubuntu I get this:

Code:

#EXTM3U
#EXT-X-VERSION:6
#EXT-X-TARGETDURATION:10
#EXT-X-MEDIA-SEQUENCE:0
#EXT-X-PLAYLIST-TYPE:VOD
#EXT-X-INDEPENDENT-SEGMENTS
#EXTINF:10.010000,
1080/seg_1658429120_1080_0.ts
#EXTINF:10.010000,
1080/seg_1658429120_1080_1.ts
#EXTINF:10.010000,
1080/seg_1658429120_1080_2.ts
#EXTINF:10.010000,
1080/seg_1658429120_1080_3.ts
....
#EXTINF:10.010000,
1080/seg_1658429120_1080_285.ts
#EXTINF:10.010000,
1080/seg_1658429120_1080_286.ts
#EXTINF:10.010000,
1080/seg_1658429120_1080_287.ts
#EXTINF:10.010000,
1080/seg_1658429120_1080_288.ts
#EXTINF:10.010000,
1080/seg_1658429120_1080_289.ts
#EXTINF:10.010000,
1080/seg_1658429120_1080_290.ts
#EXTINF:2.936267,
1080/seg_1658429120_1080_291.ts
#EXT-X-ENDLIST

which does not give me any ideas on what to do with it....

Quote

29th Jul 2022 03:41 #10

ElCap

Member

Recreate the url for index_480.m3u8 then...

It's a m3u8 url, feed it into kodi

Have a read up about hls streaming and how the master and submanifests work.

Quote

29th Jul 2022 03:47 #11

ElCap

Member

@bosseb Wait a minute, aren't you after the m3u8 stream from http://www.freeintertv.com/view/id-2308?

That reply was for @10362227 for a different website

Quote

29th Jul 2022 04:44 #12

BosseB

Member

Originally Posted by ElCap

@bosseb Wait a minute, aren't you after the m3u8 stream from http://www.freeintertv.com/view/id-2308?

That reply was for @10362227 for a different website

That is correct, it is my target URL...

Quote

29th Jul 2022 04:48 #13

ElCap

Member

Originally Posted by ElCap

Originally Posted by jack_666

Ok, we got this json response from https://tokens.playmakerservices.com/

Code:

{
  "akamai": [
    {
      "token": "hdnts=st=1659060522~exp=1659060642~acl=/hls/live/723429/NBCNewsNowa418c506/clear/*~hmac=de0e1d94b5b0a6074aa98af32b26d8ab40dca4f893b0c19c66e2c03776563c9c",
      "tokenizedUrl": "https://nbcnews3.akamaized.net/hls/live/723429/NBCNewsNowa418c506/clear/master.m3u8?hdnts=st=1659060522~exp=1659060642~acl=/hls/live/723429/NBCNewsNowa418c506/clear/*~hmac=de0e1d94b5b0a6074aa98af32b26d8ab40dca4f893b0c19c66e2c03776563c9c",
      "sourceUrl": "https://nbcnews3.akamaized.net/hls/live/723429/NBCNewsNowa418c506/clear/master.m3u8"
    }
  ]
}

tokenizedUrl :
https://nbcnews3.akamaized.net/hls/live/723429/NBCNewsNowa418c506/clear/master.m3u8?hd...e2c03776563c9c

Access Denied when trying to download it.

You don't have permission to access "http://nbcnews3.akamaized.net/hls/live/723429/NBCNewsNowa418c506/clear/master.m3u8?" on this server.

http?? shouldnt it be https? ie.

Code:

https://nbcnews3.akamaized.net/hls/live/723429/NBCNewsNowa418c506/clear/master.m3u8?hdnts=st=1659060522~exp=1659060642~acl=/hls/live/723429/NBCNewsNowa418c506/clear/*~hmac=de0e1d94b5b0a6074aa98af32b26d8ab40dca4f893b0c19c66e2c03776563c9c

works fine for me.
i copied the full request from the browser into a cmd window.
the response is this:

Code:

{
   "akamai":[
      {
         "token":"hdnts=st=1659078990~exp=1659079110~acl=/hls/live/723429/NBCNewsNowa418c506/clear/*~hmac=86b526ad336d1255ba7bdfac4bc4c43e4d39d3dc2a42732e367f37b825567fe4",
         "tokenizedUrl":"https://nbcnews3.akamaized.net/hls/live/723429/NBCNewsNowa418c506/clear/master.m3u8?hdnts=st=1659078990~exp=1659079110~acl=/hls/live/723429/NBCNewsNowa418c506/clear/*~hmac=86b526ad336d1255ba7bdfac4bc4c43e4d39d3dc2a42732e367f37b825567fe4",
         "sourceUrl":"https://nbcnews3.akamaized.net/hls/live/723429/NBCNewsNowa418c506/clear/master.m3u8"
      }
   ]
}

Code:

>curl "https://nbcnews3.akamaized.net/hls/live/723429/NBCNewsNowa418c506/clear/master.m3u8?hdnts=st=1659078990~exp=1659079110~acl=/hls/live/723429/NBCNewsNowa418c506/clear/*~hmac=86b526ad336d1255ba7bdfac4bc4c43e4d39d3dc2a42732e367f37b825567fe4"
#EXTM3U
#EXT-X-PUBLISHED-TIME:2022-07-29T07:15:44.984Z
#EXT-X-INDEPENDENT-SEGMENTS
#EXT-X-MEDIA:TYPE=CLOSED-CAPTIONS,GROUP-ID="cc",LANGUAGE="en",NAME="Closed Captions",DEFAULT=YES,AUTOSELECT=YES,INSTREAM-ID="CC1"
#EXT-X-STREAM-INF:BANDWIDTH=8238624,AVERAGE-BANDWIDTH=6596000,CODECS="mp4a.40.2,avc1.640028",RESOLUTION=1920x1080,FRAME-RATE=29.97,CLOSED-CAPTIONS="cc"
hdntl=exp=1659165406~acl=%2f*~data=hdntl~hmac=d1acca20b28a8c0dfcffe1926015f9e1b1d3cf3e86bcf5ac6e5786bae50a5b3e/VIDEO_0_6596000.m3u8
.......

Have you looked at this reply then?

Quote

29th Jul 2022 05:17 #14

BosseB

Member

Originally Posted by ElCap

Have you looked at this reply then?

Yes, but I did not understand it...
I am looking for a command I could put into my script to generate the m3u8 URL to be used in ffmpeg to do the actual download (or display in KODI).

If I use FireFox to inspect (with F12) the video and select the item "VIDEO_1_4596000.m3u8" item I can use the rightclick menu to copy the URL.
If I then save this to a text file url-nbc.txt I can then read this URL inside my script and use it to download the video.
But it expires regularly so I would like a command that can do what FireFox and myself interactively do to retrieve it.

Or else I could put such a command as a crontab entry and write the URL to the file at some appropriate time for later use by my download script or KODI.

I am not very versed in the behind the scenes working of video transfers over the internet so I would never succeed in figuring it out by myself.

Last edited by BosseB; 29th Jul 2022 at 10:58.

Quote

29th Jul 2022 08:13 #15

jack_666

Member

the tokenised url is only valid for 3 seconds when you generate it

@ElCap, thank you very much. Three seconds, very interesting. I'll check it out. Thanks again.

Quote

29th Jul 2022 14:11 #16

BosseB

Member

I have checked how the m3u8 URL varies and it looks like this particular site's player is setting the expiration timestamp to 24 hours from the time the page was accessed.

I.e. if I open the page in Firefox and then use F12 and in Network/Media locate the m3u8 item I can right-click it and use "Copy URL" to get a long URL, which looks like this right now:
Code:
https://nbcnews3.akamaized.net/hls/live/723429/NBCNewsNowa418c506/clear/hdntl=exp=1659206812~acl=%2f*~data=hdntl~hmac=1274f3f1bb4374b9f1c92d1701bdad3995a746907c2af156d021bc30e66dc842/VIDEO_1_4596000.m3u8
This contains the expiration timestamp: 1659206812, which happens to be 24 hours ahead of the time the page was accessed.

So if I save the URL to a text file I can read this from within my video download script and also make a strm file for Kodi using this as the URL.
The downside to this is that I have to manually do this every 24 hours...

So the problem for me is how to do from within a script on Ubuntu what Firefox does on its F12 screen when I select the m3u8 URL and ask it copy the URL.
Once on the clipboard I can save it to a file...
Such a script could be called from crontab once per day and all would be fine.

Is this possible or not?
I am not able to solve the issue myself.

Quote

29th Jul 2022 17:26 #17

LZAA

Member

https://forum.videohelp.com/threads/405402-Need-to-script-extraction-of-m3u8-URL-on-a-...se-with-ffmpeg ?!

Quote

30th Jul 2022 03:09 #18

BosseB

Member

Originally Posted by LZAA

https://forum.videohelp.com/threads/405402-Need-to-script-extraction-of-m3u8-URL-on-a-...se-with-ffmpeg ?!

Yes, I know...
This works for the download but it does not produce a viable m3u8 url which can be used also in a KODI strm file to play the NBC stream live...
I am grateful for your previous solution for my automatic download system!

AFAICT the resulting output of the suggested command is time limited to 120 seconds after which it expires...

I have modified the NBC download script so it prints the content of the extracted URL to a text file just before it is used in the ffmpeg call.
And then I ran a test download to see what would happen.
This is the resulting URL used by ffmpeg:
Code:
https://nbcnews3.akamaized.net/hls/live/723429/NBCNewsNowa418c506/clear/master.m3u8?hdnts=st=1659162073~exp=1659162193~acl=/hls/live/723429/NBCNewsNowa418c506/clear/*~hmac=c4d8cda51a0f53ad18a19cc7e187e2b553307fa2159da037f48feffb402d34f1
This works with ffmpeg but not with KODI strm files...

Such a file looks like this, containing the m3u8 URL extracted manually using FireFox:
Code:
https://nbcnews3.akamaized.net/hls/live/723429/NBCNewsNowa418c506/clear/hdntl=exp=1659247084~acl=%2f*~data=hdntl~hmac=7ffa14d8dac47c29cadf3d63299ba6fac5a3592bb74526fa0cf671f3c9d64ec2/VIDEO_1_4596000.m3u8|referer=http://www.freeintertv.com/view/id-2308
and Kodi plays it just fine unless the expiration time has not been reached.

So what I am looking for is a way to automatically extract the same kind of m3u8 URL as can be found using FireFox on the F12 page...

Quote

30th Jul 2022 04:39 #19

LZAA

Member

'ffmpeg -i URL -c copy -f mpegts - | %cody% -' ?

Last edited by LZAA; 31st Jul 2022 at 16:42.

Quote

30th Jul 2022 18:55 #20

dark125

Member

Using python

Code:

#! /usr/bin/python
import os
import ffmpy
import requests
import re
import json

hds = {
    'accept': 'application/json, text/plain, */*',
    'accept-language': 'es,en-US;q=0.9,en;q=0.8,pt;q=0.7,cs;q=0.6,fr;q=0.5,zh-TW;q=0.4,zh;q=0.3',
    'content-type': 'application/json',
    'origin': 'https://www.nbcnews.com',
    'referer': 'https://www.nbcnews.com/',
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36'
    }

headers = {
    'Accept': '*/*',
    'Accept-Language': 'es,en-US;q=0.9,en;q=0.8,pt;q=0.7,cs;q=0.6,fr;q=0.5,zh-TW;q=0.4,zh;q=0.3',
    'Connection': 'keep-alive',
    'Sec-Fetch-Dest': 'empty',
    'Sec-Fetch-Mode': 'cors',
    'Sec-Fetch-Site': 'none',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36',
    'sec-ch-ua': '".Not/A)Brand";v="99", "Google Chrome";v="103", "Chromium";v="103"',
    'sec-ch-ua-mobile': '?0',
    'sec-ch-ua-platform': '"Windows"'
    }


  
url_canal = 'https://tokens.playmakerservices.com/'
data_raw = '{\n  "requestorId": "nbcnews",\n  "pid": "2007524",\n  "application": "NBCSports",\n  "version": "v1",\n  "platform": "desktop",\n  "token": "",\n  "resourceId": "",\n  "inPath": "false",\n  "authenticationType": "unauth",\n  "cdn": "akamai",\n  "url": "https://nbcnews3.akamaized.net/hls/live/723429/NBCNewsNowa418c506/clear/master.m3u8"\n}'


rs = requests.post(url_canal, headers=hds, data=data_raw)
url_ch = rs.json()['akamai'][0]['tokenizedUrl'].strip()

data_m3u8 = requests.get(url_ch, headers=headers, allow_redirects=True)
data_m3u8.encoding = 'ISO-8859-1'

#select resolution=896x504 
search_896_504 = re.findall(r'(^ht.*VIDEO_3.*8)',data_m3u8.text,re.MULTILINE)
stream = search_896_504[0]

#download
ff = ffmpy.FFmpeg(inputs={stream: '-headers "User-Agent:  Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36"'}, outputs={'NBC.mp4': '-acodec copy -vcodec copy -t 35'}, global_options="-y -hide_banner -loglevel warning")
ff.run()

Quote

30th Jul 2022 18:59 #21

dark125

Member

In php

Code:

<?php
//http://127.0.0.1/nbc/nbc.php
header("Content-Type: text/plain");
$url_canal = 'https://tokens.playmakerservices.com/';
$data = array('requestorId' =>'nbcnews','pid' =>'2007524','application' =>'NBCSports','version' =>'v1','platform' =>'desktop','token' =>'','resourceId' =>'','inPath' =>'false','authenticationType' =>'unauth','cdn' =>'akamai','url' =>'https://nbcnews3.akamaized.net/hls/live/723429/NBCNewsNowa418c506/clear/master.m3u8');

$data_raw= json_encode($data, JSON_PRETTY_PRINT | JSON_UNESCAPED_SLASHES);
$rs = Data($url_canal,$data_raw);
$url_ch = json_decode($rs,true)['akamai'][0]['tokenizedUrl'];


header("Location: $url_ch");

function Data($url,$data_raw){
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
    'accept: application/json, text/plain, */*',
    'accept-language: es,en-US;q=0.9,en;q=0.8,pt;q=0.7,cs;q=0.6,fr;q=0.5,zh-TW;q=0.4,zh;q=0.3',
    'content-type: application/json',
    'origin: https://www.nbcnews.com',
    'referer: https://www.nbcnews.com/',
    'user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36'
));
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $data_raw);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
$result = curl_exec($ch);
curl_close($ch);
return $result;
}

kodi leia strm

Code:

#EXTM3U
#KODIPROP:inputstreamaddon=inputstream.adaptive
#KODIPROP:inputstream.adaptive.manifest_type=hls
#KODIPROP:inputstream.adaptive.stream_headers=User-Agent=Mozilla%2F5.0%20%28Windows%20NT%2010.0%3B%20Win64%3B%20x64%29%20AppleWebKit%2F537.36%20%28KHTML%2C%20like%20Gecko%29%20Chrome%2F61.0.3163.100%20Safari%2F537.36
#EXTINF:-1,NBC
http://127.0.0.1/nbc/nbc.php

Quote

6th Aug 2022 04:51 #22

daint

Member

Hello,

Given that you're trying to stream the output to Kodi, you could do something like this...

1. Download and run a RTSP server such as Simple RTSP Server

2. Slightly amend the Python script provided above to publish the output to your newly created RTSP server, something like this:

Code:

#! /usr/bin/python
import os
import ffmpy
import requests
import re
import json

hds = {
    'accept': 'application/json, text/plain, */*',
    'accept-language': 'es,en-US;q=0.9,en;q=0.8,pt;q=0.7,cs;q=0.6,fr;q=0.5,zh-TW;q=0.4,zh;q=0.3',
    'content-type': 'application/json',
    'origin': 'https://www.nbcnews.com',
    'referer': 'https://www.nbcnews.com/',
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36'
    }

headers = {
    'Accept': '*/*',
    'Accept-Language': 'es,en-US;q=0.9,en;q=0.8,pt;q=0.7,cs;q=0.6,fr;q=0.5,zh-TW;q=0.4,zh;q=0.3',
    'Connection': 'keep-alive',
    'Sec-Fetch-Dest': 'empty',
    'Sec-Fetch-Mode': 'cors',
    'Sec-Fetch-Site': 'none',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36',
    'sec-ch-ua': '".Not/A)Brand";v="99", "Google Chrome";v="103", "Chromium";v="103"',
    'sec-ch-ua-mobile': '?0',
    'sec-ch-ua-platform': '"Windows"'
    }


  
url_canal = 'https://tokens.playmakerservices.com/'
data_raw = '{\n  "requestorId": "nbcnews",\n  "pid": "2007524",\n  "application": "NBCSports",\n  "version": "v1",\n  "platform": "desktop",\n  "token": "",\n  "resourceId": "",\n  "inPath": "false",\n  "authenticationType": "unauth",\n  "cdn": "akamai",\n  "url": "https://nbcnews3.akamaized.net/hls/live/723429/NBCNewsNowa418c506/clear/master.m3u8"\n}'


rs = requests.post(url_canal, headers=hds, data=data_raw)
url_ch = rs.json()['akamai'][0]['tokenizedUrl'].strip()

data_m3u8 = requests.get(url_ch, headers=headers, allow_redirects=True)
data_m3u8.encoding = 'ISO-8859-1'

#select resolution=896x504 
search_896_504 = re.findall(r'(^ht.*VIDEO_3.*8)',data_m3u8.text,re.MULTILINE)
stream = search_896_504[0]

#download
ff = ffmpy.FFmpeg(inputs={stream: '-headers "User-Agent:  Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36"'}, outputs={'rtsp://<RTSP SERVER IP ADDRESS>:8554/live.sdp': '-f rtsp -rtsp_transport tcp'}, global_options="-y -hide_banner -loglevel warning")
ff.run()

3. Point your Kodi strm file to your RTSP stream, the stream should be available at rtsp://<RTSP SERVER IP ADDRESS>:8554/live.sdp, providing you used the code above.

There may be alternative ways without a RTSP server, but this should work - although only tested for a few seconds.

Quote

23rd Nov 2022 10:48 #23

BosseB

Member

Originally Posted by daint

Hello,

Given that you're trying to stream the output to Kodi, you could do something like this...
...
There may be alternative ways without a RTSP server, but this should work - although only tested for a few seconds.

A bit late but:

Well that was not my main usage...
My main use is for a Linux server to automatically download live streaming news shows in 1 hour chunks from a number of sites.
Then it needs to use the m3u8 URL to feed into ffmpeg and this is OK to have in a resource file if it does not change.
But some sites change this m3u8 URL frequently, sometimes the time to live for the URL is only a few minutes or maybe at best times a full 24 hour day.
In such a case I need a working script that can be automatically run for a certain video site's main URL to produce the stream's m3u8 URL right before it will be used.

Once that is found it can be used both for my ffmpeg download script and in the strm Kodi file to allow direct playing on Kodi.

In both cases a recent automatic extraction of the working m3u8 URL is needed.

Note:
I have actually a second Linux server on a different location using the same scripting system to download the shows and it turns out that at least one of these show sites is using an m3u8 URL that is tied to the network address (I believe).

What happens is that if I manually extract the m3u8 using Firefox or similar on my home LAN and give that to the script on the Linux server on my home LAN then the download works fine. Note that in such a case I extract the m3u8 on my Windows laptop and write it to a file on the Linux server doing the download.

But If I give it to the second server on my remote LAN it fails miserably.
The only reason that could happen as far as I can understand is if the URL is somehow tied to the external IP address of the router for the network on which the device creating the URL is connected.
How else could this even happen?

Quote

7th Feb 2023 20:13 #24

googol

Member

Can you tell me how to automatically get the mpd link containing the token?

Quote

How to automatically extract the m3u8 stream URL from this site?

Thread Tools

Similar Threads

Is there anyway to extract webm url from agora based stream ?

How to extract and use an m3u8 URL to download from a 24/7 stream?

How to extract stream URL??

How to extract stream URL

how to grab m3u8 stream from this eurosport site