VideoHelp Forum

+ Reply to Thread
Results 1 to 8 of 8
Thread
  1. I'm new here so don't know if this is the best place to ask but I am specifically looking for a way to do this from a Linux command prompt on a headless server, without using a browser or anything that requires a GUI or desktop environment, or software that only runs in Windows. And I'm pretty certain this will only work in the USA, and probably only if you have an account with a cable or satellite TV provider (some streaming accounts may also work but I don't know for sure).

    If I were using a desktop, in a web browser I would open the "Developer Tools" and under the Network tab put m3u8 (or possibly master.m3u8 to be more specific) in the search field. I would then go to https://www.nbc.com/live?brand=nbc&callsign=nbc and if necessary log in (it seems like it doesn't always ask you to log in if you have a previously stored cookie) and then in the Developer Tools pane select the very first m3u8 link that appears. It has a format similar to this:

    Code:
    https://tvenbcaffgroup4-x.akamaized.net/v1/master/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/PROD_NBC_AFFILIATE_G4_PASSTHROUGH/master.m3u8?aws.sessionId=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx&hdnts=exp=unixtimeseconds~acl=/v1/master/*!/Content/*~hmac=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    where the x's replace hexadecimal digits (which may be tied to a specific account, which is why I replaced them) and "unixtimeseconds" replaces a Unix timestamp (in seconds). I am try to find out is if there is a way to obtain this same URL from a Linux command prompt. The reason is that if you can get this URL it will play in VLC and possibly other media players, without the necessity of playing it in a browser window.

    This link does change every so often so you can't just grab it once and keep using it - by the next day it will have changed. It may change every few minutes for all I know. So I would like to find a way to get this link from the command prompt using one or more Linux commands, or even a shell script. I did try yt-dlp and it does not work, it falls back to the generic extractor and then fails, but I am not trying to save the stream anyway, I just need a way to get that m3u8 link. I'd be grateful for any suggestions as long as they don't require using a desktop environment or running Windows.
    Last edited by Prairie Dog; 1st Nov 2022 at 14:37.
    Quote Quote  
  2. Member
    Join Date
    Feb 2006
    Location
    United States
    Search Comp PM
    Originally Posted by Prairie Dog View Post
    I'm new here so don't know if this is the best place to ask but I am specifically looking for a way to do this from a Linux command prompt on a headless server, without using a browser or anything that requires a GUI or desktop environment, or software that only runs in Windows. And I'm pretty certain this will only work in the USA, and probably only if you have an account with a cable or satellite TV provider (some streaming accounts may also work but I don't know for sure).

    If I were using a desktop, in a web browser I would open the "Developer Tools" and under the Network tab put m3u8 (or possibly master.m3u8 to be more specific) in the search field. I would then go to https://www.nbc.com/live?brand=nbc&callsign=nbc and if necessary log in (it seems like it doesn't always ask you to log in if you have a previously stored cookie) and then in the Developer Tools pane select the very first m3u8 link that appears. It has a format similar to this:

    Code:
    https://tvenbcaffgroup4-x.akamaized.net/v1/master/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/PROD_NBC_AFFILIATE_G4_PASSTHROUGH/master.m3u8?aws.sessionId=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx&hdnts=exp=unixtimeseconds~acl=/v1/master/*!/Content/*~hmac=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    where the x's replace hexadecimal digits (which may be tied to a specific account, which is why I replaced them) and "unixtimeseconds" replaces a Unix timestamp (in seconds). I am try to find out is if there is a way to obtain this same URL from a Linux command prompt. The reason is that if you can get this URL it will play in VLC and possibly other media players, without the necessity of playing it in a browser window.

    This link does change every so often so you can't just grab it once and keep using it - by the next day it will have changed. It may change every few minutes for all I know. So I would like to find a way to get this link from the command prompt using one or more Linux commands, or even a shell script. I did try yt-dlp and it does not work, it falls back to the generic extractor and then fails, but I am not trying to save the stream anyway, I just need a way to get that m3u8 link. I'd be grateful for any suggestions as long as they don't require using a desktop environment or running Windows.
    try using the firefox add-on called the stream detector - https://www.videohelp.com/software/The-Stream-Detector
    Quote Quote  
  3. Originally Posted by october262 View Post
    Thanks but did you perhaps not read what I wrote?

    "I am specifically looking for a way to do this from a Linux command prompt on a headless server, without using a browser or anything that requires a GUI or desktop environment, or software that only runs in Windows."

    If I can get this to work it's going to be part of a bash script. If I have to have a browser open it won't work on the machine I want to run it on (which has no desktop and therefore no ability to run Firefox) but also if I have to go into a browser then I can just get the stream link using Developer Tools, therefopre I would not need any Firefox addons. But that would defeat the whole purpose.
    Quote Quote  
  4. I'll give you some pointers.


    Study the site code. Start from the top url and work your way down. The m3u8 link is within the code, your challenge is to find it. Once you have found the page that contains the link, work your way back up. Use curl together with the appropriate referer/header info to get the pages. Use a combination of awk, sed and grep to capture the appropriate urls from the pages.


    That's all there is to it.
    Quote Quote  
  5. Member
    Join Date
    Feb 2006
    Location
    United States
    Search Comp PM
    Originally Posted by Prairie Dog View Post
    Originally Posted by october262 View Post
    Thanks but did you perhaps not read what I wrote?

    "I am specifically looking for a way to do this from a Linux command prompt on a headless server, without using a browser or anything that requires a GUI or desktop environment, or software that only runs in Windows."

    If I can get this to work it's going to be part of a bash script. If I have to have a browser open it won't work on the machine I want to run it on (which has no desktop and therefore no ability to run Firefox) but also if I have to go into a browser then I can just get the stream link using Developer Tools, therefopre I would not need any Firefox addons. But that would defeat the whole purpose.
    see this reddit thread - https://www.reddit.com/r/DataHoarder/comments/bjpz2h/anyway_to_get_a_m3u8_directly_from_the_command/
    Quote Quote  
  6. Originally Posted by jack_666 View Post
    I'll give you some pointers.


    Study the site code. Start from the top url and work your way down. The m3u8 link is within the code, your challenge is to find it. Once you have found the page that contains the link, work your way back up. Use curl together with the appropriate referer/header info to get the pages. Use a combination of awk, sed and grep to capture the appropriate urls from the pages.


    That's all there is to it.
    I tried getting the 'https://www.nbc.com/live?brand=nbc&callsign=nbc' link using wget and it got a file of more than a million characters(!) but when I searched it for strings that would be present in the m3u8 link (such as "tvenbcaffgroup4-x" or "akamaized" or even "m3u8") they are not there. So I am not certain of how to proceed since there are MANY links and partial links in there, but none of them obviously what I am looking for.
    Quote Quote  
  7. I looked at it and it appears that guy was wanting to do what I'm trying to do but I don't see any responses that provide a clue. I see a mention of a headless browser and Selenium but my understanding of those is that they automate tasks on web pages and can read the results from web pages. So in other words you could (as an example) automate the process of going to Google, searching for a specific topic, and then capture the first result to a variable. The reason I don't see how that would work is because the link I am looking for is not on any web page. It only appears in the browser's "Developer Tools" and then only after you have started the stream playing.

    So, as I understand it, any automated process would need to go to 'https://www.nbc.com/live?brand=nbc&callsign=nbc', look to see if it is asking for a login and if so supply the login information, and click anything else necessary to start the stream playing - that much might be doable in a headless browser. But concurrently with that, it would need to go into the browser's menu structure and select "Developer Tools" (would that even exist in a "headless" browser?), click on the Network tab, put 'master.m3u8' in the search box, find the first result once it appears, right click on it and find the item to copy the link and click that, and then save that link to someplace it can be accessed in the calling script. It's that last part that I suspect may not work, because that information is not part of a web page, it is something that is in the browser's memory. It may have arrived as part of the setup for playing the stream, but it is never displayed in the browser's main window.

    And all that assumes that you can figure out how to even use a headless browser (and again I would ask if any of those even include Developer Tools) and Selenium. I ran into this once before when trying to figure out how to do something completely different and no one was ever able to point me to any simple setup instructions for use on a Linux server with no desktop. I guess I will ask, has anyone here ever installed and used a headless browser and Selenium under the server (no desktop) version of Ubuntu/Debian/Raspberry Pi OS, or any of their derivatives, for a task even remotely similar to this? If so, how did you get it working? Please keep in mind that not everyone that uses Linux is a Linux expert, and I'm definitely not (nor am I a programmer, unless you call having cobbled together a few short bash scripts programming) so at this point the workings of a headless browser are a complete mystery to me.

    Now if there were a web page where one could put in a link like 'https://www.nbc.com/live?brand=nbc&callsign=nbc' and it would cut through all the crap and give you the m3u8 link (kind of like what https://wheregoes.com/ does with shortened URLs) then I could see where a headless browser and Selenium might be useful. But unfortunately in this case I don't even think a service like that would work due to the need for a login (or perhaps a stored cookie).

    Beyond that, in that Reddit link all they talked about was what to do with the .m3u8 link once you already have it (for example using it with ffmpeg). I already know what to do with it so that's not the issue, the issue is that I want to be able to get it in a bash script that might be run as part of an unattended or remote process. But I see that guy never got a useful answer to his question either (unless I am completely missing it).
    Quote Quote  
  8. To emulate a real browser from command line I recommend you to use "curl" instead "wget".

    You must get the same response from curl than from a real browser if you send the same http headers and treat the cookies in the same way, curl allows to do that easily.

    Use curl with the same http headers (like User Agent) than the browser you get the m3u8.

    If you need to do a login you can do in curl too, and send data through POST.

    You can use mitmproxy to know exactly every byte that your browser sends to server.

    https://github.com/mitmproxy/mitmproxy
    Quote Quote  



Similar Threads