Is it possible to download from this streaming site?

[BosseB]

I am watching MSDN news, which is made available as a stream (certain times of day only) on this site.
(Note that it is usually not streaming during USA nights...)
They used to have a YouTube channel too, where youtube-dl worked so I could automate the downloads on a Ubuntu 20.04 Server during the night in Sweden (evening shows in USA). Then I could view them in the morning.

But the Youtube channel was taken down a week or so ago so now the only available source is the video player on the home page.

I have found no way to download from this so I have been forced to set up a screen capture on a computer running over night using ffmpeg in a script. Not so good because the computer is blocked for other use and no sound except the playing video may be generated because it gets into the captured video too...

So I wonder if there is a way to actually download the stream data directly from the URL, like youtube-dl can for streams on YouTube?

[LZAA]

https://www.youtube.com/watch?v=dmny0wOF3W8&feature=emb_logo

???

[BosseB]

https://www.youtube.com/watch?v=dmny0wOF3W8&feature=emb_logo
???

No, that brings you to the CNN stream!
What I am after is the MSNBC stream, and that has been discontinued on YouTube, it was like the CNN stream for many months after I found this site. But about 2 weeks ago they discontinued the MSNBC stream on YouTube due to complaints from NBC, I guess.
But on the base page I linked to there are two video players, the top one is MSNBC and is the one I want to download from.
The other is CNN and plays the exact same content as the YouTube link on the page does.

When the MSNBC YouTube link was active it worked fine with youtube-dl and I could script getting the nightly (in Europe) evening shows.
Not so anymore, now all there is is this videoplayer page object which plays streamed MSNBC.
What I need is a way to download from the source feeding the player but using for example youtube-dl or ffmpeg directly.

[october262]

https://www.youtube.com/watch?v=dmny0wOF3W8&feature=emb_logo
???

No, that brings you to the CNN stream!
What I am after is the MSNBC stream, and that has been discontinued on YouTube, it was like the CNN stream for many months after I found this site. But about 2 weeks ago they discontinued the MSNBC stream on YouTube due to complaints from NBC, I guess.
But on the base page I linked to there are two video players, the top one is MSNBC and is the one I want to download from.
The other is CNN and plays the exact same content as the YouTube link on the page does.

When the MSNBC YouTube link was active it worked fine with youtube-dl and I could script getting the nightly (in Europe) evening shows.
Not so anymore, now all there is is this videoplayer page object which plays streamed MSNBC.
What I need is a way to download from the source feeding the player but using for example youtube-dl or ffmpeg directly.

play the video, press F12 on your keyboard to bring up the debug menu, go to network, you should see master.m3u8, copy the URL
and feed it into youtube-dl to download the video.

[LZAA]

Use 'streamlink', not 'youtude-dl'.

[jagabo]

https://hlss.goodgame.ru/hls/183955.m3u8

[BosseB]

I tried to use youtube-dl, but when I tried to break the download with ctrl-C it resulted in myvideo.part and no mp4 file.
So cannot play it.

I will try streamlink too, but I don't know the call syntax for streamlink and its help does not contain some simple examples...
Off to google.
Maybe streamlink URL outputfile?

[LZAA]

streamlink URL best -o video.ts

[BosseB]

streamlink URL best -o video.ts

Thanks now running.
How do I stop download from live stream at the end? Ctrl-C or q or something else?

[jagabo]

I tried to use youtube-dl, but when I tried to break the download with ctrl-C it resulted in myvideo.part and no mp4 file.
So cannot play it.

You can often just rename the file with .mp4 and it will play. Or:

youtube-dl --hls-use-mpegts --no-part -o output.ts https://hlss.goodgame.ru/hls/183955.m3u8

You can Ctrl+C or Ctrl+Break at any time. You can even play the output.ts while it's still being downloaded.

[BosseB]

I tried to use youtube-dl, but when I tried to break the download with ctrl-C it resulted in myvideo.part and no mp4 file.
So cannot play it.

You can often just rename the file with .mp4 and it will play. Or:

youtube-dl --hls-use-mpegts --no-part -o output.ts https://hlss.goodgame.ru/hls/183955.m3u8

You can Ctrl+C or Ctrl+Break at any time. You can even play the output.ts while it's still being downloaded.

So I tried it and noticed that in the messages flying by on the screen I found this:

[https @ 0x564ab91d10c0] HTTP error 402 Payment Required   <=== Hmm, how can this be handled?
[hls @ 0x564ab8e12cc0] Failed to open next segment 163187 of playlist 0
[hls @ 0x564ab8e12cc0] Opening 'https://hlss.goodgame.ru/hls/183955-163187.ts' for reading03x
[https @ 0x564ab99b40c0] Opening 'https://hlss.goodgame.ru/hls/183955.m3u8' for reading

The file played with short interrutions along the way, irritating...

I only changed the ts extension to mp4 for the output file.
After 5.5 minutes the file is rather big already 151 Mbytes, which is a lot more than my usual downloads.
I normally download at 480p resolution.
But it seems like this stream does not carry any info on available resolutions...

[jagabo]

Why would you change the extension to mp4? It's a transport stream, .TS. As far as I can tell there's only a 720p version of the MSNBC stream. Even in a web browser that's all you can get.

[BosseB]

Why would you change the extension to mp4? It's a transport stream, .TS. As far as I can tell there's only a 720p version of the MSNBC stream. Even in a web browser that's all you can get.

The reason I changed it is that I want the file on disk to be compressed, i.e. smaller in size.
AFAIU the downloaders (youtube-dl or ffmpeg) adjusts the compression according to the file type. But I may be wrong...
Recoding it after the download is a time consuming operation.

[jagabo]

If you want to downscale to 480p and re-encode to mp4 you want a command line like:

youtube-dl "https://hlss.goodgame.ru/hls/183955.m3u8" -o - | ffmpeg -i - -vf scale=w=856:h=480 -c:v libx264 -preset veryfast -crf 25 -c:a copy output.mp4

[jagabo]

Here's how you can download with ffmpeg :

ffmpeg -user_agent "Mozilla"  -i https://hlss.goodgame.ru/hls/183955.m3u8 -c copy output.ts

Or to resize and reencode to mp4:

ffmpeg -user_agent "Mozilla"  -i https://hlss.goodgame.ru/hls/183955.m3u8 -vf scale=w=856:h=480 -c:v libx264 -preset veryfast -crf 25 -c:a copy output.mp4

Terminate by pressing q on the keyboard.

Or play the video with ffplay:

ffplay -user_agent "anything"  -i https://hlss.goodgame.ru/hls/183955.m3u8

Or to play in VLC, put this in a local m3u8 file:

#EXTM3U

#EXTINF:0 MSNBC
#EXTVLCOPT:http-user-agent=Android
https://hlss.goodgame.ru/hls/183955.m3u8

[BosseB]

Just to clarify so I understand it correctly, if I use this command:

ffmpeg -user_agent "Mozilla"  -i https://hlss.goodgame.ru/hls/183955.m3u8 -vf scale=w=856:h=480 -c:v libx264 -preset veryfast -crf 25 -c:a copy output.mp4

then the size conversion from 1280x720 (which is the only resolution offered by the site) to 856x480 is done on the fly while the video downloads?

If so it is a real boon since the "after the download" conversion takes a long time...

Since I am using Ubuntu for the downloads:

To get the right time span I need to hit q with ffmpeg, but if I script this and put it into an at job how can I send the q into ffmpeg?
Currently when I have used youtube-dl in scripts I set the download time by using the timeout command:

timeout --signal=2 65m youtube-dl.....

This sends a Ctrl-C to youtube-dl after 65 min which interrupts youtube-dl and it cleans up the download by finalizing the video.
Probably youtube-dl sends q to ffmpeg...

I like to have the download+conversion scripted into at jobs so they are not forgotten...

[BosseB]

REPLY TO SELF:
Obviously if one knows the length of the download one just puts

-t  <length in seconds>

just before the output file name...

@jagabo:
I am really grateful for your reply, it seems to solve most of my problems!

[jagabo]

Just to clarify so I understand it correctly, if I use this command:

ffmpeg -user_agent "Mozilla"  -i https://hlss.goodgame.ru/hls/183955.m3u8 -vf scale=w=856:h=480 -c:v libx264 -preset veryfast -crf 25 -c:a copy output.mp4

then the size conversion from 1280x720 (which is the only resolution offered by the site) to 856x480 is done on the fly while the video downloads?

Yes.

If so it is a real boon since the "after the download" conversion takes a long time...

Be aware that a slow computer might not be able to keep up with a live broadcast. That's why I chose the veryfast preset (fast, ok quality) rather than something slower. To get the best quality (per bitrate) you may want to do a slower re-compression after downloading.

[BosseB]

Or to resize and reencode to mp4:

ffmpeg -user_agent "Mozilla"  -i https://hlss.goodgame.ru/hls/183955.m3u8 -vf scale=w=856:h=480 -c:v libx264 -preset veryfast -crf 25 -c:a copy output.mp4

Just a final detail which I am wondering about:
In the ffmpeg command the resize argument is this:

-vf scale=w=856:h=480

But given that 1280 * 480 / 720 = 53.3333, would not the proper horizontal even number argument be 854?
Otherwise the aspect ratio would have to be changed and that might be a harder computational task, or not?

Is there a way to use something like a wildcard for the horizontal so ffmpeg can figure out by itself the proper output width of the resulting video?

[jagabo]

I just picked the nearest mod8 width. You're not going to notice the difference between 854x480 and 856x480. You can use a negative value to specify the nearest mod N that matches the other dimension:

-vf scale=w=-4:h=480

That will deliver the nearest mod4 width for a height of 480, 852x480.

-vf scale=w=640:h=-4

That will deliver the the nearest mod4 height for a width of 640, 640x360.

[BosseB]

Now that I have downloaded a number of videos using the command shown here I have an additional question:
It seems like I am getting a video-audio time difference so lip-sync is bad.
To fix this I can apply an ffmpeg command like this (for a 340 ms difference):

ffmpeg -i input.mp4 -itsoffset -0.34 -i input.mp4 -map 1:v -map 0:a -c copy output.mp4

Is it possible to add this processing into the capture command? It seems like the time difference is always about 350 ms...
I am using this capture cmd in my script at the moment:

ffmpeg -user_agent \"Mozilla\"  -i $VIDEOURL -vf scale=w=-4:h=480 -c:v libx264 -preset veryfast -crf 25 -c:a copy -t $CAPTURETIME $TARGETFILE

If so what is the correct syntax?

[jagabo]

Using itsoffset requires that you have two source videos. Opening the stream twice would use twice the bandwidth and you can't be sure the two streams will be synchronized. If you don't mind re-encoding the audio you can use the adelay audio filter:

ffmpeg -y -user_agent "Mozilla" -i https://hlss.goodgame.ru/hls/183955.m3u8 -vf scale=w=-4:h=480 -c:v libx264 -preset veryfast -crf 25 -c:a aac -af adelay=350ms:all=1 -async 1 output.mp4

[BosseB]

I thought as much given the two sources even if they are one and the same...

So I would probably be better off launching the audio delay from the download script once it has finished downloading.
That would be the more stable solution, I think.
I have noted that the audio resync is pretty quick, only takes a couple of seconds on a downloaded 1 hour video, so it would not be a big problem.
I have a script already that does this so I just have to call it from the download script to make this happen.

Thanks for the information!

[BosseB]

I have now used the commands suggested here to download the video stream to file for a number of time slots (using the at command).
What I have found is that the download itself seems to incur irregular pauses in the resulting file possibly caused by computational overload of the computer.
Since the conversion to ffmpeg and the simultaneous size change might be very computationally intensive I am now looking at the different commands:

1) Capture the video to file in the raw ts format. You already gave that command in the replies above.

ffmpeg -user_agent "Mozilla"  -i https://hlss.goodgame.ru/hls/183955.m3u8 -c copy output.ts

This seems to work fine but results in a very big ts file at 1280x720 resolution. As far as I can tell the audio lapses are gone.

2) Convert the data now on disk from ts to mp4, I found this to just convert into compressed mp4 without the change of scale:

ffmpeg -i input.ts -map 0 -c copy output.mp4

This runs pretty fast, but it seems not to apply any extra compression, the output file is almost the same size as the input.

3) Convert the data now on disk to mp4 compression AND change of scale:

ffmpeg -i input.ts -vf scale=w=-4:h=480 -c:v libx264 -preset veryfast -crf 25 -c:a copy output.mp4

This took 56% of the playing time of a video and produced a file reduction from 546 to 58 Mb...

4) Only compress the video:

ffmpeg -i input.ts -c:v libx264 -preset veryfast -crf 25 -c:a copy output720.mp4

This took a lot longer: 92% of the playing time of the video

I can speed up the conversion times by not using the Ubuntu server itself but instead run the command in a command window on my Windows main computer with the files still located on the Ubuntu server via a shared folder.

But it still takes a pretty long time to get from the 1280x720 ts file to a 852x480 mp4 file.

In any case it seems like I should download to a ts file so as not to inflict capture losses due to heavy load on the Ubuntu server.
Then I need to launch the conversion process after the download finishes.

Now I just wonder if the download of the next video, which started when the first was done, will be affected by the processing load of the first video's conversion?

[jagabo]

Your computer must be pretty slow. Downloading as a TS file takes almost no CPU time. You can be sure a simultaneous conversion doesn't disturb the downloading of another video by running the conversion at a low priority. I don't know how to do it on Linux but on Windows you can do it with:

start /b /low "title" ffmpeg -i input.ts -vf scale=w=-4:h=480 -c:v libx264 -preset veryfast -crf 25 -c:a copy output.mp4

Since the download process is running at higher priority it will get all the CPU time it wants, whenever it wants it. But since it uses so little all the rest will be available to the conversion.

[Perezdaniel]

youtube downloader is https://televzr.com/video-downloader/youtube but just blank black in VCL I tried with youtube updated to latest. Would not work.

[jagabo]

By the way, if you have a GPU with hardware video encoding you can use that with ffmpeg to reduce CPU load. Intel QSV, Nvidia NVENC, etc.

[BosseB]

As it is now I am doing this:
- File store on a Ubuntu 18.04 server running on low-end hardware, available via nfs to the other computers..

- Download videos to a dir on the nfs server executed as 1-hour at jobs on an Ubuntu 20.04 laptop (HP Workstation from 2010).
No conversion done, just store the ts stream.

- Conversion from ts to mp4 format and reduction from 1280x720 to 852x48 done by a batch job on my Windows 10 new HP workstation
The conversion is done at a speed of about 7-8X so it takes 6-8 min to do for each video and the Win10 CPU load raises to about 80% when it is running.

Then the edit is manual on my Win10 computer...

Before I downloaded to ts format and at full resolution I tried the command to download and convert both format and size on the fly but it resulted in audio gaps...
No longer so.

[metis]

@jagabo

I'm pretty far away from Your Knowledges about FFmpeg-CommandLines.
I just wonder, why don't You use -preset ultrafast instead of -preset veryfast ?

[jagabo]

I just wonder, why don't You use -preset ultrafast instead of -preset veryfast ?

Ultrafast isn't hugely faster than veryfast but it's much less efficient (2x to 3x more bitrate at the same crf). The OP can try it if he wants.