VideoHelp Forum


Try DVD-Fab Video Downloader and rip Netflix video! Or Try DVD-Fab and copy Blu-rays! or rip iTunes movies!
+ Reply to Thread
Page 1 of 2
1 2 LastLast
Results 1 to 30 of 31
Thread
  1. Member
    Join Date
    Feb 2021
    Location
    Sweden
    Search Comp PM
    I am watching MSDN news, which is made available as a stream (certain times of day only) on this site.
    (Note that it is usually not streaming during USA nights...)
    They used to have a YouTube channel too, where youtube-dl worked so I could automate the downloads on a Ubuntu 20.04 Server during the night in Sweden (evening shows in USA). Then I could view them in the morning.

    But the Youtube channel was taken down a week or so ago so now the only available source is the video player on the home page.

    I have found no way to download from this so I have been forced to set up a screen capture on a computer running over night using ffmpeg in a script. Not so good because the computer is blocked for other use and no sound except the playing video may be generated because it gets into the captured video too...

    So I wonder if there is a way to actually download the stream data directly from the URL, like youtube-dl can for streams on YouTube?
    Last edited by BosseB; 28th Feb 2021 at 05:49.
    Quote Quote  
  2. Member
    Join Date
    Feb 2021
    Location
    Sweden
    Search Comp PM
    No, that brings you to the CNN stream!
    What I am after is the MSNBC stream, and that has been discontinued on YouTube, it was like the CNN stream for many months after I found this site. But about 2 weeks ago they discontinued the MSNBC stream on YouTube due to complaints from NBC, I guess.
    But on the base page I linked to there are two video players, the top one is MSNBC and is the one I want to download from.
    The other is CNN and plays the exact same content as the YouTube link on the page does.

    When the MSNBC YouTube link was active it worked fine with youtube-dl and I could script getting the nightly (in Europe) evening shows.
    Not so anymore, now all there is is this videoplayer page object which plays streamed MSNBC.
    What I need is a way to download from the source feeding the player but using for example youtube-dl or ffmpeg directly.
    Quote Quote  
  3. Member
    Join Date
    Feb 2006
    Location
    United States
    Search Comp PM
    Originally Posted by BosseB View Post
    No, that brings you to the CNN stream!
    What I am after is the MSNBC stream, and that has been discontinued on YouTube, it was like the CNN stream for many months after I found this site. But about 2 weeks ago they discontinued the MSNBC stream on YouTube due to complaints from NBC, I guess.
    But on the base page I linked to there are two video players, the top one is MSNBC and is the one I want to download from.
    The other is CNN and plays the exact same content as the YouTube link on the page does.

    When the MSNBC YouTube link was active it worked fine with youtube-dl and I could script getting the nightly (in Europe) evening shows.
    Not so anymore, now all there is is this videoplayer page object which plays streamed MSNBC.
    What I need is a way to download from the source feeding the player but using for example youtube-dl or ffmpeg directly.
    play the video, press F12 on your keyboard to bring up the debug menu, go to network, you should see master.m3u8, copy the URL
    and feed it into youtube-dl to download the video.
    Quote Quote  
  4. Use 'streamlink', not 'youtude-dl'.
    Quote Quote  
  5. Member
    Join Date
    Feb 2021
    Location
    Sweden
    Search Comp PM
    I tried to use youtube-dl, but when I tried to break the download with ctrl-C it resulted in myvideo.part and no mp4 file.
    So cannot play it.

    I will try streamlink too, but I don't know the call syntax for streamlink and its help does not contain some simple examples...
    Off to google.
    Maybe streamlink URL outputfile?
    Quote Quote  
  6. streamlink URL best -o video.ts
    Quote Quote  
  7. Member
    Join Date
    Feb 2021
    Location
    Sweden
    Search Comp PM
    Originally Posted by LZAA View Post
    streamlink URL best -o video.ts
    Thanks now running.
    How do I stop download from live stream at the end? Ctrl-C or q or something else?
    Quote Quote  
  8. Originally Posted by BosseB View Post
    I tried to use youtube-dl, but when I tried to break the download with ctrl-C it resulted in myvideo.part and no mp4 file.
    So cannot play it.
    You can often just rename the file with .mp4 and it will play. Or:

    Code:
    youtube-dl --hls-use-mpegts --no-part -o output.ts https://hlss.goodgame.ru/hls/183955.m3u8
    You can Ctrl+C or Ctrl+Break at any time. You can even play the output.ts while it's still being downloaded.
    Quote Quote  
  9. Member
    Join Date
    Feb 2021
    Location
    Sweden
    Search Comp PM
    Originally Posted by jagabo View Post
    Originally Posted by BosseB View Post
    I tried to use youtube-dl, but when I tried to break the download with ctrl-C it resulted in myvideo.part and no mp4 file.
    So cannot play it.
    You can often just rename the file with .mp4 and it will play. Or:

    Code:
    youtube-dl --hls-use-mpegts --no-part -o output.ts https://hlss.goodgame.ru/hls/183955.m3u8
    You can Ctrl+C or Ctrl+Break at any time. You can even play the output.ts while it's still being downloaded.
    So I tried it and noticed that in the messages flying by on the screen I found this:
    Code:
    [https @ 0x564ab91d10c0] HTTP error 402 Payment Required   <=== Hmm, how can this be handled?
    [hls @ 0x564ab8e12cc0] Failed to open next segment 163187 of playlist 0
    [hls @ 0x564ab8e12cc0] Opening 'https://hlss.goodgame.ru/hls/183955-163187.ts' for reading03x
    [https @ 0x564ab99b40c0] Opening 'https://hlss.goodgame.ru/hls/183955.m3u8' for reading
    The file played with short interrutions along the way, irritating...

    I only changed the ts extension to mp4 for the output file.
    After 5.5 minutes the file is rather big already 151 Mbytes, which is a lot more than my usual downloads.
    I normally download at 480p resolution.
    But it seems like this stream does not carry any info on available resolutions...
    Quote Quote  
  10. Why would you change the extension to mp4? It's a transport stream, .TS. As far as I can tell there's only a 720p version of the MSNBC stream. Even in a web browser that's all you can get.
    Quote Quote  
  11. Member
    Join Date
    Feb 2021
    Location
    Sweden
    Search Comp PM
    Originally Posted by jagabo View Post
    Why would you change the extension to mp4? It's a transport stream, .TS. As far as I can tell there's only a 720p version of the MSNBC stream. Even in a web browser that's all you can get.
    The reason I changed it is that I want the file on disk to be compressed, i.e. smaller in size.
    AFAIU the downloaders (youtube-dl or ffmpeg) adjusts the compression according to the file type. But I may be wrong...
    Recoding it after the download is a time consuming operation.
    Quote Quote  
  12. If you want to downscale to 480p and re-encode to mp4 you want a command line like:

    Code:
    youtube-dl "https://hlss.goodgame.ru/hls/183955.m3u8" -o - | ffmpeg -i - -vf scale=w=856:h=480 -c:v libx264 -preset veryfast -crf 25 -c:a copy output.mp4
    Quote Quote  
  13. Here's how you can download with ffmpeg :

    Code:
    ffmpeg -user_agent "Mozilla"  -i https://hlss.goodgame.ru/hls/183955.m3u8 -c copy output.ts
    Or to resize and reencode to mp4:

    Code:
    ffmpeg -user_agent "Mozilla"  -i https://hlss.goodgame.ru/hls/183955.m3u8 -vf scale=w=856:h=480 -c:v libx264 -preset veryfast -crf 25 -c:a copy output.mp4
    Terminate by pressing q on the keyboard.

    Or play the video with ffplay:

    Code:
    ffplay -user_agent "anything"  -i https://hlss.goodgame.ru/hls/183955.m3u8
    Or to play in VLC, put this in a local m3u8 file:

    Code:
    #EXTM3U
    
    #EXTINF:0 MSNBC
    #EXTVLCOPT:http-user-agent=Android
    https://hlss.goodgame.ru/hls/183955.m3u8
    Last edited by jagabo; 28th Feb 2021 at 21:27.
    Quote Quote  
  14. Member
    Join Date
    Feb 2021
    Location
    Sweden
    Search Comp PM
    Just to clarify so I understand it correctly, if I use this command:
    Code:
    ffmpeg -user_agent "Mozilla"  -i https://hlss.goodgame.ru/hls/183955.m3u8 -vf scale=w=856:h=480 -c:v libx264 -preset veryfast -crf 25 -c:a copy output.mp4
    then the size conversion from 1280x720 (which is the only resolution offered by the site) to 856x480 is done on the fly while the video downloads?

    If so it is a real boon since the "after the download" conversion takes a long time...

    Since I am using Ubuntu for the downloads:

    To get the right time span I need to hit q with ffmpeg, but if I script this and put it into an at job how can I send the q into ffmpeg?
    Currently when I have used youtube-dl in scripts I set the download time by using the timeout command:
    Code:
    timeout --signal=2 65m youtube-dl.....
    This sends a Ctrl-C to youtube-dl after 65 min which interrupts youtube-dl and it cleans up the download by finalizing the video.
    Probably youtube-dl sends q to ffmpeg...

    I like to have the download+conversion scripted into at jobs so they are not forgotten...
    Quote Quote  
  15. Member
    Join Date
    Feb 2021
    Location
    Sweden
    Search Comp PM
    REPLY TO SELF:
    Obviously if one knows the length of the download one just puts
    Code:
    -t  <length in seconds>
    just before the output file name...

    @jagabo:
    I am really grateful for your reply, it seems to solve most of my problems!
    Quote Quote  
  16. Originally Posted by BosseB View Post
    Just to clarify so I understand it correctly, if I use this command:
    Code:
    ffmpeg -user_agent "Mozilla"  -i https://hlss.goodgame.ru/hls/183955.m3u8 -vf scale=w=856:h=480 -c:v libx264 -preset veryfast -crf 25 -c:a copy output.mp4
    then the size conversion from 1280x720 (which is the only resolution offered by the site) to 856x480 is done on the fly while the video downloads?
    Yes.

    Originally Posted by BosseB View Post
    If so it is a real boon since the "after the download" conversion takes a long time...
    Be aware that a slow computer might not be able to keep up with a live broadcast. That's why I chose the veryfast preset (fast, ok quality) rather than something slower. To get the best quality (per bitrate) you may want to do a slower re-compression after downloading.
    Quote Quote  
  17. Member
    Join Date
    Feb 2021
    Location
    Sweden
    Search Comp PM
    Originally Posted by jagabo View Post
    Or to resize and reencode to mp4:

    Code:
    ffmpeg -user_agent "Mozilla"  -i https://hlss.goodgame.ru/hls/183955.m3u8 -vf scale=w=856:h=480 -c:v libx264 -preset veryfast -crf 25 -c:a copy output.mp4
    Just a final detail which I am wondering about:
    In the ffmpeg command the resize argument is this:
    Code:
    -vf scale=w=856:h=480
    But given that 1280 * 480 / 720 = 53.3333, would not the proper horizontal even number argument be 854?
    Otherwise the aspect ratio would have to be changed and that might be a harder computational task, or not?

    Is there a way to use something like a wildcard for the horizontal so ffmpeg can figure out by itself the proper output width of the resulting video?
    Quote Quote  
  18. I just picked the nearest mod8 width. You're not going to notice the difference between 854x480 and 856x480. You can use a negative value to specify the nearest mod N that matches the other dimension:
    Code:
    -vf scale=w=-4:h=480
    That will deliver the nearest mod4 width for a height of 480, 852x480.

    Code:
    -vf scale=w=640:h=-4
    That will deliver the the nearest mod4 height for a width of 640, 640x360.
    Quote Quote  
  19. Member
    Join Date
    Feb 2021
    Location
    Sweden
    Search Comp PM
    Now that I have downloaded a number of videos using the command shown here I have an additional question:
    It seems like I am getting a video-audio time difference so lip-sync is bad.
    To fix this I can apply an ffmpeg command like this (for a 340 ms difference):
    Code:
    ffmpeg -i input.mp4 -itsoffset -0.34 -i input.mp4 -map 1:v -map 0:a -c copy output.mp4
    Is it possible to add this processing into the capture command? It seems like the time difference is always about 350 ms...
    I am using this capture cmd in my script at the moment:
    Code:
    ffmpeg -user_agent \"Mozilla\"  -i $VIDEOURL -vf scale=w=-4:h=480 -c:v libx264 -preset veryfast -crf 25 -c:a copy -t $CAPTURETIME $TARGETFILE

    If so what is the correct syntax?
    Last edited by BosseB; 2nd Mar 2021 at 01:51.
    Quote Quote  
  20. Using itsoffset requires that you have two source videos. Opening the stream twice would use twice the bandwidth and you can't be sure the two streams will be synchronized. If you don't mind re-encoding the audio you can use the adelay audio filter:

    Code:
    ffmpeg -y -user_agent "Mozilla" -i https://hlss.goodgame.ru/hls/183955.m3u8 -vf scale=w=-4:h=480 -c:v libx264 -preset veryfast -crf 25 -c:a aac -af adelay=350ms:all=1 -async 1 output.mp4
    Quote Quote  
  21. Member
    Join Date
    Feb 2021
    Location
    Sweden
    Search Comp PM
    I thought as much given the two sources even if they are one and the same...

    So I would probably be better off launching the audio delay from the download script once it has finished downloading.
    That would be the more stable solution, I think.
    I have noted that the audio resync is pretty quick, only takes a couple of seconds on a downloaded 1 hour video, so it would not be a big problem.
    I have a script already that does this so I just have to call it from the download script to make this happen.

    Thanks for the information!
    Quote Quote  
  22. Member
    Join Date
    Feb 2021
    Location
    Sweden
    Search Comp PM
    I have now used the commands suggested here to download the video stream to file for a number of time slots (using the at command).
    What I have found is that the download itself seems to incur irregular pauses in the resulting file possibly caused by computational overload of the computer.
    Since the conversion to ffmpeg and the simultaneous size change might be very computationally intensive I am now looking at the different commands:

    1) Capture the video to file in the raw ts format. You already gave that command in the replies above.
    Code:
    ffmpeg -user_agent "Mozilla"  -i https://hlss.goodgame.ru/hls/183955.m3u8 -c copy output.ts
    This seems to work fine but results in a very big ts file at 1280x720 resolution. As far as I can tell the audio lapses are gone.

    2) Convert the data now on disk from ts to mp4, I found this to just convert into compressed mp4 without the change of scale:
    Code:
    ffmpeg -i input.ts -map 0 -c copy output.mp4
    This runs pretty fast, but it seems not to apply any extra compression, the output file is almost the same size as the input.

    3) Convert the data now on disk to mp4 compression AND change of scale:
    Code:
    ffmpeg -i input.ts -vf scale=w=-4:h=480 -c:v libx264 -preset veryfast -crf 25 -c:a copy output.mp4
    This took 56% of the playing time of a video and produced a file reduction from 546 to 58 Mb...

    4) Only compress the video:
    Code:
    ffmpeg -i input.ts -c:v libx264 -preset veryfast -crf 25 -c:a copy output720.mp4
    This took a lot longer: 92% of the playing time of the video

    I can speed up the conversion times by not using the Ubuntu server itself but instead run the command in a command window on my Windows main computer with the files still located on the Ubuntu server via a shared folder.

    But it still takes a pretty long time to get from the 1280x720 ts file to a 852x480 mp4 file.

    In any case it seems like I should download to a ts file so as not to inflict capture losses due to heavy load on the Ubuntu server.
    Then I need to launch the conversion process after the download finishes.

    Now I just wonder if the download of the next video, which started when the first was done, will be affected by the processing load of the first video's conversion?
    Quote Quote  
  23. Your computer must be pretty slow. Downloading as a TS file takes almost no CPU time. You can be sure a simultaneous conversion doesn't disturb the downloading of another video by running the conversion at a low priority. I don't know how to do it on Linux but on Windows you can do it with:

    Code:
    start /b /low "title" ffmpeg -i input.ts -vf scale=w=-4:h=480 -c:v libx264 -preset veryfast -crf 25 -c:a copy output.mp4
    Since the download process is running at higher priority it will get all the CPU time it wants, whenever it wants it. But since it uses so little all the rest will be available to the conversion.
    Last edited by jagabo; 3rd Mar 2021 at 18:38.
    Quote Quote  
  24. youtube downloader is https://televzr.com/video-downloader/youtube but just blank black in VCL I tried with youtube updated to latest. Would not work.
    Quote Quote  
  25. By the way, if you have a GPU with hardware video encoding you can use that with ffmpeg to reduce CPU load. Intel QSV, Nvidia NVENC, etc.
    Quote Quote  
  26. Member
    Join Date
    Feb 2021
    Location
    Sweden
    Search Comp PM
    As it is now I am doing this:
    - File store on a Ubuntu 18.04 server running on low-end hardware, available via nfs to the other computers..

    - Download videos to a dir on the nfs server executed as 1-hour at jobs on an Ubuntu 20.04 laptop (HP Workstation from 2010).
    No conversion done, just store the ts stream.

    - Conversion from ts to mp4 format and reduction from 1280x720 to 852x48 done by a batch job on my Windows 10 new HP workstation
    The conversion is done at a speed of about 7-8X so it takes 6-8 min to do for each video and the Win10 CPU load raises to about 80% when it is running.

    Then the edit is manual on my Win10 computer...

    Before I downloaded to ts format and at full resolution I tried the command to download and convert both format and size on the fly but it resulted in audio gaps...
    No longer so.
    Quote Quote  
  27. Member metis's Avatar
    Join Date
    Jan 2021
    Location
    Spain
    Search Comp PM
    @jagabo

    I'm pretty far away from Your Knowledges about FFmpeg-CommandLines.
    I just wonder, why don't You use -preset ultrafast instead of -preset veryfast ?
    Life could be so easy, if there weren't those f*** Details.
    Quote Quote  
  28. Originally Posted by metis View Post
    I just wonder, why don't You use -preset ultrafast instead of -preset veryfast ?
    Ultrafast isn't hugely faster than veryfast but it's much less efficient (2x to 3x more bitrate at the same crf). The OP can try it if he wants.
    Quote Quote  



Similar Threads