VideoHelp Forum



Support our site by donate $5 directly to us Thanks!!!

Try StreamFab Downloader and download streaming video from Netflix, Amazon!



+ Reply to Thread
Results 1 to 11 of 11
  1. 20th Jun 2018 00:10 #1
    Sapito
    Sapito is offline
    Member
    Join Date
    Jun 2018
    Location
    Americas
    Search Comp PM
    Hi!

    There's certain streaming I'd like to save but it won't be available till the moment it starts. I have the link though (it says unavailable if you open it now), and it's in this format: http://vmf.edge-apps.net/embed/live.php?streamname=<name of the stream>&autoplay=true

    I've installed livestreamer and wanted to try recording something else from "vmf.edge-apps.net" to try it out, since there's no time for trying things when the stream I want starts.

    So, there's this TV channel, http://www.tvpublica.com.ar/vivo/, that if you look in the page's source you can find a link in the same format as above. Here's the link: http://vmf.edge-apps.net/embed/live.php?streamname=c7live01-20034&autoplay=true

    I looked around and found this thread: https://forum.videohelp.com/threads/379186-Tricky-stream-to-download-%28

    A kind user there shows the OP the command line for livestreamer, but I'm curious to know how did he extract the "hlsvariant://" link, given that the OP only posted a link from vmf.edge-apps.net


    Thanks for any help
    Quote Quote  
  2. 20th Jun 2018 01:39 #2
    ponens
    ponens is offline
    Member Emeritus
    Join Date
    May 2014
    Search PM
    When the stream is active look for the playlist url in dev tools network.
    Last edited by ponens; 20th Jun 2018 at 02:41.
    Quote Quote  
  3. 20th Jun 2018 02:25 #3
    Sapito
    Sapito is offline
    Member
    Join Date
    Jun 2018
    Location
    Americas
    Search Comp PM
    I didn't find anything like that, and neither with 'playback, 'session' or 'json' (there's something that contains the latter but doesn't seem like anything relevant). I even tried looking manually for a long string of characters, but nothing similar shows up.


    Edit: I made some "progress". I installed the VideoDownloadHelper addon and it does show me an m3u8 playlist on the TV channel. The playlist however doesn't seem to contain anything if I download it.

    <!DOCTYPE html>
    <html>
    <head>
    <title>403 Access denied.</title>
    </head>
    <body>
    <h1>Error 403 Access denied.</h1>
    <p>Access denied.</p>
    <h3>Guru Meditation:</h3>
    <p>XID: 93265376</p>
    <hr>
    <p>Varnish cache server</p>
    </body>
    </html>
    A 403 error. If I check the details I get the playlist's link. So I tried to loading it with livestreamer, replacing http:// with hlsvariant://

    livestreamer --http-header "Referer=http://vmf.edge-apps.net/embed/live.php?streamname=c7live01-20034" "hlsvariant://g3.vxral-slo.transport.edge-access.net/b17/ngrp:c7cam2-20034_all/c7cam2-20034_720p.m3u8" best -o Livestream.ts

    and livestreamer's response:

    [cli][info] Found matching plugin stream for URL hlsvariant://g3.vxral-slo.transport.edge-access.net/b17/ngrp:c7cam2-20034_all/c7cam2-20034_720p.m3u8
    error: No streams found on this URL: hlsvariant://g3.vxral-slo.transport.edge-access.net/b17/ngrp:c7cam2-20034_all/c7cam2-20034_720p.m3u8
    Dead end?



    Edit 2: Even MORE progress. Actually that add-on is not really needed. By having the TV channel's page open with video playing you can toggle the developer tools and go to the network tab. Now *this* is interesting. Every few seconds 3 files show up. An .m3u8, .key an .ts files. Right clicking the m3u8 and "copy as curl" gets me the original link and the playlist link

    curl "https://g2.mc-hor.transport.edge-access.net/b17/ngrp:c7cam2-20034_all/c7cam2-20034_720p.m3u8" -H "Host: g2.mc-hor.transport.edge-access.net" -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0" -H "Accept: */*" -H "Accept-Language: en-GB,en-US;q=0.7,en;q=0.3" --compressed -H "Referer: http://vmf.edge-apps.net/embed/live.php?streamname=c7live01-20034&autoplay=true" -H "Origin: http://vmf.edge-apps.net" -H "Connection: keep-alive" -H "If-Modified-Since: Wed, 20 Jun 2018 07:28:28 GMT"
    The playlist is different from what the add-on had, but whatever. Livestreamer still doesn't work. If I try downloading that playlist I get the 403 error. BUT if I choose "see response" in the network logger I get *something*

    Code:
    #EXTM3U
#EXT-X-VERSION:3
#EXT-X-MEDIA-SEQUENCE:32871
#EXT-X-TARGETDURATION:4
#EXT-X-KEY:METHOD=AES-128,URI="https://vmf.edge-apps.net/keys/00D2A735511A71B0D8449A57CF2520AA/c7cam2-20034_720p-1529475921665.key",IV=0x0000000000000000000001641BDE7701
#EXTINF:4.000,
c7cam2-20034_720p-1529475924580.ts
#EXTINF:4.000,
c7cam2-20034_720p-1529475928078.ts
#EXTINF:4.000,
c7cam2-20034_720p-1529475931744.ts
#EXTINF:4.000,
c7cam2-20034_720p-1529475936245.ts
#EXTINF:4.000,
c7cam2-20034_720p-1529475939701.ts
#EXTINF:4.000,
c7cam2-20034_720p-1529475943736.ts
#EXTINF:4.000,
c7cam2-20034_720p-1529475947698.ts
#EXTINF:4.000,
c7cam2-20034_720p-1529475952382.ts
#EXTINF:4.000,
c7cam2-20034_720p-1529475955940.ts
#EXT-X-KEY:METHOD=AES-128,URI="https://vmf.edge-apps.net/keys/00D2A735511A71B0D8449A57CF2520AA/c7cam2-20034_720p-1529475959689.key",IV=0x0000000000000000000001641BDF0B89
#EXTINF:4.000,
c7cam2-20034_720p-1529475959689.ts
#EXTINF:4.000,
c7cam2-20034_720p-1529475963695.ts
#EXTINF:4.000,
c7cam2-20034_720p-1529475967708.ts
#EXTINF:4.000,
c7cam2-20034_720p-1529475971696.ts
#EXTINF:4.000,
c7cam2-20034_720p-1529475975705.ts
#EXTINF:4.000,
c7cam2-20034_720p-1529475979691.ts
#EXTINF:4.000,
c7cam2-20034_720p-1529475983700.ts
#EXTINF:4.000,
c7cam2-20034_720p-1529475987740.ts
#EXTINF:4.000,
c7cam2-20034_720p-1529475991692.ts
#EXTINF:4.000,
c7cam2-20034_720p-1529475995693.ts
#EXT-X-KEY:METHOD=AES-128,URI="https://vmf.edge-apps.net/keys/00D2A735511A71B0D8449A57CF2520AA/c7cam2-20034_720p-1529475999748.key",IV=0x0000000000000000000001641BDFA804
    It was much longer, I trimmed it here. That does look like a playlist now. "AES-128" pops up, along with the .key file it seems there's some kind of encryption here, and I have no idea what to do with livestreamer now.
    Last edited by Sapito; 20th Jun 2018 at 03:35.
    Quote Quote  
  4. 20th Jun 2018 03:52 #4
    ponens
    ponens is offline
    Member Emeritus
    Join Date
    May 2014
    Search PM
    It can't be downloaded unless it is some way that sets the Referer. Any direct download that doesn't set the Referer will result in 403.

    720p.m3u8 is not a variant playlist. So, if you're going to use livestreamer with the 720p.m3u8 playlist, use "hls://" instead of "hlsvariant://"

    If you used the variant playlist with livestreamer directly (instead of trying to download it), you wouldn't get a 403 error as long as the Referer is properly set.
    Last edited by ponens; 20th Jun 2018 at 11:32.
    Quote Quote  
  5. 20th Jun 2018 08:40 #5
    jagabo
    jagabo is offline
    Member
    Join Date
    Dec 2005
    Search Comp PM
    I was able (with a VPN in Argentina) to download with youtube-dl with:

    Code:
    youtube-dl --referer "http://vmf.edge-apps.net/embed/live.php?streamname=c7live01-20034&autoplay=true" https://g1.proy-slo.transport.edge-access.net/b17/ngrp:c7cam2-20034_all/playlist.m3u8
    I got the url and referrer from Internet Explorer's Developer Tools.
    Last edited by jagabo; 20th Jun 2018 at 11:16.
    Quote Quote  
  6. 20th Jun 2018 14:47 #6
    jimdagys
    jimdagys is offline
    Member
    Join Date
    Nov 2003
    Search Comp PM
    Code:
    youtube-dl --referer "http://vmf.edge-apps.net/embed/live.php?streamname=c7live01-20034&autoplay=true" https://g1.proy-slo.transport.edge-access.net/b17/ngrp:c7cam2-20034_all/playlist.m3u8
    Jagabo,
    What exactly does the "referer" do? Why is it necessary? You say you got it from browser developer tools. How did you know the above (ending in "true") was the "referer"?
    Quote Quote  
  7. 20th Jun 2018 15:17 #7
    David Banner
    David Banner is offline
    Member
    Join Date
    Nov 2011
    Search Comp PM
    Originally Posted by jimdagys View Post
    Code:
    youtube-dl --referer "http://vmf.edge-apps.net/embed/live.php?streamname=c7live01-20034&autoplay=true" https://g1.proy-slo.transport.edge-access.net/b17/ngrp:c7cam2-20034_all/playlist.m3u8
    Jagabo,
    What exactly does the "referer" do? Why is it necessary? You say you got it from browser developer tools. How did you know the above (ending in "true") was the "referer"?
    I may be wrong but I think true means yes as in "autoplay is yes" - it will play automatically. https://github.com/rg3/youtube-dl/issues/1854 may explain the refeer bit
    Quote Quote  
  8. 20th Jun 2018 18:08 #8
    jagabo
    jagabo is offline
    Member
    Join Date
    Dec 2005
    Search Comp PM
    The --referer field of an http request is site from which the link was obtained.

    https://en.wikipedia.org/wiki/List_of_HTTP_header_fields
    https://en.wikipedia.org/wiki/HTTP_referer

    So in that command line you are saying "I want to open this file: https://g1.proy-slo.transport.edge-access.net/b17/ngrp:c7cam2-20034_all/playlist.m3u8, and "I got that link from http://vmf.edge-apps.net/embed/live.php?streamname=c7live01-20034&autoplay=true.

    In cases like this the --referer is to verify that you got the link from a legit web page -- other sites that link to it will not be able to play the video. With this particular request only the "http://vmf.edge-apps.net" part of the referer field is really needed.

    I got the -referer field from Developer Tools. Look at the Details of the request:
    Image Attached Thumbnails Click image for larger version Name: details.png Views: 547 Size: 18.0 KB ID: 45910  

    Last edited by jagabo; 20th Jun 2018 at 18:26.
    Quote Quote  
  9. 20th Jun 2018 19:58 #9
    Sapito
    Sapito is offline
    Member
    Join Date
    Jun 2018
    Location
    Americas
    Search Comp PM
    WE MADE IT

    So, livestreamer refuses to work. See:
    Code:
    C:\Program Files (x86)\Livestreamer>livestreamer --http-header "Referer=http://vmf.edge-apps.net/embed/live.php?streamname=c7live01-20034&autoplay=true" "hls://g1.proy-slo.transport.edge-access.net/b17/ngrp:c7cam2-20034_all/playlist.m3u8" best -o Livestream.ts
[cli][info] Found matching plugin stream for URL hls://g1.proy-slo.transport.edge-access.net/b17/ngrp:c7cam2-20034_all/playlist.m3u8
[cli][info] Available streams: live (worst, best)
[cli][info] Opening stream: live (hls)
[cli][error] Could not open stream: Unable to open URL: http://g1.proy-slo.transport.edge-access.net/b17/ngrp:c7cam2-20034_all/playlist.m3u8 (HTTPConnectionPool(host='g1.proy-slo.transport.edge-access.net', port=80): Max retries exceeded with url: /b17/ngrp:c7cam2-20034_all/playlist.m3u8 (Caused by <class 'socket.gaierror'>: [Errno 11001] getaddrinfo failed))
    But youtube-dl as jagabo says works wonderfully. It needed a ffmpeg to do anything, I downloaded the "static" Windows build. It works so I guess that's the one.

    Code:
    youtube-dl --referer "http://vmf.edge-apps.net/embed/live.php?streamname=c7live01-20034&autoplay=true" https://g1.mc-hor.transport.edge-access.net/b17/ngrp:c7cam2-20034_all/c7cam2-20034_1080p.m3u8
    Now there's a slight problem. There's no way to stop recording. If I close the cmd window or press ctrl+c I end up with a .mp4.part file that's unplayable (since live TV is infinite I have to trigger the "stop" myself). The only way to have youtube-dl produce a playable file at the end is to disconnect the network adapter, so I get this:

    Code:
    [hls,applehttp @ 061f4ac0] Opening 'crypto+https://g1.mc-hor.transport.edge-access.net/b17/ngrp:c7cam2-20034_all/c7cam2-20034_1080p-1529537100290.ts' for reading
[https @ 062db240] Opening 'https://g1.mc-hor.transport.edge-access.net/b17/ngrp:c7cam2-20034_all/c7cam2-20034_1080p.m3u8' for reading
[tls @ 06299fc0] Error in the pull function.
[hls,applehttp @ 061f4ac0] keepalive request failed for 'https://g1.mc-hor.transport.edge-access.net/b17/ngrp:c7cam2-20034_all/c7cam2-20034_1080p.m3u8', retrying with new connection: I/O error
[tcp @ 06b07d00] Failed to resolve hostname g1.mc-hor.transport.edge-access.net: The name does not resolve for the supplied parameters
[hls,applehttp @ 061f4ac0] Failed to reload playlist 0
[tcp @ 06b08180] Failed to resolve hostname g1.mc-hor.transport.edge-access.net: The name does not resolve for the supplied parameters
[hls,applehttp @ 061f4ac0] Failed to reload playlist 0
frame= 3800 fps= 25 q=-1.0 Lsize=   60191kB time=00:02:31.97 bitrate=3244.5kbits/s speed=   1x
video:57669kB audio:2386kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.226303%
[ffmpeg] Downloaded 61635404 bytes
[download] 100% of 58.78MiB in 02:34
    There the file gets renamed (finalized?) to .mp4 and it plays perfect. Any ideas? I tried outputting a TS file with "-o testvideo.ts" but it's just the same mp4 as before with a .ts extension (the player identifies it as an mp4 container).


    On the other side the -F switch to list video formats doesn't do much either. Of course, since I'm using the 1080 playlist there's bound to be an only option. I just see that the container is mp4 (why? aren't the individual streamed packets in .ts?). I expected that it could throw the video/audio codecs at least.

    Code:
    youtube-dl -F --referer "http://vmf.edge-apps.net/embed/live.php?streamname=c7live01-20034&autoplay=true" https://g1.mc-hor.transport.edge-access.net/b17/ngrp:c7cam2-20034_all/c7cam2-20034_1080p.m3u8
[generic] c7cam2-20034_1080p: Requesting header
[generic] c7cam2-20034_1080p: Downloading m3u8 information
[info] Available formats for c7cam2-20034_1080p:
format code  extension  resolution note
0            mp4        unknown
    Thanks for for helping me to get this running
    Quote Quote  
  10. 20th Jun 2018 20:59 #10
    jagabo
    jagabo is offline
    Member
    Join Date
    Dec 2005
    Search Comp PM
    Yes, live streams never end so you have to manually stop the download. Pressing 'q' while downloading used to stop youtube-dl gracefully (?) but the option was removed at some point. I find that very short downloads tend not to play after CTRL+C. Longer ones work more often. Of course, risking a one-time-only live recording with CTRL+C isn't great. Maybe you can switch to using ffmpeg for downloading instead ('q' is supported there).

    Code:
    ffmpeg.exe -headers "Referer: http://vmf.edge-apps.net" -i "https://g1.proy-slo.transport.edge-access.net/b17/ngrp:c7cam2-20034_all/playlist.m3u8" -c copy output.mp4
    Quote Quote  
  11. 23rd Jun 2018 21:18 #11
    Sapito
    Sapito is offline
    Member
    Join Date
    Jun 2018
    Location
    Americas
    Search Comp PM
    Originally Posted by jagabo View Post
    Yes, live streams never end so you have to manually stop the download. Pressing 'q' while downloading used to stop youtube-dl gracefully (?) but the option was removed at some point. I find that very short downloads tend not to play after CTRL+C. Longer ones work more often. Of course, risking a one-time-only live recording with CTRL+C isn't great. Maybe you can switch to using ffmpeg for downloading instead ('q' is supported there).

    Code:
    ffmpeg.exe -headers "Referer: http://vmf.edge-apps.net" -i "https://g1.proy-slo.transport.edge-access.net/b17/ngrp:c7cam2-20034_all/playlist.m3u8" -c copy output.mp4
    I see...

    I found out that by using the "--no-part" command youtube-dl writes directly into the .mp4, avoiding the nasty .part file. In this case I can CTRL+C safely and I always end with a playable file. Are they any drawbacks of using "--no-part"? I can't see any so far. Or what's the actual purpose of making the .part file by default.


    Anyway, now I have another doubt. The TV channel above allows you to go back 1 hour in the live stream. I tried going back in the player at the page and checked the playlist but it's the same as if I were watching the live stream. How do I tell youtube-dl to start capturing 1 hour before the current transmission?

    If my net connection goes down and the streaming is interrupted, it can be useful to go back to the point it stopped and resume capturing from there. Then I can cut and append the mp4 files to get the whole thing together.
    Quote Quote  



Similar Threads

« Previous Thread | Next Thread »
Visit our sponsor! Try DVDFab and backup Blu-rays!