"The Platform" is a service used by the NBC owned networks. Whenever I try to download something with youtube-dl, all I get are ads in the output file.
Here is an example of a show that does not require auth (from a US IP):
https://www.nbc.com/streets-of-dreams/video/diamond-district-new-york/4287008
The m3u file contains a ton of ads, so I understand why youtube-dl is doing that, but what I don't understand is how the player on the page actually shows video if the file only has ads:How can I download stuff from NBC?Code:#EXTM3U #EXT-X-VERSION:4 #EXT-X-TARGETDURATION:9 #EXT-X-PLAYLIST-TYPE:VOD #EXT-X-MEDIA-SEQUENCE:1 #ANVATO-SEGMENT-INFO: type=master #EXTINF:4.004, https://nbcuadops-vh.akamaihd.net/i/ads/tvn/... #EXTINF:4.337, https://nbcuadops-vh.akamaihd.net/i/ads/tvn/... #EXTINF:4.004, https://nbcuadops-vh.akamaihd.net/i/ads/tvn/... #EXTINF:4.338, https://nbcuadops-vh.akamaihd.net/i/ads/tvn/... #EXTINF:4.004, https://nbcuadops-vh.akamaihd.net/i/ads/tvn/... #EXTINF:4.338, https://nbcuadops-vh.akamaihd.net/i/ads/tvn/... #EXTINF:4.004, https://nbcuadops-vh.akamaihd.net/i/ads/tvn/... #EXTINF:1.156, https://nbcuadops-vh.akamaihd.net/i/ads/tvn/... #EXT-X-DISCONTINUITY #EXTINF:9.009, https://redirect.manifest.theplatform.com/r/... #ANVATO-SEGMENT-INFO: type=master #EXTINF:9.009, https://tvecnbchls.nbcuni.com/tve-adstitch/... #ANVATO-SEGMENT-INFO: type=master #EXTINF:9.009, https://tvecnbchls.nbcuni.com/tve-adstitch/... #ANVATO-SEGMENT-INFO: type=master #EXTINF:9.009, https://tvecnbchls.nbcuni.com/tve-adstitch/... #ANVATO-SEGMENT-INFO: type=master #EXTINF:9.009, https://redirect.manifest.theplatform.com/r/... #ANVATO-SEGMENT-INFO: type=master #EXTINF:9.009, https://tvecnbchls.nbcuni.com/tve-adstitch/... #ANVATO-SEGMENT-INFO: type=master #EXTINF:9.009, https://tvecnbchls.nbcuni.com/tve-adstitch/... #ANVATO-SEGMENT-INFO: type=master #EXTINF:9.009, https://tvecnbchls.nbcuni.com/tve-adstitch/... #ANVATO-SEGMENT-INFO: type=master #EXTINF:9.009, https://redirect.manifest.theplatform.com/r/... #ANVATO-SEGMENT-INFO: type=master #EXTINF:9.009, https://tvecnbchls.nbcuni.com/tve-adstitch/... #ANVATO-SEGMENT-INFO: type=master #EXTINF:9.009, https://tvecnbchls.nbcuni.com/tve-adstitch/... #ANVATO-SEGMENT-INFO: type=master #EXTINF:9.009, https://tvecnbchls.nbcuni.com/tve-adstitch/... #ANVATO-SEGMENT-INFO: type=master #EXTINF:9.009, https://redirect.manifest.theplatform.com/r/... #ANVATO-SEGMENT-INFO: type=master #EXTINF:9.009, https://tvecnbchls.nbcuni.com/tve-adstitch/... #ANVATO-SEGMENT-INFO: type=master #EXTINF:9.009, ...
+ Reply to Thread
Results 1 to 9 of 9
-
-
-
OK I will admit it: I was wrong. Unauthenticated videos work fine.
But take another episode from the same series that requires auth, and you get what you expect: "ERROR: This video is only available for users of participating TV providers"
I am using an auth that is not supported by youtube-dl. So I have used the cookies.txt extension (which was removed from the Chrome WebStore!?) to access.
Code:youtube-dl --cookies cookies.txt https://www.nbc.com/streets-of-dreams/video/calle-ocho-miami/4293059 ERROR: This video is only available for users of participating TV providers
HTML Code:document.getElementById('player').getAttribute('src')
First, let's take that url and do this:
Code:youtube-dl --get-duration --cookies cookies.txt "https://player.theplatform.com/..."
Take the same URL and do this:
Code:youtube-dl --verbose --write-pages --print-traffic --cookies cookies.txt "https://player.theplatform.com/...
Code:[debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: ['--verbose', '--write-pages', '--print-traffic', '--cookies', 'cookies.txt', 'https://player.theplatform.com/p/...#playerurl=https%3A//www.nbc.com/streets-of-dreams/video/calle-ocho-miami/4293059'] [debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252 [debug] youtube-dl version 2020.12.31 [debug] Python version 3.4.4 (CPython) - Windows-10-10.0.18362 [debug] exe versions: ffmpeg N-91481-gb8c4d2b2ed, ffprobe N-91481-gb8c4d2b2ed [debug] Proxy map: {} [ThePlatform] 4293059: Downloading webpage [ThePlatform] Saving request to 4293059_https_-_player.theplatform.com_p_jujdhC_xkaAQrhkr9IU_select_media_guid_2304981409_4293059mute=false___.dump [ThePlatform] 4293059: Downloading SMIL data [ThePlatform] Saving request to 4293059_https_-_link.theplatform.com_s_HNK2IC_media_e06vXD_jas_Scsid=oneapp_desktop_computer_web_ondemand_am_buildv=null_userAgent=___.dump [ThePlatform] 4293059: Downloading m3u8 information [ThePlatform] Saving request to 4293059_https_-_east.manifest.na.theplatform.com_m_HNK2IC_iIS84zsO9brm_yZfd0Uh8GvOY_mOI___.dump [ThePlatform] 4293059: Downloading JSON metadata [ThePlatform] Saving request to 4293059_http_-_link.theplatform.com_s_HNK2IC_media_e06vXD_jas_Sformat=preview.dump [debug] Default format spec: bestvideo+bestaudio/best [debug] Invoking downloader on 'https://east.manifest.na.theplatform.com/m/HNK2IC/iIS84zsO9brm,.../6.m3u8?sid=...' [download] Destination: Calle Ocho, Miami-4293059.mp4 [debug] ffmpeg command line: ffmpeg -y -loglevel verbose -headers "Accept-Language: en-us,en;q=0.5 Cookie: ssuid=1234 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Accept-Encoding: gzip, deflate User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3519.1 Safari/537.36 " -i "https://east.manifest.na.theplatform.com/m/HNK2IC/iIS84zsO9brm..." -c copy -f mp4 "-bsf:a" aac_adtstoasc "file:Calle Ocho, Miami-4293059.mp4.part" ffmpeg version N-91481-gb8c4d2b2ed Copyright (c) 2000-2018 the FFmpeg developers built with gcc 7.3.1 (GCC) 20180710 configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-bzlib --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --enable-libmfx --enable-amf --enable-ffnvcodec --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth libavutil 56. 18.102 / 56. 18.102 libavcodec 58. 21.105 / 58. 21.105 libavformat 58. 17.101 / 58. 17.101 libavdevice 58. 4.101 / 58. 4.101 libavfilter 7. 26.100 / 7. 26.100 libswscale 5. 2.100 / 5. 2.100 libswresample 3. 2.100 / 3. 2.100 libpostproc 55. 2.100 / 55. 2.100 [hls,applehttp @ 000002721980b380] HLS request for url 'https://redirect.manifest.theplatform.com/r/HNK2IC/iIS84zsO9brm,yZfd0Uh8GvOY', offset 0, playlist 0 [hls,applehttp @ 000002721980b380] Opening 'https://redirect.manifest.theplatform.com/r/HNK2IC/iIS84zsO9brm,yZfd0Uh8GvOY' for reading [https @ 0000027219e93600] overlong headers [hls,applehttp @ 000002721980b380] Failed to open segment 1 of playlist 0 [hls,applehttp @ 000002721980b380] HLS request for url 'https://nbcuadops-vh.akamaihd.net/i/ads/tvn/CISCOWEB30V2H_ENT_MEZZ/.../segment2_6_av.ts?set-akamai-hls-revision=5', offset 0, playlist 0 [hls,applehttp @ 000002721980b380] Opening 'https://nbcuadops-vh.akamaihd.net/i/ads/tvn/CISCOWEB30V2H_ENT_MEZZ/.../segment2_6_av.ts?set-akamai-hls-revision=5' for reading [hls,applehttp @ 000002721980b380] HLS request for url 'https://redirect.manifest.theplatform.com/r/HNK2IC/iIS84zsO9brm,yZfd0Uh8GvOY,...', offset 0, playlist 0 [hls,applehttp @ 000002721980b380] Opening 'https://redirect.manifest.theplatform.com/r/HNK2IC/iIS84zsO9brm,yZfd0Uh8GvOY,...' for reading [h264 @ 0000027219f25840] Reinit context to 1920x1088, pix_fmt: yuv420p Input #0, hls,applehttp, from 'https://east.manifest.na.theplatform.com/m/HNK2IC/iIS84zsO9brm,yZfd0Uh8GvOY,,,/6.m3u8?sid=1234': Duration: 00:49:35.65, start: 4.140767, bitrate: N/A Program 0 Metadata: variant_bitrate : 0 Stream #0:0: Video: h264 (Main), 1 reference frame ([27][0][0][0] / 0x001B), yuv420p(tv, bt709, left), 1920x1080 (1920x1088) [SAR 1:1 DAR 16:9], Closed Captions, 29.97 tbr, 90k tbn, 59.94 tbc Metadata: variant_bitrate : 0 Stream #0:1: Audio: aac (HE-AAC) ([15][0][0][0] / 0x000F), 44100 Hz, stereo, fltp Metadata: variant_bitrate : 0 Output #0, mp4, to 'file:Calle Ocho, Miami-4293059.mp4.part': Metadata: encoder : Lavf58.17.101 Stream #0:0: Video: h264 (Main), 1 reference frame (avc1 / 0x31637661), yuv420p(tv, bt709, left), 1920x1080 (0x0) [SAR 1:1 DAR 16:9], q=2-31, 29.97 tbr, 90k tbn, 90k tbc Metadata: variant_bitrate : 0 Stream #0:1: Audio: aac (HE-AAC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp Metadata: variant_bitrate : 0 Stream mapping: Stream #0:0 -> #0:0 (copy) Stream #0:1 -> #0:1 (copy) Press [q] to stop, [?] for help [hls,applehttp @ 000002721980b380] HLS request for url 'https://nbcuadops-vh.akamaihd.net/i/ads/tvn/CISCOWEB30V2H_ENT_MEZZ/CISCOWEB30V2H_ENT_MEZZ_....ts?set-akamai-hls-revision=5', offset 0, playlist 0 [https @ 0000027219e93600] Opening 'https://nbcuadops-vh.akamaihd.net/i/ads/tvn/CISCOWEB30V2H_ENT_MEZZ/CISCOWEB30V2H_ENT_MEZZ_.../segment4_6_av.ts?set-akamai-hls-revision=5' for reading [hls,applehttp @ 000002721980b380] HLS request for url 'https://redirect.manifest.theplatform.com/r/HNK2IC/iIS84zsO9brm,yZfd0Uh8GvOY...', offset 0, playlist 0 [https @ 000002721a0d1dc0] Cannot reuse HTTP connection for different host: nbcuadops-vh.akamaihd.net:-1 != redirect.manifest.theplatform.com:-1 [AVIOContext @ 000002721a19bf00] Statistics: 2169708 bytes read, 0 seeks [hls,applehttp @ 000002721980b380] keepalive request failed for 'https://redirect.manifest.theplatform.com/r/HNK2IC/iIS84zsO9brm,yZfd0Uh8GvOY...', retrying with new connection: Invalid argument [hls,applehttp @ 000002721980b380] Opening 'https://redirect.manifest.theplatform.com/r/HNK2IC/iIS84zsO9brm,yZfd0Uh8GvOY...' for reading [hls,applehttp @ 000002721980b380] HLS request for url 'https://nbcuadops-vh.akamaihd.net/i/ads/tvn/CISCOWEB30V2H_ENT_MEZZ/CISCOWEB30V2H_ENT_MEZZ_....ts?set-akamai-hls-revision=5', offset 0, playlist 0 [https @ 0000027219e93600] Opening 'https://nbcuadops-vh.akamaihd.net/i/ads/tvn/CISCOWEB30V2H_ENT_MEZZ/CISCOWEB30V2H_ENT_MEZZ_.../segment6_6_av.ts?set-akamai-hls-revision=5' for reading [hls,applehttp @ 000002721980b380] HLS request for url 'https://redirect.manifest.theplatform.com/r/HNK2IC/iIS84zsO9brm,yZfd0Uh8GvOY,...', offset 0, playlist 0 [https @ 0000027219f24980] Cannot reuse HTTP connection for different host: nbcuadops-vh.akamaihd.net:-1 != redirect.manifest.theplatform.com:-1 [AVIOContext @ 0000027219813480] Statistics: 2187756 bytes read, 0 seeks [hls,applehttp @ 000002721980b380] keepalive request failed for 'https://redirect.manifest.theplatform.com/r/HNK2IC/iIS84zsO9brm,yZfd0Uh8GvOY,...', retrying with new connection: Invalid argument [hls,applehttp @ 000002721980b380] Opening 'https://redirect.manifest.theplatform.com/r/HNK2IC/iIS84zsO9brm,yZfd0Uh8GvOY,...' for reading [hls,applehttp @ 000002721980b380] HLS request for url 'https://nbcuadops-vh.akamaihd.net/i/ads/tvn/CISCOWEB30V2H_ENT_MEZZ/CISCOWEB30V2H_ENT_MEZZ_....ts?set-akamai-hls-revision=5', offset 0, playlist 0 [https @ 0000027219e93600] Opening 'https://nbcuadops-vh.akamaihd.net/i/ads/tvn/CISCOWEB30V2H_ENT_MEZZ/CISCOWEB30V2H_ENT_MEZZ_.../segment8_6_av.ts?set-akamai-hls-revision=5' for reading [hls,applehttp @ 000002721980b380] HLS request for url 'https://redirect.manifest.theplatform.com/r/HNK2IC/iIS84zsO9brm,yZfd0Uh8GvOY,...', offset 0, playlist 0 [https @ 0000027219f24980] Cannot reuse HTTP connection for different host: nbcuadops-vh.akamaihd.net:-1 != redirect.manifest.theplatform.com:-1 [AVIOContext @ 0000027219f19980] Statistics: 2226860 bytes read, 0 seeks [hls,applehttp @ 000002721980b380] keepalive request failed for 'https://redirect.manifest.theplatform.com/r/HNK2IC/iIS84zsO9brm,yZfd0Uh8GvOY,...', retrying with new connection: Invalid argument [hls,applehttp @ 000002721980b380] Opening 'https://redirect.manifest.theplatform.com/r/HNK2IC/iIS84zsO9brm,yZfd0Uh8GvOY,...' for reading [hls,applehttp @ 000002721980b380] HLS request for url 'https://tvecnbchls.nbcuni.com/tve-adstitch/6579/21/01/10/...-2.ts', offset 0, playlist 0 [https @ 0000027219e89d40] Cannot reuse HTTP connection for different host: nbcuadops-vh.akamaihd.net:-1 != tvecnbchls.nbcuni.com:-1 [AVIOContext @ 000002721982c740] Statistics: 7912544 bytes read, 0 seeks [hls,applehttp @ 000002721980b380] keepalive request failed for 'https://tvecnbchls.nbcuni.com/tve-adstitch/6579/21/01/10/...-2.ts', retrying with new connection: Invalid argument [hls,applehttp @ 000002721980b380] Opening 'https://tvecnbchls.nbcuni.com/tve-adstitch/6579/21/01/10/...-2.ts' for reading [hls,applehttp @ 000002721980b380] New video stream 0:2 at pos:14497244 and DTS:1.442s [mp4 @ 000002721a68b7c0] Non-monotonous DTS in output stream 0:1; previous: 1150995, current: -120868; changing to 1150996. This may result in incorrect timestamps in the output file. (snip) [mp4 @ 000002c00fa6c040] Non-monotonous DTS in output stream 0:1; previous: 35427983, current: 7020507; changing to 35427984. This may result in incorrect timestamps in the output file. No more output streams to write to, finishing. frame= 779 fps=2.4 q=-1.0 Lsize= 46101kB time=00:13:23.35 bitrate= 470.1kbits/s speed=2.51x video:13382kB audio:32159kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 1.230501% Input file #0 (https://east.manifest.na.theplatform.com/m/HNK2IC/iIS84zsO9brm,yZfd0Uh8GvOY,,,/6.m3u8?... Input stream #0:0 (video): 779 packets read (13702968 bytes); Input stream #0:1 (audio): 115634 packets read (33740250 bytes); Total: 116413 packets (47443218 bytes) demuxed Output file #0 (file:Calle Ocho, Miami-4293059.mp4.part): Output stream #0:0 (video): 779 packets muxed (13702968 bytes); Output stream #0:1 (audio): 115634 packets muxed (32930812 bytes); Total: 116413 packets (46633780 bytes) muxed [AVIOContext @ 000002c00f37ee00] Statistics: 14 seeks, 191 writeouts [AVIOContext @ 000002c0149e4dc0] Statistics: 35624496 bytes read, 0 seeks [AVIOContext @ 000002c0149ecfc0] Statistics: 5224144 bytes read, 0 seeks [AVIOContext @ 000002c00ef63ec0] Statistics: 422320 bytes read, 0 seeks [ffmpeg] Downloaded 47207609 bytes [download] 100% of 45.02MiB in 05:21
Code:ffprobe -hide_banner "Calle Ocho, Miami-4293059.mp4" Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'Calle Ocho, Miami-4293059.mp4': Metadata: major_brand : isom minor_version : 512 compatible_brands: isomiso2avc1mp41 encoder : Lavf58.17.101 Duration: 00:13:23.40, start: 0.000000, bitrate: 470 kb/s Stream #0:0(und): Video: h264 (Main) (avc1 / 0x31637661), yuv420p(tv, bt709), 1920x1080 [SAR 1:1 DAR 16:9], Closed Captions, 4218 kb/s, 29.97 fps, 29.97 tbr, 90k tbn, 59.94 tbc (default) Metadata: handler_name : VideoHandler Stream #0:1(und): Audio: aac (HE-AAC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 327 kb/s (default) Metadata: handler_name : SoundHandler
So let's take a look at the dumped files.
4293059_https_-_east.manifest.na.theplatform.com_m_HNK2IC_iIS84zs O9brm_yZfd0Uh8GvOY_mOI___.dump
JSON file with links to the captions, chapter markers, etc. Same as what you see on any other video. Nothing interesting here; no links to the actual video
4293059_https_-_east.manifest.na.theplatform.com_m_HNK2IC_iIS84zs O9brm_yZfd0Uh8GvOY_mOI___.dump
This is thee master.m3u8 file. I'd post it but the URLs have expired by now. You can see in the later requests that 6.m3u8 was downloaded, which is the 1080p copy
4293059_https_-_friendship.nbc.co_v2_graphqlquery=query+bonanzaPa ge%28%0A++%24app%3A+___.dump
More JSON related to the episode. No links
4293059_https_-_friendship.nbc.co_v2_graphqlvariables=%7B%22oneAp p%22%3A+true%2C+%22userId%22%3A+____.dump
More JSON related to the episode. No links
4293059_https_-_link.theplatform.com_s_HNK2IC_media_e06vXD_jas_Sd id=_____.dump
Here's where we get the link to master.m3u8:
Code:<smil xmlns="http://www.w3.org/2005/SMIL21/Language"> <head> <meta name="manifestServiceUrl" content="https://east.manifest.na.theplatform.com/m/HNK2IC/iIS84zsO9brm,yZfd0Uh8GvOY,..."/> <meta name="interstitialUrl" content="https://east.manifest.na.theplatform.com/m/HNK2IC/iIS84zsO9brm,yZfd0Uh8GvOY,..."/> <meta name="midroll.adPattern" content="time:30000|30000"/> <meta name="startingBitrate" content="2500000"/> </head> <body> <seq> <par> <video src="https://east.manifest.na.theplatform.com/m/HNK2IC/iIS84zsO9brm,yZfd0Uh8GvOY,..." title="Calle Ocho, Miami" abstract="Marcus Lemonis gets the inside scoop on Calle Ocho, Little Havana, the booming capital of Cuban American culture in Miami." dur="2599000ms" guid="4293059" categories="Series/Streets of Dreams" keywords="cnbc,prime,reality,marcus lemonis,the profit,ceo,entrepreneur,small business,diamond distrcit,new york,nyc,denver,san diego,nashville,music row,miami,calle ocho,usa,economy,business" ratings="urn%3Av-chip:tv-pg" provider="CNBC" type="application/x-mpegURL" height="720" width="1280"> <param name="dayPart" value="Primetime"/> <param name="entitlement" value="auth"/> <param name="episodeNumber" value="103"/> <param name="externalAdvertiserId" value="CNBC_ANV_1234567"/> <param name="fullEpisode" value="true"/> <param name="seasonNumber" value="1"/> <param name="show" value="Streets of Dreams"/> <param name="advertisingGenre" value="Reality and Game Show"/> <param name="programmingType" value="Full Episode"/> <param name="seriesTmsId" value="SH036725710000"/> </video> <textstream src="https://tve-static-cnbc.akamaized.net/prod/caption/241/891/210110_4293059_Calle_Ocho__Miami.tt" lang="en" type="application/smptett+xml"/> <imagestream src="https://tve-static-cnbc.akamaized.net/prod/image/241/891/210110_4293059_Calle_Ocho__Miami_1200.fs" width="190" height="107" type="application/filmstrip+json"/> </par> </seq> </body> </smil>
The HTML of the page. Too big to post -
https://www.nbc.com/streets-of-dreams/video/calle-ocho-miami/4293059
Hulu
Lots of other episodes I want to download, so am interested in the methodology -
Bump - still looking for ways of getting the actual m3u8 instead of just the ads
Similar Threads
-
how to download video files with "mpd" (mpeg dash manifest)?
By nzhd in forum Video Streaming DownloadingReplies: 59Last Post: 17th Apr 2021, 05:26 -
help to download videos DRM or with "mpd" (mpeg dash manifest)?
By Marven in forum ProgrammingReplies: 4Last Post: 17th Apr 2021, 05:24 -
"Recover"/Convert/Download Local Silverlight files to video
By enzeru in forum Video Streaming DownloadingReplies: 6Last Post: 15th Jan 2018, 03:07 -
View and Download portion of YouTube video that's "boxed".
By Barrington in forum Newbie / General discussionsReplies: 3Last Post: 4th Jun 2017, 03:25 -
how to download caption of streaming video in "srt" format?
By jj20 in forum SubtitleReplies: 6Last Post: 3rd Mar 2017, 10:01