Why streamlink downloads this but ytdlp says 404 not found

[[ss]vegeta]

Hello
Here's a link
https://transfer.sh/Uo0mw7/a.m3u8
Why streamlink can download, but ytdlp says 404 not found?

[jack_666]

The below should be accessible in the browser, but it's not
https://5fd718c03c904.streamlock.net/Mtv1vod/_definst_/smil:2022-10-15_Mtv1_1405698301..._b2500000_0.ts
Access to 5fd718c03c904.streamlock.net was denied


[https @ 00000208447e1c00] HTTP error 403 Forbiddenq= 0B f=0/0
[hls @ 000002083e7c7f00] Failed to open segment 0 of playlist 0/0

streamlink https://transfer.sh/Uo0mw7/a.m3u8 best -o video.ts

[cli][info] Found matching plugin hls for URL https://transfer.sh/Uo0mw7/a.m3u8
[cli][info] Available streams: live (worst, best)
[cli][info] Opening stream: live (hls)
[cli][info] Writing output to
xxxxx\video.ts
[stream.hls][error] Failed to fetch segment 0: Unable to open URL: https://5fd718c03c904.streamlock.net/Mtv1vod/_definst_/smil:2022-10-15_Mtv1_1405698301..._b2500000_0.ts (403 Client Error: Forbidden for url: https://5fd718c03c904.streamlock.net/Mtv1vod/_definst_/smil:2022-10-15_Mtv1_1405698301..._b2500000_0.ts)


Suspect that a referer is reqiured

[[ss]vegeta]

The link prolly expired or something. It worked without any header. Let me grab it again.
https://transfer.sh/gCASzy/b.m3u8
Might need North Macedonia VPN

[Attachment 67228 - Click to enlarge]

[thedebutent59]

you can try this with:

yt-dlp --allow-unplayable-formats -F "https://transfer.sh/gCASzy/b.m3u8"

here is a picture:

[Attachment 67230 - Click to enlarge]

[[ss]vegeta]

you can try this with:

yt-dlp --allow-unplayable-formats -F "https://transfer.sh/gCASzy/b.m3u8"

here is a picture:

[Attachment 67230 - Click to enlarge]

No, I can't try with that, that only lists available formats, it doesn't download.

[lfer94]

Did you tried using the manifest link instead the direct link?

[AbortRetryFail]

C:\Users\me>yt-dlp --no-check-certificate --add-header "Accept:*/*" "https://transfer.sh/gCASzy/b.m3u8"
[generic] b: Downloading webpage
[generic] b: Downloading m3u8 information
[info] b: Downloading 1 format(s): 0
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 1202
[download] Destination: b [b].mp4
[download]   0.2% of ~   1.67GiB at  933.80KiB/s ETA 28:01 (frag 2/1202)
ERROR: Interrupted by user

(For whatever reason, both streamlink and yt-dlp on my system can't handle transfer.sh's LE certificate. EDIT: Oh, duh, they don't read Windows' certificate store and wouldn't see Fiddler's cert.)

My guess is it's to do with the host you've put the file on. I hooked up Fiddler to both streamlink and yt-dlp and noticed the latter tries a bit harder to pretend to be a browser and sends "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" and transfer.sh responds by sending it the webpage you would get in a browser if viewing the link. With "Accept:*/*" (or the lack of that header entirely) I assume transfer.sh assumes it's being used by cURL or something and sends the file directly. A perusal of the transfer.sh server code might confirm this or not.

[[ss]vegeta]

Did you tried using the manifest link instead the direct link?

Yes, I was getting 40X kinds of errors. I tried with a few headers with yt-dlp, still no luck.

Actual playlist.m3u8

https://live.neotv.mk/EPG_LoadBalancer_production/inellipse/v1/tvprograms/140569830/playlist.m3u8?authorization=TokenWebUser%20T1RFMU1EZzFPR1l0T1RnMU1DMDBZVGhtTFRnd1pXVXROVEkzWldObE1qY3daV0V6X1QxUldhRTFVYkcxTmJVa3pYekV6T0RZNU16TV8xNjY1ODUwNzUyMzExX1QxUldhRTFVYkcxTmJVa3pYekV6T0RZNU16TV9aR0psWXpka1l6TXRNVE13WkMwMFlUUmxMV0UwTmpNdFpEazVabVEzWkdJMVltVTM=;Secret%20MKVRl1GrjKj6I4gMCFbQ6WgpQx8=&DeviceKind=Web

Actual chunks.m3u8

https://5fd718c03c904.streamlock.net/Mtv1vod/_definst_/smil:2022-10-15_Mtv1_1405698301.smil/chunklist_w1974613399_b2500000.m3u8

Headers for playlist.m3u8

cookies = {
    'ppu': 'U2FsdGVkX19GKSwThji9JdkQtISaEkzVbRarhiu7AJc=',
    'ppp': 'U2FsdGVkX1+Aa31EolJ0UrZT5ueECHMLGzxeFtIvSpc=',
    'resellerID': '0001',
    'firstName': 'p',
    'lastName': 'k',
    'userPin': '0000',
    'leftMenuCategories': 'false',
    'logoPath': 'https://stream3.neotel.com.mk/uploads/resellers/neotel_logo.png',
    'currency': 'MKD',
    'ver': '2',
    'authorization': 'OTE1MDg1OGYtOTg1MC00YThmLTgwZWUtNTI3ZWNlMjcwZWEz',
}

headers = {
    'Accept': '*/*',
    'Accept-Language': 'en-US,en;q=0.6',
    'Connection': 'keep-alive',
    # Requests sorts cookies= alphabetically
    # 'Cookie': 'ppu=U2FsdGVkX19GKSwThji9JdkQtISaEkzVbRarhiu7AJc=; ppp=U2FsdGVkX1+Aa31EolJ0UrZT5ueECHMLGzxeFtIvSpc=; resellerID=0001; firstName=p; lastName=k; userPin=0000; leftMenuCategories=false; logoPath=https://stream3.neotel.com.mk/uploads/resellers/neotel_logo.png; currency=MKD; ver=2; authorization=OTE1MDg1OGYtOTg1MC00YThmLTgwZWUtNTI3ZWNlMjcwZWEz',
    'Referer': 'https://live.neotv.mk/',
    'Sec-Fetch-Dest': 'empty',
    'Sec-Fetch-Mode': 'cors',
    'Sec-Fetch-Site': 'same-origin',
    'Sec-GPC': '1',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36',
}

Headers for chunks.m3u8

headers = {
    'Accept': '*/*',
    'Accept-Language': 'en-US,en;q=0.6',
    'Connection': 'keep-alive',
    'Origin': 'https://live.neotv.mk',
    'Referer': 'https://live.neotv.mk/',
    'Sec-Fetch-Dest': 'empty',
    'Sec-Fetch-Mode': 'cors',
    'Sec-Fetch-Site': 'cross-site',
    'Sec-GPC': '1',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36',
}

I want to download with yt-dlp

My guess is it's to do with the host you've put the file on. I hooked up Fiddler to both streamlink and yt-dlp and noticed the latter tries a bit harder to pretend to be a browser and sends "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" and transfer.sh responds by sending it the webpage you would get in a browser if viewing the link. With "Accept:*/*" (or the lack of that header entirely) I assume transfer.sh assumes it's being used by cURL or something and sends the file directly. A perusal of the transfer.sh server code might confirm this or not.

Thanks for this investigation. Adding that header does the job to download with yt-dlp.

Now, can anyone figure out how to download with yt-dlp without me first downloading the m3u8, adding the prefix to the ts files, uploading to transfer.sh...

[jack_666]

... can anyone figure out how to download with yt-dlp without me first downloading the m3u8, adding the prefix to the ts files, uploading to transfer.sh...

I fully realize that this in not what you are asking. This is a temporary work around.


1) streamlink can work with locally stored m3u8 manifest ... therefore there is no need to upload to transfer.sh
2) For a temporary workaround, use a script.
2a) Script will download the m3u8 and add the required data on the fly. Script will then load the modified m3u8 file locally.
2B) Script will then spawn streamlink using the modified m3u8 file


Ok, let's resume what is required to enable yt-dlp to work. I assume that cookies were already tried.

[[ss]vegeta]

I assume that cookies were already tried.

Don't, because I was using a browser that yt-dlp doesn't support

[jack_666]

I have a tiny challenge. Unable to access either of the two m3u8.



Wish to use curl "url" -H bla bla bla to download the manifest

[AbortRetryFail]

Now, can anyone figure out how to download with yt-dlp without me first downloading the m3u8, adding the prefix to the ts files, uploading to transfer.sh...

Heh, yt-dlp really doesn't want you using a local manifest (so piping it through whatever text manipulation tool is out): "ERROR: [generic] Unable to download webpage: <urlopen error file:// scheme is explicitly disabled in yt-dlp for security reasons> (caused by URLError('file:// scheme is explicitly disabled in yt-dlp for security reasons'))" edit: there's https://github.com/yt-dlp/yt-dlp/issues/3675#issuecomment-1127122803 as an option too

yt-dlp has many of them and I could've missed an option, but the only remote possibility I saw of doing what you want is writing your own quick extractor plugin: https://github.com/yt-dlp/yt-dlp#plugins

In theory you can have the plugin match requests for whatever, retrieve the m3u8 in the plugin, manipulate the text as needed and pass it off to the generic extractor. I can't really check any of this - not surprisingly, I don't have any access to North Macedonian VPNs. If the videos the site serves aren't DRMed, consider opening a yt-dlp issue.

[firaso]

Hello
Here's a link
https://transfer.sh/Uo0mw7/a.m3u8
Why streamlink can download, but ytdlp says 404 not found?

[Attachment 67232 - Click to enlarge]
bro use this tool

https://github.com/der3318/m3u8-dlwrapper

[Quint]

Did you try to pass the headers via --add-header FIELD:VALUE ?