drm free content batch downloader, widefrog

[stabbedbybrick]

I'm looking at a way to see if you can find a way for definite if its emulated or not the second one is android 14 sdk 34 gey got with ghydra added xml

Outside of testing the CDM against something where having a real device matters, I don't think there is a 100% definitive way. Using the build info is probably the easiest way.

You can include more keywords to check for:

from pywidevine.device import Device

device = Device.load("device.wvd")
build = next(x.value for x in device.client_id.client_info if "build_info" in x.name).lower()

if any(word in build for word in ["sdk", "emu", "userdebug"]):
    do_something(

[StrangeX]

Hi, I get this error for mediaset infinity, maybe I missing something?

PS C:\Users\Administrator\Desktop\scripts\mediaset\mediaset> widefrog.exe "https://mediasetinfinity.mediaset.it/video/giocosporco/prima-puntata--maradona-e-pistorius_F313159202000101"
[INFO] Starting the service initialization stage.
[APP_ERROR] Failed to download: https://mediasetinfinity.mediaset.it/video/giocosporco/prima-puntata--maradona-e-pisto...13159202000101. Reason: Failed to initialize the mediasetinfinity_mediaset_it service. Solution: Debug the service.
[INFO] Current progress: [1/1]
[INFO] Starting the collections extraction stage.
[INFO] Starting the media data extraction stage.
[INFO] Finished generating the download commands.
[WARNING] Saved the failed URLs to app_files\cmds_failed.txt
[WARNING] No download commands have been generated.

[2nHxWW6GkN1l916N3ayz8HQoi]

Make sure you're using the latest version and try again (at least 2-3 times in a row). If it keeps failing add the parameter --debug and show the output please

[StrangeX]

oh, great, you're right!
now I solved it with the latest version released by you and it works, here's the proof


[Attachment 81828 - Click to enlarge]

[2nHxWW6GkN1l916N3ayz8HQoi]

v2.4.0:
=> + channel4.com (thanks @psxman for testing)
=> + france.tv
=> + netflix.com (only the free non-DRM /tudum/ fan section)
=> + play.xumo.com
=> + watch.blaze.tv (thanks again @psxman for testing this one as well)

Minor changes:
=> updated the requirements.txt file (only relevant if you use python)

Off topic: @brick, @obo, @psxman, thanks a lot for your information regarding some possible ways for detecting if a l3 cdm is emulated or not. Sadly, from what I see, no matter what criteria you choose, you risk to consider some emulated cdms to not be emulated. It's best if I leave it as it is right now until some new techniques may appear in the future.

[bLuBlUB]

Hi, great code!

when I tried NPO.nl for the first time, the code crashed. however, another URL from NPO worked fine. I did some research myself, and it turns out there's a "CALL_TO_ACTION" banner that causes your script to crash. it seems that this banner isn't present at every video. I resolved it by adding this code:

if d.get('type') == 'CALL_TO_ACTION':
    continue

on line 205 in services/npo_nl.py.

                for d in data:
                    if d.get('type') == 'CALL_TO_ACTION':
                        continue
                        
                    if d["slug"] == slug and type(d["productId"]) is str:
                        try:
                            d["programKey"]
                        except:
                            continue

                        response = d["productId"]
                        break
                else:
                    continue
                break

this is the url that otherwise doesn't work: https://npo.nl/start/serie/alles-op-tafel/seizoen-1/alles-op-tafel/afspelen


[Attachment 81840 - Click to enlarge]

hopefully, this information is valuable. thanks again for your fantastic work!

[2nHxWW6GkN1l916N3ayz8HQoi]

hopefully, this information is valuable

Thanks, not only for finding an error, but also for coming up with a fix and going the extra mile by making an account just to let me know this (seeing as this is your first post). I really appreciate it. A minor improvement to your fix is to just check if the key "slug" exists and if not, ignore that element. Instead of trying to find all possible types, CALL_TO_ACTION being only one of them for the moment.

I'll add this fix in the next version. Thanks again!

[colebri]

hello

what a powerfull an amazing tool, thank so much to create it! it allows me to download more than 50 videos in half a day where it took me 2 full days staying in front of the computer to download them manually. A wonderfull job!

I juste have 2 questions for my knowledge and my future downloads.

Since I'm not familiar with Regex, I would like to select 2 audio tracks (fre, eng) on videos which have 3 tracks (fre, eng, int).

In cmds.txt file automatically generated, the command concerning audio tracks was:

-sa best -da id=audio_div

So, first question, what the meaning of "audio_div"?

And second question, for selecting the 2 wanted tracks, I tried several solutions, like:

-sa "lang=fr|en"

-sa "lang=fr,en"

-sa "lang=fr&en"

and maybe other ones, and either it download no audio tracks, or display an error message indicating a wrong command.

The only way I managed to do it was to do

-sa all -da "lang=in"

but if there are more than 3 tracks, I would encouter the same issue.

I guess the the solution is quite simple, but I was not able to find it.

Any idea on how to do it?

thanks a lot!

[whs912km]

hi 2nHxWW6GkN1l91, first of all thanks for this really awesome tool.

also i would like to know if it is possible to implement it for amz service too. Thanks

[2nHxWW6GkN1l916N3ayz8HQoi]

Always glad to hear it's being put to good use. -da id=audio_div means drop audio with id "audio_div". When I added the "watch.globaltv.com" and was testing it, some of their videos had descriptive audio meant for blind/visually impaired users. And that audio was a little annoying so I dropped it. You can always get rid of that option if you want it. Those default parameters you see are meant to be customized according to the user's needs.

Now for your issue. Let's see first what N_m3u8DL-RE tells us (select-audio is equivalent to sa)

N_m3u8DL-RE --morehelp select-audio
More Help:

--select-audio

Select audio streams by regular expressions. ref --select-video

Examples:
# select all
-sa all
# select best eng audio
-sa lang=en:for=best
# select best 2, and language is ja or en
-sa lang="ja|en":for=best2
-sa role="main":for=best

So you were on the right track but you also needed something else:

-sa lang="spa|eng":for=best2

Now let's see how we can add that in the json config file. You will notice all the parameters are already between double quotes since that's how a json is formatted. To include "spa|eng" you must add \ for each quote. So it becomes

-sa lang=\"spa|eng\":for=best2

Now if you run the tool and you have the value of MEDIA_DOWNLOADER to 1, thus no terminals are spawned and everything happens in the same window, everything will work like this. However if you let the tool spawn a lot of parallel terminals, you also need to add this change

-sa lang=\"spa^|eng\":for=best2

So the json becomes

.
.
.
"VOD": "-sv best -sa lang=\"spa^|eng\":for=best2 -da id=audio_div -ss all -mt --check-segments-count false --no-log",
.
.
.

With or without that ^ character depending on the case. I hope this makes sense. Thanks for bringing this issue since I didn't expect that special character to be in the input. I'll fix it in the next version so you won't have to worry about ^

Additional note, the tool is designed to apply these set of parameters to ALL the found manifests no matter the site/service. So if you want only spanish and you have 10 shows and only 9 of them have spanish, that last 1 may fail. You can investigate the manifests by removing the auto selection parameters (-sv / -sa / -ss) and see what's there to offer.

hi 2nHxWW6GkN1l91, first of all thanks for this really awesome tool.

also i would like to know if it is possible to implement it for amz service too. Thanks

Thanks! Unfortunately I don't add sites by request. I only add them from time to time on random criteria. Also, isn't amz=amazon? That's a paid service.

[whs912km]

yes, amz=amazon
anyway ok, i've understand. thanks

[bLuBlUB]

I'll add this fix in the next version. Thanks again!

you're welcome!

I also tested Rakuten.tv with this URL: https://www.rakuten.tv/nl/player/movies/stream/air-bud-golden-receiver
This movie had both Dutch and English audio, and I wanted the Dutch audio. However, with the current code, I only received the English audio.

After some research on Rakuten's site, I noticed that the Dutch audio is labeled as "NLD" I entered that here:

        for a in audio_lang:
            response = json.loads(requests.post(
                rakuten_tv.STREAM_URL.format(
                    category="avod" if content_category in ["stream", "live_channels"] else "me"
                ),
                params={"device_identifier": rakuten_tv.DEVICE_ID},
                json={
                    'audio_language': "NLD",
                    'audio_quality': audio_quality,
                    'classification_id': rakuten_tv.CLASSIFICATION_ID,
                    'content_id': json_content_id,
                    'content_type': content_type,
                    'device_serial': 'device_serial',
                    'device_stream_video_quality': video_quality,
                    'player': f'{rakuten_tv.DEVICE_ID}:{manifest_type}',
                    'subtitle_language': subtitle_language,
                    'support_closed_captions': True,
                    'video_type': "stream" if content_category in ["stream", "live_channels"] else content_category
                }
            ).content.decode())

This way, I did get the Dutch audio for the movie. As a result, I took a step back and checked the output of audio_lang:

[{'type': 'languages', 'id': 'ENG', 'numerical_id': 18, 'name': 'Engels', 'abbr': 'ENG'}]

So not all audio options were being passed correctly. After a lot of research, I was able to resolve that with these changes:

new package:

import numpy as np

On line 215 (or 216 if you add the import), "response[0]" should be without the [0].

response = response

Then, I changed all values except for audio to response[0], thus using the English version. For audio_lang, however, I combined all audio languages into one array.

Starting from line 244 (or 245):

        drm_types = response[0]["streaming_drm_types"] #edit response to response[0]
        manifest_type = None
        for drm_type in drm_types:
            if drm_type["id"].lower().startswith("dash-"):
                manifest_type = drm_type["id"]
                break
        if manifest_type is None:
            manifest_type = drm_types[0]["id"]

        audio_quality = response[0]["audio_qualities"] #edit response to response[0]
        audio_quality = sorted(audio_quality, key=lambda ad: ad["numerical_id"], reverse=True)
        audio_quality = f'{audio_quality[0]["numerical_id"]}.0'

        audio_lang = np.concatenate([res["audio_languages"] for res in response]) #edit audio languages to 1 array
        subtitle_language = response[0]["subtitle_languages"][-1]["id"] #edit response to response[0]

        video_quality = response[0]["video_qualities"] #edit response to response[0]
        video_quality = sorted(
            video_quality, reverse=True,
            key=lambda v: rakuten_tv.RES_PRIORITY[v["id"]]
        )
        video_quality = video_quality[0]["id"]

        json_content_id = content_id[0]
        if content_type == "episodes":
            json_content_id = content_id[1]

        manifest = []
        subtitles = []
        pssh_values = []
        additional = {}
        has_non_mpd = False

And as a result, audio_lang receives all audio outputs:

[{'type': 'languages', 'id': 'ENG', 'numerical_id': 18, 'name': 'Engels', 'abbr': 'ENG'}
 {'type': 'languages', 'id': 'NLD', 'numerical_id': 17, 'name': 'Nederlands / Vlaams', 'abbr': 'NLD'}]

And after that, your code works smoothly with both audio sources! I hope this helps. Feel free to let me know if anything is unclear! still thanks for your great work.

[2nHxWW6GkN1l916N3ayz8HQoi]

That's kinda going into undefined territory. I always received a list with a single element on my rakuten region that's why I did response[0]. I'll add that, but I have no way of knowing if it'll behave as expected across all regions. I think it's easier to store in a separate variable the original list and iterating through it + append all audio languages, than replacing all with response[0] + another import to slow down the script.

[colebri]

Hello

thanks for this fast answer!

Always glad to hear it's being put to good use. -da id=audio_div means drop audio with id "audio_div". When I added the "watch.globaltv.com" and was testing it, some of their videos had descriptive audio meant for blind/visually impaired users. And that audio was a little annoying so I dropped it. You can always get rid of that option if you want it. Those default parameters you see are meant to be customized according to the user's needs.

ok I got it, I didn't thought that it could be related to a specific site, now it's clear

So you were on the right track but you also needed something else:

-sa lang="spa|eng":for=best2

I tried "morehelp" on --select-audio but since there was only one quality per language, I thought it was useless to specify the value "best" with the field "for"

Now let's see how we can add that in the json config file. You will notice all the parameters are already between double quotes since that's how a json is formatted. To include "spa|eng" you must add \ for each quote. So it becomes

-sa lang=\"spa|eng\":for=best2

Now if you run the tool and you have the value of MEDIA_DOWNLOADER to 1, thus no terminals are spawned and everything happens in the same window, everything will work like this. However if you let the tool spawn a lot of parallel terminals, you also need to add this change

-sa lang=\"spa^|eng\":for=best2

not the first time that I have trouble with these f**king \ !! thanks for the trick!

I tested right now and it works fine! thanks a lot!

[colebri]

directly on N_m3u8DL-RE with your code, and just for see, I tried with:

 -sa id="6|8"

since 6 and 8 are the id of the tracks I want to grab

but it didn't work, when using just one track (6 or 8) it works, but not with 2. Is it only possible to select multitrack with the "lang" field?

[2nHxWW6GkN1l916N3ayz8HQoi]

try

-sa id="6|8":for=best2

general formula

-sa id="id1|id2|...|idn":for=bestn

[colebri]

[QUOTE=2nHxWW6GkN1l916N3ayz8HQoi;2748827]try

-sa id="6|8":for=best2

yes it's ok, if thought I tried already but obviously not. Since the id identify exactly the track I thought that specifying "best" was unnecessary

thanks once again!

[2nHxWW6GkN1l916N3ayz8HQoi]

v2.5.0:
=> + ninateka.pl
=> + video.telequebec.tv

Minor changes:
=> fixed some bugs for npo.nl and rakuten.tv (thanks @blub for finding those issues)
=> fixed some problems with using the character | as an input for n_m3u8dl-re parameters

[diesel]

hello I can't download a video from rmcbfm I added my email and password and added the license in the rmcbfmplay_com service file

[2nHxWW6GkN1l916N3ayz8HQoi]

Works fine for me.

>python widefrog.py "https://www.rmcbfmplay.com/video/rmc-story/ma-maman-a-disparu-lhistoire-vraie-de-jennifer-dulos?contentId=Product::NEUF_NUM23_N23808420883527&universe=PROVIDER"
[INFO] Current app version: 2.5.0
[INFO] Running on: windows/64bit/python 3.12.2
[INFO] Starting the service initialization stage.
[INFO] Current progress: [1/1]
[INFO] Starting the collections extraction stage.
[INFO] Current progress: [1/1]
[INFO] Starting the media data extraction stage.
[INFO] Current progress: [1/1]
[INFO] Finished generating the download commands.
[INFO] Saved the generated download commands to app_files\cmds.txt
[INFO] If you used a VPN, turn it off to avoid wasting data for the downloading stage. Also you may edit the app_files\cmds.txt file if necessary. Type yes when ready: no
[INFO] Cancelling the downloading stage.

N_m3u8DL-RE "https://ncdn-cu-kairos.pfd.sfr.net/dashcenc/catchup/NEUF_NUM23/56c/N23_f3a559c182_799fc772ca894198a/manifest.mpd" --key aa8be47a53e629b0e0533b28450a1cbb:a8f3e30c8fc28072e8545cda37a997d7 --save-dir "media\rmcbfmplay_com" --save-name "Ma_maman_a_disparu_Lhistoire_vraie_de_Jennifer_Dulos" -M format=mkv

Try deleting the folder with the downloader and download it again from scratch. I notice you're using the python code. Make sure:
- you installed properly the python packages as it was detailed in the guide
- "rmcbfmplay_com.py" is found in the services folder
- you're using a french IP with a VPN that's not detected (see if you can play the video in browser)
- you're using an email/password that is not affiliated directly with google or other big providers (just the classic way that you login using their site directly)

I have no idea what "license" you added. There's no need to provide anything. Everything comes with the latest zip. Only the account is needed.

[diesel]

add license : LICENSE_URL = 'https://ws-backendtv.rmcbfmplay.com/asgard-drm-widevine/public/licence'
files rmcbfmplay_com.py

i'm french so french ip
debug command :




in scratch it works but you put the normal l3 key

[2nHxWW6GkN1l916N3ayz8HQoi]

Yep. It tells you exactly whats the issue. chomp_js is not installed. And when you try to install, it tells that you need visual c++ 14.0 and from where to get it. If you want to solve this issue for the future, there are already forum posts at a google search away that explain more in detail. Google is your friend
https://stackoverflow.com/questions/44951456/pip-error-microsoft-visual-c-14-0-is-required

But if you notice this is only needed for veeps_com. So if you're not interested in that service, delete it manually from the service folder and you should have no problem anymore with chomp js

Edit: no idea why you need to add that license manually in the code since it's obtained at runtime through some api calls, but ok

[diesel]

I listened to your advice I deleted veep
but still have weird errors

[2nHxWW6GkN1l916N3ayz8HQoi]

When you download from scratch and run, always make sure you delete the previous app_files folder. After it's generated again, set your credentials.

[alian]

Hello, ninateka does not work for me, and 35mm works normally

PS C:\widefrog> python widefrog.py "https://ninateka.pl/movies,1/katarzyna-nosowska--made-in-polska,9141"
[WARNING] External tool is not installed: N_m3u8DL-RE. Download commands may not be launched properly.
[WARNING] The service channel4.com cannot be used because it is not properly implemented, has package errors or has syntax errors. After fixing the issue delete the app_files folder.
[WARNING] The service open.spotify.com cannot be used because it is not properly implemented, has package errors or has syntax errors. After fixing the issue delete the app_files folder.
[WARNING] The service veeps.com cannot be used because it is not properly implemented, has package errors or has syntax errors. After fixing the issue delete the app_files folder.
[INFO] Current app version: 2.4.0
[INFO] Running on: windows/64bit
[INFO] Starting the service initialization stage.
[USER_ERROR] Failed to download: https://ninateka.pl/movies,1/katarzyna-nosowska--made-in-polska,9141. Reason: No service available for ninateka.pl. Solution: Implement it.
[INFO] Current progress: [1/1]
[INFO] Starting the collections extraction stage.
[INFO] Starting the media data extraction stage.
[INFO] Finished generating the download commands.
[WARNING] Saved the failed URLs to app_files\cmds_failed.txt
[WARNING] No download commands have been generated.

[2nHxWW6GkN1l916N3ayz8HQoi]

Hello. Works fine for me.

python widefrog.py "https://ninateka.pl/movies,1/katarzyna-nosowska--made-in-polska,9141"
[INFO] Current app version: 2.5.0
[INFO] Running on: windows/64bit/python 3.12.2
[INFO] Starting the service initialization stage.
[INFO] Current progress: [1/1]
[INFO] Starting the collections extraction stage.
[INFO] Current progress: [1/1]
[INFO] Starting the media data extraction stage.
[INFO] Current progress: [1/1]
[INFO] Finished generating the download commands.
[INFO] Saved the generated download commands to app_files\cmds.txt
[INFO] If you used a VPN, turn it off to avoid wasting data for the downloading stage. Also you may edit the app_files\cmds.txt file if necessary. Type yes when ready: no
[INFO] Cancelling the downloading stage.

N_m3u8DL-RE "https://r.dcs.redcdn.pl/dash/o2/fina/ninateka/prod/vod/139c4edc-d980-429a-8f07-ae96e5a6075e/MOVIE/LIBCODER_SMOOTH_1080_FINA_320A/playlist.ism?indexMode" --key d56e57420a518646a03ab71a1730bc59:2a0c336f39918af659e0c1a748d2527b --save-dir "media\ninateka_pl" --save-name "Katarzyna_Nosowska_Made_in_Polska" -M format=mkv

Delete entirely the current tool version and make sure you're using the latest version and that you have N_m3u8DL-RE in system env variables. Also, since you're using python, make sure you installed all of the current dependencies in the requirements.txt file as it is explained and detailed in the guide.

If you don't want to install all of them, you can get rid of the services that can't be used due to this like channel4, spotify etc by manually deleting them from the services folder if you don't need them. Then delete the app_files folder, run the tool to generate it again and fill any credentials or input you may need.

[diesel]

hello

I didn't really understand how to download an entire season, in particular season 14 of alien fiction on bfm-rmc

https://www.rmcbfmplay.com/info-programme/rmc-bfm-play/alien-fiction-s14?contentId=Pro...verse=PROVIDER

otherwise your python script is great!

[2nHxWW6GkN1l916N3ayz8HQoi]

Hallo. It's explained in the site use cases. I see you're trying to pass a season URL as input. The tool only works with URLs that point to the base show.
From
https://www.rmcbfmplay.com/info-programme/rmc-bfm-play/alien-fiction-s14?contentId=Pro...verse=PROVIDER
click on touts sasons and you'll get to
https://www.rmcbfmplay.com/info-programme/rmc-bfm-play/alien-fiction?contentId=Product...verse=PROVIDER

then you use the query option

python widefrog.py "https://www.rmcbfmplay.com/info-programme/rmc-bfm-play/alien-fiction?contentId=Product::NEUF_BFMAVOD_BAV_496618363527&universe=PROVIDER" --query=14::14:

[diesel]

thank you very much

[2nHxWW6GkN1l916N3ayz8HQoi]

v2.6.0:
=> + cineverse.com (thanks @psxman for helping)
=> + player.pl
=> + stirr.com
=> + uefa.tv
=> + watch.shortly.film

Minor changes:
=> updated the requirements.txt file

Off topic: if anyone knows of any other big streaming sites that are only focused on football like uefa/fifa, let me know