[Release] WidevineFetch - Keys from just the License URL; Any Lic. Wrapping

[2nHxWW6GkN1l916N3ayz8HQoi]

@larley if you're curious about rakuten, https://forum.videohelp.com/threads/414775-problem-with-rakuten-tv#post2737874

I'm curios if there's a smart way that you can include this site

[larley]

Requires a custom script. I just sent this to lomero:

from urllib.parse import urlparse, parse_qs, urljoin

import requests

# Metadata URL RegEx: https://gizmo\.rakuten\.tv/.*/avod/streamings.*

def filter_dict(
        dictionary: dict,
        keys: list
) -> dict:
    return dict(
        (k, v)
        for k, v in dictionary.items()
        if k in keys
    )


def create_metadata(
        metadata: str,
        metadata_body
) -> tuple[str, dict, dict] | None:
    try:
        parsed_body = json.loads(metadata_body)
    except Exception:
        return

    parsed_url = urlparse(metadata)
    query = parse_qs(parsed_url.query)

    url = urljoin(metadata, parsed_url.path)

    params = {
        "device_identifier": "android",
        **filter_dict(
            query,
            [
                'classification_id'
            ]
        )
    }

    json_body = {
        "player": "android:DASH-CENC",
        "device_stream_video_quality": "HD",
        **filter_dict(
            parsed_body,
            [
                "hdr_type",
                "audio_quality",
                "content_id",
                "audio_language",
                "video_type",
                "device_serial",
                "content_type",
                "subtitle_language",
            ]
        )
    }

    return url, params, json_body


def create_manifest(metadata: str, metadata_body):
    if new_metadata := create_metadata(metadata, metadata_body):
        url, params, json_body = new_metadata
    else:
        return

    metadata_request = requests.post(
        url=url,
        params=params,
        json=json_body
    )
    if metadata_request.status_code != 200:
        return

    return metadata_request.json().get('data', {}).get('stream_infos', [{}])[0].get('url')


def create_license(metadata: str, metadata_body):
    if new_metadata := create_metadata(metadata, metadata_body):
        url, params, json_body = new_metadata
    else:
        return

    metadata_request = requests.post(
        url=url,
        params=params,
        json=json_body
    )
    if metadata_request.status_code != 200:
        return

    return metadata_request.json().get('data', {}).get('stream_infos', [{}])[0].get('license_url')

call create_license and create_manifest respectively to get 540p (max) compatible license and manifest URLs. The input is the gizmo metadata URL (its RegEx is at the top of the script)

EDIT: regarding Pepe's answer:
I was thinking about creating modules that are activated by a regex of a specific license URL. The problem is that their usage is quite limited, because all the data that the custom module gets has to be already present in the fetch data. (I guess VdoCipher could work)

Maybe build an API that modules can call to request additional information via custom dialogues?

[whs912km]

wow thank you for your very fast answer! and your custom script

I will try it as soon, thanks

[whs912km]

mmm ... how run this script ??

i've saved script as rakuten.py, put on folder with others scripts (and device.wvd too) but run
python rakuten.py

nothing appears ...

[2nHxWW6GkN1l916N3ayz8HQoi]

I was thinking about creating modules that are activated by a regex of a specific license URL. The problem is that their usage is quite limited, because all the data that the custom module gets has to be already present in the fetch data. (I guess VdoCipher could work)

Maybe build an API that modules can call to request additional information via custom dialogues?

Is that API going to be hosted locally by the user?

[larley]

An API doesn't have to be over the network. An api can also just be an 'endpoint' (like a class, not accessible to the user) in the code that can be imported by modules to open a dialog or whatever.

[2nHxWW6GkN1l916N3ayz8HQoi]

Oh yeah. You talk about the general sense of an API. In that case, it's a good choice. You could have a drop down list with custom sites (vdo/rakuten). The license URL regex may break. I've seen sites where they change their license URLs. Maybe not rakuten, but I've seen other sites do that. But that's up to you, the intention is good. I don't think you can "generalize" it without custom classes for those issue sites.

[whs912km]

mmm ... sorry for my intrusion into your technical details, but how do you run the above script ?

look my post#124

[larley]

append this to your script:

if __name__ == '__main__':
    url = input("gizmo url: ")
    json_string = input("json string: ")
    print(create_manifest(url, json_string))
    print(create_license(url, json_string))

and prepend:

import json

the 'json string' can be obtained from curlconverter. It's json as a string, not as a json object.

[whs912km]

maybe it's easy for you but not for all of us ...

i've add code into script. search and found gizmo url from this link https://www.rakuten.tv/it/player/movies/stream/amores-perros?q=amores%20perros

https://gizmo.rakuten.tv/v3/me/start?device_identifier=web&device_stream_audio_quality...market_code=it

from here copied as curlbash, paste on curlconverter.com and converted into json
then copied everything on script, when json string appears. but i get NameError: name 'create_manifest' is not defined


[Attachment 81575 - Click to enlarge]

[2nHxWW6GkN1l916N3ayz8HQoi]

I think it's easier to copy the raw json from the request tab of the detected license URL. It works with that. I think you just dropped it in curlconverter, and copy paste all. Which is wrong.

[larley]

Use the URL that matches the RegEx in the script i provided:

https://gizmo\.rakuten\.tv/.*/avod/streamings.*

[Attachment 81577 - Click to enlarge]

[whs912km]

more easy to filter 'streamings?' only

and this is the right url:

https://gizmo.rakuten.tv/v3/avod/streamings?classification_id=36&device_identifier=web&device_stream_audio_quality=2.0&device_stream_hdr_type=NONE&device_stream_video_quality=FHD&disable_dash_legacy_packages=false&locale=it&market_code=it

now the issue is about json string
from the url above, i've copied copy as cURL (windows)
then go to curlconverter.com, pasted and converted into JSON

{
    "url": "https://gizmo.rakuten.tv/v3/avod/streamings",
    "raw_url": "https://gizmo.rakuten.tv/v3/avod/streamings?classification_id=36&device_identifier=web&device_stream_audio_quality=2.0&device_stream_hdr_type=NONE&device_stream_video_quality=FHD&disable_dash_legacy_packages=false&locale=it&market_code=it",
    "method": "post",
    "headers": {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/115.0",
        "Accept": "application/json, text/plain, */*",
        "Accept-Language": "en-US,en;q=0.5",
        "Accept-Encoding": "gzip, deflate, br",
        "Content-Type": "application/json",
        "Origin": "https://www.rakuten.tv",
        "Sec-GPC": "1",
        "Connection": "keep-alive",
        "Referer": "https://www.rakuten.tv/",
        "Sec-Fetch-Dest": "empty",
        "Sec-Fetch-Mode": "cors",
        "Sec-Fetch-Site": "same-site",
        "Pragma": "no-cache",
        "Cache-Control": "no-cache",
        "TE": "trailers"
    },
    "queries": {
        "classification_id": "36",
        "device_identifier": "web",
        "device_stream_audio_quality": "2.0",
        "device_stream_hdr_type": "NONE",
        "device_stream_video_quality": "FHD",
        "disable_dash_legacy_packages": "false",
        "locale": "it",
        "market_code": "it"
    },
    "data": "{audio_language:ITA,audio_quality:2.0,classification_id:36,content_id:amores-perros,content_type:movies,device_make:firefox,device_model:GENERIC,device_serial:not implemented,device_stream_video_quality:HD,device_uid:a8abf951-f6af-450b-99a5-dc4b2b757a78,device_year:1970,gam_correlator:662535663383928,gdpr_consent_opt_out:0,gdpr_consent:CQDjD8AQDjD8AAcABBENBCFgAAAAAEPgACiQAAAUVghgAIAAWABUAC4AHAAPAAgABIADKAGgAagA4AB4AEQAI4ATAApQBcAF0AMQAaAA3gB6AD8AIQAQ0AiACJAEsAMMAZQAzQBogDnAHwAP0AhABFgCOgElALqAYoA14BuADiAHUAP6Af8BEwCLwEiAJWATEApEBXYC2AF5gMEAYsAywBqoDWQHjgP3Af2BH8CbwE5gJ3AUVAAA.YAAACHwAAAAA,hdr_type:NONE,ifa_subscriber_id:null,ifa_type:ppid,slinf:95,player_height:1080,player_width:1920,player:web:DASH-CENC:WVM,publisher_provided_id:1fa77255-18f8-4d76-b9e9-4cdcfa37219d,strict_video_quality:false,subtitle_formats:^[vtt^],subtitle_language:MIS,support_closed_captions:true,video_type:stream,support_thumbnails:true,app_bundle:com.rakutentv.web,app_name:RakutenTV,url:rakuten.tv,requestedContent:{live:false,duration:9248,genres:^[Drammatico,Film indipendente,Thriller^],id:125366,rating:14,title:Amores Perros,adOptions:{channelID:3273,gdprAccepted:false,gdprConsentString:CQDjD8AQDjD8AAcABBENBCFgAAAAAEPgACiQAAAUVghgAIAAWABUAC4AHAAPAAgABIADKAGgAagA4AB4AEQAI4ATAApQBcAF0AMQAaAA3gB6AD8AIQAQ0AiACJAEsAMMAZQAzQBogDnAHwAP0AhABFgCOgElALqAYoA14BuADiAHUAP6Af8BEwCLwEiAJWATEApEBXYC2AF5gMEAYsAywBqoDWQHjgP3Af2BH8CbwE5gJ3AUVAAA.YAAACHwAAAAA,limitTargetedAds:1,playerOrigin:playerpage,gdprConsentOptOut:0,publisherProvidedId:1fa77255-18f8-4d76-b9e9-4cdcfa37219d}}}",
    "compressed": true
}

then copy to clipboard and paste into script (on request json string: )
but again i get: NameError: name 'create_manifest' is not defined

[larley]

The RegEx looks like that in order to avoid false positives.

NameError: name 'create_manifest
That's not even possible, share the full script.

[whs912km]

here my script. i've copied your script from post 122 and append main section from post 129
i think mistake isn't from script but from json data

from urllib.parse import urlparse, parse_qs, urljoin

import requests
import json

# Metadata URL RegEx: https://gizmo\.rakuten\.tv/.*/avod/streamings.*

if __name__ == '__main__':
    url = input("gizmo url: ")
    json_string = input("json string: ")
    print(create_manifest(url, json_string))
    print(create_license(url, json_string))

def filter_dict(
        dictionary: dict,
        keys: list
) -> dict:
    return dict(
        (k, v)
        for k, v in dictionary.items()
        if k in keys
    )


def create_metadata(
        metadata: str,
        metadata_body
) -> tuple[str, dict, dict] | None:
    try:
        parsed_body = json.loads(metadata_body)
    except Exception:
        return

    parsed_url = urlparse(metadata)
    query = parse_qs(parsed_url.query)

    url = urljoin(metadata, parsed_url.path)

    params = {
        "device_identifier": "android",
        **filter_dict(
            query,
            [
                'classification_id'
            ]
        )
    }

    json_body = {
        "player": "android:DASH-CENC",
        "device_stream_video_quality": "HD",
        **filter_dict(
            parsed_body,
            [
                "hdr_type",
                "audio_quality",
                "content_id",
                "audio_language",
                "video_type",
                "device_serial",
                "content_type",
                "subtitle_language",
            ]
        )
    }

    return url, params, json_body


def create_manifest(metadata: str, metadata_body):
    if new_metadata := create_metadata(metadata, metadata_body):
        url, params, json_body = new_metadata
    else:
        return

    metadata_request = requests.post(
        url=url,
        params=params,
        json=json_body
    )
    if metadata_request.status_code != 200:
        return

    return metadata_request.json().get('data', {}).get('stream_infos', [{}])[0].get('url')


def create_license(metadata: str, metadata_body):
    if new_metadata := create_metadata(metadata, metadata_body):
        url, params, json_body = new_metadata
    else:
        return

    metadata_request = requests.post(
        url=url,
        params=params,
        json=json_body
    )
    if metadata_request.status_code != 200:
        return

    return metadata_request.json().get('data', {}).get('stream_infos', [{}])[0].get('license_url')

[larley]

You didn't read my post correctly. The "if __name__ == '__main__':" has to be at the end of the script

[whs912km]

i've put 'main' at the end of the script but now run script and closes immediately ... please share here full working script. thanks

from urllib.parse import urlparse, parse_qs, urljoin

import requests
import json

# Metadata URL RegEx: https://gizmo\.rakuten\.tv/.*/avod/streamings.*

def filter_dict(
        dictionary: dict,
        keys: list
) -> dict:
    return dict(
        (k, v)
        for k, v in dictionary.items()
        if k in keys
    )


def create_metadata(
        metadata: str,
        metadata_body
) -> tuple[str, dict, dict] | None:
    try:
        parsed_body = json.loads(metadata_body)
    except Exception:
        return

    parsed_url = urlparse(metadata)
    query = parse_qs(parsed_url.query)

    url = urljoin(metadata, parsed_url.path)

    params = {
        "device_identifier": "android",
        **filter_dict(
            query,
            [
                'classification_id'
            ]
        )
    }

    json_body = {
        "player": "android:DASH-CENC",
        "device_stream_video_quality": "HD",
        **filter_dict(
            parsed_body,
            [
                "hdr_type",
                "audio_quality",
                "content_id",
                "audio_language",
                "video_type",
                "device_serial",
                "content_type",
                "subtitle_language",
            ]
        )
    }

    return url, params, json_body


def create_manifest(metadata: str, metadata_body):
    if new_metadata := create_metadata(metadata, metadata_body):
        url, params, json_body = new_metadata
    else:
        return

    metadata_request = requests.post(
        url=url,
        params=params,
        json=json_body
    )
    if metadata_request.status_code != 200:
        return

    return metadata_request.json().get('data', {}).get('stream_infos', [{}])[0].get('url')


def create_license(metadata: str, metadata_body):
    if new_metadata := create_metadata(metadata, metadata_body):
        url, params, json_body = new_metadata
    else:
        return

    metadata_request = requests.post(
        url=url,
        params=params,
        json=json_body
    )
    if metadata_request.status_code != 200:
        return

    return metadata_request.json().get('data', {}).get('stream_infos', [{}])[0].get('license_url')
        
    if __name__ == '__main__':
        url = input("gizmo url: ")
        json_string = input("json string: ")
        print(create_manifest(url, json_string))
        print(create_license(url, json_string))

[larley]

Indent it so it touches the left side. Right now, it inside the create_license method

[whs912km]

done. now from script i get:
None
None

here edited script

from urllib.parse import urlparse, parse_qs, urljoin

import requests
import json

# Metadata URL RegEx: https://gizmo\.rakuten\.tv/.*/avod/streamings.*

def filter_dict(
        dictionary: dict,
        keys: list
) -> dict:
    return dict(
        (k, v)
        for k, v in dictionary.items()
        if k in keys
    )


def create_metadata(
        metadata: str,
        metadata_body
) -> tuple[str, dict, dict] | None:
    try:
        parsed_body = json.loads(metadata_body)
    except Exception:
        return

    parsed_url = urlparse(metadata)
    query = parse_qs(parsed_url.query)

    url = urljoin(metadata, parsed_url.path)

    params = {
        "device_identifier": "android",
        **filter_dict(
            query,
            [
                'classification_id'
            ]
        )
    }

    json_body = {
        "player": "android:DASH-CENC",
        "device_stream_video_quality": "HD",
        **filter_dict(
            parsed_body,
            [
                "hdr_type",
                "audio_quality",
                "content_id",
                "audio_language",
                "video_type",
                "device_serial",
                "content_type",
                "subtitle_language",
            ]
        )
    }

    return url, params, json_body


def create_manifest(metadata: str, metadata_body):
    if new_metadata := create_metadata(metadata, metadata_body):
        url, params, json_body = new_metadata
    else:
        return

    metadata_request = requests.post(
        url=url,
        params=params,
        json=json_body
    )
    if metadata_request.status_code != 200:
        return

    return metadata_request.json().get('data', {}).get('stream_infos', [{}])[0].get('url')


def create_license(metadata: str, metadata_body):
    if new_metadata := create_metadata(metadata, metadata_body):
        url, params, json_body = new_metadata
    else:
        return

    metadata_request = requests.post(
        url=url,
        params=params,
        json=json_body
    )
    if metadata_request.status_code != 200:
        return

    return metadata_request.json().get('data', {}).get('stream_infos', [{}])[0].get('license_url')
        
if __name__ == '__main__':
    url = input("gizmo url: ")
    json_string = input("json string: ")
    print(create_manifest(url, json_string))
    print(create_license(url, json_string))

here error


[Attachment 81632 - Click to enlarge]

and here copy gizmo url as curlbash and convert on JSON

{
    "url": "https://gizmo.rakuten.tv/v3/avod/streamings",
    "raw_url": "https://gizmo.rakuten.tv/v3/avod/streamings?classification_id=36&device_identifier=web&device_stream_audio_quality=2.0&device_stream_hdr_type=NONE&device_stream_video_quality=FHD&disable_dash_legacy_packages=false&locale=it&market_code=it",
    "method": "post",
    "headers": {
        "authority": "gizmo.rakuten.tv",
        "accept": "application/json, text/plain, */*",
        "accept-language": "en,it-IT;q=0.9,it;q=0.8,en-US;q=0.7,ru;q=0.6,pt;q=0.5,es;q=0.4,nl;q=0.3,fr;q=0.2,ha;q=0.1,bg;q=0.1,am;q=0.1,hu;q=0.1,pl;q=0.1,zh-TW;q=0.1,zh;q=0.1,vi;q=0.1",
        "content-type": "application/json",
        "dnt": "1",
        "origin": "https://www.rakuten.tv",
        "priority": "u=1, i",
        "referer": "https://www.rakuten.tv/",
        "sec-ch-ua": "\"Not(A:Brand\";v=\"24\", \"Chromium\";v=\"122\"",
        "sec-ch-ua-mobile": "?0",
        "sec-ch-ua-platform": "\"Windows\"",
        "sec-fetch-dest": "empty",
        "sec-fetch-mode": "cors",
        "sec-fetch-site": "same-site",
        "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
    },
    "queries": {
        "classification_id": "36",
        "device_identifier": "web",
        "device_stream_audio_quality": "2.0",
        "device_stream_hdr_type": "NONE",
        "device_stream_video_quality": "FHD",
        "disable_dash_legacy_packages": "false",
        "locale": "it",
        "market_code": "it"
    },
    "data": {
        "audio_language": "ITA",
        "audio_quality": "2.0",
        "classification_id": "36",
        "content_id": "amores-perros",
        "content_type": "movies",
        "device_make": "chrome",
        "device_model": "GENERIC",
        "device_serial": "not implemented",
        "device_stream_video_quality": "HD",
        "device_uid": "e0eaeb4f-15b3-4aba-b533-75f7681d6d29",
        "device_year": 1970,
        "gam_correlator": 420359007407631,
        "gdpr_consent_opt_out": "0",
        "gdpr_consent": "CQDiyXAQDiyXAAcABBENBCFgAAAAAAAAACiQAAAA4gAgA4AA.YAAAAAAAAAAA",
        "hdr_type": "NONE",
        "ifa_subscriber_id": null,
        "ifa_type": "ppid",
        "slinf": "95",
        "player_height": 1080,
        "player_width": 1920,
        "player": "web:DASH-CENC:WVM",
        "publisher_provided_id": "677c6164-d200-4be2-93ac-ef9fbbae58fd",
        "strict_video_quality": false,
        "subtitle_formats": [
            "vtt"
        ],
        "subtitle_language": "MIS",
        "support_closed_captions": true,
        "video_type": "stream",
        "support_thumbnails": true,
        "app_bundle": "com.rakutentv.web",
        "app_name": "RakutenTV",
        "url": "rakuten.tv",
        "requestedContent": {
            "live": false,
            "duration": 9248,
            "genres": [
                "Drammatico",
                "Film indipendente",
                "Thriller"
            ],
            "id": 125366,
            "rating": "14",
            "title": "Amores Perros",
            "adOptions": {
                "channelID": 3273,
                "gdprAccepted": false,
                "gdprConsentString": "CQDiyXAQDiyXAAcABBENBCFgAAAAAAAAACiQAAAA4gAgA4AA.YAAAAAAAAAAA",
                "limitTargetedAds": 1,
                "playerOrigin": "playerpage",
                "gdprConsentOptOut": "0",
                "publisherProvidedId": "677c6164-d200-4be2-93ac-ef9fbbae58fd"
            }
        }
    }
}

[larley]

Paste the json as a string (found at the bottom of curlconverter in the comment)

[Attachment 81633 - Click to enlarge]

[whs912km]

same error


[Attachment 81634 - Click to enlarge]

[larley]

copy the curly braces too {}

[whs912km]

oh yesss finally the script work fine! thank you

however i get mpd and lic url. and the key ??


[Attachment 81647 - Click to enlarge]

[larley]

Write a script to get it, I can't do everything for you

[whs912km]

but sorry, all this mess with this script to get mpd only ?

i can find easy on network tab ... i thought your script was get the key, the most thing

[lukhitt1992]

[Attachment 81650 - Click to enlarge]

Got this error from nowtv.com !!!
little help sir

[larley]

but sorry, all this mess with this script to get mpd only ?

i can find easy on network tab ... i thought your script was get the key, the most thing

You've got the License URL and Manifest URL. That's all that is required to get the keys.
You can't use the License URL from the network tab, that is only used for higher resolutions.

[larley]

Got this error from nowtv.com !!!
little help sir

Send the fetch string in a .txt file

[lukhitt1992]

Got this error from nowtv.com !!!
little help sir

Send the fetch string in a .txt file

hello , thanks for reply. i sent fetch string via DM

[whs912km]

You've got the License URL and Manifest URL. That's all that is required to get the keys.
You can't use the License URL from the network tab, that is only used for higher resolutions

maybe for you, but for many users here like me, this goal it's very hard ...
so, if you implement the script to get the key, amazing, otherwise rakuten will remain unsolved for us