yt-dlp - download segmented video?

[ampersand]

After fiddling around with http toolkit for some time I believe I've found a simpler way to get the mpd url without ads - find the mpd url WITH ads and replace http with https. Or find the widevine mpd url with "https:" while looping through sources in the json that the episode info URL returns after a request. It worked with Insoupconnable E01.

This is the part of the script that deals with that:

import requests
import json
import helpers
from urllib.parse import unquote

episode_url = "https://player.stv.tv/episode/4aa8/banijay1-insouponnable"
stv_episode_info_url = "https://edge.api.brightcove.com/playback/v1/accounts/6204867266001/videos/EPISODE_ID?ad_config_id=7fb30a85-56ee-49f8-b645-900876e7c09c&config_id=edbc4bd0-6a34-46f2-8055-45d823d66886"
stv_episode_info_headers = {
    "accept": "application/json;pk=BCpkADawqM1fQNUrQOvg-vTo4VGDTJ_lGjxp2zBSPcXJntYd5csQkjm7hBKviIVgfFoEJLW4_JPPsHUwXNEjZspbr3d1HqGDw2gUqGCBZ_9Y_BF7HJsh2n6PQcpL9b2kdbi103oXvmTNZWiQ"}


def find_str(str, char):
    index = 0
    if char in str:
        c = char[0]
        for ch in str:
            if ch == c and str[index:index + len(char)] == char:
                return index
            index += 1
    return -1


def extract_json_from_html(html_string):
    json_begin = 'type="application/json">'  # +24 chars
    json_end = '}</script>'  # +1 char

    json_begin_pos = find_str(html_string, json_begin)
    temp = html_string[json_begin_pos+24:]
    json_end_pos = find_str(temp, json_end)
    json_string = temp[:json_end_pos+1]
    json_parsed = json.loads(json_string)

    return json_parsed


def get_links(episode_url):
    res = requests.get(episode_url)
    json_res = helpers.extract_json_from_html(res.text)

    title = json_res['props']['pageProps']['title'].split(" | ")[1]
    temp_id = json_res['props']['pageProps']['episodeId']
    helper_prop = f'/episodes/{temp_id}'

    episode_id = json_res['props']['initialReduxState']['playerApiCache'][
        helper_prop]['results']['video']['id']

    link_info = stv_episode_info_url.replace(
        "EPISODE_ID", f'{episode_id}')
    res_info = requests.get(link_info, headers=stv_episode_info_headers)
    info_json = res_info.json()

    sources = info_json['sources']
    for source in sources:
        dict_keys = dict.keys(source['key_systems'])
        for key in dict_keys:
            if "widevine" in key:
                widevine_source = source
                break

    mpd_url = unquote(widevine_source['src'])
    mpd_url = mpd_url.split("&")[0]
    mpd_url = mpd_url.replace("http:", "https:")
    license_url = unquote(
        widevine_source['key_systems']['com.widevine.alpha']
        ['license_url'])

    return title, mpd_url, license_url

[A_n_g_e_l_a]

After fiddling around with http toolkit for some time I believe I've found a simpler way to get the mpd url without ads - find the mpd url WITH ads and replace http with https. Or find the widevine mpd url with "https:" while looping through sources in the json that the episode info URL returns after a request. It worked with Insoupconnable E01.

You've got to dig quite deep to get the 'stv_episode_info_url' and I assume also the 'stv_episode_info_header' before you can download. Whereas a vmap is far easier to grab, single video or group of them. So for Soupconnable go the series link, with The Stream Detector in your browser visit each page in the series so The Stream Detector captures the vmap. When finished click on 'copy all visible urls' in The Stream Detector and run stv.py. No need for HTTPTollkit out and intercepting each page.

Updated stv.py with sanity checks. Does single or series videos - whatever vmap is in clipboard - ignores other stuff there...

#!/usr/bin/env python3
 
import  pyperclip as PC
import xml.etree.ElementTree as ET
import requests
import os
from pywidevine.L3.cdm import deviceconfig
from base64 import b64encode
from pywidevine.L3.decrypt.wvdecryptcustom import WvDecrypt
import shutil

'''
For stv.tv  - a series downloader

This program loads a short form of content.vmap which, during testing, provided an advert free stream from 'content_uri'
Full headers. with origin and referrer defined, produces a full vmap - the 'content_uri' from which which serves adverts.
Needs N_m3u8DL-RE and shaka-packager in $PATH

The program reads the clipboard of urls saved from 'The Stream Detector' browser plugin.
'pip install pyperclip shutil' may be needed before you start

Use 'The Stream Detector' browser plugin (Chrome + Firefox)
Find here: Firefox version - https://addons.mozilla.org/en-US/firefox/addon/hls-stream-detector/
Chrome version: https://github.com/rowrawer/stream-detector/releases  choose  hls_stream_detector-2.1X.XX.crx 
and in chrome://extensions/  set Developer mode (top-right window) and drag and drop the crx file into the window to install

Once installed configure with an additional filter set for 'vmap'.

Under Options:
'Detect additional file extensions:' 	set checkbox ticked and enter 'vmap' in box
'Detect additional Content-Type headers:' tick checkbox and add 'application/xml' in box

Now just visit any number of video pages at STV and the vmap urls will be listed by
The Stream Detector. Copy the urls to the clipboard from 'The Stream Detector' window when finished selecting videos.
This program ignores all the thumbnail (VTT) links - just leave them be.
'''

headers = {
    'Accept': '*/*',
    'Accept-Language': 'en-GB,en;q=0.7',
    'Connection': 'keep-alive',
}

def WV_Function(pssh, lic_url, cert_b64=None):
    wvdecrypt = WvDecrypt(init_data_b64=pssh, cert_data_b64=cert_b64, device=deviceconfig.device_android_generic)                   
    widevine_license = requests.post(url=lic_url, data=wvdecrypt.get_challenge(), headers=None)
    license_b64 = b64encode(widevine_license.content)
    wvdecrypt.update_license(license_b64)
    Correct, keyswvdecrypt = wvdecrypt.start_process()
    if Correct:
        return keyswvdecrypt  

def divides(text):
    text = text
    l = len(text)
    count = int(shutil.get_terminal_size().columns) - 2
    if count <= 120:
        count = int(count)
    else:
        count = int(count/2)
    count = count - (l)   
    line = ('-' * int(count/2))
    #line = (chr(9604) * int(count/2))
    print('\n'+ line + " " + text + " " + line)


divides('STV Downloader from a vmap url')
print("Ready to read from clipboard")
input("Press Enter when ready... ")
print('reading from the clipboard ...')

mylines = PC.paste().split('\n')
for line in mylines:
    vmap_url = line
    if 'thumbnail' in vmap_url:
        continue
    if 'mpd' in vmap_url:
        print('mpd found .. rejecting')
        continue
    if 'vmap' in vmap_url:
        videomap = requests.get(vmap_url, headers = headers).text
    else:
        print(f'No vmap found in link(s).')
        exit(0)
        
    # parse xml
    root = ET.fromstring(videomap)
    # get uri to m3u8 contained in vmap
    uri_dict =  root[0][0].attrib
    content_uri = uri_dict["contenturi"] 
    #videoname = input('Name of the video? ')
    videoname = vmap_url.rsplit('&', 5)
    videoname = videoname[1].split('=')
    videoname = videoname[1].title()
    divides(videoname)
    if 'mpd' in content_uri:
        # encrypted stream
        divides('encrypted video')
        mpdmap = requests.get(content_uri, headers=headers).text
        root = ET.fromstring(mpdmap)
        mpd_dict = root[0][0][1].attrib
        lic_url = mpd_dict['{urn:brightcove:2015}licenseAcquisitionUrl']
        divides('license url')
        print(lic_url)
        pssh = root[0][0][1][0].text
        divides('pssh')
        print(pssh)
        keys =  WV_Function(pssh, lic_url)
        key = keys[0]
        divides('keys found')
        print(key)
        divides('downloading video')
        os.system(f"N_m3u8DL-RE '{content_uri}' --auto-select --save-name '{videoname}' --tmp-dir './cache/' --save-dir './' --use-shaka-packager --key '{key}'  -M 'format=mkv:muxer=mkvmerge'")
        
    else:  
        divides('downloading an encryption free video')
        os.system(f"N_m3u8DL-RE '{content_uri}' --auto-select  --save-name '{videoname}' --tmp-dir './cache/' --save-dir './'  --use-shaka-packager   -M 'format=mkv:muxer=mkvmerge'")

This is for linux. Windows will need a few .exe's added. Runs with your own CDM if the program is encrypted thus needs to be in WKS-KEYS folder. Will work for non encrypted STV stuff for those without their own CDM

[codehound]

This is for linux. Windows will need a few .exe's added. Runs with your own CDM if the program is encrypted thus needs to be in WKS-KEYS folder. Will work for non encrypted STV stuff for those without their own CDM

Brilliant