Unshackle - Modular Movie, TV, and Music Archival Software

[vidsrme]

I tried downloading motd from iplayer with unshackle and it failed due to my vpn but I can download the same episode using yt-dlp with the same vpn active.

Both VT and UK-FTA get it.

Yeah getting it isnt a problem, the point I was making is unshackle cant get it

[phased]

I tried downloading motd from iplayer with unshackle and it failed due to my vpn but I can download the same episode using yt-dlp with the same vpn active.

Both VT and UK-FTA get it.

Yeah getting it isnt a problem, the point I was making is unshackle cant get it

iP is a service written by SBB, and he has said he likely won't update stuff much more. So your kind offer of a load of work, for someone else, probably needs a new target.
But since you can get mot, there's no real problem.

[vidsrme]

iP is a service written by SBB, and he has said he likely won't update stuff much more. So your kind offer of a load of work, for someone else, probably needs a new target.

I wasn't offering someone work I was giving feedback on the software, which is what the threads for and many other people do so your unpleasant attitude definitely needs a new target.

[stabbedbybrick]

I tried downloading motd from iplayer with unshackle and it failed due to my vpn but I can download the same episode using yt-dlp with the same vpn active.

yt-dlp encounters some '403: forbidden' errors but it's able to workaround them

Maybe the method from yt-dlp could be copied to unshackle?

I don't think there's a good way to do that since yt-dlp approaches things differently. It's using the "PC" endpoint, which is less strict with VPN/proxies and doesn't require any modification to the URLs, so it can just iterate through all the manifests from all hosts and simply skip the ones that fail. The downside is that this endpoint maxes out at 720p.

The Devine/Unshackle service is written to get the best available streams (up to 2160p), which comes at a cost of them being more restricted overall. And you really don't want to make a script worse by falling back to a lower quality when the intention is to get the best - especially when the limiting factor is the user's own setup.

[phased]

iP is a service written by SBB, and he has said he likely won't update stuff much more. So your kind offer of a load of work, for someone else, probably needs a new target.

I wasn't offering someone work I was giving feedback on the software, which is what the threads for and many other people do so your unpleasant attitude definitely needs a new target.

The issue was likely a BBC fault. Envied downloaded the file today but couldn't yesterday.

[Tom Saurus]

I tried downloading motd from iplayer with unshackle and it failed due to my vpn but I can download the same episode using yt-dlp with the same vpn active.

yt-dlp encounters some '403: forbidden' errors but it's able to workaround them

Maybe the method from yt-dlp could be copied to unshackle?

I don't think there's a good way to do that since yt-dlp approaches things differently. It's using the "PC" endpoint, which is less strict with VPN/proxies and doesn't require any modification to the URLs, so it can just iterate through all the manifests from all hosts and simply skip the ones that fail. The downside is that this endpoint maxes out at 720p.

The Devine/Unshackle service is written to get the best available streams (up to 2160p), which comes at a cost of them being more restricted overall. And you really don't want to make a script worse by falling back to a lower quality when the intention is to get the best - especially when the limiting factor is the user's own setup.

Thank you for Freevine and the services. Thank you for the information about PC Endpoint being less strict for VPN. I am usually happy with 720p and less. Maybe someone one day will make a iP service for the PC endpoint and say this is for 720p and less that will be more successful for those who have VPN issues.

[bharry29]

Can someone provide me a working service which I can use to build my own? Stuggling hard to understand how to.

[phased]

Can someone provide me a working service which I can use to build my own? Stuggling hard to understand how to.

Take a look at the code for the downloader. devine/core/service.py lists the abstract methods that your service code MUST implement. All the instructions are in the source code. Following an existing service wont be too much help as each implementation is different.

A forum search will reveal services I have posted here for devine and unshackle if that's what you want.

Your service class. say, class BigTv(Service): needs to inherit the Service class in devine/unshackle and implement all the abstract methods and over-ride some/all of the others.


service.py code for devine - unshackle will be a close match.

import base64
import logging
from abc import ABCMeta, abstractmethod
from collections.abc import Generator
from http.cookiejar import CookieJar
from pathlib import Path
from typing import Optional, Union
from urllib.parse import urlparse

import click
import m3u8
import requests
from requests.adapters import HTTPAdapter, Retry
from rich.padding import Padding
from rich.rule import Rule

from devine.core.cacher import Cacher
from devine.core.config import config
from devine.core.console import console
from devine.core.constants import AnyTrack
from devine.core.credential import Credential
from devine.core.drm import DRM_T
from devine.core.search_result import SearchResult
from devine.core.titles import Title_T, Titles_T
from devine.core.tracks import Chapters, Tracks
from devine.core.utilities import get_ip_info


class Service(metaclass=ABCMeta):
    """The Service Base Class."""

    # Abstract class variables
    ALIASES: tuple[str, ...] = ()  # list of aliases for the service; alternatives to the service tag.
    GEOFENCE: tuple[str, ...] = ()  # list of ip regions required to use the service. empty list == no specific region.

    def __init__(self, ctx: click.Context):
        console.print(Padding(
            Rule(f"[rule.text]Service: {self.__class__.__name__}"),
            (1, 2)
        ))

        self.config = ctx.obj.config

        self.log = logging.getLogger(self.__class__.__name__)

        self.session = self.get_session()
        self.cache = Cacher(self.__class__.__name__)

        if not ctx.parent or not ctx.parent.params.get("no_proxy"):
            if ctx.parent:
                proxy = ctx.parent.params["proxy"]
            else:
                proxy = None

            if not proxy:
                # don't override the explicit proxy set by the user, even if they may be geoblocked
                with console.status("Checking if current region is Geoblocked...", spinner="dots"):
                    if self.GEOFENCE:
                        # no explicit proxy, let's get one to GEOFENCE if needed
                        current_region = get_ip_info(self.session)["country"].lower()
                        if any(x.lower() == current_region for x in self.GEOFENCE):
                            self.log.info("Service is not Geoblocked in your region")
                        else:
                            requested_proxy = self.GEOFENCE[0]  # first is likely main region
                            self.log.info(f"Service is Geoblocked in your region, getting a Proxy to {requested_proxy}")
                            for proxy_provider in ctx.obj.proxy_providers:
                                proxy = proxy_provider.get_proxy(requested_proxy)
                                if proxy:
                                    self.log.info(f"Got Proxy from {proxy_provider.__class__.__name__}")
                                    break
                    else:
                        self.log.info("Service has no Geofence")

            if proxy:
                self.session.proxies.update({"all": proxy})
                proxy_parse = urlparse(proxy)
                if proxy_parse.username and proxy_parse.password:
                    self.session.headers.update({
                        "Proxy-Authorization": base64.b64encode(
                            f"{proxy_parse.username}:{proxy_parse.password}".encode("utf8")
                        ).decode()
                    })

    # Optional Abstract functions
    # The following functions may be implemented by the Service.
    # Otherwise, the base service code (if any) of the function will be executed on call.
    # The functions will be executed in shown order.

    @staticmethod
    def get_session() -> requests.Session:
        """
        Creates a Python-requests Session, adds common headers
        from config, cookies, retry handler, and a proxy if available.
        :returns: Prepared Python-requests Session
        """
        session = requests.Session()
        session.headers.update(config.headers)
        session.mount("https://", HTTPAdapter(
            max_retries=Retry(
                total=15,
                backoff_factor=0.2,
                status_forcelist=[429, 500, 502, 503, 504]
            ),
            pool_block=True
        ))
        session.mount("http://", session.adapters["https://"])
        return session

    def authenticate(self, cookies: Optional[CookieJar] = None, credential: Optional[Credential] = None) -> None:
        """
        Authenticate the Service with Cookies and/or Credentials (Email/Username and Password).

        This is effectively a login() function. Any API calls or object initializations
        needing to be made, should be made here. This will be run before any of the
        following abstract functions.

        You should avoid storing or using the Credential outside this function.
        Make any calls you need for any Cookies, Tokens, or such, then use those.

        The Cookie jar should also not be stored outside this function. However, you may load
        the Cookie jar into the service session.
        """
        if cookies is not None:
            if not isinstance(cookies, CookieJar):
                raise TypeError(f"Expected cookies to be a {CookieJar}, not {cookies!r}.")
            self.session.cookies.update(cookies)

    def search(self) -> Generator[SearchResult, None, None]:
        """
        Search by query for titles from the Service.

        The query must be taken as a CLI argument by the Service class.
        Ideally just re-use the title ID argument (i.e. self.title).

        Search results will be displayed in the order yielded.
        """
        raise NotImplementedError(f"Search functionality has not been implemented by {self.__class__.__name__}")

    def get_widevine_service_certificate(self, *, challenge: bytes, title: Title_T, track: AnyTrack) \
            -> Union[bytes, str]:
        """
        Get the Widevine Service Certificate used for Privacy Mode.

        :param challenge: The service challenge, providing this to a License endpoint should return the
            privacy certificate that the service uses.
        :param title: The current `Title` from get_titles that is being executed. This is provided in
            case it has data needed to be used, e.g. for a HTTP request.
        :param track: The current `Track` needing decryption. Provided for same reason as `title`.
        :return: The Service Privacy Certificate as Bytes or a Base64 string. Don't Base64 Encode or
            Decode the data, return as is to reduce unnecessary computations.
        """

    def get_widevine_license(self, *, challenge: bytes, title: Title_T, track: AnyTrack) -> Optional[Union[bytes, str]]:
        """
        Get a Widevine License message by sending a License Request (challenge).

        This License message contains the encrypted Content Decryption Keys and will be
        read by the Cdm and decrypted.

        This is a very important request to get correct. A bad, unexpected, or missing
        value in the request can cause your key to be detected and promptly banned,
        revoked, disabled, or downgraded.

        :param challenge: The license challenge from the Widevine CDM.
        :param title: The current `Title` from get_titles that is being executed. This is provided in
            case it has data needed to be used, e.g. for a HTTP request.
        :param track: The current `Track` needing decryption. Provided for same reason as `title`.
        :return: The License response as Bytes or a Base64 string. Don't Base64 Encode or
            Decode the data, return as is to reduce unnecessary computations.
        """

    # Required Abstract functions
    # The following functions *must* be implemented by the Service.
    # The functions will be executed in shown order.

    @abstractmethod
    def get_titles(self) -> Titles_T:
        """
        Get Titles for the provided title ID.

        Return a Movies, Series, or Album objects containing Movie, Episode, or Song title objects respectively.
        The returned data must be for the given title ID, or a spawn of the title ID.

        At least one object is expected to be returned, or it will presume an invalid Title ID was
        provided.

        You can use the `data` dictionary class instance attribute of each Title to store data you may need later on.
        This can be useful to store information on each title that will be required like any sub-asset IDs, or such.
        """

    @abstractmethod
    def get_tracks(self, title: Title_T) -> Tracks:
        """
        Get Track objects of the Title.

        Return a Tracks object, which itself can contain Video, Audio, Subtitle or even Chapters.
        Tracks.videos, Tracks.audio, Tracks.subtitles, and Track.chapters should be a List of Track objects.

        Each Track in the Tracks should represent a Video/Audio Stream/Representation/Adaptation or
        a Subtitle file.

        While one Track should only hold information for one stream/downloadable, try to get as many
        unique Track objects per stream type so Stream selection by the root code can give you more
        options in terms of Resolution, Bitrate, Codecs, Language, e.t.c.

        No decision making or filtering of which Tracks get returned should happen here. It can be
        considered an error to filter for e.g. resolution, codec, and such. All filtering based on
        arguments will be done by the root code automatically when needed.

        Make sure you correctly mark which Tracks are encrypted or not, and by which DRM System
        via its `drm` property.

        If you are able to obtain the Track's KID (Key ID) as a 32 char (16 bit) HEX string, provide
        it to the Track's `kid` variable as it will speed up the decryption process later on. It may
        or may not be needed, that depends on the service. Generally if you can provide it, without
        downloading any of the Track's stream data, then do.

        :param title: The current `Title` from get_titles that is being executed.
        :return: Tracks object containing Video, Audio, Subtitles, and Chapters, if available.
        """

    @abstractmethod
    def get_chapters(self, title: Title_T) -> Chapters:
        """
        Get Chapters for the Title.

        Parameters:
            title: The current Title from `get_titles` that is being processed.

        You must return a Chapters object containing 0 or more Chapter objects.

        You do not need to set a Chapter number or sort/order the chapters in any way as
        the Chapters class automatically handles all of that for you. If there's no
        descriptive name for a Chapter then do not set a name at all.

        You must not set Chapter names to "Chapter {n}" or such. If you (or the user)
        wants "Chapter {n}" style Chapter names (or similar) then they can use the config
        option `chapter_fallback_name`. For example, `"Chapter {i:02}"` for "Chapter 01".
        """

    # Optional Event methods

    def on_segment_downloaded(self, track: AnyTrack, segment: Path) -> None:
        """
        Called when one of a Track's Segments has finished downloading.

        Parameters:
            track: The Track object that had a Segment downloaded.
            segment: The Path to the Segment that was downloaded.
        """

    def on_track_downloaded(self, track: AnyTrack) -> None:
        """
        Called when a Track has finished downloading.

        Parameters:
            track: The Track object that was downloaded.
        """

    def on_track_decrypted(self, track: AnyTrack, drm: DRM_T, segment: Optional[m3u8.Segment] = None) -> None:
        """
        Called when a Track has finished decrypting.

        Parameters:
            track: The Track object that was decrypted.
            drm: The DRM object it decrypted with.
            segment: The HLS segment information that was decrypted.
        """

    def on_track_repacked(self, track: AnyTrack) -> None:
        """
        Called when a Track has finished repacking.

        Parameters:
            track: The Track object that was repacked.
        """

    def on_track_multiplex(self, track: AnyTrack) -> None:
        """
        Called when a Track is about to be Multiplexed into a Container.

        Note: Right now only MKV containers are multiplexed but in the future
        this may also be called when multiplexing to other containers like
        MP4 via ffmpeg/mp4box.

        Parameters:
            track: The Track object that was repacked.
        """

__all__ = ("Service",)

[bharry29]

Thanks I will take a look at it

[phased]

Thanks I will take a look at it

And the divine gihub points to a detailed explanation about creating classes.

[bamboobali]

Is anyone able to get Blue Planet II, episodes 3, 4 and 5 in UHD? The other episodes are fine, but these ones don't show up for me in UHD using the iP service.

I get a 404 error on a URL generated by the iP service, so maybe these episodes are an anomaly (if so I don't know what the right URL should be).

These episodes 'look' like they are in UHD if I just stream them directly on the TV.

[iamghost]

Is anyone able to get Blue Planet II, episodes 3, 4 and 5 in UHD? The other episodes are fine, but these ones don't show up for me in UHD using the iP service.

I get a 404 error on a URL generated by the iP service, so maybe these episodes are an anomaly (if so I don't know what the right URL should be).

These episodes 'look' like they are in UHD if I just stream them directly on the TV.

uv run unshackle dl -w s01e3 -r hlg ip p04tjbtx

[Attachment 88514 - Click to enlarge]

[magicbacon]

I have a service working, now I would like to download two audio language tracks to be muxed into the final MKV. Has anyone successfully gotten multi-lang downloads working? Are there any examples I can look at? I've looked at the source code for the "dl" function and the flag for language looks like it only supports one at a time. My goal is to download video, English audio, Japanese audio, and English subtitles into one MKV file. Thanks!

[bamboobali]

@iamghost Thanks - I'll try again with that exact command

[mickmars]

Quick question, hoping someone has the right answer

I started downloading somehitg from ITV that has 13 episodes, I got the first four episodes one by one,I know how to get a whole series but how would I go about batch downloading episodes 5-13 in one go?


Cheers

[Tom Saurus]

Quick question, hoping someone has the right answer

I started downloading somehitg from ITV that has 13 episodes, I got the first four episodes one by one,I know how to get a whole series but how would I go about batch downloading episodes 5-13 in one go?


Cheers

You would use the -w command

uv run unshackle dl -w S01E05-S01E13 ITV url

[Tom Saurus]

I need some guidance about UKTV. It is difficult as there are three audio qualities available and they are fluctuating a bit with lets say I choose "-ab 122" the next file might need "-ab 121" so it is stopping one from getting an entire season one episode after another. I don't want it downloading all three audio quality selections and it might be a case where it suppose to download the best quality audio and it doesn't it might try to get all three. The video quality is fluctuating episode to episode as well as sometimes it is 540 and next it might 576. I am going to test with 720p and see if that is constant. Thanks in advance for any advice. I thought I would try to get caught up on The Chelsea Detective.

I was able to get it downloaded the first season anyway at 720 and it selected the best audio itself which is good so it turned out.

[mickmars]

Quick question, hoping someone has the right answer

I started downloading somehitg from ITV that has 13 episodes, I got the first four episodes one by one,I know how to get a whole series but how would I go about batch downloading episodes 5-13 in one go?


Cheers

You would use the -w command

uv run unshackle dl -w S01E05-S01E13 ITV url

Brilliant, Cheers Mr Tom

[stabbedbybrick]

I don't know how many of you are still using my services, but I've set up a new repository on Github that will work interchangeably with Devine and Unshackle.

For those who've cloned the earlier repo from CDM-project, you can simply update the origin URL and pull the new updates:

# update URL
git remote set-url origin https://github.com/stabbedbybrick/services.git

# pull updates
git pull

I don't know how active I'll be, but I'll try to fix broken services and possibly add new ones. But to avoid headaches and drama, I'll continue to keep it focused on non-premium services for now.

[A_n_g_e_l_a]

Updated envied and TwinVine to reflect the recent update to unshackle 1.4.4. Both come with services and each offers options not available in unshackle. TwinVine is a two package install of VineFeeder and Envied; a simple uv installs both packages and it is ready to run.

Each install provides 28 services (not all tested)

Remember to copy your existing envied.yaml across if you do a new clone. Git pull should work ok.
In envied it is at envied/unshackle/unashackle.yaml
in TwinVine it is at packages/envied/src/envied/envied.yaml. For a new install copy envied-working-example to envied.yaml.

https://github.com/vinefeeder/envied/tree/main
https://github.com/vinefeeder/TwinVine/tree/main

[Tom Saurus]

Quick question, hoping someone has the right answer

I started downloading somehitg from ITV that has 13 episodes, I got the first four episodes one by one,I know how to get a whole series but how would I go about batch downloading episodes 5-13 in one go?


Cheers

You would use the -w command

uv run unshackle dl -w S01E05-S01E13 ITV url

Brilliant, Cheers Mr Tom

I am glad that I was able to help. So often it is myself asking for help so it is nice to help when I can.

[Tom Saurus]

I don't know how many of you are still using my services, but I've set up a new repository on Github that will work interchangeably with Devine and Unshackle.

For those who've cloned the earlier repo from CDM-project, you can simply update the origin URL and pull the new updates:

# update URL
git remote set-url origin https://github.com/stabbedbybrick/services.git

# pull updates
git pull

I don't know how active I'll be, but I'll try to fix broken services and possibly add new ones. But to avoid headaches and drama, I'll continue to keep it focused on non-premium services for now.

Thank you for all that you have done and for your willingness to continue to use your skills and to put in the effort to keep the services working. As for new services I am one hoping for Plex as they have the long running soaps "Knots Landing" and "Falcon Crest".

[Tom Saurus]

Updated envied and TwinVine to reflect the recent update to unshackle 1.4.4. Both come with services and each offers options not available in unshackle. TwinVine is a two package install of VineFeeder and Envied; a simple uv installs both packages and it is ready to run.

Each install provides 28 services (not all tested)

https://github.com/vinefeeder/envied/tree/main
https://github.com/vinefeeder/TwinVine/tree/main

Thank you and I think I might put Twinvine on my other Mini PC.

[Tom Saurus]

It is important in my opinion to have a text document and write down little hints and such on how to do things in Unshackle. I find that it is easy to forget things and you have a document and write down about things such how to update Unshackle you can refer to that. Also get the information about usage arguments for Deviine where it shows the various commands and save that to refer to. This being said having information and understanding is a different animal all together as at times information is overwhelming and you have to set about trying to get a grasp. But don't be afraid to ask questions as you learn and others learn along with you.

For example Unshackle informed me that a new version was available and I had written down how to update and also that I was suppose to move my .yaml so it wouldn't be overwritten so that saved me a lot of searching around. My method most of the time in using Freevine, Devine and Unshackle is to write out the lines in a text document and then just copy and paste that over in to the program. Sometimes I have multiple lines I copy over and it batches for me most of the time.