Devine - Modular Movie, TV, and Music Archival Software

[billybanana]

Starting a new Devine discussion to keep it out of stabby's Freevine thread.

https://github.com/devine-dl/devine

Discuss away

[billybanana]

I'm getting an error on all Channel 4 downloads. I can't figure out what file is can't find.

D:\Downloads\Devine>devine dl -w S01E01 ALL4 helsinki-crimes

                       / __ \/ ____/ |  / /  _/ | / / ____/
                      / / / / __/  | | / // //  |/ / __/
                     / /_/ / /___  | |/ // // /|  / /___
                    /_____/_____/  |___/___/_/ |_/_____/
                    ⠀
                    v3.3.1 Copyright © 2019-2024 rlaphoenix
                      https://github.com/devine-dl/devine

     Service Config loaded
     Loaded Cdm Widevine CDM: 4464 (L3)
     Loaded 1 Vaults

  ────────────────────────────── Service: ALL4 ───────────────────────────────

     Service is Geoblocked in your region, getting a Proxy to gb
      + Using cached Tokens...
     Authenticated with Service

  ───────────────────────── Series: Helsinki Crimes ──────────────────────────

     1 Season, 8 Episodes

  ────────────────────────── Helsinki Crimes S01E01 ──────────────────────────

     Requesting ANDROID assets...
     Requesting WEB assets...

     1 Video
     └── [H.264, SDR] | en | 1920x1080 @ 5019 kb/s, 25.000 FPS
         ⠼ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ • --:-- • -
     1 Audio
     └── [AAC] | 2.0 | 133 kb/s | en
         ⠼ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ • --:-- • -
     2 Subtitles
     ├── [VTT] | en
     │    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ • 00:03 • Downloaded
     └── [VTT] | en | SDH
          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ • 00:05 • Downloaded
     5 Chapters
     ├── 00:00:00.000 | Chapter 01
     ├── 00:08:55.480 | Chapter 02
     ├── 00:17:39.040 | Chapter 03
     ├── 00:27:13.480 | Chapter 04
     └── 00:37:41.160 | Credits

╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ C:\Python312\Lib\site-packages\devine\commands\dl.py:539 in result           │
│                                                                              │
│   536 │   │   │   │   │   │   │   )                                          │
│   537 │   │   │   │   │   │   │   for i, track in enumerate(title.tracks)    │
│   538 │   │   │   │   │   │   )):                                            │
│ ❱ 539 │   │   │   │   │   │   │   download.result()                          │
│   540 │   │   │   except KeyboardInterrupt:                                  │
│   541 │   │   │   │   console.print(Padding(                                 │
│   542 │   │   │   │   │   ":x: Download Cancelled...",                       │
│                                                                              │
│ C:\Python312\Lib\concurrent\futures\_base.py:449 in result                   │
│                                                                              │
│   446 │   │   │   │   if self._state in [CANCELLED, CANCELLED_AND_NOTIFIED]: │
│   447 │   │   │   │   │   raise CancelledError()                             │
│   448 │   │   │   │   elif self._state == FINISHED:                          │
│ ❱ 449 │   │   │   │   │   return self.__get_result()                         │
│   450 │   │   │   │                                                          │
│   451 │   │   │   │   self._condition.wait(timeout)                          │
│   452                                                                        │
│                                                                              │
│ C:\Python312\Lib\concurrent\futures\_base.py:401 in __get_result             │
│                                                                              │
│   398 │   def __get_result(self):                                            │
│   399 │   │   if self._exception:                                            │
│   400 │   │   │   try:                                                       │
│ ❱ 401 │   │   │   │   raise self._exception                                  │
│   402 │   │   │   finally:                                                   │
│   403 │   │   │   │   # Break a reference cycle with the exception in self._ │
│   404 │   │   │   │   self = None                                            │
│                                                                              │
│ C:\Python312\Lib\concurrent\futures\thread.py:58 in run                      │
│                                                                              │
│    55 │   │   │   return                                                     │
│    56 │   │                                                                  │
│    57 │   │   try:                                                           │
│ ❱  58 │   │   │   result = self.fn(*self.args, **self.kwargs)                │
│    59 │   │   except BaseException as exc:                                   │
│    60 │   │   │   self.future.set_exception(exc)                             │
│    61 │   │   │   # Break a reference cycle with the exception 'exc'         │
│                                                                              │
│ C:\Python312\Lib\site-packages\devine\core\tracks\track.py:199 in download   │
│                                                                              │
│   196 │   │   │   │   │   license_widevine=prepare_drm                       │
│   197 │   │   │   │   )                                                      │
│   198 │   │   │   elif self.descriptor == self.Descriptor.DASH:              │
│ ❱ 199 │   │   │   │   DASH.download_track(                                   │
│   200 │   │   │   │   │   track=self,                                        │
│   201 │   │   │   │   │   save_path=save_path,                               │
│   202 │   │   │   │   │   save_dir=save_dir,                                 │
│                                                                              │
│ C:\Python312\Lib\site-packages\devine\core\manifests\dash.py:404 in          │
│ download_track                                                               │
│                                                                              │
│   401 │   │   │   │   res = session.get(url=rep_base_url, headers=init_range │
│   402 │   │   │   │   res.raise_for_status()                                 │
│   403 │   │   │   │   init_data = res.content                                │
│ ❱ 404 │   │   │   │   track_kid = track.get_key_id(init_data)                │
│   405 │   │   │   │   total_size = res.headers.get("Content-Range", "").spli │
│   406 │   │   │   │   if total_size:                                         │
│   407 │   │   │   │   │   media_range = f"{len(init_data)}-{total_size}"     │
│                                                                              │
│ C:\Python312\Lib\site-packages\devine\core\tracks\track.py:364 in get_key_id │
│                                                                              │
│   361 │   │   if not isinstance(init_data, bytes):                           │
│   362 │   │   │   raise TypeError(f"Expected init_data to be bytes, not {ini │
│   363 │   │                                                                  │
│ ❱ 364 │   │   probe = ffprobe(init_data)                                     │
│   365 │   │   if probe:                                                      │
│   366 │   │   │   for stream in probe.get("streams") or []:                  │
│   367 │   │   │   │   enc_key_id = stream.get("tags", {}).get("enc_key_id")  │
│                                                                              │
│ C:\Python312\Lib\site-packages\devine\core\utils\subprocess.py:23 in ffprobe │
│                                                                              │
│   20 │   elif isinstance(uri, bytes):                                        │
│   21 │   │   args.append("pipe:")                                            │
│   22 │   try:                                                                │
│ ❱ 23 │   │   ff = subprocess.run(                                            │
│   24 │   │   │   args,                                                       │
│   25 │   │   │   input=uri if isinstance(uri, bytes) else None,              │
│   26 │   │   │   check=True,                                                 │
│                                                                              │
│ C:\Python312\Lib\subprocess.py:548 in run                                    │
│                                                                              │
│    545 │   │   kwargs['stdout'] = PIPE                                       │
│    546 │   │   kwargs['stderr'] = PIPE                                       │
│    547 │                                                                     │
│ ❱  548 │   with Popen(*popenargs, **kwargs) as process:                      │
│    549 │   │   try:                                                          │
│    550 │   │   │   stdout, stderr = process.communicate(input, timeout=timeo │
│    551 │   │   except TimeoutExpired as exc:                                 │
│                                                                              │
│ C:\Python312\Lib\subprocess.py:1026 in __init__                              │
│                                                                              │
│   1023 │   │   │   │   │   self.stderr = io.TextIOWrapper(self.stderr,       │
│   1024 │   │   │   │   │   │   │   encoding=encoding, errors=errors)         │
│   1025 │   │   │                                                             │
│ ❱ 1026 │   │   │   self._execute_child(args, executable, preexec_fn, close_f │
│   1027 │   │   │   │   │   │   │   │   pass_fds, cwd, env,                   │
│   1028 │   │   │   │   │   │   │   │   startupinfo, creationflags, shell,    │
│   1029 │   │   │   │   │   │   │   │   p2cread, p2cwrite,                    │
│                                                                              │
│ C:\Python312\Lib\subprocess.py:1538 in _execute_child                        │
│                                                                              │
│   1535 │   │   │                                                             │
│   1536 │   │   │   # Start the process                                       │
│   1537 │   │   │   try:                                                      │
│ ❱ 1538 │   │   │   │   hp, ht, pid, tid = _winapi.CreateProcess(executable,  │
│   1539 │   │   │   │   │   │   │   │   │   │    # no special security        │
│   1540 │   │   │   │   │   │   │   │   │   │    None, None,                  │
│   1541 │   │   │   │   │   │   │   │   │   │    int(not close_fds),          │
╰──────────────────────────────────────────────────────────────────────────────╯
FileNotFoundError: [WinError 2] The system cannot find the file specified

     ❌ Download Failed...
        One of the track downloads had an error!
        See the error trace above for more information.

[LastResort]

Billy, May I ask how you managed to get your installation to see your service config please...

I'm on windows 10, pip installed Devine.
Created a folder on my C drive simply called Devine.
images of installed files:



[Attachment 78261 - Click to enlarge]


[Attachment 78262 - Click to enlarge]


[Attachment 78263 - Click to enlarge]

I remember (i think it was iamghost) mention about the weird / \ change of these when using windows. so the Devine.yaml is using the reverse /.

upon running an elevated cmd prompt: and using devine env info i get the following:


[Attachment 78264 - Click to enlarge]

So I moved the folders cookies, WVD's cache, logs, to the AppData Local devine folder, Hoping this would achieve my goal. And correct this error:

(this is a test to search)


[Attachment 78265 - Click to enlarge]

Due to the "no service added" obviously something is not being seen.

And it seems that devine.yaml is being read from the appdata\local\devine\ folder. (which it shouldn't be doing.)
So my next attempt was to move my devine.yaml to appdata\local\devine\ folder to see what would happen. that's when things didn't really get exciting.
devine env info gives the following:


[Attachment 78266 - Click to enlarge]

I'm really hoping there's going to be a simple explanation because I've got a banging headache.... lol

[PSXman_uk]

Devine.yaml needs to be in appdata devine in my case i need to edit it with Notepad ++

Everytime i make a change I keep a copy there also as its funny with the indents

Services: are in this folder

C:\Scripts\devinenv\Lib\site-packages\devine\services



[Attachment 78268 - Click to enlarge]

All the other folders like cookies Wds are there also works for me

[PSXman_uk]

[Attachment 78269 - Click to enlarge]

billy are all your bin files in path ? works for me with your command

[billybanana]

billy are all your bin files in path ? works for me with your command

I don't believe so and this is where the documentation/help file is about as clear as mud. I've got the below files in my D:\Downloads\Devine directory:

aria2c.exe
ccextractor.exe
ffmpeg.exe
mkvmerge.exe
packager-win-x64.exe

but given the majority of the links for those files on git are not just the executables, I'm not really sure what files are required and where to put them. I'm sure I won't be alone in this, so if you can confirm what your set up is, I should be able to resolve the problem.

[PSXman_uk]

Not all is needed for Devine but i keep it all in one place

The rest is all in the environment that I setup from the start

[stabbedbybrick]

I don't believe so and this is where the documentation/help file is about as clear as mud. I've got the below files in my D:\Downloads\Devine directory:

aria2c.exe
ccextractor.exe
ffmpeg.exe
mkvmerge.exe
packager-win-x64.exe

but given the majority of the links for those files on git are not just the executables, I'm not really sure what files are required and where to put them. I'm sure I won't be alone in this, so if you can confirm what your set up is, I should be able to resolve the problem.

You're missing ffprobe.exe.

I would HIGHLY recommend all windows users to simply create a C:\bin and put that folder in PATH and just place everything needed in there. That way you don't have to mess up any individual project folders.

[PSXman_uk]

I don't believe so and this is where the documentation/help file is about as clear as mud. I've got the below files in my D:\Downloads\Devine directory:

aria2c.exe
ccextractor.exe
ffmpeg.exe
mkvmerge.exe
packager-win-x64.exe

but given the majority of the links for those files on git are not just the executables, I'm not really sure what files are required and where to put them. I'm sure I won't be alone in this, so if you can confirm what your set up is, I should be able to resolve the problem.

You're missing ffprobe.exe.

I would HIGHLY recommend all windows users to simply create a C:\bin and put that folder in PATH and just place everything needed in there. That way you don't have to mess up any individual project folders.

Yep makes sense I have a backup that runs at night for all the envs anyway

[achilles]

                                                                              │
│ C:\Python312\Lib\site-packages\devine\core\utils\subprocess.py:23 in ffprobe │
│                                                                              │
│   20 │   elif isinstance(uri, bytes):                                        │
│   21 │   │   args.append("pipe:")                                            │
│   22 │   try:                                                                │
│ ❱ 23 │   │   ff = subprocess.run(                                            │
│   24 │   │   │   args,                                                       │
│   25 │   │   │   input=uri if isinstance(uri, bytes) else None,              │
│   26 │   │   │   check=True,                                                 │
│                                                                              │
│ C:\Python312\Lib\subprocess.py:548 in run                                    │
│                                                                              │
│    545 │   │   kwargs['stdout'] = PIPE                                       │
│    546 │   │   kwargs['stderr'] = PIPE                                       │
│    547 │                                                                     │
│ ❱  548 │   with Popen(*popenargs, **kwargs) as process:                      │
│    549 │   │   try:                                                          │
│    550 │   │   │   stdout, stderr = process.communicate(input, timeout=timeo │
│    551 │   │   except TimeoutExpired as exc:                                 │
│                                                                              │
│ C:\Python312\Lib\subprocess.py:1026 in __init__                              │
│                                                                              │
│   1023 │   │   │   │   │   self.stderr = io.TextIOWrapper(self.stderr,       │
│   1024 │   │   │   │   │   │   │   encoding=encoding, errors=errors)         │
│   1025 │   │   │                                                             │
│ ❱ 1026 │   │   │   self._execute_child(args, executable, preexec_fn, close_f │
│   1027 │   │   │   │   │   │   │   │   pass_fds, cwd, env,                   │
│   1028 │   │   │   │   │   │   │   │   startupinfo, creationflags, shell,    │
│   1029 │   │   │   │   │   │   │   │   p2cread, p2cwrite,                    │
│                                                                              │
│ C:\Python312\Lib\subprocess.py:1538 in _execute_child                        │
│                                                                              │
│   1535 │   │   │                                                             │
│   1536 │   │   │   # Start the process                                       │
│   1537 │   │   │   try:                                                      │
│ ❱ 1538 │   │   │   │   hp, ht, pid, tid = _winapi.CreateProcess(executable,  │
│   1539 │   │   │   │   │   │   │   │   │   │    # no special security        │
│   1540 │   │   │   │   │   │   │   │   │   │    None, None,                  │
│   1541 │   │   │   │   │   │   │   │   │   │    int(not close_fds),          │
╰──────────────────────────────────────────────────────────────────────────────╯
FileNotFoundError: [WinError 2] The system cannot find the file specified

     ❌ Download Failed...
        One of the track downloads had an error!
        See the error trace above for more information.

For future reference, if you look at this error output, you will see that the traceback starts in subprocess.py within the python library itself so that's unlikely to be causing the problem. This is the most recent call and going up the chain you can see that the first error that occurs within the devine code is at a different subprocess.py.

 C:\Python312\Lib\site-packages\devine\core\utils\subprocess.py:23 in ffprobe

It stopped in a module called ffprobe which is a dead giveaway for what you are missing. And if you look in that part of the devine code, it says "Use ffprobe on the provided data to get stream information." so that's obviously where it's have the problem.

stabbedbybrick mentioned this but I am posting this so next time, the error output might be more readable to help figure out what the issue is.

BTW, it's not a bad idea to be running devine in a virtual environment. It looks like you installed devine into your global python library.

[achilles]

And it seems that devine.yaml is being read from the appdata\local\devine\ folder. (which it shouldn't be doing.)

This is where devine.yaml is read from. It doesn't read from a devine.yaml in wherever you running devine from.

And to save on future headaches, "wvds" should be in all lowercase. It won't work otherwise. And it should be the directory name only, don't include the .wvd file. So, "C:/devine/WVDs" just like the other directories.

https://github.com/devine-dl/devine/blob/master/CONFIG.md gives the expected syntax of the config file.

[iamghost]

new update for iplayer> https://cdm-project.com/stabbedbybrick/devine-services/src/branch/main/services/iP/__init__.py
just replace __init__.py file

[A_n_g_e_l_a]

new update for iplayer

Did you get spotify (SPOT) to work? Mine just hangs at the download stage. Back to spot-dl I guess.

Edit; forget that. It does download but seems not to show it to screen.

[ringy9]

Hi trying an ALL4 download and keeps failing at VTT subtitles. I don't need them. Can anyone tell me a command so it doesn't download subtitles? I have tried --video-only --audio-only --subs only srt but I'm obviously not getting correct command.

[A_n_g_e_l_a]

Hi trying an ALL4 download and keeps failing at VTT subtitles. I don't need them. Can anyone tell me a command so it doesn't download subtitles? I have tried --video-only --audio-only --subs only srt but I'm obviously not getting correct command.

--sub-format SRT ... put it before the ALL4 tag

[iamghost]

new update for iplayer

Did you get spotify (SPOT) to work? Mine just hangs at the download stage. Back to spot-dl I guess.

I didn’t try it, I just merge tpd and stabbybrick services. Tpd other 2 services I have removed cos of error when I start Devine. Might be not work for python 3.9

[A_n_g_e_l_a]

new update for iplayer

Did you get spotify (SPOT) to work? Mine just hangs at the download stage. Back to spot-dl I guess.

I didn’t try it, I just merge tpd and stabbybrick services. Tpd other 2 services I have removed cos of error when I start Devine. Might be not work for python 3.9

I've got SPOT sort of working. It hangs if you ask for an Album download but works OK for single downloads. Still doesn't beat spot-dl.
The problem is all these CDM guys have forked rlaphoenix's repository and made changes to the structure or are behind synchronizing rlaphoenix's updates . Pheonix himself suggests that as a way forward. What it means is that there will be services of the same tag-name knocking around that will only work on the forked platform. I had to carry over a file and folder to get SPOT to run.
I predict it is going to get very messy!

[ringy9]

Thank you @ Angela

[stabbedbybrick]

The problem is all these CDM guys have forked rlaphoenix's repository and made changes to the structure or are behind synchronizing rlaphoenix's updates . Pheonix himself suggests that as a way forward. What it means is that there will be services of the same tag-name knocking around that will only work on the forked platform. I had to carry over a file and folder to get SPOT to run.
I predict it is going to get very messy!

Yeah, this is the reason I decided to host only a service folder. People can just set it directly in the config or grab the services individually, and keep the Devine installation isolated.

I'd recommend cloning the original repo, set up poetry, and just add services to it as you see fit.

[peterb999]

Thanks for the help everyone. I've got it working for ALL4 on windows and it downloads fine, including the subtitles.

However, iPlayer seems to fail for me as I get:

AttributeError: 'Segment' object has no attribute 'keys'

It seems to be failing on line 473 in hls.py. I'm trying to work out why it can't find keys.

[A_n_g_e_l_a]

Thanks for the help everyone. I've got it working for ALL4 on windows and it downloads fine, including the subtitles.

However, iPlayer seems to fail for me as I get:

AttributeError: 'Segment' object has no attribute 'keys'

It seems to be failing on line 473 in hls.py. I'm trying to work out why it can't find keys.

For the time being BBC in UK-FTA is hours ahead faster and you've got that. I tried 'This Town' not only did it download the trailor but I also something else to do with fashion was on the list. However it bailed out after episode 4. Using the default downloader, (requests), devine seems to create dozens of threads; it confuses itself and fails waiting for download. So still work in progress.

[peterb999]

Thanks Angela, yes, your UK-FTA is an excellent tool and I use that a lot.

To be honest I quite enjoy just playing around with new stuff and as stabbedbybrick had kindly created an iPlayer one I thought I'd try and get it to work.
I think I actually enjoy getting this sort of stuff working more than actually watching the shows. It's like doing a jigsaw.

[A_n_g_e_l_a]

Thanks Angela, yes, your UK-FTA is an excellent tool and I use that a lot.

To be honest I quite enjoy just playing around with new stuff and as stabbedbybrick had kindly created an iPlayer one I thought I'd try and get it to work.
I think I actually enjoy getting this sort of stuff working more than actually watching the shows. It's like doing a jigsaw.

Yes I think we are all doing that, but it is bug-city at the moment.

[A_n_g_e_l_a]

Yeah, this is the reason I decided to host only a service folder.

Well! Knock me down with a feather! There was I thinking you'd got fed up explaining how to use Windoze, for anything useful, to wanna-be keyboard warriors.

[achilles]

Hi trying an ALL4 download and keeps failing at VTT subtitles. I don't need them. Can anyone tell me a command so it doesn't download subtitles? I have tried --video-only --audio-only --subs only srt but I'm obviously not getting correct command.

--sub-format SRT ... put it before the ALL4 tag

Can also add

dl:
   sub_format: srt

to config file if you want to change the default sub format for all services.

[A_n_g_e_l_a]

Can also add

dl:
   sub_format: srt

to config file if you want to change the default sub format for all services.

You know I feel this is quite like the old days learning to use yt-dlp with its conundrum of settings to get the correct streams. Devine is very clever software but using it does feel like a step backwards in time. It shouldn't be this way.

[A_n_g_e_l_a]

[Attachment 78274 - Click to enlarge]
Slow.

[achilles]

One thing I really wish devine had was an option for N_m3u8DL-RE as a downloader. Fast and reliable. Handles errors/timeouts well.

[stabbedbybrick]

[Attachment 78274 - Click to enlarge]
Slow.

[Attachment 78275 - Click to enlarge]

But yes, in general I feel like N_m3u8DL-RE is the most stable downloader. And you could theoretically implement it to a service, but I feel like it would be a nightmare to handle the output and events so it's not worth it.

[LastResort]

Win 10, fully updated. python all up to date.

I could do with some help please chaps..... Things seem to have taken a radical turn for the worse... if i try to run devine --help from an elevated cmd prompt from my devine folder I no longer get:

[Attachment 78282 - Click to enlarge]

but instead all i get is this:


[Attachment 78283 - Click to enlarge]

If i do not use "poetry shell"

I still get this:


[Attachment 78284 - Click to enlarge]

I re-installed devine, no error messages appeared. So I am now at a loss as to what I should do.

I do have a .venv folder within my Devine folder,... and my devine.yaml has been reversed back to this:

directories:
services: C:\devine\devine\services
temp: C:\devine\temp
downloads: C:\devine\downloads
cookies: C:\devine\cookies
cache: C:\devine\cache
logs: C:\devine\Logs
WVDs: C:\devine\WVDs\8162_l3.wvd
cdm:
default: 8162_l3

credentials:
ALL4:
ROKU: emailassword

key_vaults:
- type: SQLite
name: Local Vault
path: C:\devine\devine\key_store.db

tag: ''

downloader: aria2c

profiles:
default: default
-------------------------------------
I tried having the devine.yaml in my C:\Devine\devine folder but still get the same error message as above.
also tried having devine.yaml in C:\Users\xxxx\AppData\Local\devine folder but this makes no difference.

Is it at all possible to reverse the installation to enable a clean re-install. Appreciate any advice/guidance please chaps.