Suggestions For Optimizing SD Animation Encoding

[drojman]

Like deciding on 10bit, when I did my initial research I settled on Bits/(Pixel*Frame) being more important than CRF (please stop laughing, all you geniuses) so I tried to make all my Simpsons and Futurama 720p encodes have ~2000Kbps video bitrate and, because of the different sources and scripts, the CRF's for Simpsons and Futurama were different.

I told myself I was going to hold off on doing the Simpsons as I spent so long doing the Futurama episodes, though a lot of that was down to me learning the programs really. But, I couldn't resist and did a quick snippet of the Simpsons pilot, and honestly the output is transformative. I need to sit down and rip them all now! I only have up to season 6, and being in the UK getting NTSC copies of the DVDs isn't always the easiest, but I'll at least make a start.

And thank you you checking that for me. Looking at the DVD, I think it's simply due to the sources and that nothing can be done - horizontal scrolling just doesn't really play nicely with interlaced content, if you take a look at this frame, specifically the stars, you can see where the problem comes from:


[Attachment 75046 - Click to enlarge]

The encodes are such a huge improvement, I just was worried I'd messed them up somehow!

[LouieChuckyMerry]

Encoding video such that the improvement is drastic has to be some kind of gateway drug .

[drojman]

Encoding video such that the improvement is drastic has to be some kind of gateway drug .

So I've come back to this after many months, having never processed my Simpsons files, and unfortunately the time away has not been beneficial to my understanding of MeGUI. Any chance you're still lingering the forums?

[LouieChuckyMerry]

"Lingering" seems creepy--what's wrong with keeping a tab open!?; everybody's doing it!--and my memory won't win any awards, but I seem to remember that you'd sorted out your script and were good to go. What can I do to help?

[drojman]

Haha, now you mention it that does sound worse than I intended!

Since I wrote that I have managed to re-teach myself some things. I have your copy of MeGUI still and I couldn't for the life of me figure out how to get plugins etc working again...that's no longer a problem and I've actually managed to re-encode all of season 1.

However! Did you ever figure out a way of speeding up the process of this all? Between splitting up the original MKV file, separating out the m2v that I need, indexing it, converting it and then putting it all back together as one MKV it takes me many, many hours per season. I wonder if you ever had any success with the methods that keep it was one MKV file? Or alternatively the OneClick encoder? I actually had a look at this and had some success, but my method required all episode files in the same folder to process, which is a nightmare for then splitting them into different folders and adding back in the audio later...

I'm rambling, season 2 is processing as we speak at least!

[LouieChuckyMerry]

However! Did you ever figure out a way of speeding up the process of this all? Between splitting up the original MKV file, separating out the m2v that I need, indexing it, converting it and then putting it all back together as one MKV it takes me many, many hours per season. I wonder if you ever had any success with the methods that keep it was one MKV file? Or alternatively the OneClick encoder? I actually had a look at this and had some success, but my method required all episode files in the same folder to process, which is a nightmare for then splitting them into different folders and adding back in the audio later...

Given that several of the required plugins for interlaced DVD's aren't built for multithreading, the best way I found to improve encoding speed was to load an even number of files to MeGUI for processing, then send them to as many separate drives as my computer has cores; for example, on my quad-core computer I'll load 4, 8, 12, 16 (any multiple of 4) files then send one fourth of the files to four different drives (internal and external, any combination) using the main MeGUI worker plus three temporary workers.

To make other processes, like (de)muxing and the such, more efficient I use the search program Everything to load multiple files to MKVToolNix, MeGUI, MKVCleaver, etc.; I have the GUI's on the left of my screen and the Everything window on the right, so that I can search the folder tree where my source files are located then select only the .mkv or .srt or .whatever files to drag-drop onto the GUI's. I tried to use the OneClick encoder when I first started using MeGUI, but quickly found it much too annoying (but that's just me).

[drojman]

Yeah, actually processing through MeGUI takes the longest in terms of time, but at least with some temporary workers I can just set and go. That at least seems to work well, if time consuming.

Great shout on Everything. That works so well, especially for MKVCleaver. I keep each episode in it's own folder so it's easier to manage once split, and being able to drag/drop them all is a lifesaver.

Here's another question then, for MKVtoolnix, when remuxing the files back together, how do you manage to label the tracks and set the default flags efficiently? Or do you not bother? I actually was able to use the command line to do it once for season 2 but that's only because that season only contains two sets of audio and one set of subtitles so you can just blanket name them. Is it really a case of otherwise going through each one?

Thank you!

[LouieChuckyMerry]

Yeah, actually processing through MeGUI takes the longest in terms of time, but at least with some temporary workers I can just set and go. That at least seems to work well, if time consuming.

I need more cores, ha ha.

Great shout on Everything. That works so well, especially for MKVCleaver. I keep each episode in it's own folder so it's easier to manage once split, and being able to drag/drop them all is a lifesaver.

I find Everything indispensable.

Here's another question then, for MKVtoolnix, when remuxing the files back together, how do you manage to label the tracks and set the default flags efficiently? Or do you not bother? I actually was able to use the command line to do it once for season 2 but that's only because that season only contains two sets of audio and one set of subtitles so you can just blanket name them. Is it really a case of otherwise going through each one?

I also have each episode in its own folder within a specific season folder within a main Simpsons folder. I've MKVToolNix set to name the output the same as the first file input; so, I'll add the properly named subtitle first, then add files via the right-click option; having the encoded video and the audio(s) in the same folder as the subtitles makes this easier. Then I queue as many episodes as necessary and run the batch. I don't name the tracks unless there's more than one audio, and when there is I copy-paste using Ctrl-V for "EnglishAudio" and an AutoHotkey combo for "Commentary". It's very tedious, but I'm basically clueless when it comes to command line use. I'd bet there's a way to batch name in MKVToolNix if you've the time to figure it out.

[drojman]

So, I don't know if this will be any help whatsoever to you, but I have found a way using Powershell to automate the remuxing processing. Essentially what this will do is:

 Scans all subfolders in D:\Video\Final folder.
    Skips folders named output (to avoid re-processing).
    Looks for:
        Video file: *_Track01.mkv.
        Audio files: At least one .ac3 file.
        Optional subtitle file: *.srt.
    Warns and skips if any required files are missing.
    Creates an output folder inside each subfolder.
    Merges detected files using mkvmerge:
        Labels first audio as "English 5.1".
        Labels extra audio as "Commentary" or "Commentary - X".
        Labels subtitle as "English - Subtitles".
    Saves the final .mkv in the output folder.

Result: A clean, merged .mkv file for every properly structured folder.

Obviously the folder locations are set for my PC, but it would be easy to change it for your purposes. And again, you might not want the commentary files named that way, or even the subtitles. I only keep the English version of each file so it works fine for me.

Anyway this is the script:

# Base folder containing all the subfolders
$BaseFolder = "D:\Video\Final folder"
$MKVToolNixPath = "D:\Video\mkvtoolnix\mkvmerge.exe"

# Loop through each subfolder recursively
Get-ChildItem -Path $BaseFolder -Recurse -Directory | Where-Object { $_.Name -ne "output" } | ForEach-Object {
    $Folder = $_.FullName
    Write-Host "Processing folder: $Folder"

    # Dynamically detect file paths in the subfolder
    $VideoFile = Get-ChildItem -Path $Folder -Filter "*_Track01.mkv" | Select-Object -ExpandProperty FullName -ErrorAction SilentlyContinue
    $AudioFiles = Get-ChildItem -Path $Folder -Filter "*.ac3" | Select-Object -ExpandProperty FullName -ErrorAction SilentlyContinue
    $SubtitleFile = Get-ChildItem -Path $Folder -Filter "*.srt" | Select-Object -ExpandProperty FullName -ErrorAction SilentlyContinue

    # Skip if the video file is missing
    if (-Not $VideoFile) {
        Write-Warning "Skipping $($Folder): Video file missing"
        return
    }

    # Skip if no audio files are detected
    if (-Not $AudioFiles -or $AudioFiles.Count -lt 1) {
        Write-Warning "Skipping $($Folder): No audio files detected"
        return
    }

    # Warn if no subtitle file is detected
    if (-Not $SubtitleFile) {
        Write-Warning "No subtitle file found in $Folder. Continuing without subtitles."
    }

    # Determine the output folder and file name
    $OutputFolder = Join-Path $Folder "output"
    if (-Not (Test-Path $OutputFolder)) {
        New-Item -ItemType Directory -Path $OutputFolder | Out-Null
    }
    $BaseOutputName = (Get-Item $VideoFile).BaseName -replace "_Track01$"
    $OutputFile = Join-Path $OutputFolder "$BaseOutputName.mkv"

    # Build the mkvmerge command
    $Command = @(
        "-o", $OutputFile,
        $VideoFile
    )

    # Process audio tracks
    for ($i = 0; $i -lt $AudioFiles.Count; $i++) {
        $AudioTrack = $AudioFiles[$i]
        $TrackLabel = if ($i -eq 0) {
            "English 5.1"
        } elseif ($i -eq 1) {
            "Commentary"
        } else {
            "Commentary - $([int]$i)"
        }
        $Command += @(
            "--default-track", "0:" + ($i -eq 0 ? "yes" : "no"),
            "--track-name", "0:$TrackLabel",
            "--language", "0:eng",
            $AudioTrack
        )
    }

    # Add the subtitle track if available
    if ($SubtitleFile) {
        $Command += @(
            "--default-track", "0:no",
            "--track-name", "0:English - Subtitles",
            "--language", "0:eng",
            $SubtitleFile
        )
    }

    # Run the command and capture output
    Write-Host "Running mkvmerge..."
    & $MKVToolNixPath @Command | Write-Host

    Write-Host "Successfully processed: $OutputFile"
}

Write-Host "All folders processed."

[LouieChuckyMerry]

So, I don't know if this will be any help whatsoever to you, but I have found a way using Powershell to automate the remuxing processing. Essentially what this will do is:

 Scans all subfolders in D:\Video\Final folder.
    Skips folders named output (to avoid re-processing).
    Looks for:
        Video file: *_Track01.mkv.
        Audio files: At least one .ac3 file.
        Optional subtitle file: *.srt.
    Warns and skips if any required files are missing.
    Creates an output folder inside each subfolder.
    Merges detected files using mkvmerge:
        Labels first audio as "English 5.1".
        Labels extra audio as "Commentary" or "Commentary - X".
        Labels subtitle as "English - Subtitles".
    Saves the final .mkv in the output folder.

Result: A clean, merged .mkv file for every properly structured folder.

Obviously the folder locations are set for my PC, but it would be easy to change it for your purposes. And again, you might not want the commentary files named that way, or even the subtitles. I only keep the English version of each file so it works fine for me.

Anyway this is the script:

# Base folder containing all the subfolders
$BaseFolder = "D:\Video\Final folder"
$MKVToolNixPath = "D:\Video\mkvtoolnix\mkvmerge.exe"

# Loop through each subfolder recursively
Get-ChildItem -Path $BaseFolder -Recurse -Directory | Where-Object { $_.Name -ne "output" } | ForEach-Object {
    $Folder = $_.FullName
    Write-Host "Processing folder: $Folder"

    # Dynamically detect file paths in the subfolder
    $VideoFile = Get-ChildItem -Path $Folder -Filter "*_Track01.mkv" | Select-Object -ExpandProperty FullName -ErrorAction SilentlyContinue
    $AudioFiles = Get-ChildItem -Path $Folder -Filter "*.ac3" | Select-Object -ExpandProperty FullName -ErrorAction SilentlyContinue
    $SubtitleFile = Get-ChildItem -Path $Folder -Filter "*.srt" | Select-Object -ExpandProperty FullName -ErrorAction SilentlyContinue

    # Skip if the video file is missing
    if (-Not $VideoFile) {
        Write-Warning "Skipping $($Folder): Video file missing"
        return
    }

    # Skip if no audio files are detected
    if (-Not $AudioFiles -or $AudioFiles.Count -lt 1) {
        Write-Warning "Skipping $($Folder): No audio files detected"
        return
    }

    # Warn if no subtitle file is detected
    if (-Not $SubtitleFile) {
        Write-Warning "No subtitle file found in $Folder. Continuing without subtitles."
    }

    # Determine the output folder and file name
    $OutputFolder = Join-Path $Folder "output"
    if (-Not (Test-Path $OutputFolder)) {
        New-Item -ItemType Directory -Path $OutputFolder | Out-Null
    }
    $BaseOutputName = (Get-Item $VideoFile).BaseName -replace "_Track01$"
    $OutputFile = Join-Path $OutputFolder "$BaseOutputName.mkv"

    # Build the mkvmerge command
    $Command = @(
        "-o", $OutputFile,
        $VideoFile
    )

    # Process audio tracks
    for ($i = 0; $i -lt $AudioFiles.Count; $i++) {
        $AudioTrack = $AudioFiles[$i]
        $TrackLabel = if ($i -eq 0) {
            "English 5.1"
        } elseif ($i -eq 1) {
            "Commentary"
        } else {
            "Commentary - $([int]$i)"
        }
        $Command += @(
            "--default-track", "0:" + ($i -eq 0 ? "yes" : "no"),
            "--track-name", "0:$TrackLabel",
            "--language", "0:eng",
            $AudioTrack
        )
    }

    # Add the subtitle track if available
    if ($SubtitleFile) {
        $Command += @(
            "--default-track", "0:no",
            "--track-name", "0:English - Subtitles",
            "--language", "0:eng",
            $SubtitleFile
        )
    }

    # Run the command and capture output
    Write-Host "Running mkvmerge..."
    & $MKVToolNixPath @Command | Write-Host

    Write-Host "Successfully processed: $OutputFile"
}

Write-Host "All folders processed."

[Attachment 83869 - Click to enlarge]

Wha?

[drojman]

Basically it's a script for MKVToolNix (well, MKVMerge). It'll batch mux the files rather than you having to do it manually. You would just need to set the folders you want to use, and any parameters. For example, I've got it set so that any audio streams beyond the first are labelled with Commentary.

[LouieChuckyMerry]

I was joking (poorly, it seems). I fully understand the concept but would need many long and intensely studious sessions before becoming capable of something like that (Translation: I'm old and learning new tricks is very difficult ). How's your encoding going?

[drojman]

Oh right, it's not as hard to manage as you think! Far simpler than your AVISynth scripts that's for sure! It's at least something. Although I have found a slightly easier tool for GUI usage, it's called jmkvpropedit. It essentially allows you to edit the headers of the MKV files. It's really simple to use, I highly recommend it.

Encoding is great, honestly, it's a shame there's no official version of them in this quality. I think with the quality of the sources it's pretty much as good as you can get. I am taking my time between seasons. I've done the first three for now, I'll come back and tackle them in 2/3 season chunks every few weeks I think.

[LouieChuckyMerry]

Oh right, it's not as hard to manage as you think! Far simpler than your AVISynth scripts that's for sure! It's at least something. Although I have found a slightly easier tool for GUI usage, it's called jmkvpropedit. It essentially allows you to edit the headers of the MKV files. It's really simple to use, I highly recommend it.

Thanks for the idea, I'll check it out. And, in case you didn't notice, MKVToolNix has a header editor--oooh, that's fun to say!--that even I've figured out how to use (because sometimes I mislabel a track and that really bothers me, ha ha).

Encoding is great, honestly, it's a shame there's no official version of them in this quality. I think with the quality of the sources it's pretty much as good as you can get. I am taking my time between seasons. I've done the first three for now, I'll come back and tackle them in 2/3 season chunks every few weeks I think.

I'm glad the encoding is progressing steadily. I can't remember which, but some channel modernized the earlier seasons some years ago; unfortunately, the 4:3 video was cropped to 16:9 and much visual information was lost--I did the calculations once, and if I remember it's about 7% on the top and 7% on the bottom--like this all time get-it-past-the-sensors gag.

Years ago I applied The Simpsons script to my Speed Racer {1967-68} DVD's, and was pretty happy with the results. Then I discovered recently that it came out on Blu-ray. Wow. They must've gone frame-by-frame regarding the interlacing. Of course, all the other noise was present but that was way easier to eliminate. Sorry for the Abe digression; I wholly agree that there should be official versions in 1080p that look really good.

[drojman]

Sorry, I should have been clearer, jmkvpropedit is just a plug-in for MKVToolNix. It's really handy because you can load all the MKVS at once and set rules for them - so for example Audio Track 2 is always flagged as a Commentary track, or named in a certain way. You can do it with all aspects, including subtitles etc. Obviously you need to be sure each MKV is typically set up the same way, but when doing lots of DVDs of the same nature they typically are.

Yeah, cropping was a real issue too. One of the worst I saw was the Duff Brewery gag.

It's so frustrating generally that a lot of this old animation is left to languish on old media. I know in the anime community, Dragon Ball Z is often a point of frustration, they started those perfect transfers of the level sets then just abandoned them.

[LouieChuckyMerry]

Sorry, I should have been clearer, jmkvpropedit is just a plug-in for MKVToolNix. It's really handy because you can load all the MKVS at once and set rules for them - so for example Audio Track 2 is always flagged as a Commentary track, or named in a certain way. You can do it with all aspects, including subtitles etc. Obviously you need to be sure each MKV is typically set up the same way, but when doing lots of DVDs of the same nature they typically are.

Damn! I wish I'd known of that ages ago, thank you.

Yeah, cropping was a real issue too. One of the worst I saw was the Duff Brewery gag.

I forgot about that one, thanks. It's a shame people can't seem to handle black bars left and right of the video image.

It's so frustrating generally that a lot of this old animation is left to languish on old media. I know in the anime community, Dragon Ball Z is often a point of frustration, they started those perfect transfers of the level sets then just abandoned them.

It's odd to me that I can make an encode that looks better than the highest quality official offering using the highest quality available offering as a source . I guess it's not profitable relative to the required resources, or something like that. Thankfully: AviSynth to the rescue!