Suggestions For Optimizing SD Animation Encoding

[drojman]

That does not cause a crash!

I think it's something wrong with the way I have plugins setup. If I run some of the other scripts, they work OK, but this one doesn't.

If I force the use of either the internal 32bit AviSynth or use the external 32bit one, it doesn't actually crash and even warns me about this errant lines, which is fixed if I remove it. However if I try and load the script it throws an error about TFM not being present.

[drojman]

It's something to do with this part of this script:

### Deinterlace-Match Fields-Decimate ###
AssumeTFF()
TFM(Chroma=False,PP=0) 
AssumeBFF()
Interleave(TFM(Mode=1,PP=0,Field=1),TFM(Mode=1,PP=0,Field=0))
TFM(Field=0,Clip2=Yadif())
vInverse()
SRestore(23.976)

If I remove it, it saves and starts encoding (after removing the previously mentioned text on the other page of course)

[LouieChuckyMerry]

TFM is part of TIVTC, which is definitely in the plugins folder I upped. Check out the link; I'll be back, for an hourish, in an hour-plus (glad we're making progress).

[drojman]

Yes it seems that the x86 and x64 version of AviSynth had the same problem but just showed them in different ways.

I updated the plugin - in case it needed it.

Still crashing.

[drojman]

IT'S ALIVEEEE!

I was missing some plugins:

Yadif
srestore
vInverse

With them installed, the file is now encoding!

Any hints etc on what profile to use? I selected the Futurama Upscale one in the encoder, I figured that should be about right?

[LouieChuckyMerry]

Wow, I'm so sorry; all my effort has been toward making the Simpsons script work; learning what was wrong reminded me that the Futurama scrip requires extra plugins. D'oh! A moment...

[drojman]

Don't say sorry! You've been incredibly helpful.

Does this info look correct to you?

(I know nothing about why the buffer is smaller than the video size)


[Attachment 74982 - Click to enlarge]

I've only done the one file so far, but I have to say it's a good deal better than the straight rips already!

[LouieChuckyMerry]

Excellent job scoring the missing plugins! Seriously. And the info looks correct; do you have MediaInfo available? The Futurama profile is for Futurama so it's surely the choice .

[drojman]

This is my MediaInfo in text form:

General
Unique ID                                : 79364049122204711427895422452880683788 (0x3BB4F5BA1EE9DF1D6B2D0890AA08B30C)
Complete name                            : D:\Test folder for trying out AviSynth scripts\Futurama - S01E01 - Space Pilot 3000\Track01_Futurama - S01E01 - Space Pilot 3000.mkv
Format                                   : Matroska
Format version                           : Version 4
File size                                : 273 MiB
Duration                                 : 23 min 18 s
Overall bit rate                         : 1 639 kb/s
Frame rate                               : 23.976 FPS
Encoded date                             : 2023-11-20 04:19:24 UTC
Writing application                      : mkvmerge v68.0.0 ('The Curtain') 64-bit
Writing library                          : libebml v1.4.2 + libmatroska v1.6.4

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High 10@L4.1
Format settings                          : CABAC / 8 Ref Frames
Format settings, CABAC                   : Yes
Format settings, Reference frames        : 8 frames
Codec ID                                 : V_MPEG4/ISO/AVC
Duration                                 : 23 min 18 s
Bit rate                                 : 1 637 kb/s
Width                                    : 960 pixels
Height                                   : 720 pixels
Display aspect ratio                     : 4:3
Original display aspect ratio            : 4:3
Frame rate mode                          : Constant
Frame rate                               : 23.976 (24000/1001) FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 10 bits
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 0.099
Stream size                              : 273 MiB (100%)
Writing library                          : x264 core 164 r3164 c196240
Encoding settings                        : cabac=1 / ref=8 / deblock=1:2:1 / analyse=0x3:0x113 / me=umh / subme=8 / psy=1 / psy_rd=0.40:0.00 / mixed_ref=1 / me_range=20 / chroma_me=1 / trellis=2 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=-2 / threads=22 / lookahead_threads=5 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / stitchable=1 / constrained_intra=0 / bframes=5 / b_pyramid=2 / b_adapt=2 / b_bias=0 / direct=3 / weightb=1 / open_gop=0 / weightp=2 / keyint=240 / keyint_min=23 / scenecut=40 / intra_refresh=0 / rc_lookahead=60 / rc=crf / mbtree=1 / crf=13.0 / qcomp=0.60 / qpmin=0 / qpmax=81 / qpstep=4 / vbv_maxrate=50000 / vbv_bufsize=50000 / crf_max=0.0 / nal_hrd=none / filler=0 / ip_ratio=1.40 / aq=1:0.60
Default                                  : Yes
Forced                                   : No
Color range                              : Limited
Color primaries                          : BT.709
Transfer characteristics                 : BT.709
Matrix coefficients                      : BT.709

[LouieChuckyMerry]

Thanks for that because it reminded me, before you proceed, you should check that your Final Viewing Location (FVL) can handle the x264 settings. I use a laptop connected to an HDTV via an HDMI cable so if my player (MPC-BE) plays it then I can watch it. I think you mentioned Plex; it might have a limit on the number of Reference frames or the peak video bitrate, or whatever, like physical DVD and Blu-ray players do. It's easy enough to adjust the x264 settings if necessary.

[drojman]

Great, thanks. I will definitely test that before I get too far ahead then. Any reason not to test the MKV thats been made without audio?

Just looking into the process of MKVToolNix rejoining them together. I noticed you can right click on a data track and send that to the title, not sure if you had seen that option. In terms of merging them, is it 'safe' to leave everything at default? I don't want it ruining all the work the encode has done!

[LouieChuckyMerry]

Great, thanks. I will definitely test that before I get too far ahead then. Any reason not to test the MKV thats been made without audio?

No, the original audio should be fine (I think it's .ac3) but the video's the issue; you're taking a standard format and changing it, so your method of playing-viewing it should be verified.

Just looking into the process of MKVToolNix rejoining them together. I noticed you can right click on a data track and send that to the title, not sure if you had seen that option. In terms of merging them, is it 'safe' to leave everything at default? I don't want it ruining all the work the encode has done!

I'm very happy with my settings after taking some time to get there. If you want, then you can take the "mkvtoolnix-gui.ini" file from my uploaded portable version, navigate to the folder containing your portable or installed "mkvtoolnix-gui.ini" file (Everything!), rename yours temporarily, paste mine, then open MKVToolNix and check the differences (if unhappy, then simply reverse the .ini file swap). MKVToolNix is, for me, one of the greatest freeware programs ever-and-for-the-future-ever. Default setting are absolutely safe.

[drojman]

Thanks, I just downloaded and looked into your settings then and very handily I spotted that you can set an output folder relative to do the source file. Should let me easily move the merged MKV seperately in an automated fashion. The only thing I was really worried about was the default audio/subtitles flags, I wanted the right ones to play first, so will see what happens there.

Sadly I have encountered a nasty bug bear as side product of the crash using MEGUI before removing those couple of lines. If you try and add more than one episode, when you get to the point where you actually save the .avs script it crashed the whole program so that the other episodes never get their scripts saved. Rather annoying.

EDIT: Ah, unchecking save and open for encoding gets around that as it doesn't try to process the script. Still, is there possibly a faster method for adding all the files?

And on the other end, is it possible to speed up adding in script? Don't get me wrong it's not so much the time consumption, I'm just more likely to miss adding a file to the queue etc this way.

[LouieChuckyMerry]

Thanks, I just downloaded and looked into your settings then and very handily I spotted that you can set an output folder relative to do the source file. Should let me easily move the merged MKV seperately in an automated fashion. The only thing I was really worried about was the default audio/subtitles flags, I wanted the right ones to play first, so will see what happens there.

Whichever track of a type you add first typically automatically becomes the default; so, highlight the main audio and the commentary together then drag with your cursor on the main audio and it should populate first.

Sadly I have encountered a nasty bug bear as side product of the crash using MEGUI before removing those couple of lines. If you try and add more than one episode, when you get to the point where you actually save the .avs script it crashed the whole program so that the other episodes never get their scripts saved. Rather annoying.

Check out this thread; halfway to a solution for the above.

EDIT: Ah, unchecking save and open for encoding gets around that as it doesn't try to process the script. Still, is there possibly a faster method for adding all the files?

I think if you setup One-Click (or some such) in MeGUI it's possible, but I never tried. This thread and the above link are your best place for MeGUI help.

And on the other end, is it possible to speed up adding in script? Don't get me wrong it's not so much the time consumption, I'm just more likely to miss adding a file to the queue etc this way.

Again the One-Click (I'll try to figure out the proper term) feature. Do a quick search, but the two above threads are excellent for learning more about MeGUI. Now to work, ugh; more later.

Edit: MeGUI/Tools/One Click Encoder Wiki

[drojman]

Ah I had a fiddle with that One-click but honestly I think I'll just throw myself off more. Best stick with what I at least can sort of understand I guess!

Just saw that solution. I tried doing the DLL swap but that still gave me the crash. However unchecking colour correct and Mpeg2 Deblocking worked nicely! I saw your comment about disabling it for all - I think if you edit the script config in MEGUI you can uncheck those and then it won't have them enabled but I'm not sure if functionally that would change anything...?

[LouieChuckyMerry]

Ah I had a fiddle with that One-click but honestly I think I'll just throw myself off more. Best stick with what I at least can sort of understand I guess!

Nice to know I'm not alone .

Just saw that solution. I tried doing the DLL swap but that still gave me the crash. However unchecking colour correct and Mpeg2 Deblocking worked nicely! I saw your comment about disabling it for all - I think if you edit the script config in MEGUI you can uncheck those and then it won't have them enabled but I'm not sure if functionally that would change anything...?

Wha? (I love Professor Farnsworth). I think I understand; were you able to sort it out? I've waaaaaay less free time on weekdays but even if I don't reply for a stretch I've not forgotten and will eventually return.

[drojman]

Sorry -

Basically if you edit the AviSynth script profile in MEGUI, you can disable those two functions. They'll appear greyed out, if you hit Config and then (something like)
Extra or External you can uncheck those boxes. I'd screenshot it now but I'm currently attempting to encode Season 1 of Futurama!

[LouieChuckyMerry]

Basically if you edit the AviSynth script profile in MEGUI, you can disable those two functions. They'll appear greyed out, if you hit Config and then (something like)
Extra or External you can uncheck those boxes

Ahhhhhh! I need to reconfigure the profile without a video file being indexed. D'oh! Now the student teaches the master(wort)... Thanks!

Edit: It's under "Extra Setup".

[drojman]

Yep! That's where I found it. Let me know if you decide to do anything and it works for you.

This whole process takes a long time, I'm dreading ripping my Simpsons DVD's, I only have the first 6 seasons right now...

[LouieChuckyMerry]

Yep! That's where I found it. Let me know if you decide to do anything and it works for you.

I have it sorted out now, thanks. In case you use them, these other profiles would need to have those two boxes unticked also:


[Attachment 75004 - Click to enlarge]

This whole process takes a long time, I'm dreading ripping my Simpsons DVD's, I only have the first 6 seasons right now...

At least MakeMKV does most of the work .

[drojman]

Thank you

I'll definitely use the Simpsons proifle, I'm currently processing S2 of Futurama atm. It doesn't seem to utilise my whole CPU but I'm not going to fiddle with anything!

[LouieChuckyMerry]

I found multithreading an issue with any script containing TIVTC--I half remember that it has something to do with TIVTC not being linear, but please correct me if I'm wrong (and I'll try to remember this time)--so instead I would queue a number of episodes that's a multiple of the number of cores in whatever computer I was using to encode and use a temporary worker for each group to utilize the entire CPU. For example, with a quad-core I'd load forty episodes, highlight ten of them in the queue, right-click on the first one and select "Run in new temporary worker". Then highlight ten more, same thing; ten more, same thing, ten more same thing. It works quite well.

[drojman]

Oh...

I had absolutely no idea the temporary worker thing was even possible!

Even to split the workload between 4 cores would save a massive amount of time. I've been letting my PC do one at a time, what an idiot I am!

I'm too worried about messing it up now to stop it, but season 4 I'll definitely try that on!

[LouieChuckyMerry]

Also, I don't know if it's necessary but it seems like a good idea to me: I send one queue to my laptop's storage drive and the other three queues to three different thumb drives; the sources are on my OS SSD. How's muxing your encoded videos with their original audios going?

[drojman]

Ah...

I didnt think to seperate the outputs, they're currently all encoding to one drive. Seems to be ok, I did 5 epsiodes a season per core.

Muxing is the final step tonight and I'm wondering how best to tackle.it honestly. From memory you have to queue each folder individually as it's own project I think?

Did you have to mess with any default flags in MKVNix in terms of commentary etc? I kept the commentary tracks for the show.

Thanks

[LouieChuckyMerry]

Honestly, it took a bit of effort to understand the the new, far more functional MKVToolNix GUI; I was running 8.3.0 because it's way less busy until I made the switch around 56.0.0 or so. And once you get the hang of it it's brilliant. Here are several important settings to consider in red boxes and the blue box is the job queue.


[Attachment 75024 - Click to enlarge]

I've a programmable mouse and found that incredibly helpful. If you already have a method to name things to your liking, then there's no need to worry about which file to add first because MKVToolNix is smart enough to always place video first even if it's added last; however, as I suggested earlier, I would highlight the two audio tracks at the same time then drag & drop with the cursor on the main audio; that way it'll populate first and be the default audio track. Before trying to queue all your episodes for muxing I'd highly recommend fiddling about was a couple-three episodes until you understand the possibilities and you're happy with your method of loading, queuing, muxing. I like my settings on the MKVToolNix I uploaded for you for batching jobs but you might not.

[drojman]

OK - I know what you were saying when you praised MKVToolNix. What a wonderful program. I actually redownloaded a copy of the portable one, I like that I can just delete the INI file and have it fresh in case I made a mistake.

What I ended up doing was pretty much as you said, I copied a version of the re-encoded first episode and played around with that, once I figured out in MKVToolNix that you can set any Unknown languages to default to a language, that made things so much easier - my audio tracks had no metadata.

Something I have noticed, in your profile it encodes the MKVs to 10 bit h264. Gives me great quality, but will it be any different to 8bit? The only reason I ask is because Plex has to transcode 10 bit down to 8 if I stream it (for h264). Makes me wonder if I would be losing quality, and I'd be better off rencoding to 8 bit. Then if I went down that route I don't know if I should start all over again with MEGUI or just re-encode what I have

Oh well, it's probably not worth fretting about.

Another question! I notice your Simpsons CRF seems to be set at 17, that's a fair bit higher than Futurama? Did you find the DVD quality didn't require the same level of encoding as Futurama?

[LouieChuckyMerry]

OK - I know what you were saying when you praised MKVToolNix. What a wonderful program. I actually redownloaded a copy of the portable one, I like that I can just delete the INI file and have it fresh in case I made a mistake.

I find that handy too being me; I actually copy all my .ini files regularly with FreeFileSync (another brilliant freeware program).

Something I have noticed, in your profile it encodes the MKVs to 10 bit h264. Gives me great quality, but will it be any different to 8bit? The only reason I ask is because Plex has to transcode 10 bit down to 8 if I stream it (for h264). Makes me wonder if I would be losing quality, and I'd be better off rencoding to 8 bit. Then if I went down that route I don't know if I should start all over again with MEGUI or just re-encode what I have

When I first started playing with video way back when I settled on 10bit because it seemed the way to go after all my research and I wasn't sharing the encodes with anybody. I'd stick with 10bit for the quality (especially to decrease banding) because better to drop down to 8bit from a better source.

Another question! I notice your Simpsons CRF seems to be set at 17, that's a fair bit higher than Futurama? Did you find the DVD quality didn't require the same level of encoding as Futurama?

I need to clean a couple houses; I'll think about this and answer ASAP.

[drojman]

From reading around too it seems like H10 is considered the best standard in the anime ripping community too. I'm definitely going to keep a H10 copy, I may use FFMPEG to transcode a 8 bit version as well just for Plex (the less transcoding the better)

Weird question. On your Futurama DVDs.or in your encodes in the last scene of the first episode when they're stood in the ship and the stars.are.scrolling by horizontally, do you notice they appear to be stuttering? I thought at first I'd made an error with the encode, but the stutter seems to be present on my DVDS too.
If you don't have time to take a look I completely understand, I've already taken up so much lf your time!

[LouieChuckyMerry]

Thanks for your patience.

Another question! I notice your Simpsons CRF seems to be set at 17, that's a fair bit higher than Futurama? Did you find the DVD quality didn't require the same level of encoding as Futurama?

Like deciding on 10bit, when I did my initial research I settled on Bits/(Pixel*Frame) being more important than CRF (please stop laughing, all you geniuses) so I tried to make all my Simpsons and Futurama 720p encodes have ~2000Kbps video bitrate and, because of the different sources and scripts, the CRF's for Simpsons and Futurama were different.

From reading around too it seems like H10 is considered the best standard in the anime ripping community too. I'm definitely going to keep a H10 copy, I may use FFMPEG to transcode a 8 bit version as well just for Plex (the less transcoding the better)

Odds are good you read some of the same things I did .

Weird question. On your Futurama DVDs.or in your encodes in the last scene of the first episode when they're stood in the ship and the stars.are.scrolling by horizontally, do you notice they appear to be stuttering? I thought at first I'd made an error with the encode, but the stutter seems to be present on my DVDS too.

Starting at 21m44s in my NTSC source and final encode, when Fry raises his arms in excitement because he's going to be a delivery boy, it appears that the Planet Express ship goes backwards then stutters a bit when once again going forward, but it might be partially an illusion due to Fry bending at the waist in the direction of the rear of the ship then quickly returning to an upright position, and the crap quality of the source.

If you don't have time to take a look I completely understand, I've already taken up so much lf your time!

I'm happy to actually offer help instead of ask for it; no problem whatsoever .