Feeling Lost with Interlaced Video

[Downgraded286]

I'm in the process of ripping some DVDs. I intend to store them on my Plex server and use that to stream media in my house through the network.

Currently, my process is to copy the DVD with MakeMKV, and from there, I use Handbrake to compress them with h.264. However, the dreaded interlacing is kicking my ass.

Right now, I'm having problems with South Park. I've tried to read up on things, but I am a bit stuck. I've tried using the Decomb (Bob) and Detelecine filters, but neither give me perfect results. Here are some examples:

These three are with the Decomb filter, set to Bob...

One problem is I will see, for lack of better words, jaggies... like on Cartman's chin here.
https://forum.videohelp.com/images/imgfiles/kttoSDf.jpg

Decomb also misses a lot... character's mouths:
https://forum.videohelp.com/images/imgfiles/IpvZ5EP.jpg

And sometimes objects, like this wrecking ball:
https://forum.videohelp.com/images/imgfiles/ye5HQGU.jpg

These are with the Detelecine filter. It looks better than Decomb (jaggies like the ones from the scene with Cartman I put up earlier don't seem to be present) but it still isn't perfect...

Interlacing still seems to be present here:
https://forum.videohelp.com/images/imgfiles/QzLL2Wk.jpg

And this seems pretty bad too:
https://forum.videohelp.com/images/imgfiles/cxL7m9w.jpg

I can't believe this is this complicated. If it weren't for this damn interlacing, I could have saved everything in a neat little h.264 file already. This is driving me nuts. Am I missing something? Is there a better way to do this?

[jagabo]

Animated material is often a mix of frame rates, blended fields, and other artifacts that won't inverse telecine cleanly. I'm pretty sure early South Park episodes fit into that description. Handbrake will not be able to handle it correctly.

The easiest thing to do in Handbrake is encode interlaced and let the TV or player deinterlace on the fly. Turn off all filtering in Handbrake and enter tff or bff (whichever is appropriate) in the x264 Extra Options.

Next best in Handbrake would be to use the bob filter and encode a 2x the frame rate. Ie, 29.97 to 59.94 fps.

[Downgraded286]

I'm pretty much dead set on deinterlacing it when I encode. Deinterlacing on the fly not only seems to look worse, but also seems to be a really crappy way of doing things IMO. Even pirates don't release interlaced video.

I'm not sure if it's the right way to do things, but I've always set my framerate to Same as Source, and Variable. I read a guide on Handbrake's site about telecined video that recommended setting it as such. I tried setting it to 59.94 constant with decomb: bob as per your suggestion, if nothing else just to see what the results looked like... and I get the same crappy artifacting I did in the examples above. Wouldn't forcing it to such a high framerate cause it to take longer to encode, and produce a result of lesser quality and higher filesize as well?

I've tried using detelecine and decomb at the same time in Handbrake, and it seems to look okay... but I can't shake this overbearing feeling that I'm doing something very wrong by doing it this way.

[Skiller]

As mentioned animated stuff is often a wild mix of different frame rates often with several "layers" of frame rates (like credits over animation). Bob-deinterlacing to 59.94 fps is what I would do (if I really feel like not encoding it interlaced) and it's what I would suggest, especially if you want to be done with it before you are old and gray.

I don't know Handbrake too good, it probably used a rather simple deinterlacer thus the jaggies. I would use AviSynth and the superior QTGMC for deinterlacing. But beware, QTGMC is not exactly fast.

[PuzZLeR]

Yes, as was mentioned, animated material can be very complicated to de-interlace due to the nature of the beast when considering its creation/origin. It's not like it was just shot or filmed like most other sources. Hence the complications, the different patterns, etc., and why many automated methods will not be effective, not necessarily a failure on your part.

Yeah, some on-the-fly de-interlacing displays can make it look shoddy. I agree. Maybe because they too use an automated set algorithm throughout that won't be the right method for all scenes. But it's the easiest solution by far.

The only true, and best, way to de-interlace such content is scene-by-scene as needed, with the right parameters for telecine or de-interlacing, etc. Of course, this is not realistic for hobbyists wishing to do this for hours of content just to watch it on their media players. But, just saying.

Then again, if there's some "intelligent" de-interlacer out there, I wonder just how good it will be in detection. Or, how it can ever even decode patterns within layers.

I don't have a source clip to work with, but personally, my gut feeling would be to just use QTGMC, as mentioned, with double the frame rate, and be done with it. It should make it look much better than what any of your previous attempts acheived. Yes, QTGMC is slow, but at least you'll be done with this before you're "old and gray".

I don't think HandBrake does QTGMC, and it doesn't accept scripts. But if you still want to use it with this, you can instead create a lossless file first, and feed that into HandBrake, with all other filters off.

As per pirates and de-interlacing, I would bet much of their de-interlacing work is bad too. It may not be apparent since you don't have the source they worked with to compare and see the major quality that may have been lost.

[jagabo]

I think Handbrake uses Yadif to bob deinterlace.

The best way is to use AviSynth, of course.

[filmboss80]

I'll toss this into the conversation for your consideration. Instead of devoting a lot of hours to making acceptable looking re-encoded versions of your DVDs, you might just want to bite the bullet and spend a few bucks just getting the newly remastered 1080p versions of the early seasons of South Park. (Blu-ray release date unknown. They may not be out yet.) Even though the original output to TV was standard 4:3 SD, the animators maintained their original digital elements (done on Maya, I think), which were in HD at 16:9. Comedy Central is now showing the new widescreen 1080p versions of these early episodes, and they look stunning.

http://forum.dvdtalk.com/hd-talk/627150-southpark-early-seasons-now-hd.html

[Downgraded286]

Don't have any teeth to bite any bullets with, if you get what I mean...

AviSynth may be an option, but looks quite complicated. Any options on Handbrake with Detelecine and Decomb at the same time?

[manono]

AviSynth may be an option, but looks quite complicated.

It's your only option if you really want to get rid of those jaggies. And, yes, it's got a learning curve.

Any options on Handbrake with Detelecine and Decomb at the same time?

No, because its 'best' deinterlacer is Yadif which can look especially bad when applied to animations, as you've discovered yourself. The delecine filter won't help much in this case because it can't be tweaked. You're stuck with what it 'sees' as needing to be deinterlaced. Better AviSynth IVTCs can be tweaked in all kinds of ways. Better AviSynth deinterlacers won't leave behind those jaggies or broken mouths.

[jagabo]

Any options on Handbrake with Detelecine and Decomb at the same time?

Check some panning shots. Are they smooth or jerky?

[PuzZLeR]

AviSynth may be an option, but looks quite complicated.

It's really lots easier than it looks. Here's all you have to do:

Just install the application (version 2.6.0 is fine currently). Then go here and just follow a couple of the guides that currently say "Start Here" starting with "Your first script", and maybe one or two more. All you're really doing is writing a couple of very short text files renamed with extension *.avs and feeding them into an app (ex:VirtualDub).

And all this will take you less than an hour. After, it's just copy and paste really, which is what you'd be ready to do to use QTGMC or telecine de-interlacers.

That small amount of time investment will be very worth it. Once you know some basic AviSynth threre are so many things you can do with it. Besides, as was mentioned, you have little/no choice to solve this problem here anyway.

[Downgraded286]

So I need to install something else along Avisynth? I installed Avisynth, and it seems very unusual. There's no .exe, or anything. I'm somewhat, but not completely clueless here.

I'm willing to learn all this, but I do want to make one thing clear before I proceed. At this point in time, my interest with Avisynth lies solely with deinterlacing. I want to do no re-encoding with it. If I have no choice, then I want to do a lossless encode and re-run it through Handbrake later.

One other related question. I am not sure whether my DVDs are just interlaced, or telecined, or what. Currently, my understanding is, if I rip with MakeMKV and Mediainfo tells me it's progressive scan, then I don't need to do any of this (just re-encode with Handbrake for h.264). But if it says Interlaced, something's up. I've never seen it say anything other than progressive or interlaced. So I'm not sure when to deinterlace or detelecine.

It seems I have to write a script for each file? Could I automate it like... instead of D:\working\South Park S01E01.mkv, write D:\working\*.mkv so it will run through all of them?

One last thought. Does Avisynth do anything to the audio track?

[PuzZLeR]

So I need to install something else along Avisynth? I installed Avisynth, and it seems very unusual. There's no .exe, or anything. I'm somewhat, but not completely clueless here.

That's perfectly fine. AviSynth is kind of in the background in your computer. It's there. Now that you've installed it, open a text file, and rename its extension from *.txt to *.avs. (Or you can do it with the right-click shell extention.) The AVS file is now kind of a "pretend video". You write commands into it, and AviSynth responds to this. And any application that supports AviSynth will treat it like a video file now.

I'm willing to learn all this, but I do want to make one thing clear before I proceed. At this point in time, my interest with Avisynth lies solely with deinterlacing. I want to do no re-encoding with it. If I have no choice, then I want to do a lossless encode and re-run it through Handbrake later.

AviSynth is very much like creating a lossless file and running it through, without that middle extra step of actually making the lossless file. Unfortunately, if you want to run a script with HandBrake, this would be the only way since it doesn't (currently) support AviSynth. You can still make the lossless file with AviSynth (ex: HuffYUV and VirtualDub).

As per your current interests with AviSynth, once you learn to use it more you will find more interests. Yes, it's that good.

One other related question. I am not sure whether my DVDs are just interlaced, or telecined, or what. Currently, my understanding is, if I rip with MakeMKV and Mediainfo tells me it's progressive scan, then I don't need to do any of this (just re-encode with Handbrake for h.264). But if it says Interlaced, something's up. I've never seen it say anything other than progressive or interlaced. So I'm not sure when to deinterlace or detelecine.

A general simple rule is that when you scrub through the video frame-by-frame, look at the interlacing. Is it a non-pattern? Then that means you de-interlace. Is it a pattern that is 3 frames progressive, 2 interlaced, over and over? Then you have film source and you'd need to de-telecine. You can still de-interlace with film sources, but it's not the best way when you know for sure it's a film source.

But with animation it's not so simple. You may find within progressive, film sources, different telecine patterns, pure interlacing, and different layers upon different layers. If it's that complicated, just treat it as interlaced and hope for the best with QTGMC.

It seems I have to write a script for each file? Could I automate it like... instead of D:\working\South Park S01E01.mkv, write D:\working\*.mkv so it will run through all of them?

You'll define your workflow eventually as you'll find AviSynth is quite flexible. Really, once you know what you're doing, it's just copy and paste after.

One last thought. Does Avisynth do anything to the audio track?

Although I rarely work with audio and AviSynth, yes, it can manipulate audio too, but only if you decide it should. It can manipulate text and image files too. (I think subtitles too, but not sure here.)

[theewizard]

way too much work for fooling with south park
just how much do you watch those DVD episodes over and over

i would rip to mpeg or vob and apply some recode, but i would not mess with trying to de-interlace

better yet, do a screen capture record of the stream, so its all 29.xx or 30 or 25fps, what ever you like
then recode/compress that ,IF i wanted a smaller file size

IMO either way would give you closer to the end product results that you desire
it won't be as fast or as easy, but somethings can't be done fast and easy

[Downgraded286]

Now I get this, when trying to open my script with VirtualDub (No experience with VirtualDub either, just using it as a way to open my script):



This is my script:
LoadPlugin ("mt_masktools-26.dll", "mvtools2.dll", "nnedi3.dll", "RemoveGrainSSE2.dll")
FFmpegSource2("South Park S03E01.mkv")
QTGMC( Preset="Slow" )
SelectEven()

It seems to be complaining about my plugin not being there, but it's right there in my avisynth\plugins folder.

@theewizard This isn't just about South Park. South Park is the subject at this point in time, but this really is about getting perfect (or close to perfect as possible) deinterlacing, regardless of what show it may be.

[jagabo]

If the dll files are in AviSynth's plugins folder they are automatically loaded, so you don't need to manually load them in your script. If they are not in that folder you should use the full path to each plugin.

LoadPlugin("x:\path\to\some\folder\somewhere\plugname.dll")

[Downgraded286]

If I remove my LoadPlugin line, this happens:

[sophisticles]

Or you can just watch South Park for free, legally, online and not drive yourself crazy with all that work.

[Downgraded286]

Well, like I said earlier... not just about South Park. Additionally, streaming is pretty well out of the question. We were heavy on Hulu/Netflix... until we started getting letters from the ISP about going over our data cap. I'd rather save stuff to a computer I have here on the network which I'll be setting up for Plex, to stream locally.

[theewizard]

its not all that hard to de-interlace film dvd or televison DVD, as yadif will work well
but your south park animae is full of variable frame rate video, its not all the same
and much of this stuff is orginally created with rates as low as 7~15 fps, with repeated frames to fill in for broadcast on TV
it just doesn't de-interlace well ,
you picked possibly the worst example to use as representation and finding the best scheme for doing ALL your DVDs

and of course you didn't really have any problems, until you got to these DVDs
you just can't do these with scripted functions, like you want to
it takes a different approach

[Downgraded286]

Okay, I'm a lot closer. I got this to work. I reinstalled avisynth and all the plugins... maybe I missed something. It's even running multithreaded! There are still a few issues that stand out, however. Before I say anything, let me point out that I ran my script by opening it with VirtualDub, and from there, I went to File... Save as AVI. The result though, looks really good.

Firstly, it seems the only way I can do this is to edit my script for each episode, and run it through VirtualDub, one at a time. Is there any way to automate this, or at least set up a batch script?

Secondly, the resulting file is enormous. It's 40GB! Handbrake readily compressed it to 80MB, however.

Thirdly, the audio is gone. Is there any to preserve the audio?

Lastly, the chapter markers are gone. Any way to preserve this?

I ran another episode through, and lost subtitles as well. I'd like to keep these...

[jagabo]

Why don't you just use the x264 command line encoder? You can set up a batch file for encoding then drag/drop AVS scripts onto it. Or put the batch file in your Send To folder, right click on an AVS script and select Send To -> batchname.bat.

I usually handle audio separately. I extract it with DgIndex (which I use to build an index for Mpeg2Source(), used to open the video in AviSynth) then mux the AC3 files with the output of x264.

The way you're doing it, you can reduce the size of the intermediate file produced by VirtualDub, without losing any qualty, by using a lossless codec like UT Video Codec. Be sure to use VIrtualDub in Video -> Fast Recompress mode to prevent any colorspace conversions.

[Downgraded286]

@Jagabo What x264 command line encoder are you talking about? Is it for AviSynth or VirtualDub? The idea of opening AVS scripts with a batch file sounds fine with an already prepared batch file. Do I need to install something else? How would I set this up? Whatever it is, I would definitely want it to be lossless, as it's going to be re-run through Handbrake later, anyway.

I'm having a new problem. What I've done is installed MKVToolNix, and used it to open the AviSynth/VirtualDub output file, as well as the original file from the DVD. I mux the video stream from the AviSynth/VirtualDub output file, and the audio and subtitle streams from the DVD, resulting in a deinterlaced video with audio and subtitles.

However, there is now a new roadblock at the last step. Handbrake doesn't pick up on the audio or subtitles! In fact, it doesn't pick up on anything. I can't even add the file to queue. MPCHC plays audio and lets me select subtitles just fine. Mediainfo shows they're there.

I just tried the latest nightly build of Handbrake, and it says this about the file produced by MKVToolNix:


I don't get it. I used MKVToolNix to merge 2 other episodes together, and Handbrake loaded it fine. Handbrake loads the .avi that AviSynth/VirtualDub makes. Handbrake will NOT load the file I make, combining the video stream from the .avi made by AviSynth/Virtualdub, and the audio stream/subtitles from the corresponding file copied from the DVD. What the hell?

[Downgraded286]

Update on the previous issue.

I tried running the AviSynth output file, by itself, through MKVToolNix. Handbrake will not load this file. It'll load the AviSynth output file, unless I run it through MKVToolNix. When I do, even if I just run it through by itself, Handbrake fails to recgonize it.

[manono]

If I have no choice, then I want to do a lossless encode and re-run it through Handbrake later.

I read that and it makes absolutely no sense to me. "My inferior encoding program doesn't work well with this difficult source so I'll use a better program to do the filtering before then running the results through the same inferior encoding program."

That's what it says to me.

Update on the previous issue.

I tried running the AviSynth output file, by itself, through MKVToolNix. Handbrake will not load this file. It'll load the AviSynth output file, unless I run it through MKVToolNix. When I do, even if I just run it through by itself, Handbrake fails to recgonize it.

Ready to try something else yet? Either take the original script or take the AVI you made (I hope it's not uncompressed and in RGB) or place the AVI in a new AviSynth script using AVISource to open it (test in VDub first) and open that in a better encoding program. One like RipBot264, and then make your MKV or whatever.

[Downgraded286]

You make a valid point. I'm only scratching the surface with AviSynth here. It still remains a complex tool to me, and its capabilities are a mystery. So far, I know I can use it to deinterlace video with QTGMC, and that's about it.

The AVI I made is indeed uncompressed RGB. I wonder if this is why Handbrake has trouble opening it, though I remain curious as to why it will open the AviSynth output file, but it won't open what is basically the same file after I run the AviSynth output file, by itself, though MKVToolNix.

Here is my script:

SetMTMode(5, 4)
FFmpegSource2("South Park S07E01.mkv")
SetMTMode(2)
QTGMC(Preset="Draft", EdiThreads=2)
SelectEven()

I open this file with VirtualDub, and then go to File > Save as AVI...

This process results in the massive ~40GB uncompressed RGB file, which is a video stream ONLY, having lost the audio, subtitles, and chapter markers. If I could do my encoding (h.264 codec, .mp4 container is the target result) at the same time as the QTGMC deinterlacing process, and keep my audio (I transcode this as well, to HE-AAC Stereo), subtitles (as a track, not burned in), and chapter markers, I would. Maybe there is a way, but I do not know of it. And if I can encode at this step, there will be no horrible 40GB result, so I would also like to set up batch processing somehow, so I didn't have to re-write my script for each individual episode and go through it all over again one by one.

[manono]

The AVI I made is indeed uncompressed RGB

There's nothing wrong with making intermediate AVIs. I do it daily when there are a series of encodes of the same video to do. Uncompressed isn't what you want, though. You want lossless. I use Lagarith. Others use the UT Video Codec, and there are others as well. They produce compressed but lossless AVIs. In addition, when using VDub for encoding stay within the same YV12 colorspace as your DVD source. This means, among other things, setting VDub for Fast Recompress (not the default Full Processing Mode) after opening AviSynth scripts. And set up your lossless AVI codec to output YV12 as well. And your video sizes, while still large, will be significantly smaller than your 40GB. I don't use Handbrake but wouldn't be surprised if it was the reason it wouldn't open your AVI.

Or just open that script in the encoder and encode directly from it.

Your script gives you 29.97fps, right? I've never worked with South Park but would bet it's supposed to be 23.976fps. Following the SelectEven you probably want to decimate (remove frames) it to 23.976fps using the TDecimate part of TIVTC.

I do my audio and subtitles outside of the AviSynth scripts. I use RipBot264 and after opening the script, then open the audio for reencoding to AAC audio. I believe the subs can be muxed in using something else. Chapter points also when remuxing with something like MKVToolNix. Others know better than I about the use of such tools.

[Downgraded286]

You may be right about the framerate. My script does give me 29.97fps, and I have other South Park DVDs that are already Progressive scan (later seasons) and mediainfo shows them as 23.976fps. It sounds like I will have to add Decimate to my script. How do I know when and when not to use it? Will that require downloading another plugin, this TIVTC I assume, or should I have this already in QTGMC?

I'll have to give that a try, as well as adding a codec. I might give UT Video Codec a try first, some quick web surfing seems to show that as the more popular/better choice. Not sure exactly what I have to do to install/use that, though.

So how about batch processing, then? Is it possible, or do I have to do each video one by one?

[jagabo]

Cartoons are almost always made on film at 24 fps. Panning shots usually move with each frame but character animation is usually at half or a third of that rate. Made for SD TV cartoons are then telecined to 30i and edited with no consideration for cutting on progressive frames. So you often end up with orphaned fields (ie, a single field from a film frame, with no matching field to complete that frame). Then some shots are sped up or slowed down with field blending to match the music or dialog, or just to make shots run longer or shorter. Titles are often added as 30p or 30i. With cartoons like this you can't get all perfect progressive frames, or perfect motion at 30p or 24p.

That's why I recommend leaving the video interlaced or smart bobbing to 60p. With the former the video shouldn't look much worse than playing the original DVD. With the latter, some frames may pick up artifacts but since they are only visible for 1/60 second you don't notice them much.

[Downgraded286]

Well, I did give your suggestion a shot (Decomb - Bob @ FPS 59.94) it just still didn't come out right. There's also other reasons I want to deinterlace at the encoding level. As an example, MPCHC's deinterlacing looked questionable at best, at least in the LAV Video Decoder internal filter. I understand there's other shit I can install to run with MPCHC, but I believe videos should just play... I shouldn't have to tweak all this relatively complicated shit to play some crappy interlaced video. Then, what happens when I load them into Plex? I don't know, but I believe it's probably best to deinterlace at the encoding level not only for quality but compatibility reasons.

Also, as I said before, this isn't just about South Park. Learning to do this now will give me options in the future should I run across more difficult material.

I'd like to thank you guys for all your guidance so far. This isn't quite over yet, though...

Well, added TDecimate without issue. Brought my FPS down to 23.976, and the filesize of the resulting uncompressed RGB file dropped quite a bit, due to the slower framerate, I imagine. Now that I have that working, time to get UT Video Codec working...

Alright, TDecimate is working, UT Video Codec is working, and the muxed file made by MKVToolNix loads into Handbrake. At this point I can actually start working!

However, it's very cumbersome. Is there any way to automate this process so I don't have to re-write my script and do each one one by one? Also, VirtualDub won't save my settings. I have to set Video > Compression and select my codec, Video > Fast Recompress, and Audio > No Audio (since it drops audio, anyway). I'd a least like VirtualDub to save my settings, it seems like that much at least should be possible. Automated AviSynth scripts I don't know about, but a program should at least be able to save my settings, right?

Alright, I found a cumbersome way to get VirtualDub to save my settings. I saved a file in VirtualDub and re-load it on startup. I even found a way to get a little bit of automation going. For the past couple hours I wrote avisynth scrips for every file I need work done on while doing some other stuff. Now, I can load scripts in VirtualDub's job control. But... it'a only using about half my CPU. I've done some testing, not sure why. But it's running. I can't see what FPS it's running at while I load it up via job control, but in individual testing it seems to be around 14FPS, which is okay. I really don't mind if it's going to take a while, as long as I reach the desired results. In this way, it seems I can!