Suggestions For Optimizing SD Animation Encoding

[LouieChuckyMerry]

Hello, and thanks in advance for any help . I've ripped my Simpsons Seasons 1-10 NTSC DVDs with MakeMKV and am now wanting to use MeGUI to shrivel and clean up each episode for portability. My objective is a final video bit rate ~1500 Kbps. Seasons 1-7 have (according to MediaInfo, at least) interlaced video (here's a sample: S1.E1-TestClip). Through trial-and-error and (mostly) the kindness of others I've come up with the following script:

# Set DAR in encoder to 6480 : 4739. The following line is for automatic signalling
global MeGUI_darx = 6480
global MeGUI_dary = 4739
LoadPlugin("F:\[0]StandAloneApps\MeGUI-2500(core)2443(data)0.3.5(libs)[Portable]\tools\ffms\ffms2.dll")
Threads=4
SetMTMode(5,Threads)
FFVideoSource("D:\Temp\zSimpTemp\[000000]MeGUITemp\S1.E1-TestClip.mkv", fpsnum=30000, fpsden=1001, threads=1)
# SelectRangeEvery(1000,66)
SetMTMode(2)
LoadPlugin("F:\[0]StandAloneApps\MeGUI-2500(core)2443(data)0.3.5(libs)[Portable]\tools\avisynth_plugin\TIVTC.dll")
TFM(Order=-1).TDecimate()
QTGMC(FPSDivisor=2)
Toon(Strength=0.3)

the results of which are reasonable given the source material. I'd very much like to maximize the quality (and minimize the edge noise) and was hoping that someone more fluent in the language of AviSynth, especially when it comes to SD animation, would suggest tweaks for the above script (or a new script altogether, if necessary). I realize that only so much quality can be squeezed from this source but I'm certain that some of the savants frequenting this great site know secrets . Thank you for your time.

[LouieChuckyMerry]

In case somebody bumps into this thread in a quest for answers, my final script for this source material is

# Set DAR in encoder to 6480 : 4739. The following line is for automatic signalling
global MeGUI_darx = 6480
global MeGUI_dary = 4739
LoadPlugin("F:\[0]StandAloneApps\MeGUI-2500(core)2443(data)0.3.5(libs)[Portable]\tools\ffms\ffms2.dll")
#SetMemoryMax(200)
#SetMTMode(5,4)
FFVideoSource("D:\Temp\zSimpTemp\[000000]MeGUITemp\S1.E1-[I-4028].mkv", fpsnum=30000, fpsden=1001, threads=1)
#SetMTMode(2)
### Deinterlace ###
LoadPlugin("F:\[0]StandAloneApps\MeGUI-2500(core)2443(data)0.3.5(libs)[Portable]\tools\avisynth_plugin\TIVTC.dll")
TFM(Order=0,Slow=2,PP=0).TDecimate(Mode=1)
Vinverse()
### Deshaker ###
Stab(Mirror=15)
### Crop ###
Crop(8,0,-8,0)
### Resize ###
RatioResize(10/11.0,"PAR")
### Gibbs Noise Block ###
Edge=MT_Edge("prewitt",thY1=20,thY2=40).RemoveGrain(17)
Mask=MT_Logic(Edge.MT_Expand().MT_Expand().MT_Expand().MT_Expand(),Edge.MT_Inflate().MT_Inpand(),"xor")
MT_Merge(dfttest(),Mask,Luma=True)
### Overall Temporal Denoise, Could Probably Be Optimized ###
SMDegrain(tr=2,thSAD=600,ContraSharp=True,RefineMotion=True,Plane=0,Lsb_Out=True,PreFilter=2,Chroma=True)
### Debanding ###
GradFun3(thR=0.55,Radius=12,Mask=2,SMode=1,Lsb=True,Lsb_In=True, StaticNoise=True,Y=3,U=3,V=3)
DitherPost(Stacked=True,Prot=False,Mode=0)
### Line Darkener And Thinner ###
FastLineDarkenMod(Strength=20,Prot=6,Thinning=0)
aWarpSharp2(Blur=4,Type=1,Depth=3,Chroma=2)

I came to this with the kind and patient help of Dogway here. (S)he deserves all the credit . Also, for what it's worth, MT AviSynth ran fine with the test clip but trying to encode an entire episode always ended in freezing; however, running two encodes simultaneously with regular AviSynth achieved ~100% CPU usage. I'll figure out MT AviSynth some day...

[LouieChuckyMerry]

With the help of the kind people here and extensive testing (special thanks to Groucho2004's AVSMeter ), I'm now able to run my script in AviSynth 2.6 MT with seeming stability on my dual-core and quad-core laptops (encoding directly to x264, no intermediary). In case someone someday finds this information useful:

1) Windows 7 64 bit, hyperthreaded i5 3320M, 8GB RAM setup. The following script successfully runs multiple queued encodes in MeGUI, achieving ~2.65 FPS, a three-times improvement over normal AviSynth's ~0.85 FPS:

SetMemoryMax(256)
SetMTMode(3,3)
LoadPlugin("F:\[0]StandAloneApps\MeGUI-2500(core)2443(data)0.3.5(libs)[Portable]\tools\lsmash\LSMASHSource.dll")
LWLibavVideoSource("D:\Temp\S1.E1-OriginalVideo[NTSC].mkv")
### Deinterlace ###
SetMTMode(5)
LoadPlugin("F:\[0]StandAloneApps\MeGUI-2500(core)2443(data)0.3.5(libs)[Portable]\tools\avisynth_plugin\TIVTC.dll")
TFM(Order=0,Slow=2,PP=0).TDecimate(Mode=1)
SetMTMode(2)
Vinverse()
### Deshaker ###
Stab(Mirror=15)
### Crop ###
Crop(8,0,-8,0)
### Resize ###
RatioResize(10/11.0,"PAR")
### Gibbs Noise Block ###
Edge=MT_Edge("prewitt",thY1=20,thY2=40).RemoveGrain(17)
Mask=MT_Logic(Edge.MT_Expand().MT_Expand().MT_Expand().MT_Expand(),Edge.MT_Inflate().MT_Inpand(),"xor")
MT_Merge(dfttest(),Mask,Luma=True)
### Overall Temporal Denoise, Could Probably Be Optimized ###
SMDegrain(tr=2,thSAD=600,ContraSharp=True,RefineMotion=True,Plane=0,Lsb_Out=True,PreFilter=2)
### Debanding ###
GradFun3(thR=0.55,Radius=12,Mask=2,SMode=1,Lsb=True,Lsb_In=True, StaticNoise=True,Y=3,U=3,V=3)
DitherPost(Stacked=True,Prot=False,Mode=0)
### Line Darkener And Thinner ###
FastLineDarkenMod(Strength=20,Prot=6,Thinning=0)
aWarpSharp2(Blur=4,Type=1,Depth=3,Chroma=2)

2) Windows 7 64 bit, hyperthreaded i7 3840QM, 16GB RAM setup. The following script successfully runs multiple queued encodes in MeGUI, achieving ~5.1 FPS, a not-quite two-times improvement over normal AviSynth's ~2.8 FPS:

SetMemoryMax(512)
SetMTMode(3,6)
LoadPlugin("F:\[0]StandAloneApps\MeGUI-2500(core)2443(data)0.3.5(libs)[Portable]\tools\lsmash\LSMASHSource.dll")
LWLibavVideoSource("D:\Temp\S1.E1-OriginalVideo[NTSC].mkv")
### Deinterlace ###
SetMTMode(5)
LoadPlugin("F:\[0]StandAloneApps\MeGUI-2500(core)2443(data)0.3.5(libs)[Portable]\tools\avisynth_plugin\TIVTC.dll")
TFM(Order=0,Slow=2,PP=0).TDecimate(Mode=1)
SetMTMode(2)
Vinverse()
### Deshaker ###
Stab(Mirror=15)
### Crop ###
Crop(8,0,-8,0)
### Resize ###
RatioResize(10/11.0,"PAR")
### Gibbs Noise Block ###
Edge=MT_Edge("prewitt",thY1=20,thY2=40).RemoveGrain(17)
Mask=MT_Logic(Edge.MT_Expand().MT_Expand().MT_Expand().MT_Expand(),Edge.MT_Inflate().MT_Inpand(),"xor")
MT_Merge(dfttest(),Mask,Luma=True)
### Overall Temporal Denoise, Could Probably Be Optimized ###
SMDegrain(tr=2,thSAD=600,ContraSharp=True,RefineMotion=True,Plane=0,Lsb_Out=True,PreFilter=2)
### Debanding ###
GradFun3(thR=0.55,Radius=12,Mask=2,SMode=1,Lsb=True,Lsb_In=True, StaticNoise=True,Y=3,U=3,V=3)
DitherPost(Stacked=True,Prot=False,Mode=0)
### Line Darkener And Thinner ###
FastLineDarkenMod(Strength=20,Prot=6,Thinning=0)
aWarpSharp2(Blur=4,Type=1,Depth=3,Chroma=2)

L-SMASHWorks proved to be much more stable than FFMS2, and the final trick was to lower the number of cores by 25%. I'd reckon that these settings could be fine-tuned to squeeze a wee bit more FPS's, especially with the quad-core, but that'll have to wait until I figure out how to use QTGMC with AviSynth 2.6 MT .


Edit: it seems that TIVTC should always be run with SetMTMode(5), so as to not be multi-threaded (thus avoiding having frames taken out of order). The above scripts have been changed to reflect this.


EditEdit: midway through Season 3 I've discovered that some of the source episodes have variable frame rate video, which causes the encoded video to have an odd frame rate and the audio to be out of sync . Several days of searching for a solution has left my eyes crossed and my brain mush. Any suggestions on how to alter the above scripts for these variable frame rate sources to output 23.976 video and synced audio would be very much appreciated. Ahhh: S3.E11-TestClip

[LouieChuckyMerry]

I've tried every TFM-TDecimate combination I can imagine; I've skipped TFM-TDecimate altogether; I've played with the original VOB file, removing the pulldown flag. The output video bit rate is still screwy and the audio is still out of sync. Perhaps the answer is so obvious that I'm completely missing it, the proverbial if-it-was-a-snake, ready to bite. Please, any help would save what's left of the hair on my head . Thanks.

[manono]

EditEdit: midway through Season 3 I've discovered that some of the source episodes have variable frame rate video...

I'm not sure what that means, as all NTSC DVDs output interlaced 29.97fps. In addition, looking at it another way, all animations are variable framerate, almost by definition, as they can be drawn at 8, 12, 24, or 30fps, and any combination of those, and others also.

Anyway, yours isn't all soft telecined so just stripping out the flags is a recipe for the audio getting out of synch and leaving some combing behind. Maybe what you meant to say with the variable framerate remark is that it's a mix of hard and soft telecine.

I extracted the M2V from your useless MKV so I could run it through DGIndex with the Field Operation set for 'Honor Pulldown Flags' and then use MPEG2Source on it. This script works well to IVTC it and return it to 23.976fps:

TFM(D2V="Test.d2v").TDecimate(Mode=1)

[LouieChuckyMerry]

EditEdit: midway through Season 3 I've discovered that some of the source episodes have variable frame rate video...

I'm not sure what that means, as all NTSC DVDs output interlaced 29.97fps. In addition, looking at it another way, all animations are variable framerate, almost by definition, as they can be drawn at 8, 12, 24, or 30fps, and any combination of those, and others also.

Anyway, yours isn't all soft telecined so just stripping out the flags is a recipe for the audio getting out of synch and leaving some combing behind. Maybe what you meant to say with the variable framerate remark is that it's a mix of hard and soft telecine.

What I meant is that for the first two-plus seasons all the individual episode .mkvs have a video MediaInfo as such:

Video
ID                                       : 1
Format                                   : MPEG Video
Format version                           : Version 2
Format profile                           : Main@Main
Format settings, BVOP                    : No
Format settings, Matrix                  : Custom
Codec ID                                 : V_MPEG2
Codec ID/Info                            : MPEG 1 or 2 Video
Duration                                 : ~23mn 00s
Bit rate mode                            : Variable
Bit rate                                 : ~ 4 000-5 000 Kbps
Maximum bit rate                         : 8 200 Kbps
Width                                    : 720 pixels
Height                                   : 480 pixels
Display aspect ratio                     : 4:3
Frame rate mode                          : Constant
Frame rate                               : 29.970 fps
Standard                                 : NTSC
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Interlaced
Scan order                               : Bottom Field First
Compression mode                         : Lossy
Bits/(Pixel*Frame)                       : 0.389
Stream size                              : 33.6 MiB (85%)
Default                                  : No
Forced                                   : No

However, starting in Season 3 I've encountered the occasional video MediaInfo like this:

Video
ID                                       : 1
Format                                   : MPEG Video
Format version                           : Version 2
Format profile                           : Main@Main
Format settings, BVOP                    : No
Format settings, Matrix                  : Custom
Codec ID                                 : V_MPEG2
Codec ID/Info                            : MPEG 1 or 2 Video
Bit rate mode                            : Variable
Bit rate                                 : 3 955 Kbps
Maximum bit rate                         : 9 800 Kbps
Width                                    : 720 pixels
Height                                   : 480 pixels
Display aspect ratio                     : 4:3
Frame rate mode                          : Variable
Original frame rate                      : 29.970 fps
Standard                                 : Component
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Scan order                               : Top Field First
Compression mode                         : Lossy
Default                                  : No
Forced                                   : No
Color primaries                          : BT.601 NTSC
Transfer characteristics                 : BT.601
Matrix coefficients                      : BT.601

I extracted the M2V from your useless MKV so I could run it through DGIndex with the Field Operation set for 'Honor Pulldown Flags' and then use MPEG2Source on it. This script works well to IVTC it and return it to 23.976fps:

TFM(D2V="Test.d2v").TDecimate(Mode=1)

Ouch . I used MakeMKV with my DVD's because I'm relatively familiar with .mkvs and haven't previously encountered any issues using them in my limited experience with MeGUI-AviSynth. Do you think that the above difference with the MediaInfos is a result of MakeMKV making a "mistake"? Do you reckon that all the episodes not playing nice need to be extracted and re-indexed with DGIndex in order to achieve proper results? As it's a hundred-some episodes that'd be a real shame, but certainly not the end of the world. Thanks for your help, manono .

[manono]

Ouch . I used MakeMKV] with my DVD's because I'm relatively familiar with .mkvs...

Live and learn.

Do you think that the above difference with the MediaInfos is a result of MakeMKV making a "mistake"?

No, the first frame of one was encoded as progressive and of the other interlaced. Unless it's 100% one way or the other those pieces of information given by MediaInfo are useless. What does MediaInfo say about the VOBs? A much better idea of what you have will be had by making a D2V and scrolling to the bottom to check the film percentage. Or run the 'Preview' in DGIndex.

Do you reckon that all the episodes not playing nice need to be extracted and re-indexed with DGIndex in order to achieve proper results?

'Not playing nice'? So you screwed up a bunch of them and you're asking if you should do them all over? Up to you.

[LouieChuckyMerry]

After a day of testing and retesting I'm more confused than I was. Here are four test clips, two from Season 3 Episode 11 (one from the original VOB file and one from the .mkv made by MakeMKV) and two from Season 3 Episode 12 (also one from the original VOB file and one from the .mkv made by MakeMKV); both episodes are on the same NTSC DVD:

1) S3.E11-TestClip[FromVOB]

2) S3.E11-TestClip[FromMkv]

3) S3.E12-TestClip[FromVOB]

4) S3.E12-TestClip[FromMkv]

If I index each of these four files with L-SMASH Works (I need to use LSW in order to use MT AviSynth; DGIndex causes problems) and run my script:

SetMemoryMax(256)
SetMTMode(3,3)
LoadPlugin("F:\[0]StandAloneApps\MeGUI-2500(core)2443(data)0.3.5(libs)[Portable]\tools\lsmash\LSMASHSource.dll")
LWLibavVideoSource("SourcePath.mkv")
### Deinterlace ###
SetMTMode(5)
LoadPlugin("F:\[0]StandAloneApps\MeGUI-2500(core)2443(data)0.3.5(libs)[Portable]\tools\avisynth_plugin\TIVTC.dll")
TFM(Order=0,Slow=2,PP=0).TDecimate(Mode=1)
SetMTMode(2)
Vinverse()
### Deshaker ###
Stab(Mirror=15)
### Crop ###
Crop(8,0,-8,0)
### Resize ###
RatioResize(10/11.0,"PAR")
### Gibbs Noise Block ###
Edge=MT_Edge("prewitt",thY1=20,thY2=40).RemoveGrain(17)
Mask=MT_Logic(Edge.MT_Expand().MT_Expand().MT_Expand().MT_Expand(),Edge.MT_Inflate().MT_Inpand(),"xor")
MT_Merge(dfttest(),Mask,Luma=True)
### Overall Temporal Denoise, Could Probably Be Optimized ###
SMDegrain(tr=2,thSAD=600,ContraSharp=True,RefineMotion=True,Plane=0,Lsb_Out=True,PreFilter=2)
### Debanding ###
GradFun3(thR=0.55,Radius=12,Mask=2,SMode=1,Lsb=True,Lsb_In=True, StaticNoise=True,Y=3,U=3,V=3)
DitherPost(Stacked=True,Prot=False,Mode=0)
### Line Darkener And Thinner ###
FastLineDarkenMod(Strength=20,Prot=6,Thinning=0)
aWarpSharp2(Blur=4,Type=1,Depth=3,Chroma=2)

the results are:

1) S3.E11-TestClip[FromVOB].m2v.mkv @ 23.976 fps

2) S3.E11-TestClip[FromMkv].mkv.mkv @ 19.951 fps

3) S3.E12-TestClip[FromVOB].m2v.mkv @ 23.976 fps

4) S3.E12-TestClip[FromMkv].mkv.mkv @ 23.976 fps


Further, if I demux the .m2v file from the S3.E11-TestClip[FromMkv].mkv file, index it with L-SMASH Works, and run my script the result is:


2) S3.E11-TestClip[FromMkv].m2v.mkv seemingly (MediaInfo reports in a different form than the three above @ 23.976 fps) @ 23.976, but the length of the clip has shrunk from ~30s to ~28s


Now, if I have to I'll take the time to dig through a hundred-some unlabeled VOB files, sort-label them, and index them with LSW, but I'd much prefer to be able to somehow demux-remux-something-anything the already sorted and labeled .mkv files created by MakeMKV and reindex them with L-SMASH Works. Unfortunately I'm not sure if this is possible.

Please, can someone--pretty please--explain to me how two .mkvs created by the same program from identical-type VOBs on the same DVD can possibly act so differently when indexed the exact same way then encoded using the exact same script? Is there some extra (wrong) information added to the bad file? An extra "bad" frame at the beginning? Bad karma? . Seriously, this has my head in knots, any clarity would be much appreciated.


Edit: I ran DGIndex for the full S3.E11 original VOB file and the original .m2v file extracted from the original MakeMKV .mkv file, just to compare. Both are "88.09% FILM" but other things are different. First the full VOB:

DGIndexProjectFile16
1
D:\Temp\S3-TestClipsTemp\S3.E11-Original[NTSC][VOB].VOB

Stream_Type=1
MPEG_Type=2
iDCT_Algorithm=6
YUVRGB_Scale=1
Luminance_Filter=0,0
Clipping=0,0,0,0
Aspect_Ratio=4:3
Picture_Size=720x480
Field_Operation=0
Frame_Rate=29970 (30000/1001)
Location=0,0,0,66d36

Too Much To Post

FINISHED  88.09% FILM

Second the full .m2v extracted from MakeMKV's .mkv:

DGIndexProjectFile16
1
D:\Temp\S3-TestClipsTemp\S3.E11-Original[NTSC][m2v].m2v

Stream_Type=0
MPEG_Type=2
iDCT_Algorithm=6
YUVRGB_Scale=1
Luminance_Filter=0,0
Clipping=0,0,0,0
Aspect_Ratio=4:3
Picture_Size=720x480
Field_Operation=0
Frame_Rate=29970 (30000/1001)
Location=0,0,0,5332d

Too Much To Post

FINISHED  88.09% FILM

[LouieChuckyMerry]

Seems that several of my files were corrupted; re-ripping the DVD's solved the problems.

[LouieChuckyMerry]

Turns out that the eighty-fifth episode I encoded, episode four of season five, suffers from what I think is called field blending. Unfortunately it appears that the problem is with the source material--S5.E4-TestClip[Source]--and not my script. Please, is there a way to fix this? If not, then is there at least a trick to minimize the blending, some super awesome plugin or filter for AviSynth? Thanks for any help.

[manono]

Yes, it has some field-blending. Try this:

Yadif(Mode=1)#or the better QTGMC
SRestore(Frate=23.976)

[jagabo]

Try this too:

Mpeg2Source("S5.E4-TestClip[Source].d2v", Info=3) 

Interleave(TFM(field=0, pp=0), TFM(field=1, pp=0))
TFM(field=0, pp=0, slow=2, cthresh=4)
TFM(field=1, pp=0, slow=2, cthresh=4)

TDecimate(mode=1, cycle=10, cycleR=6)

[LouieChuckyMerry]

Thank you both very much for your suggestions, I appreciate them .

manono: I've your Yadif(Mode=1)/SRestore(23.976) idea successfully queued and would like to try with QTGMC, what would you suggest for QTGMC settings?

jagabo: I've a test clip with your replacement script for TIVTC successfully queued and will know more in the morning.

[manono]

manono: I've your Yadif(Mode=1)/SRestore(23.976) idea successfully queued and would like to try with QTGMC, what would you suggest for QTGMC settings?

Under the theory that it's usually better to field match than to deinterlace, I'd now go with jagabo's script, or at least a variation of it. First, Yadif, especially when used in anime, leaves behind nasty artifacts, especially aliasing. QTGMC is slow, and unless you want to use it for its cleaning (and your source could use a little cleaning), I'd go with the field matching. Somehow jagabo's TFM voodoo seems to get rid of the field-blending all by itself. And it's much faster. I've saved it and hope to use it on some of my own partially field-blended projects. I do use QTGMC quite a bit with my own projects and usually use it as:

QTGMC(Preset="Fast")

I might decimate it differently but it's six of one, half dozen of another, pretty much:

Interleave(TFM(field=0, pp=0), TFM(field=1, pp=0))
TFM(field=0, pp=0, slow=2, cthresh=4)
TFM(field=1, pp=0, slow=2, cthresh=4)
BlendBob()
TDecimate(Mode=1)

I like that bobbing you did with this sample, jagabo.

[jagabo]

It was something I read a long time ago and forgot about:

https://forum.videohelp.com/threads/329754-Help-deinterlacing-IVTC-ing-VHS-material-con...=1#post2042633

With this source two TFMs seem to be enough:

TFM(field=1, mode=5, pp=1)
TFM(field=0, mode=5, pp=1)

Using mode=1 and/or pp=0 works too. It does leave an odd duplicate sequence of 4 unique frames followed by one duplicate, 5 unique frames followed by one duplicate, 3 unique frames followed by one duplicate.

[LouieChuckyMerry]

My day certainly hasn't gone as planned. Anyway, after testing both suggestions I think jagabo's idea looks better, especially checking frame-by-frame, but as I type I'm running a test with manono's updated suggestion. Thanks again, both of you.

It was something I read a long time ago and forgot about:

https://forum.videohelp.com/threads/329754-Help-deinterlacing-IVTC-ing-VHS-material-con...=1#post2042633

In the above linked thread you type "Yadif is a special case. It doesn't auto load like other filters... I have that in an AVSI file so it automatically loads every time an AVS script is opened." Is the .avsi file literally named something like "Jag'sAutoLoader" with a series of calls? This sounds very useful.

With this source two TFMs seem to be enough:

TFM(field=1, mode=5, pp=1)
TFM(field=0, mode=5, pp=1)

Using mode=1 and/or pp=0 works too. It does leave an odd duplicate sequence of 4 unique frames followed by one duplicate, 5 unique frames followed by one duplicate, 3 unique frames followed by one duplicate.

The emphasis on "this" is mine; are you typing of my source? If so, are you suggesting:

Interleave(TFM(field=0, pp=0), TFM(field=1, pp=0))
TFM(field=1, mode=5, pp=1)
TFM(field=0, mode=5, pp=1)
TDecimate(mode=1, cycle=10, cycleR=6)

instead of your original suggestion:

Interleave(TFM(field=0, pp=0), TFM(field=1, pp=0))
TFM(field=0, pp=0, slow=2, cthresh=4)
TFM(field=1, pp=0, slow=2, cthresh=4)

TDecimate(mode=1, cycle=10, cycleR=6)

Thanks again for your time.


Edit: manono's updated suggestion looks as good as jagabo's original suggestion (at least to my eyes on my 14" laptop screen), but it takes about 50% longer to run.

[jagabo]

My day certainly hasn't gone as planned. Anyway, after testing both suggestions I think jagabo's idea looks better, especially checking frame-by-frame, but as I type I'm running a test with manono's updated suggestion. Thanks again, both of you.

Blending pairs of identical (except for noise) frames together should give you some natural noise reduction. You can get even better noise reduction using Dup(blend=true). That will blend multiple identical (again, except for noise) frames together (up to 20 by default). You may have to play with the threshold value. Too high a threshold will sometimes blend truly different frames together. Too low a value and noise will appear as differences and prevent blending. Adding Dup() to my original script:

Interleave(TFM(field=0, pp=0), TFM(field=1, pp=0))
TFM(field=1, mode=5, pp=1)
TFM(field=0, mode=5, pp=1)
Dup(blend=true)
TDecimate(mode=1, cycle=10, cycleR=6)

It was something I read a long time ago and forgot about:

https://forum.videohelp.com/threads/329754-Help-deinterlacing-IVTC-ing-VHS-material-con...=1#post2042633

In the above linked thread you type "Yadif is a special case. It doesn't auto load like other filters... I have that in an AVSI file so it automatically loads every time an AVS script is opened." Is the .avsi file literally named something like "Jag'sAutoLoader" with a series of calls? This sounds very useful.

Put this in AviSynth's plugins folder as Yadif.avsi:

Load_Stdcall_plugin("C:\Program Files (x86)\AviSynth 2.5\plugins\yadif.dll")

Change the path to yadif.dll if necessary.

With this source two TFMs seem to be enough:

TFM(field=1, mode=5, pp=1)
TFM(field=0, mode=5, pp=1)

Using mode=1 and/or pp=0 works too. It does leave an odd duplicate sequence of 4 unique frames followed by one duplicate, 5 unique frames followed by one duplicate, 3 unique frames followed by one duplicate.

The emphasis on "this" is mine; are you typing of my source? If so, are you suggesting:

Interleave(TFM(field=0, pp=0), TFM(field=1, pp=0))
TFM(field=1, mode=5, pp=1)
TFM(field=0, mode=5, pp=1)
TDecimate(mode=1, cycle=10, cycleR=6)

instead of your original suggestion:

Interleave(TFM(field=0, pp=0), TFM(field=1, pp=0))
TFM(field=0, pp=0, slow=2, cthresh=4)
TFM(field=1, pp=0, slow=2, cthresh=4)

TDecimate(mode=1, cycle=10, cycleR=6)

No I meant you can skip the bob doubling with interleaved TFMs and just use a pair of TFMs. And since the video is no longer bob'd the decimation needs to change to 1 out of 5, or 2 out of 10. Or maybe 3 out of 15 because of the odd dup pattern.

TFM(field=1, mode=5, pp=1)
TFM(field=0, mode=5, pp=1)
TDecimate(mode=1, cycle=10, cycleR=2)

That might leave a little combing here and there. Adding "slow=2, cthresh=4" may help with that (that's why I had it in my original script). All of my scripts left one combed frame at a scene transition about 15.3 seconds into the sample video. I think this is from a problem in the video but I haven't checked it closely.

[jagabo]

By the way, if you're wondering how multiple TFMs can remove blended fields consider a sequence like:

[1T 1B] [1+2T 2B] [2T 2+3B] [3T 3B]

where brackets indicate the pairing of fields in the video frame, the numbers indicate the original film frames, T and B represent top and bottom fields, and N+M indicates a blended field from film frames N and M. So this sequence includes four TFF video frames, two of which are already progressive, and two of which look interlaced and contain a blended field.

Assume you start TFM matching top fields. The first pair is already progressive and represent film frame 1. But at the next top field there is no match because of the blending. Neither the 1B before it nor the 2B after it match. So that frame is left interlaced. The next top field 2T can be matched with the 2B before it so 2+3B is discarded and 2B is substituted in its place. Finally 3T and 3B are already paired. So after the first TFM we have:

[1T 1B] [1+2T, 2B] [2T 2B] [3T 3B]

Now at the second pass of TFM we attempt to match the bottom fields. [1T 1B] is already matched.. now at 2B it searches the field before and finds no match because that field is blended. So it looks at the next field and sees 2T is a match. So 1+2T is discarded and 2T is substituted. The rest of the frames are already progressive. Now we have all progressive frames:

[1T 1B] [2T 2B] [2T 2B] [3T 3B]

It's important to use pp=0 or pp=1 to prevent TFM from deinterlacing the fields without matches.

[ndjamena]

Does disabling post processing in TFM leave the output chroma in interlaced format?

[jagabo]

Does disabling post processing in TFM leave the output chroma in interlaced format?

I didn't think of that. But apparently not. I saw no chroma blending while viewing the video in VirtualDub. It might leave the chroma interlaced if there's an orphaned field. So it might be best to run TFM one last time with the default pp setting to deinterlace that field.

[LouieChuckyMerry]

Thanks for your very informative replies, jagabo, and your patience. I've now a solution for any other source-based field blending during this project (I've still 100+ episodes to encode) and a much better understanding of field blending in general (and possible future fixes). Plus, I've a jagabo'sYadifAutoLoadScript© .

[LouieChuckyMerry]

Does disabling post processing in TFM leave the output chroma in interlaced format?

I didn't think of that. But apparently not. I saw no chroma blending while viewing the video in VirtualDub. It might leave the chroma interlaced if there's an orphaned field. So it might be best to run TFM one last time with the default pp setting to deinterlace that field.

Do you mean something like this:

TFM(Field=0,Mode=5,PP=1,Slow=2,CThresh=4)
TFM(Field=1,Mode=5,PP=1,Slow=2,CThresh=4)
TFM(Order=0,Slow=2,CThresh=4)
TDecimate(Mode=1,Cycle=10,CycleR=2)

Sorry, I'm a bit slow.

[jagabo]

Does disabling post processing in TFM leave the output chroma in interlaced format?

I didn't think of that. But apparently not. I saw no chroma blending while viewing the video in VirtualDub. It might leave the chroma interlaced if there's an orphaned field. So it might be best to run TFM one last time with the default pp setting to deinterlace that field.

Do you mean something like this:

TFM(Field=0,Mode=5,PP=1,Slow=2,CThresh=4)
TFM(Field=1,Mode=5,PP=1,Slow=2,CThresh=4)
TFM(Order=0,Slow=2,CThresh=4)
TDecimate(Mode=1,Cycle=10,CycleR=2)

Yes, that should work. During the first two TFMs frames with no complimentary fields are passed through without being touched. During the third TFM() any of those interlaced frames will be deinterlaced.

Multiple TFMs like this will only work well when there are enough clean fields to reconstruct all the original film frames. You occasionally get videos where there aren't enough clean fields to reconstruct the original film frames. With the latter the usual "smart bob and srestore" method should work better.

[LouieChuckyMerry]

Yes, that should work. During the first two TFMs frames with no complimentary fields are passed through without being touched. During the third TFM() any of those interlaced frames will be deinterlaced.

Multiple TFMs like this will only work well when there are enough clean fields to reconstruct all the original film frames. You occasionally get videos where there aren't enough clean fields to reconstruct the original film frames. With the latter the usual "smart bob and srestore" method should work better.

Thanks for the reply, jababo. Of course, I've now encountered a couple of spots where I'm in need of "the usual 'smart bob and srestore'" but I'm not sure exactly what that is. I've searched about for "avisynth smart bob srestore" but don't find much, as SmartBob seems to be a VirtualDub Plugin (well, I reckon I could use VirtualDub if necessary, but would prefer sticking with MeGUI as I'm much more familiar with it). With this sample (field-blending begins at frame 893 and lasts for only a few frames) neither the multiple TFM nor the interleave-multiple TFM method works, and trying:

Yadif(Mode=1,Order=1)
SRestore(FRate=23.976)

also fails to improve the blending. Is there a, er, smarter way to attack? I'll search more tomorrow but now it's past my bedtime.

[jagabo]

A simple bob separates the two fields of a frame into two half height images then resizes them giving a double frame rate video at the original frame size -- for example, 720x480i30 becomes 720x480p60. A smart bob uses information from other fields when possible to fill in the missing lines. So Yadif() and QTGMC() are examples of smart bobs in AviSynth. There are several smart bobbers in VirtualDub, even the built in deinterlace filter includes Yadif with double frame rate.

I only saw a few blended fields in that sample video. With a video like that you can simply copy adjacent frames over them:

######################################################

function ReplaceFrameNext(clip Source, int N)
{
  # Replace frame at N with frame at N+1
  # N is the frame to replace
  # with frame at N+1

  loop(Source, 0, N, N)
  loop(last, 2, N, N)
}

######################################################

function ReplaceFramePrev(clip Source, int N)
{
  # Replace frame at N with frame at N-1
  # N is the frame to replace
  # with frame at N-1

  loop(Source, 0, N, N)
  loop(last, 2, N-1, N-1)
}

######################################################

Mpeg2Source("S5.E19-TestClip.d2v", Info=3) 
TFM(d2v="S5.E19-TestClip.d2v") 
TDecimate() 
ReplaceFrameNext(447)
ReplaceFrameNext(449)
ReplaceFrameNext(451)

Of course, to do it this way you have to manually locate all the blended frames.

[LouieChuckyMerry]

Happy Friday! After perusing the video in VirtualDub I came up with three blended frames and, incorporating your script into mine, this script:

# Set DAR in encoder to 3 : 2. The following line is for automatic signalling
global MeGUI_darx = 3
global MeGUI_dary = 2
SetMemoryMax(512)
SetMTMode(3,6)
LoadPlugin("F:\[0]StandAloneApps\MeGUI-2500(core)2443(data)0.3.5(libs)[Portable]\tools\lsmash\LSMASHSource.dll")
LWLibavVideoSource("D:\Temp\[0000]ReEncTemp\S5.E19-SweetSeymourSkinner'sBaadasssssSong[S][Commentary][480p]\[00]ParTemp\S5.E19-TestClip.mkv")
SetMTMode(2)
Function ReplaceFrameNext(Clip Source,int N)
{
Loop(Source,0,N,N)
Loop(Last,2,N,N)
 }
 Function ReplaceFramePrev(Clip Source,int N)
 {
 Loop(Source,0,N,N)
 Loop(Last,2,N-1,N-1)
 }
SetMTMode(5)
### Deinterlace ###
LoadPlugin("F:\[0]StandAloneApps\MeGUI-2500(core)2443(data)0.3.5(libs)[Portable]\tools\avisynth_plugin\TIVTC.dll")
TFM(Order=0,PP=0,Slow=2).TDecimate(Mode=1)
ReplaceFrameNext(558)
ReplaceFrameNext(561)
ReplaceFrameNext(563)
SetMTMode(2)
Vinverse()
### Deshaker ###
Stab(Mirror=15)
### Crop ###
Crop(8,0,-8,0)
### Resize ###
RatioResize(10/11.0,"PAR")
### Gibbs Noise Block ###
Edge=MT_Edge("prewitt",thY1=20,thY2=40).RemoveGrain(17)
Mask=MT_Logic(Edge.MT_Expand().MT_Expand().MT_Expand().MT_Expand(),Edge.MT_Inflate().MT_Inpand(),"xor")
MT_Merge(dfttest(),Mask,Luma=True)
### Overall Temporal Denoise ###
SMDegrain(tr=2,thSAD=600,ContraSharp=True,RefineMotion=True,Plane=0,Lsb_Out=True,PreFilter=2)
### Debanding ###
GradFun3(thR=0.55,Radius=12,Mask=2,SMode=1,Lsb=True,Lsb_In=True, StaticNoise=True,Y=3,U=3,V=3)
DitherPost(Stacked=True,Prot=False,Mode=0)
### Line Darkener And Thinner ###
FastLineDarkenMod(Strength=20,Prot=6,Thinning=0)
aWarpSharp2(Blur=4,Type=1,Depth=3,Chroma=2)

which runs fine but doesn't seem to change anything . Perhaps I've done something wrong?

Also, this is but a single, very short instance of source-based field-blending. There are several more in other episodes that are quite a bit longer, too long to make a frame-by-frame check practical. Is there any filter-plugin that could solve this problem, even if it's slow? This would be OK because I can cut out the short sections, encode them, then stitch them back into the final video without too much effort.

[jagabo]

What does "doesn't seem to change anything" mean? If you mean the blended frames are still there you have the wrong frame numbers. After TDecimate() the frame numbers should be around 447, 449 and 451. Different decimation settings may lead to slight differences so you should check.

SRestore() and FixBlendIVTC() are the two automated deblenders I know of. Neither of which works for this clip.

[LouieChuckyMerry]

What does "doesn't seem to change anything" mean? If you mean the blended frames are still there you have the wrong frame numbers. After TDecimate()(emphasis LouieChuckyMerry's) the frame numbers should be around 447, 449 and 451. Different decimation settings may lead to slight differences so you should check.

My apologies, I'd no idea it was the frame number(s) after processing that were needed. Once I changed the frame numbers in my script accordingly the results are great, thank you .

SRestore() and FixBlendIVTC() are the two automated deblenders I know of. Neither of which works for this clip.

That's a real shame. For whatever reason, the sources (from the DVD's) seem to be growing lower in quality as I progress from Season 1 to Season 11. Seriously, the source quality of Season 1 is noticeably higher than the source quality of Season 5. If there's no filter-plugin way to fix these occasional instances of blending, then I'll have to engage in some willful ignorance, perhaps closing one eye every time I notice blending...

[LouieChuckyMerry]

Hello again, and Happy Saturday . The source video has changed type starting with Season 8 and I was hoping for confirmation that I've the new deinterlacing script right. Here's a test clip of the new-type source video. For the first seven seasons the deinterlace line was:

TFM(Order=0,Slow=2,PP=0).TDecimate(Mode=1)

which worked very well (except for the infrequent exception). Given my limited (but growing ) knowledge, I think that the change in source video type requires:

TFM(Order=1,Slow=2,PP=0).TDecimate(Mode=1)

because it's changed to top field first but is still predominantly-film hybrid that's interlaced. I want to be certain, however, because MeGUI's "Analyse" option calls for "TDecimate(Mode=1,Hybrid=3)" when muxed as an .mkv then indexed with FFMS but calls for "TDecimate(Mode=1,Hybrid=1) when demuxed to .m2v then indexed with DGIndex .

Furthermore, running the test clip with the FFMS index-Hybrid=3 combo outputs video suffering from severe field-blending but running the DGIndex-Hybrid=1 test clip combo results in seeming fine video, as does running the test clip with the FFMS index-no Hybrid combo.

Sorry if this is unclear. I guess my basic question is: does the test clip warrant "TDecimate(Mode=1,Hybrid=1)" or is "TDecimate(Mode=1)" enough? As always thanks for your time .

[jagabo]

The pulldown pattern is very irregular at the start of the clip. But it decimates down to 23.976 fps smoothly with:

Mpeg2Source("S8.E2-TestClip.d2v", Info=3) 
TFM(d2v="S8.E2-TestClip.d2v") 
TDecimate(Cycle=30, CycleR=6)

You might be able to get away with "Cycle=15, CycleR=3" or even "Cycle=10, CycleR=2" with a longer clip.