Simple MDegrain Mod v3.1.2d - A Quality Denoising Solution

[LouieChuckyMerry]

Hi Dogway, I hope you're well . I was wondering if SMDegrain with this call:

SMDegrain(TR=0,ThSAD=000,ContraSharp=True,RefineMotion=True,Plane=0,Lsb=True,Lsb_Out=True,PreFilter=2,Chroma=False)

would literally do no denoising, or if SMDegrain has at least a bit of denoising as a byproduct of being used (like QTGMC)? Thanks.

Also: I bumped into the SMDegrain Avisynth wiki and noticed that the version is 3.0d, in case you wanted to update it.

[Dogway]

What's your goal with that call, you want to sharpen?

I don't have any relation with the avisynth wiki but in any case I'm going to update to v3.2 soon if nothing is done after my find on the mvtools2 bug. EDIT: cretindesalpes fixed it already, looks like a ditherpost issue, so for now no v3.2.

[LouieChuckyMerry]

What's your goal with that call, you want to sharpen?

Actually, the script you helped me with some months ago, for the early season Simpsons, was eventually "finalized" as:

LoadPlugin("F:\[0]StandAloneApps\MeGUI-2500(core)2443(data)0.3.5(libs)[Portable]\tools\DGIndexNV\DGDecodeNV.dll")
DGSource("SourcePath")
### Deinterlace-Match Fields-Decimate ###
LoadPlugin("F:\[0]StandAloneApps\MeGUI-2500(core)2443(data)0.3.5(libs)[Portable]\tools\avisynth_plugin\TIVTC.dll")
Function FieldMatch(Clip C) {
  Global PP = C.DuplicateFrame(0)
  Global CC = C
  Global NN = C.DeleteFrame(0)
  P2 = PP.SeparateFields()
  C2 = CC.SeparateFields()
  N2 = NN.SeparateFields()
  Global PC = Interleave(P2.SelectEven(),C2.SelectOdd()).Weave()
  Global CP = Interleave(C2.SelectEven(),P2.SelectOdd()).Weave()
  Global CN = Interleave(C2.SelectEven(),N2.SelectOdd()).Weave()
  Global NC = Interleave(N2.SelectEven(),C2.SelectOdd()).Weave()
  Global Deint = QTGMC(CC).SelectEven()
  Return ScriptClip(CC, \
    "!CC.IsCombedTIVTC(CThresh=12,Chroma=True,BlockX=16,BlockY=32) ? CC : " + \
    "!NN.IsCombedTIVTC(CThresh=12,Chroma=True,BlockX=16,BlockY=32) ? NN : " + \
    "!CN.IsCombedTIVTC(CThresh=12,Chroma=True,BlockX=16,BlockY=32) ? CN : " + \
    "!NC.IsCombedTIVTC(CThresh=12,Chroma=True,BlockX=16,BlockY=32) ? NC : " + \
    "!PP.IsCombedTIVTC(CThresh=12,Chroma=True,BlockX=16,BlockY=32) ? PP : " + \
    "!CP.IsCombedTIVTC(CThresh=12,Chroma=True,BlockX=16,BlockY=32) ? CP : " + \
    "!PC.IsCombedTIVTC(CThresh=12,Chroma=True,BlockX=16,BlockY=32) ? PC : Deint")
}
TFM(Order=-1,Mode=5,PP=2,Clip2=FieldMatch(),Slow=2,MChroma=False,Ubsco=False,CThresh=12,Chroma=True)
TDecimate(Mode=1)
### Fix Line-Doubled Fields ###
NNEDI3(Field=-2)
Merge(SelectEven(),SelectOdd())
### Reduce Shimmering ###
QTGMC(InputType=1)
### Stabilize ###
Stab(Mirror=15)
### Crop ###
Crop(8,0,-8,0)
### Gibbs Noise Block ###
Edge=MT_Edge("prewitt",ThY1=20,ThY2=40).RemoveGrain(17)
Mask=MT_Logic(Edge.MT_Expand().MT_Expand().MT_Expand().MT_Expand(),Edge.MT_Inflate().MT_Inpand(),"xor")
MT_Merge(DFTTest(),Mask,Luma=True)
### Overall Temporal Denoise ###
SMDegrain(TR=2,ThSAD=400,ContraSharp=True,RefineMotion=True,Plane=0,Lsb=True,Lsb_Out=True,PreFilter=2,Chroma=False)
### Resize ###
LinearResize(640,480,Lsb_In=True,Lsb_Out=True)
### Darken-Thin Lines ###
F=DitherPost(Mode=-1)
S=F.FastLineDarkenMod(Strength=20,Prot=6).aWarpSharp2(Blur=4,Type=1,Depth=3,Chroma=2)
D=MT_MakeDiff(S,F).Dither_Convert_8_To_16()
Dither_Add16(Last,D,Dif=True,U=2,V=2)
### Deband ###
GradFun3(thR=0.55,SMode=2,Lsb_In=True,Lsb=True,StaticNoise=True)
### Preview Source OR Send 16-bit Output To x264 10-bit ###
# DitherPost()
Dither_Out()

which outputs very nice results, all things considered. Then recently I was reminded that QTGMC denoises by default, which is making me reconsider the SMDegrain denoise settings as being, at best, redundant, or, at worst, overkill. Probably I think too much.

I don't have any relation with the avisynth wiki but in any case I'm going to update to v3.2 soon if nothing is done after my find on the mvtools2 bug. EDIT: cretindesalpes fixed it already, looks like a ditherpost issue, so for now no v3.2.

Strange to realize that a wiki page can be made without someone's consent, good thing I'm not famous . I'll test the new Dither.dll this weekend; I hope it solves the problem.

Thanks for your help.

[Dogway]

If you want my opinion QTGMC is probably overdone. But due to the nature of the content (flat colors without textures/shades) you can probably be allowed. It's just that you might be reinterpreting the content lineart with so many antialiasers and filters, probably okey since the source was so crappy...

In your case I would play it simple, remove everything and replace with KNLMeansCL spatio-temporal, it works very nice with these kind of sources (similar but better effect than "surface blur" in Photoshop). Then resize, lineart darkening, thinning, and gradfun if you want. I don't like how KNLMeansCL blurs/damages lineart but in this case you are dealing with gibbs noise, darkening, thinning... so it might be fine.

[LouieChuckyMerry]

Happy Friday . I've run multiple tests with the above script as is, with it minus the QTGMC, with it minus the NEEDI3, and with it minus QTGMC and NEEDI3, and the results always look better with the entire script. S5.E21-TestClip is a prime example. Without the QTGMC the shimmering is almost unbearable. Presently I'm unable to use KNLMeansCL (there's a problem with the OpenCL compiler), but when I can I'd be curious. Are you suggesting:

LoadPlugin("F:\[0]StandAloneApps\MeGUI-2500(core)2443(data)0.3.5(libs)[Portable]\tools\DGIndexNV\DGDecodeNV.dll")
DGSource("SourcePath")
### Deinterlace-Match Fields-Decimate ###
LoadPlugin("F:\[0]StandAloneApps\MeGUI-2500(core)2443(data)0.3.5(libs)[Portable]\tools\avisynth_plugin\TIVTC.dll")
Function FieldMatch(Clip C) {
  Global PP = C.DuplicateFrame(0)
  Global CC = C
  Global NN = C.DeleteFrame(0)
  P2 = PP.SeparateFields()
  C2 = CC.SeparateFields()
  N2 = NN.SeparateFields()
  Global PC = Interleave(P2.SelectEven(),C2.SelectOdd()).Weave()
  Global CP = Interleave(C2.SelectEven(),P2.SelectOdd()).Weave()
  Global CN = Interleave(C2.SelectEven(),N2.SelectOdd()).Weave()
  Global NC = Interleave(N2.SelectEven(),C2.SelectOdd()).Weave()
  Global Deint = QTGMC(CC).SelectEven()
  Return ScriptClip(CC, \
    "!CC.IsCombedTIVTC(CThresh=12,Chroma=True,BlockX=16,BlockY=32) ? CC : " + \
    "!NN.IsCombedTIVTC(CThresh=12,Chroma=True,BlockX=16,BlockY=32) ? NN : " + \
    "!CN.IsCombedTIVTC(CThresh=12,Chroma=True,BlockX=16,BlockY=32) ? CN : " + \
    "!NC.IsCombedTIVTC(CThresh=12,Chroma=True,BlockX=16,BlockY=32) ? NC : " + \
    "!PP.IsCombedTIVTC(CThresh=12,Chroma=True,BlockX=16,BlockY=32) ? PP : " + \
    "!CP.IsCombedTIVTC(CThresh=12,Chroma=True,BlockX=16,BlockY=32) ? CP : " + \
    "!PC.IsCombedTIVTC(CThresh=12,Chroma=True,BlockX=16,BlockY=32) ? PC : Deint")
}
TFM(Order=-1,Mode=5,PP=2,Clip2=FieldMatch(),Slow=2,MChroma=False,Ubsco=False,CThresh=12,Chroma=True)
TDecimate(Mode=1)
### Fix Line-Doubled Fields ###
NNEDI3(Field=-2)
Merge(SelectEven(),SelectOdd())
### Stabilize ###
Stab(Mirror=15)
### Crop ###
Crop(8,0,-8,0)
### Overall Spatio-Temporal Denoise ###
KNLMeans(??????,Lsb_InOut=True)
### Resize ###
LinearResize(640,480,Lsb_In=True,Lsb_Out=True)
### Darken-Thin Lines ###
F=DitherPost(Mode=-1)
S=F.FastLineDarkenMod(Strength=20,Prot=6).aWarpSharp2(Blur=4,Type=1,Depth=3,Chroma=2)
D=MT_MakeDiff(S,F).Dither_Convert_8_To_16()
Dither_Add16(Last,D,Dif=True,U=2,V=2)
### Deband ###
GradFun3(thR=0.55,SMode=2,Lsb_In=True,Lsb=True,StaticNoise=True)
### Preview Source OR Send 16-bit Output To x264 10-bit ###
# DitherPost()
Dither_Out()

What would you recommend for the KNLMeans settings, given that test clip? And would this cause you to change the FastLineDarkenMod and-or GradFun3 settings?

[Dogway]

QTGMC on progressive for denoising purposes is the poor man's way to denoise, it antialias and smudges everything so much that it's easy for non-experienced to like, but it's the wrong tool for the job, in this case (due to being animation) you are creating ghosting artifacts, also having a NNEDI3 call above (which looks necessary for the double lines) is enough antialias for lineart, so adding QTGMC on top (NNEDI3 based) is further destroying details. I didn't see any shimmering, and also find that the stab() call is unnecesary (it barely shakes), so you are probably doing more harm than good. You can use KNLMeansCL, it will even fix some small jitter and will get rid of gibbs noise. If you can't, SMDegrain (+the Gibbs Noise block) gives out good results as well, although probably KNLmeans would run fairly faster since it runs on GPU. You can use this call: KNLMeansCL(D=1, A=1, h=5.0,lsb_inout=true,device_type="GPU")

[LouieChuckyMerry]

Happy Saturday! I understand your valid points about QTGMC ("surgery with a hammer" ). Perhaps "shimmering" is the improper term; if I run S5.E21-TestClip with the full script minus QTGMC all the lines look like they're alive (aliasing?). Is there any way to minimize this other than QTGMC? As for the Stab, the first couple seasons shake quite a bit, so I'll pay better attention and only use it when necessary. Also, it seems I can run KNLMeansCL on my newer laptop so I'll check that out ASAP, thanks for the call suggestion.

Edit: I ran some tests and, as usual, you're right. KNLMeansCL with your suggested settings looks the same to my eyes as the Gibbs Block-SMDegrain combo and once I moved to viewing distance from my screen (instead of typing distance) the output video is noticeably crisper without the QTGMC call (and the aliasing is much less bothersome). Plus, the new way is three (3) times faster than the old way. Thanks again for all your help Dogway . In case someone searching finds their way here, full details of the new script are here.

[Lynx_TWO]

Good Morning!

I am currently attempting to use this with StaxRip x64 and Vapoursynth.

clip = havsfunc.SMDegrain(input = clip)

The above works as default settings, but when I attempt to add any arguments, such as:

clip = havsfunc.SMDegrain(input = clip, contrasharp=true,refinemotion=true,lsb=true)

I get an error:

Python exception: name 'true' is not define
Traceback (most recent call last):
File "vapoursynth.pyx", line 1467, in vapoursynth.vpy_evaluateScript (src\cython\vapoursynth.c:24719)

I am brand new to VapourSynth, so thanks for any help!

[Dogway]

It's likely what it's failing is the implementation (or the way you call it).
SMDegrain in an avisynth function so maybe ask in the havsfunc thread for the vapoursynth version.

[poisondeathray]

The reason is vapoursynth is very picky about case sensitivity, punctuation

For example , it has to be RefineMotion=True, not refinemotion=true

[Lynx_TWO]

The reason is vapoursynth is very picky about case sensitivity, punctuation

For example , it has to be RefineMotion=True, not refinemotion=true

Yep, that was exactly it. Also, Lsb gives an error because:

The VapourSynth version doesn't use these parameters; it automatically outputs the same bitdepth as the input.

So, this works:

clip = havsfunc.SMDegrain(input = clip,tr=6,contrasharp=True,RefineMotion=True)

I'll test a bunch of setting and then post the code-blocks for anyone else to use

[real.finder]

hi

for subpixel=3 and according to mvtools2 manual it say

pelclip clip =
Optional upsampled source clip for using instead of internal subpixel interpolation (for pel>1).
Pixels at rows and colunms positions multiple to pel (0,2,4,... for pel=2) (without padding) must be original source pixels, other pixels must be interpolated.

Example for pel=2: LanczosResize(width*2,height*2,src_left=0.25, src_top=0.25).

and in SMDegrain it use nnedi3_rpow2(rfactor=pel,cshift="spline16resize",n ns=4)

nnedi3_rpow2(rfactor=2) without cshift is already top-left aligned and cshift="spline16resize" will not leave it top-left aligned

and for rfactor=4 according to SEt it should be nnedi3_rpow2(rfactor=2).nnedi3_rpow2(rfactor=2)

[Dogway]

I need more information. I don't know why I implemented spline16 as cshift since I did not document that in the changelog, it happened in v3.0d and I think it's because before that I wasn't doing anything at all (from documentation -no chroma shift correction is performed when cshift is blank-), my choice for spline16 is due to speed.

If jpsdr's version is working as nice as it claims it shouldn't be introducing any of the problems you describe (?). I fixed SMDegrain's documentation though to link to the jpsdr's version.

[real.finder]

I need more information. I don't know why I implemented spline16 as cshift since I did not document that in the changelog, it happened in v3.0d and I think it's because before that I wasn't doing anything at all (from documentation -no chroma shift correction is performed when cshift is blank-), my choice for spline16 is due to speed.

If jpsdr's version is working as nice as it claims it shouldn't be introducing any of the problems you describe (?). I fixed SMDegrain's documentation though to link to the jpsdr's version.

jpsdr fix cshift, and without cshift the result will be shift up and left for both luma and chroma (it will be top-left aligned)

so without cshift jpsdr and original one is almost the same

[real.finder]

                 (prefilter==3)  ?  (!planar && lsb ? Dither_merge16_8(inputP.Dfttest(sstring="0.0:4.0 0.2:9.0 1.0:15.0",tbsize=1,u=chroma,v=chroma,lsb=true,lsb_in=lsb_in,quiet=true), lsb_in?inputP:inputP.Dither_convert_8_to_16(),                                \
                                                                   lsb_in?inputP.Dither_lut16("x 4096 < 65535 x 19200 > 0 65535 x 4096 - 65535 19200 4096 - / * - ? ?",u=1,v=1).Ditherpost(mode=6, slice=slices, u=Chr,   v=Chr)                                           \
                                                                         :inputP.mt_lut(      "x 16 < 255 x 75 > 0 255 x 16 - 255 75 16 - / * - ? ?",u=1,v=1), luma=chroma,                       u=Chr2,  v=Chr2)                                                       : \
                                                             mt_merge( planar ?  Dfttest(!lsb_in?inputP.Planar2Interleaved(!chroma):input8y,sstring="0.0:4.0 0.2:9.0 1.0:15.0",tbsize=1,u=chroma,v=chroma,dither=1).Interleaved2Planar(!chroma)                          : \
                                                                                 Dfttest(        inputP,                                    sstring="0.0:4.0 0.2:9.0 1.0:15.0",tbsize=1,u=chroma,v=chroma,dither=1),!lsb_in?input8y:input8y.Interleaved2Planar(!chroma),   \
                                                                          input8y.mt_lut(      "x 16 < 255 x 75 > 0 255 x 16 - 255 75 16 - / * - ? ?",u=1,v=1), luma=planar?false:chroma,          u=bug_wa,v=bug_wa))                                                   : \

I think it should be "mt_merge( !planar..." or there is something I missed?

edit: after I see this "planar = IsYUY2(input) ? true : false" it shouldn't, so never mind what I said about that

anyway, I have some suggestions

1. accept lsb clip in the prefilter as http://pastebin.com/SMiMiaUi did
2. dispense masktools for 2.5 and make adjustments to support masktools for 2.6 as http://pastebin.com/2feAt0NF did

thanks

[Dogway]

Thanks for the notes.

1. MAnalyse doesn't accept stacked 16-bit so allowing 16-bit prefilter clips is not only misleading but it gives the impression you are gaining more quality when in fact it's not (it would be dithered down to 8-bit either way), moreover when I'm planning to convert the internal luma expansion function in an optional feature. Doing this only adds to confusion.

2. masktools for 2.6 (you mean tp7's?) isn't different than for 2.5, it only has speed improvements and supports more color format inputs, is this what you mean?

[real.finder]

Thanks for the notes.

1. MAnalyse doesn't accept stacked 16-bit so allowing 16-bit prefilter clips is not only misleading but it gives the impression you are gaining more quality when in fact it's not (it would be dithered down to 8-bit either way), moreover when I'm planning to convert the internal luma expansion function in an optional feature. Doing this only adds to confusion.

2. masktools for 2.6 (you mean tp7's?) isn't different than for 2.5, it only has speed improvements and supports more color format inputs, is this what you mean?

1. yes but it make sense as you use Dither_Luma_Rebuild, so just add lsb_in=true in Dither_Luma_Rebuild if input8 height =\= prefilter clip height or if prefilter clip height = 2*input8 height

2. masktools for 2.6 was there before tp7 edit, and it necessary for Dither tools in avs 2.6

[real.finder]

the main different is masktools for 2.6 didn't supports yuy2 at all so you get error message with yuy2, so to make supports for masktools for 2.6 you must do some workaround and convert yuy2 to yv16 and then back it to yuy2, convert from yuy2 to yv16 or yv16 to yuy2 is lossless and fast

[Dogway]

1. I am going to make Dither_Luma_Rebuild() optional in v3.2d, and as said such improvement for lsb_in prefilter is negligible for the added confusion (one more parameter, etc). If you read the note on v3.1.2d's prefilters block you can already see I'm doing concesions for simplicity sake, mt_merge should never be used (it has inherent limitations that e.g. Dither_merge16() doesn't) but I don't want to force anyone to download plugins out of the box. Prefilters are rough approximations after all not surgical precision work. If all you want is to parse the stacked 16-bit prefilter to Dither_Luma_Rebuild() wait for the next version where you can disable the internal luma expansion and do both externally (ie. pre=prefilter(lsb_out=true).Dither_Luma_Rebuild(ls b_in=true) )

2. I see. Yes I noticed YUY2 is not supported anymore on latest versions. As noted in the documentation I recommended to use old Vit's 2.5 version for this color format. This is a pain to change because I must ensure format consistency through the whole pipeline, so I will delay it to v3.3d. I want to put out v3.2 sooner than later that's why.

[Khanattila]

I added support to fast chroma processing (YV24 required) in the v0.6.* brunch of KNLMeansCL.
Maybe you want to take advantage of this.

[Dogway]

Yes I noticed, I haven't introduced it for that very same reason (chroma resize).
In any case I have to do the current chroma plane work for YUY2 so it's not going to make the script any leaner.

[real.finder]

in the next ver. I think it will be better if you add tv_range bool

so you can deal with such cases https://forum.videohelp.com/threads/371283-Random-avisynth-grammar-questions?p=2408582&...=1#post2408582 correctly

[Dogway]

ColorYUV(levels="TV->PC") didn't fix the issue, the only solution is to use Dither_merge16_8() but that forces everyone to download one more plugin, which I don't want since it's only a tiny prefilter issue for dfttest (prefilter=3). Also the issue in question is going to be no more in case you use the autogain luma expansion I plan to add for next version.

[Skiller]

Would adding YV16 support be any easier than fixing YUY2 support?
If yes why not just drop YUY2?

[Dogway]

YUY2 is not going to be fixed because all developers are dropping this format, implementing yv16 is not difficult only time consuming, I have to go through all the code. I will include the newer yv16 approach after I give it a look, although mvtools still doesn't support it natively so a similar hacky approach seems to be in need.

[real.finder]

if all needed plugins support yv16 it will work with no workaround, even if you do workaround to resupport YUY2 back

the YUY2 workaround need only in masktools, other plugins still support YUY2 if planar=true

I use tp7 plugins with http://pastebin.com/2feAt0NF in YUY2 and it work well

SMDegrain work with YUY2 in avs 2.6 if I use masktools2-25 (from tp7 too), but that annoying and make some filters that need masktools2-26 not work like Dither_convert_yuv_to_rgb()

[Reel.Deal]

YUY2 is not going to be fixed because all developers are dropping this format, implementing yv16 is not difficult only time consuming, I have to go through all the code. I will include the newer yv16 approach after I give it a look, although mvtools still doesn't support it natively so a similar hacky approach seems to be in need.

MVTools does support regular YUY2 though. So to process YV16 just use ConvertToYUY2() for MVTools2 and ConvertToYV16() after. Is not as elegant as native support but It's much less hacky than SSE*Tools's InterleavedToPlanar/PlanarToInterleave. Doing it this way means you would no longer be depandant on the old MaskTools2 for AviSynth 2.5 and SSE*Tools would no longer be needed. Regarding performance, I'm not sure if there's any substantial differences between the 2 methods. Would be interesting to find out.

mt_merge should never be used (it has inherent limitations that e.g. Dither_merge16() doesn't)

I think 'never' is a bit strong, I still use on trivial things. Anyways, tp7 contemplated fixing this issue but ultimately decided "masktool's approximation is reasonable." A possible workaround was suggested here and there's also MaskCL which uses CLExpr.

[Dogway]

Thanks for the input both, currently I'm finishing a project then I will deal with script updates and all, and your suggestions will come handy.

On the YV16 update I guess I can YUY2/YV16 interchangeably but there's more to it than that, I deal with stacked 16-bit YUY2 and filters that need interleaved or planar, so I have to check what is in need for each case and test. In theory it sounds good... removing dependency filters, etc also as I deem AVS 2.6 mature enough that I can legitely "force" SMDegrain to be compatible only on latest versions...

As for the 255 mask thing it isn't currently used at any meaningful place in SMDegrain. But it will for 3.2d so I might think on something, overlay (used in MaskCL, which in turn needs CLExpr) is a filter that I don't like much... so I could even make Dither a mandatory plugin. My opinion is that it is a critical issue 8albeit mathematically correct...) since I target so much for quality, or at least consistency, working around all limitations/bugs so you don't have to. mt_merge will still be used on said trivial areas (show panel, prefilter...), just not on direct output.

[Reel.Deal]

On the YV16 update I guess I can YUY2/YV16 interchangeably but there's more to it than that, I deal with stacked 16-bit YUY2 and filters that need interleaved or planar, so I have to check what is in need for each case and test. In theory it sounds good... removing dependency filters, etc also as I deem AVS 2.6 mature enough that I can legitely "force" SMDegrain to be compatible only on latest versions...

If I'm not mistaken all plugins required by SMDegrainMod support either YUY2 or YV16 so you should be good to go. It probably would be easier (scripting wise) to internally process YUY2 as YV16 and then just use ConvertToYUY2() at the very end (only if input was YUY2). Maybe you can even add support for Y8 by disabling all chroma processing, just food for thought . Chikuzen released a brand new plugin today that might be worth a look. It's called PlanarTools and it claims (haven't tried it yet) to be faster than the internal AviSynth interleaved to planar functions. I think making AviSynth 2.6 mandatory is a wise choice (less variables to worry about). 2.6 has been considered more stable than 2.58 for some time now and now that 2.6 is the official stable release it should be the obvious thing to do.

overlay (used in MaskCL, which in turn needs CLExpr) is a filter that I don't like much...

I'm not sure if I understanding correctly but MaskCL does not use Overlay(), overlay is just the clip name in the script.

----

Edit:

The reason is vapoursynth is very picky about case sensitivity, punctuation

For example , it has to be RefineMotion=True, not refinemotion=true

Fow lowercase use core = vs.get_core(accept_lowercase=True).

[Dogway]

@Reel.Deal: sorry I just gave it a quick glance and mistook the variable "overlay" for the actual internal filter "overlay", expecting it to do some mt_merge kind of work as I saw on other linked examples. In any case the goal is to demand less and less external plugins, at this point I think to ask for mvtools, masktools and Dither is not much of a query. So leveraging the Dither tools, the easier and logic approach would be to simply use Dither_merge16().

Y8 is not directly supported (mvtools can't support), but SMDegrain can work in the luma plane exclusively turning chroma off in all places it can be turned off (prefilters,mvtools,dithering,etc). I think this should show speed wise.