[AviSynth] CombMask alternative in MaskTools 2?

[8day]

Does anyone know of any alternative to CombMask in MaskTools 2? I need a non-binary combing mask, but CombMask(thY1=0, thY2=255) seems to be broken and produce same results as CombMask(thY1=255, thY2=255) (in fact, same thing goes for EdgeMask). It seems that mt_luts() may be what I want, but I'm not sure.

[poisondeathray]

CombMask for Masktools2 doesn't use Thy1, Thy2; only the original CombMask for masktools (1) uses that syntax

CombMask for masktools2
https://github.com/chikuzen/CombMask/tree/master/avisynth

some compiled versions
http://forum.doom9.org/showthread.php?t=171018

[8day]

Didn't know there were CombMask for MaskTooks2... Still, I need combing mask that will be non-binary, to combine it with non-binary edge mask (like mt_edge("prewitt", thY1=0, thY2=255)). Without it I get less fluid, more abrupt results. I'd like to amplify high values and decrease small ones, instead of simply clipping them.

Nonetheless, thank you for trying to help poisondeathray.

[poisondeathray]

I get different results when using masktools1 with CombMask, with different thy1 and thy2 values. Same with EdgeMask. Maybe try downloading another dll ?

[8day]

Nope, tried 1.5.7 and got same old binary mask... Here's screenshots:

original


combmask(thy1=0, thy2=255)


edgemask(thy1=0, thy2=255)

[8day]

Ok, so if someone will want something like this, just use this snippet:
sf = org.SeparateFields()
e = sf.SelectEvery(2, 0)
o = sf.SelectEvery(2, 1)
Overlay(e, o, mode="difference")
BilinearResize(org.width, org.height)
Levels(127, 1.000, 255, 0, 255).Inpand().Greyscale() # Inpand() is a function from MaskTools.

[8day]

Ok, so I found the reason why I had binary comb mask: binarization was embedded into source code (http://manao4.free.fr/MaskTools-v1.5.8.zip). To fix it, comment out MMX optimization function CM_MMX() in CM_func.h (I'm not familiar with Assembly, so I cannot provide working code for it as well), then change these lines in CM_func.cpp:

if ( prod < thresinf ) *d = 0;
else if ( prod > thressup ) *d = 255;
else *d = (prod >> 8);

to the ones below:

if ( prod < thresinf ) *d = prod < 0 ? 0 : prod;
else if ( prod > thressup ) *d = prod > 255 ? 255 : prod;
else *d = (prod >> 8);

Of course "prod < 0 ? 0 : prod" may be too much, but I didn't look at the code so good so that to know for sure that "prod" cannot go outside of 8 bit uint.

For those in need I have attached custom DLLs with the above fix (dynamically and statically linked; dynamically linked version relies on MSVCR80.dll from Microsoft Visual C++ 2005 Redistributable Package). I'm not a pro, so don't expect it to be a 100% safe: all I did is opened project in Visual Studio 2005 (with the aid of automatic conversion tool), then fixed those lines and pressed "Build".