need assistance with creating a line mask for a cartoon

[Betelman]

Hi all. I think I'm losing my Midas touch here big time. I used to have this really good script that allowed me to protect lines and details while filtering flat areas and well...no idea where it went. And I can't remember for the life of me how to create one. This was years ago- but I'll be more careful now. Anyway, this is the script I have (the color orange is just to see where the actual filter would be applied. It's for testing purposes)

MPEG2Source("blah")
Assumetff()
TFM()
TDecimate(mode=1)

a=Last
b=blankclip(a, color=color_orange)
mask1=mt_binarize()
mask2=mt_edge("cartoon")
Overlay(mask1, mask2)

And below is what it's producing. Again, I'm simply attempting to protect my lines and filter the rest. Can anyone lend a hand here? Thanks

[jagabo]

As usual, post a sample of the video.

One thing to note: mt_edge() and mt_binarize() have threshold values you can play with. Also, you can expand the mask with mt_expand() and inpand with mt_inpand(). It can be helpful to blur the mask too -- to soften the edge between process an unprocessed areas.

[Atak_Snajpera]

MPEG2Source("blah")
Assumetff()
TFM()
TDecimate(mode=1)
video=last

Loadplugin("Plugins_JPSDR.dll")
LevelLimit=(video.BitsPerComponent==8) ? 255 : 1023
IntensityMask=ConvertToY(video).Levels(0,2,LevelLimit,0,LevelLimit,coring=false)
EdgeMask=aSobel(IntensityMask,chroma=0,thresh=255).invert.Levels(0,2,LevelLimit,0,LevelLimit,coring=false).Blur(1)
FilteredVideo=invert(video) #USE YOUR OWN FILTERING HERE. INVERT IS JUST AN EXAMPLE!
video=Overlay(video,FilteredVideo,mask=EdgeMask,opacity=1.0)
return video

[Betelman]

As usual, post a sample of the video.

Done.

One thing to note: mt_edge() and mt_binarize() have threshold values you can play with. Also, you can expand the mask with mt_expand() and inpand with mt_inpand(). It can be helpful to blur the mask too -- to soften the edge between process an unprocessed areas.

Yes, it all sounds very nice. Can you please show me how that would look like on a script please? As I stated, I forgot how to do it.

Atak, I do appreciate your help but your script is not exactly what I'm trying to do here. And it doesn't even load.

[jagabo]

First, a note on your script in the first post:

mask1=mt_binarize()
mask2=mt_edge("cartoon")
Overlay(mask1, mask2)

You are creating two masks, mask1 and mask2. Then you overlay mask2 into mask1, thereby overwriting mask1 with mask2. The result of Overlay() is simply mask2.

Here's one way to animate the mask creation with mt_edge() so you can quickly determine what values you want to use:

function anim_mask(clip c, int thresh)
{
    mt_edge(c, "hprewitt", thy1=thresh, thy2=thresh)
    Subtitle(string(thresh)) # show threshold value
}

AviSource("bridge.avi")  # RGB source?
ConvertToYV12()
Trim(22,0).Loop(100,0,0) # only frame 22, repeated 100 times

emask = Animate(0,100, "anim_mask", last,0, last,100).GreyScale()
return(emask)

That will show you the result of mt_edge() (for frame 22 only) over a range of 0 to 100 for thy1 and thy2. Try the different modes to see what the differences are (sobel, roberts, etc).

http://avisynth.nl/index.php/MaskTools2/Mt_edge

mt_expand() can expand the mask to fill in small holes and cover areas near edges. You can soften the mask (to make the transition from masked to not masked areas less obvious:

function anim_mask(clip c, int thresh)
{
    mt_edge(c, "hprewitt", thy1=thresh, thy2=thresh)
    Subtitle(string(thresh)) # show threshold value
}

AviSource("bridge.avi")  # RGB source?
ConvertToYV12()
Trim(22,0).Loop(100,0,0) # only frame 22, repeated 100 times

emask = Animate(0,100, "anim_mask", last,0, last,100).mt_expand().Blur(1.0).GreyScale()
return(emask)

[Attachment 59909 - Click to enlarge]

Conversely, you can use mt_inpand() to thin the mask and remove small dots.

[Betelman]

Ok, yes now I remember. Awesome- I'll try this out. Thanks.