RemoveDirtMC confused by fast lights/blowing wind

[skh]

hello,
I'm having problems dirt filtering specific scenes. It seems that RemoveDirtMC is somewhat confused by faster movements in the style of flickering but not flickering as a deffective video more like the wind is blowing or the light is flashing.
I also tried Despot and spotless but they worked far worse than the various variants of Removedirt.
Maybe some of the experts here who are very good in understanding special configurations of functions could help. I would be very grateful.

here is a sample: https://www.transfernow.net/dl/20231014rCKBNzmE


and here are 2 screenshots of the issues which are caused


[Attachment 74335 - Click to enlarge]

[Attachment 74336 - Click to enlarge]

thank you very much

[jagabo]

You'll find all motion compensated filters are confused by such things. Is that video before or after RemoverDirtMC?

You can sometimes get less damage and more spot removal by calling the filter more than once but with milder settings.

Or, if there is something about the spots that can be used to identify them, use a mask to protect other parts of the picture. For example, if the spots are always black, create a mask of black areas and use Overlay() to apply RemoveDirtMC only to black areas.

Another trick is to limit the spot removal to only the parts of the picture where big changes were made:

# absolute value of v1 - v2, luma only, planar format required by mt_lutxy
function AbsSubtractY(clip v1, clip v2)
{
    mt_lutxy(v1, v2,"x y - abs", chroma="-128")
}

LWLibavVideoSource("RemoveDirtConfusion.avi", cache=false, prefer_hw=2) 
src = last

RemoveDirtMC(15, false)
diffmask = AbsSubtractY(src, last).ColorYUV(gain_y=3000, off_y=-100).mt_expand().Blur(1.0).GreyScale()
Overlay(src, last, mask=diffmask)

Interleave(src, last, diffmask) # show before, after, mask

[skh]

thanks for that suggestion. I'm not really shure if I understand it but I will definetley give it a go.
The video I uploaded is without RemoveDirtMC. Only the photos I posed are.

[skh]

ok I tested it and it seems that I'm still having the issues.

Frame 273 still shows weirdness.


[Attachment 74341 - Click to enlarge]


I used

 RestoreMotionBlocks(clensed,input,alternative=alt,pthreshold=30,cthreshold=8, gmthreshold=40,dist=3,dmode=2,debug=false,noise=limit,noisy=4, grey=_grey)

and also tried the Alternative settings which looked worse.

The stuff I want to remove is the majority of the white dust spots. For example in this frame 166 it worked fine

before

[Attachment 74342 - Click to enlarge]

after

[Attachment 74343 - Click to enlarge]

But as said in other frames it was generating theese ugly things shown on the first screen shot.

thanks a lot for your help

[jagabo]

ok I tested it and it seems that I'm still having the issues.

Frame 273 still shows weirdness.


[Attachment 74341 - Click to enlarge]

I hadn't noticed that earlier. It's definitely a problem with the motion estimation. I'll see if I can find an automated workaround...

[jagabo]

I figured out the problem. RemoveDirtMC() doesn't like the mod4 frame width. Try this:

LWLibavVideoSource("RemoveDirtConfusion.avi") 
AddBorders(0,0,4,0)
RemoveDirtMC(15, false)
Crop(0,0,-4,-0)

Add the diffmask and overlay lines if you still want to protect other parts of the picture.

[skh]

thanks a lot man that is nice to know. So the removedirt bugs didn't occure because of the flickering they were somethings else. I tend to always crop everything away I don't need so from now on I will just leave it in standard resolution and crop after remove dirt.
The weirdness shown on the 2nd picture in my main post remain but I will definetley play around with the diffmask. thanks a lot for your help

[jagabo]

You might find this diffmask calculation a little easier to understand:

diffmask = AbsSubtractY(src, last).Levels(8, 1, 24, 0, 255, coring=false).mt_expand().Blur(1.0).GreyScale()

AbsSubtractY(src, last) gives the absolute value of "src - last" Y (luma) values.

Levels(8, 1, 24, 0, 255, coring=false) sets all Y values 8 and below to 0, all values 24 and above to 255. Values between 8 and 24 are interpolated between 0 and 255.

mt_expand() expands the pixels in the mask. For example a single white pixel on a black background will expand vertically and horizontally by one pixel creating a bigger white spot.

Blur(1.0) blurs the alpha mask we're building so that it has soft edges.

Overlay(src, last, mask=diffmask) Overlays the RemoveDirtMC image (last) over the original image (src) using the alpha mask we just built. Areas where the alpha mask is 0 remain the same as the the source (src). Areas where the alpha mask is 255 get fully over written with the RemoveDirtMC image. Where the alpha mask is between those extremes the RemoveDirtMC image gets blended with the source image according to their weighting (the brightness of the pixel in the alpha mask).

[skh]

thanks for the great explanation. The mask works fantastic. One last question to make shure I did everything right.

AbsSubtractY(src, last) gives the absolute value of "src - last" Y (luma) values.

Is here in this command "last" the RemoveDirtMC image and (src) the original image just like it is in Overlay ?

thanks so much

[jagabo]

AviSynth filters usually take an input and produce an output. In AviSynth, if you don't explicitly give a name AviSynth will assume the name "last". So a sequence like:

AviSource("filename.avi")
Filter1()
Filter2()

is really saying:

last = AviSource("filename.avi")
last = Filter1(last)
last = Filter2(last)
return(last)

I should have given you an entire script to make it clear:

LWLibavVideoSource("RemoveDirtConfusion.avi") 
AddBorders(0,0,4,0)
src = last
RemoveDirtMC(15, false)
diffmask = AbsSubtractY(src, last).Levels(8, 1, 24, 0, 255, coring=false).mt_expand().Blur(1.0).GreyScale()
Overlay(src, last, mask=diffmask)
Crop(0,0,-4,-0)

I explicitly created a stream called src on the third line. It's a copy of the source video after the AddBorders() has been applied. Since I made no further changes to src after that, it has remained the same throughout the rest of the script. Overlay explicitly gets that stream as its first argument, and "last", the output of RemoveDirtMC(), as the second argument. If you had specified return(src) at the end of the script, you would get the original video with the extra border at the right, not the cleaned up video.

[skh]

ah ok thanks a lot that makes a lot of sense. I usually work with clips and have to get used to the "last" term. It is a bit confusing as I am never shure what status so to speak I declare as last. For a single video where I just apply the overlay on the entire thing it is totally logical of course. As I just work in a certain pattern there.
But for section based filtering I always work with clips which is easier for me to remember which section is filtered in which way.
Here is an example for your awesome idea. I'm curious what you think of it

Source = LWLibavVideoSource("").AddBorders(0,0,4,0)
clip1=Trim(Source,0,12566).RemoveDirtMC(15,false) 
clip2a=Trim(Source,12567,15398)
clip2b=Trim(Source,12567,15398).RemoveDirtMC(15, false)
clip3=Trim(Source,15399,72235).RemoveDirtMC(15,false) 

diffmask = AbsSubtractY(Clip2a).Levels(8, 1, 24, 0, 255, coring=false).mt_expand().Blur(1.0).GreyScale()
Overlay(Clip2a, clip2b, mask=diffmask)
clip2c =last

clip1 ++ clip2c ++ clip3 

Crop(0,0,-4,-0)

By this I see totally clear which clip is the filtered and which is the unfiltered variant and by splitting them further it is easier for me to understand that 2a is unfiltered, 2b is filtered and 2c is the overlayed variant of the unfiltered and the mask filtering.

The funny thing is that I had to specify "clip2c" =last in order to make this work.

[jagabo]

Clip2b is the same Trim() of video as clip2a. So you could use clip2a when building clip2b (instead of trimming again). AbsSubtractY() requires two arguments: AbsSubtractY(clip2a, clip2b). The output of Overlay() near the end is "last". To use it without explicitly naming it Clip2c you would use "clip1 ++ last ++ clip3". Or you could just name the output of Overlay() clip2c -- that would be more consistent of your script.

Source = LWLibavVideoSource("").AddBorders(0,0,4,0)
clip1=Trim(Source,0,12566).RemoveDirtMC(15,false) 
clip2a=Trim(Source,12567,15398)
clip2b=clip2a.RemoveDirtMC(15, false)
clip3=Trim(Source,15399,72235).RemoveDirtMC(15,false) 

diffmask = AbsSubtractY(clip2a, clip2b).Levels(8, 1, 24, 0, 255, coring=false).mt_expand().Blur(1.0).GreyScale()
clip2c = Overlay(clip2a, clip2b, mask=diffmask)

clip1 ++ clip2c ++ clip3 

Crop(0,0,-4,-0)

[skh]

you are a master of your craft. Thanks a lot. That totally makes sense to me