RemoveDirtMC motion artifact mitigation?

[BabaG]

i've been experimenting with RemoveDirtMC and have gotten some very good dirt removal on old 16mm film transfers. there are some areas, though, in which the artifacting is pretty severe. here's an image of a frame i composited from three tests i did. the fourth image is unprocessed as a reference. all three of the tests use a technique jagabo suggested in another thread that reduces the image to 50% for processing then, after processing, uses nnedi3 to bring it back to the original size.



upper left = two passes of RemoveDirtMC, extra blurring added while image reduced
upper right = two passes of RemoveDirtMC, no blurring added
lower left = one pass of RemoveDirtMC, no blurring added
lower right = unprocessed image

the files i'm working with are 2880x2160p@23.976fps. i've reduced the frame size here for posting.

obviously, there's a lot of temporal artifacting in all three of the test images. as a still image it's worse than as running footage but i'm still wondering if there's anything i can do to mitigate the temporal smearing.

here's the code for the test that employs the blurring. the other tests just comment out lines of this code. the blurring test is the only one that uses dithering. i used that to confuse the algorithm into mitigating some of the extreme grain. (this footage was shot on film whose expiration date was eight years prior to this exposure leading to extreme grain and an extremely low d-max. it was then developed by hand, resulting in a LOT of embedded dirt.)

FFMPEGSource2("Buddies_Galore_2880x2160p_GV-HQX_23976fps.mov", atrack=1)
z_ConvertFormat(pixel_type="YUV420P16")

BilinearResize(width/2, height/2)

ReduceFlicker(strength=3, aggressive=true)
dfttest(dither=50)      #   Accepts values 0 - 100
Tweak(0, 0, 0)          #   hue -180.0 - 180.0, sat 0 - 10.0, bright -255.0 - 255.0
Blur(.29)               #   Accepts values -1.0 to 1.58
Blur(.29)               #   Accepts values -1.0 to 1.58
Blur(.29)               #   Accepts values -1.0 to 1.58

RemoveDirtMC(60, false)
RemoveDirtMC(60, false)

UnsharpMask2(strength=50, radius=2, threshold=0)
UnsharpMask2(strength=50, radius=2, threshold=0)

dfttest(dither=7)      #   Accepts values 0 - 100
Tweak(0, 0, 0)          #   hue -180.0 - 180.0, sat 0 - 10.0, bright -255.0 - 255.0

nnedi3_rpow2(2, cshift="Spline36Resize")

Sharpen(1.0)            #   Accepts values -1.58 to 1.0
Sharpen(0.75)           #   Accepts values -1.58 to 1.0
Levels(5140,1,61680,0,65535, false)     # 16 bit values

i'm sure this is all very odd as i really don't know what i'm doing but it does seem to work most of the time. the large frame sizes seem also to call for some extreme measures to address the issues.

by cutting between the various tests i seem to be able to put together a pretty credible cleanup but would love to be able to have something that mitigates the temporal smearing in some shots.

thanks for any suggestions,
babag

[poisondeathray]

Those do not look like typical RemoveDirtMC artifacts . (Or maybe you've resized the image and it's hiding the artifacts) . RemoveDirtMC artifacts typically look more like splotches, or areas that are missing details compared to adjacent areas

I suspect they are temporal "ghosting"/smearing artifacts from ReduceFlicker(strength=3, aggressive=true) . If you comment out the ReduceFlicker, do you get similar problems ?

How are you examining it ? Filters like RemoveDirtMC require linear access. If you jump around, you will get different results everytime.

You have to go back about 15-20 frames and go frame by frame. It also sometimes helps if source filter uses threads=1

This is a slightly different topic but a generally more accurate dirt removal method and recovery of detail is StainlessS' Spotless + Delta Restore. Have a look at the thread at doom9 . But if you have lots of dirt , it might not be suitable approach

[BabaG]

thanks, pdr. you're right about ReduceFlicker. ran a quick test on the section the previously posted frame came from and the smearing disappears when i comment out the ReduceFlicker line. problem is, the dirt seems to have gotten worse. the smearing seems to have helped get rid of or obscure the dirt. without ReduceFlicker, it's back. what's worse, my RemoveDirt values now seem to have been invalidated. it looks like the tests i derived them from had the ReduceFlicker filter in the chain, invalidating my numbers. you don't happen to know what the valid ranges are for RemoveDirt, do you? i haven't found them anywhere.

if i can get this sorted, it looks like i'd drop one of my three tests and replace it with one that doesn't use ReduceFlicker. i could then cut to the non-ReduceFlicker file when motion artifacts become too noticeable.

one last thing, the smearing actually helped the extreme grain problem. it got rid of most of the grain better than any of the purpose-built filters i tried. it seems that the grain is so severe in this material that the algorithms don't see it as part of the film but as part of the content, as though it represents objects in the frame. will take a couple of days to get this new render done so i'll report back after that.

thanks again,
babag

[poisondeathray]

You're dealing with 2880x2160p , but even resizing by BilinearResize(width/2, height/2) , that might be to large for the default blksize . It also depends on how large the "dirt" "objects" are. You might have to increase blksize . You mention "grain too" ; "grain" typically is defined as smaller than "dirt" and has a more continuous distribution spatially

Do you have unique frames ? No duplicates ? Duplicates are one reason why dirt and grain removal filters don't work well. You'd need to decimate to unique film frames.

Also, if the "dirt" objects last more than 1 frame, they will not be registered as "dirt"

[BabaG]

all frames are individual, no duplicates. captured originally on a red camera. that's why the huge frame sizes. the material i have is gv-hqx that was rendered out of the red originals.

i know the basic idea of blksize but not how to address it or whether it needs to be addressed multiple times, like for each filter in the script.

thanks again,
babag

[poisondeathray]

i know the basic idea of blksize but not how to address it or whether it needs to be addressed multiple times, like for each filter in the script.

Which version ore RemoveDirtMC are you using ?

There are dozens of "RemoveDirtMC" variants. Some are (unfortunately) named exactly the same.

In general you would have to adjust the blksize for each call of RemoveDirtMC, if it's different from the default specified in the function. blksize is a property specific to mvtools2 based functions - so things like dfttest, tweak wouldn't accept blksize . Unsharpmask shouldn't either, but there are dozens of variations too - some of them motion compensated which might use mvtools2. (Again some of them named inappropriately.)

If the function does not have the blksize as a variable option , that means the blksize value is hardcoded in the function (less flexible if you're dealing with SD, vs HD, vs. UHD). You'd have to edit the function defintion (maybe rename it something else that is appropriate to prevent confusion) . Some "RemoveDirtMC" variations have more than 1 stage , using mrecalculate using a smaller blksize than the 1st manalyze - for those you'd want to adjust them both

[BabaG]

the RemoveDirtMC that i have is this:

function RemoveDirt(clip input, int limit, bool _grey)
{
  clensed=input.Clense(grey=_grey, cache=4)
  alt=input.RemoveGrain(2)
  return RestoreMotionBlocks(clensed,input,alternative=alt,pthreshold=6,cthreshold=8, gmthreshold=40,dist=3,              dmode=2,debug=false,noise=limit,noisy=4, grey=_grey)

  # Alternative settings
  # return RestoreMotionBlocks(clensed,input,alternative=alt,pthreshold=4,cthreshold=6, gmthreshold=40,dist=1,dmode=2,debug=false,noise=limit,noisy=12,grey=_grey,show=true)
  # return RestoreMotionBlocks(clensed,input,alternative=alt,pthreshold=6,cthreshold=8, gmthreshold=40,dist=3,tolerance= 12,dmode=2,debug=false,noise=limit,noisy=12,grey=_grey,show=false)
}

function RemoveDirtMC(clip,int limit, bool "_grey")
{
  _grey=default(_grey, false)
  limit = default(limit,6)
  i=MSuper(clip,pel=2)
  bvec = MAnalyse(i,isb=false, blksize=8, delta=1, truemotion=true)
  fvec = MAnalyse(i,isb=true, blksize=8, delta=1, truemotion=true)
  backw = MFlow(clip,i,bvec)
  forw  = MFlow(clip,i,fvec)
  clp=interleave(backw,clip,forw)
  clp=clp.RemoveDirt(limit,_grey)
  clp=clp.SelectEvery(3,1)
  return clp
}

posted here:

https://forum.videohelp.com/threads/347583-Color-correction-%28or-grading-%29?p=2174602&viewfull=1#post2174602

by somebody called "poisondeathray" back in 2012.

i'll probably be stopping my render early as it's doing funny things so i guess i really need some further tests.

thanks,
babag

[poisondeathray]

That one has the blksize set to 8 . You can try changing the blksize (from your description of the source, I'm guessing it might not do much), and/or prefiltering , and/or using another dirt removal filter. It might be some of the heavy grain is confusing the dirt removal

Not sure what you're dealing with exactly, since the source looks ok in the 1st post (but it's obviously resized picture) .

If you post a video sample of the source, you might get some suggestions

[BabaG]

i think the grain is, indeed, confusing the dirt removal. that's why i added extra blur in addition to the resizing suggested by jagabo. after the resize i added a bit of dither to try to fool the filtering into thinking the grain was smaller, then blurred things more in hopes of getting the actual film grain to sort of disappear into the fuzziness. then the dirt removal. then bringing it back up to size. works pretty well except now the dirt is not removing.

how do blksize values work? do they double? add by 2? would 16 or 32 be appropriate things to try? or, would 10 or 12 be better? or, even 64 or 128?

is there a guide or info on how to 'pre-filter'? seen it mentioned but no idea how to do it.

thanks,
babag

[poisondeathray]

how do blksize values work? do they double? add by 2? would 16 or 32 be appropriate things to try? or, would 10 or 12 be better? or, even 64 or 128?

You can try 16 or 32 or 64

There are other combinations of blksize and blksizeV (vertical) in newer mvtools2 . If it was something you were going to change frequently, you could make blksize (or any parmater/setting) a variable for the function. There are dozens of settings in mvtools that you can adjust - you can have a look at the documentation. There are newer RemoveDirtMC variants that probably have a few of them as adjustable parameters (instead of hardcoded values)

blksize

Size of a block (horizontal). Larger blocks are less sensitive to noise, are faster, but also less accurate.
List of available block sizes (blksize x blksizeV)
64x64, 64x48, 64x32, 64x16
48x64, 48x48, 48x24, 48x12
32x64, 32x32, 32x24, 32x16, 32x8
24x48, 24x24, 24x32, 24x12, 24x6
16x64, 16x32, 16x16, 16x12, 16x8, 16x4, 16x2
12x48, 12x24, 12x16, 12x12, 12x6, 12x3
8x32, 8x16, 8x8, 8x4, 8x2, 8x1
6x24, 6x12, 6x6, 6x3
4x8, 4x4, 4x2
3x6, 3x3
2x4, 2x2
Note: some smaller block sizes can only be available in 4:4:4 formats, due to block size division (chroma subsampling)

blksizeV

Vertical size of a block. Default is equal to horizontal size.

is there a guide or info on how to 'pre-filter'? seen it mentioned but no idea how to do it.

Not really; if you think the grain is a problem, reduce the grain first using various methods

The general idea is to use a different , "cleaner" clip to do the calculations on for motion vectors. If the clip you are using is "noisy" or "grainy", that fluctuating "noise/grain" might interfere with the motion vectors required for dirt removal. Some filters have options for prefilters built in for the main filtering routine - an example would be smdegrain, where it has several built in options, or to use a custom/external clip

[jagabo]

Flickering will also confuse the dirt removal. Increasing the scene change threshold might help.

[BabaG]

doing more tests today. did a very brief one last night and found that changing blksize from 8 to 4 inside the RemoveDirtMC function seemed to make a huge difference in catching the dirt.

babag

[hydra3333]

doing more tests today. did a very brief one last night and found that changing blksize from 8 to 4 inside the RemoveDirtMC function seemed to make a huge difference in catching the dirt.

Changing to 4 caught more dirt or caught less dirt ?

[BabaG]

seemed to have removed more dirt but the tests i'm doing now will hopefully clarify what are the best solutions for this, rather unusual, material. the brief test seemed to clean better at 4 but a lot of dirt is still getting through now that i'm doing more rigorous tests. depends on the scene, i suppose.

i've selected a half dozen scenes and am rendering each at a dozen or so different combinations of settings, trying to come up with the three or four most practical ones. if i can get it down to three or four, i'll render each of those and edit between them.

for a lot of the material, most really, ReduceFlicker with strength at 3 and aggressive at true works really well. it just has bad motion artifacting sometimes. the upside to that, though, is that it actually seems to be the best way i've yet found to reduce grain. it seems to effectively superimpose several frames, the effect of which is to give each particle of grain only a fractional exposure in the composite frame, which greatly smooths the image. the cost is double (and more) imaging of parts of the frame. it's probably going to come down to a judgement call as to whether, for any given scene, the artifacting is preferable to the intense grain. the other benefit is that any dirt that slips through also seems to disappear, at least to some degree, into the layering of frames.

babag

[poisondeathray]

for a lot of the material, most really, ReduceFlicker with strength at 3 and aggressive at true works really well. it just has bad motion artifacting sometimes. the upside to that, though, is that it actually seems to be the best way i've yet found to reduce grain. it seems to effectively superimpose several frames, the effect of which is to give each particle of grain only a fractional exposure in the composite frame, which greatly smooths the image. the cost is double (and more) imaging of parts of the frame. it's probably going to come down to a judgement call as to whether, for any given scene, the artifacting is preferable to the intense grain. the other benefit is that any dirt that slips through also seems to disappear, at least to some degree, into the layering of frames.

Use whatever works...but surely there is some better way than blending frames, or simple blurring, or smearing ?

You mentioned unique characteristics, and extreme grain, plus dirt - but it can't be the only time someone encountered this combination. "Extreme heavy grain" can mean different things to different people. Can you post a sample?

There are other tools that might help . eg. There are denoisers that have some overkill options, such as Neat Video. Or depending on the "grain" characteristics (which film stock, grain size, distribution, variance, color pattern/ monochromatic , etc...), there are some machine learning approaches that might help as well. Some of them are pretty amazing for heavy smaller sized grain (gaussian like noise) .

[jagabo]

KNLMeansCL() might be good for prefiltering.