which filters to remove dropout or comets, and where get them

[spiritgumm]

Searching the forum, there are many posts for removal of this kind of thing using despot, deVCR, depulse, Removespots, but I cant find all the scripts and required plugins to try most of them. If anyone can suggest what filters would be appropriate and where to find them, I'd appreciate it.

[LMotlow]

I made two versions (attached mp4's), one using RemoveSpotsMC3, another using RemoveDirtMC.

RemoveSpotsMC3 cleaned up most spots except the leading and trailing frames. With more frames the temporal filter will clean more of the first and last frames. RemoveSpotsMC3 is very slow (less than 2fps) and does remove some detail from ragged adegs, such as the picture frame in the background, and left one or two blips along the way.

RemoveDirtMC left a couple of very tiny spots here and there but cleaned with fewer side effects. Runs much faster. Increase the first parameter in RemoveDirtMC to make it stronger, but watch for disappearing detail.

A modified RemoveSpotsMC script that most people use is attached as RemoveSpotsMC4.avsi. From the avsi you can call any of 5 different RemoveSpotsMC functions, including RemoveSpotsC3.

A modified RemoveDirtMC that most people use is attached as RemoveDirtMC.avsi.

The avsi's require these support files:

# RemoveDirt.dll (v.09) http://web.archive.org/web/20140723074402/http://home.arcor.de/kassandro/RemoveDirt/RemoveDirt.zip
# RgTools.dll http://avisynth.nl/index.php/RgTools
# mvTools2.dll http://avisynth.org.ru/mvtools/mvtools2.html#download
# DeFlicker.dll https://avisynth.org.ru/deflicker/deflicker04.zip
# Despot.dll http://avisynth.nl/users/fizick/despot/despot3610.zip
# masktools2.dll http://avisynth.nl/index.php/MaskTools2
# fftd3dFilter.dll https://avisynth.org.ru/fft3dfilter/fft3dfilter.html
# 32-bit FFTW3.dll ftp://ftp.fftw.org/pub/fftw/fftw3win32mingw.zip. Place FFTW3.dll in system32 in 32-bit windows. In 64-bit Windows place FFTW3.dll in the SysWOW64 folder.
# libfftw3f-3.dll. Make a copy of the above named fftw3.dll and name the copy as "libfftw3f-3.dll". Place it in the same Windows folder with FFTW3.dll. Avisynth plugins will look for one of those named dll's, even though they are the same dll.

GradFun2DBmod and AddGrainC were also used, to mask hard and noisy gradient edges in the background walls. Cleaning those artifacts likely wouldn't have been necessary if the analog source had been capped to lossless media instead of lossy MPEG2. I still don't know why people capture analog tape to lossy codecs when they intend to do cleanup. It just makes the job tougher.

RemoveSpotsMC3 version:

MPEG2Source("VTS_01_1.d2v")
AssumeTFF()
SeparateFields()
a=last
e=a.SelectEven().RemoveSpotsMC3()
o=a.SelectOdd().RemoveSpotsMC3()
Interleave(e,o)
Weave()
GradFun2DBmod(thr=1.8,mask=false)
AddGrainC(1.5,1.5)
Crop(14,-0,-10,-8).AddBorders(12,4,12,4)
return last

RemoveDirtMC version:

MPEG2Source("VTS_01_1.d2v")
AssumeTFF()
SeparateFields()
a=last
e=a.SelectEven().RemoveDirtMC(40,false)
o=a.SelectOdd().RemoveDirtMC(40,false)
Interleave(e,o)
Weave()
GradFun2DBmod(thr=1.8,mask=false)
AddGrainC(1.5,1.5)
Crop(14,-0,-10,-8).AddBorders(12,4,12,4)
return last

[jagabo]

RemoveDirtMC()

[LMotlow]

Yep. RemoveDirtMC had some small imperfections but did cleaner work. Another video might have needed something else. Never can tell.

[spiritgumm]

LMotlow, thank you very much for the all the help. Both video samples looked fine (except for interlacing, original is BFF) so I chose RemoveDirtMC avsi since it would run faster. I ran it in AvsP before downloading any plugins and it worked right away, video looked clean as your sample. I started adding the other plugins. I'm still using Avisynth 2.5 so maybe that's why RgTools wouldnt run. Masktools2 and MVtools2 wouldnt work either so I used my existing versions. I'm not sure there's much improvement with all the extra plugins.
The video isnt as bright as your sample, maybe that's from your MKV encoder settings?

I wanted to encode back to dvd using HCenc. The field order is BFF so I changed that in the script, set HCenc for interlacing but still got the wrong field order outputted. Had to change a HCenc setting to BFF to correct output. I'm uploading it - does it seem comparable to yours, or am I missing some improvements you made?

[jagabo]

Another way to use RemoveDirtMC on interlaced video:

SeparateFields()
even = SelectEven().RemoveDirtMC(45)
odd = SelectOdd().RemoveDirtMC(45)
Interleave(even, odd)
Weave()

[spiritgumm]

Isnt that essentially the same as LMotlow's script except for the strength?

SeparateFields()
a=last
e=a.SelectEven().RemoveDirtMC(40,false)
o=a.SelectOdd().RemoveDirtMC(40,false)
Interleave(e,o)
Weave()

[jagabo]

Sorry, I didn't look at his scripts.

[LMotlow]

LMotlow, thank you very much for the all the help. Both video samples looked fine (except for interlacing, original is BFF) so I chose RemoveDirtMC avsi since it would run faster. I ran it in AvsP before downloading any plugins and it worked right away, video looked clean as your sample. I started adding the other plugins. I'm still using Avisynth 2.5 so maybe that's why RgTools wouldnt run. Masktools2 and MVtools2 wouldnt work either so I used my existing versions. I'm not sure there's much improvement with all the extra plugins.
The video isnt as bright as your sample, maybe that's from your MKV encoder settings?

I wanted to encode back to dvd using HCenc. The field order is BFF so I changed that in the script, set HCenc for interlacing but still got the wrong field order outputted. Had to change a HCenc setting to BFF to correct output. I'm uploading it - does it seem comparable to yours, or am I missing some improvements you made?

Sorry, I didn't realize you were using Avisynth 2.5. Guess I should have asked. If you later decide to use one of the RemoveSpots functions on a different video, the "unnoticed" plugins would be activated.

TFF/BFF shouldn't matter in this case, I was too rushed to change it in the script once I had noticed. Why the original is BFF seems odd to me, since only DV is commonly BFF.

The slightly brighter mp4 came from using a very mild contrast filter in my original script, which I didn't think would make enough difference to be noticed. Darks in the original sample are crushed. The filter brightens only the darks and lower midtones. It brought out a few puny remnants of earlier detail and kept the original from looking so grim. The filter used is ContrastMask.avsi (attached). The parameter that sets the brightening strength is "enhance", which I had set to a low 2.0 (the default is 10.0). Strong values for the enhance parameter can make a video look foggy. The avsi has some basic user notes and a link to the original doom9 thread. ContrastMask needs the VariableBlur plugin, also attached, and wants the FFTW3.dll mentioned earlier..

[spiritgumm]

I know this is an old thread, but I was revisiting the video, and noticed that if I wanted the video deinterlaced (and deblend to either 29.97 or 23.976), and followed with the Removespots or Removedirt, they didn't look as clean as the interweave script here. So I tried them before deinterlacing, and the results were like the cleaner version. However, I'm confused because I have used the deinterlaced/cleaner order in other scripts and it appeared work okay. Maybe I should have cleaned those first as well? And why does the order actually matter - shouldn't they clean the spots with the same effectiveness either way?

[jagabo]

Deinterlacing can spread the comets over multiple scan lines and multiple frames, making the more difficult to erase. So you normally want to remove comets first, then deinterlace.

[spiritgumm]

Thanks, jagabo. Would this also apply to other frame processing, like TFM or IVTC?

Another thing I can't figure out is why HCenc crashes with the RemoveDirtMC filter (works fine with RemoveDirt). I have to make a large lossless AVI with Vdub, and encode that with HCenc (to make dvd video).

[jagabo]

Thanks, jagabo. Would this also apply to other frame processing, like TFM or IVTC?

It depends on the filters and the video.

Another thing I can't figure out is why HCenc crashes with the RemoveDirtMC filter (works fine with RemoveDirt).

I believe mvtools isn't perfectly thread safe. Or maybe has a stray memory write. You may be able to work around the problem by specifying fewer threads.

[johnmeyer]

With all MVTools2 scripts, I seldom am able to go beyond 1/2 the total number of threads available. This is true for AVISynth and AVISynth+.

You can also play around with SetMemoryMax(). It can sometimes help with stability.

If you have time to do a few tests, start with threads=1. See if that works. Then, keep increasing until it won't run, or until you notice that the fps starts out fast, but then slows down. This problem usually takes about 10-30 seconds. If you can get to the 30 second mark without a marked reduction in fps, then the number of threads you specified is OK, and you can try the next larger number.