RemoveDirtMC

[Mephesto]

So I can't get this to work. TBH, I'll be happy with getting even the older non-SE version to work as I was impressed by the results I've seen a few years back.

AVS script:

LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\AvsFilterNet.dll") 
LoadNetPlugin("C:\Program Files\AviSynth 2.5\plugins\NLMeansCL_netautoload.dll")
Avisource("J:\test.avi")
RemoveDirtMC_SE()

I believe I got all the required DLLs.

Error message attached.

[poisondeathray]

Do you have openCL GFX card?

There are several versions floating around, but you can find the (non GPU accelerated) John Meyer modified RemoveDirtMC from videofred's thread at doom9, in the film restoring zip file

http://forum.doom9.org/showthread.php?t=144271

[Mephesto]

Do you have openCL GFX card?

I do. Why, do you think my problem is related to openCL?

[poisondeathray]

Because that RemoveDirtMC_SE uses NLmeansCL , which uses openCL and your error message says "NLMeansCL initialization error"

Did you check to see if NLmeansCL runs ok by itself first ? There are some tips and problem solving things to do in that thread
http://forum.doom9.org/showthread.php?t=158925

[Mephesto]

Yes, NLmeansCL loads fine by itself. I did see the initialization error but what caught my attention was "Could not find part of the path on C:\Temp" because C drive is not my main drive.

[sanlyn]

Because that RemoveDirtMC_SE uses NLmeansCL , which uses openCL and your error message says "NLMeansCL initialization error"

Did you check to see if NLmeansCL runs ok by itself first ? There are some tips and problem solving things to do in that thread
http://forum.doom9.org/showthread.php?t=158925

Doom9 tips still don't work for me, and this is the third time around since I first saw that post. Figured out, I'd have to build a new PC to run NLmeansCL and can't use my AIW card. Can't see the point of building a PC just to run one plugin.

[poisondeathray]

You can use the non GPU version. It's actually not that slow, and works better in some cases.

[Mephesto]

I tried the non-GPU version and it did nothing. No spot removal. x.X

[poisondeathray]

did you try using a higher strength?

post your script and any error messages

do the "spots" occur on duplicate frames ?

[Mephesto]

I'm back. I wanted to re-touch this issue, but now I've completely forgotten the workflow of RemoveDirtMC. I have two locations of RemoveDirtMC in my avisynth plugin folder, one in its own folder. So to make sure everything's in the right place, can I have a link to your RemoveDirtMC setup so it matches mine?

Right now my script (RemoveDirtMC(4,false) says there is no function named RemoveDirt. Do I gotta call up a DLL?

[lordsmurf]

I'm back. I wanted to re-touch this issue, but now I've completely forgotten the workflow of RemoveDirtMC. I have two locations of RemoveDirtMC in my avisynth plugin folder, one in its own folder. So to make sure everything's in the right place, can I have a link to your RemoveDirtMC setup so it matches mine?

Right now my script (RemoveDirtMC(4,false) says there is no function named RemoveDirt. Do I gotta call up a DLL?

I'm having this same issue. It's been too long, and I've forgotten what I used to do.

[johnmeyer]

Any reason to necro this three-year-old thread? Also, I don't understand the post.

If you are having a problem with RemoveDirtMC, check out my various posts in the doom9.org AVISynth forum in the two very long "restoring 8mm film" threads. I found that RemoveDirtMC completely blows away anything else when it comes to removing dirt on old amateur film, although it can also remove lots of other small objects, like balls being thrown. You have to pay attention to how far you crank up the dirt removal settings.

[lordsmurf]

I was reading old posts to try and remember what I had done 6+ months ago. And I posted because I was amused that I'm not the only person to whom this happens.

RemoveDirtMC actually was not very useful. I had confused it with my other method, which involved a median and degrain.

It took a few hours, but I found my last script.

[PuzZLeR]

Speaking of Doom9, they rarely gave me anything useful without plowing through piles of long and boring scripts (that didn't work) from amateurs/wannabes showboating. In the years since I've been avoiding them, I believe I've saved much time.

The most useful one IMO is from here on VideoHelp, RemoveSpotsMC3x(). It will remove many white spots and dropouts (as long as they don't stretch over multiple frames) and will not blur/soften video like other similar methods.

To run it, you will need RemoveSpotsMC(), which RemoveSpotsMC3x() calls. You may also have to download and install the latest RemoveGrainT package IIRC (and place it in the AviSynth plugins folder).

To run Clense, which is an integral part of this script, you will need certain files in your system32 folder: AvsRecursion.dll, and may also need Visual C++ library runtime components/modules: vcredist_x86, msvcp80.dll, msvcr71.dll.

To run the script, use the following:

Avisource("Y:\Spotty Clip.avi")
converttoyv12() #This depends on source.

function RemoveSpots(clip input, bool "_grey", int "repmode")
{
_grey=default(_grey, false)
repmode=default(repmode, 16)
clmode=17
clensed=Clense(input, grey=_grey, cache=4)
sbegin = ForwardClense(input, grey=_grey, cache=-1)
send = BackwardClense(input, grey=_grey, cache=-1)
alt=Repair(SCSelect(input, sbegin, send, clensed, debug=true), input, mode=repmode, modeU = _grey ? -1 : repmode )
restore=Repair(clensed, input, mode=repmode, modeU = _grey ? -1 : repmode)
corrected=RestoreMotionBlocks(clensed, restore, neighbour=input,  alternative=alt, gmthreshold=70, dist=1, dmode=2, debug=false, noise=10,  noisy=12, grey=_grey)
return corrected
} 

function RemoveSpotsMC3x(clip clip, int "limit", bool "_grey")
{
  _grey = default(_grey, false)
  limit = default(limit, 6)
  i=MSuper(clip,pel=2)
  bvec = MAnalyse(i,isb=false, blksize=8, delta=1, truemotion=true)
  fvec = MAnalyse(i,isb=true, blksize=8, delta=1, truemotion=true)
  backw = MFlow(clip,i,bvec)
  forw  = MFlow(clip,i,fvec)
  clp=interleave(backw,clip,forw)
  clp=clp.RemoveSpots().RemoveSpots().RemoveSpots()
  clp=clp.SelectEvery(3,1)
  return clp
}

SeparateFields() 
Ev=SelectEven().RemoveSpotsMC3x(0,false)
Od=SelectOdd().RemoveSpotsMC3x(0,false)
Interleave(Ev,Od) 
Weave()

The above separating of fields and weaving is necessary for interlaced material in the function calls. You may have motion issues otherwise.

Warning: Be careful when using this script. A good example clip to test it on is, for example, a tennis game or soccer game (the latter is football in other parts of the world). Watch the white ball in motion - you may lose it in faster scenes as the script may interpret it as a "bad white dot".

To rectify this, weaken the strength in your testing, by trying any combination of the following (together or individually each):

Try gmthreshold with lower number parameters like 30, 10, 1.

Reduce the calls of RemoveSpots() in RemoveSpotsMC3x down to only two calls (making it 2x actually) such as:

clp=clp.RemoveSpots().RemoveSpots()

.

Hope it helps.

Note: 64 bit is still not stable with many plugins/scripts (as of this post), maybe even with these, so you're on your own here in 2015-09.

[lordsmurf]

Speaking of Doom9, they rarely gave me anything useful without plowing through piles of long and boring scripts (that didn't work) from amateurs/wannabes showboating.

I agree completely. It's probably 98% useless blah-blah, dead links, and scripts that never worked to begin with. There are some good posts there, but you have to wade through mountains of crap to find them. I've often found myself wasting 5-6 hours there, and walking away with nothing to show for it.

Tip: One of the best ways to search there is to find a member that gives helpful (and current) info, and just search his posts. It helps cut down on the noise. But even that can yield no results at times.

Hope it helps.

I'll surely test it. Thanks.

[johnmeyer]

Speaking of Doom9, they rarely gave me anything useful without plowing through piles of long and boring scripts (that didn't work) from amateurs/wannabes showboating.

I agree completely. It's probably 98% useless blah-blah, dead links, and scripts that never worked to begin with. There are some good posts there, but you have to wade through mountains of crap to find them. I've often found myself wasting 5-6 hours there, and walking away with nothing to show for it.

I have to disagree.

Yes, there are some major-league prima donnas that lurk there, but most of them are straight out of central casting: the guy who drinks jolt cola, eats pizza, hasn't bathed in a week, talks geek ... but ... writes code that does thing that no one else thought could be done.

I just posted, in another thread in this forum, my disagreement with another post made by the owner of the excellent DigitalFAQ site. I offered lots of evidence to back up my claims, and I'll do the same here.

For instance, the doom9.org guru StainlessS, has written some very difficult to understand, but ultimately amazing code that lets you analyze the content of a frame of video in ways most people wouldn't even have thought to do. As a result of his work, and his amazing willingness to help me, I created something I don't think anyone has done before: I was able to remove the noise bars that result from using a video or film camera to capture the screen of a traditional CRT television. Here is the result of my work:

1955 NFL Football Noise Bars Before-After

With a little more work I could have made it near-perfect, but this was for an amateur film collector, with no money involved, so I had to get on to other things.

I already posted today, in that other thread in this forum, my "before/after" results with film restoration, using a heavily modified script, produced at doom9.org, by "VideoFred." His script worked fine, but I was able to engineer a 4x speed improvement, and also significantly reduced the amount of dirt using ... RemoveDirtMC !! For those that didn't read the other thread here in this forum, here is a link to a before/after torture test that shows off what this amazing technology can do. When I posted this link in that other thread today, I was trying to illustrate the stabilization I achieved. In this case, forget about the stabilization and instead please concentrate on the "subtle" dirt removal

1941 Film Restoration of Buick Aviation Engine Plant Construction, Melrose Park, Illinois

I could go on, but it would take me all day to list all the amazing things I have been able to do as a result of the help I have received there. Those scripts have let me:

• stabilize video;
• remove dirt;
• reduce grain;
• change film speed;
• produce the most amazing NTSC to PAL and PAL to NTSC conversions you've ever seen;
• create stunning slow motion;
• interpolate frames for smoother playback of low framerate video;
• remove pulldown frames from film captured by a film projector from which the shutter has been removed;
• resize video in a far better manner than my NLE (Sony Vegas Pro) could do;
• remove flicker from video;
• deinterlace video with almost no artifacts
• remove all sorts of noise from VHS (and other) videos, including noise in low-light video
• find duplicate frames
• find gaps where frames have been dropped, and interpolate a new frame to fill the gap
• create excellent scene detection (i.e., detect where the camera was stopped and then re-started)
• remove pulldown from 29.97 interlaced video in order to obtain the original 23.976 progressive film
• make old Kinescopes look like the video had been saved on videotape instead of film

To put it bluntly, my media restoration business would not exist if it weren't for the help I've received at doom9.org.

[lordsmurf]

I don't see how we're talking about the same forum. We must visit different threads.

Like I said, there's definitely some amazing stuff there -- but it's under a ton of crap to find (dead links, useless blahblah posts, non-working scripts, etc). Part of the issue is the forum organization and too many crap thread titles that make it unfriendly to browse. I also totally hate 25+ (100+!!!) page threads that go back 5-10 years. Start a new thread, for god's sake! (For the record, I can't stand VH threads that go on this long either. After 25 pages, I just don't give a shit anymore. Thankfully, we have very few of those.)

StainlessS is one of those I refer to. Search his past posts, or threads he's started. You'll find good stuff that way. Even Google-searching Doom9 can be fruitless by comparison.

FYI: I've searched your past posts as well.

I never said the site was useless.

[manono]

I also appreciate the good work StainlessS has done. Unfortunately, I find his descriptions and docs incomprehensible, so I've been unable to make use of his work.

Back on track, I have no problem using RemoveDirtMC and it's good at removing more/bigger dirt/spots than RemoveSpotsMC (or 3X). But Mephisto was using something called RemoveDirtMC_SE so I can't help at all there. And, apparently, lordsmurf has solved his problem himself, so I've nothing to contribute, I guess.

[johnmeyer]

One last comment on doom9 and then, as they say, back to the program.

The only section I visit is the AVISynth usage section. Nothing else. I don't find the thread headings any worse than any other forum I visit, but I most definitely have to agree that something must be done about the 25-page threads. I've never run a BBS or forum, so I have no clue how to correct this problem. The more popular your forum becomes, the bigger the problem.

And, for the record, nothing tops the AVS Forum, where 200+ page threads, spanning four years, are actually quite common.

As far as RemoveDirtMC, and the various offshoots, the biggest problem I have is that I have never figured out how to control the size of the spot it is going to remove, nor does there seem to be any way to add a threshold so that it only removes really dark spots. I spent dozens of hours trying to get the Despot function to work, including wrapping it inside of motion estimation. I could never get it to work reliably, but it did have the ability to specify spot size, contrast, and all sorts of other useful things. Unfortunately, even by the poor standards of AVISynth plugins, the RemoveDirt and RemoveDirtMC documentation is virtually incomprehensible, so I've never been able to tweak it successfully.

Finding spot size control would be really useful because I do a LOT of work for a prominent sports film collector, with ties to NFL Films, and when I restore football films, the ball often disappears when kicked or passed. I end up having to do two restorations, one with dirt removal and the other without, and then either cut between them in my NLE (simple by time consuming) or create a frame by frame mask to re-insert the ball (complicated, and incredibly time consuming, but excellent results). If I could tell the program to ignore gray objects, and only remove almost-black objects, and only those below a certain size, I'd have a lot less work to do.

[poisondeathray]

+1 for spot size control - very useful addition if it can be implemented

one way to do dark vs light, or limit to a given Y range , etc.. is composite with masktools luma mask

(not sure if anyone noticed but there are 2 banned posters in this thread )

[PuzZLeR]

I never said Doom9 is (completely) useless, but the mentioned 98% useless does indeed sound accurate to me, which means that you may (I said "MAY") find something useful, maybe even unique, if you're willing to sludge though hours (or MANY HOURS AND DAYS) of garbage, kind of like swimming through thick mud.

Furthermore, on top of enduring exhaustive threads with its members trying to win some "best programmer" award, and the fact that much of that board needs some serious updating and cleaning up, what further compounds the problem is that much of the info is not practical due to the bias to please Mods (for fear of "strikes") and the many posts overly edited (by Mods) which compromise even the comprehensibility of a thread's flow.

Meh. I'll stick with VideoHelp. I have found awesome - and concise - solutions for this problem here, and many solutions to the example problems mentioned a couple of posts ago too can be found similarly here, and in some other - also very good - Forums.

As far as RemoveDirtMC, and the various offshoots, the biggest problem I have is that I have never figured out how to control the size of the spot it is going to remove, nor does there seem to be any way to add a threshold so that it only removes really dark spots. I spent dozens of hours trying to get the Despot function to work, including wrapping it inside of motion estimation. I could never get it to work reliably, but it did have the ability to specify spot size, contrast, and all sorts of other useful things. Unfortunately, even by the poor standards of AVISynth plugins, the RemoveDirt and RemoveDirtMC documentation is virtually incomprehensible, so I've never been able to tweak it successfully.

Finding spot size control would be really useful because I do a LOT of work for a prominent sports film collector, with ties to NFL Films, and when I restore football films, the ball often disappears when kicked or passed. I end up having to do two restorations, one with dirt removal and the other without, and then either cut between them in my NLE (simple by time consuming) or create a frame by frame mask to re-insert the ball (complicated, and incredibly time consuming, but excellent results). If I could tell the program to ignore gray objects, and only remove almost-black objects, and only those below a certain size, I'd have a lot less work to do.

Ouch. This scene-by-scene and frame-by-frame stuff sounds awfully time consuming. I certainly hope the compensation is worth it for you.

Then again, I've been guilty of conducting some (very) time consuming stuff like this before - for personal hobby stuff, I shamefully admit.

As for the black spots, I forgot to mention that the script I've mentioned, RemoveSpotsMC, does handle them effectively, just like the white spots. Again, as I also mentioned, be careful or you will lose them too if they're important. A good test for this is a hockey game - watch the puck in fast motion. Again, reduce the value of gmthreshold if the filter is too aggressive by removing the puck in fast motion (you will lose a good slap shot).

As for RemoveDirtMC, I'd like to help, but all I remember, a while back, is that it didn't work for me. Either it was one of those that blurred too much, or the another that wouldn't load for me. You have encouraged me to test it again though.

You raise some nice issues, like actual spot sizes. Also, I would love to lengthen the detection of a spot over multiple frames. RemoveSpotsMC works excellently with the spot if it's in one frame. However, will only diminish, at best, a dropout that stretches beyond that.

[-Habanero-]

If anyone has trouble getting RemovedirtMC() to work I'll gladly upload all my files so you don't have to waste time scrounging thru multiple posts and dead links. I remember finding this after I was looking for something better than removespots and it was great.

[PuzZLeR]

If anyone has trouble getting RemovedirtMC() to work I'll gladly upload all my files so you don't have to waste time scrounging thru multiple posts and dead links. I remember finding this after I was looking for something better than removespots and it was great.

Can you post more info on this? I, and a few others, haven't had any success with it.

And yes, I have no appetite for sludging through garbage to get this working.

[-Habanero-]

Here you go.

[johnmeyer]

One line in the RemoveDirtMC.avsi file needs to be changed. This is the corrected version:

clp=interleave(forw,clip,backw)

The "forw" and "backw" variables were interchanged in the original code.

Here is a link to the middle of a long post at doom9.org where StainlessS discovered this error:

RemoveDirtMC Script Errors

[Reel.Deal]

Why do people here at VideoHelp keep attaching plugins which are readily available in the AviSynth wiki? http://avisynth.nl/index.php/RemoveDirt. Copy and paste the link and you're done...

The one on the wiki is also compiled with a newer compiler (MSVC 2010) and does not need the msvcr71.dll runtime component from the very ancient Microsoft Visual C++ .NET 2003. Due to using a more modern compiler it's also a bit faster (at least on my system).

[-Habanero-]

One line in the RemoveDirtMC.avsi file needs to be changed. This is the corrected version:

clp=interleave(forw,clip,backw)

The "forw" and "backw" variables were interchanged in the original code.

Here is a link to the middle of a long post at doom9.org where StainlessS discovered this error:

RemoveDirtMC Script Errors

Thanks for that, I corrected it. What difference will it make to the quality, though?
I noticed as of recently that I can no longer do removedirtMC on select frames. On a cartoon this script will sometimes remove legitimate detail so my best best bet was to process the whole movie with a fixed value and then use a very high value to get rid of the much larger spot occurrences here and there. Original video is combined with Trim()s of the removedirtMC(50) one. But this doesn't work anymore. I'm not sure what happened but I hope your correction fixes this.

[johnmeyer]

Thanks for that, I corrected it. What difference will it make to the quality, though?
I noticed as of recently that I can no longer do removedirtMC on select frames. On a cartoon this script will sometimes remove legitimate detail so my best best bet was to process the whole movie with a fixed value and then use a very high value to get rid of the much larger spot occurrences here and there. Original video is combined with Trim()s of the removedirtMC(50) one. But this doesn't work anymore. I'm not sure what happened but I hope your correction fixes this.

My correction makes little difference in most cases, and is certainly not the cause of your problems.

The problems you describe are well-known. There is no easy way (that I know of) to control the size and color of the spots removed. I do a lot of work on old sports films, and the football, when thrown, sometimes disappears. I have to do a second pass, without the dirt removal, and switch to that during the time the ball is in the air.

[-Habanero-]

My correction makes little difference in most cases, and is certainly not the cause of your problems.

The problems you describe are well-known. There is no easy way (that I know of) to control the size and color of the spots removed. I do a lot of work on old sports films, and the football, when thrown, sometimes disappears. I have to do a second pass, without the dirt removal, and switch to that during the time the ball is in the air.

You are doing it differently. You are processing with a high value in the hope of crushing every artifact and restoring select frames where the damage was too much. I process everything with a low value and then process select frames with a higher value where some large spots were missed.

Here's a sample:

a=import("M:\cartoon.avs")
b=a.RemovedirtMC(24,false)

c=a.trim(0,21346)
d=b.trim(21347,21347)
e=a.trim(21348,0)

c+d+e

Basically one large spot on frame 21347 was missed so I processed only that one with the high 24 value.

This used to work but it doesn't anymore for some reason. It works if I don't Trim but I don't understand because the Trim is done after the RemoveDirt processing. If it was before then I would understand cuz it would have nothing to reference but that shouldn't be the case.

[lordsmurf]

Why do people here at VideoHelp keep attaching plugins which are readily available in the AviSynth wiki? [.

It's NOT the first time in the past 10-15 years that an Avisynth site went tits up, and VH (doom9, digitalFAQ, others) were the only options left. The wiki just hotlinks, from what I've seen.