Hi, folks.
I'm trying to remove a big amount of noise in this video:
http://www.4shared.com/video/PHQ6Ta47/vigilante_or.html
First, I use crop and addBorder to repair the top, like this
I'd like to hear from you the best choice to apply here, from this list:Code:crop(last,0,6,720,474).AddBorders(0, 6, 0, 0, color=$000000)
NeatVideo
MCTemporalDenoise
DynamicNoiseReduction (vdub plugin)
MSUDenoiser
MDegrain2
RemoveGrain
Median2
Note that the noise is constant and similar along the video. Always, black dots flashing. There are some gray and white dots, but the big amount are black ones.
I'm not worried about to fix blocks or compression artifacts here. Color balance is not the point also, but the black dots flickering.
Do you, guys, suggest some scripts?
Try StreamFab Downloader and download from Netflix, Amazon, Youtube! Or Try DVDFab and copy Blu-rays! or rip iTunes movies!
+ Reply to Thread
Results 1 to 29 of 29
Thread
-
Thank you.
-
RemoveSpots() will eliminate lots of them.
Code:function RemoveSpots(clip input, bool "_grey", int "repmode") { _grey=default(_grey, false) repmode=default(repmode, 16) clmode=17 clensed=Clense(input, grey=_grey, cache=4) sbegin = ForwardClense(input, grey=_grey, cache=-1) send = BackwardClense(input, grey=_grey, cache=-1) alt=Repair(SCSelect(input, sbegin, send, clensed, debug=true), input, mode=repmode, modeU = _grey ? -1 : repmode ) restore=Repair(clensed, input, mode=repmode, modeU = _grey ? -1 : repmode) corrected=RestoreMotionBlocks(clensed, restore, neighbour=input, alternative=alt, gmthreshold=70, dist=1, dmode=2, debug=false, noise=10, noisy=12, grey=_grey) return corrected }
-
Clense is part of RemoveGrain. You can run RemoveSpots() multiple times to remove even more spots. You have to decimate the video first. There's also a scratch remover -- I forget the name.
-
removedirtmc is really good as well
you should use tdecimate() to get rid of duplicate frames before applying any dirt removal filter
all those types of spot/dirt removal type filters look at differences between frames, so if you have duplicates, it will screw up the algorithm and you will get duplicate dirty frames
also scene changes will usually remain "dirty" , unless you fix them manually or apply by scene
There's also a scratch remover -- I forget the name.
http://avisynth.org.ru/descratch/descratch.html -
jagabo,
The problem now is SCSelect. I put all RemoveGrain family in plugin folder, but SCSelect is not part of them.
Also, how I decimate.
poisondeathray,
I've already grabbed descratch.Thank you. -
Thank you.
-
-
Thank you.
-
On the sample clip I used SelectEvery(5,0,1,2,4). But that probably won't work over the long term (a dropped or added frame will change the phase). Just use TDecimate():
Code:Mpeg2Source("filename.d2v") TDecimate()
-
As I had read about Decimate and Telecine I guess I need to generate a new source before going on with the workflow. Or not?
If TDecimate get rid of duplicate frames, it do so in real time? And what about that RemoveSpots? It is a temporal function, i.e., it needs back and forward frames to remove spots. Then, I cannot use TDecimate at the same script.
If I understand right, I need to apply TDecimate first, generate a new source and then, apply a second script with RemoveSpots. Right? But, doing so, I have to take care about the encode bit rate to avoid too much loss of quality during the first pass.
Am I talking bullshit?Thank you. -
Yes, you are talking bullshit! Each line in a AviSynth script only gets the video after being processed by the lines above it. So you can do everything in one script:
Mpeg2Source("filename.d2v")
TDecimate()
RemoveSpots() -
@jairo:
In a previous thread I presented some opening titles from the movie "LILI". This is a long ongoing one, so rather than duplicate the first 2 pages I refer you to it here. It starts with some captures of the spot problems:
https://forum.videohelp.com/threads/340688-YAVHSR-%28Yet-Another-VHS-Restoral%29-LILI
A direct link to an abbreviated (edited) MPEG (12 MB) of the opening title section is posted under under the captures:
http://dc426.4shared.com/download/PqmKlPjf/Sample_Spots.mpg
In the first post in that thread I presented a script using one method for clearing spots without IVTC. This is NOT the easiest way, and is time-consuming. So in the course of this thread I had to get familiar with IVTC and RemoveSpots().
Here's a brief rundown of what I did. First, you need a section of the original source that you want to work with. The source that I IVTC'd was an MPEG recorded off digital cable. I created a .d2v of the original source (referred to in the script below). The audio was a separate .wav joined later to the resulting AVI in VirtualDub. I did this to clean the audio elsewhere and for denoising, etc., with smaller AVI's.
The IVTC script uses two functions contained in TIVTC.dll. TFM() is a preparatory step to help spot duplicate frames, dropped frames, etc., then TDecimate() does the heavy work. The script with IVTC and RemoveSpots (don't forget to change the path and filename for your video and its location):
Code:MPEG2Source("E:\Videos\LIL2\LIL13_56k\TitleE\LIL56k_TitleApcm.d2v") TFM(d2v="E:\Videos\LIL2\LIL13_56k\TitleE\LIL56k_TitleApcm.d2v",order=1).TDecimate() RemoveSpots() ConvertToRGB32(matrix="Rec601",interlaced=true) return last function RemoveSpots(clip input, bool "_grey", int "repmode") { _grey=default(_grey, false) repmode=default(repmode, 16) clmode=17 clensed=Clense(input, grey=_grey, cache=4) sbegin = ForwardClense(input, grey=_grey, cache=-1) send = BackwardClense(input, grey=_grey, cache=-1) alt=Repair(SCSelect(input, sbegin, send, clensed, debug=true), input, mode=repmode, modeU = _grey ? -1 : repmode ) restore=Repair(clensed, input, mode=repmode, modeU = _grey ? -1 : repmode) corrected=RestoreMotionBlocks(clensed, restore, neighbour=input, alternative=alt, gmthreshold=70, dist=1, dmode=2, debug=false, noise=10, noisy=12, grey=_grey) return corrected }
RemoveDirt requires RemoveGrain() v.09, available here: http://home.pages.at/kassandro/RemoveGrain/RemoveGrain.zip
From this package copy RemoveGrain.dll and Repair.dll into Avisynth plugins. I also copied SSE2Tools.dll, but I don't think it matters. The html documentation that comes with RemoveGrain is here: http://www.removegrain.de.tf/
All these parts and pieces seem like a headache (they are), but it's easier that cleaning one spot at a time the way I did at first. The video on which I used this script had a few remaining spots that I cleared later. RemoveSpots often leaves a few remnants during fade-in and fade-out or some scene changes. I use another method to clean the few remaining spots, which I can explain later. You might not require additional de-spot. -
@sanlyn
as usual, your explanations are excellent! I was away from computer for a couple days and could not reply your suggestions.
@all
I still have some doubts. I tought that temporal filters, like denoise ones, had working using more than one frame, back and foward. I do know that Avisynth scripts process one line each time and gets the video after being processed by the previous line. It is very clear. Then, that's the why I put the question at #11. This make sense to have another video source, to give room for the temporal filter work more than one frame, back and forward. Having more than only one frame, IMHO, it's more accurate to detect the actual noise and to avoid remove the actual details.
But, as it was explained, that's not that way that the things work.
You guys, talked about TDecimate to remove duplicated frames. Will I get a shorter video after that? If I have 1 duplicated frame each 4, after TDecimate my video will be 25% shorter than the original one. How to deal with the sound? (After apply TFM and TDecimate I realized that the size of the video was the same).
Applying RemoveSpots with AvsPmod I put some sliders to follow variables:
repmode, clmode, gmthreshold, noise and noisy.
I'm looking for the one that could handle the size of the dot. I couldn't get better results altering that values. As it was said before, RemoveSpots have to be applied with default settings. But, if it has parameters, and if each video has its own characteristics, why not try to change them, trying to improve the action of the filter?
And, finally, what do you thing about to use NeatVideo after RemoveSpots? That's to remove some soft vertical stripes along the video.
I'm trying to find time to read and understand the RemoveDirt's and RemoveGrain's documentation. Too many parameters and very complex material.
At overall, I could good results with the suggestions given here, but I'm trying still better ones.
Thank you. -
@sanlyn,
parameter for TFM here isn't order=1. It return and error (The field order does not match). I used order=0.
Code:Mpeg2Source("vigilante_or1.d2v") a=TFM(d2v="vigilante_or1.d2v",order=0).tdecimate() crop(a,0,6,720,474).AddBorders(0, 6, 0, 0, color=$000000) function RemoveSpots(clip input, bool "_grey", int "repmode") { _grey=default(_grey, false) repmode=default(repmode, [<"repmode", 1, 18, 16>]) clmode=[<"clmode", 10, 20, 17>] clensed=Clense(input, grey=_grey, cache=4) sbegin = ForwardClense(input, grey=_grey, cache=-1) send = BackwardClense(input, grey=_grey, cache=-1) alt=Repair(SCSelect(input, sbegin, send, clensed, debug=true), input, mode=repmode, modeU = _grey ? -1 : repmode ) restore=Repair(clensed, input, mode=repmode, modeU = _grey ? -1 : repmode) corrected=RestoreMotionBlocks(clensed, restore, neighbour=input, alternative=alt, gmthreshold=[<"gmthreshold", 10, 100, 70>], dist=1, dmode=2, debug=false, noise=[<"noise", 0, 16, 10>], noisy=[<"noisy", 0, 20, 12>], grey=_grey) return corrected } b=RemoveSpots(RemoveSpots(RemoveSpots(RemoveSpots(RemoveSpots(RemoveSpots(RemoveSpots(RemoveSpots(RemoveSpots(RemoveSpots(RemoveSpots(last))))))))))) ConvertToRGB32(matrix="Rec601",interlaced=true) b #StackHorizontal(a,b)
Thank you. -
jairo: Field order will differ for different sources, so adjust for each case. The documentation for TFM() gives these values for "order":
Code:-1 = auto (use avisynth's internal parity value) 0 = bottom field first (bff) 1 = top field first (tff)
Last edited by sanlyn; 11th Jan 2012 at 06:00.
-
-
Crop is fixing the noise at the top of the video. Look the comparison images at post #7.
Multiple RemoveSpots is to improve the final result. If I apply the function only one time, some dots still remain.
First image is the use of the function as you said (one time).
Second image is the use of multiple functions applied recursively:
frame #56
Note the big dot pointed by the arrow. Only one use of RemoveSpot didn't remove it. Several uses did that.
If I'm misunderstanding the use of the function, let me know. Or if there is another way to do, please, tell me.
@gavino
The number of frames remains the same. The duration too. I thought the number of frames would be decreased. The best news is the audio won't be affected.
@jagabo,
I'll study DeScratch and find out how to apply it here. Thx for the tip. Would you mind to be more specific and write some examples of DeScratch and GrayScale?Thank you. -
It's been a long time since I used DeScratch() and I only used it once or twice. So I don't remember much about it. GrayScale() is easy -- there are no parameters to set if the video is in YUV form. If it's in RGB form you can specify the conversion matrix.
-
You probably see that RemoveSpots can leave a few remnants. It tends to fall short during scene transitions, dissolves, fades, and some instances of movement. There's an "MC" version of RemoveDirt that reportedly handles these problems, but I've never seen anyone who can make "MC" load except programmers who understand .NET filter creation. There are tips on making MC work, but it doesn't happen for most users. It isn't developed with the same programming tools used by most Avisynth plugins.
That said, your iteration seems to work. I haven't seen the video, but judging from your captures the bottom image shows that multiple runs have somewhat softened the video and removed some detail. The method I used to clear remaining spots is explained in the link to the thread I referred to earlier. It's somewhat complicated. But it doesn't re-filter the whole video or the change the entire frame. Here it is:
Apparently you've measured where the spot is located in the frame. My method would replace only the cropped portion instead of the whole frame. In playback the cleaned spot would be difficult if not impossible to detect. Using the "before" frame that you posted, I would isolate a portion of the frame with these coordinates:
top left corner: x=322, y=166
top right corner: x=334, y=166
bottom left corner: x=322, y=174
bottom right corner: 334, y=174
These x,y coordinates define a rectangle that is 12 pixels wide by 8 pixels high. The image below has a small pink rectangle (not the exact size measured here) that outlines the portion to be covered up.
If you crop the 720x480 frame to make a small frame that isolates that rectangle, the statement would be Crop(322,166,-386,-306). This statement crops 322 pixels from the left, 166 pixels from the top, 334 pixels from the right, 306 pixels from the bottom.
This script takes in a source file, calls it "a1", then uses Avisynth variables to represent different parts of the original a1 and build a new video (Out2), replacing only bad rectangles instead of the entire video. The example below with 5 bad spots in 4 consecutive frames is a bit extreme, but I did use it and it did work. The 5 bad spots occurred during a fade-in from black.
Code:# ###################################################### # # This example cleans spots from 5 frames in clip "a1" # The frames with spots are 100, and 1136 thru 1139 # Required AviSynth plugin: RemapFrames.dll # Required AviSynth plugin: BadFrames.dll # Works in YV12, YVY2, RGB # ###################################################### AviSource("enter path to AVI previously run with RemoveSpots\filename.avi") a1=last # <- original source a2=a1 # <- temporary clip for cropping, discarded later # -- As a naming convention, "Out" clips are accumulated cleaned frames from "a1" # -- example of operation on 1 spot in 1 frame (frame 100). # -- b0 = temporary copy of accumulated frames overlaid with cleaned frames # -- b1 = temporary clip that is a rectangle taken from a CLEAN frame using Crop() # -- b2 = temporary clip that will be overlaid with the clean rectangle # -- Out1 = clip that will contain the cleaned-frame version of a1 b0=a1 # ----- BadFrames() replaces 100 with blended clean frames 99 + 100 ----- b1=a2.BadFrames(100).Crop(322,166,-386,-306) # - (replace 100, then crop clean rectangle) b2=Overlay(b0,b1,x=322,166) Out1=ReplaceFramesSimple(a1,b2,mappings="100") # - Out1 = all of a1 with 1 frame replaced # -- example of operation on 5 bad spots that appear on 4 consecutive # -- frames. "Out1" below is clip of accumulated "clean" frames from # -- previous steps. The frames are consecutive, so FrezeFrame is used # -- to move clean frames into 2 areas, then BadFrames to slightly blur # -- the remaining 2. # -- First get 4 clean frames, then get 5 clean rectangles. b0=Out1 # <- start by re-setting b0 # ---- First, use b01 to contain 4 "cleaned" frames ---- b01=a2.FreezeFrame(1136,1136,1135).FreezeFrame(1138,1138,1140).BadFrames(1137,1139) # ---- Now save 5 clean rectangles from b01 ----- b02=b01.Crop(162,246,-522,-212) b03=b01.Crop(198,254,-508,-206) b04=b01.Crop(212,268,-474,-206) b05=b01.Crop(246,258,-418,-152) b06=b01.Crop(312,338,-360,-128) # ---- Build a clip that replaces 5 "bad" rectangles with 5 "clean" ones ---- # ---- Note that this replaces every respective rectangle in the entire ---- # ---- "work copy" of the video. So the last step is to accumulate clean ---- # ---- frames into B11, which is used to replace 4 frames in Out2 output ---- b07=Overlay(b0,b02,x=162,y=246) b08=Overlay(b07,b03,x=198,y=254) b09=Overlay(b08,b04,x=212,y=268) b10=Overlay(b09,b05,x=246,y=258) b11=Overlay(b10,b06,x=312,y=338) Out2=ReplaceFramesSimple(Out1,b11,mappings="[1136 1139]") Return Out2 # <- new copy of a1 with 5 bad frames replaced, all others intact.
http://dc426.4shared.com/download/PqmKlPjf/Sample_Spots.mpg
This is the first few seconds of that clip after RemoveSpots, RemoveDirt, and 3 or 4 added routines like the script above. Note: this isn't the IVTC'd version, it's the original telecine.
http://dc234.4shared.com/download/_feM7ikb/Jackie_fix.mpgLast edited by sanlyn; 11th Jan 2012 at 11:32.
-
Last edited by jagabo; 11th Jan 2012 at 11:07.
-
sanlyn,
using your script:
Sript error: The named argument "x" was passed more than once to Overlay,
line:
b2=Overlay(b0,b1,x=322,166)
What is wrong? -
That's what's known as OOOPS! Gotta be careful when changing old scripts for new purposes. Cut-and-paste can be dangerous. I corrected the line in my post. It should read:
Code:b2=Overlay(b0,b1,x=322,y=166)
-
Before you said, I've already tryed with y=166. But I got another error:
RemapFrameSimple: value out of bounds in <mapping> string: 1136 (line offset 1)
at the same line.
That line with Overlay caught my attention and I didn't see the duration of the video reffered in RemapFrameSimple and BadFrames.
Now it is working fine. I'd adjust the values according. Thx.
(Too many "books" to study tonight: RemoveGrain, RemoveDirt, RemapFrameSimple, DeScratch, BadFrames...)Thank you. -
I'm with you, jairo, I find some of it beyond my expertise. Keep in mind that RemoveSpots is just one of many modifications of RemoveDirt(). This one with default values is a contribution from jagabo. I find different versions of RemoveDirt() all over the internet, though the logic is the same. Another script/plugin that does great work but can drive you crazy is MCTemporalDenoise(). Just one look at the "Required Filters" on MCTD's download page is enough to frighten one away. Fortunately, the "default" versions I've found of these heavy-duty scripts work well. It just takes time to learn complex plugins a small piece at a time. Don't feel alone.
-
Yes, it occurred to me you might try running the script unchanged on another video, so I should have posted a warning earlier. The line that reveals how many frames the script wants to work with is in the 2nd routine with 5 spots on 4 frames. The highest frame number in the script is 1140:
Code:b01=a2.FreezeFrame(1136,1136,1135).FreezeFrame(1138,1138,1140).BadFrames(1137,1139)
ReplaceFramesSimple is a command in RemapFrames(). I know exactly what you mean! incompreensível !!!!Last edited by sanlyn; 11th Jan 2012 at 13:00.
-
sanlyn
yes, I want to test your other suggestion, Bob_And_CleanSpot. I'll adapt the values accordingly. It's not so incompreensível. I think I could understand that part of re-used variable names. And here, I do want to repeat:
Your explanations are excellent! They are not only good for me, but for anyone else who visit this thread in the future. It's a complete guide, step by step, in the thread and also within the script, full with comments.
In previous life you were a teacher and didn't sleep well until your pupils had got all the lessons.
About the video, I put "easy way" in the title because the dots behave in the same way along all the video duration, i.e., small dots blinking fast around the screen. Here is not possible to use a individual treatment for each dot, like you did with Lili. Because the dots blink fast, during few frames, they are different from the details, that keep the same (or almost the same) along the time. For this reason, I visualized one treatment that could get more than one frame at a time (back and foward from the current one) to detect just blinking noise, leaving the details alone. The problem is that the dots appear for more than one frame, and RemoveSpots cannot know if it's noise or detail. Comparing just some few frames it is possible to know who is the noise.
But I understand that it is not the way the things work. The current results are good enough.
That's the reason for an "easy way". If I catch the right script, it could be applied for the whole video, because the behaviour of the dots is the same.
But, like my father used to say when we want only easy things, "Don't think that the Heaven is so near"
(I'm not sure if I translated correctly)
"Não vá pensando que o Céu é perto."Thank you. -
Uh, well, I've been known to pass on some confusing and sometimes just plain incorrect stuff , but I try. Ask our friends jagabo and Gavino (they catch me every time!).
The script is really a post-processing cleanup. Try IVTC and RemoveSpots first, then pick out the most annoying spots later. I don't think you can clean every dot and nit of grain, probably the movie looks better with a sort of grainy, gritty effect. Too much denoising looks unnatural, anyway. Later, I did IVTC the "LILI" titles, and that and RemoveSpots saved a lot of work! I attached the long script to demonstrate various methods for cleaning spots and junk from dissolves, motion, etc. that RemoveSpots refused to fix. The script also shows how multiple frames can be despotted (after all, the spot is usually in the same place for every frame).
You might look into MCTemporalDenoise referenced earlier. Its download page has an installer package with the ton of filters MCTD requires. A headache to set up, but -- it's usually run at simple default settings, a short statement like MCTemporalDenoise(settings="high") or MCTemporalDenoise(settings="medium") should clean up quite a bit without softening everything. Run MCTD first followed by RemoveSpots in the same script for grimy video. You'll be surprised at how many of those spots are gone before RemoveSpots is run. If you think NeatVideo is slow, wait until you see MCTD.Last edited by sanlyn; 11th Jan 2012 at 15:51.
Similar Threads
-
Scripts and filters for an Opera (sample videos included)
By jairovital in forum RestorationReplies: 317Last Post: 31st Oct 2011, 09:23 -
Are these VHS tapes fixable? youtube sample included.
By kcr in forum RestorationReplies: 11Last Post: 29th Sep 2010, 07:37 -
Need help in deinterlacing an old VHS converted video (Sample included) ..
By ahmadka in forum RestorationReplies: 22Last Post: 30th Jun 2010, 12:43 -
Help with Color Correction (Sample Included)
By mlong30 in forum RestorationReplies: 9Last Post: 13th Mar 2008, 23:01 -
Magnetized VHS - How to fix it? (Sample included)
By djnice in forum RestorationReplies: 16Last Post: 18th Sep 2007, 13:14