Waviness in image - can anything be done about this?

[brassplyer]

I find I might reduce that waviness in spots on video but not consistently throughout.

You need to train NV for the noise in each specific shot. And only apply it to those shots.

Ah. I don't know if you looked at the sample I posted but the noise moves, it's not static as in the screen shot you treated. This might explain why I'm not having great success.

[jagabo]

I find I might reduce that waviness in spots on video but not consistently throughout.

You need to train NV for the noise in each specific shot. And only apply it to those shots.

Ah. I don't know if you looked at the sample I posted but the noise moves, it's not static as in the screen shot you treated. This might explain why I'm not having great success.

Moving makes it easier for NV, not harder.



That was without NV's temporal filter. The hum bars are still slightly visible after NV but that's the least of the problems with the clip! I didn't address any other issues, just the hum bars.

[sanlyn]

Here's a 7 sec sample I carved out with Vegas Pro. It's been reconstituted but looks like the original for all intents and purposes.

Horrible mpg. But a good example of creative ways to wreck a video. I didn't need NeatVideo to clear the floating lines, but I did use it as a mild sharpener.

[brassplyer]

Horrible mpg. But a good example of creative ways to wreck a video.

What can I say it's been through unknown generations of copying over decades in addition to probably conversion from the original format. Whoever made the DVD this came from used a variable bitrate of only about 4000kbs for who knows what reason.

Thanks for the script, I'll check it out.

[sanlyn]

No need to apologize for someone else. Hope the script proves useful.

[brassplyer]

ED: Sorry, I forgot to post the text file with script notes. Might prove helpful.

Hunted down what Avisynth wiki says is needed for SmoothLevels, getting this error:

[sanlyn]

You are using a later version of SmoothAdjust than I did. The later version has auto smooth-dithering (which many find annoying in some cases).
Remove the "smooth=" parameter from the line. Change the original:

SmoothLevels(8,0.75,255,20,255,chroma=200,limiter=0,tvrange=true,smooth=200,dither=100,protect=6)

to this:

SmoothLevels(8,0.75,255,20,255,chroma=200,limiter=0,tvrange=true,dither=100,protect=6)

SmoothAdjust is pretty cool sometimes, but the last two versions are giving some users fits. I use v2.6.2.0 from July/2013.
And getting tired of a new one showing up every couple of months.

[brassplyer]

[Attachment 22112 - Click to enlarge]

That was without NV's temporal filter. The hum bars are still slightly visible after NV but that's the least of the problems with the clip! I didn't address any other issues, just the hum bars.

I see the file you're processing has an .avs extension, what are you loading into Virtualdub?

[sanlyn]

[Attachment 22112 - Click to enlarge]

That was without NV's temporal filter. The hum bars are still slightly visible after NV but that's the least of the problems with the clip! I didn't address any other issues, just the hum bars.

I see the file you're processing has an .avs extension, what are you loading into Virtualdub?

In the image of VirtualDub in jagabo's post, the script being run is an .avs file, and at the same time virtualDub filters are loaded and filtering the script's output. You can run the output of an avs script with VirtualDub filters -- test a few frames first, because a slow script coupled with some combinations of VDub filters can slow processing to a crawl. You also have to manage the YUV->RGB->YUV conversions, which can sometimes be a problem (and sometimes not). I used the dither plugin for those conversions, mainly because I already had a skeleton script set up for it. In this case, the video has been encoded and re-recorded so many times, it likely wouldn't make a difference.

[brassplyer]

You are using a later version of SmoothAdjust than I did. The later version has auto smooth-dithering (which many find annoying in some cases).
Remove the "smooth=" parameter from the line. Change the original:

Making progress. After hunting down things to make Mpeg2Source work, DGindex etc, I'm now getting this error.

[brassplyer]

I see the file you're processing has an .avs extension, what are you loading into Virtualdub?

In the image of VirtualDub in jagabo's post, the script being run is an .avs file, and at the same time virtualDub filters are loaded and filtering the script's output.

I was wondering what specific script he was using, if it's something besides a simple ffmpegsource2 conversion or whatever. I like the results in the frame grab but haven't managed to duplicate it with NV so far.

[brassplyer]

[Attachment 22112 - Click to enlarge]

That was without NV's temporal filter. The hum bars are still slightly visible after NV but that's the least of the problems with the clip! I didn't address any other issues, just the hum bars.

Can you give an idea of what section of noise you're sampling here and what settings you're using? So far I haven't managed to duplicate this with NeatVideo

Follow-up - I see going to the original mpg makes a difference. I was trying it on a previously cleaned section that was exhibiting the waviness. Loading the original mpg file with ffmpegsource2 I get a lot closer to the results in your screenshot.

[jagabo]

That was without NV's temporal filter. The hum bars are still slightly visible after NV but that's the least of the problems with the clip! I didn't address any other issues, just the hum bars.

I see the file you're processing has an .avs extension, what are you loading into Virtualdub?

Only:

 Mpeg2Source("noise sample2.d2v", CPU=2, Info=3)

In VirtualDub I ran Bob Doubler, NV, Interlace.

I didn't fine tune NV much (I just cranked all the high and mid frequency sliders all the way down, and the low frequency sliders all the way up) so Sanlyn's script gives less blurring. The part of his script that does the hum bar removal is

AssumeBFF().QTGMC(preset="fast")
c=last
e=c.SelectEven().TurnRight().RemoveDirt(20,0,false).TemporalSoften(4,4,8,15,2).TurnLeft()
o=c.SelectOdd().TurnRight().RemoveDirt(20,0,false).TemporalSoften(4,4,8,15,2).TurnLeft()
Interleave(e,o)

[sanlyn]

Making progress. After hunting down things to make Mpeg2Source work, DGindex etc, I'm now getting this error.


[Attachment 22129 - Click to enlarge]

I sometimes get that same QTGMC error from its support files when using Avisynth.dll 2.6 instead of 2.5.8. 2.6 is inconsistent, and different defaults in different plugins give different results. I really get tired of fiddling with 2.6; I load it up now and then, but eventually I encounter problems with many plugins. If you can work your way through the maze of MT settings and whatnot, it usually works. But IMO life is too short for all that. QTGMC has a set of special mods for MT versions of Avisynth -- but you have to keep fiddling with those as well, so it soon isn't worth the hassle.

I have a subfolder in System32 (or SYSWOW64 if you're in newer Windows) that has avisynth.dll 2.5 in one subfolder and AVisynth.dll 2.6 in the other. I keep switching them back and forth into the "normal" system folder, but 2.6 will eventually give me a problem every time. I really wish they'd get 2.6 straightened out.

The dither plugins will work with both versions.If dither gives you a problem, just don't use the "dither" lines. Do this:
- Change the dither YV12 and RGB conversions to plain old "ConvertToRGB32...." and "ConvertToYV12...".
- Replace GradFun3 with GrandFun2DBMod(thr=1.5, mask=false)"
- This means that you won't have to load any of the dither support plugins.

If you have problems with any of the plugins, give a holler. All of them have been uploaded to recent threads and I've kept links to them.

The part of the script that got rid of about 90% of the rolling bars is:

AssumeBFF().QTGMC(preset="fast")
#....................
c=last
e=c.SelectEven().TurnRight().RemoveDirt(20,0,false).TemporalSoften(4,4,8,15,2).TurnLeft()
o=c.SelectOdd().TurnRight().RemoveDirt(20,0,false).TemporalSoften(4,4,8,15,2).TurnLeft()
Interleave(e,o)

Low-power NeatVideo was used mainly to just clean up some of the remnants, which wasn't very much, so really strong (often too destructive) NeatVideo wasn't required. Unfortunately NV was wasn't really designed for that kind of noise.

[jagabo]

You are using a later version of SmoothAdjust than I did. The later version has auto smooth-dithering (which many find annoying in some cases).
Remove the "smooth=" parameter from the line. Change the original:

Making progress. After hunting down things to make Mpeg2Source work, DGindex etc, I'm now getting this error.


[Attachment 22129 - Click to enlarge]

Use SetMtMode() at the start of your script. If you're using Mpeg2Source:

SetMtMode(5,x) # x = number of threads you want
Mpeg2Source(...)
SetMtMode(2) # if all your filters support mode 2
filters...

[brassplyer]

The part of the script that got rid of about 90% of the rolling bars is:

AssumeBFF().QTGMC(preset="fast")
#....................
c=last
e=c.SelectEven().TurnRight().RemoveDirt(20,0,false).TemporalSoften(4,4,8,15,2).TurnLeft()
o=c.SelectOdd().TurnRight().RemoveDirt(20,0,false).TemporalSoften(4,4,8,15,2).TurnLeft()
Interleave(e,o)

Low-power NeatVideo was used mainly to just clean up some of the remnants, which wasn't very much, so really strong (often too destructive) NeatVideo wasn't required. Unfortunately NV was wasn't really designed for that kind of noise.

Once it finally dawned on me that your original script is actually *2* scripts, I wrestled with it some to get QTGMC to work and ended up with this combined with the elements from my basic QTGMC deinterlace script. Couldn't get DeHalo to cooperate either, I just got rid of the Dehalo & dither altogether, got tired of getting error messages.

Ended up with this.

SetMTMode(3, 4)
MPEg2Source("F:\Chase test.d2v")
AssumeBFF()
ConverttoYv12(interlaced=true)
SetMtMode(2) # uses less memory than mode 3
QTGMC (Preset="medium")
c=last
e=c.SelectEven().TurnRight().TemporalSoften(4,4,8, 15,2).TurnLeft()
o=c.SelectOdd().TurnRight().TemporalSoften(4,4,8,1 5,2).TurnLeft()
Interleave(e,o)
GradFun2DBmod(thr=1.5,mask=true)
AddGrainC(2.0,2.0)
ConvertToRGB32
Crop(8,22,-42,-18).Spline36ReSize(720,480)
return last

So, I combined the above .avs with NeatVideo and got a pretty decent looking result.

However, I haven't been successful getting the 2nd part of the process - the DeBlock_QED which I gather is the crucial part of the script is hanging me up.

Went to this page

http://avisynth.nl/index.php/Deblock_QED

...copied the text, saved as an .avsi file put it in the plugins folder...

...then got the elements it references, DeBlock, (already have MaskTools2) and DctFilter. For DeBlock and DctFilter - I only copy the .dll files to the Avisynth plugins folder, correct?

Attempt to open this script

AviSource("E:\chase test lagarith.avi")
ConverttoYv12()
ColorYuv(gamma_y=-5,cont_u=-30)
SmoothTweak(brightness=-3,contrast=1.1,saturation=1.15,hue1=-1)
DeBlock_QED(quant1=12,quant2=14)
LSFMod(strength=150)
# --- help to mask low bitrate mpg artifacts from Vegas encode ---
GradFun3(thr=1.2,mask=0)
AddGrainC(1.5, 1.5)
AssumeBFF().SeparateFields().SelectEvery(4,0,3).We ave()

...and get this error:

[jagabo]

See post 45.

[sanlyn]

Many plugins are non-MT versions that usually work with MT mode 2. That's just a big hassle for me, and if you move the filter in the sequence you have to remember to change modes when necessary.

Once it finally dawned on me that your original script is actually *2* scripts....

Yes, I saved the results of VirtualDub as RGB and then ran that output with a second script. With a very slow script, that also can make many scripts run 3 to 4 times faster due the way Avisynth stacks stuff in memory. Actually you can load many VirtualDub filters inside an avs script using LoadVirtualDubPlugin(). NeatVideo and Camcorder Color Denoise are two such filters. That way, you can run everything in one script. The downside is that the VDub filter parameters are hard-coded in the script, so every time you tweak the settings you have to change the script. That can be a hassle with a few filters. For example, this is how you would load a gradation curve filter in an avs script:

# ----------- Load Curves plugin -----------------------
# ---- set the path to VirtualDub plugin and give a nickname to the filter -------
LoadVirtualDubPlugin("D:\VirtualDub\plugins\gradation.vdf","Curves1",1)
# .....
# ....later in the same script ...
#
ConvertToRGB32(matrix="Rec601",interlaced=true)  #<- or interlaced = false,as required)

Curves1(1,"0001010202030404050606070809090a0b0c0d0e0f1011121314151618191a1c1d1e20212224252628292a2b2d2e2f30313334353637383"\
+"a3b3c3d3e4041424344454748494a4b4d4e4f50515354555657595a5b5c5d5f6061626365666768696a6c6d6e6f7071737475767778797a7c7d7e7"\
+"f808182838485868788898a8b8c8d8e8f90919293949596969798999a9b9c9d9e9e9fa0a1a2a3a4a4a5a6a7a8a8a9aaabacadadaeafb0b0b1b2b3b"\
+"4b4b5b6b7b7b8b9babbbbbcbdbebebfc0c1c1c2c3c4c4c5c6c6c7c8c9c9cacbcccccdcececfd0d1d1d2d3d3d4d5d5d6d7d7d8d9d9dadbdbdcdcddd"\
+"ededfdfe0e1e1e2e2e3e4e4e5e5e6e6e7e8e8e9e9eaeaeb00010203040405060708090a0b0c0d0d0e0f101112131415161718191a1b1c1d1e1f202"\
+"1222324252627282a2b2c2d2e2f3031323435363738393a3c3d3e3f40414344454647484a4b4c4d4e5051525354565758595a5c5d5e5f606263646"\
+"56768696a6b6d6e6f70717374757677797a7b7c7d7f8081828385868788898a8c8d8e8f9091929495969798999a9b9d9e9fa0a1a2a3a4a5a6a7a8aa"\
+"abacadaeafb0b1b2b3b4b5b6b7b8b9babbbcbdbebfc0c1c2c3c4c5c6c7c8c9cacacbcccdcecfd0d1d1d2d3d4d5d5d6d7d8d8d9dadadbdcdcdddeded"\
+"fdfe0e0e1e1e2e2e3e3e4e4e4e5e5e6e6e6e7e7e7e8e8e8e9e9e9e9eaeaeaebebebebebebebebebebebeb00010203040506070708090a0b0c0d0e0f"\
+"101112131415161718191a1b1c1d1e1f2022232425262728292a2b2d2e2f3031323334363738393a3b3c3e3f4041424345464748494a4b4d4e4f505"\
+"1525455565758595b5c5d5e5f60626364656667696a6b6c6d6e6f71727374757678797a7b7c7d7e8081828384858687898a8b8c8d8e8f9092939495"\
+"969798999a9b9c9e9fa0a1a2a3a4a5a6a7a8a9abacadaeafb0b1b2b3b5b6b7b8b9babbbdbebfc0c1c3c4c5c6c7c9cacbcccdcfd0d1d2d3d4d5d6d7d8"\
+"d9dadbdcdddedfdfe0e1e1e2e3e3e4e4e5e5e6e6e6e7e7e7e8e8e8e8e9e9e9e9e9eaeaeaeaeaeaeaeaebebebebebebebebebebebebebebebebebebeb"\
+"00010203040405060708090a0b0c0d0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f3132333435363738393b"\
+"3c3d3e3f4041424445464748494a4c4d4e4f50515354555657585a5b5c5d5e6061626364656768696a6b6d6e6f70717374757677797a7b7c7d7f8081"\
+"828385868788898a8c8d8e8f90929394959697999a9b9c9d9e9fa1a2a3a4a5a6a7a9aaabacadaeafb0b1b2b4b5b6b7b8b9babbbcbdbebfc0c1c2c3c4c"\
+"5c6c7c8c9cacbcbcccdcecfd0d1d1d2d3d4d5d5d6d7d8d8d9dadadbdcdcdddddedfdfe0e0e1e1e2e2e3e3e4e4e5e5e5e6e6e7e7e7e8e8e9e9e9eaeaee"\
+"bebebecececededededeeeeeeefef000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2"\
+"d2e2f303132333435363738393a3b3c3d3e3f404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f6061626364656667686"\
+"96a6b6c6d6e6f707172737475767778797a7b7c7d7e7f808182838485868788898a8b8c8d8e8f909192939495969798999a9b9c9d9e9fa0a1a2a3a4a"\
+"5a6a7a8a9aaabacadaeafb0b1b2b3b4b5b6b7b8b9babbbcbdbebfc0c1c2c3c4c5c6c7c8c9cacbcccdcecfd0d1d2d3d4d5d6d7d8d9dadbdcdddedfe0e"\
+"1e2e3e4e5e6e7e8e9eaebecedeeeff0f1f2f3f4f5f6f7f8f9fafbfcfdfeff,15222220a06070702000019142625393c586176839ca6c4c5dcd6ffeb"\
+"00001614818fafc0d2ddf5eb0000161537398997adbfc8ddf1eb00001513727b9dadb5c6d4deffef0000ffff")

Fortunately most VDUb plugins don't require that much verbiage for loading! When you have set up the curves filter to get what you want, save the VDub settings in a .vcf file. Then open the file (a .vcf file is a plain text file), copy the values, paste them into the script, and format that long string as shown. You can guess what happens if you decide to change the filter settings. What that plugin will do with MT modes is anyone's guess. NeatVideo v3 is a multithreading plugin. This is the sort of complication that prevents me from using Avisynth MT all the time. If you keep two separate copies of Avisynth.dll in a separate location and keep switching them into the system folder back and forth, it solves a lot of problems -- for me, anyway.

BTW, GradFun3 is the 16-bit version of GradFun2DBmod. For all practical purposes, they are two versions of the same plugin. There are other versions floating around. I think GradFun3 is more effective against banding and has a smoother look to it. When tryi9ng different temporal smoothers, I first used TTempSmooth instead of TemporalSoften. I was surprised that TTempSmooth didn't work as well even at strong settings. The video was too soft for it anyway.

[brassplyer]

ED: Sorry, I forgot to post the text file with script notes. Might prove helpful.

In this part of your script I take it the deblocking is supposed to do exactly what it sounds like, minimize the little squares/delineations that can be seen, correct? Dunno, I don't see a difference. Two screen grabs from VirtualDub - one is with the deblocking. Can you tell which is which? Virtualdub deblock filters never seen to do much either. I disabled addgrain and the reinterlace. I disabled Gradfun3 - still working on figuring out how to successfully install the dither package which I gather GradFun3 is part of. Also disabled LSFMod sharpener since it seems to make the blocking worse.

SetMtMode(5,2) # x = number of threads you want
SetMtMode(2) # if all your filters support mode 2 filters
AviSource("E:\chase test lagarith.avi")
ConverttoYv12()
ColorYuv(gamma_y=-5,cont_u=-30)
SmoothTweak(brightness=-3,contrast=1.1,saturation=1.15,hue1=-1)
DeBlock_QED(quant1=12,quant2=14)
#LSFMod(strength=150)
# --- help to mask low bitrate mpg artifacts from Vegas encode ---
#GradFun3(thr=1.2,mask=0)
#AddGrainC(1.5, 1.5)
#AssumeBFF().SeparateFields().SelectEvery(4,0,3).W eave()

[sanlyn]

Some scenes might need deblocking, some might not. DeBlock does some smoothing of large areas of solids and gradations to prevent banding when you encode. You can
strengthen deblocking where needed by replacing Deblock_QED(quant1=12,quant2=14) with DeBlock_QED(), which uses the defauilt values for quant1 and quant2
(the defaults are around 24, I think). You can set either or both up to 64, which is pretty strong.

You can try the mods you listed, but watch out for banding in those smooth background areas in the encode. That's what the GradFun's and AddGrainC were used
to prevent. Some scenes might need it, some might not.

The bright levels in the images are clipped to the point of being completely blown out and discolored, invalid even for PC display.

[brassplyer]

Some scenes might need deblocking, some might not. DeBlock does some smoothing of large areas of solids and gradations to prevent banding when you encode. You can
strengthen deblocking where needed by replacing Deblock_QED(quant1=12,quant2=14) with DeBlock_QED(), which uses the defauilt values for quant1 and quant2
(the defaults are around 24, I think). You can set either or both up to 64, which is pretty strong.

You can try the mods you listed, but watch out for banding in those smooth background areas in the encode. That's what the GradFun's and AddGrainC were used
to prevent. Some scenes might need it, some might not.

The bright levels in the images are clipped to the point of being completely blown out and discolored, invalid even for PC display.

Thanks for all the input.

There's no question the video has problems that are unfixable short of hiring Industrial Light & Magic. There's a chance I might be able to track down the tape the DVD was made from which would of course make for a better cleaning experience than going off a so-so quality DVD.

[jagabo]

I take it the deblocking is supposed to do exactly what it sounds like, minimize the little squares/delineations that can be seen, correct?

Yes. For example, a 4x enlargement of a blocky MPEG video:



After deblock_qed():


This type of deblocking works best if the blocks are at their original mod8 locations (because it looks for 8x8 blocks, starting at a corner). And it's usually best to run it early in a script, before the blocks get messed up.

[brassplyer]

Some questions:

Re: MPEg2Source("Drive:\path\noise sample2.d2v") Why use a .d2v file? Also, what's the difference between using MPeg2Source and ffmpeg2Source?

Also why the various colorspace conversions between the two scripts?

ColorYUV(cont_y=-45,off_y=-15,gamma_y=-10)
ColorYUV(cont_v=-105,cont_u=-90,off_u=-2)
Dither_convert_rgb_to_yuv(matrix="601",interlaced= false,tv_range=true,cplace="MPEG2",lsb=false,outpu t="YV12")
ColorYuv(gamma_y=-5,cont_u=-30)

[sanlyn]

Some questions:

Re: MPEg2Source("Drive:\path\noise sample2.d2v") Why use a .d2v file? Also, what's the difference between using MPeg2Source and ffmpeg2Source?

A d2v index file is created because MEG2Source uses the d2v index to decode the MPEG2 encoded file.

"ffmpeg2Source" is not a command or function. It's really ffmpegsource2 (or FFMS2), a library of ffmpeg functions used to decode a variety of formats, including MPEG2. When used with straight MPEG2 (aka "DVD"), it is not as frame-accurate as DGIndex/MPEG2Source. As the ffmpegSource2 User Manual states on page 1, paragraph 1, with MPEG2 it "May be frame accurate on good days" (http://ffmpegsource.googlecode.com/svn/trunk/doc/ffms2-avisynth.html). FFMS2 requires more resources and support plugins than does MPEG2Source.

Also why the various colorspace conversions between the two scripts?

ColorYUV(cont_y=-45,off_y=-15,gamma_y=-10)
ColorYUV(cont_v=-105,cont_u=-90,off_u=-2)
Dither_convert_rgb_to_yuv(matrix="601",interlaced= false,tv_range=true,cplace="MPEG2",lsb=false,outpu t="YV12")
ColorYuv(gamma_y=-5,cont_u=-30)

ColorYUV() does not convert colorspaces. http://avisynth.org.ru/docs/english/corefilters/coloryuv.htm . Or, on your own system go to All Programs, find the "Avisynth 2.5" program listing, open its program folder and click on "Avisynth Documentation".

The first script uses filters that work in YV12.
Virtualdub works with RGB.
Script #2 uses YV12.

If you want MPEG, it will be encoded as YV12. You can let your encoder make a quickie/garbage/hidden/anything-goes conversion to YV12, or you can use Avisynth to do it more carefully. If you're using a better class of encoder such as HCenc, it wants to see YV12 as input for MPEG. When one consistently uses the usual NLE for what they like to call "editing", the NLE makes these conversions in the background without telling you. Maybe that's another reason why the noise and color in the sample mpg are so ugly.