anime filtering via avisynth

[beav]

Hey guys! I'm new to this forum; would anybody be willing to help me out? I have this crappy R2 DVD source and am trying to remove the ample amounts of grain. So far, no luck. The more I try, the more detail I loose to the point where it just looks washed out or over-blurred. This is what I have in my script so far:
global MeGUI_darx = 4
global MeGUI_dary = 3
LoadPlugin("C:\Users\Beav\Desktop\MeGUI_dev[0.3.4.12]\MeGUI_dev\tools\dgindex\DGDecode.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\RemoveGrainS.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\dfttest.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\gradfun2db.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\MSharpen.dll")

DGDecode_mpeg2source("C:\Users\Beav\Downloads\Sail or Moon Collection\Sailor Moon Video\Japanese\Anime (DVD)\Season 1\Disc 1\VIDEO_TS\VTS_01_1.d2v")
RemoveGrain(mode=1)
Dfttest(sigma=64)
gradfun2db(thr=3)
MSharpen(threshold=1, strength=70, mask=false)
crop(4, 0, -4, 0)
Spline36Resize(640,480) # Spline36 (Neutral)

[Selur]

your source is telecined (MeGui should be able to detect this through it's interlace analysis option)
-> you should run IVTC before doing anything else, otherwise you totally break you source
other than that a line darkening filter like Hysteria, might be good idea to make the lines a bit darker.
I wouldn't use gradfun2db or mSharpen and I wouldn't resize to but these are more a matter of taste, the main thing is that you need to run some sort of IVTC, before doing anything else.

[x264]

Use TIVTC or AnimeIVTC

TFM().TDecimate()

or
AnimeIVTC(mode=1#if it's hard telecined), also try switching removegrain and dfftest with mctemporaldenoise.

[beav]

Correct me if I'm wrong, but I believe my source is 97% film. Therefore, I should be able to force film in dgindex right?

[Selur]

yup, if FILM is at a percentage higher or equal to 95% or if it only shows FILM normally you can simply activate Forced FILM.
If your content is anime, I would recommend to do a visual check to be sure everything is fine. (animes tend to be mastered horrible )

[beav]

Just to be certain, I'll do an IVTC instead.

[beav]

What kind of denoiser or smoothener should I use? Any recommendations?

[sanlyn]

What kind of denoiser or smoothener should I use? Any recommendations?

MCTemporalDenoise might work, but won't clean the lines enough or get the mosquito noise on edges. It's really noisy, not with grain so much as with compression artifacts, mosquito noise, etc. Lines are a bit ragged, but colors and levels are wrecked. No simple fix here. As this comes from tape (yechh!), color balance and levels change from one camera shot to another. Could use a little more middle red, but I ran out of time.


[Attachment 17419 - Click to enlarge]

[Attachment 17420 - Click to enlarge]

[Attachment 17421 - Click to enlarge]

Main tools were ColorYUV, Smooth Levels, Smooth Tweak, GradFun2DBmod, and these heavy hitters for lines and noise:

TemporalDeGrain()
w = width
h = height
nnedi3_rpow2(opt=2,rfactor=2,cshift="spline64resize").TurnLeft().NNEDI3().TurnRight().NNEDI3().spline64resize(w,h)

The script is slow. And I mean slow. Quickies won't fix this nightmare. Color and levels were tweaked in VirtualDub with gradation curves and Color Mill. Encoded with HCenc. It's not telecined, so you'll need to apply 3:2 pulldown on re-encode. Could still use some tweaks: there are some spots and frame jitter. Try Stab(). Remove !00% of all grain, and you'll get banding from the original.

[beav]

Think it would be possible to make it look like this? As in the smoothening? without the warpsharpening of course.

[sanlyn]

Sure, just use MCTD at "very high" and leave the levels and chroma super-high to mask the banding (start looking at the kid's hair at about frame 330 in your "007" sample"). The two mkv's are over filtered and washed out, but that makes banding tougher to see for those who don't know better. I got better results with NeatVideo on the earlier sample, but I don't like to recommend filters that many people don't have. I don't think I'd like watching distorted color like that for more than a few seconds. That's just me.

[sanlyn]

Even better, after TFM/Decimate run QTGMC with InputType=1 and medium or fast preset. It would look smoother than TemporalDeGrain.

[jagabo]

Some global compromise adjustments in AviSynth:

import("C:\Program Files (x86)\AviSynth 2.5\plugins\FastLineDarkenMod.avs")
import("C:\Program Files (x86)\AviSynth 2.5\plugins\Stab.avs")

Mpeg2Source("VTS_01_1.d2v", CPU=6, Info=3) 
TFM(d2v="VTS_01_1.d2v") 
TDecimate() 
ColorYUV(cont_y=50, off_u=6, off_v=-13) # contrast, white balance
ColorYUV(cont_u=75, cont_v=75) # saturation
Stab().Crop(2,2,-0,-0).AddBorders(2,2,0,0) #stabilize
McTemporalDenoise(settings="very high") # denoise
FastLineDarkenMod(75) # darken outlines

You can tweak each of the filters to your liking. The biggest remaining problem may be that blacks are dark blue rather than black. But the only shots with a significant amount of black (near the end) are otherwise very blue so I'm not sure they were supposed to be black. It looks like I crushed those blacks too. A little less contrast stretch is in order. Maybe: change the first ColorYUV() to ColorYUV(cont_y=35, off_y=10, off_u=6, off_v=-13).

[beav]

How about this? It is not over-smoothened, but I need a second opinion.

[beav]

Also, is there a filter that allows me too change the amount of red? The pinkish tint is annoying.

[jagabo]

Both Sanlyn's and my samples removed the red tint from the whites.

[beav]

What do you think of the final?

[sanlyn]

The last sample is a big improvement. Nice work!

[beav]

Thank-you! It's all thanks to your help! One problem, the lines look kinda ghosted.

[jagabo]

the lines look kinda ghosted.

They have slight over sharpening halos.

[beav]

LSFmod() did that. I just don't like linedarkeners.

[beav]

script

[jagabo]

LanczosResize() is responsible for some of it.

[beav]

Changed it to spline36

[beav]

Also, do I detect some color bleeding?

[Brad]

Boo white balance. Sailor Moon was always tinted.

[sanlyn]

Judging from the VOB samples posted earlier, the lines in the source aren't in such great shape. It's very difficult to repair that sort of problem if it exists in the source. Things can be improved somewhat, but basically you're asking a filter to re-draw the image. Filters can do some good work, but I don't think they're that smart.

[beav]

Hey guys! The sample I'm about to post is from the Italian DVDs, which are waaayyyy better quality then the R2J. One problem with the source, It looks like a frame has been dropped, which creates a jump forward in the motion, followed by a duplicate frame to bring everything back in sync. Therefore, ruining the framerate is there any way to fix this?

[poisondeathray]

Hey guys! The sample I'm about to post is from the Italian DVDs, which are waaayyyy better quality then the R2J. One problem with the source, It looks like a frame has been dropped, which creates a jump forward in the motion, followed by a duplicate frame to bring everything back in sync. Therefore, ruining the framerate is there any way to fix this?

I don't see any foward jumping, only duplicate frames added

The only sequence where there is steady motion is the last few seconds , there the duplicates are easily seen ; for the other parts it' s difficult to tell what is a "normal" duplicate vs. added duplicate

MPEG2Source()
AssumeTFF()
TFM()
TDecimate(mode=2, rate=23.976)

or TDecimate(mode=1, cycleR=1, cycle=25) , might be ok too, but it returns 24.0 FPS

[beav]

Is it possible to make the Italian footage look like this: the script below is what I have so far, I used the same grain cleaners as the person who made the sample I'm posting, but don't want to sharpen it the same way. I want the lines to be thin and dark but without all the halos and oversharpening. I tried LSFmod but it doesn't make the affect I'm looking for.
# Set DAR in encoder to 4 : 3. The following line is for automatic signalling
global MeGUI_darx = 4
global MeGUI_dary = 3
LoadPlugin("C:\Users\Ryan\Desktop\MeGUI_dev[0.3.4.12]\MeGUI_dev\tools\dgindex\DGDecode.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\RemoveGrainS.dll")
LoadPlugin("C:\Users\Ryan\Desktop\MeGUI_dev[0.3.4.12]\MeGUI_dev\tools\avisynth_plugin\FluxSmooth.dll")

DGDecode_mpeg2source("C:\BISHOJO_SENSHI_SAILOR_MOO N_01\VIDEO_TS\VTS_03_1.d2v", cpu=4, info=3)
AssumeTFF()
TFM()
TDecimate(Mode=1,Cycle=25,CycleR=1)
RemoveGrain(mode=1)
FluxSmooth_FluxSmoothST()
Spline36Resize(720,540) # Spline36 (Neutral)

[poisondeathray]

but don't want to sharpen it the same way. I want the lines to be thin and dark but without all the halos and oversharpening.

Thin , dark lines in animated content usually means awarpsharp (or awarpsharp2) , + line darkeners (e.g. toon, fastlinedarken/mod, hysteria etc...)