Tweak Saturation Causing Color Issues

[killerteengohan]

You shouldn't be generating illegal colors in the first place. In that sense, clamping isn't necessary.

So would clamping even if their are no illegal values just to be sure, hurt anything noticeable?


Sure, Ill take the modded tool your offering.

[LMotlow]

Aside from levels issues, your sample shows other problems -- especially a lot of low-bitrate compression noise and artifacts. It seems the original video has been through too many re-encodes. So those areas of flat color won't ever look smooth without some cleanup. If those defects aren't addressed, they'll look a lot worse when you encode. I'll try to post some ideas for fixing later today.

[jagabo]

You shouldn't be generating illegal colors in the first place. In that sense, clamping isn't necessary.

So would clamping even if their are no illegal values just to be sure, hurt anything noticeable?

Clamping doesn't change anything if there are no YUV values outside their respective legal ranges. Ie, if there are no Y values less than 16 or greater than 235, and no U or V values less than 16 or greater than 240, clamping doesn't change anything at all. If there are Y, U, or V values below 16 they will be changed to 16. If there are Y values above 235 they will be changed to 235. If there are U or V values above 240 they will be changed to 240.

Sure, Ill take the modded tool your offering.

Starting with the ShowBadRGB() script in this post:

https://forum.videohelp.com/threads/360935-Capturing-Correct-Chroma-and-Hue-Levels-From...=1#post2289269

I added ShowGoodRGB():

function ShowGoodRGB(clip c, bool "pcRange", int "color") {
  mask = c.RGBMask(pcRange).Invert()
  mt_merge(c, c.BlankClip(color=color), mask, U=3, V=3, luma=true)
}

[LMotlow]

Man, that m2v sample is a real critter. It looks like it's been re-encoded and denoised so much, flat areas have major potholes where data has been scrubbed away. Some mottling effects, too, and even some flicker. Low bitrate artifacts are the devil to clean up. You have to smooth 'em out and then add fine noise to fill in the gaps. Likely it'll never look totally clean. I used six popular plugins to make the smooth areas look smooth:

Cnr2()
MCTemporalDenoise(settings="medium")
DeBlock() #(quant1=40,quant2=40)
TemporalSoften(5,4,8,15,2)
GradFun3(thr=0.5,mask=0)
AddGrainC(1.5, 1.5)

Added the usual aWarpSharp2 and DeHalo_Alpha. You can set color any way you want, but I used SmoothAdjust to control levels and gamma (which is too high in most scenes and looks a bit washed out) and added about 7% saturation with ColorMill, which let me control saturation in the darks, midtones, and brights separately. This style of anime doesn't usually use pumped color, so it looks a little out of place to me. Some of those scenes, you know, are supposed to be in dim light and use softer colors. But you can set it up however you want. The attached clip is soft-telecined.

[killerteengohan]

Aside from levels issues, your sample shows other problems -- especially a lot of low-bitrate compression noise and artifacts. It seems the original video has been through too many re-encodes. So those areas of flat color won't ever look smooth without some cleanup. If those defects aren't addressed, they'll look a lot worse when you encode. I'll try to post some ideas for fixing later today.

Its hard to believe they would release DVD's with video of re-encoded quality like that. They must not have cared about the buyers or decided to compress it to dvd5 instead of dvd9 .

Thanks for the suggestions when you get around to them.

[killerteengohan]

Thanks for the tool Jagabo!

[killerteengohan]

Man, that m2v sample is a real critter. It looks like it's been re-encoded and denoised so much, flat areas have major potholes where data has been scrubbed away. Some mottling effects, too, and even some flicker. Low bitrate artifacts are the devil to clean up. You have to smooth 'em out and then add fine noise to fill in the gaps. Likely it'll never look totally clean. I used six popular plugins to make the smooth areas look smooth:

Cnr2()
MCTemporalDenoise(settings="medium")
DeBlock() #(quant1=40,quant2=40)
TemporalSoften(5,4,8,15,2)
GradFun3(thr=0.5,mask=0)
AddGrainC(1.5, 1.5)

Added the usual aWarpSharp2 and DeHalo_Alpha. You can set color any way you want, but I used SmoothAdjust to control levels and gamma (which is too high in most scenes and looks a bit washed out) and added about 7% saturation with ColorMill, which let me control saturation in the darks, midtones, and brights separately. This style of anime doesn't usually use pumped color, so it looks a little out of place to me. Some of those scenes, you know, are supposed to be in dim light and use softer colors. But you can set it up however you want. The attached clip is soft-telecined.

wow that really darkened the lines up, and the awarp threw the colors around to where their bleeding over lines in several places.

It does look a bit cleaner though.

[LMotlow]

Some of that bleed was reduced with awarpsharp, DeHalo_Alpha and FixChromaBleeding plugins. Clearing it 100% isn't likely, as some of it came from a low bitrate. The sample was around 4700 kbps, which is about a 2-hour bitrate. That's not unusual for commercial anime, but it looks like re-encoded source to me. I've seen other work from that series, which is usually not so saturated. Juicing up the colors makes errors more obvious.

I didn't crop or resize. Cropping off the edges and resizing stretches the image horizontally and alters the original aspect ratio.

[2Bdecided]

Aside from levels issues, your sample shows other problems -- especially a lot of low-bitrate compression noise and artifacts. It seems the original video has been through too many re-encodes. So those areas of flat color won't ever look smooth without some cleanup. If those defects aren't addressed, they'll look a lot worse when you encode. I'll try to post some ideas for fixing later today.

Its hard to believe they would release DVD's with video of re-encoded quality like that.

IMO they didn't. It's perfectly normal SD video. When watched in the normal way (i.e. how people did in 1997 when DVDs were invented) it looks virtually flawless. It doesn't upscale very well, but it's better than 95% of DVDs, and 99% of SD broadcasts.

Cheers,
David.

P.S. Am I imagining it, or does that cross-fade during the second scene cut-across the 3-2 pulldown pattern? I haven't checked properly, I just watched it in real time.

[LMotlow]

The clip was IVTC'd for working. There's something wrong with the field structure anyway, and I saw what looked like a dropped frame near the end.

Yep, a lot of commercial anime is horrible.

[jagabo]

It's pretty obvious the DVD was made by capturing a studio video tape. The tape is responsible for the chroma smearking, red/green splotches (typical chroma noise from tape), and noise. MPEG encoding has accentuated the problems.

It's not uncommon for fade effects to be added after the film has been telecined, with no regard for the pulldown pattern.

[killerteengohan]

Would anyone be able to tell me why it is everytime I try to adjust brightness with smoothtweak(brightness=1.5) I get this error?

"The named argument "brightness" to smoothtweak had the wrong type"

I cant get it to adjust brightness at all no matter what number I pick

it works just fine when changing saturation like this smoothtweak(saturation=1.40)

[jagabo]

SmoothTweak(brightness) requires an integer, not a float. It's equivalent to off_y = ColorYUV().