Red and Blue Climbing Histogram Walls

[koberulz]

I'm constantly running into this issue, which I think is red and blue clipping? Luma seems to be fine, but the only way I can get the red and blue back into range is to almost entirely desaturate the picture.

Original frame attached, as well as a Color Tools histogram from my result, which is after running ColorYUV(off_u=-11,off_v=2,cont_u=-50,cont_v=-50).

Any thoughts?

[jagabo]

If you look at where the illegal colors are you'll probably see most of them are in the halos from the analog sharpening filter, some from YV12 chroma subsampling, probably very few from real picture elements. Basically, nothing to worry about. But, as usual, you should provide a video sample.

[koberulz]

How does one find "where the illegal colors are"?

[jagabo]

Many editors have an out-of-range function. In AviSynth you can use ShowBadRGB() or HightlightBadRGB(). The former is very slow to start up (it may take a minute or more depending on the speed of your CPU) but is very accurate. The latter is fast but a little less accurate. ShowBadRGB() can be found here:

https://forum.videohelp.com/threads/360935-Capturing-Correct-Chroma-and-Hue-Levels-Fro...HS#post2289269

HighlightBadRGB() can be found a few posts later. Here's a slightly improved version:

function HighlightBadRGB(clip vid, int "color")
{
  color = default(color, $ff0000)

  badcolor = BlankClip(vid, color=color)
  Subtract(ConvertToYV24(vid), ConvertToYV24(vid).ConvertToRGB().ConvertToYV24())
  absY = Overlay(ColorYUV(off_y=-126), Invert().ColorYUV(off_y=-130), mode="add")
  absU = Overlay(UtoY().ColorYUV(off_y=-128), UtoY().Invert().ColorYUV(off_y=-128), mode="add")
  absV = Overlay(VtoY().ColorYUV(off_y=-128), VtoY().Invert().ColorYUV(off_y=-128), mode="add")
  Overlay(absU,absV, mode="add")
  Overlay(last,absY, mode="add")
  ColorYUV(gain_y=65000)
  Overlay(vid,badcolor,0,0,last)
}

It assumes limited range rec.601 video.

Note that the former function shows only out of range colors, the rest of the image is black. The latter shows the image with out-of-range areas in bright red (or other specified color).

[koberulz]

This looks like pretty significant portions of the image to me, in the darks moreso than in the highlights.

[jagabo]

This looks like pretty significant portions of the image to me

Not really. Those are all from oversharpening halos, the blurry chroma of VHS, and chroma subsampling. What you want to avoid is larger areas of illegal colors. For example, if you pump up the saturation you'll see the yellow around the bird in the logo at center court becomes illegal.

If you want, you can purify some of the brights with something like:

Overlay(last, last.greyscale(), mask=ColorYUV(off_y=-200).ColorYUV(gain_y=4000).Blur(1.5))

Adjust the ColorYUV parameters to suit your video.

You can do something similar for dark areas.

[koberulz]

Could you explain what that script is doing?

[jagabo]

It's desaturating bright areas by overlaying a greyscale image over the original using a mask so that only the brightest areas are overlaid. The mask is constructed by first subtracting 200 from the Y values (everything below 200 becomes 0) then brightening the mask so that the brightest parts of what's left become 255. gain_y=4000 is roughly equivalent to multiplying by 16. So basically, in the mask, everything below 200 becomes black, every thing above 216 becomes full white.

[koberulz]

And what would the equivalent for the darks be? Could I just precede the ColorYUV portion with an Invert("Y")?

What would I be looking for to tell me I need to tweak the numbers you use?

[jagabo]

And what would the equivalent for the darks be? Could I just precede the ColorYUV portion with an Invert("Y")?

Yes, you can do it that way. But just Invert() which inverts Y, U, and V. Only Y is used by the mask function of Overlay() so it doesn't matter what happens to U and V.

What would I be looking for to tell me I need to tweak the numbers you use?

Look at the mask. I usually do something like:

WhateverSource()
bmask = ColorYUV(off_y=-200).ColorYUV(gain_y=4000)
Interleave(last, bmask)

Then open the script in VirtualDub and make sure the mask corresponds to the areas I want. Once I have the mask I change the code to use it rather than view it.

[koberulz]

Even 200 on the low end affects way too much, almost completely desaturating the blue jerseys. And yet, there's still a ton of out-of-range stuff. How does that work if the jerseys themselves are in-range?

[jagabo]

Even 200 on the low end affects way too much

Then use a higher value. Or use further masking to protect the dark blues. Protecting dark blues will be hard because blue contributes very little to luma. RGB ramps, luma graph at top:


[Attachment 49500 - Click to enlarge]

And purifying the blacks and whites won't help at the border between red and yellow for example.

like I said, you'll always have this problem with VHS caps. Nobody bothers "fixing" it.