Interpreting VirtualDub Capture Histogram

[jagabo]

There seems to be some confusion here. From what I have read, the darkest luma level (black) should be set around RGB=16. Correct?

No. The darkest luma should be Y=16. When converted to RGB that becomes RGB=0. The brightest luma is Y=235. When converted to RGB that becomes 255. And all the luma levels in between are spread out equally. I explained this in post #16. And referenced it several times since then.

But, the letterbox area of the battleship image needs to be 0? Yet, there is only a 4-5 RGB point difference between the letterbox black and the black of the active image. This would indicate that the letterbox black should be set around RGB=11. Which is it?

The letterbox bars should be Y=16 and RGB=0.

[Sharc]

There seems to be some confusion here. From what I have read, the darkest luma level (black) should be set around RGB=16 .....

Hmmm confusion yes....
The conversion YUV <-> RGB depends on the conversion matrix which is in effect.
As I understand it:

a. according to ITU-R BT-601 Studio encoding parameters of digital television for standard 4:3 and wide-screen 16:9 aspect ratios:
Black: Y'CbCr(16,128,128) <-> RGB(16,16,16)
White: Y'CbCr(235,128,128) <-> RGB(235,235,235)
This seems to leave headroom for RGB also.

b. according to Intel and some other documents (https://software.intel.com/content/www/us/en/develop/documentation/ipp-dev-reference/t...?language=en):
Black: Y'CbCr(16,128,128) <-> RGB(0,0,0)
White: Y'CbCr(235,128,128) <-> RGB(255,255,255)

[jagabo]

a. according to ITU-R BT-601 Studio encoding parameters of digital television for standard 4:3 and wide-screen 16:9 aspect ratios:
Black: Y'CbCr(16,128,128) <-> RGB(16,16,16)
White: Y'CbCr(235,128,128) <-> RGB(235,235,235)
This seems to leave headroom for RGB also.

Limited range RGB is a special editing mode that retains out of gamut YUV on conversion to RGB. This allows you to make RGB adjustments without crushing out of gamut colors. You can get this conversion in AviSynth with ConvertToRGB(matrix="PC.601") or ConvertToRGB(matrix="PC.709").

But when you view video and images on a computer monitor black is at RGB=0, white at RGB=255.

[Sharc]

a. according to ITU-R BT-601 Studio encoding parameters of digital television for standard 4:3 and wide-screen 16:9 aspect ratios:
Black: Y'CbCr(16,128,128) <-> RGB(16,16,16)
White: Y'CbCr(235,128,128) <-> RGB(235,235,235)
This seems to leave headroom for RGB also.

Limited range RGB is a special editing mode that retains out of gamut YUV on conversion to RGB. This allows you to make RGB adjustments without crushing out of gamut colors. You can get this conversion in AviSynth with ConvertToRGB(matrix="PC.601") or ConvertToRGB(matrix="PC.709").

But when you view video and images on a computer monitor black is at RGB=0, white at RGB=255.

Ah this special editing mode explains it and makes sense. Thank you for clarification.

[jagabo]

If you want to see actual YUV values on your computer screen use a script that looks like this:

AviSource("testXP4a.avi") 
StackHorizontal(ColorYUV(cont_u=-256, cont_v=-256).Subtitle("Y"), UtoY().Subtitle("U"), VtoY().Subtitle("V"))
ConvertToRGB(matrix="PC.601")

[Attachment 58729 - Click to enlarge]

You can then read YUV values directly off the screen with csamp, VirtualDub, or some other screen color tool.

In VirtualDub add the crop filter and hold down a Shift key while moving the mouse cursor over the preview image:


[Attachment 58730 - Click to enlarge]

You can't see the mouse cursor in the screenshot but it was over the top letterbox bar. The RGB values in the image reflect the Y values in the Y portion of the image, the U values in the U portion of the image, and the V values in the V portion of the image. So the Y value of the top letterbox bar is 35.

[anachronon]

Yes, this is craziness. I even found an entire web page, that is devoted to discussing this confusion:

https://www.fourcc.org/fccyvrgb.php

Then, there is the question of which standard a particular capture diver uses. PC.601 is the standard for old SD analog video, and PC.709 is the standard for HD. Does that mean that an analog SD capture device is using PC.601? Not necessarily! Which, gets into a lot of second guessing that could make one's brain explode.

So, from what I see, it is best to ignore how YCbCr is being handled. The ultimate objective is to obtain the best RGB output. This gets back to doing test captures of the most problematic scenes, examining individual frames, and adjusting settings to yield the most acceptable RGB histograms:


[Attachment 58734 - Click to enlarge]

Note that in the top image, the "black" of the letterbox area is still around RGB 8. However, the heel of the histogram is at 0. Nothing is being crushed, other than a bit of noise and a few sharpening artifacts. The blacks of the second image still appear a little muddy---even though the histogram shows some details being crushed. Old analog video was never very good at handling darker areas.

[Sharc]

@jagabo: Nice! Thank you for these tricks. I must bookmark this.

[anachronon]

I fear that I am a bit lost, as to how to use scripts. I have seen scripts mentioned many times. But, I have never seen where they come into play.

[jagabo]

Yes, this is craziness. I even found an entire web page, that is devoted to discussing this confusion:

https://www.fourcc.org/fccyvrgb.php

The first pair is technically YCbCr but many programs, including AviSynth, call it YUV. That's what you're dealing with here. limited range rec.601 standard definition video.

full range RGB to limited range YCbCr:

Y  =      (0.257 * R) + (0.504 * G) + (0.098 * B) + 16
Cr = V =  (0.439 * R) - (0.368 * G) - (0.071 * B) + 128
Cb = U = -(0.148 * R) - (0.291 * G) + (0.439 * B) + 128

limited range YCbCr to full range RGB:

B = 1.164(Y - 16)                  + 2.018(U - 128)
G = 1.164(Y - 16) - 0.813(V - 128) - 0.391(U - 128)
R = 1.164(Y - 16) + 1.596(V - 128)

The others deal with unit values (Y = 0.0 to 1.0, Cb/CR -0.5 to +0.5) but are otherwise the same. MJPEG uses full range YUV internally. What comes out of the MJPEG decoder is full range RGB.

Then, there is the question of which standard a particular capture diver uses. PC.601 is the standard for old SD analog video, and PC.709 is the standard for HD. Does that mean that an analog SD capture device is using PC.601?

SD video cap will almost always be rec.601. Whether is full range or limited range depends on your proc amp settings. But you want your final output to be limited range YCbCr.

So, from what I see, it is best to ignore how YCbCr is being handled.

No. Limited range YCbCr (YUV) is universally used with DVD, BD, online streaming, etc. If you follow the rules for YCbCR/YUV you won't have problems with out of gamut RGB when the video is displayed (assuming you equipment is set up properly).

The ultimate objective is to obtain the best RGB output.

Only if you are doing the conversion correctly. If you are doing them incorrectly. And just because you think something looks better doesn't mean it is correct. People of like extreme contrast and saturation, even when it's far outside of spec and unrealistic. In some cases (hill/tree silhouette home video) it's up to you what you want it to look like. The LD presumably conforms to international video standards. The letterbox bars should be black, not dark grey.

[anachronon]

Yes, I understand the formulas mentioned. My question involves how to apply scripts in software, like VDub. I have worked with programs capable of running scripts. However, I have yet to find this feature in either VDub or GraphStudio.

Though I have an extensive background in both analog video production and TV/VCR repair, digital video capture is new to me. That said, I do have an extensive background in scanning and editing still images from film. It just took me a minute to comprehend all of the arcane and esoteric concepts being thrown around in digital video capture. For one, I had to realize that the YUV colorspace was not the same as the YIQ colorspace, with which I was familiar. YUV is the newer analog "component" output standard, used by digital devices like DVD players. YIQ is the older standard, still used in NTSC composite and S-video. Though 'Y' is the same in both standards, the I and Q color axes are not the same as the U and V axes. Everyone thoroughly confused now?

Which, brings me to the confusion over 'Y' value numbers being thrown around. Previously, I had only seen the 'Y' component in analog form, on a waveform scope. I had to mentally convert the numbers into a visual image, to comprehend what was being described. "Black" is set to Y=16, so that the horizontal and vertical blanking could be set at Y=0 (below black). This was done to keep the electron gun retrace lines from becoming visible on the old CRT's. Y=235 is equivalent to the "yellow" on a VU meter in audio recording, and Y=255 is the red (though in setting video 'Y' levels, avoiding the "yellow" can be more critical, due to possible later effects of adding chroma).

Finally, there is the question of the capture device's purpose in the workflow. Because the capture device is part of a video production, one tends to view it as a piece of studio equipment. In fact, the capture device is actually a television receiver! It converts an analog video signal into its RGB components, just as a TV would. So, I have to approach setting levels, as I would with a customer's set on the test bench. As a final step with any customer repair, I would calibrate the picture tube. I would connect the set to a colorbar generator, set on grayscale. I would then adjust the "screen" (brightness) to get a good black, and the "gain" (contrast) to get a good corresponding white. This had to be done visually.

Luckily, with a digital RGB output, I now have an RGB histogram to work with. This merely involves exporting a video frame to any image editor. Here is where my experience with image editing comes into play. In image editing, a good "black" is defined as anything between 2 and 10, and a good "white" between 245 and 253---not 0/255! In fact, most pro-level film scanners will set black/white at 5/250. Many print houses require ranges no greater than 2/253 This is to avoid clipping the heel and toe of the histogram, where a few trailing values could reach 0 and/or 255. This may seem trivial. But, hard-smashing blacks/whites against the 0/255 limits produces a harshness clearly visible to the naked eye. Leaving the heel and toe of the histogram produces an image that seems much-more natural (note histograms in #36 above). I don't see why the same concepts shouldn't be applied to video.

[anachronon]

By the way, if anyone has an old, working analog colorbar generator (NTSC capable) that they wish to sell for cheap, I would be interested. Preferably, one with S-video out, and the ability to display a step grayscale.

[jagabo]

I fear that I am a bit lost, as to how to use scripts. I have seen scripts mentioned many times. But, I have never seen where they come into play.

You can open AviSynth scripts direclty with VirtualDub as if they are video files, File -> Open Video File... VirutualDub2 has a built in AviSynth script editor. AVSPmod is a third part AviSynth script editor that provides a lot of hand holding. You need to install AviSynth and probably a lot of third party filters. I recommend AviSynth+ and VirtualDub2.

https://www.videohelp.com/software/AviSynth-Plus
http://avisynth.nl/index.php/AviSynth%2B
https://www.videohelp.com/software/VirtualDub2

One thing to beware of with VirtualDub (all versoin) is that all filtering used to work only in RGB. More recent versions allow filters to work in YUV but many filters have not been updated. So you may end up with a rec.601 conversion (crushing superblacks and superwhites) before many filters.

[anachronon]

Thanks for that. I'll have to look into it. Eventually, I want to start editing, and putting together presentations from my master copies. Scripts may help with more complex tweaks.

In my version VDub, YUV adjustments can be applied to the capture preview. But, all filters are disabled, once capture is started. I have been using GraphStudio to set my levels.

[jagabo]

Here's a frame from a commercial DVD:


[Attachment 58756 - Click to enlarge]

The letterbox bars are at Y=16. The black level of the active picture is around Y=18 in this frame, the white level around Y=235. Obviously, not every shot will have full black and full white. But this is typical of the extremes.

[anachronon]

That looks pretty good. Here are the resulting RGB histograms of the active image:


[Attachment 58757 - Click to enlarge]

Though the blacks and whites aren't being truly crushed, they are rather crowded. Mostly, it is the reds in the shadows, and the greens and blues in the sky. For making master copies, I would try backing-off the proc-amp contrast by one notch. However, proc-amp settings are rather course. So, if the new setting leaves too much space at the ends of the histogram, I would have to live with the current setting.

For me, the idea of a master copy is to capture as much info from the tape as possible. I can always make minor tweaks to the blacks and whites in post, when I make scene-by-scene edited copies. Of course, if I leave too much space at the ends of the histogram, the resulting edited copies could look posterized. It's a compromise, no matter what.

Which again brings up the question of the difference between DVD's and camcorder tapes. Do camcorder tapes have the same range of 'Y' as a commercial DVD? If the camcorder Y-range is slightly less, than the setting that you currently have, might be perfect. My reason for trying to set range with a DVD, is to avoid unnecessary wear on my old tapes.

[anachronon]

i just realized another reason for calibrating levels by RGB output---the issue of "tolerances". Consumer analog components typically come with a 10% tolerance. A 10% tolerance could mean an error of as much +/- 25 RGB steps! Professional gear and test equipment may go with the pricier 5%. Still, even that could mean an error of +/- 12 steps. Just because the input is by-the-book, doesn't guarantee that the output will be right.

[jagabo]

Which again brings up the question of the difference between DVD's and camcorder tapes. Do camcorder tapes have the same range of 'Y' as a commercial DVD?

DVDs are digital of course. Whereas old consumer camcorders were analog. In theory they represent the same thing. In reality, analog devices are all over the place.

[Sharc]

So, from what I see, it is best to ignore how YCbCr is being handled.

No. Limited range YCbCr (YUV) is universally used with DVD, BD, online streaming, etc. If you follow the rules for YCbCR/YUV you won't have problems with out of gamut RGB when the video is displayed (assuming you equipment is set up properly).

But as I understand it even when we adjust the proc amp to deliver valid YCbCr/YUV within limited range (16...235/240), it is still possible that we get out of gamut RGB after conversion, no? (Inner/outer cube). It happens for example to my VHS captures where RED and BLUE is sometimes getting clipped at 255 or 0, while the captured YUV triplets are all sane. Not to say that it looks bad or that I would see color distortions. When I apply post capture corrections to make RGB become legal the picture may look washed or 'milky'. Do I miss something and should I revisit the proc amp settings?

[jagabo]

Yes, RGB is where the rubber hits the road. The ultimate goal is to get good RGB images on-screen. And yes, most YUV combinations, even within the stated limited range of Y=16 to 235, UV=16 to 240, lead to illegal RGB colors. But if you adjust only Y to get legal RGB you will end up with washed out (low contrast) video. Sometimes you need to work in YUV, sometimes in RGB. You can also work in limited range RGB to retain colors that would have fallen slightly outside the full range RGB cube. And yes small excursions usually don't lead to visual problems on modern equipment and software. Long ago there would sometimes be problems where values would "wrap around". RGB values that would have been negative become large positive values (-1 becomes, 255, -2 becomes 254, -3 becomes 253, etc.) and RGB values that would have been over 255 become small positive values (256 becomes 0, 257 becomes 1, 258 becomes 2, etc). That's a catastrophe, visually.

One way to check for out-of-gamut YUV is to convert to RGB then convert back to YUV. Values that change by more than a unit or two were outside the RGB cube (integer rounding and chroma subsampling errors will cause some of the values within the cube to change by a unit or two).

[Sharc]

Thanks. Very much what I thought. Proc amp settings can be quite laborious to satisfy both legal YUV and legal RGB for a good visual experience at the same time. Fortunately I don't get the wrap-around RGB problem with current playback devices, so I can live with minor RGB violations.

[lollo]

When I apply post capture corrections to make RGB become legal the picture may look washed or 'milky'

I experienced the same.

For example, I have an YUV input range after capture 17-254, that I move to YUV 16-235 before AviSynth filtering to respect the "rules".

I apply then TemporaleDegrain2 and LSFmode (for example) and the YUV levels after filtering are 0-255: the filters expanded the range!

If now I re-apply the rules and shrink to YUV 16-235 the video look washed when watched on PC or on TV.

My (maybe wrong) conclusion is that I have to fix the levels for a proper filtering but it is not necessary to do it for the watching experience at the end.

Some image/sample of what I experienced here: http://www.digitalfaq.com/forum/video-restore/10776-levels-before-after.html

[jagabo]

When I apply post capture corrections to make RGB become legal the picture may look washed or 'milky'

I experienced the same.

For example, I have an YUV input range after capture 17-254, that I move to YUV 16-235 before AviSynth filtering to respect the "rules".

I apply then TemporaleDegrain2 and LSFmode (for example) and the YUV levels after filtering are 0-255: the filters expanded the range!

Yes, of course. Sharpening produces halos.

If now I re-apply the rules and shrink to YUV 16-235 the video look washed when watched on PC or on TV.

Clamp rather than adjust brightness/contrast/levels. ColorYUV(opt="coring") or Tweak(coring=true) or Levels(coring=true). Of just leave the small peaks there -- that's one reason why there's headroom and footroom in the limited range spec.

[lollo]

Yes, of course. Sharpening produces halos.

Well, in reality TD2 alone already expands the levels from 16-235 to 8-248, even if it declares "to do not sharp more than the original".

The next filter, LSFmod, further expand 8-248 to 0-255

Or just leave the small peaks there

That what my conclusion as well. Actually, when I talk of expanded range 0-255 it only concerns small peaks, the 99% of the Histogram is inside 16-235 range even after filtering.

Just to say that I learned to do not blindly trust the values of i.e. ColorYUV(analize=true) and build a sort of "automatic flow" for levels correction, but always look at the Histograms as well, to understand the distribution and the peaks location, and decide case by case what to do.

[Sharc]

Also, it is usually a good idea to switch sharpeners or other "picture quality enhancers" in the VCR and/or passthrough device (TBC) and capture device OFF (if possible) because these tend do produce halos and sharpen the noise. It better to leave this to postprocessing.

[anachronon]

Yes, this is one of the conundrums of setting by RGB. A "normal" outdoor, daylight scene is best for judging RGB levels---one with both bright areas and deep shadows, but no heavily saturated colors. Conversely, night scenes with bright saturated colors, like night neon, will have unavoidable illegal RGB values. Ironically, the illegal values are not usually found in the bright lights themselves---but in the saturated halos around the lights. Counterintuitively, some of the RGB values in these areas can be driven below 0 by the saturation. I am sure that we have all seen police videos, where the flashing blue lights produce areas that look unnaturally darker than their surroundings. Here is an example that I found online. Note the RGB separations that I have added:


[Attachment 58772 - Click to enlarge]

Colorful sunsets can also be problematic. When deciding whether to use a certain image for RGB reference, it is best to view the RGB channels first (as above) to make sure that there aren't any unnatural dark halos. A good test frame is one like the trees and hillside that I posted in #36. Or, a scene like the one that Jagabo posted in #44.

[jagabo]

Well, in reality TD2 alone already expands the levels

FFT (used by TD2) will create halos too: https://www.chemicool.com/definition/fourier_transform.html

Encoding with DCT based codecs will create them too.

[lollo]

FFT (used by TD2)

I do not use FFT3DFilter in post-processing (postFFT=0) and I thought that FFT used by TD2 was only to generate internal "calm/denoised clip" for detecting motion vectors.

I will look again to TD2 script, just for my understanding

[_Al_]

Ironically, the illegal values are not usually found in the bright lights themselves

That is what jagabo and Sharc are trying to tell. jagabo posts that RGB cube within YUV coordinates.
Look here fore example: https://forum.videohelp.com/threads/395939-ffmpeg-Color-Range/page6#post2574683 , there are illegal blue color values (very low values) within green grass color on the picture. The theme of that thread is how to eliminate it to have less than 3% of those illegal colors, but that's another story.

There is a script to show those illegal values as red, jagabo posted a script to show those illegal colors recently in avisynth, Sharc posted some scripts too. I have it here for vapoursynth, which would not help you. Question is though, if it makes any sense now, if video is not for distribution.

[anachronon]

The lossless master copies (of home videos) will not be for distribution---though I may send edited/restored/compressed clips to family/friends. In testing DVD's though, I have noticed a great variance in the settings of black/white levels. For example, the "black" of the letterbox area can vary as much as 8 levels, from one movie to another.

[poisondeathray]

The lossless master copies (of home videos) will not be for distribution---though I may send edited/restored/compressed clips to family/friends. In testing DVD's though, I have noticed a great variance in the settings of black/white levels. For example, the "black" of the letterbox area can vary as much as 8 levels, from one movie to another.

Professional retail DVD's ? Letterbox is always Y=16 on every one I've seen

Or did you mean "home video" ? Levels can be anything, color can be anything