Interpreting VirtualDub Capture Histogram

[anachronon]

I have a (hopefully) simple question. VDub has a histogram function, that can be used during capture. The histogram has no numbers/values. There is only the blue area (good) and the red areas (too low/high). So, what is the range of this blue area? Is it 0-255? Or, 16-235?

[Sharc]

As I understand it the blue range represents luma 16 ....235, assuming one captures in YUV (YCbCr) color format.

[anachronon]

Thanks. That is what I have been hoping.

[jagabo]

Yes, the blue portions represents Y = 16 to 235. Red (at either end) is outside that range. If you're going to make adjustments later it doesn't really matter if the cap has exactly the right levels. What you really want is to avoid crushing values at the left (black) and right (white) ends. Not all capture device crush at 0 and 255. Some crush at 16 and 235, or some other values.

[jagabo]

I made a quick simulation with AviSynth. Here's a greyscale ramp with a little noise:


[Attachment 58637 - Click to enlarge]

At the top right is the equivalent of VirtualDub's capture histogram -- Histogram(mode="levels"). You can see the roughly equal distribution of Y values from 0 to 255. The illegal values at the low and high ends (yellow) can't be differentiated in this image (standard rec.601 conversion to RGB) but they can easily be made legal (and the detail restored) with a few adjustments. But a capture device that crushes at Y=16 and Y=235 will yield a histogram that looks like:


[Attachment 58638 - Click to enlarge]

Notice how there are now big peaks at 16 and 235. That's because all the pixels that were below 16 now have the value 16. So there are now many more pixels with the value 16, leading to that big peak. All those pixels can no longer be differentiated so there is no way to restore the lost detail. The same happens above 235.

Proc amp adjustments on a device like this usually happen before the values are crushed. Adjusting the proc amp will let you retain the over-dark and over-bright pixels.

[anachronon]

Luckily, the Hauppauge USB-2 that I have been testing, does not destroy the values that cross into the red. And yes, the histogram in VDub does appear to show the 16-235 range. My only real gripe is with the Hauppauge driver. It wants to reset the proc amp values to default, every capture start/stop. I am using GraphStudio to set the proc amp, and that has kept the values from resetting during scene changes. But, the problem is still quite annoying/frustrating.

By the way, I have been using DVD's to experiment with proc amp settings---to avoid excessive wear on my old tapes. Is there a significant luma difference between a DVD and a Hi8/8mm tape? I will be using a Panasonic ES-15 on pass-through, to stabilize the tape signal. I am using that same ES-15 to play the DVD's that I am using in my tests.

[jagabo]

Is there a significant luma difference between a DVD and a Hi8/8mm tape?

In theory they are all the same. In reality every device/tape/shot is different.

[anachronon]

Thanks. Theory was all that I needed. I plan to set for the worst cases, as I will be capturing entire tapes. Specific scenes can then be fine-tuned later.

[jagabo]

I plan to set for the worst cases, as I will be capturing entire tapes. Specific scenes can then be fine-tuned later.

That's a reasonable approach.

[TimA-C]

Luckily, the Hauppauge USB-2 that I have been testing, does not destroy the values that cross into the red. ... It wants to reset the proc amp values to default, every capture start/stop. I am using GraphStudio to set the proc amp, and that has kept the values from resetting during scene changes. But, the problem is still quite annoying/frustrating.

YES!!!! I'm (fairly) sure that it never used to do this before last year? Are you using Windows 10? (And before people start jumping in and telling me that I shouldn't be using Win 10 . . . If I was using it seriously then I probably wouldn't be using Win 10. Alright? )

[anachronon]

I have actually experienced the Hauppauge proc-amp issue on two different computers, with two different driver versions. Yes, one is on a Win10 machine, 64-bit version. However, it also does it on an old WinXP, 32-bit machine.

[jagabo]

One other reason histograms can be misleading is oversharpening halos. Consider a fuzzy image as it might appear without VHS sharpening:


[Attachment 58655 - Click to enlarge]

The black and white lines at the far left have perfect levels, Y=16 for black Y=235 for white. With the typical VHS sharpening filters you get:


[Attachment 58656 - Click to enlarge]

The black and white levels are still at 16 and 235 but at the high contrast edges there are overshoots that extend into the illegal range. If you only look at a histogram you might be tempted to change levels to bring those pixels within the legal range:


[Attachment 58657 - Click to enlarge]

But then the overall image will have blacks that are two bright, and whites that are too dark.

[anachronon]

Yes, I have noticed this problem, even with sharpening set at the default level. To fine-tune my settings, I have been making additional test captures, and analyzing individual frames in Photoshop. Using this method, I can ignore the sharpening artifacts, and focus on the areas of importance. This has been a very tedious process.

Below is a frame from the trailer for "Dunkirk" (captured off of a DVD). The top is the full-color capture, and the bottom is a desaturated copy that I used for analysis. The black areas of the letterbox are around 23 (ignore the black retrace bar at right, as it is around 5). The glare of the explosion is around 240. So, I have a couple/few more tweaks to go. Or, is this "good enough"?


[Attachment 58677 - Click to enlarge]

I know that this is a rather "creative" approach. So, am I on the right track? I am also attaching the clip in question (it is only about one second long).

[lollo]

The glare of the explosion is around 240. So, I have a couple/few more tweaks to go. Or, is this "good enough"?

I don't know if I understand correctly, but I think that you are not considering the behavior of Hauppauge USB-Live 2 in your set-up flow: the card captures in the YUV range 16-254, so if the signal on the tape is outside this range, i.e. 5-255, you should tune the procamp to shrink the incoming range to 16-254 and capture everything with no "crushed" levels.

Later in AviSynth postprocessing you will adjust the YUV 16-254 range to 16-235 range, to stay inside "legal" YUV range for filter processing and eventually for RGB conversion if needed, for example with ColorYUV() and/or Levels()

P.S.: for the "whites" the card is able to capture up to 254 so you could ignore the high part of the levels and focus only on the low part (the "blacks")

An example showing that the card is able to captures 235-254; there is nothing to do at capture on the input levels for the high part, except if you have an input level at 255 that you want to shrink to 254



For this particolar video I adjust after capture the levels for AviSynth restoration using:

ColorYUV(cont_y=-20,off_y=-9,off_v=-5,off_u=5).\
Tweak(cont=1.08,sat=1.3,dither=true,coring=false).\
LevelsLumaOnly(17,1.05,254,16,235,dither=true,coring=false)

Another example, where you can see the (small) clamp at Y=16; here the input levels should be be adjusted at capture to avoid the spike at Y=16, that shows that some levels below 16 have been "crushed" and cannot be recovered (not a clear example of real bad capture, but it's all I have)

[lollo]

A better example of clipping at Y=16 (not mine)



http://www.digitalfaq.com/forum/video-restore/8820-first-steps-restoring.html#post54977

[jagabo]

The black areas of the letterbox are around 23 (ignore the black retrace bar at right, as it is around 5). The glare of the explosion is around 240. So, I have a couple/few more tweaks to go. Or, is this "good enough"?

You have a few problems here. The normal process of viewing YUV video is to convert limited range YUV (16-235) to full range RGB (0-255). So if you are exporting frames of video and analyzing them in Photoshop as RGB you expecting blacks to be 0,0,0 and white to be 255,255,255. So your black level is way off.

Letterbox bars are normally full black, RGB=0, not 22. Here's a waveform monitor of the first frame of your video sample (I used this frame because the active picture area has the full range intensities from black to white):


[Attachment 58686 - Click to enlarge]

And after adjustments with SmoothTweak(contrast=1.14, brightness=-28, Limiter=false):


[Attachment 58687 - Click to enlarge]

The brightest parts of the active picture are now just below Y=235 (RGB=255) and the darkest parts just above Y=16 (RGB=0). I let the letterbox bars and front porch go below 16 as they aren't parts of the active picture (and it's more instructive to see them in the waveform here) and having no noise in them will make them compress much better. In practice, you might want to clamp those excursions (Limiter=true). You should verify that the black level is correct across more parts of the movie. Maybe just this one scene has it slightly elevated from the letterbox bars.

[anachronon]

Yes, but... Notice in the "before" image, the details in the dark at lower left. There is atmospheric glow around the distant explosion, and some reflection off of the water. Those same details are completely crushed in the "after" image.

[jagabo]

I don't know exactly what you're seeing on your monitor. But I still see almost all of that glow around the distant explosion in the after image. Of course, it's darker than in the before image. If you really think it should be lighter you can use gamma to bring out more shadow detail. That will leave blacks at black but increase the brightness of other darker shades. Just as an example, following SmoothTweak() with SmoothLevels(gamma=1.2, TVRange=true):


[Attachment 58688 - Click to enlarge]

[Sharc]

There is perhaps a very minor loss on the ground beyond the distant explosion because the RED component is clipped at zero when converting to RGB (illegal RGB). Illegal RGB happen to other parts of the clip as well, e.g. towards the end of the clip. Maybe the tweak is a bit too agressive?

Edit:
I noticed only now that you added SmoothLevels which solved the issue basically

[jagabo]

Actually, there are lots of slightly out of gamut colors, even with the SmoothLevels(). If you look at the RGB cube inside the rec.601 YUV cube:



you can see that the RGB cube is standing on it's black corner. Only YUV combinations within the inner cube lead to valid RGB colors. If you pull Y levels down some pixels near the edge of the RGB cube end up outside the RGB cube. The errors are pretty small here and probably not worth worrying about -- especially given how important it is to have deep blacks. Or, as you suggested, you can use less aggressive settings. Or legalize them by converting to RGB and back to YUV.

[jagabo]

There is atmospheric glow around the distant explosion, and some reflection off of the water. Those same details are completely crushed in the "after" image.

Can you see every outer/inner block in this image?


[Attachment 58692 - Click to enlarge]

The inner blocks have RGB values 2 units higher than the outer blocks. View the image full screen with a black/dark background. The last white block is solid white (RGB=255) so you definitely won't see the inner block there. The second to last white block is 254 and 255. It may be hard to see the inner block there.

Overlay(BlankClip(width=48, height=48), BlankClip(width=24, height=24, color=$020202), x=12, y=12)

StackHorizontal(last, last.RGBAdjust(rb=1, gb=1, bb=1))
StackHorizontal(last, last.RGBAdjust(rb=2, gb=2, bb=2))
StackHorizontal(last, last.RGBAdjust(rb=4, gb=4, bb=4))
StackHorizontal(last, last.RGBAdjust(rb=8, gb=8, bb=8))

StackVertical(last, last.RGBAdjust(rb=16, gb=16, bb=16))
StackVertical(last, last.RGBAdjust(rb=32, gb=32, bb=32))
StackVertical(last, last.RGBAdjust(rb=64, gb=64, bb=64))
StackVertical(last, last.RGBAdjust(rb=128, gb=128, bb=128))
return(last)

Overlay(BlankClip(width=1920, height=1080), last, x=210, y=284)

[Brad]

Possibly worth noting, if anyone isn't aware, how much your ambient light levels will affect this as well. There's a lot in that image I can't see with afternoon sunlight in this room, but when viewed again at night I should be able to discern more levels, on the same monitor.

[jagabo]

Yes, ambient lighting definitely makes a difference.

[anachronon]

Well, perhaps this image comparison of the same frame from 2 tape captures will help to explain what I am messing with the proc amp:


[Attachment 58702 - Click to enlarge]

And, the associated histograms:


[Attachment 58703 - Click to enlarge]

The top image is from a capture made with a cheap Dazzle device, using default settings. The bottom image was made with a Hauppauge USB-Live2, with default settings. Note how Hauppauge is crushing the details in the shadow areas. The Dazzle appears to have the correct gamma and contrast, though it is a bit too bright (approx range 23-242). Truly, the Dazzle captures are much more watchable, than those of the Hauppauge (under default settings). The drawbacks of the Dazzle include: softening of the image, to hide noise and interlacing, and (more importantly) intermittent loss of video capture during scene changes and tape blank spots. The difference between the two results, is very similar to the two images posted by Jagabo, in #16 above.

Here is another example:


[Attachment 58704 - Click to enlarge]

[jagabo]

Your Hauppauge caps have obvious black crush. But your Dazzle caps have the black level way too high.

You didn't address the issue in post #21. Can you see the inner boxes in the top row of the image?

[Sharc]

Well, perhaps this image comparison of the same frame from 2 tape captures will help to explain what I am messing with the proc amp:

Dazzle: The black level is off the kilter. It's way too high

[Attachment 58714 - Click to enlarge]

Suggested proc amp tweaks: Try lowering the brightness and increasing the contrast.
It could also be corrected in post-processing.


Hauppauge: Blacks and whites are crushed

[Attachment 58715 - Click to enlarge]

Suggested proc amp tweaks: Try reducing the contrast and re-adjust the brightness.
More difficult to make the corrections in post processing I think, but I didn't try.

Also, the dazzle seems to deinterlace, but AFAIK this has been discussed before in the other thread.

[jagabo]

Revisiting Dunkirk, here's a slightly less aggressive adjustment:

AviSource("testXP4a.avi") 
TFM()
ConvertToYV16()
Crop(4,108,-12,-108)

#SmoothTweak(contrast=1.14, brightness=-28, Limiter=false, dither=-1)
SmoothTweak(contrast=1.12, brightness=-24, Limiter=false, dither=-1)

TurnRight().Histogram().TurnLeft()
Overlay(last, BlankClip(last, color=$0000ff), mask=mt_binarize(15).Invert()) # highlight Y<16 with  blue
Overlay(last, BlankClip(last, color=$ff00ff), mask=mt_binarize(235)) # highlight Y>235 with magenta

[Attachment 58719 - Click to enlarge]

Again, I'm not saying this is the perfect adjustment for the video. This particular shot may be different than the rest. You need to look a wide variety of shots from the film and use some compromise adjustment that works for most of it.

You might find the lines that highlight superblacks and superwhites useful. There are only a few bad pixels in the above imate. With more excursions the highlights are more obvious:


[Attachment 58720 - Click to enlarge]

[anachronon]

Yes, I have been making the proc-amp tweaks suggested. Which, brings this back to the original question of this thread---the reliability of the VDub histogram. Using the VDub histogram as a guide, setting the bright/contrast so that the two ends barely clip the red, the Hauppauge yields a result very similar to that of the Dazzle (note my capture of the battleship explosion above). It is obvious that the histogram is being fooled by noise and sharpening artifacts. The AGC of the Dazzle is likely being fooled in the same way. Hence, the need to take a more artistic approach to setting levels.

From what I gather, the reason for setting luma levels between 16 and 235, is to give the chroma room to work. Theoretically, this will yield RGB values between 0 and 255, without clipping. Applying that, using the single-frame analysis that I described above in #13, I should get results that look more like this:


[Attachment 58721 - Click to enlarge]

To answer Jagabo's question in #21, I can't really see the inner boxes of the top row, unless I am in a totally darkened room. This is true on both my laptop, and the calibrated monitor of my XP machine.

As for the Dazzle, I don't think that it deinterlaces. From tests that I have made, it appears that the Dazzle softens the image, to hide the interlacing artifacts. A close look at one of the Dazzle frames shows ghosting, where the interlacing would show.

[jagabo]

It is obvious that the histogram is being fooled by noise and sharpening artifacts.

The histogram isn't being fooled. It's showing you what's there. You just need to be aware of the limitations of a histogram.

From what I gather, the reason for setting luma levels between 16 and 235, is to give the chroma room to work.

No. With limited range YUV full black is defined as Y=16, U=V=128. And full white is defined as Y=235, U=V=128. Any other chroma values are illegal for those luma values. So Y=16, U=128, V=129 is technically an illegal triad.

Theoretically, this will yield RGB values between 0 and 255, without clipping.

The main reason limited range is used for all commercial digital video (DVD, BD, all the streaming services) is because it leaves headroom and footroom for oversharpening halos, noise, etc. Some old devices would develop severe ringing artifacts at sharp cutoffs (clipped peaks) -- analog display circuits deal better small excursions into superblack/superwhites than with clipping.

Applying that, using the single-frame analysis that I described above in #13, I should get results that look more like this:


[Attachment 58721 - Click to enlarge]

You still have elevated black levels, leaving a hazy washed out picture. Even the black letterbox bars (darker than anything in the active picture) are at RGB=14, not RGB=0. You cropped away the front porch on those images (the black vertical bar at the right of the frame in post #13). That was probably what you saw at the dark end of the histogram while capturing.

To answer Jagabo's question in #21, I can't really see the inner boxes of the top row, unless I am in a totally darkened room.

My office is at a medium brightness but I can see almost all the dark boxes in the first row (at full size with a dark background, not the little thumbnail you might see in the post). Here's what I mean by that, Firefox displaying the image at it's original scale, surrounded by a dark background:


[Attachment 58722 - Click to enlarge]

I have to look very closely to see the darkest boxes (0+2, and 1+3) but all the others are pretty clearly visible. Some of the brightest ones at the bottom of the image are a little difficult to differentiate.

As for the Dazzle, I don't think that it deinterlaces.

I see comb artifacts in the sample images you posted. So it's not deinterlacing -- or at least not well!

From tests that I have made, it appears that the Dazzle softens the image

From what I can see (and I only have the images you posted, no video), I think it may be the other way around. Dazzle's cap is closer to what the VCR is putting out. The Hauppuage is further sharpening it, exacerbating the oversharpening halos of the VCR.

[anachronon]

There seems to be some confusion here. From what I have read, the darkest luma level (black) should be set around RGB=16. Correct? But, the letterbox area of the battleship image needs to be 0? Yet, there is only a 4-5 RGB point difference between the letterbox black and the black of the active image. This would indicate that the letterbox black should be set around RGB=11. Which is it?