AviSynth 'Histogram' is Suspicious

[WinUser]

It's frustrating when a simple testing project turns out to require its own testing.

Background.
To determine what kind of changes may need to be corrected when using ConverToYV12(), I began by creating a simple RGB test card:



I very purposefully set the red, green and blue regions to level 128 (and compliments to 16) to straddle the center lines when the data are plotted. Additionally, I added a 235 white and 16 black along the top and bottom to possibly catch other irregularities as my project data moves from editor stage to encoder stage.

I'm editing in Sony Vegas, so the test card starts its life on the timeline in 8-bit mode with zero filters. The video scope shows everything is perfect: The white strip is located exactly at 235, the black strip is located exactly at 16, and there are R-G-B values at exactly 128 and nowhere else.




Problems and Confusion.
The test image is then fed to the Debug Frameserver using RGB24. This data is then processed into AviSynth thusly:

AviSource("Test.avi")
ConvertToYV12(matrix="Pc.709")
Histogram()

This is the point where I had hoped to start testing for possible video adjustments. Instead, AviSynth's histogram results were so ridiculous that I had to question whether AviSynth could even be trusted as a tool in the process:



The results clearly show the white strip data at 235 and the black strip data at 16. However, not only have the red, green and blue regions now magically separated from each other, but AviSynth wants me to believe that they are now much darker. Seriously? WTF?

Can anyone explain to me what is going on here?

[WinUser]

Ahhhh! I think I've intercepted my own problem: I need to be comparing against an RGB Parade scope; but apparently there isn't one for YV12 data. Start over...

[raffriff42]

Histogram("levels") seems to work OK when used like this:

ImageSource("rgbTestCard.png")
Crop(0, 2, 0, 224) ## remove 255 white, 0 black, text
ShowRed("YV12") ## or ShowGreen, ShowBlue 
Histogram("levels")
return Last

But classic Histogram is a bit off:

The VirtualDub Colortools plugin has a good (accurate) RGB scope, among other cool things.

And Avisynth has this set of scripts, which need care to get accurate results:
http://avisynth.nl/index.php/Histograms_in_RGB_%26_CMY

Finally, the image with dull reds and bright greens (or vice versa) is a classic example of the wrong color matrix (601/709) being used, at some point.

[jagabo]

The results clearly show the white strip data at 235 and the black strip data at 16. However, not only have the red, green and blue regions now magically separated from each other, but AviSynth wants me to believe that they are now much darker. Seriously? WTF?

That's exactly what is expected. Histogram() shows the luminance values (the Y in YUV). Luma is a weighted average of the RGB components. With the pc.709 the equation is:

Y = 0.2126*R + 0.7152*G + 0.0722*B

So a pure blue bar at 128 gives:

Y = 0.2126*0 + 0.7152*0 + 0.0722*128
Y = 0        + 0        + 0.0722*128
Y = 9.2416

And after converting to an inter:

Y = 9

Since the weighting is different for each RGB component looking at Y of pure color bars gives different values for each bar. Only with pure grey bars will Y=R=G=B.

HistorgramRGBParade() gives something like Sony's RGB parade (RGB waveforms).
http://avisynth.nl/index.php/Histograms_in_RGB_%26_CMY

Or you can do as riffraff42 suggested, use ShowRed(), ShowGreen(), and ShowBlue(), followed by Histogram().

[WinUser]

The VirtualDub Colortools plugin has a good (accurate) RGB scope, among other cool things.

Very nice! Thank you for that

[WinUser]

On the issue of unwanted color shifts, that V'Dub color graph has just confused me!

I revised the test card to include blacker-than-black [0,0,0] and whiter-than-white [255,255,255] along the strips. What I'm doing is simulating a project where I have full-range CGI material that needs to eventually be encoded as high-definition blu-ray media:




Within Sony Vegas, that "CGI" footage is subjected to the Computer RGB to Studio RGB levels filter.

The data is then frameserved to AVISynth and converted to YV12.

If the Rec709 matrix is specified, V'Dub reports a green shift:




If the Rec601 matrix is specified, V'Dub reports what I interpret to be the correct, intended result:



I'm terribly confused at this point, because I thought that HD-720p material was supposed to be associated with the BT.709 specification. Is V'Dub telling me that I need to produce this HD footage and flag the H264 stream as BT.601?

[shekh]

Not clear what do you open in VD, avs script? In any case Rec709 matrix is lost in this communication, you must use "alias format" filter with old VD, or change options in "decode format" dialog with FilterMod.
Cannot comment on ColorTools output, since the filter is RGB all what it says about colorspace makes no sense to me.

[WinUser]

Not clear what do you open in VD, avs script? In any case Rec709 matrix is lost in this communication, you must use "alias format" filter with old VD, or change options in "decode format" dialog with FilterMod.
Cannot comment on ColorTools output, since the filter is RGB all what it says about colorspace makes no sense to me.

VD imports a simple AVISynth script:

DirectShowSource("Test.avi")
ConvertToYV12(matrix="Rec709" or "Rec601")

I was intending to trust VD and the ColorTools filter to measure whether or not the CGI (0-255) source material has been correctly transformed for studio media (16-235) right before the point of AVC encoding.

I am now confused, because the VD results are telling me that every H264 encode I have done in the past was incorrectly green-shifted.

I'm not quite sure the purpose of the alias-filter, because the point is not to make the graphs look the way I want; the point is to correctly measure the output so that I know my production chain is correct. Are you saying VD's behavior in this situation is known to flawed? I am supposed to be using Rec709?

[jagabo]

Yes, VirtualDub normally converts YUV to RGB (for display or filtering) with a rec.601 matrix.

Don't use VirtualDub to examine levels. Use Histogram() in AviSynth.

ImageSource("RGB Test Pattern.png") 
ConvertToYV12(matrix="pc.709")
Histogram()
ConvertToRGB(matrix="rec709") # so the video displayed correctly in VirtualDub

[Attachment 40688 - Click to enlarge]

[WinUser]

Yes, VirtualDub normally converts YUV to RGB with a rec.601 matrix.

Ah, so you mean that VD is introducing an incorrect RGB interpretation for its own display and processing, but since my H264 encoder only wants YV12, then Rec601 would mess up the final product.

It is, then, proper to use ConvertToYV12(matrix="Rec709") when delivering studio-range HD content to the encoder.

Alas, I think I understand the nature of this mess. Thank you, everyone, for all the clarifications.

[jagabo]

It is, then, proper to use ConvertToYV12(matrix="Rec709") when delivering studio-range HD content to the encoder.

rec709 is for full range RGB (where 0,0,0 is black, 255,255,255 is full white) to limited range YUV (where Y=16 is black, Y=235 is full white) and vice versa. Full range RGB is the norm for display. Limited range YUV is the norm for all consumer distribution formats (DVD, Blu-ray, broadcast TV, etc.).

[WinUser]

rec709 is for full range RGB (where 0,0,0 is black, 255,255,255 is full white) to limited range YUV (where Y=16 is black, Y=235 is full white) and vice versa.

I'm reading your words literally, so I now have a new concern: If the full-range CGI footage has already been filtered in Vegas (Computer RGB to Studio RGB) to have no color values below 16 or above 235, then what is the effect of using Rec709 in the AVISynth script downstream? Surely, the production chain is now messed up.

It's like I damned if I do and I'm damned if I don't.

Is there a matrix setting for "DON'T CHANGE MY FRIGGIN' COLOR DATA!!!" ?

Has it ever been clarified what the matrix instruction actually means? Does it refer to the source color range, the target color range, or as you hinted: the CHANGE of color ranges?

It's apparent to me now that I do not understand the following:

Rec709: IN = 0-255, OUT = ?
Rec709: IN = 16-235, OUT = ?
PC709: IN = 0-255, OUT = ?
PC709: IN = 16-235, OUT =?

If the CGI data already exist as 16-235 (say, through prior levels adjustment in Vegas), then what is the correct ConverToYV12() process before AVC encoding?

This stuff is really infuriating

[jagabo]

rec709 is for full range RGB (where 0,0,0 is black, 255,255,255 is full white) to limited range YUV (where Y=16 is black, Y=235 is full white) and vice versa.

I'm reading your words literally, so I now have a new concern: If the full-range CGI footage has already been filtered in Vegas (Computer RGB to Studio RGB) to have no color values below 16 or above 235, then what is the effect of using Rec709 in the AVISynth script downstream?

If those same levels are passed through to AviSynth rec709 will give you bad levels. The video will look dull and washed out at play back because it won't have full blacks or full whites. Use ConvertToYV12(matrix="pc.709") instead. Then RGB=16 will be converted to Y=16, RGB=235 will become Y=235.

[poisondeathray]

CGI is usually higher bit depth RGB. If you're doing any post production, grading (even tiny adjustments) , you typically don't want to squish the range from the very beginning with a computer to studio RGB filter, and definitely not in 8bit (you introduce artifacts, banding especially with CG material)

So the typical answer for vegas is stay with computer RGB until the final end delivery YUV formats. You export out RGB out of vegas . You use something else like avisynth or other tools to control the RGB => YUV conversion and final encoding. Typically other filtering is applied such as dithering - especially for CG material.

But the other common concern is if you're compositing and mixing live action footage which will be treated as studio RGB in vegas (mixed asset timeline) - then it depends on the project specifics