Finding reason for differing colors

[CryGuy]

Hello everyone,

whenever you're transcoding movies and colors of the result appear not exactly like the original, there are several place where to search for the reason. Instead of making a long story, I kindly ask whomever would like to, to tell me which of the colors are the "real" colors: the ones shown in MPC-HC or the ones shown in VirtualDub? This is the screenshot: (EDIT: MPC is top, VirtualDub is bottom)



And this is the video: (6,5 MB, AVI, Motion JPEG)
www.file-upload.net/download-4011991/tintenpatronen-cut.zip.html

I'll tell more about it if of interest. I've learned that long OPs are discouraging.

TIA!

[Cornucopia]

The colors in the upper shot look more "natural" to me. Are they the "real" ones? I have little way of knowing.
White balance on the upper tends toward the yellow, whereas WB on the lower tends to either neutral or blue. How was it shot?
Although, I will say that the lower shot seems to be "blooming" (aka high luma values are blown/clamped).

Why/How is a very LONG, involved answer I won't go into right now.

Scott

[CryGuy]

First, thanks for your answer.

The colors in the upper shot look more "natural" to me

I agree - but see below.

I have little way of knowing.

That's why I uploaded the video so that you can have a look yourself.

Ok, I must add some info: The AVI file has been recorded from a Webcam. I use to transcode the huge M-JPG encoded video to H.264 using AviDemux. No problem actually. However, this time I noticed the differing colors. I started searching for the reason. The original played in MPC-HC looked "better", or more natural, than the H.264 encoded result. The latter looks as "blooming" as you see in the screenshot in the bottom. Some edges of the shown box are not even visible. I wondered why and started analyzing the original material. But even that is shown differently in VirtualDub and MPC-HC. So, I have two differing views of the original video. If I don't even know which one is the "right" original, I can't continue. That's why I asked here.

So, next step was saving it as an image sequence (Bitmap) from VirtualDub. I know the "Color depth" settings there. I'm pretty sure, VirtualDub does a good job in saving the frames unmodified (if no filter is applied). And what did I get? Blooming pictures. I don't believe this! Because: That would mean that MPC-HC "recreates" the good looking edges from that "bad" original. Almost impossible. So, VirtualDub is fooling me? Something's going wrong here...

[CryGuy]

Gottcha! Tried many things before but not this one: Opened the original in AviDemux and exported as Jpg-Sequence. Result: Nice looking frames! So, again, who is wrong? This is just M-JPEG, so why are the decompression results so different between avidemux and vdub?

[poisondeathray]

None of the above are necessarily "correct." You're looking at an RGB represetntation of the Y"CbCr data ( your original mjpeg is in Y'CbCr colorspace) . How that software converts to RGB for display, or any filters, will affect what you see on the monitor or any display

For example, in MPC, change the renderer to vrm9 (renderless) close it, reopen the video . Or try overlay mixer. It will look different, but the underlying video is the same

Graphics card settings for overlay mixer can affect how it is rendered. For example, if you have directx enabled in the vdub options, the output pane will rely on those settings you specify. If you turn directx off, it will look different.

Some decoders will clamp or clip to Y' 16-235 (or legal levels) while others will let it pass through. So you lose your superbrights in the former, but not the latter (which then can be recovered) . How the RGB conversion is done might also clip the data

[jagabo]

Start VirtualDub and select: Options -> Preferences -> AVI (left pane) -> and enable "Prefer internal video decoders..." and Options -> Preferences -> Display (left pane) and disable "Use Direct X for display...". Exit and restart VirtualDub for the change to take effect. That will bypass the graphics card's video proc amp settings which can cause brightness and colors to be rendered incorrectly.

Now go to Video -> Color Depth... in the Decompression format specify 4:2:2 YCbCr (YUY2). Save that as the default.

That pariticular video has luma (brightness) values above the standard max of 235. That is causing the blown out whites. The black level may also be a little to high (it's hard to tell because there aren't many dark areas). In recent versions of VirtualDub apply the Brightness/Contrast filter before any other filter to bring it down. Set Brightness to about -8 percent. That will bring out a little more detail in the bright areas:



Of course, you can go darker if you want. But that 8 percent will get you into the legal luma range. It's important to do this first in VirtualDub because the Brightness/Contrast filter can work while the video is still in YUV. Once it's converted to RGB (most other filters) the highlights will be lost and irrecoverable.

[CryGuy]

poisondeathray,

I know all these influences. That's why I narrowed it down to the origin. I must start somewhere to find the reason for the bad result of the final transcoded video. I must get a view of the original, unmodified frame. If that's even "wrong", I can not complain about the final transcoding result. The problem is that I still don't know what's stored in the orignal. The closest I can get is exporting the frames as bitmap files. Using Windows' Paint displaying RGB-bmps is the rawest look I can get, but if even these results differ between vdub and avidemux, what can I do? Currently I'm unable to say if the transcoding result is not as expected because of the original material or because of any other processing step afterwards - because I can not even evaluate the orignal.

[poisondeathray]

MPCHC clamping the levels to 16-235 Y' in the output pin (ffdshow and internal mjpeg decoder does this for mpchc) so when the Rec709 conversion is applied , Y'CbCr [16-235] => RGB 0,0,0 - 255,255,255 . Thus you see no clipping, because the decoder squeezes the info to 16-235 Y' before the RGB conversion for display . I mentioned earlier some decoders squeeze to 16-235, others output 0-255 Y' from the output pin.

Vdub is preserving levels, but you didnt correct Y' overshoots and probably forgot to use fast recompress. This means you incur rec601 RGB conversion (you lose Y' <16, Y' > 235 before you export) . This is a clipping (so data is lost). But even if you used fast recompress, h.264 doesn't get clamped by MPCHC unlike MJPEG (levels are 0-255 in the output pin before the RGB conversion for h.264), but the rec709 conversion to RGB for display causes the clipping h.264. The difference is MJPEG is clamped to 16-235 Y' before the RGB conversion with MPCHC, but h.264 is passed through with levels untouched

In jagabo's example, he brought down Y' to legal levels before exporting, so when you incur the RGB conversion in MPCHC, you see the details in the box

We're just talking about luminance so far, but there are other differences , due to renderer choice in your screenshots

[CryGuy]

jagabo,

I've already disabled DirectX cause GDI doesn't do any of these modifications.
The proposed filter "tweaks" the result, but actually I'm looking for a way to get the "untweaked" picture.

[jagabo]

I've already disabled DirectX cause GDI doesn't do any of these modifications.
The proposed filter "tweaks" the result, but actually I'm looking for a way to get the "untweaked" picture.

There's no such thing as an "untweaked picture". You are dealing with YUV video. It has to be converted to RGB for display. Those YUV values can be converted to RGB in a number of ways. You need to use the right way. And of course, if your monitor isn't calibrated nothing you see will be correct.

If you don't apply the Brightness/Contrast tweak you will be seeing what properly operating players will display (aside from any calibration issues with your monitor). The yellow cast in your MPCHC image is because your graphics card's video proc amp settings are incorrect. Or you're using MPCHC's internal proc amp to change the colors.

[CryGuy]

MPCHC clamping the levels to 16-235 Y' in the output pin (ffdshow and internal mjpeg decoder does this for mpchc) so when the Rec709 conversion is applied , Y'CbCr [16-235] => RGB 0,0,0 - 255,255,255 . Thus you see no clipping, because the decoder squeezes the info to 16-235 Y' before the RGB conversion for display

Vdub is preserving levels, but you didnt correct Y' overshoots and probably forgot to use fast recompress. This means you incur rec601 RGB conversion (you lose Y' <16, Y' > 235 before you export) . This is a clipping (so data is lost)

It's clear that I can't use mpc to see what's stored "in the file". After narrowing it down, the question is why Avidemux exports different frames than vdub. I don't see any influence of the chosen color depth or compression method (fast, normal) in vdub.

To make it simple, by looking at my screen, Avidemux seems to do it right because the exported bmps look more natural. Ok, so let's assume Avidemux sends the expected picture to the encoder. However, the result played in mpc doesn't look like the original. Again, I know this has many influences. So let's eliminate them by loading the result into avidemux again and save a frame as a bitmap. But this frame is also too bright. So, does this mean a) the frames are brightened while encoding or b) the frames are encoded correctly but avidemux brightens them when exporting to bmp? How can I know?

[poisondeathray]

You can't use anything to determine what's "stored" in the file. Because you are viewing RGB, while the file is Y'CbCr.

How you do that conversion determines what you see, even if the underlying video is the same

There can be differences at multiple points, from decoding, any filters, to RGB conversion and renderer and any filters. For example the decoder choice might clip, clamp, or leave levels untouched. All 3 are different. For example FFDShow Mjpeg decoder will clamp, but FFMpeg will pass through full levels.

You need a waveform monitor that operates in Y'CbCr to see what levels the data is at, but even before that, if you choose 1 decoder or another, the levels might be different.

Saving a frame as a bitmap, means you convert Y'CbCr=>RGB (again, how you do that conversion determines how it looks) . The bitmap is an RGB representation of that Y'CbCr data . It's not the same thing. RGB and Y'CbCr are color models that do not overlap completely

[CryGuy]

There's no such thing as an "untweaked picture"

That's why I put it into quotation marks. At least it doesn't get better if you put another filter on it, that was my point.

You are dealing with YUV video. It has to be converted to RGB for display.

I know I know.

And of course, if your monitor isn't calibrated nothing you see will be correct.

Doesn't matter as long as I use the same screen for looking at all results.

The yellow cast in your MPCHC image is because your graphics card's video proc amp settings are incorrect. Or you're using MPCHC's internal proc amp to change the colors.

Video proc amp? It's a screenshot (copy from screen memory), not a photograph of the screen. The screenshot of MPC should only show that it can't be correct what is displayed by vdub. If information was lost, it couldn't be recreated by MPC in any case.


Well, the whole point of my question was if the video has already been recorded as bright. I always want the transcoded video to be as close to the original as possible. Assuming that, I can live well with the "bright" result if I know that the original is also as bright. The latter one is what I have/had to find out.

[jagabo]

The yellow cast in your MPCHC image is because your graphics card's video proc amp settings are incorrect. Or you're using MPCHC's internal proc amp to change the colors.

Video proc amp? It's a screenshot (copy from screen memory), not a photograph of the screen. The screenshot of MPC should only show that it can't be correct what is displayed by vdub. If information was lost, it couldn't be recreated by MPC in any case.

Since you already know everything I shouldn't have to explain this to you.

A screen shot is capturing RGB from the graphics card, not the original YUV video. When using video overlay (or the myriad of other video renderers available in Windows) the media player is sending YUV to the the graphics card. It's the video proc amp settings that control how that YUV video is displayed on the RGB monitor. If your proc amp settings are incorrect (and it's very common for the defaults to be wrong when installing graphics drivers) the player will display the wrong colors.

Beyond that you have other issues. If the YUV matrix isn't flagged in the video most players will assume rec.601 for standard definition video, rec.709 for high definition video. What happens between 720x576 and 1280x720 is player dependent. Some will use one matrix, some the other. (And some player even ignore the matrix flag, always using a particular matrix for the conversion.) VirtualDub always uses the rec.601 matrix to convert YUV to RGB. But that is wrong for HD video or video that's flagged as rec.709.

https://forum.videohelp.com/threads/329866-incorrect-collor-display-in-video-playback?p...=1#post2045830

I always want the transcoded video to be as close to the original as possible.

That is wrong headed. Your source video has illegal YUV values. You should correct them before encoding. Some devices will render illegal YUV values unpredictably. I've seen over-bright whites show up as black, bright red, etc. Usually on very old DVD and software players.

[poisondeathray]

Well, the whole point of my question was if the video has already been recorded as bright. I always want the transcoded video to be as close to the original as possible. Assuming that, I can live well with the "bright" result if I know that the original is also as bright. The latter one is what I have/had to find out.

No you don't. You usually want to correct levels (bring them into legal range) , so you preserve that detail on any player, any program, all hardware

The reason is most software, video players, TV's will use Rec601/709 for conversion for display. So even if you have data in Y'CbCr in the "super" range , when it gets converted to RGB for display, it doesn't appear in the display (i.e you don't see it, but that data is there!)

Rec601/709 Y'CbCr=>RGB conversion means only that data between Y'16-235 will be seen. All that other data Y'<16, Y'>235 will not be seen. But if you bring that data back into legal range, then (almost) everything will be seen

The reason why opening the native video in MPCHC shows all the detail (for example the lower box folds , ignore the color differences for now), is because the directshow MJPEG decoder clamps Y' to 16-235 BEFORE the RGB conversion. So (almost) everything is seen

[CryGuy]

You can't use anything to determine what's "stored" in the file.

Please, why do you take it so literally? I put quotation marks around it. In the context I wrote, I only made clear that I'm aware of the fact that looking at mpc doesn't show any unmodified file content 1:1 because the processing chain from file to display has several parameters. Or, in short, "It's clear that I can't use mpc to see what's stored 'in the file'".

At least I can tell by comparing vdub and mpc that vdub obviously does not show the best representation of whats stored in the file. If it would, some information (see the edges in the middle of the box) wouldn't exist in the file. MPC prooves the opposite because it can't recreate what's not there.

Saving a frame as a bitmap, means you convert Y'CbCr=>RGB (again, how you do that conversion determines how it looks) . The bitmap is an RGB representation of that Y'CbCr data . It's not the same thing. RGB and Y'CbCr are color models that do not overlap completely

Yes, I still know this.

[jagabo]

Your webcam may be using the PC.601/709 matrices rather than the rec.601/709 matrices (that would explain the over bright areas). But in all likelihood it just doesn't control brightness very well.

And, if your webcam is using a PC matrix the display in VirtualDub will be wrong. Again, because VirtualDub always uses the rec.601 matrix to display YUV video.

[jagabo]

Or, in short, "It's clear that I can't use mpc to see what's stored 'in the file'".

Sure you can. Just fix your graphics card's video proc amp settings. Your graphics card has two proc amps, one for the Desktop, another for video.

Normal proc amp settings:


Way too bright proc amp settings:


The video proc amp settings have no effect on VirtualDub's display when DirectX output is disabled. Because VirtualDub is performing the YUV to RGB conversion when you do that. If you want another player that's immune to proc amp settings use VLC with the output device set to Windows GDI.

[poisondeathray]

At least I can tell by comparing vdub and mpc that vdub obviously does not show the best representation of whats stored in the file. If it would, some information (see the edges in the middle of the box) wouldn't exist in the file. MPC prooves the opposite because it can't recreate what's not there.

The principal reason for the observed levels was explained earlier, the output pin of the decoder.

MPCHC, VDub, as "showing a represenatation" - they are meaninless unless you specify what decoder, what renderer, etc... because different decoder choice will output different levels ... but you know this



In this example, both screenshots were taken with vdub, both using rec601 conversion

The difference is different decoder. FFDShow VFW Mjpeg decoder outputs levels clamped to 16-235, but Vdub's internal Mjpeg decoder outputs levels untouched 0-255 Y' - all this BEFORE the RGB conversion (but you know this , I'm just explaining for other people who might have same question)

Use vdub file=>file information to see the decoder you are using when video is loaded in vdub

(images resized for display purposes, make sure you view at the same level in the browser, or save images to desktop and flip back & forth)

[jagabo]

MJPEG is different than most video codecs. It usually uses the PC.601 matrix because JPEG stills use that. That means that MJPEG should have the luma range compressed from 0-255 to 16-235 before encoding as MPEG, Divx, h.264, etc.

Some decoders and conversion software account for this, some not.

[CryGuy]

EDIT: I see you wrote more in the meantime, so this may already be outdated....

I stop at this point replying to every single statement. Would get exhausting for everyone, and I don't want to waste your time. So, to sum up what's important to me:

• I didn't say I know everything. Don't know why people feel stepped on their toes just because I want to save them from unnecessarily wasting their time by telling me things I already know. How else can I say it without saying it? More smileys? Focussing just keeps the discussion shorter. I don't doubt any of this information, but much of it is just not necessary for answering my question. Maybe it's my English; I don't know how to express in other words.
  
• I thought the video proc amp is the amp for generating the video signal of the graphics card. Therefore the misunderstanding. Now I know the term in this context.
  
• I've already eliminated as many influences as possible by saving images from vdub/avidemux. (yes, and I do know that even that process has influences and is obviously done differently in vdub/avidemux; no need to repeat it once agein). That's why I'm also not interested in any amp settings or whatever.
  
• If I consider the original video looks ok, there is no single reason to make the transcoded video look different. That's reasonable, not "wrong headed".
  
• "A screen shot is capturing RGB from the graphics card". Sorry, but that's what I already said before. No need to repeat it in other words as if I told the opposite.
  
• If I have a bad recording, I don't want to "tweak" anything afterwards. Instead, I make a new recording. But again, to evaluate the recording I must know what has been recorded by looking at it. If I have two results, (at least) one is wrong. I asked here because I didn't know which one. If my cam always does it "the wrong way" (please note the quotation marks) I can't help; I have to live with it. BTW, it's a "Microsoft LifeCam Cinema HD".
  
• "You usually want to correct levels". I really agree in general, but on the other side, I correct only if something is wrong. Imagine my webcam outputs RGB images and they are stored unmodified on my hard disk: Would the brighter or the darker box have been recorded? Whatever would have been recorded, that's the original. (no need to discuss the quality of my webcam and the lighting scnario etc pp). After decompression of my recording, I just want to get as close to the picture that has been recorded by my cam. Consequently, the same mapping process in the other direction must take place. Naturally. If I have to correct levels for that, I'll do so. As long as values are not already clipped (= lost information) in the file, everything's fine.
  
• Thx for the information about vdub using the rec.601 matrix to display yuv video. No, I didn't know this, like many other things.

[CryGuy]

In this example, both screenshots were taken with vdub, both using rec601 conversion

I make it short: Thx for trying this. (as you can imagine, I understand your demonstration. )

So, finally, please give me instructions now. EDIT: not necessary, I'll find it myself.

1. I load the AVI in avidemux.
2. ?

Thanks for your help!

[poisondeathray]

In this example, both screenshots were taken with vdub, both using rec601 conversion

I make it short: Thx for trying this. (as you can imagine, I understand your demonstration. )

So, finally, please give me instructions now. EDIT: not necessary, I'll find it myself.

1. I load the AVI in avidemux.
2. ?

Thanks for your help!

Avidemux uses ffmpeg decoder for mjpeg. Translation: Levels are untouched, 0-255 much like the vdub internal ffmpeg decoder in the example above

So if you encode using something like x264, xvid without any filters, levels will be identical to original, Y' 0-255

In other words, when you play it back in any media player, you will not see the detail from 0-15, 236-255 (e.g. the box folds)

But if you "legalize" the levels to Y' 16-235 with a filter, then you will see the details come into view similar to the ffdshow picture when playing the Y'CbCr export in a media player

[CryGuy]

5 AM now. Need some sleep.
Thanks again.
"I'll be back"

[Cornucopia]

I still don't think you're getting what pdr and jagbo have been trying to explain (so I don't think my input here would help much), but let me add just 2 other things:

1. the "Real" color is in the eye of the beholder, but just as "all men are created equal, but some more equal than others", being able to identify what a good or correct image is is something one needs to be trained to do. This includes a lot of practice and referencing of "known good" images.

2. speaking of "known good", professional videographers/cinematographers/photographers have understood the problem you're having (how to tweak -or not- a multi-variable display chain to arrive at a linear, zero-sum-gain final presentation) for quite some time now. They've gotten around this by the simple but clever expedient of shooting, along with their target images, a "known good" reference image (and then processing them identically). IOW, the means to decode correctly is included in the data itself. Once the reference image is correctly arrived at, the target image will ipso facto also be correct.

Scott

[CryGuy]

Corn,

the kind version of what I'm feeling is: I'm not satisfied with the situation. IOW, I don't appreciate it if one thinks a discussion is a fight based on allegations, emphasized by all knowledge available - related or not - and he must be the winner. That's not my attitude. I'm thinking cooperatively and solution oriented instead. That's my 2c about most of this discussion.

The most helpful answer I got was the last one from poison. Now where at it!

Avidemux uses ffmpeg decoder for mjpeg. Translation: Levels are untouched, 0-255 much like the vdub internal ffmpeg decoder in the example above

So if you encode using something like x264, xvid without any filters, levels will be identical to original, Y' 0-255

I do believe you, but I'm surprised that a H.264 encoder doesn't always output TV levels. I thought it violates the standard it's made for. Instead, we must adjust the RGB to TV level before encoding to ensure TV level in the target video? Well, in that case, I've really learned something new.

However, there's one single thing left that makes me unsure: In avidemux, if I save the image as a Bitmap (without a filter added), I expect the picture right after decompression to be saved. But if you look at the saved bitmap (in Paint or whatever), there's no fold visible. However, you know that there is a fold! That's the point I'm desperately trying to make clear: Obviously information got lost during decompression. This is not explainable by any other influence because the only processing step was YCbCr -> Decompression -> RGB. I'm really not dumb! I'm an (experienced) programmer and have done more sophistiacted stuff in the past 25 years, so I can only ask you to believe me. This result tells me that avidemux doesn't use the correct matrix for decompression. IMO, if information got lost, something went wrong. Don't you think so, too?
The other possible explanation is: I know from the Avidemux forum that the preview inside avidemux is not correct. That's an ol' problem. Not a problem for me, but maybe the saved bitmap suffers from the same problem as the preview, whereas the correct image is sent to the encoder. It must work this way. Everything else doesn't make sense.

[poisondeathray]

Corn,

the kind version of what I'm feeling is: I'm not satisfied with the situation. IOW, I don't appreciate it if one thinks a discussion is a fight based on allegations, emphasized by all knowledge available - related or not - and he must be the winner. That's not my attitude. I'm thinking cooperatively and solution oriented instead. That's my 2c about most of this discussion.

The most helpful answer I got was the last one from poison. Now where at it!

Avidemux uses ffmpeg decoder for mjpeg. Translation: Levels are untouched, 0-255 much like the vdub internal ffmpeg decoder in the example above

So if you encode using something like x264, xvid without any filters, levels will be identical to original, Y' 0-255

I do believe you, but I'm surprised that a H.264 encoder doesn't always output TV levels. I thought it violates the standard it's made for. Instead, we must adjust the RGB to TV level before encoding to ensure TV level in the target video? Well, in that case, I've really learned something new.

However, there's one single thing left that makes me unsure: In avidemux, if I save the image as a Bitmap (without a filter added), I expect the picture right after decompression to be saved. But if you look at the saved bitmap (in Paint or whatever), there's no fold visible. However, you know that there is a fold! That's the point I'm desperately trying to make clear: Obviously information got lost during decompression. This is not explainable by any other influence because the only processing step was YCbCr -> Decompression -> RGB. I'm really not dumb! I'm an (experienced) programmer and have done more sophistiacted stuff in the past 25 years, so I can only ask you to believe me. This result tells me that avidemux doesn't use the correct matrix for decompression. IMO, if information got lost, something went wrong. Don't you think so, too?
The other possible explanation is: I know from the Avidemux forum that the preview inside avidemux is not correct. That's an ol' problem. Not a problem for me, but maybe the saved bitmap suffers from the same problem as the preview, whereas the correct image is sent to the encoder. It must work this way. Everything else doesn't make sense.

The reason for your avidemux bmp observation was explained earlier

Y'CbCr gets converted to RGB with a Rec matrix , so Y' 0-15 and 236-255 get clipped (i.e. 0-15 becomes all black instead of seeing gradations, and 236-255 becomes all white, instead of "shades" of white) . That's why you don't see a fold

"Correct" matrix is all relative. If you wanted 0-255 Y' to 0,0,0,-255,255,255 RGB , that would be a PC matrix, not a Rec matrix . Rec matrix converts 16-235 Y' to 0,0,0-255,255,255, so you don't see those values 0-15, 236-255. I feel like I'm repeating myself

Maybe we have done a poor job explaining this. If it still doesn't make sense I can make some other examples and waveforms

[poisondeathray]

I do believe you, but I'm surprised that a H.264 encoder doesn't always output TV levels. I thought it violates the standard it's made for. Instead, we must adjust the RGB to TV level before encoding to ensure TV level in the target video? Well, in that case, I've really learned something new.

When you are feeding Y'CbCr material to most encoders, there is no intermediate colorspace conversion (you're not converting to RGB at the encoding stage) . So input = output levels. The difference is most DEcoders will use Rec matrix to convert to RGB for display - so if you don't legalize Y' values, you won't see 0-15 or 235-255. They will be a big black bar, and a big white bar = no details . In Y' => Y' there is no information lost, except when you apply lossy compression. In fact, when you encode using h.264 or xvid, the information is still there, you are just not seeing it when it gets converted to RGB for display. I have a strong feeling you're not understanding what was being discussed earlier. You wouldn't be asking these questions.

In your original file , you have values that are superwhite. Normally white is Y' 235. But you have values greater than that. If you don't legalize values, they will clipped upon display by the RGB conversion

BMP's are RGB. So in the case of avidemux you are converting to RGB with a Rec matrix. You don't have control over this conversion in avidemux, but you do in other programs like avidemux. If you wanted to map full range using a PC matrix for example.

[jagabo]

I'm surprised that a H.264 encoder doesn't always output TV levels. I thought it violates the standard it's made for.

The job of a codec is to reproduce what you give it, not fix your video for you. The super blacks and super bright regions are available for occasional overshoot, to prevent ringing from hard clipping, not for routine use.

Here's a waveform (brightness) graph of your video:



All that yellow stuff in the brown bars is not supposed to be there. The brightness should be lowered it below the brown bar:



Notice how you can now see details on the bottom of the box? You have a color problem too but that's a different issue.

[CryGuy]

Y'CbCr gets converted to RGB with a Rec matrix , so Y' 0-15 and 236-255 get clipped (i.e. 0-15 becomes all black instead of seeing gradations, and 236-255 becomes all white, instead of "shades" of white) . That's why you don't see a fold

You don't get it. The final video does contain the folds. That's not possible if clipping has taken place before. Lost information is not recoverable!

"Correct" matrix is all relative.

No. If information got lost, it's just wrong. But I've said this already one, two times.