Can't get the right colors when doing RGB -> YV12 on Rec709 source

[rebus_x]

Hello there!
What I have is a file recorded with OBS into x264, and the recording settings are NV12 pixel format, colorspace 709 and limited range, see here my OBS settings

Here is the OBS recording (source). By the way, in the source picture colors are pure green (0-255-0) and pure yellow (255-255-0).
Screenshot

Now, when feeding this directly to x264, the colors after encoding are the same, which makes sense as the encoder would not need to convert anything.
Screenshot (x264 source -> x264)

But in my project, I need to edit the file in Vegas pro, so the final conversion to YV12 is required. I output the frames with Debugmode Frameserver in RGB24.
So the problem is, I tried both feeding RGB to x264, and doing ConvertToYV12() with Avisynth (with different matrices), but unfortunately I can't get the same colors as in the source.

And what I can't comprehend is this: obviously, when recording, OBS would have to convert from RGB to NV12 (which is 4:2:0 just like YV12), and how is it that it does better job at converting and preserving color? So maybe if I find out how OBS does that, I can do the better conversion too.

The rest of the screenshots. For this example, I transformed the source (OBS recording) into png frames and did RGB->YV12 with avisynth
RGB->YV12 (rec709)
RGB->YV12 (PC.709)
RGB->YV12 (rec601)
RGB->YV12 (PC.601)

Script

ImageSource("C:\...\desktop\frames\%03d.png", 0, 100, 30)
ConvertToYV12(matrix="rec709")		#and other matrices

Any help is greatly appreciated.

[Selur]

To be frank I have never used OBS, but reading

By the way, in the source picture colors are pure green (0-255-0) and pure yellow (255-255-0).

and looking at your OBS settings, I'm wondering: Why are you using color range 'partial' (16-235) and not 'full' (0-255) ?

[rebus_x]

Well, my reasons are simple:
1. it was the default setting (color space was not, by default it's 601 and i changed it after seeing results)
2. doesn't seem to make any visual difference
3. produces x264 stream which says in mediainfo:

...
Color range                 : Full
Color primaries             : BT.709
...

, instead of Limited, which I'm used to seeing with x264 everywhere
4. it should indeed be a simple matter of 0-255 vs 16-235, but everything on the internet about this setting is highly confusing and makes no sense to me, i.o.w. I haven't found any good explanations, so why even bother if it seems to not affect anything

upd. Okay, I double checked, and when recording in OBS with Full range, the colors are tiny bit closer to original RGB image, but they are almost identical to Source screenshot anyway. So i doubt that would greatly help in a conversion problem...

[Sharc]

I didn't check all your variants, but starting with the .mkv source I am getting for the green color in 709:
RGB(0,255,1) YUV(173,42,26) which is compliant with the matrix 709 conversion for full range (means RGB(0...255) <-> Y(16....235))
It does not change when I add converttoYV12(matrix="Rec709").

Creating a .png snapshot from the .mkv and watching it with the ImageSource filter I am getting
RGB(0,255,1) YUV(144,54,34), means a shift in YUV, because the .mkv YUV source was converted to RGB24 for viewing.
Adding converttoYV12(matrix="Rec709") I am getting again
RGB(0,255,1) YUV(173,42,26)

Edit: Now checking your screenshots they seem to have been color shifted using the wrong matrix (601). So your problem is with the screenshots, the video is ok.

Edit2: oops, poisondeathray was faster

[Selur]

okay,...
May be someone here can explain it in a way you understand, or may be not.

If your source is 0-255 and you captured with limited luma range the result is either compressed or cut to 16-235.
If that content is flagged as limited range (and the playback device that does the YUV->RGB conversion follows the flag) the result will be 16-235 with either compressed or cut colors.
If that content is flagged as full range (and the playback device that does the YUV->RGB conversion follows the flag) the result will be 0-255 and if the colors will be off too (unless the compression and uncompression method luckily match and no luma info got lost).
-> My guess is that either you are Vegas at some point assumes that something is limited/full luma range that actually isn't (reason could be that Vegas only supports tv range or that the content isn't flagged properly).

You probably have to wait for someone who actually used OBS+Vegas+Avisynth the way to do that knows that he is doing and looked into.
Finger crossed.

Cu Selur

[rebus_x]

I didn't check all your variants, but starting with the .mkv source I am getting for the green color in 709:
RGB(0,255,1) YUV(173,42,26) which is compliant with the matrix 709 conversion for full range (means RGB(0...255) <-> Y(16....235))
It does not change when I add converttoYV12(matrix="Rec709").

Creating a .png snapshot from the .mkv and watching it with the ImageSource filter I am getting
RGB(0,255,1) YUV(144,54,34), means a shift in YUV.
Adding converttoYV12(matrix="Rec709") I am getting again
RGB(0,255,1) YUV(173,42,26)

So what do you mean, "I'm getting"? I presume, you mean some way of showing color values, and i think you mean, before passing to avisynth (RGB) and after (YUV), right? Because i'd like to see myself

Selur
I've got no problem with Vegas and the range, meaning black/white levels and consequences for all colors, in fact in Vegas preview window you could see washed out colors clearly when there is a problem.
That's because I illustrate my problem without referring to Vegas, as to me it looks like my problem is RGB->YV12. RGB24 frames from Vegas look fine.

[Sharc]

So what do you mean, "I'm getting"? I presume, you mean some way of showing color values, and i think you mean, before passing to avisynth (RGB) and after (YUV), right? Because i'd like to see myself

Opening the .mkv in AvsPmod for example and using a RGB color picker and compute the YUV of it using the 709 full range matrix equations , or easier customize the status bar of AvsPmod (Options->Program Settings->Video->Customize Video Status Bar .....) to show RGB and YUV values of the pixel under the mouse pointer.
In AvsPmod you can then add convertto(whatever) and see what happens. The conversion matrix defaults according to the resolution, but can also be set by right clicking in the preview -> Display -> YUV -> RGB -> etc.

[poisondeathray]

Here is the OBS recording (source). By the way, in the source picture colors are pure green (0-255-0) and pure yellow (255-255-0).
Screenshot

Now, when feeding this directly to x264, the colors after encoding are the same, which makes sense as the encoder would not need to convert anything.
Screenshot (x264 source -> x264)

If you look at your screenshot "1_recording.png " . "green" is RGB 20,255,10 , "yellow" is RGB 251,255,11 in your screenshot - ie. you don't have pure green or yellow in the screenshot. The reason is you're using 601 matrix for the screenshot, not 709, so the screenshot is the wrong color. This means OBS used Limited range 709 for the conversion, and it's tagged correctly

If you used 709 matrix for "recording.mkv" screenshot, you get 0,255,1 and 253,254,0 - a lot closer. There are slight expected differences when perofrming 8bit RGB/YUV conversions . If you record in RGB directly (you can't with OBS), or 10bit YUV (I think you can in OBS, not sure), then the RGB values will be more accurate

It's the same thing for "2_enc_x264-x264.png", wrong color because wrong matrix used to convert YUV to RGB for screenshot

But in my project, I need to edit the file in Vegas pro, so the final conversion to YV12 is required. I output the frames with Debugmode Frameserver in RGB24.
So the problem is, I tried both feeding RGB to x264, and doing ConvertToYV12() with Avisynth (with different matrices), but unfortunately I can't get the same colors as in the source.

By default, the convention is HD videos use 709, SD 601.

What video are you importing into vegas ? Is it 640x360 ? How are you importing a MKV into vegas? What version of vegas?

By default, vegas uses studio RGB for most native imported YUV formats. You can use the levels presets: studio to computer RGB preset or computer RGB to studio RGB in vegas , but that does not change 601 vs. 709. There are other ways, but it depends on what you are importing

[jagabo]

Here is the OBS recording (source). By the way, in the source picture colors are pure green (0-255-0) and pure yellow (255-255-0).
Screenshot

The mkv file is limited range rec.709. But your png file is the result of convert to RGB with a limited range rec.601 matrix. The green patch is r=20, g=255, b=10. The yellow patch is r=252, g=255, b=10. The greens would have been greater than 255 but they were clamped because 8 bit RGB can't have values over 255. If you convert the source video in AviSynth with ConvertToRGB(matrix="rec709" you get the correct colors, r=0, g=255, b-1, and r=254, g=255, b=0. If you incorrectly use ConvertToRGB(matrix="rec601") you will get the same values as in the PNG image.

So basically, you're method of determining if the colors are correct wrong. This would have been much more obvious if you had used 75% patches (rgb values of 0 and 191). You would have seen that the green patches come back much higher than g=191.

Oops, pdr beat me to it. I paused to play fetch with the dog while composing this message!

[rebus_x]

Sharc
That's very cool, thanks!

If you look at your screenshot "1_recording.png " . "green" is RGB 20,255,10 , "yellow" is RGB 251,255,11 in your screenshot - ie. you don't have pure green or yellow in the screenshot. The reason is you're using 601 matrix for the screenshot, not 709, so the screenshot is the wrong color. This means OBS used Limited range 709 for the conversion, and it's tagged correctly

If you used 709 matrix for "recording.mkv" screenshot, you get 0,255,1 and 253,254,0 - a lot closer. There are slight expected differences when perofrming 8bit RGB/YUV conversions . If you record in RGB directly (you can't with OBS), or 10bit YUV (I think you can in OBS, not sure), then the RGB values will be more accurate

It's the same thing for "2_enc_x264-x264.png", wrong color because wrong matrix used to convert YUV to RGB for screenshot

To clarify, i meant that pure green and yellow were in the source material (image), which i did not show, but here it is. Colors are 0-255-0 and 255-255-0, because in my project the problem showed itself mostly with exactly those.
All snapshots are taken from encoded mkvs in VirtualDub, as i heard that would be more precise and i did see a small difference compared to vlc, for example.

The recording.mkv is generated by OBS, who got that source image on input. That scene was built right inside OBS.

[poisondeathray]

If you look at your screenshot "1_recording.png " . "green" is RGB 20,255,10 , "yellow" is RGB 251,255,11 in your screenshot - ie. you don't have pure green or yellow in the screenshot. The reason is you're using 601 matrix for the screenshot, not 709, so the screenshot is the wrong color. This means OBS used Limited range 709 for the conversion, and it's tagged correctly

If you used 709 matrix for "recording.mkv" screenshot, you get 0,255,1 and 253,254,0 - a lot closer. There are slight expected differences when perofrming 8bit RGB/YUV conversions . If you record in RGB directly (you can't with OBS), or 10bit YUV (I think you can in OBS, not sure), then the RGB values will be more accurate

It's the same thing for "2_enc_x264-x264.png", wrong color because wrong matrix used to convert YUV to RGB for screenshot

To clarify, i meant that pure green and yellow were in the source material (image), which i did not show, but here it is. Colors are 0-255-0 and 255-255-0, because in my project the problem showed itself mostly with exactly those.
All snapshots are taken from encoded mkvs in VirtualDub, as i heard that would be more precise and i did see a small difference compared to vlc, for example.

The recording.mkv is generated by OBS, who got that source image on input. That scene was built right inside OBS.

Yes, "Colors are 0-255-0 and 255-255-0" - is understood

That is why screenshot #1 is incorrect (601 vs 709) . Vdub by default uses 601 to convert YUV to RGB for screenshots, so you get the wrong color if 709 was used for the original RGB to YUV conversion (which OBS is doing)

If you don't understand the explanation earlier, ask for clarification

[rebus_x]

I didn't check all your variants, but starting with the .mkv source I am getting for the green color in 709:
RGB(0,255,1) YUV(173,42,26) which is compliant with the matrix 709 conversion for full range (means RGB(0...255) <-> Y(16....235))
It does not change when I add converttoYV12(matrix="Rec709").

Creating a .png snapshot from the .mkv and watching it with the ImageSource filter I am getting
RGB(0,255,1) YUV(144,54,34), means a shift in YUV, because the .mkv YUV source was converted to RGB24 for viewing.
Adding converttoYV12(matrix="Rec709") I am getting again
RGB(0,255,1) YUV(173,42,26)

At first I did not get these numbers, but turns out, ConvertToYV12(matrix="Rec709") was doing 601 matrix until I remade the example in HD resolution :\

Anyway... Further tests show that "large" pathes of color stay close to original, and thin stripes degrade with each iteration. So what happened is
1. OBS converted the recording RGB->YV12 just like first iteration of ConvertToYV12(matrix="Rec709"). First color degradation
2. In Vegas, it's RGB again
3. On output, it's new ConvertToYV12(matrix="Rec709") and second color degradation

Sad. I was hoping that on further iterations, if everything done right, it somehow wouldn't have to degrade again

Well then. That means, for cases such as this, one needs either source material in RGB, or an editor which works in YUV.

[jagabo]

At first I did not get these numbers, but turns out, ConvertToYV12(matrix="Rec709") was doing 601 matrix until I remade the example in HD resolution

No, it did not. Maybe you just did ConvertToYV12() -- that defaults to rec601.

Anyway... Further tests show that "large" pathes of color stay close to original, and thin stripes degrade with each iteration.

YV12 has subsampled chroma -- the chroma channels are at half the resolution of the luma on each axis. So a 1920x1080 frame has the luma at 1920x1080 but the chroma at 960x540. If you convert from YV12 to RGB the chroma has to be upscaled from 960x540 to 1920x1080. That resizing causes a little blurring/distortion of the chroma. If you then convert back to YV12 the chroma has to be downscaled back to 960x540 resulting in another round of blurring/distortion. You should avoid multiple conversions if they aren't necessary. You can specify the resizing algorithm and chroma placement that ConvertToYV12() uses. See the specs.

http://avisynth.nl/index.php/Convert

Or you can work in YV24 with many filters (no subsampling of the chroma).

Beyond that, 8 bit limited range YUV can only represent about 1/6 as many colors as 8 bit RGB (about 2.7 million different colors vs. 16 million). The first conversion from RGB to YUV causes the most damage. But further conversions back and forth may cause a little more, even in large patches of solid colors. Even looking just at greys: RGB has 256 shades of grey, limited range YUV has only 220.

[rebus_x]

jagabo
Thanks.

You can specify the resizing algorithm and chroma placement that ConvertToYV12() uses.

Yeah, does not help here, unfortunately.

No, it did not. Maybe you just did ConvertToYV12()

Oh yes, it certainly did and I didn't I took my source 640x360 image (not mkv) and did Converttoyv12 right on it, then i took this one to compare and it is HD by height... and RGB numbers for green/yellow after conversion were different