ffplay playback BT.709 yuv

[colibri]

I download a video from youtube with youtube-dl tool. and check the file format is AVC BT.709.

Then decode first frame to YUV with below command:

ffmpeg -i 256x144.mp4 -vcodec rawvideo -pix_fmt yuv420p -vframes 1 256x144.yuv

When I try below command to playback the yuv:

ffplay -v info -f rawvideo -pixel_format nv12 -video_size 256x144 256x144.yuv

I saw the color is totally wrong, but if I decode it to png file, then open the png, the color is good.

So I guess ffplay use BT.601 playback the BT.709 yuv file wrongly.

How should I use ffplay to playback BT.709 YUV?

[poisondeathray]

I download a video from youtube with youtube-dl tool. and check the file format is AVC BT.709.

How did you check or verify this? Youtube can make mistakes, or uploader can make mistake. Sometimes videos are encoded or tagged incorrectly.

If you want to verify what ffmpeg or ffplay does, you should use known colors, knonw test videos, known values

Then decode first frame to YUV with below command:

ffmpeg -i 256x144.mp4 -vcodec rawvideo -pix_fmt yuv420p -vframes 1 256x144.yuv

When I try below command to playback the yuv:

ffplay -v info -f rawvideo -pixel_format nv12 -video_size 256x144 256x144.yuv

What is "totally wrong?" 709 vs 601 is a slightly color shift, not "totally wrong" . "Totally wrong" implies major difference. Maybe wrong pixel format.

nv12 is not the same as yuv420p

You should use ffplay -pixel_format yuv420p, not nv12

I saw the color is totally wrong, but if I decode it to png file, then open the png, the color is good.

How did you "decode to png". What method did you use to convert to RGB ?

[poisondeathray]

If you wanted to test nv12, convert the input pixel format for yuv video to nv12. Not yuv420p

eg.

ffmpeg -i input.ext -vcodec rawvideo -pix_fmt nv12 -vframes 1 nv12.yuv

ffplay -f rawvideo -pixel_format nv12 -video_size 256x144 nv12.yuv
ffplay -f rawvideo -pixel_format yuv420p -video_size 256x144  nv12.yuv

Now "yuv420p" looks "wrong" and "nv12" looks correct . Because the yuv file was converted nv12, not yuv420p

[colibri]

@poisondeathray Thanks, after change it to nv12, it looks good now. but I still need to make below points clear:

1. If the input file is BT.709 format (view it with mediainfo tool), does the ffmpeg decoded output yuv file is in BT.709 format?
2. If so, ffplay should not know what's the yuv format (BT.709 or BT.601), how can ffplay playback it correctly?

[poisondeathray]

1. If the input file is BT.709 format (view it with mediainfo tool), does the ffmpeg decoded output yuv file is in BT.709 format?

You're assuming this; it can be wrong; e.g. wrong metadata, wrong flags

For example, SD should be 601, HD should be 709 by convention . If your 256x144 file was flagged 709, somewhere there is likely a problem. Maybe converted incorrectly upstream, maybe tagged incorrectly etc...

2. If so, ffplay should not know what's the yuv format (BT.709 or BT.601), how can ffplay playback it correctly?

ffmpeg decodes yuv as yuv . There is no "matrix" until you convert to RGB . "matrix" refers to YUV<=>RGB conversions

ffmpeg reads some flags automatically and sometimes the behaviour changes on certain releases (for example it autoscales full range), but you can explictly control the matrix for RGB conversion with -vf scale flags . eg.

e.g. convert to RGB24 using 709 matrix

-vf scale=in_color_matrix=bt709,format=rgb24

The direction points to the "YUV" side. So if you were converting RGB to YUV, it would be out_color_matrix=bt709,format=yuv420p

For ffmpeg, when saving to a RGB image format, be careful about ffmpeg png output, it's writes png flags such as gama and colr tags, and different applications can read the colors differently (sometimes input flags affect the png flags as well). For 8bit RGB, if you save as bmp, it should look the same everywhere

I think ffplay uses 709 for HD, 601 for SD by default for the RGB preview (it still decodes YUV as YUV). It might obey some flags , so if you have wrong flags, it might show you the wrong results (again, it varies by release or ffmpeg binary). But to explictly control the conversion to rgb for the ffplay RGB preview, use -vf scale flags

for YUV to RGB using 601 matrix

-vf scale=in_color_matrix=bt601,format=rgb24

for YUV to RGB using 709 matrix

-vf scale=in_color_matrix=bt709,format=rgb24

[colibri]

Agree with you, matrix is for YUV to RGB.

But the problem is: when use ffmpeg decode BT709 to yuv, then we should use BT709 matrxi to convert yuv to RGB, but when playback yuv with ffplay, nobody give such informaton to ffplay. how ffplay know current yuv is BT601 or BT709?

[poisondeathray]

Agree with you, matrix is for YUV to RGB.

But the problem is: when use ffmpeg decode BT709 to yuv, then we should use BT709 matrxi to convert yuv to RGB, but when playback yuv with ffplay, nobody give such informaton to ffplay. how ffplay know current yuv is BT601 or BT709?

When you decode some compressed YUV video file to raw YUV, the matrix does not matter. Matrix is only for YUV<=>RGB conversions

raw YUV does not have any metadata

YUV is YUV (not RGB)

I think ffplay's RGB converted representation of YUV depends on the HD vs. SD by default. HD 709, SD 601. That's the assumption it makes.

But you can control it explictly with -vf scale as above, same as for ffmpeg

[poisondeathray]

ffplay ignores the colormatrix flag

eg. a compressed sd mp4 , with purposely incorrect 709 flag (as read by mediainfo), is displayed as RGB in ffplay using 601 by default (correct colors, because I had know values) . If you force the conversion to RGB using 709, the colors are incorrect as expected

eg. a compressed hd mp4 , is displayed correctly with or without flags, or incorrect flags, as 709 by default . (again, known values to verify)

It seems to be resolution dependent. Many software media players behave like this too. The cutoff is usually 576 height

[colibri]

Oh, that means flags is useless but use video size to decide use BT601 or BT709 ( < 576,BT601, >576, BT709). which tool we can use to convert BT601 YUV to BT709 YUV or vice versa?

[poisondeathray]

Oh, that means flags is useless but use video size to decide use BT601 or BT709 ( < 576,BT601, >576, BT709). which tool we can use to convert BT601 YUV to BT709 YUV or vice versa?

For ffplay's automatic RGB conversion, that' s what it looks like

Convert what to what ? to RGB ?

YUV to RGB using 601 ? YUV to RGB using 709 ? RGB to YUV using 601? RGB to YUV using 709 ? YUV to YUV using 601<=>709 (ie. "swap" colors in YUV as if it used the other matrix from a prior RGB conversion e.g. -vf colormatrix) ?

eitherway for any of those you can use ffmpeg , avisynth , vapoursynth etc...



e.g. YUV 601 to YUV 709 (just swap them for the opposite 709=>601)

(ie. convert in YUV, "as if" it was converted from RGB using a prior RGB=>YUV conversion using 709. This conversion occurs in YUV, not RGB)

using -vf colormatrix

-vf colormatrix=bt601:bt709

or

using -vf zscale

-vf zscale=matrixin=470bg:matrix=709

[colibri]

Thanks, to be clear, I am checking how much difference if BT601 has been used as BT709 wrongly.

1. sample file nv21.yuv is 3840x2160 sample, it should be a BT.601 yuv file.
2. run below command to convert the nv21.yuv to bt709.yuv (BT601->BT709).

ffmpeg -y -f rawvideo -pix_fmt nv12 -s 3840x2160 -r 25 -i nv12.yuv \
-vf colormatrix=bt601:bt709 \
-c:v rawvideo -pix_fmt nv12 bt709.yuv


3. convert nv21.yuv and bt709.yuv to png, then merge it together (just for easy compare).
I do not see much different between this two png file.

nv21.yuv is here
merge.png is here

Not sure if I am doing it wrongly.

[poisondeathray]

It's right. Colors are clearly different. As mentioned earlier, a 601 /709 mismatch is just a slight shift in colors. You can usually see it on reds/greens and skin tones. But "totally wrong" is usually something else, like wrong pixel format or not decoding correctly

It's usually easier to see differences when superimposed (swap between images), rather than stacked, because of human visual memory

The top is YUV converted to RGB for the PNG with Rec601

The bottom is YUV converted with colormatrix to 709 then converted to RGB for the PNG with Rec601 . If you instead converted to RGB using Rec709 instead after the colormatrix filter, it would then look similar to top (minor rounding differences)

You can also convert to RGB directly with Rec709 . All 3 should give slightly different results.

(And UHD commonly uses Rec2020 for many sources)

[colibri]

Thanks,

Oh, maybe I made a mistake again!

I use below command to convert the so-called (fake) bt709.yuv to png:

ffmpeg -f rawvideo -pixel_format nv12 -video_size 3840x2160 -i bt709.yuv -vframes 1 out.png

But I am thinking about if it's BT.709 yuv, then we need to convert it back to BT.601 firstly, then encode with above command to png.

Since we do not tell ffmpeg which yuv format it is, by default, BT.601 is used I guess.

[poisondeathray]

When you converted to png, yes 601 was used.

That's why I asked in the 1st reply "How did you "decode to png". What method did you use to convert to RGB ? " If you didn't specify anything and there were no flags, 601 would be used


You can control how ffmpeg converts YUV=>RGB (for PNG, or other purposes). You can tell it to use 709 for the YUV=>RGB conversion, for example. See post #5

The -vf colormatrix filter is only for YUV<=>YUV conversions

Because these are calculated 8bit precision, there are rounding errors. It's better, more accurate, faster to go directly (convert to RGB using 709) , instead of going through colormatrix, than converting to RGB . Assuming that's what you wanted

It's not clear what you're doing or why. Or why "NV12" instead of yuv420p ? The original most likely would be yuv420p (although most YUV 4:2:0 fourcc's are interconvertible losslessly in ffmpeg). yuv420p is "YV12", it's far more common

[colibri]

When you converted to png, yes 601 was used.

That's why I asked in the 1st reply "How did you "decode to png". What method did you use to convert to RGB ? " If you didn't specify anything and there were no flags, 601 would be used

I decode to png with below command firstly:

ffmpeg -i bt709.mp4 -vframes 1 out.png

But after reading your comment, I do a compare with below 3 commands:
A). ffmpeg -i in.mp4 -vframes 1 out.png
B). ffmpeg -i in.mp4 -vframes 1 -vf scale=in_color_matrix=bt601,format=rgb24 out.png
C). ffmpeg -i in.mp4 -vframes 1 -vf scale=in_color_matrix=bt709,format=rgb24 out.png

The result is A almost equal to B, but C a little bit brighter than A and B. so I think ffmpeg by default using BT.601 to convert YUV to RGB.(flag is in mp4 file, not sure why ffmpeg not use that information)

It's not clear what you're doing or why. Or why "NV12" instead of yuv420p ? The original most likely would be yuv420p (although most YUV 4:2:0 fourcc's are interconvertible losslessly in ffmpeg). yuv420p is "YV12", it's far more common

Since I am using android device, android use NV12 instead of yuv420p. what I am trying to do is figure out what color format is output from android Camera. some device output from android is BT.601 and others is BT.709. now my understanding is: from encoder, in and out is YUV, so encoder should not care about BT601 or BT709. then it should be camera output, if camera output is BT601, after encoder decode, output will be BT601, the mp4 file should be taged as BT601. otherwise, if the tag is BT709, color will be wrong.

I am thinking about if we can get a method to verify if the android device tag BT601 or BT709 correct or not.

[poisondeathray]

When you converted to png, yes 601 was used.

That's why I asked in the 1st reply "How did you "decode to png". What method did you use to convert to RGB ? " If you didn't specify anything and there were no flags, 601 would be used

I decode to png with below command firstly:

ffmpeg -i bt709.mp4 -vframes 1 out.png

But after reading your comment, I do a compare with below 3 commands:
A). ffmpeg -i in.mp4 -vframes 1 out.png
B). ffmpeg -i in.mp4 -vframes 1 -vf scale=in_color_matrix=bt601,format=rgb24 out.png
C). ffmpeg -i in.mp4 -vframes 1 -vf scale=in_color_matrix=bt709,format=rgb24 out.png

The result is A almost equal to B, but C a little bit brighter than A and B. so I think ffmpeg by default using BT.601 to convert YUV to RGB.(flag is in mp4 file, not sure why ffmpeg not use that information)

It's not clear what you're doing or why. Or why "NV12" instead of yuv420p ? The original most likely would be yuv420p (although most YUV 4:2:0 fourcc's are interconvertible losslessly in ffmpeg). yuv420p is "YV12", it's far more common

Since I am using android device, android use NV12 instead of yuv420p. what I am trying to do is figure out what color format is output from android Camera. some device output from android is BT.601 and others is BT.709. now my understanding is: from encoder, in and out is YUV, so encoder should not care about BT601 or BT709. then it should be camera output, if camera output is BT601, after encoder decode, output will be BT601, the mp4 file should be taged as BT601. otherwise, if the tag is BT709, color will be wrong.

I am thinking about if we can get a method to verify if the android device tag BT601 or BT709 correct or not.

But do you access "raw" camera data, or a compressed recording such as MP4 on a card ?

If you can only access the MP4, all that matters is what the MP4 is (many steps can occur between raw camera data, debayering, conversion to YUV, subsampling) . If you can't access any of those prior steps, it's irrelevant if android uses NV12 or something else

All you would do it check the MP4 converted to RGB with 601 or 709, compared to what you actually see.

Again, don't use png, because there can be png flags written by ffmpeg. Depending on what you are using to view the png, the colors can be different. For 8bit RGB , use BMP