Weird Artifacts when extracting video frames to png in ffmpeg

[Blackcero]

Hello there,

I've been having issues converting a 720p/yuv420p video into its frames as a png image.
While at first everything seems fine, a closer inspection reveals a lot of really weird artifacts in the produced images.
The command I've been using is:

ffmpeg -i "in2.mp4" -r "1.00" -vf scale=out_color_matrix=bt709 "frames\f_%05d.png"

Examples:


I have no clue what is causing this, and after 2 nights of researching this I can simply not find an answer.
Note that using bmp has no impact on the issue. Neither has taking out "-vf scale=out_color_matrix=bt709".

Any help would be greatly appreciated.

Additional ffmpeg output/info:

PS D:\ffmpeg\bin> ffmpeg -i "in2.mp4" -r "1.00" -vf scale=out_color_matrix=bt709 "frames\f_%05d.png"
ffmpeg version N-91399-g9f0077cc03 Copyright (c) 2000-2018 the FFmpeg developers
built with gcc 7.3.0 (GCC)
configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-bzlib --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --enable-libmfx --enable-amf --enable-ffnvcodec --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth
libavutil 56. 18.102 / 56. 18.102
libavcodec 58. 20.104 / 58. 20.104
libavformat 58. 17.101 / 58. 17.101
libavdevice 58. 4.101 / 58. 4.101
libavfilter 7. 25.100 / 7. 25.100
libswscale 5. 2.100 / 5. 2.100
libswresample 3. 2.100 / 3. 2.100
libpostproc 55. 2.100 / 55. 2.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'in2.mp4':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2mp41
Duration: 00:03:23.63, start: 0.000000, bitrate: 1047 kb/s
Stream #0:0(und): Video: h264 (Main) (avc1 / 0x31637661), yuv420p(tv, bt709), 1280x720 [SAR 1:1 DAR 16:9], 868 kb/s, 30 fps, 30 tbr, 15360 tbn, 60 tbc (default)
Metadata:
handler_name : VideoHandler
Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 173 kb/s (default)
Metadata:
handler_name : SoundHandler
Stream mapping:
Stream #0:0 -> #0:0 (h264 (native) -> png (native))
Press [q] to stop, [?] for help
Output #0, image2, to 'frames\f_%05d.png':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2mp41
encoder : Lavf58.17.101
Stream #0:0(und): Video: png, rgb24, 1280x720 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 1 fps, 1 tbn, 1 tbc (default)
Metadata:
handler_name : VideoHandler
encoder : Lavc58.20.104 png
frame= 205 fps= 19 q=-0.0 Lsize=N/A time=00:03:25.00 bitrate=N/A dup=0 drop=5903 speed=19.2x
video:29757kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown

[amaipaipai]

This looks like the effects of YV12 to me.

[Blackcero]

This looks like the effects of YV12 to me.

Thank you for your answer, but as I am quite unexperienced with ffmpeg I can't really apply this information to my situation.
If the above is the case, is there any way to circumvent this behavior?

Edit:
Trying out this command:

ffmpeg -i "in2.mp4" -r "1.00" -vf scale=out_color_matrix=bt709 -pix_fmt rgb24 "frames\f_%05d.png"

to bypass any YV12 conversion which might happen by default produces the same issues as in the original post.

[amaipaipai]

I don't use ffmpeg for my stuff, but wait some more, maybe some one could help you out with ffmpeg options.
From what I can see your stream is yuv420p, maybe if you could use yuv422p instead, please notice that some devices might not read or understand yuv422p, high422 for ffmpeg.

With avisynth I always use at the end of my script:

ConvertToRGB32(matrix="Rec709")

But, maybe this could help:
https://trac.ffmpeg.org/wiki/Encode/H.264
https://video.stackexchange.com/questions/13164/encoding-422-in-10-bit-with-libx264

[Blackcero]

I don't use ffmpeg for my stuff, but wait some more, maybe some one could help you out with ffmpeg options.
From what I can see your stream is yuv420p, maybe if you could use yuv422p instead, please notice that some devices might not read or understand yuv422p, high422 for ffmpeg.

With avisynth I always use at the end of my script:

ConvertToRGB32(matrix="Rec709")

But, maybe this could help:
https://trac.ffmpeg.org/wiki/Encode/H.264
https://video.stackexchange.com/questions/13164/encoding-422-in-10-bit-with-libx264

Thank you very much for your help!
I have converted the video to yuv422p and extracted the frames from that without any issues.
However, I have one last side question, if it's not too much, since I did experiment around in avisynth a little bit.
My script was quite simple, same yuv420p origin video:

LSMASHVideoSource(source ="D:\ffmpeg\bin\in2.mp4")
ConvertToRGB(matrix="Rec709")
ImageWriter(file = "D:\ffmpeg\bin\f_%05d.bmp", type="bmp")

this produced quite a nice result at first glance yet again, but zooming in I saw this:


It seems like at the right and left borders of the orange the color brightness somehow got reversed.
Do you have any idea what could be causing this?

[pandy]

Please test if you observing same issue when those option are used by scale:

scale=iw:ih:sws_flags=spline+accurate_rnd+full_chroma_int+full_chroma_inp:out_color_matrix=bt709

i've struggle with similar issues on ffmpeg from long time - and i have mixed results - ffmpeg has nasty habit to call automatically libswscale in background without telling anything - check report thoroughly. Also it is good to explicitly control quantization range in ffmpeg (separately for input and output) frequently prefer to manually rector limited quantization range. If absolute processing accuracy is mandatory for you then Avisynth behave way better than ffmpeg.

[poisondeathray]

How are you taking the "original" screenshot ? What software or algorithm ?

It's not really the "original"; you're looking at a RGB converted representation of the original . (You started with YUV 420) . It's not necessarily "correct" either.

There is a difference in chroma upsampling method. ie. How ffmpeg or avisynth is converting YUV 420 to RGB is different than the method you used for taking the "original" screenshot . If you have 1280x720 4:2:0, this means the U,V planes which contain the color information are only 640x360 in dimensions. How they get scaled to 1280x720 in RGB determines what you see

There are dozens of different ways you can do this, even within avisynth or ffmpeg. The biggest difference will be in the chroma center interpretation, and which scaling algorithm you choose) . You can choose different algorithms eg. spline, bicubic, etc... Some are sharper but cause aliasing, some are softer, some cause ringing etc...

(Although there shouldn't be a difference between PNG and BMP output for ffmpeg; there is one known difference - it writes a PNG gAMA tag, which causes some software to display PNG differently - but this is in terms of colors (R,G,B values) , not chroma upsampling which your screenshots show)

[Blackcero]

Please test if you observing same issue when those option are used by scale:

scale=iw:ih:sws_flags=spline+accurate_rnd+full_chroma_int+full_chroma_inp:out_color_matrix=bt709

i've struggle with similar issues on ffmpeg from long time - and i have mixed results - ffmpeg has nasty habit to call automatically libswscale in background without telling anything - check report thoroughly. Also it is good to explicitly control quantization range in ffmpeg (separately for input and output) frequently prefer to manually rector limited quantization range. If absolute processing accuracy is mandatory for you then Avisynth behave way better than ffmpeg.

Hmm, I've tried your suggestion and to me it seems like if I convert the video to yuv444p before splitting it into its frames I get better results, though slightly off-color which however seems to be fixed when turning the frames back into a video. It is quite hard to tell though.

At this point I am quite interested in just using avisynth for this process, however in my previous post I have detailed
another issue I'm encountering when using avisynth to extract the frames as bmp.

How are you taking the "original" screenshot ? What software or algorithm ?

It's not really the "original"; you're looking at a RGB converted representation of the original . (You started with YUV 420) . It's not necessarily "correct" either.

There is a difference in chroma upsampling method. ie. How ffmpeg or avisynth is converting YUV 420 to RGB is different than the method you used for taking the "original" screenshot . If you have 1280x720 4:2:0, this means the U,V planes which contain the color information are only 640x360 in dimensions. How they get scaled to 1280x720 in RGB determines what you see

There are dozens of different ways you can do this, even within avisynth or ffmpeg. The biggest difference will be in the chroma center interpretation, and which scaling algorithm you choose) . You can choose different algorithms eg. spline, bicubic, etc... Some are sharper but cause aliasing, some are softer, some cause ringing etc...

(Although there shouldn't be a difference between PNG and BMP output for ffmpeg; there is one known difference - it writes a PNG gAMA tag, which causes some software to display PNG differently - but this is in terms of colors (R,G,B values) , not chroma upsampling which your screenshots show)

I'm using MPC-HC with http://www.cccp-project.net/ to screenshot the frame over the File->Save Image option.

Edit: Upon further testing, I guess the effect I'm seeing in AviSynth is most likely caused by the chroma upsampling method, same as when I'm using pandy's suggested scaling options.
I guess this means there is basically no way to avoid this?

[poisondeathray]

You're starting with 1/2 width, 1/2 height color information. There is no way unless you start with 4:4:4. That's pixel perfect color , no subsampling. Which algorithm you choose determines how that gets converted to RGB (full color)

You can get the same results as that MPCHC with CCCP is using if you choose the same and settings algorithm that it is using. It's chroma upsampling too. Notice the colors don't line up perfectly either. The point is what you "think" is "original", isn't really "original" either. It's just a representation

[Blackcero]

You're starting with 1/2 width, 1/2 height color information. There is no way unless you start with 4:4:4. That's pixel perfect color , no subsampling. Which algorithm you choose determines how that gets converted to RGB (full color)

You can get the same results as that MPCHC with CCCP is using if you choose the same and settings algorithm that it is using. It's chroma upsampling too. Notice the colors don't line up perfectly either. The point is what you "think" is "original", isn't really "original" either. It's just a representation

Thank you for your help as well!

It turns out that the weird brightness issues with AviSynth are simply a product of my video player after all.
I consider the results to be as close to perfect as I could get them now, so no further replies are needed.

Thank you everyone for your help, this community is great!

[pandy]

There is also other issue with 4:2:0 YCbCr used today - chroma samples location - in MPEG-1 chroma sample was located "between" luma samples (bilinear interpolation) later (MPEG-2 and up) chroma samples are located only on even luma samples (for interlace it is even more complex) as such some leading/trailing edge issues may occur https://en.wikipedia.org/wiki/Chroma_subsampling