[SOLVED] Ghost image issues with ffmpeg -filter_complex displace

[Crul]

Hi, noob here

I've (almost) been able to apply a displacement based on 2 animated gaussian noise videos, but I'm having issues with a ghost image. A picture is worth a thousand words

Here you have a script to replicate the issue:

ffmpeg -y -t 2 -f lavfi -i color=c=blue:s=160x120 -c:v libx264 -tune stillimage -pix_fmt rgb24 00_empty.mp4
ffmpeg -y -i 00_empty.mp4 -vf "drawtext=text=string1:y=h/2:x=w-t*w/2:fontcolor=white:fontsize=60" 01_text.mp4
ffmpeg -y -t 2 -f lavfi -i color=c=gray:s=160x120 -c:v libx264 -tune stillimage -pix_fmt rgb24 02_gray.mp4
ffmpeg -y -i 01_text.mp4 -i 02_gray.mp4 -i 02_gray.mp4 -filter_complex "[0][1][2]displace=edge=mirror" 03_displaced_text.mp4

It creates a test video with a scrolling text and a gray dummy video. Then it applies a displacement based on the gray video. If I understand correctly, because the gray video is 100% gray, it should leave the video unchanged (or maybe displace everything by a fixed ammount of pixels), but it creates a "shadow". I tried with 3 different pixel formats (yuv420p, yuv444p, rgb24) because I found this question on stackoverflow talking about that:

- Why are Cb and Cr planes displaced differently from lum by the displace complex filter in ffmpeg?

Windows 10
ffmpeg version 5.0.1-full_build-www.gyan.dev

EDIT: Thanks to @poisondeathray for the solution, using -pix_fmt rgb24 and adding -c:v libx264rgb fixes it. This works:

ffmpeg -hide_banner -y -t 2 -f lavfi -i color=c=blue:s=160x120 -tune stillimage -c:v libx264rgb -pix_fmt rgb24 00_empty.mp4
ffmpeg -hide_banner -y -i 00_empty.mp4 -vf "drawtext=text=string1:y=h/2:x=w-t*w/2:fontcolor=white:fontsize=60" -c:v libx264rgb -pix_fmt rgb24 01_text.mp4
ffmpeg -hide_banner -y -t 2 -f lavfi -i color=c=gray:s=160x120 -tune stillimage -c:v libx264rgb -pix_fmt rgb24 02_gray.mp4
ffmpeg -hide_banner -y -i 01_text.mp4 -i 02_gray.mp4 -i 02_gray.mp4 -filter_complex "[0][1][2]displace=edge=mirror" -c:v libx264rgb -pix_fmt rgb24 03_displaced_text.mp4

Any idea will be welcome.
Thanks!

[poisondeathray]

you tried rgb24, but probably didn't specify libx264rgb (or another RGB format), so it converts to yuv 4:2:0 , this causes subsampled chroma planes (1/2 width, 1/2 height)

ffmpeg -y -t 2 -f lavfi -i color=c=blue:s=160x120 -c:v libx264rgb -tune stillimage -pix_fmt rgb24 04_empty.mp4
ffmpeg -y -i 00_empty.mp4 -vf "drawtext=text=string1:y=h/2:x=w-t*w/2:fontcolor=white:fontsize=60" -c:v libx264rgb 05_text.mp4
ffmpeg -y -t 2 -f lavfi -i color=c=gray:s=160x120 -c:v libx264rgb -tune stillimage -pix_fmt rgb24 06_gray.mp4
ffmpeg -y -i 05_text.mp4 -i 06_gray.mp4 -i 06_gray.mp4 -filter_complex "[0][1][2]displace=edge=mirror" -c:v libx264rgb 07_displaced_text.mp4

[poisondeathray]

Actually, looking at the code, the real reason is the 8it RGB to YUV gray conversion for 02_gray.mp4 becomes YUV 126,128,128 (loss of accuracy from the 8bit conversion)
https://github.com/FFmpeg/FFmpeg/blob/master/libavfilter/vf_displace.c

If you supply a true gray YUV 128,128,128 video it works ok too

[Crul]

you tried rgb24, but probably didn't specify libx264rgb (or another RGB format), so it converts to yuv 4:2:0

You're right that it gives an error, but it says it uses you444p: Incompatible pixel format 'rgb24' for codec 'libx264', auto-selecting format 'yuv444p'

Anyway, with you444p it should work, but....

Actually, looking at the code, the real reason is the 8it RGB to YUV gray conversion for 02_gray.mp4 becomes YUV 126,128,128

This makes perfect sense! I'll try to specify RGB instead of YUV.

Thank you very much!

[Crul]

Confirmed, this works:

ffmpeg -hide_banner -y -t 2 -f lavfi -i color=c=blue:s=160x120 -tune stillimage -c:v libx264rgb -pix_fmt rgb24 00_empty.mp4
ffmpeg -hide_banner -y -i 00_empty.mp4 -vf "drawtext=text=string1:y=h/2:x=w-t*w/2:fontcolor=white:fontsize=60" -c:v libx264rgb -pix_fmt rgb24 01_text.mp4
ffmpeg -hide_banner -y -t 2 -f lavfi -i color=c=gray:s=160x120 -tune stillimage -c:v libx264rgb -pix_fmt rgb24 02_gray.mp4
ffmpeg -hide_banner -y -i 01_text.mp4 -i 02_gray.mp4 -i 02_gray.mp4 -filter_complex "[0][1][2]displace=edge=mirror" -c:v libx264rgb -pix_fmt rgb24 03_displaced_text.mp4

Thanks again!

[poisondeathray]

And that's the actually the expected YUV value (Y=126) for gray RGB (RGB 128,128,128) to YUV using a (standard) limited range conversion. If you specified full range, you would get YUV 128,128,128 and it would work too in YUV

Another way would be to use -vf lutyuv to force the y value to 128. I don't think ffmpeg has a YUV color generator where you can specify Y,U,V values directly

[Crul]

All these options are confusing, but I'm learning... slowly.

Thanks for the info.