ffmpeg compositing?

[scottpix]

It seems to me ffmpeg has a composite function maskedmerge which takes 2 videos and a mask to combine.
They also have overlay filter that takes two videos and assumes one might have an alpha so requires a video codec that can hold an alpha or requires alphamerge fucntion to add one.

From my limited experiements it seems that neight one of these produces a solid composite. If I take 2 mp4 video and a grayscale png file as a mask that goes from full white to black and used it with mergedmask I woudl get a solid image of one video in areas of pure black and the other video in areas of pure white.
Unfortuantley it looks liek the opacity never is 100% and simple results in a partial composite with ghosting.

Overlay seems to exhibit the same problem. I've tried different formats and materials knowing that ffmpeg sometimes deals with colorspaces in not obvious ways.

Do these work and if so how can I do a simple a over b composite with a mask (still or video)?

[jagabo]

My guess is you're converting the mask image from RGB to YUV using a limited range matrix. That puts RGB black at Y=15, and RGB white at Y=235. So when the Y values is used as an alpha mask it is never completely transparent or opaque. Convert the RGB to YUV with a full range matrix to avoid that.

[poisondeathray]

^ yes . It's what jagabo said. PNG is in RGB, you need a full range YUV mask if your inputs are YUV video. It should work as-is if all inputs are RGB, or if your mask is full range YUV video

It would look something like this for a png image mask

eg

ffmpeg -i input1.mp4 -i input2.mp4 -r 24 -i mask.png -filter_complex "[2:v]zscale=range=full:matrix=709,format=yuv420p[y]; [0:v][1:v][y]maskedmerge[out]" -map "[out]" -c:v libx264 -crf 18 -an out.mp4

[scottpix]

Thanks 03jagabo and poisondeathray, I will certainly give that a try.
The wonders of video.

[poisondeathray]

Looking at it more closely, maskedmerge does not work as expected. You're supposed to be able to use the Y plane, plane 0 as the mask. I tried converting to gray instead of yuv420p, and specifying maskedmerge=planes= . The syntax for the vapoursynth maskedmerge version works as expected, because it has a first_plane=True argument

A workaround is to use overlay, and mergeplanes to generate the "a" or alpha channel (in this example, yuva420p). You still need full range if working with YUV and RGB assets. In my example I had to reset the SAR, that's what the setsar is for in case you're wondering. You might not need to for your videos

ffmpeg -i base.ext -i overlay.ext -r 24 -i mask.png -filter_complex "[2:v]setsar=sar=1/1,zscale=range=full:matrix=709,format=yuv420p[m]; [1:v]setsar=sar=1/1[s]; [s][m]mergeplanes=0x00010210:yuva420p[alpha]; [0:v][alpha]overlay[out]" -map "[out]" -c:v libx264 -crf 18 -an out5.mp4 -y

[pandy]

Had in past some exercise with alpha in ffmpeg - for picture with alpha embedded and working as a mask:

@ffmpeg.exe -hide_banner -v 32 -stats -strict -1 -i %filename% -an  -filter_complex "[0:v]scale=773:439:sws_flags=spline+accurate_rnd+full_chroma_int+full_chroma_inp:interl=-1:out_range=full,pad=1920:1080:990:224[a0];movie=bg.png,premultiply=inplace=1[b0];[a0][b0]overlay=0:0:alpha=premultiplied,scale=960:-2,format=pix_fmts=yuv420p" -c:v ffv1 -vsync 0 -f NUT - | ffplay.exe -hide_banner -loglevel 32 -autoexit -i -

Above allow to work with ffmpeg and get visual feedback from it trough pipe to ffplay - dirty but works - key is premultiply filter

[richardpl]

Looking at it more closely, maskedmerge does not work as expected. You're supposed to be able to use the Y plane, plane 0 as the mask. I tried converting to gray instead of yuv420p, and specifying maskedmerge=planes= . The syntax for the vapoursynth maskedmerge version works as expected, because it has a first_plane=True argument

A workaround is to use overlay, and mergeplanes to generate the "a" or alpha channel (in this example, yuva420p). You still need full range if working with YUV and RGB assets. In my example I had to reset the SAR, that's what the setsar is for in case you're wondering. You might not need to for your videos

ffmpeg -i base.ext -i overlay.ext -r 24 -i mask.png -filter_complex "[2:v]setsar=sar=1/1,zscale=range=full:matrix=709,format=yuv420p[m]; [1:v]setsar=sar=1/1[s]; [s][m]mergeplanes=0x00010210:yuva420p[alpha]; [0:v][alpha]overlay[out]" -map "[out]" -c:v libx264 -crf 18 -an out5.mp4 -y

maskedmerge in FFmpeg works as expected, please do not tell otherwise, you need proper mask, and not only one plane acting as mask. If you only have mask as single plane use overlay filter. You still can use maskedmerge, even for YUV, but you will need to properly clone and rescale mask plane to other planes. OP never provided input mask file.