ffmpeg - alpha channel posterization

[iisperi]

Hi! I am trying to extract alpha channel from a video, and then posterize it so that every pixel that is not completely transparent is converted to a solid color that I set. Otherwise, it should be left transparent.

I am constructing a vf for that, but it's my first time working with geq, and it doesn't seem to work the way I do it:

ffmpeg -i "input" -vf "[in]alphaextract[alph];[alph]geq='if(gt(alpha(X,Y),0),255,0)'[out]" "output"

Could you please help me achieve the result with ffmpeg?
Thank you!

[Cornucopia]

That isn't really how alpha channels work. It is just an additional grayscale channel.
You can of course extract it - if you don't have other proper dedicated tools, you can always use Avisynth.
You can also posterize that.
But a posterized grayscale is just a posterized grayscale. If that is being used as the alpha channel, that just means you have stepped levels of transparency. It doesn't allow you to isolate elements other than perhaps as masking of borders. And changing other elements to particular colors is a multi-step procedure combining repetitions of masking selections and inpainting.


Scott

[iisperi]

That isn't really how alpha channels work. It is just an additional grayscale channel.
You can of course extract it - if you don't have other proper dedicated tools, you can always use Avisynth.
You can also posterize that.
But a posterized grayscale is just a posterized grayscale. If that is being used as the alpha channel, that just means you have stepped levels of transparency. It doesn't allow you to isolate elements other than perhaps as masking of borders. And changing other elements to particular colors is a multi-step procedure combining repetitions of masking selections and inpainting.


Scott

Thank you, Scott!

I understand that an alpha channel is but another grayscale channel; however, I hoped I could work with it after the alpha extract.
Do you think I can still combine this multi-step operation into the ffmpeg vf? If that is a grayscale video, I should be able to use any of the r/g/b channels to get the pixel value and then output the target value using if and gt operations. However, I am failing at that.

[Cornucopia]

I don't often work with ffmpeg's vf module, because I consider the procedure awkward and unintuitive (which can lead to mistakes), and there are better alternatives such as Avisynth or AE. So, I would not recommend doing it that way, nor am I the person to go to for that.

If you truly expect to script it, you'll have to be able to describe your selection targets precisely and uniquely for this to not end up a mess.


Scott

[poisondeathray]

If that is a grayscale video, I should be able to use any of the r/g/b channels to get the pixel value and then output the target value using if and gt operations.

No , it's a separate "A" channel, independent of RGB or YUV channels

alphaextract outputs a full range Y8 channel (assuming you're dealing with 8bit), even from RGB sources

The mask portion using lutyuv (instead of geq would look like)

alphaextract,lutyuv=y='if(gt(val,0),255,val)

You would use -filter_complex since you have a flowchart of inputs/output nodes, not 1 linear input/output. One you have the binarized mask output node, you can overlay any color over the original video, using the new binarized mask.

There are some "gotchas" depending if you start with YUVA or RGBA , because the "A" channel is always full range, but YUV "video" is normally converted from limited range YUV to full range RGB . FFmpeg can sometimes have some autoinsert behaviour issues

[pandy]

I use something like this:

alphaextract,colorlevels=aimin=127/256:aimax=129/256

[poisondeathray]

Here is one way of doing it , using alphaextract,lutyuv, alphamerge

It will look something like this, you have to adjust it for your case

ffmpeg -i input -f lavfi -i color=color=0xFF000000:size=(width)x(height):r=(framerate) -filter_complex "[0:v]alphaextract,lutyuv=y='if(gt(val,0),255,val)'[m]; [1:v][m]alphamerge[coloralpha]" -map [coloralpha] output

input


Alphaextact, you can see the partial transparency of the filmstrip (just 1 frame for demo)

every pixel that is not completely transparent is converted to a solid color that I set. Otherwise, it should be left transparent.

alphaextract+Lutyuv - Because it's a binarized hard mask (all values >0 become 1) , the edges will be aliased and sharp - you can blur the alpha or increase the threshold value (just 1 frame for demo)




Alphamerge merges the modifed alpha with the color layer input (red in this example)

[iisperi]

Very cool approach, thank you!
However, when I run it on my machine, after setting time limit for the color input, I get a completely red frame on the prores4444 footage:
https://www.dropbox.com/scl/fi/jauu60e19a7h32232qpxb/Comp-1_1.mov?rlkey=91o47ywhwkrxdf...hr2ofh489&dl=0

What else do I have to adjust in this case?

Here is one way of doing it , using alphaextract,lutyuv, alphamerge

ffmpeg -i input -f lavfi -i color=color=0xFF000000:size=(width)x(height):r=(framerate) -filter_complex "[0:v]alphaextract,lutyuv=y='if(gt(val,0),255,val)'[m]; [1:v][m]alphamerge[coloralpha]" -map [coloralpha] output

[poisondeathray]

I get a completely red frame on the prores4444 footage:

8bit values go from 0 to 255
proresHQ is 10bit . So the values go from 0 to 1023
prores4444 is 12bit . So the values go from 0 to 4095 . The lutyuv expression should be 4095 instead of 255 (for 8bit)

[iisperi]

I get a completely red frame on the prores4444 footage:

8bit values go from 0 to 255
proresHQ is 10bit . So the values go from 0 to 1023
prores4444 is 12bit . So the values go from 0 to 4095 . The lutyuv expression should be 4095 instead of 255 (for 8bit)

I did try to do that:

ffmpeg -i "Comp 1_1.mov" -f lavfi -i color=color=0xFF000000:size=1920x1080:r=25 -t 1 -filter_complex "[0:v]alphaextract,lutyuv=y='if(gt(val,0),4095,val)'[m]; [1:v][m]alphamerge[coloralpha]" -map '[coloralpha]' "Comp 1_1_out.mov"

However, still the same result

[poisondeathray]

What is your full commandline and desired output format ? Did you encode the export with alpha channel ?

e.g. PNG sequence works ok , -frames:v 1 to test 1 frame

ffmpeg -i "Comp 1_1.mov" -f lavfi -i color=color=0xFF000000:size=1920x1080:r=25 -filter_complex "[0:v]alphaextract,lutyuv=y='if(gt(val,0),4095,val)'[m]; [1:v][m]alphamerge[coloralpha]" -map [coloralpha] -frames:v 1 -start_number 0 "test%03.png"

[poisondeathray]

If you specify no export options , the default encoder is libx264 in 8bit 4:2:0 mode, which has no alpha channel

If you wanted an 8bit RGBA file in MOV, you could use PNG in MOV . 8bit is more than sufficient for a binarized alpha visualization for 1 color (10bit and 12bit are overkill)

ffmpeg -i "Comp 1_1.mov" -f lavfi -i color=color=0xFF000000:size=1920x1080:r=25 -filter_complex "[0:v]alphaextract,lutyuv=y='if(gt(val,0),4095,val)'[m]; [1:v][m]alphamerge[coloralpha]" -map [coloralpha] -c:v png -t 1 test_png.mov

What is the usage scenario / what applications are you viewing it in ?

[iisperi]

I just tried with a non-PRORES video, and everything works. Thank you!
However, how do I make it work for any arbitrary video with alpha (e.g. prores or not)?

My usage scenario is to mimic the Photoshop alpha matting for GIFs, by first binarizing the alpha channel, filling it with a given color, then overlaying an original video on top, and then feeding this composite video to palettegen/paletteuse command so that there are no semi-transparent pixels.

[poisondeathray]

how do I make it work for any arbitrary video with alpha (e.g. prores or not)?

maxval should work for any input bitdepth, instead of hardcoding 255/1023/4095...

lutyuv=y='if(gt(val,0),maxval,val)

[iisperi]

how do I make it work for any arbitrary video with alpha (e.g. prores or not)?

maxval should work for any input bitdepth, instead of hardcoding 255/1023/4095...

lutyuv=y='if(gt(val,0),maxval,val)

That works, however for a stream with these parameters:

Stream #0:0[0x1](eng): Video: prores (4444) (ap4h / 0x68347061), yuva444p12le(bt709, progressive), 1920x1080, 57032 kb/s, SAR 1:1 DAR 16:9, 25 fps, 25 tbr, 25 tbn (default)

I can use alphaextract separately to simply extract alpha as a grayscale video, but the command you constructed still fills the whole frame with solid red.

[poisondeathray]

The frame is supposed to be solid red - that's the color you chose. The alpha channel when retained (you have to encode a format with alpha channel), should be 100% black in the transparent areas, 100% white in the non transparent areas . When displayed properly , the white section will be cut out of the red . It's a red frame base, with an alpha channel

If you wanted to, you can composite it - "flatten" the frame so it's RGB24 (so the cutout section is truly cutout, without alpha channel), then merge back the alpha channel .

[iisperi]

The frame is supposed to be solid red - that's the color you chose. The alpha channel when retained (you have to encode a format with alpha channel), should be 100% black in the transparent areas, 100% white in the non transparent areas . When displayed properly , the white section will be cut out of the red . It's a red frame base, with an alpha channel

If you wanted to, you can composite it - "flatten" the frame so it's RGB24 (so the cutout section is truly cutout, without alpha channel), then merge back the alpha channel .

Thank you. You're 100% the best!

[poisondeathray]

Is that what you wanted ?

Your original RGB background in Comp 1_1.mov was a solid dark grey. It's not visible when rendered properly with the alpha (those dark grey areas are "transparent") . It's only visible if you ignore the alpha and render as RGB

Is that what you want ? Keep the original BG color that is actually "transparent" and not visible when viewing RGBA ?

ie. What do you want the transparent areas to be, when not rendered as transparent (what do you want the RGB channels to show) ?

[poisondeathray]

This example overlays back onto the original video (to keep the original BG, if alpha channel is not used when rendering), and merges back the binarized alpha.

 
ffmpeg -i "Comp 1_1.mov" -f lavfi -i color=color=0xFF000000:size=1920x1080:r=25 -filter_complex "[0:v]alphaextract,lutyuv=y='if(gt(val,0),maxval,val)',split[m][m2]; [1:v][m]alphamerge[coloralpha]; [0:v][coloralpha]overlay[ov]; [ov][m2]alphamerge[out]" -map [out] -c:v png -frames:v 1 test2.mov

[iisperi]

This example overlays back onto the original video (to keep the original BG, if alpha channel is not used when rendering), and merges back the binarized alpha.

 
ffmpeg -i "Comp 1_1.mov" -f lavfi -i color=color=0xFF000000:size=1920x1080:r=25 -filter_complex "[0:v]alphaextract,lutyuv=y='if(gt(val,0),maxval,val)',split[m][m2]; [1:v][m]alphamerge[coloralpha]; [0:v][coloralpha]overlay[ov]; [ov][m2]alphamerge[out]" -map [out] -c:v png -frames:v 1 test2.mov

I have slightly updated the code by adding format=rgba, which allowed me to ingest any video with alpha, from Quick Time to Prores4444:

ffmpeg -i "Comp 1_1.mov" -f lavfi -i color=color=0xFF000000:size=640x360:r=25 -t 999 -filter_complex "[0:v]scale=w=640:h=-1,split[resized],format=rgba,alphaextract,lutyuv=y='if(gt(val,0),255,val)'[m]; [1:v][m]alphamerge[coloralpha]; [coloralpha][resized]overlay,split[v],palettegen[p]; [v][p]paletteuse" -t 1 -c:v gif "test.gif"

However this results in a bunch of warnings (repeated multiple times) both on macOS and Windows:

[swscaler @ 0x139330000] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x15ab60000] [swscaler @ 0x15abd0000] No accelerated colorspace conversion found from yuva420p to bgra.
[swscaler @ 0x15a9e0000] deprecated pixel format used, make sure you did set range correctly

On macOS I can just click "Ctrl-C" and the process will finish successfully. Windows does not allow for this and no file is created.

Can you suggest a workaround here?

Thank you!

[poisondeathray]

oops I missed your post

You can hide warnings in the console

-hide_banner -loglevel error

But is format=rgba necessary ? You can also try gbrpa - planar RGB(A)

[iisperi]

oops I missed your post

You can hide warnings in the console

-hide_banner -loglevel error

But is format=rgba necessary ? You can also try gbrpa - planar RGB(A)

You are right, I did not need the format=rgba after all. I reformatted the whole command and now it still throws warnings, but they are not blocking:

ffmpeg -i "Comp 1_1.mov" -f lavfi -i color=0xFF000000:size=640x360:r=25:d=1 -filter_complex "[0:v]scale=w=640:h=-1,split[resized],alphaextract,lutyuv=y='if(gt(val,0),255,val)'[m]; [1:v][m]alphamerge[coloralpha]; [coloralpha][resized]overlay,split[v],palettegen[p]; [v][p]paletteuse" -t 1 -c:v gif "test.gif"

Thank you!