Can you remove green background with Avisynth?

[Bencuri]

I am trying to remove the green background from this video with Vegas, but no success. See the attached mask. You can see the problem is that Vegas considers my fingers be part of the backgorund. And this is what you can see infuelncing the outcome. I can set the balance so that the background is visible and also me, but around my hand the two overlaps and the background silhouettes are visible.

Can you remove this more effectively with avisynth? I know the video is not optimal either. The problem is that in the room where the green wall is, if I turn on lights, there will be a shadow appearing on the wall. If I place led light source in front of me on the floor, with the LED panel facing upwards, the shadow goes but I only have 1 panel, and if I place more, it would cast another shadow anyway I think. So these are the conditons that you can see on the video where I can avoid shadows.

Clip attached.

[hello_hello]

I'm not great with mask creation, and I couldn't manage to create one based on color due to the amount of green in the pickup and frets, however it's easy enough to at least change most of the green to grey, or to change it to another color. Maybe that'll help if you're still needing to create a mask using Vegas.
Maybe converting the green to grey will be sufficient. I don't know exactly what you want to the final output to be.

Changing the hue of the green (no doubt not what you need, but to show you what I mean about the pickup & fret color).
Tweak(hue=40, startHue=180, endHue=210)


[Attachment 93265 - Click to enlarge]

Changing the green to greyscale.
Tweak(sat=0, startHue=180, endHue=210)


[Attachment 93266 - Click to enlarge]

[jagabo]

Go to youtube and look up how to do your lighting properly for chroma keying:

https://www.youtube.com/results?search_query=lighting+for+green+screen

You're going to need more than one light.

With AviSynth you're going to need MaskHS() and maybe a brightness mask. Then Overlay() or Merge() with the mask. But you will need to fix your lighting first.

[Bencuri]

Not wanting to diminish the importance of the lighting, but it seems to me a summary of different approaches may succed. When I was experimenting with Vegas and Avisynth, the biggest problem seemed to be isolating the guitar neck and my hands. Picsart AI seems to be able to deal with that, in return it misses parts on my T-shirt. Couldn't that be compensated by avisynth? So that I can place the output from Picsart on top, and below it I could place another video where the t-shirt is okay.

[Bencuri]

Look!!!!

[hello_hello]

So how did you do it?

[Bencuri]

So how did you do it?

The way I mentioned. I noticed when using different tools to try removing the background that Vegas correctly respects my T-shirt boundaries, and there is Picsart AI that is the only one I found that respects my hand and the guitar neck boundaries. So I placed the output from Picsart as the top layer in Vegas, and another layer below it that was affected by the Vegas plugin. The two complements each other. I know the green border around my hand could be thinner but the video as is will be useable.

I tried this code in Avisynth, but the problem is that it fails to track the borders in both cases: my T-shirt and my hands+guitar neck. So this one couldn't be used:

mask_threshold = 127


src = FFVideoSource("Greenbackground.mp4")


yuv = ConvertToYUV420(src)
v_chan = ExtractV(yuv)


mask = Expr(v_chan, "x " + String(mask_threshold) + " < 255 0 ?")

mask = mask.BilinearResize(src.width, src.height).Blur(1.5)

bg = BlankClip(src, color=$0000FF)


return Overlay(src, bg, mask=mask)

This one is close but still Vegas tracks the T-shirt boundaries better.

[jagabo]

You're getting a lot of green light reflecting off the walls to the side and behind the camera. Cover those walls with black cloth or paint to get cleaner foreground colors (ie, on the shirt, frets, hands, etc.)

[Bencuri]

You're getting a lot of green light reflecting off the walls to the side and behind the camera. Cover those walls with black cloth or paint to get cleaner foreground colors (ie, on the shirt, frets, hands, etc.)

I understand what you mean but at the moment this is the only place where I can record with a solid background, and I only have one LCD light panel. I need my audio recording equipment around me so it limits places that can be suitable. I am trying to negotiate with a friend, but until then this is what is available. I tried to play with the lamps in the room, curtains, but this is the only way how I could achieve shadows behind me disappear.

[jagabo]

Here's a crude version with VirtualDub:

LWLibavVideoSource("Green background.mp4", cache=false, prefer_hw=2) 
bg = BlankClip(width=width, height=height, color=$0000ff) # blue for background

cmask = MaskHS(StartHue=185, EndHue=200, MinSat=8, MaxSat=20).mt_inpand().mt_inpand().mt_expand().mt_expand().mt_expand().BinomialBlur(5).Spline36Resize(width, height) # a mask based on chroma
bmask = MaskBrightnessRange(60, 135) # a  mask based on brightness
gmask = Overlay(cmask, bmask, mode="multiply") # logical and of the chroma and brightness masks
gmask = gmask.Crop(600,0,-0,-0).AddBorders(600,0,0,0) # clear out the left edge to cover the pickup

Overlay(last, bg, mask=gmask)

Some more tweaking should clean it up even more. It will be hard to beat the dedicated green screen filters in other programs though.

[Bencuri]

Here's a crude version with VirtualDub:

LWLibavVideoSource("Green background.mp4", cache=false, prefer_hw=2) 
bg = BlankClip(width=width, height=height, color=$0000ff) # blue for background

cmask = MaskHS(StartHue=185, EndHue=200, MinSat=8, MaxSat=20).mt_inpand().mt_inpand().mt_expand().mt_expand().mt_expand().BinomialBlur(5).Spline36Resize(width, height) # a mask based on chroma
bmask = MaskBrightnessRange(60, 135) # a  mask based on brightness
gmask = Overlay(cmask, bmask, mode="multiply") # logical and of the chroma and brightness masks
gmask = gmask.Crop(600,0,-0,-0).AddBorders(600,0,0,0) # clear out the left edge to cover the pickup

Overlay(last, bg, mask=gmask)

Some more tweaking should clean it up even more. It will be hard to beat the dedicated green screen filters in other programs though.

Thanks! Trying to test it but got a report Binominalblur is missing. Do you have it from the VariableBlur plugin? I also get the report: no function like MaskBrightnessRange. And I don't find any related plugin to it. I paste this in Google and it finds nothing.

[_Al_]

just fyi, lates Vegas versions, I use 21, which has it, uses Z depth or ai mask creation, so both basically do what was called rotoscoping of an object (that was done tediously and manually in after effects etc.) , which even does not have to be in a green background. Borders could be adjusted etc.
Things move fast ...

[jagabo]

I have the VariableBlur 64 bit plugin. But I'm not at home right now so I won't be able to provide it until tomorrow. MaskBrightnessRange() is a function that I wrote and forgot to include. Again, I can't provide it until tomorrow.

[Bencuri]

just fyi, lates Vegas versions, I use 21, which has it, uses Z depth or ai mask creation, so both basically do what was called rotoscoping of an object (that was done tediously and manually in after effects etc.) , which even does not have to be in a green background. Borders could be adjusted etc.
Things move fast ...

Maybe I will give a try. I tried Davinci Resolve and CapCut today. Davinci doesn't support my hardware as it revealed, since the 2016 version, so it is useless to try. Capcut seemed to be effective, but didnT turn out to do better job than Picsart in the end. But almost. With Custom Removal it could track the bacground borders very correctly, except for holes between my fingers where the background shows through. You just cannot have the AI to detect them by removing those on a frame. You should go frame by frame and cut them out manually.

[Bencuri]

I have the VariableBlur 64 bit plugin. But I'm not at home right now so I won't be able to provide it until tomorrow. MaskBrightnessRange() is a function that I wrote and forgot to include. Again, I can't provide it until tomorrow.

I am waiting for them

[Bencuri]

I have done some more tests since than, I I found that Pixelcut AI does perfect job. No "holes" in the background removal process, removes the wall color from between my fingers as well. Unfortunately you need to pay for anything longer than about 15 sec. But very smart AI.

[jagabo]

I have the VariableBlur 64 bit plugin. But I'm not at home right now so I won't be able to provide it until tomorrow. MaskBrightnessRange() is a function that I wrote and forgot to include. Again, I can't provide it until tomorrow.

I am waiting for them

function MaskBrightnessRange(clip v, int min, int max) # make a mask, if between min and max return 255, else 0
{
    minmask = mt_binarize(v, min) # above min?
    maxmask = mt_binarize(v, max, upper=true) # below max?
    Overlay(minmask, maxmask, mode="multiply") # logical and
    BilinearResize(v.width, v.height) # same size as source video
}

64 bit VariableBlur for AviSynth attached.

[Bencuri]

function MaskBrightnessRange(clip v, int min, int max) # make a mask, if between min and max return 255, else 0
{
    minmask = mt_binarize(v, min) # above min?
    maxmask = mt_binarize(v, max, upper=true) # below max?
    Overlay(minmask, maxmask, mode="multiply") # logical and
    BilinearResize(v.width, v.height) # same size as source video
}

64 bit VariableBlur for AviSynth attached.

Thanks! The problem is that I still couldn't get it do it properly like Pixelcut. Not far though. Your version produces patches on the background side. The best I can achieve is this. This one has no patches on the background side, but between my fingers the green is still there sometimes, and blue appears on the guitar. However this is way much better than Chroma Key in many editors. And there is a zone on the left that is less saturated, why is it there?

LWLibavVideoSource("Greenbackground.mp4", cache=false) 

bg = BlankClip(width=width, height=height, color=$0000ff) # blue for background

cmask = MaskHS(StartHue=175, EndHue=210, MinSat=5, MaxSat=40).mt_inpand().mt_expand().mt_expand().mt_expand().BinomialBlur(5).Spline36Resize(width, height) 

bmask = MaskBrightnessRange(30, 145) 

gmask = Overlay(cmask, bmask, mode="multiply") 

gmask = gmask.Crop(250,0,-0,-0).AddBorders(250,0,0,0)

Overlay(last, bg, mask=gmask)


[Attachment 93353 - Click to enlarge]

[jagabo]

And there is a zone on the left that is less saturated, why is it there?

Sorry! I didn't notice that AddBorders() would add Y=16 (limited range YUV) by default (letting some of the blue image bleed through to the picture during Overlay()). You can fix that by adding color_yuv=$007f7f to AddBorders(). Smoothing the chroma before building the mask helps too. So my earlier script becomes:

function MaskBrightnessRange(clip v, int min, int max) # make a mask, if between min and max return 255, else 0
{
    minmask = mt_binarize(v, min) # above min?
    maxmask = mt_binarize(v, max, upper=true) # below max?
    Overlay(minmask, maxmask, mode="multiply") # logical and
    BilinearResize(v.width, v.height) # same size as source video
}

LWLibavVideoSource("Green background.mp4", cache=false, prefer_hw=2) 
bg = BlankClip(width=width, height=height, color=$0000ff) # blue for background

smoothed_chroma = KNLMeansCL(d=0, a=4, h=4, channels="UV")
cmask = MaskHS(smoothed_chroma, StartHue=185, EndHue=200, MinSat=8, MaxSat=20).mt_inpand().mt_inpand().mt_expand().mt_expand().mt_expand().BinomialBlur(5).Spline36Resize(width, height)
bmask = MaskBrightnessRange(60, 135)
gmask = Overlay(cmask, bmask, mode="multiply")
gmask = gmask.Crop(600,0,-0,-0).AddBorders(600,0,0,0,color_yuv=$007f7f)

Overlay(last, bg, mask=gmask)

I didn't do so here, but changing the mt_inpand() and mt_expand() sequence might be helpful now that the chroma is smoother.

[Bencuri]

function MaskBrightnessRange(clip v, int min, int max) # make a mask, if between min and max return 255, else 0
{
minmask = mt_binarize(v, min) # above min?
maxmask = mt_binarize(v, max, upper=true) # below max?
Overlay(minmask, maxmask, mode="multiply") # logical and
BilinearResize(v.width, v.height) # same size as source video
}

LWLibavVideoSource("Green background.mp4", cache=false, prefer_hw=2)
bg = BlankClip(width=width, height=height, color=$0000ff) # blue for background

smoothed_chroma = KNLMeansCL(d=0, a=4, h=4, channels="UV")
cmask = MaskHS(smoothed_chroma, StartHue=185, EndHue=200, MinSat=8, MaxSat=20).mt_inpand().mt_inpand().mt_expand().mt_ expand().mt_expand().BinomialBlur(5).Spline36Resiz e(width, height)
bmask = MaskBrightnessRange(60, 135)
gmask = Overlay(cmask, bmask, mode="multiply")
gmask = gmask.Crop(600,0,-0,-0).AddBorders(600,0,0,0,color_yuv=$007f7f)

Overlay(last, bg, mask=gmask)
[/CODE]

I didn't do so here, but changing the mt_inpand() and mt_expand() sequence might be helpful now that the chroma is smoother.

Thanks! Unfortunately the gaps between my fingers and the outer border of my hand are still a problem, there are still heavy green patches there. I tried to tweak it a little, but once the patches got reduced the blue appears again on the brightest part of the guitar: the frets and reflections.My arm border is also fluctuating a little bit. The level where it is this way equals to what many editing software can achieve. Pixelcut somehow is more precise. It seems to me I need to stick to that if I want the best result from this.


[Attachment 93371 - Click to enlarge]

This is the code I tried after yours:

LWLibavVideoSource("Greenbackground.mp4", cache=false)
bg = BlankClip(width=width, height=height, color=$0000ff) 
smoothed_chroma = KNLMeansCL(d=0, a=4, h=4, channels="UV")
cmask = MaskHS(smoothed_chroma, StartHue=175, EndHue=205, MinSat=6, MaxSat=25).mt_inpand().mt_expand().mt_expand().mt_expand().BinomialBlur(5).Spline36Resize(width, height)
bmask = MaskBrightnessRange(35, 135)
gmask = Overlay(cmask, bmask, mode="multiply")
gmask = gmask.Crop(600,0,-0,-0).AddBorders(600,0,0,0,color_yuv=$007f7f)
Overlay(last, bg, mask=gmask)

[jagabo]

Yes, AviSynth only provides fundamental tools. It's up to you to piece together those simple filters to perform what you want. Dedicated Chromakey filters like Pixelcut have already done that hard work for you.

[Bencuri]

Yes, AviSynth only provides fundamental tools. It's up to you to piece together those simple filters to perform what you want. Dedicated Chromakey filters like Pixelcut have already done that hard work for you.

In the future I will try to find a place where I can record with a more distinctive background. In this case it is well over my knowledge to solve it and get a high quality outcome.

Thanks for your help anyway! BY the way it provides the same result as some factory softwares or online AI, and even better than many. Incl the latest CapCut for example.