Chaining calls to ColorKeyMask in Avisynth...

[pippas]

Trying to cope with a less than perfect green screen clip, I've been trying out 'chaining' several calls to ColorKeyMask as per the screen capture from my script below shows.

I appreciate this example is a bit over the top, but is this the way to approach the problem?..... Obviously as you add more and more they need to become more and more selective - and I suspect I'm already chasing rainbows (...not perhaps the best description? ) by the time you add more than 4 or 5 calls to a less than perfect 'key'...

..And is there any point in separating out the tolerances of the colours when trying to chain the calls?

[poisondeathray]

But what is the actual problem ? Can you clarify what "less than perfect" green screen clip means ? There are several common specific green screen issues that you might have to address separately

I think that is the way you 're supposed to use colorkeymask in avisynth. And if it works, great. But my guess is it's getting you less than stellar results ?

There is other free green screen software you might want to look at, including cinegobs keyer, blender, free versions of blackmagic fusion and nuke.

[jagabo]

You can always use ShowAlpha() to show the alpha channel. Interleaving the original video with the alpha channel will let you see where the transparent areas are relative to the picture. With your script names:

Interleave(clip1, ShowAlpha(transparent))

[pippas]

The actual problem is poor technique (on my part!)... I think most of the errors I'm having problems with are compression artifacts around the edges of objects. The attached jpg is a close up of a single frame from a clip I later included in this you tube clip: https://www.youtube.com/watch?v=HaYWybqTQms&feature=youtu.be

As you can see the edge of the hat is pretty messy, and I've been playing around with chaining colorkeymask calls to try and improve the key. But I think I may need to look at one of the other options you suggest -- or better still, try and reduce the edge artifacts on any original recording by improving my shooting settings (if it's possible?)

As you can see in that you tube clip, I have 'blurred' the insert slightly, to try and match the background resolution (at the original sharpness, the insert really does look even more artificial)
I'm quite happy to carry on playing about with multiple colorkeymask calls - while appreciating that I can't expect miracles form such a poor source - but if my basic notion of how those calls are scripted is fundamentally wrong, it was pointless even trying. Hence my original question...

Thanks for the suggestions guys - I shall investigate further....

[jagabo]

Yes, the sample script is the right way to use ResetMask(), ColorKeyMask(), and Layer().

[jagabo]

Probably the easiest way for you to build up your mask is to use a script like this:

function PackRGB(int r, int g, int b)
{
   return(r*256*256 + g*256 + b)
}

video = ImageSource("hat edge 1.jpg", start=0, end=23, fps=23.976, pixel_type="RGB32")
black = BlankClip(video)

trans = video.ResetMask()
trans = trans.ColorKeyMask(PackRGB(11, 167, 104), 20,20,20)
trans = trans.ColorKeyMask(PackRGB(17, 194, 124), 20,20,20)
trans = trans.ColorKeyMask(PackRGB(39, 213, 150), 20,20,20)

trans = MergeARGB(trans.Blur(1.0).Blur(1.0), trans, trans, trans)

Layer(black, trans)

Then view the result of the script with VirtualDub, use CSamp to read RGB values off VirtualDub's display, and enter those as subsequent refinements with ColorKeyMask(). Ie, this makes it easy to pick out other shades of green to remove.

<edit>
I added MergeARGB() to soften the edges of the mask. I would leave that out until you're done building your mask.
</edit>

[poisondeathray]

The actual problem is poor technique (on my part!)... I think most of the errors I'm having problems with are compression artifacts around the edges of objects. The attached jpg is a close up of a single frame from a clip I later included in this you tube clip: https://www.youtube.com/watch?v=HaYWybqTQms&feature=youtu.be

As you can see the edge of the hat is pretty messy, and I've been playing around with chaining colorkeymask calls to try and improve the key. But I think I may need to look at one of the other options you suggest -- or better still, try and reduce the edge artifacts on any original recording by improving my shooting settings (if it's possible?)

That sample was a jpg (lossy , some some compression artifacts might be from jpg compression), but I'm sure everyone gets the rough idea

Yes, part of the difficulty is compression artifacts and using a higher bitrate will help to reduce those ; but another large problem is the keying setup and lighting. Notice there is green everywhere on the foreground and especially around the edges of the foreground - i.e "green spill" - this is reflected light from the green screen . As you key out different "shades" or ranges of hues, there will be some overlapping with the foreground subject, especially at the edges where the spill is the most severe. This causes "ratty" edges in the alpha channel matte, instead of a smooth contour where the hat edge should have been. If you were to examine the post key alpha channel (one way to do this is with what jagabo suggested above) you would see noisy flickering edges where it should have been a perfectly smooth white shape. You don't have flowing fine hair blowing in the wind on this shot, so you don't need a delicatepartial transparency a soft matte.

Notice there is a green tint on the final result. The skin tones, the cap, clothing - that's from the partial spill. There are ways to control post key spill in various keying programs (including the free ones mentioned), but it's also possible to "homebrew" spill removal in avisynth using the channel mixer. I can show you how you might do that in avisynth if you don't want to use the other programs, but you usually want to fix the edges before you work on fixing the overall spill and tone

[pippas]

Thank you both for your comments and suggestions. Most helpful - as always!

I shall try out the ideas and experiment further..

PDR - I may have problems addressing the green spill problem personally (I do not have very good color vision) but I can get help from someone who has!......I should appreciate you making some suggestions on using the channel mixer in Avisynth to help with the problem, when you get a moment.

Again, thanks both for the help and ideas..

[raffriff42]

I love AviSynth and all, but Cinegobs Keyer (which I just learned about from poisondeathray in post #2) is better for this task. A quick visual demo:

A sample green screen image (source: poisondeathray again; link below)


Avisynth, seven calls to ColorKeyMask:


Using CineGob and no skill, just skimming the tutorials:


poisondeathray's result using Primatte:


Obviously, the last is superior, but CineGob gives acceptable results. Who knows what might be achieved if actual skill was applied.

[poisondeathray]

PDR - I may have problems addressing the green spill problem personally (I do not have very good color vision) but I can get help from someone who has!......I should appreciate you making some suggestions on using the channel mixer in Avisynth to help with the problem, when you get a moment.

If you can , it's better to handle things as best you can during the shooting. Manipulations in post are usually not as ideal . So if you can get a better setup, lighting etc... (there are plenty of tutorials out there), it will make the keying much easier

cinegobs has simple roto and masks - so if you need garbage masks for isolation, it's 100x easier to use than something like avisynth for those types of tasks

It's spill supression is good, even better than even some retail solutions

Earlier I mentioned avisynth can also do spill supression with channel mixer. It actually uses the same formula as cinegobs spill suppressor "channel average". It's known as "green limited by channel average of red and blue" . If you use other common methods to zap the remaining green, often you get ugly skin tones or hue shifts like purple skin and hair etc.... This method tends to preserve things better. You can also modulate the strength of the effect by using the opacity parameter in overlay (0-1, where 1 is default "100%") so it gives you a lot of control

http://avisynth.nl/index.php/ChannelMixer

This method zeros out the GG , and (mean) averages GR,GB . You use it in "darken" mode with layer or overlay

ChannelMixer(100, 0, 0,    50, 0, 50,    0, 0, 100)

A practical example

a=whateversource() #whatever source that still has spill

a
ChannelMixer(100, 0, 0,    50, 0, 50,    0, 0, 100)
b=last

overlay(a,b, mode="darken", opacity=1)

If we take rr42's 2nd picture and apply the avisynth spill suppression (just pay attention to the green and residual tones)



This is from your video in the other thread with overlay/layer . (I did a rough roto just to isolate the FG)

Notice the residual spill on the shoulders, the mic, the hat . It works against that BG composite with the plane, water, because the lighting and color of that BG is different. Ideally you'd want to match the lighting, shadows, reflections of that chosen BG scene. Keying is only part of the job, compositing is the other


Later in the video when the head is turned, the beard is green, glasses catch the green light reflections - those are tell tale signs that it's a green screenshot and it works against the illusion of the composite, that's where spill suppression really helps

[pippas]

@riffraff42 - I saw your clip in your other thread....and I have to say I'm very impressed with the result PDR got with Primatte.... although as you note, CineGob looks to be very useful too.....

@PDR - many thanks for your notes and suggestions...... Lots to look at and try out.

This was my first venture into a green screen project, and was started with a budget of almost $0

My 1st attempt at trying ensure no shadows was to record in natural light - (no sun) - in a glass roofed conservatory (not sure if you use that word for a greenhouse attached to the house on your side of the pond? ).
But as you say, without any shadow, the sunlight in the BG makes the foreground shot look artificial..... although I suspect that trying to improve and match that lighting would make the project costs start to rocket.....

I do enjoy experimenting with the various concepts, so I have plenty to try out with your and Jagabo's suggestions..

Some parts of the project were just a hard slog of course - like removing the fishermen from the original background sequence - but others look like being a bit more interesting.

Thanks again for the ideas and suggestions....

[pippas]

I've tried out some short clips with CineGobs keyer, and it certainly seems to offer a much better way of trying out various options than trying to script those same options with Avisynth ... So thanks for that suggestion, PDR...

I am having trouble trying to output video directly from it though... It keeps either crashing (exporting as uncompressed) or bringing up the error message: "Could not save the frame with the selected codec. Make sure the codec are able to handle the selected output channels" . ...That seems to apply to Lagarigh, HQX - or even x.264.

So the only way I can export from CineGobs at the moment is as a png image sequence......Not too difficult of course, but an extra process to convert to a .avi video clip.
Probably just something strange with my system ... and I'm not sure how well CineGobs works with Windows 10?....

[poisondeathray]

Not sure about cinegobs + win10 ; I don't have any computers with win10 yet

Did you try running as administrator ?

I think there is an installed version and a portable version, one might work better than the other

Being able to export a PNG image sequence ,but not being able to export an "AVI" - suggests a system communication issue with VFW codecs. But did you make sure lagarith was in RGBA mode ? (alpha selected) ? If not try UT video in RGBA . If you only selected RGB that might explain that error message with "Make sure the codec are able to handle the selected output channels"

[pippas]

Well I'm having a bit more luck now (not sure why - I've not changed anything). Using Lagarith (RGBA mode has been set all along) I can now output some of the time (it still crashes on occasion).
I'll try the installed version, see if that's more robust....thanks for the suggestions

[pippas]

I've tried out some of the options using CineGobs and I have to say it is certainly a very useful piece of software..... There are lots of parameters to try, but I found the spill suppression quite useful. ...As PDR pointed out, there was quite lot of green 'spill' on my original edit......It must have been bad because even I could see it - and I don't have very good colour vision!

I've attached a 'de-spilled' version of my clip to this post. Not perfect of course - there's only so much with you can do with a poor source - but better than before I think..
And the whole project is only a bit of fun...

My thanks to PDR for the suggestion to try out CineGobs ....

[poisondeathray]

Yes, I think it looks better

Some other suggestions -
1) The area around the top of the hat is still "noisy", the alpha channel probably wasn't clean there. You can view the alpha post key using the method jagabo suggested, or even in cinegobs if using that. I realize you're battling problematic footage but there might be some other manipulations you can do to improve that pre-key such as denoising, blurring or other manipulations . It's important to view frames in sequence (not a single frame), because that's where fluctuations are visible. Sure it's great if a single frame key looks good, but in motion there might be other problems not evident on a single frame

2) You can try some other manipulations to match the composite to make it blend in more. The FG is more neutral looking now (or at least less green) but it looks like standing in the shade - the chosen BG has sun is on the viewer's left . So some people would add some light or brightness to the FG using layers, also make it "warmer" to match the chosen BG.

[poisondeathray]

And last resort of course is rotoscoping, which cinegobs has. It's roto/masks capability is primitive compared to something like AE or mocha, but roto be a live saver. Rotoscoping is basically "moving masks" so if there was a problem area that you just couldn't fix with other methods, then you could roto. For example, the noise around the head area could be fixed that way. It's "last resort" because it can become very tedious. But on a short clip with limited movement, no complex shapes it should be pretty easy on this one. For other free alternatives - Blender has roto , as does an ancient program called claxa which also has primitive motion tracking

Maybe too much for a first adventure, but something to keep in mind maybe if you have time later or for future projects



Also how are you loading the video into cinegobs ? and what format was it ? e.g. AVI import, avisynth import, etc.. that might be partially responsible for the crashes. Although if it is stable with PNG export, probably not. Probably still the VFW interface issue

[pippas]

Thanks for the further suggestions....I shall take a look at the mask using Jagabo's script, and see what transpires from that.... lots more to still try!

I think probably the best way to try and 'match' the FG to the BG lighting might be to re-shoot outdoors in appropriate sunlight? (that can mean a bit of a wait here in the UK! )

- although I imagine that might bring a whole new set of issues?

I'm loading the video into CineGobs as an AVI input, and outputting as a PNG image sequence. Slightly more long winded that AVI output of course... but not when the AVI output crashes!....still looking into that...

As I say, more to try out... thanks again for the tips...

[poisondeathray]

Thanks for the further suggestions....I shall take a look at the mask using Jagabo's script, and see what transpires from that.... lots more to still try!

It's meant for debugging. You have to look at the alpha to understand how compositing and transparency work. Manipulating the alpha channel is the "key" (nice pun) to everything in green screening. The alpha determines what is transparent, where and how much

I think probably the best way to try and 'match' the FG to the BG lighting might be to re-shoot outdoors in appropriate sunlight? (that can mean a bit of a wait here in the UK! )

- although I imagine that might bring a whole new set of issues?

In a perfect world, sure. But that's a major reason why green screening is used. Instead of flying to mars, or travelling to some exotic location, you can "fake" it. That's what "the art compositing" is all about. It's quite a big complex topic and there are many tutorials of all levels out there (youtube, vimeo similar sites)

What I was referring to is adjusting the FG to match the chosen BG . Most often this involves a light wrap (edge pixels of the FG are derived taken from the BG) , but also global color correction of the FG layer, some other compositing tricks like adding lights, shadows . Basically you want to manipulate the FG to make it blend into the BG - as if you really shot it there

[pippas]

Playing around a bit more, and I'm beginning to understand some of the observations you've made. I think I've pretty much exhausted what I can get from my 'less than ideal' FG shot....I find that trying to improve one part of the mask can make another part worse!...

So I think I'll settle with the best I've managed to get so far (which I've posted up as a You tube clip here: https://www.youtube.com/watch?v=9VedaqTmG-g )

I think maybe I'll have a look at re-shooting the FG shot outdoors - with some 'appropriate' sunlight.

I think I may have to find another sea scape as well - the one I'm currently using has the 'calm' look I wanted, but no real sunlight on it.... The original sea scape is quite unusable of course -- too many fishermen on it! ...

As you point out, there'a lot more to compositing this kind of shot than might first appear, and I think I'm already into 'diminishing returns' territory.... but it has certainly given me an insight into how these things are done.... and just how good some folk are at it.

Thanks again for the tips and suggestions...

[poisondeathray]

Playing around a bit more, and I'm beginning to understand some of the observations you've made. I think I've pretty much exhausted what I can get from my 'less than ideal' FG shot....I find that trying to improve one part of the mask can make another part worse!...

You can use different settings for different parts of the mask and composite them together in layers . Think of it like a big overlapping jigsaw puzzle

This is why you hear people stress "evenly lit" green screen ; it reduces the amount of work you have to do when keying. With "perfect" lighting, you might only need 1 key, 1 layer.

e.g. If you have one part that is "bright green", but another is "dark green", a single key is not going to work very well. You have to divide it up and use settings appropriate for each