Avisynth - Overlay and Layer - what is the difference?

[pippas]

If I need to apply a mask to only show certain parts of 'mixed' video clips, should I use Layer or Overlay? - They seem to provide similar functions...
Is one better than the other, or are there other factors which determine the best filter to use in specific circumstances?

[pippas]

From what I've been reading so far, it would seem that these two functions are essentially interchangeable in many cases.
Certainly applying the two simple scripts below to the same clips produces identical results.

back=AVISource("C:\Users\R\test clips\601 test\fade blursea2256.avi").converttorgb32

msk=AVISource("C:\Users\R\test clips\601 test\w&b9256.avi").converttorgb32

front=AVISource("C:\Users\R\test clips\601 test\601new hqx.avi").converttorgb32.mask(msk)

Layer(back,front)

back=AVISource("C:\Users\R\test clips\601 test\fade blursea2256.avi").converttorgb32

msk=AVISource("C:\Users\R\test clips\601 test\w&b9256.avi").converttorgb32

front=AVISource("C:\Users\R\test clips\601 test\601new hqx.avi").converttorgb32

Overlay(back,front,mask=msk)

The only difference seems to be that the 'Overlay' script will work without the conversion to RGB32 - although the resulting video clip appears to be quite seriously degraded, quality wise.

The 'Layer' version only seems to work with the RBG32 conversion, which is explained in the Avisynth notes of course.

So it would seem that as long as you create RGB32 clips to work with, either function produces the same results...... so I can use either?....

[jagabo]

Every time you convert from chroma subsampled YUV (YUY2, YV12, etc.) to RGB and back you get some blurring of the colors. Avoid it when possible.

I don't know what you mean by

The only difference seems to be that the 'Overlay' script will work without the conversion to RGB32 - although the resulting video clip appears to be quite seriously degraded, quality wise.

Maybe your mask wasn't prepared correctly.

[pippas]

Maybe your mask wasn't prepared correctly.

I have been trying different masks with different amounts of 'feathering', so my observations on quality loss were down to that -- not comparing like with like -- sorry about that.

So, to avoid unnecessary colorspace conversion loss with YUY2 clips, Overlay it is then?.....

Thanks for the tips...

[jagabo]

One thing you have to keep in mind is that the mask in YUV needs to be full range, Y=0 to Y=255, not the usual video range of 16-235. So, for example, to convert an RGB mask to YUY2 you need to specify ConvertToYUY2(matrix="pc.601");

And if you do convert to RGB and back to YUY2, use ConvertBackToYUY2().

[pippas]

The masks for my current project are certainly already 0-255 (#000000 to#FFFFFF) so there should be no problems. I'll remember to check if I need to convert an RGB mask.
Thanks again for the information....

[pippas]

Further to my posts above, i have discovered that in certain cases - where the masks involved are not 'pure' 0-255 black and whites, then using a RGB32 conversion does help to overcome one problem I've encountered.

I made a overlay mix of a 'clean' and 'blurred' version of a seascape, to create a shallow depth of field illusion (in focus in the foreground, out of focus in the background). To achieve this, I used a 'graduated' mask (sample attached). It seems to work quite well - except that the resulting video clip is not completely opaque.

The effect is that when using the new 'mixed' clip to overlay another seascape which has intrusive elements I want to cover, then using a #000000 mask does not completely mask out those features in the underlying clip.
However, if I convert the clips involved to RGB32 and use 'Layer', the situation is much improved (although probably still not perfect - difficult to tell)

So, as Jagabo quite rightly points out, not making the masks correctly with pure #000000 and #FFFFFF can cause problems...

Despite introducing some colourspace conversion loss of course, the RGB conversion helps in the case of 'bleed though', when using grey scale graduated masks to mix or 'morph' two clips.........

[poisondeathray]

I made a overlay mix of a 'clean' and 'blurred' version of a seascape, to create a shallow depth of field illusion (in focus in the foreground, out of focus in the background). To achieve this, I used a 'graduated' mask (sample attached). It seems to work quite well - except that the resulting video clip is not completely opaque.

The effect is that when using the new 'mixed' clip to overlay another seascape which has intrusive elements I want to cover, then using a #000000 mask does not completely mask out those features in the underlying clip.
However, if I convert the clips involved to RGB32 and use 'Layer', the situation is much improved (although probably still not perfect - difficult to tell)

Can you clarify what you mean by "not completely opaque" ?

There shouldn't be any problems doing it in YUV. 100% black Y=0 mask should completely show the underlying base clip, 100% white Y=255 ,should completely show those sections of the overlay clip

Or maybe post an example to clarify what you mean

[pippas]

The project in question is to modify a short section of the intro to a famous old movie - "Always" - by removing the fishermen from the sequence... it's just to create a clean source for a fun 'spoof' version of the famous clip....
To illustrate the problem I encountered, I've attached a couple of frames and short clips..

This frame from the original source shows the fishermen - note the 'bright' dangling fishing rod, the right shoulder of that same guy, plus the right trouser (sorry pants) leg of the second guy;


The 'mixed focus' sea overlay was created by applying the mask in my post #7 above to 2 clips -- one of an in focus seascape, and one of a 'box blurred' version of the same clip. It is this mask which seems to allow some transparency of the final clip?.. There is a sample image from that clip here:


When I then overlay the mixed sea clip over the original source, using a 0-255 black and white mask, the resultant clip appears to allow some 'bleed through' of the original source. All the clips were created using Grass Valley HQX as a working intermediate format.
You can still see the bright 'dangling' fishing rod and that guy's right shoulder quite clearly in the attached 'demoYUV' clip. You can also (just)see the other guys right leg as well........ I'm guessing all of the part of the image that falls within the 'grey' area of the sea mix mask will be a little bit transparent?

If I simply add the instruction '.ConverttoRGB' to all 3 lines of my overlay AVIsynth script, the second 'demoRGB' clip is the result. The 'bleed through' seems to be reduced (if not completely removed?)
(The clips are cropped samples from a 1280x720 working file size)

I can only assume that the 'mixed focus' sea clip is not completely opaque where my graduated mask was grey rather than true black or white?

Not really a serious problem for a simple 'spoof' clip of course -- and the blurred seascape certainly looks more natural where the water meets the cliffs in the modified clip - but it has been interesting to note the improvement with the simple addition of the RGB conversion instruction.

I'm still checking to see if Layer produces a different result from Overlay in this case. I don't think it does?......

[jagabo]

You can still see the bright 'dangling' fishing rod and that guy's right shoulder quite clearly in the attached 'demoYUV' clip.

Because your YUV mask was not full range (0-255). YUV video normally has Y values that range from 16-235. During playback as RGB the player expands the video back to RGB=0-255.

If you convert an RGB image to YUV with ConvertToYV12() the 0-255 RGB values will be changed to Y values from 16-235. If you use that as an alpha mas there will be no fully transparent or full opaque areas. You need to use ConvertToYV12(matrix="PC.601") with your alpha mask to get Y values from 0-255. Then you won't have this problem.

[pippas]

I think maybe I'll try and convert the working files and the 'mask' pngs using a different codec.
Grass Valley HQX uses a YUY2 colourspace, so there's always going to be some loss when trying to work within an RGB colourspace. I'll try Grass Valley lossless or Lagarith and create RGB32 original working files and masks, and see how that pans out...There's no point in worrying about correcting potential YUV range problems if I don't actually need to....

Thanks once again for the suggestions and pointers....

[jagabo]

Here's a histogram of your mask converted to YUY2 with ConvertToYUY2():



This is the standard way of converting RGB to YUV, most programs will do it like this. Here's the same mask converted to YUY2 with ConvertToYUY2(matrix="PC.601") to be used as a mask for Overlay():



Notice how the blacks now reach all the way to zero and the whites all the way to 255.

[pippas]

Thanks for illustrating that - I can see how converting the YUY2(Matrix=PC.601) extends the range to the full limits.
In my case, it's the second mask I need to extend to the full limits. The one I have illustrated here is designed to have grey areas of course. My 'Overlay' black and white mask differntiates between the upper and lower halves of the picture. That's the one I need to extend the range of...

[pippas]

I've created a new mask by inputting my full range black and white pngs into Vdub, and exporting as a Lagarith RGB file. Using the ConvertToYUY2(matrix="PC.601") instruction I can create a full range mask using my usual HQX codec which, as you have advised, solves the problem completely.

I had convinced myself that it was my graduated 'sea mix' mask that was creating the problem only in the centre of the image. Wrong.
It was - as you say - the 16-235 range of my YUY2 main mask that was the problem

So, I can now use Overlay without the extra RGB conversion....

Many thanks for your help here (and elsewhere!) Jagabo ... (and to PDR of course for suggesting I post a sample!)

[pippas]

One final query --- as my source is HD (720p), I wonder if I should be using (matrix="PC.709") rather than (matrix="PC.601"). I've tried both, and they both seem to work OK?......

[jagabo]

One final query --- as my source is HD (720p), I wonder if I should be using (matrix="PC.709") rather than (matrix="PC.601"). I've tried both, and they both seem to work OK?......

It doesn't matter for a greyscale alpha mask, only the Y channel is used and that's identical for both.

[pippas]

Thanks again...

[pippas]

The original project which was the intended target for the modified 'Always' intro sequence didn't happen, so having gone to the trouble of 'removing' the fishermen, I wondered if I could use it as part of a 'spoof' demo clip for my RECORDING MIC project which I've already mentioned in another thread?.....So I spent a whole $5 on a green screen backing cloth from Ebay and dived in.... A copy of the resulting video clip is attached....

All the video editing - chromakeying - dissolves - masks etc were done with free software (Avisynth and Virtualdub).

My thanks to those on this forum whose advice helped me to understand more about all the techniques involved - especially Jagabo, PDR and racerx - thanks guys...