Creating a talking mouth to match with voice over

[xiaok23]

I need some advice or tutorial example to help me edit this video I have. This is a product video with a host talking about the product, the host missed an important word on part of his script so I need to make his mouth move in the video to fix the issue. This video was shot a several times, so I do have audio clip if the missing word as well as additional footage of videos that the host did read the script correctly, it's just that the best shot I want to use has that missing word mistake. I have been on the web searching for ways to fix this and I see several mouth moving tutorials that's based on still image so they won't work... The current best bet I have in mind is to mask the part of good clip, feather it, and see if that blends in or not, but this video was shot in 1080p, so it seems like it's going to be a little bit hard. I have not tried the method I had in mind because I want to see if anybody's got any better advice?

[smrpix]

Is there something you can cut away to, or does the speaker have to appear on-camera during this line?

If it's the latter, unless you want to get into some extremely heavy-duty CGI you've identified the two methods for dealing with this:

1. Use judicious masking and trimming to make a seamless edit, programs such as After Effects can help with this.
2. Use an alternate take.

edit: Or option 3, handle the overall video stylistically in such a way that jump cuts are expected and acceptable.

[budwzr]

I would try to mask just the lips and mouth, not facial skin. Secondary Color Corrector. You might need two or three in a chain.

1080 should be easier than 720. More pixels there to select from. But you might want to upsample to 2K+ first. Do it the same way you would if it were done in PShop. Do a magic wand selection with the right tolerance. SecColCor is a lot like that.

You can adjust bright/contrast and hue to help create the mask, doesn't affect it, it's a B/W mask. There's some other tools like "Median" and "Channel Blend" that can pull like colors together. Sort of like Dodge/Burn.

Make a garbage mask, then zoom in as far as you can. Let us know how it comes out.

You don't need to feather this type of mask. It will comp as an Over. You're gonna extract by color, not outline type selection mask. The NLE will make the layers when you chain the color selector tool.

To simplify, either choose individual hues per iteration of the SCC on the chain, or widen the selected hue in a single SCC. Depending on what you've got, it can be relatively easy, or extremely tedious. That's why I say try contrast/bright/dodge/burn first.

If you can create a garbage mask that leaves only mouth and skin that would work. You could create a luminance mask instead. Easier. Just bump up the contrast until the skin tone is equal and subtract the mouth.

Any way you do it, you're gonna need to do some child/parenting and couple different composite modes. Even the crude feathered outline mask requires three vertical tracks.

Give me a still of the video and I can show you easier than tell you. Something like this:



Note: This is a crude single pass just to convey the concept. I can't spend all day on this.

[xiaok23]

To smrpix: I really don't want to use a cut away to fix the issue, that would be my last choice if all others failed.

To budwzr: After going over your solution, it seems like the best and most detailed solution I got on the net through out a few other forums. I've looked up all the tools and modifiers that you've mentioned in your post and I'm assuming all this must be achieved by using After Effects. This is going to take me sometime to work at because I have never used After Effects yet, so far I've been using Sony Vegas for all my video editing... however, I understand MOST of what you meant in your post since I'm a daily Photoshop user. I didn't quite understand what you meant by "unsample to 2K+ though".


Thank you for your response, I'm going to give this a try when I get back to my office tomorrow, but like I said, I've never used After Effects so this might be a bit challenging for me. I'll probably PM you for some advice once I start trying this in After Effects...


Appreciated for all your help!

[budwzr]

I'm using Vegas Pro 12.

By "upsample to 2K", I mean set the project to 2X horizontal. Smaller pixels. more ppi.

Something like this can take hours, or even days. You have to do your outline mask either way, so rough that out first. That way you have something to default to if you get pressed for time.

Something like this really gets down into the microscopic level, hahaha. NLE's totally relate to raster editors, but the terminology is different.

I never used AE, but some AE users here. Give a holler when you need help.

Vegas Pro is very capable, but you have to "infer" what it's doing. Like the background layering. There's no layer list with little eyes for off/on. The check marks do it. And the layers are cumulative and recursive. The order matters.

Track compositing is totally dependent on the individual clip fX too. Changes to clip fX change the composite at that lower clip level. Slight changes at the clip level can significantly change the whole composite in real time. Vegas is VERY robust in certain ways.

What's my friggin point? When pulling your mask, tweak the basic tools first, like contrast/bright/saturation/hue and watch for the preview to go in the right direction. That's AFTER you set up your tracks. Let the preview guide you. Don't keep adding more and more filters unless it's going the right way.

Whew! That's why I only do one show a night. - Beetlegeuse