Merging two almost-identical frames in video

[Violinist1228]

Hi everyone! I am pretty new to editing, and was interested in editing one of my music video recordings. My goal is to merge two sequential excerpts together, but the end frame of the first excerpt does not match perfectly with the beginning of the second (but it's very close). Is there any good way to combine the two clips while making it as unnoticeable as possible?

[poisondeathray]

It depends on the specific characteristics

It what way doesn't it match specifically ? e.g. is lighting different, camera angle different etc.. ?

What about the audio ? What did you plan to do with the audio ? eg. Was it supposed to be an uninterrupted piece , continuous from one to the other ?

It might help to post a sample from the end of the first and the beginning of the second to describe it more clearly

[Violinist1228]

Hi poisondeathray,

I edited the audio portion in Audacity, and the audio is connected pretty well. The audio will be attached to the video after everything's done. It's just the video part that I'm having problems with.

The lighting and camera are all the same; it's just my posture and angle is a little bit different (same with pianist). I guess it's kind of impossible to be in the same posture on two different takes when playing an instrument haha.

Here are the two frames that I was talking about:

[poisondeathray]

One approach would be to synthesize "in between" frames using motion interpolation techniques. But this messes with the audio timing because you're inserting frames - you'd have to readjust the audio. Essentially you're inserting "inbetween" frames from point A to point B, as if you rotated into the new frame (and the pianist leaned back or forward depending on which shot is first) - this smooths over the motion. Nor is it "automatic" for this type of motion because there is a rotational component (motion vectors don't predict object rotations very well), there is some masking involved to make it look good

The other way is to make some creative edits to the video - but it's more difficult with timed audio like this; you're limited in what you can do in terms of styles and edits, especially if the shot was meant to be contiguous. If you think of some music videos you've seen on MTV etc.., you can place cuts , and different scenes , transitions, etc.. but that doesn't really "fit" in to this type of video unless that style was used throughout the video in the first place - what I'm trying to say is that editing style will look out of place unless other sections had that type of style used as well

[jagabo]

A crossfade between the two clips probably wouldn't look too bad. Or you could try using motion interpolation techniques. This is three interpolated frames between your two images, and then reversed:



You would really want to cut the first clip a few frames shorter, and the second clip a few frames later, then replace the missing frame with motion interpolated frames -- so the total number of frames is the same and the audio will match.

[Violinist1228]

Thanks to helpful replies! jagobo, which software/program do you use to get those interpolated frames? And which software do you suggest for the easiest insert of these frames?

[poisondeathray]

Yes, jagabo is right - it would probably work if you interpolated from not from the very ends. You would have to "interpolate over" some original frames, but - that way you can preserve the audio timing. But the larger the differences in frames, the more difficult to get better result

Your screenshots show that there are slight lighting , and frame alignment differences (camera isn't locked off) ; or it might be partially from the way you took the screenshots (they definitely weren't 1:1 - they are different sizes). Is each shot from a locked off tripod , or handheld ? It helps if you align the frames and match lighting, even if it's only over the transition period (gradual change over time won't be noticed as much by human eye)

Jagabo's gif demonstrates some typical problems with "automatic" motion interpolation - those edge morphing artifacts around the pianist's head, the violin bow, etc... There are prototypical scenarios where motion estimation fails, but in realtime video, you might not notice it too much as compared to a slowed down gif. Rotational movements and objects crossing over another are common causes of failure (the algorithm cannot distinguish between object boundaries).

You can use mvtools2 in avisynth for free to interpolate frames. But avisynth has a bit of a learning curve at first (it's all script based). The "pro" versions in some FX software like twixtor use mattes, motion tracking , and spline guides to minimize those artifacts (essentially you guide the motion estimation, "telling" the algorithm what is foreground or background or edge for a cleaner result), but it takes more user guidance, and there is quite a bit of a learning curve there as well.

[jagabo]

Yes, I used mvtools2 with InsertFramesMC(). But you would concatenate the two videos and use ReplaceFramesMC():

https://forum.videohelp.com/threads/352741-Frame-interpolation?p=2226119&viewfull=1#post2226119

By carefully choosing the starting and ending points of the interpolation you may be able to minimize the artifacts.