Auto Align/Stablize/Tranform One Video to Another

[E-Sy]

Good Day, I have a interesting problem I would like to solve with premiere pro (or vapoursynth)

My Footage:

#F1: 30fps - 1280x720 - shaking

#F2: 30fps - 1920x1080 - stable



They have the exact same content from the exactly the same spot, Only resolution, crop and shakiness is different.

How can I tranform/warp/crop/align/stablize my #F1 to match my #F2 extacly(!) ? Preferably it would align itself with #F2 as reference.

[poisondeathray]

You might be able to make use autooverlay in avisynth , but it won't perform warp transformations. Only position,scale,z-axis rotation. There is no vapoursynth version yet

https://forum.doom9.org/showthread.php?t=175247
https://github.com/introspected/AutoOverlay


Are they from same camera and lens too? Because there can be differences in optics, rolling shutter, camera settings, a bunch of other variables that will make it difficult to match . Or are these 2 different cameras, and not "exactly" the same spot

Or can you provide more background information? Why is one shaking , and one stable? Was something done to one but not the other ?

[E-Sy]

Position, scale and z-axis rotation should work too, but I think it needs also x and y rotation to match 100% (warp may not be needed)
But thank you for the tip! I will try to use avisynth with premier pro (it seems there is a way to do so) and give autooverlay a try.

I also thought maybe using 'rotoscoping' and some way to align them automaticly. I also hoped Premiere Pro would have some built-in features, as it would help with the workflow


No, there are not, but there is no complex background and the important part of the video would align perfectly. ( I tried it with one frame of each video and cropped and stretched etc it in photoshop, to see if it would match. The only problem is the shakiness between the frames. Otherwise, I could try to align it manually, what would not be perfect and efficient)
*edit I know if they not match exactly I would get ghosting in the resulting video, but if they match 99% I can hide some ghosting with other artifact removing scripts

[poisondeathray]

, but I think it needs also x and y rotation to match 100% (warp may not be needed)

If so , this is a problem. Even small angles of either image x, y rotation , or camera x,y rotation can appear to be large differences, especially at the periphery they will be magnified. You will not be able to get it to align nicely because of perspective differences

( I tried it with one frame of each video and cropped and stretched etc it in photoshop, to see if it would match. The only problem is the shakiness between the frames. Otherwise, I could try to align it manually, what would not be perfect and efficient)

You might be able to combine methods (e.g. get it close with one of the other methods, ) , and finish with photoshop auto align . The benefit of photoshop is it can handle small perspective distortions. But it's tedious to do because each frame needs it's reference

[poisondeathray]

How "stable" is F2 ? is it tripod or handheld ?

You might be able to stabilize F1 normally around a reference point (e.g. a wall or object) in after effects or similar; and then use that as the reference offset from F2

Can you upload a small sample clip from F1, F2 ?

[E-Sy]

If so , this is a problem. Even small angles of either image x, y rotation , or camera x,y rotation can appear to be large differences, especially at the periphery they will be magnified. You will not be able to get it to align nicely because of perspective differences

The background is plain, everthing that is on the same 'plane' / distance away, would match, they had a hard time to do so but it worked.
In a perfect world there would be a script that would first to try to align+rotate as much as possible and than use eg. a grid and make sure each cell has the right transformation/warp
(Just to understand what I initially searched)

You might be able to combine methods (e.g. get it close with one of the other methods, ) , and finish with photoshop auto align . The benefit of photoshop is it can handle small perspective distortions. But it's tedious to do because each frame needs it's reference

Do Photoshop can handle videos? Or should I convert to png and make a batch conversion?

How "stable" is F2 ? is it tripod or handheld ?

Camera built-in stabilizer (so handheld) without moving of the camera

You might be able to stabilize F1 normally around a reference point (e.g. a wall or object) in after effects or similar; and then use that as the reference offset from F2

I need to say premier pro is not my home editing software, but required for this. I'm not sure if I can stabilize with a refrence point (?) but with media encoder I think I can pipe the workflow if it would work with after effects

Can you upload a small sample clip from F1, F2 ?

Sadly not, it's all proprietary footage.

[poisondeathray]

The background is plain, everthing that is on the same 'plane' / distance away, would match, they had a hard time to do so but it worked.
In a perfect world there would be a script that would first to try to align+rotate as much as possible and than use eg. a grid and make sure each cell has the right transformation/warp
(Just to understand what I initially searched)

What do you mean by "plain" ? Is it a blank wall, or no objects like trees, cars, rocks ? Buildings ?

Do Photoshop can handle videos? Or should I convert to png and make a batch conversion?

PS can import some types of video directly; but that's not the problem. You need to align per frame (e.g. F1 frame 10 to F2 frame 10, F1 frame 11 to F2 frame 11, etc...). But auto align requires an image stack of a given frame - it won't do this on a "video", and you have to lock the reference frame F2. Batching is more difficult to do for what you need here, as it's not it's intended purpose. But if it's important and you really have to, PS can match small perspective distortion differences

Camera built-in stabilizer (so handheld) without moving of the camera

So F1 is a different camera. (And they are not "Exactly" in the same spot - physically impossible).

"without moving of the camera" - did you mean camera translation ? ie. Operator is not walking or in a moving car or something. Stationary. It's just the handheld movements that have been in-camera stabilized?

You might be able to stabilize F1 normally around a reference point (e.g. a wall or object) in after effects or similar; and then use that as the reference offset from F2

I need to say premier pro is not my home editing software, but required for this. I'm not sure if I can stabilize with a refrence point (?) but with media encoder I think I can pipe the workflow if it would work with after effects

Can you upload a small sample clip from F1, F2 ?

Sadly not, it's all proprietary footage.

It difficult to give more suggestions without a better description or "seeing" the videos. But I would stabilize F2 more on a static reference object (e.g. a wall or rock, whatever, in the background) . That is your reference. Then stabilize F1 on that same reference. The stabilization I'm talking about is locked off (VFX stabilize) . That eliminates the camera motion entirely (As if it's on a tripod) . You can add back some of the normal camera motion to both shots after (destabilize , as if it was "moving" handheld like F1)

[E-Sy]

What do you mean by "plain" ? Is it a blank wall, or no objects like trees, cars, rocks ? Buildings ?

To be more precise: The important part is an object in the center of the screen with a spotlight, background is a black blanket (This is the setup, it may be necessary to do this again in the future with different objects varying in sizes)

You need to align per frame (e.g. F1 frame 10 to F2 frame 10, F1 frame 11 to F2 frame 11, etc...)

This is already been done with the right camera settings and a shuffle frame script. All i-Frames match in order

But I would stabilize F2 more on a static reference object (e.g. a wall or rock, whatever, in the background) . That is your reference. Then stabilize F1 on that same reference. The stabilization I'm talking about is locked off (VFX stabilize) . That eliminates the camera motion entirely (As if it's on a tripod) . You can add back some of the normal camera motion to both shots after (destabilize , as if it was "moving" handheld like F1)

I need to see if premiere would stabilize both videos the same way, but with your suggestion to add motion back; this could actually do the trick

So F1 is a different camera. (And they are not "Exactly" in the same spot - physically impossible).
"without moving of the camera" - did you mean camera translation ? ie. Operator is not walking or in a moving car or something. Stationary. It's just the handheld movements that have been in-camera stabilized?

The visual primary plane is more or less exatcly, both cameras get a connection to a pc, where sample footage frames are blended together with 'difference blending' to view differences and get no (as less as possible) distortion on this plane. (We need do adjust both camera distances per object to record)
Yes the camera operator is not moving (as much as a human can hold still) and yes exatcly only the handheld movments got stabilized



(btw thank you for your time, you helped me already with giving me ideas to try!)

[poisondeathray]

What do you mean by "plain" ? Is it a blank wall, or no objects like trees, cars, rocks ? Buildings ?

To be more precise: The important part is an object in the center of the screen with a spotlight, background is a black blanket (This is the setup, it may be necessary to do this again in the future with different objects varying in sizes)

If you're doing this in the future, I would consider adding tracking reference markers if you can . Such as "+" strips of colored tape on the background black blanket . Even static background objects help, such as a chair, whatever . ( That is, if it's not some "performance" in front of an audience such as a stage performance, and it doesn't "distract" from what you are doing)

Because a black blanket can be difficult to track. It can be almost featureless in the shadows of the stage light. If you can track the floor / ground plane (if it's not featureless), then that can work too. You want to track stationary objects as the reference to stabilize (not the subject in the center of the screen if they are moving) . If the subject isn't moving/changing, than that can be a suitable target too

I need to see if premiere would stabilize both videos the same way, but with your suggestion to add motion back; this could actually do the trick

You can try in PP; but if it doesn't work use After Effects (e.g. point tracker) or Mocha .

Your success in how well you can stabilize (to a perfect locked off VFX shot) depends on how well you can track. If the "important part" is moving, that can interfere with tracking data. You can exclude it with mattes, for example.

[E-Sy]

If you're doing this in the future, I would consider adding tracking reference markers if you can . Such as "+" strips of colored tape on the background black blanket . Even static background objects help, such as a chair, whatever . ( That is, if it's not some "performance" in front of an audience such as a stage performance, and it doesn't "distract" from what you are doing)

We actually do have some consistent points in the setup that we could use as tracking reference, I will try do use these as reference if I know how to do so in premiere

You can try in PP; but if it doesn't work use After Effects (e.g. point tracker) or Mocha .
Your success in how well you can stabilize (to a perfect locked off VFX shot) depends on how well you can track. If the "important part" is moving, that can interfere with tracking data. You can exclude it with mattes, for example.

This might be the best solution for stabilizing. I will try it With After Effects.
Do you maybe also know wich keywords I need to google for auto position, scale and z-axis rotation with my F2 as reference? Doing this first, than stabilize with tracking should hopefully be 99% accurate (if tracking works well ofcourse, if not this might be a problem; Does after effects allow temporary blurring both videos for the stabilizing? Blurring the full HD clip a litte more should give the same 'blur-radius' as the HD one so it might work better.. But I need to test it first before I think about solutions for problems I might not have)

[poisondeathray]

I will try do use these as reference if I know how to do so in premiere

Does the foreground subject move ? Are their things like moving shadows or reflections, or maybe the back audience heads moving ? ie. is there any other motion that might interfere with motion tracking? If not, and entire field of view is "clean", you might be able to use the regular warp stabilizer in PP with the output set to "no motion" . If there are "contaminants" then the result might not be as accurate

Do you maybe also know wich keywords I need to google for auto position, scale and z-axis rotation with my F2 as reference? Doing this first, than stabilize with tracking should hopefully be 99% accurate (if tracking works well ofcourse, if not this might be a problem; Does after effects allow temporary blurring both videos for the stabilizing? Blurring the full HD clip a litte more should give the same 'blur-radius' as the HD one so it might work better.. But I need to test it first before I think about solutions for problems I might not have)

There is no (easy) way to use your F2 as a reference directly and have F1 "follow along" . (There are some complex algorithms that do millions of iterations to find best match, but it's very difficult to implement in practice)

This type of stabilization eliminates all camera motion. It's not like smoothing motion. It eliminates motion completely. It will be as if you shot it with a locked off tripod. The scale of each shot will be static if there was manual or auto zooming. If there was hand panning or tilt, it will be eliminated. You can choose a reference frame to stabilize. The delta between F1 and F2 will be static, even if it's cropped or field of view is different

So after each of F1, F2 are stabilized separately, both have their camera motions removed - all you need to do is match 1 frame (position, scale, rotation) and both videos should align and match for the entire length of video. This is easy to do manually and you can check with "difference" mode to get the most accurate match. eg. you might need + 1.236 pixels to the left etc...

Blurring almost never helps with tracking itself. It's usually very detrimental, except if you have very bad noise. Blurring usually only helps with compositing if you're trying to "degrade" the F2 to match the F1 in a composite