Deshaker wont deshake....

[fxmodels]

I tried the search to see if anyone came across this but the search is not a linked object and doesnt go to a search window.

My problem with DeshakeR is stems from a video taken with a P900 nikon superzoom cam and a plane. The plane is flying in blue sky, and leaving a contrail. The image of the aircraft is approximately 80 pixels across and is nice and zoomed but of course its bouncy even with the stabilization built into the lens and sensor on the P900.

Running Deshaker on it doesnt do a thing whereas I run Deshaker on terrestrial subjects and it works every time (Pass1, Pass2 , result etc). I think it has to do with the plentitude of blue featureless sky around the aircraft so the question is this:

Is it possible to deshake such a video with the free Deshaker? I am teaching a video class and understand there are other products such as Mercalli etc that work but I am constrained to use VirtualDub64 and deshaking aircraft video plays an important role in the class...

Any help with parameters or thoughts would be great. As I said, I cant use anything else so I am stuck with Deshaker and Virtual Dub.

Thanks,
Marc

[manono]

It might be better were you to post a small sample. No one can advise on settings without having a sample with which to work.

[JVRaines]

From the manual:

- I get zero-length motion vectors (appearing as dots) in clear areas, such as in the sky, or on a wall.
Try increasing the value for Discard motion of blocks that have maximum pixel value difference less than X.

I have no idea if this will work, but if you can get Deshaker to completely ignore the sky, maybe it will stop perceiving the plane as being in motion relative to it.

[-Habanero-]

Change "have 2nd best match > best" setting in first pass from 4 to 1. Lower the "skip frame if < X % of all blocks are ok" if necessary.

[fxmodels]

I made the changes suggested and it had no effect... As suggested earlier I will upload a sample file of an aircraft flying against blue sky to see if we can figure out the parameters that would make the deshaking work ... It is attached...
Thanks in advance!
Marc

[-Habanero-]

I see the problem now. I assumed there would be clouds in the sky to work on but this is a completely uniform sky and on top of that there's intermittent blurring. I was able to get a somewhat sensible first pass but I don't have enough memory for the second. This can't be fixed without a professional stabilizer.

[fxmodels]

Can you tell me the settings for the first pass? That could help... I may have enough memory....
Thanks,
Marc

[johnmeyer]

I've done a lot of work with Deshaker (you can Google my name and you'll find a guide I wrote, along with various scripts for automating the task). I downloaded your clip and tried to stabilize it both in Deshaker and in Mercalli. I was able to get Mercalli to actually do something, although the result was not very satisfactory.

The problem is that the airplane is wandering around the entire frame. Therefore, even if you can stabilize it, you'll either have to zoom in about 90%, leaving you with an incredibly fuzzy image, or you are going to end up with some very weird borders. When I did stabilize it in Mercalli and had it "fix" the borders, the plane actually disappeared a few times into the border fix.

If it were me, I'd do manual stabilization using keyframes. Here is the result of three minutes work. If I spent another ten minutes, I could get it stabilized better.

Stabilized Plane

However, I simply put a still image, with the airplane and contrail airbrushed out, as the background to use for fixing up the borders. There are better ways to do this, but if you watch the video and ignore the mismatch on the background, note this much more important fact: the contrail is getting truncated as the plane moves left and right. Even with a border fix algorithm which uses border information from adjacent frames, this is going to be difficult to fix.

[-Habanero-]

Can you tell me the settings for the first pass? That could help... I may have enough memory....
Thanks,
Marc

I could do up to 94 frames before my memory maxed. The results are only marginally better. I don't think deshaker can do that much for this clip period because there's so little to be worked on. I did put very minimal effort with mocha with somewhat better results.

[fxmodels]

Hi john,

I see the result. It's I teresting...Your result was the Mercalli result I assume? I realize I had a bland background but my work is aerial in nature... That clip was from a superzoom camera so it's handheld and zoomed. The camera, a Nikon P900 has a stabilized zoom out to 2000 mm but it can't handle the bounce as I move a bit.

[fxmodels]

Can you tell me the settings for the first pass? That could help... I may have enough memory....
Thanks,
Marc

I could do up to 94 frames before my memory maxed. The results are only marginally better. I don't think deshaker can do that much for this clip period because there's so little to be worked on. I did put very minimal effort with mocha with somewhat better results.

Can you tell me the settings you used for the pass so I can try it? Thanks...

[johnmeyer]

I see the result. It's I teresting...Your result was the Mercalli result I assume? I realize I had a bland background but my work is aerial in nature... That clip was from a superzoom camera so it's handheld and zoomed. The camera, a Nikon P900 has a stabilized zoom out to 2000 mm but it can't handle the bounce as I move a bit.

No, that clip is the result of hand stabilization, not Mercalli and not Deshaker.

The problem you have is that the object you want to track is so small. You might have better luck using a motion tracker. There are some free ones.

A Deshaker trick is to add a contrast filter prior to Deshaker. Set the contrast horrifically high in order to make the plane and contrail an obvious object, and make the sky almost black. You might try some other effects that add even more emphasis. Have these filters prior to Deshaker. Then, run Pass 1. Make sure to set Scale to Full and Use Pixels to All. You need all the help you can get trying to track that tiny little plane.

After running Pass 1, remove the contrast filters, and switch to Pass 2. Make sure to set the correction limits for horizontal and vertical to 99 because you need to let Deshaker move things like crazy.

Bottom line: I don't think that any motion stabilization program is going to work, which is why I suggested doing it manually, and why I uploaded the results of my manual tracking test in the previous post. The problem is that motion stabilization works by "fastening onto" the background, while ignoring foreground objects. What you want to do is precisely the opposite of that. This is why, if you want to fix this using an automatic tool rather than manually keyframing, you should look into motion tracking. It does exactly what you are trying to do.

[-Habanero-]

Can you tell me the settings for the first pass? That could help... I may have enough memory....
Thanks,
Marc

I could do up to 94 frames before my memory maxed. The results are only marginally better. I don't think deshaker can do that much for this clip period because there's so little to be worked on. I did put very minimal effort with mocha with somewhat better results.

Lower block size to 8 pixels or less
have max. pixel value diff to 10 since the video isn't noisy.
have 2nd best match > best to 1
Skip frame if to lowest 0%

But as I said, this wont improve it that much. I didn't use deshaker for the video I produced and even then it's not that good.

[poisondeathray]

For reference, this is what it looks like with a point tracker, correcting only for x,y position, without any border fill . This is what John was talking about - motion tracking. You tell it to track a spot on the plane (and it's difficult in a few spots because of blur - the spot which you told it to track, no longer exists because it changed shape in a few frames. So you might have to adjust a few frames, because motion trackers aren't "perfect", you sometimes need human guidance in a few spots on footage like this)

But as you can imagine, a more stabilized frame lends itself to more options such as border fill patching nearby stabilized offset adjacent frames (ie they "fit" better, or you have fewer positional artifacts) . But you still have other issues like lighting differences, motion blur differences, warping differences - ie. there will never be a "perfect" fix for any border fill approach, especially "automatic" ones

Or if it was "too" stable for certain goals, there are ways to "ease" the stabilization to re-introduce the original motion (ie you can modulate how much stabilization)

[johnmeyer]

For reference, this is what it looks like with a point tracker, correcting only for x,y position, without any border fill.

Wow !! What a great result. I was pretty sure that a motion tracker would work much better, but didn't realize that it would be this good.

I have motion tracking software, but haven't yet had the need to use it (it used to be an expensive program, but the company quit developing it and made it a free download). Out of curiosity, which tracker did you use?

[poisondeathray]

For reference, this is what it looks like with a point tracker, correcting only for x,y position, without any border fill.

Wow !! What a great result. I was pretty sure that a motion tracker would work much better, but didn't realize that it would be this good.

I have motion tracking software, but haven't yet had the need to use it (it used to be an expensive program, but the company quit developing it and made it a free download). Out of curiosity, which tracker did you use?

After Effects, the generic point tracker - which has been there since the beginning (maybe 15 years or so) . It's the most basic type of tracker used in visual effects, all FX / compositing programs have it e.g. hitfilm, blender, shake, nuke, etc..., because you need rock stable tracking for compositing . Blender is free, but learning curve is steep. After Effects is very common, it can be accessed freely in public libraries, schools, universities etc...



Once you have a semi decent track, you have the most control and options open -

The reason why you want to stabilize it "rock stable" (at least temporarily) is usually time/workflow related. It's easier to "fix" things on a non moving target. Patches align up automatically (or less error/ better fit, at least on a 2d scene without perspective shift, or wobble distortions), rotoscoping is much faster, less work to do

You can ease the stabilization after you've done the compositing or repairs (e.g. maybe reduce it a bit to re-introduce the natural camera motion if that was your goal, partly because it looks unnatural as it goes in/out of focus from motion blur, yet the camera "appears" stationary, not hand held), or maybe the goal is to keep it centered like that - it really depends on the situation/goals

You can "frame" the shot however you want by spatially offsetting it (e.g maybe the jet stream was an important feature, and maybe you wanted to show more of it, so move the plane more to the viewer's left)

You can time offset copies of the same stabilized layer to cover some of the "holes" . It might sound difficult but it's quite easy , you just copy a few layers and move them left and right almost randomly to plug up the holes. They align up because this is decently stable track. This is "almost" automatic filling , but it's more accurate in the sense that the data is "real" from adjacent frames. (ie. not synthesized with content aware fill methods, or blurred from averaged frames). It would be analgous to deshakers' past/future frames, but more accurate. But then you also have the problems with using real data - such as lighting/exposure changes, not enough data for some areas etc... There are some ways around that with automatic color matching filters, but anyways I've attached a spatially offset, time offset layer copy example below and you can see the "fit" looks decent but the color/lighting is off. I've purposely left some "holes", so you can see the effect of the "moving rectangles" which are the time offset copies.

For scenes like this, content aware filling / inpainting the peripheral sky usually works very well. Because it's bland and semi featureless, semi random. Synthesis fill usually doesn't work when there is complex objects or movement (e.g. people, faces are a common problem). But on something like this I typically use batch photoshop fill on the inverted alpha channel "holes", but all of them will work decently well, including avisynth ones like exinpaint, gimp resynthesizer, many more photo type apps for fill. There are other VFX oriented ones like rig removal and wire removal tools which a re a little more complex as well

[johnmeyer]

Thanks for the great explanation. Very helpful.

I was going to post the results of my efforts to do the border fix. I simply copied the original video onto another track in my NLE (Vegas) and using that, either offset by a few frames or directly, I used the black borders created by your motion tracking as a mask to let the blue sky show through from the original track. This worked well, but I then started improving it with some feathering so you don't see horizontal and vertical "borders" where the masked video shows through from below.

Because the sky is featureless and roughly the same color from one frame to the next, this worked pretty well.

If I get a chance tomorrow, I'll try to remember to upload the results.

[poisondeathray]

Yes, in the host application, when you stabilize, the alpha channel is automatically present. You can blur the alpha channel and it will feather the borders. You actually have more control in AE, if you generate an internal mask in AE, because it has more control and variable edge directed feathering (you can control the direction, amount of each line to feather)

[-Habanero-]

Johnmeyer, meet poisondeathray, the guru whose meat you'll never beat.

Seriously PDR, you did all this with a point tracker? I wouldn't think that was possible. I realize the video has no noise but how did you get past the blurring? Manual editing?

[johnmeyer]

Here's a result of taking poisondeathray's motion tracking result and then masking it with a copy of one frame of the original video with the plane and contrail painted out. I used a tight mask and feathered it a lot. If I had more time I'd adjust the mask at points where it fails.

By making the mask quite small, the contrail is always clipped. What this does is reduce the problem of what happens when the plane goes to the edge of the frame and there isn't any contrail on the frame.

It ain't perfect, but it is pretty watchable. Here's the link:

Motion Tracked, with masked border fix

[poisondeathray]

Seriously PDR, you did all this with a point tracker? I wouldn't think that was possible. I realize the video has no noise but how did you get past the blurring? Manual editing?

Yes a point tracker. I increased the search region (which slows down processing time) because there was large motion when you preview the clip , but that' s it, otherwise default settings. I mentioned that above - there were a couple frames where the tracker would slip off because of the blur (the point would change shape or characteristics) - that requires you to manually reposition the tracker on the frame and continue. It's still faster, more accurate (less jitter) than doing it manually without a tracker. Accurate hand tracking is actually very difficult and takes enormous amounts of time, because you tend to have micro x,y deviations.

I know you like using some of the fancier trackers, but often the "ideal" approach is to use the most basic one.

[racer-x]

Another way to do it all within Virtualdub is to use the very excellent "Window Cropper" plugin by Khaver. You can manually move and insert key-frames using Jump, Linear,Eased, and Smooth. Here is a quick example. You can get better results if you add more key-frames or run it through DeShaked afterwards.