Algorithm to fix unperfect video camera motion?

[gil900]

Hello,
This is my situation:

I work at a professional recording studio as camera man.
In the recording studio we producers "high end" videos materials for students. I record a person who writes things on the board(Board on the wall. For students .. in case the english word I chose is wrong word).

Sometimes, during the recording process it is necessary to move the camera Right or left (ONLY Right or left), because the frame is not big enough. this is in order to capture the extra information on the board ("information" = what the teacher continues to write or something written in the first place).

In this case, I have to move the camera (with my own hands) in perfect motion. by saying "perfect motion" I mean to slow motion (left or right) without any minor stops. the problem is that it is very hard to meet this requirement and unfortunately sometimes there are minor stops in the motion and we don't know how to fix this in software. The only option we know is to not do this at the first place.

I truly believe that there must be something that can deal with this. I know about the tool avisynth and I know that it is very powerful tool. Is there any algorithm for avisynth that can deal with this (remove the minor stops in the motion). It does not have to be performed with avisynth (I just think that this tool may be the only answer because I know from my little experience that it have a lot of exotic video algorithms. But I may be wrong and maybe there is something better for my needs).

This is the workflow I need:
1) Select time range(s) of where there is the moving motion (again, only left or right)
2) Execute the algorithm on this time range (so the algorithm will fix any minor stops in the motion)
3) Save the new video.

More info:
* There is no move up or down in the motion (mechanical mechanism prevents it)
* The conditions of studio and the lighting are very high. The quality of the image is HD. So the videos are High-quality material. so it should be more easy to work with..


I hope you understand my situation and my need.

Thank you!

[poisondeathray]

If I understand your description correctly, one approach is to interpolate over the low/no motion frames (essentially they would be considered duplicates or near duplicates, and when viewed in motion would look like a "choppy pan") , by using mvtools2 based on information of surrounding good frames (RX() or ReplaceFramesMC() ) . It's not automatic, although there are algorithms that can autodetect single duplicates (filldrops) and interpolate over them. You have to specify the affected framerange that you want to replace to be accurate. In commercial editors like premiere it's known as a "morph cut" . You're basically replacing bad frames with motion interpolated frames

e.g

A B C D D E F G

Here the 2nd "D" is a stutter, low motion, can be though as a duplicate frame , it would result in

A B C D D* E F G

Where D* is an "inbetween" frame of D , E

[johnmeyer]

Motion stabilization may do what you want. Deshaker is the free tool; Mercalli is the paid software. Both of them are designed to remove unwanted camera movement that results from taking video with a hand-held camera. However, the algorithms they use can also take a "stop and go" pan, like you described, and make it smoother. In both programs, you can independently set the amount of correction in the horizontal, vertical, rotational, and zoom (Z-axis) directions. You will want to set the horizontal parameters high, and the others low.

BTW, to avoid this problem in the future, get a fluid head for your tripod. This is what all professional videographers use. It completely eliminates the jerkiness due to hanging up on the friction points of your tripod.

https://www.youtube.com/watch?v=oFTPxQMdmho

[manono]

If I understand your description correctly...

I interpret what he's saying the same way as you do. But there's a second problem, I think, and that is, in addition to the duplicate frames where the movement 'seizes up' briefly, you'll also get these faster 'jumps' when it picks up again. So, while Filldrops might find and fix the dupe frames, you still have the problem of the jumps. Any frame interpolation has to include both in order to smooth out the overall movement. Perhaps if a sample were provided we could have a look.

And johnmeyer's solution will certainly work. But for an AviSynth and VDub beginner, I'm not sure having to learn Deshaker at the same time might be asking too much. Mercalli Prodad can certainly do the job, but it is expensive and might not be worth it unless there's some real money being spent on the project.

[gil900]

Thank you, I'll look deep into the methods you propose when I have time

[poisondeathray]

I interpret what he's saying the same way as you do. But there's a second problem, I think, and that is, in addition to the duplicate frames where the movement 'seizes up' briefly, you'll also get these faster 'jumps' when it picks up again. So, while Filldrops might find and fix the dupe frames, you still have the problem of the jumps. Any frame interpolation has to include both in order to smooth out the overall movement. Perhaps if a sample were provided we could have a look.

Yes, you're right - you mean taking care of acceleration / decel curves

But for an AviSynth and VDub beginner, I'm not sure having to learn Deshaker at the same time might be asking too much.

He's definitely not a beginner. He's used some complex scripts, autoit , some complex programming in previous posts



What is the rig that you are using ? Rails or tripod or something else? The motion characteristics are quite different. Where is it located in relation to the subject and board?

Yes, maybe a sample video would help

[manono]

Yes, you're right - you mean taking care of acceleration / decel curves

I chuckled when i read that. I don't know the lingo but that sounds about right.

He's definitely not a beginner.

Oh, okay, I didn't know. I apologize to gil900. I know plenty about interpolating frames and if the movement is slow enough and not so many frames have to be interpolated at a time, then interpolation might work, but maybe not nearly as automatically as he might wish. I've never bothered to learn how to use DeShaker properly and Stab handles most of the shaking I encounter. For the rest I let Mercalli do the heavy lifting.

[SameSelf]

Sounds like a perfect application for a 4K camera that allows you to cut/pan/zoom as needed with having to physically move the camera yet maintain HD quality.