Content skew auto correction, Avisynth or other?

[lordsmurf]

Image if you took a video of a TV screen, not dead-on and square.

Like this (simulated image):



You're unsteady, shaking, so it's never the same skew frame to frame.

This can be extracted frame by frame, run some sort of auto distortion photo software (like OCR does). Then heavy deshake to smooth it out. But can that very manual process of extract + deskew be simplified in Avisynth (or otherwise)? Essentially use a video software workflow, not a photo software workflow.

What options am I missing here, if any?

These are small clips, 5 minutes max each, so tedious work is not unreasonable work.

The monkey wrench is that sometimes the video borders blend into the non-video "background". Sort of like what you'd expect from the concert I used in the faked sample image above.

[poisondeathray]

In general you would use tracker/stabilizer that tracks perspective and applies a corrective perspective corner pin

Options depend on the actual shot characteristics.

What is in the shot, is it consistent?

Are there stable areas to track ? e.g. if it was a TV screen - usually a TV has a physical border/frame which you could track

Is the "non-video" background really "black" as in the simulated image? That would imply some (failed) transform was already applied. If that's the case, do you have the original video before the transform ?

Post a sample video

[dietboby]

Perhaps you would like to take a look at DaVinci Resolve.
Especially this function here:

https://youtu.be/Dd9UXb-YjFU?si=OtR-ErmX-O7q-N30

To achieve the desired outcome, you try to use "Perspective Positioning" instead of "Corner Positioning".

[lordsmurf]

All 4 edges of the TV are never seen. Sometimes you get 0, 1, 2, 3 screen edges.

Beyond the edge of the TV screen was crushed to 0 black.

- 480p widescreen camera was used, so VHS-ish at best, low resolution, often blurry
- all handheld shot, thus erratic movements on all three X-Y-Z axis
- the movement created lots of image frame blending
- some CMOS "jello vision"
- sometimes camera re-focus

It has the overall effect of lots of "focus breathing". The images is only stable-ish if you excise lots of bad frames manually, as well as run a deshake to realign. But the perspectives frame-to-frame are still wrong after realign.

If it could have some automatic perspective adjustment, with guestimation (and ability to easily manual readjust), then it may have a shot in the video domain.

I'm not clear on how DaVinci can accomplish this. The guide show adding a video, and warping it to points, to a background video. I have the backgrounud video now, nothing is being added. I guess that I'd have to manually set point, frame by frame, and ... something something, hit the go button?

I have After Effects CS4 still. I'm sure there's a way to manually select point to warp perspective, but we arrive back at nothing having 4 point (TV screen corners).

So the only path I can see is to extract to 1000s if still images, manually correct each in a layered PSD, and attempting to manually transform. I'm just hoping that another way exists.

Thoughts?