Partial frame stab() as basis for whole frame? (Avisynth)

[lordsmurf]

stab() stabilizes. But it does so against the whole frame.

When you have on-screen dates/clocks, this is a problem.
(If you thought the unstable video was hard to watch, just wait until you see that date/clock dance around! )

So is there are way to analyze just the portion of the image containing the on-screen date/clock, and then use only that analysis data to control the movements of the frame to apply stab()?

I would assume this is possible using stab(), since stab() is just a strand/sub-script. It seems that DePan/DePanEstimate are doing the actual analysis?

I got the idea from my other thread: https://forum.videohelp.com/threads/398713-QTGMC-image-vibration-motion#post2594368
That tape has deinterlace issues, due to the date.
Another tape is unstable, so wondering if that same idea can be applied here.

I'm thinking that "wleft" and "wright" need to be specified on this stab() line:

mdata = DePanEstimate(inter,range=range,trust=0,dxmax=dxmax,dymax=dymax)

But I'm not sure that "windows it", but rather shifts the target. And thus it would still be susceptible to non-date/clock image data. (Although this would at least be better than the current center-image -1 targeting.)

Not sure if I'm even going in the right direction here. The documentation here really sucks (I know, what else is new?).
ie, "wtop" not documented. Safe to guess there is a "wbottom"?
Reading this, I have more questions than answers: http://avisynth.nl/index.php/DePanEstimate

Obi-jagabo/manono/poison/johnmeyer, are you my only hope ... again?

A sample can be uploaded, if needed. But I don't think we're there yet.

BTW, I tried to do this in After Effects already, but it's just not very accurate. It needs far too many manual tweaks, and I would wager that a box-targeted stab() would easily outperform even my manually-adjusted attempt.

[poisondeathray]

stab() stabilizes. But it does so against the whole frame.

When you have on-screen dates/clocks, this is a problem.
(If you thought the unstable video was hard to watch, just wait until you see that date/clock dance around! )

So is there are way to analyze just the portion of the image containing the on-screen date/clock, and then use only that analysis data to control the movements of the frame to apply stab()?

It's possible, but -

Normally a date/time stamp is already stable. It's overlaid in the camera after the video is taken. If you stabilize using the stamp as the analysis data... the stabilization should do nothing if working 100% perfectly.

And, if you're trying to fix problems introduced by something else - where there is local motion estimation and different blocks can be translated differently in different directions (e.g. QTGMC can do this, rolling shutter repairs can do this), that can introduce other motions in other parts of the frame, making it less stable

[Cornucopia]

You could do a fork in your script where you crop to, say 200% the size of your date/timestamp, then re-fill w black to get a same size frame, call that reference, then do the operation on the target stream using the analysis gathered from your reference stream.


Scott

[johnmeyer]

stab() stabilizes. But it does so against the whole frame.

When you have on-screen dates/clocks, this is a problem.
(If you thought the unstable video was hard to watch, just wait until you see that date/clock dance around! )

So is there are way to analyze just the portion of the image containing the on-screen date/clock, and then use only that analysis data to control the movements of the frame to apply stab()?

That is a really good idea.

I would do it for you, but I never learned how to use the AVISynth masking functions.

However, for someone who knows how to mask in AVISynth, this would not be hard:

1. Remove the date/time using Delogo. This will create a blurry set of pixels where the date/time code used to be.

2. Stabilize this Delogo'd video.

3. Overlay the date/time from the original video back over the now-stabilized video, preferably using some feathering around the edges.

You have to remove the logo temporarily because, as you already pointed out, when you stabilize the entire video, the date/time will appear to be moving around wildly in the now-stabilized video. However, if you simply mask the date/time and then stabilize, that "black hole" where the date/time should be will now wander into the frame, just like the text from the date/time would do.

It is the same problem as what happens at the edge of the frame when you apply any stabilization. Depan/Stab has an edge-correction feature which attempts to handle that, doing almost exactly what I just described. Mercallig and Deshaker also have similar facilities for correcting the edges.

[lordsmurf]

And, if you're trying to fix problems introduced by something else

Heavy tape jitter (laymen vertical jitter aka up/down motion, not technical jitter aka timing errors), mostly vertical, but not entirely. Separation of fields, followed by some sort of windowed/targeted stabilization against the date/clock, would in theory fix it. It worked in After Effects, but poorly, even with exhaustive manual assistance.

that can introduce other motions in other parts of the frame, making it less stable

There's always trade-offs with restoration. Make it better, not make it perfect. A stable date/clock is best, even if it does reveal other problems that had been hidden.

You could do a fork in your script where you crop to, say 200% the size of your date/timestamp, then re-fill w black to get a same size frame, call that reference, then do the operation on the target stream using the analysis gathered from your reference stream.

Hmm...
Any suggestions on script syntax?

the AVISynth masking functions.

Hmm...
But if you haven't figured it out, that doesn't bode well for me. I'm decent at Avisynth, but some of you have better Avisynth-fu that I currently do.

1. Remove the date/time using Delogo. This will create a blurry set of pixels where the date/time code used to be.
2. Stabilize this Delogo'd video.
3. Overlay the date/time from the original video back over the now-stabilized video, preferably using some feathering around the edges.
You have to remove the logo temporarily because, as you already pointed out, when you stabilize the entire video, the date/time will appear to be moving around wildly in the now-stabilized video. However, if you simply mask the date/time and then stabilize, that "black hole" where the date/time should be will now wander into the frame, just like the text from the date/time would do.

I don't really understand this. Wouldn't delogo need an already-stable video? Having a hard time visualizing this one.

[poisondeathray]

And, if you're trying to fix problems introduced by something else

Heavy tape jitter (laymen vertical jitter aka up/down motion, not technical jitter aka timing errors), mostly vertical, but not entirely. Separation of fields, followed by some sort of windowed/targeted stabilization against the date/clock, would in theory fix it. It worked in After Effects, but poorly, even with exhaustive manual assistance.

Does the time/date overlay move accordingly, mostly up/down ?

The problem with "only" separating fields is the even/odd field offset. 1px Up/down is normal when stepping through fields. You'd have to compensate for, or preserve that relationship, if you want to keep interlace. eg. maybe stabilize even and odd separately , but then you can introduce inconsistencies between even/odd fields. Progressive is better, but then you have to deinterlace.

[lordsmurf]

What is your concern about the offset? (Some sort of 1px image vibration, maybe? That would still be less distracting.)

As far as keeping the interlace, it doesn't matter. Very often, deinterlacing is required for restorations. QTGMC is often not allowed, and whatever less-than-ideal method gets used that has to be further compensated for. For something like this, I'd assume a pure field separation, squashing the video to 720x240. Maybe bob, maybe.

I think it's now sample time, so I'll prep that and upload it in a bit.

[poisondeathray]

What is your concern about the offset? (Some sort of 1px image vibration, maybe? That would still be less distracting.)

As far as keeping the interlace, it doesn't matter. Very often, deinterlacing is required for restorations. QTGMC is often not allowed, and whatever less-than-ideal method gets used that has to be further compensated for. For something like this, I'd assume a pure field separation, squashing the video to 720x240. Maybe bob, maybe.

I think it's now sample time, so I'll prep that and upload it in a bit.

It's only a concern if you're trying to preserve interlace - and you have to keep in mind that 1 pixel up/down is normal if you're trying to stabilize if you only use separatefields() or bob()

If you take a normal interlaced video and use separatefields() and step through it goes up/down/up/down/up/down . Because you're viewing even scanlines, odd scanlines, even ,odd, even,odd.... They are offset by 1 pixel. That' s normal. bob() does basically the same thing, but it interpolates the missing scan lines by resampling. The field offset is not compensated for. So you still get up/down/up/down . That slightly complicates things if you're trying to stabilize a vertical jitter

"Smart" deinterlacers compensate for that up/down field offset . Yadif does, QTGMC does, etc... So you get interpolated line doubling (the missing lines are filled in), and they are positioned correctly, instead of up/down/up/down.

[lordsmurf]

I'm not really focused too much on the interlace aspect. I know that non-interlace input is required, and output can be progressive. Whatever interlace method is chosen will simply have to be compensated for (AA, etc, as needed).

The bouncy-bouncy date is my main concern.

[johnmeyer]

I should have looked at your sample before I posted. I am not sure that stabilizing the video is the answer. I have an afternoon appointment I have to go to, but if I get a chance tonight I'll look at this again.

[Cornucopia]

Looks to me like the whole frame is jittering together.


Scott

[lordsmurf]

Looks to me like the whole frame is jittering together.
Scott

Yes, the entire frame is jittering, not just the date.

This is the idea to restore the jittering video:
- track motion based on the date
- apply correction, based on the target track, to the entire frame



I thought I'd been clear about this, but apparently not?

There appears to be ways to target DePanEstimate, therefore stab(), with commands mentioned earlier.
Or maybe it can be done with some sort of cropping, then somehow applying stab() to the entire frame, not just the crop box within that frame.

I know this can be done, just not sure exactly how. Again, After Effects does it, just lousy.

[poisondeathray]

Your sample has more than just x,y translation, rotation and zoom - there is warping of the date overlay (and other parts of the picture)



This is stab modifed to analyze a cropped area of interest ("cropped") , and apply that to the whole frame

The example uses yadif to get around the other issues mentioned earlier

You can probably tweak it a bit , such as the area of interest or stab parameters to get slightly better results

AVISource("date-jitter-sample2.avi")
yadif(1,1)
ConvertToYV16()
d=last

cropped = d.crop(72,344,-368,-54,true)

d
stabC(d, cropped, range=1, dxmax=8, dymax=8)




function StabC (clip clp, clip cr, int "range", int "dxmax", int "dymax") {
#cr is cropped area of interest clip
range = default(range, 1)
dxmax = default(dxmax, 4)
dymax = default(dymax, 4)

temp  = clp.TemporalSoften(7,255,255,25,2)
tempc  = cr.TemporalSoften(7,255,255,25,2)
inter = Interleave(temp.Repair(clp.TemporalSoften(1,255,255,25,2)),clp)
interc = Interleave(tempc.Repair(cr.TemporalSoften(1,255,255,25,2)),cr)
#mdata = DePanEstimate(inter,range=range,trust=0,dxmax=dxmax,dymax=dymax)
mdatac = DePanEstimate(interc,range=range,trust=0,dxmax=dxmax,dymax=dymax)


DePan(inter,data=mdatac,offset=-1)
SelectEvery(2,0) }

[lordsmurf]

<sigh> I was all excited to try the script, and then...

Yadif gave an error.

Script error: there is no function named "yadif"

So I added this:

LoadCPlugin("C:\Program Files (x86)\AviSynth\plugins\yadif.dll")

And now I get:

Not An Avisynth 2 C Plugin: C:\Program Files (x86)\AviSynth\plugins\yadif.dll

But it came from here: http://avisynth.nl/index.php/Yadif
This file: http://avisynth.org.ru/yadif/yadif17.zip

We can either:
- troubleshoot Yadif
- change to Yadifmod, which is working, but I'm not able to guess the correct syntax

[poisondeathray]

x86 ?

I think the x64 version was the one I was using from that link, with avs x64+ 3.6.2

yadifmod2, actually produces worse results on this.

You can use AssumeTFF().QTGMC too, but there is some weird unnatural motion compensation going on because of the warping in the clip

The other option is bwdif. Fast and slightly better quality than yadif, has edeint options like yadifmod/yadifmod2.
https://github.com/Asd-g/AviSynth-BWDIF

ConvertToYV16(interlaced=true)
bwdif(field=3)

[lordsmurf]

I don't know why I was thinking that stab() didn't work in x64. It does.

Tested. Works!

Some mild vibrations still exist, I shall attempt to tweak.

EDIT: I just tried this on a really bad sample. Impressive results.

[jagabo]

Yes, I was going to say that DepanEstimate() doesn't require the same clip as Depan Stabilize(). So you can feed it a smaller clip:

LWlibavVideoSource("date-jitter-sample2.avi") 
AssumeTFF()
ConvertToYV12(interlaced=true)
Yadif(mode=1)

boxx = 116
boxy = 362
boxw = 176
boxh = 40

testclip = Crop(boxx, boxy, boxw, boxh)

maxstabH=4
maxstabV=4
mdata=DePanEstimate(testclip,trust=1.0,dxmax=maxstabH,dymax=maxstabV)
DePanStabilize(last,data=mdata,dxmax=maxstabH,dymax=maxstabV,method=1,mirror=15,cutoff=1,damping=1,prev=0,next=0,blur=0)

[lordsmurf]

The example uses yadif to get around the other issues mentioned earlier

Actually, a plain default bob() may look just as good. Follow up with SelectEven(). The Yadif still had too much vibration, it must have been baked into the recording. Maybe an AA, but I just didn't see any need at this point. It looks really stable as is, and the video is now quite decent for what it is.

Yes, I was going to say that DepanEstimate() doesn't require the same clip as Depan Stabilize().

Ah! That's what I was missing.

________

For others reading this, and possible note-to-self at a later date...

- poison's script was fairly rock-steady, impressively so. Any tweaks I tried would just make stabilization worse.
- jagabo's script still had a bit too much leeway, jitters leaked through, so it needs some more tweaking.

I like that two approaches were taken here, those may be useful for different scenarios someday.

[jagabo]

- jagabo's script still had a bit too much leeway, jitters leaked through, so it needs some more tweaking.

4 lines (maxstabH=4) might not be enough. I didn't see anywhere in the sample clip where it was larger than that but your longer clip may have larger shakes. Note that pdr's script uses 8.

[oln]

Depanestimate has settings for what area of each field to use for analysis, the show parameter can be used to visualize it which is useful (when it's on it outputs a clip with the area highlighted rather than motion data so you have to view the output directly from it directly.)

[poisondeathray]

double post

[poisondeathray]

If you're not "happy", and if stabilizing the "date" is the goal, the date can be stabilized farther with AE point tracker (or free ones like natron) . Stabilizing the "date" does not necessarily give you a more stable picture overall, because of the nature of the source and the warping

You can overlay a grid to visualize how "stable" something is (or how stable certain sections are) . You can see the date deviate with the modified stab in some sections that, that AE is able to tighten up.

eg. around frame 147-148 there is up/down motion that the modified stab didn't catch, and a few other sections are a bit wobbly

[lordsmurf]

Stabilizing the "date" does not necessarily give you a more stable picture overall, because of the nature of the source and the warping

Correct, it does not.
But it does make it truer to the shooting. And having a stable date is arguably the least obnoxious viewing experience.

You can overlay a grid to visualize how "stable" something is (or how stable certain sections are) . You can see the date deviate with the modified stab in some sections that, that AE is able to tighten up.
eg. around frame 147-148 there is up/down motion that the modified stab didn't catch, and a few other sections are a bit wobbly

Interesting. Thanks. I'll try Natron.

[poisondeathray]

You already have AE; there is no reason to learn something else (unless you want to...)

Using the point tracker to stabilize (the point tracker has been in AE since the beginning, so very old versions have it too) , just center it on one of the "periods" in the date text . Use only position (don't track rotation, scale). ie. you're only correcting for x,y translation

[lordsmurf]

AE is cumbersome, and not 100% stable (what Adobe app is?). If Natron is lightweight, that'd be a win.

Also, when I last used AE, the point tracker got confused. However, I guess I need to try it with this post-Avisynth footage.

[poisondeathray]

AE is cumbersome, and not 100% stable (what Adobe app is?). If Natron is lightweight, that'd be a win.

AE is pretty stable . Natron isn't exactly lightweight, and a bit of learning curve if you're not used to nodal workflows (nodal based, not layer based)

Also, when I last used AE, the point tracker got confused. However, I guess I need to try it with this post-Avisynth footage.

You would expect it to get confused on clips like this because of noise, warping - on this one objects/letters warp .

But the "periods" in the text are fairly consistent and can function as decent tracking markers

It's really not that much different than using point tracker stabilizing directly (pre deinterlaced with yadif), vs. using the modified stab beforehand

[johnmeyer]

This is the idea to restore the jittering video:
- track motion based on the date
- apply correction, based on the target track, to the entire frame


[Attachment 54829 - Click to enlarge]

I quit tracking this thread, but came back and saw this.

What you ask for, and what you show in the diagram is exactly what Deshaker can do. In the guide I wrote for it fifteen years ago, that was one of the things I highlighted. The guide used to be all over the Internet, but it's hard to find now so I've attached both the DOC file, as well as a zip file that has the doc AND the images.

The short version is that you specify your box using the "ignore image area" settings in pass 1. You will want to ignore everything outside the date. This will force Deshaker to create stabilization vectors using only the information from the area occupied by the date. Deshaker saves this data in a LOG file after you do pass 1. The actually stabilization is done in pass 2 using the vectors in that LOG file. Before you actually run pass 2, you eliminate the numbers you put in the "ignore" boxes so that pass 2 affects the entire video.

[lordsmurf]

But isn't Deshaker really, really, really slow compared to the stab() method used here?
And doesn't it overlook the interlace aspect?
So not quite the solution needed here.

That said, this is good to know abut Deshaker, for potential future needs. I like Mercalli far more than Deshaker, but it's not possible (that I know of) to target. That Deshaker can target (something I may have known years ago, but forgot) is worth giving it another look.

So thanks.

[johnmeyer]

Deshaker is not particularly slow. It is one of the better stabilization programs out there. I own Mercalli, and I still use Deshaker for some projects. It works with interlaced: if you took the time to glance at the image files that I uploaded in my last post, in the very first set of controls, you'll see that it is set for interlaced, bottom field first. In fact, it is one of the more "interlace aware" stabilization programs out there.

Even if you don't like the workflow, I'd recommend at least trying it since, as I pointed out, it does EXACTLY what you asked for.