Stabilization Tools Pack v2.1

[johnmeyer]

... johnmeyer said over the DBSC thread that he used to import scene change markers to vegas.

I just use a "writefileif" statement.

Here's the script I start with. I had five different approaches before this one, and was working with StainlessS' approach before I came up with this. If nothing else, it sure is fast.

I right click on the the Vegas Pro ruler (above the timeline) and change it to "Absolute Frames." Then, I copy the scene frame numbers from the file this script creates, and then paste it into the Edit Details --> Markers spreadsheet in Vegas.

Loadplugin("C:\Program Files\AviSynth 2.5\plugins\MVTools\mvtools2.dll")

filename     = "e:\Scenes.txt"
global dFact = 2.0
global dMin  = 2.0
BLOCKSIZE    = 8
thSCD1       = (BLOCKSIZE*BLOCKSIZE) * 64

setMTMode(5,6)
source = AVISource("e:\fs.avi").convertTOYV12().killaudio()
setMTMode(2)

prefiltered =  source.blur(1.0)
superfilt   =  MSuper(prefiltered,pel=2)
super       =  MSuper(source,     pel=2)

back_vec = MAnalyse(superfilt,isb=true,  blksize=BLOCKSIZE,overlap=2,search=0)
forw_vec = MAnalyse(superfilt,isb=false, blksize=BLOCKSIZE,overlap=2,search=0)

backcmp  = source.MCompensate(super, back_vec, thSCD1=thSCD1, thSCD2=255)
forwcmp  = source.MCompensate(super, forw_vec, thSCD1=thSCD1, thSCD2=255)

global mcomp_clip = interleave(forwcmp, source, backcmp)

#---------------------------------
#Next six lines let you view the metrics in order to set detection metrics.

global script  = """Subtitle("\nScene = " + String( (dFact * YDifferenceToNext (mcomp_clip) + dMin ) ) + \
          "\nYDiffP = " +  String( YDifferenceFromPrevious(mcomp_clip) ), lsp=0)"""
global metrics = ScriptClip(mcomp_clip,script)
final = ScriptClip(mcomp_clip,"""YDifferenceFromPrevious(mcomp_clip) > (dFact * YDifferenceToNext(mcomp_clip) \
        + dMin ) && (current_frame %3 == 1) ? Subtitle(metrics,"New Scene",size=Height/4,align=5) : metrics""")
return selectevery(final,3,1)

#---------------------------------
#Evaluate every 3rd frame using Mod(3) = 1
#Comment out next three lines when viewing metrics with code above

#WriteFileIf(mcomp_clip, filename, "(YDifferenceFromPrevious(mcomp_clip) > \
#           (dFact * YDifferenceToNext(mcomp_clip) + dMin )) && (current_frame %3 == 1)", \
#            "current_frame/3", append = false)

[poisondeathray]

Do you mean that there is no other filter like stab()? (scene change aware automatic rock stable filter), not even near? Well, that's a pity... this justifies all the talk we are having here due to lack of solutions.

There are filters similar to stab(), but they will fail under the same conditions that predispose stab() to fail.

Understand that the biggest reason those "rock stable" ones are rock stable in the first place is the user input and guidance. There is nothing magical going on. Human eyes can identify potential problem areas that will interfere with the type of tracking being used. It's the same idea as the mask idea you played with earlier, or the foreground/background object discussion. In my long post, the singer/tripod drifting clip was only an "easy" track because I knew exactly where to track. Someone might have chosen the flat piano face, it's a flat plane right? No - because there was a strip of reflected light and reflections are always problem for almost all types of trackers. A person new to mocha might have had to attempt / trial and error a few places before getting a decent track. So the "automatic" is going to be an issue. That's what I think the main obstacle boils down to - defining inclusion/exclusion areas for normal clips +/- warping for scenarios like yours

I don't know if you can tell me but johnmeyer said over the DBSC thread that he used to import scene change markers to vegas. I ignore if the same can be done on these other stabilization solutions to automate rock stable filtering, or if in the other hand it's just a shorcut to setup manual filtering by scene. On a related note I found yesterday a useful tool by Chikuzen for avspmod to import scene change as bookmarks, it uses x264. Cool stuff.
Also found useful macros by tp7 to deal with ReplaceFramesSimple or ClipClop replacement strings, here the article, and here the updated macros.

I think addressing scene changes is lower on the priority list. If a filter/method can't even get it stable within a given scene...then it's not much use worrying about the transition and next scene.

But it's good to think ahead. There are various scene change detection methods and scripts for programs like AE , premiere etc.. . You can import markers with scripts if you want to use external detection, an there is one that even uses ffmpeg scene change detection to create an EDL

But built in scene change detection/compensation within a filter/ filter chain/ or process would be more ideal IMO

What do you need, false-positive bound trembling, jitter, (film gate) wobble, etc? Give me a day to collect them all, it can be of huge help to see how they all react to these seemingly impossible to fix artifacts.

I played with that clip in mercalli v4 with various rolling shutter options - not helpful

Actually if you can, include a couple of each. So just a zip file of various clips would be helpful.

I know for certain that I have some of your clips (and others) scattered on HDD's from various threads that I didn't delete, but not all of them are labelled it will take forever to find. I was taking a quick look and found one, I'm 99% certain it was from you - I hope you don't mind if I upload it. I sort of remember it but don't have the original thread link, but it's typical of the small motion + warping

[Dogway]

@johnmeyer: Yes, I saw some of your posts, specially one where you described various ways to detect Scene Changes, at some point I thought that although overkill a very precise way to grab only real SC would be to run all the SC workflows (maybe output as mask), then with mt_logic() leave only those that match a SC on the three or four masks. Output could be too conservative, who knows.

What variable do you use to adjust SC threshold, thSCD1 only? With the defaults I got many more matches than SCXvid on a 20000 frames test anime clip. These are my results.

SCx264 40% = 272 matches
SCx264 50% = 318 matches
SCXvid     = 351 matches
johnme     = 634 matches

Many false positives but I guess it grabs all scene changes, SCXvid works fine, but I decided to use SCx264 (50%) because it's very convenient and fast, ~400fps vs ~150fps of your script, but this isn't a surprise x264 has all types of speed related CPU instructions built in. To note SCx264 (40%) missed a few real scene changes.

I have never used Vegas, only Premiere and Nuke, but my point is, after you add these SC markers, can Vegas or stabilization filters for that matter automatically look at these markers and say "hey, there's a scene change, let's stop stabilizating/deshaking here". Or is it just a shortcut so you can easily navigate and do manual stuff?


@poisondeathray: My perception was that having such seemingly crappy plugin like Depan being capable of decent stabilization work, it wouldn't be difficult to find something alike (that is, without user intervention) that performs a little better. I can't talk for the big standalone packs, from my ignorance I'd expect they all performed similar under good tracking choices, but from my stand point if you have to start tracking scene by scene (or worse, a static then going tilt/pan) it's going to be crazy stuff not worth the effort if not paid. It's basically post-production, which coming from the CG field produces a totally different state of mind for me.
You say these other packages can import SC markers, but as I said above they are not leveraged automatically by the deshakers to limit temporal corrections?

Anyway even if they can't (and while I'm still highly interested), the mere fact of a filter performing better than Stab() is worth a look, specially if tracking is not in need (ie only needs configuring for scenes).

That clip you uploaded is great, didn't happen to think of it for uploading (thread?). Is this the one you tested in Mercalli?
I'm going to try uploading not only the conflicting parts but longer footage to check how it performs when it works and also when it doesn't.
I haven't had time to read your long post but will try later after I gather all samples.

[the_E_y_Es]

Hello, I'm new to the forum and this topic caught my eye. I'm looking into stabilization options at the moment so this is a very timely opportunity for me to get some insight into all this.

@Dogway

I see you're working with 2D animation, I'm trying to stabilize an old HDTV video of that kind. Thing is, I want to stabilize it while preserving its other qualities as much as possible. Is Avisynth the best option for this task (in your opinion)? I've heard that stab() somewhat alters the video quality. Ideally I'd like to keep the original untouched while improving stabilization. To be honest, if degradation can't be avoided I'd rather not stabilize it at all. What are your thoughts on this?

I've experimented a bit with Deshaker, and was able to get some improvements, although not 100%. As others have mentioned, Deshaker doesn't really provide "rock stable" results. But it seemed not to mess with the video quality which I thought was a good thing and my primary concern in this case.

I realize the discussion is still up in the air as to what the best option is for different situations/problems. I'm still trying to process all the information you guys have posted here. Just thought I'd join the club and hopefully get some advice for my specific purposes.

Also, I attached a small bit of the video I was messing with in Deshaker. First sample is the original, second is processed with Deshaker. The first few seconds didn't improve much, but the rest seemed more stable. I also got a much larger output file, which means I'm probably not using the codec settings correctly. In any case, I'm putting this here to illustrate what kind of material and problem I'm working with. Feedback is appreciated.

[Dogway]

Avisynth is the best option if you want to preserve the color space as NLE, etc convert to float-point RGB and back to YUV. Either way I'm unsure how you want to keep the original untouched, in any case a (lossy) encode is typically involved.

With any filtering there's always a degree of degradation, the point is whether it's worthy and in this case the nervous letters tell me it is. The only rock stable filter for avisynth is stab() (and derived). There's also stabilise() but that one never was anything more than experimental.

Deshaker is stabilization of the "global motion" type so it doesn't work (at all) for this case. I passed it through stab2() so you can check how it compares. Probably professional solutions (Mercalli, or the ones named above) can do some better, but at the cost of manual set up by scenes.

The larger size of your second clip is due to higher bitrate, unless you encoded it with some kind of rate factor it should be the same size. Normally we use lossless codecs here to test quality, for example I used FFV1 for the clip below.

110 Mb using stab2()
http://www.mediafire.com/?d88qk4me3dt0zdx

[johnmeyer]

What variable do you use to adjust SC threshold, thSCD1 only? With the defaults I got many more matches than SCXvid on a 20000 frames test anime clip.

No, thSCD1 is not being used in the way you might think, and it really doesn't directly control scene detection. Detection is instead controlled with dfact and dmin, two variable names that crept into my script from some things that StainlessS was doing.

The original idea for using MVTools2, but not using its scene detection function, came from my discussion with StainlessS in his scene detection thread over in doom9.org.

DBSC v0.051, Scene Change Detection - Work-in-Progress

The idea was to use motion estimation for two things. First, I wanted to eliminate the false detection I was getting when the camera gets panned quickly. That fast motion often got misinterpreted as a scene change. This is something motion estimation is generally pretty good at and I was able to almost completely eliminate this problem by using motion estimation.

The second thing is that if you adjust the two thSCD parameters, you can effectively turn OFF the scene detection within MVTools2. Yes, MVTools2 does have its own scene detection, and I have written scripts using the function that it exposes (MSCDetection), but I could never get that to work even as well as the first scene detection script I wrote years ago using just the simple YDifferenceFromPrevious() call. I've written scene detection using DepanEstimate, MSCDetection, various RT_Stats functions, SCSelect (part of RemoveDirt), and various combinations of these things. I'm now getting the best results, at least with my film transfers, with the code I posted. If nothing else, it is really fast. I often get 8x real time performance with it.

By turning off scene detection, MVTools2 creates a mess at scene boundaries, and when you feed that to the simple, built-in AVISynth YDifferenceFromPrevious() function, you get very clear spikes that exceed the scene change threshold.

... after you add these SC markers, can Vegas or stabilization filters for that matter automatically look at these markers and say "hey, there's a scene change, let's stop stabilizating/deshaking here". Or is it just a shortcut so you can easily navigate and do manual stuff?

It is a way to import the scene changes into Vegas which, as you may know, has its own scripting language. I can then combine the interface of an NLE with the advantages and automation of scripting.

IMHO, trying to completely automate everything is a recipe for disaster, at least for most things. You just can't anticipate all problems. For scene changes, what do you do for crossfades? Nothing I've seen (pun intended) will flag that as a scene change. Maybe it doesn't matter, because the gradual nature of the transition will not cause other functions to malfunction, but for some things, you really do need to know that a scene change happened at this point. What about strobe lights? Do you want to end up with 100 scene changes in five seconds? Of course strobe lights are rare, but what about a photographer's flash?

I can go on, but my point is that you will end up getting to the finish line (project is done) faster, and with better results, if you use scripting to automate those things that can be done with assurance and the high probability that nothing gets screwed up, but leave the tough things for manual correction. I learned a long time ago that it makes no sense to spend hours writing code when I could have just spent 10-15 minutes slogging through something manually and getting a perfect result. I'm not only done several hours earlier, but I know that the result is correct.

Of course feel free to ignore the advice from an old man.

[the_E_y_Es]

@Dogway

Thanks for the prompt answer, and for the new sample; the improvement is really disconcerting. I'm so accostumed to the original shakiness, this one looks like another video entirely.

I forgot to mention, I'm looking at the custom functions you posted in the OP. I'm not very familiar with Avisynth but if this small sample is anything to go by, it's a lot more adequate than Deshaker for what I need to do.

You mentioned Mercalli -- I thought it wasn't the proper tool for the job, but I'll look into it. I even considered Warp Stabilizer for AdobeAE. There's a lot of comments out there describing it as "magic" and whatnot. I'm thinking it's probably accessible and user-friendly, but I'm not sure it's the best option for this.

Either way I'm unsure how you want to keep the original untouched, in any case a (lossy) encode is typically involved.

With any filtering there's always a degree of degradation, the point is whether it's worthy and in this case the nervous letters tell me it is.

I see.

If a lossy encode is inevitable, is it possible to obtain a transparent encode in this case? I know this probably depends on many factors, but anyways... it is the "filmic" appearance of the movie what I'd like to keep.

Finally, there are only a few choice scenes which need stabilizing. I'm assuming that even if I don't mess at all with 80% of the movie, the whole thing will still be subject to the same lossy encode. In that case, maybe I could clip out the bits of interest, fix them, clip them back in? Would that work, in theory?

If you don't mind, could I provide you another sample from the movie -- this time a flashier bit of animation -- so i can have a better look at the effects of stab()? I don't mean to impose, though.

[poisondeathray]

@poisondeathray: My perception was that having such seemingly crappy plugin like Depan being capable of decent stabilization work, it wouldn't be difficult to find something alike (that is, without user intervention) that performs a little better. I can't talk for the big standalone packs, from my ignorance I'd expect they all performed similar under good tracking choices, but from my stand point if you have to start tracking scene by scene (or worse, a static then going tilt/pan) it's going to be crazy stuff not worth the effort if not paid. It's basically post-production, which coming from the CG field produces a totally different state of mind for me.
You say these other packages can import SC markers, but as I said above they are not leveraged automatically by the deshakers to limit temporal corrections?

Markers by themselves don't do anything; they are just "signs". You can navigate by markers for ease of editing or make comments etc...

The way I would have done it is use a script that splits at markers. So you have individual segments, then each segment has a filter instance applied eg. select all then copy /paste . Or use an XML or EDL which will basically do the same thing (specifies edit in/out points that cut into segments). Different programs have slightly different syntax and formatting - there is no universally compatible format.

Since you have premiere here is one for premiere that uses ffmpeg for the detection ( but you can still use another method for detection, but you might have to adjust the syntax or formatting slightly) and convert it to EDL using edlgenerator
http://www.danielbachler.de/Automatic-scene-detect-via-ffmpeg-and-create-edl

I agree with what John says in the post above about markers and the way you seem to be leaning - it's going to be a pain and more problematic in the end

You might save some time doing the splits and applying filter instances , but you're still going to manually double check everything, at least every "scenechange". It will only work on hard cuts that are identified correctly - and that doesn't even address the issues of the filter itself - I think you're still getting a bit ahead, assuming that whatever this filter is actually works .

Anyway even if they can't (and while I'm still highly interested), the mere fact of a filter performing better than Stab() is worth a look, specially if tracking is not in need (ie only needs configuring for scenes).

For those rock stable variety, once a scene is set up, it's pretty much automatic. It's the setup that can be time consuming part. For some clips, it's actually faster than something like deshaker (like the singer/tripod clip) - but for complex clips with lots of problem areas, interactions, it might take very long time.

Believe me, I want to get things done faster too, I'm always looking for other better or faster ways. But sometimes you need to get your hands dirty.

Where I can see some progress being made , is somehow improving stab or stab2 itself . The automatic, absolutely no intervention (for the scanning/tracking portion) filters won't be much more accurate than stab . You can make adjustments after the analysis to tweak results, but all these filters have some sort of analysis pass to do the tracking work first (like deshaker). Even with mercalli or warp stabilizer there are broad categories narrowing down the options first before the analysis. They have things like "glide camera" or "rock steady" - representing broad categories of motion. John already mentioned this - if you choose the wrong choice, the results will be poor. So already there is some user intervention to narrow down the choices even before the analysis.

That's where I think the main problem is - that analysis step - motions or objects that interfere with the chosen algorithm. On "regular" clips, stab works decently most of the time, right ? Even with the "rock stable" variety there are conditions that make each fail, where another that uses a different tracking method will work. So I think for any type of "automation" that replaces at least part of the human guidance part - there needs to be a way to distinguish between them. Maybe some fuzzy logic or ability to identify those sections where problems might occur then apply another algorithm. Or maybe have a different version modified to treat different broad motion categories. In general, when you lump everything into one, it might work ok for everything , but not exceptionally for everything. Sort of like a swiss army knife. A dedicated tool for a specific task will produce better results. That's partially the reason for requesting a collection of clips - to categorize and differentiate them - and also determine what types of characteristics are causing various filters to fail . Why is it working on clip A, but not on clip B ?

[Dogway]

@the_E_y_Es: Use NLE or standalone tools if you have the spare time and think the improvement may worth it. We are still here checking what can be done in what tools, below I just prepared a sample collection for poisondeathray to test.
The problem with Mercalli is not the algorithm (it probably has both) but the fact that it doesn't have an automatic mode so you have to do everything by hand.

For those type of filmic (grain right?) encodes you need a high bitrate, maybe along a grain preset in x264. Also some people denoise the very fine grain in an attempt to save some bitrate there, with filters like SPresso() for example. You can append several clips in a matroska container, but this practice is not well seen, it might have some incompatibilities outside a PC. I think a requisite for the clips is to share same encoding settings, if you know them.

Post a sample clip but you know you can simply download the function (original stab or stab2) and test by yourself.


@johnmeyer: It grabbed many simple character head turns as SC in my clip, maybe I need to tweak those settings. So far I didn't observe the other SC detection tools catch pan scenes... you might give x264 a go, it's really fast x4 times your mvtools2 version. Can't tell about quality since my experience with it is of only one day. I'm curious to check and compare how all these different SC tools perform against each other in reference to your described scenarios. Maybe grabbing all kind of SC problematic scenes and stitching together and compare manually what tool does what, if you have some sort of sample pool I wouldn't mind to have a stab at it. One thing is for certain, there will be false-positives, but matches like flashes, or pans can be dealt at script level. I'm still not sure how useful this might be for me on relation to stabilization filters, I guess I won't know until I try.

"trying to completely automate everything is a recipe for disaster"
Generally yes, but Stab() is pretty much automatic and I would say that far from disaster it does a rather good job, it can be improved, yes, and that's what we are talking here, but if an amateur tool can do this kind of automatic work, what couldn't do a professional one? That's the big question in my opinion. The same applies to Deshaker, I think it works fine out of the box. Now the gist is that "automatic" and "professional" sound kind of contradictory, right? So we are in that context where amateur tools are good but not good enough, and professional ones are too professional and the middle ground is vacant.

@poisondeathray: here my tests, I don't collect samples so this is what I had around, the one you uploaded is a very good example for wobble/warp artifacts, I added a few more and appended a txt to describe in order the most problematic clips. It's 700Mb, so uploaded to mega for speed, ask me if you prefer mediafire (not sure if size allows it). I'm curious on bare deshaking/dewarping performance (not avisynth involved), I know animation is prone to foreground/background cel misalignments, that's impossible to fix so that part is fine.

700Mb jitter/wobble samples:
MEGA link

[poisondeathray]

Thanks for the link dogway, mega is fine and fast.

I tried mercalli v4 with various rolling shutter settings , and warp stablizer with various settings on your first clip, and the one I uploaded above - doesn't look too promising.

I'll test out some in the collection and report back . Even if some work ok, where others fail miserably, it will provide some more information as to what is going on / how things might be tweaked

[Dogway]

Thanks a lot poisondeathray, that's some great insight. The concept of EDL is what I had in mind, makes a lot of sense and now I recall watching something of that months ago, I just learned edition a year ago so everything is a bit new to me. Actually yesterday I created a post on that regard about exporting scenes from avisynth. But obviously this approach is better. I wonder why you don't recommend it, checking by eye is still better than splitting by hand and checking by eye. And except when the filter is very sensitive (could be) it shouldn't be problematic on some action scenes, or isn't? Well, at least all scene changes (and a few false-positives) are now hard cuts, so what could go wrong, in any case I uploaded the samples so we can see.

"once a scene is set up, it's pretty much automatic. It's the setup that can be time consuming part"
Do you mean that camera work must be avoided in all cases? That is, hard cuts just before and after all turns, tilts, pans, etc. Or can these rock stable filters work around most of them?

Scene change is becoming a topic on its own so I will probably open a new thread comparing all solutions, with some prone to fail clips, speed, usage, etc. Damn, I need a break lol.

----------------------

In my opinion, stab works like say 90% perfect for regular animation. Tons of static scenes, and then some pans, some action, then again lots of static. I haven't seen huge problems with stab in relation to these not so ideal scenes, at worst it will just create some very shifted frames at times (think earthquakes, action scenes), but that is already fixed in stab2(FixFalPos=true). My only complain is that it could be a bit more 10% rock stable (maybe some denoising aid can help) and another type of false-positives (deshake when it shouldn't) at some repetitive local motion (tremble, clapping...). Warping is out this context.

Live action is totally different. If it's unprocessed footage, like handheld video, deshaker type is the way to go, then rock stable method for certain scenes, so, yes, it's a combination, although for the sake of mental sanity I did get away with only deshaker for my trip video, it works fine... For film we assume the camera men did shot everything quite stable so what could come to us as shakiness probably is film gate wave. Stab() might work there too, but to be honest I don't have enough experience to tell for sure. I'm not sure 40 years ago, but nowadays you have this type of shooting where the camera just floats, real static scenes might be scarce, so a combination approach similar to handheld video could be viable, we only might turn off deshaking when high action kicks in, I guess "limiting" is something included in most solutions, just like Deshaker does in the last settings box. BTW, now that I'm at it let me post my personal Deshaker chart with settings and notes in case someone finds it useful.



I will check tomorrow if something looks promising and maybe check the workflow with EDL on Premiere, and some stabilization plugin for it. I also use Nuke but the editing tools there are limited.

[johnmeyer]

BTW, one thing I mentioned in my Deshaker guide, but is worth mentioning in this thread. This is somewhat unique to Deshaker, but may be true of Stab pac and other algorithms.

Here it is: Deshaker does not alter the first and last frame of the clip.

So what? Well, that means that you can cut up a scene into several clips, and apply stabilization to only some of those "sub-clips," but leave the others alone.

What good is this?

Well, a lot has been made of the fact that most stabilization fails -- and fails badly -- under some circumstances. This has certainly been my experience. When these failures happen, the result is actually worse (IMHO) than the original. My solution, when stabilization fails, is to note which portions of the clip are bad, and then re-do the stabilization, but don't apply stabilization to the bad sections: I only stabilize the portions immediately before and after the "bad" section. Then, because of what I just mentioned (that the beginning and ending frames of the stabilized clips are not altered at all), the stabilized version joins up perfectly with the unstabilized version.

This obviously won't work with any stabilization tool or algorithm that crops or zooms to deal with the border issue.

This may not be useful for some of the work being discussed here, but I thought it was worth mentioning for those who are trying to do work with the tools we have at our disposal, but don't want to completely give up on problem clips.

[poisondeathray]

Thanks a lot poisondeathray, that's some great insight. The concept of EDL is what I had in mind, makes a lot of sense and now I recall watching something of that months ago, I just learned edition a year ago so everything is a bit new to me. Actually yesterday I created a post on that regard about exporting scenes from avisynth. But obviously this approach is better. I wonder why you don't recommend it, checking by eye is still better than splitting by hand and checking by eye. And except when the filter is very sensitive (could be) it shouldn't be problematic on some action scenes, or isn't? Well, at least all scene changes (and a few false-positives) are now hard cuts, so what could go wrong, in any case I uploaded the samples so we can see.

@dogway - I'm just saying the whole process isn't going to be as "automatic" as you think.

You're double checking anyway so you don't end up saving that much time by automatically splitting (doing edits/splits in a NLE is lighting fast when you use keyboard shortcuts), and sometimes it will actually takes longer to "fix" the mistakes in auto scene detection and splitting. If everything is 100% clear cut , no transition, then yes it should be slightly faster. But definitely play with it

"once a scene is set up, it's pretty much automatic. It's the setup that can be time consuming part"
Do you mean that camera work must be avoided in all cases? That is, hard cuts just before and after all turns, tilts, pans, etc. Or can these rock stable filters work around most of them?

No it's fine - the point of tracking for the stabilization scenario is to identify the camera work characteristics (or virtual camera I guess for animation). You're essentially examining the motion characteristics of the camera. The stuff in front of the camera ie characters/actors are the "foreground" objects, the stuff in the static background like walls , trees etc.. those are "background." You need to keep the idea of camera motion separate from foreground object motions vs. background object motions. 3 different things. Sometimes a foreground object motion will interfere with you examining the background - like a kid running into the camera view - because that will skew the analysis - that's why you "block" out the kid with a mask for the analysis.

If you have a rock stable track, you have every option open - can re-introduce any motion, even zooms, tilts,pans etc..ie. you can stabilize it as much or as little as you want. If you just want to limit it to reducing up/down Y-axis movement, no problem, etc... Since the tracking data has the camera work characteristics figured out, if you nullify everything, it will essentially be static, but the objects in front will still have their motions - you're just taking the camera motion out. (Tracking of course, has other applications, I'm just talking about tracking for generic stabilization)

So what I meant by that statement is it can take a long time to setup your blocking/holdout masks, and have tracking areas marked (ie. before you hit the "track" button). I only posted very simple examples earlier. And cross your fingers that that is all you'll need. The singer/tripod is really the most basic. The autozoom is the next most basic (there is only 1 garbage mask blocking the hand). Complex scenes have a lot more going on. You might have to do a few roto/garbage mattes to block out some objects. The "fastest" way is to be smart about it and pick a suitable tracking spot that is clear the whole way through the scene , that way you don't have to do the other stuff. Remember it's the objects that obscure your tracking that causes the problem in most common cases. In the other thread I talked about "disloging trackers" - in mocha, because it tracks planes, you can change the tracking area spline (ie. you can alter the shape of the area over time) , as long as it's on the same plane or coplanar

Sometimes you want to start in the middle of a scene and go fowards to the end, the go back, then backwards to the beginning. Recall what I said about "poster" frame. The rock stable trackers reference a single frame in a scene. So on some shots, you might want to pick out the least zoomed in frame to start with. Each shot is different, and you have to have an idea/preview of the scene before you setup an attack plan. The planning is the most important by far step and will save you the most time

Scene change is becoming a topic on its own so I will probably open a new thread comparing all solutions, with some prone to fail clips, speed, usage, etc. Damn, I need a break lol.

Yes it's tough

In my opinion, stab works like say 90% perfect for regular animation. Tons of static scenes, and then some pans, some action, then again lots of static. I haven't seen huge problems with stab in relation to these not so ideal scenes, at worst it will just create some very shifted frames at times (think earthquakes, action scenes), but that is already fixed in stab2(FixFalPos=true). My only complain is that it could be a bit more 10% rock stable (maybe some denoising aid can help) and another type of false-positives (deshake when it shouldn't) at some repetitive local motion (tremble, clapping...). Warping is out this context.

The warping in some of your clips is contributing to the difficulty - that also interferes with tracking because you usually track objects, or features , or patterns, or points. If the "feature" deforms and changes shape, it's no longer considered the same "feature." The problem is worse on yours, because normally "static" or "solid" or "stable" features like walls or rocks - even those deform

[poisondeathray]

Finally, there are only a few choice scenes which need stabilizing. I'm assuming that even if I don't mess at all with 80% of the movie, the whole thing will still be subject to the same lossy encode. In that case, maybe I could clip out the bits of interest, fix them, clip them back in? Would that work, in theory?

Is the sample you uploaded the original, original file and format ? Because that determines how you can cut and append clips losslessly or not, what limitations you have (e.g. cutting on keyframes) and what tools you might use.

Format : AVI
Format/Info : Audio Video Interleave
File size : 5.30 MiB
Duration : 17s 309ms
Overall bit rate : 2 568 Kbps

Video
ID : 0
Format : VC-1
Codec ID : WMV3
Codec ID/Info : Windows Media Video 9
Codec ID/Hint : WMV3
Duration : 17s 309ms
Bit rate : 2 495 Kbps
Width : 1 920 pixels
Height : 1 040 pixels
Display aspect ratio : 1.85:1
Frame rate : 23.976 fps
Bit depth : 8 bits
Compression mode : Lossy
Bits/(Pixel*Frame) : 0.052
Stream size : 5.15 MiB (97%)

[the_E_y_Es]

Is the sample you uploaded the original, original file and format ? Because that determines how you can cut and append clips losslessly or not, what limitations you have (e.g. cutting on keyframes) and what tools you might use.

Yes, it's the exact original. I guess you mean there limitations inherent to the format. You're right, I hadn't considered that. It could turn out a lot more complicated process than I thought. And it does feel a little... sloppy, I suppose. Like a bunch of pieces strewn together. Not a very elegant solution.

[poisondeathray]

Is the sample you uploaded the original, original file and format ? Because that determines how you can cut and append clips losslessly or not, what limitations you have (e.g. cutting on keyframes) and what tools you might use.

Yes, it's the exact original. I guess you mean there limitations inherent to the format. You're right, I hadn't considered that. It could turn out a lot more complicated process than I thought. And it does feel a little... sloppy, I suppose. Like a bunch of pieces strewn together. Not a very elegant solution.

Not at all - I would consider doing that too if only certain scenes needed treatment. But the format you have is non standard - there might be other issues arising

WMV3 is usually in an .ASF/WMV stream, not AVI . asfbin is what would normally be used to cut and append pieces together , but it won't work on AVI container

It should be possible by using vdub, with a wmv3 vfw decoder - e.g. you can use ffdshow with it enabled in the vfw configuration. But you must make the cuts on keyframes

But it might not work, you might have to re-encode it anyways - you'll probably have to do a bunch of trial and error to get the "stitching" approach working. Other techniques would be to re-wrap it into ASF/WMV or transport stream and try to do the appending there

And do you have audio ?

[xekon]

I got an idea for your stab2() but i am not sure if it would work.

instead of FixFalPos=true, would it be possible to have FixFalPos=true take a float value instead? so for film that only moves a VERY small amount, I could specify FixFalPos=1 or FixFalPos=1.25 etc.... this would help in the automatic detection of failed stabilization...

[Dogway]

@poisondeathray: I have done tracking before, but for a different purpose, to remove some streetlights on a night scene for a videoclip project, so I had to track some objects for the replacement patches to follow. The thing is, it's not fun... If I have to do manual tracking in my opinion it's not worth it, unless I get paid for that. I enjoy doing filtering job, masking, etc, but that other is more like monkey work.

By the way, I hadn't read your "long post" (no time) until I tested today with Mercalli, so I found out what you already did. Mercalli is not up for the job. I even tested the first small 13Mb sample I uploaded and it fails as Stab() does. Mocha instead looks superb but you employed active tracking.

Your statement of requiring a "poster" frame in these filters is basically due to lack of internal scene change detection. To deal with that and as you pointed out I created an EDL file, and imported to Premiere, but I couldn't bulk "link media" to my file, despite using CS6 (some people seem to have same issue). This is something I have to find out, at worst I can automate the successive media linking with an AHK script, I can also convert any SC stat file to EDL compliant using regex and AHK as well so I can go the SCx264 workflow (direct SC bookmarks in avspmod for visual analysis), convert to EDL, import to Premiere and there use Mocha plugin with autotrack. I think this can be a great workflow. Yes, with autotrack mocha will go crazy on some scenes, but if we don't fix borders we can use in avisynth "scene replacement" for thick black borders, as I have done in stab2(), and replace with stab2() if it works better than mocha or nothing at all. What do you think?

@johnmeyer: You mention your guide often but you haven't posted a link yet. Yes, I googled it and found several guides from you. Reading on I found many posts of you talking about a modified script of videoFred's, it took me a lot of time to find among the sea of posts, and also found many versions (updated the same day 8th July 2015). Link, link, link.

By the way I read this post you wrote from 3 years ago that goes well with the thread's topic, and felt like sharing the quote.
"if I really need to do serious stabilization, I do that in Vegas using either Deshaker or Mercalli. Therefore, my use of Depan is only to remove film gate weave and minor camera movement. I find limiting Depan to small corrections lets me keep more of the original frame, and when I really need stabilization, both Deshaker and (to a lesser extent) Mercalli provide edge restoration so that no border cropping is done."

@xekon: It doesn't work like that, 99% of corrected frames produce a border of only 1 or 2 pixels because most stabilization happens in subpixel precision [1 <> -1]. So if you have a shift of 1.25 pixels the border will be rounded to 2 pixels. 3 pixel borders or more are rare (less than 1%) and I consider them false positives so that's what FixFalPos fixes.

[poisondeathray]

Your statement of requiring a "poster" frame in these filters is basically due to lack of internal scene change detection. To deal with that and as you pointed out I created an EDL file, and imported to Premiere, but I couldn't bulk "link media" to my file, despite using CS6 (some people seem to have same issue). This is something I have to find out, at worst I can automate the successive media linking with an AHK script, I can also convert any SC stat file to EDL compliant using regex and AHK as well so I can go the SCx264 workflow (direct SC bookmarks in avspmod for visual analysis), convert to EDL, import to Premiere and there use Mocha plugin with autotrack. I think this can be a great workflow. Yes, with autotrack mocha will go crazy on some scenes, but if we don't fix borders we can use in avisynth "scene replacement" for thick black borders, as I have done in stab2(), and replace with stab2() if it works better than mocha or nothing at all. What do you think?

Sounds good, keep me updated on how it's working out

I might be misunderstanding what you are trying to say, but when I'm referring to "poster" frame, it's not due to lack scene change limitations - it's the opposite: The scene change limitations are due to the poster frame and requirement of scene setup. Simple example: hard cut, completely different scene, one daytime shot in Miami, next a night time shot in Las Vegas. You're not even tracking the same object, plane, features... etc... You have to "restart" on the nighttime shot and redefine what is being tracked. The 1st frame isn't always the one you want to reference. For example if it's scratched, or warped (like in many of your samples) you're not going to be wanting to use that for the "reference". If your goal is to preserve borders you usually want to pick the "least zoomed in" or "most centered" to start with. You want to pick the "ideal" frame in the scene to start on (depends on what your goals are for that scene). You need to preview the entire scene to generate a plan of attack (what to track, what not to track, what type of track, what characteristics to use) . That planning is the "smart" way to do it and will end up being easily 5-10x faster (probably more like 20-30x faster, not a hyperbole). If you don't set it up optimally, you're going to have lots of headache and "redos".

This might be review for you, so ignore it if you already familiar with it - If you know for certain there are only specific characteristics that you need, you can disable the other ones. For example if there is only x,y translation, there is no need to enable zoom , sheer , perspective, etc... . Or if you only need a small search region, you can define that - much faster than scanning entire frames. That will speed processing. You can do those sorts of things in deshaker too in the configuration. In mercalli, on some rolling shutter settings, the processing can become 10-20x slower! Not a hyperbole. Brute force indeed. You only turn on the things that you need

What exactly do you mean by "mocha autotrack"? AFAIK, you're still doing the tracking and setup in mocha (it's a separate program), but I don't use that PP version so something might be slightly different

For the bulk link media issue - what sorts of media you test, and how were they arranged (same folder?)






When you were poking around depan/depanestimate for the subpixel / green uv issue, did you see an easy way to redefine the search region ? The docs say "It uses some central region of every frame.. " and when you use show=true I think that's the area that it's using (the correlation surface).

[Dogway]

I googled for mocha autotrack and found a post that made me think that it could, but it turns out that probably there's no such thing. Moreover there's not even a Mocha plugin for Premiere, and it seems by reading Mocha documentation that standalone requires Quicktime to load video (really!?). So the EDL workflow cannot be applied here unless I want to use Premiere's Warp Stabilizer... (tested it, blah).

I think I'm going back to good old Depan, lol. Automatic and 90% there, include some trims and there you go.

So many hurdles that I'm not going to bother. I'm severely fascinated how things like Quicktime still plague today's software, it's so 90's crapware that I'm amazed how it has survived.

I'm going to try to keep improving stab2() where it can. I didn't get your last paragraph, I have one crazy idea... moving geometry creates great motion masks from mvtools2, those can be used to mask out action areas, not everything will be corrected like the tremble, etc, but everything else.

[poisondeathray]

@Dogway
Yes for sure that's how I'd approach it in general . You "pick your battles" . If stab or stab2() can fix adequately say 70-90%, then just consider using another method of the rest or only the really bad section. Unless it's critical shot or you have lots of free time, it's usually not worth it - especially when you have lots of quick cuts like some styles of cartoon animation

Quicktime is only for loading MOV files directly. But Quicktime...ugh don't even get me started. It's extra silly on Windows (32bit, single threaded, gamma shifts, unstable)

My question about depanestimate is - can you modify it to use another region for the analysis? When you use show=true a rectangle appears in the middle, and I'm guessing that's the area of interest (the docs say something about a "correlation surface".) You can probably do that manually by cropping or reframing a clip, but I mean within the actual code are they any options. Even like a top left vs. middle, vs. top right etc... In your stab false positive example, the "shaking" of the character fools the algorithm . If you compare only the lower portion it might not get as confused - that the line of thinking I was exploring. It's probably shot down somewhere already at doom9 but I couldn't find any posts with a quick search

I have one crazy idea... moving geometry creates great motion masks from mvtools2, those can be used to mask out action areas, not everything will be corrected like the tremble, etc, but everything else.

If you were thinking to apply the correction, not the detection, through the mask, the repair usually won't "line up" , picture will "break apart" sort of like the rock stable vs. non rock stable patch repair issue I was talking about earlier .

If you were thinking to use mask to limit the analysis /search ie. guide the motion estimation and detection (sort of like you were playing with masks earlier), then yes, I think that's a good area to explore

[the_E_y_Es]

Not at all - I would consider doing that too if only certain scenes needed treatment. But the format you have is non standard - there might be other issues arising

WMV3 is usually in an .ASF/WMV stream, not AVI . asfbin is what would normally be used to cut and append pieces together , but it won't work on AVI container

It should be possible by using vdub, with a wmv3 vfw decoder - e.g. you can use ffdshow with it enabled in the vfw configuration. But you must make the cuts on keyframes

But it might not work, you might have to re-encode it anyways - you'll probably have to do a bunch of trial and error to get the "stitching" approach working. Other techniques would be to re-wrap it into ASF/WMV or transport stream and try to do the appending there

The keyframes limitation wouldn't be much of a problem (I think). I can already cut the stream in VirtualDub, haven't tried appending/combining pieces together yet.

Already a few programs have refused to work with this file. AdobeAE for one. It will be a troublesome thing.

And do you have audio ?

Yes.

[poisondeathray]

The keyframes limitation wouldn't be much of a problem (I think). I can already cut the stream in VirtualDub, haven't tried appending/combining pieces together yet.

Already a few programs have refused to work with this file. AdobeAE for one. It will be a troublesome thing.

It only "appears" to be troublesome because it's a non standard container for the compression used.

Importing it into another problem such as AE isn't an issue. (And you can import it into AE if you re-wrap it into the proper WMV/ASF format, and probably all the other ones that support WMV3 - which is almost all windows programs)

Hypothetically, even if that didn't work (it does work - I tried it) , you can encode your "to be filtered" segments (that were previously split on GOP boundaries) to a lossless or some other compatible format. Exporting isn't a problem either. Then you have to convert to WMV3, using the same settings used in your source file if you want the highest chance of it working . The appending should work in theory if all segments were split on GOP boundaries

The appending is the part where things might not work (there might be "glitches" at the seams, it might not append, etc.... ) - In that case you can still re-encode and if you do it correctly you won't notice any loss, or even use a lossless codec (large filesizes). I would still exhaust some of the other suggestions first (different containers, etc..) before "giving up" and re-encoding the whole thing

[Dogway]

I just found this thread by chance they talk about image stabilization, etc. I don't think I can find anything useful from it but might be interesting.

Quicktime is only for loading MOV files directly.

On MOCHA I couldn't open any file regardless of container/codec, so I think it uses the Quicktime API for loading video.

[poisondeathray]

I just found this thread by chance they talk about image stabilization, etc. I don't think I can find anything useful from it but might be interesting.

Sometimes there are tidbits posted in obscure places, but not at the usual places. g-force posts at doom9 also

Quicktime is only for loading MOV files directly.

On MOCHA I couldn't open any file regardless of container/codec, so I think it uses the Quicktime API for loading video.

Typically what is used image files formats like exr, ,dpx, tiff, png. MOV was supported even way back when they were mokey before mocha became more integrated - that depended on quicktime

But you might be right - I assumed it used VFW, because it loads AVI directly such as UT video, but I forgot UT has QT components now. It doesn't load lagarith AVI which would suggest otherwise. But lagarith doesn't have a native 64bit decoder - that might be why... I'll poke around and see what I can find

[Auliya90]

Relax i just Removed my question , and i will never ask anything when no one here can help.

[poisondeathray]

RE: depan "green frame" bug that affects related function like stab()

Sorry I forgot to post this earlier - Fizick recently came back to avisynth/doom9 after a long absence and fixed it "DePan Version 1.13.1, April 6, 2016 build"

I don't think he visits here, so if you have a sample that still exhibits the bug, please report it to doom9 , while he is still around.

This is one of the few threads dealing with that bug, but he posted in this one recently so you can probably report it to this one
http://forum.doom9.org/showthread.php?t=159478

[super8boy]

Very interesting thread.
I see a lot of talk and examples about camcorder videos here, but my main interest is in film gate weave.
One of the main reasons Dogway designed these tools seems to be to automatically improve unsteady film position from film telecine/scanning.

I scan 8mm, 9.5mm and 16mm at 12/16bit uncompressed and for pro clients, avisynth isn't really so much of an avenue to stabilize/degrain/sharpen/digitally remove dirt beyond 8bit.
My preference is to overscan (as original transport on the scanner isn't the steadiest) and I usually scan cine film at 1920x1200 (to fit the stabilized version into a 1440x1080 or 1920x1080 sequence later)
While 8bit is fine for most individuals, pros do require a minimum of 10bit at the very least (if not 12 or 16). I wish I could ignore this fact, but this is the way it is.

I have been investigating the stack16 features, which may be ok hopefully for the use of VideoFred's scripts or Dogway's script tools which also seem great, but I've read from other pros on forums that the workaround method to split a 16bit tif/dpx file into a stack16 format will destroy some information on the stacked part when doing temporal filtering.
Would a stabilization feature (where the whole image is moved vertically/horizontally) also affect an 8+8bit stacked source as significantly?

Mug Funky wrote on doom9.org that "avisynth is limited to 8-bit (we know that much already!). 16 using some hacky methods, but they necessarily don't lend themselves to heavy temporal filtering without destroying the extra bits (the lower bits are separated and stacked below the upper bits, giving a frame of twice the height. but filtering doesn't respect this arithmetic at all and the halves wont merge together meaningfully)."

Results don't have to be perfect, but the transport is too unsteady as it is (straight out of the scanner) to be delivered without software stabilization.
For pro & archival clients (who require 10/12/16 bit files) I am looking to stabilize scans with these constraints:
- from 12 & 16bit .tif sequences and dpx files
- with batching & automated processing capabilities (ideally from x64 software to make use of i7 quad core and lots of ram, gpu doesn't help much there I imagine)
- outputting to dpx, .tif sequences of at least 10 bit, up to 16bit (and/or 10bit dnxHD via ffmpeg - ProRes is a no go on windows unless you have DigitalVision software such as Phoenix)

I understand I may need to do some dithering as well in some cases.

I looked at alternatives such as Davinci Revival (now discontinued by Blackmagic, with no development over the last 6 years) and a couple of other pro software like DigitalVision's powerful Phoenix and Loki (system with watch folders and automated/batched settings).
Loki is very interesting and once you have the license (couple of thousands), you pay a yearly subscription for the various processing tools (probably another couple of thousand euros a year)
One of the great things about the windows based digitalvision software is that you can actually encode to ProRes (they must be paying Apple a fair chunk of change for this officially supported feature)

To give you an idea of the amount of film to process, we would be talking a couple of hours of film to scan every day, some of it in 8bit, some of it in 12/16bit.
This is where a powerful batch stabilizing application and a heavy duty workstation would do wonders.

What do you reckon?

[shekh]

Would a stabilization feature (where the whole image is moved vertically/horizontally) also affect an 8+8bit stacked source as significantly?

The pixels will be destroyed if image is moved by fractional amount (interpolated).
But the function to move stack16 image must be very simple, maybe it does exist (I have no idea).

[johnmeyer]

What have you actually tried, and what were the results? You sound like you've done a lot of reading, but not a lot of "doing." I have both transferred and then restored hundreds of hours of 8mm, Super 8, and 16mm amateur film. I developed my own version of VideoFred's script, and have posted both that script as well as the results. For gate weave, the Depanstabilize function works pretty darned well.

Have you tried this, and can you post some of the results of your efforts?

I just did a quick restoration for a major sports film collector. Here is a brief before/after clip on one of the graphics at the end of the show. Graphics and titles do the best job showing gate weave, and are really useful for determining the quality of the gate weave removal. While this isn't as good as what Mercalli can do, it is extremely good:

1964 All-American Team Excerpt