i have some old Super8 footage that i have had transferred to avi.
heres the thing. the film is of my very first steps. so im sure you can appreciate that its important footage and something i would like to restore. but somewhere along the line, around 15 years ago or something, the film was chewed by the projector in a screening. and my father spliced the chewed film back together. but he lost frames here and there.
so what you now see is me (age 1) trying to get from the floor to the couch, but instead of it being smooth its fairly jerky, with a few frames (18fps film) being dropped every second or so.
i would like to try and smooth this out. i was thinking of using Twixtor to maybe try and "re-create" the missing frames? but because my movements are slow and then SUDDEN (when the frame skips) i am thinking it will look smoother but still jerky? so i need to add some time ...exactly on the splice points, so that the extra frames (eg between my arm being back to being forward) is not sudden.
so heres my question, if i know (on average) that i am missing 3 frames in a spot, do i leave a gap in the Sony Vegas timeline (ie do a split in the footage) that is blank for Twixtor to fill in?or will it need just 3 frames added of the last frame before the split? or something else?
+ Reply to Thread
Results 1 to 21 of 21
Thread: Twixtor to help invent frames
1) I use the After Effects version, so I think it would be similar to the Vegas version of twixtor.
It's difficult to do what you want in vegas with variable frequencies. If you just do an average, sorry that's a half assed job and you 'll get a judder cadence, motion won't be smooth. I'm assuming your frame drops are not perfectly spaced in time, nor are they always the same number. You need to put in the correct number and placement missing frames where they were missing in the first place
But you're correct - basically you'd have to insert some missing frames, but with a variable cadence. So you would have to do several interpolations (different FPS versions) and mix & match frames on the timeline . For example if you're missing 1 inbetween frame , you would need a 2xFPS version to fill. If you're missing 2 inbetween frames, you would need a 3xFPS version etc... Moreover - It doesn't do it automatically. You need to take the higher FPS version (it has more frames, so it's no longer aligned on the timeline) and manually place it where you need to, shifting the timeline to fill in the missing frames
It's actually easier to do this sort of thing in avisynth IMO. You also need a "blank inserted" clip - ie. a clip where you have blanks inserted where frames were originally dropped to smooth out the motion where you want the algorithm to fill, but you just specify the frame ranges to interpolate over. It takes the good "end points" and interpolates the frames in between (over the blanks, replacing them)
2) Certain types of motions are difficult to interpolate using optical flow in general with any software. One prototypical "failure" is when objects cross over another (just as in walking - legs crossing over each other) . The reason is the algorithm cannot disinguish between foreground and background or between what object edge is what - so you get edge morphing artifacts . But my opinion is that even if you can get a 50-60% artifact ridden ugly job that requires some photoshopping, that's 50-60% less work that you would have had to do in photoshop or after effects alone.
3) Interpolation quality - In general all the optical flow methods produce similar results overall. In some frames, one technique yield slightly better results, but worse in other frames. They all tend to fail to some extent in those common failure scenarios (fast motions, low frame rates, object crossing over another, rotational vectors) .
The difference between the "pro" versions of these tools is they have track points and mattes that can guide the motion estimation. For example you can use motion trackers to follow the edge of objects, this helps with the problem of losing edges do a degree. Vegas doesn't have a motion tracker so I don't know if that version has track point assist. You can use foreground and background mattes to help tell the algorithm what is what for better separation and thus better results
Last edited by poisondeathray; 3rd Aug 2014 at 12:45.
If you want to learn AviSynth you can use InsertFramesMC() to insert motion interpolated frames, or ReplaceFramesMC() to overwrite a range of frames with motion interpolated frames (eg, to replace corrupted frames).
They work pretty well for panning shots or otherwise simple motions. Not so well with complex motions or very large motions.
hi poison / jagabo
thanks for your posts and taking the time to respond.
I have question, is the theory of using something like Twixtor in this case to create smoother video, that you use it to slow the video down, say 3 times slower, and then once its finished processing you then drag the file back in and speed it back up 3x to the original speed...but then with 3 times the framerate?
(im using Vegas)...is there a way to do this in one render? or is it always a double render process?
That's the general idea - you're only using twixtor to generate the inbetween frames.
I would probably do the twixtored version separately (export, then reimport a lossless version). That will make editing easier. You have 2 tracks. Bottom is the original. Top is the interpolated. Top will be a longer in the timeline because it has extra frames. You have to ripple edit so that you cover up , or insert those gaps , pushing the rest of the frames down the timeline , so that the final version will have more frames from the inserts. You're using bits from twixtored version to insert into your main video. You might need several versions, if you have different rates of frame drops (e.g. a 2x version won't be able to fill in a gap with 2 missing consecutive frames, you need a 3x version. Similarly, a 3x version can't fill a single gap perfectly, because the frames aren't spaced correctly)
I haven't done this in vegas, I use a different version. In AE, it's not necessary to render out, but it's still easier to edit that way. There can be a slight lag as twixtor is making calculations on the fly (it depends on how you have it set,some settings are slower than others), so that can slow down the actual editing. In AE, what you can do is nest or precompose timelines, so that the effects/filters become part of that video before you edit. I suppose you could nest veggies in vegas, but not sure if it will work, try it out and make sure the frames are the same as if you rendered it out. I suppose another way is keyframe the changes to twixtor on the fly at different positions (so it generates variable rates of frames), but to me that looks much more difficult to do
Last edited by poisondeathray; 10th Aug 2014 at 08:33.
youre right poison. once you get Twixtor fired up in Vegas it grinds to a halt and its really hard to do any kind of editing. it takes a few seconds to update the preview screen so a render is really the way to go.
would the "blank inserted clip" always be the last frame before the lost frames? so it would be the image of me standing as a baby taking a step... so if i have 3 lost frames, then i insert the last frame before the edit three times into the blank "gap" that is missing?
Last edited by Blackout; 11th Aug 2014 at 10:00.
Another technique is you can set it on "low" quality settings for the rough editing part, just to get the spacing and timing down so editing is more responsive, the turn up the settings on final render
What I've done in the past is generate about a dozen different versions with different settings. You will find that you can get iterations of some combinations of settings, where some settings "solve" the artifacts on some frames, but worse on others. So the solution is to use a bunch of different techniques and settings (e.g a few from avisynth, some from twixtor on different settings, and others like MSU FRC, kronos, there are a bunch of others...) , and combine them by compositing bits and pieces of each. It sounds like a lot of work but it's actually faster than doing things from scratch or photoshop
You really have to make use of the advanced parameters like using mattes and tracking points to make the most of twixtor. Just leaving it on one set of settings and hoping for the best with invariably leave you with those ugly morphing edge artifacts in many situations
The generation of frames is a separate step from spacing or "ripple editing" the inserts. Finding where the gaps are in your original, and editing so there are "blank" or placeholder frames is one of the steps. On way to do it is to split the main video on timeline and slide it over "x" frames. The blank inserts are just where the original frames would have been, had there been no frame drops. If you want red instead, just put a red solid on the track below
Twixtor (or whatever interpolation tool) will be another step. Again , you need 2x, 3x, 4x versions etc... depending on how many missing frames you have in the gaps
As a general rule, you always end and start with "good" frames as the "end points" which interpolated frames are derived from. You don't want to interpolate from damaged or artifact ridden frames, because those artifacts/damage will propogate into your "new" frames. So you're not going to interpolate from the "blank inserted" version, you're going to use the original version, probably with some cleanup beforehand (e.g. if you have noise/grain/scratches/damage, clean those up before interpolating)
Reinserting the various versions is the last step . It won't fit nicely/automatically, because each 2x, 3x, version etc... will have different number of frames... 2x, 3x more frames. And if you want to combine the masking/ multiple iterations technique you have to use some compositing / masking (possible to do in vegas, but definitely easier to do in dedicated compositing programs like AE)
Last edited by poisondeathray; 11th Aug 2014 at 10:30.
poision, thanks again.
one question, when Twixtor re-generates frames between the existing ones, does it change the existing ones as well? does it mess with the whole stream and do an overall blend? if my Super8 is running at 18fps, and i get Twixtor to make 2 new frames (ie new framerate = 54) and then i removed those two new frames in-between the originals after Twixtor has "done its thing"...framerate back to 18fps, will i be left with the original identical video?
It doesn't manipulate the original frames, if you use "whole frame multiples" . In twixtor (at least in the AE version), you set a speed% based on the input framerate. So 50% would mean a 2x version with 1 "inbetween frame" sandwiched between 2 "original frames" . 33% would mean 3x version with 2 "inbetween frames" per 2 original frames, 25% would be 4x version etc...
The output framerate is a different matter - you can assume any framerate - that's sort of the point of twixtor. If you assume the same input framerate but kept the duplicates, the motion will become slo motion. If you assume the higher frame rate, motion will be smoother (as if you shot a higher fps in the first place)
Last edited by poisondeathray; 16th Aug 2014 at 23:25.
Do you have it set on full frames or interpolation? Twixtor can use duplicates, blends or interpolation as well
What are you looking at? The twixtored original , or the the re-inserted version?
I shouldn't have confused you by adding that last paragraph - I was just describing in general how twixtor is used by most people for various scenarios.
You're going to have jittery pans if you're looking at the twixtored version of the original, because the input clip is essentially a variable FPS clip. You have frame drops, and you're interpolating from those original frames. That's normal and expected
For your case - You don't need to select the output framerate in twixtor. It's irrelevant. You're only using twixtor to generate the inbetween frames. You're inserting those frames back into the original sequence
Assuming you've reinserted correct number of missing frames, the "new fixed" version is going to have more total frames than the original 18FPS original. The "original" 18FPS version really isn't 18FPS (it's really a variable FPS lower than 18, because of the frame drops). Once you re-insert those missing frames, it becomes true 18 FPS (as if you had no frame drops in the first place)
I'm also making a whole bunch of assumptions that the Vegas version of twixtor has the same interface as the AE version - can you post a screenshot of the GUI or settings ?
actually the panning shot i was referring to was not the one of me as a baby (with the missing frames) it was a nice and clean one of just some countryside and trees. when i twixtored it to 33%, then sped it up 3x and then looked at it with 50fps avi file it was "sticking" every second and jittery. its a really tough one with 18fps Super8 film, how to get it to twixtor and run smoothly on an avi that is a "typical" rate of 50 or 60 without actually speeding up the footage and making people walk faster etc. i havent really figured that one out yet.
so just discussing the clean countryside shot for a moment, i guess the easy way is to speed up 18fps to 20 and then twixtor 33% and then speed up 3x and make it a nice even 60fps NTSC. Thing is, i am in Australia and we use PAL so ideally i would like to end up at 50fps so it is more compatible across the board, so all the family can watch it safely on their TVs and systems. (headscratch?). hehe.
Thank you for your responses poison, they are very knowledgeable and have helped me understand things a lot thus far
Ok, one goal is to fill in the missing frames (that's the baby footage) . Once you get that to true 18FPS by filling in the gaps, then you could try and use twixtor again to make it 59.94 or 50 or whatever
For the countryside footage - if it wasn't missing frames and had a native FPS of 18 , and constant acceleration/deceleration in the first place (ie. it was only as "jerky" as 18FPS normally is) , a 33% speed setting in twixtor will give you the equivalent of "54 FPS" (because 18x3=54) . (Note it's not really, "54FPS", recall I said the framerate is not really relevant - it's more accurately described as a 3x frames version.) If you then assume a rate of 50, it will be the same thing as a ~7.4% slowdown (the motion will be slowed down because 50/54 = ~92.6%) - that's probably what you're talking about ?
To keep constant motion as the original (but only smoother, because higher FPS), we have to use the LCM (lowest common multiple) , so the frames are evenly divisible. So the LCM of 18, and 50 (if that's your desired rate) is 450 (I think if my math is correct)
So you interpolate to the equivalent of 450 FPS, then take every 9th frame, so they are "evenly spaced" in time.
Now the math behind it is explained, to do the same thing in twixtor you would use 36% (because 18/50 = 0.36 or 36%) . Note you won't keep all original frames (many will be dropped) because it's not a "whole frame" multiple like a 2x version (50% speed), 3x version (33% speed), etc....
Not trying to "sell" avisynth here (since it's free ) , but it does all that automatically, you just enter the desired output rate in many of the interpolation functions. It even has SelectEvery() fuctions that can select every nth frame for example if you wanted to do it manually or process the twixtor footage more easily. Or mix & match some mvtools function frames with twixtor frames. Believe it or not, some of those seemingly "simple" manipulations like selecting every nth frame is almost impossible to do in NLE's
Last edited by poisondeathray; 16th Aug 2014 at 23:45.
Sounds like whoever transferred the film to video screwed it up. 18fps is an odd framerate, but has a common denominator of 8fps compared with 24fps. So it seems like you're missing every 4th frame.
24fps is considered the minimum framerate without noticeable judder. So are you SURE the original film was 18fps?
Vegas can retime any video and preview live with no render. Right click the event and choose "properties" and you will find two framerate tools, one is over/under crank, the other is slomo/fastmo.
Why don't you try overcranking 25% to make 24fps and see if there's any improvement. When you hit the right cadence, any judder should be spread out evenly across all frames. If that's the case, then you can go back and recapture the film properly.
P.S. This is just a theory. Lets see if it gets shot down.
Last edited by budwzr; 17th Aug 2014 at 09:56.
You can do that in the project settings, then make a custom render template.
thanks guys. much appreciated . ive learnt a lot from this discussion cheers!