Frame interpolation

[themaster1]

I'm looking for a script to replace bad frames of my choice using SVPflow.
Thanks for your inputs

[manono]

Me too. So far I'm using a script that still does it the 'old-fashioned' way. My understanding is that the improvement won't be all that much. I don't believe one updated for SVPFlow has been made available yet. And I have no idea how to convert my current script.

[jagabo]

This is renamed from some other script that was posted here before, RX().

function ReplaceFramesMC(clip Source, int N, int X)
{
 # N is number of the 1st frame in Source that needs replacing. 
 # X is total number of frames to replace
 #e.g. RX(101, 5) would replace 101,102,103,104,105 , by using 100 and 106 as reference points for mflowfps interpolation
 
start=Source.trim(N-1,-1) #one good frame before, used for interpolation reference point
end=Source.trim(N+X,-1) #one good frame after, used for interpolation reference point
 
start+end
AssumeFPS(1) #temporarily FPS=1 to use mflowfps
  
super = MSuper()
backward_vec = MAnalyse(super, isb = true)
forward_vec = MAnalyse(super, isb = false)
MFlowFps(super, backward_vec, forward_vec, blend=false, num=X+1, den=1) #num=X+1
AssumeFPS(FrameRate(Source)) #return back to normal source framerate for joining
Trim(1, framecount-1) #trim ends, leaving replacement frames
  
Source.trim(0,-N) ++ last ++ Source.trim(N+X+1,0)
}

There was a simpler version which only replaced one frame. R() maybe?

https://forum.videohelp.com/threads/322848-Scripts-and-filters-for-an-Opera-%28sample-v...=1#post2091091

[themaster1]

Right on, this works with MFlowFps though it's about time we update this no ?
I was looking at this page and i thought it was self apparent svpflow is best.

[jagabo]

Yes, I was thinking you might be able to modify it to use SVPFlow(). Never used SVPFlow() myself.

[manono]

I haven't ever seen that page before and, yes, there's some pretty good evidence there.

So, do you know how to convert that script jagabo put up (which, together with another one by Gavino for single frame interpolation are the ones I use) to using SVPFlow? I have no use at all for Interframe.

[poisondeathray]

Haha RX() was my first function ever! (I had massive help from Gavino of course)

I tried rewriting it with SVPFlow about a year ago, the problems IIRC were SVSmoothFPS wouldn't accept a variable and converting the vectors to MFlowFPS format using SVConvert (instead of using SVSmoothFPS for the interpolation) didn't work either

I would PM Gavino or ask on Doom9 avisynth forum, because I am pure hack at writing these things. It's probably something simple I overlooked

[jagabo]

I see one way to replace individual frames with SVPFlow -- use it on SelectEven() and SelectOdd() to make two videos with motion interpolated frames at every other frame, then ReplaceFramesSimple() to select frames from those. This won't work if two or more sequential frames are bad though. And it's expensive in terms of CPU usage.

[manono]

Haha RX() was my first function ever!

You made it? I had no idea. Well, thanks a lot. I use it virtually every day in my VHS to DVD projects.

[jagabo]

I almost have it working. Here's a script to replace a single frame with SVPFlow.

function ReplaceFrameSVPFlow(clip Source, int N)
{
# N is number of the 1st frame in Source that needs replacing.
#e.g. ReplaceFrameSVPFlow(101) would replace 101 , by using 100 and 102 as reference points for interpolation

start=Source.trim(N-1,-1) #one good frame before, used for interpolation reference point
end=Source.trim(N+1,-1) #one good frame after, used for interpolation reference point

start+end
AssumeFPS(1) #temporarily FPS=1 to use mflowfps

super=SVSuper("{gpu:0}")
vectors=SVAnalyse(super, "{}")
SVSmoothFps(super, vectors, "{rate:{num:2, den:1}}", url="www.svp-team.com", mt=1)

AssumeFPS(FrameRate(Source)) #return back to normal source framerate for joining
Trim(1, framecount-1) #trim ends, leaving replacement frames

Source.trim(0,-N) ++ last ++ Source.trim(N+2,0)
}

It should probably be modified so that one could pass other arguments like gpu:1 and mt=4.

The problem I'm having for replacing multiple frames is that the rate argument won't take a named variable:

function ReplaceFramesSVPFlow(clip Source, int N, int X)
{
...
SVSmoothFps(super, vectors, "{rate:{num:X+1, den:1}}", url="www.svp-team.com", mt=1)
...
}

[jagabo]

OK, I think I got it:

function ReplaceFramesSVPFlow(clip Source, int N, int X)
{
 # N is number of the 1st frame in Source that needs replacing. 
 # X is total number of frames to replace
 #e.g. ReplaceFramesSVPFLow(101, 5) would replace 101,102,103,104,105 , by using 100 and 106 as reference points for SVPFlow interpolation
 
start=Source.trim(N-1,-1) #one good frame before, used for interpolation reference point
end=Source.trim(N+X,-1) #one good frame after, used for interpolation reference point
 
start+end
AssumeFPS(1) #temporarily FPS=1 to use mflowfps

super=SVSuper("{gpu:0}")
vectors=SVAnalyse(super, "{}")
SVSmoothFps(super, vectors, "{rate:{num:"+String(X+1)+", den:1}}", url="www.svp-team.com", mt=1).Subtitle("SVPFlow")

AssumeFPS(FrameRate(Source)) #return back to normal source framerate for joining
Trim(1, framecount-1) #trim ends, leaving replacement frames
  
Source.trim(0,-N) ++ last ++ Source.trim(N+X+1,0)
}

Oh, take out the Subtitle() call. I was using it just make the interpolated frames obvious.

[poisondeathray]

Nice work jagabo! That's the problem I was referring to - the named variable "X+1" with SVSmoothFPS

LOL... It probably only took you a couple of minutes to figure it out. I spend a whole night on the original, and probably even longer trying to adapt it to SVPFLow

[themaster1]

Nice guys now i have to try this but i get an error svpflow_cpu.dll "not an avisynth plugin" same for the gpu dll

[jagabo]

Maybe it requires AviSynth 2.6? MT version?

[themaster1]

sh... i hate these versions changes, i'll try this J

[jagabo]

I just have svpflow_cpu.dll in AviSynth's plugins folder. It's being used -- I get an error if I remove it.

[jagabo]

I'm not always getting real smooth output from ReplaceFramesSVPFlow(). I wonder if the granularity of the numerator and denominator should be higher?

<edit>
The problem seems to be in SVPFlow itself. Maybe I'm using it wrong but it gives jerky motion at times. I'm using some smooth scrolling credits from the end of a movie. Just as a test I decimate to 1/4 the number of frames and use SVPFlow() to restore the original frame rate and smooth motion:

AviSource("SmoothScrolling.avi", audio=false) # 23.976 fps
SelectEven()
SelectEven()
super=SVSuper("{gpu:0}")
vectors=SVAnalyse(super, "{}")
SVSmoothFps(super, vectors, "{rate:{num:4, den:1}}", url="www.svp-team.com", mt=1)

The result is jerky. An equivalent script using MVTools:

function SmoothFPS2(clip source, int num, int den) { 
super = MSuper(source, pel=2, hpad=0, vpad=0, rfilter=4)
backward_1 = MAnalyse(super, chroma=false, isb=true, blksize=16, searchparam=3, plevel=0, search=3, badrange=(-24))
forward_1 = MAnalyse(super, chroma=false, isb=false, blksize=16, searchparam=3, plevel=0, search=3, badrange=(-24))
backward_2 = MRecalculate(super, chroma=false, backward_1, blksize=8, searchparam=1, search=3)
forward_2 = MRecalculate(super, chroma=false, forward_1, blksize=8, searchparam=1, search=3)
backward_3 = MRecalculate(super, chroma=false, backward_2, blksize=4, searchparam=0, search=3)
forward_3 = MRecalculate(super, chroma=false, forward_2, blksize=4, searchparam=0, search=3)
MBlockFps(source, super, backward_3, forward_3, num, den, mode=0)

AviSource("SmoothScrolling.avi", audio=false) # 23.976 fps
SelectEven()
SelectEven()
SmoothFPS2(24000, 1001)
}

is much smoother.

I'll have to look deeper into the use of SVPFlow.

</edit>

[themaster1]

Overall is it better than MFlowFps Jagabo ? Post some pix please

[jagabo]

Overall is it better than MFlowFps Jagabo ? Post some pix please

From what I've seen, SVPFlow has the potential to return better results than MvTools. It's less prone to generating bent lines and edges (it will blend those lines and edges instead) and other weird artifacts. But it has some problems with the timing of the generated frames. This becomes especially noticeable when generating several frames in a row.

For example, say an object moves from position 0 to 8 between two frames. And you want to generate 3 intermediate frames. MvTools will position the object at 2, 4, and 6 in the intermediate frames. But SvTools might position it at 1.8, 3.6, and 6.2. So the motion won't be as smooth.

You can't see this in still images so here are some short video samples. I started with some smooth scrolling subtitles and encoded them with x264 to MKV:

source.mkv:

AviSource("SmoothScrolling.avi", audio=false)

mvtools:

AviSource("SmoothScrolling.avi", audio=false) 
SelectEven()
SelectEven()
SmoothFPS2(24000,1001)

svpflow:

AviSource("SmoothScrolling.avi", audio=false) 
SelectEven()
SelectEven()
super=SVSuper("{gpu:0}")
vectors=SVAnalyse(super, "{}")
SVSmoothFps(super, vectors, "{rate:{num:4, den:1}}", url="www.svp-team.com", mt=1)

As you can see the source video (encoded with x264) is smooth. After reducing the video to 1/4 as many frames, using mvtools to restore the original frame rate gave fairly smooth results. But doing the same with SVPFlow gave jerky results.

[jagabo]

Another example where neither works very well:

Two consecutive frames from the original Tron:




The intermediate frame from doubling the frame rate with MvTools:


The intermediate frame from doubleing the frame rate with SVPFlow:

[Anonymous344]

Can either SVPFlow or MvTools interpolate frames without replacing them? For example, if one had a sequence of frames that ran 1,2,3,5 and wanted to interpolate the missing fourth frame using the the third and fifth frames, could either of the scripts above be modified to do this?

[jagabo]

Can either SVPFlow or MvTools interpolate frames without replacing them? For example, if one had a sequence of frames that ran 1,2,3,5 and wanted to interpolate the missing fourth frame using the the third and fifth frames, could either of the scripts above be modified to do this?

The original function was named RX(). I found it to be non-descriptive and hard to search for so I changed the name to ReplaceFramesMC():

function ReplaceFramesMC(clip Source, int N, int X)
{
 # N is number of the 1st frame in Source that needs replacing. 
 # X is total number of frames to replace
 #e.g. RX(101, 5) would replace 101,102,103,104,105 , by using 100 and 106 as reference points for mflowfps interpolation
 
start=Source.trim(N-1,-1) #one good frame before, used for interpolation reference point
end=Source.trim(N+X,-1) #one good frame after, used for interpolation reference point
 
start+end
AssumeFPS(1) #temporarily FPS=1 to use mflowfps
  
super = MSuper()
backward_vec = MAnalyse(super, isb = true)
forward_vec = MAnalyse(super, isb = false)
MFlowFps(super, backward_vec, forward_vec, blend=false, num=X+1, den=1) #num=X+1
AssumeFPS(FrameRate(Source)) #return back to normal source framerate for joining
Trim(1, framecount-1) #trim ends, leaving replacement frames
  
Source.trim(0,-N) ++ last ++ Source.trim(N+X+1,0)
}

That replaces X frames starting at N. So using your example or replacing only frame 4:

ReplaceFramesMC(video, 4, 1)

There also was a simpler version which replaced only one frame but I didn't keep it around. If you search for ReplaceFramesMC in my posts you'll probably find links to the original functions.

Here's the earliest reference I found with a quick search:

https://forum.videohelp.com/threads/322848-Scripts-and-filters-for-an-Opera-%28sample-v...=1#post2091091

That thread has references earlier but that particular post has a cleaned up version.

[Anonymous344]

Thanks. I'll search your posts for the original functions, but I'm not sure I've made myself clear. I wasn't talking about replacing the fourth frame in a sequence. I was talking about adding a frame where one was missing, so in my example there is a missing frame that has to be reconstructed using its neighbors, not a bad fourth frame that needs replacing.

[jagabo]

Ah, I see. I think the function can pretty easily be modified to do what you want.

[poisondeathray]

Can either SVPFlow or MvTools interpolate frames without replacing them? For example, if one had a sequence of frames that ran 1,2,3,5 and wanted to interpolate the missing fourth frame using the the third and fifth frames, could either of the scripts above be modified to do this?

In this example, let's pretend the source was missing 2,7,12....etc

So the frame numbers are 0,1,3,4,5,6,8, ....

To modify, you just adjust the period and the offset . In this example, 5,2 means 5 is the period (every 5th frame), offset is 2 (start at 2, because numbering in avisynth starts at zero) . So all those lines that say 5,2 , you replace with whatever values match your clip

(you need applyevery.dll for this)
http://avisynth.org/stickboy/

Orig = WhateverSource()

# Make a black clip, the same characteristics as your source
BlankClip(Orig)
Black=last

# Insert black frame in those missing frames (just temporary placeholder, will replace later)
Orig
InterleaveEvery(Black, 5,2)
BlackInserted=last

# Generates Missing Frames by interpolation
BlackInserted
sup = MSuper()
bv = MAnalyse(sup, isb=true, delta=2)
fv = MAnalyse(sup, isb=false, delta=2)
interpolated = MFlowInter(sup, bv, fv, time=50.0, ml=100).DuplicateFrame(0)
replace = interpolated.SelectEvery(5,2) 

# Delete Black Frames
DeleteEvery(5,2) 

# Replace Deleted Frames With Interpolated Frames
InterleaveEvery(replace, 5,2)

[Anonymous344]

Ah! That's a clever work-around. So you add black frames where frames are missing and then replace them. In effect, one is inserting "fake" bad frames so that they can be replaced.

I do have one concern. In your example, it seems as if your source is missing the second frame every five frames. The situation I envisage is if a single frame is missing in the length of a video.

But I suppose if ReplaceFramesMC only replaces the frames that one specifies, I could add black frames where needed and then just use that.

Thanks

[poisondeathray]

Ah! That's a clever work-around. So you add black frames where frames are missing and then replace them.

I do have one concern. In your example, it seems as if your source is missing the second frame every five frames. The situation I envisage is if a single frame is missing in the length of a video.

But I suppose if ReplaceFramesMC only replaces the frames that one specifies, I could add black frames where needed and then just use that.

Thanks

LOL , I misunderstood, I thought you meant EVERY 5th frame ... as in when someone accidentally decimates a clip that shouldn't have been

Yes, if it's just 1 frame, you can insert a blank frame (it doesn't have to be blank or black, can be anything) , then use one of the other functions to interpolate/replace it

[Anonymous344]

That's all right. I thought I was being clear but obviously not. Between you, you and jagabo have given me what I was looking for. (All the other threads I found envisaged large-scale frame interpolation e.g. doubling the frame-rate for smooth playback: that's why I asked here instead.) And your solution brings this thread back on topic after I derailed it slightly, so it's all good. Thanks again.

[jagabo]

I'm working on a mod of ReplaceFramesMC() that inserts motion interpolated frames at the indicated position. I'm calling it InsertFramesMC(). It's basically working but the question is what should be done with the audio?

[Anonymous344]

It's basically working but the question is what should be done with the audio?

In my situation, I plan to use audio from another source. The mod sounds fantastic! It should save a lot of work.