Vapoursynth slow motion

[awgcooper]

Let's say I'm creating a video composed of three trimmed clips from the same source file, i.e. clip1 + clip2 +clip3. In addition, I want clip 2 to be slow motion. What is the best way to do this for someone who's not a particularly sophisticated user?

For avisynth+, I discovered a script that created a 'slowdn' function which abstracted away any complications to do with framerates and interpolation (see attached). It just worked and there were no difficulties combining the slow-mo clip with the others.

It may be that I need to learn more to do the same in vapoursynth but either way, I'd like to know what the options are.

[jagabo]

You want to use RIFE in VapourSynth. RIFE does much better motion interpolation than MVTools. I don't know if anyone has a simplified front end for it though.

[awgcooper]

You want to use RIFE in VapourSynth. RIFE does much better motion interpolation than MVTools. I don't know if anyone has a simplified front end for it though.

Funnily enough I've just come across this since posting. You're right though, it may be a bit beyond my 'pay grade' without a simplified front end. The item I was looking at is here: https://github.com/HomeOfVapourSynthEvolution/VapourSynth-RIFE-ncnn-Vulkan. Even if I managed to apply this function correctly, I'm not quite sure what to do overall to produce a slow motion clip. This leads me to...

...a basic question about the principal of creating a slow motion clip. Is this how it's done: (i) additional frames are interpolated/inserted to increase the framerate; (ii) then the modified clip is has its framerate reduced which gives the appearance of slow motion.

[jagabo]

...a basic question about the principal of creating a slow motion clip. Is this how it's done: (i) additional frames are interpolated/inserted to increase the framerate; (ii) then the modified clip is has its framerate reduced which gives the appearance of slow motion.

Yes. Say you have a 25 fps video and you want to slow it to 1/4 speed. First interpolate to 100 fps, then just assume the video is 25 fps. You will have 4 times as many frames, played at the original frame rate, so it's 4 times longer. Use RIFE to create the interpolated frames, AssumeFPS to slow them down.

[awgcooper]

Got it, that's very clear, thanks a lot.

By the way, I just found this: https://github.com/may-son/RIFE-FixDropFrames-and-ConvertFPS. It's a non-VS way of accessing RIFE and not too difficult to use. I'm currently testing it out. One pretty obvious point: hardware acceleration is fairly essential to get anything done within even a semi-reasonable timeframe. It's currently running at just under 2fps on a laptop RTX 3080. The frames are 4k resolution though.

[_Al_]

It should not be difficult to use with vapoursynth, like this fork: https://github.com/styler00dollar/VapourSynth-RIFE-ncnn-Vulkan/releases, download that librife.dll to vapoursynth plugin directory , then download models (better download whole repozitory zip with models in it, click green Code button and download zip). Models are large bins, perhaps hundreds of MB together.

Unfortunately , I do not have CUDA, cannot test it myself (only mvtools), it should be something like this:

import vapoursynth as vs
from vapoursynth import core
clip = core.lsmas.LibavSMASHSource('video.mp4')
clip = clip[300:400] # get trimmed part from frame 300 to 399 for slow down

#RIFE, needs RGB floating point
clip = clip.resize.Point(format=vs.RGBS, matrix_in_s='709')
model_path=r'H:\downloads\RIFE models\rife-v4.1_ensembleFalse_fastTrue'
slowdown = core.rife.RIFE(clip, model=9, model_path=model_path) #default is twice length for slow down
slowdown = slowdown.resize.Point(format=vs.YUV420P8, matrix_s='709')

#MV   would be something:
##super = core.mv.Super(clip, pel=4)
##backward_vec = core.mv.Analyse(super, overlap=4, isb = True, blksize=8, truemotion=True, search=3)
##forward_vec  = core.mv.Analyse(super, overlap=4, isb = False, blksize=8, truemotion=True, search=3)
##slowdown = core.mv.FlowFPS(clip, super, backward_vec, forward_vec, num=round(clip.fps.numerator/0.5), den=clip.fps.denominator) #slows down to twice length

slowdown = slowdown.std.AssumeFPS(fpsnum=clip.fps.numerator, fpsden=clip.fps.denominator)

[awgcooper]

@_AI_: thank you very much indeed, this seemed to work just fine. Just to check I've understood this, am I correct in saying that RIFE 'increases' the frame rate by inserting interpolated frames and then MVtools returns the frame rate to that of the original thereby creating the slow motion effect and, in this case, doubling the length of the clip?

[jagabo]

mvtools isn't being used at all in that script. Notice the pound sign at the start of each of the lines that use mvtools? That makes everything behind the # a comment, not executed. Just like the line "#RIFE, needs RGB floating point" is just a comment included for clarity.

So you could disable all the RIFE lines by putting a # at the start of each line, enable all the mvtools lines by removing the # signs, then render to see how mv compares to RIFE.

In that script RIFE is used to double the frame rate. Then on the last line AssumeFPS is used to slow the frame rate back to that of the orignal video. The end result is you have twice as many frames that play at the same frame rate as the original video, doubling it's length.

[awgcooper]

Got it. I did notice the comments and at first thought it was an 'either, or' (as you intended). I'm not sure why I then adopted the erroneous interpretation!

Is there any reason to go with this version of RIFE - https://github.com/HolyWu/vs-rife? I'm not quite sure as to the plus or minuses/subtleties of having it as a Python library vs VS plugin (if that is the correct characterisation of the differences).

[poisondeathray]

Or this version, which has several machine learning implementations including RIFE
https://github.com/AmusementClub/vs-mlrt

The main difference is speed if you have a Nvidia card , and especially if you have Tensor cores. RIFE is generally faster using Nvidia and Tensor cores

The NCNN version uses Vulkan, and is accessible by all modern hardware

The main reason to use it as a VS plugin is if you have other filters/transforms to apply. You can preview in vsedit

[awgcooper]

Many thanks for the link (and also for the explanation of what using VS plugin additionally allows). I do have a Nvidia card with tensor cores so I guess I should look at the implementations that support it.

[_Al_]

That Slowndn.avsi from post #1 can dynamically slow down or speed up video, that looks very cool!

Possible slowdown , keep it that way for a while, then gradually speed up again to normal.
I put it into python for vapoursynth.
Functions rife() and mvtools() can be customized, I do not know what the best settings are or a particular RIFE version could be used, not that ncnn-vulcan version . Conversion to RGBS and from is not a bulletproof ,etc ...
It looks it is working ...

#ported to python for vapoursynth from Avisynth script:  https://www.slomo.jp137.com/index-edit.html
#just added RIFE as a default instead of mvtools. If wanting mvtools, use isRife=False

import vapoursynth as vs
from vapoursynth import core
import math

def rife(clip, fpsnum, fpsden):
    if clip.format.name.startswith('RGB'):
        matrix=None
    else:
        matrix='709'
    rgbs_clip = clip.resize.Bicubic(format=vs.RGBS, matrix_in_s=matrix)
    #https://github.com/styler00dollar/VapourSynth-RIFE-ncnn-Vulkan
    model_path=r'H:\downloads\RIFE models\rife-v4.1_ensembleFalse_fastTrue'
    interpolated = core.rife.RIFE(rgbs_clip, model_path=model_path, fps_num=fpsnum, fps_den=fpsden)    
    return interpolated.resize.Bicubic(format=clip.format.id, matrix_s=matrix)

def mvtools(clip, fpsnum, fpsden):
    sup = core.mv.Super(clip, pel=4)
    backward_vec = core.mv.Analyse(sup, overlap=4, isb = True, blksize=8, truemotion=True, search=3)
    forward_vec  = core.mv.Analyse(sup, overlap=4, isb = False, blksize=8, truemotion=True, search=3)
    return core.mv.FlowFPS(clip, sup, backward_vec, forward_vec, num=fpsnum, den=fpsden)    

def Slowdn(source: vs.VideoNode, startFrame: int=None, endFrame: int=None, startRatio: float=1.0, endRatio: float=1.0, isRife:bool=True):
    if startFrame is None:    startFrame = 0
    if endFrame   is None:    endFrame = source.num_frames-1
    if endFrame < startFrame: endFrame, startFrame = startFrame, endFrame
    mfpsn = source.fps.numerator
    mfpsd = source.fps.denominator
    mclip = source[startFrame : endFrame+1]   
    if isRife:  interpolated = rife(mclip, fpsnum=round(mfpsn/endRatio), fpsden=mfpsd)
    else:       interpolated = mvtools(mclip, fpsnum=round(mfpsn/endRatio), fpsden=mfpsd)
    slowdown = core.std.AssumeFPS(interpolated, fpsnum=mfpsn, fpsden=mfpsd)

    startFrames = (mfpsn/startRatio)/mfpsd
    endFrames = (mfpsn/endRatio)/mfpsd
    framesRange = endFrames - startFrames
    currentFrame = 0
    counter = 0
    lastFrame = slowdown.num_frames
    desiredFrames = 0
    removeFrameCounter = 0
    removeFrameThreshold = endFrames / startFrames
    overflow = 0
    
    while counter < lastFrame:
        removeFrameCounter += 1
        if removeFrameCounter >= math.floor(removeFrameThreshold + overflow):
            if overflow > 1: overflow = overflow - 1.0
            else:            overflow = overflow + removeFrameThreshold - math.floor(removeFrameThreshold)
            # Calculate the desired frames. You could use a higher power to make the transition even smoother.
            desiredFrames = startFrames + math.pow(counter/lastFrame, 4) * framesRange
            removeFrameThreshold = endFrames/desiredFrames
            if removeFrameThreshold < 1.1:
                removeFrameThreshold = 1.0
                overflow = 0
            removeFrameCounter = 0
        if removeFrameCounter == 0:
            currentFrame += 1
        else:
            slowdown = core.std.DeleteFrames(slowdown, currentFrame)
        counter += 1
    return slowdown

def Speedup(source: vs.VideoNode, startFrame: int=None, endFrame: int=None, startRatio: float=1.0, endRatio: float=1.0, isRife:bool=True):
    if startFrame is None:    startFrame = 0
    if endFrame   is None:    endFrame = source.num_frames-1
    if endFrame < startFrame: endFrame, startFrame = startFrame, endFrame
    mfpsn = source.fps.numerator
    mfpsd = source.fps.denominator
    mclip = source[startFrame : endFrame+1]   
    if isRife:  interpolated = rife(mclip, fpsnum=round(mfpsn/startRatio), fpsden=mfpsd)
    else:       interpolated = mvtools(mclip, fpsnum=round(mfpsn/startRatio), fpsden=mfpsd)
    slowdown = core.std.AssumeFPS(interpolated, fpsnum=mfpsn, fpsden=mfpsd)

    startFrames = (mfpsn/startRatio)/mfpsd
    endFrames = (mfpsn/endRatio)/mfpsd
    framesRange = startFrames - endFrames
    currentFrame = slowdown.num_frames
    counter = 0
    lastFrame = slowdown.num_frames
    desiredFrames = 0
    removeFrameCounter = 0
    removeFrameThreshold = startFrames / endFrames
    overflow = 0

    while counter < lastFrame:
        removeFrameCounter += 1
        if removeFrameCounter >= math.floor(removeFrameThreshold + overflow):
            if overflow > 1: overflow = overflow - 1.0
            else:            overflow = overflow + removeFrameThreshold - math.floor(removeFrameThreshold)
            # Calculate the desired frames. You could use a higher power to make the transition even smoother.
            desiredFrames = endFrames + math.pow(counter/lastFrame, 4) * framesRange
            removeFrameThreshold = startFrames/desiredFrames
            if removeFrameThreshold < 1.1:
                removeFrameThreshold = 1.0
                overflow = 0
            removeFrameCounter = 0
        if removeFrameCounter:
            try:
                slowdown = core.std.DeleteFrames(slowdown, currentFrame)
            except vs.Error:
                pass
        currentFrame -= 1
        counter += 1
    return slowdown

if __name__ in ['__main__', '__vapoursynth__']:
    '''
    Possible usage:
    '''
    clip = core.lsmas.LibavSMASHSource(r"G:\some_path\video.mp4")    
    clip1 = clip[:100]    #for regular speed from start
    clip2 = clip[100:400] #slowdown from regular speed to slomo
    clip3 = clip[400:500] #hold slowdown
    clip4 = clip[500:700] #gradually back to normal speed
    clip5 = clip[700:]    #regular speed to the end

    clip2 = Slowdn(clip2,  startRatio=1,   endRatio=0.2)
    clip3 = Slowdn(clip3,  startRatio=0.2, endRatio=0.2) #just keeping slow-mo for a while
    clip4 = Speedup(clip4, startRatio=0.2, endRatio=1)
    out = clip1+clip2+clip3+clip4+clip5
    out.set_output()

[awgcooper]

That's great to have a vapoursynth version, thanks. I will try it out later.

Also, I should make clear that the Slowdn function isn't my work (although that's probably quite clear from the level of my questioning in this thread!). I came across it on Doom9 but I can't find the thread right now. There is a reference in the comments at the top of the file to 'lecapitan' from Doom 9 as being the primary author.

[_Al_]

This is better link that references that avisynth script: https://www.slomo.jp137.com/index-edit.html, maybe that GScript is not needed for latest avisynth, not sure, I have actually never ran that avisynth script, only that ported vapoursynth version

I put that reference into that python file.

[_Al_]

went to here: https://www.shutterstock.com/video/clip-1020551014-young-male-worker-trips-on-puddle-water and downloaded stock footage (right click video and Save Video As (using Firefox)
Changed video name to worker.webm because name is too long, then using above script with:

if __name__ in ['__main__', '__vapoursynth__']:
    source = r'H:\downloads\worker.webm'
    clip = core.ffms2.Source(source)
    clip = clip.resize.Bicubic(width=640,height=360)  #getting to proper mod for mvtools, I think mvtools needs 8 (or 16?) the way I set it up
    clip1 = clip[:70]     #for regular speed from start
    clip2 = clip[70:90]   #slowdown from regular speed to slomo
    clip3 = clip[90:120]  #hold slowdown
    clip4 = clip[120:150] #gradually back to normal speed
    clip5 = clip[150:]    #regular speed to the end

    clip2 = Slowdn(clip2,  startRatio=1,   endRatio=0.2, isRife=False)
    clip3 = Slowdn(clip3,  startRatio=0.2, endRatio=0.2, isRife=False)
    clip4 = Speedup(clip4, startRatio=0.2, endRatio=1, isRife=False)
    out = clip1+clip2+clip3+clip4+clip5
    out.set_output()

vspipe.exe -c y4m slow_mo.py -  | x264.exe --demuxer y4m --crf 18 --vbv-maxrate 30000 --vbv-bufsize 30000 --keyint 60 --tune film --colorprim bt709 --transfer bt709 --colormatrix bt709 --output output.264 -
MP4Box.exe -add output.264  -new output.mp4

So that was mvtools. RIFE perhps would give better result.

[awgcooper]

Here's the RIFE version using rife-v4.4_ensembleFalse_fastTrue. To my untrained eye it looks smoother.


[@_AI_: a couple of very basic questions if I may:
(i) why is your script a python script rather than .vpy extension
(ii) if I wanted to make slowdn and speedup importable into my main script so I could leave the definitions out of the file, what should I do and does this vary depending on whether my main script is .py or .vpy. I believe I need to place a copy of the function definition in the python site-packages dir or something like that but I'd like to clarify this once and for all.
(iii) why MP4Box rather than, say, ffmpeg to remux the x264 file]

[_Al_]

Thanks, yes, it is like a generation or two better.

(i)
vspipe takes vpy or py, it does not matter. If having py, you can run it as you would any python script using python editor like IDLE or pycharm ..etc., for example you can add at the end of your script:

import view #download here, it also needs opencv installed
view.Preview(out)

and you can just preview your clip right away

or think of it like any python scrip, any, can turn into a vapoursynth script if you just add to it:
import vapoursynth as vs
from vapoursynth import core
that is just a massive difference as oppose to avisynth. You can also use portable python and vapoursynth in a directory and just freeze it to an exe file for some utility. You cannot do that if having script written in vpy.
When using py, you can edit your script in that python editor of your choice and you have a color highlights and all other comfort, not needing vsedit.
Another huge advantage is you can use those py scripts as python modules and import them into other python scripts, it is explained below.

Downsides of using py: ffmpeg or virtualdub would not directly load script with py extension, for that you need vpy perhaps. If using ffmpeg you can use vspipe and pipe output into ffmpeg. But that you cannot do with virtualdub I guess. Maybe there is more, not sure now.

(ii)
save that posted file as for example slow_mo.py
It has to be either in a working directory or python site-packages directory.
Then just import it into your script:

import vapoursynth as vs
from vapoursynth import core

import slow_mo

clip = core.ffms2.Source(r'H:\downloads\worker.webm')    
clip = clip.resize.Bicubic(width=640,height=360)    
clip1 = clip[:70]     #for regular speed from start
clip2 = clip[70:90]   #slowdown from regular speed to slomo
clip3 = clip[90:120]  #hold slowdown
clip4 = clip[120:150] #gradually back to normal speed
clip5 = clip[150:]    #regular speed to the end

clip2 = slow_mo.Slowdn(clip2,  startRatio=1,   endRatio=0.2)
clip3 = slow_mo.Slowdn(clip3,  startRatio=0.2, endRatio=0.2)
clip4 = slow_mo.Speedup(clip4, startRatio=0.2, endRatio=1)
out = clip1+clip2+clip3+clip4+clip5
out.set_output()

##if __name__=='__main__':
##    import view
##    view.Preview(out)

Note, if you import a module and that module (python script) has some code in that if __name__== __main__: block, than that code is not run. That is the purpose of that code. Variable __name__ is '__main__' only if you run it as a main script. If it is imported, variable __name__ is NOT '__main__'.


Also vspipe (rather vsscript that is used) sets variable __name__ as '__vapoursynth__'. So if you put at the end of your script :

if __name__=='__main__':
    import view
    view.Preview(out)

that block is not run also if using that script with vspipe, it is very convenient, I just leave it at the bottom of every script. When I run script manually, I can preview it on screen. But as soon that script is loaded into vspipe, that block is ignored (it would be a problem if it would not)


if your slow_mo.py is not in a working directory or python site-packages directory, you can also register that directory manually so python would look into that directory while importing modules:

import sys
some_module_directory = r'D:\modules'         #directory where your slow_mo.py is
sys.path.append(some_module_directory)

import slow_mo
import vapoursynth as vs
.
.
.

(iii)
it does not matter, just what you prefer , I am afraid of using ffmpeg , it tends to complain all the time about something ....

[awgcooper]

Thanks, that's a very helpful set of explanations. It seems the advantages of using straight python files are pretty compelling.

Regarding importing, I tried saving slo_mo.py in ..\AppData\Roaming\Python\Python311\site-packages and ..\AppData\Local\Programs\Python\Python311\Lib as well as creating a defined directory. In all cases I was unable to do 'import slo_mo' however it would work if I used 'from slo_mo import *'. Any ideas why this is happening? It's no big deal but it would be good to know.

P.S. I love the viewer, I'm using it from within VS Code.

[_Al_]

it was a syntax mistake perhaps, I made a mistake in my example above and typed slo_mo couple of times instead of slow_mo. I fixed that already.