Interpolate a video from 23.976fps to 71.928fps or 60fps?

[Platos]

Im thinking about what is better (and results in less stutter) for a 60Hz monitor when interpolating with hybrid RIFE.

If you would have to interpolate a 23.976 video with a 3x multiplier, what would you do? Just let it there or "scale it down" to 60FPS to match the monitors refreshrate?

I mean when the video has 71.928FPS, there will come a frame, where it get's dropped from my monitor, because it can't be displayed. Then it will just take the next one. On the other hand if i "downscale" the framerate to 60FPS after interpolation, it will definitifly drop some frames (i think, that is what hybrid does in "speed change" ?). Or how does that work, when you display a 71.928FPS video on a 60Hz monitor? How will it get "downscaled" to 60FPS ?

So what would you do, if you would have to decide?

Will it result in the exact same behavior ? So better let it on 71.928FPS instead of reducing it's framerate to 60fps?

[poisondeathray]

A 60Hz monitor will effectively drop frames too

The RIFE 4.x models can achieve any framerate. eg. You can use 60/1 or 60000/1001

[Bwaak]

If you would have to interpolate a 23.976 video with a 3x multiplier, what would you do?

I would not.

I would rather get a 72 Hz monitor that can display 24p with 3x frame repeat without judder. As for characteristic 24 fps stutter, it is a part of traditional movie experience.

[Platos]

A 60Hz monitor will effectively drop frames too

The RIFE 4.x models can achieve any framerate. eg. You can use 60/1 or 60000/1001

Thanks for your answer.

Sadly this is not possible in Hybrid with mlrt RIFE. So im forced to use 2x or 3x.

If i would do a 2x time Interpolation from 23.976 to 47.952, is there a way how i can sync the refresh-rate from the monitor to my video-file so to the media-player ?

If you would have to interpolate a 23.976 video with a 3x multiplier, what would you do?

I would not.

I would rather get a 72 Hz monitor that can display 24p with 3x frame repeat without judder. As for characteristic 24 fps stutter, it is a part of traditional movie experience.

When did i ask for your opinion about using or not using interpolation? Not possible on a forum to ask a question without coming someone telling you their opinion about sth, you never asked ?

No, i guess not, because some people are so hardcore in their opinion, they have to spread it everywhere... Just annoying

[poisondeathray]

Sadly this is not possible in Hybrid with mlrt RIFE. So im forced to use 2x or 3x.

Other implementations like vs-rife support it, check to see if it's available in hybrid, or ask Selur to support it

mvtools can achieve any framerate too, another option is to perform a 2nd step mvtools . This is what people did before RIFE 4.x models were available.

If i would do a 2x time Interpolation from 23.976 to 47.952, is there a way how i can sync the refresh-rate from the monitor to my video-file so to the media-player ?

Depends on the monitor;

But if it's a 60Hz monitor - unlikely that it will be able to do that

[Platos]

Other implementations like vs-rife support it, check to see if it's available in hybrid, or ask Selur to support it

mvtools can achieve any framerate too, another option is to perform a 2nd step mvtools . This is what people did before RIFE 4.x models were available.

Yeah, im sure it is not possible. It has a multiplier, where i can choose only full steps and i also can not write sth else in it. Hmm, maybe i ask him.

You mean mvtoolsFPS? What is it doing actually? I mean if you reccomend it, i guess it does not just drop frames. So how does it make it not stutter when reducing frames from 71.928fps to 60 fps?

Would be nice, if you could explain, because then i can decide, if the quality is good enough for me.

Depends on the monitor;

But if it's a 60Hz monitor - unlikely that it will be able to do that

Let's say my monitor is vrr compatible (freesync, gsync this stuff). Do you know a tool or sth like that?

[poisondeathray]

Other implementations like vs-rife support it, check to see if it's available in hybrid, or ask Selur to support it

mvtools can achieve any framerate too, another option is to perform a 2nd step mvtools . This is what people did before RIFE 4.x models were available.

Yeah, im sure it is not possible. It has a multiplier, where i can choose only full steps and i also can not write sth else in it. Hmm, maybe i ask him.

It's possible with vs-rife . See line 36,37 fps_num, fps_den

https://github.com/HolyWu/vs-rife/blob/master/vsrife/__init__.py

You mean mvtoolsFPS? What is it doing actually? I mean if you reccomend it, i guess it does not just drop frames. So how does it make it not stutter when reducing frames from 71.928fps to 60 fps?

MVtools uses traditional motion interpolation using vectors instead of machine learning. It too synthesizes "inbetween" frames - or "tweening". You can use FrameRateConverter for more advanced options, artifact reduction/masks . The benefit is it's much faster than RIFE, but the quality is generally worse (not always, on some frames it's actually better. You can mix/match models and MVtools2 settings to get the best results on specific frames)

Would be nice, if you could explain, because then i can decide, if the quality is good enough for me.

Most of the quality is from the 1st step . RIFE can be used from 23.976 => 47.952 => 59.94 (60000/1001)

You don't gain much benefit from going higher 3x or 4x then down to 59.94.

[Platos]

I meant in hybrid it is not possible. But will quote your post in selurs thread

ah ok, so actually i could also use vmlrt RIFE to 47.952 and then use "normal" RIFE for 59.94 ? Would be the same principal, with the difference, that RIFE is quality-wise better and slower in speed? Because in Hybrid the normal RIFE can be adjustet normally.

Other question: Should i really take 59.94 and not 60? nvidia driver says it is 60 and also windows say that in the settings. Both, for my tv and monitor.

[poisondeathray]

ah ok, so actually i could also use vmlrt RIFE to 47.952 and then use "normal" RIFE for 59.94 ? Would be the same principal, with the difference, that RIFE is quality-wise better and slower in speed? Because in Hybrid the normal RIFE can be adjustet normally.

Should be the same result and same speed if you use the same model and same architecture. vs-rife can use tensorrt too . But vsmlrt supports vulkan (other GPU's, ONNX, and even CPU)

If you could use "normal RIFE" (whatever that means), then do it in 1 step to 59.94 using 4.x models. But generally , on most sources - the 2.3, 2.4 model are the best - however they don't support arbitrary framerates. They are also slower models. The 4.x models are faster . You have to experiment to see what works better for your source

Other question: Should i really take 59.94 and not 60? nvidia driver says it is 60 and also windows say that in the settings. Both, for my tv and monitor.

If it's set to 60/1 then use 60/1

[Selur]

Hybrid differentiates between:

• RIFE: https://github.com/HomeOfVapourSynthEvolution/VapourSynth-RIFE-ncnn-Vulkan
• TorchRIFE: https://github.com/HolyWu/vs-rife (does support TRT)
• RIFEmlrt: https://github.com/AmusementClub/vs-mlrt/
  

  def RIFE(
      clip: vs.VideoNode,
      multi: int = 2,
      scale: float = 1.0,
      tiles: typing.Optional[typing.Union[int, typing.Tuple[int, int]]] = None,
      tilesize: typing.Optional[typing.Union[int, typing.Tuple[int, int]]] = None,
      overlap: typing.Optional[typing.Union[int, typing.Tuple[int, int]]] = None,
      model: typing.Literal[40, 42, 43, 44, 45, 46] = 44,
      backend: backendT = Backend.OV_CPU(),
      ensemble: bool = False,
      _implementation: typing.Optional[typing.Literal[1, 2]] = None
  )

  see: https://github.com/AmusementClub/vs-mlrt/blob/b20fc6e726aaa89ba147163d946cfa1579f1fcf2...#L973C1-L984C2
  only support integer multipliers. (Hybrid limited this to 2-10).
  => if vs-mlrt support float multipliers for the frame rate (or better fractions), I'll add support for it in Hybrid.

Cu Selur

Ps.:TorchRIFE and RIFEmlrt require torch- and vsmlrt-addons for Hybrid.

[Platos]

Ahh, i did not know that 2.3 and 2.4 are better. Thought newest is best

@Selur:

When i first testet RIFEmlrt i wrote that in another thread:

It works now perfectly in the first tests.

With 1 Steam i have 50% GPU und 60% VPU Usage (Encoder-Unit). Then i get ~185 FPS (in normal RIFE i get ~55FPS with 2 GPU Threads)
With 2 Streams i have 80%GPU and 80% VPU Usage. Then i get ~245FPS
With 3 Streams i have 90-95% GPU and 85% VPU Usage. Then i get 255FPS.

I retestet it now with some videos and i get results from 10%-70% faster (RIFEmlrt) than RIFE, if i use 4 GPU-Threads with RIFE and 3 Streams in RIFEmlrt (best setting for my RTX 4070Ti). And i use model 4.6. But most of time i get sth around 25-45% faster.

I wonder now, how i could get a 5 times higher framerate like back on the first test. Did RIFE in Hybrid got improved or do i sth wrong? Sadly i don't remember which video i used for that test.

I interpolated this video (only 1 minute): https://uploadnow.io/de

With RIFE, 4 GPU-Threads, model 4.6 and 2x multiplier i get 85FPS. With RIFEmlrt, model 4.6 and 3 Streams and also 2x multiplier i get 109FPS. GPU is fully loaded and CPU is not completly 100% on RIFEmlrt (but quite high). It is faster, no question, but i really wonder, why i get "only" 28% faster and how i could get 5x faster like on my quote. I know, i used 2 GPU Threads there, but i tested that. That can not be the reason for such a big difference.

Would you test RIFEmlrt vs RIFE with this 1 minute video? It took me only ~30sec for each RIFE Version.

BTW: Do you have sth like a changelog for your dev-editions? To know what changed/got added.

[Selur]

I add a new changelog whenever I release a new version, over at https://www.selur.de/changelog.
There is no public available changelog for my dev versions.

about RIFE speed:

short version: no real clue

long version:
RIFE (VapourSynth-RIFE-ncnn-Vulkan) hasn't changed for over a year, neither has the supporting code in Hybrid.
vs-mlrt support is newer.
Speed changes are probably due to:
a. driver changes
b. changes in Vapoursynth (https://github.com/AmusementClub/vs-mlrt/releases/tag/v13.2 uses vapoursynth-classic R57.A8 in their benchmarks, Hybrid uses R63)
c. simply that small differences in the processing of the filters (RIFE vs RIFEmlrt) to can cause larger differences depending on the source.
d. you are doing something wrong
e. something else changed on your system (for me atm. Hybrid does take 10 seconds to load up, if it is under f:\Hybrid, but only 2 seconds if it's under f:\Hybrid2 and I can't figure out why, since I'm too lazy to do a full system reinstall and re-check on each change)

Cu Selur

[Platos]

Hmm ok, i have no clue. Could check a newer/older driver. Hardware did not change and i don't use additional filter (checked on filter order). Maybe it was only this specific video which gave a 500% boost (but actually i can't imagine). Fp16 is even slower on this video, so this can also not be the reason.

I have btw the same thing with 10s loading-time.

I probably can not just change the foldername from hybrid into hybrid2 ? Will this kill the installation?

Can i downgrade with installing an older hybrid version ? Or will that not work ?

[Selur]

You will have to uninstall and reinstall Hybrid when chaning the folder name otherwise the deinstaller will probably not work properly.
Hybrid itself will still work.
You can always deinstall Hybrid and install an older version; you might have to delete your settings.

[Platos]

@ Selur:

ah ok, so actually i could also use vmlrt RIFE to 47.952 and then use "normal" RIFE for 59.94 ? Would be the same principal, with the difference, that RIFE is quality-wise better and slower in speed? Because in Hybrid the normal RIFE can be adjustet normally.

Can i do that someone with Hybrid? I mean i know i can order the filter i apply, but can i interpolate 2 times (with 2 different interpolation-technics) in hybrid in one encoding? So first i interpolate with RIFEmlrt from 23.976 to 47.952FPS and then from there to 60FPS with RIFE in Hybrid (in one encoding)?

Is this possible somehow?

[Selur]

@Platos: In theory: yes; In praxis: no, since I just found a bug that RIFEmlrt doesn't work with Vapoursynth filter queue

Update:
possible now with latest dev, here's an example using 25fps source with RIFEmlrt to go to 50fps and then RIFE to go to 60fps:

# Imports
import vapoursynth as vs
import os
import sys
os.environ["CUDA_MODULE_LOADING"] = "LAZY"
# getting Vapoursynth core
core = vs.core
# Import scripts folder
scriptPath = 'F:/Hybrid/64bit/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))
# Loading Plugins
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/FrameFilter/RIFE/RIFE.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/vs-mlrt/vstrt.dll")
import site
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/Support/fmtconv.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/MiscFilter/MiscFilters/MiscFilters.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/Support/akarin.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/vslsmashsource.dll")
# Import scripts
from importlib.machinery import SourceFileLoader
vsmlrt = SourceFileLoader('vsmlrt', 'F:/Hybrid/64bit/vs-mlrt/vsmlrt.py').load_module()
# source: 'G:\TestClips&Co\files\test.avi'
# current color space: YUV420P8, bit depth: 8, resolution: 640x352, fps: 25, color matrix: 470bg, yuv luminance scale: limited, scanorder: progressive
# Loading G:\TestClips&Co\files\test.avi using LWLibavSource
clip = core.lsmas.LWLibavSource(source="G:/TestClips&Co/files/test.avi", format="YUV420P8", stream_index=0, cache=0, prefer_hw=0)
# Setting detected color matrix (470bg).
clip = core.std.SetFrameProps(clip, _Matrix=5)
# Setting color transfer info (470bg), when it is not set
clip = clip if not core.text.FrameProps(clip,'_Transfer') else core.std.SetFrameProps(clip, _Transfer=5)
# Setting color primaries info (), when it is not set
clip = clip if not core.text.FrameProps(clip,'_Primaries') else core.std.SetFrameProps(clip, _Primaries=5)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
clip = core.std.SetFrameProp(clip=clip, prop="_FieldBased", intval=0) # progressive
from vsmlrt import Backend
clip = core.misc.SCDetect(clip=clip,threshold=0.150)
# adjusting color space from YUV420P8 to RGBS for vsRIFEmlrt
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="470bg", range_s="limited")
# adjusting frame count&rate with RIFE (mlrt)
clip = vsmlrt.RIFE(clip, model=44, backend=Backend.TRT(fp16=True, device_id=0,verbose=True,use_cuda_graph=False, num_streams=1, workspace=1 << 30)) # new fps: 50
# adjusting frame count&rate with RIFE
clip = core.rife.RIFE(clip, model=22, fps_num=60, fps_den=1) # new fps: 60
# adjusting output color from: RGBS to YUV420P10 for QSVEncModel
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, matrix_s="470bg", range_s="limited", dither_type="error_diffusion")
# set output frame rate to 60fps (progressive)
clip = core.std.AssumeFPS(clip=clip, fpsnum=60, fpsden=1)
# Output
clip.set_output()

using Hybrids Vapoursynth FilterQueue, see: [INFO] About Vapoursynth Filter Order/Queue
But I have no clue how demanding that is on the system,...

[Platos]

Thanks, i try to check it in the next days.

Would you give me the latest dev then ?

[Selur]

send you a link to latest dev and addons,...
(links will be valid ~24hrs)

[Samson_stages]

I would choose to downscale the interpolated video to 60FPS to match the monitor's framerate. This approach may introduce some judder and occasional frame artifacts when the frame rate doesn't evenly divide into 60FPS.

[Platos]

@Platos: In theory: yes; In praxis: no, since I just found a bug that RIFEmlrt doesn't work with Vapoursynth filter queue

Update:
possible now with latest dev, here's an example using 25fps source with RIFEmlrt to go to 50fps and then RIFE to go to 60fps:

using Hybrids Vapoursynth FilterQueue, see: [INFO] About Vapoursynth Filter Order/Queue
But I have no clue how demanding that is on the system,...

Question: I don't get it, how i can add filters to the FilterQueue. Can you explain that further? Because i could not see how to do it on the Link. There is no Button or sth similair in Hybrid.

[Selur]

There is no Button or sth similair in Hybrid.

If you enable 'Use Filter Queue', there are '+' and '-' buttons for each filter on the right side of the filter,....

and yes, there are quite a few other options and settings in Hybrid that are not visible by default, since most users do not use them


Cu Selur

[Platos]

Ahhh, i searched it on the FilterQueue-Tab. Thank you.

I tried it, but got an error. I can do both, RIFE and mlrt-RIFE alone with exactly this video. But not when i do mlrtRIFE and then RIFE.

Here is the debug:

And here is the vapoursynth preview:

2023-10-05 18:55:49.312
Failed to evaluate the script:
Python exception: Failed to load C:/Program Files/Hybrid/64bit/vsfilters/FrameFilter/RIFE/RIFE.dll. GetLastError() returned 126. The file you tried to load or one of its dependencies is probably missing.

Traceback (most recent call last):
File "src\cython\vapoursynth.pyx", line 3115, in vapoursynth._vpy_evaluate
File "src\cython\vapoursynth.pyx", line 3116, in vapoursynth._vpy_evaluate
File "C:\Users\Gaming-Tower\AppData\Local\Temp\tempPreviewVapoursynthFile18_55_48_675.vpy", line 12, in 
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/FrameFilter/RIFE/RIFE.dll")
File "src\cython\vapoursynth.pyx", line 2847, in vapoursynth.Function.__call__
vapoursynth.Error: Failed to load C:/Program Files/Hybrid/64bit/vsfilters/FrameFilter/RIFE/RIFE.dll. GetLastError() returned 126. The file you tried to load or one of its dependencies is probably missing.

[Selur]

Seems like a bug, the dll is named librife.dll iirc.
As a quick fix:
copy

C:/Program Files/Hybrid/64bit/vsfilters/FrameFilter/RIFE/librife.dll

to

C:/Program Files/Hybrid/64bit/vsfilters/FrameFilter/RIFE/RIFE.dll

Cu Selur

[Platos]

Do i have to copy this file and then paste it as "RIFE.dll" or just rename the file to "RIFE.dll" ?

But i tried it now and i realized, that it is (sadly) quite useless because:

When i use a 1min testvideo i get:

112FPS with mlrtRIFE (2x multiplier)
48FPS with RIFE (60/1 Framerate)

38FPS with mlrtRIFE + RIFE (with Filter Queue and same settings). So in short: It is slower than when i use just RIFE.

When i manually use RIFE over a 48FPS-output (which i made with mlrtRIFE) i have ~the same FPS, like when i directly use RIFE from 24FPS to 60FPS. I thought it would be faster with RIFE, when i interpolate a video from 48FPS to 60FPS. But it is ~same speed as from 24FPS to 60FPS.

So i did this (mlrtRIFE from 24FPS to 48FPS and then in a second run RIFE to 60FPS). Then i calculated the average FPS and i got 35.75FPS, so similair to the 38FPS i get, when i use the Filter-Queue. So it is quite useless (sadly).

So i better use RIFE and then i have 48FPS. I really thought RIFE would be faster, when i have to interpolate from 48FPS to 60FPS instead of interpolate from 24FPS to 60FPS.

Someone did a request here: https://github.com/AmusementClub/vs-mlrt/issues/59

But probably this will not make the devs implement it