Hybrid(Windows/Linux/Mac): Input -> x264/x265/Xvid/VP8/VP9

[Platos]

Ok, thanks for information.

So but with an RTX 4070Ti shouldn't i be faster with Tensor TorchRIFE? Are you faster with TorchRIFE than with normal Rife? If yes, i do sth. wrong, because my CPU is not fully used with both, TorchRIFE and normal RIFE. Because with the vsmlrt tensor-thing im way faster in upscaling. Shouldn't it be the same for interpolation?

Can i somehow edit the tooltips for my specific hybrid-installation ? So i can add some things with clarification i wrote it (than i dont forget that, like ensemble increses quality, but lowers speed) ?

[Selur]

Can i somehow edit the tooltips for my specific hybrid-installation ?

No, you can post over at https://forum.selur.net/Thread-HELP-Missing-Better-tool-tips and I might add them.

About the speed, from my experience: RIFE is usually faster than TorchRIFE. Wild guess would be this might be due to some overhead from using Python bindings for torch or somehow optimized models,...
If you want to know why, you would have to ask the authors of the plugins:
https://github.com/styler00dollar/VapourSynth-RIFE-ncnn-Vulkan
https://github.com/HolyWu/vs-rife
I haven't tested how fast vs-mlrts RIFE version is.

Because with the vsmlrt tensor-thing im way faster in upscaling.

okay,...

[Platos]

Ahh, i thought that have to be the vsmlrt version, but then i understand. When i see the FPS Gain in Upscaling, I would say this should be alot faster.

Maybe i have to search this Thread again from you. Didn't you do there sth. with RIFE?

And ok, i will have a look on the tooltipps-link, thanks.

[Selur]

Didn't you do there sth. with RIFE?

in https://forum.videohelp.com/threads/409482-vs-mlrt-a-bit-after-Getting-started-Backend-trouble I did test what was possible, but I only added generic model support in Hybrids vs-mlrt addon in my dev version.

[firest]

Hey Selur, when I try to preview an AviSynth script in "Filtering -> Avisynth -> Show current Avisynth script", Hybrid shows several error warnings like this:


After ignoring them all by closing the warning window or clicking "ok", the script preview is shown as usual.
I've tried resetting the settings to default, as well as deleting the settings folder so that Hybrid can generate them again, but it has no effect, the warnings keep popping up. Does this happen on your end?

Also, any news regarding the subtitles extraction? i.e.:

According to my tests, subtitle extraction via "Subtitle -> Extract" only doesn't cause Hybrid to close unexpectedly after extraction if a source is loaded in the main tab first.

[Selur]

Ah, those popups are caused by something I'm working on in my dev version, will send you a new version once I'm finished with what I'm working on atm.. (~2hrs)
Regarding the crash, create a debug output with that new version, and I'll look into it.

Cu Selur

[Selur]

send you a new link

[Platos]

Didn't you do there sth. with RIFE?

in https://forum.videohelp.com/threads/409482-vs-mlrt-a-bit-after-Getting-started-Backend-trouble I did test what was possible, but I only added generic model support in Hybrids vs-mlrt addon in my dev version.

Ahh ok, you mean generic model support for vs-mlrt with VSGAN models ? Because it's quite good, so i would not call it generic

[Selur]

Generic, since it allows using any model vs-mlrt supports, but it won't work properly with models which require additional stuff like RIFE&Co.
Btw. the latest dev I send you also contains basic RIFEmlrt support (some params are missing).

[Platos]

ah wow, cool. I will test it right now

(But i did not recieve a message)

[Platos]

Maybe you sent it accidently someone else.

Btw. Question: If there is someday a new public release of Hybrid, will it contain the vsmlrt stuff, modles and so on you sent me or is it thought, that users download it self from github?

I saved it anyway on backup-memory, but just asking.

[firest]

send you a new link

Fixed, thanks.
The problem with subtitle extraction doesn't happen anymore either.

[Selur]

@Platos: send you a link.

Btw. Question: If there is someday a new public release of Hybrid, will it contain the vsmlrt stuff, modles and so on you sent me or is it thought, that users download it self from github?

I have no plan to share tthe orch- and vsmlrt-addon and the models that belong to them through the homepage or github.
Those are only available for folks that ask me, about them.

Cu Selur

[Platos]

Ok, i testet it. I got an error because of sth in HEVC Encoding i believe. Here the Debug:

Debug1 is with 5MBit/s variable Bitrate.
Debug 2 ist with 20 constant rate factor. But the Job-Tab shows still variable bitrate encoding (but i take constant rate factor. testet it 3 times).

(Edit: it happens also on NVEnc h.264 and also with CPU-H.264)

Btw. the default model was 4.4 instead of 4.6 (like showed in the tooltip).

Btw2: In "C:\Program Files\Hybrid\64bit\vs-mlrt\models" there is one time "rife" and one time "rife_v2".

[Selur]

in both caases, NVEnc fails decoding the Vapoursynth script:

# Imports
import vapoursynth as vs
import os
import sys
os.environ["CUDA_MODULE_LOADING"] = "LAZY"
# getting Vapoursynth core
core = vs.core
# Import scripts folder
scriptPath = 'C:/Program Files/Hybrid/64bit/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))
# Loading Plugins
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vs-mlrt/vstrt.dll")
import site
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/Support/akarin.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/MiscFilter/MiscFilters/MiscFilters.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/vslsmashsource.dll")
# Import scripts
from importlib.machinery import SourceFileLoader
vsmlrt = SourceFileLoader('vsmlrt', 'C:/Program Files/Hybrid/64bit/vs-mlrt/vsmlrt.py').load_module()
# source: 'C:\Users\Gaming-Tower\Videos\test.mp4'
# current color space: YUV420P8, bit depth: 8, resolution: 1280x720, fps: 23.976, color matrix: 709, yuv luminance scale: limited, scanorder: progressive
# Loading C:\Users\Gaming-Tower\Videos\test.mp4 using LWLibavSource
clip = core.lsmas.LWLibavSource(source="C:/Users/Gaming-Tower/Videos/test.mp4", format="YUV420P8", stream_index=0, cache=0, fpsnum=24000, fpsden=1001, prefer_hw=0)
# Setting detected color matrix (709).
clip = core.std.SetFrameProps(clip, _Matrix=1)
# Setting color transfer info (709), when it is not set
clip = clip if not core.text.FrameProps(clip,'_Transfer') else core.std.SetFrameProps(clip, _Transfer=1)
# Setting color primaries info (), when it is not set
clip = clip if not core.text.FrameProps(clip,'_Primaries') else core.std.SetFrameProps(clip, _Primaries=1)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# making sure frame rate is set to 23.976
clip = core.std.AssumeFPS(clip=clip, fpsnum=24000, fpsden=1001)
clip = core.std.SetFrameProp(clip=clip, prop="_FieldBased", intval=0) # progressive
from vsmlrt import Backend
clip = core.misc.SCDetect(clip=clip,threshold=0.150)
# adjusting color space from YUV420P8 to RGBS for VsRIFEmlrt
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="709", range_s="limited")
# adjusting frame count&rate with RIFE (mlrt)
clip = vsmlrt.RIFE(clip, model=46, backend=Backend.TRT(fp16=True, device_id=0,use_cuda_graph=False, num_streams=1, workspace=1 << 30)) # new fps: 47.952
# adjusting output color from: RGBS to YUV420P10 for NVEncModel
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, matrix_s="709", range_s="limited", dither_type="error_diffusion")
# set output frame rate to 47.952fps (progressive)
clip = core.std.AssumeFPS(clip=clip, fpsnum=48000, fpsden=1001)
# Output
clip.set_output()

1st glance, the script seems okay to me.
Does the Vapoursynth preview work for you?

-> Got it, RIFEmlrt requires mod32 content, I'll adjust Hybrid to take this into account. (the test files I usually use are mod32.)

Cu Selur

[Platos]

So the Vapoursynth video-preview does not work. But it actually also did never work with vsmlrt-upscaling (btw).

The Script-preview does work.

And here is a debug when i use x264 insteat of NVEnc:

But what do i have to do now or what do you need? Do you want the script-preview ?

[Selur]

But it actually also did never work with vsmlrt-upscaling (btw).

would have been nice to know to earlier,....

The mod32 requirement will not change if you change the encoder,...

[Platos]

You helped me so much, i did not want to bother you with more things, sorry.

The Upscaling worked, it was only the preview, i never used. So i did not want to bother you with such a small feature.

[Selur]

If the Vapoursynth Preview does not work, encoding should not work either.
-> create a debug output from something where vsmlrt resizing fails, and I can look at it.

[Platos]

If the Vapoursynth Preview does not work, encoding should not work either.
-> create a debug output from something where vsmlrt resizing fails, and I can look at it.

But the resizing worked always fine. It was only the preview.

Edit, but now it works. When i try now, it works with the preview in upsclaling. Maybe this was in an earlier release.
So forget that. But i swear i had the exactly same failure a week ago or so.

sorry for confusing. I really swear id always didnt work a week before or so.

Edit2: I will try some other upscaling models and look, if then the preview works.

[Platos]

Update: i did now try to use VSMLRT TenorRT with Multi-Model (LD-Anime COmpact and Futsuu Anime Compact). Then i tried a preview. It came a command-window and after it came the same window, like on RIFE VSMLRT. Then i tried to reproduce it and make a Debug-File, but it did not give me the same error again (so then it worked).

So i remember right, i had some errors in Vapoursynt Preview with VSMLRT TensorRT Upscaling. But atm im not able to reproduce it (and as i said before, the upscaling did work, just the preview not).

I will try to reproduce it.

[Platos]

I have it now:

If i use a new model in VSLMRT TensorRT Upscaling and press on previev, it opens up a CMD window. But it's just black and nothing happens for 2 minutes. And that's why i always clicked it away, because i thought it is not working. When i then close the CMD Window, then it opened this Error, like in RIFE from before.

But i saw now, that my GPU is in 100% usage, when the CMD Windows open up and after 2 minutes or so it open up a normal preview.

So it was my fault. I thought it's not working, but i had to wait 2 minutes ( i mean, i did not thought i have to wait 2 minutes in front of a empty cmd-window).

So, it actually would have always worked i guess, but you have to wait some minutes, to use the prview and don't get confused by a black, empty CMD-Window and nothing happens for 2 minutes.

Sorry for that.

(But to come back to RIFE: This one does really not work. Because it gives me directly the error without a cmd-window, where i have to wait. So RIFE is really not working. I tried again.)

[Selur]

The command window appears while the .engine file is created for vstrt.
Hybrid will show something like:

Running TensorRT.trtexec for .engine creation,..
Vapoursynth preview error: 2023-19:00:
Vapoursynth preview error: 08] [W] Could not read timing cache from: F:/Hybrid/64bit/onnx_models/1x_Alias_200000_G.onnx.1440x1088_fp16_workspace1073741824_trt-8501_cudnn_I-fp32_O-fp32_NVIDIA-GeForce-RTX-4080_d3d7bd0.engine.cache. A new timing cache will be generated and written.

when the .engine creation starts.
Depending on the resolution (and your gpu), this can take quite some time. (worst thing I encountered so far, is that the creation of the .engine file took ~30min for HD content)


For the future: if you find a bug, report it properly so I at least know about it.
Something like "oh,... and xy doesn't work for ages,.." really just pisses me off and convinces me that xy might be a feature that should be removed.

Send you a new link to a dev version which makes sure both vsmlrt and rife mlrt both get mod32 content.

Cu Selur

[Platos]

Ok, sorry man. I did not want to bother you with more and more questions and things. Really sorry.

But it was my fault on the upscaling-thing, as i found out. Sorry. It worked actually, so i just did not wait long enough.

I will now report everything i find.

i try the new dev version.

sorry

[Selur]

Okay, I will then remove the mod32 requirement for vlsmlrt in the future.

[Platos]

It works now perfectly in the first tests.

With 1 Steam i have 50% GPU und 60% VPU Usage (Encoder-Unit). Then i get ~185 FPS (in normal RIFE i get ~55FPS with 2 GPU Threads)
With 2 Streams i have 80%GPU and 80% VPU Usage. Then i get ~245FPS
With 3 Streams i have 90-95% GPU and 85% VPU Usage. Then i get 255FPS.

So it's very fast! I will now interpolate a Video and compare with normal RIFE. But im sure it will look exactly same.

Thank you very much Selur for your hard work!

And im really so sorry i did not tell you from this before (which was my fault and not the programms issue).

Thank you very much!

[Selur]

Once the .engine file is created it speeds up the processing quite a bit, but like I wrote the creation of the .engine file can take ages.

[Platos]

Yes, at the beginning it took a bit of time for that. But the numbers from above are after creating the .engine file.

But on Upscaling i never had again a waiting-time for already used models and resolutions. I guess, they are saved now longer or so ? In "C:\Program Files\Hybrid\64bit\onnx_models" there are still some .engine files from models i did not use for a longer time. So they are now saved longer, which i like personally.

Just want to tell you, that now i have actually never a waiting-time for .eninge-file in Upscaling with VSMLRT. with my favorite models. Probably i just have used all Resolutions i use so far.

RIFE VSMLRT works also very good. I dont have to wait, when i already have used this configuration.

And i want to say you again thank you very much for this software !

[Platos]

When i want to interpolate a 24/25 FPS Video to 60FPS with mlrtRIFE, can i just use a 3x multiplier and then go to "Filter->Speed Change" and use "scale output frame rate" with 60FPS ?

Will it then produce a 75FPS Video (with mlrtRIFE) and after that throw out some Frames to reach 60FPS (that's what i want). Or does that not work together (and/or it will not use RIFE anymore) ?

[Selur]

When i want to interpolate a 24/25 FPS Video to 60FPS with mlrtRIFE, can i just use a 3x multiplier and then go to "Filter->Speed Change" and use "scale output frame rate" with 60FPS ?

That should work, alternatively you could use one of the methods under Frame->Reduction and adjust the filter order so that the reduction is applied after the interpolation.
As a general side note: when using RIFE check whether the scene detection threshold default 0.15 is 'okay' for your content.

Will it then produce a 75FPS Video (with mlrtRIFE) and after that throw out some Frames to reach 60FPS (that's what i want). Or does that not work together (and/or it will not use RIFE anymore) ?

It should work fine together, since "scale output frame rate" is applied later in the filter queue. (see the Vapoursynth Script Preview, to see whats happening.)

Cu Selur