Hybrid(Windows/Linux/Mac): Input -> x264/x265/Xvid/VP8/VP9

[Platos]

Ahh, now it works. Sorry for the problem. I did not now (or forgot), that i have to deinstall hybrid before i use a new dev.

But i have another problem ^^

When i use a fp16 model wit vsmlrt and TensorRT RTX Backend, i got an error. When i use TensorRT, i do not get an error. I tried it with 32Bit checkbox enabled and disabled.

Failed to evaluate the script:
Python exception: module 'onnx.helper' has no attribute 'float32_to_bfloat16'

Traceback (most recent call last):
File "C:/Program Files/Hybrid/64bit/vs-mlrt/vsmlrt.py", line 2415, in tensorrt_rtx
model = convert_float_to_float16(model, keep_io_types=not fp16_io)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\program files\hybrid\64bit\vapoursynth\Lib\site-packages\onnxconverter_common\float16.py", line 220, in convert_float_to_float16
raise ValueError(
ValueError: The model is already converted to float16, if convert again, the model might be wrong. 
If you are sure to convert again, please set check_fp16_ready=False.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "vapoursynth.pyx", line 3524, in vapoursynth._vpy_evaluate
File "vapoursynth.pyx", line 3525, in vapoursynth._vpy_evaluate
File "C:\Users\Gaming-Tower\Documents\Hybrid\Tempfolder\tempPreviewVapoursynthFile17_40_23_932.vpy", line 55, in 
clip = vsmlrt.inference([clip],network_path="C:/Users/Gaming-Tower/Videos/$$$AI-Models$$$/SISR Models/[StinkyBread]/2x-AnimeFilm/2x_animefilm_light_161k_fp16_dynamic.onnx", backend=Backend.TRT_RTX(fp16=True,device_id=0,num_streams=2,verbose=True,use_cuda_graph=False,workspace=1073741824,builder_optimization_level=3,engine_folder="C:/Users/Gaming-Tower/Documents/Hybrid/Engine-Files")) # 3840x2160
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:/Program Files/Hybrid/64bit/vs-mlrt/vsmlrt.py", line 3113, in inference
return inference_with_fallback(
^^^^^^^^^^^^^^^^^^^^^^^^
File "C:/Program Files/Hybrid/64bit/vs-mlrt/vsmlrt.py", line 3087, in inference_with_fallback
raise e
File "C:/Program Files/Hybrid/64bit/vs-mlrt/vsmlrt.py", line 3064, in inference_with_fallback
ret = _inference(
^^^^^^^^^^^
File "C:/Program Files/Hybrid/64bit/vs-mlrt/vsmlrt.py", line 2977, in _inference
engine_path = tensorrt_rtx(
^^^^^^^^^^^^^
File "C:/Program Files/Hybrid/64bit/vs-mlrt/vsmlrt.py", line 2418, in tensorrt_rtx
from modelopt.onnx.autocast import convert_to_f16, configure_logging
File "c:\program files\hybrid\64bit\vapoursynth\Lib\site-packages\modelopt\onnx\__init__.py", line 23, in 
from . import quantization
File "c:\program files\hybrid\64bit\vapoursynth\Lib\site-packages\modelopt\onnx\quantization\__init__.py", line 19, in 
from .int4 import quantize as quantize_int4
File "c:\program files\hybrid\64bit\vapoursynth\Lib\site-packages\modelopt\onnx\quantization\int4.py", line 30, in 
import onnx_graphsurgeon as gs
File "c:\program files\hybrid\64bit\vapoursynth\Lib\site-packages\onnx_graphsurgeon\__init__.py", line 1, in 
from onnx_graphsurgeon.exporters.onnx_exporter import export_onnx
File "c:\program files\hybrid\64bit\vapoursynth\Lib\site-packages\onnx_graphsurgeon\exporters\onnx_exporter.py", line 134, in 
np.uint16, onnx.helper.float32_to_bfloat16
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'onnx.helper' has no attribute 'float32_to_bfloat16'

This is the error then. Do you need more information?
Is this the same installation-error? Should i completly deinstall hybrid, delete all hybrid folders and reinstall ? I did now "only" a normal deinstallation with the normal deinstaller.

[Selur]

RTX sometimes works with vs-mlrt and sometimes it does not, form what I gather it's a problem of the onnx and torch versions that are used.
The vs-mlrt addon in Hybrid uses the v15.16 release which currently is the latest version.
=> nothing I can do about it, you report it over at https://github.com/AmusementClub/vs-mlrt/issues and maybe they can fix it.

Cu Selur

[Platos]

Ok, i did open a issue there. Thanks

[Platos]

When i use MultiModel in vsmlrt and one of them is a VSR model, i get a crash. When i use only one model and set corret the "MultiInput" number to 5, it works. But when i use multimodel and the other model is not a VSR model, i get a crash. I guess this is because then hybrid thinks the second model also uses 5 frames at once, right ? Im pretty sure (Because it works in Vapourkit)

[Selur]

You can use the FilterQueue to have multiple model calls with full control over the settings and order. Which is why Other->vsMLRT has no 'Multi' option.
The Multi-option under Resizer->VSMLRT is meant to chain multiple models that can be called with the same parameters.

I guess this is because then hybrid thinks the second model also uses 5 frames at once, right ?

No need to guess.
Hybrid allows you to see the generated script, so it's easy to see what is happening.

Because it works in Vapourkit

¯\_(ツ)_/¯
Never used it, but if it does what you want, simply use that instead of Hybrid.

Cu Selur

[Platos]

Ahh, ok thanks. Then i use "other", thanks. I did not know that vsmlrt is also there.

And no, i like Hybrid more, i wan't to use hybrid

thank you

Edit: How is that done with "other", in terms of the resize? When i use on "crop/resize" double the resolution and then usw first a 1x model and then a 2x model, does it then resize already with the 1x model? Or when/how do i tell hybrid, when it does the actual upscale with double resolution ? Because on frame-> Resize i can see the resolution which is applied, but on other i can not see it. Do you know how that works ?

[Selur]

You can always look at the generated script in the Vapoursynth Script View to see what Hybrid does.
Together with the Gimmick-options (Filtering->Vapousynth->Misc->UI->Show 'Gimmick'-controls, you can set for the filters whether the source should be resized before the filter is applied,...
If you enable all 'hidden'-elements there are a lot of otherwise hidden option available.

Combined with the filter queue you can do a lot of stuff,... alternatively you could also write a custom code in custom section,...
=> there are tons of options, you just have to look around an experiment a bit and use the Vapoursynth Script View to see what happens. (I tried to add a lot of comments inside the scripts so that they are easier to understand, but they require a basic understanding and motivation,..)

Cu Selur

[Platos]

Ok, thank you. I will have a look.

I did now find out, that using DAAMod AntiAliasing before Upscaling makes the output looks less Aliasied (which is clear) but also the Upscaling-model can upscale "better" (it looks like a higher upscale). At least on my (Comic) source now. I will need to test on more sources.

[Selur]

Always depends on the source, sometimes even applying AA before deinterlacing on separated fields can make sense,...

[Platos]

Question: There are different Resizermethodeds:

Scenario: I want to upscale with a 2x model with vsmlrt and then resize it to 1440p.

My plan was to resize with spline36 or DPID or Bicbuic (so all three and then compare).

But: On the crop/resize tab there is "only" spline and bicubic, no DPID and also no spline 36. I wonder, on WHICH resize the resize-methods from crop/resize get applied.

Edit: I found it out:

- When i'am right, the resize-method used in crop/resize is only used, if you do not use the resizer in Filtering->Vapoursynth->Frame->Resize.
- When you use The vsmlrt resizer and you set the same resolution in crop/resize as the source, then no resize (with spline or sth like that) is applyied. Only Upscaling.
- When you use higher or lower resolution in crop/resize, then you use the resize-method from the Filtering->Vapoursynth->Frame->Resize Section where you have to chose the Resizer in the drop down menue.
(That also means, if you do only want to resize in an encode, you can only use the methods on the crop/resize tabs, unless you do a custom script, which also works. With a code like this: "clip = core.resize.Spline36(clip, width=2560, height=1440)")

-> I like spline36 most for 2160p to 1440p downscaling.

At least that's my interpretation, when reading the vapoursynth-code-preview.

[Selur]

- When i'am right, the resize-method used in crop/resize is only used, if you do not use the resizer in Filtering->Vapoursynth->Frame->Resize.

Correct

- When you use The vsmlrt resizer and you set the same resolution in crop/resize as the source, then no resize (with spline or sth like that) is applyied. Only Upscaling.

When you use VSMLRT as Resizer under Vapoursynth->Frame->Resize->Resizer and don't change the resize resolution, Hybrid will undo any resizing done by the used model to match the resize resolution.
In example:

# resizing using VSMLRT, target: 640x352
clip = vsmlrt.inference([clip],network_path="F:/Hybrid/64bit/onnx_models/4x_BSRGAN.onnx", backend=Backend.TRT(fp16=True,device_id=0,bf16=False,num_streams=1,verbose=True,use_cuda_graph=False,workspace=1073741824,builder_optimization_level=3,engine_folder="J:/TRT")) # 2560x1408
# resizing 2560x1408 to 640x352
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS)
clip = core.fmtc.resample(clip, w=640, h=352, kernel="spline64", interlaced=False, interlacedd=False)

the resizer here is the one you select as 'Resizer' unter 'VSLMRT'.

- When you use higher or lower resolution in crop/resize, then you use the resize-method from the Filtering->Vapoursynth->Frame->Resize Section where you have to chose the Resizer in the drop down menue.

You can chose a specific resizer to adjust the output resolution caused by VSMLRT to your desired target resolution,...

Resizer:
Choose the resizer which will be used in case VLMLRT does not perfectly hit the desired resolution.

is what the tool-tip says, which seem accurate.

(That also means, if you do only want to resize in an encode, you can only use the methods on the crop/resize tabs, unless you do a custom script, which also works. With a code like this: "clip = core.resize.Spline36(clip, width=2560, height=1440)")

Since your assumtions are wrong, so is your conclusion, aside from that if you don't overwrite the Resizer unter Crop/Resize through Avisynth/Vapoursynth->Frame->Resize->Resizer the selected resizer will be used.

At least that's my interpretation, when reading the vapoursynth-code-preview.

=> you might want to try that a few more times, to really understand what is happening.

Question ...

your whole post has no question in it,...

Cu Selur

[Platos]

The question was, when does which resize-setting get applied, but i basically answered it myself. But i don't understand, what in my interpretation was wrong: When i want only to resize with spline36, nothing else (no vsmlrt), then i have to use the crop/resize part, and there i can't use spline36 (it is not there in the dropdown). And so that's why i guess i have to use a custom script to do that, because spline 36 is only available, when i use Vapoursynth->Frame->Resize->Resizer, but then i have to use the resizer (vsmlrt or another of them), which i don't want, when i only want to use a spline36 downsize/resize. Or can i use VSMLRT and then don0t choose a ai-model (let it empty) and then resize?

Anyway: I think i found an error/bug in hybrid (not sure, if it is a problem of hybrid): When i encode with x265 main10, i always get a green-tone. when i use main (8bit), then it is no problem. I do use nothing else. Starting hybrid, use x265 crf 18, main 10 and then the output has a green tone. It does not happen, when i set it to "main". Why is that? Is this a problem of hybrid, the encoder or the player? Maybe the player think, when it is main10 (1obit), it must be HDR. WOuld you reccomend to never use main10, if it is not 10 bit source (native HDR or so) ? Or can i fix this somehow, that it wil work on normal player like MPC-HC oder VLC or so ? Or is it generally a bad idea to use main10 if the source is 8bit ?

[Selur]

about your green tone: no clue, without details
about the resizer: Crop/Resize uses Resizers which are available in mencoder/mplayer/avisynth/vapoursynth, Filtering->Avisynth/Vapoursynth->Frame->Resizer does offer to overwrite these with specific resizers. i.e. fmtconv which uses also offers spline36.
It is correct, that Hybrid does not offer a way to use the resizers from https://www.vapoursynth.com/doc/functions/video/resize.html directly. I could probably write code to add a 'Vapoursynth native'-resizer under Filtering->Vapoursynth->Frame->Resizer, but I have no plans for it atm., I simply do not see any gain from doing it.

[Platos]

About the old Answer: I guess, the green tone is just how x265 works. When the input is 10bit, main10 works fine. if the input is 8bit, main10 makes greentone. I guess it is not hybrid.

New Error:

Vsmlrt in vapoursynth-->other-->vsmlrt does not work. I believe, the problem is, that it does not really use the settings, i say. i can uncheck 32 bit but it is still "output_format=0" (while on vapoursynth-->frame-->resize-->resizer-->vsmlrt it works and there i have "output_format=1" in the vapoursynth script). Because when i use "32bit" it works. When i un-thick the 32bit box it crashes. But it works on the frame-tab with both versions.

Here is the crash:

2026-06-26 18:10:17.916
Plugin C:/Program Files/Hybrid/64bit/vs-mlrt/vstrt.dll is using API3 which is deprecated and will be removed shortly.
Plugin C:/Program Files/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/LSMASHSource.dll is using API3 which is deprecated and will be removed shortly.
2026-06-26 18:10:18.122
Failed to evaluate the script:
Python exception: input format must be equal to output format

Traceback (most recent call last):
File "vapoursynth.pyx", line 3623, in vapoursynth._vpy_evaluate
File "vapoursynth.pyx", line 3624, in vapoursynth._vpy_evaluate
File "C:\Users\Gaming-Tower\Documents\Hybrid\Tempfolder\tempPreviewVapoursynthFile18_10_17_479.vpy", line 53, in 
clip = vsmlrt.inference([clip],network_path="C:/Program Files/Hybrid/64bit/onnx_models/2x_Ani4Kv2_G6i2_Compact_107500_fp32.onnx",backend=Backend.TRT(fp16=True,device_id=0,bf16=False,num_streams=1,verbose=True,use_cuda_graph=False,workspace=1073741824,builder_optimization_level=3,engine_folder="C:/Users/Gaming-Tower/Documents/Hybrid/Engine-Files",output_format=0)) # 2880x2160
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:/Program Files/Hybrid/64bit/vs-mlrt/vsmlrt.py", line 3137, in inference
return inference_with_fallback(
^^^^^^^^^^^^^^^^^^^^^^^^
File "C:/Program Files/Hybrid/64bit/vs-mlrt/vsmlrt.py", line 3110, in inference_with_fallback
raise e
File "C:/Program Files/Hybrid/64bit/vs-mlrt/vsmlrt.py", line 3087, in inference_with_fallback
ret = _inference(
^^^^^^^^^^^
File "C:/Program Files/Hybrid/64bit/vs-mlrt/vsmlrt.py", line 2904, in _inference
engine_path = trtexec(
^^^^^^^^
File "C:/Program Files/Hybrid/64bit/vs-mlrt/vsmlrt.py", line 2087, in trtexec
convert_model(
File "C:/Program Files/Hybrid/64bit/vs-mlrt/vsmlrt.py", line 1958, in convert_model
raise ValueError("input format must be equal to output format")
ValueError: input format must be equal to output format

And here the script from the "others" tab (the crashing one):

# Imports
import logging
import site
import sys
import os
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# Import scripts folder
scriptPath = 'C:/Program Files/Hybrid/64bit/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))
os.environ["CUDA_MODULE_LOADING"] = "LAZY"
# Force logging to std:err
logging.StreamHandler(sys.stderr)
# loading plugins
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/Vapoursynth/Lib/site-packages/vapoursynth/plugins2/vszip/vszip.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vs-mlrt/vstrt.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/Vapoursynth/Lib/site-packages/vapoursynth/plugins2/fmtconv.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/LSMASHSource.dll")
# Import scripts
from importlib.machinery import SourceFileLoader
vsmlrt = SourceFileLoader('vsmlrt', 'C:/Program Files/Hybrid/64bit/vs-mlrt/vsmlrt.py').load_module()
from vsmlrt import Backend
import validate
# Source: 'C:\Filme-Serien-Dokus&Animes\Encoding-Filter_Tests\XXXXX\XXXXX.mkv'
# clip current meta; color space: YUV420P10, bit depth: 10, resolution: 1440x1080, fps: 23.976, color matrix: 709, color primaries: Unspecific, color transfer: Unspecified, yuv luminance scale: limited, scanorder: progressive, full height: true (Source)
# Loading 'C:\XXXXX\XXX.mkvÄ using LWLibavSource
clip = core.lsmas.LWLibavSource(source="C:/XXXX/XXXXX.mkv", format="YUV420P10", stream_index=0, cache=0)
frame = clip.get_frame(0)
# setting color matrix to 709.
clip = core.std.SetFrameProps(clip, _Matrix=vs.MATRIX_BT709)
# setting color transfer (vs.TRANSFER_BT709), if it is not set.
if validate.transferIsInvalid(clip):
  clip = core.std.SetFrameProps(clip=clip, _Transfer=vs.TRANSFER_BT709)
# setting color primaries info (to vs.PRIMARIES_BT709), if it is not set.
if validate.primariesIsInvalid(clip):
  clip = core.std.SetFrameProps(clip=clip, _Primaries=vs.PRIMARIES_BT709)
# setting color range to TV (limited) range.
prop_name = '_Range' if core.core_version.release_major >= 74 else '_ColorRange'
clip = core.std.SetFrameProps(clip=clip, **{prop_name: vs.RANGE_LIMITED})
# making sure frame rate is set to 23.976fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=24000, fpsden=1001)
# making sure the detected scan type is set (detected: progressive)
clip = core.std.SetFrameProps(clip=clip, _FieldBased=vs.FIELD_PROGRESSIVE) # scan type: progressive
# changing range from limited to full range for vsVSMLRTFilter
clip = core.resize.Bicubic(clip, format=vs.YUV420P10, range_in_s="limited", range_s="full")
# setting color range to PC (full) range.
prop_name = '_Range' if core.core_version.release_major >= 74 else '_ColorRange'
clip = core.std.SetFrameProps(clip=clip, **{prop_name: vs.RANGE_FULL})
# adjusting color space from YUV420P10 to RGBH for vsVSMLRTFilter
clip = core.resize.Bicubic(clip=clip, format=vs.RGBH, matrix_in_s="709", range_in_s="full", range_s="full")
# filtering using VSMLRT
clip = vsmlrt.inference([clip],network_path="C:/Program Files/Hybrid/64bit/onnx_models/2x_Ani4Kv2_G6i2_Compact_107500_fp32.onnx",backend=Backend.TRT(fp16=True,device_id=0,bf16=False,num_streams=1,verbose=True,use_cuda_graph=False,workspace=1073741824,builder_optimization_level=3,engine_folder="C:/Users/Gaming-Tower/Documents/Hybrid/Engine-Files",output_format=0)) # 2880x2160
# making sure 0-1 limits are respected
clip = core.vszip.Limiter(clip=clip, min=[0,0,0], max=[1,1,1])
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS)
# Resizing using 10 - bicubic spline
clip = core.fmtc.resample(clip, kernel="spline16", w=1440, h=1080, interlaced=False, interlacedd=False) # resolution 1440x1080
# adjusting output color from RGBS to YUV420P8 for x265Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P8, matrix_s="709", range_in_s="full", range_s="full", dither_type="error_diffusion")
# changing range from full to limited range for matching filter target luma scale
clip = core.resize.Bicubic(clip, format=vs.YUV420P8,range_in_s="full", range_s="limited")
# setting color range to TV (limited) range.
prop_name = '_Range' if core.core_version.release_major >= 74 else '_ColorRange'
clip = core.std.SetFrameProps(clip=clip, **{prop_name: vs.RANGE_LIMITED})
# set output frame rate to 23.976fps (progressive)
clip = core.std.AssumeFPS(clip=clip, fpsnum=24000, fpsden=1001)
# output
clip.set_output()
# clip current meta; color space: YUV420P8, bit depth: 8, resolution: 1440x1080, fps: 23.976, color matrix: 709, color primaries: Unspecific, color transfer: Unspecified, yuv luminance scale: limited, scanorder: progressive, full height: true (Meta)
# script was created by Hybrid 2026.06.25.1

And here the script from the frame-resize tab (the working one):

# Imports
import sys
import os
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# Import scripts folder
scriptPath = 'C:/Program Files/Hybrid/64bit/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))
# loading plugins
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/Vapoursynth/Lib/site-packages/vapoursynth/plugins2/fmtconv.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/LSMASHSource.dll")
# Import scripts
import validate
# Source: 'C:\XXX\XXXX.mkv'
# clip current meta; color space: YUV420P10, bit depth: 10, resolution: 1440x1080, fps: 23.976, color matrix: 709, color primaries: Unspecific, color transfer: Unspecified, yuv luminance scale: limited, scanorder: progressive, full height: true (Source)
# Loading 'C:\XXXX\XXX.mkvÄ using LWLibavSource
clip = core.lsmas.LWLibavSource(source="C:/XXXX/XXXX.mkv", format="YUV420P10", stream_index=0, cache=0)
frame = clip.get_frame(0)
# setting color matrix to 709.
clip = core.std.SetFrameProps(clip, _Matrix=vs.MATRIX_BT709)
# setting color transfer (vs.TRANSFER_BT709), if it is not set.
if validate.transferIsInvalid(clip):
  clip = core.std.SetFrameProps(clip=clip, _Transfer=vs.TRANSFER_BT709)
# setting color primaries info (to vs.PRIMARIES_BT709), if it is not set.
if validate.primariesIsInvalid(clip):
  clip = core.std.SetFrameProps(clip=clip, _Primaries=vs.PRIMARIES_BT709)
# setting color range to TV (limited) range.
prop_name = '_Range' if core.core_version.release_major >= 74 else '_ColorRange'
clip = core.std.SetFrameProps(clip=clip, **{prop_name: vs.RANGE_LIMITED})
# making sure frame rate is set to 23.976fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=24000, fpsden=1001)
# making sure the detected scan type is set (detected: progressive)
clip = core.std.SetFrameProps(clip=clip, _FieldBased=vs.FIELD_PROGRESSIVE) # scan type: progressive
# Not using VSMLRT since no model is set
# adjusting output color from YUV420P10 to YUV420P8 for x265Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P8, dither_type="error_diffusion")
# set output frame rate to 23.976fps (progressive)
clip = core.std.AssumeFPS(clip=clip, fpsnum=24000, fpsden=1001)
# output
clip.set_output()
# clip current meta; color space: YUV420P8, bit depth: 8, resolution: 1440x1080, fps: 23.976, color matrix: 709, color primaries: Unspecific, color transfer: Unspecified, yuv luminance scale: limited, scanorder: progressive, full height: true (Meta)
# script was created by Hybrid 2026.06.25.1

[Selur]

try latest dev and vs-mlrt

[Platos]

This is the latest. vs-mlrt from today and dev from yesterday. i Did first deinstall hybrid and then reinstall it and then unzip the newest vapoursynth and vsmlrt into the 64 bit folder and overwrite.

I see no newer one.

[Selur]

And here the script from the frame-resize tab (the working one):

+

# Not using VSMLRT since no model is set

=> ¯\_(ツ)_/¯

This is the latest. vs-mlrt from today and dev from yesterday.

okay.

Does it work if you replace the vsmlrt.py with: https://raw.githubusercontent.com/AmusementClub/vs-mlrt/refs/heads/trt-latest/scripts/vsmlrt.py

Cu Selur

Ps.: Also make sure to delete old engine and cache files,..

[Platos]

Ah shit, that must be a copy paste failure.

It does not work with the new .py file and here ist the script from it:

# Imports
import logging
import site
import sys
import os
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# Import scripts folder
scriptPath = 'C:/Program Files/Hybrid/64bit/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))
os.environ["CUDA_MODULE_LOADING"] = "LAZY"
# Force logging to std:err
logging.StreamHandler(sys.stderr)
# loading plugins
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/Vapoursynth/Lib/site-packages/vapoursynth/plugins2/vszip/vszip.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vs-mlrt/vstrt.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/Vapoursynth/Lib/site-packages/vapoursynth/plugins2/fmtconv.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/LSMASHSource.dll")
# Import scripts
from importlib.machinery import SourceFileLoader
vsmlrt = SourceFileLoader('vsmlrt', 'C:/Program Files/Hybrid/64bit/vs-mlrt/vsmlrt.py').load_module()
from vsmlrt import Backend
import validate
# Source: 'C:\Users\Gaming-Tower\Videos\test.mkv'
# clip current meta; color space: YUV420P10, bit depth: 10, resolution: 1440x1080, fps: 23.976, color matrix: 709, color primaries: Unspecific, color transfer: Unspecified, yuv luminance scale: limited, scanorder: progressive, full height: true (Source)
# Loading 'C:\Users\Gaming-Tower\Videos\test.mkvÄ using LWLibavSource
clip = core.lsmas.LWLibavSource(source="C:/Users/Gaming-Tower/Videos/test.mkv", format="YUV420P10", stream_index=0, cache=0)
frame = clip.get_frame(0)
# setting color matrix to 709.
clip = core.std.SetFrameProps(clip, _Matrix=vs.MATRIX_BT709)
# setting color transfer (vs.TRANSFER_BT709), if it is not set.
if validate.transferIsInvalid(clip):
  clip = core.std.SetFrameProps(clip=clip, _Transfer=vs.TRANSFER_BT709)
# setting color primaries info (to vs.PRIMARIES_BT709), if it is not set.
if validate.primariesIsInvalid(clip):
  clip = core.std.SetFrameProps(clip=clip, _Primaries=vs.PRIMARIES_BT709)
# setting color range to TV (limited) range.
prop_name = '_Range' if core.core_version.release_major >= 74 else '_ColorRange'
clip = core.std.SetFrameProps(clip=clip, **{prop_name: vs.RANGE_LIMITED})
# making sure frame rate is set to 23.976fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=24000, fpsden=1001)
# making sure the detected scan type is set (detected: progressive)
clip = core.std.SetFrameProps(clip=clip, _FieldBased=vs.FIELD_PROGRESSIVE) # scan type: progressive
# changing range from limited to full range for vsVSMLRTFilter
clip = core.resize.Bicubic(clip, format=vs.YUV420P10, range_in_s="limited", range_s="full")
# setting color range to PC (full) range.
prop_name = '_Range' if core.core_version.release_major >= 74 else '_ColorRange'
clip = core.std.SetFrameProps(clip=clip, **{prop_name: vs.RANGE_FULL})
# adjusting color space from YUV420P10 to RGBH for vsVSMLRTFilter
clip = core.resize.Bicubic(clip=clip, format=vs.RGBH, matrix_in_s="709", range_in_s="full", range_s="full")
# filtering using VSMLRT
clip = vsmlrt.inference([clip],network_path="C:/Program Files/Hybrid/64bit/onnx_models/2x_Ani4Kv2_G6i2_Compact_107500_fp32.onnx",backend=Backend.TRT(fp16=True,device_id=0,bf16=False,num_streams=1,verbose=True,use_cuda_graph=False,workspace=1073741824,builder_optimization_level=3,engine_folder="C:/Users/Gaming-Tower/Documents/Hybrid/Engine-Files",output_format=0)) # 2880x2160
# making sure 0-1 limits are respected
clip = core.vszip.Limiter(clip=clip, min=[0,0,0], max=[1,1,1])
# adjusting output color from RGBH to YUV420P8 for x265Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P8, matrix_s="709", range_in_s="full", range_s="full", dither_type="error_diffusion")
# changing range from full to limited range for matching filter target luma scale
clip = core.resize.Bicubic(clip, format=vs.YUV420P8,range_in_s="full", range_s="limited")
# setting color range to TV (limited) range.
prop_name = '_Range' if core.core_version.release_major >= 74 else '_ColorRange'
clip = core.std.SetFrameProps(clip=clip, **{prop_name: vs.RANGE_LIMITED})
# set output frame rate to 23.976fps (progressive)
clip = core.std.AssumeFPS(clip=clip, fpsnum=24000, fpsden=1001)
# output
clip.set_output()
# clip current meta; color space: YUV420P8, bit depth: 8, resolution: 2880x2160, fps: 23.976, color matrix: 709, color primaries: Unspecific, color transfer: Unspecified, yuv luminance scale: limited, scanorder: progressive, full height: true (Meta)
# script was created by Hybrid 2026.06.25.1

And here now the real code from the frame tab (with the new .py file):

# Imports
import logging
import site
import sys
import os
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# Import scripts folder
scriptPath = 'C:/Program Files/Hybrid/64bit/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))
os.environ["CUDA_MODULE_LOADING"] = "LAZY"
# Force logging to std:err
logging.StreamHandler(sys.stderr)
# loading plugins
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/Vapoursynth/Lib/site-packages/vapoursynth/plugins2/vszip/vszip.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vs-mlrt/vstrt.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/Vapoursynth/Lib/site-packages/vapoursynth/plugins2/fmtconv.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/LSMASHSource.dll")
# Import scripts
from importlib.machinery import SourceFileLoader
vsmlrt = SourceFileLoader('vsmlrt', 'C:/Program Files/Hybrid/64bit/vs-mlrt/vsmlrt.py').load_module()
from vsmlrt import Backend
import validate
# Source: 'C:\Users\Gaming-Tower\Videos\test.mkv'
# clip current meta; color space: YUV420P10, bit depth: 10, resolution: 1440x1080, fps: 23.976, color matrix: 709, color primaries: Unspecific, color transfer: Unspecified, yuv luminance scale: limited, scanorder: progressive, full height: true (Source)
# Loading 'C:\Users\Gaming-Tower\Videos\test.mkvÄ using LWLibavSource
clip = core.lsmas.LWLibavSource(source="C:/Users/Gaming-Tower/Videos/test.mkv", format="YUV420P10", stream_index=0, cache=0)
frame = clip.get_frame(0)
# setting color matrix to 709.
clip = core.std.SetFrameProps(clip, _Matrix=vs.MATRIX_BT709)
# setting color transfer (vs.TRANSFER_BT709), if it is not set.
if validate.transferIsInvalid(clip):
  clip = core.std.SetFrameProps(clip=clip, _Transfer=vs.TRANSFER_BT709)
# setting color primaries info (to vs.PRIMARIES_BT709), if it is not set.
if validate.primariesIsInvalid(clip):
  clip = core.std.SetFrameProps(clip=clip, _Primaries=vs.PRIMARIES_BT709)
# setting color range to TV (limited) range.
prop_name = '_Range' if core.core_version.release_major >= 74 else '_ColorRange'
clip = core.std.SetFrameProps(clip=clip, **{prop_name: vs.RANGE_LIMITED})
# making sure frame rate is set to 23.976fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=24000, fpsden=1001)
# making sure the detected scan type is set (detected: progressive)
clip = core.std.SetFrameProps(clip=clip, _FieldBased=vs.FIELD_PROGRESSIVE) # scan type: progressive
# changing range from limited to full range for vsVSMLRT
clip = core.resize.Bicubic(clip, format=vs.YUV420P10, range_in_s="limited", range_s="full")
# setting color range to PC (full) range.
prop_name = '_Range' if core.core_version.release_major >= 74 else '_ColorRange'
clip = core.std.SetFrameProps(clip=clip, **{prop_name: vs.RANGE_FULL})
# adjusting color space from YUV420P10 to RGBH for vsVSMLRT
clip = core.resize.Bicubic(clip=clip, format=vs.RGBH, matrix_in_s="709", range_in_s="full", range_s="full")
# resizing using VSMLRT, target: 2880x2160
clip = vsmlrt.inference([clip],network_path="C:/Program Files/Hybrid/64bit/onnx_models/2x_Ani4Kv2_G6i2_Compact_107500_fp32.onnx",backend=Backend.TRT(fp16=True,device_id=0,bf16=False,num_streams=2,output_format=1,verbose=True,use_cuda_graph=False,workspace=1073741824,builder_optimization_level=3,engine_folder="C:/Users/Gaming-Tower/Documents/Hybrid/Engine-Files")) # 2880x2160
# making sure 0-1 limits are respected
clip = core.vszip.Limiter(clip=clip, min=[0,0,0], max=[1,1,1])
# adjusting output color from RGBH to YUV420P8 for x265Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P8, matrix_s="709", range_in_s="full", range_s="full", dither_type="error_diffusion")
# changing range from full to limited range for matching filter target luma scale
clip = core.resize.Bicubic(clip, format=vs.YUV420P8,range_in_s="full", range_s="limited")
# setting color range to TV (limited) range.
prop_name = '_Range' if core.core_version.release_major >= 74 else '_ColorRange'
clip = core.std.SetFrameProps(clip=clip, **{prop_name: vs.RANGE_LIMITED})
# set output frame rate to 23.976fps (progressive)
clip = core.std.AssumeFPS(clip=clip, fpsnum=24000, fpsden=1001)
# output
clip.set_output()
# clip current meta; color space: YUV420P8, bit depth: 8, resolution: 2880x2160, fps: 23.976, color matrix: 709, color primaries: Unspecific, color transfer: Unspecified, yuv luminance scale: limited, scanorder: progressive, full height: true (Meta)
# script was created by Hybrid 2026.06.25.1

Edit: With the ne .py file vsmlrt from the frame-tab does also not work anymore. it works, when i use the old .py file:

2026-06-26 19:35:20.761
Plugin C:/Program Files/Hybrid/64bit/vs-mlrt/vstrt.dll is using API3 which is deprecated and will be removed shortly.
Plugin C:/Program Files/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/LSMASHSource.dll is using API3 which is deprecated and will be removed shortly.
2026-06-26 19:35:21.221
Failed to evaluate the script:
Python exception: trtexec execution fails, log has been written to C:\Users\GAMING~1\AppData\Local\Temp\trtexec_260626_193520.log

Traceback (most recent call last):
File "vapoursynth.pyx", line 3623, in vapoursynth._vpy_evaluate
File "vapoursynth.pyx", line 3624, in vapoursynth._vpy_evaluate
File "C:\Users\Gaming-Tower\Documents\Hybrid\Tempfolder\tempPreviewVapoursynthFile19_35_20_310.vpy", line 53, in 
clip = vsmlrt.inference([clip],network_path="C:/Program Files/Hybrid/64bit/onnx_models/2x_Ani4Kv2_G6i2_Compact_107500_fp32.onnx",backend=Backend.TRT(fp16=True,device_id=0,bf16=False,num_streams=2,output_format=1,verbose=True,use_cuda_graph=False,workspace=1073741824,builder_optimization_level=3,engine_folder="C:/Users/Gaming-Tower/Documents/Hybrid/Engine-Files")) # 2880x2160
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:/Program Files/Hybrid/64bit/vs-mlrt/vsmlrt.py", line 2437, in inference
return inference_with_fallback(
^^^^^^^^^^^^^^^^^^^^^^^^
File "C:/Program Files/Hybrid/64bit/vs-mlrt/vsmlrt.py", line 2413, in inference_with_fallback
raise e
File "C:/Program Files/Hybrid/64bit/vs-mlrt/vsmlrt.py", line 2392, in inference_with_fallback
return _inference(
^^^^^^^^^^^
File "C:/Program Files/Hybrid/64bit/vs-mlrt/vsmlrt.py", line 2287, in _inference
engine_path = trtexec(
^^^^^^^^
File "C:/Program Files/Hybrid/64bit/vs-mlrt/vsmlrt.py", line 1935, in trtexec
raise RuntimeError(f"trtexec execution fails, log has been written to {log_filename}")
RuntimeError: trtexec execution fails, log has been written to C:\Users\GAMING~1\AppData\Local\Temp\trtexec_260626_193520.log

And this is the error from the "others" tab with the new .py file:

2026-06-26 19:37:11.867
Plugin C:/Program Files/Hybrid/64bit/vs-mlrt/vstrt.dll is using API3 which is deprecated and will be removed shortly.
Plugin C:/Program Files/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/LSMASHSource.dll is using API3 which is deprecated and will be removed shortly.
2026-06-26 19:37:12.303
Failed to evaluate the script:
Python exception: trtexec execution fails, log has been written to C:\Users\GAMING~1\AppData\Local\Temp\trtexec_260626_193712.log

Traceback (most recent call last):
File "vapoursynth.pyx", line 3623, in vapoursynth._vpy_evaluate
File "vapoursynth.pyx", line 3624, in vapoursynth._vpy_evaluate
File "C:\Users\Gaming-Tower\Documents\Hybrid\Tempfolder\tempPreviewVapoursynthFile19_37_11_416.vpy", line 53, in 
clip = vsmlrt.inference([clip],network_path="C:/Program Files/Hybrid/64bit/onnx_models/2x_Ani4Kv2_G6i2_Compact_107500_fp32.onnx",backend=Backend.TRT(fp16=True,device_id=0,bf16=False,num_streams=2,verbose=True,use_cuda_graph=False,workspace=1073741824,builder_optimization_level=3,engine_folder="C:/Users/Gaming-Tower/Documents/Hybrid/Engine-Files",output_format=0)) # 2880x2160
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:/Program Files/Hybrid/64bit/vs-mlrt/vsmlrt.py", line 2437, in inference
return inference_with_fallback(
^^^^^^^^^^^^^^^^^^^^^^^^
File "C:/Program Files/Hybrid/64bit/vs-mlrt/vsmlrt.py", line 2413, in inference_with_fallback
raise e
File "C:/Program Files/Hybrid/64bit/vs-mlrt/vsmlrt.py", line 2392, in inference_with_fallback
return _inference(
^^^^^^^^^^^
File "C:/Program Files/Hybrid/64bit/vs-mlrt/vsmlrt.py", line 2287, in _inference
engine_path = trtexec(
^^^^^^^^
File "C:/Program Files/Hybrid/64bit/vs-mlrt/vsmlrt.py", line 1935, in trtexec
raise RuntimeError(f"trtexec execution fails, log has been written to {log_filename}")
RuntimeError: trtexec execution fails, log has been written to C:\Users\GAMING~1\AppData\Local\Temp\trtexec_260626_193712.log

With this one it also does not work in the "others" tab, when i use 32 bit (before it worked with 32bit, but not with unchecked 32-bit box).

Edit2: I also did empty temp folder and .engine folder (but i don't know where a cache folder is. i think you mean temp folder?)

[Selur]

inside the engine-folder should be .engine and .cache files

Python exception: trtexec execution fails, log has been written to C:\Users\GAMING~1\AppData\Local\Temp\trtexec_260626_193712.log

and what does the log file say?


Strangly for me the new vsmlrt does work

# Imports
import logging
import site
import sys
import os
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# Limit frame cache to 48449MB
core.max_cache_size = 48449
# Import scripts folder
scriptPath = 'F:/Hybrid/64bit/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))
os.environ["CUDA_MODULE_LOADING"] = "LAZY"
# Force logging to std:err
logging.StreamHandler(sys.stderr)
# loading plugins
core.std.LoadPlugin(path="F:/Hybrid/64bit/Vapoursynth/Lib/site-packages/vapoursynth/plugins2/vszip/vszip.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/vs-mlrt/vstrt.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/Vapoursynth/Lib/site-packages/vapoursynth/plugins2/fmtconv.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/Vapoursynth/Lib/site-packages/vapoursynth/plugins2/libbestsource.dll")
# Import scripts
from importlib.machinery import SourceFileLoader
vsmlrt = SourceFileLoader('vsmlrt', 'F:/Hybrid/64bit/vs-mlrt/vsmlrt.py').load_module()
from vsmlrt import Backend
import validate
# Source: 'G:\TestClips&Co\files\test.avi'
# clip current meta; color space: YUV420P8, bit depth: 8, resolution: 640x352, fps: 25, color matrix: 470bg, color primaries: Unspecific, color transfer: Unspecified, yuv luminance scale: limited, scanorder: progressive, full height: true (Source)
# Loading 'G:\TestClips&Co\files\test.avi' using BestSource
clip = core.bs.VideoSource(source="G:/TestClips&Co/files/test.avi", cachepath="J:/tmp/test_bestSource", track=0, hwdevice="opencl")
frame = clip.get_frame(0)
# setting color matrix to 470bg.
clip = core.std.SetFrameProps(clip, _Matrix=vs.MATRIX_BT470_BG)
# setting color transfer (vs.TRANSFER_BT601), if it is not set.
if validate.transferIsInvalid(clip):
  clip = core.std.SetFrameProps(clip=clip, _Transfer=vs.TRANSFER_BT601)
# setting color primaries info (to vs.PRIMARIES_BT470_BG), if it is not set.
if validate.primariesIsInvalid(clip):
  clip = core.std.SetFrameProps(clip=clip, _Primaries=vs.PRIMARIES_BT470_BG)
# setting color range to TV (limited) range.
prop_name = '_Range' if core.core_version.release_major >= 74 else '_ColorRange'
clip = core.std.SetFrameProps(clip=clip, **{prop_name: vs.RANGE_LIMITED})
# making sure frame rate is set to 25fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# making sure the detected scan type is set (detected: progressive)
clip = core.std.SetFrameProps(clip=clip, _FieldBased=vs.FIELD_PROGRESSIVE) # scan type: progressive
# changing range from limited to full range for vsVSMLRTFilter
clip = core.resize.Bicubic(clip, format=vs.YUV420P8, range_in_s="limited", range_s="full")
# setting color range to PC (full) range.
prop_name = '_Range' if core.core_version.release_major >= 74 else '_ColorRange'
clip = core.std.SetFrameProps(clip=clip, **{prop_name: vs.RANGE_FULL})
# adjusting color space from YUV420P8 to RGBH for vsVSMLRTFilter
clip = core.resize.Bicubic(clip=clip, format=vs.RGBH, matrix_in_s="470bg", range_in_s="full", range_s="full")
# filtering using VSMLRT
clip = vsmlrt.inference([clip],network_path="F:/Hybrid/64bit/onnx_models/1x_artifacts_bc1_free_alsa.onnx_fp16.onnx",backend=Backend.TRT(fp16=True,device_id=0,bf16=False,num_streams=1,verbose=True,use_cuda_graph=False,workspace=1073741824,builder_optimization_level=3,engine_folder="J:/TRT",output_format=1)) # 640x352
# making sure 0-1 limits are respected
clip = core.vszip.Limiter(clip=clip, min=[0,0,0], max=[1,1,1])
# adjusting output color from RGBH to YUV420P8 for x265Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P8, matrix_s="470bg", range_in_s="full", range_s="full", dither_type="error_diffusion")
# changing range from full to limited range for matching filter target luma scale
clip = core.resize.Bicubic(clip, format=vs.YUV420P8,range_in_s="full", range_s="limited")
# setting color range to TV (limited) range.
prop_name = '_Range' if core.core_version.release_major >= 74 else '_ColorRange'
clip = core.std.SetFrameProps(clip=clip, **{prop_name: vs.RANGE_LIMITED})
# set output frame rate to 25fps (progressive)
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# output
clip.set_output()
# clip current meta; color space: YUV420P8, bit depth: 8, resolution: 640x352, fps: 25, color matrix: 470bg, color primaries: Unspecific, color transfer: Unspecified, yuv luminance scale: limited, scanorder: progressive, full height: true (Meta)
# script was created by Hybrid 2026.06.25.1

Cu Selur

Ps.: using nvidia 610.62 studio drivers (newer vsmlrt use newer TensorRT 11.1 and CUDA 13.3.0, so up-to-date drivers might be needed)
https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html + https://docs.nvidia.com/deeplearning/tensorrt/latest/getting-started/support-matrix.ht...support-matrix

[Platos]

And it is wanted, that the new vsmlrt.py you uploaded has 87k letters while the old had 115k ?

And i did upload the file.

Edit: i try new gpu driver, 1 sec..

[Selur]

vsmlrt __version__ = "3.23.2" is the latest official onem you can also try whether older versions work
https://github.com/AmusementClub/vs-mlrt/commits/master/scripts/vsmlrt.py

"Invalid TensorFormat fp16:chw" -> check that your engine folder is cleared, that you us the latest studio drivers and vsmlrt.py

[Platos]

OK, i did install newest studio driver and restart computer and it does still not work. It does throw the same error with the new .py file and also with the .old (same as the old before the new driver)
I dont know, what i do wrong.

Do i have to delete anything else than the engine-and temp folder-files ? and vapoursynth and vsmlrt .zip was correct to unzip over the old one (copy and overwrite basically) ? or should i delete the old one first and then copy fresh ?

Edit: It did not work before with an older dev and older vapoursynth and older vsmlrt. That's why i updated to the newer version from your server. It was the same there (wrong tensor format. Output was always set to 32bit, input to 16bit, even if i uncheck the 32bit checkbox). On the frame-tab this is not the case (with the old .py file).

Which version of vsmlrt is your uploaded one in the experimental folder?

[Selur]

Yes, you should delete everything in the engine folder.
Do not overwrite vs-mlrt folders, delete the old folder replace it with the new version.

Output was always set to 32bit, input to 16bit, even if i uncheck the 32bit checkbox

latest dev should set: output_format "=0" for 32bit and "=1" for 16bit.

Which version of vsmlrt is your uploaded one in the experimental folder?

Now the vs-mlrt_2026.06.26 should contain https://github.com/AmusementClub/vs-mlrt/releases/tag/v16.1.test1 + https://github.com/AmusementClub/vs-mlrt/blob/master/scripts/vsmlrt.py which is "3.23.2"

As a side note not all onnx models do work with all precisions (FP16, FP32, BF16).

Cu Selur

[Selur]

Do i have to delete anything else than the engine-and temp folder-files ? and vapoursynth and vsmlrt .zip was correct to unzip over the old one (copy and overwrite basically) ? or should i delete the old one first and then copy fresh ?

Delete old, extract new.

Which version of vsmlrt is your uploaded one in the experimental folder?

Current vs-mlrt_2026.06.26 (this was updated), contains vsmlrt "3.23.2" and the files from https://github.com/AmusementClub/vs-mlrt/releases/tag/v16.1.test1

Also note that not all models will work with all precisions (FP16, FP32, BF16). (some need specific mods, tile sizes or other stuff,...)
Judging by the name "2x_Ani4Kv2_G6i2_Compact_107500_fp32.onnx" might only work with FP32 and maybe BF16, but not FP16.

Cu Selur

Ps.: maybe the folks over at https://github.com/AmusementClub/vs-mlrt/issues have some additional ideas.

[Platos]

Ok, i did use the newest and delete old vapoursynth and vsmlrt and extract new. Sadly it did not work (still setting output_format "=0").

Anyway: i will just use lossless encoding and do it in two steps, then i don't need the FilterQueue (that's why i tried to use it). But the Model is not the problem. It works, when i use the same settings on vapoursynth->frame->resize->resizer->vsmlrt instead of vapoursynth->others->vsmlrt. And when i remember correctly, when you produce .engine files, it will automaticly produce 16Bit from the 32bit file (maybe it would not work, if i use .onnx backend, but with tensor i never had problems with that. BF16 is sth else, there i have to make sure it is bf16 i guess, when i check it). I also manually produced .engine files in ChaiNNer with exactly this model and it works.

Still thank you very, very much for your help!

[Selur]

Ok, i did use the newest and delete old vapoursynth and vsmlrt and extract new. Sadly it did not work (still setting output_format "=0").

might be a bug in Hybrid, I'll look at it.

[Selur]

Uploaded a new dev, which should fix an output_format mix-up in Hybrids call of vsmlrt.

Cu Selur

[Platos]

Ah, nice, this Works!

Thank you very much!

[Platos]

Is it possible, that there is an error with Line Thinning after the latest updates? i tried the dev from 25. again, but it was not the problem. Because when i use glsl A4k line thinning with a retinex edge mask in vapoursynth-->Line-->Misc-->Thin, it crashes, when i use it after resiziing.

This is the error:

2026-06-27 11:57:41.956
Plugin C:/Program Files/Hybrid/64bit/vsfilters/ColorFilter/Retinex/Retinex.dll is using API3 which is deprecated and will be removed shortly.
Plugin C:/Program Files/Hybrid/64bit/vs-mlrt/vstrt.dll is using API3 which is deprecated and will be removed shortly.
Plugin C:/Program Files/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/LSMASHSource.dll is using API3 which is deprecated and will be removed shortly.
2026-06-27 11:57:42.879
Failed to evaluate the script:
Python exception: retinex.MSRCP: Invalid input clip, only 8-16 bit int formats supported

Traceback (most recent call last):
File "vapoursynth.pyx", line 3623, in vapoursynth._vpy_evaluate
File "vapoursynth.pyx", line 3624, in vapoursynth._vpy_evaluate
File "C:\Users\Gaming-Tower\Documents\Hybrid\Tempfolder\tempPreviewVapoursynthFile11_57_41_146.vpy", line 64, in 
clipMask = masked.retinex_edgemask(src=clipMask, sigma=1)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\vsscripts\masked.py", line 22, in retinex_edgemask
ret = core.retinex.MSRCP(luma, sigma=[50, 200, 350], upper_thr=0.005)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "vapoursynth.pyx", line 3353, in vapoursynth.Function.__call__
vapoursynth.Error: retinex.MSRCP: Invalid input clip, only 8-16 bit int formats supported

2026-06-27 11:57:42.881
Core freed but 13 filter instance(s) still exist
Core freed but 6227200 bytes still allocated in framebuffers

This is the vapoursynth code preview:

# Imports
import logging
import site
import sys
import os
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# Import scripts folder
scriptPath = 'C:/Program Files/Hybrid/64bit/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))
os.environ["CUDA_MODULE_LOADING"] = "LAZY"
# Force logging to std:err
logging.StreamHandler(sys.stderr)
# loading plugins
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/Vapoursynth/Lib/site-packages/vapoursynth/plugins2/libvs_placebo.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/Support/TCanny.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/ColorFilter/Retinex/Retinex.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/Vapoursynth/Lib/site-packages/vapoursynth/plugins2/akarin/libakarin.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/Vapoursynth/Lib/site-packages/vapoursynth/plugins2/vszip/vszip.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vs-mlrt/vstrt.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/Vapoursynth/Lib/site-packages/vapoursynth/plugins2/fmtconv.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/LSMASHSource.dll")
# Import scripts
import masked
from importlib.machinery import SourceFileLoader
vsmlrt = SourceFileLoader('vsmlrt', 'C:/Program Files/Hybrid/64bit/vs-mlrt/vsmlrt.py').load_module()
from vsmlrt import Backend
import validate
# Source: 'C:\Users\Gaming-Tower\Videos\test.mkv'
# clip current meta; color space: YUV420P8, bit depth: 8, resolution: 1920x1080, fps: 23.976, color matrix: 709, color primaries: Unspecific, color transfer: Unspecified, yuv luminance scale: limited, scanorder: progressive, full height: true (Source)
# Loading 'C:\Users\Gaming-Tower\Videos\test.mkvÄ using LWLibavSource
clip = core.lsmas.LWLibavSource(source="C:/Users/Gaming-Tower/Videos/test.mkv", format="YUV420P8", stream_index=0, cache=0, prefer_hw=0)
frame = clip.get_frame(0)
# setting color matrix to 709.
clip = core.std.SetFrameProps(clip, _Matrix=vs.MATRIX_BT709)
# setting color transfer (vs.TRANSFER_BT709), if it is not set.
if validate.transferIsInvalid(clip):
  clip = core.std.SetFrameProps(clip=clip, _Transfer=vs.TRANSFER_BT709)
# setting color primaries info (to vs.PRIMARIES_BT709), if it is not set.
if validate.primariesIsInvalid(clip):
  clip = core.std.SetFrameProps(clip=clip, _Primaries=vs.PRIMARIES_BT709)
# setting color range to TV (limited) range.
prop_name = '_Range' if core.core_version.release_major >= 74 else '_ColorRange'
clip = core.std.SetFrameProps(clip=clip, **{prop_name: vs.RANGE_LIMITED})
# making sure frame rate is set to 23.976fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=24000, fpsden=1001)
# making sure the detected scan type is set (detected: progressive)
clip = core.std.SetFrameProps(clip=clip, _FieldBased=vs.FIELD_PROGRESSIVE) # scan type: progressive
# changing range from limited to full range for vsVSMLRT
clip = core.resize.Bicubic(clip, format=vs.YUV420P8, range_in_s="limited", range_s="full")
# setting color range to PC (full) range.
prop_name = '_Range' if core.core_version.release_major >= 74 else '_ColorRange'
clip = core.std.SetFrameProps(clip=clip, **{prop_name: vs.RANGE_FULL})
# adjusting color space from YUV420P8 to RGBH for vsVSMLRT
clip = core.resize.Bicubic(clip=clip, format=vs.RGBH, matrix_in_s="709", range_in_s="full", range_s="full")
# resizing using VSMLRT, target: 3840x2160
clip = vsmlrt.inference([clip],network_path="C:/Program Files/Hybrid/64bit/onnx_models/2x_Ani4Kv2_G6i2_Compact_107500_fp32.onnx",backend=Backend.TRT(fp16=True,device_id=0,bf16=False,num_streams=2,output_format=1,verbose=True,use_cuda_graph=False,workspace=1073741824,builder_optimization_level=3,engine_folder="C:/Users/Gaming-Tower/Documents/Hybrid/Engine-Files")) # 3840x2160
# making sure 0-1 limits are respected
clip = core.vszip.Limiter(clip=clip, min=[0,0,0], max=[1,1,1])
## Starting applying 'EdgeMask (Retinex)' masked filtering for vsGLSLThinLine
# Creating mask for clip
clipMask = clip
clipMask = masked.retinex_edgemask(src=clipMask, sigma=1)
clipMask = core.resize.Bicubic(clip=clipMask, format=vs.GRAY16, matrix_s="709", range_in_s="full", range_s="full")
clipFiltered = clip
# adjusting color space from RGBH to YUV444P16 for vsGLSLThinLine
clipFiltered = core.resize.Bicubic(clip=clipFiltered, format=vs.YUV444P16, matrix_s="709", range_in_s="full", range_s="full")
# Line thinning using GLSL Anime4K-Thin
with open("C:/Program Files/Hybrid/64bit/vsfilters/GLSL/parameterized/Anime4K_Thin_HQ.glsl") as glslf:
  glsl = glslf.read()
glsl = glsl.replace('#define STRENGTH 0.6', '#define STRENGTH 0.2')
glsl = glsl.replace('#define ITERATIONS 1', '#define ITERATIONS 1')
clipFiltered = core.placebo.Shader(clipFiltered, shader_s=glsl, width=clipFiltered.width, height=clipFiltered.height)
# adjusting clip color before mask merge
clip = core.resize.Bicubic(clip=clip, format=vs.YUV444P16, matrix_s="709", range_in_s="full", range_s="full")
# Resolution: clip: 3840x2160 and clipFiltered: 3840x2160 and clipMask: 3840x2160
# Merging: clip and clipFiltered based on clipMask
clip = core.std.MaskedMerge(clip, clipFiltered, clipMask) # RetinexEdgeMask
## Finished applying 'RetinexEdgeMask' masked filtering for vsGLSLThinLine
# adjusting output color from YUV444P16 to YUV420P8 for x265Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P8, dither_type="error_diffusion")
# changing range from full to limited range for matching filter target luma scale
clip = core.resize.Bicubic(clip, format=vs.YUV420P8,range_in_s="full", range_s="limited")
# setting color range to TV (limited) range.
prop_name = '_Range' if core.core_version.release_major >= 74 else '_ColorRange'
clip = core.std.SetFrameProps(clip=clip, **{prop_name: vs.RANGE_LIMITED})
# set output frame rate to 23.976fps (progressive)
clip = core.std.AssumeFPS(clip=clip, fpsnum=24000, fpsden=1001)
# output
clip.set_output()
# clip current meta; color space: YUV420P8, bit depth: 8, resolution: 3840x2160, fps: 23.976, color matrix: 709, color primaries: Unspecific, color transfer: Unspecified, yuv luminance scale: limited, scanorder: progressive, full height: true (Meta)
# script was created by Hybrid 2026.06.26.1

I did use retinex-masked before (i even set it as default).

LLM says the error comes from there:
"
# 1. Clip wird zu RGBH konvertiert für TensorRT
clip = core.resize.Bicubic(clip=clip, format=vs.RGBH, ...)
# 2. VSMLRT Upscale auf 4K
clip = vsmlrt.inference([clip], ...) # Output: RGBH
# 3. EdgeMask wird auf RGBH angewendet
clipMask = clip # ← RGBH (Float!)
clipMask = masked.retinex_edgemask(src=clipMask, sigma=1) # ← FEHLER!
"

I did a test again with line thinning before the upscaling and it does not look the same. And one test with a lossless encode between to make the line thinning after the resize. It looks the same, as my test before. So i feel like it was working before. But im not sure, if i did it before also with a losslessencode between. Is it possible, that now hybrid does not convert back from RGBH before it uses the line thin retinex mask stuff ? Or was this always the case?

[Selur]

According to the script and the error, the issue is that RetinexEdgeMask like Retinex does not support float color spaces, but only integer based color spaces.
so calling:

## Starting applying 'EdgeMask (Retinex)' masked filtering for vsGLSLThinLine
# Creating mask for clip
clipMask = clip
clipMask = masked.retinex_edgemask(src=clipMask, sigma=1)

fails with:

clipMask = masked.retinex_edgemask(src=clipMask, sigma=1)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\vsscripts\masked.py", line 22, in retinex_edgemask
ret = core.retinex.MSRCP(luma, sigma=[50, 200, 350], upper_thr=0.005)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "vapoursynth.pyx", line 3353, in vapoursynth.Function.__call__
vapoursynth.Error: retinex.MSRCP: Invalid input clip, only 8-16 bit int formats supported

since clip is RGBH.
I did refactor VsFilter a while ago, not sure whether I missed this case there or whether Hybrid never checked this.
=> I'll look at it.