Halo/ring effect around objects in video frames

[SVAPrjm]

I am encoding with staxrip/qtgmc progressive settings for progressive input videos, mkv to mp4, and I keep getting this halo/ring around objects (see the snowboarder on the rail top of the image in the example below). Is there a dehalo/dering filter I should use to minimize or eliminate this? staxrip encode log with avisynth script is attached as well.


[Attachment 80887 - Click to enlarge]

[Selur]

DeHalo_alpha, FineDehalo probably work if you adjust the settings.
If your source does not have these, better simply don't overdo the sharpening.
If you hope for folks to suggest some more specific settings, share a sample of your source.

[SVAPrjm]

DeHalo_alpha, FineDehalo probably work if you adjust the settings.
If your source does not have these, better simply don't overdo the sharpening.
If you hope for folks to suggest some more specific settings, share a sample of your source.

Source attached.

[Selur]

Is that the source of the source processed by 'regular' qtgmc settings?
That file was over sharpened, you might have some success by blurring and resharpening.
=> assuming this is not the source, start from the source and do not over-sharpen it blindly

[SVAPrjm]

Is that the source of the source processed by 'regular' qtgmc settings?
That file was over sharpened, you might have some success by blurring and resharpening.
=> assuming this is not the source, start from the source and do not over-sharpen it blindly

Yes, that's the output from the original source. Are there any settings you recommend to use to not oversharpen?

[Selur]

If the source has halos: remove them before sharpening
Disable sharpening in QTGMC.
Try something like CAS, don't use LSFMod but NNED3 for resizing.
Share a snippet of the source if you need more than general help.

[SVAPrjm]

If the source has halos: remove them before sharpening
Disable sharpening in QTGMC.
Try something like CAS, don't use LSFMod but NNED3 for resizing.
Share a snippet of the source if you need more than general help.

Ran a bunch of tests with your suggestions and I think I am getting the output I want. The only thing I want to clean up is is blurring effect that is present in films with high motion like this. I ran the same segment through Topaz and cleaned it up using the topaz motion deblur setting (you can see in the pics) but unfortunately it would take me 30+ hours to encode a 35min full video so that's not an option. Looking for recommendations on how to clean up this blurring effect in Hybrid/Staxrip.

[Selur]

Looking at the log that file the attached video is the output, not the input.
=> no clue, nothing I can suggest

[SVAPrjm]

If the source has halos: remove them before sharpening
Disable sharpening in QTGMC.
Try something like CAS, don't use LSFMod but NNED3 for resizing.
Share a snippet of the source if you need more than general help.

Ran a bunch of tests with your suggestions and I think I am getting the output I want. The only thing I want to clean up is is blurring effect that is present in films with high motion like this. I ran the same segment through Topaz and cleaned it up using the topaz motion deblur setting (you can see in the pics) but unfortunately it would take me 30+ hours to encode a 35min full video so that's not an option. Looking for recommendations on how to clean up this blurring effect in Hybrid/Staxrip.

Shoot sorry, here's the input.

[Selur]

Okay, that source is wrongly flagged as interlaced.
I would use CAS + DPIR Deblock + DeHalo_alpha as a base: https://imgsli.com/MjgxNjU1

# contrast sharpening using CAS
clip = core.cas.CAS(clip=clip, sharpness=0.700)
from vsdpir import dpir as DPIR
# adjusting color space from YUV420P8 to RGBH for vsDPIRDeblock
clip = core.resize.Bicubic(clip=clip, format=vs.RGBH, matrix_in_s="470bg", range_s="limited")
# deblocking using DPIRDeblock
clip = DPIR(clip=clip, strength=15.000, task="deblock", device_index=0, num_streams=3, trt=True, trt_cache_dir="J:/tmp")
# adjusting color space from RGBH to YUV444P16 for vsDeHalo_Alpha
clip = core.resize.Bicubic(clip=clip, format=vs.YUV444P16, matrix_s="470bg", range_s="limited")
# applying dehalo using DeHalo_alpha
clip = havsfunc.DeHalo_alpha(clip, rx=2.50)
# resizing using NNEDI3CL
# current: 720x480 target: 1920x1280 -> pow: 4
clip = edi_rpow2.nnedi3cl_rpow2(clip=clip, rfactor=4, nsize=3, nns=4) # 2880x1920
# adjusting resizing
clip = core.fmtc.resample(clip=clip, w=1920, h=1280, kernel="spline64", interlaced=False, interlacedd=False)# before YUV444P16 after YUV444P16

Depending on the scene I would also add additional denoising, line darkening and adjust the deblocking and sharpening strength.

Since there is a source now, others probably will suggest alternatives.

Cu Selur

[SVAPrjm]

I’m pretty new to this kind of coding (coding in general) and can’t figure out how to install the pyrepo in Vapoursynth or run this script in Hybrid. Any help would be greatly appreciated. I usually use the interface and adjust settings from there.

[Selur]

Hybrid does come with Vapoursynth, installing Vapoursynth, Python and similar manually is probably wrong.
The torch-addon for Hybrid is a separate download, from my GoogleDrive which is linked on the official download site.
(the above snippet is code that Hybrid would generate if you configure it correctly; it's not something to load into Hybrid)

[SVAPrjm]

Ah ok. That worked as intended. Getting this error now with the script you graciously provided.

Failed to evaluate the script:
Python exception: No attribute with the name cas exists. Did you mistype a plugin namespace or forget to install a plugin?

Traceback (most recent call last):
File "src\\cython\\vapoursynth.pyx", line 3365, in vapoursynth._vpy_evaluate
File "src\\cython\\vapoursynth.pyx", line 3366, in vapoursynth._vpy_evaluate
File "", line 4, in
File "src\\cython\\vapoursynth.pyx", line 2873, in vapoursynth._CoreProxy.__getattr__
File "src\\cython\\vapoursynth.pyx", line 2696, in vapoursynth.Core.__getattr__
AttributeError: No attribute with the name cas exists. Did you mistype a plugin namespace or forget to install a plugin?

[SVAPrjm]

Was able to manually set the VS filters instead of running a script but two things: 1) it was a very very slow encode running at about .24fps. Is that a combination of the filters used and my OS? Is there a way to get it to run faster without completely upgrading my hardware? 2) the file that did encode will play on my computer but won’t play on my TV now like everything else I’ve encoded. Output container is an mp4. Did I miss a setting somewhere? Can upload the output file if needed.

[Selur]

Knowing nothing about your hardware (or what settings you used), you can try using DeblockPP7 instead of DPIR deblock. (won't be as good, but faster)

[SVAPrjm]

I can't recall specifically what my hardware is but I used the settings you provided in the script above, just manually by clicking through the VS filter tabs. Main question I have is what would cause it to be unable to playback the file on my TV? Never ran into that issue with anything else I encoded. Output file is attached, without sound.

[SVAPrjm]

Knowing nothing about your hardware (or what settings you used), you can try using DeblockPP7 instead of DPIR deblock. (won't be as good, but faster)

I’m running AMD Ryzen 5 3600 6-core processor 3.6GHz and 32GB installed RAM

[Selur]

No filter can cause playback issues, those are probably due to some incompatible encoding settings.
Will look at the file later (if I don't forget it ), working now

[SVAPrjm]

Here's the output from Hybrid with the code you provided. It's pixelated/blocky. Not sure what's going. I entered it manually and through each tab on the interface. Should have copied the script over to a txt doc to upload but can't right now. Appreciate all your help.

[Selur]

At least post your script,... my guess is you didn't tell Hybrid the source isn't interlaced even when it's progressive, thus Hybrid will apply some deinterlacing which will cause issues,...

[SVAPrjm]

# Imports
import vapoursynth as vs
# getting Vapoursynth core
import sys
import os
core = vs.core
# Import scripts folder
scriptPath = 'C:/Program Files/Hybrid/64bit/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))
# loading plugins
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/ResizeFilter/nnedi3/vsznedi3.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/GrainFilter/RemoveGrain/RemoveGrainVS.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/Support/fmtconv.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/SharpenFilter/CAS/CAS.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/LSMASHSource.dll")
# Import scripts
import edi_rpow2
import havsfunc
import validate
# Source: 'C:\Users\Computer\Videos\RJ videos\Videograss\(2017) Visitors (1)-003.mkv'
# Current color space: YUV420P8, bit depth: 8, resolution: 720x480, frame rate: 29.97fps, scanorder: progressive, yuv luminance scale: limited, matrix: 470bg
# Loading C:\Users\Computer\Videos\RJ videos\Videograss\(2017) Visitors (1)-003.mkv using LWLibavSource
clip = core.lsmas.LWLibavSource(source="C:/Users/Computer/Videos/RJ videos/Videograss/(2017) Visitors (1)-003.mkv", format="YUV420P8", stream_index=0, cache=0, prefer_hw=0)
frame = clip.get_frame(0)
# Setting detected color matrix (470bg).
clip = core.std.SetFrameProps(clip=clip, _Matrix=5)
# setting color transfer (170), if it is not set.
if validate.transferIsInvalid(clip):
clip = core.std.SetFrameProps(clip=clip, _Transfer=6)
# setting color primaries info (to 470), if it is not set.
if validate.primariesIsInvalid(clip):
clip = core.std.SetFrameProps(clip=clip, _Primaries=5)
# setting color range to TV (limited) range.
clip = core.std.SetFrameProps(clip=clip, _ColorRange=1)
# making sure frame rate is set to 29.97fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=30000, fpsden=1001)
# making sure the detected scan type is set (detected: progressive)
clip = core.std.SetFrameProps(clip=clip, _FieldBased=0) # progressive
from vsdpir import dpir as DPIR
# adjusting color space from YUV420P8 to RGBS for vsDPIRDeblock
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="470bg", range_s="limited")
# deblocking using DPIRDeblock
clip = DPIR(clip=clip, strength=15.000, task="deblock", device_index=0, num_streams=3)
# contrast sharpening using CAS
clip = core.cas.CAS(clip=clip, sharpness=0.700)
# adjusting color space from RGBS to YUV444P16 for vsDeHalo_Alpha
clip = core.resize.Bicubic(clip=clip, format=vs.YUV444P16, matrix_s="470bg", range_s="limited", dither_type="error_diffusion")
# applying dehalo using DeHalo_alpha
clip = havsfunc.DeHalo_alpha(clip, rx=2.50)
# resizing using ZNEDI3
# current: 720x480 target: 1920x1280 -> pow: 4
clip = edi_rpow2.nnedi3_rpow2(clip=clip, rfactor=4) # 2880x1920
# adjusting resizing
clip = core.fmtc.resample(clip=clip, w=1920, h=1280, kernel="spline64", interlaced=False, interlacedd=False)# before YUV444P16 after YUV444P16
# adjusting output color from: YUV444P16 to YUV420P8 for x264Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P8, range_s="limited", dither_type="error_diffusion")
# set output frame rate to 29.97fps (progressive)
clip = core.std.AssumeFPS(clip=clip, fpsnum=30000, fpsden=1001)
# output
clip.set_output()

[Selur]

You used a different filter order than I.
your script:

from vsdpir import dpir as DPIR
# adjusting color space from YUV420P8 to RGBS for vsDPIRDeblock
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="470bg", range_s="limited")
# deblocking using DPIRDeblock
clip = DPIR(clip=clip, strength=15.000, task="deblock", device_index=0, num_streams=3)
# contrast sharpening using CAS
clip = core.cas.CAS(clip=clip, sharpness=0.700)
# adjusting color space from RGBS to YUV444P16 for vsDeHalo_Alpha
clip = core.resize.Bicubic(clip=clip, format=vs.YUV444P16, matrix_s="470bg", range_s="limited", dither_type="error_diffusion")
# applying dehalo using DeHalo_alpha
clip = havsfunc.DeHalo_alpha(clip, rx=2.50)
# resizing using ZNEDI3
# current: 720x480 target: 1920x1280 -> pow: 4
clip = edi_rpow2.nnedi3_rpow2(clip=clip, rfactor=4) # 2880x1920
# adjusting resizing
clip = core.fmtc.resample(clip=clip, w=1920, h=1280, kernel="spline64", interlaced=False, interlacedd=False)# before YUV444P16 after YUV444P16

my script:

# contrast sharpening using CAS
clip = core.cas.CAS(clip=clip, sharpness=0.700)
from vsdpir import dpir as DPIR
# adjusting color space from YUV420P8 to RGBH for vsDPIRDeblock
clip = core.resize.Bicubic(clip=clip, format=vs.RGBH, matrix_in_s="470bg", range_s="limited")
# deblocking using DPIRDeblock
clip = DPIR(clip=clip, strength=15.000, task="deblock", device_index=0, num_streams=3, trt=True, trt_cache_dir="J:/tmp")
# adjusting color space from RGBH to YUV444P16 for vsDeHalo_Alpha
clip = core.resize.Bicubic(clip=clip, format=vs.YUV444P16, matrix_s="470bg", range_s="limited")
# applying dehalo using DeHalo_alpha
clip = havsfunc.DeHalo_alpha(clip, rx=2.50)
# resizing using NNEDI3CL
# current: 720x480 target: 1920x1280 -> pow: 4
clip = edi_rpow2.nnedi3cl_rpow2(clip=clip, rfactor=4, nsize=3, nns=4) # 2880x1920
# adjusting resizing
clip = core.fmtc.resample(clip=clip, w=1920, h=1280, kernel="spline64", interlaced=False, interlacedd=False)# before YUV444P16 after YUV444P16

=> use the "Vapoursynth Script Preview" and adjust the "Filter Order".
I would also recommend using the Vapoursynth preview before encoding.

Cu Selur

Ps.: you might want ot use 'code'-tags to make your posts more readable when posting code.

[SVAPrjm]

That worked! And increased the speed of the encoding too. Thank you so much!

A couple other questions I have is 1) I still can't get the mp4 output to play on Roku media player, says "the codec used to compress the video is not supported." I've never had this problem in the past until last night. 2) For deinterlacing, where you recommend that filter go in the filter order?

[Selur]

1) I still can't get the mp4 output to play on Roku media player, says "the codec used to compress the video is not supported." I've never had this problem in the past until last night.

I don't own a Roku media player. I don't know what encoder and setting you used. Seems like the encoder settings (and may be resolution & frame rate) combination is not supported by that player.

2) For deinterlacing, where you recommend that filter go in the filter order?

first

Cu Selur

[SVAPrjm]

Awesome, thank you for the filter tip!

As far as the codec, I'm not entirely sure what encoders come installed with Hybrid so I just used whatever it defaults to after installing the torch add-on. Didn't mess with any of the container settings or anything.

[Selur]

Depending on your OS and your hardware, Hybrid support the following video encoders:

• x264 (H.264)
• XxiD (MPEG-4 ASP)
• VP8, VP9
• aomenc (AV-1)
• rav1e (AV-1)
• x265 (H.265)
• kvazaar (H.265)
• ProRes
• FFV1
• DNxHD (Digital Nonlinear Extensible High Definition)
• UT Video
• FFVHuff
• SVT-HEVC (H.265)
• SVT-AV1 (AV-1)
• FFNVENC (H.264, H.265)
• CFHD (Cineform High Definition)
• PNG (for image output)
• DivX265 (H.265)
• QSVEncC (H.264, H.265, AV-1)
• VCEEncC (H.264, H.265, AV-1)
• X265-16bit (H.265)
• NVEncC (H.264, H.265, AV-1)

the default encoder should be x264 with (8bit, High Profile 5.1) and 2pass encoding aiming for an average bitrate of 1500 kBit/s, which would be really low for HD content, but should be compatible with normal players.
No clue why your Roku doesn't play the file. (it probably won't support 10bit H.264, but since you didn't change the setting,..)
Did a quick GoogleSearch and from the looks of it, Roku might have some issues with some x264 encodes.
=> no clue, not a bug in Hybrid, but a mediocre player from the looks of it.

Cu Selur

[SVAPrjm]

Ahhh ok. Where or how do I change it from 10 to 8 bit 264? I’m not seeing it.

[Selur]

'x264->Base->General Settings->Calculation precision', but with 'AVC Profile' set to 'High' '10-bit' is not available, and thus that option is disabled by default.
So unless you changed some settings, your encode should be 8-bit.

[SVAPrjm]

Okay, thanks. I’ll keep playing around with the settings to see what I can figure out. Really appreciate all your help with this.

[SVAPrjm]

Encoded a full video last night and it came out perfect. This morning, not sure why I messed with a good thing, but I went and updated Hybrid to the 2024.06.16 version and added the torch add-on files and now it's back to running at .25fps encode even with the filter script in the correct order and running 1 pass CRF.

vspipe "C:\Users\Computer\AppData\Local\Temp\encodingTempSynthSkript_2024-07-27@15_11_24_5610_0.vpy" - -c y4m | x264 --preset veryfast --crf 18.00 --profile high --level 4.2 --ref 3 --direct auto --b-adapt 0 --sync-lookahead 18 --qcomp 0.50 --rc-lookahead 40 --qpmax 51 --partitions i4x4,p8x8,b8x8 --no-fast-pskip --subme 5 --aq-mode 0 --vbv-maxrate 62500 --vbv-bufsize 78125 --sar 32:27 --qpfile "C:\Users\Computer\AppData\Local\Temp\Visitors test14_1_2024-07-27@15_11_24_5610_03.qp" --non-deterministic --range tv --colormatrix bt470bg --demuxer y4m  --input-range tv --fps 30000/1001 --output-depth 8 --output "C:\Users\Computer\AppData\Local\Temp\2024-07-27@15_11_24_5610_04.264" -

# Imports
import vapoursynth as vs
# getting Vapoursynth core
import ctypes
import sys
import os
core = vs.core
# Import scripts folder
scriptPath = 'C:/Program Files/Hybrid/64bit/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))
# Loading Support Files
Dllref = ctypes.windll.LoadLibrary("C:/Program Files/Hybrid/64bit/vsfilters/Support/libfftw3f-3.dll")
# loading plugins
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/Support/EEDI3m.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/ResizeFilter/nnedi3/vsznedi3.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/MiscFilter/MiscFilters/MiscFilters.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/GrainFilter/RemoveGrain/RemoveGrainVS.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/GrainFilter/AddGrain/AddGrain.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/DenoiseFilter/NEO_FFT3DFilter/neo-fft3d.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/DenoiseFilter/DFTTest/DFTTest.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/Support/libmvtools.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/Support/scenechange.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/Support/fmtconv.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/SharpenFilter/CAS/CAS.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/LSMASHSource.dll")
# Import scripts
import edi_rpow2
import havsfunc
import validate
# Source: 'C:\Users\Computer\Videos\RJ videos\Videograss\(2017) Visitors (1)-003.mkv'
# Current color space: YUV420P8, bit depth: 8, resolution: 720x480, frame rate: 29.97fps, scanorder: progressive, yuv luminance scale: limited, matrix: 470bg
# Loading C:\Users\Computer\Videos\RJ videos\Videograss\(2017) Visitors (1)-003.mkv using LWLibavSource
clip = core.lsmas.LWLibavSource(source="C:/Users/Computer/Videos/RJ videos/Videograss/(2017) Visitors (1)-003.mkv", format="YUV420P8", stream_index=0, cache=0, prefer_hw=0)
frame = clip.get_frame(0)
# Setting detected color matrix (470bg).
clip = core.std.SetFrameProps(clip=clip, _Matrix=5)
# setting color transfer (170), if it is not set.
if validate.transferIsInvalid(clip):
  clip = core.std.SetFrameProps(clip=clip, _Transfer=6)
# setting color primaries info (to 470), if it is not set.
if validate.primariesIsInvalid(clip):
  clip = core.std.SetFrameProps(clip=clip, _Primaries=5)
# setting color range to TV (limited) range.
clip = core.std.SetFrameProps(clip=clip, _ColorRange=1)
# making sure frame rate is set to 29.97fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=30000, fpsden=1001)
# making sure the detected scan type is set (detected: progressive)
clip = core.std.SetFrameProps(clip=clip, _FieldBased=0) # progressive
# contrast sharpening using CAS
clip = core.cas.CAS(clip=clip, sharpness=0.700)
from vsdpir import dpir as DPIR
# adjusting color space from YUV420P8 to RGBS for vsDPIRDeblock
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="470bg", range_s="limited")
# deblocking using DPIRDeblock
clip = DPIR(clip=clip, strength=30.000, task="deblock", device_index=0, num_streams=3, trt=True, trt_cache_dir="")
# adjusting color space from RGBS to YUV444P16 for vsQTGMCFilter
clip = core.resize.Bicubic(clip=clip, format=vs.YUV444P16, matrix_s="470bg", range_s="limited", dither_type="error_diffusion")
# Denoising using QTGMC
clip = havsfunc.QTGMC(Input=clip, Preset="Fast", InputType=1, TR2=1, SourceMatch=0, Lossless=0, EZKeepGrain=1.00, NoisePreset="Fast")
# applying dehalo using DeHalo_alpha
clip = havsfunc.DeHalo_alpha(clip, rx=2.50)
# resizing using ZNEDI3
# current: 720x480 target: 1920x1280 -> pow: 4
clip = edi_rpow2.nnedi3_rpow2(clip=clip, rfactor=4, nsize=3, nns=4) # 2880x1920
# adjusting resizing
clip = core.fmtc.resample(clip=clip, w=1920, h=1280, kernel="spline64", interlaced=False, interlacedd=False)# before YUV444P16 after YUV444P16
# adjusting output color from: YUV444P16 to YUV420P8 for x264Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P8, range_s="limited", dither_type="error_diffusion")
# set output frame rate to 29.97fps (progressive)
clip = core.std.AssumeFPS(clip=clip, fpsnum=30000, fpsden=1001)
# output
clip.set_output()