Assistance With Vapoursynth KNLMeansCL Plugin

[FuegoJohnson]

Hello. I am trying to use Vapoursynth's KNLMeansCL plugin to denoise a 1080p 8-bit source, and I'm running into issues. My goal is to take the 8-bit source, denoise, and convert it to 10-bit to use with x265's Main10 profile.

Initially, I was running into the same problem Selur posted about on the KNLMeansCL GitHub page: https://github.com/Khanattila/KNLMeansCL/issues/42.

There, the plugin author states:

The easiest and fastest thing is to promote a video from the YUV420P10 to the YUV420P16 and work with it. Then if you prefer you can go back to 10-bit.

With that information, I then tried:

YUV420P8 -> YUV420P16 -> Denoise -> YUV420P10

Which runs, but the test video's luma appears to be messed up, and is really dark:


[Attachment 62947 - Click to enlarge]

The test script I'm using:

import vapoursynth as vs
from vapoursynth import core
import x265

in_src = r"M:\Source\Source.mkv"
in_cache = r"M:\Source\Source.lwi"

out_dir = r"M:\Source"

# Start with YUV420P8 source, 1920x1080
clip = core.lsmas.LWLibavSource(source=in_src, cachefile=in_cache)
# Change to YUV420P16 so KNLMeans can process clip
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P16, range_s="limited")
# Cropping produces a very small, centered image for some reason. Move it after KNLMeans maybe?
# clip = core.std.Crop(clip, 0, 0, 396, 396)

# Filter with KNLMeans
clip = core.knlm.KNLMeansCL(clip=clip, d=1, a=2, s=4, h=0.5, channels="Y",
                            device_type="gpu", device_id=0)
# Attempting on chroma to see if anything changes
# clip = core.knlm.KNLMeansCL(clip=clip, d=1, a=2, s=4, h=0.5, channels="UV", wref=1.0,
#                             device_type="gpu", device_id=0)

# Try to convert down to 10-bit from 16-bit...issue happens here maybe?
clip = core.resize.Bicubic(clip, format=vs.YUV420P10, range_s="limited")

# Test script provided by AI with a few modifications
x265.encode(clip=clip, crf=17, file=in_src, directory=out_dir, deblock='-2:-2', num_frames=300, start=8000) 

Where x265.encode is a slightly modified version of a script _AI_ wrote, and I borrowed it to test this (before I spent time writing my own, thanks for that ). Script is here: https://forum.videohelp.com/threads/392480-Easy-way-to-encode-vpy-scripts#post2544851.

Any ideas on what I'm doing wrong? I'm relatively new to Vapoursynth, so I'm guessing it's a user error. For reference, I'm running an AMD RX 5600 XT with OpenCL library installed - based on the Github page for KNLMeansCL, this is far above the minimum requirements.

I thought about converting to RGB instead, but based on some things I've read, that may not be the best idea? I also thought about going YUV420P8 -> YUV420P10 -> YUV420P16 -> Denoise -> YUV420P10, but that seems redundant.

Any help would be greatly appreciated!

[poisondeathray]

Is the source "normal" SDR ?

Is that screenshot the encode or the preview before the encode ?

Does the script itself preview ok without encoding ? e.g. in vsedit ?

That script works ok for me, before encoding on a 1920x1080p SDR source

Temporarily comment out the YUV420P16 line, and for KNLMeansCL, add info=True and to see if device_id=0 is the correct GPU device and that platform info looks correct when previewing

[FuegoJohnson]

Is the source "normal" SDR ?

Yep. It's my Blu-ray copy of Twister Special Edition

Is that screenshot the encode or the preview before the encode ?

That's a screenshot of the encode

Does the script itself preview ok without encoding ? e.g. in vsedit ?

Good call. I was nearly certain I checked this earlier, but after running it again, there does seem to be a runtime error:

Error forming pixmap from frame. Expected format CompatBGR32. Instead got 'YUV420P10'.

If I comment out the conversion back to YUV420P10:

Error forming pixmap from frame. Expected format CompatBGR32. Instead got 'YUV420P16'.

VSEdit script check passes, though, for whatever that's worth. I'm not quite sure what this means, but I can infer that it is expecting RGB and instead is receiving YUV?

Temporarily comment out the YUV420P16 line, and for KNLMeansCL, add info=True and to see if device_id=0 is the correct GPU device and that platform info looks correct when previewing

Awesome, thank you for the tip. I will try that right now.

[poisondeathray]

Those messages are for the preview generation in vsedit (they get converted to RGB for display , the underlying data, script is not affected)

Are you using R57, API4 ? If so, vsedit requires the mod version
https://github.com/YomikoR/VapourSynth-Editor/releases

[FuegoJohnson]

I am on R57, but I did not have that mod - much appreciated! Here is my output from Info (looks like it is pointing at my AMD card as I have an Intel chip):


[Attachment 62957 - Click to enlarge]

[poisondeathray]

Is there any change with the preview or error message ?

[Selur]

Did a quick test:

Loading 8bit 1080p content and denoising it with KNLMeansCL

# Imports
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# Loading Plugins
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/DenoiseFilter/KNLMeansCL/KNLMeansCL.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/vslsmashsource.dll")
# source: 'G:\TestClips&Co\files\MPEG-4 H.264\8.Birds_1080p24fpsRef4-112Mbps.mkv'
# current color space: YUV420P8, bit depth: 8, resolution: 1920x1080, fps: 23.976, color matrix: 709, yuv luminance scale: limited, scanorder: progressive
# Loading G:\TestClips&Co\files\MPEG-4 H.264\8.Birds_1080p24fpsRef4-112Mbps.mkv using LWLibavSource
clip = core.lsmas.LWLibavSource(source="G:/TestClips&Co/files/MPEG-4 H.264/8.Birds_1080p24fpsRef4-112Mbps.mkv", format="YUV420P8", cache=0, prefer_hw=0)
# making sure input color matrix is set as 709
clip = core.resize.Bicubic(clip, matrix_in_s="709",range_s="limited")
# making sure frame rate is set to 23.976
clip = core.std.AssumeFPS(clip=clip, fpsnum=24000, fpsden=1001)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# denoising using KNLMeansCL
clip = core.knlm.KNLMeansCL(clip=clip, channels="Y")
# adjusting output color from: YUV420P8 to YUV420P10 for x265Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, range_s="limited")
# set output frame rate to 23.976fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=24000, fpsden=1001)
# Output
clip.set_output()

-> looks fine.

Going from YUV420P8 to YUV420P16 before the filtering

# Imports
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# Loading Plugins
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/DenoiseFilter/KNLMeansCL/KNLMeansCL.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/vslsmashsource.dll")
# source: 'G:\TestClips&Co\files\MPEG-4 H.264\8.Birds_1080p24fpsRef4-112Mbps.mkv'
# current color space: YUV420P8, bit depth: 8, resolution: 1920x1080, fps: 23.976, color matrix: 709, yuv luminance scale: limited, scanorder: progressive
# Loading G:\TestClips&Co\files\MPEG-4 H.264\8.Birds_1080p24fpsRef4-112Mbps.mkv using LWLibavSource
clip = core.lsmas.LWLibavSource(source="G:/TestClips&Co/files/MPEG-4 H.264/8.Birds_1080p24fpsRef4-112Mbps.mkv", format="YUV420P8", cache=0, prefer_hw=0)
# making sure input color matrix is set as 709
clip = core.resize.Bicubic(clip, matrix_in_s="709",range_s="limited")
# making sure frame rate is set to 23.976
clip = core.std.AssumeFPS(clip=clip, fpsnum=24000, fpsden=1001)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# adjusting color from: YUV420P8 to YUV420P16 for for filtering
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, range_s="limited")
# denoising using KNLMeansCL
clip = core.knlm.KNLMeansCL(clip=clip, h=0.50, channels="Y", device_type="gpu", device_id=0)
# adjusting output color from: YUV420P16 to YUV420P10 for x265Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, range_s="limited")
# set output frame rate to 23.976fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=24000, fpsden=1001)
# Output
clip.set_output()

looks fine too. (also adjusted the KNLMeansCL values to match yours, 'd=1, a=2, s=4' are the defaults)

-> Seems like the actual Vapoursynth part seems to be correct.

In case the preview itself is correct and it's just the encoding:
"x265.encode(clip=clip, crf=17, file=in_src, directory=out_dir, deblock='-2:-2', num_frames=300, start=8000) " <- is this correct? Does x265 know that it is fed 10bit content?
in a normal command line I would use something like:

x265 --input - --output-depth 10 --y4m --profile main10 --crf 18.00 --qpfile GENERATED_QP_FILE --range limited --colormatrix bt709 --output OUTPUTFILE

note the '--output-depth 10 --y4m'.

Cu Selur

[FuegoJohnson]

Preview is no longer yelling at me with the error above, but it seems the problem exists there, too. I wasn't even sure it was playing at first, until I saw some outlines of shapes. So something is wrong with luma.

Selur, thanks for checking! To answer your question, input depth should be passed to the encode function based on the transform back to 10-bit. I use Vapoursynth's clip format property as the input depth. Here is the test encode script:

import subprocess
import shutil
import os
import vapoursynth as vs
from vapoursynth import core


def encode(clip=None, crf=18, file='', directory='', deblock='-3:-3', num_frames=None, start=None, crop=None):
    if not clip or not file or not directory:
        raise ValueError("clip, file or directory are mandatory arguments")
    if not isinstance(clip, vs.VideoNode):
        raise ValueError("'clip' is not a vapoursynth clip")
    if not isinstance(crf, int):
        raise ValueError("'crf' needs to be an integer")
    if not os.path.isfile(file):
        raise ValueError("'file' is not a file")
    if not os.path.isdir(directory):
        raise ValueError("'directory' does not exist")

    is_in_path = shutil.which('x265.exe') is not None
    if is_in_path:
        x265 = shutil.which('x265.exe')
        print(f"x265 path is: {x265}")
    else:
        raise ValueError("put x265 executable in working directory or in the PATH")

    if not clip.format.color_family == vs.YUV:
        raise ValueError("not a YUV clip")

    # set test parameter options
    num_frames = len(clip) if num_frames is None else num_frames
    start = 0 if start is None else start

    # Set crop values
    # crop = [clip.width if crop is None else crop[0], clip.height if crop[1] is None else crop[1]]

    output = os.path.join(directory, os.path.basename(file) + '_test P16_denoise.hevc')

    x265_cmd = [x265, '--frames', f'{num_frames}',
                '--seek', f'{start}',
                '--y4m',
                '--preset', 'slow',
                '--profile', 'main10',
                '--input-depth', f'{clip.format.bits_per_sample}',
                '--output-depth', f'{clip.format.bits_per_sample}',
                '--input-res', f'{clip.width}x{clip.height}',
                '--fps', f'{clip.fps_num}/{clip.fps_den}',
                '--crf', f'{str(crf)}',
                '--deblock', f'{deblock}',
                '--bframes', '8',
                '--subme', '4',
                '--aq-mode', '3',
                '--aq-strength', '0.80',
                '--b-intra',
                '--no-cutree',
                '--no-sao',
                '--output', output,
                '-']  # input

    print("Preparing to run encode...opening process")
    process = subprocess.Popen(x265_cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE)

    clip.output(process.stdin, y4m=True)
    process.communicate()
    return 

I'm currently ripping a new source to test with, in case there happens to be an issue with my current one. It seems very odd that both of you haven't had any issues, so maybe it's a problem with the source? Or perhaps the graphics library, I'm not sure At least you've confirmed the code works, so thank you for that.

[poisondeathray]

If you comment everything out, except the source filter, does preview look "normal ?" ie. Just the source loaded

If it's still off, did you try other source filters ? ffms2 ?

Is the source "normal" in a media player ? Just playing it in mpv, vlc etc...

[FuegoJohnson]

If you comment everything out, except the source filter, does preview look "normal ?" ie. Just the source loaded

Commenting out everything but the KNLMeansCL filter looks like this:


[Attachment 62963 - Click to enlarge]

I think this is what you were asking? Let me know if I misinterpreted.

I tested every line independently, and then turned them on 1 by 1. It seems the common denominator is KNLMeansCL.

If it's still off, did you try other source filters ? ffms2 ?

Same issue using ffms2, unfortunately.

Is the source "normal" in a media player ? Just playing it in mpv, vlc etc...

Opened in mpv, VLC, and MPC-HC. No issues.

[poisondeathray]

I tested every line independently, and then turned them on 1 by 1. It seems the common denominator is KNLMeansCL.

1) To be clear - I mean does video alone look normal , with that source filter only. ie. Does it ever look normal in vapoursynth/vsedit - or is it a source filter problem to begin with ?

in_src = r"M:\Source\Source.mkv"
in_cache = r"M:\Source\Source.lwi" 
clip = core.lsmas.LWLibavSource(source=in_src, cachefile=in_cache) 
clip.set_output()

2) Also try the other openCL device (intel), device_id=1 , to rule out some driver issue with AMD


3) What version of KNLMeansCL ?

If you haven't tried, try the dualsynth version , pinterf

https://github.com/pinterf/KNLMeansCL/releases

[FuegoJohnson]

To be clear - I mean does video alone look normal, with that source filter only. ie. Does it ever look normal in vapoursynth/vsedit - or is it a source filter problem to begin with ?

Oh I see. Sorry. Yes, the video looks normal in preview, unless KNLMeansCL is introduced.

Also try the other openCL device (intel), device_id=1 , to rule out some driver issue with AMD

Small rant - I changed to device_id=1, and received an error that no OpenCL supported device was found...fine. I tried to download the OpenCL runtime from Intel, jumped through their insane account creation process/requirements, and it says my login info is incorrect. So I haven't been able to test this.

What version of KNLMeansCL ?

1.1.1, which appears to be the latest.

If you haven't tried, try the dualsynth version , pinterf

Awesome! I was unaware of this, I will try it. Thank you again. If you have any other suggestions, please let me know. The source is horribly noisy, so I want to test with a proper denoised clip to compare.

Also, I finished my remux earlier, so I will test shortly to see if the issue is with the source or not.


EDIT: I just tried with source #2, same issue. So I'm guessing there must be a problem with my dll or maybe my graphics driver.

I'll grab the one you suggested, and compile from source (just in case) the KNLMeansCL lib. The version I have was shoplifted from the VS Fatpack after I did a proper Vapoursynth install, but it's hard to imagine something is wrong with it, or I'd think someone would have raised an issue with Chaos in his repo.

Thanks again for all of your help.

[poisondeathray]

Not sure, they all work for me, even the old .dll versions. The pinterf version has a bunch of bug fixes

What about using 8bit, YUV420P8 to feed into KNLMeans

Otherwise it sounds like something specific to your setup . Can you run other OpenCL tests /benchmarks / programs successfully ?

[FuegoJohnson]

What about using 8bit, YUV420P8 to feed into KNLMeans

Same issue, unfortunately. I tried the modded version and it did appear to be an improvement, but luma is still all messed up. So it must be something specific to my setup like you said. I was able to successfully run a few OpenCL benchmarks, the first few using something called GeekBench. Scores rated fairly well, but I wasn't able to see any "visual" feedback on screen.

I guess I'll try uninstalling/reinstalling my graphics driver and see if that helps. I appreciate your time trying to help me troubleshoot this.