Issues Deinterlacing DVCam Footage

[Selur]

you can try increasing SAD&SAC of SMDegrain or try mClean instead.
Also add some DeHalo filtering.

[Selur]

You could also use a motion mask and some heavy smoothing/denoising to hide the artifacts,..
script example: https://pastebin.com/1iGtzFFq

[Tom619]

Is there any way to do that using Hybrid 2022.03.20.1? I don't think this version has DGDenoise.

P.S. I have installed the MotionMask plugin though so if there's a way to use it with this version of Hybrid maybe that would work?

[Selur]

you would have to write a custom part in the script.

[Tom619]

I've tried to add some of the code in the script you posted but I can't see any difference with my output. Here is the full code and maybe you can tell me where I've gone wrong:

# Imports
import os
import sys
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# defining beforeDeinterlace-function - START
def beforeDeinterlace(clip):
  clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="709", range_s="limited")
  clip = core.descale.Debicubic(clip, 720,576,0.333,0.333)
  clip = core.resize.Bicubic(clip=clip, format=vs.YUV422P10, matrix_s="709", range_s="limited", dither_type="error_diffusion")
  return clip
# defining beforeDeinterlace-function - END

# defining beforeResize-function - START
def beforeResize(clip):
  clipMask = clip
  clipMask = core.motionmask.MotionMask(clip=clipMask, th1=10, th2=10, tht=25)
  clip = core.resize.Bicubic(clip=clip, format=vs.YUV444P16, range_s="limited")
  clipMask = core.resize.Bicubic(clip=clipMask, format=vs.GRAY16, range_s="limited")
  return clip
# defining beforeResize-function - END

# Import scripts folder
scriptPath = '/Applications/Hybrid.app/Contents/MacOS/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))
# Import scripts
import edi_rpow2
import nnedi3_resample
import mvsfunc
import smdegrain
import havsfunc
# source: '/Users/tom/Desktop/*EXTRAS/Hybrid Tests/Original File.mov'
# current color space: YUV422P10, bit depth: 10, resolution: 1280x720, fps: 25, color matrix: 709, yuv luminance scale: limited, scanorder: top field first
# Loading /Users/tom/Desktop/*EXTRAS/Hybrid Tests/Original File.mov using LibavSMASHSource
clip = core.lsmas.LibavSMASHSource(source="/Users/tom/Desktop/*EXTRAS/Hybrid Tests/Original File.mov")
# Setting color matrix to 709.
clip = core.std.SetFrameProps(clip, _Matrix=1)
clip = clip if not core.text.FrameProps(clip,'_Transfer') else core.std.SetFrameProps(clip, _Transfer=1)
clip = clip if not core.text.FrameProps(clip,'_Primaries') else core.std.SetFrameProps(clip, _Primaries=1)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
clip = beforeDeinterlace(clip)
# setting field order to what QTGMC should assume (top field first)
clip = core.std.SetFrameProp(clip=clip, prop="_FieldBased", intval=2)
# Deinterlacing using QTGMC
clip = havsfunc.QTGMC(Input=clip, Preset="Slower", InputType=0, TFF=True, TR2=1, Sharpness=0.5, SourceMatch=0, Lossless=0, opencl=True) # new fps: 25
# make sure content is preceived as frame based
clip = core.std.SetFieldBased(clip, 0)
clip = clip[::2]
# removing grain using SMDegrain
clip = smdegrain.SMDegrain(input=clip, interlaced=False, opencl=True, device=-1)
# Denoising using QTGMC
clip = havsfunc.QTGMC(Input=clip, Preset="Fast", InputType=1, TR2=1, SourceMatch=0, Lossless=0, opencl=True)
clip = beforeResize(clip)
# resizing using NNEDI3CL
clip = edi_rpow2.nnedi3cl_rpow2(clip=clip, rfactor=2, nns=1) # 2560x1440
# adjusting resizing
clip = core.fmtc.resample(clip=clip, w=720, h=576, kernel="lanczos", interlaced=False, interlacedd=False)
# adjusting output color from: YUV422P16 to YUV422P10 for ProResModel
clip = core.resize.Bicubic(clip=clip, format=vs.YUV422P10, range_s="limited")
# set output frame rate to 25.000fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Output
clip.set_output()

[Selur]

Okay.
What my script added before SMDegrain was:

## Starting applying 'MotionMask' masked filtering for vsDGDenoise
clipMask = clip
clipMask = core.motionmask.MotionMask(clip=clipMask, th1=10, th2=10, tht=25)
clipFiltered = clip
# adjusting color space from YUV422P10 to YUV444P16 for vsDGDenoise
clipFiltered = core.resize.Bicubic(clip=clipFiltered, format=vs.YUV444P16, range_s="limited")
# denoising using DGDenoise
clipFiltered = core.dgdenoise.DGDenoise(clip=clipFiltered, strength=1.00)
clip = core.resize.Bicubic(clip=clip, format=vs.YUV444P16, range_s="limited")
clipMask = core.resize.Bicubic(clip=clipMask, format=vs.GRAY16, range_s="limited")
clip = core.std.MaskedMerge(clip, clipFiltered, clipMask) # MotionMask
## Finished applying 'MotionMask' masked filtering for vsDGDenoise

to do the same you would need to add this in a custom section.
And, yes DGDenoise is needed. (or another really strong denoiser; I just used DGDenoise since it's fast on my harware)

Cu Selur

[Tom619]

I'll list the DeNoiser's I have on this version of Hybrid: VagueDenoiser, DFTTest, FluxSmooth, FFT3DFilter, KNLMeansCL, CTMF, MC Temporal Denoise, TTempSmooth, HQDN3D, MiniDeen, QTGMC, CnReducer2, Despot, SpotLess, RemoveDirt, and KillerSpots. I attempted to use QTGMC but I couldn't get an improved image. I've attached the video and here is the code I used:

# Imports
import os
import sys
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# defining beforeDeinterlace-function - START
def beforeDeinterlace(clip):
  clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="709", range_s="limited")
  clip = core.descale.Debicubic(clip, 720,576,0.333,0.333)
  clip = core.resize.Bicubic(clip=clip, format=vs.YUV422P10, matrix_s="709", range_s="limited", dither_type="error_diffusion")
  return clip
# defining beforeDeinterlace-function - END

# defining beforeResize-function - START
def beforeResize(clip):
  ## Starting applying 'MotionMask' masked filtering for vsDGDenoise
  clipMask = clip
  clipMask = core.motionmask.MotionMask(clip=clipMask, th1=10, th2=10, tht=25)
  clipFiltered = clip
  # adjusting color space from YUV422P10 to YUV444P16 for vsDGDenoise
  clipFiltered = core.resize.Bicubic(clip=clipFiltered, format=vs.YUV444P16, range_s="limited")
  # denoising using DGDenoise
  clipFiltered = havsfunc.QTGMC(Input=clipFiltered, Preset="Slow", InputType=1, TR2=1, SourceMatch=0, Lossless=0)
  clip = core.resize.Bicubic(clip=clip, format=vs.YUV444P16, range_s="limited")
  clipMask = core.resize.Bicubic(clip=clipMask, format=vs.GRAY16, range_s="limited")
  clip = core.std.MaskedMerge(clip, clipFiltered, clipMask) # MotionMask
  ## Finished applying 'MotionMask' masked filtering for vsDGDenoise
  return clip
# defining beforeResize-function - END

# Import scripts folder
scriptPath = '/Applications/Hybrid.app/Contents/MacOS/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))
# Import scripts
import edi_rpow2
import nnedi3_resample
import mvsfunc
import smdegrain
import havsfunc
# source: '/Users/tom/Desktop/*EXTRAS/Hybrid Tests/Original File.mov'
# current color space: YUV422P10, bit depth: 10, resolution: 1280x720, fps: 25, color matrix: 709, yuv luminance scale: limited, scanorder: top field first
# Loading /Users/tom/Desktop/*EXTRAS/Hybrid Tests/Original File.mov using LibavSMASHSource
clip = core.lsmas.LibavSMASHSource(source="/Users/tom/Desktop/*EXTRAS/Hybrid Tests/Original File.mov")
# Setting color matrix to 709.
clip = core.std.SetFrameProps(clip, _Matrix=1)
clip = clip if not core.text.FrameProps(clip,'_Transfer') else core.std.SetFrameProps(clip, _Transfer=1)
clip = clip if not core.text.FrameProps(clip,'_Primaries') else core.std.SetFrameProps(clip, _Primaries=1)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
clip = beforeDeinterlace(clip)
# setting field order to what QTGMC should assume (top field first)
clip = core.std.SetFrameProp(clip=clip, prop="_FieldBased", intval=2)
# Deinterlacing using QTGMC
clip = havsfunc.QTGMC(Input=clip, Preset="Slower", InputType=0, TFF=True, TR2=1, Sharpness=0.5, SourceMatch=0, Lossless=0, opencl=True) # new fps: 25
# make sure content is preceived as frame based
clip = core.std.SetFieldBased(clip, 0)
clip = clip[::2]
# removing grain using SMDegrain
clip = smdegrain.SMDegrain(input=clip, interlaced=False, opencl=True, device=-1)
clip = beforeResize(clip)
# resizing using NNEDI3CL
clip = edi_rpow2.nnedi3cl_rpow2(clip=clip, rfactor=2, nns=1) # 2560x1440
# adjusting resizing
clip = core.fmtc.resample(clip=clip, w=720, h=576, kernel="lanczos", interlaced=False, interlacedd=False)
# adjusting output color from: YUV422P16 to YUV422P10 for ProResModel
clip = core.resize.Bicubic(clip=clip, format=vs.YUV422P10, range_s="limited")
# set output frame rate to 25.000fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Output
clip.set_output()

[Selur]

Try:
a. adding the code before SMDegrain, like I did
b. if you use for example KNLMeansCL you need to increase its strength a lot
The whole point of this is to blur the stuff that is detected as motion, but only that.

Here's an example where I used:
Descale before the deinterlacing and then just motionmask+strong KNLMeansCL,..
script: https://pastebin.com/9LXWdRRq
image: https://imgsli.com/MjE3Mjgz
you might want to play around with the KNLMeansCL parameters

Cu Selur

[Tom619]

Thanks, that's really helpful! The only issue I have is that I'm getting this error now:

Failed to evaluate the script:
Python exception: knlm.KNLMeansCL: only YUV444P10 and RGB30 are supported!

I can't seem to find a way to change the colour space, do you know how I can do this?

[Selur]

Okay, that is strange.
What does your script look like now?
Do you maybe this have:

# adjusting color space from YUV422P10 to YUV444P16 for vsDGDenoise
clipFiltered = core.resize.Bicubic(clip=clipFiltered, format=vs.YUV444P16, range_s="limited")

in your script?
-> recheck what I did,...

[Tom619]

Here is the script with your custom code inserted:

# Imports
import os
import sys
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# defining beforeDeinterlace-function - START
def beforeDeinterlace(clip):
  clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="709", range_s="limited")
  clip = core.descale.Debicubic(clip, 720,576,0.333,0.333)
  clip = core.resize.Bicubic(clip=clip, format=vs.YUV422P10, matrix_s="709", range_s="limited", dither_type="error_diffusion")
  return clip
# defining beforeDeinterlace-function - END

# defining beforeSMDegrain-function - START
def beforeSMDegrain(clip):
  # denoising using KNLMeansCL
  ## Starting applying 'MotionMask' masked filtering for vsKNLMeansCL
  clipMask = clip
  clipMask = core.motionmask.MotionMask(clip=clipMask, th1=10, th2=10, tht=25)
  clipFiltered = clip
  clipFiltered = havsfunc.KNLMeansCL(clip=clipFiltered, d=3, a=3, h=20.00)
  clip = core.std.MaskedMerge(clip, clipFiltered, clipMask) # MotionMask
  ## Finished applying 'MotionMask' masked filtering for vsKNLMeansCL
  return clip
# defining beforeSMDegrain-function - END

# Import scripts folder
scriptPath = '/Applications/Hybrid.app/Contents/MacOS/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))
# Import scripts
import edi_rpow2
import nnedi3_resample
import mvsfunc
import smdegrain
import havsfunc
# source: '/Users/tom/Desktop/*EXTRAS/Hybrid Tests/Original File.mov'
# current color space: YUV422P10, bit depth: 10, resolution: 1280x720, fps: 25, color matrix: 709, yuv luminance scale: limited, scanorder: top field first
# Loading /Users/tom/Desktop/*EXTRAS/Hybrid Tests/Original File.mov using LibavSMASHSource
clip = core.lsmas.LibavSMASHSource(source="/Users/tom/Desktop/*EXTRAS/Hybrid Tests/Original File.mov")
# Setting color matrix to 709.
clip = core.std.SetFrameProps(clip, _Matrix=1)
clip = clip if not core.text.FrameProps(clip,'_Transfer') else core.std.SetFrameProps(clip, _Transfer=1)
clip = clip if not core.text.FrameProps(clip,'_Primaries') else core.std.SetFrameProps(clip, _Primaries=1)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
clip = beforeDeinterlace(clip)
# setting field order to what QTGMC should assume (top field first)
clip = core.std.SetFrameProp(clip=clip, prop="_FieldBased", intval=2)
# Deinterlacing using QTGMC
clip = havsfunc.QTGMC(Input=clip, Preset="Slow", InputType=0, TFF=True, TR2=1, Sharpness=0.5, SourceMatch=0, Lossless=0, opencl=True) # new fps: 25
# make sure content is preceived as frame based
clip = core.std.SetFieldBased(clip, 0)
clip = clip[::2]
clip = beforeSMDegrain(clip)
# removing grain using SMDegrain
clip = smdegrain.SMDegrain(input=clip, thSAD=400, thSADC=160, interlaced=False, opencl=True, device=-1)
# resizing using NNEDI3CL
clip = edi_rpow2.nnedi3cl_rpow2(clip=clip, rfactor=2, nns=1) # 2560x1440
# adjusting resizing
clip = core.fmtc.resample(clip=clip, w=720, h=576, kernel="lanczos", interlaced=False, interlacedd=False)
# adjusting output color from: YUV422P16 to YUV422P10 for ProResModel
clip = core.resize.Bicubic(clip=clip, format=vs.YUV422P10, range_s="limited")
# set output frame rate to 25.000fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Output
clip.set_output()

[Selur]

Strange, the color space before KNLMeansCL should be YUV422P10 which it should support fine.
Try if it works if you replace:

clipFiltered = havsfunc.KNLMeansCL(clip=clipFiltered, d=3, a=3, h=20.00)

with:

clipFiltered = core.resize.Bicubic(clip=clipFiltered , format=vs.YUV444P10, range_s="limited")
clipFiltered = havsfunc.KNLMeansCL(clip=clipFiltered, d=3, a=3, h=20.00)
clipFiltered = core.resize.Bicubic(clip=clipFiltered , format=vs.YUV422P10, range_s="limited")

Cu Selur

[Tom619]

I think it's working now (in the sense that the script is correct), but now whenever I try to preview it, my computer crashes! I've tried just starting the job, but then Hybrid tells me the job has crashed. I'm not really sure why this is. Any ideas? If there is another DeNoiser that can get similar results it might be worth trying that, but I've tried to fit them into the code you gave and nothing gave results as good as the ones you have.

[Selur]

You could probably use any temporal denoiser if you adjust the settings,...
Can't say anything without a debug output why it might crash for you.

[Tom619]

How do I access a debug output?

[Selur]

read: https://www.selur.de/support

[Tom619]

This is the report from me starting the job.

[Selur]

Jobs stops with:

FFmpeg output: pipe:: Invalid argument

when calling:

"/usr/local/bin/vspipe" "/private/var/folders/dq/8pw_btt901n5041n73jv6bl80000gp/T/encodingTempSynthSkript_2023-10-31@15_33_42_5110.vpy" - -c y4m | "/Applications/Hybrid.app/Contents/MacOS/ffmpeg" -y -noautorotate -nostdin -threads 8 -f yuv4mpegpipe -i - -an -sn -vf zscale=rangein=tv:range=tv -pix_fmt yuv422p10le -strict -1 -vsync 0 -vcodec prores_ks -profile:v 2 -vtag apcn -aspect 720:576 -metadata encoding_tool="Hybrid 2022.03.20.1" -f mov "/Users/tomgregson/Desktop/Original File_2023-10-31@15_33_42_5110_02.mov"

1. Check that the Vapoursynth Preview works.
2. set your temp folder to an actual folder not some dynamically created folder by MacOS.

[Tom619]

Vapoursynth Preview works for all other scripts fine, and I created a new dedicated Temp folder but it still crashed. I just don't think it likes KNLMeansCL for some reason. From the experiments I did, MC Temporal Denoise seemed like it got closest to your file. There was a point where I felt like I got diminishing returns (i.e. smoothest footage, but loss of detail was too great). I'll post the results, thanks again for all your help!

[Selur]

Vapoursynth Preview works for all other scripts fine,

okay, the question was more whether it works on this script or whether it throws an error which shows where the issues are, but happy you found something that works for you.

Cu Selur

[Tom619]

When trying to Vapoursynth Preview with the adapted KNLMeansCL my whole computer crashed every time so there was no way to see any error, so I don't think there was any way to work out what was going wrong.

[Selur]

Does it work with KNLMeansCL at all?

[Tom619]

No, like I mentioned earlier, it wants the colour space changed, and then if I add the custom script that you mentioned to change the colour space it always crashes.

[Selur]

Okay, then KNLMeansCL doesn't work with your hardware/drivers.
Sticking to a masked MCTemporaDenoise is probably a good choice then (remember to try adjusting the Denoise and the masking parameters)