Help with denoising old betacam digitalized video

[dogmydog]

Dear forum friends,

I have been using Hybrid, vapoursynth and ffmpeg for a short time and learning a lot!
What do you think it would be the best way to clean up this footage?


[Attachment 60648 - Click to enlarge]
https://drive.google.com/file/d/1OyiYOeYqCgUz3S2vSN8_OjghObBTM2AZ/view?usp=sharing

Really appreciate the help.
Cheers from Rio,

[Selur]

I would recommand to use differend filtering settings for [0 317] and [318 617]. You might want to enable "Filtering->Vapoursynth->UI->Show 'Apply only to'-controls" and use them if you use Hybrid. (not that there is a mistake in the description you need to use [a b] instead of [a-b] to filter just the frames a-b.

Did you add the timer to that source and the source you want to filter does not have that timer on it or is it a fixed part of the source?
If it's a fixed part it's probably more compilcated since you should keep it out of filtering (which Hybrid atm. does not support), to avoid filters like DeDot or Spotless to break apart the timer when trying to remove straches etc.

Cu Selur

Ps.: Question in the round: Where do people get such crappy HD content from?

[dogmydog]

I would recommand to use differend filtering settings for [0 317] and [318 617]. You might want to enable "Filtering->Vapoursynth->UI->Show 'Apply only to'-controls" and use them if you use Hybrid. (not that there is a mistake in the description you need to use [a b] instead of [a-b] to filter just the frames a-b.

Did you add the timer to that source and the source you want to filter does not have that timer on it or is it a fixed part of the source?
If it's a fixed part it's probably more compilcated since you should keep it out of filtering (which Hybrid atm. does not support), to avoid filters like DeDot or Spotless to break apart the timer when trying to remove straches etc.

Cu Selur

Ps.: Question in the round: Where do people get such crappy HD content from?

Haha, my thoughts exactly on the crappy content part
Thankfully, I added the timer because it's copyrighted, so no need to bother with that. I will try your settings, thank you!

[Selur]

btw. if you want to see the real horror of the artifacts enable Color->Misc->Retinex and check the Vapoursynth Preview.

Going to bed now, but might look at it a bit tomorrow.
My guess is the main problem is to get a handle on the flickering and scratches/dirt.

Cu Selur

[Selur]

tested two settings:
HQDN3D:

# Imports
import os
import sys
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# Import scripts folder
scriptPath = 'I:/Hybrid/64bit/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))
# Loading Plugins
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/SharpenFilter/AWarpSharp2/libawarpsharp2.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/SharpenFilter/CAS/CAS.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/DenoiseFilter/HQDN3D/libhqdn3d.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/Support/libtemporalmedian.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/Support/libmvtools.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/vslsmashsource.dll")
# Import scripts
import havsfunc
import mvsfunc
import lostfunc
import SpotLess
# source: 'C:\Users\Selur\Desktop\2121160_trecho.mp4'
# current color space: YUV420P8, bit depth: 8, resolution: 1920x1080, fps: 29.97, color matrix: 709, yuv luminance scale: limited, scanorder: progressive
# Loading C:\Users\Selur\Desktop\2121160_trecho.mp4 using LibavSMASHSource
clip = core.lsmas.LibavSMASHSource(source="C:/Users/Selur/Desktop/2121160_trecho.mp4")
# making sure input color matrix is set as 709
clip = core.resize.Bicubic(clip, matrix_in_s="709",range_s="limited")
# making sure frame rate is set to 29.970
clip = core.std.AssumeFPS(clip=clip, fpsnum=30000, fpsden=1001)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# cutting from frame 318 to end - WARNING: This might cause synch issues
clip = core.std.Trim(clip=clip, first=318)
# cropping the video to 1432x974
clip = core.std.CropRel(clip=clip, left=244, right=244, top=106, bottom=0)
clip = SpotLess.SpotLess(clip=clip, chroma=False, radT=3, ablksz=64, aoverlap=32, asearch=4)
# denoising using HQDN3D
clip = core.hqdn3d.Hqdn3d(clip=clip)
clip = lostfunc.DeSpot(o=clip)
clip = lostfunc.DeSpot(o=clip)
clip = lostfunc.DeSpot(o=clip)
clip = lostfunc.DeSpot(o=clip)
# contrast sharpening using CAS
clip = core.cas.CAS(clip=clip, sharpness=0.800)
# line darkening using Toon
clip = havsfunc.Toon(input=clip)
# adjusting output color from: YUV420P8 to YUV420P10 for x265Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, range_s="limited")
# set output frame rate to 29.970fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=30000, fpsden=1001)
# Output
clip.set_output()

DPIR: (a lot slower)

# Imports
import os
import sys
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# Import scripts folder
scriptPath = 'I:/Hybrid/64bit/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))
# Loading Plugins
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/SharpenFilter/AWarpSharp2/libawarpsharp2.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/SharpenFilter/CAS/CAS.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/Support/libtemporalmedian.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/Support/libmvtools.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/vslsmashsource.dll")
# Import scripts
import havsfunc
import mvsfunc
import lostfunc
import SpotLess
# source: 'C:\Users\Selur\Desktop\2121160_trecho.mp4'
# current color space: YUV420P8, bit depth: 8, resolution: 1920x1080, fps: 29.97, color matrix: 709, yuv luminance scale: limited, scanorder: progressive
# Loading C:\Users\Selur\Desktop\2121160_trecho.mp4 using LibavSMASHSource
clip = core.lsmas.LibavSMASHSource(source="C:/Users/Selur/Desktop/2121160_trecho.mp4")
# making sure input color matrix is set as 709
clip = core.resize.Bicubic(clip, matrix_in_s="709",range_s="limited")
# making sure frame rate is set to 29.970
clip = core.std.AssumeFPS(clip=clip, fpsnum=30000, fpsden=1001)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# cutting from frame 318 to end - WARNING: This might cause synch issues
clip = core.std.Trim(clip=clip, first=318)
# cropping the video to 1432x974
clip = core.std.CropRel(clip=clip, left=244, right=244, top=106, bottom=0)
clip = SpotLess.SpotLess(clip=clip, chroma=False, radT=3, ablksz=64, aoverlap=32, asearch=4)
from vsdpir import DPIR
# adjusting color space from YUV420P8 to RGBS for vsDPIRDenoise
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="709", range_s="limited")
# denoising using DPIRDenoise
clip = DPIR(clip=clip, strength=10.000, device_index=0)
# adjusting color space from RGBS to YUV444P16 for vsDeSpot
clip = core.resize.Bicubic(clip=clip, format=vs.YUV444P16, matrix_s="709", range_s="limited")
clip = lostfunc.DeSpot(o=clip)
clip = lostfunc.DeSpot(o=clip)
clip = lostfunc.DeSpot(o=clip)
clip = lostfunc.DeSpot(o=clip)
# contrast sharpening using CAS
clip = core.cas.CAS(clip=clip, sharpness=0.800)
# line darkening using Toon
clip = havsfunc.Toon(input=clip)
# adjusting output color from: YUV444P16 to YUV420P10 for x265Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, range_s="limited")
# set output frame rate to 29.970fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=30000, fpsden=1001)
# Output
clip.set_output()

also made a HQDN3D + DPIR version,...

and attached a reencode where I stacked the version. (tl=original, tr=spotless_hqdn3d_despot_cas, bl=spotless_dpir_despot_cas, br=spotless_hqdn3d_dpir_despot_cas)

Not totally happy with any of them, but I thought sharing the results wouldn't hurt and might help to find a better solution.

Cu Selur

[dogmydog]

Dear @selur, thank you so much for the tests taken.
I tried to replicate them here, but Hybrid unfortunately has crashed. It is very possible that I'm doing something wrong

Attached to this post is the log, which I followed the instructions from forum.selur.net

Thanks again,

[poisondeathray]

This isn't the cause for the crash, but the 1st step should be decimating the duplicates . Every 5th frame is a duplicate. It's "23.976p in 29.97p"

Because they are not 100% true duplicates (there is a bit of noise and compression artifact difference between them), you need to adjust the settings for vdecimate, such as increase the block size. At least on the version you uploaded. The log file says "prores" for the actual file - so you might have to adjust the settings for that

It would look something like this, where VDecimate is inserted after AssumeFPS

And the 2nd last line before the output node should say fpsnum=24000, fpsden=1001, or you can comment it out

# making sure frame rate is set to 29.970
clip = core.std.AssumeFPS(clip=clip, fpsnum=30000, fpsden=1001)
# decimate duplicates
clip = core.vivtc.VDecimate(clip, dupthresh=1.1, blockx=256, blocky=256)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
.
.
.

# set output frame rate to 23.976fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=24000, fpsden=1001)
# Output
clip.set_output()

[Selur]

Reason for the crash is due to a bug in Hybrid that it forgot to at the end of the LibavSMASHSource-call. (since it returns an array if the input has an alpha channel)
-> will send you a link to my current dev version, which fixes this.


Cu Selur

[dogmydog]

Thanks for the fix! It worked, but I lost sync between audio and video in the h265 lossless convert.
Is there a quick way to cut the videos generated in Hybrid (without having to reconvert) so i dont have to upload the whole video?

[Selur]

a. Need a debug output to see what you did and whether Hybrid something wrong.
b. Need a step-by-step to reproduce the issue
c. Hybrid does offer a rough cut support when (Internals->Cut Support; read the top-tips of the options) is enabled. It's experimental, but might work. That said, the async output isn't really interessting since it does not show why it's async.
If you for example added the suggested lines from poisondeathray without telling Hybrid about the frame rate and frame count change async output is to be expected.
-> a sample of the source and step-by-step instructions which allow to reproduce the problem with that sample will help to reproduce this, a sample of the async source not.

Cu Selur

[dogmydog]

Dear forum friends, sorry for taking this long to reply, I am still learning so it took me a while to organize the assets.
Here is a google folder with the tests
https://drive.google.com/drive/folders/1VA0S-LadGmEL1dR03W-4gciMH9o1-eP6?usp=sharing

Since this is copyrighted material, I can only share a small 20s example.
I numbered them chronological, being

00 ORIGINAL - original files in PRORES

01 HYBRID HQDN3D - it was the test i did using @selur's first idea:

tested two settings:
HQDN3D:
[code]# Imports
import os
import sys
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# Import scripts folder

02 NEAT (40percent) then HYBRID QTMG deinterlace - I used Neat Video plugin on Premiere then Hybrid only for Deinterlace QTGCM.

03 HYBRID h265 change speed - I came back to the forum and tried this:

(...)you need to adjust the settings for vdecimate, such as increase the block size. At least on the version you uploaded. The log file says "prores" for the actual file - so you might have to adjust the settings for that(...)

@poisondeathray, I dunno how to set the vdecimate on Hybrid, so I checked the box on
Filtering > Speed Change > Change Speed > 23.976
I believe this is why my h265 was really out of sync. Here attached is the log again:

04 HYBRID QTMG deinterlace - then I realized I shoulda deinterlace on Hybrid BEFORE using Neat Video on Premiere, which lead to:

05 HYBRID QTMG deinter then NEAT -10% - and I used Neat Video on Premiere Pro to capture the noise and this is the best result I got so far.

I really appreciate all the help and support given!
I think the bad quality it's understandable since the source material is an old SD Betacam SP Ampex.
Also the dust and scratches do not bother me, but the constant noise on the entire video, even in the color parts, they do.

Let me know what you think, if you want of course.
Cheers from Rio,

[Selur]

Sorry, but I won't look at a 197MB debug output. No clue how long it would take me to read all that.

Using 'Change Speed' was wrong and not what poisondeathray intended.
His comment basically said to drop every 5th frame since it's a duplicate and adjust the frame rate accordingly.
This does not really change the general denoising.
To archive the dropping of the frame you could use one of these:

• enable "Filtering->(De)Interlace/Telecine->Deinterlace/Teleince settings->Overwrite the input scan type to" set it to "telecine" and select either "VIVTC (Vapoursynth)" or "TIVTC (Vapoursynth)"
• enable "Filtering->Vapoursynth->Frame->Reduction->SelectEvery" and enter as offsets "0 1 2 3"
• set "Filtering->Vapoursynth->Custom->Insert Berfore" to "Deinterlace" enabled the custom section and enter:
  

  clip = core.vivtc.VDecimate(clip, dupthresh=1.1, blockx=256, blocky=256)
  # framerate 23.976

  Check the Vapoursynth preview move the slider to the far right to see the number of the last frame (frame count = last frame number +1)
  Enable "Filtering->Misc->Overwirte Output"
  Enable "Filtering->Misc->Overwirte Output->Frame count to" and enter the new frame count (whatever the Vapoursynth Preview showed)
  Enable "Filtering->Misc->Overwirte Output->Frame rate to" and enter the new frame rate (23.976)

What confuses me is that you want to deinterlace the video. When I load the video in Hybrid it's reported as progressive and since the Vapoursynth Preview shows no combing, I would assume that it is progressive.

As a side note: what ever you used to cut the mov didn't do a good job.

Video
Count                                    : 378
Count of stream of this kind             : 1
Kind of stream                           : Video
Kind of stream                           : Video
Stream identifier                        : 0
StreamOrder                              : 0
ID                                       : 1
ID                                       : 1
Format                                   : ProRes
Format                                   : ProRes
Commercial name                          : ProRes
Format version                           : Version 0
Format profile                           : 422 HQ
Codec ID                                 : apch
Codec ID/Url                             : http://www.apple.com/quicktime/download/standalone.html
Duration                                 : 211011
Duration                                 : 3 min 31 s
Duration                                 : 3 min 31 s 11 ms
Duration                                 : 3 min 31 s
Duration                                 : 00:03:31.011
Duration                                 : 00:03:30:24
Duration                                 : 00:03:31.011 (00:03:30:24)
Source duration                          : 20020
Source duration                          : 20 s 20 ms
Source duration                          : 20 s 20 ms
Source duration                          : 20 s 20 ms
Source duration                          : 00:00:20.020
Bit rate mode                            : VBR
Bit rate mode                            : Variable
Bit rate                                 : 20232915
Bit rate                                 : 20.2 Mb/s
Width                                    : 1920
Width                                    : 1 920 pixels
Height                                   : 1080
Height                                   : 1 080 pixels
Sampled_Width                            : 1920
Sampled_Height                           : 1080
Pixel aspect ratio                       : 0.909
Display aspect ratio                     : 1.616
Display aspect ratio                     : 16:10
Rotation                                 : 0.000
Frame rate mode                          : CFR
Frame rate mode                          : Constant
Frame rate                               : 29.970
Frame rate                               : 29.970 (30000/1001) FPS
FrameRate_Num                            : 30000
FrameRate_Den                            : 1001
Frame count                              : 6324
Source frame count                       : 600
Color space                              : YUV
Chroma subsampling                       : 4:2:2
Chroma subsampling                       : 4:2:2
Scan type                                : Progressive
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 0.326
Delay                                    : 3478800
Delay                                    : 57 min 58 s
Delay                                    : 57 min 58 s 800 ms
Delay                                    : 57 min 58 s
Delay                                    : 00:57:58.800
Delay_Settings                           : DropFrame=No / 24HourMax=No / IsVisual=No
Delay_DropFrame                          : No
Delay, origin                            : Container
Delay, origin                            : Container
Stream size                              : 533670979
Stream size                              : 509 MiB (89%)
Stream size                              : 509 MiB
Stream size                              : 509 MiB
Stream size                              : 509 MiB
Stream size                              : 508.9 MiB
Stream size                              : 509 MiB (89%)
Proportion of this stream                : 0.89223
Source stream size                       : 594268624
Source stream size                       : 567 MiB (99%)
Source stream size                       : 567 MiB
Source stream size                       : 567 MiB
Source stream size                       : 567 MiB
Source stream size                       : 566.7 MiB
Source stream size                       : 567 MiB (99%)
Source_StreamSize_Proportion             : 0.99354
Writing library                          : adb0
Writing library                          : adb0
Language                                 : en
Language                                 : English
Language                                 : English
Language                                 : en
Language                                 : eng
Language                                 : en
colour_description_present               : Yes
colour_description_present_Source        : Container / Stream
Color primaries                          : BT.709
colour_primaries_Source                  : Container / Stream
Transfer characteristics                 : BT.709
transfer_characteristics_Source          : Container / Stream
Matrix coefficients                      : BT.709
matrix_coefficients_Source               : Container / Stream
mdhd_Duration                            : 211011

Length and delay of the container are totally wrong.

-> will look at the video tomorrow.

Cu Selur

[dogmydog]

Hahah sorry for the 200mb file, I had no idea!
Gonna delete it

About the cutting, Hybrid did a good job on cutting the PRORES, but I didn't manage to do the rest.
So I used
ffmpeg -ss 00:03:11 -i input.mov -to 00:03:31 -c copy -copyts output.mov

[Selur]

I adjusted Hybrid to use the 'Source duration' and the 'Source frame count'.

Using:

# Imports
import os
import sys
import ctypes
# Loading Support Files
Dllref = ctypes.windll.LoadLibrary("I:/Hybrid/64bit/vsfilters/Support/libfftw3f-3.dll")
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# Import scripts folder
scriptPath = 'I:/Hybrid/64bit/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))
# Loading Plugins
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/SharpenFilter/CAS/CAS.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/DenoiseFilter/CTMF/CTMF.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/DenoiseFilter/HQDN3D/libhqdn3d.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/DenoiseFilter/NEO_FFT3DFilter/neo-fft3d.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/Support/libtemporalmedian.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/Support/libmvtools.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/Support/RemapFramesVapoursynth.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/Support/fmtconv.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/vslsmashsource.dll")
# Import scripts
import havsfunc
import SpotLess
import lostfunc
import adjust
# source: 'C:\Users\Selur\Desktop\00 ORIGINAL.mov'
# current color space: YUV422P10, bit depth: 10, resolution: 1920x1080, fps: 29.97, color matrix: 709, yuv luminance scale: limited, scanorder: progressive
# Loading C:\Users\Selur\Desktop\00 ORIGINAL.mov using LibavSMASHSource
clip = core.lsmas.LibavSMASHSource(source="C:/Users/Selur/Desktop/00 ORIGINAL.mov")
# making sure input color matrix is set as 709
clip = core.resize.Bicubic(clip, matrix_in_s="709",range_s="limited")
# making sure frame rate is set to 29.97
clip = core.std.AssumeFPS(clip=clip, fpsnum=30000, fpsden=1001)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# adjusting frame with SelectEvery
clip = core.std.SelectEvery(clip=clip, cycle=5, offsets=[0, 1, 2, 3])
clip = core.std.AssumeFPS(clip=clip,fpsnum=24000, fpsden=1001)# new fps: 23.976
# cropping the video to 1580x1080
clip = core.std.CropRel(clip=clip, left=170, right=170, top=0, bottom=0)
clip = core.fmtc.resample(clip=clip, kernel="spline16", w=1580, h=1080, interlaced=False, interlacedd=False)
# Color Adjustment
clip = core.remap.Rfs(baseclip=clip, sourceclip=adjust.Tweak(clip=clip, hue=0.00, sat=0.00, cont=1.00, coring=True), mappings="[228 479]")
clip = lostfunc.DeSpot(o=clip)
clip = lostfunc.DeSpot(o=clip)
clip = lostfunc.DeSpot(o=clip)
clip = lostfunc.DeSpot(o=clip)
clip = core.remap.Rfs(baseclip=clip, sourceclip=SpotLess.SpotLess(clip=clip, radT=1), mappings="[227 479]")
# adjusting color space from YUV422P16 to YUV444P8 for vsTemporalDegrain
clip = core.resize.Bicubic(clip=clip, format=vs.YUV444P8, range_s="limited")
# removing grain using TemporalDegrain
clip = core.remap.Rfs(baseclip=clip, sourceclip=havsfunc.TemporalDegrain(inpClip=clip), mappings="[0 227]")
# contrast sharpening using CAS
clip = core.remap.Rfs(baseclip=clip, sourceclip=core.cas.CAS(clip=clip, sharpness=0.700), mappings="[0 227]")
# adjusting output color from: YUV444P8 to YUV420P10 for x265Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, range_s="limited")
# set output frame rate to 23.976fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=24000, fpsden=1001)
# Output
clip.set_output()

It mainly filters the first part:

• Select Every
• Crop
• DeSport
• TemporalDegrain
• CAS

and does some adjustments to the second part:

• Select Every
• Crop
• Tweak to limit the colors to gray scale
• DeSpot

will look into some further filtering of the second part.

btw. did you upscale the original to 1080p (noise reduction is easier when done in the original resolution)?

Cu Selur

[lordsmurf]

Ps.: Question in the round: Where do people get such crappy HD content from?

The main culprit, when you have HD from videotapes, is by using crap like OBS, HDMI converters, and Blackmagic cards. That method is essentially putting the video content in a blender, then taking a dump on it, then trying to watch it. Everything is screwed, from interlaced to luma values.

Trying to "fix" it in post is ridiculous. Quality starts with a quality capture. Not a crappy capture and Avisynth.

[Selur]

Had another go at it:

# Imports
import os
import sys
import ctypes
# Loading Support Files
Dllref = ctypes.windll.LoadLibrary("I:/Hybrid/64bit/vsfilters/Support/libfftw3-3.dll")
Dllref = ctypes.windll.LoadLibrary("I:/Hybrid/64bit/vsfilters/Support/libfftw3f-3.dll")
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# Import scripts folder
scriptPath = 'I:/Hybrid/64bit/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))
# Loading Plugins
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/SharpenFilter/CAS/CAS.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/DebandFilter/Flash3kDeband/flash3kyuu_deband.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/SharpenFilter/AWarpSharp2/libawarpsharp2.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/Support/TCanny.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/Support/vcm.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/Support/libmvtools_sf_em64t.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/DenoiseFilter/CTMF/CTMF.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/DenoiseFilter/HQDN3D/libhqdn3d.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/DenoiseFilter/NEO_FFT3DFilter/neo-fft3d.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/Support/libtemporalmedian.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/Support/libmvtools.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/Support/RemapFramesVapoursynth.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/Support/fmtconv.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/vslsmashsource.dll")
# Import scripts
import mvsfunc
import muvsfunc
import G41Fun
import havsfunc
import SpotLess
import lostfunc
import adjust
# source: 'C:\Users\Selur\Desktop\00 ORIGINAL.mov'
# current color space: YUV422P10, bit depth: 10, resolution: 1920x1080, fps: 29.97, color matrix: 709, yuv luminance scale: limited, scanorder: progressive
# Loading C:\Users\Selur\Desktop\00 ORIGINAL.mov using LibavSMASHSource
clip = core.lsmas.LibavSMASHSource(source="C:/Users/Selur/Desktop/00 ORIGINAL.mov")
# making sure input color matrix is set as 709
clip = core.resize.Bicubic(clip, matrix_in_s="709",range_s="limited")
# making sure frame rate is set to 29.970
clip = core.std.AssumeFPS(clip=clip, fpsnum=30000, fpsden=1001)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# adjusting frame with SelectEvery
clip = core.std.SelectEvery(clip=clip, cycle=5, offsets=[0, 1, 2, 3])
clip = core.std.AssumeFPS(clip=clip,fpsnum=24000, fpsden=1001)# new fps: 23.976
# cropping the video to 1580x1080
clip = core.std.CropRel(clip=clip, left=170, right=170, top=0, bottom=0)
clip = core.fmtc.resample(clip=clip, kernel="spline16", w=1580, h=1080, interlaced=False, interlacedd=False)
# Color Adjustment
clip = core.remap.Rfs(baseclip=clip, sourceclip=adjust.Tweak(clip=clip, hue=0.00, sat=0.00, cont=1.00, coring=True), mappings="[228 479]")
clip = lostfunc.DeSpot(o=clip)
clip = lostfunc.DeSpot(o=clip)
clip = SpotLess.SpotLess(clip=clip, chroma=False, radT=1, ablksz=16, aoverlap=8)
# adjusting color space from YUV422P16 to YUV444P8 for vsTemporalDegrain
clip = core.resize.Bicubic(clip=clip, format=vs.YUV444P8, range_s="limited")
# removing grain using TemporalDegrain
clip = core.remap.Rfs(baseclip=clip, sourceclip=havsfunc.TemporalDegrain(inpClip=clip), mappings="[0 227]")
# denoising using mClean
clip = core.remap.Rfs(baseclip=clip, sourceclip=G41Fun.mClean(clip=clip), mappings="[228 479]")
# contrast sharpening using CAS
clip = core.remap.Rfs(baseclip=clip, sourceclip=core.cas.CAS(clip=clip, sharpness=0.700), mappings="[0 227]")
# adjusting output color from: YUV444P16 to YUV420P10 for x265Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, range_s="limited")
# set output frame rate to 23.976fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=24000, fpsden=1001)
# Output
clip.set_output()

attached a lossless encode again.

Cu Selur

Ps.: I'll also compile and upload my current dev version so you can do the things I did with Hybrid.

[dogmydog]

Dear @selur thanks for your tests, your results look AMAZING!
I tried to replicate it what you did here, but I couldnt manage to select which part to filter so I mainly applied to the whole film
Video: PRORES
Audio: passthrough all

Vapoursynth
Denoise:
FFT3DFILTER (default values)
CTMF (default values)
HQDN3D (default values)

Deband:
Flash3kDB (default values)

Sharpen:
aWarpSharp2 (default values)
CAS (sharpness 0,700)

https://drive.google.com/file/d/1bQsgqmaf58QfHEFAn9qtfGIyljMoyzeC/view?usp=sharing

The result is far from what you got, so maybe i need further instructions to apply them correctly.

Oh, you asked before why I used QTCMG to deinterlace even tho the source was progressive it was because I could still see the interlacing lines that remained from the betacam capture. Was that an useless extra step? Thank you!

[Selur]

Oh, you asked before why I used QTCMG to deinterlace even tho the source was progressive it was because I could still see the interlacing lines that remained from the betacam capture. Was that an useless extra step?

Normally adjusting the settings in the deinterlacing should help and make the second call unnecessary.

I tried to replicate it what you did here,...
...
Vapoursynth
Denoise:
FFT3DFILTER (default values)
CTMF (default values)
HQDN3D (default values)

Deband:
Flash3kDB (default values)

Sharpen:
aWarpSharp2 (default values)
CAS (sharpness 0,700)

Those are totally different filters than I used.
I used:
- SelectEvery
- CropRel
- Tweak
- DeSpot
- Spotless
- TemporalDegrain
- mClean
- CAS
also I only applied some of them in a specific range ("Filtering->Vapoursynth->Misc->Ui->Show 'Apply only to'- controls") of the video and in that order ("Filtering->Vapoursynth->Misc->Filter Order/Queue") and most of them with the adjusted values.
So since you used totally different filters and parameters, it's no surprise your result looks different.

Cu Selur

[dellsam34]

Was the Betacam tape shot with a Betacam camcorder or was it a uMatic dub? What the OP needs to do is get a player with SDI output and transfer the raw footage using a SDI to USB adapter. Betacam decks with SDI have a good ADC and image stabilizer inside them.

[dogmydog]

Dear Selur, thanks for the quick answer!
I will have another go at it, I appreciate the patience.

Was the Betacam tape shot with a Betacam camcorder or was it a uMatic dub? What the OP needs to do is get a player with SDI output and transfer the raw footage using a SDI to USB adapter. Betacam decks with SDI have a good ADC and image stabilizer inside them.

Dear dellsam, thanks for joining the conversation!
According to the producer it was an old SD Betacam SP Ampex.
I do not have access to the original in SD, only to the original I linked before, a PRORES upscaled 1080p with crappy quality.

[Selur]

sorry, wrong thread.

[Truthler]

Dear forum friends,

I have been using Hybrid, vapoursynth and ffmpeg for a short time and learning a lot!
What do you think it would be the best way to clean up this footage?


[Attachment 60648 - Click to enlarge]
https://drive.google.com/file/d/1OyiYOeYqCgUz3S2vSN8_OjghObBTM2AZ/view?usp=sharing

Really appreciate the help.
Cheers from Rio,

Use the latest Premiere pro and Neat Video. All other "free" solutions are amateur, so their quality willl be amateur too.

[dogmydog]

[list][*]enable "Filtering->(De)Interlace/Telecine->Deinterlace/Teleince settings->Overwrite the input scan type to" set it to "telecine" and select either "VIVTC (Vapoursynth)" or "TIVTC (Vapoursynth)"[*]enable "Filtering->Vapoursynth->Frame->Reduction->SelectEvery" and enter as offsets "0 1 2 3"[*]set "Filtering->Vapoursynth->Custom->Insert Berfore" to "Deinterlace" enabled the custom section and enter:

clip = core.vivtc.VDecimate(clip, dupthresh=1.1, blockx=256, blocky=256)
# framerate 23.976

Check the Vapoursynth preview move the slider to the far right to see the number of the last frame (frame count = last frame number +1)

Dear @selur, I was following this guide to input the SelectEvery, and when I press the button for the Vapousynth preview, I get the error:
Failed to evaluate the script:
Python exception: There is no function named Median

Traceback (most recent call last):
File "src\cython\vapoursynth.pyx", line 2242, in vapoursynth.vpy_evaluateScript
File "src\cython\vapoursynth.pyx", line 2243, in vapoursynth.vpy_evaluateScript
File "D:\Hybrid\Temp\tempPreviewVapoursynthFile22_05_27_464.vpy", line 89, in <module>
clip = G41Fun.mClean(clip=clip)
File "C:\Program Files\Hybrid\64bit\vsscripts\G41Fun.py", line 2539, in mClean
c = core.vcm.Median(clip, plane=[0, 1, 1]) if chroma else clip
File "src\cython\vapoursynth.pyx", line 1932, in vapoursynth.Plugin.__getattr__
AttributeError: There is no function named Median

Script exceeded memory limit. Consider raising cache size.

Any idea on how to solve this? Thanks!

[Selur]

@dogmydog: try whether replacing Hybrid\64bit\vsfilters\Support\vcm.dll with the x64/vcm.dll from http://www.avisynth.nl/users/vcmohan/vcm/vcm.7z
If it does not I would need a debug output to see the script.
@Truhler: at least have the decency to show what Neat Video can to for that video source if you send him to buy something for xy $,... (also try to not oversharpen things like you seem to like doing)

Cu Selur

[dogmydog]

Dear @selur, thanks for the fix, it worked!
I followed your steps here:

• enable "Filtering->(De)Interlace/Telecine->Deinterlace/Teleince settings->Overwrite the input scan type to" set it to "telecine" and select either "VIVTC (Vapoursynth)" or "TIVTC (Vapoursynth)"
• enable "Filtering->Vapoursynth->Frame->Reduction->SelectEvery" and enter as offsets "0 1 2 3"
• set "Filtering->Vapoursynth->Custom->Insert Berfore" to "Deinterlace" enabled the custom section and enter:
  

  clip = core.vivtc.VDecimate(clip, dupthresh=1.1, blockx=256, blocky=256)
  # framerate 23.976

  Check the Vapoursynth preview move the slider to the far right to see the number of the last frame (frame count = last frame number +1)
  Enable "Filtering->Misc->Overwirte Output"
  Enable "Filtering->Misc->Overwirte Output->Frame count to" and enter the new frame count (whatever the Vapoursynth Preview showed)
  Enable "Filtering->Misc->Overwirte Output->Frame rate to" and enter the new frame rate (23.976)

and then here:

- SelectEvery
- CropRel
- Tweak
- DeSpot
- Spotless
- TemporalDegrain
- mClean
- CAS

in that order ("Filtering->Vapoursynth->Misc->Filter Order/Queue"), but didn't specify to any range or changed default values, I just input in the whole file.

I am more pleased with the look result, but however the final file is out of sync.
https://drive.google.com/file/d/16NOtkyeIc4AcKnK2PXE9mRkDpPpvtG6L/view?usp=sharing

I would input the debug file, but for some reason it jumped from 20mb to 128mb, so I deleted it so I didn't want to waste your time.
Thanks again for all the help!

[Selur]

About the sync: If your audio is synched to 29.97 you need to reencode the audio (Audio->Base->Audio Encoding Options->Filtering->Speed change) and compensate that change. Alternatively you could use some frame interpolation to get back to the original frame rate.

Cu Selur

[poisondeathray]

29.97 to 23.976 via decimation should not change sync. It's the same duration.

There are framedrops (jumps) in the file in post 25. Make sure you only chose one of the options (vdecimate, vs. selectevery) , not all of them

This is what selur posted, which your quote cuts off

To archive the dropping of the frame you could use one of these:

[Selur]

Also post your Vapoursynth script (Filtering->Vapoursynth->Show Vapoursynth Script)

Cu Selur

[dogmydog]

29.97 to 23.976 via decimation should not change sync. It's the same duration.

There are framedrops (jumps) in the file in post 25. Make sure you only chose one of the options (vdecimate, vs. selectevery) , not all of them

This is what selur posted, which your quote cuts off

To archive the dropping of the frame you could use one of these:

Dear poisondeathray, that is exactly my error!! I did all of them combined.
Sorry, english is not my first language, so I need to read the instructions more carefully. I will give it another shot and post results.
Thanks for the patience,

[dogmydog]

Dear friends, sorry for the long late reply.
I have been struggling with work, but thanks to Selur and poisondeathray tips, I managed to succeed.

Here i post the results of Hybrid (filters are here attached)
https://drive.google.com/open?id=13n10DPbfxtDFlrwKKZwEpSH7dxQ1WY5r

and Neat Video.
https://drive.google.com/open?id=1VMrsSZyIi0sWlBax9Alb5dJLc5L0vwS-

I'm not a tech savvy, but I believe I was able to achieve better results with the filters and procedures here suggested than with Neat Video.

Long live Hybrid and this amazing forum
Thanks again for all the help and support