floating chroma shift issue?

[Selur]

Second parameter is float "varC" = variance chroma, which should be sigma_vU, sigma_vV in vsTCanny. Not sigmaU.
more something like:

v2= vsTCanny(v2, sigmaY=50.0, sigma_vU=50.0 + gblur, sigma_vV=50.0 + gblur, mode=-1)

(using 50 instead of 7)

[cinephil]

Well, "=50" - rises the highlights (unlike "=7"), but either way not resolve "border" issue.

Not sure what "variance chroma" mean exactly, but I guess it somewhat different from "Standard deviation of vertical gaussian blur" - which in vsTCanny is sigma_vY, sigma_vU, sigma_vV (and sigmaY, sigmaU, sigmaV - Standard deviation of horizontal gaussian blur.)

[Selur]

It should be the deviation for U and V as I understood it. )
I assume you did simplify your script to stop after the ContrastMask calls to be sure this is not caused by something else, right?

[cinephil]

I assume you did simplify your script to stop after the ContrastMask calls to be sure this is not caused by something else, right?

Well, if I will not tweak after ContrastMask - the picture will be wash out and I won't see any difference.
Right now it look like this:

old ContrastMask

SeparateFields()
MergeChroma(last, Spline36Resize(width/2, height).aWarpSharp2(depth=30).nnedi3_rpow2(2, cshift="Spline36Resize", fwidth=width, fheight=height))
Weave()

ConvertToYV12(matrix="rec601", interlaced=true)

SeparateFields() 
even = SelectEven(last).ContrastMask(enhance=3.0)
odd = SelectOdd(last).ContrastMask(enhance=3.0)
Interleave(even, odd)
Weave()

ConvertBits(16) 

ColorYUV(gamma_y=-35, cont_y=46, gain_y=-10, off_y=0)
Tweak(bright=3, coring=false)
Tweak(cont=0.9)
Tweak(Hue=-16)

FFT3DFilter(bt=-1, sharpen=0.3, interlaced=true)

ConvertBits(bits=8, dither=1)

new ContrastMask

ConvertBits(16) 

SeparateFields()
MergeChroma(last, Spline36Resize(width/2, height).aWarpSharp2(depth=30).nnedi3_rpow2(2, cshift="Spline36Resize", fwidth=width, fheight=height))
Weave()

SeparateFields()
even = SelectEven(last).ContrastMask(enhance=3.0)
odd = SelectOdd(last).ContrastMask(enhance=3.0)
Interleave(even, odd)
Weave()

ColorYUV(gamma_y=-35, cont_y=46, gain_y=-10, off_y=0)
Tweak(bright=3, coring=false)
Tweak(cont=0.9)
Tweak(Hue=-16)

FFT3DFilter(bt=-1, sharpen=0.3, interlaced=true)

ConvertBits(bits=8, dither=1)
ConvertToYV12(matrix="rec601", interlaced=true)

[Selur]

With 'old ContrastMask' you used MergeChroma on 8bit, not you use it on 16bit.

As a general side node:
Your field handling seems odd / wrong to me, can you explain why you do it that way?

• Convert to 16bit (okay)
  Unsure why since afaik. none of the filters you use here does benefit from it, so it's just a slow-down)
• You separate the fields. (okay)
• You do some chroma adjustments on the separated fields. (okay)
• You weave the fields and then separate them again => this seems wrong. Can you explain why you do that?
• You apply ContrastMask on the fields. (okay)
• You re-interlace the fields using Interleave(even, odd) and Weave(). (okay)
• You change the colors of the interlaced content. (okay)
• You apply FFT3DFilter it the interlaced content. (okay)
• You convert to 8bit, while dithering (okay)
• You convert to YV12 (okay)

Personally I would, deinterlace the content, apply the filters and at the end re-interlace the file, instead of working on the separated fields.
The first step for me would be deinterlacing and the last step would be the re-interlacing.

Cu Selur

[cinephil]

With 'old ContrastMask' you used MergeChroma on 8bit, not you use it on 16bit.

MergeChroma part handling some chroma ringing issue, I can't lose that step.

As a general side node:
Your field handling seems odd / wrong to me, can you explain why you do it that way?
[*]Convert to 16bit (okay)
Unsure why since afaik. none of the filters you use here does benefit from it, so it's just a slow-down)

It's for dithering while converting to 8bit, - to remove the banding.

[*]You weave the fields and then separate them again => this seems wrong. Can you explain why you do that?

I think, there was something between this, but not anymore.

Personally I would, deinterlace the content, apply the filters and at the end re-interlace the file, instead of working on the separated fields.
The first step for me would be deinterlacing and the last step would be the re-interlacing.

There is mixed content of film and video (true 29.97 fps) [but not in this 03.avi clip]. So to avoid complication/problems (and not to lose video frames) I stay interlaced.

[Selur]

MergeChroma part handling some chroma ringing issue, I can't lose that step.

You should do that in the original bit depth,...

It's for dithering while converting to 8bit, - to remove the banding.

use a debanding filter instead

There is mixed content of film and video (true 29.97 fps) [but not in this 03.avi clip]. So to avoid complication/problems (and not to lose video frames) I stay interlaced.

bob deinterlace then. You are not staying interlaced, why you do, atm. isn't keeping the content interlaced,..

[cinephil]

So it will be something like this?

AssumeTFF()
Bob(0.0, 1.0)

MergeChroma(last, Spline36Resize(width/2, height).aWarpSharp2(depth=30).nnedi3_rpow2(2, cshift="Spline36Resize", fwidth=width, fheight=height))

ContrastMask(enhance=3.0)

ColorYUV(gamma_y=-35, cont_y=46, gain_y=-10, off_y=0) #cont_y=46
Tweak(bright=3, coring=false)
Tweak(cont=0.9)
Tweak(Hue=-16)

FFT3DFilter(bt=-1, sharpen=0.3)

neo_f3kdb(range=31, keep_tv_range=true, grainY=0, grainC=0, output_depth=8)

ConvertToYV12(matrix="rec601")

AssumeTFF.SeparateFields().SelectEvery(4,0,3).Weave()

I will do some testing tomorrow. But debanding filter, as I see on the Histogram - not doing good enough to replace ->16bit->8bit(dithering).

[Selur]

I would probably use QTGMC in fast mode, but yeah, something like that.

[cinephil]

MergeChroma part handling some chroma ringing issue, I can't lose that step.

You should do that in the original bit depth,...

That definitely fix the "border" issue!

There is mixed content of film and video (true 29.97 fps) [but not in this 03.avi clip]. So to avoid complication/problems (and not to lose video frames) I stay interlaced.

bob deinterlace then. You are not staying interlaced, why you do, atm. isn't keeping the content interlaced

With BOB the picture looks softer. I thought it would be the other way around.
I bumped sharpen from 0.3 to 0.5 - to get similar to the old script look.
* Although sharpen- 0.5 is a bit too much (bring more noise), - 0.4 is fine, I guess...

AssumeTFF()
Bob(0.0, 1.0)

MergeChroma(last, Spline36Resize(width/2, height).aWarpSharp2(depth=30).nnedi3_rpow2(2, cshift="Spline36Resize", fwidth=width, fheight=height))

ConvertBits(16)

ContrastMask(enhance=3.0)

ColorYUV(gamma_y=-35, cont_y=46, gain_y=-10, off_y=0) #cont_y=46
Tweak(bright=3, coring=false)
Tweak(cont=0.9)
Tweak(Hue=-16)

FFT3DFilter(bt=-1, sharpen=0.5)

AssumeTFF.SeparateFields().SelectEvery(4,0,3).Weave()

ConvertBits(bits=8, dither=1)
ConvertToYV12(matrix="rec601")

[Selur]

That definitely fix the "border" issue!

so, that was not caused by ContrastMask

With BOB the picture looks softer.

]
Like I said, I would not use the simple default Bob filter here. Also for light sharpening, try CAS.

[cinephil]

Turns out that BOB is not that bad. But sharpen with FFT3DFilter work better after reinterlacing.
------------------
Now getting back to the clip from my first post - after taken care of the chroma bleeding, I'm still trying to fix halo/ghost issue.
With FineDehalo it's look better, but I still have one bright halo/ghost.

[Attachment 82903 - Click to enlarge]

Is it possible to get rid of that line without degrade picture too much furter?

AssumeTFF()
Bob(0.0, 1.0)

FineDehalo2(hconv = "-1 -1 1 1 40 1 1 -1 -1", vconv = "-1 -1 1 1 40 1 1 -1 -1", showmask=0)
FineDehalo(rx=3.0, ry=2.0, thmi=30, thma=158, thlimi=50, thlima=100, darkstr=1.2, brightstr=1.5, showmask=0, contra=0.0, excl=true)

ConvertBits(32)
ConvertToPlanarRGB(matrix="Rec601")
mlrt_ncnn(network_path="1x_BleedOut_Compact_300k_net_g.onnx")

ConvertToYUV444(matrix="Rec601")

ColorYUV(gamma_y=-35, cont_y=48, gain_y=-10, off_y=0)
Tweak(bright=12)

AssumeTFF.SeparateFields().SelectEvery(4,0,3).Weave()

FFT3DFilter(bt=-1, sharpen=1.0, interlaced=true)

ConvertBits(8)
ConverttoYv12(matrix="Rec601", interlaced=true)

[jagabo]

FineDehalo doesn't work well with wide halos. Try something like dehalo_alpha(rx=7, ry=1, BrightStr=0.8, DarkStr=0.8). Use an edge mask to limit it to the strongest edges (thus protecting the rest of the image).

Even dehalo_alpha works better with smaller halos. So try downscaling horizontally, dehalo_alpha, upscale horizontally and an edges mask.

[cinephil]

Thanks, jagabo!

Maybe I do something wrong with dehalo_alpha (or with edge mask), since don't got much improvement using it?

Spline36Resize(width/2, height)
emask=mt_edge(mode="prewitt", thy1=0, thy2=255, U=2, V=2)
Overlay(last, dehalo_alpha(rx=2.0, ry=1, BrightStr=0.8, DarkStr=1.2), mask=emask)
Spline36Resize(width*2,height)

But I got much better results from FineDehalo (although don't eliminated this bright line completely).

Spline36Resize(width/2, height)
FineDehalo2(hconv = "-1 -1 1 1 40 1 1 -1 -1", vconv = "-1 -1 1 1 40 1 1 -1 -1", showmask=0)
FineDehalo(rx=2.0, ry=1.0, thmi=60, thma=128, thlimi=50, thlima=100, darkstr=1.2, brightstr=0.8, showmask=0, contra=0.0, excl=true)
Spline36Resize(width*2,height)

[Attachment 82954 - Click to enlarge]

[Selur]

You used a different rx value than jagabo suggested.

[cinephil]

You used a different rx value than jagabo suggested.

Well, rx=7 was suggested for no downscale-upscale involving. And it not look good.

emask=mt_edge(mode="prewitt", thy1=0, thy2=255, U=2, V=2)
Overlay(last, dehalo_alpha(rx=7, ry=1, BrightStr=0.8, DarkStr=0.8), mask=emask)

[Attachment 82956 - Click to enlarge]

With downscale-upscale

Spline36Resize(width/2, height)
emask=mt_edge(mode="prewitt", thy1=0, thy2=255, U=2, V=2)
Overlay(last, dehalo_alpha(rx=7, ry=1, BrightStr=0.8, DarkStr=0.8), mask=emask)
Spline36Resize(width*2,height)

[Attachment 82959 - Click to enlarge]

I tried rx=3.5 too

Spline36Resize(width/2, height)
emask=mt_edge(mode="prewitt", thy1=0, thy2=255, U=2, V=2)
Overlay(last, dehalo_alpha(rx=3.5, ry=1, BrightStr=0.8, DarkStr=0.8), mask=emask)
Spline36Resize(width*2,height)

[Attachment 82960 - Click to enlarge]

[Selur]

Have you tried applying YAHR two times?

YAHR()
YAHR()

[jagabo]

Here's what I used:

LWLibavVideoSource("FineDeHalo-before-after.png", cache=false, prefer_hw=2) 
ConvertToYV12(interlaced=true)
Crop(0,0,width/2,height)
src = last

emask = mt_edge(thy1=8, thy2=8).mt_expand().mt_expand().mt_expand().mt_expand().mt_expand().Blur(1.5).Blur(1.5)
dh = dehalo_alpha(rx=7, ry=1, BrightStr=0.8, DarkStr=0.8)
Overlay(src, dh, mask=emask)

[Attachment 82962 - Click to enlarge]

I didn't supply the script earlier because I didn't have the original video to work with.

[Selur]

@jagabo: afaik. the video should be 05-5D[JP]-01.avi from the first post

DeHalo_alpha()
YAHR()
YAHR()
seesm fine too. since YAHR with larger depth and stronger blur is also possible

[cinephil]

Here's what I used:

LWLibavVideoSource("FineDeHalo-before-after.png", cache=false, prefer_hw=2) 
ConvertToYV12(interlaced=true)
Crop(0,0,width/2,height)
src = last

emask = mt_edge(thy1=8, thy2=8).mt_expand().mt_expand().mt_expand().mt_expand().mt_expand().Blur(1.5).Blur(1.5)
dh = dehalo_alpha(rx=7, ry=1, BrightStr=0.8, DarkStr=0.8)
Overlay(src, dh, mask=emask)

I didn't supply the script earlier because I didn't have the original video to work with.

@jagabo: afaik. the video should be 05-5D[JP]-01.avi from the first post

Sorry for inconvenience. Yes the video source is in #1 post 05-5D[JP]-01.avi

[jagabo]

With that source, something like this:

LWLibavVideoSource("05-5D[JP]-01.avi", cache=false, prefer_hw=2) 
TFM()
TDecimate()
vInverse()
MergeChroma(last, Spline36Resize(width/2, height).aWarpSharp2(depth=30).nnedi3_rpow2(2, cshift="Spline36Resize", fwidth=width, fheight=height))
SMDegrain(tr=2, thsad=200, PreFilter=6)
Tweak(sat=2.0, bright=-15, cont=1.15)

# dehalo with edge mask
emask = Spline36Resize(400, height).mt_edge(thy1=10, thy2=10).mt_expand().mt_expand(mode="horizontal").Blur(1.0).GreyScale().nnedi3_rpow2(2, cshift="spline36Resize", fwidth=720, fheight=height)
dh = Spline36Resize(400, height).dehalo_alpha(rx=3.5, ry=1, BrightStr=0.0, DarkStr=0.8).Spline36Resize(width, height).Sharpen(0.5, 0.0)
Overlay(last, dh, mask=emask)

[jagabo]

An even more selective edge mask would be:

emask = Spline36Resize(400, height).mt_edge(mode="-1 2 -1    -1 2 -1   -1 2 -1   6").mt_expand().mt_expand(mode="horizontal").Blur(1.0).GreyScale().nnedi3_rpow2(2, cshift="spline36Resize", fwidth=720, fheight=height)

It only detects vertical edges.