Multiple side by side lines of halo's fix?

[killerteengohan]

This Blu-Ray I'm working on has a weird halo issue where it looks like multiple lines of halo's side by side. I have tried a combination of dehalo filters and had no luck in fixing it.

Here are some screenshots.

https://forum.videohelp.com/images/imgfiles/k0U8uja.png (It seems to be visible everywhere in this image)
https://forum.videohelp.com/images/imgfiles/CIm68bY.png (Very visible on the Blue and Grays)
https://forum.videohelp.com/images/imgfiles/Tl5XEEM.png (Orange on her shirt has it easily visible)
https://forum.videohelp.com/images/imgfiles/H78lKkU.png (All of the red areas have it easily visible)

What would you suggest as a fix if this can be fixed?

[Selur]

What would you suggest as a fix if this can be fixed?

as long as it's for simple anime/cartoon: MiniDeen
(try 5x5 radius)

[killerteengohan]

While that does appear to be helping, it also appears to be smearing lines and other things from DNR. It's also making banding worse. I tried regular deen as well, and it also helps, but the DNR is smudging things just the same.

I even tried something Jagabo taught me not too long ago where I can filter only the chroma, but it's not really working when I use it with that method. It barely made an impact when compared to just using it on everything.

U = UtoY().deen(mode="c3d", rad=2, thrY=5, thrUV=9, tthY=2, tthUV=9, min=0.5, scd=9.0, fcf="", borderfix=false)
V = VtoY().deen(mode="c3d", rad=2, thrY=5, thrUV=9, tthY=2, tthUV=9, min=0.5, scd=9.0, fcf="", borderfix=false)
YtoUV(U, V, last)


I also tried this so I would not be using DNR, and while it did help, it's not really working well enough.

MergeChroma(BicubicResize(width/2,height/2).BicubicResize(1920,1080).aWarpSharp2(depth=10)) .Sharpen(0.3)


Is there no chroma only or luma only fix where I do not need to use DNR?

[poisondeathray]

While that does appear to be helping, it also appears to be smearing lines and other things from DNR. It's also making banding worse. I tried regular deen as well, and it also helps, but the DNR is smudging things just the same.

Anything that "denoises" will cause banding on this type of cartoon in certain areas. On the 1st pic, on the dark areas of the jacket - notice a dither or grain pattern in the original. If you denoise and remove that - that will usually become "splotches" and a form of banding



If you want to reduce the halos, but minimize softening of the picture, you can use waifu2x with mild denoising settings and line preservation. Line preservation is a major aspect in what it was trained for.

Model 1, Noise 1 works ok. I ran it twice with bicubic . waifu2xcaffe works well in vapoursynth if you have a nvidia card. There are other methods, other GUI's that can use waifu2x too, some with vulcan . I'm not up to date on all the waifu2x variations, but there are dozens (outside of vapoursynth too) . The only thing you should avoid is waifu2x with avisynth. It's not optimized at all , about 100x slower . If you choose to go this route, you'd generally want to dither or add grain or fix the areas that were denoised that left banding


[Attachment 58219 - Click to enlarge]

I even tried something Jagabo taught me not too long ago where I can filter only the chroma, but it's not really working when I use it with that method. It barely made an impact when compared to just using it on everything.

U = UtoY().deen(mode="c3d", rad=2, thrY=5, thrUV=9, tthY=2, tthUV=9, min=0.5, scd=9.0, fcf="", borderfix=false)
V = VtoY().deen(mode="c3d", rad=2, thrY=5, thrUV=9, tthY=2, tthUV=9, min=0.5, scd=9.0, fcf="", borderfix=false)
YtoUV(U, V, last)


I also tried this so I would not be using DNR, and while it did help, it's not really working well enough.

MergeChroma(BicubicResize(width/2,height/2).BicubicResize(1920,1080).aWarpSharp2(depth=10)) .Sharpen(0.3)


Is there no chroma only or luma only fix where I do not need to use DNR?

This strategy of separating Y,U,V processing won't help much in this case, because it affects Y,U,V

You can use Greyscale , or UtoY, or VtoY separately to visualize each plane, and you will see the artifact affects all 3 planes

#original YV12 source
ConvertToYV24() # resample to 4:4:4
Greyscale()
#UtoY
#VtoY

[killerteengohan]

I know how to fix the banding, and I know that's a side effect of DNR, that's not really a problem. The problem is the DNR is destroying the image in certain scenes or spaces.

Look what it's doing to the image when looking out the cars back window.

https://slow.pics/c/ntz5fjuS (Flip through images with keyboard arrows or mouse click)

Entire lines are disappearing off buildings, and the DNR is too strong. I like how it cleans the halo lines I wanted cleaned, but it's being too destructive.

[Selur]

The problem is the DNR is destroying the image in certain scenes or spaces.

I thought this is about image filtering not video filtering,...

Entire lines are disappearing off buildings, and the DNR is too strong. I like how it cleans the halo lines I wanted cleaned, but it's being too destructive.

adjusting the thresholds and applying different filter setting to different parts of the video might help,...

Cu Selur

[killerteengohan]

The problem is the DNR is destroying the image in certain scenes or spaces.

I thought this is about image filtering not video filtering,...

Entire lines are disappearing off buildings, and the DNR is too strong. I like how it cleans the halo lines I wanted cleaned, but it's being too destructive.

adjusting the thresholds and applying different filter setting to different parts of the video might help,...

Cu Selur

I already did that and played with the settings from the beginning. Even at its much lower settings, it was still doing more harm than I wanted and began to be less effective at removing the halo lines as I lowered them. Deen is typically a destructive to detail DNR filter, especially in darker areas. It's why I have not used it in years, and replaced it with other DNR filters.

I tried lessening the smearing by doing a mergeluma and it helps, but its not really good enough.

source = last
deen(mode="c3d", rad=2, thrY=5, thrUV=9, tthY=2, tthUV=6, min=0.5, scd=9.0, fcf="", borderfix=false)
MergeLuma(last, source, weight=0.5)

If I raise the weight to about 0.7 or more, its no longer doing as much damage to lines, but it is also not removing those halo lines anymore.

This is why I was hoping someone had a non DNR solution to suggest.

[jagabo]

Try this:

#ImageSource("k0U8uja.png", start=0, end=23, fps=23.976) 
ImageSource("unknown_27069.png", start=0, end=23, fps=23.976) 
ConvertToYV12()

lmask1 = mt_edge(thy1=10, thy2=10, mode="0 0 0   -4 8 -4  0 0 0") # vertical edges

lmask2 = lmask1.mt_edge(mode="0 0 0 1 1 1 0 0 0") # expand horizontally
lmask2 = lmask2.mt_edge(mode="0 0 0 1 1 1 0 0 0") # expand horizontally
lmask2 = lmask2.mt_edge(mode="0 0 0 1 1 1 0 0 0") # expand horizontally
lmask2 = lmask2.mt_edge(mode="0 0 0 1 1 1 0 0 0") # expand horizontally
lmask2 = lmask2.mt_edge(mode="0 0 0 1 1 1 0 0 0") # expand horizontally
lmask2 = lmask2.mt_edge(mode="0 0 0 1 1 1 0 0 0") # expand horizontally
lmask2 = lmask2.mt_edge(mode="0 0 0 1 1 1 0 0 0") # expand horizontally
lmask2 = lmask2.mt_edge(mode="0 0 0 1 1 1 0 0 0") # expand horizontally
lmask2 = lmask2.mt_edge(mode="0 0 0 1 1 1 0 0 0") # expand horizontally
lmask2 = lmask2.mt_edge(mode="0 0 0 1 1 1 0 0 0") # expand horizontally
lmask2 = lmask2.mt_edge(mode="0 0 0 1 1 1 0 0 0") # expand horizontally
lmask2 = lmask2.mt_edge(mode="0 0 0 1 1 1 0 0 0") # expand horizontally
lmask2 = lmask2.mt_edge(mode="0 0 0 1 1 1 0 0 0") # expand horizontally
lmask2 = lmask2.mt_inpand().mt_expand().mt_expand().mt_expand() # inpand to remove lone dots, expand some more

lmask = Overlay(lmask2, lmask1, mode="Subtract").Blur(1.0).Blur(1.0).Blur(1.0).GreyScale() # protect original lines

Overlay(last, MiniDeen(2), mask=lmask)