Removing dark artifacts around halo's

[killerteengohan]

I have a source that has a pretty bad case of halos. I can do fairly decent at getting rid of the white with dehalo_alpha(rx=1.9, ry=1.9, darkstr=0.4, lowsens=0, highsens=50, ss=1.0), but there's black lines, and or dark artifacts along with them, and I cannot seem to clean those.

Here is a couple screenshots from the source
https://forum.videohelp.com/images/imgfiles/RUvU5wx.png
https://forum.videohelp.com/images/imgfiles/O2eRGvO.png

You can see what I am referring to best if you look at characters mouth area, and lips.
Anyone have any suggestions on how I clean those darker artifacts around the halo's as well?

[jagabo]

Those are secondary halos -- halos of the halos. Here's what I came up with:

Spline36Resize(width/2, height/2)
dehalo_alpha(rx=2.0, ry=2.0, brightstr=1.5, lowsens=0, highsens=0)
FastLineDarkenMod()
nnedi3_rpow2(2, cshift="Spline36Resize", fwidth=src.width, fheight=src.height)
aWarpSharp(depth=5)
Sharpen(0.5)

I only looked at the second image.

[SaurusX]

finedehalo2() is meant to specifically address the dark halos on the outside of bright halos.

[killerteengohan]

Can someone give me the avsi for finedehalo2? No matter what I try, mine keeps saying there's a problem with line9 and it will not work at all.

EDIT: Nevermind. It wont work by putting loadplugin in the script for whatever reason. It does however work when I put it into the autoload folder. Im not sure why using loadplugin gives me problems. It was doing the same thing when I tried to load fastlinedarkenmod as well. I had to put them both into autoload.

[jagabo]

FineDehalo is a script, not a dll. Use Import(), not LoadPlugin().

[jagabo]

This is about the best I could do with FineDehalo() and FineDehalo2()

BilinearResize(1440, 612)
FineDehalo(rx=2.0, ry=2.0, thmi=0, brightstr=1.5)
FineDehalo2()
nnedi3_rpow2(2, cshift="Spline36Resize", fwidth=src.width, fheight=src.height)
aWarpSharp(depth=5)
Sharpen(0.3)

FineDehalo2() is reducing the residual dark lines but it's also creating some mild bright halos.

[SaurusX]

This is about the best I could do with FineDehalo() and FineDehalo2()

BilinearResize(1440, 612)
FineDehalo(rx=2.0, ry=2.0, thmi=0, brightstr=1.5)
FineDehalo2()
nnedi3_rpow2(2, cshift="Spline36Resize", fwidth=src.width, fheight=src.height)
aWarpSharp(depth=5)
Sharpen(0.3)

FineDehalo2() is reducing the residual dark lines but it's also creating some mild bright halos.

Findehalo2 is best used in this fashion:

FineDehalo2(hconv = "-1 -1 1 1 40 1 1 -1 -1", vconv = "-1 -1 1 1 40 1 1 -1 -1")

Adjust the positive 1's in the hconv and vconv parameters upward if you need more dark halo removal strength, but usually you want to keep things symmetric. So, something like this if you need more horizontal dark halo removal and just light vertical halo removal:

FineDehalo2(hconv = "-1 -1 5 5 40 5 5 -1 -1", vconv = "-1 -1 1 1 40 1 1 -1 -1")

I wouldn't go much above 7 unless you absolutely have to. I've gone to 9 before, but watch out for side effects. The "40" value can be adjusted downward if you need a wider spread of the affected halo areas.

And yes, it can cause some very mild bright halos, which is why I'd advise to use it first, before dehalo_alpha(), and then compensate.

[killerteengohan]

@Jagabo
That certainly cleaned them that's for sure, but it horribly ruined the video, and smeared everything to death. Id rather keep the black artifacts than do that excessive amount of detail loss and smearing lol.
https://slow.pics/c/HfEU3AAi

@SaurusX
It did a little bit of cleaning, but it's not all that noticeable. What is VERY noticeable is this filter is adding white halo like glowing lines along the darker lines and other places.
https://slow.pics/c/dktjv72p
Look at the guys eyebrows for easiest viewing when flipping between the comparison images. Its doing that just above his belt over the white shirt. It gets a white glowing area added in when this filter is put in. It does this as well on several other places, even when I use it before dehalo alpha.

[SaurusX]

@SaurusX
It did a little bit of cleaning, but it's not all that noticeable. What is VERY noticeable is this filter is adding white halo like glowing lines along the darker lines and other places.
https://slow.pics/c/dktjv72p
Look at the guys eyebrows for easiest viewing when flipping between the comparison images. Its doing that just above his belt over the white shirt. It gets a white glowing area added in when this filter is put in. It does this as well on several other places, even when I use it before dehalo alpha.

I don't need to look. I know what you're talking about. Check my message above for proper usage. Good luck!

[killerteengohan]

FineDehalo is a script, not a dll. Use Import(), not LoadPlugin().

Ah, that explains it. Thanks!

[killerteengohan]

@SaurusX
It did a little bit of cleaning, but it's not all that noticeable. What is VERY noticeable is this filter is adding white halo like glowing lines along the darker lines and other places.
https://slow.pics/c/dktjv72p
Look at the guys eyebrows for easiest viewing when flipping between the comparison images. Its doing that just above his belt over the white shirt. It gets a white glowing area added in when this filter is put in. It does this as well on several other places, even when I use it before dehalo alpha.

I don't need to look. I know what you're talking about. Check my message above for proper usage. Good luck!

Yeah, that reply was not up when I was making mine and I did not see it until after I posted the reply. I will tinker with those settings and see what I can get. Thanks!

[killerteengohan]

Yeah SaurusX those settings you suggested were much better. I barely even notice what it does introduce compared to before without comparison to source.
The amount of glowing introduced is further reduced with these settings ((hconv = "-1 0 1 1 40 1 1 0 -1", vconv = "0 -1 1 1 40 1 1 -1 0)) It also kept lines slightly more in place.

With ideas from both you and Jagabo this was about the best middle ground I could come up with. I think it looks much better.

https://slow.pics/c/rqycwasC

Thanks for the ideas guys.