Dehalo and Chroma Sharpening?

[killerteengohan]

I been using the halo remover HQDering(Parameters vary) and its been working great for me with little side effects or damage usually.

Problem with this thing is, it also uses a smoother/denoise in it. I can lessen the strength of it by changing parameters or using an alternate smoother/denoise method. Usually the lower I make it, the less it works and I can also lessen the strength of the denoise it does to where its just enough for removal and retain more detail. If I turn off the smoother/denoiser then the halo removal doesn't even work anymore so I haven't been able to do this without smoothing/denoising as well using this filter.

I have a source that I do not wish to denoise at all and want to retain all the details. Is there a halo remover I can use that will not denoise/smooth or is that a requirement of any dehalo/dering filter?

The one I usually use isn't doing me any good when it comes to retaining fine details. For example, its smearing everything around the letters on this book in his hand even when I turned the strength of the smoother/denoiser lower. Any lower than this setting and it has no real effect at removing the halo's. It was also doing this in other places as well. A common side effect of using deen.

BEFORE


AFTER

Here is a sample of the source I believe needs halo removal and I'm trying to get them off of.

https://mega.co.nz/#!4tI2WKrZ!q5tWVLpEp-1lwolqAIun7sgupx2WhuO9IWV7GqkNsB4

https://mega.co.nz/#!ggA1iIxS!HL9UL9wJtGHb-BcRYi4iqKFt4MquZGVv9N67eW0fQoc

The first sample seems to have it alot worse than the majority of the other episodes.



Also, would you say this needs chroma sharpened? I'm not 100% sure how to tell when it's needed, I just try it when lines look blurry, but it certainly looks like it makes it look a bit better along the lines when I tried it.
What I tried was MergeChroma(awarpsharp2(depth=4)) for Chroma sharpening and HQDering(smoother=deen("a3d",1,5,2)) for Dehalo.



My current script I'm using is this.

tfm(order=1).tdecimate(mode=1,hybrid=0)
crop( 2, 2, -4, 0)
LanczosResize(848,480)
maa2(ss=3.00)
LimitedSharpenFaster(ss_x=1.0, ss_y=1.0, strength=6, overshoot=0, undershoot=0, soft=0, edgemode=0)

[jagabo]

Dehalo_alpha(rx=3.0, ry=0.01)

You'll want to play around with the thresholds and strengths too.

One easy way to see the effect of chroma sharpening is to crank up the saturation. You can also use UtoY() and VtoY().

ColorYUV(cont_u=500, cont_v=500)
StackHorizontal(last, StackVertical(UtoY(), VtoY()))

[Skiller]

I don't see any halos in both images.

Maybe the softness of light (blooming/diffusion) is what you think is what dehalo_alpha is meant to treat but that's not what it does – it's to reduce oversharpening by removing artifical contours around real ones.

[jagabo]

There are very obvious halos in the sample video he uploaded.

[Skiller]

I see.
Downloading from MEGA just doesn't work for me (takes forever, never starts) so I just look at the images.

[newpball]

There are very obvious halos in the sample video he uploaded.

Very obvious?

[jagabo]

Example:


Even the halos have halos.

[newpball]

Oh the video, I only looked at the image he posted.

[killerteengohan]

Oh the video, I only looked at the image he posted.

The images were only meant to show the damage my current dehalo filter I always use was doing to the fine details. Those images werent showing the halo's, thats why I put up the video's.

[killerteengohan]

There are very obvious halos in the sample video he uploaded.

What about that second video Jagabo? The first one has it worse than any other episode in the series. That second one is the usual I have to deal with and didn't look nearly as bad. Same filter?

Also, I still haven't been able to get the Dehalo_Alpha to work for me but once in the past on a computer I no longer have. I think I need a specific Microsoft C++ for it to work. Do you know the exact year I need? I know I have the needed plugins and avsi file.

[jagabo]

The haloing in the second video is so light it's not worth worrying about. The damage caused by dehaloing will be far worse than the problem.

Dehalo_alpha.avsi uses masktools2. Be sure you have that. If you are getting a message about a missing dll download and install that.

But if you're concerned about the small amount of damage in your sample images you're not going to be happy with Dehalo_alhpa(). Halos are caused by sharpening filters. So removing them involves selectively blurring them away. So you will get loss of detail near the halos.

Well, here's something that helps with the light halos of the second video without causing as much damage as other dehalo filters:

blurred = Blur(0.5, 0.0) # enough horizontal blur to reduce halos
vedges=mt_edge(mode="0 0 0 -1 2 -1 0 0 0").mt_expand().mt_inpand().mt_edge(mode="0 0 0 -1 2 -1 0 0 0").ColorYUV(gain_y=-100).GreyScale()
Overlay(last,  blurred,  mask=vedges)

[killerteengohan]

One easy way to see the effect of chroma sharpening is to crank up the saturation. You can also use UtoY() and VtoY().

ColorYUV(cont_u=500, cont_v=500)
StackHorizontal(last, StackVertical(UtoY(), VtoY()))

So how do I use this properly? I put it at the top of the script and the saturation rose ALOT on the picture then theres 2 greyscale looking images on the right. Does this go at the top or bottom of the script?

Do I apply the chroma sharpen right after it or right before it?

Would you use chroma sharpening on this source?

[killerteengohan]

The haloing in the second video is so light it's not worth worrying about. The damage caused by dehaloing will be far worse than the problem.

Dehalo_alpha.avsi uses masktools2. Be sure you have that. If you are getting a message about a missing dll download and install that.

But if you're concerned about the small amount of damage in your sample images you're not going to be happy with Dehalo_alhpa(). Halos are caused by sharpening filters. So removing them involves selectively blurring them away. So you will get loss of detail near the halos.

Well, here's something that helps with the light halos of the second video without causing as much damage as other dehalo filters:

blurred = Blur(0.5, 0.0) # enough horizontal blur to reduce halos
vedges=mt_edge(mode="0 0 0 -1 2 -1 0 0 0").mt_expand().mt_inpand().mt_edge(mode="0 0 0 -1 2 -1 0 0 0").ColorYUV(gain_y=-100).GreyScale()
Overlay(last,  blurred,  mask=vedges)

Thanks!! I think I will go with just leaving them alone and only using it on that one episode that had it horribly.

[jagabo]

One easy way to see the effect of chroma sharpening is to crank up the saturation. You can also use UtoY() and VtoY().

ColorYUV(cont_u=500, cont_v=500)
StackHorizontal(last, StackVertical(UtoY(), VtoY()))

So how do I use this properly? I put it at the top of the script and the saturation rose ALOT on the picture then theres 2 greyscale looking images on the right. Does this go at the top or bottom of the script?

Do I apply the chroma sharpen right after it or right before it?

The increased saturation makes bleeding or shifted colors more obvious. The two small grey images are the U and V channels (YV12 video has U and V at half the width and half the height of the Y channel) viewed as greyscale. You put this after the chroma sharpening to see the result of the sharpening. Once your satisfied with the sharpened colors you remove this code.

Would you use chroma sharpening on this source?

The first sample could use a little chroma sharpening. You need to crank aWarpSharp up to 10 or so to get significant sharpening.

[killerteengohan]

The haloing in the second video is so light it's not worth worrying about. The damage caused by dehaloing will be far worse than the problem.

Dehalo_alpha.avsi uses masktools2. Be sure you have that. If you are getting a message about a missing dll download and install that.

I have masktools2 and should have all .dll's needed.

The error I seem to be getting at the moment is this

Script error: there is no function named "expand"
(DehHalo_alpha.avsi, line 21)

Usually when it says no function, I'm missing a plugin but I'm not sure why its saying that. I have every single plugin I had on my other computer because I copied them all over and it worked fine on my other computer.

This isn't a Microsoft C++ problem?

[killerteengohan]

I just tried another computer after copying all my plugins over and it works fine on that computer, so somethings wrong.

Anyways I tried what you suggested Jagabo and it removes them, though I prefer the rx at 1.6 - 2.0 because 3 was too strong for my liking.

Why did you make ry so low compared to rx? is ry a bad thing?

[jagabo]

I just tried another computer after copying all my plugins over and it works fine on that computer, so somethings wrong.

expand() is part of the older version of masktools. My guess is you have an older version of dehalo_alpha() on the computer with problems. The new version uses mt_expand() instead.

Anyways I tried what you suggested Jagabo and it removes them, though I prefer the rx at 1.6 - 2.0 because 3 was too strong for my liking.

You need to use about 3.0 to get rid of the halos. 1.6 to 2.0 won't completely get rid of them. Of course, the higher the radius the more damage you get. You should try playing with the sensitivity and strength values too. See the docs.

Why did you make ry so low compared to rx? is rx a bad thing?

There's no ringing on the Y axis, only on the X axis. You can't set RY to 0.0 so I set it to a very small value to "disable" it. Analog tape captures usually only have oversharpening halos on the X axis. Usually only digital sharpening filters will create halos on the Y axis.