Creating an edgemask to protect edges/detail when denoising

[leonsk]

Been trying to achieve this in VapourSynth for hours, but it doesn't seem to work.
First I prepare an edgemask like this:

edgemask = kf.retinex_edgemask(src)
luma = core.std.ShufflePlanes(edgemask, 0, colorfamily=vs.GRAY)

Then denoise:

denoise = mvf.BM3D(src, sigma=[100.0,100.0,100.0], radius1=1)

Finally merging:

core.std.MaskedMerge(src, denoise, edgemask)

All I get is the source plus white lines, no denoising (That's why the sigma is ridiculously high - to be sure it's working or not).

The source is YUV420P10, if that matters.

[_Al_]

Looking at the code, should not be luma clip actually your mask? Not edgemask.

core.std.MaskedMerge(src, denoise, luma)

[leonsk]

Yep, it was like that, but getting the same result I thought the problem was the format conversion so I tried using it directly and forgot to change back.
It's the same with both luma and edgemask.

[poisondeathray]

The retinex_edgemask function should automatically return vs.GRAY anyways, "luma" is redundant there


what does denoise look like by itself ?

what does your edgemask look like by itself?

You can try replacing "denoise" temporarily with a colored clip to visualize what is going on (and where) , then adjusting the mask

[leonsk]

@pdr
At first Sobel was used.
The way it should, heavily blurred and washed out.
BW with thicker lines for retinex and thinner lines for Sobel+luma
Any clip? Or I should generate it inside VS?

[poisondeathray]

That function's edgemask values look incorrect. Trying on some random 10 bit videos, I see G values over 2000 . Should be 0-1023 for Gray10

[poisondeathray]

Any clip? Or I should generate it inside VS?

you can generate it in vs . e.g. for YUV 10bit, a "lime green" clip, or pick another color. Something easy to see

test = core.std.BlankClip(src, color=[800, 400,400])

core.std.MaskedMerge(src, test, edgemask)


The out of range values can make the affected areas "white" . (in my clip, the lines are super white, from the retinex_edgemask function > 1023 for the edgemask) So add limiter for 10bit

edgemask = core.std.Limiter(edgemask, min=0, max=1023)

eg. in my lime green viz test, the final lines were white, until the mask values were brought into 10bit 0-1023 range, then they were green, which is what you would expect


That's filtering lines. (100% white mask would be 100% filtered). So if you want to "protect" lines as the title suggets, you'd normally want to invert the mask



It's late here, I'll have to revisit tomorrow if you and _Al_ haven't figured it out tonight. But I think it's that function with out of range values

[leonsk]

Mind you I tried using format='YUV420P8' as LSMASH argument and then the white lines were gone, but couldn't figure out why that fixed it. Thank you for suggesting the Limiter.

I think I've misunderstood what MaskedMerge actually does. I though it takes clip A then adds clip B and then the mask.
Based on your explanation above I guess it applies the filter from clip B based on the selection mask to clip A, am I correct?
So I should take the edgemask then invert it so that it would be my selection where I want to filter, and that wouldn't touch the edges because they wouldn't be selected?

edgemask = core.std.Limiter(edgemask, min=0, max=1023).std.Invert()

[poisondeathray]

Mind you I tried using format='YUV420P8' as LSMASH argument and then the white lines were gone, but couldn't figure out why that fixed it. Thank you for suggesting the Limiter.

I think I've misunderstood what MaskedMerge actually does. I though it takes clip A then adds clip B and then the mask.
Based on your explanation above I guess it applies the filter from clip B based on the selection mask to clip A, am I correct?
So I should take the edgemask then invert it so that it would be my selection where I want to filter, and that wouldn't touch the edges because they wouldn't be selected?

edgemask = core.std.Limiter(edgemask, min=0, max=1023).std.Invert()

If you check retinex_edgemask results in 8bit, it only goes to 255. In the same areas, it goes >1023 in 10bit for some reason. In 12bit it goes > 4095 (I'm seeing values like 8000) . You can examine the function in more detail if you want, but it seems like a bug to me. Maybe I have an older version, not sure

Yes, 100% white mask means 100% of clip B is applied to those areas. 0% white (100% black) means clip A. So white lines means the lines are clip B . Intermediate values means intermediate values of A and B

Another approach is composite the original lines back, or a lightly filtered version of the lines back on "top" (think of it as multiple layers, as if you were using photoshop or gimp or image editor)

[leonsk]

How do you check for those exceeded values?
I see that the function uses "retinex.MSRCP", "get_y", "kirsch" and "TCanny", maybe a bug in one of those? I'm not sure where "get_y" comes from tho, probably from one of the imports. I used VSRepo to install it and assuming it installs the latest version and considering I get the same result I think it's a bug as well.

Got it, thank you for the explanation. Only small issue is that not all the edges are masked. What would you suggest to improve the edge detection? Run the function twice or mix it with other functions?

I originally thought of doing it like this, but couldn't figure out how:
1. Create an edgemask
2. Denoise the luma plane
3. Add the edgemask on top of the denoie
4. Put that as the luma plane and get the other 2 planes from the original clip

[poisondeathray]

How do you check for those exceeded values?
I see that the function uses "retinex.MSRCP", "get_y", "kirsch" and "TCanny", maybe a bug in one of those? I'm not sure where "get_y" comes from tho, probably from one of the imports. I used VSRepo to install it and assuming it installs the latest version and considering I get the same result I think it's a bug as well.

I moused over in vsedit, and it shows the pixel values. But you can use PlaneStats too

edgemask = core.std.PlaneStats(edgemask).text.FrameProps()

In my clip, it showed a max of 2046 for the 10bit mask. Interesting that it's exactly 2*1023 . My 12 bit clip showed max of 8190. Exactly 2*4095. So that' s not a coincidence. I can't look into it right now; I'm busy with other stuff - but this should be reported to the developer of that filter if you can't see what's causing it in the function

Got it, thank you for the explanation. Only small issue is that not all the edges are masked. What would you suggest to improve the edge detection? Run the function twice or mix it with other functions?

You can try tweaking the parameters, and/or use other edge detection filters. Many of them are ported from avisynth

If you use the VSDB , search for "edge detection", it lists 24 filters . Some of them are duplicated, but it's a good place to start
http://vsdb.top/

[_Al_]

I just vent to github for latest kagefunc.py, fvsfunc.py, vsutil.py and mvsfunc.py I used some older version, also getting latest plugins BM3D.dll, Retinex.dll and YUV420P10 turned out good. I mean it worked. Result was not good at all because blocks of colors were changed in denoised parts, so some tweaking needs to be done. But edgemask values did not go over 1023 limit . I used:

import vapoursynth as vs
core = vs.core
import kagefunc
import mvsfunc
src = core.lsmas.LibavSMASHSource(r'C:\YUV420P8_video.mp4')
src = core.resize.Point(src, format = vs.YUV420P10)
edgemask = kagefunc.retinex_edgemask(src).std.Invert()
denoise = mvsfunc.BM3D(src, sigma=[100.0,100.0,100.0], radius1=1)
result = core.std.MaskedMerge(src, denoise, edgemask)

that get_y() is from vsutil.py and it just gets Y plane, nothing else. With these modules they start to pull codes from across each other, so if things get settled (type of denoise , etc.) it is better do go back to basics and pull actual lines from those and use them.

[_Al_]

images in this order:
src, edgemask, denoise, result

[_Al_]

to break down retinex_edgemask:
this line:

edgemask = kagefunc.retinex_edgemask(src).std.Invert()

is this:

def kirsch(src: vs.VideoNode) -> vs.VideoNode:
    """
    Kirsch edge detection. This uses 8 directions, so it's slower but better than Sobel (4 directions).
    more information: https://ddl.kageru.moe/konOJ.pdf
    """
    weights = [5] * 3 + [-3] * 5
    weights = [weights[-i:] + weights[:-i] for i in range(4)]
    clip = [core.std.Convolution(src, (w[:4] + [0] + w[4:]), saturate=False) for w in weights]
    return core.std.Expr(clip, 'x y max z max a max')

Y = core.std.ShufflePlanes(src, 0, vs.GRAY)
retinex = core.retinex.MSRCP(Y, sigma=[50, 200, 350], upper_thr=0.005)
kirsch_edge_mask  = kirsch(Y)
tcanny_edge_mask = core.tcanny.TCanny(retinex, mode=1, sigma=1)\
            .std.Minimum(coordinates=[1, 0, 1, 0, 0, 1, 0, 1])
edgemask = core.std.Expr(clips = [kirsch_edge_mask, tcanny_edge_mask], expr=['x y +'])\
               .std.Invert()

so looking at that last expression,
kirsch_edge mask value and tcanny_edge value are added up, and guessing it should be clipped to max value where for some reason it was not and those values were doubled?
or it could be limited right there: expr=['x y + 1023 min']

[leonsk]

I moused over in vsedit, and it shows the pixel values.

Color panel must be enabled, didn't know that even existed.
Quick question: How does one print to vsedit's log? Is that possible or things must be printed on the frame?

@ _Al_
I got them using VSRepo so I assume they are the latest version. Try without Invert().
Or maybe it's because you're using an 8-bit clip. My source is YUV420P10.

[_Al_]

I changed that clip to vs.YUV420P10 in that script.

You can print to log in VSEdit raising Error. That will stop the script in that spot, but you get a value and then you'd delete that line.
For example:

src = core.lsmas.LibavSMASHSource(path)
raise ValueError(f'{src}')

that would print :

[11/04/2020 02:38:43]  Vapoursynth Script: <class 'ValueError'>VideoNode
	Format: YUV420P10
	Width: 1920
	Height: 1080
	Num Frames: 2645
	FPS: 60000/1001
	Flags: NoCache IsCache

that was a neat trick from WolframRhodeum(I think)

Or you can write a values to a log, to a disk:

with open('error.log', 'a') as log:
    src = core.lsmas.LibavSMASHSource(path)
    log.write(f'{src}')
    new_clip = core.resize.Bicubic(src,640,360)
    log.write(f'{new_clip}')

that has an advantage of printing and not stopping script

Or you can run your script as *.py in your Python console, IDLE or other, where simple print(something) would work. You'd need to request frames at the bottom of script though:

for frame in (0, src.num_frames): #(or your custom range)
    src.get_frame(frame)

that would not show frames, those frames would just be requested and script be running, therefore printing if some prints present. Running script in VsEdit you do not have to request frames, because VsEdit does it for you, getting clip from vs.get_output(), changing it to RGB frame by frame, therefore requesting them and running script.

Or you can request frames ,same as above AND see actual frames as well using numpy and opencv for preview:

#you script here
#last line :    result = ....
import numpy as np
import cv2
rgb = core.resize.Point(result, format=vs.RGB24,    matrix_in_s = '709')
for frame in range(0, rgb.num_frames):     
    img = np.dstack([np.asarray(rgb.get_frame(frame).get_read_array(p)) for p in [2,1,0]])
    cv2.imshow('title name for your window here', img)
    cv2.waitKey(0) #press space bar to step thru frames, or use 1 for nonstop playback

this is why it goes miles further than avisynth, using Python modules ...

Or you can redirect sys.stdout or sys.stderr.

[leonsk]

I see, but your source is 8-bit, anyway as I said try without Invert(), in my case the problem disappears with it.

Seems to be fixed here:
https://github.com/Irrational-Encoding-Wizardry/kagefunc/commit/b87017b1beb0eb50978354...4aeab974c21ce3

Thanks for the tips.

[_Al_]

They fixed that expression to 'x y + 1023 min'. Pythonic equivalent of min(x+y, 1023). Except not hard-coding it to 1023 (10bit max value) but whatever max value for bitdepth there is.
But what is weird, I did not have to fixed it and it was limited anyway. Even without Invert() it works ok, values are limited. Don't know what's going on.