Using Avisyth Overlay for Contrast Masking

[MattiasN]

my script do look like this

function deflicker(clip e)
{
o   = e
sm  = o.bicubicresize(88,64) #  can be altered, but ~25% of original resolution seems reasonable
smm = sm.temporalsoften(1,32,255,24,2).merge(sm,0.25)
smm = smm.temporalsoften(2,12,255,20,2)
o2  = o.mt_makediff(mt_makediff(sm,smm,U=3,V=3).bicubicresize(width(o),height(o),0,0),U=3,V=3)
return o2
}

you have to change the size in bicubicresize(88,64) to the right resolution if you do not have it correct then it did soften even more if i remember right

the resolution my film did have was 1600x1200 so bicubicresize(88,64) was the best for me

[jagabo]

my script do look like this

function deflicker(clip e)
{
o   = e
sm  = o.bicubicresize(88,64).grayscale()  #  can be altered, but ~25% of original resolution seems reasonable
smm = sm.temporalsoften(1,32,255,24,2).merge(sm,0.25)
#smm = smm.temporalsoften(2,12,255,20,2)
o2  = o.mt_makediff(mt_makediff(sm,smm,U=3,V=3).bicubicresize(width(o),height(o),0,0),U=3,V=3)
return o2
}

you have to change the size in bicubicresize(88,64) to the right resolution if you do not have it correct then it did soften even more if i remember right

the resolution my film did have was 1600x1200 so bicubicresize(88,64) was the best for me

Interesting technique. I think you need to increase the temporalsoften radius (the first argument) for this clip though. Try something like 4.

[MattiasN]

Thanks i have to test that

[jagabo]

Oh wait, I was testing a video from another thread. I don't know if changing the temporalsoften radius will help with the video in this thread. In any case, you can remove slower fluctuations by using a larger radius.

[sanlyn]

Revisited some old and new masking and overlay routines, and several revisions of them: http://forum.doom9.org/showthread.php?t=161986 .
Much better results with two of the functions in that thread:

#  === http://forum.doom9.org/showpost.php?p=1514814&postcount=2
function ContrastMask(clip v, float "gblur", float "enhance")
{
enhance = default (enhance, 10.0)
#tvcolor = default (tvcolor, true)  #<-- don't use this.
gblur = default (gblur, 20.0)
enhance = (enhance>=0.0 && enhance<=10.0) ? float(enhance*0.1) : 1.0
v2=v.Tweak(sat=0)
v2=v2.invert()
v2=v2.gaussianblur(50.0,50.0+gblur)
photoshop_overlay=mt_lutxy(v,v2,"x 127.5 > y 255 x - 127.5 / * x 255 x - - + y x 127.5 / * ? ")
merged=overlay(v,photoshop_overlay,opacity=enhance)
return merged
}

The above is handy bringing up shadow detail without amplifying noise, banding, mottling, crushed garbage -- wshich is a huge problem with this Trader Horn project. It's one reason by plain vanilla overaly and twisted gradation curves didn't work so well. The function below also came in handy for curbing blown out brights:

# === http://forum.doom9.org/showpost.php?p=1523448&postcount=30
function Highlighter(clip v, float "gblur", bool "gradient", int "threshold", bool "twopass", int "amount", bool "softlimit", int "method")
{
gradient = default (gradient,true)
gblur = (gradient==true) ? default (gblur,100) : default (gblur,5.0) 
threshold = default (threshold,100) #The lower the value, the more sensitive the filter will be.
twopass = default (twopass,false)
amount = default (amount,10)
softlimit = default (softlimit,false)
method = default (method, 1)

amount = (amount<0) ? abs(amount) : amount

lighten=v.Tweak(sat=0).mt_lut("x "+string(threshold)+" > 0 x ?")
blurred= (gradient==true) ? lighten.gaussianblur(gblur).gaussianblur(gblur+100).gaussianblur(gblur+200) : lighten.gaussianblur(gblur)
#blurred= (gradient==true) ? lighten.Blur(0.5).Blur(0.5).Blur(0.5) : lighten.Blur(0.5)

fuzziness_mask=blurred.mt_edge(mode="prewitt", Y=3, U=2, V=2).mt_expand(mode="both", Y=3, U=2, V=2)
photoshop_overlay = (method==1) ? mt_lutxy(v,blurred.invert(),"x 127.5 > y 255 x - 127.5 / * x 255 x - - + y x 127.5 / * ? ") : \
                    (method==2) ? mt_lutxy(v,blurred.invert(),"255 255 x - 255 y - * 255 / -") : v.Tweak(bright=amount)
#photoshop_overlay = (twopass==true) ? photoshop_overlay.SmoothLevels(gamma=2.0,smode=2) : photoshop_overlay
photoshop_overlay = (twopass==true) ? photoshop_overlay.SmoothLevels(gamma=2.0) : photoshop_overlay

merged=mt_merge(v,photoshop_overlay,blurred)
fuzzy= (softlimit==true) ? mt_merge(merged,mt_lutxy(v,merged,"x y + 2 /"),fuzziness_mask) : merged
return fuzzy
}

The result with the above, attached as horn_Levels4a.mkv, with a really teentsy-weentsy tweak from gradation curves at about RGB 60. The parameter settings can be mighty tricky. As the opening frames in the river shot show, this puppy has a lot of work ahead. I've stretched the darker stuff about as far as one can: you already see mottling and noise from crushed darks (look at the shadows under the helmet of the guy on the right).

Talk about levels: Every shot in this MPEG telecined sample from the recording ( The Captive Scene mpeg ) has a different levels problem. Note a big luma bump about 15 frames in. More of these elsewhere. Think you'll get clean detail from those darks? You'd be surprised at what lurks therein. I've struggled with it all day. The guy getting slugged is Duncan Renaldo (aka "The Cisco Kid" from the 1950's). And all he did was politely ask the Princess to let them escape. I get a definite impression with this gal that "no" means "no". And this grimy sample is one of the better-looking scenes from the film.

[manono]

I'm getting nowhere fast trying out the different AviSynth solutions proposed in the MattaisN link for deflickering. Haven't tried the VDub one, the MSU one doesn't work well (or I can't get it to work), and the TimeLapseDF.dll mentioned by pdr is for scene only deflickering. Although I'm willing to go through it scene-by scene (and do anyway), it's not well designed to do what I want. Anyone want to take a stab at it? It's from a very poor DVD converted to NTSC from PAL by adding a dupe frame in every 6. There's lots else wrong with it, but here I'd be really interested in getting rid of the constant luma flickering:

[jagabo]

Deflickering techniques based on average luma won't work well with that video. This did reduce some of the flicker though (based on MattiasN's function):

function DeFlicker_mt(clip e)
{
   o   = e
   sm  = o.bicubicresize(88,64).grayscale()
   # Try different radii for temporalsoften (first argument):
   smm = sm.temporalsoften(16, 64, 255, 24, 2)
   o2  = o.mt_makediff(mt_makediff(sm,smm,U=3,V=3).bicubicresize(width(o),height(o),0,0),U=3,V=3)
   return o2
}

Mpeg2Source("D:\Downloads\FlickerSample.d2v", CPU=2, Info=3) 
TDecimate()
StackHorizontal(last, DeFlicker_mt())

There are still long term fluctuations in brightness.

[manono]

Thanks for having a look at the problem. The luma declines by 5-6 points during mild brightness peaks and as much as 10-11 when there are real bright peaks. It's a significant improvement. I was thinking that putting it on two or three times might work out even better. But no. Even with it at your default settings and using it just once, during movement there's ghosting created and during great movement there's real temporal artifacts created. Putting it on multiple times, while it steadies the luma even more, makes the ghosting/artifacting much worse. I purposely uploaded a fairly static scene as a sample.

I don't know if you've ever used the BadFrames filter. You name the frames to be filtered and it merges the good frames on either side. On static sequences like credits, it nips the luma fluctuations in the bud. The only problem is that you have to type in all the 'bad frames', sometimes dozens in a row, and it's useful only with static frames, but it does a perfect job with steadying the luma. It's also good on static fade outs, and brings the luma down to black perfectly. It performs similarly on fadeins.

So I was thinking that by naming a beginning and end frame it should be possible to steady the luma for all frames in between, whether for an entire scene, or just for some luma rise and fall. And without using temporal filtering which, no matter how much it's 'MC'd', is still liable to ghosting.

It might use the ReplaceFramesSimple syntax, something like:

A=Last
B=A.SteadyLuma()
ReplaceFramesSimple(A,B,Mappings="[0 400] [401 999] [1000 1199] [1240 1250]")

Something like that. Gavino?

Thanks again. I'll try it for this movie and then study the results later on.

[jagabo]

Even with it at your default settings and using it just once, during movement there's ghosting created and during great movement there's real temporal artifacts created.

I did notice the ghosting (it's a temporal filter after all) but thought it wasn't too bad for the given clip. I can see where it would be more problematic with large motions.

[sanlyn]

There are many similar luma burps in the Trader Horn flick, so I'm following this and keep watching this scene and scenes in Horn again and again. Something similar in both clips attached in post #35. The "levels" mkv has it in the opening frames of the river scene. Then again, the mpg sample about 15 frames in has a big gamma and highlight bump, but it doesn't drop back to the previous level--levels stay where they bumped to. Deflickers have no effect, nor does the function discussed here. I've even tried autogain/autolevel plugins from Avisynth and VDub alike--no decent effect, but a ton of unwanted side effects. I don't care for MSU's filters at all, but I tried them anyway; they're haywire and unpredictable, and just didn't work. The list of plugins I've tried would fill half this page, and even includes RemoveDirt (!).

If the techs at Turner Classic Movies seem to have no answer to this, I wouldn't know where to go next. It seems that MaskTools could be used to build some sort of matrix for something like a "typical" frame that keeps the mavericks steady. But I just don't understand much of how MaskTools works. I've loaded frames in Photoshop and AfterEffects but still can't find a common factor to work with.

Two nights ago TCM had some obscure silent films found in a warehouse somewhere and were "restored", but still had similar damage effects and worse. I guess the guys at ColourLab and UCLA can't figure it out, either.

[manono]

I did notice the ghosting (it's a temporal filter after all) but thought it wasn't too bad for the given clip. I can see where it would be more problematic with large motions.

It's fine for the low-movement sample. When there's movement it becomes unusable. When Didee made his original version he used TemporalSoften only to do the luma smoothing, but it wasn't used at all in the final output so there wasn't any ghosting. But there wasn't nearly as much luma smoothing, either, compared to yours:

A small clip is created. The small clip is temporal-softened. The difference achieved by temporalsoften is taken from the small clip, then re-scaled, and applied to the original clip.
Effectively, this is (very similar to) a spatial lowpass on the difference of a normal temporalsoften.

http://forum.doom9.org/showthread.php?p=1599701#post1599701

I'll fool around with it some more. I have several films to work on that need this problem solved. Before now I've been using your ReplaceFramesMC, but that has obvious and major drawbacks too.

[manono]

Then again, the mpg sample about 15 frames in has a big gamma and highlight bump, but it doesn't drop back to the previous level

I get that too. For those I use Levels or YLevels combined with ReplaceFramesSimple to limit the filtering to just the frames I want, usually between scene changes. The idea is to try and match the 'before' with the 'after' better and then 'bridge' the gap with ReplaceFramesMC, or sometimes with Dissolve. But Dissolve is a pain in the ass to set up. And I'm sure you've already figured all this out for yourself.

And I'm glad you're interested in the problem, too, because I was feeling a bit guilty for hijacking your thread.

[sanlyn]

No problem. The TCM video needs all the help it can get. BTW, I've seeen this movie on film and the big screen many times. Apparently commercial prints had these problems long before TCM was around to show it on TV.

[jagabo]

I did notice the ghosting (it's a temporal filter after all) but thought it wasn't too bad for the given clip. I can see where it would be more problematic with large motions.

It's fine for the low-movement sample. When there's movement it becomes unusable. When Didee made his original version he used TemporalSoften only to do the luma smoothing, but it wasn't used at all in the final output so there wasn't any ghosting. But there wasn't nearly as much luma smoothing, either, compared to yours:

The slight modification I used does the exact same thing. It's just that the averaging is done over a much wider range of frames and I had to use a very high scene change threshold because of the low frequency and magnitude of the flicker. Since mt_makediff() works on a pixel by pixel basis (rather than a frame by frame basis) you get ghosting from motion.

[poisondeathray]

Hey... did manono do a thread hijack??

manono's clip is a disaster. That's no ordinary flicker, it has both intraframe and interframe components .



Is that supposed to be the same scene? The "scenecut" is really a 180 on the camera isn't it ? Wow that is disorienting

But notice the lady has different earrings: long ones on her left vs. short ones right....

[sanlyn]

That sudden "flash" oif inner componments in manonoi's scene has plenty of those explosions, but the mpg from Horn has them as well....faryher apart, and not as severe. In working my project I fixed a couple of them with ReplaceFramesMC, but they were only 1 or 2 frame duration and not much motion. manono's cut has too many for manual fixes, and some last for several frames.

[manono]

Is that supposed to be the same scene?

Yeah, it was taken straight from the retail DVD with no cutting out or rearranging of the scene. I made it a static scene on purpose to make that luma pumping more obvious. In retrospect maybe that wasn't such a good idea because you don't see the ghosting you get during more pronounced movement after using the filter.

In working my project I fixed a couple of them with ReplaceFramesMC, but they were only 1 or 2 frame duration and not much motion.

Hehe, I already worked on a different film with this luma crap all through it and used ReplaceFramesMC hundreds of times to improve it. I'd happily go back and do it over again with a better filter, as well as with the film from which the sample was taken, which I haven't done all that much with yet.

[jagabo]

I've started looking at some way of using using AverageLuma and YPlaneMin/Median/Max within ScriptClip to deflicker. I don't know if I'll get anywhere. It would probably just be easier to write a dll.

[manono]

I've started looking at some way of using using AverageLuma and YPlaneMin/Median/Max within ScriptClip to deflicker. I don't know if I'll get anywhere. It would probably just be easier to write a dll.

It's frustrating, isn't it, wishing to know more than one does? I appreciate your efforts in this and your other filter-writing projects. As you can tell from my sometimes ignorant statements, I don't know the first thing about filter writing. I'm only a filter user and sometimes not even a good filter user. Gavino has been very helpful at Doom9 with people attempting to write filters, so maybe you can enlist him in the project, if not at Doom9 then by messaging him and asking to exchange ideas with him here or via PM.

[jagabo]

This is mostly a learning experience for me -- to get a better understanding of ScriptClip() and the runtime functions. I've come up with something that works well for "pure gain" flicker, with no risk of blending/ghosting. By "pure gain" I mean the flicker you can produce with a function like:

function _MakeFlicker(clip vid, int "amp")
{
  #ColorYUV(vid, gain_y=Sin(current_frame)*amp) # sinusoidal flicker
   ColorYUV(vid, gain_y=Rand(amp)) # random flicker
}

function MakeFlicker(clip vid, int "amp")
{
  amp = default(amp, 30)
  ScriptClip(vid, "_MakeFlicker("+string(amp)+")")
}

My deflicker function currently looks like:

function _MyDeFlicker(clip vid)
{
  smooth = Crop(vid, vid.width-80, 0, 80, -0)
  savg = AverageLuma(smooth)

  base = Crop(vid, 0, 0, -80, -0)
  bavg = AverageLuma(base)

  base = base.ColorYUV(gain_y=((savg/bavg)-1.0)*256.0)
  oavg = AverageLuma(base)

  # debugging information
  base = base.Subtitle("savg="+String(savg)+" bavg="+String(bavg))
  base = Subtitle(base, "oavg="+String(oavg), y=20)
  StackHorizontal(base,smooth)
}

function MyDeFlicker(clip vid, int "radius")
{
  radius = default(radius, 6)
  sm = BilinearResize(vid, 80, vid.height).temporalsoften(radius,255,255,255,2)
  both = StackHorizontal(vid, sm)
  ScriptClip(both, "_MyDeflicker")
  Crop(0,0,-80,-0)
}

I didn't see any way of passing two videos via ScriptClip() so I stacked the smoothed reference video onto the side of the main video. The inner _MyDeFlicker() function separates the main video and the reference into two videos, then adjusts the gain of the main video to match the average luma of the reference video. It has to return a frame the same size as the input so it stacks the reference filter back onto the side of the main video.

Using a still frame from manono's FlickerSample.m2v and:

Mpeg2Source("FlickerSample.d2v") 
TDecimate()
Loop(1000,0,0).ColorYUV(gain_y=-20).MakeFlicker() # still frame, a little darker, sinusoidal flicker
src=last # remember our flickery video

MyDeFlicker(src, 6) # video, temporal radius for TemporalSoften
StackHorizontal(src, last)
ConvertToYUY2().VideoScope("bottom")

gives sin.mkv and rnd.mkv, attached to this message. I used 6 as the radius in sin.mkv, 10 in rnd.mkv. For the latter, the larger the radius better the deflickering. Note that this code makes no attempt to detect scene changes or other events that change the average brightness (like someone in a white shirt walking into a dark scene).

Of course, this is a case where I created a flicker with known properties, hence I was able undo that flicker. So don't get your hopes up. The function doesn't do much for manono's actual video. The brightness fluctuations there are much more complex than simple gain changes. There are probably very few videos where this will work. And I suspect everyone else who's written a deflicker filter started with something like this then added more smarts. Things like including the darkest and lightest parts of the picture in the calculations, eliminating motion based brightness changes, etc.

In any case, I thought others might find the code instructional.

[manono]

If there's no scene change detection (and I don't think there should be as an all-of-a-sudden luma flareup might trigger it when there's no scene change), then I'd think there should be a way to name a range of frames for it to operate, either the beginning and end of a scene (perhaps also including a frame number from which the luma values are to be taken) for films where this crap occurs all the way through, or the beginning and end of a luma fluctuation for films like sanlyn's Trader Horn where this stuff is fairly rare. There are already deflicker filters out there for the very mild stuff.

For us anal-retentives who put ungodly amounts of time into these projects, having to fill in FirstFrame/LastFrame isn't asking too much. Or have multiple modes where one is more-or-less automatic (maybe for minor fluctuations) and the others are more hands-on. One names beginning and end frames and the luma adjusts linearly from one to the other (as in a fade out). Another names beginning and end frames with a third frame from which to get the luma values for the entire thing. See, I'm full of bright ideas to create even more work for you.

And if you'd like more real world samples, just ask.

[jagabo]

If there's no scene change detection (and I don't think there should be as an all-of-a-sudden luma flareup might trigger it when there's no scene change), then I'd think there should be a way to name a range of frames for it to operate

You could Trim() before the deflicker. Or you can re-enable the scene change threshold in TemporalSoften(). I disabled it (set it to 255) so that I wouldn't have to worry about whether a big flicker was disabling the deflicker function while testing. Or you could add first/last frame arguments to the functions.

(perhaps also including a frame number from which the luma values are to be taken)

You can change the savg (avg luma of the smoothed video) to a constant. For example change "savg = AverageLuma(smooth)" to "savg=75", or whatever. I was doing that during testing. With the debugging information printed onto the frame you can see the average luma of the base (original) frame (bavg) and the smoothed reference frame (savg). That could be added as an optional argument to functions.

But I don't see any way to add it as a frame number argument because the function called by ScriptClip() only sees one frame at a time. Ie, if you want to apply it to frames 100 to 200 using frame 150 as the reference brightness, the function needs to know the brightness of frame 150 before it can work on the other frames. But it won't see frame 150 until after it's already processed frames 100 to 149. It may be possible using GScript() or some other addon but I'm not familiar with that at all.

I noticed a bug setting the default amplitude in MakeFlicker() in my earlier post. I'll fix it in case anyone ever tries to use it.

Here's a little sample of the technique used on some moving video:

ffVideoSource("Golf.mkv") 
BicubicResize(720,400)
MakeFlicker(30) # make it flickery
StackHorizontal(last, MyDeflicker(6))
ConvertToYUY2().VideoScope("bottom")
Trim(520,620)

[unclescoob]

Who's the ***** in the background?