Stabilize temporal flickering?

[Selur]

Hi, I'm playing a bit around with vsCodeFormer.
I start out with suzie_lossless and then apply 'SwinIR-L x4' and vsCodeFormer, which frame-by-frame looks nice (see: SwinIR+CodeFormer_lossless).
The problem is the temporal instability of the process.

Anyone got suggestions on how to stabilize this flickering?
(QTGMC maybe?)

Cu Selur

[poisondeathray]

The temporal aliasing is a common problem with single image algorithms, especially upscaling ones

The differences in prediction each frame are too large in that example. QTGMC inputtype 1/2/3 can only do so much, and there are usually side effects incl. detail loss, possible ghosting . If you stack some temporal filters to combat the temporal aliasing/inconsistencies , you begin losing too much detail and blurring everything to mush

Another approach is to pre scale with something temporally consistent. You can experiment BasicVSR++ or Tecogan somewhere in the chain, either before /after codeformer, +/- downscaling

Here is an experiment with BasicVSR++ 4x => bicubic downscale /2 => GPEN *2 => QTGMCp inputtype=1 , compared to lanczos3 at 1/2 speed. I like the finer hair details improved on BasicVSR++ . I don't like that the BG textures are smoothed away, eye specular reflections are way too enhanced, contrast and saturation changed (in this example, they were crudely matched back) , other small details smoothed away compared to BasicVSR++ alone

Not a fan overall of the "look" of codeformer , or most "face" upscalers - I find that they change the "facts" and actual details too much. In that codeformer example, the hair (e.g. hair style changes, curls are straighened), eyes (eyes change, look almost transplanted), nose & lip shape change too much

My cutoff for acceptable tolerance is if the image is "plausible" and similar in underlying structure compared to say a lanczos4x

[Selur]

Yeah, BasicVSR++ often simply is too smooth/aggressive and codeFormer for normal content only is useful when the weight is near 1.
Here's an example of what weight 0 can do: https://imgsli.com/MTY2NDI2 *gig*

Cu Selur

[poisondeathray]

What would be ideal is training a model with the same person for inference. e.g. if you were doing a family member or some famous person - train a specific model using other videos of the same family member or famous person . The temporal flicker would still be there for single image algorithms to some extent, but at least the face would match

[Selur]

For those interested in temporal stabilization, I just stumbled over: https://github.com/pifroggi/vs_temporalfix
Haven't tested it yet, but I hope for the best.

Cu Selur

[poisondeathray]

I just played with it for a bit and vs_temporalfix tends not to do that well on live action content like "suzie's" moving hair and other types of live action where there are moving details . It does better when there are static textures like some types of anime

[Selur]

Damn, I feared something like that.

[Selur]

Added another file to the first post.