Hi, I'm playing a bit around with vsCodeFormer.
I start out with suzie_lossless and then apply 'SwinIR-L x4' and vsCodeFormer, which frame-by-frame looks nice (see: SwinIR+CodeFormer_lossless).
The problem is the temporal instability of the process.
Anyone got suggestions on how to stabilize this flickering?
+ Reply to Thread
Results 1 to 4 of 4
Last edited by Selur; 2nd Apr 2023 at 06:27.users currently on my ignore list: deadrats, Stears555
The temporal aliasing is a common problem with single image algorithms, especially upscaling ones
The differences in prediction each frame are too large in that example. QTGMC inputtype 1/2/3 can only do so much, and there are usually side effects incl. detail loss, possible ghosting . If you stack some temporal filters to combat the temporal aliasing/inconsistencies , you begin losing too much detail and blurring everything to mush
Another approach is to pre scale with something temporally consistent. You can experiment BasicVSR++ or Tecogan somewhere in the chain, either before /after codeformer, +/- downscaling
Here is an experiment with BasicVSR++ 4x => bicubic downscale /2 => GPEN *2 => QTGMCp inputtype=1 , compared to lanczos3 at 1/2 speed. I like the finer hair details improved on BasicVSR++ . I don't like that the BG textures are smoothed away, eye specular reflections are way too enhanced, contrast and saturation changed (in this example, they were crudely matched back) , other small details smoothed away compared to BasicVSR++ alone
Not a fan overall of the "look" of codeformer , or most "face" upscalers - I find that they change the "facts" and actual details too much. In that codeformer example, the hair (e.g. hair style changes, curls are straighened), eyes (eyes change, look almost transplanted), nose & lip shape change too much
My cutoff for acceptable tolerance is if the image is "plausible" and similar in underlying structure compared to say a lanczos4x
Yeah, BasicVSR++ often simply is too smooth/aggressive and codeFormer for normal content only is useful when the weight is near 1.
Here's an example of what weight 0 can do: https://imgsli.com/MTY2NDI2 *gig*
Cu Selurusers currently on my ignore list: deadrats, Stears555
What would be ideal is training a model with the same person for inference. e.g. if you were doing a family member or some famous person - train a specific model using other videos of the same family member or famous person . The temporal flicker would still be there for single image algorithms to some extent, but at least the face would match
Which best software to reduce or stabilize shaky video ?By vhfm in forum EditingReplies: 8Last Post: 16th Feb 2023, 09:08
Struggling to Deflicker/StabilizeBy stephenjosey in forum RestorationReplies: 3Last Post: 21st Nov 2022, 09:46
Deshake/Stabilize Suggestions NeededBy RABinMS in forum RestorationReplies: 7Last Post: 30th May 2021, 11:26
Remove stabilize from scriptBy lordsmurf in forum RestorationReplies: 9Last Post: 22nd Sep 2020, 00:39
How to stabilize this video?By TeNSoR in forum Newbie / General discussionsReplies: 2Last Post: 28th Jul 2020, 05:00