Video stabilization with timestamp

[theseeker2]

I recently captured a very shaky Hi8 video using Firewire and WinDV on my Sony Digital8 camcorder. Fortunately, there are plenty of great stabilization tools out there, and I've had success stabilizing the footage with both the DeShaker plugin in VirtualDub and also the paid program Mercalli (just using the free version for now).

Here's the problem: the original Hi8 videotape also has a timestamp printed on it. This is on the tape itself and cannot be removed on playback. Whenever I run DeShaker, Mercalli, or even Warp Stabilizer in Premiere Pro or Stabilizer in DaVinci, I end up with a bouncing timestamp that is extremely distracting and ruins the stabilized footage.

I have been scouring the internet for some type of tool that can remove this timestamp before I apply the stabilization. VoodooFX graciously ran the video through InpaintDelogo, but the results were not great (VoodooFX warned me of this).

Can anyone recommend a tool that can help me remove this timestamp? I've attached my sample and also the DeShaker output so you can see the bouncing timestamp. I am hoping that where there is a will, there is a way

[poisondeathray]

"Temporal inpainting" - if there is usable data in other frames, there are algorithms that can make use of those nearby frames to help fill .

I posted about "ProPainter" and a few other temporal inpainters in other threads before at doom9 and videohelp. Propainter is probably one of the better ones right now, not perfect, but definitely a generation better than older ones a few years ago. Short version: it's machine learning , slow, and difficult to use unless you're used to these types of projects . They are limited by GPU memory so you have to do things like crop to region of interest , perhaps divide up into frame ranges depending on how much GPU memory you have. There are quite a few options and settings to tweak in propainter , and the default model works fairly well on most sources
https://github.com/sczhou/ProPainter

This was preprocessed with QTGMCed, light denoise with SMDegrain before feeding into ProPainter (If you have noisy frame outside of mask area, not good for temporal inpainters in general because they can fill with that noise)

[theseeker2]

This was preprocessed with QTGMCed, light denoise with SMDegrain before feeding into ProPainter (If you have noisy frame outside of mask area, not good for temporal inpainters in general because they can fill with that noise)

Wow! This is really amazing!

Could you share the script you used on my sample? Thanks again.

[Selur]

Impressive.

[poisondeathray]

Could you share the script you used on my sample? Thanks again.

The script is just for prep work - deinterlace, light denoise . I also used "full range" to convert to RGB (Propainter works internally in RGB) as to not clip the levels (you should do the same when using deshaker or any RGB filter, and/or legalize the levels), and made a gamma adjustment to brighten it a bit . Although Propainter can use input video, I use PNG sequences for this type of work. I reversed the full range and gamma steps when taking the propainter output to encode the video

AVISource("hth_edit.avi")
AssumeBFF().QTGMC(border=true)
Levels(0,1.5,255,0,255,false)
SMDegrain()
ConvertToRGB24(matrix="pc.601")

<ProPainter>

#for video
ConvertToYV12(matrix="pc.601")
Levels(0,0.75,255,0,255,false)

[theseeker2]

The script is just for prep work - deinterlace, light denoise . I also used "full range" to convert to RGB (Propainter works internally in RGB) as to not clip the levels (you should do the same when using deshaker or any RGB filter, and/or legalize the levels), and made a gamma adjustment to brighten it a bit . Although Propainter can use input video, I use PNG sequences for this type of work. I reversed the full range and gamma steps when taking the propainter output to encode the video

Fantastic, thanks again. I can make PNG out of the AVI, so that part should be no problem, and thanks for the tip on that.

Can you provide the command line you used for Propainter? Propainter looks like it's written in python, and I have actually written some python code myself, so I should have no trouble running it. But I am having trouble figuring out from the docs what the correct options are for doing the type of work you've done here. Would appreciate your help with that last detail.

[poisondeathray]

The script is just for prep work - deinterlace, light denoise . I also used "full range" to convert to RGB (Propainter works internally in RGB) as to not clip the levels (you should do the same when using deshaker or any RGB filter, and/or legalize the levels), and made a gamma adjustment to brighten it a bit . Although Propainter can use input video, I use PNG sequences for this type of work. I reversed the full range and gamma steps when taking the propainter output to encode the video

Fantastic, thanks again. I can make PNG out of the AVI, so that part should be no problem, and thanks for the tip on that.

Can you provide the command line you used for Propainter? Propainter looks like it's written in python, and I have actually written some python code myself, so I should have no trouble running it. But I am having trouble figuring out from the docs what the correct options are for doing the type of work you've done here. Would appreciate your help with that last detail.

I use avisynth ImageWriter eg.

Imagewriter("PATH\prefiltered_%04d.png", type="png")

Look at inference_propainter.py for the full options .

Propainter takes 2 inputs: (1) the image sequence or video (2) the matching mask image sequence or video. The mask is a binary format white/black, where 100% white denotes the fill areas - so you'd cover the timestamp area with white on a black background. Zoom in and look closely because often there are compression artifacts around the borders of logos and such

--save_frames will also save a png sequence , but by default it saves a lossy h264/mp4 and a mask preview h264/mp4 . The video export will get resized to mod16, so it's better to use mod16 dimensions as input, but the png sequence will be the same dimensions as input

The "tighter" the mask, the more potentially detailed the results because there are more areas to take from. But if you "miss" areas , or there are problems with the usable areas (e.g. noise, artifacts that you didn't cover with the mask) - the fill regions might get contaminated and not be as good

Beware that you when you apply QTGMC to a video with static overlay - it often is no longer completely static. Preview the mask (e.g. overlay with partial transparency in avisynth) to see that you get complete coverage (either animate the mask or expand/dilate it) . But there are mask dilation options in Propainter too . The 1st test I did, I missed some spots (because the timestamp moves slightly with QTGMC, and it moved slightly beyond the mask) , so I cheated and dilated the mask and redid one section using propainter

The more reference frames , the more potential fill areas that can be used . e.g. if camera 90 frames away moves to a clean area on stage that could have been used as a fill region, but you limit to 60 frames, you miss out on that opportunity for a good fill. But it's a "double edged sword": the longer the "window" , also the potential for more contamination , lighting differences, perspective differences, time differences. Close references usually work better if possible - ideally you would adjust the settings appropriately per scene. Many of these machine learning temporal inpainting algorithms use a variation on Optical Flow called RAFT - that's partially how they predict the fill regions of objects. e.g. if some guy is walking "behind" the fill area, the synthesized areas use RAFT based on nearby frames to predict how his arms and legs would be. If the references are not good in close proximity, or too far - you can get distortions and errors. So "usable" data in (hopefully) nearby frames is the key

python inference_propainter.py --video test/input --mask test/input_mask --mask_dilation 0 --subvideo_length 60 --neighbor_length 30 --ref_stride 30 --scale_w 1 --scale_h 1 --fp16 --save_frames --output results/output

[theseeker2]

I will give it a try this evening. Thanks again for all of the info, poisondeathray!

[lollo]

This was preprocessed with QTGMCed, light denoise with SMDegrain before feeding into ProPainter (If you have noisy frame outside of mask area, not good for temporal inpainters in general because they can fill with that noise)

Impressive achievement, master pdr!