EMA-VFI-DR for frame interpolation - "Picket Fence" problems improved

[poisondeathray]

A well known issue is repeating patterns such as picket fences, window blinds, etc... cause problem from all video frame interpolation algorithms. There seems to be some improvement in this area... finally

EMA-VFI is a bit newer , and score slightly higher on benchmarks than RIFE
https://github.com/MCG-NJU/EMA-VFI

There are 4 EMA-VFI variants in InterpAny-Clearer (Along with 6 RIFE, 4 AMT, 4 IFRNet variants)
https://github.com/zzh-tech/InterpAny-Clearer

The EMA-VFI-DR variant tends to does better on "picket fence" sources that cause problems for other video frame interpolation algorithms like RIFE, DAIN, mvtools2, svpflow, Resolve OF, Adobe OF, etc...



3 Test clips which typically cause "freak out" artifacts from motion interpolation algorithms:

a) camera moving up with horizontal shutters, man partially occluding. Taken from "Fensterladen.mkv" courtesy of Selur

Test clip prep

LWLibavVideoSource("Fensterladen.mkv")
Crop(2,0,-14,0,true)
Trim(40,97)

EMA-VFI-DR result is clean. That practically never happens with this type of scene for video frame interpolation algorithms

b) boy riding bike horizontally across vertical fence bars, camera tracking shot (black and white). Taken from The Bad Sister DVD
https://forum.videohelp.com/threads/397425-A-Thank-You-to-all-an-critiques-welcomed!#post2584649

Test clip prep

MPEG2Source("Picket Fence.demuxed.d2v")
TFM(pp=0)
Trim(3,0) #trim off duplicates
Trim(0,25)

Preprocessed with TFM since src was field shifted. For EMA-VFI-DR You get some "smash face" (face looks punched in on interpolated frames), but overall the artifacts are much more palatable (especially at normal speed viewing) than the usual interpolation algorithms which have more distracting artifacts. There are still some minor artifacts with the default settings, increasing iterations to 4 (from default 2) seems to improve them


c) Selur fence clip. Girl riding bike in front of fence. Camera movement around.
https://forum.doom9.org/showthread.php?t=174541

Test clip prep

LSmashVideoSource("forInterpolation.mp4")
ResampleHQ(1280,720)
Trim(0,75) #trim off last frame (black)

Original clip was a soft UHD. This was downscaled with ResampleHQ to 720p. The uneven curvature lens blur combined with "picket fence" tends to cause artifacts on the leading and trailing fence edges for most algorithms. EMA-VFI-DR has them to a lesser extent, but it blurs them slightly and it's more "calm" compared to other algorithms which freak out with distortions. Tests at higher resolutions produced worse artifacts for EMA-VFI-DR , maybe from those blur/lens distortion - or perhaps the training set data was not done at those resolutions - have to do more testing and on other clips an other motions like circular/diagonal.

Another negative is the brake wire shadow on the pavement "flickers" more with EMA-VFI-DR compare to RIFE ; and RIFE is worse than mvtools2. The wire shadow gets reduced or blurred on interpolated frames .


Fensterladen.mkv and forInterpolation.mp4 are from selur's test clip for interpolation collection ( I hope it's ok to share this folder link selur)
https://drive.google.com/drive/folders/1hG4-vEDn4l_B0QTlsrUnyl8JrOAuRgBy

Comparison clips are slowed down

InterpAny-Clearer has some minor color/levels shifting and some legal range clipping for video inputs, possibly some YUV/RGB conversion issues. I reported issue at github.

This is a useful tool , especially for the specific repeating patterns/picket fence problem scenario during video frame interpolation. There are many video frame interpolation projects, but they are only marginally better or worse than RIFE. I'm hoping it gets a vapoursynth and eventually avisynth port of InterpAny-Clearer (Yes, I'm shouting out to: HolyWu, styler00dollar, WolframRhodium and others....)

[lollo]

Thanks for the info!

[Selur]

I hope it's ok to share this folder link selur

Sure, I posted the link back then, so others can use them to test interpolation methods.

Cu Selur

Ps.: how well does EMA-VFI perform with the other clips?
PPs.: here's an alternative link to the test files: https://drive.filejump.com/drive/s/3IjwAK2K2E2dfUPs0oX2wnpXv0cG3a