Video interpolation question

[Santuzzu]

I would like to convert some HD soccer videos with progressive scan from 25fps to 50fps. Since with fast motion scenes it's hard to expect that artifacts will be negligible, I wanted to ask is there any sophisticated semi-automatic software where I can choose frame by frame which created frames to keep and which ones to correct or discard? I would like to read from experienced people what is the best solution for this.

[poisondeathray]

I would like to convert some HD soccer videos with progressive scan from 25fps to 50fps. Since with fast motion scenes it's hard to expect that artifacts will be negligible, I wanted to ask is there any sophisticated semi-automatic software where I can choose frame by frame which created frames to keep and which ones to correct or discard? I would like to read from experienced people what is the best solution for this.

You cannot "drop" a frame, otherwise you would lose sync; it would have to be a replacement frame of a duplicate or another interpolated frame, or using different interpolation settings

Some interpolation plugins in NLE's have the ability to keyframe toggle between interpolation, and duplication. This makes it easy, but only using that internal optical flow algorithm

But there are dozens of interpolation software algorithms. There are external ones that use "AI"/ neural net processing that excel on certain types of frames (e.g. DAIN), but poor results on others . There might be a clean frame with one of the algorithms or settings . A given algorithm might have dozens or even hundreds of settings. That's easily thousands of iterations where a given algorithm and setting might produce a good clean interpolated frame

A general simple way to manually fine tune the results is to use a NLE. Almost all can do a 2 track edit easily. You place your "automatic" interpolated layer on the bottom track and keyframe the opacity of a 50p duplicated (clean) version on top track. Whenever there are "bad" interpolated frame, you "cover" them up with the top layer. The opacity controls the visibility of the top layer

Some offer multicam edit capability. You load several interpolated versions and choose between them (essentially a multicam edit, usually hotkeys are used to switch). e.g you might load a DAIN version, a SVPFlow version, a Twixtor version, a framerateconverter small block size, and framerateconverter large block size, and a clean duplicate version

Those are full frame edits. If you want more advanced, you can mask certain parts of certain versions to composite a clean interpolated version

Some offer manual guidance and mask/trackpoint control to help guide the "automatic" motion estimation. These help improve the results, but are quite advanced and require lots of user involvement

[johnmeyer]

I think what you're asking is whether you can create 50 fps from 25 fps using motion interpolation but, when that interpolation creates a frame with awful artifacts, have the software automatically detect that and for that frame use instead either a duplicate of the previous frame, or a frame blend of the two adjacent frames.

You might want to take a look at this thread over at doom9.org:

FrameRateConverter

The OP was not entirely successful in building something that is similar to what you are describing, but he got a LOT of advice on different ways to achieve the goal, including my suggestion of switching manually between motion estimated and frame-blended outputs.

[abolibibelot]

If going the Avisynth route, a useful tool would be StainlessS's “Sawbones”, which was meant to accelerate the tedious task of writing down interpolation commands for each frame that needs to be fixed when reviewing them one by one in quick succession. I used it once to make a list of about 5000 frame interpolation commands out of about 50000 frames (about 30min of unique footage from a funeral recorded by a camera with a defective stabilizer, which added intermittent jerkiness rather than smoothing out movements — a nightmare) and it was a huge time saver. It creates commands for the companion frame interpolation function named FrameSurgeon, but the text file can then be edited to match the syntax of another filter / function if needed. In my tests about 2 years ago, though, FrameSurgeon was among the best interpolation functions available for Avisynth, and I relied on it for the bulk of that task, only for a few problematic frames did I use other functions : Morpheus (also by “StainlessS”, also based on MVTools2, but tends to produce less “edgy” artifacts) and Morph (less sophisticated, simply blends adjacent frames, less pleasant than a good interpolation but doesn't produce ugly artifacts so it can be preferable to a poor interpolation if nothing else yields a satisfying result, but it also has the drawback of being poorly optimized when it comes to memory usage and whatnot so Avisynth will choke with more than a few dozen Morph() calls, FrameSurgeon and Morpheus don't have such issues even with thousands of calls), and for some very problematic frames I had to resort to manually blend two or three versions with GIMP...

https://forum.doom9.org/showthread.php?t=173158

Another useful function, also from “StainlessS”, is SelectRange (mentioned there, requires the “Prune” plugin), which allows to display only frames which were marked for interpolation (from a list of FrameSurgeon commands). So the idea would be to run a first step with Sawbones to mark the problematic frames, then in a second step review those frames only with SelectRange, and try different possible functions or settings, using AVSPMod with either two tabs (tabs with videos of the exact same length and resolution are synchronized automatically) or with StackHorizontal / StackVertical to display two versions at the same time.