How to achieve frame-blending/motion blur?

[eko]

I've been trying to create a "motion blur" effect for a little while now but have not found a method that works for my use case. The footage either has extreme ghosting with sharp "after images" with no blur or blur with artifacts from still objects. FFmpeg, Smart-Resample, and ReelSmart Motion Blur are some methods that I have tried. Whether that would be interpolating the footage to a higher framerate, slowing the footage down while interpolating it, and then speeding it up to create motion blur or blending frames. I have tried numerous methods in FFmpeg but the render always has the same issue.

Here is what I mean by sharp "after images":


As you can see, the motion blur is sharp and not blurred. The issue is also that there are not enough frames to fill the space in between meaning that it would not look convincing.

What I am trying to achieve is this:
- https://www.youtube.com/watch?v=IE7pY5AJhCI
- https://www.youtube.com/watch?v=4nJ9u5ANcEw
- https://youtu.be/q1Ev3QxbdUY?t=44
- https://streamable.com/glzxly

The issue with most methods is that non-moving elements are being affected by the motion blur, for example with ReelSmart Motion Blur:


So what I am trying to find is a method that can produce motion blur with high fps footage (180fps) without artifacts and blurred "after images". Thanks!

[poisondeathray]

What is your "use case" ? What is the source of the 180fps footage? What is the target output fps ?

If it's CG or video game, do you have access to game engine/ render engine motion vectors and z-depth data ?

What elements are static? Are they overlays, HUD, those sorts of things ? Did you try masking them out ?

[eko]

What is your "use case" ? What is the source of the 180fps footage? What is the target output fps ?

If it's CG or video game, do you have access to game engine/ render engine motion vectors and z-depth data ?

What elements are static? Are they overlays, HUD, those sorts of things ? Did you try masking them out ?

So my use case is game footage shot in 180fps at 1080p using OBS, and no I do not have access to its game nor rendering engine, just plain 2D footage. The target output fps would be 600 after frame interpolation, which I've seen been done with good results (for gameplay) but have no clue how they did it. As you said, the static elements would be the game's HUD, and masking those out would be too much of a hassle for something that I am looking to be automatic and would probably still show up (after masking) using ReelSmart Motion Blur.

[poisondeathray]

Does your monitor refresh as 600Hz ? How did you "see" good results ?

Did you mean interpolated to 600fps then converted to 24 or 30 or 60 ? None of the ones your posted above are remotely close to 600fps

There might are other ways to mask depending on what the HUD design setup is like.

The problem with calculated motion vectors is they might go over static areas. They are error margins in doing this in 2D post afterwards , compared to actual accurate motion vectors, object data, z-depth data

Can you post a sample of your source recording ?

[johnmeyer]

Here is some AVISynth code I have lying around. This is used for things like providing the proper look to time lapse where, if you don't add motion blur, the video looks too stroboscopic. The same is true of video that was shot with high shutter speed.

source=AVISource("E:\fs.avi").ConvertToYV12()
Assumefps(source,23.976)
super = MSuper()
backward_vectors = MAnalyse(super, isb = true)
forward_vectors = MAnalyse(super, isb = false)
blurred=MFlowBlur(super, backward_vectors, forward_vectors, blur=48)

#stackhorizontal(source,blurred) #Enable if you want to compare input to output
return blurred

[eko]

Does your monitor refresh as 600Hz ? How did you "see" good results ?

Did you mean interpolated to 600fps then converted to 24 or 30 or 60 ? None of the ones your posted above are remotely close to 600fps

There might are other ways to mask depending on what the HUD design setup is like.

The problem with calculated motion vectors is they might go over static areas. They are error margins in doing this in 2D post afterwards , compared to actual accurate motion vectors, object data, z-depth data

Can you post a sample of your source recording ?

Yeah, it should be interpolated to 600fps then rendered at a lower fps, probably 24fps, sorry for not clarifying that. The thing is, I do not mind if there are some overlapping motion blur with static elements in the footage but my goal is for it to be minimized.

Here is an example of my source recording:
https://mega.nz/embed/pkcATbCR#COupmaJ2BMYvOfAi2hf0whsTJnW-MIFpU2SVRrxFf8c
(And sorry for posting a Mega link, I could not think of a service that supported 180fps footage)

[eko]

Here is some AVISynth code I have lying around. This is used for things like providing the proper look to time lapse where, if you don't add motion blur, the video looks too stroboscopic. The same is true of video that was shot with high shutter speed.

source=AVISource("E:\fs.avi").ConvertToYV12()
Assumefps(source,23.976)
super = MSuper()
backward_vectors = MAnalyse(super, isb = true)
forward_vectors = MAnalyse(super, isb = false)
blurred=MFlowBlur(super, backward_vectors, forward_vectors, blur=48)

#stackhorizontal(source,blurred) #Enable if you want to compare input to output
return blurred

I've tried to use AVISynth together with VirtualDub before but I keep getting "AVI Import Filter error: (Unknown) (80040154)" once I try to load an AVISynth script.

[johnmeyer]

I've tried to use AVISynth together with VirtualDub before but I keep getting "AVI Import Filter error: (Unknown) (80040154)" once I try to load an AVISynth script.

Replace the AVISource line with another method to load the video into AVISynth.

[eko]

I've tried to use AVISynth together with VirtualDub before but I keep getting "AVI Import Filter error: (Unknown) (80040154)" once I try to load an AVISynth script.

Replace the AVISource line with another method to load the video into AVISynth.

Now I get a different error. "LoadPlugin: unable to load "mvtools2.dll", Module not found. Install missing library?".
Weird, since I have mvtools2 inside the plugins folder.

[poisondeathray]

Now I get a different error. "LoadPlugin: unable to load "mvtools2.dll", Module not found. Install missing library?".
Weird, since I have mvtools2 inside the plugins folder.

Make sure you have matching x86 or x64 version same as your avisynth install

pinterf branch is the most up to date, has most bugfixes
https://github.com/pinterf/mvtools/releases

I recommend using mflowblur with mrecalculate - it reduces the edge and borders artifacts around static areas

You might want to consider increasing the antialiasing settings in game. You can apply AA in post too, but it's generally less effective than in game.

The example below does not use AA, it just demonstrates some settings you might start with using mflowblur with mrecalculate.

 LSmashVideoSource("Replay 2020-10-28 20-03-41.mp4")
 AssumeFPS(180)
 super = MSuper(pel=2, hpad=0, vpad=0, rfilter=4)
 backward_1 = MAnalyse(super, chroma=false, isb=true, blksize=32, searchparam=3, plevel=0, search=3, badrange=(-24))
 forward_1 = MAnalyse(super, chroma=false, isb=false, blksize=32, searchparam=3, plevel=0, search=3, badrange=(-24))
 backward_2 = MRecalculate(super, chroma=false, backward_1, blksize=16, searchparam=1, search=3)
 forward_2 = MRecalculate(super, chroma=false, forward_1, blksize=16, searchparam=1, search=3)
 backward_3 = MRecalculate(super, chroma=false, backward_2, blksize=8, searchparam=0, search=3)
 forward_3 = MRecalculate(super, chroma=false, forward_2, blksize=8, searchparam=0, search=3)
 MFlowBlur(super, backward_3, forward_3,  blur=200)
 ChangeFPS(30)

Interpolating to a higher framerate first is not necessarily a good way to go - because you introduce additional interpolation artifacts first, and the motionblur artifacts are added on top

Ideally an integer multiple like 30fps derived from 180fps would work out better and you only have motionblur artifacts

https://forum.videohelp.com/attachment.php?attachmentid=55652&d=1603930410

The main issue is the edges of part of the HUD - parts are semi transparent and it causes confusion with the motion vectors "underneath" . But it's difficult to mask parts of that HUD effectively, because some parts animate on/off, some are semi-transparent

[manono]

I'd use the motion blur feature of QTGMC, but getting QTGMC set up can be a bit tricky.

[eko]

Now I get a different error. "LoadPlugin: unable to load "mvtools2.dll", Module not found. Install missing library?".
Weird, since I have mvtools2 inside the plugins folder.

Make sure you have matching x86 or x64 version same as your avisynth install

pinterf branch is the most up to date, has most bugfixes
https://github.com/pinterf/mvtools/releases

I recommend using mflowblur with mrecalculate - it reduces the edge and borders artifacts around static areas

You might want to consider increasing the antialiasing settings in game. You can apply AA in post too, but it's generally less effective than in game.

The example below does not use AA, it just demonstrates some settings you might start with using mflowblur with mrecalculate.

 LSmashVideoSource("Replay 2020-10-28 20-03-41.mp4")
 AssumeFPS(180)
 super = MSuper(pel=2, hpad=0, vpad=0, rfilter=4)
 backward_1 = MAnalyse(super, chroma=false, isb=true, blksize=32, searchparam=3, plevel=0, search=3, badrange=(-24))
 forward_1 = MAnalyse(super, chroma=false, isb=false, blksize=32, searchparam=3, plevel=0, search=3, badrange=(-24))
 backward_2 = MRecalculate(super, chroma=false, backward_1, blksize=16, searchparam=1, search=3)
 forward_2 = MRecalculate(super, chroma=false, forward_1, blksize=16, searchparam=1, search=3)
 backward_3 = MRecalculate(super, chroma=false, backward_2, blksize=8, searchparam=0, search=3)
 forward_3 = MRecalculate(super, chroma=false, forward_2, blksize=8, searchparam=0, search=3)
 MFlowBlur(super, backward_3, forward_3,  blur=200)
 ChangeFPS(30)

Interpolating to a higher framerate first is not necessarily a good way to go - because you introduce additional interpolation artifacts first, and the motionblur artifacts are added on top

Ideally an integer multiple like 30fps derived from 180fps would work out better and you only have motionblur artifacts

https://forum.videohelp.com/attachment.php?attachmentid=55652&d=1603930410

The main issue is the edges of part of the HUD - parts are semi transparent and it causes confusion with the motion vectors "underneath" . But it's difficult to mask parts of that HUD effectively, because some parts animate on/off, some are semi-transparent

Hurray!
After some time I finally got AVISynth working. I had the x86 and x64 versions mixed up for both mvtools and LSMASH.

I spent some time writing a script and this is my result:
https://files.videohelp.com/u/297159/a.avi
Not perfect but I think it has minimal artifacts, especially around the HUD.
Thanks!

Taken to the extreme :
https://files.videohelp.com/u/297159/a123.avi
https://files.videohelp.com/u/297159/a4.avi (With more blur)
And yeah, as you can see when the footage changes a lot the artifacts become more apparent. I do not know if there is a way around this though.
The good thing is that the crosshair, for example, seems to not be affected by the motion blur that much. Probably because of the contrast.

[poisondeathray]

There is no way around having at least some artifacts - they are part of motion vector estimation and object boundary edge estimation. They are just estimates - they are not actual measurements and data like you have from a real 3d game engine. If you play with the settings you might be able to reduce some, but the other sections might exhibit more artifacts

You can overlay a crosshair on top from the source video since it doesn't change position. Since it's a small area, having the real crosshair won't affect the motion blurred appearance that much, and it will be consistently "solid"

But the semi transparent health meters are a relatively large area . If you overlaid the original data there, there might be discontinunities with the blurred data adjacent and underneath. But if you have some games or gameplay where that area is becomes a problem, that might be an option to deal with it


I just looked at a123.avi - I noticed some duplicate frames, choppier motion than expected . Did you use the same framerate for recording ? Where there framedrops or what source filter did you use

[eko]

There is no way around having at least some artifacts - they are part of motion vector estimation and object boundary edge estimation. They are just estimates - they are not actual measurements and data like you have from a real 3d game engine. If you play with the settings you might be able to reduce some, but the other sections might exhibit more artifacts

You can overlay a crosshair on top from the source video since it doesn't change position. Since it's a small area, having the real crosshair won't affect the motion blurred appearance that much, and it will be consistently "solid"

But the semi transparent health meters are a relatively large area . If you overlaid the original data there, there might be discontinunities with the blurred data adjacent and underneath. But if you have some games or gameplay where that area is becomes a problem, that might be an option to deal with it


I just looked at a123.avi - I noticed some duplicate frames, choppier motion than expected . Did you use the same framerate for recording ? Where there framedrops or what source filter did you use

I recorded that in 180fps as well and I do not think I had any frame drops during that recording. I also used LSMASH like you did, the reason it looks like that is because I did not use the exact script you gave me.

Take a look at a4.avi where I changed the blur amount, looks better than what a123.avi did.

This one as well:
https://files.videohelp.com/u/297159/21a.mp4

The reason behind why it looked so bad could be the excessive motion in the footage.

[poisondeathray]

a4.avi has the same problems with duplicate frames . Ideally you want to preserve 30 unique frames per second for 30fps for smooth motion. About 2 out of every 5 frames are duplicates

[eko]

a4.avi has the same problems with duplicate frames . Ideally you want to preserve 30 unique frames per second for 30fps for smooth motion. About 2 out of every 5 frames are duplicates

It is probably because I am using the SelectEvery() parameter. Going to try rendering it without it.

[poisondeathray]

a4.avi has the same problems with duplicate frames . Ideally you want to preserve 30 unique frames per second for 30fps for smooth motion. About 2 out of every 5 frames are duplicates

It is probably because I am using the SelectEvery() parameter. Going to try rendering it without it.

You don't have to "render" anything. It's usually a waste of time. You can preview the script in avspmod or vdub2 . Step through with the arrow keys

When debugging scripts, start at the top, comment out lines. First, check to make sure source filter is working correctly. Maybe you are using bad or old version of plugin

[eko]

Okay it does not look like the SelectEvery() parameter was causing that.

[eko]

a4.avi has the same problems with duplicate frames . Ideally you want to preserve 30 unique frames per second for 30fps for smooth motion. About 2 out of every 5 frames are duplicates

It is probably because I am using the SelectEvery() parameter. Going to try rendering it without it.

You don't have to "render" anything. It's usually a waste of time. You can preview the script in avspmod or vdub2 . Step through with the arrow keys

When debugging scripts, start at the top, comment out lines. First, check to make sure source filter is working correctly. Maybe you are using bad or old version of plugin

I'll check the source filter.

[poisondeathray]

This branch of L-Smash is stable and updated fairly regularly.
https://github.com/HolyWu/L-SMASH-Works/releases

[eko]

I have the latest LSMASHSource.dll so it probably is not the source filter that's causing this.

[poisondeathray]

I have the latest LSMASHSource.dll so it probably is not the source filter that's causing this.

If you step though a section with motion, with only that 1 line, and there are duplicates, then it's likely the recording. Your MIN in-game FPS might not be hitting 180 fps. You have FPS dips. You might need to turn down some settings or upgrade hardware , or record lower FPS

LSmashVideoSource("asdasd.mp4")

If that's the case, I would decimate to unique frames (to get the actual frame rate) before adding motion blur


You can try alternative source filters such as ffms2, and directshowsource or dss2, but it's unlikely to be a source filter issue if you recorded with the same setup as that earlier video (Replay 2020-10-28 20-03-41.mp4) . That earlier one had true 180 FPS to start with . So it's likely the asdasd.mp4 video

[eko]

I have the latest LSMASHSource.dll so it probably is not the source filter that's causing this.

If you step though a section with motion, with only that 1 line, and there are duplicates, then it's likely the recording. Your MIN in-game FPS might not be hitting 180 fps. You have FPS dips. You might need to turn down some settings or upgrade hardware , or record lower FPS

LSmashVideoSource("asdasd.mp4")

If that's the case, I would decimate to unique frames (to get the actual frame rate) before adding motion blur


You can try alternative source filters such as ffms2, and directshowsource or dss2, but it's unlikely to be a source filter issue if you recorded with the same setup as that earlier video (Replay 2020-10-28 20-03-41.mp4) . That earlier one had true 180 FPS to start with . So it's likely the asdasd.mp4 video

Yeah, the game in the recording largely variates in framerate depending on what "level" you're playing on so it is probably the recording that's the issue as you said.