How to improve the video with jerky/choppy motion in background scenes?

[ls_milkyway]

Hi !

I am new to video related problems ....hence I don't even know what to call this problem? (Jerky/choppy)...It's a mobile game play recording using AZ Recorder.

The game was set to play at 30fps initially...(galaxy M30S) ..but later I changed it to 60fps (turned off Anti-Aliasing) as gameplay was smoother and AZ recorder was also set to record at 60fps with max possible bitrate (12 kbits/s).

But the video recorded is in VFR please check ffprobe (mediainfo) .... I am also attaching 10 sec clip from the recorded clip.

What I have done till now:

I am very comfortable with staxrip, after converting VFR -> CFR using 60/1 as per these instructions (to avoid audio sync issues) ...

Used EEDI3 filter to encode ....found very slight improvement.

I know it's very hard to improve unless original recording is not good ....but kindly suggest which will the best filter to use in my case?

For best possible improvement.....I thought of trying QTGMC repair ....but sadly my CPU does not support AVX instructions.

[rrats]

it says 26.xxxx fps on VLC media player. Definitely not a constant framerate

[jagabo]

If you ignore the timestamps and assume a constant frame rate the result is pretty smooth.

ffmpeg -r 30.000 -i 10-seconds.mp4 -c:v libx264 -c:a copy output30.000.mkv

That will play the smoothest on a 60 Hz monitor as every frame will be displayed for the same amount of time (2/60 second). But audio and video will not be in sync.

ffmpeg -r 26.919 -i 10-seconds.mp4 -c:v libx264 -c:a copy output26.919.mkv

That will be more jerky on a 60 Hz monitor because different frames will be displayed for different amounts of time (2/60 or 3/60 second). But if the frame rate is correct audio and video will stay in sync.

[ls_milkyway]

it says 26.xxxx fps on VLC media player. Definitely not a constant framerate

Yeah, I know it's variable frame rate, Mediainfo (by mediainfo.sourcefouge.net) says:

Frame rate : 26.054 FPS
Minimum frame rate : 0.786 FPS
Maximum frame rate : 3 461.539 FPS


I have also attached detailed mediainfo in json using ffprobe.......but I need to know which is the best filter to use to reduce the problem (if possible)....if I encode this recorded video in x264 using staxrip?

[ls_milkyway]

If you ignore the timestamps and assume a constant frame rate the result is pretty smooth.

ffmpeg -r 30.000 -i 10-seconds.mp4 -c:v libx264 -c:a copy output30.000.mkv

That will play the smoothest on a 60 Hz monitor as every frame will be displayed for the same amount of time (2/60 second). But audio and video will not be in sync.

ffmpeg -r 26.919 -i 10-seconds.mp4 -c:v libx264 -c:a copy output26.919.mkv

That will be more jerky on a 60 Hz monitor because different frames will be displayed for different amounts of time (2/60 or 3/60 second). But if the frame rate is correct audio and video will stay in sync.

Thanks......but I need video/audio synced (probably to upload on a streaming platform dailymotion, youtube etc.) ......you mean to say that the video is JERKY only on 60HZ monitor? FFmpeg out put does look good ....I have also tried increasing FPS to 120 using SVPFlow filter in staxrip ....but the output is not better than FFmpeg?

[jagabo]

What I was pointing out is that the VFR timestamps appear to be wrong. Here's the duration of the first few dozen frames (in ~milliseconds):

0.024
0.033
0.062
0.001
0.064
0.032
0.049
0.035
0.024
0.032
0.041
0.043
0.054
0.041
0.028
0.059
0.030
0.038
0.001
0.046
0.039
0.032
0.051
0.024
0.023
0.056
0.031
0.029
0.055
0.023
0.048
0.019
0.014
0.044
0.035
0.070
0.025
0.035

Note how they vary from 1 ms to 70 ms. Ignoring the time codes and assuming a constant frame rate gives much smoother (though not perfect) results.

Video is always jerky if the monitor's refresh rate isn't an integer multiple of the video's frame rate. This is a separate issue from the smoothness of the video itself.

Reading the frames one by one and assuming a constant 26.919 fps (assuming MediaInfo's reported average frame rate is correct) and using a motion interpolating frame rate convertor to make it 60 fps will give smooth results on a 60 fps monitor and keep the audio in sync.

a = LWLibavAUdioSource("10-seconds.mp4") 
v = LWLibavVideoSource("10-seconds.mp4") # reads frame by frame, ignoring timecodes
AudioDub(v,a)
AssumeFPS(26.919) # the average frame rate according to MediaInfo (may note be right but looks close)
FrameRateConverter(newnum=60000, newden=1000) # motion interpolation to 60.000 fps

Interpolating to 30 fps will not be as smooth -- because 30p video is inherently jerky compared to 60p.

Motion interpolation works fairly well for simple, slow moving stuff like this. But will not work when there is a lot of motion, or complex motions.

[ls_milkyway]

What I was pointing out is that the VFR timestamps appear to be wrong. Here's the duration of the first few dozen frames (in ~milliseconds):

0.024
0.033
0.062
0.001
0.064
0.032
0.049
0.035
0.024
0.032
0.041
0.043
0.054
0.041
0.028
0.059
0.030
0.038
0.001
0.046
0.039
0.032
0.051
0.024
0.023
0.056
0.031
0.029
0.055
0.023
0.048
0.019
0.014
0.044
0.035
0.070
0.025
0.035

Note how they vary from 1 ms to 70 ms. Ignoring the time codes and assuming a constant frame rate gives much smoother (though not perfect) results.

Video is always jerky if the monitor's refresh rate isn't an integer multiple of the video's frame rate. This is a separate issue from the smoothness of the video itself.

Reading the frames one by one and assuming a constant 26.919 fps (assuming MediaInfo's reported average frame rate is correct) and using a motion interpolating frame rate convertor to make it 60 fps will give smooth results on a 60 fps monitor and keep the audio in sync.

a = LWLibavAUdioSource("10-seconds.mp4") 
v = LWLibavVideoSource("10-seconds.mp4") # reads frame by frame, ignoring timecodes
AudioDub(v,a)
AssumeFPS(26.919) # the average frame rate according to MediaInfo (may note be right but looks close)
FrameRateConverter(newnum=60000, newden=1000) # motion interpolation to 60.000 fps

Interpolating to 30 fps will not be as smooth -- because 30p video is inherently jerky compared to 60p.

Motion interpolation works fairly well for simple, slow moving stuff like this. But will not work when there is a lot of motion, or complex motions.

that I think is the best I can get....THANKS! for providing the details.
Other than the script you mentioned, X264 encoding parameters I am planning to use is:

--crf 16 --preset veryslow --tune grain --aq-mode 2 --aq-strength 0.7 --chroma-qp-offset -2 --rc-lookahead 250 --min-keyint 60 --keyint 300 --colorprim bt709 --colormatrix bt709 --transfer bt709

I will be highly obliged if you provide suggestions on x264 parameters to maintain the smoothness. I hope the parameters (--tune grain, --keyint 300, --min-keyint 60) will not prevent smooth output.
Thanks.

[jagabo]

x264 parameters won't make any difference in the smoothness of playback. Unless your device can't handle the settings. For example, the settings you list aren't compatible with Blu-ray discs. Old Blu-ray players will probably choke on them.

[ls_milkyway]

x264 parameters won't make any difference in the smoothness of playback. Unless your device can't handle the settings. For example, the settings you list aren't compatible with Blu-ray discs. Old Blu-ray players will probably choke on them.

....I understand ....my old TV was not able to play any x264 video with reference frames > 4.

[ls_milkyway]

a = LWLibavAudioSource("10-seconds.mp4") 
v = LWLibavVideoSource("10-seconds.mp4") # reads frame by frame, ignoring timecodes
AudioDub(v,a)
AssumeFPS(26.919) # the average frame rate according to MediaInfo (may note be right but looks close)
FrameRateConverter(newnum=60000, newden=1000) # motion interpolation to 60.000 fps

is throwing error:

Script Error

System exception - Illegal Instruction
(F:\Audio Video\StaxRip-v2.13.0-x64\Apps\Plugins\AVS\FrameRateConverter\FrameRateC onverter.avsi, line 157)
(G:\Infinity\Originals\Final_temp\Final.avs, line 14)

The Line 157 is:

fwd = Recalculate ? MRecalculate(super, fwd, blksize=BlkSize/2, blksizev=BlkSizeV/2, overlap = BlkSize/2>4?(BlkSize/8+1)/2*2:0, overlapv = BlkSizeV/2>4?(BlkSizeV/8+1)/2*2:0, thSAD=100, dct=DctRe) : fwd

What i have tried?
1) Updated FrameRateConverter in staxrip
2)

FrameRateConverter(NewNum=60000, NewDen=1000)

3)

FrameRateConverter(NewNum=60, NewDen=1)

[davexnet]

System exception - Illegal Instruction

Which CPU do you have?

[jagabo]

And make sure you have all the required plugins: masktools2, mvtools, maybe grunt.

http://avisynth.nl/index.php/FrameRateConverter#Required_Plugins

And masktools2 requires the Visual Studio 2017 runtime.

[ls_milkyway]

System exception - Illegal Instruction

Which CPU do you have?

One without AVX instruction as mentioned in the first question....plus there is a small icon below profile name which shows "System Information" in this forum.

[ls_milkyway]

And make sure you have all the required plugins: masktools2, mvtools, maybe grunt.

http://avisynth.nl/index.php/FrameRateConverter#Required_Plugins

And masktools2 requires the Visual Studio 2017 runtime.

Yep. I have installed avisynth+ with required plugins (except grunt as it is needed for debugging....I guess) ...& the output is:

[Attachment 66203 - Click to enlarge]

All visual studios present on the system... is attached as a screen shot to this message.......it seems to me:

FrameRateConverter(NewNum=60000, NewDen=1000)

requires AVX instructions.

[poisondeathray]

...it seems to me:

FrameRateConverter(NewNum=60000, NewDen=1000)

requires AVX instructions.

It does not require AVX . The majority of avisynth plugins autodetect what is available and use that. e.g If you only have SSE2, they will use that

You can force a max cpu instruction set by using SetMaxCPU at the beginning of the script

eg

SetMaxCPU("SSE2")

http://avisynth.nl/index.php/Internal_functions#SetMaxCPU

[ls_milkyway]

You can force a max cpu instruction set by using SetMaxCPU at the beginning of the script

eg

SetMaxCPU("SSE2")

http://avisynth.nl/index.php/Internal_functions#SetMaxCPU

Ok! Good to know that ...So the current set:

SetMaxCPU("SSE2") # also tried SetMaxCpu("SSE4.2") as G3220 supports max SSE4.2
v = LWLibavVideoSource("G:\Infinity\Originals\Final.mp4")
a = LWLibavAudioSource("G:\Infinity\Originals\Final.mp4")
AudioDub(v,a)
AssumeFPS(26.054)
FrameRateConverter(NewNum=60000, NewDen=1000)

has output:

[Attachment 66208 - Click to enlarge]

Plugins installed in plugins64+ folder are:

[Attachment 66209 - Click to enlarge]

[jagabo]

I tried FrameRateConverter 1.2.1 (the version I had installed ealier), 1.3, and 2.0 beta, dll and avsi files. All worked properly in 64 bit AviSynth+.

[poisondeathray]

Your last error message is "€" ??? and before that it was "z" ???

What version of avs+ ?

A one line script will report it

version

[ls_milkyway]

Your last error message is "€" ??? and before that it was "z" ???

What version of avs+ ?

A one line script will report it

version

Output:

[Attachment 66217 - Click to enlarge]

[jagabo]

Try AvsInfoTool to look for installation problems.

That's the same version I'm using.

[ls_milkyway]

Try AvsInfoTool to look for installation problems.

That's the same version I'm using.

[Plugin errors/warnings]
__________________________________________________ ________________________________________________

"E:\Programs\AviSynth+\plugins64+\DePanEstimate.dll" requires the FFTW library for some functions.

Note: 'libfftw3f-3.dll' can be downloaded here:
http://www.fftw.org/install/windows.html

'libfftw3f-3.dll' should be placed in "System32" and/or "SysWoW64"
__________________________________________________ ________________________________________________

"E:\Programs\AviSynth+\plugins64+\mvtools2.dll " requires the FFTW library for some functions.

Note: 'libfftw3f-3.dll' can be downloaded here:
http://www.fftw.org/install/windows.html

'libfftw3f-3.dll' should be placed in "System32" and/or "SysWoW64"
__________________________________________________ ________________________________________________

The error no longer persists! .....Although virustotal flags these DLLs as safe...I hope these DLLs to be placed on "System32" and/or "SysWoW64" are really very safe?
Strange installation method....why avisynth+ did not install them in first place??

[jagabo]

Yes, they are safe -- as long as you got them from fftw.org, not just some random web site in Russia.

Those are not AviSynth filters, they are system wide libraries (fast Fourier transform) for Windows (many programs may use them).

https://en.wikipedia.org/wiki/Fast_Fourier_transform

[ls_milkyway]

I have tried this successfully on staxrip:

v = LWLibavVideoSource("%source_file%")
a = LWLibavAudioSource("%source_file%")
AudioDub(v,a)
AssumeFPS(26.054)
FrameRateConverter(NewNum=60000, NewDen=1000)

Note: AssumeFPS is 26.054 because that's what mediainfo is reporting for full video.

Output has tremendously improved....thanks! ...but audio is still out of sync?
I guess re-adjusting time-stamps of the recorded clip by using one of the following FFMPEG command (no re-encoding & changing container to mkv):

ffmpeg -fflags +genpts -i input.mp4 -c:v copy -c:a copy output.mkv
OR
ffmpeg -i input.mp4 -codec copy output.mkv

& then encoding will solve the problem ....Because I guess I used 1 of them & quickly encoded with fastest x264 settings to find that the audio is indeed in SYNC....else it's taking around 7-8 hours for full video with x264's very slow preset & avisynth+ settings.

[jagabo]

I have tried this successfully on staxrip:

v = LWLibavVideoSource("%source_file%")
a = LWLibavAudioSource("%source_file%")
AudioDub(v,a)
AssumeFPS(26.054)
FrameRateConverter(NewNum=60000, NewDen=1000)

Note: AssumeFPS is 26.054 because that's what mediainfo is reporting for full video.

Output has tremendously improved....thanks! ...but audio is still out of sync?

That's why I said earlier "assuming MediaInfo's reported average frame rate is correct". There's not much reason to believe it is correct. You need to figure out in what way the audio is out of sync and adjust things. If it's constant throughout the video adjust the initial offset. If it starts out in sync and gets gradually out of sync change the frame rate to get them to match. It gets harder if the sync changes suddenly at several points during the video.

I guess re-adjusting time-stamps of the recorded clip by using one of the following FFMPEG command (no re-encoding & changing container to mkv):

ffmpeg -fflags +genpts -i input.mp4 -c:v copy -c:a copy output.mkv
OR
ffmpeg -i input.mp4 -codec copy output.mkv

& then encoding will solve the problem ....Because I guess I used 1 of them & quickly encoded with fastest x264 settings to find that the audio is indeed in SYNC....else it's taking around 7-8 hours for full video with x264's very slow preset & avisynth+ settings.

Don't those leave the video just as jerky as it started out?

[ls_milkyway]

That's why I said earlier "assuming MediaInfo's reported average frame rate is correct". There's not much reason to believe it is correct. You need to figure out in what way the audio is out of sync and adjust things. If it's constant throughout the video adjust the initial offset. If it starts out in sync and gets gradually out of sync change the frame rate to get them to match. It gets harder if the sync changes suddenly at several points during the video.

I have already tried setting a delay (as audio starts before the video) in mkv container of the encoded video....but it later gets out sync again....clearly meaning the sync changes suddenly at several points.......as VFR fluctuates to much in low/high range.

Don't those leave the video just as jerky as it started out?

Nope, because SOURCE (recorded video is in MP4 with wrong audio time-stamps...I guess) -----> Changes to MKV container (without encoding) & having rectified time-stamps by FFMPEG ------>Finally filtered by Avisynth+ & encoded by staxrip in x264....also this guide (scroll down to the last paragraph) states FFMPEG is the last/only resort to handle out of sync audio due to VFR.

Hence, passing the source to FFMPEG ....sort of repairs the time-stamps.....best part is we have only changed the container to mkv without any quality loss.