Using ffmpeg to make an html5 webm video scroll smoothly

[ScottyBe]

Hi. I posted this question at StackOverflow but didn't get an answer, so I figured I should have come to the experts here in the first place!

I used the sample code at https://codepen.io/ollieRogers/pen/lfeLc to show frames of a (background) video as the user scrolls the page. The webm video used there plays very smoothly, but my video at https://codepen.io/Scottybe/pen/PBEOzG stutters quite a bit, so I'm assuming the problem is with the video itself.

I've used ffprobe to see if I can tell the difference between the two videos with no luck. Both have similar frame sizes, bit rates, frame rates, and lengths.

Is there a recommended way to create html5 video files that will "scroll" smoothly, i.e, that will respond quickly to the window.requestAnimationFrame() call? (I got a hint it may have something to do with "fragmentation" but not sure what to do with it.) An ffmpeg command to encode correctly would be even better! Thanks!

[pandy]

I've used ffprobe to see if I can tell the difference between the two videos with no luck. Both have similar frame sizes, bit rates, frame rates, and lengths.

You need more keyframes - more keyframes = better seekability of your file (adverse effect is higher bitrate requirement) - perhaps you can favor more P frames and prevent to encode B frames - tested both video files for key frames with ffprobe and hey are dramatically different:

@ffprobe -hide_banner -v 32 -stats -y -skip_frame nokey -i filename -select_streams v:0 -print_format compact -show_entries "frame=key_frame,pkt_pts_time" > "filename.txt"

for your video this is result:

frame|key_frame=1|pkt_pts_time=0.003000
frame|key_frame=1|pkt_pts_time=4.274000

and for reference:

frame|key_frame=1|pkt_pts_time=0.033000
frame|key_frame=1|pkt_pts_time=0.434000
frame|key_frame=1|pkt_pts_time=0.834000
frame|key_frame=1|pkt_pts_time=1.235000
frame|key_frame=1|pkt_pts_time=1.635000
frame|key_frame=1|pkt_pts_time=2.035000
frame|key_frame=1|pkt_pts_time=2.436000
frame|key_frame=1|pkt_pts_time=2.836000
frame|key_frame=1|pkt_pts_time=3.237000
frame|key_frame=1|pkt_pts_time=3.637000
frame|key_frame=1|pkt_pts_time=4.037000
frame|key_frame=1|pkt_pts_time=4.438000
frame|key_frame=1|pkt_pts_time=4.838000
frame|key_frame=1|pkt_pts_time=5.239000
frame|key_frame=1|pkt_pts_time=5.639000
frame|key_frame=1|pkt_pts_time=6.039000
frame|key_frame=1|pkt_pts_time=6.440000
frame|key_frame=1|pkt_pts_time=6.840000
frame|key_frame=1|pkt_pts_time=7.241000
frame|key_frame=1|pkt_pts_time=7.641000
frame|key_frame=1|pkt_pts_time=8.041000
frame|key_frame=1|pkt_pts_time=8.442000
frame|key_frame=1|pkt_pts_time=8.842000
frame|key_frame=1|pkt_pts_time=9.243000
frame|key_frame=1|pkt_pts_time=9.643000
frame|key_frame=1|pkt_pts_time=10.043000
frame|key_frame=1|pkt_pts_time=10.444000
frame|key_frame=1|pkt_pts_time=10.844000
frame|key_frame=1|pkt_pts_time=11.245000
frame|key_frame=1|pkt_pts_time=11.645000
frame|key_frame=1|pkt_pts_time=12.045000
frame|key_frame=1|pkt_pts_time=12.446000
frame|key_frame=1|pkt_pts_time=12.846000
frame|key_frame=1|pkt_pts_time=13.247000
frame|key_frame=1|pkt_pts_time=13.647000
frame|key_frame=1|pkt_pts_time=14.047000
frame|key_frame=1|pkt_pts_time=14.448000
frame|key_frame=1|pkt_pts_time=14.848000
frame|key_frame=1|pkt_pts_time=15.249000
frame|key_frame=1|pkt_pts_time=15.649000
frame|key_frame=1|pkt_pts_time=16.049000
frame|key_frame=1|pkt_pts_time=16.450000
frame|key_frame=1|pkt_pts_time=16.850000
frame|key_frame=1|pkt_pts_time=17.251000
frame|key_frame=1|pkt_pts_time=17.651000
frame|key_frame=1|pkt_pts_time=18.051000
frame|key_frame=1|pkt_pts_time=18.452000
frame|key_frame=1|pkt_pts_time=18.852000
frame|key_frame=1|pkt_pts_time=19.253000
frame|key_frame=1|pkt_pts_time=19.653000
frame|key_frame=1|pkt_pts_time=20.053000
frame|key_frame=1|pkt_pts_time=20.454000
frame|key_frame=1|pkt_pts_time=20.854000
frame|key_frame=1|pkt_pts_time=21.255000
frame|key_frame=1|pkt_pts_time=21.655000
frame|key_frame=1|pkt_pts_time=22.055000
frame|key_frame=1|pkt_pts_time=22.456000
frame|key_frame=1|pkt_pts_time=22.856000
frame|key_frame=1|pkt_pts_time=23.257000
frame|key_frame=1|pkt_pts_time=23.657000
frame|key_frame=1|pkt_pts_time=24.057000
frame|key_frame=1|pkt_pts_time=24.458000
frame|key_frame=1|pkt_pts_time=24.858000
frame|key_frame=1|pkt_pts_time=25.259000
frame|key_frame=1|pkt_pts_time=25.659000
frame|key_frame=1|pkt_pts_time=26.059000
frame|key_frame=1|pkt_pts_time=26.460000
frame|key_frame=1|pkt_pts_time=26.860000
frame|key_frame=1|pkt_pts_time=27.261000
frame|key_frame=1|pkt_pts_time=27.661000
frame|key_frame=1|pkt_pts_time=28.061000
frame|key_frame=1|pkt_pts_time=28.462000
frame|key_frame=1|pkt_pts_time=28.862000
frame|key_frame=1|pkt_pts_time=29.263000
frame|key_frame=1|pkt_pts_time=29.663000
frame|key_frame=1|pkt_pts_time=30.063000
frame|key_frame=1|pkt_pts_time=30.464000
frame|key_frame=1|pkt_pts_time=30.864000
frame|key_frame=1|pkt_pts_time=31.098000
frame|key_frame=1|pkt_pts_time=31.498000
frame|key_frame=1|pkt_pts_time=31.899000
frame|key_frame=1|pkt_pts_time=32.299000
frame|key_frame=1|pkt_pts_time=32.699000
frame|key_frame=1|pkt_pts_time=33.100000
frame|key_frame=1|pkt_pts_time=33.500000
frame|key_frame=1|pkt_pts_time=33.901000

in your case distance between keyframes is 4 seconds, where in reference there is approx .5 sec distance - decoder need o decode all frames between keyframes so you pushing CPU (and memory) load to max.

So you need to control so called GOP size (or better introduce more keyframes within longer GOP) and force video codec to produce video with plenty of Random Access Points.
More I and P frames, reduce or no B frames should improve seekability. Have no experience with VP8 (it is clearly seen why H.264 is more superior).
Perfectly smooth random access require only Intra frames (all frames are keyframes) but bitrate requirements are very high for such video - so you need to find optimal ratio between I and P frames (perhaps B too) - for older codecs it was around .5seconds (that's why for DVD GOP is 12 - 15 max 18 frames long - depend on framerate).

[pandy]

made simple script to encode webm with vorbis and vp8 - perhaps it will be helpful

@rem set audio quality
@set aq=3

@rem set video quality
@set vq=8

@rem set GOP size
@set GOP=15

@rem set number of B frames
@set B=2

@set /A k_min=(%GOP%/4)+1
@set /A qpmin=(%vq%/4)+3

@set filename=%~1

@rem audio signal processing
@set aproc="pan=stereo|FL < FL+1.414FC+0.5BL+0.5SL+0.25LFE|FR < FR+1.414FC+0.5BR+0.5SR+0.25LFE,firequalizer=gain='if(gte(f,30),0,-INF)+if(lte(f,14000),0,-INF)',dynaudnorm=p=1/sqrt(2),firequalizer=gain='if(gte(f,30),0,-INF)+if(lte(f,14000),0,-INF)',aresample=resampler=soxr:osr=44100:cutoff=0.990:precision=20:dither_method=0"

@rem video signal processing
@set vproc="pp=ac,scale=480:-4:sws_flags=spline+accurate_rnd+full_chroma_int+full_chroma_inp:interl=-1:out_range=auto:out_color_matrix=bt601,format=pix_fmts=yuv420p"

@ffmpeg -hide_banner -loglevel 32 -stats -color_range 2 -y -i "%filename%" -c:v libvpx -tune 0 -deadline best -quality best -rc_lookahead %GOP% -q:v %vq% -crf %vq% -qmin %qpmin% -bf %B% -keyint_min %k_min% -g %GOP% -vf %vproc% -color_range 2 -pix_fmt yuv420p -af %aproc% -c:a libvorbis -q:a %aq% -f webm -dash 0 "%~n1.webm"

[ScottyBe]

Thanks Pansy! I’ll try this out when I get home (as well as create the other html5 video formats) and let you know how it goes.

[ScottyBe]

Pansy, I spent a little time converting your script from Windows to Mac!

newname="new_$1"

# set audio quality
aq=3

# set video quality
vq=8

# set GOP size
GOP=15

# set number of B frames
B=2

let k_min=(GOP/4)+1
let qpmin=(vq/4)+3

#audio signal processing
aproc="pan=stereo|FL < FL+1.414FC+0.5BL+0.5SL+0.25LFE|FR < FR+1.414FC+0.5BR+0.5SR+0.25LFE,firequalizer=gain=\"expr:if(gte(f,30),0,-INF)+if(lte(f,14000),0,-INF)\",dynaudnorm=p=\"expr:1/sqrt(2)\",firequalizer=gain=\"expr:if(gte(f,30),0,-INF)+if(lte(f,14000),0,-INF)\",aresample=resampler=soxr:osr=44100:cutoff=0.990:precision=20:dither_method=0"

# video signal processing
vproc="pp=ac,scale=480:-4:sws_flags=spline+accurate_rnd+full_chroma_int+full_chroma_inp:interl=-1:out_range=auto:out_color_matrix=bt601,format=pix_fmts=yuv420p"

command="ffmpeg -hide_banner -loglevel 32 -stats -color_range 2 -y -i $1 -c:v libvpx -tune 0 -deadline best -quality best -rc_lookahead $GOP -q:v $vq -crf $vq -qmin $qpmin -bf $B -keyint_min $k_min -g $GOP -vf $vproc -color_range 2 -pix_fmt yuv420p -af $aproc -c:a libvorbis -q:a $aq -f webm -dash 0 $newname"

echo $command

output:

ffmpeg -hide_banner -loglevel 32 -stats -color_range 2 -y -i test.webm -c:v libvpx -tune 0 -deadline best -quality best -rc_lookahead 15 -q:v 8 -crf 8 -qmin 5 -bf 2 -keyint_min 4 -g 15 -vf pp=ac,scale=480:-4:sws_flags=spline+accurate_rnd+full_chroma_int+full_chroma_inp:interl=-1:out_range=auto:out_color_matrix=bt601,format=pix_fmts=yuv420p -color_range 2 -pix_fmt yuv420p -af pan=stereo|FL < FL+1.414FC+0.5BL+0.5SL+0.25LFE|FR < FR+1.414FC+0.5BR+0.5SR+0.25LFE,firequalizer=gain="expr:if(gte(f,30),0,-INF)+if(lte(f,14000),0,-INF)",dynaudnorm=p="expr:1/sqrt(2)",firequalizer=gain="expr:if(gte(f,30),0,-INF)+if(lte(f,14000),0,-INF)",aresample=resampler=soxr:osr=44100:cutoff=0.990:precision=20:dither_method=0 -c:a libvorbis -q:a 3 -f webm -dash 0 new_test.webm

Unfortunately, when I execute the ffmpeg command, I get:

-bash: FL+1.414FC+0.5BL+0.5SL+0.25LFE: No such file or directory
-bash: FR+1.414FC+0.5BR+0.5SR+0.25LFE,firequalizer=gain=expr:if(gte(f,30),0,-INF)+if(lte(f,14000),0,-INF),dynaudnorm=p=expr:1/sqrt(2),firequalizer=gain=expr:if(gte(f,30),0,-INF)+if(lte(f,14000),0,-INF),aresample=resampler=soxr:osr=44100:cutoff=0.990:precision=20:dither_method=0: No such file or directory
Trailing options were found on the commandline.
Input #0, matroska,webm, from 'test.webm':
  Metadata:
    encoder         : Lavf52.62.0
  Duration: 00:00:34.18, start: 0.033000, bitrate: 473 kb/s
    Stream #0:0(eng): Video: vp8, yuv420p(pc, progressive), 640x360, SAR 1:1 DAR 16:9, 29.97 fps, 29.97 tbr, 1k tbn, 1k tbc (default)
    Stream #0:1(eng): Audio: vorbis, 44100 Hz, stereo, fltp (default)
At least one output file must be specified

I'm stuck. Any idea where I went wrong? Thanks!

[poisondeathray]

You're probably missing an escape or parentheses. But I don't see why you need all those options. Just way overcomplicating things. I don't think you have 5.1 audio right ? I'd just cut all that crap out to the most simplest and go from there if you still want to add options or customize things

If you just want to set the max keyframe interval it would be -g in ffmpeg . e.g. -g 15 would be a max keyframe interval of 15

ffmpeg -i input.ext -c:v libvpx -b:v 1.5M -g 15 -c:a libvorbis -b:a 128k output.webm

I downloaded your video and tested it locally, and indeed there was high latency scrolling (tested locally, to rule out bandwidth issues)

So I took your video and re-encoded it using those settings (really only the -g 15 is the most important) , plugged it back into the HTML5/JS code and it scrolled smoothly (the quality of the video was low, but it was low to begin with)

If you're still experiencing stutter or high latency scrolling, then i would do a local host test first (to rule out CDN bandwidth issues) . If there it's still too slow for you (it was about the same as the original demo for me, maybe even smoother) , you can look at other things like reducing the keyframe interval even more, or capping bitrate (VBV). Report back if you still need help with optimizing encoding settings, but I think that should be good enough

[poisondeathray]

and I have to quote this because it's funny

Thanks Pansy! I’ll try this out when I get home (as well as create the other html5 video formats) and let you know how it goes.

Pansy, I spent a little time converting your script from Windows to Mac!

It's "pandy", not "Pansy"

Where I come from, calling someone a "pansy" has other connotations - you can look it up! It's the same in the USA

[ScottyBe]

Sorry bout that! I was on my phone...

And I figured out my problem with my shell script. I ended up running your script on a PC to get the actual final ffmpeg command, and I went from there. Thanks again.

[ScottyBe]

More questions Pandy!

Here's my working shell script for others who may want to do the same thing:

newname="new_$1"

# set audio quality
aq=3

# set video quality
vq=8

# set GOP size
GOP=15

# set number of B frames
B=2

let k_min=(GOP/4)+1
let qpmin=(vq/4)+3

#audio signal processing
aproc="pan=stereo|FL < FL+1.414FC+0.5BL+0.5SL+0.25LFE|FR < FR+1.414FC+0.5BR+0.5SR+0.25LFE,firequalizer=gain='if(gte(f,30),0,-INF)+if(lte(f,14000),0,-INF)',dynaudnorm=p=1/sqrt(2),firequalizer=gain='if(gte(f,30),0,-INF)+if(lte(f,14000),$

# video signal processing
vproc="pp=ac,scale=480:-4:sws_flags=spline+accurate_rnd+full_chroma_int+full_chroma_inp:interl=-1:out_range=auto:out_color_matrix=bt601,format=pix_fmts=yuv420p"

command="ffmpeg -hide_banner -loglevel 32 -stats -color_range 2 -y -i \"$1\" -c:v libvpx -tune 0 -deadline best -quality best -rc_lookahead $GOP -q:v $vq -crf $vq -qmin $qpmin -bf $B -keyint_min $k_min -g $GOP -vf \"$vproc\" -color_ra$

eval $command

I'm very much a beginner with ffmpeg. What settings would keep the output at the same quality as the original video?

Also, how can I do the same thing for mp4 version?

[pandy]

More questions Pandy!

I'm very much a beginner with ffmpeg. What settings would keep the output at the same quality as the original video?

Also, how can I do the same thing for mp4 version?

You can adjust video quality by CRF factor - this is recommended way - generally for modern codecs short GOP's may deliver suboptimal quality so ability to seek video may be contradictory to "best quality at particular bitrate". VP8 is very limited (i was not aware this but seem VP8 doesn't support B frames ).
Keeping "same quality" is usually impossible as loosing quality is unavoidable when lossy codec is used - you can only try to keep quality sufficiently close to original but at some point you may end in bitrates higher than source - depends on you...

Webm, MP4 are video containers - in ffmpeg video containers are controlled by option '-f' so if you decide to change container you can use simply '-f mp4' (also ffmpeg is sufficiently smart to automatically select desired container based on your output file extension - so using 'filename.mp4' there is high chance that ffmpeg will use mp4 container not for example wav or avi) - usually container may have additional features controlled by flags or commands - for example YT recommends to use flag faststart (Run a second pass to put the index (moov atom) at the beginning of the file) so i use:

-f mp4 -movflags faststart

Not sure what codec will be used by you with mp4 container - don't know if VP8 is supported by mp4 anyway unless VP8 is absolute must (web browser & HTML5 video object capabilities) then i would recommend something modern like H.264 or VP9 (beware that VP8, VP9 encoders are usually way slower than H.264).

And yes, my scripts share similar structure so they are "full of crap" (by principle - they work for me so they should work for you too).

And yes, i was aware of 'pansy' meaning but i assumed this is not intentional typo thus don't care.