VP8 adding frames, what did I do wrong?

[Selur]

What did I do wrong?

I wanted to do a little VP8 vs. x264 comparision using http://pdl.stream.aol.com/pdlext/aol/brightcove/ame/201204/04/42907/totalrecall_trlr_01_1080p_dl.mov as source.

1. I wrote a small script for decoding, colormatrix adjustemt and resizing:

AvisynthScript:

D:\Encoding Temp\encodingTempAvisynthSkript.avs ### SetMemoryMax(768)
SetMTMode(5,8)
LoadPlugin("G:\Hybrid\avisynthPlugins\ffms2.dll")
LoadPlugin("G:\Hybrid\avisynthPlugins\ColorMatrix.dll")
FFVideoSource("D:\comparison\totalrecall_trlr_01_1080p_dl.mov",cachefile="D:\Encoding Temp\totalrecall_trlr_01_1080p_dl_19_02_39_961.ffindex",threads=1)
SetMTMode(2)
ConvertToYv12(interlaced=false, matrix="Rec.601")
ColorMatrix(mode="Rec.709->Rec.601",clamp=3,interlaced=false)
Crop(0,140,1920,800)
Spline16Resize(720,364)
distributor()
return(last)

2. I encoded the clip with VP8 and muxed the output with mkvmerge:

VP8 1st pass:

mencoder -lavdopts threads=8 -really-quiet -ofps 24000/1001 -of rawvideo -o - -ovc raw -noskip -vf scale,format=i420 -forcedsubsonly -nosub -nosound -mc 0 "D:\Encoding Temp\encodingTempAvisynthSkript.avs" | ffmpeg -v -10 -threads 8  -r 23.976 -pix_fmt yuv420p -s 720x364 -f rawvideo -i - -an -r 23.976 -f yuv4mpegpipe - | vpxenc --codec=vp8 --passes=2 --pass=1 --target-bitrate=800 --end-usage=0 --fpf="D:\Encoding Temp\VP8_800_720x364_19_02_39_961_03.stats" --profile=0 --best --cpu-used=0 --bias-pct=70 --minsection-pct=1 --maxsection-pct=10000 --min-q=0 --max-q=63 --lag-in-frames=16 --undershoot-pct=0 --buf-sz=6 --buf-initial-sz=4 --buf-optimal-sz=5 --drop-frame=0 --resize-allowed=0 --kf-min-dist=0 --kf-max-dist=250 --auto-alt-ref=1 --noise-sensitivity=0 --sharpness=0 --static-thresh=0 --token-parts=1 --arnr-maxframes=5 --arnr-strength=3 --threads=16 --width=720 --height=364 --timebase=1001/24000 --yv12 -o NUL -

VP8 2nd pass:

mencoder -lavdopts threads=8 -really-quiet -ofps 24000/1001 -of rawvideo -o - -ovc raw -noskip -vf scale,format=i420 -forcedsubsonly -nosub -nosound -mc 0 "D:\Encoding Temp\encodingTempAvisynthSkript.avs" | ffmpeg -v -10 -threads 8  -r 23.976 -pix_fmt yuv420p -s 720x364 -f rawvideo -i - -an -r 23.976 -f yuv4mpegpipe - | vpxenc --codec=vp8 --passes=2 --pass=2 --target-bitrate=800 --end-usage=0 --fpf="D:\Encoding Temp\VP8_800_720x364_19_02_39_961_03.stats" --profile=0 --best --cpu-used=0 --bias-pct=70 --minsection-pct=1 --maxsection-pct=10000 --min-q=0 --max-q=63 --lag-in-frames=16 --undershoot-pct=0 --buf-sz=6 --buf-initial-sz=4 --buf-optimal-sz=5 --drop-frame=0 --resize-allowed=0 --kf-min-dist=0 --kf-max-dist=250 --auto-alt-ref=1 --noise-sensitivity=0 --sharpness=0 --static-thresh=0 --token-parts=1 --arnr-maxframes=5 --arnr-strength=3 --threads=16 --width=720 --height=364 --timebase=1001/24000 --yv12 -o "D:\Encoding Temp\VP8_800_720x364.vp8" -

Muxing:

mkvmerge --ui-language en -o "D:\Encoding Output\VP8_800_720x364.webm" --webm -d 0 --default-track 0:yes --default-duration 0:24000/1001fps --aspect-ratio-factor 0:40/33 --no-chapters --forced-track 0:yes --no-audio --no-subtitles "D:\Encoding Temp\VP8_800_720x364.vp8"

result: http://www.embedupload.com/?d=7GMUA0MTRN

3. I encoded the clip with x264 and muxed the output with mkvmerge:

x264 1st pass:

mencoder -lavdopts threads=8 -really-quiet -ofps 24000/1001 -of rawvideo -o - -ovc raw -noskip -vf scale,format=i420 -forcedsubsonly -nosub -nosound -mc 0 "D:\Encoding Temp\encodingTempAvisynthSkript.avs" | x264 --pass 1 --bitrate 800 --bframes 8 --direct auto --b-adapt 2 --ratetol 2 --rc-lookahead 60 --aq-mode 2 --sar 40:33 --deblock -1:-1 --stats "D:\Encoding Temp\x264_800_720x364_19_08_58_731_03.stats" --input-res 720x364 --output NUL -

x264 2nd pass:

mencoder -lavdopts threads=8 -really-quiet -ofps 24000/1001 -of rawvideo -o - -ovc raw -noskip -vf scale,format=i420 -forcedsubsonly -nosub -nosound -mc 0 "D:\Encoding Temp\encodingTempAvisynthSkript.avs" | x264 --preset veryslow --pass 2 --bitrate 800 --ratetol 2 --aq-mode 2 --sar 40:33 --deblock -1:-1 --colormatrix bt470bg --stats "D:\Encoding Temp\x264_800_720x364_19_08_58_731_03.stats" --input-res 720x364 --output "D:\Encoding Temp\x264_800_720x364_19_08_58_731_04.264" -

Muxing:

mkvmerge --ui-language en -o "D:\Encoding Output\x264_800_720x364.mkv" -d 0 --default-track 0:yes --default-duration 0:24000/1001fps --aspect-ratio-factor 0:40/33 --fourcc 0:MP4V --no-chapters --forced-track 0:yes --no-audio --no-subtitles "D:\Encoding Temp\x264_800_720x364_19_08_58_731_04.264"

result: http://www.embedupload.com/?d=2XI3DZZDAW

4. I wrote a small script to compare the results

Avisynth script for comparison:

LoadPlugin("G:\Hybrid\avisynthPlugins\ffms2.dll")
LoadPlugin("G:\Hybrid\avisynthPlugins\ColorMatrix.dll")
Import("G:\Hybrid\avisynthPlugins\FFMS2.avsi")
vp8 = FFVideoSource("VP8_800_720x364.webm", threads=1).FFInfo()
x264 = FFVideoSource("x264_800_720x364.mkv", threads=1).FFInfo()
source = FFVideoSource("D:\comparison\totalrecall_trlr_01_1080p_dl.mov", threads=1).ColorMatrix(mode="Rec.709->Rec.601",clamp=3,interlaced=false).Crop(0,140,1920,800).Spline16Resize(720,364)
source = source.FFInfo()
vp8 = vp8.subtitle("vp8", align=9)
x264 = x264.subtitle("x264", align=9)
source = source.subtitle("source", align=9)
StackHorizontal(StackVertical(vp8, source), StackVertical(source, x264))

So far so good, I thought until I saw the first frame and had to realize that the VP8 clip was longer than the source!



If anyone knows what's going wrong please post.

Cu Selur

[Selur]

I also tried the following:

1st pass VP8:

mencoder -lavdopts threads=8 -really-quiet -ofps 24000/1001 -of rawvideo -o - -ovc raw -noskip -vf crop=720:364:0:140,scale=1920:1178,format=i420 -sws 10 -forcedsubsonly -nosub -nosound -mc 0 "D:\comparison\totalrecall_trlr_01_1080p_dl.mov" | ffmpeg -v -10 -threads 8  -r 23.976 -pix_fmt yuv420p -s 1920x1178 -f rawvideo -i - -an -r 23.976 -f yuv4mpegpipe - | vpxenc --codec=vp8 --passes=2 --pass=1 --target-bitrate=800 --end-usage=0 --fpf="D:\Encoding Temp\VP8_800_720x364_withoutAvisynth_19_35_28_501_01.stats" --profile=0 --best --cpu-used=0 --bias-pct=70 --minsection-pct=1 --maxsection-pct=10000 --min-q=0 --max-q=63 --lag-in-frames=16 --undershoot-pct=0 --buf-sz=6 --buf-initial-sz=4 --buf-optimal-sz=5 --drop-frame=0 --resize-allowed=0 --kf-min-dist=0 --kf-max-dist=250 --auto-alt-ref=1 --noise-sensitivity=0 --sharpness=0 --static-thresh=0 --token-parts=1 --arnr-maxframes=5 --arnr-strength=3 --threads=16 --width=1920 --height=1178 --timebase=1001/24000 --yv12 -o NUL -

2nd pass VP8:

mencoder -lavdopts threads=8 -really-quiet -ofps 24000/1001 -of rawvideo -o - -ovc raw -noskip -vf crop=720:364:0:140,scale=1920:1178,format=i420 -sws 10 -forcedsubsonly -nosub -nosound -mc 0 "D:\comparison\totalrecall_trlr_01_1080p_dl.mov" | ffmpeg -v -10 -threads 8  -r 23.976 -pix_fmt yuv420p -s 1920x1178 -f rawvideo -i - -an -r 23.976 -f yuv4mpegpipe - | vpxenc --codec=vp8 --passes=2 --pass=2 --target-bitrate=800 --end-usage=0 --fpf="D:\Encoding Temp\VP8_800_720x364_withoutAvisynth_19_35_28_501_01.stats" --profile=0 --best --cpu-used=0 --bias-pct=70 --minsection-pct=1 --maxsection-pct=10000 --min-q=0 --max-q=63 --lag-in-frames=16 --undershoot-pct=0 --buf-sz=6 --buf-initial-sz=4 --buf-optimal-sz=5 --drop-frame=0 --resize-allowed=0 --kf-min-dist=0 --kf-max-dist=250 --auto-alt-ref=1 --noise-sensitivity=0 --sharpness=0 --static-thresh=0 --token-parts=1 --arnr-maxframes=5 --arnr-strength=3 --threads=16 --width=1920 --height=1178 --timebase=1001/24000 --yv12 -o "D:\Encoding Temp\VP8_800_720x364_withoutAvisynth.vp8" -

Muxing:

mkvmerge --ui-language en -o "D:\Encoding Output\VP8_800_720x364_withoutAvisynth.webm" --webm -d 0 --default-track 0:yes --default-duration 0:24000/1001fps --aspect-ratio-factor 0:40/33 --no-chapters --forced-track 0:yes --no-audio --no-subtitles "D:\Encoding Temp\VP8_800_720x364_withoutAvisynth.vp8"

result: http://www.embedupload.com/?d=6QAEORMGHA

Now the output has 3457 frames,...

-> Why is VP8 adding frames,... or if it's not VP8 what is adding the frames and why?

Cu Selur

[Selur]

encoding with x264 without Avisynth:

x2641st pass:

mencoder -lavdopts threads=8 -really-quiet -ofps 24000/1001 -of rawvideo -o - -ovc raw -noskip -vf crop=1920:800:0:140,scale=720:364,format=i420 -sws 10 -forcedsubsonly -nosub -nosound -mc 0 "D:\comparison\totalrecall_trlr_01_1080p_dl.mov" | x264 --pass 1 --bitrate 800 --bframes 16 --direct auto --b-adapt 2 --ratetol 2 --rc-lookahead 60 --aq-mode 2 --sar 40:33 --deblock -1:-1 --stats "D:\Encoding Temp\x264_800_720x364_withoutAvisynth_19_57_49_961_01.stats" --input-res 720x364 --output NUL -

x264 2nd pass:

mencoder -lavdopts threads=8 -really-quiet -ofps 24000/1001 -of rawvideo -o - -ovc raw -noskip -vf crop=1920:800:0:140,scale=720:364,format=i420 -sws 10 -forcedsubsonly -nosub -nosound -mc 0 "D:\comparison\totalrecall_trlr_01_1080p_dl.mov" | x264 --preset veryslow --tune film --pass 2 --bitrate 800 --bframes 16 --ratetol 2 --aq-mode 2 --sar 40:33 --colormatrix bt709 --stats "D:\Encoding Temp\x264_800_720x364_withoutAvisynth_19_57_49_961_01.stats" --input-res 720x364 --output "D:\Encoding Temp\x264_800_720x364_withoutAvisynth_19_57_49_961_02.264" -

Muxing:

mkvmerge --ui-language en -o "D:\Encoding Output\x264_800_720x364_withoutAvisynth.mkv" -d 0 --default-track 0:yes --default-duration 0:24000/1001fps --aspect-ratio-factor 0:40/33 --fourcc 0:MP4V --no-chapters --forced-track 0:yes --no-audio --no-subtitles "D:\Encoding Temp\x264_800_720x364_withoutAvisynth_19_57_49_961_02.264"

result: http://www.embedupload.com/?d=0HIXAGEURV

frame count: 3377 wtf!?
-> something is fishy here

[Selur]

Okay, think I found the route of the problem
Analysing the input:

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : Main@L4.0
Format settings, CABAC                   : No
Format settings, ReFrames                : 2 frames
Format settings, GOP                     : M=2, N=60
Codec ID                                 : avc1
Codec ID/Info                            : Advanced Video Coding
Duration                                 : 2mn 21s
Bit rate                                 : 6 623 Kbps
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Variable
Frame rate                               : 23.976 fps
Minimum frame rate                       : 23.970 fps
Maximum frame rate                       : 24.212 fps
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 0.133
Stream size                              : 111 MiB (98%)
Language                                 : English
Encoded date                             : UTC 2012-04-04 23:58:38
Tagged date                              : UTC 2012-04-05 00:51:31
Color primaries                          : BT.709-5, BT.1361, IEC 61966-2-4, SMPTE RP177
Transfer characteristics                 : BT.709-5, BT.1361
Matrix coefficients                      : BT.709-5, BT.1361, IEC 61966-2-4 709, SMPTE RP177

didn't see that the input was vfr! (wtf? since when is are mov trailers vfr?)

[Selur]

did some further testing and it seems, vfr is not the root of the problem,...

Switching to ffmpeg as decoder

source: http://pdl.stream.aol.com/pdlext/aol/brightcove/ame/201204/04/42907/totalrecall_trlr_01_1080p_dl.mov

Avisynth script used for decoding, resize and color adjustment:

SetMemoryMax(768)
SetMTMode(5,8)
LoadPlugin("G:\Hybrid\avisynthPlugins\ffms2.dll")
LoadPlugin("G:\Hybrid\avisynthPlugins\ColorMatrix.dll")
FFVideoSource("D:\comparison\totalrecall_trlr_01_1080p_dl.mov",threads=1)
SetMTMode(2)
ConvertToYv12(interlaced=false, matrix="Rec.601")
ColorMatrix(mode="Rec.709->Rec.601",clamp=3,interlaced=false)
Crop(0,140,1920,800)
Spline16Resize(720,364)
distributor()
return(last)

VP8 1st pass:

ffmpeg -v -10 -i "D:\Encoding Temp\encodingTempAvisynthSkript_01_12_30_711.avs" -r 24000/1001 -threads 8 -vsync 0 -an -r 24000/1001 -pix_fmt yuv420p -f yuv4mpegpipe - | vpxenc --codec=vp8 --passes=2 --pass=1 --target-bitrate=800 --end-usage=0 --fpf="D:\Encoding Temp\vp8_800_720x364_01_12_30_711_03.stats" --profile=0 --best --cpu-used=0 --bias-pct=70 --minsection-pct=1 --maxsection-pct=10000 --min-q=0 --max-q=63 --lag-in-frames=16 --undershoot-pct=0 --buf-sz=6 --buf-initial-sz=4 --buf-optimal-sz=5 --drop-frame=0 --resize-allowed=0 --kf-min-dist=0 --kf-max-dist=250 --auto-alt-ref=1 --noise-sensitivity=0 --sharpness=0 --static-thresh=0 --token-parts=1 --arnr-maxframes=5 --arnr-strength=3 --threads=2 --width=720 --height=364 --timebase=1001/24000 --yv12 -o NUL -

-> Pass 1/2 frame 3385/3386 487584B 1152b/f 27628b/s 24211 ms (139.81 fps)

VP8 2nd pass:

ffmpeg -v -10 -i "D:\Encoding Temp\encodingTempAvisynthSkript_01_12_30_711.avs" -r 24000/1001 -threads 8 -vsync 0 -an -r 24000/1001 -pix_fmt yuv420p -f yuv4mpegpipe - | vpxenc --codec=vp8 --passes=2 --pass=2 --target-bitrate=800 --end-usage=0 --fpf="D:\Encoding Temp\vp8_800_720x364_01_12_30_711_03.stats" --profile=0 --best --cpu-used=0 --bias-pct=70 --minsection-pct=1 --maxsection-pct=10000 --min-q=0 --max-q=63 --lag-in-frames=16 --undershoot-pct=0 --buf-sz=6 --buf-initial-sz=4 --buf-optimal-sz=5 --drop-frame=0 --resize-allowed=0 --kf-min-dist=0 --kf-max-dist=250 --auto-alt-ref=1 --noise-sensitivity=0 --sharpness=0 --static-thresh=0 --token-parts=1 --arnr-maxframes=5 --arnr-strength=3 --threads=2 --width=720 --height=364 --timebase=1001/24000 --yv12 -o "D:\Encoding Temp\vp8_800_720x364.vp8" -

-> Pass 2/2 frame 3385/3502 14090043B 33299b/f 798400b/s 178531 ms (18.96 fps)

if I multiplex the output using:

mkvmerge --ui-language en -o "D:\Encoding  Output\vp8_800_720x364.webm" --webm -d 0 --default-track 0:yes  --default-duration 0:24000/1001fps --aspect-ratio-factor 0:40/33  --no-chapters --forced-track 0:yes --no-audio --no-subtitles  "D:\Encoding Temp\vp8_800_720x364.vp8"

3502 frames are reported

Now as a last attempt I used the file directly with ffmpeg:

ffmpeg -v -10 -i "D:\comparison\totalrecall_trlr_01_1080p_dl.mov" -r 24000/1001 -threads 8 -vsync 0 -an -r 24000/1001 -pix_fmt yuv420p -f yuv4mpegpipe - | vpxenc --codec=vp8 --passes=2 --pass=1 --target-bitrate=800 --end-usage=0 --fpf="D:\Encoding Temp\vp8_800_720x364_01_12_30_711_03.stats" --profile=0 --best --cpu-used=0 --bias-pct=70 --minsection-pct=1 --maxsection-pct=10000 --min-q=0 --max-q=63 --lag-in-frames=16 --undershoot-pct=0 --buf-sz=6 --buf-initial-sz=4 --buf-optimal-sz=5 --drop-frame=0 --resize-allowed=0 --kf-min-dist=0 --kf-max-dist=250 --auto-alt-ref=1 --noise-sensitivity=0 --sharpness=0 --static-thresh=0 --token-parts=1 --arnr-maxframes=5 --arnr-strength=3 --threads=2 --width=720 --height=364 --timebase=1001/24000 --yv12 -o NUL -

-> Pass 1/2 frame 3385/3386 487584B 1152b/f 27628b/s 165460 ms (20.46 fps)

ffmpeg -v -10 -i "D:\comparison\totalrecall_trlr_01_1080p_dl.mov" -r 24000/1001 -threads 8 -vsync 0 -an -r 24000/1001 -pix_fmt yuv420p -f yuv4mpegpipe - | vpxenc --codec=vp8 --passes=2 --pass=2 --target-bitrate=800 --end-usage=0 --fpf="D:\Encoding Temp\vp8_800_720x364_01_12_30_711_03.stats" --profile=0 --best --cpu-used=0 --bias-pct=70 --minsection-pct=1 --maxsection-pct=10000 --min-q=0 --max-q=63 --lag-in-frames=16 --undershoot-pct=0 --buf-sz=6 --buf-initial-sz=4 --buf-optimal-sz=5 --drop-frame=0 --resize-allowed=0 --kf-min-dist=0 --kf-max-dist=250 --auto-alt-ref=1 --noise-sensitivity=0 --sharpness=0 --static-thresh=0 --token-parts=1 --arnr-maxframes=5 --arnr-strength=3 --threads=2 --width=720 --height=364 --timebase=1001/24000 --yv12 -o "D:\Encoding Temp\vp8_800_720x364.vp8" -

-> Pass 2/2 frame 3385/3506 14206317B 33574b/f 804989b/s 461389 ms (7.34 fps)

if I multiplex the output using:

mkvmerge --ui-language en -o "D:\Encoding Output\vp8_800_720x364.webm" --webm -d 0 --default-track 0:yes --default-duration 0:24000/1001fps --aspect-ratio-factor 0:40/33 --no-chapters --forced-track 0:yes --no-audio --no-subtitles "D:\Encoding Temp\vp8_800_720x364.vp8"

3506 frames are reported


=> neither feeding vpxenc with ffmpeg or mencoder resulted in a result without to many frames


Cu Selur

[jimbankoski]

VP8 has something called alt ref frames ( http://blog.webmproject.org/2010/05/inside-webm-technology-vp8-alternate.html ) which are frames that are created from one or more frames for the sole purpose of better prediction, they are decoded but never shown to the user. The appearance of these packets is expected behavior. If for some reason it messes up synchronization of audio and video or other calculations - then its a bug.

[jagabo]

VP8 has something called alt ref frames ( http://blog.webmproject.org/2010/05/inside-webm-technology-vp8-alternate.html ) which are frames that are created from one or more frames for the sole purpose of better prediction, they are decoded but never shown to the user. The appearance of these packets is expected behavior.

Interesting. Good to know.

Selur, how are your comparisons going? You might find this type of comparison useful:

# If Videos start at different frames use frameadjust to align them
frameadjust=0
name1="video1.mkv"
name2="video2.mkv"

v1 = ffVideoSource(name1)
v2 = ffVideoSource(name2).trim(frameadjust,0)
sub = v1.subtract(v2) 
substrong = sub.levels(112,1,144,0,255) 

StackVertical(StackHorizontal(v1.subtitle(name1),v2.subtitle(name2)),StackHorizontal(sub.subtitle("Difference"),substrong.subtitle("Difference amplified 8x")))

[Selur]

Not so good, since I now have to contact the libav/ffmpegsource team since when I open:

LoadPlugin("G:\Hybrid\avisynthPlugins\ffms2.dll")
Import("G:\Hybrid\avisynthPlugins\FFMS2.avsi")
FFVideoSource("VP8_800_720x364.webm", threads=1).FFInfo()

to look at the file, all the 3500+ frames are visible,.. (which would make a frame-by-frame or a reencoding of vp8 material using ffmpegsource and Avisynth a real problem,..)
-> since my temp files got lost in a hdd crash this morning I have to redo the whole thing and then report the problem,..

[Selur]

added an issue report to http://code.google.com/p/ffmpegsource/issues/detail?id=88

Cu Selur