content affecting playback rate

[cheyrn]

I made an overlay video that is 3 frames of solid white then n frames of solid black, the point being to help me with timing the video which because of my weird project is something I have to do a few times per second... I was distressed to find that the time appeared to be way off. Like it appeared as though the first frame would appear a fraction of a second after the audio at the start, in windows media player or VLC or premiere or quicktime player.

It occurred to me try a dot instead of an entire frame of different color and the delay stopped happening.

So, my question is, do you think this is what my GPU having problems rendering looks like, or is there actually an affect on playback rate based on the amount of pixels that have to change in a frame? With the delay being out of sync with the audio.

[poisondeathray]

No, content won't affect a software player's playback (if you have decent hardware)

Complex content (e.g. action scenes, explosions) can cause playback problems with some hardware players, due to bitrate spikes and buffering issues . But plain solid colors won't elicit a large bitrate response if you used video compression

[cheyrn]

This is happening consistently. I don't have decent hardware. A different clip that switches dramatically from one frame to the next plays much slower, perhaps twice as slow.

For example, I have 2 seconds of audio and 2 seconds of video and then append another clip of 1 second so that something comes after it.

When the 2 seconds is a continuous scene the two clips take 3 seconds and appear to play continuously.

When the 2 seconds has no 2 frames next to each other that are from the same scene, the 2 seconds of audio stops while the rest of the clip plays, then the 1 second clip after it starts.

I know there is some type of compression that is based on not repeating pixels that are the same from one period to the next. If that is what the xvid mpeg encoder in virtual dub does, maybe it explains why since there would be no common pixels or almost no common pixels between frames.

[LMotlow]

If that is what the xvid mpeg encoder in virtual dub does...

I wouldn't think so. There's no such thing as an "Xvid mpeg encodr". XVid and MPEG are two different codecs. VirtualDub doesn't "come with" those encoders. They are part of the codec inventory you've installed in your system or with other software.

[cheyrn]

If I use adobe media encoder to encode it using h264 the result takes the correct amount of time and is in sync. So, this also makes me think that the content makes a difference. If it is my hardware struggling to keep up it still means that it's a difference based on content. If these examples are not coincidences.

[poisondeathray]

If I use adobe media encoder to encode it using h264 the result takes the correct amount of time and is in sync. So, this also makes me think that the content makes a difference. If it is my hardware struggling to keep up it still means that it's a difference based on content. If these examples are not coincidences.

How does that logic make any sense ? You encode the same content with AME and it's in sync - doesn't that suggest it's content INdependent ?

xvid = MPEG4-ASP or MPEG4 Part 2

I know there is some type of compression that is based on not repeating pixels that are the same from one period to the next. If that is what the xvid mpeg encoder in virtual dub does, maybe it explains why since there would be no common pixels or almost no common pixels between frames.

Both h.264 and xvid can use b-frames which store the residuals (what's left over between what's predicted, ie. essentially the differences between frames). If that's what you mean by "content" affecting it, basically all end distribution formats use some sort of long GOP compression . Default h.264 AME settings certainly use it. But b-frames can sometimes cause "b-frame decoder lag" with AVI container, but it's very unusual to experience this in modern software players . If you want to rule this out, encode some tests without b-frames . b-frames in general help with compression . b and p frames provide temporal compression

For example, I have 2 seconds of audio and 2 seconds of video and then append another clip of 1 second so that something comes after it.

Well "appending" introduces another variable that migth be causing it

[poisondeathray]

Do you want to get to the bottom of this quickly ? Upload some of your test clips

[cheyrn]

My reasoning is that if what I am seeing is to do with my hardware and not the video content then I would probably see the same thing regardless of the encoding.

I assume I can't actually upload clips right? Vimeo scales the videos and most likely changes the encoding.

I have a laborious process with which the clips could be created.... If there is a way to upload, I should do that.

Laborious process:

strobe.avs:

function blink(int len)
{
blankclip(height=1080 , width=1920, color=$FFFFFF, fps=60, length=1)
\ + blankclip(height=1080, width=1920, color=$000000, fps=60, length=(len - 1))
}

blink(12) + blink(12) + blink(12) + blink(23)
\ + blink(12) + blink(12) + blink(36)
\ + blink(24) + blink(24)


The above creates a video with the weird playback issu. If I add audio, the video continues on past the end of the audio for some time. I get the same result if I encode with b-vop unchecked in the xvid hd 1080 encoder (which I have in virtualdub).

I created an mp4 like so:

ffmpeg -i strobe.avi -vcodec libx264 strobe.mp4

The result plays in sync and at what appears to be the intended speed.

I made another where every frame has some pixels in common with surrounding frames and it was actually worse. So, I guess my theory is wrong. But, it still seems to me like content is making the difference. If I create a video with a roughly 100 pixel diameter dot, instead of bars or solid frames, it appear to play at the correct speed.

This makes the worse video (the two avis are 1 frame that is 1920 x135 one solid white the other solid black):

b = AVISource("black bar (135).avi")
w = AVISource("white bar (135).avi")

s = stackvertical( w, w, w, w, w, w, w, w)
\ + stackvertical( b, w, w, w, w, w, w, w)
\ + stackvertical( w, b, w, w, w, w, w, w)
\ + stackvertical( b, b, w, w, w, w, w, w)
\ + stackvertical( w, w, b, w, w, w, w, w)
\ + stackvertical( b, w, b, w, w, w, w, w)
\ + stackvertical( w, b, b, w, w, w, w, w)
\ + stackvertical( b, b, b, w, w, w, w, w)
\ + stackvertical( w, w, w, b, w, w, w, w)
\ + stackvertical( b, w, w, b, w, w, w, w)
\ + stackvertical( w, b, w, b, w, w, w, w)
\ + stackvertical( b, b, w, b, w, w, w, w)
\ + stackvertical( w, w, b, b, w, w, w, w)
\ + stackvertical( b, w, b, b, w, w, w, w)
\ + stackvertical( w, b, b, b, w, w, w, w)
\ + stackvertical( b, b, b, b, w, w, w, w)
\ + stackvertical( w, w, w, w, b, w, w, w)
\ + stackvertical( b, w, w, w, b, w, w, w)
\ + stackvertical( w, b, w, w, b, w, w, w)
\ + stackvertical( b, b, w, w, b, w, w, w)
\ + stackvertical( w, w, b, w, b, w, w, w)
\ + stackvertical( b, w, b, w, b, w, w, w)
\ + stackvertical( w, b, b, w, b, w, w, w)
\ + stackvertical( b, b, b, w, b, w, w, w)
\ + stackvertical( w, w, w, b, b, w, w, w)
\ + stackvertical( b, w, w, b, b, w, w, w)
\ + stackvertical( w, b, w, b, b, w, w, w)
\ + stackvertical( b, b, w, b, b, w, w, w)
\ + stackvertical( w, w, b, b, b, w, w, w)
\ + stackvertical( b, w, b, b, b, w, w, w)
\ + stackvertical( w, b, b, b, b, w, w, w)
\ + stackvertical( b, b, b, b, b, w, w, w)
\ + stackvertical( w, w, w, w, w, b, w, w)
\ + stackvertical( b, w, w, w, w, b, w, w)
\ + stackvertical( w, b, w, w, w, b, w, w)
\ + stackvertical( b, b, w, w, w, b, w, w)
\ + stackvertical( w, w, b, w, w, b, w, w)
\ + stackvertical( b, w, b, w, w, b, w, w)
\ + stackvertical( w, b, b, w, w, b, w, w)
\ + stackvertical( b, b, b, w, w, b, w, w)
\ + stackvertical( w, w, w, b, w, b, w, w)
\ + stackvertical( b, w, w, b, w, b, w, w)
\ + stackvertical( w, b, w, b, w, b, w, w)
\ + stackvertical( b, b, w, b, w, b, w, w)
\ + stackvertical( w, w, b, b, w, b, w, w)
\ + stackvertical( b, w, b, b, w, b, w, w)
\ + stackvertical( w, b, b, b, w, b, w, w)
\ + stackvertical( b, b, b, b, w, b, w, w)
\ + stackvertical( w, w, w, w, b, b, w, w)
\ + stackvertical( b, w, w, w, b, b, w, w)
\ + stackvertical( w, b, w, w, b, b, w, w)
\ + stackvertical( b, b, w, w, b, b, w, w)
\ + stackvertical( w, w, b, w, b, b, w, w)
\ + stackvertical( b, w, b, w, b, b, w, w)
\ + stackvertical( w, b, b, w, b, b, w, w)
\ + stackvertical( b, b, b, w, b, b, w, w)
\ + stackvertical( w, w, w, b, b, b, w, w)
\ + stackvertical( b, w, w, b, b, b, w, w)
\ + stackvertical( w, b, w, b, b, b, w, w)
\ + stackvertical( b, b, w, b, b, b, w, w)
\ + stackvertical( w, w, b, b, b, b, w, w)
\ + stackvertical( b, w, b, b, b, b, w, w)
\ + stackvertical( w, b, b, b, b, b, w, w)
\ + stackvertical( b, b, b, b, b, b, w, w)
\ + stackvertical( w, w, w, w, w, w, b, w)
\ + stackvertical( b, w, w, w, w, w, b, w)
\ + stackvertical( w, b, w, w, w, w, b, w)
\ + stackvertical( b, b, w, w, w, w, b, w)
\ + stackvertical( w, w, b, w, w, w, b, w)
\ + stackvertical( b, w, b, w, w, w, b, w)
\ + stackvertical( w, b, b, w, w, w, b, w)
\ + stackvertical( b, b, b, w, w, w, b, w)
\ + stackvertical( w, w, w, b, w, w, b, w)
\ + stackvertical( b, w, w, b, w, w, b, w)
\ + stackvertical( w, b, w, b, w, w, b, w)
\ + stackvertical( b, b, w, b, w, w, b, w)
\ + stackvertical( w, w, b, b, w, w, b, w)
\ + stackvertical( b, w, b, b, w, w, b, w)
\ + stackvertical( w, b, b, b, w, w, b, w)
\ + stackvertical( b, b, b, b, w, w, b, w)
\ + stackvertical( w, w, w, w, b, w, b, w)
\ + stackvertical( b, w, w, w, b, w, b, w)
\ + stackvertical( w, b, w, w, b, w, b, w)
\ + stackvertical( b, b, w, w, b, w, b, w)
\ + stackvertical( w, w, b, w, b, w, b, w)
\ + stackvertical( b, w, b, w, b, w, b, w)
\ + stackvertical( w, b, b, w, b, w, b, w)
\ + stackvertical( b, b, b, w, b, w, b, w)
\ + stackvertical( w, w, w, b, b, w, b, w)
\ + stackvertical( b, w, w, b, b, w, b, w)
\ + stackvertical( w, b, w, b, b, w, b, w)
\ + stackvertical( b, b, w, b, b, w, b, w)
\ + stackvertical( w, w, b, b, b, w, b, w)
\ + stackvertical( b, w, b, b, b, w, b, w)
\ + stackvertical( w, b, b, b, b, w, b, w)
\ + stackvertical( b, b, b, b, b, w, b, w)
\ + stackvertical( w, w, w, w, w, b, b, w)
\ + stackvertical( b, w, w, w, w, b, b, w)
\ + stackvertical( w, b, w, w, w, b, b, w)
\ + stackvertical( b, b, w, w, w, b, b, w)
\ + stackvertical( w, w, b, w, w, b, b, w)
\ + stackvertical( b, w, b, w, w, b, b, w)
\ + stackvertical( w, b, b, w, w, b, b, w)
\ + stackvertical( b, b, b, w, w, b, b, w)
\ + stackvertical( w, w, w, b, w, b, b, w)
\ + stackvertical( b, w, w, b, w, b, b, w)
\ + stackvertical( w, b, w, b, w, b, b, w)
\ + stackvertical( b, b, w, b, w, b, b, w)
\ + stackvertical( w, w, b, b, w, b, b, w)
\ + stackvertical( b, w, b, b, w, b, b, w)
\ + stackvertical( w, b, b, b, w, b, b, w)
\ + stackvertical( b, b, b, b, w, b, b, w)
\ + stackvertical( w, w, w, w, b, b, b, w)
\ + stackvertical( b, w, w, w, b, b, b, w)
\ + stackvertical( w, b, w, w, b, b, b, w)
\ + stackvertical( b, b, w, w, b, b, b, w)
\ + stackvertical( w, w, b, w, b, b, b, w)
\ + stackvertical( b, w, b, w, b, b, b, w)
\ + stackvertical( w, b, b, w, b, b, b, w)
\ + stackvertical( b, b, b, w, b, b, b, w)
\ + stackvertical( w, w, w, b, b, b, b, w)
\ + stackvertical( b, w, w, b, b, b, b, w)
\ + stackvertical( w, b, w, b, b, b, b, w)
\ + stackvertical( b, b, w, b, b, b, b, w)
\ + stackvertical( w, w, b, b, b, b, b, w)
\ + stackvertical( b, w, b, b, b, b, b, w)
\ + stackvertical( w, b, b, b, b, b, b, w)
\ + stackvertical( b, b, b, b, b, b, b, w)
\ + stackvertical( w, w, w, w, w, w, w, b)
\ + stackvertical( b, w, w, w, w, w, w, b)
\ + stackvertical( w, b, w, w, w, w, w, b)
\ + stackvertical( b, b, w, w, w, w, w, b)
\ + stackvertical( w, w, b, w, w, w, w, b)
\ + stackvertical( b, w, b, w, w, w, w, b)
\ + stackvertical( w, b, b, w, w, w, w, b)

changefps(assumefps(s, s.framecount / 2.788), 60)

[jagabo]

I know there is some type of compression that is based on not repeating pixels that are the same from one period to the next.

All high compression codecs do this.

With constant quantizer or CRF encooding content does make a difference. Big changes mean more bitrate is needed for those frames. More bitrate means more CPU power is required to decode. Two videos, both encoded at CRF 18:

function blinkbig(int len)
{
  ColorBars(height=1080 , width=1920).AssumeFPS(60).Trim(0,-1)
  \ + ColorBars(height=1080, width=1920).AssumeFPS(60).Trim(1,len-1).Amplify(0.0).RGBAdjust(r=0.0, g=0.0, b=0.0)
}

function blinksmall(int len)
{
  ColorBars(160,120).AddBorders(880,480,880,480).AssumeFPS(60).Trim(0,-1)
  \ + ColorBars(height=1080, width=1920).AssumeFPS(60).Trim(1,len-1).Amplify(0.0).RGBAdjust(r=0.0, g=0.0, b=0.0)
}

Per frame bitrate from bitrate viewer, big on top, small on bottom:

[poisondeathray]

My reasoning is that if what I am seeing is to do with my hardware and not the video content then I would probably see the same thing regardless of the encoding.

No, because different compression schemes and settings could be more taxing on the CPU.

For example, HEVC takes a lot more "juice" to decode than say MPEG2 . All I-frame encodes within h.264 with CAVLC will be easier to decode than very long GOP's with CABAC, etc...

Bitrate does affect CPU consumption , but CABAC, deblocking and resolution are higher up on the "totem pole" of CPU consumption hierarchy (in terms of h.264 decoding)

You said you "don't have decent hardware" , well that's where small changes can make the difference between smooth playback and stuttering. What is your system configuration?

You can upload clips directly to this site (there is an upload button) , or at the very least post more information about the encoding settings for each (mediaino => view text) . On one hand you say h.264 AME plays ok, on the other you say the xvid has problems. Generally xvid is less CPU intensive, so those results would be unexpected. Was it even the same test clip? Same resolution ? If one was 4k and the other 1080p , obviously that invalidates your testing. Test 1 parameter at a time. So be more specific and clear about your tests, and the settings used

[cheyrn]

The idea that different compression schemes can be more taxing implies that content compressed in different ways are taxing to different amounts, which means that the content affects playback.

It's sort of like how if a golf cart and a tractor are going across town and they encounter a hill. If the golf cart can't make it up the hill but the tractor can, it means that the hill impedes progress, even though the tractor can make it over the hill.

I attached files.

I noticed this comment from ffmpeg:

[mpeg4 @ 05600ac0] Video uses a non-standard and wasteful way to store B-frames
('packed B-frames'). Consider using a tool like VirtualDub or avidemux to fix it
.

The strobe files were made with the blink function in the first avs script posted above. Stacks is made with the second. The point was that it has common data between frames. black has all common data between frames. Dot (ignore that it's not a dot) has fewer pixels than the strobe that change. They are also aligned vertically.

So, the point is if the content is affecting playback, what about it and how I encoded it causes the problem.

Oh yeah, my hardware. I am using a laptop with 8GB RAM an AMD E-300 CPU and radeon hd 6310 display adapter and 5400 rpm drive

[poisondeathray]

In the very first reply (if you have decent hardware....) well that's exactly what I'm trying to say about your "golf cart" (or "goats" that you've filled in your computer description ), when you should be using a semi trailer .

1080p60 is easy to decode for modern hardware with videos using common compression schemes like xvid, h264. A few years ago it was almost impossible.

So what is your computer configuration ? If you're on the cusp , then small changes and optimising settings might enable you to playback smoothly without dropping frames . Or consider using a lower resolution like 720p60

[poisondeathray]

Oh yeah, my hardware. I am using a laptop with 8GB RAM an AMD E-300 CPU and radeon hd 6310 display adapter and 5400 rpm drive

Ok I missed this. This is underpowered. This is the "goat"... with 1 bad leg.

There is GPU accelerated playback for certain types of files . They have to be DXVA compliant. xvid doesn't fall under that category for AMD cards (unless they added it recently), so that might help to explain some of the observations . It is possible to get GPU acceleration with LAV filters with Nvidia cards (and maybe ffdshow added it too, not sure)

For pure CPU playback, xvid is almost always easier to decode than h.264

Can you list which ones playback fine for you? For example , do all the MP4's playback fine, yet all the AVI's playback poorly ?

[cheyrn]

So, poisondeathray and jagabo are confirming that the type of content affects playback.

All of the mp4 examples play as I would expect. What does it mean then, other than that you can overcome some kinds of problems by getting faster hardware.

I also have a desktop computer that has an i7 cpu, 32GB of RAM and two Nvidia GTX 570 graphics cards. It also exhibits the play back differences between the examples that I attached. Did you try playing strobe avi and mp4 files and didn't see a difference?

It would be useful to know what the problem is exactly that the h264 encoding avoids in these examples.

[cheyrn]

So, poisondeathray and jagabo are confirming that the type of content affects playback.

All of the mp4 examples play as I would expect. What does it mean then, other than that you can overcome some kinds of problems by getting faster hardware.

I also have a desktop computer that has an i7 cpu, 32GB of RAM and two Nvidia GTX 570 graphics cards. It also exhibits the play back differences between the examples that I attached. Did you try playing strobe avi and mp4 files and didn't see a difference?

It would be useful to know what the problem is exactly that the h264 encoding avoids in these examples.

[poisondeathray]

These observations don't really demonstrate "content affecting playback", or maybe it's the interpretation of what you mean by that phrase.

Content does affect things like frametype allocation, bitrate allocation, - and those can affect playback to a mild extent but it's usually not an issue for playback on a decent system. If you can decode a 30- 40Mb/s bu-ray streams without an issue, surely can decode something 100x less - so bitrates in the typical range are rarely a contributing factor unless you have a borderline system.

So the difference here appears to be attributed to xvid/avi/pcm vs. h.264/mp4/aac, since the same content plays fine when encoded as h.264/mp4 . If content affected playback, then you would expect that within the h.264 encodes, there would be playback differences . So I think your "problem" is more specifically about why can't you play the AVI videos as expected in this scenario.

Other factors and differences that might be contributing : container differences, muxing differences, compressed audio vs uncompressed audio. It's not very scientific when a bunch of different things are changed, because you don't know what is affecting what. You have to change 1 thing at a time. There will be muxing AV delay differences and audio padding with compressed audio. You can see these differences when you zoom in with audacity for example (10ms difference, but it would be barely audible to human). (This shouldn't affect it but you also converted mono to stereo). You might "stream copy" the mono pcm wav into mkv container instead if you wanted to test the affect of video codec, so that the only variable that is different would be video codec, instead of a dozen other things

If you really wanted to test "content affecting playback" , you have to control all these other variables that might be confounding

If you wanted to rule CPU "horsepower" issues, you can do your tests at a lower resolution

[cheyrn]

I think jagabo's example demonstrates unambiguously that there are different performance requirements depending on content. Among other things, this means that a playback problem doesn't necessarily mean that there is something wrong with the video. It could be that it also implies that you can address some potential playback problems that are based on characteristics of the content by choosing an appropriate encoding.

The extent of my knowledge at this point is that which characteristics of content have what playback performance differences is not necessary to know if all you care about is that a computer is fast enough to make the differences unapparent. I think I also know that if I use libx264 to encode these particular videos the differences are unapparent to me right now on a slow laptop and a faster than average desktop computer, but they are apparent on both using some xvid profile and settings.

If all I cared about was "how do I make it go", I would not be reading this forum.

I don't have to use the blinking overlay video. But, other clips I am working with playback at different speeds in some situations that I haven't sorted out. I address the problem by randomly trying things while also asking questions like those in this thread. It is impractical for me to continue that way and so I am pursuing and will continue to refine my understanding of what the specifics are that relate to these problems. This thread is a case for study, I think.