HCenc: VBV at critical level, VBV underflow.

[transcendent]

Hi there, so I've been re-encoding a DVD .m2v demuxed stream, avisynth filtered and all.
I have come across the problem as in the title in a very specific part of the film.

The scene is very simple essentially black screen with some white text, then some flashes of orange so this should be fairly easy to encode, but I'm guessing due to this, the VBV buffer is getting very low. HCenc will just die on me at this point. I've tried things like raising average bit rate, increasing minimum bit rate, increasing bit rate for selected section of the frames in that section, but i seem to get the buffer underflow (almost) regardless. I found that lowering the Adaptive Quantiser to zero seems to fix this problem. Is this a valid fix? Or is there something else I should be changing instead.

If need be I can encode the entire film in sections to the chapter breakpoints with closed GOPS and then re-encode this specific section with different settings.

I've attached a screenshot of settings. I can make a sample video on request but it will take me a little time.





I'd also like to state that I'm going for highest possible quality encode. Any recommendations would be welcome regarding settings, bit rate etc. For example, the original video averaged about 8250 and had 1.5kbps DTS stream+ small AC3 stream. So I've kept inline with this. However, calculators seem to say 9.8 - 1.5-0.16 - overhead should be around 7800kb/s. Am I pushing my luck?

I chose FOX2 matrix on a pretty poorly informed intuitive approach but from the lower numbers in the matrices, I was assuming some sections in the film which lack contrast and look pretty washed out, would benefit from it. There isn't all that much motion though, it's only frame by frame drawn anime. So I think maybe the matrix (from action shows i believe) is not 100% optimal. Just doin what I can for best possible image quality while remaining under 9.8mbps.
Capacity is a DVD-9 so I have plenty of space for a 86minute video.

[aedipuss]

first thing i'd try is updating to the latest version of hcenc and seeing how that goes.

[jman98]

If you truly have a DVD source for this, why are you re-encoding it at all? Since you intend to burn it to a DVD-9, there is no reason to re-encode. You may want to post your AviSynth script as you may be doing something there that is the real reason for your problems.

[transcendent]

If you truly have a DVD source for this, why are you re-encoding it at all? Since you intend to burn it to a DVD-9, there is no reason to re-encode. You may want to post your AviSynth script as you may be doing something there that is the real reason for your problems.

Best video is remastered which only came in italian/japanese. I have english subs/english commentary on another DVD. These all need to be remuxed to make one final copy. Even the remastered version I thought could do with some filtering on noise, some added sharpness etc.

If you're interested: Note, i cobbled this together, I'm pretty sure I end up doing a twice super sample in some of the triple frame processing or something horribly inefficient. But I'm not too bothered at the moment.


LoadPlugin("C:\Users\Samantha Carter\Desktop\dgmpgdec158_SSE\dgmpgdec158\DGDecode.dll")
MPEG2Source("filtertestclip.d2v")

AssumeTFF()
Telecide(guide=0,post=0)
super = MSuper(pel=2, sharp=1,rfilter=4)
backward_vec2 = MAnalyse(super, isb = true, delta = 2, overlap=4)
backward_vec1 = MAnalyse(super, isb = true, delta = 1, overlap=4)
forward_vec1 = MAnalyse(super, isb = false, delta = 1, overlap=4)
forward_vec2 = MAnalyse(super, isb = false, delta = 2, overlap=4)
MDegrain2(super, backward_vec1,forward_vec1,backward_vec2,forward_v ec2,thSAD=325)

Deen(mode="a2d", rad=3, thrY=7, thrUV=8, min=0.15)

LanczosResize(2160,1728)
#warpsharp(40,150)
xsharpen(200,15)
LanczosResize(720,576)

[manono]

Any recommendations would be welcome regarding settings, bit rate etc. For example, the original video averaged about 8250 and had 1.5kbps DTS stream+ small AC3 stream. So I've kept inline with this. However, calculators seem to say 9.8 - 1.5-0.16 - overhead should be around 7800kb/s.

I'd say your average bitrate is too close to the max bitrate, and that's why you're getting underflows. But as aedipuss suggested, you should probably upgrade to the newer 0.26 version.

However, even if it did encode successfully, it would never author because your 8600 max bitrate plus the full bitrate DTS track, plus the AC3 audio, plus the subs, plus the muxing overhead, will go well over the 10080 limit.

In my opinion, DTS is a waste of space.

[transcendent]

Any recommendations would be welcome regarding settings, bit rate etc. For example, the original video averaged about 8250 and had 1.5kbps DTS stream+ small AC3 stream. So I've kept inline with this. However, calculators seem to say 9.8 - 1.5-0.16 - overhead should be around 7800kb/s.

I'd say your average bitrate is too close to the max bitrate, and that's why you're getting underflows. But as aedipuss suggested, you should probably upgrade to the newer 0.26 version.

However, even if it did encode successfully, it would never author because your 8600 max bitrate plus the full bitrate DTS track, plus the AC3 audio, plus the subs, plus the muxing overhead, will go well over the 10080 limit.

In my opinion, DTS is a waste of space.

Hi again manono, and I'd like to say thank you for all the help recently

However, conceptually why does the buffer UNDERflow when bit rate is high? Have I got it the wrong way round thinking that too high a bit rate would cause a buffer to OVERflow i.e. the buffer cannot deal with the amount of information coming in?

And regarding bit rate, say all streams excluding video come to 2mbps. And for simplicity, lets call the max 10mbps combined. Does that mean my MAX bit rate should never ever exceed 8mbps? Or is there some leeway from buffering and also if the audio-streams are variable bit rate, won't my re-encodes (which have smoothing/denoisers) follow a similar pattern of bit rate rise and falls of the original source therefore not exceed 10mbps combined as long as I keep the same average/max of the original source (I know that changing quantisation matrices will invalidate this somewhat as well, but I'm interested in the theory as well). Can their ability to encode all the audio streams + average 8.25mbps video be explained ..because they encoded both the audio and video streams at the sametime and made sure the total bit rate never exceeded 10mbps?

[transcendent]

So I've tried 0.26 beta. As well as arbitrarily setting the bit rate to low to very low values. I still get the buffer underflow. Infact, it reaches critical even sooner when I set bit rate to 2000kb/s


edit: If I push the bit rate to 10mbps, it manages to fix the underflow and carry on...

[Cornucopia]

First, note that with the DVD-Video spec, NO types of audio streams or subtitle streams are allowed to be Variable bitrate. ALL are CBR. This makes the calculations easier. So, yes, if (for example) your audio+subs assets = 2Mbps and your Total Max of all assets <= 10Mbps, then your Total Max of your Video stream must be <=8Mbps!. No ifs, ands or buts.

VBV is NOT just about bitrate, it's also about how those bits are packetized and what bits were available to the encoder at encode time. If you're getting an underflow, it's probably not just that you have a bitrate set too high or too low, it's that you have it set too high (or low) compared to ANOTHER setting that would normally force that bitrate to be a different setting. Maybe GOP length, or MIN setting or your MATRIX.

If you're encoding with hcEnc, are you using a default settings profile? or are you starting with a default profile and customizing those settings? If so, try NOT customizing (except the bitrate).

BTW, for DVD-legal compatibility, your highest bitrate for your video stream asset is 9.8Mbps, and it really shouldn't be this high as many players have fits with it that high. Use 9.0Mbps as your limit. Don't use ANY minimum (let it decide automatically). If your average is within ~500kbps from your peak/max, you probably should just be using CBR (yeah, I know hcEnc doesn't do CBR - use a different encoder).

Scott

[manono]

One definition of a buffer underflow (or underrun) I found is:

These errors usually indicate that your elementary MPEG video stream is not being compressed enough in sections of the video to meet the stream’s elementary MPEG video bit-rate.

Apparently it can't keep to the max 8600, and goes over. Again, that could point to the average bitrate being too close to the max. Do you have CCE or another encoder you can try?

And regarding bit rate, say all streams excluding video come to 2mbps. And for simplicity, lets call the max 10mbps combined. Does that mean my MAX bit rate should never ever exceed 8mbps?

Yes, that's what it means, assuming the encoder never goes over the max set. TMPGEnc goes over sometimes, but I thought HCEnc did keep to the max. Anyway, it's often a good idea to keep it somewhat below the max, not only for underflow reasons, but also because (I've been told) some players might choke on streams too close to the max for too long of a period, something that might result in stuttering playback or short pauses or freezes.

...and also if the audio-streams are variable bit rate...

Which they aren't, if for DVD.

Can their ability to encode all the audio streams + average 8.25mbps video be explained ..because they encoded both the audio and video streams at the sametime and made sure the total bit rate never exceeded 10mbps?

How did you determine the average bitrate of this video? Because, if it really has a full bitrate DTS track, some sort of AC3 audio (192, maybe?), with subs, then there's no way in hell this commercial release could ever be authored with a max bitrate of 8600 or thereabouts. The max has to be somewhat less than what you're using.

There are ways to get an accurate reading of the average bitrate. Since you use DGIndex, you could open the VOBs, make sure it's set to play back at the fastest possible rate, and then run the Preview. It might take a while, though. I think GSpot can also parse the VOBs and give an accurate reading of the average bitrate.

Can their ability to encode all the audio streams + average 8.25mbps video be explained ..because they encoded both the audio and video streams at the sametime and made sure the total bit rate never exceeded 10mbps?

They're not using HCEnc, that's for sure. Not that HCEnc isn't a very good encoder. But I guess it chokes on your settings.

Also, you're just encoding a small portion of a video, only 302 frames? You might have better luck encoding a much longer video. In fact, if you're opening that slow script in HCenc, I think I'd make a lossless AVI from it in VDub before then feeding that into HCEnc via an AviSynth script and AVISource.

[transcendent]

Cornucopia:If you're encoding with hcEnc, are you using a default settings profile? or are you starting with a default profile and customizing those settings? If so, try NOT customizing (except the bitrate).

Thanks, this piece of advice did the trick. Obviously I was attempting to optimise my output quality. But it actually seems the main problem was the Scene detection options. If I have it disabled, I have to raise my bit rate (at least the max bit rate) to levels which are outside DVD spec/feasibility. To be honest, I doubt this is how it's supposed to function, closing the GOP on a detected scene change (except for some freaky marginal case) shouldn't suddenly fix all the issues. But there we have it.

Thanks both you can manono for correcting me on the CBR nature of audio streams on DVD.

How did you determine the average bitrate of this video? Because, if it really has a full bitrate DTS track, some sort of AC3 audio (192, maybe?), with subs, then there's no way in hell this commercial release could ever be authored with a max bitrate of 8600 or thereabouts. The max has to be somewhat less than what you're using.

I ran the first VOB of the main movie from the actual DVD through DGindex. Here's what is muxed for that VOB:
Ending bit rate: 8.420Mbps
Average bit rate: 8.250Mbps
Max bit rate: 9.122Mbps

Audio:
AC3 3/2 384
AC3 3/2 384
DTS Audio

I guess I must be wrong about assuming it's full bit rate DTS. I know the japanese version is in full DTS and then it has a stereo stream in japanese as well of what I assume is of a fairly low bit rate. So this Italian version has 2 x 384 + maybe a 700 odd bitrate DTS giving a total of 1.5mbps audio. 9.8 - 1.5 gives 8.3. Average bit rate is 8.250. Then How is it allowable to have max bit rate spikes over Mbps on the video stream?

Also, you're just encoding a small portion of a video, only 302 frames? You might have better luck encoding a much longer video. In fact, if you're opening that slow script in HCenc, I think I'd make a lossless AVI from it in VDub before then feeding that into HCEnc via an AviSynth script and AVISource.

I picked out the small number of frames because it's not fun testing with an already cut down section which takes over an hour and a half to encode at 4fps...(my previous script used to do it at 10fps...but I guess the super sampling killed it)
I picked the 302 around the problem area- giving it enough frames beforehand to "calibrate" the VBV buffer to go into the problem area correctly to reproduce the problem. I may have an "unofficial" look into some other encoders but I'm fairly happy so far with HCenc though I will have a look at converting to uncompressed AVI. Hopefully 250gb of HD space(all I have left) will be enough.

So final setting wise I calculate my hardlimit to be 9.8 - 1.5 - 0.16 = 8.14Mbps
An average of 8 or 8.05 is inline with the kind of headroom the italian release gave. But the max spike bit rate still confuses me slightly. It seems I can push it up to about 9 and intuitively I assume the encoder must keep track of the VBV to buffer these high bit rate spikes during lower activity previous frames i.e. giving the maximum possible quality. Obviously this is a posteriori information so it comes from the multiple passes required to encode it. This is the only way I can imagine the DVD spec being followed.

Anyway, this has been most enlightening. I'd have liked maybe to be in the thick of this discovery some years back while it was stil the most relevant media. It's mostly moving onto H.264 and blu-ray. But hopefully the knowledge gained here is transferrable, at least in concepts and basics into the newer formats. I can't imagine AVI synth gets used quite as much for blu-ray though - much less requirements for post processing etc. Could be wrong ^^

thanks again ^^