Does a middle encode harm the quality of the end result?

[PuzZLeR]

A bit of a theoretical question that I'm trying to determine myself, so hopefully it's interesting to the forum.

Given a source S, let's say you encode it to a target clip T with a lossy format (MPEG-2, Xvid, DivX, H.264, etc) given a certain bitrate b.

Scheme 1:
S --> T, with T encoded at b.

Now let's introduce another middle format M, also lossy, and encoded with a higher bitrate B, where B>b.

Scheme 2:
S--> M --> T, with M encoded at B, and T encoded at b.

Let's assume all settings are the same for the encoder(s). And to keep this simple, let's assume all codecs used have a similar compression efficiency or the middle clip has a justifiably high enough bitrate from the target.

Now my question: Is Scheme 2 necessary a more lossier encode than Scheme 1? If so, would it be by much? After all, the end result is with the same settings and same bitrate. I know there would be several factors involved among different encoders and codecs, etc, but let's just keep it hypothetical for the moment.

What do you think?

Thanks for reading.

[PuzZLeR]

Ok, in case this is over anyone’s head, here’s an example with the Xvid settings the same in both encodes.

Scheme 1:

DV encoded to Xvid with a bitrate of 5000kbps.

Scheme 2:

DV encoded to Xvid with a bitrate of 7500kbps, then this in turn gets encoded to Xvid with a bitrate of 5000kbps.

I could introduce different codecs for the mid format, but that would complicate things with the parity in efficiency so I kept it simple.

Given that the end result is the same target format would the final result be that much more lossier with Scheme 2?

Thanks again.

Note to Mods: I've just realized this may be in the wrong section of the forum. Please move it if need be.

[jagabo]

Each time you decompress a video compressed with a lossy codec, and recompress it with a lossy codec, you will lose some quality.

[PuzZLeR]

Hey Jagabo. Thanks for answering.

Of course you will lose (at least some) quality. But do you, or anyone else, think that you will lose (alot) more with a middle format given that the target is the same?

I guess I'm not following what you mean by "decompress".

[jagabo]

You can't compress a second time without decompressing the first decompression.

Uncompressed video -> (xvid) compessed video -> uncompressed video -> (xvid) compressed video.

If you use sufficient bitrate (or a low enough quantizer level) the losses may not be very visible. I find Xvid to do reasonably well through multiple generations. You'll have to decide what's acceptable for yourself. If you really want to be careful use lossless compression (HuffYUV, Lagarith) for the intermediate file.

If you're just doing cut and paste editing use VirtualDub in Smart Rendering mode. It will only recompress those frames that need it (cut GOPs). The rest of the video will pass through without being recompressed.

[PuzZLeR]

Ah Ok. Didn't realize a video stream goes through a decompression during the encoding to a (further) compressed format, much like audio being decompressed to wave before being recompressed.

I'm hoping the losses aren't too visible, and it's more than just Xvid I'm working with. And I know a lossless middle format would be for best results but it wouldn't work for what I want to do.

The main reason I asked this question is that I have alot of DV footage, that would otherwise need too many TBs to store (as archived source), that I'd like to compress to another smaller "source". I'm not talking the "precious" stuff, just the trivial not-so-important stuff that can pile up.

I'd like to have this "source" available to keep my options open - to encode to several formats as I need them: DvD, iPod, DivX/Xvid, whatever, etc.

To keep the "source" manageable, I encode to a high bitrate MPEG-2 (~8000kbps) and audio to FLAC, which then becomes the new "source" after I then delete the DV and save tons of space (with only a tiny hit in quality, which I accept).

I was just wondering how much quality I'm losing when using this MPEG-2 as "source" instead of the actual original DV since the target formats are even lossier anyway.

DV --> target format

Vs.

DV --> high bitrate MPEG-2 --> target format

(The target format also includes the DvD choice, with MPEG-2 being less than 8000kpbs.)

In other words, does the intermediate encode (MPEG-2~8000kbps) make the target (much) more lossy?

My eyes have failed to see much difference so far, but I'm still wondering...

[guns1inger]

One of the issues I have with Xvid and Divx, even at high bitrates, is that they cause flattening of the colours in low contrast areas. This causes blocking artifacts that are visible, and aren't restricted to dark areas.

DV generational loss if far less than mpeg-2 or mpeg-4 compression, and with the right DV codec (Sony or Mainconcept, for example) you can go up to 4 or 5 generations before visual loss becomes an issue.

For DV footage, especially anything shot handheld or in low light, I would not consider 8000 kbps Mpeg-2 to be "high bitrate". Perhaps 12 - 15 mbps would be closer.

My solution is to keep the tapes and get it back from there when and if necessary

[vhelp]

Situation A -- picture pixelation

First, ruling out pixelation because that is the most obvious in terms of quantifying it as lossy.
For me, anyting sign of pixelation and right away I determine it, lossy.

Situation B -- color space conversion and decoding/encoding

If were talking about a undistorted image (above) then the next ruling would be how much with regards to how much Pic A differs from Pic B. What quantifies that is how much of the rounding errors are produced in the conversion(s) processes, and/or compression errors. This also includes the decoding aspects, then you also have to consider the "decoder" used in such cases because not every single decoder are alike--they have varying generation algorithm-mic optimization aspects and so on.

I say processes because several things are going on: decoding->rounding errors, color space conversion->rounding errors, and so on. Then, re-encoding again: color space conversion->rounding errors, conversion to format->rounding errors. There are other things also, like DCT and Quantizing etc.etc., each used in each decoding/encoding passes.

The other errors come in the form of the compression aspects, DCT; gibblits; floating x/x blocks, and so on and so forth. The order of error rate would prob be something like, decodeing; color space conversion; pixelation; rounding errors; compression errors/artifacts, etc.etc. not that there is any particular order, but a general idea there of.

Personally though, I've since changed my opinion on the lossy versus lossless scenarios because when we first review a video we almost imediately call a video lossy at the first sign of pixelation. The other levels of lossy would be those with artifacts that look similar to pixelation but are compression related, such as DCT errors and too low a bitrate, etc.etc., plus other decoding/encoding aspects not mentioned here.

But, just because you may not see any pixelation or even compression related errors (above) with regards to rounding errors (in all levels) even this can be considered lossless provided that the end result is accurate enough reproduction of the target video (detail) left through the transfer or conversion proceses. You could give a video (not all videos are alike) a guage or percentage as to how much you are willing to consider a video lossless before finally calling it lossy. True, any video that is not an exact (pixel-for-pixel) representation of the original source is not lossless--it is lossy. But a few pixels from frame to frame (who's gonna really notice this) is hardly lossy to the eyes. No one sits in front of a tv and analyizes a pictures as it is playing 30 frames per second and can spot flaws or lossyness.

But I think that puzzler was looking for something along those lines of "how much or when should one say" is lossless or lossy given the above points..a few rounding errors at best though who would really notice this as lossy. As long as the picture is still maintained to a certain level that your mind can dicern as lossless video, that is all the really matters, I think.

-vhelp 5159

[jagabo]

I was curious about some things so I ran some experiments. I ran some encodings through several generations of Xvid with VirtualDub. I started with some daylight 29.97 fps progressive DV footage (YUY2 from Cedocida), resized it to 320x240, made a darker version (to examine dark area problems), stacked them side by side, and saved with Lagarith (lossless YUY2) compression. This gave me a very clean starting video with both light and dark areas.

After five successive Xvid encodes at target quantizer 3 the video was still very good -- as long as the Xvid decoder's deblocking filter was not enabled, and VirtualDub was in Fast Recompress mode (to avoid conversion to RGB in VirtualDub). By successive Xvid encodes I mean I opened the Lagarith file then compressed with Xvid. Opened that Xvid AVI and compressed it again with Xvid. Etc.

Here's a sample frame from the original Lagarith file:



The first generation of Xvid compression (no deblocking, fast recompress mode) looks pretty good except for posterization in the dark areas (as expected):



The fifth generation doesn't look too much worse (a little less saturated):



With Xvid's deblocking option enabled for all generations the 5th generation showed a lot of detail loss and blurring in dark areas:



With VirtualDub in Full Processing mode (decompression to RGB) and without deblocking there were noticeable color shifts in both the light and dark areas:



I recommend you download these images and examine them with a 4x nearest neighbor enlargment. Here's the lower left corner of four of them enlarged 4x:



All these png files were made by copying frames from VirtualDub. So they all have the usual rec.601 contrast expansion. The original DV file had some blown out bright areas (the dog's foot in the lower left, for example) -- common in DV camcorders.

Because some artifacts are only seen when viewing the animations, here are some of the AVI files:

generation 1: gen1.avi
geneartion 5 with deblocking: deblock5.avi
generation 5 with RGB conversion: rgb5.avi

Some take-home lessons:

1) Even the first generation of Xvid encoding will create some posterization artifacts in dark areas.

2) Successive conversions between YUV and RGB (with Xvid compression in between) will have a very negative effect on your video. Do all your filtering in YUV (AviSynth) when possible.

3) Deblocking should be used sparingly and not through several generations of Xvid (and probably other high compression codecs) encoding.