Archiving Footage with de-bloated files

[JDługosz]

I have a Canon 60D, which I've started using to shoot video. However, the files it writes appear to be quite bloated because it doesn't use any motion analysis or advanced features — essentially it just compresses each frame, as they are all marked as “I-FRM” and the data rate is 44 to 48 Megabits/sec.

I posted a message on avidemux which includes details given by MediaInfo.

I'm thinking that it should be possible to re-save using other frame types and motion analysis, but tuning the quality settings so as not to make it any lossier than it was already, while reducing the file size substantially.

Does anybody know how to do this, or in general how to gleam the necessary information from an original file about what the quantization settings were?

Any advice on which tools I should be using (on Windows) is good too. My initial use has been using avidemux, which is easy enough to trim losslessly, and allows most everything to be specified for the underlying x264 library. My copy of Adobe Premere 1.5 is quite out of date and useless for modern files.

Thanks,
—John

[poisondeathray]

I'm thinking that it should be possible to re-save using other frame types and motion analysis, but tuning the quality settings so as not to make it any lossier than it was already, while reducing the file size substantially.

That's not how lossy compression works

You no longer have the original sensor data from the camera.

Certainly x264 is a much better encoder than the hardware encoder used by Canon and you can reduce the filesize, but there will be some quality loss , unless you use the lossless mode (which will make the filesize many times larger than the original)

[JDługosz]

That's not how lossy compression works

I know that with a still (JPEG) file, if it is loaded and re-saved using the same Q-matrix as before, then it will be changed only by a very tiny amount, if any. Each block is decoded with loss from the true original, but re-encoding will quantize the same as before, and in fact floating point rounding errors will often be erased by the quantization and I get the exact same encoded data as before, for each block that was not changed.

Why is that not essentially true for AVC as well?

[poisondeathray]

No, lossy means you lose quality . Period . Each generation you throw away data. If you re-encode jpeg even at 100% quality you incur loss. Do it 100x and it will look like mush

There is a point on the compression curve using AVC (using x264 encoder), where it becomes lossless. At that point, it will be several times larger than the original. You can choose some point in between

You can get "visually lossless" (you can't see the difference under normal viewing conditions) with these Canon DSLR files with significant compression (roughly ~1/2 to 2/3 the size) - the reason is they are poorly compressed files to begin with, and x264 is a very good encoder . Eitherway you lose quality, unless you use a quantizer of 0 (lossless) .

In video compression, you have to decode before you re-encode. Thus you're starting with an uncompressed version. Think of it analgous to a TIFF image (except it's in YUV).

[Cornucopia]

Also, there is lossy quality that looks ok as final output to the display & eyes, and there is lossy that is still workable for edit/post. They are NOT the same. If you choose as priority the former for size savings, you will give greater problems to the latter. Then when it gets to its final stage, it WILL be noticeable.

edit: Think of it this way...
What is the natural state of a sponge? Full of water? Only a very small amount of water? or No water and squeezed real tight?

Camera outputs come "full of water". You can edit that way, or maybe squeeze out some of the water (during editing/post), but if you squeeze out too much water prior to distribution, your product will look more like a balled-up thing than a normal sponge.
Sure, it is much better to have small things to DISTRIBUTE, but that's not how you want to WORK with them.

Scott

[JDługosz]

No, lossy means you lose quality . Period . Each generation you throw away data. If you re-encode jpeg even at 100% quality you incur loss. Do it 100x and it will look like mush ⋯
In video compression, you have to decode before you re-encode. Thus you're starting with an uncompressed version. Think of it analgous [sic] to a TIFF image (except it's in YUV).

Hmm, either you don't understand what I'm saying, you don't believe my (factual and verifiable) JPEG assertions, or you are saying that it's not the same for AVC without explaining why it would be different?

Yes, you decode. You re-encode, with the SAME q-matrix.

A (original) saves to B (lossy). B decodes to A2, which is in general not the same as A. There is a large number of values Ax which encode to the same B. Encoding any of those gives B. Decoding B gives A2 in particular. Repeat any number of times and it sticks with A2<-->B. That is if the same settings are used (same color transform, subsamping, and Q-matrix).

I did not ask for some way to reduce the file size without altering the content at all. I mean there should be recommended settings that give vanishingly small change to the existing (lossy) data, by doing things like color space transforms the same way, using the same block boundaries, and using the same quantization on those blocks.

—John

[poisondeathray]

I did not ask for some way to reduce the file size without altering the content at all. I mean there should be recommended settings that give vanishingly small change to the existing (lossy) data, by doing things like color space transforms the same way, using the same block boundaries, and using the same quantization on those blocks.

Yes you can reduce the size with minimal quality loss . This is what "visually lossless" means . Under normal playing conditions you won't be able to see the difference. If you go frame by frame and zoom in - you will see the difference . You can choose any point in between from crap quality to lossless. At the lossless end, the filesize will be larger than the original

You asked for something different, not to make it any "lossier" while reducing the filesize - that's not possible

I'm thinking that it should be possible to re-save using other frame types and motion analysis, but tuning the quality settings so as not to make it any lossier than it was already, while reducing the file size substantially.

Cheers

[Cornucopia]

No, lossy means you lose quality . Period . Each generation you throw away data. If you re-encode jpeg even at 100% quality you incur loss. Do it 100x and it will look like mush ⋯
In video compression, you have to decode before you re-encode. Thus you're starting with an uncompressed version. Think of it analgous [sic] to a TIFF image (except it's in YUV).

Hmm, either you don't understand what I'm saying, you don't believe my (factual and verifiable) JPEG assertions, or you are saying that it's not the same for AVC without explaining why it would be different?

Yes, you decode. You re-encode, with the SAME q-matrix.

A (original) saves to B (lossy). B decodes to A2, which is in general not the same as A. There is a large number of values Ax which encode to the same B. Encoding any of those gives B. Decoding B gives A2 in particular. Repeat any number of times and it sticks with A2<-->B. That is if the same settings are used (same color transform, subsamping, and Q-matrix).

I did not ask for some way to reduce the file size without altering the content at all. I mean there should be recommended settings that give vanishingly small change to the existing (lossy) data, by doing things like color space transforms the same way, using the same block boundaries, and using the same quantization on those blocks.

—John

If you start with 100% quality and encode using 90% quality-efficiency, you are left with 90% quality as your "A2" decode.
If you start with 90% quality and encode using 97% quality-efficiency (because of re-quantization & matrix efficiencies), you are now left with 87.3% as your "A3" decode.
Even if your quality-efficiency is 99% next time, you'll be left with 86.42% as your "A4" decode.

Doesn't matter which way you roll it, you are still LOSING something. How much does depend on tweaking, yes, but you are talking about your ORIGINAL MASTERS here, why skimp when storage is CHEAP? I certainly want my masters to be at 95-100% of the possible quality, not starting off from the beginning at 87% or less like you seem to be advocating, just for the sake of saving space.

Scott

[2Bdecided]

That's not how lossy compression works

I know that with a still (JPEG) file, if it is loaded and re-saved using the same Q-matrix as before, then it will be changed only by a very tiny amount, if any.

Encoders don't work like that. They could, but they don't. They assume the image/video is uncompressed (or unknown provenance), and try to do the best they can with it. They don't assume it's compressed, try to figure out the compression, and reverse the decompression to get back the original compressed file.

That wouldn't be an encoder, it would be a reverse decoder. Apart from a few university/research projects, these don't exist.

Transcoders that do not fully decode, and/or use elements of the original encoding in the new output, do exist - but it's usually done this way for efficient implementation rather than maximum quality.

If you do find someone has done this, please let us know. I've always thought that a near-lossless DV-AVI > MPEG-2 I-frame only transcoder would be useful, but as far as I know no-one has ever done it. You want to improve the efficiency by allowing B-frames. I don't know what efficiency gains you could get; I know that when I transcode 25Mbps I-frame only DV-AVI to 9Mbps IPB-frame MPEG-2 for DVD, the results can sometimes be nowhere near visually lossless, never mind mathematically lossless.

Cheers,
David.

[Cornucopia]

I've seen these in the lab at places like NCSA, but these aren't ready-for-prime-time. They aren't optimized well, just Proof-of-Concept kinds of things.

BTW, DV vs. MPEG2-I has other things involved which would probably preclude a simple conversion (not least of which is the color subsampling differences).

Scott