old vs new technology - which is actually better?

[zieske]

I am in the market for a consumer level camcorder. I have been researching various models and technologies for weeks. I have gone through spec sheets on all of the current models of all of the major players (and some of the minors).
I am a very big fan of canon digital cameras. I have owned many and they always produce superior quality images. Naturally I gravitated to canon camcorders assuming the quality would be in the same range. From everything I read, top image quality is between canon and sony. good enough.
I am obviously looking at an HD camcorder. It is not a fad - got it.
The avchd format used by (it appears) all of the new model hd camcorders leads me to part of my question here. My understanding is that avchd is akin to mpeg4 compression. In the digital world, all compression is bad as it relates to an algorithm used to throw data away to make the file smaller. Just like having the ability to pull "raw" images off my canon digital camera, if only there was a way to capture "raw" video for editing from a camcorder, that would be the golden cow right there, right? not going to happen at a consumer level - got it.
I have read several opinions stating that HDV to tape actually produces superior quality results. The basis of this argument is that the HDV format is akin to mpeg2 compression.
so part 1 of my question is: is mpeg2 better because it compresses the original data less than mpeg4? Or is mpeg4 so much better a codec that the loss experienced in the compression does not produce any worse results than the older mpeg2 format?
What got me on this track was finding the JVC GZ-HD5. To look at the specs of this thing, you would think they really had this figured out. All the ratings I have read unfortunately tell a different story.
The avchd spec claims to have a top bit rate of 24mbps. There is only one brand on the market that meets those specs; canon. And from what I read, they are performing a little voodoo to get to that spec. The obvious benefit of recording at the max possible bit rate is a (technically) more detailed image. Like I always say about digital cameras with multiple resolution settings.. why would you ever think that the picture you are taking is not worth capturing at the max resolution. If you don’t need that detail, throw it away later when you are editing. You can’t make up the data later (without interpolation).
Looking at the JVC, it claims to record 1080/60p at 30mbps max (26mbps average). Holy crap, there are tons of prosumer cameras that can’t meet those specs. The apparent ability for this camera to perform this video miracle is its recording format of mpeg2-ts. If you look at the newer JVC models which have switched over to the avchd (almost a) standard, their cameras drop down to 1080/60i at 17mbps like all the rest.
So part 2 of my long winded question is: is avchd really better? I can apparently do anything under the sun with the video in editing. There are tons of apps/plugins that will let me convert to 60p or 30p or 24p and apply any number of effects to the video. What they can’t do is create more data in the stream (without interpolation). And as you know, interpolation does not create detail, it takes the cruddy detail you have and makes it bigger.
I would love to hear other opinions on this. I do not have experience with these devices. I am only going by the info I have been able to research. Who knows, just like science, half of what I think I know may be wrong..

Thanks

[AlanHK]

so part 1 of my question is: is mpeg2 better because it compresses the original data less than mpeg4? Or is mpeg4 so much better a codec that the loss experienced in the compression does not produce any worse results than the older mpeg2 format?

I can't help with the hardware, but on this, MPEG4 is more efficient, it encodes the same quality in a smaller file.
With either MPEG2 or 4, you can choose a compression/quality trade off. So what you see on screen can be the same.
As you can get a PNG image identical to a BMP in 1% of the size, because it's a smarter format.

A more efficent encoding probably requires more intensive computation to encode or decode, but with Moore's Law this is not a problem.

[2Bdecided]

MPEG-4 AVC is (potentially) dramatically more efficient than MPEG-2, but realising that advantage in real-time on a consumer device isn't possible (yet).

So forget about the specs, and how it does it, and just look at the actual picture quality each particular camera can deliver.

Cheers,
David.

P.S. If a camera recorded 1920x1080p60 at 26Mbps, then all other things being equal, you'd get about as many compression artefacts as you'd get recording 1920x1080i60 at about 18Mbps, or 1440x1080i60 at about 14Mbps. In comparison, HDV records 1440x1080i60 at 25Mbps. So, all other things being equal, 1920x1080p60 26Mbps MPEG-2 will have more visible artfacts than 1440x1080i60 25Mbps HDV (albeit on a higher resolution, progressive image). Are "all other things equal"? Probably not. For one thing, HDV is CBR, while it sounds like this other camera is using VBR. More importantly, there is lens-, sensor-, and processing technology involved.

[zieske]

2bdecided,

Yes, I agree with the image quality statement. The issue with the JVC I quoted is that all reviews say the image is not that great. What is the point of having this level of technology if you put a $10 ccd on the camera?

What really interests me is the info in your PS. can you expand on what you are saying there? Mathematically, I can understand your point about the probability of catching more artifacts if you start with more data, but mentally my reaction to that is: yeah but I have so much more data that I can afford to throw away during editing. My mind is still in the digital still world. If I start with a huge pic and make it smaller, it looks really good. If I start with a small pic and try to make it bigger, I end up with a really big crappy pic.

[JohnnyMalaria]

Both formats are poor acquisition formats but good distribution formats. So, which has the most appropriate/acceptable tools available to you (definition of which depends on exactly 'you' means)? A great distribution format may be a lousy choice for editing.

[zieske]

any editing of video requires decompression of the stream. I am not sure of the relevance as it is just as easy to find wintel based utilities to edit m2t based files as it is to edit avchd files. everything has its drawbacks and the editing process itself will likely cause more damage than the original compression did.
I have no intention of trying to edit on the fly or on camera. I am also not a pro (obviously) so it isn't like I am going to drop 20G on a hardware/software editing solution.
I am just trying to think through the technology logically to come up with pros and cons to steer me in a direction. As far as I can tell, none of the manufacturers have "the" solution in a consumer level device. So without that, I am trying to decide what will best serve me for the next x years of use.

I hope I didn't totally misunderstand your comment.

[2Bdecided]

John made the most important practical point: AVCHD takes more computing power to decode, so it's painful to edit.

You can't edit lossless files.
You can edit near-lossless files, but near-lossless editing codec cost money. If you intend to use one, then for editing there's no difference between HDV and AVCHD.
You can edit HDV. You can even do it losslessly with some editors (i.e. parts that you don't change go through the edit as bit-perfect copies). With any other output format, the input files can be HDV.
You can edit AVCHD, not losslessly yet (AFAIK), but here's the problem: if the input files are AVCHD, and you do a cross fade or picture-in-picture etc, then when you preview that section, your PC has to decode to AVCHD streams at once, and process them. You need a very powerful PC.

As for the question you asked me: in the still picture world, the equivalent would be chosing between a 1Mpixel image at 100kB JPEG or a 2Mpixel image at 100kB JPEG. The latter has more pixels, but is much lower quality, because 100kB isn't enough for a decent 1Mpixel JPEG image, never mind a 2Mpixel one!

Cheers,
David.

[zieske]

uhhhg.. you are making my brain hurt.
If you have two different compression techniques whereby one can compress a 1 megapixel image to 100k and the second can only compress it to 200k, when the image is uncompressed, it is still the same image (you know, depending on how good the compression algorithm is.. blah, blah).
If on the other hand you use the same compression to turn a 1 megapixel file into 100k and a 2 megapixel file into 100k than something additional was thrown away from the 2m file, like color depth.
If (all things being equal) you capture two hd 1920x1080 streams, one at 17mbps and the other at 26mbps than the 26mbps file should be bigger.
so just as an aside, I found a review that tells me more about the camera operation than jvc will. while the camera captures at 60p, it has a dedicated asic to downconvert to 60i before it writes the file. you can get the 60p image if you hook up the camera using hdmi and it will upconvert on the way out. so to get the file using usb, you get the 60i data. funnier still is that the combination of the 3 ccds in this camera only add up to 1.6m. so they are doing interpolation (sorry, pixel-shifting) to get to 1920 x 1080 resolution.
ok, so I started in my first post saying that the consensus was this camera was no good. but lets follow through with your point.. Sony says it captures full hd 60i at 17m avchd using a 4mp ccd at a rate of 8.25GB/hour on its new xr series. Canon says it captures full hd 60i at 24m avchd using a 6mp ccd at a rate of 10.5GB/hour on its new hfs series. whereas the jvc is capturing interpolated full hd 60p at 26m hdv using a 1.5mp ccd at a rate of 12GB/hour.
Thanks for making me think through that. it all makes a little more sense. math is your friend.

As to your other point, I have seen the system requirements/recommendations for editing avchd (dual quad core with 8gb and a fast hard drive). I don't have that yet which was one of the other reasons I thought the older technology stuff might be a little more convenient.

thanks for all the comments.

[2Bdecided]

If on the other hand you use the same compression to turn a 1 megapixel file into 100k and a 2 megapixel file into 100k than something additional was thrown away from the 2m file, like color depth.

That's it exactly. Except what's thrown away are DCT coefficients, or at least they're more aggressively quantised.

Which means loss of high frequency details, ringing, and blockiness.


As you say, the sensor numbers alone don't look great for that camcorder!

Cheers,
David.

[hazydave]

The avchd format used by (it appears) all of the new model hd camcorders leads me to part of my question here. My understanding is that avchd is akin to mpeg4 compression.

No "akin" about it.. AVCHD is using the AVC video CODEC, which is also known as H.264 and MPEG-4 Part 10. So, as one of the two CODECs in MPEG-4, this _is_ MPEG-4 by definition. AVC is largely accepted as the successor to MPEG-2.. it's the basis for things as tiny as iPod and cellphone video (DVB-H for example), and it's also the basis for most HD satellite video... my Dish Network STB gets its HD channels in AVC.

Like MPEG-2, AVC wasn't really intended as an editing format, but it is currently the offering the best quality video at the lowest bitrate.. at least versus anything not still in a lab somewhere.

The AVCHD format itself is a decendent of Sony's professional XDCAM, as well as the Blu-Ray specification.

In the digital world, all compression is bad as it relates to an algorithm used to throw data away to make the file smaller.

Not really. When you're taking about psychoacoustic/psychovisual compression, like MPEG-2 or AVC, you're talking about an enabling technology... digital video cameras in the consumer/prosumer market wouldn't exist without it. And they exist simply because they delivered better overall video than analog cameras.

What you don't like is visible compression... can you see compression artifacts? How do they compare, one CODEC to another? The big win of AVC is that, when well encoded, it's 2-3x the "coding efficiency" of MPEG-2. In other words, I can get the same quality with 1/2 to 1/3 the bitrate using AVC vs. MPEG-2. This isn't even open to debate... the fiasco of the early Blu-Ray discs, still using MPEG-2, compares to the early HD-DVD discs, all based on Microsoft's VC-1 CODEC (VC-1's coding efficiency is between AVC and MPEG-2, but closer to AVC).

The deal with camcorder AVC is simple: AVC is still a new technology, MPEG-2 is old, and thus, mature. Your camcorder isn't delivering the best possible AVC compression, it's delivering whatever the on-board video DSP can manage to deliver in realtime. In older AVCHD camcorders, and perhaps still in off-brands, video suffered due to the algorithms employed, not AVC itself. By most accounts, recent AVC camcorders with higher bitrates rival HDV. Going to 24Mb/s, they're pretty close to HDV's 25Mb/s anyway (though most of the time, the AVC camcorder is doing a full 1920x1080 vs. HDV's 1440x1080). My experience is that it can actually be better, but there are also conditions that show up more artifacts with AVC (though I'll admit most of my experiences are with a somewhat older AVC camcorder, compared to two HDVs I also use).

Just like having the ability to pull "raw" images off my canon digital camera, if only there was a way to capture "raw" video for editing from a camcorder, that would be the golden cow right there, right?

RAW from digital still cameras actually gives you two things: no compression artifacts, and the original 12-bit or 14-bits per pixel capture from your sensor, rather than chopping it down to 8-bits/color JPEG. Higher quality JPEGs are much of a problem, noise wise, but there's no way you're going to get the additional dynamic range available in RAW shots.

I have read several opinions stating that HDV to tape actually produces superior quality results. The basis of this argument is that the HDV format is akin to mpeg2 compression.

Until recently, it wasn't debatable.. HDV produced better results, at least in some circumstances, all else being equal. Again, HDV uses MPEG-2, there's no "akin" to be found here. HDV also uses MPEG Layer 2 audio, which is compressed... DV used uncompressed audio, AVCHD specs out AC-3, which is also a higher-coding efficiency format than MPEG Layer 2. It could actually be better, but mics in consumer camcorders are so wretched, no one's likely to notice.

so part 1 of my question is: is mpeg2 better because it compresses the original data less than mpeg4? Or is mpeg4 so much better a codec that the loss experienced in the compression does not produce any worse results than the older mpeg2 format?

Yes.

The real answer is that AVCHD is only just recently starting to compare in quality, on camcorders, to HDV.. that doesn't mean always the same, it means sometimes better, sometimes worse. As the encoding technology improves, AVC should be expected to increasingly best HDV, and at increasingly lower bitrates. You can see this in action on Blu-Ray discs.. but I know, when I encode HDV for Blu-Ray, I spend 3-4 hours per hour of video, or more.. on a Q6600 Quad core processor. You're not getting that kind of power on a camcorder yet.. AVC is pretty complex to do well. But eventually, sure.

The avchd spec claims to have a top bit rate of 24mbps. There is only one brand on the market that meets those specs; canon. And from what I read, they are performing a little voodoo to get to that spec. The obvious benefit of recording at the max possible bit rate is a (technically) more detailed image. Like I always say about digital cameras with multiple resolution settings.. why would you ever think that the picture you are taking is not worth capturing at the max resolution. If you don’t need that detail, throw it away later when you are editing. You can’t make up the data later (without interpolation).

The AVCHD spec was defined with various consumer parameters to make it a well defined, easy to deal with thing for the whole consumer tool chain... 24Mb/s was considered all you'd need for consumer video, and they're probably right. There's nothing wonky about Canon's 24Mb/s.. of course, if you're using a flash card camera, you're going to need fast enough media, but it's not such a big deal.

Lowering bitrate isn't a matter of lowering pixel rate.. it's a matter of increasing the low pass filtering of the video. All DCT (Discrete Cosine Transform) CODECs, including MPEG-2 and AVC, work their magic along similar lines. They run the transform on specific blocks of video (in MPEG-2 it's usually 16x16, in AVC it's more flexible), which transform spatial data to frequency data. Then they run a low-pass filter, chopping out the high frequency data, and then a lossless compression, like Huffmann encoding. The key here is the LPF... as you do more filtering, you get smaller data out of the lossless routines, but you also eliminate the high frequency data... sharpness. There's no way to get that back.

The apparent ability for this camera to perform this video miracle is its recording format of mpeg2-ts. If you look at the newer JVC models which have switched over to the avchd (almost a) standard, their cameras drop down to 1080/60i at 17mbps like all the rest.

No miracles.... modern HDDs (which is JVCs area) can usually handle 25MB/s, not simply 25Mb/s or whatever. AVCHD also uses the MPEG-2 Transport Stream format.. it's typically used for things like this. ATSC, DVN, and most satellite TV uses it, too... it's based on small packets, so there's little buffering needed and a continuous stream of digital data to store.

So part 2 of my long winded question is: is avchd really better?

Not yet. It probably will be some day. The purpose today was not to deliver better video, but more or less the same quality video on "computer" media: BD/DVD, HDD, Flash cards. This is a big advantage... consumers find these easier to work with. And the conversion was already underway.. in 2007, consumers bought more DVD camcorders than tape, first time ever. So the HD folks needed to enable HD on consumer media. The AVCHD spec was a good way, since its leveraging so much of the Blu-Ray infrastructure, rather than going some other odd way.

And as you know, interpolation does not create detail, it takes the cruddy detail you have and makes it bigger.

No one's really talking about interpolation, anyway. Systems that "uprez" do some sort of interpolation, true, but they also do line detection and other things which make the video look better to humans... it has nothing to do with restoring the original video. It's a trick, but one most people like (I was very skeptical, but the PS3's upscaling algorithms really do improve the overall look of many DVDs). You can't use that for improving lower bitrate video, because the pixel count isn't altered. You might be able to punch it up a bit using sharpening algorithms (unsharp mask, edge detection and enhancement, etc).