XVID better solution for backup and editing than MPEG2?

[kb1985]

Hi,
my collection of DV tapes is growing rapidly. I've decided to convert it to MPEG2 and record on DVD, but then i thought that maybe MPEG4 (xvid) will be better idea (because I'm planning to buy dvd set with xvid/divx playback). I don't mind size of file unless it doesn't exceed 2 GB (that's the max size of file in dvd specification). So it would be about two 2 GB xvid files on one DVD - which is fine for me.

But is xvid the right choice?
- is it easy to edit? i'd like to often cut part of the footage and reedit it. Should i set max iframe interval to some value - like 6 or 12?
- does it recompress better/worse/same like mpeg2?
- "layer2" in DVD seems easily editable, but "razor mp3" from xvid gives me problems in virtual dub - what should i chose to be sure that stantadlone dvd player won't make any problems?
- on my test footage it seemed little bit darker than original DV

Anybody uses xvid as their backup/editing solution?
How do you configure it for best results?
Do you use B frames?
Do you use "quality" or "2pass"?
etc.

thanks for your time
k.

[junkmalle]

Xvid is generally not a good backup/editing solution.

One of the main reasons it gets better compression than MPEG2 is because it can have very long stretches between key frames. This makes editing a pain. If you reduce the keyframe interval to 6 or 12 frames you won't get much more compression than MPEG2.

I don't know any editors that give you frame accurate editing and are smart enough not to reencode the whole video when you make a change. So you'll loose quality every time you edit. Of course VirtualDub's Direct Stream Copy mode doesn't loose quality -- but you don't get frame accurate editing.

If you use only key frames and quantization mode at Q=1 you will get the highest quality and, in theory, frame accurate cutting. The quality at this setting will be very similar to DV -- and so will the size!

B frames will make the files smaller but they also reduce the picture quality.

If your DV footage is irreplacable and important to you, just bite the bullet and archive as DV AVI. If I remember correctly, you can put 4GB files on UDF format DVDs. If you're talking about backing up DVDs Xvid is a reasonable choice -- you can always go back to the original DVD if you need a pristine copy.

[guns1inger]

If you are serious about it, buy a big hard drive and external enclosure and store DV-AVI on that. Cheaper solution is to put your edits back to tape. Tapes are pretty cheap now anyway.

Any compression algorythm based on temporal compression is a poor solution for editing (Divx, Xvid, Mpeg)

[kb1985]

Unfortunatelly I can't afford buying more and more HDD's because by the end of 2005 total size of my DV sizes will be overwhelming. The only solution would be HD-DVD but I'm not that naive that I think that I will get one by the end of 2005.

So I have to choose different solution. I was thinking about MPEG2 and making them DVD's, but because my new dvd player will also be able to play MPEG4, so I was thinking that maybe MPEG4 is better choice than MPEG2, because I've heared rumours that MPEG4 gives better quality and is easier editable than mpeg. I've posted here to verify those rumours.

1. Comparing 2 files of the same size MPEG4 wins right?

2. Will standalone dvd players read "lame mp3" or do i need "freunhaufer mp3", "PCM" or somthing else?

3. Why do I need "frame accurate editing."? Google says that it is something with keyframes but the whole idea seems still blurry to me.

4. MPEG2 or MPEG4?

thx

[Ward River]

MPEG2 is the standard. Hard drives aren't that expensive anymore. Buy one if you can afford it. Find a cheaper hobby you can afford if you can't.

[kb1985]

circa 12 GB weekly - that makes a lot of hard drives per year

[guns1inger]

You need audio that is comptaible with your player - only your manual can tell you what that is.

When you start editing, that when you will realise how important being able to cut on a specific frame is. If you don't do real editing, then there is no way to convince you yet.

The other problem with mpeg4 as a pre-edit compression (and mpeg2 for that matter) is that it does not like being compressed again. And if your edits are anything more than simple cuts, you will have to reencode. Your quality will suffer substantially if you are working with heavily compressed mpeg4 source material.

Mpeg4, like Mpeg2, was designed and implemented as a high quality, low volume, PLAYBACK. At this it excels, and it is amazing the quality that can be squeezed into such a small space at times. But as a source for editing it is a poor substitute for a real solution.

[kb1985]

So maybe i should save my mpeg4 as uncompressed, reedit and then compress again for better results than just editing the xvid file?

[guns1inger]

Too late. Once you compress the video to mpeg4 you cant go back. Converting it to uncompressed afterwards for editing might get you frame accuracy, but the compression damge is permanent, and when you reencode you compound the damage to some degree.

[drjtech]

circa 12 GB weekly - that makes a lot of hard drives per year

52 x 12GB = 624GB = only 2 320GB drives

Buy one now and the prices will drop before you have to buy your next one in six months.

Or, do as guns1inger advised and simply save back to DV tape.

-drj

[edDV]

You didn't state the importance of this material for future editing. DV is your best archive format.

52 tapes / yr = ~ $150 in bulk
52 x 12GB = 624GB HDD costs ~ $300-500 (cheaper every month)
Later you can dub all this directly to ~30 HD-DVDs and reuse or sell the drives.

Next best strategy, edit and keep best stuff on DV tape. Encode the rest to DVD MPeg2 (1 hr/ DVD).

[guns1inger]

You could also be more selective. On most tapes, for every hour of footage you get maybe 10 minutes of good stuff, another 10 of OK stuff, and the rest is dross. Don't just grab the whole tape. Keep what's good, toss the rest. You'll probably find you can fit most weeks on a DVD as DV-AVI.

[junkmalle]

1. Comparing 2 files of the same size MPEG4 wins right?

Assuming you performed the encodings properly, yes.

2. Will standalone dvd players read "lame mp3" or do i need "freunhaufer mp3", "PCM" or somthing else?

The players I have experience with can all play MP3 and MP2 audio from any compressor. They typically don't like very low bitrates (below ~96 kbps). And sometimes don't like really high bitrates above ~256).

3. Why do I need "frame accurate editing."? Google says that it is something with keyframes but the whole idea seems still blurry to me.

Because you want to cut your video where you want to cut it, not somewhere the codec decides is convenient (a nearby keyframe) -- which, with XVID, could be as much as 10 seconds from where you requested it.

[kb1985]

Next best strategy, edit and keep best stuff on DV tape. Encode the rest to DVD MPeg2 (1 hr/ DVD).

But still this topic leads to conslusion that it seems better to encode "the rest" to mpeg4 than mpeg2.



Thanx for all your replies!

[Ward River]

Next best strategy, edit and keep best stuff on DV tape. Encode the rest to DVD MPeg2 (1 hr/ DVD).

But still this topic leads to conslusion that it seems better to encode "the rest" to mpeg4 than mpeg2.

No, it doesn't.

[guns1inger]

The conclusion is don't use a lossy temporal encoder (mpeg4, mpeg2 etc) until the very end of the process. If you don't need to edit, do it now. If you do need to edit, do it after the editing is done.

[edDV]

The conclusion is don't use a lossy temporal encoder (mpeg4, mpeg2 etc) until the very end of the process. If you don't need to edit, do it now. If you do need to edit, do it after the editing is done.

Exactly.

I recommend keeping the original DV material if you want to edit in the future.

If that is too inconvenient, use a high I frame rate MPeg2 as the next best choice. At least you can play the MPeg2 on any DVD player.

Save the highly compressed formats for the finished product that will not be edited further. As said, editable frames are widely spaced. Reconstruction of missing frames is a very messy low quality business.

The purpose of archiving is to keep the original quality master for future use.

[LRD]

I don't see the choice: MPEG2 is the standard and encodes look better than anything I have been able to accomplish with MPEG4/DivX. Audio is also superior in MPEG2. When you consider DVD's can be purchased for less than 40 cents each, it costs only $1.20 to preserve 12 gigs of data. Also, your DVD's will play anywhere; your MPEG4 will play on only a handful of players--at least at present.

LRD

[steve2713]

I don't see the choice: MPEG2 is the standard and encodes look better than anything I have been able to accomplish with MPEG4/DivX. Audio is also superior in MPEG2. When you consider DVD's can be purchased for less than 40 cents each, it costs only $1.20 to preserve 12 gigs of data. Also, your DVD's will play anywhere; your MPEG4 will play on only a handful of players--at least at present.

LRD

Totally APPLES to ORANGES comparison. You can't compare an MPEG4 video at 500-1000kbps bitrate to a MPEG2 video with a bitrate of 4-8MBps. Same with the audio - most/all of the MPEG4 videos you've probably seen have been re-encoded from a lossy source, usually DVD, to begin with and are highly compressed, BY CHOICE. There's no reason I couldn't use audio bitrates and codecs that are SUPERIOR to the DVD format, let alone equal to, with MPEG4. It just happens that most MPEG4 videos floating around are using lower bitrate audio for space saving reasons.

[jimmalenko]

I don't see the choice: MPEG2 is the standard and encodes look better than anything I have been able to accomplish with MPEG4/DivX.

Two words - source and bitrate.

Audio is also superior in MPEG2

Should be exactly the same. You can still have 5.1ch 448kbps AC3 or PCM on AVIs. Dunno about DTS though.

Also, your DVD's will play anywhere; your MPEG4 will play on only a handful of players--at least at present.

That is true, but given the context of the question - editing - settop box playability doesn't really come into it, as these don't need to be playable since they're only going to be used for editing further down the track.

At the end of the day, if you want the absolute best, it'll cost you, in some way, shape or form. In this case, it would either be to buy a couple of hard drives or some DV tapes, as the absolute best case scenario here IMO would be to keep it as DV.

[DarrellS]

But still this topic leads to conslusion that it seems better to encode "the rest" to mpeg4 than mpeg2.

Exactly. Burn your excellant quality stuff to MPEG2/DVD and everything else to DivX or XviD as a data DVD.

[rgs_uk]

Isn't DV just a variation of MPEG2 which is designed for editing??

The bitrate for MPEG2 which will play in your DVD player is way lower than the bitrate for DV. So you will be throwing away some quality. Plus you then end up with a version of MPEG2 which is primarily designed for playing, not editing.

It's worth mentioning that you can push up the bitrate of 'DVD' MPEG2 way above the limit that stand-alone DVD players will handle and you can still play it on your PC.

If I were you I would leave it all in DV format for archiving. Split it and put 20 minutes (?) of DV on each DVD disc. A blank DVD only costs 20p (35c). Even if you make two copies the costs is almost nothing compared to a DV tape.

Plus you could make an additional 'viewing copy' at a normal DVD bitrate (6000 or something) for watching on your TV.

[SaSi]

It is easy to make wrong choices based on sound theoretical grounds.

Yes, DV is prefferable to keep because it coded in a lossless compression scheme, so you can edit at your heart's desire and lose nothing. (Faster too as no re-encoding is done).

On the other hand, MPEG2 at 8Mbps and above is considered (not by me but by experts) to have no considerable loss in quality.

If you don't believe this, try this.

Take a 1 minute ultra high quality video clip and encode it to MPEG2 at 8Mbps CBR with a descent encoder. Then decode this to uncompressed AVI (use VirtualDUB Mod).
Then repeat the above cycle as many times as you like, prefferably renaming each pass file with a sequence number to keep all generations.

I have done this and I have noticed negligible image quality loss after 20 cycles.

The "trick" is that once the first encoding is done, all that is going to be lost is lost. Quantization is done, some chrominance values are gone (the eye cannot notice them and the typical TV screen won't be able to show them anyway) and every cycle has ready-made quantizized material that goes through the loop.

All this goes to point out that re-encoding is not nececarily bad. Certainly a waste of time, but not a catastrophy.

In terms of the original question, DV vs MPEG4, I will give both faces of issue.

Using MPEG4 as an intermediate file with a Key frame interval of 5 frames or less (even 1) and a Q factor of 3 or less (2 if you have space) is fine for editing and quality is not lost.

However, the file will become invariably larger than any final version MPEG2 destined for DVD authoring.

At low bitrates, both xVid and DivX tend to quantizize chrominance badly and when viewed on high quality monitors or flat TVs, such videos tend to show high contrast and darker colours.

As for the archival question, I will agree that the best approach will be to:

1. Transfer DV from camera to the PC
2. Trim the video footage to remove garbage and also split the footage into distinct segments, keeping DV as the video format.
3. Record the split footage segments on DVDRs at about 4Gb per disk
WHY? It's cheaper and you also keep a reference archive:
One 200Gb HD actually holds 186Gb, or the contents of 46 DVDRs (at 4Gb each).
Where I leave, a 200Gb disk costs about 115 Euro and a DVDr costs 0.3 Euro. (In other countries exact prices differ but I trust the relations are the same).

Therefore, storing the material onto 46 DVDRs cost 14 Euro, i.e. 1/8th of the cost of keeping footage on disk. I also find that Read Only disks are less volatile than a Hard Drive

[guns1inger]

Totally different algorythms. DV isn't temporal in nature. That is, every frame is discreet, and the compression is done in the way the colour is stored. DV compression is fixed at around 5:1, and is a contant bitrate. It is the fact that every frame is, for all intents and purposes, an I frame, that DV is a good editing codec.

Mpeg-2, on the other hand, is a temporal compression codec. Compression occurs not just intra-frame, as with DV, but also interframe. Only one frame per group of pictures (GOP) contains a complete frame. The intervening frames hold only the changes between that frame and the last I frame (OK, this is a little simplified, but basically sound). Many editors struggle to cut between I-Frames because the information isn't complete. For PAL DVD the GOP is 15 - 18 frames long, so any edit could be out by up to 17 frames. That is almost a second.

Mpeg4 takes this concept further and enhances it to get better compression with less quality loss, but has the same basic issues. One way of getting more compression is to have less I frames. However this means that when you edit you might be several seconds out from where you really want to be.

As I said earlier, mpeg-2 and mpeg-4 are great delivery algorythms. Good quality in a small space, ideal for playback.

DV is designed for editing. It eats a lot more space (12 -13GB/hour), but has frame accurate freedom.

Horse for course. Just make sure you back the right one at the time.

[rgs_uk]

I wouldn't recommend hard-drives for longterm storage. Over the years I have had several that worked fine when they were put into the cupboard but then wouldn't power up properly when I took them out a year later (200Gb is a lot to lose). They are great if they are being used regularly or constantly.

If you want to archive your material safely then you need two copies (even if you are using a hard-drive). For me that means keeping the tape and a means of playing it, plus a DVD containing DV AVI or, in the case of Hi8 material, a good quality MPEG2 transfer on DVD.

If there is no tape, I store two copies on different brands of DVD in different locations. It would be madness to have just one DVD copy of something and for that to be your 'viewing' copy, as discs are so easily damaged. The archive copies (or at least one of them) should not be used as viewing copies.

As for editing, I have had a similar experience wth high bitrate MPEG2. I have opened it in Vegas, edited and then encoded it back to MPEG2 and I can't see any difference in image quality.

Also, if you want simple editing using just straight cuts, you CAN achieve frame accurate cuts with Womble MPEG VCR with no recompression on almost all of each shot. Most of the frames in the shot will be identical (not recompressed), with just a few frames re-encoded around the edit point. And you will really not notice it.

One minus is that MPEG2 takes longer to open up for editing, in Vegas for example, and is more sluggish to work with than DV.

However, I really do think that DV split into 20 min chunks on DVDs, and two copies, is the way to go.

For archive purposes MPEG2 or DV is preferable right now because the two formats are so widespread. You are much more likely to be able to play MPEG2 or DV 20 years from now, than some version of DivX or XVid.

[edDV]

For archive purposes MPEG2 or DV is preferable right now because the two formats are so widespread. You are much more likely to be able to play MPEG2 or DV 20 years from now, than some version of DivX or XVid.

So true. I've learned from past experience to encode a player on every MPeg4 or other oddball compressed format DVD.

[guns1inger]

Ultimately, do what suits you. Having had discourse here with people who believe that you can get 8 hours on a single sided, single layer disk, and have it indistinguishable from the commercial original, I know people will do what they want, regardless of what they are told. I do a fair bit of editing, and would never, out of choice, use mpeg as a source. DV is faster, easier, recompresses better, and is designed to do just that.

I suggest though, as a test before committing yourself to something you can't undo, that you try it out. Make an informed decision by doing, rather than trusting that anyone here knows what's right. You have half a dozen different and conflicting views. Each of us knows where we stand (rightly or wrongly), except you, because you haven't actually done it yet.

Have a go. It costs you nothing but time, and you will learn a lot from the experience.