Best storable formatt for future editing projects

[mpalm887]

My question spans multiple topics so I thought this would be the best place for it...

I'm capturing all of my old 8mm footage using a Canopus ADVC-110 (which I highly recommend btw) to DV-AVI on my PC. Space isn't too much of an issue, I've got 1.5TB of space, but of course the raw footage is unreasonably large. I've been reencoding them to MPEG-2 .mpg files using Premiere Elements before dumping them onto one of my external hard drives. Although this formatt uses more space then I anticipated, the quality is much more acceptable than the only other alternative I'm aware of: divx/xvid which I've tested and find to be too much of quality hit in both color and clarity.

My issue is that someone just tipped me off that MPEG encoded video is a pain in the a$$ to edit and that I should consider another formatt for storing footage unless I know I won't edit it again. Although most of it will not be edited, certain footage definitley will be such as old Christmas mornings, and sports footage.

It's the editing that I love, and I'm no technician when it comes to video conversion. So I'd like to find a formatt that I can store my home movies as which will allow me to

-not use an insane amount of space
-have a good quality picture
-edit footage again in the future

Quality is more important than space saving, and I'm prepared to use the space that MPEG-2 uses.

What are my options?

[redwudz]

For what you are doing, I would stick with good quality MPEG-2. It's really not that bad to edit, at least compared to higher compressed formats like Divx/Xvid. DV is one of the better choices, but you do have a problem with the space it takes up.

I would just use a decent MPEG-2 editor and be happy. I like MPEG Video Wizard myself for MPEG editing, but others also have their favorites.

[mpalm887]

That's reassuring, I have already captured and encoded about 25 tapes. I also considered just going the .divx/xvid route and passing them through a light filter to brighten up some of the dimmer footage.

One other concern I have is making sure I go with a timeless format. Is there such a thing? I'm going to be storing these for, well, potentially the rest of my life. Is it possible that divx/xvid will unpopularize in favor of some of the newer formats? Should I consider .mkv?

Sorry, I play devils advocate too much, I just like to get confirmation on all my haunting suspicions.

[yoda313]

As long as you keep your files in a retreivable media you will always be able to play them. Stick with the big guns like mpeg2 or divx/xvid. They will be here to stay. Look at mpeg 1, that has been around the longest and you can still use it.

If you are really concerned about it you could always download the codecs and burn to a backup disc and then you'll be able to use them no matter what. But I have a feeling divx/xvid and mpeg 2 will be around for a very long time. They may not be the 'popular' flavor at any given time but there will almost certainly be some online repository for those favored codecs.

[jagabo]

That's reassuring, I have already captured and encoded about 25 tapes. I also considered just going the .divx/xvid route and passing them through a light filter to brighten up some of the dimmer footage.

I think you should stick with MPEG2. But Divx/Xvid (MPEG4) can deliver just as good quality as MPEG2, and typically at half the size. MPEG4 is not as easily edited because of the long GOP sizes. And a lot of editing software simply doesn't support it as well, or at all.

I suspect what's happening is the player you are using to view the MPEG2 files is automatically making the video brighter as it is being played. This is because it knows that computer monitors are very different from televisions and it is trying to simulate how the MPEG2 data will look in TV. The actual content of the MPEG2 and MPEG4 files is probably very similar. That is, if the player did the same thing to the MPEG4 video it would look just as bright.

[daamon]

One question that hasn't been asked that is important is "What do you mean by editing?"

If you just want to cut bits out and moves things around then you'll be OK with good quality MPEG2 and something like MPEG Video Wizard, as per redwudz.

If you want to do colour correction, loads of effects etc. then you're better sticking with DV AVI as each frame is an I frame (i.e. a complete picture, not partial to achieve the compression).

[jagabo]

Of course he could use MPEG2 or MPEG4 with all I frames too. But then he won't get as much compression. Or if he still uses high compression the quality will suffer.

If he plans on a lot of filtering later on, the bigger issue is the macroblock artifacts and loss of detail with high compression. Filtering will likely exacerbate those problems.

[thecoalman]

If they are important like family video stick with the DV-AVI. 20 years from now you'll be happy you did. the best quality format you have will be whatever the most original is. We have no idea what is in store for the fiutre for restoring video but whatever it is having the highest quality video to use as a source is going to be important. You always lose a little bit when you reencode and that little bit may very well make difference in the future, besides 14 gigs per hour is really not that much and storage space costs are dropping every day.

The way I look at it is not keeping the original capture is like throwing negatives away. Once its gone its gone for good because those VHS tapes are not going to be in the same shape 20 years from now and the equipment is either going to be unavailable or very scarce.

[mpalm887]

If they are important like family video stick with the DV-AVI. 20 years from now you'll be happy you did. the best quality format you have will be whatever the most original is. We have no idea what is in store for the fiutre for restoring video but whatever it is having the highest quality video to use as a source is going to be important. You always lose a little bit when you reencode and that little bit may very well make difference in the future, besides 14 gigs per hour is really not that much and storage space costs are dropping every day.

These are the very thoughts that have been haunting me. I definitely want to only do this once. I don't want to compress alot and then try to restore video quality using whatever new software is out 10 years from now to make them more watchable because 1TB hard drives are $20 at WalMart (acually they'll probably be 1TB USB Sticks or something.

[mpalm887]

One question that hasn't been asked that is important is "What do you mean by editing?"

Good point. This is all family videos so they are important to me but I won't be adding special effects to them. Just some random clipping, adding transitions, and at times timing footage to music. My brother is an After Effects wizard but doesn't know a lot about compressed formats as he ALWAYS works with DV-AVI or the native format captured/transfered from the camcorder.

[mpalm887]

At this point I'm fairly happy with the MPEG 2 except with sports footage. I have some old footage from the lake and the really fast moving stuff makes me sick to watch in MPEG 2 format but in original DV-AVI I it's fine. What happens during the conversion that makes to fast-moving scenes so dizzying?

[thecoalman]

What happens during the conversion that makes to fast-moving scenes so dizzying?

Shouldn't be a huge diffrence if you're using an adequate bitrate. Having said that MPEG compression relies on being able to convert the information from one frame to the next, you have I frmes which are full frames and the other are made up from infopmation from previous frames. With 8mm you have so much amount of noise which eats up the bitrate and if it was not shot on a tripod even those little movements eat up a lot of bitrate because every frame is constantly changing.

Here's a good example from a clip using a very inadequate bitrate that may help you understand this., note the lights which are flashing and moving which produces huge macroblocks, yet the rest of the footage doesn't. That's because the lights change quite bit from one frame to the next but the rest does not.



So if you have action footage, with noisy hi-8 and its not on tripod basically ervery pixel for the entire length of the video is never the same so you need a lot of bitrate to get anything decent.


-----------------------

Note that screenshot is from 3000kbps encode, noise in the original DV-AVI is nearly non-existent, it's shot on tripod so the encode is very good with adequate bitrate. This is about as action packed as video can get... http://www.nepadigital.com/reencode/8000cbr.mpg

[gadgetguy]

At this point I'm fairly happy with the MPEG 2 except with sports footage. I have some old footage from the lake and the really fast moving stuff makes me sick to watch in MPEG 2 format but in original DV-AVI I it's fine. What happens during the conversion that makes to fast-moving scenes so dizzying?

In addition to adequate bitrate you must also maintain the proper field order and encode as interlaced.

[mpalm887]

Coalman that is extremely enlightening. I see a huge difference in those pictures. I've just convinced my dad to buy me a 1TB external purely for the purpose of storing raw captured footage. Now I can keep my 1/2TB for my personal documents and photos. Most of this captured footage was shot when I was a kid and since it's essentially his camera work of me and my siblings, he felt compelled. Looks like the core of my problem is solved.

I do have a couple of other questions about compression that I should probably just do more of my own research on, but since the thread is active if anyone doesn't mind commenting or at least pointing me to some info I'd be greatly appreciative.

In the future I plan on using my PC in conjunction with my home theater, but for now I use a PS3 for viewing optical media and have tried to dump a few of my home videos onto the HD in MP4 format (divx is also playable now). While this is extremely convenient since I can simply scroll through the videos without having to insert a disk or connect external devices, the quality suffers (naturally). I used PS3 Video 9 but a standard divx conversion looks essentially as bad. Jagabo mentioned:

Of course he could use MPEG2 or MPEG4 with all I frames too. But then he won't get as much compression. Or if he still uses high compression the quality will suffer.

If he plans on a lot of filtering later on, the bigger issue is the macroblock artifacts and loss of detail with high compression. Filtering will likely exacerbate those problems.

What software can I use (I'll probably need a guide too) to convert my AVI stuff to MP4 for viewing on my PS3 but not with so much compression/quality loss. I use AutoGK to convert to divx but even with a target quality of 100% is looks pretty crappy in comparisson. I'd actually love to compress some of my .avi stuff to the quality of my MPEG2 stuff, and leave the MPEG2 stuff at it's current size/quality but change the format to mp4 or divx for viewing on my ps3.

[thecoalman]

Coalman that is extremely enlightening. I see a huge difference in those pictures.

Just so it's clear the first one is only 3000kbps that I made specifically to create macroblocking. It's entirely inadequate for MPEG at 720x480, especially that clip. The second is 8000kbps but its still MPEG, there's more clips and screen shots here: https://forum.videohelp.com/topic257651.html

The difference between the DV and MPEG is noticeable but its not that much.

[mpalm887]

The 8000kbps sample is excellent. I've been chopping off some unwanted fuzz from the first and last few frames, dropping some quick-fade transitions and lead-ins/outs and exporting from Premiere Elements as MPEG 2 with almost stock settings. I'm going to do some tweaking and export some samples to compare to eachother.

The difference between the DV and MPEG is noticeable but its not that much.

Agreed, and for the quality difference MPEG2 is worth the space saving IMO. I'm definitely not technically minded in this stuff (not yet at least) but seem to have quite a discriminating eye. It's a blessing and a curse.

[jagabo]

What software can I use (I'll probably need a guide too) to convert my AVI stuff to MP4 for viewing on my PS3 but not with so much compression/quality loss. I use AutoGK to convert to divx but even with a target quality of 100% is looks pretty crappy in comparisson. I'd actually love to compress some of my .avi stuff to the quality of my MPEG2 stuff, and leave the MPEG2 stuff at it's current size/quality but change the format to mp4 or divx for viewing on my ps3.

The biggest problem you'll have is the interlacing of your source. Players generally don't handle interlaced Xvid/Divx well.

Here's a sample using VirtualDub with the same frame size at about half the bitrate of thecoalman's 8000 kbps MPEG2 file (note that I was starting with his MPEG file -- starting with his DV source could have given better results):

smart.avi

I used Donald Graft's Smart Deinterlaced filter and MSU Deblocking (to remove the MPEG macroblocking, deblocking wouldn't be necessary with the original DV). This uses Xvid's DAR setting to specify 16:9 playback. Some players do not support this and will playback at the SAR of 3.2:

Here's the same video at 576x320 (square pixel) and 1/4 the MPEG2 bitrate:

576x320.avi

These have a little more macroblocking than the source MPEG2. Using Xvid's deblocking function (on playback) will eliminate most of it.

[thecoalman]

The difference between the DV and MPEG is noticeable but its not that much.

Agreed, and for the quality difference MPEG2 is worth the space saving IMO.

But again that is absolutley clean source, noise is almost non existent and it's shot on stable platform. I'd first do some tests with your own footage before settling on MPEG. You could also as suggested above go with I-frame only MPEG, file will be bigger than normal but you'll have all single frames. In the end though I really think it's fruitless effort to encode a lightly compressed format like DV-AVI to MPEG for long term archiving

[thecoalman]

Here's a sample using VirtualDub with the same frame size at about half the bitrate of thecoalman's 8000 kbps MPEG2 file (note that I was starting with his MPEG file -- starting with his DV source could have given better results):

The source is in the link I posted above.

[JohnnyMalaria]

Personally, I keep all my original tapes - Video8, Hi8 and DV.

My oldest recordings are from 1994. At the time, the idea of DV quality video at consumer prices was quite literally incredible. My only archiving option (other than the original tapes) was to record to another analog format or capture to MPEG1 with my 486DX100 Win3.1 system - both unacceptable.

14 years on I can take the original tapes, play them in a better camcorder (a Hi8 one I got in 1996) and convert to DV. The quality is much better than I could have dreamed of. I would have surely regretted "archiving" them in 1994.

The same is true with my DV tapes - some from 1997. I keep the tapes. In another 15 years time, the chances are very high that I will be able to play the tapes in one of my DV camcorders/decks and I won't need to worry about getting a computer to play what will be an archaic DVD format and/or hard drive.

Currently, I have about 150GB or DV format AVI files representing ten tapes (DVCAM) that are being edited. The final edited video will be recorded back to DV tape as well as converted to MPEG2 for DVD authoring.

The cost of mass storage is so cheap today, it doesn't make sense to throw away quality for the sake of a few GB.

[mpalm887]

I exported a 1 minute clip of my brother and I body surfing in Hawaii and even with the kbps turned up to 8000 the raw avi footage looked much better. Again the MPEG quality was very acceptable for viewing randomly.

You could also as suggested above go with I-frame only MPEG, file will be bigger than normal but you'll have all single frames.

Can this be done in Premeire Elements? In the export settings you can adjust "M Frames" and "N Frames" but I'm not sure exactly what that means.

I'm going to play around with some of these other settings. You can also select 2-pass VBR bitrate encoding, which I'm assuming would make a big difference in the quality output. As wll as "Field Order" settings which you can change to "none (progressive)" but I don't know if that will help. These are things that I should read about in my help manuals of course.

In the end though I really think it's fruitless effort to encode a lightly compressed format like DV-AVI to MPEG for long term archiving

If what you mean is that I may as well leave it uncompressed I'm with you there. At this point I'm likely going to stick with DV-AVI and take my pop's up on the free 1TB HD. I'm still interested in finding the best balance (for me) of compression/quality for exporting projects to PS3, optical media etc. for sharing with family/friends, I suppose that's what this is really about now.

[thecoalman]

I'm going to play around with some of these other settings. You can also select 2-pass VBR bitrate encoding, which I'm assuming would make a big difference in the quality output.

It will make a difference but i wouldn't expect miracles, the bitrate gets disbused to where it needsit the most.

As wll as "Field Order" settings which you can change to "none (progressive)" but I don't know if that will help.

DV-AVI is always bottom field first so you would want to encode the MPEG as bottom field first. This setting alone can really make it look bad if you use the wrong field order. Not necessarily macroblocking but you'll get a ghosting effect during high motion when you view it on TV.

While on the topic you should be testing on DVD player being played to TV.