Video for natural resource monitoring - technical questions

[pdowty]

Sorry for the long message - any help would be greatly appreciated.

I work for a state natural resource agency. We conduct regional monitoring of aquatic vegetation in Puget Sound using underwater video. We are faced with some video technology issues.

Current situation: We fill over 100 Digital8 tapes with underwater video every annual field season. We also put copies of this video on authored DVDs as a backup. Previously we used VHS tapes and have a large library of old VHS tapes. We need to archive all video.

Issues:
· We are concerned about degradation in the older tapes (up to 10 years since recording) and need a strategy for long-term archiving.
· We have found that the Digital8 video is far superior to DVD for vegetation classification. The video quality is noticeably better but far more important we get much better results when looking at individual frames or a slow sequence of frames.
· We sense that the Digital8 format is fading (tapes and video equipment less plentiful) and are beginning to think about transitioning to a newer technology. We are working on the assumption that this new technology will involve disk-based storage media, or perhaps solid-state media.

Questions:
1. Am I correct in thinking that the relatively poor performance from authored DVDs is a result inherent in the MPEG-2 compression (relative to DV used in Digital8) which loses a lot of information relative to the larger DV video? In other words, adjusting parameters - e.g. frame rate, etc. – would not do much to affect this discrepancy between DVD/MPEG-2 and DV.
2. Are there other obvious options of encoding (codec) and container that might be more appropriate for us than DV-AVI that we should consider for storage on disk? An online video storage calculator indicates to me that DV-AVI files are much larger than MPEG-2 files on DVD, but we want to retain the DV quality.
3. Is 40 GB per Digital8 tape a reasonable working number for planning purposes for needed disk storage capacity? This was taken from a source that discussed 8mm tapes in the context of computer data storage (e.g. using old Exabyte tape drives).
4. Is our assumption reasonable that we need to move from tape-based solutions to disk-based solutions?
5. Given our large storage needs and limited resources, are there digital video storage solutions out there that would be more appropriate than putting 2TB 3.5 inch drives in a 4-bay hard drive enclosure to get 8 TB storage units? What do video professionals use?

Thank you very much for any comments or suggestions.
Pete

[aedipuss]

1-no. a well made dvd made from a one hour dv tape will be virtually indistinguishable from the tape.

2- and 3-DVavi is 13 gb per hour or 25mbps. dvd spec mpeg-2 maxes out at ~ 10mbps with audio or about 5 gb per hour. mpeg-4 is a little lighter on storage space for about the same quality output, but will not play in most dvd players if that is a requirement for storage on disc.

4-d8 is dead. wasn't a high quality system to start with. miniDV was better and there were nice high quality SD cams available, but it's also mostly done. HDV uses the same miniDV tape and 25mbps bitrate on a one hour tape but is 1080 HD. the quality of the camera is much more important than the storage medium.

5-nas storage is fine as long as you set them up using a raid system that allows rebuilding arrays when a drive dies, and they will. so 8gb of drives results in more like 5gb of usable storage after formatting the raid array.

[pdowty]

Thanks for the response aedipuss.

Your response surprised me. I'm hoping you can help me with two followup questions.

1) If MPEG-2 has such lighter storage needs than DV-AVI (5 GB/hr vs. 13 GB/hr) for virutally indistinguishable quality, is it correct to attribute this to simply a much greater efficiency in the MPEG-2 format? If so, that is remarkable.

2) We have had significantly lower quality DVD/MPEG-2 video as compared to Digital8/DV for all combinations of DVD/MPEG-2 settings that we have tried. Is it more likely that we just haven't properly optimized the settings, or is the hardware encoder in our unit just of limited quality?

We have been using a Sony DVD Recorder RDR-GX7
(http://esupport.sony.com/US/perl/model-home.pl?mdl=RDRGX7&LOC=3#/manualsTab ). I don't see anything in the specs that could be used as a basis of comparison with other units to assess quality of encoding.

Thanks for your help.
Pete

[edDV]

1) If MPEG-2 has such lighter storage needs than DV-AVI (5 GB/hr vs. 13 GB/hr) for virutally indistinguishable quality, is it correct to attribute this to simply a much greater efficiency in the MPEG-2 format? If so, that is remarkable.

2) We have had significantly lower quality DVD/MPEG-2 video as compared to Digital8/DV for all combinations of DVD/MPEG-2 settings that we have tried. Is it more likely that we just haven't properly optimized the settings, or is the hardware encoder in our unit just of limited quality?

DV is a frame by frame recording format with ~5x intraframe DCT compression. As such, it is an ideal format for stop-frame analysis.

MPeg2 uses both intraframe and interframe GOP based compression where one full I frame is recorded out of every 15 frames and change data for the rest.

I've had experience with underwater video recording for the US Navy. The big issue that affects video compression is light scatter off floating particles in the water (aka "backscatter"). This adds considerable random noise to the recorded video making interframe compression difficult and inefficient. This is why you are seeing lower quality after encoding DV to DVD MPeg2.

All the low end HD camcorder formats are GOP rather than frame based. HDV format uses 25 Mbps MPeg2, AVCHD uses 24 Mbps h.264. Both use realtime hardware encoders in the camera. The HD resolution of these cameras may partially compensate for the inefficient interframe encoding caused by back-scatter. You need to do a test in your typical water conditions. Further compression from the camera native bit rates will be destructive to quality again depending on water quality and lighting. Plan to archive HD material at the 24 Mbps (or higher) rate.

Assuming your HD camera tests are successful, here are some further considerations.

AVCHD format records to flash media. A 64GB Class 10 flash card can record over 5 hours of 24 Mbps video. This will minimize opening the the underwater case as often.

The lowest cost frame based HD camcorders will cost over $5k. See DVCProHD HVX series* from Panasonic. These record HD frames at 100 Mbps rate. DVCProHD is the HD version of the DV format.

Any artificial lighting should be at 35-45 degree angles (h and v) to minimize backscatter.



*Panasonic has a newer HVX-170 DVCProHD solid state camcorder priced from $3395. A 64GB P2 flash card prices around $700. You should buy at least 2.

http://www.bhphotovideo.com/c/product/751649-REG/Panasonic_AGHPX170PJ_AG_HPX170_P2HD_S...Camcorder.html
http://www.fullcompass.com/product/369684.html?utm_source=googleps&utm_medium=shopping...FUcHRQodvFbhBQ


PS: Since you are near Puget Sound, you can see some of the equipment my team used at the Naval Undersea Museum in Keyport, WA. They may help you with local contacts to help your program.
http://www.history.navy.mil/museums/keyport/index1.htm

A deep water camcorder housing.

[pdowty]

Thanks edDV for the response. Very helpful - especially to hear of your underwater experience.

I now understand your point about the importance of frame-based vs. GOP-based compression and I assume this is much more important than the specific hardware (Brand, model) used to do the compression.

I'm hoping I might get input on three more issues.

1) Estimating data rates. I have the data rates you quoted, but I'm trying to come up with more numbers to help weigh video quality against storage requirements. So far I've put together the numbers in the attachment digitial_video_compression.pdf. I realize my sources are not authoritative and I suspect these numbers are not accurate. For example, std definition MPEG-4 has a higher data rate than MPEG-2/DVD in these numbers. Also, the resolution (720 vs 1080) does not affect numbers for the DVCPRO HD data rate. If these numbers are grossly inaccurate, is there a more reliable source for these numbers?

2) After getting a demo from my colleague that does our work with DVD and Digital8 viewing, it seems that some differences we're experiencing could be due to equipment (DVD player vs. Sony tape deck to TV) rather than the compression/video quality. The DVD player doesn't seem flexible in terms of stop frame and slow motion viewing. If we get all our video to computer files, are there standard software options we should be considering for stop frame comparisons of file-based video?

3) Do you have experience with video enhancement to address the backscatter issue? We have certainly seen the benefits of simple Photoshop enhancement in still images (see image below). Increasing the contrast would certainly aid video classification. The image below is perhaps not the best example because the water is pretty clear, but you get the idea. We recently learned of hardware designed to perform video enhancement (www. lyyn.com ) and are now interested in looking at enhancement options. We currently now collect video in standard definition with an NTSC camera and are thinking that some form of enhancment might be more powerful than a move to HD.


Once again, any thoughts would be greatly appreciated.