LTO5 real-world performance

[vu2srm]

Hi all,

I am working on the project wherein we will be capturing around 4000 hrs of video data.

The initial test of video capturing with Declink extreme capture card in uncompressed video format at 720x570 resolution 25fps.
With this settings the 60 mins video captured file size goes upto 100 GB.

I am thinking of LTO5 as a long term archive media.

First question is, is that fine to go ahead with LTO5?
and what is the real-world experience of data transfer rate achieved during the data transfer with LTO5 Ultrium 3000 with SAS interface?

The data source is Raid5 1 TB internal storage where the videos are captured.

SAS interface to be used: HP SC44Ge Host Bus Adapter - 8 Channel - SATA-300 / SAS - 300 MBps - PCI Express x8

LTO Drive: HP LTO5 Ultrium 3000

Can someone help?

vu2srm

[SLK001]

Wow... 4000 hours of video at 100GB per hour? You're talking about 400 TERABYTES of needed storage. Why do you use some lossless codec, like HUFFYUV to save some space? What makes you think that you have to store everything without any compression?

[vu2srm]

SLK001: What makes you think that you have to store everything without any compression?

Idea is to store the content in "AS IS" condition as almost 10% of the content is on 30 yrs old VHS which need to be restored (artefact removal). We are being more optimistic about the would be technology in restoration, intend to retain the original content with whatever artefacts it has, to be processed with would be technology. I know its a costly affair but let that be.

So it is clear that the stored stuff wont be accessed that regularly hence looking for long term archival solution. do you think LTO is a perfect fit for the purpose. While googling yesterday I came across thread which talks about IBM and FUJI FILM is working on 35TB tape cartridge, no idea about its developmental status and when that will be available.

So as per the currently available technology, guess LTO5 is the best fit.
Can some one address the original question of real-world performance of LTO5?

vu2srm

[tkliphuis]

Hi,

We are using LTO-5 for archiving video. From an Intel MAC to half height HP LTO-5 drive we get over 130 MBytes per Second, from an internal RAID. From an external HFS disk connected eSATA the MBytes per second drops to around 85. An external disk over FireWire 800 will run around 65 MBytes per second.

Best case to fill a 1.5 TB tape is three hours, worst case is seven hours.

We are using ATTO H680 6GBPS SAS cards, and HP LTFS software.

Good luck.

Tony,

[vu2srm]

Thanks Tony,

The figures are pretty impressive. Can I expect improvement with H6F0 ExpressSAS [Maximum Transfer Rate (Full Duplex) 19.2 GB/s]?


Can you please as well tell us the max size file you have transferred and the time taken for the same. As I mentioned earlier I would be handling the RAW Video of 100 GB per file.

Thanks again.

vu2srm

[pandy]

Try different approch - resize video (pointsampling) 4 - 8x, use some decent encoder like MJPEG, Lossless H.264 etc - i think that You not loose any detail and You reduce bitrate for sure. With modern CPU's speed impact should be acceptable.

[vu2srm]

Sure...

As suggested by SLK001, test with HUFFYUV codec is already in pipeline. Will post the stats soon.

Thanks

[vu2srm]

Wow... 4000 hours of video at 100GB per hour? You're talking about 400 TERABYTES of needed storage. Why do you use some lossless codec, like HUFFYUV to save some space? What makes you think that you have to store everything without any compression?

Will carry out the tests with what you have suggested. Just got Bluefish 422 (the discontinued one), will be comparing the results with bluefish422 and declink extreme. Any experience with bluefish422?

[tkliphuis]

vu2srm,

Using LTO-5 we have backed up single files as big as 400 GB (2+ hours HD). That one file took around one hour to copy. I have never seen a copy to tape run faster than 140 MBytes per second (500 GB per hour). This is related to the maximum speed of the tape drive. HP LTO-5 drives have 6Gbps SAS interfaces, so I don't think a different controller will help.

If you are looking for a "real" standard storage format MJPEG-2000 can be encoded to support reversible lossless format, that would require around 50 GBytes per hour. We do archive to MJPEG, but include IMX50 and H.264 in archive sets as well, because MJPEG is difficult to work with as compared to something Final Cut can easily work with.

Good luck.

[vu2srm]

vu2srm,

Using LTO-5 we have backed up single files as big as 400 GB (2+ hours HD). That one file took around one hour to copy. I have never seen a copy to tape run faster than 140 MBytes per second (500 GB per hour). This is related to the maximum speed of the tape drive. HP LTO-5 drives have 6Gbps SAS interfaces, so I don't think a different controller will help.

If you are looking for a "real" standard storage format MJPEG-2000 can be encoded to support reversible lossless format, that would require around 50 GBytes per hour. We do archive to MJPEG, but include IMX50 and H.264 in archive sets as well, because MJPEG is difficult to work with as compared to something Final Cut can easily work with.

Good luck.

Thanks again Tony,

Is this HP StorageWorks LTO-5 Ultrium 3280?

What would be the windows alternative to LTFS?

Regards

vu2srm

[tkliphuis]

Hi,

IBM supports LTFS on Windows. The software is free. You should not have any big problems with an HP drive using IBM software.

http://publib.boulder.ibm.com/infocenter/ltfs/cust/index.jsp?topic=/com.ibm.storage.ho...l_upgrade.html

IBM has plenty of instructions and help information.

You might also want to look at the Wikipedia page for LTFS.

We are using the Ultrium 3000 half height drives.

Tony