VideoHelp Forum

Our website is made possible by displaying online advertisements to our visitors.
Consider supporting us by disable adblocker or buy Replay Video Capture or PlayON and record Netflix!
+ Reply to Thread
Page 1 of 2
1 2 LastLast
Results 1 to 30 of 41
Thread
  1. Everything I say is false koberulz's Avatar
    Join Date
    Oct 2006
    Location
    Australia
    Search Comp PM
    What's the best storage option for a large VHS-to-digital project? I initially got an 8TB external, thinking that would last at least a good chunk of it, but I've run out of space almost immediately.

    I also currently have no backups, which isn't ideal.

    EDIT: It's important to pay attention when using multiple tabs for different subforums. Oops. Meant to put this in Newbie/General.
    Last edited by koberulz; 10th Apr 2018 at 10:43.
    Quote Quote  
  2. Mod Neophyte Super Moderator redwudz's Avatar
    Join Date
    Sep 2002
    Location
    USA
    Search Comp PM
    Maybe a external box with several internal HDDs. Use USB 3 or eSATA connections.
    You could also add RAID, but that would eat up a lot of drive space.
    Just an example, not a suggestion:
    https://www.amazon.com.au/TerraMaster-D5-300-External-Enclosure-Diskless/dp/B01DW1X348...ive+enclosures

    I'll move this thread to Newbie/General Forum.
    Quote Quote  
  3. Member
    Join Date
    Jul 2007
    Location
    United States
    Search Comp PM
    +1 to redwudz suggestion of an external box.

    I've been using the non-RAID version Mediasonic Probox(es) for years http://="https://www.amazon.com/Mediasonic-ProBox-HF2-SU3S2-SATA-Enclosure/dp/B003X26V...dp/B003X26VV4/ and highly recommend them. There's also RAID versions, though I haven't used them.

    At least in the U.S., 8tb HDDs are the sweet spot for price (<$20/tb). As discussed in numerous threads, at the risk of voiding the warranty, it's highly recommended to remove the drives from the case due to issues with overheating and USB interface failure.*

    *The WD Easystore drives are really easy to open without breaking any internal tabs and at least through the end of 2017 contained WD NAS drives.

    Currently the only other viable options are optical discs (Blu-Ray) and cloud. Neither as convenient or cost effective and HDD storage.

    Also, BACKUP, BACKUP, BACKUP! I've had multiple gigs of data disappear in an instant many times and while it's no fun restoring a backup, it's better than having to start from scratch.
    Quote Quote  
  4. What video format and what resolution are you using? I use DV to transfer and store VHS video. It is a constant 13 GB/hour. Basic math says that your 8 TB drive will store 615 hours of video. That is over 300 2-hour cassettes, each filled all the way up. Perhaps your project is bigger than that.

    Many people now archive using long-GOP delivery formats, like h.264, and you can actually get very good quality at less than half this size, which means you would get well over 1,000 hours onto 8 TB.

    So, it would be interesting to know how much space one hour of your video is consuming, and if it is more than these numbers, perhaps you should reconsider your workflow.

    And yes, without backups, you have nothing, because eventually the primary will fail.

    Without knowing how many hours or how many VHS cassettes are involved, my final recommendations may not make sense, but for me, I always make a DVD of everything, because that is the closest thing to true archival digital media that we have. It could last 100+ years. For VHS, you can easily get 1.5 hours on a single layer DVD, and with low-motion content can stretch that to two hours, if you have a good multi-pass MPEG-2 encoder.
    Quote Quote  
  5. Mod Neophyte Super Moderator redwudz's Avatar
    Join Date
    Sep 2002
    Location
    USA
    Search Comp PM
    I used DVD and DVD DL, then BD and DL BDs discs for quite a while for backups. But then I had a lot of discs to store.
    Lately, I've been backing up to external USB 3 drives, usually the WD 3.5" externals. Not the best for security,
    but most of my backups are from my DVD/BD collections, so I could just convert them again.
    For more important video/data, I still use the DL BDs as they are seem most effective per GB of storage, more so than DVDs, IMO.
    Quote Quote  
  6. Everything I say is false koberulz's Avatar
    Join Date
    Oct 2006
    Location
    Australia
    Search Comp PM
    lingyi, that link doesn't work.

    "BACKUP, BACKUP, BACKUP!" is obvious, but I'm not sure the best way to go about it. Making multiple copies of dozens of terabytes on a daily basis seems suboptimal, for example.

    john, I have a too-large-to-count number of tapes (definitely well over 300), at between 1.5 and 3 hours. Capturing Lagarith, and also doing restoration work. Some tapes require multiple captures. I intend to retain the original capture in Lagarith for flexibility and the uncompressed result of my AviSynth and VDub processes for cross-platform compatibility of the Premiere files that will create the final result.

    They're all basketball games, so no low-motion content here.
    Quote Quote  
  7. Capture and short term storage: a lossless format like UTVideo
    ~4 Bits/(Pixel*Frame)

    Long term storage: a "near-lossless" format like H.264 CRF 14, preset 'medium'
    ~0.3 Bits/(Pixel*Frame) (less than 1/10 the size of lossless)

    If you need to go back and edit the footage, re-convert to lossless.
    Quote Quote  
  8. [QUOTE=lingyi;2516962As discussed in numerous threads, at the risk of voiding the warranty, it's highly recommended to remove the drives from the case due to issues with overheating and USB interface failure.*[/QUOTE]

    Here's a timely note from the Federal Trade Commission. Opening cases, replacing parts, breaking "Warranty void if broken" seals, etc. do not void warranties.
    FTC Staff Warns Companies that it is Illegal to Condition Warranty Coverage on the Use of Specified Parts or Services

    In fact, those warranty void stickers may actually be against the law:
    FTC Warns Manufacturers That 'Warranty Void If Removed' Stickers Break the Law
    They that give up essential liberty to obtain a little temporary safety deserve neither liberty or safety.
    --Benjamin Franklin
    Quote Quote  
  9. Everything I say is false koberulz's Avatar
    Join Date
    Oct 2006
    Location
    Australia
    Search Comp PM
    What would you do with it after removing it from there anyway? Use it as an internal? I don't have that many bays and portability is important.
    Quote Quote  
  10. Member
    Join Date
    Jul 2009
    Location
    United States
    Search Comp PM
    Get a drive dock, something like this https://www.newegg.com/Product/Product.aspx?Item=9SIA1DS28Y2007 and use regular internal drives. If it is only for storage and not while editing then you can user slower and cheaper drives.
    Quote Quote  
  11. Member
    Join Date
    Jul 2007
    Location
    United States
    Search Comp PM
    Here's a corrected link to the Probox: https://www.amazon.com/Mediasonic-ProBox-HF2-SU3S2-SATA-Enclosure/dp/B003X26VV4/ref=sr...rds=mediasonic One of the nice things about his line is that they can be used as USB 3.0 or eSata.

    For portability, there are external cases with fans for better cooling or SATA to USB adapters and HDD carry cases (individual and multiple).
    Quote Quote  
  12. Member
    Join Date
    Apr 2018
    Location
    Italy
    Search Comp PM
    Hi,
    the most convient price/performance solution is to buy external hard disc (2.5 inches) with 4 TB disc to be connected by usb. Keep them twice: every file should be save twice on different device.
    Usb is quite universal and I suppose could be last a couple of decades. So I think you'llread them within 20 years.
    Quote Quote  
  13. Everything I say is false koberulz's Avatar
    Join Date
    Oct 2006
    Location
    Australia
    Search Comp PM
    I've got through about 5% of the project on this 8TB drive, I doubt a 4TB drive is the solution.
    Quote Quote  
  14. Dinosaur Supervisor KarMa's Avatar
    Join Date
    Jul 2015
    Location
    US
    Search Comp PM
    Originally Posted by koberulz View Post
    I've got through about 5% of the project on this 8TB drive, I doubt a 4TB drive is the solution.
    Well at that rate you can either get a total of 160TB of HDD space, or you can go for LTO-6 tape drives. LTO-6 is usually around half the price of HDDs but the investment in the tape drive is the most expensive part usually, I've seen deals on ebay for LTO6 and LTO7 drives for below $1000. You would also need a proper SAS card connected to the motherboard via PCI-E, and probably 3 HDDs in RAID0 to be able to feed the tape drives a constant ~300MB per second.

    Personally I don't have this level of data needs and so just use single layer BD-R for cold storage, with added par2 data.
    Quote Quote  
  15. Member
    Join Date
    Jul 2007
    Location
    United States
    Search Comp PM
    Originally Posted by koberulz View Post
    lingyi, that link doesn't work.

    "BACKUP, BACKUP, BACKUP!" is obvious, but I'm not sure the best way to go about it. Making multiple copies of dozens of terabytes on a daily basis seems suboptimal, for example.

    john, I have a too-large-to-count number of tapes (definitely well over 300), at between 1.5 and 3 hours. Capturing Lagarith, and also doing restoration work. Some tapes require multiple captures. I intend to retain the original capture in Lagarith for flexibility and the uncompressed result of my AviSynth and VDub processes for cross-platform compatibility of the Premiere files that will create the final result.

    They're all basketball games, so no low-motion content here.
    It takes ~14-18 hours to backup 6TB of data from from drive to another, ~100-130MB/s in my USB 3.0 enclosure, slightly quicker with both drives on internal SATA. This is in line with others report online. The bottleneck being the mechanical nature of regular HDDs. Seagate Archive drives take about 50% longer because of their slow write speeds.

    Assuming you're generating several terabytes of data a day (given capture and editing time), you could set backup the drive overnight or upon completion of a project, set up that drive for backup while using a third drive for the next project. Rinse and repeat.

    The only alternative I can think of is RAID (1 or 10) which will give you an "one the fly" copy, but is no better than a single drive if data is corrupted. You'll still need a backup solution (ideally at least two, one local and one offsite), so this is a costlier solution than the one above.

    Unless something drastically changes in the very near future, your choices are HDDs for capture and HDDs, Blu-Ray or LTO for backup. I put cloud storage as secondary backup because of upload transfer speed.
    Quote Quote  
  16. Member
    Join Date
    Aug 2010
    Location
    San Francisco, California
    Search PM
    Originally Posted by raffriff42 View Post
    If you need to go back and edit the footage, re-convert to lossless.
    I don't think you can become a virgin again.
    Quote Quote  
  17. Member
    Join Date
    Jul 2007
    Location
    United States
    Search Comp PM
    Originally Posted by JVRaines View Post
    Originally Posted by raffriff42 View Post
    If you need to go back and edit the footage, re-convert to lossless.
    I don't think you can become a virgin again.
    They have surgeries for that now!
    Quote Quote  
  18. It's not about being a virgin again, it's about having "good enough" quality for long term storage -- this is VHS footage we're talking about, and H.264/CRF 14 is plenty good enough.

    Converting back to lossless is done simply because intra-only lossless formats work a lot better in an NLE -- they seek better and decode faster than long-GOP video does. (to state what should be very very obvious)
    Quote Quote  
  19. I still think, with this much material, and with the original quality being VHS, that the OP really ought to re-think using lossless (Lagarith) compression and, if I read his posts correctly, also rethink storing uncompressed. I would use neither of these things. If I wanted to store something that is editable, and can be edited several generations, then Cineform is clearly the best option. I say "clearly," because this format was developed specifically for having great editing speed on an NLE timeline; for have extremely small initial degradation; and for having extremely low generational loss when saved (re-compressed) multiple times.

    What's more, it used to cost several thousand dollars, but is now free, via the GoPro site (they purchased the Cineform company).

    I would strongly recommend taking 1-2 minutes of sample footage and then save it using uncompressed, Lagarith, Cineform, and h.264. Make sure to save using the same color space so there is no detectable color shift. Then, find at least three willing souls to look at thirty seconds from each saved video, preferably on a big screen, and have them vote which is the best. I expect that no one will be able to tell the difference between any of them even, possibly, the very lossy h.264.

    The amount of storage you are talking about is absolutely massive and, having archived a LOT of sports footage (1,000+ hours), I can tell you that NO ONE will ever watch it all, so it is crazy to spend this much money, and take up considerable storage space, for something where only small bits will ever be looked at closely.

    I am not arguing to not do the project, but instead try to come up with a solution that is appropriate for what you are doing. VHS just doesn't call for this workflow.
    Quote Quote  
  20. Everything I say is false koberulz's Avatar
    Join Date
    Oct 2006
    Location
    Australia
    Search Comp PM
    I don't think I could find three willing souls who can tell the difference between SD and HD, comparing lossless and near-lossless video would be a waste of time.

    Here's my workflow:
    Capture to Lagarith
    Restore using AviSynth and VDub, with Lagarith intermediate files where necessary
    Export to uncompressed to bring into Premiere (needs to be compatible with both Mac and Windows, and properly treated as YUV by Premiere CS6)
    Color-correction in Premiere, along with overlaying graphics, editing out commercials, etc
    Export to Lagarith AVI, 576p50 (QTGMC used to double-rate deinterlace during restoration step)
    Use MeGUI to generate x264 files in 576p25 (square-pixel), 576i25 (anamorphic), 720p50 (square-pixel)
    Can also go to DVD or Blu-ray via the latter two AVS files in either MeGUI or HCEnc
    Delete any intermediate Lagarith files
    Quote Quote  
  21. Dinosaur Supervisor KarMa's Avatar
    Join Date
    Jul 2015
    Location
    US
    Search Comp PM
    You capturing your tapes in YUY2?
    Quote Quote  
  22. Everything I say is false koberulz's Avatar
    Join Date
    Oct 2006
    Location
    Australia
    Search Comp PM
    Yes.
    Quote Quote  
  23. Dinosaur Supervisor KarMa's Avatar
    Join Date
    Jul 2015
    Location
    US
    Search Comp PM
    Originally Posted by koberulz View Post
    Export to Lagarith AVI, 576p50 (QTGMC used to double-rate deinterlace during restoration step)
    Use MeGUI to generate x264 files in 576p25 (square-pixel), 576i25 (anamorphic), 720p50 (square-pixel)
    Can also go to DVD or Blu-ray via the latter two AVS files in either MeGUI or HCEnc
    Delete any intermediate Lagarith files
    Well if you are keeping the x264 versions as your final masters then there really is nothing more you can do besides use a higher CRF (lower bitrate) to save space. Not sure if you consider your final Lagarith copy to be an intermediate and if you are keeping those too, if so then you might want to delete those if you are creating duplicate lossy x264 copies. Otherwise you simply need more storage, whether it be HDD/LTO/BD-R.
    Quote Quote  
  24. Member Cornucopia's Avatar
    Join Date
    Oct 2001
    Location
    Deep in the Heart of Texas
    Search PM
    Originally Posted by koberulz View Post
    I don't think I could find three willing souls who can tell the difference between SD and HD, comparing lossless and near-lossless video would be a waste of time...
    Hahaha!

    My mother can tell the difference between SD & HD, and she's 98.

    Scott
    Quote Quote  
  25. Everything I say is false koberulz's Avatar
    Join Date
    Oct 2006
    Location
    Australia
    Search Comp PM
    I feel like half the time I've spent watching TV with others has been spent trying to steal the remote and change to the HD channel.

    Originally Posted by KarMa View Post
    Originally Posted by koberulz View Post
    Export to Lagarith AVI, 576p50 (QTGMC used to double-rate deinterlace during restoration step)
    Use MeGUI to generate x264 files in 576p25 (square-pixel), 576i25 (anamorphic), 720p50 (square-pixel)
    Can also go to DVD or Blu-ray via the latter two AVS files in either MeGUI or HCEnc
    Delete any intermediate Lagarith files
    Well if you are keeping the x264 versions as your final masters then there really is nothing more you can do besides use a higher CRF (lower bitrate) to save space. Not sure if you consider your final Lagarith copy to be an intermediate and if you are keeping those too, if so then you might want to delete those if you are creating duplicate lossy x264 copies. Otherwise you simply need more storage, whether it be HDD/LTO/BD-R.
    Yes, the Lagarith out of Premiere is deleted.

    I've also received the suggestion of MagicYUV for VDub-to-Premiere for multi-OS compatibility.

    But yes, I'm not here to find out how to fit this whole thing into 8TB. I'm asking what the storage options are well above that, and backups as daily backups would mean I spend more time backing up than working. Creating a new backup each time is incredibly space-expensive, but overwriting the one backup repeatedly doesn't seem ideal either as you're basically deleting your backup every day and if a file gets corrupted without you noticing or something...
    Quote Quote  
  26. >I'm asking what the storage options are well above that,
    Which people have been giving you

    >and backups as daily backups would mean I spend more time backing up than working.
    Overnight backups are a thing.

    overwriting the one backup repeatedly doesn't seem ideal either
    >as you're basically deleting your backup every day and if a file gets
    >corrupted without you noticing or something...

    That's why you need at least 2 backups in rotation (plus the original data)

    There is a saying:
    If you can't find the time to do it right, when will you find the time to do it over?
    (because you will lose at least part of your data at some point in this project)
    Quote Quote  
  27. You can't find a lossless codec that Premiere on both Windows and Mac can use properly? How about editing out the commercials in the second stage of your workflow in Vdub? (Could you not do the colour correction at the same time?) Seems like an awful lot of work for basketball games on VHS, but each to their own.

    As to your assertion that you don't think you could find three willing souls who can tell the difference between SD and HD, well, either stop tormenting your blind friends with stupid questions, or get a decent screen! Seriously, I'd be interested in seeing a small sample of your finished work.
    Quote Quote  
  28. Everything I say is false koberulz's Avatar
    Join Date
    Oct 2006
    Location
    Australia
    Search Comp PM
    Originally Posted by raffriff42 View Post
    >and backups as daily backups would mean I spend more time backing up than working.
    Overnight backups are a thing.
    Someone mentioned 14-18 hours per backup earlier.

    overwriting the one backup repeatedly doesn't seem ideal either
    >as you're basically deleting your backup every day and if a file gets
    >corrupted without you noticing or something...

    That's why you need at least 2 backups in rotation (plus the original data)
    Does that not just give you one extra backup period before you run into the same problem, though?

    You can't find a lossless codec that Premiere on both Windows and Mac can use properly?
    Premiere is bad at YUV, apparently, but every lossless codec I'd seen mentioned prior to MagicYUV was one OS or the other. Haven't tried Magic yet, but I might give it a shot next time I get to that stage of the process.

    How about editing out the commercials in the second stage of your workflow in Vdub? (Could you not do the colour correction at the same time?)
    I much prefer Premiere's color-correction tools, and editing of commercials includes inserting fades to black, and can also mean adding graphics, inserting music, etc.
    Image Attached Files
    Quote Quote  
  29. Dinosaur Supervisor KarMa's Avatar
    Join Date
    Jul 2015
    Location
    US
    Search Comp PM
    How were you cleaning up the VHS drop outs/errors in the 1982_GEE file?
    Quote Quote  
  30. Originally Posted by koberulz View Post
    1984_GEE.mp4
    1984_MEL.mp4
    You're encoding interlaced video as if it's progressive?
    Quote Quote  



Similar Threads