Looking for a 10+ Hard Drive Storage Solution

[holygamer]

I do a lot of video restoration. Once I've converted my videos I put them on a 4TB USB 3.0 hard drive (or whatever is the largest & cheapest hard drive for the money). Once a hard drive is filled, I rarely access the drive again. The problem is that USB 3.0 isn't exactly 10 times (or whatever) faster than USB 2.0. I get around 60 - 110 MB/s and the technology itself isn't very reliable (windows 8 has lots of issues for example). On a couple of occasions I've noticed some files missing on my USB 3.0 drive even though I used the safely remove tool. Also I've got loads of power bricks in my room. eSATA III is faster and doesn't have any reliablilty problems as it's just like using an internal SATA III drive.

eSATA III drives don't seem to exist (at least not in an affordable way). NAS's are expensive and only support a few drives - I want something that is infinitly expandable.

The 4TB hard drives I'm using were cut out of the Hitachi Touro Desk DX3 4TB USB 3.0 Hard Drive case. I noticed that the enclosure didn't have a fan in it.

Possible Solutions

If you have any suggestions then please let me know. Here are my thoughts on the issue:

NAS: I don't want to use a NAS because that means the drives will always be ON and will wear out after several years (please tell me if this is wrong). However eSATA drives will last a lifetime as I only connect them to my PC on rare occasions to perform backups.

eSATA III Rack Solution: Two eSATA III cables from the back of my PC into a Highpoint RocketStor 5322 SATA III twin hard drive dock. I would then connect a combined power & sata cable to the dock and the other end would be connected to my SATA III hard drives which could be inside an open hard drive rack like this: http://www.span.com/product/5-Bay-HD...rd-Drive~27840 £19 for a 5 bay rack and it's stackable. Because it's an open rack I can easily connect the combined power & sata cable to each hard drive without needing to open the case.

Is it safe to have hard drives in the open air (in my bedroom) like on that rack? I assume that I won't need a fan on the rack since the hard drive inside the USB 3.0 enclosure didn't have a fan on it?

OR I could avoid using the Dock altogether and just connect the eSata cables from my PC to my SATA III hard drives on the rack and connect a power supply to the rack?

Tower Solution: I could get a 9 bay tower case: http://www.span.com/product/9-Bay-5-...-2xPorts~26264 and a power supply to connect the hard drives to. I'm not sure how the sata cables would fit out the back of the case though.


Questions

If a hard drive is connected to a power supply which is always on but the SATA cable isn't attached to anything then does that mean that the hard drive will NOT suffer from wear and tear because the drive isn't being accessed?

If I had a power supply which was only used to power hard drives. What's the maximum number of hard drives I could power with one power supply? If I'm only accessing 2 drives at once when I plug the sata cables into my dock does that mean I can connect as many hard drives as I want to a power supply - and power is only used by 2 drives at once?

[Agent]

You can simply switch a NAS on and off as you need to access it. There is no requirement for it to be on all the time (no access to the files though ).
WOL solutions is another option and one of the things I'm planning to use. Keep the system in S3 / whatever a lot of the time, and then just send the magic packet to wake it up

It depends how serious you are about your data as to what solution you take. Some people go with a Microserver, ECC RAM, a filesystem like ZFS - where I'm personally likely to just have a couple of drives that are mirrored (and I don't mean RAID, but rather scheduled each night). Important stuff that's not confidential gets mirrored to the cloud overnight in a Truecrypt volume for extra safety.

[Splash]

Couple of points: HDDs are mechanical devices, all will fail eventually, however they tend to exhibit a bathtub curve failure rate (ie you would normally expect a failure reasonably quickly, or they last ages). With the amount of data you're talking about you should *definitely* be planning to accommodate those disk failures, regardless of how you do so. You've not mentioned any form of RAID, so I'll start by recommending a solid RAID controller.

As far as a solution goes... I'm not sure you know what you want. A half decent NAS will have power save functionality to spin disks down when not in use, and NAS grade disks will be fine with 24/7 use anyways (though again: plan for failure, and have a raid array with a cold spare in a drawer and ideally a hot spare ready to start rebuilding when a disk drops out.)

How much *actual* storage do you require, and is this data the kind of thing that you need to backup (and if so how have you thought of that? all those disks in a single box is no good if a power surge takes out the controller and a few disks, nor is it any use if your house burns down), or is it something where you need a lot of storage short term, then the space is freed up once you have finished a project?

What degree of performance do you need? You'll generally get better performance with a larger number of smaller disks than a smaller number of large disks, assuming that they make up the same amount of usable storage (and that you're writing/reading across those spindles and you're not having to calculate parity)
Maximum number of disks you can power will depend on the quality of the power supply, how much juice it can handle and the number of connectors on the cabling (though you can buy splitters). As for leaving a disk powered but not data connected I'm pretty sure that's a bad idea: the SATA connector isn't necessarily hot-pluggable.


So there we go: more questions than answers I'm afraid, but fill in the gaps for us and we'll do what we can to help.

[holygamer]

What degree of performance do you need? You'll generally get better performance with a larger number of smaller disks than a smaller number of large disks, assuming that they make up the same amount of usable storage (and that you're writing/reading across those spindles and you're not having to calculate parity)
Maximum number of disks you can power will depend on the quality of the power supply, how much juice it can handle and the number of connectors on the cabling (though you can buy splitters). As for leaving a disk powered but not data connected I'm pretty sure that's a bad idea: the SATA connector isn't necessarily hot-pluggable.

I have an internal hard drive that is my project drive. I do restoration on my videos on that drive then I put the originals and the restored videos on a 4TB or bigger external hard drive once a month. The external hard drive is filled within a year then the drive will rarely be accessed again so it should last a lifetime. The only time I will be accessing the hard drive again is when a successor to Bluray comes out every 10 years or so as I'll need to re-encode the videos to a new format.

Whenever I backup something to an external drive, I also manually back it up to another drive.

I have over 10 4TB hard drives and the number grows all the time so I don't think I can use a NAS as they only seem to support a few drives and they're quite expensive. And honestly I don't know anything about NAS, servers and RAID, etc.

Note sure what you mean by performance but I need to use 4TB drives as they give the best value for money to TB ratio and I want SATA III or Gigabit Ethernet transfer speeds.

Could you please clarify a few things for me please.

1. Would the open rack I mentioned be safe, I would have it in my bedroom. Just wondering if it's safe to have a hard drive in open air like that?
2. I'm using Windows 7 on a recent motherboard - eSATA is hot-pluggable isn't it? I would like to have my SATA III hard drives on a rack beside my PC. A power supply would be connected to the drives on the rack. When I want to access a drive I would just connect a Sata cable from the drive to the eSATA hard drive dock or directly into the eSATA ports on my PC. Is that OK or would that cause problems?
3. Also does wear and tear occur to a drive if I have it connected to a power supply but not data connected via a sata cable to my PC?

[peterb]

The external hard drive is filled within a year then the drive will rarely be accessed again so it should last a lifetime.

Whenever I backup something to an external drive, I also manually back it up to another drive.

I have over 10 4TB hard drives and the number grows all the time so I don't think I can use a NAS as they only seem to support a few drives and they're quite expensive. And honestly I don't know anything about NAS, servers and RAID, etc.

Note sure what you mean by performance but I need to use 4TB drives as they give the best value for money to TB ratio and I want SATA III or Gigabit Ethernet transfer speeds.

Could you please clarify a few things for me please.

1. Would the open rack I mentioned be safe, I would have it in my bedroom. Just wondering if it's safe to have a hard drive in open air like that?
2. I'm using Windows 7 on a recent motherboard - eSATA is hot-pluggable isn't it? I would like to have my SATA III hard drives on a rack beside my PC. A power supply would be connected to the drives on the rack. When I want to access a drive I would just connect a Sata cable from the drive to the eSATA hard drive dock or directly into the eSATA ports on my PC. Is that OK or would that cause problems?
3. Also does wear and tear occur to a drive if I have it connected to a power supply but not data connected via a sata cable to my PC?

Snipped a couple of bits

It is a dangerous assumption that a drive left on the shelf will last a life time - as a mechanical device, it could still fail - there are lubricated bearings in there, they can degrade over time, even if not in use - and go course it only has to be dropped.

Backing up to a second hard drive does mitigate against that of course, but it might be wise to spin up the drives every (say) 3 months or so.

RAID is not a backup solution in itself, but it does provide some insurance against a single drive failure. RAID1 provides mirroring, so the contents of one drive are duplicated on a second. It doesn't prevent data corruption, or single point failures of things like the controller, but the hard drive is probably the device most likely to fail in a computer system.

No electrical or physical reason why a drive can't be left in the open - apart from a slightly increased risk of physical damage (or spilling coffee over it! )

eSATA is only hot pluggable if the driver controller is designed for hot plugging.

If the drive is spinning without the data cable connected (and it probably will be) then there is wear and tear occurring.

I don't know how valuable your data is (or your budget0 but a similar system I built for someone as a server has 4 $TB drives as two mirrored RAID arrays of $TB each (total 8TB) and a separate drive for the operating system. (It isn't the most efficient use of the drives, but for this person, resilience was the most important criteria)

The backup solution is to tape - which are very resilient, the unit cost is low per tape, so you can make multiple backups, but the downside is that the initial cost of a tape system is very high, and of course is an offline system. But it is infinitely expandable.

In your case, I'd be thinking about a 4TB RAID 1 array for work in progress, backing the finished video to tape (and maybe a hard drive as well for quicker access) But you are looking at relatively big bucks for tape systems.

[shaithis]

Questions

If a hard drive is connected to a power supply which is always on but the SATA cable isn't attached to anything then does that mean that the hard drive will NOT suffer from wear and tear because the drive isn't being accessed?

If I had a power supply which was only used to power hard drives. What's the maximum number of hard drives I could power with one power supply? If I'm only accessing 2 drives at once when I plug the sata cables into my dock does that mean I can connect as many hard drives as I want to a power supply - and power is only used by 2 drives at once?

You can set a drive to spindown. Most NASes and NAS-software allow the configuration of this. There are a number of settings for granularity in the advanced power management spec.

The max number of drives will be determined by the PSU. If you plug into a drive, it will try and power up.

For 10 drives I would definitely recommend a NAS unit (Synology is my favourite but even the 8 bay is quite pricey) or build your own and install FreeNAS/OMV/OpenFiler/Windows Server

[jim]

If you want to put drives on a shelf, then all you need is a dock to put the media onto them.

However, your assumption that they'll last forever is very risky. I've had files corrupted on shelved drives before, and had to restore them from duplicate. It's all a question of duplication. I do use a large number of shelved disks for long-term archival purposes, but I always have multiple copies of the files.

I typically have one set live on a server, one set safe in a cupboard, updated whenever I can be bothered, and one set safe at another address, updated maybe every 2-3 years so that in the event of fire or whatever I've still got the archival stuff to go back to.

[herulach]

Just a couple of points to add to what peterb has said. Firstly, don't worry about sata 2/3 with whatever solution you end up with. 4TB WD Reds have a sequential write speed only marginally faster than SATA 1. SATA 2 is barely a restriction on even the fastest SSDs.

Secondly, you really need to think about how important this data is, and really the rational for doing it. We don't know what the videos are, but presumably they started life as VHS or similar? In which case you should consider whether you're actually missing out on anything by archiving at BR quality rather than in raw or similar.

Certainly if its at all important, or completely irrecoverable then off site storage is essential, maybe you could consider backblaze or something like that for offsite in a compressed format?

Knocking together a pretty low power consumption server with capacity for 10 drives that spin up and down on demand isn't too difficult, and by having it in a separate machine you can comfortably have it automatically compressing and offsiting data whilst your editing machine is off.

[Agent]

Try to also be realistic with the importance of data too, as it can quickly get expensive.

Important documents, photos and such might well come to under 100GB. That is easily within a cloud hosting solution like Google drive (Again, I always Truecrypt it though) as well as a drive backup.

Music files are simply mirrored to another drive without any other sort of backup. Worst case - both drives die and I need to rerip them all from CD / download any digital purchases. Annoying, yes, but at that point it's a time vs money thing.

If you can separate it into 3 sections of importance, you'll find that backup solutions tend to scale reasonably well with it. Stuff I have on Steam simply isn't backed up and the directory is excluded from any backup images, for example. Again, download time vs money to backup. Re-downloading a game is an inconvenience. Losing work files could be job destroying.

[holygamer]

OK I think for now I'll just go with a rack as it's the cheapest and simplest solution. I'll have my hard drives in the rack then I'll connect the drives to a dock. I will keep mirror drives in my cupboard.

Can you guys recommend a rack with a bigger capacity than this or should I just go with this?:

http://www.span.com/product/5-Bay-HD...rd-Drive~27840

Can you suggest a cheap online backup service where my 4TB internal project drive is automatically backed up whenever there's a new or changed file. I have a 19 Mbps upload speed so it shouldn't be a problem.

[jim]

For your usage, I suspect Crashplan might be a good bet as they had unlimited storage. Not used their client though.