Triple redundancy setup for storing photos

Hey,

Basically I'm a photographer, and I take a ton of photos (can hit 1tb in a few weeks easily). As my images are my livelihood, I'm trying to make sure everything is backed up and stored securely.

At the minute I've just got a bunch of disks running on Raid 1, but I know this isn't the most secure method.

So essentially I'm looking to drop some cash and make a bomb-proof setup (not literally).

My current idea is to buy a 2 bay external drive, that will connect to my PC through e-sata. I'll run this in Raid 1. Then also purchase a 1 bay e-sata drive.

I'll use the drives in my computer for initial sorting and processing, then back the files up on to the 2 bay drive. The once a week I'll backup the 2 drive bay onto a single drive, using some sort of mirroring software. That drive will then be stored off-site in a fireproof safe.


I'm sure this setup would probably have some weak spots though. Or perhaps there's a more efficient way to do it? I was also thinking of buying a UPS incase there is some sort of problem with the electrics (not that it has ever happened before). Or would a special fused plug be good enough to protect me from power surges frying my equipment?

Thanks!
-agour

A UPS ain't a bad idea, especially if your building electrics are as shoddy as mine. The single disk for offsite backup strikes me as your weak point: if you can budget for it is it maybe worth buying either 3 or 4, and running your backup on a rotating basis? That way if the single disk fails when restoring your backup you can then recover from the previous weekly. Other than that... do you have some sort of versioning on your 2 bay device in order to be able to recover a previous version of an image? Also: how are you handling the catalogue (assuming Lightroom or something similar) - is this being backed up too? If not I'd strongly recommend you do so.


Other than that: it's ultimately about balancing cost and benefits. You could spend a fortune on an full-on archive solution, but one man's bomb-proof is another man's failboat :)



As an example for mine: images are stored on a network drive which lives on a Microserver running RAID10 (using an HP P410 RAID controller). The images volume is backup up to a second NAS which is configured in RAID5, and the backups are rotated once a week over 8 weeks. The images are also backed up to Spideroak for one set of offsite storage, as well as across a VPN to a single bay NAS running at both my parents and sisters houses. I'm not a pro photographer (hell, I'm not even an amateur) but these images have a lot of sentimental value to me - family members who have passed away and the like so I'm *really* keen that they not disappear.

Personally, I use a tape set up - tapes are cheap, pretty rugged and econmical.

The downside is that the initial investment in the tape drive is high. Current sweet spot is LTO4 , but if you can afford it, LTO6 is the current generation with over 1T storage per tape, and pretty fast write speeds via an SAS interface card.

A tape drive and SAS card will set you back about £1500 (inc VAT), tapes are around £30 for 6TB storage (compressed - depending on the format, the images may may not compress much - although it is a lossless compression algorithm) but uncompressed tape capacity is 2.5TB.
Compression is hardware based in the drive. Tapes are smaller, cheaper and more rugged than drives, so makes multiple off-site backups a realistic option.

Have a look at

http://www.imagestore.co.uk/lto-and-...rnal-kit-6gbs/

I get the impression you're (original poster) quite focused on RAID. Don't be. If this is your day-to-day then you need enough redundancy to keep working, but when you're talking storage then it's backups you need to concentrate on, not RAID.

I would work out what you need to carry on working, and cover that with redundancy, be that by having a redundant hard drive in RAID, or even a redundant system which you can switch to in case of emergency. Then regarding your data backing up via a NAS is a pretty good idea - and rather than using RAID use back up and incrementals etc. Then if there's a problem with your working setup you can restore from latest backup on the spare system and carry on.

For serious back up tapes are hard to beat, as suggested by peterb above. Alternatively/as well you might want to consider if you can afford cloud storage, though I don't think it works out for the volumes you're talking about yet.

I'm not sure cloud is an option - apart from anything else, you are relying on a service over which you have no direct control.

But as Kalniel says - RAID is NOT in any way, shape or form a back-up method or a substiturte for backup - it is more about reliability and to some extent availability.

Splash - yeah I thought about the single disk backup thing after posting. I'll probably have two rotating disks instead.

The 2 bay devices I was looking at don't appear to have that option. I've not been using lightroom, everything is just stored away into folders and labelled appropriately. Most of my work is timelapses, so a few hundred photos might only be 1 sequence.. which makes it easier to catalogue. I should really look into using lightroom though.. especially as I already have it installed.

Haha yeah that sounds like a pretty full-on system! I'd love to have an off-site backup that's automated, but unfortunately my internet is so shoddy that it makes it impossible. Manually taking disks off site is the only way.

Peterb - That's a lot more than I wanted to invest right now, but it does seem like a really good option. Particularly as the disks are so cheap.

kalniel - yeah it's mainly day-to-day that I'll be using it for. That's why I was thinking of having working files on my pc, move them to the 2-bay once they are processed, and then back them up again onto the other disk once a week. So worst case scenario both drives in my pc die, losing 1-2 days work. Or the drives in my pc, and the dual bay dies, which loses a maximum of a weeks work.
Once the dual-bay is full, both drives (and one of the backups) will go in to long-term storage. Three drives in separate locations should be enough redundancy I think.

Cloud is definitely not an option, my connection isn't fast enough to send the files, and also the cost to store multiple TBs's is too high.


Gotcha about RAID, I just like the idea knowing that there's 2 drives in case of failure.

No, tape isn't cheap, and does require an investment - but look on it as insurance - and what is the value of the photographs you are storing, and how much would it cost to re-create them (assuming that you can) if they were lost?

But it is (as Splash said) a balance between risk, costs and budget. The problem with hard drives is that they will fail - you just don't know when, and they are fragile - drop one and there is a good chance it will sustain some damage.

Tapes are a great call if you can stomach the initial investment - you may be able to pick up a (relatively) cheap LTO4 drive second hand - they're pretty robust so long as they're cleaned regularly so you shouldn't have too many concerns about one failing on you.

To come back to the cloud though: you have concerns about your upstream bandwidth, but what is your rate of change of data (ie how much data would you expect to have to upload per day)? There's a decent chance that it's less than you might think, and you could host your own "cloud" storage at another premises by building a small Owncloud setup or similar which doesn't need to cost the earth.

As others have hinted at though: offline backups are where you should be focussing. Online backups/replicas/synch jobs are too susceptible to one corrupted file being replicated to all backups and at that point they're more or less useless without versioning (and you've probably seen the recent malware that encrypts data - including any network drives that it can access),

I started tape backups after a failure of my RAId1 setup on a linux server. fortunately I lost nothing, there was a problem with a drive connector (intermittent) and Linux disabled the array to prevent any data corruption. Frightening at the time, and after cloning both disks to aid any recovery, I eventually fixed the problem without any data loss or ned for data recovery - and thats when I bought a second hand LTO2 - when the latest generation was LTO4. apart from the drive, I had to get a SCI card and SCSI cable. The card I got second hand 9from a HEXUS member!) the cable was harder to find.

I have recently upgraded by buying a second hand LTO4 system, still using SCSI, although there are some SAS drives about.

You do need a fastish system - LTO drives like a steady supply of fast data - they are streaming devices and don't like frequent start/stop. LTO 5 and 6 drives are either SAS or fibre channel. SAS is plenty fast enough - and you can run SATA drives off them too, and the cables are less expensive.

If you can find a used LTO4 (preferably SAS) it would be a good entry point - but again be prepared to spend £600 or so. LTO4 tapes are cheaper, but the data capacity is less (800Gb native capacity).

There's perhaps another, slightly lateral, way to look at this.

I think perhaps you ought to consider the difference between "backup" and "archive".

For a backup solution, I'd be looking at something where, on whatever time basus data volumes require. I was rotating the media set. So, something like the grandfather-father-son sequencr, then overwrite the grandfather. That's fine for data, like perhaps accounting software files, that constantly changing and, beyond a certain point, is too old to be of much use.

BUT .... I have photo files from years ago. I'm a bit of a squirrel when it comes to my photo files. Once I've decided an image is worth keeping, I pretty much want to keep it forever.

So .... my "backup" strategy uses a blend of both backup and "archive". Regular, normal working data files get the "backup" treatment, but my photos, and for that matter, a document-image archive of scanned document images, get "archived".

And for the archive process, my criteria are that, first, it's for files where it's the original version, or ginal ginished version, I archive, and secondly, that archive medium must be capable of long-term storage, not degrading over time .... or not if it can be helped.

And, personally, I distrust anything fundamentally magnetic, be it hard drive or tape, for long-term storage.

So, I use a variety of optical, or magneto-optical methods. In my case, as data volumes aren't anything like TBs per few weeks, I get away with DVD-RAM and MO disks, designed for archiving, over long periods. The downside is that I needed to come up with a way of indexing archived files, so I can find them again, when I want them be it 6 weeks or 6 years later.

So here's the "lateral thinking" suggestion.

By all means have files stored locally on your machine. By all means have a copy on a second machine, be it another networked PC, or external HD, or NAS. I do that, and use Sync software to automatically copy from primary to secondary.

But, periodically, I also copy to archive disk. So, what about "archiving" onto Bluray write-once disks? They're only 25GB or 50GB at a time ( well, 100GB disks and dtives exist, but currently, very expensive.

A Bluray burner for your PC is, what, £60? And 50GB blanks are about 50p each. So a box of 50 gives you 1.25TB for about £25. And keep thos BD disks, offsite, in a fireproof safe.

It's a fairly low-tech solution. And it does require self-discipline on your part to, periodically, burn files to disk .... which I do as part of my process for taking inages off camera memory cards. And you MUST devise a ststem for naming, tracking and locating inages/projects on archive disk. So it's not a setup-and-forget option, but it does keep upfront costs to a minimum, uses easily readable media that ought to be very resilient if stored carefully, and gives you a long-term archive copy.

It's just a thought, but it's been working for me for about 20 years .... since my first CD burner when they were £4k a time, and 650MB blanks were £10 a piece .... though I've updated the technology I use about 3 or 4 times over the years, and have, as a result, yet to lose a single archived image. Not a single one. ;)