Business Server - Data Backup Methods

[cptwhite_uk]

This is just a general question about data backup methods. I work for a medium sized business, with maybe 40 people in the offices and we use one central server for all our data storage. Currently holding about 0.5Tb of information we need about 1Tb of storage capacity for future growth. The weekend saw one of the hard drives in the server die apparently and we are now resorting to "backup tapes" which are saved every night to recover the lost data. However this takes multiple tapes and as of Tuesday morning they're still restoring the data and I'm still without e-mail / access to our files.

This basically means we cannot produce jobs or receive them via e-mail. Even though it's likely we'll recover the data the down time is a considerable problem. We don't have an IT guy as such - the company buying dept. manager oversees the server and while he's more knowledgeable than me on server lingo and hardware I just don't get the impression he's particuarly confident or as fluent with PCs as me in general.

This makes me believe we probably aren't using the best method of data back-up that's available. So what kind of set-up should a company like ours be using - surely there's a way to backup the server without such long down time periods. By the way this is about the 3rd crash in the last 12 months with considerable down time.

My limited knowledge in this area just makes me think RAID1 array - is this a stupid idea for a server, or maybe a PC on the network in another location that simply keeps a backup of all the data on the server in real time, just like a RAID1, but a different physical location would ensure it's less likely to be taken out if the whole server were to go - thus if a hard drive failure occurs you simply swop drives.

Am I talking out of my arse, am I mad? Can someone enlightne me

[Splash]

If you're having significant downtime that regularly I'd be asking why you don't have somebody (internal or external to the company, there are plenty of managed services companies that will look after it) to look after it properly? A server without proper hardware support should not be in a production (hell, I'd be dubious about using it in test) environment.

It's not the answer you want, but it's the harsh truth - somebody somewhere needs to realise that a server that has been down 3 times in 12 months is costing money. A maintenance contract with a decent SLA (or a new server with a nice 3 year onsite warranty) would probably have cost less than one period of downtime.

EDIT - Also (having just re-read your post) I should point out that while RAID1 will allow you an amount of redundancy (if it literally is a disk failure) if the problem is something else it won't help you - corrupt files are corrupt on both disks in the array. RAID is not backups, and should never be used as a substitute. Do you keep offsite backups?

[mikerr]

I'm always amazed as how much time and money companies lose to IT, when it could all be avoided with a little investment.

What annoys me is that 40-employee company probably has 3-4 accountants, but no full time, or even outsourced IT bod :/

Joe from the office who knows how to use a mouse better than most, is not an IT bod and will be lost on a server....

Always check how long your backups will take to restore too... many incremental tape systems would take days to restore ....

Time for the good ole tao of backup link

[Splash]

It's mostly because IT is invisible when done well, so bean counters like to think it doesn't exist. A salesman makes some big sales and gets a big bonus, a sysadmin runs a faultless network and... so he should!

[cptwhite_uk]

I should have been more clear we are serviced by an IT company who are involved with the maintenance, to what extent I'm not sure - server upgrades, data backup support, but it's our guy who seems to do all the running about fixing it while they're generally directing from the other end of the phone. That's my perception anyway

The server just seems notoriously unreliable. We do deal with potentially large files which I believe are e-mailed???!?!

We need to set-up some kind of Anon FTP server for people to upload files too I believe. Thanks for the responces

[gss03]

Has the IT company servicing this down'd server visited to assist?

I would be screaming down the phone till someone came onsite and took control of this problem and had an idea of what went wrong and how to fix it!

Is the server a Dell/ HP server with any warranty?

[Saracen]

Any backup system is a case of balancing risk and inconvenience against cost.

RAID is not intended as a backup solution. It's about resilience. It's about keeping that server up and running, albeit perhaps with a performance hit, if a hard drive goes down. But there are things that will take multiple hard drives in one machine down. I know - it happened to me.

So any sensible backup strategy is about a mix of layers. It has to be designed in such a way that the admin overhead isn't too high, or procedures get ignored. It has to be sufficient to protect against an adequate level of risk. And the cost has to be proportionate to the degree of risk. And that risk involves determining what happens to the company in a given event.

You can have duplicated servers, with every single disk write duplicated on the backup. You can have automatic duplexing, with fallover to the backup if the primary goes down, with email/SMS warnings to the tech person when it happens. But if you put both those servers n the same room (or building) and it burns down, where's the backup?

You can backup those servers to tape (or hard disk, or optical disk, or ... etc) once a day and keep that backup in a fire-proof safe. And if you get a flood, are the tapes/disks water-proof? Is the safe?

You can keep the backups off-site. But does anyone ever test that the backup can actually be restored? How old are the tapes? Are they ever tested/replaced? An off-site backup isn't a lot of help if, when the need arises, it fails to restore because the tape is old, or corrupted because whoever took it home rested it on top of their car's subwoofer every night on the way home.

You can have insurance policies, or you can have a completely separate data centre. You can put decent UPS everywhere, and find a PSU fails. You can have standby generators, and have them break down ... or get flooded.

It's very difficult for anyone to suggest a "best" solution, because the "best" will almost certainly be so expensive as to be unjustified for a small company. Instead, it's about assessing what risks your company faces, and coming up with sensible, practical and affordable solutions for addressing those risks ....., but knowing where to stop.

And it's about decent insurance.

[Splash]

Sounds to me like you need to either

a) upgrade your server support contract or
b) get yourself a server support contract with another company

You should be reporting a fault like this, they should be sending someone to fix it. End of. I suspect what has actually happened is either that nobody has pointed out how much this downtime is costing your company OR that somebody has, and a bean counter doesn't think that an enhanced support package is value for money.

Interestingly as I view this page there's a banner for Arcserve r12 at the top.

Also - someone might want to do some form of risk assessment before putting an entirely anonymous FTP server up...

[peterb]

you didn't say how often backups were taken, or what the tape medium is.

lto is a very effective medium for backup - and fast - but not cheap to buy upfront 9but worth t if it means downtime is reduced)

RAID 1 or 5 would certainly improve resiliance, but as others have said, it is NOT a substitute for a proper backup strategy.

If the server has failed 3 time is a year - I would be asking what failed n it - and again I would be looking hard at my support contract.

[cptwhite_uk]

Well unfortunately it's not my place to critisise However I think this has been an interesting read so many thanks for your thoughts Splash / Saracen. For your info. an IT bod has been out this afternoon.

[Saracen]

.....

Also - someone might want to do some form of risk assessment before putting an entirely anonymous FTP server up...

Amen to that.

[Stoo]

For what it's worth, this is the system more or less that we use on some of our internal systems:

Server: 5 disk RAID 5 array

Backup Server: 5 disk RAID 5 array

Tape Backup: LTO-4 (1.6TB compressed) daily backup

Weekly off-site tape rotation (every week we shift all the backup tapes to an off-site DR centre, and bring the previous weeks tapes into the building for re-use)

Not to mention the odd ad-hoc backup before you make any big changes to anything.

That's for a company of ~250 employees, and our e-commerce section takes in ~£4million a month

But as Saracen said, it's all about the level of risk you're both willing, and financially able to take

oh yeah, anything can and will happen - we had one of the tape drives fail and chew up one of the tapes, which both knocked out the tape drive and the ability to backup for a couple of days before the engineer could come out and fix it.. we now have 2 tape drives, and plenty of spare, new tapes

[badass]

You need to talk to your IT services company. They will be in a far better position to advise you. With 40 employees and that much data your service contract should at least involve a monthly half day site visit from one of their engineers. If you have that facility, nab them then and get some advice