Raid 5 - Dead

[Vini]

Say i had a server with raid 5, and 2 of the HDD's decided to give an error, yet continue to work....

Then say i rebooted the server and then the server refused to boot, giving errors for:

2 Failed drives 'and/or' Unresolved mismatch between disk(s) and NVRAM.

What would you do?

[badass]

Say i had a server with raid 5, and 2 of the HDD's decided to give an error, yet continue to work....

Then say i rebooted the server and then the server refused to boot, giving errors for:

2 Failed drives 'and/or' Unresolved mismatch between disk(s) and NVRAM.

What would you do?

Replace dead drives, restore from backup.
Some of the data is most likely corrupted anyway.

[Vini]

aye thats where we're heading, disk replacements are en route.

[CrimsonAvenger]

Yup - once you have 2 dead drives you've lost the array

[directhex]

Yup - once you have 2 dead drives you've lost the array

unless you built the array with more than 1 drive's worth of parity data.

we had this happen once on a Sun T3 array. the data was gone.

[shaithis]

You can sometimes reboot a few times and a disk will come back to life for a while...

Also, some raid controllers will allow you to force the disks back online, this MAY work but dont count on it...

Depending on what exactly failed on the drives and wether the replacements are identical or not, you could try switching the controller (PCB) from one of the new drives over to one of the dead ones.

Had this happen once when our operations team wasn't monitoring our servers corrently, nasty business.

[CrimsonAvenger]

bit of a bummer for 2 disks to fail at the same time........

[badass]

Just to qualify my first statement further - I meant even if you can get one of teh disks back online, the data is still too lokely to be corrupted to make anything but a restore worthwile.
That said, if you do get one back online, back up the lot, even though its corrupted, and then restore from backups pre disks dying
At least if people are desperate for things between backups you can try to restore them.

[Vini]

right, well the 2 new hdd's we got wouldnt rebuild, understandible. once 2 disks are down the arrays gone.

we have the option to check consistency and rebuild manually, but im happy, as the 'urgency' doesnt seem to be there. i was expecting a fun fuelled weekend of file finding and server building. instead we've postponed until monday \o/

seems like the controller could be up the creek and not actually the discs. with everything crossed - this would be a preferable 'fault'.

[chinny]

I've seen this on a batch of HP DL380s (3 out of 4) and it turned out it was the raid controller causing it. Re-seated cables and it came back up fine - no problem with the disks.
Good luck!

[Vini]

it is a hp server, have been called in tomorrow to play.

after exploring the tape backup from the night before, it seems the most vital folder is missing, typical!

though the veritas log shows it as being backed up, and the folder size, shows it as being backed up. when looking to restore, it just isnt visible!

oh joys.