Is data recovery possible? Intel Matrix Raid failure

[mroz]

City Link did indeed try to deliver yesterday. According to tracking, I was carded. I'd love to get my hands on one of those cards, as they possess the attribute of invisibility. Anyhow the guy's recently turned up today, yay, so I have my order. I left a note on the door telling him we were in & to ring if he didn't get an immediate response. Apparently, he explained, that would be impossible as their phones don't allow outgoing calls. Makes me wonder why their depot never mentioned this during any of the previous twenty odd deliveries, when I've asked them to pass my phone number on to the driver in case he couldn't find us. Particularly in the couple of cases where someone claiming to be the driver did actually phone me, coincidentally ten or so minutes before the real driver turned up. Strange are the ways of the economy courier.

My new Spinpoint is now nestled in it's cosy Icy Box & slowly formatting itself into a sublime state of data recovery readiness.

I'll post back later when it all goes tragically wrong.

[mroz]

Having recovered most, if not all, of the data, rather than continue with rebuilding I thought I'd check out other possible recovery strategies while I had the chance - It's not everyday, I hope, that one has a broken RAID to hand.

Along the way I realised GetDataBack does allow saving of scan results, so the recovery can be continued even after the application is closed down. I've updated my earlier post to correct this error.

After my data was backed up I went into the bios & used it to delete/recreate the arrays. Booting back into XP I find all partition tables are missing from the array volumes, however Active@ Partition Recovery was able to recover them, so this looks like a promising fast strategy for recovery.

Unfortunately in my case the sequence of errors/problems which initiated the corruption were resulting in blue screens shortly after login, immediately prior to the RAID structure corruption. This situation was also, not too surprisingly, restored.

As I wanted to change my arrays, to increase allocation of 1 relative to 0 & as I had my data backed up, rather than pursue the BSOD further I simply deleted the arrays.

I'm now rebuilding. Another hour of formatting followed by the joy of XP installation & 20 to 40 hours of installing applications await me as punishment for my failure to back up.

[peterb]

======snip

I'm now rebuilding. Another hour of formatting followed by the joy of XP installation & 20 to 40 hours of installing applications await me as punishment for my failure to back up.

Ah yes! I had RAID 1 array - with the intention of backing up "sometime" and I hada memory failure that corrupted the file system. To make matters more interesting, the raid supported logical volumes.

However to cut a long story short, having investiogated data recovery companies, I then cloned the two drives and did a bit more investigating, and discovered that the OS kernel (Linux) had disabled the LVM - presumably as a safety measure, and once that was re-enabled, I could read the filesystem and repair it (it is journalled)

I now have a Ultrium 2 tape drive in one machine which I use as a backup server. Possibly a bit OTT - but very very good.

The lessons I learned were

1. Don't rely on hard drives - ever - even in a raid configuration
2. back up critical/irreplaceable data regularly - how regularly depends on your risk assessment of the value and how frequently it changes.
3. When it all goes wrong - STOP. Don't do anything immediately (circumstances will dictate if you should turn off the machine at that point - 9 times out of 10 you should) but walk away and think about your recovery options and strategies. Cloning the damaged disk gives you a fallback position if the first attenmpt fails and keeps the original as it was.

[mroz]

Certance LTO Ultrium-2 Tape Drive 200/400gb + 19 Tapes on eBay

Ouch - £800 for the unit & 19 tapes.

What's the storage life for a tape compared to a hard drive (with no rewrites)? Cost per GB is roughly comparable. Not that I can afford one atm. I assume lifespan & reliability are high, given that's key to their purpose.

I'm still reinstalling atm. Probably another full day's work left.

Oh, while I'm here, anyone suggest software for backups under XP? Both for imaging my system drive & for backing up groups of files. Reliability is most important. I'd be willing to use separate applications for each case, though an integrated solution would be nice. I'll be backing up to an eSata drive, to a local dvd writer & also across a lan. Recommendations?

[peterb]

Well I didn't pay that much - and a 400GB tape (with hardware compression - 200GB native) costs about £15. Under good storage conditions they should last at least 20 years! However the capital outlay for the drive and a SCSI card is high.

If you are using Windows, then the basic Windowsd backup utility is OK - it is installed by default on Win2K - on XP you have to install it manually from the Install disk (in extras, or options, or wherever the utility software is located)

For cloning a drive to another drive (bit by bit logical copy) I used to use caspar XP, but actually a linux Live CD is the cheapest way (and IMHO is better because you boot off the CD so the system disk isn't in use - no changing data so itis an exact copy). It does require you to learn some (very) basic Linux - mainly how to identify the drives) bvut it is very simple.

As for imaging a system disk, most people seem to recommend one of the Acronis products - but I haven't (needed to ) use it myself.

If you are using Linux, then there is a whole range op options. I use dump either locally or to back up over a lan to the machine with the tape drive innstalled. I get about 1.1GB/min which isn't too bad. However it isn't optimum because my 64 bit PCI SCSI card is in a 32 bit slot, which reduces throuput. (but not many Mobos available with 64 bit PCI slots) But it doesn't really matter - its just fire up and forget.

For a system image, I use Mondo Rescue - so in the event of a system failure, I reckon I would be up and running again in about 2 hours - reload the system from the image on a bootable CD - then restore the working directories from the tape drive.

[mroz]

Well I didn't pay that much - and a 400GB tape (with hardware compression - 200GB native) costs about £15.

Sounds about right for what I've seen. Roughly 65% of the cost of hard drive storage - my 500GB spinpoint was £60.

Under good storage conditions they should last at least 20 years! However the capital outlay for the drive and a SCSI card is high.

Think I'll have to settle for three 500GB spinpoints, which can be installed into the Icy Box enclosure in seconds, for grandfather father son backup; with the data being rewritten every month or so, they should last quite a while. For longer term I guess I'm stuck with dvd-r until blu-ray or hd-dvd media drop massively in price.

If you are using Windows, then the basic Windowsd backup utility is OK - it is installed by default on Win2K - on XP you have to install it manually from the Install disk (in extras, or options, or wherever the utility software is located)

For cloning a drive to another drive (bit by bit logical copy) I used to use caspar XP, but actually a linux Live CD is the cheapest way (and IMHO is better because you boot off the CD so the system disk isn't in use - no changing data so itis an exact copy). It does require you to learn some (very) basic Linux - mainly how to identify the drives) bvut it is very simple.

I use linux occasionally (SuSE usually) & cygwin more often, so it's not entirely alien to me. Never got deeply into it as there's still a lot it won't let me do - lack of video tools being one; I couldn't live without AVISynth.

Thanks for the ideas

As for imaging a system disk, most people seem to recommend one of the Acronis products - but I haven't (needed to ) use it myself.

Google suggested Acronis True Image & Norton Ghost as two popular commercial products. I've used Ghost; might take a look at True Image to see how it compares.

If you are using Linux, then there is a whole range op options. I use dump either locally or to back up over a lan to the machine with the tape drive innstalled. I get about 1.1GB/min which isn't too bad. However it isn't optimum because my 64 bit PCI SCSI card is in a 32 bit slot, which reduces throuput. (but not many Mobos available with 64 bit PCI slots) But it doesn't really matter - its just fire up and forget.

For a system image, I use Mondo Rescue - so in the event of a system failure, I reckon I would be up and running again in about 2 hours - reload the system from the image on a bootable CD - then restore the working directories from the tape drive.

Again, thanks for the feedback.

[mroz]

Three weeks later & guess what?



I was hoping the drive with the error would be the same as last time, so I could hopefully rma it & fix this once & for all, however as one can see, this time it's the other drive & mobo port on which an error is reported.

I'll rebuild, hopefully quickly & with minimal loss, in the next few hours, but obviously this can't go on. Anyone got a clue as to what the problem might be?

The machine has seemed fine for the last three weeks with the exception that a couple of days ago I had an inexplicable reboot - no clues in the logs. Then this morning I wake to a pretty blue screen:
Kernel_Stack_InPage_Error
Stop 0x00000077 (0xC000000E, 0xC000000E, 0x00000000, 0x01EB2000)

Upon reset the machine is unable to boot. Bios reports the Raid failure as shown above.

When the machine must have died Diskkeeper would have been running a continuous defrag on C: (I had this set for Friday 4am to 11am). I was also downloading some material via Emule. No other clues.

Is there likely to be a fault with the mobo or is this going to be a firmware/driver issue? All such is up to date.

I'm somewhat at a loss. Seriously thinking about abandoning Raid on this box.

[peterb]

A long shot but... try running memtest86+ and make sure you aren't getting odd memory errors.

[mroz]

I booted into XP on the recovery install I created last time using an old 20GB drive. The Intel storage manager offered to rebuild MrReliable, which I let it do. It also gave me a warning that there were some read problems with MrFast, which explains the bsod, but that I might be able to backup some of its data.

I can seemingly access all the data. A cursory glance plus chkdsk /r on my three partitions has so far reported no problems - the Data partition on MrReliable still has about 30m more to go.

After this I was in fact planning on checking if System on MrFast is now bootable & then running a recent memtest86+ overnight.

I ran memtest86+ as part of post build testing, before (& I hope immediately after ) overclocking it, with no reported errors, but it's obviously worth repeating now.

Is there any painless way to determine if this is a hardware or driver problem? I'm concerned it's going to be impossible to determine if the fault is with the raid drivers or a drive specific hardware failure, without risking data loss & a repeat of these problems from continued testing.

If I suspect a drive fault, what evidence will I need to present to Scan, in order to get a quick exchange? I've never had drive problems in the past with any of my machines, but then I've never used Samsung drives or Intel Matrix Raid before

[peterb]

If you can access the SMART data from the drive, that might give you a clue - but the problem is pro=ving a negative - a fault showing up in the SMART data indicates a fault - but nothing showing doesn't necessarily indicate that all is well...

My gut feeling is that it is more likely to be a driver issue - however (as I think I mentioned earlier) I had a similar experience which proved to be memory failure - a year after it had been fitted and working without problem, so these things do happen.

I have also just noticed an array problem on my RAID 1 setup - so that machine will be offline in about 5 minutes from now doing a system backup before I investigate the cause! Once bitten...

[mroz]

Caution is always justified. I guess one could go too far, but I know my impatience often gets the better of me, so it's very unlikely I'll ever be too cautious when it comes to computers.

Hope you find & fix the cause of your problem.

Any idea if it's possible to get data via SMART when the drives are in an Intel Matrix Raid config? (Edit: apparently the Matrix Storage manager will report drive SMART events, though I can't find where these are logged).

What tools does one use to monitor SMART data - something specific to the mobo or drive manufacturer, or a generic tool like SpeedFan? I'm ashamed tp say I've never utilised SMART before, though I usually leave it enabled in the bios - just another thing I never got around to

BTW chkdsk returned no errors on my drives. I was also able to subsequently boot from my System partition on MrFast, with no further recovery actions necessary.

Memtest86+ ran for nearly 12hrs & returned no errors.

I'm just starting backing up my partitions now - a little anxiously as atm I only have the disk space to do this by deleting my last backups first. After that I'm abandoning Raid & will restore the data directly to the two drives.

My plan is to run the system like this for a few months, keeping an eye on SMART. Hopefully same usage patterns with no problems will mean it was likely an Intel Raid issue, in which case I'll never touch it again. Drive errors otoh & I can look into replacing the drive/s & possibly going back to Matrix Raid.

Thanks for the advice, once more

I have looked into other Raid solutions. I'd go with software Raid if I was running linux. Being stuck with XP, I'd want a reliable hardware solution. The cheap options (XFX Revo £10; Highpoint around £100) seem far from ideal. £200-300 for Areca or Adaptec I can't justify or afford (I'd want 4 ports to do Raid 5). Looks like if Intel's onboard solution won't work for me I'll have to drop the idea.

[mroz]

I've rebuilt my machine. The drives are now no longer in a Raid, though I did leave the controller in Raid mode somewhat unintentionally, when I meant to switch it to AHCI. XP gets upset if I try to make the change, I assume as it doesn't have what it needs to boot from such a volume. Something else to look into.

[mroz]

I finally found the info needed to switch the drive addressing from RAID to AHCI. Here it is for the record:

See HOWTO: enable AHCI mode after installing Windows - PC Perspective Forums

To recap, I was running XP on a Raid volume (via Intel ICH9R on a Gigabyte P35-DS4). After problems I wanted to switch to AHCI, so I backed all partitions up, deleted the array, repartioned & restored my data.

All well & good: the system boots & functions, however the system is still addressing the drives via the Raid driver (under which they are classed as non-Raid drives).

To switch to AHCI requires a bios config change, however this alone will bluescreen XP during the subsequent boot attempt as it doesn't have the necessary drivers which would usually be installed immediately prior to the OS install.

The solution is to directly modify the registry so that it knows which drivers to use. If one had been running in IDE mode one would additionally need to install the IaStor.sys driver, as described in the above referenced post, however in my case of moving from Raid to AHCI, the file was already installed.

Thus all I needed to do, aside from tracking this procedure down, was execute the ahci.reg file:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Control\CriticalDeviceDatabase\pci#ven_8086&dev_2922&cc_0106]
"Service"="iaStor"
"ClassGUID"="{4D36E96A-E325-11CE-BFC1-08002BE10318}"

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\iaStor]
"Type"=dword:00000001
"Start"=dword:00000000
"Group"="SCSI miniport"
"ErrorControl"=dword:00000001
"ImagePath"="system32\\drivers\\iaStor.sys"
"tag"=dword:00000019
"DisplayName"="Intel AHCI Controller"

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\iaStor\Parameters]
"queuePriorityEnable"=dword:00000000

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\iaStor\Enum]
"0"="PCI\\VEN_8086&DEV_2922&SUBSYS_B0051458&REV_02\\3&13c0b0c5&0&FA"
"Count"=dword:00000001
"NextInstance"=dword:00000001

Reboot, enter the bios & switch from RAID to AHCI, reboot & let XP start up. At this point it finds the 'new' hardware; close the wizard & rerun the Intel Matrix Storage Manager install - this will install the remaining driver related files needed, as well as the storage manager gui itself.

NB For Gigabyte P35-DS4 users, along the way I discovered the IMSM & drivers linked to from GIGABYTE - Support - Motherboard - Driver - GA-P35-DS4 (rev. 1.1) are not the most up to date. I went with the direct download from Intel via Download Software, Drivers and Utilities Filters

As a final amusing aside, despite the fact no new physical hardware was installed during this process, my copy of XP Pro SP2 decided not only did it need reactivating afterwards due to all this new kit, but that I was a bad person & was actually trying to do bad things (tm), so it wouldn't allow an Internet activation. I had to phone some wretched lifeless machine to get a new activation code which seems to have appeased it. Perhaps it was in a rotten mood, what with it being 4:30am.

[peterb]

Glad it is all working! (I died post earlier - but it must have got lost as it didn't appear!)

You might want to look at this

SourceForge.net: S.M.A.R.T. Monitoring Tools

for SMART monitoring. Originally a Linux tool, it looks as if it has been ported to Windows.

I am using the XFX revo as my RAID controller (but bought two - one as a spare "just in case" and it seems OK, the main reason was that the mobo I am using in that machine doesn't support SATA - but for preference I think I would use the Linux native RAID drivers.

(BTW - I think the problem on mty RAID was a loose power connector - removing and reseating it meant the drive was recognised again, so it just need re-initialising and the array is rebuilding now - in fact it should be complete. That is the main problem with teh revo board and Linux - it is only possible to monitor the state of the array at boot time - and the smatmon tools won't work with it either)

[mroz]

Thanks, I'll take a look in a moment.

I'm not sure if it's changing to AHCI, or the driver update, but previously when my eSata device was powered up, I'd get the usual 'new hardware' pop ups. Now however I get both those & a pop up from the IMSM.

The slightly worrying fact is that I'm getting the IMSM pop up at other times too - when no new device is connected. Had it about three times today that I've noticed.

It reads:
Storage Device Connected
A storage device was connected. If this device is a port multiplier, only port 0 of the port multiplier will be active

Obviously I'm wondering what's going on. It occurs to me if, perhaps, I'm losing either power or data connections to one drive, it dropping out could be the source of my problems, while it coming back would explain the pop ups.

An odd coincidence. I'll look for loose connectors when there's daylight. This machine is also my first Sata box. I wasn't hugely impressed by the connectors & even less so now.

Edit: i've just got the message again & noticed a defrag had started. I suppose an intermittent contact could be triggered by vibration increases with increased activity, perhaps.

Edit2, Reports on the drives' fitness:
S.M.A.R.T. hard disk status and hard disk failure prevention
S.M.A.R.T. hard disk status and hard disk failure prevention

Currently both fine.

[jamsbong]

Hi MROZ

I think i have the same problem you had. recently, my 2 hard drive in raid started failing. and it became obvious when the tv recording pauses for an extended period of time. luckily i managed to backup most of my data.

In my experience, harddrive is usually robustly reliable and i certainly was not expecting it to fail after just 1 year. I do overclock my machine but I always monitor the temperature.

However, I'm struggling to pin point where the error comes from. is it from the Intel's raid hardware? or is it from the 2 samsung harddrive? in your case, was it the harddisk or the motherboard?

Bong