N5200Pro setup problems - Keep losing RAID5

[Stewie Gilligan]

PLEASE NOTE the cause doesn't appear to be the N5200 rather a 'bad' disk - See later posts.


Hello I've just taken delivery on Friday (all from Scan) and I'm trying to setup RAID5 only it doesn't seem to survive a power cycle and I'm now onto my 3rd rebuild which takes over 9 hours. I havent even had a chance to configure the storage.

You can see from the log while it rebuilt OK the RAID didn't last past a scheduled power down and lost a disk.

The drives are on the Thecus compatible list but I've noted the SMART all showing Hardware ECC Recovered errors (paticularly 3 & 5). No idea if thats normal, though I suspect not.

Bad hardware that I should contact Scan about, or idiot user?

If it's the latter any tips for this idiot user?

Cheers for any help,
Stewie

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Current setup
3x Samsung f1 1TB (more when needed) Model HD103UJ, Firmware 1AA0, bay 1,3,5 (better cooling)
N5200Pro Firmware 2.00.04.

Log
2008/05/04 11:14:42 N5200 : The system N5200 is recovering the RAID and rebuilding is in progress.
2008/05/04 11:12:45 N5200 : User admin logged in from 192.168.1.3
2008/05/04 11:11:59 N5200 : The system N5200 change to degrade mode.
2008/05/04 11:11:57 N5200 : N5200 boot
2008/05/04 11:11:54 N5200 : mdadm: /dev/md1 has been started with 2 drives (out of 3).
2008/05/04 11:11:54 N5200 : [Disk sdc] UUID[raid5;e14bd758:2912513e:b590994c:8f099a84;Sat May 3 14:06:30 2008]
2008/05/04 11:11:54 N5200 : [Disk sdb] UUID[raid5;e14bd758:2912513e:b590994c:8f099a84;Sat May 3 14:06:30 2008]
2008/05/04 11:11:54 N5200 : [Disk sda] UUID[raid5;e14bd758:2912513e:b590994c:8f099a84;Sat May 3 14:06:30 2008]
2008/05/04 11:11:54 N5200 : Assemble RAID [md1] from [ /dev/sda2 /dev/sdb2 /dev/sdc2 ]
2008/05/04 11:08:51 N5200 : User admin logged in from 192.168.1.3
2008/05/04 11:08:38 N5200 : The system N5200 change to degrade mode.
2008/05/04 11:08:37 N5200 : N5200 boot
2008/05/04 11:08:34 N5200 : mdadm: /dev/md1 has been started with 2 drives (out of 3).
2008/05/04 11:08:34 N5200 : [Disk sdc] UUID[raid5;e14bd758:2912513e:b590994c:8f099a84;Sat May 3 14:06:30 2008]
2008/05/04 11:08:34 N5200 : [Disk sdb] UUID[raid5;e14bd758:2912513e:b590994c:8f099a84;Sat May 3 14:06:30 2008]
2008/05/04 11:08:34 N5200 : [Disk sda] UUID[raid5;e14bd758:2912513e:b590994c:8f099a84;Sat May 3 14:06:30 2008]
2008/05/04 11:08:34 N5200 : Assemble RAID [md1] from [ /dev/sda2 /dev/sdb2 /dev/sdc2 ]
2008/05/04 10:40:44 N5200 : The system N5200 change to degrade mode.
2008/05/04 10:40:37 N5200 : N5200 boot
2008/05/04 10:40:34 N5200 : mdadm: /dev/md1 has been started with 2 drives (out of 3).
2008/05/04 10:40:34 N5200 : [Disk sdb] UUID[raid5;e14bd758:2912513e:b590994c:8f099a84;Sat May 3 14:06:30 2008]
2008/05/04 10:40:34 N5200 : [Disk sda] UUID[raid5;e14bd758:2912513e:b590994c:8f099a84;Sat May 3 14:06:30 2008]
2008/05/04 10:40:34 N5200 : Assemble RAID [md1] from [ /dev/sda2 /dev/sdb2 ]
2008/05/04 01:00:02 N5200 : Schedule Power Off
2008/05/03 23:22:15 N5200 : Healthy: The system N5200 is healthy now.
2008/05/03 15:05:53 N5200 : User admin logged in from 192.168.1.1
2008/05/03 14:19:05 N5200 : User admin logged in from 192.168.1.3
2008/05/03 14:06:30 N5200 : The system N5200 is recovering the RAID and rebuilding is in progress.
2008/05/03 14:00:25 N5200 : Swap Disk Damage

Drive Info
Tray Number 1
Model HD103UJ
Power On Hours 26 Hours
Temperature Celsius 26
Reallocated Sector Count 0
Current Pending Sector 0
Raw Read Error Rate 0
Seek Error Rate 0
Hardware ECC Recovered 124075

Tray Number 3
Model HD103UJ
Power On Hours 27 Hours
Temperature Celsius 27
Reallocated Sector Count 0
Current Pending Sector 0
Raw Read Error Rate 0
Seek Error Rate 0
Hardware ECC Recovered 235401994

Tray Number 5
Model HD103UJ
Power On Hours 28 Hours
Temperature Celsius 29
Reallocated Sector Count 0
Current Pending Sector 0
Raw Read Error Rate 0
Seek Error Rate 0
Hardware ECC Recovered 242190514

[Stewie Gilligan]

Well, after 9 hours of rebuilding, I assigned some iSCSI space, but didn't want to spend too much time on it (just in case).

Did a system shutdown via the web menu, powered off for a minute and rebooted.

Yet again a disk is missing, though the lights show as powered Disk 1 is missing from the array and an annoying constant tone is comming from the N5200 as I type.

Aggggghhhhhhh!

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
System log
2008/05/04 20:07:11 N5200 : User admin logged in from 192.168.1.3
2008/05/04 20:06:13 N5200 : The system N5200 change to degrade mode.
2008/05/04 20:06:06 N5200 : N5200 boot
2008/05/04 20:06:03 N5200 : mdadm: /dev/md1 has been started with 2 drives (out of 3).
2008/05/04 20:06:03 N5200 : [Disk sdb] UUID[raid5;e14bd758:2912513e:b590994c:8f099a84;Sat May 3 14:06:30 2008]
2008/05/04 20:06:03 N5200 : [Disk sda] UUID[raid5;e14bd758:2912513e:b590994c:8f099a84;Sat May 3 14:06:30 2008]
2008/05/04 20:06:03 N5200 : Assemble RAID [md1] from [ /dev/sda2 /dev/sdb2 ]
2008/05/04 19:58:46 N5200 : Healthy: The system N5200 is healthy now.
2008/05/04 19:58:45 N5200 : The system N5200 change to degrade mode.
2008/05/04 16:10:14 N5200 : User admin logged in from 192.168.1.3
2008/05/04 11:14:42 N5200 : The system N5200 is recovering the RAID and rebuilding is in progress.
2008/05/04 11:12:45 N5200 : User admin logged in from 192.168.1.3
2008/05/04 11:11:59 N5200 : The system N5200 change to degrade mode.
2008/05/04 11:11:57 N5200 : N5200 boot

[Stewie Gilligan]

After some experimentaion of rebuilding the RAID, swapping the drives to other bays to make sure it's not a faulty connector issue with the N5200, I've been able to conclude that a single drive appears faulty.

The RAID succesfully survives a hot reboot, but after every cold start the same physical drive always gets dropped (regardless of bay) and the array degraded meaning 9 hours of down time whilst it's rebuilt. Yet more when other drives are added later.

No errors (other than ECC) appear in the SMART which made it difficult to issolate.

Hope no one else ever has this, it's not a problem I've ever seen before, but at least now it's logged here.

Log and trouble shooting approach follow.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This is 3rd attempt at building the array – Previous logs do not seem accessible (conjecture, they are stored somewhere on the array and lost when reformatted)

Here the disk in Drive1 is continually being lost during power down then up, causing the RAID5 array to degrade and over 9 hours of rebuilding, whilst the device is useless (data can’t be accessed).

2008/05/04 11:14:42 N5200 : The system N5200 is recovering the RAID and rebuilding is in progress.
2008/05/04 11:12:45 N5200 : User admin logged in from 192.168.1.3
2008/05/04 11:11:59 N5200 : The system N5200 change to degrade mode.
2008/05/04 11:11:57 N5200 : N5200 boot
2008/05/04 11:11:54 N5200 : mdadm: /dev/md1 has been started with 2 drives (out of 3).
2008/05/04 11:11:54 N5200 : [Disk sdc] UUID[raid5;e14bd758:2912513e:b590994c:8f099a84;Sat May 3 14:06:30 2008]
2008/05/04 11:11:54 N5200 : [Disk sdb] UUID[raid5;e14bd758:2912513e:b590994c:8f099a84;Sat May 3 14:06:30 2008]
2008/05/04 11:11:54 N5200 : [Disk sda] UUID[raid5;e14bd758:2912513e:b590994c:8f099a84;Sat May 3 14:06:30 2008]
2008/05/04 11:11:54 N5200 : Assemble RAID [md1] from [ /dev/sda2 /dev/sdb2 /dev/sdc2 ]
2008/05/04 11:08:51 N5200 : User admin logged in from 192.168.1.3
2008/05/04 11:08:38 N5200 : The system N5200 change to degrade mode.
2008/05/04 11:08:37 N5200 : N5200 boot
2008/05/04 11:08:34 N5200 : mdadm: /dev/md1 has been started with 2 drives (out of 3).
2008/05/04 11:08:34 N5200 : [Disk sdc] UUID[raid5;e14bd758:2912513e:b590994c:8f099a84;Sat May 3 14:06:30 2008]
2008/05/04 11:08:34 N5200 : [Disk sdb] UUID[raid5;e14bd758:2912513e:b590994c:8f099a84;Sat May 3 14:06:30 2008]
2008/05/04 11:08:34 N5200 : [Disk sda] UUID[raid5;e14bd758:2912513e:b590994c:8f099a84;Sat May 3 14:06:30 2008]
2008/05/04 11:08:34 N5200 : Assemble RAID [md1] from [ /dev/sda2 /dev/sdb2 /dev/sdc2 ]
2008/05/04 10:40:44 N5200 : The system N5200 change to degrade mode.
2008/05/04 10:40:37 N5200 : N5200 boot
2008/05/04 10:40:34 N5200 : mdadm: /dev/md1 has been started with 2 drives (out of 3).
2008/05/04 10:40:34 N5200 : [Disk sdb] UUID[raid5;e14bd758:2912513e:b590994c:8f099a84;Sat May 3 14:06:30 2008]
2008/05/04 10:40:34 N5200 : [Disk sda] UUID[raid5;e14bd758:2912513e:b590994c:8f099a84;Sat May 3 14:06:30 2008]
2008/05/04 10:40:34 N5200 : Assemble RAID [md1] from [ /dev/sda2 /dev/sdb2 ]
2008/05/04 01:00:02 N5200 : Schedule Power Off
2008/05/03 23:22:15 N5200 : Healthy: The system N5200 is healthy now.
2008/05/03 15:05:53 N5200 : User admin logged in from 192.168.1.1
2008/05/03 14:19:05 N5200 : User admin logged in from 192.168.1.3
2008/05/03 14:06:30 N5200 : The system N5200 is recovering the RAID and rebuilding is in progress.

This is 4th Attempt – The disk that was previously in Drive1 (that kept disappearing from the Array) has been swapped with Drive in bay 3, and the array rebuilt. This should help confirm if it’s a bad connector due to manufacturing issues (quality control) as some of the forum posts suggest they have suffered.

I’m attempting to isolate when the device fails so will incrementally alter the environment using typical trouble shooting methods.

Some minor configuration of the storage has been performed (RAID Data set to, USB Target, iSCSC and Access Control List is modified.

A Reboot ONLY is performed – This is the 1st successful reboot rebuilding of the RAID. Note this turns out to be onlyy a partial success, see below and impact of a power cycle.

2008/05/05 09:54:01 N5200 : User admin logged in from 192.168.1.3
2008/05/05 09:52:56 N5200 : N5200 boot
2008/05/05 09:52:55 N5200 : Healthy: The system N5200 is healthy now.
2008/05/05 09:52:52 N5200 : mdadm: /dev/md1 has been started with 3 drives.
2008/05/05 09:52:52 N5200 : [Disk sdc] UUID[raid5;ab634f59:6e3bfd7b:ba04e73b:5e364ffc;Sun May 4 21:00:51 2008]
2008/05/05 09:52:52 N5200 : [Disk sdb] UUID[raid5;ab634f59:6e3bfd7b:ba04e73b:5e364ffc;Sun May 4 21:00:51 2008]
2008/05/05 09:52:52 N5200 : [Disk sda] UUID[raid5;ab634f59:6e3bfd7b:ba04e73b:5e364ffc;Sun May 4 21:00:51 2008]
2008/05/05 09:52:52 N5200 : Assemble RAID [md1] from [ /dev/sda2 /dev/sdb2 /dev/sdc2 ]
2008/05/05 09:50:40 N5200 : The ACL of folder usbhdd has been modified.
2008/05/05 09:50:18 N5200 : The ACL of folder usbhdd has been modified.
2008/05/05 09:49:27 N5200 : You have change folder [usbcopy] info in RAID : [ Public = no ]
2008/05/05 09:46:48 N5200 : [ Quota ] Folder [ usbcopy ] add to Quota Management.
2008/05/05 09:46:48 N5200 : You have change folder [usbcopy] info in RAID : [ Public = yes ][ Share Folder Limit = 1 GB ]
2008/05/05 09:28:19 N5200 : User admin logged in from 192.168.1.3
2008/05/05 06:38:50 N5200 : Healthy: The system N5200 is healthy now.
2008/05/05 00:53:52 N5200 : User admin logged in from 192.168.1.1
2008/05/04 21:00:51 N5200 : The system N5200 is recovering the RAID and rebuilding is in progress.
2008/05/04 20:56:40 N5200 : User admin logged in from 192.168.1.3
2008/05/04 20:55:39 N5200 : N5200 boot
2008/05/04 20:55:36 N5200 : [Disk sdb] UUID[raid5;e14bd758:2912513e:b590994c:8f099a84;Sat May 3 14:06:30 2008]

At this point, I’ve decided to turn Disk Power Management from OFF to 30 minutes (action doesn’t get recorded in the logs), another reboot is performed (in the unlikely event the disk management settings may not apply until reboot). This appears to be successful as below.

2008/05/05 10:12:21 N5200 : User admin logged in from 192.168.1.3
2008/05/05 10:12:14 N5200 : N5200 boot
2008/05/05 10:12:14 N5200 : Healthy: The system N5200 is healthy now.
2008/05/05 10:12:11 N5200 : mdadm: /dev/md1 has been started with 3 drives.

At this point I’ve left the drive for at least 40 minutes, and on return attempted to access the drives. Only accessed SMART info – All appeared OK, but unable to say if the drives actually span down and up again.

Performed a controlled shut down via the browser at 11:10.

As you can see below the RAID is lost (Degraded), this time Disk 3 is missing. NOTE Disk 3 here is the one transferred from Disk 1 that used to go missing previously. It would seem that disk will not survive a power cycle and maintain the RAID.

Rebooted device from a cold start. As usual there is a continuous monotone warning (very annoying, can’t work next to it) coming from the N5200. RAID is still showing as degraded. Disk 3 (same hardware that was Disk1 in earlier errors) is OK but not within the RAID. It is NOW available as a Spare so could be added (though it should still be part of RAID), BUT I’m not repeating this as that takes another 9 hours where data is vulnerable to loss due to hardware failure during re-build after every power down or power cut. Drive is not usable in RAID while the other members of the ‘triplets’ appear to be working OK.

2008/05/05 11:21:29 N5200 : User admin logged in from 192.168.1.3
2008/05/05 11:20:21 N5200 : The system N5200 change to degrade mode.
2008/05/05 11:20:14 N5200 : N5200 boot
2008/05/05 11:20:11 N5200 : mdadm: /dev/md1 has been started with 2 drives (out of 3).
2008/05/05 11:20:11 N5200 : mdadm: /dev/md1 has been started with 2 drives (out of 3).
2008/05/05 11:20:11 N5200 : [Disk sdc] UUID[raid5;ab634f59:6e3bfd7b:ba04e73b:5e364ffc;Sun May 4 21:00:51 2008]
2008/05/05 11:20:11 N5200 : [Disk sdc] UUID[raid5;ab634f59:6e3bfd7b:ba04e73b:5e364ffc;Sun May 4 21:00:51 2008]
2008/05/05 11:20:11 N5200 : [Disk sdb] UUID[raid5;ab634f59:6e3bfd7b:ba04e73b:5e364ffc;Sun May 4 21:00:51 2008]
2008/05/05 11:20:11 N5200 : [Disk sdb] UUID[raid5;ab634f59:6e3bfd7b:ba04e73b:5e364ffc;Sun May 4 21:00:51 2008]
2008/05/05 11:20:11 N5200 : [Disk sda] UUID[raid5;ab634f59:6e3bfd7b:ba04e73b:5e364ffc;Sun May 4 21:00:51 2008]
2008/05/05 11:20:11 N5200 : [Disk sda] UUID[raid5;ab634f59:6e3bfd7b:ba04e73b:5e364ffc;Sun May 4 21:00:51 2008]
2008/05/05 11:20:11 N5200 : Assemble RAID [md1] from [ /dev/sda2 /dev/sdb2 /dev/sdc2 ]
2008/05/05 11:20:11 N5200 : Assemble RAID [md1] from [ /dev/sda2 /dev/sdb2 /dev/sdc2 ]
2008/05/05 11:15:42 N5200 : User admin logged in from 192.168.1.3
2008/05/05 11:15:35 N5200 : The system N5200 change to degrade mode.
2008/05/05 11:15:28 N5200 : N5200 boot
2008/05/05 11:15:25 N5200 : mdadm: /dev/md1 has been started with 2 drives (out of 3).
2008/05/05 11:15:25 N5200 : mdadm: /dev/md1 has been started with 2 drives (out of 3).
2008/05/05 11:15:25 N5200 : [Disk sdb] UUID[raid5;ab634f59:6e3bfd7b:ba04e73b:5e364ffc;Sun May 4 21:00:51 2008]
2008/05/05 11:15:25 N5200 : [Disk sdb] UUID[raid5;ab634f59:6e3bfd7b:ba04e73b:5e364ffc;Sun May 4 21:00:51 2008]
2008/05/05 11:15:25 N5200 : [Disk sda] UUID[raid5;ab634f59:6e3bfd7b:ba04e73b:5e364ffc;Sun May 4 21:00:51 2008]
2008/05/05 11:15:25 N5200 : [Disk sda] UUID[raid5;ab634f59:6e3bfd7b:ba04e73b:5e364ffc;Sun May 4 21:00:51 2008]
2008/05/05 11:15:25 N5200 : Assemble RAID [md1] from [ /dev/sda2 /dev/sdb2 ]
2008/05/05 11:15:25 N5200 : Assemble RAID [md1] from [ /dev/sda2 /dev/sdb2 ]
2008/05/05 11:01:15 N5200 : User admin logged in from 192.168.1.3
2008/05/05 10:12:21 N5200 : User admin logged in from 192.168.1.3
2008/05/05 10:12:14 N5200 : N5200 boot
2008/05/05 10:12:14 N5200 : Healthy: The system N5200 is healthy now.

[banana]

Hi Stewie,

thanks for your detailed report.
I'm having almost exactly the same issues here.
Same system, same harddisks, same raid.
The disks are ok according to SMART and tested with Samsung's es-tool.

So I clearly put the blame on Thecus. You are not an "idiot user", nor am I.

I wish Thecus would concentrate on making their products reliable instead of putting out new ones.

My system currently is doing the funny 9-hour-rebuild. Maybe I can add something tomorrow.

Greets
banana

[Stewie Gilligan]

I wasn't going to do a diagnostic scan but I did anyway, unsuprisingly no errors found using current version of Samsung tools.

Can't understand what the difference betwwen the good and bad drives, I know it doesn't mean a great deal but the serials are very close to each other, so probably the same batch etc, only this one refuses to work....but only after a cold boot.

I'm loathed to keep the drive knowing I could go thru this all again when I buy/add another 2TB, only I imagine it's going to take almost twice the time

[banana]

All seems fine now.
What I did was replacing the suspicious drive and doing a filesystem check.

I brought back the drive to where I bought it and the dealer was kind enough to exchange it for another of the same kind, really very nice.

Then after the rebuild I still had problems with a non-responding web-GUI and all the folders and data were gone, which wasn't a problem, because it was all testing stuff, but I don't like that to happen with real data.

Then I did a filesystem check and afterwards everything was fine, the data were back again and the web-gui was working as it should.

I tried reboots also, warm and cold, no problems anymore.

So, knock on wood, I'm happy again with my Thecus.

Conclusions:
-The Samsung HD103UJ don't seem to be 100% compatible, and there doesn't seem to be a way of knowing beforehand if a drive will work in the Thecus or not. It's a matter of chance.

-Do a filesystem check, if the Thecus isn't working as it should.
But I would expect from a user-friendly machine to be doing a check automatically.

@Stewie
Maybe you could try it the other way round, doing the filesystem check and if that doesn't help, replace the drive.

Good luck
banana

[Itlan]

The Samsung Spinpoints seem to have a nasty bug where they are constantly losing the Raid. There is rumored to be a beta firmware that fixes this issue referenced in several of the threads below, so I'd contact Thecus and see if that is the case.

See the following for more people living with your pain:
Well since I dont have 5 posts I cant post links so here are the thread numbers you can call up or just search on Spinpiint and degrade and youll get the jist of it...

thecususergroup.proboards106.com

=2301

=2404

=2140

=2445

=172

=2106

[Stewie Gilligan]

Originally Posted by Itlan

The Samsung Spinpoints seem to have a nasty bug where they are constantly losing the Raid. There is rumored to be a beta firmware that fixes this issue referenced in several of the threads below, so I'd contact Thecus and see if that is the case.

Cheers Itlan will have a look at that - Looks excellent news. Sorry for the delay, I never got the email saying a post had been made.

The Thecus has been with Scan for about 3 weeks while they did their tests. Just got it back a few days ago, they ignored my test results, sent me a new N5200 and the old drives back. Of course the drive still drops out of the RAID, so I'm waiting another RMA from them - v v v poor customer service from Scan, all they seem to have done is a stress test on the drives, and built the array watched it drop on a cold boot and assume it's the Thecus, even when I've written it all down, phoned and spoke to them explaining the situation before they even got the drives back.

As it happens I bought another couple of F1 before I read this, and the Thecus has been behaving normally - Did a fair bit testing to make sure the issue didn't repeat - I've just finished transfering about 1.5 TB of data to it last night, all seems well, surviving numerous boots/power downs correctly getting ot this stage.

I thought I'd do a file system check before I deleted the data from the original sources......imagine my supprise when I came back to find array had degraded and a previousley good drive had been dropped from the array (no smart errors etc).

The arrays being rebuilt now, but I have to say while both the N5200 Pro and the Samsung F1 have had excellent reviews independantly I'm less than impressed when both are together - It's starting to look like £1000+ down the toilet.

[peterfu]

Originally Posted by Itlan

The Samsung Spinpoints seem to have a nasty bug where they are constantly losing the Raid. There is rumored to be a beta firmware that fixes this issue referenced in several of the threads below, so I'd contact Thecus and see if that is the case.

May be You should check this thread:
http://www.thecusforum.eu/showthread.php?t=345
br
Peter

[Stewie Gilligan]

Originally Posted by peterfu

May be You should check this thread:
http://www.thecusforum.eu/showthread.php?t=345
br
Peter

Thanks Peter, I've downloaded the beta firmware, seems to be running OK for now, and I quite like the ISO function, as I never upgraded from the pre-installed ~.04 firmware.

<OT>
I'm not so keen on the way ISO Mount is implemented, I'd have suggested including the ability for the user to specify a virtual folder (neater) rather than having to have the .iso and folder side by side. Also all the files show as hidden (using XP Pro) - Perhaps it's me (not seen any instructions), though I have noted other forum posts mention the hidden attribute is always set for them also.
</OT>
Regards,
Stewie