N8800 Froze; Rebooted and RAID is GONE!!!

Hello All -
Our N8800 was running for about 100 days, then got frozen today and the only thing I could do was to cut the power and rebooted the machine.

Well, it booted up now, showing all the drives, but no RAID!

I cannot SSH into the system. I cannot select the web-based module management section because it complains about "You don't have any RAID system". I received several email messages from the N8800 about the system is "healthy". Not!

Did I just lost all the data? All 5+ TB of it?

I'm panicking! Can someone please throw me a rope to pull me out of this hole!

Many thanks.

If one look at the information log below, where I highlighted in Red ...

... we can see the N8800 know something about the raid volume created earlier. I think if I can somehow ssh into the machine, I should be able to use mdadm to reassemble it (?). Possible? I don't know.

Right now, I cannot do that because I cannot click on the Module Mgnt section to install any new module.

HELLO THECUS - Please help! I tried to open a ticket, but the support website asked me to register first. I'm waiting for registration to confirm before seeking any support from Thecus.

Yvon @ Thecus - can you hear me?/??? :crazy:

Here's my error log:
2009/10/01 20:04:32 landaufs02 : Can't mount data volume, and you can do filesystem check try to recovery your data.
2009/10/01 20:05:49 landaufs02 : Can't mount data volume, and you can do filesystem check try to recovery your data.
2009/10/01 20:05:51 landaufs02 : Swap Disk Damage
2009/10/01 20:06:45 landaufs02 : User admin logged in fail from xxx.xxx.xxx.xxx

Warning Log:
2009/10/01 20:04:32 landaufs02 : Assemble RAID warning, try to force assemble RAID

Information Log:
2009/10/01 20:04:24 landaufs02 : [Disk sda] UUID[raid6;560cb50a:abdf8f39:96cc7eda:b898bfb6;Thu Aug 27 14:19:04 2009]
2009/10/01 20:04:24 landaufs02 : [Disk sdb] UUID[raid6;560cb50a:abdf8f39:96cc7eda:b898bfb6;Thu Aug 27 14:19:04 2009]
2009/10/01 20:04:24 landaufs02 : [Disk sdc] UUID[raid6;560cb50a:abdf8f39:96cc7eda:b898bfb6;Thu Aug 27 14:19:04 2009]
2009/10/01 20:04:24 landaufs02 : [Disk sdd] UUID[raid6;560cb50a:abdf8f39:96cc7eda:b898bfb6;Thu Aug 27 14:19:04 2009]
2009/10/01 20:04:24 landaufs02 : [Disk sde] UUID[raid6;560cb50a:abdf8f39:96cc7eda:b898bfb6;Thu Aug 27 14:19:04 2009]
2009/10/01 20:04:24 landaufs02 : [Disk sdf] UUID[raid6;560cb50a:abdf8f39:96cc7eda:b898bfb6;Thu Aug 27 14:19:04 2009]
2009/10/01 20:04:24 landaufs02 : [Disk sdg] UUID[raid6;560cb50a:abdf8f39:96cc7eda:b898bfb6;Thu Aug 27 14:19:04 2009]
2009/10/01 20:04:24 landaufs02 : [Disk sdh] UUID[raid6;560cb50a:abdf8f39:96cc7eda:b898bfb6;Thu Aug 27 14:19:04 2009]
2009/10/01 20:04:24 landaufs02 : Assemble RAID [md1] from [ /dev/sda2 /dev/sdb2 /dev/sdc2 /dev/sdd2 /dev/sde2 /dev/sdf2 /dev/sdg2 /dev/sdh2 ]
2009/10/01 20:04:25 landaufs02 : mdadm: /dev/md1 has been started with 8 drives.
2009/10/01 20:05:46 landaufs02 : Assemble RAID [md1] from [ /dev/sda2 /dev/sdb2 /dev/sdc2 /dev/sdd2 /dev/sde2 /dev/sdf2 /dev/sdg2 /dev/sdh2 ]
2009/10/01 20:05:46 landaufs02 : mdadm: /dev/md1 has been started with 8 drives.
2009/10/01 20:05:50 landaufs02 : landaufs02 boot
2009/10/01 20:05:50 landaufs02 : Healthy: The system landaufs02 is healthy now.
2009/10/01 20:05:51 landaufs02 : Healthy: The system landaufs02 is healthy now.
2009/10/01 20:06:52 landaufs02 : User admin logged in from xxx.xxx.xxx.xxx

Dear rhumba,

Please wait for the registration confirmation and submit a ticket for our technical team to assist with your query.

Also provide us with your config.bin for our tech support team to better assist with your query.

You can download your config bin under System-ConfigMgnt-download

_____________________________

Official Thecus Support Representative

Dear rhumba,
we suffer from an equal problem (http://forums.hexus.net/thecus-care-...0-06-ajax.html).
I have submitted a support call 3 days ago. But nobody is caring about. No reaction at all. We have 3 remote branch offices who can not work properly now for 3 days. All data is gone (backup exist, but it can not be restored).

Regards,
Rüdiger

It must be the crappy N8800. As of today, I have six N5500 and four N7700 stationed across different remote sites and all are working very well. All these boxes are running 24/7 for hundreds of days! I was pretty confident when I purchase this N8800 for the capacity and it only had been around 45 days!

When I tried to ssh into the box, it gave me a "ssh: connect to host nas02 port 22: Connection refused". Wondering if the flash storage has problem or not.

Please keep me posted on any progress you receive from Thecus because I cannot open a ticket myself.

Thanks.

What kind of disks do you use with N8800?? I have new N8800 with 3.00.06 FW and have so many problems with this NAS :/ I use Samsung Spinpoint 1TB HD103UJ

Dear rhumba,

Can you tell me the problem you are experiencing when trying to open a ticket?

_____________________________

Official Thecus Support Representative

Hello Ivan - Please locate my support ticket# KLH-555519

Thank you.

Hello rhumba,
Thecus support team did contact me and was trying to point out that the used Samsung disks were not certified. Since we could not afford the down time any more, we replaced the Samsung disks by Seagate ES.2 (now 8x ES.2 1TB) yesterday - but it did not help.

The N8800 froze always when creating the raid (only power off was possible), the raid became degraded and disk dropped randomly (were marked red).
Now we have spent a lot of money and time but still we are offline.

And due to Mr. Murphy, the complete backup is also lost, since the N5200PRO containing the backup lost its RAID 6 after a reboot (5 x Samsung 1TB certified; Display: NO RAID). We tested the disk with Samsung test tool: Result "No Error".
We were mirroring over locations. It is a absolute disaster. 800 GB of 3,5TB can not be recovered. The financial loss is tremendous.

Greetings,
Rüdiger

We have tried to recreate RAID6 on N8800 over night. After 3 hours one disk was labeled as "disk was removed". The Raid became degraded. After reboot the disk was available as expected.
I feel, that the new firmware has a huge problem.
At the moment the N8800 is absolutely unusable and support still has not provided any idea to solve the problem.

Rüdiger

I think the linux OS on the flash chip is corrupted.

I'm wondering if we can download a clean content of the flash chip and replace it on our flash chip (???)

We have a couple of N8800s at work, the first of which displayed VERY similar symptoms. We bought in a second N8800 to use to recover the data as it was felt that the DOM had failed however the new unit displayed the same symptoms.

It turned out in the end that one of the disks wasn't being picked up correctly, and was reporting a small number of bad sectors. This disk was the hot spare for the RAID5 array, so there's not really any way it should have affected the array however when I pulled the disk while the N8800 was up the array appeared. I reinserted the disk and added it back as hotspare and all has been fine since.

We've got data backed up between the 2 units now at both sites but all important data is backed up to tape too - these units basically give us business continuity - it's quicker to restore data from these than to call the tapes back on site for a restore, but even if both went pop we can restore the data within a day or 2. Using a NAS for backup is all well and good for online backups, but for business critical data (such as it sounds like you've lost) you *really* need a tried and tested offline backup. And by tested I mean *regularly* tested.

Hello, I had the same last night, after a proper and normal shutdown on my N7700 by the webinterface and a normal boot later on the total RAID5 configuration doesn't exist anymore,

Please tell me if you were able to get your data back, :crazy::crazy:

Hi wouterb,
sorry to tell you - but we have lost all our data on both units.

Everyone will tell you that RAID craps out when you're rebooting/re-initializing the array so not sure what's the point of pushing your luck? Unless you obviously don't have a choice cuz it froze etc.

My box ran pretty much 24/7 for the past year and no issues. Saving on leccy bills by turning it off is not worth it imo. You can cause yourself lots of problems.

/RANT

Businesses are spending thousands of pounds to get reliable storage and I would not bother with dodgy boxes for up to £1000 quid... Don't get me wrong here I do have my box and I'm very happy with it BUT for home/personal use.

Deffo I'd never use one as live server in production as risks of loosing data is always greater than the price to pay.

/RANT OFF

Tonight, I finally got connected with a support person at Thecus via Teamviewer.

Thecus transferred a ssh patch into my N8800. From there, they went about checking and repairing the xfs partition and in about ten minutes, I got all my data back!!!

I'm a happy camper!

Thanks Thecus. !!!