N8800 reliability issue

[Joe Wells]

I have a client that is trying to use a N8800, but they're having trouble keeping it on-line.

The current issue is that the SMB shares, along with the Web interface and SNMP all fail, although the iSCSI targets stay on-line. This seems to happen most often during times of heavy I/O, such as during nightly backup. Actually, the Web interface is responsive long enough to prompt for login, but does not respond after that.

This is an unmodified unit, no modules installed, with firmware 2.01.09. AD integration is enabled. SMB and iSCSI are the only transports running.

Thecus folks: This ties back to ticket BEH-62042-858

Any thoughts from the community on this would be most appreciated.

Thanks,
-Joe

[Joe Wells]

At the recommendation of Thecus support, we installed 3.00.07a. Since then, it appears that I can no longer use ADS/NT Support. When I fill in the fields and Apply, it creates the computer account in our domain, but then returns an error "Incorrect administrator ID or password Fail to join ADS domain". The username and password are correct (if they weren't, the computer account wouldn't have been created).

On the domain controllers, I get Event IDs 672, 675, 529, 680 associated with the logon attempt.

AD integration was working correctly for us with 2.01.09.

[MarkTTU]

I have a client that is trying to use a N8800, but they're having trouble keeping it on-line.

The current issue is that the SMB shares, along with the Web interface and SNMP all fail, although the iSCSI targets stay on-line. This seems to happen most often during times of heavy I/O, such as during nightly backup. Actually, the Web interface is responsive long enough to prompt for login, but does not respond after that.

This is an unmodified unit, no modules installed, with firmware 2.01.09. AD integration is enabled. SMB and iSCSI are the only transports running.

Thecus folks: This ties back to ticket BEH-62042-858

Any thoughts from the community on this would be most appreciated.

Thanks,
-Joe

I have 3.00.06 and have the same issues. Did 3.00.07a fix these issues? I don't need/use AD integration so if that's all that's broken in the beta I might give it a shot.

[Joe Wells]

I have 3.00.06 and have the same issues. Did 3.00.07a fix these issues? I don't need/use AD integration so if that's all that's broken in the beta I might give it a shot.

Difficult to say at this point. Without AD integration we really can't use the SMB shares at all. SNMP and the Web Interface haven't failed since upgrading, though. That's about all I can tell conclusively at this point.

-Joe

[MarkTTU]

Thanks Joe. After another crash this morning while attempting to back up my data I took the risk and updated to the beta... I'll everyone know how it goes later.

[MarkTTU]

Well after several days running on the beta firmware I haven't crashed again so it looks like its fixed my problems

I don't know how related it was to the crashing, but shortly after upgrading to the beta firmware I got a notification that I had a bad disk. I went ahead and replaced it, but oddly enough I had to power down and then power back up before the Thecus acknowledged the new disk existed

[Joe Wells]

I'm still having trouble with SMB going off-line during heavy I/O. In fact, it's worse now than before. I can't even copy the data off of the SMB shares in order to bring it on-line elsewhere. Effectively, the NAS is holding ~100GB of data hostage right now.

iSCSI continues to function just fine, though, and the Web Interface and SNMP continue to work when SMB falls over.

I'd really like to have ssh, telnet, or even a monitor and keyboard on this thing so I can see what on Earth is going on. I may try to install an old PCI video card in it just to get terminal access.

My last response from Thecus indicated that 3.00.08 is "days away " from release, but I don't know that we can wait that long.

[MarkTTU]

My last response from Thecus indicated that 3.00.08 is "days away " from release, but I don't know that we can wait that long.

Interesting; when was that? Wonder how many days are in that "days away"....

[Joe Wells]

Interesting; when was that? Wonder how many days are in that "days away"....

Yesterday. What "days away" means in their release schedule is anyone's guess, though.

They sent me the procedure on how to downgrade the firmware today. It appears to involve pulling the firmware from a USB drive. Our spare N8800 is due to return from repair soon (it failed completely). I'd like to just move the drives into that unit and see how it goes rather than downgrade, but at least I have the option now.

[shug]

I'd really like to have ssh, telnet, or even a monitor and keyboard on this thing so I can see what on Earth is going on. I may try to install an old PCI video card in it just to get terminal access.

I've also had problems with a N8800SAS. There are unoffical packages to enable ssh access. As i can't post links atm, you have to use the search function in this forum: "thecus n8800 sas ssh"
I can confirm that those packages are working for the N8800SAS with FW 3.00.07a.
Having console access helped me a lot, i'm wondering why they don't enable this feature by default?!

regards
- shug

[Joe Wells]

I can confirm that those packages are working for the N8800SAS with FW 3.00.07a.

Thank you very much for that. I had looked at those modules before, but they're working now.

It looks like when this system flakes out, zfs-fuse is burning up all of the CPU. More digging to do.

[Joe Wells]

Yuck. ZFS is falling over after almost any I/O. The ZFS shares partition just becomes unavailable. Doing a ls at /raid0 returns the error "./data: Transport endpoint is not connected". /raid0/data is where the shares should be mounted. zfs-fuse isn't a running process right now.

[Joe Wells]

More fun:

We received out backup N8800 unit from repair today, so I wanted to try to move the disks into that one instead. Upon power-up, I discovered that it was running 3.00.06 (rather than 2.01.09 that it left here with). So I attempted to use the firmware downgrade instructions provided.

I checked the md5sum of the N8800_FW_2.01.09.bin file both after download as well as after copying to the USB drive. I prepared the USB drive as directed, shut down the N8800 and connected it (with no hard drives installed).

The LCD display showed a typical boot-up until just after it displayed the firmware rev, then the display briefly showed HHHH all across both lines. Then the screen went to:

FTST EEPROM
Addr: 0x000

Then to:

FTST ---- 00:01
Addr: 0x1ff OK

After a couple of minutes the display changed to:

Self Testing... (with a brief beep shortly after this appeared).

This stayed on for about 4 min, then changed to a typical boot-up:

Thecus NAS
Booting... (and so forth)

The firmware is still at 3.00.06 version, although the config has been set to factory defaults.

I upgraded the fw to 3.00.07a to see if the firmware downgrade procedure was specific to that version, but the result was the same.

Has anyone else successfully downgraded fw? This is the procedure that was provided to me:

1. Prepare an USB flash disk and format as FAT16/32 file system.
2. Download specific firmware file
http://www.thecus.com.tw/Downloads/N8800_FW_2.01.09.bin
3. Extract attached ZIP file and get "factory_test.sh"
4. Copy both "N8800_FW_2.01.09.bin" and "factory_test.sh" into the USB flash disk.
5. Power off the N8800.
6. Remove all drives.
7. Insert the USB flash disk into the N8800 and then power on it
8. Once LCD display "Start FW upgrade", then remove the USB flash disk and wait for the upgrade process complete. The N8800 will automatically reboot itself once it done.
9. Wait the flashing firmware process for 4~6 minutes
10. After reboot, power off the N8800.
11. Install all drives into N8800.

I never see the Start FW upgrade message.

-Joe

[Joe Wells]

OK, it looks like the ZFS snapshot feature went a little crazy. It's supposed to remove the oldest snapshot once it gets to 16, but my shares had ~25 snapshots each. I've removed all snaps from all shares and disabled scheduled snapshots on each. I'm able to copy data so far.

I don't know how long I'll be able to access the shares, but they've remained available far longer than they have in a couple of weeks.