Thecus N7700 NFS/portmap issues with linux clients

[chrisskelsey]

Hello,

I'm experiencing trouble with NFS shares from my Thecus N7700. Firmware is 2.01.09.

The directories are exported to linux clients, rw to any host (*) with full access from client root accounts. This all works well. The problem occurs when one of the NFS clients reboots (or just umounts the share): when the client tries to mount the share, I get permission denied errors:

mount: mount to NFS server 'metnas' failed: System Error: Connection refused(retrying)

Typical fstab entry on the client is:

# Home directories
metnas:/raid1/data/swdev/home /usr1/home nfs hard,intr,bg 0 0

I resolve this by stopping and starting NFS on the N770 then 'mount -a'. A simple workaround, but I can't have this weakness in our network so I'd like to get to the bottom of it.

Perhaps related are these messages in /var/log/messages on the client:

Feb 2 22:01:59 geosat-dps kernel: portmap: server 172.16.0.33 not responding, timed out
Feb 2 22:01:59 geosat-dps kernel: lockd: server 172.16.0.33 not responding, still trying

rpcinfo -p 172.16.0.33 (the N7700) gives:

rpcinfo: can't contact portmapper: RPC: Remote system error - Connection refused

I may be wrong, but I expect any NFS server to be running portmapper and a bunch of rpc services. The following processes are running on the N7700:

root@127.0.0.1:~# ps|grep rpc
4958 root SW< [rpciod/0]
15396 root 188 S /opt/nfs-utils/sbin/rpc.rquotad
15408 root 804 S /opt/nfs-utils/sbin/rpc.mountd
15410 root 688 S /opt/nfs-utils/sbin/rpc.statd
15423 root 476 S /opt/nfs-utils/sbin/rpc.idmapd

root@127.0.0.1:~# ps |grep lock
35 root SW< [kblockd/0]
15398 root SW [lockd]

but no portmap.

The binary does exist in /sbin/portmap and running it manually invokes the process, but no joy from rpcinfo on the clients. The process exits shortly afterwards.

Any suggestions most appreciated.

Cheers,

Chris.

[chrisskelsey]

Some more info on this.

Starting portmap manually causes rpcinfo -p 172.16.0.33 to respond with:

program vers proto port
100000 2 tcp 111 portmapper
100000 2 udp 111 portmapper

but umounting any directory kills portmapper.

dmesg shows the process is segfaulting:

portmap[21873]: segfault at b7e3a5f4 eip 0804d182 esp bfbe82f4 error 7
nfsd: last server has exited
nfsd: unexporting all filesystems
RPC: failed to contact local rpcbind server (errno 512).
portmap[23672]: segfault at b7efd5f4 eip 0804d182 esp bf819f24 error 7
nfsd: last server has exited
nfsd: unexporting all filesystems
RPC: failed to contact local rpcbind server (errno 512).
portmap[15223]: segfault at b7eb15f4 eip 0804d182 esp bfe96da4 error 7
portmap[6456]: segfault at b7ed15f4 eip 0804d182 esp bfeff184 error 7
portmap[9309]: segfault at b7e435f4 eip 0804d182 esp bff8ea14 error 7
nfsd: last server has exited
nfsd: unexporting all filesystems
RPC: failed to contact local rpcbind server (errno 512).
portmap[10978]: segfault at b7e3b5f4 eip 0804d182 esp bfa41154 error 7
portmap[12151]: segfault at b7e615f4 eip 0804d182 esp bfb55dd4 error 7
portmap[12794]: segfault at b7e2e5f4 eip 0804d182 esp bfeff984 error 7
portmap[22610]: segfault at b7ea45f4 eip 0804d182 esp bfb855f4 error 7

Any ideas here? Thanks.

[chrisskelsey]

Further updates.

Tried FW 3.00.08c and b but no improvement.

Can anyone confirm that the N7700 can successfully NFS share (read/write) folders to linux clients please? Processes on my unit seg fault preventing normal NFS sharing.

So further to the above, here's an example error in 'dmesg' output on the 7700:

portmap[10305]: segfault at b7ecb5f4 eip 0804d182 esp bfbd9694 error 7

which appeared at some point after NFS mounts were made. Subsequently on the client:

rpcinfo -p metnas
rpcinfo: can't contact portmapper: RPC: Remote system error - Connection refused

and related problems NFS mounting other directories and obtaining NFS locks on existing mounts. These problems seem to occur without pattern: sometimes immediately upon NAS boot (so no portmap running when the unit comes up); sometimes only after u/mounting directories a few times; and others after u/mounts and extensive file accesses on the mount. I've tried, and failed, to find any pattern whatsoever.

uname is:

Linux N7700 2.6.23N7700 #1 SMP Wed Dec 23 17:32:50 CST 2009 i686 unknown

and an strace on portmapper follows below.

I'm stumped now. I'm sure this basic NFS usage should work. I opened a ticket on this issue (QEN-912923) two weeks ago but I've had no reply.

BTW, might not be related, but when I upgrade firmware via the web, the progress bar only gets to 17% (or so) before reporting the upgrade's successful - normal? The unit reboots and certainly reports the correct FW version. Also, I'm connected via the WAN port (LAN unplugged), which I assume is innocuous.

strace just after portmapper starts:

t@127.0.0.1:~# strace -p 13199
Process 13199 attached - interrupt to quit
poll([{fd=3, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}, {fd=4, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND, revents=POLLIN|POLLRDNORM}], 2, -1) = 1
accept(4, {sa_family=AF_INET, sin_port=htons(940), sin_addr=inet_addr("172.16.2.32")}, [16]) = 5
poll([{fd=3, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}, {fd=4, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}, {fd=5, events=POLLIN|POLLPRI|POLLRDNORM|P OLLRDBAND, revents=POLLIN|P
OLLRDNORM}], 3, -1) = 1
poll([{fd=5, events=POLLIN, revents=POLLIN}], 1, 35000) = 1
read(5, "\200\0\0(<\311o\0\0\0\0\0\0\0\0\2\0\1\206\240\0\0\0\2\0"..., 400) = 44
open("/etc/hosts.allow", O_RDONLY) = -1 EACCES (Permission denied)
time([1265688422]) = 1265688422
open("/etc/localtime", O_RDONLY) = -1 EACCES (Permission denied)
open("/etc/localtime", O_RDONLY) = -1 EACCES (Permission denied)
open("/etc/localtime", O_RDONLY) = -1 EACCES (Permission denied)
open("/etc/localtime", O_RDONLY) = -1 EACCES (Permission denied)
socket(PF_FILE, SOCK_DGRAM, 0) = 6
fcntl64(6, F_SETFD, FD_CLOEXEC) = 0
connect(6, {sa_family=AF_FILE, path="/dev/log"}, 16) = -1 ENOENT (No such file or directory)
close(6) = 0
write(5, "\0\0\1\214<\311o\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 400) = 400
write(5, "\200\0\0H\0\0\0\21\0\0\3\225\0\0\0\1\0\1\206\245\0\0\0"..., 76) = 76
poll([{fd=3, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}, {fd=4, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}, {fd=5, events=POLLIN|POLLPRI|POLLRDNORM|P OLLRDBAND, revents=POLLIN|P
OLLRDNORM}], 3, -1) = 1
poll([{fd=5, events=POLLIN, revents=POLLIN}], 1, 35000) = 1
read(5, "", 400) = 0
close(5) = 0
poll(



then it sits there polling. Without me doing anything:

poll([{fd=3, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND, revents=POLLIN|POLLRDNORM}, {fd=4, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}, {fd=-1}], 3, -1) = 1
recvmsg(3, {msg_name(16)={sa_family=AF_INET, sin_port=htons(32769), sin_addr=inet_addr("172.16.1.25")}, msg_iov(1)=[{"%\21\213K\0\0\0\0\0\0\0\2\0\1\206\240\0\0\0\2\0\0\0\5\0"..., 88
00}], msg_controllen=24, msg_control=0x8051060, , msg_flags=0}, 0) = 100
open("/etc/hosts.allow", O_RDONLY) = -1 EACCES (Permission denied)
time([1265688438]) = 1265688438
open("/etc/localtime", O_RDONLY) = -1 EACCES (Permission denied)
open("/etc/localtime", O_RDONLY) = -1 EACCES (Permission denied)
open("/etc/localtime", O_RDONLY) = -1 EACCES (Permission denied)
socket(PF_FILE, SOCK_DGRAM, 0) = 5
fcntl64(5, F_SETFD, FD_CLOEXEC) = 0
connect(5, {sa_family=AF_FILE, path="/dev/log"}, 16) = -1 ENOENT (No such file or directory)
close(5) = 0
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
Process 13199 detached
root@127.0.0.1:~#

Now that's just not right.

Thanks all - any help much appreciated,

Chris.

[unclebobuk]

Hi,
I'm running 2.1.09 and use NFS to mount stuff to my Sat box running a enigma2 linux image which seems to work. I also mount via NFS to my Popcorn Hour media streamer (which I believe is linux based). I've stuck with 2.1.09 as it seems to work (well most of the time!! ).

If i get a chance i'll try a test on my macbook for you... (running OSX)...

hope that helps. (probably not!)
regards,
bob.