NFS broken again

davygrvy · June 1, 2019, 1:22am

NFS was still broken for me before I noticed it had an upgrade

jeff · June 1, 2019, 1:37am

I recall a bunch of mailing-list activity around NFSv4, some of which might be related to NFS in terms of upgrading the packages.

On master

jeff@deb-devel:~/devel/openwrt/feeds/packages$ git log --pretty='%h %cd %s' --date=short --grep=nfs
c86fdd679 2019-05-26 nfs-kernel-server: compile with internal rpcgen
c01f3bf0b 2019-05-12 nfs-kernel-server: update to 2.3.4
20468cfef 2019-04-22 nfs-kernel-server: create nfs user and group
46fc281e9 2019-04-20 nfs-utils: Fix compilation on some platforms
6d65505c2 2019-03-21 nfs-kernel-server: fix dependencies
edbac15a5 2019-02-19 nfs-kernel-server: update patches
274ce493e 2019-02-17 nfs-kernel-server: fix freeaddrinfo usage in nfs-kernel-server, because freeaddrinfo in musl after the 1.1.21 update, doesn't handly NULL pointers (which seems to spec conform) see https://www.openwall.com/lists/musl/2019/02/03/3 for more info
95db98bd7 2018-09-29 nfs-kernel-server: add support for NFSv4
3fc7f7b8c 2018-09-11 nfs-kernel-server: fix missing libbsd dependency
e5216bb0d 2018-09-09 nfs-kernel-server: update to 2.3.3
cfba4f087 2018-08-31 nfs-kernel-server: Switch to xz tarball
deaa180a5 2018-08-13 netatalk: remove librpc depends
22ebb5a8d 2018-08-13 nfs-kernel-server: switch to libtirpc, enable ipv6
9b3c6720a 2018-08-11 Merge pull request #6706 from Andy2244/nfs-kernel-server-res_querydomain-fix
19dfe3b17 2018-08-08 nfs-kernel-server: fix missing host symbol res_querydomain/missing-include-dir
460dbee22 2018-08-06 samba4: add nfs-kernel-server/host build dependency
b1be3f9c0 2018-08-02 nfs-kernel-server: add -Wno-error=format-security to TARGET_CFLAGS to a fix compily error
0d1f48893 2018-05-28 nfs-kernel-server: update to 2.3.2

artemklevtsov · June 4, 2019, 5:27am

Same here. I built a fresh firmware with image builder (18.06.2). After restart nfsd:

Tue Jun  4 12:27:06 2019 kern.warn kernel: [80189.144502] svc: failed to register nfsdv3 RPC service (errno 124).
Tue Jun  4 12:27:06 2019 kern.warn kernel: [80189.182449] NFSD: the nfsdcld client tracking upcall will be removed in 3.10. Please transition to using nfsdcltrack.
Tue Jun  4 12:27:06 2019 kern.info kernel: [80189.193261] NFSD: starting 90-second grace period (net 80444040)
Tue Jun  4 12:27:06 2019 daemon.warn rpc.statd[9290]: Running as root.  chown /var/lib/nfs to choose different user```

davygrvy · June 4, 2019, 5:35pm

If anyone finds the magic to making NFS work, please post the solution. Thanks. Although the DLNA music server is working on the router, I can't add anything, at the moment

Hekarath · June 4, 2019, 9:07pm

Same issue here:

Tue Jun  4 20:47:50 2019 kern.info kernel: [    7.562924] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
Tue Jun  4 20:47:52 2019 kern.warn kernel: [   16.544397] svc: failed to register nfsdv3 RPC service (errno 97).
Tue Jun  4 20:47:52 2019 kern.warn kernel: [   16.633794] NFSD: the nfsdcld client tracking upcall will be removed in 3.10. Please transition to using nfsdcltrack.

Related logs:

Tue Jun  4 20:47:52 2019 daemon.err rpc.mountd[2845]: Could not bind socket: (98) Address in use
Tue Jun  4 20:47:52 2019 daemon.warn rpc.mountd[2845]: mountd: No V2 or V3 listeners created!

Tue Jun  4 20:47:52 2019 daemon.err rpc.statd[2844]: failed to create RPC listeners, exiting

I can't use PXE boot (which is serving large files over NFS) anymore. Running OpenWrt 18.06.2 r7676-cddd7b4c77.

@jeff do you mean that the upcoming 'nfs-kernel-server' update might fix the issue? It doesn't look that I have it at the latest version and I do have a rather fresh OpenWRT install:

# opkg list-installed | grep nfs-kernel-server
nfs-kernel-server - 2.3.3-3

artemklevtsov · June 6, 2019, 2:52am

Related github issue: https://github.com/openwrt/packages/issues/8930

davygrvy · June 6, 2019, 2:53pm

root@OpenWrt:~# mv /etc/rc.d/S20network /etc/rc.d/S14network
root@OpenWrt:~# reboot
root@OpenWrt:~# Connection to openwrt.lan closed by remote host.
Connection to openwrt.lan closed.

waited a bit... then on the client:

davygrvy@kahaluia:~$ sudo mount /media/BigMedia
[sudo] password for davygrvy: 
mount.nfs: requested NFS version or transport protocol is not supported

Examining the router:

root@OpenWrt:~# dmesg |tail
[   16.515162] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[   18.470155] br-lan: port 1(eth0.1) entered learning state
[   18.528762] svc: failed to register nfsdv3 RPC service (errno 97).
[   18.612527] svc: failed to register lockdv1 RPC service (errno 97).
[   18.619027] NFSD: the nfsdcld client tracking upcall will be removed in 3.10. Please transition to using nfsdcltrack.
[   18.629709] NFSD: starting 90-second grace period (net c092a080)
[   21.520133] br-lan: port 1(eth0.1) entered forwarding state
[   21.525861] IPv6: ADDRCONF(NETDEV_CHANGE): br-lan: link becomes ready
[  140.640080] NFSD: Unable to end grace period: -110
[  355.116753] NFSD: Unable to create client record on stable storage: -110
root@OpenWrt:~#

I thought starting portmap after the network was up would have solved this, but nope. I did fix minidlna coming up before the disk was ready by reordering, though

davygrvy · June 7, 2019, 10:32pm

This isn't shown as a running process. Is portmap managing this?

/usr/sbin/rpc.statd -p 32778 -o 32779

This is the error message when manually starting in foreground

root@OpenWrt:~# /usr/sbin/rpc.statd -p 32778 -o 32779 -F
sm-notify: Version 2.3.3 starting
sm-notify: Already notifying clients; Exiting!

Normal or abnormal?

Hekarath · June 8, 2019, 10:02am

For those of you not willing to wait for a permanent fix, here's a temporary one. It looks that the 'nfs-kernel-server_2.1.1-1' version build for OpenWRT 17.01 is working perfectly fine with OpenWRT 18.06.2 stable build which I'm using.

OK, step by step, in a way which even less advanced users should understand what they're doing

Remove the current version of 'nfs-kernel-server':

opkg remove --autoremove nfs-kernel-server

Update opkg repo, we'll still need the dependencies for 'nfs-kernel-server_2.1.1-1' to be fetched and installed:

opkg update

Check the architecture you're using by executing:

grep DISTRIB_ARCH /etc/openwrt_release

Get the 'nfs-kernel-server_2.1.1-1' for your architecture. In my case it's 'arm_cortex-a9_vfpv3' as I'm using Linksys WRT32X. Replace the <DISTRIB_ARCH> with proper value in command below:

wget http://downloads.openwrt.org/releases/packages-17.01/<DISTRIB_ARCH>/packages/nfs-kernel-server_2.1.1-1_<DISTRIB_ARCH>.ipk

Install the package and it's dependencies:

opkg install nfs-kernel-server_2.1.1-1_<DISTRIB_ARCH>.ipk

Enable services:

/etc/init.d/portmap enable
/etc/init.d/nfsd enable

Reboot:

reboot

Now everything seems to be fine. Logs are OK:

logread | grep nfsd
Sat Jun  8 09:23:42 2019 kern.info kernel: [    7.555859] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
Sat Jun  8 09:23:44 2019 kern.warn kernel: [   16.805921] NFSD: the nfsdcld client tracking upcall will be removed in 3.10. Please transition to using nfsdcltrack.

logread | grep rpc
Sat Jun  8 09:23:44 2019 daemon.notice rpc.mountd[2872]: Version 2.1.1 starting
Sat Jun  8 09:23:44 2019 daemon.notice rpc.statd[2871]: Version 2.1.1 starting
Sat Jun  8 09:23:44 2019 daemon.warn rpc.statd[2871]: Flags: No-Daemon
Sat Jun  8 09:23:44 2019 daemon.notice rpc.statd[2871]: Initializing NSM state
Sat Jun  8 09:23:44 2019 daemon.warn rpc.statd[2871]: Running as root.  chown /var/lib/nfs to choose different user

I'm also able to PXE boot with resources shared over NFS.

Hope that helps!

artemklevtsov · June 8, 2019, 11:15am

Thank you for the instruction. You can import variables fro the /etc/openwrt_release.

Simply do the following:

opkg update
BASE_URL=http://downloads.openwrt.org/releases/packages-17.01
source /etc/openwrt_release
opkg remove --autoremove nfs-kernel-server
wget $BASE_URL/$DISTRIB_ARCH/packages/nfs-kernel-server_2.1.1-1_$DISTRIB_ARCH.ipk
opkg install nfs-kernel-server_2.1.1-1_$DISTRIB_ARCH.ipk
rm nfs-kernel-server_2.1.1-1_$DISTRIB_ARCH.ipk
/etc/init.d/portmap enable
/etc/init.d/nfsd enable
/etc/init.d/portmap start
/etc/init.d/nfsd start

Also you should solve the conflicts:

# opkg install nfs-kernel-server_2.1.1-1_$DISTRIB_ARCH.ipk
Installing nfs-kernel-server (2.1.1-1) to root...
Collected errors:
 * satisfy_dependencies_for: Cannot satisfy the following dependencies for nfs-kernel-server:
 *      libwrap
 *      librpc
 *      kmod-fs-nfsd
 *      kmod-fs-nfs
 *      portmap
 * opkg_install_cmd: Cannot install package nfs-kernel-server.

Hekarath · June 8, 2019, 6:09pm

For the "conflicts" - you have to run 'opkg update' prior to installing the .ipk package - the error you get informs you that opkg wasn't able to find required packages in the repository. It is covered by point 2. in the instruction I wrote. You have to remember that opkg local package repository is cleared every time you reboot.

artemklevtsov · June 8, 2019, 6:42pm

Thank for this point. I missed step 2 without a code listing.

rutu · June 14, 2019, 9:17am

I have the same problem - downgrade help to me. Did someone know, if the bugfix is planned?

artemklevtsov · June 14, 2019, 9:33am

Track the github issue posted above.

rutu · June 14, 2019, 9:45am

thank You - I didn't notice, sorry

chunkeey · June 14, 2019, 4:42pm

For people with !SMALL_FLASH (i.e NOT 4MiB Flash devices) please test again with a up-to-date snapshot. Because

https://git.openwrt.org/?p=openwrt/openwrt.git;a=commit;h=fcb41decf6c622482b20af45a77e62db8d95046e

Enabled FHANDLE and friends, which from what I know prevented the nfs daemon from working in later releases.

rutu · June 17, 2019, 7:13am

in the dev verzion of openwrt works nfs fine now

Magnets · June 21, 2019, 5:27pm

Doesn't look like the fix made it in 18.06.3

https://openwrt.org/releases/18.06/changelog-18.06.3

chunkeey · June 21, 2019, 7:04pm

@magnets no, nobody posted a patch for 18.06.

system · July 1, 2019, 7:04pm

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.