NFS broken again

[522138.025198] svc: failed to register nfsdv3 RPC service (errno 97).
[522138.073102] svc: failed to register lockdv1 RPC service (errno 97).
[522138.079548] NFSD: the nfsdcld client tracking upcall will be removed in 3.10. Please transition to using nfsdcltrack.
[522138.090304] NFSD: starting 90-second grace period (net c092a080)
[522261.874420] NFSD: Unable to end grace period: -110

updated to nfs-kernel-server 2.3.3-3 on 18.06.2, but still broken with the same error messages. nfsd not shown in ps.

1 Like

And from a client:

davygrvy@kahaluia:~$ sudo mount /media/BigMedia
mount.nfs4: requested NFS version or transport protocol is not supported

Upgrading packages is generally a bad idea, as OpenWrt does not yet have ABI versioning in its package-management system.

Simplest now is probably to save your config, list of self-installed packages, flash fresh, and install a self-consistent set of packages.

When you install a package, is the baseline package for that release installed or does opkg upgrade to the latest by default.

Edit:

Looking at the Packages Download page it seems they are continuously built for current release. It seems opkg has no choice but to install what is available.

How does the advice not to upgrade packages apply to installing a packages

@mbo2o raises a good point. It all "works" when the dependencies are either not installed, or provide the same ABI as what is being expected. If the already-installed TLS library is "different" (which has been a recent problem), then installing a "newly built" package against a newer API would cause a problem as well.

Does upgrading all upgradable packages workaround it

In the past, people who have used scripts to "upgrade all upgradable packages" have run into problems.

I've scratched my head as to what the failure mechanism there is, but its unfortunately not obvious to me.

With it being relatively easy to flash and install a "full set" of needed packages, I still consider that the best option for anyone not building their own images.

sysupgrade on master supplies

	-k           include in backup a list of current installed packages at
	             /etc/backup/installed_packages.txt

which it accomplishes with

                # Format: pkg-name<TAB>{rom,overlay,unkown}
                # rom is used for pkgs in /rom, even if updated later
                find /usr/lib/opkg/info -name "*.control" \( \
                        \( -exec test -f /rom/{} \; -exec echo {} rom \; \) -o \
                        \( -exec test -f /overlay/upper/{} \; -exec echo {} overlay \; \) -o \
                        \( -exec echo {} unknown \; \) \
                        \) | sed -e 's,.*/,,;s/\.control /\t/' > ${INSTALLED_PACKAGES}

(at least as of late May, 2019, prior to any future v19 release)

NFS was still broken for me before I noticed it had an upgrade

I recall a bunch of mailing-list activity around NFSv4, some of which might be related to NFS in terms of upgrading the packages.

On master

jeff@deb-devel:~/devel/openwrt/feeds/packages$ git log --pretty='%h %cd %s' --date=short --grep=nfs
c86fdd679 2019-05-26 nfs-kernel-server: compile with internal rpcgen
c01f3bf0b 2019-05-12 nfs-kernel-server: update to 2.3.4
20468cfef 2019-04-22 nfs-kernel-server: create nfs user and group
46fc281e9 2019-04-20 nfs-utils: Fix compilation on some platforms
6d65505c2 2019-03-21 nfs-kernel-server: fix dependencies
edbac15a5 2019-02-19 nfs-kernel-server: update patches
274ce493e 2019-02-17 nfs-kernel-server: fix freeaddrinfo usage in nfs-kernel-server, because freeaddrinfo in musl after the 1.1.21 update, doesn't handly NULL pointers (which seems to spec conform) see https://www.openwall.com/lists/musl/2019/02/03/3 for more info
95db98bd7 2018-09-29 nfs-kernel-server: add support for NFSv4
3fc7f7b8c 2018-09-11 nfs-kernel-server: fix missing libbsd dependency
e5216bb0d 2018-09-09 nfs-kernel-server: update to 2.3.3
cfba4f087 2018-08-31 nfs-kernel-server: Switch to xz tarball
deaa180a5 2018-08-13 netatalk: remove librpc depends
22ebb5a8d 2018-08-13 nfs-kernel-server: switch to libtirpc, enable ipv6
9b3c6720a 2018-08-11 Merge pull request #6706 from Andy2244/nfs-kernel-server-res_querydomain-fix
19dfe3b17 2018-08-08 nfs-kernel-server: fix missing host symbol res_querydomain/missing-include-dir
460dbee22 2018-08-06 samba4: add nfs-kernel-server/host build dependency
b1be3f9c0 2018-08-02 nfs-kernel-server: add -Wno-error=format-security to TARGET_CFLAGS to a fix compily error
0d1f48893 2018-05-28 nfs-kernel-server: update to 2.3.2
1 Like

Same here. I built a fresh firmware with image builder (18.06.2). After restart nfsd:

Tue Jun  4 12:27:06 2019 kern.warn kernel: [80189.144502] svc: failed to register nfsdv3 RPC service (errno 124).
Tue Jun  4 12:27:06 2019 kern.warn kernel: [80189.182449] NFSD: the nfsdcld client tracking upcall will be removed in 3.10. Please transition to using nfsdcltrack.
Tue Jun  4 12:27:06 2019 kern.info kernel: [80189.193261] NFSD: starting 90-second grace period (net 80444040)
Tue Jun  4 12:27:06 2019 daemon.warn rpc.statd[9290]: Running as root.  chown /var/lib/nfs to choose different user```

If anyone finds the magic to making NFS work, please post the solution. Thanks. Although the DLNA music server is working on the router, I can't add anything, at the moment

Same issue here:

Tue Jun  4 20:47:50 2019 kern.info kernel: [    7.562924] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
Tue Jun  4 20:47:52 2019 kern.warn kernel: [   16.544397] svc: failed to register nfsdv3 RPC service (errno 97).
Tue Jun  4 20:47:52 2019 kern.warn kernel: [   16.633794] NFSD: the nfsdcld client tracking upcall will be removed in 3.10. Please transition to using nfsdcltrack.

Related logs:

Tue Jun  4 20:47:52 2019 daemon.err rpc.mountd[2845]: Could not bind socket: (98) Address in use
Tue Jun  4 20:47:52 2019 daemon.warn rpc.mountd[2845]: mountd: No V2 or V3 listeners created!
Tue Jun  4 20:47:52 2019 daemon.err rpc.statd[2844]: failed to create RPC listeners, exiting

I can't use PXE boot (which is serving large files over NFS) anymore. Running OpenWrt 18.06.2 r7676-cddd7b4c77.

@jeff do you mean that the upcoming 'nfs-kernel-server' update might fix the issue? It doesn't look that I have it at the latest version and I do have a rather fresh OpenWRT install:

# opkg list-installed | grep nfs-kernel-server
nfs-kernel-server - 2.3.3-3
1 Like

Related github issue: https://github.com/openwrt/packages/issues/8930

1 Like
root@OpenWrt:~# mv /etc/rc.d/S20network /etc/rc.d/S14network
root@OpenWrt:~# reboot
root@OpenWrt:~# Connection to openwrt.lan closed by remote host.
Connection to openwrt.lan closed.

waited a bit... then on the client:

davygrvy@kahaluia:~$ sudo mount /media/BigMedia
[sudo] password for davygrvy: 
mount.nfs: requested NFS version or transport protocol is not supported

Examining the router:

root@OpenWrt:~# dmesg |tail
[   16.515162] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[   18.470155] br-lan: port 1(eth0.1) entered learning state
[   18.528762] svc: failed to register nfsdv3 RPC service (errno 97).
[   18.612527] svc: failed to register lockdv1 RPC service (errno 97).
[   18.619027] NFSD: the nfsdcld client tracking upcall will be removed in 3.10. Please transition to using nfsdcltrack.
[   18.629709] NFSD: starting 90-second grace period (net c092a080)
[   21.520133] br-lan: port 1(eth0.1) entered forwarding state
[   21.525861] IPv6: ADDRCONF(NETDEV_CHANGE): br-lan: link becomes ready
[  140.640080] NFSD: Unable to end grace period: -110
[  355.116753] NFSD: Unable to create client record on stable storage: -110
root@OpenWrt:~# 

I thought starting portmap after the network was up would have solved this, but nope. I did fix minidlna coming up before the disk was ready by reordering, though

1 Like

This isn't shown as a running process. Is portmap managing this?

/usr/sbin/rpc.statd -p 32778 -o 32779

This is the error message when manually starting in foreground

root@OpenWrt:~# /usr/sbin/rpc.statd -p 32778 -o 32779 -F
sm-notify: Version 2.3.3 starting
sm-notify: Already notifying clients; Exiting!

Normal or abnormal?

For those of you not willing to wait for a permanent fix, here's a temporary one. It looks that the 'nfs-kernel-server_2.1.1-1' version build for OpenWRT 17.01 is working perfectly fine with OpenWRT 18.06.2 stable build which I'm using.

OK, step by step, in a way which even less advanced users should understand what they're doing :wink:

  1. Remove the current version of 'nfs-kernel-server':
opkg remove --autoremove nfs-kernel-server
  1. Update opkg repo, we'll still need the dependencies for 'nfs-kernel-server_2.1.1-1' to be fetched and installed:
opkg update
  1. Check the architecture you're using by executing:
grep DISTRIB_ARCH /etc/openwrt_release
  1. Get the 'nfs-kernel-server_2.1.1-1' for your architecture. In my case it's 'arm_cortex-a9_vfpv3' as I'm using Linksys WRT32X. Replace the <DISTRIB_ARCH> with proper value in command below:
wget http://downloads.openwrt.org/releases/packages-17.01/<DISTRIB_ARCH>/packages/nfs-kernel-server_2.1.1-1_<DISTRIB_ARCH>.ipk
  1. Install the package and it's dependencies:
opkg install nfs-kernel-server_2.1.1-1_<DISTRIB_ARCH>.ipk
  1. Enable services:
/etc/init.d/portmap enable
/etc/init.d/nfsd enable
  1. Reboot:
reboot

Now everything seems to be fine. Logs are OK:

logread | grep nfsd
Sat Jun  8 09:23:42 2019 kern.info kernel: [    7.555859] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
Sat Jun  8 09:23:44 2019 kern.warn kernel: [   16.805921] NFSD: the nfsdcld client tracking upcall will be removed in 3.10. Please transition to using nfsdcltrack.
logread | grep rpc
Sat Jun  8 09:23:44 2019 daemon.notice rpc.mountd[2872]: Version 2.1.1 starting
Sat Jun  8 09:23:44 2019 daemon.notice rpc.statd[2871]: Version 2.1.1 starting
Sat Jun  8 09:23:44 2019 daemon.warn rpc.statd[2871]: Flags: No-Daemon
Sat Jun  8 09:23:44 2019 daemon.notice rpc.statd[2871]: Initializing NSM state
Sat Jun  8 09:23:44 2019 daemon.warn rpc.statd[2871]: Running as root.  chown /var/lib/nfs to choose different user

I'm also able to PXE boot with resources shared over NFS.

Hope that helps!

5 Likes

Thank you for the instruction. You can import variables fro the /etc/openwrt_release.

Simply do the following:

opkg update
BASE_URL=http://downloads.openwrt.org/releases/packages-17.01
source /etc/openwrt_release
opkg remove --autoremove nfs-kernel-server
wget $BASE_URL/$DISTRIB_ARCH/packages/nfs-kernel-server_2.1.1-1_$DISTRIB_ARCH.ipk
opkg install nfs-kernel-server_2.1.1-1_$DISTRIB_ARCH.ipk
rm nfs-kernel-server_2.1.1-1_$DISTRIB_ARCH.ipk
/etc/init.d/portmap enable
/etc/init.d/nfsd enable
/etc/init.d/portmap start
/etc/init.d/nfsd start

Also you should solve the conflicts:

# opkg install nfs-kernel-server_2.1.1-1_$DISTRIB_ARCH.ipk
Installing nfs-kernel-server (2.1.1-1) to root...
Collected errors:
 * satisfy_dependencies_for: Cannot satisfy the following dependencies for nfs-kernel-server:
 *      libwrap
 *      librpc
 *      kmod-fs-nfsd
 *      kmod-fs-nfs
 *      portmap
 * opkg_install_cmd: Cannot install package nfs-kernel-server.

For the "conflicts" - you have to run 'opkg update' prior to installing the .ipk package - the error you get informs you that opkg wasn't able to find required packages in the repository. It is covered by point 2. in the instruction I wrote. You have to remember that opkg local package repository is cleared every time you reboot.

1 Like

Thank for this point. I missed step 2 without a code listing.

I have the same problem - downgrade help to me. Did someone know, if the bugfix is planned?