Zerotier segmentation fault on upgrade

openwrt 21.02.2

Just upgraded to zerotier_1.8.6-1_arm_cortex-a7_neon-vfpv4.ipk.

Now I get a Segmentation fault when doing zerotier-cli info

2 Likes

good on you!

on what hw ?

Linksys EA6350v3

that package appear to come from the snapshot dir, I don't think it's wise
to install snapshot ipks on a stable release, esp if you don't take care of any
dependencies there might be.

https://downloads.openwrt.org/snapshots/packages/arm_cortex-a7_neon-vfpv4/packages/

This package is also in the release packages.

https://downloads.openwrt.org/releases/21.02.2/packages/arm_cortex-a7_neon-vfpv4/packages/

opkg info zerotier
Package: zerotier
Version: 1.8.6-1
Depends: libc, libpthread, libstdcpp6, kmod-tun, ip, libminiupnpc, libnatpmp1
Status: install user installed
Section: net
Architecture: arm_cortex-a7_neon-vfpv4
Size: 447623
Filename: zerotier_1.8.6-1_arm_cortex-a7_neon-vfpv4.ipk

Interesting, I actually googled it, there was only one hit, the snapshots.

Did you by any chance try to update the dependencies too?

opkg install zerotier
Installing zerotier (1.8.6-1) to root...
Downloading https://downloads.openwrt.org/releases/21.02.2/packages/arm_cortex-a7_neon-vfpv4/packages/zerotier_1.8.6-1_arm_cortex-a7_neon-vfpv4.ipk
Installing kmod-tun (5.4.179-1) to root...
Downloading https://downloads.openwrt.org/releases/21.02.2/targets/ipq40xx/generic/packages/kmod-tun_5.4.179-1_arm_cortex-a7_neon-vfpv4.ipk
Installing libmnl0 (1.0.4-2) to root...
Downloading https://downloads.openwrt.org/releases/21.02.2/packages/arm_cortex-a7_neon-vfpv4/base/libmnl0_1.0.4-2_arm_cortex-a7_neon-vfpv4.ipk
Installing ip-tiny (5.11.0-3) to root...
Downloading https://downloads.openwrt.org/releases/21.02.2/packages/arm_cortex-a7_neon-vfpv4/base/ip-tiny_5.11.0-3_arm_cortex-a7_neon-vfpv4.ipk
Installing libminiupnpc (2.2.1-1) to root...
Downloading https://downloads.openwrt.org/releases/21.02.2/packages/arm_cortex-a7_neon-vfpv4/packages/libminiupnpc_2.2.1-1_arm_cortex-a7_neon-vfpv4.ipk
Installing libnatpmp1 (20150609-3) to root...
Downloading https://downloads.openwrt.org/releases/21.02.2/packages/arm_cortex-a7_neon-vfpv4/packages/libnatpmp1_20150609-3_arm_cortex-a7_neon-vfpv4.ipk
Configuring kmod-tun.
Configuring libmnl0.
Configuring ip-tiny.
Configuring libminiupnpc.
Configuring libnatpmp1.
Configuring zerotier.

zerotier-cli info
Segmentation fault

Has anyone found out what is causing the Segmentation fault issue that I am having? Thanks,

I upgraded to another router and got the same problem.
Version 1.6.6-2 has been working for months on both my routers.
How can I down grade back to 1.6.6-2?

Model Linksys WRT3200ACM
Architecture ARMv7 Processor rev 1 (v7l)
Target Platform mvebu/cortexa9
Firmware Version OpenWrt 21.02.1 r16325-88151b8303
Kernel Version 5.4.154

Upgrade
zerotier 1.6.6-2 to 1.8.6-1

zerotier-cli info
Segmentation fault

Mixing and matching kmods is guaranteed to crash. Mixing and matching userspace packages may or may not work.

If you want to get back to a working 21.02.1, reinstall the whole OS, and don't run opkg upgrade

Is there a reason you don't want to upgrade the whole OS to 21.02.2? That would be the official way to get the Zerotier that is included with 21.02.2.

The first router I installed Zerotier on is at 21.02.2 and it failed.
I have not had time to upgrade the other router yet, but clearly something is wrong with Zerotier 1.8.6-1.

Zerotier 1.8.6-1 does not work with OpenWrt 21.02.2 or OpenWrt 21.02.01.

So can someone please fix Zerotier 1.8.6-1.
Thanks

1 Like

I cannot replicate the problem here. I have just installed 21.02.2 and its Zerotier 1.8.6-1 on a Mi Router 4A Gigabit (MT7621 device), and it works.

I do not have any ipq401X or a WRT3200 hardware to test on. Suggest making a complete clean install on your device, and if it still does not work then that is a reportable bug.

1 Like

^^^ this.

Fresh instal of the official openwrt image, do not preserve settings. Do not restore a backup. Only configure the minimum required items relative to the default configuration (I.e enable WiFi, adjust wan settings if required).

Then
opkg update; opkg install zerotier

Similar story with GL-B1300 (ipq40xx). Upgraded packages while at 21.02.1. Zerotier started to give segfault. Flashed 21.02.02 with "Keep settings and retain the current configuration" and opkg-extras. opkg-extras hotplug installed every package that i was using . Zerotier gave segfault again.

Examined library dependencies with ldd (seems everything ok):
/lib/ld-musl-armhf.so.1 (0xb6ea1000)
libminiupnpc.so.17 => /usr/lib/libminiupnpc.so.17 (0xb6d13000)
libnatpmp.so.1 => /usr/lib/libnatpmp.so.1 (0xb6d00000)
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0xb6c38000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0xb6c1c000)
libc.so => /lib/ld-musl-armhf.so.1 (0xb6ea1000)

According to opkg Zerotier depends on those:
Package: zerotier
Version: 1.8.6-1
Depends: libc, libpthread, libstdcpp6, kmod-tun, ip, libminiupnpc, libnatpmp1

Evey application within router except zerotier has no problem:
config opkg 'auto'
list rpkg 'dnsmasq'
list ipkg 'avahi-dbus-daemon'
list ipkg 'chat'
list ipkg 'comgt'
list ipkg 'comgt-ncm'
list ipkg 'dbus'
list ipkg 'dnsmasq-full'
list ipkg 'etherwake'
list ipkg 'fwknopd'
list ipkg 'getdns'
list ipkg 'ip-full'
list ipkg 'ip-tiny'
list ipkg 'ipset'
list ipkg 'iptables-mod-conntrack-extra'
list ipkg 'iptables-mod-ipopt'
list ipkg 'kmod-crypto-hash'
list ipkg 'kmod-crypto-kpp'
list ipkg 'kmod-crypto-lib-blake2s'
list ipkg 'kmod-crypto-lib-chacha20'
list ipkg 'kmod-crypto-lib-chacha20poly1305'
list ipkg 'kmod-crypto-lib-curve25519'
list ipkg 'kmod-crypto-lib-poly1305'
list ipkg 'kmod-ifb'
list ipkg 'kmod-ipt-conntrack-extra'
list ipkg 'kmod-ipt-ipopt'
list ipkg 'kmod-ipt-ipset'
list ipkg 'kmod-ipt-nat6'
list ipkg 'kmod-ipt-raw'
list ipkg 'kmod-mii'
list ipkg 'kmod-nf-conntrack-netlink'
list ipkg 'kmod-nf-nat6'
list ipkg 'kmod-nfnetlink'
list ipkg 'kmod-sched-cake'
list ipkg 'kmod-sched-core'
list ipkg 'kmod-tun'
list ipkg 'kmod-udptunnel4'
list ipkg 'kmod-udptunnel6'
list ipkg 'kmod-usb-ehci'
list ipkg 'kmod-usb-net'
list ipkg 'kmod-usb-net-cdc-ether'
list ipkg 'kmod-usb-net-cdc-ncm'
list ipkg 'kmod-usb-net-huawei-cdc-ncm'
list ipkg 'kmod-usb-net-rndis'
list ipkg 'kmod-usb-ohci'
list ipkg 'kmod-usb-serial'
list ipkg 'kmod-usb-serial-option'
list ipkg 'kmod-usb-serial-wwan'
list ipkg 'kmod-usb-uhci'
list ipkg 'kmod-usb-wdm'
list ipkg 'kmod-usb2'
list ipkg 'kmod-wireguard'
list ipkg 'libatomic1'
list ipkg 'libavahi-dbus-support'
list ipkg 'libbpf0'
list ipkg 'libcap'
list ipkg 'libcap-ng'
list ipkg 'libdaemon'
list ipkg 'libdbus'
list ipkg 'libelf1'
list ipkg 'libevent2-7'
list ipkg 'libevent2-core7'
list ipkg 'libevent2-pthreads7'
list ipkg 'libexpat'
list ipkg 'libfko'
list ipkg 'libgd'
list ipkg 'libgmp10'
list ipkg 'libgnutls'
list ipkg 'libipset13'
list ipkg 'libjpeg-turbo'
list ipkg 'liblzo2'
list ipkg 'libminiupnpc'
list ipkg 'libmnl0'
list ipkg 'libnatpmp1'
list ipkg 'libnetfilter-conntrack3'
list ipkg 'libnettle8'
list ipkg 'libnfnetlink0'
list ipkg 'libopenssl1.1'
list ipkg 'libpcap1'
list ipkg 'libpng'
list ipkg 'libqrencode'
list ipkg 'librt'
list ipkg 'libstdcpp6'
list ipkg 'libusb-1.0-0'
list ipkg 'libusb-compat4'
list ipkg 'libuuid1'
list ipkg 'libwebp'
list ipkg 'libyaml'
list ipkg 'luci-app-fwknopd'
list ipkg 'luci-app-nut'
list ipkg 'luci-app-openvpn'
list ipkg 'luci-app-sqm'
list ipkg 'luci-app-upnp'
list ipkg 'luci-app-vpn-policy-routing'
list ipkg 'luci-app-wol'
list ipkg 'luci-compat'
list ipkg 'luci-proto-ncm'
list ipkg 'luci-proto-wireguard'
list ipkg 'miniupnpd'
list ipkg 'msmtp'
list ipkg 'nut'
list ipkg 'nut-common'
list ipkg 'nut-driver-blazer_usb'
list ipkg 'nut-driver-safenet'
list ipkg 'nut-server'
list ipkg 'nut-upsc'
list ipkg 'nut-upscmd'
list ipkg 'nut-upslog'
list ipkg 'nut-upsmon'
list ipkg 'nut-upsrw'
list ipkg 'nut-upssched'
list ipkg 'nut-web-cgi'
list ipkg 'obfs4proxy'
list ipkg 'openvpn-openssl'
list ipkg 'qrencode'
list ipkg 'resolveip'
list ipkg 'smcroute'
list ipkg 'sqm-scripts'
list ipkg 'stubby'
list ipkg 'tc-mod-iptables'
list ipkg 'tc-tiny'
list ipkg 'tor'
list ipkg 'usb-modeswitch'
list ipkg 'vpn-policy-routing'
list ipkg 'wireguard-tools'
list ipkg 'wwan'
list ipkg 'zerotier'
list ipkg 'zlib'

Any hint?
Sorry for long post :grinning:

As was stated previously, try resetting your router to defaults. Do not keep/restore settings. Do not make any changes to the default configuration that are not absolutely necessary (things such as changing your LAN subnet, WAN connection type, and enabling wifi are all fine, don't touch anything else for now). Do not install or upgrade any packages except for zerotier.

Does it still segfault?

Just did a clean update on the Linksys WRT3200ACM. Same problem.

root@OpenWrt:~# zerotier-cli info
Segmentation fault

After your suggestion flashed 21.02.02 withOUT "Keep settings and retain the current configuration". Except setting root password I did not configure anything. Did "opkg update" and installed zerotier with "opkg install zerotier". Still having segfault. Any more suggestions?

At this point, I think the following applies:

Could you install either strace and/or gdb, and try to see where it crashes?

Here is the strace:

root@gl-b1300:~# strace zerotier-cli
execve("/usr/bin/zerotier-cli", ["zerotier-cli"], 0xbe823de0 /* 13 vars */) = 0
set_tls(0xb6f995d8)                     = 0
set_tid_address(0xb6f9a18c)             = 21237
open("/etc/ld-musl-armhf.path", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/lib/libminiupnpc.so.17", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/libminiupnpc.so.17", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libminiupnpc.so.17", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fcntl64(3, F_SETFD, FD_CLOEXEC)         = 0
fstat64(3, {st_mode=S_IFREG|0644, st_size=40963, ...}) = 0
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\214\32\0\0004\0\0\0"..., 936) = 936
mmap2(NULL, 110592, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0xb6f04000
mmap2(0xb6f1d000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x9000) = 0xb6f1d000
close(3)                                = 0
open("/lib/libnatpmp.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/libnatpmp.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libnatpmp.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fcntl64(3, F_SETFD, FD_CLOEXEC)         = 0
fstat64(3, {st_mode=S_IFREG|0644, st_size=8195, ...}) = 0
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\370\7\0\0004\0\0\0"..., 936) = 936
mmap2(NULL, 77824, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0xb6ef1000
mmap2(0xb6f02000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x1000) = 0xb6f02000
close(3)                                = 0
open("/lib/libstdc++.so.6", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/libstdc++.so.6", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libstdc++.so.6", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fcntl64(3, F_SETFD, FD_CLOEXEC)         = 0
fstat64(3, {st_mode=S_IFREG|0755, st_size=743643, ...}) = 0
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0 v\5\0004\0\0\0"..., 936) = 936
mmap2(NULL, 819200, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0xb6e29000
mmap2(0xb6ee9000, 32768, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0xb0000) = 0xb6ee9000
mmap2(0xb6eef000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb6eef000
close(3)                                = 0
open("/lib/libgcc_s.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fcntl64(3, F_SETFD, FD_CLOEXEC)         = 0
fstat64(3, {st_mode=S_IFREG|0644, st_size=45059, ...}) = 0
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0000E\0\0004\0\0\0"..., 936) = 936
mmap2(NULL, 114688, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0xb6e0d000
mmap2(0xb6e27000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0xa000) = 0xb6e27000
close(3)                                = 0
mprotect(0xb6f1d000, 4096, PROT_READ)   = 0
mprotect(0xb6f02000, 4096, PROT_READ)   = 0
mprotect(0xb6ee9000, 16384, PROT_READ)  = 0
mprotect(0xb6e27000, 4096, PROT_READ)   = 0
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_ACCERR, si_addr=0x471d18} ---
+++ killed by SIGSEGV +++
Segmentation fault

I will look into gdb tomorrow.
If its possible to generate and collect core file on platform, I will happily put it here.