25.12.x / BPI-R4 - netbird service fails to start upon boot due to missing /var/log/netbird folder (tmpfs)

Installing and Using OpenWrt
21

Keep us informed if you find anything, this really is a mystery

22

IIRC, this is related more specifically to the netbird service command. Never ever use it, or you’re on your own. The service in OpenWrt is managed by procd, and the correct command is service netbird. However, this command is only useful for disabling, enabling, starting, and stopping the netbird service. Generally speaking, netbird down/up is good enough to make the necessary configurations. Also, like @egc, I tried to reproduce this bug on three targets, if I’m not mistaken, in the past, and it never happened on OpenWrt OOTB.

Besides being possible, it is not recommended to upgrade or install packages in OpenWrt with apk add or apk upgrade, sometimes this can cause headaches. The recommended way is to use owut, luci-app-attendedsysupgrade, or firmware-selector, and flash a new image.

Maybe this can help:

  • Make a backup of your OpenWrt config and any other important files.
  • If netbird is running and working, check whether you have any files in /var/lib/netbird (This dir is a volatile, don't use, more info in this commit) and none in /root/.config/netbird. Backup all the files inside /var/lib/netbird.
  • Use one of the tools above to generate a firmware for your target with the packages you need.
  • Flash the new firmware using LuCI or SSH, after the boot, stop the netbird service if it is running, and do not use netbird service stop, the correct command is service netbird stop. Check whether /root/.config/netbird exists. If it does not, create the directory. If it exists, remove any files and move the backup files from /var/lib/netbird there. The state.json file is usually unnecessary and can be removed. Start the netbird service (service netbird start).

Also, if you don’t know it yet, I recommend reading the excellent guide by @egc, if you want to go YOLO and configure everything from scratch.

Maybe in the future I’ll patch the netbird service command so it simply doesn’t work and prints a message telling users to use procd instead.

23

I can't speak for anyone else, naturally, but the reconfigure command came in clutch for me. I have no idea why it kept double-parking on hostnames, but deleting the peer, running reconfigure, and up'ing the interface with the appropriate setup key again was how I was ultimately able to fix that. So my humble vote is to keep it in some comparable flavor of function.

As for the issue at-large, blowing the image away and starting from scratch without hand-carrying any configs over was ultimately enough. I had to reconfigure each device from scratch and employ some fairly fancy footwork in the process since they're all geographically dislocated and netbird is my typical in-band management facility across geographies. I suspect some unaccounted-for change from 10.x to 12.x or cross-pollination with netbird was the brass tack.

Thanks for all your help, both of you.

24

Yeah, that’s what I thought before, if someone wants to use it, just use it. In the end, it’s your device, your rules. But now I think this will end up causing headaches for people, myself included, because I’ll be trying to solve something that is outside my scope and beyond what most people know.

My personal opinion is that netbird service <something> is broken, and it will probably bite everyone in the ass sooner or later. The problem is that netbird service <something> seems to assume that everyone uses systemd (which, in some ways, is also questionable in how it is managed, or maybe not anymore, I manually manage my systemd services, so maybe it has been fixed), or SysV (while OpenWrt uses procd).

netbird also treats every Linux system as just Linux, but OpenWrt is a very specific kind of Linux. For example, /var is a volatile directory.

Also, in the future, I’ll try to do this for the next OpenWrt release, I’ve been working on making the procd service more useful and better integrated with uci and OpenWrt in general. I already shared a very old snippet with @egc (I’m busy, but I haven’t abandoned the package), but I don’t like the direction this is going, so I decided to create a netbird protocol, managed by netifd, which seems like the right approach. This will also fix one bug that no one reported, if you restart the network, for example with service network restart, netbird will never reconnect the tunnel.

About the hostnames, as far as I know, the only time you can set one is during the first run, with netbird up --login/--setup-key --hostname <my_fancy_hostname_here>. If you omit --hostname during netbird up --login/--setup-key, the router’s hostname will be used instead. After that, it can only be changed in the management interface, never by the client. So I don’t know what netbird service reconfigure is doing for you. Are you using some --service-env flag?

25

Nope, no extra flags or switches besides --setup-key, and there were at least a few instances recently where the up command was assigning the hostname of a different, geographically dislocated device of the same model and sysupgrade image (already up and listed under Peers on the Netbird dashboard), even though I used a different setup key than the device it double-parked on used.

26

I'm not following. See my test with three clean OpenWrt 25.12 images:

root@openwrt-foo:~# netbird up --setup-key E093CE97-3659-4476-AB40-EDF4E889F6B0
Connected
root@openwrt-foo:~# netbird status --json | jsonfilter -e '@.fqdn'
openwrt-foo.netbird.cloud

==================================================

root@openwrt-bar:~# netbird up --setup-key E093CE97-3659-4476-AB40-EDF4E889F6B0
Connected
root@openwrt-bar:~# netbird status --json | jsonfilter -e '@.fqdn'
openwrt-bar.netbird.cloud

==================================================

root@openwrt-baz:~# netbird up --setup-key E093CE97-3659-4476-AB40-EDF4E889F6B0
Connected
root@openwrt-baz:~# netbird status --json | jsonfilter -e '@.fqdn'
openwrt-baz.netbird.cloud

root@openwrt-foo:~# netbird up --setup-key E093CE97-3659-4476-AB40-EDF4E889F6B0 --hostname wearethesame
Connected
root@openwrt-foo:~# netbird status --json | jsonfilter -e '@.fqdn'
wearethesame-198-98.netbird.cloud

==================================================

root@openwrt-bar:~# netbird up --setup-key E093CE97-3659-4476-AB40-EDF4E889F6B0 --hostname wearethesame
Connected
root@openwrt-bar:~# netbird status --json | jsonfilter -e '@.fqdn'
wearethesame.netbird.cloud

==================================================

root@openwrt-baz:~# netbird up --setup-key E093CE97-3659-4476-AB40-EDF4E889F6B0 --hostname wearethesame
Connected
root@openwrt-baz:~# netbird status --json | jsonfilter -e '@.fqdn'
wearethesame-51-90.netbird.cloud

In the first test, the three OpenWrt devices with unique hostnames take the correct hostname/name and fqdn:

  • openwrt-foo --> openwrt-foo.netbird.cloud
  • openwrt-bar --> openwrt-bar.netbird.cloud
  • openwrt-baz --> openwrt-baz.netbird.cloud

In the second test, the three OpenWrt devices with "conflicting hostnames" use the same name in management, but the fqdn does not, because of the conflict.

  • openwrt-foo --> wearethesame --> wearethesame-198-98.netbird.cloud
  • openwrt-bar --> wearethesame --> wearethesame.netbird.cloud
  • openwrt-baz --> wearethesame --> wearethesame-51-90.netbird.cloud

All of these results are expected. Can you show exactly what you are doing, step by step?