25.12.x / BPI-R4 - netbird service fails to start upon boot due to missing /var/log/netbird folder (tmpfs)

I'm not entirely sure if this is a regression as of 25.12.1, or some version following that, but it most definitely wasn't happening on 25.10.x.

Issue is confirmed on two of my BPI-R4 devices. The netbird service fails to start due to the non-existent "netbird" folder in /var/log, and naturally the service doesn't start upon boot either, which is a big problem. If you create the netbird folder manually in /var/log and reboot, the netbird folder gets deleted due to /var/log being mounted to /tmp (tmpfs) But interestingly enough, the samba and ddns folders get recreated upon boot just fine.................. netbird is effectively broken because of this.

Image 941

Is this an openwrt issue or a netbird issue?

I see https://github.com/netbirdio/netbird/pull/6108, but I don't show it committed to a release yet...

I use Netbird on several OpenWRT routers and linux clients and have not encountered this problem
The log writes to /var/log/netbird/client.log which is the default log path

So not sure what the root cause is in your case.

Did you have an older version installed? I so remove that first.

Furthermore since two days Netbird is on version 0.73.2 so strange you still did get 0.66.2

So I would update again and see if you get the new 0.73.2

It is possible to set an other path for the logfile or have it write to syslog but it would be better if we can find the root cause.

0.66.2-r1 was the latest aarch64_cortex-a53 package as of yesterday. I do see it was updated to 0.73.2-r1 sometime today and went ahead and upgraded. Unfortunately, the issue persists. The netbird folder does not get recreated upon boot due to the issue cited by the linked PR 6108 which has yet to be committed.

Is it possible that /var/log isn't mounted to a tmpfs on your devices, because if it was, I suspect you'd be having the same problem. Best as I can tell, this is a two-part root cause. I suspect treatment of the /var/log directory for filogic devices changed for 25.12.x, and coupled with the longer-standing netbird issue cited by the open PR, you have a successful Swiss cheese alignment.

Please replace your /etc/init.d/netbird with the following code (make a backup first)

#!/bin/sh /etc/rc.common

START=99
STOP=10

USE_PROCD=1

start_service() {
	mkdir -p /var/log/netbird
	mkdir -p /root/.config/netbird
	mkdir -p /var/lib/netbird

	# There is no max number of rotated log files so delete those:
	find /var/log/netbird -name '*.log.gz' -mtime +2 -delete 2>/dev/null
	# However this only runs on restart so on low ram routers better write to syslog,
	# see below or make a cron job doing it daily

	procd_open_instance
	procd_set_param command /usr/bin/netbird
	procd_append_param command service run

	procd_set_param env NB_STATE_DIR="/root/.config/netbird"
	procd_append_param env NB_DISABLE_SSH_CONFIG="1"
	procd_append_param env NB_DNS_STATE_FILE="/var/lib/netbird/state.json"
	procd_set_param pidfile /var/run/netbird.pid

	# Set logging to syslog (logread), recommended for low RAM routers as the client log can grow,
	# remove the # on the line below to write to syslog
	#procd_append_param env NB_LOG_FILE=syslog

	# Set max log file size
	procd_append_param env NB_LOG_MAX_SIZE_MB="5"
	procd_close_instance
}

I attempted a simple insert of the mkdir commands into my existing init.d entry:

# For RedHat and cousins:
# chkconfig: - 99 01
# description: NetBird mesh network client
# processname: /usr/bin/netbird

### BEGIN INIT INFO
# Provides:          /usr/bin/netbird
# Required-Start:
# Required-Stop:
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Netbird
# Description:       NetBird mesh network client
### END INIT INFO

mkdir -p /var/log/netbird
mkdir -p /root/.config/netbird
mkdir -p /var/lib/netbird

cmd="/usr/bin/netbird "service" "run" "--log-level" "info" "--daemon-addr" "unix:///var/run/netbird.sock" "--log-file" "/var/log/netbird/client.log""

name=$(basename $(readlink -f $0))
pid_file="/var/run/$name.pid"
stdout_log="/var/log/netbird/$name.log"
stderr_log="/var/log/netbird/$name.err"

export SYSTEMD_UNIT=netbird
[ -e /etc/sysconfig/$name ] && . /etc/sysconfig/$name

get_pid() {
    cat "$pid_file"
}

is_running() {
    [ -f "$pid_file" ] && cat /proc/$(get_pid)/stat > /dev/null 2>&1
}

case "$1" in
    start)
        if is_running; then
            echo "Already started"
        else
            echo "Starting $name"
            
            $cmd >> "$stdout_log" 2>> "$stderr_log" &
            echo $! > "$pid_file"
            if ! is_running; then
                echo "Unable to start, see $stdout_log and $stderr_log"
                exit 1
            fi
        fi
    ;;
    stop)
        if is_running; then
            echo -n "Stopping $name.."
            kill $(get_pid)
            for i in $(seq 1 10)
            do
                if ! is_running; then
                    break
                fi
                echo -n "."
                sleep 1
            done
            echo
            if is_running; then
                echo "Not stopped; may still be shutting down or shutdown may have failed"
                exit 1
            else
                echo "Stopped"
                if [ -f "$pid_file" ]; then
                    rm "$pid_file"
                fi
            fi
        else
            echo "Not running"
        fi
    ;;
    restart)
        $0 stop
        if is_running; then
            echo "Unable to stop, will not attempt to start"
            exit 1
        fi
        $0 start
    ;;
    status)
        if is_running; then
            echo "Running"
        else
            echo "Stopped"
            exit 1
        fi
    ;;
    *)
    echo "Usage: $0 {start|stop|restart|status}"
    exit 1
    ;;
esac
exit 0

as well as a verbatim replacement with yours provided:

#!/bin/sh /etc/rc.common

START=99
STOP=10

USE_PROCD=1

start_service() {
	mkdir -p /var/log/netbird
	mkdir -p /root/.config/netbird
	mkdir -p /var/lib/netbird

	# There is no max number of rotated log files so delete it 
	# but this only runs on restart but so either use syslog on low ram routers,
	# see comment out line below or make a cron job doing it daily
	find /var/log/netbird -name '*.log.gz' -mtime +2 -delete 2>/dev/null

	procd_open_instance
	procd_set_param command /usr/bin/netbird
	procd_append_param command service run

	procd_set_param env NB_STATE_DIR="/root/.config/netbird"
	procd_append_param env NB_DISABLE_SSH_CONFIG="1"
	procd_append_param env NB_DNS_STATE_FILE="/var/lib/netbird/state.json"
	procd_set_param pidfile /var/run/netbird.pid

	# Set logging to syslog (logread), recommended for low RAM routers as the client log can grow 
	#procd_append_param env NB_LOG_FILE=syslog
	# Set max log file size
	procd_append_param env NB_LOG_MAX_SIZE_MB="5"
	procd_close_instance
}

but unfortunately neither results in /var/log/netbird or /var/lib/netbird being created at boot.

I also tried rebinding /var to /mnt to get it off the tmpfs:

mkdir -p /mnt/var
rsync -aHAX /var/ /mnt/var/
umount /var
mount --bind /mnt/var /var

/etc/fstab :

/mnt/var  /var  none  bind  0  0

but it just reverts to the tmpfs upon reboot. Something's gotta give here..........

Do you have the config file:

root@X86:~# ls -la /root/.config/netbird
drwxr-xr-x    3 root     root          4096 Jul  1 15:58 .
drwxr-xr-x    3 root     root          4096 Jun 29 12:57 ..
-rw-------    1 root     root            45 Mar  8 17:38 active_profile.json
-rw-------    1 root     root             7 Jun 16 15:57 active_profile.txt
-rw-------    1 root     root          2030 Jul  1 15:58 default.json
-rw-r--r--    1 root     root            47 Jul  1 16:00 resolv.conf
drwx------    2 root     root          4096 Jul  1 15:58 root
root@X86:~#

I deleted the /var/log/netbird directory and rebooted after reboot the log file and directory are recreated.

Next thing to try is to write to syslog, remove the comment on this line

#procd_append_param env NB_LOG_FILE=syslog

It is strange Netbird is used by a number of OpenWRT users and this has not surfaced yet.

What you show does not seem to be the OpenWRT Netbird implementation ?

:man_shrugging: Not sure. I only install packages from the repo using the UI. That's the init.d entry it populated for the aarch64_cortex-a53 package.

It looks like something went really wrong while installing.

/usr/bin/netbird is a binary file

/etc/init.d/netbird should look like this:

#!/bin/sh /etc/rc.common

START=99
STOP=10

USE_PROCD=1

start_service() {
	procd_open_instance
	procd_set_param command /usr/bin/netbird
	procd_append_param command service run

	procd_set_param env NB_STATE_DIR="/root/.config/netbird"
	procd_append_param env NB_DISABLE_SSH_CONFIG="1"
	procd_append_param env NB_DNS_STATE_FILE="/var/lib/netbird/state.json"

	procd_set_param pidfile /var/run/netbird.pid
	procd_close_instance
}

If it's not idiosyncratic to the package for that architecture, I really can't say. As for the directory tree, generating a backup archive creates the /root/.config/ directory with that file structure, but on the working filesystem on-device, those files reside in /var/lib/netbird, not /root/.config.

Does your router not use the arm A73 cores?

Edit: it is no wonder it is not working for you there is something seriously wrong with your installation

Does your router not use the arm A73 cores?

Yes it does, and I did find that somewhat odd. But the a53 repo is the one configured as distributed, and I do not purport to know these devices or package architectures better than the various OpenWRT developers.

I will say that this is the first time I'm having such issues. 25.10.x was working just fine on all my BPI-R4's.

Could be, I do not have your router but the way it is installed looks not correct.

No idea what the cause is but the way it is installed cannot work.

Maybe something went wrong while building the package, but that is beyond my knowledge.

I will see if i can install the package from source but even if that works it could be something specific for your router.

On My MT6000 I deleted Netbird and all its config and log files and directory and removed the package from the router, rebooted and confirmed all Netbird belongings are gone.

I used the GUI to update the software and install Netbird again, everything works, the log file is installed, directory created and config files in /root/.config/netbird (which was also deleted before)

So everything looks fine, so no idea what went wrong on your side :frowning:

I'm also having a separate issue where unfathomably, one of my setup keys keeps assigning a netbird hostname that 1) doesn't match the hostname of the device I'm configuring, and 2) is already in use at a different site. Strange AF, but that's entirely beyond my control, and running netbird service reconfigure fixes it well enough. That seems to account for the /root/.config weirdness, but the directory issue manifests local to the device at boot.

Can you run findmnt -T /var and tell me where it's mounted?

root@X86:~# findmnt -T /var
TARGET SOURCE FSTYPE OPTIONS
/tmp tmpfs tmpfs rw,nosuid,nodev,noatime
root@X86:~#

That doesn't make a damned bit of sense.... Maybe someone else with a BPI-R4 can chime in with a confirmation. All I can tell you is that it's not just one device, it's two of my BPI-R4's at different locations. Later today I should be able to confirm or deny a third that I've yet to upgrade to 25.12.x.

Installed mount-utils on another router (MT6000)

root@MT-6000:~# findmnt -T /var
TARGET SOURCE FSTYPE OPTIONS
/tmp tmpfs tmpfs rw,nosuid,nodev,noatime

So the same and looks good to me /var is symlinked to /tmp/var

root@MT-6000:~# ls -la /var
lrwxrwxrwx    1 root     root             3 Jun 30 09:11 /var -> tmp
root@MT-6000:~#

In any case, thanks for your help.