How can I make nlbwmon persist data across reboot

Installing and Using OpenWrt
1

So. I know nlbwmon is saving to RAM by default to avoid flash wear. This is not about that.

I connected a USB flash drive to the router, and changed nlbwmon settings to save the file there. Still, whenever the router reboots, all data for the current accounting period gets reset. Older accounting periods are still there, and completely fine.

My main suspicion is that it has to do with system time, i.e because nlbwmon starts before ntp, the database files 'from the future' gets deleted. My question is, how can I prevent that from happening?

2

Does changing the startup sequence of nlbwmon after the ntp fix your issue?

3

how do I do that though?

4

ls -la /etc/rc.d/ ?

5

What for? nlbwmon is 60, sysntpd is 98.

Either way, some insight I got so far:

  • ntp has to wait for pppoe link to be up, so even changing nlbwmon to 99, it will still load before ntp sync.
  • current time gets save to disk on shutdown and loaded on reboot - so the problem will only happen on unexpected power loss.
  • even pulling the plug now will not help test this... I need to wait a few days until the saved time is older than the current accounting period.
  • according to the log, vnstat does act to migitate this problem, so why can't nlbwmon do it to?
Wed Jan  4 13:56:17 2023 daemon.info vnstatd[1581]: Info: Latest database update is in the future (db: 2023-01-04 13:57:00 > now: 2023-01-04 13:56:17). Giving the system clock up to 5 minutes to sync before continuing.
Wed Jan  4 13:58:10 2023 daemon.info vnstatd[1581]: Info: Latest database update is no longer in the future (db: 2023-01-04 13:57:00 <= now: 2023-01-04 13:58:10), continuing.

:point_up: This is so beautiful

6

You can use an internal lan ntp if you don't want to wait for pppoe.
You can change the startup sequence to start nlbwmon after ntp.
You can add some sleep XX inside the script to delay it even more.

7

If there's power loss to the whole house (or the whole neighborhood), the local lan ntp will be down too, and may take longer than openwrt to start up.

Changing the startup sequence won't help, as I mentioned.

adding sleep somewhere might or might not help... but I have no idea where it can be safely added, and to what effect.

8

Disable the autostart of the nlbwmon service and install ntpdate.

/etc/init.d/nlbwmon disable
opkg update; opkg install ntpdate

Use this script to start nlbwmon only after successful time synchronization.

cat << "EOF" > /root/nlbwmon.scr
#!/bin/sh

while true; do

ntpdate 0.openwrt.pool.ntp.org >/dev/null 2>&1

  if [ "$?" = "0" ]; then 
  /etc/init.d/nlbwmon start
  break
  else
  sleep 5
  fi

done
EOF

Insert the following into /etc/rc.local above exit 0.

/root/nlbwmon.scr &
1 Like
9

What value have you set for the db commit interval?

uci export nlbwmon

10

Use with an external disk :

mkdir -p /mnt/disk/tools/nlbwmon
uci set nlbwmon.@nlbwmon[0].database_directory='/mnt/disk/tools/nlbwmon'
uci commit
service nlbwmon restart
11

Ah.
Add these lines on /etc/sysctl.conf

# nlbwmon
net.core.rmem_max=104857600
net.core.wmem_max=104857600

run:

sysctl -p

You will not lost anymore data.

12

No, nlbwmon doesn't rely on ntp. It tracks summary data over the specified period. The data doesn't use a timestamp.

The db structure and data it captures is:

{"columns":["family","proto","port" "mac" "ip",		    "conns"  "rx_bytes"  "rx_pkts"  "tx_bytes" "tx_pkts" "layer7"]  # Data Captured Follows
               4      "TCP"   443   "mac" "10.10.1.115"  52782   1985037313   1564434   220182927  527160    "HTTPS"]

nlbwmon doesn't need to do it.

Unlike nlbwmon, vnstat tracks data over time ie: how much traffic in the last 5 minutes/hour/day/week/month/year, so it does rely on accurate time.

Try changing option commit_interval '24h' to something like every hour
option commit_interval '60m' in /etc/config/nlbwmon`.

You can just add this to /etc/config/nlbwmon. This will survive an upgrade while sysctrl won't.

`option netlink_buffer_size '10485760'`
1 Like
13

Allow me to disagree. Sure, the database entries are not date dependent, but the accounting periods themselves are (and so the separate database files). If the current system time is set to before the latest database file, then nlbwmon will update a previous one. Then when time rights itself (by ntp), nlbwmon will assume a new accounting period had started and create the latest database from scratch - losing all my data.

it's set to 6h for a while now. still I lost 4 days of data after the last power failure.

I have both set for a while too. This is a completely different issue.

14

Faced this issue . Looking into code to see what could be the reason. First step is to probably add logging and see which part of code decides to delete data for current accounting period.

15

Based on these two commits, it seems that code "intends" to handle this scenario :