Hello,
I’m experiencing daily reboots with my Netgear R7800 with stable OpenWrt firmware. As I’m monitoring all my OpenWRT devices with Prometheus, I realized that when the reboot occurs, the memory is very high. As you can see in the graph below, the memory always keep growing, till next reboot. How can I fix this ?
Start by disabling all non essential services it's running.
That is be a bit ambiguous what version exactly, 23.05.5? 24.10.1? Are you using an NSS version?
ubus call system board
will show the necessary information
But follow @frollic's advice, so also disable Prometheus etc.
These devices used to have random crashes if they weren’t set to “performance” mode on the cpu governor.
Is that still a thing?
At some point before it reboots (at around 384 MiB, for example), run the ps command on the OpenWrt device via SSH and save its output to a file on your PC for analysis. You can probably set up some kind of alerting mechanism in Prometheus to do this automatically for you.
Example command:
ssh root@OpenWrt.lan ps > processes.txt
Sort processes.txt by the VSZ[1] column and this should easily tell you the offending process(es).
OpenWrt's ps doesn't seem to output the RSS column, but VSZ should be good enough for this purpose. ↩︎
Here is the versions. I recently moved to 24.10 (by changing parameters to DSA). But the problem was already there in 23.05
And I also have ( and need ) :
Nothing more installed that the base firmware + these 4 features.
Check where these programs are writing their logs and databases. If they're writing to somewhere in /var, this will lead to increasing RAM usage because OpenWrt symlinks /var to /tmp by default.
I’ll try to make a script to get process information when RAM is near saturation. I’ll tell you what I found.
After it has been running for a while the other parts of the status page would help - memory and storage. Especially the latter to see how large Temp space can grow and how much is being consumed.
Hello all,
I finally found the problem, and I’m guilty ! It was related to a script that I deployed some months ago, that I forgot.
To sync the devices name accross my OpenWRT devices, I deployed a script based on this discussion : Automatically Sync DHCP Leases to Dumb Access Points
And to avoid using cron, I placed it in hotplug.d/dhcp, to trigger it when a device connects. But It seems that the scp hangs sometimes (I didn’t analyse why), and as there are many devices connecting to the network, the scp processes start to increase dramatically till the device reboot.
I just deactivate this script, and no more memory increase. As this feature is not really necessary, I will not try to debug it.
This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.
