Hunting a Memory Leak in 19.07.1 WRT1900ACS

No, in my case 802.11w is disabled

I did some more measurements - there is no change at all if 25M or 45M memory is used. The top task is always hostapd with 1.3% memory used - sorted by RES or SHR.


ieee80211w is also disabled on my config

Just saw that yesterday a new firmware (19.07.2) came out for my router.
I'll upgrade and update you with the results.

Just updated to 19.07.2 - still the same.

Just checked, last memory leak fix (3e11ddaf2ede4f105bc9ac91229623526371a7a2) was added on Feb 20. Therefore I would say we're up to date with the latest memory leak fixes.

I've seen with htop that the process uhttpd is getting quite a lot of MINFLT (Minor Page Faults), IDK if that may be related to the memory leak

Is valgrind an option?

There are differences to be found, I assume they were not tagged knowingly, but for example, master search, and 19.x tag search

Sure, I just need some guidance for setting up valgrind (AFAIK i would need to compile my firmware myself, right?).

74.1M/500M is not a leak. htop will NOT give you anything useful until you reach 90% memory usage.

300M/500M as by now

On a consumer router?!. I doubt it.

Can you order by VIRT / RES / SHR ?

luci-bwc 9 do you have LuCI open with Realtime Graphs?

UPDATE: if you suspect that it is uhttpd, then ssh into the router and stop it /etc/init.d/uhttps stop. Then check the memory usage. If it does not drop, stop other services one at a time: dnsmasq, nlbwmon, collect, etc.

Sorted by VIRT (sorting by RES/SHR won't change much).

Yes I do. I can close it to check if the memory gets freed.

Stop uhttpd and if nothing changes, run ps -w | grep -v "\[" to get a list of processes you are running.

Have you actually caught a OOM killer message in log output. I doubt very much that you meet with any success in trying to catch this with (h)top.

Have not used valgrind myself, packages are sitting there for download, but I assume you would have to instrument in some way, but then you have to contend with heisenberg.

That is a way too heavy hammer :wink: There are still things to learn by stopping services one at a time.

A constantly open LuCi session (or many) with auto refresh could do a lot of damage....

Stop of uhttpd:
root@OpenWrt:~# service uhttpd stop
no change in memory consumption
I start uhttpd again:
root@OpenWrt:~# service uhttpd start

ps not giving anything different than htop as far as i see

root@OpenWrt:~# ps -w | grep -v "\["
  PID USER       VSZ STAT COMMAND
    1 root      1356 S    /sbin/procd
  921 root      1008 S    /sbin/ubusd
  922 root       696 S    /sbin/askfirst /usr/libexec/login.sh
  939 root       808 S    /sbin/urngd
 1454 root      1028 S    /sbin/logd -S 64
 1479 root      1960 S    /sbin/rpcd -s /var/run/ubus.sock -t 30
 1625 root      1484 S    /sbin/netifd
 1663 root      1244 S    /usr/sbin/odhcpd
 1689 root      1076 S    /usr/sbin/crond -f -c /etc/crontabs -l 5
 1926 root      1072 S    udhcpc -p /var/run/udhcpc-eth1.2.pid -s /lib/netifd/dhcp.script -f -t 0 -i eth1.2 -x hostname:OpenWrt -C
 2012 root      1076 S<   /usr/sbin/ntpd -n -N -S /usr/sbin/ntpd-hotplug -p 0.openwrt.pool.ntp.org -p 1.openwrt.pool.ntp.org -p 2.o
 2359 root       836 S    /usr/sbin/dropbear -F -P /var/run/dropbear.1.pid -p 192.168.0.2:22 -p fdd3:fee7:5b46::1:22 -K 300 -T 3
 2730 dnsmasq   1172 S    /usr/sbin/dnsmasq -C /var/etc/dnsmasq.conf.cfg01411c -k -x /var/run/dnsmasq/dnsmasq.cfg01411c.pid
 2731 root      1700 S    /usr/sbin/hostapd -s -P /var/run/wifi-phy1.pid -B /var/run/hostapd-phy1.conf
 2751 root      1700 S    /usr/sbin/hostapd -s -P /var/run/wifi-phy0.pid -B /var/run/hostapd-phy0.conf
 3019 root       900 S    /usr/sbin/dropbear -F -P /var/run/dropbear.1.pid -p 192.168.0.2:22 -p fdd3:fee7:5b46::1:22 -K 300 -T 3
 3020 root      1080 S    -ash
 3293 root      2208 S    htop
 3388 root       900 S    /usr/sbin/dropbear -F -P /var/run/dropbear.1.pid -p 192.168.0.2:22 -p fdd3:fee7:5b46::1:22 -K 300 -T 3
 3393 root      1080 S    -ash
19954 root      1104 S    /usr/sbin/uhttpd -f -h /www -r OpenWrt -x /cgi-bin -t 60 -T 30 -k 20 -A 1 -n 3 -N 100 -R -p 0.0.0.0:80 -p
19961 root      1072 R    ps -w

I closed all the LuCi tabs (no other PCs are logged in via LuCi) but no change in memory consumption.

Run these one at a time, wait for a bit, and check memory:

/etc/init.d/uhttpd stop
/etc/init.d/odhcpd stop
/etc/init.d/cron stop
/etc/init.d/log stop
wifi down