Low memory issues

Installing and Using OpenWrt
1

I'm running into memory issues with my (dumb) AP. Running latest-ish snapshot:

root@RP-AC56:~# ubus call system board
{
        "kernel": "6.6.35",
        "hostname": "RP-AC56",
        "system": "MediaTek MT7621 ver:1 eco:3",
        "model": "Asus RP-AC56",
        "board_name": "asus,rp-ac56",
        "rootfs_type": "squashfs",
        "release": {
                "distribution": "OpenWrt",
                "version": "SNAPSHOT",
                "revision": "r26815-b4069fad02",
                "target": "ramips/mt7621",
                "description": "OpenWrt SNAPSHOT r26815-b4069fad02"
        }
}

wsl@DESKTOP-IESHI3C:~/openwrt$ ./scripts/diffconfig.sh
CONFIG_TARGET_ramips=y
CONFIG_TARGET_ramips_mt7621=y
CONFIG_TARGET_ramips_mt7621_DEVICE_asus_rp-ac56=y
CONFIG_DEVEL=y
CONFIG_TOOLCHAINOPTS=y
CONFIG_CCACHE=y
CONFIG_CLEAN_IPKG=y
CONFIG_EXPERIMENTAL=y
# CONFIG_GCC_USE_VERSION_13 is not set
CONFIG_GCC_USE_VERSION_14=y
CONFIG_GCC_VERSION="14.1.0"
CONFIG_GCC_VERSION_14=y
# CONFIG_GDB is not set
# CONFIG_KERNEL_WERROR is not set
CONFIG_PACKAGE_cgi-io=y
# CONFIG_PACKAGE_dnsmasq is not set
# CONFIG_PACKAGE_firewall4 is not set
CONFIG_PACKAGE_irqbalance=y
# CONFIG_PACKAGE_jansson is not set
# CONFIG_PACKAGE_kmod-lib-crc-ccitt is not set
# CONFIG_PACKAGE_kmod-nft-fib is not set
# CONFIG_PACKAGE_kmod-ppp is not set
CONFIG_PACKAGE_liblucihttp=y
CONFIG_PACKAGE_liblucihttp-ucode=y
# CONFIG_PACKAGE_libmnl is not set
# CONFIG_PACKAGE_libnftnl is not set
# CONFIG_PACKAGE_libpthread is not set
CONFIG_PACKAGE_luci-base=y
CONFIG_PACKAGE_luci-mod-admin-full=y
CONFIG_PACKAGE_luci-mod-network=y
CONFIG_PACKAGE_luci-mod-status=y
CONFIG_PACKAGE_luci-mod-system=y
CONFIG_PACKAGE_luci-proto-ipv6=y
CONFIG_PACKAGE_luci-theme-bootstrap=y
# CONFIG_PACKAGE_nftables-json is not set
# CONFIG_PACKAGE_odhcpd-ipv6only is not set
# CONFIG_PACKAGE_opkg is not set
# CONFIG_PACKAGE_ppp is not set
# CONFIG_PACKAGE_procd-ujail is not set
CONFIG_PACKAGE_rpcd=y
CONFIG_PACKAGE_rpcd-mod-file=y
CONFIG_PACKAGE_rpcd-mod-iwinfo=y
CONFIG_PACKAGE_rpcd-mod-luci=y
CONFIG_PACKAGE_rpcd-mod-rrdns=y
CONFIG_PACKAGE_rpcd-mod-ucode=y
CONFIG_PACKAGE_ucode-mod-html=y
CONFIG_PACKAGE_ucode-mod-math=y
CONFIG_PACKAGE_uhttpd=y
CONFIG_PACKAGE_uhttpd-mod-ubus=y

root@RP-AC56:~# ps
  PID USER       VSZ STAT COMMAND
    1 root      2392 S    /sbin/procd
    2 root         0 SW   [kthreadd]
    3 root         0 SW   [pool_workqueue_]
    4 root         0 IW<  [kworker/R-rcu_g]
    5 root         0 IW<  [kworker/R-rcu_p]
    6 root         0 IW<  [kworker/R-slub_]
    7 root         0 IW<  [kworker/R-netns]
   12 root         0 IW<  [kworker/R-mm_pe]
   13 root         0 IW   [rcu_tasks_trace]
   14 root         0 SW   [ksoftirqd/0]
   15 root         0 IW   [rcu_sched]
   16 root         0 SW   [migration/0]
   17 root         0 SW   [cpuhp/0]
   18 root         0 SW   [cpuhp/1]
   19 root         0 SW   [migration/1]
   20 root         0 SW   [ksoftirqd/1]
   23 root         0 IW<  [kworker/R-inet_]
   24 root         0 IW   [kworker/u4:1-ph]
   25 root         0 IW   [kworker/1:1-eve]
   26 root         0 SW   [oom_reaper]
   27 root         0 IW<  [kworker/R-write]
   28 root         0 SW   [kcompactd0]
   29 root         0 IW<  [kworker/R-pencr]
   30 root         0 IW<  [kworker/R-pdecr]
   31 root         0 IW<  [kworker/R-kbloc]
   32 root         0 SW   [watchdogd]
   40 root         0 SW   [kswapd0]
   41 root         0 IW<  [kworker/0:1H-kb]
   42 root         0 IW<  [kworker/R-kthro]
   84 root         0 IW   [kworker/u4:2-ev]
   96 root         0 RW   [kworker/u4:3-ev]
  108 root         0 IW   [kworker/0:2-eve]
  125 root         0 SW   [spi0]
  212 root         0 SW   [napi/mtk_eth-5]
  213 root         0 SW   [napi/mtk_eth-6]
  235 root         0 IW<  [kworker/R-mld]
  237 root         0 IW<  [kworker/R-ipv6_]
  238 root         0 IW   [kworker/1:4-eve]
  239 root         0 IW<  [kworker/1:1H-kb]
  240 root         0 IW<  [kworker/R-dsa_o]
  350 root         0 SW   [irq/16-mt7530]
  375 root         0 SW   [irq/22-keys]
  376 root         0 SW   [irq/23-keys]
  507 root         0 SWN  [jffs2_gcd_mtd6]
  562 ubus      1568 S    /sbin/ubusd
  563 root      1112 S    /sbin/askfirst /usr/libexec/login.sh
  598 root      1264 S    /sbin/urngd
  684 root         0 IW<  [kworker/R-cfg80]
  689 root         0 RW   [napi/phy0-7]
  690 root         0 SW   [napi/phy0-8]
  697 root         0 RW   [mt76-tx phy0]
  715 root         0 SW   [napi/phy1-9]
  716 root         0 SW   [napi/phy1-10]
  717 root         0 SW   [mt76-tx phy1]
  881 logd      2312 S    /sbin/logd -S 64
  935 root      3808 S    /sbin/rpcd -s /var/run/ubus/ubus.sock -t 30
 1046 root      1332 S    /usr/sbin/dropbear -F -P /var/run/dropbear.1.pid -p 22 -K 300 -T 3 -W 262144
 1097 root      4788 S    /usr/sbin/hostapd -s -g /var/run/hostapd/global
 1098 root      4772 S    /usr/sbin/wpa_supplicant -n -s -g /var/run/wpa_supplicant/global
 1160 root      2868 S    /sbin/netifd
 1342 root      2064 S    /usr/sbin/uhttpd -f -h /www -r RP-AC56 -x /cgi-bin -u /ubus -t 60 -T 30 -k 20 -A 1 -n 3 -N 100 -R -p 0.0.0.0:80 -p [::]:80
 1529 root      1388 S    /usr/sbin/irqbalance -f -c 2 -t 10
 1935 root      1428 S    udhcpc -p /var/run/udhcpc-br-lan.pid -s /lib/netifd/dhcp.script -f -t 0 -i br-lan -x hostname:RP-AC56 -C -R -O 121
 1991 root      1432 S<   /usr/sbin/ntpd -n -N -S /usr/sbin/ntpd-hotplug -p 192.168.178.1
 2871 root         0 IW<  [kworker/1:0H]
 2888 root         0 IW   [kworker/0:0-rcu]
 3105 root         0 IW<  [kworker/0:2H]
 3263 root      1352 R    /usr/sbin/dropbear -F -P /var/run/dropbear.1.pid -p 22 -K 300 -T 3 -W 262144 -2 9
 3264 root      1432 S    -ash
 3275 root      1436 R    ps

root@RP-AC56:~# free
              total        used        free      shared  buff/cache   available
Mem:          54392       37944       10860         248        5588        1244
Swap:             0           0           0

root@RP-AC56:~# cat /proc/meminfo
MemTotal:          54392 kB
MemFree:           10860 kB
MemAvailable:       1236 kB
Buffers:               0 kB
Cached:             4156 kB
SwapCached:            0 kB
Active:             5260 kB
Inactive:           1696 kB
Active(anon):       2244 kB
Inactive(anon):      816 kB
Active(file):       3016 kB
Inactive(file):      880 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:          2848 kB
Mapped:             2944 kB
Shmem:               248 kB
KReclaimable:       1432 kB
Slab:              14908 kB
SReclaimable:       1432 kB
SUnreclaim:        13476 kB
KernelStack:         576 kB
PageTables:          312 kB
SecPageTables:         0 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:       27196 kB
Committed_AS:       6436 kB
VmallocTotal:    1048372 kB
VmallocUsed:        3220 kB
VmallocChunk:          0 kB
Percpu:              120 kB

Running without procd-ujail because I suspected it to be the issue, but it's not. I've got another device with similar specs and setup but got plenty of available memory there (main difference: mt76x8 instead of mt7621).

Slab and SUnreclaim are somewhat noticeably higher, but I'm not sure how to debug this further.

2

https://openwrt.org/es/supported_devices/864_warning

2 Likes
3

Hi.
To enhance @frollic 's answer, your device is too short in memory to run kernel 6.6 correctly. You should consider running 23.05 on such device, and keep it this way.

2 Likes
4

Well yes, I do know that. Hence I usually remove everything I don't need. The wiki entry alludes to that as well:

  • 64 MB RAM may have some issues with stability, depending on your hardware and use cases, although it is enough for basic usage

The 64MB are plenty for my use case with other devices (and the same packages), it just appears to be this one (or its platform?). The processes alone would yield around ~35MB, shouldn't there be enough left?

5

Modern wireless drivers for modern wifi standards need more RAM than their predecessors, mt76 is even considerably better in this regard than ath10k/ ath11k/ ath12k (which eat RAM for breakfast). Additionally each connected client does drive up the RAM requirements (again less of a problem with mt76, than with ath10k/ ath11k/ ath12k where this effect is considerably more pronounced), with IoT and smarthome adding a lot of wireless stations to modern networks, you do see where quite a bit of the RAM is going to.

1 Like
6

I understand this, but I still don't understand why this device now has these memory issues. Like I said I have another one, also ~64MB RAM, also mt76, also kernel 6.6:

root@MR200:~# cat /proc/meminfo
MemTotal:          58892 kB
MemFree:           12880 kB
MemAvailable:      11636 kB
Buffers:               0 kB
Cached:            20540 kB
SwapCached:            0 kB
Active:             7464 kB
Inactive:          16116 kB
Active(anon):       3392 kB
Inactive(anon):        0 kB
Active(file):       4072 kB
Inactive(file):    16116 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:          3056 kB
Mapped:             4796 kB
Shmem:               352 kB
KReclaimable:       1896 kB
Slab:               9552 kB
SReclaimable:       1896 kB
SUnreclaim:         7656 kB
KernelStack:         416 kB
PageTables:          368 kB
SecPageTables:         0 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:       29444 kB
Committed_AS:       7052 kB
VmallocTotal:    1048372 kB
VmallocUsed:        1352 kB
VmallocChunk:          0 kB
Percpu:               32 kB

This one even has some more processes running (+ relayd & odhcpd), while the other one is a simple dumb AP (with WDS uplink).

I'd like to understand what's consuming the available RAM with the RP-AC56.

1 Like
7

You can try remove this patch .

1 Like
8

Thank you, yes omitting that one helps a lot. So, it's not mt76, but the mediatek ethernet driver being assigned more memory for the TX path for QDMA?

This kind of makes ~64MB memory for mt7621 (borderline) unusable - more than it already was for "normal" router operations, but even for a dumb AP with stripped packages it's not sufficient anymore. However, thank you for the information, nevertheless. I'll go ahead and replace this one then.

9

You can always add zram-swap, you have only 8/59m strictly resident rest can be freed on spot or swapped out.

10

That’s why the wiki is calling out 8/64 devices as potential problems

11

you can keep this patch, just change some numbers

search for static const struct mtk_soc_data mt7621_data

change for .tx
.dma_size = MTK_DMA_SIZE(1K),
.fq_dma_size = MTK_DMA_SIZE(512),

change for .rx
dma_size = MTK_DMA_SIZE(128), //even 16 could be safe here

You will get even more free RAM than you had before.

2 Likes
12

also in patch https://github.com/openwrt/openwrt/blob/27b6838afacfcd79be52b2563b870ae7b3caf8f6/target/linux/generic/backport-6.6/752-29-v6.10-net-ethernet-mtk_ppe-Change-PPE-entries-number-to-16.patch

decrease MTK_PPE_ENTRIES_SHIFT from 4 back to 3 or 2

Also you can decrease size for fq_codel qdiscs or replace them with pfifo_fast

1 Like
13

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.