I made another change to my build: I enabled CONFIG_BUSYBOX_CONFIG_FEATURE_PS_LONG
to be able to read RSS and not just the virtual memory size (VSZ). Looking at VSZ probably overestimated the memory cost of OpenSSL because the 3.5mb libcrypto.so.3 is mapped into https-dns-proxy, hostapd, wpa_supplicant and uhttpd, but most of those pages should be R/RX and shared across all those processes.
3 days in I have a memory use graph from luci-statistics, and the difference is pretty stunning: Memory use hovers around 40mb and is stable plus/minus 5 MB as WiFi clients come and go.
The change I hacked into my build is this:
--- openwrt-orig/build_dir/target-mipsel_24kc_musl/linux-ramips_mt7621/linux-6.6.100/drivers/net/ethernet/mediatek/mtk_eth_soc.c 2025-08-11 21:05:42.020814611 +0300
+++ openwrt/build_dir/target-mipsel_24kc_musl/linux-ramips_mt7621/linux-6.6.100/drivers/net/ethernet/mediatek/mtk_eth_soc.c 2025-08-11 21:51:09.364872361 +0300
@@ -5424,14 +5424,14 @@
.desc_size = sizeof(struct mtk_tx_dma),
.dma_max_len = MTK_TX_DMA_BUF_LEN,
.dma_len_offset = 16,
- .dma_size = MTK_DMA_SIZE(2K),
+ .dma_size = MTK_DMA_SIZE(512),
.fq_dma_size = MTK_DMA_SIZE(2K),
},
.rx = {
.desc_size = sizeof(struct mtk_rx_dma),
.irq_done_mask = MTK_RX_DONE_INT,
.dma_l4_valid = RX_DMA_L4_VALID,
- .dma_size = MTK_DMA_SIZE(2K),
+ .dma_size = MTK_DMA_SIZE(512),
.dma_max_len = MTK_TX_DMA_BUF_LEN,
.dma_len_offset = 16,
},
I have now reverted it to see if the problem comes back or if something else fixed the problem by accident.
You can get detailed PS info via htop or btop without rebuilding.
I get slightly less than 1MB per client added using default -mbedtls hostapd.
Here’s a graph of the memory use without the patch. Most of it was idle overnight hours.
The mem use starts at 42mb but went up to 60mb even though the device was idle. It went down, then back up to 70 mb as one of the WiFi clients was downloading Windows updates. The memory was eventually freed and during the do-nothing hours it oscillated between ~45 and ~55.
Again note that the busybox applet free and this graph have a different idea of what is considered used memory. While LuCI says “Used: Last: 48.4M”, free says “used 67584” (kb). (I think this is a choice in how they treat the different categories in /proc/meminfo and of no deep concern.)
Comparing the /proc/meminfo and process list of the original and patch kernel suggests that the extra memory use is accounted to userspace processes - all of the processes that deal with network in one way or the other have a 1 to 5 MB larger resident size. In total, 13MB more are used by userspace processes (49192 kb orig, 36052 patched). Virtual size is pretty much unchanged, only 1MB larger (91660 vs 90432).
The biggest hogs(non-exhaustive list) are (+1920kb), hostapd (+2044), https-dns-proxy (+4868), wpa_supplicant (+1536), uhttp (916). Weirdly there is alo the kernel helper [watchdogd] that is 1228kb larger.
I think the stock build has the same fundamental issue, but it doesn’t show up in actually observable crashes. https-dns-proxy is not standard issue. I am pushing my device a bit further by using openssl instead of mbedtls. And it almost makes it anyhow - it only visibly crashes when high network use, clients coming and going and LuCI are in use at the same time.
I think I’ve found the answer to What? Now the question is Why? Is mtk_eth_soc.c also involved with wifi and not just wired ethernet (Wouldn’t surprise me). Do large ethernet-allocated buffers get reused for the WiFi driver even though the WiFi driver wants smaller ones?
So re-reading the patches and MRs shows that at least half of my change was in commit 15887235, which got cherry-picked into the 24.10 branch in 642b5b61. However, the mt7621 specific change got lost in 3a2a2c99 (kernel: refresh patches), presumably by accident.
I’ll send an MR to restore it and see if the rx size change is enough to stabilise the router.
For comparisons sake here are graphs of 1 day and 7 day memory use history with the current openwrt24.10 branch. You can see memory use going up to 70 MB at times. The dynamic use is not as obvious in the 7 day graph though. Unless there is sustained heavy traffic it goes down fast enough that the graph doesn’t show it.
I didn’t have particularly heavy network traffic in those 24 hours. A Windows update on one box, otherwise just generic web surfing.
Pull request created: https://github.com/openwrt/openwrt/pull/19784 .
In the pull request I only set the RX DMA size to 512 because that’s what the patch did before it was partially lost in a rebase. I am running this build myself now and I’ll see how the memory use of tx=2K, rx=512 compares to tx=rx=512 and rx=rx=2K. I expect it to be somewhere in the middle, which should be good enough to keep the device stable.
The patch was merged today. Thanks everyone for the assistance.
Here is a graph of the memory use of the patched kernel. There wasn’t any major wan→lan/wifi traffic though, so judgement is still out. At first glance it looks good, average used at 41mb vs50 before. It is slightly higher than the 40 mb rx=tx=512, and it is going up and down a bit more than my graph from post 22, but it is certainly an improvement.
I’ll mark this topic solved. If I still run out of memory I’ll either have to find savings elsewhere or convince devs to lower the tx dma buffer to e.g. 1K - which I expect to be a tough sell considering that it was raised to work around difficult to reproduce hardware hangs.
Out of curiosity I tested if the DMA buffer size has a noticeable impact on network performance. The answer is no. I tested this with iperf running on my wire-connected raspberry pi 4 and the router itself. Average throughput over 100 seconds was:
rx=tx=512:
[ 5] 0.00-100.00 sec 7.93 GBytes 681 Mbits/sec 0 sender
[ 5] 0.00-100.01 sec 7.93 GBytes 681 Mbits/sec receiver
rx=512,tx=2k:
[ 5] 0.00-100.00 sec 8.15 GBytes 700 Mbits/sec 0 sender
[ 5] 0.00-100.01 sec 8.15 GBytes 700 Mbits/sec receiver
rx=tx=2k
[ 5] 0.00-100.00 sec 7.98 GBytes 685 Mbits/sec 0 sender
[ 5] 0.00-100.01 sec 7.98 GBytes 685 Mbits/sec receiver
Generally the CPU of the MR600 can't keep up with a full duplex gigabit line, even if it is just simple iperf.
Grateful you followed up that your patch is correctly integrated. The difference indded is within noise floor, like air humidity etc.
Those tests were over wire, so air quality shouldn't matter I hope 
. Since I didn't turn off wifi and kick out all other devices though the router's CPU likely had other small tasks to do while I ran those benchmarks. So I agree that the differences I saw are just noise.
Incidentally the 700 mbit/s are also the maximum I get between a WiFi client and the wire-attached RPI. So I wonder if WiFi speed is actually limited by the CPU rather than the WiFi hardware and signal quality itself. But that's different topic than the memory issue.
And before the thread gets frozen: Here's a ~6 day memory graph. Average 45 mb. But the router has seen plenty of traffic, had 10+ clients connected and has been stable all the time - even when using LuCI during heavy traffic.
So while I guess further optimization might be possible if we find an alternative to the 2K transmit DMA buffer, things are good enough for my use.
