NordLynx causing OOM in combination with SQM

Installing and Using OpenWrt
1

Hello everyone, my first post here, so sorry if it's not according to all the rules...

TP-Link Archer AX23 v1 with OpenWrt 24.10.0
VDSL connection 100/10
SQM CAKE installed so I can overcome bufferbloat

When I'm using NordVPN on my desktop, and if I'm connected via NordLynx protocol, I can't download anything... CPU Load spikes to 5+, WiFi connections die, internet dies, and in system logs I can find some OOM messages... I'll post some of them here, but it's often a different process killed, but never mind, the point is in OOM, right?

Sat Mar  1 23:59:08 2025 kern.info kernel: [  110.666490] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=hostapd,pid=1568,uid=101
Sat Mar  1 23:59:08 2025 kern.err kernel: [  110.679365] Out of memory: Killed process 1568 (hostapd) total-vm:5012kB, anon-rss:384kB, file-rss:0kB, shmem-rss:0kB, UID:101 pgtables:24kB oom_score_adj:0

Sun Mar  2 12:50:17 2025 kern.info kernel: [46361.190405] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=rpcd,pid=1161,uid=0
Sun Mar  2 12:50:17 2025 kern.err kernel: [46361.202814] Out of memory: Killed process 1161 (rpcd) total-vm:3944kB, anon-rss:760kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:24kB oom_score_adj:0

Sun Mar  2 12:50:17 2025 kern.info kernel: [46361.727455] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=hostapd,pid=4306,uid=101
Sun Mar  2 12:50:17 2025 kern.err kernel: [46361.740290] Out of memory: Killed process 4306 (hostapd) total-vm:4928kB, anon-rss:368kB, file-rss:0kB, shmem-rss:0kB, UID:101 pgtables:20kB oom_score_adj:0

This does not happen on NordLynx if I'm not using CAKE SQM, and it's not happening if I'm using CAKE SQM but with NordVPN using OpenVPN (both TCP and UDP)... So, only NordLynx+CAKE is killing my connection...

Anything I should configure differently?

2

Your router only sees to have 128 MiB of ram, and apparently you are surfing close to the edge. No real idea why the combination of cake and wireguard traffic causes the OOM though.
Could I convince you to post the following (copy and paste as preformatted text as you did with the error messages, please):

Do not run a VPN, ideally reboot the rputer
2) tc -s qdisc
3) start some mild rate limited traffic through the VPN (maybe some radio streaming)
4) tc -s qdisc
5) start one of the downloads that are know to cause the OOM but without the VPN
6) try to get a tc -s qdisc from after the download started for real
7) start one of the downloads that are know to cause the OOM but through NordLynx
8) try to get a tc -s qdisc from after the download started for real but befor the OOM hits (might be tricky)

Repeat, but connect the NordLynx computer via ethernet cable to the router instead of WiFi.

The goal is to get an idea of the load passing through your cake instances and the resulting memory requirements for the different scenarios.

3

I tried to post full logs, but I got error: An error occurred: Body is limited to 32000 characters; you entered 80118.

So I cut the parts I think are not important...
If you need full logs, tell me, and I'll upload to pastebin

16:05 no vpn, no traffic

qdisc cake 8005: dev pppoe-wan root refcnt 2 bandwidth 9700Kbit besteffort triple-isolate nonat nowash no-ack-filter split-gso rtt 100ms noatm overhead 34 mpu 68
 Sent 920016 bytes 2895 pkt (dropped 4, overlimits 1095 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 44800b of 4Mb
 capacity estimate: 9700Kbit
 min/max network layer size:           40 /    1460
 min/max overhead-adjusted size:       74 /    1494
 average network hdr offset:            0

                  Tin 0
  thresh       9700Kbit
  target            5ms
  interval        100ms
  pk_delay       2.11ms
  av_delay        252us
  sp_delay         22us
  backlog            0b
  pkts             2899
  bytes          924632
  way_inds           29
  way_miss          355
  way_cols            0
  drops               4
  marks               0
  ack_drop            0
  sp_flows            0
  bk_flows            1
  un_flows            0
  max_len         16060
  quantum           300

qdisc ingress ffff: dev pppoe-wan parent ffff:fff1 ----------------
 Sent 2610948 bytes 3676 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc cake 8006: dev ifb4pppoe-wan root refcnt 2 bandwidth 86Mbit besteffort triple-isolate nonat wash no-ack-filter split-gso rtt 100ms noatm overhead 34 mpu 68
 Sent 2609500 bytes 3675 pkt (dropped 1, overlimits 1701 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 158112b of 4300000b
 capacity estimate: 86Mbit
 min/max network layer size:           40 /    1460
 min/max overhead-adjusted size:       74 /    1494
 average network hdr offset:            0

                  Tin 0
  thresh         86Mbit
  target            5ms
  interval        100ms
  pk_delay        154us
  av_delay         40us
  sp_delay          8us
  backlog            0b
  pkts             3676
  bytes         2610948
  way_inds            1
  way_miss          340
  way_cols            0
  drops               1
  marks               0
  ack_drop            0
  sp_flows            1
  bk_flows            1
  un_flows            0
  max_len          1460
  quantum          1514

16:07 (two minutes of radio streaming)

qdisc cake 8005: dev pppoe-wan root refcnt 2 bandwidth 9700Kbit besteffort triple-isolate nonat nowash no-ack-filter split-gso rtt 100ms noatm overhead 34 mpu 68
 Sent 2608612 bytes 9861 pkt (dropped 12, overlimits 4364 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 56000b of 4Mb
 capacity estimate: 9700Kbit
 min/max network layer size:           29 /    1460
 min/max overhead-adjusted size:       68 /    1494
 average network hdr offset:            0

                  Tin 0
  thresh       9700Kbit
  target            5ms
  interval        100ms
  pk_delay       1.48ms
  av_delay        257us
  sp_delay         11us
  backlog            0b
  pkts             9873
  bytes         2620971
  way_inds           90
  way_miss          764
  way_cols            0
  drops              12
  marks               0
  ack_drop            0
  sp_flows            1
  bk_flows            1
  un_flows            0
  max_len         16614
  quantum           300

qdisc ingress ffff: dev pppoe-wan parent ffff:fff1 ----------------
 Sent 18591736 bytes 19776 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc cake 8006: dev ifb4pppoe-wan root refcnt 2 bandwidth 86Mbit besteffort triple-isolate nonat wash no-ack-filter split-gso rtt 100ms noatm overhead 34 mpu 68
 Sent 18581482 bytes 19768 pkt (dropped 8, overlimits 15414 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 272160b of 4300000b
 capacity estimate: 86Mbit
 min/max network layer size:           36 /    1460
 min/max overhead-adjusted size:       70 /    1494
 average network hdr offset:            0

                  Tin 0
  thresh         86Mbit
  target            5ms
  interval        100ms
  pk_delay        270us
  av_delay         86us
  sp_delay         17us
  backlog            0b
  pkts            19776
  bytes        18591736
  way_inds           95
  way_miss          756
  way_cols            0
  drops               8
  marks               0
  ack_drop            0
  sp_flows            1
  bk_flows            1
  un_flows            0
  max_len          1460
  quantum          1514

16:09 1gb file download (75% downloaded) - no vpn

qdisc cake 8005: dev pppoe-wan root refcnt 2 bandwidth 9700Kbit besteffort triple-isolate nonat nowash no-ack-filter split-gso rtt 100ms noatm overhead 34 mpu 68
 Sent 17859869 bytes 364333 pkt (dropped 14, overlimits 20184 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 56000b of 4Mb
 capacity estimate: 9700Kbit
 min/max network layer size:           29 /    1460
 min/max overhead-adjusted size:       68 /    1494
 average network hdr offset:            0

                  Tin 0
  thresh       9700Kbit
  target            5ms
  interval        100ms
  pk_delay         21us
  av_delay         11us
  sp_delay          9us
  backlog            0b
  pkts           364347
  bytes        17875146
  way_inds          108
  way_miss          909
  way_cols            0
  drops              14
  marks               0
  ack_drop            0
  sp_flows            0
  bk_flows            1
  un_flows            0
  max_len         16614
  quantum           300

qdisc ingress ffff: dev pppoe-wan parent ffff:fff1 ----------------
 Sent 978095215 bytes 678350 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc cake 8006: dev ifb4pppoe-wan root refcnt 2 bandwidth 86Mbit besteffort triple-isolate nonat wash no-ack-filter split-gso rtt 100ms noatm overhead 34 mpu 68
 Sent 977431971 bytes 677894 pkt (dropped 414, overlimits 1061651 requeues 0)
 backlog 61320b 42p requeues 0
 memory used: 285120b of 4300000b
 capacity estimate: 86Mbit
 min/max network layer size:           36 /    1460
 min/max overhead-adjusted size:       70 /    1494
 average network hdr offset:            0

                  Tin 0
  thresh         86Mbit
  target            5ms
  interval        100ms
  pk_delay       5.93ms
  av_delay       3.87ms
  sp_delay       3.76ms
  backlog        61320b
  pkts           678350
  bytes       978095215
  way_inds          215
  way_miss          895
  way_cols            0
  drops             414
  marks               0
  ack_drop            0
  sp_flows            0
  bk_flows            1
  un_flows            0
  max_len          1460
  quantum          1514

16:10 1gb file download (100% downloaded) - no vpn

qdisc cake 8005: dev pppoe-wan root refcnt 2 bandwidth 9700Kbit besteffort triple-isolate nonat nowash no-ack-filter split-gso rtt 100ms noatm overhead 34 mpu 68
 Sent 24848318 bytes 527795 pkt (dropped 14, overlimits 27822 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 56000b of 4Mb
 capacity estimate: 9700Kbit
 min/max network layer size:           29 /    1460
 min/max overhead-adjusted size:       68 /    1494
 average network hdr offset:            0

                  Tin 0
  thresh       9700Kbit
  target            5ms
  interval        100ms
  pk_delay        481us
  av_delay         21us
  sp_delay         14us
  backlog            0b
  pkts           527809
  bytes        24863595
  way_inds          116
  way_miss          952
  way_cols            0
  drops              14
  marks               0
  ack_drop            0
  sp_flows            1
  bk_flows            1
  un_flows            0
  max_len         16614
  quantum           300

qdisc ingress ffff: dev pppoe-wan parent ffff:fff1 ----------------
 Sent 1419120934 bytes 980802 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc cake 8006: dev ifb4pppoe-wan root refcnt 2 bandwidth 86Mbit besteffort triple-isolate nonat wash no-ack-filter split-gso rtt 100ms noatm overhead 34 mpu 68
 Sent 1418235770 bytes 980194 pkt (dropped 608, overlimits 1535353 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 285120b of 4300000b
 capacity estimate: 86Mbit
 min/max network layer size:           36 /    1460
 min/max overhead-adjusted size:       70 /    1494
 average network hdr offset:            0

                  Tin 0
  thresh         86Mbit
  target            5ms
  interval        100ms
  pk_delay       7.93ms
  av_delay       6.72ms
  sp_delay         17us
  backlog            0b
  pkts           980802
  bytes      1419120934
  way_inds          291
  way_miss          940
  way_cols            0
  drops             608
  marks               0
  ack_drop            0
  sp_flows            1
  bk_flows            1
  un_flows            0
  max_len          1460
  quantum          1514

16:11 1gb file download (126 mb downloaded - connection speed dropped to zero and I immediately canceled the download, no OOM) - vpn

qdisc cake 8005: dev pppoe-wan root refcnt 2 bandwidth 9700Kbit besteffort triple-isolate nonat nowash no-ack-filter split-gso rtt 100ms noatm overhead 34 mpu 68
 Sent 36064348 bytes 609877 pkt (dropped 14, overlimits 43660 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 751680b of 4Mb
 capacity estimate: 9700Kbit
 min/max network layer size:           28 /    1460
 min/max overhead-adjusted size:       68 /    1494
 average network hdr offset:            0

                  Tin 0
  thresh       9700Kbit
  target            5ms
  interval        100ms
  pk_delay       5.81ms
  av_delay        3.9ms
  sp_delay        552us
  backlog            0b
  pkts           609891
  bytes        36079625
  way_inds          140
  way_miss          993
  way_cols            0
  drops              14
  marks               0
  ack_drop            0
  sp_flows            0
  bk_flows            1
  un_flows            0
  max_len         16614
  quantum           300

qdisc ingress ffff: dev pppoe-wan parent ffff:fff1 ----------------
 Sent 1557825062 bytes 1165852 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc cake 8006: dev ifb4pppoe-wan root refcnt 2 bandwidth 86Mbit besteffort triple-isolate nonat wash no-ack-filter split-gso rtt 100ms noatm overhead 34 mpu 68
 Sent 1555367758 bytes 1163038 pkt (dropped 2197, overlimits 1595470 requeues 0)
 backlog 463460b 619p requeues 0
 memory used: 3190752b of 4300000b
 capacity estimate: 86Mbit
 min/max network layer size:           36 /    1460
 min/max overhead-adjusted size:       70 /    1494
 average network hdr offset:            0

                  Tin 0
  thresh         86Mbit
  target            5ms
  interval        100ms
  pk_delay       51.5ms
  av_delay       33.4ms
  sp_delay         33ms
  backlog       463500b
  pkts          1165852
  bytes      1557825062
  way_inds          339
  way_miss          987
  way_cols            0
  drops            2197
  marks               0
  ack_drop            0
  sp_flows            0
  bk_flows            1
  un_flows            0
  max_len          1460
  quantum          1514

16:14 1gb file download (342 mb downloaded - oom, wifi got killed, etc) - vpn

qdisc cake 8005: dev pppoe-wan root refcnt 2 bandwidth 9700Kbit besteffort triple-isolate nonat nowash no-ack-filter split-gso rtt 100ms noatm overhead 34 mpu 68
 Sent 69932350 bytes 859631 pkt (dropped 20, overlimits 91521 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 813888b of 4Mb
 capacity estimate: 9700Kbit
 min/max network layer size:           28 /    1460
 min/max overhead-adjusted size:       68 /    1494
 average network hdr offset:            0

                  Tin 0
  thresh       9700Kbit
  target            5ms
  interval        100ms
  pk_delay       2.14ms
  av_delay        792us
  sp_delay         16us
  backlog            0b
  pkts           859651
  bytes        69949787
  way_inds          166
  way_miss         1220
  way_cols            0
  drops              20
  marks               1
  ack_drop            0
  sp_flows            2
  bk_flows            1
  un_flows            0
  max_len         16614
  quantum           300

qdisc ingress ffff: dev pppoe-wan parent ffff:fff1 ----------------
 Sent 1997765386 bytes 1753853 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc cake 8006: dev ifb4pppoe-wan root refcnt 2 bandwidth 86Mbit besteffort triple-isolate nonat wash no-ack-filter split-gso rtt 100ms noatm overhead 34 mpu 68
 Sent 1992109042 bytes 1746607 pkt (dropped 7246, overlimits 1790017 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 3190752b of 4300000b
 capacity estimate: 86Mbit
 min/max network layer size:           36 /    1460
 min/max overhead-adjusted size:       70 /    1494
 average network hdr offset:            0

                  Tin 0
  thresh         86Mbit
  target            5ms
  interval        100ms
  pk_delay       1.34ms
  av_delay       1.34ms
  sp_delay         12us
  backlog            0b
  pkts          1753853
  bytes      1997765386
  way_inds          361
  way_miss         1198
  way_cols            0
  drops            7246
  marks               0
  ack_drop            0
  sp_flows            2
  bk_flows            1
  un_flows            0
  max_len          1460
  quantum          1514
4

This seems to be a common problem with some devices like mt7621 there's another thread I posted in with this but I can't find it right now

It seems like you have to make sure the base device has an mtu higher than the ones you put on top of it ? If the bottom device is 'wan' you need to set the mtu higher, i'm not sure exactly how much more possibly ? Maybe 1524 or 1512 or something or maybe you just need to lower the mtu on the device you've made off "pppoe-wan" to 1492 ?

Perhaps the router gets confused when you make these devices and use pppoe.

pppoe has an overhead of 8 bytes, so if the 'wan' device has an mtu of 1500 then that means the mtu of your pppoe device will be 1492
If you then go and make some qos device on top of that with an mtu of 1500, this might be how memory starts leaking away somehow

1 Like
5

So cake takes around 4 MiB (for both ingress and egress combined, where the theoretical maximum would be 8 MiB) which is IMHO not really bad. Without the VPN cake only takes around 300 KiB. This implies that with the VPN there is more memory pressure, but whether that is enough to trigger the OOM is hard to say. Now what would be interesting is to know whether with the VPN you suffer packet fragmentation, as that would be a noticeable load... for that try to get a packet capture from the wan interface while doing a download through the VPN...
See here for how to use wireshark over SSH so you are sure to get a capture even if the router OOMs.... the idea is to look whether you get signs of packet fragmentation... (you might need to do this both on the wan and br-lan depending on where the fragmentation happens).