OOM-Killer spree despite huge swap

Installing and Using OpenWrt
1

I'm having a lot of fun with this so I welcome everybody to the party.
Performing is a GL-AX100, Qualcomm IPQ6018 SoC, 512MB RAM,
1GB swap partition on fast-ish NVMe SSD attached to USB3.0, uap driver loaded.

Edit: I wonder if the default -2 swap priority is getting in the way. Changed it to 100 and will play some more.

Same issue on SNAPSHOT r31645-3b21f97641 and 25.12.0-rc4.
Running:
3 VLans, 1 5GHz Wi-Fi, 2 ETH
adblock-fast with small list - 81983 domains
https-dns-proxy
transmission: Starting with 83735200 virt mem
Stopped as unnecessary: uhttpd, odhcpd
Swap gets max 60MB use
Swappiness and minimum memory unchanged

root@flint:~# free
              total        used        free      shared  buff/cache   available
Mem:         418676      272736       39584           4      106356       95596
Swap:       1023996       22080     1001916

root@flint:~# uptime
 17:15:57 up  3:34,  load average: 0.05, 0.27, 0.29

Then this pops-up:
dnsmasq invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP) , order=0, oom_score_adj=0

Log available @ GL-AX1800 OOM

Am I missing something? Expecting the system to use swap as available memory for running processes. Is there something in config? Or is it compiled to act this way?

Been looking for a while and couldn't find anything, any ideas appreciated.

2

512MB is tight running on IPQ devices, AdBlock doesn't make it better.

Try kmod-zram.

There was also a discussion about putting the block lists on (external) disk, instead of RAM.

2 Likes
3

Again, Thanks!
That's next on my list. Still, would like to know why swap is ignored and how to, if possible.

4

Should have been item number zero

What filesystem on usb? How big? eg btr needs lots of mem

1 Like
5

SSD is 256 GB with F2FS, 1GB swap partition

6

The list being loaded into dnsmasq and taking only 2.2MB in /tmp I thought it's worth keeping in RAM

root@flint:/tmp# du -h | grep adblock
2.2M	./run/adblock-fast

root@flint:/# ps | grep dnsmasq
 4133 root      2832 S    {dnsmasq} /sbin/ujail -t 5 -n dnsmasq -u -l -r /bin/ubus -r /etc/TZ -r /etc/dnsmasq.conf -r /etc/ethers -r /etc/group -r /etc/hosts -r /etc/
 4173 dnsmasq   7876 S    /usr/sbin/dnsmasq -C /var/etc/dnsmasq.conf.cfg01411c -k -x /var/run/dnsmasq/dnsmasq.cfg01411c.pid
7

I guess every old dog is told that about every new trick. :wink:

Might be useful to include a strong suggestion in the guides for all those wanting to run any extra services to first install zram.

8

Try other filrsystems, seems like open files mmap-ed by transmission are taking all ram and then some.

9

Having transmission running is the only time I see the swap being used.

I wonder if the default -2 swap priority is getting in the way.
Changed it to 100 and will play some more.

10

Mauybe you feel placebo cure with single swap file being prioritized against nothing else.

11

:rofl: :rofl: :rofl: :rofl:

Did you actually write "FEEL"?

I wrote "WONDER"!

rotfwpmsl

12

First add zram-swap, it will buffer small bursts of memory excess, 10x faster than USB.
If it gets again into usb swap well look further if anything needs be done.

1 Like
13

Will keep at it.

root@flint:~# swapon -s
Filename				Type		Size		Used		Priority
/dev/sda2               partition	1023996		0		    90
/dev/zram0             partition	208892		9040		100
root@flint:~# free
              total        used        free      shared  buff/cache   available
Mem:         418676      267316       35252          52      116108      101148
Swap:       1232888        9040     1223848
14

You know, check in a day, then in a week then in a month if it is growing or stabilises.

15

It's still not playing nice
dnsmasq invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0

although plenty of swap available

0 pages in swap cache
Free swap  = 1208016kB
Total swap = 1232888kB

With my limited understanding, I'll say this only to be said:
Looks like it suddenly realizes it's low on memory and, without having time to even consider swap, the killer gets the job done by... killing someone.

Looks like a very sophisticated party :smiley:

16

You can vary vm.swappiness between 1 and 200 and vm.min_free_kbytes between 1024 and 65536
Documentation at upstream kernel.

17

Solved by load balancing the cpu - tuning smp irq

Looking at htop or cat /proc/interrupts CPU0 was doing all the lifting by itself. I very rarely seen all 4 cores active in htop.

Note: irqbalance package does nothing, I suppose it's a naming problem, there is no eth0

The solution was to create /etc/hotplug.d/net/50-custom-net-smp-affinity

# bam_dma to core 1
echo 2 > /proc/irq/17/smp_affinity
echo 2 > /proc/irq/18/smp_affinity
echo 2 > /proc/irq/19/smp_affinity

# edma_* to core 2                       
echo 4 > /proc/irq/33/smp_affinity
echo 4 > /proc/irq/34/smp_affinity
echo 4 > /proc/irq/35/smp_affinity
echo 4 > /proc/irq/36/smp_affinity

# usb to core 3
echo 8 > /proc/irq/39/smp_affinity

And blimey, Happy Days, it can do a LOT more now!

I managed to get it down to 7168, the OOMs killed dnsmasq and transmission quite frequently yet they could restart, but it did throw rare page allocation errors.

18

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.