Tailscale on OpenWrt Freezes: "tailscaled invoked oom-killer"

[Audioctor]

Hi,
I am using tailscale on OpenWrt on Netgear R6350. I use this device exclusively for advertising my subnet and connecting to another network at a remote site. There are no other devices connected to this device.
I recently updated my OpenWrt version to 23.05.5. Since the update, my tailscale freezes after every few hours of running. After a reboot, it runs for a few hours only to go down again.
The system log shows : "tailscaled invoked oom-killer" message.

I am posting an excerpt from the Log for reference:

Sun Jan 26 12:23:07 2025 kern.warn kernel: [44493.587439] tailscaled invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Sun Jan 26 12:23:07 2025 kern.warn kernel: [44493.607445] CPU: 3 PID: 8705 Comm: tailscaled Not tainted 5.15.167 #0
Sun Jan 26 12:23:07 2025 kern.warn kernel: [44493.620337] Stack : 00000000 8081a3d8 00000840 80083cc4 00000000 00000000 00000000 00000000
Sun Jan 26 12:23:07 2025 kern.warn kernel: [44493.637064]         00000000 00000000 00000000 00000000 00000000 00000001 84245b60 80c7d5c0
Sun Jan 26 12:23:07 2025 kern.warn kernel: [44493.653747]         84245bf8 00000000 00000000 84245a08 00000038 803a19c4 ffffffea 00000000
Sun Jan 26 12:23:07 2025 kern.warn kernel: [44493.670455]         84245a14 000002f2 80822ab0 ffffffff 8073fdd4 84245b40 81988c40 00000000
Sun Jan 26 12:23:07 2025 kern.warn kernel: [44493.687160]         808a34c0 01100cca 00000000 8081a3d8 00000018 804139d8 0000000c 809e000c
Sun Jan 26 12:23:07 2025 kern.warn kernel: [44493.703851]         ...
Sun Jan 26 12:23:07 2025 kern.warn kernel: [44493.708729] Call Trace:
Sun Jan 26 12:23:07 2025 kern.warn kernel: [44493.708746] [<80083cc4>] 0x80083cc4
Sun Jan 26 12:23:07 2025 kern.warn kernel: [44493.720560] [<803a19c4>] 0x803a19c4
Sun Jan 26 12:23:07 2025 kern.warn kernel: [44493.727585] [<804139d8>] 0x804139d8
Sun Jan 26 12:23:07 2025 kern.warn kernel: [44493.734529] [<80007898>] 0x80007898
Sun Jan 26 12:23:07 2025 kern.warn kernel: [44493.741467] [<800078a0>] 0x800078a0
Sun Jan 26 12:23:07 2025 kern.warn kernel: [44493.748409] [<80385a98>] 0x80385a98
Sun Jan 26 12:23:07 2025 kern.warn kernel: [44493.755352] [<80141f08>] 0x80141f08
Sun Jan 26 12:23:07 2025 kern.warn kernel: [44493.762309] [<80142760>] 0x80142760
Sun Jan 26 12:23:07 2025 kern.warn kernel: [44493.769254] [<80143150>] 0x80143150
Sun Jan 26 12:23:07 2025 kern.warn kernel: [44493.776211] [<80195334>] 0x80195334
Sun Jan 26 12:23:07 2025 kern.warn kernel: [44493.783198] [<8013dc1c>] 0x8013dc1c
Sun Jan 26 12:23:07 2025 kern.warn kernel: [44493.790139] [<8013c6bc>] 0x8013c6bc
Sun Jan 26 12:23:07 2025 kern.warn kernel: [44493.797136] [<8013e30c>] 0x8013e30c
Sun Jan 26 12:23:07 2025 kern.warn kernel: [44493.804088] [<8013b9b0>] 0x8013b9b0
Sun Jan 26 12:23:07 2025 kern.warn kernel: [44493.811066] [<801778fc>] 0x801778fc
Sun Jan 26 12:23:07 2025 kern.warn kernel: [44493.818026] [<8017d3d4>] 0x8017d3d4
Sun Jan 26 12:23:07 2025 kern.warn kernel: [44493.825005] [<80721210>] 0x80721210
Sun Jan 26 12:23:07 2025 kern.warn kernel: [44493.832009] [<80017528>] 0x80017528
Sun Jan 26 12:23:07 2025 kern.warn kernel: [44493.838982] [<800b2900>] 0x800b2900
Sun Jan 26 12:23:07 2025 kern.warn kernel: [44493.845944] [<8001c9e8>] 0x8001c9e8
Sun Jan 26 12:23:07 2025 kern.warn kernel: [44493.852912]
Sun Jan 26 12:23:07 2025 kern.warn kernel: [44493.855992] Mem-Info:
Sun Jan 26 12:23:07 2025 kern.warn kernel: [44493.860583] active_anon:5333 inactive_anon:6330 isolated_anon:0
Sun Jan 26 12:23:07 2025 kern.warn kernel: [44493.860583]  active_file:46 inactive_file:186 isolated_file:0
Sun Jan 26 12:23:07 2025 kern.warn kernel: [44493.860583]  unevictable:0 dirty:0 writeback:0
Sun Jan 26 12:23:07 2025 kern.warn kernel: [44493.860583]  slab_reclaimable:565 slab_unreclaimable:4918
Sun Jan 26 12:23:07 2025 kern.warn kernel: [44493.860583]  mapped:27 shmem:19 pagetables:156 bounce:0
Sun Jan 26 12:23:07 2025 kern.warn kernel: [44493.860583]  kernel_misc_reclaimable:0
Sun Jan 26 12:23:07 2025 kern.warn kernel: [44493.860583]  free:4092 free_pcp:2 free_cma:0
Sun Jan 26 12:23:07 2025 kern.warn kernel: [44493.929890] Node 0 active_anon:25320kB inactive_anon:21332kB active_file:296kB inactive_file:16kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:108kB dirty:0kB writeback:0kB shmem:76kB writeback_tmp:0kB kernel_stack:1024kB pagetables:624kB all_unreclaimable? no
Sun Jan 26 12:23:07 2025 kern.warn kernel: [44493.978717] Normal free:16280kB min:16384kB low:20480kB high:24576kB reserved_highatomic:0KB active_anon:23252kB inactive_anon:23400kB active_file:440kB inactive_file:300kB unevictable:0kB writepending:0kB present:131072kB managed:120468kB mlocked:0kB bounce:0kB free_pcp:212kB local_pcp:8kB free_cma:0kB
Sun Jan 26 12:23:07 2025 kern.warn kernel: [44494.032317] lowmem_reserve[]: 0 0
Sun Jan 26 12:23:07 2025 kern.warn kernel: [44494.039053] Normal: 178*4kB (UME) 166*8kB (UME) 106*16kB (UME) 43*32kB (UME) 15*64kB (UME) 4*128kB (ME) 6*256kB (UM) 14*512kB (UME) 2*1024kB (UM) 0*2048kB 0*4096kB = 17336kB
Sun Jan 26 12:23:07 2025 kern.warn kernel: [44494.070185] 208 total pagecache pages
Sun Jan 26 12:23:07 2025 kern.warn kernel: [44494.077589] 0 pages in swap cache
Sun Jan 26 12:23:07 2025 kern.warn kernel: [44494.084232] Swap cache stats: add 0, delete 0, find 0/0
Sun Jan 26 12:23:07 2025 kern.warn kernel: [44494.094845] Free swap  = 0kB
Sun Jan 26 12:23:07 2025 kern.warn kernel: [44494.100791] Total swap = 0kB
Sun Jan 26 12:23:07 2025 kern.warn kernel: [44494.106570] 32768 pages RAM
Sun Jan 26 12:23:07 2025 kern.warn kernel: [44494.112129] 0 pages HighMem/MovableOnly
Sun Jan 26 12:23:07 2025 kern.warn kernel: [44494.119842] 2651 pages reserved
Sun Jan 26 12:23:07 2025 kern.info kernel: [44494.126202] Tasks state (memory values in pages):
Sun Jan 26 12:23:07 2025 kern.info kernel: [44494.135720] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Sun Jan 26 12:23:07 2025 kern.info kernel: [44494.153116] [    721]    81   721      376       25    20480        0             0 ubusd
Sun Jan 26 12:23:07 2025 kern.info kernel: [44494.169624] [    722]     0   722      265        8    16384        0             0 askfirst
Sun Jan 26 12:23:07 2025 kern.info kernel: [44494.186731] [    757]     0   757      303       15    16384        0             0 urngd
Sun Jan 26 12:23:07 2025 kern.info kernel: [44494.203197] [   1118]   514  1118      371       36    20480        0             0 logd
Sun Jan 26 12:23:07 2025 kern.info kernel: [44494.219442] [   1178]     0  1178      911      100    24576        0             0 rpcd
Sun Jan 26 12:23:07 2025 kern.info kernel: [44494.235642] [   1399]     0  1399      319       14    16384        0             0 dropbear
Sun Jan 26 12:23:07 2025 kern.info kernel: [44494.252566] [   1503]     0  1503      728       14    16384        0             0 hostapd
Sun Jan 26 12:23:07 2025 kern.info kernel: [44494.269308] [   1504]     0  1504      728       14    20480        0             0 wpa_supplicant
Sun Jan 26 12:23:07 2025 kern.info kernel: [44494.287251] [   1506]   101  1506     1081       88    24576        0             0 hostapd
Sun Jan 26 12:23:07 2025 kern.info kernel: [44494.303995] [   1507]   101  1507     1076       83    20480        0             0 wpa_supplicant
Sun Jan 26 12:23:07 2025 kern.info kernel: [44494.321989] [   1568]     0  1568      530       42    16384        0             0 netifd
Sun Jan 26 12:23:07 2025 kern.info kernel: [44494.338532] [   1778]     0  1778      468       32    16384        0             0 odhcpd
Sun Jan 26 12:23:07 2025 kern.info kernel: [44494.355090] [   1838]     0  1838      345       12    20480        0             0 crond
Sun Jan 26 12:23:07 2025 kern.info kernel: [44494.371537] [   1904]     0  1904      725       32    24576        0             0 uhttpd
Sun Jan 26 12:23:07 2025 kern.info kernel: [44494.388225] [   2166]     0  2166      305       15    16384        0             0 odhcp6c
Sun Jan 26 12:23:07 2025 kern.info kernel: [44494.404975] [   2173]     0  2173      344        8    20480        0             0 udhcpc
Sun Jan 26 12:23:07 2025 kern.info kernel: [44494.421632] [   2262]     0  2262      728       28    20480        0             0 dnsmasq
Sun Jan 26 12:23:07 2025 kern.info kernel: [44494.438399] [   2271]   453  2271      412       29    20480        0             0 dnsmasq
Sun Jan 26 12:23:07 2025 kern.info kernel: [44494.455221] [   3194]     0  3194      728       28    20480        0             0 ntpd
Sun Jan 26 12:23:07 2025 kern.info kernel: [44494.471486] [   3236]   123  3236      344       12    20480        0             0 ntpd
Sun Jan 26 12:23:07 2025 kern.info kernel: [44494.487769] [   8704]     0  8704   156893    10676   147456        0             0 tailscaled
Sun Jan 26 12:23:07 2025 kern.info kernel: [44494.505098] [   9185]     0  9185      394       63    20480        0             0 dhcpv6.script
Sun Jan 26 12:23:07 2025 kern.info kernel: [44494.523742] [   9190]     0  9190      349        9    20480        0             0 fw3
Sun Jan 26 12:23:07 2025 kern.info kernel: [44494.539798] [   9191]     0  9191      344        9    20480        0             0 flock
Sun Jan 26 12:23:07 2025 kern.info kernel: [44494.556251] [   9192]     0  9192      665       93    20480        0             0 utpl
Sun Jan 26 12:23:07 2025 kern.info kernel: [44494.572507] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=tailscaled,pid=8704,uid=0
Sun Jan 26 12:23:07 2025 kern.err kernel: [44494.598344] Out of memory: Killed process 8704 (tailscaled) total-vm:627572kB, anon-rss:42704kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:144kB oom_score_adj:0
Sun Jan 26 12:23:07 2025 daemon.notice netifd: Network device 'tailscale0' link is down

Any help is appreciated.
Thanks in advance.

[frollic]

@Audioctor

"tailscaled invoked oom-killer" message.

OOM means Out of Memory, you figure out the rest...

[Audioctor]

Thanks for the reply. Sadly, I am not much of an expert at this. I have been using tailscale on Openwrt as more of a hobby. So I would need more help on this.

As I mentioned I am not using the device as a router nor have I installed any other applications. So, I am not sure what's eating up the memory. How do I find out?

Thanks.

[fdauti]

Take a look here, the solution posted might help in your case too:

[Solved] Tailscale alternative recommendation

I was looking for a way to quickly access the router from the phone either web or ssh. I tried Wireguard but the peer was never able to connect. My IP is CGNat'ed so port forwarding form WAN is no go either. I tried tailscale and was pleasantly surprised how easy it was to get it up and running though you are limited to 3 devices for the Free plan. What i didn't like is the high RAM usage at almost 50MB and 30MB of disk space with all the relevant dependent packages. The disk space is not yet a…

[mk24]

Zerotier is also rather big but it still works for me on 120Mb RAM.

[brada4]

Your tailscale is just 40M resident, likely zram-swap can hrlp you jump over small memory shortage.

[Audioctor]

Thanks for the comment.
Now, as I am not any good at using the command line, I am asking another silly question:

procd_set_param env TS_DEBUG_FIREWALL_MODE="$fw_mode" GOGC=10

Would it be sufficient to just copy and paste the above code in command line or are there any more steps to be followed?

Thanks in advance.

[Audioctor]

Thanks for the advice. Again, as I mentioned previously. I do not know how to do that exactly. Can you me through ?

Thanks .

[Audioctor]

I thank everyone for your comments.
I have a few questions:

1. I have been using tailscale on OpenWrt on the same device for more than a year. Why did this problem start only after I updated OpenWrt to Ver. 23.05.5?

2. I am using exactly the same configuration on a second device on my remote network. Why is it not having this problem?

3. Can anyone, please, post a step-step-guide to sort this issue out? It would be of help to others who might facing the same issue.

Thanks.

[frollic]

@Audioctor

Why did this problem start only after I updated OpenWrt to Ver. 23.05.5?

New version with new everything uses more RAM, router's RAM was the only thing not upgraded.

@Audioctor

I am using exactly the same configuration on a second device on my remote network

Identical device ?

There could also be a memory leak somewhere.

[Audioctor]

@frollic

Identical device ?

Same Netgear R6350 with same version of Openwrt.

[Audioctor]

@frollic

There could also be a memory leak somewhere.

How do I check?

[brada4]

Find zram-swap in packages list. it adds 1/2 RAM sized swap backed by compressed RAM, what it does is compresses less/not used memory 2-10x and prevents "small" memory shortages.

[Audioctor]

Thanks. I'll try that.

[Audioctor]

Thanks. It has worked.

[brada4]

Just curious - was it tailscale daemon or something else considered small?