3466

I have the qsdk partition layout and I can flash own build qsdk and also official xiaomi firmware, but I would prefer to go back to official layout?
You can see the difference in the wiki ( https://openwrt.org/inbox/toh/xiaomi/xiaomi_ax3600 ) or just use panko of mtd12 is different size then mtd13

3467

For ipq806x, the nss-crypto is pretty much standalone. Once it gets loaded, I can use its internal benchmark tool to check if the crypto core is running.

nss-cfi just provide kernel crypto integration for higher level kernel drivers to use. nss-cfi needs nss-crypto.

3468

Ok, that was my logic as well.
I can load the benchmark kmod, but as soon as start command is issued it will cause a NSS core1 coredump and then kernel panic and reboot.

I will try bumping the crypto log level

1 Like
3469

I remember pulling my hairs out when toying with the crypto drivers, as I was really hoping to accelerate OpenVPN. Alas …

Anyway, my issue turns out to be compatibility between driver and firmware versions. Once I restart and ensure QSDK and firmware version matches, everything started working.

3470

I feel so incredibly stupid right now. Had to to the flag_ota_reboot to get the international fw running and stuff so my brain automatically did the same for openwrt. Thanks!

3471

I did like that too.

3472

Yeah, we got burned by the mismatched FW version as well earlier.

However, I tried using 11.4 FW as well and the end result is the same.

BTW, my AX9000 finally arrived, with packaging its like 3.4kg.
Box is huge compared to the AX3600, much more fancy.
Its gonna be a fun weekend trying to get UART TX working.
Horrible adapters as well, these are completely useless as the pins are completely bent.
Google Photos

2 Likes
3473

That one AX9000 is on my watch list... Curious to know how much it can handle.

3474

During the process of soldering the pins for the UART access I noticed that the quality of the thermal pads is quite improvable, so I took the opportunity to change them.

Before:

The old ones fall apart when touched, the residue must be cleaned with isopropanol:

New ones:

I use those pads:

2 Likes
3475

From left to right:

RX, GND, TX (I guess ;-), VCC

ax9000_uart
ax9000_uart1148×908 166 KB

3476

Thanks, did they mark the pins on the silkscreen like on AX3600?

3477

Not really, only the right pin has some mark.

I used an oscilloscope to find the RX

3478

Ok, then we will just document it in the wiki.

3479

And i already had a look if we can use the same nvram inject method like the ax3600.

But "set_config_iotdev" is missing. But there are a lot of other lua api methods.

But this is really not my area.

EDIT

set_config_iotdev is still there in misystem.lua

3480

cfg_backup is encrypted (of cause)

/lua/xiaoqiang/module/XQBackup.lua seems to handle this, but is was only able to partially decrypt the lua. (with luadec)

3481

There are two additional screws (to the other ones under the rubber feet) under the device label, lift the top carefully, there is a small cable/plug from the leds to the board.

3482

@robimarko

Massive OOM, after just 24 hours of uptime (in case you want to take a look):

[102645.375788] netifd invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
[102645.375833] CPU: 1 PID: 1863 Comm: netifd Not tainted 5.10.42 #0
[102645.383290] Hardware name: Xiaomi AX3600 (DT)
[102645.389278] Call trace:
[102645.393624]  dump_backtrace+0x0/0x1b0
[102645.396310]  show_stack+0x18/0x30
[102645.399869]  dump_stack+0xdc/0x11c
[102645.403253]  dump_header+0x44/0x194
[102645.406642]  oom_kill_process+0x1d0/0x1d8
[102645.410459]  out_of_memory+0x20c/0x320
[102645.414366]  __alloc_pages_slowpath.constprop.150+0x6c8/0x9b0
[102645.418101]  __alloc_pages_nodemask+0x1f4/0x238
[102645.424001]  __get_free_pages+0x1c/0x68
[102645.428775]  proc_pid_readlink+0x74/0x2b8
[102645.432683]  vfs_readlink+0x11c/0x138
[102645.436589]  do_readlinkat+0xac/0x128
[102645.440318]  __arm64_sys_readlinkat+0x20/0x30
[102645.444056]  el0_svc_common.constprop.3+0xac/0x170
[102645.448482]  do_el0_svc+0x70/0x90
[102645.453255]  el0_svc+0x18/0x28
[102645.456726]  el0_sync_handler+0x90/0xb8
[102645.459764]  el0_sync+0x184/0x1c0
[102645.464040] Mem-Info:
[102645.467148] active_anon:69 inactive_anon:677 isolated_anon:0
[102645.467148]  active_file:15 inactive_file:76 isolated_file:0
[102645.467148]  unevictable:0 dirty:0 writeback:0
[102645.467148]  slab_reclaimable:1878 slab_unreclaimable:11436
[102645.467148]  mapped:15 shmem:70 pagetables:138 bounce:0
[102645.467148]  free:4801 free_pcp:194 free_cma:0
[102645.478778] Node 0 active_anon:276kB inactive_anon:2708kB active_file:60kB inactive_file:304kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:60kB dirty:0kB writeback:0kB shmem:280kB writeback_tmp:0kB kernel_stack:1392kB all_unreclaimable? yes
[102645.501543] DMA free:19204kB min:16384kB low:20480kB high:24576kB reserved_highatomic:0KB active_anon:276kB inactive_anon:2708kB active_file:188kB inactive_file:432kB unevictable:0kB writepending:0kB present:524288kB managed:375880kB mlocked:0kB pagetables:552kB bounce:0kB free_pcp:688kB local_pcp:0kB free_cma:0kB
[102645.529551] lowmem_reserve[]: 0 0 0 0
[102645.551798] DMA: 452*4kB (UME) 207*8kB (UME) 124*16kB (UME) 56*32kB (UME) 24*64kB (UME) 9*128kB (UME) 4*256kB (M) 6*512kB (UM) 2*1024kB (UM) 1*2048kB (U) 0*4096kB = 18120kB
[102645.555366] 300 total pagecache pages
[102645.570885] 0 pages in swap cache
[102645.574526] Swap cache stats: add 0, delete 0, find 0/0
[102645.577912] Free swap  = 0kB
[102645.583382] Total swap = 0kB
[102645.586237] 131072 pages RAM
[102645.589213] 0 pages HighMem/MovableOnly
[102645.592145] 37102 pages reserved
[102645.596137] Tasks state (memory values in pages):
[102645.599351] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[102645.604088] [    842]    81   842      304       28    36864        0             0 ubusd
[102645.612740] [    843]     0   843      205        9    28672        0             0 askfirst
[102645.620996] [    877]     0   877      237       11    24576        0             0 urngd
[102645.629662] [   1326]   514  1326      296       38    32768        0             0 logd
[102645.637754] [   1397]     0  1397      566       84    32768        0             0 rpcd
[102645.646000] [   1682]     0  1682      254       10    28672        0             0 dropbear
[102645.654155] [   1802]     0  1802      943       52    45056        0             0 hostapd
[102645.662728] [   1803]     0  1803      922       31    36864        0             0 wpa_supplicant
[102645.671185] [   1863]     0  1863      452       54    32768        0             0 netifd
[102645.679985] [   1917]     0  1917      358       29    36864        0             0 odhcpd
[102645.688255] [   2037]     0  2037      850       73    40960        0             0 uhttpd
[102645.696627] [   2585]     0  2585      306       15    28672        0             0 ntpd
[102645.704921] [   3075]   453  3075      346       34    32768        0             0 dnsmasq
[102645.713291] [   3327]     0  3327      319      117    32768        0             0 pppd
[102645.721762] [   3442]     0  3442      393      103    32768        0             0 sh
[102645.729870] [  19354]     0 19354      305       10    28672        0             0 sleep
[102645.737648] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=dnsmasq,pid=3075,uid=453
[102645.746036] Out of memory: Killed process 3075 (dnsmasq) total-vm:1384kB, anon-rss:132kB, file-rss:4kB, shmem-rss:0kB, UID:453 pgtables:32kB oom_score_adj:0
[102645.758871] oom_reaper: reaped process 3075 (dnsmasq), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[102771.797710] hostapd invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[102771.797755] CPU: 2 PID: 1802 Comm: hostapd Not tainted 5.10.42 #0
[102771.806254] Hardware name: Xiaomi AX3600 (DT)
[102771.812415] Call trace:
[102771.816848]  dump_backtrace+0x0/0x1b0
[102771.819533]  show_stack+0x18/0x30
[102771.823093]  dump_stack+0xdc/0x11c
[102771.826476]  dump_header+0x44/0x194
[102771.829865]  oom_kill_process+0x1d0/0x1d8
[102771.833682]  out_of_memory+0x20c/0x320
[102771.837588]  __alloc_pages_slowpath.constprop.150+0x6c8/0x9b0
[102771.841340]  __alloc_pages_nodemask+0x1f4/0x238
[102771.847227]  pagecache_get_page.part.71+0xf8/0x2b8
[102771.852000]  filemap_fault+0x774/0x978
[102771.856600]  __do_fault+0x3c/0xf8
[102771.860418]  handle_mm_fault+0x6b4/0xc38
[102771.863890]  do_page_fault+0x1ac/0x3d8
[102771.867968]  do_translation_fault+0x50/0x60
[102771.871616]  do_mem_abort+0x40/0xa0
[102771.876132]  el0_ia+0x64/0xd0
[102771.879690]  el0_sync_handler+0x78/0xb8
[102771.882554]  el0_sync+0x184/0x1c0
[102771.886749] Mem-Info:
[102771.889861] active_anon:70 inactive_anon:671 isolated_anon:0
[102771.889861]  active_file:20 inactive_file:58 isolated_file:0
[102771.889861]  unevictable:0 dirty:0 writeback:0
[102771.889861]  slab_reclaimable:1874 slab_unreclaimable:11442
[102771.889861]  mapped:43 shmem:70 pagetables:138 bounce:0
[102771.889861]  free:4076 free_pcp:5 free_cma:0
[102771.901133] Node 0 active_anon:280kB inactive_anon:2684kB active_file:132kB inactive_file:24kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:20kB dirty:0kB writeback:0kB shmem:280kB writeback_tmp:0kB kernel_stack:1456kB all_unreclaimable? no
[102771.924067] DMA free:16816kB min:16384kB low:20480kB high:24576kB reserved_highatomic:0KB active_anon:280kB inactive_anon:2684kB active_file:220kB inactive_file:508kB unevictable:0kB writepending:0kB present:524288kB managed:375880kB mlocked:0kB pagetables:552kB bounce:0kB free_pcp:148kB local_pcp:20kB free_cma:0kB
[102771.951994] lowmem_reserve[]: 0 0 0 0
[102771.974226] DMA: 656*4kB (UME) 242*8kB (UME) 127*16kB (UME) 69*32kB (UME) 28*64kB (UME) 9*128kB (ME) 5*256kB (M) 5*512kB (M) 2*1024kB (UM) 0*2048kB 0*4096kB = 17632kB
[102771.977893] 139 total pagecache pages
[102771.992715] 0 pages in swap cache
[102771.996533] Swap cache stats: add 0, delete 0, find 0/0
[102771.999922] Free swap  = 0kB
[102772.005385] Total swap = 0kB
[102772.008251] 131072 pages RAM
[102772.011195] 0 pages HighMem/MovableOnly
[102772.014156] 37102 pages reserved
[102772.018147] Tasks state (memory values in pages):
[102772.021361] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[102772.026079] [    842]    81   842      304       28    36864        0             0 ubusd
[102772.034656] [    843]     0   843      205        9    28672        0             0 askfirst
[102772.042988] [    877]     0   877      237       11    24576        0             0 urngd
[102772.051584] [   1326]   514  1326      296       38    32768        0             0 logd
[102772.059742] [   1397]     0  1397      566       84    32768        0             0 rpcd
[102772.067989] [   1682]     0  1682      254       10    28672        0             0 dropbear
[102772.076150] [   1802]     0  1802      943       52    45056        0             0 hostapd
[102772.084656] [   1803]     0  1803      922       31    36864        0             0 wpa_supplicant
[102772.093163] [   1863]     0  1863      452       54    32768        0             0 netifd
[102772.101930] [   1917]     0  1917      358       29    36864        0             0 odhcpd
[102772.110261] [   2037]     0  2037      850       73    40960        0             0 uhttpd
[102772.118597] [   2585]     0  2585      306       15    28672        0             0 ntpd
[102772.126929] [   3327]     0  3327      319       37    32768        0             0 pppd
[102772.135262] [   3442]     0  3442      393      103    32768        0             0 sh
[102772.143424] [  19354]     0 19354      305       10    28672        0             0 sleep
[102772.151238] [  19363]   453 19363      341       28    32768        0             0 dnsmasq
[102772.159569] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=sh,pid=3442,uid=0
[102772.168187] Out of memory: Killed process 3442 (sh) total-vm:1572kB, anon-rss:408kB, file-rss:4kB, shmem-rss:0kB, UID:0 pgtables:32kB oom_score_adj:0
[102772.180271] oom_reaper: reaped process 3442 (sh), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[102784.049579] netifd invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[102784.049628] CPU: 3 PID: 1863 Comm: netifd Not tainted 5.10.42 #0
[102784.058140] Hardware name: Xiaomi AX3600 (DT)
[102784.064199] Call trace:
[102784.068544]  dump_backtrace+0x0/0x1b0
[102784.071230]  show_stack+0x18/0x30
[102784.074790]  dump_stack+0xdc/0x11c
[102784.078173]  dump_header+0x44/0x194
[102784.081563]  oom_kill_process+0x1d0/0x1d8
[102784.085380]  out_of_memory+0x20c/0x320
[102784.089287]  __alloc_pages_slowpath.constprop.150+0x6c8/0x9b0
[102784.093021]  __alloc_pages_nodemask+0x1f4/0x238
[102784.098924]  pagecache_get_page.part.71+0xf8/0x2b8
[102784.103697]  filemap_fault+0x774/0x978
[102784.108299]  __do_fault+0x3c/0xf8
[102784.112115]  handle_mm_fault+0x6b4/0xc38
[102784.115587]  do_page_fault+0x1ac/0x3d8
[102784.119665]  do_translation_fault+0x50/0x60
[102784.123314]  do_mem_abort+0x40/0xa0
[102784.127830]  el0_ia+0x64/0xd0
[102784.131408]  el0_sync_handler+0x78/0xb8
[102784.134252]  el0_sync+0x184/0x1c0
[102784.138480] Mem-Info:
[102784.141575] active_anon:68 inactive_anon:558 isolated_anon:0
[102784.141575]  active_file:14 inactive_file:93 isolated_file:0
[102784.141575]  unevictable:0 dirty:0 writeback:0
[102784.141575]  slab_reclaimable:1873 slab_unreclaimable:11441
[102784.141575]  mapped:41 shmem:70 pagetables:123 bounce:0
[102784.141575]  free:4080 free_pcp:1 free_cma:0
[102784.152834] Node 0 active_anon:272kB inactive_anon:2232kB active_file:48kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:12kB dirty:0kB writeback:0kB shmem:280kB writeback_tmp:0kB kernel_stack:1472kB all_unreclaimable? yes
[102784.175806] DMA free:16564kB min:16384kB low:20480kB high:24576kB reserved_highatomic:0KB active_anon:272kB inactive_anon:2232kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:524288kB managed:375880kB mlocked:0kB pagetables:492kB bounce:0kB free_pcp:908kB local_pcp:0kB free_cma:0kB
[102784.203263] lowmem_reserve[]: 0 0 0 0
[102784.225537] DMA: 359*4kB (UME) 251*8kB (UME) 132*16kB (UME) 75*32kB (UME) 27*64kB (UME) 10*128kB (ME) 7*256kB (UM) 6*512kB (UM) 1*1024kB (M) 0*2048kB 0*4096kB = 16852kB
[102784.229090] 175 total pagecache pages
[102784.244240] 0 pages in swap cache
[102784.247882] Swap cache stats: add 0, delete 0, find 0/0
[102784.251265] Free swap  = 0kB
[102784.256737] Total swap = 0kB
[102784.259593] 131072 pages RAM
[102784.262552] 0 pages HighMem/MovableOnly
[102784.265505] 37102 pages reserved
[102784.269511] Tasks state (memory values in pages):
[102784.272710] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[102784.277429] [    842]    81   842      304       28    36864        0             0 ubusd
[102784.286027] [    843]     0   843      205        9    28672        0             0 askfirst
[102784.294340] [    877]     0   877      237       11    24576        0             0 urngd
[102784.302934] [   1326]   514  1326      296       42    32768        0             0 logd
[102784.311118] [   1397]     0  1397      566       84    32768        0             0 rpcd
[102784.319339] [   1682]     0  1682      254       10    28672        0             0 dropbear
[102784.327501] [   1802]     0  1802      943       52    45056        0             0 hostapd
[102784.336007] [   1803]     0  1803      922       31    36864        0             0 wpa_supplicant
[102784.344515] [   1863]     0  1863      452       54    32768        0             0 netifd
[102784.353303] [   1917]     0  1917      358       29    36864        0             0 odhcpd
[102784.361719] [   2037]     0  2037      850       73    40960        0             0 uhttpd
[102784.369955] [   2585]     0  2585      306       15    28672        0             0 ntpd
[102784.378366] [   3327]     0  3327      319       37    32768        0             0 pppd
[102784.386627] [  19363]   453 19363      341       28    32768        0             0 dnsmasq
[102784.394871] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=rpcd,pid=1397,uid=0
[102784.403343] Out of memory: Killed process 1397 (rpcd) total-vm:2264kB, anon-rss:332kB, file-rss:4kB, shmem-rss:0kB, UID:0 pgtables:32kB oom_score_adj:0
[102784.415872] oom_reaper: reaped process 1397 (rpcd), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[102827.829785] pppd invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[102827.829828] CPU: 3 PID: 3327 Comm: pppd Not tainted 5.10.42 #0
[102827.837981] Hardware name: Xiaomi AX3600 (DT)
[102827.843882] Call trace:
[102827.848401]  dump_backtrace+0x0/0x1b0
[102827.851086]  show_stack+0x18/0x30
[102827.854646]  dump_stack+0xdc/0x11c
[102827.858029]  dump_header+0x44/0x194
[102827.861418]  oom_kill_process+0x1d0/0x1d8
[102827.865235]  out_of_memory+0x20c/0x320
[102827.869143]  __alloc_pages_slowpath.constprop.150+0x6c8/0x9b0
[102827.872876]  __alloc_pages_nodemask+0x1f4/0x238
[102827.878779]  pagecache_get_page.part.71+0xf8/0x2b8
[102827.883553]  filemap_fault+0x774/0x978
[102827.888154]  __do_fault+0x3c/0xf8
[102827.891971]  handle_mm_fault+0x6b4/0xc38
[102827.895444]  do_page_fault+0x1ac/0x3d8
[102827.899522]  do_translation_fault+0x50/0x60
[102827.903170]  do_mem_abort+0x40/0xa0
[102827.907687]  el0_ia+0x64/0xd0
[102827.911242]  el0_sync_handler+0x78/0xb8
[102827.914108]  el0_sync+0x184/0x1c0
[102827.918324] Mem-Info:
[102827.921428] active_anon:70 inactive_anon:535 isolated_anon:0
[102827.921428]  active_file:23 inactive_file:59 isolated_file:0
[102827.921428]  unevictable:0 dirty:0 writeback:0
[102827.921428]  slab_reclaimable:1872 slab_unreclaimable:11441
[102827.921428]  mapped:0 shmem:70 pagetables:129 bounce:0
[102827.921428]  free:4104 free_pcp:261 free_cma:0
[102827.933068] Node 0 active_anon:280kB inactive_anon:2140kB active_file:92kB inactive_file:564kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:40kB dirty:0kB writeback:0kB shmem:280kB writeback_tmp:0kB kernel_stack:1488kB all_unreclaimable? yes
[102827.955697] DMA free:16956kB min:16384kB low:20480kB high:24576kB reserved_highatomic:0KB active_anon:280kB inactive_anon:2140kB active_file:0kB inactive_file:40kB unevictable:0kB writepending:0kB present:524288kB managed:375880kB mlocked:0kB pagetables:516kB bounce:0kB free_pcp:208kB local_pcp:120kB free_cma:0kB
[102827.983395] lowmem_reserve[]: 0 0 0 0
[102828.005626] DMA: 334*4kB (UME) 229*8kB (UME) 138*16kB (UME) 73*32kB (UME) 29*64kB (ME) 11*128kB (UME) 7*256kB (UM) 6*512kB (UM) 1*1024kB (M) 0*2048kB 0*4096kB = 16864kB
[102828.009447] 304 total pagecache pages
[102828.024618] 0 pages in swap cache
[102828.028274] Swap cache stats: add 0, delete 0, find 0/0
[102828.031641] Free swap  = 0kB
[102828.037112] Total swap = 0kB
[102828.039977] 131072 pages RAM
[102828.042922] 0 pages HighMem/MovableOnly
[102828.045882] 37102 pages reserved
[102828.049874] Tasks state (memory values in pages):
[102828.053089] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[102828.057808] [    842]    81   842      304       28    36864        0             0 ubusd
[102828.066389] [    843]     0   843      205        9    28672        0             0 askfirst
[102828.074765] [    877]     0   877      237       11    24576        0             0 urngd
[102828.083343] [   1326]   514  1326      296       38    32768        0             0 logd
[102828.091601] [   1682]     0  1682      254       10    28672        0             0 dropbear
[102828.099722] [   1802]     0  1802      943       52    45056        0             0 hostapd
[102828.108320] [   1803]     0  1803      922       31    36864        0             0 wpa_supplicant
[102828.116736] [   1863]     0  1863      452       54    32768        0             0 netifd
[102828.125600] [   1917]     0  1917      358       29    36864        0             0 odhcpd
[102828.133840] [   2037]     0  2037      850       73    40960        0             0 uhttpd
[102828.142310] [   2585]     0  2585      306       14    28672        0             0 ntpd
[102828.150506] [   3327]     0  3327      319       37    32768        0             0 pppd
[102828.158924] [  19363]   453 19363      341       28    32768        0             0 dnsmasq
[102828.167001] [  19368]     0 19368      532       48    32768        0             0 rpcd
[102828.175647] [  19371]     0 19371      305       12    24576        0             0 sh
[102828.183576] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=uhttpd,pid=2037,uid=0
[102828.191408] Out of memory: Killed process 2037 (uhttpd) total-vm:3400kB, anon-rss:292kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:40kB oom_score_adj:0
[102828.204104] oom_reaper: reaped process 2037 (uhttpd), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[103370.343728] netifd invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
[103370.343773] CPU: 1 PID: 1863 Comm: netifd Not tainted 5.10.42 #0
[103370.351232] Hardware name: Xiaomi AX3600 (DT)
[103370.357219] Call trace:
[103370.361564]  dump_backtrace+0x0/0x1b0
[103370.364251]  show_stack+0x18/0x30
[103370.367811]  dump_stack+0xdc/0x11c
[103370.371194]  dump_header+0x44/0x194
[103370.374582]  oom_kill_process+0x1d0/0x1d8
[103370.378400]  out_of_memory+0x20c/0x320
[103370.382307]  __alloc_pages_slowpath.constprop.150+0x6c8/0x9b0
[103370.386042]  __alloc_pages_nodemask+0x1f4/0x238
[103370.391942]  __get_free_pages+0x1c/0x68
[103370.396716]  proc_pid_readlink+0x74/0x2b8
[103370.400623]  vfs_readlink+0x11c/0x138
[103370.404529]  do_readlinkat+0xac/0x128
[103370.408258]  __arm64_sys_readlinkat+0x20/0x30
[103370.411997]  el0_svc_common.constprop.3+0xac/0x170
[103370.416421]  do_el0_svc+0x70/0x90
[103370.421197]  el0_svc+0x18/0x28
[103370.424666]  el0_sync_handler+0x90/0xb8
[103370.427706]  el0_sync+0x184/0x1c0
[103370.432069] Mem-Info:
[103370.435104] active_anon:69 inactive_anon:492 isolated_anon:0
[103370.435104]  active_file:0 inactive_file:21 isolated_file:0
[103370.435104]  unevictable:0 dirty:0 writeback:0
[103370.435104]  slab_reclaimable:1867 slab_unreclaimable:11472
[103370.435104]  mapped:35 shmem:70 pagetables:123 bounce:0
[103370.435104]  free:4920 free_pcp:0 free_cma:0
[103370.446371] Node 0 active_anon:276kB inactive_anon:1968kB active_file:0kB inactive_file:84kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:140kB dirty:0kB writeback:0kB shmem:280kB writeback_tmp:0kB kernel_stack:1456kB all_unreclaimable? yes
[103370.469211] DMA free:19680kB min:16384kB low:20480kB high:24576kB reserved_highatomic:4096KB active_anon:276kB inactive_anon:1968kB active_file:308kB inactive_file:160kB unevictable:0kB writepending:0kB present:524288kB managed:375880kB mlocked:0kB pagetables:492kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[103370.497190] lowmem_reserve[]: 0 0 0 0
[103370.519386] DMA: 581*4kB (UMEH) 212*8kB (UMEH) 110*16kB (UMEH) 100*32kB (UMEH) 28*64kB (UMEH) 13*128kB (UME) 3*256kB (M) 4*512kB (UM) 1*1024kB (M) 0*2048kB 1*4096kB (M) = 20372kB
[103370.523046] 94 total pagecache pages
[103370.538910] 0 pages in swap cache
[103370.542726] Swap cache stats: add 0, delete 0, find 0/0
[103370.546027] Free swap  = 0kB
[103370.551491] Total swap = 0kB
[103370.554356] 131072 pages RAM
[103370.557306] 0 pages HighMem/MovableOnly
[103370.560254] 37102 pages reserved
[103370.564253] Tasks state (memory values in pages):
[103370.567471] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[103370.572188] [    842]    81   842      304       28    36864        0             0 ubusd
[103370.580764] [    843]     0   843      205        9    28672        0             0 askfirst
[103370.589096] [    877]     0   877      237       11    24576        0             0 urngd
[103370.597689] [   1326]   514  1326      296       38    32768        0             0 logd
[103370.605849] [   1682]     0  1682      254       10    28672        0             0 dropbear
[103370.614099] [   1802]     0  1802      943       52    45056        0             0 hostapd
[103370.622604] [   1803]     0  1803      922       31    36864        0             0 wpa_supplicant
[103370.631110] [   1863]     0  1863      452       54    32768        0             0 netifd
[103370.639875] [   1917]     0  1917      358       29    36864        0             0 odhcpd
[103370.648209] [   2585]     0  2585      306       14    28672        0             0 ntpd
[103370.656541] [   3327]     0  3327      319       37    32768        0             0 pppd
[103370.664880] [  19363]   453 19363      341       28    32768        0             0 dnsmasq
[103370.673035] [  19368]     0 19368      532       48    32768        0             0 rpcd
[103370.681546] [  19381]     0 19381      827       36    40960        0             0 uhttpd
[103370.689620] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=hostapd,pid=1802,uid=0
[103370.697797] Out of memory: Killed process 1802 (hostapd) total-vm:3772kB, anon-rss:208kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:44kB oom_score_adj:0
[103370.710894] oom_reaper: reaped process 1802 (hostapd), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[103370.724642] netifd invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
[103370.733868] CPU: 2 PID: 1863 Comm: netifd Not tainted 5.10.42 #0
[103370.742120] Hardware name: Xiaomi AX3600 (DT)
[103370.748104] Call trace:
[103370.752451]  dump_backtrace+0x0/0x1b0
[103370.755136]  show_stack+0x18/0x30
[103370.758694]  dump_stack+0xdc/0x11c
[103370.762079]  dump_header+0x44/0x194
[103370.765482]  oom_kill_process+0x1d0/0x1d8
[103370.769288]  out_of_memory+0x20c/0x320
[103370.773192]  __alloc_pages_slowpath.constprop.150+0x6c8/0x9b0
[103370.776927]  __alloc_pages_nodemask+0x1f4/0x238
[103370.782828]  __get_free_pages+0x1c/0x68
[103370.787602]  proc_pid_readlink+0x74/0x2b8
[103370.791509]  vfs_readlink+0x11c/0x138
[103370.795415]  do_readlinkat+0xac/0x128
[103370.799146]  __arm64_sys_readlinkat+0x20/0x30
[103370.802883]  el0_svc_common.constprop.3+0xac/0x170
[103370.807309]  do_el0_svc+0x70/0x90
[103370.812084]  el0_svc+0x18/0x28
[103370.815552]  el0_sync_handler+0x90/0xb8
[103370.818590]  el0_sync+0x184/0x1c0
[103370.822796] Mem-Info:
[103370.825981] active_anon:69 inactive_anon:454 isolated_anon:0
[103370.825981]  active_file:16 inactive_file:21 isolated_file:0
[103370.825981]  unevictable:0 dirty:0 writeback:0
[103370.825981]  slab_reclaimable:1867 slab_unreclaimable:11472
[103370.825981]  mapped:35 shmem:70 pagetables:123 bounce:0
[103370.825981]  free:5051 free_pcp:74 free_cma:0
[103370.837614] Node 0 active_anon:276kB inactive_anon:1816kB active_file:0kB inactive_file:272kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:140kB dirty:0kB writeback:0kB shmem:280kB writeback_tmp:0kB kernel_stack:1488kB all_unreclaimable? yes
[103370.860350] DMA free:19692kB min:16384kB low:20480kB high:24576kB reserved_highatomic:0KB active_anon:276kB inactive_anon:1816kB active_file:460kB inactive_file:0kB unevictable:0kB writepending:0kB present:524288kB managed:375880kB mlocked:0kB pagetables:492kB bounce:0kB free_pcp:580kB local_pcp:0kB free_cma:0kB
[103370.887959] lowmem_reserve[]: 0 0 0 0
[103370.910182] DMA: 335*4kB (UME) 152*8kB (UME) 115*16kB (UME) 96*32kB (UME) 29*64kB (UME) 13*128kB (UME) 3*256kB (M) 4*512kB (UM) 1*1024kB (M) 0*2048kB 1*4096kB (M) = 18924kB
[103370.913949] 151 total pagecache pages
[103370.929465] 0 pages in swap cache
[103370.933083] Swap cache stats: add 0, delete 0, find 0/0
[103370.936477] Free swap  = 0kB
[103370.941940] Total swap = 0kB
[103370.944805] 131072 pages RAM
[103370.947757] 0 pages HighMem/MovableOnly
[103370.950709] 37102 pages reserved
[103370.954703] Tasks state (memory values in pages):
[103370.957916] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[103370.962636] [    842]    81   842      304       71    36864        0             0 ubusd
[103370.971234] [    843]     0   843      205        9    28672        0             0 askfirst
[103370.979623] [    877]     0   877      237       11    24576        0             0 urngd
[103370.988141] [   1326]   514  1326      296       42    32768        0             0 logd
[103370.996380] [   1682]     0  1682      254       10    28672        0             0 dropbear
[103371.004555] [   1803]     0  1803      922       31    36864        0             0 wpa_supplicant
[103371.013130] [   1863]     0  1863      452       54    32768        0             0 netifd
[103371.021907] [   1917]     0  1917      358       29    36864        0             0 odhcpd
[103371.030304] [   2585]     0  2585      306       14    28672        0             0 ntpd
[103371.038576] [   3327]     0  3327      319       37    32768        0             0 pppd
[103371.046912] [  19363]   453 19363      341       28    32768        0             0 dnsmasq
[103371.055130] [  19368]     0 19368      532       48    32768        0             0 rpcd
[103371.063574] [  19381]     0 19381      827       36    40960        0             0 uhttpd
[103371.071739] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=netifd,pid=1863,uid=0
[103371.079832] Out of memory: Killed process 1863 (netifd) total-vm:1808kB, anon-rss:216kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:32kB oom_score_adj:0
[103371.092648] oom_reaper: reaped process 1863 (netifd), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
3483

@kirdes Thanks, I hope to have some time even later today to finally open the box.

Damn, this thing is like 2+ kgs on its own.

@dchard Yeah, we are aware, unfortunately ath11k is really complicated

3484

Thought maybe you can dig something out of the log that helps :slight_smile:

3485

Thanks for sharing, I will check it in detail later but it's just OOM panic.
And it will print only the userspace usage per app so that's not useful.

I mean, we narrowed the leak to ath11k WMI buffer which never gets freed.
Now it's a matter of spending a lot of man-hours trying to figure out why is the callback not being triggered to clear the buffer.
Unfortunately since we do this after work there is never enough time.

4 Likes