Random reboots APU2 x86/64

I have an APU2 running OpenWrt 22.03.5 x86/64.
I noticed some days ago that the router was disconnecting.
I managed to capture this log.
It seems that the router is rebooting due to a kernel problem.
Can someone help me why it keeps crashing?

192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65922.992588] ------------[ cut here ]------------
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	crit	kernel	[65922.997308] Kernel BUG at 0xffffffff8111c11a [verbose debug info unavailable]
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.004515] invalid opcode: 0000 [#1] SMP NOPTI
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.009077] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 5.10.176 #0
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.015188] Hardware name: PC Engines apu2/apu2, BIOS v4.13.0.1 11/25/2020
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.022083] RIP: 0010:0xffffffff8111c11a
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.026026] Code: 00 49 c7 c4 ff ff ff ff 48 83 c4 58 4c 89 e0 5b 41 5c 41 5d 41 5e 41 5f 5d e9 b2 47 ce 00 4c 2b 70 18 49 21 f6 e9 ef fe ff ff <0f> 0b 8b 45 98 44 8d 34 03 44 39 f3 73 16 48 8b 45 d0 31 f6 4c 89
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.044809] RSP: 0018:ffffc90000120460 EFLAGS: 00010246
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.050057] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.057206] RDX: ffff8881008eb078 RSI: 0000000000200000 RDI: 0000000000000000
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.064356] RBP: ffffc900001204e0 R08: 0000000000000001 R09: 0000000000000000
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.071506] R10: 0000000104978804 R11: 0000000000000000 R12: 0000000000200000
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.078656] R13: 0000000104978804 R14: 00000000cbe87000 R15: ffff8881026fb554
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.085809] FS:  0000000000000000(0000) GS:ffff88812ad80000(0000) knlGS:0000000000000000
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.093913] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.099672] CR2: 00007fd2242a9000 CR3: 00000001022d2000 CR4: 00000000000406e0
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.106831] Call Trace:
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.109299]  <IRQ>
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.111331]  ? 0xffffffff81a48789
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.114676]  ? 0xffffffffa047ccec [mac80211@00000000a4ce113f+0x7b000]
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.121132]  0xffffffff8111c461
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.124295]  0xffffffff8111a2a5
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.127468]  0xffffffffa06ad932 [ath9k@00000000205c32fd+0x1a000]
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.133495]  0xffffffffa06af12f [ath9k@00000000205c32fd+0x1a000]
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.139514]  ? 0xffffffff810e1630
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.142847]  ? 0xffffffff81a48789
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.146189]  ? 0xffffffffa047aa35 [mac80211@00000000a4ce113f+0x7b000]
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.152648]  0xffffffffa06af5b6 [ath9k@00000000205c32fd+0x1a000]
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.158674]  0xffffffffa06afb59 [ath9k@00000000205c32fd+0x1a000]
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.164703]  0xffffffffa047a574 [mac80211@00000000a4ce113f+0x7b000]
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.170987]  0xffffffffa047fdb3 [mac80211@00000000a4ce113f+0x7b000]
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.177270]  ? 0xffffffffa0058187 [igb@000000001f4b3a8d+0x27000]
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.183307]  0xffffffffa047ffe9 [mac80211@00000000a4ce113f+0x7b000]
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.189587]  ? 0xffffffff81866343
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.192925]  0xffffffff81865e6e
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.196088]  0xffffffff81866671
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.199252]  ? 0xffffffff819c5cdc
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.202590]  0xffffffff8186689b
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.205750]  0xffffffff819c5b50
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.208913]  ? 0xffffffff819dc493
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.212258]  0xffffffff819c5cdc
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.215412]  ? 0xffffffff819dc493
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.218744]  0xffffffff819c5e22
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.221904]  ? 0xffffffff819dc299
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.225233]  ? 0xffffffff819dc493
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.228569]  0xffffffff819c6071
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.231732]  0xffffffff819c2f15
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.234898]  0xffffffff81865e6e
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.238061]  0xffffffff81866671
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.241223]  0xffffffff8186689b
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.244386]  0xffffffff819e3bdb
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.247551]  0xffffffff81865e6e
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.250715]  0xffffffff818b1b6b
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.253878]  0xffffffff818b1e15
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.257041]  0xffffffff818662f2
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.260201]  ? 0xffffffff81200478
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.263530]  ? 0xffffffff811c8c4f
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.266858]  ? 0xffffffff818afc46
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.270195]  0xffffffff8186689b
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.273357]  0xffffffff81922ca1
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.276513]  0xffffffff81922d00
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.279678]  0xffffffff81922e0a
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.282840]  ? 0xffffffff8111ee84
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.286179]  0xffffffff8186e47d
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.289339]  0xffffffff818729ab
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.292503]  ? 0xffffffff818727d0
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.295839]  0xffffffff8111e296
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.299004]  0xffffffff8111ec65
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.302168]  ? 0xffffffff8112e5ae
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.305503]  0xffffffff81e000be
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.308666]  0xffffffff81c010a2
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.311828]  </IRQ>
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.313943]  0xffffffff8102ce6c
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.317098]  0xffffffff810bfdbf
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.320254]  0xffffffff81a402cd
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.323417]  0xffffffff81c00d42
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.326580] RIP: 0010:0xffffffff817f025b
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.330526] Code: 54 8f ff 65 8b 3d 05 54 82 7e e8 30 53 8f ff 31 ff 49 89 c6 e8 d6 5f 8f ff 80 7d d7 00 0f 85 9e 01 00 00 fb 66 0f 1f 44 00 00 <45> 85 ff 0f 88 cf 00 00 00 49 63 cf 48 8d 04 49 48 8d 14 81 48 c1
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.349304] RSP: 0018:ffffc90000093e80 EFLAGS: 00000246
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.354549] RAX: ffff88812ada52c0 RBX: ffff888100f01c00 RCX: 000000000000001f
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.361699] RDX: 0000000000000000 RSI: 00000000803d74ed RDI: 0000000000000000
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.368860] RBP: ffffc90000093eb8 R08: 00003bf4e48a766b R09: 0000000000000000
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.376006] R10: 0000000000000008 R11: 0000000000000000 R12: 0000000000000002
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.383157] R13: ffffffff824cb2e0 R14: 00003bf4e48a766b R15: 0000000000000002
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.390316]  0xffffffff817f04bf
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.393479]  0xffffffff810e6448
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.396643]  0xffffffff810e65eb
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.399814]  0xffffffff8104be87
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.402977]  0xffffffff810000f5
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.406143] Modules linked in: ath9k ath9k_common xt_connlimit pppoe ppp_async nft_fib_inet nf_flow_table_ipv6 nf_flow_table_ipv4 nf_flow_table_inet nf_conncount iptable_nat ath9k_hw ath10k_pci ath10k_core ath xt_state xt_nat xt_helper xt_conntrack xt_connmark xt_connbytes xt_REDIRECT xt_MASQUERADE xt_FLOWOFFLOAD xt_CT wireguard pppox ppp_generic nft_reject_ipv6 nft_reject_ipv4 nft_reject_inet nft_reject nft_redir nft_quota nft_objref nft_numgen nft_nat nft_masq nft_log nft_limit nft_hash nft_flow_offload nft_fib_ipv6 nft_fib_ipv4 nft_fib nft_ct nft_counter nft_compat nft_chain_nat nf_tables nf_nat nf_flow_table nf_conntrack mac80211 libchacha20poly1305 ipt_REJECT curve25519_x86_64 chacha_x86_64 cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_recent xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_ecn xt_dscp xt_comment xt_TCPMSS xt_LOG xt_HL xt_DSCP xt_CLASSIFY slhc sch_cake r8169 poly1305_x86_64 nfnetlink nf_reject_ipv4 nf_log_ipv6 nf_log_ipv4 nf_log_common nf_defrag_ipv6
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.406428]  nf_defrag_ipv4 libcurve25519_generic libcrc32c libchacha iptable_raw iptable_mangle iptable_filter ipt_ECN ip_tables igc forcedeth e1000e crc_ccitt compat bnx2 sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred act_gact pcengines_apuv2 i2c_dev snd_pcsp ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 ixgbe e1000 amd_xgbe ifb ip6_udp_tunnel udp_tunnel mdio snd_rawmidi snd_seq_device snd_pcm_oss snd_mixer_oss snd_hwdep snd_compress snd_pcm snd_timer snd soundcore vfat fat nls_utf8 nls_iso8859_1 nls_cp437 ena sha256_ssse3 sha1_ssse3 seqiv jitterentropy_rng drbg kpp ccp_crypto rsa_generic mpi asn1_decoder akcipher ccp sha256_generic sha1_generic libsha256 hmac cmac authenc xhci_plat_hcd fsl_mph_dr_of ehci_platform ehci_fsl igb button_hotplug gpio_amd_fch tg3 ptp realtek pps_core mii
192.168.1.1	Jul 26 20:47:24	OpenWrt	kern	warning	kernel	[65923.573503] ---[ end trace 2bd54b645b9d887f ]---

My APU2 has been up for 42 days now, kernel 5.10.176, BIOS 4.0.7 2017-02-28. See if you're on the same kernel or not.

Have you tried running memtest? That could explain invalid opcodes.

What power supply are you using? Have you tried swapping it for a different supply? A flaky PSU could cause memory/bus/data corruption too.

Have you updated the BIOS or any other firmware at any point?

Have you owned this unit for a long time (so you know this is a new fault), or was it recently obtained second hand?

Thank you for your reply.
I have the following answers.

My APU2 has the following versions installed:

  • BIOS: 4.13.0.1 11/25/2020
  • Kernel: 5.10.176

I have ordered a new power supply (12V, 2A - the standard one sold by PCEngines).
It wil take some days before the supply arrives. In the mean time I will try to run a memtest.

I didn't change any firmware (apart from installing OpenWrt).
I bought the board 2 to 3 years ago.

Like you and @CopperCassette, I've got an APU2 as my edge router. I'm running 22.03.5 (5.10.176), BIOS 4.7.0.1 06/22/2022. I have not seen any issues like that, I've been running 5.10 kernels since last Sep when fw4 came out in the SNAPSHOT, then installed 22.03 when it was finalized...

+1 on running memtest, at least rule that out with zero cost before anything else.

What "extras" are you running? For example, I have stubby upstream of dnsmasq and apcupsd to monitor the UPS it's on...

Memtest has not found any errors.
Extras are:

  • acme
  • ddns-scripts
  • wireguard

Maybe I should try a clean install of OpenWrt and see if it runs stable.

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.