Random reboots after x86 upgrade to 23.05.3

johndg · April 4, 2024, 8:50am

Hi,
following my thread about x86 upgrading, I tried the approach of upgrading (from 22.03.5, to 23.05.3) while increasing root size, and using image builder to maintain my packages and settings during upgrades.
while that was no picnic, I'm facing a few issues in the new version.
one of them is this strange kernel stack trace (which to me indicates a serious issue), that appeared twice since the upgrade last night:

Thu Apr  4 10:50:00 2024 kern.warn kernel: [10773.780089] ------------[ cut here ]------------
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10773.785199] WARNING: CPU: 1 PID: 404 at 0xffffffffa03dc75f [nf_conntrack@00000000956ca192+0x13000]
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10773.795091] Modules linked in: pppoe ppp_async nft_fib_inet nf_flow_table_ipv6 nf_flow_table_ipv4 nf_flow_table_inet wireguard pppox ppp_generic nft_reject_ipv6 nft_reject_ipv4 nft_reject_inet nft_reject nft_redir nft_quota nft_objref nft_numgen nft_nat nft_masq nft_log nft_limit nft_hash nft_flow_offload nft_fib_ipv6 nft_fib_ipv4 nft_fib nft_ct nft_counter nft_chain_nat nf_tables libchacha20poly1305 ipt_REJECT curve25519_x86_64 chacha_x86_64 btusb btintel zstd xt_time xt_tcpudp xt_state xt_quota xt_pkttype xt_physdev xt_owner xt_nat xt_multiport xt_mark xt_mac xt_limit xt_conntrack xt_comment xt_cgroup xt_addrtype xt_TCPMSS xt_REDIRECT xt_MASQUERADE xt_LOG xt_CT slhc rfcomm regmap_core r8169 poly1305_x86_64 nfnetlink nf_reject_ipv4 nf_log_syslog nf_flow_table macvlan lzo_rle lzo libcurve25519_generic libchacha kvm_intel kvm irqbypass iptable_nat iptable_mangle iptable_filter ip_tables igc hidp hci_uart forcedeth e1000e crc_ccitt btrtl br_netfilter bnx2 bnep bluetooth fuse evdev i2c_dev
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10773.795223]  ip6table_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip6t_NPT ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 ixgbe e1000 amd_xgbe ip6_udp_tunnel udp_tunnel mdio veth tun cifs oid_registry cifs_md4 cifs_arc4 asn1_decoder dns_resolver nls_utf8 ena ecdh_generic ecc sha512_ssse3 sha512_generic sha256_ssse3 sha256_generic libsha256 seqiv jitterentropy_rng drbg md5 kpp hmac des_generic libdes cmac crypto_acompress nls_iso8859_1 nls_cp437 uas fsl_mph_dr_of ehci_platform ehci_fsl igb vfat fat btrfs zstd_decompress zstd_compress xxhash xor raid6_pq lzo_decompress lzo_compress libcrc32c button_hotplug tg3 realtek mii
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10773.955649] CPU: 1 PID: 404 Comm: kworker/1:2 Not tainted 5.15.150 #0
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10773.962745] Hardware name: Techvision TVI7309X/TVI7309X, BIOS 5.19 09/07/2022
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10773.970609] Workqueue: events 0xffffffffa05cede0 [macvlan@000000002b7e2bdb+0x5000]
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10773.978937] RIP: 0010:0xffffffffa03dc75f [nf_conntrack@00000000956ca192+0x13000]
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10773.987068] Code: 00 00 48 8b 91 a0 0a 00 00 40 0f 95 c6 48 85 d2 74 dd 0f b6 50 04 84 d2 74 d5 48 01 c2 74 d0 40 0f b6 f6 f0 48 0f ab 32 eb c5 <0f> 0b 44 89 f6 44 89 ff 31 db e8 e2 d6 ff ff be 00 02 00 00 48 c7
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10774.007705] RSP: 0018:ffffc9000007cd08 EFLAGS: 00010202
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10774.013455] RAX: 0000000000000188 RBX: 0000000000000000 RCX: 0000000000000000
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10774.021302] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffffffffa03ea694
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10774.029163] RBP: ffffc9000007cd58 R08: 0dde34d04bd2aa49 R09: 0000000000000000
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10774.037027] R10: ffffc9000007ccd8 R11: 5d82c1f27a5d9c69 R12: ffff88810c777e00
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10774.044868] R13: 0000000000000000 R14: 0000000000005275 R15: 0000000000028da8
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10774.052756] FS:  0000000000000000(0000) GS:ffff888277e80000(0000) knlGS:0000000000000000
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10774.061662] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10774.068011] CR2: 00007f35e370b748 CR3: 000000000340a000 CR4: 0000000000350ee0
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10774.075848] Call Trace:
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10774.078547]  <IRQ>
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10774.080768]  ? 0xffffffff81a88afd
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10774.084418]  ? 0xffffffff81a88c27
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10774.088062]  ? 0xffffffff810bd8d4
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10774.091711]  ? 0xffffffffa03dc75f [nf_conntrack@00000000956ca192+0x13000]
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10774.099170]  ? 0xffffffff813f6f56
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10774.102813]  ? 0xffffffff81ad8b28
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10774.106458]  ? 0xffffffff81ad8c38
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10774.110119]  ? 0xffffffff81c00b4b
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10774.113773]  ? 0xffffffffa03dc75f [nf_conntrack@00000000956ca192+0x13000]
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10774.121259]  ? 0xffffffffa03dc36a [nf_conntrack@00000000956ca192+0x13000]
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10774.128732]  0xffffffffa03e0eb4 [nf_conntrack@00000000956ca192+0x13000]
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10774.135997]  0xffffffffa03e0f1c [nf_conntrack@00000000956ca192+0x13000]
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10774.143268]  0xffffffff8195e4fc
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10774.146740]  0xffffffff81968aff
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10774.150204]  ? 0xffffffff81968a10
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10774.153851]  0xffffffff81968f97
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10774.157308]  ? 0xffffffff81968d80
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10774.160971]  0xffffffff818dfaa6
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10774.164424]  0xffffffff818dfc65
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10774.167884]  0xffffffff818de6b3
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10774.171356]  0xffffffff818deb04
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10774.174809]  0xffffffff81e000c2
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10774.178285]  0xffffffff810c2b78
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10774.181743]  </IRQ>
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10774.184057]  <TASK>
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10774.186366]  0xffffffff818da8fa
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10774.189824]  0xffffffffa05cec47 [macvlan@000000002b7e2bdb+0x5000]
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10774.196531]  0xffffffffa05cef33 [macvlan@000000002b7e2bdb+0x5000]
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10774.203230]  0xffffffff810d5f28
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10774.206688]  0xffffffff810d60de
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10774.210144]  ? 0xffffffff810d6090
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10774.213790]  0xffffffff810dd1b8
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10774.217250]  ? 0xffffffff810dd090
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10774.220904]  0xffffffff81003c2f
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10774.224361]  </TASK>
Thu Apr  4 10:50:00 2024 kern.warn kernel: [10774.226778] ---[ end trace 3c4de4fbb6c8eb60 ]---

system details:

❯ ubus call system board
{
        "kernel": "5.15.150",
        "hostname": "OpenWrt",
        "system": "Intel(R) Celeron(R) N5105 @ 2.00GHz",
        "model": "Techvision TVI7309X",
        "board_name": "techvision-tvi7309x",
        "rootfs_type": "ext4",
        "release": {
                "distribution": "OpenWrt",
                "version": "23.05.3",
                "revision": "r23809-234f1a2efa",
                "target": "x86/64",
                "description": "OpenWrt 23.05.3 r23809-234f1a2efa"
        }
}

there are no other suspicions log lines, except for docker issues, don't know if it's related.
the system also randomly rebooted at an unknown time overnight.

can anyone help with this?

LilRedDog · April 4, 2024, 10:40pm

The only help I can provide is to ask you to give more information; if you don't mind.

Please connect to your OpenWrt device using ssh and copy the output of the following commands and post it here using the "Preformatted text </> " button:

Remember to redact passwords, MAC addresses and any public IP addresses you may have:
cat /etc/config/network
cat /etc/config/wireless
cat /etc/config/dhcp
cat /etc/config/firewall

I removed the system board call since you have already provided it.
It may help but may be a waste of ~20 seconds of typing, copying and pasting.

Oh, and did you try to save settings when you upgraded or did you configure after?

johndg · April 5, 2024, 6:00am

well I had to reformat, so settings were backed up and later restored.
yesterday had 2 reboots, and another stack trace "warning".

here are the logs:
❯ cat /etc/config/network

config interface 'loopback'
        option device 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'fd85:c145:3d1a::/48'

config device
        option name 'br-lan'
        option type 'bridge'
        list ports 'eth1'
        list ports 'eth2'
        list ports 'eth3'

config interface 'lan'
        option device 'br-lan'
        option ipaddr '192.168.1.1'
        option netmask '255.255.255.0'
        option ip6assign '60'
        option proto 'static'
        list dns '192.168.1.1'

config interface 'wan'
        option proto 'dhcp'
        option device 'eth0'

config interface 'wan6'
        option proto 'dhcpv6'
        option device 'eth0'
        option reqaddress 'try'
        option reqprefix 'auto'
        option auto '0'

config interface 'docker'
        option device 'docker0'
        option proto 'none'
        option auto '0'

config device
        option type 'bridge'
        option name 'docker0'

config interface 'iot_online'
        option device 'eth1.10'
        option ipaddr '192.168.10.1'
        option netmask '255.255.255.0'
        option proto 'static'

config interface 'iot_offline'
        option device 'eth1.11'
        option ipaddr '192.168.11.1'
        option netmask '255.255.255.0'
        option proto 'static'

config interface 'guest'
        option device 'eth1.12'
        option ipaddr '192.168.12.1'
        option netmask '255.255.255.0'
        option proto 'static'

config interface 'macvlan'
        option proto 'static'
        option device 'br-lan.15'
        option ipaddr '192.168.15.1'
        option netmask '255.255.0.0'

config device
        option type 'macvlan'
        option ifname 'br-lan'
        option mode 'bridge'
        option name 'br-lan.15'
        option acceptlocal '1'
        option promisc '1'

the router has no wireless (I do have a "dumb ap")

❯ cat /etc/config/dhcp

config dnsmasq
        option domainneeded '1'
        option localise_queries '1'
        option local '/lan/'
        option domain 'lan'
        option expandhosts '1'
        option authoritative '1'
        option readethers '1'
        option leasefile '/tmp/dhcp.leases'
        option localservice '1'
        option ednspacket_max '1232'
        option noresolv '1'
        option cachesize '1000'
        option rebind_protection '0'
        option port '54'
        list server '192.168.1.1'

config dhcp 'lan'
        option interface 'lan'
        option start '100'
        option limit '150'
        option leasetime '12h'
        option dhcpv4 'server'
        option dhcpv6 'server'
        option ra 'server'
        list ra_flags 'managed-config'
        list ra_flags 'other-config'
        list dhcp_option '6,192.168.1.1'
        list dhcp_option '3,192.168.1.1'
        list dns '2a0d:6fc2:4820:3d00::1'
        list dns 'fd85:c145:3d1a::1'

config dhcp 'wan'
        option interface 'wan'
        option ignore '1'

config odhcpd 'odhcpd'
        option maindhcp '0'
        option leasefile '/tmp/hosts/odhcpd'
        option leasetrigger '/usr/sbin/odhcpd-update'
        option loglevel '4'

config dhcp 'iot_online'
        option interface 'iot_online'
        option start '100'
        option limit '150'
        option leasetime '12h'
        list dhcp_option '6,8.8.8.8'

config dhcp 'iot_offline'
        option interface 'iot_offline'
        option start '100'
        option limit '150'
        option leasetime '12h'

config dhcp 'guest'
        option interface 'guest'
        option start '100'
        option limit '150'
        option leasetime '12h'
        list dhcp_option '6,192.168.1.1'

plus a bunch of "static leases"

❯ cat /etc/config/firewall

config defaults
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option synflood_protect '1'

config zone
        option name 'lan'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'
        list network 'lan'

config zone
        option name 'wan'
        option input 'REJECT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option masq '1'
        option mtu_fix '1'
        list network 'wan'
        list network 'wan6'

config forwarding
        option src 'lan'
        option dest 'wan'

config rule
        option name 'Allow-DHCP-Renew'
        option src 'wan'
        option proto 'udp'
        option dest_port '68'
        option target 'ACCEPT'
        option family 'ipv4'

config rule
        option name 'Allow-Ping'
        option src 'wan'
        option proto 'icmp'
        option icmp_type 'echo-request'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-IGMP'
        option src 'wan'
        option proto 'igmp'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-DHCPv6'
        option src 'wan'
        option proto 'udp'
        option dest_port '546'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-MLD'
        option src 'wan'
        option proto 'icmp'
        option src_ip 'fe80::/10'
        list icmp_type '130/0'
        list icmp_type '131/0'
        list icmp_type '132/0'
        list icmp_type '143/0'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Input'
        option src 'wan'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        list icmp_type 'router-solicitation'
        list icmp_type 'neighbour-solicitation'
        list icmp_type 'router-advertisement'
        list icmp_type 'neighbour-advertisement'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Forward'
        option src 'wan'
        option dest '*'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-IPSec-ESP'
        option src 'wan'
        option dest 'lan'
        option proto 'esp'
        option target 'ACCEPT'

config rule
        option name 'Allow-ISAKMP'
        option src 'wan'
        option dest 'lan'
        option dest_port '500'
        option proto 'udp'
        option target 'ACCEPT'

config zone 'docker'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'
        option name 'docker'
        list network 'docker'

config zone
        option name 'iot_online'
        option network 'iot_online'
        option output 'ACCEPT'
        option forward 'REJECT'
        option input 'REJECT'

config zone
        option name 'iot_offline'
        option network 'iot_offline'
        option output 'ACCEPT'
        option forward 'REJECT'
        option input 'REJECT'

config zone
        option name 'guest'
        option network 'guest'
        option output 'ACCEPT'
        option forward 'REJECT'
        option input 'REJECT'

config forwarding
        option src 'iot_online'
        option dest 'wan'

config forwarding
        option src 'lan'
        option dest 'iot_online'

config forwarding
        option src 'lan'
        option dest 'iot_offline'

config forwarding
        option src 'guest'
        option dest 'wan'

config forwarding
        option src 'lan'
        option dest 'guest'

config rule
        option name 'Allow-iot-offline-DHCP'
        list proto 'udp'
        option dest_port '67-68'
        option target 'ACCEPT'
        option src 'iot_offline'

config rule
        option name 'Allow-iot-online-DNS'
        option dest_port '53'
        option target 'ACCEPT'
        list proto 'tcp'
        list proto 'udp'
        option src 'iot_online'

config rule
        option name 'Allow-iot-online-DHCP'
        list proto 'udp'
        option dest_port '67-68'
        option target 'ACCEPT'
        option src 'iot_online'

config rule
        option name 'Allow-guest-DNS'
        option dest_port '53'
        option target 'ACCEPT'
        list proto 'tcp'
        list proto 'udp'
        option src 'guest'

config rule
        option name 'Allow-guest-DHCP'
        list proto 'udp'
        option dest_port '67-68'
        option target 'ACCEPT'
        option src 'guest'

config rule
        option name 'Allow-iot-HA'
        list proto 'udp'
        list proto 'tcp'
        option dest_port '80 8123'
        option target 'ACCEPT'
        option src 'iot_offline'
        option dest 'lan'
        list dest_ip '192.168.1.179'

config include 'pbr'
        option fw4_compatible '1'
        option type 'script'
        option path '/usr/share/pbr/pbr.firewall.include'

config zone
        option name 'macvlan'
        list network 'macvlan'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'

config forwarding
        option src 'macvlan'
        option dest 'lan'

config forwarding
        option src 'lan'
        option dest 'macvlan'

decisoft · June 10, 2025, 7:38am

Hi!

Did you manage to solve this issue? I'm having these random reboots on same motherboard, on two fresh OpenWRT installations (24.10.1). Even I reinstalled twice just in case.
Thanks

brada4 · June 10, 2025, 8:18am

Do you have intel ucode installed?

decisoft · June 10, 2025, 8:40am

I have installed intel-microcode, don't know if that's the package you point out. I also have installed other intel related packages such as i915-firmware-dmc, kmod-drm-i915, kmod-dwmac-intel, kmod-e1000...

But I see that iucode-tool is not installed (Description: intel microcode loader). Maybe that's the one?

Thanks for the quick reply, really appreciated

brada4 · June 10, 2025, 9:57am

dmesg | grep microcode

it is just the cpu microvode package. Others are for peripherials.

decisoft · June 10, 2025, 3:42pm

# dmesg | grep microcode
[    1.301963] Register File Data Sampling: Vulnerable: No microcode
[    1.301963] SRBDS: Vulnerable: No microcode
[    3.143886] microcode: Current revision: 0x24000024
[    3.148978] microcode: Microcode Update Driver: v2.2.

It has rebooted three times in the last hour, twice this morning (plus more last night). I'm kinda frustrating. I'll try to research deeper tonight.

brada4 · June 10, 2025, 3:44pm

Can you boot ubuntu or fedora live USB and wait for reboots (or verify they are stable)?

frollic · June 10, 2025, 3:56pm

It isn't used for anything else but routing?