Kernel oops, Netgear R7800, 22.03.3

atownlede · February 2, 2023, 3:16am

I built my system with image-builder, so it's stock kernel and modules.

Anything useful in the oops traces?

ubus call system board:

{
	"kernel": "5.10.161",
	"hostname": "LEDE",
	"system": "ARMv7 Processor rev 0 (v7l)",
	"model": "Netgear Nighthawk X4S R7800",
	"board_name": "netgear,r7800",
	"rootfs_type": "squashfs",
	"release": {
		"distribution": "OpenWrt",
		"version": "22.03.3",
		"revision": "r20028-43d71ad93e",
		"target": "ipq806x/generic",
		"description": "OpenWrt 22.03.3 r20028-43d71ad93e"
	}
}

and dmesg-ramoops-0 (From the "cut here" section onwards):

<1>[281002.003918] 8<--- cut here ---
<1>[281002.003953] Unable to handle kernel paging request at virtual address c0068574
<1>[281002.005877] pgd = a47da87d
<1>[281002.013164] [c0068574] *pgd=4201141e(bad)
<0>[281002.015948] Internal error: Oops: 8000000d [#1] SMP ARM
<4>[281002.020105] Modules linked in: pppoe ppp_async nft_fib_inet nf_flow_table_ipv6 nf_flow_table_ipv4 nf_flow_table_inet ath10k_pci ath10k_core ath wireguard pppox ppp_generic nft_reject_ipv6 nft_reject_ipv4 nft_reject_inet nft_reject nft_redir nft_quota nft_objref nft_numgen nft_nat nft_masq nft_log nft_limit nft_hash nft_flow_offload nft_fib_ipv6 nft_fib_ipv4 nft_fib nft_ct nft_counter nft_chain_nat nf_tables nf_nat nf_flow_table nf_conntrack mac80211 libchacha20poly1305 iptable_mangle iptable_filter ipt_REJECT ipt_ECN ip_tables curve25519_neon cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_ecn xt_dscp xt_comment xt_TCPMSS xt_LOG xt_HL xt_DSCP xt_CLASSIFY x_tables slhc sch_cake poly1305_arm nfnetlink nf_reject_ipv6 nf_reject_ipv4 nf_log_ipv6 nf_log_ipv4 nf_log_common nf_defrag_ipv6 nf_defrag_ipv4 libcurve25519_generic libcrc32c crc_ccitt compat chacha_neon sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_tcindex cls_route
<4>[281002.020675]  cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred act_gact ledtrig_usbport ifb ip6_udp_tunnel udp_tunnel sit tunnel4 ip_tunnel seqiv kpp cmac uas usb_storage leds_gpio xhci_plat_hcd xhci_pci xhci_hcd dwc3 dwc3_qcom ohci_platform ohci_hcd phy_qcom_ipq806x_usb ahci fsl_mph_dr_of ehci_platform ehci_fsl sd_mod ahci_platform libahci_platform libahci libata scsi_mod ehci_hcd ramoops reed_solomon pstore gpio_button_hotplug f2fs ext4 mbcache jbd2 crc32c_generic crc32_generic
<4>[281002.134085] CPU: 1 PID: 21037 Comm: kworker/1:2 Not tainted 5.10.138 #0
<4>[281002.156310] Hardware name: Generic DT based system
<4>[281002.163086] Workqueue: events 0xc07b7aac
<4>[281002.167675] PC is at 0xc0068574
<4>[281002.171838] LR is at 0xc062e318
<4>[281002.175139] pc : [<c0068574>]    lr : [<c062e318>]    psr: a0000193
<4>[281002.178357] sp : c37cbd70  ip : 00000000  fp : c1aeef00
<4>[281002.184694] r10: c10e4c18  r9 : 00000000  r8 : c37cbdd4
<4>[281002.189986] r7 : 00000002  r6 : e96766d9  r5 : 00000006  r4 : c0ccf440
<4>[281002.195288] r3 : c0068574  r2 : c0c9b31c  r1 : 00000000  r0 : e96766d9
<4>[281002.201624] Flags: NzCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment none
<4>[281002.208217] Control: 10c5787d  Table: 48e4006a  DAC: 00000051
<0>[281002.215516] Process kworker/1:2 (pid: 21037, stack limit = 0x2c2b1da2)
<0>[281002.221416] Stack: (0xc37cbd70 to 0xc37cc000)
<0>[281002.227931] bd60:                                     c10f8fc0 00000001 20000113 c06962f4
<0>[281002.232455] bd80: c10f8fd8 00000001 ffffffff c0696374 c10f8fe4 c0c80314 ffffffff c0697ac8
<0>[281002.240708] bda0: 00000000 c0c80314 ffffffff c0341a40 c10e4c00 c0c80314 c10ed840 00000002
<0>[281002.248954] bdc0: 23c34600 c1188900 00000000 c068672c c1196e00 c10fb4c0 23c34600 2faf0800
<0>[281002.257200] bde0: c1196e00 2faf0800 c10ed840 00000000 c1184240 c068ab98 c10ed728 c1184240
<0>[281002.265449] be00: c0696820 c10ca3c0 23c34600 c1188900 00000000 c068abdc c10ed840 00000000
<0>[281002.273692] be20: 2faf0800 c1184240 c1188880 c1188900 00000000 c068ae08 000f387c 2faf0800
<0>[281002.281934] be40: 00000000 ffffffff 2faf0800 c0c825c0 c1188938 c1ae9800 2faf0800 dd99a010
<0>[281002.290184] be60: 23c34600 c1188880 c1188900 c068af88 c1916800 2faf0800 dd99a010 23c34600
<0>[281002.298427] be80: c1188880 c07ae5d0 00000000 c0cd2c08 c1188938 c11888b8 00000000 2faf0800
<0>[281002.306674] bea0: c37ca000 c1917200 00000000 c0cd2bd0 00000002 000c3500 00000000 00000000
<0>[281002.314923] bec0: c37ca000 c07b3864 c1917200 000927c0 000c3500 000000a1 c1917200 c1af0180
<0>[281002.323168] bee0: c1aeee80 c1af0180 c1ae9d80 c1aeee80 00000000 c07b6dfc c1af01b8 00000000
<0>[281002.331416] bf00: c1af0184 c0c90358 00000000 00000040 00000000 c07b7ae4 c1af01b8 c1397000
<0>[281002.339657] bf20: dd9a0940 dd9a3b00 00000000 c0338880 00000008 dd9a0958 c1397000 c1397014
<0>[281002.347907] bf40: dd9a0940 00000008 dd9a0958 c0c03d00 dd9a0b00 c0338b68 c0c9bab8 c0c0c0e8
<0>[281002.356152] bf60: c1397000 c2fa7380 c6b32e80 00000000 c37ca000 c0338af4 c1397000 c953dec4
<0>[281002.364400] bf80: c2fa73a4 c033ea4c 00000000 c6b32e80 c033e8f0 00000000 00000000 00000000
<0>[281002.372645] bfa0: 00000000 00000000 00000000 c0300148 00000000 00000000 00000000 00000000
<0>[281002.380895] bfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<0>[281002.389139] bfe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
<0>[281002.397376] Function entered at [<c062e318>] from [<c06962f4>]
<0>[281002.405615] Function entered at [<c06962f4>] from [<c0696374>]
<0>[281002.411426] Function entered at [<c0696374>] from [<c0697ac8>]
<0>[281002.417331] Function entered at [<c0697ac8>] from [<c0341a40>]
<0>[281002.423232] Function entered at [<c0341a40>] from [<c068672c>]
<0>[281002.429141] Function entered at [<c068672c>] from [<c068ab98>]
<0>[281002.435038] Function entered at [<c068ab98>] from [<c068abdc>]
<0>[281002.440939] Function entered at [<c068abdc>] from [<c068ae08>]
<0>[281002.446843] Function entered at [<c068ae08>] from [<c068af88>]
<0>[281002.452744] Function entered at [<c068af88>] from [<c07ae5d0>]
<0>[281002.458653] Function entered at [<c07ae5d0>] from [<c07b3864>]
<0>[281002.464553] Function entered at [<c07b3864>] from [<c07b6dfc>]
<0>[281002.470455] Function entered at [<c07b6dfc>] from [<c07b7ae4>]
<0>[281002.476359] Function entered at [<c07b7ae4>] from [<c0338880>]
<0>[281002.482262] Function entered at [<c0338880>] from [<c0338b68>]
<0>[281002.488162] Function entered at [<c0338b68>] from [<c033ea4c>]
<0>[281002.494071] Function entered at [<c033ea4c>] from [<c0300148>]
<0>[281002.499969] Exception stack(0xc37cbfb0 to 0xc37cbff8)
<0>[281002.505881] bfa0:                                     00000000 00000000 00000000 00000000
<0>[281002.511101] bfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<0>[281002.519347] bfe0: 00000000 00000000 00000000 00000000 00000013 00000000
<0>[281002.527590] Code: 6765725f 65747369 666e0072 676f6c5f (736e755f) 
<4>[281002.534434] ---[ end trace 1e5aa5b68b159554 ]---

and dmesg-ramoops-1:

<1>[281002.003918] 8<--- cut here ---
<1>[281002.003953] Unable to handle kernel paging request at virtual address c0068574
<1>[281002.005877] pgd = a47da87d
<1>[281002.013164] [c0068574] *pgd=4201141e(bad)
<0>[281002.015948] Internal error: Oops: 8000000d [#1] SMP ARM
<4>[281002.020105] Modules linked in: pppoe ppp_async nft_fib_inet nf_flow_table_ipv6 nf_flow_table_ipv4 nf_flow_table_inet ath10k_pci ath10k_core ath wireguard pppox ppp_generic nft_reject_ipv6 nft_reject_ipv4 nft_reject_inet nft_reject nft_redir nft_quota nft_objref nft_numgen nft_nat nft_masq nft_log nft_limit nft_hash nft_flow_offload nft_fib_ipv6 nft_fib_ipv4 nft_fib nft_ct nft_counter nft_chain_nat nf_tables nf_nat nf_flow_table nf_conntrack mac80211 libchacha20poly1305 iptable_mangle iptable_filter ipt_REJECT ipt_ECN ip_tables curve25519_neon cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_ecn xt_dscp xt_comment xt_TCPMSS xt_LOG xt_HL xt_DSCP xt_CLASSIFY x_tables slhc sch_cake poly1305_arm nfnetlink nf_reject_ipv6 nf_reject_ipv4 nf_log_ipv6 nf_log_ipv4 nf_log_common nf_defrag_ipv6 nf_defrag_ipv4 libcurve25519_generic libcrc32c crc_ccitt compat chacha_neon sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_tcindex cls_route
<4>[281002.020675]  cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred act_gact ledtrig_usbport ifb ip6_udp_tunnel udp_tunnel sit tunnel4 ip_tunnel seqiv kpp cmac uas usb_storage leds_gpio xhci_plat_hcd xhci_pci xhci_hcd dwc3 dwc3_qcom ohci_platform ohci_hcd phy_qcom_ipq806x_usb ahci fsl_mph_dr_of ehci_platform ehci_fsl sd_mod ahci_platform libahci_platform libahci libata scsi_mod ehci_hcd ramoops reed_solomon pstore gpio_button_hotplug f2fs ext4 mbcache jbd2 crc32c_generic crc32_generic
<4>[281002.134085] CPU: 1 PID: 21037 Comm: kworker/1:2 Not tainted 5.10.138 #0
<4>[281002.156310] Hardware name: Generic DT based system
<4>[281002.163086] Workqueue: events 0xc07b7aac
<4>[281002.167675] PC is at 0xc0068574
<4>[281002.171838] LR is at 0xc062e318
<4>[281002.175139] pc : [<c0068574>]    lr : [<c062e318>]    psr: a0000193
<4>[281002.178357] sp : c37cbd70  ip : 00000000  fp : c1aeef00
<4>[281002.184694] r10: c10e4c18  r9 : 00000000  r8 : c37cbdd4
<4>[281002.189986] r7 : 00000002  r6 : e96766d9  r5 : 00000006  r4 : c0ccf440
<4>[281002.195288] r3 : c0068574  r2 : c0c9b31c  r1 : 00000000  r0 : e96766d9
<4>[281002.201624] Flags: NzCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment none
<4>[281002.208217] Control: 10c5787d  Table: 48e4006a  DAC: 00000051
<0>[281002.215516] Process kworker/1:2 (pid: 21037, stack limit = 0x2c2b1da2)
<0>[281002.221416] Stack: (0xc37cbd70 to 0xc37cc000)
<0>[281002.227931] bd60:                                     c10f8fc0 00000001 20000113 c06962f4
<0>[281002.232455] bd80: c10f8fd8 00000001 ffffffff c0696374 c10f8fe4 c0c80314 ffffffff c0697ac8
<0>[281002.240708] bda0: 00000000 c0c80314 ffffffff c0341a40 c10e4c00 c0c80314 c10ed840 00000002
<0>[281002.248954] bdc0: 23c34600 c1188900 00000000 c068672c c1196e00 c10fb4c0 23c34600 2faf0800
<0>[281002.257200] bde0: c1196e00 2faf0800 c10ed840 00000000 c1184240 c068ab98 c10ed728 c1184240
<0>[281002.265449] be00: c0696820 c10ca3c0 23c34600 c1188900 00000000 c068abdc c10ed840 00000000
<0>[281002.273692] be20: 2faf0800 c1184240 c1188880 c1188900 00000000 c068ae08 000f387c 2faf0800
<0>[281002.281934] be40: 00000000 ffffffff 2faf0800 c0c825c0 c1188938 c1ae9800 2faf0800 dd99a010
<0>[281002.290184] be60: 23c34600 c1188880 c1188900 c068af88 c1916800 2faf0800 dd99a010 23c34600
<0>[281002.298427] be80: c1188880 c07ae5d0 00000000 c0cd2c08 c1188938 c11888b8 00000000 2faf0800
<0>[281002.306674] bea0: c37ca000 c1917200 00000000 c0cd2bd0 00000002 000c3500 00000000 00000000
<0>[281002.314923] bec0: c37ca000 c07b3864 c1917200 000927c0 000c3500 000000a1 c1917200 c1af0180
<0>[281002.323168] bee0: c1aeee80 c1af0180 c1ae9d80 c1aeee80 00000000 c07b6dfc c1af01b8 00000000
<0>[281002.331416] bf00: c1af0184 c0c90358 00000000 00000040 00000000 c07b7ae4 c1af01b8 c1397000
<0>[281002.339657] bf20: dd9a0940 dd9a3b00 00000000 c0338880 00000008 dd9a0958 c1397000 c1397014
<0>[281002.347907] bf40: dd9a0940 00000008 dd9a0958 c0c03d00 dd9a0b00 c0338b68 c0c9bab8 c0c0c0e8
<0>[281002.356152] bf60: c1397000 c2fa7380 c6b32e80 00000000 c37ca000 c0338af4 c1397000 c953dec4
<0>[281002.364400] bf80: c2fa73a4 c033ea4c 00000000 c6b32e80 c033e8f0 00000000 00000000 00000000
<0>[281002.372645] bfa0: 00000000 00000000 00000000 c0300148 00000000 00000000 00000000 00000000
<0>[281002.380895] bfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<0>[281002.389139] bfe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
<0>[281002.397376] Function entered at [<c062e318>] from [<c06962f4>]
<0>[281002.405615] Function entered at [<c06962f4>] from [<c0696374>]
<0>[281002.411426] Function entered at [<c0696374>] from [<c0697ac8>]
<0>[281002.417331] Function entered at [<c0697ac8>] from [<c0341a40>]
<0>[281002.423232] Function entered at [<c0341a40>] from [<c068672c>]
<0>[281002.429141] Function entered at [<c068672c>] from [<c068ab98>]
<0>[281002.435038] Function entered at [<c068ab98>] from [<c068abdc>]
<0>[281002.440939] Function entered at [<c068abdc>] from [<c068ae08>]
<0>[281002.446843] Function entered at [<c068ae08>] from [<c068af88>]
<0>[281002.452744] Function entered at [<c068af88>] from [<c07ae5d0>]
<0>[281002.458653] Function entered at [<c07ae5d0>] from [<c07b3864>]
<0>[281002.464553] Function entered at [<c07b3864>] from [<c07b6dfc>]
<0>[281002.470455] Function entered at [<c07b6dfc>] from [<c07b7ae4>]
<0>[281002.476359] Function entered at [<c07b7ae4>] from [<c0338880>]
<0>[281002.482262] Function entered at [<c0338880>] from [<c0338b68>]
<0>[281002.488162] Function entered at [<c0338b68>] from [<c033ea4c>]
<0>[281002.494071] Function entered at [<c033ea4c>] from [<c0300148>]
<0>[281002.499969] Exception stack(0xc37cbfb0 to 0xc37cbff8)
<0>[281002.505881] bfa0:                                     00000000 00000000 00000000 00000000
<0>[281002.511101] bfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<0>[281002.519347] bfe0: 00000000 00000000 00000000 00000000 00000013 00000000
<0>[281002.527590] Code: 6765725f 65747369 666e0072 676f6c5f (736e755f) 
<4>[281002.534434] ---[ end trace 1e5aa5b68b159554 ]---
<0>[281002.556387] Kernel panic - not syncing: Fatal exception
<2>[281002.556436] CPU0: stopping
<4>[281002.560770] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G      D           5.10.138 #0
<4>[281002.563285] Hardware name: Generic DT based system
<4>[281002.570927] Function entered at [<c030e3ac>] from [<c030a138>]
<4>[281002.575698] Function entered at [<c030a138>] from [<c0630908>]
<4>[281002.581601] Function entered at [<c0630908>] from [<c030d0d0>]
<4>[281002.587504] Function entered at [<c030d0d0>] from [<c030d130>]
<4>[281002.593405] Function entered at [<c030d130>] from [<c037168c>]
<4>[281002.599309] Function entered at [<c037168c>] from [<c0649c68>]
<4>[281002.605212] Function entered at [<c0649c68>] from [<c0300b8c>]
<4>[281002.611114] Exception stack(0xc0c01ee0 to 0xc0c01f28)
<4>[281002.617038] 1ee0: 00000000 0000ff92 1ce4a000 dd991d40 00000000 01787a60 c11c7040 00000000
<4>[281002.622250] 1f00: dd991030 0000ff92 00000000 0000ff92 c31d3540 c0c01f30 c07bab1c c07bab3c
<4>[281002.630480] 1f20: 60000013 ffffffff
<4>[281002.638717] Function entered at [<c0300b8c>] from [<c07bab3c>]
<4>[281002.642453] Function entered at [<c07bab3c>] from [<c07bad8c>]
<4>[281002.648094] Function entered at [<c07bad8c>] from [<c034e5b0>]
<4>[281002.653998] Function entered at [<c034e5b0>] from [<c034e908>]
<4>[281002.659898] Function entered at [<c034e908>] from [<c0b00f48>]

psherman · February 2, 2023, 3:48am

A whole bunch of questions...

What version of OpenWrt did you upgrade from? Did you keep settings?

Why did you use image builder? Did you add or remove packages?
When you built your image, did you change the default configurations (i.e. the 'files' directory)?
Do you need to use your own image, or can you use the standard release builds and then customize from there?

atownlede · February 2, 2023, 4:08am

Previous system was OpenWrt 22.03.2. Yes I kept settings.
System was up from approximately January 10th until the system crash/reboot sometime the morning of January 27th.

I use image-builder since it's so much easier to keep all my packages together and simplifies upgrade, no need to go reinstall them all and deal with that mess.
(see package list below, if you're interested)

I do have one custom package, it installs a startup script (rrdbackup) that handles saving/restoring luci statistics files during reboot. It's got nothing kernel related.

No files directory added to the image built for this system.

I need the image to properly save/restore the luci statistics across upgrades. (Need to have the backup/restore init script in place on reboot, otherwise the statistics restarts and there's lost data/history.)
But even without that, I would build my own images for the simplicity of upgrade without having to reinstall packages. I build images for remotely administered systems as well, and those absolutely require images with preinstalled packages so that a remote unattended reboot/upgrade comes up with everything still configured (including wireguard packages and interfaces).

So, is there anything to be gained from the oops reports...anything else that would help with a bug report and any sort of clue as to what caused the crash?

List of packages:

root@LEDE:~# opkg list-installed
6in4 - 28
ath10k-board-qca9984 - 20220411-1
ath10k-firmware-qca9984-ct - 2020-11-08-1
avahi-daemon-service-http - 0.8-7
avahi-daemon-service-ssh - 0.8-7
avahi-dbus-daemon - 0.8-7
avahi-utils - 0.8-7
base-files - 1498-r20028-43d71ad93e
bind-dig - 9.18.10-2
bind-libs - 9.18.10-2
bind-rndc - 9.18.10-2
bind-server - 9.18.10-2
bind-server-filter-aaaa - 9.18.10-2
block-mount - 2022-06-02-93369be0-2
busybox - 1.35.0-5
ca-bundle - 20211016-1
cgi-io - 2022-08-10-901b0f04-21
collectd - 5.12.0-33
collectd-mod-cpu - 5.12.0-33
collectd-mod-csv - 5.12.0-33
collectd-mod-interface - 5.12.0-33
collectd-mod-iwinfo - 5.12.0-33
collectd-mod-load - 5.12.0-33
collectd-mod-lua - 5.12.0-33
collectd-mod-memory - 5.12.0-33
collectd-mod-network - 5.12.0-33
collectd-mod-ping - 5.12.0-33
collectd-mod-rrdtool - 5.12.0-33
collectd-mod-write-graphite - 5.12.0-33
dbus - 1.13.18-12
ddns-scripts - 2.8.2-25
ddns-scripts-services - 2.8.2-25
diffutils - 3.8-1
dnsmasq - 2.86-15
dropbear - 2022.82-2
e2fsprogs - 1.46.5-2
f2fs-tools - 1.14.0-3
f2fsck - 1.14.0-3
firewall4 - 2022-10-18-7ae5e14b-1
fstools - 2022-06-02-93369be0-2
fwtool - 2019-11-12-8f7fe925-1
gdisk - 1.0.8-3
getrandom - 2021-08-03-205defb5-2
hostapd-common - 2022-01-16-cff80b4f-15.2
iperf3 - 3.11-1
iptables-mod-ipopt - 1.8.7-7
iptables-zz-legacy - 1.8.7-7
irqbalance - 1.9.2-1
iw - 5.16-1
iwinfo - 2022-12-15-8d158096-1
jansson4 - 2.13.1-2
jshn - 2022-05-15-d2223ef9-1
jsonfilter - 2018-02-04-c7e938d6-1
kernel - 5.10.161-1-5a2645570c3344ff9ebb222dc1c7f2a0
kmod-ata-ahci - 5.10.161-1
kmod-ata-ahci-platform - 5.10.161-1
kmod-ata-core - 5.10.161-1
kmod-ath - 5.10.161+5.15.81-1-1
kmod-ath10k-ct - 5.10.161+2022-05-13-f808496f-1
kmod-cfg80211 - 5.10.161+5.15.81-1-1
kmod-crypto-aead - 5.10.161-1
kmod-crypto-ccm - 5.10.161-1
kmod-crypto-cmac - 5.10.161-1
kmod-crypto-crc32 - 5.10.161-1
kmod-crypto-crc32c - 5.10.161-1
kmod-crypto-ctr - 5.10.161-1
kmod-crypto-gcm - 5.10.161-1
kmod-crypto-gf128 - 5.10.161-1
kmod-crypto-ghash - 5.10.161-1
kmod-crypto-hash - 5.10.161-1
kmod-crypto-hmac - 5.10.161-1
kmod-crypto-kpp - 5.10.161-1
kmod-crypto-lib-chacha20 - 5.10.161-1
kmod-crypto-lib-chacha20poly1305 - 5.10.161-1
kmod-crypto-lib-curve25519 - 5.10.161-1
kmod-crypto-lib-poly1305 - 5.10.161-1
kmod-crypto-manager - 5.10.161-1
kmod-crypto-null - 5.10.161-1
kmod-crypto-rng - 5.10.161-1
kmod-crypto-seqiv - 5.10.161-1
kmod-crypto-sha256 - 5.10.161-1
kmod-fs-ext4 - 5.10.161-1
kmod-fs-f2fs - 5.10.161-1
kmod-gpio-button-hotplug - 5.10.161-3
kmod-hwmon-core - 5.10.161-1
kmod-ifb - 5.10.161-1
kmod-ipt-core - 5.10.161-1
kmod-ipt-ipopt - 5.10.161-1
kmod-iptunnel - 5.10.161-1
kmod-iptunnel4 - 5.10.161-1
kmod-leds-gpio - 5.10.161-1
kmod-lib-crc-ccitt - 5.10.161-1
kmod-lib-crc16 - 5.10.161-1
kmod-lib-crc32c - 5.10.161-1
kmod-mac80211 - 5.10.161+5.15.81-1-1
kmod-nf-conntrack - 5.10.161-1
kmod-nf-conntrack6 - 5.10.161-1
kmod-nf-flow - 5.10.161-1
kmod-nf-ipt - 5.10.161-1
kmod-nf-log - 5.10.161-1
kmod-nf-log6 - 5.10.161-1
kmod-nf-nat - 5.10.161-1
kmod-nf-reject - 5.10.161-1
kmod-nf-reject6 - 5.10.161-1
kmod-nfnetlink - 5.10.161-1
kmod-nft-core - 5.10.161-1
kmod-nft-fib - 5.10.161-1
kmod-nft-nat - 5.10.161-1
kmod-nft-netdev - 5.10.161-1
kmod-nft-offload - 5.10.161-1
kmod-nls-base - 5.10.161-1
kmod-phy-qcom-ipq806x-usb - 5.10.161-1
kmod-ppp - 5.10.161-1
kmod-pppoe - 5.10.161-1
kmod-pppox - 5.10.161-1
kmod-pstore - 5.10.161-1
kmod-ramoops - 5.10.161-1
kmod-reed-solomon - 5.10.161-1
kmod-sched-cake - 5.10.161-1
kmod-sched-core - 5.10.161-1
kmod-scsi-core - 5.10.161-1
kmod-sit - 5.10.161-1
kmod-slhc - 5.10.161-1
kmod-udptunnel4 - 5.10.161-1
kmod-udptunnel6 - 5.10.161-1
kmod-usb-core - 5.10.161-1
kmod-usb-dwc3 - 5.10.161-1
kmod-usb-dwc3-qcom - 5.10.161-1
kmod-usb-ehci - 5.10.161-1
kmod-usb-ledtrig-usbport - 5.10.161-1
kmod-usb-ohci - 5.10.161-1
kmod-usb-storage - 5.10.161-1
kmod-usb-storage-uas - 5.10.161-1
kmod-usb-xhci-hcd - 5.10.161-1
kmod-usb2 - 5.10.161-1
kmod-usb3 - 5.10.161-1
kmod-wireguard - 5.10.161-1
libatomic1 - 11.2.0-4
libavahi-client - 0.8-7
libavahi-dbus-support - 0.8-7
libblkid1 - 2.37.4-1
libblobmsg-json20220515 - 2022-05-15-d2223ef9-1
libbpf20220308 - 2022-03-08-04c465fd-1
libc - 1.2.3-4
libcap - 2.63-1
libcomerr0 - 1.46.5-2
libdaemon - 0.14-5
libdbus - 1.13.18-12
libelf1 - 0.186-1
libevdev - 1.13.0-1
libevent2-core7 - 2.1.12-1
libevent2-pthreads7 - 2.1.12-1
libexpat - 2.4.9-1
libext2fs2 - 1.46.5-2
libf2fs6 - 1.14.0-3
libgcc1 - 11.2.0-4
libgdbm - 1.21-1
libip4tc2 - 1.8.7-7
libip6tc2 - 1.8.7-7
libiptext0 - 1.8.7-7
libiptext6-0 - 1.8.7-7
libiwinfo-data - 2022-12-15-8d158096-1
libiwinfo-lua - 2022-12-15-8d158096-1
libiwinfo20210430 - 2022-12-15-8d158096-1
libjson-c5 - 0.15-2
libjson-script20220515 - 2022-05-15-d2223ef9-1
libltdl7 - 2.4.6-2
liblua5.1.5 - 5.1.5-10
liblucihttp-lua - 2022-07-08-6e68a106-1
liblucihttp0 - 2022-07-08-6e68a106-1
libmnl0 - 1.0.5-1
libncurses6 - 6.3-2
libnftnl11 - 1.2.1-2
libnghttp2-14 - 1.44.0-1
libnl-tiny1 - 2021-11-21-8e0555fb-1
libopenssl-conf - 1.1.1s-1
libopenssl1.1 - 1.1.1s-1
liboping - 1.10.0-3
libpcap1 - 1.10.1-3
libpcre - 8.45-3
libpthread - 1.2.3-4
libqrencode - 4.1.1-1
librrd1 - 1.0.50-5
librt - 1.2.3-4
libss2 - 1.46.5-2
libstdcpp6 - 11.2.0-4
libubox20220515 - 2022-05-15-d2223ef9-1
libubus-lua - 2022-06-01-2bebf93c-1
libubus20220601 - 2022-06-01-2bebf93c-1
libuci-lua - 2021-10-22-f84f49f0-6
libuci20130104 - 2021-10-22-f84f49f0-6
libuclient20201210 - 2021-05-14-6a6011df-1
libucode20220812 - 2022-12-02-46d93c9c-1
libudev-zero - 1.0.1-1
libusb-1.0-0 - 1.0.24-5
libustream-wolfssl20201210 - 2022-12-08-9217ab46-2
libuuid1 - 2.37.4-1
libuv1 - 1.41.1-1
libwolfssl5.5.4.ee39414e - 5.5.4-stable-1
libxtables12 - 1.8.7-7
logd - 2021-08-03-205defb5-2
lua - 5.1.5-10
luci - git-20.074.84698-ead5e81
luci-app-ddns - git-22.339.34822-6da87df
luci-app-firewall - git-22.089.67563-7e3c1b4
luci-app-opkg - git-22.273.29004-9f6876b
luci-app-sqm - git-21.164.25180-005c09a
luci-app-statistics - git-23.002.71119-463850a
luci-app-tinyproxy - git-20.201.65687-354e4cb
luci-app-uhttpd - git-22.029.54222-e3fbfe9
luci-app-watchcat - git-22.210.22699-010119f
luci-app-wireguard - git-21.322.66896-8ae208d
luci-base - git-22.361.69894-438c598
luci-compat - git-22.069.45071-03bb0e2
luci-lib-base - git-20.232.39649-1f6dc29
luci-lib-ip - git-20.250.76529-62505bd
luci-lib-jsonc - git-22.097.61921-7513345
luci-lib-nixio - git-20.234.06894-c4a4e43
luci-mod-admin-full - git-19.253.48496-3f93650
luci-mod-network - git-22.345.48602-4853e7b
luci-mod-status - git-22.302.42268-2878b35
luci-mod-system - git-22.339.34970-004e47c
luci-proto-ipv6 - git-21.148.48881-79947af
luci-proto-ppp - git-21.158.38888-88b9d84
luci-proto-wireguard - git-22.327.45657-14403fe
luci-theme-bootstrap - git-22.288.45147-96ec0cd
luci-theme-openwrt-2020 - git-22.287.85459-68ff77c
mkf2fs - 1.14.0-3
mtd - 26
nano - 7.1-1
netifd - 2022-08-25-76d2d41b-1
nftables-json - 1.0.2-2.1
ntp-utils - 4.2.8p15-4
odhcp6c - 2022-08-05-7d21e8d8-18
odhcpd-ipv6only - 2023-01-02-4a673e1c-2
openssl-util - 1.1.1s-1
openwrt-keyring - 2022-03-25-62471e69-3
opkg - 2022-02-24-d038e5b6-1
ppp - 2.4.9.git-2021-01-04-3
ppp-mod-pppoe - 2.4.9.git-2021-01-04-3
procd - 2022-06-01-7a009685-2
procd-seccomp - 2022-06-01-7a009685-2
procd-ujail - 2022-06-01-7a009685-2
qrencode - 4.1.1-1
rpcd - 2022-12-15-7de4820c-1
rpcd-mod-file - 2022-12-15-7de4820c-1
rpcd-mod-iwinfo - 2022-12-15-7de4820c-1
rpcd-mod-luci - 20210614
rpcd-mod-rrdns - 20170710
rrdbackup - 1-1
rrdtool1 - 1.0.50-5
shadow-common - 4.8.1-3
shadow-groupadd - 4.8.1-3
shadow-groupmod - 4.8.1-3
shadow-useradd - 4.8.1-3
shadow-usermod - 4.8.1-3
socat - 1.7.4.1-1
sqm-scripts - 1.5.2-1
swconfig - 12
tc-full - 5.15.0-3
tc-mod-iptables - 5.15.0-3
tcpdump - 4.9.3-4
terminfo - 6.3-2
tinyproxy - 1.10.0-2
ubi-utils - 2.1.4-1
uboot-envtools - 2022.01-36
ubox - 2021-08-03-205defb5-2
ubus - 2022-06-01-2bebf93c-1
ubusd - 2022-06-01-2bebf93c-1
uci - 2021-10-22-f84f49f0-6
uclient-fetch - 2021-05-14-6a6011df-1
ucode - 2022-12-02-46d93c9c-1
ucode-mod-fs - 2022-12-02-46d93c9c-1
ucode-mod-ubus - 2022-12-02-46d93c9c-1
ucode-mod-uci - 2022-12-02-46d93c9c-1
uhttpd - 2022-10-31-23977554-1
uhttpd-mod-ubus - 2022-10-31-23977554-1
urandom-seed - 3
urngd - 2020-01-21-c7f7b6b6-1
usbutils - 014-1
usign - 2020-05-23-f1f65026-1
watchcat - 1-17
wget-ssl - 1.21.3-2
wireguard-tools - 1.0.20210424-3
wireless-regdb - 2022.08.12-1
wpad-basic-wolfssl - 2022-01-16-cff80b4f-15.2
xtables-legacy - 1.8.7-7
zlib - 1.2.11-6
root@LEDE:~#

flygarn12 · February 2, 2023, 7:07am

Is it possible that you as default save this data (that as default are supposed to be saved in ram and lost at reboot) during runtime on the flash memory?

If that’s the case you could simply have worn out the flash memory.

jdwl1o1 · February 2, 2023, 9:23am

Does the oops occur with the default images on openwrt.org? That should indicate whether your imagebuilder created image has anything to do with it.

Or does rolling back to your previous image fix it?

atownlede · February 2, 2023, 11:36am

Unlikely, because the init script only writes the data to flash during orderly shutdown, which (in the absence of crashes) happens on this system only every few months, when I'm installing a new OpenWrt release.

atownlede · February 2, 2023, 11:41am

I don't know if the oops would happen on a stock image, nor do I know if it happens on a roll-back to previous image.
This is my home's primary router, I don't plan on frequent outages and I'm not willing to go back and forth between releases/images.

The question I'm wanting to explore is: what additional data are available about the kernel oops, and how do we get those data?

I gathered /sys/fs/pstore/dmesg-ramoops-0 (and 1), but as we see those reports don't have symbols in the kernel stack traces.
Is there additional crash data somewhere that has a stack trace with symbols? Is there a tool that can analyze the crash data to generate a better stack trace?
Is there something I need to include in image-builder images to capture stack traces in the kernel oops reports?

flygarn12 · February 2, 2023, 12:24pm

Have you searched the forum?
And google?

We don't even know if it is a local build fault for you or a general fault which I doubt since you are pretty alone with the error as of this time.

Faultfinding is only a small part log reading and humongous big part removing things and trial and error and some experience.
You also need to do something yourself for us to find out. Like building a new image and reinstall it to see any changes appears. Or install something else older or newer to see if the problem is logic or hardware.

flygarn12 · February 2, 2023, 12:27pm

Openwrt doesn’t really have a meaningful orderly shutdown function. Other than the classic orderly power off maneuver.

jdwl1o1 · February 2, 2023, 8:12pm

Ok, so how are you using it now if you’re having kernel panics?

atownlede · February 2, 2023, 11:19pm

Sorry, should have said "orderly reboot", where the /etc/init.d/ scripts get to run

atownlede · February 2, 2023, 11:20pm

I've had just the one kernel panic. If it's not happening regularly, I can live with it. I'm hoping to gather diagnostics to at least get a kernel stack trace with symbols to have some idea of what was going on when it crashed.

atownlede · February 2, 2023, 11:29pm

Yes, no hits on anything related to why kernel oops traces would not have symbols on them.
That to me is step #1: How do I diagnose why I didn't have kernel symbols, and then get a build with proper symbols in place so if this recurs, I get something useful in the oops output.

Here's the steps I used with image-builder to construct the image. Should these steps generate a 22.03.3 system with symbols in place that will show during an oops kernel stack backtrace?

If not, then what did I miss to get this capability available in an image-builder system?

    mkdir -p packages
    rm -f packages/*
    cp /media/sf_nbu/rrdbackup_1-1_all.ipk packages
    # that is the init script that copies statistics data to flash during orderly reboot, and restores to the working copy during boot

PACKAGES_BASE="\
 6in4\
 avahi-utils\
 avahi-daemon-service-http\
 avahi-daemon-service-ssh\
 bind-dig\
 collectd-mod-csv\
 collectd-mod-lua\
 collectd-mod-ping\
 collectd-mod-rrdtool\
 collectd-mod-write-graphite\
 diffutils\
 kmod-nft-netdev\
 iperf3\
 irqbalance\
 luci-app-ddns\
 luci-app-sqm\
 luci-app-statistics\
 luci-app-tinyproxy \
 luci-app-uhttpd\
 luci-app-watchcat\
 luci-app-wireguard\
 luci-proto-ipv6\
 luci-proto-wireguard\
 luci-theme-bootstrap\
 luci-theme-openwrt-2020\
 luci\
 nano\
 ntp-utils\
 openssl-util\
 rrdbackup\
 qrencode\
 shadow-groupadd\
 shadow-groupmod\
 shadow-useradd\
 shadow-usermod\
 socat\
 tc-full\
 tcpdump\
 tinyproxy\
 watchcat\
 wget-ssl\
 wireguard-tools\
"

PACKAGES_NETGEAR="${PACKAGES_BASE} \
 bind-rndc\
 bind-server-filter-aaaa\
 bind-server\
 block-mount\
 e2fsprogs\
 f2fs-tools\
 gdisk\
 kmod-fs-ext4\
 kmod-fs-f2fs\
 kmod-usb-storage-uas\
 kmod-usb-storage\
 usbutils\
"

DISABLED_SERVICES_HOME="sqm named"
PROFILE=netgear_r7800
IMAGE_NAME=jtk
time make manifest PROFILE=$PROFILE PACKAGES="$PACKAGES_NETGEAR" DISABLED_SERVICES="$DISABLED_SERVICES_HOME" |&tee MANIFEST.log

IMAGE_SUFFIX_HOME=-home
time make image PROFILE=$PROFILE EXTRA_IMAGE_NAME=$IMAGE_NAME$IMAGE_SUFFIX_HOME PACKAGES="$PACKAGES_NETGEAR" DISABLED_SERVICES="$DISABLED_SERVICES_HOME" |& tee BUILD.home.log

Bill · February 2, 2023, 11:47pm

I had to look this up, and compare to you oops printouts and now I have learned two things.

Forgive my bookmark post