Aarch64 linux: Seeing kernel panic with openwrt 18.06 after upgrade glibc to 2.32

Hi I am using openwrt 18.06 for my box running aarch64 linux and recently updated the glibc from 2.30 to 2.32 version.
Sine then, I am seeing a random reboot of my box following a kernel panic. Unable to find any recent matching signature or fix online. There are several spi related errors and kernel panic issues fixed in linux but none of them recent. Crash seems to be in dspi_transfer_one_message() and seeing these messages on console prior to crash
223.164543] spi_master spi3: failed to transfer one message from queue

Is this a known issue in openwrt?
Thank you,
Charu
Log:

[  235.164532] spi_master spi3: failed to transfer one message from queue

[  235.164551] Unable to handle kernel NULL pointer dereference at virtual

address 00000008

[  235.164553] pgd = ffff000008f77000

[  235.164560] [00000008] *pgd=00000000bfdfe003, *pud=00000000bfdfd003,

*pmd=0000000000000000

[  235.164564] Internal error: Oops: 96000006 [#1] SMP

[  235.164638] Modules linked in: rpd_mdio_kmod(O) nl_irq_pfail(O)

nl_irq_wdog(O) nl_irq_ts_lo(O) nl_irq_ts_hi(O) nl_irq_1pps(O) nl_irq(O)

rpd_watchdog_sys(O) rpd_fpga_dnld(O) rpd_cpld_reg(O) i2c_mux_psio(O)

spi_mux_rpd(O) nf_conntrack_ipv6 lz4 iptable_nat ipt_REJECT ipt_MASQUERADE

xt_time xt_tcpudp xt_state xt_policy xt_nat xt_multiport xt_mark xt_mac

xt_limit xt_esp xt_conntrack xt_comment xt_TCPMSS xt_REDIRECT xt_LOG xt_CT

nf_reject_ipv4 nf_nat_redirect nf_nat_masquerade_ipv4 nf_conntrack_ipv4

nf_nat_ipv4 nf_nat_ftp nf_nat nf_log_ipv4 nf_defrag_ipv6 nf_defrag_ipv4

nf_conntrack_ftp lzo lz4_compress iptable_mangle iptable_filter ipt_ah

ip_tables crc_ccitt ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 nf_log_common

ip6table_mangle ip6table_filter ip6_tables x_tables ah6 ipcomp xfrm4_tunnel

ah4 zram zsmalloc

[  235.164641]  rpd_cpld_dev(O)

[  235.164647] CPU: 0 PID: 980 Comm: spi0 Tainted: G           O    4.9.140 #0

[  235.164649] Hardware name: LS1043A RDB Board (DT)

[  235.164652] task: ffff80003956a300 task.stack: ffff800021ce8000

[  235.164662] PC is at dspi_transfer_one_message+0xac/0x310

[  235.164665] LR is at dspi_transfer_one_message+0x38/0x310

[  235.164668] pc : [<ffff0000085e9584>] lr : [<ffff0000085e9510>] pstate:

60000145

[  235.164669] sp : ffff800021cebcd0

[  235.164673] x29: ffff800021cebcd0 x28: 0000000000000000 

[  235.164677] x27: 0000000000000001 x26: ffff000008b2b000 

[  235.164681] x25: 000000000000000a x24: ffff800021cc4800 

[  235.164684] x23: 0000000000000001 x22: ffff800039a8f000 

[  235.164688] x21: ffff80002f2abd48 x20: ffff800039434600 

[  235.164691] x19: ffff800021cc4dd0 x18: 00000000000007e3 

[  235.164695] x17: 0000ffffade2bf40 x16: ffff0000081c1b18 

[  235.164699] x15: 00000000018bfc54 x14: 0000000000a680e5 

[  235.164703] x13: 0000000000eea7c5 x12: 0000000000000000 

[  235.164706] x11: 00000000fa83b2da x10: 0000000000000820 

[  235.164710] x9 : ffff800021cebd40 x8 : ffff80003956ab80 

[  235.164714] x7 : ffff80003956a440 x6 : 000000362bb4a008 

[  235.164717] x5 : 0000000000000000 x4 : 0000000000000000 

[  235.164720] x3 : ffff000008587888 x2 : 0000000000001996 

[  235.164724] x1 : ffff80002f2abd48 x0 : 0000000000000000 

[  235.164725] 

[  235.164728] Process spi0 (pid: 980, stack limit = 0xffff800021ce8000)

[  235.164731] Stack: (0xffff800021cebcd0 to 0xffff800021cec000)

[  235.164735] bcc0:                                   ffff800021cebd70

ffff0000085e5e9c

[  235.164739] bce0: ffff800021cc4800 ffff800021cc4c18 ffff80002f2abd48

ffff80002f2abd88

[  235.164743] bd00: ffff800021cc4b28 ffff000008ebb000 ffff800021cc4bb8

ffff000008ebb380

[  235.164747] bd20: 0000000000000000 0000000000000000 ffff800021cebd60

ffff000008b2b718

[  235.164752] bd40: 6ab000003fd89600 ffff800039864180 ffff800021cebd70

ffff0000085e5d08

[  235.164756] bd60: ffff800021cc4800 ffff0000085e5cdc ffff800021cebdc0

ffff0000085e60bc

[  235.164760] bd80: ffff800021cc4bf0 ffff80003956a300 ffff800021cc4bb0

ffff800021cc4bb4

[  235.164764] bda0: 0000000000000001 ffff000008ebb000 ffff800021cebe10

ffff00000897a144

[  235.164768] bdc0: ffff800021cebdd0 ffff0000080c21bc ffff800021cebe20

ffff0000080c20f0

[  235.164773] bde0: ffff800021ca11c0 ffff80003956a300 ffff000008adb350

ffff800021cc4bb0

[  235.164777] be00: ffff0000080c2120 0000000000000000 0000000000000000

0000000000000000

[  235.164781] be20: 0000000000000000 ffff000008083990 ffff0000080c2008

ffff800021ca11c0

[  235.164784] be40: 0000000000000000 0000000000000000 0000000000000000

0000000000000000

[  235.164788] be60: 0000000000000000 0000000000000000 ffff800021cebea0

ffff800021cc4bb0

[  235.164793] be80: ffff000000000000 ffff800000000000 ffff800021cebe90

ffff800021cebe90

[  235.164797] bea0: 0000000000000000 ffff000000000000 ffff800021cebeb0

ffff800021cebeb0

[  235.164800] bec0: 0000000000000000 0000000000000000 0000000000000000

0000000000000000

[  235.164804] bee0: 0000000000000000 0000000000000000 0000000000000000

0000000000000000

[  235.164807] bf00: 0000000000000000 0000000000000000 0000000000000000

0000000000000000

[  235.164811] bf20: 0000000000000000 0000000000000000 0000000000000000

0000000000000000

[  235.164815] bf40: 0000000000000000 0000000000000000 0000000000000000

0000000000000000

[  235.164818] bf60: 0000000000000000 0000000000000000 0000000000000000

0000000000000000

[  235.164822] bf80: 0000000000000000 0000000000000000 0000000000000000

0000000000000000

[  235.164826] bfa0: 0000000000000000 0000000000000000 0000000000000000

0000000000000000

[  235.164829] bfc0: 0000000000000000 0000000000000005 0000000000000000

0000000000000000

[  235.164833] bfe0: 0000000000000000 0000000000000000 0000000000000000

0000000000000000

[  235.164835] Call trace:

[  235.164839] Exception stack(0xffff800021cebb00 to 0xffff800021cebc30)

[  235.164843] bb00: ffff800021cc4dd0 0000ffffffffffff ffff800021cebcd0

ffff0000085e9584

[  235.164847] bb20: ffff800021cebb50 ffff0000080cb5a4 ffff800039452680

ffff800039452c84

[  235.164851] bb40: 0000000000000000 00000000000001c0 ffff800021cebbb0

ffff0000080cb700

[  235.164855] bb60: ffff80002f7fd5e0 ffff80002f2abb58 0000000000000000

0000000000000000

[  235.164859] bb80: 0000000000000001 0000000000000001 0000000000000000

ffff000008ebb380

[  235.164863] bba0: 0000000000000000 ffff80002f2abd48 0000000000001996

ffff000008587888

[  235.164867] bbc0: 0000000000000000 0000000000000000 000000362bb4a008

ffff80003956a440

[  235.164871] bbe0: ffff80003956ab80 ffff800021cebd40 0000000000000820

00000000fa83b2da

[  235.164875] bc00: 0000000000000000 0000000000eea7c5 0000000000a680e5

00000000018bfc54

[  235.164878] bc20: ffff0000081c1b18 0000ffffade2bf40

[  235.164883] [<ffff0000085e9584>] dspi_transfer_one_message+0xac/0x310

[  235.164887] [<ffff0000085e5e9c>] __spi_pump_messages+0x2f4/0x500

[  235.164890] [<ffff0000085e60bc>] spi_pump_messages+0x14/0x20

[  235.164897] [<ffff0000080c21bc>] kthread_worker_fn+0x9c/0x180

[  235.164901] [<ffff0000080c20f0>] kthread+0xe8/0xf0

[  235.164906] [<ffff000008083990>] ret_from_fork+0x10/0x40

[  235.164911] Code: 54000060 39412281 36000041 b9006e77 (79401001) 

[  235.164916] ---[ end trace 308a28a88f5f94b8 ]---

[  235.164920] Kernel panic - not syncing: Fatal exception

[  235.164922] SMP: stopping secondary CPUs

[  235.171075] Kernel Offset: disabled

Define 'upgrade' please. Through opkg? Your C library is an essential piece of code and opkg does not have the bells and whistles a full-fledged normal Linux package manager has to handle such system upgrades, like your C library and the kernel.

OpenWrt cannot be upgraded the way you can a regular Linux distribution. You should flash a new image.

1 Like

19.xx should be flashed.

Thank you for the quick responses - apologies for the confusion but I am not upgrading openwrt. I am still using 18.06 version. I updated only my glibc package from 2.30 to 2.32. Post that, I see a kernel panic crash on bootup occassionally.
Update glibc procedure - download glib2.32 stable release tar from gnu/ftp glibc site. Build the source code under openwrt/toolchain and then build the linux image.
The kernel panic is not seen after several retries when glibc was downgraded from 2.32 to 2.30 in toolchain. Hence wondering if there are any known issues with openwrt 18.06+ gllibc 2.32. Are these compatible and if I can find a compatibility matrix for the libc pkg for this openwrt version?
this is the stack trace on reboot (not seen every time though, maybe 3/10 times)

[  335.683493] spi_master spi3: failed to transfer one message from queue
[  335.690230] list_del corruption. prev->next should be ffff800031d37d88, but was ffff800031d37e20
[  335.699064] ------------[ cut here ]------------
[  335.703695] WARNING: CPU: 3 PID: 982 at lib/list_debug.c:59 __list_del_entry+0xa4/0xc8
[  335.711616] Modules linked in: rpd_mdio_kmod(O) nl_irq_pfail(O) nl_irq_wdog(O) nl_irq_ts_lo(O) nl_irq_ts_hi(O) nl_irq_1pps(O) nl_irq(O) rpd_watchdog_sys(O) rpd_fpga_dnld(O) rpd_cpld_reg(O) i2c_mux_psio(O) spi_mux_rpd(O) nf_conntrack_ipv6 lz4 iptable_nat ipt_REJECT ipt_MASQUERADE xt_time xt_tcpudp xt_state xt_policy xt_nat xt_multiport xt_mark xt_mac xt_limit xt_esp xt_conntrack xt_comment xt_TCPMSS xt_REDIRECT xt_LOG xt_CT nf_reject_ipv4 nf_nat_redirect nf_nat_masquerade_ipv4 nf_conntrack_ipv4 nf_nat_ipv4 nf_nat_ftp nf_nat nf_log_ipv4 nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_ftp lzo lz4_compress iptable_mangle iptable_filter ipt_ah ip_tables crc_ccitt ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables x_tables ah6 ipcomp xfrm4_tunnel ah4 zram zsmalloc rpd_cpld_dev(O)

[  335.784136] CPU: 3 PID: 982 Comm: spi0 Tainted: G           O    4.9.140 #0
[  335.791098] Hardware name: LS1043A RDB Board (DT)
[  335.795802] task: ffff8000394921c0 task.stack: ffff800021d10000
[  335.801728] PC is at __list_del_entry+0xa4/0xc8
[  335.806262] LR is at __list_del_entry+0xa4/0xc8
[  335.810793] pc : [<ffff000008453df4>] lr : [<ffff000008453df4>] pstate: 400001c5
[  335.818190] sp : ffff800021d13d60
[  335.821500] x29: ffff800021d13d60 x28: 0000000000000000 
[  335.826815] x27: 0000000000000000 x26: ffff000008ebb380 
[  335.832130] x25: ffff800021cda3b8 x24: ffff000008ebb000 
[  335.837446] x23: ffff800021cda328 x22: ffff800031d37d88 
[  335.842762] x21: 0000000000000140 x20: ffff800021cda418 
[  335.848078] x19: ffff800021cda000 x18: 0000000000000010 
[  335.853393] x17: 0000ffff8268e2c0 x16: ffff000008113b70 
[  335.858707] x15: ffffffffffffffff x14: 6666666620736177 
[  335.864022] x13: 20747562202c3838 x12: 6437336431333030 
[  335.869335] x11: 3038666666662065 x10: 00000000000001ec 
[  335.874650] x9 : 0000000000000064 x8 : 657270202e6e6f69 
[  335.879966] x7 : 74707572726f6320 x6 : ffff000008539e10 
[  335.885281] x5 : ffff80003fdaea50 x4 : 0000000000000001 
[  335.890597] x3 : 0000000000000007 x2 : 0000000000000006 
[  335.895911]