Background: previously I was dealing with a level 2 cache issue on an Armada 370 (cortex A9) subtarget:
The problem was the idle power save mechanism which was leaving the L2 cache invalidated. After diabling the power save the L2 cache came to life. So this issue was solved.
But now with the L2 cache enabled i noticed sporadic kernel panics (after 1 day or more). So I took the device for making more tests.
I was able to reproduce the error under certain conditions:
- I installed an USB Gbit ethernet adapter (RTL8153)
- I made a big transfer test to /dev/null
while true; do wget -O- http://192.168.1.7:8000/bigfile.bin > /dev/null;done
On the first or second transfer I got allways a kernel panic:
[ 766.509561] Unhandled prefetch abort: unknown 25 (0x409) at 0xc071b6a0
[ 766.516135] Internal error: : 409 [#1] SMP ARM
[ 766.520599] Modules linked in: ksmbd iptable_nat xt_state xt_nat xt_conntrack xt_REDIRECT xt_MASQUERADE xt_FLOWOFFLOAD nf_nat nf_flow_table_hw nf_flow_table nf_conntrack_rtcache nf_conntrack ipt_REJECT asix xt_time xt_tcpudp xt_multiport xt_mark xt_mac xt_limit xt_comment xt_TCPMSS xt_LOG usbnet r8152 nf_reject_ipv4 nf_log_ipv4 nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle iptable_filter ip_tables exfat crc_ccitt dump_cp15_regs gpio_fan drivetemp ledtrig_heartbeat nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 vfat fat raid10 raid1 raid0 linear md_mod nls_utf8 nls_iso8859_1 nls_cp437 sha512_generic sha256_generic libsha256 seqiv jitterentropy_rng drbg md5 md4 hmac ghash_generic gf128mul gcm ecb ctr cmac ccm arc4 rtc_rs5c372 linkstation_poweroff gpio_button_hotplug xfs libcrc32c mii [last unloaded: ksmbd]
[ 766.597207] CPU: 0 PID: 4304 Comm: wget Not tainted 5.4.85 #0
[ 766.602977] Hardware name: Marvell Armada 370/XP (Device Tree)
[ 766.608848] PC is at __copy_to_user_std+0x1f0/0x378
[ 766.613748] LR is at 0xf36d7cbf
[ 766.616902] pc : [<c071b6a0>] lr : [<f36d7cbf>] psr: 60000013
[ 766.623195] sp : c1335d0c ip : 00000000 fp : dfba3d20
[ 766.628441] r10: 000019b1 r9 : c1309e22 r8 : c1335ef0
[ 766.633688] r7 : 00000471 r6 : c1335ea8 r5 : 00000471 r4 : 49222b93
[ 766.640244] r3 : bfc8e9ea r2 : ffffff91 r1 : c1309e24 r0 : 0034d480
[ 766.646801] Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user
[ 766.653968] Control: 10c5387d Table: 1d954019 DAC: 00000055
[ 766.659742] Process wget (pid: 4304, stack limit = 0xc3a07c61)
[ 766.665600] Stack: (0xc1335d0c to 0xc1336000)
[ 766.669979] 5d00: 0034d010 00000471 00000000 00000051 c03f53f8
[ 766.678196] 5d20: 00000471 c03f93c4 c13099b1 c023f664 00000000 c12e2de4 00000b70 c1a945e0
[ 766.686412] 5d40: 00001c48 00000471 000017d7 00000009 00001b11 000019b1 dfba3d20 c05d1640
[ 766.694629] 5d60: dce331a0 00000000 00002000 c0662fdc de82a000 de82a000 c1335ea8 00000000
[ 766.702845] 5d80: 000017d7 c0947974 df66f148 de82a000 000017d7 dce33100 00001b11 dce3350c
[ 766.711061] 5da0: dce331a0 00000000 00000000 c05d17d4 00000000 c05d17dc 00000000 00000000
[ 766.719278] 5dc0: dce33100 c065241c e09fbd80 c0932f8c 00000001 1dbf41a0 de9a1d88 00000001
[ 766.727494] 5de0: 00002000 dce33604 00000000 c1335ea0 c1334000 c094f880 c0b31dd2 c094f6fc
[ 766.735710] 5e00: c094f854 00000000 00000000 be97d458 00000004 00000000 00000004 c0259608
[ 766.743927] 5e20: e0a86000 7fffffff c1335e40 c1335e44 c1335e48 c1335e4c c1335e50 c1335e54
[ 766.752143] 5e40: 00000008 00000000 00000000 00000008 00000000 00000000 00002000 c1335ea0
[ 766.760359] 5e60: c06521e8 dcdea840 df1b7200 c1335f80 00002000 00000003 00000000 c0683fa4
[ 766.768576] 5e80: 00000000 c1335e8c 00000000 00000000 c1335ef8 c0683f4c dcdea840 c05bf1d4
[ 766.776791] 5ea0: 00000000 00000000 00000004 00000000 00002000 c1335ef0 00000001 ffffffff
[ 766.785008] 5ec0: 00000000 00000000 00000000 c1335f10 dcdea840 dcdea840 00000000 00000000
[ 766.793224] 5ee0: 00000000 c0244238 00002000 be97d420 0034d010 00002000 00000004 00000000
[ 766.801440] 5f00: 00002000 c1335ef0 00000001 ffffffff dcdea840 00000000 00000000 00000000
[ 766.809656] 5f20: 00000000 00000000 00000000 00000000 00000000 00000000 389fcc2a 0034d010
[ 766.817872] 5f40: 00000001 00002000 c1335f80 dcdea840 00000000 c0244344 c1335f78 0000008e
[ 766.826088] 5f60: 00000000 dcdea840 dcdea840 00000000 00000000 0034d010 00002000 c0244624
[ 766.834304] 5f80: 00000000 00000000 b6f8c120 00000000 00000000 00000000 00000003 c0101204
[ 766.842520] 5fa0: c1334000 c0101000 00000000 00000000 00000003 0034d010 00002000 00000000
[ 766.850736] 5fc0: 00000000 00000000 00000000 00000003 00000000 00000000 0710e000 00000000
[ 766.858952] 5fe0: be97d4b8 be97d4a8 b6f6c05f b6f6bd16 60000030 00000003 00000000 00000000
[ 766.867187] [<c071b6a0>] (__copy_to_user_std) from [<c03f53f8>] (copyout+0x4c/0x60)
[ 766.874885] [<c03f53f8>] (copyout) from [<c03f93c4>] (_copy_to_iter+0x90/0x41c)
[ 766.882239] [<c03f93c4>] (_copy_to_iter) from [<c05d1640>] (__skb_datagram_iter+0x170/0x2b0)
[ 766.890718] [<c05d1640>] (__skb_datagram_iter) from [<c05d17d4>] (skb_copy_datagram_iter+0x24/0x2c)
[ 766.899818] [<c05d17d4>] (skb_copy_datagram_iter) from [<c065241c>] (tcp_recvmsg+0x234/0xa88)
[ 766.908397] [<c065241c>] (tcp_recvmsg) from [<c0683fa4>] (inet_recvmsg+0x58/0xd4)
[ 766.915918] [<c0683fa4>] (inet_recvmsg) from [<c05bf1d4>] (sock_read_iter+0x94/0xd0)
[ 766.923710] [<c05bf1d4>] (sock_read_iter) from [<c0244238>] (__vfs_read+0x11c/0x19c)
[ 766.931492] [<c0244238>] (__vfs_read) from [<c0244344>] (vfs_read+0x8c/0x114)
[ 766.938662] [<c0244344>] (vfs_read) from [<c0244624>] (ksys_read+0x48/0xb4)
[ 766.945659] [<c0244624>] (ksys_read) from [<c0101000>] (ret_fast_syscall+0x0/0x54)
[ 766.953260] Exception stack(0xc1335fa8 to 0xc1335ff0)
[ 766.958335] 5fa0: 00000000 00000000 00000003 0034d010 00002000 00000000
[ 766.966551] 5fc0: 00000000 00000000 00000000 00000003 00000000 00000000 0710e000 00000000
[ 766.974765] 5fe0: be97d4b8 be97d4a8 b6f6c05f b6f6bd16
[ 766.979844] Code: e491e004 e25cc004 e1833c0e e4803004 (cafffff9)
[ 766.985965] ---[ end trace dd34b70afa368f98 ]---
[ 766.990604] Kernel panic - not syncing: Fatal exception
[ 766.995858] Rebooting in 3 seconds..
With the L2 cache disabled this never happens. Also almost always the error occur on the same address or slightly different: 0xc071b6a0
I also enabled the I/O coherency which wasn't enabled as default in this particular SoC. Apparently this feature has no effect, with or without the I/O coherency the kernel panic also occurs.
I remembered that I made tests in the past with the Thumb2 mode with good results, so I compiled the kernel with this feature:
And voila the kernel panic dissapeared in kernel Thumb2 mode with the cache enabled. While writing this post it continues with the infinite download loop without any kernel panic.
root@LinkStation:~# wget -O- http://192.168.1.7:8000/bigfile.bin > /dev/null
--2020-12-29 19:40:28-- http://192.168.1.7:8000/bigfile.bin
Connecting to 192.168.1.7:8000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2602501255 (2.4G) [application/octet-stream]
Saving to: 'STDOUT'
- 100%[===================================>] 2.42G 64.5MB/s in 39s
2020-12-29 19:41:07 (64.0 MB/s) - written to stdout [2602501255/2602501255]
root@LinkStation:~# while true; do wget -O- http://192.168.1.7:8000/bigfile.bin > /dev/null;done
--2020-12-29 19:41:19-- http://192.168.1.7:8000/bigfile.bin
Connecting to 192.168.1.7:8000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2602501255 (2.4G) [application/octet-stream]
Saving to: 'STDOUT'
- 100%[===================================>] 2.42G 62.0MB/s in 40s
2020-12-29 19:41:59 (61.8 MB/s) - written to stdout [2602501255/2602501255]
--2020-12-29 19:41:59-- http://192.168.1.7:8000/bigfile.bin
Connecting to 192.168.1.7:8000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2602501255 (2.4G) [application/octet-stream]
Saving to: 'STDOUT'
- 100%[===================================>] 2.42G 62.2MB/s in 40s
2020-12-29 19:42:39 (62.6 MB/s) - written to stdout [2602501255/2602501255]
--2020-12-29 19:42:39-- http://192.168.1.7:8000/bigfile.bin
Connecting to 192.168.1.7:8000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2602501255 (2.4G) [application/octet-stream]
Saving to: 'STDOUT'
- 100%[===================================>] 2.42G 65.6MB/s in 39s
2020-12-29 19:43:18 (64.3 MB/s) - written to stdout [2602501255/2602501255]
--2020-12-29 19:43:18-- http://192.168.1.7:8000/bigfile.bin
Connecting to 192.168.1.7:8000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2602501255 (2.4G) [application/octet-stream]
Saving to: 'STDOUT'
- 100%[===================================>] 2.42G 64.9MB/s in 39s
2020-12-29 19:43:56 (64.1 MB/s) - written to stdout [2602501255/2602501255]
--2020-12-29 19:43:56-- http://192.168.1.7:8000/bigfile.bin
Connecting to 192.168.1.7:8000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2602501255 (2.4G) [application/octet-stream]
Saving to: 'STDOUT'
- 100%[===================================>] 2.42G 63.1MB/s in 40s
2020-12-29 19:44:36 (62.5 MB/s) - written to stdout [2602501255/2602501255]
--2020-12-29 19:44:36-- http://192.168.1.7:8000/bigfile.bin
Connecting to 192.168.1.7:8000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2602501255 (2.4G) [application/octet-stream]
Saving to: 'STDOUT'
- 100%[===================================>] 2.42G 64.7MB/s in 39s
2020-12-29 19:45:15 (64.0 MB/s) - written to stdout [2602501255/2602501255]
--2020-12-29 19:45:15-- http://192.168.1.7:8000/bigfile.bin
Connecting to 192.168.1.7:8000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2602501255 (2.4G) [application/octet-stream]
Saving to: 'STDOUT'
- 100%[===================================>] 2.42G 63.0MB/s in 40s
2020-12-29 19:45:55 (62.0 MB/s) - written to stdout [2602501255/2602501255]
--2020-12-29 19:45:55-- http://192.168.1.7:8000/bigfile.bin
Connecting to 192.168.1.7:8000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2602501255 (2.4G) [application/octet-stream]
Saving to: 'STDOUT'
- 100%[===================================>] 2.42G 62.0MB/s in 40s
2020-12-29 19:46:35 (61.5 MB/s) - written to stdout [2602501255/2602501255]
--2020-12-29 19:46:35-- http://192.168.1.7:8000/bigfile.bin
Connecting to 192.168.1.7:8000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2602501255 (2.4G) [application/octet-stream]
Saving to: 'STDOUT'
- 100%[===================================>] 2.42G 62.7MB/s in 40s
2020-12-29 19:47:15 (62.3 MB/s) - written to stdout [2602501255/2602501255]
--2020-12-29 19:47:15-- http://192.168.1.7:8000/bigfile.bin
Connecting to 192.168.1.7:8000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2602501255 (2.4G) [application/octet-stream]
Saving to: 'STDOUT'
- 100%[===================================>] 2.42G 64.8MB/s in 39s
2020-12-29 19:47:54 (64.1 MB/s) - written to stdout [2602501255/2602501255]
--2020-12-29 19:47:54-- http://192.168.1.7:8000/bigfile.bin
Connecting to 192.168.1.7:8000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2602501255 (2.4G) [application/octet-stream]
Saving to: 'STDOUT'
- 100%[===================================>] 2.42G 65.2MB/s in 39s
2020-12-29 19:48:32 (64.4 MB/s) - written to stdout [2602501255/2602501255]
--2020-12-29 19:48:32-- http://192.168.1.7:8000/bigfile.bin
Connecting to 192.168.1.7:8000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2602501255 (2.4G) [application/octet-stream]
Saving to: 'STDOUT'
- 100%[===================================>] 2.42G 62.1MB/s in 40s
2020-12-29 19:49:13 (61.7 MB/s) - written to stdout [2602501255/2602501255]
--2020-12-29 19:49:13-- http://192.168.1.7:8000/bigfile.bin
Connecting to 192.168.1.7:8000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2602501255 (2.4G) [application/octet-stream]
Saving to: 'STDOUT'
- 100%[===================================>] 2.42G 62.3MB/s in 40s
2020-12-29 19:49:53 (62.0 MB/s) - written to stdout [2602501255/2602501255]
--2020-12-29 19:49:53-- http://192.168.1.7:8000/bigfile.bin
Connecting to 192.168.1.7:8000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2602501255 (2.4G) [application/octet-stream]
Saving to: 'STDOUT'
- 100%[===================================>] 2.42G 65.8MB/s in 39s
2020-12-29 19:50:31 (64.5 MB/s) - written to stdout [2602501255/2602501255]
--2020-12-29 19:50:31-- http://192.168.1.7:8000/bigfile.bin
Connecting to 192.168.1.7:8000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2602501255 (2.4G) [application/octet-stream]
Saving to: 'STDOUT'
- 100%[===================================>] 2.42G 65.2MB/s in 39s
2020-12-29 19:51:10 (64.3 MB/s) - written to stdout [2602501255/2602501255]
--2020-12-29 19:51:10-- http://192.168.1.7:8000/bigfile.bin
Connecting to 192.168.1.7:8000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2602501255 (2.4G) [application/octet-stream]
Saving to: 'STDOUT'
- 100%[===================================>] 2.42G 61.5MB/s in 41s
2020-12-29 19:51:50 (61.1 MB/s) - written to stdout [2602501255/2602501255]
--2020-12-29 19:51:50-- http://192.168.1.7:8000/bigfile.bin
Connecting to 192.168.1.7:8000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2602501255 (2.4G) [application/octet-stream]
Saving to: 'STDOUT'
- 100%[===================================>] 2.42G 62.2MB/s in 40s
2020-12-29 19:52:31 (61.8 MB/s) - written to stdout [2602501255/2602501255]
--2020-12-29 19:52:31-- http://192.168.1.7:8000/bigfile.bin
Connecting to 192.168.1.7:8000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2602501255 (2.4G) [application/octet-stream]
Saving to: 'STDOUT'
- 100%[===================================>] 2.42G 65.0MB/s in 39s
2020-12-29 19:53:09 (64.5 MB/s) - written to stdout [2602501255/2602501255]
--2020-12-29 19:53:09-- http://192.168.1.7:8000/bigfile.bin
Connecting to 192.168.1.7:8000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2602501255 (2.4G) [application/octet-stream]
Saving to: 'STDOUT'
- 100%[===================================>] 2.42G 61.6MB/s in 40s
2020-12-29 19:53:50 (61.4 MB/s) - written to stdout [2602501255/2602501255]
--2020-12-29 19:53:50-- http://192.168.1.7:8000/bigfile.bin
Connecting to 192.168.1.7:8000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2602501255 (2.4G) [application/octet-stream]
Saving to: 'STDOUT'
.....
--- output deleted ----
It's also worth mentioning that the board hangs at some point when loading the kernel if compiled in Thumb2 mode with the L2 cache disabled. Therefore the Thumb2 mode requires a working L2 cache.
Not sure if this can be considered a workaround or a fix. But the kernel panic shouldn't never occur.
If someone also experiments a similar issue in his/her device please post here. (I don't discard a hardware problem with my board).
Regards