Hey everyone,
My R3P seemed to be going into a crash loop a couple of times a week and losing internet connection. I mounted an external drive to get the logs and was able to get this:
Feb 10 04:14:20 rtr-1 kernel: [ 7662.000809] INFO: rcu_sched self-detected stall on CPU
Feb 10 04:14:20 rtr-1 kernel: [ 7662.006072] 3-...: (78019 ticks this GP) idle=372/140000000000001/0 softirq=519402/519402 fqs=36371
Feb 10 04:14:20 rtr-1 kernel: [ 7662.015291] (t=78021 jiffies g=260316 c=260315 q=8197)
Feb 10 04:14:20 rtr-1 kernel: [ 7662.020680] NMI backtrace for cpu 3
Feb 10 04:14:20 rtr-1 kernel: [ 7662.024189] CPU: 3 PID: 19587 Comm: kworker/3:0 Not tainted 4.14.215 #0
Feb 10 04:14:20 rtr-1 kernel: [ 7662.030806] Workqueue: events_power_efficient 0x8f0a4cac [nf_flow_table@8f0a4000+0x3370]
Feb 10 04:14:20 rtr-1 kernel: [ 7662.038862] Stack : 00000000 00000000 804d8568 8fc0fd24 00000000 00000000 00000000 00000000
Feb 10 04:14:20 rtr-1 kernel: [ 7662.047210] 00000000 00000000 00000000 00000000 00000000 00000001 8fc0fce0 1cc28237
Feb 10 04:14:20 rtr-1 kernel: [ 7662.055557] 8fc0fd78 00000000 00000000 0000c1f0 00000038 8049e0f8 00000007 00000000
Feb 10 04:14:20 rtr-1 kernel: [ 7662.063904] 00000000 80550000 00007856 00000000 8fc0fcc0 00000000 00000000 00000003
Feb 10 04:14:20 rtr-1 kernel: [ 7662.072251] 80554500 8054c0cc 000000e0 80550000 00000003 802ae4f0 0000000c 806b000c
Feb 10 04:14:20 rtr-1 kernel: [ 7662.080599] ...
Feb 10 04:14:20 rtr-1 kernel: [ 7662.083038] Call Trace:
Feb 10 04:14:20 rtr-1 kernel: [ 7662.083097] [<8049e0f8>] 0x8049e0f8
Feb 10 04:14:20 rtr-1 kernel: [ 7662.089011] [<802ae4f0>] 0x802ae4f0
Feb 10 04:14:20 rtr-1 kernel: [ 7662.092495] [<8000c1a0>] 0x8000c1a0
Feb 10 04:14:20 rtr-1 kernel: [ 7662.095969] [<8000c1a8>] 0x8000c1a8
Feb 10 04:14:20 rtr-1 kernel: [ 7662.099443] [<80486f44>] 0x80486f44
Feb 10 04:14:20 rtr-1 kernel: [ 7662.102916] [<80072ce4>] 0x80072ce4
Feb 10 04:14:20 rtr-1 kernel: [ 7662.106391] [<8048df64>] 0x8048df64
Feb 10 04:14:20 rtr-1 kernel: [ 7662.109864] [<80008f30>] 0x80008f30
Feb 10 04:14:20 rtr-1 kernel: [ 7662.113338] [<80008f30>] 0x80008f30
Feb 10 04:14:20 rtr-1 kernel: [ 7662.116811] [<8048e050>] 0x8048e050
Feb 10 04:14:20 rtr-1 kernel: [ 7662.120286] [<80086678>] 0x80086678
Feb 10 04:14:20 rtr-1 kernel: [ 7662.123772] [<80085ac8>] 0x80085ac8
Feb 10 04:14:20 rtr-1 kernel: [ 7662.127269] [<8f0a0000>] 0x8f0a0000 [x_tables@8f0a0000+0x3290]
Feb 10 04:14:20 rtr-1 kernel: [ 7662.133113] [<8f0a0000>] 0x8f0a0000 [x_tables@8f0a0000+0x3290]
Feb 10 04:14:20 rtr-1 kernel: [ 7662.138919] [<80089238>] 0x80089238
Feb 10 04:14:20 rtr-1 kernel: [ 7662.142394] [<8009a288>] 0x8009a288
Feb 10 04:14:20 rtr-1 kernel: [ 7662.145867] [<80083a2c>] 0x80083a2c
Feb 10 04:14:20 rtr-1 kernel: [ 7662.149342] [<80332ab4>] 0x80332ab4
Feb 10 04:14:20 rtr-1 kernel: [ 7662.152826] [<800793b8>] 0x800793b8
Feb 10 04:14:20 rtr-1 kernel: [ 7662.156302] [<80073670>] 0x80073670
Feb 10 04:14:20 rtr-1 kernel: [ 7662.159776] [<8025c038>] 0x8025c038
Feb 10 04:14:20 rtr-1 kernel: [ 7662.163249] [<8025bee4>] 0x8025bee4
Feb 10 04:14:20 rtr-1 kernel: [ 7662.166724] [<8025c0a4>] 0x8025c0a4
Feb 10 04:14:20 rtr-1 kernel: [ 7662.170199] [<80073670>] 0x80073670
Feb 10 04:14:20 rtr-1 kernel: [ 7662.173673] [<804a4dbc>] 0x804a4dbc
Feb 10 04:14:20 rtr-1 kernel: [ 7662.177148] [<8025b760>] 0x8025b760
Feb 10 04:14:20 rtr-1 kernel: [ 7662.180624] [<80007488>] 0x80007488
Feb 10 04:14:20 rtr-1 kernel: [ 7662.184094]
The logs are filled with messages like these and it seems the router is unable of doing anything else until it is manually restarted. I've caught other reboots in the logs that don't seem related to this.
I found this bug
https://bugs.openwrt.org/index.php?do=details&task_id=1170&order=tasktype&sort=asc
Which seems to fit the bill (nf_flow_table), so I disabled hw offloading on the firewall configs.
I wanted someone more knowledgeable than me with openwrt to help confirm this is actually the case.
Thanks