Ubiquiti EdgeRouter X CPU stalls r4561

Installing and Using OpenWrt
1

I bought an EdgeRouter X to play with and I'm getting a lot of big, scary looking errors prior to the device deteriorating into an unusable state (SSH goes, then uhttpd, etc etc).

(Using LEDE master/snapshot build r4561 from downloads.lede-project.org/snapshots)

This is the error I manged to catch before it fell over last time;

Wed Jul 12 20:14:25 2017 kern.err kernel: [ 88.830000] INFO: rcu_sched detected stalls on CPUs/tasks: Wed Jul 12 20:14:25 2017 kern.err kernel: [ 88.840000] 3-...: (0 ticks this GP) idle=0e6/0/0 softirq=1115/1115 fqs=0 Wed Jul 12 20:14:25 2017 kern.err kernel: [ 88.850000] (detected by 1, t=6004 jiffies, g=119, c=118, q=1951) Wed Jul 12 20:14:25 2017 kern.info kernel: [ 88.860000] Task dump for CPU 3: Wed Jul 12 20:14:25 2017 kern.info kernel: [ 88.870000] swapper/3 R running task 0 0 1 0x00100000 Wed Jul 12 20:14:25 2017 kern.warn kernel: [ 88.880000] Stack : 8fc77fe0 8fc54680 00000000 f73d9003 00000013 00000000 00000013 00000000 Wed Jul 12 20:14:25 2017 kern.warn kernel: [ 88.900000] 804e148c 00000008 8047f8e0 804f0000 00000001 00000001 bfbf8000 00000003 Wed Jul 12 20:14:25 2017 kern.warn kernel: [ 88.920000] 00000000 803e8234 00000000 00000000 8fc76000 804e148c 00000008 800c50a4 Wed Jul 12 20:14:25 2017 kern.warn kernel: [ 88.930000] 804e148c 00000008 8047f8e0 804f0000 804e148c 80062ebc 00000001 00000001 Wed Jul 12 20:14:25 2017 kern.warn kernel: [ 88.950000] 00000001 00000001 00000001 00000001 00000001 00000001 00000001 a020046c Wed Jul 12 20:14:25 2017 kern.warn kernel: [ 88.970000] ... Wed Jul 12 20:14:25 2017 kern.warn kernel: [ 88.970000] Call Trace: Wed Jul 12 20:14:25 2017 kern.warn kernel: [ 88.980000] [<803e7ff8>] __schedule+0x5d4/0x7a4 Wed Jul 12 20:14:25 2017 kern.warn kernel: [ 88.980000] [<803e8234>] schedule+0x6c/0x84 Wed Jul 12 20:14:25 2017 kern.warn kernel: [ 88.990000] [<800c50a4>] quiet_vmstat+0x54/0x80 Wed Jul 12 20:14:25 2017 kern.warn kernel: [ 89.000000] [<80062ebc>] cpu_startup_entry+0xec/0x150 Wed Jul 12 20:14:25 2017 kern.err kernel: [ 89.010000] rcu_sched kthread starved for 6020 jiffies! g119 c118 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x1 Wed Jul 12 20:14:25 2017 kern.info kernel: [ 89.030000] rcu_sched S 0 7 2 0x00100000 Wed Jul 12 20:14:25 2017 kern.warn kernel: [ 89.040000] Stack : 804e15f4 8fc52340 8fc6be50 81234320 8fc6be50 00000000 00000000 8007b4c8 Wed Jul 12 20:14:25 2017 kern.warn kernel: [ 89.060000] 81234320 ffff9612 81234320 8fc6be50 804e0000 00000001 804f0000 804e0000 Wed Jul 12 20:14:25 2017 kern.warn kernel: [ 89.070000] 804e96b0 803e8234 ffff9612 804e0000 8fc6be50 81234320 ffff9612 803eab98 Wed Jul 12 20:14:25 2017 kern.warn kernel: [ 89.090000] 804e15f4 804f0000 804e9580 804e96b0 81236240 00000001 00000000 812343c4 Wed Jul 12 20:14:25 2017 kern.warn kernel: [ 89.110000] ffff9612 8007b9c4 8fc52340 04c00003 804e9580 00000001 804e0000 00000000 Wed Jul 12 20:14:25 2017 kern.warn kernel: [ 89.120000] ... Wed Jul 12 20:14:25 2017 kern.warn kernel: [ 89.130000] Call Trace: Wed Jul 12 20:14:25 2017 kern.warn kernel: [ 89.130000] [<803e7ff8>] __schedule+0x5d4/0x7a4 Wed Jul 12 20:14:25 2017 kern.warn kernel: [ 89.140000] [<803e8234>] schedule+0x6c/0x84 Wed Jul 12 20:14:25 2017 kern.warn kernel: [ 89.150000] [<803eab98>] schedule_timeout+0x160/0x19c Wed Jul 12 20:14:25 2017 kern.warn kernel: [ 89.160000] [<80078f00>] rcu_gp_kthread+0x7f4/0x7fc Wed Jul 12 20:14:25 2017 kern.warn kernel: [ 89.170000] [<80044be8>] kthread+0xd8/0xec Wed Jul 12 20:14:25 2017 kern.warn kernel: [ 89.180000] [<8000a318>] ret_from_kernel_thread+0x14/0x1c Wed Jul 12 20:15:30 2017 kern.err kernel: [ 154.250000] INFO: rcu_sched detected stalls on CPUs/tasks: Wed Jul 12 20:15:30 2017 kern.err kernel: [ 154.260000] 3-...: (58 GPs behind) idle=5ce/0/0 softirq=1115/1115 fqs=1 Wed Jul 12 20:15:30 2017 kern.err kernel: [ 154.270000] (detected by 1, t=6004 jiffies, g=177, c=176, q=1285) Wed Jul 12 20:15:30 2017 kern.info kernel: [ 154.280000] Task dump for CPU 3: Wed Jul 12 20:15:30 2017 kern.info kernel: [ 154.290000] swapper/3 R running task 0 0 1 0x00100000 Wed Jul 12 20:15:30 2017 kern.warn kernel: [ 154.300000] Stack : 8fc77fe0 0000fc00 00000000 0031b71a 7fcc8850 00000000 00000010 77c02564 Wed Jul 12 20:15:30 2017 kern.warn kernel: [ 154.320000] 804e148c 00000008 8047f8e0 804f0000 00000001 00000001 bfbf8000 00000003 Wed Jul 12 20:15:30 2017 kern.warn kernel: [ 154.340000] 000010d9 8000c1cc 00000000 00000000 8fc76000 8fc77f00 00000000 80062ec4 Wed Jul 12 20:15:30 2017 kern.warn kernel: [ 154.350000] 1100fc03 00000003 00000001 804f0000 804e148c 80062ebc 00000001 00000001 Wed Jul 12 20:15:30 2017 kern.warn kernel: [ 154.370000] 00000001 00000001 00000001 00000001 00000001 00000001 00000001 a020046c Wed Jul 12 20:15:30 2017 kern.warn kernel: [ 154.390000] ... Wed Jul 12 20:15:30 2017 kern.warn kernel: [ 154.390000] Call Trace: Wed Jul 12 20:15:30 2017 kern.warn kernel: [ 154.400000] [<803e7ff8>] __schedule+0x5d4/0x7a4 Wed Jul 12 20:15:30 2017 kern.warn kernel: [ 154.400000] [<8000c1cc>] r4k_wait_irqoff+0x0/0x20 Wed Jul 12 20:15:30 2017 kern.err kernel: [ 154.410000] rcu_sched kthread starved for 6016 jiffies! g177 c176 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x1 Wed Jul 12 20:15:30 2017 kern.info kernel: [ 154.430000] rcu_sched S 0 7 2 0x00100000 Wed Jul 12 20:15:30 2017 kern.warn kernel: [ 154.440000] Stack : 804e15f4 8fc52340 8fc6be50 81228320 8fc6be50 804e0000 00000001 8007b4c8 Wed Jul 12 20:15:30 2017 kern.warn kernel: [ 154.460000] 81234320 ffffafa2 81234320 8fc6be50 804e0000 00000000 804f0000 804e0000 Wed Jul 12 20:15:30 2017 kern.warn kernel: [ 154.480000] 804e96b0 803e8234 ffffafa2 804e0000 8fc6be50 81234320 ffffafa2 803eab98 Wed Jul 12 20:15:30 2017 kern.warn kernel: [ 154.490000] 804e15f4 804f0000 804e9580 804e96b0 8122a240 00000001 00000000 812346d8 Wed Jul 12 20:15:30 2017 kern.warn kernel: [ 154.510000] ffffafa2 8007b9c4 8fc52340 36000003 804e9580 00000001 804e0000 00000000 Wed Jul 12 20:15:30 2017 kern.warn kernel: [ 154.530000] ... Wed Jul 12 20:15:30 2017 kern.warn kernel: [ 154.530000] Call Trace: Wed Jul 12 20:15:30 2017 kern.warn kernel: [ 154.540000] [<803e7ff8>] __schedule+0x5d4/0x7a4 Wed Jul 12 20:15:30 2017 kern.warn kernel: [ 154.540000] [<803e8234>] schedule+0x6c/0x84 Wed Jul 12 20:15:30 2017 kern.warn kernel: [ 154.550000] [<803eab98>] schedule_timeout+0x160/0x19c Wed Jul 12 20:15:30 2017 kern.warn kernel: [ 154.560000] [<80078f00>] rcu_gp_kthread+0x7f4/0x7fc Wed Jul 12 20:15:30 2017 kern.warn kernel: [ 154.570000] [<80044be8>] kthread+0xd8/0xec Wed Jul 12 20:15:30 2017 kern.warn kernel: [ 154.580000] [<8000a318>] ret_from_kernel_thread+0x14/0x1c Wed Jul 12 20:16:39 2017 kern.err kernel: [ 223.110000] INFO: rcu_sched detected stalls on CPUs/tasks: Wed Jul 12 20:16:39 2017 kern.err kernel: [ 223.120000] 3-...: (72 GPs behind) idle=e92/0/0 softirq=1115/1115 fqs=1 Wed Jul 12 20:16:39 2017 kern.err kernel: [ 223.130000] (detected by 1, t=6004 jiffies, g=191, c=190, q=931) Wed Jul 12 20:16:39 2017 kern.info kernel: [ 223.140000] Task dump for CPU 3: Wed Jul 12 20:16:39 2017 kern.info kernel: [ 223.150000] swapper/3 R running task 0 0 1 0x00100000 Wed Jul 12 20:16:39 2017 kern.warn kernel: [ 223.160000] Stack : 8fc77fe0 0000fc00 00000000 d4a850ca 00000032 ffffffff 0000002e 772762c0 Wed Jul 12 20:16:39 2017 kern.warn kernel: [ 223.180000] 804e148c 00000008 8047f8e0 804f0000 00000001 00000001 bfbf8000 00000003 Wed Jul 12 20:16:39 2017 kern.warn kernel: [ 223.200000] 00000000 8000c1cc 00000000 00000000 8fc76000 8fc77f00 00000000 80062ec4 Wed Jul 12 20:16:39 2017 kern.warn kernel: [ 223.210000] 1100fc03 00000003 00000001 804f0000 804e148c 80062ebc 00000001 00000001 Wed Jul 12 20:16:39 2017 kern.warn kernel: [ 223.230000] 00000001 00000001 00000001 00000001 00000001 00000001 00000001 a020046c Wed Jul 12 20:16:39 2017 kern.warn kernel: [ 223.250000] ... Wed Jul 12 20:16:39 2017 kern.warn kernel: [ 223.250000] Call Trace: Wed Jul 12 20:16:39 2017 kern.warn kernel: [ 223.250000] [<803e7ff8>] __schedule+0x5d4/0x7a4 Wed Jul 12 20:16:39 2017 kern.warn kernel: [ 223.260000] [<8000c1cc>] r4k_wait_irqoff+0x0/0x20 Wed Jul 12 20:16:39 2017 kern.err kernel: [ 223.270000] rcu_sched kthread starved for 6016 jiffies! g191 c190 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x1 Wed Jul 12 20:16:39 2017 kern.info kernel: [ 223.290000] rcu_sched S 0 7 2 0x00100000 Wed Jul 12 20:16:39 2017 kern.warn kernel: [ 223.300000] Stack : 804e15f4 8fc52340 8fc6be50 81228320 8fc6be50 804e0000 00000001 8007b4c8 Wed Jul 12 20:16:39 2017 kern.warn kernel: [ 223.320000] 81234320 ffffca88 81234320 8fc6be50 804e0000 00000000 804f0000 804e0000 Wed Jul 12 20:16:39 2017 kern.warn kernel: [ 223.340000] 804e96b0 803e8234 ffffca88 804e0000 8fc6be50 81234320 ffffca88 803eab98 Wed Jul 12 20:16:39 2017 kern.warn kernel: [ 223.350000] 804e15f4 804f0000 804e9580 804e96b0 8122a240 00000001 00000000 81234710 Wed Jul 12 20:16:39 2017 kern.warn kernel: [ 223.370000] ffffca88 8007b9c4 8fc52340 39800003 804e9580 00000001 804e0000 00000000 Wed Jul 12 20:16:39 2017 kern.warn kernel: [ 223.390000] ... Wed Jul 12 20:16:39 2017 kern.warn kernel: [ 223.390000] Call Trace: Wed Jul 12 20:16:39 2017 kern.warn kernel: [ 223.400000] [<803e7ff8>] __schedule+0x5d4/0x7a4 Wed Jul 12 20:16:39 2017 kern.warn kernel: [ 223.400000] [<803e8234>] schedule+0x6c/0x84 Wed Jul 12 20:16:39 2017 kern.warn kernel: [ 223.410000] [<803eab98>] schedule_timeout+0x160/0x19c Wed Jul 12 20:16:39 2017 kern.warn kernel: [ 223.420000] [<80078f00>] rcu_gp_kthread+0x7f4/0x7fc Wed Jul 12 20:16:39 2017 kern.warn kernel: [ 223.430000] [<80044be8>] kthread+0xd8/0xec Wed Jul 12 20:16:39 2017 kern.warn kernel: [ 223.440000] [<8000a318>] ret_from_kernel_thread+0x14/0x1c Wed Jul 12 20:18:06 2017 kern.err kernel: [ 309.530000] INFO: rcu_sched detected stalls on CPUs/tasks: Wed Jul 12 20:18:06 2017 kern.err kernel: [ 309.540000] 3-...: (116 GPs behind) idle=6e0/0/0 softirq=1115/1115 fqs=1 Wed Jul 12 20:18:06 2017 kern.err kernel: [ 309.550000] (detected by 2, t=6004 jiffies, g=235, c=234, q=984) Wed Jul 12 20:18:06 2017 kern.info kernel: [ 309.560000] Task dump for CPU 3: Wed Jul 12 20:18:06 2017 kern.info kernel: [ 309.570000] swapper/3 R running task 0 0 1 0x00100000 Wed Jul 12 20:18:06 2017 kern.warn kernel: [ 309.580000] Stack : 8fc77fe0 0000fc00 00000000 0000b898 00000047 00000013 00000000 00000000 Wed Jul 12 20:18:06 2017 kern.warn kernel: [ 309.600000] 804e148c 00000008 8047f8e0 804f0000 00000001 00000001 bfbf8000 00000003 Wed Jul 12 20:18:06 2017 kern.warn kernel: [ 309.620000] 00000000 8000c1cc 1100fc03 00000003 8fc76000 8fc77f00 00000000 80062ec4 Wed Jul 12 20:18:06 2017 kern.warn kernel: [ 309.630000] 1100fc03 00000003 00000001 804f0000 804e148c 80062ebc 00000001 00000001 Wed Jul 12 20:18:06 2017 kern.warn kernel: [ 309.650000] 00000001 00000001 00000001 00000001 00000001 00000001 00000001 a020046c Wed Jul 12 20:18:06 2017 kern.warn kernel: [ 309.670000] ... Wed Jul 12 20:18:06 2017 kern.warn kernel: [ 309.670000] Call Trace: Wed Jul 12 20:18:06 2017 kern.warn kernel: [ 309.680000] [<803e7ff8>] __schedule+0x5d4/0x7a4 Wed Jul 12 20:18:06 2017 kern.warn kernel: [ 309.680000] [<8000c1cc>] r4k_wait_irqoff+0x0/0x20 Wed Jul 12 20:18:06 2017 kern.err kernel: [ 309.690000] rcu_sched kthread starved for 6016 jiffies! g235 c234 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x1 Wed Jul 12 20:18:06 2017 kern.info kernel: [ 309.710000] rcu_sched S 0 7 2 0x00100000 Wed Jul 12 20:18:06 2017 kern.warn kernel: [ 309.720000] Stack : 804e15f4 8fc52340 8fc6be50 81228320 8fc6be50 804e0000 00000001 8007b4c8 Wed Jul 12 20:18:06 2017 kern.warn kernel: [ 309.740000] 81234320 ffffec4a 81234320 8fc6be50 804e0000 00000000 804f0000 804e0000 Wed Jul 12 20:18:06 2017 kern.warn kernel: [ 309.760000] 804e96b0 803e8234 ffffec4a 804e0000 8fc6be50 81234320 ffffec4a 803eab98 Wed Jul 12 20:18:06 2017 kern.warn kernel: [ 309.770000] 804e15f4 804f0000 804e9580 804e96b0 00000001 00000001 00000000 81234754 Wed Jul 12 20:18:06 2017 kern.warn kernel: [ 309.790000] ffffec4a 8007b9c4 8fc52340 3dc00003 804e9580 00000001 804e0000 00000000 Wed Jul 12 20:18:06 2017 kern.warn kernel: [ 309.810000] ... Wed Jul 12 20:18:06 2017 kern.warn kernel: [ 309.810000] Call Trace: Wed Jul 12 20:18:06 2017 kern.warn kernel: [ 309.820000] [<803e7ff8>] __schedule+0x5d4/0x7a4 Wed Jul 12 20:18:06 2017 kern.warn kernel: [ 309.820000] [<803e8234>] schedule+0x6c/0x84 Wed Jul 12 20:18:06 2017 kern.warn kernel: [ 309.830000] [<803eab98>] schedule_timeout+0x160/0x19c Wed Jul 12 20:18:06 2017 kern.warn kernel: [ 309.840000] [<80078f00>] rcu_gp_kthread+0x7f4/0x7fc Wed Jul 12 20:18:06 2017 kern.warn kernel: [ 309.850000] [<80044be8>] kthread+0xd8/0xec Wed Jul 12 20:18:06 2017 kern.warn kernel: [ 309.860000] [<8000a318>] ret_from_kernel_thread+0x14/0x1c Wed Jul 12 20:19:21 2017 kern.err kernel: [ 385.140000] INFO: rcu_sched detected stalls on CPUs/tasks: Wed Jul 12 20:19:21 2017 kern.err kernel: [ 385.150000] 3-...: (0 ticks this GP) idle=9ec/0/0 softirq=1115/1115 fqs=0 Wed Jul 12 20:19:21 2017 kern.err kernel: [ 385.160000] (detected by 1, t=6004 jiffies, g=268, c=267, q=938) Wed Jul 12 20:19:21 2017 kern.info kernel: [ 385.170000] Task dump for CPU 3: Wed Jul 12 20:19:21 2017 kern.info kernel: [ 385.180000] swapper/3 R running task 0 0 1 0x00100000 Wed Jul 12 20:19:21 2017 kern.warn kernel: [ 385.190000] Stack : 8fc77fe0 0000fc00 00000000 0001257e 00000059 ffffffff 00000000 00000000 Wed Jul 12 20:19:21 2017 kern.warn kernel: [ 385.210000] 804e148c 00000008 8047f8e0 804f0000 00000001 00000001 bfbf8000 00000003 Wed Jul 12 20:19:21 2017 kern.warn kernel: [ 385.230000] 00000000 8000c1cc 1100fc03 00000003 8fc76000 8fc77f00 00000000 80062ec4 Wed Jul 12 20:19:21 2017 kern.warn kernel: [ 385.240000] 1100fc03 00000003 00000001 804f0000 804e148c 80062ebc 00000001 00000001 Wed Jul 12 20:19:21 2017 kern.warn kernel: [ 385.260000] 00000001 00000001 00000001 00000001 00000001 00000001 00000001 a020046c Wed Jul 12 20:19:21 2017 kern.warn kernel: [ 385.280000] ... Wed Jul 12 20:19:21 2017 kern.warn kernel: [ 385.280000] Call Trace: Wed Jul 12 20:19:21 2017 kern.warn kernel: [ 385.290000] [<803e7ff8>] __schedule+0x5d4/0x7a4 Wed Jul 12 20:19:21 2017 kern.warn kernel: [ 385.290000] [<8000c1cc>] r4k_wait_irqoff+0x0/0x20 Wed Jul 12 20:19:21 2017 kern.err kernel: [ 385.300000] rcu_sched kthread starved for 6018 jiffies! g268 c267 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x1 Wed Jul 12 20:19:21 2017 kern.info kernel: [ 385.320000] rcu_sched S 0 7 2 0x00100000 Wed Jul 12 20:19:21 2017 kern.warn kernel: [ 385.330000] Stack : 804e15f4 8fc52340 8fc6be50 81228320 8fc6be50 804e0000 00000000 8007b4c8 Wed Jul 12 20:19:21 2017 kern.warn kernel: [ 385.350000] 81234320 000009d1 81234320 8fc6be50 804e0000 00000001 804f0000 804e0000 Wed Jul 12 20:19:21 2017 kern.warn kernel: [ 385.370000] 804e96b0 803e8234 000009d1 804e0000 8fc6be50 81234320 000009d1 803eab98 Wed Jul 12 20:19:21 2017 kern.warn kernel: [ 385.380000] 804e15f4 804f0000 804e9580 804e96b0 81236240 00000001 00000000 8123468c Wed Jul 12 20:19:21 2017 kern.warn kernel: [ 385.400000] 000009d1 8007b9c4 8fc52340 31400003 804e9580 00000001 804e0000 00000000 Wed Jul 12 20:19:21 2017 kern.warn kernel: [ 385.420000] ... Wed Jul 12 20:19:21 2017 kern.warn kernel: [ 385.420000] Call Trace: Wed Jul 12 20:19:21 2017 kern.warn kernel: [ 385.430000] [<803e7ff8>] __schedule+0x5d4/0x7a4 Wed Jul 12 20:19:21 2017 kern.warn kernel: [ 385.440000] [<803e8234>] schedule+0x6c/0x84 Wed Jul 12 20:19:21 2017 kern.warn kernel: [ 385.440000] [<803eab98>] schedule_timeout+0x160/0x19c Wed Jul 12 20:19:21 2017 kern.warn kernel: [ 385.450000] [<80078f00>] rcu_gp_kthread+0x7f4/0x7fc Wed Jul 12 20:19:21 2017 kern.warn kernel: [ 385.460000] [<80044be8>] kthread+0xd8/0xec Wed Jul 12 20:19:21 2017 kern.warn kernel: [ 385.470000] [<8000a318>] ret_from_kernel_thread+0x14/0x1c Wed Jul 12 20:22:28 2017 kern.err kernel: [ 571.940000] INFO: rcu_sched detected stalls on CPUs/tasks: Wed Jul 12 20:22:28 2017 kern.err kernel: [ 571.950000] 3-...: (319 GPs behind) idle=ab2/0/0 softirq=1115/1115 fqs=1 Wed Jul 12 20:22:28 2017 kern.err kernel: [ 571.960000] (detected by 1, t=6004 jiffies, g=587, c=586, q=1025) Wed Jul 12 20:22:28 2017 kern.info kernel: [ 571.970000] Task dump for CPU 3: Wed Jul 12 20:22:28 2017 kern.info kernel: [ 571.980000] swapper/3 R running task 0 0 1 0x00100000 Wed Jul 12 20:22:28 2017 kern.warn kernel: [ 571.990000] Stack : 8fc77fe0 0000fc00 00000000 1ec361d8 00000082 ffffffff 00000074 772762c0 Wed Jul 12 20:22:28 2017 kern.warn kernel: [ 572.010000] 804e148c 00000008 8047f8e0 804f0000 00000001 00000001 bfbf8000 00000003 Wed Jul 12 20:22:28 2017 kern.warn kernel: [ 572.030000] 00000000 8000c1cc 1100fc03 00000003 8fc76000 8fc77f00 00000000 80062ec4 Wed Jul 12 20:22:28 2017 kern.warn kernel: [ 572.040000] 1100fc03 00000003 00000001 804f0000 804e148c 80062ebc 00000001 00000001 Wed Jul 12 20:22:28 2017 kern.warn kernel: [ 572.060000] 00000001 00000001 00000001 00000001 00000001 00000001 00000001 a020046c Wed Jul 12 20:22:28 2017 kern.warn kernel: [ 572.080000] ... Wed Jul 12 20:22:28 2017 kern.warn kernel: [ 572.080000] Call Trace: Wed Jul 12 20:22:28 2017 kern.warn kernel: [ 572.090000] [<803e7ff8>] __schedule+0x5d4/0x7a4 Wed Jul 12 20:22:28 2017 kern.warn kernel: [ 572.090000] [<8000c1cc>] r4k_wait_irqoff+0x0/0x20 Wed Jul 12 20:22:28 2017 kern.err kernel: [ 572.100000] rcu_sched kthread starved for 6015 jiffies! g587 c586 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x1 Wed Jul 12 20:22:28 2017 kern.info kernel: [ 572.120000] rcu_sched S 0 7 2 0x00100000 Wed Jul 12 20:22:28 2017 kern.warn kernel: [ 572.130000] Stack : 804e15f4 8fc52340 8fc6be50 81228320 8fc6be50 804e0000 00000001 8007b4c8 Wed Jul 12 20:22:28 2017 kern.warn kernel: [ 572.150000] 81234320 000052cc 81234320 8fc6be50 804e0000 00000000 804f0000 804e0000 Wed Jul 12 20:22:28 2017 kern.warn kernel: [ 572.170000] 804e96b0 803e8234 000052cc 804e0000 8fc6be50 81234320 000052cc 803eab98 Wed Jul 12 20:22:28 2017 kern.warn kernel: [ 572.180000] 804e15f4 804f0000 804e9580 804e96b0 8122a240 00000001 00000000 81234790 Wed Jul 12 20:22:28 2017 kern.warn kernel: [ 572.200000] 000052cc 8007b9c4 8fc52340 41800003 804e9580 00000001 804e0000 00000000 Wed Jul 12 20:22:28 2017 kern.warn kernel: [ 572.220000] ... Wed Jul 12 20:22:28 2017 kern.warn kernel: [ 572.220000] Call Trace: Wed Jul 12 20:22:28 2017 kern.warn kernel: [ 572.230000] [<803e7ff8>] __schedule+0x5d4/0x7a4 Wed Jul 12 20:22:28 2017 kern.warn kernel: [ 572.230000] [<803e8234>] schedule+0x6c/0x84 Wed Jul 12 20:22:28 2017 kern.warn kernel: [ 572.240000] [<803eab98>] schedule_timeout+0x160/0x19c Wed Jul 12 20:22:28 2017 kern.warn kernel: [ 572.250000] [<80078f00>] rcu_gp_kthread+0x7f4/0x7fc Wed Jul 12 20:22:28 2017 kern.warn kernel: [ 572.260000] [<80044be8>] kthread+0xd8/0xec Wed Jul 12 20:22:28 2017 kern.warn kernel: [ 572.270000] [<8000a318>] ret_from_kernel_thread+0x14/0x1c Wed Jul 12 20:23:30 2017 kern.err kernel: [ 633.560000] INFO: rcu_sched detected stalls on CPUs/tasks: Wed Jul 12 20:23:30 2017 kern.err kernel: [ 633.570000] 3-...: (347 GPs behind) idle=070/0/0 softirq=1115/1115 fqs=1 Wed Jul 12 20:23:30 2017 kern.err kernel: [ 633.580000] (detected by 1, t=6004 jiffies, g=615, c=614, q=1011) Wed Jul 12 20:23:30 2017 kern.info kernel: [ 633.590000] Task dump for CPU 3: Wed Jul 12 20:23:30 2017 kern.info kernel: [ 633.600000] swapper/3 R running task 0 0 1 0x00100000 Wed Jul 12 20:23:30 2017 kern.warn kernel: [ 633.610000] Stack : 00000068 0000003c 00000001 00000001 00000020 00000000 00000000 00000001 Wed Jul 12 20:23:30 2017 kern.warn kernel: [ 633.630000] 804e148c 00000008 8047f8e0 804f0000 00000001 00000001 bfbf8000 00000003 Wed Jul 12 20:23:30 2017 kern.warn kernel: [ 633.650000] 00000000 8000c1cc 00000000 00000000 8fc76000 8fc77f00 00000000 80062ec4 Wed Jul 12 20:23:30 2017 kern.warn kernel: [ 633.660000] 1100fc03 00000003 00000001 804e148c 804e148c 80062ebc 00000001 00000001 Wed Jul 12 20:23:30 2017 kern.warn kernel: [ 633.680000] 00000001 00000001 00000001 00000001 00000001 00000001 00000001 a020046c Wed Jul 12 20:23:30 2017 kern.warn kernel: [ 633.700000] ... Wed Jul 12 20:23:30 2017 kern.warn kernel: [ 633.700000] Call Trace: Wed Jul 12 20:23:30 2017 kern.warn kernel: [ 633.710000] [<803e7ff8>] __schedule+0x5d4/0x7a4 Wed Jul 12 20:23:30 2017 kern.warn kernel: [ 633.710000] [<8000c1cc>] r4k_wait_irqoff+0x0/0x20 Wed Jul 12 20:23:30 2017 kern.err kernel: [ 633.720000] rcu_sched kthread starved for 6015 jiffies! g615 c614 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x1 Wed Jul 12 20:23:30 2017 kern.info kernel: [ 633.740000] rcu_sched S 0 7 2 0x00100000 Wed Jul 12 20:23:30 2017 kern.warn kernel: [ 633.750000] Stack : 804e15f4 8fc52340 8fc6be50 81228320 8fc6be50 804e0000 00000001 8007b4c8 Wed Jul 12 20:23:30 2017 kern.warn kernel: [ 633.770000] 81234320 00006ade 81234320 8fc6be50 804e0000 00000000 804f0000 804e0000 Wed Jul 12 20:23:30 2017 kern.warn kernel: [ 633.790000] 804e96b0 803e8234 00006ade 804e0000 8fc6be50 81234320 00006ade 803eab98 Wed Jul 12 20:23:30 2017 kern.warn kernel: [ 633.800000] 804e15f4 804f0000 804e9580 804e96b0 00000001 00000001 00000000 81234794 Wed Jul 12 20:23:30 2017 kern.warn kernel: [ 633.820000] 00006ade 8007b9c4 8fc52340 41c00003 804e9580 00000001 804e0000 00000000 Wed Jul 12 20:23:30 2017 kern.warn kernel: [ 633.840000] ... Wed Jul 12 20:23:30 2017 kern.warn kernel: [ 633.840000] Call Trace: Wed Jul 12 20:23:30 2017 kern.warn kernel: [ 633.850000] [<803e7ff8>] __schedule+0x5d4/0x7a4 Wed Jul 12 20:23:30 2017 kern.warn kernel: [ 633.850000] [<803e8234>] schedule+0x6c/0x84 Wed Jul 12 20:23:30 2017 kern.warn kernel: [ 633.860000] [<803eab98>] schedule_timeout+0x160/0x19c Wed Jul 12 20:23:30 2017 kern.warn kernel: [ 633.870000] [<80078f00>] rcu_gp_kthread+0x7f4/0x7fc Wed Jul 12 20:23:30 2017 kern.warn kernel: [ 633.880000] [<80044be8>] kthread+0xd8/0xec Wed Jul 12 20:23:30 2017 kern.warn kernel: [ 633.890000] [<8000a318>] ret_from_kernel_thread+0x14/0x1c
This is one I got trying to refresh a page in LUCI/usr/lib/lua/luci/dispatcher.lua:466: Failed to execute function dispatcher target for entry '/'. The called action terminated with an exception: /usr/lib/lua/luci/util.lua:623: Unable to establish ubus connection stack traceback: [C]: in function 'assert' /usr/lib/lua/luci/dispatcher.lua:466: in function 'dispatch' /usr/lib/lua/luci/dispatcher.lua:121: in function </usr/lib/lua/luci/dispatcher.lua:120>

Is this likely to be something I've caused? Or is this worthy of a ticket on the bug tracker? I saw this device has only just been updated to kernel 4.9 in LEDE master. Should I just put this down to teething issues?

Installed packages

adblock - 2.8.3-1 base-files - 174-r4561-254f0da busybox - 1.26.2-7 dnscrypt-proxy - 1.9.5-4 dnscrypt-proxy-resolvers - 1.9.5+git-20170530-60baef4-4 dnsmasq - 2.77-7 dropbear - 2017.75-3 firewall - 2017-05-27-a4d98aea-1 fstools - 2017-05-09-c43ae11e-2 fwtool - 1 ip6tables - 1.6.1-1 iptables - 1.6.1-1 iptables-mod-conntrack-extra - 1.6.1-1 iptables-mod-ipopt - 1.6.1-1 jshn - 2017-06-17-fd57eea9-1 jsonfilter - 2016-07-02-dea067ad-1 kernel - 4.9.34-1-7f0de30d5b73958cb146494d8e5b2ef4 kmod-gpio-button-hotplug - 4.9.34-2 kmod-ifb - 4.9.34-1 kmod-ip6tables - 4.9.34-1 kmod-ipt-conntrack - 4.9.34-1 kmod-ipt-conntrack-extra - 4.9.34-1 kmod-ipt-core - 4.9.34-1 kmod-ipt-ipopt - 4.9.34-1 kmod-ipt-ipset - 4.9.34-1 kmod-ipt-nat - 4.9.34-1 kmod-ipt-nat6 - 4.9.34-1 kmod-leds-gpio - 4.9.34-1 kmod-lib-crc-ccitt - 4.9.34-1 kmod-nf-conntrack - 4.9.34-1 kmod-nf-conntrack6 - 4.9.34-1 kmod-nf-ipt - 4.9.34-1 kmod-nf-ipt6 - 4.9.34-1 kmod-nf-nat - 4.9.34-1 kmod-nf-nat6 - 4.9.34-1 kmod-nfnetlink - 4.9.34-1 kmod-ppp - 4.9.34-1 kmod-pppoe - 4.9.34-1 kmod-pppox - 4.9.34-1 kmod-sched-cake - 4.9.34+2017-01-28-9789742c-1 kmod-sched-core - 4.9.34-1 kmod-slhc - 4.9.34-1 lede-keyring - 2017-01-20-a50b7529-1 libblobmsg-json - 2017-06-17-fd57eea9-1 libc - 1.1.16-1 libgcc - 5.4.0-1 libip4tc - 1.6.1-1 libip6tc - 1.6.1-1 libiwinfo - 2016-09-21-fd9e17be-1 libiwinfo-lua - 2016-09-21-fd9e17be-1 libjson-c - 0.12.1-1 libjson-script - 2017-06-17-fd57eea9-1 liblua - 5.1.5-1 libmbedtls - 2.5.1-1 libnl-tiny - 0.1-5 libpthread - 1.1.16-1 libsodium - 1.0.12-1 libubox - 2017-06-17-fd57eea9-1 libubus - 2017-02-18-34c6e818-1 libubus-lua - 2017-02-18-34c6e818-1 libuci - 2017-04-12-c4df32b3-1 libuci-lua - 2016-07-04-e1bf4356-1 libuclient - 2016-12-09-52d955fd-1 libustream-mbedtls - 2016-07-02-ec80adaa-2 libxtables - 1.6.1-1 logd - 2017-03-03-21a4bd04-1 lua - 5.1.5-1 luci - git-17.190.70526-d6360bf-1 luci-app-adblock - git-17.190.70526-d6360bf-1 luci-app-firewall - git-17.190.70526-d6360bf-1 luci-app-sqm - 1.1.3-1 luci-app-uhttpd - git-17.190.70526-d6360bf-1 luci-base - git-17.190.70526-d6360bf-1 luci-lib-ip - git-17.190.70526-d6360bf-1 luci-lib-jsonc - git-17.190.70526-d6360bf-1 luci-lib-nixio - git-17.190.70526-d6360bf-1 luci-mod-admin-full - git-17.190.70526-d6360bf-1 luci-proto-ipv6 - git-17.190.70526-d6360bf-1 luci-proto-ppp - git-17.190.70526-d6360bf-1 luci-ssl - git-17.190.70526-d6360bf-1 luci-theme-bootstrap - git-17.190.70526-d6360bf-1 luci-theme-material - git-17.190.70526-d6360bf-1 mtd - 21 netifd - 2017-06-13-ef5f7a09-1 odhcp6c - 2017-03-22-0463b057-2 odhcpd - 2017-07-04-f0d78e7e-1 openssh-sftp-server - 7.5p1-2 opkg - 2017-07-11-52fc0069-1 ppp - 2.4.7-12 ppp-mod-pppoe - 2.4.7-12 procd - 2017-06-22-e5e99c46-3 px5g-mbedtls - 4 rpcd - 2016-12-03-0577cfc1-1 sqm-scripts - 1.1.3-1 swconfig - 11 tc - 4.11.0-1 ubi-utils - 1.5.2-1 ubox - 2017-03-03-21a4bd04-1 ubus - 2017-02-18-34c6e818-1 ubusd - 2017-02-18-34c6e818-1 uci - 2017-04-12-c4df32b3-1 uclient-fetch - 2016-12-09-52d955fd-1 uhttpd - 2016-10-25-1628fa4b-2 uhttpd-mod-ubus - 2016-10-25-1628fa4b-2 usign - 2015-07-04-ef641914-1

Just to be clear, not complaining, I know it's git master/snapshot/WIP and I'm bound to encounter issues. Mainly I would like to know if I'm causing it!

Cheers

2

This is mt7621 right? Are you using SQM by any chance?

3

It is and I am. I turned it off (disabled init) but didn't remove it... Should I?

Edit: Removed it in anticipation of your answer. I'll check on it in a few hours

4

That seems to have fixed it. This looks similar to my issue https://bugs.lede-project.org/index.php?do=details&task_id=804&order=dateopened&sort=desc

Also, I've seen mention of issues with SQM on dir-860l. Just noticed it's the same target.

Does SQM work on mt7621 in 17.01.2?

5

No it shouldn't cause any problems if it's disabled... It's just that SQM with Cake makes mt7621 unstable, but if it's not active it shouldn't matter.

6

Disabling it was the first thing I did but I think I would have needed to disable all the associated packages to do it properly (e.g. kmod-sched-core).

Anyway, removing them did the same thing :slight_smile:

By the way, is that ticket I linked about the same issue? If not, does one exist?

7

I'm not sure if it is the same issue. I wasn't aware of that bug report, this one is the main report for the mt7621 stability problem.

8

Thanks. Voted for both.