Mtk_soc_eth watchdog timeout after r11573

SirToffski · 2020-03-04T03:17:12.386Z

Come to think of it, before switching from the master to the stable branch, I did flash ERX 2.0.x recovery image via TTL, not sure if this made any deference IRL; worth mentioning maybe.

apocalypse · 2020-03-04T07:47:21.807Z

My switch VLAN configuration is similary.

Maybe when updating the bootloader with the recovery method the problems disappear. Before using the recovery method did you have problems with the kernel or reboots/freezes with the master branch?

SirToffski · 2020-03-04T20:20:10.772Z

Yep, before using recovery method I did experience random freezes/kernel panics. It's interesting as I've presumed moving to stable branches must have been the solution - however now it's not as clear.

I suppose ERX recovery images v1.10.x + and v2.0.x would update the boot-loader because these introduced SSH-recovery option.

I'm compiling 19.07.02 now from source, hopefully it would be stable as well.

P.S. I used to send all kernel logs to a syslog-ng server on my LAN, I'll dig though the logs to see if I can find some kernel events while running on master.

SirToffski · 2020-03-04T21:27:48.755Z

All I could find in old dmesg messages are kernel oops events presumably caused by flow-offloading, here are some examples. No messages matching (mtk_soc_eth): transmit queue 0 timed out

Here are the events: https://gist.github.com/SirToffski/7041c0191cdfd0ab3c4ac8546007f4d6

SirToffski · 2020-03-05T19:30:20.355Z

Just built and flashed today's master, will report if there are stability issues in ERX.

apocalypse · 2020-03-13T02:00:39.752Z

19.07.1 with this patch https://patchwork.ozlabs.org/patch/1186211/ and without if condition.

Uptime
8d 9h 10m 32s and no kernel error, reboots or freezes.

There are also no interrupts errors:

root @ UBNT-ERX: ~ # cat / proc / interrupts
           CPU0 CPU1 CPU2 CPU3
  8: 72430831 72430828 72430820 72430815 MIPS GIC Local 1 timer
  9: 1248747 0 0 0 MIPS GIC 63 IPI call
 10: 0 248950327 0 0 MIPS GIC 64 IPI call
 11: 0 0 304656537 0 MIPS GIC 65 IPI call
 12: 0 0 0 470619082 MIPS GIC 66 IPI call
 13: 1565767 0 0 0 MIPS GIC 67 IPI resched
 14: 0 2153705 0 0 MIPS GIC 68 IPI resched
 15: 0 0 2398922 0 MIPS GIC 69 IPI resched
 16: 0 0 0 1220371 MIPS GIC 70 IPI resched
 19: 12 0 0 0 MIPS GIC 33 ttyS0
 20: 2106289666 0 0 0 MIPS GIC 10 1e100000.ethernet
 21: 53 0 0 0 MIPS GIC 30 gsw
ERR: 0

I don't want to sing victory, but things look very, very good. I have never had an uptime of more than 5 days without problems.

Thanks neheb for linking the patch. I saw it a few months ago but i didn't try it, i thought it wouldn't fix this.

Enig123 · 2020-03-13T06:13:09.017Z

Wow, that's impressive, 0 ERRs!

Mushoz · 2020-03-13T15:34:17.996Z

When is this patch hitting Master? Was a pull request already created?

Enig123 · 2020-03-13T21:02:03.510Z

I am wondering if Archer C7 v2 has similar interrupt handling issues, there’s also a lot of ERRs in the interrupt output.

Enig123 · 2020-03-14T03:50:34.642Z

apocalypse:

19.07.1 with this patch https://patchwork.ozlabs.org/patch/1186211/ and without if condition.

What do you mean by 'without if condition', were you meaning modify the patch by eliminate the added 'if' line in routine fe_poll_tx?

BTW, I am compiling master branch with this patch you referred to, and will start testing tomorrow.

neheb · 2020-03-14T06:48:04.091Z

Just saw those errors on my Archer C7v3. Guess I need to swap it out...

apocalypse · 2020-03-14T16:32:18.178Z

Enig123:

apocalypse:

19.07.1 with this patch https://patchwork.ozlabs.org/patch/1186211/ and without if condition.

What do you mean by 'without if condition', were you meaning modify the patch by eliminate the added 'if' line in routine fe_poll_tx?

BTW, I am compiling master branch with this patch you referred to, and will start testing tomorrow.

No, i am talking about this patch. This disable Flow Control. https://github.com/openwrt/mt76/issues/211#issuecomment-569944489. Remove all conditional and leave alone:

/ * (GE1, Force 1000M / FD, FC OFF, MAX_RX_LENGTH 1536) * /
mtk_switch_w32 (gsw, 0x2305e30b, GSW_REG_MAC_P0_MCR);
mt7530_mdio_w32 (gsw, 0x3600, 0x5e30b);

you don't need to apply this patch, just edit the file at ./target/linux/ramips/files-4.14/drivers/net/ethernet/mediatek/gsw_mt7621.c

Enig123 · 2020-03-16T01:07:23.893Z

I can't get zero interrupt ERRs, with hw offloading on at least.

           CPU0       CPU1       CPU2       CPU3
  8:     225649     225616     225626     225614  MIPS GIC Local   1  timer
  9:      10551          0          0          0  MIPS GIC  63  IPI call
 10:          0       3335          0          0  MIPS GIC  64  IPI call
 11:          0          0      10829          0  MIPS GIC  65  IPI call
 12:          0          0          0       3295  MIPS GIC  66  IPI call
 13:    3403753          0          0          0  MIPS GIC  67  IPI resched
 14:          0      96741          0          0  MIPS GIC  68  IPI resched
 15:          0          0      35819          0  MIPS GIC  69  IPI resched
 16:          0          0          0      27239  MIPS GIC  70  IPI resched
 19:         14          0          0          0  MIPS GIC  33  ttyS0
 20:          0          0          0          0  MIPS GIC  29  xhci-hcd:usb1
 21:    1205402          0          0          0  MIPS GIC  10  1e100000.ethernet
 22:          2          0          0          0  MIPS GIC  30  gsw
 23:          2          0      58652          0  MIPS GIC  11  mt76x2e
 24:     206807          0          0          0  MIPS GIC  31  mt76x2e
 26:          0          0          0          0      GPIO   7  keys
 27:          0          0          0          0      GPIO  18  keys
ERR:          7

Just run it for less than 40 minutes.

apocalypse · 2020-03-16T02:16:12.541Z

Surely those interruption errors are due to the wifi interface.

Anyway, have you applied both patches? (220-mt7621-disable-flow-control and OpenWrt-Devel-PATCHv2-2-2-ramips-ethernet-fix-to-interrupt-handling)?

Copy patches to root of build path and apply with the commands:

patch -p1 < 220-mt7621-disable-flow-control.patch
patch -p1 < OpenWrt-Devel-PATCHv2-2-2-ramips-ethernet-fix-to-interrupt-handling.patch

If you are a building master, you can skip the first one.

I have uploaded both patches: https://www.mediafire.com/file/kzcmkazpsntny0b/openwrt_patches.zip/file

Enig123 · 2020-03-16T02:51:48.614Z

I am using the master branch, in which the other patch you were referring to already included.

BTW, what’s your use case, is it without the WiFi?

apocalypse · 2020-03-16T03:06:54.808Z

Ubiquiti Edgerouter X, no WiFi.

Enig123 · 2020-03-16T03:10:52.675Z

OK, I will have to test it with my DIR-860L B1 for a longer time to see if the timeout issues gone with this patch, then.

apocalypse · 2020-03-16T19:44:35.471Z

12 days without errors and the router was restarted 1 hour ago. I have updated the ER-X bootloader (it had the factory version) to see if this fixed. In addition, that bootloader had a very serious security issue, at boot the switch ports communicated with each other until the system boots.

yoonsikp · 2020-03-17T02:09:35.453Z

Hello, I am wondering if these patches will make it to master?

Enig123 · 2020-03-17T09:41:53.181Z

Hard to say in which way it will be adopted.

Now the developers are busy transitioning to the next major release which is based on Linux 5.4 kernel, for my understanding it’s a whole new story than just patches.

Best hope is, this particular patch, as critical as is, will be in 19.07.x branch eventually.