Mt7621 / mt7530 programming: Disabling Flow Control on all ports

For Developers
#25

Yes, you are right. MAC 0 through 4 are all connected to the external PHYs and shouldn't be touched I think. Because bit 4&5 only work if bit 15 is set to 1, but setting bit 15 to 1 will make the MAC use all the bits with "FORCE_" prepended, and thus will force the link to a certain speed. The patch should probably be changed to:

/* (GE1, Force 1000M/FD, FC OFF, MAX_RX_LENGTH 1536) */
mtk_switch_w32(gsw, 0x2305e30b, GSW_REG_MAC_P0_MCR);
for (i = 5; i <= 6; i++) {
	mt7530_mdio_w32(gsw, 0x3000 + (i * 0x100), 0x5e30b);
}

/* Disable Flow Control Globally */
val = mt7530_mdio_r32(gsw, 0x1FE0);
val &= ~BIT(31);
mt7530_mdio_w32(gsw, 0x1FE0, val);

/* turn off pause advertisement on all PHYs */
for (i = 0; i <= 4; i++) {
	val = _mt7620_mii_read(gsw, i, 4);
	val &= ~BIT(10);
	_mt7620_mii_write(gsw, i, 4, val);
}
#26

An update on the flashed router by the way: It has been running for a week now, but unfortunately I did see the transmit queue has timed out error. It didn't result in a reboot. I am not sure if the connection was interrupted and for how long, since I wasn't using the connection when it happened. Still, it's not very promising to see the issue crop up again.

#27

Since May there had been no problem. It has rebooted after 12 hours working with ports 1~4 in the same VLAN. I give up, it is best to use an external Switch and use each port of the mediatek in a different VLAN (with Software Bridge if necessary).

It is ironic that being a SoC intended for use as a switch it cannot be used as such. I hope that in the official firmwares the same does not happen (for example in EdgeOS from Ubiquiti), if not this SoC is rubbish.

I leave it as before: GMAC Port 5 (6) FC Off, interrupt handling patch and separate VLAN for each port. In this way is rock solid.

#28

Should be this topic kept as solved?

Or the problem is now covered again by the original Mtk_soc_eth watchdog timeout after r11573 topic?

#29

Well, I am not entirely sure yet. The problematic router has been running for 2.5 weeks now without crashing. Previously, it would crash once a week on average. While 2.5 weeks without crashing was possible before, it was pretty rare. So I am going to let it run for a bit longer to see if it can get 4 weeks uptime. If so, I will flash it again with the latest version of the patch (Which doesn't force 1000 mbit on all ports) and see if that is still sufficient to prevent it from crashing. I'm starting to become cautiously optimistic at the moment.

#30

So to add to the work you had did, I've allowed flow control on ports 0-4 and left ports 5,6 as off for flow control.

Without flow control the cpu is working twice as hard and can cause kernel crashes.

But i've added a reset function on the switch, as pause frames can still be sent from other switches. I've experienced complete lockups on the switch, so with this new reset function it clears itself automatically.

1 Like
#31

Interesting. I did some digging and found that this is an old patch by Kristian: https://patchwork.ozlabs.org/project/lede/patch/20170831082245.22255-1-kristian.evensen@gmail.com/

This is definitely something I will add to my builds as a precaution. I haven't experienced any crashes yet (thankfully!), but improving the reset functionality isn't a bad idea.

Which flow control option causes this exactly? I agree that the flow control options on the MAC 0 through 4 should be disabled, since they need to be able to auto-negotiate things like link speed, so forcing everything there isn't a viable option. But as far as I know, disabling flow control globally shouldn't be an issue, right?

At the moment I haven't had any kernel crashes, and I didn't see worse performance / higher CPU usage either. What exactly where you seeing? And was this still an issue with my second version which doesn't touch MAC 0 through 4?

#32

So thats the patch i used that you had found, it never got applied in mainstream. Which is a shame as it does alleviate when the mt7530 goes mental and hard locks.

A user from my xiamio thread had reported

<4>[85798.489144] CPU: 3 PID: 18589 Comm: kworker/3:0 Not tainted 4.14.201 #0
<4>[85798.495745] Workqueue: events_power_efficient 0x86944304 [xt_FLOWOFFLOAD@86944000+0xc10]
<4>[85798.503801] task: 87f9aca0 task.stack: 8703c000
<4>[85798.508308] $ 0   : 00000000 00000001 00828260 00000001
<4>[85798.513526] $ 4   : 00828260 8703de4f 8703de4f ffff00fe
<4>[85798.518744] $ 8   : 8703dfe0 00007c00 00004e08 00000001
<4>[85798.523957] $12   : 0000031f 00000000 ffffffff 000044ac
<4>[85798.529169] $16   : 8703de4f 86944000 00000000 fffffff5
<4>[85798.534380] $20   : 8056fba0 00000001 000000c0 fffffffe
<4>[85798.539592] $24   : 00000000 80008e34                  
<4>[85798.544803] $28   : 8703c000 8703ddf0 80550000 86f203ac
<4>[85798.550016] Hi    : 0000000a
<4>[85798.552880] Lo    : 66666669
<4>[85798.555759] epc   : 86f20384 0x86f20384 [nf_flow_table@86f20000+0x3350]
<4>[85798.562350] ra    : 86f203ac 0x86f203ac [nf_flow_table@86f20000+0x3350]
<1>[85798.563012] CPU 2 Unable to handle kernel paging request at virtual address 00828264, epc == 8049ef08, ra == 86f20268
<4>[85798.568931] Status: 11007c03	KERNEL EXL IE 
<4>[85798.568950] Cause : 40800008 (ExcCode 02)
<4>[85798.568963] BadVA : 0082828e
<4>[85798.590512] PrId  : 0001992f (MIPS 1004Kc)
<4>[85798.594586] Modules linked in: pppoe ppp_async pppox ppp_generic nf_conntrack_ipv6 mt76x2e mt76x2_common mt76x02_lib mt7603e mt76 mac80211 iptable_nat ipt_REJECT ipt_MASQUERADE cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_recent xt_quota xt_pkttype xt_owner xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_helper xt_geoip xt_ecn xt_dscp xt_conntrack xt_connmark xt_connlimit xt_connbytes xt_comment xt_addrtype xt_TCPMSS xt_REDIRECT xt_LOG xt_HL xt_FLOWOFFLOAD xt_DSCP xt_CT xt_CLASSIFY wireguard spidev slhc nf_reject_ipv4 nf_nat_redirect nf_nat_masquerade_ipv4 nf_conntrack_ipv4 nf_nat_ipv4 nf_nat nf_log_ipv4 nf_flow_table_hw nf_flow_table nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_rtcache nf_conntrack_netlink macremapper iptable_raw iptable_mangle iptable_filter ipt_ECN
<4>[85798.665182]  ip_tables crc_ccitt compat br_netfilter sch_cake nf_conntrack sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_tcindex cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 ifb ip6_udp_tunnel udp_tunnel leds_gpio gpio_button_hotplug
<4>[85798.724816] Process kworker/3:0 (pid: 18589, threadinfo=8703c000, task=87f9aca0, tls=00000000)
<4>[85798.733384] Stack : 80550000 80088698 805acda0 8054e1e8 86944b80 00828260 0000001b 00000100
<4>[85798.741733]         00000200 85fdfc00 0000000f 00000001 80550000 00000000 86944b70 86940000
<4>[85798.750080]         86940000 86944374 8054dd00 804a15a0 00000001 87f9aca0 8113fa20 01142a00
<4>[85798.758429]         00000000 86944b40 86efc780 8113fa20 81142c00 00000000 00000000 800460a8
<4>[85798.766776]         8113fbc0 8113fa38 80550000 8113fbc0 80550000 fffffffe 86efc780 8113fa20
<4>[85798.775125]         ...
<4>[85798.777564] Call Trace:
<4>[85798.777577] [<80088698>] 0x80088698
<4>[85798.783503] [<86940000>] 0x86940000 [xt_LOG@86940000+0x300]
<4>[85798.789058] [<86940000>] 0x86940000 [xt_LOG@86940000+0x300]
<4>[85798.794614] [<86944374>] 0x86944374 [xt_FLOWOFFLOAD@86944000+0xc10]

Enabling back offload globally stopped this from happening. i'm guessing the cpu got overwhelmed?

I havent experienced it myself, but since putting it back no reported issues.

#33

That stack trace seems to be about flow offload, which is something completely different than flow control. In flow control, pause frames can be sent at the ethernet level to signify the other end of the link to pause transmission when overwhelmed to clear up the queues. Without flow control, there is no way to signify the other end of the link, which will lead to dropped frames. Layers on top (TCP/IP) should slow down and do a retransmission, basically shifting the responsibility of slowing down to higher layers. Ultimately, having flow control enabled / disabled shouldn't impact the CPU usage at all:

  1. With flow control the other side is asked to slow down.
  2. Without flow control the other side isn't asked to slow down and the frames are simply dropped instead. These frames will never reach the CPU, and will therefor have zero impact on the CPU usage.
1 Like
#34

It's been up for 3 weeks now without reboots or lockups with the first version of these patches (the one that forces ethernet to 1Gb/s). It does have a few transmit queue has timed out errors in the log, but it didn't result in anything breaking. Without these patches, it would lock up or hard reboot once every week on average. Going to let it run for one more week to reach the 1 month mark. I will then flash it with the second version of the patch that doesn't force the ethernet speeds and reevaluate stability. Looking good so far though!

2 Likes
#35

Another update: Four weeks without a single hardlock/reboot now. I am not sure if this fixes it for everyone, but for me personally I consider it fixed since this used to happen once a week on average. My steps will be:

I will keep everyone posted!

#36

I assume the best was if this patch could use parameter in DTS,
then we could have some devices with enabled patch and some without it.

It would be IMHO the best solution and will finally make stable support for MT7621

#37

This patch only touches mt7621 devices, which all have a mt7530 switch and hence all probably have this bug. The reason why not all mt7621 devices require it (for example, I have a mt7621 device at home that doesn't) is that the bug is triggered by other devices sending pause frames in a certain way. So if there aren't incompatible devices that are connected, you won't ever see this bug pop-up.

In my opinion, the best way to implement this would be to add in support so that ethtool can be used for toggling pause frames on/off, so that by default this functionality is in one state, and people are able to change it if they run into issues. However, this driver is only used in the 19.07 branch, since the master branch is using DSA drivers. Therefor, I think it's unlikely a patch implementing this new feature would be accepted in a stable branch. So for me personally, this small patch is the best solution for getting the 19.07 branch stable.

I will evaluate again once a stable 20.xx branch is out to see what the best course of action is for that particular branch. I haven't tested the master branch myself on the problematic device, since it's used in a small business in production, so I cannot really tinker with it.

#38

I have been using that patch (interrupt handling) since May, without it transmit timed out happens every day. But this alone does not cause reboots or hangs.

I have tested your patch on 19.07.4 and more than one port in the same VLAN and in 12 hours it rebooted.

For me the only solution to reboots or hangs is to separate each port in a VLAN, and if you don't want to see more trasmit timed outs use the interrupt handling patch.

Do you have multiple ports on the same VLAN on the problematic router?

I am currently on 19.07.4 with no problems, but i can't put more than one port on the same VLAN, this is what triggers the reboots.

#39 
Fri Nov 13 19:56:37 2020 kern.warn kernel: [  600.152978] ------------[ cut here ]------------
Fri Nov 13 19:56:37 2020 kern.warn kernel: [  600.162196] WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:320 0x8038c0d0
Fri Nov 13 19:56:37 2020 kern.info kernel: [  600.176268] NETDEV WATCHDOG: eth0 (mtk_soc_eth): transmit queue 0 timed out
Fri Nov 13 19:56:37 2020 kern.warn kernel: [  600.190171] Modules linked in: pppoe ppp_async pppox ppp_generic nf_nat_pptp nf_conntrack_pptp nf_conntrack_ipv6 mt76x2e mt76x2_common mt76x02_lib mt7603e mt76 mac80211 iptable_nat ipt_REJECT ipt_MASQUERADE ftdi_sio cfg80211 xt_time xt_tcpudp xt_state xt_nat xt_multiport xt_mark xt_mac xt_limit xt_conntrack xt_comment xt_TCPMSS xt_REDIRECT xt_LOG xt_FLOWOFFLOAD xt_CT usbserial usblp ts_fsm ts_bm slhc nf_reject_ipv4 nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_redirect nf_nat_proto_gre nf_nat_masquerade_ipv4 nf_nat_irc nf_conntrack_ipv4 nf_nat_ipv4 nf_nat_h323 nf_nat_amanda nf_nat nf_log_ipv4 nf_flow_table_hw nf_flow_table nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_tftp nf_conntrack_snmp nf_conntrack_sip nf_conntrack_rtcache nf_conntrack_proto_gre nf_conntrack_irc nf_conntrack_h323 nf_conntrack_broadcast
Fri Nov 13 19:56:37 2020 kern.warn kernel: [  600.332706]  ts_kmp nf_conntrack_amanda nf_conntrack iptable_raw iptable_mangle iptable_filter ip_tables crc_ccitt compat ledtrig_usbport nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 tun vfat fat ntfs nls_utf8 nls_iso8859_1 nls_cp437 usb_storage leds_gpio xhci_plat_hcd xhci_pci xhci_mtk xhci_hcd sd_mod scsi_mod gpio_button_hotplug ext4 mbcache jbd2 usbcore nls_base usb_common crc32c_generic
Fri Nov 13 19:56:37 2020 kern.warn kernel: [  600.412146] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.14.195 #0
Fri Nov 13 19:56:37 2020 kern.warn kernel: [  600.424264] Stack : 00000000 00000000 00000000 87f43f40 00000000 00000000 00000000 00000000
Fri Nov 13 19:56:37 2020 kern.warn kernel: [  600.440902]         00000000 00000000 00000000 00000000 00000000 00000001 87c09d60 53261622
Fri Nov 13 19:56:37 2020 kern.warn kernel: [  600.457537]         87c09df8 00000000 00000000 00006938 00000038 8049c858 00000008 00000000
Fri Nov 13 19:56:37 2020 kern.warn kernel: [  600.474171]         00000000 80550000 000649f2 00000000 87c09d40 00000000 00000000 8050aed8
Fri Nov 13 19:56:37 2020 kern.warn kernel: [  600.490804]         8038c0d0 00000140 00000000 87f43f40 00000000 802ad210 00000000 806b0000
Fri Nov 13 19:56:37 2020 kern.warn kernel: [  600.507438]         ...
Fri Nov 13 19:56:37 2020 kern.warn kernel: [  600.512299] Call Trace:
Fri Nov 13 19:56:37 2020 kern.warn kernel: [  600.512319] [<8049c858>] 0x8049c858
Fri Nov 13 19:56:37 2020 kern.warn kernel: [  600.524094] [<8038c0d0>] 0x8038c0d0
Fri Nov 13 19:56:37 2020 kern.warn kernel: [  600.531028] [<802ad210>] 0x802ad210
Fri Nov 13 19:56:37 2020 kern.warn kernel: [  600.537964] [<8000c1a0>] 0x8000c1a0
Fri Nov 13 19:56:37 2020 kern.warn kernel: [  600.544893] [<8000c1a8>] 0x8000c1a8
Fri Nov 13 19:56:37 2020 kern.warn kernel: [  600.551820] [<804856b4>] 0x804856b4
Fri Nov 13 19:56:37 2020 kern.warn kernel: [  600.558751] [<80071ab0>] 0x80071ab0
Fri Nov 13 19:56:37 2020 kern.warn kernel: [  600.565679] [<8002e608>] 0x8002e608
Fri Nov 13 19:56:37 2020 kern.warn kernel: [  600.572609] [<8038c0d0>] 0x8038c0d0
Fri Nov 13 19:56:37 2020 kern.warn kernel: [  600.579544] [<8002e690>] 0x8002e690
Fri Nov 13 19:56:37 2020 kern.warn kernel: [  600.586477] [<800884b4>] 0x800884b4
Fri Nov 13 19:56:37 2020 kern.warn kernel: [  600.593456] [<8038c0d0>] 0x8038c0d0
Fri Nov 13 19:56:37 2020 kern.warn kernel: [  600.600386] [<8038bf24>] 0x8038bf24
Fri Nov 13 19:56:37 2020 kern.warn kernel: [  600.607311] [<80088568>] 0x80088568
Fri Nov 13 19:56:37 2020 kern.warn kernel: [  600.614240] [<8005f214>] 0x8005f214
Fri Nov 13 19:56:37 2020 kern.warn kernel: [  600.621174] [<80088824>] 0x80088824
Fri Nov 13 19:56:37 2020 kern.warn kernel: [  600.628102] [<80079158>] 0x80079158
Fri Nov 13 19:56:37 2020 kern.warn kernel: [  600.635039] [<804a3658>] 0x804a3658
Fri Nov 13 19:56:37 2020 kern.warn kernel: [  600.641971] [<80032fb4>] 0x80032fb4
Fri Nov 13 19:56:37 2020 kern.warn kernel: [  600.648898] [<8025a5f0>] 0x8025a5f0
Fri Nov 13 19:56:37 2020 kern.warn kernel: [  600.655827] [<80007488>] 0x80007488
Fri Nov 13 19:56:37 2020 kern.warn kernel: [  600.662755]
Fri Nov 13 19:56:37 2020 kern.warn kernel: [  600.665905] ---[ end trace 78f661a8159a772d ]---
Fri Nov 13 19:56:38 2020 kern.err kernel: [  600.675119] mtk_soc_eth 1e100000.ethernet eth0: transmit timed out
Fri Nov 13 19:56:38 2020 kern.info kernel: [  600.687465] mtk_soc_eth 1e100000.ethernet eth0: dma_cfg:80000065
Fri Nov 13 19:56:38 2020 kern.info kernel: [  600.699465] mtk_soc_eth 1e100000.ethernet eth0: tx_ring=0, base=071c0000, max=0, ctx=4057, dtx=4057, fdx=3895, next=4057
Fri Nov 13 19:56:38 2020 kern.info kernel: [  600.721137] mtk_soc_eth 1e100000.ethernet eth0: rx_ring=0, base=06160000, max=0, calc=3266, drx=3314
Fri Nov 13 19:56:38 2020 kern.info kernel: [  600.766136] mtk_soc_eth 1e100000.ethernet: 0x100 = 0x6060000c, 0x10c = 0x80818
Fri Nov 13 19:56:38 2020 kern.info kernel: [  600.786493] mtk_soc_eth 1e100000.ethernet: PPE started
Fri Nov 13 19:58:11 2020 kern.warn kernel: [  693.872519] vlan egress tag control neither untag nor tag.
Fri Nov 13 19:58:33 2020 kern.warn kernel: [  716.202205] vlan egress tag control neither untag nor tag.

Netgear R6220 with openwrt 19.07.4 Hardware Offloading set
Tela marinera si este firm esta en la rama estable.

#40

Thanks for all your investigations into this, and documenting it all clearly.

I was wondering if anyone knew how to force a particular port to a set speed and negotiation state?

What I want to do is this, but that does not work on 19.07.4 / MT7621

swconfig dev switch0 port 4 set link "speed 100 duplex full autoneg off"

Or an equivalent via UCI.
#41

I don't think the driver in its current form allows you to force a specific speed at runtime. You'd have to compile it in as a hardcoded value, or add the functionality to the driver to be able to set this at runtime.

#42

Thanks, that's what I feared would be the case.

#43

A Archer C50v1 causes network issues of a r8169 here, both are connected to the same 5 port TP-Link switch:

[3278753.848768] r8169 0000:03:00.0 enp3s0: rtl_txcfg_empty_cond == 0 (loop: 666, delay: 100).
[3278763.848268] r8169 0000:03:00.0 enp3s0: rtl_txcfg_empty_cond == 0 (loop: 666, delay: 100).

An other Archer C50v1 which is only connected to a Pi2 does not create these issues. So, I'm wondering if this issue is also related.

#44

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.