Mtk_soc_eth watchdog timeout after r11573

I have temporarily configured syslog in flash, to see what happens when the Switch hangs. And this is what I found after manual reboot.

Mon Dec 14 10:18:19 2020 kern.warn kernel: [  270.132232] br-lan: received packet on eth0.1 with own address as source address (addr:80:2a:a8:xx:xx:xx, vlan:0)
Mon Dec 14 10:18:26 2020 kern.warn kernel: [  277.141007] ------------[ cut here ]------------
Mon Dec 14 10:18:26 2020 kern.warn kernel: [  277.150244] WARNING: CPU: 1 PID: 0 at net/sched/sch_generic.c:320 dev_watchdog+0x1ac/0x324
Mon Dec 14 10:18:26 2020 kern.info kernel: [  277.166739] NETDEV WATCHDOG: eth0 (mtk_soc_eth): transmit queue 0 timed out
Mon Dec 14 10:18:26 2020 kern.warn kernel: [  277.180658] Modules linked in: pppoe ppp_async pppox ppp_generic nf_nat_pptp nf_conntrack_pptp nf_conntrack_ipv6 iptable_nat ipt_REJECT ipt_MASQUERADE xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_recent xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_helper xt_ecn xt_dscp xt_conntrack xt_connmark xt_connlimit xt_connbytes xt_comment xt_TCPMSS xt_REDIRECT xt_LOG xt_HL xt_FLOWOFFLOAD xt_DSCP xt_CT xt_CLASSIFY ts_fsm ts_bm slhc nf_reject_ipv4 nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_rtsp nf_nat_redirect nf_nat_proto_gre nf_nat_masquerade_ipv4 nf_nat_irc nf_conntrack_ipv4 nf_nat_ipv4 nf_nat_h323 nf_nat_ftp nf_nat_amanda nf_nat nf_log_ipv4 nf_flow_table_hw nf_flow_table nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_tftp nf_conntrack_snmp nf_conntrack_sip nf_conntrack_rtsp nf_conntrack_rtcache
Mon Dec 14 10:18:26 2020 kern.warn kernel: [  277.324460]  nf_conntrack_proto_gre nf_conntrack_netlink nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp nf_conntrack_broadcast ts_kmp nf_conntrack_amanda nf_conntrack iptable_raw iptable_mangle iptable_filter ipt_ECN ip_tables crc_ccitt xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 tun nls_utf8 nls_iso8859_15 nls_cp852 nls_cp850 nls_cp437 nls_base leds_gpio gpio_button_hotplug
Mon Dec 14 10:18:26 2020 kern.warn kernel: [  277.455322] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 4.14.167 #0
Mon Dec 14 10:18:26 2020 kern.warn kernel: [  277.467450] Stack : 00000000 8fead540 80580000 8007265c 805a0000 80546510 00000000 00000000
Mon Dec 14 10:18:26 2020 kern.warn kernel: [  277.484085]         80512100 8fc0bdc4 8fc3c33c 805808e7 8050cef0 00000001 8fc0bd68 53261646
Mon Dec 14 10:18:26 2020 kern.warn kernel: [  277.500718]         00000000 00000000 806e0000 00004490 00000000 000000e7 00000008 00000000
Mon Dec 14 10:18:26 2020 kern.warn kernel: [  277.517350]         00000000 80580000 0006f29a 00000000 00000000 805a0000 00000000 80540718
Mon Dec 14 10:18:26 2020 kern.warn kernel: [  277.533982]         80370050 00000140 00000001 8fead540 00000000 80299210 00000004 806e0004
Mon Dec 14 10:18:26 2020 kern.warn kernel: [  277.550613]         ...
Mon Dec 14 10:18:26 2020 kern.warn kernel: [  277.555475] Call Trace:
Mon Dec 14 10:18:26 2020 kern.warn kernel: [  277.560366] [<8000c7b0>] show_stack+0x58/0x100
Mon Dec 14 10:18:26 2020 kern.warn kernel: [  277.569229] [<8044f8c4>] dump_stack+0xa4/0xe0
Mon Dec 14 10:18:26 2020 kern.warn kernel: [  277.577905] [<8002f5f8>] __warn+0xe0/0x138
Mon Dec 14 10:18:26 2020 kern.warn kernel: [  277.586052] [<8002f680>] warn_slowpath_fmt+0x30/0x3c
Mon Dec 14 10:18:26 2020 kern.warn kernel: [  277.595938] [<80370050>] dev_watchdog+0x1ac/0x324
Mon Dec 14 10:18:26 2020 kern.warn kernel: [  277.605316] [<8008932c>] call_timer_fn.isra.25+0x24/0x84
Mon Dec 14 10:18:26 2020 kern.warn kernel: [  277.615883] [<800895e8>] run_timer_softirq+0x1bc/0x248
Mon Dec 14 10:18:26 2020 kern.warn kernel: [  277.626126] [<8046d770>] __do_softirq+0x128/0x2ec
Mon Dec 14 10:18:26 2020 kern.warn kernel: [  277.635491] [<80033f84>] irq_exit+0xac/0xc8
Mon Dec 14 10:18:26 2020 kern.warn kernel: [  277.643829] [<8024c1c0>] plat_irq_dispatch+0xfc/0x138
Mon Dec 14 10:18:26 2020 kern.warn kernel: [  277.653880] [<80007588>] except_vec_vi_end+0xb8/0xc4
Mon Dec 14 10:18:26 2020 kern.warn kernel: [  277.663755] [<80008f50>] r4k_wait_irqoff+0x1c/0x24
Mon Dec 14 10:18:26 2020 kern.warn kernel: [  277.673449] ---[ end trace 77cac57743d47a00 ]---
Mon Dec 14 10:18:26 2020 kern.err kernel: [  277.682707] mtk_soc_eth 1e100000.ethernet eth0: transmit timed out
Mon Dec 14 10:18:26 2020 kern.info kernel: [  277.695106] mtk_soc_eth 1e100000.ethernet eth0: dma_cfg:80000067
Mon Dec 14 10:18:26 2020 kern.info kernel: [  277.707162] mtk_soc_eth 1e100000.ethernet eth0: tx_ring=0, base=0ed50000, max=0, ctx=3950, dtx=3761, fdx=3761, next=3950
Mon Dec 14 10:18:26 2020 kern.info kernel: [  277.728879] mtk_soc_eth 1e100000.ethernet eth0: rx_ring=0, base=0dcf0000, max=0, calc=391, drx=392
Mon Dec 14 10:18:27 2020 kern.info kernel: [  278.164010] mtk_soc_eth 1e100000.ethernet: 0x100 = 0x3c60180c, 0x10c = 0x80818
Mon Dec 14 10:18:27 2020 kern.info kernel: [  278.178416] mtk_soc_eth 1e100000.ethernet: reset pse
Mon Dec 14 10:18:27 2020 kern.info kernel: [  278.203701] mtk_soc_eth 1e100000.ethernet: PPE started
Mon Dec 14 10:18:38 2020 kern.err kernel: [  289.218283] mtk_soc_eth 1e100000.ethernet eth0: transmit timed out
Mon Dec 14 10:18:38 2020 kern.info kernel: [  289.230629] mtk_soc_eth 1e100000.ethernet eth0: dma_cfg:80000067
Mon Dec 14 10:18:38 2020 kern.info kernel: [  289.242623] mtk_soc_eth 1e100000.ethernet eth0: tx_ring=0, base=0dcf0000, max=0, ctx=3072, dtx=0, fdx=0, next=3072
Mon Dec 14 10:18:38 2020 kern.info kernel: [  289.263289] mtk_soc_eth 1e100000.ethernet eth0: rx_ring=0, base=0f0b0000, max=0, calc=3079, drx=3080
Mon Dec 14 10:18:38 2020 kern.info kernel: [  289.687138] mtk_soc_eth 1e100000.ethernet: 0x100 = 0x3c60180c, 0x10c = 0x80818
Mon Dec 14 10:18:38 2020 kern.info kernel: [  289.701548] mtk_soc_eth 1e100000.ethernet: reset pse
Mon Dec 14 10:18:38 2020 kern.info kernel: [  289.726294] mtk_soc_eth 1e100000.ethernet: PPE started
Mon Dec 14 10:18:48 2020 kern.err kernel: [  299.136375] mtk_soc_eth 1e100000.ethernet eth0: transmit timed out
Mon Dec 14 10:18:48 2020 kern.info kernel: [  299.148711] mtk_soc_eth 1e100000.ethernet eth0: dma_cfg:80000067
Mon Dec 14 10:18:48 2020 kern.info kernel: [  299.160707] mtk_soc_eth 1e100000.ethernet eth0: tx_ring=0, base=0f140000, max=0, ctx=3072, dtx=0, fdx=0, next=3072
Mon Dec 14 10:18:48 2020 kern.info kernel: [  299.181370] mtk_soc_eth 1e100000.ethernet eth0: rx_ring=0, base=0f3d0000, max=0, calc=1581, drx=1582
Mon Dec 14 10:18:48 2020 kern.info kernel: [  299.605233] mtk_soc_eth 1e100000.ethernet: 0x100 = 0x3c60180c, 0x10c = 0x80818
Mon Dec 14 10:18:48 2020 kern.info kernel: [  299.619649] mtk_soc_eth 1e100000.ethernet: reset pse
Mon Dec 14 10:18:48 2020 kern.info kernel: [  299.644538] mtk_soc_eth 1e100000.ethernet: PPE started
Mon Dec 14 10:18:58 2020 kern.err kernel: [  309.134745] mtk_soc_eth 1e100000.ethernet eth0: transmit timed out
Mon Dec 14 10:18:58 2020 kern.info kernel: [  309.147088] mtk_soc_eth 1e100000.ethernet eth0: dma_cfg:80000067
Mon Dec 14 10:18:58 2020 kern.info kernel: [  309.159066] mtk_soc_eth 1e100000.ethernet eth0: tx_ring=0, base=0eb20000, max=0, ctx=3072, dtx=0, fdx=0, next=3072
Mon Dec 14 10:18:58 2020 kern.info kernel: [  309.179711] mtk_soc_eth 1e100000.ethernet eth0: rx_ring=0, base=0d330000, max=0, calc=2066, drx=2067
Mon Dec 14 10:18:58 2020 kern.info kernel: [  309.603558] mtk_soc_eth 1e100000.ethernet: 0x100 = 0x3c60180c, 0x10c = 0x80818
Mon Dec 14 10:18:58 2020 kern.info kernel: [  309.617975] mtk_soc_eth 1e100000.ethernet: reset pse
Mon Dec 14 10:18:58 2020 kern.info kernel: [  309.642755] mtk_soc_eth 1e100000.ethernet: PPE started
Mon Dec 14 10:19:08 2020 kern.err kernel: [  319.133366] mtk_soc_eth 1e100000.ethernet eth0: transmit timed out
Mon Dec 14 10:19:08 2020 kern.info kernel: [  319.145733] mtk_soc_eth 1e100000.ethernet eth0: dma_cfg:80000067
Mon Dec 14 10:19:08 2020 kern.info kernel: [  319.157718] mtk_soc_eth 1e100000.ethernet eth0: tx_ring=0, base=0ba80000, max=0, ctx=3072, dtx=0, fdx=0, next=3072
Mon Dec 14 10:19:08 2020 kern.info kernel: [  319.178390] mtk_soc_eth 1e100000.ethernet eth0: rx_ring=0, base=0b840000, max=0, calc=2076, drx=2077
Mon Dec 14 10:19:08 2020 kern.info kernel: [  319.602157] mtk_soc_eth 1e100000.ethernet: 0x100 = 0x3c60180c, 0x10c = 0x80818
Mon Dec 14 10:19:08 2020 kern.info kernel: [  319.616612] mtk_soc_eth 1e100000.ethernet: reset pse
Mon Dec 14 10:19:08 2020 kern.info kernel: [  319.641355] mtk_soc_eth 1e100000.ethernet: PPE started
Mon Dec 14 10:19:18 2020 kern.err kernel: [  329.132170] mtk_soc_eth 1e100000.ethernet eth0: transmit timed out
Mon Dec 14 10:19:18 2020 kern.info kernel: [  329.144511] mtk_soc_eth 1e100000.ethernet eth0: dma_cfg:80000067
Mon Dec 14 10:19:18 2020 kern.info kernel: [  329.156508] mtk_soc_eth 1e100000.ethernet eth0: tx_ring=0, base=0eac0000, max=0, ctx=3072, dtx=0, fdx=0, next=3072
Mon Dec 14 10:19:18 2020 kern.info kernel: [  329.177172] mtk_soc_eth 1e100000.ethernet eth0: rx_ring=0, base=0d040000, max=0, calc=2073, drx=2074
Mon Dec 14 10:19:18 2020 kern.info kernel: [  329.600996] mtk_soc_eth 1e100000.ethernet: 0x100 = 0x3c60180c, 0x10c = 0x80818
Mon Dec 14 10:19:18 2020 kern.info kernel: [  329.615412] mtk_soc_eth 1e100000.ethernet: reset pse
Mon Dec 14 10:19:18 2020 kern.info kernel: [  329.640025] mtk_soc_eth 1e100000.ethernet: PPE started
Mon Dec 14 10:19:28 2020 kern.err kernel: [  339.131152] mtk_soc_eth 1e100000.ethernet eth0: transmit timed out
Mon Dec 14 10:19:28 2020 kern.info kernel: [  339.143492] mtk_soc_eth 1e100000.ethernet eth0: dma_cfg:80000067
Mon Dec 14 10:19:28 2020 kern.info kernel: [  339.155492] mtk_soc_eth 1e100000.ethernet eth0: tx_ring=0, base=0f250000, max=0, ctx=3072, dtx=0, fdx=0, next=3072
Mon Dec 14 10:19:28 2020 kern.info kernel: [  339.176157] mtk_soc_eth 1e100000.ethernet eth0: rx_ring=0, base=0f1a0000, max=0, calc=2071, drx=2072
Mon Dec 14 10:19:28 2020 kern.info kernel: [  339.599992] mtk_soc_eth 1e100000.ethernet: 0x100 = 0x3c60180c, 0x10c = 0x80818
Mon Dec 14 10:19:28 2020 kern.info kernel: [  339.614404] mtk_soc_eth 1e100000.ethernet: reset pse
Mon Dec 14 10:19:28 2020 kern.info kernel: [  339.638988] mtk_soc_eth 1e100000.ethernet: PPE started
Mon Dec 14 10:19:38 2020 kern.err kernel: [  349.130285] mtk_soc_eth 1e100000.ethernet eth0: transmit timed out
Mon Dec 14 10:19:38 2020 kern.info kernel: [  349.142628] mtk_soc_eth 1e100000.ethernet eth0: dma_cfg:80000067
Mon Dec 14 10:19:38 2020 kern.info kernel: [  349.154628] mtk_soc_eth 1e100000.ethernet eth0: tx_ring=0, base=0f180000, max=0, ctx=3072, dtx=0, fdx=0, next=3072
Mon Dec 14 10:19:38 2020 kern.info kernel: [  349.175289] mtk_soc_eth 1e100000.ethernet eth0: rx_ring=0, base=0f240000, max=0, calc=2092, drx=2093
Mon Dec 14 10:19:38 2020 kern.info kernel: [  349.599117] mtk_soc_eth 1e100000.ethernet: 0x100 = 0x3c60180c, 0x10c = 0x80818
Mon Dec 14 10:19:38 2020 kern.info kernel: [  349.613538] mtk_soc_eth 1e100000.ethernet: reset pse
Mon Dec 14 10:19:38 2020 kern.info kernel: [  349.638386] mtk_soc_eth 1e100000.ethernet: PPE started
Mon Dec 14 10:19:48 2020 kern.err kernel: [  359.129526] mtk_soc_eth 1e100000.ethernet eth0: transmit timed out
Mon Dec 14 10:19:48 2020 kern.info kernel: [  359.141867] mtk_soc_eth 1e100000.ethernet eth0: dma_cfg:80000067
Mon Dec 14 10:19:48 2020 kern.info kernel: [  359.153848] mtk_soc_eth 1e100000.ethernet eth0: tx_ring=0, base=0d390000, max=0, ctx=3072, dtx=0, fdx=0, next=3072
Mon Dec 14 10:19:48 2020 kern.info kernel: [  359.174460] mtk_soc_eth 1e100000.ethernet eth0: rx_ring=0, base=0f1b0000, max=0, calc=2068, drx=2069
Mon Dec 14 10:19:48 2020 kern.info kernel: [  359.598366] mtk_soc_eth 1e100000.ethernet: 0x100 = 0x3c60180c, 0x10c = 0x80818
Mon Dec 14 10:19:48 2020 kern.info kernel: [  359.612779] mtk_soc_eth 1e100000.ethernet: reset pse
Mon Dec 14 10:19:48 2020 kern.info kernel: [  359.637611] mtk_soc_eth 1e100000.ethernet: PPE started
Mon Dec 14 10:19:58 2020 kern.err kernel: [  369.128902] mtk_soc_eth 1e100000.ethernet eth0: transmit timed out
Mon Dec 14 10:19:58 2020 kern.info kernel: [  369.141239] mtk_soc_eth 1e100000.ethernet eth0: dma_cfg:80000067
Mon Dec 14 10:19:58 2020 kern.info kernel: [  369.153240] mtk_soc_eth 1e100000.ethernet eth0: tx_ring=0, base=0f380000, max=0, ctx=3072, dtx=0, fdx=0, next=3072
Mon Dec 14 10:19:58 2020 kern.info kernel: [  369.173905] mtk_soc_eth 1e100000.ethernet eth0: rx_ring=0, base=0f240000, max=0, calc=2063, drx=2066
Mon Dec 14 10:19:58 2020 kern.info kernel: [  369.597778] mtk_soc_eth 1e100000.ethernet: 0x100 = 0x3c60180c, 0x10c = 0x80818
Mon Dec 14 10:19:58 2020 kern.info kernel: [  369.612214] mtk_soc_eth 1e100000.ethernet: reset pse
Mon Dec 14 10:19:58 2020 kern.info kernel: [  369.636904] mtk_soc_eth 1e100000.ethernet: PPE started
Mon Dec 14 10:20:08 2020 kern.err kernel: [  379.128371] mtk_soc_eth 1e100000.ethernet eth0: transmit timed out
Mon Dec 14 10:20:08 2020 kern.info kernel: [  379.140715] mtk_soc_eth 1e100000.ethernet eth0: dma_cfg:80000067
Mon Dec 14 10:20:08 2020 kern.info kernel: [  379.152714] mtk_soc_eth 1e100000.ethernet eth0: tx_ring=0, base=0c3d0000, max=0, ctx=3072, dtx=0, fdx=0, next=3072
Mon Dec 14 10:20:08 2020 kern.info kernel: [  379.173389] mtk_soc_eth 1e100000.ethernet eth0: rx_ring=0, base=0b9d0000, max=0, calc=2093, drx=2094
Mon Dec 14 10:20:08 2020 kern.info kernel: [  379.597206] mtk_soc_eth 1e100000.ethernet: 0x100 = 0x3c60180c, 0x10c = 0x80818
Mon Dec 14 10:20:08 2020 kern.info kernel: [  379.611623] mtk_soc_eth 1e100000.ethernet: reset pse
Mon Dec 14 10:20:08 2020 kern.info kernel: [  379.636491] mtk_soc_eth 1e100000.ethernet: PPE started
Mon Dec 14 10:20:18 2020 kern.err kernel: [  389.127904] mtk_soc_eth 1e100000.ethernet eth0: transmit timed out
Mon Dec 14 10:20:18 2020 kern.info kernel: [  389.140243] mtk_soc_eth 1e100000.ethernet eth0: dma_cfg:80000067
Mon Dec 14 10:20:18 2020 kern.info kernel: [  389.152242] mtk_soc_eth 1e100000.ethernet eth0: tx_ring=0, base=0b9f0000, max=0, ctx=3072, dtx=0, fdx=0, next=3072
Mon Dec 14 10:20:18 2020 kern.info kernel: [  389.172908] mtk_soc_eth 1e100000.ethernet eth0: rx_ring=0, base=0d040000, max=0, calc=2072, drx=2073
Mon Dec 14 10:20:18 2020 kern.info kernel: [  389.596757] mtk_soc_eth 1e100000.ethernet: 0x100 = 0x3c60180c, 0x10c = 0x80818
Mon Dec 14 10:20:18 2020 kern.info kernel: [  389.611171] mtk_soc_eth 1e100000.ethernet: reset pse
Mon Dec 14 10:20:18 2020 kern.info kernel: [  389.636110] mtk_soc_eth 1e100000.ethernet: PPE started

I thought it was not giving any transmit timed out error while working, only gives it when it freezes and couldn't see the syslog in ram after a reboot. I had never seen the error of the first line where a packet is received from the router itself (?).

It seems that transmit timed out repeats every 10 seconds until i reboot the router.

I have left everything exactly as it was a few months ago, when I had no reboots or crashes. But I get nothing. Also i tried patch from Mushoz (Mt7621 / mt7530 programming: Disabling Flow Control on all ports) to disable pause frames advertisement but without positive result.

I can recreate this situation quickly, just using iperf between LAN - WAN (or two routing interfaces) with no more than 20 threads is enough ...

The truth is that I do not know if the problem has always been there. With a "normal" use of the connection does not happen (even with P2P or large downloads). I can only crash it with iperf.