NETDEV WATCHDOG on MT7688AN Soc Board when trying to network boot

Hi,

I'm trying to perform a network boot on our MT7688AN SoC based board and U-Boot can TFTP the kernel and boot it but the kernel cannot mount the NFS root filesystem, it looks like the initial DHCP packets get stuck.

[    0.000000] Linux version 4.14.179 (martin@beeker) (gcc version 7.5.0 (OpenWrt GCC 7.5.0 r11050-a8c92e9)) #0 Fri May 8 18:35:50 2020
[    0.000000] Board has DDR2
[    0.000000] Analog PMU set to hw control
[    0.000000] Digital PMU set to hw control
[    0.000000] SoC Type: MediaTek MT7688 ver:1 eco:2
[    0.000000] bootconsole [early0] enabled
[    0.000000] CPU0 revision is: 00019655 (MIPS 24KEc)
[    0.000000] MIPS: machine is Octopuck
[    0.000000] Determined physical RAM map:
[    0.000000]  memory: 08000000 @ 00000000 (usable)
[    0.000000] User-defined physical RAM map:
[    0.000000]  memory: 08000000 @ 00000000 (usable)
[    0.000000] Initrd not found or empty - disabling initrd
[    0.000000] Primary instruction cache 64kB, VIPT, 4-way, linesize 32 bytes.
[    0.000000] Primary data cache 32kB, 4-way, PIPT, no aliases, linesize 32 bytes
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x0000000000000000-0x0000000007ffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x0000000007ffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x0000000007ffffff]
[    0.000000] random: get_random_bytes called from start_kernel+0x90/0x478 with crng_init=0
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 32512
[    0.000000] Kernel command line:  console=ttyS1,57600 root=/dev/nfs nfsroot=192.168.0.210:/export/octave-octopuck-mt7688an-devel-rootfs,nolock,v3 rw ip=dhcp mtdoops.mtddev= loglevel=7 mem=128M
[    0.000000] PID hash table entries: 512 (order: -1, 2048 bytes)
[    0.000000] Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
[    0.000000] Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
[    0.000000] Writing ErrCtl register=0007a005
[    0.000000] Readback ErrCtl register=0007a005
[    0.000000] Memory: 123276K/131072K available (4015K kernel code, 177K rwdata, 940K rodata, 1196K init, 213K bss, 7796K reserved, 0K cma-reserved)
[    0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] NR_IRQS: 256
[    0.000000] intc: using register map from devicetree
[    0.000000] CPU Clock: 580MHz
[    0.000000] timer_probe: no matching timers found
[    0.000000] clocksource: MIPS: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 6590553264 ns
[    0.000011] sched_clock: 32 bits at 290MHz, resolution 3ns, wraps every 7405115902ns
[    0.015376] Calibrating delay loop... 385.84 BogoMIPS (lpj=1929216)
[    0.087589] pid_max: default: 32768 minimum: 301
[    0.096946] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.109910] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.129482] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.148859] futex hash table entries: 256 (order: -1, 3072 bytes)
[    0.160962] pinctrl core: initialized pinctrl subsystem
[    0.172249] NET: Registered protocol family 16
[    0.184209] rt2880-pinmux pinctrl: error claiming hogs: -22
[    0.195150] rt2880-pinmux pinctrl: could not claim hogs: -22
[    0.206441] rt2880-pinmux: probe of pinctrl failed with error -22
[    0.245702] mt7621_gpio 10000600.gpio: registering 32 gpios
[    0.256922] mt7621_gpio 10000600.gpio: registering 32 gpios
[    0.268047] mt7621_gpio 10000600.gpio: registering 32 gpios
[    0.283742] clocksource: Switched to clocksource MIPS
[    0.294867] NET: Registered protocol family 2
[    0.304263] TCP established hash table entries: 1024 (order: 0, 4096 bytes)
[    0.317967] TCP bind hash table entries: 1024 (order: 0, 4096 bytes)
[    0.330485] TCP: Hash tables configured (established 1024 bind 1024)
[    0.343189] UDP hash table entries: 256 (order: 0, 4096 bytes)
[    0.354658] UDP-Lite hash table entries: 256 (order: 0, 4096 bytes)
[    0.367268] NET: Registered protocol family 1
[    0.378521] RPC: Registered named UNIX socket transport module.
[    0.390158] RPC: Registered udp transport module.
[    0.399392] RPC: Registered tcp transport module.
[    0.408680] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    0.422684] Crashlog allocated RAM at address 0x3f00000
[    0.434857] workingset: timestamp_bits=30 max_order=15 bucket_order=0
[    0.453487] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    0.474097] jffs2: version 2.2 (NAND) (SUMMARY) (LZMA) (RTIME) (CMODE_PRIORITY) (c) 2001-2006 Red Hat, Inc.
[    0.496108] io scheduler noop registered
[    0.503784] io scheduler deadline registered (default)
[    0.514768] gdma-rt2880 10002800.gdma: revision: 3, channels: 16
[    0.528456] Serial: 8250/16550 driver, 3 ports, IRQ sharing disabled
[    0.541787] of_serial 10000c00.uartlite: could not find pctldev for node /pinctrl/uartlite, deferring probe
[    0.561024] of_serial 10000d00.uart1: could not find pctldev for node /pinctrl/uart1, deferring probe
[    0.580105] spi-mt7621 10000b00.spi: could not find pctldev for node /pinctrl/spi_pins, deferring probe
[    0.599215] libphy: Fixed MDIO Bus: probed
[    0.623196] rt3050-esw 10110000.esw: link changed 0x00
[    0.634210] mtk_soc_eth 10100000.ethernet: generated random MAC address 66:c3:43:de:8c:a0
[    0.651000] mtk_soc_eth 10100000.ethernet eth0: mediatek frame engine at 0xb0100000, irq 5
[    0.669101] NET: Registered protocol family 10
[    0.682000] Segment Routing with IPv6
[    0.689295] NET: Registered protocol family 17
[    0.698014] 8021q: 802.1Q VLAN Support v1.8
[    0.706989] of_serial 10000c00.uartlite: could not find pctldev for node /pinctrl/uartlite, deferring probe
[    0.726284] of_serial 10000d00.uart1: could not find pctldev for node /pinctrl/uart1, deferring probe
[    0.744496] spi-mt7621 10000b00.spi: could not find pctldev for node /pinctrl/spi_pins, deferring probe
[    0.793890] Sending DHCP requests .
[    2.643739] random: fast init done
[    3.153734] .
[    6.313734] ------------[ cut here ]------------
[    6.325858] WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:320 dev_watchdog+0x164/0x274
[    6.342166] NETDEV WATCHDOG: eth0 (mtk_soc_eth): transmit queue 0 timed out
[    6.355924] Modules linked in:
[    6.361944] CPU: 0 PID: 0 Comm: swapper Not tainted 4.14.179 #0
[    6.373642] Stack : 80500000 804b2014 00000000 00000000 804882f8 87c07e34 804dfd4c 804df967
[    6.390151]         80484340 00000000 80633660 00000140 804f3e94 80630000 87c07de8 f7cf2cff
[    6.406662]         00000000 00000000 80630000 00001e68 00000000 00000065 00000007 00000000
[    6.423173]         00000000 804e0000 000585d8 00000000 00000000 00000009 00000000 804aadf4
[    6.439686]         802d69d0 00000140 804f3e94 804f3e54 00000002 804e0000 00000000 80630000
[    6.456198]         ...
[    6.461006] Call Trace:
[    6.465856] [<8000af90>] show_stack+0x58/0x100
[    6.474616] [<80026990>] __warn+0xe4/0x13c
[    6.482685] [<80026a18>] warn_slowpath_fmt+0x30/0x3c
[    6.492504] [<802d69d0>] dev_watchdog+0x164/0x274
[    6.501791] [<800603d0>] call_timer_fn.isra.26+0x24/0x84
[    6.512281] [<800605a4>] run_timer_softirq+0x174/0x1ec
[    6.522444] [<803eab90>] __do_softirq+0xe8/0x2bc
[    6.531547] [<80005f90>] except_vec_vi_end+0xb8/0xc4
[    6.541353] [<8000790c>] r4k_wait_irqoff+0x18/0x24
[    6.550809] ---[ end trace 3cef26ae9dfb1ebf ]---
[    6.559930] mtk_soc_eth 10100000.ethernet eth0: transmit timed out
[    6.572136] mtk_soc_eth 10100000.ethernet eth0: dma_cfg:00000057
[    6.584012] mtk_soc_eth 10100000.ethernet eth0: tx_ring=0, base=07ddc000, max=1024, ctx=2, dtx=1, fdx=1, next=2
[    6.603962] mtk_soc_eth 10100000.ethernet eth0: rx_ring=0, base=07fd8000, max=1024, calc=1023, drx=0
[    7.273731] .
[   12.713748] mtk_soc_eth 10100000.ethernet eth0: transmit timed out
[   12.728946] mtk_soc_eth 10100000.ethernet eth0: dma_cfg:00000057
[   12.740816] mtk_soc_eth 10100000.ethernet eth0: tx_ring=0, base=07cd4000, max=1024, ctx=3, dtx=0, fdx=0, next=3
[   12.760767] mtk_soc_eth 10100000.ethernet eth0: rx_ring=0, base=07ee8000, max=1024, calc=1023, drx=0
[   14.473730] .
[   24.233747] mtk_soc_eth 10100000.ethernet eth0: transmit timed out
[   24.248945] mtk_soc_eth 10100000.ethernet eth0: dma_cfg:00000057
[   24.260814] mtk_soc_eth 10100000.ethernet eth0: tx_ring=0, base=07ddc000, max=1024, ctx=3, dtx=0, fdx=0, next=3
[   24.280766] mtk_soc_eth 10100000.ethernet eth0: rx_ring=0, base=07f68000, max=1024, calc=1023, drx=0
[   27.053730] .
[   34.153749] mtk_soc_eth 10100000.ethernet eth0: transmit timed out
[   34.168949] mtk_soc_eth 10100000.ethernet eth0: dma_cfg:00000057
[   34.180819] mtk_soc_eth 10100000.ethernet eth0: tx_ring=0, base=07cd4000, max=1024, ctx=1, dtx=0, fdx=0, next=1
[   34.200770] mtk_soc_eth 10100000.ethernet eth0: rx_ring=0, base=07ee8000, max=1024, calc=1023, drx=0

I've read a few posts about disable flow control but they are for the GSW whereas it looks like this board uses the ESW. Some other posts mention configuring the switch using swconfig but I can't do this either. Maybe there is something I need to include in the switch code though as I see

rt3050-esw 10110000.esw: link changed 0x00

in the boot log but no other link change events. Also is the base address correct for the line

[    0.651000] mtk_soc_eth 10100000.ethernet eth0: mediatek frame engine at 0xb0100000, irq 5

I thought it would be 0x10100000.

All I have in my device tree for the Ethernet is

&ethernet {
	status = "okay";
	/* 1 is LAN, 0 is WAN */
	/*mediatek,portmap = "0x3e";*/
	/*mtd-mac-address = <&factory 0x28>;*/
};

I think the ESW is enabled by default. My understanding from the driver code is that the portmap is initialised so all ports are LAN, not sure if this disables PAUSE frames or not.

From a HW perspective the ethernet is connected to Port 0 and is a single port ethernet device.

Any help appreciated,
Martin.

This is a unfinished project in trunk version,please switch to stable version,19.07,etc.