That's one test, the other is too see what the stock firmware does (see below).
Turns out the router has an USB port, so getting files on and off is easy 
I copied both /sbin/devmem and /lib/ld-musl-mips-sf.so.1 from OpenWRT into the stock firmware (into the proper places - I tried keeping them elsewhere and use LD_LIBRARY_PATH which didn't work, probably because that is used by the dynamic linker to find libraries, not by the kernel to find the linker in the first place).
So the stock firmware actually sets the GMAC delay registers:
# devmem 0x18070000
0x0003C001
This means ETH_RXDV_DELAY=3 and ETH_RXD_DELAY=3, and both of the TX delays at 0.
I will also check the working board to see if that sets different values (but I'm going to guess it doesn't, and that board is just lucky that it works because of subtle changes in silicon between different chip revisions or even different chips).
I also looked in the stock firmware kernel sources to see if they ever set these values, and I couldn't find anything (but maybe this happens in a non-obvious place using hardcoded values instead of proper constants, or maybe it happens in some script rather than the kernel.., who knows...).
Checking the pll register value in the stock firmware shows it also matches the OpenWRT value:
# devmem 0x1805002C
0x06000000
(for reference, the 0x2C register offset comes from the pll reg attribute plus the second element of the eth0 pll-reg attribute, the pll-data defined in the OpenWRT DTS is for Gbps, 100Mbps, 10Mbps respectively, and this is an internal link that is always 1Gbps, so only the first value, 0x06000000, is relevant)
Looking more closely at the kernel code, it seems the ethreg global register reading code (triggered by -p 15, but failing) actually almost matches the QCA8337 datasheet (except for some magic numbers)ull, but the port-specific PHY register reading code (which does seem to work) does not match the datasheets (it uses register 0x98 for accessing the PHY-port registers, but that is the HEADER_CTRL register according to the datasheet.
I guess I might be looking at the wrong bit of code, or the sources I'm looking at are not the right sources for this board. So I'm going to be slightly distrustful of the ethreg output. Also, this means I cannot really verify whether the stock firmware sets up any (TX) delays in the PHY.
I've also tried bitbanging the right MII registers based on the 9344 and 8337 datasheets and current kernel sources using devmem, which seems to work. Here's the script for that, in case it's useful for anyone else:
# readreg.sh
# This script reads QCA8337 (and compatible) switch global registers
# through AR9344 (and compatible) MII control registers by poking directly
# in memory. This is probably slightly dangerous and will probably crash
# things if the kernel also happens to access these registers at the same
# time.
REG=$1
devmem 0x19000028 32 0x1800 # MII_ADDR
devmem 0x1900002C 32 $((REG >> 9)) # MII_CTRL
PHYNUM=$(((REG >> 6) & 0x7 | 0x10))
PHYREG=$(((REG >> 1) & 0x1e))
devmem 0x19000024 32 0 # MII_CMD
devmem 0x19000028 32 $((PHYNUM << 8 | PHYREG)) # MII_ADDR
devmem 0x19000024 32 1 # MII_CMD
LO=$(devmem 0x19000032 16) # MII_STATUS + 2
PHYREG=$((PHYREG | 0x1))
devmem 0x19000024 32 0 # MII_CMD
devmem 0x19000028 32 $((PHYNUM << 8 | PHYREG)) # MII_ADDR
devmem 0x19000024 32 1 # MII_CMD
HI=$(devmem 0x19000032 16) # MII_STATUS + 2
printf "0x%08x = 0x%04x%04x\n" "$REG" "$HI" "$LO"
I verified it works by reading rregister 0x0 (MASK_CTRL), bits 15:8 are the device ID which should be 0x13:
# ./readreg.sh 0x0
0x00000000 = 0x00001302
Here's a full dump of the QCA8337 registers in the stock firmware on the "broken" board:
# for r in 0 4 8 12 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76 80 84 88 92 96 100 104 108 112 116 120 124 128 132 136 140 144 148 152 156 160 164 168 172 176 180 184 188 192 196 200 204 208 212 216 220 224; do ./readreg.sh $r; done
0x00000000 = 0x00001302
0x00000004 = 0x87a00000
0x00000008 = 0x01000000
0x0000000c = 0x01000000
0x00000010 = 0x80000000
0x00000014 = 0xf0107650
0x00000018 = 0x0000a980
0x0000001c = 0x00003f1f
0x00000020 = 0x3f170a00
0x00000024 = 0x000100ff
0x00000028 = 0x00000000
0x0000002c = 0x00000000
0x00000030 = 0x80000304
0x00000034 = 0x00000000
0x00000038 = 0x0f000000
0x0000003c = 0x00000000
0x00000040 = 0x20700aaa
0x00000044 = 0x00000000
0x00000048 = 0x000088a8
0x0000004c = 0x00000000
0x00000050 = 0xcf35cf35
0x00000054 = 0xcf35cf35
0x00000058 = 0xcf35cf35
0x0000005c = 0x03ffff00
0x00000060 = 0x00000001
0x00000064 = 0x00000000
0x00000068 = 0x00000000
0x0000006c = 0x00000000
0x00000070 = 0xb00ee060
0x00000074 = 0x03707f07
0x00000078 = 0x00002400
0x0000007c = 0x0000007e
0x00000080 = 0x80001ffd
0x00000084 = 0x00001280
0x00000088 = 0x00001280
0x0000008c = 0x00001280
0x00000090 = 0x00001280
0x00000094 = 0x0000007e
0x00000098 = 0x0001aaaa
0x0000009c = 0x00000002
0x000000a0 = 0x00000000
0x000000a4 = 0x00000000
0x000000a8 = 0x00000000
0x000000ac = 0x00000000
0x000000b0 = 0x00000000
0x000000b4 = 0x00000000
0x000000b8 = 0x00000000
0x000000bc = 0x00000000
0x000000c0 = 0x00000000
0x000000c4 = 0x00000000
0x000000c8 = 0x80901040
0x000000cc = 0x00000000
0x000000d0 = 0xfffbff7e
0x000000d4 = 0x00000001
0x000000d8 = 0x00000100
0x000000dc = 0x000303ff
0x000000e0 = 0xc70164c0
Some interesting observations:
- The port connected to the CPU is probably PORT0, since its CTL register 0x4 has some values, while PORT5_PAD_CTL (0x8) and PORT6_PAD_CTL (0xC) have just one bit set. I couldn't actually find in the OpenWRT DTS file where it is define what switch port is which (even more, I couldn't even find the
compatible = "qca,ar8327" that I think would trigger the right driver for the switch, so maybe there are some hardcoded defaults, or I didn't look in the right places).
- Bit 31 of PORT0_PAD_CTL is set, which means MAC06_EXCHANGE_EN, which exchanges MAC0 and MAC6 (so maybe the hardware is wired to port 6 after all?).
- The OpenWRT dts file sets some of these registers as well, to identical values as the stock firmware (except for bit 8 of the STATUS registers, but that is a read-only link status value so should not make a difference)
- The PORT0_PAD_CTRL register (0x4) has value 0x87a00000 which means
MAC0_RGMII_EN | MAC0_RGMII_TXCLK_DELAY_EN | MAC0_RGMII_RXCLK_DELAY_EN | MAC0_RGMII_TXCLK_DELAY_SEL(2) | MAC0_RGMII_RXCLK_DELAY_SEL (2) (note that the datasheet does not show MAC0_RGMII_RXCLK_DELAY_EN, but according to kernel sources that is bit 24). This means that the PHY is configured to apply delays on TX and RX, but apparently that is not actually enough? Also, OpenWRT applies the same delays (at least they are listed in the dts file), so nothing to be gained there.
- It is actually weird that both the PHY and the GMAC apply a delay for the RX signal. I'm not sure about the units for this delay, but I understand that the goal is 2ns delay, so maybe 2 in the PHY means 2ns? If 3 in the GMAC also means 3ns, then that would add up to 5ns delay, which is more than half the RGMII clock period (8ns), given this is DDR, that would be too much. Weird...
- I suspect that some if not all of the settings in the qca,ar8327-initvals attribute can be more semanticaly set using other DT properties (such as using
phy_mode = "rgmii-rxid" for enabling a RX/TX delay value of 2, or using qca,ignore-power-on-sel to set that one bit in PWS_REG_VALUE, but I'm not sure if this is actually worth the trouble...
For good measure, here's also a dump of the GMAC0 config registers from stock firmware which might or might not be relevant (I left out the second half of the registers, which seem to be mostly status registers and counters, and the registers for GMAC1 which is unused):
# for r in 419430400 419430404 419430408 419430412 419430416 419430420 419430424 419430428 419430432 419430436 419430440 419430444 419430448 419430452 419430456 419430460 419430464 419430468 419430472 419430476 419430480 419430484 419430488 419430492; do printf "%08x = " $r; devmem $r; done
19000000 = 0x0000002F
19000004 = 0x00007215
19000008 = 0x40605060
1900000c = 0x00A1F037
19000010 = 0x00000600
19000014 = 0x00000000
19000018 = 0x00000000
1900001c = 0x00000000
19000020 = 0x0000000B
19000024 = 0x00000000
19000028 = 0x00000411
1900002c = 0x00004007
19000030 = 0x00000010
19000034 = 0x00000000
19000038 = 0x00000000
1900003c = 0x00000008
19000040 = 0x00026FB2
19000044 = 0x10010000
19000048 = 0x001F1F00
1900004c = 0x0010FFFF
19000050 = 0x03FF0155
19000054 = 0x01F00140
19000058 = 0x0003FFFF
1900005c = 0x000E6BE2
Ok, so a lot of things learned. Next steps are:
- Dump registers with stock firmware on my other board, to confirm that uses the same values
- Create an image with the right gmac delays (copied from stock) and test that on both my boards. I'll also publish the image somewhere, would be great if @Dock can then also test on their two boards.
- If this works everywhere, submit the patch to OpenWRT.
