Support for RTL838x based managed switches

These are SFP+ ports, have a look at how port 27 is defined in

things like 10GBit need to be defined explicitly. However, the first thing necessary is to set up i2c. You need to figure out which GPIO pins are used for I2C SDA and SCL (see the reference to the i2c node in the .dts). This should be possible from u-boot. We only have a bit-banged I2C over GPIO driver in OWRT for now, which works nicely on the XGS 12xx devices where the original firmware also uses the bit-banged version. SDK 3 introduced support for a dedicated I2C controller on the 9300. I am not sure what your switch is using, but it should be sufficient to read out the SFP module. However, we need to figure out which GPIO pins are being used. If the I2C controller is used, the configuration is in 0xbb000360 - 039c, register RTL9300_I2C_MST_GLB_CTRL for example allows GPIO pins 9-16 to be used for SDA. SCL seems to be always pin 8. If the firmware uses bit-banging, then maybe u-boot can still show this via the rtk i2c commands.

I just see your edit. This is amazing. You are the first to get an SFP+ port working on these devices. An impressive piece of hacking! And I wanted you to look into i2c...

I'll share my hacky patch, and I haven't tested speeds yet. SFP+ hotplug detection is definitely not working :slight_smile: I'm getting EINVAL (-22) from dsa switch driver for some reason, looking into that first.

dmesg currently is here, still hacking https://pastebin.com/MdQ0P95z

The EINVAL probably comes from an issue with the .dts and is produced by phylink.
If you want to provide a not so hacky patch, then there is already some work done to support MAC to SerDes mapping for the RTL931x here:

The sds property is read out and is used by the ethernet driver code to do port to SDS mapping in the mdio reset routine I mentioned previously. The logic should be the same for the 9300. Brainslayer and I figured the Edgecore out by dumping registers before (done by u-boot) and after the configuration, which need to be identical.

I've hacked up the SWITCH_SFP_PHY to use rxaui (a guess) and a 10000 fixed link, and I have working hotplug on port 1 & 2, but I'm not sure how or why :slight_smile: Link is negotiating at 10GBase-T on copper according to the leds on a different switch.

I get a lot of MDIO errors still in dmesg:

[    0.947579] mdio_bus mdio-bus: MDIO device at address 16 is missing.
[    0.954778] mdio_bus mdio-bus: MDIO device at address 17 is missing.
[    0.961966] mdio_bus mdio-bus: MDIO device at address 18 is missing.
[    0.969170] mdio_bus mdio-bus: MDIO device at address 19 is missing.
[    0.976358] mdio_bus mdio-bus: MDIO device at address 20 is missing.
[    0.983554] mdio_bus mdio-bus: MDIO device at address 21 is missing.
[    0.990758] mdio_bus mdio-bus: MDIO device at address 22 is missing.
[    0.997946] mdio_bus mdio-bus: MDIO device at address 23 is missing.

And hotplug only works on port 1 & 2, for some reason. Current DTS is here (note, mmc partitions are wrong for this device)

Here is the current DTS that sort of kind of works :slight_smile: https://pastebin.com/Mqqnp0NZ
I should note, that like the OEM firmware, I do not see the ports, just the eth0 from port28. Not sure if that's because of lack of DSA or something.

dmesg so far is here: https://pastebin.com/bgGuU3V0

I made a lot of progress, dsa comes up successfully, and reboot works. For some reason, this only occurred after enabling GPIO controller (fixing interrupt to 13 from 31 in dtsi). I am trying to get GPIO working to get I2C working to talk to the GPIO expander, which I think is either part of the i2c-muxer or at least is used to handle GPIOs for the SFPs.

Current hacky patch is here (note it still identifies as a Zyxel, and there's lots to clean up): https://pastebin.com/gtKEADT3

Indeed, good progress!
Regarding the MDIO device missing problems: this is due to the internal PHY that goes with the SerDes not being read. This happens here:

If you look below for the routine for the 931x, you can see that the mapping of port to type of PHY (internal with SerDes or external) is fixed unlike for the rtl931x. You need to add the same way of making this dynamic via the priv->sds_id[mii_id] array and populate the .dts with mapping of port number to SerDes like for the EdgeCore switch, so that rtl930x_read_sds_phy() is being called for all your ports. For understanding: the code uses the same way of addressing a PHY as the SoC does, namely mii_id is the port number, not a PHY address (the SoC itself maps port number to SMI-bus-id / PHY-id via the configuration registers). But the SoC does not know when to talk to the SerDes PHYs, they do not sit on an (emulated) mdio/SMI bus.

Could you post photos of the board. You said there is an I2C muxer and a GPIO expander, it would be interesting to see which and how they are connected to SoC/SFPs. The standard GPIO expander for the RTL-based boards is the RTL8231. Is this there? If you post the I2C configuration registers dumped from within U-boot, then it should be possible to identify the GPIO configuration used for I2C. You should download the SDK from biot's wiki for the xgs1210, which contains the register definitions in swcore_rtl9300.h so that you can read the GPIO configuration from RTL9300_I2C_MST_GLB_CTRL_ADDR.

To get working GPIOs, with interrupts, you will need two things.
Use this node in your DTS (note the reg offset, and indeed the interrupt):

		gpio0: gpio-controller@3300 {
			compatible = "realtek,rtl9300-gpio", "realtek,otto-gpio";
			reg = <0x3300 0x1c>, <0x3338 0x8>;

			gpio-controller;
			#gpio-cells = <2>;
			ngpios = <24>;

			interrupt-controller;
			#interrupt-cells = <2>;
			interrupt-parent = <&intc>;
			interrupts = <13>;
		};

To get everything working, you will then need to drop these three patches into the patches-5.10 directory:

  1. gpio: realtek-otto: Support reversed port layouts
  2. gpio: realtek-otto: Support per-cpu interrupts
  3. gpio: realtek-otto: Add RTL930x support
3 Likes

I've tried cleaning this up quite a bit. I moved the driver and phy code from the brainslayer branch linked above into openwrt (some changes were required, such as max ring count causing a malloc failure), and annotated the DTS. I probably need to carefully review the diff, but I am trying to bring up serdes "correctly" now. I've enabled GPIO using the legacy driver, rather than otto driver as above. Monitoring these 31 GPIOs shows no changes when inserting or removing SFPs.

I'm not sure if this matches our earlier discussion, but here is the annotated DTS: https://pastebin.com/h8GPN5gb. I think the SMI address values are incorrect, but trying various things there seems to have no effect. I also suspect the serdes values. I'll post board pictures when I get a chance. Notably, there is a small IC near each SFP cage, a larger IC I cannot read that is related by traces to the cages, and a PCA9555A that also appears to be related to the cages. The OEM firmware has 8 muxed i2c buses, the first of which appears identical to the i2c bus. I'll see what I can do with reading registers as well there.

[    0.895566] libphy: Fixed MDIO Bus: probed
[    0.902005] rtl83xx-switch switch@1b000000: Error registering switch: -517
[    0.909889] Probing RTL838X eth device pdev: 8146c800, dev: 8146c810
[    0.917033] Allocate 3315364 bytes for DMA
[    0.935341] Found SoC ID: 9303: RTL9303, family 9300
[    0.940884] Using MAC 000000e04c000000
[    0.945074] rtl838x_mdio_init: looking at port 16
[    0.950343] rtl838x_mdio_init: looking at port 17
[    0.955566] rtl838x_mdio_init: looking at port 18
[    0.960790] rtl838x_mdio_init: looking at port 19
[    0.966011] rtl838x_mdio_init: looking at port 20
[    0.971256] rtl838x_mdio_init: looking at port 21
[    0.976484] rtl838x_mdio_init: looking at port 22
[    0.981705] rtl838x_mdio_init: looking at port 23
[    0.987589] c45_mask: 00080000
[    0.991045] libphy: rtl930x-eth-mdio: probed
[    1.005332] mdio_bus mdio-bus: MDIO device at address 16 is missing.
[    1.022002] mdio_bus mdio-bus: MDIO device at address 17 is missing.
[    1.038669] mdio_bus mdio-bus: MDIO device at address 18 is missing.
[    1.055338] mdio_bus mdio-bus: MDIO device at address 19 is missing.
[    1.072006] mdio_bus mdio-bus: MDIO device at address 20 is missing.
[    1.088674] mdio_bus mdio-bus: MDIO device at address 21 is missing.
[    1.105334] mdio_bus mdio-bus: MDIO device at address 22 is missing.
[    1.122009] mdio_bus mdio-bus: MDIO device at address 23 is missing.
[    1.129550] rtl93xx_set_features called
[    1.135096] NET: Registered protocol family 10
[    1.170593] Segment Routing with IPv6
[    1.174777] NET: Registered protocol family 17
[    1.180036] 8021q: 802.1Q VLAN Support v1.8
[    1.198759] rtl930x_setup called
[    1.202378] In rtl83xx_vlan_setup
[    1.206054] In rtl930x_vlan_profile_setup
[    1.210555] In rtl930x_vlan_profile_setup
[    1.215011] UNKNOWN_MC_PMASK: 000000001fffffff
[    1.219945] VLAN 0: L2 learn: 0; Unknown MC PMasks: L2 1fffffff, IPv4 1fffffff, IPv6: 1fffffff
[    1.219952]   Routing enabled: IPv4 UC y, IPv6 UC y, IPv4 MC y, IPv6 MC y
[    1.237087]   Bridge enabled: IPv4 MC n, IPv6 MC n,
[    1.242501] VLAN profile 0: raw 00033000 00000000 1fffffff 1fffffff 1fffffff
[    3.368446] rtl930x_pie_init
[    3.373201] libphy: dsa slave smi: probed
[    3.377767] rtl83xx-switch switch@1b000000: Unsupported interface: 25 for port 1
[    3.386007] rtl83xx-switch switch@1b000000 lan1 (uninitialized): no phy at 1
[    3.393895] rtl83xx-switch switch@1b000000 lan1 (uninitialized): failed to connect to PHY: -ENODEV
[    3.403863] rtl83xx-switch switch@1b000000 lan1 (uninitialized): error -19 setting up PHY for tree 0, switch 0, port 1
[    3.415894] rtl83xx-switch switch@1b000000: Unsupported interface: 25 for port 2
[    3.424142] rtl83xx-switch switch@1b000000 lan2 (uninitialized): no phy at 2
[    3.432017] rtl83xx-switch switch@1b000000 lan2 (uninitialized): failed to connect to PHY: -ENODEV
[    3.441982] rtl83xx-switch switch@1b000000 lan2 (uninitialized): error -19 setting up PHY for tree 0, switch 0, port 2
[    3.454007] rtl83xx-switch switch@1b000000: Unsupported interface: 25 for port 3
[    3.462252] rtl83xx-switch switch@1b000000 lan3 (uninitialized): no phy at 3
[    3.470127] rtl83xx-switch switch@1b000000 lan3 (uninitialized): failed to connect to PHY: -ENODEV
[    3.480092] rtl83xx-switch switch@1b000000 lan3 (uninitialized): error -19 setting up PHY for tree 0, switch 0, port 3
[    3.492116] rtl83xx-switch switch@1b000000: Unsupported interface: 25 for port 4
[    3.500362] rtl83xx-switch switch@1b000000 lan4 (uninitialized): no phy at 4
[    3.508200] rtl83xx-switch switch@1b000000 lan4 (uninitialized): failed to connect to PHY: -ENODEV
[    3.518199] rtl83xx-switch switch@1b000000 lan4 (uninitialized): error -19 setting up PHY for tree 0, switch 0, port 4
[    3.530227] rtl83xx-switch switch@1b000000: Unsupported interface: 25 for port 5
[    3.538473] rtl83xx-switch switch@1b000000 lan5 (uninitialized): no phy at 5
[    3.546309] rtl83xx-switch switch@1b000000 lan5 (uninitialized): failed to connect to PHY: -ENODEV
[    3.556307] rtl83xx-switch switch@1b000000 lan5 (uninitialized): error -19 setting up PHY for tree 0, switch 0, port 5
[    3.568317] rtl83xx-switch switch@1b000000: Unsupported interface: 25 for port 6
[    3.576606] rtl83xx-switch switch@1b000000 lan6 (uninitialized): no phy at 6
[    3.584447] rtl83xx-switch switch@1b000000 lan6 (uninitialized): failed to connect to PHY: -ENODEV
[    3.594432] rtl83xx-switch switch@1b000000 lan6 (uninitialized): error -19 setting up PHY for tree 0, switch 0, port 6
[    3.606439] rtl83xx-switch switch@1b000000: Unsupported interface: 25 for port 7
[    3.614726] rtl83xx-switch switch@1b000000 lan7 (uninitialized): no phy at 7
[    3.622565] rtl83xx-switch switch@1b000000 lan7 (uninitialized): failed to connect to PHY: -ENODEV
[    3.632550] rtl83xx-switch switch@1b000000 lan7 (uninitialized): error -19 setting up PHY for tree 0, switch 0, port 7
[    3.644560] rtl83xx-switch switch@1b000000: Unsupported interface: 25 for port 8
[    3.652845] rtl83xx-switch switch@1b000000 lan8 (uninitialized): no phy at 8
[    3.660685] rtl83xx-switch switch@1b000000 lan8 (uninitialized): failed to connect to PHY: -ENODEV
[    3.670670] rtl83xx-switch switch@1b000000 lan8 (uninitialized): error -19 setting up PHY for tree 0, switch 0, port 8
[    3.682731] rtl83xx-switch switch@1b000000: configuring for fixed/internal link mode
[    3.691673] DSA: tree 0 setup
[    3.695013] LINK state irq: 20
[    3.698427] In rtl83xx_setup_qos
[    3.702058] L3_IPUC_ROUTE_CTRL 00002000, IPMC_ROUTE 0000057e, IP6UC_ROUTE 00014580, IP6MC_ROUTE 00013b7e
[    3.712638] L3_IPUC_ROUTE_CTRL 00002001, IPMC_ROUTE 00000501, IP6UC_ROUTE 00014581, IP6MC_ROUTE 00012881
[    3.723171] L3_IP_ROUTE_CTRL 00000130
[    3.727296] rtl930x_dbgfs_init called
[    3.732588] rtl83xx-switch switch@1b000000: Link is Up - 1Gbps/Full - flow control off

EDIT: added 10GBASER to rt93xx_phy_validate, but it's not clear if that or KR or something else is the more appropriate configuration.

1 Like
md.l 0xbb000360                         
bb000360: 00000000 00000000 00000000 00000000    ................
bb000370: 00005003 00000000 00000000 00000000    ..P.............
bb000380: 00000000 00000000 00000000 00005003    ..............P.
bb000390: 00000000 00000000 00000000 00000000    ................
bb0003a0: 00340000 00000000 00000000 00000000    .4..............
bb0003b0: 00000000 00000000 00000000 00000000    ................
bb0003c0: 00000000 00000000 00000000 00000000    ................
bb0003d0: 00000000 00000000 00000000 00000000    ................
bb0003e0: 00000000 00000000 00000000 00000000    ................
bb0003f0: 00000000 00000000 00000000 00000000    ................
bb000400: 00000000 00000000 00000000 00000000    ................
bb000410: 00000000 00000000 00000000 00000000    ................
bb000420: 00000000 00000000 00000000 00000000    ................
bb000430: 00000000 00000000 00000000 00000000    ................
bb000440: 00000000 00000000 00000000 00000000    ................
bb000450: 00000000 00000000 00000000 00000000    ................

The OEM fw uses the rtk-i2c and rtk-i2c-mux drivers, so it's possible the i2c mux is in the soc, but I'm hoping simple-i2c-mux or similar can be used here

I have i2c-0 working, took a guess at GPIO 8 and 9 (can't find that XGS SDK link)

root@OpenWrt:/# i2cdetect 0
i2cdetect: WARNING! This program can confuse your I2C bus
Continue? [y/N] y
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:          -- -- -- -- -- -- -- -- -- -- -- -- -- 
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
50: 50 51 -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
60: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
70: -- -- -- -- -- -- -- --   

root@OpenWrt:/# i2cdump 0 0x50
i2cdump: WARNING! This program can confuse your I2C bus
Continue? [y/N] y
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: 03 04 07 10 00 00 00 00 00 00 00 06 67 00 00 00    ????.......?g...
10: 08 02 00 1e 4f 45 4d 20 20 20 20 20 20 20 20 20    ??.?OEM         
20: 20 20 20 20 00 00 90 65 53 46 50 2d 31 30 47 2d        ..?eSFP-10G-
30: 53 52 20 20 20 20 20 20 30 32 20 20 03 52 00 70    SR      02  ?R.p
40: 00 1a 00 00 43 53 46 31 30 31 4c 38 30 33 33 37    .?..CSF101L80337
50: 20 20 20 20 32 31 30 38 30 35 20 20 00 80 03 4c        210805  .??L
60: 00 00 11 b9 d8 d7 ad dd 60 fc 70 40 27 62 d3 d3    ..??????`?p@'b??
70: 27 6f a1 00 00 00 00 00 00 00 00 00 c1 b7 8b 32    'o?.........???2
80: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
90: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
c0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................

This looks like the SFP+ does in OEM firmware

1 Like

It does not look as if the I2C IP core is used. Only RTL9300_I2C_MST1_CTRL2_ADDR (0x0370) is set and it is identical to RTL9300_I2C_MST2_CTRL2_ADDR (0x038c). This looks more like a default value after reset (some frequency is set and a device address, whatever that means). rtk-i2c does not necessarily use the i2c IP core, it can also do bit-banging, which is the only option on the rtl83xx SoCs. Bit-Banging allows to use any GPIO pin, not just the few dedicated to i2c. I cannot find any reference to an i2c mux driver in the SDK we have.
I just see your edit, GPIOs 8 and 9 would allow the I2C IP core to be used.

The SMI addresses are not used when the device attaches to an SFP module. The module itself does normally not have a PHY, so it is the SoC internal one that is being used. The phy-address in rtl930x_read_sds_phy() is actually the SerDes ID, and this is set in the .dts via the < sds > property.

Does this DTS look reasonable? Getting ENODEV on every phy still, and ifconfig eth0 up hangs the board.

bootlog: https://pastebin.com/VYJz4c4g

Yes, but the code to pick it up is only there for the 931x, you need to copy this into the 930x functions such as from rtl931x_mdio_read() to rtl930x_mdio_read(). Check directly within rtl930x_read_sds_phy() what values are actually read. Registers 2/3 should contain the strange PHY-ID 0x70d03106.

Ah, I see. It is used in rst code, but not there. Ok, back to hacking :slight_smile:

I found a few bugs here and there, the biggest was brainslayer's 0x86 instead of openwrt's 0x4 in the CPU PORT reset logic (this hard hangs 9303). SDS read from the dts was always 0 as well, but I haven't correctly fixed that yet...just hardcoded and trying to bring up one port. I now have it booting, detecting the phy, seems to work, except no traffic is passing. I'm assuming this might be simple, like SFP gpios missing, or it might be harder, like figuring out what exact protocols the SerDes phy needs to be using to talk to the 10GBase-T (Copper, Marvell) phy in the SFP.

The 0x86 might have been a typo from me, could you try 0x6, this will reset both the switch queues and the NIC, see also the patch submitted by @svanheule on the owrt mailing list for the RTL83xx devices?
If traffic is passing under u-boot, then there are two places this can be messed up: either by writes to the PHY or by incorrectly configuring the SerDes/port configuration in mdio_reset(). For the first you could prevent writes to the PHY by phylink (just disable the sds version of phy writes) and for the second check the registers after the configuration for changes with respect to what u-boot did.

I noticed a possible SDS problem, which is that sds_id does not appear to be read from the DTS properly, and the phy code doesn't get sds_num from anywhere. Hardcoding both, the registers seem to match the uboot values:
[ 1.385404] SDS_MODE_SEL_0: 0069a7df SDS_MODE_SEL_1: 000007df SDS_MODE_SEL_2: 0069a69a SDS_MODE_SEL_3: 0000069a

I'll try disabling the SDS version of writes next, and I'm trying to fix the sds_id always being zero at first in the mdio code (although I've hacked around that for now). I'll also change to 0x6 in the reset code.

I'd really like to figure out the i2c-mux at some point as well, since my SFP is reliably detected, but only works on port 1.

I'll clean up and share current patch and boot log later today.

Forgot to mention, I found at least one RTL8231 is detected successfully.

The RTL8231 page on the wiki has a list of pins you can check, to determine if a chip will run in shift register mode or as a real GPIO expander: https://biot.com/switches/rtl8231