DSA: two-switch setup in devicetree

I am trying to hook an ar8334 switch to the lantiq switch on a Fritzbox 5490 not sure how to correctly setup. The mdio bus of the lantiq chip probably needs to be chained to the mdio-bus of the qca8k switch... I am not sure.

Any help would be much appreciated! Maybe someone with more experience in this field can give some guidance here.

This is the setup:

Lantiq-gswip --- port0 - fiber
              +- port1 - switch-qca8k  --- port2 (lan3)
                                        +- port4 (lan4)
              +- port2 (lan2)
              +- port4 (lan1)
              +- port5 (wasp)

This my network related setup in the devicetree:

&gswip_mdio {
	// lan2 - port2
	phy5: ethernet-phy@5 {
		reg = <0x05>;
	};

	// lan1 - port4
	phy9: ethernet-phy@9 {
		reg = <0x09>;
	};

	// wasp - port5
//	phy7: ethernet-phy@7 {
//		reg = <0x07>;
//		reset-gpios = <&gpio 34 GPIO_ACTIVE_LOW>;
//	};

	// fiber - qca8033 (serdes transciever) - port0
	phy6: ethernet-phy@6 {
		reg = <0x06>;
		reset-gpios = <&gpio 32 GPIO_ACTIVE_LOW>;
	};
	// switch - qca8334 - port1
	switch18: switch@18 {
		compatible = "qca,qca8334";
		#address-cells = <1>;
		#size-cells = <0>;

		dsa,member = <0 1>;

//		reg = <0x10>;
		reg = <0x18>;
		reset-gpios = <&gpio 44 GPIO_ACTIVE_LOW>;

		ports {
			#address-cells = <1>;
			#size-cells = <0>;
			port@0 {
				reg = <0>;
				label = "switch";
				fixed-link {
					speed = <1000>;
					full-duplex;
				};
			};

			port@2 {
				reg = <1>;	// 2
				label = "lan3";
			};

			port@3 {
				reg = <2>;	// 3
				label = "lan4";
			};
		};
	};
};

&gswip_ports {
	port@0 {
		reg = <0>;
		label = "fiber";
		phy-mode = "rgmii-rxid";
		phy-handle = <&phy6>;
	};
	port@1 {
		reg = <1>;
		label = "switch";
		phy-mode = "rgmii-id";
		phy-handle = <&switch18>;

//		fixed-link {
//			speed = <1000>;
//			full-duplex;
//		};
	};
	port@2 {
		reg = <2>;
		label = "lan2";
		phy-mode = "internal";
		phy-handle = <&phy5>;
	};

	port@4 {
		reg = <4>;
		label = "lan1";
		phy-mode = "internal";
		phy-handle = <&phy9>;
	};

	port@5 {
		reg = <5>;
		label = "wasp";
		phy-mode = "rgmii-txid";
		fixed-link {
			speed = <1000>;
			full-duplex;
		};
	};
};

Producing this output:

[    8.124154] libphy: Fixed MDIO Bus: probed
[    8.451618] libphy: lantiq,xrx200-mdio: probed
[    8.545569] gswip 1e108000.switch: dsa switch register failed: -517
[    8.556733] membase: ae10b100
[    8.568965] NET: Registered protocol family 10
[    8.579258] Segment Routing with IPv6
[    8.581864] NET: Registered protocol family 17
[    8.586161] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[    8.602882] 8021q: 802.1Q VLAN Support v1.8
[    8.611614] pcie-xrx200 1d900000.pcie: failed to get the PCIe PHY
[    8.931647] libphy: lantiq,xrx200-mdio: probed
[    9.060468] DSA: tree 0 has no CPU port
[    9.062909] qca8k: probe of 1e108000.switch-mii:18 failed with error -22
[    9.181142] gswip 1e108000.switch fiber (uninitialized): PHY [1e108000.switch-mii:06] driver [Qualcomm Atheros AR8031/AR8033] (irq=POLL)
[    9.193671] gswip 1e108000.switch switch (uninitialized): no phy at 1
[    9.198669] gswip 1e108000.switch switch (uninitialized): failed to connect to port 1: -19
[    9.207089] gswip 1e108000.switch switch (uninitialized): error -19 setting up PHY for tree 0, switch 0, port 1
[    9.220294] gswip 1e108000.switch lan2 (uninitialized): PHY [1e108000.switch-mii:05] driver [Intel XWAY PHY11G (xRX v1.2 integrated)] (irq=POLL)
[    9.236991] gswip 1e108000.switch lan1 (uninitialized): PHY [1e108000.switch-mii:09] driver [Intel XWAY PHY11G (xRX v1.2 integrated)] (irq=POLL)
[    9.254228] DSA: tree 0 setup
[    9.255822] gswip 1e108000.switch: probed GSWIP version 21 mod 0
1 Like

Hmm this mdio tool seem usefull!

root@OpenWrt:/# mdio bus 1e108000.switch-mii
 DEV      PHY-ID  LINK
0x00  0x004dd036  down     <- qca8k-id
0x01  0x004dd036  down          "                         lan3
0x02  0x004dd036  down          "                         lan4
0x03  0x004dd036  down          "
0x04  0x004dd036  down          "
0x05  0xd565a409  down    <- PHY_ID_PHY11G_VR9_1_2
0x06  0x004dd074  down    <- ATH8033_PHY_ID
0x07  0x00000000  down
0x09  0xd565a409  down    <- PHY_ID_PHY11G_VR9_1_2
0x10  0x00000000  down
0x11  0x00000000  down
0x12  0x12800000  down    <-  
0x13  0x00000000  down
0x14  0x00000000  down
0x15  0x00000000  up
0x16  0x00000000  down
0x17  0x00000000  down
0x18  0x00000000  down

Assuming now switch ar8334 located at phy 0...
Lets retry..

1 Like

In for updates. I don't have a Fritz box (so I'm not any help with the actual issue), but I'm working on a device that has an mt7621/rtl8367rb switch combined. I'm mixing DSA for the mt7621, but swconfig for the rtl, so anything I can learn would be helpful :smiley:

1 Like

That sounds like a recipe for pain. lol. Also the luci-situation might turn into one these "I knew it would be painful, but was certain the gained knowledge bliss would outweigh the pain of the voyage"'-projects.

Another possible approach perhaps: How feasible is DSA-support for RTL8367? IIUC, they are the dumb(without builtin CPU) variant of the 8380. I have one or two as well, and plan to hook them up to some "Single-Lan-Port"-Openwrt-devices as a "Port extender", when/if DSA-support becomes available.

small correction, the switch is mt7530.

In the Edgerouter ER10x? I use the MT7621 defines and added the RTL8367RB after. (Edit: Right, you're talking the switch chip and I'm talking the SoC? I used whatever the default mt7621 set was for the ER-X)

So far, I've gotten All the ports on the MT76 board to work, and 4 of the 5 on the RTL to work, connected via switch0. I get connectivity, except for the dead port.. I've been fighting it for a while now (which is why the initial post intrigues me)

I've been looking into this, but I'm not a programmer nor a hardware engineer. the RTL SMI that OpenWrt uses should be usable, but I never figured out how. I do have the Tech datasheet pdf for the hardware though

Someone posted a patch set for RTL8365MB on netdev mailing list. RTL8367 uses the same code base.

The problem is that DSA relies on the switch's special tag, but MediaTek tag and Realtek tag are incompatible with each other, nor are they designed for D (Distributed) in DSA. So cross-chip operations such as bridging ports of different switches together may not work.

1 Like

Interesting discussions indeed, but my question is more related to the devicetree configuration.

What is puzzling to me is that the mdio tool is apparently able to see the ports of the QCA8334 switch, so we should be almost there.

I have tried different combinations and parameters but I am unable to probe the qca8k switch correctly using my dts config... :sob:

[    8.139646] libphy: Fixed MDIO Bus: probed
[    8.472197] libphy: lantiq,xrx200-mdio: probed
[    8.475574] mdio_bus 1e108000.switch-mii: MDIO device at address 0 is missing.
[    8.482780] mdio_bus 1e108000.switch-mii: MDIO device at address 1 is missing.
[    8.489832] mdio_bus 1e108000.switch-mii: MDIO device at address 2 is missing.
[    8.497120] mdio_bus 1e108000.switch-mii: MDIO device at address 3 is missing.
[    8.504382] mdio_bus 1e108000.switch-mii: MDIO device at address 4 is missing.
[    8.604710] gswip 1e108000.switch: dsa switch register failed: -517
[    8.616058] membase: ae10b100
[    8.632092] NET: Registered protocol family 10
[    8.643112] Segment Routing with IPv6
[    8.645487] NET: Registered protocol family 17
[    8.649870] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[    8.663694] 8021q: 802.1Q VLAN Support v1.8
[    8.672898] pcie-xrx200 1d900000.pcie: failed to get the PCIe PHY
[    9.001940] libphy: lantiq,xrx200-mdio: probed
[    9.005797] mdio_bus 1e108000.switch-mii: MDIO device at address 0 is missing.
[    9.012866] mdio_bus 1e108000.switch-mii: MDIO device at address 1 is missing.
[    9.019563] mdio_bus 1e108000.switch-mii: MDIO device at address 2 is missing.
[    9.026862] mdio_bus 1e108000.switch-mii: MDIO device at address 3 is missing.
[    9.034147] mdio_bus 1e108000.switch-mii: MDIO device at address 4 is missing.
[    9.171044] DSA: tree 0 has no CPU port
[    9.173482] qca8k: probe of 1e108000.switch-mii:08 failed with error -22
[    9.291615] gswip 1e108000.switch fiber (uninitialized): PHY [1e108000.switch-mii:06] driver [Qualcomm Atheros AR8031/AR8033] (irq=POLL)
[    9.307906] gswip 1e108000.switch lan2 (uninitialized): PHY [1e108000.switch-mii:05] driver [Intel XWAY PHY11G (xRX v1.2 integrated)] (irq=POLL)
[    9.324652] gswip 1e108000.switch lan1 (uninitialized): PHY [1e108000.switch-mii:09] driver [Intel XWAY PHY11G (xRX v1.2 integrated)] (irq=POLL)
[    9.341974] DSA: tree 0 setup
[    9.343571] gswip 1e108000.switch: probed GSWIP version 21 mod 0

Hello,

I'm also in a similar situation with a mt7621 and a rtl8367s in a TpLink Archer C5v4 (ramips/mt7620a). I'm modifying the driver @LGA1150 mentioned to port it from SMI/gpio interface to a MDIO interface, converting the driver from a platform driver into a mdio driver. My branch is https://github.com/luizluca/linux/tree/realtek_mdio_refactor

I'm working with kernel 5.15. I needed a hack in DTS file to disable all resets as, in 5.15, all devices where resets were applied stopped to work. @arinc9 did most the 5.15 port. Here is our current state: https://github.com/luizluca/openwrt/tree/arinc9 . Many things in the DTS file are guesses as I'm not really sure what to set where.

First I was trying to modify the platform_driver to use MDIO instead of SMI. I went as far as I was able to create the interfaces in OS and also detect link when a cable was connected to one of them. However, there was no traffic between ports and with CPU. However, as all upstream dsa drivers that use mdio are mdio driver, I started over. Now, with the mdio driver, I still cannot initialize the lan/wan ports. I get:

[  225.759377] realtek-mdio mdio-bus:1d lan1 (uninitialized): failed to connect to PHY: -ENODEV
[  225.768034] realtek-mdio mdio-bus:1d lan1 (uninitialized): error -19 setting up PHY for tree 0, switch 0, port 0
[  225.778668] realtek-mdio mdio-bus:1d lan2 (uninitialized): failed to connect to PHY: -ENODEV
[  225.787360] realtek-mdio mdio-bus:1d lan2 (uninitialized): error -19 setting up PHY for tree 0, switch 0, port 1
[  225.797864] realtek-mdio mdio-bus:1d lan3 (uninitialized): failed to connect to PHY: -ENODEV
[  225.806583] realtek-mdio mdio-bus:1d lan3 (uninitialized): error -19 setting up PHY for tree 0, switch 0, port 2
[  225.817166] realtek-mdio mdio-bus:1d lan4 (uninitialized): failed to connect to PHY: -ENODEV
[  225.825916] realtek-mdio mdio-bus:1d lan4 (uninitialized): error -19 setting up PHY for tree 0, switch 0, port 3
[  225.836463] realtek-mdio mdio-bus:1d wan (uninitialized): failed to connect to PHY: -ENODEV
[  225.845039] realtek-mdio mdio-bus:1d wan (uninitialized): error -19 setting up PHY for tree 0, switch 0, port 4

MediaTek tag and Realtek tag are incompatible but MediaTek mt7530 is really just used as a dumb switch connecting the CPU port to the external switch. If we leave it alone, would it pass the Realtek tag to the CPU port just like any other unknown ethertype? That is hope :wink:

Any help is appreciated. Thanks.

1 Like

I am working on the Edgerouter 10x, and I can get it to work with the Swconfig (mostly). I thought I had found the 8367r/s/rb driver, but integrating it into the source tree is mostly beyond my abilities. I may revisit it in the future, but I've been trying to track down other issues because I hit a wall with this.

if you have any suggestions on how to move forward, certainly send me a message!

If I'm not mistaken you don't init slave_mii_bus member of dsa switch structure.
realtek-smi-core.c defines function realtek_smi_setup_mdio and rtl8366rb.c call it.
I think that definition of phy_{read,write} function in rtl8365mb.c is not enough for dsa to work with realtek internal phy.

I intentionally didn't init slave_mii_bus. If I provide phy_read/write to ds_ops, dsa switch will create the slave_mii_bus by its own. Wouldn't it be enough for a switch with MDIO interface?

The current state is that the switch is able to configure all ports correctly with a little hack to define supported phy modes. However, the issue is that the switch is stuck in a block-all state, even between external ports. I think that the initialization code sequence is different for my switch. It might be missing some commands.

Default phy_{read,write} check phys_mii_mask (mask for available phy device on mii-bus) before calling of original phy_{read.write}

Yes, ds->phys_mii_mask is preinit with a bitmap of user ports. In my case, it has the value of '0000001f' (first five ports).

AFAIK, it looks like regs are being written as expected. I dumped the reads/writes and decode them here:

I'm new to this DSA world but it looks like the switch is configured to isolate ports. So, any traffic between LAN1 and LAN2 will be validated by the CPU port. If CPU port is not working, LAN ports will be isolated.

I think my CPU port is not working, either because I have something wrong with the CPU port setup, the internal switch mt7530 (inside a mt7620a) or the ethernet port. There is also another issue with MTU limit, as the ethernet driver failed to accept an MTU of 1508 (defined by DSA). I simply add a hack to pass that limit but it might be an issue when I get full length ethernet frames. Or it might be just an issue with the 5.15 kernel I'm using.

I don't find this code. But this error:

[  225.759377] realtek-mdio mdio-bus:1d lan1 (uninitialized): failed to connect to PHY: -ENODEV
[  225.768034] realtek-mdio mdio-bus:1d lan1 (uninitialized): error -19 setting up PHY for tree 0, switch 0, port 0

can be explained by zero value of phys_mii_mask.

Or it may be simpler. IIRC after reset phys are in disabled state. Can you use ethtool for checking phy state?

I think that the part of reading/writing to registers seems to be working. If I comment the code where the driver isolates the ports, it forwards traffic between lan ports. I can read mibs statistics, detect a connected cable (except for wan port). If I configure an lan port as the CPU, I can sniff externally the traffic with the DSA tag. The issue is only when I send that same data through the internal switch, either because it is dropping them (swconfig does not show statistics of received data) or because the CPU port setup is wrong.

The only issue is that I have no traffic between the local interface (eth0) and the switch. I don't know if the issue is a wrong CPU port or something the new driver still lacks. I think that the cpu port would be either EXT_PORT0 or EXT_PORT1. EXT_PORT0 is referenced in the original switch driver from realtek while docs says that only EXT_PORT1 supports rgmii. Anyway, both didn't work. Is it possible that the swconfig driver was working even with a wrong CPU port? Is there a quick way to detect which port is used to connect to the internal switch?

root@OpenWrt:~# ethtool lan3
Settings for lan3:
        Supported ports: [ TP    MII ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Advertised pause frame use: Symmetric Receive-only
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Link partner advertised link modes:  10baseT/Half 10baseT/Full
                                             100baseT/Half 100baseT/Full
                                             1000baseT/Full
        Link partner advertised pause frame use: No
        Link partner advertised auto-negotiation: Yes
        Link partner advertised FEC modes: Not reported
        Speed: 1000Mb/s
        Duplex: Full
        Auto-negotiation: on
        master-slave cfg: preferred master
        master-slave status: master
        Port: Twisted Pair
        PHYAD: 2
        Transceiver: external
        MDI-X: Unknown
        Supports Wake-on: d
        Wake-on: d
        Link detected: yes

Port 5 of internal mediatek switch is connected to port7 (EXT_PORT1?) of external realtek switch. So both ports must be configure as rgmii ports and internal mediatek switch as dumb switch (with port5 enabled). See PR 4327.
swconfig don't use CPU_TAG and don't config cpu port at all.

BTW I find another logical error in your code.
From SDK driver point of view:
port0-port4 - ports with internal gphy
port5 - digital interface 0 (is absent for rtl8367)
port6 - digital interface 1 (logical extended port 0 for rtl8367)
port7 - digital interface 2 (logical extended port 1 for rtl8367)
So your definition here

#define RTL8367C_DIGITAL_INTERFACE_FORCE_REG(_extport)   \
		(RTL8367C_DIGITAL_INTERFACE_FORCE_REG0 + \
		 ((_extport) & 0x1) +                     \
		 ((((_extport) >> 1) & 0x1) * (0x13C4 - 0x1310)))

is only correct if _extport is digital interface number.
But definition here

#define RTL8367C_EXT_RGMXF_REG(_extport)   \
		(RTL8367C_EXT_RGMXF_REG1 + \
		 (((_extport) >> 1) * (0x13C5 - 0x1307)))

assumes extended port 0/1 correspond digital interface 1/2.
Compare with definitions here

#define RTL8367B_EXT_RGMXF_REG(_x)		\
	  ((_x) == 2 ? 0x13c5 : 0x1306 + (_x))
#define   RTL8367B_EXT_RGMXF_DUMMY0_SHIFT	5
#define   RTL8367B_EXT_RGMXF_DUMMY0_MASK	0x7ff
#define   RTL8367B_EXT_RGMXF_TXDELAY_SHIFT	3
#define   RTL8367B_EXT_RGMXF_TXDELAY_MASK	1
#define   RTL8367B_EXT_RGMXF_RXDELAY_MASK	0x7

#define RTL8367B_DI_FORCE_REG(_x)		\
	  ((_x) == 2 ? 0x13c4 : 0x1310 + (_x))

So than you force ext_port to 1 you really set rgmii mode for digital interface 1 and rgmii parameters for digital interface 2.
But as I said before you need configure port7 (or digital interface 2).
And I think that it's better to change extended port to digital interface totally in your source for the sake of clarity.

I'm working on a device that has the MT7621 DSA and the RTL8367RB using swconfig. I've gotten everything to work except eth5, which is dead under OpenWrt and doesn't even register a cable event. I'm not trying to hijack/redirect the thread, but it isn't often I find folks talking about the internals of the RTL8367 :smiley:

I have already spotted that strange macro. It is not my code but the one accepted upstream (for 5.16).

There are these three values:
#define RTL8367C_REG_DIGITAL_INTERFACE0_FORCE 0x1310
#define RTL8367C_REG_DIGITAL_INTERFACE1_FORCE 0x1311
#define RTL8367C_REG_DIGITAL_INTERFACE2_FORCE 0x13c4

And that macro simply selects the INTERFACE<_extport> with a quite complex binary arithmetic. It does work but I would use a normal switch or a inline if. That macro is only used with _extport==1. It will work as expected for that value. I would simply use the macro RTL8367C_REG_DIGITAL_INTERFACE1_FORCE, but only after I get something working.

My issue seems to be the setup of that ext1 interface. I've tried ports 5, 6, 7, 16, 17 and none of them worked. I'm trying to troubleshoot it since then.

First I wanted to make sure I was configuring the switch correctly. I defined LAN2 as CPU port and I could see the tagged traffic. Maybe the internal switch was dropping frames with tag? I set tagging to DSA_TAG_PROTO_NONE and now the traffic on LAN2 was clear of tags, but again nothing when I change CPU to ports 5,6,7,16,17. Then I removed the isolation and "no learning" part of port setup, permitting switching between lan ports. Again, nothing got into internal switch (according to swconfig mib) when port 7 was CPU and no traffic was forwarded from/to the internal switch when CPU port was LAN2. So, the issue looks like it is with that connection between MT7530 port 5 and RTL8367S port 7.

                        ...
                        port@3 {
                                reg = <3>;
                                label = "lan4";
                        };

                        port@4 {
                                reg = <4>;
                                label = "wan";
                        };

                        port@7 {
                                reg = <7>;
                                label = "port7";
                                ethernet = <&ethernet>;
                                phy-mode = "rgmii";
                                tx-internal-delay-ps = <2000>;
                                rx-internal-delay-ps = <2000>;

                                fixed-link {
                                        speed = <1000>;
                                        full-duplex;
                                        pause;
                                };
                        };

Oh, and wan port also does not work either. I never get link status (when a cable is connected) nor it forwards traffic with lan when I added it to br-lan. Maybe the driver has a problem with >=4 ports? It is not a HW issue as wan works during the boot loader (I can load an image)

I tried port 16 and 17 because the rtl8367c from realtek defines ports as:

typedef enum rtk_port_e
{
    UTP_PORT0 = 0,
    UTP_PORT1,
    UTP_PORT2,
    UTP_PORT3,
    UTP_PORT4,
    UTP_PORT5,
    UTP_PORT6,
    UTP_PORT7,

    EXT_PORT0 = 16,
    EXT_PORT1,
    EXT_PORT2,

    UNDEFINE_PORT = 30,
    RTK_PORT_MAX = 31
} rtk_port_t;

And that is used at:

    /* Physical to Logical */
    {UTP_PORT0, UTP_PORT1, UTP_PORT2, UTP_PORT3, UTP_PORT4, UNDEFINE_PORT, EXT_PORT0, EXT_PORT1,

EXT_PORT1 is at index 7 but it is a "logical" position.