Need help understanding the internal network topology of the Linksys ea9500 (bcm53xxx)

Unlike the Linksys ea8500, whose switch ports are not directly exposed, the ea9500 makes network interfaces for each of the ports on the two internal switches. This is due to the driver being part of the distributed switch architecture (DSA). I'm trying to understand the topology of the devices so I know how I can best utilize the abundance of ports on this device.

So far I understand that there are two switches. One switch is a bcm53125 which is managed over the SRAB interface (/sys/devices/platform/18007000.srab) and another is a bcm53012 which is managed over MDIO (/sys/devices/platform/1800c1c0.mdio-mii-mux).

Broadcom documentation on these two devices (and the SoC) is really sparse. I wasn't able to find a datasheet.

Digging around in the OS, I can see that there's some notion that the lan ports are split between the two switches, with lan{1,2,3,5,6} on one switch and lan{4,7,8} on the other

root@OpenWrt:/# ip link | egrep -o '^[0-9][^<]+' | egrep 'eth|exswitch|lan[0-9]'
2: eth0:
3: eth1:
4: eth2:
5: extsw@eth0:
6: lan7@eth0:
7: lan4@eth0:
8: lan8@eth0:
9: wan@eth0:
10: lan1@extsw:
11: lan5@extsw:
12: lan2@extsw:
13: lan6@extsw:
14: lan3@extsw:

I found the device tree bindings for this board in the linux source, which helps solidify the mapping of lan ports to switches. It also reveals the switch ports that aren't getting used for the lan. These are the ports for which I'm interested in understanding the interconnects for.

switch@0  {
  compatible = "brcm,bcm53125";
  #address-cells = <1>;
  #size-cells = <0>;
  reset-gpios = <&chipcommon 10 GPIO_ACTIVE_LOW>;
  reset-names = "robo_reset";
  reg = <0>;
  dsa,member = <1 0>;
  ports{
    port@0 {
      reg = <0>;
      label = "lan1";
    };
    port@1 {
      reg = <1>;
      label = "lan5";
    };
    port@2 {
      reg = <2>;
      label = "lan2";
    };
    port@3 {
      reg = <3>;
      label = "lan6";
    };
    port@4 {
      reg = <4>;
      label = "lan3";
    };
    sw1_p8: port@8 {
      reg = <8>;
      ethernet = <&sw0_p0>;
      label = "cpu";
    };
  };
};

&srab {
  compatible = "brcm,bcm53012-srab", "brcm,bcm5301x-srab";
  status = "okay";
  dsa,member = <0 0>;
  ports {
    #address-cells = <1>;
    #size-cells = <0>;
    port@1 {
      reg = <1>;
      label = "lan7";
    };
    port@2 {
      reg = <2>;
      label = "lan4";
    };
    port@3 {
      reg = <3>;
      label = "lan8";
    };
    port@4 {
      reg = <4>;
      label = "wan";
    };
    port@8 {
      reg = <8>;
      ethernet = <&gmac2>;
      label = "cpu";
    };
    sw0_p0: port@0 {
      reg = <0>;
      label = "extsw";
    };
  };
};

(from: https://github.com/torvalds/linux/blob/master/arch/arm/boot/dts/bcm47094-linksys-panamera.dts)

In the configuration for switch@0 > port@8, there's a reference to sw0p0 from srab > port@0. I'm not certain if this is suggesting the two are wired to each other or not - but that would line up with how ip link describes the ports (i.e., lan1@extsw and extsw@eth0).

What I'm still unclear on is that there's an eth1 and eth2 as well. If all 8 lan ports and the wan port are coming in via the switches, and the switches are chained behind eth0, then what are these other two interfaces wired to?

The community build for the ea9500 adds two VLANs on eth2 and there is some userspace configuration suggesting these VLANs are used for the lan and the wan port(s):

root@OpenWrt:/# grep -A25 panamera /etc/board.d/02_network 
linksys,panamera)

        et0macaddr="$(nvram get et0macaddr)"
        et1macaddr="$(nvram get et1macaddr)"
        et2macaddr="$(nvram get et2macaddr)"

        ip link set eth0 address $et0macaddr
        ip link set eth1 address $et1macaddr
        ip link set eth2 address $et2macaddr

        for p in lan1 lan2 lan3 lan4 lan5 lan6 lan7 lan8 extsw wan
        do
                ip link set $p address $et2macaddr
        done

        ucidef_set_interface_lan "lan1 lan2 lan3 lan4 lan5 lan6 lan7 lan8 extsw eth2.101"
        ucidef_set_interface_wan "wan eth2.102"

        board_config_flush
        
        exit 0
        ;;
esac

So I'm clearly missing a few links (figuratively and literally) in my understanding.

By randomly digging through /etc for what looked like userspace initialization stuff, I located this script, which is called from /etc/rc.local. It explains where the vlans are coming from but doesn't really help with understanding the topology much. It explicitly sets eth0 down, but ip link suggests that all the switches are hanging off eth0. This suggests they're all via eth2 instead...

root@OpenWrt:/# cat /etc/fix_interfaces.sh
#!/bin/sh

vlan101_ifnames='lan1 lan2 lan3 lan4 lan5 lan6 lan7 lan8 wlan0 wlan1 wlan2 extsw eth2.101'
vlan102_ifnames='wan eth2.102'
br_ifnames='br-wan br-lan'

# Create VLAN $2 on interface $1
create_vif() {
    ip link add link $1 name $1.$2 type vlan id $2
}

set_bridge_stp_fdd() {
    ip link set dev $1 type bridge stp_state 0
    ip link set dev $1 type bridge forward_delay 0
}

enslave_ifaces() {
    for p in $2
    do
        ip link set $p up
        ip link set $p master $1
        bridge vlan add vid $3 dev $p pvid untagged
    done
}


delete_vids() {
    for p in $2
    do
        bridge vlan del dev $p vid $1 self
        bridge vlan del dev $p vid $1 master
    done
}


echo '#################################'
echo '      Fixing DSA Interfaces'
echo '#################################'

ip link set eth0 down
ip lonk set eth1 down

create_vif eth2 101
create_vif eth2 102

set_bridge_stp_fdd br-lan
set_bridge_stp_fdd br-wan

enslave_ifaces br-lan "$vlan101_ifnames" 101

# Make extsw tagged member of eth0.101
bridge vlan add vid 101 dev extsw pvid tagged

enslave_ifaces br-wan "$vlan102_ifnames" 102

delete_vids 1 "$vlan101_ifnames $vlan102_ifnames $br_ifnames"

exit 0

@npcomplete Probably knows, or lease could point me in the right direction.

Found this diagram on npcomplete's blog.

This helps a lot. It doesn't express what the interface names are to the CPU per say. There are three of them and three eth[0-9] so one could make an educated guess that eth2 is port 8 on the internal switch.

All eth0 (port5), eth1 (port7) and eth2(port8) are on internal switch. Please read: https://github.com/RMerl/asuswrt-merlin/blob/master/release/src-rt-7.x.main/src/include/hndfwd.h

Also, correction to your first post - 53012 is internal switch and 53125 is external.

Let me know if there any more questions.

Thanks,

@npcomplete when you say eth2(port8) is on internal switch, do you mean the internal switch port 8's internal MAC is what is designated as eth2 or is there a separate host ethernet controller (external to the internal switch BCM53012) that is wired to port 8?

From some DSA documentation online (https://netdevconf.info/2.1/papers/distributed-switch-architecture.pdf), it would appear the host ethernet controller (master interface) in our EA-9500 example would be eth2, and hence external to the switch and not the GMAC internal switch port itself?