DSA Device Tree (For the watchguard T35)

I am working on the device tree for the Watchguard Firebox T35. The device has 5 physical, external Ethernet ports connected to a Marvell mvE88E6176 DSA switch (same driver as a mvE88E6085).

I have link lights on the switch that my firebox is connected to. I see in the OpenWRT console that the interfaces respond to a cable being physically inserted or removed by going up or down. I see all of my external interfaces enslaved to a bridge. Yet, from my PC connected to the same LAN, I'm scanning for the IP of the Firebox, I don't see it. I can't ping the assigned IP, and my Firebox doesn't get a DHCP address if I set it to automatic.

I think I have the control plane in my device tree configured correctly but I'm missing the data plane.

  1. Does someone have working DTS for the T35? I seems like there's active, but incomplete development for the Watchguard T series
  2. Is someone able to explain the dataplane connection between by SoC and the MV switch and or inspect my device tree for errors?

I have been basing my device tree off of the stock images' DTS and the OpenWRT watchguard Firebox m300 DTS
(and of course the linux/Documentation/devicetree/... files in the kernel, thank you ElektromAn )

I'll share my DTS in a moment, I copied the t1024rdb.dts file and made appropriate edits since unlike the rdb, the firebox doesn't have it's Ethernet ports connected directly to the SoC.

This is my current version of the DTS

To elaborate on my network, this is my config:

config interface 'loopback'
        option device 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'fdfd:3c63:d126::/48'

config device 'lan_br'
        option name 'br-lan'
        option type 'bridge'
        list ports 'eth1'
        list ports sweth0
        list ports sweth1
        list ports sweth2
        list ports sweth3
        list ports sweth4

config interface 'lan'
        option device 'br-lan'
        option proto 'static'
        option netmask '255.255.255.0'
        option ip6assign '60'
        option ipaddr '192.168.10.25'

config interface 'wan'
        option device 'eth0'
        option proto 'dhcp'

config interface 'wan6'
        option device 'eth0'
        option proto 'dhcpv6'

I have this output from ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 66:01:55:e7:6d:b9 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::6401:55ff:fee7:6db9/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br-lan state UP qlen 1000
    link/ether aa:a4:69:45:ef:e8 brd ff:ff:ff:ff:ff:ff
4: sweth0@eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master br-lan state LOWERLAYERDOWN qlen 1000
    link/ether aa:a4:69:45:ef:e8 brd ff:ff:ff:ff:ff:ff
5: sweth1@eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master br-lan state LOWERLAYERDOWN qlen 1000
    link/ether aa:a4:69:45:ef:e8 brd ff:ff:ff:ff:ff:ff
6: sweth2@eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-lan state UP qlen 1000
    link/ether aa:a4:69:45:ef:e8 brd ff:ff:ff:ff:ff:ff
7: sweth3@eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master br-lan state LOWERLAYERDOWN qlen 1000
    link/ether aa:a4:69:45:ef:e8 brd ff:ff:ff:ff:ff:ff
8: sweth4@eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master br-lan state LOWERLAYERDOWN qlen 1000
    link/ether aa:a4:69:45:ef:e8 brd ff:ff:ff:ff:ff:ff
9: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether aa:a4:69:45:ef:e8 brd ff:ff:ff:ff:ff:ff
    inet 192.168.10.25/24 brd 192.168.10.255 scope global br-lan
       valid_lft forever preferred_lft forever
    inet6 fdfd:3c63:d126::1/60 scope global noprefixroute
       valid_lft forever preferred_lft forever
    inet6 fe80::a8a4:69ff:fe45:efe8/64 scope link
       valid_lft forever preferred_lft forever

Right now, I have an Ethernet cable psyically plugged into port2 = sweth2

I have this from ethtool:

root@OpenWrt:/# ethtool sweth2
Settings for sweth2:
        Supported ports: [ TP    MII ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Link partner advertised link modes:  10baseT/Half 10baseT/Full
                                             100baseT/Half 100baseT/Full
                                             1000baseT/Half 1000baseT/Full
        Link partner advertised pause frame use: No
        Link partner advertised auto-negotiation: Yes
        Link partner advertised FEC modes: Not reported
        Speed: 1000Mb/s
        Duplex: Full
        Auto-negotiation: on
        Port: Twisted Pair
        PHYAD: 2
        Transceiver: external
        MDI-X: Unknown
        Supports Wake-on: d
        Wake-on: d
        Link detected: yes

It don't think it is, but it's possible it's a VLAN issue. My router that my Firebox is connected to uses IPFire - it's on the green (trusted LAN) interface. The switch between my Firebox and IPfire router is a Netgear GS108Ev3 that can do VLAN tagging, but I haven;t configure it to do so

See Watchguard M200 firewall appliance: could we run OpenWrt on it? - #8 by stintel where stintel who added the M300 support discusses his issues with getting networking to work on the M200 which has some similarities to the T35.

1 Like

About your dts-file.

Are your sure that marvell switch have 0 address at mdio0 (mdio@fc000) bus?

        switch0: switch@0 {
      		compatible = "marvell,mv88e6085";
          reg = <0x0>;

M200 use 0x10 address. If this address is wrong than switch can not be correctly configured. (Check log?)

For rgmii connection type usaully you need to set phy-mode = “rgmii-txid”; for both end of connection

			      port@6 {
			              reg = <6>;
			              label = "cpu";
			              ethernet = <&enet3>;
			              phy-mode = "rgmii-txid";
			              fixed-link {
					      speed = <1000>;
					      full-duplex;
				      };
			      };

and

  		  enet3: ethernet@e6000 {
		  	phy-connection-type = "rgmii";
		        phy-mode = "rgmii-txid";
		        sleep = <&rcpm 0x10000000>;
		        fixed-link {
		          speed = <1000>;
		          full-duplex;
		        };
		  };

About network config. Exclude dsa-port eth1from br-lanbridge.

the switch address is correct

Linksys WRT3200ACM uses the same switch chip

here is my network config

root@OpenWrt:~# cat /etc/config/network 

config interface 'loopback'
	option device 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option ula_prefix 'fdff:7794:93a5::/48'

config device
	option name 'br-lan'
	option type 'bridge'
	list ports 'lan1'
	list ports 'lan2'
	list ports 'lan3'
	list ports 'lan4'

config interface 'lan'
	option device 'br-lan'
	option proto 'dhcp'

config device
	option name 'wan'
	option macaddr '62:38:e0:bc:32:30'

config interface 'wan'
	option device 'wan'
	option proto 'dhcp'

config interface 'wan6'
	option device 'wan'
	option proto 'dhcpv6'

dts for the switch is here

I recall trying address switch@10 because that's what I saw in the m300 DTS and it not working. I've had the switch@0 for the last few iterations and I've been able to bring up the switch.

Nevertheless, I will try switch@10 again and report back as soon as I can.

I guess my rudimentary understanding was that MDIO addresses didn't correspond to a physical connection or set of pins on the board, and so were arbitrary?

I do see in the device-tree documentation for the switch chip

marvell,mv88e6085: This switch uses base address 0x10.
            This switch and its siblings will be autodetected from
            ID registers found in the switch, so only "marvell,mv88e6085" should be
            specified. This includes the following list of MV88Exxxx switches:
            6085, 6095, 6097, 6123, 6131, 6141, 6161, 6165, 6171, 6172, 6175, 6176,
            6185, 6240, 6320, 6321, 6341, 6350, 6351, 6352
          marvell,mv88e6190: This switch uses base address 0x00.
            This switch and its siblings will be autodetected from
            ID registers found in the switch, so only "marvell,mv88e6190" should be
            specified. This includes the following list of MV88Exxxx switches:
            6190, 6190X, 6191, 6290, 6361, 6390, 6390X

I see 88E6176 printed on the switch chip, so that means it has the first driver for the marvell,mv88e6085. It says it has base address 0x10, so I was also suprised that I got it to come up with base address 0 but not 10. I will check again, as I have a habit of making multiple changes at once and testing instead of one at a time

switch driver report result of marvell chip autodetection in kernel log.

Then simple combinatorics:

SoC mac enet2,enet3is connected to port 5, port 6 of switch using connection type rgmii, sgmii :slight_smile:

I am finding that something is transmitting on the cable, at least at level 2

I have managed switch connected to the other end that has a bytes-received counter for each port.
When try to ping my other router or PC from the OpenWRT router, I can see the bytes received go up by increments of 64 byte in sync with my OpenWRT router sending out ping requests

My two leading theories are that its a VLAN issue or somthing similar.
I tried setting everything to use VLAN1 and still couldn't see my OpenWRT router's IP.

I also assured everything was untagged and still couldn't see my OpenWRT router's IP.

Theory two is that the interface is able to transmit, but not receive. I see 88 tx_packets on my lan4 interface where I have my cable plugged in, but 0 rx_packets:

root@OpenWrt:~# cat /sys/class/net/lan4/statistics/tx_packets
88
root@OpenWrt:~# cat /sys/class/net/lan4/statistics/rx_packets
0

Has anyone got pointers? I've tried tcpdumping the interface on my PC and working router to see no arp packets and I woutdn't expect to see the ICMP ping packets.

With respect to the device tree, I found that the switch comes up, at at least on the management plae, whether I use MDIO address 0x10 or 0x0.

Right now I have it assigned to MDIO address 0x10 since that's what the documentation says to do.

I also noticed that in my T35 stock image's device tree, the mac node has tx and rx ports opposite of the linux kernel T1024rdb's port assignment.
Also, the stock DTS uses the fman v2 port rx and tx drivers, which makes no sense to me since it's an v3 frame manager

Stock DTS pseudocode:

fman@400000/ethernet@e4000{
   compatible = "fsl,fman-memac";
   rx = port@8b000 {  compatible = "fsl,fman-v2-port-rx"; }
   tx = port@ab000 {  compatible = "fsl,fman-v2-port-tx"; }
}
fman@400000/ethernet@e6000{
   compatible = "fsl,fman-memac";
   rx = port@8a000 {  compatible = "fsl,fman-v2-port-rx"; }
   tx = port@aa000 {  compatible = "fsl,fman-v2-port-tx"; }
}

compared to the linux kernel t1024rdb DTS:

fman@400000/ethernet@e4000{
   compatible = "fsl,fman-memac";
   rx = port@8a000 {  compatible = "fsl,fman-v3-port-rx"; }
   tx = port@aa000 {  compatible = "fsl,fman-v3-port-tx"; }
}
fman@400000/ethernet@e6000{
   compatible = "fsl,fman-memac";
   rx = port@8b000 {  compatible = "fsl,fman-v3-port-rx"; }
   tx = port@ab000 {  compatible = "fsl,fman-v3-port-tx"; }
}

I think that rx traffic is more informative. Initiate traffic from switch to SoC (ping or dhcp session from external client) and analyze output of ifconfig for both enet3 (linux eth1 network device) and enet2 (eth0).

To solve our combinatics quiz follow these 4 simple steps:

  1. Disable port@5 of switch (status = "disabled";), set connection type to rgmii (phy-mode = "rgmii-id";) for all enet1 , enet2 and port@6. Really it's current configuration. Initiate traffic from switch to SoC (As example connect any client to switch, set static ip address 192.168.1.2 and ping router 192.168.1.1 or start dhcp client).
    Show output of ifconfig command of router console keeping attention to rx packets.

  2. Disable port@6 and enable port@5, set connection type to rgmii phy-mode = "rgmii-id";for all enet1 , enet2 and port@6. Initiate traffic from switch to SoC.
    Show output of ifconfig.

  3. Disable port@6 and enable port@5, set connection type to rgmii phy-mode = "sgmii";for all enet1 , enet2 and port@6. Initiate traffic from switch to SoC.
    Show output of ifconfig.

  4. Disable port@5 and enable port@6, set connection type to rgmii phy-mode = "sgmii";for all enet1 , enet2 and port@6. Initiate traffic from switch to SoC.
    Show output of ifconfig.

I have tried those permutations (combinations?) to no effect. I'm investigating other causes of the DPAA hardware not working.

I recall at one point, the SoC macs had fized addresses, but I'm seeing that they get random MAC addresses at runtime. I think there is something wrong with my device tree w.r.t. the MACs:

[    1.351869] fsl_dpaa_mac ffe4e4000.ethernet: of_get_mac_address(/soc@ffe000000/fman@400000/ethernet@e4000) failed
[    1.363529] fsl_dpaa_mac ffe4e4000.ethernet: FMan MEMAC
[    1.375111] fsl_dpaa_mac ffe4e6000.ethernet: of_get_mac_address(/soc@ffe000000/fman@400000/ethernet@e6000) failed
[    1.390645] fsl_dpaa_mac ffe4e6000.ethernet: FMan MEMAC
[    1.403174] fsl_dpaa_mac ffe4e4000.ethernet: Using random MAC address: 66:e0:3d:ce:ba:ee
[    1.414593] fsl_dpaa_mac ffe4e4000.ethernet eth0: Probed interface eth0
[    1.423789] fsl_dpaa_mac ffe4e6000.ethernet: Using random MAC address: 46:03:e4:46:49:49
[    1.435290] fsl_dpaa_mac ffe4e6000.ethernet eth1: Probed interface eth1

the working device-tree i have is from kernel v3, so it's was not trivial getting it to work with v5 and I just recently switched to v6.

Does anyone know where the MAC fixed addresses are physically stored? The question is somewhat ill formed.
I know the external Ethernet ports have the same MAC address as the SoC's DSA cpu port's MAC address, so my question is where does the SoC get it's MAC address from.

I'm seeing on the Freescale/NXP forums a question "Where are the MAC address stored"
https://community.nxp.com/t5/Layerscape/LS1046A-Where-MAC-addresses-are-stored/m-p/911830 (I believe this link requires an account to view the post)
with response

LS1046A does not have fuses for MAC address. In the LS1046ARDB board MAC addresses are stored in external EEPROM connected via I2C interface....

My current understanding is that the bootloader UBoot loads the MAC addresses from EEPROM and passes those to the kernel before it starts?

You do not see rx packet on linux network device eth0 or eth1? Or ping/dhcp client fail for all 4 case?

It do not concern SoC to switch connectivity problem.