MT7531 LAN Switch issues

thatsme · June 9, 2024, 1:49pm

Hi,

I am using an Asus AX 4200 (mediatek FiLogic with MT7531 Switch) and have
some issues with LAN to LAN connectivity.

I use one of the Switch Ports (LAN4) as an uplink to the Network, while I am using
LAN Ports 1 to 3 for an WIIM Media Streamer, an VU+ Settop Box and an FireTV Cube.
In the upstream Network there are two more AX4200s with Openwrt, all AX4200 are
running as dumb access points.

The issue is with LAN to LAN Port connectivity. ARP Packets coming from the upstream Network don't get a reply (tough with tcpdump I have seen them going
out to the device, and also the ARP Reply comes back to the device but does
not get forwarded to the upstream network.)

In practice it causes the iPhone App not to discover the WIIM Media streamer when the iPhone ist associated to another AX4200 (traffic to the device is then LAN to LAN
on the AX4200 in Question), while it works fine when the iPhone is accociated to the same AX4200 where the WIIM is connected to. Means in summary: WLAN to LAN traffic is
ok, LAN to LAN traffic is faulty.

Using Openwrt 23.05-Snaphot with Kernel 5.15 is working as expected, also LAN to LAN
traffic is working good. Using an master build with Kernel 6.6 (and earlier also with 6.1)
shows this failure.

A workaround is to include Ethernet eth1 into the br-lan bridge and use that as the uplink
port, forcing the traffic though the software bridge bypassing the hardware switch.

So I am using the 23.05-Snaphot now but I'd like to go to master because of some
newer features and because WED with Bridger works as expected with master while it
doesn't with 23.05-Snaphot.

I'd assume it is related to the kernel and the DSA subsystem.

Does anyone else made the same observation?
By intent I used the thread title "MT7531 LAN Switch issue" because the error may or may not occour on other devices using the same chipset, which would be interesting to know.

If you google for MT7531 arp lan etc you'll find some possibly related issue, but none of
them explicitely is about LAN to LAN traffic.

Thanks for reading,
best regards,
Stephan

thatsme · June 17, 2024, 9:04am

... seems to be related to

github.com/openwrt/openwrt

Regression: DSA breaks roaming to WLAN bridged to VLAN

opened 06:07PM - 30 Dec 22 UTC

dahopem

bug

### Describe the bug Subject: Regression: DSA breaks roaming to WLAN bridged to… VLAN # Summary When bridging a WLAN with a VLAN on DSA-enabled OpenWRT versions, an erroneous FDB entry creates 100% packet loss for 300 seconds. # Summary analysis When a VLAN-tagged packet is received by the device from the an Ethernet (LAN) port side, 2 FDB entries for the source MAC address of the packet are created in the forwarding database (FDB) of the bridge: 1. one VLAN-tagged FDB entry and 2. one VLAN-untagged FDB entry Both FDB entries tell the switch to forward packets whose destination is that MAC address to that Ethernet port. When then a packet from the same source is received by the device from the WiFi (WLAN) side (e.g. after roaming a client device to that WiFi) and forwarded to the Ethernet (LAN) side (with a VLAN tag), only 1 of these 2 FDB entries are updated (namely the VLAN-tagged FDB entry). The other VLAN-untagged FDB entry remains unchanged, erroneously. When then another VLAN-tagged packet (e.g. an ARP reply packet or a DHCP reply packet) is received by the device from the Ethernet (LAN) side and that packet's destination is the client device's MAC address (=the VLAN-untagged FDB entry's MAC address), then the switch believes that this packet ought to be forwarded to the Ethernet port (even although it is received from exactly this port), while it actually should be forwarded to the CPU of the router (and then further to the WLAN). This packet then gets dropped. This packet loss continues until, after about 300 seconds (5 minutes), the erroneous FDB entry expires. # Impact of this bug This bug affects potentially all DSA-enabled platforms, but at least a sizable subset of all DSA-enabled platforms. It has been verified to exist on: 1. OpenWRT 22.03 on BT HomeHub 5a (hh5a). 2. Turris OS 4.0 on [Turris Omnia](https://forum.turris.cz/t/omnia-vlan-on-dsa-port-breaks-arp-responses-tos-4-0-5/12584/37) already since the year 2020. # How to reproduce ## Steps 1. Ensure that your OpenWRT device A and software supports Distributed Switch Architecture (DSA). 2. Setup your OpenWRT device A to have at least one WiFi interface and at least one ethernet port with VLAN support: 1. Create bridge device "br-switch", with the list of bridge ports only consisting of port "lan2". 2. Enable "VLAN filtering" for that bridge-device and create a VLAN with VLAN ID 31, local enabled and Egress tagged for port "lan2". 3. Create OpenWRT interface "users" with device "br-switch.31". 4. Create a OpenWRT WiFi interface with SSID "test". Ensure (under Interface Configuration/General Setup/Network) that it is connected to network "users". 5. Verify by running "brctl show" on the command line (e.g. using SSH) that there is a bridge called "br-switch" with at least 2 members (one member being "lan2" and the other member being a WiFi interface, for example "wlan1-1"). 6. Reboot the OpenWRT device A. 3. Connect your OpenWRT device A to another WiFi router B 1. Ensure that WiFi router B also support VLANs 2. Connect an Ethernet cable to port "lan2" of OpenWRT device A and a suitable Ethernet port of WiFi router B. 3. Setup WiFi router B such that it has a VLAN with VLAN ID 31 which is available (tagged) at the Ethernet port of WiFi router B. 4. Setup a WiFi interface on WiFi router B with the same SSID "test". Similarly to OpenWRT device A, ensure that that this WiFi interface is bridged to VLAN ID 31. 4. Connect a client device C to OpenWRT device A. 1. Run "ping" from the client device C to the IP address of WiFi router B. 2. Observe that client device C receives ping replies. 5. Roam to WiFi router B. 1. Change the physical position of client device C to be close to WiFi router B and away from OpenWRT device A. (You may need to reduce the output power of OpenWRT device A if the devices are close to each other.) 2. Wait 15 seconds. 3. Verify that client device C is now associated with WiFi router B. (You may verify this looking at the UI of WiFi router B or looking at the output of running "iw dev wlan0 link" on client device C.) 4. Observe that client device C still receives ping replies although its WLAN association has changed. 6. Roam to OpenWRT device A again. 1. Change the physical position of client device C to be close to OpenWRT device A and away from WiFi router B . (You may need to reduce the output power of WiFi router B if the devices are close to each other.) 2. Wait 15 seconds. 3. Verify that client device C is now associated with OpenWRT device A. (You may verify this looking at the UI of WiFi router B or looking at the output of running "iw dev wlan0 link" on client device C.) ## Expected result It is expected that client device C still receives ping replies although its WLAN association has changed again. ## Observed result It can be observed that client device C does not receive ping replies once its WLAN association has changed again. # How to analyze 1. Install OpenWRT packages "ip-bridge" on your OpenWRT device A. 2. Re-run the reproduction steps 4, 5, and 6. Assume that the MAC address of client device C is "02:ff:04:05:06:07". 3. Run "bridge fdb show | grep 02:03:04:05:06:07" after step 3 but before step 4. Observe that the output is empty. 4. Run "bridge fdb show | grep 02:03:04:05:06:07" after step 4 but before step 5. Observe that the output is akin ``` 02:ff:04:05:06:07 dev wlan1-1 vlan 31 master br-switch ``` This shows that there is one FDB entry which says that if there is a packet which is tagged with VLAN 31 and which has the destination MAC address 02:03:04:05:06:07, then it should be forwarded through device "wlan1-1". 5. Run "bridge -statistics fdb show | grep 02:03:04:05:06:07" after step 5 but before step 6. Observe that the output is akin ``` 02:ff:04:05:06:07 dev lan2 vlan 31 master br-switch 02:ff:04:05:06:07 dev lan2 self ``` This shows that there is one FDB entry which says that if there is a packet which is tagged with VLAN 31 and which has the destination MAC address 02:03:04:05:06:07, then it should be forwarded through device "lan2", and another FDB entry which says that if there is a packet which has the destination MAC address 02:03:04:05:06:07, then it should be forwarded through device "lan2". 6. Run "bridge -statistics fdb show | grep 02:03:04:05:06:07" after step 6. Observe that the output is akin ``` 02:ff:04:05:06:07 dev lan2 self 02:ff:04:05:06:07 dev wlan1-1 vlan 31 master br-switch ``` This shows that there is one FDB entry which says that if there is a packet which is tagged with VLAN 31 and which has the destination MAC address 02:03:04:05:06:07, then it should be forwarded through device "wlan1-1", and another FDB entry which says that if there is a packet which has the destination MAC address 02:03:04:05:06:07, then it should be forwarded through device "lan2". # Working workarounds ## Wait 5 minutes After 5 minutes, the erroneous FDB entry expires automatically. ## Delete the erroneous FDB entry explicitly once Run "bridge fdb del 02:ff:04:05:06:07 dev lan2". Once you do this, you will observe immediately that the packets are not dropped anymore. ## Delete the erroneous FDB entry explicitly automatically This bug is so pervasive that [somehone has created a workaround](https://forum.turris.cz/t/omnia-vlan-on-dsa-port-breaks-arp-responses-tos-4-0-5/12584/44) which deletes the erroneous FDB entries automatically. ## Downgrade to swconfig-enabled-OpenWRT versions (instead of DSA-enabled OpenWRT versions) This is possible and works perfectly. However, not upgrading is not viable in the long run. # Non-working workarounds ## Enabling learning_sync Running ``` bridge link set dev lan2 learning_sync on ``` has no effect on the bug, it still occurs. ## Disabling learning Running ``` bridge link set dev lan2 learning off ``` fails with ``` RTNETLINK answers: Not supported ``` # Analysis The root cause seems to be the confusion between whether VLAN-untagged FDB entries should also apply to VLAN-tagged packets. This confusion results in unequal treatment for VLAN-tagged packets incoming through the Ethernet port and forwarded to the CPU on the one hand and VLAN-tagged packets outgoing from the CPU through the Ethernet port on the other hand: 1. When a VLAN-tagged packet is incoming, 2 FDB entries are created (or updated) 2. When a VLAN-tagged packet is outgoing, only 1 FDB entry is created (or updated) ## Solution 1: VLAN-untagged FDB entries should not apply to VLAN-tagged packets In this case, 1. when a VLAN-tagged packet is incoming, only 1 FDB entries should be created (or updated), 2. when a VLAN-tagged packet is outgoing, only 1 FDB entry should be created (or updated). ## Solution 2: VLAN-untagged FDB entries should also apply to VLAN-tagged packets In this case, 1. when a VLAN-tagged packet is incoming, 2 FDB entries should be created (or updated), 2. when a VLAN-tagged packet is outgoing, 2 FDB entry should be created (or updated). This confusion is evidently software-based, as non-DSA versions of OpenWRT do not exhibit this bug. The exact location of this confusion is (currently) unknown. ### OpenWrt version r19803-9a599fee93 ### OpenWrt target/subtarget lantiq/xrx200 ### Device hh5a (BT HomeHub 5a) ### Image kind Official downloaded image ### Steps to reproduce 1. Ensure that your OpenWRT device A and software supports Distributed Switch Architecture (DSA). 2. Setup your OpenWRT device A to have at least one WiFi interface and at least one ethernet port with VLAN support: 1. Create bridge device "br-switch", with the list of bridge ports only consisting of port "lan2". 2. Enable "VLAN filtering" for that bridge-device and create a VLAN with VLAN ID 31, local enabled and Egress tagged for port "lan2". 3. Create OpenWRT interface "users" with device "br-switch.31". 4. Create a OpenWRT WiFi interface with SSID "test". Ensure (under Interface Configuration/General Setup/Network) that it is connected to network "users". 5. Verify by running "brctl show" on the command line (e.g. using SSH) that there is a bridge called "br-switch" with at least 2 members (one member being "lan2" and the other member being a WiFi interface, for example "wlan1-1"). 6. Reboot the OpenWRT device A. 3. Connect your OpenWRT device A to another WiFi router B 1. Ensure that WiFi router B also support VLANs 2. Connect an Ethernet cable to port "lan2" of OpenWRT device A and a suitable Ethernet port of WiFi router B. 3. Setup WiFi router B such that it has a VLAN with VLAN ID 31 which is available (tagged) at the Ethernet port of WiFi router B. 4. Setup a WiFi interface on WiFi router B with the same SSID "test". Similarly to OpenWRT device A, ensure that that this WiFi interface is bridged to VLAN ID 31. 4. Connect a client device C to OpenWRT device A. 1. Run "ping" from the client device C to the IP address of WiFi router B. 2. Observe that client device C receives ping replies. 5. Roam to WiFi router B. 1. Change the physical position of client device C to be close to WiFi router B and away from OpenWRT device A. (You may need to reduce the output power of OpenWRT device A if the devices are close to each other.) 2. Wait 15 seconds. 3. Verify that client device C is now associated with WiFi router B. (You may verify this looking at the UI of WiFi router B or looking at the output of running "iw dev wlan0 link" on client device C.) 4. Observe that client device C still receives ping replies although its WLAN association has changed. 6. Roam to OpenWRT device A again. 1. Change the physical position of client device C to be close to OpenWRT device A and away from WiFi router B . (You may need to reduce the output power of WiFi router B if the devices are close to each other.) 2. Wait 15 seconds. 3. Verify that client device C is now associated with OpenWRT device A. (You may verify this looking at the UI of WiFi router B or looking at the output of running "iw dev wlan0 link" on client device C.) ### Actual behaviour It can be observed that client device C does not receive ping replies once its WLAN association has changed again. ### Expected behaviour It is expected that client device C still receives ping replies although its WLAN association has changed again. ### Additional info _No response_ ### Diffconfig _No response_ ### Terms - [X] I am reporting an issue for OpenWrt, not an unsupported fork.

thatsme · June 28, 2024, 9:40am

It boils down to a bridger issue which is already known:

Disabling bridger restores normal operation. I have been looking to it
from the wrong perspective: I was asking myself: why can the devices on the local AX4200 switch not be reached, but the real question is: why can the wifi device not reach those devices.

I also used tcpdump on the Asus AX4200 directly on the LAN ports, but I could
only see received packets, not sent packets ( at least if they have been forwarded though the switch hardware).

But now the situation is clear, I can live without bridger until it has been fixed,the CPU is strong enough to handle the pakets without hardware
acceleration (WED)

Thanks for reading,
regards.
Stephan

wilsonyan · June 30, 2024, 1:06am

vlan support is still busted on ipq40xx
tagging seems to work for the first / lan vlan
but if you make another one nothing flows between two tagged ports

stuck to using 5.15.x kernel still