IPQ40xx Switch Config "Strangeness"

Let me follow up on this topic in May 2022.
I have a summary of what status quo I'm facing, I have two sketches, and I also have a possible workaround :slight_smile: in terms of user-space configuration (no recompilation needed).

My HW:
Mikrotik hAP AC2

Caveat #1: on the plastic shell of this hardware, on the printed labels, the RJ45 ports are labeled "in reverse order" compared to the port numbers reported by swconfig etc. I.e., the cursed WAN port, internally known as port 5, is labeled "#1 - Internet/PoE in" on the outside.

Caveat #2: if you use the DHCP+TFTP method to boot the initial "initrd+kernel" as an ephemeral foothold for an actual permanent flash install of OpenWRT (sysupgrade), mind the following caveat:

  • the box asks for DHCP and TFTP on the WAN port, labeled #1 on the outside (aka the cursed port #5 internally) ! BUT !
  • once booted, you need to look for OpenWRT at !!! at one of the LAN ports !!! i.e. ports #2 to #5

To persuade the RouterBoard to boot via TFTP, I couldn't seem to get it done using the Reset button, so I ended up using the Mikrotik method No.3: boot RouterOS, connect to the WebFig and configure the board to try "Ethernet once, then NAND" - but contrary to the aforementioned guide, I had to choose BOOTP (rather than DHCP - my ISC dhcpd can do either) and I did not check the mark "Force Backup Booter".
The rest went as described by the guide - once I had OpenWRT net-booted, the sysupgrade via the LUCI HTTP GUI was a piece of cake.

The OpenWRT image that I'm using:

uname -a
Linux AP007 5.4.188 #0 SMP Sat Apr 16 12:59:34 2022 armv7l GNU/Linux

My original desired /etc/config/network that does not quite work (mind the scroll bar on the right):

config device
	option name 'br-lan'
	option type 'bridge'
	list ports 'eth0.1'

config device
	option name 'br-guest'
	option type 'bridge'
	list ports 'eth0.6'

config device
	option name 'eth0'
	option macaddr 'dc:2c:6e:28:60:76'

config interface 'lan'
	option device 'br-lan'
	option proto 'static'
	option ipaddr ''
	option netmask ''
	option gateway ''
	option ip6assign '60'

config switch
	option name 'switch0'
	option reset '1'
	option enable_vlan '1'

config switch_vlan
	option device 'switch0'
	option vlan '1'
	option vid '1'
	option ports '1 2 3 4 0t 5t'

config switch_vlan
	option device 'switch0'
	option vlan '6'
	option vid '6'
	option ports '0t 5t'

config interface 'WLAN'
	option proto 'static'
	option ipaddr ''
	option netmask ''

config interface 'guest'
	option device 'br-guest'
	option proto 'none'

Indeed the infamous "switch port #5 weirdness" is giving me trouble.

Mind the definition of interface "WLAN", which I then mapped to a particular
ESSID in /etc/config/wireless. I was using this for management access while
messing with the config of the physical Ethernet switch.

Port 0 is the internal CPU-bound trunk, port 5 is the cursed WAN/uplink.
Using further manual intervention against VLAN #2 (see below), I can get
swconfig as clean as this (I'm skipping the traffic counters for clarity, and again mind the scrollbar):

Global attributes:
	enable_vlan: 1
Port 0:
	pvid: 0
	link: port:0 link:up speed:1000baseT full-duplex txflow rxflow 
Port 1:
	pvid: 1
	link: port:1 link:down
Port 2:
	pvid: 1
	link: port:2 link:up speed:1000baseT full-duplex auto
Port 3:
	pvid: 1
	link: port:3 link:down
Port 4:
	pvid: 1
	link: port:4 link:down
Port 5:
	pvid: 0
	link: port:5 link:up speed:1000baseT full-duplex auto
	vid: 1
	ports: 0t 1 2 3 4 5t 
	vid: 6
	ports: 0t 5t 

The observed (mis)behavior:

I get egress traffic VLAN-tagged correctly on the output of physical port 5 i.e. the "cursed uplink".

Having a PC connected to the untagged port #2, I can ping just fine to a network connected via the trunk port #5.
I.e. the HW switch matrix does forward, tag and untag my traffic for VLAN 1 just fine between the untagged port #2 and tagged port #5, in both directions.

From the PC on untagged port #2, I can access (ping and ssh) the management of my hAP_AC2 on the inner CPU-bound trunk port #0, via VLAN1 software-decapsulated via eth0.1.
I.e. again the tagging and forwarding works fine for this path, within VLAN1, between port #2 and switch port #0 and Linux eth0.1, in both directions.

What does NOT work: traffic incoming from the cursed trunk port #5, with whatever VLAN tag, gets forwarded (including the tag) obediently to the inner port #0, but ends up internally on eth1, including the VLAN tag as received from the physical cable outside.
I.e., incoming traffic from outer port #5 does not ever reach eth0 !

Curiously enough, the opposite direction works: traffic dispatched by OpenWRT down eth0.1 or down eth0.6 ends up as egress traffic on the cursed trunk port #5, including the desired VLAN tags, and is duly delivered further down my broader network.

My conclusion is: you are essentially free to configure any VLAN tags of your choice in /etc/config/network or via swconfig. Maybe except for VLAN 2 (i.e. 802.1Q VID=2) - more on that below.

The following is possible but does not help (not all the way, or possibly not at all):

echo 0 > /proc/sys/net/edma/default_wan_tag
echo 0 > /proc/sys/net/edma/default_group2_vlan_tag
echo 0 > /proc/sys/net/edma/default_lan_tag
echo 0 > /proc/sys/net/edma/default_group1_vlan_tag

(or an equivalent via /etc/sysctl.conf)
Also tried

echo 3 > /proc/sys/net/edma/default_wan_tag
echo 3 > /proc/sys/net/edma/default_group2_vlan_tag

...which changed literally nothing in the observed misbehavior.

The assignment of VLAN's and tagged/untagged mode per switch port can be configured via /etc/config/network or at runtime just by direct access via swconfig (which is a generic tool, not qualcomm-proprietary).
Things to try in swconfig (various invocations) to get in the picture:

swconfig list
swconfig dev switch0 show
swconfig dev switch0 help

The latter tells you what KEYwords you can set or get.

Somehow miraculously, port 5 gets automagically configured at boot for Vlan=2 vid=2 pvid=2.
This config can be canceled at runtime by swconfig - or can it?

Yes you can un-configure the VLAN using swconfig:

swconfig dev switch0 vlan 2 set ports ''

so that it no longer gets reported back by

swconfig dev switch0 show

You can and you should remove the pvid=2 from the cursed port #5 as well (supposed to control ingress tagging):

swconfig dev switch0 port 5 set pvid 0

Yet, if you then try

tcpdump -e -n -i eth1

and you provide some untagged ingress traffic at port #5, you'll see it coming tagged with vid=2
in your tcpdump on eth1. I haven't checked, but maybe a VLAN tag with vid=2 would get stripped in the egress direction on port #5 (regardless of what current config gets reported by swconfig).

Other than that, from a slightly different angle, maybe the vid=2 does not have any special role in actual switch hardware or driver. So perhaps the advice to strictly avoid VID=2 in your config is a little too harsh...
Is this just a residual artifact of some greater past mischief, or has this always been just a partial misunderstanding?
I mean to suggest that the obsession of LUCI with avoidance of port5 / VLAN2 is possibly moot. I mean LUCI trying to tiptoe around that issue, thus preventing LUCI from embracing a workaround that I dare to suggest below.

One more example of swconfig -- VLAN1 done my way, in a one-liner:

swconfig dev switch0 vlan 1 set ports '0t 1 2 3 4 5t'

And in spite of all efforts to get a clean config, some ingress traffic gets forwarded from the cursed port #5 to the inner gmac #2 aka eth1 and this forwarding is "transparent" / ignorant of the VLAN config accessible to me.
After all, gmac #1 = eth0 and gmac #2 = eth1 both correspond to the switch matrix inner CPU-bound port #0 - so there's no bloody violation of rules, right :slight_smile:

Clearly, swconfig does not have full control over the HW peculiarities of this particular switch chip model (or NIC driver?), namely of the essedma driver's magic in forwarding traffic from port5 strictly to "gmac #2".

I've noticed someone's forum post (cannot find it now) that the switch matrix subsystem internally uses a proprietary packet header or tag, to implement this strict source-PHY-based delivery to the desired GMAC port. Surprise, surprise. If you were a magical proprietary per-packet forwarding mechanism, how would you be implemented to disguise yourself :slight_smile:

I've also tried following up on some patches suggested earlier (already in 2017/2019) and I have to say that I'm too lazy to learn to use the build environment, try to patch new kernel source with old patches (in the way of a script kid) and hope for a positive result...
Just in case someone was interested, here is a summary of relevant keywords and pointers to
get you jump-started:

Keywords: edma essedma ipq4019 qcom,num_gmac

Chunkeey's legendary patch:

Instead, I have:
A suggested workaround (call it a practical hack if you will):

While poking at the switch, and realizing that the gremlins are possibly hiding elsewhere (in how the switch port 0 gets bifurcated into two GMAC interfaces), I kept wondering: so if I have gmac2 for the uplink, and gmac1 for "everything else", could I just bridge the two inner GMAC interfaces together?
I mean - in software, using the soft-bridge available in Linux, used by OpenWRT for quite a bit of its mischief. Actually I prefer to set up an explicit soft-bridge instance for each VLAN-come-virtual-ESSID anyway, for the sake of config clarity. So maybe I could just add eth1 too, in addition to eth0, to the bridge. Or some dotted VLAN subinterfaces on both eth0 and eth1, for that matter. What could ever go wrong, right? :slight_smile:

Well the obvious worry popping up on my mind was: could this bring about a broadcast storm? I would be bridging=returning broadcasts (and unlearned unicasts) back to the switch subsystem. On the surface, this sounds like a classic recipe for a storm.
Looking closer, the two ports are actually just an unholy bifurcation of a single port 0 on the switch. So if I send a broadcast packet to switch port 0, the switch will not return it to me immediately on port 0, which is how an actual storm gets prevented. (A sigh of relief.)
I did test the idea right away, and it did seem to work - so far so good.


All I needed was to add the eth1.VLAN subinterfaces to the soft-bridge instances, in /etc/config/network:

config device
        option name 'br-lan'
        option type 'bridge'
        list ports 'eth0.1'
        list ports 'eth1.1'

config device
        option name 'br-guest'
        option type 'bridge'
        list ports 'eth0.6'
        list ports 'eth1.6'

The config of the internal HW switch can remain the same! because it does not address the bifurcation of gmac1/gmac2 in any way.

The catch

It all seemd to work, except that... somehow, my production traffic suddenly suffered from hiccups / interruptions. And, in some directions, ARP (arping) showed frequent outages or even did not seem to work at all (no responses). And printing went awry. And, then I realized that some machines in my network would intermittently suffer a general loss of connectivity. Lots of lost packets.

By that time the principal error in my hack has dawned on me: by bouncing every broadcast
packet back to the switch (actually up the whole network) once again, I was forcing all the switches to re-learn the source MAC instantly, often to appear to come from a different direction than where the originating machine actually was.
Such as, if my fileserver or my default gatway sent an "ARP who-has", the L2 forwarding entries across the whole network would instantly point to my OpenWRT AP somewhere at the network's edge...

Which made me wonder: could I somehow prevent the soft-bridge instances from forwarding traffic "traversing from eth0 to eth1 and vice versa" ? Brctl doesn't seem to have native commands for that sort of functionality. I'd better keep MAC address learning basically intact. I just needed to prevent that one combination of source and destination. I checked if maybe veth, macvlan or the bonding driver had something to offer, but no luck. I also remembered ebtables, and gave their manpage a fumble. And, this is where apparently I found the clue I needed.

EBtables to the rescue

Long story short, this is what I have in my rc.local (for the lack of a more appropriate place to put these rules):

ebtables -F FORWARD
ebtables -A FORWARD -i eth0.1 -o eth1.1 -j DROP
ebtables -A FORWARD -i eth1.1 -o eth0.1 -j DROP
ebtables -A FORWARD -i eth0.6 -o eth1.6 -j DROP
ebtables -A FORWARD -i eth1.6 -o eth0.6 -j DROP

I.e., prevent forwarding from eth0 to eth1 or the other direction, and each VLAN has a soft-bridge instance of its own. I've tried setting the rules on "physical" eth0 and eth1 (omitting the dotted subinterfaces) and that did not work. Which makes sense: if the soft-bridge instance gets the VLAN subinterfaces as members.

While configuring this, I was worried if that ebtables rule is perhaps too broad a broom for a packet carrying a broadcast destination. The question to me was, at what stage of forwarding of the broadcast packet, does the rule get applied:

A) before the packet gets split to individual bridge egress ports, or
B) after the multiplication, i.e. applied per egress port?

How does the "-o" rule even apply to a broadcast destination?
As far as I'm aware, there's no KPTD for ebtables... but, long story short, based on a simple
practical test, it does work the way I need :slight_smile:
The broadcast packets do get forwarded in all egress directions, except the one that I have banned using those specific ebtables rules.
Gives you an insight into how "broadcast forwarding" actually works under the hood.

Yes it does mean some extra CPU-bound processing, in a "gigabit-grade network element".
But, note that the situation is not necessarily all that bad:

  • traffic between outer RJ45 ports of the hAP AC2 does not get affected = does get switched and tagged/untagged by the switch hardware
  • the workaround/hack only applies to traffic destined to the WiFi radio segment (or the local management CPU of the hAP AC2).
  • i.e., this workaround is different from "using the cursed port #5 as a dedicated L3 interface, and routing between this one and the rest using the CPU" ! Again my workaround needs zero CPU horsepower for traffic passing "from RJ45 to RJ45".
  • the box has a quad-core CPU! Lots of horsepower (remember to install irqbalance and enable it in /etc/config/irqbalance, check its effect in /proc/interrupts)
  • this setup with two interfaces bridged could actually yield some load balancing between the two GMAC interfaces: as mentioned before, one caters for the cursed uplink, the other for "the rest". I.e., at a "peripheral location where you need a VLAN-aware switch", you get a dedicated Gigabit bandwidth for your L2 LAN uplink and another dedicated Gb for the "nearest local metallic neighborhood" of the hAP AC (any local computer / NAS / printer, you name it). The radioes in the hAP AC aren't exactly slow

CAVEAT: if you wondered about using the cursed port #5 obediently as an exclusive passthrough to the inner eth1, and just bridging eth1 to eth0 at layer 2 in software (= using CPU horsepower for every packet), note that my "pitfall of bounced packets wreaking havoc to learned L2 FDB entries" applies to your scenario as well :slight_smile:
This is because the traffic would still be passing twice through the same L2 switch, with the same source MAC address, even if just the inner switch of the hAP AC2, this time not your whole network. So the failures would be more "localized" :slight_smile:
This is because the switch does not feature an isolated per-VLAN FDB. Note that this is different from machines that have a dedicated WAN port on a dedicated NIC, separate from a built-in LAN switch (no problem to soft-bridge in the latter scenario).

Speaking of FDB's, I haven't found a way to dump the dynamic FDB of the hardware switch in the ipq40xx. The swconfig does not provide such an option. If you know about a way, let me know. It is my #1 serious downside of these "managed" built-in switches...

BTW, failsafe mode works just right on his hardware. Saved my day once or twice :slight_smile:

1 Like

I suggest you check out https://github.com/openwrt/openwrt/pull/4721

This PR adds DSA support for the ipq40xx target and fixes the issue mentioned in this thread. Check if your board is migrated (its easy to migrate) and give it a try. I have it running for more than 6 months and it seems pretty stable.


Is it possible to use WAN port as another LAN port? Need the DSA support patch?

Yes, you can. You can bridge eth1 with eth0.

Cannge in /etc/config/network

config device
	option name 'br-lan'
	option type 'bridge'
	list ports 'eth0'


config device
	option name 'br-lan'
	option type 'bridge'
	list ports 'eth0 eth1'

I guess vlans 1 and 2 are not needed in this case, but that requires testing.

And you obviously need to remove wan and wan6 interfaces if they are using eth1.


1 Like

Should be

list ports 'eth0'
list ports 'eth1'
1 Like

That's correct. I forgot the syntax. Couldn't find it in the docs easily.

1 Like

That's great, I already had interfaces WAN and WAN6 disabled, so I only had to add eth1 to bridge br-lan as per your advice, Thank you

config device
	option name 'br-lan'
	option type 'bridge'
	list ports 'eth0'
	list ports 'eth1'

Thank you for the syntax