IPQ40xx Switch Config "Strangeness" (swconfig)

Let me follow up on this topic in May 2022.
I have a summary of what status quo I'm facing, I have two sketches, and I also have a possible workaround :slight_smile: in terms of user-space configuration (no recompilation needed).

My HW:
Mikrotik hAP AC2
RBD52G-5HacD2HnD-TC

Caveat #1: on the plastic shell of this hardware, on the printed labels, the RJ45 ports are labeled "in reverse order" compared to the port numbers reported by swconfig etc. I.e., the cursed WAN port, internally known as port 5, is labeled "#1 - Internet/PoE in" on the outside.

Caveat #2: if you use the DHCP+TFTP method to boot the initial "initrd+kernel" as an ephemeral foothold for an actual permanent flash install of OpenWRT (sysupgrade), mind the following caveat:

  • the box asks for DHCP and TFTP on the WAN port, labeled #1 on the outside (aka the cursed port #5 internally) ! BUT !
  • once booted, you need to look for OpenWRT at 192.168.1.1 !!! at one of the LAN ports !!! i.e. ports #2 to #5

To persuade the RouterBoard to boot via TFTP, I couldn't seem to get it done using the Reset button, so I ended up using the Mikrotik method No.3: boot RouterOS, connect to the WebFig and configure the board to try "Ethernet once, then NAND" - but contrary to the aforementioned guide, I had to choose BOOTP (rather than DHCP - my ISC dhcpd can do either) and I did not check the mark "Force Backup Booter".
The rest went as described by the guide - once I had OpenWRT net-booted, the sysupgrade via the LUCI HTTP GUI was a piece of cake.

The OpenWRT image that I'm using:
https://downloads.openwrt.org/releases/21.02.3/targets/ipq40xx/mikrotik/openwrt-21.02.3-ipq40xx-mikrotik-mikrotik_hap-ac2-squashfs-sysupgrade.bin

uname -a
Linux AP007 5.4.188 #0 SMP Sat Apr 16 12:59:34 2022 armv7l GNU/Linux

My original desired /etc/config/network that does not quite work (mind the scroll bar on the right):

config device
	option name 'br-lan'
	option type 'bridge'
	list ports 'eth0.1'

config device
	option name 'br-guest'
	option type 'bridge'
	list ports 'eth0.6'

config device
	option name 'eth0'
	option macaddr 'dc:2c:6e:28:60:76'

config interface 'lan'
	option device 'br-lan'
	option proto 'static'
	option ipaddr '192.168.2.147'
	option netmask '255.255.254.0'
	option gateway '192.168.2.1'
	option ip6assign '60'

config switch
	option name 'switch0'
	option reset '1'
	option enable_vlan '1'

config switch_vlan
	option device 'switch0'
	option vlan '1'
	option vid '1'
	option ports '1 2 3 4 0t 5t'

config switch_vlan
	option device 'switch0'
	option vlan '6'
	option vid '6'
	option ports '0t 5t'

config interface 'WLAN'
	option proto 'static'
	option ipaddr '192.168.6.1'
	option netmask '255.255.255.0'

config interface 'guest'
	option device 'br-guest'
	option proto 'none'

Indeed the infamous "switch port #5 weirdness" is giving me trouble.

Mind the definition of interface "WLAN", which I then mapped to a particular
ESSID in /etc/config/wireless. I was using this for management access while
messing with the config of the physical Ethernet switch.

Port 0 is the internal CPU-bound trunk, port 5 is the cursed WAN/uplink.
Using further manual intervention against VLAN #2 (see below), I can get
swconfig as clean as this (I'm skipping the traffic counters for clarity, and again mind the scrollbar):

Global attributes:
	enable_vlan: 1
Port 0:
	pvid: 0
	link: port:0 link:up speed:1000baseT full-duplex txflow rxflow 
Port 1:
	pvid: 1
	link: port:1 link:down
Port 2:
	pvid: 1
	link: port:2 link:up speed:1000baseT full-duplex auto
Port 3:
	pvid: 1
	link: port:3 link:down
Port 4:
	pvid: 1
	link: port:4 link:down
Port 5:
	pvid: 0
	link: port:5 link:up speed:1000baseT full-duplex auto
VLAN 1:
	vid: 1
	ports: 0t 1 2 3 4 5t 
VLAN 6:
	vid: 6
	ports: 0t 5t 

The observed (mis)behavior:

I get egress traffic VLAN-tagged correctly on the output of physical port 5 i.e. the "cursed uplink".

Having a PC connected to the untagged port #2, I can ping just fine to a network connected via the trunk port #5.
I.e. the HW switch matrix does forward, tag and untag my traffic for VLAN 1 just fine between the untagged port #2 and tagged port #5, in both directions.

From the PC on untagged port #2, I can access (ping and ssh) the management of my hAP_AC2 on the inner CPU-bound trunk port #0, via VLAN1 software-decapsulated via eth0.1.
I.e. again the tagging and forwarding works fine for this path, within VLAN1, between port #2 and switch port #0 and Linux eth0.1, in both directions.

What does NOT work: traffic incoming from the cursed trunk port #5, with whatever VLAN tag, gets forwarded (including the tag) obediently to the inner port #0, but ends up internally on eth1, including the VLAN tag as received from the physical cable outside.
I.e., incoming traffic from outer port #5 does not ever reach eth0 !

Curiously enough, the opposite direction works: traffic dispatched by OpenWRT down eth0.1 or down eth0.6 ends up as egress traffic on the cursed trunk port #5, including the desired VLAN tags, and is duly delivered further down my broader network.

My conclusion is: you are essentially free to configure any VLAN tags of your choice in /etc/config/network or via swconfig. Maybe except for VLAN 2 (i.e. 802.1Q VID=2) - more on that below.

The following is possible but does not help (not all the way, or possibly not at all):

echo 0 > /proc/sys/net/edma/default_wan_tag
echo 0 > /proc/sys/net/edma/default_group2_vlan_tag
echo 0 > /proc/sys/net/edma/default_lan_tag
echo 0 > /proc/sys/net/edma/default_group1_vlan_tag

(or an equivalent via /etc/sysctl.conf)
Also tried

echo 3 > /proc/sys/net/edma/default_wan_tag
echo 3 > /proc/sys/net/edma/default_group2_vlan_tag

...which changed literally nothing in the observed misbehavior.

The assignment of VLAN's and tagged/untagged mode per switch port can be configured via /etc/config/network or at runtime just by direct access via swconfig (which is a generic tool, not qualcomm-proprietary).
Things to try in swconfig (various invocations) to get in the picture:

swconfig
swconfig list
swconfig dev switch0 show
swconfig dev switch0 help

The latter tells you what KEYwords you can set or get.

Somehow miraculously, port 5 gets automagically configured at boot for Vlan=2 vid=2 pvid=2.
This config can be canceled at runtime by swconfig - or can it?

Yes you can un-configure the VLAN using swconfig:

swconfig dev switch0 vlan 2 set ports ''

so that it no longer gets reported back by

swconfig dev switch0 show

You can and you should remove the pvid=2 from the cursed port #5 as well (supposed to control ingress tagging):

swconfig dev switch0 port 5 set pvid 0

Yet, if you then try

tcpdump -e -n -i eth1

and you provide some untagged ingress traffic at port #5, you'll see it coming tagged with vid=2
in your tcpdump on eth1. I haven't checked, but maybe a VLAN tag with vid=2 would get stripped in the egress direction on port #5 (regardless of what current config gets reported by swconfig).

Other than that, from a slightly different angle, maybe the vid=2 does not have any special role in actual switch hardware or driver. So perhaps the advice to strictly avoid VID=2 in your config is a little too harsh...
Is this just a residual artifact of some greater past mischief, or has this always been just a partial misunderstanding?
I mean to suggest that the obsession of LUCI with avoidance of port5 / VLAN2 is possibly moot. I mean LUCI trying to tiptoe around that issue, thus preventing LUCI from embracing a workaround that I dare to suggest below.

One more example of swconfig -- VLAN1 done my way, in a one-liner:

swconfig dev switch0 vlan 1 set ports '0t 1 2 3 4 5t'

And in spite of all efforts to get a clean config, some ingress traffic gets forwarded from the cursed port #5 to the inner gmac #2 aka eth1 and this forwarding is "transparent" / ignorant of the VLAN config accessible to me.
After all, gmac #1 = eth0 and gmac #2 = eth1 both correspond to the switch matrix inner CPU-bound port #0 - so there's no bloody violation of rules, right :slight_smile:

Clearly, swconfig does not have full control over the HW peculiarities of this particular switch chip model (or NIC driver?), namely of the essedma driver's magic in forwarding traffic from port5 strictly to "gmac #2".

I've noticed someone's forum post (cannot find it now) that the switch matrix subsystem internally uses a proprietary packet header or tag, to implement this strict source-PHY-based delivery to the desired GMAC port. Surprise, surprise. If you were a magical proprietary per-packet forwarding mechanism, how would you be implemented to disguise yourself :slight_smile:

I've also tried following up on some patches suggested earlier (already in 2017/2019) and I have to say that I'm too lazy to learn to use the build environment, try to patch new kernel source with old patches (in the way of a script kid) and hope for a positive result...
Just in case someone was interested, here is a summary of relevant keywords and pointers to
get you jump-started:

Keywords: edma essedma ipq4019 qcom,num_gmac

Chunkeey's legendary patch:

Instead, I have:
A suggested workaround (call it a practical hack if you will):


While poking at the switch, and realizing that the gremlins are possibly hiding elsewhere (in how the switch port 0 gets bifurcated into two GMAC interfaces), I kept wondering: so if I have gmac2 for the uplink, and gmac1 for "everything else", could I just bridge the two inner GMAC interfaces together?
I mean - in software, using the soft-bridge available in Linux, used by OpenWRT for quite a bit of its mischief. Actually I prefer to set up an explicit soft-bridge instance for each VLAN-come-virtual-ESSID anyway, for the sake of config clarity. So maybe I could just add eth1 too, in addition to eth0, to the bridge. Or some dotted VLAN subinterfaces on both eth0 and eth1, for that matter. What could ever go wrong, right? :slight_smile:

Well the obvious worry popping up on my mind was: could this bring about a broadcast storm? I would be bridging=returning broadcasts (and unlearned unicasts) back to the switch subsystem. On the surface, this sounds like a classic recipe for a storm.
Looking closer, the two ports are actually just an unholy bifurcation of a single port 0 on the switch. So if I send a broadcast packet to switch port 0, the switch will not return it to me immediately on port 0, which is how an actual storm gets prevented. (A sigh of relief.)
I did test the idea right away, and it did seem to work - so far so good.

Bridging

All I needed was to add the eth1.VLAN subinterfaces to the soft-bridge instances, in /etc/config/network:

config device
        option name 'br-lan'
        option type 'bridge'
        list ports 'eth0.1'
        list ports 'eth1.1'

config device
        option name 'br-guest'
        option type 'bridge'
        list ports 'eth0.6'
        list ports 'eth1.6'

The config of the internal HW switch can remain the same! because it does not address the bifurcation of gmac1/gmac2 in any way.

The catch

It all seemd to work, except that... somehow, my production traffic suddenly suffered from hiccups / interruptions. And, in some directions, ARP (arping) showed frequent outages or even did not seem to work at all (no responses). And printing went awry. And, then I realized that some machines in my network would intermittently suffer a general loss of connectivity. Lots of lost packets.

By that time the principal error in my hack has dawned on me: by bouncing every broadcast
packet back to the switch (actually up the whole network) once again, I was forcing all the switches to re-learn the source MAC instantly, often to appear to come from a different direction than where the originating machine actually was.
Such as, if my fileserver or my default gatway sent an "ARP who-has", the L2 forwarding entries across the whole network would instantly point to my OpenWRT AP somewhere at the network's edge...

Which made me wonder: could I somehow prevent the soft-bridge instances from forwarding traffic "traversing from eth0 to eth1 and vice versa" ? Brctl doesn't seem to have native commands for that sort of functionality. I'd better keep MAC address learning basically intact. I just needed to prevent that one combination of source and destination. I checked if maybe veth, macvlan or the bonding driver had something to offer, but no luck. I also remembered ebtables, and gave their manpage a fumble. And, this is where apparently I found the clue I needed.

EBtables to the rescue

Long story short, this is what I have in my rc.local (for the lack of a more appropriate place to put these rules):

ebtables -P FORWARD ACCEPT
ebtables -F FORWARD
ebtables -A FORWARD -i eth0.1 -o eth1.1 -j DROP
ebtables -A FORWARD -i eth1.1 -o eth0.1 -j DROP
ebtables -A FORWARD -i eth0.6 -o eth1.6 -j DROP
ebtables -A FORWARD -i eth1.6 -o eth0.6 -j DROP

I.e., prevent forwarding from eth0 to eth1 or the other direction, and each VLAN has a soft-bridge instance of its own. I've tried setting the rules on "physical" eth0 and eth1 (omitting the dotted subinterfaces) and that did not work. Which makes sense: if the soft-bridge instance gets the VLAN subinterfaces as members.

While configuring this, I was worried if that ebtables rule is perhaps too broad a broom for a packet carrying a broadcast destination. The question to me was, at what stage of forwarding of the broadcast packet, does the rule get applied:

A) before the packet gets split to individual bridge egress ports, or
B) after the multiplication, i.e. applied per egress port?

How does the "-o" rule even apply to a broadcast destination?
As far as I'm aware, there's no KPTD for ebtables... but, long story short, based on a simple
practical test, it does work the way I need :slight_smile:
The broadcast packets do get forwarded in all egress directions, except the one that I have banned using those specific ebtables rules.
Gives you an insight into how "broadcast forwarding" actually works under the hood.

Yes it does mean some extra CPU-bound processing, in a "gigabit-grade network element".
But, note that the situation is not necessarily all that bad:

  • traffic between outer RJ45 ports of the hAP AC2 does not get affected = does get switched and tagged/untagged by the switch hardware
  • the workaround/hack only applies to traffic destined to the WiFi radio segment (or the local management CPU of the hAP AC2).
  • i.e., this workaround is different from "using the cursed port #5 as a dedicated L3 interface, and routing between this one and the rest using the CPU" ! Again my workaround needs zero CPU horsepower for traffic passing "from RJ45 to RJ45".
  • the box has a quad-core CPU! Lots of horsepower (remember to install irqbalance and enable it in /etc/config/irqbalance, check its effect in /proc/interrupts)
  • this setup with two interfaces bridged could actually yield some load balancing between the two GMAC interfaces: as mentioned before, one caters for the cursed uplink, the other for "the rest". I.e., at a "peripheral location where you need a VLAN-aware switch", you get a dedicated Gigabit bandwidth for your L2 LAN uplink and another dedicated Gb for the "nearest local metallic neighborhood" of the hAP AC (any local computer / NAS / printer, you name it). The radioes in the hAP AC aren't exactly slow
    :slight_smile:

CAVEAT: if you wondered about using the cursed port #5 obediently as an exclusive passthrough to the inner eth1, and just bridging eth1 to eth0 at layer 2 in software (= using CPU horsepower for every packet), note that my "pitfall of bounced packets wreaking havoc to learned L2 FDB entries" applies to your scenario as well :slight_smile:
This is because the traffic would still be passing twice through the same L2 switch, with the same source MAC address, even if just the inner switch of the hAP AC2, this time not your whole network. So the failures would be more "localized" :slight_smile:
This is because the switch does not feature an isolated per-VLAN FDB. Note that this is different from machines that have a dedicated WAN port on a dedicated NIC, separate from a built-in LAN switch (no problem to soft-bridge in the latter scenario).

Speaking of FDB's, I haven't found a way to dump the dynamic FDB of the hardware switch in the ipq40xx. The swconfig does not provide such an option. If you know about a way, let me know. It is my #1 serious downside of these "managed" built-in switches...

BTW, failsafe mode works just right on his hardware. Saved my day once or twice :slight_smile:

1 Like

I suggest you check out https://github.com/openwrt/openwrt/pull/4721

This PR adds DSA support for the ipq40xx target and fixes the issue mentioned in this thread. Check if your board is migrated (its easy to migrate) and give it a try. I have it running for more than 6 months and it seems pretty stable.

4 Likes

Is it possible to use WAN port as another LAN port? Need the DSA support patch?

Yes, you can. You can bridge eth1 with eth0.

Cannge in /etc/config/network

config device
	option name 'br-lan'
	option type 'bridge'
	list ports 'eth0'

to

config device
	option name 'br-lan'
	option type 'bridge'
	list ports 'eth0 eth1'

I guess vlans 1 and 2 are not needed in this case, but that requires testing.

And you obviously need to remove wan and wan6 interfaces if they are using eth1.

No.

1 Like

Should be

list ports 'eth0'
list ports 'eth1'
3 Likes

That's correct. I forgot the syntax. Couldn't find it in the docs easily.

2 Likes

That's great, I already had interfaces WAN and WAN6 disabled, so I only had to add eth1 to bridge br-lan as per your advice, Thank you

config device
	option name 'br-lan'
	option type 'bridge'
	list ports 'eth0'
	list ports 'eth1'

Thank you for the syntax

Thanks for this. I had a strange issue with a Linksys ea6350 where mDNS traffic wasn't being propagated between the lan and wan ports, even though I'd combined them with swconfig. I'll see if this fixes it.

This won't fix anything with wan port. This setting is for a case where there is no wan interface.

Yes, that's my use case.

I wasted my time porting DSA configuration back to swconfig to make it work with a router with IPQ4019 chipset. It took me another hour of headache only to find that the switch had serious problem and it did not follow the configuration file at all.

Tested on 22.03.0.

The DSA PR was merged to master, it will be included in a future release :slight_smile:

3 Likes

From what I have heard, DSA cannot support devices with 2 physical interfaces. Mine is the MobiPromo CM520-79F which has eth0 (CPU port) and eth1 (dedicated WAN port). I don't know if it will be in the list of DSA.

Are you sure that's the case? I believe all ipq40xx boards reported two interfaces due to how the driver is written. They have hardcoded VLANs for each interface which is what causes the "strangeness". With DSA you are going to see one interface and multiple ports. Also no more hardcoded VLANs. I suggest you open an issue in GitHub and get it converted, it should be rather easy!

After a fresh install of OpenWrt 22.03.0, here is what it reports:

root@OpenWrt:~# swconfig dev switch0 show
[...]
VLAN 1:
    vid: 1
    ports: 0 3 4

VLAN 2:
vid: 2
ports: 0t 5
  • Port 0 is the switch port connected to eth0
  • Port 3 is the switch port labeled LAN2
  • Port 4 is the switch port labeled LAN1
  • Port 5 is nowhere to be found. I suspect it to be a hidden, internal, unused port.
  • WAN port is reported as eth1

I've found a workaround. The router completely freezes if I try to add option vlan '2' to /etc/config/network. Avoiding VLAN 2 and it will work properly. I can still configure VLAN 3 with option vid '2' without making it freeze, but I don't have time to test if VID 2 actually works or not.

I believe all ipq40xx boards reported two interfaces due to how the driver is written. They have hardcoded VLANs for each interface which is what causes the "strangeness".

With your suggestion, now I am suspecting Port 5 is actually the WAN port with a separated hardcoded VLAN 2 (probably with VID 2 as well). I am now using the WAN port for my local network, and 2 LAN ports for 2 remote sites. One of the 2 remote sites has a VLAN with VID=2. I don't know if trying to create a VLAN with VID=2 on the LAN port (connected to remote site) will mix that L2 network with my local network (WAN port of the router) or not.

Yes, port 5 is the WAN port and also its not shown in the UI. You can specify it in the config manually and it will work. Do not use VLAN 1 and 2 and you should be fine.

Here's mine with 22.03.0 (non DSA!):

VLAN 1:
        vid: 1
        ports: 0 1 2 3 4
VLAN 2:
        vid: 2
        ports: 0t 5
VLAN 20:
        vid: 20
        ports: 0t 3t 4t
VLAN 40:
        vid: 40
        ports: 0t 3t 4t
VLAN 50:
        vid: 50
        ports: 0t 3t 4t

Alternatively, just port it to DSA. Much simpler, works like charm and no hidden magic.

This looks exactly like my GL.iNET GL-B1300, before the DSA:

root@GL-B1300-078 ~ # swconfig dev switch0 show|tail
TxLateCol   : 0

	pvid: 2
	link: port:5 link:down
VLAN 1:
	vid: 1
	ports: 0 3 4 
VLAN 2:
	vid: 2
	ports: 0t 5 

GL-B1300 is converted, so you could get a hint.

ipq40xx physically only has a single CPU port, what you're seeing is just the fake port invented by the old swconfig driver - so no regression here (for ipq40xx, ipq806x would have a real second CPU port).

Do not use VLAN 1 and 2 and you should be fine.

By avoiding VLAN 2, I was able to get the router working instead of freezing. I still tried to use VID 1 on my router because I need to connect to VLAN/VID 1 of a Cisco switch. At first it seemed alright. However I could connect to some address on the Cisco side, while some just gave no responses.

All IP addresses I tried to connect are in the same subnet. Further investigation showed that ARP just failed on some IP, randomly, with the result being 00:00:00:00:00:00. I configured the Cisco to output VLAN 1 untagged, to receive it on my router as VLAN 751 untagged (can be any random VLAN number) and it works now.

The conclusion is: VID 1 and VLAN 2 should be avoided. VLAN 1 and VID 2 are still good to use. Although I've encountered no problem so far, I need some more testing on VID 2.

1 Like