HOWTO: EdgeRouter X (ERX) as managed DSA switch for a router-on-a-stick

Documenting my adventures with EdgeRouter X (ERX), which was originally used as the main router on my home network, but ended up being a managed switch for a "router-on-a-stick" Mini-PC. Both the Mini-PC router and the ERX run under OpenWrt.

The original problem I had with my ERX is that it was often freezing while running under OpenWrt 19.07.0x, which seems to be a common issue with ERX. I did try troubleshooting it but to no avail.

I then repurposed a spare x86 Mini PC as a "router-on-a-stick", and tried using the ERX as a managed switch for it, like this:

image

That worked, but the ERX was still hanging. I then asked Redditors for recommendations of a decent managed SOHO switch, other than ERX, with an option to isolate the management port. Apparently, the today's choice of those are rather limited. Presumably, TRENDnet TEG-S80ES is capable of isolating its management UI (emulator).

I've also tried reverting the ERX back to the latest sock Ubiquiti EdgeOS and configuring it as a managed Layer-2 switch. I couldn't get it working for the VLAN setup as on the picture above, I gave up after an hour of fiddling.

Then I stumbled upon this Q/A. Apparently, the freezing stops being an issue with the newer MT7621 SoC driver for the DSA mode, which is used by the Linux kernel v5+ in current OpenWrt snapshot builds.

So, I've decided to give my ERX a final try, now under the latest OpenWrt snapshot build. And so far, so good! It's been stable for over a week now.

The snapshot builds of OpenWrt currently don't have the Switch menu for configuring DSA switches in the Luci UI. A manual configuration is required, and it took me quite a while to figure out how to apply it to EdgeRouter X. Here's what I've come up with, for the future reference:

  • Updated, install ip-bridge and ip-full packages (for proper functioning of bridge and ip link set dev $SW type bridge vlan_filtering 1 commands):

    opkg update && opkg install ip-full ip-bridge
    

  • /etc/config/network

I use eth0 for emergency management, it isn't a part of the switch. Normally, the switch can also be managed via a DHCP address it obtains from the Mini-PC router's LAN segment:

config interface 'loopback'
	option ifname 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option packet_steering '1'

config interface 'man'
	option ifname 'eth0'
	option proto 'static'
	option ipaddr '192.168.2.1'
	option netmask '255.255.255.0'
	option delegate '0'

config interface 'switch'
	option ifname 'sw0'
	option proto 'dhcp'
	option delegate '0'

  • /etc/hotplug.d/iface/21-switch

This script does the actual job on manually creating the new sw0 bridge interface and configuring its VLANs. I couldn't think of a better place for running this logic, other than upon the ifup hotplug hook for localhost interface.

#!/bin/sh

[ "$INTERFACE" = loopback ] && [ "$ACTION" = ifup ] || exit 0

# name of the bridge
SW="sw0"

logger "Configuring VLANs on $SW..."

# create the SW bridge
if [ ! -d "/sys/class/net/$SW" ]
then
  ip link add name $SW type bridge
else
  ip link set $SW down
fi

# enable VLAN filtering
ip link set dev $SW type bridge vlan_filtering 1
ip link set dev $SW type bridge vlan_default_pvid 1

logger "VLAN filtering enabled: $(cat /sys/class/net/$SW/bridge/vlan_filtering)"

ip link set dev eth1 master $SW
ip link set dev eth2 master $SW
ip link set dev eth3 master $SW
ip link set dev eth4 master $SW

# clear out the default vlan id 1 on SW
bridge vlan del dev eth1 vid 1
bridge vlan del dev eth2 vid 1
bridge vlan del dev eth3 vid 1
bridge vlan del dev eth4 vid 1
bridge vlan del dev $SW self vid 1

# set vlan for eth1 (WAN1)
bridge vlan add dev eth1 vid 10 pvid untagged

# set vlan for eth2 (WAN2)
bridge vlan add dev eth2 vid 20 pvid untagged

# set vlan for eth3 (LAN)
bridge vlan add dev eth3 vid 30 pvid untagged

# set vlans for eth4 (trunk)
bridge vlan add dev eth4 vid 10
bridge vlan add dev eth4 vid 20
bridge vlan add dev eth4 vid 30

# set vlans of the CPU port (LAN)
bridge vlan add dev $SW self vid 30 pvid untagged

# bring up interfaces
ip link set eth1 up
ip link set eth2 up
ip link set eth3 up
ip link set eth4 up

  • /etc/sysupgrade.conf

We need to add 21-switch to the list of persistent scripts:

## This file contains files and directories that should
## be preserved during an upgrade.

# /etc/example.conf
# /etc/openvpn/

/etc/hotplug.d/iface/21-switch

For completeness, here is /etc/config/network from my Mini-PC router, which has only one Ethernet port and uses driver-level tagged VLANs for WAN1, WAN2 and LAN:

config interface 'loopback'
	option ifname 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'

config device
	option type '8021q'
	option ifname 'eth0'
	option vid '10'
	option name 'vlan10'

config device
	option type '8021q'
	option ifname 'eth0'
	option vid '20'
	option name 'vlan20'

config device
	option type '8021q'
	option ifname 'eth0'
	option vid '30'
	option name 'vlan30'

config interface 'lan'
	option ifname 'vlan30'
	option proto 'static'
	option netmask '255.255.255.0'
	option ipaddr '192.168.3.1'
	option delegate '0'

config interface 'wan'
	option ifname 'vlan10'
	option proto 'dhcp'
	option delegate '0'
	option hostname 'router'

Hope this may help others and my future self. Alternatively, a Raspberry Pi could be used in place of the x86/x64 Mini PC.

5 Likes

I have a selection of "managed" switches here, with various degrees of "managed"/"smart" features. All of them support 802.1q VLANs, ideal for a router-on-a-stick scenario.

All of them are fanless, so they're quiet. Depending on your needs, you might find there are more VLAN-capable switches than initially thought. And you might get lucky and score a bargain on eBay.

Cisco WS-C2960X-8TC-L
Pro: It's Cisco; everyone knows how to use it. Gigabit to all ports. Uplink ports take optional SFPs. Management features include a serial console in case the administrator is a moron and screws up the VLAN config and can no longer manage the thing over the network.
Con: It's Cisco; it's bloody expensive. I got lucky on eBay and picked a working one up cheaply as "spares or repair" (the seller didn't know what he had, but the photo of the LEDs told me it was working).

Cisco WS-C2960C-8TC-L
Pro: It's Cisco; everyone knows how to use it. Gigabit uplinks. Uplink ports take optional SFPs. Management features include a serial console in case the administrator is a moron and screws up the VLAN config and can no longer manage the thing over the network.
Con: It's Cisco; it's bloody expensive. I got lucky on eBay again. Main ports are only 100Mbps, not gigabit.

Netgear M4100-D12G
Pro: 12 gigabit ports, plus 2x SFP uplinks. Serial console port to help the moron administrator recover from the inevitable screw-up.
Con: It's Netgear; it's a bit byzantine to configure compared to Cisco. SFPs are mandatory for uplinks; no choice of RJ45/SFP as with Cisco. It's also expensive, unless you can score a bargain on eBay.

TP-Link T1500G-10PS
Pro: Cheap, dirt cheap, for what it is. Utter bargain. Loaded with management protocols. Gigabit across the board. 8 port RJ45 PoE + 2x SFP uplinks. Trivially easy to configure; TP-Link rips off, er, I mean "pays homage to" Cisco's style.
Con: PoE budget is a measly 53W, not enough to run all 8 ports at full power; fine if your power demands are modest, but less so if you need full 802.3af across the board. TP-Link's attitude to security is poor; the SSH daemon intentionally runs old, weak ciphers, TP-Link openly admits this and deliberately refuses to upgrade the firmware with modern, strong ciphers. No serial console port; if the administrator is a moron the administrator can expect to spend plenty of time with a paperclip. Uplinks are SFP-only; no choice of built-in RJ45.

Zyxel GS1920-8HPv2
Pro: Like the TL1500G-10PS, cheap, dirt cheap for what it is. Also loaded with management protcols. Gigabit across the board. 2x uplinks with choice of RJ45 or SFP for both. PoE budget is correct, unlike TP-Link, with 130W enough to feed all 8 ports with 802.3af.
Con: No serial console port; the boneheaded administrator can expect to spend quality time with a paperclip.

TP-Link TL-SG108Ev4
Pro: Astonishingly cheap for something with 8 ports, 802.1q support and port-mirroring.
Con: No management protocols to speak of. Web admin only; no SSH, SNMP, or other monitoring goodness. No serial console port either, although this switch makes it very difficult for the administrator to be a moron. Still, a paperclip reset isn't too onerous.

Netgear GS108Tv2
Pro: Also astonishingly cheap for an 8-port switch with 802.1q support and port mirroring. Dual firmware slots for failed upgrade recovery. A few management protocols including SNMP. Netgear also provides its own centralised management tool.
Con: No SSH, and the web admin page is a bit... slow. No serial console port, so keep a paperclip handy.

Ubiquiti USW Flex Mini
Pro: It's tiny, the size of a pack of cards. It's ridiculously cheap, too. Powered by either 5V USB-C, or PoE. 802.1q VLAN support. Integrates well with Ubiquiti management tools.
Con: No PoE out, only PoE in. No direct management possible; can only be managed by a UniFi controller.

5 Likes

Thank you, that's a great write-up! I have a TL-SG108E v3, and as much as I like the price and the hardware, my biggest problem with it is that it's discoverable from any port using their Easy Smart Configuration Utility.

With a setup like on my pic, the switch could be discovered from any host on the WAN network (verified by using another router to simulate my ISP).

As you've mentioned, TP-Link's approach to security is subpar, and they don't seem to be interested in fixing this, here's an interesting take: Not So Smart: TP-Link TL-SG105E V3.0 5-Port Gigabit Easy Smart Switch.

I don't mind using TL-SG108E behind the firewall, but it's not good for the router-on-a-stick scenario.

1 Like

Hi @noseratio
I love your HOWTO article.
You helped me very much.
I suffered from the same reason.
I use the same configuration - MAN, 2xWAN, LAN, tagged VLAN
My device is Mikrotik hEX (750gr3)](https://openwrt.org/toh/mikrotik/mikrotik_rb750gr3)
Tons of hours spent with frozen/timed out 19.07.x, lost OpenWrt credibility at home (office) ...

Thank you
odon

1 Like

I love your HOWTO article.

I appreciate your comment, glad it helped :slight_smile: After all, it wasn't an OpenWrt issue, but rather a MediaTek's driver issue. My ERX has been stable for weeks since I've done this.

1 Like

Nice overview. But maybe time to consider the OpenWrt realtek target? This brings the cheap switches to another level wrt stability and features, IMHO. Almost anything you can imagine. Still WiP, but definitely worth checking out if you are in the market for a new switch.

You could for example use the ZyXEL GS1900 series instead of the 1920. And the Netgear GS108Tv3 is also a realtek target device - directly replacing the GS108Tv2

1 Like