HOWTO: EdgeRouter X (ERX) as managed DSA switch for a router-on-a-stick

Documenting my adventures with EdgeRouter X (ERX), which was originally used as the main router on my home network, but ended up being a managed switch for a "router-on-a-stick" Mini-PC. Both the Mini-PC router and the ERX run under OpenWrt.

The original problem I had with my ERX is that it was often freezing while running under OpenWrt 19.07.0x, which seems to be a common issue with ERX. I did try troubleshooting it but to no avail.

I then repurposed a spare x86 Mini PC as a "router-on-a-stick", and tried using the ERX as a managed switch for it, like this:

image

That worked, but the ERX was still hanging. I then asked Redditors for recommendations of a decent managed SOHO switch, other than ERX, with an option to isolate the management port. Apparently, the today's choice of those are rather limited. Presumably, TRENDnet TEG-S80ES is capable of isolating its management UI (emulator).

I've also tried reverting the ERX back to the latest sock Ubiquiti EdgeOS and configuring it as a managed Layer-2 switch. I couldn't get it working for the VLAN setup as on the picture above, I gave up after an hour of fiddling.

Then I stumbled upon this Q/A. Apparently, the freezing stops being an issue with the newer MT7621 SoC driver for the DSA mode, which is used by the Linux kernel v5+ in current OpenWrt snapshot builds.

So, I've decided to give my ERX a final try, now under the latest OpenWrt snapshot build. And so far, so good! It's been stable for over a week now.

The snapshot builds of OpenWrt currently don't have the Switch menu for configuring DSA switches in the Luci UI. A manual configuration is required, and it took me quite a while to figure out how to apply it to EdgeRouter X. Here's what I've come up with, for the future reference:

  • Updated, install ip-bridge and ip-full packages (for proper functioning of bridge and ip link set dev $SW type bridge vlan_filtering 1 commands):

    opkg update && opkg install ip-full ip-bridge
    

  • /etc/config/network

I use eth0 for emergency management, it isn't a part of the switch. Normally, the switch can also be managed via a DHCP address it obtains from the Mini-PC router's LAN segment:

config interface 'loopback'
	option ifname 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option packet_steering '1'

config interface 'man'
	option ifname 'eth0'
	option proto 'static'
	option ipaddr '192.168.2.1'
	option netmask '255.255.255.0'
	option delegate '0'

config interface 'switch'
	option ifname 'sw0'
	option proto 'dhcp'
	option delegate '0'

  • /etc/hotplug.d/iface/21-switch

This script does the actual job on manually creating the new sw0 bridge interface and configuring its VLANs. I couldn't think of a better place for running this logic, other than upon the ifup hotplug hook for localhost interface.

#!/bin/sh

[ "$INTERFACE" = loopback ] && [ "$ACTION" = ifup ] || exit 0

# name of the bridge
SW="sw0"

logger "Configuring VLANs on $SW..."

# create the SW bridge
if [ ! -d "/sys/class/net/$SW" ]
then
  ip link add name $SW type bridge
else
  ip link set $SW down
fi

# enable VLAN filtering
ip link set dev $SW type bridge vlan_filtering 1
ip link set dev $SW type bridge vlan_default_pvid 1

logger "VLAN filtering enabled: $(cat /sys/class/net/$SW/bridge/vlan_filtering)"

ip link set dev eth1 master $SW
ip link set dev eth2 master $SW
ip link set dev eth3 master $SW
ip link set dev eth4 master $SW

# clear out the default vlan id 1 on SW
bridge vlan del dev eth1 vid 1
bridge vlan del dev eth2 vid 1
bridge vlan del dev eth3 vid 1
bridge vlan del dev eth4 vid 1
bridge vlan del dev $SW self vid 1

# set vlan for eth1 (WAN1)
bridge vlan add dev eth1 vid 10 pvid untagged

# set vlan for eth2 (WAN2)
bridge vlan add dev eth2 vid 20 pvid untagged

# set vlan for eth3 (LAN)
bridge vlan add dev eth3 vid 30 pvid untagged

# set vlans for eth4 (trunk)
bridge vlan add dev eth4 vid 10
bridge vlan add dev eth4 vid 20
bridge vlan add dev eth4 vid 30

# set vlans of the CPU port (LAN)
bridge vlan add dev $SW self vid 30 pvid untagged

# bring up interfaces
ip link set eth1 up
ip link set eth2 up
ip link set eth3 up
ip link set eth4 up

  • /etc/sysupgrade.conf

We need to add 21-switch to the list of persistent scripts:

## This file contains files and directories that should
## be preserved during an upgrade.

# /etc/example.conf
# /etc/openvpn/

/etc/hotplug.d/iface/21-switch

For completeness, here is /etc/config/network from my Mini-PC router, which has only one Ethernet port and uses driver-level tagged VLANs for WAN1, WAN2 and LAN:

config interface 'loopback'
	option ifname 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'

config device
	option type '8021q'
	option ifname 'eth0'
	option vid '10'
	option name 'vlan10'

config device
	option type '8021q'
	option ifname 'eth0'
	option vid '20'
	option name 'vlan20'

config device
	option type '8021q'
	option ifname 'eth0'
	option vid '30'
	option name 'vlan30'

config interface 'lan'
	option ifname 'vlan30'
	option proto 'static'
	option netmask '255.255.255.0'
	option ipaddr '192.168.3.1'
	option delegate '0'

config interface 'wan'
	option ifname 'vlan10'
	option proto 'dhcp'
	option delegate '0'
	option hostname 'router'

Hope this may help others and my future self. Alternatively, a Raspberry Pi could be used in place of the x86/x64 Mini PC.

5 Likes

I have a selection of "managed" switches here, with various degrees of "managed"/"smart" features. All of them support 802.1q VLANs, ideal for a router-on-a-stick scenario.

All of them are fanless, so they're quiet. Depending on your needs, you might find there are more VLAN-capable switches than initially thought. And you might get lucky and score a bargain on eBay.

Cisco WS-C2960X-8TC-L
Pro: It's Cisco; everyone knows how to use it. Gigabit to all ports. Uplink ports take optional SFPs. Management features include a serial console in case the administrator is a moron and screws up the VLAN config and can no longer manage the thing over the network.
Con: It's Cisco; it's bloody expensive. I got lucky on eBay and picked a working one up cheaply as "spares or repair" (the seller didn't know what he had, but the photo of the LEDs told me it was working).

Cisco WS-C2960C-8TC-L
Pro: It's Cisco; everyone knows how to use it. Gigabit uplinks. Uplink ports take optional SFPs. Management features include a serial console in case the administrator is a moron and screws up the VLAN config and can no longer manage the thing over the network.
Con: It's Cisco; it's bloody expensive. I got lucky on eBay again. Main ports are only 100Mbps, not gigabit.

Netgear M4100-D12G
Pro: 12 gigabit ports, plus 2x SFP uplinks. Serial console port to help the moron administrator recover from the inevitable screw-up.
Con: It's Netgear; it's a bit byzantine to configure compared to Cisco. SFPs are mandatory for uplinks; no choice of RJ45/SFP as with Cisco. It's also expensive, unless you can score a bargain on eBay.

TP-Link T1500G-10PS
Pro: Cheap, dirt cheap, for what it is. Utter bargain. Loaded with management protocols. Gigabit across the board. 8 port RJ45 PoE + 2x SFP uplinks. Trivially easy to configure; TP-Link rips off, er, I mean "pays homage to" Cisco's style.
Con: PoE budget is a measly 53W, not enough to run all 8 ports at full power; fine if your power demands are modest, but less so if you need full 802.3af across the board. TP-Link's attitude to security is poor; the SSH daemon intentionally runs old, weak ciphers, TP-Link openly admits this and deliberately refuses to upgrade the firmware with modern, strong ciphers. No serial console port; if the administrator is a moron the administrator can expect to spend plenty of time with a paperclip. Uplinks are SFP-only; no choice of built-in RJ45.

Zyxel GS1920-8HPv2
Pro: Like the TL1500G-10PS, cheap, dirt cheap for what it is. Also loaded with management protcols. Gigabit across the board. 2x uplinks with choice of RJ45 or SFP for both. PoE budget is correct, unlike TP-Link, with 130W enough to feed all 8 ports with 802.3af.
Con: No serial console port; the boneheaded administrator can expect to spend quality time with a paperclip.

TP-Link TL-SG108Ev4
Pro: Astonishingly cheap for something with 8 ports, 802.1q support and port-mirroring.
Con: No management protocols to speak of. Web admin only; no SSH, SNMP, or other monitoring goodness. No serial console port either, although this switch makes it very difficult for the administrator to be a moron. Still, a paperclip reset isn't too onerous.

Netgear GS108Tv2
Pro: Also astonishingly cheap for an 8-port switch with 802.1q support and port mirroring. Dual firmware slots for failed upgrade recovery. A few management protocols including SNMP. Netgear also provides its own centralised management tool.
Con: No SSH, and the web admin page is a bit... slow. No serial console port, so keep a paperclip handy.

Ubiquiti USW Flex Mini
Pro: It's tiny, the size of a pack of cards. It's ridiculously cheap, too. Powered by either 5V USB-C, or PoE. 802.1q VLAN support. Integrates well with Ubiquiti management tools.
Con: No PoE out, only PoE in. No direct management possible; can only be managed by a UniFi controller.

5 Likes

Thank you, that's a great write-up! I have a TL-SG108E v3, and as much as I like the price and the hardware, my biggest problem with it is that it's discoverable from any port using their Easy Smart Configuration Utility.

With a setup like on my pic, the switch could be discovered from any host on the WAN network (verified by using another router to simulate my ISP).

As you've mentioned, TP-Link's approach to security is subpar, and they don't seem to be interested in fixing this, here's an interesting take: Not So Smart: TP-Link TL-SG105E V3.0 5-Port Gigabit Easy Smart Switch.

I don't mind using TL-SG108E behind the firewall, but it's not good for the router-on-a-stick scenario.

1 Like

Hi @noseratio
I love your HOWTO article.
You helped me very much.
I suffered from the same reason.
I use the same configuration - MAN, 2xWAN, LAN, tagged VLAN
My device is Mikrotik hEX (750gr3)](https://openwrt.org/toh/mikrotik/mikrotik_rb750gr3)
Tons of hours spent with frozen/timed out 19.07.x, lost OpenWrt credibility at home (office) ...

Thank you
odon

1 Like

I love your HOWTO article.

I appreciate your comment, glad it helped :slight_smile: After all, it wasn't an OpenWrt issue, but rather a MediaTek's driver issue. My ERX has been stable for weeks since I've done this.

1 Like

Nice overview. But maybe time to consider the OpenWrt realtek target? This brings the cheap switches to another level wrt stability and features, IMHO. Almost anything you can imagine. Still WiP, but definitely worth checking out if you are in the market for a new switch.

You could for example use the ZyXEL GS1900 series instead of the 1920. And the Netgear GS108Tv3 is also a realtek target device - directly replacing the GS108Tv2

1 Like

Hello @noseratio,

I apologize for bringing up this topic again after such a long time, but I really want to express my sincere gratitude for the incredible guide you have put together. I am also working on a similar solution. Recently, I bought a Raspberry Pi 4 with the intention of using it as a router-on-a-stick, just like you. But I think I need some help at this point.

My ISP provides a FTTH (Fiber-to-the-Home) connection where a fiber cable comes into my living room and connects to the ONT (Optical Network Terminal) device. They use PPPoE with the default VLAN, so I don't need to change the VLAN ID to connect. However, I need to clone the WAN MAC address of my original ISP router, but that's a different story.

Despite trying various methods over the past week, I haven't been able to achieve any results yet. I have a Ubiquiti EdgeRouter X, currently running OpenWrt 23.05.0 RC2, and I want to configure it as a managed switch. Here's what I want to accomplish:

eth0   eth1   eth2   eth3   eth4
wan    trunk  lan    lan   management

Therefore, I made some minor modifications to the script you shared. The final version is as follows. Could you please review it? Additionally, I should mention that I updated the management port in the /etc/config/network file to eth0 -> eth4 to match this configuration.

#!/bin/sh

[ "$INTERFACE" = loopback ] && [ "$ACTION" = ifup ] || exit 0

# name of the bridge
SW="sw0"

logger "Configuring VLANs on $SW..."

# create the SW bridge
if [ ! -d "/sys/class/net/$SW" ]
then
  ip link add name $SW type bridge
else
  ip link set $SW down
fi

# enable VLAN filtering
ip link set dev $SW type bridge vlan_filtering 1
ip link set dev $SW type bridge vlan_default_pvid 1

logger "VLAN filtering enabled: $(cat /sys/class/net/$SW/bridge/vlan_filtering)"

ip link set dev eth0 master $SW
ip link set dev eth1 master $SW
ip link set dev eth2 master $SW
ip link set dev eth3 master $SW

# clear out the default vlan id 1 on SW
bridge vlan del dev eth1 vid 1
bridge vlan del dev eth2 vid 1
bridge vlan del dev eth3 vid 1
bridge vlan del dev eth4 vid 1
bridge vlan del dev $SW self vid 1

# set vlan for eth0 (WAN1)
bridge vlan add dev eth0 vid 1 pvid untagged

# set vlan for eth2 (WAN2)
# bridge vlan add dev eth2 vid 20 pvid untagged

# set vlan for eth3 (LAN)
bridge vlan add dev eth2 vid 10 pvid untagged
bridge vlan add dev eth3 vid 10 pvid untagged

# set vlans for eth1 (trunk)
bridge vlan add dev eth1 vid 1
# bridge vlan add dev eth1 vid 20
bridge vlan add dev eth1 vid 10

# set vlans of the CPU port (LAN)
bridge vlan add dev $SW self vid 10 pvid untagged

# bring up interfaces
ip link set eth0 up
ip link set eth1 up
ip link set eth2 up
ip link set eth3 up

And the default network file is as follows:

config interface 'loopback'
	option device 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option ula_prefix 'fd05:86a9:b25e::/48'
	option packet_steering '1'

config device
	option name 'br-lan'
	option type 'bridge'
	list ports 'eth1'
	list ports 'eth2'
	list ports 'eth3'
	list ports 'eth4'

config interface 'lan'
	option device 'br-lan'
	option proto 'static'
	option ipaddr '192.168.1.1'
	option netmask '255.255.255.0'
	option ip6assign '60'

config interface 'wan'
	option device 'eth0'
	option proto 'dhcp'

config interface 'wan6'
	option device 'eth0'
	option proto 'dhcpv6'

I hope I can repurpose this device so that I won't have to buy a switch.. Thank you :slight_smile:

ulpian

There is no need for a script any more. The /etc/config/network file can set up such a configuration and the existing standard netifd / UCI scripts will deploy it properly.

  • Put all 5 ports into the same bridge.
  • Create a bridge-vlan for each network, with unique VLAN number for each. Since VLAN tags are not needed on the external ports, these numbers can be arbitrary.
  • Attach ports to the apropriate bridge-vlan
  • Create interfaces for each network, with the bridge device (br-lan etc) dot VLAN number as the device (e.g. option device 'br-lan.3') The ones that merely pass through the switch still need a network-- it should be of proto none. The management interface can be the existing lan interface.
2 Likes

Hey @ulpian, I agree with what @mk24 says. I had to struggle with manual switch scripting back in OpenWrt v18 days, but ERX/DSA configuration is very well supported via Luci as of v22.03.5 (I haven't tried v23 RC yet, so can speak about it).

That's great! I'll try to configure it using LuCi. Thank you so much to both of you! @mk24 @noseratio

ulpian

1 Like

Hello again,

Today I had a little time to work on this project. It was sufficient to make the following configurations to use the Ubiquiti EdgeRouter X model with OpenWrt installed as a managed switch only:

eth0 == WAN uplink (VLAN ID 10)
eth1 == Raspberry Pi 4 trunk port (VLAN ID 10 and VLAN ID 20)
eth2 == LAN (VLAN ID 20)
eth3 == LAN (VLAN ID 20)
eth4 == LAN (VLAN ID 20)

In addition, the network configuration on the compatible Raspberry Pi 4 is as follows:

config interface 'loopback'
	option device 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option ula_prefix 'fdc9:279a:5e68::/48'
	option packet_steering '1'

config device
	option name 'br-lan'
	option type 'bridge'
	list ports 'eth0'

config interface 'lan'
	option device 'br-lan.20'
	option proto 'static'
	option ipaddr '10.16.0.1'
	option netmask '255.255.255.0'
	option ip6assign '60'

config bridge-vlan
	option device 'br-lan'
	option vlan '10'
	list ports 'eth0:t'

config bridge-vlan
	option device 'br-lan'
	option vlan '20'
	list ports 'eth0:t'

config interface 'wan'
	option proto 'dhcp'
	option device 'br-lan.10'

For now, I'm using it as a secondary router. I haven't completed the installation as the main router yet. Therefore, it is currently obtaining internet access via DHCP from the main router.

ulpian