VXLAN performance impact

Installing and Using OpenWrt Network and Wireless Configuration
1

Hello folks,

I have a garage that I’m connecting wirelessly into my main house network. At this stage, I’m testing the setup - not yet fully installed.

I have a working setup with VXLAN over a backhaul 5GHz wireless connection. On each end this is attached to the main bridge, which has VLANs set on it (thus VXLAN, so that I can extend those VLANs into the garage). The VXLAN port on the bridge has all VLANs configured as tagged.

Sadly, I’m seeing performance impact of the VXLAN connection when compared to the backhaul wireless connection. This doesn’t feel right.

iperf3 numbers between the OpenWRT APs:

  • garage → house, backhaul: 350 Mbps
  • garage → house, vxlan: 280 Mbps (~20% loss of bandwidth)
  • house → garage: backhaul: 585 Mbps
  • house → garage, vxlan: 290 Mbps (~50% loss of bandwidth)

I understand that VXLAN is widely used at scale, so wouldn’t expect this level of overhead. Please could somebody offer some pointers?

Thank you!

Hardware/OpenWRT specs:

Hostname wap-garage
Model TP-Link Archer AX23 v1
Architecture MediaTek MT7621 ver:1 eco:3
Target Platform ramips/mt7621
Firmware Version OpenWrt 24.10.4 r28959-29397011cc / LuCI openwrt-24.10 branch 25.292.66247~75e41cb
Hostname wap-eaves
Model Cudy WR3000S v1
Architecture ARMv8 Processor rev 4
Target Platform mediatek/filogic
Firmware Version OpenWrt 24.10.4 r28959-29397011cc / LuCI openwrt-24.10 branch 25.307.78403~d3100e2
2

The impact of vxlan should not be that high. I suspect the usual and known limitations of wireless links.

3

Yes - though I had expected wireless limitations to be accounted for in the “backhaul” numbers. Sync speed is > 1Gbps.

4

You will have significant loss of bandwidth if you have not set mtu accordingly.
I have had great success with 1600 in the backhaul vif and 1500 on the vxlan.
BUT I found setting this in the uci config (or Luci) did not always work - seems to vary with hardware. However setting via ip link set <ifname> mtu xxxx always works.

5

I have had great success with 1600 in the backhaul vif and 1500 on the vxlan.

Thank you. I had thought I’d accounted for this with an MTU of 2048 on the vif. Interestingly, dropping this to 1554 (handles the VXLAN + VLAN overhead) brings marginally increased numbers across the board, but still a similar performance impact:

  • garage → house, backhaul: 395 Mbps
  • garage → house, vxlan: 310 Mbps (~20% loss of bandwidth)
  • house → garage: backhaul: 665 Mbps
  • house → garage, vxlan: 310 Mbps (~50% loss of bandwidth)

Still hoping to track down this issue. Any ideas very welcome.

6

Yes, there is a sweet spot there somewhere - I ended up with 1600 because it also works well and is a nice round number :grin:
Depending on how you have set it, it is worth checking what mtu is actually running at, using ip link

How are you configuring your vxlan tunnel? I found the ipv4 method in (L)uci was very poor, and last time I looked vxlan on ipv6 was not supported in uci.

I ended up setting it up with a script using ipv6 link local to get a high performance point to multi-point tunnel.

You may well have other issues as well, related to the signal quality etc. as @_bernd suggested/suspects.

Note: I also found the ip-full package is required for full vxlan support. Without it things just did not work as expected.

Edit: Here is a typical ip link snippet of my working test system:

16: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1600 qdisc noqueue state UP qlen 1000
    link/ether 94:83:c4:a4:e3:77 brd ff:ff:ff:ff:ff:ff
17: br-tun69: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 96:83:c4:a9:e3:77 brd ff:ff:ff:ff:ff:ff
23: vxlan69: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-tun69 state UNKNOWN qlen 1000
    link/ether 96:83:c4:ac:e3:77 brd ff:ff:ff:ff:ff:ff

There are currently nine devices on the vxtunnel, any one can talk to any other through br-tun69 that functions as a "virtual bridge" connecting all nodes in a true peer to multi-peer network.

Of course you can overlay vlans onto the vxlan tunnel, but if you only have two networks using vlans, you can actually do away with them and just use the tunnel and backhaul as is.
Also, you can configure multiple vxlan tunnels (by tunnel id) and do away with all your vlans (assuming you have multiple).

7

Thank you. I’m actually configuring via Ansible script, dumping a templated /etc/config/network file onto the access point.

ip link shows the correct MTU:

13: phy1-sta0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1554 qdisc noqueue state UP qlen 1000
    link/ether 54:af:97:5d:c2:21 brd ff:ff:ff:ff:ff:ff
14: vxl_1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-eth state UNKNOWN qlen 1000
    link/ether 36:dc:d3:4d:99:0d brd ff:ff:ff:ff:ff:ff

I think I’m going to stick with tunnelled VLANs for simplicity’s sake.

You mention ip-full is necessary for full support - can you elaborate on the issues you were seeing, please?

8

I am configuring a multi-point to multi-point tunnelled network (aka a virtual layer 2 switch) using ipv6 link-local as the transport.
For that ip-full was essential.

That was my initial thought, but setting up multiple independent tunnels is so simple.

Here is the ash script function I use:

create_vxlan_if() {

	ip -6 link add "vxlan$tun_id" \
		address "$vxmac" \
		type vxlan \
		id "$tun_id" \
		dstport 4789 \
		local "$link_local_addr" \
		group "ff02::$tun_id" \
		dev "$transport_bridge_device" \
		ttl 5

	sleep 1 # depends on your hardware, may not be needed or may need to be longer

	ip link set "vxlan$tun_id" up
	ip link set "$transport_ifname" mtu 1600
	ip link set "$transport_bridge_device" mtu 1600
	ip link set "vxlan$tun_id" mtu 1500
}

This creates a vxlan interface vxlan$tun_id
where

  1. $tun_id is the tunnel id eg 69
  2. $vxmac is a unique locally administered mac address for the vxlan interface
  3. $link_local_addr is the link local address of the transport bridge device eg br-lan
  4. ff02::$tun_id is the multicast group address for this tunnel
  5. $transport_bridge_device is the name of the transport bridge device eg br-lan
  6. $transport_ifname is the transport interface name eg phy0-ap0