Slow network with VLAN tagging

Installing and Using OpenWrt
1

I have configured a TP-Link Archer C6 v2 as dumb AP, and have OPNsense in Proxmox running as the firewall. However the Proxmox box only have one ethernet port, so I have VLAN tagging as this:

image
image1297×335 17.4 KB

Upstream Internet connected to WAN port, Proxmox/OPNsense into LAN1.

After the setup, things are working, but the network speed is slow. I have 1Gbps symmetrical internet, but end devices connected to the AP only gets like 10Mbps and it fluctuates a lot, sometimes down to 1Mbps.

So for testing I have used iperf3.

Devices' IPs:
OPNsense in Proxmox to LAN1 - 192.168.1.1 (1)
OpenWRT dumb AP - 192.168.1.2 (2)
VM1 in Proxmox to LAN1 - 192.168.1.51 (51)
VM2 in Proxmox to LAN1 - 192.168.1.52 (52)
Laptop connected to LAN2 - 192.168.1.99 (99)

iperf3 results:
(51) to (99): 950Mbps (expected speed)
(99) to (51): 3Mbps (fluctuates between 0 - 7 Mbps)

(51) to (52): 5Gbps (because only internally without going out any ethernet cables)

(51) to (2): 360Mbps
(2) to (51): 70Mbps (fluctuates between 45 - 115 Mbps)

(99) to (2): 340Mbps
(2) to (99): 260Mbps

(51) to Internet: 350 Up/150 Down

All ethernet connections are 1Gbps ports with at least CAT5e cables.

So obviously all connections should be at near 1Gbps, but they are not. Is this because of VLAN tagging or did I configure something wrongly? I already have software offloading enabled.

2

I don’t immediately think that this is related to openwrt (but it could be). We need more data.

Wha happens if you connect your laptop directly to the proxmox device and don’t use the openwrt device at all. If your laptop supports VLANs and can be configured with tagged networks, that is ideal since you can test directly without any changes on the proxmox side. If not, set one of the networks (on proxmox/pfsense) to untagged so that your laptop can connect to it.

3

Hmm, indeed even if I connect my laptop to proxmox with VLAN tagging directly, I'm still getting similar speed. I will have to diagnose proxmox then.

Do you know that if VLAN tagging on OpenWRT will have performance penalty? Because every packets have to go from OpenWRT to proxmox and back and then only to internet and vice versa, does this cut my ethernet speed in half? I'm still trying to decide if I should continue this setup or get another network interface for the proxmox server.

4

With one Ethernet port, using the Internet - yes.

5

Yeah, that was my thinking. It could be any number of things going on, but it is not related to OpenWrt.

As long as the ports that you are using on your OpenWrt device are all connected to the same physical switch chip, there should be no performance penalty with respect to switching VLANs. If the ports are individually routed (sometimes the physical WAN port is not part of the switch on a 4+1 port all-in-one type router), there could be significant penalties, but it depends on the hardware.

Actually, yes and no. In some tests (such as internet speed tests) with an otherwise quiet network (no significant traffic between any two networks aside from the test itself), you can actually approach full line speeds, provided that your interface is running in full duplex (which is required for gigabit). This is because the interface will actually be able to support 1Gbps in each direction (full duplex). However, in real world, practical use, you could see your speeds suffer... cut as much as half in some cases, depending on the nature of the inter-VLAN traffic.

If your internet speed is <500Mbps and you don't have much inter-VLAN traffic, you will be fine with the single cable arrangement (sometimes called a "router on a stick").

But regardless of what I said above, yes, I think you should... you'll get better performance this way.

Something is wrong with your proxmox system (or maybe your laptop or your test methodology) in terms of the speeds it can achieve over the wire. You need to figure that out first, then you can worry about the rest. In the current state, even if you do add another physical interface to your proxmox system, you still won't get good speeds because something is not quite right.

Regarding your tests -- consider that iPerf is only good for testing between devices with general purpose CPUs. This is because an embedded CPU/SoC (such as the ones you'll find in most router hardware) are great at routing (i.e. passing traffic through the chip), but not good at sourcing/sinking the packets at the same speeds. Case in point, I have multiple routers that can route at full gigabit speeds, but speed tests run on the device will typically max out around 250Mbps.