WireGuard performance Site to Site - Flint 2 - 1 Gig internet connections

Installing and Using OpenWrt
1

I have 2x Flint 2 routers, one per location, and both are on Verizon FiOS 1 Gig symmetric connections. The locations are 5 miles apart in the same state. All I’m trying to do is establish a site to site VPN connection with WireGuard between them. And I have already done that. The problem is the performance between the two sites via WireGuard is terrible. Meaning 25-50MBps, using iPerf3. Same speeds in both directions.

Standard internet speed tests put both locations at about 660MBps down and 880MBps up.

I have adjusted MTUs on both sides of the tunnel to all values I’ve seen online with no notable changes in upload/download within the tunnel.

For reference with the same ISP circuits and using 2x ER-4 Edgerouters, I can achieve 300-350MBps without a problem, but that’s because the ER-4s don’t have the CPU capability that the Flint 2s do.

Flint 2 routers are on 23.05.4 Stable (no Gl.Inet firmware in place). I’ve tried all variations of firmware (the Gl.Inet stable and their op24 version) with no performance change.

At this point I’m looking for any ideas on what can be done next, or if anyone has had a similar issue with these routers.

1 Like
2

Maybe your provider caps wireguard thinking it is torrent. Make it 443/udp or something.

2 Likes
3

Make sure to enable mtu fix and make mtu the same on both sides.

Start with 1280 but I have seen reports of users who had to go as low as 400 (although that seems unbelievably low)

My DL-WRX 36 can do 800 Mb/s so yours should easily do over 700.

4
5

So did more testing with MTU values. Started with the lowest OpenWRT would let me use, 576. On the iperf -R (reverse mode) we get to 137MBps. With the iperf -s (non reverse test) we get to 72.5MBps.
I test from Site B to Site A using iperf3 -c server name, then iperf3 -c server name -R. The iperf server is in Site A.
Also tried some wild values in the 9000 range and got good upload from Site A. Was also decent near MTU 1050

@egc you mention a MTU Fix, what is that and how to adjust that?

For reference I am changing the MTU on the Network - Interfaces - Devices tab, then change the MTU for the device Wireguard. Then I do it on both routers, then restart the interface.

@brada4 I did implement your nft rule on both devices, don't know if it made any real difference.

Thanks!

6

I used a MTU of 576 at site A (with the iperf server) and a MTU of 9000 at Site B and that yields 140MBps from B to A, and about the same the other way. No idea how this makes any sense.

7

You do not need to set LAN side MTU, check via nft list ruleset that postrouting mtu fixup includes wireguard interface. Obviously you have to use default unspecified MTU.

8

You set the MTU on the Interfaces > Advanced section of the WG interface or in /etc/config/network > WG interface: option mtu '1420'

But I think what you do amounts to the same

The MTU fix is set on the Firewall zone of the WG interface enable MSS clamping (/etc/config/firewall > WG zone: `option mtu_fix '1').
But if you have the WG interface in the LAN zone you have to create a separate zone for the WG interface

9

That still does not work on stable, you need manual fixup rule.

10

Is that still not backported to 23.05.4?

If not will it be in the upcoming 23.05.5?

(I know I can check the commits but lazy today)

11 
                "version": "23.05-SNAPSHOT",
                "revision": "r24099-dd4b05ee8a",
firewall4 - 2023-09-01-598d9fbb-1
1 Like
12

So what is the correct way to create the Wireguard zone in the firewall and allow traffic to pass in the site to site VPN manner? I normally just put it in the LAN zone.

13

If you want to filter site to site traffic make a separate zone with separate /30 subnet for interconnect.

14

That is the easy (and correct) way but if you want to apply the MTU fix than make a separate zone with the WG interface and with the same settings (so all ACCEPT) and add MTU fix to this zone

1 Like
15

Ok think I did that right, but no notable change in performance. Still around 100-200MBps

Screenshot 2024-09-23 111031
Screenshot 2024-09-23 111031976×355 19.4 KB

16

As brada4 already pointed out it is possible that the MTU fix is not working correctly and you need his rule to accomplish that.

Question I forgot to ask (or maybe it is somewhere), are you using PPPoE or USB or 4G WAN, or IPv4 over IPv6 etc.?

17

the WAN is IPv4 DHCP provided by the carrier. Plain IPv4, not ipv4 over ipv6 or any of that stuff.

18

The detail - $wan_devices match default checkbox, you need to check fw4 print | less and use respective interface variable or interface, since wg is software interface it should be oifname in place of oif

1 Like
19 
chain input {
                type filter hook input priority filter; policy accept;
                iifname "lo" accept comment "!fw4: Accept traffic from loopback"
                ct state established,related accept comment "!fw4: Allow inbound established and related flows"
                tcp flags & (fin | syn | rst | ack) == syn jump syn_flood comment "!fw4: Rate limit TCP syn packets"
                iifname { "lan5", "br-lan.2", "br-lan.4", "br-lan" } jump input_lan comment "!fw4: Handle lan IPv4/IPv6 input traffic"
                iifname { "eth1", "lan1" } jump input_wan comment "!fw4: Handle wan IPv4/IPv6 input traffic"
                iifname "br-lan.3" jump input_guest comment "!fw4: Handle guest IPv4/IPv6 input traffic"
                iifname "Wireguard" jump input_Wireguard comment "!fw4: Handle Wireguard IPv4/IPv6 input traffic"
        }
chain forward {
                type filter hook forward priority filter; policy drop;
                meta l4proto { tcp, udp } flow offload @ft;
                ct state established,related accept comment "!fw4: Allow forwarded established and related flows"
                iifname { "lan5", "br-lan.2", "br-lan.4", "br-lan" } jump forward_lan comment "!fw4: Handle lan IPv4/IPv6 forward traffic"
                iifname { "eth1", "lan1" } jump forward_wan comment "!fw4: Handle wan IPv4/IPv6 forward traffic"
                iifname "br-lan.3" jump forward_guest comment "!fw4: Handle guest IPv4/IPv6 forward traffic"
                iifname "Wireguard" jump forward_Wireguard comment "!fw4: Handle Wireguard IPv4/IPv6 forward traffic"
                jump handle_reject
        }
chain output {
                type filter hook output priority filter; policy accept;
                oifname "lo" accept comment "!fw4: Accept traffic towards loopback"
                ct state established,related accept comment "!fw4: Allow outbound established and related flows"
                oifname { "lan5", "br-lan.2", "br-lan.4", "br-lan" } jump output_lan comment "!fw4: Handle lan IPv4/IPv6 output traffic"
                oifname { "eth1", "lan1" } jump output_wan comment "!fw4: Handle wan IPv4/IPv6 output traffic"
                oifname "br-lan.3" jump output_guest comment "!fw4: Handle guest IPv4/IPv6 output traffic"
                oifname "Wireguard" jump output_Wireguard comment "!fw4: Handle Wireguard IPv4/IPv6 output traffic"
        }
chain prerouting {
                type filter hook prerouting priority filter; policy accept;
                iifname { "lan5", "br-lan.2", "br-lan.4", "br-lan" } jump helper_lan comment "!fw4: Handle lan IPv4/IPv6 helper assignment"
                iifname "br-lan.3" jump helper_guest comment "!fw4: Handle guest IPv4/IPv6 helper assignment"
                iifname "Wireguard" jump helper_Wireguard comment "!fw4: Handle Wireguard IPv4/IPv6 helper assignment"
        }
        chain input_Wireguard {
                jump accept_from_Wireguard
        }
        chain output_Wireguard {
                jump accept_to_Wireguard
        }
        chain forward_Wireguard {
                jump accept_to_lan comment "!fw4: Accept Wireguard to lan forwarding"
                jump accept_to_Wireguard
        }
        chain helper_Wireguard {
        }
        chain accept_from_Wireguard {
                iifname "Wireguard" counter accept comment "!fw4: accept Wireguard IPv4/IPv6 traffic"
        }
        chain accept_to_Wireguard {
                oifname "Wireguard" counter accept comment "!fw4: accept Wireguard IPv4/IPv6 traffic"
}

Hopefully something useful in there. Tried to find any section related to Wireguard.

Thanks!

20

The postrouting lists are not dispatched to interface-based zone jumps