Wireguard Site-to-site

fcecamor · February 5, 2021, 5:17am

Hello all,

I'm trying to set up a WG S2S connection between two OpenWRT devices.

Site A: 192.168.1.0/24
Site B: 192.168.10.0/24
VPN: 192.168.9.0/24

Site A:
/etc/config/network

[...]
config interface 'vpn'
        option proto 'wireguard'
        option private_key 'DELETED'
        option listen_port '51820'
        list addresses '192.168.9.1/24'
[...]
config wireguard_vpn 'spare02'
        option public_key ''
        option preshared_key ''
        list allowed_ips '192.168.10.0/24'
        list allowed_ips '192.168.9.17/32'
        option description 'spare02'
        option route_allowed_ips '1'
[...]

/etc/config/firewall

[...]
config zone 'lan'
        option name 'lan'
        list network 'lan'
        list network 'vpn'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'
        list device 'tun0'
[...]

ip r

default via 192.168.1.1 dev br-lan proto static
192.168.1.0/24 dev br-lan proto kernel scope link src 192.168.1.2
192.168.8.0/24 dev tun0 proto kernel scope link src 192.168.8.1
192.168.9.0/24 dev vpn proto kernel scope link src 192.168.9.1
192.168.9.17 dev vpn proto static scope link <--------------------------------- ??
192.168.10.0/24 dev vpn proto static scope link

Site B:
/etc/config/network

[...]
config interface 'vpn'
        option proto 'wireguard'
        option private_key 'DELETED'
        list addresses '192.168.9.17/24'
        list addresses 'fdf1:e8a1:8d3f:9::1'

config wireguard_vpn 'wgserver'
        option public_key 'DELETED'
        option preshared_key 'DELETED'
        option endpoint_host 'DELETED'
        option endpoint_port '51820'
        option route_allowed_ips '1'
        option persistent_keepalive '25'
        list allowed_ips '192.168.1.0/24'
[...]

/etc/config/firewall

[...]
config zone 'lan'
        option name 'lan'
        list network 'lan'
        list network 'vpn'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'
[...]

ip r

default via 192.168.4.1 dev eth0.2 proto static src 192.168.4.214
EXT_IP_DELETED via 192.168.4.1 dev eth0.2 proto static
192.168.1.0/24 dev vpn proto static scope link
192.168.4.0/24 dev eth0.2 proto kernel scope link src 192.168.4.214
192.168.9.0/24 dev vpn proto kernel scope link src 192.168.9.17
192.168.10.0/24 dev br-lan proto kernel scope link src 192.168.10.1

Now, there are some things that I don't understand, for example:

Why in the route table of the site A, there is a line with the VPN address of the site B?

192.168.9.17 dev vpn proto static scope link <--------------------------------- ??

The two OpenWRT devices cannot ping each other 192.168.9.0/24 class ip but the clients can ping the other site.

router on site A

PING 192.168.9.17 (192.168.9.17): 56 data bytes
^C
--- 192.168.9.17 ping statistics ---
2 packets transmitted, 0 packets received, 100% packet loss

router on site B

PING 192.168.9.1 (192.168.9.1): 56 data bytes
ping: sendto: No error information

Cheers

faser · February 5, 2021, 5:32am

Because you listed it in the config

And it would be needed if you just would allow communication between the two devices without the attached LAN's

On B you should also add list allowed_ips '192.168.9.1/32' so that you at least on a device to device communication you are fine.

The two devices would use the WG IP as source address (unless told them differently and as you were missing a route from B to A it would not work.

fcecamor · February 5, 2021, 5:56am

I'm not sure to understand. In the other peer configurations I've the same settings: each one has its own IP and it doesn't appear on the route table of the router on site A.

It doesn't work

Site A

PING 192.168.9.17 (192.168.9.17): 56 data bytes
ping: sendto: No error information

Site B

PING 192.168.9.1 (192.168.9.1): 56 data bytes
^C
--- 192.168.9.1 ping statistics ---
2 packets transmitted, 0 packets received, 100% packet loss

Site A

config wireguard_vpn 'spare02'
        option public_key 'DELETED'
        option preshared_key 'DELETED'
        list allowed_ips '192.168.10.0/24'
#       list allowed_ips '192.168.9.17/32'
        list allowed_ips 'fdf1:e8a1:8d3f:9::17/128'
        option description 'spare02'
        option route_allowed_ips '1'

Site B

config wireguard_vpn 'wgserver'
        option public_key 'DELETED'
        option preshared_key 'DELETED'
        option endpoint_host 'DELETED'
        option endpoint_port '51820'
        option route_allowed_ips '1'
        option persistent_keepalive '25'
        list allowed_ips '192.168.1.0/24'
        list allowed_ips '192.168.9.1/32'

faser · February 5, 2021, 5:58am

Why have you suddenly removed that? it is needed if you want to ping device to device unless you change the SRC IP.

Also need to allow firewall for WG to WG Zone.
Unless as mentioned you change SRC IP (e.g. ping -I 192.168.1.2 192.168.9.17)

fcecamor · February 5, 2021, 3:48pm

un-commenting that line it started working perfectly. Thanks

Now I would like to understand why it's so slow:

Site A is a 1000/300mbits fiber line on a WDR4900 (decently performing router).
Most of the time, the speed I get is around 355kB/s during file transferring on CIFS (windows share), sometimes is so slow that it interrupts the file transferring.

It's the same if I connect via the Site B (then connecting a client to it), or directly as a VPN client.
I also had the exactly same issue with openvpn.

If I try to upload something without VPN (like using PLEX with public open port) it works perfectly. It's just the VPN extremely slow.

CPU usage on the router is very low. Really I've no idea what the bottleneck is.

iplaywithtoys · February 6, 2021, 9:50pm

Crypto is slow. There's no getting around it. If you want fast crypto, for wire-speed VPNs, you need faster hardware and possibly dedicated crypto chips. Or, in other words, cheap crypto is slow, while fast crypto is expensive.

vgaetera · February 6, 2021, 9:59pm

In addition, some ISPs violate network neutrality by using DPI and traffic shaping for specific types of traffic which can include VPN protocols, or provide connectivity by utilizing methods which negatively impact UDP-based connections.

fcecamor · February 7, 2021, 6:09am

Hello,
I'm actually using two devices: a WDR4900 with a "powerful" Freescale P1014 800MHz as WG server and a WR1043NDv2 as WG client.

The first is able to stream over 100mbits, I really don't think that is the problem.

fcecamor · February 7, 2021, 6:11am

This morning 06:00L AM I was able to download up to 3.2mB/s my personal record, when yesterday night the maximum speed was about 355kB/s.

I don't understand why using other protocols work all the time, like I said before, PLEX with external port (not via VPN) works always perfectly.

I tried to change from UDP to TCP, I also tried to change the ports, it's exactly the same.

vgaetera · February 7, 2021, 11:21am

There are external factors you cannot control:

iplaywithtoys · February 7, 2021, 12:14pm

Shunting packets around takes minimal processing. In addition, some switch ASICs are optimised for switching.

Crypto takes maths, lots of maths. It's unlikely the WDR4900's guts are sufficient for wire-speed cryptography.

As a real-world example, I also run a site-to-site VPN over the Internet, using WireGuard, on a pair of Raspberry Pi 3B+ devices, one at each end. Those Raspberry Pi units have faster CPUs and more RAM than the WDR4900.

The obvious bottleneck is one site's upstream bandwidth; it tops out at 20Mbps. VPN or no, inter-site traffic cannot exceed 20Mbps. I've previously tested direct site-to-site transfer, without crypto, and topped out at just under 20Mbps.

However, the WireGuard VPN doesn't saturate the link. It tops out around 15-16Mbps, even on hardware demonstrably more powerful than the WDR4900. Some times it doesn't even reach that. On occasion it's dropped to 10Mbps. As @vgaetera notes, there are other factors outside your - and my - control.

Here's a suggestion of a good test you could carry out, to help see the effects of crypto in action.

Set up your own lab with a pair of identical routers attached to a switch to simulate the Internet. Put two identical computers behind those routers, to generate traffic to cross your "Internet". iperf and iperf3 are good for this sort of thing.

Observe transfer speeds without crypto. Then enable crypto (e.g. WireGuard, IPsec, OpenVPN, etc.) and carry out the same test. See how the speeds compare with and without crypto.

At home I'm fortunate enough to have a Palo Alto PA-220 firewall (my employer works with Palo Alto, so I was able to wheedle one out of the company). It's a very expensive bit of kit to buy for the typical home; it's priced for businesses, not private individuals. This firewall supports both IPsec and SSL VPNs. Palo Alto's data sheets for the firewall and its bigger siblings can be found here:

All firewalls: https://www.paloaltonetworks.co.uk/resources/datasheets/product-summary-specsheet
PA-220: https://www.paloaltonetworks.co.uk/resources/datasheets/pa-220-specsheet

The PA-220's ports are all gigabit-capable, but the VPNs cannot reach gigabit speeds. And this is on a device which is far more expensive and powerful than the WDR4900. It also has dedicated hardware for various features, rather than a single CPU doing everything. And it still can't saturate the wire if crypto is turned on.

In a nutshell, if you want wire-speed crypto, you'll need deep pockets. The WDR4900 might be a fine and fast router indeed, but it's not optimised for crypto.

krazeh · February 7, 2021, 1:38pm

While I don't disagree generally with your comments about crypto being punishing on CPUs, you shouldn't have an issue getting 20Mbps over Wireguard on a Pi 3B+. I have a rPi4 which barely registers a CPU load when uploading over Wireguard at 35(ish)Mbps (line speed). When downloading I hit 75% on one core and 50% on the 3 other cores (approximately) but it is receiving traffic at 500Mbps.

As for the OP, while the routers aren't super powerful, I would expect better than sub-1Mbps speeds. Never gonna hit line speed, but the hardware should be capable of more. I'd suspect the issue is more in line with vgaetera's viewpoint, i.e. external factors at play. I would be contacting my ISP to clarify if they have any traffic shaping in place.

system · February 16, 2021, 9:59pm

This topic was automatically closed 0 minutes after the last reply. New replies are no longer allowed.