I am testing/troubleshooting/benchmarking Routed IPSec performance.
PC (i7-4790 with aes-ni) IP 2.2.2.1 <----> WAN (IP 2.2.2.2) <----> LAN (DHCP 192.168.88.188) Mac mini (i5)
IPSec tunnel between PC "acting as internet" and the router.
Routed on PC: 0.0.0.0/0 selectors: XFRM0 device (192.168.77.2) route 192.168.88.0/24 via XFRM0
Routed IPSec on the router: XFRM0 device (192.168.88.2). Routing 192.168.77.0/24 via xfrm0.
OOTB firewall settings plus additional rules to allow IPSec on WAN to device and allow PING. XFRM0 device on additional VPN zone with forwarding VPN->LAN and LAN->VPN
This works as expected: I can ping any device on the LAN side of the router from my PC and the other way around.
Running iperf3 tests it seems I am having performance issues:
No matter what I do I can't seem to get more "download speed" (meaning the router is receiving on WAN) than around 325mbps and I can't seem to get more "upload speed" (meaning the router is sending on WAN) than 210mbps. Doing iperf3 --bidir I can get around 200/100.
Interestingly I can get around 325mbps down in a single stream, but I need 2 streams to get 210mbps up.
For full duplex 200/100 I also need 2 streams.
I tried everything between null-md5, 3des-md5, aes128-sha1 to aes256-sha256 and only the 3des is slower (probably because the i7 can't keep up). I would expect aes128-sha1 to be faster vs aes256-sha256 if I am hitting a crypto limitation.
Is this a routing issue? Would policy based help performance? Any setting I could adjust? How to troubleshoot where the bottleneck is?
I will add some iperf3 results to more expand on my problem:
I am getting "nice" performance, I start the iperf3 again almost immediately and performance is "bad". Retry again a few times and performance is "good" again??
This is between my PC (external IP 2.2.2.1 xfrm0 ip 192.168.77.2) and mac mini connected to LAN (IP 192.168.88.188). IPSec tunnel is between PC and Router.
root@debian:~# iperf3 -c 192.168.88.188
Connecting to host 192.168.88.188, port 5201
[ 5] local 192.168.77.2 port 39452 connected to 192.168.88.188 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 45.4 MBytes 380 Mbits/sec 0 1.66 MBytes
[ 5] 1.00-2.00 sec 43.8 MBytes 367 Mbits/sec 48 1.35 MBytes
[ 5] 2.00-3.00 sec 43.8 MBytes 367 Mbits/sec 0 1.47 MBytes
[ 5] 3.00-4.00 sec 43.8 MBytes 367 Mbits/sec 0 1.56 MBytes
[ 5] 4.00-5.00 sec 45.0 MBytes 378 Mbits/sec 21 1.15 MBytes
[ 5] 5.00-6.00 sec 43.8 MBytes 367 Mbits/sec 0 1.23 MBytes
[ 5] 6.00-7.00 sec 45.0 MBytes 378 Mbits/sec 0 1.28 MBytes
[ 5] 7.00-8.00 sec 43.8 MBytes 367 Mbits/sec 0 1.32 MBytes
[ 5] 8.00-9.00 sec 43.8 MBytes 367 Mbits/sec 0 1.34 MBytes
[ 5] 9.00-10.00 sec 45.0 MBytes 377 Mbits/sec 0 1.36 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 443 MBytes 371 Mbits/sec 69 sender
[ 5] 0.00-10.03 sec 441 MBytes 369 Mbits/sec receiver
iperf Done.
root@debian:~# iperf3 -c 192.168.88.188
Connecting to host 192.168.88.188, port 5201
[ 5] local 192.168.77.2 port 39456 connected to 192.168.88.188 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 32.4 MBytes 272 Mbits/sec 0 1.48 MBytes
[ 5] 1.00-2.00 sec 31.2 MBytes 262 Mbits/sec 60 1.34 MBytes
[ 5] 2.00-3.00 sec 31.2 MBytes 262 Mbits/sec 0 1.46 MBytes
[ 5] 3.00-4.00 sec 31.2 MBytes 262 Mbits/sec 0 1.56 MBytes
[ 5] 4.00-5.00 sec 30.0 MBytes 252 Mbits/sec 0 1.63 MBytes
[ 5] 5.00-6.00 sec 31.2 MBytes 262 Mbits/sec 3 1.19 MBytes
[ 5] 6.00-7.00 sec 31.2 MBytes 262 Mbits/sec 0 1.27 MBytes
^C[ 5] 7.00-7.37 sec 11.2 MBytes 256 Mbits/sec 0 1.29 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-7.37 sec 230 MBytes 262 Mbits/sec 63 sender
[ 5] 0.00-7.37 sec 0.00 Bytes 0.00 bits/sec receiver
iperf3: interrupt - the client has terminated
root@debian:~# iperf3 -c 192.168.88.188
Connecting to host 192.168.88.188, port 5201
[ 5] local 192.168.77.2 port 39460 connected to 192.168.88.188 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 33.5 MBytes 281 Mbits/sec 0 1.46 MBytes
[ 5] 1.00-2.00 sec 31.2 MBytes 262 Mbits/sec 60 1.33 MBytes
^C[ 5] 2.00-2.79 sec 23.8 MBytes 252 Mbits/sec 0 1.43 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-2.79 sec 88.5 MBytes 266 Mbits/sec 60 sender
[ 5] 0.00-2.79 sec 0.00 Bytes 0.00 bits/sec receiver
iperf3: interrupt - the client has terminated
root@debian:~# iperf3 -c 192.168.88.188
Connecting to host 192.168.88.188, port 5201
[ 5] local 192.168.77.2 port 39464 connected to 192.168.88.188 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 45.4 MBytes 381 Mbits/sec 0 1.67 MBytes
[ 5] 1.00-2.00 sec 45.0 MBytes 377 Mbits/sec 68 1.36 MBytes
[ 5] 2.00-3.00 sec 43.8 MBytes 367 Mbits/sec 0 1.48 MBytes
[ 5] 3.00-4.00 sec 43.8 MBytes 367 Mbits/sec 0 1.57 MBytes
[ 5] 4.00-5.00 sec 45.0 MBytes 377 Mbits/sec 38 1.15 MBytes
[ 5] 5.00-6.00 sec 43.8 MBytes 367 Mbits/sec 0 1.23 MBytes
[ 5] 6.00-7.00 sec 43.8 MBytes 367 Mbits/sec 0 1.29 MBytes
[ 5] 7.00-8.00 sec 45.0 MBytes 377 Mbits/sec 0 1.34 MBytes
[ 5] 8.00-9.00 sec 45.0 MBytes 377 Mbits/sec 0 1.36 MBytes
[ 5] 9.00-10.00 sec 43.8 MBytes 367 Mbits/sec 0 1.38 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 444 MBytes 373 Mbits/sec 106 sender
[ 5] 0.00-10.03 sec 442 MBytes 370 Mbits/sec receiver
iperf Done.