MIPS optimized IPsec crypto modules (mcespi) on Archer C7

[defanator]

Hi everyone,

I've been using a couple of Archer C7 devices (v2 and v4) for a while, and recently I moved IPsec setup from HP miniserver behind a router to the router itself (it is Archer C7 v2 device running OpenWRT 18.06.1).

I noticed that the VPN bandwidth is quite low comparing to previous setup, charon process is running at 97-99% CPU. I did a quick googling and found this page:

https://openwrt.org/docs/guide-user/services/vpn/ipsec/strongswan/performance

The mcespi.c module mentioned there wasn't able to compile in my environment, but I managed to get it done - for the reference, here's the code:

defanator/mcespi

So I got the mcespi.ko, copied it to the device, did "insmod mcespi" and checked that it actually was loaded ("lsmod" and "cat /proc/crypto").

However, I don't see any changes - charon still consumes almost all the CPU, bandwidth is the same. I have the following ciphers configured in ipsec.conf:
ike=aes128-sha-modp1536,aes128-sha-modp1024,aes128-md5-modp1536,aes128-md5-modp1024,3des-sha-modp1536,3des-sha-modp1024,3des-md5-modp1536,3des-md5-modp1024
esp=aes128-sha1,aes128-md5,3des-sha1,3des-md5

Is something wrong with the setup? Should the mcespi module be "switched on" somehow? How can I ensure that the module is actually working?

Appreciate any input.

Thanks!

[jeff]

Assembler or C, a typical MIPS-based SoC is never going to be fast for VPN. Depending on the VPN bandwidth you need and security requirements, I’d put the end point back on the mini-server, in a VM, or on another device with appropriate hardware and crypto support.

I’d also evaluate your cipher list if you stay with IPSec

[mpa]

I second everything said by Jeff above. Setting up software crypto on this device may not be worth the trouble.

@defanator

How can I ensure that the module is actually working?

To ensure that the module is being used:

• For the esp parameter in ipsec.conf, list the algorithms supported by the mcespi driver, or a subset of them. End the list with an exclamation mark to make it exclusive. You can find the supported algorithms in /proc/crypto by looking at the driver field.
• Dump /proc/crypto before and after starting the connection. Compare the files, look for changes in the refcnt field.

[mk24]

Charon should only demand CPU during the key exchange. After that, encryption of data packets is done in the kernel.

[defanator]

That's interesting. I'm not sure tools like perf are available directly on the device, but I'm sure the charon process is the only one hitting CPU during the transfer.

[defanator]

I totally understand the reasons of a drop, I just want to ensure I did my best to get higher possible bandwidth with IPsec on the device.

So far my only guess is that I'm using wrong cipher names in ipsec.conf, investigating this.

[defanator]

Here's the ipsec.conf:

config setup
  uniqueids=no

conn %default
  auto=add
  keyexchange=ikev2
  fragmentation=yes
  forceencaps=yes
  reauth=no
  rekey=no
  mobike=yes
  dpdaction=restart
  dpddelay=300s
  ike=aes128-sha1-md5-modp1024!
  esp=aes128-sha1!
  left=%any
  leftcert=vpn-host-certificate.pem
  leftsendcert=always
  leftsubnet=10.7.7.0/24
  right=%any
  rightid=%any
  rightsourceip=10.89.1.0/24
  eap_identity=%identity

conn IPSec-IKEv2-EAP
  leftid=@my.domain
  rightauth=eap-mschapv2

Here's the contents of /proc/crypto with mcespi.ko loaded:

name         : sha224
driver       : sha224-generic
module       : sha256_generic
priority     : 0
refcnt       : 1
selftest     : passed
internal     : no
type         : shash
blocksize    : 64
digestsize   : 28

name         : sha256
driver       : sha256-generic
module       : sha256_generic
priority     : 0
refcnt       : 1
selftest     : passed
internal     : no
type         : shash
blocksize    : 64
digestsize   : 32

name         : sha1
driver       : sha1-generic
module       : sha1_generic
priority     : 0
refcnt       : 1
selftest     : passed
internal     : no
type         : shash
blocksize    : 64
digestsize   : 20

name         : jitterentropy_rng
driver       : jitterentropy_rng
module       : jitterentropy_rng
priority     : 100
refcnt       : 1
selftest     : passed
internal     : no
type         : rng
seedsize     : 0

name         : stdrng
driver       : drbg_nopr_hmac_sha256
module       : drbg
priority     : 207
refcnt       : 1
selftest     : passed
internal     : no
type         : rng
seedsize     : 0

name         : stdrng
driver       : drbg_nopr_hmac_sha512
module       : drbg
priority     : 206
refcnt       : 1
selftest     : passed
internal     : no
type         : rng
seedsize     : 0

name         : stdrng
driver       : drbg_nopr_hmac_sha384
module       : drbg
priority     : 205
refcnt       : 1
selftest     : passed
internal     : no
type         : rng
seedsize     : 0

name         : stdrng
driver       : drbg_nopr_hmac_sha1
module       : drbg
priority     : 204
refcnt       : 1
selftest     : passed
internal     : no
type         : rng
seedsize     : 0

name         : stdrng
driver       : drbg_pr_hmac_sha256
module       : drbg
priority     : 203
refcnt       : 1
selftest     : passed
internal     : no
type         : rng
seedsize     : 0

name         : stdrng
driver       : drbg_pr_hmac_sha512
module       : drbg
priority     : 202
refcnt       : 1
selftest     : passed
internal     : no
type         : rng
seedsize     : 0

name         : stdrng
driver       : drbg_pr_hmac_sha384
module       : drbg
priority     : 201
refcnt       : 1
selftest     : passed
internal     : no
type         : rng
seedsize     : 0

name         : stdrng
driver       : drbg_pr_hmac_sha1
module       : drbg
priority     : 200
refcnt       : 1
selftest     : passed
internal     : no
type         : rng
seedsize     : 0

name         : md5
driver       : md5-generic
module       : md5
priority     : 0
refcnt       : 1
selftest     : passed
internal     : no
type         : shash
blocksize    : 64
digestsize   : 16

name         : cbc(aes)
driver       : cbc-mcespi
module       : mcespi
priority     : 300
refcnt       : 1
selftest     : passed
internal     : no
type         : blkcipher
blocksize    : 16
min keysize  : 16
max keysize  : 32
ivsize       : 16
geniv        : <default>

name         : md5
driver       : md5-mcespi
module       : mcespi
priority     : 200
refcnt       : 1
selftest     : passed
internal     : no
type         : shash
blocksize    : 64
digestsize   : 16

name         : sha1
driver       : sha1-mcespi
module       : mcespi
priority     : 200
refcnt       : 1
selftest     : passed
internal     : no
type         : shash
blocksize    : 64
digestsize   : 20

name         : aes
driver       : aes-mcespi
module       : mcespi
priority     : 200
refcnt       : 4
selftest     : passed
internal     : no
type         : cipher
blocksize    : 16
min keysize  : 16
max keysize  : 32

name         : des3_ede
driver       : des3_ede-generic
module       : des_generic
priority     : 100
refcnt       : 1
selftest     : passed
internal     : no
type         : cipher
blocksize    : 8
min keysize  : 24
max keysize  : 24

name         : des
driver       : des-generic
module       : des_generic
priority     : 100
refcnt       : 1
selftest     : passed
internal     : no
type         : cipher
blocksize    : 8
min keysize  : 8
max keysize  : 8

name         : deflate
driver       : deflate-generic
module       : deflate
priority     : 0
refcnt       : 1
selftest     : passed
internal     : no
type         : compression

name         : digest_null
driver       : digest_null-generic
module       : crypto_null
priority     : 0
refcnt       : 1
selftest     : passed
internal     : no
type         : shash
blocksize    : 1
digestsize   : 0

name         : compress_null
driver       : compress_null-generic
module       : crypto_null
priority     : 0
refcnt       : 1
selftest     : passed
internal     : no
type         : compression

name         : ecb(cipher_null)
driver       : ecb-cipher_null
module       : crypto_null
priority     : 100
refcnt       : 1
selftest     : passed
internal     : no
type         : blkcipher
blocksize    : 1
min keysize  : 0
max keysize  : 0
ivsize       : 0
geniv        : <default>

name         : cipher_null
driver       : cipher_null-generic
module       : crypto_null
priority     : 0
refcnt       : 1
selftest     : passed
internal     : no
type         : cipher
blocksize    : 1
min keysize  : 0
max keysize  : 0

name         : crc32c
driver       : crc32c-generic
module       : crc32c_generic
priority     : 100
refcnt       : 3
selftest     : passed
internal     : no
type         : shash
blocksize    : 1
digestsize   : 4

name         : ecb(arc4)
driver       : ecb(arc4)-generic
module       : kernel
priority     : 100
refcnt       : 1
selftest     : passed
internal     : no
type         : blkcipher
blocksize    : 1
min keysize  : 1
max keysize  : 256
ivsize       : 0
geniv        : <default>

name         : arc4
driver       : arc4-generic
module       : kernel
priority     : 0
refcnt       : 5
selftest     : passed
internal     : no
type         : cipher
blocksize    : 1
min keysize  : 1
max keysize  : 256

name         : aes
driver       : aes-generic
module       : kernel
priority     : 100
refcnt       : 1
selftest     : passed
internal     : no
type         : cipher
blocksize    : 16
min keysize  : 16
max keysize  : 32

I tried to compare refcounts a) between restarting ipsec, b) between performing actual connections - nothing changes at all.

[mpa]

@mk24

Charon should only demand CPU during the key exchange. After that, encryption of data packets is done in the kernel.

Good point.

@defanator

That's interesting. I'm not sure tools like perf are available directly on the device, but I'm sure the charon process is the only one hitting CPU during the transfer .

Restart charon and check the log for loaded plugins. Make sure it is using the kernel-netlink plugin. Do not use the kernel-libipsec plugin, which handles IPsec in userspace.

However, CPU load caused by the kernel implementation of IPsec does not appear in the process list, only in the summary. Look at idle, for example, which decreases under load.

[defanator]

Removing strongswan-mod-kernel-libipsec and restarting ipsec did the trick! I'm now seeing bandwidth improvements even without mcespi loaded, but for some reasons I can't reach hosts behind the router directly if ipsec is operating in kernel. However, port forwarding works fine.

Ah, just checked the rules and it seems like there's no ipsec0 interface anymore, hence the firewall rules also do not work.

Could someone point me to the right direction on how to resolve this?

TIA!

[defanator]

Ok, I finally figured this out.

Had to change this:

config zone                  
        option forward 'REJECT'
        option name 'vpn'      
        option output 'ACCEPT'
        option network 'ipsec'
        option input 'ACCEPT'

to this:

config zone                  
        option forward 'REJECT'
        option name 'vpn'      
        option output 'ACCEPT'
        option network 'wan'
        option input 'ACCEPT'
        option subnet '10.89.1.0/24'

where 10.89.1.0/24 is my rightsourceip subnet from /etc/ipsec/ipsec.conf.

[mpa]

@defanator

config zone                  
        option forward 'REJECT'
        option name 'vpn'      
        option output 'ACCEPT'
        option network 'wan'
        option input 'ACCEPT'
        option subnet '10.89.1.0/24'

I suggest this firewall configuration:

IPsec site-to-site tunnel

The wan interface is set up with masquerading (source NAT) by default. This affects both plaintext traffic to the internet and VPN traffic towards the remote LAN. When IPsec decides whether to pass through or to tunnel a packet, it compares the packet's source IP, destination IP, and optionally other header fields with its own configured policy. Masquerading changes the source IP, which causes a mismatch in this comparison, and the packet is not sent through the VPN. You have to make sure the …

[defanator]

@mpa

I suggest this firewall configuration:

Adding the following:

	option extra_src '-m policy --dir in --pol none'
	option extra_dest '-m policy --dir out --pol none'

to wan and lan zones configuration resulted in the following error during firewall reload and total router unavailability:

Oct 21 22:37:15 archer-c7-v2 kernel: [ 7976.111557] xt_policy: input policy not valid in POSTROUTING and OUTPUT
Oct 21 22:37:15 archer-c7-v2 kernel: [ 7976.137766] xt_policy: input policy not valid in POSTROUTING and OUTPUT
Oct 21 22:37:15 archer-c7-v2 kernel: [ 7976.179248] xt_policy: input policy not valid in POSTROUTING and OUTPUT

Anyway, thanks a lot for you help! I guess firewall issues aren't relevant in this particular topic. I'll try to handle it further.

Cheers!

[mpa]

@defanator

Oct 21 22:37:15 archer-c7-v2 kernel: [ 7976.111557] xt_policy: input policy not valid in POSTROUTING and OUTPUT

I can reproduce this error only with the following setting, which has the directions mixed up:

option extra_dest '-m policy --dir in --pol none'

[lucize]

@defanator

there's no ipsec0 interface anymore

use VTI interfaces or XFRMi interfaces to send marked packets in strongswan to them. there are template examples in openwrt

[defanator]

Just for the reference, some numbers from iperf3 output, client is running on a laptop connected by wifi in one location, server is either HP miniserver connected by wire to the Archer C7 (v2), or Archer C7 itself (latest measurement) in another location, ISP is the same in both locations, bandwidth is limited to 50 Mbits/s by ISP, units is Mbits/sec, results go through ministat(1):

ipsec (no mcespi):

    N           Min           Max        Median           Avg        Stddev
x  60          29.6          37.5          35.2     34.933333     1.5985869

ipsec (mcespi):

    N           Min           Max        Median           Avg        Stddev
x  60          38.8          45.4         44.15     43.868333      1.079154

directly to the router:

    N           Min           Max        Median           Avg        Stddev
x  60          36.8          48.1          46.5     46.083333     2.0081332

I'll make another test with a router, server and client in the same room and wired connections, but overall I'm quite fine with the numbers for my particular purposes.