BT Home Hub 5A: configuring protonVPN via openVPN

Installing and Using OpenWrt Network and Wireless Configuration
1

Hi,
I just bougth a BT Homehub 5A router and I found it a great router together with latest openWRT.
I would like to configure it as VPN client in order to protect all my LAN. I'm using protonVPN as service both free and paid versions. I followed this guide in order to setup it into my openWRT.
If I use standard openVPN client from my GNU/linux OS, I can connect to protonVPN and it works well.
I used different protonVPN accounts with different servers and via UDP.
Then, I tried to use the same accounts and the same servers via openVPN on my openWRT router and in this case, I can connect to protonVPN, but I cannot navigate. When the router finishes the boot I can navigate on Internet for less than one minute.
I think that sources of the problem could be two:

  1. Insufficient processing power
  2. Wrong configuration

For the first point. The router has SoC: Lantiq XWAY VRX268 (PSB 80910 EL V1.2) MIPS 34Kc and 128 MB RAM with openWRT 17.01.4 and kernel 4.4.92.
I also ran benchmark in order to test encryption performance following this guide:

| r3560 | xRX200 rev 1.2 | BTHOMEHUBV5A - BT Home Hub 5A | MIPS 34Kc V5.6 | 332.54 | 1.0.2o | 25504330 | 22153860 | 12252030 | 3649200 | 3882310 | 1475740 | 6657450 | 5786680 | 5134420 | 5.5 | 206.2 15.8 | 16.7 |

This is my laptop (i7- i7-3537U) in which openVPN works well:

| | 1.0.2o | 424338410 | 344252780 | 199024370 | 228348100 | 57664760 | 18779730 | 96382760 | 91406340 | 57513530 | 455.4 | 18359.5 1639.3 | 1371.7 |

Both are using the same version of openSSL 1.0.2o. i7 is 10-15 times faster than SoC.
Do you think that the bottleneck is into CPU?
In other thread, an user suggested that the SoC has a crypto CPU embedded compatible with openWRT via package kmod-crypto-user even if it requires openSSL 1.1.0 instead of 1.0.2 currently used by openWRT.
Do you think it could be a CPU bottleneck problem?
Regarding the second point. The configuration seems correct, however the behaviour it is strange since the connection is dropped quite instantly.
Do you have any advice?
Thank you

P.S.
output of cat /proc/crypto here
Real time load after boot no openVPN

Load%20no%20openVPN
Load%20no%20openVPN1185Ă—792 42.7 KB

Real time load after boot with openVPN
Load%20with%20openVPN
Load%20with%20openVPN1171Ă—792 42.3 KB

2

Yes, the CPU is much slower than x86, and it can limit your VPN throughput. However, this should not cause all VPN traffic to stop. You can deal with these issues separately.

Regarding the integrated crypto hardware, please ignore the kmod-crypto-user part of my previous posting, and try cryptodev instead as suggested by @drbrains, since this should work with the OpenSSL version available in OpenWrt and even have better performance.

Now about your failing VPN connection, when you say "navigate on Internet", do you mean fetching web pages with your browser? I guess we should diagnose this on a lower level.

Please run ping -n IP_ADDRESS in parallel to your browsing to see if the VPN problem affects all IP traffic or just your browser. IP_ADDRESS should be located outside your network and reachable through the VPN, and obviously it should respond to ping (ICMP echo requests).

Also in parallel, log into your router with ssh, run logread -f and watch for anything related to your VPN interface.

How is your firewall set up? Is the VPN interface included in the wan zone, a zone of its own, or not mentioned at all?

3

When you get back to evaluating CPU load, I'd look at CPU utilization rather than the "load average". top or htop can provide better insight than the metric that uptime or equivalent outputs.

4

fwiw, if you do successfully achieve a stable openvpn connection, you will discover the maximum throughput is about 9 mbps on HH5A running OpenVPN client.

Worth a read:
https://www.dropbox.com/sh/c8cqmpc6cacs5n8/AAA2f8htk1uMitBckDW8Jq88a?dl=0

5

Now about your failing VPN connection, when you say "navigate on Internet", do you mean fetching web pages with your browser? I guess we should diagnose this on a lower level.

You are right, I directly translated form my mother language. I should have said browse.
However, output of cat logread -f here
I just tried with ping -n 1.1.1.1 or ping -n openwrt.org
You can see my configurations attached. They are the same of the links first or second.

When you get back to evaluating CPU load, I'd look at CPU utilization rather than the "load average". top or htop can provide better insight than the metric that uptime or equivalent outputs.

Realtime Load on openWRT luci interface is the same of top or htop.

fwiw, if you do successfully achieve a stable openvpn connection, you will discover the maximum throughput is > > about 9 mbps.

Worth a read:
https://openwrt.ebilan.co.uk/viewtopic.php?f=7&t=279

Thank you for your link. I also tried to follow the procedure described, that is similar to the other one, but I'm not able to have a working connection. Moreover, the CPU described into the procedure has 400 MHz while the one of BT Home Hub 5A is Lantiq PSB 80910 and it has 500 MHz.
I would like to see the difference.
First procedure

Firewall%20setting%20first%20configuration
Firewall%20setting%20first%20configuration1183Ă—830 58.7 KB

Second procedure
Firewall%20setting%20second%20configuration%20
Firewall%20setting%20second%20configuration%201296Ă—810 63.2 KB

Both procedures
Inter%20zone%20forwarding
Inter%20zone%20forwarding965Ă—838 51.9 KB

6

If you are still struggling to get a working connection, perhaps you can consider factory reset the hub, and go through setting up the HH5A as described in the PDF exactly as described in the instructions:
https://openwrt.ebilan.co.uk/viewtopic.php?f=7&t=279
for use with free 'vpnbook' VPN provider that was used in the example.

I know it also works with mullvad too who offer free 3 hour connections on demand. Once you are successful setting up a free provider, then adapt it for ProtonVPN.

9 mbps is about the maximum throughput you will get through HH5A running OpenVPN client on LEDE. The quote about '400MHz' was from a VPN provider, mullvad, commenting on the throughput of budget routers in general.

I notice Proton offer free 7 day trials. If I can find some spare time, I'll try setting up Proton on my existing HH5A openvpn client (setup as described in PDF) which I have been using for past two months with another VPN provider without any issues.

7

If you are still struggling to get a working connection, perhaps you can consider factory reset the hub, and go through setting up the HH5A as described in the PDF exactly as described in the instructions:
https://openwrt.ebilan.co.uk/viewtopic.php?f=7&t=279 2
for use with free 'vpnbook' VPN provider that was used in the example.

Ok, I will try. Maybe could be some package that does not allow to openVPN to work properly.

I notice Proton offer free 7 day trials. If I can find some spare time, I'll try setting up Proton on my existing HH5A openvpn client (setup as described in PDF) which I have been using for past two months with another VPN provider without any issues.

ProtonVPN provides life time free plan for one device and slow speed from few different servers:

P.S.

9 mbps is about the maximum throughput you will get through HH5A running OpenVPN client on LEDE. The quote about '400MHz' was from a VPN provider, mullvad, commenting on the throughput of budget routers in general.

These are the benchmark with openSSL.
The 'numbers' are in 1000s of bytes per second processed.

type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-128 cbc       5759.11k     6218.64k     6507.54k     6778.70k     6609.45k
aes-192 cbc       5136.88k     5631.40k     5862.05k     5833.63k     5811.20k
aes-256 cbc       4526.87k     4990.54k     5067.33k     5090.04k     5116.41k
sha256            2387.42k     5494.11k     9726.54k    11977.48k    12824.72k
sha512             497.24k     2061.73k     2744.67k     3692.60k     4070.50k

Since protonVPN uses aes-256 cbc for data and sha512 for authentication, the router should achieve at least about 4526.87k ~ 4.5 MByte/s ~ 36 Mbit/s. So 9 Mbit/s are limited by something else.

8

I just tried protonvpn using these two ovpn files extracted from the zip file containing free server config files for Japan, USA and Netherlands.

nl-108.protonvpn.com.udp1194.ovpn
us-ca-101.protonvpn.com.udp1194.ovpn

The only amendments I made to above ovpn files is:

auth-user-pass /etc/openvpn/userpass.txt

I then updated the userpass.txt with the protonvpn openvpn/ikev2 username & password obtained from the Accounts page.

I'm unable to authenticate according to the logs when connecting to a free US server:

Sat Jun 23 14:49:10 2018 daemon.notice openvpn(VPNclient)[3241]: OpenVPN 2.4.4 mips-openwrt-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD]
Sat Jun 23 14:49:10 2018 daemon.notice openvpn(VPNclient)[3241]: library versions: OpenSSL 1.0.2o  27 Mar 2018, LZO 2.10
Sat Jun 23 14:49:10 2018 daemon.warn openvpn(VPNclient)[3241]: WARNING: --ping should normally be used with --ping-restart or --ping-exit
Sat Jun 23 14:49:10 2018 daemon.warn openvpn(VPNclient)[3241]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Sat Jun 23 14:49:10 2018 daemon.notice openvpn(VPNclient)[3241]: Outgoing Control Channel Authentication: Using 512 bit message hash 'SHA512' for HMAC authentication
Sat Jun 23 14:49:10 2018 daemon.notice openvpn(VPNclient)[3241]: Incoming Control Channel Authentication: Using 512 bit message hash 'SHA512' for HMAC authentication
Sat Jun 23 14:49:10 2018 daemon.notice openvpn(VPNclient)[3241]: TCP/UDP: Preserving recently used remote address: [AF_INET]209.58.142.154:1194
Sat Jun 23 14:49:10 2018 daemon.notice openvpn(VPNclient)[3241]: Socket Buffers: R=[163840->163840] S=[163840->163840]
Sat Jun 23 14:49:10 2018 daemon.notice openvpn(VPNclient)[3241]: UDP link local: (not bound)
Sat Jun 23 14:49:10 2018 daemon.notice openvpn(VPNclient)[3241]: UDP link remote: [AF_INET]209.58.142.154:1194
Sat Jun 23 14:49:10 2018 daemon.notice openvpn(VPNclient)[3241]: TLS: Initial packet from [AF_INET]209.58.142.154:1194, sid=61aaf23d 7a218e40
Sat Jun 23 14:49:10 2018 daemon.warn openvpn(VPNclient)[3241]: WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
Sat Jun 23 14:49:10 2018 daemon.notice openvpn(VPNclient)[3241]: VERIFY OK: depth=2, C=CH, O=ProtonVPN AG, CN=ProtonVPN Root CA
Sat Jun 23 14:49:10 2018 daemon.notice openvpn(VPNclient)[3241]: VERIFY OK: depth=1, C=CH, O=ProtonVPN AG, CN=ProtonVPN Intermediate CA 1
Sat Jun 23 14:49:10 2018 daemon.notice openvpn(VPNclient)[3241]: VERIFY KU OK
Sat Jun 23 14:49:10 2018 daemon.notice openvpn(VPNclient)[3241]: Validating certificate extended key usage
Sat Jun 23 14:49:10 2018 daemon.notice openvpn(VPNclient)[3241]: ++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication
Sat Jun 23 14:49:10 2018 daemon.notice openvpn(VPNclient)[3241]: VERIFY EKU OK
Sat Jun 23 14:49:10 2018 daemon.notice openvpn(VPNclient)[3241]: VERIFY OK: depth=0, CN=us-ca-101.protonvpn.com
Sat Jun 23 14:49:13 2018 daemon.notice openvpn(VPNclient)[3241]: Control Channel: TLSv1.2, cipher TLSv1/SSLv3 ECDHE-RSA-AES256-GCM-SHA384, 2048 bit RSA
Sat Jun 23 14:49:13 2018 daemon.notice openvpn(VPNclient)[3241]: [us-ca-101.protonvpn.com] Peer Connection Initiated with [AF_INET]209.58.142.154:1194
Sat Jun 23 14:49:14 2018 daemon.notice openvpn(VPNclient)[3241]: SENT CONTROL [us-ca-101.protonvpn.com]: 'PUSH_REQUEST' (status=1)
Sat Jun 23 14:49:14 2018 daemon.notice openvpn(VPNclient)[3241]: AUTH: Received control message: AUTH_FAILED
Sat Jun 23 14:49:14 2018 daemon.notice openvpn(VPNclient)[3241]: SIGTERM[soft,auth-failure] received, process exiting

I tried changing/creating an simpler openvpn/ikeV2 username & password, but above problem persists. I've given up for the moment.

I've dropped a message to ProtonVPN support. Wait and see what they say.

btw, you have no chance achieving 36 Mbit/s with a HH5A imho.
If you want those sorts of OpenVPN speeds, you would need to use something like Asus RT-AC68u (AsusWRT) or Netgear R7000.

Update: no response from ProtonVPN support on this issue.

9

Hi,
thank you very much for your help.

nl-108.protonvpn.com.udp1194.ovpn
us-ca-101.protonvpn.com.udp1194.ovpn

Be aware that these servers are not free and you could not use them after one week.

I finally found the source of problem and it is not related to the openWRT router, but to the configuration of openVPN and protonVPN.
I followed the protonVPN official guide and I could connect by using network-manager-openvpn-gnome log file. Whereas I could not connect via command line see manual authentication log file and file authentication log log.
I'm using ubuntu mate 16.04 LTS and so now it seems a problem that is related to protonVPN, I also contacted the official support.
Moreover, I replicated the same configuration on the router and now I get the same errors as you:


Sun Jun 24 07:12:07 2018 daemon.notice openvpn(protonvpn_autoconf)[19083]: OpenVPN 2.4.4 mips-openwrt-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD]
Sun Jun 24 07:12:07 2018 daemon.notice openvpn(protonvpn_autoconf)[19083]: library versions: OpenSSL 1.0.2o  27 Mar 2018, LZO 2.10
Sun Jun 24 07:12:07 2018 daemon.warn openvpn(protonvpn_autoconf)[19083]: WARNING: --ping should normally be used with --ping-restart or --ping-exit
Sun Jun 24 07:12:07 2018 daemon.warn openvpn(protonvpn_autoconf)[19083]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Sun Jun 24 07:12:07 2018 daemon.notice openvpn(protonvpn_autoconf)[19083]: Outgoing Control Channel Authentication: Using 512 bit message hash 'SHA512' for HMAC authentication
Sun Jun 24 07:12:07 2018 daemon.notice openvpn(protonvpn_autoconf)[19083]: Incoming Control Channel Authentication: Using 512 bit message hash 'SHA512' for HMAC authentication
Sun Jun 24 07:12:07 2018 daemon.notice openvpn(protonvpn_autoconf)[19083]: TCP/UDP: Preserving recently used remote address: [AF_INET]89.39.107.204:1194
Sun Jun 24 07:12:07 2018 daemon.notice openvpn(protonvpn_autoconf)[19083]: Socket Buffers: R=[163840->163840] S=[163840->163840]
Sun Jun 24 07:12:07 2018 daemon.notice openvpn(protonvpn_autoconf)[19083]: UDP link local: (not bound)
Sun Jun 24 07:12:07 2018 daemon.notice openvpn(protonvpn_autoconf)[19083]: UDP link remote: [AF_INET]89.39.107.204:1194
Sun Jun 24 07:12:07 2018 daemon.notice openvpn(protonvpn_autoconf)[19083]: TLS: Initial packet from [AF_INET]89.39.107.204:1194, sid=0a444e26 77724359
Sun Jun 24 07:12:07 2018 daemon.warn openvpn(protonvpn_autoconf)[19083]: WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
Sun Jun 24 07:12:07 2018 daemon.notice openvpn(protonvpn_autoconf)[19083]: VERIFY OK: depth=2, C=CH, O=ProtonVPN AG, CN=ProtonVPN Root CA
Sun Jun 24 07:12:07 2018 daemon.notice openvpn(protonvpn_autoconf)[19083]: VERIFY OK: depth=1, C=CH, O=ProtonVPN AG, CN=ProtonVPN Intermediate CA 1
Sun Jun 24 07:12:07 2018 daemon.notice openvpn(protonvpn_autoconf)[19083]: VERIFY KU OK
Sun Jun 24 07:12:07 2018 daemon.notice openvpn(protonvpn_autoconf)[19083]: Validating certificate extended key usage
Sun Jun 24 07:12:07 2018 daemon.notice openvpn(protonvpn_autoconf)[19083]: ++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication
Sun Jun 24 07:12:07 2018 daemon.notice openvpn(protonvpn_autoconf)[19083]: VERIFY EKU OK
Sun Jun 24 07:12:07 2018 daemon.notice openvpn(protonvpn_autoconf)[19083]: VERIFY OK: depth=0, CN=nl-107.protonvpn.com
Sun Jun 24 07:12:10 2018 daemon.notice openvpn(protonvpn_autoconf)[19083]: Control Channel: TLSv1.2, cipher TLSv1/SSLv3 ECDHE-RSA-AES256-GCM-SHA384, 2048 bit RSA
Sun Jun 24 07:12:10 2018 daemon.notice openvpn(protonvpn_autoconf)[19083]: [nl-107.protonvpn.com] Peer Connection Initiated with [AF_INET]89.39.107.204:1194
Sun Jun 24 07:12:11 2018 daemon.notice openvpn(protonvpn_autoconf)[19083]: SENT CONTROL [nl-107.protonvpn.com]: 'PUSH_REQUEST' (status=1)
Sun Jun 24 07:12:12 2018 daemon.notice openvpn(protonvpn_autoconf)[19083]: AUTH: Received control message: AUTH_FAILED
Sun Jun 24 07:12:12 2018 daemon.notice openvpn(protonvpn_autoconf)[19083]: SIGTERM[soft,auth-failure] received, process exiting

P.S.
I added the file update-resolv-conf to /etc/openvpn/ on openWRT.

10

Hi,
I made some progresses. The supports of protonVPN told me that the servers do not accept too frequent reconnection requests (at least 2 minutes). This could be the reason for which we get many AUTH_FAILED.
Now, the script provided by protonVPN works on ubuntu and on openWRT is executed up to:

up /etc/openvpn/update-resolv-conf
down /etc/openvpn/update-resolv-conf

Then the update-resolv-conf script requires to run /sbin/resolvconf program. However, the openWRT does not have any program in that folder and I could not find it in any package.
I tried to comment these lines and then the router established a stable connection, but the DNS does not work as expected and I could not browser the Web.
Do you have any idea? We are close to a solution and then we can have a working guide.

P.S. the first procedure does not use file.openvpn and so does not requires resolvconf. What about the https://openwrt.ebilan.co.uk/viewtopic.php?f=7&t=279, the one that works? How does it manage resolvconf? It is very strange...

11

Could it be the remote DNS servers are hard coded and provided by DHCP server as mentioned in section 2.3 of the HH5A guide?

eg. this is pushed out to connected clients:
DHCP-Options 6,8.8.8.8,8.8.4.4

12

Hi,
I finally solved all the problems. I summarise the procedure in order to allow to other user to avoid a waste of time.

  1. You can follow the procedure posted by @bill888 here. Remember to setup the VPN for the all networks and not only for a specific one.
  2. You have to manually edit the configuration file of protonVPN server-name.ovpn and comment the following lines since openWRT does not have resolvconf program
up /etc/openvpn/update-resolv-conf
down /etc/openvpn/update-resolv-conf
  1. You have to modify DNS-DHCP server by adding your DNS entries (suggested DNS 1.1.1.1, 208.67.222.222, 208.67.220.220)
    LUCI->network->DNS and DHCP->DNS forwardings
  2. You have to disable resolv conf file
    LUCI->network->DNS and DHCP->resolv and hosts files->ignore resolve file

Then I made several benchmark with speedtest and nperf: by not using protonVPN, by using it with my PC and finally by using it with openWRT.
I noticed that after the boot and some operations performed by openWRT that required all the CPU, the router was in idle. OpenVPN reached about 90% of CPU usage during the benchmark.
The values seems to confirm that the maximum speed should be about 9 Mbit/s as suggested by @bill888. However, adding the option option engine 'dynamic' to /etc/config/openvpn does not produce any effect @mpa @drbrains .
Do you have further suggestion in order to improve connection speed?
Thank you

not using protonVPN speedtest
no%20protonVPN%20speedtest!
not using protonVPN nperf
no%20protonVPN%20nperf!Ăą
using protonVPN PC speedtest
protonVPN%20PC%20speedtest
using protonVPN PC nperf
protonVPN%20PC%20nperf
using protonVPN openWRT speedtest
protonVPN%20openWRT%20speedtest
using protonVPN openWRT nperf
protonVPN%20openWRT%20nperf
CPU load after boot
htop%20after%20boot
CPU load during benchmark
htop%20load%20protonVPN

13

Your /proc/crypto shows you have hardware crypto available. But...in order for you to use that you need to have “cryptodev” loaded AND OpenSSL needs to be compiled with the cryptodev feature.

lsmod should show cryptodev loaded (or a message in dmesg).

You can test if OpenSSL has the cryptodev enabled with OpenVPN itself.

ssh into your router
cd /tmp
openvpn —genkey —secret key
openvpn —test-crypto —secret key —cipher aes-256-cbc —engine cryptodev

The output will do a selftest and give you all the information about OpenSSL.

14

It seems that cryptodev is not loaded.

lsmod | grep crypto
crypto_null             2544  1 aead
cryptomgr               2080  0

dmesg | grep crypto
[   13.040312] ath10k_pci 0000:02:00.0: htt-ver 2.1 wmi-op 5 htt-op 2 cal file max-sta 128 raw 0 hwcrypto 1

openvpn test

openvpn --test-crypto --secret key --cipher aes-256-cdb --engine cryptodev
Tue Jul  3 10:45:35 2018 disabling NCP mode (--ncp-disable) because not in P2MP client or server mode
Tue Jul  3 10:45:35 2018 OpenVPN 2.4.4 mips-openwrt-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD]
Tue Jul  3 10:45:35 2018 library versions: OpenSSL 1.0.2o  27 Mar 2018, LZO 2.10
Tue Jul  3 10:45:35 2018 OpenVPN 2.4.4 mips-openwrt-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD]
Tue Jul  3 10:45:35 2018 OpenSSL: error:25066067:lib(37):func(102):reason(103)
Tue Jul  3 10:45:35 2018 OpenSSL: error:25070067:lib(37):func(112):reason(103)
Tue Jul  3 10:45:35 2018 OpenSSL: error:260B6084:lib(38):func(182):reason(132)
Tue Jul  3 10:45:35 2018 OpenSSL: error:2606A074:lib(38):func(106):reason(116)
Tue Jul  3 10:45:35 2018 OpenSSL: error:25066067:lib(37):func(102):reason(103)
Tue Jul  3 10:45:35 2018 OpenSSL: error:25070067:lib(37):func(112):reason(103)
Tue Jul  3 10:45:35 2018 OpenSSL: error:260B6084:lib(38):func(182):reason(132)
Tue Jul  3 10:45:35 2018 OpenSSL error: cannot load engine 'cryptodev'
Tue Jul  3 10:45:35 2018 Exiting due to fatal error

Is there any guide, readme to follow?
Thank you

P.S.

openssl engine
(dynamic) Dynamic engine loading support

openssl version -f
compiler: ccache_cc -I. -I.. -I../include  -fPIC -DOPENSSL_PIC -DZLIB_SHARED -DZLIB -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -I/build/lede-17.01/slaves/phase2/mips_24kc/build/sdk/staging_dir/target-mips_24kc_musl-1.1.16/usr/include -I/build/lede-17.01/slaves/phase2/mips_24kc/build/sdk/staging_dir/target-mips_24kc_musl-1.1.16/include -I/build/lede-17.01/slaves/phase2/mips_24kc/build/sdk/staging_dir/toolchain-mips_24kc_gcc-5.4.0_musl-1.1.16/usr/include -I/build/lede-17.01/slaves/phase2/mips_24kc/build/sdk/staging_dir/toolchain-mips_24kc_gcc-5.4.0_musl-1.1.16/include/fortify -I/build/lede-17.01/slaves/phase2/mips_24kc/build/sdk/staging_dir/toolchain-mips_24kc_gcc-5.4.0_musl-1.1.16/include -znow -zrelro -DOPENSSL_SMALL_FOOTPRINT -DHAVE_CRYPTODEV -DUSE_CRYPTODEV_DIGESTS -DOPENSSL_NO_ERR -DTERMIOS -Os -pipe -mno-branch-likely -mips32r2 -mtune=24kc -fno-caller-saves -fno-plt -fhonour-copts -Wno-error=unused-but-set-variable -Wno-error=unused-result -msoft-float -iremap/build/lede-17.01/slaves/phase2/mips_24kc/build/sdk/build_dir/target-mips_24kc_musl-1.1.16/openssl-1.0.2o:openssl-1.0.2o -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro -fpic -I/build/lede-17.01/slaves/phase2/mips_24kc/build/sdk/feeds/base/package/libs/openssl/include -ffunction-sections -fdata-sections -fomit-frame-pointer -Wall -DOPENSSL_BN_ASM_MONT -DSHA1_ASM -DSHA256_ASM -DAES_ASM
15

This is how I prepared my Home Hub 5a for an OpenSSL benchmark with hardware crypto acceleration:

# download the SDK matching the OpenWrt version installed on your router:
wget https://downloads.openwrt.org/releases/18.06.0-rc1/targets/lantiq/xrx200/openwrt-sdk-18.06.0-rc1-lantiq-xrx200_gcc-7.3.0_musl.Linux-x86_64.tar.xz
tar xf openwrt-sdk-18.06.0-rc1-lantiq-xrx200_gcc-7.3.0_musl.Linux-x86_64.tar.xz
cd openwrt-sdk-18.06.0-rc1-lantiq-xrx200_gcc-7.3.0_musl.Linux-x86_64
scripts/feeds update base packages
scripts/feeds install openssl cryptodev-linux
sed -i -e 's/^PKG_RELEASE:=1$/PKG_RELEASE:=1.1/' feeds/base/package/libs/openssl/Makefile
make menuconfig
  Global build settings -> # disable all 4 options
  Kernel modules -> Cryptographic API modules -> <M> kmod-cryptodev
  # return to top level menu: "Linux Kernel Configuration"
  Libraries -> SSL -> <M> libopenssl -> 
    [*] Crypto acceleration support
    [*] Digests acceleration support
  Utilities -> <M> openssl-util
make download
make -j5
# find build results under bin/
bin/targets/lantiq/xrx200/packages/kmod-cryptodev_4.9.109+1.9.git-2017-10-04-lantiq-1_mips_24kc.ipk
bin/packages/mips_24kc/base/libopenssl_1.0.2o-1.1_mips_24kc.ipk
bin/packages/mips_24kc/base/openssl-util_1.0.2o-1.1_mips_24kc.ipk
# copy packages to router, install with opkg

Now to the benchmark results. First with OpenSSL built-in crypto (no acceleration):

# rmmod cryptodev
# openssl engine -c -t
(dynamic) Dynamic engine loading support
     [ unavailable ]

# openssl speed md5 sha1 sha256 aes-128-cbc aes-256-cbc 
[...]
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
md5               1004.44k     3689.23k    11875.46k    26415.79k    40771.58k
sha1              1221.11k     4219.29k    11907.67k    21875.71k    28983.30k
aes-128 cbc       5853.58k     6478.76k     6663.00k     6718.12k     6725.63k
aes-256 cbc       4589.95k     4977.60k     5083.90k     5106.35k     5114.54k
sha256            2342.86k     5436.57k     9545.22k    11838.67k    12626.60k

Second, with hardware acceleration through cryptodev:

# modprobe cryptodev
# openssl engine -c -t
(cryptodev) BSD cryptodev engine
 [RSA, DSA, DH, DES-CBC, AES-128-CBC, AES-192-CBC, AES-256-CBC, hmacWithMD5, hmacWithSHA1, MD5, SHA1]
     [ available ]
(dynamic) Dynamic engine loading support
     [ unavailable ]

# openssl speed -engine cryptodev md5 sha1 sha256 aes-128-cbc aes-256-cbc
engine "cryptodev" set.
[...]
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
md5                358.83k     1438.15k     5096.50k    19664.49k   163187.98k
sha1               385.79k     1484.09k     6103.97k    23343.18k   211105.93k
aes-128 cbc       5834.75k     6467.58k     6670.76k     6727.68k     6745.39k
aes-256 cbc       4592.57k     4973.93k     5084.07k     5131.64k     5134.38k
sha256            2344.61k     5447.53k     9553.07k    11782.83k    12679.79k

Bulk data transfers over VPN have a payload near 1500 bytes, so the most relevant column for this usecase should be 1024 bytes. For md5, cryptodev even brings a slowdown, and for sha1, a small speedup.
[EDIT: Please don't rely on these numbers. openssl speed should be invoked with option -elapsed as suggested by @drbrains below.]
However, aes-cbc and sha256 seem to be completely unaffected by the acceleration, perhaps it was not used at all. Maybe something about my setup is wrong, or maybe cryptodev is not fully supported by OpenSSL, I don't know.
You can also see that encryption is multiple times faster than 9 Mbit/s. The low performance of OpenVPN is likely also caused by other processing, for example by copying data from and to userspace.

Use faster hardware, or use a VPN software which encrypts/decrypts in kernel space. With IPsec on the TP-Link TD-W8980, I get a throughput of 12..15 Mbits/s without acceleration, and 27 Mbits/s with crypto acceleration enabled (all measurements with conntrack disabled). Wireguard might also have good performance, I have not tried it.

1 Like
16

IPSeC should use the hardware crypto by default. It doesn’t need the cryptodev. The same for DM-crypt to encrypt Storage devices attached to e.g. USB.

As for AES and OpenSSL, try with the “-evp” switch. With cryptodev loaded it should give better performance:

time -v openssl speed -elapsed -evp aes-256-cbc

On the MT7628 improved driver that Im optimizing I use a fallback to software below a blocksize of 200 bytes. (Thats about the break even point). I got the idea from the OMAP driver. That uses the same thing.

Once you have your device setup properly to use the driver with VPN you might look at the possibility to patch the driver to do the same.

One note on the cryptodev driver. The standard package with OpenWRT is from last year. It’s missing some patches from this April. Specifically a patch to have the zero-copy work properly which improves throughput.

1 Like
17

Thank you, this looks much better now:

# modprobe cryptodev
# openssl speed -elapsed -evp aes-128-cbc
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-128-cbc       1310.57k     3403.29k    10522.97k    22138.95k    31667.54k

# openssl speed -elapsed -evp aes-256-cbc
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256-cbc       1292.61k     3334.76k    10087.17k    20084.39k    27779.07k

/proc/crypto does not show any hardware support for sha256, so it cannot be offloaded.

18

Here's the complete benchmark redone with -elapsed, without and with acceleration.

# rmmod cryptodev
# openssl speed -elapsed md5 sha1 sha256 aes-128-cbc aes-256-cbc
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
md5                995.42k     3676.82k    11776.43k    26266.28k    40716.97k
sha1              1198.31k     4153.34k    11765.33k    21776.38k    28915.03k
aes-128 cbc       5840.58k     6461.93k     6663.08k     6702.76k     6733.82k
aes-256 cbc       4597.81k     4983.08k     5092.69k     5124.10k     5117.27k
sha256            2342.94k     5439.96k     9554.09k    11801.94k    12642.99k

# modprobe cryptodev
# for a in md5 sha1 sha256 aes-128-cbc aes-256-cbc; do openssl speed -elapsed -evp $a; done
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
md5                149.26k      559.04k     2142.72k     7494.31k    26681.34k
sha1               149.08k      563.35k     2173.27k     7643.14k    28999.68k
aes-128-cbc       1305.31k     3407.77k    10531.67k    22115.67k    31612.93k
aes-256-cbc       1275.35k     3265.73k     9839.96k    19615.74k    27699.88k
sha256             540.41k     1831.89k     5089.62k     9220.78k    12050.43k
(output edited for readability)

I wonder why accelerated md5 and sha1 are so slow, since the hardware does support them.

19

For “light” encryption or hashing the overhead of context switching and copying buffers for user space to kernel space will actually make the hardware slower. It depends on the hardware and the driver.

At first glance it seems the hardware can’t do scatter/gather itself so the driver has to provide the block per segment. Also the way it’s queue manager is written: it’s waiting for one request to finish before it can process the next request. (Not possible to queue in hardware),

Even with the same overall performance, take into consideration CPU usage. While the encryption is offloaded your CPU can do something else. Since resources and performance is limited on the average router, every little bit can help.

On different hardware, the general improvement for OpenVPN is only between 10-15%. This says a lot about OpenVPN. Like I mentioned: If you can, try changing to IPSeC. This should show much better performance.

20

Thank you for the help, the procedure and the benchmark. Crypto accelerator should help a lot at least with aes-256-cbc and 256-8192 bytes interval.

You can also see that encryption is multiple times faster than 9 Mbit/s. The low performance of OpenVPN is likely also caused by other processing, for example by copying data from and to userspace.

I found the same conclusion in a previous message. Even by using only CPU without crypto accelerator the worst openSSL performance are about 4526.87k ~ 4.5 MByte/s ~ 36 Mbit/s. So 9 Mbit/s are limited by something else...

On the MT7628 improved driver that Im optimizing I use a fallback to software below a blocksize of 200 bytes. (Thats about the break even point). I got the idea from the OMAP driver. That uses the same thing.
Once you have your device setup properly to use the driver with VPN you might look at the possibility to patch the driver to do the same.
One note on the cryptodev driver. The standard package with OpenWRT is from last year. It’s missing some patches from this April. Specifically a patch to have the zero-copy work properly which improves throughput.

Does the new coming release 18.06 of openWRT include updated driver version?

Unfortunately, protonVPN does not provide IPsec and this protocol is often blocked by the firewall.
TD-W8980 has the same SoC Lantiq XWAY VRX268 as BT Home Hub 5A. I read about wireguard, but at the moment is not well supported.

Even with the same overall performance, take into consideration CPU usage. While the encryption is offloaded your CPU can do something else. Since resources and performance is limited on the average router, every little bit can help.

You are right and this is the reason for which I'm spending a lot of time on this.