Ipsec differences between devices: is kmod-crypto-ctr the problem?

For Developers
1

Two different devices (GLinet-B1300 on ipq40xx and GLinet MT300nv2 on ramips) when configured identically behave differently whilst creating an ipsec tunnel. The former device fails with a netlink error, and the later succeeds. Both are built from imagebuilder with the same config and same packages.

Analysis shows that on target ipq40xx the kmod-crypto-ctr package does not seem to work. E.g.

modprobe ctr 
failed to find a module named ctr

whereas this works on ramips device.

Looking in more detail I can see on the ipq40xx device that:

opkg files kmod-crypto-ctr 
Package kmod-crypto-ctr (4.14.131-1) is installed on root and has the following files: /etc/modules.d/09-crypto-ctr

whereas on the other device:

opkg files kmod-crypto-ctr 
Package kmod-crypto-ctr (4.14.95-1) is installed on root and has the following files: /lib/modules/4.14.95/ctr.ko /etc/modules.d/09-crypto-ctr

This can also be looked by inspecting the ipk files which do not contain a ko file within the data.tar.gz. I have checked this on build 18.06.4, but it seems to also be a problem on 18.06.3 and snapshot.

Is this a problem with kmod-crypto-ctr or is this the way it is meant to work?

I have raised (perhaps prematurely) a defect at https://bugs.openwrt.org/index.php?do=details&task_id=2486&order=dateopened&sort=desc

1 Like
2

Hi, leoc, just an additional datapoint: On a freshly compiled 4.19 ath79 snapshot, if I select kmod-crypto-ctr in make menuconfig, I do get the module at /lib/modules/4.19.71/ctr.ko. So ramips seems to be the outlier.

I'm going to compile a snapshot for a MT76 for one of my devices, and I also have a B1300 somewhere, that I can maybe reflash tonight. I will post results after the compile/test

3

Thanks for checking. If you do get module for ath79 then I think it is ipq40xx that is outlier, not ramips?

I'm rebuilding 18.06.4 from source at moment to see if I can figure out what's happening

4

Oops, i mis-remembered this detail of your post when i was replying.

Why are you compiling the old version, and not master, or 19.07?

5

I wanted to see if I could work out why it was failing in a version I knew didn't work. My hope was if I could emulate the problem by compiling from source I could then debug.

6

I've done a bit more digging; there is a patch for the ipq40xx that seems to be the culprit (https://github.com/openwrt/openwrt/blob/master/target/linux/ipq40xx/patches-4.19/181-crypto-qce-add-CRYPTO_ALG_KERN_DRIVER_ONLY-flag.patch) as it "Set the CRYPTO_ALG_KERN_DRIVER_ONLY flag to all algorithms exposed by the qce driver", which are then removed from the build in https://github.com/torvalds/linux/blob/81160dda9a7aad13c04e78bb2cfd3c4630e3afab/drivers/crypto/ixp4xx_crypto.c.
However rfc3686 isn't on this list so I can't figure out why that is not being built into the module

7

ipq40xx != ixp4xx

8

My mistake. The registration of the qce kernel algs seems to happen here: https://github.com/torvalds/linux/blob/81160dda9a7aad13c04e78bb2cfd3c4630e3afab/drivers/crypto/qce/sha.c

I can't figure out what stops the kernel modules being built though, and particularly why there is no implementation of rfc3686(ctr(aes))

9

@cotequeiroz

10

crypto-ctr is built into the kernel for ipq40xx, because it has the qce hw-crypto module built-in, so it pulls crypto-ctr as a dependency. ramips & ath79 do not have that, so it is built as a module.

The patch adding the CRYPTO_ALG_KERN_DRIVER_ONLY should not cause too much trouble. The flag is used to identify a hw-accelerated algorithm: kernel-driver-only means something that is only available through the kernel--such as DMA access by the hw-crypto engine, not some CPU instruction that userspace can use directly.
The flag is perhaps being used somewhere to select, or filter-out, an algorithm--openssl devcrypto engine, for example, uses the flag to only enable kernel-hw-accelarated algorithms--so that the qce driver was not being used before the patch, but is now, and is not working properly. It's not clear to me what versions have you tested, but the 18.06 branch does not have that patch, so test it to find out if it works.

It may be worth checking out and comparing the contents of /proc/crypto for each device. It will show you available crypto algorithms, among other things, the driver & module being used, and their priority. You may verify ctr presence there, by looking for an algorithm named ctr(aes).
Cheers

1 Like
11

Thank you. That is super useful already. I'll test with 18.06 soon, but for now running with 18.04

more /proc/crypto | grep ctr

on the ipq40xx gives:

name         : ctr(aes)
driver       : ctr-aes-qce

and on the ramips the same command gives:

name         : rfc3686(ctr(aes))
driver       : rfc3686(ctr(aes-generic))
module       : ctr
name         : ctr(aes)
driver       : ctr(aes-generic)
module       : ctr

It's the rfc3686(ctr(aes)) that I care about.

My guess from the code is the gce code is adding ctr(aes) but not rfc3686(ctr(aes)), and yet the whole ctr module is not being compiled. What I can't figure out is why, and whether it is possible to mix kernel-hw-accelarated algorithms and kernel modules (ideally I wouldn't just disable hardware acceleration) to add rfc3686(ctr(aes)) support.

Any guidance really valued.

12

I'm running on self compiled branch v18.06.4 which I can confirm doesn't contain the patch; so it isn't the patch that is stopping the ctr.ko from being built

1 Like
13

I see that the qce driver sets the ctr-mode blocksize to 16 (confirm it with /proc/crypto), when the correct one should be 1 (it's a stream cipher). ctr.c checks for this before creating a rfc3686 instance. I'm not sure if this is enough to make it work, since other drivers publish rfc3686 separately.

Apply this as target/linux/ipq40xx/patches-4.14/182-crypto-qce-fix-ctr-blocksize.patch and recompile your image:

--- a/drivers/crypto/qce/ablkcipher.c
+++ b/drivers/crypto/qce/ablkcipher.c
@@ -299,7 +299,7 @@ static const struct qce_ablkcipher_def ablkcipher_def[] = {
                .flags          = QCE_ALG_AES | QCE_MODE_CTR,
                .name           = "ctr(aes)",
                .drv_name       = "ctr-aes-qce",
-               .blocksize      = AES_BLOCK_SIZE,
+               .blocksize      = 1,
                .ivsize         = AES_BLOCK_SIZE,
                .min_keysize    = AES_MIN_KEY_SIZE,
                .max_keysize    = AES_MAX_KEY_SIZE,

Ensure that /proc/crypto shows the blocksize for ctr(aes) to be 1 after applying the patch.

14

I don't have the hardware, so I won't be able to check this on my own. I caught the bug by looking at the output of /proc/crypto that @chunkeey posted here

@jeff, @chunkeey can you also confirm if the patch works? I just mean that /proc/crypto should show blocksize=1 for ctr(aes) with the ctr-aes-qce driver.

I don't necessarily think that this will bring up rfc3686 support on its own (I'm just hoping, really); it may require more effort, but the block size should be fixed, so, I will send patches to openwrt and upstream after confirmation.

15

I forgot, as part of testing, we need to ensure it encrypts right. Please use openssl with the devcrypto engine enabled, and the AES-128-CTR cipher enabled as well, then run:

# echo '0123456789abcdefgh' | openssl aes-128-ctr -e -nopad -iv 0f0e0d0c0b0a09080706050403020100 -a -K 000102030405060708090a0b0c0d0e0f
EJjLoYB5bd88Jp2+D8r8DCDOrg==

PS: This should preferably be done in 19.07 or master, using openssl 1.1.1, so we can confirm this is actually using the qce driver with openssl engine -t -c -pre DUMP_INFO -- if not, try to confirm it via /proc/interrupts.

16

Thanks. I'll apply the patch and run the test over the next couple of days.

17

I used the patch on master (191c3e49b9) which didn't work until I modified

@@ -299,7 +299,7 @@ static const struct qce_ablkcipher_def ablkcipher_def[] = {

to become

@@ -291,7 +291,7 @@

once applied the patch seemed to work and blocksize becomes 1 (instead of 16, which I verified was the previous result)

name         : ctr(aes)
driver       : ctr-aes-qce
module       : kernel
priority     : 300
refcnt       : 1
selftest     : passed
internal     : no
type         : ablkcipher
async        : yes
blocksize    : 1
min keysize  : 16
max keysize  : 32
ivsize       : 16
geniv        : <default>

However rfc3686 still doesn't appear.

I was also unable to run the test, because it prompted for a password and when provided with one completed with no obvious output.

Running openssl engine -t -c -pre DUMP_INFO produced the following (which doesn't look good)

3069834596:error:260AC089:engine routines:int_ctrl_helper:invalid cmd name:crypto/engine/eng_ctrl.c:87:
3069834596:error:260AB089:engine routines:ENGINE_ctrl_cmd_string:invalid cmd name:crypto/engine/eng_ctrl.c:255:
     [ unavailable ]
18

I forgot that, in order to minimize the use of ioctl, openssl will always send a full 16-byte block when performing AES-CTR, so it ends up not being a good test for this. I will provide instructions to run openssl nonetheless.

It seems your openssl is not configured to use /dev/crytpo. You must have libopenssl-devcrypto installed first. Then, edit your /etc/ssl/openssl.cnf and add these lines right before the first [section] line (it's usually [ new_oids ]). Technically, the openssl_conf=openssl_conf line needs to be placed in the first unnamed section, the other lines can go elsewhere:

openssl_conf=openssl_conf

[openssl_conf]
engines=engines

[engines]
devcrypto=devcrypto

[devcrypto]
default_algorithms = ALL

Then, try the openssl engine command again.

PS:
These instructions are for openssl 1.1.1--at least here, it does not ask for a password, are you using 1.0.2?

I used the patch on master (191c3e49b9) which didn't work until I modified

Just to be clear: changing the offset is fine. It won't affect the outcome.

19

Thanks for the guidance.

The test now runs, but produces a different result to yours:

# echo '0123456789abcdefgh' | openssl aes-128-ctr -e -nopad -iv 0f0e0d0c0b0a09080706050403020100 -a -K 00010203
0405060708090a0b0c0d0e0f 
EJjLoYB5bd88Jp2+D8r8DEfB8w==

and:

openssl engine -t -c -pre DUMP_INFO

gives:

(dynamic) Dynamic engine loading support
[Failure]: DUMP_INFO
3069621604:error:260AC089:engine routines:int_ctrl_helper:invalid cmd name:crypto/engine/eng_ctrl.c:87:
3069621604:error:260AB089:engine routines:ENGINE_ctrl_cmd_string:invalid cmd name:crypto/engine/eng_ctrl.c:255:
     [ unavailable ]
(devcrypto) /dev/crypto engine
Information about ciphers supported by the /dev/crypto engine:
Cipher DES-CBC, NID=31, /dev/crypto info: id=1, driver=cbc-des-qce (hw accelerated)
Cipher DES-EDE3-CBC, NID=44, /dev/crypto info: id=2, driver=cbc-3des-qce (hw accelerated)
Cipher BF-CBC, NID=91, /dev/crypto info: id=3, driver=cbc(blowfish-generic) (software)
Cipher CAST5-CBC, NID=108, /dev/crypto info: id=4, CIOCGSESSION (session open call) failed
Cipher AES-128-CBC, NID=419, /dev/crypto info: id=11, driver=cbc-aes-qce (hw accelerated)
Cipher AES-192-CBC, NID=423, /dev/crypto info: id=11, driver=cbc-aes-qce (hw accelerated)
Cipher AES-256-CBC, NID=427, /dev/crypto info: id=11, driver=cbc-aes-qce (hw accelerated)
Cipher RC4, NID=5, /dev/crypto info: id=12, CIOCGSESSION (session open call) failed
Cipher AES-128-CTR, NID=904, /dev/crypto info: id=21, driver=ctr-aes-qce (hw accelerated)
Cipher AES-192-CTR, NID=905, /dev/crypto info: id=21, driver=ctr-aes-qce (hw accelerated)
Cipher AES-256-CTR, NID=906, /dev/crypto info: id=21, driver=ctr-aes-qce (hw accelerated)
Cipher AES-128-ECB, NID=418, /dev/crypto info: id=23, driver=ecb-aes-qce (hw accelerated)
Cipher AES-192-ECB, NID=422, /dev/crypto info: id=23, driver=ecb-aes-qce (hw accelerated)
Cipher AES-256-ECB, NID=426, /dev/crypto info: id=23, driver=ecb-aes-qce (hw accelerated)

Information about digests supported by the /dev/crypto engine:
Digest MD5, NID=4, /dev/crypto info: id=13, driver=md5-generic (software), CIOCCPHASH capable
Digest SHA1, NID=64, /dev/crypto info: id=14, driver=sha1-qce (hw accelerated), CIOCCPHASH capable
Digest RIPEMD160, NID=117, /dev/crypto info: id=102, driver=rmd160-generic (software), CIOCCPHASH capable
Digest SHA224, NID=675, /dev/crypto info: id=103, driver=sha224-generic (software), CIOCCPHASH capable
Digest SHA256, NID=672, /dev/crypto info: id=104, driver=sha256-qce (hw accelerated), CIOCCPHASH capable
Digest SHA384, NID=673, /dev/crypto info: id=105, driver=sha384-generic (software), CIOCCPHASH capable
Digest SHA512, NID=674, /dev/crypto info: id=106, driver=sha512-generic (software), CIOCCPHASH capable

[Success]: DUMP_INFO
 [DES-CBC, DES-EDE3-CBC, AES-128-CBC, AES-192-CBC, AES-256-CBC, AES-128-CTR, AES-192-CTR, AES-256-CTR, AES-128-ECB, AES-192-ECB, AES-256-ECB]
     [ available ]

I can confirm that I'm on OpenSSL 1.1.1d 10 Sep 2019.

20

I've started adding debug statements inside ctr.c which shows that crypto_ctr_alloc is executed, but neither crypto_rfc3686_create or crypto_rfc3686_create are.