Strongswan can not work with kernel-netlink, we must use inefficient libipsec instead.
You can test it quickly as below.
on debian 10:
root@debian:~# ip xfrm state add src 188.8.131.52 dst 184.108.40.206 proto esp spi 111 enc aes 0x117283a372113790a178721c36520021
root@debian:~# ip xfrm state
src 220.127.116.11 dst 18.104.22.168
proto esp spi 0x000000c9 reqid 0 mode transport
enc cbc(aes) 0x117283a372113790a178721c36520021
anti-replay context: seq 0x0, oseq 0x0, bitmap 0x00000000
sel src 0.0.0.0/0 dst 0.0.0.0/0
on OpenWrt trunk:
root@OpenWrt:~# opkg list-installed|grep kmod-ipsec4
kmod-ipsec4 - 5.4.80-1
root@OpenWrt:~# ip xfrm state add src 22.214.171.124 dst 126.96.36.199 proto esp spi 111 enc aes 0x117283a372113790a178721c36520021
RTNETLINK answers: No such file or directory
I tested x86, rockchip and ipq40xx target, only ipq40xx could work, the others failed.
strongswan has two implemention of ipsec. one is processed by linux kernel's xfrm framework, this is recommended, but is broken now. the other one is processed by userspace process like openvpn, it works, but not efficiently.
I doubt there is a bug, as strongSwan works just fine for me without strongswan-mod-kernel-libipsec. I have multiple tunnels on multiple x86/64 devices, and also on a mips64 device, and they have been working just fine for years.
As this is unlikely to be a bug, please don't open an issue on Github. I would suggest to go on IRC and ask for help there or use the ML.
I can confirm that IPsec works just fine on an MT7621 target. Maybe "broken" means that there is a missing package which is not selected by default for your use case.
@stintel, since we are on this subject: Do you have an alternate init.d script for "swanctl" vs "ipsec and starter". This is the new default method according Strongswan but OpenWrt is still using the old starter. Users should convert their ipsec.conf and ipsec.secrets into swanctl.conf.
Thanks to @drbrains, manually installing kmod-crypto-ctr fixed this issue. For some reasons, after upgrading to latest snapshot, it's not be selected along with kmod-ipsec or strongswan-default like before. In fact, when installing kmod-crypto-ctr, kmod-crypto-seqiv, kmod-crypto-rng, kmod-crypto-sha256 also be installed.