Strongswan can not work with kernel-netlink, we must use inefficient libipsec instead.
You can test it quickly as below.
on debian 10:
root@debian:~# ip xfrm state add src 1.2.3.4 dst 5.6.7.8 proto esp spi 111 enc aes 0x117283a372113790a178721c36520021
root@debian:~# ip xfrm state
src 1.2.3.4 dst 5.6.7.8
proto esp spi 0x000000c9 reqid 0 mode transport
replay-window 0
enc cbc(aes) 0x117283a372113790a178721c36520021
anti-replay context: seq 0x0, oseq 0x0, bitmap 0x00000000
sel src 0.0.0.0/0 dst 0.0.0.0/0
on OpenWrt trunk:
root@OpenWrt:~# opkg list-installed|grep kmod-ipsec4
kmod-ipsec4 - 5.4.80-1
root@OpenWrt:~# ip xfrm state add src 1.2.3.4 dst 5.6.7.8 proto esp spi 111 enc aes 0x117283a372113790a178721c36520021
RTNETLINK answers: No such file or directory
I tested x86, rockchip and ipq40xx target, only ipq40xx could work, the others failed.
strongswan has two implemention of ipsec. one is processed by linux kernel's xfrm framework, this is recommended, but is broken now. the other one is processed by userspace process like openvpn, it works, but not efficiently.
I doubt there is a bug, as strongSwan works just fine for me without strongswan-mod-kernel-libipsec. I have multiple tunnels on multiple x86/64 devices, and also on a mips64 device, and they have been working just fine for years.
As this is unlikely to be a bug, please don't open an issue on Github. I would suggest to go on IRC and ask for help there or use the ML.
I can confirm that IPsec works just fine on an MT7621 target. Maybe "broken" means that there is a missing package which is not selected by default for your use case.
@stintel, since we are on this subject: Do you have an alternate init.d script for "swanctl" vs "ipsec and starter". This is the new default method according Strongswan but OpenWrt is still using the old starter. Users should convert their ipsec.conf and ipsec.secrets into swanctl.conf.
I'm trying to reproduce your problem but have had no luck so far. If I remove the esp4 or xfrm_user modules, I'm getting a different error:
root@wr0:~# ip xfrm state add src 1.2.3.4 dst 5.6.7.8 proto esp spi 111 enc aes 0x117283a372113790a178721c36520021
Cannot open netlink socket: Protocol not supported
What version of OpenWrt are you running (please post content of /etc/openwrt_release)? Do you use (snapshot) images or are you building yourself?
Thanks to @drbrains, manually installing kmod-crypto-ctr fixed this issue. For some reasons, after upgrading to latest snapshot, it's not be selected along with kmod-ipsec or strongswan-default like before. In fact, when installing kmod-crypto-ctr, kmod-crypto-seqiv, kmod-crypto-rng, kmod-crypto-sha256 also be installed.