I am trying to use SCTP on my network, as a part of which I want to set up the firewall for IPv6 SCTP.
Here is more detail on what I’ve been trying to do https://community.ui.com/questions/SCTP-support-for-EdgeOSv2/7b50504f-8966-45f2-a1dd-f5571d3a7873 . I need to compile the equivalent of nf_conntrack with SCTP support. But when I make menuconfig I can’t find the necessary option to add SCTP support to netfilter.
Can someone help me here?
ubnt@ubnt:~$ sudo modprobe sctp
modprobe: FATAL: Module sctp not found in directory /lib/modules/4.14.54-UBNT
this isn't openwrt, we know nothing about the stock firmware on your Uni device.
It appears you are using firmware that is not from the official OpenWrt project.
When using forks/offshoots/vendor-specific builds that are "based on OpenWrt", there may be many differences compared to the official versions (hosted by OpenWrt.org). Some of these customizations may fundamentally change the way that OpenWrt works. You might need help from people with specific/specialized knowledge about the firmware you are using, so it is possible that advice you get here may not be useful.
You may find that the best options are:
If you believe that this specific issue is common to generic/official OpenWrt and/or the maintainers of your build have indicated as such, please feel free to clarify.
Nice crashes. Your device supports OpenWrt, when you install it we can talk.
Oh I’m sorry I did not mention this! I have installed OpenWrt on my EdgeRouter-X device (version string : OpenWrt 23.05.6 r24232-539228933c / LuCI openwrt-23.05 branch git-25.222.75657-7ce34fe ). This, along with other IPv6-related lacunae, was the final straw for me to bite the bullet and make the jump.
It’s an officially supported router (https://downloads.openwrt.org/releases/23.05.6/targets/ramips/mt7621/openwrt-23.05.6-ramips-mt7621-ubnt_edgerouter-x-initramfs-kernel.bin ).
OK, now sysupgrade to supported version from interim installer image. Then make kernel_menuconfig and find sctp in firewall
I have not yet upgraded to the supported version, thinking this is fine for me now - since there are some complications relating to the partition sizing on my device for 24.05. Does the supported version have more kernel compilation options or something?
I just cannot find SCTP in firewall section. Not even a search with / revealed anything useful, like "sctp" protocol match support (NEW) that I found for the ER-X kernel source (only found the module that allows the router itself to use the protocol, which is not that useful). Can you perhaps check and let me know what I’m missing here?
Isn't it just included in nftables?
https://wiki.nftables.org/wiki-nftables/index.php/Quick_reference-nftables_in_10_minutes#Sctp
Have you checked on a current OpenWrt version first?
It is hidden under netfilter (not enabled bu default in OpenWrt)
CONFIG_NF_CT_PROTO_SCTP
not to confuse with xt (iptabled-legacy) sctp support.
And stateless filter works without it
#!/usr/sbin/nft -c -d netlink -f
table inet t {
chain c {
meta l4proto sctp th dport 123
}
}
The manual nft command worked! Does this mean SCTP is already part of the base nftables/xtables/netfilter? And how can this be propagated to the LuCI GUI such that it can accept port numbers for SCTP?
I couldn’t find anything with CONFIG_NF_CT_ in the menuconfig though (note that I’m not too acquainted with this kernel compilation thing yet). Could you help me a little with whether I should find this in the menuconfig application? Or should I find it in .config? Or should I manually type it in and not expect any other sign of its presence? Because I expect that in future I might need stateful tracking.
Well, helps if you re-read the thread. make kernel_menuconfig
You can try to complete fw4 to interpret all protocols with ports and all protocols with query-id. I dont see much need for those in a common home. That would need a backward incompatibility ie config v2.0. Significantly bigger than writing your own rules.
Apologies for missing it, but I had already tried both in the beginning : kernel_menuconfig did not yield any difference in the available options for my purpose, couldn’t find any such option… I don’t know what I’m doing wrong though!
Writing my own rules will certainly work for me, but if the OpenWrt firewall supports SCTP properly, is there any reason not to have the UI allow its proper usage?
Option is network / network options / netfilter firewall / advanced netfilter options [check] -> find sctp
If the OpenWrt firewall supports sctp it will be pretty lonely out there as nobody else will use it.
I kind of very precisely said that you have to fix up in luci and fw4 that if selected protocol is with ports like udplite then ports are presented in ui and parsed into rules. Otherwise those protocols are handled like ipsec esp without ports.
Understood mostly. Just trying to redo from scratch because something is off with my make kernel_menuconfig now (why is it compiling stuff when I just wanted the menu selection??). Brb.
So I ran into this kind of error (FAILED: lib/Support/CMakeFiles/LLVMSupport.dir/Caching.cpp.o) while makeing kernel_menuconfig which involved building llvm-bpf… apparently the toolchain has to be built before even configuring the kernel build options! Even after I fixed that one I ran into further strange errors involving AMD HSA (!!) I “solved” it by choosing to use the system llvm instead of compiling the one sourced by OpenWrt. Now I’m able to see the actual kernel options! (Earlier the menu was identical to menuconfig options , with some vague errors!)
I’ll report back after compiling — I’ll still need help to install the module in a proper way.
I was finally able to compile the kernel and the relevant modules… Thanks a lot for the help @anon63541380. I’m using https://blog.blaok.me/patching-an-openwrt-kernel-module/ as a reference in addition to the build system documentation (which is quite good actually).
Now my biggest fear is that installing the kernel (I’m presuming I can’t just insmod my compiled module to the distribution kernel) might cause it to bootloop and lock me out of the device permanently.
Does OpenWrt have any facilities to help with recovering from a bootloop? And of course, requesting some best practices around replacing the distribution kernel with a custom compiled one.
Good guidance.
You have to use sysupgrade file from your build, nor kernel. Also OpenWrt kmods will not work, make a copy of kmod-*-*pk arttifacts from build tree should you need to add one.
Normal packages work. Also note kmod-sctp and sctp-conntrack do not need eachother by any means.
The change from kernel parameter is that sctp will have proper NAT state teacker, like tcp dropping out-of-line packets.
Feel free to report fw4 bugs if * protocol is interpreted as tcp(,udp(,icmp).
Update for anyone who cares, I got back to it recently, and my worry was about how to test out the newly compiled nf-conntrack. Long story short, I was able to unload all the modules that depended upon it, unload it, and then try to load the new module. It consistently gave me a kernel freeze, which justifies my fear that if I just replaced the on-disk module it would have me stuck in a bootloop. Now I have to figure out what's wrong with the module and fix that.
Alright so a moderate update on this activity, with both good news (this post) and bad news (next post).
Good news: Thanks to OpenWRT's strong reproducible build practices, I was able to replicate the factory kernel module to assure myself that I was on the right track. There were a few tough spots and rough edges that needed a lot of reading and experimenting, but now I'm pretty comfortable with the OpenWRT build system. For future reference of people who are just starting out, here are the steps I followed to obtain a SCTP-enabled conntrack kernel module:
Clone the exact openwrt version branch
Copy device-specific config.buildinfo to buildroot
Place precompiled llvm-bpf in buildroot and set the corresponding option in menuconfig
Then make download, make tools, make toolchain
Run make target/linux/compile, make package/kernel/linux/compile
Get the .ipk at bin/targets///packages/kmod-nf-conntrack*
Extract the contained data.tar.gz, then further extract the contained module at data/lib/modules/..../*
Verify they are identical to factory
Now make kernel_menuconfig CONFIG_TARGET=subtarget, narrow the subtarget to specific device, enable CONFIG_NF_CT_PROTO_SCTP (enabling any gating options necessary to access this setting, explained below).
Edit nf-conntrack KernelPackage definition in netfilter.mk to add kmod-lib-crc32c dependency conditional to NF_CT_PROTO_SCTP
DEPENDS:= +NF_CT_PROTO_SCTP:kmod-lib-crc32c
Rerun step 5, then do steps 6 & 7
Steps 4&5, and step 10 were quite time-consuming to figure out:
I didn't want to build the world, but the build instructions aren't too clear what's the minimum needed for what. Once that was done, the make targets target/linux/compile and package/kernel/linux/compile were easier but still unintuitive and unfamiliar, so I had to double check even after getting search results.
In the middle, I spent a lot of time working with target/linux/prepare so that I could get a more "real" kernel .config. Because initially, even netfilter itself was not even enabled in the kernel menuconfig. The following thread helped me a lot to understand how OpenWRT's build system works, especially in the context of a kernel build: When building a firmware image, is there a way to apply scripts/diffconfig.sh to the kernel’s .config? . It turns out that the actual kernel config file is formed by merging several different pieces, so it wasn't really affecting anything if I enabled netfilter myself in kernel_menuconfig, since it'd get enabled later anyway. I only need it to show conntrack settings, so I only enabled what I need.
Enabling CONFIG_NF_CT_PROTO_SCTP and compiling resulted in
Package kmod-nf-conntrack is missing dependencies for the following libraries:
libcrc32c.ko
This was strange to me because this module was already enabled. Took me a while to figure out that OpenWRT packaging was a manual layer over what the kernel already has, and that CONFIG_ must not be prepended to the netfilter.mk dependency spec.
Another note is that I had to extract the .ko from the .ipk to compare, since the bare <arch>/<linux-version>/net/netfilter/nf_conntrack.ko was vastly bigger than the factory .ko on the router itself. It took me some searching to figure out that the factory kernel modules are stripped but the bare ones are not (yet) stripped.
@anon63541380 maybe kmod-lib-crc32c should be part of the official package dependency for kmod-nf-conntrack. If my understanding is correct, should I open an issue/PR/?
Now for the bad news.
As mentioned earlier, inserting the new nf_conntrack was crashing the kernel, so I tried to unload more modules, including nf_defrag_ipv4 and nf_defrag_ipv6 (which are actually compiled together with nf_conntrack, that should have given me a hint but I was too focused). Now when I tried to insert nf_conntrack, I got a failure, not a hang.
root@OpenWrt:~# insmod /tmp/nf_conntrack.ko
failed to insert /tmp/nf_conntrack.ko
root@OpenWrt:~# dmesg
[ 1265.707805] nf_conntrack: Unknown symbol nf_defrag_ipv4_disable (err -2)
[ 1265.721470] nf_conntrack: Unknown symbol nf_defrag_ipv4_enable (err -2)
[ 1265.734795] nf_conntrack: Unknown symbol nf_defrag_ipv6_disable (err -2)
[ 1265.748588] nf_conntrack: Unknown symbol nf_defrag_ipv6_enable (err -2)
Obviously it depends on nf_defrag, so I inserted the new modules. The new ipv4 defrag one went off perfectly well, but inserting the new nf_defrag_ipv6 caused a kernel hang. And as far as I could tell, there were no modules it depended on. To my disappointment, it turned out that the new nf_defrag_ipv4 was bit-identical to the factory one, while the ipv6 module was only different in a few very specific offsets.
Turns out that CONFIG_NF_CT_PROTO_SCTP causes data structures (starting fromnf_ip_net within include/net/netns/conntrack.h, up to a struct net within net/net_namespace.h) that reside within the kernel vmlinux to change in size, breaking things that are not even related to conntrack.
So its not going to be possible for me to try out these modules without flashing a whole image. And I can't risk that until I get my serial port interfacing setup going. Till then no NAT support for SCTP, so no IPv4 SCTP for me. IPv6 though …