Vsock.ko kernel module needed (aarch64/X86_64 targets)

For Developers
62

Can you share the profile corresponding to the instance that works for you? I have just confirmed virtiofsd works when manually mapping a volume into an instance. But when starting your test images, no such mappings appear to exist. I am trying to understand where the missing channel is supposed to be configured in the first place.

edit:
@tregubovav.dev I have narrowed down my issue to the call to mount_incus agent "${MNT_P}" in /lib/functions/incus_functions.sh. The config mount does work.

I can reproduce it with:

root@OpenWrt:~# mount -t 9p agent -o access=0,trans=virtio,size=1048576 /run/inc
us_agent/.mnt/
[17082.391687] 9pnet_virtio: no channels available for device agent
mount: mounting agent on /run/incus_agent/.mnt/ failed: No such file or directory

So, the question is, what on the host configures the availability of this agent mount over 9p. And moreover, if it fails, but everything still works, why is it needed in the first place?

FWIW, the documentation only mentions config, not agent.

63

So, I had a go at packaging: first attempt. It's based on the init script that @tregubovav.dev shipped in his test images. Like that image, it does not actually package incus-agent, but merely an init script to copy the agent from the 9p 'config' mount. All feedback welcome, including offers of co-maintenance. I guess we can time the PR to coincide with the kernel bits?

@tregubovav.dev can you clarify the applicable license on those init scripts?

In my second commit, I deleted the copy_agent() function and solved the error I shared previously. The incus-agent is already copied over from the config mount by copy_config(), so it turned out there was no need (and no more error messages about non-existing mount points).

64

Hello @Sagi!

Thank you for the investigation and feedback. Your feedback is helpful.
See my answers and comments below:

  1. Incus does not ship the incus-agent as a separate package. The incus-agent for Linux and Windows are the part of the incus software packages.
  2. incus-agent initialization scripts should not be delivered as an OpenWRT package. It should be integrated to the incus OpenWRT image by lxc-distrobuilder.
  3. I built the incus-agent initialization script based on alpine Linux OpenRC scripts: from lxc distrobuilder
  4. The copy_agent() call is not needed anymore as the config virtual disk contains the incus-agent' bootstrap script now. This script does the same thing as copy_agent()` function (which was originally written in the middle of January 2025).
    I will see how I can simplify and rewrite the init script in a couple days.
  5. About script license: As this could be a part of images distributed through the images.linuxcontainers.org - it would be Apache 2.0 license.
65

Just to inform everyone in the thread.

I published PR today.

66

I would appreciate it if someone could give me a suggestion on how to resolve the build failure.
The build process stops with a explicit requests to disable or enable kernel config key .

67

Hi, I would also really like to see an Incus vm image for OpenWRT.

I have looked at your work, @tregubovav.dev, and I think we are on the right
track here. However, as Hauke said, it might be a better idea to build the drivers as modules, instead of building them into the kernel.

OpenWRT always tries to keep the kernel as small as possible and as not all devices deployments of the x86/64 or armsr targets will be running as a vm guest, having extra modules specific to vm guests built-in is not desirable.

I propose we build the drivers as modules and ship them as a kmod-* package. This does not affect installations that don't explicitly install these modules, so it has a low chance of causing regressions and does not add unnecessary overhead on the systems that don't need it.

A first attempt at packaging the relevant modules can be found here.
(I dropped theCONFIG_VSOCKETS_LOOPBACK option, as I could not find why it was needed and my vm worked fine without it).

For me incus-agent works when building for the x86/64 subtarget and selecting the following options in openwrt's make menuconfig:

CONFIG_PACKAGE_kmod-9pnet=y
CONFIG_PACKAGE_kmod-9pvirtio=y
CONFIG_PACKAGE_kmod-fs-9p=y
CONFIG_PACKAGE_kmod-fs-netfs=y
CONFIG_PACKAGE_kmod-vsock-virtio=y

This builds the kmod packages and includes them in the final rootfs image.

I have not tested this for armsr/armv8 as I don't have the hardware readily available, but I expect it to work without major problems.

It is good to note that on armsr/armv8 (as you already mentioned) vmw_vsock_virtio_transport_common and vsock are not built-in, so those kmods would automatically be selected as dependencies of vmw_vsock_virtio_transport by the build system.

The 9p modules are built-in on armsr/armv8, so those would not be needed.

As for distrobuilder, I wanted to start porting the openrc script, but then I realized some work has already been done on that front.
I searched a bit, but I could not find it. Would you mind sharing your progress on the changes to distrobuilder?

P.S. I also found that virtio-console-helper (a subpackage of qemu) does exactly what we need for the virtio-ports symlinks. One less problem to tackle.

68

I was just about to ask if there is any progress on this. :slight_smile: It is unfortunatly over my knowledge level at the moment to help in building but for sure I can assist in testing it. Thanks for the work everyone.

69

Hello Folks,

The major issue that almost nobody is interested in this change. My Pull Request is stuck for 5+ month as nobody wants to review it. I couldn’t wait for so long time and I have to move my customers to another solution outside of OpenWRT.

Of course, we can discuss how to better build modules (build-in or via kmod) and how to run Incus Agent automatically. But before that we need to engage six Maintainers to review proposed changes and get their feedback. Nothing will happen without that.

If you like to build image that runs agent automatically you can execute following command into files directory in OpenWRT build system. It will extract OpenRC file support function module, plus adds a links for Incus Agent autostart.

cat <<EOF | base64 -d | tar xz
H4sIAAAAAAAAA+1ZbW/bNhD21+hXMKpQJ0WtN9tx20DYisTIgi4vsNN+2QZDkehEiExqotSmTf3f
dyQlWZLlplnrtMN8SGKJd7w73nM8hmfdaK2dTKBBv88/rUHfLH/m1LJ6g/4Afm2r2zItc69rtlB/
/a61WilL3BihVoJZguPVcvfx/6OkG2FwueYceDj+Vnewt8H/MUjiP02JlwSUsLVkwsPx7/bMDf6P
QnX8A+KlbFK86+z6221wgPd6vVX49yzblvjDOH8GrtW1W8j8dtP30/8c/yfbxmVADHatKMenB2/H
k9dHw9OLyfnri98c1YhTkmWEe4VJolZlRmdHjirYnWW2o2p3dY1zozYGGuaqouTphkJ6tbOL7hQE
dD46Bh2WKl5ODsGUZsuXP1DnEwL1IDFX0V/o6dNi6GR8lA/FOEljIiaA2isco06EfBfPKNHlXNRJ
kFpeHtdgg0PzkktTNwgLn0C9o1m5E6RmkXPbb8kNoR8IwnFM43ZuHrXlezFDMPBtkCCrYo6B19Hk
fUTjhBVmpzRGAQoI0nauYhyhdhsZ7CMzvNBlzHgfxDC1I6YYzwzizvCumOZTpGy9Oz8bcTB2Ll2G
OQ+U+EEsn+6C+e6uCkK/H5++qQpxlrqPJEe7E59Pnr2aK1uzG5jPg6kaPn5fNa/yOPypbGUR8jIZ
mM/dyAOVCYQEddgUqbpeEmhQmlnPguZTgish82LsJnhSAnICDhbBW7jbkJBS5YymJOHZkMyiKcv+
NoqjDgVpHzsmFKvnLPiEHbt3UnFHKJPeFD4cDt/leXPCdTmaLV6Go5GjSh+CaZHEIC0i9fnzIq/5
LD4oZJNrTABanlcirRBknXsZYpTQbC2gTYU4Bh5WkUt8A2RUyYlowDM9cmOAGSoaQ26MEaEJTz29
rWzJfYPsfl/mXlB4qu0UgXoZLRyFkLiehxlzzOdJ7BLmSPRkeCyz96I/2CstQibn8MDRfsl2knY3
PIDdiP9GpkyQpbiIrQajlRCUI6BqwFUL77XhQe59JVVo9HHiUTINFnVGAtJYrfQZL2r1JMqWscgc
CTaSeusSXoQ6I7c0ajz7QiamMsI1HfGMW68rvuZlpjNCMaXJK/5nhd6lAIg98v3XL9Q2LX+6GGw6
APQwIOmtru2kovJ0ZrtqbSVfu+KHBRHisghMuXpAWWal6gvp56+oBvnZc7vkSPkMgqyGrZw9V8t9
2Sz/R2QBi7fC5j5UT87XjTpbHqagvaqURosj9fhwDLs4Cnw6XdIupu9WzjYuP9eytfBDCAbEMcQZ
+/x8uQnCEAnB+X5WmX/0fzQbegjpBk68n+/+3zO7m/vfY5DEP/Z0f31J8HD8bbtrbfB/DCrhP7bN
0snxHW0I/AeDr8ffNvv9QQvZcDEISJCAb+txjNMG/xz/Ny/7Pw/+e90N/o9BEv8symuy8S/qf3+w
+f7nUaiC/5p22T34272eVev/9gb8+59N/3f9VPR/UX4QeHQ2o0RR3o6H/FJ4cOhYyvji9ejCsU14
ODt3XvaVardYsts60o7P3xxNjk/HF6Ozs4t7v1rYX77/thXlEi72xZVVDMJd9FdxsxVvE4Zj3tkq
ZB5sV8wqWrM+vkyvkDrGSRKQK7Tcf0RppOu67CKU+7P8VlxS0DARpDtphCCkUYgT7G+rvBPAW8pf
Eud8Ltvo5sE19m64n8EUNdzgUcAgqBCnEFQUXoPsdkOLo9ZJyyys0MpbhIXm5/kjeMKtIGWruQ1b
aveWmm/1Udk6KnWGv+zNtcvQJcZk4Y8MbDZbxrdxGVLeFb2XQksl3jhk+GvCUZhGbggr9z/qaIRT
JpBJisBPA0V8RjH1/AmNMJmIicTDpXHAfSIasjxTZi7xUb4pO7de3QOx2bKeVn06S3yaJshcxcRx
vIIZYxa5HwiyXpjI4j8lKS+kDC/8FvuQRt9pG/JIB2RKYQOC0ojHr6lHWGRyrbeV70GpogkrLhXB
XhAZUm4YN8jKTOBtdD5rm9ecH12gN7ShDW1oTfQPJFaFcAAoAAA=
EOF
70

I’m completly agree with the solution, not to add extra and unneeded/useless kernel modules on default, this makes the image only bigger.

And yes, I count bytes, some people may use the image for TFTP

So what you can do, to make people happy ?

add

define Device/container

in

target/linux/x86/image/64.mk
target/linux/armsr/image/Makefile

with the needd changes and kmods

add extra kmods into

target/linux/x86/modules.mk
target/linux/armsr/modules.mk

Sample code for this are in the files

71

I have a couple concerns:

  1. List of already built-in modules for arm8 and x86_64 platforms is different. This means that some kmods will be just placeholders. (and it will bring concern from maintainers).
  2. Making modules built-in from my list will not impact to the image size significantly. It adds a few dozen KBytes only and will not impact to the performance at all.

However, I would appreciate it if someone can do experiment with modules as kmods. I do not have enough experience in that.

72

Maybe I wasn’t clear in my original post, but I have packaged the modules needed for incus-agent to work as kmods in package/kernel/linux/modules/virt.mk

This adds them for all architectures, as opposed to adding them in target/Linux/<target>/modules.mk, which is only for target specific modules. These modules might be most used on x86/64 and armsr, but they have no reason not to work on riscv64.

When building openwrt like this and installing the modules, incus-agent works.

List of already built-in modules for arm8 and x86_64 platforms is different. This means that some kmods will be just placeholders. (and it will bring concern from maintainers).

I am afraid I don’t quite follow you here. What do you mean by placeholder?

The list of built-in modules is indeed different between the targets, but I solved it by having different dependencies based on what target you are building for:

The +(!TARGET_x86_64&&!TARGET_loongarch64):kmod-vsock line means that kmod-vsock is only a dependency if we are not on the x86/64 or loongarch64 targets (they have the vsock module built-in to the kernel, so the separate kmod package is not needed). On all other targets, vsock is not built-in, so it is a dependency there (e.g. on armsr).

Note this is used extensively in other kmod packages throughout OpenWRT.

I hope this clarifies things :slight_smile:

73

After some more research, it turned out that I was overcomplicating all of this.

I assumed that trying to package a kmod that has been built into the kernel would fail, as there would be no .ko file to package. When browsing the packages for armsr/armv8, however, I found that there was a package for kmod-fs-9p. This is odd, as that module is built into the kernel on this subtarget. When extracting the package I found that it did not contain the module, only the file for auto loading it (in /etc/modules-load.d). Then I started digging and found a lot of kmod packages which were dummies. It turns out that the build system handles all this transparantly. The convention seems to be just to let your kmod package depend on the other kmods it needs. If a module is built-in on some target, the package is just empty, but the dependency relation stays the same.

When applying this to our case, we get the following behaviour:

  • on x86/64, installing kmod-vsock-virtio pulls in kmod-vsock and kmod-vsock-virtio-common, which are both empty.
  • on armsr/armv8 it also installs all three packages, but because the modules are not built-in, kmod-vsock-virtio-common and kmod-vsock actually contain the modules.

This is a lot cleaner and significantly simplifies the dependency tracking and maintenance. Should the kernel config change on some target, the kmod package now automagically includes the module file, resulting in everything continuing to work without a hitch. Looking for this in the source tree revealed that there are tons of kmods packages that are really just dummies on some targets. I was unaware of this, as the wiki page on packaging kmods makes no mention of this whatsoever.

I have updated the kmods to be packaged like this (drop the target specific depends).
Additionally kmod-vsock-virtio needs CONFIG_VIRTIO set in the kernel config, which can be checked by depending on VIRTIO_SUPPORT in OpenWRT, so package is only visible on targets that support virtio (see also kmod-9pvirtio).

Let me know what you guys think of the current state. If there are no new problems, I think this is ready to be submitted as a pr.

74

Concept pr can be found here.

75

Thank you for the investigation. You are correct, the OpenWRT have tons of kmod-* packages that contains the only metadata. This is known behavior.

Have you checked that the kernel modules packages can be installed with the default image?
I had issues in that. Even modules are marked as loadable, the kernel config is changes and enforces the new signature making official images not compatible with the ne kernel modules.

76

I applied the patch to the openwrt-24.10 branch and it compiled successfully. Running it works, but installing the *.ipk packages from my branch on a vanilla image of openwrt 24.10 does not work, because indeed, the kmod packages depend on the specific kernel they were compiled with (by the hash in the package version).

When extracting the *.ipk packages and trying to manually load the module with insmod, it fails with the following error:

root@OpenWrt:~# insmod /vmw_vsock_virtio_transport.ko 
[  290.425358] module vmw_vsock_virtio_transport: .gnu.linkonce.this_module section size must match the kernel's built struct module size at run time
failed to insert (null)

So it seems like the ipk files depend on the specific kernel build for a reason.

In conclusion, the modules don't load on a stock openwrt image.

However, I think we should not expect them to. If I understand correctly, in-tree modules are compiled in the same run as the kernel, and are only loadable by the kernel resulting from that build. For a module to be loadable by another kernel, it should explicitly be build out-of-tree with something like dkms. This is not the case with our modules, so I am not suprised the modules don't load on other kernels (even though they are the same version).

77

Search the forum for vermagic.

78

This is why the incus support require kernel changes, neverminded if these changes adds statically linked or dynamically loaded modules. My PR unifies a bit list of static modules between armv8 and x86_64 platforms and increases the the kernel image size for a few KiB only. In fact it much simpler change than to switch to kmod approach.

79

You got review feedback on your PR that @rietbergenm is now acting upon (yay!).

80

I have opened a pull request on github with my changes. Let's see what the maintainers have to say :slight_smile:

81

Build the whole firmware image from source, flash new image - the vermagic isn't just there for decoration and annoy you (subsystem changes like these dooo change the kernel ABI, considerably and in incompatible ways).