Building OpenWRT to x86_64 with Btrfs support

Community Builds, Projects & Packages
1

Hello guys! I've been using OpenWRT on a micro-ATX PC for some years, I'm now buying a MOGINSOK appliance to have a router with 2,5 and 10 GbE. I decided to use this upgrade as chance to add to it a secondary SATA SSD and use RAID1.

I also decided to do a sort of A/B deploy: have 3 partitions to hold 3 distinct rootfs+kernel and 3 Grub2 entries, so that I can deploy a new version on one of the partitions and leave previous one unchanged, so that if some issue raises I can just switch entry and get to the working environment.

I decided to use Btrfs instead of mdadm + ext4 due to advanced features it has, specially be able to clone a subvolume into a readonly snapshot.

I've been working on last days on a QEMU VM to get it working. Soon enough I learned it can't be done from imagebuilder. The issue is that, to have rootfs on Btrfs, kernel needs the driver before it mounts the partition. On most distros the driver kmod package can just be added to initramfs, but OpenWRT imagebuilder doesn't support that.

Having seen that I need to compile the kernel, and therefore (AFAIK) all packages, the 2 possibilities I found is to either compile the driver on the kernel or build a kernel+initramfs that run on RAM and somehow mount the folders I need to be persistent.

I didn't try the 2nd option yet, it seems more complex. If I can compile a kernel with Btrfs support, I can just pass to it the rootfs subvolume and I'll have all changes persist.

But I keep failing to get that to work. The build system seems to be too much complex, with multiple config files, leading to a config in one file being overwritten by the same config in another, and it seems that there are also Makefile files that hardcode some actions and ignore or override configs.

From what I've seen, recently a new config KERNEL_BTRFS_FS was even created for buildroot/.config but it applies to only a few targets.

I added CONFIG_BTRFS_FS=y to buildroot/target/linux/x86/64/config-6.6 but then for some reason when buildroot/build_dir/target-x86_64_musl/linux-x86_64/linux-6.6.93/.config is created during make process it has CONFIG_BTRFS_FS=m

I'm even starting to think it'd be easier to switch to the initramfs solution, so that kernel has access to the driver as kmod before it needs to mount anything. I had studied some solutions to mount partitions early during init process, specially /etc, but haven't tried yet.

2

During the night I got error that CONFIG_PACKAGE_btrfs-progs=y requires kmod-fs-btrfs but I had disabled it suspecting that it enforces kernel's CONFIG_BTRFS_FS=m.

I then disabled CONFIG_PACKAGE_btrfs-progs but now got the same error and found it enabled. I suspect that adding docker package enabled it.

I'm adding DOCKER_STO_BTRFS=n, let's see if now it works. I can't believe that now docker will force enabling Btrfs kmod which would lead to breaking enable Btrfs compiled to kernel.

I remember I read somewhere that when a kmod is set to be added to kernel the builder adds an empty version of the package so other packages that depend on it have their dependency solved, but that's not the behavior I've been seeing. Anybody knows how to do it?

I'd need CONFIG_BTRFS_FS=y on kernel while kmod-fs-btrfs=y is also enabled on .config but with the package empty, or at least somehow not conflicting with the kmod alrdy being on the kernel.

3

I did it.

I'm far from knowing what exactly made it work, I tried a lot of things. What I can confirm is that it works, yes, without initramfs, OpenWRT built kernel is able to boot from Grub2 running on a Btrfs subvolume and mount and start rootfs on another subvolume on the same partition.

What I did:

  • a file with kernel configs that's appended (not overwrite) to target/linux/x86/64/config-${KERNEL_VERSION} and target/linux/generic/config-${KERNEL_VERSION}
  • a file with OpenWRT build config that's placed (overwriting if exists) on .config
  • execute make defconfig to read the .config and create a new one with all configs
  • a serie of sed -i commands to enforce some configs, I'm not sure which ones are actually changed, I certainly know that CONFIG_PACKAGE_dropbear is disabled
  • a set of verifications: use of grep on a list of configs to verify if they're present on these 2 files and a few configs that are verified if not present too, many many times the process was halted on this step until I got everything working
  • make -j"$(nproc)" to compile it all, make -j1 V=s provides better log readability and takes a few hours to run, that's why I took so long on the work
  • watch -n 5 'grep CONFIG_BTRFS_FS build_dir/target-x86_64_musl/linux-x86_64/linux-6.6.93/.config' on another terminal to monitor when the file is created and if Btrfs config is properly set, many many times it came as =m even though on the previous file it was =y, idk what exactly made it finally come with =y, which is what reliably confirms that Btrfs driver will be compiled into the kernel
  • validation if bin/targets/x86/64/openwrt-x86-64-generic-kernel.bin and bin/targets/x86/64/openwrt-x86-64-generic-rootfs.tar.gz are present, followed by a nice log to make it clear if the process succeeded or not

For Grub2, I installed it from a Debian Live I built for my needs, which I also used to format the Btrfs partition. I had to do nothing special to install Grub2 with Btrfs support when Debian has it installed. When it does, I can just place /boot on a Btrfs subvolume and it works fine.

I didn't try yet but I'm sure I'll work to have multiple subvolumes for multiple rootfs instances. I placed vmlinuz on the rootfs instead of /boot and it's loaded normally.

Curiously, I didn't explicitly add btrfs-progs package but it's present. dockerd and containerd are also present, I didn't test creating container but I saw no sign they're broken.