VRF support? Testing out EVPN at home

Testing out a few EVPN concepts at home now that OpenWRT has the excellent FRR toolset available.

Hitting a blocker when it comes to routing due to the lack of VRF support. I can see that some kernel options were enabled back in the 22.03 release but there's no sign of a kmod-vrf or similar. Is this something that is expected to be added or do VRFs require a custom kernel build?

At a glance it looks like that should work, provided the compiled kernel had its option set to Y

@hauke ?

By the way this is what I see:

# ip link add vrf-1 type vrf table 1001
Error: Unknown device type.

Using a Ubiquiti EdgeRouter X on 23.05.2. I installed the ip-full package already.

Yes, looks like the problem is that it depends on the new NET_L3_MASTER_DEV symbol which defaults to n. So the kmod-vrf package won't be built by the builders.

But it should work fine if you build your own images and select "L3 Master device support" under " Global build settings -> Kernel build options", and then enable the "kmod-vrf" package showing up under "Kernel modules -> Network Support"

I don't know why NET_L3_MASTER_DEV ended up disabled by default on all targets, but I assume it's because it adds too much size overhead for a very specialized use case.

1 Like

I'll compile with it enabled and disabled and report what the size difference is.

Looks like the functionality is purely in a kmod, because when I enable that option I get exactly the same sized image built. Can you validate that finding and perhaps we can include it by default if the size is not increased?
Actually - the sysupgrade is the same size but initramfs grew a small amount:

$ ls -al bin/targets-default/ath79/generic/openwrt-23.05-snapshot-r23672-fd6831731b-ath79-generic-glinet_gl-ar750-squashfs-sysupgrade.bin 
-rw-r--r-- 1 100999 100999 6423349 Dec 19 09:35 bin/targets-default/ath79/generic/openwrt-23.05-snapshot-r23672-fd6831731b-ath79-generic-glinet_gl-ar750-squashfs-sysupgrade.bin
$ ls -al bin/targets-vrf/ath79/generic/openwrt-23.05-snapshot-r23672-fd6831731b-ath79-generic-glinet_gl-ar750-squashfs-sysupgrade.bin 
-rw-r--r-- 1 100999 100999 6423349 Dec 19 10:49 bin/targets-vrf/ath79/generic/openwrt-23.05-snapshot-r23672-fd6831731b-ath79-generic-glinet_gl-ar750-squashfs-sysupgrade.bin
$ ls -al bin/targets-default/ath79/generic/openwrt-23.05-snapshot-r23672-fd6831731b-ath79-generic-glinet_gl-ar750-initramfs-kernel.bin 
-rw-r--r-- 1 100999 100999 5617238 Dec 19 09:35 bin/targets-default/ath79/generic/openwrt-23.05-snapshot-r23672-fd6831731b-ath79-generic-glinet_gl-ar750-initramfs-kernel.bin
$ ls -al bin/targets-vrf/ath79/generic/openwrt-23.05-snapshot-r23672-fd6831731b-ath79-generic-glinet_gl-ar750-initramfs-kernel.bin 
-rw-r--r-- 1 100999 100999 5617320 Dec 19 10:49 bin/targets-vrf/ath79/generic/openwrt-23.05-snapshot-r23672-fd6831731b-ath79-generic-glinet_gl-ar750-initramfs-kernel.bin

Only 82 bytes larger tho, which seems negligible.

Nice. Then there should be no reason to disable this option. Maybe send a PR enabling KERNEL_NET_L3_MASTER_DEV by default? Or even removing it and simply set CONFIG_NET_L3_MASTER_DEV in target/linux/generic/config-*?

Looking at the kernel I see that the option hides code in a large number of files, but it's mostly a single line. The most significant code contribution is

net/Makefile:obj-$(CONFIG_NET_L3_MASTER_DEV)    += l3mdev/

but it's not that big either so you're numbers are probably correct:

bjorn@miraculix:/usr/local/src/git/linux$ ls -l net/l3mdev
total 16
-rw-r--r-- 1 bjorn bjorn  298 Feb 24  2023 Kconfig
-rw-r--r-- 1 bjorn bjorn 6743 Jul 13 08:43 l3mdev.c
-rw-r--r-- 1 bjorn bjorn   96 Feb 24  2023 Makefile

I could be wrong, but it doesn't seem like this causes any major increase in object sizes either. It does add some fields here and there but nothing big.

1 Like

Editing the default kernel file for 5.15 in target/linux/generic/config-5.15 and running defconfig doesn't seem to result in the option appearing in the .config file and I verified that installing the resulting image doesn't enable that option in kernel 5.15.

## select my architecture and target, do nothing else.
$ make menuconfig


*** End of the configuration.
*** Execute 'make' to start the build or try 'make help'.


$ grep -R L3_MASTER_DEV target/
target/linux/generic/config-5.15:CONFIG_NET_L3_MASTER_DEV=y
$ grep L3_MASTER_DEV .config
# CONFIG_KERNEL_NET_L3_MASTER_DEV is not set

Is the kernel looking elsewhere for build options? Docs seem to indicate that there's three sources of options - built-in Linux defaults, generic config and per-target/subtarget config. Seems like the generic config isn't being inserted into the .config file.

Editing the kernel config templates won't change the OpenWrt .config, only the generated kernel .config.

But you'll have to remove the config/Config-kernel.in entry and the dependency in KernelPackage/vrf too if you go this route.

The alternative is modifying the config/Config-kernel.in entry with a "default y"

Sorry for asking so many questions - which is the correct place to make the kernel config change for a PR? I have been reading the many pages of build instructions but I'm not entirely sure how my local builds are differing from the release builds. If you can link to documentation that would be great.

Sorry, but I don't really know this. Your answer is as good as mine.

1 Like

Opened a PR for this

1 Like

Looks good, I have been manually adding the module for a couple of platforms but this makes a lot of sense where we don't have storage pressure.

1 Like