Are there any good guides for customizing/optimizing the kernel config when building OpenWrt (via toolchain)?

For Developers
1

TL;DR: Are there any good guides (or even good general advice) for how to optimize the kernel (via make kernel_menuconfig) in a way that doesnt break it?

My device (netgear R9000) isn't officially supported, but egorenar made it possible to build an image that works for the R9000 using the full openwrt build system and toolchain (i.e., not the imagebuilder).

After (way too) many rebuilds over the past few years I got my config pretty well set up and customized. This includes:

  • using unbound+odhcpd+adblock for DNS+DHCP (with things like libevent and pthreads and the subnetcache module compiled into unbound)
  • running a plex media server instance on the router (with media on a usb-attached external HDD)
  • using ksmbd for a smb3 server
  • setting up 802.11r/v/k + neighbor reports + usteer for fast roaming / band steering (this worked not so well, but I think that is due to the STA's, not the router)
  • compiling in eBPF support and using things like bridger and qosify
  • compiling the target with CFLAGS optimized for my R9000 (currently I use "-O3 -pipe -funsafe-math-optimizations -mfpu=neon-vfpv4 -mcpu=cortex-a15 -mtune=cortex-a15 -mfloat-abi=hard -mtls-dialect=gnu2 -marm")
  • compiling in most of the required custom configuration (using the files dir in the buildroot)
  • etc.

So, at some point I started tweaking the kernel's configuration too (via make kernel_menuconfig). In doing so, I've discovered it is quite easy to tweak the kernel in ways that make the image either not compile or compile and not run (bootloop) or compile and run but be unstable.

Through trial and error (mostly error) Ive figured out a little of what I can/cant (or should/shouldnt) do to get an image that works (and hopefully works better than if I hadnt screwed with the kernel config), but for most of the config options (and there are a lot of kernel config options) I basically just scan through and see what looks interesting and try it and see if it works or not. Which sorta sucks and is a rather slow process.

Also, I have a couple of specific questions:

QUESTION: When something in the kernel config has an analogous/related option in the userspace .config (set by make menuconfig), how can I figure out which of the following options is the case:

  • Option 1: The kernel config item is required by the userspace config item.
  • Option 2: The kernel config item enhances the userspace one (i.e., it is optional to add to the kernel config but could potentially make things work better or give extra functionality)
  • Option 3: The kernel config item is incompatible with the userspace one, and enabling it will cause things to break
  • If option 1 or 2: how do i know if it needs to be a module or a built-in (or if either is OK)?

I'm fairly sure that most of the kmod- userspace .config options require the analogous kernel option and (sometimes?) require it as a module, and that adding support for additional crypto mostly falls in the "it enhances things" category, but for most of the stuff under the "device drivers" and "networking" and "filesystems" kernel config submenus I dont know which of the above options in the case. A few examples would be:

  • adding additional netfilter/nftables functionality
  • adding in support for various additional networking protocols
  • adding in support (or additional functionality) for filesystems I know I will use, including UBIFS, overlayfs, ntfs[3] and (compressed) squashfs filesystems
  • adding in device-specific drivers, both with and without analagous userspace .config options. example: the annapurna labs ethernet driver or crypto hw acceleration driver (both of which have analogous userspace config options), and the annapurna error-correcting ECC driver (which doesnt have a userspace /.config option).

QUESTION: Which compilation warnings/errors can safely be overlooked/ignored, and which should really be looked into (even if they dont stop the compile process) since they mean the image probably wont work?

  • can-be-overlooked/ignored example: a few packages (especially those that use cmake instead of make) will put a -Werror after the user-specified compile-flags and then break on a harmless error (like "this feature will be depreciated in the future, use _____ instead") because you are using a newer gcc.

  • should-probably-look-into example: ive found that when the kernel is compiling (e.g., when you run make after running make kernel_menuconfig) and incorrect type -- expected ____, got ____ warnings typically mean something is wrong with or missing from the kernel config, even though those often dont actually stop the compile.

side note: I use the logging feature (enabled in make menuconfig) so I can more easily search through the compilation logs for specific warnings/errors and when an error happens I can easily figure out which make` command was responsible.

Any insights on how to work this out would be much appreciated. Thanks in advance.