Extra-Extra Qdiscs

You can run make kernel_menuconfig and then search for SCH_HHF using / and then statically build it in to your kernel as [*]. If you want to make it a module, you'd have to modify the build system: Make kernel_menuconfig and modules

Thanks. It's been awhile since I've built an image. The documentation seems pieced together. Any updated instructions?

Same as always. Just be running a linux distro with the prerequisites.

git clone https://github.com/openwrt/openwrt.git
cd openwrt
./scripts/feeds update -a
./scripts/feeds install -a

Copy the text below into /openwrt as (whatevername).patch

diff --git a/package/kernel/linux/modules/netsupport.mk b/package/kernel/linux/modules/netsupport.mk
index 7d8a57d11a..4569bbc8e4 100644
--- a/package/kernel/linux/modules/netsupport.mk
+++ b/package/kernel/linux/modules/netsupport.mk
@@ -750,7 +750,7 @@ $(eval $(call KernelPackage,mppe))
 
 SCHED_MODULES = $(patsubst $(LINUX_DIR)/net/sched/%.ko,%,$(wildcard $(LINUX_DIR)/net/sched/*.ko))
 SCHED_MODULES_CORE = sch_ingress sch_fq_codel sch_hfsc sch_htb sch_tbf cls_basic cls_fw cls_route cls_flow cls_tcindex cls_u32 em_u32 act_mirred act_skbedit cls_matchall
-SCHED_MODULES_FILTER = $(SCHED_MODULES_CORE) act_connmark act_ctinfo sch_netem sch_mqprio em_ipset cls_bpf cls_flower act_bpf act_vlan
+SCHED_MODULES_FILTER = $(SCHED_MODULES_CORE) act_connmark act_ctinfo sch_netem sch_hhf sch_mqprio em_ipset cls_bpf cls_flower act_bpf act_vlan
 SCHED_MODULES_EXTRA = $(filter-out $(SCHED_MODULES_FILTER),$(SCHED_MODULES))
 SCHED_FILES = $(patsubst %,$(LINUX_DIR)/net/sched/%.ko,$(filter $(SCHED_MODULES_CORE),$(SCHED_MODULES)))
 SCHED_FILES_EXTRA = $(patsubst %,$(LINUX_DIR)/net/sched/%.ko,$(SCHED_MODULES_EXTRA))
@@ -788,6 +788,20 @@ endef
 
 $(eval $(call KernelPackage,sched-core))
 
+define KernelPackage/sched-hhf
+  SUBMENU:=$(NETWORK_SUPPORT_MENU)
+  TITLE:=Heavy Hitter Filter
+  DEPENDS:=+kmod-sched-core
+  KCONFIG:=CONFIG_NET_SCH_HHF
+  FILES:=$(LINUX_DIR)/net/sched/sch_hhf.ko
+  AUTOLOAD:=$(call AutoProbe, sch_hhf)
+endef
+
+define KernelPackage/sched-hhf/description
+ The first size-based qdisc that attempts to differentiate between small flows and heavy-hitters.
+endef
+
+$(eval $(call KernelPackage,sched-hhf))
 
 define KernelPackage/sched-flower
   SUBMENU:=$(NETWORK_SUPPORT_MENU)

Then

patch -p 1 -i (whatevername).patch
make menuconfig

I have not tested compilation for this. However, it should add sched HHF to OpenWrt's buildroot.
It will be located in Kernel Modules -> Network Support -> kmod-sched-hhf after running make menuconfig. You will have to manually invoke it/the commands for it, I don't know how to use it.

This patch will work as of OpenWrt master commit 3f5cf3872d078ca50849e08db3890718624bd0fb.

EDIT:
Some configuration options

Some configurable qdisc settings through 'tc':
- hhf_reset_timeout: period to reset counter values in the multi-stage
                     filter (default 40ms)
- hhf_admit_bytes:   threshold to classify heavy-hitters
                     (default 128KB)
- hhf_evict_timeout: threshold to evict idle heavy-hitters
                     (default 1s)
- hhf_non_hh_weight: Weighted Deficit Round Robin (WDRR) weight for
                     non-heavy-hitters (default 2)
- hh_flows_limit:    max number of heavy-hitter flow entries
                     (default 2048)

Thanks very much! I'm setting up CYGIN on my Windows 10 laptop and giving it a go. If that doesn't work I'll install Alpine Linux and go native to practice compiling and testing.

My understanding is that hhf_reset_timeout and hhf_admit_timeout correlate as in the bandwidth delay product.

128 x 1024 x 8 / .040 = 26214400 bits per second
Heavy Hitter Filter

Cygwin would be dicey, at best. I’d use a VM with Debian. A 32 MB “expandable” disk is sufficient. (It takes a little more than 16 MB, as I recall.)

Any particular version of Debian is recommended?

I run the latest, without graphics, enabling SSH and adding sudo with a user in the sudo group. With VirtualBox I can then SSH into the VM from the host and use a host-native terminal app, rather than the “video” console.

1 Like

So far I installed Ubuntu 18.06 VM since it's debian based. Will finish the setup tonight. I'm hoping to setup a lab with a bunch of servers in the future for testing stuff.

Ubuntu 18.04.3 LTS is installed replacing Windows 10 instead of virtual machine. Windows 10 does support case sensitive directories BUT each folder must be set. Which means sub directories don't inherit the changes applied to be case sensitive. I'm running 18.06.5 tickless and preemptive @ 1000hz. Compiling seems very stable without issues every time ( I tried others like voluntary and periodic...meh ). I'll try the hhf patch and some other stuff next!

I'm not sure if preemptive is for the best anymore:

That thread in general has a good discussion on preempt vs. none in networking applications.

Thanks but I've read that thread and many many others for months on end. No two groups seem to agree on what is best or not. I figure it can't hurt to try for myself right? I'm also digging to find a way to benchmark it properly. Everything does feel snappy now having switched back and forth to different partitions with the official release and my builds. voluntary and periodic tick @ 1k hz made no difference. Tickless may be the key as dtaht has mention many times. I'll be happy to perform some test and post the results if anyone is interested or send the *.bin out.

config.seed

CONFIG_TARGET_mvebu=y
CONFIG_TARGET_mvebu_cortexa9=y
CONFIG_TARGET_mvebu_cortexa9_DEVICE_linksys-wrt3200acm=y
CONFIG_DEVEL=y
CONFIG_PACKAGE_ethtool=y
CONFIG_PACKAGE_iptables-mod-conntrack-extra=y
CONFIG_PACKAGE_iptables-mod-ipopt=y
CONFIG_PACKAGE_kmod-ifb=y
CONFIG_PACKAGE_kmod-ipt-conntrack-extra=y
CONFIG_PACKAGE_kmod-ipt-ipopt=y
CONFIG_PACKAGE_kmod-sched-cake=y
CONFIG_PACKAGE_kmod-sched-core=y
CONFIG_PACKAGE_libiwinfo-lua=y
CONFIG_PACKAGE_liblua=y
CONFIG_PACKAGE_liblucihttp=y
CONFIG_PACKAGE_liblucihttp-lua=y
CONFIG_PACKAGE_libncurses=y
CONFIG_PACKAGE_libpcap=y
CONFIG_PACKAGE_libubus-lua=y
CONFIG_PACKAGE_lua=y
CONFIG_PACKAGE_luci=y
CONFIG_PACKAGE_luci-app-advanced-reboot=y
CONFIG_PACKAGE_luci-app-firewall=y
CONFIG_PACKAGE_luci-app-sqm=y
CONFIG_PACKAGE_luci-base=y
CONFIG_PACKAGE_luci-lib-ip=y
CONFIG_PACKAGE_luci-lib-jsonc=y
CONFIG_PACKAGE_luci-lib-nixio=y
CONFIG_PACKAGE_luci-mod-admin-full=y
CONFIG_PACKAGE_luci-proto-ipv6=y
CONFIG_PACKAGE_luci-proto-ppp=y
CONFIG_PACKAGE_luci-theme-bootstrap=y
CONFIG_PACKAGE_nano=y
CONFIG_PACKAGE_rpcd=y
CONFIG_PACKAGE_rpcd-mod-rrdns=y
CONFIG_PACKAGE_sqm-scripts=y
CONFIG_PACKAGE_tc=y
CONFIG_PACKAGE_tcpdump=y
CONFIG_PACKAGE_terminfo=y
CONFIG_PACKAGE_uhttpd=y
CONFIG_TARGET_OPTIONS=y

openwrt-mvebu-cortexa9-device-linksys-wrt3200acm.manifest

base-files - 194.2-r7897-9d401013fc
busybox - 1.28.4-3
dnsmasq - 2.80-1.4
dropbear - 2017.75-7.1
ethtool - 4.19-1
firewall - 2018-08-13-1c4d5bcd-1
fstools - 2019-03-28-ff1ded63-5
fwtool - 1
hostapd-common - 2018-05-21-62566bc2-8
ip6tables - 1.6.2-3
iptables - 1.6.2-3
iptables-mod-conntrack-extra - 1.6.2-3
iptables-mod-ipopt - 1.6.2-3
iw - 4.14-1
iwinfo - 2018-07-31-65b8333f-1
jshn - 2018-07-25-c83a84af-2
jsonfilter - 2018-02-04-c7e938d6-1
kernel - 4.14.151-1-4a44d0180192bcb03cdb76e2beca2846
kmod-bluetooth - 4.14.151-1
kmod-btmrvl - 4.14.151-1
kmod-cfg80211 - 4.14.151+2017-11-01-10
kmod-crypto-acompress - 4.14.151-1
kmod-crypto-aead - 4.14.151-1
kmod-crypto-cmac - 4.14.151-1
kmod-crypto-ecb - 4.14.151-1
kmod-crypto-ecdh - 4.14.151-1
kmod-crypto-hash - 4.14.151-1
kmod-crypto-kpp - 4.14.151-1
kmod-crypto-manager - 4.14.151-1
kmod-crypto-null - 4.14.151-1
kmod-crypto-pcompress - 4.14.151-1
kmod-gpio-button-hotplug - 4.14.151-2
kmod-hid - 4.14.151-1
kmod-i2c-core - 4.14.151-1
kmod-ifb - 4.14.151-1
kmod-input-core - 4.14.151-1
kmod-input-evdev - 4.14.151-1
kmod-ip6tables - 4.14.151-1
kmod-ipt-conntrack - 4.14.151-1
kmod-ipt-conntrack-extra - 4.14.151-1
kmod-ipt-core - 4.14.151-1
kmod-ipt-ipopt - 4.14.151-1
kmod-ipt-nat - 4.14.151-1
kmod-ipt-offload - 4.14.151-1
kmod-lib-crc-ccitt - 4.14.151-1
kmod-lib-crc16 - 4.14.151-1
kmod-lib-lzo - 4.14.151-1
kmod-mac80211 - 4.14.151+2017-11-01-10
kmod-mmc - 4.14.151-1
kmod-mwifiex-sdio - 4.14.151+2017-11-01-10
kmod-mwlwifi - 4.14.151+2018-11-14-81413aa9-1
kmod-nf-conntrack - 4.14.151-1
kmod-nf-conntrack6 - 4.14.151-1
kmod-nf-flow - 4.14.151-1
kmod-nf-ipt - 4.14.151-1
kmod-nf-ipt6 - 4.14.151-1
kmod-nf-nat - 4.14.151-1
kmod-nf-reject - 4.14.151-1
kmod-nf-reject6 - 4.14.151-1
kmod-nls-base - 4.14.151-1
kmod-ppp - 4.14.151-1
kmod-pppoe - 4.14.151-1
kmod-pppox - 4.14.151-1
kmod-regmap - 4.14.151-1
kmod-sched-cake - 4.14.151+2019-01-08-331ac70c-1
kmod-sched-core - 4.14.151-1
kmod-slhc - 4.14.151-1
kmod-usb-core - 4.14.151-1
libblobmsg-json - 2018-07-25-c83a84af-2
libc - 1.1.19-2
libgcc - 7.3.0-2
libip4tc - 1.6.2-3
libip6tc - 1.6.2-3
libiwinfo - 2018-07-31-65b8333f-1
libiwinfo-lua - 2018-07-31-65b8333f-1
libjson-c - 0.12.1-2
libjson-script - 2018-07-25-c83a84af-2
liblua - 5.1.5-1
liblucihttp - 2019-07-05-a34a17d5-1
liblucihttp-lua - 2019-07-05-a34a17d5-1
libncurses - 6.1-1
libnl-tiny - 0.1-5
libpcap - 1.9.1-1
libpthread - 1.1.19-2
libubox - 2018-07-25-c83a84af-2
libubus - 2018-10-06-221ce7e7-1
libubus-lua - 2018-10-06-221ce7e7-1
libuci - 2019-05-17-f199b961-1
libuclient - 2018-11-24-3ba74ebc-1
libxtables - 1.6.2-3
logd - 2018-02-14-128bc35f-2
lua - 5.1.5-1
luci - git-19.309.48729-bc17ef6-1
luci-app-advanced-reboot - git-19.309.48729-bc17ef6-42
luci-app-firewall - git-19.309.48729-bc17ef6-1
luci-app-sqm - 1.2.4-1
luci-base - git-19.309.48729-bc17ef6-1
luci-lib-ip - git-19.309.48729-bc17ef6-1
luci-lib-jsonc - git-19.309.48729-bc17ef6-1
luci-lib-nixio - git-19.309.48729-bc17ef6-1
luci-mod-admin-full - git-19.309.48729-bc17ef6-1
luci-proto-ipv6 - git-19.309.48729-bc17ef6-1
luci-proto-ppp - git-19.309.48729-bc17ef6-1
luci-theme-bootstrap - git-19.309.48729-bc17ef6-1
mtd - 23
mwifiex-sdio-firmware - 2017-09-06-a61ac5cf-1
mwlwifi-firmware-88w8964 - 2018-11-14-81413aa9-1
nano - 4.5-1
netifd - 2019-01-31-a2aba5c7-2.1
odhcp6c - 2018-07-14-67ae6a71-15
odhcpd-ipv6only - 1.15-3
openwrt-keyring - 2018-05-18-103a32e9-1
opkg - 2019-06-14-dcbc142e-1
ppp - 2.4.7-12
ppp-mod-pppoe - 2.4.7-12
procd - 2018-03-28-dfb68f85-1
rpcd - 2018-11-28-3aa81d0d-2
rpcd-mod-rrdns - 20170710
sqm-scripts - 1.2.4-1
swconfig - 11
tc - 4.16.0-8
tcpdump - 4.9.3-1
terminfo - 6.1-1
ubi-utils - 2.0.2-1
uboot-envtools - 2018.03-1
ubox - 2018-02-14-128bc35f-2
ubus - 2018-10-06-221ce7e7-1
ubusd - 2018-10-06-221ce7e7-1
uci - 2019-05-17-f199b961-1
uclient-fetch - 2018-11-24-3ba74ebc-1
uhttpd - 2018-11-28-cdfc902a-3
usign - 2019-08-06-5a52b379-1
wireless-regdb - 2017-10-20-4343d359
wpad-mini - 2018-05-21-62566bc2-8

Preemptive kernel is really important when you're using the real-time scheduler. In fact, it's use case is when the administrator has fine-tuned the scheduler policies.
It makes no sense to interrupt a network daemon with collectd but it does make sense to interrupt collectd when hostapd wants to handshake a new wireless connection.
This is why the RR and FIFO queue exists, and that's why some daemons (i.e. rpcd, procd, ubus, irqbalance, hostapd, netifid and haveged) want to be real-time.
For example, OpenSSL has depleted the entropy pool, and in that moment haveged must have to become ready to execute and preempt the current process (say OpenVPN). It will lead to more overhead but OpenVPN will stay less time blocked waiting entropy in the moment that haveged ends. And if it's time hasn't end yet, it will run immediately. The opposite would be to block OpenVPN and wait for haveged to be scheduled and then wait again for OpenVPN to become scheduled again.

The preemptive kernel by itself won't improve performance when not tuned correctly to use the real-time scheduler.

You'd have to give us the contents of target/linux/mvebu/config-4.19.

I didn't even know the Armada 38x series was capable of running tickless.

Hell, mwlwifi on your 3200ACM is configured to run:
#define SYSADPT_TIMER_WAKEUP_TIME 10 /* ms */

Or 100Hz tick.

contents of target/linux/mvebu/config-4.14

CONFIG_AHCI_MVEBU=y
CONFIG_ALIGNMENT_TRAP=y
CONFIG_ARCH_CLOCKSOURCE_DATA=y
CONFIG_ARCH_HAS_DEBUG_VIRTUAL=y
CONFIG_ARCH_HAS_ELF_RANDOMIZE=y
CONFIG_ARCH_HAS_GCOV_PROFILE_ALL=y
CONFIG_ARCH_HAS_SET_MEMORY=y
CONFIG_ARCH_HAS_SG_CHAIN=y
CONFIG_ARCH_HAS_STRICT_KERNEL_RWX=y
CONFIG_ARCH_HAS_STRICT_MODULE_RWX=y
CONFIG_ARCH_HAS_TICK_BROADCAST=y
CONFIG_ARCH_HAVE_CUSTOM_GPIO_H=y
CONFIG_ARCH_HIBERNATION_POSSIBLE=y
CONFIG_ARCH_MIGHT_HAVE_PC_PARPORT=y
CONFIG_ARCH_MULTIPLATFORM=y
# CONFIG_ARCH_MULTI_CPU_AUTO is not set
CONFIG_ARCH_MULTI_V6_V7=y
CONFIG_ARCH_MULTI_V7=y
CONFIG_ARCH_MVEBU=y
CONFIG_ARCH_NR_GPIO=0
CONFIG_ARCH_OPTIONAL_KERNEL_RWX=y
CONFIG_ARCH_OPTIONAL_KERNEL_RWX_DEFAULT=y
# CONFIG_ARCH_SELECT_MEMORY_MODEL is not set
# CONFIG_ARCH_SPARSEMEM_DEFAULT is not set
CONFIG_ARCH_SUPPORTS_ATOMIC_RMW=y
CONFIG_ARCH_SUPPORTS_BIG_ENDIAN=y
CONFIG_ARCH_SUPPORTS_UPROBES=y
CONFIG_ARCH_SUSPEND_POSSIBLE=y
CONFIG_ARCH_USE_BUILTIN_BSWAP=y
CONFIG_ARCH_USE_CMPXCHG_LOCKREF=y
CONFIG_ARCH_WANT_GENERAL_HUGETLB=y
CONFIG_ARCH_WANT_IPC_PARSE_VERSION=y
CONFIG_ARM=y
CONFIG_ARMADA_370_CLK=y
CONFIG_ARMADA_370_XP_IRQ=y
CONFIG_ARMADA_370_XP_TIMER=y
CONFIG_ARMADA_38X_CLK=y
CONFIG_ARMADA_THERMAL=y
CONFIG_ARMADA_XP_CLK=y
CONFIG_ARM_APPENDED_DTB=y
# CONFIG_ARM_ARMADA_37XX_CPUFREQ is not set
CONFIG_ARM_ATAG_DTB_COMPAT=y
# CONFIG_ARM_ATAG_DTB_COMPAT_CMDLINE_EXTEND is not set
# CONFIG_ARM_ATAG_DTB_COMPAT_CMDLINE_FROM_BOOTLOADER is not set
CONFIG_ARM_ATAG_DTB_COMPAT_CMDLINE_MANGLE=y
CONFIG_ARM_CPU_SUSPEND=y
CONFIG_ARM_CRYPTO=y
CONFIG_ARM_ERRATA_720789=y
CONFIG_ARM_ERRATA_764369=y
CONFIG_ARM_GIC=y
CONFIG_ARM_GLOBAL_TIMER=y
CONFIG_ARM_HAS_SG_CHAIN=y
CONFIG_ARM_HEAVY_MB=y
CONFIG_ARM_L1_CACHE_SHIFT=6
CONFIG_ARM_L1_CACHE_SHIFT_6=y
# CONFIG_ARM_LPAE is not set
CONFIG_ARM_MVEBU_V7_CPUIDLE=y
CONFIG_ARM_PATCH_IDIV=y
CONFIG_ARM_PATCH_PHYS_VIRT=y
CONFIG_ARM_PMU=y
CONFIG_ARM_THUMB=y
# CONFIG_ARM_THUMBEE is not set
CONFIG_ARM_UNWIND=y
CONFIG_ARM_VIRT_EXT=y
CONFIG_ASYNC_TX_ENABLE_CHANNEL_SWITCH=y
CONFIG_ATA=y
CONFIG_ATAGS=y
CONFIG_AUTO_ZRELADDR=y
CONFIG_BLK_DEV_LOOP=y
CONFIG_BLK_DEV_SD=y
CONFIG_BLK_MQ_PCI=y
CONFIG_BLK_SCSI_REQUEST=y
CONFIG_BOUNCE=y
CONFIG_BUILD_BIN2C=y
# CONFIG_CACHE_FEROCEON_L2 is not set
CONFIG_CACHE_L2X0=y
CONFIG_CLKDEV_LOOKUP=y
CONFIG_CLKSRC_ARM_GLOBAL_TIMER_SCHED_CLOCK=y
CONFIG_CLKSRC_MMIO=y
CONFIG_CLONE_BACKWARDS=y
CONFIG_COMMON_CLK=y
CONFIG_CONTEXT_TRACKING=y
# CONFIG_CONTEXT_TRACKING_FORCE is not set
CONFIG_CPUFREQ_DT=y
CONFIG_CPUFREQ_DT_PLATDEV=y
CONFIG_CPU_32v6K=y
CONFIG_CPU_32v7=y
CONFIG_CPU_ABRT_EV7=y
# CONFIG_CPU_BIG_ENDIAN is not set
# CONFIG_CPU_BPREDICT_DISABLE is not set
CONFIG_CPU_CACHE_V7=y
CONFIG_CPU_CACHE_VIPT=y
CONFIG_CPU_COPY_V6=y
CONFIG_CPU_CP15=y
CONFIG_CPU_CP15_MMU=y
CONFIG_CPU_FREQ=y
CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE=y
# CONFIG_CPU_FREQ_GOV_CONSERVATIVE is not set
# CONFIG_CPU_FREQ_GOV_ONDEMAND is not set
CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
# CONFIG_CPU_FREQ_GOV_POWERSAVE is not set
# CONFIG_CPU_FREQ_GOV_USERSPACE is not set
CONFIG_CPU_FREQ_STAT=y
CONFIG_CPU_HAS_ASID=y
# CONFIG_CPU_HOTPLUG_STATE_CONTROL is not set
# CONFIG_CPU_ICACHE_DISABLE is not set
CONFIG_CPU_IDLE=y
CONFIG_CPU_IDLE_GOV_LADDER=y
CONFIG_CPU_IDLE_GOV_MENU=y
CONFIG_CPU_PABRT_V7=y
CONFIG_CPU_PJ4B=y
CONFIG_CPU_PM=y
CONFIG_CPU_RMAP=y
CONFIG_CPU_SPECTRE=y
CONFIG_CPU_THERMAL=y
CONFIG_CPU_THUMB_CAPABLE=y
CONFIG_CPU_TLB_V7=y
CONFIG_CPU_V7=y
CONFIG_CRC16=y
CONFIG_CRYPTO_ACOMP2=y
CONFIG_CRYPTO_AEAD=y
CONFIG_CRYPTO_AEAD2=y
CONFIG_CRYPTO_AES_ARM=y
CONFIG_CRYPTO_AES_ARM_BS=y
# CONFIG_CRYPTO_AES_ARM_CE is not set
# CONFIG_CRYPTO_CHACHA20_NEON is not set
CONFIG_CRYPTO_CRC32=y
CONFIG_CRYPTO_CRC32C=y
# CONFIG_CRYPTO_CRC32_ARM_CE is not set
CONFIG_CRYPTO_CRYPTD=y
CONFIG_CRYPTO_DEFLATE=y
CONFIG_CRYPTO_DES=y
CONFIG_CRYPTO_DEV_MARVELL_CESA=y
# CONFIG_CRYPTO_GHASH_ARM_CE is not set
CONFIG_CRYPTO_HASH=y
CONFIG_CRYPTO_HASH2=y
CONFIG_CRYPTO_HW=y
CONFIG_CRYPTO_LZO=y
CONFIG_CRYPTO_MANAGER=y
CONFIG_CRYPTO_MANAGER2=y
CONFIG_CRYPTO_NULL2=y
CONFIG_CRYPTO_RNG2=y
CONFIG_CRYPTO_SHA1=y
CONFIG_CRYPTO_SHA1_ARM=y
# CONFIG_CRYPTO_SHA1_ARM_CE is not set
CONFIG_CRYPTO_SHA1_ARM_NEON=y
CONFIG_CRYPTO_SHA256_ARM=y
# CONFIG_CRYPTO_SHA2_ARM_CE is not set
CONFIG_CRYPTO_SHA512_ARM=y
CONFIG_CRYPTO_SIMD=y
CONFIG_CRYPTO_WORKQUEUE=y
CONFIG_DCACHE_WORD_ACCESS=y
CONFIG_DEBUG_ALIGN_RODATA=y
CONFIG_DEBUG_INFO=y
CONFIG_DEBUG_LL=y
CONFIG_DEBUG_LL_INCLUDE="debug/8250.S"
CONFIG_DEBUG_MVEBU_UART0=y
# CONFIG_DEBUG_MVEBU_UART0_ALTERNATE is not set
# CONFIG_DEBUG_MVEBU_UART1_ALTERNATE is not set
CONFIG_DEBUG_UART_8250=y
# CONFIG_DEBUG_UART_8250_FLOW_CONTROL is not set
CONFIG_DEBUG_UART_8250_SHIFT=2
# CONFIG_DEBUG_UART_8250_WORD is not set
CONFIG_DEBUG_UART_PHYS=0xd0012000
CONFIG_DEBUG_UART_VIRT=0xfec12000
CONFIG_DEBUG_UNCOMPRESS=y
CONFIG_DEBUG_USER=y
CONFIG_DMADEVICES=y
CONFIG_DMA_ENGINE=y
CONFIG_DMA_ENGINE_RAID=y
CONFIG_DMA_OF=y
CONFIG_DTC=y
CONFIG_EARLY_PRINTK=y
CONFIG_EDAC_ATOMIC_SCRUB=y
CONFIG_EDAC_SUPPORT=y
CONFIG_EXT4_FS=y
CONFIG_EXTCON=y
# CONFIG_F2FS_CHECK_FS is not set
CONFIG_F2FS_FS=y
# CONFIG_F2FS_FS_SECURITY is not set
CONFIG_F2FS_FS_XATTR=y
CONFIG_F2FS_STAT_FS=y
CONFIG_FIXED_PHY=y
CONFIG_FIX_EARLYCON_MEM=y
CONFIG_FS_MBCACHE=y
CONFIG_GENERIC_ALLOCATOR=y
CONFIG_GENERIC_ARCH_TOPOLOGY=y
CONFIG_GENERIC_BUG=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
CONFIG_GENERIC_CPU_AUTOPROBE=y
CONFIG_GENERIC_EARLY_IOREMAP=y
CONFIG_GENERIC_IDLE_POLL_SETUP=y
CONFIG_GENERIC_IO=y
CONFIG_GENERIC_IRQ_CHIP=y
CONFIG_GENERIC_IRQ_EFFECTIVE_AFF_MASK=y
CONFIG_GENERIC_IRQ_MIGRATION=y
CONFIG_GENERIC_IRQ_SHOW=y
CONFIG_GENERIC_IRQ_SHOW_LEVEL=y
CONFIG_GENERIC_MSI_IRQ=y
CONFIG_GENERIC_MSI_IRQ_DOMAIN=y
CONFIG_GENERIC_PCI_IOMAP=y
CONFIG_GENERIC_PHY=y
CONFIG_GENERIC_SCHED_CLOCK=y
CONFIG_GENERIC_SMP_IDLE_THREAD=y
CONFIG_GENERIC_STRNCPY_FROM_USER=y
CONFIG_GENERIC_STRNLEN_USER=y
CONFIG_GLOB=y
CONFIG_GPIOLIB=y
CONFIG_GPIOLIB_IRQCHIP=y
CONFIG_GPIO_GENERIC=y
CONFIG_GPIO_GENERIC_PLATFORM=y
CONFIG_GPIO_MVEBU=y
CONFIG_GPIO_PCA953X=y
CONFIG_GPIO_PCA953X_IRQ=y
CONFIG_GPIO_SYSFS=y
# CONFIG_GRO_CELLS is not set
CONFIG_HANDLE_DOMAIN_IRQ=y
CONFIG_HARDEN_BRANCH_PREDICTOR=y
CONFIG_HARDIRQS_SW_RESEND=y
CONFIG_HAS_DMA=y
CONFIG_HAS_IOMEM=y
CONFIG_HAS_IOPORT_MAP=y
# CONFIG_HAVE_64BIT_ALIGNED_ACCESS is not set
CONFIG_HAVE_ARCH_AUDITSYSCALL=y
CONFIG_HAVE_ARCH_BITREVERSE=y
CONFIG_HAVE_ARCH_JUMP_LABEL=y
CONFIG_HAVE_ARCH_KGDB=y
CONFIG_HAVE_ARCH_PFN_VALID=y
CONFIG_HAVE_ARCH_SECCOMP_FILTER=y
CONFIG_HAVE_ARCH_TRACEHOOK=y
CONFIG_HAVE_ARM_SCU=y
CONFIG_HAVE_ARM_SMCCC=y
CONFIG_HAVE_ARM_TWD=y
# CONFIG_HAVE_BOOTMEM_INFO_NODE is not set
CONFIG_HAVE_CC_STACKPROTECTOR=y
CONFIG_HAVE_CLK=y
CONFIG_HAVE_CLK_PREPARE=y
CONFIG_HAVE_CONTEXT_TRACKING=y
CONFIG_HAVE_C_RECORDMCOUNT=y
CONFIG_HAVE_DEBUG_KMEMLEAK=y
CONFIG_HAVE_DMA_API_DEBUG=y
CONFIG_HAVE_DMA_CONTIGUOUS=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_HAVE_DYNAMIC_FTRACE_WITH_REGS=y
CONFIG_HAVE_EBPF_JIT=y
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
CONFIG_HAVE_FUNCTION_TRACER=y
CONFIG_HAVE_GENERIC_DMA_COHERENT=y
CONFIG_HAVE_HW_BREAKPOINT=y
CONFIG_HAVE_IDE=y
CONFIG_HAVE_IRQ_TIME_ACCOUNTING=y
CONFIG_HAVE_MEMBLOCK=y
CONFIG_HAVE_MOD_ARCH_SPECIFIC=y
CONFIG_HAVE_NET_DSA=y
CONFIG_HAVE_OPROFILE=y
CONFIG_HAVE_OPTPROBES=y
CONFIG_HAVE_PERF_EVENTS=y
CONFIG_HAVE_PERF_REGS=y
CONFIG_HAVE_PERF_USER_STACK_DUMP=y
CONFIG_HAVE_PROC_CPU=y
CONFIG_HAVE_REGS_AND_STACK_ACCESS_API=y
CONFIG_HAVE_SMP=y
CONFIG_HAVE_SYSCALL_TRACEPOINTS=y
CONFIG_HAVE_UID16=y
CONFIG_HAVE_VIRT_CPU_ACCOUNTING_GEN=y
CONFIG_HIGHMEM=y
# CONFIG_HIGHPTE is not set
CONFIG_HOTPLUG_CPU=y
CONFIG_HWBM=y
CONFIG_HWMON=y
CONFIG_HZ=1000
# CONFIG_HZ_100 is not set
CONFIG_HZ_1000=y
CONFIG_HZ_FIXED=0
CONFIG_I2C=y
CONFIG_I2C_BOARDINFO=y
CONFIG_I2C_CHARDEV=y
CONFIG_I2C_MV64XXX=y
# CONFIG_I2C_PXA is not set
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
CONFIG_INITRAMFS_SOURCE=""
CONFIG_IOMMU_HELPER=y
CONFIG_IRQCHIP=y
CONFIG_IRQ_DOMAIN=y
CONFIG_IRQ_DOMAIN_DEBUG=y
CONFIG_IRQ_DOMAIN_HIERARCHY=y
CONFIG_IRQ_FORCED_THREADING=y
CONFIG_IRQ_WORK=y
# CONFIG_IWMMXT is not set
CONFIG_JBD2=y
CONFIG_LEDS_GPIO=y
CONFIG_LEDS_PCA963X=y
CONFIG_LEDS_TLC591XX=y
CONFIG_LEDS_TRIGGER_DISK=y
CONFIG_LIBFDT=y
CONFIG_LOCALVERSION_AUTO=y
CONFIG_LOCK_SPIN_ON_OWNER=y
CONFIG_LZO_COMPRESS=y
CONFIG_LZO_DECOMPRESS=y
CONFIG_MACH_ARMADA_370=y
# CONFIG_MACH_ARMADA_375 is not set
CONFIG_MACH_ARMADA_38X=y
# CONFIG_MACH_ARMADA_39X is not set
CONFIG_MACH_ARMADA_XP=y
# CONFIG_MACH_DOVE is not set
CONFIG_MACH_MVEBU_ANY=y
CONFIG_MACH_MVEBU_V7=y
CONFIG_MAGIC_SYSRQ=y
CONFIG_MANGLE_BOOTARGS=y
CONFIG_MARVELL_PHY=y
CONFIG_MDIO_BUS=y
CONFIG_MDIO_DEVICE=y
CONFIG_MDIO_I2C=y
CONFIG_MEMORY=y
CONFIG_MIGHT_HAVE_CACHE_L2X0=y
CONFIG_MIGHT_HAVE_PCI=y
CONFIG_MIGRATION=y
CONFIG_MMC=y
CONFIG_MMC_BLOCK=y
CONFIG_MMC_MVSDIO=y
CONFIG_MMC_SDHCI=y
# CONFIG_MMC_SDHCI_PCI is not set
CONFIG_MMC_SDHCI_PLTFM=y
CONFIG_MMC_SDHCI_PXAV3=y
# CONFIG_MMC_TIFM_SD is not set
CONFIG_MODULES_TREE_LOOKUP=y
CONFIG_MODULES_USE_ELF_REL=y
CONFIG_MTD_CFI_STAA=y
CONFIG_MTD_M25P80=y
CONFIG_MTD_NAND=y
CONFIG_MTD_NAND_ECC=y
CONFIG_MTD_NAND_PXA3xx=y
CONFIG_MTD_SPI_NOR=y
CONFIG_MTD_SPLIT_FIRMWARE=y
CONFIG_MTD_UBI=y
CONFIG_MTD_UBI_BEB_LIMIT=20
CONFIG_MTD_UBI_BLOCK=y
# CONFIG_MTD_UBI_FASTMAP is not set
# CONFIG_MTD_UBI_GLUEBI is not set
CONFIG_MTD_UBI_WL_THRESHOLD=4096
CONFIG_MULTI_IRQ_HANDLER=y
CONFIG_MUTEX_SPIN_ON_OWNER=y
CONFIG_MVEBU_CLK_COMMON=y
CONFIG_MVEBU_CLK_COREDIV=y
CONFIG_MVEBU_CLK_CPU=y
CONFIG_MVEBU_DEVBUS=y
CONFIG_MVEBU_MBUS=y
CONFIG_MVMDIO=y
CONFIG_MVNETA=y
CONFIG_MVNETA_BM=y
CONFIG_MVNETA_BM_ENABLE=y
CONFIG_MVPP2=y
CONFIG_MVSW61XX_PHY=y
CONFIG_MV_XOR=y
CONFIG_NEED_DMA_MAP_STATE=y
CONFIG_NEON=y
CONFIG_NET_DSA=y
CONFIG_NET_DSA_MV88E6XXX=y
CONFIG_NET_DSA_MV88E6XXX_GLOBAL2=y
CONFIG_NET_DSA_TAG_DSA=y
CONFIG_NET_DSA_TAG_EDSA=y
CONFIG_NET_FLOW_LIMIT=y
CONFIG_NET_SWITCHDEV=y
CONFIG_NLS=y
CONFIG_NOP_USB_XCEIV=y
CONFIG_NO_BOOTMEM=y
CONFIG_NO_HZ_COMMON=y
CONFIG_NO_HZ_FULL=y
CONFIG_NO_HZ_FULL_ALL=y
CONFIG_NR_CPUS=4
CONFIG_OF=y
CONFIG_OF_ADDRESS=y
CONFIG_OF_ADDRESS_PCI=y
CONFIG_OF_EARLY_FLATTREE=y
CONFIG_OF_FLATTREE=y
CONFIG_OF_GPIO=y
CONFIG_OF_IRQ=y
CONFIG_OF_MDIO=y
CONFIG_OF_NET=y
CONFIG_OF_PCI=y
CONFIG_OF_PCI_IRQ=y
CONFIG_OF_RESERVED_MEM=y
CONFIG_OLD_SIGACTION=y
CONFIG_OLD_SIGSUSPEND3=y
CONFIG_ORION_WATCHDOG=y
CONFIG_OUTER_CACHE=y
CONFIG_OUTER_CACHE_SYNC=y
CONFIG_PADATA=y
CONFIG_PAGE_OFFSET=0xC0000000
CONFIG_PCI=y
CONFIG_PCI_DOMAINS=y
CONFIG_PCI_DOMAINS_GENERIC=y
CONFIG_PCI_MSI=y
CONFIG_PCI_MSI_IRQ_DOMAIN=y
CONFIG_PCI_MVEBU=y
CONFIG_PERF_EVENTS=y
CONFIG_PERF_USE_VMALLOC=y
CONFIG_PGTABLE_LEVELS=2
CONFIG_PHYLIB=y
CONFIG_PHYLINK=y
# CONFIG_PHY_MVEBU_CP110_COMPHY is not set
CONFIG_PINCTRL=y
CONFIG_PINCTRL_ARMADA_370=y
CONFIG_PINCTRL_ARMADA_38X=y
CONFIG_PINCTRL_ARMADA_XP=y
CONFIG_PINCTRL_MVEBU=y
# CONFIG_PINCTRL_SINGLE is not set
CONFIG_PJ4B_ERRATA_4742=y
# CONFIG_PL310_ERRATA_588369 is not set
# CONFIG_PL310_ERRATA_727915 is not set
CONFIG_PL310_ERRATA_753970=y
# CONFIG_PL310_ERRATA_769419 is not set
CONFIG_PLAT_ORION=y
CONFIG_PM_OPP=y
CONFIG_PREEMPT=y
CONFIG_PREEMPT_COUNT=y
# CONFIG_PREEMPT_NONE is not set
CONFIG_PREEMPT_RCU=y
CONFIG_PRINTK_SAFE_LOG_BUF_SHIFT=11
CONFIG_PWM=y
CONFIG_PWM_SYSFS=y
CONFIG_RATIONAL=y
# CONFIG_RCU_BOOST is not set
CONFIG_RCU_NEED_SEGCBLIST=y
CONFIG_RCU_NOCB_CPU=y
CONFIG_RCU_STALL_COMMON=y
CONFIG_REGMAP=y
CONFIG_REGMAP_I2C=y
CONFIG_REGMAP_MMIO=y
CONFIG_REGMAP_SPI=y
CONFIG_REGULATOR=y
CONFIG_REGULATOR_FIXED_VOLTAGE=y
CONFIG_RFS_ACCEL=y
CONFIG_RPS=y
CONFIG_RTC_CLASS=y
CONFIG_RTC_DRV_ARMADA38X=y
CONFIG_RTC_DRV_MV=y
CONFIG_RTC_I2C_AND_SPI=y
CONFIG_RTC_MC146818_LIB=y
CONFIG_RWSEM_SPIN_ON_OWNER=y
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
CONFIG_SATA_AHCI_PLATFORM=y
CONFIG_SATA_MV=y
CONFIG_SATA_PMP=y
# CONFIG_SCHED_INFO is not set
CONFIG_SCHED_MC=y
CONFIG_SCSI=y
CONFIG_SENSORS_PWM_FAN=y
CONFIG_SENSORS_TMP421=y
CONFIG_SERIAL_8250_DW=y
CONFIG_SERIAL_8250_FSL=y
CONFIG_SERIAL_MVEBU_CONSOLE=y
CONFIG_SERIAL_MVEBU_UART=y
CONFIG_SFP=y
CONFIG_SG_POOL=y
CONFIG_SMP=y
CONFIG_SMP_ON_UP=y
CONFIG_SOC_BUS=y
CONFIG_SPARSE_IRQ=y
CONFIG_SPI=y
# CONFIG_SPI_ARMADA_3700 is not set
CONFIG_SPI_MASTER=y
CONFIG_SPI_ORION=y
CONFIG_SRAM=y
CONFIG_SRAM_EXEC=y
CONFIG_SRCU=y
CONFIG_SWCONFIG=y
CONFIG_SWIOTLB=y
CONFIG_SWPHY=y
CONFIG_SWP_EMULATE=y
CONFIG_SYS_SUPPORTS_APM_EMULATION=y
CONFIG_TASKS_RCU=y
CONFIG_THERMAL=y
CONFIG_THERMAL_DEFAULT_GOV_STEP_WISE=y
CONFIG_THERMAL_EMERGENCY_POWEROFF_DELAY_MS=0
CONFIG_THERMAL_GOV_STEP_WISE=y
CONFIG_THERMAL_HWMON=y
CONFIG_THERMAL_OF=y
# CONFIG_THUMB2_KERNEL is not set
CONFIG_TIMER_OF=y
CONFIG_TIMER_PROBE=y
CONFIG_TREE_SRCU=y
CONFIG_UBIFS_FS=y
# CONFIG_UBIFS_FS_ADVANCED_COMPR is not set
CONFIG_UBIFS_FS_LZO=y
CONFIG_UBIFS_FS_ZLIB=y
CONFIG_UNCOMPRESS_INCLUDE="debug/uncompress.h"
CONFIG_UNINLINE_SPIN_UNLOCK=y
CONFIG_USB=y
CONFIG_USB_COMMON=y
CONFIG_USB_EHCI_HCD=y
CONFIG_USB_EHCI_HCD_ORION=y
CONFIG_USB_EHCI_HCD_PLATFORM=y
CONFIG_USB_LEDS_TRIGGER_USBPORT=y
CONFIG_USB_PHY=y
CONFIG_USB_STORAGE=y
CONFIG_USB_SUPPORT=y
CONFIG_USB_XHCI_HCD=y
CONFIG_USB_XHCI_MVEBU=y
CONFIG_USB_XHCI_PLATFORM=y
CONFIG_USE_OF=y
CONFIG_VECTORS_BASE=0xffff0000
CONFIG_VFP=y
CONFIG_VFPv3=y
CONFIG_VIRT_CPU_ACCOUNTING=y
CONFIG_VIRT_CPU_ACCOUNTING_GEN=y
CONFIG_WATCHDOG_CORE=y
CONFIG_XPS=y
CONFIG_XZ_DEC_ARM=y
CONFIG_XZ_DEC_BCJ=y
CONFIG_ZBOOT_ROM_BSS=0x0
CONFIG_ZBOOT_ROM_TEXT=0x0
CONFIG_ZLIB_DEFLATE=y
CONFIG_ZLIB_INFLATE=y

Yes it's full dyn tickless

So then how do we tune the preemptive kernel? How do we tune scheduler policies on OpenWrt? I've been looking at this: https://doc.opensuse.org/documentation/leap/archive/42.1/tuning/html/book.sle.tuning/cha.tuning.taskscheduler.html but it seems like OpenWrt isn't configured to be this tuneable in its current state.

In my build/tests, kernel preempt solely improved LuCI interactivity. It made things worse on my end in terms of SQM and latency. The stdev from mean ping times became more variable.

Granted, it was untuned. Daemons/applications weren't put into CGROUPS/etc.

I know what you're saying makes sense for other OSs (e.g. Debian, Arch, or Gentoo, Clear). But until:

  1. You're able to provide data from an OpenWrt device and;
  2. It shows a definite improvement across multiple archs that support PREEMPT (i.e. additional tests on MIPS, ARM, IPQ, etc.)

It's still just a theory for OpenWrt devices. You've provided us with a hypothesis:

So please run some experiments by tuning it and crunch/provide the numbers :slight_smile:

The two main benchmarks will be (1) latency and (2) throughput. (3) can be stability.

I may even help you out on this by providing numbers from an mvebu based 1900ACS (provided I have the time in the busy holidays).

The diff from mine and yours:

@@ -57,6 +57,7 @@
 CONFIG_ARM_MVEBU_V7_CPUIDLE=y
 CONFIG_ARM_PATCH_IDIV=y
 CONFIG_ARM_PATCH_PHYS_VIRT=y
+CONFIG_ARM_PMU=y
 CONFIG_ARM_THUMB=y
 # CONFIG_ARM_THUMBEE is not set
 CONFIG_ARM_UNWIND=y
@@ -70,6 +71,7 @@
 CONFIG_BLK_MQ_PCI=y
 CONFIG_BLK_SCSI_REQUEST=y
 CONFIG_BOUNCE=y
+CONFIG_BUILD_BIN2C=y
 # CONFIG_CACHE_FEROCEON_L2 is not set
 CONFIG_CACHE_L2X0=y
 CONFIG_CLKDEV_LOOKUP=y
@@ -77,6 +79,8 @@
 CONFIG_CLKSRC_MMIO=y
 CONFIG_CLONE_BACKWARDS=y
 CONFIG_COMMON_CLK=y
+CONFIG_CONTEXT_TRACKING=y
+# CONFIG_CONTEXT_TRACKING_FORCE is not set
 CONFIG_CPUFREQ_DT=y
 CONFIG_CPUFREQ_DT_PLATDEV=y
 CONFIG_CPU_32v6K=y
@@ -102,6 +106,7 @@
 # CONFIG_CPU_ICACHE_DISABLE is not set
 CONFIG_CPU_IDLE=y
 CONFIG_CPU_IDLE_GOV_LADDER=y
+CONFIG_CPU_IDLE_GOV_MENU=y
 CONFIG_CPU_PABRT_V7=y
 CONFIG_CPU_PJ4B=y
 CONFIG_CPU_PM=y
@@ -244,6 +249,7 @@
 CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
 CONFIG_HAVE_FUNCTION_TRACER=y
 CONFIG_HAVE_GENERIC_DMA_COHERENT=y
+CONFIG_HAVE_HW_BREAKPOINT=y
 CONFIG_HAVE_IDE=y
 CONFIG_HAVE_IRQ_TIME_ACCOUNTING=y
 CONFIG_HAVE_MEMBLOCK=y
@@ -265,14 +271,17 @@
 CONFIG_HOTPLUG_CPU=y
 CONFIG_HWBM=y
 CONFIG_HWMON=y
-CONFIG_HW_RANDOM=y
+CONFIG_HZ=1000
+# CONFIG_HZ_100 is not set
+CONFIG_HZ_1000=y
 CONFIG_HZ_FIXED=0
-CONFIG_HZ_PERIODIC=y
 CONFIG_I2C=y
 CONFIG_I2C_BOARDINFO=y
 CONFIG_I2C_CHARDEV=y
 CONFIG_I2C_MV64XXX=y
 # CONFIG_I2C_PXA is not set
+CONFIG_IKCONFIG=y
+CONFIG_IKCONFIG_PROC=y
 CONFIG_INITRAMFS_SOURCE=""
 CONFIG_IOMMU_HELPER=y
 CONFIG_IRQCHIP=y
@@ -288,6 +297,7 @@
 CONFIG_LEDS_TLC591XX=y
 CONFIG_LEDS_TRIGGER_DISK=y
 CONFIG_LIBFDT=y
+CONFIG_LOCALVERSION_AUTO=y
 CONFIG_LOCK_SPIN_ON_OWNER=y
 CONFIG_LZO_COMPRESS=y
 CONFIG_LZO_DECOMPRESS=y
@@ -317,6 +327,7 @@
 CONFIG_MMC_SDHCI_PLTFM=y
 CONFIG_MMC_SDHCI_PXAV3=y
 # CONFIG_MMC_TIFM_SD is not set
+CONFIG_MODULES_TREE_LOOKUP=y
 CONFIG_MODULES_USE_ELF_REL=y
 CONFIG_MTD_CFI_STAA=y
 CONFIG_MTD_M25P80=y
@@ -357,7 +368,10 @@
 CONFIG_NLS=y
 CONFIG_NOP_USB_XCEIV=y
 CONFIG_NO_BOOTMEM=y
-CONFIG_NR_CPUS=2
+CONFIG_NO_HZ_COMMON=y
+CONFIG_NO_HZ_FULL=y
+CONFIG_NO_HZ_FULL_ALL=y
+CONFIG_NR_CPUS=4
 CONFIG_OF=y
 CONFIG_OF_ADDRESS=y
 CONFIG_OF_ADDRESS_PCI=y
@@ -383,6 +397,7 @@
 CONFIG_PCI_MSI=y
 CONFIG_PCI_MSI_IRQ_DOMAIN=y
 CONFIG_PCI_MVEBU=y
+CONFIG_PERF_EVENTS=y
 CONFIG_PERF_USE_VMALLOC=y
 CONFIG_PGTABLE_LEVELS=2
 CONFIG_PHYLIB=y
@@ -401,11 +416,17 @@
 # CONFIG_PL310_ERRATA_769419 is not set
 CONFIG_PLAT_ORION=y
 CONFIG_PM_OPP=y
+CONFIG_PREEMPT=y
+CONFIG_PREEMPT_COUNT=y
+# CONFIG_PREEMPT_NONE is not set
+CONFIG_PREEMPT_RCU=y
 CONFIG_PRINTK_SAFE_LOG_BUF_SHIFT=11
 CONFIG_PWM=y
 CONFIG_PWM_SYSFS=y
 CONFIG_RATIONAL=y
+# CONFIG_RCU_BOOST is not set
 CONFIG_RCU_NEED_SEGCBLIST=y
+CONFIG_RCU_NOCB_CPU=y
 CONFIG_RCU_STALL_COMMON=y
 CONFIG_REGMAP=y
 CONFIG_REGMAP_I2C=y
@@ -426,6 +447,7 @@
 CONFIG_SATA_MV=y
 CONFIG_SATA_PMP=y
 # CONFIG_SCHED_INFO is not set
+CONFIG_SCHED_MC=y
 CONFIG_SCSI=y
 CONFIG_SENSORS_PWM_FAN=y
 CONFIG_SENSORS_TMP421=y
@@ -451,6 +473,7 @@
 CONFIG_SWPHY=y
 CONFIG_SWP_EMULATE=y
 CONFIG_SYS_SUPPORTS_APM_EMULATION=y
+CONFIG_TASKS_RCU=y
 CONFIG_THERMAL=y
 CONFIG_THERMAL_DEFAULT_GOV_STEP_WISE=y
 CONFIG_THERMAL_EMERGENCY_POWEROFF_DELAY_MS=0
@@ -458,16 +481,15 @@
 CONFIG_THERMAL_HWMON=y
 CONFIG_THERMAL_OF=y
 # CONFIG_THUMB2_KERNEL is not set
-CONFIG_TICK_CPU_ACCOUNTING=y
 CONFIG_TIMER_OF=y
 CONFIG_TIMER_PROBE=y
-CONFIG_TREE_RCU=y
 CONFIG_TREE_SRCU=y
 CONFIG_UBIFS_FS=y
 # CONFIG_UBIFS_FS_ADVANCED_COMPR is not set
 CONFIG_UBIFS_FS_LZO=y
 CONFIG_UBIFS_FS_ZLIB=y
 CONFIG_UNCOMPRESS_INCLUDE="debug/uncompress.h"
+CONFIG_UNINLINE_SPIN_UNLOCK=y
 CONFIG_USB=y
 CONFIG_USB_COMMON=y
 CONFIG_USB_EHCI_HCD=y
@@ -484,6 +506,8 @@
 CONFIG_VECTORS_BASE=0xffff0000
 CONFIG_VFP=y
 CONFIG_VFPv3=y
+CONFIG_VIRT_CPU_ACCOUNTING=y
+CONFIG_VIRT_CPU_ACCOUNTING_GEN=y
 CONFIG_WATCHDOG_CORE=y
 CONFIG_XPS=y
 CONFIG_XZ_DEC_ARM=y

Look. I won't give you raw performance test with numbers. I'm not here to trying to convince anyone because I've done this before in many environments from servers to Android devices. It's a pure matter of the fact that fine tuning is needed. Not a single solution is feasible for all use cases, but they all have something in common: fine tuning.

So, perform your tests. Do some investigation of what daemons and kernel threads need to be tuned and test them yourself.

But in order to do so, I'll give you the "magic tool". This patch allows to build the chrt tool from util-linux which in turn will allow you to fine tune the Linux scheduler. Hint: critical daeoms want to be real-time (the policy, either RR or FIFO, and the relative priority in their queue matters): sugov, procd, kswapdX, ubusd, heveged, irqbalance, rpcd, netifid, hostapd/wpa_supplicant, unbound/dnsmasq... Also you want to move the priority of the preemptive RCU to the highest possible (irq/XX-keys is 51, so you want rcuc/X/rcub/X to be priority 50) and to lower the boost time of the grace period to a prime member (fine tuning needed).
Some daemons want to run in the batch queue: minidlna, samba, dropbear/sshd, etc...
Others want to be in the idle queue: knetatop, avahi-daeom, dbus-daeom, nmbd/winbind, collectd, tcpdump, etc...

The patch needed to build and use chrt: https://github.com/NoTengoBattery/openwrt/commit/7363cc0f3ec40cb132f31b7ebef3551b3c8f174f
The manpage of chrt: http://man7.org/linux/man-pages/man1/chrt.1.html
The man page of the Linux scheduler: http://man7.org/linux/man-pages/man7/sched.7.html
The manpage of the OTHER/BATCH relative priority: http://man7.org/linux/man-pages/man2/nice.2.html

In the RR/FIFO queues you need to tune the absolute priority. In OTHER/BATCH you must tune the relative priority (niceness) and the IDLE queue is not tunable.


You need a custom boot parameter to move the RCU priority and a custom configuration to reduce the time of the boost to the blocked readers: https://github.com/NoTengoBattery/openwrt/commit/154598b4585c72efa34169f30105b1b10ee49572 and https://lwn.net/Articles/220677/


Adding more to the discussion: network packets are sent/received within the SOFTIRQ. I do not recommend it, but you can boost the ksoftirqd/X threads below the RCU thread in the FIFO queue to see if that helps: https://www.oreilly.com/library/view/understanding-the-linux/0596002130/ch18s03.html