I've trying investigate or find info what exactly doing Packet Steering ,i've installed htop,
when i chekd'ON Packet Steering, gigabit connection uses 2 cores 20 and 30%, ~50% summary
when i chekd'OFF Packet Steering, gigabit connection uses 2 cores 25 and 12%, ~40% summary
Is this function share load or each core have full load?
Receive/Transmit Packet Steering is software based. It's functions (if the nic has more queues than the system's cpus e.g. ethtool -x eth0) is to help spread the network load across all processors. YMMV but my experience is that manually pinning an interrupt to a core & rps is very beneficial.
find /sys/class/net/*/queues/[rt]x-[01234567]/[rx]ps_cpus -exec sh -c '[ -w {} ] && echo f > {} 2>/dev/null' \;
2 years later) but thank you, i will try your script just today sir!
Receive/Transmit Packet Steering is a software technology intended for systems having NO hardware packet processing acceleration.
But it looks like your hardware can do it better.
Server motherboard ASUS P9D-C/4L
https://www.asus.com/Commercial-Servers-Workstations/P9DC4L/specifications/
Enterprise-level Intel I210-AT NIC's:
https://www.intel.com/content/www/us/en/products/sku/64400/intel-ethernet-controller-i210at/specifications.html
According to specification above, all of your NIC's supports hardware-accelerated network processing called RSS (Receive-Side Scaling).
Receive-Side Scaling (RSS), also known as multi-queue receive, distributes network receive processing across several hardware-based receive queues, allowing inbound network traffic to be processed by multiple CPUs. RSS can be used to relieve bottlenecks in receive interrupt processing caused by overloading a single CPU, and to reduce network latency.
Intel I210-AT NIC's supports up to 4 hardware-based queues.
To utilize RSS you can try to use Ethernet drivers from the following package repository:
After some hardcore studying and research, my system with a j1900 is set to this simplified config:
RSS is better than irq affinity or RPS, when the driver supports it.
Disable irqbalance or packet steering.
# rss defaults are not good.
interfaces='eth0 eth1 eth2 eth3'
for interface in $interfaces
do
ethtool -X "$interface" equal 2;
ethtool -N "$interface" rx-flow-hash udp4 sdfn # udp4 uses only the addresses by default
done
The journey and knowledge gained was well worth the effort.
Linux Scaling
Tuning a very similiar unit
Thank you for sharing your experience!
It sucks not being able to set the rxhash key, as I found what appears to be a good one with good flows and spreading per the author...
can i ask what device you use with j1900?
I don't have a x86 system myself, but I'm wondering if your network driver supports RSS?
Did you have to install a special driver version with RSS support included?
https://github.com/csrutil/OpenWrt-NIC-Drivers
Or maybe OpenWRT supports RSS on Intel NIC "out of the box" now?
QOTOM-Q190G4N mini pc
I believe even when I was using v19, it supported RSS
DISTRIB_ID='OpenWrt'
DISTRIB_RELEASE='22.03.2'
DISTRIB_REVISION='r19803-9a599fee93'
DISTRIB_TARGET='x86/64'
DISTRIB_ARCH='x86_64'
DISTRIB_DESCRIPTION='OpenWrt 22.03.2 r19803-9a599fee93'
DISTRIB_TAINTS=''
Would you please to run the commands below and share us the results.
This will show how many queues your NIC has.
ethtool -S eth0 | grep queue_
cat /proc/interrupts
cat /etc/modules.d/35-igb-intel
Thank you in advance!
I think maybe you're confusing me with the OP, but here are my specs
cpu: J1900
nic: I211-AT
RSS queues: 2 rx 2 tx
driver ver:
version: 5.10.146
firmware-version: 0. 6-1
I've tried to install the latest driver from the posted link, but it's giving me errors:
root@ROUTER:~# opkg install /tmp/kmod-igb-intel_5.10.161+5.11.4-1_x86_64\ \(1\).ipk
Unknown package 'kmod-igb-intel'.
Collected errors:
* pkg_hash_check_unresolved: cannot find dependency kernel (= 5.10.161-1-9309db0b721b9fb3fe826181264a78c4) for kmod-igb-intel
* pkg_hash_fetch_best_installation_candidate: Packages for kmod-igb-intel found, but incompatible with the architectures configured
* opkg_install_cmd: Cannot install package kmod-igb-intel.
root@ROUTER:~# cat /etc/os-release
NAME="OpenWrt"
VERSION="22.03.2"
ID="openwrt"
ID_LIKE="lede openwrt"
PRETTY_NAME="OpenWrt 22.03.2"
VERSION_ID="22.03.2"
HOME_URL="https://openwrt.org/"
BUG_URL="https://bugs.openwrt.org/"
SUPPORT_URL="https://forum.openwrt.org/"
BUILD_ID="r19803-9a599fee93"
OPENWRT_BOARD="x86/64"
OPENWRT_ARCH="x86_64"
OPENWRT_TAINTS=""
OPENWRT_DEVICE_MANUFACTURER="OpenWrt"
OPENWRT_DEVICE_MANUFACTURER_URL="https://openwrt.org/"
OPENWRT_DEVICE_PRODUCT="Generic"
OPENWRT_DEVICE_REVISION="v0"
OPENWRT_RELEASE="OpenWrt 22.03.2 r19803-9a599fee93"
root@ROUTER:~# ubus call system board
{
"kernel": "5.10.146",
"hostname": "ROUTER",
"system": "Intel(R) Celeron(R) CPU J1900 @ 1.99GHz",
"model": "Default string Default string",
"board_name": "default-string-default-string",
"rootfs_type": "ext4",
"release": {
"distribution": "OpenWrt",
"version": "22.03.2",
"revision": "r19803-9a599fee93",
"target": "x86/64",
"description": "OpenWrt 22.03.2 r19803-9a599fee93"
}
}
DOH!
I download and tried to install the wrong kernel version,
Success!
root@ROUTER:~# ethtool -i eth0 | grep version
version: 5.11.4
firmware-version: 0. 6-1
expansion-rom-version:
now to give it a spin...
i've also tryed to install custom RSS driver from link provided, but it was not work great for me, to fix your error, first delete kmod-igb already installed in your system, 100% will work)), with this custom driver i had some strange cpu behaviour even my x86 router was working great..
Now I'm seeing only 1 queues for rx/tx vs 2. I'm not understanding the directions besides using ethtool to change the number of queues?
root@ROUTER:/etc/modules.d# ethtool -l eth0
Channel parameters for eth0:
Pre-set maximums:
RX: 0
TX: 0
Other: 1
Combined: 2
Current hardware settings:
RX: 0
TX: 0
Other: 1
Combined: 1
i'm not sure how works that driver and why you have 1 less queue, but as i mentioned, i also had strange behaviour.. but with i210
but we can create a git-issue and ask developer))
So, to set the RSS no. of queues in openwrt for the I211-AT loadable module at boot. I believe the I210-AT has 4?
This is interesting stuff. I stumbled on that RSS github driver collection back in 2022 while I was searching for some related topics on QOS. I thought hardware acceleration works great in theory...but in real world its a big bag of unknown.
Went to openwrt forums, nobody has created a topic of this until now....
Pretty cool to see you guys test if RSS hardware acceleration has some benefits -- most notably, if it improves CAKE qos. Will watch this thread with great interest.
It is not recommended to set the number of RSS queues greater than the number of physical processor cores:
When configuring RSS, Red Hat recommends limiting the number of queues to one per physical CPU core. Hyper-threads are often represented as separate cores in analysis tools, but configuring queues for all cores including logical cores such as hyper-threads has not proven beneficial to network performance.
https://www.kernel.org/doc/Documentation/networking/scaling.txt
note that on processors with hyperthreading (HT), each hyperthread is represented asa separate CPU. For interrupt handling, HT has shown no benefit in initial tests, so limit the number of queues to the number of CPU cores in the system.
This means that only 2 of the 4 available queues should be used when configuring RSS for I210-AT on a 2-Core CPU.