Sirq bottleneck on x86

Installing and Using OpenWrt
1

I'm using OpenWrt 23.05 on a relatively powerful x86 box - a Qotom Q350G4 with an Intel i5-4200U CPU and four Intel I211 NICs.

I've recently upgraded to a symmetric 1Gb FTTP internet connection, and when downloading at full speed the CPU usage on one core jumps to nearly 100%. top and htop both show that this is due to software interrupts:

image
image1471×348 28.7 KB

Interrupts shown in /proc/interrupts are pretty evenly distributed across hardware queues and cores by default:

# cat /proc/interrupts | grep eth
 48:          0          0          0          1   PCI-MSI 524288-edge      eth0
 49:    6295892          0          1          0   PCI-MSI 524289-edge      eth0-rx-0
 50:          0    5737640          0          1   PCI-MSI 524290-edge      eth0-rx-1
 51:          1          0   12203297          0   PCI-MSI 524291-edge      eth0-tx-0
 52:          0          1          0   10926988   PCI-MSI 524292-edge      eth0-tx-1
 58:          0          1          0          0   PCI-MSI 1572864-edge      eth2
 59:          0          0      38863          0   PCI-MSI 1572865-edge      eth2-rx-0
 60:          0          0          0      35382   PCI-MSI 1572866-edge      eth2-rx-1
 61:      36564          0          0          0   PCI-MSI 1572867-edge      eth2-tx-0
 62:          0      37690          0          0   PCI-MSI 1572868-edge      eth2-tx-1
 63:          0          0          1          0   PCI-MSI 2097152-edge      eth3
 64:          0          0          0    9603999   PCI-MSI 2097153-edge      eth3-rx-0
 65:    8765842          0          0          0   PCI-MSI 2097154-edge      eth3-rx-1
 66:          0    7544189          0          0   PCI-MSI 2097155-edge      eth3-tx-0
 67:          0          0    8767616          0   PCI-MSI 2097156-edge      eth3-tx-1

(eth0 is WAN, eth2 is the guest network, and eth3 is the main LAN)

I've tried enabling packet steering, irqbalance, and even manual balancing as described in this doc, but none of it makes any difference to the sirq load.

I also thought this might be due to SQM, but nothing changes even if I stop the sqm service :thinking:

So while the box can handle a 1Gb WAN, it seems that it can only just manage it, which is surprising to me. Is there something else I can do to distribute the sirq load across cores?

2

It turns out the speed test I was using sends everything over a single UDP connection, despite the interface indicating that the number of "threads" is 5.

When I run a reverse iperf test with 10 streams (on the router itself) the results are much nicer:

image
image1470×275 18.2 KB

(EDIT: swapped screenshot for one that uses the same colour scheme as the one in the OP)

Is it working as designed for a single stream's sirq load to be confined to a single core rather than being distributed across multiple cores?

3

Did you take a packet capture? I would be interested in the packet sie and hence the packet rate of the test

4

Basic tcpdump output while running this test looks like this:

...
11:13:58.733667 IP 172.67.157.175.443 > XXXXXX.61837: UDP, length 1200
11:13:58.733667 IP 172.67.157.175.443 > XXXXXX.61837: UDP, length 1200
11:13:58.733667 IP 172.67.157.175.443 > XXXXXX.61837: UDP, length 1200
11:13:58.733667 IP 172.67.157.175.443 > XXXXXX.61837: UDP, length 1200
11:13:58.733667 IP 172.67.157.175.443 > XXXXXX.61837: UDP, length 1200
...

So at 900Mbps that would be around 94,000 pps?

1 Like
5

Yes, roughly, assuming an IP length of 1200
0.9 * 1000^3 / ((1200 + 38) * 8) = 90872 pps

But with full MTU packets that would still be
0.9 * 1000^3 / ((1500 + 38) * 8) = 73147 pps

But maybe the difference is packet steering, with multiple flows rx-0 and rx-1 get distributed to different CPUs, but with the single stream test all processing ends up on the same CPU?

If that is true you should similar abysmal results when doing a single flow iperf test?

6

Well here around the same SIRQ loads exists (as expected for the same throughput) as in the UDP test, albeit distributed over two CPUs which as far as I understand packet steering is as good as it can get for your NIC.

I am puzzled a bit that a "normal" 900 Mbps load should tax a decent x86 CPU that badly... especially since intel NICs are often claimed to be quite efficient.

7

If that is true you should similar abysmal results when doing a single flow iperf test?

Yes, a single flow iperf test shows a very similar sirq load on a single CPU.

Well here around the same SIRQ loads exists (as expected for the same throughput) as in the UDP test, albeit distributed over two CPUs which as far as I understand packet steering is as good as it can get for your NIC.

Agreed. My understanding is that this is because there are only two physical cores.

But maybe the difference is packet steering, with multiple flows rx-0 and rx-1 get distributed to different CPUs, but with the single stream test all processing ends up on the same CPU?

I was wondering the same thing. I'm not sure exactly how packet steering works though.

I am puzzled a bit that a "normal" 900 Mbps load should tax a decent x86 CPU that badly... especially since intel NICs are often claimed to be quite efficient.

Me too.

It's probably worth mentioning that the core clock frequency doesn't max out during the test (it spends most of its time under 2Ghz, and the max is 2.6Ghz), so there's probably slightly more headroom here than the htop output suggests, but it still seems like a very high sirq load.

8

I personally run an x86 (i7-6700T) with performance governor all the time: additional power consumption is non detectable and no need for the CPU to spend time switching frequencies.