will take a look. it was interesting fiddling around with the rps settings, I tried 3 and c to allow each NIC to have two CPUs but as I remember that worked worse, though I'll try it again. one issue is I'm trying to route just one iperf3 stream, so I think it hashes the stream and always sends it to the same processor anyway. with several streams like a dslreports speed test you'd probably get different results. But I'm trying to route a gigabit, so getting the WAN involved is not optimal.
I tried adjusting xps and rps on both nics. the xps doesn't seem to be supported on the USB nic, but rps is... fiddling around with these didn't seem to make any major difference so long as I had them sufficiently enabled. I'm currently echoing c to all of them, so I get CPU 2 and 3 available.
the issue I think is I'm routing a single stream, which means that it all gets hashed to the same place and so there is just one softirq thread involved and it maxes out at 100%.
Good news is that doing that produces 580+ Mbps sustained routed, firewalled, and HFSC shaped! Hey that's not bad.
I do suspect that if I had multiple streams going that they would spread over multiple CPUs and I'd be able to saturate the LAN interfaces. It also might help to use a bond between the two NICs rather than one for "wan" and one for "lan".
in my opinion the results show that the RPi is a capable router when combined with a smart switch. There is probably a better USB NIC than the amazonbasics one I got. Maybe I'll try a TP-Link which uses a Realtek chipset.
If I were looking for an inexpensive router for anything less than 500Mbps I would choose the RPI 4 and a tp-link sg108e smart switch. Total cost less than $150 and even if all you have is say 10Mbps, it will grow to almost anything you can get today... probably gigabit included in aggregate, though single stream does seem to be limited to around 500 reliably with shaping, maybe 800 without shaping.