Yeah, this is a definitely weird aspect of the Pi4, it's like it's magic. I suspect that what is going on is the realtek driver is somehow batching the IRQs, like it just wakes up every 1ms and handles all the packets all at once. By doing this, or something like that, it avoids a huge amount of overhead, and therefore can handle crazy high packet counts.
I wonder if someone puts USB3 NICs on an x86 if it does the same thing? or is it somehow ARM/broadcom CPU specific?
Worth trying on for anyone who has an APU2, unfortunately this Sophos only has USB2 ports, even though the processor has USB3 support.
Aaaand it's gone! 
Did you try installing irqbalance package?
Not sure it would make a lot of difference but you might as well try that too
By comparison the Pi 4 just destroys it performance-wise, even with a Broadcom and dual Realtek USB3 NICs. In the same tests Pi's CPU barely wakes up at all: perhaps 5% sirq flat out on PPPoE/NAT at 941mpbs, and iperf3 tests routed between LANs you can't even tell it's doing anything.
I'd be more inclined to believe the reporting of some metrics is just broken and the number you see is wrong. But this is just a feeling, no proof.
I can try, but I only have USB dongles with ASIX chipsets, no Realtek USB dongles in this house as you probably remember from other discussions.
No need. As I mentioned, the IRQs are distributed amongst the cores on both platforms; as far as possible I like to compare apples with apples. On the Pi it's done with my own script, but as of 21.02.0 on the Intel box I didn't have to do anything: at boot the interrupts land by default on both cores; RX and TX for each queue on same core, the queue pairs on separate cores. This was not the case in 19.07. I haven't looked at the code, this seems to be a kernel thing. It is not done through affinities: the smp affinity remains the all-cores mask (in this case "3").
Hi guys, sorry for the dumb question, is there any chance the APU2 get support from the OpenWRT 21.x stable series?
thanks,
cheers 
Thanks for your tip, but why on ToH the device is listed as "Current Supported Rel -> 19.07.8"?
because the release happened a few days ago and the wiki wasn't fully updated yet. A lot of other devices are still showing "Current Supported Rel -> 19.07.8" too.
Afaik this is a manual process that involves checking if the download server has a firmware download for this device and updating the entry so it's not instantaneous.
This device will run fine on 21.02, I have been running mine on development snapshot (the most latest possible OpenWrt) for years.
Thanks a lot for the clarification. 
The update script didn't play well this time.
If you find discrepancies between dataentry and real support, feel free to correct the dataentry.
yeah if it was one device here and there I would, but for example all entries about PCEngines (both APU and ALIX) are still using the wrong version https://openwrt.org/toh/pcengines/start so I thought you were still working on that.
Hm, that means those SPI chips use the same ID code so the tool does not know exactly what chip that is. Some chip brands do that.
It is better to check what name is written on the SPI chip with your eyes (maybe use a lens because it is small) and then select that manually with -c
for example
-c "MX25L1605"
-c "MX25L1605A/MX25L1606E/MX25L1608E"
-c "MX25L1605D/MX25L1608D/MX25L1673E"
Afaik the BIOS chip on the APU1 is near a mini-pcie slot and near a header. It should be the one with the red dot in the image of the manufacturer website https://www.pcengines.ch/apu1c4.htm
Since you are flashing APU1 BIOS make sure you downloaded the right bios image (it has "apu1" in the name).
SPI chips are not the same on all boards. They are the same size but not always the same brand/model. Different batches of boards will get whatever is cheapest or most available at the time. Just like with a lot of other generic components (resistors, inductors and capacitors).
So I think other people should still use the normal command WITHOUT the -c "XXXXXXXXXXXXXX" and then if it fails should still do the manual check with their eyes and adjust the command to the chip in their own board.
Just forcing the SPI chip ID like that can be very bad if their board has a different chip.
I got a lone APU3 from the batches of simplewans I bought, and yes the same procedure as APU2 (with different BIOS image of course) works fine.
APU3 is not "better" it is just "optimized for LTE modems" aka 2 of the 3 mini pcie slots are wired with USB and it has two SIM slots (one for each mini pcie slot).
Everything else is the same as APU2.
APU4 is also the same as APU2 but with 4 ethernet ports, so this is easy to see from the back.
Yes that page has a link to the board schematics of the custom adapter for APU2/3/4.
The page for the custom adapter for APU1 has also the schematics https://pcengines.ch/lpc1aapu.htm
Reading the schematics you know how things are wired and what components you need.
is the simplewan number different or the production date?
regards alex
No it is not. Same sticker as all others in the batch. Production date was in 2017, and product number was SW302DA. Only difference was mac address
I also got a lone APU2D2, which is identical to the APU2C2 but with a slight board design rework
"Reduce leakage current between V3 and V3A power rails (can cause problems with SD cards). Add options for better compatibility with LTE modem modules."
But this is 2 outliers for around 30 units I've opened, so I still think the bulk of them is APU2C2.
My current theory is that some odd boards were just sitting in PCEngines inventory when a large bulk order from Simplewan went through and so they were pressed into service to fulfill the order, since Simplewan isn't using the mini pcie slots anyway so for them there was no difference between them.
Can I DD a duplicate image from the SDcard to a mSATA drive and just removed the SD card?
Yeah, using OpenWrt on the SDcard to DD a new image to the mSata drive works fine.
It works the same for all x86 (or any architecture that can boot from multiple sources, really) see https://openwrt.org/docs/guide-user/installation/openwrt_x86#installing_openwrt_in_an_internal_drive
Although there is no real need to use a mSata drive for a normal OpenWrt install, since it does not write to disk unless you are saving the configuration.
mSata is required for pfSense or OPNSense or a normal Linux distro afaik as those do write logs and stuff constantly to the drive.
Or if you want to install applications that write to disk on OpenWrt, of course.
At least for pfSense, the storage requirements are indeed insane. A few years ago I wanted to quickly test something using pfSense running from a USB stick (ivy-bridge pentium g2020, 16 GB RAM, internal HDD disconnected, to keep its installed data safe) - it was unusable. Booting took ages, even once fully booted up, it remained extremely slow (and I had more than enough RAM to cache the entire USB stick). OpenWrt works just fine in a setup like that - and even full-blown desktop linux tends to run quite well from USB sticks, if there's enough RAM for caching (yes, you notice and want to use fast USB sticks, but even with slower ones it remains usable).
Yeah it's not particularly optimized, boot time is measured in minutes even on Sata SSD for FreBSD-based projects. Also FreeNAS/TrueNAS take a while to boot and it's just a NAS system with a web interface.
On the same hardware (a somewhat lousy Atom S1260 with 8GB of ECC ram) an OpenSUSE 15.3 Linux distro with similar services (ZFS storage array, web server, Syncthing, a few SMB shared folders) is up and running in less than 10 seconds.
Between this, the fail to do multicore routing properly (that results in Linux-based firewalls eating their lunch and dinner with more than double the performance), and the extremely pickyness on hardware I'm not a fan of anything based on *BSD.
Yes they have a much more web polished interface, but boy they run bad.