I'm still there.
is there a good reason to want to do SQM for speeds above 1Gbps with an opensource home router ???
It will be very difficult to find a remote service capable of providing +2Gbps... so it will take a lot of simultaneous connections to completely saturate this type of plan. don't know if SQM is needed in that case.
SInce I have FTTH I don't think I need SQM anymore ... but my sons are still too young to use the internet a lot. We'll see when they're teenagers.
But when I had ADSL the SQM was mandatory in order to have a correct experience during work meetings (Teams, skype ....) from home
My router (a new one, with I226 nic,because the i225's nic router is now dead because of a bad hdmi cable that shortcuted the CPU ... no luck) and my 2.5Gbps equiped computers are "in production" at home. I have a 300Mbps plan so I can't test using internet.
Trying this for yourself involves making all the IT in the house unavailable for a while (router, file server with Hyper-V hosting few VMs like my mail server) so I think you can understand I will not do that.
From what I've seen at home, the Realtek 2.5Gb PCIe cards (not the USB !) seem to perform better than the I225/226 that equip these mini PCs.
And anyway I tend to think that it's not the power of the CPUs that is limiting but rather the IRQ (SIRQ) when we do routing, switching (these are 4 independent cards, not a 4-port switch) or SQM so you need higher freq (or better offloading). And these small CPUs heat up a lot (in poorly cooled fanless cases) at high frequency under load.
Multigig equipment start to become more available, prices will drop (slowly) and I hope we will have more (and better) choices soon. I especially hope that the next generations of hardware will consume less, I find that they currently consume far too much.
I have an unused nic on the router, so I did some quick tests anyway (iperf3 -P4 -t20 -bidir) without having to degrade the service at home:
Environment:
N5105 4x I226V router - 8GB DDR4@2933 - slowest NVMe I had in stock - Proxmox + OpenWRT VM 22.03.0 stable - 4 Cores + 192MB RAM - virtio nics (bridged + queues=4)
iperf3 server: Debian VM on a W10 host - Core i5 8600k - RTL8125B
iperf3 client: Core i7 9700k - W10 - RTL8125B
no Software flow offloading (otherwise SQM no longer works)
switching lan to/from lan (2.5Gb TRendnet switch): 2.33 Gbits/sec
switching lan to/from lan (via the router): 2.20 Gbits/sec
routing lan to/from my iot-lan (via the router) no SQM: 2.20 Gbits/sec (normal, it's the same job for the router to do routing or switching between 2 independent network cards)
and then with SQM (cake/piece_of_cake) configured at 2Gbps (tested with an upper limit without change):
upload from client to server: 1.1 Gbits/sec
download from client to server: 1.55 Gbits/sec was the best I was able to achieve
SQM at these speeds seems to causes instabilities, I suffered unexpected reboots (pve pveproxy[26367]: worker exit) ... resulting in a read-only FS in openwrt
The need for competent queue management is not a direct function of a link's speed... Now, if you do not encounter bufferbloat/increased-latency-under-load/decreased responsiveness under working conditions no need for sqm independent of access speed, if you do encounter these sqm might be beneficial (assuming your CPU can keep up with sqm's demands). Typically queues build up where link speeds change (e.g. internal LAN 1 Gbps + 2.4GHz WiFi 0.6 Gbps + 5GHz WiFi 1.2 Gbps = 2.8 Gbps over a 1 Gbps access link) and then being able to deal with eventual queues gracefully can help.
Well, easy enough to test, use your network without and with sqm and figure out how responsive it feels when using interactive applications during peak usage times. If you feel/see no difference sqm probably is not helping enough to justify it for your use-case.
Your network, your rules.
With internal wiring for 2.5 Gbps and a 0.3 Gbps internet access, I would assume situations where sqm could help are not completely unlikely...
For 2.5 Gbps ethernet in a unidirectional test you can expect at most: 2.5 * ((1500-20-20)/(1500+38)) = 2.37 Gbps
for a bi-directional test you need to subtract the ACK traffic. For normal ACKing after two full MSS segments you expect ~1/40 of the data volume for ACKs. So 2.37/40 = 0.05925 Gbps 2.37 - (2.37/40) = 2.31 Gbps
more modern TCP stacks will do GSO/GRO resulting in less ACK traffic.
With the defaults the expected best IPv4-speed would be:
2 * ((1500-20-20)/(1500+14)) = 1.93 Gbps
Interesting... I am in no position to test this, but this might be related to the VM setup more than to sqm/cake itself.
very interesting post @moeller0, thanks.
I didn't do this test for my own need, but for @JonP, as I don't need to analyze the behavior of this CPU under SQM at such speeds
I finished at 4am, I would have liked to test with the lan cards in PCI passthrough rather than in virtio bridge to try to define if the problem encountered came from the network drivers on the PVE host or something in the genre . But at that time I no longer had the courage
Just for information, these measurements were made "by default" in IPv6. It seems that the speeds are a little better in IPv4.
For example when I get 2.20 Gbits/sec in IPv6 I have 2.25 Gbits/sec in IPv4 (switching via the router in the same subnet).
It makes me laugh when we know that IPv6 is "supposed" to be faster than IPv4. None that none of my testing on this for over a year has led me to ONE case where IPv6 happens to be faster than IPv4.
but hey, the most important thing is that it works fast and well
(Mo/s == MB/s)
Well IPv6 headers are 40 bytes, where IPv4 headers are 20 bytes, so each IPv6 packet has a higher overhead and hence a slightly lower IP-payload throughput:
But that really only affects throughput. For internet connections routing can be different for IPv4 and IPv6 connections to the same servers, and sometimes one or the other has a shorter path with lower latency, but that really depends on the routing and hence is not a sign of one of the two IP versions being generally faster.
IMHO the argument for IPv6 is not "faster than IPv4" but really that for ~8000 million humans with probably more than a single device each IPv4's 4000 Million possible addresses are simply not enough, no ifs no buts...
So,
I installed an alternative kernel on my PVE box (seems to help with N5105 support):
Since that I didn't suffer any openwrt resets under load with SQM.
Results with SQM set à 2Gbps symmetric with queues=4 on each openwrt VM's nic (for network packet balancing between CPUs):
sending: don't know why the 4th core is unused viewed from openwrt VM !?!?!
if I take a look on the PVE box the load is fine:
receiving ... same problem:
If I remove the "queues=4" flag I have worst core balancing (on the VM) but more symmetrical iperf3 results:
sending:
on the PVE box ... looks good:
receiving:
So far, so good
I won't do more tests on this side (because I don't need for my use case, because during these tests the box draws 18~20W instead of 11~12W, because CPU temp is 70°C instead of 45~50°C ....)
I hope this may have helped those who asked
Traffic shapers like the one in cake are quite CPU hungry, and I are serialized behind a single lock, so at best each sqm instance runs on a single core... if we could off load the traffic shaper we might be better off on the qdisc side.
I226v has better perf than I225v b3 (only my experience with 2 boxes)
The N5105 can do SQM with the default configuration at just over 1Gbps but not much more ... at least on a PVE VM (and I don't want to reinstall barebone to test)
routing/switching at 2.5Gbps is doable without SQM (but +10W to achieve that)
Basically for SQM @1Gbps:
Classically: Routing + DNS + DHCP ...
SQM piece_off_cake
Adblock
No Wi-Fi
11~13W idling & 18~21W loaded (1x8GB + NVME PCIe 2X [Kingston A1000 256GB]) depending on number of cables plugged in. Arround 9W with PVE or Ubuntu only running (OpenWRT standalone or in PVE seems to consumme a lot of W ... don't know why)
45/65°C after spending 20 min on the machine to pass a metal file and apply a better thermal paste. Temp takes a long time to drop (fanless in my case)
How did you manage that ??? what storage do you have ?
with only one 1G link active I read 9.9W (with servethehome's bios tweaks) on my Changwang v4 board also latest bios
Either my wattmeter is wrong (probably) or it's yours ... or both perhaps
post has a link to a video of power consumption of i226v nic being driven to limit.
Unit is currently stuffed with a 480GB Corsair MP510, and 2x8G-2400 Corsair, waiting on a single 16G-3200 to replace the 2 SODIMMS. I did notice with the latest BIOS a slight drop in idle consumption, but under load it tops out much higher, currently working on getting a fan into the unit.
I assume both meters are wrong, but by way of comparison, this meter on a idle mamba (WRT1900AC), both radios disabled, sits at ~12.7w.
Recently, some UP masters reported that the old version of BIOS running score data is relatively low.
This updated version removes the power consumption wall limit.
Since there is no power consumption limit, when running at full speed,
please enhance the cooling work by yourself.
After all, this is only a fanless.
It's a mini-host that can't be compared with a desktop computer.
You can play on the bios with 'power limit' and 'platform limit' (PL1 & PL2) for that
Wow... Quite the test session(s) I provoked!! Wasn't expecting such detail, and an all-nighter! You really went above and beyond, vevere!!
Thoughts...
Probably any time there's a big mismatch in speeds, networks bottling into/funneling out... there's potential for flow problems...
I noticed you were running 21.02.3. I'm not sure where along the way, but now that I've updated to 22.03.0, I see a "Intel 2.5G" driver listed in the log on my Zotac box with the Realtec chips. There might be a driver update more specific to your 2.5Gbit chips. Or maybe this came out back in 21.x.x.x sometime? Might be something to check for, when convenient.
Edit: I looked, seems like there was an addition on 6-28, for x86, so post 22.03.0-rc5 would seem to have it, but not in 21.02.3.... some Realtek 2.5Gb support was already in there before that time...
Did some poking around with my system... I also suffer from the "too many experiments, the family comes after me with pitchforks" situation... so I feel your pain... I have a cable connection of 1Gb/35Mb, and as yet don't have a lot of hardware set up to do "real" testing, as well as everything is 1Gbe.
Have some Speedtest-netperf quick tests I did against my internet connection last night, which maybe tells me I have some issues... My machine is lowlier than yours though (N3450, Realtek 1Gbe) and can just handle my connection with SQM. Some odd behavior, as well. I'll post some of that later.
. I corrected it, thanks for your attention 