Here's my attempt at bufferbloat mitigation. I'm running a Comcast cable connection (1 Gbps down / 30 Mbps up) through a Motorola MB8611 Cable Modem, and an OpenWRT One router.
It works pretty well, keeps router CPU utilization low, and is compatible with Software Offloading and Hardware Offloading.
The wan_throttle_autorate script is very much still a work in progress. I'm in the 'testing and finding edge cases' phase. I've tested it down to 100 kbps minimum upload speed limit (with DECAY_TRIGGER=50). When doing the BufferBloat test, it ramps from 100 kbps to 33000 kbps (my set maximum) pretty quickly, then ramps back down on network quietude.
The way it works: It'll start up at your set maximum, then ramp down until it hits just under your current upload rate, whereupon it'll hold. If your network is quiet and it ramps all the way to the minimum, when the data rate spikes, it'll ramp up to just below your current upload rate.
It spares NAND write cycles, because it's injecting Linux Traffic Control (tc) Token Bucket Filter (tbf) data into volatile memory. The state file (to print out the status) and PID file are also in volatile memory.
If you stop it (wan_throttle_autorate stop), it'll clear the tbf data, which completely unthrottles the data rate.
Standard solutions like SQM (Smart Queue Management) add too much CPU overhead, and standard Firewall4 (fw4 / nftables) rules get completely bypassed (for already-established connections) if Software or Hardware Flow Offloading is enabled.
But if I look at sqm-scripts, all they do is set up an IFB and call into tc to instantiate the desired qdisc hierarchy... pretty much what you are doing as well, so puzzled how you avoid the CPU overhead of SQM? And since you are doing the same as SQM you will have pretty much the same issues with software and hardware offloads. So by all means develop your own solution and tout your horn here, that is IMHO well-deserved, but maybe restrict claims about sqm-scripts to those that you actually did empirically confirm.
I did empirically confirm it. luci-app-sqm and sqm-scripts spiked CPU utilization on my OpenWRT One. I'm not sure why, I didn't trace it down. I uninstalled it.
I have Hardware Offloading enabled... upload speed is still throttled, and given that by throttling upload speed I can throttle download speed, so is download speed.
I'm not "touting" anyone's "horn". I developed a solution to a problem, the best way to ensure there are no bugs is to have others see it, run it, debug it. More eyes and more braincells means faster code evolution toward optimum.
Well, sqm-scripts really has no real run time component that consumes CPU aside from the qdiscs, so if you use the same qdisc with sqm and with your approach you need to end up with the same CPU load during comparable traffic loads, if that is not the case one of the solutions differs from the other, as being involved in sqm-scripts I would love to learn if sqm-scripts has an issue. Maybe you remember which of the scripts you used, and whether you tried any special configuration?
That does not bode well, unless you also offload the traffic shaper and qdisc...
How that?
Mmmh, sorry, this was not intended as insult, in opensource you need to talk about yur solution if you want feed-back, and that is what I wanted to convey. For me "touting ine's horn" ist not a bad thing.
moeller0 wrote:
"That does not bode well, unless you also offload the traffic shaper and qdisc..."
That doesn't even make sense.
Hardware Flow Offloading operates at Layer 3... prerouting, forward and postrouting are here.
Linux Traffic Control operates at Layer 2, and it can read Layer 3 headers.
IOW, Linux Traffic Control is prior to, superior to, Hardware Flow Offloading.
moeller0 wrote:
"How that?"
TCP flow control.
You've never used Linux tc to clamp upload to, say, 50 kbps, then done a bufferbloat test or attempted downloading a large file? Your download will only be single-digit mbps. It's an exchange of data... limit one leg of that, and you limit the whole process.
Your upload : download ratio is going to be ~23.72 if you have a strict 1:1 ACK for every packet, or ~47.44 if TCP Delayed ACKS is enabled (it is). With upload clamped to 50 kbps, your download is going to be ~2.3 mbps.
So we're using TCP flow control against itself, to limit download rate by limiting upload rate.
Conversely, sqm-scripts and luci-app-sqm create an Intermediate Functional Block into which all packets are shuttled. The CPU must then manually context-switch, prioritize, queue, schedule, and (in the case of bufferbloat) delay those packets (ie: traffic shaping) inside the IFB before passing them back to the network stack. And that is the reason behind the high CPU utilization.
Just wondering, have you tried QoSmate? Because from my experience, I have the same internet provider, and using standard SQM didn't really fix bufferbloat even when throttling the connection, so I tried a different SQM script that already has cake autorate, which, from my understanding, adjusts based on parameters you give it but is always dynamically adjusting to reduce latency, which seems similar to what you're doing. The future is definitely a script that can dynamically adapt to the connection, which I do see others making similar stuff, and/or waiting for technology that will take ages to be implemented from the ISP side like L4S.
I do think if you had actual graphical data your claims would look a bit more valid because seeing A+ doesn't neccesarily mean it stable at all times. I like seeing receipts am a latency fanatic.
Hardware flow offloading steals packets from the rest of the network stack, and a qdisc can only operate on packets it sees. If the shaper is offloaded into the same offload engine, it sees the packets and can operate on those.
This is easy to confirm e.g. witb cake in besteffort mode:
reset the counters for smaller numbers and take a pre-test snapshot of tc -s qdisc
after the test finished take another cake statistiks snapshot tc -s qdisc
compare the increase in package and byte counts
Do this twice once with hardware offloads enabled and once with hardware offloads disabled, then compare the delta counts between the two tests. If I am correct the offload enabled test will show noticeably fewer packets and bytes for the same test.
If that would be true, sqm-scripts would have zero issues with offloads as well, at least in upload direction, as it uses the same tc system as your solution. But honestly, you might want to research this topic a bit deeper as there are plenty reports of tc qdiscs being side stepped by offloadschard or soft.
Well, let's be frank here, this is not incorrect, at least for TCP Reno, but repeat your test with BBR or QUIC, but also pretty useless. Unless you throttle the Uplink hard it will have little efrect on Downloads, especially if you use a head dropping flow fair aqm like fq-codel or cake (ACKs are cummulative, so head dropping will not do what you want to slow flows down).
Well, if you can not stomach the cost for the IFB and do not want/need download traffic shaping and AQM just set the ingress rate in sqm to 0... in thst case no IFB is instantiated and no traffic is passed over one.
Typically an IFB adds something like 5% to the load, small compared to the actual traffic shaping.
About context switching, qdiscs do not run in userspace...
So I had another look at the project and I am puzzled about the logic behind the autorate setting... this is driven from the qdisc dropped stats (so at the very least it should also evaluate marked, but I digress) so purely from the input/LAN side if used on your WAN link. How should the LAN have reliable information about the WAN side congestion? cake-autorate and sqm-autorate (and purple and perl-autorate) all resort to actively measuring delay changes over the wan link (libreqos uses passive RTT measurements via the pping approach, but that still measures the expected effect of congestion/bufferbloat on the relevant WAN link), exactly they need to be able to detect the WAN link's current capacity. But the shapers drop rate really only tells you something about the LAN side. So respectfully, this seems not to be best available proxy for WAN side congestion.
How did you come up with this design? And how much of the apparent confusion is AI?
I first tried ecn_marked, but that didn't work. Even under heavy congestion, pushing bandwidth as hard as I could, ecn_marked didn't change much.
But Overlimits changed by quite a bit, and when it got up into the thousands, Drops started ramping up, as well.
So I set the script up to trigger first on Overlimits, then on Drops.
It works... if I run with Linux tc at defaults (enabled on OpenWRT for FQ_CODEL out of the box, but unthrottled on wan because the OpenWRT developers have no way of knowing what bandwidth you've paid your ISP for) and max out the connection, latency more than doubles from no-load; whereas with the script, download latency increases by less than +10ms, and upload latency remains at +0ms.
moeller0 wrote:
"Hardware flow offloading steals packets from the rest of the network stack, and a qdisc can only operate on packets it sees."
We set the hardware's transmit ring buffer parameters in the script. When offloaded traffic enters the ring buffer faster than it can handle, a tx ring exhaustion event occurs. Those packets over and above what the tx ring buffer can handle are evicted from the hardware accelerator and dumped back into the software stack. By setting the transmit ring buffer in script we control the offloader.
ECN marking is restricted to flows actually using/negotiating ECN in the first place, typically that needs to be enabled in the endpoints, but if in use packets are marked and only dropped in emergency cases, so for rfc3168 compliant flows marks are equivalent with drops.
Now, I doubt that driving a WAN-shaper from the LAN traffic drops/marks is a usefull aprroach to begin with, but if you do it, you should not only look at drops.
Overlimits really just tells you that the qdisc engaged with the traffic, if overlimits stays low, this just means traffic was really sparse and the qdisc did not do much.
This really just tells you that there is LAN side traffic, but that is exactly what the combination of a fixed traffic shaper with an flow queueing AQM is designed to handle well, without needing to ramp up aggressiveness (and if you want a more aggressive shaper in cake just configure the "ingress" keyword).
Arguably...
Yes, but that is not the relevant comparison if you claim to be better than sqm-scripts, is it? Mind you, it is totally fine rolling your own and ignoring sqm-scripts, shit-talking sqm-scripts however is considerably less cool.
moeller0 wrote:
"Yes, but that is not the relevant comparison if you claim to be better than sqm-scripts, is it? Mind you, it is totally fine rolling your own and ignoring sqm-scripts, shit-talking sqm-scripts however is considerably less cool."
I didn't claim to be better than sqm-scripts. I claimed lower CPU utilization than sqm-scripts. No one is "shit-talking" (your words) anything. Pointing out the fact that sqm-scripts and luci-app-sqm spikes CPU utilization is fact, not "shit-talking" (your words).
moeller0 wrote:
"This really just tells you that there is LAN side traffic"
Yes, the drops/overcommits qdisc on eth0 will count the "pressure" the LAN side generates in the WAN direction... that is not really diagnostic for WAN side congestion, especially with a traffic-shaper, -scheduler, and AQM on the WAN interface that is already designed to deal with that.
The whole idea behind traffic shaping in SQM is to make sure the queues stay unter control of fq_codel/cake as these AQM/scheduler combinations generally deal with that gracefully.
But you did not really demonstrate it... you at best were comparing apples with bananas, or egress-only shaping with your solution versus bidirectional shaping with SQM... (now, as I said you can tell sqm to not shape in ingress, I bet if you would do this the differences would mostly go away, especially if you used simplest_tbf.qos).
I am inclined to let this rest here, as this not getting any more productive, and I generally want to encourage people to explore this problem space and come up with their own solutions.
tc -s qdisc show dev eth0 is showing LAN-side "pressure" only because that data is going out over the wire... that "pressure" is in the tx ring buffer of the wan physical interface.
Better that we manage that "pressure" locally in the router than allow it in the modem or upstream at the ISP, where we have no control.
moeller0 wrote:
"you at best were comparing apples with bananas"
Thus far, you've stated that I'm "shit-talking" (your words) sqm when in fact I merely stated that my solution reduces CPU utilization as compared to sqm; and now you're claiming I'm "comparing apples to bananas" (your words), when I'm merely working from different operational constraints that you, apparently, refuse to even consider.
You want a sqm-managed packet flow, regardless of CPU utilization. I want low CPU utilization for reasons you've not considered, despite my having explicated those operational constraints in the very first sentence of the very first post pertaining to this topic, to wit:
"You want to mitigate bufferbloat by capping upload speed to just below the ISP’s limit, but for whatever reason, you need to limit CPU utilization."
Your increasingly going ad hominem can only stand as testament that you are an sqm fanboy who will defend it at all costs, disregarding the very operational constraints which caused myself (and likewise likely cause others) to not use sqm, and hence my casting about for a solution that doesn't spike CPU utilization.
I'm currently working on one that looks at the acceleration of Overlimits and Backlog. We'll see how it works.
Yes, but once you do that there is no real value in flexibly adapting the shaper, let alone the interval and target settings of the AQM, to the amount of WAN-directed traffic from the LAN. The whole idea is to get the bottleneck queues under control of a competent shaper and AQM, once you have that best not interfere with that scheduler and AQM unless you need to.
Well, again, your solution will not use less CPU than sqm-scripts, if you configure sqm appropriately... really these are just two slightly different ways of setting up the same qdisc/qdisc hierarchy, after they are instantiated TBF+fq-codel will not differ whether started from your script or from sqm-scripts...
What I want is irrelevant here, what we get is the pretty much the same qdisc set-up, and these will have pretty much the same CPU-cost for the same traffic, assuming you do not compare a bidirectional traffic shaper with a uni-directional one.
Yes, I am clearly getting annoyed by our apparent inability to find common ground in this discussion. Not sure where you find an ad hominem, but I take you by your word and accept that you feel that way.
No, I am one of sqm-scripts' junior developers, and I had my fingers in it when fq-codel and cake were created, in short I am meddling with this topic for over a decade and consider myself to be reasonably knowledgeable about the details.
Well, I have a prediction, that I am not going to share.
But explain this, why do you think that these (Overlimits and Backlog) are indicative of upstream congestion and why do you think that either fq-codel or cake fail to deal properly with that situation? Both were designed exactly to handle that situation, or better yet, the increase in drop rates you see comes exactly from fq-codel/cake dealing with excessive and/or under-responsive traffic originated on the LAN-side.
Let me ask a question though, how much of the research behind this project is from good old searching and reading, and how much is from AI summaries?
Except for that IFB, and the concomitant CPU-based processing of packets by sqm. And the fact that sqm doesn't work well with Hardware Flow Offloading, so the CPU is processing all packets.
Whereas my solution merely limits the wan physical interface's tx ring buffer such that if it's saturated, excess packets are shunted back to CPU-based processing. The rest of the packets are handled by the Packet Processing Engine. Hence we control the extent of hardware offloading.
For you to say the CPU utilization in both cases will be strictly identical is head-scratching... especially considering that I wouldn't have embarked upon attempting to find a different solution if sqm hadn't spiked CPU utilization.
moeller0 wrote:
"Yes, I am clearly getting annoyed by our apparent inability to find common ground..."
If by "find common ground" you mean "use sqm", right?
Now you're backpedaling and attempting to minimize your having gone ad hominem. How's that working for ya?
moeller0 wrote:
"But explain this, why do you think that these (Overlimits and Backlog) are indicative of upstream congestion and why do you think that either fq-codel or cake fail to deal properly with that situation?"
To the contrary, I'm using FQ_CODEL for just that reason. Are you attempting to imply that I'm not using any scheduler?
But out-of-the-box, while FQ_CODEL is enabled, it is unthrottled on wan, by necessity (because the OpenWRT developers have no way of knowing what bandwidth you've paid for). It can do nothing to mitigate bufferbloat. I'm merely using FQ_CODEL to manipulate Linux Traffic Control by changing the parameters of the tx ring buffer (and thus controlling hardware offloading) in an attempt at doing same.
"Let me ask a question though, how much of sqm is from good old searching and reading, and how much is from AI summaries?" (your words, paraphrased)
You see how ad hominem your sqm fanboyism has caused you to become?
Well, as I keep saying, if you do not want ingress shaping and IFB, sqm-scripts allows you to not use that, simply by configuring the download/ingress rate as 0. The fact that I need to repeat this and you still do not get it, implies, that you did not look too closely at what sqm-scripts actually does. And that is A-OK, but please refrain making statements about what sqm-scripts does n that case.
So, the same will be true for your solution, either the hardware offload steals the packets before your qdiscs sees them and acts upon them, OR you will pay the full CPU cost of that qdiscs processing per packet. As I described above that is relatively easy to test and confirm, but excuse me for not doing these tests for your solution.
Puzzled, you are instantiating e.g. a cake qdisc in shaper mode and adjust the shaper rate by your weird control law, that is different from changing the TX buffer size as you seem to claim, please describe your theory of operation?
In your dreams... that is not how this works, I wish it was different, but that is not how this operates. That makes me wonder how diligently did you measure and confirm that your solution actually improves upon anything on your link?
Well, basic logic, either the qdisc processes the packets and then the same qdisc will n(for similar enough configurations) have the same CPU cost whether it was instantiated via sqm-scripts or via your script, as in both cases at run time we are dealing with exactly the same qdisc. OR we do not see CPU spikes if the qdisc sim ply does not get hold of the packets when hardware offloading steals the packets away from the qdisc. But humor me and perform the test I sketched above and show that for the same capacity test cake reports similar numbers of packets and bytes during the test both with and without hardware offloads enabled. We can postpone measuring CPU costs until this test, as I have a strong assumption, that this test will show massively fewer packets hitting cake with hardware offloads enabled.
No, as I said, I fully endorse people exploring this space by themselves and finding their own solutions, and I am happy if they exceed sqm-scripts and will recommend them, e.g. I recommend both qosify and qosmate over sqm-sxcripts for people that want to do per flow DSCP based prioritisation. But I do not like people spreading incorrect information.
No, I am not back-paddling, as I still do not see an ad hominem here, what I do see is an attempt to derail a technical discussion by claiming ad hominem attacks, something I will not continue to engage with.
So how do yu think fq-codel tells a flow that is above its capacity share how to slow down? It does so either by setting an ECN mark (IFF ECN is enabled in fq-codel, AND the flow in question actually uses ECN, or rather has either ECHT(0) or ECT(1) set in the IP header) OR by DROPPING a packet. If flows do not respond quickly enough to that signal they will accumulate quite a lot of drops... the fq component will make sure that such drops mostly accumulate in glows that stay above their capacity share for longer times. So dropping packets is a sign of fq-codel cake actually fulfilling their control function. This is why I consider it to be at best confused to use those drops as signal to change the shaper rate... This looks like a recipe for a chain reaction... you see a lot of drops then you reduce the shaper rate which at least transently will increase the number of packets that will queue up and hence the drop rate, if you are unlucky this will initiate the next deceleration with concurrent transient drop increase... this dies not seem all that stable and frankly driven by the wrong measure.
Yes and no. If the WAN link is limited by an upstream bottleneck below line-rate than indeed our AQM will never see enough back-pressure to engage, but if you truely have a 1 Gbps ethernet link then BQL is sufficient to generate that back-pressure for fq-codel or cake. On typical home internet access links that is exceedingly rare and so traffic shapers to the resue it is (a concept that is quite old, e.g. see the venerable wondershaper script, that combined traffic shaping with stochastic fair queuing, one of the giants on whose shoulders the current generation of bufferbloat mitigation attemps stands, including sqm-scripts).
Regarding the TX ring buffer, I fail to see in https://forum.banana-pi.org/t/low-cpu-utilization-bufferbloat-mitigation-with-hardware-offloading-enabled/27828/19 where you change the TX ring buffer. Yet you keep repeating that claim, is that a result of your own research or is that coming from AI?
Nope, that is not an ad hominem that is a genuine question whose answer tells me how to proceed... I note that you refrained from actually answering that. Mind you using AI as a research tool is OK, if done in moderation and care is taken to confirm claims made by the AI, what isa a problem is putting too much trust in solutions that AIs provide without properly checking them. But my question is less about ranting for or against AI but rather I want to understand how you developed your theory of operations, that tro repeat it is in my opinion incorrect.
But hey, feel free to assign all to me having a pro sqm-scripts bias instead of considering whether your solution actually does whjat you claim it does.
Upload is absolutely limited, as is upload bufferbloat. Without FQ_CoDel, upload rate is ~42 Mbps and upload bufferbloat usually doubles the unloaded latency. But because FQ_CoDel (or any other shaper) can only operate on packets at the egress of an interface, I'm attempting to find a way to mitigate download bufferbloat... without spiking the CPU. An IFB merely creates a 'shadow' interface in software, then works upon the packets at that 'shadow' interface's egress... same as a veth pair does, same as a VLAN pipeline does, same as a loopback patch cable does, same as a clsact qdisc does.
I've dissociated eth1 from the rest of the machine and I'm working on setting up qdisc to shunt inbound packets from eth0 to eth1, then have FQ_CoDel operate on those packets on the egress of eth1, then shunt them to br-lan.
This is the script I use:
QUEUE_PATH="/sys/devices/platform/soc/15100000.ethernet/net/eth0/queues"
for i in $(seq 0 15); do
BQL_DIR="${QUEUE_PATH}/tx-${i}/byte_queue_limits"
if [ -d "${BQL_DIR}" ]; then
echo "0" > "${BQL_DIR}/limit_min" 2>/dev/null
echo "4542" > "${BQL_DIR}/limit_max" 2>/dev/null
echo "0" > "${QUEUE_PATH}/tx-${i}/tx_maxrate" 2>/dev/null
fi
done
ifconfig eth0 txqueuelen 0
Whereas tx has 16 queues (tx-0 through tx-15), rx only has one (rx-0). And there are only two files there:
Sorry, BQL will only engage:
a) if it is supported by the NIC's driver
b) if there is data queueing up at line rate.
If you put a traffic shaper in front of it, it will never engage.
Not argueing that the combination of a traffic-shaper and a competent AQM will take out bufferbloat at all. I am arguing that your presented theory of how this works in combination with hardware offload is incorrect. And I maintain that.
Good luck, not sure there are realistic options. Traffic shaping is unfortunately quite costly, if you have to run that on the CPU you will have to pay CPU cycles. The best bet would be to offload the traffic shaper to an offload engine (which would need to allow to do this for ingress traffic, as for most users ingress will have higher contracted rates than egress), or offload the whole enchilada to hardware (for QualComm's NSS cores there is, IIRC, an fq-codel implementation and a shaper, nit sure whether it allow operation for ingress traffoc though).
Yes, I am well aware, in practice both options I tested (IFB and VETH) have a similar CPU cost. I note that typically that cost for the IFB pales in comparison for the cost of the ingress side traffic shaper (in some old measurements IFB added around 5%, that is achievable shaper rate increased by 5% when the traffic was not going through an IFB but through the egress side of a LAN facing interface).
Seems quite complicated... but with DSA the issue clearly is that we can not address the CPU to switch port individually anymore, and more importantly this port is shared by LAN traffic between the router's LAN and WLAN ports, and we typically do not want to throttle these. So the obvious alternatives are essentially:
A) full wifi-router using IFB or VETH (or similar) to allow ingress traffic shaping
B) wired-only router with two ethernet interfaces (or 2 VLANS with shaper rates per direction <= half the interface rate and managed switch to split out egress and ingress VLAN traffic to different switch ports (also doable vis VLAN trickery if need be)) implementing internet-download traffic shaping as egress-side shaping of the LAN facing interface.
But really if you want ingress shaping you need to put in the CPU cycles for ingress shaping somewhere TANSTAAFL, even with offload engines the work is just moved somewhere else, but still needs to be performed.
And this again is a different kettle of fish, receive side packet scaling, a good idea to help spreading multiple qdiscs over multiple CPUs, but it will not allow to split one traffic shaper instance over multiple CPUs.
And that will help on a server terminating connections much more than on a router that mainly just handles packets between inside and outside but that does not "consume" those packets.
See here: https://docs.kernel.org/networking/scaling.html
RFS: Receive Flow Steering
While RPS steers packets solely based on hash, and thus generally provides good load distribution, it does not take into account application locality. This is accomplished by Receive Flow Steering (RFS). The goal of RFS is to increase datacache hitrate by steering kernel processing of packets to the CPU where the application thread consuming the packet is running. RFS relies on the same RPS mechanisms to enqueue packets onto the backlog of another CPU and to wake up that CPU.
In RFS, packets are not forwarded directly by the value of their hash, but the hash is used as index into a flow lookup table. This table maps flows to the CPUs where those flows are being processed. The flow hash (see RPS section above) is used to calculate the index into this table. The CPU recorded in each entry is the one which last processed the flow. If an entry does not hold a valid CPU, then packets mapped to that entry are steered using plain RPS. Multiple table entries may point to the same CPU. Indeed, with many flows and few CPUs, it is very likely that a single application thread handles flows with many different flow hashes.
