Ultimate SQM settings: Layer_cake + DSCP marks

[warchief]

# This file is interpreted as shell script.
# Put your custom iptables rules here, they will
# be executed with each firewall (re-)start.

# Internal uci firewall chains are flushed and recreated on reload, so
# put custom rules into the root chains e.g. INPUT or FORWARD or into the
# special user chains, e.g. input_wan_rule or postrouting_lan_rule.

# This file is interpreted as shell script.
# Put your custom iptables rules here, they will
# be executed with each firewall (re-)start.

# Internal uci firewall chains are flushed and recreated on reload, so
# put custom rules into the root chains e.g. INPUT or FORWARD or into the
# special user chains, e.g. input_wan_rule or postrouting_lan_rule.

## flush mangle table
iptables -t mangle -F FORWARD
iptables -t mangle -A FORWARD -j ACCEPT

## defaulttraffic
iptables -t mangle -A FORWARD -I br-lan -m connmark --mark 0x0/0xf0 -j ACCEPT
iptables -t mangle -A FORWARD -I br-lan -j DSCP --set-dscp 0

##ACCEPT ROUTERIP
iptables -t mangle -A FORWARD -i eth+ -s \10.0.0.1/32 -j ACCEPT

##packetlimit
iptables -t mangle -A FORWARD -p udp -s 192.168.1.2 -m hashlimit --hashlimit-name gamingudp --hashlimit-upto 250/s --hashlimit-burst 250 --hashlimit-mode srcip,srcport,dstip,dstport --hashlimit-htable-gcinterval 1000 --hashlimit-htable-expire 1000 -j CONNMARK --set-mark 0x46 -j ACCEPT
iptables -t mangle -A FORWARD -p udp -s 192.168.1.2 -m connmark ! --mark 0x46 -m conntrack -m multiport ! --ports 53,5353,60887 -m connbytes --connbytes 0:770 --connbytes-dir both --connbytes-mode avgpkt -j DSCP --set-dscp 46

## game traffic
iptables -t mangle -A FORWARD -p all -s 192.168.1.2 -m multiport ! --ports 53,5353,60887 -j DSCP --set-dscp 46
iptables -t mangle -A FORWARD -p all -d 192.168.1.2 -m multiport ! --ports 53,5353,60887 -j DSCP --set-dscp 46

#DNS traffic both udp and tcp
iptables -t mangle -A FORWARD -p udp -s 192.168.1.2 -m multiport --port 53,5353 -j DSCP --set-dscp 26
iptables -t mangle -A FORWARD -p tcp -s 192.168.1.2 -m multiport --port 53,5353 -j DSCP --set-dscp 26
iptables -t mangle -A FORWARD -p udp -d 192.168.1.2 -m multiport --port 53,5353 -j DSCP --set-dscp 26
iptables -t mangle -A FORWARD -p tcp -d 192.168.1.2 -m multiport --port 53,5353 -j DSCP --set-dscp 26

##firewall
iptables -A FORWARD -s 192.168.1.2 -j ACCEPT
iptables -A FORWARD -d 192.168.1.2 -j ACCEPT

##resetlinks
iptables -t filter -A FORWARD -m conntrack --ctstate established,related -m dscp ! --dscp 0 -j ACCEPT

#tcpdump
#tcpdump -i br-lan and udp and portrange 1-65535 and !port 53 and ! port 80 and ! port 443 -vv -X -w /root/cap-name.pcap
 
Powered by LuCI openwrt-18.06 branch (git-19.020.41695-6f6641d) / OpenWrt 18.06.2 r7676-cddd7b4c77

I'm running into dropped packets abnormally. I am confident this is a TCP connection issue is there any way to clean up the 1506kb TCP connection using hashlimit and conntrack. This is my current ruleset and it is not working the switch from advancedtomato to openwrt cake cleaned up the UDP transmissions but it just doesnt transmit and receive to the game servers like video, it transmits like a .gif looking stutter when I input user actions and it has been happening consistently only during any type of my inputs. One thing i've noticed in Wireshark is that there are 0 UDP issues occuring they are all TCP.

I have sqm configured to allow for 1.5mb because thats what my tcp is reading in wireshark, and my udp is generally around 770kb avg. So I just opened up my sqm for 3 mb to account for 1.5mb packets each way.

The config I want to use is:
EF/46 for all protocols for my gaming rig
MED/26 for DNS
LOW/0 for all lan traffic

And basically this bit:

iptables -t mangle -A FORWARD -p udp -d 192.168.1.2 -m hashlimit --hashlimit-name gamingudp --hashlimit-above 220/sec --hashlimit-burst 100 --hashlimit-mode srcip,srcport,dstip,dstport -j CONNMARK --set-mark 0x46
iptables -t mangle -A FORWARD -p udp -d 192.168.1.2 -m connmark ! --mark 0x46 -m multiport ! --ports 53,5353,60887 -m connbytes --connbytes 0:700 --connbytes-dir both --connbytes-mode avgpkt -j DSCP --set-dscp-class CS5

The main point here is there is no other upload traffic on the lan whatsoever so my game traffic should be transmitting fine but of course it is hitting congestion. Now when I look at hashlimit and connmark I don't feel like this applies to my config because I can just literally cs0 everything and cs5 my rig by ip/mac -so do I have this in there for nothing?

Another question is since my ISP doesn't require the VLAN do I need to use VLAN? Could that be a problem, but then my wifi would not be able to be used right?

The dsl line should be clean with very little local traffic day or night - none of the other lines would be torrenting so this is not an issue either. Right now I am using a zyxel dsl modem instead of the Asus because for some reason the Asus will knock out the dsl line and I have to get a tech out to the dslam just to get my signal back up even though I configure them similarly.

Now I have cake/layer cake on egress for interface on eth1.2(wan, wan6 - not on the pppoe-wan tunnel interface) and cake/piece of cake on egress for interface eth0.1(lan - not on the br-lan bridge). So I begin to adjust my sqm settings and this was not a problem because there was just no changes on one sqm (one egress+ingress) vs. 2 sqm(wan egress layer cake, lan egress piece of cake). The numbers were adjusted to allow for 33% or 95% sqm headroom and there was no change. No change for the sqm type(cake vs. layercake) on LAN egress, keeping layercake on WAN egress for DSCP marking. I could not or as dlakelan has stated it is impossible to vethpair on OpenWRT and it involves using Debian linux coding - I looked into that and for my Linksys32xx it is literally a page where you copy the code and build your entire modem config using linux and it is a complete mess. It would just end bad. I would not even be able manage the scripts if I had a problem, especially if I brick it over and over without knowing why. I cannot use linux without knowing excatly what would cause the brick if I even manage to recover the $250 device -and I imagine I will be soldering some special part in in the middle of all this and i'm sure that there are other devices running this type of user interface with a better support system on a different modem but I have this modem and OpenWRT.

WAN above, LAN below

WAN

LAN

LAN w/dual-dshost ingress and ECN enabled

VDSL2 no VLAN from ISP config

Switches

Wireshark

One >1 min fortnite session:
Statistics

Measurement	Captured	Displayed	Marked
Packets	18630	18630 (100.0%)	—
Time span, s	425.171	425.171	—
Average pps	43.8	43.8	—
Average packet size, B	294	294	—
Bytes	5473387	5473387 (100.0%)	0
Average bytes/s	12 k	12 k	—
Average bits/s	102 k	102 k	—

Interfaces: pppoe-wan/wan/wan6, br-lan(lan,wireless)

PC(capable of sustaining 144fps) > 5 ft cat5e > Zyxel/ASUS DSL68u(new firmware)> 5 ft cat 5e > Linksys 3200ACM(stock firmware)/ OpenWRT install(50down/10up) > 6 ft cat 5e phoneline to jack > 25 ft cat5e line > ~1/4 mile to ISP node(cable and DSL functionality)

cFos doesn't do anything I can't do in windows and the VyperVPN gave me frame drops. I do not have hardware limitations but rather some type of linux issue to adjust the code or look for other avenues to stop the stuttering/congestion. Now essentially it is a transmission issue which I feel like can be remedied through the OpenWRT firewall config. pls help

Main picture is a streamer with fiber/vdsl line im not sure but 32 ping
Picture in picture is mine about 64 ping

[shm0]

Switched back to simple.qos and fq_codel to test somethings.
With the default hard limit of 1001 packets, my router is unable to maintain the full shaping bandwidth.
It reaches the limit of ~360 Mbit/s and then falls down to ~320 Mbit/s.
With the default hard limit of 10240 there no such "problem".
Latency increase is also still acceptable 10 to 20ms increase.
But 1000 packets limit should be enough even for Gbit speeds?
What is going on there?

[warchief]

like with cake in general or openwrt in general

OHHHHH in like a custom rule lol

[moeller0]

@shm0

But 1000 packets limit should be enough even for Gbit speeds?
What is going on there?

(1000 * 2048)/1000^2 = 2.048 MB, multiply by ten and it becomes clear that for a 32 or even 64 MiB router such a queue size will easily lead to OOM. Therefore sqm takes the default down by an order of magnitude. If your router has more than 32 MiB of RAM I would just increase the limit, to get the bandwidth back, hopefully while still researching what is going on.
Tl;Dr: SQM uses limit as a safety to avoid OOM and it is taylored to the smallest devices running OpenWrt when sqm was developed, it should have no bad effects to increase this number. A total of 1001 packets however should work for Gbps speeds:
(1001 * 1538 * 8) / 1000^3=12 milliseconds
A 1001 packet buffer will take 12 Ms to fill at Gbps Ethernet speed, so either you have lots of small packets clogging that or the service interval exceeds 12 Ms...
Time to consider switching sqm from using limit to using memory_limit/mem_limit for this safety feature...

[shm0]

Thanks for the detailed answer again
What about quantum?
Especially for the single queue scripts (simplest, piece of cake), isn't better to set a lower quantum in general?
As I understand this, using a lower quantum makes the qdisc favor smaller packets and most latency sensitive traffic uses small packets, in most cases.

[moeller0]

By now we auto-scale quantum based on bandwidth. In theory the smaller the quantum the less lumpy the fairness is distributed so a quantum of say 100B would be desirable. But this comes at a computational price which at higher bandwidths becomes too steep for most user's taste.
In recent sqm-scripts we allow the user to set the acceptable lumpiness for both quantum and burst in microseconds (we default to using the burst value also for quantum, but that can be overridden in /usr/lib/sqm/defaults.sh). The idea is that you now configure the maximally acceptable additional delay caused by this (this is mostly for burst, but also applies to quantum). Hope this helps.

P.S.: cake will automatically select both quantum and its own burst equivalent.

[shm0]

I don't know if I understand the burst value correctly.
Does it mean, by default, no packets drops are allowed for 1 ms?

Looking at the draft for docsis pie:
https://tools.ietf.org/html/draft-ietf-aqm-docsis-pie-02
I know Pie != fq_codel != cake
But:

target: 10ms
interval: 16ms
max burst: 142 ms
mean packet size: 1024 (quantum?)

And it works actually quite good.

I have one more question.
On a device with one or more cpu ports (eth0, eth1).
And, for example, eth0 has the following sub-interfaces:
lan1 lan2 lan3 lan4 wan.
If now, a qdisc like fq_codel is attached to the cpu port (eth0, eth1).
Will this also affect traffic flowing between, for example, lan1 and lan2? I guess yes?
Will it also affect traffic flowing between lan1-4 to wan? I guess yes too?
So what are good settings here for 1Gbit speeds? target 50us interval 1ms? (cake lan preset?)

There is a fq_codel Implementation. What are the defaults there?
target 35ms interval 150ms?

[moeller0]

So burst is a feature of HTB, not of fq_codel. Effectively this variable describes the maximum amount of data htb is willing to release into the next layer when it runs. If HTB is guaranteed to always run just before the deadline for sending the next packet a buffer of one packet would be enough, independent of packet size. This is computationally expensive and will overtax a small router's CPU pretty quickly, resulting (for HTB/fq_codel) in an effective shaper bandwidth well below the shapers set rate. Increasing the burst buffer effectively allows HTB to maintain the shaper rate at a much lower CPU cycle cost.
We recently changed upstream sqm-scripts to default to a burst-buffer size equivalent to 1 millisecond service time or at least a full MTU packet +overhead at the configured shaper rate, this effectively bounds the latency increase due to the burst buffer in a way that the user can immediately see the worst case latency price.
DOCSIS pie, which only works for the upstream is in a much easier position, as it's bandwidth effectively will be at most in the 50-100 Mbps range, it uses queue size instead of the more costly sojourn time (avoid running a high resolution time function often was one of the design goals for DOCSIS pie IIRC) and it is linked with the DOCSIS egress "MAC" such that it will only be called if the MAC requires new packages if I recall correctly. As a consequence the docsis-pie numbers will not help that much in the generic case with a software rate limiter like HTB. Does this help?

The fq_codel defaults are interval 100ms and target 5ms, these will be active on all interfaces. Setting interval much lower will only work if all that nterface sees is local traffic, once internet traffic also hits the interface interval 100 is a sane default. Target should be 5-10% of interval, for tcp-friendly congestion controllers.

[shm0]

Thanks.

Always thought target defines the maximum acceptable delay of a packet in the queue.
But this is not the case? It is actually the minimum delay?
And the maximum delay is actually defined by the interval parameter?
So with the default values, the worst-case delay can be 104.9... ms?
And setting interval = worst-case RTT makes sense?
Waiting for an entire RTT seems a bit long.
And isn't it already too late then to start dropping/marking packets?

[moeller0]

@shm0

Always thought target defines the maximum acceptable delay of a packet in the queue.
But this is not the case? It is actually the minimum delay?

If for the duration of interval the minimal experienced sojurn time of packets in the queue exceeds target the next drop is scheduled, in a sense target defines the maximally acceptable standing queue.

@shm0

And the maximum delay is actually defined by the interval parameter?

Not really. I tend to think as interval as the time codel/fq_codel give a flow to respond to a signal; it takes one RTT for the signal to the receive and back to the sender and from there as reduced sending rate back to the codel/fq_codel instance that dropped the packet. Itt is the sender that needs to change its bahavior. Dropping more than one packet and not waiting until the sender actually has a chance to react seems counter productive especially given tat the sender will respond with halving its congestion window (effectively halving its sending rate).

RFC 8289 (https://tools.ietf.org/html/rfc8289#page-14) has sections on both parameters that also show that target is best understood/set as a fixed fraction of interval.

@shm0

So with the default values, the worst-case delay can be 104.9... ms?

The worst case can be actually worse than that, but on average the standing queue will approximate target (so with bi-directional shapers the sum of target(ingress) and targer(egress)).

@shm0

And setting interval = worst-case RTT makes sense?

It turns out, interval does not need to be too exact as long as the order of magnitude (power of 2) is roughly correct things will just work, so no need to aim for worst-case RTT.

@shm0

Waiting for an entire RTT seems a bit long.

Well, this is the earliest time the senders changed behavior will reach the codel instance and effectively telling the flow to half its sending rate is rather drastic, so codel can afford to wait a bit for gentler shaping.
But the codel RFC has more details and more importantly was written by people who actually understand things unlike me.

[warchief]

You guys are discussing this? Would you also tell what they're purpose is and what their values mean such as this gentleman here?

SQM, Flow Offloading, VLAN Tagging and Gaming

In theory, requesting low latency from AQM/SQM should result in small buffers and reduced bufferbloat. However, some interval settings should be more than your average RTT, if I've understood correctly. In addition, if you uplink is limited (e.g. ADSL) I think that your uplink (egress) traffic should not use ECN in any situation because it's better to drop the packet early instead of trying to push it through the limited uplink. On the other hand ECN should be okay for downloads (ingress) in all…

[warchief]

Also the issue I have with Fortnite I have come to conclude is that I am CPU limited due to Fortnite making use of 6 cores better than 4. Which is why just Fortnite lagged no other games.

Ultimate SQM settings: Layer_cake + DSCP marks

[openwrt1] [openwrt2] # This file is interpreted as shell script. # Put your custom iptables rules here, they will # be executed with each firewall (re-)start. # Internal uci firewall chains are flushed and recreated on reload, so # put custom rules into the root chains e.g. INPUT or FORWARD or into the # special user chains, e.g. input_wan_rule or postrouting_lan_rule. # This file is interpreted as shell script. # Put your custom iptables rules here, they will # be executed with each firew…

[shm0]

Thanks again

[moeller0]

tcMTU only matters if you use HTB/fq_codel and the tc_stab link layer adjustment method, cake does not care for this.
MPU however is evaluated by both cake and HTB/fq_codel (in not too old sqm-scipts) this is basically the minimal size of a packet. All encapsulations that re-package ethernet frames will have at least the ethernet minimum of 64 bytes here, often a bit more.

[warchief]

also what these purposes are and what their values mean

[moeller0]

Well, limit sets a limit on how many packets total the qdisc will accept. To deal with low memory routers we default this to 1001 packets. If your router has more than say 64 MB ram you could increase that number. Just note by default each packet will require a full skb (socketbuffer) requiring 2048 Bytes of memory (same size for a 64B packet as for a 1500B one).

The latency target is related to codel/fq_codel, see https://tools.ietf.org/html/rfc8289#page-14 for details. Effectively, target needs to be >= the time required to send one packet through the link. The default target of 5ms will get into troubles once a links rate gets lower than around 2 Mbps, automatic selection will set target to be the maximum of the default 5ms or the time required to send 1.5 full MTU packets.

The advanced option strings are just there so that users can pass on options to the selected qdisc that are not already handled in sqm's GUI (like setting per internal IP-address fairness modes).

Hope that helps.

BTW the OpenWrt documentation has some pages on SQM that are worth reading....

[Happi]

is there any update on any scripts to further improve the Latency/Bufferbloat? please do share, am currently trying to improve my WRT3200, i have Comcast/Xfinity cable internet, speed around 60 Down, 5 Up. please do share some scripts that i could try and play around with, i usually play around with the settings to see how far i could push it.

[warchief]

iptables firewall scripts for established/related connections generally have a positive effect

[Happi]

can you post some scripts to try like @hisham2630 did? i tried several scripts and still not able to get the best possible connection.

[warchief]

Post your config and values!