Ultimate SQM settings: Layer_cake + DSCP marks

Thanks for the info.
So in diffserv4, if i reached the 25% of bandwidth in voip tin, then packets will go to video tin?, i just think
it's better to borrow a none used bandwidth from the next tin?!

Why i should have ton of rules, if one or two rules can do everything, Like using connbytes to detect downloads it's working great and can stop torrent hogging the bandwidth!.

Most of game operators use TCP SYN to measure latency, they block ICMP to save their servers from DDOS
etc.
While they can use udp to measure latency; As you said:

I think it's OK, it's just a number, i need to test for a few days, i play another game which the ping is low and stable, because it's use UDP to measure latency, also on the same ip for the game session.

Yeah, it's a robust like the bulk rule that use connbytes, i completely removed port based prioritizing for
the realtime traffic.
Thank you for the valued help and diagnosing .

Yeah sure, but let me clean it first, wait a few minutes!


########################################
# Latency Sensitive (gaming/voip)
########################################
##ICMP, to prioritize pings
$IPT -t mangle -A PREROUTING -p icmp -j DSCP --set-dscp-class CS5 -m comment --comment "ICMP-pings"

#DNS traffic both udp and tcp
$IPT -t mangle -A PREROUTING -p udp -m multiport --port 53,5353 -j DSCP --set-dscp-class CS5 -m comment --comment "DNS udp"
$IPT -t mangle -A PREROUTING -p tcp -m multiport --port 53,5353 -j DSCP --set-dscp-class CS5 -m comment --comment "DNS tcp"

#give a high priority for PS4 and xbox
$IPT -t mangle -A PREROUTING -p tcp -m conntrack --ctorigsrc 192.168.1.150 -m multiport ! --ports 80,443 -j DSCP --set-dscp-class CS6 -m comment --comment "PS4" #for PS4 or xbox etc, change the ip according to your ip setting.
$IPT -t mangle -A PREROUTING -p udp -m conntrack --ctorigsrc 192.168.1.150 -m multiport ! --ports 80,443 -j DSCP --set-dscp-class CS6 -m comment --comment "PS4"

#High priority ipset, i use for pubgM
$IPT -t mangle -A PREROUTING -p udp -m set --match-set latsens src -j DSCP --set-dscp-class CS6  -m comment --comment "latency sensitive ipset" ## set dscp tag for Latency Sensitive (latsens) ipset

#A robust 2 rules to detect realtime traffic
$IPT -t mangle -A PREROUTING -p udp -m hashlimit --hashlimit-name udp_high_prio --hashlimit-above 120/sec --hashlimit-burst 50 --hashlimit-mode srcip,srcport,dstip,dstport -j CONNMARK --set-mark 0x55 -m comment --comment "connmark for udp"
$IPT -t mangle -A PREROUTING -p udp -m connmark ! --mark 0x55 -m conntrack --ctorigsrc 192.168.1.0/24 -m multiport ! --ports 53,5353,80,443,8080,60887 -m connbytes --connbytes 0:700 --connbytes-dir both --connbytes-mode avgpkt -j DSCP --set-dscp-class CS6 -m comment --comment "small udp connection"

##################
#TCP ACK flows
##################
# prioritize inbound and outbound ACK pkts according to size
# empty ipv4 ack pkts <64 appear very often but empty ipv6 ack pkts are larger and fall into smaller priority class
$IPT -t mangle -A PREROUTING -p tcp -m tcp --tcp-flags ALL ACK -m length --length :64 -j DSCP --set-dscp-class CS4
$IPT -t mangle -A PREROUTING -p tcp -m tcp --tcp-flags ALL ACK -m length --length 65:89 -j DSCP --set-dscp-class CS4

#mid size ACK
$IPT -t mangle -A PREROUTING -p tcp -m tcp --tcp-flags ALL ACK -m length --length 90:159 -j DSCP --set-dscp-class CS3
$IPT -t mangle -A PREROUTING -p tcp -m tcp --tcp-flags ALL ACK -m length --length 160:255 -j DSCP --set-dscp-class CS3

# large ack pkts carry data and dont need the same priority
$IPT -t mangle -A PREROUTING -p tcp -m tcp --tcp-flags ALL ACK -m length --length 256:511 -j DSCP --set-dscp-class CS2
$IPT -t mangle -A PREROUTING -p tcp -m tcp --tcp-flags ALL ACK -m length --length 512:1500 -j DSCP --set-dscp-class CS2
$IPT -t mangle -A PREROUTING -p tcp -m tcp --tcp-flags ALL ACK -m length --length :1500 -j DSCP --set-dscp-class CS2

#Make sure ACK packets get priority (to avoid upload speed limiting our download speed)
$IPT -t mangle -A PREROUTING -p tcp -m length --length :128 --tcp-flags SYN,RST,ACK ACK -j DSCP --set-dscp-class CS3
########################################
# Streaming Media (videos/audios)
########################################
#Know video streams sites like youtube,netflix
$IPT -t mangle -A PREROUTING -m set --match-set vidstream src -j DSCP --set-dscp-class AF41 -m comment --comment "vidstream ipset"

##set dscp tag for our ipset usrcdn #those cdn's have an isolated speeds by my ISP
$IPT -t mangle -A PREROUTING -m set --match-set usrcdn src -j DSCP --set-dscp-class AF21 -m comment --comment "usrcdn ipset"

# some iptv provider's use this port
$IPT -t mangle -A PREROUTING -p tcp -m multiport --dport 1935 -j DSCP --set-dscp-class AF41 -m comment --comment "some iptv streaming service"

########################################
# Best Effort (browsing/services)
########################################
#rule to make browsing, games login faster, for both tcp and udp
$IPT -t mangle -A PREROUTING -p tcp -m multiport --ports 80,443,853,3455,8080,27015 -j DSCP --set-dscp-class CS3 -m comment --comment "priority for faster browsing-tcp"
$IPT -t mangle -A PREROUTING -p udp -m multiport --ports 80,443,853,3455,8080 -j DSCP --set-dscp-class CS3 -m comment --comment "priority for faster browsing-udp"

#########################################
# Background Traffic (Bulk/file transfer)
#########################################
#bulk traffic ipset, like windows udates and steam updates/downloads
$IPT -t mangle -A PREROUTING -m set --match-set bulk src -j DSCP --set-dscp-class CS1 -m comment --comment "bulk traffic ipset"

#A robust rule to detect downloads/uploads and torrents!
$IPT -t mangle -A PREROUTING -p tcp -m multiport --ports 21,80,443,8080,60887,27014:27050 -m connbytes --connbytes 1048576: --connbytes-dir both --connbytes-mode bytes -j DSCP --set-dscp-class CS1 #60887 bittorrent
$IPT -t mangle -A PREROUTING -p udp -m multiport --ports 21,80,443,8080,60887 -m connbytes --connbytes 1048576: --connbytes-dir both --connbytes-mode bytes -j DSCP --set-dscp-class CS1

#tcpdump rule, copy and paste this rule into terminal, this rule is used to capture realtime traffic, you can change ip to what you like
#tcpdump -i br-lan host 192.168.1.126 and udp and portrange 1-65535 and !port 53 and ! port 80 and ! port 443 -vv -X -w /root/cap-name.pcap

Most of these seem reasonable, there are a few where I imagine you could go both directions rather than just download direction for example (matching ipset src rather than dest). I do think probably some of your rules overlap each other, like your ACK rules might set TCP connections to CS2 and then later your streaming media rules might reset it to AF41 or whatever. That might be something you want it to do, but you might look at that to see if the interaction of rules is what you want.

Overall, I think this is a pretty robust set of rules and together with an appropriate mapping to buckets should work well. If you have remaining latency issues with VOIP or games it's probably either inherent issues (ie. problems between you and the server) or transient over-committing to high priority buckets, like if your PS4 suddenly starts updating some software over port 8080 or some other weird port and that all gets prioritized CS6

1 Like

i fixed that by using src,dst.

Still not sure which one's is overlap with each other's, it's a tcp flag, so not sure if it will make a problem
with streaming media. i think it's fine if streaming media reset it to AF41.
Hahha, i'm not sure what i want to do and what i don't want to do!!!
Sometimes interaction is not bad!?

Yeah it's still working fine for all cases until now, it happens sometimes there's a problem between me and
server , duo to loaded isp routers/peering issues.
I don't have PS4 or any console so i can't test that behavior!
I exclude those bulky ports like(80,443,8080,torrent port), sometimes when i start a download a game ping
will go up by 20ms for 2 sec then back normal, i think this is because i have a small bandwidth.
at early morning my speed will go up to 8mbps, browsing is much faster and there's no ping jump while
initiate a download, but in general i tried running a download and uploaded while during a game session
and there's no lag or ping jump at all!
I will keep maintain this script, and let you know; if you have any ideas or suggestions please share it.

That's great. I definitely had a similar improvement when I set up my QoS for my voip server, and recently I had my voip server cloud kernel changed by the VPS managers and they left out the HFSC qdisc, so reboot made me lose the QoS things went terrible, so I fixed it by using my own kernel and continuing to use my QoS system. The point is, even when you have hundreds of megabits of bandwidth, it's still perfectly possible to get terrible realtime performance for VOIP or games unless you make some effort to tag/prioritize/queue properly.

1 Like

Bandwidth division for diffserv8 is:

+static int cake_config_diffserv8(struct Qdisc *sch)
+{
+/*	Pruned list of traffic classes for typical applications:
+ *
+ *		Network Control          (CS6, CS7)          ----> 39.27%
+ *		Minimum Latency          (EF, VA, CS5, CS4)  ----> 44.87%
+ *		Interactive Shell        (CS2, TOS1)         ----> 51.29%
+ *		Low Latency Transactions (AF2x, TOS4)        ----> 58.61%
+ *		Video Streaming          (AF4x, AF3x, CS3)   ----> 66.99%
+ *		Bog Standard             (CS0 etc.)          ----> 76.56%
+ *		High Throughput          (AF1x, TOS2)        ----> 87.5%
+ *		Background Traffic       (CS1)               ----> 100%
+ *
+ *		Total 8 traffic classes.
+ */

@dlakelan
i'm planing to change my tags like that, correct me if it's wrong:

  1. CS6 for voip/gaming
  2. CS5 for dns and icmp
  3. CS2 some tcp flags and
  4. AF21 another set of tcp flags
  5. AF31 for youtube and other streaming
  6. CS0 for none prioritized traffic, also mail ports?!
  7. AF11 internet browsing and games login?
  8. CS1 for downloads and torrents
    Having a look at Cisco DSCP values https://community.cisco.com/kxiwq67737/attachments/kxiwq67737/discussions-collaboration-apps/4531/1/dscpphb_values.jpg

so, all this discussion is great...

but, has anything been distilled down to normal jargon I could learn from?

like...
yea download this package aside from SQM and do this commands, better net!

1 Like

You can read here:


@dlakelan
I found something interested about my wifi Accidentally !

                            BE         BK        VI        VO

MPDUs Queued:             3006        439         2      4354
MPDUs Completed:          2477        413         4      4746
MPDUs XRetried:            912        171         0        42
Aggregates:              24664      23446         0         0
AMPDUs Queued HW:            0          0         0         0
AMPDUs Completed:       196461     155442        36         0
AMPDUs Retried:          36099       6076         5         0
AMPDUs XRetried:         12035       1822         2         0
TXERR Filtered:            196         15         0         0
FIFO Underrun:               0          0         0         0
TXOP Exceeded:               0          0         0         0
TXTIMER Expiry:              0          0         0         0
DESC CFG Error:              0          0         0         0
DATA Underrun:               0          0         0         0
DELIM Underrun:              0          0         0         0
TX-Pkts-All:            211885     157848        42      4788
TX-Bytes-All:        250282717  199528107      4040    996801
HW-put-tx-buf:          160262      72233        47      4788
HW-tx-start:                 0          0         0         0
HW-tx-proc-desc:        160262      72174        47      4788
TX-Failed:                   0          0         0         0

(VO):  qnum: 3 qdepth:  0 ampdu-depth:  0 pending:   0
(VI):  qnum: 2 qdepth:  0 ampdu-depth:  0 pending:   0
(BE):  qnum: 1 qdepth:  0 ampdu-depth:  0 pending:   0
(BK):  qnum: 0 qdepth:  0 ampdu-depth:  0 pending:   0
(CAB): qnum: 8 qdepth:  0 ampdu-depth:  0 pending:   0

So nothing for the common man here lol

This is really more an experimental thread, nothing finalized into a "package" really. If you install and set up SQM with very basic settings then bam... Better network. If you want to go beyond that, requires some experiments, which is what this thread is about

1 Like

Stats on wmm usage are interesting how did you get that?

hehehe, too simple

 cat /sys/kernel/debug/ieee80211/phy1/ath9k/xmit
 cat /sys/kernel/debug/ieee80211/phy1/ath9k/queues

and this one for errors and frags:

cat /sys/kernel/debug/ieee80211/phy1/ath9k/recv

can you have a look at post

Diffserv8 may have 8 queues, but that doesn't mean you need to use all of them. I'd limit it to

  1. Realtime stuff, tagged CS6 (VOIP, games, NTP)
  2. Low Latency small stuff tagged CS5 (flags, DNS)
  3. Video Streaming tagged AF41 (this matches with WMM Video queue)
  4. Best Effort Stuff tagged CS0 (goes into BE WMM queue)
  5. Background CS1 (Goes into BK WMM queue)

Beyond that trying to use other tags will wind up getting you conflicts between cake and WMM and some cheap managed switches and various other things that might use DSCP, one thing will treat say CS2 as prioritized, another one will treat it as Background, etc.

1 Like

Thanks.
I see that when 2 clients connected to the same lan, assume client A call client B using viber voice call app
viber will tag packets with AF41 for both outbound and inbound, also the call is p2p using the private ip of
those 2 clients.
My ISP is tagging NTP packets with AF41 ?!

Since there are no hard standards each app or provider uses their own system, it's a bit of a mess. But af41 is reasonable as a tag for low latency data. It won't go into voice WMM queue though

1 Like

I saw there's something wrong, can you catch it?! @dlakelan Screenshot%20(2)
Screenshot%20(5)
Screenshot%20(6)
There's an interested things here

Hi again guys! Perhaps you can help me extend this setup. (if any1 has any other tips to optimize tcp over openvpn be my guest)

I'm trying to add OpenVPN to this for my game and im planning to route the traffic from my windows through the OpenVPN client on OpenWRT by destination IP, or policy based routing)

I tried Windows VPN clients with split-tunneling to try and only route specific apps through VPN but it seem mostly dirty fixes with firewall rules i believe, and just feels dirty.

Currently the OpenVPN works (tun0) and connection established, turned off nagling with tcp-nodelay.

The default routes work to the clients depending whether i use 'route-nopull' or not, obviously use it as i want to do some manual route.

But first I gotta get the Tun0 to work with the veth setup so also ingress from tun0 goes through veth!

Thank you!

with a vpn you'll have two kinds of ingress and two kinds of egress, one "into / out of the tunnel" and one the actual encrypted packet sent out or in on ethernet. If you prioritize the unencrypted (pre/post tunnel) packet, but the encrypted packet waits... you won't get the prioritization you want. It's a bit tricky because you can't look at the encrypted packet at all so likely you want to restrict the tunnel to just gaming, and prioritize all the encrypted packets.

also, if you can use a VPN provider who uses wireguard, it is a better choice due to lower CPU requirements for the router.

Yep, that will be hard to prioritize inside tunnel too, but... the VPN is intended for solely one game! I know the IP destination range, but im having trouble setting it up in the same veth setup as @hisham2630

So at least the tunnel should not be hard to prioritize now and I can just prioritize the whole port of my choosing, i think? (Currently just port 443, but I can change to a different one later)

If I can just have both Eth1.2 (WAN) and Tun0 both let egress go over Veth0/1, and the tunnel sends egress back the regular route to Eth1.2 (WAN)

Or is this too simple thinking?

Edit: I think I already managed it working, but not sure if OK. When I do speedtest (Download) i can see equal Tun0 and Veth data flow!

What i did:

ip route add default dev veth0 table 100
ip rule add iif eth1.2 table 100 priority 100
ip rule add iif tun0 table 100 priority 100

Just added the last line. I use this package to specify route atm the easy way: