Yes there were some large packets in your VoIP session but the avg was still quite small, it's probably some rekeying or changing codec parameters or similar packets... What matters is more the limited packet rate and the avg size, that's what identifies realtime flows. When transferring bulk bytes packet size will max out and rate will increase to try to fill the bandwidth. Realtime just sends whatever new limited info is generated in the most recent time interval, so maintains a relatively steady rate and size on avg.
1 Like
I set up a similar rule using nftables, it marks flows with small udp packets using a similar technique. I use over 200 packets per second and 100 packet burst as my threshold (but I have a fast connection so I can afford it).
So far it's caught several tens of thousands of packets and seems to be working.
1 Like
Congratulations and happy new year for you and others.
It's work better than port based tagging, but for some games they use tcp to measure the latency, so should i
use the same rule for tcp.
So here is the comparable output for diffserv8:
sudo tc qdisc replace root dev eth0 cake diffserv8 overhead 34 mpu 64 bandwidth 100mbit ; tc -s qdisc
[...]
qdisc cake 8003: dev eth0 root refcnt 6 bandwidth 100Mbit diffserv8 triple-isolate nonat nowash no-ack-filter split-gso rtt 100.0ms noatm overhead 34 mpu 64
Sent 60176 bytes 316 pkt (dropped 0, overlimits 102 requeues 0)
backlog 0b 0p requeues 0
memory used: 9984b of 5000000b
capacity estimate: 100Kbit
min/max network layer size: 102 / 282
min/max overhead-adjusted size: 136 / 316
average network hdr offset: 10
Tin 0 Tin 1 Tin 2 Tin 3 Tin 4 Tin 5 Tin 6 Tin 7
thresh 100Mbit 87500Kbit 76562Kbit 66992Kbit 58618Kbit 51290Kbit 44879Kbit 39269Kbit
target 5.0ms 5.0ms 5.0ms 5.0ms 5.0ms 5.0ms 5.0ms 5.0ms
interval 100.0ms 100.0ms 100.0ms 100.0ms 100.0ms 100.0ms 100.0ms 100.0ms
pk_delay 18.4ms 0us 154.9ms 0us 0us 0us 0us 0us
av_delay 402us 0us 8.8ms 0us 0us 0us 0us 0us
sp_delay 1us 0us 1us 0us 0us 0us 0us 0us
backlog 0b 0b 0b 0b 0b 0b 0b 0b
pkts 124 0 189 2 0 0 0 1
bytes 20887 0 39163 84 0 0 0 42
way_inds 0 0 0 0 0 0 0 0
way_miss 2 0 10 1 0 0 0 1
way_cols 0 0 0 0 0 0 0 0
drops 0 0 0 0 0 0 0 0
marks 0 0 0 0 0 0 0 0
ack_drop 0 0 0 0 0 0 0 0
sp_flows 0 0 1 0 0 0 0 1
bk_flows 1 0 0 0 0 0 0 0
un_flows 0 0 0 0 0 0 0 0
max_len 1078 0 1146 42 0 0 0 42
quantum 1514 1514 1514 1514 1514 1514 1369 1198
[...]
I just want to note that cake will overflow packets into lower priority cans/tiers if one exceeds the maximum bandwidth. Also I have no clue how much real-world exposure diffserv8 has gotten.
IMHO you are close to overdoing it, QoS schemes can become quite brittle if one piles up too many rules 
(I guess this is due to the fact that is easy to loose track of the subtle interactions between the different marking heuristics, now, both you and @dlakelan seem careful and methodological enough to avoid that trap, but in general it seems easy to get hooked on too many bespoke qos-rules).
Maybe you should have a look at the precedence scheme, which is much less forgiving (lower tiers are only serviced if no higher tier has packets to send) but also a lot more predictable...
Happy New Year!
1 Like
I don't think so. You don't want to prioritize every small TCP flow because there could be a lot of them, and TCP is usually used only for transferring non-realtime traffic. Measuring latency with TCP is basically laziness I think. As long as the game play itself works well, I think you're ok. If the game play suffers, perhaps you can prioritize just those specific game server destinations or something?
I think this is actually an advantage of identifying the realtime flows via their rate/size behavior since it's a robust rule that captures most of the traffic you want and you don't have too many rules to keep track of.
@hisham2630 can you post the current rule set (just the iptables rules, not the whole script)
Thanks for the info.
So in diffserv4, if i reached the 25% of bandwidth in voip tin, then packets will go to video tin?, i just think
it's better to borrow a none used bandwidth from the next tin?!
Why i should have ton of rules, if one or two rules can do everything, Like using connbytes to detect downloads it's working great and can stop torrent hogging the bandwidth!.
Most of game operators use TCP SYN to measure latency, they block ICMP to save their servers from DDOS
etc.
While they can use udp to measure latency; As you said:
I think it's OK, it's just a number, i need to test for a few days, i play another game which the ping is low and stable, because it's use UDP to measure latency, also on the same ip for the game session.
Yeah, it's a robust like the bulk rule that use connbytes, i completely removed port based prioritizing for
the realtime traffic.
Thank you for the valued help and diagnosing .
Yeah sure, but let me clean it first, wait a few minutes!
########################################
# Latency Sensitive (gaming/voip)
########################################
##ICMP, to prioritize pings
$IPT -t mangle -A PREROUTING -p icmp -j DSCP --set-dscp-class CS5 -m comment --comment "ICMP-pings"
#DNS traffic both udp and tcp
$IPT -t mangle -A PREROUTING -p udp -m multiport --port 53,5353 -j DSCP --set-dscp-class CS5 -m comment --comment "DNS udp"
$IPT -t mangle -A PREROUTING -p tcp -m multiport --port 53,5353 -j DSCP --set-dscp-class CS5 -m comment --comment "DNS tcp"
#give a high priority for PS4 and xbox
$IPT -t mangle -A PREROUTING -p tcp -m conntrack --ctorigsrc 192.168.1.150 -m multiport ! --ports 80,443 -j DSCP --set-dscp-class CS6 -m comment --comment "PS4" #for PS4 or xbox etc, change the ip according to your ip setting.
$IPT -t mangle -A PREROUTING -p udp -m conntrack --ctorigsrc 192.168.1.150 -m multiport ! --ports 80,443 -j DSCP --set-dscp-class CS6 -m comment --comment "PS4"
#High priority ipset, i use for pubgM
$IPT -t mangle -A PREROUTING -p udp -m set --match-set latsens src -j DSCP --set-dscp-class CS6 -m comment --comment "latency sensitive ipset" ## set dscp tag for Latency Sensitive (latsens) ipset
#A robust 2 rules to detect realtime traffic
$IPT -t mangle -A PREROUTING -p udp -m hashlimit --hashlimit-name udp_high_prio --hashlimit-above 120/sec --hashlimit-burst 50 --hashlimit-mode srcip,srcport,dstip,dstport -j CONNMARK --set-mark 0x55 -m comment --comment "connmark for udp"
$IPT -t mangle -A PREROUTING -p udp -m connmark ! --mark 0x55 -m conntrack --ctorigsrc 192.168.1.0/24 -m multiport ! --ports 53,5353,80,443,8080,60887 -m connbytes --connbytes 0:700 --connbytes-dir both --connbytes-mode avgpkt -j DSCP --set-dscp-class CS6 -m comment --comment "small udp connection"
##################
#TCP ACK flows
##################
# prioritize inbound and outbound ACK pkts according to size
# empty ipv4 ack pkts <64 appear very often but empty ipv6 ack pkts are larger and fall into smaller priority class
$IPT -t mangle -A PREROUTING -p tcp -m tcp --tcp-flags ALL ACK -m length --length :64 -j DSCP --set-dscp-class CS4
$IPT -t mangle -A PREROUTING -p tcp -m tcp --tcp-flags ALL ACK -m length --length 65:89 -j DSCP --set-dscp-class CS4
#mid size ACK
$IPT -t mangle -A PREROUTING -p tcp -m tcp --tcp-flags ALL ACK -m length --length 90:159 -j DSCP --set-dscp-class CS3
$IPT -t mangle -A PREROUTING -p tcp -m tcp --tcp-flags ALL ACK -m length --length 160:255 -j DSCP --set-dscp-class CS3
# large ack pkts carry data and dont need the same priority
$IPT -t mangle -A PREROUTING -p tcp -m tcp --tcp-flags ALL ACK -m length --length 256:511 -j DSCP --set-dscp-class CS2
$IPT -t mangle -A PREROUTING -p tcp -m tcp --tcp-flags ALL ACK -m length --length 512:1500 -j DSCP --set-dscp-class CS2
$IPT -t mangle -A PREROUTING -p tcp -m tcp --tcp-flags ALL ACK -m length --length :1500 -j DSCP --set-dscp-class CS2
#Make sure ACK packets get priority (to avoid upload speed limiting our download speed)
$IPT -t mangle -A PREROUTING -p tcp -m length --length :128 --tcp-flags SYN,RST,ACK ACK -j DSCP --set-dscp-class CS3
########################################
# Streaming Media (videos/audios)
########################################
#Know video streams sites like youtube,netflix
$IPT -t mangle -A PREROUTING -m set --match-set vidstream src -j DSCP --set-dscp-class AF41 -m comment --comment "vidstream ipset"
##set dscp tag for our ipset usrcdn #those cdn's have an isolated speeds by my ISP
$IPT -t mangle -A PREROUTING -m set --match-set usrcdn src -j DSCP --set-dscp-class AF21 -m comment --comment "usrcdn ipset"
# some iptv provider's use this port
$IPT -t mangle -A PREROUTING -p tcp -m multiport --dport 1935 -j DSCP --set-dscp-class AF41 -m comment --comment "some iptv streaming service"
########################################
# Best Effort (browsing/services)
########################################
#rule to make browsing, games login faster, for both tcp and udp
$IPT -t mangle -A PREROUTING -p tcp -m multiport --ports 80,443,853,3455,8080,27015 -j DSCP --set-dscp-class CS3 -m comment --comment "priority for faster browsing-tcp"
$IPT -t mangle -A PREROUTING -p udp -m multiport --ports 80,443,853,3455,8080 -j DSCP --set-dscp-class CS3 -m comment --comment "priority for faster browsing-udp"
#########################################
# Background Traffic (Bulk/file transfer)
#########################################
#bulk traffic ipset, like windows udates and steam updates/downloads
$IPT -t mangle -A PREROUTING -m set --match-set bulk src -j DSCP --set-dscp-class CS1 -m comment --comment "bulk traffic ipset"
#A robust rule to detect downloads/uploads and torrents!
$IPT -t mangle -A PREROUTING -p tcp -m multiport --ports 21,80,443,8080,60887,27014:27050 -m connbytes --connbytes 1048576: --connbytes-dir both --connbytes-mode bytes -j DSCP --set-dscp-class CS1 #60887 bittorrent
$IPT -t mangle -A PREROUTING -p udp -m multiport --ports 21,80,443,8080,60887 -m connbytes --connbytes 1048576: --connbytes-dir both --connbytes-mode bytes -j DSCP --set-dscp-class CS1
#tcpdump rule, copy and paste this rule into terminal, this rule is used to capture realtime traffic, you can change ip to what you like
#tcpdump -i br-lan host 192.168.1.126 and udp and portrange 1-65535 and !port 53 and ! port 80 and ! port 443 -vv -X -w /root/cap-name.pcap
Most of these seem reasonable, there are a few where I imagine you could go both directions rather than just download direction for example (matching ipset src rather than dest). I do think probably some of your rules overlap each other, like your ACK rules might set TCP connections to CS2 and then later your streaming media rules might reset it to AF41 or whatever. That might be something you want it to do, but you might look at that to see if the interaction of rules is what you want.
Overall, I think this is a pretty robust set of rules and together with an appropriate mapping to buckets should work well. If you have remaining latency issues with VOIP or games it's probably either inherent issues (ie. problems between you and the server) or transient over-committing to high priority buckets, like if your PS4 suddenly starts updating some software over port 8080 or some other weird port and that all gets prioritized CS6
1 Like
i fixed that by using src,dst.
Still not sure which one's is overlap with each other's, it's a tcp flag, so not sure if it will make a problem
with streaming media. i think it's fine if streaming media reset it to AF41.
Hahha, i'm not sure what i want to do and what i don't want to do!!!
Sometimes interaction is not bad!?
Yeah it's still working fine for all cases until now, it happens sometimes there's a problem between me and
server , duo to loaded isp routers/peering issues.
I don't have PS4 or any console so i can't test that behavior!
I exclude those bulky ports like(80,443,8080,torrent port), sometimes when i start a download a game ping
will go up by 20ms for 2 sec then back normal, i think this is because i have a small bandwidth.
at early morning my speed will go up to 8mbps, browsing is much faster and there's no ping jump while
initiate a download, but in general i tried running a download and uploaded while during a game session
and there's no lag or ping jump at all!
I will keep maintain this script, and let you know; if you have any ideas or suggestions please share it.
That's great. I definitely had a similar improvement when I set up my QoS for my voip server, and recently I had my voip server cloud kernel changed by the VPS managers and they left out the HFSC qdisc, so reboot made me lose the QoS things went terrible, so I fixed it by using my own kernel and continuing to use my QoS system. The point is, even when you have hundreds of megabits of bandwidth, it's still perfectly possible to get terrible realtime performance for VOIP or games unless you make some effort to tag/prioritize/queue properly.
1 Like
Bandwidth division for diffserv8 is:
+static int cake_config_diffserv8(struct Qdisc *sch)
+{
+/* Pruned list of traffic classes for typical applications:
+ *
+ * Network Control (CS6, CS7) ----> 39.27%
+ * Minimum Latency (EF, VA, CS5, CS4) ----> 44.87%
+ * Interactive Shell (CS2, TOS1) ----> 51.29%
+ * Low Latency Transactions (AF2x, TOS4) ----> 58.61%
+ * Video Streaming (AF4x, AF3x, CS3) ----> 66.99%
+ * Bog Standard (CS0 etc.) ----> 76.56%
+ * High Throughput (AF1x, TOS2) ----> 87.5%
+ * Background Traffic (CS1) ----> 100%
+ *
+ * Total 8 traffic classes.
+ */
@dlakelan
i'm planing to change my tags like that, correct me if it's wrong:
- CS6 for voip/gaming
- CS5 for dns and icmp
- CS2 some tcp flags and
- AF21 another set of tcp flags
- AF31 for youtube and other streaming
- CS0 for none prioritized traffic, also mail ports?!
- AF11 internet browsing and games login?
- CS1 for downloads and torrents
Having a look at Cisco DSCP values https://community.cisco.com/kxiwq67737/attachments/kxiwq67737/discussions-collaboration-apps/4531/1/dscpphb_values.jpg
so, all this discussion is great...
but, has anything been distilled down to normal jargon I could learn from?
like...
yea download this package aside from SQM and do this commands, better net!
1 Like
You can read here:
@dlakelan
I found something interested about my wifi Accidentally !
BE BK VI VO
MPDUs Queued: 3006 439 2 4354
MPDUs Completed: 2477 413 4 4746
MPDUs XRetried: 912 171 0 42
Aggregates: 24664 23446 0 0
AMPDUs Queued HW: 0 0 0 0
AMPDUs Completed: 196461 155442 36 0
AMPDUs Retried: 36099 6076 5 0
AMPDUs XRetried: 12035 1822 2 0
TXERR Filtered: 196 15 0 0
FIFO Underrun: 0 0 0 0
TXOP Exceeded: 0 0 0 0
TXTIMER Expiry: 0 0 0 0
DESC CFG Error: 0 0 0 0
DATA Underrun: 0 0 0 0
DELIM Underrun: 0 0 0 0
TX-Pkts-All: 211885 157848 42 4788
TX-Bytes-All: 250282717 199528107 4040 996801
HW-put-tx-buf: 160262 72233 47 4788
HW-tx-start: 0 0 0 0
HW-tx-proc-desc: 160262 72174 47 4788
TX-Failed: 0 0 0 0
(VO): qnum: 3 qdepth: 0 ampdu-depth: 0 pending: 0
(VI): qnum: 2 qdepth: 0 ampdu-depth: 0 pending: 0
(BE): qnum: 1 qdepth: 0 ampdu-depth: 0 pending: 0
(BK): qnum: 0 qdepth: 0 ampdu-depth: 0 pending: 0
(CAB): qnum: 8 qdepth: 0 ampdu-depth: 0 pending: 0
So nothing for the common man here lol
This is really more an experimental thread, nothing finalized into a "package" really. If you install and set up SQM with very basic settings then bam... Better network. If you want to go beyond that, requires some experiments, which is what this thread is about
1 Like
Stats on wmm usage are interesting how did you get that?
hehehe, too simple
cat /sys/kernel/debug/ieee80211/phy1/ath9k/xmit
cat /sys/kernel/debug/ieee80211/phy1/ath9k/queues
and this one for errors and frags:
cat /sys/kernel/debug/ieee80211/phy1/ath9k/recv
can you have a look at post
Diffserv8 may have 8 queues, but that doesn't mean you need to use all of them. I'd limit it to
- Realtime stuff, tagged CS6 (VOIP, games, NTP)
- Low Latency small stuff tagged CS5 (flags, DNS)
- Video Streaming tagged AF41 (this matches with WMM Video queue)
- Best Effort Stuff tagged CS0 (goes into BE WMM queue)
- Background CS1 (Goes into BK WMM queue)
Beyond that trying to use other tags will wind up getting you conflicts between cake and WMM and some cheap managed switches and various other things that might use DSCP, one thing will treat say CS2 as prioritized, another one will treat it as Background, etc.
1 Like
Thanks.
I see that when 2 clients connected to the same lan, assume client A call client B using viber voice call app
viber will tag packets with AF41 for both outbound and inbound, also the call is p2p using the private ip of
those 2 clients.
My ISP is tagging NTP packets with AF41 ?!