Reading @easyteacher's rule it seems to be connmarking connections based on them having a rate between 15 and 128 packets per second. I'm not quite clear on how it prevents connections from bouncing back and forth if they're near to the edge of those ranges, but maybe it doesn't matter. Doesn't seem to take packet size into account at all which could be fine too as long as you're in that range of packets per second the max bandwidth is 1500 bytes * 128 packets/s = 1.5 Mbps. So if you have at least 15Mbps then you're probably fine ignoring packet size. Only issue is if you get a LAN party going with 20 people all wanting to play CS:GO and wind up clogging your entire pipe 
Here is without sqm on
Here is my result for the speed test with sqm on
Now I'm not sure what's going on with the fluctuations in the speed but typically without sqm on I would hit around 800 mbps for download and around higher 800 mbps for upload. I'm not sure if its the ethernet cable is to blame or not. and unfortunately I don't have another pc or laptop to test it on...
What is your speed setting for sqm I agree your results are strange.
Speed setting for download is set to 0
Upload speed is set to 20000kbps
I suspect the wrt3200 will handle around 250Mbps to 300Mbps shaping without running out of CPU. Try setting it to up and down speeds of 200 and watch top -d 1 for the percentage idle, it should stay above say 10% to get good performance, from there bump it up in speed until you get to around 10% idle and then show that speed test result.
20000 is 20 Mbps you need an extra zero 
Set 200000 both directions. Download may be suffering due to bufferbloat.
Connbyte with avgpkt has already taken packet length into account. Specifically, upload and download traffic need to be separately considered.
Take LOL for example.
ipv4 2 udp 17 179 src=192.168.1.121 dst=182.254.35.166 sport=8786 dport=7080 packets=55483 bytes=4841699 src=182.254.35.166 dst=172.27.77.225 sport=7080 dport=8786 packets=65694 bytes=12491560 [ASSURED] mark=80 zone=0 use=2
Average upload packet length: 4841699/55483≈87
Average download packet length: 12491560 / 65694≈190
Games like DNF will have larger download packet length around 300-400.
I am using connmark to avoid those connections whose rates are bouncing back and forth. The following two rules are used to save CPU cycles by only checking connections with no connmark and remove connmark with a probability of 0.5% in order to let some connections checked again later.
$ipt -A PREROUTING -m conntrack --ctstate RELATED,ESTABLISHED -m connmark ! --mark 0xf0/0xf0 -m connmark ! --mark 0x60/0xf0 -m connmark ! --mark 0x70/0xf0 -m statistic --mode random --probability 0.005 -j CONNMARK --and-mark 0xff0f
$ipt -A PREROUTING -i br-lan -m connmark --mark 0x0/0xf0 -j mark_DSCP_ul
That rule seems reasonably good provided you've tested it and it works with your games. It seems pretty specialized, the one @hisham2630 and I tried is going to be somewhat more generic, will catch VOIP, most games, and probably catch a few other things, maybe some of which shouldn't be gotten, but I think in general it's pretty robust. Yours seems to be tuned more tightly.
The idea of a low-probability of resetting the connmark is maybe a good one in case stuff hits a threshold briefly, and never will come back. probability 0.005 gives you something like a 50% probability of getting reset after 1 second at 100 pps. So that seems reasonable you won't bounce back and forth too much.
@dlakelan
Hi
Have a look here, he used connection rate and connection bytes for some games, messengers and other services; this may help us
detect more traffic correctly!
I think that qualifies as too baroque for my tastes 
I really like our rate/size filter, and a few ipsets or proxy rules... at this point because I have much faster bandwidth than you it's probably just fine for me. I understand when you have limited bandwidth it can be worth it to try to squeeze better responsiveness out of it. I still don't think anything like that ruleset is a good idea, all those manually entered ip addresses etc... not good.
In your current system, what issues do you still face? delay? stuttering in games/streams? voip breakup?
2 Likes
I'm like you, i don't like a lot of rules(why i should have a bunch of rules, when one rule can do everything).
i like keep things simple and tidy, ip and port based prioritization is not good most of times and eat a lot
of cpu, also it's hard to maintain. but what i want to say from that link is to get some idea about how he did
used connbytes and connection rate to prioritize important services and some games without using ip' or
ports.
Games and Voip's is fine, even i don't have a problem with two voip running at the same time while download
and upload is running, the only problem is delay when do web browsing or when opening some apps
like youtube, it will take some time to load, i think you didn't had a look at this post
download won't fall into bulk connbytes (i see only a few packets).
i tried to use those two rules to speed up browsing and other things:
$IPT -t mangle -A PREROUTING -m length --length 0:500 -j DSCP --set-dscp-class CS3
$IPT -t mangle -A PREROUTING -m connbytes --connbytes 0:250 --connbytes-dir both --connbytes-mode avgpkt -j DSCP --set-dscp-class CS3
It's not specific to tcp or udp, i think it's need some tuning, also some VOIP will fall into the second rule!
@dlakelan I see it's better to isolate rules, so they won't interact with each other, because i see that sometime youtube packets tagged with CS1 instead of AF41, to fix the problem i will use something like this:
$IPT -t mangle -A PREROUTING -m set --match-set vidstream src,dst -j CONNMARK --set-mark 0x41 -m comment --comment "vidstream ipset"
$IPT -t mangle -A PREROUTING -m connmark --mark 0x41 -j DSCP --set-dscp-class AF41
$IPT -t mangle -A PREROUTING -p tcp -m connmark ! --mark 0x41 -m multiport --ports 21,25,80,81,443,444,554,8000,8080,8409,60887,27014:27050 -m connbytes --connbytes 500000: --connbytes-dir both --connbytes-mode bytes -j DSCP --set-dscp-class CS1 #60887 bittorrent
$IPT -t mangle -A PREROUTING -p udp -m connmark ! --mark 0x41 -m multiport --ports 21,25,80,81,443,444,554,8000,8080,8409,60887 -m connbytes --connbytes 500000: --connbytes-dir both --connbytes-mode bytes -j DSCP --set-dscp-class CS1
Yes, I couldn't make heads or tails of those images on my phone 
screen too small. I'm on my computer now, but still dont' know what I'm looking for. Do I understand that you have a rule to down-prioritize large total byte flows like CS1 for background downloads and it's not matching? This would be in the last image right?
Your image: https://forum.openwrt.org/uploads/default/original/2X/3/3aa2dfa0c51a46808ffefa40bb8b3e97bad2b06c.png
Seems to indicate about 300 MB of total traffic marked using those rules, so what was it you were thinking should have happened?
I notice you're marking 1.3 GB of TCP flags, so that seems probably not ideal. but it may be remarked by later rules anyway, like the rule that marks port 80/443 etc "priority for faster browsing-tcp"
Here's what I suggest: simplify your rule set for testing down to this:
########################################
# Latency Sensitive (gaming/voip)
########################################
##ICMP, to prioritize pings
$IPT -t mangle -A PREROUTING -p icmp -j DSCP --set-dscp-class CS5 -m comment --comment "ICMP-pings"
#DNS traffic both udp and tcp
$IPT -t mangle -A PREROUTING -p udp -m multiport --port 53,5353 -j DSCP --set-dscp-class CS5 -m comment --comment "DNS udp"
$IPT -t mangle -A PREROUTING -p tcp -m multiport --port 53,5353 -j DSCP --set-dscp-class CS5 -m comment --comment "DNS tcp"
#High priority ipset, i use for pubgM
$IPT -t mangle -A PREROUTING -p udp -m set --match-set latsens src -j DSCP --set-dscp-class CS6 -m comment --comment "latency sensitive ipset" ## set dscp tag for Latency Sensitive (latsens) ipset
#A robust 2 rules to detect realtime traffic
$IPT -t mangle -A PREROUTING -p udp -m hashlimit --hashlimit-name udp_high_prio --hashlimit-above 120/sec --hashlimit-burst 50 --hashlimit-mode srcip,srcport,dstip,dstport -j CONNMARK --set-mark 0x55 -m comment --comment "connmark for udp"
$IPT -t mangle -A PREROUTING -p udp -m connmark ! --mark 0x55 -m conntrack --ctorigsrc 192.168.1.0/24 -m multiport ! --ports 53,5353,80,443,8080,60887 -m connbytes --connbytes 0:700 --connbytes-dir both --connbytes-mode avgpkt -j DSCP --set-dscp-class CS6 -m comment --comment "small udp connection"
########################################
# Streaming Media (videos/audios)
########################################
#Know video streams sites like youtube,netflix
$IPT -t mangle -A PREROUTING -m set --match-set vidstream src,dst -j CONNMARK --set-mark 0x41 -m comment --comment "vidstream ipset"
$IPT -t mangle -A PREROUTING -m connmark --mark 0x41 -j DSCP --set-dscp-class AF41
##set dscp tag for our ipset usrcdn #those cdn's have an isolated speeds by my ISP
$IPT -t mangle -A PREROUTING -m set --match-set usrcdn src -j DSCP --set-dscp-class AF21 -m comment --comment "usrcdn ipset"
# some iptv provider's use this port
$IPT -t mangle -A PREROUTING -p tcp -m multiport --dport 1935 -j DSCP --set-dscp-class AF41 -m comment --comment "some iptv streaming service"
#########################################
# Background Traffic (Bulk/file transfer)
#########################################
#bulk traffic ipset, like windows udates and steam updates/downloads
$IPT -t mangle -A PREROUTING -m set --match-set bulk src -j DSCP --set-dscp-class CS1 -m comment --comment "bulk traffic ipset"
#A robust rule to detect downloads/uploads and torrents!
$IPT -t mangle -A PREROUTING -p tcp -m connmark ! --mark 0x41 -m multiport --ports 21,25,80,81,443,444,554,8000,8080,8409,60887,27014:27050 -m connbytes --connbytes 500000: --connbytes-dir both --connbytes-mode bytes -j DSCP --set-dscp-class CS1 #60887 bittorrent
$IPT -t mangle -A PREROUTING -p udp -m connmark ! --mark 0x41 -m multiport --ports 21,25,80,81,443,444,554,8000,8080,8409,60887 -m connbytes --connbytes 500000: --connbytes-dir both --connbytes-mode bytes -j DSCP --set-dscp-class CS1
I just removed some of the ack rules and browsing rules... I included your modifications to the connmark to prevent downgrading YouTube etc.
I think upgrading browsing and acks... may be a bad idea. Those seem like they should be "best effort" unless you are really starving acks, but even so I would use CS3 for acks not higher if you're using diffserv8.
1 Like
Np, i don't like small screens, that's why i always answering from computer.
Those large downloads rules is not working properly sometimes and only a few MB fall into connbytes TCP,
while udp connbytes, i don't know why it's have a big MB cause downloads usually use TCP (maybe youtube)
fallen into connbytes udp.
I think it's a bad idea to marking big tcp flags, that's why i comment it.
Thanks, but i'm planing to give a browsing some priority like CS2 or CS3, this will help speedup browsing and
opening apps login's etc ?!.
I will prioritize only small ACK's !, and i think it's better to use connmark !--mark xxx to avoid downgrading and
interact between rules
Yes, this is a good idea. Are you using mostly wired, or also wifi? Because the interaction between CS2 and CS3 and soforth with the WMM queue is not the same as with the diffserv8 queues. CS2 is the same as CS1 in WMM!!! background! CS3 is the same as best effort in WMM! (at least these are defaults). Currently we can't adjust the queue settings using hostapd because that feature isn't compiled into the version in OpenWrt. There's a thread from about 6 mo ago about this, but no progress.
So, my general take is try to use CS0 for "normal" stuff, CS1 for "background" and then bump all the way to say AF41 for video / DNS, and CS5 for games (goes in VIDEO queue) and CS6 for voip (goes in VOICE queue), or maybe CS6 for both games and voip, depending on how important games are for you.
Trying to use a wide variety of queues doesn't work well because different parts of the system (wifi, cake, maybe managed switches etc) will treat them differently.
Make sure to test this carefully, though. In general it seems advisable to put ACKs into the same priority tier as the respective data packets, but this is from first principle and might be not 100% optimal. Anyway make sure to test this, and make sure that the ACKs of high priority flows also end up in the high priority tier...
This will avoid problems, mostly i use wifi, i only use wire when something screwed up or my router crazy
(old router),
That's what I've wanted to know for a long time, strange default's!!!
I know about that thread it's by @shm0, i wish him the best.
I have a usb wifi act as AP(to extended range), but this one doesn't support WMM queue(i think it's driver problem), it's a ralink chipset.
Maybe i will use CS6 for games and voip(cause you know i don't have voip phone, i use whatsapp and viber as voip), also we can't differ between voip and games(maybe we need a method like @easyteacher ).
CS5 for DNS and NTP and AF41 for video.
You know that voip is more important but we can't catch it Specifically (at least NDPI can do it, but compile
code is pain).
So for that reason diffserv4 is enough, and there's no standards that force a proper use of marking.but i like diffserv8 cause it offer more bandwidth for each tin, so you
this will reduce the probability one tin can overflow to the next one! , because it run out of bandwidth.
I will try to isolate those ACK's by using connmark, there's no ideal things in this wold, but at least we can
make things work better, like video stream fall from AF41 to CS1 this is not acceptable, i done some test's
that's why i mentioned here to isolate rules and prevent them from interact with each other, games and voip
was sometimes downgraded from CS6 to CS3, so marking the connection fixed the problem.
i will keep you all updated, and sorry for my bad English.
@dlakelan
Can i use the another veth method (bridged mode) .
like bridge my eth0.4 (wan interface) with veth0 and run a pppoe on veth1 to get internet, then inbound sqm on
veth1 and outbound sqm on veth0, is this correct?
Interesting point, sevices like youtube that use dash, should IMHO not be placed higher than CS0 as they show a really nasty bursty behaviour (at reaching a watermark level in the draining buffer they will request data for a few more seconds which will rush in in a rather high-bandwidth burst, placing that above CS0 will wreck havoc to all traffic in CS0, that is unless the higher tier is much smaller in bandwidth). Now you could go and elevate traditional CS0 uses like interactive web browsing above CS0, but then rising youtube dash priority starts to appear somewhat busy-work, YMMV.
1 Like
Youtube is isolated by all isp, it will not affect the main poor bandwidth at all. sometimes there's no internet
access (isp issue), and youtube work normally!
Fair enough. Now I just used youtube as a generic example for any DASH using streaming service (but failed to mention the intended generality, sorry for not being clear).