Ultimate SQM settings: Layer_cake + DSCP marks

That is partly my point; it is much easier to screw this kind of prioritization up than do it well; both you and @hisham2630 are way ahead of the curve in not-screwing-it-up though :wink:

1 Like

Thanks, we win vs Vietnam!!!
Do not spare us your advice, have a nice day!

1 Like

@dlakelan @moeller0
I think it's possible to use qos with software offloading friendly by using this rule before the offloading rule?!:
iptables -t filter -A FORWARD -m conntrack --ctstate established,related -m dscp ! --dscp 0 -j ACCEPT

Finally!, now my downloads is suffering a lot(5KB/s) while browsing webpages, and webpage load instantly like there's
no download at all, also isolate rules was a perfect idea many thanks @easyteacher

Great, can you post the rules you're using now!

Sure i can't share them (copyright issues), just kidding.
@dlakelan @moeller0 I think it's better to use
-m dscp ! --dscp 48 instead of -m connmark ! --mark 0x6 right ???

########################################
# Latency Sensitive (gaming/voip)
########################################
##ICMP, to prioritize pings
$IPT -t mangle -A PREROUTING -p icmp -j CONNMARK --set-mark 0x5 -m comment --comment "ICMP-pings"

#DNS traffic both udp and tcp
$IPT -t mangle -A PREROUTING -p udp -m multiport --port 53,123,5353 -j CONNMARK --set-mark 0x5 -m comment --comment "DNS udp and NTP"
$IPT -t mangle -A PREROUTING -p tcp -m multiport --port 53,5353 -j CONNMARK --set-mark 0x5 -m comment --comment "DNS tcp"
$IPT -t mangle -A PREROUTING -m connmark --mark 0x5 -j DSCP --set-dscp-class CS5

#give a high priority for PS4 and xbox
$IPT -t mangle -A PREROUTING -p tcp -m conntrack --ctorigsrc 192.168.1.150 -m multiport ! --ports 80,443,8080 -j CONNMARK --set-mark 0x6 -m comment --comment "PS4" #for PS4 or xbox etc, change the ip according to your ip setting.
$IPT -t mangle -A PREROUTING -p udp -m conntrack --ctorigsrc 192.168.1.150 -m multiport ! --ports 80,443,8080 -j CONNMARK --set-mark 0x6 -m comment --comment "PS4"

#High priority ipset, i use for pubgM
$IPT -t mangle -A PREROUTING ! -p tcp -m set --match-set latsens src,dst -j CONNMARK --set-mark 0x6 -m comment --comment "latency sensitive ipset" ## set dscp tag for Latency Sensitive (latsens) ipset

#A robust 2 rules to detect realtime traffic
$IPT -t mangle -A PREROUTING -p udp -m hashlimit --hashlimit-name udp_high_prio --hashlimit-above 120/sec --hashlimit-burst 50 --hashlimit-mode srcip,srcport,dstip,dstport -j CONNMARK --set-mark 0x55 -m comment --comment "connmark for udp"
$IPT -t mangle -A PREROUTING -p udp -m connmark ! --mark 0x55 -m conntrack --ctorigsrc 192.168.1.0/24 -m multiport ! --ports 53,5353,80,443,8080,60887 -m connbytes --connbytes 0:700 --connbytes-dir both --connbytes-mode avgpkt -j CONNMARK --set-mark 0x6 -m comment --comment "small udp connection"
$IPT -t mangle -A PREROUTING -m connmark --mark 0x6 -j DSCP --set-dscp-class CS6
##################
#TCP ACK flows
##################
# prioritize inbound and outbound ACK pkts according to size
# empty ipv4 ack pkts <64 appear very often but empty ipv6 ack pkts are larger and fall into smaller priority class
$IPT -t mangle -A PREROUTING -p tcp -m tcp --tcp-flags ALL ACK -m length --length :64 -j CONNMARK --set-mark 0x3
$IPT -t mangle -A PREROUTING -p tcp -m tcp --tcp-flags ALL ACK -m length --length 65:89 -j CONNMARK --set-mark 0x3

#Make sure ACK packets get priority (to avoid upload speed limiting our download speed)
$IPT -t mangle -A PREROUTING -p tcp -m length --length :128 --tcp-flags SYN,RST,ACK ACK -j CONNMARK --set-mark 0x3
$IPT -t mangle -A PREROUTING -m connmark --mark 0x3 -j DSCP --set-dscp-class CS3

#mid size ACK
$IPT -t mangle -A PREROUTING -p tcp -m tcp --tcp-flags ALL ACK -m length --length 90:159 -j CONNMARK --set-mark 0x2
$IPT -t mangle -A PREROUTING -p tcp -m tcp --tcp-flags ALL ACK -m length --length 160:255 -j CONNMARK --set-mark 0x2
$IPT -t mangle -A PREROUTING -m connmark --mark 0x2 -j DSCP --set-dscp-class CS2

# large ack pkts carry data and dont need the same priority
$IPT -t mangle -A PREROUTING -p tcp -m tcp --tcp-flags ALL ACK -m length --length 256:511 -j CONNMARK --set-mark 0x2
########################################
# Streaming Media (videos/audios)
########################################
#Known video streams sites like youtube,netflix
$IPT -t mangle -A PREROUTING -m set --match-set vidstream src,dst -j CONNMARK --set-mark 0x41 -m comment --comment "vidstream ipset"
# some iptv provider's use this port
$IPT -t mangle -A PREROUTING -p tcp -m multiport --port 1935 -j CONNMARK --set-mark 0x41 -m comment --comment "some iptv streaming service"
$IPT -t mangle -A PREROUTING -m connmark --mark 0x41 -j DSCP --set-dscp-class AF41

##set dscp tag for our ipset usrcdn #those cdn's have an isolated speeds by my ISP
$IPT -t mangle -A PREROUTING -m set --match-set usrcdn src,dst -j CONNMARK --set-mark 0x21 -m comment --comment "usrcdn ipset"
$IPT -t mangle -A PREROUTING -m connmark --mark 0x21 -j DSCP --set-dscp-class AF21

########################################
# Best Effort (browsing/services)
########################################
#Small packet is probably interactive or flow control
$IPT -t mangle -A PREROUTING -m connmark ! --mark 0x2 -m connmark ! --mark 0x3 -m connmark ! --mark 0x5 -m connmark ! --mark 0x6 -m connmark ! --mark 0x21 -m connmark ! --mark 0x41 -m length --length 0:500 -j CONNMARK --set-mark 0x3

#Small packet connections: multi purpose (don't harm since not maxed out)
$IPT -t mangle -A PREROUTING -m connmark ! --mark 0x2 -m connmark ! --mark 0x3 -m connmark ! --mark 0x5 -m connmark ! --mark 0x6 -m connmark ! --mark 0x21 -m connmark ! --mark 0x41 -m connbytes --connbytes 0:250 --connbytes-dir both --connbytes-mode avgpkt -j CONNMARK --set-mark 0x3
#########################################
# Background Traffic (Bulk/file transfer)
#########################################
#bulk traffic ipset, like windows udates and steam updates/downloads
$IPT -t mangle -A PREROUTING -m set --match-set bulk src,dst -j CONNMARK --set-mark 0x1 -m comment --comment "bulk traffic ipset"

#A robust rule to detect downloads/uploads and torrents!
$IPT -t mangle -A PREROUTING -m connmark ! --mark 0x2 -m connmark ! --mark 0x3 -m connmark ! --mark 0x5 -m connmark ! --mark 0x6 -m connmark ! --mark 0x21 -m connmark ! --mark 0x41 -m connmark --mark 0x1 -j CONNMARK --set-mark 0xB

$IPT -t mangle -A PREROUTING -p tcp -m multiport --ports 21,25,80,81,443,444,554,8000,8080,8409,60887,27014:27050 -m connbytes --connbytes 500000: --connbytes-dir both --connbytes-mode bytes -j CONNMARK --set-mark 0xB #60887 bittorrent
$IPT -t mangle -A PREROUTING -p udp -m multiport --ports 21,25,80,81,443,444,554,8000,8080,8409,60887 -m connbytes --connbytes 500000: --connbytes-dir both --connbytes-mode bytes -j CONNMARK --set-mark 0xB
$IPT -t mangle -A PREROUTING -m connmark --mark 0xB -j DSCP --set-dscp-class CS1

#tcpdump rule, copy and paste this rule into terminal, this rule is used to capture realtime traffic, you can change ip to what you like
#tcpdump -i br-lan host 192.168.1.126 and udp and portrange 1-65535 and !port 53 and ! port 80 and ! port 443 -vv -X -w /root/cap-name.pcap

One question @dlakelan @moeller0 can CAKE handle a lot of users, like 100 ~ 400 users ?

Hi @Ernieelias i forgot to check in and find out the results of those tests. Were you able to try 200000 kbps and get more stable bandwidth? I think your previous tests were suffering from ack starvation on download because the upload was configured so low.

I think this all depends on bandwidth and router hardware. Obviously with say a megabit and 1000 users the bandwidth sharing will probably not work well because just a kilobit or so per user is not much and you can only send one MTU sized packet every 8-10 seconds.... But with a gigabit and 1000 users it should handle it ok. Obviously more users means more connections and more RAM needed to track them all. So you need more hardware. But I'd be surprised if an x86 couldn't split a gigabit among 1000 users. I wonder if maybe some university or medium business has done it.

1 Like

I'd like to see this too, i know an ISP operator, i'm not sure how much bandwidth he have but i think it's between
10~100 mbps and he have about 100 users or more, i did subscribed to his service but it was too slow, not
only slow but also unresponsive, i can't play any games, login failed, loading... and webpage loading take
long time sometimes failed with tcp_reset error etc... at night i was getting about 30KB/s sometimes goes to
10KB/s.
@dlakelan i want to switch to veth bridge mode instead of routing mode?
i assume i should bridge veth0 with eth0.4(wan interface), then ran pppoe on veth1 ?
after this sqm egress on veth0 and sqm ingress on veth1?

One side question: @dlakelan can you tell me how much current your mini pc consume?!
Cause my dc ups can give up to 2A, my dc ups can out multiple voltages (5v,9v,12v,15v,24v), and can provide
2A for each output!.
And is core i3 mini pc better than celeron n3350

Yes, but you have to create a separate namespace and move the bridge over there otherwise the kernel decides it needs to route incoming packets and it never bridges into veth0.

It runs a 4 disk RAID array as a NAS that provides home directories and media to 4 other computers so it's more than just a router... but I think all together it's about 50-60 watts. The PC itself is probably only 10-15 watts. I'll check what my UPS says and get back to you. I bet 12v 2A is enough for a Celeron mini PC.
I am liking my espressobin that I just got this week, you should look into that if you want v low power and are going to have less than 200Mbps. I'm running Armbian on it. I've read that it idles at 1W.

My UPS is putting out 72 watts for the mini PC, a 4 spinning disk RAID, a 24 port smart switch, and a monitor turned off. I'd guess the PC itself is probably close to 20-30 watts

1 Like

I will stick with routing method for now, because names pace is not enabled on openwrt (i think it's need to change a setting in the kernel config)

I want something that can handle 1gbit/s , this will avoid me future upgrading, 2amp @12v =24 watts
so it's enough, i'm not planing to connect any HDD.

Is it a 20-30 watts @ 12v ?, does your smart switch got it's power from the mini pc or from ups(i'm sure it's from UPS?)

Congratulations :smiling_face_with_three_hearts: , in my country it's pain to buy something online, so i will
stick to what is available in the shopes

No, it's 120V AC house current, the PC has a power supply in it. The smart switch is plugged in separately to the UPS.

Now i understand, the pc powered by an ac/dc adapter ?

It's a standard ATX power supply, I put it together from parts before these mini PCs we're common, it's a mini itx board and a case with ATX power supply.

If I send you a shipment of 100 espressobins you can set up Hisham's Hollywood Router Emporium and sell them there. I'll drive the shipment past the walk of stars in Hollywood so you can honestly claim it's the router to the stars :joy:

1 Like

HHAh, Thank you for your generosity.

Then assume it will be out of fuel after sometime then not more router:sob:
I just told people here to use openwrt in their routers, they like the features but they said opnewrt doesn't
support our native speaking language so most can't use it.
i deiced to build a translation pack in my language for openwrt, but i have two problems.

  1. does openwrt(LUCI) support my language letter encoding.
  2. i don't know from where should i start, i read somewhere it's poorly documented.

I know that Luci is translated to vietnamese and chinese and things, so I assume it uses utf8 so it should handle arabic characters at least. I don't know about how to handle right to left vs left to right. I'm woefully ignorant about that, you should start a new thread and ask here if someone can help you figure out what is needed.

1 Like

Why does all isp's here use MTU=1480 and not 1500 ?
http://n2.netalyzr.icsi.berkeley.edu/summary/id=36a470be-31160-6e4f1570-a314-4a48-a11c#tcp_connectivity

It might be because of something like pppoE ?

But if you see some ISP's use 1492 or 1500(jumbo frame) as MTU