Start with diffserv8 and see if that works for you. It at least gives you more than one latency sensitive tier
1 Like
I will try to use it later.
There's one problem when packet length reach 260 it will stop tag packet with CS6, then packets between
260 and 304 some is tagged and some is not tagged.i will share .pcap when voip call is finish
Can you post the full ruleset you are using?
The conntrack rule is not working properly, but port tagging is working.
conntrack only script:
IPT="iptables"
tc qdisc add dev wlan0 root mq #enable multi queue on wlan0
## set up a 2 pair of veth devices to handle inbound and outbound traffic
ip link show | grep veth0 || ip link add type veth
## get new veth interfaces up
ip link set veth0 up
ip link set veth1 up
## trun on promisc mode,sometimes it's needed to make bridge work
ip link set veth1 promisc on
## add veth1,veth3 to bridge
brctl addif br-lan veth1
## just to make sure there's nothing inside those 2 tables
ip rule del priority 100
ip route flush table 100
##ipset for streming sites.they are bening filled by dnsmasq
ipset create vidstream hash:ip
ipset create usrcdn hash:ip
ipset create bulk hash:ip
ipset create latsens hash:ip
## flush mangle table
$IPT -t mangle -F PREROUTING
## add routing for veth0 this will handle all slow traffic
ip route add default dev veth0 table 100
ip rule add iif pppoe-wan table 100 priority 100
########################################
# Latency Sensitive (gaming/voip)
########################################
##Latency Sensitive
$IPT -t mangle -A PREROUTING -p icmp -j DSCP --set-dscp-class CS5 -m comment --comment "ICMP-pings"
$IPT -t mangle -A PREROUTING -p udp -m multiport --port 53,5353 -j DSCP --set-dscp-class CS5 -m comment --comment "DNS udp"
$IPT -t mangle -A PREROUTING -p tcp -m multiport --port 53,5353 -j DSCP --set-dscp-class CS5 -m comment --comment "DNS tcp"
$IPT -t mangle -A PREROUTING -p tcp -m conntrack --ctorigsrc 192.168.1.150 -m multiport ! --ports 80,443 -j DSCP --set-dscp-class CS6 -m comment --comment "PS4" #for PS4, change the ip according to your ip setting.
$IPT -t mangle -A PREROUTING -p udp -m conntrack --ctorigsrc 192.168.1.150 -m multiport ! --ports 80,443 -j DSCP --set-dscp-class CS6 -m comment --comment "PS4"
$IPT -t mangle -A PREROUTING -p udp -m set --match-set latsens src -j DSCP --set-dscp-class CS6 -m comment --comment "latency sensitive ipset" ## set dscp tag for Latency Sensitive (latsens) ipset
#new
$IPT -t mangle -A PREROUTING -p udp -m hashlimit --hashlimit-name udp_high_prio --hashlimit-above 75/sec --hashlimit-mode srcip,srcport,dstip,dstport -j CONNMARK --set-mark 0x55 -m comment --comment "connmark for udp"
$IPT -t mangle -A PREROUTING -p udp -m connmark ! --mark 0x55 -m conntrack --ctorigsrc 192.168.1.0/24 -m multiport ! --ports 53,80,443,60887 -m length --length 0:700 -j DSCP --set-dscp-class CS6 -m comment --comment "small udp flows"
##################
#TCP ACK flows
##################
# prioritize outbound ACK pks according to size
# empty ipv4 ack pkts <64 appear very often but empty ipv6 ack pkts are larger and fall into smaller priority class
$IPT -t mangle -A PREROUTING -p tcp -m tcp --tcp-flags ALL ACK -m length --length :64 -j DSCP --set-dscp-class CS4
$IPT -t mangle -A PREROUTING -p tcp -m tcp --tcp-flags ALL ACK -m length --length 65:89 -j DSCP --set-dscp-class CS4
#
$IPT -t mangle -A PREROUTING -p tcp -m tcp --tcp-flags ALL ACK -m length --length 90:159 -j DSCP --set-dscp-class CS3
$IPT -t mangle -A PREROUTING -p tcp -m tcp --tcp-flags ALL ACK -m length --length 160:255 -j DSCP --set-dscp-class CS3
# Make sure ACK packets get priority (to avoid upload speed limiting our download speed)
$IPT -t mangle -A PREROUTING -p tcp -m length --length :128 --tcp-flags SYN,RST,ACK ACK -j DSCP --set-dscp-class CS3
# large ack pkts carry data and dont need the same priority
$IPT -t mangle -A PREROUTING -p tcp -m tcp --tcp-flags ALL ACK -m length --length 256:511 -j DSCP --set-dscp-class CS2
$IPT -t mangle -A PREROUTING -p tcp -m tcp --tcp-flags ALL ACK -m length --length 512:1500 -j DSCP --set-dscp-class CS2
$IPT -t mangle -A PREROUTING -p tcp -m tcp --tcp-flags ALL ACK -m length --length :1500 -j DSCP --set-dscp-class CS2
########################################
# Streaming Media (videos/audios)
########################################
$IPT -t mangle -A PREROUTING -m set --match-set vidstream src -j DSCP --set-dscp-class AF41 -m comment --comment "vidstream ipset"
$IPT -t mangle -A PREROUTING -m set --match-set usrcdn src -j DSCP --set-dscp-class AF21 -m comment --comment "usrcdn ipset" #set dscp tag for our ipset usrcdn #those cdn's are isolated by my ISP.
$IPT -t mangle -A PREROUTING -p tcp -m multiport --dport 1935 -j DSCP --set-dscp-class AF41 -m comment --comment "some iptv streaming service"
########################################
# Best Effort (browsing/services)
########################################
$IPT -t mangle -A PREROUTING -p tcp -m multiport --ports 80,443,853,3455,8080,27015 -j DSCP --set-dscp-class CS3 -m comment --comment "priority for faster browsing-tcp"
$IPT -t mangle -A PREROUTING -p udp -m multiport --ports 80,443,853,3455,8080 -j DSCP --set-dscp-class CS3 -m comment --comment "priority for faster browsing-udp"
#########################################
# Background Traffic (Bulk/file transfer)
#########################################
$IPT -t mangle -A PREROUTING -m set --match-set bulk src -j DSCP --set-dscp-class CS1 -m comment --comment "bulk traffic ipset" ##set dscp tag for our bulk ipset
$IPT -t mangle -A PREROUTING -p tcp -m multiport --ports 21,80,443,8080,60887,27014:27050 -m connbytes --connbytes 1048576: --connbytes-dir both --connbytes-mode bytes -j DSCP --set-dscp-class CS1 #60887 bittorrent
$IPT -t mangle -A PREROUTING -p udp -m multiport --ports 21,80,443,8080,60887 -m connbytes --connbytes 1048576: --connbytes-dir both --connbytes-mode bytes -j DSCP --set-dscp-class CS1
port tagging + conntrack:
IPT="iptables"
tc qdisc add dev wlan0 root mq #enable multi queue on wlan0
####Latency Sensitive Ports
##league of legends,PUBGM,some voip service like whatsapp.
sport1="37,81,88,123,500,1500,1863,3074:3076,3478:3481,3544,3659,4244"
sport2="4379,4380,4500,5222,5223,5228,5242,5000:5500,7985,8000,8888,9339,9785,34784"
sport3="10000:10300,19132,25565,27000:27030,28960,59172,39499,34141,59437:59581,45395,50318,59234"
sport4="1579,20000"
dport1="37,81,88,123,500,1500,1863,3005,3074:3076,3101,3478:3481,3544"
dport2="3659,4244,4379,4380,4500,5222,5223,5228,5242,5000:5500,7985,8000,8888,34784"
dport3="9339,9785,10000:10300,19132,25565,27000:27030,28960,59172,39499,34141,59437:59581,45395"
dport4="1119,1579,3724,7000:7999,8011,8081:8090,9030,17000,20000,50318,59234"
## set up a 2 pair of veth devices to handle inbound and outbound traffic
ip link show | grep veth0 || ip link add type veth
## get new veth interfaces up
ip link set veth0 up
ip link set veth1 up
## trun on promisc mode,sometimes it's needed to make bridge work
ip link set veth1 promisc on
## add veth1,veth3 to bridge
brctl addif br-lan veth1
## just to make sure there's nothing inside those 2 tables
ip rule del priority 100
ip route flush table 100
##ipset for streming sites.they are bening filled by dnsmasq
ipset create vidstream hash:ip
ipset create usrcdn hash:ip
ipset create bulk hash:ip
ipset create latsens hash:ip
## flush mangle table
$IPT -t mangle -F PREROUTING
## add routing for veth0 this will handle all slow traffic
ip route add default dev veth0 table 100
ip rule add iif pppoe-wan table 100 priority 100
#Clear interface dscp marks, we don't trust ISP marks(also to use our own marks).
#$IPT -t mangle -A PREROUTING -i pppoe-wan -j DSCP --set-dscp 0
########################################
# Latency Sensitive (gaming/voip)
########################################
##Latency Sensitive
$IPT -t mangle -A PREROUTING -p icmp -j TOS --set-tos 0x10 ##TOS tag for ping packets
#dns traffic
$IPT -t mangle -A PREROUTING -p udp -m multiport --port 53,5353 -j DSCP --set-dscp-class CS5
$IPT -t mangle -A PREROUTING -p tcp -m multiport --port 53,5353 -j DSCP --set-dscp-class CS5
$IPT -t mangle -A PREROUTING -p udp -m set --match-set latsens src -j DSCP --set-dscp-class CS6 ## set dscp tag for our Latency Sensitive (latsens) ipset
$IPT -t mangle -A PREROUTING -p tcp -m conntrack --ctorigsrc 192.168.1.150 -m multiport ! --ports 80,443 -j DSCP --set-dscp-class CS6 -m comment --comment "PS4" #for PS4, change the ip according to your ip setting.
$IPT -t mangle -A PREROUTING -p udp -m conntrack --ctorigsrc 192.168.1.150 -j DSCP --set-dscp-class CS6 -m comment --comment "PS4"
$IPT -t mangle -A PREROUTING -p udp -m multiport --sport $sport1 -j DSCP --set-dscp-class CS6
$IPT -t mangle -A PREROUTING -p udp -m multiport --sport $sport2 -j DSCP --set-dscp-class CS6
$IPT -t mangle -A PREROUTING -p udp -m multiport --sport $sport3 -j DSCP --set-dscp-class CS6
$IPT -t mangle -A PREROUTING -p udp -m multiport --sport $sport4 -j DSCP --set-dscp-class CS6
$IPT -t mangle -A PREROUTING -p tcp -m multiport --sport 22,2099,4244,5222,5223,5228,5242,8088,8393:8400,50318,59234 -j DSCP --set-dscp-class CS6
$IPT -t mangle -A PREROUTING -m dscp --dscp 48 -j TOS --set-tos 0xc0
$IPT -t mangle -A PREROUTING -p udp -m multiport --dport $dport1 -j DSCP --set-dscp-class CS6
$IPT -t mangle -A PREROUTING -p udp -m multiport --dport $dport2 -j DSCP --set-dscp-class CS6
$IPT -t mangle -A PREROUTING -p udp -m multiport --dport $dport3 -j DSCP --set-dscp-class CS6
$IPT -t mangle -A PREROUTING -p udp -m multiport --dport $dport4 -j DSCP --set-dscp-class CS6
$IPT -t mangle -A PREROUTING -p tcp -m multiport --dport 22,2099,3013,4244,5222:5228,5242,10012,17000,17500,8088,8393:8400,50318,59234 -j DSCP --set-dscp-class CS6
#test
#$IPT -t mangle -A PREROUTING -p udp -m multiport ! --ports 53,80,443,60887 -m length --length 28:1500 -j DSCP --set-dscp-class CS6
#$IPT -t mangle -A PREROUTING -p udp -m conntrack --ctorigsrc 192.168.1.0/24 -m multiport ! --ports 53,80,443,60887 -m connbytes --connbytes 0:700 --connbytes-dir both --connbytes-mode avgpkt -j DSCP --set-dscp-class CS6
#new
$IPT -t mangle -A PREROUTING -p udp -m hashlimit --hashlimit-name udp_high_prio --hashlimit-above 75/sec --hashlimit-mode srcip,srcport,dstip,dstport -j CONNMARK --set-mark 0x55
$IPT -t mangle -A PREROUTING -p udp -m connmark ! --mark 0x55 -m conntrack --ctorigsrc 192.168.1.0/24 -m multiport ! --ports 53,80,443,60887 -m length --length 0:700 -j DSCP --set-dscp-class CS6
###############
#TCP ACK flows
###############
# prioritize outbound ACK pks according to size
# empty ipv4 ack pkts <64 appear very often but empty ipv6 ack pkts are larger and fall into the next band
$IPT -t mangle -A PREROUTING -p tcp -m tcp --tcp-flags ALL ACK -m length --length :64 -j DSCP --set-dscp-class CS3
$IPT -t mangle -A PREROUTING -p tcp -m tcp --tcp-flags ALL ACK -m length --length 65:89 -j DSCP --set-dscp-class CS3
# Make sure ACK packets get priority (to avoid upload speed limiting our download speed)
$IPT -t mangle -A PREROUTING -p tcp -m length --length :128 --tcp-flags SYN,RST,ACK ACK -j DSCP --set-dscp-class CS3
#
$IPT -t mangle -A PREROUTING -p tcp -m tcp --tcp-flags ALL ACK -m length --length 90:159 -j DSCP --set-dscp-class CS3
$IPT -t mangle -A PREROUTING -p tcp -m tcp --tcp-flags ALL ACK -m length --length 160:255 -j DSCP --set-dscp-class CS3
# Make sure ACK packets get priority (to avoid upload speed limiting our download speed)
$IPT -t mangle -A PREROUTING -p tcp -m length --length :128 --tcp-flags SYN,RST,ACK ACK -j DSCP --set-dscp-class CS3
# large ack pkts carry data and dont need the same priority these rules are here temporarily for accounting
# to see quite how many pkts drop into each size band
$IPT -t mangle -A PREROUTING -p tcp -m tcp --tcp-flags ALL ACK -m length --length 256:511 -j DSCP --set-dscp-class CS2
$IPT -t mangle -A PREROUTING -p tcp -m tcp --tcp-flags ALL ACK -m length --length 512:1500 -j DSCP --set-dscp-class CS2
$IPT -t mangle -A PREROUTING -p tcp -m tcp --tcp-flags ALL ACK -m length --length :1500 -j DSCP --set-dscp-class CS2
########################################
# Streaming Media (videos/audios)
########################################
$IPT -t mangle -A PREROUTING -m set --match-set vidstream src -j DSCP --set-dscp-class AF41 ##set dscp tag for our ipset vidstearm
$IPT -t mangle -A PREROUTING -m dscp --dscp 34 -j TOS --set-tos 0x88
$IPT -t mangle -A PREROUTING -m set --match-set usrcdn src -j DSCP --set-dscp-class AF21 ##set dscp tag for our ipset usrcdn #those cdn's are isolated by my ISP.
$IPT -t mangle -A PREROUTING -p tcp -m multiport --dport 1935 -j DSCP --set-dscp-class AF41
$IPT -t mangle -A PREROUTING -m dscp --dscp 18 -j TOS --set-tos 0x48
########################################
# Best Effort (browsing/services)
########################################
$IPT -t mangle -A PREROUTING -p tcp -m multiport --ports 80,443,853,3455,8080,27015 -j DSCP --set-dscp-class CS3
$IPT -t mangle -A PREROUTING -p udp -m multiport --ports 80,443,853,3455,8080 -j DSCP --set-dscp-class CS3
$IPT -t mangle -A PREROUTING -m dscp --dscp 24 -j TOS --set-tos 0x60
#########################################
# Background Traffic (Bulk/file transfer)
#########################################
$IPT -t mangle -A PREROUTING -m set --match-set bulk src -j DSCP --set-dscp-class CS1 ##set dscp tag for our bulk ipset
$IPT -t mangle -A PREROUTING -p tcp -m multiport --ports 21,80,443,8080,60887,27014:27050 -m connbytes --connbytes 1048576: --connbytes-dir both --connbytes-mode bytes -j DSCP --set-dscp-class CS1 #60887 bittorrent
$IPT -t mangle -A PREROUTING -p udp -m multiport --ports 21,80,443,8080,60887 -m connbytes --connbytes 1048576: --connbytes-dir both --connbytes-mode bytes -j DSCP --set-dscp-class CS1
$IPT -t mangle -A PREROUTING -m dscp --dscp 8 -j TOS --set-tos 0x20
I'm not sure if conntrack is enabled for all flows in the standard firewall, though I'd guess it's required for NAT? You might look in Luci if there's a place to force enable it
Also check the counters on the rules are the hashlimit rules or conntrack rules ever being matched?
I think i need a sysctl rule.
3.84 K 4.35 MB CONNMARK udp * * 0.0.0.0/0 0.0.0.0/0 limit: above 75/sec burst 5 mode srcip-srcport-dstip-dstport CONNMARK set 0x55 -
9.26 K 2.00 MB DSCP udp * * 0.0.0.0/0 0.0.0.0/0 connmark match ! 0x55 ctorigsrc 192.168.1.0/24 multiport ports !53,80,443,60887 length 0:700 DSCP set 0x30
conntrack is already enabled, but for some reason it won't match.
here's a voip cap: https://a.uguu.se/sULknFikWhyE_mia2w.pcap
EDIT:
I think i know where is the problem, let me check!
Instead of:
-m connmark ! --mark 0x55
It should be:
-m mark ! --mark 0x55
Conntrack for udp packets only:
https://a.uguu.se/wkRewN8sTspl_ct
I think you should use connmark because it marks the whole connection so no issues with some packets marked and some not.
I suggest you set your hashlimit burst to at least half a second worth of packets so burst = 37 this avoids having your hashlimit match after the first 5 packets due to jitter and extra setup packets etc.
2 Likes
it's tag all packets now even size like 557, but i still face a lag when download is running.
EDIT:
It's working good, but let me check outbound.
Should be only UDP packets, and not port 80 or 443, so QUIC won't be included
1 Like
Yeah, i already excluded both port 80 and 443, also qbittorrent port!
On outbound biggest packet size was 400 and got tagged properly, so all packets got tag without the need
to tag ports, this is a big profit!
let me test on long voip sessions, then let you know, many thanks
Hi @moeller0
I hope you are fine 
Can you tell us about the bandwidth sharing between each diffserv8 tins?
Also some info about diffserv-llt "Latency-Loss Tradeoff"
I know that bandwidth sharing in diffserv4 is:
bulk best-effort video voice
6.25% 100% 50% 25%
@dlakelan
Our conntrack and hashlimit rule is working great, but there's only one problem, packets will get tagged until
length=737, any packet bugger than that size will stay CS0. any suggestion?
packets length was 871 for 24min of game session, smallest length is 49. i don't know why packets grow by time
i have netlimiter on my laptop, so i watched how much KB/s the game was used, it's about 6KB/s
Can you post the exact two rules you are using for hashlimit and connmark rule?
Sure:
$IPT -t mangle -A PREROUTING -p udp -m hashlimit --hashlimit-name udp_high_prio --hashlimit-above 75/sec --hashlimit-burst 37 --hashlimit-mode srcip,srcport,dstip,dstport -j CONNMARK --set-mark 0x55 -m comment --comment "connmark for udp"
$IPT -t mangle -A PREROUTING -p udp -m connmark ! --mark 0x55 -m conntrack --ctorigsrc 192.168.1.0/24 -m multiport ! --ports 53,80,443,60887 -m length --length 0:700 -j DSCP --set-dscp-class CS6 -m comment --comment "small udp flows"
Yes we talked about using conntrack avgpkt length, so the whole flow gets tagged but your match is just looking at length of the individual packet
So i should add avgpkt to conntrack, but how much is the length?
I understand that i should replace length with conntrack avgpkt, and avgpkt=700 ?
I think it's ok to use connbytes 0:700 bytes avgpkt mode. Together with the hashlimit it means the avg rate is less than 700bytes *75pkts/sec
1 Like
$IPT -t mangle -A PREROUTING -p udp -m connmark ! --mark 0x55 -m conntrack --ctorigsrc 192.168.1.0/24 -m multiport ! --ports 53,80,443,60887 -m connbytes --connbytes 0:700 --connbytes-dir both --connbytes-mode avgpkt -j DSCP --set-dscp-class CS6 -m comment --comment "small udp flows"
1 Like
Yes try that, if it needs tuning I'd adjust the hashlimit, maybe up to 120 pps with say 50 burst. VoIP will probably stay close to 50 packets each direction, but games might be closer to 100 or be more bursty during fast action etc.
1 Like
I will test it tomorrow, thanks.
the maximum packet size was 1208 for a voip or game, can't remember.