Ultimate SQM settings: Layer_cake + DSCP marks

Installing and Using OpenWrt Network and Wireless Configuration
502

Yes, I understand that concept! Perhaps I didn't explain myself properly as it can get pretty complicated at times. Let me try and reiterate.

Got egress QoS working, Client OS tags DSCP -> Router -> WAN

Got ingress QoS working with Veth0/1, Router tags DSCP -> Client.

So now I added a OpenVPN (tun0) connection (Hopefully properly) and sending the corresponding ingress over Veth0/1 as well, as I assume if I dont do this, the VPN would circumvent my global QoS ingress setup, no? My game would no longer get prioritized if the VPN runs outside of the QoS setup.

Or is the preferred way to not let VPN run over our existing veth0/1 and instead make a seperate SQM instance for tun0 (vpn interface)

In that case, i could imagine they are not 'aware' of each other and global QoS would not work as expected as the game and all my other traffic is not QoS'ed in the same tins.

Question 2
Ok, theres that, the other thing im trying to figure out in the current 'VPN ingress over veth0/1' method, before i used VPN at all, and had everything setup, i could see on my client with wireshark both incoming and outgoing UDP packets as EF marked, but I dont see incoming packets EF marked with the current VPN setup.

I assumed since the Openwrt has the VPN connection initiated on TCP port 443, i could simply make a rule to mark packets orinating from port 443 to be marked EF, but I dont see it happening when I run the game now,

So either I'm missing something, or not fully grasping it. The iptables code are in my previous post.

I hope I sort of clarified my previous post.

503

Yes correct for exactly the reasons you say.

It seems like you should, maybe check your rules to see if there is something about them that excludes your game by accident or whatever.

504

Hi
For testing purpose duplicate your iptables rules but change -A PREROUTING to -A POSTROUTING then
capture with tcpdump on tun0, br-lan and wan interface. then look at dscp marks.

505

Edit: Ok so think I know what is going on (My theory anyway).

So when I mark OpenVPN packets on port 443 with EF they do get marked, but that is the encapsulated traffic, the de-encapsulated traffic forwarded to my client comes from within the tunnel and thus not get marked by EF.

Since the tunnel is only occupied by the game data stream, the tunnel itself is still QoS'ed but not the data inside so thats why on client side i dont see the incoming packets with EF DSCP, and also not see port 443 as source port, and still see the actual external ip of game servers in wireshark. Either way the result should be that the VPN is in fact QoS'ed.

$IPT -t mangle -A POSTROUTING -p TCP --sport 443 -j DSCP --set-dscp-class EF

I can still mark the de-encapsulated traffic originating from the tunnel with the following, even if it is just to see them in Wireshark:

$IPT -t mangle -A FORWARD -i tun+ -j DSCP --set-dscp-class EF

At least the tunneling works, kind of nice having both EF (46) be high priority and at the same time function as a tool to redirect traffic to VPN :slight_smile:

1 Like
506

Yes you must mark the tunnel traffic with your -i tun rule as well as the encrypted packets with your port 443 rule otherwise incoming packets over the tunnel could get stalled in the veth device.

1 Like
507

Hi again.
I'm still have a small problem which is rules interact like CS6 and AF41, the only problem now is bulk traffic rule:

$IPT -t mangle -A PREROUTING -p tcp -m multiport --ports 21,25,80,81,443,444,554,8000,8080,8409,60887,27014:27050 -m connbytes --connbytes 500000: --connbytes-dir both --connbytes-mode bytes -j CONNMARK --set-mark 0x1 #60887 bittorrent
$IPT -t mangle -A PREROUTING -p udp -m multiport --ports 21,25,80,81,443,444,554,8000,8080,8409,60887 -m connbytes --connbytes 500000: --connbytes-dir both --connbytes-mode bytes -j CONNMARK --set-mark 0x1
$IPT -t mangle -A PREROUTING -m connmark --mark 0x1 -j DSCP --set-dscp-class CS1

I tried something like this:
$IPT -t mangle -A PREROUTING -m connmark ! --mark 0x2 -m connmark ! --mark 0x3 -m connmark ! --mark 0x5 -m connmark ! --mark 0x6 -m connmark ! --mark 0x21 -m connmark ! --mark 0x41 -m connmark --mark 0x1 -j CONNMARK --set-mark 0xB

and i added a match to the original bulk rule:
$IPT -t mangle -A PREROUTING -p tcp -m connmark --mark 0xB -m multiport --ports 21,25,80,81,443,444,554,8000,8080,8409,60887,27014:27050 -m connbytes --connbytes 500000: --connbytes-dir both --connbytes-mode bytes -j CONNMARK --set-mark 0x1

but the problem is bulk rule will never catch anything.

One nice thing is that my isp is support ECN, i saw some sites have ECN enabled like this drama site.
I can say this setup is robust, cause i can get A+ with wrong bandwidth value!!! , but this is when i have speed above 1mbps. i ran this test
at morning when isp is giving near 8mbps.

One more strange thing, when SQM and script is off, while running a download + upload, icmp pings will remain low, what matter i do it will say low, i think isp is doing a prioritization to icmp packets, but the question
is which DSCP value they did use?
I tired to ping -Q $value 8.8.8.8 , but there's no difference, but i see a lower ping time by 2ms when using
0x80 and 0x88 !, also NTP packets is tagged with AF41.
The most strange thing, yesterday my family called someone via whatsapp, but the call ran over TCP :scream:
and not over udp, i don't know whyyyyy!!!

508

@moeller0
Here my MTU is back again to 1480, because my isp is using the main upstream provider.

« SpeedGuide.net TCP Analyzer Results » 
Tested on: 2019.01.16 10:12 
IP address: 185.51.221.124
Client OS/browser: Windows 10 (Chrome 61.0.3163.79) 
 
TCP options string: 020405a00103030801010402 
MSS: 1440 
MTU: 1480 
TCP Window: 132352 (not multiple of MSS) 
RWIN Scaling: 8 bits (2^8=256) 
Unscaled RWIN : 517 
Recommended RWINs: 63360, 126720, 253440, 506880, 1013760 
BDP limit (200ms): 5294kbps (662KBytes/s)
BDP limit (500ms): 2118kbps (265KBytes/s) 
MTU Discovery: ON 
TTL: 235 
Timestamps: OFF 
SACKs: ON 
IP ToS: 00000010 (2) 
    Precedence: 000 (routine)
    Delay: 0 (normal delay)
    Throughput: 0 (normal throughput)
    Reliability: 0 (normal reliability)
    Cost: 1 (low cost)
    Check bit: 0 (correct)
DSCP (DiffServ): CS0 000000 (0) - class 0, default traffic (RFC 2474).
509

It's hard to know why without the full rule set. If you look at the firewall status, are these bulk rules getting hit at all? Perhaps if they're not last they are getting hit, but then overwritten?

Here's a strategy that might be useful:

start by marking everything CS0

then have rules that check if a packet is marked CS0 and if it matches some rule then mark it CS6 or AF41 etc with your existing rules (this avoids the confusion of a bazillion different connmarks)

at the end, check to see if it matches the bulk category and mark it CS1 (you might want to downgrade some marks, so you need to write this rule carefully not just check for CS0 mark and then go)

This might be easier than your connmark method as you won't have to check so many different interactions. The assumption is that almost all rules are supposed to hit just ONCE. Only the bulk rule is meant to downgrade from previous mark, and then only for things that you think are eligible.

1 Like
510

this is the none working full rule set(i mean bulk rule):
i just removed some rules to make script shorter here.

Script 
#A robust 2 rules to detect realtime traffic
$IPT -t mangle -A PREROUTING -p udp -m hashlimit --hashlimit-name udp_high_prio --hashlimit-above 120/sec --hashlimit-burst 50 --hashlimit-mode srcip,srcport,dstip,dstport -j CONNMARK --set-mark 0x55 -m comment --comment "connmark for udp"
$IPT -t mangle -A PREROUTING -p udp -m connmark ! --mark 0x55 -m conntrack --ctorigsrc 192.168.1.0/24 -m multiport ! --ports 53,5353,80,443,8080,60887 -m connbytes --connbytes 0:700 --connbytes-dir both --connbytes-mode avgpkt -j CONNMARK --set-mark 0x6 -m comment --comment "small udp connection"
$IPT -t mangle -A PREROUTING -m connmark --mark 0x6 -j DSCP --set-dscp-class CS6
##################
#TCP SYN,ACK flows
##################
# prioritize inbound and outbound ACK pkts according to size
# empty ipv4 ack pkts <64 appear very often but empty ipv6 ack pkts are larger and fall into smaller priority class
#$IPT -t mangle -A PREROUTING -p tcp -m tcp --tcp-flags ALL ACK -m length --length :64 -j CONNMARK --set-mark 0x3
#$IPT -t mangle -A PREROUTING -p tcp -m tcp --tcp-flags ALL ACK -m length --length 65:89 -j CONNMARK --set-mark 0x3

#Make sure ACK packets get priority (to avoid upload speed limiting our download speed)
$IPT -t mangle -A PREROUTING -p tcp --tcp-flags ALL ACK -m length --length :128 -j CONNMARK --set-mark 0x3
$IPT -t mangle -A PREROUTING -p tcp --tcp-flags ALL SYN -m length --length :666 -j CONNMARK --set-mark 0x3
$IPT -t mangle -A PREROUTING -m connmark --mark 0x3 -j DSCP --set-dscp-class CS3

#mid size ACK
$IPT -t mangle -A PREROUTING -p tcp -m tcp --tcp-flags ALL ACK -m length --length 90:159 -j CONNMARK --set-mark 0x2
$IPT -t mangle -A PREROUTING -p tcp -m tcp --tcp-flags ALL ACK -m length --length 160:255 -j CONNMARK --set-mark 0x2
$IPT -t mangle -A PREROUTING -m connmark --mark 0x2 -j DSCP --set-dscp-class CS2

# large ack pkts carry data and dont need the same priority
$IPT -t mangle -A PREROUTING -p tcp -m tcp --tcp-flags ALL ACK -m length --length 256:511 -j CONNMARK --set-mark 0x2
########################################
# Streaming Media (videos/audios)
########################################
#Known video streams sites like youtube,netflix
$IPT -t mangle -A PREROUTING -m set --match-set vidstream src,dst -j CONNMARK --set-mark 0x41 -m comment --comment "vidstream ipset"
# some iptv provider's use this port
$IPT -t mangle -A PREROUTING -p tcp -m multiport --port 1935 -j CONNMARK --set-mark 0x41 -m comment --comment "some iptv streaming service"
$IPT -t mangle -A PREROUTING -m connmark --mark 0x41 -j DSCP --set-dscp-class AF41

##set dscp tag for our ipset usrcdn #those cdn's have an isolated speeds by my ISP
$IPT -t mangle -A PREROUTING -m set --match-set usrcdn src,dst -j CONNMARK --set-mark 0x21 -m comment --comment "usrcdn ipset"
$IPT -t mangle -A PREROUTING -m connmark --mark 0x21 -j DSCP --set-dscp-class AF21

########################################
# Best Effort (browsing/services)
########################################
#Small packet is probably interactive or flow control
$IPT -t mangle -A PREROUTING -m connmark ! --mark 0x2 -m connmark ! --mark 0x3 -m connmark ! --mark 0x5 -m connmark ! --mark 0x6 -m connmark ! --mark 0x21 -m connmark ! --mark 0x41 -m length --length 0:500 -j CONNMARK --set-mark 0x3

#Small packet connections: multi purpose (don't harm since not maxed out)
$IPT -t mangle -A PREROUTING -m connmark ! --mark 0x2 -m connmark ! --mark 0x3 -m connmark ! --mark 0x5 -m connmark ! --mark 0x6 -m connmark ! --mark 0x21 -m connmark ! --mark 0x41 -m connbytes --connbytes 0:250 --connbytes-dir both --connbytes-mode avgpkt -j CONNMARK --set-mark 0x3
#########################################
# Background Traffic (Bulk/file transfer)
#########################################
#bulk traffic ipset, like windows udates and steam updates/downloads
$IPT -t mangle -A PREROUTING -m set --match-set bulk src,dst -j CONNMARK --set-mark 0x1 -m comment --comment "bulk traffic ipset"

#A robust rule to detect downloads/uploads and torrents!
$IPT -t mangle -A PREROUTING -m connmark ! --mark 0x2 -m connmark ! --mark 0x3 -m connmark ! --mark 0x5 -m connmark ! --mark 0x6 -m connmark ! --mark 0x21 -m connmark ! --mark 0x41 -m connmark --mark 0x1 -j CONNMARK --set-mark 0xB

$IPT -t mangle -A PREROUTING -p tcp -m connmark --mark 0xB -m multiport --ports 21,25,80,81,443,444,554,8000,8080,8409,60887,27014:27050 -m connbytes --connbytes 500000: --connbytes-dir both --connbytes-mode bytes -j CONNMARK --set-mark 0x1 #60887 bittorrent

$IPT -t mangle -A PREROUTING -m connmark --mark 0x1 -j DSCP --set-dscp-class CS1

Nothing is hitting bulk rules(stay 0), i don't think they are getting overwritten!

I think you mean something like this:

$IPT -t mangle -A PREROUTING -p tcp -m multiport --port 1935 -j DSCP --set-dscp-class AF41 -m comment --comment "some iptv streaming service"

$IPT -t mangle -A PREROUTING -p tcp -m dscp ! --dscp  34 -m multiport --ports 21,25,80,81,443,444,554,8000,8080,8409,60887,27014:27050 -m connbytes --connbytes 500000: --connbytes-dir both --connbytes-mode bytes -j DSCP --set-dscp-class CS1

I see the connmark method is complicated and eat more cpu for nothing, i see that bulk rule should downgrade CS0 cause i need other rules stay as it.

EDIT:
I think it's working now with DSCP method instead of connamark, but let me test for sometime!

511

@dlakelan
I think i catch it, my isp is using DSCP (50) for ICMP (showed in the wireshark, but i don't know if wireshark shows values as
hex or dec)

512

For the first time, here is my flent results:
SQM off:
sqm-off
SQM on:
SQM-on
I'm not sure how to read it, but it's look's funny!

1 Like
513

These look both somewhat terrible and note how the periods of low download rates coincide with (painfully) increased latency.

514

I think this duo to my vm, i have some connectivity issues inside the vmware(i feel vmware v15 is buggy).
That's why results is terrible!

515

what about this:
____

516

This looks better, but do you notice the weird periodic modulation of the mean latency? I wonder what drives this.
Also the upload for all plots is less stable than I have seen in the past.

517

Is there a DASH stream going on your network simultaneously? That'd modulate the ping periodically.

518

I think this is because i have connectivity issues on the latest vmware(also vm seems buggy), second thing is
something related to isp, at morning before the 9am my speed is about 8Mbps(unstable speed) after this time it will be 1 mbps, when speed is above 1mbps bufferbloat is A+, and when speed is 1mbps bufferbloat is C.
*Note: i don't use actual values for my download and upload, cause you all know that i have different speeds
for each service like(normal download,youtube, facebook and cache), the same is applied to upload.
i use 16mbps for download and 8mbps for upload(i didn't changed bandwidth values for all tests!).
One more strange thing: When my script and SQM off bufferbloat= C, when SQM on and script off bufferbloat=A, when SQM on + script on bufferbloat=A+, so in this case script is helping to maintain bufferbloat

here's my tests:
SQM on + script ON

SQM off script off

SQM on Script off:

No, network was idle when i took those measurements!
Also remember youtube is isolated!

519

As long as your latency is stable for latency sensitive streams, it doesn't much matter of it spiked for other streams. Can you plot just the latency for high priority tags? That seems tricky.

Because of your scenario with multiple time varying bandwidths it is much harder to know how well your scheme works. You might try virtualbox or even an SBC as your testing platform if you can't stabilize vmware

520

I can say it's very stable for sensitive streams(with wrong bandwidth set), even the max spike i saw is about 30ms.

i want to know something, does flent use HTTPS to do a measurement ?, cause https is really bad for unknown reason:

I will download virtual box, SBC= server based computing?

521

Single board computer... Like espressobin, odroid, orange pi, something with a gig Ethernet port that isn't crippled, not a raspberry pi with it's USB based Ethernet.