QoSmate: (Yet Another) Quality of Service Tool for OpenWrt

No QoSmate: https://www.waveform.com/tools/bufferbloat?test-id=4eb2d423-b5b3-41a5-9d65-8fa9c40294d4

No QoSmate w SW Offload: https://www.waveform.com/tools/bufferbloat?test-id=f4830bd3-a138-4693-b66b-44a750991408

QoSmate (without offload) set to 150000kbps / 150000kbps: https://www.waveform.com/tools/bufferbloat?test-id=09cd4093-daf9-4dd5-8ab2-e456a2a89c0f

QoSmate (without offload) set to 150000kbps / 150000kbps:

.\iperf3.exe --bidir -c ping.online.net -p 5206
Connecting to host ping.online.net, port 5206
[  5] local 192.168.7.137 port 56384 connected to 51.158.1.21 port 5206
[  7] local 192.168.7.137 port 56385 connected to 51.158.1.21 port 5206
[ ID][Role] Interval           Transfer     Bitrate
[  5][TX-C]   0.00-1.01   sec  7.62 MBytes  63.1 Mbits/sec
[  7][RX-C]   0.00-1.01   sec  4.75 MBytes  39.3 Mbits/sec
[  5][TX-C]   1.01-2.00   sec  2.88 MBytes  24.4 Mbits/sec
[  7][RX-C]   1.01-2.00   sec  5.50 MBytes  46.7 Mbits/sec
[  5][TX-C]   2.00-3.00   sec  1.88 MBytes  15.7 Mbits/sec
[  7][RX-C]   2.00-3.00   sec  7.75 MBytes  64.8 Mbits/sec
[  5][TX-C]   3.00-4.01   sec  2.12 MBytes  17.8 Mbits/sec
[  7][RX-C]   3.00-4.01   sec  8.50 MBytes  71.1 Mbits/sec
[  5][TX-C]   4.01-5.01   sec  1.88 MBytes  15.7 Mbits/sec
[  7][RX-C]   4.01-5.01   sec  7.88 MBytes  65.9 Mbits/sec
[  5][TX-C]   5.01-6.01   sec  2.25 MBytes  18.8 Mbits/sec
[  7][RX-C]   5.01-6.01   sec  8.12 MBytes  68.0 Mbits/sec
[  5][TX-C]   6.01-7.01   sec  1.88 MBytes  15.7 Mbits/sec
[  7][RX-C]   6.01-7.01   sec  7.50 MBytes  62.7 Mbits/sec
[  5][TX-C]   7.01-8.00   sec  1.00 MBytes  8.49 Mbits/sec
[  7][RX-C]   7.01-8.00   sec  12.4 MBytes   105 Mbits/sec
[  5][TX-C]   8.00-9.00   sec  1.25 MBytes  10.5 Mbits/sec
[  7][RX-C]   8.00-9.00   sec  7.25 MBytes  60.7 Mbits/sec
[  5][TX-C]   9.00-10.00  sec  1.62 MBytes  13.6 Mbits/sec
[  7][RX-C]   9.00-10.00  sec  9.88 MBytes  82.7 Mbits/sec

No QoSmate:

.\iperf3.exe --bidir -c ping.online.net -p 5206
Connecting to host ping.online.net, port 5206
[  5] local 192.168.7.137 port 56404 connected to 51.158.1.21 port 5206
[  7] local 192.168.7.137 port 56405 connected to 51.158.1.21 port 5206
[ ID][Role] Interval           Transfer     Bitrate
[  5][TX-C]   0.00-1.00   sec  12.0 MBytes   101 Mbits/sec
[  7][RX-C]   0.00-1.00   sec  13.5 MBytes   113 Mbits/sec
[  5][TX-C]   1.00-2.00   sec  15.5 MBytes   130 Mbits/sec
[  7][RX-C]   1.00-2.00   sec  17.2 MBytes   144 Mbits/sec
[  5][TX-C]   2.00-3.00   sec  16.5 MBytes   138 Mbits/sec
[  7][RX-C]   2.00-3.00   sec  17.2 MBytes   144 Mbits/sec
[  5][TX-C]   3.00-4.00   sec  15.1 MBytes   127 Mbits/sec
[  7][RX-C]   3.00-4.00   sec  17.9 MBytes   150 Mbits/sec
[  5][TX-C]   4.00-5.00   sec  16.2 MBytes   136 Mbits/sec
[  7][RX-C]   4.00-5.00   sec  17.6 MBytes   148 Mbits/sec
[  5][TX-C]   5.00-6.00   sec  17.9 MBytes   150 Mbits/sec
[  7][RX-C]   5.00-6.00   sec  17.6 MBytes   148 Mbits/sec
[  5][TX-C]   6.00-7.01   sec  16.8 MBytes   140 Mbits/sec
[  7][RX-C]   6.00-7.01   sec  16.5 MBytes   138 Mbits/sec
[  5][TX-C]   7.01-8.01   sec  17.4 MBytes   145 Mbits/sec
[  7][RX-C]   7.01-8.01   sec  18.8 MBytes   157 Mbits/sec
[  5][TX-C]   8.01-9.01   sec  17.0 MBytes   142 Mbits/sec
[  7][RX-C]   8.01-9.01   sec  17.0 MBytes   142 Mbits/sec
[  5][TX-C]   9.01-10.01  sec  17.2 MBytes   144 Mbits/sec
[  7][RX-C]   9.01-10.01  sec  18.1 MBytes   152 Mbits/sec

And for reference waveform tests with sqm-scripts, @Hudra

sqm-scripts fq_codel/simplest set to 150000kbps / 150000kbps: https://www.waveform.com/tools/bufferbloat?test-id=3ee45eaf-56f9-4fe9-9d64-f7777d075bf0

sqm-scripts fq_codel/simplest set to 135000kbps / 135000kbps: https://www.waveform.com/tools/bufferbloat?test-id=ae4a4e60-3e2b-4419-8091-8a1190c14104

sqm-scripts fq_codel/simplest set to 120000kbps / 120000kbps: https://www.waveform.com/tools/bufferbloat?test-id=8cad27be-1e84-4f69-b6b7-a67b2d9df524

It's consistent, has happened everytime I have used it and happened again those three times I gave it a test. It gave me the results below:

  • Download 131202kbps / Upload 130878kbps
  • Download131283kbps / Upload 130437kbps
  • Download 68292kbps / Upload 27936kbps
cat /tmp/qosmate_auto_setup_output.txt
Starting qosmate non-interactive auto-setup...
Stopping qosmate for accurate speed test results...
Stopping service qosmate...
Error: Could not process rule: No such file or directory
delete table inet dscptag
                  ^^^^^^^
Reloading network service...
Detected WAN interface: eth0
speedtest-go is already installed. Using it for the speed test.
Running speed test... This may take a few minutes.
Speed test results:
Download speed: 75.88
92.52 Mbit/s
Upload speed: 31.04
38.43 Mbit/s
QoS configuration:
DOWNRATE: 68292 kbps (90% of measured download speed)
UPRATE: 27936 kbps (90% of measured upload speed)
No gaming device IP added.
Configuration updated. New settings:
    option WAN 'eth0'
    option DOWNRATE '68292'
    option UPRATE '27936'
Auto-setup complete. qosmate has been configured with detected settings.
To apply these changes, please restart qosmate by running: /etc/init.d/qosmate restart
root@routteri:~# ubus call system board
{
        "kernel": "6.6.69",
        "hostname": "routteri",
        "system": "MediaTek MT7621 ver:1 eco:3",
        "model": "Ubiquiti EdgeRouter X",
        "board_name": "ubnt,edgerouter-x",
        "rootfs_type": "squashfs",
        "release": {
                "distribution": "OpenWrt",
                "version": "24.10.0-rc5",
                "revision": "r28304-6dacba30a7",
                "target": "ramips/mt7621",
                "description": "OpenWrt 24.10.0-rc5 r28304-6dacba30a7",
                "builddate": "1736026537"
        }
}
root@routteri:~# service qosmate auto_setup
Starting qosmate auto-setup...
Stopping qosmate for accurate speed test results...
Stopping service qosmate...
Error: Could not process rule: No such file or directory
delete table inet dscptag
                  ^^^^^^^
Reloading network service...
Detected WAN interface: eth0
Do you want to run a speed test or enter speeds manually? [test/manual]
test
This will run a speed test to configure qosmate. Do you want to continue? [y/N]
y
speedtest-go is already installed. Using it for the speed test.
Running speed test... This may take a few minutes.
Speed test results:
Download speed: 74.27
92.03 Mbit/s
Upload speed: 31.60
39.06 Mbit/s
QoS configuration:
DOWNRATE: 66843 kbps (90% of measured download speed)
UPRATE: 28440 kbps (90% of measured upload speed)
Would you like to add a gaming device IP for prioritization? [y/N]

No gaming device IP added.
Configuration updated. New settings:
    option WAN 'eth0'
    option DOWNRATE '66843'
    option UPRATE '28440'
Auto-setup complete. qosmate has been configured with detected settings.
To apply these changes, please restart qosmate by running: /etc/init.d/qosmate restart
root@routteri:~# echo $?
0

If you’re using sqm-scripts with simplest, then HTB is used as the shaper. It’s possible that HTB requires fewer resources than HFSC, and the lower performance of your router might have a greater impact when using HFSC. For example, Cake would require even more resources than both.

Could you quickly share your configuration so we can rule out whether ACK limiting in the download direction is affecting your bandwidth?

cat /etc/config/qosmate

I’ll look into the speed test issue later when I have more time.

1 Like

It is clearly capped by CPU resources. IF any QoS setting improves over A 150/150 with offload go for it, but I doubt it is likely to get any improvement.

1 Like
root@routteri:~# cat /etc/config/qosmate

config global 'global'
        option enabled '1'

config settings 'settings'
        option WAN 'eth0'
        option DOWNRATE '65000'
        option UPRATE '65000'
        option ROOT_QDISC 'hfsc'

config advanced 'advanced'
        option PRESERVE_CONFIG_FILES '1'
        option WASHDSCPUP '1'
        option WASHDSCPDOWN '1'
        option BWMAXRATIO '20'
        option UDP_RATE_LIMIT_ENABLED '0'
        option TCP_UPGRADE_ENABLED '1'
        option UDPBULKPORT '51413,6881-6889'
        option TCPBULKPORT '51413,6881-6889'
        option NFT_HOOK 'forward'
        option NFT_PRIORITY '0'

config hfsc 'hfsc'
        option LINKTYPE 'ethernet'
        option OH '44'
        option gameqdisc 'pfifo'
        option nongameqdisc 'fq_codel'
        option nongameqdiscoptions 'besteffort ack-filter'
        option MAXDEL '24'
        option PFIFOMIN '5'
        option PACKETSIZE '450'
        option netemdelayms '30'
        option netemjitterms '7'
        option netemdist 'normal'
        option pktlossp 'none'
        option netem_direction 'both'

config cake 'cake'
        option COMMON_LINK_PRESETS 'ethernet'
        option PRIORITY_QUEUE_INGRESS 'diffserv4'
        option PRIORITY_QUEUE_EGRESS 'diffserv4'
        option HOST_ISOLATION '1'
        option NAT_INGRESS '1'
        option NAT_EGRESS '1'
        option ACK_FILTER_EGRESS 'auto'
        option AUTORATE_INGRESS '0'

config custom_rules 'custom_rules'

Ok, thanks for the analysis. I knew it's not a power house but I was just wondering why the discrepancy between what I get down and up. I'd happily have the values switched over.

You have to do measurements over NAT,(iperf says 150/150 - enable qosmate, check iperf again and proportionate down) local ones may not represent whats achievable in normal data path.

1 Like

@sandberg I think what @brada4 means is that auto-setup is not the best method of checking effective bandwidth, because this way your router has to source or sink the data, which loads its CPU, so the measured result does not represent the effective bandwidth you will be getting in normal use. Rather, measure the effective bandwidth using iperf on a machine connected to your LAN.

2 Likes

@Hudra check new PR#35, now OpenWRT 22 derivatives can be supported by deoptimizing dscp | 0x80 into value map as proposed by affected user passing by.

1 Like

Thanks, @brada4, I saw it and left a comment.

For everyone's information: Once I/we have implemented the changes, there will be a legacy branch in the QoSmate repository, where OpenWrt versions < 23.05 (with nftables) will also be supported.

2 Likes

@sandberg To rule out that ACK rate limiting in the download direction is reducing your bandwidth, please try setting the "ACK Rate" to 0, then run your tests again and let me know if anything changes. If not, I’m afraid your router might not be powerful enough. In that case, I suggest the following options:

  1. Stick with sqm-scripts.
  2. Use QoSmate with HFSC, go through the fine-tuning process, and accept some bandwidth loss. Details: Fine-Tuning Process (Optional).
  3. Upgrade to a newer device.

Regarding your issues with the auto-setup: I looked into it and discovered a small bug that likely crept in during recent code optimizations. The auto-setup function checks which speed test tool is installed. If none is found, it installs either speedtest-go or python3-speedtest-cli based on available space.

When using speedtest-go (which was your case), the parser was capturing both lines of the speedtest output, leading to an invalid format in the output file. I’ve added a head -n1 command to capture only the first speed value, which matches the expected format for the UI. So this issue should now be fixed after updating to the latest version.

1 Like

I'm using QoSmate with CAKE. I would like to know if I have to use the wash command on ingress and egress or is it not necessary?

IIUC qosmate defaults to copyiung the egress DSCP mark to the conntrack database and then to ingress packets of the same connection/flow. This means these will affect which of cake's priority tins a packet will land up in.
If you configure wash for ingress then inside your lan you will only see DSCP 0 for packets coming from the internet, which might or might not be what you want.
If you configure wash for egress your ISP is not going to see your internal DSCPs but just CS0, it really depends on your ISP what is better, but often all CS0 is preferred by end users, but only you can know/test on your own link.

So here even with wash some packages are marked with cs6 and cs7, shouldn't they all be cs0?

I also used this command. And it says that the packages that are not cs0 are af21, not cs6 and cs7.

tcpdump -i br-lan -v -n '(ip and (ip[1] & 0xfc) >> 2 != 0)' or '(ip6 and (ip6[0:2] & 0xfc0) >> 4  != 0)'

Please show the packet capture... packets coming from your router and not the internet might still be marked, only traffic that goes through cake will be washed... so what shaper are you using cake or HFSC?

Which will show both incoming and outgoing packets... but ingress wash will not affect DSCPs set by clients in your network...

So I'm using CAKE and when I use tc -s qdisc even with the wash on upload and download some traffic still drops in cs6 and cs7.

Now, I am really confused about what you are running and what you are seeing ;).
Maybe help me out and show me:
a) the output of tc -s qdisc (so I can see the active cake confoiguration and also whether some other shapers might be active).
b) a bit of your packet caputers that show packets on your br-lan with unexpected DSCP values.

What do you mean with this? That you see CS6/7 on packet captures or that you have packets in cake's Voice tin?

I have packets falling into network control. Doesn't qosmate manipulate the packets before cake sees them?