Help me update my HFSC shaper scripts for fw4/nftables

Installing and Using OpenWrt Network and Wireless Configuration
565

Nice

If you have your GAMEUP and GAMEDOWN set to the default formula, it uses 15 percent of your bandwidth and adds 400 kbps to it as a buffer. This bandwidth is reserved for your gaming realtime class.

When using bfifo, the script uses the following tc command for the realtime gaming queue:

tc qdisc add dev "$DEV" parent 1:11 handle 10: bfifo limit $((MAXDEL * gamerate / 8))

This command adds a byte-limited FIFO queue, with the queue size being determined by the specified maximum delay (MAXDEL) and the defined gamerates for upload (GAMEUP) and download (GAMEDOWN).

For example:

Assume your download rate (DOWNRATE) is 100000 kbps and MAXDEL is 10 ms:

GAMEDOWN=$((DOWNRATE*15/100+400)) = 15400
GAMEDOWN=$((100000*15/100+400)) = 15400


Bfifo = (MAXDEL * gamerate / 8) = 19250
Bfifo = (10 * 15400 / 8) = 19250

So in this example, the byte-limited FIFO queue would have a size of 19250 bytes. This setup ensures that the gaming traffic has a reserved bandwidth of 15400 kbps, with a queue that can buffer up to 19250 bytes.

But depending on your bandwidth be careful how low you set your MAXDEL

Example with Low Bandwidth and Low MAXDEL

Assume your download rate (DOWNRATE) is 5000 kbps and MAXDEL is set to 1 ms:

GAMEDOWN=$((5000*15/100+400)) = 1150

Bfifo = (1 * 1150 / 8) = 143,75

In this example, the bfifo queue would be limited to 143.75 bytes and gaming packets are probably are bigger than that. Such a small queue size is not sufficient to ensure a stable network connection for gaming, as it would fill up instantly, leading to packet loss and higher latency.

I was considering implementing a formula to prevent the bfifo size from being set too low when dealing with low bandwidths.

You can also test if the bfifo MAXDEL is set too low.

  1. Install watch: opkg update && opkg install procps-ng-watch
  2. Start and play your game
  3. Use this command and see if packets are dropped frequently watch -n 2 'tc -s qdisc | grep -A 2 "parent 1:11"'
    This executes the tc command every 2 seconds and filters only the realtime gaming class (1:11) for up and download. If you see your sent packets increasing but no packets dropped, you are probably good to go...
qdisc bfifo 10: dev ifb-eth1 parent 1:11 limit 19250b
 Sent 130 bytes 1 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
566

This is unloaded latency I guess... this then simply shows that cake is considerably more resource (read CPU) hungry than HFSC+bfifo... Question, which queue's latency did you probe here, that is how are externally incoming ICMP probes classified, especially those terminating at the router?

However the IMHO more interesting test is what happens under full load... my guess is that, assuming this tests the gaming class latency, HFSC will still deliver lower latency than cake, at least with default cake parameters.

That said, I doubt that a few (single digit) milliseconds up or down are going to make a big difference, but I am not an avid on-line gamer so might more relaxed in this regard.

567

in that picture i ran several speed tests and bufferbloat tests latency barely increase at all with hfscscript script. x86 doesn't break sweat, system load is very low with or without cake running.

Zia

568

Well, in what priority class did you put ICMP responses (what thinkbroadband uses for its latency probes) in the HFSC and in the cake case? It looks like you put that into the high priority class with HFSC?

But that seems also true for the cake case, what did happen is that the baseline delay increased by a few milliseconds with cake.

569 

qdisc bfifo 10: dev ifb-pppoe-wan parent 1:11 limit 84625b
 Sent 125198671 bytes 128708 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
--
qdisc bfifo 10: dev pppoe-wan parent 1:11 limit 8593b
 Sent 48652263 bytes 97083 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0

biffo set to 5ms
Zia

570

For generic traffic classes the rule of thumb is to always allow for 1-2 full sized packets in the queue, if your GAME class only ever sees e.g. X byte packets than 2*X is a sane lower limit for a bfifo... BTW, nit sure what happens if the bfifo is smaller than a packet to be squeezed in...

571

Hmm not sure i put icmp in a class, just defualt hfsc script.
only thing i tweaked was bfifo maxdel,went from 20,10 then 5.

Zia

572

HFSC and cake work in a similar manner with respect to the shaper but with some differences. The difference with HFSC is that its service curve can be arranged to be nonlinear and have an 'initial rate' different from the long term rate, whereas cakes appears to be purely linear with a constant rate for all time.

Imagine both are connected to a quiescent link. Two packets come in essentially at the same time. Both shapers will immediately send the first packet, then HFSC will use the short-term rate to calculate the time at which the second packet goes out, whereas cake has a single rate at which it operates and will schedule the second packet at that rate.

HFSC can have a brief period where its sending rate is either faster or slower than the long-term rate. So if the temporary rate is higher, HFSC will send the packet out sooner, if the temporary rate is lower, HFSC will delay the packet longer than cake.

In addition to this behavior, the HFSC has a "realtime" mode where it blocks all non-realtime traffic when realtime traffic is available, until the realtime traffic has blocked due to exhausting its service curve.

What this means is if you have a game download or surfing at the "non-realtime" priority, and a game control packet comes in and is scheduled into the realtime queue... no matter what else is going on HFSC is going to stop all other traffic and service the realtime game traffic... until the service curve is hit.

If your game generates regular clock-ticks, and you've set your GAMEUP rate above the rate that the game generates packets, this means your game packets will never be delayed by anything other than other game packets.

Also, if your game sends say 2 packets in short succession, HFSC will space those packets out in time a little bit rather than dumping them both one immediately after the other the way a token bucket filter would (cake also does this but generally with a longer delay).

HFSC is, in theory and when properly configured, a better qdisc for handling hard-real-time requirements, such as gaming, or controlling a robot or delivering audio packets for realtime conversations. In general the advantage will primarily occur when you have lower bandwidth. For example if you need to send 500kbps of gaming packets on a gigabit link, even if you want to have the maximum delay be 1ms you could send 83 full sized TCP packets first and then send the gaming packet, and still have no issue.

On the other hand, on a 10Mbps uplink from a DOCSIS provider... that'd be more like 0.83 full size TCP packets, meaning HFSC's advantages will be more likely to be real rather than purely theoretical. As you transition to something like 30 or 100Mbps uplink, you can get more sloppy, at 30 Mbps you can have 2.5 full MTU packets of slop, at 100 you can have 8 of them...

If you have an uplink below say 50Mbps you are likely to actually feel the HFSC advantage, if you have an uplink faster than 100Mbps you are less likely to feel the advantages.

1 Like
573

But that is not exactly clear, as cake will also 'burst' into the next layer a bit (you need to do this to allow for a bit of scheduling slack for cake, otherwise you can not achieve the requested shaper rate). Also this does not explain the static increase in RTT seen in the thinkbroadband plots, as such a mechanism would introduce primarily more jitter (which at the resolution of the thinkbroadband trace seems not to be the case).
Cake also offers higher priority tiers, but for ICMP probes coming from the outside as far as I can tell cake and this HFSC script will just keep them inside the default (non-realtime?) tier.

I severely doubt that. Sure when I read the HFSC paper years ago, I was convinced it was the best shaper since sliced bread as well, but that was in a time when CFQ and TBF where often driven by a 100Hz ticks... by now at least HTB and cake are not driven by periodic timers but schedule themselves based on their next transmit slot requirement. IMHO that removes much of the differences in macroscopic behavior.
That said, I see and accept the, let's say ~5ms larger delay in the thinkbroadband traces with cake versus HFSC, but these might be caused by the competing traffic in the default priority tier and the resulting drr++ round robin of the active flows (but to verify/falsify that hypothesis I would need to see the 'tc -s qdisc' output from a quit epoch as well as an epoch showing the same amount of background traffic, as well as information about how the ingress traffic was handled for HFSC and cake respectively, as IFB or veth or ingress on br-lan, can make a difference as well independent of the actual traffic shaper).

574

Right, I said "in theory and when properly configured". It also will make a bigger difference if you have considerable bandwidth constraints. Like, for example, DSL at 768kbps uplink and are trying to play a game with 300kbps packet stream on a 15ms tick while your roomate uploads stuff via TCP to a cloud storage or something. which was the original case when I built the script.

If you have 10Gbps links you can literally put hundreds of packets ahead of your game packet and no-one will ever notice (1.2 microseconds per 1500 byte packet, so you could do nearly a thousand of those and not even induce an extra ms of delay).

But if you're gaming at say 300kbps on a 768 kbps uplink, one extra 1500 byte packet is about 15.6ms of delay, which is about equivalent to one dropped packet. Under those conditions, the realtime link freezing out the TCP packet entirely and sending the game packet on a slightly accelerated schedule to ensure you have a chance to make your control deadline is valuable.

As mentioned in other threads, if you have 3000kbps uplink or so you should be able to game with pretty much any modern qdisc and keep your game from going too far out of whack.

The tight tolerances HFSC is potentially capable of make much less difference at speeds in the 10Mbps + range which are fortunately relatively common these days. Though there are still people who have low-tier DOCSIS who might have somewhat less than that on their uplink.

For higher speeds, HFSC still allows you to have somewhat more control over the behavior of the various tiers. This is valuable if you want to control stuff somewhat more tightly, such as intentionally stalling out torrenting or you want hard real-time on your audio packets and more soft real time on video packets or whatever.

575

@Hudra - in dscptag.nft

#downgrade udp going faster than 450 pps, probably not realtime traffic ip protocol udp ip dscp > cs2 add @udp_meter4 {ip saddr . ip daddr . udp sport . udp dport limit rate over 450/second} counter ip dscp set cs2 counter ip6 nexthdr udp ip6 dscp > cs2 add @udp_meter6 {ip6 saddr . ip6 daddr . udp sport . udp dport limit rate over 450/second} counter ip6 dscp set cs2 counter

needs to be removed from the script else when $udp_rate_limit_rules are "yes" appears twice.

576

You are right, thanks man! Just fixed it.

The issue only occurred on my master branch. When I made a few changes on the fly last time, I forgot to remove the hardcoded rules in the dscptag.nft file. Should be working as intended now...

1 Like
577 

root@OpenWrt:~# /etc/SimpleHFSCgamerscript.sh
Config files have been added to sysupgrade.conf for preservation.
Error: Failed to load TC action module.
We have an error talking to the kernel

This script prioritizes the UDP packets from / to a set of gaming
machines into a real-time HFSC queue with guaranteed total bandwidth

Based on your settings:

Any idea what this error means?

Does uninstall script still work?

578

It seems like you don't have kmod-sched-ctinfo installed. This module should actually be automatically installed by the init script. Is it possible that you are using a "new" version of the main script (with ctinfo) with an old version of the init script?
What is the output of:

grep 'REQUIRED_PACKAGES=' /etc/init.d/SimpleHFSCgamerscript

Please do not use the uninstall script when using my version. Instead, refer to the uninstallation section in the README of my repository:

579 

root@OpenWrt:~# grep 'REQUIRED_PACKAGES=' /etc/init.d/SimpleHFSCgamerscript
REQUIRED_PACKAGES="kmod-sched ip-full kmod-veth tc-full kmod-netem kmod-sched-ctinfo kmod-ifb kmod-sched-cake"
root@OpenWrt:~#

i see have must copied the experimental cake install script.
580

mmmh, ctinfo is there an should be installedā€¦

Actually you donā€™t need the install script anymore. Refer to the readme on GitHub for setup instructions

581

i think there is a issue keep giving me same eror when running script as above, following Github instructions
doesn't install any of the packages for some reason
Also can you create install script for Dev one soo when you install that version in auto install all necessay files,packages?

Zia

582

So I thought everything was working fine for you. What has changed since then?

I just ran some tests and uninstalled/installed the script a few times with both versions (master branch and dev branch). I never encountered any issues following the instructions in the GitHub readme (uninstall/install).

Are you using official OpenWrt? What is the ouput of:

ubus call system board

As I told you before there is no need for an install script. Everytime you start the init script it checks if all necessary packages are installed... If not they will be installed. This normally happens when starting the script the first time....

The version on my dev branch should install all the necessary packages in the same way.

1 Like
583

Hmm sry for being a pain am just testing.i wiped the drive reinstalled 23.5.3 and then installed dev branch,and got the above error, for some reason not pulling the files for me..obviously something i am doing wrong,can manaually install packages works fine but will try again.
thank you

root@OpenWrt:~# ubus call system board
{
        "kernel": "5.15.150",
        "hostname": "OpenWrt",
        "system": "Intel(R) Core(TM) i3-6100 CPU @ 3.70GHz",
        "model": "Dell Inc. OptiPlex 5050",
        "board_name": "dell-inc-optiplex-5050",
        "rootfs_type": "ext4",
        "release": {
                "distribution": "OpenWrt",
                "version": "23.05.3",
                "revision": "r23809-234f1a2efa",
                "target": "x86/64",
                "description": "OpenWrt 23.05.3 r23809-234f1a2efa"
        }
}
root@OpenWrt:~#

Zia

584

Do the following:

Uninstall the script:

Uninstallation

To completely remove SimpleHFSCgamerscript from your OpenWrt router:

  1. Stop the script:

/etc/init.d/SimpleHFSCgamerscript stop

  1. Disable the script from starting on boot:

/etc/init.d/SimpleHFSCgamerscript disable

  1. Remove the script files:

rm /etc/init.d/SimpleHFSCgamerscript && rm /etc/SimpleHFSCgamerscript.sh && rm /etc/config/hfscscript

  1. Reboot your router to clear any remaining settings: reboot

Then reinstall the script again as instructed in the readme:

When you reach step 6 and start the script for the first time, show me the entire output.