CAKE w/ Adaptive Bandwidth [August 2022 to March 2024]

[moeller0]

@Lynx

True, but there are already other checks in place like the use of the checksum based on timestamp, and the saving in CPU cycles seemed too hard to resist:

https://github.com/lynxthecat/cake-autorate/commit/3e0037f527826b5df6f5743934d4e70028319b41

If you check the timing data validity in some other way, sure one can go, either the belt or the suspenders

@Lynx

Is the solution to simply increase connection_active_thr_kbps to 2000 or reduce the sustained_idle_sleep_thr_s to 30?

I would guess so, if the background load truely is occasionally bumping across 1Mbps then gently increasing this threshold seems the cheapest solution...

@Lynx

This means that any time the traffic exceeds connection_active_thr_kbps, even just for a single 200ms monitoring interval, the logic is reset, and in this way exfiltration from IoT devices can potentially preclude any sleep.

I guess my concern was/is that waking up from sleep is not instantaneous so we really do not want to do it too often, but maybe even once every 10 seconds might not be too often.... so I would probably set up logging with a long log size and just capture overnight what happens... the achieved load curve should be able to tell you whether gently increasing the threshold might help and whether 60 seconds might just be patently ill-fitting... say if you get a 5 Mbps spike every 60 seconds, maybe simply reducing sustained_idle_sleep_thr_s to 10 or even 5 might be enough.

[Lynx]

@moeller0

so I would probably set up logging with a long log size and just capture overnight what happens

I'll pipe tail -f /var/log/cake-autorate.primary.log to /tmp via gzip overnight and see what happens.

[Lynx]

So I obtained data spanning from 21:52 to 09:06. The network was quiet save for: some initial TV streaming in the evening; an adblock-lean blocklist update at 05:24; and some browsing with video traffic in the morning.

Load:

Load clipped to 10Mbit/s and below:

Average latency delta with my very lenient settings:

dl_owd_delta_thr_ms=80.0 # (milliseconds)
ul_owd_delta_thr_ms=80.0 # (milliseconds)

dl_avg_owd_delta_thr_ms=150.0 # (milliseconds)
ul_avg_owd_delta_thr_ms=150.0 # (milliseconds)

Close up of the typical 'exfiltration' data pattern:

Full data here: https://easyupload.io/bemcr5

[moeller0]

I guess for that pattern simply increasing the rate threshold to 1800 Kbps might do?

[buggz]

This is probably a dumb question...

Are you supposed to have "regular" sqm running, and then the cake-autorate.sh running also?
Or, just cake-autorate.sh?
Seems some of the sqm settings in luci affects, or is also used via cake-autorate.sh?

I'm "starting out" again, with a different router...

[Lynx]

cake-autorate simply continually updates the bandwidth of existing instances of cake, which can be setup using the existing traditional sqm packages in OpenWrt or my own lean and simple cake-qos-simple service script, which supports DSCPs.

[moeller0]

@buggz

Are you supposed to have "regular" sqm running, and then the cake-autorate.sh running also?

To elaborate, cake-autorate is agnostic to how you got your initial cake instances intitiated, leaving you with a wide choice between cake-qos-simple, qosify, or sqm-scripts, just to name the most well-known options.

[buggz]

I find if I disable sqm, and just run cake-autorate, I don't get the luci cake stats, only the sqm stat.

I also see that, auc, attended sysupgrade, wipes out the cake-autorate directory and service.
I have to use auc to upgrade, as my router is only OpenWRT available via SNAPSHOT release.

[efahl]

@buggz

I also see that, auc, attended sysupgrade, wipes out the cake-autorate directory and service.

Add the directory that you wish to save to /etc/sysupgrade.conf and test that it will be backed up with sysupgrade -l. Both auc and LuCI Attended Sysupgrade use this mechanism to carry files across upgrades.

$ echo "/etc/cake-autorate/" >> /etc/sysupgrade.conf
$ sysupgrade -l | grep 'cake'
...

[buggz]

Thanks!
I will try this next time.

@efahl

Add the directory that you wish to save to /etc/sysupgrade.conf and test that it will be backed up with sysupgrade -l. Both auc and LuCI Attended Sysupgrade use this mechanism to carry files across upgrades.

$ echo "/etc/cake-autorate/" >> /etc/sysupgrade.conf
$ sysupgrade -l | grep 'cake'

[Lynx]

Also do check out the README;

lynxthecat/cake-autorate/blob/master/INSTALLATION.md#preserving-cake-autorate-files-for-backup-or-upgrades

[Simon10362]

Hey again,

Hmm, I'm having some weird behaviour on my end with the program.

It's behaving odd right after boot up. It configures itself to the lowest limits set in the config and then takes a while to start actual shaping.

As in, 3 minutes after boot up, the speeds go to where they should be. Otherwise they sit near the bottom and it occurs at random, sometimes I've seen it on the outbound interfaces, and sometimes on the inbound.

My topology is such that I have a 4G antenna - LAN - internal firewall.

The 4G antenna is a router, the internal firewall is a checkpoint device, SMB 1590.
The network between them contains an openwrt machine on x86, running a i7-3770k.
It has 2 interfaces configured in bridge mode.

2 separate cake instances on either interface.

Any idea what could be causing a problem with the delay post-boot?
Restarting the service exhibits a similar situation.

I'm using 3.1.1 with the following conf:

root@qos-gw1:~/cake-autorate# cat config.primary.sh
#!/bin/bash

# *** STANDARD CONFIGURATION OPTIONS ***

### For multihomed setups, it is the responsibility of the user to ensure that the probes
### sent by this instance of cake-autorate actually travel through these interfaces.
### See ping_extra_args and ping_prefix_string

dl_if=eth0 # download interface
ul_if=eth1 # upload interface

# Set either of the below to 0 to adjust one direction only
# or alternatively set both to 0 to simply use cake-autorate to monitor a connection
adjust_dl_shaper_rate=1 # enable (1) or disable (0) actually changing the dl shaper rate
adjust_ul_shaper_rate=1 # enable (1) or disable (0) actually changing the ul shaper rate

min_dl_shaper_rate_kbps=7000  # minimum bandwidth for download (Kbit/s)
base_dl_shaper_rate_kbps=40000 # steady state bandwidth for download (Kbit/s)
max_dl_shaper_rate_kbps=200000  # maximum bandwidth for download (Kbit/s)

min_ul_shaper_rate_kbps=5000  # minimum bandwidth for upload (Kbit/s)
base_ul_shaper_rate_kbps=10000 # steady state bandwidth for upload (KBit/s)
max_ul_shaper_rate_kbps=50000  # maximum bandwidth for upload (Kbit/s)

# *** OVERRIDES ***

### See defaults.sh for additional configuration options
### that can be set in this configuration file to override the defaults.
### Place any such overrides below this line.

no_pingers=2
reflectors=("9.9.9.9" "9.9.9.10")
pinger_binary=tsping

The tc statistics:

root@qos-gw1:~/cake-autorate# tc -s qdisc
qdisc noqueue 0: dev lo root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc cake 8001: dev eth0 root refcnt 2 bandwidth 40Mbit besteffort triple-isolate nonat nowash ingress no-ack-filter split-gso rtt 100ms raw overhead 0
 Sent 229955299 bytes 220164 pkt (dropped 448, overlimits 480421 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 544000b of 4Mb
 capacity estimate: 40Mbit
 min/max network layer size:           42 /    1514
 min/max overhead-adjusted size:       42 /    1514
 average network hdr offset:           14

                  Tin 0
  thresh         40Mbit
  target            5ms
  interval        100ms
  pk_delay       6.94ms
  av_delay       2.69ms
  sp_delay          3us
  backlog            0b
  pkts           220612
  bytes       230546169
  way_inds        16712
  way_miss        32416
  way_cols           50
  drops             448
  marks               1
  ack_drop            0
  sp_flows            2
  bk_flows            1
  un_flows            0
  max_len          3028
  quantum          1220

qdisc cake 8002: dev eth1 root refcnt 2 bandwidth 5Mbit besteffort triple-isolate nonat nowash ack-filter split-gso rtt 100ms raw overhead 0
 Sent 14816266 bytes 86383 pkt (dropped 477, overlimits 70932 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 87808b of 4Mb
 capacity estimate: 10Mbit
 min/max network layer size:           42 /    1454
 min/max overhead-adjusted size:       42 /    1454
 average network hdr offset:           14

                  Tin 0
  thresh          5Mbit
  target            5ms
  interval        100ms
  pk_delay       1.15ms
  av_delay        192us
  sp_delay          1us
  backlog            0b
  pkts            86860
  bytes        14883345
  way_inds         7873
  way_miss        32624
  way_cols           74
  drops              35
  marks               2
  ack_drop          442
  sp_flows           30
  bk_flows            1
  un_flows            0
  max_len          1454
  quantum           300

qdisc noqueue 0: dev br-lan root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0

40mb is my normal, but the 5 on upload is not.
This is a 100/25 from the ISP, it shapes pretty well, but yeah...

root@qos-gw1:~/cake-autorate# uptime
 20:48:12 up 9 min,  load average: 0.02, 0.02, 0.00

[Lynx]

Could you export data file from boot up to where things even out again? And upload it somewhere, e.g. here: https://easyupload.io/. You could stop the cake-autorate service. Then delete log file. Then reboot with service enabled. And then wait for things to settle down. You might need to increase the log duration to 300s temporarily.

[Simon10362]

I'll try and see if I can get it done tomorrow but I can't guarantee it. But I'll get it done ASAP.

[Lynx]

@moeller0 I have been working on cake-autorate a little just for fun to see if I can reduce CPU cycles more.

But something that is puzzling me is this.

When comparing runs using 'time -v' I see a newer script report an increase in CPU usage, whereas when monitoring:

cat /sys/fs/cgroup/services/cake-autorate/cpu.stat | awk 'NR==2,NR==3{sum+=$2};END{print sum;}'

I see reduced CPU usage.

Which can I trust?

[moeller0]

@Lynx

Which can I trust?

Sorry, no idea.

[Lynx]

From playing around I think the:

/sys/fs/cgroup/services/${service}/cpu.stat

approach seems to give accurate readings. At least they compare with 'atop' when selecting the 'p' option to group threads together.

It looks like time may give unrealiable results depending upon priority as explained here:

Is the UNIX `time` command accurate enough for benchmarks?

[Lynx]

In any case, I was able to reduce total CPU consumption by ~5% on my RT3200 by folding the pinger parsing into the main process thereby reducing some of the overhead associated with exchanging data between the separate processes:

Fold pinger parsing into main process

[buggz]

I just downloaded the new one, edited config, restarted and get this:

Sun Mar  3 13:49:11 2024 user.notice cake-autorate.primary: INFO: 1709491751.211159 Starting cake-autorate with PID: 19587 and config: /root/cake-autorate/config.primary.sh
Sun Mar  3 13:49:14 2024 user.notice cake-autorate.primary: ERROR: 1709491754.333903 /root/cake-autorate/cake-autorate.sh: line 1117: ewma: unbound variable

ewma_iteration()
{
        local value="${1}"
        local alpha="${2}" # alpha must be scaled by factor of 1000000
        local -n ewma="${3}"

1117      prev_ewma=${ewma}
        ewma=$(( (alpha*value+(1000000-alpha)*prev_ewma)/1000000 ))
}

[Lynx]

That commit was pretty hefty. I think I know what might have happened there and I think I might have already ironed out the issue; I force pushed over that commit a few times today to iron out a few things. Please can you verify that the cake-autorate.sh version you tested is slightly different from the latest on the master branch? Hopefully the latest version resolves this issue. If not, there’s something else needing attention.