CAKE w/ Adaptive Bandwidth [August 2022 to March 2024]

patrakov · 2022-12-04T09:51:16.194Z

No preference either way. But still please always use mkdir -p.

moeller0 · 2022-12-04T10:02:29.444Z

Lynx:

But then it depends whether the instance id is set or not what the path should look like

Let's default to an instance_id even in the single configuration case and get rid of the alternative code paths?

patrakov:

Any mkdir without the -p switch` is a bug, unless the directory is used as a lock (and we don't do it).

Nit-pickingly I disagree, if I know a directory exists (or needs to exist) I can use unadorned mkdir (in any case I should evaluate the return value and report any errors to the user). Mkdir without p has valid uses, but I agree for our intent of creating nested directories -p is the right thing to do.

patrakov · 2022-12-04T10:17:36.780Z

moeller0:

Let's default to an instance_id even in the single configuration case and get rid of the alternative code paths?

This might actually be a good idea, if we accept that the instance ID should be a part of the log filename, as opposed to the directory name. But then the instance ID should also capture the leading dot.

Plain old single-instance configuration (empty ID): /var/log/cake-autorate.log

LTE (ID = .lte) + GPON (ID = .gpon): /var/log/cake-autorate.lte.log + /var/log/cake-autorate.gpon.log

Lynx · 2022-12-04T10:19:29.911Z

Quick question @moeller0 - I've gone back to luci-app-sqm because I've decided to try ditching VPN and because most users will use it so it seems better that I work with it. How do I go back from triple-isolate to the per lan IP host thingy:

image1362×256 22.7 KB

patrakov:

Plain old single-instance configuration (empty ID): /var/log/cake-autorate.log

LTE (ID = .lte) + GPON (ID = .gpon): /var/log/cake-autorate.lte.log + /var/log/cake-autorate.gpon.log

Yes I think this looks neater.

patrakov · 2022-12-04T10:22:52.412Z

image985×406 34.3 KB

Lynx · 2022-12-04T10:26:42.071Z

@moeller0 this OK for LTE?

tc qdisc ls
qdisc noqueue 0: dev lo root refcnt 2
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1518 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc noqueue 0: dev lan1 root refcnt 2
qdisc noqueue 0: dev lan2 root refcnt 2
qdisc noqueue 0: dev lan3 root refcnt 2
qdisc noqueue 0: dev lan4 root refcnt 2
qdisc cake 800d: dev wan root refcnt 2 bandwidth 10Mbit besteffort dual-srchost nat nowash no-ack-filter split-gso rtt 100ms noatm overhead 10
qdisc ingress ffff: dev wan parent ffff:fff1 ----------------
qdisc noqueue 0: dev br-lan root refcnt 2
qdisc noqueue 0: dev br-guest root refcnt 2
qdisc noqueue 0: dev wlan0 root refcnt 2
qdisc noqueue 0: dev wlan1 root refcnt 2
qdisc noqueue 0: dev wlan1.sta1 root refcnt 2
qdisc noqueue 0: dev wlan1.sta2 root refcnt 2
qdisc cake 800e: dev ifb4wan root refcnt 2 bandwidth 10Mbit besteffort dual-dsthost nat wash no-ack-filter split-gso rtt 100ms noatm overhead 10

Overhead '10' as per:

CAKE w/ Adaptive Bandwidth [Historic] Installing and Using OpenWrt

I also came across this: [image] I used to actually work on Intel/Samsung LTE patent applications, but every single application was on one very detailed aspect. I recall so many hacks to make each new innovation work with the (revised) 3GPP standard. Anyway that was a fair while ago now. I also have the fun complexity that I am applying CAKE to interfaces (veth-lan for download, and wan for upload) that see both WireGuard and non-WireGuard traffic. So presumably I should go for worst case…

I forget the default logic wash and nowash. Why do we want not to wash outgoing packets to ISP?

patrakov · 2022-12-04T10:32:16.400Z

Sorry, I don't see the word ingress in your copy-paste. I think you need to add it after dual-dsthost (and the same applies to my router). What it does is to count the bandwidth used up by dropped packets as used-up.

moeller0 · 2022-12-04T10:33:34.346Z

patrakov:

But then the instance ID should also capture the leading dot.

No, I disagree, we should set these dots explicitly when we construct compound names, otherwise we end up with pathnames like /var/run/cake-autorate/.lte which can be hidden from sight for normal ls, let's not do this, please.

Lynx · 2022-12-04T10:39:21.728Z

I'm a bit lost - any chance you could spell out what you have in mind and I'll code that up. It doesn't help that I completely reset my router back to basics so have been setting everything back up at the same time. Bit fiddly but pretty much there now.

moeller0 · 2022-12-04T10:44:20.040Z

Lynx:

@moeller0 this OK for LTE?

Regarding nat and dual-xxxhost this looks fine. About overhead 10 I am less sure, maybe time to get the experts summoned.
@Lochnair and @bmork could you maybe explain how much overhead on top of IP packets end-users would need to account for? I think that you might be able to contact folks that should know about such details intimately
(@Lynx however keep in mind that for a variable rate link with a running autorate instance the exact overhead becomes essentially irrelevant).

Lynx:

I forget the default logic wash and nowash. Why do we want not to wash outgoing packets to ISP?

Cake will always see the "original" DSCPs (quotes as if qosify was used cake will see qosify's output DSCP) if set to besteffort it will however not treat packets differentially by DSCP, if you set wash cake will re-mark all packets to DSCP 0 before passing them on.

patrakov:

Sorry, I don't see the word ingress in your copy-paste. I think you need to add it after dual-dsthost (and the same applies to my router). What it does is to count the bandwidth used up by dropped packets as used-up.

Ingress will essentially try to shape for the rate of packets flowing into cake while no-ingress will control for cake's egress rate. Which works as described above, with ingress even dropped packets are counted. This has two effects:
a) the shaper becomes more aggressive if the ingress traffic is less responsive (which is great)
b) less resonsive (e.g. longer RTT flows) tend to pay a larger price, so that per flow fairness for cake's egress is not as perfect as without the ingress keyword

for most links I tested ingress was a winner IMHO, but it is a policy each network needs to decide on individually.

rb1 · 2022-12-04T10:49:36.050Z

My first time using cake-autorate and second time testing cake. I have been using Gargoyle for ~10 years on a number of routers with its QoS Active Congestion Controller qosmon.c (ACC) which works similar to this, measuring RTT to estimate max download bandwidth. I just received my RT3200 and Gargoyle doesn't support it yet so I'm testing this out.

I managed to get cake-autorate to work well. Compared to ACC, I like how it works on both downloads and uploads. When running ping to 1.1.1.1, the ping is good when downloading from Steam. Anecdotally, cake-autorate seems fluctuate the download bandwidth a lot more than ACC. I noticed in the cake-autorate_config.sh, it appears possible to adjust the speed RTT responsiveness. Not sure which would result in a higher total download throughput over a hour. See screenshot below of WAN speeds. I need to save the same screenshot with ACC.

Screenshot960×1097 200 KB

One point that annoys me (which is probably more a cake thing) is when downloading in Steam while trying to watch a youtube video. Steam is probably using 10 connections while youtube probably 1. So while RTT is good, steam would download at say 5 MBytes/sec and youtube 0.5MB/s. Or if I try a speedtest.net, I would get 1MB/sec. Or if I download octave, it would download at 0.2MB/s but if I pause steam, octave would download at 3MB/s. What I want is cake to give steam 50% and youtube 50%. It seems like cake doesn't fairly distribute bandwidth via service.

I found some examples here which lead me to try various combinations of (srchost | dsthost | hosts) in the advanced cake ingress and egress settings which seems to help. dsthost in ingress seems to be the best to me as I can do a speedtest on my phone while steam downloading and bandwidth is a bit more equal.

What cake settings are recommended to get cake to more fairly distribute bandwidth?

moeller0 · 2022-12-04T10:52:24.454Z

The idea is something along the lines of:

instance_id="lte"
...
log_file_FQN="/var/log/cake-autorate.${instance_id}.log"
...
run_dir="/var/run/cake-autorate/${instance_id}/"
...

And do something like sqm-scripts does

    LAST_ERROR=$( ${CMD_BINARY} "$@" 2>&1 )
    RET=$?

    if [ "$RET" -eq "0" ] ; then
        sqm_debug "cmd_wrapper: ${CALLERID}: SUCCESS: ${CMD_BINARY} $@"
    else
        # this went south, try to capture & report more detail
        $ERRLOG "cmd_wrapper: ${CALLERID}: FAILURE (${RET}): ${CMD_BINARY} $@"
        $ERRLOG "cmd_wrapper: ${CALLERID}: LAST ERROR: ${LAST_ERROR}"
    fi

to capture and report failed commend invocations?

moeller0 · 2022-12-04T11:03:43.517Z

rb1:

It seems like cake doesn't fairly distribute bandwidth via service.

Yes, because it can not do this. Per application fairness would need to be implemented on the end-device running the applications. If you run the applications on different end-devices then cake's per-internal-IP-fairness can help you, but if you run things on the same machine cake simply is lacking information about your desired "fairness-grouping"

rb1:

What cake settings are recommended to get cake to more fairly distribute bandwidth?

Again, can not be done outside of the end-host running your applications, and even there cake would not able to do this easily. Cake is aimed to be run on routers/gateways where application-origin of packets is not available.

bmork · 2022-12-04T13:21:06.490Z

moeller0:

could you maybe explain how much overhead on top of IP packets end-users would need to account for?

I'm not sure. There isn't really any overhead on a QMI raw-ip interface. On MBIM we add an ethernet header, so you get that. I don't think cake will see the USB overhead at all here, but I might be wrong.

Lynx · 2022-12-04T13:43:19.172Z

So @moeller0 with that info should LTE users set:

image1438×217 20.9 KB

moeller0 · 2022-12-04T14:08:59.602Z

Well, do you know it is an QMI raw-ip interface or MBIM?

Lynx · 2022-12-04T14:15:55.377Z

Not exactly sure since I don't understand these concepts. I have this setup:

if that's relevant?

patrakov · 2022-12-04T16:21:51.654Z

What's relevant is whether the interface has a MAC address. If it does, it is an Ethernet-like interface. If it doesn't, it is a raw IP interface.

Lynx · 2022-12-04T16:39:49.517Z

Ah, thanks. Is this definitive:

ifconfig output:

wwan0     Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00

patrakov · 2022-12-04T16:42:15.975Z

Yes. So your device has a raw IP interface.

EDIT: actually even that is irrelevant. What's relevant is that LTE does not have MAC addresses in the radio packets. So, even though my modem (based on cdc-wdm) does have a MAC address, it's just a dummy needed for the fake DHCP server inside the modem to work, and its size and overhead should be disregarded. Of course, the LTE-specific overhead should be included, but that's on top of raw IP.