CAKE w/ Adaptive Bandwidth [August 2022 to March 2024]

[Lynx]

@moeller0 I have updated the link above to include the JSON.

I am very curious to understand what the cause of this is. Could it be some form of traffic shaping by my ISP?

Does this make the use of timestamp ICMPs a little harder to meaningfully leverage in terms of identifying which direction is responsible for causing bufferbloat?

[moeller0]

@Lynx

@moeller0 I have updated the link above to include the JSON.

Thanks, currently running, might take a while... the parser I am using (jsonlab) is really slow under octave...

@Lynx

I am very curious to understand what the cause of this is. Could it be some form of traffic shaping by my ISP?

No idea.

@Lynx

Does this make the use of timestamp ICMPs a little harder to meaningfully leverage in terms of identifying which direction is responsible for causing bufferbloat?

Not really, the increased delay indicates that there is increased queuing happening in that direction and that means that the offered traffic exceeds to bottleneck rate so this is indicative of having to throttle (if your ISP requires a certain standing queue to schedule more transmit slots you might need to increase the delay threshold for the upload direction).

P.S.: I notice that this seems to only happen for traffic in the download direction, correct?

[Lynx]

@moeller0

Not really, the increased delay indicates that there is increased queuing happening in that direction and that means that the offered traffic exceeds to bottleneck rate so this is indicative of having to throttle (if your ISP requires a certain standing queue to schedule more transmit slots you might need to increase the delay threshold for the upload direction).

But since the uplink delay increases with saturating download, lowering the upload cake bandwidth isn't going to work, or?

So what I mean is that splitting out the overall delay to try to independently control directions may not work in my case?

Hitherto I've always just worked with RTTs and split it 50/50, and this has proved pretty reliable. Here is just a random run of the waveform test from my mobile phone:

Bufferbloat Test Result: A+

This is typical - cake-autorate with fping and my rather aggressive settings tends to keep download bandwidth at this time of day at circa 10Mbit/s and upload bandwidth at circa 25Mbit/s. Latency is kept in check, although I have noticed some stuttering in video in Teams calls lately. Not sure what's causing that.

But I'm not sure how to meaningfully work with OWDs if download saturation increases uplink delay.

@moeller0

P.S.: I notice that this seems to only happen for traffic in the download direction, correct?

Yes it seems that saturating download mostly increases the uplink delay whereas saturating upload does not increase the downlink delay.

In both the tsping and irtt tests I ran a standard waveform bufferbloat test that saturates download and then saturates upload.

[moeller0]

@Lynx

But since the uplink delay increases with saturating download, lowering the upload cake bandwidth isn't going to work, or?

That depends on what we are seeing here, a rate reduction of the ISP or some odd sort of requiring a larger queue to schedule transmit opportunities.

@Lynx

So what I mean is that splitting out the overall delay to try to independently control directions may not work in my case?

Not sure I agree... this really depends on why we observe what we observe...

@Lynx

Hitherto I've always just worked with RTTs and split it 50/50, and this has proved pretty reliable. Here is just a random run of the waveform test from my mobile phone:

https://www.waveform.com/tools/bufferbloat?test-id=6404b3b6-7ac9-40b0-b4b0-6eeccabcc335

Yes, we worked with RTTs so far (splitting 50/50 was just a side-effect free way to generate correlated pretend OWDs) but that means we always throttled both directions in unison...

@Lynx

Yes it seems that saturating download mostly increases the uplink delay whereas saturating upload does not increase the downlink delay.

Yes, how about you double the latency threshold for the upload direction? And test again...

BTW the plots from the json are almost identical to your plots, but I added the directional loss:

[Lynx]

@moeller0

Yes, how about you double the latency threshold for the upload direction? And test again...

To be clear, the tsping and irtt tests were conducted with cake not enabled. So these tests are for saturating loads. Does that make these results even weirder?

With these plots I can't figure out how we work out which shaper rate to reduce. Can one unambiguously discern that the first half is download and the second half is upload?

[moeller0]

Well, could you run a bidirectionally saturating throughput test as well, so we can see what happens?

[moeller0]

@Lynx

With these plots I can't figure out how we work out which shaper rate to reduce. Can one unambiguously discern that the first half is download and the second half is upload?

Well, it might be necessary to increase the delay threshold for the upload direction...
However even in the current configuration you are already doing better than when using RTTs, for download you see the same linked reduction in up- and download as with RTT/2, but for upload traffic the OWDs are decoupled and you will see independent shaping per direction.

[Lynx]

@moeller0

Well, could you run a bidirectionally saturating throughput test as well, so we can see what happens?

Is there a netperf way to make say 30s DL, 30s UL and then 30s DL+UL? Or can this even be done with irtt itself?

[moeller0]

@Lynx

Is there a netperf way to make say 30s DL, 30s UL and then 30s DL+UL?

Good question. I tend to use flent to run netperf tests:
RRUL_var: allows bi directionally saturatimg tests with configurable number of flows and per-flow DSCP configuration:

date ; ping -c 10 test.server.com ; ./run-flent --ipv4 -l 180 -H test.server.com rrul_var --remote-metadata=root@192.168.42.1 --test-parameter=cpu_stats_hosts=root@192.168.42.1 --step-size=.05 --test-parameter bidir_streams=8 --test-parameter markings=CS0,CS1,CS2,CS3,CS4,CS5,CS6,CS7 --test-parameter ping_hosts=1.1.1.1 -D . -t my_test_result_name --log-file

upload only:

date ; ping -c 10 test.server.com ; ./run-flent --ipv4 -l 180 -H test.server.com tcp_nup --remote-metadata=root@192.168.42.1 --test-parameter=cpu_stats_hosts=root@192.168.42.1 --step-size=.05 --test-parameter upload_streams=8 --test-parameter ping_hosts=1.1.1.1 -D . -t my_test_result_name --log-file

download only:

date ; ping -c 10 test.server.com ; ./run-flent --ipv4 -l 180 -H test.server.com tcp_ndown --remote-metadata=root@192.168.42.1 --test-parameter=cpu_stats_hosts=root@192.168.42.1 --step-size=.05 --test-parameter download_streams=8 --test-parameter ping_hosts=1.1.1.1 -D . -t my_test_result_name --log-file

I typically only run the RRUL_var test myself so the other command lines are not tested yet.

However flent supports batch files, maybe it is time to create one for N flow upload, N flow download, N flow bidirectional traffic...

[Lynx]

Looks like I can just tweak @richb-hanover-priv's betterspeedtest.sh:

richb-hanover/OpenWrtScripts/blob/285883a2335d82c92c53583ce0e8c61274d1ed51/betterspeedtest.sh#L276

[moeller0]

You could, but flent offers really nice additional features like an easy way to generate plots (in python) and can e.g. also collect TCP statistics if running on a Linux host.

Here is an example from a bidirectional rrul_var test:

P.S.: The cyclic dips in the download direction came to be as others were streaming video during the test, so each dip corresponds to the streaming client fetching the next "segment" and cake's per-IP-fairness throttling my test download capacity... I want to add that the video streamers did not complain about degraded quality or buffering... so clearly on a fixed-rate cake-shaped 105/37 Mbps link, sqm does work quite well out of the box

[Lynx]

True, but I'm just looking for something quick.

@richb-hanover-priv might you be tempted to add a bidirectional option to betterspeedtest.sh? It's not quite as simple as just:

measure_direction "Download" & measure_direction " Upload"

I'd need to look at it in detail to get that to work properly, but for now it's good enough to generate the loads.

@moeller0 here are the files:

easyupload.io

Here is a screen capture of the loads reported by OpenWrt:

And excel plot of the irtt data:

I have a hard time understanding what's going on here.

[richb-hanover-priv]

@Lynx

@richb-hanover-priv might you be tempted to add a bidirectional option to betterspeedtest.sh?

A while back, I added a --idle parameter to measure "background latency" without sending any traffic. I suppose this option would be the complement - measuring latency with traffic in both directions.

@Lynx I would accept a PR, especially from such a bash expert as you

[Lynx]

Well for sure I think I know most of the bash tricks by now. I've come to love bash with its rich feature set and the speed at which you can completely restructure things. Your script uses: #!/bin/sh, but that avoids the bash dependency. I'll see if I can add in the bidirectional capability when I get a chance.

@moeller0 in the irtt graph above it strikes me that the downlink latency simply could not be relied upon to ascertain bufferbloat arising from download. So this would mean in my case having to work with rtt for download and ul-owd for upload, or?

[moeller0]

@Lynx

Here is a screen capture of the loads reported by OpenWrt:

That is nice as a graphic, but too course to be really useful for deeper analysis...

@Lynx

in the irtt graph above it strikes me that the downlink latency simply could not be relied upon to ascertain bufferbloat arising from download.

I respectfully disagree, I see increases in the orange line whenever there is download traffic, and increases in the gray line during upload... yes, the gray also increases during downloads, but considerably less than when there is saturating upload traffic. You "simply" need considerably larger delay thresholds for the upload direction than the download direction.... there is no reason why these should be symmetric anyway...

[moeller0]

@Lynx

True, but I'm just looking for something quick.

Assuming you have a linux host or linux VM, getting flent up and running is pretty quick... as reward you get persistent logs and nicer plotting capabilities...

If all you want/need is a staggered, download, upload, bidirectional test with a nice GUI have a look at:

GitHub - Zoxc/crusader: A network bandwidth and latency tester.

however you need to set-up your own servers in the internet to act as remote measurement points.

[Lynx]

How about this:

Add bidirectional measurement

I've done this rather quickly because I really should be working. It certainly needs testing. Works for me though. Any thoughts?

---

@moeller0

I respectfully disagree, I see increases in the orange line whenever there is download traffic, and increases in the gray line during upload... yes, the gray also increases during downloads, but considerably less than when there is saturating upload traffic. You "simply" need considerably larger delay thresholds for the upload direction than the download direction.... there is no reason why these should be symmetric anyway...

Music to my ears - let me play with adjusting the thresholds as you have suggested.

---

By the way, @Lochnair and @moeller0 shouldn't tsping itself compensate for the midnight rollover issue? Otherwise the lines it prints:

root@OpenWrt-1:~/cake-autorate# tsping 9.9.9.9 --print-timestamps --machine-readable
Starting tsping 0.2.3 - pinging 1 targets
1681088125.024088,9.9.9.9,0,3324984,3325004,3325004,3325024,40,20,20
1681088125.124105,9.9.9.9,1,3325084,3325104,3325104,3325124,40,20,20
1681088125.224088,9.9.9.9,2,3325184,3325204,3325204,3325224,40,20,20
1681088125.323094,9.9.9.9,3,3325285,3325304,3325304,3325323,38,19,19
1681088125.425049,9.9.9.9,4,3325385,3325405,3325405,3325425,40,20,20
1681088125.524247,9.9.9.9,5,3325485,3325504,3325504,3325524,39,20,19
1681088125.624081,9.9.9.9,6,3325585,3325604,3325604,3325624,39,20,19
1681088125.725051,9.9.9.9,7,3325686,3325704,3325704,3325725,39,21,18
1681088125.824067,9.9.9.9,8,3325786,3325804,3325804,3325824,38,20,18
1681088125.924038,9.9.9.9,9,3325886,3325904,3325904,3325924,38,20,18
1681088126.024109,9.9.9.9,10,3325986,3326004,3326004,3326024,38,20,18
1681088126.123025,9.9.9.9,11,3326087,3326104,3326104,3326123,36,19,17

will presumably be phoney at the rollover time?

@tievolu set out the recipe for dealing with this here:

CAKE w/ Adaptive Bandwidth

I have a chunk of code in my perl implementation to try and deal with this stuff, which is essential with my approach of working with hundreds/thousands of reflector IPs whose reliability is unknown. I "correct" the timestamps to get absolute OWD values, rather than dealing with deltas and baselines. This probably isn't as efficient for the most part, but it does mean that I can correct the numbers in these scenarios: Reflector's timer is not in sync with the local timer, but the offset is …

[moeller0]

@Lynx

will presumably be phoney at the rollover time?

tsping should veridically report its data, and autorate needs to learn to detect and ignore/handle roll-over events. ICMP timestamps are designed to roll-over so consumers need to deal with that fact gracefully, especially since due to mis-set clocks such roll-overs can happen anytime (and the two relevant clocks, local and remote can roll-over at quite different times).

@Lynx

@tievolu set out the recipe for dealing with this here:

Without looking at the details, that is something on autorate's plate, not tsping's... in our case things are not that tricky since all we need to do is:
a) ignore samples where one of the clocks rolled-over during the sample (these will be rare so just ignoring those samples seems OK)
b) adjust our baselines quickly after clock-roll-overs, however we already have code to adjust baselines, so I think even doing nothing should get us back into acceptable delay performance territory again.

I have however not tried to think this through in detail, so I could e out for lunch here....

[Lynx]

Hmm but shouldn't then tsping not report OWDs or take in clock to ensure it won't output phoney OWDs?

Would baselines need to be able to be negative to deal with the overrun issue?

[moeller0]

@Lynx

Hmm but shouldn't then tsping not report OWDs or take in clock to ensure it won't output phoney OWDs?

Ah, you propose to have tsping report the "true" timestamps but fix up/down values with internal roll-overs? IMHO it is autorate's onus to sanitize its inputs. Before putting something on tsping's plate, let's first go through all relevant cases and see what can/should be done, no?

@Lynx

Would baselines need to be able to be negative to deal with the overrun issue?

Yes, they would... if the remote ensided rolls-over before the local side we end up with negative numbers (negative but still with informative deltas)...
(and since clocks can be arbitrarily off, I am sure there will be reflectors that will report negative differences for most of the day)...