CAKE w/ Adaptive Bandwidth [August 2022 to March 2024]

Lynx · 2023-04-10T08:31:44.074Z

easyupload.io

Waveform

Bufferbloat Test Result: D

View the full results, and test your own bufferbloat

moeller0 · 2023-04-10T08:36:59.776Z

I need the .json file for my parser...
but the file you posted showd the following aggregate statistics:

                         Min     Mean   Median      Max   Stddev
                         ---     ----   ------      ---   ------
                RTT  48.77ms  148.4ms  134.9ms  310.7ms  76.78ms
         send delay  20.69ms    103ms  58.47ms  280.5ms   80.7ms
      receive delay  27.06ms  45.31ms  32.29ms  166.2ms  26.97ms
                                                                
      IPDV (jitter)    311ns    7.9ms   4.91ms  120.4ms   9.47ms
          send IPDV    137ns   7.65ms   5.66ms  107.2ms      9ms
       receive IPDV    129ns   2.09ms   1.24ms  38.57ms   2.95ms
                                                                
     send call time     63µs    145µs            1.74ms   90.4µs
        timer error     60ns    272µs            5.83ms    231µs
  server proc. time    710ns   11.3µs             579µs     11µs

                duration: 1m18s (wait 0s)
   packets sent/received: 7764/7207 (7.17% loss)
 server packets received: 7579/7764 (2.38%/4.91% loss up/down)
     bytes sent/received: 465840/432420
       send/receive rate: 47.6 Kbps / 44.3 Kbps
           packet length: 60 bytes
             timer stats: 57/7821 (0.73%) missed, 2.72% error

With higher send delay than receive delay, which to me indicates that the tsping data sees believable as well...

Lynx · 2023-04-10T08:37:51.003Z

image1850×1043 87.3 KB

@moeller0 I have updated the link above to include the JSON.

I am very curious to understand what the cause of this is. Could it be some form of traffic shaping by my ISP?

Does this make the use of timestamp ICMPs a little harder to meaningfully leverage in terms of identifying which direction is responsible for causing bufferbloat?

moeller0 · 2023-04-10T08:56:53.502Z

Lynx:

@moeller0 I have updated the link above to include the JSON.

Thanks, currently running, might take a while... the parser I am using (jsonlab) is really slow under octave...

Lynx:

I am very curious to understand what the cause of this is. Could it be some form of traffic shaping by my ISP?

No idea.

Lynx:

Does this make the use of timestamp ICMPs a little harder to meaningfully leverage in terms of identifying which direction is responsible for causing bufferbloat?

Not really, the increased delay indicates that there is increased queuing happening in that direction and that means that the offered traffic exceeds to bottleneck rate so this is indicative of having to throttle (if your ISP requires a certain standing queue to schedule more transmit slots you might need to increase the delay threshold for the upload direction).

P.S.: I notice that this seems to only happen for traffic in the download direction, correct?

Lynx · 2023-04-10T09:27:33.151Z

moeller0:

Not really, the increased delay indicates that there is increased queuing happening in that direction and that means that the offered traffic exceeds to bottleneck rate so this is indicative of having to throttle (if your ISP requires a certain standing queue to schedule more transmit slots you might need to increase the delay threshold for the upload direction).

But since the uplink delay increases with saturating download, lowering the upload cake bandwidth isn't going to work, or?

So what I mean is that splitting out the overall delay to try to independently control directions may not work in my case?

Hitherto I've always just worked with RTTs and split it 50/50, and this has proved pretty reliable. Here is just a random run of the waveform test from my mobile phone:

Waveform

Bufferbloat Test Result: A+

View the full results, and test your own bufferbloat

This is typical - cake-autorate with fping and my rather aggressive settings tends to keep download bandwidth at this time of day at circa 10Mbit/s and upload bandwidth at circa 25Mbit/s. Latency is kept in check, although I have noticed some stuttering in video in Teams calls lately. Not sure what's causing that.

But I'm not sure how to meaningfully work with OWDs if download saturation increases uplink delay.

moeller0:

P.S.: I notice that this seems to only happen for traffic in the download direction, correct?

Yes it seems that saturating download mostly increases the uplink delay whereas saturating upload does not increase the downlink delay.

In both the tsping and irtt tests I ran a standard waveform bufferbloat test that saturates download and then saturates upload.

moeller0 · 2023-04-10T09:42:52.238Z

Lynx:

But since the uplink delay increases with saturating download, lowering the upload cake bandwidth isn't going to work, or?

That depends on what we are seeing here, a rate reduction of the ISP or some odd sort of requiring a larger queue to schedule transmit opportunities.

Lynx:

So what I mean is that splitting out the overall delay to try to independently control directions may not work in my case?

Not sure I agree... this really depends on why we observe what we observe...

Lynx:

Hitherto I've always just worked with RTTs and split it 50/50, and this has proved pretty reliable. Here is just a random run of the waveform test from my mobile phone:

https://www.waveform.com/tools/bufferbloat?test-id=6404b3b6-7ac9-40b0-b4b0-6eeccabcc335

Yes, we worked with RTTs so far (splitting 50/50 was just a side-effect free way to generate correlated pretend OWDs) but that means we always throttled both directions in unison...

Lynx:

Yes it seems that saturating download mostly increases the uplink delay whereas saturating upload does not increase the downlink delay.

Yes, how about you double the latency threshold for the upload direction? And test again...

BTW the plots from the json are almost identical to your plots, but I added the directional loss:

irtt_LTE_20230410_092841.irtt_timecourse1920×1390 165 KB

Lynx · 2023-04-10T09:45:59.264Z

moeller0:

Yes, how about you double the latency threshold for the upload direction? And test again...

To be clear, the tsping and irtt tests were conducted with cake not enabled. So these tests are for saturating loads. Does that make these results even weirder?

With these plots I can't figure out how we work out which shaper rate to reduce. Can one unambiguously discern that the first half is download and the second half is upload?

moeller0 · 2023-04-10T09:51:27.345Z

Well, could you run a bidirectionally saturating throughput test as well, so we can see what happens?

moeller0 · 2023-04-10T09:53:47.006Z

Lynx:

With these plots I can't figure out how we work out which shaper rate to reduce. Can one unambiguously discern that the first half is download and the second half is upload?

Well, it might be necessary to increase the delay threshold for the upload direction...
However even in the current configuration you are already doing better than when using RTTs, for download you see the same linked reduction in up- and download as with RTT/2, but for upload traffic the OWDs are decoupled and you will see independent shaping per direction.

Lynx · 2023-04-10T09:57:10.435Z

moeller0:

Well, could you run a bidirectionally saturating throughput test as well, so we can see what happens?

Is there a netperf way to make say 30s DL, 30s UL and then 30s DL+UL? Or can this even be done with irtt itself?

moeller0 · 2023-04-10T10:21:39.643Z

Lynx:

Is there a netperf way to make say 30s DL, 30s UL and then 30s DL+UL?

Good question. I tend to use flent to run netperf tests:
RRUL_var: allows bi directionally saturatimg tests with configurable number of flows and per-flow DSCP configuration:

date ; ping -c 10 test.server.com ; ./run-flent --ipv4 -l 180 -H test.server.com rrul_var --remote-metadata=root@192.168.42.1 --test-parameter=cpu_stats_hosts=root@192.168.42.1 --step-size=.05 --test-parameter bidir_streams=8 --test-parameter markings=CS0,CS1,CS2,CS3,CS4,CS5,CS6,CS7 --test-parameter ping_hosts=1.1.1.1 -D . -t my_test_result_name --log-file

upload only:

date ; ping -c 10 test.server.com ; ./run-flent --ipv4 -l 180 -H test.server.com tcp_nup --remote-metadata=root@192.168.42.1 --test-parameter=cpu_stats_hosts=root@192.168.42.1 --step-size=.05 --test-parameter upload_streams=8 --test-parameter ping_hosts=1.1.1.1 -D . -t my_test_result_name --log-file

download only:

date ; ping -c 10 test.server.com ; ./run-flent --ipv4 -l 180 -H test.server.com tcp_ndown --remote-metadata=root@192.168.42.1 --test-parameter=cpu_stats_hosts=root@192.168.42.1 --step-size=.05 --test-parameter download_streams=8 --test-parameter ping_hosts=1.1.1.1 -D . -t my_test_result_name --log-file

I typically only run the RRUL_var test myself so the other command lines are not tested yet.

However flent supports batch files, maybe it is time to create one for N flow upload, N flow download, N flow bidirectional traffic...

Lynx · 2023-04-10T10:28:26.199Z

Looks like I can just tweak @richb-hanover-priv's betterspeedtest.sh:

github.com

richb-hanover/OpenWrtScripts/blob/285883a2335d82c92c53583ce0e8c61274d1ed51/betterspeedtest.sh#L276


      
          then
            echo "$DATE Testing idle line while pinging $PINGHOST ($TESTDUR seconds)"
            SPEEDFILE=$(mktemp /tmp/netperfUL.XXXXXX) || exit 1
            start_pings
            sleep "$TESTDUR"
            summarize_pings "$PINGFILE"
            clean_up
          
          
else
            echo "$DATE Testing against $TESTHOST ($PROTO) with $MAXSESSIONS simultaneous sessions while pinging $PINGHOST ($TESTDUR seconds in each direction)"
            measure_direction "Download" 
            measure_direction "  Upload" 
          fi

moeller0 · 2023-04-10T10:38:25.434Z

You could, but flent offers really nice additional features like an easy way to generate plots (in python) and can e.g. also collect TCP statistics if running on a Linux host.

Here is an example from a bidirectional rrul_var test:

rrul_var_-_IPv4_noSQM_cake_layer-cake_LLA-ETH_OH34_U097pct36000of36999K-D090pct105000of116790K_hms-beagle2_en3USB3_TurrisOmnia-TurrisOS.6.2.3-pppoe-wan-eth2.7_2_bridged-FB7520-OpenWrt-r22036-1a145ccb0a-Hvt-VDSL100_2_netperf-eu.bufferbloat.net1920×738 150 KB

P.S.: The cyclic dips in the download direction came to be as others were streaming video during the test, so each dip corresponds to the streaming client fetching the next "segment" and cake's per-IP-fairness throttling my test download capacity... I want to add that the video streamers did not complain about degraded quality or buffering... so clearly on a fixed-rate cake-shaped 105/37 Mbps link, sqm does work quite well out of the box

Lynx · 2023-04-10T10:52:00.007Z

True, but I'm just looking for something quick.

@richb-hanover-priv might you be tempted to add a bidirectional option to betterspeedtest.sh? It's not quite as simple as just:

measure_direction "Download" & measure_direction " Upload"

I'd need to look at it in detail to get that to work properly, but for now it's good enough to generate the loads.

@moeller0 here are the files:

easyupload.io

Here is a screen capture of the loads reported by OpenWrt:

image1178×676 64.3 KB

And excel plot of the irtt data:

image2418×1351 150 KB

I have a hard time understanding what's going on here.

richb-hanover-priv · 2023-04-10T11:05:31.484Z

Lynx:

@richb-hanover-priv might you be tempted to add a bidirectional option to betterspeedtest.sh?

A while back, I added a --idle parameter to measure "background latency" without sending any traffic. I suppose this option would be the complement - measuring latency with traffic in both directions.

@Lynx I would accept a PR, especially from such a bash expert as you

Lynx · 2023-04-10T11:15:28.515Z

Well for sure I think I know most of the bash tricks by now. I've come to love bash with its rich feature set and the speed at which you can completely restructure things. Your script uses: #!/bin/sh, but that avoids the bash dependency. I'll see if I can add in the bidirectional capability when I get a chance.

@moeller0 in the irtt graph above it strikes me that the downlink latency simply could not be relied upon to ascertain bufferbloat arising from download. So this would mean in my case having to work with rtt for download and ul-owd for upload, or?

moeller0 · 2023-04-10T11:39:17.837Z

Lynx:

Here is a screen capture of the loads reported by OpenWrt:

That is nice as a graphic, but too course to be really useful for deeper analysis...

Lynx:

in the irtt graph above it strikes me that the downlink latency simply could not be relied upon to ascertain bufferbloat arising from download.

I respectfully disagree, I see increases in the orange line whenever there is download traffic, and increases in the gray line during upload... yes, the gray also increases during downloads, but considerably less than when there is saturating upload traffic. You "simply" need considerably larger delay thresholds for the upload direction than the download direction.... there is no reason why these should be symmetric anyway...

moeller0 · 2023-04-10T11:42:32.043Z

Lynx:

True, but I'm just looking for something quick.

Assuming you have a linux host or linux VM, getting flent up and running is pretty quick... as reward you get persistent logs and nicer plotting capabilities...

If all you want/need is a staggered, download, upload, bidirectional test with a nice GUI have a look at:

GitHub

GitHub - Zoxc/crusader: A network bandwidth and latency tester.

A network bandwidth and latency tester. Contribute to Zoxc/crusader development by creating an account on GitHub.

however you need to set-up your own servers in the internet to act as remote measurement points.

Lynx · 2023-04-10T12:13:02.406Z

How about this:

github.com/lynxthecat/OpenWrtScripts

Add bidirectional measurement

committed 12:12PM - 10 Apr 23 UTC

lynxthecat

+13 -4

I've done this rather quickly because I really should be working. It certainly needs testing. Works for me though. Any thoughts?

moeller0:

I respectfully disagree, I see increases in the orange line whenever there is download traffic, and increases in the gray line during upload... yes, the gray also increases during downloads, but considerably less than when there is saturating upload traffic. You "simply" need considerably larger delay thresholds for the upload direction than the download direction.... there is no reason why these should be symmetric anyway...

Music to my ears - let me play with adjusting the thresholds as you have suggested.

By the way, @Lochnair and @moeller0 shouldn't tsping itself compensate for the midnight rollover issue? Otherwise the lines it prints:

root@OpenWrt-1:~/cake-autorate# tsping 9.9.9.9 --print-timestamps --machine-readable
Starting tsping 0.2.3 - pinging 1 targets
1681088125.024088,9.9.9.9,0,3324984,3325004,3325004,3325024,40,20,20
1681088125.124105,9.9.9.9,1,3325084,3325104,3325104,3325124,40,20,20
1681088125.224088,9.9.9.9,2,3325184,3325204,3325204,3325224,40,20,20
1681088125.323094,9.9.9.9,3,3325285,3325304,3325304,3325323,38,19,19
1681088125.425049,9.9.9.9,4,3325385,3325405,3325405,3325425,40,20,20
1681088125.524247,9.9.9.9,5,3325485,3325504,3325504,3325524,39,20,19
1681088125.624081,9.9.9.9,6,3325585,3325604,3325604,3325624,39,20,19
1681088125.725051,9.9.9.9,7,3325686,3325704,3325704,3325725,39,21,18
1681088125.824067,9.9.9.9,8,3325786,3325804,3325804,3325824,38,20,18
1681088125.924038,9.9.9.9,9,3325886,3325904,3325904,3325924,38,20,18
1681088126.024109,9.9.9.9,10,3325986,3326004,3326004,3326024,38,20,18
1681088126.123025,9.9.9.9,11,3326087,3326104,3326104,3326123,36,19,17

will presumably be phoney at the rollover time?

@tievolu set out the recipe for dealing with this here:

CAKE w/ Adaptive Bandwidth Community Builds, Projects & Packages

I have a chunk of code in my perl implementation to try and deal with this stuff, which is essential with my approach of working with hundreds/thousands of reflector IPs whose reliability is unknown. I "correct" the timestamps to get absolute OWD values, rather than dealing with deltas and baselines. This probably isn't as efficient for the most part, but it does mean that I can correct the numbers in these scenarios: Reflector's timer is not in sync with the local timer, but the offset is …

moeller0 · 2023-04-10T12:38:08.593Z

Lynx:

will presumably be phoney at the rollover time?

tsping should veridically report its data, and autorate needs to learn to detect and ignore/handle roll-over events. ICMP timestamps are designed to roll-over so consumers need to deal with that fact gracefully, especially since due to mis-set clocks such roll-overs can happen anytime (and the two relevant clocks, local and remote can roll-over at quite different times).

Lynx:

@tievolu set out the recipe for dealing with this here:

Without looking at the details, that is something on autorate's plate, not tsping's... in our case things are not that tricky since all we need to do is:
a) ignore samples where one of the clocks rolled-over during the sample (these will be rare so just ignoring those samples seems OK)
b) adjust our baselines quickly after clock-roll-overs, however we already have code to adjust baselines, so I think even doing nothing should get us back into acceptable delay performance territory again.

I have however not tried to think this through in detail, so I could e out for lunch here....