CAKE w/ Adaptive Bandwidth

This seems to be outside of what your router can actually achieve with cake...

Interesting, will take a peek into these files, thanks.

1 Like

Here is a plot from @gba's autorate run:

@moeller0 notice how closely the achieved rate follows the shaper rate and am I right that right before the autorate algorithm reduces the shaper rate the achieved rate has already started to drop?

so during bufferbloat choking on download saturation RTT increases and the achieved rate begins to drop off compared to the shaper rate? Is there anything in the latter (discrepency between shaper rate and achieved rate) that could be further exploited in terms of bufferbloat detection or otherwise? I suppose there is no way of knowing if the achieved rate drops off because download has stopped or sender stops sending as much data or bufferbloat has occurred? But from graph by eye knowing that download is saturated I think we can identify the drop off as bufferbloat related.

We already exploit discrepancy between shaper rate and achieved rate during bufferbloat detection by setting the punished shaper rate to the minimum of:

a) a fraction (0.9 by default) of the achieved rate; and

b) a fraction (0.9 by default) of the shaper rate

during bufferbloat.

I wonder if with Starlink the achieved rate fraction should be set to 1.2 or something such that the punished rate is the less of 1.2 * achieved_rate and 0.9 * shaper_rate. What do you think?

Also notice that just before the shaper rate drops it is held because load is detected as medium since the achieved rate drops below the high load threshold but above the low load threshold.

@gba there is a different autorate implementation that uses LUA linked at the top of this thread. You could try that to see how it compares. It might work better.

2 Likes

Side note, for b) I really think we should aim for 80% or even less.

But generally we already have logic to regress back to the baserate with sub-threshold loads, maybe just increase the regression rate somewhat (for decreases, not for increases)? Other than that these reductions in achieved goodput are hard to interpret, we can not disambiguate reduction, say due to dropped packets or delayed ACKs and the sender simply not having as much data to send right now...

Again, part of the challenge is the generally high variability in idle RTT measurements which makes it hard to define a tight RTT threshold...

If at all go to a factor of 0.8 for both and stick to using the minimum of the two...

This indicates that the medium load "stay the course" heuristic does not work as intended on this link, so maybe we can effectively disable it by setting a high threshold for medium_load? This heuristic clearly is one which has some hidden assumptions (namely that the bottleneck rate does not change all that quickly) which sinmply seem not correct for starlink, no?

BTW< the achieved rate does not drop during bufferbloat (at least not fast enough to not over-fill the buffers) otherwise we would not see bufferbloat. So the question is what is going on here.
Potential options could be (totally made up):

  1. starlink delays the reverse ACK streams to employ TCP's ACK clocking to throttle the sender
  2. starlink might actually drop or ECN mark packets it already received to tell the sender to slow down
  3. @gba's testload might use a better behaved TCP like BBR that notices increased RTT and reduces its sending rate.

Mind you these are wild speculations, 1) and 2) might be detectable in packet traces though.

1 Like

The RTT does seem pretty wild with Starlink. Do you think these events are actually even bufferbloat events?

Keep in mind this is for all reflectors so there will be some localized differences given the different paths between return samples.

1 Like

Hard to say. The idle RTT spikes (occasionally) are higher than the raised plateau during upload/download saturation, so we are in a bad spot here. Either be too cautious and reduce the rate for accidental RTT spikes (and sacrificing throughput even if that sacrifice would not have been necessary) or be to insensitive and only start reducing the rate way after congestion started and hence creating a deeper bufferbloat hole to dig out of (and risking a longer and more pronounced high delay condition).

Yes, I wonder whether with starlink it might not be worth to try to only ping the first starling hop (or alternative include that node in the reflector list and react if min(all RTT samples) > threshold), that at least should cut out all the additional RTT variability on the rest of the paths (seems acceptable here as the aim is to control for the actual over-the-air segment which appears the be the biggest problem...)

1 Like

Here are some of the lines around a detected bufferbloat event - does this reveal anything:

1655931202.447609 34782  560    73  24  [1655931202.431839] 8.8.4.4         797    36439  72900  36497  60480  0 medium         low            47524  2340  
1655931202.503006 34782  560    73  23  [1655931202.491725] 1.1.1.1         801    35522  52700  17195  60380  0 medium         low            47524  2340  
1655931202.559077 46434  1042   97  44  [1655931202.548824] 1.0.0.1         800    35843  65700  29886  60380  0 high           medium         47999  2340  
1655931202.598527 46434  1042   96  44  [1655931202.586186] 8.8.8.8         799    32793  63300  30537  60378  0 high           medium         48478  2340  
1655931202.629117 46434  1042   95  44  [1655931202.610185] 8.8.4.4         798    36449  47200  10761  60375  0 high           medium         48962  2340  
1655931202.715702 41268  1058   84  45  [1655931202.702896] 1.1.1.1         802    35546  59600  24078  60373  0 high           medium         49451  2340  
1655931202.763512 40762  767    82  32  [1655931202.750082] 1.0.0.1         801    35870  62900  27057  60370  0 high           medium         49945  2340  
1655931202.800530 40762  767    81  32  [1655931202.788620] 8.8.8.8         800    32821  61500  28707  60368  0 high           medium         50444  2340  
1655931202.829007 40762  767    80  32  [1655931202.814406] 8.8.4.4         799    36459  47100  10651  60365  0 high           medium         50948  2340  
1655931202.914274 44651  1091   87  46  [1655931202.903321] 1.1.1.1         803    35566  56300  20754  60363  0 high           medium         51457  2340  
1655931202.981465 45160  1258   87  53  [1655931202.965989] 1.0.0.1         802    35908  74800  38930  60361  0 high           medium         51971  2340  
1655931203.013607 45160  1258   86  53  [1655931202.983103] 8.8.8.8         801    32840  52100  19279  60358  0 high           medium         52490  2340  
1655931203.055998 45160  1258   86  53  [1655931203.040585] 8.8.4.4         800    36491  69400  32941  60356  0 high           medium         53014  2340  
1655931203.127514 42652  988    80  42  [1655931203.115683] 1.1.1.1         804    35595  64800  29234  60354  0 high           medium         53544  2340  
1655931203.176150 42652  988    79  42  [1655931203.162658] 1.0.0.1         803    35939  67500  31592  60352  0 high           medium         54079  2340  
1655931203.239964 47828  1059   88  45  [1655931203.224995] 8.8.8.8         802    32897  90000  57160  60349  0 high           medium         54619  2340  
1655931203.283513 47828  1059   87  45  [1655931203.268654] 8.8.4.4         801    36548  93700  57209  60347  0 high           medium         55165  2340  
1655931203.335508 55316  923    100 39  [1655931203.320799] 1.1.1.1         805    35625  65800  30205  60345  0 high           medium         55716  2340  
1655931203.373933 55316  923    99  39  [1655931203.356972] 1.0.0.1         804    35960  57600  21661  60343  0 high           medium         56273  2340  
1655931203.402400 51734  932    91  39  [1655931203.387227] 8.8.8.8         803    32912  48000  15103  60341  0 high           medium         56835  2340  
1655931203.450167 51734  932    91  39  [1655931203.438060] 8.8.4.4         802    36570  59000  22452  60339  0 high           medium         57403  2340  
1655931203.573119 46088  889    80  37  [1655931203.562170] 1.1.1.1         806    35692  103000 67375  60337  1 high           medium         57977  2340  
1655931203.615563 34534  730    59  31  [1655931203.603549] 1.0.0.1         805    36025  101000 65040  60334  2 medium         medium         57977  2340  
1655931203.665238 34534  730    59  31  [1655931203.654404] 8.8.8.8         804    32990  111000 78088  60334  3 medium         medium         57977  2340  
1655931203.690525 34534  730    59  31  [1655931203.680552] 8.8.4.4         803    36630  97500  60930  60334  4 medium         medium         57977  2340  
1655931203.749230 34463  644    59  27  [1655931203.737905] 1.1.1.1         807    35731  75000  39308  60334  4 medium         medium         57977  2340  
1655931203.791011 34463  644    59  27  [1655931203.780518] 1.0.0.1         806    36062  73300  37275  60334  4 medium         medium         57977  2340  
1655931203.818000 47318  976    81  41  [1655931203.804427] 8.8.8.8         805    33018  61400  28410  60334  4 high           medium         58556  2340  
1655931203.866335 47318  976    80  41  [1655931203.853917] 8.8.4.4         804    36660  66700  30070  60332  4 high           medium         59141  2340  
1655931203.980917 35872  626    60  26  [1655931203.968490] 1.1.1.1         808    35797  102000 66269  60330  4 medium         medium         59141  2340  
1655931204.043897 32991  787    55  33  [1655931204.031496] 1.0.0.1         807    36145  120000 83938  60330  4 medium         medium         59141  2340  
1655931204.085434 32991  787    55  33  [1655931204.074240] 8.8.8.8         806    33111  127000 93982  60330  4 medium         medium         59141  2340  
1655931204.123116 23435  493    39  21  [1655931204.107319] 8.8.4.4         805    36739  116000 79340  60330  4 medium         idle           59141  2386  
1655931204.162293 23435  493    39  20  [1655931204.149830] 1.1.1.1         809    35840  78800  43003  60231  4 medium         idle           59141  2386  
1655931204.207526 23435  493    39  20  [1655931204.193959] 1.0.0.1         808    36187  79000  42855  60231  4 medium         idle           59141  2386  
1655931204.233719 43668  904    73  37  [1655931204.221903] 8.8.8.8         807    33148  70900  37789  60231  4 medium         medium         59141  2386  
1655931204.283811 43668  904    73  37  [1655931204.269658] 8.8.4.4         806    36776  74700  37961  60231  4 medium         medium         59141  2386  
1655931204.397239 32233  683    54  28  [1655931204.385714] 1.1.1.1         810    35915  111000 75160  60231  4 medium         medium         59141  2386  
1655931204.422877 32233  683    54  28  [1655931204.409611] 1.0.0.1         809    36241  90700  54513  60231  3 medium         medium         59141  2386  
1655931204.435636 35407  1067   59  44  [1655931204.420900] 8.8.8.8         808    33180  65800  32652  60231  2 medium         medium         59141  2386  
1655931204.477866 35407  1067   59  44  [1655931204.467458] 8.8.4.4         807    36807  68300  31524  60231  1 medium         medium         59141  2386  
1655931204.601772 37939  1079   64  45  [1655931204.589685] 1.1.1.1         811    35990  111000 75085  60231  2 medium         medium         59141  2386  
1655931204.663261 24603  490    41  20  [1655931204.651376] 1.0.0.1         810    36332  128000 91759  60231  3 medium         idle           59141  2386  
1655931204.686361 24603  490    41  20  [1655931204.675003] 8.8.8.8         809    33262  116000 82820  60231  4 medium         idle           59141  2386  
1655931204.730511 24603  490    41  20  [1655931204.717683] 8.8.4.4         808    36885  115000 78193  60231  5 medium_delayed idle_delayed   22142  2000  
1655931204.804628 28275  870    127 43  [1655931204.792413] 1.1.1.1         812    36063  109000 73010  61541  5 high_delayed   medium_delayed 22142  2000  
1655931204.842413 27368  696    123 34  [1655931204.829921] 1.0.0.1         811    36402  107000 70668  61541  6 high_delayed   medium_delayed 22142  2000  
1655931204.857660 27368  696    123 34  [1655931204.847230] 8.8.8.8         810    33317  88300  55038  61541  6 high_delayed   medium_delayed 22142  2000  
1655931204.917674 27368  696    123 34  [1655931204.907033] 8.8.4.4         809    36952  104000 67115  61541  7 high_delayed   medium_delayed 22142  2000  
1655931204.984757 23127  649    104 32  [1655931204.972329] 1.1.1.1         813    36112  85400  49337  61541  6 high_delayed   medium_delayed 22142  2000  
1655931205.010052 23127  649    104 32  [1655931204.999021] 1.0.0.1         812    36437  72000  35598  61541  5 high_delayed   medium_delayed 22142  2000 

Full data here: https://pastebin.com/w44KgjmG

The upload shaper rate is very low but there seems to be significant upload achieved rate compared to the shaper rate, e.g. around 50%. Is that significant or is upload not playing a significant part here? I recall that for satellite type connections upload is typically an issue. Is the 50% owing to acknowledgements from the modem for each received packet (my knowledge is pretty weak in this area).

Here is another graph depicting the delay classifications:

I am thinking the adjusted delay_thr may be a little too low - so perhaps delay_thr should be increased from 55ms to 75ms?

1 Like

I'm still reading all your posts to see what I can figure out to try, but I thought I would get you the results from the overnight mtr. I ran it for 5 hours during a time which should have not been too congested. There were only a few short blips of intense network activity during that time period but it was mostly very quiet:

HOST: OpenWrt                                    Loss%   Snt   Last   Avg  Best  Wrst StDev
@Not a TXT record
  1. AS???    100.64.0.1 (100.64.0.1)             1.1% 18000   34.9  46.0  20.9 982.9  16.1
@Not a TXT record
  2. AS???    172.16.249.2 (172.16.249.2)         1.1% 18000   49.4  46.1  19.1 836.6  15.9
        172.16.249.76 (172.16.249.76)     
@Not a TXT record
     AS???    172.16.249.76 (172.16.249.76)
  3. AS14593  149.19.108.23 (149.19.108.23)       1.3% 18000   36.1  45.8  21.6 692.4  15.6
  4. AS15169  142.250.171.128 (142.250.171.128)   1.4% 18000   45.4  48.6  20.7 589.6  15.7
  5. AS15169  209.85.142.117 (209.85.142.117)     1.2% 18000   48.1  47.0  19.9 445.5  15.7
        142.251.64.201 (142.251.64.201)   
     AS15169  142.251.64.201 (142.251.64.201)
  6. AS15169  216.239.47.87 (216.239.47.87)       1.4% 18000   36.1  46.5  18.9 260.4  14.8
        142.251.60.5 (142.251.60.5)       
     AS15169  142.251.60.5 (142.251.60.5)
        142.251.60.203 (142.251.60.203)   
     AS15169  142.251.60.203 (142.251.60.203)
  7. AS15169  dns.google (8.8.8.8)                1.0% 18000   38.8  46.0  18.9 1128.  16.9

There has been a lot of speculation about how Starlink actually works at a lower level, but my rudimentary understanding is that there are time slots of transmission given to each dish and it would make sense that during low usage time one dish may be allocated fewer time slots and during times of heavy use they may allocate more time slots to a dish. But if a user starts transferring data for too long they may then reduce the time slots allocated to them, to be fair to everyone, since there are a fixed number of possible transmission time slots that need to be shared.

So it kind of makes sense that we see those bandwidth fluctuations, and also it might not scale very neatly, as they could have an idle dish get X time slots every Y period, then double it for more bandwidth, then double that again, so it might not be very granular in how it responds. There could just be a few bandwidth buckets, I believe I've seen speculation about that before but nothing confirmed.

It would be a lot easier if Starlink just implemented this on their side since they know their algorithm!

1 Like

I haven't had a chance to test much but I wanted to note that I did a brief test with the lua version, to see if it was different, and found that it doesn't work at all because of this issue (which is Starlink's fault)

I am beginning to think I would not want (current) starlink as my ISP... IMHO an ISP should be in the business of transporting the users valid IP packets, and leave the content of the packets alone while doing that...

Thanks for the long term mtr data, unfortunately this really does not seem to help as all hops look similarly bad (judged from the StDev). I guess instead of mtr we really would need something storing the individual ICMP results and then plot them over time.
Sorry for making you do busy work... on a reasonably stable fixed rate link such multi-hour mtr runs can be quite revealing, but on your link I really only see that the link is very variable, but you knew that before :wink:

1 Like

This reminds me that I think that ideally CAKE-autorate code could be switched between different delay data sources...
For the RTT data that mainly just means to report the RTT duplicated as OWDs, and have the rest of the code act independently on uplink and downlink data.
The next step would be to include new delay sources like ICMP timestamps, and (my favorite) NTP data samples, or even old-school IPv4 timestamps.

1 Like

I gave up on ICMP type 13 because OpenWrt has no package that supports it well. nping is broken. hping3 did a good job but is not an official package and there is insufficient motivation to make it one. And it seems ICMP type 13 is not all that well supported by reflectors or blocked for security reasons. And then we now see that Starlink filters it out.

NTP seems to work reliably but that package you identified above blocks rapid fire. It would be great if that package could be modified to allow rapid fire.

Good old RTT suffers from lack of directionality but since I think managing one-way (mostly download) saturation mostly dominates anyway, this doesn't seem to be such a big deal.

I did wonder about firing off an NTP packet to get the directionality of bufferbloat and then punishing the specific direction, but I imagined the lag that would introduce would be undesirable - so better to just punish both. Does it seem reasonable to you that waiting for this NTP response would hurt performance? One could detect delays based on RTT and then when those exceed a threshold fire off NTP packet to determine direction and then punish the offending direction.

Whereas with the plots I've done on LTE connections with CAKE-aurorate I feel like the right thing is happening more or less, I feel uneasy about the Starlink ones. Something doesn't feel quite right. I'm hoping this can be improved with tweaking the default parameters which were, after all, based on my own LTE connection.

I should perhaps summarize a set of parameters for @gba to try because I've fired off too many suggestions above.

1 Like

I wonder how such cuts in bandwidth would compare with the bufferbloat effect of bandwidth variation in LTE owing to signal conditions or cell tower saturation. I am guessing such cuts are more like steps that cause big latency spikes and make autorate hard because it is reacting to big step changes than more gradual changes but this is speculation.

1 Like

@gba I wonder if you could produce another output exactly like you did, but please set:

output_processing_stats=1 # enable (1) or disable (0) output monitoring lines showing processing stats
output_cake_changes=0     # enable (1) or disable (0) output monitoring lines showing cake bandwidth changes
debug=0			  # enable (1) or disable (0) out of debug lines

with the following settings:

reflector_ping_interval_s=0.15 # (seconds, e.g. 0.2s or 2s)
no_pingers=6

delay_thr_ms=75 # (milliseconds)

bufferbloat_detection_window=8  # number of samples to retain in detection window
bufferbloat_detection_thr=5     # number of delayed samples for bufferbloat detection

achieved_rate_adjust_bufferbloat=0.85 # how rapidly to reduce achieved rate upon detection of bufferbloat 
shaper_rate_adjust_bufferbloat=0.85   # how rapidly to reduce shaper rate upon detection of bufferbloat 
shaper_rate_adjust_load_high=1.05    # how rapidly to increase shaper rate upon high load detected 
shaper_rate_adjust_load_low=0.8     # how rapidly to return to base shaper rate upon idle or low load detected 

Ping interval at 0.15s with 6 reflectors means 40 responses per second i.e. one response every 25ms. So that should increase responsiveness.

The increased delay_thr_ms will serve to raise the light orange threshold line here:

this will serve to increase the threshold at which a response sample is considered delayed relative to the baseline for the given reflector.

With:

bufferbloat_detection_window=8  # number of samples to retain in detection window
bufferbloat_detection_thr=5     # number of delayed samples for bufferbloat detection

this means that out of the last 8 samples at least 5 must be delayed to detect bufferbloat (which is indicated in the data by appending _delayed after low, medium or high load). You have used this before and it seemed to work OK.

Then these:

achieved_rate_adjust_bufferbloat=0.85 # how rapidly to reduce achieved rate upon detection of bufferbloat 
shaper_rate_adjust_bufferbloat=0.85   # how rapidly to reduce shaper rate upon detection of bufferbloat 

are more aggressive in terms of punishing the shaper rate upon bufferbloat detection.

And these:

shaper_rate_adjust_load_high=1.05    # how rapidly to increase shaper rate upon high load detected 
shaper_rate_adjust_load_low=0.8     # how rapidly to return to base shaper rate upon idle or low load detected 

will allow more rapid ascent during high load and also more rapid descent on low load.

@moeller0 do these seem reasonable suggestions?

1 Like

Well thanks to both of you for being willing to look at this, even though you don't have Starlink. If this works then I'm sure there will be other Starlink users that will be interested in this autorate script.

I'll give your parameters a try and post the results.

1 Like

OK, I ran those parameters twice:

test1:

test1.log

I forgot to record the average download speed for this 60s period but I think it was around 50 Mbps?

Then I ran it again since I forgot the download speed the first time and it was faster:

test2.log

For that one the reported average download speed was 68 Mbps and upload 6.5 Mbps.

I did a couple tests before and after, with autorate disabled but cake just set to 200/30 and I got right around 100 Mbps download average. Thanks.

I'll plot the data and see how that looks visually when I get back home.

How does it feel to you compared to the previous settings?

For me the main purpose of the autorate script is to allow responsiveness when the connection is not in use (base rate is the safe harbour) but allow managed turbo excursions for when I need sustained heavy download or upload in a way that retains responsiveness to keep browsing snappy and Zoom and Teams working well.

Thanks. I haven't really run it much in actual use, so I'll have to try that out, especially in the evenings when Starlink gets congested and even more variable. What you are describing is exactly what I would desire as well.

I'm trying to do a check first to make sure it's not running too fast to be sucking too much CPU on the router. I don't see it pegging any core in htop, but it is still like 50% on all 4 cores when in use at 0.1s interval, and one test I did made it seem like the CPU may be constraining it. Not really CAKE itself, as it can do CAKE without breaking a sweat, but when I run the autorate script it really takes a lot more CPU on it.

BTW if you haven't installed and enabled irqbalance you should. Also I was thinking rather than 0.1s and 4 reflectors 0.15s and 6 reflectors would be preferable in terms of not sending too many pings out per second to each reflector. It could be that 40 pings per second is not really giving any benefit and just eating up unecessary CPU cycles.

Another consideration is that reported CPU use can be quite high if governor is not performance and has down clocked CPU. Since then the percentage would be lower if the CPU is clocked up to full rate.

I worked hard to try to get the CPU consumption down as much as possible but admittedly this is coded up in bash. That allows really easy tweaking and it's probably good enough for most cases.

On my RT3000 with sub 100Mbit/s rates this is no issue at all, but perhaps with Starlink rates where even just CAKE at 300Mbit/s will break typical routers this is more of an issue at such high rates.

I haven't tested show much increasing the ping response frequency actually helps. I originally assumed the more the better but on my connection halving it from 40 per second to 20 per second did not make much of a difference, whilst halving CPU consumption of the bash processes.

This just doesn't look right to me:

Zoomed in on first detected bufferbloat event:

Data right before first detected bufferbloat event:

1656084983	65212	1349	44	26	[1656084983.122905]	8.8.4.4	76	38515	99800	61346	77482	0	medium	medium	146248	5000
1656084983	65212	1349	44	26	[1656084983.148015]	1.1.1.1	80	39679	97000	57378	77482	0	medium	medium	146248	5000
1656084983	65212	1349	44	26	[1656084983.175446]	1.0.0.1	79	40474	76500	36062	77482	0	medium	medium	146248	5000
1656084983	65212	1349	44	26	[1656084983.179084]	8.8.8.8	78	41781	73000	31250	77482	0	medium	medium	146248	5000
1656084983	65212	1349	44	26	[1656084983.190818]	8.8.4.4	77	38543	66800	28285	77482	0	medium	medium	146248	5000
1656084983	117383	2159	80	43	[1656084983.212981]	1.1.1.1	81	39701	62000	22321	77482	0	high	medium	153560	5000
1656084983	117383	2159	76	43	[1656084983.298165]	8.8.8.8	79	41830	91000	49219	77478	0	high	medium	161238	5000
1656084983	88958	1646	55	32	[1656084983.301289]	1.0.0.1	80	40531	97800	57326	77474	0	medium	medium	161238	5000
1656084983	88958	1646	55	32	[1656084983.310131]	8.8.4.4	78	38587	83200	44657	77474	0	medium	medium	161238	5000
1656084983	88958	1646	55	32	[1656084983.334161]	1.1.1.1	82	39739	78700	38999	77474	0	medium	medium	161238	5000
1656084983	88958	1646	55	32	[1656084983.387807]	8.8.8.8	80	41868	80700	38870	77474	0	medium	medium	161238	5000
1656084983	88958	1646	55	32	[1656084983.404128]	1.0.0.1	81	40586	96000	55469	77474	0	medium	medium	161238	5000
1656084983	88958	1646	55	32	[1656084983.415769]	8.8.4.4	79	38633	84700	46113	77474	0	medium	medium	161238	5000
1656084983	93659	1916	58	38	[1656084983.445029]	1.1.1.1	83	39785	85900	46161	77474	0	medium	medium	161238	5000
1656084984	93659	1916	58	38	[1656084983.488994]	8.8.8.8	81	41902	76700	34832	77474	0	medium	medium	161238	5000
1656084984	93659	1916	58	38	[1656084983.489751]	1.0.0.1	82	40626	81500	40914	77474	0	medium	medium	161238	5000
1656084984	122687	1503	76	30	[1656084983.503660]	8.8.4.4	80	38662	68500	29867	77474	0	high	medium	169299	5000
1656084984	122687	1503	72	30	[1656084983.533508]	1.1.1.1	84	39813	68600	28815	77470	0	medium	medium	169299	5000
1656084984	122687	1503	72	30	[1656084983.585787]	8.8.8.8	82	41930	70600	28698	77470	0	medium	medium	169299	5000
1656084984	122687	1503	72	30	[1656084983.600551]	1.0.0.1	83	40674	89400	48774	77470	0	medium	medium	169299	5000
1656084984	65955	1340	38	26	[1656084983.645116]	8.8.4.4	81	38728	105000	66338	77470	0	medium	medium	169299	5000
1656084984	65955	1340	38	26	[1656084983.681551]	1.1.1.1	85	39887	114000	74187	77470	0	medium	medium	169299	5000
1656084984	65955	1340	38	26	[1656084983.693534]	8.8.8.8	83	41962	74300	32370	77470	0	medium	medium	169299	5000
1656084984	65955	1340	38	26	[1656084983.694251]	1.0.0.1	84	40711	78400	37726	77470	0	medium	medium	169299	5000
1656084984	65955	1340	38	26	[1656084983.702332]	8.8.4.4	82	38748	59200	20472	77470	0	medium	medium	169299	5000
1656084984	98389	2004	58	40	[1656084983.748416]	1.1.1.1	86	39923	76100	36213	77470	0	medium	medium	169299	5000
1656084984	98389	2004	58	40	[1656084983.770665]	8.8.8.8	84	41967	47300	5338	77470	0	medium	medium	169299	5000
1656084984	98389	2004	58	40	[1656084983.775347]	1.0.0.1	85	40730	60400	19689	77470	0	medium	medium	169299	5000
1656084984	79587	1653	47	33	[1656084983.862325]	8.8.4.4	83	38828	119000	80252	77470	1	medium	medium	169299	5000
1656084984	79587	1653	47	33	[1656084983.910456]	1.1.1.1	87	40018	135000	95077	77470	2	medium	medium	169299	5000
1656084984	79587	1653	47	33	[1656084983.921600]	1.0.0.1	86	40792	103000	62270	77470	2	medium	medium	169299	5000
1656084984	112570	1884	66	37	[1656084983.927185]	8.8.8.8	85	42025	100000	58033	77470	2	medium	medium	169299	5000
1656084984	112570	1884	66	37	[1656084983.930750]	8.8.4.4	84	38872	83700	44872	77470	2	medium	medium	169299	5000
1656084984	112570	1884	66	37	[1656084983.950517]	1.1.1.1	88	40049	71500	31482	77470	2	medium	medium	169299	5000
1656084984	112570	1884	66	37	[1656084983.984427]	1.0.0.1	87	40813	62500	21708	77470	2	medium	medium	169299	5000
1656084984	112570	1884	66	37	[1656084983.989364]	8.8.8.8	86	42044	61500	19475	77470	2	medium	medium	169299	5000
1656084984	112570	1884	66	37	[1656084983.999595]	8.8.4.4	85	38881	48400	9528	77470	1	medium	medium	169299	5000
1656084984	78323	1250	46	25	[1656084984.101923]	1.1.1.1	89	40127	119000	78951	77470	1	medium	low	169299	5000
1656084984	78323	1250	46	25	[1656084984.147923]	1.0.0.1	88	40897	125000	84187	77470	2	medium	low	169299	5000
1656084984	71074	1224	41	24	[1656084984.150398]	8.8.8.8	87	42120	119000	76956	77470	2	medium	low	169299	5000
1656084984	71074	1224	41	24	[1656084984.172852]	1.1.1.1	90	40172	85800	45673	77470	2	medium	low	169299	5000
1656084984	71074	1224	41	24	[1656084984.172852]	8.8.4.4	86	38960	118000	79119	77470	3	medium	low	169299	5000
1656084984	71074	1224	41	24	[1656084984.196598]	8.8.8.8	88	42139	61400	19280	77470	3	medium	low	169299	5000
1656084984	71074	1224	41	24	[1656084984.196861]	1.0.0.1	89	40925	69600	28703	77470	3	medium	low	169299	5000
1656084984	71074	1224	41	24	[1656084984.224259]	8.8.4.4	87	38986	65100	26140	77470	3	medium	low	169299	5000
1656084984	97604	1855	57	37	[1656084984.270739]	1.1.1.1	91	40211	79700	39528	77470	2	medium	medium	169299	5000
1656084984	57081	1027	33	20	[1656084984.392066]	8.8.8.8	89	42249	153000	110861	77470	2	medium	low	169299	5000
1656084984	57081	1027	33	20	[1656084984.397966]	1.0.0.1	90	41051	167000	126075	77470	3	medium	low	169299	5000
1656084984	57081	1027	33	20	[1656084984.411982]	8.8.4.4	88	39096	149000	110014	77470	4	medium	low	169299	5000
1656084984	57081	1027	33	20	[1656084984.427582]	1.1.1.1	92	40303	133000	92789	77470	4	medium	low	169299	5000
1656084984	57081	1027	33	20	[1656084984.435454]	8.8.8.8	90	42299	92300	50051	77470	4	medium	low	169299	5000
1656084984	57081	1027	33	20	[1656084984.436915]	1.0.0.1	91	41111	102000	60949	77470	4	medium	low	169299	5000
1656084984	97870	1609	57	32	[1656084984.445415]	8.8.4.4	89	39134	78000	38904	77470	4	medium	medium	169299	5000
1656084985	97870	1609	57	32	[1656084984.463905]	1.1.1.1	93	40323	60900	20597	77470	4	medium	medium	169299	5000
1656084985	97870	1609	57	32	[1656084984.511785]	8.8.8.8	91	42325	68500	26201	77470	3	medium	medium	169299	5000
1656084985	97870	1609	57	32	[1656084984.526300]	1.0.0.1	92	41158	88700	47589	77470	2	medium	medium	169299	5000
1656084985	97870	1609	57	32	[1656084984.563247]	8.8.4.4	90	39187	92300	53166	77470	1	medium	medium	169299	5000
1656084985	72543	1763	42	35	[1656084984.616516]	1.1.1.1	94	40391	109000	68677	77470	0	medium	medium	169299	5000
1656084985	72543	1763	42	35	[1656084984.646978]	8.8.8.8	92	42382	99800	57475	77470	0	medium	medium	169299	5000
1656084985	72543	1763	42	35	[1656084984.651752]	1.0.0.1	93	41229	113000	71842	77470	0	medium	medium	169299	5000
1656084985	72543	1763	42	35	[1656084984.657824]	8.8.4.4	91	39234	86800	47613	77470	0	medium	medium	169299	5000
1656084985	114623	2433	67	48	[1656084984.680588]	1.1.1.1	95	40418	68000	27609	77470	0	medium	medium	169299	5000
1656084985	114623	2433	67	48	[1656084984.706659]	8.8.8.8	93	42398	58600	16218	77470	0	medium	medium	169299	5000
1656084985	114623	2433	67	48	[1656084984.709042]	1.0.0.1	94	41253	65500	24271	77470	0	medium	medium	169299	5000
1656084985	114623	2433	67	48	[1656084984.736381]	8.8.4.4	92	39255	60800	21566	77470	0	medium	medium	169299	5000
1656084985	93433	1560	55	31	[1656084984.794925]	1.1.1.1	96	40457	80000	39582	77470	0	medium	medium	169299	5000
1656084985	93433	1560	55	31	[1656084984.813749]	1.0.0.1	95	41280	68900	27647	77470	0	medium	medium	169299	5000
1656084985	93433	1560	55	31	[1656084984.830412]	8.8.8.8	94	42432	77200	34802	77470	0	medium	medium	169299	5000
1656084985	93433	1560	55	31	[1656084984.840269]	8.8.4.4	93	39280	65200	25945	77470	0	medium	medium	169299	5000
1656084985	93433	1560	55	31	[1656084984.872799]	1.1.1.1	97	40470	53800	13343	77470	0	medium	medium	169299	5000
1656084985	126336	1873	74	37	[1656084984.904116]	1.0.0.1	96	41295	57100	15820	77470	0	medium	medium	169299	5000
1656084985	126336	1873	74	37	[1656084984.917717]	8.8.8.8	95	42451	61600	19168	77470	0	medium	medium	169299	5000
1656084985	126336	1873	74	37	[1656084984.936709]	8.8.4.4	94	39298	57800	18520	77470	0	medium	medium	169299	5000
1656084985	72570	1514	42	30	[1656084985.044595]	1.1.1.1	98	40550	121000	80530	77470	1	medium	medium	169299	5000
1656084985	72570	1514	42	30	[1656084985.067464]	8.8.8.8	96	42516	108000	65549	77470	1	medium	medium	169299	5000
1656084985	72570	1514	42	30	[1656084985.067782]	1.0.0.1	97	41370	117000	75705	77470	1	medium	medium	169299	5000
1656084985	72570	1514	42	30	[1656084985.070858]	8.8.4.4	95	39346	87900	48602	77470	1	medium	medium	169299	5000
1656084985	110021	1953	64	39	[1656084985.107164]	1.1.1.1	99	40589	80200	39650	77470	1	medium	medium	169299	5000
1656084985	110021	1953	64	39	[1656084985.118814]	8.8.8.8	97	42528	55500	12984	77470	1	medium	medium	169299	5000
1656084985	110021	1953	64	39	[1656084985.130292]	1.0.0.1	98	41401	73200	31830	77470	1	medium	medium	169299	5000
1656084985	110021	1953	64	39	[1656084985.135716]	8.8.4.4	96	39355	48600	9254	77470	1	medium	medium	169299	5000
1656084985	90519	1929	53	38	[1656084985.222148]	1.1.1.1	100	40639	91000	50411	77470	0	medium	medium	169299	5000
1656084985	90519	1929	53	38	[1656084985.237743]	1.0.0.1	99	41437	78400	36999	77470	0	medium	medium	169299	5000
1656084985	90519	1929	53	38	[1656084985.260986]	8.8.8.8	98	42579	93900	51372	77470	0	medium	medium	169299	5000
1656084985	90519	1929	53	38	[1656084985.268001]	8.8.4.4	97	39392	76900	37545	77470	0	medium	medium	169299	5000
1656084985	106844	2065	63	41	[1656084985.322282]	1.1.1.1	101	40688	90400	49761	77470	0	medium	medium	169299	5000
1656084985	106844	2065	63	41	[1656084985.325753]	1.0.0.1	100	41458	62700	21263	77470	0	medium	medium	169299	5000
1656084985	106844	2065	63	41	[1656084985.326970]	8.8.8.8	99	42596	59900	17321	77470	0	medium	medium	169299	5000
1656084985	106844	2065	63	41	[1656084985.369995]	8.8.4.4	98	39423	70700	31308	77470	0	medium	medium	169299	5000
1656084985	109546	1873	64	37	[1656084985.404978]	1.1.1.1	102	40720	73400	32712	77470	0	medium	medium	169299	5000
1656084985	109546	1873	64	37	[1656084985.427246]	1.0.0.1	101	41476	60000	18542	77470	0	medium	medium	169299	5000
1656084985	109546	1873	64	37	[1656084985.439351]	8.8.8.8	100	42621	68300	25704	77470	0	medium	medium	169299	5000
1656084985	109546	1873	64	37	[1656084985.466254]	8.8.4.4	99	39446	63300	23877	77470	0	medium	medium	169299	5000
1656084986	99062	1831	58	36	[1656084985.516703]	1.1.1.1	103	40760	80800	40080	77470	0	medium	medium	169299	5000
1656084986	99062	1831	58	36	[1656084985.545582]	1.0.0.1	102	41508	74300	32824	77470	0	medium	medium	169299	5000
1656084986	99062	1831	58	36	[1656084985.551295]	8.8.8.8	101	42654	76100	33479	77470	0	medium	medium	169299	5000
1656084986	99062	1831	58	36	[1656084985.573793]	8.8.4.4	100	39472	66200	26754	77470	0	medium	medium	169299	5000
1656084986	99062	1831	58	36	[1656084985.608307]	1.1.1.1	104	40788	69300	28540	77470	0	medium	medium	169299	5000
1656084986	130491	2977	77	59	[1656084985.635377]	8.8.8.8	102	42667	56400	13746	77470	0	high	medium	177763	5000
1656084986	130491	2977	73	59	[1656084985.646114]	1.0.0.1	103	41537	70700	29192	77467	0	medium	medium	177763	5000
1656084986	63743	1095	35	21	[1656084985.717096]	8.8.4.4	101	39538	106000	66528	77467	0	medium	low	177763	5000
1656084986	63743	1095	35	21	[1656084985.777200]	1.1.1.1	105	40881	134000	93212	77467	1	medium	low	177763	5000
1656084986	63743	1095	35	21	[1656084985.787940]	8.8.8.8	103	42729	105000	62333	77467	1	medium	low	177763	5000
1656084986	63743	1095	35	21	[1656084985.790703]	1.0.0.1	104	41607	112000	70463	77467	1	medium	low	177763	5000
1656084986	63743	1095	35	21	[1656084985.802407]	8.8.4.4	102	39585	87400	47862	77467	1	medium	low	177763	5000
1656084986	112286	1754	63	35	[1656084985.812891]	1.1.1.1	106	40906	65900	25019	77467	1	medium	medium	177763	5000
1656084986	112286	1754	63	35	[1656084985.838012]	8.8.8.8	104	42737	50800	8071	77467	1	medium	medium	177763	5000
1656084986	112286	1754	63	35	[1656084985.843660]	1.0.0.1	105	41625	60100	18493	77467	1	medium	medium	177763	5000
1656084986	112286	1754	63	35	[1656084985.880803]	8.8.4.4	103	39606	61500	21915	77467	1	medium	medium	177763	5000
1656084986	62656	1128	35	22	[1656084985.946510]	1.1.1.1	107	40960	95400	54494	77467	0	medium	low	177763	5000
1656084986	62656	1128	35	22	[1656084985.959354]	1.0.0.1	106	41655	72200	30575	77467	0	medium	low	177763	5000
1656084986	62656	1128	35	22	[1656084985.967652]	8.8.8.8	105	42770	76700	33963	77467	0	medium	low	177763	5000
1656084986	62656	1128	35	22	[1656084985.999754]	8.8.4.4	104	39643	76900	37294	77467	0	medium	low	177763	5000
1656084986	62656	1128	35	22	[1656084986.015364]	1.1.1.1	108	40983	64400	23440	77467	0	medium	low	177763	5000
1656084986	130250	2567	73	51	[1656084986.087269]	8.8.8.8	106	42819	91900	49130	77467	0	medium	medium	177763	5000
1656084986	130250	2567	73	51	[1656084986.110704]	1.0.0.1	107	41732	119000	77345	77467	0	medium	medium	177763	5000
1656084986	130250	2567	73	51	[1656084986.131780]	8.8.4.4	105	39707	104000	64357	77467	0	medium	medium	177763	5000
1656084986	75380	1419	42	28	[1656084986.144190]	1.1.1.1	109	41031	89200	48217	77467	0	medium	medium	177763	5000
1656084986	75380	1419	42	28	[1656084986.164289]	1.0.0.1	108	41759	69300	27568	77467	0	medium	medium	177763	5000
1656084986	75380	1419	42	28	[1656084986.166222]	8.8.8.8	107	42845	69300	26481	77467	0	medium	medium	177763	5000
1656084986	75380	1419	42	28	[1656084986.206948]	8.8.4.4	106	39742	74900	35193	77467	0	medium	medium	177763	5000
1656084986	113128	2278	63	45	[1656084986.246199]	1.1.1.1	110	41077	87100	46069	77467	0	medium	medium	177763	5000
1656084986	113128	2278	63	45	[1656084986.277804]	8.8.8.8	108	42881	78900	36055	77467	0	medium	medium	177763	5000
1656084986	113128	2278	63	45	[1656084986.284659]	1.0.0.1	109	41802	85700	43941	77467	0	medium	medium	177763	5000
1656084986	113128	2278	63	45	[1656084986.335627]	8.8.4.4	107	39803	101000	61258	77467	0	medium	medium	177763	5000
1656084986	101334	1999	57	39	[1656084986.359275]	1.1.1.1	111	41131	95200	54123	77467	0	medium	medium	177763	5000
1656084986	101334	1999	57	39	[1656084986.385609]	8.8.8.8	109	42920	82200	39319	77467	0	medium	medium	177763	5000
1656084986	101334	1999	57	39	[1656084986.389213]	1.0.0.1	110	41845	85600	43798	77467	0	medium	medium	177763	5000
1656084986	101334	1999	57	39	[1656084986.407850]	8.8.4.4	108	39834	71600	31797	77467	0	medium	medium	177763	5000
1656084986	101334	1999	57	39	[1656084986.440251]	1.1.1.1	112	41166	77100	35969	77467	0	medium	medium	177763	5000
1656084986	104855	2121	58	42	[1656084986.465417]	8.8.8.8	110	42935	58100	15180	77467	0	medium	medium	177763	5000
1656084987	104855	2121	58	42	[1656084986.477971]	1.0.0.1	111	41872	69000	27155	77467	0	medium	medium	177763	5000
1656084987	104855	2121	58	42	[1656084986.507540]	8.8.4.4	109	39862	68300	28466	77467	0	medium	medium	177763	5000
1656084987	121772	2308	68	46	[1656084986.585051]	8.8.8.8	111	42965	73800	30865	77467	0	medium	medium	177763	5000
1656084987	121772	2308	68	46	[1656084986.595783]	1.0.0.1	112	41914	84800	42928	77467	0	medium	medium	177763	5000
1656084987	121772	2308	68	46	[1656084986.608028]	8.8.4.4	110	39887	64900	25038	77467	0	medium	medium	177763	5000
1656084987	114533	2000	64	40	[1656084986.660606]	1.1.1.1	114	41214	89400	48234	77467	0	medium	medium	177763	5000
1656084987	114533	2000	64	40	[1656084986.685594]	8.8.8.8	112	42992	70600	27635	77467	0	medium	medium	177763	5000
1656084987	114533	2000	64	40	[1656084986.691064]	1.0.0.1	113	41947	75800	33886	77467	0	medium	medium	177763	5000
1656084987	114533	2000	64	40	[1656084986.701655]	8.8.4.4	111	39901	54700	14813	77467	0	medium	medium	177763	5000
1656084987	114533	2000	64	40	[1656084986.748995]	1.1.1.1	115	41250	77800	36586	77467	0	medium	medium	177763	5000
1656084987	100203	1343	56	26	[1656084986.812303]	1.0.0.1	114	42002	97200	55253	77467	0	medium	medium	177763	5000
1656084987	100203	1343	56	26	[1656084986.816606]	8.8.8.8	113	43046	97300	54308	77467	0	medium	medium	177763	5000
1656084987	100203	1343	56	26	[1656084986.854872]	8.8.4.4	112	39969	108000	68099	77467	0	medium	medium	177763	5000
1656084987	98687	1113	55	22	[1656084986.868924]	1.1.1.1	116	41306	97600	56350	77467	0	medium	low	177763	5000
1656084987	98687	1113	55	22	[1656084986.899070]	8.8.8.8	114	43082	79400	36354	77467	0	medium	low	177763	5000
1656084987	98687	1113	55	22	[1656084986.901888]	1.0.0.1	115	42046	86900	44898	77467	0	medium	low	177763	5000
1656084987	98687	1113	55	22	[1656084986.906133]	8.8.4.4	113	39984	55200	15231	77467	0	medium	low	177763	5000
1656084987	98687	1113	55	22	[1656084986.948386]	1.1.1.1	117	41341	77000	35694	77467	0	medium	low	177763	5000
1656084987	124468	2734	70	54	[1656084986.979090]	1.0.0.1	116	42063	59300	17254	77467	0	medium	medium	177763	5000
1656084987	124468	2734	70	54	[1656084986.983225]	8.8.8.8	115	43097	58300	15218	77467	0	medium	medium	177763	5000
1656084987	21681	594	12	11	[1656084987.159135]	8.8.4.4	114	40147	203000	163016	77467	1	low	low	142210	5000
1656084987	39903	907	28	18	[1656084987.205064]	1.1.1.1	118	41529	230000	188659	77484	2	medium	low	142210	5000
1656084987	39903	907	28	18	[1656084987.207511]	8.8.8.8	116	43233	180000	136903	77484	3	medium	low	142210	5000
1656084987	39903	907	28	18	[1656084987.212609]	1.0.0.1	117	42210	190000	147937	77484	4	medium	low	142210	5000
1656084987	39903	907	28	18	[1656084987.226405]	8.8.4.4	115	40273	167000	126853	77484	5	medium_delayed	low_delayed	33917	3000
1656084987	83138	1687	245	56	[1656084987.227202]	1.1.1.1	119	41634	147000	105471	79353	6	high_delayed	medium_delayed	33917	3000
1656084987	83138	1687	245	56	[1656084987.267149]	1.0.0.1	118	42307	140000	97790	79353	7	high_delayed	medium_delayed	33917	3000
1656084987	83138	1687	245	56	[1656084987.268828]	8.8.4.4	116	40340	108000	67727	79353	7	high_delayed	medium_delayed	33917	3000
1656084987	83138	1687	245	56	[1656084987.269064]	8.8.8.8	117	43326	137000	93767	79353	7	high_delayed	medium_delayed	33917	3000
1656084987	83138	1687	245	56	[1656084987.278085]	1.1.1.1	120	41687	95000	53366	79353	6	high_delayed	medium_delayed	33917	3000
1656084987	83138	1687	245	56	[1656084987.300100]	1.0.0.1	119	42333	69100	26793	79353	5	high_delayed	medium_delayed	33917	3000
1656084987	83138	1687	245	56	[1656084987.300786]	8.8.8.8	118	43348	65500	22174	79353	4	high	medium	33917	3000
1656084987	39819	1185	117	39	[1656084987.340592]	8.8.4.4	117	40376	76600	36260	79353	3	high	medium	33917	3000
1656084987	39819	1185	117	39	[1656084987.359885]	1.1.1.1	121	41722	76900	35213	79353	2	high	medium	33917	3000
1656084987	39819	1185	117	39	[1656084987.391101]	8.8.8.8	119	43356	52000	8652	79353	1	high	medium	33917	3000
1656084987	39819	1185	117	39	[1656084987.405305]	1.0.0.1	120	42360	70100	27767	79353	1	high	medium	33917	3000
1656084987	39819	1185	117	39	[1656084987.434271]	8.8.4.4	118	40402	66800	26424	79353	0	high	medium	33917	3000
1656084987	39819	1185	117	39	[1656084987.449201]	1.1.1.1	122	41742	61900	20178	79353	0	high	medium	33917	3000

If anything this looks worse to me - what do you think @moeller0? What gives here?

I'm not sure what to try now.