CAKE w/ Adaptive Bandwidth

This is a bit tricky, but what I think you should do with both versions to compare them is to run a lot of speedtests (uni- and bidirectional ones if possible) and then compare the high-load CDFs plots between both approaches, the middle tracking approach should result in a larger right-ward tail of the high-load CDFs, as we trade in lower latency for higher throughput resulting in more samples with higher RTTs, note when comparing the CDFs look at the raw plots for absolute values, if you look at the delta CDFs the middle-tracking values will absolutely appear smaller (since you subtract a larger baseline) also the code currently expects positive deltaDelays IIRC), however the "long tail area should still be larger.

We can, but what do you intend to do with SMA(mid)-EWMA(min)? This is a genuine part of the queuing driven delay increase range (at least we need to assume that lacking a decent theory what else is hiding behind that part of the experienced delay). As might be visible from my principled objections, I am not a fan of ignoring part of potential bufferbloat without having a decent idea what else these delay increases might be. And as long as I consider that part of "organic bufferbloat" I think autorate should respond to it with rate reduction.

One challenge though, try simply setting the baseline increase EWMA factor close to what I propose above and try again, you should get something that tracks the minimum of the fluctuating line better than the mid SMA (as tracking the middle lacks so far a theoretical foundation*)

*) Not saying there is no theoretical foundation, but at least so far I fail to understand what that would be)

What would you recommend altering this to for testing? BTW is your main objection to SMA that we have to pfaff about with keeping track of rtt/owd history? I find in my mind it easier to think about SMA period than the alpha(s) of EWMA. And I think computationally they are much of a muchness in bash.

I am speculating here but could it be that bufferbloat associated with connection between modem and cell tower results in the very short term increase say from 35ms to 200ms I am used to, and latency between cell tower and internet is associated with the average increase from 35ms to 80ms? The cell tower latency I can't do anything about, but we can ensure connection to cell tower doesn't result in bufferbloat.

My thinking is that we can't ignore spread for the reason that it will serve to significantly reduce our minimum and thus render the expected values very high and ever closer or passed to beyond our delta threshold, and this presumably varies with time. So either we work with tracking minimums + spread or fudge and track middles. The kind of correctable bufferbloat I have seen cake-autorate address is corrected very quickly as in circa 300ms.

So say at the times I've seen average latency go from 35ms (with I think tight spread) to 80ms (with I think loose spread), it's no use holding down bandwidth to 5 Mbit/s because this doesn't actually help reduce latency?

But now with tracking middle the issue is not so bad (half as bad?) given:

With the red the delta_thr ought really to be pushed out to the right (taking variance into account). With the orange we can get away more with keeping delta_thr static? Because the increase from A to B is not as big as the increase from X to Y (which from my observations has resulted in pinning bandwidth to a minimum in settings that otherwise work fine for much of the time).

Y=2X=2B=4A; B=2A

A=10ms; B=20ms; B-A=10ms

X=20ms; Y=40ms; Y-X=20ms

I think the following:
alpha_baseline_increase=0.9 # how rapidly baseline RTT is allowed to increase
should be close to your 20 sample window (but it will not be identical ), but I do not think that either makes a reasonable default. Also I would not use alpha_baseline_decrease=0.9 but would always set the baseline to the sample value if the sample is smaller than the baseline, in fact I though we actually did that?

My main objection to SMA is that is much costlier especially for larger windows without actually bringing any significant advantage over EWMA ether for conservative minimum tracking or even for mid-tracking. By playing with alpha you can change the influence of the past in EWMA without an increase in operations, but for SMA you need to maintain more sample data and perform more operations. So my point is SMA does not buy us much for considerable higher effort, and I mention that e.g. TCP and cake internally also use EWMA to track things like RTTs so there is precedence that EWMA is "good enough".

From your perspective it makes little sense to differentiate between the over-the-air-segment and the over-fiber segment of your access link if these are essentially always in series, no? Especially since "cell tower" latency very much also depends on how much data you add to the load of the base station (assuming that over-the-air is indeed faster than basestation to internet).

We do not ignore the spread, we use it to tailor our thresholds, the goal is to find a threshold that nicely separates between the idle-RTT distribution and the congested-RTT distribution, the smaller the spread of both distributions the closer their medians/means/centers can be for a desired classification performance. But the variance of the idle and loaded RTT is a fact that we should accept as "data" we have to deal with (unless we can explain and model some part of that away). The spread has zero effect ob the minimum, as the minimum is a physical fact: over a given network path the speed of signal propagation limits put in a hard stop in how low we can go. Anything slower than that must be caused by the stuff we are trying to control for... so to repeat the variance has no effect on the true baseline, but it affects in where we can set our congestion detection threshold for deltaDelay values, the larger the variance the larger a threshold we need to set to avoid unacceptable high rates of false congestion detections. So yes, the spread/variance is important and we should not ignore it, but simply stuffing half of in into the 'baseline" does IMHO not solve the problem at its root.

Well then you set your minimum rate and or your threshold too low, if you are willing to accept noticeable increased latency for higher throughput. Mind you setting policy on the responsiveness/throughput trade-off is very much your prerogative and all choices a user is happy with are fine, but I think if you are willing to accept 45ms more latency just set the threshold 45ms higher?

I fail to see how tracking the minimum or the "middle" (caveat these RTT distributions are not nicely symmetric but tend to show considerable skewness) make the problem of differentiating unloaded and loaded samples?

Yes, because we are "hiding" some of the variable change in RTT into the baseline, but by doing so we implicitly accept the increase in latency-under-load decrease in responsiveness that comes with it, without having a simple control parameter that allows us to predict the total increase in latency-under-load with the current mode we can argue, if we have X values > than Y ms the controller triggers and we counter act bufferbloat withY being conceptually easy to replace by a pretty constant number per reflector, e.g. if 1.1.1.1 is > baseline+threshold = (20+15) = 35ms we detect a congestion vote, with the new mode with a variable baseline we can't do that anymore, making it much harder to reason about what a congestive epoch actually is, no?
Yes the current mode already allows to adjust the baselines slowly, but that is because we have to empirically detect the true baseline from empiric data with an unknown load state and we know that the ground truth baseline per reflector very much is not going to be static for all time, network paths can and do change affecting the pure propagation delay between controller and reflector.

Again we already have a toggle to allow faster convergence of the baseline estimate, it is just I do not think it helps us doing a better job of keeping responsiveness high under (sustained) load.

BTW, sqm-autorate went the route of allowing the threshold to be auto-adjusted, which IMHO has similar issues as accounting part of the variable delay as baseline dynamically, both suffer from the fact that we have no ground truth about true path congestion and hence can end up folding in true congestive delay increases into the "expected normal" state, which in both cases is probably fine for throughput, but detrimental for responsiveness. In both cases it is mostly the automatic part that concerns me. If your link behaves as shown, I have zero issues if you would change the threshold e.g. based on time of day (assuming the 80ms RTTs happen at a predictable time) with the rationale that during most of the day you prefer higher responsiveness, but at peak-hour you rather want higher quality streaming; it is really mostly the make this by automatically adjusting some delay related control variable that gives me pause*.

*) Again mainly because I can see this easily resulting in catastrophic failures, where autorate adjusts too much resulting in a link with atrocious latency under load...

But isn't the point that setting just one static number will not work well given the dynamically changing variance? We need something that will take into account the changing variance.

But wait, cell tower latency I cannot control. The cell tower presumably has a ton of users and my contribution may be peanuts compared to the overall flow. And in this condition I've seen that with SQM off I get 60 Mbit/s and with cake-autorate on I see 5 Mbit/s. So in this situation where the latency has increased to 80ms from 30ms (and variance increased I think) that increase I cannot control, but I can stop the bufferbloat relating to link between modem and cell tower. Again this is speculation on my part but it fits in my head and with what I have experienced. I have asked Vodafone to look into this, but it's next to impossible to get them to have an Engineer look at a cell tower. It involves about 10 or 20 calls and pestering on various channels and waiting in queues with infuriatingly stupid music and then answering questions about wifi and speed tests and postcode for signal in the area, etc.

Anyway if we do not get the lift from middle increase associated with variance increase to baseline we otherwise need a way to compensate for that if we are working with the minimum that gets displaced horizontally significantly and so then we'd need two EWMAs - one to track minimum and one to track middle or maximum and use that somehow to adjust the delta threshold?

I'm open minded about EWMA vs SMA but I really think just working with minimum and threshold may not be cutting it.

Also I'm getting really nice performance in my tests throughout the day - how can I square that? By the way everything is over WireGuard save for the pings from the router. I know that complicates things a bit but I think at this bandwidth / latency level NordVPN doesn't mess things up too much. At least this way Vodafone can't mess about with my data so much as I saw without VPN.

OK here is some data having run various speed tests, etc:

I suppose we should take into account I just swapped out EWMA min with my SMA mid without changing any thresholds, and I suppose with SMA mid the delta thresholds ought to have been lowered a bit for any fair comparison.

Also I should verify that it's not just that cake-autorate is never even hitting the bandwidth ceiling given my fancy new outdoor antenna/router. Because otherwise it could just be those speedtests are not long enough to actually result in oscillation around the maximum. That's certainly possible given my bandwidth has increased so much now.

This is where we disagree, this changing variance is indicative of bufferbloat somewhere along the path, if it affects all reflectors reducing the rate seems imminently reasonable... if your ISP expects more "pressure" from your UE to assign more airtime you will need to increase the threshold to allow more pressure to build up.

No, but your traffic does affect cell tower latency just not alone so if the cell tower uplink is congested reducing your rate is still a reasonable action.

Not really, but we need OWDs to disentangle this, you have decent control over your uplink, but much less over the downlink. But here the point is you are willing to accept an increase in latency under load of 45ms, so why not make that explicit and change the threshold?

As I said, fine if yo do that manually or wit a timed script, but auto-adjustment requires a reliable why to assign components of the variable delays to different root causes and I fail to see a robust and reliable solution for that. I really think the current heuristic we use does its job pretty well, and am not sure more automatism is going to help more or hurt more (in general, for each individual link we probably can find some arbitrary changes that improve the perceived autorate performance, but for all/most links I predict these changes will not be the same).

That is a reasonable policy position to take, so increase alpha_baseline_increase and maybe remove the reset on negative sample RTT. I do not think however that this is a generally applicable change so would prefer to keep the default as is.

Great! Well your autotracking approach will start with something not-inflated after periods of idleness, then a quick speedtest is pushing up the SMA a bit, but with speedtest legs only taking ~10 seconds most of the oneway speedtests will not push the baseline up high enough to result in a big enough loaded latency increase to matter (assuming that the speedtest actually manages to drive your link into saturation to begin with, depending on the range from current shaper rate to max achievable shaper rate a speedtest leg might not be long enough to cause reliable saturation).
It might be time to switch to longer running speedtests, like fast.co which can be configured to run for 60 seconds I believe, or iperf2/3/netperf/flent to see how things behave if the load hits sauration and stays there.

Yes, that is a reasonable step forward to test how robust and reliable the new mode behaves. I might well be off my rocker and things just work as fine as with the old mode, only one way to find out :wink:

You probably noticed, this input file triggered a small bug in the plotting routine (actually in fn_propose_delay_thresholds() )

Not sure what to make out of it though the raw delay samples look pretty constant and no clear indicator that the data transfers reached a ceiling (not saying this was without sufficient saturation, only that the timecourse and CFDs do not allow me to recognize unambiguous signs of saturation.)

Sorry, wrong file I believe, the parser chokes on today's file and needs more work.
The log file contains unadorned entries like:

rtt_baselines_us[208.67.222.123]=66815+(62500-66800)/20

Maybe these could be prefixed with something like:
DEBUG; 2022-11-25-14:34:11; 1669386851.324215;
to parse as proper DEGUG type records?

Here is the new plot:

OK since it seems I needed longer tests I just went for tons of random tests including bidirectional 240 netperf-eu and concurrent Google internet speed tests and waveform bufferbloat. I presume this may have thrashed things and resulted in some bufferbloat given the present settings:

The output from the betterpseedtest.sh looked very ugly with mention of 300ms or so.

Ah crap yes sorry about those lines. Can't we strip everything out that doesn't have DATA_HEADER OR DATA or something?

I am bracing myself for a damning report that marks the end of my SMA project :smiley:.

Here is what I see on Excel:

The bits I've looked at seem OK:

These are functionally ignored already and do not crash the parser (I was wrong about that above) they just produce tons of WARNINGs.

cake-autorate.log.timecourse:

cake-autorate.log.old.timecourse:

cake-autorate_2022-11-25_15_35_46.log.timecourse:

All of these show negative deltaDelay values (some more some less).

OKAY, look at the third plot the starting period up (to say 100 sec), here there is a load that hits the ceiling and gets throttled back with the expected series of above threshold deltaDelays (note the spikes in the raw Delay data in darker ochre) indicating autorate actually engaging... not sure about the later data. Do you have by chance MTR results over the VPN to the speedtests endpoints, I am curious how far these appear to be away.

H


ere an example from a speedtest on my link, note how during bi-directionally saturating load (first test epoch, later two sequential speedtests) there is a stream of above-threshold deltaDelays that seem mostly absent in your plots or at least not correlated with the load. That makes me a bit dubious on what is limiting your speedtests above.
Note, also how during speedtests the raw sample RTTs (darker shades of ochre and moss*) increase during the test, while in your tests this increase seems mostly missing...
Not sure what to make out of this....

*) These are looking so broad as there was an outlying relfector with considerably higher baseline that just dominates the plot.

Agreed the plots of my data look weird to me? What's going on? Maybe some 4G thing is kicking in now that I have a much better connection. But this is all UDP over WireGuard (NordVPN). Hmm. I'm not seeing anything wrong with my RTTs though...

Isn't this for example some healthy oscillation around my 80 Mbit/s maximum:

DATA	 2022-11-25-15:30:36	1669390236	1669390236	59543	3330	98	36	1669390236	 208.67.222.123	858	40271	43250	3163	15208	40271	43250	3163	16401	0	0	 dl_high	 ul_low	61097	9090
DATA	 2022-11-25-15:30:36	1669390236	1669390236	59543	3330	97	36	1669390236	 94.140.14.140	859	34970	32800	-2400	15206	34970	32800	-2400	16401	0	0	 dl_high	 ul_low	61707	9090
DATA	 2022-11-25-15:30:36	1669390237	1669390237	59543	3330	96	36	1669390236	 208.67.220.220	859	37725	38450	989	15204	37725	38450	989	16401	0	0	 dl_high	 ul_low	62324	9090
DATA	 2022-11-25-15:30:36	1669390237	1669390237	57339	3250	92	35	1669390237	 156.154.70.4	859	55946	51000	-4996	15202	55946	51000	-4996	16401	0	0	 dl_high	 ul_low	62947	9090
DATA	 2022-11-25-15:30:36	1669390237	1669390237	57339	3250	91	35	1669390237	 156.154.71.3	859	37953	38700	1029	15200	37953	38700	1029	16401	0	0	 dl_high	 ul_low	63576	9090
DATA	 2022-11-25-15:30:36	1669390237	1669390237	57339	3250	90	35	1669390237	 8.8.8.8	859	41152	45500	5030	15198	41152	45500	5030	16401	0	0	 dl_high	 ul_low	64211	9090
DATA	 2022-11-25-15:30:36	1669390237	1669390237	57339	3250	89	35	1669390237	 94.140.15.16	859	50990	50500	-347	15196	50990	50500	-347	16401	0	0	 dl_high	 ul_low	64853	9090
DATA	 2022-11-25-15:30:36	1669390237	1669390237	57943	3355	89	36	1669390237	 156.154.70.5	859	36613	34600	-2003	15194	36613	34600	-2003	16401	0	0	 dl_high	 ul_low	65501	9090
DATA	 2022-11-25-15:30:36	1669390237	1669390237	57943	3355	88	36	1669390237	 185.228.168.10	859	37076	33350	-3576	15192	37076	33350	-3576	16401	0	0	 dl_high	 ul_low	66156	9090
DATA	 2022-11-25-15:30:36	1669390237	1669390237	57943	3355	87	36	1669390237	 208.67.222.123	859	40789	51500	11228	15190	40789	51500	11228	16387	0	0	 dl_high	 ul_low	66817	9180
DATA	 2022-11-25-15:30:36	1669390237	1669390237	57943	3355	86	36	1669390237	 94.140.14.140	860	35293	43650	8679	15188	35293	43650	8679	16387	0	0	 dl_high	 ul_low	67485	9180
DATA	 2022-11-25-15:30:37	1669390237	1669390237	61600	3339	91	36	1669390237	 208.67.220.220	860	37663	36550	-1175	15186	37663	36550	-1175	16387	0	0	 dl_high	 ul_low	68159	9180
DATA	 2022-11-25-15:30:37	1669390237	1669390237	61600	3339	90	36	1669390237	 9.9.9.10	860	35908	25800	-10258	15185	35908	25800	-10258	16387	0	0	 dl_high	 ul_low	68840	9180
DATA	 2022-11-25-15:30:37	1669390237	1669390237	61600	3339	89	36	1669390237	 156.154.70.4	860	55261	45300	-10646	15183	55261	45300	-10646	16387	0	0	 dl_high	 ul_low	69528	9180
DATA	 2022-11-25-15:30:37	1669390237	1669390237	61600	3339	88	36	1669390237	 156.154.71.3	860	37910	36150	-1803	15181	37910	36150	-1803	16387	0	0	 dl_high	 ul_low	70223	9180
DATA	 2022-11-25-15:30:37	1669390237	1669390237	63789	3648	90	39	1669390237	 8.8.8.8	860	41707	43800	2648	15179	41707	43800	2648	16387	0	0	 dl_high	 ul_low	70925	9180
DATA	 2022-11-25-15:30:37	1669390237	1669390237	63789	3648	89	39	1669390237	 94.140.15.16	860	50652	45750	-5240	15177	50652	45750	-5240	16387	0	0	 dl_high	 ul_low	71634	9180
DATA	 2022-11-25-15:30:37	1669390237	1669390237	63789	3648	89	39	1669390237	 156.154.70.5	860	36250	29300	-7313	15176	36250	29300	-7313	16387	0	0	 dl_high	 ul_low	72350	9180
DATA	 2022-11-25-15:30:37	1669390237	1669390237	63789	3648	88	39	1669390237	 185.228.168.10	860	37026	34250	-2826	15174	37026	34250	-2826	16387	0	0	 dl_high	 ul_low	73073	9180
DATA	 2022-11-25-15:30:37	1669390237	1669390237	68179	3828	93	41	1669390237	 208.67.222.123	860	40846	39500	-1289	15172	40846	39500	-1289	16387	0	0	 dl_high	 ul_low	73803	9180
DATA	 2022-11-25-15:30:37	1669390237	1669390237	68179	3828	92	41	1669390237	 94.140.14.140	861	35135	30300	-4993	15170	35135	30300	-4993	16387	0	0	 dl_high	 ul_low	74541	9180
DATA	 2022-11-25-15:30:37	1669390238	1669390238	68179	3828	91	41	1669390238	 9.9.9.10	861	35353	25150	-10758	15169	35353	25150	-10758	16387	0	0	 dl_high	 ul_low	75286	9180
DATA	 2022-11-25-15:30:37	1669390238	1669390238	69105	3764	91	41	1669390238	 156.154.70.4	861	55219	47650	-7611	15167	55219	47650	-7611	16387	0	0	 dl_high	 ul_low	76038	9180
DATA	 2022-11-25-15:30:37	1669390238	1669390238	69105	3764	90	41	1669390238	 156.154.71.3	861	38145	34300	-3610	15165	38145	34300	-3610	16387	0	0	 dl_high	 ul_low	76798	9180
DATA	 2022-11-25-15:30:37	1669390238	1669390238	69105	3764	89	41	1669390238	 8.8.8.8	861	42254	39550	-2157	15164	42254	39550	-2157	16387	0	0	 dl_high	 ul_low	77565	9180
DATA	 2022-11-25-15:30:37	1669390238	1669390238	69105	3764	89	41	1669390238	 94.140.15.16	861	50705	44100	-6552	15162	50705	44100	-6552	16387	0	0	 dl_high	 ul_low	78340	9180
DATA	 2022-11-25-15:30:37	1669390238	1669390238	69105	3764	88	41	1669390238	 156.154.70.5	861	36380	32600	-3650	15160	36380	32600	-3650	16387	0	0	 dl_high	 ul_low	79123	9180
DATA	 2022-11-25-15:30:37	1669390238	1669390238	72326	3803	91	41	1669390238	 185.228.168.10	861	37301	37300	274	15159	37301	37300	274	16387	0	0	 dl_high	 ul_low	79914	9180
DATA	 2022-11-25-15:30:37	1669390238	1669390238	72326	3803	90	41	1669390238	 208.67.222.123	861	41041	38800	-2046	15159	41041	38800	-2046	16387	0	0	 dl_high	 ul_low	80000	9180
DATA	 2022-11-25-15:30:37	1669390238	1669390238	72326	3803	90	41	1669390238	 94.140.14.140	862	34905	32050	-3085	15159	34905	32050	-3085	16373	0	0	 dl_high	 ul_low	80000	9271
DATA	 2022-11-25-15:30:38	1669390238	1669390238	72326	3803	90	41	1669390238	 208.67.220.220	862	38078	38100	437	15159	38078	38100	437	16373	0	0	 dl_high	 ul_low	80000	9271
DATA	 2022-11-25-15:30:38	1669390238	1669390238	67859	3795	84	40	1669390238	 9.9.9.10	862	35295	27500	-7853	15159	35295	27500	-7853	16373	0	0	 dl_high	 ul_low	80000	9271
DATA	 2022-11-25-15:30:38	1669390238	1669390238	67859	3795	84	40	1669390238	 156.154.70.4	862	55169	51000	-4219	15159	55169	51000	-4219	16373	0	0	 dl_high	 ul_low	80000	9271
DATA	 2022-11-25-15:30:38	1669390238	1669390238	67859	3795	84	40	1669390238	 156.154.71.3	862	38245	35200	-2945	15159	38245	35200	-2945	16373	0	0	 dl_high	 ul_low	80000	9271
DATA	 2022-11-25-15:30:38	1669390238	1669390238	67859	3795	84	40	1669390238	 8.8.8.8	862	42562	38300	-3954	15159	42562	38300	-3954	16373	0	0	 dl_high	 ul_low	80000	9271
DATA	 2022-11-25-15:30:38	1669390238	1669390238	60693	3252	75	35	1669390238	 94.140.15.16	862	50875	51500	795	15176	50875	51500	795	16373	0	0	 dl_low	 ul_low	72000	9271
DATA	 2022-11-25-15:30:38	1669390238	1669390238	60693	3252	84	35	1669390238	 156.154.70.5	862	36150	36250	-130	15175	36150	36250	-130	16373	0	0	 dl_high	 ul_low	72720	9271
DATA	 2022-11-25-15:30:38	1669390238	1669390238	60693	3252	83	35	1669390238	 185.228.168.10	862	37236	37750	449	15173	37236	37750	449	16373	0	0	 dl_high	 ul_low	73447	9271
DATA	 2022-11-25-15:30:38	1669390238	1669390238	67866	3703	92	39	1669390238	 208.67.222.123	862	42186	45100	4058	15171	42186	45100	4058	16373	0	0	 dl_high	 ul_low	74181	9271
DATA	 2022-11-25-15:30:38	1669390238	1669390238	67866	3703	91	39	1669390238	 94.140.14.140	863	34798	32400	-2505	15169	34798	32400	-2505	16373	0	0	 dl_high	 ul_low	74922	9271
DATA	 2022-11-25-15:30:38	1669390239	1669390239	67866	3703	90	39	1669390238	 208.67.220.220	863	38143	34550	-3528	15168	38143	34550	-3528	16373	0	0	 dl_high	 ul_low	75671	9271
DATA	 2022-11-25-15:30:38	1669390239	1669390239	67866	3703	89	39	1669390239	 9.9.9.10	863	35673	36300	1004	15166	35673	36300	1004	16373	0	0	 dl_high	 ul_low	76427	9271
DATA	 2022-11-25-15:30:38	1669390239	1669390239	66038	3400	86	36	1669390239	 156.154.71.3	863	38325	37350	-895	15164	38325	37350	-895	16373	0	0	 dl_high	 ul_low	77191	9271
DATA	 2022-11-25-15:30:38	1669390239	1669390239	66038	3400	85	36	1669390239	 8.8.8.8	863	42964	42200	-362	15163	42964	42200	-362	16373	0	0	 dl_high	 ul_low	77962	9271
DATA	 2022-11-25-15:30:38	1669390239	1669390239	66038	3400	84	36	1669390239	 94.140.15.16	863	50452	47050	-3825	15161	50452	47050	-3825	16373	0	0	 dl_high	 ul_low	78741	9271
DATA	 2022-11-25-15:30:38	1669390239	1669390239	66038	3400	83	36	1669390239	 156.154.70.5	863	36353	35100	-1050	15160	36353	35100	-1050	16373	0	0	 dl_high	 ul_low	79528	9271
DATA	 2022-11-25-15:30:38	1669390239	1669390239	71368	3956	89	42	1669390239	 185.228.168.10	863	37043	35900	-1336	15159	37043	35900	-1336	16373	0	0	 dl_high	 ul_low	80000	9271
DATA	 2022-11-25-15:30:38	1669390239	1669390239	71368	3956	89	42	1669390239	 208.67.222.123	863	42861	47800	5613	15159	42861	47800	5613	16373	0	0	 dl_high	 ul_low	80000	9271
DATA	 2022-11-25-15:30:38	1669390239	1669390239	71368	3956	89	42	1669390239	 94.140.14.140	864	34763	34450	-348	15159	34763	34450	-348	16360	0	0	 dl_high	 ul_low	80000	9363
DATA	 2022-11-25-15:30:38	1669390239	1669390239	71368	3956	89	42	1669390239	 208.67.220.220	864	37905	33850	-4293	15159	37905	33850	-4293	16360	0	0	 dl_high	 ul_low	80000	9363
DATA	 2022-11-25-15:30:39	1669390239	1669390239	71963	3877	89	41	1669390239	 9.9.9.10	864	35838	35400	-273	15159	35838	35400	-273	16360	0	0	 dl_high	 ul_low	80000	9363
DATA	 2022-11-25-15:30:39	1669390239	1669390239	71963	3877	89	41	1669390239	 156.154.70.4	864	55869	65500	10331	15159	55869	65500	10331	16360	0	0	 dl_high	 ul_low	80000	9363
DATA	 2022-11-25-15:30:39	1669390239	1669390239	71963	3877	89	41	1669390239	 156.154.71.3	864	39188	46750	8424	15159	39188	46750	8424	16360	0	0	 dl_high	 ul_low	80000	9363
DATA	 2022-11-25-15:30:39	1669390239	1669390239	68843	3719	86	39	1669390239	 8.8.8.8	864	44114	59500	16535	15159	44114	59500	16535	16360	1	1	 dl_high	 ul_low	80000	9363
DATA	 2022-11-25-15:30:39	1669390239	1669390239	68843	3719	86	39	1669390239	 94.140.15.16	864	51602	77000	26547	15159	51602	77000	26547	16360	2	2	 dl_high	 ul_low	80000	9363
DATA	 2022-11-25-15:30:39	1669390239	1669390239	68843	3719	86	39	1669390239	 156.154.70.5	864	37230	55500	19147	15159	37230	55500	19147	16360	3	3	 dl_high	 ul_low	80000	9363
DATA	 2022-11-25-15:30:39	1669390239	1669390239	68843	3719	86	39	1669390239	 185.228.168.10	864	38213	60000	22956	15159	38213	60000	22956	16360	4	4	 dl_high	 ul_low	80000	9363
DATA	 2022-11-25-15:30:39	1669390240	1669390240	70789	3737	88	39	1669390239	 208.67.222.123	864	44461	67500	24638	15199	44461	67500	24638	16511	5	5	 dl_high_bb	 ul_low_bb	63710	8426
DATA	 2022-11-25-15:30:39	1669390240	1669390240	70789	3737	111	44	1669390239	 94.140.14.140	865	35793	55000	20237	15199	35793	55000	20237	16511	6	6	 dl_high_bb	 ul_low_bb	63710	8426
DATA	 2022-11-25-15:30:39	1669390240	1669390240	70789	3737	111	44	1669390240	 208.67.220.220	865	39468	60500	22594	15199	39468	60500	22594	16511	7	7	 dl_high_bb	 ul_low_bb	63710	8426
DATA	 2022-11-25-15:30:39	1669390240	1669390240	70789	3737	111	44	1669390240	 9.9.9.10	865	37380	62000	26162	15199	37380	62000	26162	16511	8	8	 dl_high_bb	 ul_low_bb	63710	8426
DATA	 2022-11-25-15:30:39	1669390240	1669390240	64382	3764	101	44	1669390240	 156.154.71.3	865	40560	60500	21312	15199	40560	60500	21312	16511	9	9	 dl_high_bb	 ul_low_bb	63710	8426
DATA	 2022-11-25-15:30:39	1669390240	1669390240	64382	3764	101	44	1669390240	 156.154.70.4	865	57369	87500	31631	15199	57369	87500	31631	16511	10	10	 dl_high_bb	 ul_low_bb	63710	8426
DATA	 2022-11-25-15:30:39	1669390240	1669390240	64382	3764	101	44	1669390240	 8.8.8.8	865	46004	74000	29885	15199	46004	74000	29885	16511	10	10	 dl_high_bb	 ul_low_bb	63710	8426
DATA	 2022-11-25-15:30:39	1669390240	1669390240	58154	3355	91	39	1669390240	 94.140.15.16	865	52252	72000	20397	15199	52252	72000	20397	16511	10	10	 dl_high_bb	 ul_low_bb	63710	8426
DATA	 2022-11-25-15:30:39	1669390240	1669390240	58154	3355	91	39	1669390240	 156.154.70.5	865	37950	51500	14269	15199	37950	51500	14269	16511	9	9	 dl_high_bb	 ul_low_bb	63710	8426
DATA	 2022-11-25-15:30:39	1669390240	1669390240	58154	3355	91	39	1669390240	 185.228.168.10	865	37958	36650	-1563	15199	37958	36650	-1563	16511	8	8	 dl_high_bb	 ul_low_bb	63710	8426
DATA	 2022-11-25-15:30:39	1669390240	1669390240	58154	3355	91	39	1669390240	 208.67.222.123	865	44944	51000	6538	15199	44944	51000	6538	16511	7	7	 dl_high_bb	 ul_low_bb	63710	8426
DATA	 2022-11-25-15:30:40	1669390240	1669390240	58154	3355	91	39	1669390240	 94.140.14.140	866	36470	45400	9607	15243	36470	45400	9607	16679	6	6	 dl_high_bb	 ul_low_bb	52338	7583
DATA	 2022-11-25-15:30:40	1669390240	1669390240	58368	3080	111	40	1669390240	 208.67.220.220	866	39688	41350	1882	15243	39688	41350	1882	16679	5	5	 dl_high_bb	 ul_low_bb	52338	7583
DATA	 2022-11-25-15:30:40	1669390240	1669390240	58368	3080	111	40	1669390240	 9.9.9.10	866	37565	39650	2269	15243	37565	39650	2269	16663	4	4	 dl_high	 ul_low	52338	7658
DATA	 2022-11-25-15:30:40	1669390240	1669390240	58368	3080	111	40	1669390240	 156.154.70.4	866	57419	59500	2131	15243	57419	59500	2131	16663	3	3	 dl_high	 ul_low	52338	7658
DATA	 2022-11-25-15:30:40	1669390240	1669390240	58368	3080	111	40	1669390240	 156.154.71.3	866	40758	40400	-160	15243	40758	40400	-160	16663	2	2	 dl_high	 ul_low	52338	7658
DATA	 2022-11-25-15:30:40	1669390240	1669390240	47708	2588	91	33	1669390240	 94.140.15.16	866	52352	52500	247	15243	52352	52500	247	16663	1	1	 dl_high	 ul_low	52338	7658
DATA	 2022-11-25-15:30:40	1669390240	1669390240	47708	2588	91	33	1669390240	 156.154.70.5	866	38050	36700	-1250	15243	38050	36700	-1250	16663	0	0	 dl_high	 ul_low	52338	7658
DATA	 2022-11-25-15:30:40	1669390240	1669390240	47708	2588	91	33	1669390240	 185.228.168.10	866	37898	35250	-2708	15243	37898	35250	-2708	16663	0	0	 dl_high	 ul_low	52338	7658
DATA	 2022-11-25-15:30:40	1669390240	1669390240	46873	2544	89	33	1669390240	 208.67.222.123	866	45191	43650	-1294	15243	45191	43650	-1294	16663	0	0	 dl_high	 ul_low	52338	7658
DATA	 2022-11-25-15:30:40	1669390240	1669390240	46873	2544	89	33	1669390240	 94.140.14.140	867	36425	33200	-3270	15243	36425	33200	-3270	16663	0	0	 dl_high	 ul_low	52338	7658
DATA	 2022-11-25-15:30:40	1669390241	1669390241	46873	2544	89	33	1669390240	 208.67.220.220	867	39570	32450	-7238	15240	39570	32450	-7238	16663	0	0	 dl_high	 ul_low	52861	7658
DATA	 2022-11-25-15:30:40	1669390241	1669390241	46873	2544	88	33	1669390241	 9.9.9.10	867	37598	28300	-9265	15238	37598	28300	-9265	16663	0	0	 dl_high	 ul_low	53389	7658
DATA	 2022-11-25-15:30:40	1669390241	1669390241	47706	2700	89	35	1669390241	 156.154.70.4	867	57594	60000	2581	15236	57594	60000	2581	16663	0	0	 dl_high	 ul_low	53922	7658
DATA	 2022-11-25-15:30:40	1669390241	1669390241	47706	2700	88	35	1669390241	 156.154.71.3	867	41138	40700	-58	15233	41138	40700	-58	16663	0	0	 dl_high	 ul_low	54461	7658
DATA	 2022-11-25-15:30:40	1669390241	1669390241	47706	2700	87	35	1669390241	 8.8.8.8	867	46322	43000	-3004	15231	46322	43000	-3004	16663	0	0	 dl_high	 ul_low	55005	7658
DATA	 2022-11-25-15:30:40	1669390241	1669390241	47706	2700	86	35	1669390241	 94.140.15.16	867	52855	55500	3147	15229	52855	55500	3147	16663	0	0	 dl_high	 ul_low	55555	7658
DATA	 2022-11-25-15:30:40	1669390241	1669390241	48067	2754	86	35	1669390241	 156.154.70.5	867	38045	32900	-5150	15226	38045	32900	-5150	16663	0	0	 dl_high	 ul_low	56110	7658
DATA	 2022-11-25-15:30:40	1669390241	1669390241	48067	2754	85	35	1669390241	 185.228.168.10	867	37688	40800	2901	15224	37688	40800	2901	16663	0	0	 dl_high	 ul_low	56671	7658
DATA	 2022-11-25-15:30:40	1669390241	1669390241	48067	2754	84	35	1669390241	 208.67.222.123	867	45386	45050	-141	15222	45386	45050	-141	16663	0	0	 dl_high	 ul_low	57237	7658
DATA	 2022-11-25-15:30:40	1669390241	1669390241	48067	2754	83	35	1669390241	 94.140.14.140	868	36398	35400	-1025	15220	36398	35400	-1025	16663	0	0	 dl_high	 ul_low	57809	7658
DATA	 2022-11-25-15:30:41	1669390241	1669390241	48067	2754	83	35	1669390241	 208.67.220.220	868	39745	37300	-2270	15218	39745	37300	-2270	16663	0	0	 dl_high	 ul_low	58387	7658
DATA	 2022-11-25-15:30:41	1669390241	1669390241	51055	2746	87	35	1669390241	 9.9.9.10	868	38413	45450	7852	15215	38413	45450	7852	16646	0	0	 dl_high	 ul_low	58970	7734
DATA	 2022-11-25-15:30:41	1669390241	1669390241	51055	2746	86	35	1669390241	 156.154.70.4	868	57844	57000	-594	15213	57844	57000	-594	16646	0	0	 dl_high	 ul_low	59559	7734
DATA	 2022-11-25-15:30:41	1669390241	1669390241	51055	2746	85	35	1669390241	 156.154.71.3	868	41775	47800	6662	15211	41775	47800	6662	16646	0	0	 dl_high	 ul_low	60154	7734
DATA	 2022-11-25-15:30:41	1669390241	1669390241	54422	2755	90	35	1669390241	 8.8.8.8	868	46627	43900	-2422	15209	46627	43900	-2422	16646	0	0	 dl_high	 ul_low	60755	7734
DATA	 2022-11-25-15:30:41	1669390241	1669390241	54422	2755	89	35	1669390241	 94.140.15.16	868	52827	49950	-2905	15207	52827	49950	-2905	16646	0	0	 dl_high	 ul_low	61362	7734
DATA	 2022-11-25-15:30:41	1669390241	1669390241	54422	2755	88	35	1669390241	 156.154.70.5	868	38065	37750	-295	15205	38065	37750	-295	16646	0	0	 dl_high	 ul_low	61975	7734
DATA	 2022-11-25-15:30:41	1669390241	1669390241	54422	2755	87	35	1669390241	 185.228.168.10	868	37331	35300	-2388	15203	37331	35300	-2388	16646	0	0	 dl_high	 ul_low	62594	7734
DATA	 2022-11-25-15:30:41	1669390241	1669390241	55322	3017	88	39	1669390241	 208.67.222.123	868	45914	47950	2563	15201	45914	47950	2563	16646	0	0	 dl_high	 ul_low	63219	7734
DATA	 2022-11-25-15:30:41	1669390241	1669390241	55322	3017	87	39	1669390241	 94.140.14.140	869	36193	36100	-298	15199	36193	36100	-298	16646	0	0	 dl_high	 ul_low	63851	7734
DATA	 2022-11-25-15:30:41	1669390242	1669390242	55322	3017	86	39	1669390242	 208.67.220.220	869	39868	38100	-1645	15197	39868	38100	-1645	16646	0	0	 dl_high	 ul_low	64489	7734
DATA	 2022-11-25-15:30:41	1669390242	1669390242	55322	3017	85	39	1669390242	 9.9.9.10	869	38423	37400	-1013	15195	38423	37400	-1013	16646	0	0	 dl_high	 ul_low	65133	7734
DATA	 2022-11-25-15:30:41	1669390242	1669390242	57353	3267	88	42	1669390242	 156.154.70.4	869	57719	54500	-3344	15193	57719	54500	-3344	16646	0	0	 dl_high	 ul_low	65784	7734
DATA	 2022-11-25-15:30:41	1669390242	1669390242	57353	3267	87	42	1669390242	 156.154.71.3	869	41668	37050	-4725	15191	41668	37050	-4725	16646	0	0	 dl_high	 ul_low	66441	7734
DATA	 2022-11-25-15:30:41	1669390242	1669390242	57353	3267	86	42	1669390242	 8.8.8.8	869	46462	39050	-7577	15189	46462	39050	-7577	16646	0	0	 dl_high	 ul_low	67105	7734
DATA	 2022-11-25-15:30:41	1669390242	1669390242	57353	3267	85	42	1669390242	 94.140.15.16	869	52827	51000	-1827	15187	52827	51000	-1827	16646	0	0	 dl_high	 ul_low	67776	7734
DATA	 2022-11-25-15:30:41	1669390242	1669390242	57353	3267	84	42	1669390242	 156.154.70.5	869	37943	32000	-6065	15186	37943	32000	-6065	16646	0	0	 dl_high	 ul_low	68453	7734
DATA	 2022-11-25-15:30:41	1669390242	1669390242	53509	2630	78	34	1669390242	 185.228.168.10	869	37028	33400	-3931	15184	37028	33400	-3931	16646	0	0	 dl_high	 ul_low	69137	7734
DATA	 2022-11-25-15:30:41	1669390242	1669390242	53509	2630	77	34	1669390242	 208.67.222.123	869	46021	39950	-5964	15182	46021	39950	-5964	16646	0	0	 dl_high	 ul_low	69828	7734
DATA	 2022-11-25-15:30:41	1669390242	1669390242	53509	2630	76	34	1669390242	 94.140.14.140	870	36155	35850	-343	15180	36155	35850	-343	16646	0	0	 dl_high	 ul_low	70526	7734
DATA	 2022-11-25-15:30:42	1669390242	1669390242	57791	3230	81	41	1669390242	 9.9.9.10	870	38240	33650	-4773	15178	38240	33650	-4773	16646	0	0	 dl_high	 ul_low	71231	7734
DATA	 2022-11-25-15:30:42	1669390242	1669390242	57791	3230	81	41	1669390242	 156.154.70.4	870	57919	59000	1281	15177	57919	59000	1281	16630	0	0	 dl_high	 ul_low	71943	7811
DATA	 2022-11-25-15:30:42	1669390242	1669390242	57791	3230	80	41	1669390242	 156.154.71.3	870	41763	37550	-4118	15175	41763	37550	-4118	16630	0	0	 dl_high	 ul_low	72662	7811
DATA	 2022-11-25-15:30:42	1669390242	1669390242	61290	3255	84	41	1669390242	 8.8.8.8	870	46632	45050	-1412	15173	46632	45050	-1412	16630	0	0	 dl_high	 ul_low	73388	7811
DATA	 2022-11-25-15:30:42	1669390242	1669390242	61290	3255	83	41	1669390242	 94.140.15.16	870	52727	56000	3172	15171	52727	56000	3172	16630	0	0	 dl_high	 ul_low	74121	7811
DATA	 2022-11-25-15:30:42	1669390242	1669390242	61290	3255	82	41	1669390242	 156.154.70.5	870	37550	37100	-843	15170	37550	37100	-843	16630	0	0	 dl_high	 ul_low	74862	7811
DATA	 2022-11-25-15:30:42	1669390242	1669390242	61290	3255	81	41	1669390242	 185.228.168.10	870	36961	36150	-878	15168	36961	36150	-878	16630	0	0	 dl_high	 ul_low	75610	7811
DATA	 2022-11-25-15:30:42	1669390242	1669390242	61290	3255	81	41	1669390242	 208.67.222.123	870	45804	40600	-5421	15166	45804	40600	-5421	16630	0	0	 dl_high	 ul_low	76366	7811
DATA	 2022-11-25-15:30:42	1669390243	1669390243	60169	2150	78	27	1669390242	 94.140.14.140	871	36085	36650	494	15165	36085	36650	494	16630	0	0	 dl_high	 ul_low	77129	7811
DATA	 2022-11-25-15:30:42	1669390243	1669390243	60169	2150	78	27	1669390243	 208.67.220.220	871	40015	44700	4832	15163	40015	44700	4832	16630	0	0	 dl_high	 ul_low	77900	7811
DATA	 2022-11-25-15:30:42	1669390243	1669390243	60169	2150	77	27	1669390243	 9.9.9.10	871	38285	47300	9059	15161	38285	47300	9059	16630	0	0	 dl_high	 ul_low	78679	7811
DATA	 2022-11-25-15:30:42	1669390243	1669390243	55027	2271	69	29	1669390243	 156.154.71.3	871	41515	40850	-913	15179	41515	40850	-913	16630	0	0	 dl_low	 ul_low	70811	7811
DATA	 2022-11-25-15:30:42	1669390243	1669390243	55027	2271	77	29	1669390243	 156.154.70.4	871	58869	77000	19081	15178	58869	77000	19081	16630	1	1	 dl_high	 ul_low	71519	7811
DATA	 2022-11-25-15:30:42	1669390243	1669390243	55027	2271	76	29	1669390243	 8.8.8.8	871	46197	40500	-6132	15176	46197	40500	-6132	16630	1	1	 dl_high	 ul_low	72234	7811
DATA	 2022-11-25-15:30:42	1669390243	1669390243	58780	3047	81	39	1669390243	 156.154.70.5	871	37565	46750	9199	15174	37565	46750	9199	16630	1	1	 dl_high	 ul_low	72956	7811
DATA	 2022-11-25-15:30:42	1669390243	1669390243	58780	3047	80	39	1669390243	 94.140.15.16	871	53502	72000	19272	15172	53502	72000	19272	16630	2	2	 dl_high	 ul_low	73685	7811
DATA	 2022-11-25-15:30:42	1669390243	1669390243	58780	3047	79	39	1669390243	 185.228.168.10	871	37046	40950	3989	15171	37046	40950	3989	16630	2	2	 dl_high	 ul_low	74421	7811
DATA	 2022-11-25-15:30:42	1669390243	1669390243	58780	3047	78	39	1669390243	 208.67.222.123	871	45574	43000	-2804	15169	45574	43000	-2804	16630	2	2	 dl_high	 ul_low	75165	7811
DATA	 2022-11-25-15:30:43	1669390243	1669390243	58851	3003	78	38	1669390243	 208.67.220.220	872	40315	45650	5634	15167	40315	45650	5634	16630	2	2	 dl_high	 ul_low	75916	7811
DATA	 2022-11-25-15:30:43	1669390243	1669390243	58851	3003	77	38	1669390243	 9.9.9.10	872	37615	43600	5314	15166	37615	43600	5314	16630	2	2	 dl_high	 ul_low	76675	7811
DATA	 2022-11-25-15:30:43	1669390243	1669390243	58851	3003	76	38	1669390243	 156.154.71.3	872	41500	41650	134	15164	41500	41650	134	16614	2	2	 dl_high	 ul_low	77441	7889
DATA	 2022-11-25-15:30:43	1669390243	1669390243	58851	3003	75	38	1669390243	 156.154.70.4	872	58969	68500	9631	15164	58969	68500	9631	16614	2	2	 dl_low	 ul_low	77441	7889
DATA	 2022-11-25-15:30:43	1669390243	1669390243	58851	3003	75	38	1669390243	 8.8.8.8	872	46092	40450	-5747	15164	46092	40450	-5747	16614	1	1	 dl_low	 ul_low	77441	7889
DATA	 2022-11-25-15:30:43	1669390243	1669390243	65733	2977	84	37	1669390243	 94.140.15.16	872	53877	63500	9997	15162	53877	63500	9997	16614	1	1	 dl_high	 ul_low	78215	7889
DATA	 2022-11-25-15:30:43	1669390243	1669390243	65733	2977	84	37	1669390243	 156.154.70.5	872	37493	41250	3684	15161	37493	41250	3684	16614	1	1	 dl_high	 ul_low	78997	7889
DATA	 2022-11-25-15:30:43	1669390243	1669390243	65733	2977	83	37	1669390243	 185.228.168.10	872	36966	37500	454	15159	36966	37500	454	16614	0	0	 dl_high	 ul_low	79786	7889
DATA	 2022-11-25-15:30:43	1669390243	1669390243	65733	2977	82	37	1669390243	 208.67.222.123	872	44934	37700	-7874	15159	44934	37700	-7874	16614	0	0	 dl_high	 ul_low	80000	7889
DATA	 2022-11-25-15:30:43	1669390243	1669390243	66907	3348	83	42	1669390243	 94.140.14.140	873	36323	38900	2814	15159	36323	38900	2814	16614	0	0	 dl_high	 ul_low	80000	7889
DATA	 2022-11-25-15:30:43	1669390244	1669390244	66907	3348	83	42	1669390244	 208.67.220.220	873	39565	35500	-4815	15159	39565	35500	-4815	16614	0	0	 dl_high	 ul_low	80000	7889
DATA	 2022-11-25-15:30:43	1669390244	1669390244	66907	3348	83	42	1669390244	 9.9.9.10	873	37460	42750	5134	15159	37460	42750	5134	16614	0	0	 dl_high	 ul_low	80000	7889
DATA	 2022-11-25-15:30:43	1669390244	1669390244	69402	3377	86	42	1669390244	 156.154.71.3	873	40993	40850	-650	15159	40993	40850	-650	16614	0	0	 dl_high	 ul_low	80000	7889
DATA	 2022-11-25-15:30:43	1669390244	1669390244	69402	3377	86	42	1669390244	 156.154.70.4	873	59444	67500	8531	15159	59444	67500	8531	16614	0	0	 dl_high	 ul_low	80000	7889
DATA	 2022-11-25-15:30:43	1669390244	1669390244	69402	3377	86	42	1669390244	 8.8.8.8	873	46657	52000	5908	15159	46657	52000	5908	16614	0	0	 dl_high	 ul_low	80000	7889
DATA	 2022-11-25-15:30:43	1669390244	1669390244	72823	4016	91	50	1669390244	 94.140.15.16	873	54645	64000	10122	15159	54645	64000	10122	16614	0	0	 dl_high	 ul_low	80000	7889
DATA	 2022-11-25-15:30:43	1669390244	1669390244	72823	4016	91	50	1669390244	 156.154.70.5	873	37933	43300	5807	15159	37933	43300	5807	16614	0	0	 dl_high	 ul_low	80000	7889
DATA	 2022-11-25-15:30:43	1669390244	1669390244	72823	4016	91	50	1669390244	 185.228.168.10	873	36811	34550	-2416	15159	36811	34550	-2416	16614	0	0	 dl_high	 ul_low	80000	7889
DATA	 2022-11-25-15:30:43	1669390244	1669390244	72823	4016	91	50	1669390244	 208.67.222.123	873	44944	45550	616	15159	44944	45550	616	16614	0	0	 dl_high	 ul_low	80000	7889
DATA	 2022-11-25-15:30:44	1669390244	1669390244	72823	4016	91	50	1669390244	 94.140.14.140	874	36413	42000	5677	15159	36413	42000	5677	16614	0	0	 dl_high	 ul_low	80000	7889
DATA	 2022-11-25-15:30:44	1669390244	1669390244	72847	3801	91	48	1669390244	 208.67.220.220	874	39825	47300	7734	15159	39825	47300	7734	16614	0	0	 dl_high	 ul_low	80000	7889
DATA	 2022-11-25-15:30:44	1669390244	1669390244	72847	3801	91	48	1669390244	 9.9.9.10	874	37195	36250	-1210	15159	37195	36250	-1210	16614	0	0	 dl_high	 ul_low	80000	7889
DATA	 2022-11-25-15:30:44	1669390244	1669390244	72847	3801	91	48	1669390244	 156.154.70.4	874	59094	60500	1056	15159	59094	60500	1056	16614	0	0	 dl_high	 ul_low	80000	7889
DATA	 2022-11-25-15:30:44	1669390244	1669390244	72847	3801	91	48	1669390244	 156.154.71.3	874	39535	37850	-3143	15159	39535	37850	-3143	16614	0	0	 dl_high	 ul_low	80000	7889
DATA	 2022-11-25-15:30:44	1669390244	1669390244	73089	3821	91	48	1669390244	 8.8.8.8	874	46232	50500	3843	15159	46232	50500	3843	16598	0	0	 dl_high	 ul_low	80000	7967
DATA	 2022-11-25-15:30:44	1669390244	1669390244	73089	3821	91	47	1669390244	 156.154.70.5	874	38305	56500	18567	15159	38305	56500	18567	16598	1	1	 dl_high	 ul_low	80000	7967
DATA	 2022-11-25-15:30:44	1669390244	1669390244	73089	3821	91	47	1669390244	 94.140.15.16	874	55745	84000	29355	15159	55745	84000	29355	16598	2	2	 dl_high	 ul_low	80000	7967
DATA	 2022-11-25-15:30:44	1669390244	1669390244	73089	3821	91	47	1669390244	 185.228.168.10	874	37543	56500	19689	15159	37543	56500	19689	16598	3	3	 dl_high	 ul_low	80000	7967
DATA	 2022-11-25-15:30:44	1669390244	1669390244	75280	3997	94	50	1669390244	 208.67.222.123	874	45169	60000	15056	15159	45169	60000	15056	16598	3	3	 dl_high	 ul_low	80000	7967
DATA	 2022-11-25-15:30:44	1669390245	1669390245	75280	3997	94	50	1669390245	 94.140.14.140	875	37425	56000	19587	15159	37425	56000	19587	16598	4	4	 dl_high	 ul_low	80000	7967
DATA	 2022-11-25-15:30:44	1669390245	1669390245	75280	3997	94	50	1669390245	 208.67.220.220	875	40720	61000	21174	15187	40720	61000	21174	16776	5	5	 dl_high_bb	 ul_low_bb	67752	7170
DATA	 2022-11-25-15:30:44	1669390245	1669390245	75280	3997	111	55	1669390245	 9.9.9.10	875	37885	60000	22804	15187	37885	60000	22804	16776	6	6	 dl_high_bb	 ul_low_bb	67752	7170
DATA	 2022-11-25-15:30:44	1669390245	1669390245	71044	3604	104	50	1669390245	 156.154.71.3	875	39633	43050	3514	15187	39633	43050	3514	16776	6	6	 dl_high_bb	 ul_low_bb	67752	7170
DATA	 2022-11-25-15:30:44	1669390245	1669390245	71044	3604	104	50	1669390245	 156.154.70.4	875	58919	69000	9906	15187	58919	69000	9906	16776	6	6	 dl_high_bb	 ul_low_bb	67752	7170
DATA	 2022-11-25-15:30:44	1669390245	1669390245	71044	3604	104	50	1669390245	 8.8.8.8	875	45607	56500	10268	15187	45607	56500	10268	16776	6	6	 dl_high_bb	 ul_low_bb	67752	7170
DATA	 2022-11-25-15:30:44	1669390245	1669390245	71044	3604	104	50	1669390245	 94.140.15.16	875	56045	57000	1255	15187	56045	57000	1255	16776	5	5	 dl_high_bb	 ul_low_bb	67752	7170
DATA	 2022-11-25-15:30:44	1669390245	1669390245	71044	3604	104	50	1669390245	 156.154.70.5	875	38433	37500	-805	15187	38433	37500	-805	16776	4	4	 dl_high	 ul_low	67752	7170
DATA	 2022-11-25-15:30:44	1669390245	1669390245	60966	3024	89	42	1669390245	 185.228.168.10	875	37558	37000	-543	15187	37558	37000	-543	16776	3	3	 dl_high	 ul_low	67752	7170
DATA	 2022-11-25-15:30:44	1669390245	1669390245	60966	3024	89	42	1669390245	 208.67.222.123	875	45151	44600	-569	15187	45151	44600	-569	16776	3	3	 dl_high	 ul_low	67752	7170
DATA	 2022-11-25-15:30:45	1669390245	1669390245	60966	3024	89	42	1669390245	 94.140.14.140	876	37520	41300	3874	15187	37520	41300	3874	16776	2	2	 dl_high	 ul_low	67752	7170
DATA	 2022-11-25-15:30:45	1669390245	1669390245	60966	3024	89	42	1669390245	 208.67.220.220	876	39900	36100	-4620	15187	39900	36100	-4620	16776	1	1	 dl_high	 ul_low	67752	7170
DATA	 2022-11-25-15:30:45	1669390245	1669390245	60524	3396	89	47	1669390245	 9.9.9.10	876	38560	54000	16114	15186	38560	54000	16114	16776	1	0	 dl_high	 ul_low	68429	7170
DATA	 2022-11-25-15:30:45	1669390245	1669390245	60524	3396	88	47	1669390245	 156.154.70.4	876	60144	79000	20081	15184	60144	79000	20081	16776	2	1	 dl_high	 ul_low	69113	7170
DATA	 2022-11-25-15:30:45	1669390245	1669390245	59889	3214	86	44	1669390245	 156.154.71.3	876	40753	58000	18367	15182	40753	58000	18367	16776	3	2	 dl_high	 ul_low	69804	7170
DATA	 2022-11-25-15:30:45	1669390245	1669390245	59889	3214	85	44	1669390245	 8.8.8.8	876	45384	44400	-1207	15180	45384	44400	-1207	16758	3	2	 dl_high	 ul_low	70502	7241
DATA	 2022-11-25-15:30:45	1669390245	1669390245	59889	3214	84	44	1669390245	 94.140.15.16	876	56492	55000	-1045	15178	56492	55000	-1045	16758	3	2	 dl_high	 ul_low	71207	7241
DATA	 2022-11-25-15:30:45	1669390245	1669390245	59889	3214	84	44	1669390245	 156.154.70.5	876	38590	32900	-5533	15177	38590	32900	-5533	16758	3	2	 dl_high	 ul_low	71919	7241
DATA	 2022-11-25-15:30:45	1669390245	1669390245	59889	3214	83	44	1669390245	 185.228.168.10	876	37746	32700	-4858	15175	37746	32700	-4858	16758	3	2	 dl_high	 ul_low	72638	7241
DATA	 2022-11-25-15:30:45	1669390245	1669390245	59889	3214	82	44	1669390245	 208.67.222.123	876	45054	40700	-4451	15173	45054	40700	-4451	16758	3	2	 dl_high	 ul_low	73364	7241
DATA	 2022-11-25-15:30:45	1669390246	1669390246	63417	3652	86	50	1669390245	 94.140.14.140	877	37725	35900	-1620	15171	37725	35900	-1620	16758	3	2	 dl_high	 ul_low	74097	7241
DATA	 2022-11-25-15:30:45	1669390246	1669390246	63417	3652	85	50	1669390246	 208.67.220.220	877	39355	31150	-8750	15170	39355	31150	-8750	16758	3	2	 dl_high	 ul_low	74837	7241
DATA	 2022-11-25-15:30:45	1669390246	1669390246	63417	3652	84	50	1669390246	 9.9.9.10	877	38460	34300	-4260	15168	38460	34300	-4260	16758	2	2	 dl_high	 ul_low	75585	7241

Well, maybe re-run the test with the old code (you might need to increase the threshold a bit to make it closer in effect to the mid-corrected baseline data, but start with the same data).

Still, sma not looking so bad here at least?

Can you elaborate w.r.t.?

Yes to me this looks great:

I mean we see clear spikes and clear corrections.

How would I get these?

tracert netperf-eu.bufferbloat.net

Tracing route to demo.tohojo.dk [193.10.227.30]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  OpenWrt.lan [192.168.1.1]
  2     *        *        *     Request timed out.
  3     *        *        *     Request timed out.
  4     *        *        *     Request timed out.
  5     *        *        *     Request timed out.
  6     *        *        *     Request timed out.
  7     *        *        *     Request timed out.
  8     *        *        *     Request timed out.
  9     *        *        *     Request timed out.
 10     *        *        *     Request timed out.
 11     *        *        *     Request timed out.
 12     *        *        *     Request timed out.

I disagree, it does what I expect it to do, but I consider that not to be the right action.

If you allow the same delta threshold (e.g. 10ms) on top of min(baseline say 100) and a spread of say 10ms you end up wih different effective thresholds:
MIN: min(baseline) + threshold: 100+10 = 110
MID: min(baseline) + spread0.5 + threshold = 100 + 100.5+10 = 115
(spread * 0.5 approximates the average deltaDelay for symmetric deltaDelay distributions).

So the same nominal threshold allows higher effective thresholds for the mid-method which will likely result in higher throughput, and higher latency over the min-method, so if you want to see these two perform more similarly in regards to throughput you should adjust the threshold at least statically.

Yes, but the question is is this actually better than what you see with the old min-method, looking at both throughput and induced latency.

Do a packet capture to get the IP addresses from the speedtest servers (taken from before the packets are encrypted). Then use 'mtr -ezb4w -c 100 $IPADDRESSOFSPEEDTESTSERVER` just make sure these mtr probes are going through the VPN tunnel and not around it :wink:

root@OpenWrt:~# mtr -ezb4w -c 100 195.89.146.24
Start: 2022-11-25T16:47:38+0000
HOST: OpenWrt                                Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. AS???    192.168.1.1 (192.168.1.1)       0.0%   100    1.6   1.5   1.3   1.9   0.1
  2. AS???    ???                            100.0   100    0.0   0.0   0.0   0.0   0.0
  3. AS???    ???                            100.0   100    0.0   0.0   0.0   0.0   0.0
  4. AS???    ???                            100.0   100    0.0   0.0   0.0   0.0   0.0
  5. AS???    ???                            100.0   100    0.0   0.0   0.0   0.0   0.0
  6. AS???    ???                            100.0   100    0.0   0.0   0.0   0.0   0.0
  7. AS???    ???                            100.0   100    0.0   0.0   0.0   0.0   0.0
  8. AS???    ???                            100.0   100    0.0   0.0   0.0   0.0   0.0
  9. AS???    ???                            100.0   100    0.0   0.0   0.0   0.0   0.0
 10. AS1273   195.89.146.24 (195.89.146.24)   1.0%   100   70.0  80.6  57.6 101.9   9.7

Mmmh there is something odd with:


Note how the shaper barely hits the ceiling (and only twice does the reflector exceed the thrsold around those moments)? I guess this is not showing all reflectors but only a single one so we do not see all deltaDelays crossing the threshold.

Thanks for the MTR results although this just shows a single node on vodafones network, making me wonder where your VPN endpoint is located? Are you sure the mtr goes through the tunnel?

That 192.168.1.1 is my main router and this is run from a client 192.168.1.2.

Yes I have 0.5s per reflector now. So it's all heavily interleaved. And hence sma at 20 points corresponds with 10s per reflector.

1 Like

Yes, but since the main router packages things into the wireguard tunnel this does not allow me to figure out whether the MTR measured through the tunnel (as it should) or around it like your latency probes for autorate?