@bmork I wonder whether you can offer any insight into 4G/5G in respect of how download and/or upload saturation is expected to affect one way delays (measurable using ICMP type 13 or irtt) in both the download and upload directions?
On my connection I am seeing a noticeable increase in upload one way delay on saturating download, whereas on saturating upload there is not a similar increase in download one way delay.
Does this make sense to you?
This seems odd because ordinarily download saturation results in primarily only an increase in download OWD and upload saturation results in primarily only an increase in upload OWD. Or at least that was my understanding up until now. @tievolu is that what you see on your connection?
Yes, this is what I normally see - i.e. exactly what you would expect to see. If you're not seeing that I suspect there's some other issue with your connection.
Right now there's a problem with my cable connection that is impacting the upstream signal, which in turn is restricting my upstream bandwidth to about 4-6Mbps, so the acks associated with the ~300Mbps downstream are sometimes enough to choke the upstream.
Thanks @tievolu I was also wondering about the possible effect of the ack filter in my situation. What does the ack filter do again / how does it work?
@moeller0 any clue why I'm seeing video stutter on download in Teams? I made a log export actually during the call. I'll run that through your plotter and post on this post.
cake's ACK filter will treat ACKs like all other packets and hash them into its (by default) 1024 buckets, but if an ACK is added to a bucket that already contains ACK to the same target IP/port combination, it will check whether the ACKs are mergeable and if they are, the earlier compatible ACKs in the queue will be replaced by the new ACK. This works as ACKs are cumulative, that is the ACK with the highest sequence number is the more important one as the other end will essentially do the same thing if ACN(n) and ACK(n++1) are delivered back-to-back or if ACK(n) is not delivered at all.
IMHO ACK filtering is a bit of a crutch, and the better solution for over-verbose ACKs would be to fix the end-point TCPs and to use less asymmetric links, but that often isn't a real option and so haven ACK filtering as an option for cake is really great!
Samples per reflector:
ReflectorID: 8.8.4.4; N: 2111
ReflectorID: 8.8.8.8; N: 2138
ReflectorID: 94.140.14.140; N: 2133
ReflectorID: 94.140.14.141; N: 2113
ReflectorID: 94.140.15.15; N: 2162
ReflectorID: 94.140.15.16; N: 2112
DL: maximum 95.000%-ile delta delay over all 6 reflectors: 9.820 ms.
DL: maximum 99.000%-ile delta delay over all 6 reflectors: 12.570 ms.
DL: maximum 99.500%-ile delta delay over all 6 reflectors: 14.305 ms.
DL: maximum 99.900%-ile delta delay over all 6 reflectors: 18.050 ms.
DL: maximum 99.950%-ile delta delay over all 6 reflectors: 20.605 ms.
DL: maximum 99.990%-ile delta delay over all 6 reflectors: 68.630 ms.
DL: maximum 99.999%-ile delta delay over all 6 reflectors: 68.630 ms.
UL: maximum 95.000%-ile delta delay over all 6 reflectors: 9.820 ms.
UL: maximum 99.000%-ile delta delay over all 6 reflectors: 12.570 ms.
UL: maximum 99.500%-ile delta delay over all 6 reflectors: 14.305 ms.
UL: maximum 99.900%-ile delta delay over all 6 reflectors: 18.050 ms.
UL: maximum 99.950%-ile delta delay over all 6 reflectors: 20.605 ms.
UL: maximum 99.990%-ile delta delay over all 6 reflectors: 68.630 ms.
UL: maximum 99.999%-ile delta delay over all 6 reflectors: 68.630 ms.
INFO: Writing plot as: ./output.timecourse.pdf
INFO: fn_parse_autorate_log took: 81.6908 seconds.
Is the issue that I can't safely cruise along at 20Mbit/s base rate because Teams can actually generate quite bursty data?
The data looks fairly unremarkable right? The bit in the middle is the call start. But the issue was just general video stutter throughout the call. Audio seemed fine and no complaints about my upload of audio/video (although I didn't ask).
I'm guessing cake rate is too high but the loading is not sufficient to generate discernable bufferbloat in the pings, albeit temporary bursts of data can still result in bufferbloat? But then wouldn't I see that in the pings? How does that work?
My limited understanding is that it effectively merges/combines acks when possible, so that fewer have to be sent. I'm probably oversimplifying it though.
Well, it has two consequences:
a) there is the potential for a little less ACK traffic. For MTU ~1500 TCP Reno will cause ~1/40 of the forward traffic as reverse ACK traffic (for wildly asymmetric links that thinning of the ACK stream that the ACK filter produces can relieve some pressure)
b) the resulting ACK stream at the receiving end will be smoother which results in better overal TCP performance.
Whether that helps in a specific situation is harder to predict than to test
I'm very curious about your thoughts on my Teams call stutter given my post above.
I'm trying to get a handle on situation where cake rate may be too high, giving stutter on bursty live video feed data, but yet not manifest in terms of pings evenly spread across in time. How is the connection capacity in terms of handling video data assessed relative to ICMPs? What I'm trying to wrap my head around is why ICMPs look fine, but cake rate was probably too high.
It seems setting CleanBrowsing family filter DNS results in the Microsoft Exchange front door resulting in Berlin. Can I direct IP lookups to certain domains to a specific DNS (presumably my ISPs DNS would be better for the microsoft domain lookups). This may or may not be a red herring.
The DNS lookups flip flop between London and Hamburg/Berlin:
I agree the autorate timecourse looks pretty innocent, nothing in there to predict stutter in the video... How bursty teams traffic is I do not know, but that is something you should be able to look at in a packet capture, no?
Yes, that looks puzzling... having your "front door" in Berlin certainly does not help, but video conferencing should work across and between continents, so just "over to the continent" should not cause big issues
Or the video sender had a shitty path and the feed came in already choppy... occasionally it is somebody else's ISP that is at fault
I honestly do not know, ever since I switched to my turris omnia under turrisOS, I opted to operate DNS as non-forwarding resolver (using DNSSEC) and never looked back to other DNS approaches. (Well, my ISP only resolves its own SIP-servers via its own DNS servers, so I configured my VoIP-base station to use O2's DNS servers and and disabled the "force local DNS" in my adblocker, which with the advent of dot and doh lost part of its teeth anyways, but I digress).
Well packet loss certainly can cause choppy video, but again this does not need to happen at your end (it is likely though that a variable-rate link like yours is part of the issue).
with instructions to use mtr. I tried mtr from my downstream router to the Microsoft Teams server (I don't have credentials for my upstream NR7101 handy), but I just get:
root@OpenWrt-1:~# mtr -ezb4w -c 100 52.112.139.57
Start: 2023-04-10T20:22:39+0100
HOST: OpenWrt-1 Loss% Snt Last Avg Best Wrst StDev
any idea why? Is it that server doesn't respond to these types of packets?
I found one that seemed associated with Teams that does respond:
So i am getting close, but no response from the end-host. However I have no SLA contract with teams that requires them to respond to pings, so not sure I have standing to complain
Yes, as I joke above responding to ICMP echo requests is mostly voluntary and something busy servers often are configured not to do (or there is a firewall that filters the requests out before they hit the server); this also seems true for at least some on-line game servers.
Not for intermediary hops, these owe us nothing and often use rate-limiting and de-prioritization so ICMP handling does not interfere with their primary job of forwarding/routing data packets.
High loss rate to the finsal host never is good!
Ah good test is to run the irtt test for say 10-30 minutes on an otherwise quiescent network and have a look at the observed packet osses per direction...
when network is quiet for ten minutes? With cake + cake-autorate running?
Stupid question but if I get a VPS can I set a tunnel with that that multiplexes every packet so it is sent ten times and I sacrifice bandwidth to ensure I get next to no packet loss? Or is that rendered moot by existing protocols - use of 'ack', etc. Could my 'ack' filtering as per:
actually be hurting? Update - oh no, I see 'no-ack-filter'. Doh, my mind is shattered. Probably this massive restructuring of cake-autorate (897 additions and 837 deletions) is a factor.
if the network is truly quiescent it should not matter that cake is running, but since this is easy to test, try both with and without cake active?
That idea generically is called Forward Error correction (short FEC), maybe something like:
would work?
No not really TCP does not tolerate random losses all that well, so any method to avoid such losses can help. But it does not come for free, FEC typically increases the size of the data (since it needs to add redundancies) and often also introduces additional delay (e.g. if interleaving is used).
You are not enabling cake's ACK filter at all ;), so I believe the answer about hurt is "no".
Could well be, large changes tend to bring in their own stabilization periods...
Does setting a fixed cake bandwidth at 5Mbit/s in both directions reduce bufferbloat for you as compared to having no cake set at all (as measurable using e.g. https://www.waveform.com/tools/bufferbloat)?
The results remain inconsistent ... but you can definitely feel a good difference with and without cake-autorate as i tend to not get massive amount of jitter and packet loss.
adjust_dl_shaper_rate=1 # enable (1) or disable (0) actually changing the dl shaper rate
adjust_ul_shaper_rate=1 # enable (1) or disable (0) actually changing the ul shaper rate
min_dl_shaper_rate_kbps=5000 # minimum bandwidth for download (Kbit/s)
base_dl_shaper_rate_kbps=525000 # steady state bandwidth for download (Kbit/s)
max_dl_shaper_rate_kbps=1200000 # maximum bandwidth for download (Kbit/s)
min_ul_shaper_rate_kbps=5000 # minimum bandwidth for upload (Kbit/s)
base_ul_shaper_rate_kbps=525000 # steady state bandwidth for upload (KBit/s)
max_ul_shaper_rate_kbps=1200000 # maximum bandwidth for upload (Kbit/s)
I'll defer to @moeller0 here as he's more familiar with your connection type (could you elaborate on that?) and how to assess it and set cake up properly on it. Are you sure the connection capacity varies with time and it's not just a question of working out a fixed cake bandwidth to set? Also even the result you post without cake doesn't look so bad. But then my standards are pretty low since I am familiar with 4G - circa 40ms to 100ms RTT - and keeping RTT under 80ms is the goal! What sort of hardware router do you have I wonder because shaping at this level is computationally expensive.
Even if @moeller0 thinks cake-autorate isn't suited to your use case, could you post some data showing a saturating download and then upload (even if just using a speed test like the waveform cycle)?
almost all of japans endpoints are using cellular like links so the latency can be up to 1000ms for an extended period of time, and during those times they will make everyones connection in my area near 128kbps... which even if you complain to their FCC they will just say too bad so sad @_@
I have almost the exact problems moeller has but im going to assume the UK has things handled a bit better
I will attempt to provide a saturated link, I will post the results asap.
