CAKE w/ Adaptive Bandwidth [October 2021 to September 2022]

moeller0 · 2021-12-20T07:17:16.000Z

As far as I can tell their aim is to automatically set shaper thresholds based on speedtests and do so and do so on the hours to minute scales.
So from far enough away they do something similar, but closer view reveals their approach probably not suitable for the track a variable rate link second by second challenge this project tackles.

Lynx · 2021-12-20T08:21:57.526Z

Great find @anon89577378. See:

https://patents.google.com/patent/WO2017058657A1/en?assignee=Evenroute&oq=Evenroute

Here is the prosecution history of their EP application:

https://register.epo.org/application?number=EP16852344&lng=en&tab=doclist

They actually describe using OpenWrt firmware on a router and adding their technology on top of that.

Pending claim 1 filed on October 11, 2021 specifies:

Screenshot_20211220-081155954×985 139 KB

It is pertinent to consider that this requires that a web service calculates the QoS values rather than the client. And initial QoS settings are determined by the web service based on connection type and protocol, and then revised by the web service based on the performance measurements conducted by the client, whereupon the determined QoS values are sent from the web service to the client, which applies them.

Can anyone think of good prior art to help the EPO Examiner along? Good prior art would be a prior publication (before September 28, 2015) that discloses all of those steps. Alternatively something that discloses all of those steps bar one would be helpful for inventive step.

I will consider the sprout algorithm:

usenix.org

nsdi13-final113.pdf

479.03 KB

anon89577378 · 2021-12-20T08:43:03.156Z

I became aware of Evenroute in 2019 when we were trying to help a guy on DSLReports with huge bandwidth issues.

They do in fact use a custom version of OpenWRT, and their version 2 router looked exactly like a TP-Link Archer C7.

They're coming out with an x86 router called the "Pro" (not cheap).

IQrouter

Optimizes responsiveness for critical applications — IQrouter

Powerful router with automatic optimization for high-speed lines. Fixes Bufferbloat with advanced traffic management for improved responsiveness on all applications.

moeller0 · 2021-12-20T09:26:10.963Z

So to take a step back here, let me start with a disclaimer:
I like evenroute and have been aware of their fine 'QoS auto-tuning' (and had a number of intersting discussions with Jonathan Foulkes, their founder on the bufferbloat lists). However, I have not been aware that they filed a patent*. But since they do operate their own measurement infrastructure and actually sell a product using that infrastructure (so are as opposite to a patent troll as one can be while still playing the patent game), I am willing to give them more than the benefit of the doubt as IMHO they have a good track record as belonging to the good guys here.

From the little text you posted (not being skilled in reading patents I wonder whether it is worth spending my layman time doing that), I would not see any problem for your/our approach, because they apparently went the speedtest route, while we are going the "if you want to minimize bufferbloat, use actual latency-(under-load-)increase as your yardstick" route. IMHO these might ultimately tackle the same higher level challenge, but sufficiently different so we do not need to bother about theirs (at least that is my understanding). This is in addition to them doing calculations remote versus locally. If at all what we are trying is more related to congestion control work like TCP Vegas, LEDBAT, and BBRv[1|2] and the arl qdisc @dtaht recently posted a link for which mostly are documented in IETF RFCs and as far as I can tell are patent-free prior art which we can refer to should we ever need to bat patent claims away, no?

If my interpretation is acceptable, why would we need to find "prior art" for their approach?

*) Personally, I am not a big fan of patents in computing, but I also understand that to persist in an environment with pervasive patents, companies, especially small ones, probably do not have many attractive alternatives. So I grudgingly shift from an absolute 'software patents are bad' to 'it depends how patents are wielded, with defensive use being acceptable'.

moeller0 · 2021-12-20T09:52:56.249Z

Lochnair:

I haven't really considered this before, but how many reflectors would we need to get good results in a geographic region? I'm not thinking about resources (CPU, bandwidth, etc.), but how many we need to get accurate results, as in do we need different paths?

Good question, even though I would frame it as, "how few reflectors we can get away with", the more the merrier that seems sure. I already know of one big European ISP that is very economical in sizing its peerings with the big transit providers resulting in large parts of the internet (large be fraction of IP address space, not necessarily large in that ISP's customers requests) routinely being affected by bufferbloat at the peering points, so all reflectors behind such a link are potentially "useless" at peak times. This is where spreading the reflectors around can help (as would finding reliable reflectors inside an ISPs network, like e.g. the ISPs own DNS servers). It is however generally hard to impossible to figure out the full network path to a given end-point without at least having traceroute/mtr data from both direction (so each end-point probing the other). That is why aiming at heavily anycasted cloud infrastructure likely served from multiple data centers seems like a workable initial starting point.

To circle back to your question, I do not know how many refelectors we need, but I would propose to start with at least 3 maybe up to ~10 (especially if we go the interleaved sampling from multiple sources always evaluating based on the most recent 3 route)... if we stick to the "minimum" deltaOWD/deltaRTT approach even or odd should not matter, if we go to a voting consensus style an odd number makes things conceptually clearer.

Lynx · 2021-12-20T11:02:21.736Z

moeller0:

I would not see any problem for your/our approach, because they apparently went the speedtest route, while we are going the "if you want to minimize bufferbloat, use actual latency-(under-load-)increase as your yardstick" route

So to be clear, the EP patent seeks to obtain a 20-year monopoly on:

image954×985 139 KB

This claim has been well crafted and is highly generalised. It is satisfied by anything that fulfils all of the requirements. Determining 'performance measurements' is highly general and not limited to speed tests. It may be argued that 'performance measurements' include ICMP measurements.

moeller0:

If my interpretation is acceptable, why would we need to find "prior art" for their approach?

I object to the breadth of this claim because it seeks to cover simply offloading computation of QoS particulars to a remote server - client conducts performance measurements, those are sent to remote server, and remote server determines QoS values based on those measurements, which are then sent to client for application by the client.

moeller0:

This is in addition to them doing calculations remote versus locally.

Yes I think this is the key point; this is not what we have implemented and been working on because rather we have the QoS calculations carried out by the client, and not by the server.

But are there not situations in which such offloading of the QoS calculations to remote server is desirable - or do you think this would never be of interest? I am guessing, but I suppose this is of interest to Evenroute because it means they can set up a subscription model in which user must pay for the router and then also pay to retain access to the server that performs the QoS calculations (and the QoS calculation code can be remotely updated and tweaked).

Generally speaking it is in the public interest to ensure patents that get granted are not overly broad, and if we know about strong prior art it will help ensure the resulting patent secures a more focused monopoly that is more valid.

moeller0:

If at all what we are trying is more related to congestion control work like TCP Vegas, LEDBAT, and BBRv[1|2] and the arl qdisc @dtaht recently posted a link for which mostly are documented in IETF RFCs and as far as I can tell are patent-free prior art which we can refer to should we ever need to bat patent claims away, no?

Do you know whether any of those comprise QoS determinations undertaken at the remote end based on performance measurements by client, which are then sent to client, and applied at client?

In the case of WiFi, is there a sense in which server calculates rate specifics based on performance measurements undertaken by client, then sends those rate specifics to client, which applies them?

This 2012 Microsoft publication looks pretty promising as a novelty knockout:

patentimages.storage.googleapis.com

US8335163.pdf

1924.68 KB

image904×596 50.5 KB

image635×1014 47.3 KB

The QOS client applications 216 monitor the communications of their corresponding computers 210 and measure QOS performance levels related to the communications 106 from their computers 210. For instance, as a communication 106 leaves (or arrives at) a particular computer 210, the QOS client application 216 measures various QOS performance levels related to the communication 106. Network performance levels such as latency, packet loss rate, transmission rate, bandwidth, etc. represent some of the QOS performance levels which the QOS client applications 216 measure.

The QOS client applications 216 also store the measured QOS performance levels in conjunction with the time, date, communication path involved, and other information pertinent to the communications 106. The QOS client applications 216 can submit the measured QOS performance levels to the QOS server application 214 as soon as the measurement occurs. In the alternative, the QOS client applications 216 can wait to submit that information. For instance, the QOS client applications 216 can wait to submit the information until they have measured some selected number of QOS performance levels; until they have monitored some selected number of communications 106; or until they have measured QOS performance levels for some selected number of communication paths (to name but a few of many possibilities).

With further reference to FIG. 3, the path assembly component 312 receives the query responses 308 from the QOS server application 214 . Moreover, the path assembly component 312 parses those query responses 308 to identify QOS performance levels for the individual path segments 220 , 222 , 224 , 226 , 228 , and 230 which might be of interest to the entities 212 within the computer 210 . The path assembly component 312 , upon detecting that an outgoing communication 106 has been created, examines the determined QOS performance levels that might be useful for routing the outgoing communication 106 to select an overall communication path suitable for that communication 106 . For instance, the communication can be routed on an optimal communication path for that communication.

_FailSafe · 2021-12-20T12:25:25.914Z

Anyone else lost in all the legalese verbiage here?

moeller0 · 2021-12-20T12:41:59.136Z

Lynx:

This claim has been well crafted and is highly generalised.

Isn't that how "patents as a game" works nowadays? Try to make as general a claim as you can get away with, giving you loads of opportunity to decide whether to not to consider your patent violated, no?

Are these claims typically interpreted at face value by courts or will courts interpret these claims back to something more specific?

Lynx:

I object to the breadth of this claim because it seeks to cover simply offloading computation of QoS particulars to a remote server - client conducts performance measurements, those are sent to remote server, and remote server determines QoS values based on those measurements, which are then sent to client for application by the client.

I would argue that most software patents can be described in such a way, that they appear pretty obvious... (again not a fan of software patents here).

Lynx:

But are there not situations in which such offloading of the QoS calculations to remote server is desirable - or do you think this would never be of interest?

For us? I do not think remote analysis that makes much sense as we would either require each user to set up their own server somewhere in the internet, or to set up servers for groups of users. The first seems pretty costly (assuming we target all OpenWrt users and not just those with money to spare) and the second a organizational nightmare. In general however something like not having to distribute the logic to all endpoints makes some sense (except the cost of our solution is mostly the traffic shaper itself, a router capable of comfortably shaping a link should also have the capacity to rund the script).

Lynx:

I am guessing, but I suppose this is of interest to Evenroute because it means they can set up a subscription model in which user must pay for the router and then also pay to retain access to the server that performs the QoS calculations (and the QoS calculation code can be remotely updated and tweaked).

I think they are not offering subscriptions, but fold the cost of operating the measurement infrastructure into the one-time purchase price of their routers... I guess they want to avoid that somebody swipes in and eats their lunch... (As I said I like them, and I think that they have a nice product, aimed at people that are no network exerts and that neither want or can spend the time required to become a network expert, but still would like to use competent AQM on their link; and no I am not a paid or unpaid spokesperson for evenroute )

Lynx:

Generally speaking it is in the public interest to ensure patents that get granted are not overly broad, and if we know about strong prior art it will help ensure the resulting patent secures a more focused monopoly that is more valid.

+1; as you realized by now, all I know about the patent system are a few isolated nuggets, leading me to wrong conclusions (Kruger-Dunning at work ) So thanks for all the patient explanation, always happy to learn from experts.

Lynx:

Do you know whether any of those comprise QoS determinations undertaken at the remote end based on performance measurements by client, which are then sent to client, and applied at client?

Well these are more like what we are doing, as these are all sender-based algorithms that take some information sent from the other side into account. IN the extreme one maybe could interpret ECN as based on remote logic.
ECN: an ECN-enabled TCP packet (EXT(0) or ECT-(1) marked) passes over a congested node with ECN-enabled AQM, which changes the ECN pattern in the IP ECN-bitmap from ECT([0|1]) to CE. This CE-marked packet gets to the remote receiver which will see the CE mark in the IP header and in turn will set the ECE ( ECN-Echo) flag in the reverse ACK* packet's TCP header and will do so until the other side (our near end client) will sent the CWR (CongestionWindowReduced) flag on the "forward" TCP header. At which point the remote end will stop sending ECEs (until it get new CE-marked packets).

Here the remote end needs to perform some transformation to get "performance data" (the CE-mark) into actionable information (the ECE flag).

But that feels a bit contrived, IMHO....

_FailSafe:

lost in all the legalese verbiage here?

That has a certain poetic ring to it, albeit poetry in a foreign language I only know a few words of and where I misinterpret others

*) I am simplyfying here, this also happens if the ACK data is not sent as pure ACK packet but piggybacked on a data packet in the reverse direction.

_FailSafe · 2021-12-20T13:16:33.066Z

anon89577378:

They're coming out with an x86 router called the "Pro" (not cheap).

I knew this chassis looked familiar. Looks like they’re probably re-branding Qotom boxes like this:
Qotom-Q355G4 Fanless Mini Industrial PC with 4 Ethernet LAN Intel Core i5 5200U Computer (8G RAM + 64G SSD) https://www.amazon.com/dp/B076PJ4S2Z/ref=cm_sw_r_cp_api_glt_fabc_3BY8CZZ7Z1NQRXZQHJK6

I know Qotom boxes are popular for other firewall solutions (pfSense, Opnsense, etc.), so well played, Evenroute.

moeller0 · 2021-12-20T13:26:36.602Z

_FailSafe:

I know Qotom boxes are popular for other firewall solutions (pfSense, Opnsense, etc.), so well played, Evenroute.

They do not build their own hardware, and as far as I can tell never claimed to do so. Their product is a third-party firmware to run on a very limited set of existing router hardware, that offers a few nice features and is targeted mostly (IMHO) at the non-techie segment of the market, where their features like update notifications, "easy self-serv updates", and "self-tuning" competent AQM meet a certain demand.

_FailSafe · 2021-12-20T13:35:41.855Z

moeller0:

They do not build their own hardware, and as far as I can tell never claimed to do so.

Not sure if I’m misinterpreting the tone here, which is easy to do when only text is exchanged and no emojis to help. Just so we’re clear, I wasn’t saying what they were doing was a bad thing.

I was simply making a statement. They are selling software on top of re-branded, and well established hardware. I was commending the approach.

FWIW, are they putting OpenWrt on these boxes and selling their “service” as their business model?

moeller0 · 2021-12-20T14:00:01.979Z

_FailSafe:

Not sure if I’m misinterpreting the tone here, which is easy to do when only text is exchanged and no emojis to help. Just so we’re clear, I wasn’t saying what they were doing was a bad thing.

Ah, sorry, I was unclear, I really just wanted to express that I think that they (in spite of the patent) belong to the good guys here. I was unsure about your "well played" comment, and instead of asking for clarification went for a statement. Which was a bad idea, so apologies to you, for "not playing it well" from my side here

_FailSafe:

FWIW, are they putting OpenWrt on these boxes and selling their “service” as their business model?

More or less, except they do not simply deploy default OpenWrt builds, but as far as I understand add quite a lot of their own code and documentation on top (a use-case that OpenWrt invites and is not opposed to IIUC). And as I said before do this not based on periodic maintenance fees but (similar to apple) based on the hardware purchase price. Their founder was and probably still is active in this forum and on the wider bufferbloat mailing lists and is often willing to share information and details that are hard to come by. But enough of my evenroute fan-fiction (disclaimer, I never used their routers so have no first hand experience).

Lynx · 2021-12-20T14:30:36.027Z

(I changed my mind).

dlakelan · 2021-12-20T15:48:20.047Z

Lynx:

Generally speaking it is in the public interest to ensure patents that get granted are not overly broad, and if we know about strong prior art it will help ensure the resulting patent secures a more focused monopoly that is more valid.

I agree, in my opinion it sounds like the dslreports speed test might be prior art. The end user runs a speed test in which some delay probes are sent to the dslreports server and then the dslreports server sends back a graph of the delay which is information you use to optimize the speed setting for sqm

Also I think "a web service that does X" when X itself is not an innovation is so blatantly obvious it's like saying "calling someone on the phone to talk to them about X"

Lynx · 2021-12-20T18:56:15.811Z

Your post on this thread tipped me over the edge.

image500×640 105 KB

dlakelan:

Also I think "a web service that does X" when X itself is not an innovation is so blatantly obvious it's like saying "calling someone on the phone to talk to them about X"

Good point and agreed.

Third party observations:

accounts.google.com

Google Drive: Sign-in

Access Google Drive with a Google account (for personal use) or Google Workspace account (for business use).

Lynx · 2021-12-20T19:55:59.857Z

@_FailSafe my first finding with testing the port form shell is as follows. For some reason my deltas stay high.

root@OpenWrt:~# lua ./sqm-autorate.lua
                      log_time;             rx_load;             tx_load;  min_downlink_delta;    min_uplink_delta;         cur_dl_rate;         cur_ul_rate;
                1640030038.973;                0.02;                0.01;               16.98;               24.00;                9900;                9900;
                         1.485;                0.02;                0.01;               16.96;               28.96;                9801;                9801;
                          2.96;                0.01;                0.01;               16.94;               25.93;                9702;                9702;
                         2.696;                0.01;                0.01;               16.92;               24.92;                9604;                9604;
                         3.209;                0.02;                0.01;               16.91;               24.83;                9507;                9507;
                         3.811;                0.02;                0.01;               16.88;               23.81;                9411;                9411;
                         4.417;                0.01;                0.01;               16.85;               21.79;                9316;                9316;
                          5.29;                0.02;                0.01;               16.84;               22.73;                9222;                9222;
                         5.626;                0.01;                0.01;               16.82;               29.71;                9129;                9129;
                         6.139;                0.02;                0.01;               16.80;               28.70;                9037;                9037;
                         6.656;                0.02;                0.01;               16.78;               28.64;                8946;                8946;
                         7.256;                0.02;                0.01;               16.79;               27.59;                8856;                8856;
                         7.855;                0.02;                0.01;               16.75;               21.54;                8767;                8767;
                         8.368;                0.02;                0.01;               16.73;               22.52;                8679;                8679;
                         8.979;                0.02;                0.01;               16.72;               25.49;                8592;                8592;
                         9.579;                0.01;                0.01;               16.71;               23.44;                8506;                8506;
                        10.187;                0.02;                0.01;               16.66;               22.44;                8420;                8420;
                        10.698;                0.02;                0.01;               16.67;               21.38;                8335;                8335;
                        11.299;                0.02;                0.00;               16.66;               23.35;                8251;                8251;
                        11.809;                0.02;                0.01;               16.61;               22.37;                8168;                8168;
                        12.417;                0.02;                0.01;               16.59;               25.29;                8086;                8086;
                         13.18;                0.02;                0.01;               16.57;               22.23;                8005;                8005;
                        13.617;                0.01;                0.01;               16.54;               21.21;                7924;                7924;
                        14.138;                0.02;                0.01;               16.55;               22.21;                7844;                7844;
^Clua: ./sqm-autorate.lua:538: interrupted!

Any idea why my baseline deltas stay so high, resulting in perpetual rate decrease?

This is with:

https://raw.githubusercontent.com/Fail-Safe/sqm-autorate/port-from-shell/sqm-autorate.lua

anon89577378 · 2021-12-20T21:34:44.460Z

dlakelan:

I agree, in my opinion it sounds like the dslreports speed test might be prior art

Sidebar...

Since the DSLReports speed test is now badly broken, it makes me wonder what they plan to replace it with.

Majority of DSLReports speed test servers offline Installing and Using OpenWrt

This has been the case for a while now... Despite numerous complaints from DSLR members, there is no indication from the moderators, or the site owner, that this will be resolved. [DSLR Test Servers] Waveform is probably the better alternative to recommend...

Their website speed test still uses it.

_FailSafe · 2021-12-20T22:30:14.966Z

anon89577378:

Since the DSLReports speed test is now badly broken, it makes me wonder what they plan to replace it with.

Not to take this thread too far off track, but this is somewhat relevant to the overall theme here...

I have been using these in place of DSLReports as of late:

Waveform

Bufferbloat and Internet Speed Test - Waveform

Is bufferbloat causing issues with your internet connection? Run this test to find out.

speed.cloudflare.com

Internet Speed Test - Measure Network Performance | Cloudflare

Test your Internet connection. Check your network performance with our Internet speed test. Powered by Cloudflare's global edge network.

_FailSafe · 2021-12-20T22:33:35.862Z

Lynx:

Any idea why my baseline deltas stay so high, resulting in perpetual rate decrease?

I'll take a look at it this evening and see what might need adjusting.

anon89577378 · 2021-12-20T22:54:39.473Z

I mentioned Waveform in the DSLReports thread.

Waveform actually uses the Cloudflare speed tests.