Possible cause of R7800 latency issues

[fantom-x]

@quarky

Probably the interrupt handlers is not handling interrupts efficiently?

Is there a way to verify/prove this theory?

[huaracheguarache]

Another strange fact worth noting is that @slh doesn't experience these spikes on his ZyXEL NBG6817, even though it also has the IPQ8065 and QCA9984.

[quarky]

@fantom-x

Is there a way to verify/prove this theory?

Well, I can’t think of how to test for interrupt issues effectively. Probably have to review the kernel codes. Maybe we can turn off all peripheral components like WiFi, UART, LEDs etc and just test the LAN ports?

[fantom-x]

@quarky

Maybe we can turn off all peripheral components like WiFi, UART, LEDs etc and just test the LAN ports?

I know how to turn off WiFi, but not the others. Can you provide some pointers?

[quarky]

Have to compile a custom firmware with all those device drivers removed. Then we’ll know for sure those are not interfering since it’ll not register the interrupts.

[lleachii]

Did you see this?

@lleachii

I’ve watched this thread for quite some time, and I’m surprised it’s flowed this far.

• Your “baseline” was connecting a PC directly to your DSL modem, versus through your R7800. This will obviously add latency.
• Another note, any software you run on the router to test - will add latency.

[huaracheguarache]

@lleachii

Did you see this?

I’ve watched this thread for quite some time, and I’m surprised it’s flowed this far.

Your “baseline” was connecting a PC directly to your DSL modem, versus through your R7800. This will obviously add latency.
Another note, any software you run on the router to test - will add latency.

Yes, but a ping test run locally on my computer also shows substantial spikes as seen in the plot with interval = 0.2 seconds.

[lleachii]

@huaracheguarache

Yes, but a ping test run locally on my computer also shows substantial spikes as seen in the plot with interval = 0.2 seconds.

Yes, I know. You also stated this:

@huaracheguarache

However, increasing the ping frequency to every 0.2 seconds gave a different picture:

@huaracheguarache

So, in trying to figure out what might cause this I had a look at what htop -d1 showed. htop -d1 updates every 1/10 of a second and I consistently spotted a kworker thread hogging 30-85 percent of core 2 approximately every 2 seconds.

So...what do you think happens to a CPU, when you multiply the given load by a factor of 5?

[huaracheguarache]

I'm not running htop -d1 while doing the ping tests - I just used it to have a look if something is hogging the CPU periodically. I assume you implied that I ran it during the pings. Sorry if I misunderstood you.

[lleachii]

But you are increasing your pings to a public DNS server though?

Have you tested something more suited to responding in a timely fashion?

[huaracheguarache]

@lleachii

But you are increasing your pings to a public DNS server though?

Since the kworker hogging the CPU appeared in short bursts I thought it was a good idea to increase the frequency of the pings to make it more likely for a ping to coincide with a CPU utilisation spike. That way any initially hard-to-spot latency issues might become more apparent.

@lleachii

Have you tested something more suited to responding in a timely fashion?

No, but I'm open to suggestions

[lleachii]

A Speedtest...perhaps?

DNS servers are configured optimize UDP/53...not ICMP Echo-Request.

[fantom-x]

@lleachii

Another note, any software you run on the router to test - will add latency.

Adding some predicable and stable overhead is understood and this router does indeed add ~1ms or so. It is the seemingly random and sizeable spikes that is a concern.

[fantom-x]

@lleachii

A Speedtest…perhaps?

A web based speed test does not run pings long enough to cache this issue. Are your suggesting pinging one of their servers?

[lleachii]

Why are you under the impression that ping can clearly provide viable results to measure your issue?

My suggestion is to use some Layer 7 test, particularly on a server designed for it. The best suggestion would be to have a server plugged into the WAN port solely for this test...but in any case...

Google Public DNS was designed and optimized for udp/53, not ICMP Echo-Request. They could be limiting your pings...delaying them, etc...

http://netalyzr.icsi.berkeley.edu/

That's all.

[fantom-x]

@lleachii

Google Public DNS was designed and optimized for udp/53, not ICMP Echo-Request. They could be limiting your pings…delaying them, etc…

This is true, but the same exact test with the same router running stock or an Archer C7 running the same master LEDE build introduce no spikes and almost unnoticeable jitter.

[anon57995562]

Take a look @richb-hanover's betterspeedtest ...

richb-hanover/CeroWrtScripts

[richb-hanover]

A few things to think about:

• The intermittent latency spikes every few minutes might be Wi-Fi Channel Scans At least you can try to rule it out...

• The charts shown in that blog posting are created by flent (Flent.org) It's a great way to make repeatable latency/traffic measurements.

• I've never heard anyone say that Google limits/delays pings. I always ping gstatic.com or ping google.com with pretty uniform results.

• Yes, betterspeedtest.sh is sort of like the DSLReports Speedtest except no GUI and run from the command line. It has parameters for how long to run, what device to ping and the number of simultaneous sessions (among others), then summarizes the latencies into min, 10th percentile, average, median, 90th percentile, and max.

[huaracheguarache]

Ok, as a baseline I've plotted the ping I get from pinging 8.8.8.8 directly connected to the modem:

The ping never goes above 15 ms and is very stable, apart from minor isolated spikes. With this in mind, I think it's reasonable to use 8.8.8.8 as a target.

Here's a plot from pinging 8.8.8.8 from my computer connected to the R7800 via ethernet:

The issue is definitely with the R7800.

Here's also a plot with both wifi interfaces disabled:

Pretty much the same story.

[lleachii]

My friend...you are in the tens of milliseconds...I stand corrected...

I'm sure you know your R7800 more than I...

What other devices are similar in:

• CPU
• Memory; and
• Switch chip (if any)?