Ipq40xx EA6350 v3 as WAP, wifi to Lan latency

Installing and Using OpenWrt
1

I recently started to use EA6350 v3 as a simple wireless access point. It is running 19.07.0-rc2.

When I ping the EA6350 on 2.4 or 5 GHz, the time taken fluctuates from 2-4 ms reported by Windows 10 laptops (Intel 7260 ac, 6300 n, Qualcomm AR5B22 n). If I ping a reliable known website such as www.bbc.co.uk, it returns 10-12ms instead of usual 9ms.

If I toggle back to using Linksys OEM Ver. 3.1.10.191322 firmware, the time taken to ping the EA6350 as a WAP is 1-2 ms, and is 2-4 ms with OpenWrt.

I've checked another EA6350 v3 with same OEM firmware, trying different wifi channel and using 20 MHz bandwidth, and it returns same 1-2ms when pinging EA6350 wap with OEM fw.

My previous WAP was a HH5A with LEDE 17.01.6. Pinging the HH5A consistently returned 1 ms. When pinging www.bbc.co.uk, it typically took 9ms. (8ms with wired ethernet connection btw).

TBH, the extra latency when using EA6350 doesn't cause any particular issues for me, but I was just wondering if any other EA6350 owner has observed the same behaviour?

2

My EA6350v3 is not set up as pure WAP, but closer to the standard configuration AP+router. I have no experience with the standard linksys firmware other than how to flash OpenWrt as soon as possible :wink:

An uncongested 80 MHz 5 GHz channel with good but not great signal strength gives me ~3-7 ms (ath10k driver+FW) and ~ 2-4 ms (ath10k-ct driver+FW) ping to the router. I use very recent master builds, yet that does not seem to change the situation as there are the same ~ 2-4 ms latency as your device with 19.07.0-rc2.

Pinging www.bbc.co.uk seems to be not very useful for me as I get different Amazon AWS hosts under this address. In order to rule out a possible source of your latency differences make sure you always ping the same IP.

For comparison: A Netgear R6220 (mt76 wifi) with the same setup (config and location) as the EA6350 delivered consistenly lower pings with less fluctuation (e.g. game traffic: 16-17 ms vs. 20-25+ ms). As the differences in latency are no big deal for me either, I stopped looking into this too deep for now. Tests at dslreports.com give all A+ results with usually little fluctuation, so it could be e.g. some power saving mechanism in the ath10k HW or the EA6350 that only shows up on sproadic load like ping.

It would be nice to get the EA6350 down to the level of the R6220 as the EA6350 is other than wifi latency (and to a lesser degree signal strength and throughput) the far superior device in my scenario.

I have no idea if this is isolated to EA6350 or if this is common with ipq40xx devices or even perhaps ath10k devices.

3

As a quick test, I configured a spare HH5A with 19.07.0-rc2 as a simple WAP. HH5a uses Qualcomm QCA9880 5 GHz radio and ath10k 'ct' driver. Pinging it returns 1ms. ie. normal.

It does look like issue may be confined to EA6350 v3 and 'may be' other ipq40xx devices.

(One of the reasons why I put OpenWrt onto EA6350v3, is because some of my Dell Windows 10 laptops with older Intel wifi cards some times encounter 'No Internet, Secured' message when they are powered up and try to connect to 5 GHz radio for first time to EA6350v3 with OEM firmware operating in WAP mode. Never had this issue with HH5A wap, and problem has not resurfaced since flashing EA6350v3 with 19.07.0-rc2)

4

I can confirm this on my zyxel nbg6617

1 Like
5

So if the EA6350 seems to have increased latency as well as the NBG6617 it seems to be related to ipq40xx, at least ipq4018 and not just the EA6350.

Are ipq4019 devices also affected?

Can someone using one of these ipq4019 devices please check for this increased latency:

OpenMesh A62
Qualcomm Technologies, Inc. IPQ40xx/AP-DK04.1-C1
Qualcomm Technologies, Inc. IPQ4019/AP-DK04.1
Qxwlan E2600AC c1
Qxwlan E2600AC c2
Qxwlan E2600AC
Linksys EA8300 (Dallas)
AVM FRITZ!Box 7530
AVM FRITZ!Repeater 1200
AVM FRITZ!Repeater 3000
ASUS Lyra MAP-AC2200
Unielec U4019 (32M)
Compex WPJ419
1 Like
6

It turns out @NoTengoBattery is aware of this issue he investigated in October. He kindly referred me to the following web links:

https://forum.openwrt.org/t/optimized-nas-latency-memory-build-for-linksys-ea6350v3-civic/44125/7

And after testing, I found this results when the device uses the "power saving" frequency:

Most recent v0.22 custom build created on 5 December:
https://github.com/NoTengoBattery/openwrt/releases/tag/157e17-v0.22

Minimum CPU frequency set to 448 MHz instead of 48 MHz (OEM firmware sets the CPU at top speed, but that's a bit aggressive)

.....This is an important setting, the governor used by default cannot deal with this because "ping localhost" (or any ping from any client) does not load the CPU enough. This results in ping spikes even with SQM and noticeable with low speed (but low latency) WAN.

As a quick and dirty test based on NoTengoBattery's above findings, I executed these commands on 19.07.0-rc2

# original value: 48000  ie. 48 MHz. New test value 200 MHz
    echo "200000" > /sys/devices/system/cpu/cpu0/cpufreq/scaling_min_freq
    echo "200000" > /sys/devices/system/cpu/cpu1/cpufreq/scaling_min_freq
    echo "200000" > /sys/devices/system/cpu/cpu2/cpufreq/scaling_min_freq
    echo "200000" > /sys/devices/system/cpu/cpu3/cpufreq/scaling_min_freq

(I did not change the /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor setting)

Ping times via wifi to EA6350 v3 reduced to 1-2 ms. Ping time to www.bbc.co.uk reduced to 9-10 ms ie. normal.

If the router has been 'idle' for a while prior to executing 'ping', the ping times start off really bad though before dropping to 1-2 ms using above 'quick fix' as a test. ie. The quick fix is clearly not optimised.

Pinging 192.168.1.248 with 32 bytes of data:
Reply from 192.168.1.248: bytes=32 time=74ms TTL=64
Reply from 192.168.1.248: bytes=32 time=60ms TTL=64
Reply from 192.168.1.248: bytes=32 time=48ms TTL=64
Reply from 192.168.1.248: bytes=32 time=1ms TTL=64
Reply from 192.168.1.248: bytes=32 time=2ms TTL=64
Reply from 192.168.1.248: bytes=32 time=2ms TTL=64
Reply from 192.168.1.248: bytes=32 time=2ms TTL=64
Reply from 192.168.1.248: bytes=32 time=2ms TTL=64

Update: I installed NoTengoBattery's latest v0.22 'sc' (standard clocked) custom build and configured it as a simple WAP. I used the 'factory' image and flashed from Linksys OEM GUI. All seems OK. pinging EA6350 takes 1-2 ms, instead of 2-4 ms. I observe the very first ping may take longer if the router has been 'idle' for a while - which is the same behaviour when using OEM firmware.

Pinging 192.168.1.247 with 32 bytes of data:
Reply from 192.168.1.247: bytes=32 time=10ms TTL=64
Reply from 192.168.1.247: bytes=32 time=2ms TTL=64
Reply from 192.168.1.247: bytes=32 time=2ms TTL=64
Reply from 192.168.1.247: bytes=32 time=1ms TTL=64
Reply from 192.168.1.247: bytes=32 time=2ms TTL=64
1 Like
7

so if you know the way to remedy this problem with ipq40xx, can it be introduced in master or in the next stable releases?
before i put my nbg6617 in the trash?

8

I'm sure the issue can be remedied but it requires a developer with knowledge of ipq40xx devices, to be able to introduce it into master.

In the mean time, if you check out the forum link provided in earlier post, perhaps you could SSH into your nbg6617 and try these settings to force the ipq4018 SoC to run at 710 MHz speed, and observe what happens....?

    echo "performance" > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor 
    echo "performance" > /sys/devices/system/cpu/cpu1/cpufreq/scaling_governor 
    echo "performance" > /sys/devices/system/cpu/cpu2/cpufreq/scaling_governor 
    echo "performance" > /sys/devices/system/cpu/cpu3/cpufreq/scaling_governor 
    echo "710000" > /sys/devices/system/cpu/cpu0/cpufreq/scaling_min_freq
    echo "710000" > /sys/devices/system/cpu/cpu1/cpufreq/scaling_min_freq
    echo "710000" > /sys/devices/system/cpu/cpu2/cpufreq/scaling_min_freq
    echo "710000" > /sys/devices/system/cpu/cpu3/cpufreq/scaling_min_freq

Update: See @shb better suggestion to edit a policy setting two posts down for the scaling_governor setting, which I can confirm works.

To survive a router reboot, adding the above commands to /etc/rc.local script?

9

The 2-4 ms latency are with performance CPU governor. With ondemand (default) pings are worse and especially SQM performance suffers. So at least for my EA6350 the problem is not a low CPU frequency.

I switched to the performance governor after measuring idle power usage compared to ondemand and there was no real difference (below 100 mW).

10

echo performance > /sys/devices/system/cpu/cpufreq/policy0/scaling_governor sets the governor for all CPUs. After that there should be no reason to set the scaling_min_freq as the perfomance governor simply sets the CPUs to max freqency, effectively disabling frequency scaling.

Update: Setting the performance governor permanently should be done with careful consideration. It is probably better to stay with ondemand if there is constant/heavy load on the system, or if the environment is warm.

1 Like
11

just extracted the zyxel nbg6617 firmware and found a powerctl file inside etc/init.d

#!/bin/sh /etc/rc.common

START=99

ipq40xx_power_auto() {
	# change scaling governor as ondemand to enable clock scaling based on system load
	echo "ondemand" > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

	# set scaling min freq as 200 MHz
	echo "200000" > /sys/devices/system/cpu/cpu0/cpufreq/scaling_min_freq

	# Change sampling rate for frequency scaling decisions to 1s, from 10 ms
	echo "1000000" > /sys/devices/system/cpu/cpufreq/ondemand/sampling_rate

	# Change sampling rate for frequency down scaling decision to 10s
	echo 10 > /sys/devices/system/cpu/cpufreq/ondemand/sampling_down_factor

	# Change the CPU load threshold above which frequency is up-scaled to
	# turbo frequency,to 50%
	echo 50 > /sys/devices/system/cpu/cpufreq/ondemand/up_threshold
}

start() {
	ipq40xx_power_auto
}

putting this on my router solves the pingproblem.

12

Can anyone explain what is the importance of 1-2 ms ping difference?

Now in 17.01 the default settings are:

root@OpenWrt:~# cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
ondemand
root@OpenWrt:~# cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_min_freq
48000
root@OpenWrt:~# cat /sys/devices/system/cpu/cpufreq/ondemand/sampling_rate
20000
root@OpenWrt:~# cat /sys/devices/system/cpu/cpufreq/ondemand/sampling_down_factor
1
root@OpenWrt:~# cat /sys/devices/system/cpu/cpufreq/ondemand/up_threshold
80

wlan to the router ping is ~4ms

With @mezo settings the ping is ~3ms.

But how the ping times affect anything in real life?

13

Added latency in the region of ~4 ms over wifi is not that big of a deal and certainly "good enough", yet I guess quite a few people use OpenWrt to not only get "good enough" performance out of their devices.

It is a little sad to see hardware clearly capable of lower latency underperforming while running OpenWrt. I got ath10k hardware explcitly because it is quite well supported and development/research regarding better wifi performance like the recent Airtime-based Queue Limit (AQL) is often initially done on/for this kind of hardware.

These few ms of latency are added to all of your traffic over wifi. Luckily it's not that much but if you e.g. setup and optimise your SQM, you probably don't want to add this unnecessary jitter.

Not the end of the world at all, yet it would be nice to get the same (or even better) performance of the OEM firmware and other ath10k based devices not having this added latency.

14

In my case I see only 1 ms added. But I didn't test on a fully idle device. That may be the reason.

That doesn't seem correct. If the problem is in the governor on an idle device, then on real traffic that won't add anything.

15

Fair point! It seems to affect sporadic traffic like surfing, ssh, games, etc. . If there is a constant load like a download, the latency drops a little for me too. Power saving of some component could be at play...

The governor does play into it, e.g. the ondemand (default) governor takes a little while to ramp up the frequency and only does so if there is enough CPU load. To eliminate this initial ramp-up latency I set the governor to performance so a change in CPU frequency cannot have any impact on performance in my case. With ondemand the latency is worse in my case.

The latency increase I have is in "real" traffic, not only while pinging the router. If I didn't have the direct comparison with the R6220, I would not even know about this and would attribute it to some wifi property.

16

It maybe only games that really may be affected. I can't imagine a case where ssh or surfing latency +2-4ms may matter.

17

It doesn't :wink:

I'd just like to get the most out of the device, it is not about it being usable or unusable.

Look at the work done within the Make Wi-Fi Fast project... they fight for every ms!

18

Joining a bit the discussion:
Let's be clear: the clock frequency is an important factor but it's not the actual solution. More fine tuning is needed, and some of it is already included in my custom build from things I've learnt from the OEM firmware.
That makes a difference, as noted by @bill888. It's a clear difference even between the rc2 with the clock a max speed. Source code of my work is available for anyone to test it but...

It will never ever reach OpenWrt upstream. See, they are really "conservative" and the latency fixed done by Linksys and tuned by me can be considered "too cutting edge". I have no problems with it, and I know what I do: I came from XDA-Developers so I understand how to tune an embedded Linux system. Still, they won't accept any of these changes ever. It's a shame to close the door to improvements in the sake of "stability".

Among other things, my custom build includes:

  1. Overclocking (optional)
  2. A brand new clock frequency scaling
  3. Fully preemptive, tickless @ 500Hz kernel
  4. Removed the double tagging of the WAN/LAN switch
  5. System daemons are actually fine tuned using a tool which is only available for desktops (chrt), boosting system responsiveness under load (read SQM/OpenVPN)
  6. A brand new subsystem for compressed memory designed and fine tuned by me to avoid memory starvation under pressure (read Samba/CIFS)

All of them works and help, but none of them will reach upstream ever. Give my snapshots a try and then you can grab some modifications to build your own image!

19

The brand new scheduler "schedutil" has proven to be superior in every aspect compared to the "ondemand". In fact, the Linux people want to replace all governors with "schedutil" some day. The problem: it requires a modification to the kernel in order to enable it. OpenWrt will not ever think of getting it upstream.

20

No positive changes with this so far:

  • ondemand settings from nbg6617 firmware as pointed out by @mezo
  • ath10k-ct driver with ath10k firmware (newest 10.4-3.5.3-00078 as well as 10.4-3.6-00140)
  • ath10k-ct-smallbuffers
  • switch to a CONFIG_PREEMPT=y kernel
  • fixing sleep_clk from 32768 to 32000 and setting this 32 kHz clock for e.g. wcss5g_rtc_clk
  • looking for something else in the kernel patches and config in the Linksys GPL code
  • irqbalance on/off