New x86 router = higher latency?

Just recently upgraded from and EdgeRouter Lite 3 to a Qotom mini PC with a Core i5-5300U.
Both routers were running OpenWRT 18.06.1 and SQM (Cake/Piece of Cake) with everything the same. However, when I ran the DSLReports speed test I noticed I have 10 - 50 ms higher latency with the newer x86 router.

Any ideas what could cause this?

I still have the ERL so I swapped routers back and forth and the ERL consistently showed lower latency on the speed test site and a few game servers on Steam.

Here are links to tests I ran today

Weird. You could try playing with ethtool -g settings to reduce queues. Also with ethtool -k. There’s definitely a bug somewhere.

@Dr_Fambo, I was under the impression that the EdgeRouters have a special NAT chip (see links below). This could explain the discrepancies. It might be that you have to decide between the speed of hardware NAT and the power of the Qotom. Were you running Openwrt on the ERL or stock?

like @neheb said; try running this for all your eth-devices and see if it helps.
/usr/sbin/ethtool -K ${DEVICE} gro off gso off tso off sg off tx off
(opkg update && opkg install ethtool if you dont have it)

@claviger-pc the erl does not use hardware acceleration if sqm is in use, so the mini-pc should be a lot faster.

@fuller - Thanks!! I did not know that, and am happy to know because I was thinking about employing some type of x86 solution instead of ERL

@fuller - Thanks!! I did not know that, and am happy to know because I was thinking about employing some type of x86 solution instead of ERL

Nah, the i5 has horsepower for NAT and shaping no problemo. I suspect cpufreq power saving stuff

1 Like

Check what frequency governor is in use for power saving. You are probably going to want performance.

1 Like

@neheb @fuller
The ping times are better after using ethtool to disable the offloading but it is still higher than the ERL. The ERL typically idles at 18 ms downloads at 36 ms and uploads at 24 ms
The Mini PC idles 29 ms downloads 41 ms uploads 35
Obviously, I don't get an A+ in bufferbloat with those numbers.

Not sure what you are asking for. In the BIOS I disabled Intel SpeedStep and any option that appeared to reduce power when idle. I can check again but it requires getting approval from the wife and kids.

cat /sys/devices/system/cpu/cpufreq/policy0/scaling_governor

will tell you which governor your CPU is currently using, doesn't require a reboot at all.

I think you should just settle for your current settings if you don't actually have any observed problems. An A+ is a totally arbitrary idea. What you really have is something like 10-15 ms of bufferbloat (increase of latency under load), which is a very acceptable amount. Consider that VOIP packets are sent every 20ms and game packets often similar. Jitter buffers will completely cover up all the latency increase you are observing.

The command returned the following;

Bizarre. Is this using Intel NICs? I would have thought they were decent.

Also try reducing ring buffers with ethtool (-G or -g to 256 or 128).

Intel NIC's
4 x Intel I211-AT- 10 100 1000 Controller

Yes so you are in powersave mode which will idle the cpus more aggressively. You can change it to performance and probably get better results. I'm not sure what the official way to do this is. It can be done in rc.local by echoing to each file.

I have a x86 fanless machine (Celeron J1900). I think it's posible to modify it from the BIOS, I don't know exactly where to find it, but one day I saw something regarding to CPU performance in my BIOS.

@Dr_Fambo my CPU is set to ondemand

I installed irqbalance and edited /etc/rc.local to start it after boot.
I removed the startup entry and I see better results now as well. It seems it is not necessary for routers due to MSI-X being used in the Linux IGB driver for interrupt handling

I also changed the scaling_governor to performance

To rule out any hardware issue try for instance pfsense on a USB stick, your issues in general seems really weird tbh.