Ping response time through router - x86 vs mvebu

wally_walrus · October 31, 2022, 2:27am

For the last few days, I kept swapping between an x86 (HP T740 with a quad i350 GbE) and a mvebu-based WRT1900AC v2. Same OpenWRT version on both (22.03.02), pretty much the same packages (including SQM), same network configuration (both wired only). My internet connection is vDSL 50Mb/s, so both should be flying. However I noticed when I ping speedtest.net via the x86 machine I get approx. 8.5 ms response time, while the mvebu router gets me 16ms so roughly double. This is from a Linux PC, and I haven't tried other sites (will do that next), but speedtest.net is consistent in this regard.

Now I wouldn't complain if I didn't read around here that mvebu drivers should be highly optimized for Ethernet routing, so I was expecting a much smaller gap... Would anyone care to take a crack at explaining why? Am I missing anything?

TIA

lleachii · October 31, 2022, 12:57pm

You're just missing it's an x86_64 you previously used. I had the same behavior when switching from a full computer to an embedded device (double response time).

But...did the speed go down?

moeller0 · October 31, 2022, 1:36pm

Mmmh.

On my mvebu/cortex a9 turris omnia (TOS 6 is based on OpenWrt 21) I get:

BusyBox v1.33.2 (2022-10-19 16:21:10 UTC) built-in shell (ash)

      ______                _         ____  _____
     /_  __/_  ____________(_)____   / __ \/ ___/
      / / / / / / ___/ ___/ / ___/  / / / /\__ 
     / / / /_/ / /  / /  / (__  )  / /_/ /___/ / 
    /_/  \__,_/_/  /_/  /_/____/   \____//____/  
                                             
 -----------------------------------------------------
 TurrisOS 6.0.1, Turris Omnia
 -----------------------------------------------------
Doh...
root@turris:~# ping -c 10 speedtest.net
PING speedtest.net(2a04:4e42:400::731 (2a04:4e42:400::731)) 56 data bytes
64 bytes from 2a04:4e42:400::731 (2a04:4e42:400::731): icmp_seq=1 ttl=61 time=15.7 ms
64 bytes from 2a04:4e42:400::731 (2a04:4e42:400::731): icmp_seq=2 ttl=61 time=16.0 ms
64 bytes from 2a04:4e42:400::731 (2a04:4e42:400::731): icmp_seq=3 ttl=61 time=15.9 ms
64 bytes from 2a04:4e42:400::731 (2a04:4e42:400::731): icmp_seq=4 ttl=61 time=15.7 ms
64 bytes from 2a04:4e42:400::731 (2a04:4e42:400::731): icmp_seq=5 ttl=61 time=15.8 ms
64 bytes from 2a04:4e42:400::731 (2a04:4e42:400::731): icmp_seq=6 ttl=61 time=15.9 ms
64 bytes from 2a04:4e42:400::731 (2a04:4e42:400::731): icmp_seq=7 ttl=61 time=15.8 ms
64 bytes from 2a04:4e42:400::731 (2a04:4e42:400::731): icmp_seq=8 ttl=61 time=16.0 ms
64 bytes from 2a04:4e42:400::731 (2a04:4e42:400::731): icmp_seq=9 ttl=61 time=16.0 ms
64 bytes from 2a04:4e42:400::731 (2a04:4e42:400::731): icmp_seq=10 ttl=61 time=16.0 ms

--- speedtest.net ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9006ms
rtt min/avg/max/mdev = 15.717/15.877/16.028/0.108 ms


root@turris:~# ping -4 -c 10 speedtest.net
PING speedtest.net (151.101.194.219) 56(84) bytes of data.
64 bytes from 151.101.194.219 (151.101.194.219): icmp_seq=1 ttl=57 time=15.9 ms
64 bytes from 151.101.194.219 (151.101.194.219): icmp_seq=2 ttl=57 time=15.6 ms
64 bytes from 151.101.194.219 (151.101.194.219): icmp_seq=3 ttl=57 time=16.0 ms
64 bytes from 151.101.194.219 (151.101.194.219): icmp_seq=4 ttl=57 time=15.7 ms
64 bytes from 151.101.194.219 (151.101.194.219): icmp_seq=5 ttl=57 time=15.9 ms
64 bytes from 151.101.194.219 (151.101.194.219): icmp_seq=6 ttl=57 time=15.9 ms
64 bytes from 151.101.194.219 (151.101.194.219): icmp_seq=7 ttl=57 time=15.7 ms
64 bytes from 151.101.194.219 (151.101.194.219): icmp_seq=8 ttl=57 time=15.7 ms
64 bytes from 151.101.194.219 (151.101.194.219): icmp_seq=9 ttl=57 time=15.7 ms
64 bytes from 151.101.194.219 (151.101.194.219): icmp_seq=10 ttl=57 time=15.7 ms

--- speedtest.net ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9011ms
rtt min/avg/max/mdev = 15.598/15.792/16.046/0.137 ms
root@turris:~# cat /etc/os-release 
NAME="TurrisOS"
VERSION="6.0.1"
ID="turrisos"
ID_LIKE="lede openwrt"
PRETTY_NAME="TurrisOS 6.0.1"
VERSION_ID="6.0.1"
HOME_URL="https://www.turris.cz/"
BUG_URL="https://gitlab.nic.cz/groups/turris/-/issues/"
SUPPORT_URL="https://www.turris.cz/support/"
BUILD_ID="r16691+128-060aa0021f"
OPENWRT_BOARD="mvebu/cortexa9"
OPENWRT_ARCH="arm_cortex-a9_vfpv3-d16"
OPENWRT_TAINTS="busybox"
OPENWRT_DEVICE_MANUFACTURER="CZ.NIC"
OPENWRT_DEVICE_MANUFACTURER_URL="https://www.turris.cz/"
OPENWRT_DEVICE_PRODUCT="Turris Omnia"
OPENWRT_DEVICE_REVISION="v0"
OPENWRT_RELEASE="TurrisOS 6.0.1 060aa0021f688d455767b553cfe95e29b6559b7a"
root@turris:~#

On my macbook pro (macos monterey, x86, quad core i7 4870HQ haswell) connected via USB3 gigabit ethernet I get:

macbook-pro:CAKE-autorate user $  ping6 -c 10 speedtest.net
PING6(56=40+8+8 bytes) 2a01:c22:8d51:1c00:854c:4102:7cae:de6f --> 2a04:4e42:600::731
16 bytes from 2a04:4e42:600::731, icmp_seq=0 hlim=60 time=16.410 ms
16 bytes from 2a04:4e42:600::731, icmp_seq=1 hlim=60 time=17.459 ms
16 bytes from 2a04:4e42:600::731, icmp_seq=2 hlim=60 time=16.994 ms
16 bytes from 2a04:4e42:600::731, icmp_seq=3 hlim=60 time=16.643 ms
16 bytes from 2a04:4e42:600::731, icmp_seq=4 hlim=60 time=16.742 ms
16 bytes from 2a04:4e42:600::731, icmp_seq=5 hlim=60 time=16.817 ms
16 bytes from 2a04:4e42:600::731, icmp_seq=6 hlim=60 time=17.184 ms
16 bytes from 2a04:4e42:600::731, icmp_seq=7 hlim=60 time=17.017 ms
16 bytes from 2a04:4e42:600::731, icmp_seq=8 hlim=60 time=17.111 ms
16 bytes from 2a04:4e42:600::731, icmp_seq=9 hlim=60 time=16.397 ms

--- speedtest.net ping6 statistics ---
10 packets transmitted, 10 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 16.397/16.877/17.459/0.323 ms


macbook-pro:CAKE-autorate user $  ping -c 10 speedtest.net
PING speedtest.net (151.101.194.219): 56 data bytes
64 bytes from 151.101.194.219: icmp_seq=0 ttl=56 time=16.506 ms
64 bytes from 151.101.194.219: icmp_seq=1 ttl=56 time=16.475 ms
64 bytes from 151.101.194.219: icmp_seq=2 ttl=56 time=17.029 ms
64 bytes from 151.101.194.219: icmp_seq=3 ttl=56 time=16.511 ms
64 bytes from 151.101.194.219: icmp_seq=4 ttl=56 time=16.890 ms
64 bytes from 151.101.194.219: icmp_seq=5 ttl=56 time=16.460 ms
64 bytes from 151.101.194.219: icmp_seq=6 ttl=56 time=16.910 ms
64 bytes from 151.101.194.219: icmp_seq=7 ttl=56 time=16.752 ms
64 bytes from 151.101.194.219: icmp_seq=8 ttl=56 time=17.078 ms
64 bytes from 151.101.194.219: icmp_seq=9 ttl=56 time=16.922 ms

--- speedtest.net ping statistics ---
10 packets transmitted, 10 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 16.460/16.753/17.078/0.232 ms
macbook-pro:CAKE-autorate user$

On my ubuntu 22.04.1 LTS (x86, quad core xeon E3-1230 V2) I get:

user@work-horse:~$ ping -6 -c 10 speedtest.net
PING speedtest.net(2a04:4e42:400::731) 56 data bytes
64 bytes from 2a04:4e42:400::731: icmp_seq=1 ttl=60 time=16.0 ms
64 bytes from 2a04:4e42:400::731: icmp_seq=2 ttl=60 time=16.0 ms
64 bytes from 2a04:4e42:400::731: icmp_seq=3 ttl=60 time=15.9 ms
64 bytes from 2a04:4e42:400::731: icmp_seq=4 ttl=60 time=16.2 ms
64 bytes from 2a04:4e42:400::731: icmp_seq=5 ttl=60 time=16.0 ms
64 bytes from 2a04:4e42:400::731: icmp_seq=6 ttl=60 time=16.1 ms
64 bytes from 2a04:4e42:400::731: icmp_seq=7 ttl=60 time=15.7 ms
64 bytes from 2a04:4e42:400::731: icmp_seq=8 ttl=60 time=15.7 ms
64 bytes from 2a04:4e42:400::731: icmp_seq=9 ttl=60 time=16.0 ms
64 bytes from 2a04:4e42:400::731: icmp_seq=10 ttl=60 time=15.9 ms

--- speedtest.net ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 90237ms
rtt min/avg/max/mdev = 15.658/15.956/16.174/0.151 ms


user@work-horse:~$ ping -4 -c 10 speedtest.net
PING  (151.101.66.219) 56(84) bytes of data.
64 bytes from 151.101.66.219: icmp_seq=1 ttl=56 time=15.9 ms
64 bytes from 151.101.66.219: icmp_seq=2 ttl=56 time=16.0 ms
64 bytes from 151.101.66.219: icmp_seq=3 ttl=56 time=16.3 ms
64 bytes from 151.101.66.219: icmp_seq=4 ttl=56 time=15.9 ms
64 bytes from 151.101.66.219: icmp_seq=5 ttl=56 time=16.2 ms
64 bytes from 151.101.66.219: icmp_seq=6 ttl=56 time=16.1 ms
64 bytes from 151.101.66.219: icmp_seq=7 ttl=56 time=16.0 ms
64 bytes from 151.101.66.219: icmp_seq=8 ttl=56 time=16.2 ms
64 bytes from 151.101.66.219: icmp_seq=9 ttl=56 time=15.9 ms
64 bytes from 151.101.66.219: icmp_seq=10 ttl=56 time=16.2 ms

---  ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 90237ms
rtt min/avg/max/mdev = 15.905/16.073/16.260/0.130 ms
user@work-horse:~$

As you might see ping from the router is a tad faster, but not more than expected... no factor of 2 differences. This was on a VDSL2 100/40 link shaped with sqm-scripts and cake-autorate during a quite period.

lleachii · October 31, 2022, 4:10pm

Wow, yes if OP is getting 2X response time on pings such as that - yea there's some issue.

Wow you have alot of routers in your network!

moeller0 · October 31, 2022, 4:21pm

I would guess the three OS simply have different defaults for TTL? Because as far as I know there is a single router in my home network

The msain difference though is for each OS between IPv6 and IPv4, which just shows different routes taken for the two protocols IMHO.

spence · October 31, 2022, 5:48pm

If you are using ICMP Pings as a measure of packet forwarding performance for all protocols it may be useful to also do RTT (round trip time) measurements for TCP and UDP. In enterprise networking situations, I've looked into suspected network problems where ICMP was slow and had many drops but testing TCP rtt and UDP rtt showed they were fast and reliable. Some servers under test had slightly slow TCP session starts but the data flow after the session start was very fast. It can get interesting.

wally_walrus · November 1, 2022, 2:30am

Apologies everyone for a late reply... Just wanted to clarify that I was referring to pinging speedtest.net through (and NOT from) 2 different routers - one x86_64 and one mvebu.

However I might have jumped the gun, as it appears even only using the x86_64 router gets me EITHER 8ms or 16ms. Here's 2 runs one right after another:

alpine:~# ping speedtest.net
PING speedtest.net (2a04:4e42:200::731): 56 data bytes
64 bytes from 2a04:4e42:200::731: seq=0 ttl=60 time=8.344 ms
64 bytes from 2a04:4e42:200::731: seq=1 ttl=60 time=8.942 ms
64 bytes from 2a04:4e42:200::731: seq=2 ttl=60 time=8.528 ms
64 bytes from 2a04:4e42:200::731: seq=3 ttl=60 time=9.099 ms
64 bytes from 2a04:4e42:200::731: seq=4 ttl=60 time=10.093 ms
64 bytes from 2a04:4e42:200::731: seq=5 ttl=60 time=8.483 ms
64 bytes from 2a04:4e42:200::731: seq=6 ttl=60 time=9.318 ms
64 bytes from 2a04:4e42:200::731: seq=7 ttl=60 time=8.542 ms
64 bytes from 2a04:4e42:200::731: seq=8 ttl=60 time=8.359 ms
64 bytes from 2a04:4e42:200::731: seq=9 ttl=60 time=9.329 ms
64 bytes from 2a04:4e42:200::731: seq=10 ttl=60 time=8.772 ms
^C
--- speedtest.net ping statistics ---
11 packets transmitted, 11 packets received, 0% packet loss
round-trip min/avg/max = 8.344/8.891/10.093 ms
alpine:~# ping speedtest.net
PING speedtest.net (2a04:4e42:600::731): 56 data bytes
64 bytes from 2a04:4e42:600::731: seq=0 ttl=59 time=16.319 ms
64 bytes from 2a04:4e42:600::731: seq=1 ttl=59 time=15.232 ms
64 bytes from 2a04:4e42:600::731: seq=2 ttl=59 time=16.993 ms
64 bytes from 2a04:4e42:600::731: seq=3 ttl=59 time=16.249 ms
64 bytes from 2a04:4e42:600::731: seq=4 ttl=59 time=15.822 ms
64 bytes from 2a04:4e42:600::731: seq=5 ttl=59 time=16.097 ms
64 bytes from 2a04:4e42:600::731: seq=6 ttl=59 time=16.406 ms
64 bytes from 2a04:4e42:600::731: seq=7 ttl=59 time=16.027 ms
64 bytes from 2a04:4e42:600::731: seq=8 ttl=59 time=16.159 ms
64 bytes from 2a04:4e42:600::731: seq=9 ttl=59 time=16.421 ms
^C
--- speedtest.net ping statistics ---
10 packets transmitted, 10 packets received, 0% packet loss
round-trip min/avg/max = 15.232/16.172/16.993 ms
alpine:~#

So it looks like I'm pinging different IPv6 addresses within the speedtest.net CDN which explains the difference. Sorry about the confusion everyone

moeller0 · November 1, 2022, 7:24am

You could install mtr and then run:

mtr -ezb6 2a04:4e42:600::731

and

mtr -ezb6 2a04:4e42:200::731

to see if/how the routing differs (keep in mind though that you only see the forward path, even if the mtr results look equal there still might be differences in the return paths.)