Speedtest: new package to measure network performance

Is this going to be added to the official repo any time soon? I'm on 18.06.1 and can't find it. Also when I add your feed to the custom feed list it fails (others work just fine):

Downloading https://raw.github.com/guidosarducci/papal-repo/master/Packages.gz
Updated list of available packages in /var/opkg-lists/papal_repo
Downloading https://raw.github.com/guidosarducci/papal-repo/master/Packages.sig
Signature check failed.
Remove wrong Signature file.

Oh the new raw link is here:

https://github.com/guidosarducci/papal-repo/blob/master/speedtest_0.9-7_all.ipk?raw=true

it would be awesome if we could get an accompaning luci interface for this! thanks so much great program

Sorry I missed your post -- looks like you found the details in the top post for directly downloading the package in any case. :slight_smile: I regularly use the repo custom feed so it definitely works. The "Signature check failed" message suggests there was a problem importing the repo public key. Try double checking my repo instructions for Online Install. And check if one of your keys in /etc/opkg/keys matches papal-repo.pub from the Github repo.

Yes, but there are still a couple of small updates I'm looking into before making a PR (and other PRs in queue). Since there's a direct package link as well as a custom feed, hopefully everyone who wants can get the package in the meantime.

Heh heh, that was one of the things I'm thinking about...

3 Likes

You might do a PR and get this accepted as a regular package to OpenWrt packages repo. Much easier to just download it from the main package repo.

I think that the package is good enough already :wink:

Download speed reported is inaccurate.
I have Gigabit ISP connection.
Upload speeds are closer to actual.
Any comments why?
My hardware is PC-Engine APU2. Running 18.06.

./speedtest.sh -H netperf-west.bufferbloat.net -p 1.1.1.1 --sequential
2019-02-16 22:17:51 Starting speedtest for 60 seconds per transfer session.
Measure speed to netperf-west.bufferbloat.net (IPv4) while pinging 1.1.1.1.
Download and upload sessions are sequential, each with 5 simultaneous streams.
.............................................................
Download: 82.55 Mbps
Latency: [in msec, 61 pings, 0.00% packet loss]
Min: 15.361
10pct: 15.410
Median: 15.569
Avg: 15.689
90pct: 15.922
Max: 17.397
CPU Load: [in % busy (avg +/- std dev) @ avg frequency, 52 samples]
cpu0: 2.9% +/- 1.9% @ 698 MHz
cpu1: 9.8% +/- 2.8% @ 698 MHz
cpu2: 10.2% +/- 3.8% @ 704 MHz
cpu3: 13.3% +/- 4.4% @ 749 MHz
Overhead: [in % used of total CPU available]
netperf: 3.3%
.............................................................
Upload: 526.89 Mbps
Latency: [in msec, 61 pings, 0.00% packet loss]
Min: 15.434
10pct: 15.558
Median: 15.852
Avg: 16.245
90pct: 17.257
Max: 22.723
CPU Load: [in % busy (avg +/- std dev) @ avg frequency, 52 samples]
cpu0: 4.3% +/- 2.2% @ 722 MHz
cpu1: 11.8% +/- 2.9% @ 748 MHz
cpu2: 19.1% +/- 4.1% @ 732 MHz
cpu3: 20.1% +/- 4.3% @ 762 MHz
Overhead: [in % used of total CPU available]

This looks fantastic , Am keen to try it out when it hits the main repo.

(Some time down the road for next major release. No urgency whatsoever :wink:). On the discussion around avoiding skewing the numbers / overloading the cpu, one possible way (Non-Trivial) might be to create a similar interface to what Speedtest uses, where the client downloads the script into their browser , it executed, then it pushes the results back to the http server by the browser. Then the server side adds this datapoint into all the rest it has and displays the result from there back to the client.. this would allow possibility of some cool stuff like grouping speed test numbers by connected device /mac , WiFi channel Id, and time of day. These things obviously are outside the true benchmarking problem, but are nonetheless still relevant for real world usage. I’ve only dabbled in this kind of thing for small tasks/mini projects at work , but I’d be interested in helping anyone that wants to explore something like this. In my view , having both metrics would the the best, on-router and on-client , with all the data able to be displayed together in a neat and pretty luci interface.

Looks like I am not CPU-bound so why the results are inaccurate?
Only @egross seems to have hit the right values.
Wonder what I am doing different?
Is it about distance from servers?

In a Speedtests like this you basically determine the minimum of the available bandwidth and CPU cycles of the remote end, bandwidth and CPU cycles of the local end, and the available bandwidth along the network path the test packets are traveling over. Unfortunately, the only one you could rule out is CPU cycles of the local end...
In reality the Backend of the buffer lost testing network, payed for by volunteers out of their own pocket, simply is not prepared for reliably testing gigabit links. Partly this is because IMHO given that servers often are only connected at 1gbps themselves, one would need to only allow a maximum number of concurrent Speedtests to make sure the server's bandwidth is not the bottleneck. Something that for example ookla's speedtest.net also struggles with. I also note that Speedtest.net also is not the most reliable for fast links (but at least there you can try to find server locations with a good reputation for fast links, but I digress)

2 Likes

That's a fair point. I suspect remote server being bottleneck as well, at least some times.
To rule that out I have setup a dedicated server with 1 Gbps and good cpu. On this server I am always getting upload speed 800 Mbps+. So I believe I should see at least that much when I am running speedtest from the router.
I shall post my results soon using this server.
However I should mention that, so far running simultaneous curl downloads large file from this server hasn't given promising results. Hopefully netperf should provide a better result.

1 Like

Results with the dedicated pvt server.
While download speed result improved , upload speed result dropped significantly; compared to netperf-west.bufferbloat.net.
PS: Same (pvt server) server reports 800+ Mbps when tested with speedtest.net in browser.
Results are consistent across tests.

Download and upload sessions are sequential, each with 5 simultaneous streams.
............................................................
Download: 329.24 Mbps
Latency: [in msec, 61 pings, 0.00% packet loss]
Min: 15.352
10pct: 15.417
Median: 15.530
Avg: 15.559
90pct: 15.721
Max: 16.127
CPU Load: [in % busy (avg +/- std dev) @ avg frequency, 51 samples]
cpu0: 9.1% +/- 2.8% @ 759 MHz
cpu1: 29.0% +/- 6.8% @ 793 MHz
cpu2: 24.9% +/- 5.1% @ 779 MHz
cpu3: 33.1% +/- 6.7% @ 811 MHz
Overhead: [in % used of total CPU available]
netperf: 9.3%
.............................................................
Upload: 129.49 Mbps
Latency: [in msec, 61 pings, 0.00% packet loss]
Min: 15.177
10pct: 15.224
Median: 15.366
Avg: 15.377
90pct: 15.518
Max: 15.806
CPU Load: [in % busy (avg +/- std dev) @ avg frequency, 52 samples]
cpu0: 3.7% +/- 2.1% @ 700 MHz
cpu1: 13.9% +/- 5.0% @ 734 MHz
cpu2: 3.3% +/- 2.0% @ 701 MHz
cpu3: 19.4% +/- 4.5% @ 754 MHz
Overhead: [in % used of total CPU available]
netperf: 1.4%

=====================

Update:
With 8 simultaneous streams I was able to hit:
Download: 577.23 Mbps
Upload: 230.73 Mbps

Results are now getting closer to actual. Thanks for your support, folks!

I am using MWAN3 and I wonder if there is a way to run this speedtest script for each wan interface.
I found there is -L option in netperf to specify local IP.
However looking at the results I can see that it's not really using the specified local interface. It uses primary wan link irrespective of what I provide in -L option.
Is a higher layer rule (metric, default route etc) over-riding this option?
Any direction appreciated.

The script doesn't define the route packets take to get to the netperf servers; that's something you need to define as part of your system configuration. It's no different than running wget and expecting it to control routing; it's not meant to.

You haven't really explained what you're trying to accomplish or how you've configured things. IIRC, MWAN3 (or another policy routing application like vpn-policy-routing) should let you configure routing based on destination. So, for example, forcing all traffic destined for netperf-east.bufferbloat.net to go through wan0 rather wan1 should be possible.

Since you're using MWAN3, why not configure an appropriate rule there? Or even simpler, how about shutting down one wan interface to force the other to be used?

That leads to network disruption for a minute while mwan3 failover happens, which I want to avoid.
My config is simple, 2 WANs, using mwan3 as fail-over. One active, other idle/standby.
Thanks for the pointers, I will check if mwan3 has interface selection option based on destination.
Also, I was hoping the netperf -L option would work, which didn't, probably mwan3 interfered here.
Wasn't expecting the speedtest script to do much on this.

==========
update:

I was able to set user rule in mwan3 and do "mwan3 restart" to take effect.
This way I could select wan links and run speedtest for that link.

config rule 'speedtest'
list comment 'to speedtest server'
option src_ip '0.0.0.0/0'
option dest_ip 'xx.xx.xx.xx/32'
option use_policy 'wan_only'

==============
update:
Even better, added a new rule for speedtest server using "ip rule add ..."
Doesn't require mwan restart.. so no network disruption at all.

Hi all, some news about the package:

I've folded in some final changes I've been trialing:

  • Updated the package name to clarify the back-end, highlight the ability to use any netperf server, and distinguish from other speedtest variants. Users should use the new speedtest-netperf_1.0.0-1_all.ipk package after uninstalling the old one. The new package will remind you of any conflict with the old one during installation.
  • Improved consistency and clarity of results and units.
  • Check for presence of netperf if running the unpackaged script on a Linux server.

Finally, the package has been merged into the openwrt/packages repo. It can now be installed on any snapshot build using opkg directly, or downloaded from the OpenWRT download site (note that the package is architecture independent).

I'll update the top post accordingly. Thanks to everyone for feedback and testing, and please continue using this topic for discussion!

5 Likes

People should keep in mind the impacts that QoS can have on CWND sizing. It takes some really big windows (in the thousands) to flood a gigabit pipe using a handful of streams.
If running QoS in the router, it tends to force the TCP stacks to back off on their CWND sizing (in the double digits or low triple digits) to achieve low-latencies. So one needs to use a LOT of concurrent streams.
Testing from browsers on clients, it could take 32 streams (or more) to get enough data on the line.

FYI this package works for 18.0.6.2 stable as well.

1 Like

Hmm I'm on OpenWrt 18.06.2 r7676-cddd7b4c77 but don't see the option to install speedtest in the software tab.

The package has not been officially backported to the stable 18.06 branch, so it only exists in the master development branch.

You can install it manually from the .ipk file (or just copy the script).

To install directly from a snapshot of the master branch:
(note the package is arch independent)

# cd /tmp
# uclient-fetch https://downloads.openwrt.org/snapshots/packages/mips_24kc/packages/speedtest-netperf_1.0.0-1_all.ipk
# opkg install speedtest-netperf_1.0.0-1_all.ipk
1 Like

same situation, the AC68U runs pretty cool and I see no realistic difference in temperature in regular use between different frequencies.