Did you post those logs as “preformated text”?
Maybe add crusader (https://github.com/Zoxc/crusader) as a nifty tool to generate load between two endpoints and also concurrent latency measurements. Not as powerful as i.e. iperf2 but it will generate simple graphs to allow assessing the measurement immediately after each run.
2 Likes
Another thought: Instead of attempting to use your router to source/sink data (which will affect its routing performance), you could use a small single-board computer with Ubuntu to run Flent, Crusader, iperf, etc.
I have a RPi 4 (purchased before the pandemic - so not too expensive) and an Odroid (don't remember what model) attached to Ethernet ports on my router...
Does anyone have recommendations/links for other (modest price) devices that could run those tests?
1 Like
Yeah, and even if you have a (spare) desktop computer (Linux/Mac/Windows) connected with gigabit Ethernet to the same network as the wifi access point, run iperf there as the other end. Ensure the Ethernet speed is faster than the highest possible speed of the wifi network.
Anti-pattern: measuring wifi by connecting to an upstream server. You'll be hard-pressed to segregate the wifi link peformance and the upstream link/general Internet's performance.
1 Like
Thank you for starting this great thread. A lot of good advice in here.
I also have a few tips to contribute.
iperf3 tips
One important thing to note here is that by default iperf3 UPLOADS data from the client to the server.
The reason this is important to consider on WiFi is that WiFi clients oftentimes have much weaker antennas than WiFi routers.
For example, on my M1 Macbook Air I can get up to 700-800 mbit when downloading from the server, but only 400-500 when uploading.
This can be solved by using this option:
-R, --reverse run in reverse mode (server sends, client receives)
I would suggest to add that to all the commands in your post, because download is probably what most people are interested in testing.
Where you run the testing tool server and client is important
A lot of the performance is affected by the processing speed of the components between the client and the server.
For example, if you have a router with a weak CPU and run the iperf3 server on the router, then it's possible the speeds will be limited, because iperf3 itself uses up a large portion of the router's available CPU and starves the router of the CPU to achieve full wifi transmission rates.
The solution would be to connect a wired computer to the router that runs the iperf3 server. And then run the test from a wifi client to the wired computer.
But this stil doesn't give you the FULL picture. Because routing from lan to wan also requires CPU power and this is not being done when testing on lan only. This is especially important if you want to test the maximum bandwidth your router can handle with SQM.
If you want to determine the routing capabilities, then the iperf3 server should be "behind the WAN port" of the router.
Realistically you would need two routers for that setup.
For example, maybe you have the ISP router that's connected to the internet and has a LAN of 192.168.1.0/24.
Then you connect the new router you want to test to it that has its own lan of 192.168.2.0/24. Plug the new router's WAN port into the ISP router's LAN port so that it is in both networks and routes between them.
Then you connect a the iperf3 server to the ISP router. And you connect the wifi client to the new router.
Then you run the test from a client connected to the new router. This fully tests the performance of your router and all components involved.
Again, this is most likely not necessary if you just want to test your current network connection as you can run speedtests directly to the internet to test the routing performance.
But if you want to determine the maximum performance your router is capable off, then that's one way to do it.
Test bufferbloat on both wired and wifi
A similar issue si when you are measuring bufferbloat (latency under load). You should always first test it on ethernet to determine the bufferbloat performance of the routing component and SQM system.
This tests the bufferbloat between router and ISP.
After that you can test it on WiFi to determine if there is any additional bufferbloat between WiFi -> Router.
Avoid WiFi scanning tools while testing
Another important thing that could trip you up if you are testing a lot of things is WiFi scanning.
If you run a software that analyses channels, then the software is going to make periodic scans of available WiFi access points.
When it does that, it oftentimes interrupts the entire WiFi connection. This causes all WiFi transmissions to stop for a few miliseconds or even seconds.
This can cause you to have huge ping spikes and reduced transmission that will give you the wrong impression of your WiFi performance.
2 Likes
@ThiloteE You are only interested in English-language texts, I suppose?
I wrote something in German a while ago, because I was active in an action group who helped to bring fiber-optical internet to our small place, and in the aftermath I got countless questions re. WiFi problems, so I created a blog post about how to debug WiFi issues... I also created a similar blog post about internet issues in general (again in German).
In case you are interested anyway, please let me know, and I'll share the links here.
Sure, i would be delighted 
although I think most people here speak English, as it seems to be the current lingua franca. I am from Germany too 
1 Like
Yeah, right. But maybe for German people less fluent in English the posts could be helpful?
Anyway, here's the blog posts, feel free to include them in your collection (or to omit them):
- WLAN-Probleme erkennen und beseitigen (detect and fix WiFi problems)
- Mein Internet funktioniert nicht, was tun?! (my internet is not working, what to do?)
(I suspected already you are German because of your username -- the leading part is a well-known German given name... )
2 Likes
Thanks for the time you spent and the info shared. I'm a user and seldom need to get down to the packet level. My Fave tool is android app "WiFiAnalyzer", the VREM version. You can watch your radio in almost real time. Handy for initial config of wrt3200acm which has too many radios.
1 Like
== Update 1 iperf3 under Windows on Arm
https://njit.io/kb/os/windows/how-to-compile-iperf3-for-windows-using-cygwin/
Followed the above post, installed Cygwin, compile iperf3 under Cygwin, run iperf3 under Cygwin, now I got same speed as running Speedtest in Edge browser. (Chrome Browser is much slower, due to X64 binary for Chrome only)
===
Where I can find out the best tools for the listed tests under:
Windows x64
Windows for Arm
Linux
iPhone
Android
I want to test with my Windows for Arm laptop, Android (Nokia 9) and iPhone 13, with Dynalink DL-WRX36 Stock / OpenWrt.
The Bufferbloat test speed is faster than iperf3 speed, so I suspect the Win64 binary is not running well under Windows for Arm.
Please open a new forum post for your issue. We can debug it over there. As I mentioned in my first post "This thread here is about teaching. This thread here is not about debugging your very personal setup of OpenWRT". When you open a new thread, I would ask you to explain your setup (where is the client, where is the server) and which commands you used for the measurements. Also, your exact configuration of OpenWRT can be important.
With regard to your questions:
- It depends on the specific tools, if they provide builds and releases for multiple operating systems. About "best" tools, I don't know, but Iperf3 should be a "good enough" tool for what you want to do.
- "The bufferbloat test is faster than iperf3 speed" depends on how and where you set up your measurements. Also, you should test wifi and the connection from your main router to your Internet service provider (ISP) separately. It could be the case, that your wifi is worse than the speed provided by your ISP. The bufferbloat test by waveform should be conducted from your main router or at least from a client connected to your main router via cable and not via wifi, as wifi uses buffers and may distort the bufferbloat measurements.
- Alternative download location for newest version of iperf3 for windows: https://community.chocolatey.org/packages/iperf3
Thank you for the reply.
I am sorry that I do not make myself clear enough.
From the Windows on ARMs laptop, the performance difference is due to the version of iperf3 used (running on that laptop).
Once I compiled iperf3 under Cygwin, the figure matches (iperf3 w/ Cygwin Vs Speedtest.net / Waveform Bufferfloat)
I am learning about the tools, and trying to find out which tool is available to which platform.
Like,
- flent seems not available on Windows
- TWAMP-GUI not available on Fedora
Until I learn to use more tools, iperf3 will be the tool I am testing for the moment.
flent itself is available on windows IIRC, but its current main data source netperf is not, so you can create plots under windows, but you can not actually take measurements.
So iperf3's biggest advantage over most of the rest of the list is that there exist some publicly open servers on the internet than can be used for some tests.
However, for real reproducible tests one probably needs to deploy one's own remote measurement points and then all tools are more or less available again...
1 Like
Sidenote: a lot of the measurement packages seem either "complete" or stalled in development...
Which is a real pity especially for the one-way delay measurement tools out there, just had a look at owping and e.g. the absence of an installable packet for ubuntu does not fill me with confidence (let alone OpenWrt)...
I wish IPv6 had mandated that ICMP timestamps would still be required (and I dream about a tighter definition/resolution of what to reply, so e.g. microseconds since midnight UTC instead of millisecond since midnight UTC and no backsies...). Alas, not much use closing that barn, as the time window to solve that is clearly well in the past by now.
1 Like
A command that queues 14 Iperf3 throughput tests and writes the output to a text file:
Today I learned how to queue commands in cmd.exe and how to write output to a textfile. Cmd.exe is the Windows 10 command line tool; The command should also work with Linux, but I have not tried it yet.
Here is my final command, which singlehandedly triggers **14 separate iperf3 throughput tests and writes the output to a textfile located in the folder the command was called from.
(echo TCP test && echo iperf3 -c IP-ADRESS-OF-IPERF-SERVER -p 5201 -t 60 -i 10 && echo. && iperf3 -c IP-ADRESS-OF-IPERF-SERVER -p 5201 -t 60 -i 10 && echo. && echo. && echo TCP test with 2 client streams && echo iperf3 -c IP-ADRESS-OF-IPERF-SERVER -p 5201 -P 2 -t 60 -i 10 && echo. && iperf3 -c IP-ADRESS-OF-IPERF-SERVER -p 5201 -P 2 -t 60 -i 10 && echo. && echo. && echo TCP test with 4 Client Streams && echo iperf3 -c IP-ADRESS-OF-IPERF-SERVER -p 5201 -P 4 -t 60 -i 10 && echo. && iperf3 -c IP-ADRESS-OF-IPERF-SERVER -p 5201 -P 4 -t 60 -i 10 && echo. && echo. && echo TCP test with 8 Client Streams && echo iperf3 -c IP-ADRESS-OF-IPERF-SERVER -p 5201 -P 8 -t 60 -i 10 && echo. && iperf3 -c IP-ADRESS-OF-IPERF-SERVER -p 5201 -P 8 -t 60 -i 10 && echo. && echo. && echo Bidirectional TCP test && echo iperf3 -c IP-ADRESS-OF-IPERF-SERVER -p 5201 -t 60 -i 10 --bidir && echo. && iperf3 -c IP-ADRESS-OF-IPERF-SERVER -p 5201 -t 60 -i 10 --bidir && echo. && echo. && echo Bidirectional TCP test with 2 Client Streams && echo iperf3 -c IP-ADRESS-OF-IPERF-SERVER -p 5201 -P 2 -t 60 -i 10 --bidir && echo. && iperf3 -c IP-ADRESS-OF-IPERF-SERVER -p 5201 -P 2 -t 60 -i 10 --bidir && echo. && echo. && echo Bidirectional TCP test with 4 Client Streams && echo iperf3 -c IP-ADRESS-OF-IPERF-SERVER -p 5201 -P 4 -t 60 -i 10 --bidir && echo. && iperf3 -c IP-ADRESS-OF-IPERF-SERVER -p 5201 -P 4 -t 60 -i 10 --bidir && echo. && echo. && echo Bidirectional TCP test with 8 Client Streams && echo iperf3 -c IP-ADRESS-OF-IPERF-SERVER -p 5201 -P 8 -t 60 -i 10 --bidir && echo. && iperf3 -c IP-ADRESS-OF-IPERF-SERVER -p 5201 -P 8 -t 60 -i 10 --bidir && echo. && echo. && echo TCP test in reverse && echo iperf3 -c IP-ADRESS-OF-IPERF-SERVER -p 5201 -t 60 -i 10 -R && echo. && iperf3 -c IP-ADRESS-OF-IPERF-SERVER -p 5201 -t 60 -i 10 -R && echo. && echo. && echo TCP test with 2 Client Streams in reverse && echo iperf3 -c IP-ADRESS-OF-IPERF-SERVER -p 5201 -P 2 -t 60 -i 10 -R && echo. && iperf3 -c IP-ADRESS-OF-IPERF-SERVER -p 5201 -P 2 -t 60 -i 10 -R && echo. && echo. && echo TCP test with 4 Client Streams in reverse && echo iperf3 -c IP-ADRESS-OF-IPERF-SERVER -p 5201 -P 4 -t 60 -i 10 -R && echo. && iperf3 -c IP-ADRESS-OF-IPERF-SERVER -p 5201 -P 4 -t 60 -i 10 -R && echo. && echo. && echo TCP test with 8 Client Streams in reverse && echo iperf3 -c IP-ADRESS-OF-IPERF-SERVER -p 5201 -P 8 -t 60 -i 10 -R && echo. && iperf3 -c IP-ADRESS-OF-IPERF-SERVER -p 5201 -P 8 -t 60 -i 10 -R && echo. && echo. && echo UDP test && echo iperf3 -c IP-ADRESS-OF-IPERF-SERVER -p 5201 -t 20 -i 2 -u -b 1000M && echo. && iperf3 -c IP-ADRESS-OF-IPERF-SERVER -p 5201 -t 20 -i 2 -u -b 1000M && echo. && echo. && echo UDP test in reverse && echo iperf3 -c IP-ADRESS-OF-IPERF-SERVER -p 5201 -t 20 -i 2 -u -b 1000M -R && echo. && iperf3 -c IP-ADRESS-OF-IPERF-SERVER -p 5201 -t 20 -i 2 -u -b 1000M -R) | tee output.txt
What to do if you want to use this command:
On Iperf server side:
- Temporarily open relevant ports (5201) in the firewall, which are necessary for Iperf to work
Iperf3 -s
On Iperf client side:
- copy 'n paste the command into a text editor
- search and replace
IP-ADRESS-OF-IPERF-SERVERwith the IP adress of your Iperf server. - (Optional) search and replace
5201with the port your Iperf server listens at. (5201 is Iperf3's default) - (Optional) search and replace
output.txtwith the filename of your choice
The long command explained:
I use following syntax:
(echo. && echo. && echo <Iperf3command1> && echo. && <Iperf3command1> && echo <Iperf3command2> && echo. && <Iperf3command2> && echo <Iperf3command3> && echo. && <command3>...) | tee <filename>
The important bits (apart from iperf3 related) are ...
-
echo.= A new line. This will make it easier to find the tests you were looking for in the output file. Simply for Aesthetics. -
echo <command>= Theechocommand in Linux is used to display a string provided by the user. The syntax isecho [option] [string](More info here: https://phoenixnap.com/kb/echo-command-linux). What is does is to print the text you entered after it. Withecho <command> && <command>it will have the command you entered precede the output. This will make it easier to find the tests you were looking for in the output file. -
&&= Example:command1 && command2= Used to run the command following && only, if the command preceding the symbol is successful. Cmd.exe runs the first command, and then runs the second command only if the first command completed successfully. -
<your-command> | tee result.txt= Show output in Windows cmd, but also send to output file (https://stackoverflow.com/questions/25702196/how-to-save-iperf-result-in-an-output-file). The()are necessary, when multiple commands are queued, otherwise only the last command before| teewill be written to the file.
What advantages brings command queuing?
You will save lots of time. Gone are the days when you have to wait in front of your computer for a test to finish, before you can start the next command. With this syntax you can trigger multiple tests with a single press of the enter button and then spend some time with friends or simply enjoy life and come back in 30 minutes and everything should be done by then
1 Like
I found the following article to be a great read and a pointer towards determining what end-users and Internet service providers should focus on when trying to detect throughput and latency bottlenecks in their network:
While I think the whole article is worth reading, there are some particular things worth noting
-
There is a point where maximum throughput over time becomes less relevant to the average end-user, because as soon as individual applications (e.g. video streaming, video conferences, audio, gaming, ...) are successfully transmitting with their maximum saturation, no higher throughput is required by the application. This holds true until applications evolve in response to changes in the underlaying technology.
Following graphics illustrate this point:
That is to say, high throughput services (e.g. 1 Gbit/s) are NOT useless. They CAN aleviate the problem of conquestion, that is when there is too much traffic (by multiple users) going on at the same time. Since 1 Gbit/s services usually are achieved by installing fiber networks, getting rid of older infrastructure (e.g. copper wires) also eliminates at least some (marginal?) latency that came with that. In any case, conquestion is not the only problem users and ISPs have.
-
In the article, there is lots of info about bufferbloat, buffers, queuing, packet loss rates, the importance of location and what latencies to expect, simply because of lower bound limitations (e.g. speed of light, latency in voice calls caused by encoding / decoding sound, ...).
I guess, the takeaway would be, if you are an end-user and want to tune your network: do the waveform bufferbloat test and checkout QoS, FQ-Qodel, Cake, AQM and so on and so forth. (Note: not all of these tools apply to wifi!). While there are some things you can do with your local device, a lot depends on your internet service provider.
-
If you require low latency, (automatic) tuning of AQM and introduction of Low Latency, Low Loss, Scalable Throughput (L4S)
are something to look forward to!are proposed to make the world a better place (but it seems like L4S also has it's deficiencies).
I respectfully disagree... L4S was designed by people that believe "believing hard enough" can replace actual engineering, and it shows (in that L4S has known problem spots that in spite of being described well have not been addressed by L4S' proponents). L4S is designed primarily to built a short RTT fast-lane, without actually confessing that. You can assign that to incompetence or to intent, I opt for the latter. In that role it seems like an attempt at avoiding network neutrality rules all the while giving close-by data centers (aka larger ISPs with captive eye-balls) more leverage when trying to sell data center close to the eyeballs to content providers (CAPs)... as with L4S a CAP close by will massively outperform a CAP further away if both compete.
L4S is too little too late*... if you use sqm-scripts/cake today it is unlikely that you will notice a switch to L4S as a worthwhile improvement in responsiveness.
*) and it was already that decade ago its design began
I still consider it unfortunate that the IETF actually decided to publish the 3 L4S RFC before the well known deficiencies had been remedied, because as it stands these deficiencies will likely never be remedied. However I predict L4S will simply stutter/fail in the market as it over-promises and under-delivers.
3 Likes
Thank you for your insightful comment!
No with L4S (at least with the default L4S AQM the dual queue AQM) there is only one of each:
a) a low latency queue
b) a non-low latency queue
all L4S flows will share queue a) all the rest will share queue b), any flow misbehaving in a) will affect all other flows in a) as well (there are some proposed optional remedies (queue protection), but these are all easily side-stepped as they are not aimed at adversarial flow behavior). The L4S specifications also allow flow-queueing schedulers that would avoid that problem, but the expected largest immediate deployment of L4S (low latency docsis) will opt for a simplistic dual queue set-up and IMHO will be far inferior to today's cake/sqm-scripts.
Note however I am hardly objective in this matter, as I tried to convince the IETF to not ratify L4S and failed, so you can interpret my objection as valid or as "sour grapes", your choice
3 Likes