I’m working on a small open-source tool called WTF — Wi-Fi Test Framework. The goal is to automate repetitive Wi-Fi throughput tests on OpenWrt-based access points.
The basic use case is:
Change Wi-Fi radio settings on the AP
Apply/reload the wireless configuration
Run iperf3 throughput measurement
Collect the result
Repeat the same process for multiple channel/bandwidth combinations
This is mainly intended for testing Wi-Fi performance under different radio configurations, for example when comparing different channels, bandwidth settings, or AP/client setups.
At the current stage, WTF is a Python CLI tool that runs from a Linux client machine and controls the OpenWrt AP remotely.
Current architecture:
Linux client machine runs WTF
WTF connects to the OpenWrt AP via SSH
Wireless settings are changed using UCI
iperf3 is used for throughput measurements
A separate control network is used for SSH/UCI access
A Wi-Fi test network is used for throughput testing
The control network and the test network are expected to use different IP address spaces.
Current functionality:
configuration loading and validation
basic CLI commands
OpenWrt AP abstraction
SSH/UCI interaction
initial iperf3 test execution flow
result saving
Planned next steps:
better error handling
more automated tests
CI/CD
improved documentation
eventually PyPI packaging
The project is still early-stage and not production-ready. I’m mainly looking for feedback from people who work with OpenWrt, Wi-Fi testing, or network automation.
I would especially appreciate comments on:
whether this client-side architecture makes sense;
better ways to apply/reload wireless settings after UCI changes;
common failure cases I should handle when changing Wi-Fi settings remotely;
whether there are existing OpenWrt tools or workflows I should study;
how to structure tests for this kind of tool without requiring real hardware for every test.
Why not use a test framework, like pytest? I think it would be beneficial, because:
Save you from inventing a machinery that runs tests - it's already there
It is familiar to a lot of people (or should be ), so it's going to be easier for people to use, and also for them to contribute to the project
It comes with builtin (or plugin-available) report generation capacity, so you can make this fancier using existing components, rather than build it all from scratch
It also provides an easier way to repeat the same test with different parameters
Realistically, pure throughput tests will only show how well a wifi link deals with bulk data transfers. Many use cases however are interactive and their quality will, above a certain capacity (often 25-50 Mbps), already plateau out, and the perceived quality will depend much stronger on latency and jitter. So I would ask you to also add concurrent latency measurements to your load testing, so you will know the latency and jitter for both idle and loaded conditions. Finally, and that might already be part of your system, I would recommend to use 3 distinct load phases with a bit of cool off between them: download only, upload only, concurrent down- and upload.
At the moment WTF is more of a benchmark-style utility, so pure pass/fail logic provided by pytest is not what I really want from it. But I agree that pytest features, like parametrization and fixtures, could be useful here.
Also I'm still newbie for Python&pytest infra/env, so I'll explore if there are suitable benchmark-style reporting plugins or patterns I'll think about pytest could be integrated into WTF
At the moment WTF is mostly focused on throughput measurement, because that was the original task that led to the idea of WTF.
This looks like a very good next step for the project. My first idea is to run ping in parallel while the iperf3 test is running, and then store latency statistics together with the throughput results. Do you think it's god enoug or there should be another execution/measurement methodology?
Have a look at what crusader (https://github.com/Zoxc/crusader) does... three load phases all with bidirectional latency measurements (and per side reporting of lost latency probes). That is a pretty decent approach to quickly characterize a link. If possible try to measure one-way delays, but that can be tricky, and then simple RTT measurements are already much better than nothing. The challenge is to make the latency probes dense enough be be interpretable but not so dense that they cause too much of a load by themselves...
I see you point, those are tests to obtain metrics not validations. But as you pointed out, you might benefit from the structure it provides to make the development of your tests easier. That beings said, I dare say that one day, in the future you will reach the point where (or asked to) thresholds will be set to determine if those metrics you have been obtaining are good or bad.
(not necessarily a problem for pytest to solve, I know)
The OWD measurements require another util, so I think I'll make some day lightweighted OWD metrics collector, but won;t include it to the WTF, at least for now.
), so it's going to be easier for people to use, and also for them to contribute to the project