Test Suite for OpenWrt

Is there any kind of standard test suite or process for OpenWRT?

I was thinking that for a new release, I could run a barrage of tests on a specific hardware device to make sure everything works.

No, because it would be basically impossible to do. How would you e.g. test WiFi-performance and -functionality in a standardized manner without standardized test-equipment to connect to and monitor with? You literally can't test such without another device or devices, because e.g. drivers could be buggy and give false readings, so you need something external to corroborate any such readings.

1 Like

The hardware should be unchanged since it was manufactured - the OEM tested it then.

Those OEM tests don't tell you anything about the device running under latest OpenWrt.

2 Likes

I guess it could be a standardized test program to use. But why? The whole purpose with RC[n] is to make wild tests in real life use.
OpenWrt is made to run in the wild and you would be surprised what users here in the forum can do to get it non operational/bricked.

It is like kindergarten. If a salesmen say his product is child proof by their CE test program. Well throw it in to a kindergarden playground and close the doors, the little wolfs will jump on it right away.
Does the product after the end of the day actually work, is in one peace and look like the original product then it is quality checked by the best test operators in the world.

No, as I explained already. Any application running on the router would only test whether the drivers respond as expected, it cannot test whether the drivers actually do with the hardware what they're supposed to do. If the driver thinks everything is hunky-dory, but e.g. there's a bug that causes the WiFi-hardware to not actually work, there is literally no way for the application to determine that -- you need an external setup to verify against.

A quality test program do usually include external tests equipment.

That's what I am saying! You can't ship standardized external test equipment to every single OpenWrt-user or expect a test-environment that conforms to specific requirements, ergo such a standardized test would be impossible.

1 Like

This seems to be a complicated topic.

I was thinking along the lines of run some commands and validate that the desired behavior occurred. Validation would be weak without assessing the external environment.

One level of validation could be that the command completed without errors and the expected configuration was implemented.

Maybe could detect something like this: https://github.com/openwrt/openwrt/issues/9460
Where the radio doesn't even start up.

Could take another approach where you have a pre-configured partner device to bounce things off. Make a wired connection, make a wireless uplink, make a vpn connections, etc. Getting a tad complicated though.

Sure, but what would the point be? I mean, if you're just testing whether the commands run without error, then you're basically duplicating what RC-releases are for. There is no reason to expect the most basic functionality to just break in stable releases.

That doesn't seem particularly useful to me, either, considering the user would immediately see that the radio isn't working anyway, if they were setting it up.

Aaaaaaand here we get to the standardized test-equipment. I mean, how do you tell that the problem isn't on the other end instead or in the environment, if you don't standardize on some very specific hardware in a specific environment with specific settings? How do you tell, which features or functionality to measure and trust and which ones to discard, when you don't have a reliable baseline?

I would like to see openwrt releases stress tested thoroughly and regularly with the rrul and rtt_fair tests.

1 Like

Well then go ahead and do it and post the results. No one has said testing is a bad idea

2 Likes

I really don’t like quoting my self, but how much more testing do we actually need😂
Or is it the actual testing job you want just for the sake of it?

You do realize it cost money for the developers to have all these test equipment and network equipment and no one gets paid to do this.

And to be honest the security track record of OpenWrt equipment in todays cyber warefare world is pretty impressive in comparison with Asus where Russia took over all the Asus home routers firewalls in the whole world a couple of months ago.

I appreciate everyone's comments and feedback. Thanks.

I was thinking more that there would be a package or script that any user could download and run some standard tests. I would not try to burden devs with this. Basically crowd source the testing and leverage the broad community with wide diversity of devices. Sure, many people test many things in the rc releases. Could also have it where many people test the same subset of all the things across many devices.

This would give people an opportunity to contribute to the community with a low bar for skills and experience.

1 Like

Putting aside the need to have specialized test equipment (which can easily be in the range of $10K - $100K USD), I'd argue that writing test routines requires a higher level of skill than is needed for 'regular' coding work. This is not to suggest the the OpenWrt dev community is lower skilled -- if anything they're an amazingly talented group and surely capable of writing test code. But this writing unit and integration tests, and capturing ground-truth data to verify that things are working properly requires so much more than basic skills... one has to have intimate knowledge of how the system functions, what data would be expected, what the tolerances are, integration with external test equipment and scripting the test equipment functions, data collection, and data analysis. Test engineering is a specialization and not something that is going to be found with a "low bar" skill set.

1 Like

I'm actually planning to setup some Fuego, kernelci and/or Lava style test lab for OpenWrt. We recently integrated similar infrastructure at work and now have more than 50 embedded devices in our automation lab running dozens of tests every night. While I agree on some challenges discussed here especially when it comes to networking and Wi-Fi functionality, I still think any such testing will be very valuable in catching issues early. We now e.g. have better test coverage at work than some of them SoC vendors we base our stuff on! Well, okay, not that hard with some such ignorant bunch. Hahaha.

2 Likes

I would so appreciate a regular set of stress testing results with flent, primarily the rrul test and secondly with sqm (cake) enabled to a value where we know it should work, against ipv6 and ipv4 at the same time. It's really really really easy to fly through and compare 1000s of results with flent....

Regrettably I had to sell off my lab 5 years ago for lack of funding.

1 Like

Wow, a lot of interesting frameworks and utilities as mentioned by sumo and dtaht. I hadn't heard of these before.

https://elinux.org/images/0/08/Primer-Testing-Your-Embedded-System-What-is-a-ptest-Lava-Fuego-KernelCI-and...-Jan-Simon-Moeller-The-Linux-Foundation.pdf
https://flent.org/
https://www.bufferbloat.net/projects/codel/wiki/RRUL_test_suite/

There is also the testing standard from broadband forum: https://www.broadband-forum.org/download/TR-398.pdf
And the wifi alliance test: https://github.com/Wi-FiTestSuite

my hope would be to compare ,1 vs the rc5 vs a vs this truly exhausting regression here: AQL and the ath10k is *lovely* - #643 by amteza

across the mt76, ath9k, ath10k.

2 Likes

There are different kinds of testing. Hardware and performance testing are a bit out of reach for most. I was looking more for something that a user could run on a single device. This may have limited value.
I'm working on a script which sets some different config values as a test.

I thought it identified a problem where I could start 5g radio on channel 36, but not 100. Then I figured out it was due to missing country setting.