The hardware should be unchanged since it was manufactured - the OEM tested it then.
Those OEM tests don't tell you anything about the device running under latest OpenWrt.
2 Likes
I guess it could be a standardized test program to use. But why? The whole purpose with RC[n] is to make wild tests in real life use.
OpenWrt is made to run in the wild and you would be surprised what users here in the forum can do to get it non operational/bricked.
It is like kindergarten. If a salesmen say his product is child proof by their CE test program. Well throw it in to a kindergarden playground and close the doors, the little wolfs will jump on it right away.
Does the product after the end of the day actually work, is in one peace and look like the original product then it is quality checked by the best test operators in the world.
No, as I explained already. Any application running on the router would only test whether the drivers respond as expected, it cannot test whether the drivers actually do with the hardware what they're supposed to do. If the driver thinks everything is hunky-dory, but e.g. there's a bug that causes the WiFi-hardware to not actually work, there is literally no way for the application to determine that -- you need an external setup to verify against.
A quality test program do usually include external tests equipment.
That's what I am saying! You can't ship standardized external test equipment to every single OpenWrt-user or expect a test-environment that conforms to specific requirements, ergo such a standardized test would be impossible.
1 Like
This seems to be a complicated topic.
I was thinking along the lines of run some commands and validate that the desired behavior occurred. Validation would be weak without assessing the external environment.
One level of validation could be that the command completed without errors and the expected configuration was implemented.
Maybe could detect something like this: https://github.com/openwrt/openwrt/issues/9460
Where the radio doesn't even start up.
Could take another approach where you have a pre-configured partner device to bounce things off. Make a wired connection, make a wireless uplink, make a vpn connections, etc. Getting a tad complicated though.
Sure, but what would the point be? I mean, if you're just testing whether the commands run without error, then you're basically duplicating what RC-releases are for. There is no reason to expect the most basic functionality to just break in stable releases.
That doesn't seem particularly useful to me, either, considering the user would immediately see that the radio isn't working anyway, if they were setting it up.
Aaaaaaand here we get to the standardized test-equipment. I mean, how do you tell that the problem isn't on the other end instead or in the environment, if you don't standardize on some very specific hardware in a specific environment with specific settings? How do you tell, which features or functionality to measure and trust and which ones to discard, when you don't have a reliable baseline?
I would like to see openwrt releases stress tested thoroughly and regularly with the rrul and rtt_fair tests.
1 Like
Well then go ahead and do it and post the results. No one has said testing is a bad idea
2 Likes
I really don’t like quoting my self, but how much more testing do we actually need😂
Or is it the actual testing job you want just for the sake of it?
You do realize it cost money for the developers to have all these test equipment and network equipment and no one gets paid to do this.
And to be honest the security track record of OpenWrt equipment in todays cyber warefare world is pretty impressive in comparison with Asus where Russia took over all the Asus home routers firewalls in the whole world a couple of months ago.
I appreciate everyone's comments and feedback. Thanks.
I was thinking more that there would be a package or script that any user could download and run some standard tests. I would not try to burden devs with this. Basically crowd source the testing and leverage the broad community with wide diversity of devices. Sure, many people test many things in the rc releases. Could also have it where many people test the same subset of all the things across many devices.
This would give people an opportunity to contribute to the community with a low bar for skills and experience.
1 Like
Putting aside the need to have specialized test equipment (which can easily be in the range of $10K - $100K USD), I'd argue that writing test routines requires a higher level of skill than is needed for 'regular' coding work. This is not to suggest the the OpenWrt dev community is lower skilled -- if anything they're an amazingly talented group and surely capable of writing test code. But this writing unit and integration tests, and capturing ground-truth data to verify that things are working properly requires so much more than basic skills... one has to have intimate knowledge of how the system functions, what data would be expected, what the tolerances are, integration with external test equipment and scripting the test equipment functions, data collection, and data analysis. Test engineering is a specialization and not something that is going to be found with a "low bar" skill set.
1 Like
I'm actually planning to setup some Fuego, kernelci and/or Lava style test lab for OpenWrt. We recently integrated similar infrastructure at work and now have more than 50 embedded devices in our automation lab running dozens of tests every night. While I agree on some challenges discussed here especially when it comes to networking and Wi-Fi functionality, I still think any such testing will be very valuable in catching issues early. We now e.g. have better test coverage at work than some of them SoC vendors we base our stuff on! Well, okay, not that hard with some such ignorant bunch. Hahaha.
2 Likes
I would so appreciate a regular set of stress testing results with flent, primarily the rrul test and secondly with sqm (cake) enabled to a value where we know it should work, against ipv6 and ipv4 at the same time. It's really really really easy to fly through and compare 1000s of results with flent....
Regrettably I had to sell off my lab 5 years ago for lack of funding.
1 Like
Wow, a lot of interesting frameworks and utilities as mentioned by sumo and dtaht. I hadn't heard of these before.
https://elinux.org/images/0/08/Primer-Testing-Your-Embedded-System-What-is-a-ptest-Lava-Fuego-KernelCI-and...-Jan-Simon-Moeller-The-Linux-Foundation.pdf
https://flent.org/
https://www.bufferbloat.net/projects/codel/wiki/RRUL_test_suite/
There is also the testing standard from broadband forum: https://www.broadband-forum.org/download/TR-398.pdf
And the wifi alliance test: https://github.com/Wi-FiTestSuite
my hope would be to compare ,1 vs the rc5 vs a vs this truly exhausting regression here: AQL and the ath10k is *lovely* - #643 by amteza
across the mt76, ath9k, ath10k.
2 Likes
There are different kinds of testing. Hardware and performance testing are a bit out of reach for most. I was looking more for something that a user could run on a single device. This may have limited value.
I'm working on a script which sets some different config values as a test.
I thought it identified a problem where I could start 5g radio on channel 36, but not 100. Then I figured out it was due to missing country setting.
I put my bft.sh script on GitHub.
Trying to do something simple for this. It just runs locally on a router. I quickly saw how it doesn't scale well and testing framework may be worth the investment.
# DO NOT RUN ON A DEVICE IN SERVICE OR A DEVICE WHERE YOU CARE ABOUT THE CURRENT CONFIG
#
# Big Friendly Test (bft)
# this is a proof of concept and may lack the desired robustness
#
# this is a script to perform tests on a fresh OpenWrt install
# assumes standard image like rc# or formal release for current config.
# snapshot or a already modified device may not work well
# assumes some things like eth0 exists radio names like radio0 radio1, etc
# not intended for device in use, config will get stomped
# you may want to change the country code or some other settings
In sample output for Netgear EX6120. The tests that rely on an internet connection failed. This device does does not have a wan port by default and thus did not have an internet connection.
In sample output for Cudy X6 some of the tests have a status of "unknown". There is a test for success and another for failure. If neither of those match it gets "unknown". The success/fail checks are weak and loose.
The tests run unattended but you can play along so to speak. If you are easily amused you may enjoy watching the different SSID come up on different channels in something like WifiMan Android APP. There is another test that creates a listener on 8080 and then blocks it with the Firewall. So, you can test access before and after the block.
I think next, I may try one of the test frameworks or a two device test where networking and SSID could be confirmed.
Another idea was to consider OpenWisp. It is not really a good fit, but I could see register a device to OpenWisp and then manually cycle through some prepared template configs as tests.
I found that there was a similar discussion on this topic.
There is BoardFarm, an existing project for automating OpenWrt test. It is Python based.