What is the "TESTERS file"?

https://openwrt.org/docs/guide-developer/source-code/start#releases

This wiki page says

We will introduce a TESTERS file that is formatted similarly to the MAINTAINERS file of the kernel. Community members can list themselves as testers for a target/profile/device.

but I couldn't find TESTERS file in the openwrt/openwrt.git repository.

Is it obsolete?
Dose anyone know?

Given that the blurb about it was added to the wiki page in 2016, without it ever materializing in the repo suggests that it has never been picked up. The concept is interesting, but probably not that easy to accomplish, nor necessarily as useful as it sounds at first glance.

Starting with the elephant in the room, someone would have to start maintaining that file (if it were part of the git repository, as suggested there), with a relatively high churn rate (testers coming and going). Considering that it's already pretty hard to keep on top of actual code changes (github PRs), I fear that's not a very realistic prospect - meaning the actual implementation might have to be different (more like self-service, e.g. wiki page); GDPR would add another dimension to this though.

While technically useful, I fear it's not as usable in practice, than it claims to be - because what does 'tester' really mean? Are $you able to build a pending PR from source, to test it ahead of time, are $you following the main branch regularly or 'only' the stable releases. For the most part, there are three large aspects of changes:

  • general, not hardware specific, changes, this needs testing by $everyone (so a testers file wouldn't help for these)
  • target specific changes or refactoring, again a large number of testers is desired here - very useful and wanted, but actively pinging testers would wear them out quite a bit
  • device specific changes, that might entail new features or more refactoring-like changes like nvmem migration or similar, testing would be very useful here, but still rather on the intrusive side towards the testers
  • and beyond these 'obvious' things, pretty much any change, regardless how small it appears, might subtly break seemingly unrelated things (OpenWrt's buildsystems is heavily Makefile based, broken whitespace/ indentation or broken escaping/ quoting has a potential to break things big time), so semi-regular testing is needed all the time.

Before we even come to the question of how to deal with stale tester entries, which would be a completely different world of bother as well…

During LEDE times, a decision was made to drop explicit maintainer markings for targets, because it didn't work out that well in practice - it's usually better to take a short look at who was active in this area of the code recently, than trying to keep up with these maintainer labels (especially if most contributions for a given target don't really have an in-team maintainer), which are plagued by the same staleness issues as for testers above. Therefore it might not be out of line to draw an analogue to the testers label as well.


That out of the way, testing is hugely important. Everyone is strongly encouraged to test their areas of interest and expertise continuously (over all maintained branches, old-release/ release, pre-release and main), at your own pace and cadence (all the more power to you, if you manage to do so regularly, e.g every 1-2 weeks, every 4-6 weeks, whenever you get around to do so). Please do speak up about problems and regressions early, depending on the kind of issue forum, github bugs or mailing list posts come to mind. Everyone if very encouraged to keep an eye on pending PRs on github (or patch series on the mailing lists), to review- and test them, as well as giving feedback. It's very helpful to get things fixed before they hit the main branch, in the PR discussion phase - don't wait for a call to arms, be proactive - and keep things fun (for you!).

5 Likes

Thank you for the detailed explanation.
So, does this mean that the "TESTERS file" system is not currently adopted, at least for now?

Currently, no it isn't.

1 Like

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.