Criteria for first LEDE stable release?

@stangri and @jow were discussing the evolution of LEDE in What should we call the current builds?)

I have renamed this discussion and brought it into the Developer category of the forum.

stangri:
So what's the next feature driving the next release of OpenWrt/LEDE?

jow:
I don't think there's any killer feature driving the progress of the entire project. Development currently is mainly driven by external contributions (device support patches, bug fixes) and polishing/extending/fixing existing sub systems as well as overall performance improvements.

stangri:
Whatever the project, unless it's feature complete, there's always a feature which can define the release criteria for the next milestone (or two).

jow:
I can't think of any suitable one. Here's some example:

  • Device tree support for ar71xx? That is months to years away and would delay the release forever.
  • Kernel 4.9? That will invalidate all stabilization work invested into the tree so far.
  • New device? New devices are added one to two times per week so on which one should we wait and why only on that and not others?
  • Bug count below 50 ? Various issues in the tracker are eternal bugs which are unlikely to be fixed within a foreseeable time frame.

To return the question what exactly would you define as a release criteria?

So far you seem to worry about new devices first but given that releases happen every two to three months, is there an actual problem? In the worst case, a new device is added shortly after a release which would imply a maximum waiting time of two to three months until the next release branch is started. But even in this situation there is nothing preventing us from cherry picking the device support patches into the current release branch and make an YY.MM.N+1 point release within a day.

stangri:
Wasn't having infrastructure in place to support first LEDE release one of the first goals of the fork? Wouldn't then having all infrastructure in place for the release warrant a first release? Is infrastructure in place now?

jow:
The hardware is in place, the release setup is doing build testing since a few days.

It seems to me that this is a good first step toward criteria that lead to our first release. I see these important points:

  • Current builds are pretty stable
  • No killer features are outstanding. (There are plenty of enhancements in the wings that can be incorporated in the future, but they're not holding up a release.)
  • Build system is in place, although only in early testing phase

Is this enough? Could we declare the current feature set (more or less) to be sufficient for our first release?

I can only answer in short, exclusively with affirmatives: Yes, and yes please. The sooner the better.

2 Likes

Based on the past 3 months I would pick 3 milestones.

  1. General tree cleanup and merging of the backed up openwrt patch queues - this seems to have slowed weeks ago, basically done?
  2. Ath9k buffer bloat improvements and added coolness - I've been running the airtime fairness patch for weeks, I don't know if this ever left staging. Would be nice if it got in.
  3. Release tooling - @jow seems to have slowed down so I'm guessing he's nearly happy with things

As soon as the tooling is done I'm guessing we'll get a beta release. Can't see anything else we need to wait on. Would be cool to have a Christmas day or New years day final release.

We decided against boose codenames right?
I think "Reboot" is fine for LEDE1.0, but has any discussion happened over what the next codename meme is going to be?
Dank memes are important for long running projects :slight_smile:

FYI, for those who don't know, my perspective is from one whose normal job is a product manager, not a developer. My usual role is sitting with the developers and, based on the feedback from "the market," saying "this is great for now, please finish that for this release, and we can leave the other thing for a future release."

I think the current feature set, based on what I have seen (I do not claim full knowledge of the features of LEDE), are quite sufficient for a release. I think the goal of a "release" for a "product" like LEDE should be to provide a well-developed, well-tested, well-documented "snapshot" of the development at a particular point in time for others who want to either just put it on a device to use or to use as a stable starting point for additional development. I have to agree with weedy, though, that getting the bufferbloat improvements into the release would be a very good thing.

Into the future, I think it would be useful to simply choose a point in time for a "proposed" release, and evaluate the ongoing projects at that time. If the core features are in good shape, then we push to a release based on rough agreement on what's in, what's out, and what needs to be hurried up into the release. The time (every 6 months? 3 months?) just stands as a flag on the horizon so everyone has a timeframe to work to (if I want this in the next release, I need to get it integrated by May for the June release...).

Having releases far apart means the releases are more consequential - more new features and capabilities - but making them closer together decreases the penalties for "missing" the release, which I believe increases the overall quality of the code in the release. More frequent releases are a boatload of work, but, if you do it frequently, it becomes much more routine. The further apart the releases are, the more "startup overhead" there is in the release mechanism.

The other thing that frequent releases potentially help with is back-porting. If there's a critical bug found, back-porting, as ugly as it is, is probably necessary. However, with frequent releases, you can just back-port to the most recent release and go forward, because nobody needs to be "married" to a year-old release to maintain compatibility.

So my two cents worth is this: let's just do a release ASAP (preferably with the bufferbloat stuff in, if everyone agrees it's stable and working properly) and then do small releases every 3 months -in other words, have a fairly constant release cycle, with one going out the door and one coming up on the horizon all the time. Let's automate it (to the maximum extent possible) and minimize back-porting.

-Bill

+1

The sum of all the small improvements that have been done since the reboot is more than sufficient reason to release. I think it would be good to have a new "baseline" for further development now. If the infrastructure is ready, then I say: release asap.

If my opinion as more of a user than developer counts for anything :wink:

I believe the understanding is that the first release will happen when there's an infrastructure in place to support release. Can't happen sooner than that. :wink:

What's missing?

Few days ago i posted this question:
https://forum.openwrt.org/t/who-wants-to-use-lede-and-is-still-not-using-it-and-why/509/2

Maybe the replies that I get there could shine a light on what certain end users like to see for LEDE's first release.

+1

The sum of all the small improvements that have been done since the reboot is
more than sufficient reason to release. I think it would be good to have a
new "baseline" for further development now. If the infrastructure is ready,
then I say: release asap.

IMO, there would have been value in making a LEDE release as soon as the
branding change from OpenWRT to LEDE was completed, even with no other
development work, just to give people a place to start with before doing any
other development.

beyond that, a release a few months later to include the initial burst of
cleanup work would have also been very nice to see.

There is no need to wait for something "significant" enough to be a release,
LEDE has that many times over.

David Lang

1 Like

Hear hear. In fact, I can imagine that collecting all improvements over the last OpenWrt release is itself quite a task. Just look at how many new devices are supported, how many libraries have been updated, the number of structural and organizational improvements. The "list of improvements" that would go into the release notes would be massive already.

I flashed a firmware today and it changed from CURRENT to SNAPSHOT :smile: how fast :laughing:

For me, I'd love to see the bufferbloat improvements on Ath9k in the first release. But that's just me. Feature-wise I think LEDE is fine at the moment for its first release. The most important thing is a well-tested and thus stable first release that can be used in home networks without fear of using a unstable snapshot on accident.

For the record, I would like current versions of both bufferbloat and make-wifi-fast/airtime-fairness fixes to be incorporated.

They would bring demonstrable benefits to the initial LEDE release since they dramatically decrease latency/lag and increase wireless performance with essentially no downsides.

My impression (which I'm sure will be vetted by those teams) is that:

  • Bufferbloat/fq_codel/cake/etc. are in very good shape and are included by default.
  • Make-wifi-fast/Airtime-fairness are also stable and built into multiple independent/community builds. Even though there may be further improvements to be had in the next few months, we would be well-served to have them in place.

@Mushoz

So this would bring in a Release Candidate test cycle?

Is there a test checklist or something that we can use to make sure important stuff is actually being tested?
What do the DEVs want to know? What can end users do? Would it be possible to have some tests automated? Not sure if that makes any sense, the idea just popped into my mind just now.

@charcoal I am not a developer, so I'm not the right person to ask. Just an enthusiastic user giving his opinion. I'm more than willing to help test release candidates once it comes to that.

And if so. Will this result in a feature freeze to "force" only commits that improve what is already there to be merged/cherrry-picked into the stable branch?

[quote="steinmb, post:16, topic:552"]

So this would bring in a Release Candidate test cycle?

Will this result in a feature freeze to "force" only commits that improve what is already there to be merged/cherrry-picked into the stable branch?
[/quote]Pretty much so on the release branch. but development goes on in the master. The release branch only gets fixes and possibly a few important changes/features.

Based on experience with Openwrt releases (AA, BB, CC), the rc phase tends to get a bit too prolonged and both devs' and enthusiasts' interests wander toward the new functionalities that are implemented in the master at the same time.

The release/branching automation work done recently by @jow should enable a more rapid rc/release cycle than earlier. And that is a major benefit as a long rc phase tends to move into a non-event.

Another factor to consider is that the Release Candidates should be widely-publicized as stable, near final, with testing encouraged.

I wasn't particularly focussed on the CC release process, but my impression was there were a few mentions from official sources of the RC1, RC2, and without a bold proclamation of what's supposed to be working.

That made it hard for me to decide "when to jump in" and try out the release candidates. Our release notes need to ask people to test, and give them reasons to do so.

I would advocate that our Release Notes for RC1 &c have the form:

  • The LEDE Project provides a powerful and stable operating system replacement for routers and embedded devices. For more information, see https://lede-project.org/about
  • This is Release Candidate #1 (RC1) for our upcoming "LEDE XX.YY" release. It contains the following new features:
    • Feature A
    • Feature B
    • ... etc...
  • We believe the software is stable, and can be used widely, although still in test situations.
  • If no bugs were found, we would ship this version. (Of course, there probably will be bugs in this release candidate, so...)
  • We plan to make an additional N Release Candidate releases, and ship on or around ddMMMyyyy
  • No new devices (beyond those already in the ToH) will be added/supported in this release
  • Please help us test!
  • [And finally, no talk about the new development/trunk release. We don't want potential testers distracted by shiny new toys that won't appear for months.]
  • This is Release Candidate #1 (RC1) for our upcoming "LEDE XX.YY" release. It contains the following new features:
  • Feature A
  • Feature B
  • ... etc...

we need volunteers to dig through the git changelog to identify what features
have been added that are worthy of being listed. One problem with releases of
distros that combine a lot of upstream changes is that the most important thing
is either support for new devices (i.e. someone can't run the earlier version)
or just the fact that it includes newer version of a lot of software, and all
the new stuff that implies.

  • If no bugs were found, we would ship this version. (Of course, there probably will be bugs in this release candidate, so...)
  • We plan to make an additional N Release Candidate releases, and ship on or around ddMMMyyyy

I would replace this with something like:

If there are no bugs found in this rc, we plan to ship it as the release on or
around ddMMMyyyy.

If there are bugs found, we will ship an additional rc instead.

David Lang

Agree with everything that @dlang said. In fact the list should have up to a dozen big features, with a final bullet item pointing people to a longer, fine-grained list of all the changes/bug fixes/etc.

With this initial LEDE release, we might forgo the list of "new devices supported", since that's the universe of devices. But that probably should become a component of subsequent releases.