Reasons for frequent major kernel upgrades

Why is kernel upgraded to a newer version despite the old ones still being supported for years?

Upgrading v19.07 to v21.02 made my router reboot frequently and drop packets, i haven't done any triage, but kernel is most likely to blame?
Not a ram/rom issue as even with custom image that takes 1.4mb rom and 10mb ram (df and free)

Kernel 4.4 way back from v17.01 is still supported.
Kernel 5.4 supported until the end of 2025 but is already being replaced.

Concerns:

  • Stability - out of hundreds of supported devices some could become unstable after a kernel upgrade and QA is costly, not even OEM does that, latest image from my router OEM from the end of 2019 ships kernel 2.6, seems patching vulnerabilities is cheaper than migrating a device to a whole new kernel

  • Resources - if there are still years of support from given kernel release, why spend resources on porting a new version instead of working on more user-facing features? i get kernel might enable newer hardware to work or even fix some bugs, but the reverse is also true and requires a lot of QA to be confident its good.

What's the reasoning behind those short cycles? Thank you.

1 Like

It's a volunteer project and most want to stay on top with upstream rather than backporting various things that add "unnecessary" workload. Feel free to fork and maintain a legacy version :slight_smile:

3 Likes

This is a known global problem that has been risen many times on government level that these oem firmwares are the part of the growing DDoS problem.
I would say the manufacturers don't give a s…it about cyber security, they do only save money by reusing old crap over and over again, don’t expect any security holes fixed in these firmwares.

1 Like

rather than backporting

maintain a legacy version

Upstream supported kernels are not legacy and receive security updates.

Volunteer project and being free to choose what to work on is implied, not being questioned, I ask for the reasoning behind certain actions.

We can always do better and discussion is a part of that.

Feel free to fork

It makes more sense to make fork to run a newer kernel as opposed to an older one unless arguments are provided.

Big if true, I thought that if they push an image, they have patched known vulnerabilities at the time of release.

Edit: that example was an extereme, kernels 4.4, 4.10, 4.14 etc. are still supported upstream.

Why would they do that?
It is to expensive to have engineers sitting around doing this on home routers in a world where the only thing that counts is mass production at minimum cost and mass sale at maximum price and that result in biggest profit.

2 Likes

Actually no, it is pretty common.

the only thing that counts is mass production at minimum cost and mass sale at maximum price and that result in biggest profit.

You've brought a great point - the role of OpenWRT (even if coincidental) is to provide competition and increase customer level of knowledge of what matters, because those OEM's don't do things because they are evil, just because they optimize for maximum output of what the consumer demands or accepts. So if openwrt userbase is bigger, easier to get into and so on, then consumers won't buy into lackluster solutions.

1 Like

It was not an extreme that the action is not common, but that the kernels i talk about are still supported.

https://www.kernel.org/category/releases.html
4.4 has only about 2,5months left.
From 19.07 to 21.02 we didn’t have any release. It was supposed to be a release in 2020 but it didn’t happened.

So a kernel we have active need to be alive for up to 4years if we include master dev, branch and the parallel running of next branch until EOL.

You are missing the point again, those are just examples. It could be 4.4 -> 5.10 or whatever, just decrease the number of bumps of major kernel versions as it serves no purpose (no arguments so far).

The reason for this as you ask in the first post is that 5.4 is to buggy but it was implemented in master right after 19.07 (to implement DSA as far as I know) was released and 21.02 had to try its wings sooner or later.

But 5.10 is only in master for now so you shouldn’t need to think about this at this point?

But it is hard to implement 5.10 at 2019 when it isn’t invented until december 2020.

4.4 in v17.01 -> 4.9 in v18.06 -> 4.14 in v19.07 -> 5.4 in v21.02
which could be say
4.4 -> 4.4 -> 4.4 -> 5.4
with more confidence that everything works for most and more free time to improve other aspects

I do not believe, that the manufacturers are back porting and patching security vulnerabilities to the 2.6 kernel. The changes since 2.6 kernel were too complex. I back ported once an uvc video driver to the 3.14 kernel on android an it was not really easy to make the driver work. It took me about 4 full weeks of source changing and debugging until I succeeded and made the driver work. Easier is to bump the kernel revision than back porting in source code and wondering why the hell the thing is not working at all. This is not so easy on Android and I found it much more organized in Openwrt. When you have control over source code you can do nearly everything and the only limiting thing is the hardware you have. In contrary - Android with its half open source half proprietary system is limited in such a way that even most linux kernel drivers working out of the box, are not working on Android and a single hardware change results in a failing condition where you have to recompile the kernel and the system from source when it is available and in most cases you are limited to a single not maintained source left alone by the hardware vendor. That is the reason why Android phones will receive few updates and then you are forced to upgrade your hardware. With Openwrt provided you have enough storage and RAM you are not limited in such condition. You can run 8 years old router with the latest software even then when your hardware vendor drops the support for the device.

1 Like

But it seems that most users actually report better functional routers with 21.02 than they had with any of the kernel 4 firmwares.

So, what is your actual fault in your router to begin with that isn’t working with kernel 5.4 that worked better before?

I've browsed bug tracker and there are plenty bugs about hardware problems occuring from updates.
Mine reboots frequently and drops packets, but it'd be better discussed in another post or a bug report instead.
This thread is to ponder about the idea of frequent major kernel upgrades where they could possibly be avoided.

I have read through this thread and I don't understand what is your problem with kernel updates?
This is a full volunteer project and the only way to make it sustainable is to upstream as much as possible and use that.
And to do that you cant be 5 years behind the mainline kernel and carry even more backports than currently, it's just not viable.
Just look at the amount of backports that we currently have just because we are on 5.4 and 5.10 instead of 5.15, and even worse you get to a point where to backport one thing you need it means needing to backport five more.
You are using more time to backport things than actually develop and that is just wasting already really limited time for development.

So what is wrong with moving from LTS to LTS, it happens usually once a year.
And even that is really slow for active development.

5 Likes

Finally we have a trade-off

Shorter kernel upgrade cycle:

  • new features
  • new hardware
  • potentially fixed bugs & improved operation
  • potentially new bugs introduced & stability might be reduced due to more changes occuring, some of which harming the operation
  • all sorts of resources spent on the upgrade happening

Longer kernel upgrade cycle:

  • new features would still be introcuded, just at a later time
  • new hardware would have to wait
  • potentially more stable due to more time on one release and less changes occuring
  • due to lower number of upgrades, some time of developers would be freed to pursue other areas of the project

I still think that you don't understand how projects like this work.
People volunteer their time so they get to decide on what would they like to do, sometimes companies will contribute support for their product but they again depend on drivers and the rest of the infrastructure that is effectively donated by developers.

So reducing kernel updates will just push away developers nothing else.
Developers usually have their field of interest and you really cant force them to work on something else

4 Likes

People volunteer their time so they get to decide on what would they like to do

More power to them, i'm just trying to influence the decision to be more informed as to what is the trade-off between certain decisions, that is how things grow.