Load balancing SMP/IRQs

The R4S thread has been discussing SMP and IRQs for better load balancing and i was looking to improve the wiki details on this as it is applicable to many devices.

The wiki entry for it is here - https://openwrt.org/toh/friendlyarm/nanopi_r4s_v1#cpu_management

However after a few searches and some hunting i was unable to find a existing wiki entry for this.

So i created the following page and am populating it with a more generalised set of explanations and details to enable others to use on other routers.

@tmomas I hope that is a reasonable place for the topic?

3 Likes

This, btw, indicates that heterogeneous SoCs are not automatically a good fit for generic routers... Setting affinities by hand is IMHO not something that should be required (let alone push all essential router duties onto two of 6 cores), but a direct consequence of big-little designs. The raspberry pi4B for all its warts, does not have that specific issue (nor does the R5S) in that it sports just one type of CPU cores.

4 Likes

maybe just better defaults required for some routers?

I'd assumed that most routers would be set to use all cores automatically. As i found out with my BT Hub 5 this is obviously not the case, and the multitude of threads I've seen shows some are in need of some tweaking.

But that means analysing and coming up with best defaults across the entire range that OpenWrt supports. Thats not something i'd like to have to do. Thus my reasoning for doing a manual tweak page and referring to IRQ Balance for a more automated catch all.

3 Likes

But such defaults are considerably simpler for equal cores... depending on what your router does putting interrupt processing on the big might be worse than putting it on the little cores...

Yes, the default is still geared towards low storage single core SoCs, were even adding irqbalance to the default builds is considered wasteful. Not a call I would make nowadays (not that I am asked), but certainly one that makes sense given OpenWrt's history, no?

+1; these are two orthogonal things and given that "better defaults" seem rather more involved (if possible at all) "arming" users to make/test changes themselves is a great way forward!

2 Likes

Having a multi core big/little design isn't a problem. Having the OS being aware of it and understanding that is. Scheduling can be hard. Intel's new chips with Big/Little have similar issues. Both windows and linux have had to have patches to fix poor performance due to imbalanced cores. A hybrid solution is always a trade off between power/efficiency.

Apple have had better luck with their A series version as well due to fully controlled systems.

Yes it is easier to balance between same type cores, but it can make sense to have low speed cores for basic background processes and only spin up highspeed cores to deal with load. Its certainly more power efficient than having x big type cores permanently running. (aka the smart phone revolution)

Eventually smarter scheduling engines will be commonplace and manual tweaks like this wont be required. Till then. Here's the wiki :slight_smile:

IRQ Balance may handle the basics nicely but if you are doing more than just routing (docker containers for instance) then you will need to do more tweaking. It may even do harm on some routers by shifting pre balanced cores where each core handles a different eth port to pushing all eth onto one core instead. I can certainly understand OpenWrt's reluctance to have that as an "out of box issue".

With todays modern world... a one size fits all never works. Thus why having choice to change defaults is always good.

3 Likes

Well, it clearly is the case that any OS will have an easier time with an equal core set-up... proper scheduling on big-little systems requires knowledge about a tasks future processing requirements and short of an oracle that knowledge is hard to come by generically, no?

Yes, but for doing that you need to divine for an arbitrary process whether it is a background task or not... sure you can employ heuristics, but these are almost guaranteed to have mis classifications.

Nope, smartphones developed on equal cores for a long time, even for ARM big+little came relatively late to the party and smartphone OSes the walled gardens they are are likely the best case for such a design as the OS has tight control over lots of things and mostly will know which processes are "background"... and still...

And yet a "universal design" is still a valid goal to strive for... "one size fits all" was always intended as a slur....

+1; but that is orthogonal whether it is a good idea to increase scheduling complexity by making cores significantly different. That said, it is clear that ship has sailed independent of what I think of of it :wink:

2 Likes

Ok. I've finally finished and would appreciate a lookover if you dont mind?

@moeller0 @walmartshopper @xShARkx

@tmomas i hope that is in right place and fits with the wiki?

(edit - fixed my titles. i'd left my introduction at top and thus wrongly titled it)

3 Likes

@tmomas ty for fixes. Also do you have a table editor for the wiki? i had to do that damn cpu table manually... I felt like i'm hacking html in notepad back when it was html3!!!

1 Like

5 posts were split to a new topic: OpenWrt wiki: Improving table editing

The exclamation box about restarting SQM has a minor grammar mistake but other than that looks good.

2 Likes

tweaked the SQM notice. should be better now.

1 Like

Seems good, from the quick look that i took :+1:

1 Like

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.