Yes, offloading the shaper onto hardware would probably help. MQPRIO requires however that the actual hardware/driver support the offload, for supported hardware however it would be interesting to see whether that would allow to reduce the shaping cost.
But that would need some testing and probably something like BQL on the driver so that an unlimited fq_codel/cake will not inject too large data bursts.
Could you have a look at this documentation: Mediatek QoS. They are comparing their implementation with HTB qdisc. I am sure I could make it work as a limited hardware HTB version (limited because of the number of queues) otherwise for sure as MQPRIO. Not using their "qdma" command, but using "tc" as it should be used.
Similar could be done on with the Atheros 8327 and very likely other targets as well, but I don't have documentation for those.
I would naively say the easiest approach to test this would be to set up a singular rate limit and attach cake in unlimited mode in front of that. That might already work but depends on giving enough push back to cake so cake still queues most of the packets and hence maintains the core of the queue.
To test is not that simple. I have a very very old linux (3.4.113) running on a MT7621 target which has all those proprietary mediatek code included. So I can test their "qdma" commands in combination with iptables as in their examples (from the document in the link above). BUT: that build does not have "tc". The "qdma" user command is using IOCTL to communicate with the raeth driver, not the now in use mtk_eth and mt7530 dsa drivers.
This HTB offload was added beginning of this year (kernel 5.12). Our router hardware will be more restrictive on the amount of queue we can have (which I guess translates in the total nodes, not just leaves?), but 8 queues should be enough to have a basic setup. As far as I understood the code, if more are needed those could still be done in software. MQPRIO is already present in the current kernel version that we are using right now.
I think I should have some proof of concept ready by tommorrow, using the old mediatek code and adding a small helper kernel mod to accept the IOCTL commands. The mtk_eth_soc needs some patching to add Queue ID to the qdma transmit code.
Going over the simple.qos script, for ingress only 1 device is used for the HTB qdisc. I assume this is "old" and should now be replaced by the bridge-device (br-lan) or probably even by every separate LAN0..3 port since we moved to a DSA system? If I would like to use hardware to do rate-limiting, it will only work on egress. So for ingress I should use "egress LAN?" correct? Anything other than wired ethernet ports will be left out? (done in software)
Another thought for OpenWrt/CeroWrt II: No-Configuration Networking
I desperately want to make it easy for a newcomer to install and configure OpenWrt. But it's still too hard - lots of little bumps in the road decrease the probability of success. It's getting better, but there remains the all-or-nothing dance on the tightrope - replacing or flashing your ISP's equipment.
So I want to focus on the use case where someone buys a new router for OpenWrt and plugs it into their ISP's gear. (Yes, they'd have double-NAT. But lots of people have double-NAT already, and it works fine for the vast majority of situations.)
This makes the instructions simple: First go to the Table of Hardware and select a suitable router. (Or have recommendations for modest price devices that work fine.)
Then go to a (single) page on the OpenWrt wiki that basically says: Use an Ethernet to connect your new router. Flash it with the OpenWrt "factory" image (from https://firmware-selector.openwrt.org). Enable Wi-Fi. Connect the new router's WAN port to the ISP gear, and connect to 192.168.1.1...
OH DRAT! What if the ISP gear is using 192.168.1.0/24? Why do we tell people to use 192.168.1.1, anyway? (It's the 21st century...) Could we simply use a mDNS name instead? Can't we make this simpler?
Here's my desire:
a) The router could use a default algorithm to assign its LAN interface address to be from a different subnet from the ISP gear.
b) OpenWrt can then use mDNS naming for its LuCI interface. We tell people: Just type OpenWrt.lan in your browser (or ssh root@OpenWrt.lan)
c) Newcomers are on the air! Profit!
Does a facility already exist that would accomplish this? Could it be incorporated into the default behavior? Thanks.
2 Likes
Getting a "zero config" to work would cost a lot of valuable flash space on most devices. Most of which would be used only "once" during setup.
Maybe we should just accept that OpenWrt is not for everyone. Similar to try and switch a Windows user to a Linux desktop: even with all the eye-candy that is available in a modern distro, most people will still find it "hard to use" compared to what they know. (Even windows to MacOs is a big step for a lot of people).
Adding mDNS for Openwrt.lan might be an option. Changing from 192.168.1.1 to something less often used is also a "cheap and easy" to implement idea. Moving to 10.0.0.1 could be an idea: most routers/modems will use something in the 192.168.x.1 range.
@dtaht and @moeller0 I noticed that the mvebu target just added patches for MQPRIO offload to hardware. Maybe not the ideal replacement (and I did see the Qosify thread), but given that a lot of our hardware is "low end": how much would having MQPRIO with FQ_CODEL help, or asked the other way around: how much worse would it be compared to CAKE but with the additional help from some hardware offloading. If we would all run OpenWrt on some x86 type of device we wouldn't have any problems, but most routers are "low-end" while FTTH (or cable) is getting 1Gbps and even higher.
Here is an (old 2012) comparison between MQPRIO and HTB in terms of latency (obviously the document is biased towards MQPRIO, since that's what he was trying to implement at that point in time)
-
I would like to explore offloads on more recent hardware than this. I note that there's a clever and underused feature in BQL that - if you can set an interrupt rate - lets you do the bandwidth limit without HTB. I keep hoping more hardware will do that (some intel nics do). I don't like how bursty htb is.
-
I saw those patches go by for the mqprio offload also. i just don't know. Given how finicy the nss offloads have been, I'd rather focus on doing 1gbit with some hardware assistance other than offloading the whole forwarding plane, on some hardware capable of it. In that case, at least, it does not appear to me that the separate queues can borrow bandwidth from each other, and that seems like a lose to me.
-
I think the single most needed feature is to shape inbound, and the only way I can think of that is to produce a better multi-core ingress path, and a cake-mq that shares the bandwidth parameter between itself. Somehow. But! I do not want to shut down speculation as to what we could do based on my opinions, (keep the ideas flowing!!!), and i'd rather find better hardware to start with.
1 Like
It already does this, a ULA ipv6 address!!
this already works doesn't it? (Though to be fair, I think it should ONLY return the Ipv6 ULA in the lookup of openwrt.lan and then avoid all issues with overlapping ipv4 subnets entirely, just move administration 100% to ipv6)
Clever idea! I like it! (And you're right that mDNS works fine with the IPv6 ULA.)
But that only solves half the problem. Other devices on the LAN still need to have an IPv4 address, and things come to tears if the ISP gear and the router both try to assign from 192.168.1.0/24...
@drbrains - I intentionally said "no-configuration" networking (not "Zeroconf™") for exactly the reason you cite - there's a lot of machinery involved in that spec.
In both cases, I'm advocating for letting OpenWrt configure the default LAN subnet to be distinct from the WAN subnet. 192.168.x.x or 172.30.x.x or 10.x.x.x subnets are all fine with me, as long as the choice is different from the WAN port. Thanks!
ipv4 meh, I ran my LAN ipv6 only for a year already back in 2017 or so, worked just fine 
But yeah, there are too many things that really want an ipv4. I suggest just like on firstboot the way it chooses a ULA, it should choose two random bytes and construct a 10.x.y.0/24 subnet. The probability of a collision is small enough.
1 Like
From the AR8327 data sheet:
2.15 Rate Limiting
In triple-play applications, the switch may need to limit the rate for all frames but continue to maintain QoS policy. The AR8327 supports ingress and egress rate limiting requirements on a per-port basis by configuring the Port Rate Limit register.
The AR8327 can also support per port per Queue based egress rate limiting. Ingress rate limit can include or exclude the consideration of Management frames and registered multicast frames, while Egress rate limit can be configured to take management frames into account.
The AR8327 can limit all frames and support rate limits from 32 Kbps to 1 Gbps at 32 Kbps granularity. The AR8327 also supports 128 counters for ACL rule based rate limits.
Mediatek MT762x can do similar. Not as precise to 32 Kbps, but should still be workable considering that we set like 85-90% of the line-rate for most SQM scripts.
What about putting cake below say a 3 class qfq and then hash flows into the three classes, each cake with 1/3 the total bandwidth. Sure no single stream could get the whole magilla but maybe that's less important.
I think all qdiscs on a given interface share the same lock, so the single qfq instance will make this behave pretty much as a single cake instance....
1 Like
This is getting dangerously close to an actual proposal that requires an actual decision. How do we collect the "political will" to make an actual change for 22.0x? Thanks.
I probably would try to avoid the 10.0.0.0/8 range as that seems to be used relatively often by ISPs for CG-NATing, and then having the homenetwork on one of the 192.168.0.0/16 gives a clear demarcation in traceroutes and is generally easier to discern internal from external addresses.
Personally I like @dtaht's 192.168.42.0/24 and still use it in my home network. Or is the argument that a /24 is too small?
this is supposed to be what 100.64.0.0/10 is for https://en.wikipedia.org/wiki/Reserved_IP_addresses
But even if the ISP uses the 10.0.0.0/8 range, the likelihood of a collision is still small, with 256*256 = 65536 possibilities for a /24 subnet. It seems unlikely that a given subscriber would "see" more than 2 or 3 subnets and it seems unlikely that a subscriber actually wants to reach a piece of ISP kit behind the CGNAT, that should be mostly a few routers right?
Don't know. Who are the most important people to convince?
Also is the reason you want mDNS because then the client doesn't need to be using the router for DNS? Or is it ok for the router to advertise openwrt.lan through dnsmasq? This would avoid having to have mDNS servers by default.
So I think the proposal for me is:
-
resolve openwrt.lan only to the ULA address of the router (probably by putting an entry in /etc/hosts ?), not any ipv4 address (ensuring extremely high reliability ability to connect to the router regardless of other ipv4 overlapping subnets)
-
on first boot read two random bytes x and y from /dev/urandom and form a 10.x.y.0/24 subnet on the default LAN.
-
Rewrite all documentation to encourage people to navigate to http://openwrt.lan by name to configure their device. And if they have actively disabled ipv6 on their computers, to re-enable it as it's required for administration. (note that every operating system I know of has worked out of the box with ipv6 since at least 2010 or so, with exception maybe of some gaming consoles like a PS4 but you probably shouldn't be administrating your router from a gaming console)
-
Note that I'm not suggesting to stop responding to ipv4 or to get rid of ipv4 on the router, just to by default only have openwrt.lan advertise its ipv6
My reading is that the 100. range was added post hoc after things got too tight in the already allotted ranges, point is I see 10.x addresses on the wan side of traces of users behind CG-NAT.
And I am not arguing about collisions (where I agree with you) but really about keeping IP addresses easier to interpret when trouble shooting. Think somebody posting a traceroute here in the forum... but admittedly that is a weak convenience argument.
yeah but 192.168.x.y appears very often on the WAN side of home routers, maybe more so, especially anyone putting their own router behind an ISP all-in-one device, and I think much more often a collision occurs since x is usually 0 or 1 or 100 or sometimes 8 or ...
we have 172.16.0.0/12 but I find that range really awkward to work with because it's on a nibble boundary but ipv4 addresses are in decimal so it always requires pulling out a calculator or looking it up to figure out what is allowed in the 2nd octet. I think for the "uninitiated" this is even more confusing. But it has some advantages so I could probably be convinced.
192.168.x.0/24 just doesn't have a big enough available space to reliably avoid collisions using a random selection of x...
So that puts us at 10.x.y.0/24 as the best scenario for using a random selection and most likely avoiding collision. Also the number 10 is easy for people to remember as compared to 192.168 or 172 (which I always have to look up). This has big advantages for telling people what to do over the phone
Me: "now type 192.168 "
Them: "hold on hold on... one... nine... what was the rest?"
Me: "type openwrt.lan"
Them: "openwrt.lan ok got it"