SQM - BB vs LEDE - major diff in performance

Installing and Using OpenWrt
10

There are a large number of "network appliances" out of China these days that use on the order of 10-15 watts, are compact, no moving parts, and will blow even the ARM routers out of the water. Of course they have no WiFi, but they also have no gotchas. You can't brick them, they will do all the routing and shaping you ever want, they will have 2-4 gigabytes of RAM, and 20-60 Gigs of SSD storage, and they will still have CPU cycles left for stuff like an NFS server or the like. Plus, even with a simple AP to provide wifi, they tend to cost about the same as high end routers.

A few years back, you'd be looking at a full x86 mini tower or something and it'd be 60 to 150 watts, and cost you $600. Today that's not the case, and with consumer bandwidth increasingly large and cost of x86 plummeting ... the case for the consumer router (as opposed to x86 appliance router and consumer AP) is quickly disappearing.

At the low end of cost for example there's this:

https://www.amazon.com/XCY-Celeron-2-41GHz-Supporting-Ethernet/dp/B0719L1VFK

and a simple AP:

https://www.amazon.com/TP-Link-AC1200-Wireless-Wi-Fi-Access/dp/B01LLAK1UG

compared to say an ARM based router like: https://www.amazon.com/Wireless-StreamBoost-Beamforming-Antennas-NBG6817/dp/B01I4223HS

Price might be a little higher, but performance of the router appliance, and support for the hardware in the kernel and un-brickability, and not having to deal with factory lock-outs and soforth is probably worth the ~ $50 or whatever for most people who have 200 mbit connections.

Since an AP isn't necessarily a security critical thing you may even be fine with sticking with the factory firmware, plus the mounting options are better so you can probably get better signal.

All this tells me that in my opinion, LEDE should focus on getting itself onto more APs and also focus on some kind of software for controlling multiple APs from a single configuration.

EDIT: another thing to consider is the consolidation options. People who get 200mbit+ connections probably want a NAS or media server or something or other like that, and so buying a slightly higher horsepower x86 router than that one and combining the two functions will give better performance and better cost and electricity savings than buying an ARM router and a separate NAS box.

11

As a data point, my closet has a J1900 based mobo mini itx router/NAS with a USB enclosure running 4 spinning drives, and a 24 port smart switch and the customer router for my ATT fiber connection all on a UPS that report 72 watts total. That costs me about $75/year in electricity. Each drive is rated between 8 and 15 watts, so about 32 watts of that 72 is probably the drives. The router box itself is likely around 10 to 20 watts.

The zyxel armor power supply is rated 12 volts 3.5A on wikidevi and is probably over-provisioned but still we're talking... 20 watts or something.

12

Also looks quite interesting. I haven't used it but it's a good price point for a combined Router/NAS and has 2 NICs (no idea if they're Intel) and specs say it supports Ubuntu which also probably means Debian.

13

Ah, yes, it performs a bit better than just the freq change would suggest, but agreed, it's not a huge step up.

I believe you are correct, the LEDE version of the custom build will use fast-path if no qdisc is set on a given interface. But the old BB version was pre-fastpath IIRC.
We also tested the 17.01.4 normal build and it had the same QoS-on results on the C7.

I agree CPU is quite possibly the limiting factor here, but the huge delta relative to the BB based tests on same model HW made us suspect something else.
Idle did not raise alarms but we were focused on sirq, so will re-test and capture the exact values.

Understood, and to validate, we ran DSLreports tests from a wired PC, and results correlated with the netperf local tests, so it does not seem the extra in-router process was the limiting factor.

I appreciate all the feedback on this topic.

14

Very Interesting stuff, pricing and form factors have indeed improved a lot. Thanks for sharing that.

The AP linked, and others in that family do make great AP's. For one, because unlike Ubiquity, they use standard POE and one can directly mange them by logging into their local web UI; no need for a central console if all you have is one or two APs on factory firmware. I have two myself and like them.

Cool, and up to what speed can you shape with QoS?

15

I seem to recall that @gwlim was a proponent of overclocking router CPUs (and in a sane fashion including stability tests); maybe the BB build you used was running the CPU at > 560 MHz?

So, I initially was distracted by the fact that idle goes down while the other go up under load, so my casual observations did not finger idle to be special until I started to pay attention to the details an realized that 0% idle actually does translate into the router has no CPU cycles to spare. That in itself is not so bad, but for that fact that "the router has actually far fewer CPU cycles available than it desires" has the exact same 0% idle "phenotype".

Ah, that is valuable information, in my limited tests I think saw some effect of running netperf on the router itself, but I did not actually research that any deeper after realizing that this was not testing what I intended to test, so thanks for the additional data point here.

I am somewhat sorry, that I will not be able to really help, that is maybe https://forum.openwrt.org/t/overclocking-router-devices/1298 has some pointer on how to overclock your wdr3600 as that might give you just enough additional cpu cycles to make sqm work at your bandwidth.

Best Regards

16

So I mostly agree with you, and many thanks for the information about recent prices and power consumption of x86, but with the consolidation argument I have some discomfort. My gut-feeling is that it probably is a good idea to have the entry point to one' own network not do too much important stuff in addition to its core duties, I simply think that not putting all my eggs into one, exposed to the internet, basket is all that cautious (especially since I tend to opt for convenience over strict security quite often, so I am not ready to vouch for the security of my main router).

17

I have 3 NICS bonded into a smart switch, I shape a gigabit fiber with a custom HFSC setup and fq_codel. It's a little hard to measure because in fact most speed tests can't measure a gigabit connection, and also because I use a squid proxy and an ipv6 only LAN with Tayga for NAT64... but I regularly get 500 mbit + from dslreports with minimal bufferbloat. Because of the proxy the user CPU usage can be high. If I just had it routing and not doing 64 translation it would shape the full gigabit fine (it did that before I changed things for various reasons).

My concern isn't really raw speed (because the gigabit is kinda fake anyway, it depends on how much the neighborhood is really using) but I really want reasonably high speeds while remaining absolutely perfect with up to 3 or 4 VOIP calls at once even while multiple devices are streaming videos or I'm downloading packages to upgrade my Debian laptop or whatnot. SO far, so good.

18

This is a good point. I do have some consolidation, but I also run additional firewalls on my laptop, desktop, HP printer, and I set randomly generated passwords on all my management interfaces, and have offsite backups of all my files in a safe deposit box I update every 3 months or so and an onsite backup I update weekly. My feeling is that assuming the router is the security moat that none shall pass is probably not the best security scenario.

To each their own of course, but there's a good argument to be made that in the future x86 boxes will continue to get cheaper or stay similar priced with more power, and so even unconsolidated, it starts to make good sense.

19

Ah, sorry I should have said "all my eggs" as I was really only talking about my personal comfort knowing the security compromises I accepts for convenience (since my file server offers less services it is easier to keep the number of compromises low :wink: ). I fully trust that there are folk, you included, who are very much on top of keeping things secure.

And I fully subscribe to that idea, and I add typically a moat was followed by a wall so security is certainly a bit like an onion.

I guess that will also show in OTC routers. For example I fully expect lantiq SoCs to be x86 based any time soon (unless they already ship these); noe grant you it will be atoms and not the real beefy x86 cores but still they should run circles around say the OP's almost 10years? old 560MHz mips cpu...

20

Just throwing my 2c in the ring, this test should be repeated with a CC build.
From what we’ve observed at Gargoyle there has been a drop in routing speed under normal operation between the large releases (i.e. kernel jumps). The biggest one was BB->CC when a con track caching option was dropped (and maybe patched back in?? Can’t remember).

In my experience this datapoint will be somewhere between the two existing tests.

21

from 2015 seems to support that idea. Basically those XCY boxes are high end OTC routers. If they just installed IPFire on them out of the box, they'd be kinda what you're talking about :wink:

So if your argument is that in the future you will get a grunty little workhorse box and it will have an Celeron or Atom processor and do a lot for very little money... then we're basically saying the same thing, except I'm saying that if you pay a little more now you can already have it, and it's worth it compared to much of what's actually available in the consumer router space.

The other aspect though, is that for LEDE enthusiasts who are flashing their own software... there is a shrinking price gap between a good consumer router, and a full on PC like the XCY or whatever. And that shrunken gap has lots of implications because much of the special purpose embedded stuff... MTD layouts and soldering your own serial ports and bootloaders and getting around region codes while flashing files and whatnot are just a pain in the ass that goes away when you have standard PC hardware. If the difference in price is $100 for a fancy new x86 Lantiq with 256MB RAM and 1G flash, and $200 for a nice standard PC with 4Gig RAM and 128 Gig removable flash or something... I know how I'm going to spend my money so I can save a lot of hassle and time.

22

You are absolutely right, we hit idle = zero at around 100Mbps with QoS on, so any headroom above that point is probably really pushing it.

Too bad about the performance step down in LEDE, but out of CPU is out of CPU. Time to look at the box linked above. I kind of like having something totally focused on routing/firewall duties and no WiFi (already have the aforementioned TP-Link EAP's).

Thanks again for all the feedback.

23

Ah, good point, my untold assumption was I need a vdsl modem which I would prefer to be integral part of my router and then the options are extremely limited (well there is lantiq/intel or nothing).
I currently use a lantiq device using lede as bridged modem in front of a ar71xx device as the main router, and just because the anemic CPU in the lantiq device is already exhausted by concurrently running the vdsl modem, nat, pppoe, firewall and wifi; so I am looking forward to a lantiq device with a somewhat more powerful cpu. Without that constraint, I agree with your assessment

24

Exactly my thoughts. Was looking at pricy OTC routers with multiple cores and all that, but boxes like the one you linked seem tailor made for the typical LEDE crowd.
However, it is great that LEDE supports a wide variety of OTC routers, as some can not swing the $200.

25

Yes, I think those boxes are perfect for the typical LEDE crowd. Running LEDE itself... well maybe or maybe not depending. IPFire or just a raw Debian or Arch or something might make sense when the embedded small-storage-space restrictions are no longer there.

But, when it comes to Access Points, this I think is where LEDE could shine in the future. Because it makes sense to me that you'd still like more flexibility and custom services and security etc together with specialized RF hardware and antennas and PoE and soforth that you're not going to replace with a custom PC kit.

So I'd really love to see more support for TP-Link APs and the like to compliment the transition to more people using grunty little 4 NIC mini Linux PCs as their dedicated routers.

1 Like
26

The problem with eliminating the router/ap in favor of a full blown PC is
getting the wan connection (cable model or dsl) interface.

27

But that is similar for LEDE on non-x86 hardware, only lantiq XDSL-modem chips are supported, and as far as I can tell there are exactly 0 LEDE supported DOCSIS modem chips. So more often than not one already needs to bridge a proper "modem" (I know, these do not exist anymore these are all routers nowadays, but one can ask them to play dumb :wink: ). In that case x86 becomes an option again...

1 Like
28

If you are absolutely certain you are able to attain far higher speeds on standard, but older builds, you can do a Git bisect to try and find which commit introduced the regression and file a bug report.

Just compile an old version, without any optimizations, and verify if the speed is indeed much higher. If so, start the bisect process until you find the commit that broke the speed.

29

There has been approximately 6000 commits since 15.05. A bisect should require to compile and test approximately 13 versions, that seems doable.