Hi, I have gigabit fiber (via Centurylink) which uses PPPoE. The provided router was easily able to get gigabit speeds, but was incredibly unstable. After doing some research I ended up buying a PCEngines apu4d4 and ran pfSense on it, and it ran very well but only gave me around 250Mbit speeds due to OpenBSD's PPPoE implementation, and Internet resources suggested switching to OpenWRT.
I have gotten OpenWRT working fine (version 21.02.3), but I'm still only getting 600Mbit. Which is a pretty good speed, but it's still not my full gigabit, and every now and then I do large file transfers where getting the full gigabit makes a big difference to me.
What performance tuning options do I have to try to improve things? Do I need to overclock the CPU, or are there some network parameters I might try? I'm just using the defaults from /etc/sysctl.d.
My interface setup is as follows:
br-lan bridges eth0/eth1/eth2
eth3.201 is an 802.1q VLAN with tag 201 (required by Centurylink)
pppoe-wan is connected to eth3.201
I'm using all the default MTUs. Also currently I'm only using the eth0 port for my local network (connected to a 24-port unmanaged switch), so if it would improve performance to bypass br-lan and just use eth0 directly that is something I'd be fine with.
[EDIT]: I just changed my LAN port to be eth0 instead of br-lan, and that got my performance up to 800Mbps. Nice. That last 200 is so tantalizing though.
Don't know exact details but shouldn't you set the mtu a little bit lower because of pppoe overhead?
May also try to enable packet steering?
BTW do you really want full bandwidth speed for a couple of clients 2 or 3 or do you want low bufferbloat with more clients and a stable connection with lower bandwidth as in 600~Mbit with SQM?
With the ISP-provided router, fast.com gives me consistent speeds of 980Mbps (which is the rated/advertised speed of the connection). With pfSense it was consistently 250. With OpenWRT it currently varies between 750-850.
htop is showing one core as being pegged at 100%, so yeah, that must be it. Dang. (But there's plenty of load on the other cores as well, 30-50% in general.)
Hm, now it's a different core that gets saturated, but the overall CPU load is still distributed the same across cores. I'd imagine that there's not a whole lot that can be done for the throughput of a single connection at a time.
I guess the real solution is probably to get a faster CPU, or to maybe overclock the one I have.
Oh well, 800Mbps is still pretty darn fast, and quite sufficient most of the time. Mostly I'm just annoyed at not getting the full bandwidth of my connection.
i have an APU2C4, and i have 1 Gbit finger in the nose, but my ISP is using DHCP which give better results compared to pppoe.
About overclocking, you can't do it. But i know you have to flash a recent firmware to enable the turbo 1.2ghz
I have already updated my firmware to the latest (4.16.0.4). Is there something I have to do in order to enable turbo mode after that? /proc/cpuinfo says my cores are only running at ~650MHz normally and ~950MHz under load.
And yeah I wish CenturyLink used DHCP instead of PPPoE, but they're stuck in the past; most of their Internet offerings are DSL and I assume that this makes their provisioning easier.
no, it's all managed by the cpu. you don't even see it at 1.2ghz because it's not reported to the OS. only benchmarks showed that the turbo was working.
PPPoE is a quite significant additional load on the CPU, which is already marginal for plain routing at 1 GBit/s (as you noticed, linux can achieve that, the xBSD drivers can't).
That's what I was afraid of. Maybe I could do something funky with the ISP-provided router and have it handle the PPPoE and then I just use the apu4d4 for the routing/DHCP/etc.
PPPoE: if your internet provider is using PPPoE protocol, and you have purchased a Fiber WAN over Gigabit, the APU2 cannot reach that 1 Gigabit in real life. More realistic is somewhere between 200 and 650 Mbit (the latter is the absolute maximum under real life condtiona). Reason: the PPPoE is single-threaded under *BSD, and the 1 Ghz cores in this SoC cannot handle that amount of traffic. PF and NAT can (and fornsure will!) decrease this value even further. There are no surprises, clock speed wins over core count in single threaded code. So this should set your expectations when your ISP is using PPPoE over that gigabit fiber connection!
I was thinking that maybe the ISP router would have a bridging mode I can put it into. I've had DSL ISPs in the past which provide that option, and that avoids the double-NAT/double-forwarding issue.
