Getting gigabit on a PCEngines apu4d4 and PPPoE

Hi, I have gigabit fiber (via Centurylink) which uses PPPoE. The provided router was easily able to get gigabit speeds, but was incredibly unstable. After doing some research I ended up buying a PCEngines apu4d4 and ran pfSense on it, and it ran very well but only gave me around 250Mbit speeds due to OpenBSD's PPPoE implementation, and Internet resources suggested switching to OpenWRT.

I have gotten OpenWRT working fine (version 21.02.3), but I'm still only getting 600Mbit. Which is a pretty good speed, but it's still not my full gigabit, and every now and then I do large file transfers where getting the full gigabit makes a big difference to me.

What performance tuning options do I have to try to improve things? Do I need to overclock the CPU, or are there some network parameters I might try? I'm just using the defaults from /etc/sysctl.d.

My interface setup is as follows:

br-lan bridges eth0/eth1/eth2
eth3.201 is an 802.1q VLAN with tag 201 (required by Centurylink)
pppoe-wan is connected to eth3.201

I'm using all the default MTUs. Also currently I'm only using the eth0 port for my local network (connected to a 24-port unmanaged switch), so if it would improve performance to bypass br-lan and just use eth0 directly that is something I'd be fine with.

[EDIT]: I just changed my LAN port to be eth0 instead of br-lan, and that got my performance up to 800Mbps. Nice. That last 200 is so tantalizing though. :slight_smile:


Fire up htop, see if you max out any of the CPU cores during your speed tests.

Internet speed test aren't very reliable, one minute you get half your max speed, one minute later you max it out.

Don't know exact details but shouldn't you set the mtu a little bit lower because of pppoe overhead?

May also try to enable packet steering?

BTW do you really want full bandwidth speed for a couple of clients 2 or 3 or do you want low bufferbloat with more clients and a stable connection with lower bandwidth as in 600~Mbit with SQM?

The defaults already take care of that; it's defaulting to 1492 on the PPPoE interface.

How do I enable packet steering?

Generally I only care about having 1 client getting the full gigabit throughput, rather than having multiple clients sharing the total throughput.

With the ISP-provided router, gives me consistent speeds of 980Mbps (which is the rated/advertised speed of the connection). With pfSense it was consistently 250. With OpenWRT it currently varies between 750-850.

htop is showing one core as being pegged at 100%, so yeah, that must be it. Dang. (But there's plenty of load on the other cores as well, 30-50% in general.)

Try installing, and enabling irqbalance.

You might also want to check the bios version of the apu4, newer ones enable cpu boost ( or something like that, read the beginning of Tips for getting cheap used x86-based firewall with full Gbit NAT (a PC Engines APU) if you are in the US ).


Hm, now it's a different core that gets saturated, but the overall CPU load is still distributed the same across cores. I'd imagine that there's not a whole lot that can be done for the throughput of a single connection at a time.

I guess the real solution is probably to get a faster CPU, or to maybe overclock the one I have.

Oh well, 800Mbps is still pretty darn fast, and quite sufficient most of the time. Mostly I'm just annoyed at not getting the full bandwidth of my connection. :slight_smile:

i have an APU2C4, and i have 1 Gbit finger in the nose, but my ISP is using DHCP which give better results compared to pppoe.
About overclocking, you can't do it. But i know you have to flash a recent firmware to enable the turbo 1.2ghz

1 Like

Network>Interfaces>Global Network Options

1 Like

I have already updated my firmware to the latest ( Is there something I have to do in order to enable turbo mode after that? /proc/cpuinfo says my cores are only running at ~650MHz normally and ~950MHz under load.

And yeah I wish CenturyLink used DHCP instead of PPPoE, but they're stuck in the past; most of their Internet offerings are DSL and I assume that this makes their provisioning easier.

I had an apu1d2 for a while, it was able to route 900mbit or so, in a lab, never used it live, with my ISP.

Yours has better NICs.
Same scenario as @shdf though, no PPPoE.

Thanks. That doesn't seem to have changed anything, unfortunately.

no, it's all managed by the cpu. you don't even see it at 1.2ghz because it's not reported to the OS. only benchmarks showed that the turbo was working.

1 Like

Force performance governor instead of ondemand for both cpu's 0/1?

PPPoE is a quite significant additional load on the CPU, which is already marginal for plain routing at 1 GBit/s (as you noticed, linux can achieve that, the xBSD drivers can't).

1 Like

Just tried that (on all four cores, not just policy0), no change.

That's what I was afraid of. Maybe I could do something funky with the ISP-provided router and have it handle the PPPoE and then I just use the apu4d4 for the routing/DHCP/etc.

Please read this thread, at least this comment :

  • PPPoE: if your internet provider is using PPPoE protocol, and you have purchased a Fiber WAN over Gigabit, the APU2 cannot reach that 1 Gigabit in real life. More realistic is somewhere between 200 and 650 Mbit (the latter is the absolute maximum under real life condtiona). Reason: the PPPoE is single-threaded under *BSD, and the 1 Ghz cores in this SoC cannot handle that amount of traffic. PF and NAT can (and fornsure will!) decrease this value even further. There are no surprises, clock speed wins over core count in single threaded code. So this should set your expectations when your ISP is using PPPoE over that gigabit fiber connection!
1 Like

You really want to terminate the PPPoE Session on your OpenWrt router, mostly for these reasons:

  • having the WAN IP and the NAT on your OpenWrt router, avoiding double NAT
  • ease of use (not having to configure port-forwardings etc. twice)
  • security

I was thinking that maybe the ISP router would have a bridging mode I can put it into. I've had DSL ISPs in the past which provide that option, and that avoids the double-NAT/double-forwarding issue.