Correct MTU for Freedom Internet (NL)

[huj3r]

Hi there

I have been struggling for a while now to figure out if my configuration is correct. I use Freedom Internet (glass fiber provider is either Cambrium or Fiber Crew) and I am reaching max speed of ~560MB/s (using Ookla speedtest).

The configuration needed for this provider should be the same as KPN; this is what I currently use:

config interface 'loopback'
	option device 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option ula_prefix '***:***:***::/48'

config device
	option name 'br-lan'
	option type 'bridge'
	list ports 'eth1.1'
	option igmp_snooping '1'

config interface 'lan'
	option device 'br-lan'
	option proto 'static'
	option netmask '255.255.255.0'
	option ip6assign '60'
	option ipaddr '192.168.1.1'
	list dns '8.8.4.4'
	list dns '1.1.1.1'

config interface 'wan'
	option proto 'pppoe'
	option ipv6 'auto'
	option device 'eth0.6'
	option username 'user@freedom.nl'
	option password '1234'
	option peerdns '0'
	list dns '1.1.1.1'
	list dns '8.8.4.4'
	list dns_search 'example.org'

config switch
	option name 'switch0'
	option reset '1'
	option enable_vlan '1'

config switch_vlan
	option device 'switch0'
	option vlan '1'
	option ports '6t 4 3 2 1'
	option vid '1'

config switch_vlan
	option device 'switch0'
	option vlan '2'
	option vid '6'
	option ports '0t 5t'

config device
	option name 'eth0.6'
	option type '8021q'
	option ifname 'eth0'
	option vid '6'
	option macaddr '***'

config route
	option interface 'lan'
	option target '192.168.2.0/24'
	option gateway '192.168.1.50'

config device
	option name 'eth0'

This is the most stable configuration I can use but I am not reaching 1GB; I have tried to change MTU with network.wan.mtu='1500' and also changing the eth0 device MTU and enabling MSS clamping on the firewall zone, but that does not increase the maximum download bandwidth by much and instead causes issues (broken connections, buffering when using some apps).

My conclusion is that I cannot use higher MTU with this provider; did anyone else have success with it?

Thanks!

[frollic]

what router device are you using ?
even though slightly outdated, the new AX SoCs could probably work for you too.

So you have 500Mbps-1Gbps fiber and need a router READ THIS FIRST

It's such a frequently asked question over the last few years that we really need a post we can point people to. So your cable company rolled out 500Mbps or 1Gbps download speeds, or you have 1Gbps symmetric fiber (GPON/EPON) from ATT or Centurylink or Orange or whoever. You realize your old all in one router from 2009 is not up to the task of handling this, but hey by now stuff should be cheap and available that will handle your new fiber connection right? So you want advice about a router for…

[moeller0]

Let's run the numbers:
"speedtest"* goodput for 1 Gbps ethernet using IPv4/PPPoE/TCP
MTU 1500: 1000 * ((1500-8-20-20)/(1500+38)) = 944.08 Mbps
MTU 1508: 1000 * ((1508-8-20-20)/(1508 +38)) = 944.37 Mbps

so the "cost" of the PPPoE header in- or out-of-line of the typical 1500 byte MTU is rather "in the noise"; playing with MTUs can have an effect when going to real jumbo frames >> 1500 MTU, but good luck getting that to actually work over the existing internet.
BTW, the underlaying ethernet interface for PPPoE traffic needs to be at MTU 1500, the "MTU" to the internet beyond the remote PPPoE-terminal will be reduced by the 8 byte the PPPoE header takes, that way it seems as if your MTU would be 1492, but be assured, it is not...

So, IMHO the true problem of PPPoE is not so much the negligible reduction in potential throughput, but the fact that tunnel en- and decapsulation costs CPU cycles (though you can sometimes pass this work on to off-load engines). And that added CPU cost means that a router just capable of routing X Mbps, will only be ale to route <<X Mbps when also tasked to handle PPPoE.

@huj3r

My conclusion is that I cannot use higher MTU with this provider; did anyone else have success with it?

So, real jumbo frames are mostly off-limits for the internet, however a few ISP support RFC4638 "baby-jumbo-frames" that are just 8 bytes larger so that the apparent MTU to the internet looks like 1500 byte. You need to talk to your ISP to figure out whether they belong to that club. E.g. in the UK baby-jumbo frames seem to be usable for VDSL links, but e.g. in Germany Deutsche Telekom piloted this but never activated/offered it to end-customers, so its seems hard to make predictions about arbitrary ISPs.

*) "speedtests" are misnamed; speed is defined as distance/time, but speedtests measure data_volume/time so are not a measure of speed, but rather of capacity, but I digress

[huj3r]

@frollic

what router device are you using ?
even though slightly outdated, the new AX SoCs could probably work for you too.

It's the R7800.

@moeller0

You need to talk to your ISP to figure out whether they belong to that club.

ISP claims so. I am waiting to see if there is some other OpenWRT user using the same provider to double-check with their configuration.

[frollic]

@huj3r

It's the R7800.

you're not going to get anywhere near a gigabit with that ...

get some cheap x86, and repurpose the 7800 as AP.

[moeller0]

I bet you simply run out of CPU cycles... you could try one of the NSS builds and try to off-load the PPPoE handling to the NSS cores, but realistically at 1 Gbps you should get a router with a CPU that can handle the data independent of off-loads.

[huj3r]

@frollic

you're not going to get anywhere near a gigabit with that ...

get some cheap x86, and repurpose the 7800 as AP.

Netgear claims that it can reach it, but I haven't verified with Netgear's firmware instead of OpenWRT (obviously I want to use OpenWRT); perhaps Netgear's firmware can use hardware acceleration that it is not available to OpenWRT?

[frollic]

@huj3r

Netgear claims that it can reach it

do they actually claim it, or do they just say it's a "gigabit router" ?
(haven't checked, just asking ,,,)

@huj3r

perhaps Netgear's firmware can use hardware acceleration that it is not available to OpenWRT?

that could be the case, yes.

install and run htop on your 7800, check the CPU load while in use.

[huj3r]

@moeller0

I bet you simply run out of CPU cycles... you could try one of the NSS builds and try to off-load the PPPoE handling to the NSS cores, but realistically at 1 Gbps you should get a router with a CPU that can handle the data independent of off-loads.

I did a quick and dirty test with htop and CPU 0 reaches max 98.6% on download and 100% on upload while running the test, so it could be that. What are the NSS builds? I guess you mean these: Ipq806x NSS build (Netgear R7800 / TP-Link C2600 / Linksys EA8500)

But they don't mention that PPPoE is off-loaded to NSS cores? (only NAT)

@frollic

install and run htop on your 7800, check the CPU load while in use.

Already did, see above

@frollic

do they actually claim it, or do they just say it's a "gigabit router" ?

They do and ISP confirmed that it should reach gigabit (although he wasn't an OpenWRT expert, but the hardware alone is listed as compliant in their list)

[huj3r]

Now that I think of it, shouldn't upload and download run on separate cores? So that they don't interfere with each other.

Gotta run now but I am going to give a deeper look into these NSS builds later, thanks!

[moeller0]

@huj3r

Netgear claims that it can reach it, but I haven't verified with Netgear's firmware instead of OpenWRT (obviously I want to use OpenWRT); perhaps Netgear's firmware can use hardware acceleration that it is not available to OpenWRT?

Well, as I said that is using the NSS core accelerator. I would not rely on that myself...

@huj3r

I am waiting to see if there is some other OpenWRT user using the same provider to double-check with their configuration.

Yeah, but look at the numbers, baby jumbo frames really only help if you have constant problems with path MTU discovery, otherwise you will likely not notice. The real prize is getting rid of PPPoE encapsulation at all. However that latter is unlikely in areas where PPPoE is extensively used to allow choice of ISPs over common/foreign-ISP-operated access networks, there some form of tunneling is needed and if ISPs already have a working solution for one such tunnel like PPPoE they are unlikely to change to another tunnel...

@huj3r

I did a quick and dirty test with htop and CPU 0 reaches max 98.6% on download and 100% on upload while running the test, so it could be that.

This is as much proof as you are going to get, you are CPU limited.

@huj3r

What are the NSS builds? I guess you mean these: Ipq806x NSS build (Netgear R7800 / TP-Link C2600 / Linksys EA8500)

But they don't mention that PPPoE is off-loaded to NSS cores? (only NAT)

Well, I do not know what off-loads are implemented there as I do not own an NSS-equipped router (and I am no fan of such off-load engines, so even if my router would support that, I likely would not use it, as most hardware acceleration is incompatible with sqm-scripts).

@huj3r

They do and ISP confirmed that it should reach gigabit (although he wasn't an OpenWRT expert, but the hardware alone is listed as compliant in their list)

That is unfortunately the game ISPs currently play, offer/recommend CPE that only allow the contracted rates extensively using accelerators. I guess this is partly customer driven, as devices that can handle higher speed on their CPUs tend to be either more expensive or less "complete". As an example for the latter, a number of x86 based solutions do not offer competent WiFi and hence are no all-in-one replacements. On the plus side when combined with an AP, you now could place the router and AP such that each is at the optimal location...

[moeller0]

@huj3r

Now that I think of it, shouldn't upload and download run on separate cores? So that they don't interfere with each other.

You could look at irqbalance and packet-steering, both of these might help (or might not, that depends a bit on the actual hardware).

[slh]

So you have 500Mbps-1Gbps fiber and need a router READ THIS FIRST

It's such a frequently asked question over the last few years that we really need a post we can point people to. So your cable company rolled out 500Mbps or 1Gbps download speeds, or you have 1Gbps symmetric fiber (GPON/EPON) from ATT or Centurylink or Orange or whoever. You realize your old all in one router from 2009 is not up to the task of handling this, but hey by now stuff should be cheap and available that will handle your new fiber connection right? So you want advice about a router for…

The NSS cores can offload PPPoE traffic, but if that's enabled enabled in those OpenWrt based builds is another question.

Neither irqbalance nor packet steering is going to make a material difference, the r7800's ARMv7 cores won't do more than that - not for routing plain ethernet (at most 550-650 MBit/s), even less for PPPoE encapsulated traffic. Software flow-offloading might push the envelope a little (less for PPPoE, than it would for plain ethernet), but that's not without quirks either - this hardware is just designed with NSS offloading to a proprietary firmware running on its two 800 MHz little-endian ubicom32 derived cores (which lay dormant on OpenWrt) in mind, without that (as on OpenWrt) it won't reach those speeds.

Faster hardware, such as x86_64, is the better option for this use case and the following might provide some ideas (more in the later parts of that thread):

Tips for getting cheap used x86-based firewall with full Gbit NAT (a PC Engines APU) if you are in the US

While I was watching for old stuff I could play with on ebay I noticed that a few sellers are offering a few "cloud managed routers" (i.e. if you don't pay a subscription they are paperweights) from a company called "simplewan" that looked oddly familiar. here some pictures (from auctions, not mine) they are bright orange. [simplewan 3] [simplewan2] [simplewan 1] The images are a dead giveaway, these devices seem to just be a branded version of PCEngines APU1 (the ones with black USB 2.0 p…

Again, PPPoE and sqm (should you need it) are demanding, so even those AMD Jaguar cores or baytrail might (PPPoE) or will not (sqm) suffice for full line speed at 1 GBit/s, but something starting around the ivy-bridge based c1037u will (but haswell and newer would be more power efficient, depending on the mainboard).
Personally I have gone this route, and I'm not looking back - the r7800 can still make a fine AP.

[huj3r]

I guess I need to open a new topic now, because this was about using the correct MTU with Freedom Internet; I already knew it's not supposed to make a big difference on reaching giga speed but I wanted to select the correct MTU independently from that.

[huj3r]

I'll be damned..I have set back MTU=1500 on the WAN interface and flashed the v22 NSS build, this is the result:

This is much faster, thanks everyone for the tips!
I'll wait for someone else with knowledge of this ISP to tell me more specifically about the MTU configuration

[moeller0]

@huj3r

but I wanted to select the correct MTU independently from that.

Any MTU <= the maximally usable MTU is "correct". But sure I can understand the desire to squash this for good. If my ISP would actually offer that I might get the same bug as well

[frollic]

Check if the Dynalink WRX36 is available on Amazon NL again.

[huj3r]

@moeller0

Any MTU <= the maximally usable MTU is "correct". But sure I can understand the desire to squash this for good. If my ISP would actually offer that I might get the same bug as well

I get what you mean; I actually want the correct MTU because I experience random connectivity issues (TLS connections fail to establish, apps like Instagram buffering but not buffering when using VPN etc) and would like to make sure it's not related to MTU.

For now I will follow this configuration since they should be technically the same as my provider: https://openwrt.org/docs/guide-user/network/wan/isp-configurations#kpn_ftth