Seeed ReRouter CM4 on Gigabit Fibre Internet - Not getting speeds I was expecting

[Le_Patenteux]

Reading the forums, I was led to believe that the Seeed ReRouter CM4 would be plenty enough to route and saturate my gigabit Internet link... Spoiler, right now, it does not!

I am new to OpenWRT and have played around with it only for a dozen hours yet...

Firmware
OpenWrt 22.03.5 r20134-5f15225c1e

Setup (had the same result without the dumb switch)

Speed test:

I tried using software offloading... it improved, but still not what I was expecting, and I read that it disables many other useful functions on the device.
Speedtest:

Then I tried using he speedtest-cli package within the router

speedtest-cli --share

And got this result. (With or without software offloading)

What this tells me, is that my issue resides in packet management between WAN and LAN...

What can I do to improve this?

I own 2 basic managed switches and a dumb switch if it can be of any help

For now, the internal network is not using VLAN

[Le_Patenteux]

Forgot to say... the CPU on the thing is barely working... 2-3% with some rare peaks around 12%

[ncompact]

add pakages:

irqbalance

and

suppose this ...

So you have 500Mbps-1Gbps fiber and need a router READ THIS FIRST

It's such a frequently asked question over the last few years that we really need a post we can point people to. So your cable company rolled out 500Mbps or 1Gbps download speeds, or you have 1Gbps symmetric fiber (GPON/EPON) from ATT or Centurylink or Orange or whoever. You realize your old all in one router from 2009 is not up to the task of handling this, but hey by now stuff should be cheap and available that will handle your new fiber connection right? So you want advice about a router for…

[Le_Patenteux]

Thank you for your response, I will add this.

As for the post cited, I read it 5 000 times... I was told a CM4 had all the power needed to route that...

Could it be the VLAN hurting the performance? The fact that the internal network does not use it?
I could put a switch between the CM4 and the ONT and make it do the VLAN tagging for the ISP

[ncompact]

if you connect a pc directly to your ONT (whit vlan set) what speed do you get?

[Le_Patenteux]

Eeeeeeuh!... this can be done? Really?

My Linux PC could tag the VLAN and register on PPPoE ???

Do you have a guide for this?

[ncompact]

https://wiki.debian.org/PPPoE

https://wiki.debian.org/it/NetworkConfiguration

Openwrt = linux (kernel) + busybox + other packages

[ncompact]

pppoe set up with VLAN configuration

https://wiki.archlinux.org/title/VLAN

PPPoE on Ubuntu 14.04

https://www.google.com/search?q=linux+tagged+vlan+interface+pppoe

ecc .....

[pythonic]

@ncompact

add pakages:

irqbalance

irqbalance doesn't help much on the Pi4 family in my experience however enabling receive packet steering does help a fair bit.

[Le_Patenteux]

SEE NEXT POST, I MADE A MISTAKE IN MY TESTING!

OK... I was able to connect directly through my laptop... network-manager GUI is not really helpful for that situation, let's just say that!

Here are the results:

I did 3 runs and got from 923 to 933 up and down... Pretty close to the advertised 940/940 (not exactly gigabit, but who's counting!)

Now... The CM4 was almost getting me the same speed, except for uploads... I will re-wire it and retest in case upload was just a transient error...

Now... how can I improve it's switching capability from there?

In my hardware, I have a tp-link dumb 8 ports gigabit switch, and a brand new TL-SG116E managed switch...
Can I alleviate the load on the CM4 by placing components differently? Like having the switch do the vlan maybe?

[Le_Patenteux]

Ok... I made a mistake in my testing so I will rectify it here...

I was able to connect my PC to the ONT with your help, thank you! (network-manager GUI is not helpful here!)

My first test was using the Website Speedtest.net in the browser... Then I realized I was not replicating the test conditions used in the router, which is to test using speedtest-cli

I re-did the test with speedtest-cli on my PC and got similar results as those on the CM4

That is, download speeds are close to max, but upload speeds are lower.

I was getting unstable results (range was outside error margin)... and narrowed-it down to forcing the server to use in speedtest-cli

speedtest-cli --server 16754 --share

I did 10 runs on each device to get the most accurate data on speed.
Upload speed difference could be due to the different architecture's implementation of the tool (AMD64 vs ARM64)... I can't tell.

This is the CPU graph for the CM4. I was repeating tests in 2 different runs.
(irqbalance is installed)

So, my understanding is that is a limitation of the tool, not the connexion itself, at least for uploads.

Now, what makes it that I can't route those speeds to the LAN port after at reasonable speeds?

If I put my managed switch between the ONT and the CM4 make it do the VLAN to ISP, would it help?

[zekica]

You are using PPPoE.

Can you test enabling "Packet Steering" in Network -> Interfaces -> Global network options and additionally type each of the following in the router's shell - testing each time:

echo 1 >/sys/class/net/pppoe-*/queues/rx-0/rps_cpus
echo 2 >/sys/class/net/pppoe-*/queues/rx-0/rps_cpus
echo 4 >/sys/class/net/pppoe-*/queues/rx-0/rps_cpus
echo 8 >/sys/class/net/pppoe-*/queues/rx-0/rps_cpus
echo 3 >/sys/class/net/pppoe-*/queues/rx-0/rps_cpus
echo c >/sys/class/net/pppoe-*/queues/rx-0/rps_cpus
echo f >/sys/class/net/pppoe-*/queues/rx-0/rps_cpus

[Le_Patenteux]

Can you please explain what those commands do?
Is it doing some traffic shaping of some kind?

[zekica]

Packet Steering is setting "f" to all ethernet devices' rps_cpus on your 4-core RPi.

When rps_cpus is set to zero, kernel processes the packet up to the point it is inserted into the output device's queue on the CPU core it originally received the interrupt on, and some ethernet chips support only one interrupt request number - always handled by a single core.

When you set rps_cpus to a number (in hexadecimal format), whenever a packet is received, it is instead dispatched to any of the cores specified in the bitmask (1 -> 1st core, 2 -> 2nd core, 4 -> 3rd core, 8 -> 4th core, 3 -> 1st and 2nd cores, c -> 3rd and 4th cores, f -> any core) by creating a software IRQ, and then continuing the processing of the packet on that (random) core.

[Le_Patenteux]

@zekica

echo 1 >/sys/class/net/pppoe-*/queues/rx-0/rps_cpus

1 - No change

2 - Worst performance (700/650)

4 - Worst-er performance (680/630)

8 - Same as 2

3 - Lower performance (880/650)

c - Even-Worst-er performance (650/630)

f - Same as 2, but download was all over the place (675-800 / 650)

I forgot to "cat" the original default value... is it 1 ?

[zekica]

The original value is "0"

[Le_Patenteux]

So from my result, we could conclude that 0 or 1 would be pretty much the same... Meaning the interface has afinity to the first core... am I right?

I did not try switching the ports on the device... could that help with something... depending on where they are "tied" to the CM4 ?

For now I am on eth0 for WAN

From the schematics, there is one conected directly to the CM4, and the other one to a PCIe to USB3 to Ethernet path...

[zekica]

You are mostly correct but not entirely, AFAIK 0 won't trigger a software interrupt while 1 will - but keep it in the same core if your interface sends IRQs to the first core.

You can try switching - if eth1 has a better chip, it can help moving the PPPoE processing there, and using eth0 for your lan. I don't know which chips this board is using.

From the diagram, one of the ethernet chips is connected via USB. This surely adds additional overhead.

[Le_Patenteux]

The USB3/LAN interface is Microchip LAN7800

I am trying to see which interface is connected where from the schematics...
(trying to Identify them)

[Le_Patenteux]

Is there a command to enter from the console that would reset all network parameters to default when you messed-up and can't access the router anymore?...

I am asking for a friend...