10gig system -> 1gig OpenWrt gets 300meg

I have an x86 mini PC (Protecli FW4A) running OpenWRT (upgraded to 21.02.2). I have some 10 gigabit ethernet connections internally - there's a 10gig switch connected to a 1gig switch connected to a 1gig port on the OpenWRT router. I noticed that my 10gig systems get the full 1gig Internet download (well, about 930-940meg, which is the real max on 1gig), but only maybe 200-300meg upload. Running iperf3 between the 10gig systems and the OpenWRT router shows the same.

Going from other 1gig devices to the OpenWRT router, or from 10gig devices to other 1gig devices, all always get the expected 1gig. The problem is specific to the 10gig devices to 1gig OpenWRT. All the other devices are Linux (Fedora 35).

Is there anything different about how OpenWRT is configuring the NICs or such that might cause this? Since I can go 10->1 and get 1 between other Linux systems, it seems like it must be something specific to OpenWRT and/or the NICs (Intel, using e1000e driver).

Do you have any qos setup at all on the router or on any of the switches

Everything is at defaults in that regards. It doesn't "feel" like it would be switch related, since I get 1gig through from 10gig device to non-OpenWRT 1gig Linux device (everything is plugged into the same 1gig switch except for the 10gig devices on the 10gig switch).

is layer three(routing) involved?
does connecting OpenWrt to the 10G switch alter the behavior?

The only router is the OpenWRT device. I connected it through the 10gig switch (did have one copper 1gig SFP on hand) and don't see any difference, so that does take the 1gig switch out of the picture.

When doing iperf3 from a 10gig to another 1gig, I see retransmits for the first few seconds (and lower throughput), but then it steadies out and gets close to 1gig the rest of the way. Going from 10gig to OpenWRT, the retransmits never stop. It's like some mechanism is disabled in OpenWRT compared to other Linux systems.

If you're running the iperf client on the router, don't - it uses enough CPU to skew the results. Plug in a wired gigabit client machine and use that to run the client on the router side.

sounds like something in the domain of layer4
(sliding window / congestion control)

are you running iperf tcp?

The router CPU is an Atom E3845 (quad core); I think it can handle it. I'm running iperf there to eliminate as much else as practical, and the results I'm getting mirror results going to the Internet.

Yes, iperf3 is TCP by default.

To add: I started looking at this because I saw slow Internet upload speeds. I have a symmetric 1gig connection, and if I connect my main system through a 1gig NIC, Ookla speedtests (using the CLI tool to avoid browser screwiness) shows around 930-940meg down and up. When I change that system to the 10gig NIC, down is the same, but up drops to 140-150meg.

That's when I started running iperf3 locally to narrow it down. 10gig->1gig works okay, except for the OpenWRT router, which seems to point to something in OpenWRT's config.

Have you tried swapping the cable to the router?

1 Like

Yes, and also NIC (since it has 4 and I'm only using 2).