Slow ethernet speed from router to desktop

stuffie · January 14, 2020, 4:18am

Hi all,

I got stuck into a performance issue after upgrading from 17.01 to 18.06.4. My router is a Netgear r7800 and when I run iperf3 on router and desktop, i get this:

root@router:/tmp# iperf3 -c 192.168.1.11
Connecting to host 192.168.1.11, port 5201
[  5] local 192.168.1.1 port 32968 connected to 192.168.1.11 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.02   sec  45.6 MBytes   376 Mbits/sec    0   61.3 KBytes
[  5]   1.02-2.04   sec  32.2 MBytes   264 Mbits/sec    0   78.4 KBytes
[  5]   2.04-3.04   sec  18.8 MBytes   158 Mbits/sec    0    103 KBytes
[  5]   3.04-4.08   sec  18.8 MBytes   151 Mbits/sec    0    128 KBytes
[  5]   4.08-5.00   sec  20.4 MBytes   187 Mbits/sec    0    144 KBytes
[  5]   5.00-6.05   sec  43.1 MBytes   345 Mbits/sec    0    153 KBytes
[  5]   6.05-7.01   sec  19.6 MBytes   172 Mbits/sec    0    207 KBytes
[  5]   7.01-8.02   sec  16.8 MBytes   140 Mbits/sec    0    294 KBytes
[  5]   8.02-9.02   sec  43.9 MBytes   367 Mbits/sec    0    294 KBytes
[  5]   9.02-10.03  sec  37.5 MBytes   313 Mbits/sec    0    294 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.03  sec   297 MBytes   248 Mbits/sec    0             sender
[  5]   0.00-10.03  sec   297 MBytes   248 Mbits/sec                  receiver

root@router:/tmp# iperf3 -c 192.168.1.11 -R
Connecting to host 192.168.1.11, port 5201
Reverse mode, remote host 192.168.1.11 is sending
[  5] local 192.168.1.1 port 34850 connected to 192.168.1.11 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   109 MBytes   912 Mbits/sec
[  5]   1.00-2.00   sec   108 MBytes   902 Mbits/sec
[  5]   2.00-3.00   sec   107 MBytes   897 Mbits/sec
[  5]   3.00-4.00   sec   102 MBytes   858 Mbits/sec
[  5]   4.00-5.00   sec   108 MBytes   909 Mbits/sec
[  5]   5.00-6.00   sec   109 MBytes   917 Mbits/sec
[  5]   6.00-7.00   sec   109 MBytes   914 Mbits/sec
[  5]   7.00-8.00   sec   106 MBytes   892 Mbits/sec
[  5]   8.00-9.00   sec   108 MBytes   910 Mbits/sec
[  5]   9.00-10.00  sec   109 MBytes   913 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  1.05 GBytes   903 Mbits/sec                  sender
[  5]   0.00-10.00  sec  1.05 GBytes   902 Mbits/sec                  receiver

So there is no problem from desktop to router (900Mbit/sec), but router to desktop is too slow and fluctuates (150-400 Mbit/sec). I tried several computers/cables/ports, always the same results. I can also confirm that desktop to desktop speed (via r7800) isn't slow.

Can someone point my in de right direction? Samba worked better before

leeandy · July 10, 2019, 1:43pm

Oh, I have a similar issue with my x86 board Celeron J3355 dual core 2Ghz.
Iperf has never exceeded ~ 600mbps in both directions.
My board's cpu is much stronger than r7800.

cbz · July 10, 2019, 3:28pm

What are the actual symptoms of your performance issue? What is iperf like via your router as opposed to from your router? [and as an aside what was the change since you upgraded] Because the scenario you are testing doesn't really correspond to a real use case.

stuffie · July 10, 2019, 4:07pm

Well, I noticed a slower downloading speed from my router to my desktop using a Samba share. On 17.01 is was 60-80 MB/s, and now it is 15-30 MB/s. Uploading speed is fine. So I investigated the problem and found out it was not the SSD USB harddrive, nor Samba. According to iperf it has to be the ethernet speed.

jeff · July 10, 2019, 4:54pm

What do you see when you route through the router, WAN to LAN? (LAN only will not touch the CPU, only the switch fabric.)

Running iperf on the router could be exhausting CPU capacity.

diizzy · July 10, 2019, 5:57pm

@stuffie
Jump to 19.X or master/trunk branch, this is where all debugging and enhancements will occur.
If you want a community build for your router this might be of interest: Build for Netgear R7800

@leeandy
Does performance improve if you use another Linux/BSD distribution?

leeandy · July 11, 2019, 12:38am

I'm try running Ubuntu 19.04 live-usb on my board, but not improve, similar speed.

I noticed that when using a iperf with cubic setting, the speed was a bit faster ~ 100Mbps than bbr setting. Noticed that the rate of increase and decrease was erratic.
After modifying some things (set cpu scaling_governor to performance, install irqbalance, set bios C state to C6 instead of C10), the speed improved slightly.
Iperf server running on my Ubuntu pc, iperf client run on router board:

root@ASR:~# iperf3 -c 192.168.16.65 -V -C bbr
iperf 3.7
Linux ASR 4.19.57
Control connection MSS 1448
Connecting to host 192.168.16.65, port 5201
      Cookie: jdlxgv6o3gdeqavmhxhtl2kmxk7vrs2mfrf3
      TCP MSS: 1448 (default)
[  5] local 192.168.16.1 port 53481 connected to 192.168.16.65 port 5201
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   842 MBytes   707 Mbits/sec  57544             sender
[  5]   0.00-10.05  sec   841 MBytes   702 Mbits/sec                  receiver
CPU Utilization: local/sender 3.9% (0.0%u/3.9%s), remote/receiver 35.4% (3.0%u/32.3%s)
snd_tcp_congestion bbr
rcv_tcp_congestion bbr

root@ASR:~# iperf3 -c 192.168.16.65 -V -C cubic
iperf 3.7
Linux ASR 4.19.57 
Control connection MSS 1448
Connecting to host 192.168.16.65, port 5201
      Cookie: gwbfl3r2wwezoqwfeffvuihstcpozj6dbmm3
      TCP MSS: 1448 (default)
[  5] local 192.168.16.1 port 53827 connected to 192.168.16.65 port 5201
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   974 MBytes   817 Mbits/sec  21385             sender
[  5]   0.00-10.04  sec   971 MBytes   811 Mbits/sec                  receiver
CPU Utilization: local/sender 4.6% (0.5%u/4.1%s), remote/receiver 33.1% (2.3%u/30.8%s)
snd_tcp_congestion cubic
rcv_tcp_congestion cubic

diizzy · July 11, 2019, 6:25am

If that's the case I think you can safely assume it's performance issue with the IPQ8***-chipset and/or possibly a faulty cable. There's a "hack/workaround" which makes the SoC go full blast all the time by changing the cpu governor from what I've read but since I don't have any IPQ8-chipset devices I haven't followed the discussion(s) closely.

stuffie · July 12, 2019, 6:22pm

Sorry, I don't have such fast internet connection. It's only 50/5.

stuffie · July 12, 2019, 8:10pm

Thanks for all suggestions. Looking for the best solution, I ran more tests.

First against different versions (clean installs) of LEDE/OpenWRT:

LEDE Reboot 17.01.4   average speed 923 Mbit/sec
LEDE Reboot 17.01.6   average speed 840 Mbit/sec
OpenWRT 18.06.0       average speed 422 Mbit/sec
OpenWRT 18.06.4       average speed 238 Mbit/sec

Those are all download speeds from router to desktop. So, newer versions give worse results. I hope version 19 does not give me 10kbit/sec...

And then I have tested tweaked versions:

stable openwrt-18.06  from https://forum.openwrt.org/t/build-for-netgear-r7800/316:
   - average speed 394 Mbit/sec

stable openwrt-19.07  from https://forum.openwrt.org/t/build-for-netgear-r7800/316:
   - could not be tested, package iperf3 not available

OpenWRT 18.06.4 with some tweaking (https://forum.openwrt.org/t/netgear-r7800-exploration-ipq8065-qca9984/285/1442): 
   - average speed 489 Mbit/sec

And tried some other suggestions found on the internet:

Software Flow offloading: no difference
Hardware Flow offloading: not available on r7800
Disable Firewall (iptables -F): slightly better performance (5%)
irqbalance: no change (edited after reply below)

What's next? What do I have to do to get 17.01.4 performance on 18.06.4?

leeandy · July 12, 2019, 7:39pm

On x86 target, 17.01 branch have ondemand scaling_governor, but it's missing from 18.06 branch, the default scaling_governor is powersave. You can checking by command:

cat /sys/devices/system/cpu/cpufreq/policy*/*

You can also manual download & install irqbalance from this repos: http://downloads.openwrt.org/snapshots/packages/arm_cortex-a15_neon-vfpv4/packages/irqbalance_1.2.0-4_arm_cortex-a15_neon-vfpv4.ipk

leeandy · July 13, 2019, 7:21pm

Well, It seems that the problem in the realtek driver on my ubuntu pc with kernel 5.0. After i'm reinstall r8168-dkms driver, the speed > 900mbps on both direction.
@stuffie what is the nic card on your computer? What is the nic driver version?

stuffie · January 11, 2020, 6:14pm

Downloads are still slow on 19.07.0:

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 -----------------------------------------------------
 OpenWrt 19.07.0, r10860-a3ffeb413b
 -----------------------------------------------------
root@router:/etc/hotplug.d/net# iperf3 -c 192.168.1.11
Connecting to host 192.168.1.11, port 5201
[  5] local 192.168.1.1 port 40582 connected to 192.168.1.11 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.06   sec  28.0 MBytes   222 Mbits/sec    0   54.2 KBytes
[  5]   1.06-2.05   sec  20.0 MBytes   170 Mbits/sec    0   78.4 KBytes
[  5]   2.05-3.02   sec  30.0 MBytes   259 Mbits/sec    0   97.0 KBytes
[  5]   3.02-4.02   sec  42.5 MBytes   356 Mbits/sec    0    115 KBytes
[  5]   4.02-5.02   sec  42.5 MBytes   356 Mbits/sec    0    127 KBytes
[  5]   5.02-6.02   sec  42.5 MBytes   357 Mbits/sec    0    127 KBytes
[  5]   6.02-7.02   sec  30.0 MBytes   253 Mbits/sec    0    127 KBytes
[  5]   7.02-8.00   sec  21.2 MBytes   181 Mbits/sec    0    127 KBytes
[  5]   8.00-9.02   sec  21.0 MBytes   174 Mbits/sec    0    202 KBytes
[  5]   9.02-10.01  sec  30.0 MBytes   254 Mbits/sec    0    202 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.01  sec   308 MBytes   258 Mbits/sec    0             sender
[  5]   0.00-10.01  sec   308 MBytes   258 Mbits/sec                  receiver

iperf Done.

root@router:~# cat /sys/devices/system/cpu/cpufreq/policy0/scaling_governor
ondemand
root@router:~# cat /sys/devices/system/cpu/cpufreq/policy1/scaling_governor
ondemand

psherman · February 8, 2020, 7:15pm

@blinton - did you read the solution from that thread? This is a hardware performance limitation if the packets originate or terminate on the router (the SoC is just not powerful enough to do this).

stuffie · February 8, 2020, 9:17pm

@psherman - then why do i get 1Gbps with LEDE Reboot 17.01.4, but only 250Mbit/s on 19.07.1?

See my post here: Slow ethernet speed from router to desktop

psherman · February 8, 2020, 9:38pm

I have no idea why you would have gotten 1gbps on lede, but the reduction could be the result of higher load on the CPU in general.

blinton · February 9, 2020, 8:36am

Hi stuffie and psherman,

Yes, thanks a lot for the explanation / solution. Indeed, when using the router as "packet through" and not "originating" from it, I get easily full speed (ie 940 MBit/s) at all time, also for VLANs.
In my case, I do not see any scenario, where I would use the router with packets originating from it (without encryption).

cheers Blinton