Speed drops to 100mbit accross VLANs

Installing and Using OpenWrt
1

Hi all, I have this really weird issue on my OpentWRT setup. Firstly, everything works as expect except for some speed issues I just noticed.

Cloud<-->Modem<-->OpentWRT (Linksys wrt1900acs) <--->VLANS via unmanaged switches.
I current have:
192.168.1.x as Main lan
192.168.10.x as Servers
192.168.20.x as ios devices

The Issue:

  • From main PC's can use speedtest.net to get gigabit speeds.
  • From VM's on server proxmox can use speedtest.net to get gigabit speeds.
  • between 2 machines on the lan (windows) can copy files to and from each other at gigabit speeds
  • Between 2 servers on the server lan can iperf between each other for gigabit speeds

So switches and routing to Internet all seems to be fine speed wise. Issue now is between servers and lan it only does 100mbit speeds. Doing iperf to router also only does 100mbit speeds.

I have no idea what is doing on with this. If is is purely VLAN related, then 192.168.1.100 to the router 192.168.1.1 should ipsec at a gbit, but its only 100mbit. Servers to router is also in the 100mbit range.

Does anybody have any clue what is going on and how to diagnose this issue?
Your help will be appreciated.

2

It's not exactly clear what you mean: You've set up the router ports as untagged and use unmanaged switches for the untagged traffic? Then it's fine.
Using tagged VLANs over unmanaged switches is not recommended (it works with some switches, doesn't with others).

This traffic has to reach the CPU of the router and needs to be handled in software. Since most routers (didn't check on yours) only have a single connection between the internal switch and the CPU, your traffic needs to pass this interface twice, halving the available bandwidth (so a theoretical 500M). The CPU needs to handle this routing speed, which I doubt it does. You can check the CPU usage of the router while you perform such a speed test - it's probably maxed out.

See here for a similar discussion with other hardware.

3

For your first question:
Yes, I have set each router port to a seperate Vlan untagged, then Switches for multiple devices. Seemed the simplest way. I do have 1 tagged port to a second openwrt router.

I was running top and and checking the Luci webpage and the cpu doesn't seem to be touched at all. Maybe sometimes hitting 1%, so either that is false or the cpu does not seem to be the issue.

I will look through the other thread, but from what I can see it does not seem to be a cpu/load on the router issue. Is there a better way of getting that sort of measurement? After copying a 1.5GB file from server to lan, uptime still shows a load average of 0.0.

EDIT: between the 2 routers on tagged port it does pop up to 0.06 load. which is still minor blip.

4

Can you confirm that every physical and virtual machine across all of your VLANs can get gigabit speeds when testing via an internet bandwidth speed test?

And is the issue that any/all iperf tests from your server network to your regular lan are limited to 100Mbps? What is the exact number you are getting?

5

All windows machines can get gigabit speeds on speedtest website. I have installed speed-cli on the linux servers (one has AMD 4650g chip, OrangePi5 and OrangePi4) and they all are approaching Gigabit speeds.

Also, can confirm that on the same subnet, iperf is working find, but across subnets it drops to 100mbit speeds. Here are the iperf results each run twice

root@proxmox:~# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  128 KByte (default)
------------------------------------------------------------
[  1] local 192.168.10.200 port 5001 connected with 192.168.1.100 port 61388 (icwnd/mss/irtt=14/1448/643)
[ ID] Interval       Transfer     Bandwidth
[  1] 0.0000-10.2510 sec   113 MBytes  92.8 Mbits/sec
[  2] local 192.168.10.200 port 5001 connected with 192.168.1.100 port 61390 (icwnd/mss/irtt=14/1448/659)
[ ID] Interval       Transfer     Bandwidth
[  2] 0.0000-10.2328 sec   113 MBytes  92.4 Mbits/sec
[  3] local 192.168.10.200 port 5001 connected with 192.168.10.100 port 54498 (icwnd/mss/irtt=14/1448/455)
[ ID] Interval       Transfer     Bandwidth
[  3] 0.0000-10.0183 sec  1.10 GBytes   941 Mbits/sec
[  4] local 192.168.10.200 port 5001 connected with 192.168.10.100 port 54608 (icwnd/mss/irtt=14/1448/282)
[ ID] Interval       Transfer     Bandwidth
[  4] 0.0000-10.0256 sec  1.10 GBytes   941 Mbits/sec

If you need more information, just ask.

6

Can you run speed tests between two windows machines? (iperf is a good option).

The figure you're getting really looks like a 100Mbps ethernet physical link, rather than a routing bottleneck.

And let's take a look at your configs:

Please connect to your OpenWrt device using ssh and copy the output of the following commands and post it here using the "Preformatted text </> " button:
grafik
Remember to redact passwords, MAC addresses and any public IP addresses you may have:

ubus call system board
cat /etc/config/network
cat /etc/config/wireless
cat /etc/config/dhcp
cat /etc/config/firewall
7

Unfortunately with WSL2 it tends to create its own subnet so can't get a iperf server running properly on it. Copying a large file between it does get up to 113MB/s as opposed to from the server which is about 11MB/s.

However here is iperf run on both windows boxes to an OrangePi4 sitting in the same LAN.

root@orangepi4-lts:~# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  128 KByte (default)
------------------------------------------------------------
[  4] local 192.168.1.10 port 5001 connected with 192.168.1.100 port 61408
[ ID] Interval       Transfer     Bandwidth
[  4] 0.0000-10.0359 sec   878 MBytes   734 Mbits/sec
[  5] local 192.168.1.10 port 5001 connected with 192.168.1.100 port 61410
[ ID] Interval       Transfer     Bandwidth
[  5] 0.0000-10.0418 sec   882 MBytes   737 Mbits/sec
[  4] local 192.168.1.10 port 5001 connected with 192.168.1.101 port 60474
[ ID] Interval       Transfer     Bandwidth
[  4] 0.0000-10.0304 sec  1.09 GBytes   937 Mbits/sec
[  5] local 192.168.1.10 port 5001 connected with 192.168.1.101 port 60476
[ ID] Interval       Transfer     Bandwidth
[  5] 0.0000-10.0257 sec  1.10 GBytes   940 Mbits/sec

Also, the logs you requested. I think I have sanitized them properly.

root@Linksys_OpenWrt:~# ubus call system board
{
        "kernel": "5.10.146",
        "hostname": "Linksys_OpenWrt",
        "system": "ARMv7 Processor rev 1 (v7l)",
        "model": "Linksys WRT1900ACS",
        "board_name": "linksys,wrt1900acs",
        "rootfs_type": "squashfs",
        "release": {
                "distribution": "OpenWrt",
                "version": "22.03.2",
                "revision": "r19803-9a599fee93",
                "target": "mvebu/cortexa9",
                "description": "OpenWrt 22.03.2 r19803-9a599fee93"
        }
}

root@Linksys_OpenWrt:~# cat /etc/config/network

config interface 'loopback'
        option device 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'fdfa:c69a:1874::/48'

config device
        option name 'br-lan'
        option type 'bridge'
        list ports 'lan1'
        list ports 'lan2'
        list ports 'lan3'
        list ports 'lan4'

config interface 'lan'
        option proto 'static'
        option ipaddr '192.168.1.1'
        option netmask '255.255.255.0'
        option device 'br-lan.99'

config device
        option name 'wan'
        option macaddr 'MAC_ADDRESS'

config interface 'wan'
        option device 'wan'
        option proto 'dhcp'

config interface 'wan6'
        option device 'wan'
        option proto 'dhcpv6'

config bridge-vlan
        option device 'br-lan'
        option vlan '99'
        list ports 'lan1:u*'
        list ports 'lan4:t'

config bridge-vlan
        option device 'br-lan'
        option vlan '10'
        list ports 'lan2'
        list ports 'lan4:t'

config bridge-vlan
        option device 'br-lan'
        option vlan '20'
        list ports 'lan3'
        list ports 'lan4:t'

config interface 'servers'
        option proto 'static'
        option device 'br-lan.10'
        option ipaddr '192.168.10.1'
        option netmask '255.255.255.0'

config interface 'iot'
        option proto 'static'
        option device 'br-lan.20'
        option ipaddr '192.168.20.1'
        option netmask '255.255.255.0'

root@Linksys_OpenWrt:~# cat /etc/config/wireless

config wifi-device 'radio0'
        option type 'mac80211'
        option path 'soc/soc:pcie/pci0000:00/0000:00:01.0/0000:01:00.0'
        option band '5g'
        option htmode 'VHT80'
        option country 'US'
        option cell_density '0'
        option channel '36'

config wifi-device 'radio1'
        option type 'mac80211'
        option path 'soc/soc:pcie/pci0000:00/0000:00:02.0/0000:02:00.0'
        option band '2g'
        option htmode 'HT20'
        option country 'US'
        option cell_density '0'
        option channel '5'

config wifi-iface 'default_radio1'
        option device 'radio1'
        option network 'lan'
        option mode 'ap'
        option macaddr 'MAC_ADDRESS'
        option ssid 'monkey_zoo'
        option encryption 'psk2'
        option key 'password'
        option ieee80211r '1'
        option mobility_domain '3FAD'
        option ft_over_ds '0'
        option ft_psk_generate_local '1'

config wifi-iface 'wifinet2'
        option device 'radio1'
        option mode 'ap'
        option ssid 'Cats_House_2.4GHz'
        option encryption 'psk2'
        option key 'password'
        option ieee80211r '1'
        option mobility_domain '2FAD'
        option ft_over_ds '0'
        option ft_psk_generate_local '1'
        option network 'iot'

config wifi-iface 'wifinet3'
        option device 'radio0'
        option mode 'ap'
        option ssid 'Cats_House_5GHz'
        option key 'password'
        option ieee80211r '1'
        option mobility_domain '2FAC'
        option ft_over_ds '0'
        option ft_psk_generate_local '1'
        option network 'iot'
        option encryption 'psk2'
        option disabled '1'

config wifi-iface 'wifinet4'
        option device 'radio0'
        option mode 'ap'
        option ssid 'monkey_zoo5g'
        option encryption 'psk2'
        option key 'password'
        option ieee80211r '1'
        option mobility_domain '3FAC'
        option ft_over_ds '0'
        option ft_psk_generate_local '1'
        option network 'lan'

root@Linksys_OpenWrt:~# cat /etc/config/dhcp

config dnsmasq
        option domainneeded '1'
        option localise_queries '1'
        option rebind_protection '1'
        option rebind_localhost '1'
        option local '/lan/'
        option domain 'lan'
        option expandhosts '1'
        option authoritative '1'
        option readethers '1'
        option leasefile '/tmp/dhcp.leases'
        option resolvfile '/tmp/resolv.conf.d/resolv.conf.auto'
        option localservice '1'
        option ednspacket_max '1232'
        list server '192.168.1.10'

config dhcp 'lan'
        option interface 'lan'
        option start '100'
        option limit '150'
        option leasetime '12h'
        option dhcpv4 'server'
        list dhcp_option '6,192.168.1.10'

config dhcp 'wan'
        option interface 'wan'
        option ignore '1'

config odhcpd 'odhcpd'
        option maindhcp '0'
        option leasefile '/tmp/hosts/odhcpd'
        option leasetrigger '/usr/sbin/odhcpd-update'
        option loglevel '4'

config dhcp 'servers'
        option interface 'servers'
        option start '100'
        option limit '150'
        option leasetime '12h'
        list dhcp_option '6,192.168.1.10'

config dhcp 'iot'
        option interface 'iot'
        option start '100'
        option limit '150'
        option leasetime '12h'
        list dhcp_option '6,192.168.1.10'

config domain
        option name 'orangepi4-lts'
        option ip '192.168.1.10'

config domain
        option name 'gorilla'
        option ip '192.168.1.100'

config domain
        option name 'tiger'
        option ip '192.168.1.101'

config domain
        option name 'seagate-d4'
        option ip '192.168.10.199'

config domain
        option name 'proxmox'
        option ip '192.168.10.200'

config host
        option name 'OrangePi4-LTS'
        option dns '1'
        option mac 'MAC_ADDRESS
        option ip '192.168.1.10'

config host
        option mac 'MAC_ADDRESS'
        option ip '192.168.1.50'
        option name 'Asus-OpenWRT'
        option dns '1'

config host
        option name 'Gorilla'
        option dns '1'
        option mac 'MAC_ADDRESS'
        option ip '192.168.1.100'
        option leasetime '24'

config host
        option name 'tiger'
        option ip '192.168.1.101'
        option mac 'MAC_ADDRESS'

config host
        option ip '192.168.1.198'
        option name 'Printer'
        option dns '1'
        option mac 'MAC_ADDRESS'

config host
        option name 'Note10Lite'
        option dns '1'
        option mac 'MAC_ADDRESS'
        option ip '192.168.1.220'

config host
        option name 'Tab6Lite'
        option dns '1'
        option mac 'MAC_ADDRESS'
        option ip '192.168.1.221'

config host
        option mac 'MAC_ADDRESS'
        option name 'seagate-D4'
        option dns '1'
        option ip '192.168.10.199'
        option leasetime '24'

config host
        option name 'proxmox'
        option dns '1'
        option ip '192.168.10.200'
        option mac 'MAC_ADDRESS'

config host
        option name 'tv-bedroom'
        option dns '1'
        option mac 'MAC_ADDRESS'
        option ip '192.168.20.102'

config host
        option name 'tv-lounge'
        option dns '1'
        option mac 'MAC_ADDRESS'
        option ip '192.168.20.114'

config host
        option name 'tv-guestroom'
        option dns '1'
        option mac 'MAC_ADDRESS'
        option ip '192.168.20.129'

config host
        option name 'IPC-Upstairs'
        option dns '1'
        option mac 'MAC_ADDRESS'
        option ip '192.168.20.148'

config host
        option name 'tv-office'
        option dns '1'
        option mac 'MAC_ADDRESS'
        option ip '192.168.20.171'

config host
        option name 'floater'
        option dns '1'
        option mac 'MAC_ADDRESS'
        option ip '192.168.20.200'


root@Linksys_OpenWrt:~# cat /etc/config/firewall

config defaults
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option synflood_protect '1'

config zone
        option name 'lan'
        list network 'lan'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'

config zone
        option name 'servers'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'
        list network 'servers'

config zone
        option name 'guest'
        option output 'ACCEPT'
        list network 'iot'
        option forward 'REJECT'
        option input 'REJECT'

config zone
        option name 'wan'
        list network 'wan'
        list network 'wan6'
        option input 'REJECT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option masq '1'
        option mtu_fix '1'

config forwarding
        option src 'lan'
        option dest 'wan'

config rule
        option name 'Allow-DHCP-Renew'
        option src 'wan'
        option proto 'udp'
        option dest_port '68'
        option target 'ACCEPT'
        option family 'ipv4'

config rule
        option name 'Allow-Ping'
        option src 'wan'
        option proto 'icmp'
        option icmp_type 'echo-request'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-IGMP'
        option src 'wan'
        option proto 'igmp'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-DHCPv6'
        option src 'wan'
        option proto 'udp'
        option dest_port '546'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-MLD'
        option src 'wan'
        option proto 'icmp'
        option src_ip 'fe80::/10'
        list icmp_type '130/0'
        list icmp_type '131/0'
        list icmp_type '132/0'
        list icmp_type '143/0'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Input'
        option src 'wan'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        list icmp_type 'router-solicitation'
        list icmp_type 'neighbour-solicitation'
        list icmp_type 'router-advertisement'
        list icmp_type 'neighbour-advertisement'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Forward'
        option src 'wan'
        option dest '*'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-IPSec-ESP'
        option src 'wan'
        option dest 'lan'
        option proto 'esp'
        option target 'ACCEPT'

config rule
        option name 'Allow-ISAKMP'
        option src 'wan'
        option dest 'lan'
        option dest_port '500'
        option proto 'udp'
        option target 'ACCEPT'

config rule
        option name 'Guest DHCP and DNS'
        option src 'guest'
        option dest_port '53 67 68'
        option target 'ACCEPT'

config forwarding
        option src 'guest'
        option dest 'wan'

config forwarding
        option src 'servers'
        option dest 'wan'

config redirect
        option target 'DNAT'
        option name 'http'
        option src 'wan'
        option src_dport '80'
        option dest_port '8001'
        option dest 'servers'
        option dest_ip '192.168.10.100'

config redirect
        option target 'DNAT'
        option name 'https'
        option src 'wan'
        option src_dport '443'
        option dest_port '443'
        option dest 'servers'
        option dest_ip '192.168.10.100'

config rule
        option name 'Guest DNS and DNS PiHole'
        option dest_port '53 67 68'
        option target 'ACCEPT'
        option src 'guest'
        list dest_ip '192.168.1.10'
        option dest 'lan'

config rule
        option src 'servers'
        option src_port '53 67 68'
        option dest 'lan'
        list dest_ip '192.168.1.10'
        option dest_port '53 67 68'
        option target 'ACCEPT'
        option name 'Server DNS and DHCP Pihole'

config rule
        option name 'Allow LAN SMB Access to TrueNAS 137-138'
        list proto 'udp'
        option src 'lan'
        option dest 'servers'
        list dest_ip '192.168.10.250'
        option dest_port '137-138'
        option target 'ACCEPT'
        option enabled '0'

config rule
        option name 'Allow LAN SMB Access to TrueNAS 139'
        list proto 'tcp'
        option src 'lan'
        option dest 'servers'
        list dest_ip '192.168.10.250'
        option dest_port '139'
        option target 'ACCEPT'
        option enabled '0'

config rule
        option name 'Allow LAN SMB Access to TrueNAS  445'
        list proto 'tcp'
        option src 'lan'
        option dest 'servers'
        list dest_ip '192.168.10.250'
        option dest_port '445'
        option target 'ACCEPT'
        option enabled '0'

config forwarding
        option src 'lan'
        option dest 'servers'

config forwarding
        option src 'servers'
        option dest 'lan'

config forwarding
        option src 'lan'
        option dest 'guest'

config redirect
        option dest 'lan'
        option target 'DNAT'
        option name 'empyrion'
        option src 'wan'
        option src_dport '30000-30004'
        option dest_port '30000-30004'
        option dest_ip '192.168.1.101'

config redirect
        option dest 'lan'
        option target 'DNAT'
        option name 'Space Engineers'
        option src 'wan'
        option src_dport '27016'
        option dest_ip '192.168.1.101'
        option dest_port '27016'

config redirect
        option dest 'servers'
        option target 'DNAT'
        option name 'v-rising'
        option src 'wan'
        option src_dport '27010-27011'
        option dest_ip '192.168.10.200'
        option dest_port '27010-27011'

config redirect
        option dest 'servers'
        option target 'DNAT'
        option name '7days'
        option src 'wan'
        option src_dport '27050-27052'
        option dest_ip '192.168.10.200'
        option dest_port '27050-27052'

config redirect
        option dest 'servers'
        option target 'DNAT'
        option name 'factorio'
        option src 'wan'
        option dest_ip '192.168.10.200'
        option src_dport '27000-27001'
        option dest_port '27000-27001'

config redirect
        option dest 'servers'
        option target 'DNAT'
        option name 'valheim'
        option src 'wan'
        option src_dport '27020-27021'
        option dest_ip '192.168.10.200'
        option dest_port '27020-27021'

config redirect
        option dest 'servers'
        option target 'DNAT'
        option name 'starbound'
        option src 'wan'
        option src_dport '27025'
        option dest_ip '192.168.10.200'
        option dest_port '27025'

config redirect
        option target 'DNAT'
        option name 'wireguard'
        list proto 'udp'
        option src 'wan'
        option src_dport '51820'
        option dest_port '51820'
        option dest 'lan'
        option dest_ip '192.168.1.10'

config rule
        option name 'tailscaleDNS'
        option dest 'servers'
        option dest_port '53 67 68'
        option target 'ACCEPT'
        option src '*'
        list dest_ip '192.168.10.10'
        option enabled '0'

config redirect
        option dest 'servers'
        option target 'DNAT'
        option name 'Stationeers'
        option src 'wan'
        option src_dport '27030-27031'
        option dest_ip '192.168.10.200'
        option dest_port '27030-27031'

config rule
        option name 'SteamLink-UDP'
        list proto 'udp'
        list src_ip '192.168.1.100'
        option src_port '27031-27036'
        option dest_port '27031-27036'
        option target 'ACCEPT'
        list dest_ip '192.168.20.114'
        list dest_ip '192.168.20.102'
        list dest_ip '192.168.20.171'
        option src 'lan'
        option dest 'guest'

config rule
        option name 'SteamLink-TCP'
        list proto 'tcp'
        list src_ip '192.168.1.100'
        option src_port '27036-27037'
        option dest_port '27036-27037'
        option target 'ACCEPT'
        option src 'lan'
        option dest 'guest'
        list dest_ip '192.168.20.114'
        list dest_ip '192.168.20.102'
        list dest_ip '192.168.20.171'

config redirect
        option dest 'servers'
        option target 'DNAT'
        option name 'Icarus'
        list proto 'udp'
        option src 'wan'
        option src_dport '27040-27042'
        option dest_ip '192.168.10.200'
        option dest_port '27040-27042'

config redirect
        option dest 'servers'
        option target 'DNAT'
        option name 'Sons Of Forest'
        option src 'wan'
        option src_dport '27060-27062'
        option dest_ip '192.168.10.200'
        option dest_port '27060-27062'
        list proto 'tcp'
        list proto 'udp'

config redirect
        option dest 'servers'
        option target 'DNAT'
        option name 'Project Zomboid'
        option src 'wan'
        option src_dport '27035-27036'
        option dest_ip '192.168.10.200'
        option dest_port '27035-27036'

config redirect
        option dest 'servers'
        option target 'DNAT'
        option name 'Astroneer'
        option src 'wan'
        option src_dport '27035'
        option dest_ip '192.168.10.200'
        option dest_port '27035'

config rule
        option name 'pialert'
        option src 'servers'
        list src_ip '192.168.10.10'
        option dest '*'
        option target 'ACCEPT'

config rule
        option name 'Allow_IoTChromecastTCP'
        list proto 'tcp'
        option src 'guest'
        option dest 'lan'
        option target 'ACCEPT'
        option src_port '8008 8009 8443'

config rule
        option name 'Allow_IoTChromecastUDP'
        list proto 'udp'
        option src 'guest'
        option dest 'lan'
        option target 'ACCEPT'
        option src_port '32768-61000'

config rule
        option name 'Allow_IoTChromecastInboundUDP'
        list proto 'udp'
        option src 'guest'
        option dest 'lan'
        option target 'ACCEPT'
        option dest_port ' 32768-61000'

config rule
        option name 'Avahi-daemonRelay'
        list proto 'udp'
        option src '*'
        option src_port '5353'
        option dest '*'
        list dest_ip '224.0.0.251'
        option target 'ACCEPT'
        option dest_port '5353'

config redirect
        option dest 'servers'
        option target 'DNAT'
        option name 'SpaceEngineersDocker'
        option src 'wan'
        option dest_ip '192.168.10.200'
        option src_dport '27032-27033'
        option dest_port '27032-27033'

Edit: a MAC adress

8

The ports should definitely be gigabit, as I was using this router quite a while back before I redid my network for VLANS and had everything on one subnet and didn't notice any issues.

root@Linksys_OpenWrt:~# ethtool eth0
Settings for eth0:
        Supported ports: [ MII ]
        Supported link modes:   1000baseT/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  1000baseT/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Link partner advertised link modes:  1000baseT/Full
        Link partner advertised pause frame use: No
        Link partner advertised auto-negotiation: No
        Link partner advertised FEC modes: Not reported
        Speed: 1000Mb/s
        Duplex: Full
        Port: MII
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: on
        Supports Wake-on: d
        Wake-on: d
        Link detected: yes
root@Linksys_OpenWrt:~# ethtool wan
Settings for wan:
        Supported ports: [ TP MII ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Link partner advertised link modes:  10baseT/Half 10baseT/Full
                                             100baseT/Half 100baseT/Full
                                             1000baseT/Half 1000baseT/Full
        Link partner advertised pause frame use: No
        Link partner advertised auto-negotiation: Yes
        Link partner advertised FEC modes: Not reported
        Speed: 1000Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 4
        Transceiver: external
        Auto-negotiation: on
        MDI-X: Unknown
        Supports Wake-on: d
        Wake-on: d
        Link detected: yes
9

If you put the windows machine on the server network, do you get the expected 1Gbps? And what about vice versa?

10

Do you mean windows to windows with one being on the server network? I can test that. But I would expect the drop down to 100mbit when copying between each other with the speedtest.net still being up to 1Gigabit.

I will post the results in a few minutes, just gotta run some cables and crawl under desks.. :smiley:

11

Windows to windows across the VLANs. Then windows to server with the two on the same network.

12

So I switched one of the windows boxes to server network and as expected it behaved like I thought.

Copy windows to windows was at the 100mbit rate, but from server was at 1gbit rate. I ran some iperf to main Server on Server Land, and OrangePi on main lan. It behaved like all the previous boxes, so it is not a windows vs linux setup.

Speedtest.net results was in the Gigabit range as well, pretty much mirroring all the previous results.

This is pretty weird. as from internet speed tests the bandwidth is there because it needs to go through the cabling and ports, but across local vlans it freaks out.

PS C:\Users\talung> ipconfig

Windows IP Configuration


Ethernet adapter Ethernet 2:

   Connection-specific DNS Suffix  . : lan
   IPv6 Address. . . . . . . . . . . : 
   Temporary IPv6 Address. . . . . . : 
   Temporary IPv6 Address. . . . . . : <removed>
   Temporary IPv6 Address. . . . . . : 
   Link-local IPv6 Address . . . . . : 
   IPv4 Address. . . . . . . . . . . : 192.168.1.101
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 192.168.1.1

Ethernet adapter vEthernet (WSL):

   Connection-specific DNS Suffix  . :
   Link-local IPv6 Address . . . . . : <removed>
   IPv4 Address. . . . . . . . . . . : 172.19.224.1
   Subnet Mask . . . . . . . . . . . : 255.255.240.0
   Default Gateway . . . . . . . . . :
PS C:\Users\talung> ipconfig

Windows IP Configuration


Ethernet adapter Ethernet 2:

   Connection-specific DNS Suffix  . : lan
   Link-local IPv6 Address . . . . . : <removed>
   IPv4 Address. . . . . . . . . . . : 192.168.10.107
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 192.168.10.1

Ethernet adapter vEthernet (WSL):

   Connection-specific DNS Suffix  . :
   Link-local IPv6 Address . . . . . : <removed>
   IPv4 Address. . . . . . . . . . . : 172.19.224.1
   Subnet Mask . . . . . . . . . . . : 255.255.240.0
   Default Gateway . . . . . . . . . :
PS C:\Users\talung>

Windows machine to Server on same VLAN

root@proxmox:~# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  128 KByte (default)
------------------------------------------------------------
[  1] local 192.168.10.200 port 5001 connected with 192.168.10.107 port 65266 (icwnd/mss/irtt=14/1448/500)
[ ID] Interval       Transfer     Bandwidth
[  1] 0.0000-10.0336 sec  1.10 GBytes   941 Mbits/sec
[  2] local 192.168.10.200 port 5001 connected with 192.168.10.107 port 65268 (icwnd/mss/irtt=14/1448/282)
[ ID] Interval       Transfer     Bandwidth
[  2] 0.0000-10.0256 sec  1.10 GBytes   941 Mbits/sec

Windows machine to OrangePi4 on lan subnet

root@orangepi4-lts:~# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  128 KByte (default)
------------------------------------------------------------
[  4] local 192.168.1.10 port 5001 connected with 192.168.10.107 port 65204
[ ID] Interval       Transfer     Bandwidth
[  4] 0.0000-10.2642 sec   114 MBytes  93.2 Mbits/sec
[  5] local 192.168.1.10 port 5001 connected with 192.168.10.107 port 65206
[ ID] Interval       Transfer     Bandwidth
[  5] 0.0000-10.2346 sec   114 MBytes  93.1 Mbits/sec
13

Can you draw a diagram of your network topology? You've got unmanaged switches in the system -- I'd love to see how these are connected and used.

Please mark each device with the brand/model and the network(s) that are carried on each link.

14

Ok Just threw something together:

CurrentLayout
CurrentLayout2484×1200 170 KB

I really didn't think this was a complex layout.

EDIT: It has been running for months without too much issue, just never noticed the speed issue between VLANS before as backups happen at night in the background using things like rsync etc. I only noticed it with a 60GB clone of one of my git repositories off Gitea.

15

To rule out the stiff downstream of the router, can you try connecting a pc directly to port 2 and the server to port 3. Then run both the Speedtest to the internet and iperf tests between the pc and server.

16

Ok, so I connected a linux Laptop to the Server Switch, and ran all the tests. Speedtest to net was 930mbits, and iperf from main pc was 92mbit.

talung@Gorilla:~$ iperf -c 192.168.10.161
------------------------------------------------------------
Client connecting to 192.168.10.161, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  1] local 172.29.156.168 port 48936 connected with 192.168.10.161 port 5001
[ ID] Interval       Transfer     Bandwidth
[  1] 0.0000-10.3528 sec   114 MBytes  92.4 Mbits/sec
talung@Gorilla:~$ iperf -c 192.168.10.161
------------------------------------------------------------
Client connecting to 192.168.10.161, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  1] local 172.29.156.168 port 48938 connected with 192.168.10.161 port 5001
[ ID] Interval       Transfer     Bandwidth
[  1] 0.0000-10.3026 sec   114 MBytes  93.0 Mbits/sec
talung@Gorilla:~$

Then I connect same laptop directly to port 3 and got the same results:

talung@Gorilla:~$ iperf -c 192.168.10.161
------------------------------------------------------------
Client connecting to 192.168.10.161, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  1] local 172.29.156.168 port 48940 connected with 192.168.10.161 port 5001
[ ID] Interval       Transfer     Bandwidth
[  1] 0.0000-10.3156 sec   114 MBytes  93.0 Mbits/sec
talung@Gorilla:~$ iperf -c 192.168.10.161
------------------------------------------------------------
Client connecting to 192.168.10.161, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  1] local 172.29.156.168 port 48942 connected with 192.168.10.161 port 5001
[ ID] Interval       Transfer     Bandwidth
[  1] 0.0000-10.2666 sec   114 MBytes  92.8 Mbits/sec
talung@Gorilla:~$

The IP remained the same because of DHCP. With PiHole doing my DNS is was a real pain trying the other way. Pihole is sitting in the 192.168.1.x subnet.

EDIT: As an aside, if it was the cabling or switches, surely I would never be able to get gigabit speeds to the internet from the "bad switches" etc.

17

Ok.. Here is something really interesting.

So with the laptop I decided to connect to the wireless network on the Router, only use for phones etc. avoid for any of my devices. Obviously was not getting gigabit speeds to the internet, round about 400mbit.

But I decided to use iperf to the server to see how it would do.

root@proxmox:~# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  128 KByte (default)
------------------------------------------------------------
[  1] local 192.168.10.200 port 5001 connected with 192.168.1.169 port 36026 (icwnd/mss/irtt=14/1448                                 /921)
[ ID] Interval       Transfer     Bandwidth
[  1] 0.0000-10.0506 sec   527 MBytes   440 Mbits/sec
[  2] local 192.168.10.200 port 5001 connected with 192.168.1.169 port 52514 (icwnd/mss/irtt=14/1448                                 /1013)
[ ID] Interval       Transfer     Bandwidth
[  2] 0.0000-10.0437 sec   545 MBytes   455 Mbits/sec
[  3] local 192.168.10.200 port 5001 connected with 192.168.1.100 port 64134 (icwnd/mss/irtt=14/1448/619)
[ ID] Interval       Transfer     Bandwidth
[  3] 0.0000-10.2557 sec   113 MBytes  92.7 Mbits/sec
[  4] local 192.168.10.200 port 5001 connected with 192.168.1.100 port 64136 (icwnd/mss/irtt=14/1448/630)
[ ID] Interval       Transfer     Bandwidth
[  4] 0.0000-10.2356 sec   113 MBytes  92.8 Mbits/sec

It was scoring way higher than the Wired connection. I am starting to think that there maybe an issue between the PC switch and router cable. Going to be testing that out. and let you know.

EDIT: Narrator: It was not the cable...

I did however try plugging into the second router and doing the tests. EVERYTHING was limited to 100mbit. That does go through a tagged line between the 2 routers. Again, this leads me to believe the VLAN is somehow throttling down to 100mbits.

18

I SOLVED IT!!!!

So after the last test, where I plugged the laptop into the other router and was getting 100mbit, I started looking at port assignments in the br-lan device.

vlan_ports
vlan_ports967×571 28 KB

And I noticed something... lan4 was only at 100FD instead of all the others at 1000FD. That connection being slow doesn't really bother me as it is just for a TV and "Guest" wifi, but I decided to unplug it try the tests again...

[ 11] local 192.168.10.200 port 5001 connected with 192.168.1.100 port 64048 (icwnd/mss/irtt=14/1448/618)
[ ID] Interval       Transfer     Bandwidth
[ 11] 0.0000-10.0377 sec   872 MBytes   728 Mbits/sec
[ 12] local 192.168.10.200 port 5001 connected with 192.168.1.100 port 64050 (icwnd/mss/irtt=14/1448/626)
[ ID] Interval       Transfer     Bandwidth
[ 12] 0.0000-10.0333 sec   876 MBytes   732 Mbits/sec

I broke the 100mbit mark!

Now I need to figure out why that is 100 vs 1000 and also why should that slow my entire network?

EDIT: Found the issue a small cable patch (20cm) that linked two ports hidden along the way down to the router was faulty... this caused the lower speeds.. all links are now running at 1000mbit.

1 Like
19

I recall a series of threads where the switch would drop down to 100Mbps if any singular port was operating at 100Mbps (even if all others were at gigabit). I think it was a bug in the switch configuration (low level code), but I don't recall exactly what happened in the end (a search may bring up more info).

But yes, a cable that is faulty could obviously cause erratic and intermittent behavior like this. I can't explain exactly why it only happened when the routing was occurring between your two VLANs and never from one of your lans to the internet... that would be an interesting thing to dig into, but probably not worth the effort except as an academic exercise.

Glad you found the culprit!

20

Thanks for you help and diagnostic skills I was able to pin it down. While I am still new to VLANS, It now appears to me that the speed of a VLAN is dependent on its lowest trunk/tagged line.

While some ports on the switches do run at 100mbit (some TV/camera) stuff, none of this was an issue, just the single tagged line. This will definitely help in future.

I have ordered a Managed switch to replace all the unmanaged ones (TP-Link TL-SG3428), so hopefully this will help make my setup a lot neater (and power efficient) and easier to diagnose. :slight_smile: