wait the RPI is on port 3? I thought it was on port 1? if it's on port 3 then you need to make port 3 be a tagged member of vlan 3... make vlan 3 be tagged on port 3 and untagged on port 5, that will fix it!!
lol yes it's always been on PORT 3. Port 1 is the TP-LINK media converter (the 1gig FTTH)
I'll give that a try!
THERE WE GO! That did it lol - It was that the entire time! I can now obtain an IP address on this new Rpi4 interface (eth1.3) !!
All this time.. lol - Oh well.. I didn't clue in on that either. Thanks so much for your help troubleshooting all of this!! 
So now I can simply do iperf3 tests between these 2 networks right? Which will make everything go through both LAN and WAN ports of my Rpi4 right?
If so, how can they communicate between each other? My linuxbox on my main LAN can't see my laptop on my GuestLAN.
it will go through both ethernet devices yes. But you have now two LANs, "testlan" which is just port 5 on your switch... and whatever existing lan you had.
if the lan and testlan interfaces are in the same firewall zone, you should be able to see each other, pinging from the machine on port 5 to a machine on your regular lan should work for example.
ok, so FROM the testlan laptop (192.168.20.121) I can ping ALL my devices on my regular LAN (192.168.10.x). From my Rpi4 I can ping the laptop on the testlan (192.168.20.121), however none of my regular LAN clients/devices can ping the testlan laptop (192.168.20.121)
since I can at least reach my regular LAN from my testlan I did a quick iperf speed test!
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.20.121, port 52140
[ 5] local 192.168.76.8 port 5201 connected to 192.168.20.121 port 52141
[ 8] local 192.168.76.8 port 5201 connected to 192.168.20.121 port 52142
[ 10] local 192.168.76.8 port 5201 connected to 192.168.20.121 port 52143
[ 12] local 192.168.76.8 port 5201 connected to 192.168.20.121 port 52144
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 6.96 MBytes 58.4 Mbits/sec
[ 8] 0.00-1.00 sec 6.38 MBytes 53.5 Mbits/sec
[ 10] 0.00-1.00 sec 30.0 MBytes 252 Mbits/sec
[ 12] 0.00-1.00 sec 28.4 MBytes 238 Mbits/sec
[SUM] 0.00-1.00 sec 71.8 MBytes 602 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 1.00-2.00 sec 28.3 MBytes 237 Mbits/sec
[ 8] 1.00-2.00 sec 28.1 MBytes 236 Mbits/sec
[ 10] 1.00-2.00 sec 26.7 MBytes 224 Mbits/sec
[ 12] 1.00-2.00 sec 28.8 MBytes 242 Mbits/sec
[SUM] 1.00-2.00 sec 112 MBytes 939 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 2.00-3.00 sec 24.4 MBytes 204 Mbits/sec
[ 8] 2.00-3.00 sec 31.8 MBytes 266 Mbits/sec
[ 10] 2.00-3.00 sec 24.2 MBytes 203 Mbits/sec
[ 12] 2.00-3.00 sec 31.6 MBytes 265 Mbits/sec
[SUM] 2.00-3.00 sec 112 MBytes 939 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 3.00-4.00 sec 24.3 MBytes 204 Mbits/sec
[ 8] 3.00-4.00 sec 31.9 MBytes 267 Mbits/sec
[ 10] 3.00-4.00 sec 24.3 MBytes 204 Mbits/sec
[ 12] 3.00-4.00 sec 31.9 MBytes 267 Mbits/sec
[SUM] 3.00-4.00 sec 112 MBytes 942 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 4.00-5.00 sec 23.8 MBytes 200 Mbits/sec
[ 8] 4.00-5.00 sec 32.4 MBytes 272 Mbits/sec
[ 10] 4.00-5.00 sec 23.6 MBytes 198 Mbits/sec
[ 12] 4.00-5.00 sec 32.2 MBytes 271 Mbits/sec
[SUM] 4.00-5.00 sec 112 MBytes 940 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 5.00-6.00 sec 17.1 MBytes 144 Mbits/sec
[ 8] 5.00-6.00 sec 21.1 MBytes 177 Mbits/sec
[ 10] 5.00-6.00 sec 16.8 MBytes 141 Mbits/sec
[ 12] 5.00-6.00 sec 21.5 MBytes 180 Mbits/sec
[SUM] 5.00-6.00 sec 76.5 MBytes 641 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 6.00-7.00 sec 25.7 MBytes 215 Mbits/sec
[ 8] 6.00-7.00 sec 30.5 MBytes 256 Mbits/sec
[ 10] 6.00-7.00 sec 25.4 MBytes 213 Mbits/sec
[ 12] 6.00-7.00 sec 30.5 MBytes 256 Mbits/sec
[SUM] 6.00-7.00 sec 112 MBytes 941 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 7.00-8.00 sec 28.1 MBytes 236 Mbits/sec
[ 8] 7.00-8.00 sec 28.1 MBytes 236 Mbits/sec
[ 10] 7.00-8.00 sec 28.2 MBytes 236 Mbits/sec
[ 12] 7.00-8.00 sec 27.9 MBytes 234 Mbits/sec
[SUM] 7.00-8.00 sec 112 MBytes 942 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 8.00-9.00 sec 24.1 MBytes 202 Mbits/sec
[ 8] 8.00-9.00 sec 31.9 MBytes 268 Mbits/sec
[ 10] 8.00-9.00 sec 24.0 MBytes 202 Mbits/sec
[ 12] 8.00-9.00 sec 31.9 MBytes 267 Mbits/sec
[SUM] 8.00-9.00 sec 112 MBytes 939 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 9.00-10.00 sec 21.6 MBytes 181 Mbits/sec
[ 8] 9.00-10.00 sec 7.90 MBytes 66.3 Mbits/sec
[ 10] 9.00-10.00 sec 21.5 MBytes 180 Mbits/sec
[ 12] 9.00-10.00 sec 27.1 MBytes 227 Mbits/sec
[SUM] 9.00-10.00 sec 78.1 MBytes 655 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 10.00-11.00 sec 25.3 MBytes 212 Mbits/sec
[ 8] 10.00-11.00 sec 27.2 MBytes 228 Mbits/sec
[ 10] 10.00-11.00 sec 25.2 MBytes 212 Mbits/sec
[ 12] 10.00-11.00 sec 34.4 MBytes 289 Mbits/sec
[SUM] 10.00-11.00 sec 112 MBytes 941 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 11.00-12.00 sec 25.5 MBytes 214 Mbits/sec
[ 8] 11.00-12.00 sec 30.9 MBytes 259 Mbits/sec
[ 10] 11.00-12.00 sec 25.4 MBytes 213 Mbits/sec
[ 12] 11.00-12.00 sec 30.2 MBytes 254 Mbits/sec
[SUM] 11.00-12.00 sec 112 MBytes 940 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 12.00-13.00 sec 26.2 MBytes 220 Mbits/sec
[ 8] 12.00-13.00 sec 29.8 MBytes 250 Mbits/sec
[ 10] 12.00-13.00 sec 26.2 MBytes 220 Mbits/sec
[ 12] 12.00-13.00 sec 29.9 MBytes 251 Mbits/sec
[SUM] 12.00-13.00 sec 112 MBytes 941 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 13.00-14.00 sec 24.3 MBytes 204 Mbits/sec
[ 8] 13.00-14.00 sec 31.7 MBytes 266 Mbits/sec
[ 10] 13.00-14.00 sec 24.2 MBytes 203 Mbits/sec
[ 12] 13.00-14.00 sec 31.6 MBytes 265 Mbits/sec
[SUM] 13.00-14.00 sec 112 MBytes 939 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 14.00-15.00 sec 16.8 MBytes 141 Mbits/sec
[ 8] 14.00-15.00 sec 23.8 MBytes 200 Mbits/sec
[ 10] 14.00-15.00 sec 17.2 MBytes 145 Mbits/sec
[ 12] 14.00-15.00 sec 52.1 MBytes 437 Mbits/sec
[SUM] 14.00-15.00 sec 110 MBytes 922 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 15.00-16.00 sec 26.0 MBytes 218 Mbits/sec
[ 8] 15.00-16.00 sec 32.8 MBytes 275 Mbits/sec
[ 10] 15.00-16.00 sec 27.6 MBytes 232 Mbits/sec
[ 12] 15.00-16.00 sec 25.6 MBytes 215 Mbits/sec
[SUM] 15.00-16.00 sec 112 MBytes 940 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 16.00-17.00 sec 28.1 MBytes 236 Mbits/sec
[ 8] 16.00-17.00 sec 28.2 MBytes 237 Mbits/sec
[ 10] 16.00-17.00 sec 28.1 MBytes 236 Mbits/sec
[ 12] 16.00-17.00 sec 27.9 MBytes 234 Mbits/sec
[SUM] 16.00-17.00 sec 112 MBytes 942 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 17.00-18.00 sec 25.0 MBytes 210 Mbits/sec
[ 8] 17.00-18.00 sec 31.1 MBytes 261 Mbits/sec
[ 10] 17.00-18.00 sec 25.0 MBytes 210 Mbits/sec
[ 12] 17.00-18.00 sec 31.1 MBytes 261 Mbits/sec
[SUM] 17.00-18.00 sec 112 MBytes 941 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 18.00-19.00 sec 26.5 MBytes 222 Mbits/sec
[ 8] 18.00-19.00 sec 29.5 MBytes 247 Mbits/sec
[ 10] 18.00-19.00 sec 26.5 MBytes 222 Mbits/sec
[ 12] 18.00-19.00 sec 29.4 MBytes 246 Mbits/sec
[SUM] 18.00-19.00 sec 112 MBytes 938 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 19.00-20.00 sec 21.0 MBytes 176 Mbits/sec
[ 8] 19.00-20.00 sec 20.2 MBytes 170 Mbits/sec
[ 10] 19.00-20.00 sec 19.8 MBytes 166 Mbits/sec
[ 12] 19.00-20.00 sec 19.2 MBytes 161 Mbits/sec
[SUM] 19.00-20.00 sec 80.3 MBytes 673 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 20.00-21.00 sec 34.0 MBytes 285 Mbits/sec
[ 8] 20.00-21.00 sec 27.4 MBytes 230 Mbits/sec
[ 10] 20.00-21.00 sec 25.1 MBytes 210 Mbits/sec
[ 12] 20.00-21.00 sec 25.5 MBytes 214 Mbits/sec
[SUM] 20.00-21.00 sec 112 MBytes 939 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 21.00-22.00 sec 28.0 MBytes 235 Mbits/sec
[ 8] 21.00-22.00 sec 28.7 MBytes 240 Mbits/sec
[ 10] 21.00-22.00 sec 22.4 MBytes 188 Mbits/sec
[ 12] 21.00-22.00 sec 32.9 MBytes 276 Mbits/sec
[SUM] 21.00-22.00 sec 112 MBytes 939 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 22.00-23.00 sec 28.1 MBytes 236 Mbits/sec
[ 8] 22.00-23.00 sec 28.2 MBytes 237 Mbits/sec
[ 10] 22.00-23.00 sec 27.9 MBytes 234 Mbits/sec
[ 12] 22.00-23.00 sec 28.1 MBytes 235 Mbits/sec
[SUM] 22.00-23.00 sec 112 MBytes 942 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 23.00-24.00 sec 29.0 MBytes 243 Mbits/sec
[ 8] 23.00-24.00 sec 28.8 MBytes 241 Mbits/sec
[ 10] 23.00-24.00 sec 26.0 MBytes 218 Mbits/sec
[ 12] 23.00-24.00 sec 28.5 MBytes 239 Mbits/sec
[SUM] 23.00-24.00 sec 112 MBytes 941 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 24.00-25.00 sec 17.6 MBytes 148 Mbits/sec
[ 8] 24.00-25.00 sec 21.0 MBytes 176 Mbits/sec
[ 10] 24.00-25.00 sec 17.4 MBytes 146 Mbits/sec
[ 12] 24.00-25.00 sec 21.4 MBytes 179 Mbits/sec
[SUM] 24.00-25.00 sec 77.4 MBytes 650 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 25.00-26.00 sec 29.2 MBytes 245 Mbits/sec
[ 8] 25.00-26.00 sec 29.8 MBytes 250 Mbits/sec
[ 10] 25.00-26.00 sec 21.4 MBytes 180 Mbits/sec
[ 12] 25.00-26.00 sec 30.9 MBytes 259 Mbits/sec
[SUM] 25.00-26.00 sec 111 MBytes 934 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 26.00-27.00 sec 25.0 MBytes 210 Mbits/sec
[ 8] 26.00-27.00 sec 31.3 MBytes 263 Mbits/sec
[ 10] 26.00-27.00 sec 24.5 MBytes 206 Mbits/sec
[ 12] 26.00-27.00 sec 31.3 MBytes 262 Mbits/sec
[SUM] 26.00-27.00 sec 112 MBytes 940 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 27.00-28.00 sec 27.8 MBytes 233 Mbits/sec
[ 8] 27.00-28.00 sec 28.4 MBytes 238 Mbits/sec
[ 10] 27.00-28.00 sec 27.6 MBytes 232 Mbits/sec
[ 12] 27.00-28.00 sec 28.4 MBytes 238 Mbits/sec
[SUM] 27.00-28.00 sec 112 MBytes 941 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 28.00-29.00 sec 27.6 MBytes 232 Mbits/sec
[ 8] 28.00-29.00 sec 28.4 MBytes 238 Mbits/sec
[ 10] 28.00-29.00 sec 27.8 MBytes 233 Mbits/sec
[ 12] 28.00-29.00 sec 28.4 MBytes 238 Mbits/sec
[SUM] 28.00-29.00 sec 112 MBytes 941 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 29.00-30.00 sec 23.8 MBytes 200 Mbits/sec
[ 8] 29.00-30.00 sec 23.4 MBytes 197 Mbits/sec
[ 10] 29.00-30.00 sec 35.4 MBytes 297 Mbits/sec
[ 12] 29.00-30.00 sec 24.2 MBytes 203 Mbits/sec
[SUM] 29.00-30.00 sec 107 MBytes 897 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 30.00-31.00 sec 14.4 MBytes 121 Mbits/sec
[ 8] 30.00-31.00 sec 19.2 MBytes 161 Mbits/sec
[ 10] 30.00-31.00 sec 27.2 MBytes 228 Mbits/sec
[ 12] 30.00-31.00 sec 46.1 MBytes 386 Mbits/sec
[SUM] 30.00-31.00 sec 107 MBytes 897 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 31.00-32.00 sec 36.2 MBytes 303 Mbits/sec
[ 8] 31.00-32.00 sec 23.7 MBytes 198 Mbits/sec
[ 10] 31.00-32.00 sec 21.2 MBytes 178 Mbits/sec
[ 12] 31.00-32.00 sec 31.0 MBytes 260 Mbits/sec
[SUM] 31.00-32.00 sec 112 MBytes 939 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 32.00-33.00 sec 28.0 MBytes 235 Mbits/sec
[ 8] 32.00-33.00 sec 30.7 MBytes 258 Mbits/sec
[ 10] 32.00-33.00 sec 22.8 MBytes 191 Mbits/sec
[ 12] 32.00-33.00 sec 30.5 MBytes 256 Mbits/sec
[SUM] 32.00-33.00 sec 112 MBytes 940 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 33.00-34.00 sec 27.7 MBytes 233 Mbits/sec
[ 8] 33.00-34.00 sec 28.4 MBytes 238 Mbits/sec
[ 10] 33.00-34.00 sec 27.5 MBytes 231 Mbits/sec
[ 12] 33.00-34.00 sec 28.3 MBytes 237 Mbits/sec
[SUM] 33.00-34.00 sec 112 MBytes 939 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 34.00-35.00 sec 24.4 MBytes 205 Mbits/sec
[ 8] 34.00-35.00 sec 22.5 MBytes 188 Mbits/sec
[ 10] 34.00-35.00 sec 17.8 MBytes 149 Mbits/sec
[ 12] 34.00-35.00 sec 23.4 MBytes 197 Mbits/sec
[SUM] 34.00-35.00 sec 88.1 MBytes 739 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 35.00-36.00 sec 27.9 MBytes 234 Mbits/sec
[ 8] 35.00-36.00 sec 30.6 MBytes 257 Mbits/sec
[ 10] 35.00-36.00 sec 23.4 MBytes 196 Mbits/sec
[ 12] 35.00-36.00 sec 30.4 MBytes 255 Mbits/sec
[SUM] 35.00-36.00 sec 112 MBytes 942 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 36.00-37.00 sec 26.6 MBytes 223 Mbits/sec
[ 8] 36.00-37.00 sec 29.4 MBytes 246 Mbits/sec
[ 10] 36.00-37.00 sec 26.7 MBytes 224 Mbits/sec
[ 12] 36.00-37.00 sec 29.6 MBytes 249 Mbits/sec
[SUM] 36.00-37.00 sec 112 MBytes 942 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 37.00-38.00 sec 26.6 MBytes 223 Mbits/sec
[ 8] 37.00-38.00 sec 29.6 MBytes 249 Mbits/sec
[ 10] 37.00-38.00 sec 26.6 MBytes 223 Mbits/sec
[ 12] 37.00-38.00 sec 29.5 MBytes 247 Mbits/sec
[SUM] 37.00-38.00 sec 112 MBytes 942 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 38.00-39.00 sec 19.9 MBytes 167 Mbits/sec
[ 8] 38.00-39.00 sec 20.6 MBytes 173 Mbits/sec
[ 10] 38.00-39.00 sec 19.6 MBytes 165 Mbits/sec
[ 12] 38.00-39.00 sec 18.1 MBytes 152 Mbits/sec
[SUM] 38.00-39.00 sec 78.2 MBytes 656 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 39.00-40.00 sec 27.8 MBytes 233 Mbits/sec
[ 8] 39.00-40.00 sec 28.4 MBytes 238 Mbits/sec
[ 10] 39.00-40.00 sec 27.8 MBytes 233 Mbits/sec
[ 12] 39.00-40.00 sec 28.4 MBytes 238 Mbits/sec
[SUM] 39.00-40.00 sec 112 MBytes 942 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 40.00-41.00 sec 24.6 MBytes 207 Mbits/sec
[ 8] 40.00-41.00 sec 31.7 MBytes 266 Mbits/sec
[ 10] 40.00-41.00 sec 24.4 MBytes 205 Mbits/sec
[ 12] 40.00-41.00 sec 31.5 MBytes 264 Mbits/sec
[SUM] 40.00-41.00 sec 112 MBytes 941 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 41.00-42.00 sec 23.8 MBytes 200 Mbits/sec
[ 8] 41.00-42.00 sec 32.4 MBytes 272 Mbits/sec
[ 10] 41.00-42.00 sec 23.6 MBytes 198 Mbits/sec
[ 12] 41.00-42.00 sec 32.4 MBytes 272 Mbits/sec
[SUM] 41.00-42.00 sec 112 MBytes 940 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 42.00-43.00 sec 27.1 MBytes 227 Mbits/sec
[ 8] 42.00-43.00 sec 28.9 MBytes 243 Mbits/sec
[ 10] 42.00-43.00 sec 27.0 MBytes 226 Mbits/sec
[ 12] 42.00-43.00 sec 29.0 MBytes 243 Mbits/sec
[SUM] 42.00-43.00 sec 112 MBytes 940 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 43.00-44.00 sec 17.2 MBytes 144 Mbits/sec
[ 8] 43.00-44.00 sec 32.8 MBytes 275 Mbits/sec
[ 10] 43.00-44.00 sec 21.3 MBytes 179 Mbits/sec
[ 12] 43.00-44.00 sec 39.1 MBytes 328 Mbits/sec
[SUM] 43.00-44.00 sec 110 MBytes 926 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 44.00-45.00 sec 27.7 MBytes 232 Mbits/sec
[ 8] 44.00-45.00 sec 30.4 MBytes 255 Mbits/sec
[ 10] 44.00-45.00 sec 27.9 MBytes 234 Mbits/sec
[ 12] 44.00-45.00 sec 26.2 MBytes 220 Mbits/sec
[SUM] 44.00-45.00 sec 112 MBytes 941 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 45.00-46.00 sec 28.0 MBytes 235 Mbits/sec
[ 8] 45.00-46.00 sec 28.1 MBytes 236 Mbits/sec
[ 10] 45.00-46.00 sec 28.0 MBytes 235 Mbits/sec
[ 12] 45.00-46.00 sec 28.1 MBytes 236 Mbits/sec
[SUM] 45.00-46.00 sec 112 MBytes 942 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 46.00-47.00 sec 22.2 MBytes 186 Mbits/sec
[ 8] 46.00-47.00 sec 34.2 MBytes 287 Mbits/sec
[ 10] 46.00-47.00 sec 21.9 MBytes 184 Mbits/sec
[ 12] 46.00-47.00 sec 32.5 MBytes 273 Mbits/sec
[SUM] 46.00-47.00 sec 111 MBytes 931 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 47.00-48.00 sec 27.2 MBytes 228 Mbits/sec
[ 8] 47.00-48.00 sec 37.5 MBytes 315 Mbits/sec
[ 10] 47.00-48.00 sec 27.2 MBytes 228 Mbits/sec
[ 12] 47.00-48.00 sec 20.2 MBytes 170 Mbits/sec
[SUM] 47.00-48.00 sec 112 MBytes 941 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 48.00-49.00 sec 16.0 MBytes 135 Mbits/sec
[ 8] 48.00-49.00 sec 17.7 MBytes 149 Mbits/sec
[ 10] 48.00-49.00 sec 22.9 MBytes 192 Mbits/sec
[ 12] 48.00-49.00 sec 20.3 MBytes 170 Mbits/sec
[SUM] 48.00-49.00 sec 77.0 MBytes 646 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 49.00-50.00 sec 24.6 MBytes 206 Mbits/sec
[ 8] 49.00-50.00 sec 20.2 MBytes 169 Mbits/sec
[ 10] 49.00-50.00 sec 32.0 MBytes 269 Mbits/sec
[ 12] 49.00-50.00 sec 34.4 MBytes 288 Mbits/sec
[SUM] 49.00-50.00 sec 111 MBytes 932 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 50.00-51.00 sec 28.1 MBytes 236 Mbits/sec
[ 8] 50.00-51.00 sec 30.3 MBytes 254 Mbits/sec
[ 10] 50.00-51.00 sec 28.0 MBytes 235 Mbits/sec
[ 12] 50.00-51.00 sec 25.8 MBytes 216 Mbits/sec
[SUM] 50.00-51.00 sec 112 MBytes 941 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 51.00-52.00 sec 28.4 MBytes 238 Mbits/sec
[ 8] 51.00-52.00 sec 28.5 MBytes 239 Mbits/sec
[ 10] 51.00-52.00 sec 26.9 MBytes 225 Mbits/sec
[ 12] 51.00-52.00 sec 28.5 MBytes 239 Mbits/sec
[SUM] 51.00-52.00 sec 112 MBytes 941 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 52.00-53.00 sec 28.1 MBytes 236 Mbits/sec
[ 8] 52.00-53.00 sec 28.1 MBytes 236 Mbits/sec
[ 10] 52.00-53.00 sec 28.0 MBytes 235 Mbits/sec
[ 12] 52.00-53.00 sec 28.0 MBytes 235 Mbits/sec
[SUM] 52.00-53.00 sec 112 MBytes 942 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 53.00-54.00 sec 16.5 MBytes 138 Mbits/sec
[ 8] 53.00-54.00 sec 23.1 MBytes 194 Mbits/sec
[ 10] 53.00-54.00 sec 16.5 MBytes 138 Mbits/sec
[ 12] 53.00-54.00 sec 18.1 MBytes 152 Mbits/sec
[SUM] 53.00-54.00 sec 74.2 MBytes 623 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 54.00-55.00 sec 15.0 MBytes 126 Mbits/sec
[ 8] 54.00-55.00 sec 53.6 MBytes 450 Mbits/sec
[ 10] 54.00-55.00 sec 14.8 MBytes 124 Mbits/sec
[ 12] 54.00-55.00 sec 26.7 MBytes 224 Mbits/sec
[SUM] 54.00-55.00 sec 110 MBytes 923 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 55.00-56.00 sec 27.2 MBytes 228 Mbits/sec
[ 8] 55.00-56.00 sec 31.5 MBytes 264 Mbits/sec
[ 10] 55.00-56.00 sec 27.3 MBytes 229 Mbits/sec
[ 12] 55.00-56.00 sec 25.9 MBytes 218 Mbits/sec
[SUM] 55.00-56.00 sec 112 MBytes 938 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 56.00-57.00 sec 26.2 MBytes 220 Mbits/sec
[ 8] 56.00-57.00 sec 29.9 MBytes 251 Mbits/sec
[ 10] 56.00-57.00 sec 26.1 MBytes 219 Mbits/sec
[ 12] 56.00-57.00 sec 30.0 MBytes 252 Mbits/sec
[SUM] 56.00-57.00 sec 112 MBytes 941 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 57.00-58.00 sec 19.6 MBytes 164 Mbits/sec
[ 8] 57.00-58.00 sec 21.5 MBytes 180 Mbits/sec
[ 10] 57.00-58.00 sec 19.6 MBytes 165 Mbits/sec
[ 12] 57.00-58.00 sec 21.4 MBytes 179 Mbits/sec
[SUM] 57.00-58.00 sec 82.1 MBytes 688 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 58.00-59.00 sec 31.7 MBytes 266 Mbits/sec
[ 8] 58.00-59.00 sec 24.3 MBytes 204 Mbits/sec
[ 10] 58.00-59.00 sec 24.9 MBytes 209 Mbits/sec
[ 12] 58.00-59.00 sec 26.3 MBytes 221 Mbits/sec
[SUM] 58.00-59.00 sec 107 MBytes 899 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 59.00-60.00 sec 27.1 MBytes 228 Mbits/sec
[ 8] 59.00-60.00 sec 29.1 MBytes 244 Mbits/sec
[ 10] 59.00-60.00 sec 26.8 MBytes 225 Mbits/sec
[ 12] 59.00-60.00 sec 29.1 MBytes 244 Mbits/sec
[SUM] 59.00-60.00 sec 112 MBytes 941 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 60.00-60.02 sec 512 KBytes 195 Mbits/sec
[ 8] 60.00-60.02 sec 768 KBytes 292 Mbits/sec
[ 10] 60.00-60.02 sec 518 KBytes 197 Mbits/sec
[ 12] 60.00-60.02 sec 688 KBytes 261 Mbits/sec
[SUM] 60.00-60.02 sec 2.43 MBytes 945 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-60.02 sec 1.45 GBytes 208 Mbits/sec receiver
[ 8] 0.00-60.02 sec 1.63 GBytes 233 Mbits/sec receiver
[ 10] 0.00-60.02 sec 1.45 GBytes 207 Mbits/sec receiver
[ 12] 0.00-60.02 sec 1.69 GBytes 242 Mbits/sec receiver
[SUM] 0.00-60.02 sec 6.22 GBytes 890 Mbits/sec receiver
If this is accurate and my traffic is in fact going through both my WAN (USB3 UE300 eth1) and LAN (onboad nic eth0) ports of my Rpi4 then everything looks pretty good.. 
note: My regular LAN ip range is back to 192.168.76.x (which it was before I ran the other tests)
1 Like
Same issue here, my cpu goes suddenly to load 1.0 after few hours of normal usage (load).
I've got rev 1.1 & rtl8153, with kernel 5.4.95
edit: could anyone test with 5.4.96 ?
Hi there,
Some breaking news (the first one is good! The second one is bringing a lot of information).
-
The rev 1.4 board (with 8GB) is able to perform like the 1.1 board (with 2GB) if you put the following line into config.txt:
total_mem=2048
which is hopefully clearly enough for OpenWrt. -
Putting the kernel 5.4.95 from r15696 (dtb, overlays, kernel8.img from boot part, and modules folder 5.4.95 into /lib/modules/ from rootfs) into the old OpenWrt r11631, does not reduces the performance. Old OpenWrt r11631 is performing as good with old and new kernels: 1000 Mbps WAN to LAN or LAN to WAN with ax88179 without regression!
(This time I used my old OpenWrt installation (r11631) for testing my kernels between 4.19 and 5.4, instead of using Raspbian)
Everything is now becoming clear. Here is a summary of the information spread in the thread:
-
On the old OpenWrt version with 4.19.86 kernel, I had no ksoftirqd/0 high usage (both on integrated NIC and USB NIC)
-
On the new OpenWrt version with 5.4 kernel, I had high ksoftirqd/0 usage (both on integrated NIC and USB NIC), bringing big performance regression. I supposed it has something in common.
-
When I saw that modern Raspbian using 5.4 kernel had the same high usage on integrated NIC (while I hadn't this high usage on my old OpenWrt 4.19), I supposed the kernel update was guilty.
-
When I tested raspbian with 4.19 kernel the high usage on integrated NIC was still here so I didn't understand, and supposed OpenWrt had a patch or a different kernel.
-
irqbalance and software offloading are giving a little bit of help but not enough to fully compensate the performance loss
-
New kernels (like 5.10) is performing better on integrated NIC. But it does nothing more than before on USB NIC!
-
rtl8153 driver is eating a little bit less ressources. On new 1.4 boards with 8GB of RAM at least, it performs better than ax88179 but still is impacted by the performance regression
-
however, with 2GB RAM, things are performing better (and the max bandwidth regression is not visible anymore for rtl8153).
Finally, no matter which kernel you are using, all of them are able to have both low of high ksoftirqd/0 usage depending on the OS settings. I still haven't found which OS settings (/etc/config/files show not differences)
OpenWrt upgrade is finally causing the trouble, not the kernel!
Just as a reminder, I'm using the following configuration:
- eth0 is integrated NIC, used for the LAN (with a DHCP server enabled on it), connected to the house using dumb switches
- eth1 is USB NIC, used for the WAN (as DHCP and DHCPv6 client), directly connected to the fiber "ONT" adapter (GPON to GbE)
- no more specific configuration
@Cylac with 939 Mbps displayed into iperf you have no performance issue anymore 
I didn't follow everything but I hope what you did with the help @dlakelan made your RPi4 working full Gigabit
@fosix I don't know if it's related, but as I'm going to soon use OpenWrt again (probably with my rtl8153 adapter) I'll keep you informed if it goes from "working fine" to "performance drops" after few hours of usage. As the CPU usage is quite high recently, even if able to route 1000 Mbps in both way (LAN to WAN or WAN to LAN), be sure your CPU isn't reaching thermal throttling temperature after few hours! it could be a reason why performance drops. Mine has a little fan on it so unless the fan stops, it won't reach thermal throttling
Conclusion:
There is just 1 question left, and then, everything will be answered about this issue. What is the difference between r11631 and r15696 which is increasing the ksoftirqd/0 usage?
How about 4096? I've been using the 4G devices so far.
1 Like
I just tested, with 4096 it's running a little bit slower (in both way, iperf3 is showing ~830 Mbps instead of 933 Mbps with 2048).
(I don't know what is causing this performance drop when rising memory amount).
(By the way It would be interesting to lower at 1024 to see if the performance is increasing again 
)
I placed the rootfs files of OpenWrt r11631 (old) and r15696 (new) into those links so that anybody can try to find what is (or are?) the difference(s) making the old one performing better than the new one (when using the same kernel and same boards)
https://pix-server-sorel.pixconfig.fr/Manual/openwrt-debug/rootfs-old.tar.gz
https://pix-server-sorel.pixconfig.fr/Manual/openwrt-debug/rootfs-new.tar.gz
The boot part is not included as I tried using exactly the same files (and it shows no differences).
And that's all folks!
I successfully found what changed:
A lot of others things changed too, but the thing that disappeared (smp tune, "smp" stands for Symmetric multiprocessing) is now "SMP packet steering" and is a new functionality (disabled by default) that is available on the same tab that the one that is used for changing LAN IPv6 prefix
(believing my test, a reboot is needed after activation, to take the activation into account)
Also, when using boards with 4GB or (even worst) 8GB, the performance decreases compared to 2GB one. So, also adding total_mem=2048 into config.txt file, will make current snapshot to perform as good as the old one, even with ax88179, even with new boards, for 1000 Mbps WAN to LAN (or 1000 Mbps LAN to WAN) with no CPU bottleneck
Thanks to everyone who participated into this thread 
(I'll probably use the rtl8153 anyway now that I have one, just to test some full duplex routing test, which means running 2000 Mbps on each NIC. The ax88179 is able to handle full duplex, when connected to a big computer, but on the RPi4 it's struggling a little bit - rtl8153 will probably struggle too, but maybe a little bit less).
(New 5.10 kernel is using less CPU ressources for integrated NIC compared to 5.4 so for extreme performance tuning, new kernel, software offload, and irqbalance added to total_mem=2048 and packet steering may do a great job).
5 Likes
I'm playing at full duplex through the Pi4 with OpenWrt SNAPSHOT r15696-17fa01bb79 with kernel "5.4.95"
~ 940 Mbps both ways (upload + download at the same time).
In order to avoid flooding the Internet I placed my actual LAN on the WAN side (eth1, rtl8153) and a computer on the Pi4 LAN side (creating a LAN behind another LAN). Traffic between the 2 LAN's is running trough the RPi4 as it would for Internet traffic.
I'll keep you informed if it's slowing down during the following hours. But for now it's performing really great 
Let's leave the speed out of the equation for the moment.
The bigger issue is that I'm getting 1.0 load on the cpu without the router doing anything.
For now the only remedy is a reboot, then it's working as usual and after a few hours this 1.0 load appears out of nowhere and stays there untill I reboot the device.
when you say you've got 1.0 load, can you show the output of top -d 1 during that period?
Sure, waiting it to happen
how about show output "before" so we have a comparison
Another interesting observation:
When doing a speedtest on fast.com I have ksoftirqd/0 and sirq at around 5-7% on download and
18-25 % on upload
after
CPU load counter stays at 1.00 untill the rest (5min, 15 min) catch up.
Performance drops after few hours (400-500mbps), after restart everything's back to normal (800-900mbps)
your CPU is still 99% idle, this does not indicate excessive cpu usage.
1 Like