100Gbps Link Running 12 Mbps

customdev · November 27, 2025, 1:39am

I have two OpenWRT flashed Mellanox SN2410 switches. The capability of the metal is incredible but the delivery is underwhelming. Link is supposedly 100Gbps but can only get 12 Mbps.

My Setup:

SN2410 ←–>SN2410

We're using two identical Mellanox compatible QSFP28 100GBASE-SR4 transcievers connected with a single MTO OM4 fiber cable 3 feet long. Each transceiver is plugged into port 8.

All ports are except eth1 are bridged to br-lan on both switches. One is 192.168.1.1 and the other is at 192.168.1.2.

iperf, iperf3, and every other method I've used suggests a max of 12 Mbps.

We are on mlxsw_spectrum-13.2010.1006.mfa for both switches per wiki instructions. OWRT is the latest stable 24.10.4.

No major errors in dmesg or the kernel log.

What am I missing here to get full wire speed here?

**As a side note I can use 1Gbps, 10Gbps, or 25Gbps SFP/SFP+ trancievers in any of the SFP+ ports and come up with the same result. 12Mbps. SImilar setup with a 3ft section of OM3 LC/LC patch.

***Will give outputs of commands on demand.

greem · November 27, 2025, 2:12am

This may be an extremely dumb question… are you sure these figures aren't 12MBps, therefore a factor of 8 above what you say and thence pretty close to 100Mbps?

The capitalisation is stupendously important.

customdev · November 27, 2025, 2:22am

We’re talking 100 Gigabit Ethernet connection here. 100GBASE. As in 5 year old datacenter spine switch level stuff. As in I recycled a couple x86/64 based Mellanox switches. As in 12.5 GBps.

We’re not talking 100 Megabit Ethernet connection. 100BASE. As in 30 year old legacy switch level stuff that was obsolete by 2000. That would be 12.5 MBps and not what I’m talking about.

psherman · November 27, 2025, 4:19am

Don’t dismiss @greem ‘s comments. It’s not about the generation of network gear, but rather a troubleshooting step.

~100Mbps would be slow and outdated, yes, but it would be very logical if you have any faulty cables/terminations/ports or if a driver was not working properly and failing back to Fast Ethernet.
-12Mbps would not have that logical consequence, but as pointed out, 12MBps is about 100Mbps which is why the question was raised.

customdev · November 27, 2025, 5:01pm

Guess the proof is in the pudding.

root@OpenWrt:\~# iperf3 -c 192.168.1.2 -P 20
Connecting to host 192.168.1.2, port 5201
\[  5\] local 192.168.1.1 port 59990 connected to 192.168.1.2 port 5201
\[  7\] local 192.168.1.1 port 60000 connected to 192.168.1.2 port 5201
\[  9\] local 192.168.1.1 port 60006 connected to 192.168.1.2 port 5201
\[ 11\] local 192.168.1.1 port 60022 connected to 192.168.1.2 port 5201
\[ 13\] local 192.168.1.1 port 60028 connected to 192.168.1.2 port 5201
\[ 15\] local 192.168.1.1 port 60036 connected to 192.168.1.2 port 5201
\[ 17\] local 192.168.1.1 port 60052 connected to 192.168.1.2 port 5201
\[ 19\] local 192.168.1.1 port 60054 connected to 192.168.1.2 port 5201
\[ 21\] local 192.168.1.1 port 60056 connected to 192.168.1.2 port 5201
\[ 23\] local 192.168.1.1 port 60058 connected to 192.168.1.2 port 5201
\[ 25\] local 192.168.1.1 port 60070 connected to 192.168.1.2 port 5201
\[ 27\] local 192.168.1.1 port 60074 connected to 192.168.1.2 port 5201
\[ 29\] local 192.168.1.1 port 60082 connected to 192.168.1.2 port 5201
\[ 31\] local 192.168.1.1 port 60092 connected to 192.168.1.2 port 5201
\[ 33\] local 192.168.1.1 port 60104 connected to 192.168.1.2 port 5201
\[ 35\] local 192.168.1.1 port 60116 connected to 192.168.1.2 port 5201
\[ 37\] local 192.168.1.1 port 60120 connected to 192.168.1.2 port 5201
\[ 39\] local 192.168.1.1 port 60130 connected to 192.168.1.2 port 5201
\[ 41\] local 192.168.1.1 port 60146 connected to 192.168.1.2 port 5201
\[ 43\] local 192.168.1.1 port 60148 connected to 192.168.1.2 port 5201

\[ ID\] Interval           Transfer     Bitrate         Retr
\[  5\]   0.00-10.00  sec   896 KBytes   734 Kbits/sec  232             sender
\[  5\]   0.00-10.00  sec   640 KBytes   524 Kbits/sec                  receiver
\[  7\]   0.00-10.00  sec  1.88 MBytes  1.57 Mbits/sec  349             sender
\[  7\]   0.00-10.00  sec  1.25 MBytes  1.05 Mbits/sec                  receiver
\[  9\]   0.00-10.00  sec   640 KBytes   524 Kbits/sec  174             sender
\[  9\]   0.00-10.00  sec   384 KBytes   314 Kbits/sec                  receiver
\[ 11\]   0.00-10.00  sec   768 KBytes   629 Kbits/sec  193             sender
\[ 11\]   0.00-10.00  sec   512 KBytes   419 Kbits/sec                  receiver
\[ 13\]   0.00-10.00  sec  1.25 MBytes  1.05 Mbits/sec  234             sender
\[ 13\]   0.00-10.00  sec   896 KBytes   734 Kbits/sec                  receiver
\[ 15\]   0.00-10.00  sec  1.12 MBytes   944 Kbits/sec  270             sender
\[ 15\]   0.00-10.00  sec   896 KBytes   734 Kbits/sec                  receiver
\[ 17\]   0.00-10.00  sec   896 KBytes   734 Kbits/sec  164             sender
\[ 17\]   0.00-10.00  sec   640 KBytes   524 Kbits/sec                  receiver
\[ 19\]   0.00-10.00  sec  1.12 MBytes   944 Kbits/sec  283             sender
\[ 19\]   0.00-10.00  sec   896 KBytes   734 Kbits/sec                  receiver
\[ 21\]   0.00-10.00  sec  1.12 MBytes   944 Kbits/sec  230             sender
\[ 21\]   0.00-10.00  sec   768 KBytes   629 Kbits/sec                  receiver
\[ 23\]   0.00-10.00  sec   640 KBytes   524 Kbits/sec  165             sender
\[ 23\]   0.00-10.00  sec   384 KBytes   314 Kbits/sec                  receiver
\[ 25\]   0.00-10.00  sec   384 KBytes   315 Kbits/sec  158             sender
\[ 25\]   0.00-10.00  sec   384 KBytes   314 Kbits/sec                  receiver
\[ 27\]   0.00-10.00  sec   384 KBytes   315 Kbits/sec  136             sender
\[ 27\]   0.00-10.00  sec   256 KBytes   210 Kbits/sec                  receiver
\[ 29\]   0.00-10.00  sec   768 KBytes   629 Kbits/sec  195             sender
\[ 29\]   0.00-10.00  sec   512 KBytes   419 Kbits/sec                  receiver
\[ 31\]   0.00-10.00  sec   896 KBytes   734 Kbits/sec  231             sender
\[ 31\]   0.00-10.00  sec   768 KBytes   629 Kbits/sec                  receiver
\[ 33\]   0.00-10.00  sec   768 KBytes   629 Kbits/sec  205             sender
\[ 33\]   0.00-10.00  sec   640 KBytes   524 Kbits/sec                  receiver
\[ 35\]   0.00-10.00  sec   512 KBytes   419 Kbits/sec  152             sender
\[ 35\]   0.00-10.00  sec   384 KBytes   314 Kbits/sec                  receiver
\[ 37\]   0.00-10.00  sec  1.25 MBytes  1.05 Mbits/sec  207             sender
\[ 37\]   0.00-10.00  sec  1.12 MBytes   943 Kbits/sec                  receiver
\[ 39\]   0.00-10.00  sec   640 KBytes   524 Kbits/sec  178             sender
\[ 39\]   0.00-10.00  sec   512 KBytes   419 Kbits/sec                  receiver
\[ 41\]   0.00-10.00  sec   640 KBytes   524 Kbits/sec  193             sender
\[ 41\]   0.00-10.00  sec   512 KBytes   419 Kbits/sec                  receiver
\[ 43\]   0.00-10.00  sec  1.62 MBytes  1.36 Mbits/sec  278             sender
\[ 43\]   0.00-10.00  sec  1.38 MBytes  1.15 Mbits/sec                  receiver
\[SUM\]   0.00-10.00  sec  18.0 MBytes  15.1 Mbits/sec  4227             sender
\[SUM\]   0.00-10.00  sec  13.5 MBytes  11.3 Mbits/sec                  receiver

iperf Done.
root@OpenWrt:\~#

customdev · November 27, 2025, 5:23pm

Little more info:

root@OpenWrt:~# ethtool eth25
Settings for eth25:
Supported ports: [ FIBRE ]
Supported link modes: 1000baseKX/Full
10000baseKR/Full
40000baseCR4/Full
40000baseSR4/Full
40000baseLR4/Full
25000baseCR/Full
25000baseSR/Full
50000baseCR2/Full
100000baseSR4/Full
100000baseCR4/Full
100000baseLR4_ER4/Full
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 1000baseKX/Full
10000baseKR/Full
40000baseCR4/Full
40000baseSR4/Full
40000baseLR4/Full
25000baseCR/Full
25000baseSR/Full
50000baseCR2/Full
100000baseSR4/Full
100000baseCR4/Full
100000baseLR4_ER4/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: 100000Mb/s
Duplex: Full
Port: FIBRE
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
Link detected: yes

customdev · November 27, 2025, 5:59pm

Output for the firmware on both devices:

root@OpenWrt:~# mstfwmanager
Querying Mellanox devices firmware ...

Device #1:

Device Type: Spectrum
Part Number: 01AD888_01AD903_Ax
Description: SN2410 Fabric Switch w/Cumulus-OS
PSID: IBM0000000015
PCI Device Name: 0000:03:00.0
Base MAC: 506b4b997ec0
Versions: Current Available
FW 13.2010.1006 N/A

Status: No matching image found

hecatae · November 27, 2025, 6:30pm

Which Openwrt version?

Bingo · November 27, 2025, 8:27pm

Are you running the iperf on the switch ?

Normally one is adviced to run the iperf on xternal equipment, and let the switch/router "just do the switching/routing"

Your switch mgmt CPU, could easily be the limiting factor here.

100Gb forwarding would have to be done in some kind of ASIC.

customdev · November 27, 2025, 11:02pm

I’m using one SN2410 as the iperf3 server and the other SN2410 as the client.

I suppose it could be the case.

Tecnolobo · December 2, 2025, 4:31pm

Sorry, I don't have experience with these type of devices, but do you have check if the software and hardware offload are enabled into firewall? If need you can try run the test with firewall off to see if it are degrading the performance.

M10 · December 2, 2025, 11:39pm

This is powerfull switch plus weak cpu combo, whole switching mafic it’s done by switch and cpu only set’s it’s parameters for work. So rake a look to this switch configuration.