interesting. I'm guessing this needs backporting.
hmm unfortunate. can't get the driver to find the NAND.
I played around with the driver last year and was able to detect the NAND of the LGS352C. I could read the data but for some reason the filesystem could not be read (maybe wrong ECC algorithm). Here my dts.
// SPDX-License-Identifier: (GPL-2.0-or-later or MIT)
/dts-v1/;
#include "rtl931x.dtsi"
#include <dt-bindings/input/input.h>
#include <dt-bindings/gpio/gpio.h>
/ {
compatible = "linksys,lgs352c", "realtek,rtl931x-soc";
model = "Linksys LGS352C";
memory@0 {
device_type = "memory";
/* highmem not yet available, start with only 384MB of 512MB */
reg = <0x0 0x18000000>;
};
};
&spi0 {
status = "okay";
flash@0 {
compatible = "jedec,spi-nor";
reg = <0>;
spi-max-frequency = <10000000>;
partitions {
compatible = "fixed-partitions";
#address-cells = <1>;
#size-cells = <1>;
partition@e0000 {
label = "u-boot-env";
reg = <0x000e0000 0x10000>;
};
partition@f0000 {
label = "u-boot-env2";
reg = <0x000f0000 0x10000>;
};
};
};
};
&snand {
status = "okay";
flash@0 {
compatible = "spi-nand";
reg = <0>;
partitions {
compatible = "fixed-partitions";
#address-cells = <1>;
#size-cells = <1>;
partition@0 {
label = "ubifs";
reg = <0x00000000 0x4000000>;
};
partition@4000000 {
label = "firmware";
compatible = "openwrt,uimage";
reg = <0x04000000 0x1e00000>;
};
partition@5e00000 {
label = "runtime2";
reg = <0x05e00000 0x1e00000>;
read-only;
};
partition@7c00000 {
label = "unknown";
reg = <0x07c00000 0x400000>;
read-only;
};
};
};
};
And here my config diff.
diff --git a/target/linux/realtek/rtl931x/config-6.6 b/target/linux/realtek/rtl931x/config-6.6
index 736f472029..ccfc326928 100644
--- a/target/linux/realtek/rtl931x/config-6.6
+++ b/target/linux/realtek/rtl931x/config-6.6
@@ -112,7 +112,6 @@ CONFIG_MDIO_DEVICE=y
CONFIG_MDIO_DEVRES=y
CONFIG_MDIO_I2C=y
CONFIG_MDIO_SMBUS=y
-CONFIG_MEMFD_CREATE=y
CONFIG_MFD_SYSCON=y
CONFIG_MIGRATION=y
CONFIG_MIPS=y
@@ -143,12 +142,18 @@ CONFIG_MTD_CFI_ADV_OPTIONS=y
CONFIG_MTD_CFI_GEOMETRY=y
CONFIG_MTD_CMDLINE_PARTS=y
CONFIG_MTD_JEDECPROBE=y
+CONFIG_MTD_SPI_NAND=y
CONFIG_MTD_SPI_NOR=y
CONFIG_MTD_SPLIT_BRNIMAGE_FW=y
CONFIG_MTD_SPLIT_EVA_FW=y
CONFIG_MTD_SPLIT_FIRMWARE=y
CONFIG_MTD_SPLIT_TPLINK_FW=y
CONFIG_MTD_SPLIT_UIMAGE_FW=y
+CONFIG_MTD_UBI=y
+CONFIG_MTD_UBI_BEB_LIMIT=20
+CONFIG_MTD_UBI_BLOCK=y
+CONFIG_MTD_UBI_NVMEM=y
+CONFIG_MTD_UBI_WL_THRESHOLD=4096
CONFIG_NEED_DMA_MAP_STATE=y
CONFIG_NET_DEVLINK=y
CONFIG_NET_DSA=y
@@ -185,7 +190,6 @@ CONFIG_QUEUED_SPINLOCKS=y
CONFIG_RATIONAL=y
# CONFIG_REALTEK_OTTO_TIMER is not set
CONFIG_REALTEK_OTTO_WDT=y
-# CONFIG_REALTEK_PHY is not set
CONFIG_REALTEK_SOC_PHY=y
CONFIG_REGMAP=y
CONFIG_REGMAP_I2C=y
@@ -209,6 +213,7 @@ CONFIG_SOCK_RX_QUEUE_MAPPING=y
CONFIG_SPI=y
CONFIG_SPI_MASTER=y
CONFIG_SPI_MEM=y
+CONFIG_SPI_REALTEK_SNAND=y
CONFIG_SRCU=y
CONFIG_SWPHY=y
CONFIG_SYNC_R4K=y
@@ -233,6 +238,7 @@ CONFIG_TIMER_OF=y
CONFIG_TIMER_PROBE=y
CONFIG_TREE_RCU=y
CONFIG_TREE_SRCU=y
+CONFIG_UBIFS_FS=y
CONFIG_USE_GENERIC_EARLY_PRINTK_8250=y
CONFIG_USE_OF=y
CONFIG_WATCHDOG_CORE=y
And the main dts diff.
diff --git a/target/linux/realtek/dts/rtl931x.dtsi b/target/linux/realtek/dts/rtl931x.dtsi
index 61599e89b5..526d7e66f8 100644
--- a/target/linux/realtek/dts/rtl931x.dtsi
+++ b/target/linux/realtek/dts/rtl931x.dtsi
@@ -86,7 +86,7 @@
compatible = "simple-bus";
#address-cells = <1>;
#size-cells = <1>;
- ranges = <0x0 0x18000000 0x10000>;
+ ranges = <0x0 0x18000000 0x20000>;
spi0: spi@1200 {
status = "okay";
@@ -97,6 +97,17 @@
#address-cells = <1>;
#size-cells = <0>;
};
+
+ snand: spi@1a400 {
+ compatible = "realtek,rtl9301-snand";
+ reg = <0x1a400 0x44>;
+ interrupt-parent = <&gic>;
+ interrupts = <GIC_SHARED 37 IRQ_TYPE_LEVEL_HIGH>;
+ clocks = <&lx_clk>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+ status = "okay";
+ };
watchdog0: watchdog@3260 {
compatible = "realtek,rtl9310-wdt";
1 Like
This is wrong. I looked further into the details, and my problem is specific to the SFP+ I am using, and similar SFP+. 1gig copper will work fine, and maybe also some of the other 10gig copper SFPs
Gory details:
My 10GBase-T SFP+ matches this quirk entry in the sfp driver:
SFP_QUIRK_F("FS", "SFP-10G-T", sfp_fixup_fs_10gt)
This fixes the eeprom data, letting the driver know that this is in fact a copper SFP and not the multimode fibre SFP it claims to be. Reading the eeprom will still show the original bogus data, but with that quirk the driver knows better:
boot@OpenWrt:~# ethtool -m lan8|head
Identifier : 0x03 (SFP)
Extended identifier : 0x04 (GBIC/SFP defined by 2-wire interface ID)
Connector : 0x07 (LC)
Transceiver codes : 0x10 0x00 0x00 0x00 0x40 0x00 0x0c 0x00 0x00
Transceiver type : 10G Ethernet: 10G Base-SR
Transceiver type : FC: short distance (S)
Transceiver type : FC: Multimode, 62.5um (M6)
Transceiver type : FC: Multimode, 50um (M5)
Encoding : 0x06 (64B/66B)
BR, Nominal : 10300MBd
So far we're good. The sfp driver knows that there is a phy there and will try to create an mdio bus for that over i2c. But the phy in this SFP isn't available on the well-known i2c address we're used to. Maybe an attempt to hide it from hosts trying to detect such fake eeprom data? Lucky for us, the phy is exposed over a debug interface accessible via the standard "eeprom" sensor address (0x51, registers 0x80 to 0x85). This protocol is implemented in the mdio-i2c bus driver, which is used by the sfp driver to access phys over i2c:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/net/mdio/mdio-i2c.c#n109
And that's where we hit the wall: We're not using mdio-i2c on rtl93xx. The SoC has a multiplexing i2c engine, which s handled by the i2c-rtl9300 driver. The problem is that this isn't implementing i2c - it's SMBus only. So we have an OpenWrt specific mdio-smbus driver to make this work with sfps. Which it does just fine for any phy talking clause 22 on the standard 0x56 i2c address. Contrary to mdio-i2c, the mdio-smbus driver does not implement clause 45 ("10gig") access. Might work anyway, using clause 22? I don't have any sfp I can test that on.
The big problem is the missing RollBall protocol implementation for SMBus.
This is only a problem because we've told the sfp driver that it should expect to talk to a phy. A quickfix is to simply ignore the phy if the protocol is RollBall and the mdio driver doesn't implement it. This is enough to make my SFP+ "work" like it did before the quirk, and like it does in switches which knows nothing about 10gig copper. Such a fix might even be upstreamable, assuming we can get them to accept that the sfp driver should be able to talk to out-of-tree mdio bus drivers.
But a nicer fix would be to implement this for our SMBus driver too. So I've been struggling with that. Without success so far, I'm afraid. Most likely due to me knowing exactly nothing about i2c, SMBus, mdio and all that.... But I will not let that stop me 
Not going to complain if someone who knows what they're doing just solves the problem, though ....
And the sensors are there? What does ethtool show? You need ethtool-full for the decoded view. The hwmon device create by the sfp driver will poll the same registers shown by ethtool. Mine looks like this (just after setting the remote end og the link up - which is why the temperature isn't higher yet. The "Receiver signal average optical power" etc depend on the link quality, even if this is copper):
root@OpenWrt:~# ethtool -m lan8
Identifier : 0x03 (SFP)
Extended identifier : 0x04 (GBIC/SFP defined by 2-wire interface ID)
Connector : 0x07 (LC)
Transceiver codes : 0x10 0x00 0x00 0x00 0x40 0x00 0x0c 0x00 0x00
Transceiver type : 10G Ethernet: 10G Base-SR
Transceiver type : FC: short distance (S)
Transceiver type : FC: Multimode, 62.5um (M6)
Transceiver type : FC: Multimode, 50um (M5)
Encoding : 0x06 (64B/66B)
BR, Nominal : 10300MBd
Rate identifier : 0x00 (unspecified)
Length (SMF,km) : 0km
Length (SMF) : 0m
Length (50um) : 30m
Length (62.5um) : 10m
Length (Copper) : 0m
Length (OM3) : 0m
Laser wavelength : 850nm
Vendor name : FS
Vendor OUI : 00:00:00
Vendor PN : SFP-10G-T
Vendor rev :
Option values : 0x00 0x1a
Option : RX_LOS implemented
Option : TX_FAULT implemented
Option : TX_DISABLE implemented
BR margin, max : 10%
BR margin, min : 88%
Vendor SN : F2220644072
Date code : 220824
Optical diagnostics support : Yes
Laser bias current : 6.000 mA
Laser output power : 0.5000 mW / -3.01 dBm
Receiver signal average optical power : 0.4000 mW / -3.98 dBm
Module temperature : 50.41 degrees C / 122.75 degrees F
Module voltage : 3.2811 V
Alarm/warning flags implemented : Yes
Laser bias current high alarm : Off
Laser bias current low alarm : Off
Laser bias current high warning : Off
Laser bias current low warning : Off
Laser output power high alarm : Off
Laser output power low alarm : Off
Laser output power high warning : Off
Laser output power low warning : Off
Module temperature high alarm : Off
Module temperature low alarm : Off
Module temperature high warning : Off
Module temperature low warning : Off
Module voltage high alarm : Off
Module voltage low alarm : Off
Module voltage high warning : Off
Module voltage low warning : Off
Laser rx power high alarm : Off
Laser rx power low alarm : Off
Laser rx power high warning : Off
Laser rx power low warning : Off
Laser bias current high alarm threshold : 15.000 mA
Laser bias current low alarm threshold : 1.000 mA
Laser bias current high warning threshold : 13.000 mA
Laser bias current low warning threshold : 2.000 mA
Laser output power high alarm threshold : 1.9952 mW / 3.00 dBm
Laser output power low alarm threshold : 0.1584 mW / -8.00 dBm
Laser output power high warning threshold : 1.5848 mW / 2.00 dBm
Laser output power low warning threshold : 0.1778 mW / -7.50 dBm
Module temperature high alarm threshold : 80.00 degrees C / 176.00 degrees F
Module temperature low alarm threshold : -10.00 degrees C / 14.00 degrees F
Module temperature high warning threshold : 75.00 degrees C / 167.00 degrees F
Module temperature low warning threshold : -5.00 degrees C / 23.00 degrees F
Module voltage high alarm threshold : 3.6000 V
Module voltage low alarm threshold : 3.0000 V
Module voltage high warning threshold : 3.5000 V
Module voltage low warning threshold : 3.1000 V
Laser rx power high alarm threshold : 1.1220 mW / 0.50 dBm
Laser rx power low alarm threshold : 0.0199 mW / -17.01 dBm
Laser rx power high warning threshold : 1.0000 mW / 0.00 dBm
Laser rx power low warning threshold : 0.0223 mW / -16.52 dBm
Some progress. The phy driver registration doesn't work yet, but this is starting to look somewhat sane:
root@OpenWrt:~# mdio smbus:sfp-p8 mmd 17:1
CTRL1(0x00): 0x2040
flags: -reset -low-power -remote-loopback -local-loopback
speed: 10g
STAT1(0x01): 0x0006
capabilities: -pias -peas +low-power
flags: -fault +link
DEVID(0x02/0x03): 0x31c31c12
SPEED(0x04): 0x6031
capabilities: -400g +5g +2.5g -200g -25g -10g-xr -100g -40g -10g/1g -10 +100
+1000 -10-ts -2-tl +10g
DEVS(0x06/0x05): 0xe000009a
devices: +vendor2 +vendor1 +c22-ext -power-unit -ofdm -pma4 -pma3 -pma2 -pma1
+aneg -tc -dte-xs +phy-xs +pcs -wis +pma/pmd -c22
CTRL2(0x07): 0x0009
flags: -pias -peas
type: 10g-t
STAT2(0x08): 0xb301
capabilities: +tx-fault +rx-fault +ext-register +tx-disable +local-loopback
-10g-sr -10g-lr -10g-er -10g-lx4 -10g-sw -10g-lw -10g-ew
flags: +present -tx-fault -rx-fault
EXTABLE(0x0B): 0x40fc
capabilities: -10g-cx4 -10g-lrm +10g-t +10g-kx4 +10g-kr +1000-t +1000-kx
+100-tx -10-t -p2mp -40g/100g -1000/100-t1 -25g -200g/400g
+2.5g/5g -1000-h
PKGID(0x0E/0x0F): 0x31c31c12
The reported id is an AQR113C, which is reasonable. Now on to figure out why I can't register that phy driver/device
1 Like
Looks like it's not supported on this adapter, which is annoying as most 10gbase-T SFP+ modules make no mention of DOM support, I see one with the Broadcom BCM84891 chipset that has support but costs as much as the switch...
root@XikeStor:~# ethtool -m lan8
Identifier : 0x03 (SFP)
Extended identifier : 0x04 (GBIC/SFP defined by 2-wire interface ID)
Connector : 0x21 (Copper pigtail)
Transceiver codes : 0x00 0x00 0x00 0x00 0x00 0x04 0x00 0x00 0x00
Transceiver type : Passive Cable
Encoding : 0x00 (unspecified)
BR, Nominal : 10300MBd
Rate identifier : 0x00 (unspecified)
Length (SMF,km) : 0km
Length (SMF) : 0m
Length (50um) : 0m
Length (62.5um) : 0m
Length (Copper) : 1m
Length (OM3) : 0m
Passive Cu cmplnce. : 0x00 (unspecified) [SFF-8472 rev10.4 only]
Vendor name : QSFPTEK
Vendor OUI : 00:40:20
Vendor PN : QT-SFP-10G-T
Vendor rev : 03
Option values : 0x00 0x00
BR margin, max : 0%
BR margin, min : 0%
Vendor SN : QT6241210634
Date code : 241211
We need some sort of decoder ring to decipher the various combinations of Broadcom BCM84891, Marvell 88X3310, AQR113C and Realtek RTL8261N chipsets and their support for multi-rate, DOM and their relative heat outputs.
1 Like
Yay! Good news:
[ 94.886292] rtl83xx-switch switch@1b000000 lan8: PHY [smbus:sfp-p8:11] driver [Aquantia AQR113C] (irq=POLL)
Only problem is that I have no idea how to properly fix the last problem: phy registration failed with -EBUSY due to a phy already attached to the port. What I did so far was simply renaming the bogus phy-handle properties on the sfp ports to "pseudo-phy-handle" and change rtl83xx_mdio_probe() to use that property instead:
phy_node = of_parse_phandle(dn, "pseudo-phy-handle", 0);
This allows all the mumbo-jumbo mac/serdes/pcs/phy confusion in this driver to continue as before without confusing dsa/sfp/phylink layer. I tried simply removing the handles first, but that causes the driver to skip lots of the mac/serdes setup so it doesn't work.
I think it's time to challenge this confusion in the drivers. I understand that some of the mac/pcs/serdes configuration stuff is hidden behind something that looks like a mdio bus with phys. But that's no reason to bring that device tree or drivers.
Sort of hoping that the upstream effort will solve this problem....
In any case, with this last hack and my RollBall over SMBus changes, I now have working phy management for the FS 10GBase-T SFP+
root@OpenWrt:~# ethtool lan8
Settings for lan8:
Supported ports: [ ]
Supported link modes: 100baseT/Half 100baseT/Full
1000baseT/Full
10000baseT/Full
10000baseKR/Full
2500baseT/Full
5000baseT/Full
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 100baseT/Half 100baseT/Full
1000baseT/Full
10000baseT/Full
10000baseKR/Full
2500baseT/Full
5000baseT/Full
Advertised pause frame use: Symmetric Receive-only
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Link partner advertised link modes: 100baseT/Full
1000baseT/Full
10000baseT/Full
Link partner advertised pause frame use: Symmetric Receive-only
Link partner advertised auto-negotiation: Yes
Link partner advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Full
Auto-negotiation: on
Port: Twisted Pair
PHYAD: 11
Transceiver: external
MDI-X: Unknown
Supports Wake-on: d
Wake-on: d
Link detected: yes
And note that we now see the correct phy address instead of the virtual realtek address
6 Likes
Posted what I have here:
Noticed one strange oddity which I believe is unrelated to my fixes: Reading the "eeprom" with normal decoding works fine, but dumping it in hex format using ethtool -m lan8 hex on fails with -EIO. A bit unexpected. But the latter command will read a larger range of registers. I guess some of the RollBall registers is causing the problem.
ethtool -m lan8 hex on length 385 works, and so does ethtool -m lan8 hex on offset 478.
The only particular difference I see is the interrupt-parent and interrupts in my node:
snand: spi@1a400 {
compatible = "realtek,rtl9301-snand";
reg = <0x1a400 0x44>;
interrupt-parent = <&intc>;
interrupts = <19>;
clocks = <&lx_clk>;
#address-cells = <1>;
#size-cells = <0>;
status = "disabled";
This one claims to support DDM: https://www.aliexpress.com/item/1005002636459209.html?spm=a2g0o.order_list.order_list_main.17.651818021Kp82S I ordered one and will report back when I get it.
Nice! I am very interested in how that SFP+ works both with the original mdio-smbus code and my modified version. I tried to implement clause 45 support based on the mdio-i2c implementation, but I have no SFP I can test it on. So it probably doesn't work....
But I also reimplemented the clause 22 support as a simplified version of clause 45, similar to how mdio-i2c does it. And that I can and will test as soon as I find my 1000Base-T SFP.
The reason I did this was because I started doubting whether the original clause 22 code worked at all. It would only read and write a single byte, which became obvious in mdio bus scans I posted earlier. Lots of 00ff 00ff phy ids found where we expected ffff ffff. I guess that code hasn't been tested much since the only supported device so far has been the Zyxel XGS1250-12. No one in their right mind would use the single SFP+ slot for a copper SFP when there are 3 10gig copper ports next to it. The Xikestor sks8300-8x is a different game, making mdio-smbus important.
Very happy to test as soon as switch and SFP arrive (also the two gigabit ones I ordered while I was at it and the DAC I already have - that one does not seem to have much in the line of DDM if the BPI R4 ethtool output is to be trusted).
Reading DDM and other data should work in any case. The problem I believe I found is related to MDIO access to phys inside SFPs, which is irrelevant for DAC and fibre. MDIO read/write and SFP read/write are independent implementations, despite being run over the same SMBus or i2c bus.
The OpenWrt patch adding SMBus support to the sfp driver is mostly OK. The only minor issue is that it addsthe bus uncondtionally, while the original i2c code only adds it for SFPs actually expected to have a phy. This is easily fixed by simply rearranging the probing code:
BTW with these DAC cables around my desk I tried using an SFP port on the HP 1920-8G-PoE (JG921A) with a SFP+ DAC over onto an Intel E810 NIC and got this error:
[ 386.933785] sfp sfp-0: module QSFPTEK QT-SFP+-CA-P2M rev 03 sn QTCU240223049 dc 240223
[ 386.965470] rtl83xx-switch switch@1b000000: Unsupported interface: 28 for port 24
[ 386.990464] rtl83xx-switch switch@1b000000: Unsupported interface: 28 for port 24
[ 387.015417] rtl83xx-switch switch@1b000000 lan9: validation with support 00,00000000,00000000,00000000 failed: -EINVAL
you need a 10gig port for DAC
1 Like
Of course, the SFP socket always stays at the native rate and the module needs to perform the rate conversion!
My suspicion is it works the other way round - you can plug SFP modules into SFP+ sockets. Source: I have definitely used 1g FS fiber SFP in the SFP+ socket of a Mikrotik RB5009 for a brief period before my 10g fiber went live.
https://community.ui.com/questions/SFP-module-in-SFP-slot/7b188e48-9bb4-4df3-a852-d6fbb8463225 seems to agree that the opposite generally does not work.
Interesting. I did initially assume that the 25/10 port would step down to 1. Maybe if I had a SFP DAC instead of SFP+ or SFP28 that would be successful?
Probably but cannot say for sure, never used a device with SFP only. For gigabit, copper works well enough in my view. Power requirements are also benign.