Wow. That's a big step forward. I was looking for a way to send SFP events into the PHY driver since more than a year. The first time this came up was when I added the initial support for the RTL8214FC, but I see now that under 5.4, there was no support in the kernel for that, it was added later and the only example seems to be the marvel10g driver. Then it came up again to do different calibrations of the internal SerDes/PHY of the RTL93xx in combination with the insertion of different DAC cables and recently again to have SFP support for the RTL8214QF. So this is a real breakthrough for considerably improved SFP support on several RTL devices.
To proceed: why don't you add the
int (*module_insert)(void *priv, const struct sfp_eeprom_id *id);
void (*module_remove)(void *priv);
members to the sfp_upstream_ops structure and have them call rtl8380_rtl8214fc_media_set(). Then you can provide a PR for your device so people can test with other devices which have an RTL8214FC?
In the meantime I will play with what you learned on the RTL8214QF and the RTL93xx PHYs.
I can confirm this is in the latest SDK we have. They do it also for the RTL839x, not just on the RTL93xx. It seems to be necessary in combination with certain PHYs. Interesting that it was working with the RTL8214FC before. The comment in the code for the RTL839x code says it is needed because of latency, but on the RTL93xx consensus seemed to be it was due to latching. But now with what we see on the 8214FC I tend again to attribute this to latency, because it seems to work in certain conditions. Also the 8214FC is 1/2 of an RTL8218B (the last digit states the number of ports provided, FC seems to be support for cable and fibre. 1/2 because the 8218b has 2 QSGMII links, the 8214s have only one. This makes the 8214QF a quad port fibre only PHY, as stated on the product page, and the RTL8214C a quad port cable only chip. The PHY registers seem to be 100% identical.), so it is really unclear why this is never necessary on the 8218B if it is latching, which is a PHY property. Stupid me to not check the latest SDK for the RTL83xx when I realized it was necessary for the RTL8226 on the RTL9300, I am pretty sure this was not in the earlier SDK copies we have, which did not have the RTL93xx support.
On this device, it is necessary for all PHYs (internal RTL8218B, RTL8218D, RTL8214FC) to read the link state correctly. It doesn't seem to make any difference in actual functionality, though.
This works.
However, there is an issue with rtl8380_rtl8214fc_media_set itself. Media switching has always been a bit buggy here when networking was not initialized by the bootloader: After switching to fiber for the first time, it is necessary to do a ifconfig down/up cycle until the port actually works (I can't tell how copper ports would behave, as this device doesn't have combo ports). The first SFP port doesn't have this issue. The problem is that the phy_package_write_paged is used for powering ports off/on, which always changes the first port. Changing that to phy_write_paged fixes this issue.
I also implemented a change to make power configuration independent from media switching. This is necessary to avoid powering up a port while its interface is down. I kept rtl8380_rtl8214fc_on_off as it is, as it uses different code (phy_modify for fiber), and I don't know why this is the case. As this device rtl8380_phy_reset is called instead, I cannot test if a change there would break anything.
Aside from that, I saw in the kernel log that the PHYs seem to be initialized twice. Is this expected behaviour?
Kernel log
[ 2.247381] Probing RTL838X eth device pdev: 82086a00, dev: 82086a10
[ 2.318991] Found SoC ID: 8382: RTL8382, family 8380
[ 2.335667] rtl8380_init_mac
[ 2.345370] Using MAC 0000d894033cc5ac
[ 2.460660] Realtek RTL8218B (internal) mdio-bus:08: Detected internal RTL8218B
[ 2.485046] Firmware loaded. Size 1184, magic: 83808380
[ 5.102427] Realtek RTL8214FC mdio-bus:18: Detected external RTL8214FC
[ 5.124195] Firmware loaded. Size 1676, magic: 83808380
[ 6.435370] i2c /dev entries driver
[ 6.450486] NET: Registered protocol family 10
[ 6.488510] Segment Routing with IPv6
[ 6.501024] NET: Registered protocol family 17
[ 6.516832] 8021q: 802.1Q VLAN Support v1.8
[ 6.532371] i2c-gpio i2c-gpio-0: Slow GPIO pins might wreak havoc into I2C/SMBus bus timing
[ 6.560683] i2c-gpio i2c-gpio-0: using lines 464 (SDA) and 465 (SCL)
[ 6.582537] i2c-gpio i2c-gpio-1: Slow GPIO pins might wreak havoc into I2C/SMBus bus timing
[ 6.610876] i2c-gpio i2c-gpio-1: using lines 474 (SDA) and 475 (SCL)
[ 6.632863] i2c-gpio i2c-gpio-2: Slow GPIO pins might wreak havoc into I2C/SMBus bus timing
[ 6.661160] i2c-gpio i2c-gpio-2: using lines 478 (SDA) and 479 (SCL)
[ 6.683150] i2c-gpio i2c-gpio-3: Slow GPIO pins might wreak havoc into I2C/SMBus bus timing
[ 6.711453] i2c-gpio i2c-gpio-3: using lines 482 (SDA) and 483 (SCL)
[ 6.734364] sfp sfp-0: Host maximum power 1.0W
[ 6.749281] sfp sfp-0: No tx_disable pin: SFP modules will always be emitting.
[ 6.775025] sfp sfp-1: Host maximum power 1.0W
[ 6.789934] sfp sfp-1: No tx_disable pin: SFP modules will always be emitting.
[ 6.815688] sfp sfp-2: Host maximum power 1.0W
[ 6.830588] sfp sfp-2: No tx_disable pin: SFP modules will always be emitting.
[ 6.856401] sfp sfp-3: Host maximum power 1.0W
[ 6.871328] sfp sfp-3: No tx_disable pin: SFP modules will always be emitting.
[ 6.951523] Realtek RTL8218B (internal) rtl838x slave mii-0:08: Detected internal RTL8218B
[ 6.979055] Firmware loaded. Size 1184, magic: 83808380
[ 7.362079] sfp sfp-3: module WTD RTXM191-400 rev 3.0 sn EC122500180652 dc 120622
[ 7.393655] rtl8380_rtl8214fc_media_set: port 27, set_fibre: 1
[ 7.453556] rtl8380_rtl8214fc_power_set: Powering off COPPER (port 27)
[ 7.495520] rtl8380_rtl8214fc_power_set: Powering on FIBRE (port 27)
[ 9.585902] Realtek RTL8214FC rtl838x slave mii-0:18: Detected external RTL8214FC
[ 9.610855] Firmware loaded. Size 1676, magic: 83808380
Also, there is still the issue of duplicated hwmons. Every time an SFP module is powered up (after plugging it in or using ifconfig up), an additional one appears. This doesn't happen if the SFP is not referenced in the PHY node in the device tree.
Since the details about realtek PoE support went to a different sister topic, just wanted to inform that since last weekend the ZyXEL GS1900-24HPv1 is having some early PoE support on ports 1-24!
The link state is something the SoC reads out of the PHY, for this, the SoC actively polls the PHY via MDIO, reads out the PHY registers and triggers the interrupt when a link change is detected. The link change bit of the port is set correctly for the ISR. But the link state bit seems to be not always correctly set, at least it needs to be read 2x in some conditions.
The wrong link state read from the SoC seems to be corrected by phylink in some cases by reading out the PHY link state directly. It also reads out the PHY link state by polling itself every couple of seconds, so it can compensate even for no poling happening at all. The way to stop polling by phylink would be to have the PHY issue an IRQ, but the RTL SoCs do not have sufficiently many GPIOs that would allow to have a HW interrupt when the PHY issues it.
The behaviour of MAC_LINK_STS does either not seem to be considered a bug by Realtek or otherwise too difficult or not worth it to fix. The PHY polling is a central feature of the SoCs as it also is used to control the LEDs. Without polling the port leds do not work automatically. On the RTL93xx the SoC can even poll c45 PHYs and has per-mdio bus registers to configure which PHY registers are being polled and how to interpret them.
I will test this on the D-Link DGS1210-16 with its combo ports.
I have written support for configuration of the LEDs on the RTL838x. It works similarly to the ones found in the RTL839x and newer SoCs. It takes information about the PHYs and combines this with configuration values for the setup of the serial control. This can be used when u-boot does not properly initialize the LEDs. The control is then via the SoC, but it could also be the basis for a separate LED manual controller, which could be limited to merely poke values into the software LED on/off registers, and which would not need to e.g. have port/PHY information.
This is the code, the commit contains instructions of use:
Thanks @anon13997276, but nothing works. I tried all modes. Those 4 combo ports only works when u-boot initialize them (i.e. loading initramfs using them).
I tried multiple configurations, using either INTERNAL_PHY or EXTERNAL_PHY, SWITCH_PORT and SWITCH_SFP_PORT.
I also have two conflicting information about those ports. My device is a rtl8393 and u-boot report those ports as RTL8214FC. However, the driver detects them as 2 "External RTL8393 SERDES" and 2 "RTL8218B (external)". I don't know who is telling me the truth. If I use SWITCH_PORT, It can detect link presence for all but lan49 port, although it still does not work.
It still does not work as expected. It is unstable, normally only working after I boot with those combo ports connected.
I was also hoping that the gpio pins were the same but in my tests, the ping related to the sfp module (mod-def0-gpio?) is a little bit different. It is one pin lower for lan50 for example. Is there a place or method to get those pin settings?
Edgecore ECS-2100 series seems to be a similar device, possibly supportable. Does anyone know how to stop u-boot? I tried many combinations and nothing worked. They also refused to publish the GPL package.
Getting the GPIO pins for the SFP ports correct is necessary for it to work properly.
There are different ways to identify the GPIOs. The easiest is if the OEM firmware offers the show tech-support command. Just issue that and it will show amongst a lot of other info the GPIO configuration. If this does not work the next thing I do is to use a special SFP module which has wires soldered to the connector and exiting a the rear end of the module so that I can trace e.g. the mod-def0-gpio to a pin typically on the RTL8231. There is a datasheet for that chip available and it will directly give you all the GPIO numbers. The least efficient is to dump gpios from within u-boot (rtk pinGet and the like) or from within linux without or with the module inserted/signal present. This gives only the input pins such as LOS and MOD-DEF0 .
I did a mistake: the pins match exactly the ones used by dgs-1210-16 (based only on mod-def0-gpio). Anyway, it does not work. As it is a different SoC, it might require to port those patches to RTL839x.
They might work nicely with dgs-1210-28. I might test them next week.
I'll submit dgs-1210-52 as living without 4 ports in 52 might be an acceptable price to pay to have OpenWrt. https://github.com/openwrt/openwrt/pull/10227
I added a DTS macro to easily flip the ports on/off for further tests.
I am not sure you mentioned it before, but if, then I did not get it. The PHY-code for the 8214FC is heavily dependent on the SoC. It starts with the fact that the PHY id of the external RTL8218B is identical to the RTL8214FC, so in order to identify it, one needs to check to which ports it is connected: only the high ports (20-27) on the RTL838x are linked to a SerDes capable of QSGMII and 1000BX, and it even depends on whether it is an RTL8382 or RTL8380. On the RTL839x these are other ports >=48. But it does not end there. There is a configuration patch of the PHY depending on its version and what the SoC is, and currently it is only configured on the RTL838x. The RTL PHYs are heavily linked to the RTL SoCs, the RTL8221B found on the RTL93xx devices even has a different PHY id than the allegedly same chip which can be found on 2.5 GBit adapters sold individually and supported by the latest linux kernels.
I just extended my branch with full support for HPE 1920-8G.
This time, the SFP ports are working right away. The only issue with them is that there is no TX disable support. The bootloader has code specifically for this device, which toggles GPIOs 27/28 on the RTL8231, but these aren't actually connected anywhere.
There was another issue though, the built-in watchdog fails to reboot this device and it hangs. As the bootloader uses the external watchdog for rebooting, I added that to the device tree as GPIO watchdog.
However, this resulted in the watchdog triggering during PHY initialization. The problem is that the PHY patching takes longer than the watchdog timeout, and as the kernel is built without preemption, the kernel fails to schedule the watchdog ping in time. After adding a call to cond_sched to rtl838x_smi_wait_op this issue is gone. I'm not sure if this is the ideal solution here?
With that change, both 1920-8G and 1920-16G should be practically fully working, so I'm thinking about submitting the changes soon. The only remaining thing that doesn't work fully is the Power LED. But I don't think there is an easy fix for that, as it is controlled via LED_SW_CTRL (at least the bootloader sets it to be permanently on).
It should very likely be safe, there is a global SMI lock held while you reschedule and polling by the SoC is disabled for the entire PHY package. So neither CPU nor SoC should be doing anything dangerous with the PHY of SMI registers during the switched context. But have you tried to simply enable preemtive scheduling? The intention was always to allow preemption, the merely voluntary yielding of the context was an oversight.