When I threw the problem at ChatGPT it suggested a new pse-core method to get a consolidated 'status', which could have a polling config associated with it (i.e. pse-core would automatically poll for the data if configured via dts)
It did seem like a relatively clean idea.
Since for a pse chip with like 8 or more ports, calling 8x admin_state + 8x status + 8x fault all individually would be inefficient both in driver, and 'network' traffic.
But it's an upstream thing... so would likely be a bit slow to progress. But we could probably carrying some submitted patches in OpenWrt whilst things proceed in Kernel upstream.
I think if we can get the stc8 MFD stuff in for the LEDs, then the basic HS104 pse stuff in, the control and some status will be there... so it'd 'mostly' work.
It would also open the door to more switches using the linux pse stuff, likely revealing more gaps in functionality required..
If you're referencing Linux Upstream stuff... that's a weeks/months long kind of process, very unlikely to be days.
I still haven't submitted anything HS104 upstream, since we're still on a much older kernel version, and things have changed quite a bit still between the 6.18 that we've got (your) backports for, and 7.0-rcN that the kernel tip is now up to.
Here's a link to the ChatGPT 'conversation' I had about the pse-pd polling
I think there is already some notification stuff that has recently gone into upstream pse-pd, so that likely only needs some tweaks, but the polling stuff I believe is still entirely missing. I'm sure the current system is based entirely on interrupts (which makes sense for some of the fancier chips already upstream, but for our incredibly cheap Hasivo chips.. no interrupts).
There's still a bit of stuff I've got to do upstream (which would be my first) for the SYSCON LEDs, and also for the PSE-Controller (so that we can have them as modules).
The SYSCON LED stuff I've already tested and got a patch ready for, so will try to submit that tonight.
The PSE-Controller bit (modularization) I'll try to get the patch for tonight, to submit tomorrow night (or maybe tonight if things work out faster).
Do you want to do the HS104 submission upstream? (it should just be the HS104, without LEDs or polling)
Once the HS104 upstream lands, then it should be easier to push for polling to be added.
The LEDS_SYSCON upstream submission, we'll have to see if there's any pushback from it.
I got a bit further with testing the PSE_CONTROLLER as a module, and that's a no go. Too many inter-dependencies. But I think with the stuff that robimarko sorted out we'll be able to proceed with this as-is.
Just wanted to say you guys are doing great work! Sorry I’ve been missing from the Realtek side of the house, I’ve been busy over on Airoha, Mediatek and Qualcomm. Looking forward to testing out your contributions on my switches.
I guess my next step is (that is what I meant with some days) to build a first prototype based on the stuff you have in your gpt conversation + the recent changes you submitted .... After we can get some feedback from other folks on what to submit to upstream or only to openwrt. The direction is clear as the general approach seems to be right
Do you have more information on the sequencing issue of the pse vs phy?
I suspect this might be more an issue around the phy / mdio aspect than anything else, since the current pse modules in upstream are all able to be built as modules, not strictly built-in, and they likely work with a range of other phy / mdio systems out there (there must be platforms that 'Kory Maincent' and 'Oleksij Rempel' use with these)..
It'll definitely be a bit of a blocker on getting it into OpenWrt like that. robimarko already NAK'd the same for the leds-syscon stuff.
With the hs104 not working as a package/module, did you see a lockup on boot like this:
[ 732.363831] probe of 1b000000.switchcore:mdio-controller returned 517 after 119490 usecs
[ 732.391225] probe of switch@1b000000 returned 517 after 288 usecs
[ 732.511069] probe of realtek-mdio:00 returned 0 after 9040 usecs
[ 732.531831] probe of 1b000000.switchcore:mdio-controller returned 517 after 120175 usecs
[ 732.559242] probe of switch@1b000000 returned 517 after 287 usecs
[ 732.679188] probe of realtek-mdio:00 returned 0 after 9287 usecs
[ 732.699989] probe of 1b000000.switchcore:mdio-controller returned 517 after 120319 usecs
[ 732.727351] probe of switch@1b000000 returned 517 after 286 usecs
[ 732.846483] probe of realtek-mdio:00 returned 0 after 8927 usecs
If so, that looks like a dependency issue around the PSE driver not being available when the switch is trying to bring in the PHYs on the MDIO.
I'm unsure if this is a DSA common issue, or particular to the Realtek DSA at this stage... I'm somewhat leaning more towards the first.
Although I would think with modular PHY drivers (like the RTL8261N) it would have been more of an issue if it was DSA common...
I think fixing this in MDIO / PHY probing is likely the right way... such that if the driver isn't yet available (because it hasn't been loaded) a callback is registered to complete the init, but the remainder can still continue.
Hmm.. ok, I think the issue is the Realtek DSA driver..
[ 732.699989] probe of 1b000000.switchcore:mdio-controller returned 517 after 120319 usecs
[ 732.727351] probe of switch@1b000000 returned 517 after 286 usecs
[ 732.846483] probe of realtek-mdio:00 returned 0 after 8927 usecs
This was caused by two combined factors:
Adding pses = <&pse0 0> to the PHY node
Having the Hasivo HS104 PSE driver as a module (through a package) instead of built-in.
Both Gemini and ChatGPT think that we should move away from the immediate attempts to register PHYs during the MDIO probe, and should instead defer them with use of devm_add_action() and phylink_of_phy_connect().
I understand those words from the AI overlords.. but they've bullshitted me numerous times before. And I'm not sure if that is indeed 'the way'.
I still haven't confirmed that my attempt to get the hs104 pse driver into the boot modules actually worked (I only tried it after the probe looping, and it still probe looped.. so I didn't get a shell that I could actually check where the hs104 module was referenced... it's on the agenda for today).
That returns with -EPROBE_DEFER with no pse driver loaded and the mdio driver tries to be initialized several times. That is totally normal as deferring means "retry after next driver load".
So do you have an issue with PSE loading as a module or not?
@bevanweiss I can confirm, my testing was only with "build-in" - by doing so I did not have any issues.
I will fix now the comments from the PR. Later I will take a look at the "as module" issue...
I got a working version (module based and not build-in).
Tbh, I am not sure if that is now required, because the way how the hs104 works is a little different to others and the source of the issue is not in the driver itself. Why should someone use the module not as build-in? I mean you don't change the hardware after setup. It will make this project bigger and bigger..
It took me some hours to figure that out, So the needed steps are:
Don't fail PHY registration on EPROBE_DEFER (fwnode_mdio.c) - register the PHY without PSE, set psec = NULL .. as you figured out
resolve PSE control on first ethtool access via pse_try_resolve() ... pse_Core.c
Prevent regulator cleanup from killing pse (pse_core.c): pse_pi_is_enabled() returns admin_state_enabled (0 at boot) instead of reading hardware, so regulator_init_complete thinks the ports are off and leaves them alone
Sync regulator refcount on consumer arrival (pse_core.c): call regulator_enable() when a PSE control is first acquired
Poll worker recovery (pse_core.c) ... if hardware gets reset externally after a consumer exists, detect the mismatch and re enable
And take it with a grain of salt... it is rather hacky and a lot of LLMs are involved.
So do you have an issue with PSE loading as a module or not?
If yes. What tells
cat /sys/kernel/debug/devices_deferred
Correct, works fine when builtin, but not as a module.
The boot in this scenario never seems to get to an interactive console. It gets stuck in that loop, so running any commands to explicitely identify the deferred device hasn't worked for me. But I think given it happens specifically around pse, it's almost certainly the pse module not getting loaded in time before the MDIO/switch requirement.
My understanding is that kernel driver loading goes:
builtin > bootable modules > (interactive console) + other modules
And things seem to be getting stuck in that builtin part. I'm unsure if the deferred probe can extend from builtin > bootable modules staging... I would have expected so, but perhaps I'm wrong (or there's another hidden dependency around the MDIO/switch interaction which prevents other things from proceeding to loading builtin modules).
I did check on the boot modules (by editing the dts to exclude the pses = <&pseX N> on each PHY).
ls /etc/modules-boot.d/*
/etc/modules-boot.d/02-crypto-hash
/etc/modules-boot.d/04-crypto-crc32c
/etc/modules-boot.d/1-pse-hasivo-hs104
/etc/modules-boot.d/30-gpio-button-hotplug
So:
Hasivo module is at least listed in the correct location for boot modules.
It's definitely about the PHY->PSE linkage (since the only change between booting, and probe looping is the pses in the devicetree)
Ahhh... the hs104 driver depends on the i2c-core... and that's a module which is NOT currently available at boot. So I think maybe it's about the i2c-core package not coming in at boot time...
because the way how the hs104 works is a little different to others
What do you mean about this part? Do you mean you think the driver is structurally different?
This PSE/PHY aspect should only be about the probe behaviours, and I think the hs104 driver is externally identical to the other existing upstream drivers in this fashion.
Why should someone use the module not as build-in?
Because the core kernel is built identically for ALL boards within a subtarget (including those without PoE). And remembering that most OpenWrt users (by volume) do not compile their own firmware. They just use the firmware selector, or download a published configuration from the OpenWrt build infrastructure. It's not reasonable to impose the rootfs size requirements of holding ALL pse drivers on ALL Realtek (and other target) boards just because our particular device happens to have this one specific pse chip on it.
So we really need to find a way to support this as a module.
Or OpenWrt would require an entire re-architecture of how the build system works... which I suspect is not viable.
root@OpenWrt:~# ls /etc/modules-boot.d/
02-crypto-hash 30-gpio-button-hotplug 52-pse-hasivo-hs104
04-crypto-crc32c 51-i2c-core
but still got the same issue with the MDIO/switch
[ 15.271451] Using MAC 00:28:24:12:92:97
[ 15.285847] probe of 1b000000.switchcore:ethernet returned 0 after 37527 usecs
[ 15.410833] probe of realtek-mdio:00 returned 0 after 8663 usecs
[ 15.431677] probe of 1b000000.switchcore:mdio-controller returned 517 after 121699 usecs
[ 15.459100] probe of switch@1b000000 returned 517 after 281 usecs
[ 15.580466] probe of realtek-mdio:00 returned 0 after 8691 usecs
[ 15.601288] probe of 1b000000.switchcore:mdio-controller returned 517 after 121781 usecs
[ 15.628704] probe of switch@1b000000 returned 517 after 286 usecs
[ 15.750132] probe of realtek-mdio:00 returned 0 after 8872 usecs
[ 15.770973] probe of 1b000000.switchcore:mdio-controller returned 517 after 121824 usecs
[ 15.798333] probe of switch@1b000000 returned 517 after 282 usecs
[ 15.919798] probe of realtek-mdio:00 returned 0 after 8674 usecs
It does appear to do init of the i2c-gpio, but I'm unsure if this includes enough to actually probe i2c devices (like the Hasivo hs104). There's no signs of hs104 in the dmesg up to the loop.
[ 15.033835] i2c-gpio i2c_scl23_sda22: using lines 534 (SDA) and 535 (SCL)
[ 15.056752] probe of i2c_scl23_sda22 returned 0 after 25132 usecs
When the pses binding isn't present, the hs104 init is occuring within boot modules.
[ 18.657166] Run /init as init process
[ 18.669370] with arguments:
[ 18.679216] /init
[ 18.686789] with environment:
[ 18.697217] HOME=/
[ 18.705064] TERM=linux
[ 19.438869] init: Console is alive
[ 19.451048] init: - watchdog -
[ 19.478317] kmodloader: loading kernel modules from /etc/modules-boot.d/*
[ 19.503103] calling init_module+0x0/0x1000 [crc32c_generic] @ 536
[ 19.523732] initcall init_module+0x0/0x1000 [crc32c_generic] returned 0 after 11 usecs
[ 19.551808] gpio_button_hotplug: loading out-of-tree module taints kernel.
[ 19.575251] calling init_module+0x0/0x1000 [gpio_button_hotplug] @ 536
[ 19.598422] probe of keys returned 0 after 1048 usecs
[ 19.615678] initcall init_module+0x0/0x1000 [gpio_button_hotplug] returned 0 after 18398 usecs
[ 19.646604] calling init_module+0x0/0x1000 [hasivo_hs104] @ 536
[ 19.668250] hs104: read reg 0x02: 0x1452, step: 10000, return_val: 52020000
[ 19.691421] hs104: voltage read: 52020000 uV
Unfortunately I don't have a different (non-Realtek) PoE device to test if the same behaviour occurs with pse devices as modules.
