I've tried cleaning this up quite a bit. I moved the driver and phy code from the brainslayer branch linked above into openwrt (some changes were required, such as max ring count causing a malloc failure), and annotated the DTS. I probably need to carefully review the diff, but I am trying to bring up serdes "correctly" now. I've enabled GPIO using the legacy driver, rather than otto driver as above. Monitoring these 31 GPIOs shows no changes when inserting or removing SFPs.
I'm not sure if this matches our earlier discussion, but here is the annotated DTS: https://pastebin.com/h8GPN5gb. I think the SMI address values are incorrect, but trying various things there seems to have no effect. I also suspect the serdes values. I'll post board pictures when I get a chance. Notably, there is a small IC near each SFP cage, a larger IC I cannot read that is related by traces to the cages, and a PCA9555A that also appears to be related to the cages. The OEM firmware has 8 muxed i2c buses, the first of which appears identical to the i2c bus. I'll see what I can do with reading registers as well there.
[ 0.895566] libphy: Fixed MDIO Bus: probed
[ 0.902005] rtl83xx-switch switch@1b000000: Error registering switch: -517
[ 0.909889] Probing RTL838X eth device pdev: 8146c800, dev: 8146c810
[ 0.917033] Allocate 3315364 bytes for DMA
[ 0.935341] Found SoC ID: 9303: RTL9303, family 9300
[ 0.940884] Using MAC 000000e04c000000
[ 0.945074] rtl838x_mdio_init: looking at port 16
[ 0.950343] rtl838x_mdio_init: looking at port 17
[ 0.955566] rtl838x_mdio_init: looking at port 18
[ 0.960790] rtl838x_mdio_init: looking at port 19
[ 0.966011] rtl838x_mdio_init: looking at port 20
[ 0.971256] rtl838x_mdio_init: looking at port 21
[ 0.976484] rtl838x_mdio_init: looking at port 22
[ 0.981705] rtl838x_mdio_init: looking at port 23
[ 0.987589] c45_mask: 00080000
[ 0.991045] libphy: rtl930x-eth-mdio: probed
[ 1.005332] mdio_bus mdio-bus: MDIO device at address 16 is missing.
[ 1.022002] mdio_bus mdio-bus: MDIO device at address 17 is missing.
[ 1.038669] mdio_bus mdio-bus: MDIO device at address 18 is missing.
[ 1.055338] mdio_bus mdio-bus: MDIO device at address 19 is missing.
[ 1.072006] mdio_bus mdio-bus: MDIO device at address 20 is missing.
[ 1.088674] mdio_bus mdio-bus: MDIO device at address 21 is missing.
[ 1.105334] mdio_bus mdio-bus: MDIO device at address 22 is missing.
[ 1.122009] mdio_bus mdio-bus: MDIO device at address 23 is missing.
[ 1.129550] rtl93xx_set_features called
[ 1.135096] NET: Registered protocol family 10
[ 1.170593] Segment Routing with IPv6
[ 1.174777] NET: Registered protocol family 17
[ 1.180036] 8021q: 802.1Q VLAN Support v1.8
[ 1.198759] rtl930x_setup called
[ 1.202378] In rtl83xx_vlan_setup
[ 1.206054] In rtl930x_vlan_profile_setup
[ 1.210555] In rtl930x_vlan_profile_setup
[ 1.215011] UNKNOWN_MC_PMASK: 000000001fffffff
[ 1.219945] VLAN 0: L2 learn: 0; Unknown MC PMasks: L2 1fffffff, IPv4 1fffffff, IPv6: 1fffffff
[ 1.219952] Routing enabled: IPv4 UC y, IPv6 UC y, IPv4 MC y, IPv6 MC y
[ 1.237087] Bridge enabled: IPv4 MC n, IPv6 MC n,
[ 1.242501] VLAN profile 0: raw 00033000 00000000 1fffffff 1fffffff 1fffffff
[ 3.368446] rtl930x_pie_init
[ 3.373201] libphy: dsa slave smi: probed
[ 3.377767] rtl83xx-switch switch@1b000000: Unsupported interface: 25 for port 1
[ 3.386007] rtl83xx-switch switch@1b000000 lan1 (uninitialized): no phy at 1
[ 3.393895] rtl83xx-switch switch@1b000000 lan1 (uninitialized): failed to connect to PHY: -ENODEV
[ 3.403863] rtl83xx-switch switch@1b000000 lan1 (uninitialized): error -19 setting up PHY for tree 0, switch 0, port 1
[ 3.415894] rtl83xx-switch switch@1b000000: Unsupported interface: 25 for port 2
[ 3.424142] rtl83xx-switch switch@1b000000 lan2 (uninitialized): no phy at 2
[ 3.432017] rtl83xx-switch switch@1b000000 lan2 (uninitialized): failed to connect to PHY: -ENODEV
[ 3.441982] rtl83xx-switch switch@1b000000 lan2 (uninitialized): error -19 setting up PHY for tree 0, switch 0, port 2
[ 3.454007] rtl83xx-switch switch@1b000000: Unsupported interface: 25 for port 3
[ 3.462252] rtl83xx-switch switch@1b000000 lan3 (uninitialized): no phy at 3
[ 3.470127] rtl83xx-switch switch@1b000000 lan3 (uninitialized): failed to connect to PHY: -ENODEV
[ 3.480092] rtl83xx-switch switch@1b000000 lan3 (uninitialized): error -19 setting up PHY for tree 0, switch 0, port 3
[ 3.492116] rtl83xx-switch switch@1b000000: Unsupported interface: 25 for port 4
[ 3.500362] rtl83xx-switch switch@1b000000 lan4 (uninitialized): no phy at 4
[ 3.508200] rtl83xx-switch switch@1b000000 lan4 (uninitialized): failed to connect to PHY: -ENODEV
[ 3.518199] rtl83xx-switch switch@1b000000 lan4 (uninitialized): error -19 setting up PHY for tree 0, switch 0, port 4
[ 3.530227] rtl83xx-switch switch@1b000000: Unsupported interface: 25 for port 5
[ 3.538473] rtl83xx-switch switch@1b000000 lan5 (uninitialized): no phy at 5
[ 3.546309] rtl83xx-switch switch@1b000000 lan5 (uninitialized): failed to connect to PHY: -ENODEV
[ 3.556307] rtl83xx-switch switch@1b000000 lan5 (uninitialized): error -19 setting up PHY for tree 0, switch 0, port 5
[ 3.568317] rtl83xx-switch switch@1b000000: Unsupported interface: 25 for port 6
[ 3.576606] rtl83xx-switch switch@1b000000 lan6 (uninitialized): no phy at 6
[ 3.584447] rtl83xx-switch switch@1b000000 lan6 (uninitialized): failed to connect to PHY: -ENODEV
[ 3.594432] rtl83xx-switch switch@1b000000 lan6 (uninitialized): error -19 setting up PHY for tree 0, switch 0, port 6
[ 3.606439] rtl83xx-switch switch@1b000000: Unsupported interface: 25 for port 7
[ 3.614726] rtl83xx-switch switch@1b000000 lan7 (uninitialized): no phy at 7
[ 3.622565] rtl83xx-switch switch@1b000000 lan7 (uninitialized): failed to connect to PHY: -ENODEV
[ 3.632550] rtl83xx-switch switch@1b000000 lan7 (uninitialized): error -19 setting up PHY for tree 0, switch 0, port 7
[ 3.644560] rtl83xx-switch switch@1b000000: Unsupported interface: 25 for port 8
[ 3.652845] rtl83xx-switch switch@1b000000 lan8 (uninitialized): no phy at 8
[ 3.660685] rtl83xx-switch switch@1b000000 lan8 (uninitialized): failed to connect to PHY: -ENODEV
[ 3.670670] rtl83xx-switch switch@1b000000 lan8 (uninitialized): error -19 setting up PHY for tree 0, switch 0, port 8
[ 3.682731] rtl83xx-switch switch@1b000000: configuring for fixed/internal link mode
[ 3.691673] DSA: tree 0 setup
[ 3.695013] LINK state irq: 20
[ 3.698427] In rtl83xx_setup_qos
[ 3.702058] L3_IPUC_ROUTE_CTRL 00002000, IPMC_ROUTE 0000057e, IP6UC_ROUTE 00014580, IP6MC_ROUTE 00013b7e
[ 3.712638] L3_IPUC_ROUTE_CTRL 00002001, IPMC_ROUTE 00000501, IP6UC_ROUTE 00014581, IP6MC_ROUTE 00012881
[ 3.723171] L3_IP_ROUTE_CTRL 00000130
[ 3.727296] rtl930x_dbgfs_init called
[ 3.732588] rtl83xx-switch switch@1b000000: Link is Up - 1Gbps/Full - flow control off
EDIT: added 10GBASER to rt93xx_phy_validate, but it's not clear if that or KR or something else is the more appropriate configuration.