ER-X-SFP - SFP (eth5) port has link-state (LED = lit), but swconfig says link: port:5 link:down

Thirsty · October 20, 2019, 5:12pm

Hi,

What kind of SFP plug are you using fiber or RJ45? Brand and part-number?
Did this one also work with ubiquity software?
Is "eth5" led on/blink on start-up/power-up of the device?

Note: swconfig always port 5 is up.
Because we force the port up.

I updated my branch. I added package phytool.
You have to enable it by selecting the package in menuconfig. Network -> phytool

With this tool you can read and write to the phy registers.
Our sfp module and port 5 of the switch are connected to at8033 phy which has address 7.

Read phy register phytool print eth0/7

ieee-phy: id:0x004dd074

   ieee-phy: reg:BMCR(0x00) val:0x1000
      flags:          -reset -loopback +aneg-enable -power-down -isolate -aneg-restart -collision-test
      speed:          10-half

   ieee-phy: reg:BMSR(0x01) val:0x616d
      capabilities:   -100-b4 +100-f +100-h -10-f -10-h -100-t2-f -100-t2-h
      flags:          +ext-status +aneg-complete -remote-fault +aneg-capable +link -jabber +ext-register

Forge 1000mbit/full-duplex: phytool write eth0/7/0 0x0140

Read phy register again phytool print eth0/7

ieee-phy: id:0x004dd074

   ieee-phy: reg:BMCR(0x00) val:0x0140
      flags:          -reset -loopback -aneg-enable -power-down -isolate -aneg-restart -collision-test
      speed:          1000-full

   ieee-phy: reg:BMSR(0x01) val:0x614d
      capabilities:   -100-b4 +100-f +100-h -10-f -10-h -100-t2-f -100-t2-h
      flags:          +ext-status -aneg-complete -remote-fault +aneg-capable +link -jabber +ext-register

I hope that works.

ptpt52 · October 21, 2019, 5:16am

hi, my device is ER-X-SFP and my sfp module works at first with factory firmware

but after installing openwrt with your patches, eth5 down.

ok I will try the phytool instruction later.

Thirsty · October 21, 2019, 11:12am

@hvandrie Do you also have time to test this?

ptpt52 · October 22, 2019, 3:23pm

sfp not work
and here is the info with phytool
also do the i2cdump

root@X-WRT:~# phytool print eth0/7
ieee-phy: id:0x004dd074

   ieee-phy: reg:BMCR(0x00) val:0x1140
      flags:          -reset -loopback +aneg-enable -power-down -isolate -aneg-restart -collision-test
      speed:          1000-full

   ieee-phy: reg:BMSR(0x01) val:0x6149
      capabilities:   -100-b4 +100-f +100-h -10-f -10-h -100-t2-f -100-t2-h
      flags:          +ext-status -aneg-complete -remote-fault +aneg-capable -link -jabber +ext-register
root@X-WRT:~# 
root@X-WRT:~# phytool write eth0/7/0 0x0140
root@X-WRT:~# 
root@X-WRT:~# phytool print eth0/7
ieee-phy: id:0x004dd074

   ieee-phy: reg:BMCR(0x00) val:0x0140
      flags:          -reset -loopback -aneg-enable -power-down -isolate -aneg-restart -collision-test
      speed:          1000-full

   ieee-phy: reg:BMSR(0x01) val:0x614d
      capabilities:   -100-b4 +100-f +100-h -10-f -10-h -100-t2-f -100-t2-h
      flags:          +ext-status -aneg-complete -remote-fault +aneg-capable +link -jabber +ext-register
root@X-WRT:~# i2cdump -y 0 0x50
No size specified (using byte-data access)
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: 03 04 00 00 00 00 08 00 00 00 00 01 0c 00 00 00    ??....?....??...
10: 00 00 64 00 46 49 4e 49 53 41 52 20 43 4f 52 50    ..d.FINISAR CORP
20: 2e 20 20 20 00 00 90 65 48 38 35 32 31 2d 33 2d    .   ..?eH8521-3-
30: 48 33 43 20 20 20 20 20 34 20 20 20 00 00 00 fa    H3C     4   ...?
40: 00 10 00 00 50 39 38 31 32 35 45 20 20 20 20 20    .?..P98125E     
50: 20 20 20 20 30 36 30 32 32 35 20 20 00 00 00 3d        060225  ...=
60: 48 75 61 77 65 69 2d 33 43 6f 6d 33 34 31 30 41    Huawei-3Com3410A
70: 30 30 30 20 20 00 00 00 00 00 00 00 00 00 00 bb    000  ..........?
80: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
90: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
c0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................

hvandrie · October 22, 2019, 4:58pm

I will share my info asap.
Using Ubiquiti SFP module. My setup was/is 2 ER-X-SFP's connected only with fiber. One ER-X-SFP running Ubiquity firmware and the other your OpenWRT build.

Thirsty · October 22, 2019, 5:22pm

Hi,

I think PHYLIB is the problem. PHYLIB knows that at8033 exits but doesn't have a driver.
You can enable the driver but that is not going to work because at8033 current driver doesn't understand the SFP cage.
But what PHYLIB does, it puts the PHY is auto negotiation mode.
But that doesn't work well with your SGMII sfp module.

What you can try is same phytool write command.
This disables the auto negotiation on the at8033 PHY.
Then take the SFP plug out and re-insert the SFP module.
So it is in a know state.
This should work.

Other option also put the PHY inside the SFP in forced mode.

i2cset -y 0 0x56 0x00 0x0140 w

ptpt52 · October 23, 2019, 5:09am

I run the phytool command

phytool write eth0/7/0 0x0140

and then take the SFP plug out and re-insert the SFP module.
and it works.

We need more improvement.

Thirsty · October 23, 2019, 7:10pm

Hi,

I redo some detection code and removed the phy-node in dts.
Now PHYLIB doesn't touch the phy anymore.

Can you please try this branch?

ptpt52 · October 25, 2019, 2:31am

thanks, I will try it latter.

ptpt52 · October 25, 2019, 3:31pm

I try the new branch and it work. but...
This time I don't need the phytool command.
And to get it work I have to plug out and re-insert the SFP module.

Thirsty · October 25, 2019, 4:19pm

So it doesn't work directly? You first have to replug the sfp module?
What happens after a power cycle?

ptpt52 · October 25, 2019, 9:23pm

everty time the device reboot, I need to do a replug-the-sfp-module action to get it work. (only) after that, unplugging the cable and plugging in again is also working.

mathiashro · December 27, 2019, 7:45pm

I've tested the current solution on two ER-X-SFP with each equipped UF-SM-1G-S transceiver. Everything worked flawlessly. Also after reboot, unplug/replug, remove/reinsert, network restart etc.

We created a backport for the current openwrt 19.07 branch (rc2 state / git), for your reference:
https://dev.opennet-initiative.de/browser/on_firmware/patches/edgerouterx-sfp.patch

As know the status of the link and SFP is not known (always up). Any plans how to proceed and bringing it upstream to openwrt (trunk or even 19.07)? Or any other tasks to be done first / cleanup?

If you need further tests or support I'm (we are) happy to follow up here or in person at 36c3.

gabri94 · February 28, 2020, 12:05pm

Hi, i have an ER-X-SFP running openwrt 19.07.01 and I'd like to use it with my Technicolor ONU.
It works out of the box using edgeOS, so it should be easy to let it work.
I followed a bit the thread, but I haven't understood if there's a way to use it and if it has been merged upstream

Edit: I took the patch published by @mathiashr, adapted it to run on openwrt 19.07 RC2 and compiled it. Unfortunately when i plug my sfp module the link state remains down.
I also tryied to force it up with the phytool command provided above, but nothing changes.

mathiashro · March 2, 2020, 7:35pm

Hi @gabri94, can you explain what kind of SFP exactly the "Technicolor ONU" is? Do you have other SFPs available to test? I've used Cisco and UBNT which worked well.

Also please kindly understand that we still maintain the above patch against the OpenWrt 19.07 branch (currently at .02). We run compile tests, last functional test was at RC2 as stated above. Would be great if you can evaluate further.

gabri94 · March 2, 2020, 9:46pm

Hi, I am trying to apply the patch to the commit with hash "c56ed72d2bc6dbee3ff82b4bd42e1768f1a2c737"
which should corresponds to the head of openwrt-19.07.

The error i get is:

edgerouterx-sfp.patch:93: trailing whitespace.
	
edgerouterx-sfp.patch:306: trailing whitespace.
		;;
error: corrupted patch at line 307

I circumvented it by removing that block and doing the changes by hand.

Regarding to the SFP, it is a Telecom Italia ONT for GPON FTTH access. Unfortunately I haven't got others SFP to try, but I tested this SFP with the latest version of EdgeOS (v1) and it works out of the box.

mathiashro · March 4, 2020, 6:20pm

Good Evening,

updated our patch to remove the trailing whitespaces (was already included in the upstream patchset). The currupted error is not visible here - patch and unpatch was successfull on our codebase.

Can you retest the patch and/or investigate further about the reason why you can't apply the patch?

Have a nice day!

Patch patches/edgerouterx-sfp.patch
patching file openwrt/target/linux/ramips/files-4.14/drivers/net/ethernet/mediatek/gsw_mt7620.h
patching file openwrt/target/linux/ramips/files-4.14/drivers/net/ethernet/mediatek/gsw_mt7621.c
patching file openwrt/target/linux/ramips/dts/mt7621.dtsi
patching file openwrt/target/linux/ramips/base-files/etc/board.d/02_network
Hunk #1 succeeded at 249 (offset -1 lines).
Hunk #2 succeeded at 259 (offset -1 lines).
patching file openwrt/target/linux/ramips/dts/UBNT-ERX-SFP.dts
patching file openwrt/package/network/utils/phytool/Makefile

gabri94 · March 4, 2020, 8:03pm

I'm not getting the trailing whitespaces messages anymore (though i think they were warning)
but I'm stil getting the last one:

error: corrupted patch at line 307

At this point i don't know could be.
I am applying the patch using the command:

git apply edgerouterx-sfp.patch

on the commit hash:

c56ed72d2bc6dbee3ff82b4bd42e1768f1a2c737

Can you confirm this is consistent with your patch?

gabri94 · March 4, 2020, 8:15pm

I have fixed the patch. You can find my version here https://pastebin.com/wKNfMqsG

This is applicable on the openwrt feed using that command.
I had to remove "openwrt/" from all the path, it looks like you were applying the patch from outside the repo.

Now, I'd like to help you troubleshoot why my SFP is not working with this patch. Is there some log or some command which would help you?

mathiashro · March 12, 2020, 1:03pm

Ok, understood. We are using Quilt (https://wiki.debian.org/UsingQuilt) in our automated build environment, therefor the path is different than using "git apply".

Tools used here and also referred in forum threads:

Link status: phytool print eth0/7 ("flags +aneg-complete +link" = link is active)
Transceiver modell und serial number: i2cdump -y 0 0x50 (via installed package "i2c-tools")

Hope this helps to get one step forwards. Also you should consider checking kernel.org for driver support of your transceiver.