Bcm63xx_enet internal phy + mdio-gpio switch. Non responsive LAN

I'm developing support for an old BCM6333 / 4-port switch via BCM5325E / 16MB RAM / 4MB SPI flash board. Just for fun and it's my first time modding the Linux kernel.

The switch's MDIO link is connected to the SOC via GPIO. I couldn't find any previous attempt to support such configuration on the bcm63xx. I seem to be successful in linking the mdio-gpio bus to the PHY in the bcm63xx_enet code and boot log doesn't indicate any obvious error. But I'm not able to ping and connect using LAN.

Boot log

CFE version 1.0.37-102.6 for BCM96338 (32bit,SP,BE)
Build Date: 2009�� 09�� 09�� ������ 16:35:36 CST (root@localhost.localdomain)
Copyright (C) 2000-2008 Broadcom Corporation.

Serial flash device: name EN25P32, id 0x1c15, size 4096KB
100 MB Full-Duplex (auto-neg)
CPU type 0x29010: 240MHz
Total memory: 16777216 bytes (16MB)
Boot Address 0xbfc00000


Board IP address                  : 192.168.1.1:ffffff00
Host IP address                   : 192.168.1.100
Gateway IP address                :
Run from flash/host (f/h)         : f
Default host run file name        : vmlinux
Default host flash file name      : bcm963xx_fs_kernel
Boot delay (0-9 seconds)          : 1
Board Id (0-11)                   : 96333AW2G
Number of MAC Addresses (1-32)    : 11
Base MAC Address                  : 00:1f:a4:cb:de:87
PSI Size (1-64) KBytes            : 64

send udp message finished.
Not response, auto boot ...
*** Press any key to stop auto run (1 seconds) ***
Auto run second count down: 0
Booting from only image (0xbfc10000) ...
Code Address: 0x80A00000, Entry Address: 0x80a00000
LZMA: Prossible old LZMA format, trying to decompress..
Decompression OK!
Entry at 0x80a00000
Closing network.
Closing DMA Channels.
Starting program at 0x80a00000
[    0.000000] Linux version 4.9.152 (lspiaw@lspiaw-Lenovo-V480) (gcc version 7.3.0 (OpenWrt GCC 7.3.0 r7676-cddd7b4c77) ) #0 Wed Jan 30 12:21:02 2019
[    0.000000] Detected Broadcom 0x6338 CPU revision a2
[    0.000000] CPU frequency is 240 MHz
[    0.000000] 16MB of RAM installed
[    0.000000] board_bcm963xx: Boot address 0xbfc00000
[    0.000000] board_bcm963xx: CFE version: unknown
[    0.000000] bootconsole [early0] enabled
[    0.000000] CPU0 revision is: 00029010 (Broadcom BCM6338)
[    0.000000] board: board name: 96333AW2G
[    0.000000] MIPS: machine is Broadcom BCM96333AW2G reference board
[    0.000000] Determined physical RAM map:
[    0.000000]  memory: 01000000 @ 00000000 (usable)
[    0.000000] Initrd not found or empty - disabling initrd
[    0.000000] Primary instruction cache 16kB, VIPT, 2-way, linesize 16 bytes.
[    0.000000] Primary data cache 8kB, 2-way, VIPT, no aliases, linesize 16 bytes
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x0000000000000000-0x0000000000ffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x0000000000ffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x0000000000ffffff]
[    0.000000] Built 1 zonelists in Zone order, mobility grouping off.  Total pages: 4064
[    0.000000] Kernel command line: rootfstype=squashfs,jffs2 noinitrd console=ttyS0,115200
[    0.000000] PID hash table entries: 64 (order: -4, 256 bytes)
[    0.000000] Dentry cache hash table entries: 2048 (order: 1, 8192 bytes)
[    0.000000] Inode-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.000000] Memory: 10908K/16384K available (3083K kernel code, 161K rwdata, 460K rodata, 1280K init, 201K bss, 5476K reserved, 0K cma-reserved)
[    0.000000] SLUB: HWalign=16, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] NR_IRQS:256
[    0.000000] clocksource: MIPS: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 15927170388 ns
[    0.000042] sched_clock: 32 bits at 120MHz, resolution 8ns, wraps every 17895697403ns
[    0.008359] Calibrating delay loop... 237.82 BogoMIPS (lpj=475648)
[    0.047410] pid_max: default: 32768 minimum: 301
[    0.052896] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.059950] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.090167] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.100501] futex hash table entries: 256 (order: -1, 3072 bytes)
[    0.107324] pinctrl core: initialized pinctrl subsystem
[    0.116328] NET: Registered protocol family 16
[    0.132721] Can't analyze schedule() prologue at 8030f57c
[    0.253117] clocksource: Switched to clocksource MIPS
[    0.267292] NET: Registered protocol family 2
[    0.276260] TCP established hash table entries: 1024 (order: 0, 4096 bytes)
[    0.283892] TCP bind hash table entries: 1024 (order: 0, 4096 bytes)
[    0.290797] TCP: Hash tables configured (established 1024 bind 1024)
[    0.298554] UDP hash table entries: 256 (order: 0, 4096 bytes)
[    0.304988] UDP-Lite hash table entries: 256 (order: 0, 4096 bytes)
[    0.312946] NET: Registered protocol family 1
[    0.356625] Crashlog allocated RAM at address 0xf00000
[    0.371927] workingset: timestamp_bits=30 max_order=12 bucket_order=0
[    0.415718] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    0.421999] jffs2: version 2.2 (NAND) (SUMMARY) (LZMA) (RTIME) (CMODE_PRIORITY) (c) 2001-2006 Red Hat, Inc.
[    0.519465] io scheduler noop registered
[    0.523755] io scheduler deadline registered (default)
[    0.538345] fffe0300.serial: ttyS0 at MMIO 0xfffe0300 (irq = 10, base_baud = 1562500) is a bcm63xx_uart
[    0.548409] console [ttyS0] enabled
[    0.548409] console [ttyS0] enabled
[    0.555776] bootconsole [early0] disabled
[    0.555776] bootconsole [early0] disabled
[    0.608812] m25p80 spi0.0: en25f32 (4096 Kbytes)
[    0.631843] bcm63xxpart: Partition 0 is CFE offset 0 and length 10000
[    0.638710] bcm63xxpart: Partition 1 is nvram offset 3f0000 and length 10000
[    0.646168] bcm63xxpart: Partition 2 is linux offset 10000 and length 3e0000
[    0.653666] 3 bcm63xxpart partitions found on MTD device spi0.0
[    0.659962] Creating 3 MTD partitions on "spi0.0":
[    0.665231] 0x000000000000-0x000000010000 : "CFE"
[    0.679316] 0x0000003f0000-0x000000400000 : "nvram"
[    0.696480] 0x000000010000-0x0000003f0000 : "linux"
[    0.720901] parser_imagetag: rootfs: CFE image tag found at 0x0 with version 6, board type 96333AW2G
[    0.730589] parser_imagetag: Partition 0 is kernel offset 100 and length 1339e8
[    0.738327] parser_imagetag: Partition 1 is rootfs offset 133ae8 and length 2ac518
[    0.746317] parser_imagetag: Spare partition is offset 250004 and length 18fffc
[    0.754080] 2 bcm963xx-imagetag partitions found on MTD device linux
[    0.760824] Creating 2 MTD partitions on "linux":
[    0.765941] 0x000000000100-0x000000133ae8 : "kernel"
[    0.780215] 0x000000133ae8-0x0000003e0000 : "rootfs"
[    0.797458] mtd: device 4 (rootfs) set to be root filesystem
[    0.806967] 1 squashfs-split partitions found on MTD device rootfs
[    0.813647] 0x000000250000-0x0000003e0000 : "rootfs_data"
[    0.828506] bcm63xx-spi fffe0c00.spi: at [mem 0xfffe0c00-0xfffe0c3f flags 0x200] (irq 9, FIFOs size 63)
[    0.843551] libphy: GPIO Bitbanged MDIO: probed
[    0.874219] libphy: Fixed MDIO Bus: probed
[    0.973570] b53_common: found switch: BCM5325, rev 0
[    0.979005] Broadcom B53 (1) gpio-0:00: attached PHY driver [Broadcom B53 (1)] (mii_bus:phy_addr=gpio-0:00, irq=-1)
[    0.991757] bcm63xx-wdt bcm63xx-wdt:  started, timer margin: 30 sec
[    1.003682] NET: Registered protocol family 17
[    1.008869] 8021q: 802.1Q VLAN Support v1.8
[    1.052090] random: fast init done
[    1.135135] VFS: Mounted root (squashfs filesystem) readonly on device 31:4.
[    1.174300] Freeing unused kernel memory: 1280K
[    1.179210] This architecture does not have kernel memory protection.
[   10.441536] init: Console is alive
[   10.446526] init: - watchdog -
[   12.331049] kmodloader: loading kernel modules from /etc/modules-boot.d/*
[   12.340742] kmodloader: done loading kernel modules from /etc/modules-boot.d/*
[   12.360417] init: - preinit -
[   12.788414] random: crng init done
Press the [f] key and hit [enter] to enter failsafe mode
Press the [1], [2], [3] or [4] key and hit [enter] to select the debug level
[   25.613306] eth0: link UP - 100/full - flow control off
[   28.650791] mount_root: jffs2 not ready yet, using temporary tmpfs overlay
[   28.737543] urandom-seed: Seed file not found (/etc/urandom.seed)
[   29.093659] procd: - early -
[   29.097470] procd: - watchdog -
[   29.709289] eth0: link DOWN
[   29.879001] procd: - watchdog -
[   29.884519] procd: - ubus -
[   30.582972] procd: - init -
Please press Enter to activate this console.
[   31.609592] kmodloader: loading kernel modules from /etc/modules.d/*
[   31.652872] nf_conntrack version 0.5.0 (1024 buckets, 4096 max)
[   31.866251] xt_time: kernel timezone is -0000
[   31.880935] ip_tables: (C) 2000-2006 Netfilter Core Team
[   31.931555] PPP generic driver version 2.4.2
[   31.948453] NET: Registered protocol family 24
[   31.971692] kmodloader: done loading kernel modules from /etc/modules.d/*
[   63.950306] jffs2_scan_eraseblock(): End of filesystem marker found at 0x0
[   63.981592] jffs2_build_filesystem(): unlocking the mtd device... [   63.988027] done.
[   63.990151] jffs2_build_filesystem(): erasing all blocks after the end marker... [   66.130898] br-lan: port 1(eth0) entered blocking state
[   66.136653] br-lan: port 1(eth0) entered disabled state
[   66.143754] device eth0 entered promiscuous mode
[   67.290106] eth0: link UP - 100/full - flow control off
[   67.312615] br-lan: port 1(eth0) entered blocking state
[   67.318338] br-lan: port 1(eth0) entered forwarding state
[  122.178090] done.
[  122.180196] jffs2: notice: (719) jffs2_build_xattr_subsystem: complete building xattr subsystem, 0 of xdatum (0 unchecked, 0 orphan) and 0 of xref (0 dead, 0 orphan) found.

ifconfig

br-lan    Link encap:Ethernet  HWaddr 00:1F:A4:CB:DE:87  
          inet addr:192.168.1.1  Bcast:192.168.1.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:342 (342.0 B)

eth0      Link encap:Ethernet  HWaddr 00:1F:A4:CB:DE:87  
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:342 (342.0 B)
          Interrupt:16 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:160 errors:0 dropped:0 overruns:0 frame:0
          TX packets:160 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1 
          RX bytes:10880 (10.6 KiB)  TX bytes:10880 (10.6 KiB)

swconfig

root@OpenWrt:/# swconfig list
Found: switch0 - eth0
root@OpenWrt:/# swconfig dev eth0 show
Global attributes:
        enable_vlan: 1
        ports: 0x002f
Port 0:
        pvid: 1
        link: port:0 link:down
Port 1:
        pvid: 1
        link: port:1 link:down
Port 2:
        pvid: 1
        link: port:2 link:up speed:100baseT full-duplex auto
Port 3:
        pvid: 1
        link: port:3 link:down
Port 4:
        pvid: 0
        link: port:4 link:down
Port 5:
        pvid: 1
        link: port:5 link:up speed:100baseT full-duplex 
VLAN 1:
        ports: 0 1 2 3 5 
root@OpenWrt:/# swconfig dev eth0 help
switch0: eth0(BCM5325), ports: 6 (cpu @ 5), vlans: 16
     --switch
        Attribute 1 (int): enable_vlan (Enable VLAN mode)
        Attribute 2 (string): ports (Available ports (as bitmask))
        Attribute 3 (none): apply (Activate changes in the hardware)
        Attribute 4 (none): reset (Reset the switch)
     --vlan
        Attribute 1 (ports): ports (VLAN port mapping)
     --port
        Attribute 1 (int): pvid (Primary VLAN ID)
        Attribute 2 (unknown): link (Get port link information)

/etc/config/network

config interface 'loopback'
        option ifname 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config interface 'lan'
        option type 'bridge'
        option ifname 'eth0'
        option proto 'static'
        option ipaddr '192.168.1.1'
        option netmask '255.255.255.0'

config switch
        option name 'switch0'
        option reset '1'
        option enable_vlan '1'

config switch_vlan
        option device 'switch0'
        option vlan '1'
        option ports '0 1 2 3 5'

dts

/dts-v1/;

#include "bcm6338.dtsi"

#include <dt-bindings/input/input.h>

/ {
	model = "Broadcom BCM96333AW2G reference board";
	compatible = "brcm,bcm96333aw2g", "brcm,bcm6338";

	chosen {
		bootargs = "rootfstype=squashfs,jffs2 noinitrd console=ttyS0,115200";
		stdout-path = "serial0:115200n8";
	};

	gpio-leds {
		compatible = "gpio-leds";
		/*0-internet_green*/
		/*1-power_green*/
		/*2-internet_red*/
		/*ppp_green {
			label = "96333AW2G:green:ppp";
			gpios = <&gpio0 0 1>;
		};*/
		power_green {
			label = "96333AW2G:green:power";
			gpios = <&gpio0 1 1>;
			default-state = "on";
		};	
		/*ppp_fail_green {
			label = "96333AW2G:green:ppp-fail";
			gpios = <&gpio0 2 1>;
		};*/
	};

	aliases {
		mdio-gpio0 = &mdio0;
	};

	mdio0: mdio {
		compatible = "virtual,mdio-gpio";
		#address-cells = <1>;
		#size-cells = <0>;
		gpios = <&gpio0 3 0 &gpio0 4 0>;
		
		phy@0 {
			reg = <0>;
			ports {
				port0 {
					reg = <0>;
					label = "wan";
				};
				port1 {
					reg = <1>;
					label = "lan1";
				};
				port2 {
					reg = <2>;
					label = "lan2";
				};
				port3 {
					reg = <3>;
					label = "lan3";
				};
				port5 {
					reg = <5>;
					label = "cpu";
				};
			};
		};
	};
};

&lsspi {
	status = "ok";

	flash@0 {
		compatible = "jedec,spi-nor";
		spi-max-frequency = <781000>;
		reg = <0>;

		#address-cells = <1>;
		#size-cells = <1>;

		linux,part-probe = "bcm63xxpart";

		partitions {
			compatible = "brcm,bcm963xx-cfe-nor-partitions";
		};
	};
};

&uart0 {
	status = "ok";
};

bcm63xx_enet.c

diff --git a/drivers/net/ethernet/broadcom/bcm63xx_enet.c b/drivers/net/ethernet/broadcom/bcm63xx_enet.c
index 6f48c3b..c75e3af 100644
--- a/drivers/net/ethernet/broadcom/bcm63xx_enet.c
+++ b/drivers/net/ethernet/broadcom/bcm63xx_enet.c
@@ -35,6 +35,8 @@
 #include <bcm63xx_dev_enet.h>
 #include "bcm63xx_enet.h"
 
+#include <linux/of_mdio.h>
+
 static char bcm_enet_driver_name[] = "bcm63xx_enet";
 static char bcm_enet_driver_version[] = "1.0";
 
@@ -1696,7 +1698,6 @@ static int bcm_enet_probe(struct platform_device *pdev)
 	struct net_device *dev;
 	struct bcm63xx_enet_platform_data *pd;
 	struct resource *res_mem, *res_irq, *res_irq_rx, *res_irq_tx;
-	struct mii_bus *bus;
 	int i, ret;
 
 	if (!bcm_enet_shared_base[0])
@@ -1818,32 +1819,42 @@ static int bcm_enet_probe(struct platform_device *pdev)
 		struct phy_device *phydev;
 		char phy_id[MII_BUS_ID_SIZE + 3];
 
-		priv->mii_bus = mdiobus_alloc();
-		if (!priv->mii_bus) {
-			ret = -ENOMEM;
-			goto out_unregister_netdev;
+		struct device_node *np;
+		np = of_find_compatible_node(NULL, NULL, "virtual,mdio-gpio");
+		if(np) {
+			priv->mii_bus = of_mdio_find_bus(np);
 		}
 
-		bus = priv->mii_bus;
-		bus->name = "bcm63xx_enet MII bus";
-		bus->parent = &pdev->dev;
-		bus->priv = priv;
-		bus->read = bcm_enet_mdio_read_phylib;
-		bus->write = bcm_enet_mdio_write_phylib;
-		sprintf(bus->id, "%s-%d", pdev->name, pdev->id);
-
-		/* only probe bus where we think the PHY is, because
-		 * the mdio read operation return 0 instead of 0xffff
-		 * if a slave is not present on hw */
-		bus->phy_mask = ~(1 << priv->phy_id);
-
-		if (priv->has_phy_interrupt)
-			bus->irq[priv->phy_id] = priv->phy_interrupt;
-
-		ret = mdiobus_register(bus);
-		if (ret) {
-			dev_err(&pdev->dev, "unable to register mdio bus\n");
-			goto out_free_mdio;
+		if(!priv->mii_bus) {
+			struct mii_bus *bus;
+
+			priv->mii_bus = mdiobus_alloc();
+			if (!priv->mii_bus) {
+				ret = -ENOMEM;
+				goto out_unregister_netdev;
+			}
+
+			bus = priv->mii_bus;
+			bus->name = "bcm63xx_enet MII bus";
+			bus->parent = &pdev->dev;
+			bus->priv = priv;
+			bus->read = bcm_enet_mdio_read_phylib;
+			bus->write = bcm_enet_mdio_write_phylib;
+			sprintf(bus->id, "%s-%d", pdev->name, pdev->id);
+
+			/* only probe bus where we think the PHY is, because
+			* the mdio read operation return 0 instead of 0xffff
+			* if a slave is not present on hw */
+			bus->phy_mask = ~(1 << priv->phy_id);
+
+			if (priv->has_phy_interrupt)
+				bus->irq[priv->phy_id] = priv->phy_interrupt;
+
+			ret = mdiobus_register(bus);
+			if (ret) {
+				dev_err(&pdev->dev, "unable to register mdio bus\n");
+				goto out_free_mdio;
+			}
 		}
 
 		/* connect to PHY */
option ports '0 1 2 3 5t' ?

Still the same result. I suspect that the SoC is not connected to the switch by MII, but through one of the switch's network interface. Ergo, SoC internal PHY <-> switch port PHY. I based my hunch on a BCM6333-based board schematic in FCC database:
https://fccid.io/YZAQA-ZBQI-W0NKP4/Schematics/schematic-1448626

The schematic did clue me in on the mdio-gpio setup so I think the SoC-switch interface will be similar as well. After all the MII interface needs more wires than ethernet. This is a very cost-effective design.

I've tried enabling use_internal_phy in the board configuration. But phy_connect failed for the switch.

I will need to phy_connect the internal PHY and switch. There is only 1 MAC in the SoC, which has to be connected to the internal PHY in this case. I would appreciate help from someone experienced in Linux kernel network device driver.

I have solved the switch issue. I would say it's a hack as I just tried whatever solution blindly until the thing works. Again, any help from someone experienced in Linux kernel network device driver would be great.

The mods will can be used on devices using board ID 96333AW2G, namely Innacomm W3400V and Aztech DSL1000EW-L. Other devices like D-Link 2640B ver. T1 might have the same SoC-switch interface design.

I've also added support for VLAN tag ID's higher than 15 on the BCM5325E switch chip. However anyone using that feature must first read the chip's datasheet and understand the its VLAN design.

My public repo:
https://github.com/spliew89/openwrt-bcm6333

2 Likes