[PoC] [WIP] DSA support for ath79 built-in switch

For Developers
1

Hi,

When playing around with my Ruckus ZF7372 and trying to get its secondary Ethernet port to report actual link status to userspace, I decided to take the plunge, and try to enable DSA ar9331 driver for built-in switch of AR9344 et al. It did probe and report the correct link status to userspace, however I could not get any traffic to get through.

Next, I dug up my old MR3420v2 with modded flash and RAM, to try on device with fuller configuration, and the results are more or less, the same:

  • DSA tree is set up properly
  • No traffic gets through on the DSA ports - I can see some traffic with Atheros header on the switch upstream port, when the link gets up, but not much else
  • GMAC1 ends up as eth0 due to being the DSA CPU port and probe ordering
  • No traffic on GMAC0 (now eth1) as well, when the appropriate PHY exposed by ar9331 gets attached to it, even though link state is reported correctly, and I get bombarded with TX timeouts.

My boot log showing this is here:

***************************************
*     U-Boot 1.1.4-7a540a78-clean     *
*          Build: 2018-02-23          *
***************************************

** Warning: bad env CRC, using default,
   use 'saveenv' to save it in FLASH

  BOARD: TP-Link TL-MR3420 v2
    SOC: AR9341 rev. 3
    CPU: MIPS 74Kc
    RAM: 64 MB DDR1 16-bit CL3-3-3-8
  FLASH: 16 MB Winbond W25Q128
    MAC: 30:B5:C2:DD:81:CC
 CLOCKS: CPU/RAM/AHB/SPI/REF
         550/400/200/ 25/ 25 MHz

Hit any key to stop booting:  0

Booting image from 0x9F020000...

   Vendor/image name:    OpenWrt r19486+166-e20de2244270
   Hardware ID:          0x34200002
   Whole image size:     15.5 MB (16252928 bytes)
   Kernel size:          2.4 MB (2565103 bytes)
   Rootfs size:          5.3 MB (5535786 bytes)
   Kernel load address:  0x80060000
   Kernel entry point:   0x80060000

   Header CRC...  skipped
   Data CRC...    skipped

Stopping network... OK!
Uncompressing Kernel... OK!
Starting kernel...

[    0.000000] Linux version 5.15.34 (lechu@buildserver) (mips-openwrt-linux-musl-gcc (OpenWrt GCC 11.2.0 r19486+138-e20de2244270) 11.2.0, GNU ld (GNU Binutils) 2.37) #0 Sun Jun 5 11:35:56 2022
[    0.000000] printk: bootconsole [early0] enabled
[    0.000000] CPU0 revision is: 0001974c (MIPS 74Kc)
[    0.000000] MIPS: machine is TP-Link TL-MR3420 v2
[    0.000000] SoC: Atheros AR9341 rev 3
[    0.000000] Initrd not found or empty - disabling initrd
[    0.000000] Primary instruction cache 64kB, VIPT, 4-way, linesize 32 bytes.
[    0.000000] Primary data cache 32kB, 4-way, VIPT, cache aliases, linesize 32 bytes
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x0000000000000000-0x0000000003ffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x0000000003ffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x0000000003ffffff]
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 16240
[    0.000000] Kernel command line: console=ttyS0,115200 rootfstype=squashfs,jffs2
[    0.000000] Dentry cache hash table entries: 8192 (order: 3, 32768 bytes, linear)
[    0.000000] Inode-cache hash table entries: 4096 (order: 2, 16384 bytes, linear)
[    0.000000] Writing ErrCtl register=00000000
[    0.000000] Readback ErrCtl register=00000000
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 55072K/65536K available (6292K kernel code, 612K rwdata, 1412K rodata, 1212K init, 218K bss, 10464K reserved, 0K cma-reserved)
[    0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] NR_IRQS: 51
[    0.000000] random: get_random_bytes called from start_kernel+0x548/0x740 with crng_init=0
[    0.000000] CPU clock: 550.000 MHz
[    0.000000] clocksource: MIPS: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 6950037990 ns
[    0.000002] sched_clock: 32 bits at 275MHz, resolution 3ns, wraps every 7809031678ns
[    0.008886] Calibrating delay loop... 274.02 BogoMIPS (lpj=1370112)
[    0.085933] pid_max: default: 32768 minimum: 301
[    0.091414] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
[    0.099747] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
[    0.115021] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.126265] futex hash table entries: 256 (order: -1, 3072 bytes, linear)
[    0.134146] pinctrl core: initialized pinctrl subsystem
[    0.141450] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[    0.148864] thermal_sys: Registered thermal governor 'step_wise'
[    0.170254] clocksource: Switched to clocksource MIPS
[    0.184018] NET: Registered PF_INET protocol family
[    0.189754] IP idents hash table entries: 2048 (order: 2, 16384 bytes, linear)
[    0.198974] tcp_listen_portaddr_hash hash table entries: 512 (order: 0, 4096 bytes, linear)
[    0.208557] TCP established hash table entries: 1024 (order: 0, 4096 bytes, linear)
[    0.217296] TCP bind hash table entries: 1024 (order: 0, 4096 bytes, linear)
[    0.225334] TCP: Hash tables configured (established 1024 bind 1024)
[    0.232713] UDP hash table entries: 256 (order: 0, 4096 bytes, linear)
[    0.240170] UDP-Lite hash table entries: 256 (order: 0, 4096 bytes, linear)
[    0.248502] NET: Registered PF_UNIX/PF_LOCAL protocol family
[    0.255037] PCI: CLS 0 bytes, default 32
[    0.263411] workingset: timestamp_bits=14 max_order=14 bucket_order=0
[    0.276372] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    0.283082] jffs2: version 2.2 (NAND) (SUMMARY) (LZMA) (RTIME) (CMODE_PRIORITY) (c) 2001-2006 Red Hat, Inc.
[    0.295506] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
[    0.308040] pinctrl-single 1804002c.pinmux: 544 pins, size 68
[    0.316003] Serial: 8250/16550 driver, 16 ports, IRQ sharing enabled
[    0.326208] printk: console [ttyS0] disabled
[    0.331228] 18020000.uart: ttyS0 at MMIO 0x18020000 (irq = 9, base_baud = 1562500) is a 16550A
[    0.341071] printk: console [ttyS0] enabled
[    0.341071] printk: console [ttyS0] enabled
[    0.350131] printk: bootconsole [early0] disabled
[    0.350131] printk: bootconsole [early0] disabled
[    0.383842] spi-nor spi0.0: w25q128 (16384 Kbytes)
[    0.388886] 3 fixed-partitions partitions found on MTD device spi0.0
[    0.395515] Creating 3 MTD partitions on "spi0.0":
[    0.400511] 0x000000000000-0x000000020000 : "u-boot"
[    0.408398] 0x000000020000-0x000000ff0000 : "firmware"
[    0.415300] 2 tplink-fw partitions found on MTD device firmware
[    0.421529] Creating 2 MTD partitions on "firmware":
[    0.426670] 0x000000000000-0x0000002725ef : "kernel"
[    0.431823] mtd: partition "kernel" doesn't end on an erase/write block -- force read-only
[    0.443492] 0x0000002725f0-0x000000fd0000 : "rootfs"
[    0.448652] mtd: partition "rootfs" doesn't start on an erase/write block boundary -- force read-only
[    0.459243] mtd: device 3 (rootfs) set to be root filesystem
[    0.466404] 1 squashfs-split partitions found on MTD device rootfs
[    0.472890] 0x0000007c0000-0x000000fd0000 : "rootfs_data"
[    0.479629] 0x000000ff0000-0x000001000000 : "art"
[    0.881368] ag71xx 19000000.eth: Could not connect to PHY device. Deferring probe.
[    1.565538] ag71xx 1a000000.eth: connected to PHY at fixed-0:00 [uid=00000000, driver=Generic PHY]
[    1.575629] eth0: Atheros AG71xx at 0xba000000, irq 5, mode: gmii
[    1.582486] i2c_dev: i2c /dev entries driver
[    1.588896] NET: Registered PF_INET6 protocol family
[    1.602777] Segment Routing with IPv6
[    1.606630] In-situ OAM (IOAM) with IPv6
[    1.610905] NET: Registered PF_PACKET protocol family
[    1.616244] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[    1.630018] 8021q: 802.1Q VLAN Support v1.8
[    1.635630] gpio-4 (tp-link:power:usb): hogged as output/high
[    1.981366] ag71xx 19000000.eth: Could not connect to PHY device. Deferring probe.
[    2.017142] DSA: failed to set STP state 3 (-122)
[    2.022101] ar9331_switch mdio.0:10: configuring for fixed/gmii link mode
[    2.030039] ar9331_switch mdio.0:10 lan4 (uninitialized): PHY [!ahb!eth@1a000000!mdio!switch@10:04] driver [Generic PHY] (irq=POLL)
[    2.044292] ar9331_switch mdio.0:10 lan1 (uninitialized): PHY [!ahb!eth@1a000000!mdio!switch@10:01] driver [Generic PHY] (irq=POLL)
[    2.058604] ar9331_switch mdio.0:10 lan2 (uninitialized): PHY [!ahb!eth@1a000000!mdio!switch@10:02] driver [Generic PHY] (irq=POLL)
[    2.072927] ar9331_switch mdio.0:10 lan3 (uninitialized): PHY [!ahb!eth@1a000000!mdio!switch@10:03] driver [Generic PHY] (irq=POLL)
[    2.086398] DSA: failed to set STP state 0 (-122)
[    2.091442] DSA: tree 0 setup
[    2.097111] ar9331_switch mdio.0:10: Link is Up - 1Gbps/Full - flow control off
[    2.432082] ag71xx 19000000.eth: connected to PHY at !ahb!eth@1a000000!mdio!switch@10:00 [uid=004dd042, driver=Generic PHY]
[    2.444743] eth1: Atheros AG71xx at 0xb9000000, irq 4, mode: mii
[    2.463364] VFS: Mounted root (squashfs filesystem) readonly on device 31:3.
[    2.478438] Freeing unused kernel image (initmem) memory: 1212K
[    2.484622] This architecture does not have kernel memory protection.
[    2.491304] Run /sbin/init as init process
[    2.580279] random: fast init done
[    3.323605] init: Console is alive
[    3.327701] init: - watchdog -
[    5.610494] kmodloader: loading kernel modules from /etc/modules-boot.d/*
[    5.739777] usbcore: registered new interface driver usbfs
[    5.745644] usbcore: registered new interface driver hub
[    5.751331] usbcore: registered new device driver usb
[    5.812347] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    5.833174] SCSI subsystem initialized
[    5.845259] ehci-fsl: Freescale EHCI Host controller driver
[    5.855123] ehci-platform: EHCI generic platform driver
[    5.861070] ehci-platform 1b000000.usb: EHCI Host Controller
[    5.866990] ehci-platform 1b000000.usb: new USB bus registered, assigned bus number 1
[    5.875312] ehci-platform 1b000000.usb: irq 3, io mem 0x1b000000
[    5.910286] ehci-platform 1b000000.usb: USB 2.0 started, EHCI 1.00
[    5.917871] hub 1-0:1.0: USB hub found
[    5.922823] hub 1-0:1.0: 1 port detected
[    5.933425] kmodloader: done loading kernel modules from /etc/modules-boot.d/*
[    5.951584] init: - preinit -
[    7.421360] random: jshn: uninitialized urandom read (4 bytes read)
[    7.620924] random: jshn: uninitialized urandom read (4 bytes read)
[    7.726699] random: jshn: uninitialized urandom read (4 bytes read)
[    8.412711] DSA: failed to set STP state 3 (-122)
[    8.417613] ar9331_switch mdio.0:10 lan1: configuring for phy/internal link mode
[    8.427048] eth0: link up (1000Mbps/Full duplex)
[    8.433440] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Press the [f] key and hit [enter] to enter failsafe mode
Press the [1], [2], [3] or [4] key and hit [enter] to select the debug level
[    8.646733] urandom_read: 4 callbacks suppressed
[    8.646761] random: procd: uninitialized urandom read (4 bytes read)
[   13.007254] mount_root: loading kmods from internal overlay
[   13.079382] kmodloader: loading kernel modules from //etc/modules-boot.d/*
[   13.088475] kmodloader: done loading kernel modules from //etc/modules-boot.d/*
[   14.181233] jffs2: notice: (414) jffs2_build_xattr_subsystem: complete building xattr subsystem, 6 of xdatum (0 unchecked, 1 orphan) and 7 of xref (1 dead, 0 orphan) found.
[   14.198098] block: attempting to load /tmp/jffs_cfg/upper/etc/config/fstab
[   14.210955] block: extroot: not configured
[   14.283941] jffs2: notice: (412) jffs2_build_xattr_subsystem: complete building xattr subsystem, 6 of xdatum (0 unchecked, 1 orphan) and 7 of xref (1 dead, 0 orphan) found.
[   14.997941] block: attempting to load /tmp/jffs_cfg/upper/etc/config/fstab
[   15.009684] block: extroot: not configured
[   15.015701] mount_root: switching to jffs2 overlay
[   15.033630] overlayfs: upper fs does not support tmpfile.
[   15.046852] urandom-seed: Seeding with /etc/urandom.seed
[   15.236571] DSA: failed to set STP state 0 (-122)
[   15.256797] procd: - early -
[   15.260235] procd: - watchdog -
[   15.969050] random: jshn: uninitialized urandom read (4 bytes read)
Failed to connect to ubus
[   16.140426] procd: - watchdog -
[   16.150735] procd: - ubus -
[   16.565665] random: ubusd: uninitialized urandom read (4 bytes read)
[   16.948754] procd: - init -
Please press Enter to activate this console.
[   19.265322] kmodloader: loading kernel modules from /etc/modules.d/*
[   20.039202] urngd: v1.0.2 started.
[   20.458289] random: crng init done
[   20.689510] fuse: init (API version 7.34)
[   20.747715] Loading modules backported from Linux version v5.15.33-0-g06f50ca83ace
[   20.755620] Backport generated by backports.git v5.15.33-1-0-g183c4ab2
[   20.935315] xt_time: kernel timezone is -0000
[   21.664940] PPP generic driver version 2.4.2
[   21.692891] NET: Registered PF_PPPOX protocol family
[   21.954802] ieee80211 phy0: Atheros AR9340 Rev:3 mem=0xb8100000, irq=2
[   22.023197] kmodloader: done loading kernel modules from /etc/modules.d/*
[   51.471944] eth0: link down
[   51.516461] eth0: link up (1000Mbps/Full duplex)
[   51.529056] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[   51.539637] DSA: failed to set STP state 3 (-122)
[   51.544608] ar9331_switch mdio.0:10 lan1: configuring for phy/internal link mode
[   51.568404] br-lan: port 1(lan1) entered blocking state
[   51.573902] br-lan: port 1(lan1) entered disabled state
[   51.579722] device lan1 entered promiscuous mode
[   51.584583] device eth0 entered promiscuous mode
[   51.714279] DSA: failed to set STP state 3 (-122)
[   51.719180] ar9331_switch mdio.0:10 lan2: configuring for phy/internal link mode
[   51.748207] br-lan: port 2(lan2) entered blocking state
[   51.753702] br-lan: port 2(lan2) entered disabled state
[   51.759510] device lan2 entered promiscuous mode
[   51.797147] DSA: failed to set STP state 3 (-122)
[   51.802119] ar9331_switch mdio.0:10 lan3: configuring for phy/internal link mode
[   51.830883] br-lan: port 3(lan3) entered blocking state
[   51.836326] br-lan: port 3(lan3) entered disabled state
[   51.842209] device lan3 entered promiscuous mode
[   51.885804] DSA: failed to set STP state 3 (-122)
[   51.890788] ar9331_switch mdio.0:10 lan4: configuring for phy/internal link mode
[   51.920917] br-lan: port 4(lan4) entered blocking state
[   51.926356] br-lan: port 4(lan4) entered disabled state
[   51.938393] device lan4 entered promiscuous mode
[   54.023317] eth1: link up (100Mbps/Full duplex)
[   54.028079] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[   69.140262] ------------[ cut here ]------------
[   69.145063] WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:477 dev_watchdog+0x298/0x2a0
[   69.153660] NETDEV WATCHDOG: eth1 (ag71xx): transmit queue 0 timed out
[   69.160419] Modules linked in: ath9k ath9k_common pppoe ppp_async iptable_nat ath9k_hw ath xt_state xt_nat xt_conntrack xt_REDIRECT xt_MASQUERADE xt_FLOWOFFLOAD xt_CT pppox ppp_generic nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_pptp nf_nat_irc nf_nat_h323 nf_nat_amanda nf_nat nf_flow_table nf_conntrack_tftp nf_conntrack_snmp nf_conntrack_sip nf_conntrack_pptp nf_conntrack_irc nf_conntrack_h323 nf_conntrack_broadcast nf_conntrack_amanda nf_conntrack mac80211 lzo ipt_REJECT cfg80211 xt_time xt_tcpudp xt_multiport xt_mark xt_mac xt_limit xt_comment xt_TCPMSS xt_LOG ts_kmp ts_fsm ts_bm slhc nf_reject_ipv4 nf_log_syslog nf_defrag_ipv6 nf_defrag_ipv4 mdio_netlink lzo_rle lzo_decompress lzo_compress iptable_raw iptable_mangle iptable_filter ip_tables crc_ccitt compat asn1_decoder fuse ledtrig_usbport xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip
[   69.160884]  ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 vfat fat nls_utf8 nls_iso8859_1 nls_cp437 sha256_generic libsha256 seqiv jitterentropy_rng drbg hmac cmac crypto_acompress fsl_mph_dr_of ehci_platform ehci_fsl sd_mod scsi_mod scsi_common ehci_hcd gpio_button_hotplug ext4 mbcache jbd2 usbcore nls_base usb_common crc16 mii crc32c_generic
[   69.296076] CPU: 0 PID: 0 Comm: swapper Not tainted 5.15.34 #0
[   69.302119] Stack : 00000000 00000000 80c09dc4 809b0000 807f0000 80726010 807f0310 807efea3
[   69.310806]         809b32d4 00000000 00000000 800c2210 8071edc8 00000001 80c09d80 6ad703c8
[   69.319483]         00000000 00000000 80726010 80c09c18 ffffefff 00000000 00000000 ffffffea
[   69.328169]         000000cd 80c09c24 000000cd 807f5fd8 809b0000 00000009 80c09e60 804f24a8
[   69.336860]         00000009 00000000 ffffa5d2 8085c174 00000018 803b1100 00000000 809b0000
[   69.345543]         ...
[   69.348076] Call Trace:
[   69.350610] [<80067068>] show_stack+0x28/0xf0
[   69.355149] [<80086d1c>] __warn+0xc0/0x12c
[   69.359402] [<80086e14>] warn_slowpath_fmt+0x8c/0xac
[   69.364550] [<804f24a8>] dev_watchdog+0x298/0x2a0
[   69.369419] [<800d2954>] call_timer_fn.constprop.0+0x1c/0x8c
[   69.375311] [<800d2da4>] run_timer_softirq+0x2ec/0x388
[   69.380645] [<80683fe0>] __do_softirq+0x118/0x2cc
[   69.385529] [<80343d50>] plat_irq_dispatch+0x94/0xc8
[   69.390686] [<80062738>] handle_int+0x138/0x144
[   69.395375] [<80683d7c>] r4k_wait_irqoff+0x18/0x24
[   69.400344] [<80683e2c>] default_idle_call+0x20/0x3c
[   69.405483] [<800b1e28>] do_idle+0x98/0x108
[   69.409820] [<800b20e4>] cpu_startup_entry+0x18/0x28
[   69.414968] [<80881ec4>] start_kernel+0x70c/0x740
[   69.419847] 
[   69.421398] ---[ end trace 4cdaf1414fc94c6d ]---
[   69.426172] eth1: tx timeout

and this is what I get when LAN1 port is enabled:

root@OpenWrt:/# tcpdump -i eth0 -nvvXX
tcpdump: listening on eth0, link-type NULL (BSD loopback), capture size 262144 bytes
[  211.944661] ar9331_switch mdio.0:10 lan1: Link is Up - 100Mbps/Full - flow control rx/tx
[  211.955578] br-lan: port 1(lan1) entered blocking state
[  211.961050] br-lan: port 1(lan1) entered forwarding state
[  211.969737] IPv6: ADDRCONF(NETDEV_CHANGE): br-lan: link becomes ready
11:40:47.345153 AF Unknown (858996850), length 172: 
        0x0000:  7240 3333 0000 0016 30b5 c2dd 81cc 86dd  r@33....0.......
        0x0010:  6000 0000 0074 0001 0000 0000 0000 0000  `....t..........
        0x0020:  0000 0000 0000 0000 ff02 0000 0000 0000  ................
        0x0030:  0000 0000 0000 0016 3a00 0502 0000 0100  ........:.......
        0x0040:  8f00 e171 0000 0005 0400 0000 ff02 0000  ...q............
        0x0050:  0000 0000 0000 0001 ffdd 81cc 0400 0000  ................
        0x0060:  ff05 0000 0000 0000 0000 0000 0001 0003  ................
        0x0070:  0400 0000 ff02 0000 0000 0000 0000 0000  ................
        0x0080:  0001 0002 0400 0000 ff02 0000 0000 0000  ................
        0x0090:  0000 0001 ff00 0001 0400 0000 ff02 0000  ................
        0x00a0:  0000 0000 0000 0000 0000 0002            ............
11:40:47.385140 AF Unknown (858996850), length 88: 
        0x0000:  7240 3333 ffdd 81cc 30b5 c2dd 81cc 86dd  r@33....0.......
        0x0010:  6000 0000 0020 3aff 0000 0000 0000 0000  `.....:.........
        0x0020:  0000 0000 0000 0000 ff02 0000 0000 0000  ................
        0x0030:  0000 0001 ffdd 81cc 8700 42a6 0000 0000  ..........B.....
        0x0040:  fe80 0000 0000 0000 32b5 c2ff fedd 81cc  ........2.......
        0x0050:  0e01 0031 8678 abc5                      ...1.x..
11:40:47.455136 AF Unknown (858996850), length 88: 
        0x0000:  7240 3333 ff00 0001 30b5 c2dd 81cc 86dd  r@33....0.......
        0x0010:  6000 0000 0020 3aff 0000 0000 0000 0000  `.....:.........
        0x0020:  0000 0000 0000 0000 ff02 0000 0000 0000  ................
        0x0030:  0000 0001 ff00 0001 8700 8de5 0000 0000  ................
        0x0040:  fd16 7dc7 1126 0000 0000 0000 0000 0001  ..}..&..........
        0x0050:  0e01 9eb4 3b1c 78e2                      ....;.x.
11:40:47.785134 AF Unknown (858996850), length 172: 
        0x0000:  7240 3333 0000 0016 30b5 c2dd 81cc 86dd  r@33....0.......
        0x0010:  6000 0000 0074 0001 0000 0000 0000 0000  `....t..........
        0x0020:  0000 0000 0000 0000 ff02 0000 0000 0000  ................
        0x0030:  0000 0000 0000 0016 3a00 0502 0000 0100  ........:.......
        0x0040:  8f00 e171 0000 0005 0400 0000 ff02 0000  ...q............
        0x0050:  0000 0000 0000 0001 ffdd 81cc 0400 0000  ................
        0x0060:  ff05 0000 0000 0000 0000 0000 0001 0003  ................
        0x0070:  0400 0000 ff02 0000 0000 0000 0000 0000  ................
        0x0080:  0001 0002 0400 0000 ff02 0000 0000 0000  ................
        0x0090:  0000 0001 ff00 0001 0400 0000 ff02 0000  ................
        0x00a0:  0000 0000 0000 0000 0000 0002            ............
11:40:48.395434 AF Unknown (858996850), length 192: 
        0x0000:  7240 3333 0000 0016 30b5 c2dd 81cc 86dd  r@33....0.......
        0x0010:  6000 0000 0088 0001 fe80 0000 0000 0000  `...............
        0x0020:  32b5 c2ff fedd 81cc ff02 0000 0000 0000  2...............
        0x0030:  0000 0000 0000 0016 3a00 0502 0000 0100  ........:.......
        0x0040:  8f00 6a77 0000 0006 0400 0000 ff02 0000  ..jw............
        0x0050:  0000 0000 0000 0001 ff00 0000 0400 0000  ................
        0x0060:  ff02 0000 0000 0000 0000 0001 ffdd 81cc  ................
        0x0070:  0400 0000 ff05 0000 0000 0000 0000 0000  ................
        0x0080:  0001 0003 0400 0000 ff02 0000 0000 0000  ................
        0x0090:  0000 0000 0001 0002 0400 0000 ff02 0000  ................
        0x00a0:  0000 0000 0000 0001 ff00 0001 0400 0000  ................
        0x00b0:  ff02 0000 0000 0000 0000 0000 0000 0002  ................
11:40:48.425256 AF Unknown (858996850), length 92: 
        0x0000:  7240 3333 0000 0016 30b5 c2dd 81cc 86dd  r@33....0.......
        0x0010:  6000 0000 0024 0001 fe80 0000 0000 0000  `....$..........
        0x0020:  32b5 c2ff fedd 81cc ff02 0000 0000 0000  2...............
        0x0030:  0000 0000 0000 0016 3a00 0502 0000 0100  ........:.......
        0x0040:  8f00 faa9 0000 0001 0400 0000 ff02 0000  ................
        0x0050:  0000 0000 0000 0001 ff00 0000            ............
11:40:48.565237 AF Unknown (858996850), length 92: 
        0x0000:  7240 3333 0000 0016 30b5 c2dd 81cc 86dd  r@33....0.......
        0x0010:  6000 0000 0024 0001 fe80 0000 0000 0000  `....$..........
        0x0020:  32b5 c2ff fedd 81cc ff02 0000 0000 0000  2...............
        0x0030:  0000 0000 0000 0016 3a00 0502 0000 0100  ........:.......
        0x0040:  8f00 faa9 0000 0001 0400 0000 ff02 0000  ................
        0x0050:  0000 0000 0000 0001 ff00 0000            ............
11:40:49.115235 AF Unknown (858996850), length 192: 
        0x0000:  7240 3333 0000 0016 30b5 c2dd 81cc 86dd  r@33....0.......
        0x0010:  6000 0000 0088 0001 fe80 0000 0000 0000  `...............
        0x0020:  32b5 c2ff fedd 81cc ff02 0000 0000 0000  2...............
        0x0030:  0000 0000 0000 0016 3a00 0502 0000 0100  ........:.......
        0x0040:  8f00 6a77 0000 0006 0400 0000 ff02 0000  ..jw............
        0x0050:  0000 0000 0000 0001 ff00 0000 0400 0000  ................
        0x0060:  ff02 0000 0000 0000 0000 0001 ffdd 81cc  ................
        0x0070:  0400 0000 ff05 0000 0000 0000 0000 0000  ................
        0x0080:  0001 0003 0400 0000 ff02 0000 0000 0000  ................
        0x0090:  0000 0000 0001 0002 0400 0000 ff02 0000  ................
        0x00a0:  0000 0000 0000 0001 ff00 0001 0400 0000  ................
        0x00b0:  ff02 0000 0000 0000 0000 0000 0000 0002  ................
11:41:04.321386 AF Unknown (858996850), length 176: 
        0x0000:  7240 3333 0000 0001 30b5 c2dd 81cc 86dd  r@33....0.......
        0x0010:  6005 a8b4 0078 3aff fe80 0000 0000 0000  `....x:.........
        0x0020:  32b5 c2ff fedd 81cc ff02 0000 0000 0000  2...............
        0x0030:  0000 0000 0000 0001 8600 bdb7 40c0 0000  ............@...
        0x0040:  0000 0000 0000 0000 0101 30b5 c2dd 81cc  ..........0.....
        0x0050:  0501 0000 0000 05dc 0304 40c0 ffff ffff  ..........@.....
        0x0060:  ffff ffff 0000 0000 fd16 7dc7 1126 0000  ..........}..&..
        0x0070:  0000 0000 0000 0000 1803 3000 0000 0708  ..........0.....
        0x0080:  fd16 7dc7 1126 0000 0000 0000 0000 0000  ..}..&..........
        0x0090:  1903 0000 0000 0708 fd16 7dc7 1126 0000  ..........}..&..
        0x00a0:  0000 0000 0000 0001 0701 0000 0009 27c0  ..............'.
11:41:20.323586 AF Unknown (858996850), length 176: 
        0x0000:  7240 3333 0000 0001 30b5 c2dd 81cc 86dd  r@33....0.......
        0x0010:  6005 a8b4 0078 3aff fe80 0000 0000 0000  `....x:.........
        0x0020:  32b5 c2ff fedd 81cc ff02 0000 0000 0000  2...............
        0x0030:  0000 0000 0000 0001 8600 bdb7 40c0 0000  ............@...
        0x0040:  0000 0000 0000 0000 0101 30b5 c2dd 81cc  ..........0.....
        0x0050:  0501 0000 0000 05dc 0304 40c0 ffff ffff  ..........@.....
        0x0060:  ffff ffff 0000 0000 fd16 7dc7 1126 0000  ..........}..&..
        0x0070:  0000 0000 0000 0000 1803 3000 0000 0708  ..........0.....
        0x0080:  fd16 7dc7 1126 0000 0000 0000 0000 0000  ..}..&..........
        0x0090:  1903 0000 0000 0708 fd16 7dc7 1126 0000  ..........}..&..
        0x00a0:  0000 0000 0000 0001 0701 0000 0009 27c0  ..............'.
[  252.501877] ar9331_switch mdio.0:10 lan1: Link is Down
[  252.507289] br-lan: port 1(lan1) entered disabled state

output of ip -a

root@OpenWrt:/# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 30:b5:c2:dd:81:cc brd ff:ff:ff:ff:ff:ff
    inet6 fe80::32b5:c2ff:fedd:81cc/64 scope link 
       valid_lft forever preferred_lft forever
3: lan4@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master br-lan state LOWERLAYERDOWN group default qlen 1000
    link/ether 30:b5:c2:dd:81:cc brd ff:ff:ff:ff:ff:ff
4: lan1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-lan state UP group default qlen 1000
    link/ether 30:b5:c2:dd:81:cc brd ff:ff:ff:ff:ff:ff
5: lan2@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master br-lan state LOWERLAYERDOWN group default qlen 1000
    link/ether 30:b5:c2:dd:81:cc brd ff:ff:ff:ff:ff:ff
6: lan3@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master br-lan state LOWERLAYERDOWN group default qlen 1000
    link/ether 30:b5:c2:dd:81:cc brd ff:ff:ff:ff:ff:ff
7: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
    link/ether 30:b5:c2:dd:81:c9 brd ff:ff:ff:ff:ff:ff
8: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/sit 0.0.0.0 brd 0.0.0.0
9: wlan0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 30:b5:c2:dd:81:cc brd ff:ff:ff:ff:ff:ff
10: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 30:b5:c2:dd:81:cc brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.1/24 brd 192.168.1.255 scope global br-lan
       valid_lft forever preferred_lft forever
    inet6 fd5c:3091:e169::1/60 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::32b5:c2ff:fedd:81cc/64 scope link 
       valid_lft forever preferred_lft forever

And now, the questions:

  • Has anyone here attempted bringing up that driver?
  • I suspect that either parsing Atheros header is broken, or the PHY-to-MAC mapping in device tree (see ar934x.dtsi is borked)? Ideas on how to diagnose that?
  • Any tips, how to trace down what causes TX timeouts on GMAC0?

My sources are here, on top of AR83x7 DSA PoC by @Ansuel: https://github.com/Leo-PL/openwrt/commits/ath79-builtin-switch-dsa-poc

1 Like
2

For the atheros header you should first check if the switch actually send data back... you should manually check the header and check the destination port...

3

Now I wonder, where does that strange address family 7240 3333 originate, and what is the actual offset of the Atheros header, at the capture it looks to me like it is at 0x14, and it should be at 0xC, but it looks shifted by the additional header: 7240 3333 0000 0016 - or it is the second, 4-byte form of Atheros header, which I read about in AR8327 datasheet - however AR9344 datasheet doesn't mention it.

4

for the switch family the header should really be the same... (about location)
what could be different is the size and how things are placed in the header...

in theory should always be 2-byte after the transportation header... but would should first check if the atheros header feature is enabled (it's disabled by default) could be that option is not present in the PoC dsa driver?

But anyway the header should have magic values to be identified so it's not that hard once you understand the logic

5

By studying the driver code, it for sure does enable the header here: https://github.com/torvalds/linux/blob/952923ddc01120190dcf671e7b354364ce1d1362/drivers/net/dsa/qca/ar9331.c#L410

And supposedly, the driver selects support for it when built-in.

6

mhhh can you link the documentation and what do you think it's the atheros header? wonder if we are missing something very obvious

7

See page 67, section 3.9.14 Atheros Header Configuration
I think it is either at 0x4 or 0x14.
I can see, that in every packet, the device MAC appears at 0x8.

8

wow the documentation is plain shit.... AHAHAH think you have to find another revision cause the current version lacks of many thing...

We don't have the transmit header documentation...
And the receive part lacks of the packet type table...

image
image654×609 70.7 KB

This is from qca8337

image
image723×737 129 KB

As you can see what we have is the receive (this is by the point of view of the system so receive = switch sends to the system)
The packet type is probably always 0 but that is at a different mask so that should be fixed if we intend to use the current qca tagger...

Also again i'm asking you what is in your opinion the entire 16 bit header that you are receiving so we can parse it and check it does contains correct data... (and also check if the switch transmit tagged packet)
Again it should always start with the bit 10

from documentation it's after the packet SourceAddress

9

I have yet to discover it, frankly. When I set up the DHCP client on the switch bridge, to get some more familiar data, this is what I get:

14:24:59.607402 AF Unknown (4294918258), length 344: 
        0x0000:  7240 ffff ffff ffff 30b5 c2dd 81cc 0800  r@......0.......
        0x0010:  4500 0148 0000 0000 4011 79a6 0000 0000  E..H....@.y.....
        0x0020:  ffff ffff 0044 0043 0134 8e5e 0101 0600  .....D.C.4.^....
        0x0030:  19e5 4332 0075 0000 0000 0000 0000 0000  ..C2.u..........
        0x0040:  0000 0000 0000 0000 30b5 c2dd 81cc 0000  ........0.......
        0x0050:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0060:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0070:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0080:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0090:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x00a0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x00b0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x00c0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x00d0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x00e0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x00f0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0100:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0110:  0000 0000 0000 0000 6382 5363 3501 0139  ........c.Sc5..9
        0x0120:  0202 4037 0801 0306 0c0f 1c2a 790c 0f74  ..@7.......*y..t
        0x0130:  706c 696e 6b2d 6d72 3334 3230 7632 3c0c  plink-mr3420v2<.
        0x0140:  7564 6863 7020 312e 3335 2e30 ff00 0000  udhcp.1.35.0....
        0x0150:  0000 0000 0000 0000                      ........

So 7240 seems to be the magic value, probably taken from AR7240, then we get dst address (ffff ffff ffff), source address(30b5 c2dd 81cc), so the offset is 0xE, but then bit 15 and 7 should be set, and here, only bit 16 is set. Doh.

AR9344 documentation seems to be a bit more complete: https://raw.githubusercontent.com/Deoptim/atheros/master/AR9344_May_2012.pdf
See page 109, section 4.9.14 Atheros Header Configuration

10

The thing is that it shouldn't be there... Also what are you using to dump the packet (i mean the option you are passing)? Are you sure we are correctly printing the DA and SA and packet type values?

11

I'm using tcpdump -i eth0 -nvvXX to get the most raw data. Now that I set up the wireless client on the device, I can connect with Wireshark, to get a reliable capture.

12

Ok much better

image
image661×611 94.1 KB

So it's a completely different implementation of the qca_tag driver we have for ar8337
(or i'm missing something and the tagger is already fixed?)

@Leo-PL i'm not finding where is the tagger driver CONFIG_NET_DSA_TAG_AR9344

13

It is here: https://github.com/torvalds/linux/blob/master/net/dsa/tag_ar9331.c
Seems as simple as it gets, but nothing mentions that magic value at the beginning :expressionless:

14

Mhhh i'm getting confused by the implementation.... i mean it's strange... wonder if we can test a variant with the values from the documentation... i can do a sample driver so you can test if you want...

(but again you should check that you are dumping the packet the correct way as with the wrong option the DA SA and packet type are not printed in the log

But again by the looks of it the mask and values doesn't match ar9344 documentation AT ALL so it does make sense that the packet are getting lost.

15

Maybe the driver wasn't tested at all at this family, as there are apparently some differences between the SoC's - it even mentions them, I need to check if I have anything with exactly AR9331 and multiple ports...
Edit: I have a MR3220v2 - how lucky! :wink:

I'll do a conversion for it and see how it goes - too bad that I wanted to use it as a wireless client for a PIU arcade machine. But I'll probably replace it with something different, then.

16

well these SoC probably where on a transition phase and the implementation wasn't stable for them... Again fixing the tagger driver shouldn't difficult at all... I can totally provide you a patch so you can test... just tell me if you want to waste some time with me. :smiley:

17

Why not, it's for science after all :wink:

18

ok i'm preparing the patch...

19

@Leo-PL here is the patch... you will porbably have to fix the makefile and kconfig thing

I also added a debug print in the driver to check if we gets recv header packets and if they reflect the documentation... (at this point i don't trust it at all)

20

Just did the conversion for MR3220v2 and guess what... it just worked. WAN port on GMAC0 also works. So we're looking at some more substantial differences here - I guess.
The great thing is that I can actually use that MR3220v2 like I wanted to, because it just works :smiley:

I probably need to modify the AR9331 DSA driver to use new tagger, am I right?
I'll pull your driver in layer today, and do necessary adjustments to the switch driver itself.

Edit: Just as I thought, the performance is abysmal :smiley:

$ iperf3 -c 192.168.1.1
Connecting to host 192.168.1.1, port 5201
[  5] local 192.168.1.4 port 41574 connected to 192.168.1.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  5.82 MBytes  48.8 Mbits/sec   10   56.6 KBytes       
[  5]   1.00-2.00   sec  6.52 MBytes  54.7 Mbits/sec   12   48.1 KBytes       
[  5]   2.00-3.00   sec  6.34 MBytes  53.2 Mbits/sec    5   56.6 KBytes       
[  5]   3.00-4.00   sec  6.40 MBytes  53.7 Mbits/sec   12   43.8 KBytes       
[  5]   4.00-5.00   sec  6.40 MBytes  53.7 Mbits/sec    5   53.7 KBytes       
[  5]   5.00-6.00   sec  6.21 MBytes  52.2 Mbits/sec    8   65.0 KBytes       
[  5]   6.00-7.00   sec  6.59 MBytes  55.3 Mbits/sec   12   46.7 KBytes       
[  5]   7.00-8.00   sec  6.34 MBytes  53.2 Mbits/sec    5   56.6 KBytes       
[  5]   8.00-9.00   sec  6.21 MBytes  52.1 Mbits/sec   12   43.8 KBytes       
[  5]   9.00-10.00  sec  6.34 MBytes  53.2 Mbits/sec   12   48.1 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  63.2 MBytes  53.0 Mbits/sec   93             sender
[  5]   0.00-10.01  sec  62.9 MBytes  52.7 Mbits/sec                  receiver

iperf Done.
$ iperf3 -Rc 192.168.1.1
Connecting to host 192.168.1.1, port 5201
Reverse mode, remote host 192.168.1.1 is sending
[  5] local 192.168.1.4 port 41578 connected to 192.168.1.1 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  4.88 MBytes  40.9 Mbits/sec                  
[  5]   1.00-2.00   sec  4.74 MBytes  39.8 Mbits/sec                  
[  5]   2.00-3.00   sec  5.64 MBytes  47.2 Mbits/sec                  
[  5]   3.00-4.00   sec  4.68 MBytes  39.3 Mbits/sec                  
[  5]   4.00-5.00   sec  2.91 MBytes  24.4 Mbits/sec                  
[  5]   5.00-6.00   sec  5.27 MBytes  44.2 Mbits/sec                  
[  5]   6.00-7.00   sec  4.84 MBytes  40.6 Mbits/sec                  
[  5]   7.00-8.00   sec  4.76 MBytes  40.0 Mbits/sec                  
[  5]   8.00-9.00   sec  3.69 MBytes  30.9 Mbits/sec                  
[  5]   9.00-10.00  sec  4.46 MBytes  37.4 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  46.0 MBytes  38.5 Mbits/sec  181             sender
[  5]   0.00-10.00  sec  45.9 MBytes  38.5 Mbits/sec                  receiver

iperf Done.

For comparison: On WAN port (out of DSA tree):

$ iperf3 -Rc 192.168.1.1
Connecting to host 192.168.1.1, port 5201
Reverse mode, remote host 192.168.1.1 is sending
[  5] local 192.168.1.4 port 41880 connected to 192.168.1.1 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  7.43 MBytes  62.3 Mbits/sec                  
[  5]   1.00-2.00   sec  8.02 MBytes  67.3 Mbits/sec                  
[  5]   2.00-3.00   sec  7.46 MBytes  62.6 Mbits/sec                  
[  5]   3.00-4.00   sec  8.19 MBytes  68.7 Mbits/sec                  
[  5]   4.00-5.00   sec  7.82 MBytes  65.6 Mbits/sec                  
[  5]   5.00-6.00   sec  8.76 MBytes  73.5 Mbits/sec                  
[  5]   6.00-7.00   sec  11.2 MBytes  93.8 Mbits/sec                  
[  5]   7.00-8.00   sec  10.0 MBytes  84.2 Mbits/sec                  
[  5]   8.00-9.00   sec  7.30 MBytes  61.2 Mbits/sec                  
[  5]   9.00-10.00  sec  7.44 MBytes  62.4 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.02  sec  83.8 MBytes  70.1 Mbits/sec    0             sender
[  5]   0.00-10.00  sec  83.6 MBytes  70.2 Mbits/sec                  receiver

iperf Done.
$ iperf3 -c 192.168.1.1
Connecting to host 192.168.1.1, port 5201
[  5] local 192.168.1.4 port 41888 connected to 192.168.1.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  7.10 MBytes  59.5 Mbits/sec   13   41.0 KBytes       
[  5]   1.00-2.00   sec  6.77 MBytes  56.9 Mbits/sec    8   43.8 KBytes       
[  5]   2.00-3.00   sec  6.90 MBytes  57.9 Mbits/sec   10   49.5 KBytes       
[  5]   3.00-4.00   sec  7.08 MBytes  59.4 Mbits/sec   14   49.5 KBytes       
[  5]   4.00-5.00   sec  6.90 MBytes  57.9 Mbits/sec   16   48.1 KBytes       
[  5]   5.00-6.00   sec  6.90 MBytes  57.8 Mbits/sec    7   46.7 KBytes       
[  5]   6.00-7.00   sec  6.71 MBytes  56.4 Mbits/sec    9   41.0 KBytes       
[  5]   7.00-8.00   sec  10.3 MBytes  86.0 Mbits/sec   16   49.5 KBytes       
[  5]   8.00-9.00   sec  11.1 MBytes  92.8 Mbits/sec   31   58.0 KBytes       
[  5]   9.00-10.00  sec  8.20 MBytes  68.8 Mbits/sec   17   43.8 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  77.9 MBytes  65.3 Mbits/sec  141             sender
[  5]   0.00-10.00  sec  77.5 MBytes  65.0 Mbits/sec                  receiver

iperf Done.

And on 19.07 with swconfig:

lechu@pumbaa-vm ~> iperf3 -c 10.0.0.254
Connecting to host 10.0.0.254, port 5201
[  5] local 10.0.0.122 port 59326 connected to 10.0.0.254 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  11.2 MBytes  93.9 Mbits/sec   23   43.8 KBytes       
[  5]   1.00-2.00   sec  11.0 MBytes  92.3 Mbits/sec   19   63.6 KBytes       
[  5]   2.00-3.00   sec  11.1 MBytes  92.8 Mbits/sec   22   56.6 KBytes       
[  5]   3.00-4.00   sec  10.7 MBytes  89.6 Mbits/sec   16   48.1 KBytes       
[  5]   4.00-5.00   sec  8.70 MBytes  73.0 Mbits/sec   14   43.8 KBytes       
[  5]   5.00-6.00   sec  11.0 MBytes  92.3 Mbits/sec   17   43.8 KBytes       
[  5]   6.00-7.00   sec  11.0 MBytes  92.3 Mbits/sec   13   58.0 KBytes       
[  5]   7.00-8.00   sec  10.9 MBytes  91.7 Mbits/sec   19   50.9 KBytes       
[  5]   8.00-9.00   sec  10.8 MBytes  90.2 Mbits/sec   20   49.5 KBytes       
[  5]   9.00-10.00  sec  8.64 MBytes  72.5 Mbits/sec   14   50.9 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   105 MBytes  88.1 Mbits/sec  177             sender
[  5]   0.00-10.01  sec   105 MBytes  87.8 Mbits/sec                  receiver

iperf Done.
lechu@pumbaa-vm ~> iperf3 -Rc 10.0.0.254
Connecting to host 10.0.0.254, port 5201
Reverse mode, remote host 10.0.0.254 is sending
[  5] local 10.0.0.122 port 59346 connected to 10.0.0.254 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  7.77 MBytes  65.2 Mbits/sec                  
[  5]   1.00-2.00   sec  6.01 MBytes  50.4 Mbits/sec                  
[  5]   2.00-3.00   sec  7.61 MBytes  63.9 Mbits/sec                  
[  5]   3.00-4.00   sec  10.4 MBytes  87.3 Mbits/sec                  
[  5]   4.00-5.00   sec  10.3 MBytes  86.7 Mbits/sec                  
[  5]   5.00-6.00   sec  10.8 MBytes  90.3 Mbits/sec                  
[  5]   6.00-7.00   sec  7.40 MBytes  62.0 Mbits/sec                  
[  5]   7.00-8.00   sec  8.82 MBytes  74.0 Mbits/sec                  
[  5]   8.00-9.00   sec  9.48 MBytes  79.5 Mbits/sec                  
[  5]   9.00-10.00  sec  9.94 MBytes  83.4 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.01  sec  88.6 MBytes  74.3 Mbits/sec   64             sender
[  5]   0.00-10.00  sec  88.5 MBytes  74.3 Mbits/sec                  receiver

iperf Done.

Edit2: since it works, I pushed that conversion to Github.