DSA and Appletalk multicast issues

I can guarantee that.
The traffic is showing up on other devices on the network. Devices that have a working setup (23.05 or Ubuntu) react to the traffic and respond to it.
Out path works.

I'll work on the capture so I can share it but I think this is the case. tcpdump -e -i br-lan (or lan1) shows multicasts from other devices on the network - MAC address corresponds to the working 23.05 router or to an Ubuntu working install.

Cool, I stand corrected :wink:

So you are not using snapshots then.....

You are not telling something....

So I stand by my comment:

Presumably you have backported/compiled your own fork of 23.05 for this.
I would guess the netatalk package has config options for atalkd to specify the network interface to use.
The problem is that the bridge, with DSA, is not an interface, rather it is a device.

Try specifying one of the bridge device interfaces eg lan1 instead of eg br-lan
Run brctl show and pick one.

That makes it sound shady. :slight_smile: No, not calling anyone into the back alley, just trying to focus the discussion.
I maintain the netatalk package. It is officially in Snapshot and in 24.10. I am hoping to get it backported to 23.05 where it works (PR open), in the meantime I have the netatalk package built for 23.05 with the official SDK. That is the userspace piece.
I opened a PR (now merged) to get the appletalk module back into OpenWrt. It's on snapshot now in standard images. PR open to backport to 24.10. Maybe to 25.05 after? That's the kernel piece.
I use a custom 23.05 image which is identical in config to the official one plus the change in kernel build scripts (modules/netsupport) to include building the appletalk module for 23.05 (details in PR).
I am also in contact with upstream netatalk but I am not part of their team.

That's probably more confusing than anything else I've sent. To summarize, I have:

  • Snapshot official image - nothing custom built - not working (ipq806x and ath97)
  • 24.10 quasi-official image (at head) - netatalk official build - not working (ipq806x and ath97)
  • 23.05 quasi-official image (at 23.05 tag) - SDK netatalk built - working (ipq806x and ath97)
  • Ubuntu with build from netatalk upstream tarball - working.

Also built a bunch of other images at different commit points and that's how I'm certain the change to DSA broke this. That's on the openwrt issue I have open.

What that means is that replicating this is easy:

  • flash snapshot, install netatalk, run atalkd with defaults
  • do the same thing on a second device (for two devices ignoring each other), or better yet, build netatalk on linux (which will work) and have it interact with the snapshot OpenWt device.

No doubt I need a new hobby.

Tried.

root@TestWrt-63:~# brctl show
bridge name	bridge id		STP enabled	interfaces
br-lan		7fff.c47154053a77	no		lan4
							lan2
							lan3
							lan1

Doesn't matter if atalkd runs against br-lan, lan1, or even eth1. Same behaviour.

I don't discount that. But put the other way around, DSA breaks this.
If some config voodoo can unbreak it, or (again, I know, long shot) some change to how OpenWrt enabled DSA allows for this to work, I'd be happy.
This might be a symptom of an underlying issue with DSA.
I would agree the priority of it is below rock bottom but it still matters to me.
Hunting for ideas... like that one:

With that in mind I'll poke at the module's code to at least add debugging all over.

:+1:

One more time for me please, I'm slow...

Inbound frames are seen in a non-promiscuous (-p) tcpdump on lan1?

And the firewall is disabled?

But atalkd does not receive anything on its socket?

No worries! Thank you for putting up with this.

Yes, and on br-lan. As in, if this device has MAC address 99:88:77:66:55:44:

root@TestWrt-63:~# tcpdump -p -e -i lan1 | grep Appletalk
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on lan1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
14:47:34.618308 11:22:33:44:55:66 (oui Unknown) > 09:00:07:ff:ff:ff (oui Unknown), 802.3, length 28: LLC, dsap SNAP (0xaa) Individual, ssap SNAP (0xaa) Command, ctrl 0x03: oui Appletalk (0x080007), pid Appletalk (0x809b), length 20: 6400.148.zip > 0.zip:  at-#6 7

It's dead, Jim. Stopped and disabled.

Exactly.

                 can see                 
     β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€   outgoing               
     β”‚       β”Œβ”€β”€β”€β”€β”€β”€β”€β”€    ───────────┐   
     β”‚       β”‚                       β”‚   
     β–Ό       β–Ό                       β–Ό   
   β”Œβ”€β”€β”€β”   β”Œβ”€β”€β”€β”                   β”Œβ”€β”€β”€β” 
   β”‚   β”‚   β”‚   β”‚                   β”‚   β”‚ 
   β”‚ a β”œβ”€β”€β–Ίβ”‚ b β”œβ”€β”€β”€β”€β”€β”€β”€LAN────────►│ o β”‚ 
   β”‚ t β”‚   β”‚ r β”‚                   β”‚ t β”‚ 
   β”‚ a β”‚   β”‚ - β”‚                   β”‚ h β”‚ 
   β”‚ l β”‚   β”‚ l β”‚                   β”‚ e β”‚ 
   β”‚ k β”‚   β”‚ a β”‚                   β”‚ r β”‚ 
   β”‚ d β”‚   β”‚ n │◄──────LAN──────────   β”‚ 
   β”‚   β”‚   β”‚   β”‚                   β”‚   β”‚ 
   β””β”€β”€β”€β”˜   β””β”€β”€β”€β”˜                   β””β”€β”€β”€β”˜ 
     β–²       β–²                       β–²   
     β”‚       β”‚                       β”‚   
nothing      └──────── can see  β”€β”€β”€β”€β”€β”˜   
from outside            incomming        

And yes... it's probably the module or atalkd... but an image built at the commit prior the device migrated to DSA works beautifully. Image built at commit when device migrated to DSA behaves like this.
I'm looking for a "C for DSA dummies" text I can read that describes how a socket on an interface needs to be changed to keep socketing on a DSA switch. Bit unfair; I would have expected DSA not to impose code changes to daemon/module code and for this to be caused by some decision to have OpenWrt just DSA IP traffic or something like that.

1 Like

Going back to your question...

Yes. The only ethernet cable plugged into this thing is on lan1. If anything is coming into this box, it's through lan1.
Maybe this is relevant then? lan1 sees it coming in, but it's not a "multicast" because it's weird 802.3 packet and not the usual ethertype IPv4? It's almost like the switch didn't know what to do with 09:00:07:ff:ff:ff or a 802.3 packet.

I mean the frame is already "in" after its received on the CPU port, is it not? Landing the frame on lan1 I would imagine is just a case of updating the interface pointer in skb to point to lan1 (after peeling the switch's proprietary tag that says something akin to "this one is inbound for lan1").

Hmm should we not see drops in the interface statistics somewhere, if the frame is weird and gets dropped?

Total shot in the dark here:

grep "" /sys/devices/virtual/net/br-lan/statistics/*
grep IpExt /proc/net/netstat

Ugh, I want to see the frame but I don't want to compile software right now haha

Hm, I briefly considered if some ethertype confusion could arise during the second pass through netif_receive_skb(). But at least on my box the ethertypes used are different.

root@OpenWrt:~# tcpdump -nni eth0 -ec1
tcpdump: listening on eth0, link-type DSA_TAG_EDSA (Marvell EDSA), snapshot length 262144 bytes
21:57:15.831085 12:34:56:78:9a:bc > de:f0:12:34:56:78, Marvell EDSA ethertype 0xdada (Unknown), rsvd 0 0, mode Forward, dev 1, port 0, tagged, VID 1, FPri 0, ethertype IPv4 (0x0800), length 134: (tos 0x48, ttl 64, id 27935, offset 0, flags [DF], proto TCP (6), length 112)
...

Looks like the CPU port receives frames with ethertype 0xDADA on my hardware.

sys/netatalk/at.h:#define ETHERTYPE_AT     0x809B
sys/netatalk/at.h:#define ETHERTYPE_AARP   0x80F3

Would have been really odd too. If one could trigger an extra netif_receive_skb() loop or a spurious detag, that would also mean you could inject frames that spoof coming from an arbitrary switch port.

Btw, did you dissect the frame captured on lan1 to verify that it looks identical to the one the Ubuntu machine sent?

Can't hurt to try :slight_smile:

Could also vlan tag the traffic between router and ubuntu, see if a bit of extra layer in the cake does anything :person_shrugging:

You should have said so in the first place :smiley:
I am surprised though that the PR was accepted without a uci config...

What do you have in your /etc/afp.conf file?

It is a long time since I had to use afp for anything and I do not have any Apple products for testing, but my OpenSuse Laptop should be able to see an afp server so I will try a few things.

Normally on OpenWrt with DSA, to add support for a new protocol, you would load a suitable kmod and then create an interface to use the protocol, using a supporting utility.
Then add that new interface to a bridge.

A typical example is vxlan layer 2 tunnelling, where a vxlan interface is created using ip-full. For example on my test router:

root@meshnode-16ad:/etc# ip -d link show vxlan69
16: vxlan69: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-tun69 state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 96:83:c4:18:16:ad brd ff:ff:ff:ff:ff:ff promiscuity 1 allmulti 1 minmtu 68 maxmtu 65535 
    vxlan id 69 group ff02::69 local fe80::9683:c4ff:fe17:16ad dev br-lan srcport 0 0 dstport 4789 ttl 5 ageing 300 

and:

root@meshnode-16ad:/etc# brctl show
bridge name	bridge id		STP enabled	interfaces
br-tun69		7fff.9683c41816ad	no		vxradio0
							vxlan69
br-lan		7fff.9483c41716ad	yes		phy0-ap0
							eth0.1
							m-11s-0

There is no utility I am aware of that will create an afp interface, so I would expect the afpd to do this as it starts.
This will be expected by DSA as far as I can see.

Another thing, reading the netatalk documentation/man, it is all biassed towards tcp/ip layer 3 rather than the ancient/original layer 2 implementation.
My guess is configuring with the router's ipv6 ula would be more appropriate as listener address...
Just thinking out loud.....

1 Like

I can't say I know everything I should be doing. If we need that, I'll eventually get it done.

That's AFP (Apple Filing Protocol). That can go on IP. And it works. That's what started this for me. I needed afpd working for Time Machine for a High Sierra Mac (High Sierra and before can do Samba shares but cannot use TM on them). We can poke at that but that's not this problem. It works on DSA no issues. It's IP.

This is about DDP or AppleTalk which was the layer 3 protocol for Macs that don't do IP (System 6 or older, System 7-9 without Open Transport, supported up to Lion I think so 10.7). In that case afpd is reached through a DDP port. afpd registers itself on atalkd, and clients talk DDP to atalkd.

I have a test setup with a System 9 PPC Mac VM speaking DDP to the afpd server in the OpenWrt 23.05 install. Works. Goes through wireless and unmanaged switches no problem (I don't have any managed ones). I've seen howto's on how to share a LaserWriter with this, relying on netatalk v2. There is little but some niche demand for this. Someone actually sells an IP gateway for localtalk.

For sure. I used to run Suse. Good memories.
If you are running Snapshot or 24.10, just pull netatalk from OpenWrt downloads for afpd. It is all userspace. The Snapshot package might just work in 23.05 provided dependencies are met (libevent, bdb47, libgcrypt). If not, packages repo has the Makefile and the service file so it is straightforward to build on a OpenWrt SDK.

To fight this DDP thing with OpenSuse, you would need to clone netatalk's repo or get their tarball and build on OpenSuse. It's a straightforward quick build. You will get an atalkd for your platform, and that can be run from shell. OpenSuse should have the appletalk module in their distro (you can modprobe it to check, Ubuntu does). If atalkd is running then afpd will open a port on it and register itself. If you are running snapshot, you can apk add kmod-appletalk. With the netatalk package that will give you the full set for DDP on the router.

As for running the System 9 VM on qemu, it took me a while but it's not complicated. If that's your thing I'll share details. I plan on documenting all this stuff on the OpenWrt wiki one day.

kmod-appletalk

Hmmm... that should be atalkd. And that's what's missing I guess. atalkd before just opened a socket on the interface and it was good to go. If DSA requires a new interface to be created, that's net new. Some of it will fall on atalkd and some other on the kext. This will be fun.

I can see why you need a new interface for a tunnel. Or for a virtual adapter (tap). But for this? I don't see every other daemon (like odhcpd or dnsmasq) creating an interface to get traffic out of the bridge. Those get out of jail free because it's IPv4/v6? More reading to do.

Yeah, because the original layer 2 stuff dropped on netatalk 3... so September 2012? It's back as of this year. Like sun spots... 12 year cycle, I guess.
For the layer 3 stuff you are good to go, even on 23.05 if you build netatalk with the SDK.
For that ancient stuff... I think I'm almost there (maybe bridge family on nftables? maybe poke at the kmod to see where it is dropping the ball?).

I'm an idiot. I tried that and locked myself out of the router.
I'll try again when I know what I'm doing with VLANs.

root@TestWrt-63:~# grep "" /sys/devices/virtual/net/br-lan/statistics/*
/sys/devices/virtual/net/br-lan/statistics/collisions:0
/sys/devices/virtual/net/br-lan/statistics/multicast:19937
/sys/devices/virtual/net/br-lan/statistics/rx_bytes:128554602
/sys/devices/virtual/net/br-lan/statistics/rx_compressed:0
/sys/devices/virtual/net/br-lan/statistics/rx_crc_errors:0
/sys/devices/virtual/net/br-lan/statistics/rx_dropped:0
/sys/devices/virtual/net/br-lan/statistics/rx_errors:0
/sys/devices/virtual/net/br-lan/statistics/rx_fifo_errors:0
/sys/devices/virtual/net/br-lan/statistics/rx_frame_errors:0
/sys/devices/virtual/net/br-lan/statistics/rx_length_errors:0
/sys/devices/virtual/net/br-lan/statistics/rx_missed_errors:0
/sys/devices/virtual/net/br-lan/statistics/rx_nohandler:0
/sys/devices/virtual/net/br-lan/statistics/rx_over_errors:0
/sys/devices/virtual/net/br-lan/statistics/rx_packets:201426
/sys/devices/virtual/net/br-lan/statistics/tx_aborted_errors:0
/sys/devices/virtual/net/br-lan/statistics/tx_bytes:44003734
/sys/devices/virtual/net/br-lan/statistics/tx_carrier_errors:0
/sys/devices/virtual/net/br-lan/statistics/tx_compressed:0
/sys/devices/virtual/net/br-lan/statistics/tx_dropped:0
/sys/devices/virtual/net/br-lan/statistics/tx_errors:0
/sys/devices/virtual/net/br-lan/statistics/tx_fifo_errors:0
/sys/devices/virtual/net/br-lan/statistics/tx_heartbeat_errors:0
/sys/devices/virtual/net/br-lan/statistics/tx_packets:59209
/sys/devices/virtual/net/br-lan/statistics/tx_window_errors:0
root@TestWrt-63:~# grep IpExt /proc/net/netstat
IpExt: InNoRoutes InTruncatedPkts InMcastPkts OutMcastPkts InBcastPkts OutBcastPkts InOctets OutOctets InMcastOctets OutMcastOctets InBcastOctets OutBcastOctets InCsumErrors InNoECTPkts InECT1Pkts InECT0Pkts InCEPkts ReasmOverlaps
IpExt: 0 0 106 56 233103 4 284138979 60906090 3392 1808 238999387 4116 0 469162 0 0 0 0

Doesn't look like dropping anything...

That does not exist as far as I can see:

root@meshnode-16ad:/etc# apk update
fetch https://downloads.openwrt.org/snapshots/targets/ramips/mt76x8/packages/packages.adb
fetch https://downloads.openwrt.org/snapshots/packages/mipsel_24kc/base/packages.adb
fetch https://downloads.openwrt.org/snapshots/targets/ramips/mt76x8/kmods/6.6.61-1-be498157c017b974fc7edc6dc515f412/packages.adb
fetch https://downloads.openwrt.org/snapshots/packages/mipsel_24kc/luci/packages.adb
fetch https://downloads.openwrt.org/snapshots/packages/mipsel_24kc/packages/packages.adb
fetch https://downloads.openwrt.org/snapshots/packages/mipsel_24kc/routing/packages.adb
fetch https://downloads.openwrt.org/snapshots/packages/mipsel_24kc/telephony/packages.adb
 [https://downloads.openwrt.org/snapshots/targets/ramips/mt76x8/packages/packages.adb]
 [https://downloads.openwrt.org/snapshots/packages/mipsel_24kc/base/packages.adb]
 [https://downloads.openwrt.org/snapshots/targets/ramips/mt76x8/kmods/6.6.61-1-be498157c017b974fc7edc6dc515f412/packages.adb]
 [https://downloads.openwrt.org/snapshots/packages/mipsel_24kc/luci/packages.adb]
 [https://downloads.openwrt.org/snapshots/packages/mipsel_24kc/packages/packages.adb]
 [https://downloads.openwrt.org/snapshots/packages/mipsel_24kc/routing/packages.adb]
 [https://downloads.openwrt.org/snapshots/packages/mipsel_24kc/telephony/packages.adb]
OK: 10619 distinct packages available
root@meshnode-16ad:/etc# apk list | grep "kmod-appletalk"
root@meshnode-16ad:/etc# 


^^ Nothing found....

No need, it is readily available on Yast. Got it.

Layer 2 or layer 3, it is all the same to the bridge (as layer 3 sits on top of layer 2).
What is missing is protocol support on an interface of the bridge device.
If that was available, the afp service would be able to pick up requests and send on the bridge, but it cannot.

They already have interfaces with ip protocol support, so yes they get out of jail free!

Needs to be v4.0.x

It's there.
https://downloads.openwrt.org/snapshots/targets/ramips/mt76x8/kmods/6.6.63-1-005e3b048dd95438397271ca431fe1c0/

Depends how recent your snapshot is. 6.6.61 doesn't have it, but 6.6.63 does.

Tried. No change in behaviour.

As far as tcpdump will say. Will get wireshark installed and capture. But tcpdump says:
Local package (given MAC address and DDP address):
20:01:51.863600 11:22:33:44:55:66:77 > 09:00:07:ff:ff:ff, 802.3, length 43: LLC, dsap SNAP (0xaa) Individual, ssap SNAP (0xaa) Command, ctrl 0x03: oui Appletalk (0x080007), pid Appletalk (0x809b), length 35: et1 300.180.1 > 0.1: at-rtmp 22
Remote:
20:01:44.614838 00:11:22:33:44:55:c9 > 09:00:07:ff:ff:ff, 802.3, length 37: LLC, dsap SNAP (0xaa) Individual, ssap SNAP (0xaa) Command, ctrl 0x03: oui Appletalk (0x080007), pid Appletalk (0x809b), length 29: et1 6400.148.1 > 0.1: at-rtmp 16
Ubuntu reports the same thing.

afp service will pick up requests because it is IP enabled. That one works right off.
For AppleTalk file sharing (or afp through DDP and not IP) the atalkd and kernel module are needed.
I'll poke around to figure out what is missing so protocol support is added to the existing interfaces as happens for IP.
It's just complicated to juggle all the different builds for this to happen.

it does look similar.

or tcpdump ... -s0 -w /tmp/my.pcap and open on another machine with wireshark already installed.

Hmm just looked up what is needed to run ftrace / bpftrace / etc and stick ones nose in net / skb / sock functions and sniff around. From Brendan Gregg's excellent homepage:

CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
CONFIG_BPF_JIT=y
CONFIG_HAVE_EBPF_JIT=y
CONFIG_BPF_EVENTS=y
CONFIG_FTRACE_SYSCALLS=y
CONFIG_FUNCTION_TRACER=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_DYNAMIC_FTRACE=y
CONFIG_HAVE_KPROBES=y
CONFIG_KPROBES=y
CONFIG_KPROBE_EVENTS=y
CONFIG_ARCH_SUPPORTS_UPROBES=y
CONFIG_UPROBES=y
CONFIG_UPROBE_EVENTS=y
CONFIG_DEBUG_FS=y

And then with fx. just debugfs and ftrace:

# cd /sys/kernel/debug/tracing
# echo netif_receive_skb > set_ftrace_filter
# echo function_graph > current_tracer
# cat trace_pipe
... enjoy ...
# echo nop > current_tracer
# echo > set_ftrace_filter

Food for thought.

1 Like

Ah yes, I'm on 6.6.61 just now.

Indeed.

I'll build a new snapshot image on the Firmware Selector and see what I get.

It's only 3.1.18.....

Nice ToDo list to work through...

@apccv
Reflashed.

Now I get:

root@meshnode-16ad:~# lsmod | grep appletalk
appletalk              28672  0 

Default configuration:

root@meshnode-16ad:~# cat /etc/afp.conf
;
; Netatalk 4.x configuration file
;

[Global]
; Global server settings

[Homes]
basedir regex = /home

; [My AFP Volume]
; path = /path/to/volume

; [My Time Machine Volume]
; path = /path/to/backup
; time machine = yes
root@meshnode-16ad:~# 

Service is running and syslog gives:

Thu Nov 28 11:26:47 2024 daemon.notice afpd[7113]: Netatalk AFP/TCP listening on 192.168.1.249:548

So it is running AFP/TCP by default, on the ipv4 address of the router.

What should be in the config to disable AFP/TCP and enable AFP/DDP?

EDIT: I tried appletalk = yes, no difference.