Netatmo HomeKit native devices, not able to update iOS Home app when isolated (but work online/using Netamo app)

[spence]

I will go through that capture and comment on a few things later.

For future captures, please add the 'v' (verbose) option unless a specific capture doesn't need it. I would like to see 'tcp' vs 'udp' clearly showing in the output.
tcpdump -ennv -i wl0-ap1 ...

Using the filter expression 'ether host 70:ee:50:6d:c6:80' and the -e option is useful in early investigation to see what the hosts are doing, ether host usually only applies when capturing on the same layer2 network. Use the appropriate ethernet address that applies on a different l2 segment to see the related traffic on the different network or use the related ip addresses etc. as appropriate.

Using the ether host expression was useful because it showed the DHCP traffic and it show that the Netatmo device (thermostat?) is also using IPv6. It could be that the Netatmo devices work with HomeKit using IPv6 when on the same L2 network. It is possible unless proven to be false.
.
.
.

@giuliomagnifico

another thing, when I try to add again the Netatmo thermostat, when it's in the netamo interface, and with the firewall that is blocking the access from Netatmo->lan, it fails, but every time I press "try again" I see some requests from a different port:

09:37:37.110929 f8:b1:dd:a6:53:80 > 70:ee:50:6d:c6:80, ethertype IPv4 (0x0800), length 54: 192.168.6.127.49406 > 192.168.6.149.25050: Flags [.], ack 5, win 65535, length 0

the next try

09:39:19.320635 70:ee:50:6d:c6:80 > f8:b1:dd:a6:53:80, ethertype IPv4 (0x0800), length 88: 192.168.6.149.25050 > 192.168.6.127.49407: Flags [P.], seq 310:344, ack 190, win 2811, length 34

Looks like those devices are randomizing the ports that they're using to search for HomeKit devices inside the LAN, this could be why they're working from the Netatmo website/app but not inside HomeKit (?).

Notice that the first is from 192.168.6.127.49406 > 192.168.6.149.25050 and the second is in the opposite direction: 192.168.6.149.25050 > 192.168.6.127.49407, not that it matters for this discussion. Port randomization usually picks non-sequential port numbers. It is common with many protocols that a different source port is used for each new connection so what you shared is expected unless you know that the protocol in use by Netatmo is different.
.
.
.
If IP addresses are working, let us dig into mDNS.
If you still have your Netatmo thermostat and weather station on the Netatmo SSID and network (192.168.6.0/24) with the WAX206 interface of wl0-ap1, please do a new, different tcpdump. Run it long enough to see repeated mDNS queries from the Netatmo devices and any responses from the other side. Same for queries and responses in the opposite direction:
tcpdump -ennv -i wl0-ap1 port 5353

If nothing is seen coming back from the "LAN" through avahi, 192.168.1.0/24 then from a new terminal window, do 2 new captures, still on WAX206 starting at the same time with a similar capture on the br-lan side in the added ssh session:
tcpdump -ennv -i br-lan port 5353
and a new one for tcpdump -ennv -i wl0-ap1 port 5353 .

Use the appropriate 'wlX-apY' for network "LAN" if needed to see wifi traffic for network 192.168.1.0/24.

Please share the complete output as quoted text. Feel free to PM them and any other communications to me if you don't want to share with the world. You can update the "solution" at the end.

EDIT: Please include any traffic from the "iot" network. It is useful for comparisons.
Please identify what host the IP addresses belong to and describe the function if you think it may not be clear to me. Netatmo thermostat, weather station, router, HomeBridge, HomeHub, Mac, etc.
We should avoid using pronouns or other terms that may be ambiguous or otherwise unclear. This one can be a challenge for me
.
.
Are all the hosts of interest on wifi?

Is Avahi still running on WAX206?
.
.
.

Info: Keeping the captures on each interface separate helps keep track of what is seen where.

I will try to work on keeping future questions / responses to a narrow focus.

[spence]

I spent a lot of time researching and writing down details of the setup so I didn't write up info on the full tcpdump session you shared earlier. In summary, the capture shows DHCP working, Netatmo devices using IPv6, doing mDNSqueries and sending replies, doing normal dns, arp requests, and having a tcp session with a server on the Internet.
.
.
.
Here are my notes so far ( time for me to eat a late supper and relax an hour before bedtime. )

Info about the full capture session and starting a list of what we know and what we assume.

Know: All traffic from "LAN" to "iot" and "Netatmo" is allowed:

Know: Apple Home (HomeKit) system natively uses HAP, HomeKit Accessory Protocol, on Ethernet/WIFI.

Assume: Communication is direct between the Home app and each Homekit (HAP) capable appliance. With your Mac or iPhone or iPad connected to your home wifi, each should be having direct communication with every managed (Home) device.

I (Spence) do not know what other intermediate devices you have or what they do. Things like a bridge, hub, pod.

Assume: Everything learns about all other devices through Bonjour / mDNS.
Assume: Each system advertises the service it provides via mDNS, including 'TXT' and 'SRV' records including the TCP port to connect on.

Firewall Rule:

config rule
	option dest_port '5353'
	option target 'ACCEPT'
	option src_port '5353'
	option src 'iot'
	list proto 'udp'
	option dest 'lan'
	option name 'IoT mDNS port 5353'
	list dest_ip '224.0.0.251'
	list dest_ip 'ff02::fb'

Note that this rule allows mDNS to the router (WAX206) on IPv6 even though IPv6 routing is not configured on the WAX206. Avahi on WAX206 should be proxying mDNS.

Know: mDNS uses local_only IP multicast.
Know: mDNS uses IPv4 multicast group address 224.0.0.251 and UDP port 5353 on Ethernet multicast address 01:00:5E:00:00:FB.
Know: mDNS uses IPv4 multicast group address ff02::fb and UDP port 5353 on Ethernet multicast address 33:33:00:00:00:FB.

I saw it stated that mDNS replies "should" be to the unicast address of the querier but it is clear that we have seen replies to 224.0.0.251. We should be open to either styles.

As an example for the Home app on your mac asking for appliance info we should expect to see packet captures with the following src and dst:

Query:
WAX206 int wl0-ap0: 192.168.1.10:5353 > 224.0.0.251:5353 proto UDP
WAX206 int wl0-ap1: 192.168.6.1:5353 > 224.0.0.251:5353 proto UDP

Reply:
WAX206 int wl0-ap1: 192.168.6.127:5353 > 224.0.0.251:5353 proto UDP
WAX206 int wl0-ap0: 192.168.1.3:5353 > 224.0.0.251:5353 proto UDP

Is wlo-ap0 bound to the LAN network, same as br-lan?

Assume: HAP and Home uses the ports advertised in mDNS.

Know: We have seen The Home protocol use TCP ports 51826 and 51827.
You have a rule on WAX206:

config rule
	option target 'ACCEPT'
	option src_port '51826-51827'
	option dest_port '51826-51827'
	option src 'iot'
	option dest 'lan'
	option name 'HomeKit connectivity'

The "LAN" network has some added complexity. There is the main interface for it on R4S with IP address 192.168.1.1 and DHCP gives .1 as the default gateway.
However, WAX206 connects via 'LAN' with address .3. Normally any packets destined for network 192.168.1.0/24 would be sent direct to the hosts ethernet address via the ARP entry on WAX206. Various security improvements over the years may cause this to not work as hosts only know their default route via 192.168.1.1 and send to it for further routing. When replies come from the MAC address of the WAX206, a host may not accept the packet. Either router may also choose not to forward the packet depending on security settings. This may not be a problem as many hosts in the 'iot' network are communicating with hosts in the 'LAN' network.
A work-around is in place, an additional route table on WAX206:

config route
	option interface 'LAN'
	option gateway '192.168.1.3'
	option source '192.168.1.2'
	option target '192.168.5.0/24'

So, if this is working, we should see unicast IP traffic from 'iot' or 'Netatmo' networks destined for 'LAN' forwarded to R4S, and R4S send to host MAC addresses based on ARP entries.
since mDNS uses local_only multicast, we need to see how it behaves when proxied by Avahi on WAX206 with the custom route.

The end.

[giuliomagnifico]

Woah, thanks for the super-exhaustive reply! I try to reply:

@spence

Using the ether host expression was useful because it showed the DHCP traffic and it show that the Netatmo device (thermostat?) is also using IPv6. It could be that the Netatmo devices work with HomeKit using IPv6 when on the same L2 network. It is possible unless proven to be false

Yes is the thermostat, I'm using it because it's easy to reset. About the IPv6, yes but it should be the same when in the other subnet and in all my network the ipv6 is disabled.

@spence

Notice that the first is from 192.168.6.127.49406 > 192.168.6.149.25050 and the second is in the opposite direction: 192.168.6.149.25050 > 192.168.6.127.49407, not that it matters for this discussion. Port randomization usually picks non-sequential port numbers. It is common with many protocols that a different source port is used for each new connection so what you shared is expected unless you know that the protocol in use by Netatmo is different

My thought indeed is that the Netatmo devices are using random ports also to communicate inside the LAN. It would be improbable because they are using standard protocols...but...

@spence

If IP addresses are working, let us dig into mDNS.
If you still have your Netatmo thermostat and weather station on the Netatmo SSID and network (192.168.6.0/24) with the WAX206 interface of wl0-ap1, please do a new, different tcpdump. Run it long enough to see repeated mDNS queries from the Netatmo devices and any responses from the other side. Same for queries and responses in the opposite direction:
tcpdump -ennv -i wl0-ap1 port 5353

Yes, I will use Pastebin now, because there's a words limit here on the forum, here is the output: https://pastebin.com/xMY2xp3D but since there's nothing from the 'lan' side, here is the tcpdump -ennv -i br-lan port 5353, and the tcpdump -ennv -i br-lan port 5353.

@spence

EDIT: Please include any traffic from the "iot" network. It is useful for comparisons.
Please identify what host the IP addresses belong to and describe the function if you think it may not be clear to me. Netatmo thermostat, weather station, router, HomeBridge, HomeHub, Mac, etc.
We should avoid using pronouns or other terms that may be ambiguous or otherwise unclear. This one can be a challenge for me

I think the name of the devices is explicit anyway the
Thermostat is at: 70:ee:50:6d:c6:80 and 192.168.6.247
Weather Station 70:EE:50:65:73:8A and 192.168.6.150

@spence

Are all the hosts of interest on wifi?

Is Avahi still running on WAX206?

Yes to both, only my TV is cabled to my Lan but is in the .1.x subnet.

Avahi is running with this config:

[server]
#host-name=foo
#domain-name=local
use-ipv4=yes
use-ipv6=yes
check-response-ttl=no
use-iff-running=no

[publish]
publish-addresses=yes
publish-hinfo=yes
publish-workstation=no
publish-domain=yes
#publish-dns-servers=192.168.1.1
#publish-resolv-conf-dns-servers=yes

[reflector]
enable-reflector=yes
reflect-ipv=no

[rlimits]
#rlimit-as=
rlimit-core=0
rlimit-data=4194304
rlimit-fsize=0
rlimit-nofile=30
rlimit-stack=4194304
rlimit-nproc=3

@spence

I (Spence) do not know what other intermediate devices you have or what they do. Things like a bridge, hub, pod.

There's only the Netgear switch between but it hasn't any configuration, it's doing nothing, only adding ports.

@spence

Note that this rule allows mDNS to the router (WAX206) on IPv6 even though IPv6 routing is not configured on the WAX206. Avahi on WAX206 should be proxying mDNS

I added the ipv6 address because only the ipv4 wasn't working and I noted that other Apple devices are using also the ipv6, I can remove it but is the same.

@spence

As an example for the Home app on your mac asking for appliance info we should expect to see packet captures with the following src and dst:

Query:
WAX206 int wl0-ap0: 192.168.1.10:5353 > 224.0.0.251:5353 proto UDP
WAX206 int wl0-ap1: 192.168.6.1:5353 > 224.0.0.251:5353 proto UDP

Reply:
WAX206 int wl0-ap1: 192.168.6.127:5353 > 224.0.0.251:5353 proto UDP
WAX206 int wl0-ap0: 192.168.1.3:5353 > 224.0.0.251:5353 proto UDP

wl0-ap0 is the 'iot' interface and has the 192.168.5.1, the Netatmo interface is just to debug without rely on the iot interface but has the same rules/config/etc... of the iot interface as I wrote before.

@spence

Is wlo-ap0 bound to the LAN network, same as br-lan?

No, is bound to nothing, is not assigned because I had weird issue if doing it, anyway I also tried to bound it to br-lan but is the same.

@spence

The "LAN" network has some added complexity. There is the main interface for it on R4S with IP address 192.168.1.1 and DHCP gives .1 as the default gateway.
However, WAX206 connects via 'LAN' with address .3. Normally any packets destined for network 192.168.1.0/24 would be sent direct to the hosts ethernet address via the ARP entry on WAX206. Various security improvements over the years may cause this to not work as hosts only know their default route via 192.168.1.1 and send to it for further routing. When replies come from the MAC address of the WAX206, a host may not accept the packet. Either router may also choose not to forward the packet depending on security settings. This may not be a problem as many hosts in the 'iot' network are communicating with hosts in the 'LAN' network.
A work-around is in place, an additional route table on WAX206:

config route
	option interface 'LAN'
	option gateway '192.168.1.3'
	option source '192.168.1.2'
	option target '192.168.5.0/24'

This is what I thought, I added the routing table at the beginning of this reply, and I also tried it some days before, but without any help.

@spence

This may not be a problem as many hosts in the 'iot' network are communicating with hosts in the 'LAN' network

Exactly, but instead it's a problem for these Netatmo devices! The only difference is that those devices don't rely on the Homebridge server at 192.168.1.5 because they're HomeKit native, so they should communicate with the hub/homepod directly. But this doesn't happen when they are in the 'iot' or 'netatmo' interface/subnet as the other devices in the same interface are doing.

thanks a lot for this extended reply, I have no idea what else I can do...other people are reporting that Netatmo devices in separated VLANs are working (after using Avahi as reflector), so this is my network issue.

[giuliomagnifico]

After investigating I'm starting think that Avahi is not working as expected with these Netatmo devices, I try to explain:

When the Netatmo devices are isolated from the lan, I absolutely can't see any query when I listen from the WAX206 with avahi-browse -arp.

As soon as I bond the Netatmo wireless interface to the br-lan interface, they appear! But note the IPs (I separated with white lines for easy readability):

root@WAX206:/etc/avahi# avahi-browse -arp
...............
=;br-lan;IPv4;MacBook\032Air;_airplay._tcp;local;MacBook-Air.local;192.168.1.103;7000;"srcvers=675.4.1" "pk=6fe401bbca6a752c126a0fbd36fbe5267a539b5bd785b0f99bd3770ac720cfcc" "psi=5CFB1E52-17CB-45A9-9D50-485E568203CF" "pi=0f45d894-46b5-421f-a59a-51e046da8964" "protovers=1.1" "at=4" "model=MacBookAir10,1" "gcgl=0" "igl=0" "gid=E3FA7F36-2BDD-4B22-8F1B-2607B9D468C4" "flags=0x204" "rsf=0x8" "features=0x4A7FCFD5,0xB8154FDE" "fex=1c9/St5PFbgmIQ" "deviceid=1C:91:80:EA:46:61" "acl=0" "act=2"
=;br-lan;IPv4;HomePodSensor\032469446;_hap._tcp;local;Salotto.local;192.168.1.127;61204;"sh=yi0Ddg==" "ci=10" "sf=0" "s#=7" "pv=1.1" "md=HomePod" "id=5A:CF:80:22:AA:E5" "ff=2" "c#=1"
=;br-lan;IPv4;homebridge-lgwebos-tv\0329992;_hap._tcp;local;homebridge.local;192.168.1.5;33600;"sh=xZ2SGg==" "ci=2" "sf=0" "s#=1" "pv=1.1" "md=homebridge" "id=0E:C9:34:9B:85:24" "ff=0" "c#=2"
=;br-lan;IPv4;RPi\0322EDD;_hap._tcp;local;homebridge.local;192.168.1.5;35521;"sh=foRhag==" "ci=2" "sf=0" "s#=1" "pv=1.1" "md=homebridge" "id=0E:AA:D3:5A:E1:5C" "ff=0" "c#=4"
=;br-lan;IPv4;Meross\0325B65;_hap._tcp;local;homebridge.local;192.168.1.5;53167;"sh=kMp34A==" "ci=2" "sf=0" "s#=1" "pv=1.1" "md=homebridge" "id=0E:66:A1:4E:B2:EA" "ff=0" "c#=4"
=;br-lan;IPv4;homebridge-prometheus-exporter\0326097;_hap._tcp;local;homebridge.local;192.168.1.5;35834;"sh=wOpYyw==" "ci=2" "sf=0" "s#=1" "pv=1.1" "md=homebridge" "id=0E:C6:E7:21:D6:C4" "ff=0" "c#=2"
=;br-lan;IPv4;Televisione\0327492;_hap._tcp;local;homebridge.local;192.168.1.5;38855;"sh=IFwVEg==" "ci=31" "sf=0" "s#=1" "pv=1.1" "md=Model Name" "id=0B:74:47:D6:70:6B" "ff=0" "c#=52"
=;br-lan;IPv4;Broadlink\032RM\0324A8F;_hap._tcp;local;homebridge.local;192.168.1.5;33440;"sh=eKzYEw==" "ci=2" "sf=0" "s#=1" "pv=1.1" "md=homebridge" "id=0E:B7:47:FD:F0:28" "ff=0" "c#=4"
=;br-lan;IPv4;SwitchBot\032E202;_hap._tcp;local;homebridge.local;192.168.1.5;49850;"sh=yE+FFA==" "ci=2" "sf=0" "s#=1" "pv=1.1" "md=homebridge" "id=0E:66:68:8A:59:5F" "ff=0" "c#=3"
=;br-lan;IPv4;Homebridge\03227ED;_hap._tcp;local;homebridge.local;192.168.1.5;51625;"sh=zeQfPA==" "ci=2" "sf=1" "s#=1" "pv=1.1" "md=homebridge" "id=0E:94:69:C8:F3:6E" "ff=0" "c#=2"
+;br-lan;IPv4;Giulio\226\128\153s\032iPhone;_rdlink._tcp;local
+;br-lan;IPv4;70\058b3\05806\0581d\0585e\058d9\064fe80\058\05872b3\0586ff\058fe1d\0585ed9-supportsRP;_apple-mobdev2._tcp;local
=;br-lan;IPv4;Giulio\226\128\153s\032iPhone;_rdlink._tcp;local;Giulios-iPhone.local;192.168.1.109;50712;"rpAD=a498ad3f4f0e" "rpVr=430.3" "rpBA=08:8E:C5:42:74:7B"
=;br-lan;IPv4;70\058b3\05806\0581d\0585e\058d9\064fe80\058\05872b3\0586ff\058fe1d\0585ed9-supportsRP;_apple-mobdev2._tcp;local;Giulios-iPhone.local;192.168.1.109;32498;
-;br-lan;IPv4;Giulio\226\128\153s\032iPhone;_rdlink._tcp;local
-;br-lan;IPv4;70\058b3\05806\0581d\0585e\058d9\064fe80\058\05872b3\0586ff\058fe1d\0585ed9-supportsRP;_apple-mobdev2._tcp;local


+;br-lan;IPv4;Netatmo\032Relay;_hap._tcp;local
=;br-lan;IPv4;Netatmo\032Relay;_hap._tcp;local;Netatmo\032Relay.local;169.254.129.198;5001;"sh=LJ/KDg==" "ci=2" "na_tkn=cd35dff9ce88" "id=70:60:BB:30:BF:3C" "md=Netatmo Relay\000" "pv=1.1" "sf=0" "ff=1" "s#=1" "c#=9"
+;br-lan;IPv4;Weather\032Station;_hap._tcp;local
=;br-lan;IPv4;Weather\032Station;_hap._tcp;local;Netatmo.local;169.254.139.115;5001;"sh=7jn59A==" "ci=2" "na_tkn=c082117652ea" "id=8C:3F:21:DD:87:8C" "md=Weather Station" "pv=1.1" "sf=0" "ff=1" "s#=1" "c#=11"


+;br-lan;IPv4;Giulio\226\128\153s\032iPhone;_rdlink._tcp;local
+;br-lan;IPv4;70\058b3\05806\0581d\0585e\058d9\064fe80\058\05872b3\0586ff\058fe1d\0585ed9-supportsRP;_apple-mobdev2._tcp;local
=;br-lan;IPv4;Giulio\226\128\153s\032iPhone;_rdlink._tcp;local;Giulios-iPhone.local;192.168.1.109;50712;"rpAD=a498ad3f4f0e" "rpVr=430.3" "rpBA=08:8E:C5:42:74:7B"
=;br-lan;IPv4;70\058b3\05806\0581d\0585e\058d9\064fe80\058\05872b3\0586ff\058fe1d\0585ed9-supportsRP;_apple-mobdev2._tcp;local;Giulios-iPhone.local;192.168.1.109;32498;
+;br-lan;IPv4;iPadPro;_rdlink._tcp;local
=;br-lan;IPv4;iPadPro;_rdlink._tcp;local;iPadPro.local;192.168.1.106;49154;"rpAD=7acbfc19fe6c" "rpVr=420.5" "rpBA=0A:D4:AA:49:DD:10"
^CGot SIGINT, quitting.
root@WAX206:/etc/avahi# 

With 169.254.129.198;5001 and 169.254.139.115;5001 as IPs, they will never e discovered by Avahi when they are separated... but why they don’t have correct IPs when in lan? From the WAX206 I see the correct IP.

I have no idea of what to do now how to fix this issue? edit the avahi config file somewhere in these points? I'm just thinking on what to do now, if you have any suggestion...

publish-dns-servers= Takes a comma seperated list of IP addresses for unicast DNS servers. You can use this to announce unicast DNS servers via mDNS. When used in conjunction with avahi-dnsconfd on the client side this allows DHCP-like configuration of unicast DNS servers.
publish-resolv-conf-dns-servers= Takes a boolean value ("yes" or "no"). If set to "yes" avahi-daemon will publish the unicast DNS servers specified in /etc/resolv.conf in addition to those specified with publish-dns-servers. Send avahi-daemon a SIGHUP to have it reload this file. Defaults to "no".

[spence]

@giuliomagnifico

With 169.254.129.198;5001 and 169.254.139.115;5001 as IPs, they will never e discovered by Avahi when they are separated... but why they don’t have correct IPs when in lan? From the WAX206 I see the correct IP.

169.254.x.x are self assigned local_only addresses.

Link-local address

Commonly used when a host has no valid ip address config or DHCP.
If you changed the network connectivity without causing a physical link loss on the hosts, this could be expected. Simple devices sometimes don't have code to deal with that situation, or perhaps any network change without a reboot.

Try causing a wifi re-association for the Netatmo devices if that is easy or just reboot them.
.
.
.

@giuliomagnifico

When the Netatmo devices are isolated from the lan, I absolutely can't see any query when I listen from the WAX206 with avahi-browse -arp.

This agrees with the packet captures.
.
.
.

@giuliomagnifico

Yes, I will use Pastebin now, because there's a words limit here on the forum, here is the output:

I looked at the traces.
I think we are close!
The capture on the Netatmo side (.6.0/24) shows proxied mDNS from the LAN (.1.0/24).
There is zero mDNS traffic from the Netatmo side showing on the LAN side.
Avahi seems to be working in one direction: LAN to Netatmo.

Once the Netatmo devices are back in the Netatmo network and you verify that they have correct IP addresses and mDNS traffic is seen by tcpdump -ennv -i wl0-ap1 port 5353 on WAX206, please do a capture on the WAX206 with this:
tcpdump -ennv -i br-lan
and verify that the IP address on the WAX206, 192.168.1.3, shows up as a source in some packets.
If your ssh session is from the LAN network then there should be plenty of traffic during a short capture.
I don't need you to share those captures with me.

If you can capture some packets "from" the interface on WAX206 directly connected to the LAN then it looks like Avahi is only working in one direction.
I noticed there was no proxied mDNS traffic from 'iot' network on the br-lan capture.
Are there any devices doing mDNS on the 'iot' network? If there are, then maybe avahi is currently only forwarding from 'Netatmo'.

I don't have any suggestions but please work on changing the Avahi config / firewall rules so Avahi forwards from 'Netatmo' to 'LAN' as well.
Check with tcpdump -ennv -i br-lan port 5353 should show nDNS traffic from source 192.168.1.3.

Good luck!
I will be away for 2 - 3 hours but I look forward to some good news from you then!

[spence]

A quick addition before I run out the door...

Look into advanced options of netstat and ps commands to see the network binding to the avahi process(es).

netstat -nluepw or other variants.
ps -w

There are other tools that may need to be added by opkg but I hope the above tools reveal the process and network bindings. They may help correlate bindings to the Avahi config changes for listening/forwarding to/from interfaces.

[giuliomagnifico]

@spence

169.254.x.x are self assigned local_only addresses.

https://en.wikipedia.org/wiki/Link-local_address

Commonly used when a host has no valid ip address config or DHCP.
If you changed the network connectivity without causing a physical link loss on the hosts, this could be expected. Simple devices sometimes don't have code to deal with that situation, or perhaps any network change without a reboot.

Try causing a wifi re-association for the Netatmo devices if that is easy or just reboot them.

Yes this is what I was saying, if they don't have an IP, how they can talk? I'll try the reboot when they are inside the lan and I will check if they get a valid

[quote="spence, post:19, topic:149133"]
please do a capture on the WAX206 with this:
tcpdump -ennv -i br-lan
and verify that the IP address on the WAX206, 192.168.1.3 , shows up as a source in some packets[/quote]

Yes but it shows up only in the (current) SSH session with the Mac I'm using, nothing else...

17:48:40.904910 80:cc:9c:eb:8d:37 > f8:e4:3b:a3:5f:54, ethertype IPv4 (0x0800), length 1326: (tos 0x4a,ECT(0), ttl 64, id 25025, offset 0, flags [DF], proto TCP (6), length 1312)
    192.168.1.3.22 > 192.168.1.10.50721: Flags [P.], cksum 0x8870 (incorrect -> 0x4fbc), seq 20940704:20941964, ack 13177, win 1002, options [nop,nop,TS val 3344280471 ecr 547442437], length 1260

@spence

If you can capture some packets "from" the interface on WAX206 directly connected to the LAN then it looks like Avahi is only working in one direction.
I noticed there was no proxied mDNS traffic from 'iot' network on the br-lan capture.
Are there any devices doing mDNS on the 'iot' network? If there are, then maybe avahi is currently only forwarding from 'Netatmo'

It's what I'm trying to say, Avahi is not working as expected, and I think the host inside 'iot' interface are working only because there's the Avahi istance running on Homebridge, because all the queries are sent to them but from them

@spence

I don't have any suggestions but please work on changing the Avahi config / firewall rules so Avahi forwards from 'Netatmo' to 'LAN' as well.
Check with tcpdump -ennv -i br-lan port 5353 should show nDNS traffic from source 192.168.1.3.

I see only traffic from other sources like smart plugs/TV/etcc..and the Homebridge server where Avahi is running and is running fine:

Homebridge server

192.168.1.5.5353 > 224.0.0.251.5353: 0 PTR (QM)? _hap._tcp.local. (33)
18:10:39.713126 e4:5f:01:b3:3a:8d > 01:00:5e:00:00:fb, ethertype IPv4 (0x0800), length 1248: (tos 0x0, ttl 255, id 38789, offset 0, flags [DF], proto UDP (17), length 1234)

Random iot device

192.168.1.109.5353 > 224.0.0.251.5353: 0*- [0q] 3/0/2 Giulios-iPhone.local. (Cache flush) AAAA fe80::109a:cc7a:4eaf:87d5, Giulios-iPhone.local. (Cache flush) A 192.168.1.109, Giulios-iPhone.local. (Cache flush) AAAA fd91:e81d:cbd3:4623:1076:22e4:f83b:8ceb (153)

About the rule... hmmm but all the traffic can already go from 'lan' to 'netatmo', what rule I can try to add other than this that is already in firewall?

config rule
	option name 'LAN to Netatmo'
	option src 'lan'
	option target 'ACCEPT'
	option dest 'Netatmo'
	list proto 'all'

I also have to go out, I'll think on it!

Thanks!

EDIt:

@spence

ps -w

Uhm, Avahi is running as I've always noticed but I noticed only now that assigned to the useless 'nobody' user! This could be the issue (?) no permissions?

19252 nobody    2684 S    avahi-daemon: running [WAX206.local]

[spence]

@giuliomagnifico

I see only traffic from other sources like smart plugs/TV/etcc..and the Homebridge server where Avahi is running and is running fine:

Homebridge server

192.168.1.5.5353 > 224.0.0.251.5353: 0 PTR (QM)? _hap._tcp.local. (33)
18:10:39.713126 e4:5f:01:b3:3a:8d > 01:00:5e:00:00:fb, ethertype IPv4 (0x0800), length 1248: (tos 0x0, ttl 255, id 38789, offset 0, flags [DF], proto UDP (17), length 1234)

Random iot device

192.168.1.109.5353 > 224.0.0.251.5353: 0*- [0q] 3/0/2 Giulios-iPhone.local. (Cache flush) AAAA fe80::109a:cc7a:4eaf:87d5, Giulios-iPhone.local. (Cache flush) A 192.168.1.109, Giulios-iPhone.local. (Cache flush) AAAA fd91:e81d:cbd3:4623:1076:22e4:f83b:8ceb (153)

These are all on the 'lan' network (192.168.1.0/24) and not the 'iot' network (192.168.5.0/24). We know Avahi proxies them to network 'Netatmo'. I am interested to know what mDNS hosts are in network 'iot' that we should see reflected to the other networks.
.
.
.
What I was suggesting is that after adjusting the config so avahi might work better / differently then look for reflected mDNS traffic exiting interfaces on the WAX206. You may have another tool to make that easier. Didn't you show me something like avahi-browse -arp ?
.
.
.

@giuliomagnifico

Uhm, Avahi is running as I've always noticed but I noticed only now that assigned to the useless 'nobody' user! This could be the issue (?) no permissions?

Since Avahi IS working reflecting mDNS from 'Netatmo' to 'lan' I would not think permissions are an issue. Why would "nobody" be allowed read to access of 'Netatmo' and write access to 'lan' but not read access from 'lan' or read/write access to 'iot'.

Are there mDNS / Homekit devices active on 'iot'?

For netstat, try a broader, more inclusive set of options but grep for '5353':
netstat -np |grep 5353
If nothing shows up then run netstat -anp and look for anything interesting. There might be something tied to ubusd or netflow etc. I don't know how avahi runs. Maybe there is revealing info in /etc/rc.d or /etc/init.d.
.
.
.
As for Avahi config, I didn't find much info on openwrt.org/docs/ but from https://linux.die.net/man/5/avahi-daemon.conf I found allow-interfaces=.

A recent posting of your Avahi config does not have a line for
allow-interfaces=
It may be good to add that back to section [server] and leave the list empty.
Try it and look for proxied mDNS, either with 'avahi-browse -arp` or tcpdump.

Have you been watching logread and dmesg for any related messages, especially avahi startup and dhcp events. If you use dnsmasq for dhcp, I think you can turn on event logging for things like unusual client requests in the config file and reload the config.

[spence]

@giuliomagnifico

After investigating I'm starting think that Avahi is not working as expected with these Netatmo devices, I try to explain:

When the Netatmo devices are isolated from the lan, I absolutely can't see any query when I listen from the WAX206 with avahi-browse -arp.

As soon as I bond the Netatmo wireless interface to the br-lan interface, they appear!

I am thinking that it is more likely that avahi is only reflecting from network 'lan' to network 'Netatmo` and not from 'Netatmo' or 'iot' to 'lan'.

To test this, identify a system that is known to be sending mDNS traffic and seen by WAX206 and move it to network 'Netatmo'. Better yet, I saw your iPhone mDNS traffic reflected from 'lan' to 'Netatmo'.

09:24:33.909046 02:0c:43:26:60:30 > 01:00:5e:00:00:fb, ethertype IPv4 (0x0800), length 304: (tos 0x0, ttl 255, id 45888, offset 0, flags [DF], proto UDP (17), length 290)
    192.168.6.1.5353 > 224.0.0.251.5353: 0 [3q] [5n] ANY (QM)? GiulioM-bM-^@M-^Ys iPhone._rdlink._tcp.local. ANY (QM)? Giulios-iPhone.local. ANY (QM)? 70:b3:06:1d:5e:d9@fe80::72b3:6ff:fe1d:5ed9-supportsRP._apple-mobdev2._tcp.local. (262)

09:24:34.160194 02:0c:43:26:60:30 > 01:00:5e:00:00:fb, ethertype IPv4 (0x0800), length 304: (tos 0x0, ttl 255, id 45899, offset 0, flags [DF], proto UDP (17), length 290)
  192.168.6.1.5353 > 224.0.0.251.5353: 
    0 [3q] [5n]
     ANY (QM)? GiulioM-bM-^@M-^Ys iPhone._rdlink._tcp.local.
     ANY (QM)? Giulios-iPhone.local.
     ANY (QM)? 70:b3:06:1d:5e:d9@fe80::72b3:6ff:fe1d:5ed9-supportsRP._apple-mobdev2._tcp.local.
    (262)

(I added some line breaks in the second packet for easy reading)
Did you have the Home app open on your iPhone at around 9:24 this morning?
.
.

Try moving your iPhone to the 'Netatmo' network, verify that it has a good IP address and see if it shows up in avahi-browse and or in tcpdumps.

Also test to see if Avahi is reflecting to 'iot'. I don't think we did that yet. I had that on my todo list from yesterday.

[giuliomagnifico]

Ehy @spence you won't trust me but.... I solved the issue. In a very stupid way, and this is not the first time I have a similar experience with OpenWrt (I also wrote it before, about bounding/assigning an interface to a device, example.

I was simply searching a way to allow the traffic rom netatmo to lan as you wrote:

I don't have any suggestions but please work on changing the Avahi config / firewall rules so Avahi forwards from 'Netatmo' to 'LAN' as well.

And I changed this, just for curiosity (since the Netatmo zone is already assigned to the wl0-ap0 interface):

That is the same as add list device

config zone
	option name 'Netatmo'
	option output 'ACCEPT'
	list network 'Netatmo'
	option forward 'ACCEPT'
	option input 'REJECT'
	list device 'wl0-ap1'

My previous setting was

config zone
	option name 'Netatmo'
	option output 'ACCEPT'
	list network 'Netatmo'
	option input 'REJECT'
	option forward 'REJECT'

This fixed all.

I have no idea of what is the difference in doing this, I mean assign a zone to an interface or specify the interface from a zone, but that is all.

Now here in Italy are the 11PM so I'm to tired to think to something else rather than sleep

Tomorrow I'll reply to your post, I'll turn off the netatmo interface and I'll put the Netatmo devices inside the iot WLAN, do some tcpdump, I'll investigate better, etc...

That's absurd... I spent one week just to add a stupid option to assign a stupid

list device 'wl0-ap1'

to a firewall zone that is already assigned to the 'wl0-ap0` interface.

That emoticon explain my mood!

I don't know if it's a weird behviour with LuCi or what else, but is way faster and simpler create a new firewall zone when you create the new interface with LuCi.... but if this don't/will not work, then is pointless, you still have to go inside the firewall and specify to which device this zone must be assigned.

Anyway I learned lots of new things/stuff doing this debug, thanks to you! So I'm happy to fixed it in this "stupid way"

[spence]

Yay!!!! Congratulations!!

[spence]

@giuliomagnifico

Ehy @spence you won't trust me but.... I solved the issue. In a very stupid way, and this is not the first time I have a similar experience with OpenWrt (I also wrote it before, about bounding/assigning an interface to a device, example.

I was simply searching a way to allow the traffic rom netatmo to lan as you wrote:

I don't have any suggestions but please work on changing the Avahi config / firewall rules so Avahi forwards from 'Netatmo' to 'LAN' as well.

And I changed this, just for curiosity (since the Netatmo zone is already assigned to the wl0-ap0 interface):

I believe it! I am not surprised that it is something like that. I was working with you to find the point where it breaks so you could focus on finding a solution for a specific problem.

Your solution looks unusual to me. I never used swconfig so I only had to learn the DSA way and other "new" config styles and did not need to unlearn old openwrt ways. It was interesting learning openwrt after 26 years of cisco and other vendors before that.
.
.
.
I'm sure it is already your plan but after testing in network 'iot', you can set mDNS reflection to be limited to 'lan' and 'iot' so 'guest' can not see nor serve records, unless you want that to work for screen casting to your tv or something.

Assuming that the rest of your setup for iot isolation goes smoothly, feel free to bring up anything you mentioned earlier that I did not address to your satisfaction or new things as well.

[giuliomagnifico]

@spence

I believe it! I am not surprised that it is something like that. I was working with you to find the point where it breaks so you could focus on finding a solution for a specific problem.

Yes thanks again!

@spence

Your solution looks unusual to me.

...don't tell it to me... this solution looks absurd, because the Netatmo interface was already assigne to the Netamo zone of the firewall. So why add a device to a firewall zone if an interface is assigned to the same zone?

@spence

I never used swconfig so I only had to learn the DSA way and other "new" config styles and did not need to unlearn old openwrt ways. It was interesting learning openwrt after 26 years of cisco and other vendors before that.

But this is a DSA router/AP, not swconfig, with my old swconfig things were "more logical", is the second time that this DSA config is incomprensibile to me. But that's the (new) way! And I like learn new things

@spence

I'm sure it is already your plan but after testing in network 'iot', you can set mDNS reflection to be limited to 'lan' and 'iot' so 'guest' can not see nor serve records, unless you want that to work for screen casting to your tv or something.

Yes thanks, already done, after assignign the same wireless interface to the 'iot' firewall zone, I've put the Netatmo device inside the 'iot' zone, now they're working as expected.

@spence

Assuming that the rest of your setup for iot isolation goes smoothly, feel free to bring up anything you mentioned earlier that I did not address to your satisfaction or new things as well

Oh thanks, at the moment everything is perfect, the only next step I'm planning is to add the TV to a VLAN, because it's wired connected to the hub, but I have to chose if I have to use the Netgear hub to make a untagged VLAN or a tagged one in the R4S I don't like both ways, the interface in the Netgear witch is ugly and I don't like the tagged VLANs but since I use the TV for airplay content sharing, maybe is better leave it inside the lan with other devices to avoid further messes

I'll write a post on my blog for this issue, maybe could be useful for someone else, since it was a very weird and not logical solution!

Edit: another weird behavior

I'm cleaning up the Netatmo interface, zone, etcc... but I discovered that if I don't specify also the subnet IP inside the firewall (not only the attached device), the setup isn't working.

config zone
	option name 'iot'
	option output 'ACCEPT'
	option input 'REJECT'
	list device 'wl0-ap0'
	list network 'iot'
	option forward 'REJECT'
	list subnet '192.168.5.0/24'

With the Netatmo zone it was working without specifying anything... becuse the wl0-ap0 device is already assigned to the iot zone that has 192.168.5.0/24 has IP. I have no idea...

[Finn]

Hi Giulio.
just picked up this thread from our wax206 thread and haven't gone through the entire post - yet.
FYI
There shall be no firewall settings/setup on the AP (and no dhcp or dns) - these things You can disable in system->startup - those services shall be handled by the router only

[giuliomagnifico]

Hi Finn, thanks for the reply, I solved the issues yesterday, you can read only the last post: Netatmo HomeKit native devices, not able to update iOS Home app when isolated (but work online/using Netamo app) - #24 by giuliomagnifico

Anyway yes, use the firewall on the router only is another way but I have to create lots of tagged VLANs and the router should handle more work. Instead using the firewall access point I demand some work to it and I can have another layer of security.

Now my network is made in this way, with 3 isolated subnets and no VLANs

[Finn]

I think I recall You as having a Nanopi R4S as Your router - it's piece of cake to run many vlans for that device.
I am running 3 vlans (found out how to set it up from reading Your posts amongst others) - lan, IOT and game and yet another vlan will be added soon.

My setup: WAN -> NanopiR4S (OpenWrt) -> Netgear 308Tgs (Openwrt) - Netgear wax206 AP (Openwrt)

Most security is in the firewall (router) and in here I control what which vlan can regarding access to the internet - Lan full access to wan and IOT, IOT only outgoing access to wan no access to Lan same with Game (no one can access IOT or game from WAN) but that requires me to allow dhcp and dns requests (in traffic rules) so IOT and Game can receive information when needed and maybe this may have caused Your issues not getting any ip's ? just a guess

Ciao

[giuliomagnifico]

Yes I also have the small great R4S (that also @spence has IIRC). And thanks. We're on the same exact hardware, except that I have another Netgear switch (very similar, the GS108E)

Your config is perfect, whatever to use VLAN or subnet is a personal choice. Both have advantages and disadvantages! My network is similar to your, only using subnets, by the way my issue wasn’t the IPs/DHCP, was something related to specific explain every subnet and device attached to the a firewall zone. Also if it’s already selected an interface that is assigned to the same subnet/device. And I had to specify a new route and gateway for the subnets to the R4S to route the mDNS packets for the Homebridge server… just a mess for Apple stuff

Anyway I’m writing a new post blog on my actual home setup, because I made some hardware changes, like two big fans inside the """server cabinet* """ I will write also the details about this issue and the configuration of the WAX206.

The R4S is rugged/dust-proof

[spence]

@giuliomagnifico

this solution looks absurd, because the Netatmo interface was already assigne to the Netamo zone of the firewall. So why add a device to a firewall zone if an interface is assigned to the same zone?

Did you reboot your WAX206 after adding or making network or firewall changes for iot or Netatmo? When I was working on adding several VLANs / L3 networks to my setup, I found that the added setup didn't work after a commit. Even reloading the firewall config and reloading the network config didn't help. Restarting network service didn't help either. Rebooting the device got the new config working each time. I repeated this a couple of times adding new networks but did not try to analyze the root issue.

@giuliomagnifico

Edit: another weird behavior

I'm cleaning up the Netatmo interface, zone, etcc... but I discovered that if I don't specify also the subnet IP inside the firewall (not only the attached device), the setup isn't working.

config zone
	option name 'iot'
	option output 'ACCEPT'
	option input 'REJECT'
	list device 'wl0-ap0'
	list network 'iot'
	option forward 'REJECT'
	list subnet '192.168.5.0/24'

With the Netatmo zone it was working without specifying anything... becuse the wl0-ap0 device is already assigned to the iot zone that has 192.168.5.0/24 has IP. I have no idea...

.
.
I'll share the basics of my working setup in case it sparks an idea for a change to your setup. I am not trying to get you to use VLANs. Just sharing my working config that does not add list device or list subnet to my zone stanzas.
I use VLANs in my config. My wired interfaces are defined in a bridge and my 5 L3 'lan' side networks/interfaces all follow this strategy:

In my /etc/config/network file, each config bridge-vlan stanza defines a 'vlan' which is also a virtual device:

config bridge-vlan
	option device 'sw_switch'
	option vlan '51'
	list ports 'eth1:t'

In my /etc/config/network file, each config interface stanza defines a "L3 network" and maps it to a virtual device for the vlan.

config interface 'vl51_guest'
	option proto 'static'
	option device 'sw_switch.51'
	list ipaddr '192.168.51.1/24'

To add a specific wifi SSID to be part of the 'network',( and zone by way of the config interface stanza), the option network line in the config wifi-iface stanza in the /etc/config/wireless file defines that binding:

config wifi-iface 'wifinet10'
	option device 'radio0'
	option mode 'ap'
	option encryption <redacted>
	option isolate '1'
	option key <redacted>
	option ieee80211w <redacted>
	option wpa_disable_eapol_key_retries '1'
	option network 'vl51_guest'
	option ssid 'Spence_guest'
	option ifname 'wl5g-vl51-gu'

I currently only have one network per zone but I did successfully test with multiple "untrusted" networks in a single zone.
In my /etc/config/firewall file, The 'list network' line is the reference to the config interface label in the network file to bind everything in that network to the zone.

config zone
	option output 'ACCEPT'
	option input 'REJECT'
	option forward 'ACCEPT'
	option name 'guest'
	list network 'vl51_guest'

config forwarding
	option src 'guest'
	option dest 'wan'

config rule
	option name 'Allow-Guest-input-DHCP'
	option src 'guest'
	option dest_port '67'
	option target 'ACCEPT'

config rule
	option name 'Allow-Guest-input-ICMP'
	list proto 'icmp'
	option src 'guest'
	option target 'ACCEPT'
	list dest_ip '192.168.51.1'

config rule
	list proto 'udp'
	option src 'guest'
	list dest_ip '192.168.51.1'
	option target 'ACCEPT'
	option name 'Allow-Guest-input-DNS'
	option dest_port '53'

.
.
.
If I remember to, I will try adding a wifi SSID to a separate IP interface/network in its own zone without involving any wired ports or dot1q VLANs and see if I run into a similar issue to yours but today I am going outside for a long walk to enjoy this warm winter afternoon.

Happy OpenWrt-ing!

[giuliomagnifico]

@spence

Did you reboot your WAX206 after adding or making network or firewall changes for iot or Netatmo? When I was working on adding several VLANs / L3 networks to my setup, I found that the added setup didn't work after a commit. Even reloading the firewall config and reloading the network config didn't help. Restarting network service didn't help either. Rebooting the device got the new config working each time. I repeated this a couple of times adding new networks but did not try to analyze the root issue

Yes I noticed this too, only a bit late maybe, less frequent for the firewall (generally a restart of it was working), but very often for the network. And that's weird because I also was restarting every time after a change the network. Maybe there's something like a "kernel cache of the routing" for the DSA devices?

@spence

I'll share the basics of my working setup in case it sparks an idea for a change to your setup. I am not trying to get you to use VLANs. Just sharing my working config that does not add list device or list subnet to my zone stanzas.
I use VLANs in my config. My wired interfaces are defined in a bridge and my 5 L3 'lan' side networks/interfaces all follow this strategy

My old setup, with the R7800 swconfig, was working without all these 'list this/that', now I have also to add the subnet of an interface that HAS the same subnet. I don't know, had I known earlier I would have started at the beginning with the VLANs, now I'm planning to rebuild all again with a VLAN from the router to the AP. Anyway I can also add now a VLAN, keep the subnets but remove the firewall zones/rules on the AP, and create the rules for the VLAN on the R4S/router.

Thanks for posting your setup, since I can use it as inspiration for this (why are you forwarding the ICMP on the Guest? )

@spence

If I remember to, I will try adding a wifi SSID to a separate IP interface/network in its own zone without involving any wired ports or dot1q VLANs and see if I run into a similar issue to yours but today I am going outside for a long walk to enjoy this warm winter afternoon.

Yes you can try, have fun and in case let me know what happens.

@spence

Happy OpenWrt-ing!

You too and thanks again for all the help again!