Apple Home/HomeKit on a isolated network results in very slow IoT devices

Installing and Using OpenWrt
1

Hi,
it's the n time that I try to figure out why if I isolate from my main network the IoT devices, then they are superslow to respond (like 3-4 secs to turn on/off a light), see gif below

Screen Recording 2023-01-19 at 08.19.41

I configured the 2.4GHz only to the IoT devices, than I created a zone for the interface like as the Guest zone, I also tried to use some firewall rules/ports (as I've read online) but they are still slow. Alexa devices or "direct devices" with their app, are working fine, only when I use the same devices via HomeKit there's this issue.
I'm using Homebridge for Apple HomeKit that is a Raspberry PI on my main LAN at port 51625.

config zone
        option name 'iot'
        option output 'ACCEPT'
        option forward 'REJECT'
        list network 'iot'
        option input 'REJECT'

config forwarding
        option src 'iot'
        option dest 'lan'

config rule
        option name 'IoT DNS'
        option src 'iot'
        option dest_port '53'
        option target 'ACCEPT'

config rule
        option src 'iot'
        option target 'ACCEPT'
        option name 'IoT 80'
        option dest_port '80'

config rule
        option name 'IoT 443'
        option src 'iot'
        option dest_port '443'
        option target 'ACCEPT'

config rule
        option name 'IoT 5353'
        option src 'iot'
        option dest_port '5353'
        option target 'ACCEPT'

config rule
        option name 'Block IoT LAN'
        option src 'iot'
        option dest 'lan'
        option target 'DROP'
        list proto 'all'
        list dest_ip '192.168.1.0/24'

Anyone has an idea of what can be the issue here?

Thanks!

2

I can't speak specifically for Homekit, but Apple (and other providers) get discovery-crazy on most networks. I had a similar issue user a different vendors system.

Some services are constantly scanning for neighbors and advertising their existence (Bonjour as an example). You can take a look on that network to see if you see broadcast or multicast traffic firing at high rates.

.And if the devices can't find/phone home or two each other (having received a unicast or broadcast notice of neighbors), behavior can be slow, non-functioning, unpredictable. In my case, the discovery traffic frequency increased rapidly with no communication (as a result of client isolation) since the designers wanted to onboard devices as soon as possible. So it winds up being a small flood.

If you look in the router logs, you may also see lots of that getting rejected by firewall rules since you (correctly, are isolating and only allow specific dns and thru to internet.

In my case I added specific firewall rules to allow select traffic, but in the end I had to remove client isolation to get everything to play. So I wound up putting all those devices in their own wireless network with isolation disabled, but isolated from my other stuff... and only added a few pinhole firewall rules if I needed to reach them from outside their bubble.

1 Like
3

Thank for the reply, yes I know for the Isolation in the wireless interface, indeed I’m not using it, and as you saying, Apple mDNS usually spams lo of messanges, but as you noted, the ports of this protocols are open. This is why I’m not understanding why this weird behavior.

Anyway now maybe I have an idea: the HomePod (hub) is on another Wi-Fi radio, with the same SSID but on the 5GHz and not assigned to the IoT firewall zone!

This could be the issue, when I’ll be at home I’ll try again to put the HomePod/hub in the same Wi-Fi and with the same subnet. I hope to fix it.

4

Hi,

firewall rules for mDNS are not enough. You also need an mDNS Proxy for it to work.

I use avahi for my setup and it is working fine. The reason, why it takes several seconds for you is probably because you are communicating via iCloud like you are in an external network.

Hope that helps

2 Likes
5

Thanks for the reply, I had thought that it might be something with Avahi as mDNS, but since I'm using Homebridge, it already has the Avahi daemon running on it

Screenshot 2023-01-19 at 15.03.29
Screenshot 2023-01-19 at 15.03.291136×648 15.9 KB

...this is why I have never installed it on my AP/router. Maybe I have to configure something else on the firewall? The tutorial you linked is outdated, umdns is not available in opkg. Did you have installed Avahi with some custom setting? Or just installed it via opkg?

My setup at the moment is:

Router 192.168.1.2
AP 192.168.1.3
Homebridge 192.168.1.5 (with Avahi daemon)
iot network 192.168.4.x

the firewall rules for the iot zone:

config zone
	option name 'iot'
	option output 'ACCEPT'
	option forward 'REJECT'
	option input 'REJECT'
	list network 'iot'

config forwarding
	option src 'iot'
	option dest 'lan'

config rule
	option name 'IoT DHCP'
	list proto 'udp'
	option src 'iot'
	option dest_port '67-68'
	option target 'ACCEPT'
	option enabled '0'

config rule
	option name 'IoT DNS'
	option src 'iot'
	option dest_port '53'
	option target 'ACCEPT'

config rule
	option target 'ACCEPT'
	option name 'IoT 80'
	option dest_port '80'
	option src 'iot'

config rule
	option name 'IoT 443'
	option dest_port '443'
	option target 'ACCEPT'
	option src 'iot'

config rule
	option name 'IoT 5353'
	option src 'iot'
	option dest_port '5353'
	option target 'ACCEPT'
	option src_port '5353'

config rule
	option name 'Block IoT LAN'
	option src 'iot'
	option dest 'lan'
	option target 'DROP'
	list proto 'all'
	list dest_ip '192.168.1.0/24'

Edit: I installed the avahi daemon following this tutorial https://blog.christophersmart.com/2020/03/30/resolving-mdns-across-vlans-with-avahi-on-openwrt/

And I edited the avahi-daemon.conf using my domain (LAN), my DNS server (192.168.1.4) and my interfaces (br-lan,wl0-ap0)

[server]
#host-name=foo
domain-name=lan
use-ipv4=yes
use-ipv6=yes
check-response-ttl=no
use-iff-running=no
allow-interfaces=br-lan,wl0-ap0

[publish]
publish-addresses=yes
publish-hinfo=yes
publish-workstation=no
publish-domain=yes
publish-dns-servers=192.168.1.4
#publish-resolv-conf-dns-servers=yes

[reflector]
enable-reflector=yes
reflect-ipv=no

but something is still not working...

6

I had to reboot my router to have the settings working. I used the same tutorial.

7

thanks, done but the daemon is running...

root@WAX206:~# /etc/init.d/avahi-daemon restart
root@WAX206:~# /etc/init.d/avahi-daemon status
running

I also rebooted it but is the same, I also restarted Homebridge. I can't figure out why Homekit is routing everything through the cloud, because if I use Alexa or the Smart Plug app (Meross), all is super-fast as normal!
And the funny thing is that: if I turn on a light using the Meross app, then the Home app update the status of the same light fast as usual, but if I use the Home app to turn on/off the same light...it's super-slow.

I don't know but to me looks something related to the subnet of the Homebridge server

8

I wonder if some packet captures would reveal some useful info. Compare captures with the setup in a "fast" config to captures with the setup in a "slow" config. Maybe collect the captures right on the Homebridge/Home Kit server (rpi).

1 Like
9

Yes but maybe I understood where is the issue:

Is the HomePod.

Because the HomePod acts like a hub, and it should be in the same subnet as the other iot devices (192.168.4.x), BUT since the HomePod hasn't the option to change the wifi, instead it gets the SSID from the iPhone, and my iPhone is not connected to the iot network (obviously), then the HomePod or is not connected to anything if I force it to use the iot wifi (by MAC address exclusion of the main wifi), or is in another subnet (192.168.1.x) of the iot devices. That is the same as my iPhone and other devices but not the iot devices, this is why it's routing everything via Cloud and is so slow.

I tried to connect my phone to the iot wifi/network, but since it gets no internet connection because this interface is isolated, then the HomePod doesn't connect to the same network, because it doesn't have internet access.

What a mess!

Maybe I have first to connect to first disable the "internet block/80" on the iot network, then connect with my iPhone to it, than wait/force for the HomePod to connect to the same network, than re-enable the firewall rules to isolate the clients and then hope that the HomePod will stay connected to this network also if it can't have internet access..

well... what a mess, damn Apple. In the end I will left the iot network not isolated.

10

Here's an overview of my setup that works almost perfectly with homebridge + HomeKit and isolated IOT VLANs.

OpenWrt:
Main VLAN: hosts homebridge server and phones/laptops, HomePod, etc
IOT VLAN: hosts all IOT lightbulbs, outlets, switches, etc; NO WAN access at all.

Firewall drops everything between IOT and main VLAN, with the exception of bidirectional flow from homebridge to IOT vlan and IOT vlan to homebridge IP on main lan

Yes, if someone did compromise an IOT device and then compromised my homebridge server I'd be screwed, but its better than nothing. Everything routes locally, bulbs are almost instantaneous.

Lastly, be sure all your plugins on HomeKit are running as child bridges; this sped things up tremendously.

1 Like
11

Uhm I need more informations, it’s the same setup as me but for me it’s slow.

Can you write your network/subnet and firewall config?

12

I checked my firewall rules for mDNS again. These look as follows for me:

config rule                                           
        option family 'ipv4'                          
        list proto 'udp'                              
        option src 'iot'                           
        option src_port '5353'                        
        list dest_ip '224.0.0.251'                    
        option dest_port '5353'                       
        option target 'ACCEPT'                                     
        option name 'Allow-iot-mDNS'

Is the rule really correct or are src and dest swapped here?

config forwarding
	option src 'iot'
	option dest 'lan'
13

It’s correct, and the rule is the same on my firewall but is not the firewall the issue, because also if I change the zone to the iot interface, doesn’t work. I think is something with the Homebridge/HomePod hub and iot devices on a different subnet. What’s/are your?

14

I've captured the packets with both the wireless interfaces assigned to the "iot network" (when HomeKit isn't working as expected) and the wireless interface assigned to the "lan network" (when it's working).

But I'm unable to find specifically the packets I'm searching in Wireshack. If you have any suggestion...

15

This is a series of mDNS packets when I'm using the separate interfaces and there's 2-3 seconds of lag, the only weird stuff is the ipv6, because I have off on all the interfaces, I don't know why there are some ipv6, and I'm not seeing the the packet I'm searching, because I'm tuning on a light/bulb (192.168.4.120) from MacBook air (192.168.1.10).

Screenshot 2023-01-20 at 15.46.45
Screenshot 2023-01-20 at 15.46.452420×300 202 KB

16

With the complexities of your network setup, routed network, Avahi on a different device from the router, Home Bridge, separation of SSID/vlan between 2.4Ghz and 5 Ghz, you need to become an expert in all the protocols and any application behavior changes that may happen when a change in communication occurs.

Start out simple and learn how a controller learns of a device and talks to it when on the same L2 network. What communication changes when it goes through iCloud.

Are your devices directly using homekit accessory protocol (HAP) or a different home automation protocol? Is your home automation bridge translating protocols?

If communication is direct with your macbook on the 192.168.4.x network, analyze the network traffic by using HomeKit on your macbook and capturing packets on the macbook with all browsers and other apps closed to minimize unrelated traffic. Look at the mDNS traffic a while, turn the light on and off taking note of the exact times. Look at the captures to learn the communication flow.

You may be better served in getting help with Homekit elsewhere like:
https://www.reddit.com/r/HomeKit

This 4 year old thread may be useful:
https://www.reddit.com/r/HomeKit/comments/a3iwwj/troubleshooting_homekit_what_tools_to_use_and_how/

Another older thread that may help:
https://medium.com/@gepeto42/using-homekit-devices-across-vlans-and-subnets-aa5ae1024939

This one talks about firewall setting, mentioning 3 ports/protocols:

Once you understand the communication flow and see it, make network changes and look for the communications in the new path, analyzing where things change or go missing.

Some of the threads I linked refer to additional tools like mdns query tools. Try manually opening the tcp port(s) to test reach-ability across a router, firewall or bridge device.

Good luck!

1 Like
17

Thanks for the reply!

Are your devices directly using homekit accessory protocol (HAP) or a different home automation protocol? Is your home automation bridge translating protocols?

If communication is direct with your macbook on the 192.168.4.x network, analyze the network traffic by using HomeKit on your macbook and capturing packets on the macbook with all browsers and other apps closed to minimize unrelated traffic. Look at the mDNS traffic a while, turn the light on and off taking note of the exact times. Look at the captures to learn the communication flow.

IoT devices are using HAP protocols and Homebridge Avahi as mDNS advertiser, but my MacBook (192.168.1.x) is on a different subnet than the iot devices (192.168.4.x). I will try to anaylze better the network but I simply don't find the packet in the mDNS traffic, what I suspect is that when the HomePod is on a different subnet, it's not using mDNS but iCloud. This is why I can't find the packets...

Instead if I use the Meross app (smart plugs) to turn on/off the same light (instead of Home app), everything is perfect, fast as it should be, and (!) the Home app that I left open in background, updates the status of the light super-fast. And I can also find the mDNS packet in Wireshark.

That's very weird, and I want to try to put my HomePod on the "iot" wireless network, but I don't think is possible to force it, and there's no options to do it: the Apple way...

Anyway thanks for the links, using:

dns-sd -Z _homekit._tcp local.

I'm able to see that when the iot devices are on the separate network they can't be seen from the MacBook, I can see only the device in the same subnet. Looks like the mDNS rule and service is not working on the access point (obviously the rule and service are active), I'll try to investigate.

18

I thought you said that Avahi is running on your rpi and not an AP. Either way, you need the mDNS reflector working properly as you know. Looks like you have step 1 in debugging decided. :slight_smile:

Use mDNS query tools on the mDNS reflector/proxy host to test mDNS there.

19

Yes at the beginning I tried to use Avahi only on Homebridge(Raspberry PI), now I installed it also on the AP, but in both ways something is not working.

Basically I just need to forward the queries that make the Homebridge server (192.168.1.5) for avahi (on port 5353) to the "iot" network on subnet 192.168.4.x.

Looks like these rules aren't working

config forwarding
	option src 'iot'
	option dest 'lan'

config rule
	option name 'IoT DHCP'
	list proto 'udp'
	option src 'iot'
	option dest_port '67-68'
	option target 'ACCEPT'

config rule
	option name 'IoT DNS'
	option src 'iot'
	option dest_port '53'
	option target 'ACCEPT'

config rule
	option target 'ACCEPT'
	option name 'IoT 80'
	option dest_port '80'
	option src 'iot'

config rule
	option name 'IoT 443'
	option dest_port '443'
	option target 'ACCEPT'
	option src 'iot'

config rule
	option name 'IoT 5353'
	option dest_port '5353'
	option target 'ACCEPT'
	option src_port '5353'
	list dest_ip '224.0.0.251'
	option src 'iot'
	option dest '*'

config rule
	option name 'IoT 51827'
	option dest_port '51827'
	option target 'ACCEPT'
	option src_port '51827'
	option src 'iot'
	option dest 'lan'

Or I don't know what could be the issue with Avahi on the AP, because it’s running on all interfaces (I see it from the logs) but I don’t think that I have to run 2 instances of avahi. The one on the Homebridge server/RaspberryPI should be enough :slight_smile:

Use mDNS query tools on the mDNS reflector/proxy host to test mDNS there.

If you mean the dns-sd -Z … command, is not available on OpenWrt, it’s a macOS executable (or I have no idea which opkg packet is)

20

I recommend you only have one instance running and work with it. :slight_smile: It will be useful to select a device that you can install and run various tools on like tcpdump and a mDNS query tool.

For clarity, I'll mention a few facts as I understand them at this point to make sure that you are aware:
mDNS uses local only IP multicast.
Local IP multicast is not routable and only is forwardable within the L2 domain (bridged Ethernet).
The mDNS reflector needs to be directly connected to the L2 (Ethernet) networks in which you need the local multicast packets to reach.

mDNS uses UDP port 5353 on multicast group ipv4 address 224.0.0.251 or ipv6 address ff02::fb.

On Ethernet, multicast uses Ethernet Multicast MAC address 01:00:5E:00:00:FB on IPv4 and Mac address 33:33:00:00:00:FB on IPv6.

Correct me if I stated something incorrectly. :slight_smile: It has been well over 10 years since I worked with this stuff.

Tcpdumps on the Avahi host should help to reveal where things are breaking down.
From one of the linked external resources earlier, I think I saw a sample packet capture that showed the MAC and IP address of the Avahi host as the source on both the controller and the controlled device in the mDNS queries and answers.

Are those rules on the Avahi host?

For tools on OpenWrt, I see this on my R4S:

opkg list |grep mdns
...
mdns-utils - IETF104-5 - Bonjour, also known as zero-configuration networking, enables automatic discovery of computers, devices, and services on IP networks. . This package contains mDNS client utilities: - dns-sd - mDNSClient - mDNSIdentify - mDNSNetMonitor - mDNSProxyResponder - mDNSResponder
...

Also opkg list |grep avahi shows an Avahi implemetation as well as a homebridge package.

On your Avahi host, try a tcpdump with a command something like tcpdump -iany host 224.0.0.251 and do an mDNS query from your mac. Hopefully you see it. :slight_smile:

Apple devices like to use IPv6 so unless you disable it on each system, you will likely see it in use as you mentioned earlier today. It might be part of the working solution with a Home controller on the same vlan as the controlled device. Since it looks like both IPv4 and IPv6 are tried in mDNS, hopefully IPv4 will work fine for you.

1 Like