I ran my network this way for a while, and it worked for the most part. Essentially, what I had was an IoT VLAN into which all chromecast, roku and similar devices were placed.
I'll say up front that while the end result allowed clients in LAN to cast to devices in IoT, certain things like chromecast audio groups wouldn't work reliably, and occasionally, the list of available chromecasts would come and go. Ultimately, I decided to go with what "just works", by tossing everything back into the LAN. I may return to this setup at some point because I'm stubborn, and it did teach me quite a bit about multicast and these clever devices.
Allowing clients on my main LAN to speak to the IoT VLAN required a handful of things:
avahi-daemon to allow SSDP between IoT and LAN
LEDE Firewall to allow communication between IoT and LAN and vice-versa
iptables to mangle TTL for multicast as it crossed IoT to LAN
My Firewall zones were set up as follows:
LAN -> WAN, IoT Accept for Input, Output and Forward
IoT -> WAN Accept for Input, Output and Forward
Traffic Rules were configured so that devices in IoT can fetch DNS and DHCP from the router itself:
IoT -> Router: DHCP
IoT -> Router: DNS
IoT -> ANY:5353
IoT -> LAN:1900
IoT -> LAN:5556-5558
IoT -> LAN/224.0.0.0/4
But wait, there's more! Because Chromecast blasts with a TTL of 1, it’ll die when it crosses a subnet. To get around this, I had a custom IPTABLES rule to keep that TTL up:
iptables -A PREROUTING -t mangle -p udp —dport 1900 -j TTL-inc 1
I also have a handful of other iptables rules that replicate those above:
iptables -A INPUT -t filter -i br-lan -p udp —sport 5353 -J ACCEPT
Finally, I configured avahi-daemon:
[server]
host-name=myrouter-gw
domain-name=myrouter.net
use-ipv4=yes
use-ipv6=no
check-response-ttl=no
use-iff-running=no
enable-dbus=yes
allow-interfaces=br-lan, br-IoT
[wide-area]
enable-wide-area=no
[publish]
publish-addresses=yes
publish-hinfo=yes
publish-workstation=no
publish-domain=yes
publish-dns-servers=192.168.0.1
publish-resolv-conf-dns-servers=yes
[reflector]
enable-reflector=yes
reflect-ipv=no
[rlimits]
#rlimit-as=
rlimit-core=0
rlimit-data=4194304
rlimit-fsize=0
rlimit-nofile=30
rlimit-stack=4194304
rlimit-nproc=3
Here are a bunch of posts by people smarter than me that helped me get to this point:
http://www.cron.dk/edgerouter-and-chromecast/
https://www.dd-wrt.com/phpBB2/viewtopic.php?p=1061122