You'll need to use a separate wifi network (SSID) for IPTV and regular Internet access. You'll also need to run Ethernet from your modems IPTV port to an OpenWrt-enabled device. You also need an OpenWrt-enabled device to receive the wireless IPTV network, and run a cable from it to the STB.
Depending on the specifics of your device, you will have one or more VLANs already configured in OpenWrt by default. As an example, my TP-Link WDR4300 has VLAN 1 (LAN) and VLAN 2 (WAN) by default. This is part of my /etc/config/network:
config switch_vlan
option device 'switch0'
option vlan '1'
option ports '1 0t'
config switch_vlan
option device 'switch0'
option vlan '2'
option ports '2 3 4 5 0t'
On both OpenWrt devices used to transmit IPTV, you need to add an additional VLAN, say VLAN 7. The ports on these devices connected to your modem and the STB should be in VLAN 7, untagged. An Ethernet port can only be untagged on one VLAN, so you need to remove it from the VLAN it's already in. Here I use port 2 (as said, you do this on both devices):
config switch_vlan
option device 'switch0'
option vlan '1'
option ports '1 0t'
config switch_vlan
option device 'switch0'
option vlan '2'
option ports '3 4 5 0t'
config switch_vlan
option device 'switch0'
option vlan '7'
option ports '2 0t'
Note that port 0 on my device is the CPU (internal) port, on my (and most) device all VLANs must be tagged ('t') on this port.
Now create an interface. Here are my WAN (default config) and IPTV interfaces:
config interface 'wan'
option ifname 'eth0.1'
option proto 'dhcp'
config interface 'iptv'
option ifname 'eth0.7'
option type 'bridge'
option proto 'none'
"Proto none" because the IPTV interface doesn't need an IP address, as IPTV traffic should just pass through.
Now create wifi-iface sections (/etc/config/wireless) on both devices. The key here is this option:
config 'wifi-iface'
...
option 'network' 'iptv'
Which will connect this wifi-iface to the IPTV interface instead of the default LAN. You'll also need to choose an appropriate 'option mode', as access point (ap) and client (sta) may not work (this depends on how your ISP communicates with your STB). If both OpenWrt devices supports WDS, that's the best (easiest to configure) option. If that isn't the case, you can try a mesh network between them, e.g. batman-adv or 802.11s.
The reason you need OpenWrt devices on both ends is because most consumer device stock firmware won't let you link an Ethernet VLAN to a particular wireless interface. Relatively few devices let you configure VLANs at all, coming to think of it.
Install the tcpdump (or tcpdump-mini) package on your OpenWrt devices, it lets you see what traffic is passing the various interfaces. This can be helpful to find out if traffic from the STB reaches the remote OpenWrt device, and if the ISP responds (based on my examples, tcpdump -i eth0.7 will show traffic on the IPTV interfaces).
(Last edited by makro on 16 Jan 2016, 16:20)