in fact, it will be enough to just block certain ip addresses. but openwrt ip subnet 192.168.11. and mesh network 192.168.5. so i tried to add block rule to the firewall for 192.168.5.5, but it didint work.
The firewall cannot filter packets on subnets that it does not see.
inet openwrt mesh wireless system local network
wan ip 192.168.11.1 192.168.11.2 - 192.168.5.1 192.168.5.0/24
The mesh system appears to be performing NAT, right? This means that the OpenWrt router can not see into that network -- everything that happens on that network that is headed towards the internet appears as traffic from a single IP address: 192.168.11.2. You cannot filter granularly in this case.
The solution is to disable NAT on the mesh device so that the local network exists on the same network (192.168.11.0/24) and then each device will have an IP that can be seen by the OpenWrt router.
MAC addresses are a layer 2 concept - a firewall can only filter based on MAC if it is able to see it as a member of the same L2 network.
Bottom line: your Tenda mesh router system uses NAT masquerading and a firewall such that the OpenWrt router (or any upstream device) sees all the traffic as originating from returning to a single MAC / IP address - the address of the Tenda device itself. This means there is no ability to selectively filter any devices on the Tenda mesh network - it is all or none. Period.
Your solution is to disable NAT (you said that this option is not present), use the firewall on the Tenda (you implied that it doesn’t have all the necessary capabilities), or remove the Tenda and use the OpenWrt router only (and figure out a non-NAT wireless solution if you need better WiFi).
No, what @lleachii wrote is correct. The firewall works at L3 but can filter by MAC addresses when those devices are directly connected to its own LAN (which is basically the boundary of L2 and L3). The reason you cannot filter by MAC address for the devices connected to the mesh system is they are no longer L2 visible since they have gone through the NAT routing layer (L2 to L3) of that mesh system.