Fundamentally, if you want the "main" openwrt router to handle all the major routing tasks for your entire network, and if you want to ensure that non-VPN'd devices are not double-NAT'd, what we've done here is a very workable method.
The other way to do this would be to use the 'dumb ap + guest network' paradigm where the upstream LTE router would handle the routing for at least one of the networks (i.e. stuff that shouldn't be double-NAT'd, presumably the trusted LAN). Then you would NAT off of that for the guest network, or tunnel through it for a VPN network.
Your guest network will break with masquerading disabled (because we don't have a route defined for it on the main router)... but now the reason your lan isn't working probably because your VPN is active (unless it has been disabled)... you don't have lan > vpn forwarding enabled.
Honestly, the easiest way to do this is to remove all the PBR and VPN stuff entirely to prove that everything is working with the symmetric routing. Then you can 'layer' the extra stuff on as you need... in other words, you'd be building up from a base of a symmetric wan routing situation, but simplifying for the first principles.
You could make a backup and then factory reset your main router... this would then work right out of the box (for the basic routing), and it should still work with the wan masquerading disabled.
I'm not sure it is worth trying to reverse engineer the routes in this situation... but most of them point to the wan (and not the VPN). For all of the guest network, that will break because there is no static route on the main router for it.
Also, you have a route that sends stuff to 192.168.0.1/32 which shouldn't be needed.
Try disabling all of your routes and the VPN. Then test.
We have too many variables in play here to know where the problem is. That's why we should simplify.
You shouldn't need any special rules to acces the Zyxel (except when the VPN is enabled)... lets start with the VPN disabled and all user-added routes and policies disabled on the main router.
The other alternative would be, if you have OpenWrt running on the NR7101 and assuming the hardware is fast enough, to make that one your router - and the rt3200 just another AP…
I have used a similar setup requiring static routes before (during the overlap of contracts with different ISPs, phones (SIP) had to be serviced via VDSL (bthub5), while everything else going over fibre (nbg6817) already), so I know it works - but your additional complexity makes it really hard to see the wood among the trees, so simplify to the extent possible and build up from the basics again.
@psherman and @slh should the guest network be setup on the Zyxel as a bridged lan interface or can the single lan interface in the Zyxel carry both normal LAN and guest LAN traffic? I suppose I just set a static route from the Zyxel for 192.168.2.0 (downstream guest subnet) via gateway 192.168.0.2 (downstream router IP) in addition to my static route for 192.168.1.0 via gateway 192.168.0.2?
Isn't there an easy way to recreate bridge mode in OpenWrt for my Zyxel? I read about kmod-trelay? I am guessing I can't just make the LTE a LAN interface and bridge with the existing LAN interface? That sounds too simple.
With a simple LTE-LAN bridge these complexities go away don't they?
I mean I relay want my downstream OpenWrt router to do all the heavy lifting. The Zyxel is only for modem.
Albeit perhaps a downside is I might lose out on the use of my WiFi antenna on the Zyxel but I'm not sure that matters much.
But I'm new to networking so I'm open minded. I can see how the approach we've been working on so far retains flexibility. And I guess spreading some of the load out between my RT3200 and Zyxel NR7101 may make some sense.
@psherman could my issues relate to my CG-NAT ISP wan / subnet? See this from Zyxel:
root@OpenWrt:~# uqmi -d /dev/cdc-wdm0 --get-current-settings
{
"pdp-type": "ipv4",
"ip-family": "ipv4",
"mtu": 1500,
"ipv4": {
"ip": "[CG-NAT WAN IP]",
"dns1": "XX",
"dns2": "YY",
"gateway": "[ISP GATEWAY]",
"subnet": "255.255.255.252"
},
"ipv6": {
},
"domain-names": {
}
}
root@OpenWrt:~# ip route show table local
broadcast 10.234.22.200 dev wwan0 scope link src [CG-NAT WAN IP]
local [CG-NAT WAN IP] dev wwan0 scope host src [CG-NAT WAN IP]
broadcast 10.234.22.203 dev wwan0 scope link src [CG-NAT WAN IP]
broadcast 127.0.0.0 dev lo scope link src 127.0.0.1
local 127.0.0.0/8 dev lo scope host src 127.0.0.1
local 127.0.0.1 dev lo scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo scope link src 127.0.0.1
broadcast 192.168.0.0 dev br-lan scope link src 192.168.0.1
local 192.168.0.1 dev br-lan scope host src 192.168.0.1
broadcast 192.168.0.255 dev br-lan scope link src 192.168.0.1
root@OpenWrt:~# ip route show table main
default via [ISP GATEWAY] dev wwan0 src [CG-NAT WAN IP]
10.234.22.200/30 dev wwan0 scope link src [CG-NAT WAN IP]
192.168.0.0/24 dev br-lan scope link src 192.168.0.1
I am finding that:
I can ping Zyxel at 192.168.0.1 from my main OpenWrt router at 192.168.1.1
I cannot ping my OpenWrt router at 192.168.1.1 from my Zyxel at 192.168.0.1
I cannot ping or access the web interface of Zyxel form any LAN client
... unless I have 'wan' firewall set to masquearing on the OpenWrt router.
Do I need a static route from my upstream Zyxel router to my downstream OpenWrt router?
No, cg nat is not affecting this routing (it appears to be on a completely different subnet).
But cg-nat also means that you are already double-nat’d behind the first router. So having another layer of nat (on the second/main router) actually has little additional negative impact.
With cg-nat in mind here, you have very little to gain from trying to avoid double nat on your side. It’s great to learn how to do symmetric routing without masquerading, but it won’t gain you anything in terms of performance or connectivity.
I agree with @slh ‘s comments. Beyond that, there are just too many variables in your setup. Starting fresh is the way to approach this problem - start with first principles, test, then add vpn, test, and then the policy routing - you’ll find out where and when things break and can focus on the specific problem rather than reverse engineering your entire network configuration.
The upstream router has no idea there is a downstream network - it doesn’t know the subnet or which host acts as the gateway. You have to tell it that information.
On the other hand, the downstream router is connected to the upstream network explicitly. It has an address on the upstream network (I.e the wan) and it sets up a default route via that network and it knows the subnet.
Think of it like a hallway and exit paths. If you have a room off the hallway, you can tell while you are in the hallway because there is a door. From the hallway (with the room door closed, you do not know if there is an en-suite bathroom in that room. However, if you are in the room, you know about both the hallway and the bathroom. if you are in the room, you know that you should exit by the hallway.
