Just wanted to quickly wrap my head around a situation that I have. I have an OpenWrt router that has a standard WAN setup, and I have been tasked with setting up a secondary WAN in case our primary connection goes down. There was a Netcomm 4G Wi-Fi Modem Router that was lying around and I figured I could use this as a gateway for a secondary connection by defining a second "WAN interface" on the LAN port (let's call it LWAN), and configuring MWAN so it can use this interface.
The LWAN interface is configured as follows:
Type = Static Address
Interface = br-lan
IP = 192.168.14.253
Gateway = 192.168.14.254 (the 4G router)
Firewall = WAN zone
Normally I would re-assign a LAN port and use this as WAN2, but the location that the main router is situated has no 4G service, and I have to instead place the router in another location which is quite far away, hence why I am in a situation where my secondary gateway is on my LAN.
I have been trying to get this working in a test environment, and I can get OpenWrt to send traffic to the LWAN interface, but something isn't right. Despite my metric configuration, it ALWAYS defaults to the LWAN interface, and never uses my WAN interface. Failover doesn't really work as a result.
What am I doing wrong here? I feel like I am missing something. Using OpenWrt 23.05.5, and MWAN is working fine thanks to the nft2ipset scripts. I am open to using VLANs but I don't want to break the LAN and WLAN clients that are currently connected to the Netcomm 4G Modem Wi-Fi Router. Advice appreciated!
There might be a way to do this with routing and configuration on the openwrt router and alias interfaces or something. However I would suggest the following:
move all the wired clients on the netcomm onto the unmanaged switch, or on one of the last remaining ports on the openwrt router, or get openwrt router for AP, unmanaged/managed switch etc.
Wifi clients can be transitioned with another SSID on the openwrt router. Or on another AP with vlans.
Disable wifi on the 4g modem router and put it on another lan port repurposed as another dhcp client port
edit1:
I guess one could start with dumping the config for the openwrt router and the mwan3 config so it's not just some simple configuration.
Thanks for the input. The 4G modem is in another area of the building - it is far from everything else. Are you saying I need to add another Wi-Fi AP with ethernet? Is there no other way?
Ah. I'm suggesting the dumb "throw parts / money" at the problem as that's how I know how to get you a working solution, that is more typical and easier to support =P
Anyway. So we should start with your mwan3 config and see if anyone has some input or done it by routing before.
What I'd think to do is make sure your mwan3 ipv4 rules are correct. Plus the members are configured correctly etc. Plus we need to check the what I assume is static IP alias/veth interface or something on your lan subnet that you will use to forward traffic to the 4G modem.
So I guess start with a uci show ? or cat your /etc/config
But It's not something I've ever tried to do before. Worst case if you can't move the 4G modem and can't run another cable you can split the pairs. But vlan trunking and managed switches and/or another openwrt AP would work too.
is metric 0 a valid metric for your wan interface? I'd think it would need to be 1.
But LWAN is a different metric of 2 which is good.
Personally I start at 10 and work my way up from there.
Other than something like creating a virtual ethernet adapter IDK where to go from here unless I actually try to replicate your setup. (But I'm doing other dev work at the moment....)
Cool. Other than check your routing tables and the mwan3 status IDK. Is metric 1 used by something else? Are you getting a port created on br-lan?
Anyway as an aside I quickly drew up two architectures on how I'd achieve the setup normally. (But I'd probably throw in two managed switches, one before the bridged AP, the other to get rid of the unmanaged switch)
The second diagram with VLANs is probably my best bet as I won't be blessed with the ability to have a direct connection with the 4G modem. That said I am a bit surprised there is no other way to achieve what I'm trying to do... will be interesting to see if anyone else has managed to get this working somehow, perhaps on a second subnet or something?
We can wait and see. I put it in the too hard basket and "just add more hardware" =P.
I can think with appropriate routing and the creation of appropriate virtual ethernet adapters one could do it. But possible and good idea are two different things =P I've picked up managed switches and openwrt capable AP's for under 20-25AUD so you can make the call on time vs money I guess as well.
edit:
Yeah also if you can live with 100mbit but can't run another cable: Split the pairs in your network cable and add a dumb AP....
Yes, you need a separate interface for the second wan, you cannot run it through the lan.
It can be a simple vlan or even a secondary interface running on the same physical interface (br-lan) or you can separate a port from the switch and allocate it to the new wan.
The latter is what I have in my setup - I suspect that this isn't sufficient and I need to have that interface on a separate VLAN. I'll do some testing and see what I find.
Thanks - I tried adjusting it to a separate subnet, unfortunately I have the same issue. All my traffic ends up through that interface when I don't want it to. If I try moving my 4G router to another subnet, I have the opposite issue and it doesn't work at all.
Please run the following commands (copy-paste the whole block) and paste the output here, using the "Preformatted text </> " button:
Remember to redact passwords, MAC addresses and any public IP addresses you may have
ubus call system board; \
uci export network; uci export mwan3; \
uci export dhcp; uci export firewall; \
ip -4 addr ; ip -4 ro li tab all ; ip -4 ru; \
mwan3 status
Apologies, I missed that. I changed it back to a /24 and LWAN is back up, but all my traffic is now forced through the LWAN interface despite having WAN as the priority via metric.
Edit: I also tried changing the interface from br-lan to eth0 and eth0.1 and neither configuration yeilded a connection. Only using br-lan allows LWAN connectivity, but at the cost of all my traffic being routed through it.
I feel like this is technically possible without additional VLANs and equipment but I am stumped. Any other ideas? Perhaps I need to be manually configuring routes? Or maybe it's an MWAN issue? Here's my MWAN troubleshooting output:
MultiWAN Manager - Troubleshooting
Software-Version
-------------------------------------------------
OpenWrt - 23.05.5
Output of "ip -4 a show"
-------------------------------------------------
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
6: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
inet 192.168.14.1/24 brd 192.168.14.255 scope global br-lan
valid_lft forever preferred_lft forever
inet 192.168.15.1/24 brd 192.168.15.255 scope global br-lan
valid_lft forever preferred_lft forever
8: eth0.2@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
inet 192.168.20.233/24 brd 192.168.20.255 scope global eth0.2
valid_lft forever preferred_lft forever
Output of "ip -4 route show"
-------------------------------------------------
default via 192.168.20.1 dev eth0.2 proto static src 192.168.20.233 metric 1
default via 192.168.15.254 dev br-lan proto static metric 2
192.168.14.0/24 dev br-lan proto kernel scope link src 192.168.14.1
192.168.15.0/24 dev br-lan proto static scope link metric 2
192.168.20.0/24 dev eth0.2 proto static scope link metric 1
Output of "ip -4 rule show"
-------------------------------------------------
0: from all lookup local
1001: from all iif eth0.2 lookup 1
1002: from all iif br-lan lookup 2
2001: from all fwmark 0x100/0x3f00 lookup 1
2002: from all fwmark 0x200/0x3f00 lookup 2
2061: from all fwmark 0x3d00/0x3f00 blackhole
2062: from all fwmark 0x3e00/0x3f00 unreachable
3001: from all fwmark 0x100/0x3f00 unreachable
3002: from all fwmark 0x200/0x3f00 unreachable
32766: from all lookup main
32767: from all lookup default
Output of "ip -4 route list table 1-250"
-------------------------------------------------
Routing table 1:
default via 192.168.20.1 dev eth0.2 proto static src 192.168.20.233 metric 1
192.168.14.0/24 dev br-lan proto kernel scope link src 192.168.14.1
192.168.20.0/24 dev eth0.2 proto static scope link metric 1
Routing table 2:
default via 192.168.15.254 dev br-lan proto static metric 2
192.168.14.0/24 dev br-lan proto kernel scope link src 192.168.14.1
192.168.15.0/24 dev br-lan proto static scope link metric 2
Output of "iptables -t mangle -w -L -v -n"
-------------------------------------------------
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
615K 711M mwan3_hook all -- * * 0.0.0.0/0 0.0.0.0/0
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
6109 1614K mwan3_hook all -- * * 0.0.0.0/0 0.0.0.0/0
Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain mwan3_connected_ipv4 (2 references)
pkts bytes target prot opt in out source destination
406K 599M MARK all -- * * 0.0.0.0/0 0.0.0.0/0 match-set mwan3_connected_ipv4 dst MARK or 0x3f00
Chain mwan3_custom_ipv4 (2 references)
pkts bytes target prot opt in out source destination
0 0 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 match-set mwan3_custom_ipv4 dst MARK or 0x3f00
Chain mwan3_dynamic_ipv4 (2 references)
pkts bytes target prot opt in out source destination
0 0 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 match-set mwan3_dynamic_ipv4 dst MARK or 0x3f00
Chain mwan3_hook (2 references)
pkts bytes target prot opt in out source destination
618K 712M CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00 CONNMARK restore mask 0x3f00
5657 1735K mwan3_ifaces_in all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00
1740 198K mwan3_custom_ipv4 all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00
1740 198K mwan3_connected_ipv4 all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00
1097 92307 mwan3_dynamic_ipv4 all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00
1097 92307 mwan3_rules all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00
621K 712M CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 CONNMARK save mask 0x3f00
490K 603M mwan3_custom_ipv4 all -- * * 0.0.0.0/0 0.0.0.0/0 mark match ! 0x3f00/0x3f00
490K 603M mwan3_connected_ipv4 all -- * * 0.0.0.0/0 0.0.0.0/0 mark match ! 0x3f00/0x3f00
85258 4771K mwan3_dynamic_ipv4 all -- * * 0.0.0.0/0 0.0.0.0/0 mark match ! 0x3f00/0x3f00
Chain mwan3_iface_in_LWAN (1 references)
pkts bytes target prot opt in out source destination
0 0 MARK all -- br-lan * 0.0.0.0/0 0.0.0.0/0 match-set mwan3_custom_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
101 8525 MARK all -- br-lan * 0.0.0.0/0 0.0.0.0/0 match-set mwan3_connected_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
0 0 MARK all -- br-lan * 0.0.0.0/0 0.0.0.0/0 match-set mwan3_dynamic_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
42 1680 MARK all -- br-lan * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00 /* LWAN */ MARK xset 0x200/0x3f00
Chain mwan3_iface_in_wan (1 references)
pkts bytes target prot opt in out source destination
0 0 MARK all -- eth0.2 * 0.0.0.0/0 0.0.0.0/0 match-set mwan3_custom_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
3238 1336K MARK all -- eth0.2 * 0.0.0.0/0 0.0.0.0/0 match-set mwan3_connected_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
0 0 MARK all -- eth0.2 * 0.0.0.0/0 0.0.0.0/0 match-set mwan3_dynamic_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
534 190K MARK all -- eth0.2 * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00 /* wan */ MARK xset 0x100/0x3f00
Chain mwan3_ifaces_in (1 references)
pkts bytes target prot opt in out source destination
5420 1696K mwan3_iface_in_LWAN all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00
5062 1620K mwan3_iface_in_wan all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00
Chain mwan3_policy_wan_LWAN (0 references)
pkts bytes target prot opt in out source destination
0 0 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00 /* wan 3 3 */ MARK xset 0x100/0x3f00
Chain mwan3_rules (1 references)
pkts bytes target prot opt in out source destination
I don't think what you want to achieve can be done this way.
You want to use br-lan as an inbound and outbound interface and that breaks mwan3, because everything arriving from br-lan is forwarded directly to routing table 2.
Also, the mwan3 rule is not created. This could be because options src_ip and ipset overlap, or because 1.1.1.1/30 is not valid CIDR notation.
