I have an OpenWrt 21.02.0 installation on a Raspberry Pi 4 Model B Rev 1.4. This set up has LAN on eth0, a broadband on eth1 and a LTE (4G) on eth2. The wan interfaces eth1 and eth2 are based on USB3 to ethernet adapters and I have configured a mwan3 module to failover from broadband (eth1) to LTE (eth2) only when the broadband becomes unavailable. Both members of mwan3 havd 8.8.4.4 and 8.8.4.4 as their tracking IPs. My broadband (eth1) goes into cable modem supplied by my ISP.
I also have a wireguard client setup that connects to a wireguard VPN provider. A VPN policy routing forces all my outbound traffic through the wireguard client interface.
When I pull out the ethernet cable between the broadband modem and the pi, I expect failover to take place and the wireguard client to start sending all outbound traffic through eth2. However, since the broadband interface on eth1 is still up (only "disconnected" as per mwan3), wireguard fails to acknowledge the difference and keeps trying to send outbound traffic through eth1, resulting in loss of internet connectivity for all LAN members. Now if I manually stop my broadband interface on eth1, the wireguard client switches to the lte interface on eth2 and internet connectivity is restored.
If I completely remove the wireguard client from the set up, I can confirm mwan3 failover works fine and all outbound traffic from my lan is sent via the lte connection.
Please, can anyone provide insights into why the wireguard client does not automatically start sending traffic through lte on eth2 when my broadband on eth1 is disconnected? Is there a workaround I can put in place?
Depending on the configuration, mwan3 might not match any locally originated traffic. Furthermore mixing VPR and mwan3 is not recommended, as they accomplish the same goal and it is a matter of which one has higher iptables entries to have precedence.
I was wondering if it could be possible to write logic in /etc/mwan3.user to alter the iptables based on my needs. This script, I understand, is called on every mwan3 hotplug event. It is just that I simply don't know what to change
Finally managed to solve this confounding problem on my own. The solution is to manipulate default routes for the wireguard connection to ensure packets are sent through the right interface depending on which link is up. An outline of this problem and its solution (on a different piece of hardware) have been described here.
Based on the above idea and after a bit of playing around with my own openwrt installation on raspberry pi, I could cook up the following script that I have added to /etc/mwan3.user. I can confirm this works like a charm. When mwan3 fails over to eth2, the new route rules inserted by this script force wireguard to connect through my lte interface. Upon restoration of eth1, this script alters routes again, in response to which wireguard seamlessly shifts back to the primary eth1 interface.
Now that I have taken a deep dive ito this, I don't agree mixing VPR with mwan3 is such a bad idea after all. In my opinion, they accomplish different things and I don't see why they can't work together to achieve a solution of the kind I have built for myself.
I hope others too can make use of my script as it is or with minor tweaks.
# Allow things to settle down a bit
sleep 5
# IP used to track interfaces
TRACKING_IP="8.8.8.8"
# A custom log prefix for debugging
LOG_PREFIX="WG_MANAGEMENT ::"
# Name of the wireguard client interface
WG_IF=wgclient
# Find the endpoint of the wireguard client interface
WG_ENDPOINT_IP=`wg show ${WG_IF} | grep endpoint | awk '{split($0,a,":"); print a[2]}' | xargs`
# Find the gateway IP address of the main (eth1/bband) interface
ETH1_GATEWAY=`ip route show | grep default | grep eth1 | awk '{split($0,a," "); print a[3]}' | xargs`
logger -t hotplug "${LOG_PREFIX} Interface: ${INTERFACE} / Device: ${DEVICE} / Action: ${ACTION} / WG_IF: ${WG_IF} / WG_ENDPOINT_IP: ${WG_ENDPOINT_IP} / ETH1_GATEWAY: ${ETH1_GATEWAY}"
# validate
if [ "${WG_ENDPOINT_IP}" = "" ] || \
[ "${ETH1_GATEWAY}" = "" ] || \
[ "${INTERFACE}" != "bband" ] || \
! ([ "${DEVICE}" = "eth1" ] || [ "${DEVICE}" = "eth2" ]) || \
! ([ "${ACTION}" = "disconnected" ] || [ "${ACTION}" = "connected" ])
then
logger -t hotplug "${LOG_PREFIX} one or more conditions not met. Cannot progress."
exit 1
else
logger -t hotplug "${LOG_PREFIX} validation ok"
fi
/bin/ping -W 2 -I eth1 -c1 ${TRACKING_IP} 1>/dev/null 2>/dev/null
ETH1_UP=$?
/bin/ping -W 2 -I eth2 -c1 ${TRACKING_IP} 1>/dev/null 2>/dev/null
ETH2_UP=$?
# Don't change anything if none of the outbound interfaces are available
if [ $ETH1_UP -ne 0 ] && [ $ETH2_UP -ne 0 ] ; then
logger -t hotplug "${LOG_PREFIX} both interfaces appear to be down. Nothing to do."
exit 1
fi
logger -t hotplug "${LOG_PREFIX} about to make route changes"
# Delete the original route first
ip route del ${WG_ENDPOINT_IP}
logger -t hotplug "${LOG_PREFIX} Original route deleted"
# Now add a route for the interface that is available
if [ $ETH1_UP -eq 0 ]; then
logger -t hotplug "${LOG_PREFIX} Routing ${WG_IF} through eth1"
ip route add ${WG_ENDPOINT_IP} via ${ETH1_GATEWAY} dev eth1 proto static
else
logger -t hotplug "${LOG_PREFIX} Routing ${WG_IF} through eth2"
ip route add ${WG_ENDPOINT_IP} dev eth2 proto static
fi
logger -t hotplug "${LOG_PREFIX} New route added"
With due respect, here's an example. In my setup, LAN members are tunnelled through a wireguard client for all kinds of traffic. This has been configured as a general rule in VPR, with one exception. When I run a speed test (on www.speedtest.net), I don't want the wireguard tunnel to show me reduced speeds and have accordingly added a bypass rule in VPR to ignore the wireguard interface for connections to www.speedtest.net. I wonder how such a thing can be achieved with mwan3?
More or less in the same way as PBR does. However since mwan3 is not supporting ipsets by default, the rules will have to be entered manually.
To be clear, I am not saying that they are completely the same. You can do some things easier with one and some with the other. However the principal is the same and PBR is more newbie friendly. On the other hand mwan3 is more focused on load balancing and can better handle multiple uplinks.
