[Guide] High availability failover SDWAN: Maintain TCP/UDP sessions across 2 or more connections

Installing and Using OpenWrt
1

I have been dabbling with a solution to maintain my internet connectivity while having 2+ semi reliable internet connections. Anyone who lives in the boonies and has starlink probably is familiar with seconds to minutes of outages, as I've found in the first week of having the service. I had planned for this and have been abusing/testing an unlimited dataplan and easy tether, this actually works great (albiet slow at 5Mbps) as long as you also use a VPN to obscure your traffic to prevent steam shaping by the carrier.

Initially I used a plain old mwan3 failover, however I quickly discovered that streams that were started on cellular continued on cellular even when starlink was available for several minutes. I also came up with a mwan user script to selectivly kill conntrack states on lower priority connections when a higher priority connection became available. This actually worked ok for streaming video, however it would occationally cause the app to just stop the session, frequently enough to be more annoying than just having to use 480p on my 4K TV.

Enter SDWAN bonding. This had been a concept I was well familiar with as a business solution, however I really had no reason to play around with this since I had access to reliable cable internet (I still do, but I'm practicing some sadomasticicm with unreliable internet to prepare for my Diogenes life in a van) There is a home user targeted service called https://speedify.com/store/ that I looked at (openwrt possible on a rpi), but for $5 a month I can host my own on https://ramnode.com/ or https://digitalocean.com/ and have full control of the solution.
I will be detailing the components I setup on my cute little OpenWRT router that combines Cellular, Wifi, and Ethernet into one logical pipe where NAT sessions are tracked from one place and failover can occur in sub second responce times in some cases.

Each of these components can be replaced with a like solution, however I have familiarity with them, so they

Pros of this solution:

  1. Since the termination of your "LAN" is in the VPS and your "Edge" is always on and available to connect to, TCP/UDP sessions will be valid regardless if your traffic is coming from 1, 2, or even 100 different ISP links
  2. You can port-forward from your VPS into any device on your network, even through CGNAT
  3. In my exprience, the biggest benefit is having an available connection 100% of the time, with merely moments of outage when one connection is unavailable. In the case of a 10 second outage of a primary connection, the secondary/backup connections will take over to maintain (slower) traffic flow before the primary connection returns and becomes the data path within secconds.. The experience shouldn't be much different than watching a streaming video and someone starts downloading a big file.

Cons:

  1. Increased complexity. I can turn off my router and turn it back on and it will connect just as well as a simple ethernet internet connection. However there are a lot of break points to troubleshoot on your own
  2. Increased security surface area: Not only do you have to be aware of firewall and updates to your home router, but you also need to maintain and manage the connection on your VPS which is probably reused from hundreds of other customers who may have attracted the attention of hackers/bot nets that will be scanning your new "public ip". However if you watch your WAN zone traffic logs at home, you likely get port scanned or unsolicited traffic often anyway
  3. Additional cost over just having two plain connections in fail over
  4. the VPN adds overhead, so if you already have existing wireguard tunnels on your home router, you will need to reduce their MTU to approx 1300 (could be a bit higher, I just shotgun this value while troubleshooting my issues)

VPN side:

  • VPS, suggested 2 core, >512MB ram. Shared CPU seems to be fine and my opinion 2 shared cores is better than 1 dedicated core
  • Vyos Router ISO from https://vyos.io (This is essentially a wrapper for 2 key technologies: FRR and WireGuard/OpenVPN)

OpenWrt side:
Router:

  • Linksys E5480, however a router with USB3 will be preferred
  • Self Powered USB hub
  • Alfa MT7612 USB Wifi Adapter
  • Easytether Driver

Cellular Link

  • 5G Android Phone with EasyTether installed
  • 2x wide band Cellular 10dbi antennas
  • 2x N-Type to SMA cable
  • 2x SMA to uFL cable adapters
  • USB cable

Wifi Link

  • Public Wifi, or any other Wifi Network. This is how I currently connect to starlink for testing

Ethernet Link

  • Any primary connection, Eventually I'll get a PoE adapter for starlink

I'll go into detail with future edits, but here is a rough outline of How I set this up

VPS Setup

  1. setup a VPS at your preferred host
  2. Download/Upload the latest Vyos image
  3. In the remote console, Create a new VM instance and install Vyos, setting a temporary password, then reboot (don't use your common password since the keys are passed through the "untrusted" host network into VNC or what ever console they provide.
  4. configure basic network configuration and test connectivity to SSH from your home computer
  5. ssh into your new Vyos router and create a new user such as admin, but not root with a secure password
  6. Setup OTP authentication for increased security
  7. set SSH to listen only on an alternate port and not on port 22
  8. Setup firewall zones for WAN, LOCAL, and SDWAN
    8a) LOCAL to WAN default Action should be accept
    8b) LOCAL to SDWAN default action should be accept
    8c) SDWAN to WAN default action should be accept
    8d) WAN to LOCAL default action should be drop
    8e) WAN to SDWAN default action should be drop
  9. Add rules to WAN to LOCAL for your ssh port from step 7, and expected UDP ports for VPN tunnels ( i.e. 1350-1352)
  10. Commit your changes and save. Then reboot. Confirm you can ssh into your router before closing your remote session
    10a) If you can SSH into your router, delete the default vyos user
    10b) if you can't SSH into your router, login via the host console to your VPS and review your settings until it's safe to delete the basic vyos login
  11. configure wireguard/OpenVPN tunnel interfaces, one on each port specified in step 9
  12. Create a dummy interface and assign it IP 10.99.99.1/32
  13. Configure OSPF with various costs and hello
    13a) Cellular VPN should have a cost of 100, hello of 1 sec, and a dead timer of at least 600 se.
    13b) WiFi VPN should have a cost of 10, hello of 1 sec and dead timer of 2. I had tried using a hello multiplier, but the connection wouldn't stay open for more than 60 seconds, so this is what I accept
    13c) Ethernet/Primary VPN should have a cost of 1, hello of 1 and a dead timer of 2. This could be a good candidate to use hello-multiplier
    13d) Any number of additional connection can be added using the same configuration and tuning the hello and dead timers will be a creative process
    13e) assign each of the VPN interfaces to ospf area 0.0.0.0, and the dummy interface area 0.0.0.5
  14. setup a nat rule with an outbound interface of eth0 using masquerade
    14a) I have not tried this, but it may be an option to use NAT 66 to translate your private fd00::/8 networks into a subnet assigned by your VPS host.

OpenWrt setup

  1. Install OpenWrt on a router, and configure the driver packages for your WiFi adapter
    1a) connect the powered USB hub so any devices, including the phone don't rob power from the router's SOC
  2. Download and install the easy tether driver. I suggest these modifications to the hotplug script to ensure the tap device is created with minimal interaction
  3. Configure and confirm that your internet uplinks work on a basic level (ping -I 1.1.1.1). Ensure each link has an appropriate metric set, (ethernet 1, Wifi 10, Cellular 100)
  4. Install the following software: frr, frr-staticd, frr-watchfrr, frr-ospfd, frr-vtysh, frr-zebra, mwan3, luci-proto-wireguard or luci-app-openvpn
  5. configure your VPN of choice and confirm you can connect and ping to/from both sides, create one tunnel device for each VPN device created on the vyos router
  6. configure mwan3 to track each interface and configure unique policies for the destination IP of your server and each VPN port, i.e. 1350-1352. Assign each policy to a rule unique to each interface. (this is where the source of the magic is!)
  7. Save and apply the configuration, and re-check the connectivity to/from each end of the tunnel
  8. configure frr either with frr.conf or via vtysh to setup OSPF similarly to step 12 of the vyos router setup. However additionally set each of your LAN interfaces to area 0.0.0.10. (this announces your local network as a route
  9. run sh ip ospf neighbor to confirm at least one of your links is connected (hopefully all of them are for verification)
    9a) also run sh ip route to confirm you see a route to 10.99.99.1/32
  10. configure two routes which will become the new default gateway
    10a) ip route 0.0.0.0/1 10.99.99.1
    10b) ip route 128.0.0.0/1 10.99.99.1
  11. check that the public IP you configured in the vyos router now appears when your check your ip at somewhere like whatsmyip.org
  12. test with your secondary/primary links, start a ping -A 1.1.1.1 and watch the stream of packets while you disconnect your connection, and then reconnect. if everything is working, you will see a momentary pause before it continues, possibly with a higher ping time. The same should be true when you reconnect
    12a) you can also verify the routing is working by running sh ip ospf neighbor and confirming you see 1 or more neighbors listed and the two default route hacks are pointed via recursive route to the lowest cost interface of your vyos router.

bonus
This is provided for educational purposes, consult your local/FCC regulations and contact your carrier

  1. choose a "cheap" 5G phone, preferably one with a plastic back. I use the Samsung S20 5G FE
  2. use a heat gun and de-lid the back of the phone and unscrew the housing until you reveal tiny coax cables
  3. typically, the connections at the top of the phone are the input, the bottom are the remote antennas
  4. remove the antenna wires and replace them with the SMA to uFL adapters, routing them in a similar way to the factory. I believe red and white are LTE/5G MiMO, blue is bluetooth, which I just remove. (some experimentation is needed to confirm on the model of phone you use_
  5. punch a small holein the back of the phone and cut a slit from the edge to the hole. then slide the wires in so they come out of the back
  6. replace the back of the phone and use strong tape to anchor them in place
  7. connect your wide band external antennas to these wires with the appropriate SMA extensions and tape these into place on the back of the phone to prevent damage
  8. confirm an improvement in signal strength and network connectivity
  9. connect your phone to your router, install the easy tether app, enable USB debugging, and ensure the default USB mode is set for transferring files

That's pretty much all there was to it. I'll add some configs to this for easier copy paste + where you will need to add your own settings. If some of these terms don't make sense, it's probably better to look at speedify, otherwise I hope this is useful to at least one other person.
This was the result of trying and experimenting with hundreds of dollars of things from v1 on a GLiNET router I had because I just wanted a drop-in and go, to a RPi CM4 router I tried to use as a VPN gateway (these are $$$now) that I now just have in my bin because I have a 1 device/router solution.

5 Likes
2

Interesting, thanks for sharing!

3

I will come back and add configs, but I've had a log of little issues come up and have been squashing them since posting.

The latest discovery is that an unlimited cellphone plan is $55 a month, however an AT&T unlimited tablet plan is $20/mo and the sim works in a Nighthawk M6 Pro, but it still gets <5Mbps after 22GB of monthly use. This is 10 times better than easy tether, let alone half the monthly cost.

I'm still trying to resolve an issue where the tunnels won't reconnect when the Linksys router is rebooted until the VPS is rebooted.

1 Like
4

any updates?

5

I will try to get back and share the configs this weekend once I find my laptop. life happened, but I can also report that this configuration is rock solid and ive improved the configuration since posting (particularly setting the cellular OSPF hello to 60 sec and dead time to 10 minutes, and then adding BFD to the primary link which will fail to cellular if latency is >300ms).
with the one caveat that when the openwrt router is rebooted, some wireguard session is broken and can only be fixed by rebooting the vyos\vps router.
Unsure if it's a openwrt (unlikely) or vyos issue, but has persisted through several updates, but havent tried 23.05 yet

2 Likes
6

Latest update,

Ive solved the wireguard issue and switched to bgp which combined were necessary to make the solution complete. Im working on dumping the config snippets

1 Like
7

I'm in a typing mood, and I've been held off because I had a breaking issue with openwrt devices that don't have a battery failing to reconnect to the wireguard because when the device came online after a reboot, the timestamps were behind the cloud router.

I now have a work around for this, I've also rebased on bgp for the routing protocol. I'm posting the generic configuration below, so knowledge of the basics will be needed to configure your devices as necessary, firewall rules, etc...

Vyos router config

#set dummy if with default gateway ip
set interfaces dummy dum0 address '10.23.22.1/32'
set interfaces dummy dum0 address 'fd00:f9a8:9a7e:399::1/128'

#configure wireguard interfaces, MTU 1340 ensures cellular networks do not drop traffic.
#all interfaces must have the same MTU to prevent larger packet streams from being dropped when they fail to cellular
set interfaces wireguard wg100 ip adjust-mss 'clamp-mss-to-pmtu'
set interfaces wireguard wg100 ipv6 adjust-mss 'clamp-mss-to-pmtu'
set interfaces wireguard wg100 mtu '1340'
set interfaces wireguard wg100 peer peername-wan allowed-ips '0.0.0.0/0'
set interfaces wireguard wg100 peer peername-wan allowed-ips '::/0'
set interfaces wireguard wg100 peer peername-wan persistent-keepalive '60'
set interfaces wireguard wg100 peer peername-wan preshared-key 'key-here'
set interfaces wireguard wg100 peer peername-wan public-key 'key-here'
set interfaces wireguard wg100 port '9468'
set interfaces wireguard wg100 private-key 'key-here'
set interfaces wireguard wg101 address '10.33.23.5/30'
set interfaces wireguard wg101 address 'fd00:f9a8:9a7e:301::1/64'
set interfaces wireguard wg101 ip adjust-mss 'clamp-mss-to-pmtu'
set interfaces wireguard wg101 ipv6 adjust-mss 'clamp-mss-to-pmtu'
set interfaces wireguard wg101 mtu '1340'
set interfaces wireguard wg101 peer peername-wwan allowed-ips '0.0.0.0/0'
set interfaces wireguard wg101 peer peername-wwan allowed-ips '::/0'
set interfaces wireguard wg101 peer peername-wwan persistent-keepalive '60'
set interfaces wireguard wg101 peer peername-wwan preshared-key 'key-here'
set interfaces wireguard wg101 peer peername-wwan public-key 'key-here'
set interfaces wireguard wg101 port '9469'
set interfaces wireguard wg101 private-key 'key-here'
set interfaces wireguard wg102 address '10.33.23.9/30'
set interfaces wireguard wg102 address 'fd00:f9a8:9a7e:302::1/64'
set interfaces wireguard wg102 ip adjust-mss 'clamp-mss-to-pmtu'
set interfaces wireguard wg102 ipv6 adjust-mss 'clamp-mss-to-pmtu'
set interfaces wireguard wg102 mtu '1340'
set interfaces wireguard wg102 peer peername-wifi allowed-ips '0.0.0.0/0'
set interfaces wireguard wg102 peer peername-wifi allowed-ips '::/0'
set interfaces wireguard wg102 peer peername-wifi persistent-keepalive '25'
set interfaces wireguard wg102 peer peername-wifi preshared-key 'key-here'
set interfaces wireguard wg102 peer peername-wifi public-key 'key-here'
set interfaces wireguard wg102 port '5643'
set interfaces wireguard wg102 private-key 'key-here'

#Ensure that all outbound traffic uses your cloud router's public IP via NAT
set nat source rule 100 outbound-interface 'eth0'
set nat source rule 100 source address '0.0.0.0/0'
set nat source rule 100 translation address 'PUBLIC IP HERE'

#Configure preferences for routes learned via each wireguard tunnel, this is the equiv of setting the cost in OSPF. This is important to ensure that the starlink connection always has priority, the Wifi network is secondary, and cellular is teriary via local-preference.
#Note these local-preferences MUST be set on each end, beacuse they aren't announced over the tunnel
set policy prefix-list BGP_PTH-prefixlist rule 10 action 'permit'
set policy prefix-list BGP_PTH-prefixlist rule 10 prefix '10.23.22.1/32'
set policy prefix-list6 BGP_PTH-prefixlist rule 10 action 'permit'
set policy prefix-list6 BGP_PTH-prefixlist rule 10 prefix 'fd00:f9a8:9a7e:399::1/128'
set policy route-map BGP_RMAP_PTH-in rule 10 action 'permit'
set policy route-map BGP_RMAP_PTH-in rule 10 match peer 'fd00:f9a8:9a7e:300::2'
set policy route-map BGP_RMAP_PTH-in rule 10 set extcommunity bandwidth '100'
set policy route-map BGP_RMAP_PTH-in rule 10 set local-preference '200'
set policy route-map BGP_RMAP_PTH-in rule 20 action 'permit'
set policy route-map BGP_RMAP_PTH-in rule 20 match peer 'fd00:f9a8:9a7e:301::2'
set policy route-map BGP_RMAP_PTH-in rule 20 set extcommunity bandwidth '5'
set policy route-map BGP_RMAP_PTH-in rule 20 set local-preference '105'
set policy route-map BGP_RMAP_PTH-in rule 30 action 'permit'
set policy route-map BGP_RMAP_PTH-in rule 30 match peer 'fd00:f9a8:9a7e:302::2'
set policy route-map BGP_RMAP_PTH-in rule 30 set extcommunity bandwidth '50'
set policy route-map BGP_RMAP_PTH-in rule 30 set local-preference '150'
set policy route-map BGP_RMAP_PTH-in rule 100 action 'permit'
set policy route-map BGP_RMAP_PTH-out rule 10 action 'permit'
set policy route-map BGP_RMAP_PTH-out rule 10 match ipv6 address prefix-list 'BGP_PTH-prefixlist'
set policy route-map BGP_RMAP_PTH-out rule 11 action 'permit'
set policy route-map BGP_RMAP_PTH-out rule 11 match ip address prefix-list 'BGP_PTH-prefixlist'
set policy route-map BGP_RMAP_PTH-out rule 999 action 'deny'
set policy route-map BGP_wg103-in rule 10 action 'permit'
set policy route-map BGP_wg103-in rule 10 match peer 'fd00:f9a8:9a7e:1::1'
set policy route-map BGP_wg103-in rule 10 set as-path prepend-last-as '10'
set policy route-map BGP_wg103-in rule 10 set extcommunity bandwidth '1'
set policy route-map BGP_wg103-in rule 10 set local-preference '101'
set policy route-map BGP_wg103-in rule 999 action 'permit'
set policy route-map BGP_wg103-out rule 10 action 'permit'
set policy route-map BGP_wg103-out rule 10 match ipv6 address prefix-list 'BGP_PTH-prefixlist'
set policy route-map BGP_wg103-out rule 10 set
set policy route-map BGP_wg103-out rule 11 action 'permit'
set policy route-map BGP_wg103-out rule 11 match ip address prefix-list 'BGP_PTH-prefixlist'
set policy route-map BGP_wg103-out rule 11 set
set policy route-map BGP_wg103-out rule 999 action 'deny'

#sets a BFD fast profile so that the starlink connection is monitored rapidly for a need to fail over and return to the route calculation
set protocols bfd profile BFDfast interval multiplier '5'
set protocols bfd profile BFDfast interval receive '300'
set protocols bfd profile BFDfast interval transmit '300'

#Announce the dummy0 address
set protocols bgp address-family ipv4-unicast network 10.23.22.1/32

set protocols bgp address-family ipv6-unicast network fd00:f9a8:9a7e:399::1/128

#Setup the remote bgp neighbors in the wireguard network
set protocols bgp neighbor fd00:f9a8:9a7e:1::1 address-family ipv4-unicast nexthop-self
set protocols bgp neighbor fd00:f9a8:9a7e:1::1 address-family ipv4-unicast route-map export 'BGP_RMAP_PTH-out'
set protocols bgp neighbor fd00:f9a8:9a7e:1::1 address-family ipv4-unicast route-map import 'BGP_wg103-in'
set protocols bgp neighbor fd00:f9a8:9a7e:1::1 address-family ipv6-unicast nexthop-self
set protocols bgp neighbor fd00:f9a8:9a7e:1::1 address-family ipv6-unicast route-map export 'BGP_RMAP_PTH-out'
set protocols bgp neighbor fd00:f9a8:9a7e:1::1 address-family ipv6-unicast route-map import 'BGP_wg103-in'
set protocols bgp neighbor fd00:f9a8:9a7e:1::1 ebgp-multihop '1'
set protocols bgp neighbor fd00:f9a8:9a7e:1::1 remote-as '65001'
set protocols bgp neighbor fd00:f9a8:9a7e:1::1 timers connect '10'
set protocols bgp neighbor fd00:f9a8:9a7e:1::1 timers holdtime '300'
set protocols bgp neighbor fd00:f9a8:9a7e:1::1 timers keepalive '60'
set protocols bgp neighbor fd00:f9a8:9a7e:2::1 address-family ipv4-unicast nexthop-self
set protocols bgp neighbor fd00:f9a8:9a7e:2::1 address-family ipv6-unicast nexthop-self
set protocols bgp neighbor fd00:f9a8:9a7e:2::1 ebgp-multihop '1'
set protocols bgp neighbor fd00:f9a8:9a7e:2::1 remote-as '65002'
set protocols bgp neighbor fd00:f9a8:9a7e:300::2 bfd check-control-plane-failure
set protocols bgp neighbor fd00:f9a8:9a7e:300::2 bfd profile 'BFDfast'
set protocols bgp neighbor fd00:f9a8:9a7e:300::2 ebgp-multihop '1'
set protocols bgp neighbor fd00:f9a8:9a7e:300::2 peer-group 'PTH_PEERGROUP'
set protocols bgp neighbor fd00:f9a8:9a7e:300::2 timers connect '10'
set protocols bgp neighbor fd00:f9a8:9a7e:300::2 timers holdtime '30'
set protocols bgp neighbor fd00:f9a8:9a7e:300::2 timers keepalive '10'
set protocols bgp neighbor fd00:f9a8:9a7e:301::2 ebgp-multihop '1'
set protocols bgp neighbor fd00:f9a8:9a7e:301::2 peer-group 'PTH_PEERGROUP'
set protocols bgp neighbor fd00:f9a8:9a7e:301::2 timers connect '10'
set protocols bgp neighbor fd00:f9a8:9a7e:301::2 timers holdtime '300'
set protocols bgp neighbor fd00:f9a8:9a7e:301::2 timers keepalive '60'
set protocols bgp neighbor fd00:f9a8:9a7e:302::2 bfd check-control-plane-failure
set protocols bgp neighbor fd00:f9a8:9a7e:302::2 ebgp-multihop '1'
set protocols bgp neighbor fd00:f9a8:9a7e:302::2 peer-group 'PTH_PEERGROUP'
set protocols bgp neighbor fd00:f9a8:9a7e:302::2 timers connect '10'
set protocols bgp neighbor fd00:f9a8:9a7e:302::2 timers holdtime '30'
set protocols bgp neighbor fd00:f9a8:9a7e:302::2 timers keepalive '10'

set protocols bgp parameters bestpath bandwidth 'skip-missing'
set protocols bgp parameters fast-convergence
set protocols bgp parameters graceful-restart stalepath-time '5'
#I put the neighbors in a peer group so common settings are set here
set protocols bgp peer-group PTH_PEERGROUP address-family ipv4-unicast nexthop-self
set protocols bgp peer-group PTH_PEERGROUP address-family ipv4-unicast route-map export 'BGP_RMAP_PTH-out'
set protocols bgp peer-group PTH_PEERGROUP address-family ipv4-unicast route-map import 'BGP_RMAP_PTH-in'
set protocols bgp peer-group PTH_PEERGROUP address-family ipv6-unicast nexthop-self
set protocols bgp peer-group PTH_PEERGROUP address-family ipv6-unicast route-map export 'BGP_RMAP_PTH-out'
set protocols bgp peer-group PTH_PEERGROUP address-family ipv6-unicast route-map import 'BGP_RMAP_PTH-in'
set protocols bgp peer-group PTH_PEERGROUP remote-as '65100'
set protocols bgp system-as '65000'

set system task-scheduler task wireguard-watchdog executable arguments '400'
set system task-scheduler task wireguard-watchdog executable path '/config/scripts/reset-vbond.sh'

contents of /config/scripts/reset-vbond.sh
This script will scan the peers of each wireguard interface and if the last handshake is longer than specifid "400 seconds" as configured above, will turn off and on the interface.
This is mandatory to ensure the tunnels can comeback up if the openwrt client does not have access to ntp when it reboots until the tunnels come up

#!/usr/bin/bash
IFS=$'\n' 

dead=$1

for iface in $(sudo wg show | grep interface | awk '{print $2}')
do
  for handshake in $(sudo wg show ${iface} latest-handshakes)
  do
    peer=$(echo ${handshake} | awk '{print $1}')
    handshake=$(sudo wg show ${iface} latest-handshakes | grep ${peer})
    time=$(echo ${handshake} | awk '{print $2}')
    if [[ ${time} == 0 ]]; then break;fi
    age=$(expr $(date +'%s') - ${time} )
    if [[ ${age} -gt ${dead} ]]; then
      echo "Resetting wireguard if ${iface}"
      vbash -c "
source /opt/vyatta/etc/functions/script-template
configure

set interface wireguard ${iface} disable
sleep 5
sudo python3 << END
import sys
sys.path.insert(1, '/usr/lib/python3/dist-packages/vyos/utils')
from commit import wait_for_commit_lock
wait_for_commit_lock()
END

sleep 5
commit
sleep 5

sudo ip link del dev ${iface}
sleep 5

del interface wireguard ${iface} disable
sleep 5
sudo python3 << END                                                                     
import sys                                                                              
sys.path.insert(1, '/usr/lib/python3/dist-packages/vyos/utils')                           
from commit import wait_for_commit_lock
wait_for_commit_lock()
END
sleep 5
commit"
    fi
  done
done

openwrt config

#Network interfaces
#Note ,the route tables for each wan must be different and are important

config interface 'wan'
	option device 'eth2'
	option proto 'dhcp'
	option peerdns '0'
	option ip4table '54'
	option ip6table '54'
	option delegate '0'
	option hostname 'asdfioasoiasdf'
	option broadcast '1'

config interface 'wwan0'
	option proto 'dhcp'
	option device 'eth3'
	option peerdns '0'
	option ip4table '55'
	option ip6table '55'
	option delegate '0'

config interface 'wwan1'
	option proto 'dhcp'
	option peerdns '0'
	option ip4table '56'
	option ip6table '56'
	option hostname 'Pithos'
	option device 'br-boondock'


config interface 'wwan0v6'
	option proto 'dhcpv6'
	option device '@wwan0'
	option reqaddress 'try'
	option reqprefix 'auto'
	option ip4table '55'
	option ip6table '55'
	option delegate '0'

config interface 'wanv6'
	option proto 'dhcpv6'
	option device '@wan'
	option reqaddress 'try'
	option reqprefix 'auto'
	option ip4table '54'
	option ip6table '54'
	option delegate '0'
	option clientid 'FE33AB10'


config interface 'dummy'
	option proto 'static'
	option device 'dummy0'
	list ipaddr '10.23.32.1/32'
	list ip6addr 'fd00:f9a8:9a7e:399::2/128'

config interface 'vbond0'
	option proto 'wireguard'
	option private_key 'keyhere'
	option nohostroute '1'
	option peerdns '0'
	option ip4table '54'
	option ip6table '54'
	option delegate '0'
	option fwmark '0x34dd'
	list addresses '10.33.23.2/30'
	list addresses 'fd00:f9a8:9a7e:300::2/64'
	list addresses 'fe80::6331:aabc:dd31:2e9e/64'
	option mtu '1340'
	option defaultroute '0'

config wireguard_vbond0
	option description 'nvrnsdwangw01-wan'
	option public_key 'keyhere'
	option preshared_key 'keyhere'
	list allowed_ips '0.0.0.0/0'
	list allowed_ips '::/0'
	option endpoint_host 'cloud router ip'
	option endpoint_port '9468'
	option persistent_keepalive '60'

config interface 'vbond1'
	option proto 'wireguard'
	option private_key 'keyhere'
	option nohostroute '1'
	option defaultroute '0'
	option peerdns '0'
	option ip4table '55'
	option ip6table '55'
	option fwmark '0x34ee'
	option delegate '0'
	option mtu '1340'
	list addresses '10.33.23.6/30'
	list addresses 'fd00:f9a8:9a7e:301::2/64'
	list addresses 'fe80::300d:af96:223c:1ad2/64'

config wireguard_vbond1
	option description 'nvrnsdwangw01-wwan'
	option public_key 'keyhere'
	option preshared_key 'keyhere'
	list allowed_ips '0.0.0.0/0'
	list allowed_ips '::/0'
	option endpoint_host 'cloud router ip'
	option endpoint_port '9469'
	option persistent_keepalive '60'

config interface 'vbond2'
	option proto 'wireguard'
	option private_key 'keyhere'
	option nohostroute '1'
	option defaultroute '0'
	option peerdns '0'
	option ip4table '56'
	option ip6table '56'
	option fwmark '0x34ff'
	option delegate '0'
	list addresses '10.33.23.10/30'
	list addresses 'fd00:f9a8:9a7e:302::2/64'
	list addresses 'fe80::dde9:2434:26fa:2bc0/64'
	option mtu '1340'

config wireguard_vbond2
	option public_key 'keyhere'
	option preshared_key 'keyhere'
	option description 'nvrnsdwangw01'
	list allowed_ips '0.0.0.0/0'
	list allowed_ips '::/0'
	option endpoint_host 'cloud router ip'
	option endpoint_port '443'
	option persistent_keepalive '25'

#These marks associate wireguard marks with the table you associate with an ISP.
#The lookup must match the routing table specified in the ISP interface above
config rule
	option lookup '54'
	option mark '0x34dd'

config rule
	option mark '0x34ee'
	option lookup '55'

config rule
	option lookup '56'
	option mark '0x34ff'

config rule
	option dest '192.168.100.1/32'
	option lookup '54'

config rule
	option dest '192.168.1.1/32'
	option lookup '54'

config rule
	option dest '192.168.117.1/32'
	option lookup '55'

#This tries to put local traffic on the default table where our default route lives
config rule
	option in 'loopback'
	option lookup 'default'


#@@Firewall

config zone
	option input 'DROP'
	option name 'wan'
	option output 'DROP'
	option log '1'
	option forward 'REJECT'
	list network 'wan'
	list network 'wwan0'
	list network 'wwan1'
	list network 'wwan0v6'
	list network 'wanv6'

config zone
	option name 'sdwan_bond'
	option output 'ACCEPT'
	option masq_allow_invalid '1'
	option mtu_fix '1'
	option log '1'
	option forward 'ACCEPT'
	option input 'REJECT'
	list network 'dummy'
	list network 'vbond0'
	list network 'vbond1'
	list network 'vbond2'

config nat
	option name 'NAT to LTE modem'
	list proto 'all'
	option src 'wan'
	option dest_ip '192.168.117.1'
	option target 'MASQUERADE'

config nat
	option name 'NAT to starlink'
	list proto 'all'
	option src 'wan'
	option dest_ip '192.168.100.1'
	option target 'MASQUERADE'


#Put these near the top of /etc/config/firewall
config rule
	option direction 'out'
	option name 'Block sdwan wan'
	option dest 'wan'
#Note this mark matches the vbond config associated with the device
	option mark '0x34dd'
	option target 'ACCEPT'
	option device 'eth2'
	list proto 'all'

#These rules link a wireguard mark to a specific interface, ensuring the connections are only allowed out the ISP interface they should be
config rule
	option direction 'out'
	option name 'Block sdwan wwan0'
	option dest 'wan'
	option target 'ACCEPT'
#Note this mark matches the vbond config associated with the device
	option mark '0x34ee'
	option device 'eth3'
	list proto 'all'

config rule
	option direction 'out'
	option device 'wwan2'
	option target 'ACCEPT'
#Note this mark matches the vbond config associated with the device
	option mark '0x34ff'
	option dest 'wan'
	option name 'Block sdwan wwan2'
	list proto 'all'

config rule
	option name 'BFD'
	option src 'sdwan_bond'
	option target 'ACCEPT'
	option dest_port '4784 3784 3785'

config rule
	option name 'BGP WAN'
	list proto 'tcp'
	option src 'sdwan_bond'
	option dest_port '179'
	option target 'ACCEPT'

config rule
	option name 'Allow access to starlink and lte router '
	list proto 'tcp'
	option src 'lan'
	option dest 'wan'
	option target 'ACCEPT'
	list dest_ip '192.168.100.1'
	list dest_ip '192.168.117.1'

#Prevents wireguard tunnels from connecting within an existing wireguard tunnel
config rule
	list proto 'udp'
	option target 'REJECT'
	option dest 'sdwan_bond'
	option name 'Block tunnel nesting'
	option dest_port '443 9468 9469 9470'
	list dest_ip 'cloud router ip'



#@@FRR

#/etc/frr/daemons
#turn on bgpd and bfdd

#/etc/frr/frr.conf
#We want all traffic going over the SDWAN interfaces, these set the dummy0 interface on the cloud router as the default gateway. if the wireguard infrerfaces are down, there's no route out
ip route 0.0.0.0/0 10.23.22.1 table 254        
ipv6 route ::/0 fd00:f9a8:9a7e:399::1 table 254

router bgp 65100                                               
 no bgp ebgp-requires-policy                                   
 no bgp hard-administrative-reset                              
 no bgp default ipv4-unicast                                   
 bgp graceful-restart stalepath-time 5                         
 no bgp graceful-restart notification                          
 bgp bestpath as-path multipath-relax                          
 bgp bestpath bandwidth default-weight-for-missing             
 no bgp network import-check                                   
 neighbor CS1_PEERGROUP peer-group                             
 neighbor CS1_PEERGROUP remote-as 65000                        
 neighbor fd00:f9a8:9a7e:300::1 peer-group CS1_PEERGROUP       
 neighbor fd00:f9a8:9a7e:300::1 bfd                            
 neighbor fd00:f9a8:9a7e:300::1 bfd profile BFDfast            
 neighbor fd00:f9a8:9a7e:300::1 bfd check-control-plane-failure
 neighbor fd00:f9a8:9a7e:300::1 interface vbond0               
 neighbor fd00:f9a8:9a7e:300::1 timers 10 30                   
 neighbor fd00:f9a8:9a7e:300::1 timers connect 10              
 neighbor fd00:f9a8:9a7e:301::1 peer-group CS1_PEERGROUP       
 neighbor fd00:f9a8:9a7e:301::1 interface vbond1               
 neighbor fd00:f9a8:9a7e:301::1 timers 0 300                   
 neighbor fd00:f9a8:9a7e:301::1 timers connect 10              
 neighbor fd00:f9a8:9a7e:302::1 peer-group CS1_PEERGROUP       
 neighbor fd00:f9a8:9a7e:302::1 bfd                            
 neighbor fd00:f9a8:9a7e:302::1 bfd check-control-plane-failure
 neighbor fd00:f9a8:9a7e:302::1 interface vbond2               
 neighbor fd00:f9a8:9a7e:302::1 timers 10 30                   
 neighbor fd00:f9a8:9a7e:302::1 timers connect 10              
 neighbor fd00:f9a8:9a7e:aedc::1 remote-as 65001               
 neighbor fd00:f9a8:9a7e:aedc::1 passive                       
 neighbor fd00:f9a8:9a7e:aedc::1 timers delayopen 5            
 bgp fast-convergence                                          
 !                                                             
 address-family ipv4 unicast                                              
  network 10.23.32.1/32                        
#also specify your LAN networks so the cloud router can route traffic replies                 
  neighbor CS1_PEERGROUP activate                              
  neighbor CS1_PEERGROUP next-hop-self                         
  neighbor CS1_PEERGROUP route-map BGP_vbond-in in             
  neighbor fd00:f9a8:9a7e:aedc::1 activate                     
  neighbor fd00:f9a8:9a7e:aedc::1 next-hop-self                
 exit-address-family                                           
 !                                                             
 address-family ipv6 unicast                                  
  network fd00:f9a8:9a7e:399::2/128            
#also specify your LAN networks so the cloud router can route traffic replies             
  neighbor CS1_PEERGROUP activate                              
  neighbor CS1_PEERGROUP next-hop-self                         
  neighbor CS1_PEERGROUP route-map BGP_vbond-in in             
  neighbor fd00:f9a8:9a7e:aedc::1 activate                     
  neighbor fd00:f9a8:9a7e:aedc::1 next-hop-self                
 exit-address-family                                           
exit

#Here are the matching route local-preferences to ensure the starlink network has priority over the cellular. These are matched by neighbor IP associated with a particular wireguard tunnel
route-map BGP_vbond-in permit 20                               
 match peer fd00:f9a8:9a7e:301::1                              
 set extcommunity bandwidth 5                                  
 set local-preference 105                                      
exit                                                           
!                                                              
route-map BGP_vbond-in permit 30                               
 match peer fd00:f9a8:9a7e:302::1                              
 set extcommunity bandwidth 50                                 
 set local-preference 150                                      
exit  

bfd                                                            
 profile BFDfast                                               
  detect-multiplier 5                                          
 exit                                                          
 !                                                             
exit
8

I have a question:
Did you consider MPTCP during the evaluation phase of a potential solution? If so, what were the drawbacks or other reasons for not taking that path?

9

I considered it briefly and did not follow that path because a) i dont think its mature enough, b) i wanted to use the same multipath tunnel for my site to site traffic b) i wanted multipath for all protocols, tcp udp icmp, etc... and wireguard udp encapsulated packets have the lowest protocol overhead of any vpn to mu understanding

1 Like
10

Hello there, this is an interesting project that is along the lines of what i'd like to do myself. of particular interest to me is the bonus section where you used a "cheap" 5G phone as a cellular link. were you able to implement this approach to adding a cellular link to your setup ? if so:

  • what wide band antennas did you use ?
  • did you confirm that the red and white connectors on the S20 5G FE are LTE/5G ?
  • can i expect this approach of adding an external antenna to 5G phones to work on any phone that has at least two coaxial connectors on its motherboard ?

Thank you.

11

If it appears in openwrt as an internet route you can use it as described. The secret sauce is putting each internet link it it's own routing table, and pinning the wireguard mark'd packets to that routing table.

I ended up just getting a pay as you go service and getting a netgear nighthawk router, because the easytether openwrt isn't maintained anymore. But i'm sure it still "works"
That being said, when I did use it, it worked pretty well, but the below changes

add this to rc.local to reset the phone adb usb on reboot, 1-1 is the usb controller-port your phone is on (needs trial and error ie: echo /sys/bus/usb/devices/1-1/vendor )

sh -c 'sleep 60; echo 1 > /sys/bus/usb/devices/1-1/remove'

here's the scripts I've dug up
hotplug modification

root@npancmgw01:~# cat /etc/hotplug.d/usb/99-easytether-usb
#!/bin/sh

[ "$ACTION" = "add" ] || exit 0

[ "$INTERFACE" = "255/66/1" ] && {
        pgrep -f launch-easytether | xargs kill -9 
        killall -9 easytether-usb
        /root/launch-easytether.sh > /dev/null &
}

script

root@npancmgw01:~# cat /root/launch-easytether.sh
#!/bin/sh

while true;do
    
    if $(pgrep easytether-usb > /dev/null) ; then
echo null
      else
     /usr/bin/easytether-usb -n -s <device serial number>
    fi
    sleep 60
 done

<device serial number> can be discovered by just running easytether-usb and noticing which device ID it outputs

As far as your other questions, since I'm not using this anymore what you need to do is look for the pictures in FCC filings that document the internals of the device you choose to use, S20 5G FE was what I used (keep in mind "5G" is just a signal management improvement and the bands are mostly the same as LTE (think marketing over engineering)

honestly, you probably could get good service from just leaving the phone as is, I just wanted to be able to run long SMA coax to a better location for the antennas, and since it was a short term experiment, I don't even know if there was a significant change in SNR be it + or -

1 Like
12

that was quick! Thank you for getting back to me so fast with a detailed post.

i was thinking of using a usb to ethernet adapter and then using ethernet tethering in the settings to connect the phone to a router so i think easy tether will not be needed. that being said, Thank you for posting the scripts needed to make easy tether work well.

the tip on looking through the FCC website is golden, i'll be taking a visit there to see if the devices i'm interested in have been documented. on 5G, i guess 4.5G was not catchy enough for marketing.

it would have been great to have had some SNR numbers comparing internal to external antennas. i could explore this avenue myself and see if the modifications are worth the effort. can you point me in the direction of the antennas that you were using ?

13

The reason easy tether is attractive is because the data uses the phone channel as an app, wherass ethernet tethering uses hotspot channel which most "unlimited" plans limit or block

Obviously rooring the phone is an option to put hotspot data on the phone channel, but easy tether works without rooting.

14

that is interesting and useful information to know/have. in my case i would not mind rooting the device in order to bypass these limits/blocks. though, where i'm from, i do not believe that our providers limit or block hotspot data so ethernet tethering is a workable option.

i appreciate your insights, they are of great help in my planning process and i have learnt a thing or two from our exchange. Thank you very much.