Round-robin between two VPNs with nftables

Hi!

I want to send all traffik through two VPNs in round-robin way.
I experimented on clean VirtualBox's Ubuntu and it worked. Here are the settings:

Firewall:

nft list ruleset
table ip filter {
	chain INPUT {
		type filter hook input priority filter; policy accept;
	}

	chain OUTPUT {
		type filter hook output priority filter; policy accept;
	}

	chain FORWARD {
		type filter hook forward priority filter; policy accept;
	}
}
table ip mangle {
	chain PREROUTING {
		type filter hook prerouting priority mangle; policy accept;
	}

	chain POSTROUTING {
		type filter hook postrouting priority mangle; policy accept;
	}

	chain OUTPUT {
		type route hook output priority mangle; policy accept;
		meta l4proto tcp ip daddr 146.70.116.194 counter packets 34010 bytes 3602368 return
		meta l4proto tcp ip daddr 103.108.231.98 counter packets 77971 bytes 10947426 return
		ct state new  counter packets 4839 bytes 405737 meta mark set 0x2 
		ct state new mark 0x0 counter packets 4828 bytes 409387 meta mark set 0x1 
		ct state new counter packets 9528 bytes 802176 ct mark set mark
		ct state related,established counter packets 33365 bytes 3373279 meta mark set ct mark
		ct state new mark 0x1 counter packets 387 bytes 29090 log prefix "marked1:" level debug
		ct state new mark 0x2 counter packets 427 bytes 40959 log prefix "marked2:" level debug
	}
}
table ip nat {
	chain POSTROUTING {
		type nat hook postrouting priority srcnat; policy accept;
		oifname "tun1" counter packets 809 bytes 67250 masquerade 
		oifname "tun0" counter packets 829 bytes 69763 masquerade 
	}
}

Route tables:

echo "200 vpn0" >> /etc/iproute2/rt_tables
echo "201 vpn1" >> /etc/iproute2/rt_tables

Table defaults:

ip route add default dev tun0 table vpn0
ip route add default dev tun1 table vpn1

Rules:

ip rule add from all fwmark 1 lookup vpn0
ip rule add from all fwmark 2 lookup vpn1

Thus, we mark NEW connections with 1 or 2, based on the mark we send the packets to tun0 or tun1 and keep the marks for packets of ESTABLISHED,RELATED connections. We also add source NAT for two VPNs interfaces.

Now I want to transfer this setup to OpenWRT router. I want all the traffik be distributed to two VPNs running on the router.

I tried just to copy-paste ct settings from my experiment to router's firewall, changing the OUTPUT chain to PREROUTING chain but that did not work: the connection from my laptop to internet through the router is lost.

table inet fw4 {
	ct helper amanda {
		type "amanda" protocol udp
		l3proto inet
	}

	ct helper RAS {
		type "RAS" protocol udp
		l3proto inet
	}

	ct helper Q.931 {
		type "Q.931" protocol tcp
		l3proto inet
	}

	ct helper irc {
		type "irc" protocol tcp
		l3proto ip
	}

	ct helper pptp {
		type "pptp" protocol tcp
		l3proto ip
	}

	ct helper sip {
		type "sip" protocol udp
		l3proto inet
	}

	ct helper snmp {
		type "snmp" protocol udp
		l3proto ip
	}

	ct helper tftp {
		type "tftp" protocol udp
		l3proto inet
	}

	chain input {
		type filter hook input priority filter; policy drop;
		iifname "lo" accept comment "!fw4: Accept traffic from loopback"
		ct state established,related accept comment "!fw4: Allow inbound established and related flows"
		tcp flags syn / fin,syn,rst,ack jump syn_flood comment "!fw4: Rate limit TCP syn packets"
		iifname "br-lan" jump input_lan comment "!fw4: Handle lan IPv4/IPv6 input traffic"
		iifname { "tun0", "eth1", "tun1" } jump input_wan comment "!fw4: Handle wan IPv4/IPv6 input traffic"
		jump handle_reject
	}

	chain forward {
		type filter hook forward priority filter; policy drop;
		ct state established,related accept comment "!fw4: Allow forwarded established and related flows"
		iifname "br-lan" jump forward_lan comment "!fw4: Handle lan IPv4/IPv6 forward traffic"
		iifname { "tun0", "eth1", "tun1" } jump forward_wan comment "!fw4: Handle wan IPv4/IPv6 forward traffic"
		jump upnp_forward comment "Hook into miniupnpd forwarding chain"
		jump handle_reject
	}

	chain output {
		type filter hook output priority filter; policy accept;
		oifname "lo" accept comment "!fw4: Accept traffic towards loopback"
		ct state established,related accept comment "!fw4: Allow outbound established and related flows"
		oifname "br-lan" jump output_lan comment "!fw4: Handle lan IPv4/IPv6 output traffic"
		oifname { "tun0", "eth1", "tun1" } jump output_wan comment "!fw4: Handle wan IPv4/IPv6 output traffic"
	}

	chain prerouting {
		type filter hook prerouting priority filter; policy accept;
		iifname "br-lan" jump helper_lan comment "!fw4: Handle lan IPv4/IPv6 helper assignment"
	}

	chain handle_reject {
		meta l4proto tcp reject with tcp reset comment "!fw4: Reject TCP traffic"
		reject comment "!fw4: Reject any other traffic"
	}

	chain syn_flood {
		limit rate 25/second burst 50 packets return comment "!fw4: Accept SYN packets below rate-limit"
		drop comment "!fw4: Drop excess packets"
	}

	chain input_lan {
		jump accept_from_lan
	}

	chain output_lan {
		jump accept_to_lan
	}

	chain forward_lan {
		jump accept_to_wan comment "!fw4: Accept lan to wan forwarding"
		jump accept_to_lan
	}

	chain helper_lan {
		udp dport 10080 ct helper set "amanda" comment "!fw4: Amanda backup and archiving proto"
		udp dport 1719 ct helper set "RAS" comment "!fw4: RAS proto tracking"
		tcp dport 1720 ct helper set "Q.931" comment "!fw4: Q.931 proto tracking"
		meta nfproto ipv4 tcp dport 6667 ct helper set "irc" comment "!fw4: IRC DCC connection tracking"
		meta nfproto ipv4 tcp dport 1723 ct helper set "pptp" comment "!fw4: PPTP VPN connection tracking"
		udp dport 5060 ct helper set "sip" comment "!fw4: SIP VoIP connection tracking"
		meta nfproto ipv4 udp dport 161 ct helper set "snmp" comment "!fw4: SNMP monitoring connection tracking"
		udp dport 69 ct helper set "tftp" comment "!fw4: TFTP connection tracking"
	}

	chain accept_from_lan {
		iifname "br-lan" counter packets 4813 bytes 853862 accept comment "!fw4: accept lan IPv4/IPv6 traffic"
	}

	chain accept_to_lan {
		oifname "br-lan" counter packets 1935 bytes 452555 accept comment "!fw4: accept lan IPv4/IPv6 traffic"
	}

	chain input_wan {
		meta nfproto ipv4 udp dport 68 counter packets 0 bytes 0 accept comment "!fw4: Allow-DHCP-Renew"
		icmp type echo-request counter packets 0 bytes 0 accept comment "!fw4: Allow-Ping"
		meta nfproto ipv4 meta l4proto igmp counter packets 177 bytes 6372 accept comment "!fw4: Allow-IGMP"
		meta nfproto ipv6 udp dport 546 counter packets 12 bytes 2508 accept comment "!fw4: Allow-DHCPv6"
		ip6 saddr fe80::/10 icmpv6 type . icmpv6 code { mld-listener-query . no-route, mld-listener-report . no-route, mld-listener-done . no-route, mld2-listener-report . no-route } counter packets 179 bytes 13604 accept comment "!fw4: Allow-MLD"
		icmpv6 type { destination-unreachable, time-exceeded, echo-request, echo-reply, nd-router-solicit, nd-router-advert } limit rate 1000/second counter packets 1275 bytes 174336 accept comment "!fw4: Allow-ICMPv6-Input"
		icmpv6 type . icmpv6 code { packet-too-big . no-route, parameter-problem . no-route, nd-neighbor-solicit . no-route, nd-neighbor-advert . no-route, parameter-problem . admin-prohibited } limit rate 1000/second counter packets 1876 bytes 127848 accept comment "!fw4: Allow-ICMPv6-Input"
		jump reject_from_wan
	}

	chain output_wan {
		jump accept_to_wan
	}

	chain forward_wan {
		icmpv6 type { destination-unreachable, time-exceeded, echo-request, echo-reply } limit rate 1000/second counter packets 0 bytes 0 accept comment "!fw4: Allow-ICMPv6-Forward"
		icmpv6 type . icmpv6 code { packet-too-big . no-route, parameter-problem . no-route, parameter-problem . admin-prohibited } limit rate 1000/second counter packets 0 bytes 0 accept comment "!fw4: Allow-ICMPv6-Forward"
		meta l4proto esp counter packets 0 bytes 0 jump accept_to_lan comment "!fw4: Allow-IPSec-ESP"
		udp dport 500 counter packets 0 bytes 0 jump accept_to_lan comment "!fw4: Allow-ISAKMP"
		jump reject_to_wan
	}

	chain accept_to_wan {
		meta nfproto ipv4 oifname { "tun0", "eth1", "tun1" } ct state invalid counter packets 100 bytes 4684 drop comment "!fw4: Prevent NAT leakage"
		oifname { "tun0", "eth1", "tun1" } counter packets 30662 bytes 10699870 accept comment "!fw4: accept wan IPv4/IPv6 traffic"
	}

	chain reject_from_wan {
		iifname { "tun0", "eth1", "tun1" } counter packets 3269 bytes 301877 jump handle_reject comment "!fw4: reject wan IPv4/IPv6 traffic"
	}

	chain reject_to_wan {
		oifname { "tun0", "eth1", "tun1" } counter packets 0 bytes 0 jump handle_reject comment "!fw4: reject wan IPv4/IPv6 traffic"
	}

	chain dstnat {
		type nat hook prerouting priority dstnat; policy accept;
		jump upnp_prerouting comment "Hook into miniupnpd prerouting chain"
	}

	chain srcnat {
		type nat hook postrouting priority srcnat; policy accept;
		oifname { "tun0", "eth1", "tun1" } jump srcnat_wan comment "!fw4: Handle wan IPv4/IPv6 srcnat traffic"
		jump upnp_postrouting comment "Hook into miniupnpd postrouting chain"
	}

	chain srcnat_wan {
		meta nfproto ipv4 masquerade comment "!fw4: Masquerade IPv4 wan traffic"
	}

	chain raw_prerouting {
		type filter hook prerouting priority raw; policy accept;
	}

	chain raw_output {
		type filter hook output priority raw; policy accept;
	}

	chain mangle_prerouting {
		type filter hook prerouting priority mangle; policy accept;
	}

	chain mangle_postrouting {
		type filter hook postrouting priority mangle; policy accept;
	}

	chain mangle_input {
		type filter hook input priority mangle; policy accept;
	}

	chain mangle_output {
		type route hook output priority mangle; policy accept;
	}

	chain mangle_forward {
		type filter hook forward priority mangle; policy accept;
		iifname { "tun0", "eth1", "tun1" } tcp flags syn tcp option maxseg size set rt mtu comment "!fw4: Zone wan IPv4/IPv6 ingress MTU fixing"
		oifname { "tun0", "eth1", "tun1" } tcp flags syn tcp option maxseg size set rt mtu comment "!fw4: Zone wan IPv4/IPv6 egress MTU fixing"
	}

	chain upnp_forward {
	}

	chain upnp_prerouting {
	}

	chain upnp_postrouting {
	}

	chain input_VPN {
	}

	chain output_VPN {
	}

	chain forward_VPN {
	}

	chain accept_from_VPN {
	}

	chain accept_to_VPN {
	}

	chain srcnat_VPN {
	}
}
table ip filter {
	chain FORWARD {
		type filter hook forward priority filter; policy accept;
	}

	chain INPUT {
		type filter hook input priority filter; policy accept;
	}
}
table ip nat {
	chain POSTROUTING {
		type nat hook postrouting priority srcnat; policy accept;
	}
}
table ip mangle {
	chain PREROUTING {
		type filter hook prerouting priority mangle; policy accept;
		ct state new meta mark set 0x2 
		ct state new mark 0x0 meta mark set 0x1 
		ct state new ct mark set mark
		ct state related,established meta mark set ct mark
		ct state new mark 0x1 log prefix "marked1:" level debug
		ct state new mark 0x2 log prefix "marked2:" level debug
	}

	chain OUTPUT {
		type route hook output priority mangle; policy accept;
	}
}

Can you advice how can I adapt this solution to OpenWRT firewall?

Do you want to load balance the traffic or do policy based routing e.g. some traffic through tun1 other through tun2

For load balancing research mwan3, for pbr:

1 Like

Thanks, but I succeeded without mwan3 which i was unable to configure.