Bypassing Wireguard IPv6 default route for specific VLAN

anon_openwrt · February 24, 2022, 5:22am

I have a network comprised of multiple VLANs. Wireguard is set as default route and router can ping6 via the wireguard network. Clients on lan network bypass Wireguard and go to wan. I can do this IPv4 using PBR, and it works great.

In the past I had enabled NAT6 in order to share my Wireguard's /128 address via the ULA on the vpn network. To prevent leaks I also disabled ULA on lan (LAN clients only get GUA). At one point this all worked great, but since upgrading to snapshot (using firewall3/ipt), I can no longer route any (lan or vpn) IPv6 on clients.

uci export network:

package network

config globals 'globals'
	option ula_prefix 'ddc2:9aea:13b1::/48'

config interface 'loopback'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'
	option device 'lo'

config interface 'lan'
	option proto 'static'
	option netmask '255.255.255.0'
	option ipaddr '192.168.1.1'
	option ip6assign '64'
	option ip6ifaceid '::1'
	option ip6hint '1'
	option device 'br-lan'
	list ip6class 'wan6'

config interface 'wan'
	option proto 'dhcp'
	option device 'eth0'

config interface 'wan6'
	option proto 'dhcpv6'
	option reqaddress 'try'
	option reqprefix '56'
	option device 'eth0'
	option ip6table '2'

config interface 'wireguard'
	option proto 'wireguard'
	option private_key 'redacted
	list addresses '10.65.5.239/32'
	list addresses 'fc00:bbbb:redacted/128'

config wireguard_wireguard
	option persistent_keepalive '25'
	option public_key 'redacted'
	option endpoint_host 'redacted'
	option description 'vpn'
	list allowed_ips '0.0.0.0/0'
	list allowed_ips '::0/0'
	option endpoint_port '51820'
	option route_allowed_ips '1'

config interface 'vpn'
	option proto 'static'
	option ipaddr '192.168.2.1'
	option netmask '255.255.255.0'
	list dns 'redacted'
	option device 'br-vpn'
	option ip6ifaceid '::1'
	option ip6assign '64'
	option ip6hint '2'
	list ip6class 'local'

config interface 'dmz'
	option proto 'static'
	option ipaddr '192.168.3.1'
	option netmask '255.255.255.0'
	option ip6assign '64'
	option ip6ifaceid '::1'
	option ip6hint '3'
	option device 'br-dmz'

config interface 'iot'
	option proto 'static'
	option ipaddr '192.168.4.1'
	option ip6assign '64'
	option ip6hint '4'
	option ip6ifaceid '::1'
	option netmask '255.255.255.0'
	option device 'br-iot'

config device
	option name 'br-lan'
	option type 'bridge'
	option stp '1'
	list ports 'eth1'

config device
	option name 'br-vpn'
	option type 'bridge'
	list ports 'eth1.2'

config device
	option name 'br-dmz'
	option type 'bridge'
	list ports 'eth1.3'

config device
	option name 'br-iot'
	option type 'bridge'
	list ports 'eth1.4'

uci export firewall:

package firewall

config defaults
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option synflood_protect '1'

config zone
	option name 'lan'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'ACCEPT'
	list network 'dmz'
	list network 'lan'

config zone
	option name 'wan'
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option masq '1'
	option mtu_fix '1'
	list network 'wan'
	list network 'wan6'

config zone
	option name 'iot'
	option output 'ACCEPT'
	list network 'iot'
	option forward 'REJECT'
	option input 'ACCEPT'

config zone
	option name 'vpn'
	option output 'ACCEPT'
	list network 'vpn'
	option forward 'REJECT'
	option input 'ACCEPT'

config zone
	option name 'wireguard'
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option masq '1'
	option mtu_fix '1'
	list network 'wireguard'
	option masq6 '1'

config forwarding
	option src 'lan'
	option dest 'wan'

config forwarding
	option src 'lan'
	option dest 'vpn'

config forwarding
	option src 'lan'
	option dest 'iot'

config forwarding
	option src 'vpn'
	option dest 'lan'

config forwarding
	option src 'vpn'
	option dest 'wireguard'

config forwarding
	option src 'vpn'
	option dest 'iot'

config forwarding
	option src 'iot'
	option dest 'wireguard'

config forwarding
	option src 'iot'
	option dest 'wan'

config rule
	option name 'Allow-DHCP-Renew'
	option src 'wan'
	option proto 'udp'
	option dest_port '68'
	option target 'ACCEPT'
	option family 'ipv4'

config rule
	option name 'Allow-Ping'
	option src 'wan'
	option proto 'icmp'
	option icmp_type 'echo-request'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-IGMP'
	option src 'wan'
	option proto 'igmp'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-DHCPv6'
	option src 'wan'
	option proto 'udp'
	option src_ip 'fc00::/6'
	option dest_ip 'fc00::/6'
	option dest_port '546'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-MLD'
	option src 'wan'
	option proto 'icmp'
	option src_ip 'fe80::/10'
	list icmp_type '130/0'
	list icmp_type '131/0'
	list icmp_type '132/0'
	list icmp_type '143/0'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Input'
	option src 'wan'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	list icmp_type 'router-solicitation'
	list icmp_type 'neighbour-solicitation'
	list icmp_type 'router-advertisement'
	list icmp_type 'neighbour-advertisement'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Forward'
	option src 'wan'
	option dest '*'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'
	option enabled ''\''0'\'''

config rule
	option name 'Allow-IPSec-ESP'
	option src 'wan'
	option dest 'lan'
	option proto 'esp'
	option target 'ACCEPT'

config rule
	option name 'Allow-ISAKMP'
	option src 'wan'
	option dest 'lan'
	option dest_port '500'
	option proto 'udp'
	option target 'ACCEPT'

config include
	option path '/etc/firewall.user'

config include 'miniupnpd'
	option type 'script'
	option path '/usr/share/miniupnpd/firewall.include'
	option family 'any'
	option reload '1'

config include 'pbr'
	option type 'script'
	option path '/usr/share/pbr/pbr.firewall.include'
	option family 'any'
	option reload '1'

config include 'nat6'
	option path '/etc/firewall.nat6'
	option reload '1'

pbr

uci export pbr:

package pbr

config pbr 'config'
	option src_ipset '0'
	list supported_interface ''
	list ignored_interface 'vpnserver wgserver'
	option boot_timeout '30'
	option iptables_rule_option 'append'
	option iprule_enabled '0'
	option webui_chain_column '0'
	option webui_sorting '1'
	list webui_supported_protocol 'tcp'
	list webui_supported_protocol 'udp'
	list webui_supported_protocol 'tcp udp'
	list webui_supported_protocol 'icmp'
	list webui_supported_protocol 'all'
	option verbosity '1'
	option webui_enable_column '1'
	option webui_protocol_column '1'
	option dest_ipset '0'
	option webui_show_ignore_target '0'
	option resolver_ipset 'dnsmasq.ipset'
	option ipv6_enabled '1'
	option strict_enforcement '1'
	option enabled '1'

config policy
	option interface 'wan'
	option proto 'all'
	option src_addr '192.168.1.0/24'
	option name 'dmz4'

In this case the 192.168.1.0/24 subnet (lan) is routed to wan. Great! But I cannot seem to figure out how to use pbr to route lan IPv6 to wan6, every configuration of CIDR I've used for the GUAs on my lan segment fail with "unknown fw_mark." I've also read about using route6 to set up routes but I don't think it (or PBR?) are designed for routing based on GUAs (or their subnets).

Right now, I don't care about NAT6 or IPv6 in my vpn vlan. I just want to have IPv6 on my LAN working correctly (routed to wan6) as if the wireguard route were not involved. No matter what I try I cannot seem to accomplish this!

/etc/init.d/pbr status:

pbr 0.9.4-10 running on OpenWrt SNAPSHOT.
============================================================
Dnsmasq version 2.86  Copyright (c) 2000-2021 Simon Kelley
Compile time options: IPv6 GNU-getopt no-DBus UBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP conntrack ipset auth cryptohash DNSSEC no-ID loop-detect inotify dumpfile
============================================================
Routes/IP Rules
default         *               0.0.0.0         U     0      0        0 wireguard

IPv4 Table 201: default via 35.143.192.1 dev eth0 
192.168.1.0/24 dev br-lan proto kernel scope link src 192.168.1.1 
192.168.2.0/24 dev br-vpn proto kernel scope link src 192.168.2.1 
192.168.3.0/24 dev br-dmz proto kernel scope link src 192.168.3.1 
192.168.4.0/24 dev br-iot proto kernel scope link src 192.168.4.1 
IPv4 Table 201 Rules:
30000:	from all fwmark 0x10000/0xff0000 lookup wan

IPv4 Table 202: default via 10.65.5.239 dev wireguard 
192.168.1.0/24 dev br-lan proto kernel scope link src 192.168.1.1 
192.168.2.0/24 dev br-vpn proto kernel scope link src 192.168.2.1 
192.168.3.0/24 dev br-dmz proto kernel scope link src 192.168.3.1 
192.168.4.0/24 dev br-iot proto kernel scope link src 192.168.4.1 
IPv4 Table 202 Rules:
29999:	from all fwmark 0x20000/0xff0000 lookup wireguard
IPv6 Table 202: default dev wireguard proto static metric 1024 pref medium
============================================================
Mangle IP Table: PREROUTING
# Warning: iptables-legacy tables present, use iptables-legacy to see them
-N PBR_PREROUTING
-A PBR_PREROUTING -s 192.168.1.0/24 -m comment --comment dmz4 -c 0 0 -g PBR_MARK0x010000
============================================================
Mangle IPv6 Table: PREROUTING
-N PBR_PREROUTING
============================================================
Mangle IP Table MARK Chain: PBR_MARK0x010000
# Warning: iptables-legacy tables present, use iptables-legacy to see them
-N PBR_MARK0x010000
-A PBR_MARK0x010000 -c 2 134 -j MARK --set-xmark 0x10000/0xff0000
-A PBR_MARK0x010000 -c 2 134 -j RETURN
============================================================
Mangle IP Table MARK Chain: PBR_MARK0x020000
# Warning: iptables-legacy tables present, use iptables-legacy to see them
-N PBR_MARK0x020000
-A PBR_MARK0x020000 -c 0 0 -j MARK --set-xmark 0x20000/0xff0000
-A PBR_MARK0x020000 -c 0 0 -j RETURN
============================================================
Current ipsets
============================================================
Your support details have been logged to '/var/pbr-support'. [✓]

Any ideas re: the best method to get IPv6 working on my lan network?

trendy · February 24, 2022, 10:36am

Roll back to the stable version you had.

anon_openwrt · February 24, 2022, 4:19pm

@trendy that is definitely sound advice but I am too persistent to take it unfortunately. I just wish I knew which of these issues were "out of my hands" so-to-speak and not rectifiable with configuration.

I will try to break down my issues into smaller chunks and address them one-by-one.

anon_openwrt · February 24, 2022, 9:18pm

My first mistake was here:

list allowed_ips '::0/0'

I have changed this to the subnet of my vpn interface ddc2:9aea:13b1:2::1/64 and now I have IPv6 connectivity on my other interfaces again. This was the major problem I think I was running into so I will try to dive back into NAT6 now.

trendy · February 25, 2022, 9:50am

So this was not a problem of the upgrade itself, rather than misconfiguration.

anon_openwrt · February 25, 2022, 1:46pm

I had it working in the past with VPR using the wireguard route of '::0/0'. I think that the weighting has changed in some manner (or rules precedence) or PBR works in a different manner. I don't want to speculate as I'm not going to do side-by-side tests, so I'm just considering that restricting allowed_ips is the working solution in snapshot.

Unfortunately, I still haven't been able to get NAT6 to work on my vpn interface using the directions outlined in the NAT6 setup guide (this definitely used to work, and no longer does).

As a workaround, I'd like to try to set up some static routes (i.e. "manual" PBR), however the documentation in this area is not very detailed. Something like this perhaps:

config route6
	option interface 'wireguard'
	option target '::/0'
	option source 'ddc2:9aea:13b1:2::1/64'

...but that doesn't seem to work either when allowed_ips '::/0'. What both of these pieces of information tell me is that the default wireguard route seems to override PBR or static routes (or perhaps PBR and/or manual static routes are not being set at all?).

trendy · February 25, 2022, 2:38pm

In the wireguard tunnel you need definitely the ::/0 in the allowed networks. Otherwise it won't be able to send to that tunnel anything else other than packets destined to whatever address you have there. Also there must be a default route via the wg tunnel and it should be visible from PBR.
Then you need a policy for the ipv6 part in the PBR. In the initial post there is nothing.
If you want to try to manually do policy routing, you need a pair of rules and routes. However disable PBR first, as they will conflict.

anon_openwrt · February 25, 2022, 2:48pm

In the wireguard tunnel you need definitely the ::/0 in the allowed networks. Otherwise it won't be able to send to that tunnel anything else other than packets destined to whatever address you have there.

OK, after reading some more, this makes sense to me (although strangely it seemed to work fine, passing several remote IPv6 tests on clients in that subnet--so presumably a lot more than just my local subnet was accessible remotely-maybe this a clue?). In the meantime, I've reverted back to ::0/0 and will try addressing this with PBR or static routes/rules.

However, when I add a PBR rule to route IPv6 from my lan subnet to wan6:

config policy
	option name 'dmz6'
	option src_addr 'ddc2:9aea:13b1:1::1/64'
	option interface 'wan6'

...I just get an "Unknown fw_mark for wan6" error and no ipv6 route is added. Am I misunderstanding how to set PBR rules?

/etc/init.d/pbr restart:

# Warning: iptables-legacy tables present, use iptables-legacy to see them
pbr 0.9.4-10 stopped ✓
Processing Interfaces ✓✓
Processing Policies ✓✓✓✗
pbr 0.9.4-10 monitoring interfaces: wan wan6 wireguard 
pbr 0.9.4-10 started with gateways:
wan/eth0/redacted/2603:9000:redacted/128
fe80::4863:a6ff:fe2b:672a/64
wireguard/redacted/fc00:bbbb:redacted/128 [✓]
ERROR: Unknown fw_mark for wan6

Assuming that PBR is broken, I am fine setting this up manually as my rules are very simple (one network subnet to wireguard, the other to wan).

Is this close to being correct?

config rule
    option in 'lan'
    option src '192.168.1.0/24'
    option lookup '101'
    
config rule
    option in 'lan'
    option src 'ddc2:9aea:13b1:1::1/64'
    option lookup '102'
        
config rule
    option in 'vpn'
    option src '192.168.2.0/24'
    option lookup '103'

config rule
    option in 'vpn'
    option src 'ddc2:9aea:13b1:2::1/64'
    option lookup '104'

config route
    option interface 'wan'
    option target '0.0.0.0'
    option netmask '0.0.0.0'
    option metric '200'
    option table '101'
        
config route6
    option interface 'wan6'
    option target '::0/0'
    option metric '200'
    option table '102'
    
config route
    option interface 'wireguard'
    option target '0.0.0.0'
    option netmask '0.0.0.0'
    option metric '200'
    option table '103'
        
config route6
    option interface 'wireguard6'
    option target '::0/0'
    option metric '200'
    option table '104'

(wireguard6 does not exist, I will handle NAT6 tunneling on vpn in the future after I get lan correctly routed around and not through my wireguard tunnel. From here it looks like I can just target the wireguard interface if masq6 is enabled?)

trendy · February 26, 2022, 9:20am

This source address is wrong. If you want to match one address use ddc2:9aea:13b1:1::1/128. If you want to use the whole subnet, use ddc2:9aea:13b1:1::/64.

The IPv6 address seems to be on wan, not wan6.

Same as before.

And here. Fix the address.

anon_openwrt · February 26, 2022, 7:18pm

Thank you, sometimes the dumbest mistakes (specifying the device and not the subnet) cause the biggest headaches!

I have a couple of remaining questions before I move onto NAT6 (in a different thread).

These are my working default routes/rules for bypassing the default wireguard route without pbr:

config route
	option interface 'wan'
	option target '0.0.0.0/0'
	option table '1'
	option metric '200'

config route6
	option interface 'wan6'
	option target '::/0'
	option table '1'
	option metric '200'

config rule
	option in 'lan'
	option src '192.168.1.1/24'
	option lookup '1'

config rule6
	option in 'lan'
	option src 'ddc2:9aea:13b1:1::/64'
	option lookup '1'

Could I have instead assigned an ip4table and ip6table to wan and wan6 respectively, and then just used a rule/rule6 to lookup that table instead of specifying a separate route/table? I know that if I want granularity (not every client on the chosen interface) then I would need to make additional routes as well--can these just be appended to the same table?
Are any of these options unnecessary (do I need to specify target, metric, src, etc or will these be inherited from the interface). The src seems redundant since I imagine it would route every address from lan by default, which is what I intend. Possibly the same with target as well. And metric...is this the equivalent of "priority" for rules, but for routes?
Is it possible to use GUAs as src IPs so I don't need to assign ULA to lan?

trendy · February 27, 2022, 11:31am

I am not sure if you want that, as it will assign the route only to the particular routing table and not on main. If some device doesn't use any of the additional routing tables, it will be blackholed.

I think only the ingress interface is needed in rules, and you need to specify the lookup table to make it work.
In routes, metric is not necessary, unless you have multiple same routes.

As long as you nat them accordingly, I don't see why not.

anon_openwrt · February 28, 2022, 6:27pm

Thank you so much for your help, I've got a working setup here I wanted to share in case anyone finds this thread again.

After setting wireguard to route allowed ips and creating custom routes/rules for lan segment to bypass the wireguard tunnel, I lost connectivity to/from lan. In order to address this I added a couple of extra rules to put local IPv4/IPv6 lan traffic back onto the main table so it wasn't just sent to wan/wan6 by default:

config route
        option interface 'wan'
        option target '0.0.0.0/0'
        option table '1'

config route6
        option interface 'wan6'
        option target '::/0'
        option table '1'

config rule
        option in 'lan'
        option dest '192.168.0.0/16'
        option lookup 'main'

config rule6
        option in 'lan'
        option dest 'ddc2:9aea:13b1::/48'
        option table 'main'

config rule
        option in 'lan'
        #option src '192.168.1.1/24'
        option lookup '1'

config rule6
        option in 'lan'
        option lookup '1'

I haven't had any luck with NAT6 yet, but I'm still working on it. Thanks again!

anon_openwrt · February 28, 2022, 10:34pm

Now I've got NAT6 working on a specific VLAN network using firewall4!

This section was still a problem:

config route6
        option interface 'wan6'
        option target '::/0'
        option table '1'

Conceptually I understand why it is pointless to route GUA (wan clients will lose connectivity), but I thought that explicitly setting the default route would not cause issues, but it does. This makes sense because while packets may go through the correct interface, the firewall has "broken" the direct GUA between the client and ISP. So don't try to route un-NAT'd IPv6 (duh, now I get it!).

Here is what works when wireguard is the default route (i.e. option route_allowed_ips '1'):

config route
        option interface 'wan'
        option target '0.0.0.0/0'
        option table '10'

config rule
        option in 'lan'
        option dest '192.168.0.0/16'
        option lookup 'main'

config rule
        option in 'lan'
        option lookup '10'

config route6
        option interface 'wireguard'
        option target '::/0'
        option table '20'

config rule6
        option in 'vpn'
        option lookup '20'

Break-down

Create route for IPv4 wan traffic:

config route
        option interface 'wan'
        option target '0.0.0.0/0'
        option table '10'

Create rule to allow local lan traffic to remain on the main table so devices have access to other network segments:

config rule
        option in 'lan'
        option dest '192.168.0.0/16'
        option lookup 'main'

Send rest of IPv4 traffic on LAN to the wan route (and put this rule second so you don't need to specify priority):

config rule
        option in 'lan'
        option lookup '10'

Create route for IPv6 wireguard traffic:

config route6
        option interface 'wireguard'
        option target '::/0'
        option table '20'

Send all IPv6 traffic on vpn vlan to wireguard:

config rule6
        option in 'vpn'
        option lookup '20'

But what about LAN IPv6 and Wireguard IPv4?
As mentioned earlier, LAN IPv6 is handled by the GUA (yay IPv6, no rules required!).
Wireguard IPv4 is handled by option route_allowed_ips '1' on the main table.

I hope this helps!

system · March 10, 2022, 10:35pm

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.