WireGuard, No Internet Access

arrmo · January 29, 2021, 11:51pm

Hi,

OK, a follow-on to this one, as promised

I have a "server" and "client" (to make this easier to follow - I know they are considered peers). The two boxes are remote from each other. The server is pfSense, client is OpenWrt. The client is set up to route all traffic to the server - that sort of works ... I can get to the server, and devices on that subnet. But not from there out to the internet. I need this, because I'm routing all traffic from the client to the server.

On the client end, the info requested in the post noted above (some items removed to shorted the capture, but also some addresses removed),

ip address show; ip route show table all; ip rule show; iptables-save; wg show
6: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether b0:4e:26:3f:9f:30 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.1/24 brd 192.168.0.255 scope global br-lan
       valid_lft forever preferred_lft forever
    inet6 fd7b:819c:917f::1/60 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::b24e:26ff:fe3f:9f30/64 scope link
       valid_lft forever preferred_lft forever
7: eth0.1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-lan state UP qlen 1000
    link/ether b0:4e:26:3f:9f:30 brd ff:ff:ff:ff:ff:ff
8: eth0.2@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether b0:4e:26:3f:9f:31 brd ff:ff:ff:ff:ff:ff
    inet X.X.X.X/24 brd 128.189.183.255 scope global eth0.2
       valid_lft forever preferred_lft forever
    inet6 fe80::b24e:26ff:fe3f:9f31/64 scope link
       valid_lft forever preferred_lft forever
9: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN qlen 1000
    link/[65534]
    inet 192.168.253.3/32 brd 255.255.255.255 scope global wg0
       valid_lft forever preferred_lft forever
10: wlan1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-lan state UP qlen 1000
    link/ether b0:4e:26:3f:9f:30 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::b24e:26ff:fe3f:9f30/64 scope link
       valid_lft forever preferred_lft forever
11: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-lan state UP qlen 1000
    link/ether b0:4e:26:3f:9f:2f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::b24e:26ff:fe3f:9f2f/64 scope link
       valid_lft forever preferred_lft forever
default dev wg0 scope link
Y.Y.Y.Y via X.X.X.X dev eth0.2
128.189.183.0/24 dev eth0.2 scope link  src X.X.X.X
192.168.0.0/24 dev br-lan scope link  src 192.168.0.1
192.168.2.0/24 dev wg0 scope link
192.168.253.0/24 dev wg0 scope link
broadcast 127.0.0.0 dev lo table local scope link  src 127.0.0.1
local 127.0.0.0/8 dev lo table local scope host  src 127.0.0.1
local 127.0.0.1 dev lo table local scope host  src 127.0.0.1
broadcast 127.255.255.255 dev lo table local scope link  src 127.0.0.1
broadcast 128.189.183.0 dev eth0.2 table local scope link  src X.X.X.X
local X.X.X.X dev eth0.2 table local scope host  src X.X.X.X
broadcast 128.189.183.255 dev eth0.2 table local scope link  src X.X.X.X
broadcast 192.168.0.0 dev br-lan table local scope link  src 192.168.0.1
local 192.168.0.1 dev br-lan table local scope host  src 192.168.0.1
broadcast 192.168.0.255 dev br-lan table local scope link  src 192.168.0.1
local 192.168.253.3 dev wg0 table local scope host  src 192.168.253.3
fd7b:819c:917f::/64 dev br-lan  metric 1024
unreachable fd7b:819c:917f::/48 dev lo  metric 2147483647
fe80::/64 dev eth0  metric 256
fe80::/64 dev eth0.2  metric 256
fe80::/64 dev br-lan  metric 256
fe80::/64 dev wlan1  metric 256
fe80::/64 dev wlan0  metric 256
local ::1 dev lo table local  metric 0
anycast fd7b:819c:917f:: dev br-lan table local  metric 0
local fd7b:819c:917f::1 dev br-lan table local  metric 0
anycast fe80:: dev eth0 table local  metric 0
anycast fe80:: dev eth0.2 table local  metric 0
anycast fe80:: dev br-lan table local  metric 0
anycast fe80:: dev wlan1 table local  metric 0
anycast fe80:: dev wlan0 table local  metric 0
local fe80::b24e:26ff:fe3f:9f2f dev wlan0 table local  metric 0
local fe80::b24e:26ff:fe3f:9f30 dev eth0 table local  metric 0
local fe80::b24e:26ff:fe3f:9f30 dev br-lan table local  metric 0
local fe80::b24e:26ff:fe3f:9f30 dev wlan1 table local  metric 0
local fe80::b24e:26ff:fe3f:9f31 dev eth0.2 table local  metric 0
ff00::/8 dev eth0 table local  metric 256
ff00::/8 dev br-lan table local  metric 256
ff00::/8 dev eth0.2 table local  metric 256
ff00::/8 dev wg0 table local  metric 256
ff00::/8 dev wlan1 table local  metric 256
ff00::/8 dev wlan0 table local  metric 256
0:      from all lookup local
32766:  from all lookup main
32767:  from all lookup default
# Generated by iptables-save v1.8.6 on Fri Jan 29 15:43:37 2021
*nat
:PREROUTING ACCEPT [3746:620436]
:INPUT ACCEPT [709:48754]
:OUTPUT ACCEPT [582:42055]
:POSTROUTING ACCEPT [1034:97758]
:postrouting_lan_rule - [0:0]
:postrouting_rule - [0:0]
:postrouting_wan_rule - [0:0]
:prerouting_lan_rule - [0:0]
:prerouting_rule - [0:0]
:prerouting_wan_rule - [0:0]
:zone_lan_postrouting - [0:0]
:zone_lan_prerouting - [0:0]
:zone_wan_postrouting - [0:0]
:zone_wan_prerouting - [0:0]
-A PREROUTING -m comment --comment "!fw3: Custom prerouting rule chain" -j prerouting_rule
-A PREROUTING -i br-lan -m comment --comment "!fw3" -j zone_lan_prerouting
-A PREROUTING -i wg0 -m comment --comment "!fw3" -j zone_lan_prerouting
-A PREROUTING -i tun0 -m comment --comment "!fw3" -j zone_wan_prerouting
-A PREROUTING -i eth0.2 -m comment --comment "!fw3" -j zone_wan_prerouting
-A POSTROUTING -m comment --comment "!fw3: Custom postrouting rule chain" -j postrouting_rule
-A POSTROUTING -o br-lan -m comment --comment "!fw3" -j zone_lan_postrouting
-A POSTROUTING -o wg0 -m comment --comment "!fw3" -j zone_lan_postrouting
-A POSTROUTING -o tun0 -m comment --comment "!fw3" -j zone_wan_postrouting
-A POSTROUTING -o eth0.2 -m comment --comment "!fw3" -j zone_wan_postrouting
-A zone_lan_postrouting -m comment --comment "!fw3: Custom lan postrouting rule chain" -j postrouting_lan_rule
-A zone_lan_postrouting -s 192.168.0.0/24 -d 192.168.0.1/32 -p tcp -m tcp --dport 22 -m comment --comment "!fw3: ssh (reflection)" -j SNAT --to-source 192.168.0.1
-A zone_lan_postrouting -s 192.168.0.0/24 -d 192.168.0.1/32 -p udp -m udp --dport 22 -m comment --comment "!fw3: ssh (reflection)" -j SNAT --to-source 192.168.0.1
-A zone_lan_postrouting -s 192.168.253.3/32 -d 192.168.0.1/32 -p tcp -m tcp --dport 22 -m comment --comment "!fw3: ssh (reflection)" -j SNAT --to-source 192.168.253.3
-A zone_lan_postrouting -s 192.168.253.3/32 -d 192.168.0.1/32 -p udp -m udp --dport 22 -m comment --comment "!fw3: ssh (reflection)" -j SNAT --to-source 192.168.253.3
-A zone_lan_postrouting -s 192.168.0.0/24 -d 192.168.0.1/32 -p tcp -m tcp --dport 80 -m comment --comment "!fw3: http (reflection)" -j SNAT --to-source 192.168.0.1
-A zone_lan_postrouting -s 192.168.253.3/32 -d 192.168.0.1/32 -p tcp -m tcp --dport 80 -m comment --comment "!fw3: http (reflection)" -j SNAT --to-source 192.168.253.3
-A zone_lan_prerouting -m comment --comment "!fw3: Custom lan prerouting rule chain" -j prerouting_lan_rule
-A zone_lan_prerouting -s 192.168.0.0/24 -d X.X.X.X/32 -p tcp -m tcp --dport 6022 -m comment --comment "!fw3: ssh (reflection)" -j DNAT --to-destination 192.168.0.1:22
-A zone_lan_prerouting -s 192.168.0.0/24 -d X.X.X.X/32 -p udp -m udp --dport 6022 -m comment --comment "!fw3: ssh (reflection)" -j DNAT --to-destination 192.168.0.1:22
-A zone_lan_prerouting -s 192.168.253.3/32 -d X.X.X.X/32 -p tcp -m tcp --dport 6022 -m comment --comment "!fw3: ssh (reflection)" -j DNAT --to-destination 192.168.0.1:22
-A zone_lan_prerouting -s 192.168.253.3/32 -d X.X.X.X/32 -p udp -m udp --dport 6022 -m comment --comment "!fw3: ssh (reflection)" -j DNAT --to-destination 192.168.0.1:22
-A zone_lan_prerouting -s 192.168.0.0/24 -d X.X.X.X/32 -p tcp -m tcp --dport 80 -m comment --comment "!fw3: http (reflection)" -j DNAT --to-destination 192.168.0.1:80
-A zone_lan_prerouting -s 192.168.253.3/32 -d X.X.X.X/32 -p tcp -m tcp --dport 80 -m comment --comment "!fw3: http (reflection)" -j DNAT --to-destination 192.168.0.1:80
-A zone_wan_postrouting -m comment --comment "!fw3: Custom wan postrouting rule chain" -j postrouting_wan_rule
-A zone_wan_postrouting -m comment --comment "!fw3" -j MASQUERADE
-A zone_wan_prerouting -m comment --comment "!fw3: Custom wan prerouting rule chain" -j prerouting_wan_rule
-A zone_wan_prerouting -p tcp -m tcp --dport 6022 -m comment --comment "!fw3: ssh" -j DNAT --to-destination 192.168.0.1:22
-A zone_wan_prerouting -p udp -m udp --dport 6022 -m comment --comment "!fw3: ssh" -j DNAT --to-destination 192.168.0.1:22
-A zone_wan_prerouting -s 192.168.0.0/16 -p tcp -m tcp --dport 80 -m comment --comment "!fw3: http" -j DNAT --to-destination 192.168.0.1:80
COMMIT
# Completed on Fri Jan 29 15:43:37 2021
# Generated by iptables-save v1.8.6 on Fri Jan 29 15:43:37 2021
*mangle
:PREROUTING ACCEPT [14035:1885022]
:INPUT ACCEPT [10346:1331167]
:FORWARD ACCEPT [2099:292697]
:OUTPUT ACCEPT [15326:5078143]
:POSTROUTING ACCEPT [17425:5370840]
-A FORWARD -o tun0 -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -i tun0 -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -o eth0.2 -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -i eth0.2 -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
COMMIT
# Completed on Fri Jan 29 15:43:37 2021
# Generated by iptables-save v1.8.6 on Fri Jan 29 15:43:37 2021
*filter
:INPUT ACCEPT [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
:forwarding_lan_rule - [0:0]
:forwarding_rule - [0:0]
:forwarding_wan_rule - [0:0]
:input_lan_rule - [0:0]
:input_rule - [0:0]
:input_wan_rule - [0:0]
:output_lan_rule - [0:0]
:output_rule - [0:0]
:output_wan_rule - [0:0]
:reject - [0:0]
:syn_flood - [0:0]
:zone_lan_dest_ACCEPT - [0:0]
:zone_lan_forward - [0:0]
:zone_lan_input - [0:0]
:zone_lan_output - [0:0]
:zone_lan_src_ACCEPT - [0:0]
:zone_wan_dest_ACCEPT - [0:0]
:zone_wan_dest_REJECT - [0:0]
:zone_wan_forward - [0:0]
:zone_wan_input - [0:0]
:zone_wan_output - [0:0]
:zone_wan_src_REJECT - [0:0]
-A INPUT -i lo -m comment --comment "!fw3" -j ACCEPT
-A INPUT -m comment --comment "!fw3: Custom input rule chain" -j input_rule
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -m comment --comment "!fw3" -j ACCEPT
-A INPUT -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m comment --comment "!fw3" -j syn_flood
-A INPUT -i br-lan -m comment --comment "!fw3" -j zone_lan_input
-A INPUT -i wg0 -m comment --comment "!fw3" -j zone_lan_input
-A INPUT -i tun0 -m comment --comment "!fw3" -j zone_wan_input
-A INPUT -i eth0.2 -m comment --comment "!fw3" -j zone_wan_input
-A FORWARD -m comment --comment "!fw3: Custom forwarding rule chain" -j forwarding_rule
-A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -m comment --comment "!fw3" -j ACCEPT
-A FORWARD -i br-lan -m comment --comment "!fw3" -j zone_lan_forward
-A FORWARD -i wg0 -m comment --comment "!fw3" -j zone_lan_forward
-A FORWARD -i tun0 -m comment --comment "!fw3" -j zone_wan_forward
-A FORWARD -i eth0.2 -m comment --comment "!fw3" -j zone_wan_forward
-A FORWARD -m comment --comment "!fw3" -j reject
-A OUTPUT -o lo -m comment --comment "!fw3" -j ACCEPT
-A OUTPUT -m comment --comment "!fw3: Custom output rule chain" -j output_rule
-A OUTPUT -m conntrack --ctstate RELATED,ESTABLISHED -m comment --comment "!fw3" -j ACCEPT
-A OUTPUT -o br-lan -m comment --comment "!fw3" -j zone_lan_output
-A OUTPUT -o wg0 -m comment --comment "!fw3" -j zone_lan_output
-A OUTPUT -o tun0 -m comment --comment "!fw3" -j zone_wan_output
-A OUTPUT -o eth0.2 -m comment --comment "!fw3" -j zone_wan_output
-A reject -p tcp -m comment --comment "!fw3" -j REJECT --reject-with tcp-reset
-A reject -m comment --comment "!fw3" -j REJECT --reject-with icmp-port-unreachable
-A syn_flood -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m limit --limit 25/sec --limit-burst 50 -m comment --comment "!fw3" -j RETURN
-A syn_flood -m comment --comment "!fw3" -j DROP
-A zone_lan_dest_ACCEPT -o br-lan -m comment --comment "!fw3" -j ACCEPT
-A zone_lan_dest_ACCEPT -o wg0 -m comment --comment "!fw3" -j ACCEPT
-A zone_lan_forward -m comment --comment "!fw3: Custom lan forwarding rule chain" -j forwarding_lan_rule
-A zone_lan_forward -m comment --comment "!fw3: Zone lan to wan forwarding policy" -j zone_wan_dest_ACCEPT
-A zone_lan_forward -m conntrack --ctstate DNAT -m comment --comment "!fw3: Accept port forwards" -j ACCEPT
-A zone_lan_forward -m comment --comment "!fw3" -j zone_lan_dest_ACCEPT
-A zone_lan_input -m comment --comment "!fw3: Custom lan input rule chain" -j input_lan_rule
-A zone_lan_input -m conntrack --ctstate DNAT -m comment --comment "!fw3: Accept port redirections" -j ACCEPT
-A zone_lan_input -m comment --comment "!fw3" -j zone_lan_src_ACCEPT
-A zone_lan_output -m comment --comment "!fw3: Custom lan output rule chain" -j output_lan_rule
-A zone_lan_output -m comment --comment "!fw3" -j zone_lan_dest_ACCEPT
-A zone_lan_src_ACCEPT -i br-lan -m conntrack --ctstate NEW,UNTRACKED -m comment --comment "!fw3" -j ACCEPT
-A zone_lan_src_ACCEPT -i wg0 -m conntrack --ctstate NEW,UNTRACKED -m comment --comment "!fw3" -j ACCEPT
-A zone_wan_dest_ACCEPT -o tun0 -m conntrack --ctstate INVALID -m comment --comment "!fw3: Prevent NAT leakage" -j DROP
-A zone_wan_dest_ACCEPT -o tun0 -m comment --comment "!fw3" -j ACCEPT
-A zone_wan_dest_ACCEPT -o eth0.2 -m conntrack --ctstate INVALID -m comment --comment "!fw3: Prevent NAT leakage" -j DROP
-A zone_wan_dest_ACCEPT -o eth0.2 -m comment --comment "!fw3" -j ACCEPT
-A zone_wan_dest_REJECT -o tun0 -m comment --comment "!fw3" -j reject
-A zone_wan_dest_REJECT -o eth0.2 -m comment --comment "!fw3" -j reject
-A zone_wan_forward -m comment --comment "!fw3: Custom wan forwarding rule chain" -j forwarding_wan_rule
-A zone_wan_forward -p esp -m comment --comment "!fw3: Allow-IPSec-ESP" -j zone_lan_dest_ACCEPT
-A zone_wan_forward -p udp -m udp --dport 500 -m comment --comment "!fw3: Allow-ISAKMP" -j zone_lan_dest_ACCEPT
-A zone_wan_forward -m conntrack --ctstate DNAT -m comment --comment "!fw3: Accept port forwards" -j ACCEPT
-A zone_wan_forward -m comment --comment "!fw3" -j zone_wan_dest_REJECT
-A zone_wan_input -m comment --comment "!fw3: Custom wan input rule chain" -j input_wan_rule
-A zone_wan_input -p udp -m udp --dport 68 -m comment --comment "!fw3: Allow-DHCP-Renew" -j ACCEPT
-A zone_wan_input -p icmp -m icmp --icmp-type 8 -m comment --comment "!fw3: Allow-Ping" -j ACCEPT
-A zone_wan_input -p igmp -m comment --comment "!fw3: Allow-IGMP" -j ACCEPT
-A zone_wan_input -m conntrack --ctstate DNAT -m comment --comment "!fw3: Accept port redirections" -j ACCEPT
-A zone_wan_input -m comment --comment "!fw3" -j zone_wan_src_REJECT
-A zone_wan_output -m comment --comment "!fw3: Custom wan output rule chain" -j output_wan_rule
-A zone_wan_output -m comment --comment "!fw3" -j zone_wan_dest_ACCEPT
-A zone_wan_src_REJECT -i tun0 -m comment --comment "!fw3" -j reject
-A zone_wan_src_REJECT -i eth0.2 -m comment --comment "!fw3" -j reject
COMMIT
# Completed on Fri Jan 29 15:43:37 2021
interface: wg0
  public key: ......
  private key: (hidden)
  listening port: 39988

peer: ......
  preshared key: (hidden)
  endpoint: Y.Y.Y.Y:yyyy
  allowed ips: 192.168.253.0/24, 192.168.2.0/24, 0.0.0.0/0
  latest handshake: 5 seconds ago
  transfer: 354.26 KiB received, 1.42 MiB sent

I have another client connected to the server, no issue at all .. can get to the server subnet, and the internet. BTW, I see this on the OpenWrt client ... perhaps that Gateway is the issue? Not sure how to set that, I admit.

Thanks!

krazeh · January 30, 2021, 1:41am

What is the wireguard config on each end of the tunnel?

If you remove 192.168.253.0/24 and 192.168.2.0/24 from the Allowed IPs on the client and then restart the WG interface are you still able to access the server end?

arrmo · January 30, 2021, 2:57am

Yes! Sorry, just saw this, but I tried that and it does seem to work - so I have removed them, cleaned it up.

A bit of tcpdump capturing, but it almost seems like the traffic coming across the WG link is going out (internet request / access), but it's with the internal (private, non-routable) IP. Still need to confirm that, but that may explain why I see no return traffic. Hmm ... for traffic routed over WG, and then "out the door" ... where is NAT applied?

Thanks!

PS: This only seems to be the case for DHCP clients. The OpenWrt box itself (ssh in) can connect. Very odd!

vgaetera · January 30, 2021, 7:48am

The allowed IPs should not overlap, that is keep only 0.0.0.0/0 on the client side, and 192.168.253.3/32 + 192.168.0.0/24 on the server side for the respective peers.

In addition, change the WAN metric on the client to preserve its default gateway:
https://openwrt.org/docs/guide-user/services/vpn/wireguard/extras#dynamic_connection

And also post the server side diagnostics.

krazeh · January 30, 2021, 11:08am

If traffic from the 'client' LAN is getting to the 'server' side (and you can access local resources on both sides of the link) then the issue is likely with the 'server'. You are probably missing allowed IPs in the wireguard config or the routing hasn't been set up correctly.

arrmo · January 30, 2021, 8:26pm

Agreed! Removed all but 0.0.0.0/0, thanks! Really seems like traffic is going out, but not coming back (only for DHCP clients, from the "router itself" it works ... arrgh!).

Yep, thinking the same thing - believe we're aligned . As above, it's fine for the WG subnet (and ssh into the router), just not for DHCP clients (different subnet). Let me poke pfSense (the "server"). Thanks!

arrmo · January 31, 2021, 4:33am

FYI, getting a different answer from the pfSense side of things ... LOL. Meaning - they say I have to NAT on the OpenWrt end, and then can't really put WG in the LAN zone ... right? Rather, I have to forward LAN to WG (which I have seen in some posts here as well). Thoughts?

FYI, the pfSense input,

Thanks!

vgaetera · January 31, 2021, 5:17am

This is entirely feasible without NAT on a Linux-based system.
I'm not sure about BSD, but that reply sounds weird.
If that's true, I'd consider replacing pfSense.

If you decide to follow their advice:

Create a separate firewall zone with masquerading enabled.
Attach the WG interface to that zone.
Allow traffic forwardings between the LAN and WG zones.

arrmo · January 31, 2021, 11:30am

Sadly, that one isn't an option. And I do have several other clients working with the pfSense box, though none of them are behind a router (so a bit different, I admit).

Right, this is what I had seen when I started the other thread - it seemed like some folks were saying this is needed, no? And that wasn't pfSense related, just in general. Just trying to understand!

And for all traffic to go over WG, forward all traffic from LAN to WG, right? Then also, no forwarding from LAN to WAN (i.e. remove it) ... correct? This worries me just a bit, as it's how I got locked out before ... I lose ssh access then (to LAN) .

Thanks!

krazeh · January 31, 2021, 1:35pm

It's not. The other end of the tunnel (the 'server' end) should be able to route traffic back to the entire 'client' subnet. There should be no need to use NAT.

At the moment (assuming you haven't made the changes suggested above) can you access the server and client subnets from the opposing end of the tunnel? If so, then I wouldn't start messing around with the OpenWRT end of the tunnel. The issue is one of routing and needs to be sorted on the pfSense box.

arrmo · January 31, 2021, 3:44pm

Agreed! And sorry for the confusion - it was one of the thoughts / options rolling around in my head, but I didn't ask it well at all. That's on me. Sheesh, can't you interpret what I'm thinking (but not writing)?!?! Apologies!

No messing yet, and I do prefer this approach as well! Dumb question, but to step back just a bit ... thinking that on the server side I need to set allowed IP's to the client WG address, and the client subnet (i.e. 192.168.0.0/24). Agreed? And on the OpenWrt side ... just 0.0.0.0/0? Or perhaps as a first step, the server WG IP, and that subnet? Baby steps .

Thanks!

krazeh · January 31, 2021, 4:00pm

The 'Allowed IPs' field has two functions. The first is to determine what traffic is allowed into and out of the tunnel. Any traffic that is entering the tunnel has it's destination IP checked to see if it matches an allowed IP. Any traffic exiting the tunnel has it's source IP checked to see if it matches. Only traffic which matches is allowed in or out, respectively.

The second function is to set up appropriate routing. How this operates depends on the specific implementation, e.g. on OpenWrt it will only add routes if you set route_allowed_ips to 1. On Linux you'd use the wg-quick command. And i'm sure pfSense will have it's own options to do the same.

So putting 0.0.0.0/0 as an allowed IP means you don't need to add any other IPs. Everything will be covered already. On the other side you're right to add the 'client' subnet and the address you assigned to the 'client' WG interface. You then just have to figure out how the pfSense implementation of WG deals with routing.

arrmo · January 31, 2021, 4:25pm

Thanks for the detailed explanation - much appreciated! OK, let me give this a shot, will report back (and will try capturing things with tcpdump, to see what gets in and out ).

FYI, heading out of town for a couple days, not sure if I can do this remotely or not - so apologies if I go "silent". Just may not have access. But I will send an update as soon as I can get the data / info.

Thanks again!

arrmo · January 31, 2021, 5:45pm

OK, some real quick (and preliminary) results, as I run out the door - but for head scratching ... LOL!

ping, from OpenWrt (ssh, so source IP = WG address, don't have access to a DHCP client right now). But this is working fine (as it was before, to the server subnet). Still need to be able to check from a DHCP client, and of course, from a DHCP client to the internet (the core issue).

ping 192.168.2.1
PING 192.168.2.1 (192.168.2.1): 56 data bytes
64 bytes from 192.168.2.1: seq=0 ttl=64 time=60.228 ms
64 bytes from 192.168.2.1: seq=1 ttl=64 time=60.685 ms

But, from the server side back to the client,

ping 192.168.0.120
PING 192.168.0.120 (192.168.0.120) 56(84) bytes of data.
From 192.168.253.1 icmp_seq=1 Redirect Host(New nexthop: 0.0.0.0)
From 192.168.253.1 icmp_seq=1 Redirect Host(New nexthop: 0.0.0.0)
From 192.168.253.1 icmp_seq=1 Redirect Host(New nexthop: 0.0.0.0)
...
--- 192.168.0.120 ping statistics ---
2 packets transmitted, 0 received, +34 errors, 100% packet loss, time 1000ms

This is a new one on me ... have you seen it before? Is this tied to the pic I sent before (I think!), showing that the Gateway for the Wireguard VPN (on OpenWrt) is 0.0.0.0?

Thanks!

krazeh · January 31, 2021, 8:06pm

From what you've posted it looks like the OpenWrt end is fine. The issue is with the pfSense end. Is 192.168.253.1 the IP for the WG interface on the pfSense box?

arrmo · February 1, 2021, 2:51am

It is! And you may be right, I admit ... not sure (yet). I did a traceroute from the pfSense box (and another box on the subnet), and it does get back to a box behind the OpenWrt router. So that does seem to work. The ping may be a red herring (searching on it a bit), as it's responding from a different IP than "expected" ... I think, may be wrong!

It's really about getting the .0.x subnet traffic, from behind the OpenWrt router out to the internet. Not sure why it's being blocked .

Thanks!

arrmo · February 1, 2021, 2:33pm

BTW, I have left LAN and WLAN (x2) bridged ... assuming that's OK? It is odd, if I remove and re-add this, then br-lan is in the routing table. If I then take WG down and back up ... br-lan is removed from the routing table.

Thanks!

krazeh · February 1, 2021, 3:01pm

Can you post your routing table? With and without WG up.

I still think the issue is at the pfSense end though. Can you try doing some traceroutes to internet hosts from devices on the OpenWrt subnet?

arrmo · February 1, 2021, 3:57pm

Sure! Below. And a couple notes, to explain

I had to add the OpenWrt subnet to the WG AllowedIP's ... or pings out from OpenWrt (to the pfSense subnet) fail. Huh? That part loses me ... LOL. Or is it as you mentioned - all incoming and outgoing IPs are checked against that list? But if so, then by adding the local subnet (which also drives the routing table), do I lose access to local subnet machines (e.g. local network printer)? Somehow it seems that routing and allowed IP's need to be independent, no?
I left the default bridge on lan and wlan0, wlan1. Is this OK?
I can get from the router itself (logged in via ssh) - through to the internet. It's only DHCP clients off of the router that fail. And I can't really get to those clients easily, they are ~ 1800 miles away.

And the tables (external internet IP replaced by x.x.x.x)
a) wg off

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         host254-183.res 0.0.0.0         UG    0      0        0 eth0.2
x.x.x.x         host254-183.res 255.255.255.255 UGH   0      0        0 eth0.2
128.189.183.0   *               255.255.255.0   U     0      0        0 eth0.2

b) wg on

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         host254-183.res 0.0.0.0         UG    0      0        0 eth0.2
x.x.x.x         host254-183.res 255.255.255.255 UGH   0      0        0 eth0.2
128.189.183.0   *               255.255.255.0   U     0      0        0 eth0.2
192.168.0.0     *               255.255.255.0   U     0      0        0 wg0
192.168.1.0     *               255.255.255.0   U     0      0        0 wg0
192.168.2.0     *               255.255.255.0   U     0      0        0 wg0
192.168.253.0   *               255.255.255.0   U     0      0        0 wg0
216.115.184.69  *               255.255.255.255 UH    0      0        0 wg0

And, here is what I have for allowed IP's (OpenWrt end) - as above, I had to add 192.168.0.0, or even ping to the pfSense subnet fails. I have only set up one IP for now (on the Internet), for testing ... not to break anything else.

216.115.184.69/32
192.168.1.0/24
192.168.2.0/24
192.168.0.0/24
192.168.253.0/24

FYI, from the web interface, Restart of WG (interface) does not seem to really work? Rather, I need to do and ifdown and if up (wg). Is this expected?

Sure! And you may be right, no argument. Here is an example. But as above, this is ssh'd in, it works. Not from a DHCP client (that I can't get to right now, not sure how to simulate that from the ssh interface?).

traceroute to 216.115.184.69 (216.115.184.69), 30 hops max, 38 byte packets
 1  192.168.253.1 (192.168.253.1)  65.116 ms  63.871 ms  60.546 ms
 2  192.168.1.1 (192.168.1.1)  60.874 ms  63.550 ms  60.919 ms
 3  *  *  *
 4  *  *  *
 5  *  *  *
 6  *  *  *
 7  *  *  *
 8  *  *  *
 9  *  *  *
10  *  *  *
11  *  *  *
12  216.115.184.69 (216.115.184.69)  99.648 ms  101.049 ms  104.673 ms

Thanks!!

krazeh · February 1, 2021, 4:05pm

I thought you were using 0.0.0.0/0 on the OpenWrt end? Why have you removed it?

What subnets are you using? Can you list them all (from both sites) please.

I'll be honest, it's been a while since I've messed with my WG configs. I can't recall if I was able to just restart the interface through Luci when making changes or if I had to restart the interface through SSH.