Strange routing on wg interface

Trying to connect two remote LANs via wireguard.

following advice for another thread, I'm not specifying gateways anywhere (apart from the main WAN's GW)

there's a large site (10.0.1.0/24) with a E8450 (UBI), and a small site (10.0.2.0/24) with a UniFi AC Pro. both on 23.05.5. They are setup the same, only the large site have individual physical ports, and the small have the weird unifi ap single logical port with vlans for two physical ports. But apart from that, same.

tunnels start fine. wg show shows the peers and handshake times OK.

both /etc/config/network entries look like (i name things xLarge and xSmall to differentiate them)

config interface 'wgLarge'
        option proto 'wireguard'
        option private_key ...
        option listen_port '51820'
        list dns '10.0.2.1' # small LAN modem
        option delegate '0'
        option dns_metric '9'

config wireguard_wgLarge
        option description 'wgSmallpeer'
        option public_key ...
        option preshared_key ...
        option route_allowed_ips '1'
        list allowed_ips '10.0.2.0/24' # small lan
        option endpoint_host '1.2.3.4'
        option endpoint_port '51820'

config device
        option name 'wgLarge'
        option ipv6 '0'

config interface 'lan'
        option device 'br-lan'
        option proto 'static'
        option netmask '255.255.255.0'
        option ipaddr '10.0.1.1' # large lan

The other lan is the same, but replace Large<->Small and 10.0.1.*<->10.0.2.*

after wg peers handshake:

openwrt-large-lan# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         192.168.1.1     0.0.0.0         UG    0      0        0 wan
10.0.1.0        *               255.255.255.0   U     0      0        0 br-lan
10.0.2.0        *               255.255.255.0   U     0      0        0 wgLarge
1.2.3.4         192.168.1.1     255.255.255.255 UGH   0      0        0 wan
192.168.1.0     *               255.255.255.0   U     0      0        0 wan
10.99.3.0       *               255.255.255.0   U     0      0        0 br-guest

openwrt-large-lan# wg show
interface: wgLarge
  public key: large....=
  private key: (hidden)
  listening port: 51820

peer: small...=
  preshared key: (hidden)
  endpoint: 1.2.3.4:51820
  allowed ips: 10.0.2.1/24
  transfer: 0 B received, 25.44 KiB sent

(large also has a guest network that small doesn't have)

openwrt-small-lan# # route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         192.168.1.1     0.0.0.0         UG    0      0        0 eth0.2
10.0.1.0        *               255.255.255.0   U     0      0        0 wgSmall
192.168.1.1     *               255.255.255.0   U     0      0        0 eth0.2
10.0.2.0        *               255.255.255.0   U     0      0        0 br-lan
22.22.22.22     192.168.1.1     255.255.255.255 UGH   0      0        0 eth0.2

openwrt-small-lan# wg show
interface: wgSmall
  public key: small....=
  private key: (hidden)
  listening port: 51820

peer: large...=
  preshared key: (hidden)
  endpoint: 22.22.22.22:51820
  allowed ips: 10.0.1.0/24
  latest handshake: 48 minutes, 39 seconds ago
  transfer: 2.54 MiB received, 4.05 MiB sent

So far so good... but now the weird part:

if I ping (all modems have the default firewall rules to allow icmp, also on both the wg* interface is assigned to the LAN zone) from:

  • large -> small: timesout
  • small -> large: works
  • client on large LAN -> small: works!

why wile on large, 10.0.1.1, i cannot ping 1.0.2.1, but a client on it's wifi network can? traceroute didn't help at all, it just goes on forever.

edit:

just noticed the client on large LAN -> small connection works only when the small lan have data going trhu the tunnel :confused:

so, from a client (e.g. 10.0.1.133):

client@large# $ tracepath -n 10.0.2.1
 1?: [LOCALHOST]                      pmtu 1500
 1:  10.0.1.1                                            0.468ms 
 1:  10.0.1.1                                            0.369ms 
 2:  10.0.1.1                                            0.372ms pmtu 1420
 2:  no reply
 3:  no reply
 4:  no reply
 5:  no reply
 6:  no reply
 7:  no reply
 8:  no reply
^C

But if I ssh to modem at small lan, and ping 10.0.1.1 (i.e. send data trhu the already active wg tunnel),

client@large# $ tracepath -n 10.0.2.1
 1?: [LOCALHOST]                      pmtu 1500
 1:  10.0.1.1                                            0.548ms 
 1:  10.0.1.1                                            0.367ms 
 2:  10.0.1.1                                            0.361ms pmtu 1420
 2:  10.0.2.1                                          48.496ms reached
     Resume: pmtu 1420 hops 2 back 2 

But, from modem at large (10.0.1.1) i get no connection at all, even after "activating" it on the other side, which seems to work for the clients, which send data trhu the modem 10.0.1.1... i'm lost on this.

one thing that is different are the ISP modems... then are forwarding port 51820u and nothing else. I don't have much info on them.

Since the wg tunnel doesn't have its own IP address to use as a source IP, pinging the other site from the router kernel itself (rather than forwarded from the LAN) is not certain. Use ping -I 10.1.0.1 10.0.2.1 (the I option is the local site's LAN IP) to be sure that the ping originates with a source IP which is an allowed_ip at the other end.

Depending on how the NAT in the main router is implemented you may need to add persistent_keepalive to keep the tunnel open. It doesn't hurt to set it.

4 Likes

This. Was my first guess too.

1 Like

I am not sure why you do not give your WG interfaces its own address, although theoretically not necessary, I have seen strange routing issues arise from it.

This is interesting,it should be 10.0.2.0/24 as you set it:

1 Like

Sure if you set your routes correctly like manually via ip route add 192.0.2.0/24 dev wg99 src 198.51.100.1 or for instance with bird2 and krt_prefsrc and setting allowed ips correctly then yes it works. Maybe 192.0.2.1/32 is needed on the interface too but yes the more common way is to set tunnel addresses and use these for next hop. (Good key words are Linux and unnumbered ipv4 interfaces.)

I didn't assign IPs mostly because you suggested they were optional before :slight_smile: and I'm trying to have the most bare bones setup initially. But you did mention assigning an IP help with debugging.... So i will do that.

meanwhile, pinging forcing interface does work! but again, only after siteSmall initiate the connection.

large# ping -I 10.0.1.1 10.0.2.1
(nothing)

ping hangs there without any success. but as soon as i ssh to site 2 and start a ping

small# ping 10.0.1.1
64 bytes from ...

then the ping at site 1(large) starts to succeed right away! (the first pings with some 10s delay or whatever the time i took me to ssh to the other side). OK that solves the "why the modem behaves differently from the clients" part, but still have to "initiate" the peerage from one specific side :frowning:


ok, now assigning IP addresses to the wg interfaces, but i don't think I understood:

I've added .9 of the remote network to each tunnel (since client dhcp starts at .100+) now the hosts have the IPs:

1: largeModem
10.0.1.1 LAN
10.0.2.9 wg (allowed ips 10.0.2.0/24)

2: smallModem
10.0.2.1 LAN
10.0.1.9 wg (allowed ips 10.0.1.0/24)

modem small (2) can still ping lan 1. but modem large (1) still cannot ping the other, and if i force the interface on ping, I still need to use the LAN ip, the wg IP won't work.

large# ping -I 10.0.2.9 10.0.2.1
... 100% packet loss
large# ping -I 10.0.1.1 10.0.2.1
(works)

also, now that i've also applied the IP change to site 2 (service network restart), i cannot ping directly:

small# ping 10.0.1.1
(used to work before assigning ip, now doesn't)
small# ping -I 10.0.1.9 10.0.1.1
(like the other site, using the wg interface also doesn't work)
small# ping -I 10.0.2.1 10.0.1.1
(now, using the LAN interface works, like the other site did before and after assigning an IP)

yeah, I definitely got the wg interface IPs wrong. I cannot ping them from anywhere, using any combination of interface as sources.

looking at docs on wireguard/openwrt/arch, i see they always use allowed IPs on the same network (a:10.0.1/24, b:10.0.2/24) is this a requirement?

The intersite tunnel IPs have to each be in a unique subnet that doesn't overlap any of the lans, nor any of the road warriors, or any other intersite tunnel. Which is why I suggested not using them for the point to point tunnels as you'll need to keep track of a lot of subnets, and the only time these IP addresses will actually be used is to send test pings through the tunnels.

To be clear, the wg interfaces of both ends of a point to point tunnel are different IPs in the same subnet, but not the same subnet as anything else. The allowed_ip set for the peer needs to include the peer's tunnel IP.

I see. So i should use ip: 10.99.99.1/32 and .2 on each wg interface? (is there a "standard" ip range for those?)

All guides mention to use /32, but you said subnet... does it have to be a subnet?

trying both:

  • using 10.99.99.[1,2]/32 nothing pings those addresses, no matter which interface i force on ping.
  • using 10.99.99.[1,2]/24, i get "ping: sendto: Required key not available"

oh, i guess i have to add 10.99.99... to the allowed ip range.
I've added 10.99.99.2/32 on 1 and .1/32 on 2. and now i can ping each other, but still have to force the LAN ip as outgoing ping interface.

openwrt set the routing as

large# route
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         192.168.1.1     0.0.0.0         UG    0      0        0 wan
10.0.1.0        *               255.255.255.0   U     0      0        0 br-lan
10.0.2.0        *               255.255.255.0   U     0      0        0 wg1
10.99.99.0      *               255.255.255.0   U     0      0        0 wg1
10.99.99.2      *               255.255.255.255 UH    0      0        0 wg1
1.2.3.4         192.168.1.1     255.255.255.255 UGH   0      0        0 wan
192.168.1.0     *               255.255.255.0   U     0      0        0 wan

large# ip a
73: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    inet 10.55.42.1/24 brd 10.55.42.255 scope global br-lan
75: wg1: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN qlen 1000
    inet 10.99.99.1/24 brd 10.99.99.255 scope global wg1

large# wg show
interface: wg1
  public key: large...=
  private key: (hidden)
  listening port: 51820

peer: smal...=
  preshared key: (hidden)
  endpoint: 22.22.22.22:51820
  allowed ips: 10.0.2.0/24, 10.99.99.2/32
  latest handshake: 7 seconds ago
  transfer: 58.89 KiB received, 33.78 KiB sent

this is not true anymore after i restarted network on both modems at the same time... now i can ping each other using the 10.99.99.[1,2] ips without any outgoing interface

2 Likes

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.