Wireguard multiple interfaces not working at the same time

Dm1 · December 4, 2018, 2:58pm

They actually are on different ports already:

option listen_port '55656'
option listen_port '55655'

There is the same port on a remote peer, but how can this be a problem?

option endpoint_port '55655'

lleachii · December 4, 2018, 3:02pm

I stand corrected, I see that's your remote peer now.

Can you provide the results of cat /proc/sys/kernel/random/entropy_avail before and after the connection issue.
Also, do you have a random number generator or Atheros ath9k WiFi enabled?

You need enough entropy for the encrypted connections.

Dm1 · December 4, 2018, 3:04pm

Also, do you have a random number generator or Atheros ath9k WiFi enabled?

Yep, this is TP-Link TL-WR1043N/ND v2 with WiFi enabled.

lleachii · December 4, 2018, 3:06pm

BTW...you did not provide the results of cat /proc/sys/kernel/random/entropy_avail

Dm1 · December 4, 2018, 3:08pm

root@ROUTER:~# cat /proc/sys/kernel/random/entropy_avail
845
root@ROUTER:~# cat /proc/sys/kernel/random/entropy_avail
846
root@ROUTER:~# cat /proc/sys/kernel/random/entropy_avail
848
root@ROUTER:~# cat /proc/sys/kernel/random/entropy_avail
849
root@ROUTER:~# cat /proc/sys/kernel/random/entropy_avail
866
root@ROUTER:~# cat /proc/sys/kernel/random/entropy_avail
868
root@ROUTER:~# cat /proc/sys/kernel/random/entropy_avail
869
root@ROUTER:~# cat /proc/sys/kernel/random/entropy_avail
870
root@ROUTER:~# cat /proc/sys/kernel/random/entropy_avail
870
root@ROUTER:~# cat /proc/sys/kernel/random/entropy_avail
871
root@ROUTER:~# cat /proc/sys/kernel/random/entropy_avail
873
root@ROUTER:~# cat /proc/sys/kernel/random/entropy_avail
874
root@ROUTER:~# cat /proc/sys/kernel/random/entropy_avail
875
root@ROUTER:~# cat /proc/sys/kernel/random/entropy_avail
876
root@ROUTER:~# cat /proc/sys/kernel/random/entropy_avail
877
root@ROUTER:~# cat /proc/sys/kernel/random/entropy_avail
878
root@ROUTER:~# cat /proc/sys/kernel/random/entropy_avail
878
root@ROUTER:~# cat /proc/sys/kernel/random/entropy_avail

lleachii · December 4, 2018, 3:11pm

Your entropy is extremely low!!!

SEE:

Also, you can try to install haveged

Dm1 · December 4, 2018, 3:20pm

Yeah, looks like it's the source of the problem. At least entropy is really too low and I've not even thought about this. Now haveged fixed this for me:

root@ROUTER:~# cat /proc/sys/kernel/random/entropy_avail
2054
root@ROUTER:~# cat /proc/sys/kernel/random/entropy_avail
2055
root@ROUTER:~# cat /proc/sys/kernel/random/entropy_avail
2059
root@ROUTER:~# cat /proc/sys/kernel/random/entropy_avail
2061
root@ROUTER:~# cat /proc/sys/kernel/random/entropy_avail
2062
root@ROUTER:~# cat /proc/sys/kernel/random/entropy_avail
2063
root@ROUTER:~# cat /proc/sys/kernel/random/entropy_avail
2064
root@ROUTER:~# cat /proc/sys/kernel/random/entropy_avail
2065

I'll test wireguard tomorrow and tell you the result. Thanks man!

Dm1 · December 5, 2018, 10:03am

Sadly this didn't solve the issue:

root@ROUTER:~# cat /proc/sys/kernel/random/entropy_avail
1655
root@ROUTER:~# cat /proc/sys/kernel/random/entropy_avail
1657
root@ROUTER:~# cat /proc/sys/kernel/random/entropy_avail
1659
root@ROUTER:~# cat /proc/sys/kernel/random/entropy_avail
1662
root@ROUTER:~# cat /proc/sys/kernel/random/entropy_avail
1665

root@ROUTER:~# wg
interface: wg0
  public key: PUBLIC_KEY3
  private key: (hidden)
  listening port: 55656

peer: PUBLIC_KEY
  endpoint: ENDPOIT_IP:55655
  allowed ips: 0.0.0.0/0, 10.0.0.3/32
  latest handshake: 5 seconds ago
  transfer: 3.35 GiB received, 596.86 MiB sent

interface: wg1
  public key: PUBLIC_KEY4
  private key: (hidden)
  listening port: 55655

peer: PUBLIC_KEY2
  endpoint: ENDPOIT_IP2:41538
  allowed ips: 172.17.1.0/24, 10.0.0.2/32
  transfer: 888 B received, 552 B sent

Because endpoint: ENDPOIT_IP2:41538 shows up by itself while it's not present in config, and some data is transmitted and received, it's still looks like a handshake problem, but entropy is high enough for it to work.

Router runs LEDE Reboot 17.01.5, maybe it's a good idea to try OpenWrt 18.06.1, but I'm not rushing with new releases usually.

lleachii · December 5, 2018, 1:02pm

Incorrect. Sadly, your entropy is still low...and sinking (if you never rebooted from the ~2065 results.

Dm1 · December 5, 2018, 1:11pm

It sinks to 1000 and then immediately raises over 2000 and so on. I've never seen embedded device with entropy over 3000, so basically wireguard is incompatible with all embedded devices without hardware accelerated encryption?

lleachii · December 5, 2018, 1:15pm

Can you test using just ONE WG interface, first?

I honestly run 3 WG interfaces with multiple peers - on devices with and without P/TRNGs (without, I needed my ath9k WiFi enabled to do so).

Dm1 · December 5, 2018, 1:18pm

One wireguard interface works absolutely fine, both wg0 and wg1 are working without a problem when the other one is down.

lleachii · December 5, 2018, 1:36pm

Are you running anything else requiring encryption on your device, aside from:

SSH
and the 2 WG interfaces?
BTW, if wg1's peer (seems to be point-to-point and not a VPN provider) is behind a firewall, make sure the remote end has a keepalive.

Dm1 · December 5, 2018, 1:46pm

I have also 2 SSH connections at the same time, 2 WiFi networks with 8 clients connected total, two wireguard interfaces with one of them working fine, nothing else uses encryption I think.

Is there a way to make wireguard more verbose to debug this?

Dm1 · December 5, 2018, 2:35pm

Looks like it is actually some kind of routing problem. Interesting why it affects only wireguard and none of other router services and forwarded ports, but without list allowed_ips '0.0.0.0/0' router now handles both wireguard connections. I've tested this before of course, but entropy was also a problem.
One of those situations when several problems combined made everything look mystical.
Will try to add packet marking to solve this.

lleachii · December 5, 2018, 3:02pm

OH!!!

Yes, I missed that.

This is because you made the VPN the default route.

In the future, you can make a route to the IP of the other peers using WAN xxx.xxx.xxx.xxx via eth0.2 src <IP_of_WAN>.

config route
	option interface 'wan'
	option target 'xxx.xxx.xxx.xxx/32'
	option gateway 'xxx.xxx.xxx.xxx'

Then use the following 2 routes to get Internet on VPN:

0.0.0.0/1
128.0.0.0/1

Dm1 · December 5, 2018, 3:08pm

And connection tracking doesn't fix this because it doesn't help for local services? This is why other services through forwarded ports were working fine with default route through VPN, but local wireguard wasn't? Or there is something else I didn't get right?

lleachii · December 5, 2018, 3:13pm

Doesn't fix what???

Connection tracking is a firewall feature, what does it have to do with an incorrect route?

If you are port forwarding through the working WG interface, of course it worked. WG also has roaming (if you don't set an endpoint IP on the remote side), so you should have had a route loop. Better to properly setup routes.

Dm1 · December 5, 2018, 3:18pm

Connection tracking ensures that replied packets of incoming connection are router through the same interface. In my case wg1 is an incoming connection from a remote peer.

lleachii · December 5, 2018, 3:21pm

Yes...I know what Connection Tracking is.

What I'm telling you is:

You setup 2 WGs
You incorrectly route all traffic through X
You SEND a packet to the endpoint on tunnel Y
It will be received at the remote end as long as you don't have an endpoint address set
Endpoint Y is free to send the reply packet!
You likely are not blocking for this, and remember you allowed 0.0.0.0/0 anyways. So connection tracking worked, it did come back on the same Interface!
Even though you allowed 0.0.0.0/0, it doesn't matter, as I said before, I believe you made a route loop over the Internet (via your VPN).

Hope this helps you understand.