Wireguard + mwan3, problem with https

Installing and Using OpenWrt Network and Wireless Configuration
1

Good day mate,

I'm trying to config Wireguard thr my wan pppoe on mwan3.

lan -> wg_1 -> wan pppoe
lan -> wg_2 -> wan pppoe
lan -> wg_3 -> wan pppoe
over mwan3.

I'm pretty close,
I could ping/nslookup with "-I {wg_1/wg_2/wg_3/pppoe}" to google.com in my openwrt terminal,
in terminal, I can wget http://aSite/index.html, but couldn't wget https://aSite/index.html.
the same from my lan, in browser, I could go to "http://" sites, but couldn't go to "https://".

I've found someone had similar issue, by changing the MTU to 1500 (which didn't work in my case)

what could be the possible reason failing with https? (I could see it's trying to do some TLS handshake and fail)

2

Please run the following commands (copy-paste the whole block) and paste the output here, using the "Preformatted text </> " button:
grafik
Remember to redact passwords, MAC addresses and any public IP addresses you may have

ubus call system board; \
uci export network; uci export mwan3; \
uci export dhcp; uci export firewall; \
ip link; mwan3 status; \
ls -l  /etc/resolv.* /tmp/resolv.* /tmp/resolv.*/* ; head -n -0 /etc/resolv.* /tmp/resolv.* /tmp/resolv.*/*
3

here you go, thanks so much!

root@OpenWrt:~# ubus call system board; \
> uci export network; uci export mwan3; \
> uci export dhcp; uci export firewall; \
> ip link; mwan3 status; \
> ls -l  /etc/resolv.* /tmp/resolv.* /tmp/resolv.*/* ; head -n -0 /etc/resolv.* /tmp/resolv.* /tmp/resolv.*
/*
{
	"kernel": "5.10.176",
	"hostname": "OpenWrt",
	"system": "Intel(R) Pentium(R) Dual  CPU  T2390  @ 1.86GHz",
	"model": "LENOVO Lenovo 3000 G410                ",
	"board_name": "lenovo-lenovo-3000-g410",
	"rootfs_type": "ext4",
	"release": {
		"distribution": "OpenWrt",
		"version": "22.03.5",
		"revision": "r20134-5f15225c1e",
		"target": "x86/64",
		"description": "OpenWrt 22.03.5 r20134-5f15225c1e"
	}
}
package network

config interface 'loopback'
	option device 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option packet_steering '1'
	option ula_prefix 'dddd:dddd:dddd::/48'

config device
	option name 'br-lan'
	option type 'bridge'
	list ports 'eth0'

config interface 'lan'
	option proto 'static'
	option ip6assign '60'
	option device 'eth0'
	option ipaddr '192.168.0.1'
	option netmask '255.255.0.0'

config interface 'HW_USB_4GLTE'
	option device 'eth2'
	option proto 'dhcp'
	option metric '99'

config interface 'eth1DialupDHCP'
	option proto 'dhcp'
	option device 'eth1'

config device
	option name 'pppoe-DialupInt'
	option macaddr 'C0:4A:00:6D:77:55'

config device
	option name 'eth1'
	option macaddr 'C0:4A:00:6D:77:55'

config interface 'wan0'
	option proto 'pppoe'
	option device 'eth1'
	option username 'xxx'
	option password 'xxx'
	option ipv6 'auto'
	option peerdns '0'
	option metric '90'
	list dns '1.1.1.1'
	list dns '8.8.8.8'

config interface 'vpnsyd'
	option proto 'wireguard'
	option private_key 'xxx='
	list addresses '10.14.0.2/16'
	option metric '11'
	list dns '149.154.159.92'
	list dns '162.252.172.57'

config wireguard_vpnsyd
	option description 'my peer'
	option public_key 'xxx/Dl5KyIk4='
	list allowed_ips '0.0.0.0/0'
	option route_allowed_ips '1'
	option endpoint_host 'au-syd.prod.surfshark.com'
	option private_key 'xxx='
	option endpoint_port '51820'

config interface 'vpnmel'
	option proto 'wireguard'
	option private_key 'xxx='
	option metric '12'
	list addresses '10.14.0.3/16'
	list dns '149.154.159.92'
	list dns '162.252.172.57'

config wireguard_vpnmel
	option description 'my peer'
	option public_key 'xxx+u21o='
	list allowed_ips '0.0.0.0/0'
	option route_allowed_ips '1'
	option endpoint_host 'au-mel.prod.surfshark.com'
	option endpoint_port '51820'
	option private_key 'xxx='

config interface 'vpnsng'
	option proto 'wireguard'
	option private_key 'xxx='
	option metric '20'
	list addresses '10.14.0.9/16'
	list dns '149.154.159.92'
	list dns '162.252.172.57'

config wireguard_vpnsng
	option description 'sng peer'
	option public_key 'xxx='
	list allowed_ips '0.0.0.0/0'
	option route_allowed_ips '1'
	option endpoint_host 'sg-sng.prod.surfshark.com'
	option private_key 'xxxx='
	option endpoint_port '51820'

config device
	option name 'vpnmel'
	option macaddr 'xx:77:00'

config device
	option name 'pppoe-wan0'
	option macaddr 'xx:77:55'

config device
	option name 'vpnsng'
	option macaddr 'xx:77:01'

config device
	option name 'vpnsyd'
	option macaddr 'xx:77:02'

package mwan3

config globals 'globals'
	option mmx_mask '0x3F00'
	option logging '1'
	option loglevel 'debug'

config policy 'balanced'
	option last_resort 'unreachable'
	list use_member 'm_vpnsyd'
	list use_member 'm_vpnmel'

config rule 'Dell_To_SG'
	option proto 'all'
	option src_ip '192.168.1.202'
	option sticky '0'
	option use_policy 'vpnsngOnly'

config rule 'https'
	option sticky '1'
	option use_policy 'balanced'
	option proto 'tcp'
	option dest_port '443'

config rule 'default_rule_v4'
	option use_policy 'balanced'
	option family 'ipv4'
	option proto 'all'
	option sticky '0'

config interface 'vpnsyd'
	option enabled '1'
	option initial_state 'online'
	option family 'ipv4'
	option track_method 'ping'
	option reliability '1'
	option count '1'
	option size '56'
	option max_ttl '60'
	option timeout '4'
	option interval '10'
	option failure_interval '5'
	option recovery_interval '5'
	option down '5'
	option up '5'
	list track_ip '1.1.1.1'

config interface 'vpnmel'
	option enabled '1'
	option initial_state 'online'
	option family 'ipv4'
	option track_method 'ping'
	option reliability '1'
	option count '1'
	option size '56'
	option max_ttl '60'
	option timeout '4'
	option interval '10'
	option failure_interval '5'
	option recovery_interval '5'
	option down '5'
	option up '5'
	list track_ip '1.1.1.1'

config member 'm_vpnsyd'
	option interface 'vpnsyd'
	option metric '10'
	option weight '3'

config member 'm_vpnmel'
	option interface 'vpnmel'
	option metric '10'
	option weight '3'

config interface 'vpnsng'
	option enabled '1'
	option initial_state 'online'
	option family 'ipv4'
	option track_method 'ping'
	option reliability '1'
	option count '1'
	option size '56'
	option max_ttl '60'
	option timeout '4'
	option interval '10'
	option failure_interval '5'
	option recovery_interval '5'
	option down '5'
	option up '5'
	list track_ip '1.1.1.1'

config member 'm_vpnsng'
	option metric '11'
	option weight '3'
	option interface 'vpnsng'

config policy 'vpnsngOnly'
	list use_member 'm_vpnsng'
	option last_resort 'unreachable'

package dhcp

config dnsmasq
	option domainneeded '1'
	option localise_queries '1'
	option rebind_protection '1'
	option rebind_localhost '1'
	option local '/lan/'
	option domain 'lan'
	option expandhosts '1'
	option authoritative '1'
	option readethers '1'
	option leasefile '/tmp/dhcp.leases'
	option resolvfile '/tmp/resolv.conf.d/resolv.conf.auto'
	option localservice '1'
	option ednspacket_max '1232'

config dhcp 'lan'
	option interface 'lan'
	option start '100'
	option limit '150'
	option dhcpv4 'server'
	option dhcpv6 'server'
	option ra 'server'
	list ra_flags 'managed-config'
	list ra_flags 'other-config'
	option leasetime '2m'

config odhcpd 'odhcpd'
	option maindhcp '0'
	option leasefile '/tmp/hosts/odhcpd'
	option leasetrigger '/usr/sbin/odhcpd-update'
	option loglevel '4'

config host
	option name 'wirelessRouterAP'
	option dns '1'
	option mac 'xxx:E0'
	option leasetime '9999999999999h'
	option ip '192.168.1.1'

 
config domain
	option name 'openwrt'
	option ip '192.168.0.1'



package firewall

config defaults
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option synflood_protect '1'

config zone
	option name 'lan'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'ACCEPT'
	list network 'lan'

config zone
	option name 'wan'
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option masq '1'
	option mtu_fix '1'
	list network 'wan6'
	list network 'HW_USB_4GLTE'
	list network 'eth1DialupDHCP'
	list network 'wan0'

config rule
	option name 'Allow-DHCP-Renew'
	option src 'wan'
	option proto 'udp'
	option dest_port '68'
	option target 'ACCEPT'
	option family 'ipv4'

config rule
	option name 'Allow-Ping'
	option src 'wan'
	option proto 'icmp'
	option icmp_type 'echo-request'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-IGMP'
	option src 'wan'
	option proto 'igmp'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-DHCPv6'
	option src 'wan'
	option proto 'udp'
	option dest_port '546'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-MLD'
	option src 'wan'
	option proto 'icmp'
	option src_ip 'fe80::/10'
	list icmp_type '130/0'
	list icmp_type '131/0'
	list icmp_type '132/0'
	list icmp_type '143/0'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Input'
	option src 'wan'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	list icmp_type 'router-solicitation'
	list icmp_type 'neighbour-solicitation'
	list icmp_type 'router-advertisement'
	list icmp_type 'neighbour-advertisement'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Forward'
	option src 'wan'
	option dest '*'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-IPSec-ESP'
	option src 'wan'
	option dest 'lan'
	option proto 'esp'
	option target 'ACCEPT'

config rule
	option name 'Allow-ISAKMP'
	option src 'wan'
	option dest 'lan'
	option dest_port '500'
	option proto 'udp'
	option target 'ACCEPT'

config zone
	option name 'vpnclient'
	option input 'REJECT'
	option forward 'REJECT'
	option output 'ACCEPT'
	option masq '1'
	list network 'vpnsyd'
	list network 'vpnmel'
	list network 'vpnsng'

config forwarding
	option src 'lan'
	option dest 'vpnclient'

config rule
	option name 'Allow-OpenVPN-Inbound'
	option target 'ACCEPT'
	option dest_port '1194'
	list proto 'udp'
	option src 'wan'

config redirect
	option dest 'lan'
	option target 'DNAT'
	option name 'Force DNS to local'
	option src 'lan'
	option src_dport '53'
	option enabled '0'

config rule
	option name 'Filter-Parental-Controls'
	option src 'lan'
	option src_mac '00:11:22:33:44:55'
	option dest 'wan'
	option start_time '21:30:00'
	option stop_time '07:00:00'
	option weekdays 'Mon Tue Wed Thu Fri'
	option target 'REJECT'
	option enabled '0'


config forwarding
	option src 'lan'
	option dest 'wan'

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 00:1e:ec:xxx brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether c0:4a:00:xxx:55 brd ff:ff:ff:ff:ff:ff permaddr 00:b5:6d:02:cc:c9
4: vpnmel: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/none 
5: vpnsng: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/none 
6: vpnsyd: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/none 
7: pppoe-wan0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1492 qdisc fq_codel state UNKNOWN mode DEFAULT group default qlen 3
    link/ppp 
Interface status:
 interface vpnsyd is online 00h:02m:38s, uptime 00h:02m:39s and tracking is active
 interface vpnmel is online 00h:02m:35s, uptime 00h:02m:36s and tracking is active
 interface vpnsng is online 00h:02m:39s, uptime 00h:02m:39s and tracking is active

Current ipv4 policies:
# Warning: iptables-legacy tables present, use iptables-legacy to see them
balanced:
# Warning: iptables-legacy tables present, use iptables-legacy to see them
# Warning: iptables-legacy tables present, use iptables-legacy to see them
# Warning: iptables-legacy tables present, use iptables-legacy to see them
 vpnmel (50%)
# Warning: iptables-legacy tables present, use iptables-legacy to see them
 vpnsyd (50%)
vpnsngOnly:
# Warning: iptables-legacy tables present, use iptables-legacy to see them
# Warning: iptables-legacy tables present, use iptables-legacy to see them
# Warning: iptables-legacy tables present, use iptables-legacy to see them
 vpnsng (100%)

Current ipv6 policies:
# Warning: ip6tables-legacy tables present, use ip6tables-legacy to see them
balanced:
# Warning: ip6tables-legacy tables present, use ip6tables-legacy to see them
# Warning: ip6tables-legacy tables present, use ip6tables-legacy to see them
 unreachable
vpnsngOnly:
# Warning: ip6tables-legacy tables present, use ip6tables-legacy to see them
# Warning: ip6tables-legacy tables present, use ip6tables-legacy to see them
 unreachable

Directly connected ipv4 networks:
10.14.0.9
192.168.0.0/16
10.14.0.0
192.168.255.255
144.48.38.xx
127.0.0.0/8
127.0.0.0
192.168.0.1
127.0.0.1
127.255.255.255
192.168.0.0
10.14.255.255
89.187.163.xxx
224.0.0.0/3
10.14.0.0/16
203.134.4.xxx
10.14.0.2
125.168.75.xxx
10.14.0.3
45.248.76.xxx

Directly connected ipv6 networks:
fe80::/64
dddd:dddd:dddd::/64

Active ipv4 user rules:
   46  5368 - vpnsngOnly  all  --  *      *       192.168.1.202        0.0.0.0/0            
  253 17493 S https  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 443 
  313 23376 - balanced  all  --  *      *       0.0.0.0/0            0.0.0.0/0            

Active ipv6 user rules:
    0     0 S https  tcp      *      *       ::/0                 ::/0                 multiport dports 443 

lrwxrwxrwx    1 root     root            16 Apr 28 06:28 /etc/resolv.conf -> /tmp/resolv.conf
-rw-r--r--    1 root     root            36 May 24 11:26 /tmp/resolv.conf
-rw-r--r--    1 root     root           268 May 24 11:26 /tmp/resolv.conf.d/resolv.conf.auto
-rw-r--r--    1 root     root            46 May 24 11:26 /tmp/resolv.conf.ppp

/tmp/resolv.conf.d:
-rw-r--r--    1 root     root           268 May 24 11:26 resolv.conf.auto
==> /etc/resolv.conf <==
nameserver 127.0.0.1
nameserver ::1

==> /tmp/resolv.conf <==
nameserver 127.0.0.1
nameserver ::1

==> /tmp/resolv.conf.d <==
head: /tmp/resolv.conf.d: I/O error

==> /tmp/resolv.conf.ppp <==
nameserver 111.220.1.1
nameserver 111.220.2.2

==> /tmp/resolv.conf.d/resolv.conf.auto <==
# Interface vpnsyd
nameserver 149.154.159.92
nameserver 162.252.172.57
# Interface vpnmel
nameserver 149.154.159.92
nameserver 162.252.172.57
# Interface vpnsng
nameserver 149.154.159.92
nameserver 162.252.172.57
# Interface wan0
nameserver 1.1.1.1
nameserver 8.8.8.8
root@OpenWrt:~#
4 
uci set firewall.@zone[-1].mtu_fix='1'
uci commit firewall
service firewall restart

Try with mtufix.

2 Likes
5

I've applied to wan, but I will be trying, in my vpnclient and lan zone, later today too,
will get back to you. thanks!

6

In lan zone is not necessary to be applied.

1 Like
7

I've added mtu_fix in vpnclient zone, it moves forward a bit, brilliant! (https://www.google.com {test}-> https://www.speedtest.net/)
in "https://www.speedtest.net/", I monitor with F12 in browser, there are some socket errors.

maybe it's MTU again? shd I change all MTU in the interfaces to 1500~ish?

8

No, 1420 is fine.
What are the errors?

1 Like
9

I had messed up the mwan3 rules, after fixing it, all looks good! (unrelated to the problem, just sorted)

thanks so much, the mtu_fix is the solution.

10

@trendy
sorry, still have some wield problem, I found that in some pages, some package might have been lost somehow,
for example, when I check an email, with some pictures attached, I needa refresh several times in order to load them.

would you please tell how could I check/debug?
dmesg isn't enough.... :pray:

11

Are you using a load balancing policy?

1 Like
12

yes, is it not appropriate to load balance 2 WG connections?
it once work flawlessly, after reboot once/twice, it lost packages somehow.

not much error message shown in my browser... may be hidden somewhere in router.

13

Not that it is not appropriate, but certain sites might not see with a good eye that the same user is coming from different connections.

1 Like
14

I thought adding https sticky flag in mwan3 rule could get around it... I will monitor it once my problem solved.

with Multiple Multiguard Configs, my case WG0, WG1, WG2,
it's possible that they will randomly un-connectable if I set all them bring up on-boot.

I guess my service provider doesn't like simultaneously connection requests.
I tried to reboot with only 1 WG auto start, and start another WG manually, it managed to get around the problem (seems).
@trendy
Is there a way to delay the WG connections and start one by one on boot please?

15

Stickiness will work when you try to reach the same IP. And there are only a handful of cases when you can really see the aggregated speed in a load balancing policy.

You can disable them from coming up on boot, then add ifup name; sleep 10; ifup name2;... in rc.local.

1 Like
16

thanks so much @trendy , I've learnt a lot from your replies.

I've tried WG with multiple interfaces (Sydney/Singapore), and it seems that with multiple connections it will not start up consistently. (it works some time, but not in another, even I manually start/stop them one by one), which gives me impression that this version / or my VPN provider isn't happy with the initial connection handshake ~with multiple WGs. (I suspect this is the similar bug/limitation of WG mentioned in some other threads)

I've just changed to PBR(with OpenVPN, instead of WG), it looks it much more stable than I've thought for my use case.....

thanks again!

1 Like
17

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.