Luci timeout when connecting from ZeroTier VPN

Suxsem · February 4, 2021, 11:26am

Hello everyone!

I think this is a tricky one.

I have a device running openwrt snapshot with Luci installed.

My main wan connection is a NordVPN tunnel with the Wireguard protocol.
The router and every host in my network can access Internet through the NordVPN public IP ( 192.145.127.xxx) and I can access the luci web panel from my browser at http://192.168.11.1/cgi-bin/luci/ (192.168.11.1 is the private router IP). I think you can ignore NordVPN, you can think of it as my ISP.

Now...i installed ZeroTier, that is basically a peer-to-peer VPN that allows to connect different devices together also if they are far away putting them in a virtual LAN (it uses various NAT traversal techniques to do that). One (not only) of the goal is to reach the router despite NordVPN shares the same public IP with different customers).
The ZeroTier virtual interface has its own IP, in my case is 172.22.98.82.

I'm able to remotely reach my router at 172.22.98.82 (obviously from another device that has joined the ZeroTier virtual network): I can ping it, I can SSH into it, I can make http requests to uhttpd (es. http://172.22.98.82/mypage.html).

Here is the problem: for some reason, /cgi-bin/luci (only this specific path!!!) refuse the connection when the request came from the ZeroTier adapter.

I cannot find any log in the router and I don't know how to troubleshoot this.

Thank you guys!

anon50098793 · February 4, 2021, 11:34am

uci show firewall | grep -A6 network

Suxsem · February 4, 2021, 11:38am

uci show firewall | grep -A6 network
firewall.@zone[0].network='lan'
firewall.@zone[1]=zone
firewall.@zone[1].name='wan'
firewall.@zone[1].input='REJECT'
firewall.@zone[1].output='ACCEPT'
firewall.@zone[1].forward='REJECT'
firewall.@zone[1].masq='1'
--
firewall.@zone[1].network='wan wan6 nordvpn wwan'
firewall.@forwarding[0]=forwarding
firewall.@forwarding[0].src='lan'
firewall.@forwarding[0].dest='wan'
firewall.@rule[0]=rule
firewall.@rule[0].name='Allow-DHCP-Renew'
firewall.@rule[0].src='wan'
--
firewall.@zone[2].network='ZeroTier'
firewall.@forwarding[1]=forwarding
firewall.@forwarding[1].src='ZeroTier'
firewall.@forwarding[1].dest='lan'
firewall.@forwarding[2]=forwarding
firewall.@forwarding[2].src='lan'
firewall.@forwarding[2].dest='ZeroTier'

anon50098793 · February 4, 2021, 11:40am

Please use the "Preformatted text </>" button for logs, scripts, configs and general console output.

Please edit your post accordingly. Thank you!

Suxsem · February 4, 2021, 11:42am

done a couple of minutes after posting, sorry for that
I hope it's clearer now

vgaetera · February 4, 2021, 12:04pm

Activate the required VPN connections and post the output to pastebin.com redacting the private parts:

ubus call system board; uci show network; uci show firewall; uci show uhttpd; \
ip address show; ip route show table all; ip rule show; iptables-save; \
netstat -l -n -p | grep -e uhttpd

Suxsem · February 4, 2021, 1:18pm

here it is: ubus call system board; uci show network; uci show firewall; uci show uhttpd; \ - Pastebin.com

please keep in mind that the problem is related to a specific path so I'm not sure it's network related.
thank you!

anon50098793 · February 4, 2021, 1:27pm

### from-remote
curl -I -S http://<IP>
curl -k -I -S https://<IP>
### on-router
ps wwww | grep uhttpd

you are not listening on 443 yet you have it enabled ( no redirect )... something is whacky somewhere...

tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      5686/uhttpd
tcp        0      0 :::80                   :::*                    LISTEN      5686/uhttpd

Suxsem · February 4, 2021, 1:43pm

I'm not interested in HTTPS right now, just http:

root@orangepipcplus:~# curl -k -I -S https://172.22.98.82/
curl: (7) Failed to connect to 172.22.98.82 port 443: Connection refused

root@orangepipcplus:~# curl -I -S http://172.22.98.82/
HTTP/1.1 200 OK
Connection: Keep-Alive
Keep-Alive: timeout=20
ETag: "14d-20a-60145980"
Last-Modified: Fri, 29 Jan 2021 18:52:48 GMT
Date: Thu, 04 Feb 2021 13:40:15 GMT
Content-Type: text/html
Content-Length: 522

root@orangepipcplus:~# curl -I -S http://172.22.98.82/cgi-bin/luci
HTTP/1.1 403 Forbidden
Connection: Keep-Alive
Keep-Alive: timeout=20
X-LuCI-Login-Required: yes
Content-Type: text/html
Cache-Control: no-cache
Expires: 0
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff

Suxsem · February 4, 2021, 1:54pm

Ok there is something really strange.

If I send the request through curl, the requested is server (with 403 but it's served).
If i try the same request from the browser of from postman the request just times out.

I need to compare the two requests in order to discover what's different (maybe some http headers?)

Suxsem · February 4, 2021, 2:01pm

Ok I inspected the curl request and it's using the HEAD http method instead of the GET method to send the request, this is why it works from curl but not from the browser.

Now I wonder why luci is rejecting the GET request if it's coming from the zerotier vpn, while the other paths are server correctly.

Suxsem · February 4, 2021, 2:42pm

I think this is a bug of Luci. Check the following output:


C:\Users\xxx\Downloads>curl -I -S http://172.22.98.82/cgi-bin/luci
HTTP/1.1 403 Forbidden
Connection: Keep-Alive
Keep-Alive: timeout=20
X-LuCI-Login-Required: yes
Content-Type: text/html
Cache-Control: no-cache
Expires: 0
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff

C:\Users\xxx\Downloads>

C:\Users\xxx\Downloads>curl -X GET -I -S http://172.22.98.82/cgi-bin/luci
HTTP/1.1 403 Forbidden
Connection: Keep-Alive
Transfer-Encoding: chunked
_

and after some time (a couple of minutes)...

C:\Users\xxx\Downloads>curl -X GET -I -S http://172.22.98.82/cgi-bin/luci
HTTP/1.1 403 Forbidden
Connection: Keep-Alive
Transfer-Encoding: chunked
curl: (56) Recv failure: Connection was reset

when using any http method other then HEAD the connection just hangs.

Please note that anything other then /cgi-bin/luci works:

C:\Users\xxx\Downloads>curl -X GET -I -S http://172.22.98.82/cgi-bin/daad
HTTP/1.1 404 Not Found
Connection: Keep-Alive
Transfer-Encoding: chunked
Keep-Alive: timeout=20
Content-Type: text/html

Can you suggest me something to the debug this behaviour?
Thanks!

anon50098793 · February 4, 2021, 3:06pm

uci show | grep '/ubus'

Suxsem · February 4, 2021, 3:51pm

> uci show | grep '/ubus'
luci.main.ubuspath='/ubus/'
rpcd.@rpcd[0].socket='/var/run/ubus/ubus.sock'
uhttpd.main.ubus_prefix='/ubus'

anon50098793 · February 4, 2021, 3:55pm

looks ok... strace uhttpd / check file permissions...?

Suxsem · February 4, 2021, 4:25pm

I don't think is a file permission issue, why it occurs only when requested from zerotier vpn?

maybe strace is the only way...

vgaetera · February 4, 2021, 6:29pm

This sounds like MTU-related issues, or incorrect proxy configuration in the system/browser.

uci set firewall.@zone[2].mtu_fix="1"
uci commit firewall
/etc/init.d/firewall restart

In addition, the current ZT tunnel appears to be built over the WG tunnel.
This can negatively affect MTU, latency and speed.
You should likely set up routing to use WAN instead of WG to run the ZT tunnel.

If you want to proceed with diagnostics, check tcpdump on OpenWrt when using browser.

Suxsem · February 4, 2021, 6:44pm

I think you're right.
There is something wrong with the ZT tunnel itself.

Unfortunately mtu_fix did not solved the issue.

It's ok for me to route ZeroTier through WAN instead of NordVPN but I'm unable to do so.
I setup the Wireguard tunnel as the default route for the system. I can tell ZeroTier to use wan instead of Wireguard?

Thank you

vgaetera · February 4, 2021, 7:25pm

Suxsem · February 5, 2021, 7:10pm

I can confirm that using WAN for ZT fixes the issue!

I ended up using luci-app-vpn-policy-routing to control the vpn policy.

Thank you for your incredible help!