Routing ssh response over wan instead of vpn

gamma-577 · December 20, 2020, 12:19pm

Hello mates!

I am a newcomer to OpenWrt. I have a problem and I am looking for an efficient solution. I have a BTHH5A running OW 19.07.04.

I installed protonvpn ( free pack ) on tun0 and it is working fine. I wanted to ssh into my router from an external connection such as my mobile. So I forwarded wan port 31749 ( fake port ) to port 22 on the router.

This arrangement works very well when the vpn is off. When I connect to vpn the reply packets from the router are sent over tun0. I saw them in tcpdump's capture. But I want these packets to be sent back to wan instead of tun0.

I thought pbr is the answer. Maybe not...? Could you please suggest a solution?

Here is my pbr policy :

Local addresses / devices : 192.168.0.1 # my OW router
Local ports : 22
Remote addresses / domains : <blank> # the mobile company keeps on changing ip addresses every now and then
Remote ports : <blank> # maybe I should choose 22 ...? No idea really.
Protocol : Auto
Chain : output # also tried prerouting and forward out of sheer ignorence
Interface : wan # should it be something else  :roll_eyes:

No matter howsoever I play with these settings, the reply coming out of port 31749 is always routed to tun0.

The tcp dump commands and outputs on tun0 and wan are :

tun0 :

tcpdump -i tun0 host 92.41.221.75
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tun0, link-type RAW (Raw IP), capture size 262144 bytes
11:51:19.776008 IP cpc74084-crdf54-0-4-cust48.7-1.cable.virginm.net.31749 > 92.41.221.75.threembb.co.uk.24734: Flags [S.], seq 3836558
508, ack 566629716, win 28960, options [mss 1460,sackOK,TS val 3357627170 ecr 828784634,nop,wscale 4], length 0
11:51:20.775896 IP cpc74084-crdf54-0-4-cust48.7-1.cable.virginm.net.31749 > 92.41.221.75.threembb.co.uk.24734: Flags [S.], seq 3836558
508, ack 566629716, win 28960, options [mss 1460,sackOK,TS val 3357628170 ecr 828784634,nop,wscale 4], length 0
11:51:21.804805 IP cpc74084-crdf54-0-4-cust48.7-1.cable.virginm.net.31749 > 92.41.221.75.threembb.co.uk.24734: Flags [S.], seq 3836558
508, ack 566629716, win 28960, options [mss 1460,sackOK,TS val 3357629199 ecr 828784634,nop,wscale 4], length 0
11:51:22.772278 IP cpc74084-crdf54-0-4-cust48.7-1.cable.virginm.net.31749 > 92.41.221.75.threembb.co.uk.24734: Flags [S.], seq 3836558
508, ack 566629716, win 28960, options [mss 1460,sackOK,TS val 3357630167 ecr 828784634,nop,wscale 4], length 0
11:51:23.776552 IP cpc74084-crdf54-0-4-cust48.7-1.cable.virginm.net.31749 > 92.41.221.75.threembb.co.uk.24734: Flags [S.], seq 3836558
508, ack 566629716, win 28960, options [mss 1460,sackOK,TS val 3357631171 ecr 828784634,nop,wscale 4], length 0
11:51:24.772732 IP cpc74084-crdf54-0-4-cust48.7-1.cable.virginm.net.31749 > 92.41.221.75.threembb.co.uk.24734: Flags [S.], seq 3836558
508, ack 566629716, win 28960, options [mss 1460,sackOK,TS val 3357632167 ecr 828784634,nop,wscale 4], length 0
11:51:26.772016 IP cpc74084-crdf54-0-4-cust48.7-1.cable.virginm.net.31749 > 92.41.221.75.threembb.co.uk.24734: Flags [S.], seq 3836558
508, ack 566629716, win 28960, options [mss 1460,sackOK,TS val 3357634166 ecr 828784634,nop,wscale 4], length 0
11:51:28.780806 IP cpc74084-crdf54-0-4-cust48.7-1.cable.virginm.net.31749 > 92.41.221.75.threembb.co.uk.24734: Flags [S.], seq 3836558
508, ack 566629716, win 28960, options [mss 1460,sackOK,TS val 3357636175 ecr 828784634,nop,wscale 4], length 0
11:51:30.772854 IP cpc74084-crdf54-0-4-cust48.7-1.cable.virginm.net.31749 > 92.41.221.75.threembb.co.uk.24734: Flags [S.], seq 3836558
508, ack 566629716, win 28960, options [mss 1460,sackOK,TS val 3357638167 ecr 828784634,nop,wscale 4], length 0

wan :

tcpdump -i eth0.2 host 92.41.221.75
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0.2, link-type EN10MB (Ethernet), capture size 262144 bytes
12:03:51.332925 IP 92.41.221.75.threembb.co.uk.24704 > cpc74084-crdf54-0-4-cust48.7-1.cable.virginm.net.31749: Flags [S], seq 35249022
99, win 65535, options [mss 1220,nop,wscale 5,nop,nop,TS val 829535634 ecr 0,sackOK,eol], length 0
12:03:52.367877 IP 92.41.221.75.threembb.co.uk.24704 > cpc74084-crdf54-0-4-cust48.7-1.cable.virginm.net.31749: Flags [S], seq 35249022
99, win 65535, options [mss 1220,nop,wscale 5,nop,nop,TS val 829536635 ecr 0,sackOK,eol], length 0
12:03:53.368229 IP 92.41.221.75.threembb.co.uk.24704 > cpc74084-crdf54-0-4-cust48.7-1.cable.virginm.net.31749: Flags [S], seq 35249022
99, win 65535, options [mss 1220,nop,wscale 5,nop,nop,TS val 829537636 ecr 0,sackOK,eol], length 0
12:03:54.368262 IP 92.41.221.75.threembb.co.uk.24704 > cpc74084-crdf54-0-4-cust48.7-1.cable.virginm.net.31749: Flags [S], seq 35249022
99, win 65535, options [mss 1220,nop,wscale 5,nop,nop,TS val 829538637 ecr 0,sackOK,eol], length 0
12:03:56.367629 IP 92.41.221.75.threembb.co.uk.24704 > cpc74084-crdf54-0-4-cust48.7-1.cable.virginm.net.31749: Flags [S], seq 35249022
99, win 65535, options [mss 1220,nop,wscale 5,nop,nop,TS val 829540638 ecr 0,sackOK,eol], length 0
12:03:58.378004 IP 92.41.221.75.threembb.co.uk.24704 > cpc74084-crdf54-0-4-cust48.7-1.cable.virginm.net.31749: Flags [S], seq 35249022
99, win 65535, options [mss 1220,nop,wscale 5,nop,nop,TS val 829542639 ecr 0,sackOK,eol], length 0
12:04:02.367775 IP 92.41.221.75.threembb.co.uk.24704 > cpc74084-crdf54-0-4-cust48.7-1.cable.virginm.net.31749: Flags [S], seq 35249022
99, win 65535, options [mss 1220,nop,wscale 5,nop,nop,TS val 829546640 ecr 0,sackOK,eol], length 0
12:04:10.378232 IP 92.41.221.75.threembb.co.uk.24704 > cpc74084-crdf54-0-4-cust48.7-1.cable.virginm.net.31749: Flags [S], seq 35249022
99, win 65535, options [mss 1220,nop,wscale 5,nop,nop,TS val 829554640 ecr 0,sackOK,eol], length 0

Where do I go from here? Any suggestions, please?

Thank you!

-Gamma

Edit : typo

vgaetera · December 20, 2020, 3:53pm

Disable gateway redirection in the VPN client profile and add a policy to route LAN to VPN by default.

ulmwind · December 20, 2020, 5:11pm

Add corresponding policy to mark packets by port number and route them via WAN, not tun interface.

ulmwind · December 20, 2020, 5:12pm

No, just policy to route via tun for special port number. If default route is set via WAN, why is VPN needed?

gamma-577 · December 20, 2020, 6:45pm

Hi @vgaetera,

Thanks for the suggestion. How do I disable gateway redirection? I did not find any such option. Perhaps that's enabled by default. On quick googling I found adding this line :

redirect-gateway def1

to client profile.

Should I do that?

And of course I still need to add a policy ( second line ) to pbr to route traffic to tun0.

Thank you!
-Gamma

Edit : typo

vgaetera · December 20, 2020, 6:52pm

Remove that:

And add this:

pull-filter ignore redirect-gateway

gamma-577 · December 20, 2020, 7:15pm

Thanks for the suggestion. Protonvpn does not like me disconnecting and reconnecting frequently. I have to be patient and wait till tomorrow.

Thank you once again.
-Gamma

gamma-577 · December 21, 2020, 9:43am

Hi,

I made a mistake in starting vpn-policy-routing without starting openvpn service. Protonvpn did not like that and it banned me from connecting for the day. I guess I can connect in the evening (GMT).

BTW, I could not locate any material on service dependencies. I read a guide here : https://openwrt.org/docs/guide-developer/procd-init-script-example

But it does not describ how to make a service dependent on other. Say if openvpn service is not running the vpn-policy-routing should not start in first place. Is there any such setting?

Thank you!
-Gamma

vgaetera · December 21, 2020, 6:14pm

VPN-PBR only affects traffic from your LAN and the order in which the services start is irrelevant.

gamma-577 · December 21, 2020, 10:51pm

Hi @vgaetera,

Thanks for the explanation.

Here are my latest findings :

Ignoring gateway pull has badly exposed my ip address. The https://dnsleaktest.com not only found out my isp's dns but also my vpn's dns!
Apart from that the things are running smoothly. The only downside is my ssh connection to router over lan is lost. Obviously every packed coming out of port 22 is going to wan instead of br-lan's client. But I can get a workaround by creating another instance of dropbear listening on some obscure port reserved for the ssh traffic port forwarded from wan.

Here are my service starting messages :

/etc/init.d/vpn-policy-routing start

Creating table 'wan/eth0.2/86.10.53.1' [✓] # fake
Creating table 'tun0if/tun0/10.16.0.11' [✓]
Routing 'ssh' via wan [✓]
Routing 'Default' via tun0if [✓]
vpn-policy-routing 0.2.1-13 started with gateways:
wan/eth0.2/86.10.53.1 [✓] # fake
tun0if/tun0/10.16.0.11
vpn-policy-routing 0.2.1-13 monitoring interfaces: wan tun0if .

And this is my pbr config :

uci show vpn-policy-routing

vpn-policy-routing.config=vpn-policy-routing
vpn-policy-routing.config.verbosity='2'
vpn-policy-routing.config.src_ipset='0'
vpn-policy-routing.config.dest_ipset='dnsmasq.ipset'
vpn-policy-routing.config.ipv6_enabled='0'
vpn-policy-routing.config.supported_interface=''
vpn-policy-routing.config.ignored_interface='vpnserver wgserver'
vpn-policy-routing.config.boot_timeout='30'
vpn-policy-routing.config.iptables_rule_option='append'
vpn-policy-routing.config.iprule_enabled='0'
vpn-policy-routing.config.webui_sorting='1'
vpn-policy-routing.config.webui_supported_protocol='tcp' 'udp' 'tcp udp' 'icmp' 'all'
vpn-policy-routing.config.webui_enable_column='1'
vpn-policy-routing.config.webui_protocol_column='1'
vpn-policy-routing.config.webui_chain_column='1'
vpn-policy-routing.config.strict_enforcement='0'
vpn-policy-routing.config.enabled='1'
vpn-policy-routing.@include[0]=include
vpn-policy-routing.@include[0].path='/etc/vpn-policy-routing.netflix.user'
vpn-policy-routing.@include[0].enabled='0'
vpn-policy-routing.@include[1]=include
vpn-policy-routing.@include[1].path='/etc/vpn-policy-routing.aws.user'
vpn-policy-routing.@include[1].enabled='0'
vpn-policy-routing.@policy[0]=policy
vpn-policy-routing.@policy[0].interface='wan'
vpn-policy-routing.@policy[0].name='ssh'
vpn-policy-routing.@policy[0].src_addr='192.168.0.1'
vpn-policy-routing.@policy[0].src_port='22'
vpn-policy-routing.@policy[0].chain='OUTPUT'
vpn-policy-routing.@policy[1]=policy
vpn-policy-routing.@policy[1].name='Default'
vpn-policy-routing.@policy[1].interface='tun0if'
vpn-policy-routing.@policy[1].src_addr='0.0.0.0/0'
vpn-policy-routing.@policy[1].dest_addr='0.0.0.0/0'
vpn-policy-routing.@policy[1].chain='OUTPUT'

How do I stop the exposure? Your guidance will be helpful.

Thank you!

-Gamma

Edit : typo

vgaetera · December 21, 2020, 11:44pm

You can provide a custom public DNS with DHCP and reconnect the LAN clients to apply changes.
Or add a route to the DNS server via the VPN interface.
Or use this method:

gamma-577 · December 22, 2020, 10:04am

Hi @vgaetera,

Sorry for my ignorence, how do I mark packets by port number? Also how do I add a route to the DNS server via the VPN interface? Are there any tutorial for that?

I tried to customise public DNS with DHCP, but something went wrong and my connectivity messed up. I will try this option in after office hours (evening GMT). In the meantime is there any reading material for the other two options?

Thank you!
-Gamma

gamma-577 · December 23, 2020, 11:21pm

Hello @vgaetera,

I added custom public DNS with DHCP and of course included your line pull-filter ignore redirect-gateway in the vpn client profile. I also commented out pull command from there.

After starting pbr service, I got a lot many arp requests on wan. Here is tcpdump.

tcpdump -i eth0.2

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0.2, link-type EN10MB (Ethernet), capture size 262144 bytes
22:58:26.715228 ARP, Request who-has 86.10.55.216 tell cpc74084-crdf54-0-4-gw.3-3.cable.virginm.net, length 46
22:58:26.743208 ARP, Request who-has 82.8.179.196 tell 82.8.176.1, length 46
22:58:26.749228 ARP, Request who-has 86.25.147.205 tell 86.25.144.1, length 46
22:58:26.803199 ARP, Request who-has 86.10.87.208 tell 86.10.84.1, length 46
22:58:26.825322 ARP, Request who-has 86.25.147.233 tell 86.25.144.1, length 46
22:58:26.850206 ARP, Request who-has 86.10.42.54 tell 86.10.40.1, length 46
22:58:26.886263 ARP, Request who-has 82.8.179.238 tell 82.8.176.1, length 46
22:58:27.003248 ARP, Request who-has 86.10.83.112 tell cpc74084-crdf54-0-4-gw.3-3.cable.virginm.net, length 46
22:58:27.018219 ARP, Request who-has 86.10.87.248 tell 86.10.84.1, length 46
22:58:27.128220 ARP, Request who-has 86.25.144.177 tell 86.25.144.1, length 46
22:58:27.131160 ARP, Request who-has 86.19.218.183 tell 86.19.216.1, length 46
22:58:27.138923 IP 80.249.99.164 > cpc74084-crdf54-0-4-cust48.7-1.cable.virginm.net: ICMP echo request, id 32871, seq 62379, length 8
22:58:27.152656 IP 80.249.99.164 > cpc74084-crdf54-0-4-cust48.7-1.cable.virginm.net: ICMP echo request, id 32791, seq 38967, length 8
22:58:27.163242 ARP, Request who-has 86.13.179.199 tell 86.13.176.1, length 46
22:58:27.220276 ARP, Request who-has 86.6.225.98 tell 86.6.224.1, length 46
22:58:27.229232 ARP, Request who-has 86.13.195.95 tell 86.13.192.1, length 46
22:58:27.244242 ARP, Request who-has 86.13.195.150 tell 86.13.192.1, length 46
22:58:27.259234 ARP, Request who-has 86.21.27.243 tell 86.21.24.1, length 46
22:58:27.386325 ARP, Request who-has 10.226.199.80 tell 10.226.196.1, length 46
22:58:27.397262 ARP, Request who-has 86.25.147.229 tell 86.25.144.1, length 46
22:58:27.399137 ARP, Request who-has 86.10.43.231 tell 86.10.40.1, length 46
22:58:27.447555 IP cpc74084-crdf54-0-4-cust48.7-1.cable.virginm.net.61993 > 104.16.248.249.443: Flags [P.], seq 3599453079:3599453135,
 ack 2268753375, win 253, length 56
22:58:27.447635 IP cpc74084-crdf54-0-4-cust48.7-1.cable.virginm.net.61993 > 104.16.248.249.443: Flags [P.], seq 56:141, ack 1, win 253
, length 85
22:58:27.468298 ARP, Request who-has 86.10.40.53 tell 86.10.40.1, length 46
22:58:27.473304 IP 104.16.248.249.443 > cpc74084-crdf54-0-4-cust48.7-1.cable.virginm.net.61993: Flags [.], ack 141, win 68, length 0
22:58:27.474165 IP cpc74084-crdf54-0-4-cust48.7-1.cable.virginm.net.62498 > 139.59.210.197.443: Flags [S], seq 2830387509, win 64240,
options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
22:58:27.477234 IP 104.16.248.249.443 > cpc74084-crdf54-0-4-cust48.7-1.cable.virginm.net.61993: Flags [P.], seq 1:177, ack 141, win 68
, length 176
22:58:27.477271 IP 104.16.248.249.443 > cpc74084-crdf54-0-4-cust48.7-1.cable.virginm.net.61993: Flags [P.], seq 177:208, ack 141, win
68, length 31
22:58:27.487080 IP cpc74084-crdf54-0-4-cust48.7-1.cable.virginm.net.61993 > 104.16.248.249.443: Flags [.], ack 208, win 253, length 0
22:58:27.489211 ARP, Request who-has 82.27.219.209 tell 82.27.216.1, length 46
22:58:27.520187 IP 139.59.210.197.443 > cpc74084-crdf54-0-4-cust48.7-1.cable.virginm.net.62498: Flags [S.], seq 2591360341, ack 283038
7510, win 29200, options [mss 1460,nop,wscale 7], length 0
22:58:27.521840 IP cpc74084-crdf54-0-4-cust48.7-1.cable.virginm.net.62498 > 139.59.210.197.443: Flags [.], ack 1, win 256, length 0
22:58:31.802141 ARP, Request who-has 10.36.164.24 tell 10.36.164.1, length 46
22:58:41.775934 ARP, Request who-has 82.27.219.189 tell 82.27.216.1, length 46
22:58:51.751759 ARP, Request who-has 86.13.179.162 tell 86.13.176.1, length 46
22:59:01.836603 ARP, Request who-has 86.7.61.220 tell 86.7.60.1, length 46
22:59:06.860577 ARP, Request who-has 82.27.219.236 tell 82.27.216.1, length 46
22:59:16.800382 ARP, Request who-has 86.10.32.59 tell 86.10.32.1, length 46
22:59:21.822432 ARP, Request who-has 86.21.27.71 tell 86.21.24.1, length 46
22:59:26.811225 ARP, Request who-has 82.27.216.72 tell 82.27.216.1, length 46
22:59:31.874129 ARP, Request who-has 86.10.83.139 tell cpc74084-crdf54-0-4-gw.3-3.cable.virginm.net, length 46
^C22:59:36.960098 ARP, Request who-has 86.10.87.228 tell 86.10.84.1, length 46

42 packets captured
5530 packets received by filter
5457 packets dropped by kernel

The normal traffic is routed via tun0, I guess. But somehow my ip is exposed. Also the dns resolution of all wifi clients and router takes a hit.

Here is the protonvpn client profile file :

client
dev tun
proto udp

remote nl-free-01.protonvpn.com 5060
remote nl-free-01.protonvpn.com 80
remote nl-free-01.protonvpn.com 1194
remote nl-free-01.protonvpn.com 443
remote nl-free-01.protonvpn.com 4569
remote nl-free-02.protonvpn.com 80
remote nl-free-02.protonvpn.com 4569
remote nl-free-02.protonvpn.com 1194
remote nl-free-02.protonvpn.com 5060
remote nl-free-02.protonvpn.com 443
remote nl-free-03.protonvpn.com 4569
remote nl-free-03.protonvpn.com 80
remote nl-free-03.protonvpn.com 443
remote nl-free-03.protonvpn.com 5060
remote nl-free-03.protonvpn.com 1194
remote nl-free-04.protonvpn.com 443
remote nl-free-04.protonvpn.com 5060
remote nl-free-04.protonvpn.com 4569
remote nl-free-04.protonvpn.com 1194
remote nl-free-04.protonvpn.com 80
remote nl-free-05.protonvpn.com 4569
remote nl-free-05.protonvpn.com 1194
remote nl-free-05.protonvpn.com 443
remote nl-free-05.protonvpn.com 80
remote nl-free-05.protonvpn.com 5060
remote nl-free-06.protonvpn.com 80
remote nl-free-06.protonvpn.com 5060
remote nl-free-06.protonvpn.com 1194
remote nl-free-06.protonvpn.com 443
remote nl-free-06.protonvpn.com 4569
remote nl-free-07.protonvpn.com 5060
remote nl-free-07.protonvpn.com 80
remote nl-free-07.protonvpn.com 1194
remote nl-free-07.protonvpn.com 4569
remote nl-free-07.protonvpn.com 443
remote nl-free-08.protonvpn.com 80
remote nl-free-08.protonvpn.com 4569
remote nl-free-08.protonvpn.com 1194
remote nl-free-08.protonvpn.com 5060
remote nl-free-08.protonvpn.com 443
remote nl-free-09.protonvpn.com 5060
remote nl-free-09.protonvpn.com 443
remote nl-free-09.protonvpn.com 1194
remote nl-free-09.protonvpn.com 80
remote nl-free-09.protonvpn.com 4569

remote-random
resolv-retry infinite
nobind
cipher AES-256-CBC
auth SHA512
comp-lzo no
verb 3

tun-mtu 1500
tun-mtu-extra 32
mssfix 1450
persist-key
persist-tun

reneg-sec 0

remote-cert-tls server
auth-user-pass /etc/openvpn/nlfreeudp.auth
# pull
fast-io
# gateway redirected to old default
# redirect-gateway def1
pull-filter ignore redirect-gateway

script-security 2
up /etc/openvpn/client.sh
down /etc/openvpn/client.sh

<ca>
-----BEGIN CERTIFICATE-----
MIIFozCCA4ugAwIBAgIBATANBgkqhkiG9w0BAQ0FADBAMQswCQYDVQQGEwJDSDEV
blah-blah-blah
DK/yPwECUcPgHIeXiRjHnJt0Zcm23O2Q3RphpU+1SO3XixsXpOVOYP6rJIXW9bMZ
A1gTTlpi7A==
-----END CERTIFICATE-----
</ca>

key-direction 1
<tls-auth>
# 2048 bit OpenVPN static key
-----BEGIN OpenVPN Static key V1-----
6acef03f62675b4b1bbd03e53b187727
blah-blah-blah
16672ea16c012664f8a9f11255518deb
-----END OpenVPN Static key V1-----
</tls-auth>

Any idea what's hapening?

Thank you!
-Gamma

Edit :

dns resolution symptom added
protonvpn client profile listed

gamma-577 · December 28, 2020, 9:49pm

Hi @vgaetera,

After enabling pull directive in openvpn profile file, I found dns working properly. I could also connect via wan over ssh. The only problem is my ip exposure. Somehow I need to hide it from wan interface.

Any idea how to proceed?

Thank you!
-Gamma