Do a restart of the dnsmasq and vpn-pbr to make it work. Take a quick snapshot of the iptables.
iptables-save -c -t mangle
Wait until it stops working and take another snapshot.
Didn’t get it to work by restarting vpn-pbr and dnsmasq. I had to restart openVPN to get it to work...
Now it is working...
# Generated by iptables-save v1.8.3 on Sun Jan 3 13:57:32 2021
*mangle
:PREROUTING ACCEPT [12400:6831658]
:INPUT ACCEPT [5491:3301210]
:FORWARD ACCEPT [5695:2691360]
:OUTPUT ACCEPT [5171:1010387]
:POSTROUTING ACCEPT [10788:3682652]
:VPR_FORWARD - [0:0]
:VPR_INPUT - [0:0]
:VPR_OUTPUT - [0:0]
:VPR_PREROUTING - [0:0]
[12406:6833377] -A PREROUTING -m mark --mark 0x0/0xff0000 -j VPR_PREROUTING
[5492:3301292] -A INPUT -m mark --mark 0x0/0xff0000 -j VPR_INPUT
[97:6196] -A FORWARD -o tun0 -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone VyprVPN MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
[91:5428] -A FORWARD -i tun0 -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone VyprVPN MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
[5557:2674479] -A FORWARD -m mark --mark 0x0/0xff0000 -j VPR_FORWARD
[5175:1011291] -A OUTPUT -m mark --mark 0x0/0xff0000 -j VPR_OUTPUT
[0:0] -A VPR_FORWARD -m set --match-set VyprVPN dst -j MARK --set-xmark 0x20000/0xff0000
[0:0] -A VPR_FORWARD -m set --match-set lan dst -j MARK --set-xmark 0x10000/0xff0000
[0:0] -A VPR_INPUT -m set --match-set VyprVPN dst -j MARK --set-xmark 0x20000/0xff0000
[0:0] -A VPR_INPUT -m set --match-set lan dst -j MARK --set-xmark 0x10000/0xff0000
[0:0] -A VPR_OUTPUT -m set --match-set VyprVPN dst -j MARK --set-xmark 0x20000/0xff0000
[0:0] -A VPR_OUTPUT -m set --match-set lan dst -j MARK --set-xmark 0x10000/0xff0000
[0:0] -A VPR_PREROUTING -m set --match-set VyprVPN_mac src -j MARK --set-xmark 0x20000/0xff0000
[0:0] -A VPR_PREROUTING -m set --match-set VyprVPN_ip src -j MARK --set-xmark 0x20000/0xff0000
[0:0] -A VPR_PREROUTING -m set --match-set VyprVPN dst -j MARK --set-xmark 0x20000/0xff0000
[0:0] -A VPR_PREROUTING -m set --match-set lan_mac src -j MARK --set-xmark 0x10000/0xff0000
[0:0] -A VPR_PREROUTING -m set --match-set lan_ip src -j MARK --set-xmark 0x10000/0xff0000
[138:16881] -A VPR_PREROUTING -m set --match-set lan dst -j MARK --set-xmark 0x10000/0xff0000
COMMIT
# Completed on Sun Jan 3 13:57:32 2021
When it stops working i will paste it here
And the ipset list lan to verify that the correct destination IPs are matched.
Name: lan
Type: hash:net
Revision: 6
Header: family inet hashsize 1024 maxelem 65536 comment
Size in memory: 951
References: 4
Number of entries: 4
Members:
52.29.58.77 comment "meineip2 www.wieistmeineip.de: 52.29.58.77"
52.29.79.39 comment "meineip: wieistmeineip.de"
35.157.134.147 comment "meineip2 www.wieistmeineip.de: 35.157.134.147"
52.28.125.65 comment "meineip2 www.wieistmeineip.de: 52.28.125.65"
So far it just works. Murphys Law. 
I'm not complaining, but it would be nice to know what the problem was.
1 Like
Okay. It worked for more than 3 hours without any problem.
So I decided to add some more domains and restart the service.
After making the changes it didn’t work and I restarted vpn-pbr, dnsmasq and openvpn several times... after some restarts and a couple of minutes it worked.
root@OpenWrt:~# iptables-save -c -t mangle
# Generated by iptables-save v1.8.3 on Sun Jan 3 17:24:08 2021
*mangle
:PREROUTING ACCEPT [44497:42257784]
:INPUT ACCEPT [19020:21257271]
:FORWARD ACCEPT [24404:20252302]
:OUTPUT ACCEPT [8789:1502034]
:POSTROUTING ACCEPT [33182:21753461]
:VPR_FORWARD - [0:0]
:VPR_INPUT - [0:0]
:VPR_OUTPUT - [0:0]
:VPR_PREROUTING - [0:0]
[44505:42264425] -A PREROUTING -m mark --mark 0x0/0xff0000 -j VPR_PREROUTING
[19020:21257271] -A INPUT -m mark --mark 0x0/0xff0000 -j VPR_INPUT
[94:6016] -A FORWARD -o tun0 -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone VyprVPN MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
[94:5592] -A FORWARD -i tun0 -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone VyprVPN MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
[24331:20242110] -A FORWARD -m mark --mark 0x0/0xff0000 -j VPR_FORWARD
[8789:1502034] -A OUTPUT -m mark --mark 0x0/0xff0000 -j VPR_OUTPUT
[0:0] -A VPR_FORWARD -m set --match-set VyprVPN dst -j MARK --set-xmark 0x20000/0xff0000
[0:0] -A VPR_FORWARD -m set --match-set lan dst -j MARK --set-xmark 0x10000/0xff0000
[0:0] -A VPR_INPUT -m set --match-set VyprVPN dst -j MARK --set-xmark 0x20000/0xff0000
[0:0] -A VPR_INPUT -m set --match-set lan dst -j MARK --set-xmark 0x10000/0xff0000
[0:0] -A VPR_OUTPUT -m set --match-set VyprVPN dst -j MARK --set-xmark 0x20000/0xff0000
[0:0] -A VPR_OUTPUT -m set --match-set lan dst -j MARK --set-xmark 0x10000/0xff0000
[0:0] -A VPR_PREROUTING -m set --match-set VyprVPN_mac src -j MARK --set-xmark 0x20000/0xff0000
[0:0] -A VPR_PREROUTING -m set --match-set VyprVPN_ip src -j MARK --set-xmark 0x20000/0xff0000
[0:0] -A VPR_PREROUTING -m set --match-set VyprVPN dst -j MARK --set-xmark 0x20000/0xff0000
[0:0] -A VPR_PREROUTING -m set --match-set lan_mac src -j MARK --set-xmark 0x10000/0xff0000
[0:0] -A VPR_PREROUTING -m set --match-set lan_ip src -j MARK --set-xmark 0x10000/0xff0000
[73:10192] -A VPR_PREROUTING -m set --match-set lan dst -j MARK --set-xmark 0x10000/0xff0000
COMMIT
# Completed on Sun Jan 3 17:24:08 2021
root@OpenWrt:~# ipset list lan
Name: lan
Type: hash:net
Revision: 6
Header: family inet hashsize 1024 maxelem 65536 comment
Size in memory: 1154
References: 4
Number of entries: 6
Members:
91.123.100.206 comment "Zattoo4: zahs.tv"
91.123.100.227 comment "Zattoo3: zattoo.com"
45.60.121.229 comment "Zattoo2: digicert.com"
45.60.196.209 comment "Zattoo1: geotrust.com"
52.29.79.39 comment "meineip: wieistmeineip.de"
52.28.125.65 comment "meineip2: www.wieistmeineip.de"
like before, two minutes later it stoped working... but the output seems to be the same...
root@OpenWrt:~# iptables-save -c -t mangle
# Generated by iptables-save v1.8.3 on Sun Jan 3 17:28:14 2021
*mangle
:PREROUTING ACCEPT [136299:136181791]
:INPUT ACCEPT [59002:70205493]
:FORWARD ACCEPT [74892:64347437]
:OUTPUT ACCEPT [24648:3782390]
:POSTROUTING ACCEPT [99515:68128340]
:VPR_FORWARD - [0:0]
:VPR_INPUT - [0:0]
:VPR_OUTPUT - [0:0]
:VPR_PREROUTING - [0:0]
[136307:136188432] -A PREROUTING -m mark --mark 0x0/0xff0000 -j VPR_PREROUTING
[59002:70205493] -A INPUT -m mark --mark 0x0/0xff0000 -j VPR_INPUT
[191:12224] -A FORWARD -o tun0 -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone VyprVPN MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
[185:11028] -A FORWARD -i tun0 -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone VyprVPN MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
[74786:64332005] -A FORWARD -m mark --mark 0x0/0xff0000 -j VPR_FORWARD
[24648:3782390] -A OUTPUT -m mark --mark 0x0/0xff0000 -j VPR_OUTPUT
[0:0] -A VPR_FORWARD -m set --match-set VyprVPN dst -j MARK --set-xmark 0x20000/0xff0000
[0:0] -A VPR_FORWARD -m set --match-set lan dst -j MARK --set-xmark 0x10000/0xff0000
[0:0] -A VPR_INPUT -m set --match-set VyprVPN dst -j MARK --set-xmark 0x20000/0xff0000
[0:0] -A VPR_INPUT -m set --match-set lan dst -j MARK --set-xmark 0x10000/0xff0000
[0:0] -A VPR_OUTPUT -m set --match-set VyprVPN dst -j MARK --set-xmark 0x20000/0xff0000
[0:0] -A VPR_OUTPUT -m set --match-set lan dst -j MARK --set-xmark 0x10000/0xff0000
[0:0] -A VPR_PREROUTING -m set --match-set VyprVPN_mac src -j MARK --set-xmark 0x20000/0xff0000
[0:0] -A VPR_PREROUTING -m set --match-set VyprVPN_ip src -j MARK --set-xmark 0x20000/0xff0000
[0:0] -A VPR_PREROUTING -m set --match-set VyprVPN dst -j MARK --set-xmark 0x20000/0xff0000
[0:0] -A VPR_PREROUTING -m set --match-set lan_mac src -j MARK --set-xmark 0x10000/0xff0000
[0:0] -A VPR_PREROUTING -m set --match-set lan_ip src -j MARK --set-xmark 0x10000/0xff0000
[106:15432] -A VPR_PREROUTING -m set --match-set lan dst -j MARK --set-xmark 0x10000/0xff0000
COMMIT
# Completed on Sun Jan 3 17:28:14 2021
root@OpenWrt:~# ipset list lan
Name: lan
Type: hash:net
Revision: 6
Header: family inet hashsize 1024 maxelem 65536 comment
Size in memory: 1154
References: 4
Number of entries: 6
Members:
91.123.100.206 comment "Zattoo4: zahs.tv"
91.123.100.227 comment "Zattoo3: zattoo.com"
45.60.121.229 comment "Zattoo2: digicert.com"
45.60.196.209 comment "Zattoo1: geotrust.com"
52.29.79.39 comment "meineip: wieistmeineip.de"
52.28.125.65 comment "meineip2: www.wieistmeineip.de"
it’s not working any more.
It’s an on and off game.
Sometimes the website www.wieistmeineip.de shows me the vpn ip address, sometimes the real one.
When it is not working:
When it is working:
which makes sense because it should include all the IPs the hostname resolves to.
Otherwise it will work only when the client uses the IP 52.28.125.65
I believe it has to do with #7 and you need to wait a bit.
I think you are right.
Every time i run the iptable mangle command these two numbers in front of the lines are counting up
[32632:24177344] -A FORWARD -m mark --mark 0x0/0xff0000 -j VPR_FORWARD
[19672:8585194] -A OUTPUT -m mark --mark 0x0/0xff0000 -j VPR_OUTPUT
If I go to www.wieistmeineip.de it sometimes shows up the vpn IP address, sometimes the real one, depending on which IP address it resolves, i think...
The numbers in the IP tables count on, till every rule is set, right?
Should these stop if every IP to bypass is set?
I even got zattoo working one time by http proxy the traffic and set the following ip range to bypass 91.123.0.0/16
I entered a second IP range to bypass Amazon Prime Video and restarted vpn-pbr.
Now I wait for it to work.
The iptables count and count....
I think it will take quite a long time with this wide range of IPs to bypass.
Let's try again with the dnsmasq.ipset . I tried it and it automatically installs all the addresses the hostname resolves, plus all subdomains.
config policy
option dest_addr 'wieistmeineip.de'
option interface 'lan'
option name 'wimi.de'
config vpn-policy-routing 'config'
option ipv6_enabled '0'
list supported_interface 'lan'
option boot_timeout '30'
option webui_sorting '1'
list webui_supported_protocol 'tcp'
list webui_supported_protocol 'udp'
list webui_supported_protocol 'tcp udp'
list webui_supported_protocol 'icmp'
list webui_supported_protocol 'all'
option webui_enable_column '1'
option webui_protocol_column '1'
option webui_chain_column '1'
list ignored_interface 'vpnserver wgserver'
option iprule_enabled '0'
option verbosity '1'
option strict_enforcement '1'
option enabled '1'
option iptables_rule_option 'append'
option dest_ipset 'dnsmasq.ipset'
option src_ipset '0'
config include
option path '/etc/vpn-policy-routing.netflix.user'
option enabled '0'
config include
option path '/etc/vpn-policy-routing.aws.user'
option enabled '0'
After you restart the service service vpn-policy-routing restart, check the ipset list lan, it should be empty. Then from a client do a nslookup www.wieinstmeineip.de 192.168.1.2 to make sure that the dnsmasq is used. Now the ipset list lan must contain all 4 addresses.
Okay. One moment...
Okay. I changed the config and restarted vpn-pbr.
ipset list lan had 0 entries.
I did the following on my mac
Dominik@Dominiks-Mac-mini ~ % nslookup www.wieistmeineip.de
Server: 192.168.1.2
Address: 192.168.1.2#53
Non-authoritative answer:
www.wieistmeineip.de canonical name = production.wieistmeineip.anw.net.
production.wieistmeineip.anw.net canonical name = public-lb-wieistmeineip-1392959777.eu-central-1.elb.amazonaws.com.
Name: public-lb-wieistmeineip-1392959777.eu-central-1.elb.amazonaws.com
Address: 52.29.58.77
Name: public-lb-wieistmeineip-1392959777.eu-central-1.elb.amazonaws.com
Address: 35.157.134.147
Name: public-lb-wieistmeineip-1392959777.eu-central-1.elb.amazonaws.com
Address: 52.28.125.65
Dominik@Dominiks-Mac-mini ~ % nslookup 192.168.1.2
Server: 192.168.1.2
Address: 192.168.1.2#53
** server can't find 2.1.168.192.in-addr.arpa: NXDOMAIN
Back on openWRT
ipset list lan still shows 0 entries...
root@OpenWrt:~# ipset list lan
Name: lan
Type: hash:net
Revision: 6
Header: family inet hashsize 1024 maxelem 65536 comment
Size in memory: 344
References: 4
Number of entries: 0
Members:
Maybe option src_ipset '0' should be '1'?
Or is this correct when using dnsmasq.ipset?
I think the nslookup command I first entered was wrong...
Here the right one. But still no entries...
Dominik@Dominiks-Mac-mini ~ % nslookup www.wieistmeineip.de 192.168.1.2
Server: 192.168.1.2
Address: 192.168.1.2#53
Non-authoritative answer:
www.wieistmeineip.de canonical name = production.wieistmeineip.anw.net.
production.wieistmeineip.anw.net canonical name = public-lb-wieistmeineip-1392959777.eu-central-1.elb.amazonaws.com.
Name: public-lb-wieistmeineip-1392959777.eu-central-1.elb.amazonaws.com
Address: 35.157.134.147
Name: public-lb-wieistmeineip-1392959777.eu-central-1.elb.amazonaws.com
Address: 52.29.58.77
Name: public-lb-wieistmeineip-1392959777.eu-central-1.elb.amazonaws.com
Address: 52.28.125.65
Dominik@Dominiks-Mac-mini ~ % ssh root@192.168.1.2
root@192.168.1.2's password:
BusyBox v1.30.1 () built-in shell (ash)
_______ ________ __
| |.-----.-----.-----.| | | |.----.| |_
| - || _ | -__| || | | || _|| _|
|_______|| __|_____|__|__||________||__| |____|
|__| W I R E L E S S F R E E D O M
-----------------------------------------------------
OpenWrt 19.07.5, r11257-5090152ae3
-----------------------------------------------------
root@OpenWrt:~# ipset list lan
Name: lan
Type: hash:net
Revision: 6
Header: family inet hashsize 1024 maxelem 65536 comment
Size in memory: 344
References: 4
Number of entries: 0
Members:
root@OpenWrt:~#
It was, but it turns out the mac was configured to ask the OpenWrt as nameserver, so the result was the same.
No, we are not using the ipset for source IPs, only for destinations.
I think the stubby is messing up something. Try to use only dnsmasq for now until we sort it out.
uci del dhcp.@dnsmasq[-1].server
uci set dhcp.@dnsmasq[-1].noresolv=0
uci set network.lan.dns='1.1.1.1'
uci commit
service dnsmasq restart
1 Like
I did all the steps above.
My internet connection gets lost.
I hardcoded the DNS 1.1.1.1 in my iPad to get access for now.
Here is what i did
root@OpenWrt:/etc/config# uci del dhcp.@dnsmasq[-1].server
root@OpenWrt:/etc/config# uci set dhcp.@dnsmasq[-1].noresolv=0
root@OpenWrt:/etc/config# uci set network.lan.dns='1.1.1.1'
root@OpenWrt:/etc/config# uci commit
root@OpenWrt:/etc/config# service dnsmasq restart
root@OpenWrt:/etc/config# ipset list lan
Name: lan
Type: hash:net
Revision: 6
Header: family inet hashsize 1024 maxelem 65536 comment
Size in memory: 344
References: 4
Number of entries: 0
Members:
root@OpenWrt:/etc/config# service vpn-policy-routing restart
Processing Interfaces ✓✓
Processing Policies ✓
vpn-policy-routing 0.2.1-13 started with gateways:
lan/br-lan/192.168.1.1
VyprVPN/tun0/10.2.23.90 [✓]
root@OpenWrt:/etc/config# service dnsmasq restart
root@OpenWrt:/etc/config# ipset list lan
Name: lan
Type: hash:net
Revision: 6
Header: family inet hashsize 1024 maxelem 65536 comment
Size in memory: 344
References: 4
Number of entries: 0
Members:
When I installed stubby, I deleted all other config resolver, I think thats why nothing works anymore now...
And I did this
uci set network.wan.peerdns='0'
uci set network.wan.dns='127.0.0.1'
What is the output of:
uci export network; uci export dhcp; ls -l /etc/resolv.* /tmp/resolv.* /tmp/resolv.*/* ; head -n -0 /etc/resolv.* /tmp/resolv.* /tmp/resolv.*/*
root@OpenWrt:~# uci export network; uci export dhcp; ls -l /etc/resolv.* /tmp/resolv.* /tmp/resolv.*/* ; head -n -0 /etc/resolv.* /tmp/resolv.* /tmp/re
solv.*/*
package network
config interface 'loopback'
option ifname 'lo'
option proto 'static'
option ipaddr '127.0.0.1'
option netmask '255.0.0.0'
config globals 'globals'
option ula_prefix 'fd7a:c322:7f86::/48'
config interface 'lan'
option type 'bridge'
option ifname 'eth0'
option proto 'static'
option ipaddr '192.168.1.2'
option netmask '255.255.255.0'
option ip6assign '60'
option gateway '192.168.1.1'
option dns '1.1.1.1'
config interface 'VyprVPN'
option ifname 'tun0'
option proto 'none'
package dhcp
config dnsmasq
option domainneeded '1'
option localise_queries '1'
option rebind_protection '1'
option rebind_localhost '1'
option local '/lan/'
option domain 'lan'
option expandhosts '1'
option readethers '1'
option leasefile '/tmp/dhcp.leases'
option localservice '1'
option confdir '/tmp/dnsmasq.d'
option dnssec '1'
list ipset '/zh2-9-hls7enc-live.zahs.tv/wieistmeineip.de/vpnbypass'
option noresolv '0'
config dhcp 'lan'
option interface 'lan'
option ra 'server'
option ignore '1'
config dhcp 'wan'
option interface 'wan'
option ignore '1'
config odhcpd 'odhcpd'
option maindhcp '0'
option leasefile '/tmp/hosts/odhcpd'
option leasetrigger '/usr/sbin/odhcpd-update'
option loglevel '4'
ls: /tmp/resolv.*/*: No such file or directory
lrwxrwxrwx 1 root root 16 Dec 6 08:31 /etc/resolv.conf -> /tmp/resolv.conf
-rw-r--r-- 1 root root 32 Jan 3 20:50 /tmp/resolv.conf
-rw-r--r-- 1 root root 39 Jan 2 23:52 /tmp/resolv.conf.auto
==> /etc/resolv.conf <==
search lan
nameserver 127.0.0.1
==> /tmp/resolv.conf <==
search lan
nameserver 127.0.0.1
==> /tmp/resolv.conf.auto <==
# Interface lan
nameserver 192.168.1.2
head: /tmp/resolv.*/*: No such file or directory
Remove:
And do service dnsmasq restart; ifup lan