VLAN + VPN Configuration

Installing and Using OpenWrt Network and Wireless Configuration
1

Hello, I'm pretty new to OpenWrt but I have some background on Linux / Network management.
I have some issues with my current configuration, and I would also like someone better than me to check if I did everything correctly (firewall, security etc - thanks in advance).

Here's my current network:

Untitled Diagram
Untitled Diagram717×462 38.6 KB

  • LAN (with VLAN ID 1) is my main network - I want traffic to go straight to WAN without any VPN
  • APP (with VLAN ID 3) is my guest network - I want traffic to go through VPN (if the VPN is connected - otherwise I will make it go through WAN directly if VPN is not working for some reason)

Of course I do not want devices on VLAN 3 to see devices on the other VLAN for any reason (and vice-versa). It's okay to see the WAN router as nothing is attached to it. So, here is the configuration I did:

2

VLANs:

image
image952×304 13.5 KB

3

Interfaces:

image
image958×507 40.7 KB

4

Firewall:

image
image960×754 34.3 KB

5

VPN Policy Routing:

image
image956×255 15.8 KB

6

Bandwitdth Limiter for APP Interface:

image
image973×557 23.2 KB

Things seems pretty much working as expected but I would like someone to double check the firewall rules to make sure everything is set up correctly (works != secure or well done).

7

Also, i'm having some issues on LAN:

  • If I try to ping the internet (8.8.8.8 or any website) I get 100% packet loss (of course I have internet connection, so it seems like it's dropping ping packets somewhere)
  • Sometimes some websites do not load correctly or take long time to load (I noticed this problem mostly with Amazon's website) - IPv6 is disabled everywhere and MSS Clamping is active in every firewall rule I have (this problem does not seem to occur on the "APP" VLAN, where everything loads correctly when on VPN). On the LAN interface I have these settings:
    image
    image469×599 21.7 KB

These are the main issues i'm experiencing, and I'm quite sure there is some wrong configuration on the Firewall. If you need any other info let me know!

Thanks a lot! :smile:

Ps. Sorry, I had to do multiple replies because have a 1 image limit per post.

8

You allow forwarding from the wan to the lan and app zones. If wan is an untrusted network then you usually don't allow that

You also use masquerading on all zones, something typically only used on wan zones (including zones with interfaces connected to a VPN provider). (Though it is sometimes needed as a work-around if routing isn't configured correctly in the local network.)

9

Thanks a lot for your answer! :slight_smile:

So, something like this should be good?

image
image957×283 20.7 KB

The ping / amazon issues are still present (but the ping works when on the APP VLAN connected to the VPN

10

Please copy the output of the following commands and post it here using the "Preformatted text </> " button:
grafik
Remember to redact passwords, MAC addresses and any public IP addresses you may have

uci export network; uci export wireless; \
uci export dhcp; uci export vpn-policy-routing; uci export firewall; \
head -n -0 /etc/firewall.user; \
iptables-save -c; \
ip -4 addr ; ip -4 ro li tab all ; ip -4 ru; \
ls -l  /etc/resolv.* /tmp/resolv.*; head -n -0 /etc/resolv.* /tmp/resolv.*

It is hard to get the whole picture from the images.

11

Hello, here's the output of the commands:

root@Main-Router:~# uci export network;
package network

config interface 'loopback'
        option ifname 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'fd9f:ee95:8521::/48'

config interface 'lan'
        option type 'bridge'
        option ifname 'eth0.1'
        option proto 'static'
        option netmask '255.255.255.0'
        option ipaddr '192.168.12.1'
        option delegate '0'

config interface 'wan'
        option ifname 'eth0.2'
        option proto 'dhcp'
        option delegate '0'

config switch
        option name 'switch0'
        option reset '1'
        option enable_vlan '1'

config switch_vlan#
root@Maioption device 'switch0'
root@Maioption vlan '1'
root@Maioption vid '1'ear
        option ports '0t 2 3 4'

config switch_vlan
        option device 'switch0'
        option vlan '2'
        option ports '0t 1'
        option vid '2'

config switch_vlan
        option device 'switch0'
        option vlan '3'
        option ports '0t 5'
        option vid '3'

config interface 'APP'
        option ifname 'eth0.3'
        option proto 'static'
        option netmask '255.255.255.0'
        option ipaddr '192.168.13.1'
        option type 'bridge'
        option delegate '0'

config interface 'VPN'
        option ifname 'tun0'
        option proto 'none'

root@Main-Router:~# uci export wireless;
package wireless

config wifi-device 'radio0'
        option type 'mac80211'
        option hwmode '11a'
        option path 'pci0000:00/0000:00:00.0'
        option htmode 'VHT80'
        option country 'IT'
        option channel '48'

config wifi-iface 'default_radio0'
        option device 'radio0'
        option network 'lan'
        option mode 'ap'
        option ft_over_ds '1'
        option ssid 'Home-WiFi'
        option encryption 'psk2'
        option ft_psk_generate_local '1'
        option key 'xxxxx'
        option ieee80211r '1'

config wifi-device 'radio1'
        option type 'mac80211'
        option hwmode '11g'
        option path 'platform/ahb/18100000.wmac'
        option htmode 'HT20'
        option country 'IT'
        option channel '6'

config wifi-iface 'default_radio1'
        option device 'radio1'
        option network 'lan'
        option mode 'ap'
        option ft_over_ds '1'
        option ssid 'Home-WiFi'
        option encryption 'psk2'
        option ft_psk_generate_local '1'
        option key 'xxxxx'
        option ieee80211r '1'

config wifi-iface 'wifinet2'
        option ssid 'App-WiFi'
        option encryption 'psk2'
        option device 'radio1'
        option mode 'ap'
        option network 'APP'
        option key 'xxxxx'

root@Main-Router:~# uci export dhcp;
package dhcp

config dnsmasq
        option domainneeded '1'
        option boguspriv '1'
        option filterwin2k '0'
        option localise_queries '1'
        option rebind_protection '1'
        option rebind_localhost '1'
        option local '/lan/'
        option domain 'lan'
        option expandhosts '1'
        option nonegcache '0'
        option authoritative '1'
        option readethers '1'
        option leasefile '/tmp/dhcp.leases'
        option resolvfile '/tmp/resolv.conf.auto'
        option nonwildcard '1'
        option localservice '1'

config dhcp 'lan'
        option interface 'lan'
        option limit '150'
        option leasetime '12h'
        option start '30'

config dhcp 'wan'
        option interface 'wan'
        option ignore '1'

config odhcpd 'odhcpd'
        option maindhcp '0'
        option leasefile '/tmp/hosts/odhcpd'
        option leasetrigger '/usr/sbin/odhcpd-update'
        option loglevel '4'

config dhcp 'APP'
        option leasetime '12h'
        option limit '150'
        option interface 'APP'
        option start '30'

root@Main-Router:~# uci export vpn-policy-routing;
package vpn-policy-routing

config vpn-policy-routing 'config'
        option verbosity '2'
        option src_ipset '0'
        option ipv6_enabled '0'
        list supported_interface ''
        list ignored_interface 'vpnserver wgserver'
        option iptables_rule_option 'append'
        option iprule_enabled '0'
        option webui_enable_column '0'
        option webui_protocol_column '0'
        option webui_chain_column '0'
        option webui_sorting '1'
        list webui_supported_protocol 'tcp'
        list webui_supported_protocol 'udp'
        list webui_supported_protocol 'tcp udp'
        list webui_supported_protocol 'icmp'
        list webui_supported_protocol 'all'
        option dest_ipset 'ipset'
        option enabled '1'
        option strict_enforcement '0'
        option boot_timeout '60'

config include
        option path '/etc/vpn-policy-routing.netflix.user'
        option enabled '0'

config include
        option path '/etc/vpn-policy-routing.aws.user'
        option enabled '0'

config policy
        option interface 'wan'
        option name 'Main'
        option src_port '0-65535'
        option dest_addr '0.0.0.0/0'
        option dest_port '0-65535'
        option src_addr '192.168.12.0/24'

config policy
        option name 'App'
        option interface 'VPN'
        option src_addr '192.168.13.0/24'
        option src_port '0-65535'
        option dest_addr '0.0.0.0/0'
        option dest_port '0-65535'

root@Main-Router:~# uci export firewall;
package firewall

config defaults
        option syn_flood '1'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'REJECT'

config zone
        option name 'lan'
        option output 'ACCEPT'
        option mtu_fix '1'
        option forward 'REJECT'
        option network 'lan'
        option family 'ipv4'
        option input 'ACCEPT'

config include
        option path '/etc/firewall.user'

config zone
        option network 'APP'
        option input 'ACCEPT'
        option forward 'REJECT'
        option name 'app'
        option output 'ACCEPT'
        option mtu_fix '1'

config zone
        option network 'VPN'
        option forward 'REJECT'
        option name 'vpn'
        option output 'ACCEPT'
        option mtu_fix '1'
        option input 'REJECT'
        option masq '1'

config forwarding
        option dest 'vpn'
        option src 'app'

config zone
        option network 'wan'
        option forward 'REJECT'
        option name 'wan'
        option output 'ACCEPT'
        option masq '1'
        option input 'REJECT'

config forwarding
        option dest 'wan'
        option src 'lan'

config forwarding
        option dest 'wan'
        option src 'app'

root@Main-Router:~# head -n -0 /etc/firewall.user;
# This file is interpreted as shell script.
# Put your custom iptables rules here, they will
# be executed with each firewall (re-)start.

# Internal uci firewall chains are flushed and recreated on reload, so
# put custom rules into the root chains e.g. INPUT or FORWARD or into the
# special user chains, e.g. input_wan_rule or postrouting_lan_rule.

root@Main-Router:~# iptables-save -c;
# Generated by iptables-save v1.8.3 on Tue Jun  9 08:16:31 2020
*nat
:PREROUTING ACCEPT [3897:667004]
:INPUT ACCEPT [951:76942]
:OUTPUT ACCEPT [524:35742]
:POSTROUTING ACCEPT [51:2624]
:postrouting_app_rule - [0:0]
:postrouting_lan_rule - [0:0]
:postrouting_rule - [0:0]
:postrouting_vpn_rule - [0:0]
:postrouting_wan_rule - [0:0]
:prerouting_app_rule - [0:0]
:prerouting_lan_rule - [0:0]
:prerouting_rule - [0:0]
:prerouting_vpn_rule - [0:0]
:prerouting_wan_rule - [0:0]
:zone_app_postrouting - [0:0]
:zone_app_prerouting - [0:0]
:zone_lan_postrouting - [0:0]
:zone_lan_prerouting - [0:0]
:zone_vpn_postrouting - [0:0]
:zone_vpn_prerouting - [0:0]
:zone_wan_postrouting - [0:0]
:zone_wan_prerouting - [0:0]
[3897:667004] -A PREROUTING -m comment --comment "!fw3: Custom prerouting rule chain" -j prerouting_rule
[3754:644951] -A PREROUTING -i br-lan -m comment --comment "!fw3" -j zone_lan_prerouting
[95:15689] -A PREROUTING -i br-APP -m comment --comment "!fw3" -j zone_app_prerouting
[0:0] -A PREROUTING -i tun0 -m comment --comment "!fw3" -j zone_vpn_prerouting
[48:6364] -A PREROUTING -i eth0.2 -m comment --comment "!fw3" -j zone_wan_prerouting
[1972:226339] -A POSTROUTING -m comment --comment "!fw3: Custom postrouting rule chain" -j postrouting_rule
[48:1920] -A POSTROUTING -o br-lan -m comment --comment "!fw3" -j zone_lan_postrouting
[3:704] -A POSTROUTING -o br-APP -m comment --comment "!fw3" -j zone_app_postrouting
[604:38982] -A POSTROUTING -o tun0 -m comment --comment "!fw3" -j zone_vpn_postrouting
[1317:184733] -A POSTROUTING -o eth0.2 -m comment --comment "!fw3" -j zone_wan_postrouting
[3:704] -A zone_app_postrouting -m comment --comment "!fw3: Custom app postrouting rule chain" -j postrouting_app_rule
[95:15689] -A zone_app_prerouting -m comment --comment "!fw3: Custom app prerouting rule chain" -j prerouting_app_rule
[48:1920] -A zone_lan_postrouting -m comment --comment "!fw3: Custom lan postrouting rule chain" -j postrouting_lan_rule
[3754:644951] -A zone_lan_prerouting -m comment --comment "!fw3: Custom lan prerouting rule chain" -j prerouting_lan_rule
[604:38982] -A zone_vpn_postrouting -m comment --comment "!fw3: Custom vpn postrouting rule chain" -j postrouting_vpn_rule
[604:38982] -A zone_vpn_postrouting -m comment --comment "!fw3" -j MASQUERADE
[0:0] -A zone_vpn_prerouting -m comment --comment "!fw3: Custom vpn prerouting rule chain" -j prerouting_vpn_rule
[1317:184733] -A zone_wan_postrouting -m comment --comment "!fw3: Custom wan postrouting rule chain" -j postrouting_wan_rule
[1317:184733] -A zone_wan_postrouting -m comment --comment "!fw3" -j MASQUERADE
[48:6364] -A zone_wan_prerouting -m comment --comment "!fw3: Custom wan prerouting rule chain" -j prerouting_wan_rule
COMMIT
# Completed on Tue Jun  9 08:16:31 2020
# Generated by iptables-save v1.8.3 on Tue Jun  9 08:16:31 2020
*raw
:PREROUTING ACCEPT [213323:176751546]
:OUTPUT ACCEPT [3373:638066]
:zone_app_helper - [0:0]
:zone_lan_helper - [0:0]
[86329:50413496] -A PREROUTING -i br-lan -m comment --comment "!fw3: lan CT helper assignment" -j zone_lan_helper
[557:89429] -A PREROUTING -i br-APP -m comment --comment "!fw3: app CT helper assignment" -j zone_app_helper
COMMIT
# Completed on Tue Jun  9 08:16:31 2020
# Generated by iptables-save v1.8.3 on Tue Jun  9 08:16:31 2020
*mangle
:PREROUTING ACCEPT [212777:176599416]
:INPUT ACCEPT [3062:868940]
:FORWARD ACCEPT [208307:175345335]
:OUTPUT ACCEPT [3238:623310]
:POSTROUTING ACCEPT [211256:175953113]
:QOS_MARK_br-APP - [0:0]
:VPR_FORWARD - [0:0]
:VPR_INPUT - [0:0]
:VPR_OUTPUT - [0:0]
:VPR_PREROUTING - [0:0]
[0:0] -A PREROUTING -i vtun+ -p tcp -j MARK --set-xmark 0x2/0xff
[1000:43664] -A PREROUTING -i br-APP -m dscp ! --dscp 0x00 -j DSCP --set-dscp 0x00
[212938:176638592] -A PREROUTING -m mark --mark 0x0/0xff0000 -j VPR_PREROUTING
[1834:777450] -A INPUT -m mark --mark 0x0/0xff0000 -j VPR_INPUT
[847:49640] -A FORWARD -o br-lan -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone lan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
[24:1440] -A FORWARD -o br-APP -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone app MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
[67:3700] -A FORWARD -o tun0 -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone vpn MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
[124791:125445393] -A FORWARD -m mark --mark 0x0/0xff0000 -j VPR_FORWARD
[109440:11413167] -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp 0x24
[3295:627042] -A OUTPUT -m mark --mark 0x0/0xff0000 -j VPR_OUTPUT
[118362:118414780] -A POSTROUTING -o br-APP -m mark --mark 0x0/0xff -g QOS_MARK_br-APP
[118362:118414780] -A QOS_MARK_br-APP -j MARK --set-xmark 0x2/0xff
[1:76] -A QOS_MARK_br-APP -m dscp --dscp 0x08 -j MARK --set-xmark 0x3/0xff
[1186:399597] -A QOS_MARK_br-APP -m dscp --dscp 0x30 -j MARK --set-xmark 0x1/0xff
[0:0] -A QOS_MARK_br-APP -m dscp --dscp 0x2e -j MARK --set-xmark 0x1/0xff
[556:130559] -A QOS_MARK_br-APP -m dscp --dscp 0x24 -j MARK --set-xmark 0x1/0xff
[556:130559] -A QOS_MARK_br-APP -m tos --tos 0x10/0x3f -j MARK --set-xmark 0x1/0xff
[0:0] -A VPR_FORWARD -m set --match-set VPN dst -j MARK --set-xmark 0x20000/0xff0000
[0:0] -A VPR_FORWARD -m set --match-set wan dst -j MARK --set-xmark 0x10000/0xff0000
[0:0] -A VPR_INPUT -m set --match-set VPN dst -j MARK --set-xmark 0x20000/0xff0000
[0:0] -A VPR_INPUT -m set --match-set wan dst -j MARK --set-xmark 0x10000/0xff0000
[0:0] -A VPR_OUTPUT -m set --match-set VPN dst -j MARK --set-xmark 0x20000/0xff0000
[0:0] -A VPR_OUTPUT -m set --match-set wan dst -j MARK --set-xmark 0x10000/0xff0000
[75:14791] -A VPR_PREROUTING -s 192.168.13.0/24 -p udp -m multiport --sports 0:65535 -m multiport --dports 0:65535 -m comment --comment App -j MARK --set-xmark 0x20000/0xff0000
[456:70956] -A VPR_PREROUTING -s 192.168.13.0/24 -p tcp -m multiport --sports 0:65535 -m multiport --dports 0:65535 -m comment --comment App -j MARK --set-xmark 0x20000/0xff0000
[10185:1696895] -A VPR_PREROUTING -s 192.168.12.0/24 -p udp -m multiport --sports 0:65535 -m multiport --dports 0:65535 -m comment --comment Main -j MARK --set-xmark 0x10000/0xff0000
[75505:48626604] -A VPR_PREROUTING -s 192.168.12.0/24 -p tcp -m multiport --sports 0:65535 -m multiport --dports 0:65535 -m comment --comment Main -j MARK --set-xmark 0x10000/0xff0000
[0:0] -A VPR_PREROUTING -m set --match-set VPN dst -j MARK --set-xmark 0x20000/0xff0000
[0:0] -A VPR_PREROUTING -m set --match-set wan dst -j MARK --set-xmark 0x10000/0xff0000
COMMIT
# Completed on Tue Jun  9 08:16:31 2020
# Generated by iptables-save v1.8.3 on Tue Jun  9 08:16:31 2020
*filter
:INPUT ACCEPT [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
:forwarding_app_rule - [0:0]
:forwarding_lan_rule - [0:0]
:forwarding_rule - [0:0]
:forwarding_vpn_rule - [0:0]
:forwarding_wan_rule - [0:0]
:input_app_rule - [0:0]
:input_lan_rule - [0:0]
:input_rule - [0:0]
:input_vpn_rule - [0:0]
:input_wan_rule - [0:0]
:output_app_rule - [0:0]
:output_lan_rule - [0:0]
:output_rule - [0:0]
:output_vpn_rule - [0:0]
:output_wan_rule - [0:0]
:reject - [0:0]
:syn_flood - [0:0]
:zone_app_dest_ACCEPT - [0:0]
:zone_app_dest_REJECT - [0:0]
:zone_app_forward - [0:0]
:zone_app_input - [0:0]
:zone_app_output - [0:0]
:zone_app_src_ACCEPT - [0:0]
:zone_lan_dest_ACCEPT - [0:0]
:zone_lan_dest_REJECT - [0:0]
:zone_lan_forward - [0:0]
:zone_lan_input - [0:0]
:zone_lan_output - [0:0]
:zone_lan_src_ACCEPT - [0:0]
:zone_vpn_dest_ACCEPT - [0:0]
:zone_vpn_dest_REJECT - [0:0]
:zone_vpn_forward - [0:0]
:zone_vpn_input - [0:0]
:zone_vpn_output - [0:0]
:zone_vpn_src_REJECT - [0:0]
:zone_wan_dest_ACCEPT - [0:0]
:zone_wan_dest_REJECT - [0:0]
:zone_wan_forward - [0:0]
:zone_wan_input - [0:0]
:zone_wan_output - [0:0]
:zone_wan_src_REJECT - [0:0]
[1:76] -A INPUT -i lo -m comment --comment "!fw3" -j ACCEPT
[3140:891186] -A INPUT -m comment --comment "!fw3: Custom input rule chain" -j input_rule
[1957:784426] -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -m comment --comment "!fw3" -j ACCEPT
[6:344] -A INPUT -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m comment --comment "!fw3" -j syn_flood
[1023:75764] -A INPUT -i br-lan -m comment --comment "!fw3" -j zone_lan_input
[87:17367] -A INPUT -i br-APP -m comment --comment "!fw3" -j zone_app_input
[0:0] -A INPUT -i tun0 -m comment --comment "!fw3" -j zone_vpn_input
[73:13629] -A INPUT -i eth0.2 -m comment --comment "!fw3" -j zone_wan_input
[208783:175473910] -A FORWARD -m comment --comment "!fw3: Custom forwarding rule chain" -j forwarding_rule
[205206:174960592] -A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -m comment --comment "!fw3" -j ACCEPT
[3541:510966] -A FORWARD -i br-lan -m comment --comment "!fw3" -j zone_lan_forward
[36:2352] -A FORWARD -i br-APP -m comment --comment "!fw3" -j zone_app_forward
[0:0] -A FORWARD -i tun0 -m comment --comment "!fw3" -j zone_vpn_forward
[0:0] -A FORWARD -i eth0.2 -m comment --comment "!fw3" -j zone_wan_forward
[136:10127] -A FORWARD -m comment --comment "!fw3" -j reject
[1:76] -A OUTPUT -o lo -m comment --comment "!fw3" -j ACCEPT
[3403:645454] -A OUTPUT -m comment --comment "!fw3: Custom output rule chain" -j output_rule
[2826:606038] -A OUTPUT -m conntrack --ctstate RELATED,ESTABLISHED -m comment --comment "!fw3" -j ACCEPT
[2:80] -A OUTPUT -o br-lan -m comment --comment "!fw3" -j zone_lan_output
[6:1710] -A OUTPUT -o br-APP -m comment --comment "!fw3" -j zone_app_output
[568:37298] -A OUTPUT -o tun0 -m comment --comment "!fw3" -j zone_vpn_output
[1:328] -A OUTPUT -o eth0.2 -m comment --comment "!fw3" -j zone_wan_output
[129:17092] -A reject -p tcp -m comment --comment "!fw3" -j REJECT --reject-with tcp-reset
[80:6664] -A reject -m comment --comment "!fw3" -j REJECT --reject-with icmp-port-unreachable
[6:344] -A syn_flood -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m limit --limit 25/sec --limit-burst 50 -m comment --comment "!fw3" -j RETURN
[0:0] -A syn_flood -m comment --comment "!fw3" -j DROP
[6:1710] -A zone_app_dest_ACCEPT -o br-APP -m comment --comment "!fw3" -j ACCEPT
[0:0] -A zone_app_dest_REJECT -o br-APP -m comment --comment "!fw3" -j reject
[36:2352] -A zone_app_forward -m comment --comment "!fw3: Custom app forwarding rule chain" -j forwarding_app_rule
[36:2352] -A zone_app_forward -m comment --comment "!fw3: Zone app to vpn forwarding policy" -j zone_vpn_dest_ACCEPT
[0:0] -A zone_app_forward -m comment --comment "!fw3: Zone app to wan forwarding policy" -j zone_wan_dest_ACCEPT
[0:0] -A zone_app_forward -m conntrack --ctstate DNAT -m comment --comment "!fw3: Accept port forwards" -j ACCEPT
[0:0] -A zone_app_forward -m comment --comment "!fw3" -j zone_app_dest_REJECT
[87:17367] -A zone_app_input -m comment --comment "!fw3: Custom app input rule chain" -j input_app_rule
[0:0] -A zone_app_input -m conntrack --ctstate DNAT -m comment --comment "!fw3: Accept port redirections" -j ACCEPT
[87:17367] -A zone_app_input -m comment --comment "!fw3" -j zone_app_src_ACCEPT
[6:1710] -A zone_app_output -m comment --comment "!fw3: Custom app output rule chain" -j output_app_rule
[6:1710] -A zone_app_output -m comment --comment "!fw3" -j zone_app_dest_ACCEPT
[87:17367] -A zone_app_src_ACCEPT -i br-APP -m conntrack --ctstate NEW,UNTRACKED -m comment --comment "!fw3" -j ACCEPT
[2:80] -A zone_lan_dest_ACCEPT -o br-lan -m comment --comment "!fw3" -j ACCEPT
[0:0] -A zone_lan_dest_REJECT -o br-lan -m comment --comment "!fw3" -j reject
[3541:510966] -A zone_lan_forward -m comment --comment "!fw3: Custom lan forwarding rule chain" -j forwarding_lan_rule
[3541:510966] -A zone_lan_forward -m comment --comment "!fw3: Zone lan to wan forwarding policy" -j zone_wan_dest_ACCEPT
[0:0] -A zone_lan_forward -m conntrack --ctstate DNAT -m comment --comment "!fw3: Accept port forwards" -j ACCEPT
[136:10127] -A zone_lan_forward -m comment --comment "!fw3" -j zone_lan_dest_REJECT
[1023:75764] -A zone_lan_input -m comment --comment "!fw3: Custom lan input rule chain" -j input_lan_rule
[0:0] -A zone_lan_input -m conntrack --ctstate DNAT -m comment --comment "!fw3: Accept port redirections" -j ACCEPT
[1023:75764] -A zone_lan_input -m comment --comment "!fw3" -j zone_lan_src_ACCEPT
[2:80] -A zone_lan_output -m comment --comment "!fw3: Custom lan output rule chain" -j output_lan_rule
[2:80] -A zone_lan_output -m comment --comment "!fw3" -j zone_lan_dest_ACCEPT
[1023:75764] -A zone_lan_src_ACCEPT -i br-lan -m conntrack --ctstate NEW,UNTRACKED -m comment --comment "!fw3" -j ACCEPT
[22:880] -A zone_vpn_dest_ACCEPT -o tun0 -m conntrack --ctstate INVALID -m comment --comment "!fw3: Prevent NAT leakage" -j DROP
[582:38770] -A zone_vpn_dest_ACCEPT -o tun0 -m comment --comment "!fw3" -j ACCEPT
[0:0] -A zone_vpn_dest_REJECT -o tun0 -m comment --comment "!fw3" -j reject
[0:0] -A zone_vpn_forward -m comment --comment "!fw3: Custom vpn forwarding rule chain" -j forwarding_vpn_rule
[0:0] -A zone_vpn_forward -m conntrack --ctstate DNAT -m comment --comment "!fw3: Accept port forwards" -j ACCEPT
[0:0] -A zone_vpn_forward -m comment --comment "!fw3" -j zone_vpn_dest_REJECT
[0:0] -A zone_vpn_input -m comment --comment "!fw3: Custom vpn input rule chain" -j input_vpn_rule
[0:0] -A zone_vpn_input -m conntrack --ctstate DNAT -m comment --comment "!fw3: Accept port redirections" -j ACCEPT
[0:0] -A zone_vpn_input -m comment --comment "!fw3" -j zone_vpn_src_REJECT
[568:37298] -A zone_vpn_output -m comment --comment "!fw3: Custom vpn output rule chain" -j output_vpn_rule
[568:37298] -A zone_vpn_output -m comment --comment "!fw3" -j zone_vpn_dest_ACCEPT
[0:0] -A zone_vpn_src_REJECT -i tun0 -m comment --comment "!fw3" -j reject
[203:9634] -A zone_wan_dest_ACCEPT -o eth0.2 -m conntrack --ctstate INVALID -m comment --comment "!fw3: Prevent NAT leakage" -j DROP
[3203:491533] -A zone_wan_dest_ACCEPT -o eth0.2 -m comment --comment "!fw3" -j ACCEPT
[0:0] -A zone_wan_dest_REJECT -o eth0.2 -m comment --comment "!fw3" -j reject
[0:0] -A zone_wan_forward -m comment --comment "!fw3: Custom wan forwarding rule chain" -j forwarding_wan_rule
[0:0] -A zone_wan_forward -m conntrack --ctstate DNAT -m comment --comment "!fw3: Accept port forwards" -j ACCEPT
[0:0] -A zone_wan_forward -m comment --comment "!fw3" -j zone_wan_dest_REJECT
[73:13629] -A zone_wan_input -m comment --comment "!fw3: Custom wan input rule chain" -j input_wan_rule
[0:0] -A zone_wan_input -m conntrack --ctstate DNAT -m comment --comment "!fw3: Accept port redirections" -j ACCEPT
[73:13629] -A zone_wan_input -m comment --comment "!fw3" -j zone_wan_src_REJECT
[1:328] -A zone_wan_output -m comment --comment "!fw3: Custom wan output rule chain" -j output_wan_rule
[1:328] -A zone_wan_output -m comment --comment "!fw3" -j zone_wan_dest_ACCEPT
[73:13629] -A zone_wan_src_REJECT -i eth0.2 -m comment --comment "!fw3" -j reject
COMMIT
# Completed on Tue Jun  9 08:16:31 2020

root@Main-Router:~# ip -4 addr ; ip -4 ro li tab all ; ip -4 ru;
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
13: br-APP: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc htb state UP group default qlen 1000
    inet 192.168.13.1/24 brd 192.168.13.255 scope global br-APP
       valid_lft forever preferred_lft forever
15: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 192.168.12.1/24 brd 192.168.12.255 scope global br-lan
       valid_lft forever preferred_lft forever
17: eth0.2@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 192.168.11.10/24 brd 192.168.11.255 scope global eth0.2
       valid_lft forever preferred_lft forever
79: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 100
    inet 10.8.1.14/24 brd 10.8.1.255 scope global tun0
       valid_lft forever preferred_lft forever
default via 192.168.11.1 dev eth0.2 table 201
192.168.13.0/24 dev br-APP table 201 proto kernel scope link src 192.168.13.1
default via 10.8.1.14 dev tun0 table 202
192.168.13.0/24 dev br-APP table 202 proto kernel scope link src 192.168.13.1
0.0.0.0/1 via 10.8.1.1 dev tun0
default via 192.168.11.1 dev eth0.2 proto static src 192.168.11.10
10.8.1.0/24 dev tun0 proto kernel scope link src 10.8.1.14
128.0.0.0/1 via 10.8.1.1 dev tun0
192.168.11.0/24 dev eth0.2 proto kernel scope link src 192.168.11.10
192.168.12.0/24 dev br-lan proto kernel scope link src 192.168.12.1
192.168.13.0/24 dev br-APP proto kernel scope link src 192.168.13.1
217.138.197.219 via 192.168.11.1 dev eth0.2
broadcast 10.8.1.0 dev tun0 table local proto kernel scope link src 10.8.1.14
local 10.8.1.14 dev tun0 table local proto kernel scope host src 10.8.1.14
broadcast 10.8.1.255 dev tun0 table local proto kernel scope link src 10.8.1.14
broadcast 127.0.0.0 dev lo table local proto kernel scope link src 127.0.0.1
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1
broadcast 192.168.11.0 dev eth0.2 table local proto kernel scope link src 192.168.11.10
local 192.168.11.10 dev eth0.2 table local proto kernel scope host src 192.168.11.10
broadcast 192.168.11.255 dev eth0.2 table local proto kernel scope link src 192.168.11.10
broadcast 192.168.12.0 dev br-lan table local proto kernel scope link src 192.168.12.1
local 192.168.12.1 dev br-lan table local proto kernel scope host src 192.168.12.1
broadcast 192.168.12.255 dev br-lan table local proto kernel scope link src 192.168.12.1
broadcast 192.168.13.0 dev br-APP table local proto kernel scope link src 192.168.13.1
local 192.168.13.1 dev br-APP table local proto kernel scope host src 192.168.13.1
broadcast 192.168.13.255 dev br-APP table local proto kernel scope link src 192.168.13.1
0:      from all lookup local
32560:  from all fwmark 0x20000/0xff0000 lookup 202
32561:  from all fwmark 0x10000/0xff0000 lookup 201
32766:  from all lookup main
32767:  from all lookup default

root@Main-Router:~# ls -l  /etc/resolv.* /tmp/resolv.*;
lrwxrwxrwx    1 root     root            16 May 16 20:32 /etc/resolv.conf -> /tmp/resolv.conf
-rw-r--r--    1 root     root            32 Jun  8 10:22 /tmp/resolv.conf
-rw-r--r--    1 root     root            54 Jun  8 22:54 /tmp/resolv.conf.auto

root@Main-Router:~# head -n -0 /etc/resolv.* /tmp/resolv.*
==> /etc/resolv.conf <==
search lan
nameserver 127.0.0.1

==> /tmp/resolv.conf <==
search lan
nameserver 127.0.0.1

==> /tmp/resolv.conf.auto <==
# Interface wan
nameserver 8.8.8.8
nameserver 8.8.4.4

Also my idea is to configure DMZ on the DSL Modem to the OpenWrt IP (and manage everything from OpenWrt Firewall).
Thanks a lot for your help!

12

Remove mtu-fix from the lan and app zones. Add it in the wan zone.
The ping is not working most likely because you are not marking it. As you can see below only tcp and udp are marked for PBR. So ICMP and the rest will be forwarded to VPN and will be dropped due to firewall limitations.

Modify the policy and use this as template


config policy
        option interface 'wan'
        option name 'Main'
        option src_addr '192.168.12.0/24'
        option proto 'all'
13

Hello, thanks for your answer!
I removed the tick from "MSS clamping" on LAN and APP zones, and I've enabled it on the WAN zone, but unfortunately some websites (like Amazon's website) have trouble loading (i'm sure this is not caused by the provider), while on VPN it works.

Also I tried to add "option proto 'all'" at the end of the "Main" policy in the /etc/config/vpn-policy-routing config file, but after doing that the result is that I have no internet access on the LAN interface.

thanks

14

Let's see the whole picture:

Please copy the output of the following commands and post it here using the "Preformatted text </> " button:
grafik
Remember to redact passwords, MAC addresses and any public IP addresses you may have

ubus call system board; uci export network; \
uci export dhcp; uci export firewall; uci export vpn-policy-routing \
head -n -0 /etc/firewall.user; \
iptables-save -c; \
ip -4 addr ; ip -4 ro li tab all ; ip -4 ru; \
ls -l  /etc/resolv.* /tmp/resolv.*; head -n -0 /etc/resolv.* /tmp/resolv.*
1 Like
15 
root@Main-Router:~# ubus call system board; uci export network;
{
        "kernel": "4.14.180",
        "hostname": "Main-Router",
        "system": "Qualcomm Atheros QCA956X ver 1 rev 0",
        "model": "TP-Link Archer C6 v2",
        "board_name": "tplink,archer-c6-v2",
        "release": {
                "distribution": "OpenWrt",
                "version": "19.07.3",
                "revision": "r11063-85e04e9f46",
                "target": "ath79/generic",
                "description": "OpenWrt 19.07.3 r11063-85e04e9f46"
        }
}
package network

config interface 'loopback'
        option ifname 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'fd9f:ee95:8521::/48'

config interface 'lan'
        option type 'bridge'
        option ifname 'eth0.1'
        option proto 'static'
        option netmask '255.255.255.0'
        option ipaddr '192.168.12.1'
        option delegate '0'

config interface 'wan'
        option ifname 'eth0.2'
        option proto 'dhcp'
        option delegate '0'

config switch
        option name 'switch0'
        option reset '1'
        option enable_vlan '1'

config switch_vlan
        option device 'switch0'
        option vlan '1'
        option vid '1'
        option ports '0t 2 3 4'

config switch_vlan
        option device 'switch0'
        option vlan '2'
        option ports '0t 1'
        option vid '2'

config switch_vlan
        option device 'switch0'
        option vlan '3'
        option ports '0t 5'
        option vid '3'

config interface 'APP'
        option ifname 'eth0.3'
        option proto 'static'
        option netmask '255.255.255.0'
        option ipaddr '192.168.13.1'
        option type 'bridge'
        option delegate '0'

config interface 'VPN'
        option ifname 'tun0'
        option proto 'none'

root@Main-Router:~# uci export dhcp; uci export firewall; uci export vpn-policy-routing
package dhcp

config dnsmasq
        option domainneeded '1'
        option boguspriv '1'
        option filterwin2k '0'
        option localise_queries '1'
        option rebind_protection '1'
        option rebind_localhost '1'
        option local '/lan/'
        option domain 'lan'
        option expandhosts '1'
        option nonegcache '0'
        option authoritative '1'
        option readethers '1'
        option leasefile '/tmp/dhcp.leases'
        option resolvfile '/tmp/resolv.conf.auto'
        option nonwildcard '1'
        option localservice '1'

config dhcp 'lan'
        option interface 'lan'
        option limit '150'
        option leasetime '12h'
        option start '30'

config dhcp 'wan'
        option interface 'wan'
        option ignore '1'

config odhcpd 'odhcpd'
        option maindhcp '0'
        option leasefile '/tmp/hosts/odhcpd'
        option leasetrigger '/usr/sbin/odhcpd-update'
        option loglevel '4'

config dhcp 'APP'
        option leasetime '12h'
        option limit '150'
        option interface 'APP'
        option start '30'

package firewall

config defaults
        option syn_flood '1'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'REJECT'

config zone
        option name 'lan'
        option output 'ACCEPT'
        option forward 'REJECT'
        option network 'lan'
        option family 'ipv4'
        option input 'ACCEPT'

config include
        option path '/etc/firewall.user'

config zone
        option network 'APP'
        option input 'ACCEPT'
        option forward 'REJECT'
        option name 'app'
        option output 'ACCEPT'

config zone
        option network 'VPN'
        option forward 'REJECT'
        option name 'vpn'
        option output 'ACCEPT'
        option mtu_fix '1'
        option input 'REJECT'
        option masq '1'

config forwarding
        option dest 'vpn'
        option src 'app'

config zone
        option network 'wan'
        option forward 'REJECT'
        option name 'wan'
        option output 'ACCEPT'
        option masq '1'
        option input 'REJECT'
        option mtu_fix '1'

config forwarding
        option dest 'wan'
        option src 'lan'

config forwarding
        option dest 'wan'
        option src 'app'

package vpn-policy-routing

config vpn-policy-routing 'config'
        option verbosity '2'
        option src_ipset '0'
        option ipv6_enabled '0'
        list supported_interface ''
        list ignored_interface 'vpnserver wgserver'
        option iptables_rule_option 'append'
        option iprule_enabled '0'
        option webui_enable_column '0'
        option webui_protocol_column '0'
        option webui_chain_column '0'
        option webui_sorting '1'
        list webui_supported_protocol 'tcp'
        list webui_supported_protocol 'udp'
        list webui_supported_protocol 'tcp udp'
        list webui_supported_protocol 'icmp'
        list webui_supported_protocol 'all'
        option dest_ipset 'ipset'
        option enabled '1'
        option strict_enforcement '0'
        option boot_timeout '60'

config include
        option path '/etc/vpn-policy-routing.netflix.user'
        option enabled '0'

config include
        option path '/etc/vpn-policy-routing.aws.user'
        option enabled '0'

config policy
        option interface 'wan'
        option name 'Main'
        option src_port '0-65535'
        option dest_addr '0.0.0.0/0'
        option dest_port '0-65535'
        option src_addr '192.168.12.0/24'

config policy
        option name 'App'
        option interface 'VPN'
        option src_addr '192.168.13.0/24'
        option src_port '0-65535'
        option dest_addr '0.0.0.0/0'
        option dest_port '0-65535'

(Note that I have removed the "option proto 'all'" from the 'Main' config policy as it gave me no internet access)

root@Main-Router:~# head -n -0 /etc/firewall.user;
# This file is interpreted as shell script.
# Put your custom iptables rules here, they will
# be executed with each firewall (re-)start.

# Internal uci firewall chains are flushed and recreated on reload, so
# put custom rules into the root chains e.g. INPUT or FORWARD or into the
# special user chains, e.g. input_wan_rule or postrouting_lan_rule.

root@Main-Router:~# iptables-save -c
# Generated by iptables-save v1.8.3 on Wed Jun 10 21:05:15 2020
*nat
:PREROUTING ACCEPT [851:143430]
:INPUT ACCEPT [77:7209]
:OUTPUT ACCEPT [71:6259]
:POSTROUTING ACCEPT [11:2032]
:postrouting_app_rule - [0:0]
:postrouting_lan_rule - [0:0]
:postrouting_rule - [0:0]
:postrouting_vpn_rule - [0:0]
:postrouting_wan_rule - [0:0]
:prerouting_app_rule - [0:0]
:prerouting_lan_rule - [0:0]
:prerouting_rule - [0:0]
:prerouting_vpn_rule - [0:0]
:prerouting_wan_rule - [0:0]
:zone_app_postrouting - [0:0]
:zone_app_prerouting - [0:0]
:zone_lan_postrouting - [0:0]
:zone_lan_prerouting - [0:0]
:zone_vpn_postrouting - [0:0]
:zone_vpn_prerouting - [0:0]
:zone_wan_postrouting - [0:0]
:zone_wan_prerouting - [0:0]
[851:143430] -A PREROUTING -m comment --comment "!fw3: Custom prerouting rule chain" -j prerouting_rule
[780:135928] -A PREROUTING -i br-lan -m comment --comment "!fw3" -j zone_lan_prerouting
[3:672] -A PREROUTING -i br-APP -m comment --comment "!fw3" -j zone_app_prerouting
[0:0] -A PREROUTING -i tun0 -m comment --comment "!fw3" -j zone_vpn_prerouting
[68:6830] -A PREROUTING -i eth0.2 -m comment --comment "!fw3" -j zone_wan_prerouting
[268:19275] -A POSTROUTING -m comment --comment "!fw3: Custom postrouting rule chain" -j postrouting_rule
[6:1423] -A POSTROUTING -o br-lan -m comment --comment "!fw3" -j zone_lan_postrouting
[1:337] -A POSTROUTING -o br-APP -m comment --comment "!fw3" -j zone_app_postrouting
[128:6947] -A POSTROUTING -o tun0 -m comment --comment "!fw3" -j zone_vpn_postrouting
[129:10296] -A POSTROUTING -o eth0.2 -m comment --comment "!fw3" -j zone_wan_postrouting
[1:337] -A zone_app_postrouting -m comment --comment "!fw3: Custom app postrouting rule chain" -j postrouting_app_rule
[3:672] -A zone_app_prerouting -m comment --comment "!fw3: Custom app prerouting rule chain" -j prerouting_app_rule
[6:1423] -A zone_lan_postrouting -m comment --comment "!fw3: Custom lan postrouting rule chain" -j postrouting_lan_rule
[780:135928] -A zone_lan_prerouting -m comment --comment "!fw3: Custom lan prerouting rule chain" -j prerouting_lan_rule
[128:6947] -A zone_vpn_postrouting -m comment --comment "!fw3: Custom vpn postrouting rule chain" -j postrouting_vpn_rule
[128:6947] -A zone_vpn_postrouting -m comment --comment "!fw3" -j MASQUERADE
[0:0] -A zone_vpn_prerouting -m comment --comment "!fw3: Custom vpn prerouting rule chain" -j prerouting_vpn_rule
[129:10296] -A zone_wan_postrouting -m comment --comment "!fw3: Custom wan postrouting rule chain" -j postrouting_wan_rule
[129:10296] -A zone_wan_postrouting -m comment --comment "!fw3" -j MASQUERADE
[68:6830] -A zone_wan_prerouting -m comment --comment "!fw3: Custom wan prerouting rule chain" -j prerouting_wan_rule
COMMIT
# Completed on Wed Jun 10 21:05:15 2020
# Generated by iptables-save v1.8.3 on Wed Jun 10 21:05:15 2020
*raw
:PREROUTING ACCEPT [9226:2641942]
:OUTPUT ACCEPT [2569:324785]
:zone_app_helper - [0:0]
:zone_lan_helper - [0:0]
[5510:666265] -A PREROUTING -i br-lan -m comment --comment "!fw3: lan CT helper assignment" -j zone_lan_helper
[5:1357] -A PREROUTING -i br-APP -m comment --comment "!fw3: app CT helper assignment" -j zone_app_helper
COMMIT
# Completed on Wed Jun 10 21:05:15 2020
# Generated by iptables-save v1.8.3 on Wed Jun 10 21:05:15 2020
*mangle
:PREROUTING ACCEPT [9227:2641982]
:INPUT ACCEPT [1799:151538]
:FORWARD ACCEPT [6856:2371769]
:OUTPUT ACCEPT [2571:325985]
:POSTROUTING ACCEPT [9224:2687719]
:QOS_MARK_br-APP - [0:0]
:VPR_FORWARD - [0:0]
:VPR_INPUT - [0:0]
:VPR_OUTPUT - [0:0]
:VPR_PREROUTING - [0:0]
[0:0] -A PREROUTING -i vtun+ -p tcp -j MARK --set-xmark 0x2/0xff
[2:397] -A PREROUTING -i br-APP -m dscp ! --dscp 0x00 -j DSCP --set-dscp 0x00
[10007:3069668] -A PREROUTING -m mark --mark 0x0/0xff0000 -j VPR_PREROUTING
[370:61444] -A INPUT -m mark --mark 0x0/0xff0000 -j VPR_INPUT
[3840:2311052] -A FORWARD -m mark --mark 0x0/0xff0000 -j VPR_FORWARD
[0:0] -A FORWARD -o tun0 -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone vpn MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
[89:5152] -A FORWARD -o eth0.2 -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
[133:19874] -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp 0x24
[2693:334352] -A OUTPUT -m mark --mark 0x0/0xff0000 -j VPR_OUTPUT
[3:1011] -A POSTROUTING -o br-APP -m mark --mark 0x0/0xff -g QOS_MARK_br-APP
[3:1011] -A QOS_MARK_br-APP -j MARK --set-xmark 0x2/0xff
[0:0] -A QOS_MARK_br-APP -m dscp --dscp 0x08 -j MARK --set-xmark 0x3/0xff
[3:1011] -A QOS_MARK_br-APP -m dscp --dscp 0x30 -j MARK --set-xmark 0x1/0xff
[0:0] -A QOS_MARK_br-APP -m dscp --dscp 0x2e -j MARK --set-xmark 0x1/0xff
[0:0] -A QOS_MARK_br-APP -m dscp --dscp 0x24 -j MARK --set-xmark 0x1/0xff
[0:0] -A QOS_MARK_br-APP -m tos --tos 0x10/0x3f -j MARK --set-xmark 0x1/0xff
[0:0] -A VPR_FORWARD -m set --match-set VPN dst -j MARK --set-xmark 0x20000/0xff0000
[0:0] -A VPR_FORWARD -m set --match-set wan dst -j MARK --set-xmark 0x10000/0xff0000
[0:0] -A VPR_INPUT -m set --match-set VPN dst -j MARK --set-xmark 0x20000/0xff0000
[0:0] -A VPR_INPUT -m set --match-set wan dst -j MARK --set-xmark 0x10000/0xff0000
[0:0] -A VPR_OUTPUT -m set --match-set VPN dst -j MARK --set-xmark 0x20000/0xff0000
[0:0] -A VPR_OUTPUT -m set --match-set wan dst -j MARK --set-xmark 0x10000/0xff0000
[2:640] -A VPR_PREROUTING -s 192.168.13.0/24 -p udp -m multiport --sports 0:65535 -m multiport --dports 0:65535 -m comment --comment App -j MARK --set-xmark 0x20000/0xff0000
[1035:172272] -A VPR_PREROUTING -s 192.168.12.0/24 -p udp -m multiport --sports 0:65535 -m multiport --dports 0:65535 -m comment --comment Main -j MARK --set-xmark 0x10000/0xff0000
[4672:514881] -A VPR_PREROUTING -s 192.168.12.0/24 -p tcp -m multiport --sports 0:65535 -m multiport --dports 0:65535 -m comment --comment Main -j MARK --set-xmark 0x10000/0xff0000
[0:0] -A VPR_PREROUTING -m set --match-set VPN dst -j MARK --set-xmark 0x20000/0xff0000
[0:0] -A VPR_PREROUTING -m set --match-set wan dst -j MARK --set-xmark 0x10000/0xff0000
COMMIT
# Completed on Wed Jun 10 21:05:15 2020
# Generated by iptables-save v1.8.3 on Wed Jun 10 21:05:15 2020
*filter
:INPUT ACCEPT [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
:forwarding_app_rule - [0:0]
:forwarding_lan_rule - [0:0]
:forwarding_rule - [0:0]
:forwarding_vpn_rule - [0:0]
:forwarding_wan_rule - [0:0]
:input_app_rule - [0:0]
:input_lan_rule - [0:0]
:input_rule - [0:0]
:input_vpn_rule - [0:0]
:input_wan_rule - [0:0]
:output_app_rule - [0:0]
:output_lan_rule - [0:0]
:output_rule - [0:0]
:output_vpn_rule - [0:0]
:output_wan_rule - [0:0]
:reject - [0:0]
:syn_flood - [0:0]
:zone_app_dest_ACCEPT - [0:0]
:zone_app_dest_REJECT - [0:0]
:zone_app_forward - [0:0]
:zone_app_input - [0:0]
:zone_app_output - [0:0]
:zone_app_src_ACCEPT - [0:0]
:zone_lan_dest_ACCEPT - [0:0]
:zone_lan_dest_REJECT - [0:0]
:zone_lan_forward - [0:0]
:zone_lan_input - [0:0]
:zone_lan_output - [0:0]
:zone_lan_src_ACCEPT - [0:0]
:zone_vpn_dest_ACCEPT - [0:0]
:zone_vpn_dest_REJECT - [0:0]
:zone_vpn_forward - [0:0]
:zone_vpn_input - [0:0]
:zone_vpn_output - [0:0]
:zone_vpn_src_REJECT - [0:0]
:zone_wan_dest_ACCEPT - [0:0]
:zone_wan_dest_REJECT - [0:0]
:zone_wan_forward - [0:0]
:zone_wan_input - [0:0]
:zone_wan_output - [0:0]
:zone_wan_src_REJECT - [0:0]
[17:3663] -A INPUT -i lo -m comment --comment "!fw3" -j ACCEPT
[1800:148595] -A INPUT -m comment --comment "!fw3: Custom input rule chain" -j input_rule
[1378:112406] -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -m comment --comment "!fw3" -j ACCEPT
[8:416] -A INPUT -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m comment --comment "!fw3" -j syn_flood
[239:23105] -A INPUT -i br-lan -m comment --comment "!fw3" -j zone_lan_input
[4:992] -A INPUT -i br-APP -m comment --comment "!fw3" -j zone_app_input
[0:0] -A INPUT -i tun0 -m comment --comment "!fw3" -j zone_vpn_input
[179:12092] -A INPUT -i eth0.2 -m comment --comment "!fw3" -j zone_wan_input
[6856:2371769] -A FORWARD -m comment --comment "!fw3: Custom forwarding rule chain" -j forwarding_rule
[6494:2328947] -A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -m comment --comment "!fw3" -j ACCEPT
[362:42822] -A FORWARD -i br-lan -m comment --comment "!fw3" -j zone_lan_forward
[0:0] -A FORWARD -i br-APP -m comment --comment "!fw3" -j zone_app_forward
[0:0] -A FORWARD -i tun0 -m comment --comment "!fw3" -j zone_vpn_forward
[0:0] -A FORWARD -i eth0.2 -m comment --comment "!fw3" -j zone_wan_forward
[5:420] -A FORWARD -m comment --comment "!fw3" -j reject
[17:3663] -A OUTPUT -o lo -m comment --comment "!fw3" -j ACCEPT
[2582:328226] -A OUTPUT -m comment --comment "!fw3: Custom output rule chain" -j output_rule
[2401:316490] -A OUTPUT -m conntrack --ctstate RELATED,ESTABLISHED -m comment --comment "!fw3" -j ACCEPT
[10:2772] -A OUTPUT -o br-lan -m comment --comment "!fw3" -j zone_lan_output
[1:337] -A OUTPUT -o br-APP -m comment --comment "!fw3" -j zone_app_output
[170:8627] -A OUTPUT -o tun0 -m comment --comment "!fw3" -j zone_vpn_output
[0:0] -A OUTPUT -o eth0.2 -m comment --comment "!fw3" -j zone_wan_output
[179:12092] -A reject -p tcp -m comment --comment "!fw3" -j REJECT --reject-with tcp-reset
[5:420] -A reject -m comment --comment "!fw3" -j REJECT --reject-with icmp-port-unreachable
[8:416] -A syn_flood -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m limit --limit 25/sec --limit-burst 50 -m comment --comment "!fw3" -j RETURN
[0:0] -A syn_flood -m comment --comment "!fw3" -j DROP
[1:337] -A zone_app_dest_ACCEPT -o br-APP -m comment --comment "!fw3" -j ACCEPT
[0:0] -A zone_app_dest_REJECT -o br-APP -m comment --comment "!fw3" -j reject
[0:0] -A zone_app_forward -m comment --comment "!fw3: Custom app forwarding rule chain" -j forwarding_app_rule
[0:0] -A zone_app_forward -m comment --comment "!fw3: Zone app to vpn forwarding policy" -j zone_vpn_dest_ACCEPT
[0:0] -A zone_app_forward -m comment --comment "!fw3: Zone app to wan forwarding policy" -j zone_wan_dest_ACCEPT
[0:0] -A zone_app_forward -m conntrack --ctstate DNAT -m comment --comment "!fw3: Accept port forwards" -j ACCEPT
[0:0] -A zone_app_forward -m comment --comment "!fw3" -j zone_app_dest_REJECT
[4:992] -A zone_app_input -m comment --comment "!fw3: Custom app input rule chain" -j input_app_rule
[0:0] -A zone_app_input -m conntrack --ctstate DNAT -m comment --comment "!fw3: Accept port redirections" -j ACCEPT
[4:992] -A zone_app_input -m comment --comment "!fw3" -j zone_app_src_ACCEPT
[1:337] -A zone_app_output -m comment --comment "!fw3: Custom app output rule chain" -j output_app_rule
[1:337] -A zone_app_output -m comment --comment "!fw3" -j zone_app_dest_ACCEPT
[4:992] -A zone_app_src_ACCEPT -i br-APP -m conntrack --ctstate NEW,UNTRACKED -m comment --comment "!fw3" -j ACCEPT
[10:2772] -A zone_lan_dest_ACCEPT -o br-lan -m comment --comment "!fw3" -j ACCEPT
[0:0] -A zone_lan_dest_REJECT -o br-lan -m comment --comment "!fw3" -j reject
[362:42822] -A zone_lan_forward -m comment --comment "!fw3: Custom lan forwarding rule chain" -j forwarding_lan_rule
[362:42822] -A zone_lan_forward -m comment --comment "!fw3: Zone lan to wan forwarding policy" -j zone_wan_dest_ACCEPT
[0:0] -A zone_lan_forward -m conntrack --ctstate DNAT -m comment --comment "!fw3: Accept port forwards" -j ACCEPT
[5:420] -A zone_lan_forward -m comment --comment "!fw3" -j zone_lan_dest_REJECT
[239:23105] -A zone_lan_input -m comment --comment "!fw3: Custom lan input rule chain" -j input_lan_rule
[0:0] -A zone_lan_input -m conntrack --ctstate DNAT -m comment --comment "!fw3: Accept port redirections" -j ACCEPT
[239:23105] -A zone_lan_input -m comment --comment "!fw3" -j zone_lan_src_ACCEPT
[10:2772] -A zone_lan_output -m comment --comment "!fw3: Custom lan output rule chain" -j output_lan_rule
[10:2772] -A zone_lan_output -m comment --comment "!fw3" -j zone_lan_dest_ACCEPT
[239:23105] -A zone_lan_src_ACCEPT -i br-lan -m conntrack --ctstate NEW,UNTRACKED -m comment --comment "!fw3" -j ACCEPT
[110:4400] -A zone_vpn_dest_ACCEPT -o tun0 -m conntrack --ctstate INVALID -m comment --comment "!fw3: Prevent NAT leakage" -j DROP
[60:4227] -A zone_vpn_dest_ACCEPT -o tun0 -m comment --comment "!fw3" -j ACCEPT
[0:0] -A zone_vpn_dest_REJECT -o tun0 -m comment --comment "!fw3" -j reject
[0:0] -A zone_vpn_forward -m comment --comment "!fw3: Custom vpn forwarding rule chain" -j forwarding_vpn_rule
[0:0] -A zone_vpn_forward -m conntrack --ctstate DNAT -m comment --comment "!fw3: Accept port forwards" -j ACCEPT
[0:0] -A zone_vpn_forward -m comment --comment "!fw3" -j zone_vpn_dest_REJECT
[0:0] -A zone_vpn_input -m comment --comment "!fw3: Custom vpn input rule chain" -j input_vpn_rule
[0:0] -A zone_vpn_input -m conntrack --ctstate DNAT -m comment --comment "!fw3: Accept port redirections" -j ACCEPT
[0:0] -A zone_vpn_input -m comment --comment "!fw3" -j zone_vpn_src_REJECT
[170:8627] -A zone_vpn_output -m comment --comment "!fw3: Custom vpn output rule chain" -j output_vpn_rule
[170:8627] -A zone_vpn_output -m comment --comment "!fw3" -j zone_vpn_dest_ACCEPT
[0:0] -A zone_vpn_src_REJECT -i tun0 -m comment --comment "!fw3" -j reject
[88:5215] -A zone_wan_dest_ACCEPT -o eth0.2 -m conntrack --ctstate INVALID -m comment --comment "!fw3: Prevent NAT leakage" -j DROP
[269:37187] -A zone_wan_dest_ACCEPT -o eth0.2 -m comment --comment "!fw3" -j ACCEPT
[0:0] -A zone_wan_dest_REJECT -o eth0.2 -m comment --comment "!fw3" -j reject
[0:0] -A zone_wan_forward -m comment --comment "!fw3: Custom wan forwarding rule chain" -j forwarding_wan_rule
[0:0] -A zone_wan_forward -m conntrack --ctstate DNAT -m comment --comment "!fw3: Accept port forwards" -j ACCEPT
[0:0] -A zone_wan_forward -m comment --comment "!fw3" -j zone_wan_dest_REJECT
[179:12092] -A zone_wan_input -m comment --comment "!fw3: Custom wan input rule chain" -j input_wan_rule
[0:0] -A zone_wan_input -m conntrack --ctstate DNAT -m comment --comment "!fw3: Accept port redirections" -j ACCEPT
[179:12092] -A zone_wan_input -m comment --comment "!fw3" -j zone_wan_src_REJECT
[0:0] -A zone_wan_output -m comment --comment "!fw3: Custom wan output rule chain" -j output_wan_rule
[0:0] -A zone_wan_output -m comment --comment "!fw3" -j zone_wan_dest_ACCEPT
[179:12092] -A zone_wan_src_REJECT -i eth0.2 -m comment --comment "!fw3" -j reject
COMMIT
# Completed on Wed Jun 10 21:05:16 2020

root@Main-Router:~# ip -4 addr ; ip -4 ro li tab all ; ip -4 ru;
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
13: br-APP: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc htb state UP group default qlen 1000
    inet 192.168.13.1/24 brd 192.168.13.255 scope global br-APP
       valid_lft forever preferred_lft forever
15: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 192.168.12.1/24 brd 192.168.12.255 scope global br-lan
       valid_lft forever preferred_lft forever
17: eth0.2@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 192.168.11.10/24 brd 192.168.11.255 scope global eth0.2
       valid_lft forever preferred_lft forever
22: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 100
    inet 10.8.2.25/24 brd 10.8.2.255 scope global tun0
       valid_lft forever preferred_lft forever
default via 192.168.11.1 dev eth0.2 table 201
192.168.13.0/24 dev br-APP table 201 proto kernel scope link src 192.168.13.1
default via 10.8.2.25 dev tun0 table 202
192.168.13.0/24 dev br-APP table 202 proto kernel scope link src 192.168.13.1
0.0.0.0/1 via 10.8.2.1 dev tun0
default via 192.168.11.1 dev eth0.2 proto static src 192.168.11.10
10.8.2.0/24 dev tun0 proto kernel scope link src 10.8.2.25
128.0.0.0/1 via 10.8.2.1 dev tun0
192.168.11.0/24 dev eth0.2 proto kernel scope link src 192.168.11.10
192.168.12.0/24 dev br-lan proto kernel scope link src 192.168.12.1
192.168.13.0/24 dev br-APP proto kernel scope link src 192.168.13.1
217.138.197.219 via 192.168.11.1 dev eth0.2
broadcast 10.8.2.0 dev tun0 table local proto kernel scope link src 10.8.2.25
local 10.8.2.25 dev tun0 table local proto kernel scope host src 10.8.2.25
broadcast 10.8.2.255 dev tun0 table local proto kernel scope link src 10.8.2.25
broadcast 127.0.0.0 dev lo table local proto kernel scope link src 127.0.0.1
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1
broadcast 192.168.11.0 dev eth0.2 table local proto kernel scope link src 192.168.11.10
local 192.168.11.10 dev eth0.2 table local proto kernel scope host src 192.168.11.10
broadcast 192.168.11.255 dev eth0.2 table local proto kernel scope link src 192.168.11.10
broadcast 192.168.12.0 dev br-lan table local proto kernel scope link src 192.168.12.1
local 192.168.12.1 dev br-lan table local proto kernel scope host src 192.168.12.1
broadcast 192.168.12.255 dev br-lan table local proto kernel scope link src 192.168.12.1
broadcast 192.168.13.0 dev br-APP table local proto kernel scope link src 192.168.13.1
local 192.168.13.1 dev br-APP table local proto kernel scope host src 192.168.13.1
broadcast 192.168.13.255 dev br-APP table local proto kernel scope link src 192.168.13.1
0:      from all lookup local
32764:  from all fwmark 0x20000/0xff0000 lookup 202
32765:  from all fwmark 0x10000/0xff0000 lookup 201
32766:  from all lookup main
32767:  from all lookup default

root@Main-Router:~# ls -l  /etc/resolv.* /tmp/resolv.*; head -n -0 /etc/resolv.* /tmp/resolv.*
lrwxrwxrwx    1 root     root            16 May 16 20:32 /etc/resolv.conf -> /tmp/resolv.conf
-rw-r--r--    1 root     root            32 Jun 10 21:01 /tmp/resolv.conf
-rw-r--r--    1 root     root            54 Jun 10 21:01 /tmp/resolv.conf.auto
==> /etc/resolv.conf <==
search lan
nameserver 127.0.0.1

==> /tmp/resolv.conf <==
search lan
nameserver 127.0.0.1

==> /tmp/resolv.conf.auto <==
# Interface wan
nameserver 8.8.8.8
nameserver 8.8.4.4

Thanks again!

16

Currently it is only classifying the APP interface for both tcp+udp, but only udp for lan.
I suspect that there is some conflict with the QoS that is running. Can you try to disable that?
Verify with iptables-save -c | grep "192.168.12.0"

17

By disabling SQM QoS, the output of the previous command is:

root@Main-Router:~# iptables-save -c | grep "192.168.12.0"
[506:91367] -A VPR_PREROUTING -s 192.168.12.0/24 -p udp -m multiport --sports 0:65535 -m multiport --dports 0:65535 -m comment --comment Main -j MARK --set-xmark 0x10000/0xff0000
[774:161125] -A VPR_PREROUTING -s 192.168.12.0/24 -p tcp -m multiport --sports 0:65535 -m multiport --dports 0:65535 -m comment --comment Main -j MARK --set-xmark 0x10000/0xff0000

The problem with websites is still there tho (Amazon.it takes forever to load and amazon.com gives me "ERR_CONNECTION_RESET"). Thanks

18

Alright, first verify that it works fine if you bring down the vpn and everything goes through wan.

Second post here the output of:

/etc/init.d/vpn-policy-routing reload
/etc/init.d/vpn-policy-routing support
19

Yes! I can confirm that stopping the "tun0" interface makes everything working again (including amazon's website and ping from both interfaces lan and app).

After that I have enabled again the VPN interface and noticed everything was still working. I've also noticed that the "default interface" on vpn-policy-routing changed to the WAN (usually the tick is near VPN):
image

Maybe it could be a good idea to set the wan interface as default for policy routing?

Here's the output for the commands:

root@Main-Router:~# /etc/init.d/vpn-policy-routing reload
Creating table 'wan/eth0.2/192.168.11.1' [✓]
Creating table 'VPN/tun0/10.8.0.5' [✓]
Routing 'Main' via wan [✓]
Routing 'App' via VPN [✓]
vpn-policy-routing 0.2.1-13 started with gateways:
wan/eth0.2/192.168.11.1 [✓]
VPN/tun0/10.8.0.5
vpn-policy-routing 0.2.1-13 monitoring interfaces: wan VPN .
root@Main-Router:~# /etc/init.d/vpn-policy-routing support
vpn-policy-routing 0.2.1-13 running on OpenWrt 19.07.3. WAN (IPv4): wan/dev/192.168.11.1.
============================================================
Dnsmasq version 2.80  Copyright (c) 2000-2018 Simon Kelley
Compile time options: IPv6 GNU-getopt no-DBus no-i18n no-IDN DHCP no-DHCPv6 no-Lua TFTP no-conntrack no-ipset no-auth no-DNSSEC no-ID loop-detect inotify dumpfile
============================================================
Routes/IP Rules
default         192.168.11.1    0.0.0.0         UG    0      0        0 eth0.2
IPv4 Table 201: default via 192.168.11.1 dev eth0.2
192.168.13.0/24 dev br-APP proto kernel scope link src 192.168.13.1
IPv4 Table 201 Rules:
32749:  from all fwmark 0x10000/0xff0000 lookup 201
IPv4 Table 202: default via 10.8.0.5 dev tun0
192.168.13.0/24 dev br-APP proto kernel scope link src 192.168.13.1
IPv4 Table 202 Rules:
32748:  from all fwmark 0x20000/0xff0000 lookup 202
============================================================
IP Tables PREROUTING
-N VPR_PREROUTING
-A VPR_PREROUTING -s 192.168.13.0/24 -p udp -m multiport --sports 0:65535 -m multiport --dports 0:65535 -m comment --comment App -c 1 320 -j MARK --set-xmark 0x20000/0xff0000
-A VPR_PREROUTING -s 192.168.13.0/24 -p tcp -m multiport --sports 0:65535 -m multiport --dports 0:65535 -m comment --comment App -c 0 0 -j MARK --set-xmark 0x20000/0xff0000
-A VPR_PREROUTING -s 192.168.12.0/24 -p udp -m multiport --sports 0:65535 -m multiport --dports 0:65535 -m comment --comment Main -c 35 4927 -j MARK --set-xmark 0x10000/0xff0000
-A VPR_PREROUTING -s 192.168.12.0/24 -p tcp -m multiport --sports 0:65535 -m multiport --dports 0:65535 -m comment --comment Main -c 102 13873 -j MARK --set-xmark 0x10000/0xff0000
-A VPR_PREROUTING -m set --match-set VPN dst -c 0 0 -j MARK --set-xmark 0x20000/0xff0000
-A VPR_PREROUTING -m set --match-set wan dst -c 0 0 -j MARK --set-xmark 0x10000/0xff0000
============================================================
IP Tables FORWARD
-N VPR_FORWARD
-A VPR_FORWARD -m set --match-set VPN dst -c 0 0 -j MARK --set-xmark 0x20000/0xff0000
-A VPR_FORWARD -m set --match-set wan dst -c 0 0 -j MARK --set-xmark 0x10000/0xff0000
============================================================
IP Tables INPUT
-N VPR_INPUT
-A VPR_INPUT -m set --match-set VPN dst -c 0 0 -j MARK --set-xmark 0x20000/0xff0000
-A VPR_INPUT -m set --match-set wan dst -c 0 0 -j MARK --set-xmark 0x10000/0xff0000
============================================================
IP Tables OUTPUT
-N VPR_OUTPUT
-A VPR_OUTPUT -m set --match-set VPN dst -c 0 0 -j MARK --set-xmark 0x20000/0xff0000
-A VPR_OUTPUT -m set --match-set wan dst -c 0 0 -j MARK --set-xmark 0x10000/0xff0000
============================================================
Current ipsets
create wan hash:net family inet hashsize 1024 maxelem 65536 comment
create VPN hash:net family inet hashsize 1024 maxelem 65536 comment
============================================================
Your support details have been logged to '/var/vpn-policy-routing-support'. [✓]

Thanks again!

20

Yes, that could be a workaround.

Here are instructions how to ignore the default route pushed by the vpn. Everything will be routed via wan by default and only the preselected in VPN-PBR will go via VPN.