Mwan3 rules with ipset

gfucka · January 13, 2020, 3:41pm

Hi,
I install and configure mwan3 and all works good.
Now, I'm going to configure a specific rule. What I want is that all traffic for a specific site address (destionation-address.com) use just wan1.
I read the tutorial and create a rule in /etc/dnsmasq.conf:

ipset=/destionation-address.com/myrule

and then I create a rule:

config rule 'myrule'
    option sticky ‘'
    option timeout ‘'
    option ipset 'myrule'
    option dest_port ''
    option proto ''
    option use_policy 'wan1_only'

but it not work.

Please can you help me?

Thankyou

ulmwind · January 13, 2020, 5:06pm

You should use ipset, Stickiness and ipset: https://openwrt.org/docs/guide-user/network/wan/multiwan/mwan3#rule_configuration

Please, give output of:

ip rule show
ip route show table all
mwan3 status

gfucka · January 13, 2020, 6:52pm

Hi thank you for answer.
This are the outputs:
ip rule show

0:      from all lookup 128 
1:      from all lookup local 
1001:   from all iif eth0.2 lookup main 
1002:   from all iif wlan0 lookup main 
2001:   from all fwmark 0x100/0xff00 lookup 1 
2002:   from all fwmark 0x200/0xff00 lookup 2 
2253:   from all fwmark 0xfd00/0xff00 blackhole
2254:   from all fwmark 0xfe00/0xff00 unreachable
32766:  from all lookup main 
32767:  from all lookup default

ip route show table all

default via 192.168.1.1 dev eth0.2  table 1 
default via 192.168.0.1 dev wlan0  table 2 
default via 192.168.0.1 dev wlan0  proto static  src 192.168.0.100 
default via 192.168.1.1 dev eth0.2  proto static  src 192.168.1.13  metric 10 
192.168.0.0/24 dev wlan0  proto kernel  scope link  src 192.168.0.100 
192.168.0.1 dev wlan0  proto static  scope link  src 192.168.0.100 
192.168.1.0/24 dev eth0.2  proto static  scope link  metric 10 
192.168.1.1 dev eth0.2  proto static  scope link  src 192.168.1.13  metric 10 
192.168.2.0/24 dev br-lan  proto kernel  scope link  src 192.168.2.1 
broadcast 127.0.0.0 dev lo  table local  proto kernel  scope link  src 127.0.0.1 
local 127.0.0.0/8 dev lo  table local  proto kernel  scope host  src 127.0.0.1 
local 127.0.0.1 dev lo  table local  proto kernel  scope host  src 127.0.0.1 
broadcast 127.255.255.255 dev lo  table local  proto kernel  scope link  src 127.0.0.1 
broadcast 192.168.0.0 dev wlan0  table local  proto kernel  scope link  src 192.168.0.100 
local 192.168.0.100 dev wlan0  table local  proto kernel  scope host  src 192.168.0.100 
broadcast 192.168.0.255 dev wlan0  table local  proto kernel  scope link  src 192.168.0.100 
broadcast 192.168.1.0 dev eth0.2  table local  proto kernel  scope link  src 192.168.1.13 
local 192.168.1.13 dev eth0.2  table local  proto kernel  scope host  src 192.168.1.13 
broadcast 192.168.1.255 dev eth0.2  table local  proto kernel  scope link  src 192.168.1.13 
broadcast 192.168.2.0 dev br-lan  table local  proto kernel  scope link  src 192.168.2.1 
local 192.168.2.1 dev br-lan  table local  proto kernel  scope host  src 192.168.2.1 
broadcast 192.168.2.255 dev br-lan  table local  proto kernel  scope link  src 192.168.2.1 
unreachable default dev lo  table unspec  proto kernel  metric 4294967295  error -128
fd06:92e0:4336::/64 dev br-lan  proto static  metric 1024 
unreachable fd06:92e0:4336::/48 dev lo  proto static  metric 2147483647  error -128
fe80::/64 dev eth0  proto kernel  metric 256 
fe80::/64 dev br-lan  proto kernel  metric 256 
fe80::/64 dev eth0.2  proto kernel  metric 256 
fe80::/64 dev wlan0  proto kernel  metric 256 
unreachable default dev lo  table unspec  proto kernel  metric 4294967295  error -128
local ::1 dev lo  table local  proto none  metric 0 
local fd06:92e0:4336:: dev lo  table local  proto none  metric 0 
local fd06:92e0:4336::1 dev lo  table local  proto none  metric 0 
local fe80:: dev lo  table local  proto none  metric 0 
local fe80:: dev lo  table local  proto none  metric 0 
local fe80:: dev lo  table local  proto none  metric 0 
local fe80:: dev lo  table local  proto none  metric 0 
local fe80::b248:7aff:fedb:9301 dev lo  table local  proto none  metric 0 
local fe80::b248:7aff:fedb:9301 dev lo  table local  proto none  metric 0 
local fe80::b248:7aff:fedb:9301 dev lo  table local  proto none  metric 0 
local fe80::b248:7aff:fedb:9302 dev lo  table local  proto none  metric 0 
ff00::/8 dev br-lan  table local  metric 256 
ff00::/8 dev eth0  table local  metric 256 
ff00::/8 dev eth0.2  table local  metric 256 
ff00::/8 dev wlan0  table local  metric 256 
unreachable default dev lo  table unspec  proto kernel  metric 4294967295  error -128

mwan3 status

Interface status:
 interface wan is online (tracking active)
 interface wwan is online (tracking active)

Policy balanced:
 wwan (50%)
 wan (50%)

Policy wan1only:
 wan (100%)

Policy wan2_only:
 unreachable

Policy wan2_wan:
 wan (100%)

Policy wan_only:
 wan (100%)

Policy wan_wan2:
 wan (100%)

Known networks:
 224.0.0.0/3
 192.168.0.0
 192.168.2.0
 192.168.2.255
 192.168.0.1
 192.168.0.100
 127.0.0.0
 127.0.0.0/8
 192.168.1.255
 192.168.0.0/24
 192.168.1.1
 127.0.0.1
 192.168.1.0
 192.168.2.1
 192.168.1.0/24
 192.168.2.0/24
 127.255.255.255
 192.168.0.255
 192.168.1.13

Active rules:
  0     0 - wan1only  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set myipset dst multiport sports 0:65535 multiport dports 0:65535
  174 14932 - balanced  all  --  *      *       0.0.0.0/0            0.0.0.0/0

ulmwind · January 13, 2020, 8:20pm

gfucka:

 0     0 - wan1only  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set myipset dst multiport sports 0:65535 multiport dports 0:65535
  174 14932 - balanced  all  --  *      *       0.0.0.0/0            0.0.0.0/0

We see your ipset, but with slightly different name.

Please, make following:

Remove ipset from /etc/dnsmasq.conf, remove your rule from mwan3 config.
Reboot router, and check output of
ip rule show
Compare it with output before.

gfucka · January 13, 2020, 8:41pm

Sorry I don't understand whay you mean.
Please can you explain me?

ulmwind · January 13, 2020, 9:51pm

I've quoted two strings from your output from mwan3 status. One string corresponds to your ipset, but it's name is 'myipset'.

gfucka · January 14, 2020, 5:40am

Ok now I understand.
Yes I make a mistake when I write the first request the correct rule in dnsmasq.conf is

ipset=/destionation-address.com/myipset.

So do you know why it not works?

Thanks

ulmwind · January 14, 2020, 8:48am

No, but we'll debug it.
What is name and IP of interface wan1(mwan3)? I suppose, eth0.2 192.168.1.1?
Please, give output of
iptables -S -t mangle
We should see your ipset and marking packets in output.

gfucka · January 14, 2020, 8:05pm

Sorry I was out of home.

yes eth0.2

-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N fwmark
-N mssfix
-N mwan3_connected
-N mwan3_hook
-N mwan3_iface_wan
-N mwan3_iface_wwan
-N mwan3_ifaces
-N mwan3_policy_balanced
-N mwan3_policy_wan1only
-N mwan3_policy_wan2_only
-N mwan3_policy_wan2_wan
-N mwan3_policy_wan_only
-N mwan3_policy_wan_wan2
-N mwan3_rules
-N mwan3_track
-A PREROUTING -j mwan3_hook
-A PREROUTING -j fwmark
-A FORWARD -j mssfix
-A OUTPUT -j mwan3_hook
-A mssfix -o eth0.2 -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "wan (mtu_fix)" -j TCPMSS --clamp-mss-to-pmtu
-A mssfix -o eth0.3 -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "wan (mtu_fix)" -j TCPMSS --clamp-mss-to-pmtu
-A mssfix -o wlan0 -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "wan (mtu_fix)" -j TCPMSS --clamp-mss-to-pmtu
-A mwan3_connected -m set --match-set mwan3_connected dst -j MARK --set-xmark 0xff00/0xff00
-A mwan3_hook -j CONNMARK --restore-mark --nfmask 0xff00 --ctmask 0xff00
-A mwan3_hook -m mark --mark 0x0/0xff00 -j mwan3_ifaces
-A mwan3_hook -m mark --mark 0x0/0xff00 -j mwan3_connected
-A mwan3_hook -m mark --mark 0x0/0xff00 -j mwan3_track
-A mwan3_hook -m mark --mark 0x0/0xff00 -j mwan3_rules
-A mwan3_hook -j CONNMARK --save-mark --nfmask 0xff00 --ctmask 0xff00
-A mwan3_hook -m mark ! --mark 0xff00/0xff00 -j mwan3_connected
-A mwan3_iface_wan -i eth0.2 -m set --match-set mwan3_connected src -m mark --mark 0x0/0xff00 -m comment --comment default -j MARK --set-xmark 0xff00/0xff00
-A mwan3_iface_wan -i eth0.2 -m mark --mark 0x0/0xff00 -m comment --comment wan -j MARK --set-xmark 0x100/0xff00
-A mwan3_iface_wwan -i wlan0 -m set --match-set mwan3_connected src -m mark --mark 0x0/0xff00 -m comment --comment default -j MARK --set-xmark 0xff00/0xff00
-A mwan3_iface_wwan -i wlan0 -m mark --mark 0x0/0xff00 -m comment --comment wwan -j MARK --set-xmark 0x200/0xff00
-A mwan3_ifaces -m mark --mark 0x0/0xff00 -j mwan3_iface_wan
-A mwan3_ifaces -m mark --mark 0x0/0xff00 -j mwan3_iface_wwan
-A mwan3_policy_balanced -m mark --mark 0x0/0xff00 -m statistic --mode random --probability 0.50000000000 -m comment --comment "wwan 1 2" -j MARK --set-xmark 0x200/0xff00
-A mwan3_policy_balanced -m mark --mark 0x0/0xff00 -m comment --comment "wan 1 1" -j MARK --set-xmark 0x100/0xff00
-A mwan3_policy_wan1only -m mark --mark 0x0/0xff00 -m comment --comment "wan 1 1" -j MARK --set-xmark 0x100/0xff00
-A mwan3_policy_wan2_only -m mark --mark 0x0/0xff00 -m comment --comment unreachable -j MARK --set-xmark 0xfe00/0xff00
-A mwan3_policy_wan2_wan -m mark --mark 0x0/0xff00 -m comment --comment "wan 3 3" -j MARK --set-xmark 0x100/0xff00
-A mwan3_policy_wan_only -m mark --mark 0x0/0xff00 -m comment --comment "wan 3 3" -j MARK --set-xmark 0x100/0xff00
-A mwan3_policy_wan_wan2 -m mark --mark 0x0/0xff00 -m comment --comment "wan 3 3" -j MARK --set-xmark 0x100/0xff00
-A mwan3_rules -m set --match-set myipset dst -m mark --mark 0x0/0xff00 -m comment --comment myrule -j mwan3_policy_wan1only
-A mwan3_rules -m mark --mark 0x0/0xff00 -m comment --comment default_rule -j mwan3_policy_balanced
-A mwan3_track -p icmp -m set --match-set mwan3_track_wan dst -m icmp --icmp-type 8 -m length --length 32 -j MARK --set-xmark 0xff00/0xff00
-A mwan3_track -p icmp -m set --match-set mwan3_track_wwan dst -m icmp --icmp-type 8 -m length --length 32 -j MARK --set-xmark 0xff00/0xff00

ulmwind · January 14, 2020, 8:18pm

OK, now we have:

matching ipset:

-A mwan3_rules -m set --match-set myipset dst -m mark --mark 0x0/0xff00 -m comment --comment myrule -j mwan3_policy_wan1only

marking packets according to mwan3_policy_wan1only:

-A mwan3_policy_wan1only -m mark --mark 0x0/0xff00 -m comment --comment "wan 1 1" -j MARK --set-xmark 0x100/0xff00

routing them by table 1:

2001:   from all fwmark 0x100/0xff00 lookup 1

default route for table 1 is eth0.2:

default via 192.168.1.1 dev eth0.2  table 1

How have you checked, that it doesn't work? Please, give output of
traceroute youtube.com

gfucka · January 14, 2020, 8:48pm

Sorry it not is youtube but a private address so I must obfuscate it.
This is the output:

traceroute to xxxxxxx.com (85.114.x.x), 30 hops max, 38 byte packets
 1  192.168.1.1 (192.168.1.1)  0.628 ms  0.534 ms  0.621 ms
 2  *  10.205.182.61 (10.205.182.61)  58.813 ms  *
 3  *  *  172.17.144.160 (172.17.144.160)  10.041 ms
 4  *  *  172.17.145.228 (172.17.145.228)  12.758 ms
 5  *  172.19.245.81 (172.19.245.81)  22.399 ms  *
 6  etrunk41.milano50.mil.seabone.net (195.22.192.80)  23.992 ms  etrunk14.milano1.mil.seabone.net (93.186.128.213)  26.199 ms  23.772 ms
 7  *  ae11.milano58.mil.seabone.net (195.22.208.79)  23.057 ms  *
 8  *  *  *
 9  ae-1-3103.edge5.Dusseldorf1.Level3.net (4.69.136.246)  38.146 ms  38.210 ms  38.430 ms
10  *  *  VtelJO3.Frankfurt1.Level3.net (212.162.19.26)  37.832 ms
11  *  eth1-1.ipcar.bb.as24961.net (62.141.47.106)  54.584 ms  *
12  xxxxxxxx.dedicated.server-hosting.expert (xx.xx.xx.xx)  35.876 ms  *  34.724 ms

ulmwind · January 14, 2020, 8:49pm

Sorry, it goes via 192.168.1.1, gateway on eth0.2, so what do you want?

gfucka · January 14, 2020, 8:55pm

Ok but if with a pc connected to openwrt router I try to access the url it is unreachable (because it is reacheable only with wan1) so I think it goes on wan2...

ulmwind · January 14, 2020, 8:57pm

Run traceroute (or tracert in Windows) from PC, check IP's of sites initially. Does PC use the same DNS? Check also traceroute by IP. Also disable wan2 (leaving only wan1) and check, whether site works.

gfucka · January 14, 2020, 9:09pm

I run traceroute from PC but it just show the openwrt router ip as hop:

traceroute to xxxxxxx.com (85.114.x.x), 64 hops max
  1   192.168.2.1  0,450ms  0,341ms  0,317ms 
  2   10.161.xxx.xx 187,092ms  214,425ms  285,287ms 
  3   10.205.xxx.xx  159,821ms  250,059ms  241,358ms 
......

I use DHCP on opewrt router so the DNS is served by router or not?

ulmwind · January 14, 2020, 9:14pm

It should be, but it is strange, that next hop is 10.161.224.85. I can't understand, what interface is used.

I've checked on my router and PC:

traceroute to 85.114.134.X (85.114.134.142), 30 hops max, 38 byte packets
 1  178.170.Y.Z (178.170.Y.Z) 

  1    <1 мс    <1 мс    <1 мс  192.168.1.1
  2    64 ms    64 ms    63 ms  178.170.Y.Z

gfucka · January 14, 2020, 9:21pm

Ok I understand why:

ping -c 1 -I eth0.1 10.161.xxx.xx
PING 10.161.xxx.xx (10.161.xxx.xx): 56 data bytes
^C
--- 10.161.xxx.xx ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
root@OpenWrt:~# ping -c 1 -I wlan0 10.161.xxx.xx
PING 10.161.xxx.xx (10.161.xxx.xx): 56 data bytes
64 bytes from 10.161.xxx.xx: seq=0 ttl=64 time=58.170 ms

--- 10.161.xxx.xx ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 58.170/58.170/58.170 ms

10.161.xxx.xxis an IP only reachable by the wan2! but why it use wan2??
It Is possible a DNS problem? Because wan1 and wan2 use differents DNS. Wan1 use 8.8.8.8 but with Wan2 I can't use it I must use a different one because it is behind a proxy

ulmwind · January 14, 2020, 9:51pm

Since they resolve the same IP, it doesn't matter.

Something interesting is here:

-A mwan3_iface_wwan -i wlan0 -m mark --mark 0x0/0xff00 -m comment --comment wwan -j MARK --set-xmark 0x200/0xff00

It marks packets coming from interface wlan0 and routes them into table 2 (wan2).

Sorry, now I don't understand, what means table 2. So what is 'wan2' from mwan3? What is interface wlan0 and why have you pinged via it?

gfucka · January 14, 2020, 11:28pm

Ok I try to explain what is my configuration.
I have eth0.2 as wan1 in mwan3 and it is a normal ADSL line. I also have wlan0 as wan2 in mwan3 and it is a WiFi LTE router but the SIM is a company SIM so it is behind proxy.
What I need is just when I request some specific URL I want to use just eth0.2 aka wan1.

ulmwind · January 15, 2020, 9:29am

OK, got it.

It is strange behavior, because tracert from PC doesn't show next gateway. Could you check tracert for another site?