Topic: mwan3; multi-wan policy routing (general topic)

The content of this topic has been archived between 22 May 2013 and 6 May 2018. Unfortunately there are posts – most likely complete pages – missing.

Page 20 of 65

Post #476

Ramesh_P

10 Sep 2013, 16:36

Hi Adze,

Yes now its seems to be working, after disabling multi wan package.

and also in MWAN3 Multi-wan page(gui) i could see both interfaces are online.

I'm very much new to this please help to complete this task.

now tell me how to test wheather mwan3 is working or not

Test case:(need test case)
=======

fail over: ????
-----------

load balancing : ????
-------------------

Here trouble shouting commands results.

root@OpenWrt:/# ping -c 1 -I nas0 www.google.com
PING www.google.com (74.125.128.99): 56 data bytes
64 bytes from 74.125.128.99: seq=0 ttl=44 time=132.718 ms

--- www.google.com ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 132.718/132.718/132.718 ms
root@OpenWrt:/# ping -c 1 -I nas1 www.google.com
PING www.google.com (74.125.128.105): 56 data bytes
64 bytes from 74.125.128.105: seq=0 ttl=44 time=172.723 ms

--- www.google.com ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 172.723/172.723/172.723 ms

#############
route
#############

root@OpenWrt:/# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 200.200.59.1 0.0.0.0 UG 10 0 0 nas0
0.0.0.0 200.200.60.1 0.0.0.0 UG 20 0 0 nas1
192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 br-lan
200.200.59.0 0.0.0.0 255.255.255.0 U 10 0 0 nas0
200.200.60.0 0.0.0.0 255.255.255.0 U 20 0 0 nas1

############
ip rule show
############

root@OpenWrt:/# ip rule show
0: from all lookup local
1001: from all fwmark 0x100/0xff00 lookup 1001
1002: from all fwmark 0x200/0xff00 lookup 1002
1016: from all fwmark 0x1000/0xff00 lookup 1016
1017: from all fwmark 0x1100/0xff00 lookup 1017
1018: from all fwmark 0x1200/0xff00 lookup 1018
1019: from all fwmark 0x1300/0xff00 lookup 1019
1020: from all fwmark 0x1400/0xff00 lookup 1020
32766: from all lookup main
32767: from all lookup default

root@OpenWrt:/# ip route list table 1001
default via 200.200.59.1 dev nas0

root@OpenWrt:/# ip route list table 1002
default via 200.200.60.1 dev nas1

root@OpenWrt:/# ip route list table 1016
default via 200.200.59.1 dev nas0 metric 1

root@OpenWrt:/# ip route list table 1017
default via 200.200.60.1 dev nas1 metric 1

root@OpenWrt:/# ip route list table 1018
default metric 1
nexthop via 200.200.60.1 dev nas1 weight 2
nexthop via 200.200.59.1 dev nas0 weight 3

root@OpenWrt:/# ip route list table 1019
default via 200.200.59.1 dev nas0 metric 1
default via 200.200.60.1 dev nas1 metric 2

root@OpenWrt:/# ip route list table 1020
default via 200.200.60.1 dev nas1 metric 1
default via 200.200.59.1 dev nas0 metric 2

##############
iptable rules
##############

root@OpenWrt:/# iptables -L mwan3_pre -t mangle -v -n
Chain mwan3_pre (2 references)
pkts bytes target prot opt in out source destination
4069 509K CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 CONNMARK restore mask 0xff00
82 6888 MARK all -- nas1 * 0.0.0.0/0 0.0.0.0/0 MARK xset 0x8200/0xff00
187 15352 MARK all -- nas0 * 0.0.0.0/0 0.0.0.0/0 MARK xset 0x8100/0xff00
1008 72183 mwan3_default all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0xff00
298 24599 mwan3_interfaces all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0xff00
13 1227 mwan3_rules all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0xff00

root@OpenWrt:/# iptables -L mwan3_post -t mangle -v -n
Chain mwan3_post (2 references)
pkts bytes target prot opt in out source destination
86 7224 MARK all -- * nas1 0.0.0.0/0 0.0.0.0/0 mark match ! 0x7f00/0xff00 MARK xset 0x200/0xff00
207 16780 MARK all -- * nas0 0.0.0.0/0 0.0.0.0/0 mark match ! 0x7f00/0xff00 MARK xset 0x100/0xff00
305 25204 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x8000/0x8000 MARK and 0xffff7fff
4363 544K CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 CONNMARK save mask 0xff00

root@OpenWrt:/# iptables -L mwan3_default -t mangle -v -n
Chain mwan3_default (1 references)
pkts bytes target prot opt in out source destination
0 0 MARK all -- * * 0.0.0.0/0 224.0.0.0/3 mark match 0x0/0xff00 MARK xset 0x7f00/0xff00
4 240 MARK all -- * * 0.0.0.0/0 127.0.0.0/8 mark match 0x0/0xff00 MARK xset 0x7f00/0xff00
514 30736 MARK all -- * * 0.0.0.0/0 192.168.1.0/24 mark match 0x0/0xff00 MARK xset 0x7f00/0xff00
134 11256 MARK all -- * * 0.0.0.0/0 200.200.59.0/24 mark match 0x0/0xff00 MARK xset 0x7f00/0xff00
87 7308 MARK all -- * * 0.0.0.0/0 200.200.60.0/24 mark match 0x0/0xff00 MARK xset 0x7f00/0xff00

root@OpenWrt:/# iptables -L mwan3_rules -t mangle -v -n
Chain mwan3_rules (1 references)
pkts bytes target prot opt in out source destination
0 0 MARK tcp -- * * 192.168.21.0/24 0.0.0.0/0 multiport sports 0:65535 multiport dports 563 mark match 0x0/0xff00 MARK xset 0x1100/0xf
f00
0 0 MARK tcp -- * * 192.168.21.0/24 0.0.0.0/0 multiport sports 0:65535 multiport dports 995 mark match 0x0/0xff00 MARK xset 0x1000/0xf
f00
0 0 MARK tcp -- * * 0.0.0.0/0 88.154.0.0/16 multiport sports 0:65535 multiport dports 1024:65535 mark match 0x0/0xff00 statistic mod
e random probability 0.60000000009 MARK xset 0x100/0xff00
0 0 MARK tcp -- * * 0.0.0.0/0 88.154.0.0/16 multiport sports 0:65535 multiport dports 1024:65535 mark match 0x0/0xff00 statistic mod
e random probability 0.99999999953 MARK xset 0x200/0xff00
0 0 MARK tcp -- * * 0.0.0.0/0 77.11.41.0/24 multiport sports 0:65535 multiport dports 1024:65535 mark match 0x0/0xff00 MARK xset 0x1
300/0xff00
0 0 MARK udp -- * * 0.0.0.0/0 112.136.0.0/16 multiport sports 0:65535 multiport dports 5352 mark match 0x0/0xff00 MARK xset 0x1400/0x
ff00
0 0 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0xff00 MARK xset 0x1200/0xff00

thanks in advance.

(Last edited by Ramesh_P on 10 Sep 2013, 16:37)

Post #477

Adze

10 Sep 2013, 17:30

Ramesh_P wrote:

I'm very much new to this please help to complete this task. Now tell me how to test wheather mwan3 is working or not.

Please take some time and read through this topic and the wiki. It has a lot of info on this.

Thnx

Post #478

n_dandanov

13 Sep 2013, 00:50

Hello, Adze,

Sorry for my late response. Once again, thank You a lot for all the support!
Currently I'm using openDNS service in order to avoid the mentioned issue.
However, wouldn't adding a DNS server "binding" option to the mwan package be a good idea? When there's an event and we're about to route traffic through one of the interfaces, we would also change the DNS configuration file to use the DNS servers provided by the ISP, connected to that same interface. Or perhaps changes should be made obly when a link goes down?

Best regards,
Nickolay

Adze wrote:

Just a small note on dns and mwan3. If you use the dns servers from your isp, it is very common that resolving fails if you try it with a wan interface (source ip address) that is not from that specific provider. I recommend to use opendns or google's dns service as they work from any ip address. It is not an mwan3 issue.

Post #479

Thomymaster

17 Sep 2013, 09:12

Another question, is the loopback trick (in rc.local) still needed in the current build to load-balance traffic coming from the router itself?

Post #480

Adze

17 Sep 2013, 10:24

Thomymaster wrote:

Another question, is the loopback trick (in rc.local) still needed in the current build to load-balance traffic coming from the router itself?

Short answer is no...

You can now use the reroute interface option to also load-balance traffic originating from the wan interface itself. However, the source ip adres of a packet originating from the router itself will be the ip address of the interface with the lowest metric default route in the main routing table. If that packet is load-balanced and leaves another wan interface it then gets natted.

If you wish to use an alternative source ip address (eg. loopback), you still need the loopback trick.

Post #481

Thomymaster

18 Sep 2013, 08:49

Sorry Adze but i don't unterstand properly.

I have interface 'wan' with a interface metric of 20 and interface 'wwan' with a interface metric of 10.
Or are you talkin about the metric in the mwan3 config?

config rule 'default'
        option proto 'all'
        option use_policy 'wan_pri_wwan_sec'
        option dest_ip '0.0.0.0/0'

config interface 'wan'
        option enabled '1'
        option reliability '1'
        option count '1'
        option timeout '2'
        option interval '5'
        option down '3'
        option up '3'
        list track_ip '173.194.69.94'
        list track_ip '8.8.8.8'

config interface 'wwan'
        option enabled '1'
        option reliability '1'
        option count '1'
        option timeout '2'
        option interval '5'
        option down '3'
        option up '3'
        list track_ip '173.194.69.94'
        list track_ip '8.8.8.8'

config member 'wwan_m1_w1'
        option interface 'wwan'
        option metric '1'
        option weight '1'

config member 'wwan_m2_w1'
        option interface 'wwan'
        option metric '2'
        option weight '1'

config policy 'wan_pri_wwan_sec'
        list use_member 'wan_m1_w1'
        list use_member 'wwan_m2_w1'

config member 'wan_m2_w1'
        option interface 'wan'
        option metric '2'
        option weight '1'

config member 'wan_m1_w1'
        option interface 'wan'
        option metric '1'
        option weight '1'

This config should be used for failover from wan to wwan which works.

So what do you mean with:

"However, the source ip adres of a packet originating from the router itself will be the ip address of the interface with the lowest metric default route in the main routing table. If that packet is load-balanced and leaves another wan interface it then gets natted."

So traffic from my subnet is load-balanced normally and has the source IP of the wan interface as well as router traffic. When the router switches to wwan, the normal traffic has the source IP of the wwan interface, but router traffic still has the source IP of the wan interface but leaves the router through the wwan interface?

Post #482

Adze

18 Sep 2013, 22:13

Hi Thomymaster,

Let me try to explain with an example. Imagine we have a router with two wan interfaces (wan1 and wan2) with ip addresses 1.1.1.1 and 2.2.2.2. These are configured in the network config with metrics 10 and 20. Now we want to telnet to 3.3.3.3 from the router.

Before a packet can be routed, it first has to be created. We know the destination address, but we don't know the source yet. The kernel determines it based on the main routing table.

In our example wan1 has the lowest metric default gateway, so the source ip address is set to 1.1.1.1. If you were to set the default gateway with the loopback trick, the source ip address of the created packets will then be the ip address you gave it. This is very important for creating mwan3 rules, as this address is used as the src-ip value. Now the packet is routed using the custom mwan3 rules and routes.

So what does the option reroute do?

A newly router originated packet with an ip address of a wan interface, is per default always routed out that interface. If you set the reroute option to "1", mwan3 will also load-balance that packet based on the rules.

Post #483

garretwcox

19 Sep 2013, 06:44

Hi everyone, I'm having a very strange issue.
Load balancing, failover, all seems to work marvelously from within and behind the mwan3 test router. (AWESOME work by the way). However, the router cannot ping or be pinged by other clients in the wan subnet besides the gateway.

Here's my layout:
Gateway to the internet:10.0.77.1

Desktop: 10.0.77.2

Mwan3 test router:
WAN1: 10.0.77.3
USBWAN: 192.168.32.10
LAN: 192.168.1.1

Laptop: 192.168.1.2

The ONLY issue I have is when I ping the mwan3 router from the deskop, I get no replies.
When I ping the desktop from the mwan3 router.. I get EXACTLY ONE reply, then no more. Crazy huh! I've never seen that before!

Even more crazy..EVERYTHING ELSE WORKS.
The Laptop behind the mwan3 router can ping everyone.
The mwan3 router can ping the gateway! The gateway can ping the mwan3 router!

Now, I know what you are thinking: same thing I was: the problem is the desktop!
Nope....I actually have several machines, running several different OSes, in the 10.0.77.0/24 subnet, and they all show the exact same symptoms.
Furthermore, I can make the issue go away by simply setting "Enabled" to "no" in the mwan3 interfaces config for wan1.

I've already tested this with and without reroute enabled on both interfaces.

does anyone have any ideas? Here's the info from the troubleshooting page:

Software versions : 

OpenWrt - OpenWrt Attitude Adjustment 12.09 (r36088)
mwan3 - 1.2-17
luci-app-mwan3 - 1.1-13

Firewall default output policy (must be ACCEPT) : 

ACCEPT

Output of "ip route show" : 

default via 192.168.32.2 dev eth2  proto static 
default via 10.0.77.1 dev eth1  proto static  metric 10 
10.0.77.0/24 dev eth1  proto static  scope link  metric 10 
192.168.1.0/24 dev br-lan  proto kernel  scope link  src 192.168.1.1 
192.168.32.0/24 dev eth2  proto kernel  scope link  src 192.168.32.10

Output of "ip rule show" : 

0:    from all lookup local 
1001:    from all fwmark 0x100/0xff00 lookup 1001 
1002:    from all fwmark 0x200/0xff00 lookup 1002 
1016:    from all fwmark 0x1000/0xff00 lookup 1016 
1017:    from all fwmark 0x1100/0xff00 lookup 1017 
1018:    from all fwmark 0x1200/0xff00 lookup 1018 
1019:    from all fwmark 0x1300/0xff00 lookup 1019 
1020:    from all fwmark 0x1400/0xff00 lookup 1020 
32766:    from all lookup main 
32767:    from all lookup default

Output of "ip route list table 1001-1099" (1001-1015 = interface tables, 1016-1099 = policy tables) : 

1001
default via 10.0.77.1 dev eth1 
1002
default via 192.168.32.2 dev eth2 
1016
default via 10.0.77.1 dev eth1  metric 1 
1017
default via 192.168.32.2 dev eth2  metric 1 
1018
default  metric 1 
    nexthop via 192.168.32.2  dev eth2 weight 2
    nexthop via 10.0.77.1  dev eth1 weight 3
1019
default via 10.0.77.1 dev eth1  metric 1 
default via 192.168.32.2 dev eth2  metric 2 
1020
default via 192.168.32.2 dev eth2  metric 1 
default via 10.0.77.1 dev eth1  metric 2

Output of "iptables -L -t mangle -v -n | awk '/mwan3/' RS=" : 

Chain PREROUTING (policy ACCEPT 3794 packets, 644K bytes)
 pkts bytes target     prot opt in     out     source               destination         
 3947  695K mwan3_pre  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain INPUT (policy ACCEPT 2705 packets, 287K bytes)
 pkts bytes target     prot opt in     out     source               destination         
 2779  296K mwan3_post  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT 2540 packets, 581K bytes)
 pkts bytes target     prot opt in     out     source               destination         
 2599  585K mwan3_pre  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain POSTROUTING (policy ACCEPT 3002 packets, 719K bytes)
 pkts bytes target     prot opt in     out     source               destination         
 3126  761K mwan3_post  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain mwan3_default (1 references)
 pkts bytes target     prot opt in     out     source               destination         
  114  6161 MARK       all  --  *      *       0.0.0.0/0            224.0.0.0/3         mark match 0x0/0xff00 MARK xset 0x7f00/0xff00 
    0     0 MARK       all  --  *      *       0.0.0.0/0            127.0.0.0/8         mark match 0x0/0xff00 MARK xset 0x7f00/0xff00 
    0     0 MARK       all  --  *      *       0.0.0.0/0            10.0.77.0/24        mark match 0x0/0xff00 MARK xset 0x7f00/0xff00 
  161  9132 MARK       all  --  *      *       0.0.0.0/0            192.168.1.0/24      mark match 0x0/0xff00 MARK xset 0x7f00/0xff00 
    0     0 MARK       all  --  *      *       0.0.0.0/0            192.168.32.0/24     mark match 0x0/0xff00 MARK xset 0x7f00/0xff00 

Chain mwan3_post (2 references)
 pkts bytes target     prot opt in     out     source               destination         
  585 45312 MARK       all  --  *      eth2    0.0.0.0/0            0.0.0.0/0           mark match !0x7f00/0xff00 MARK xset 0x200/0xff00 
 1062  143K MARK       all  --  *      eth1    0.0.0.0/0            0.0.0.0/0           mark match !0x7f00/0xff00 MARK xset 0x100/0xff00 
 1807  240K MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0           mark match 0x8000/0x8000 MARK and 0xffff7fff 
 5905 1058K CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0           CONNMARK save mask 0xff00 

Chain mwan3_pre (2 references)
 pkts bytes target     prot opt in     out     source               destination         
 6546 1280K CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0           CONNMARK restore mask 0xff00 
  714 93234 MARK       all  --  eth2   *       0.0.0.0/0            0.0.0.0/0           MARK xset 0x8200/0xff00 
 1613  361K MARK       all  --  eth1   *       0.0.0.0/0            0.0.0.0/0           MARK xset 0x8100/0xff00 
 1373  109K mwan3_default  all  --  *      *       0.0.0.0/0            0.0.0.0/0           mark match 0x0/0xff00 
 1078 91538 mwan3_wan  all  --  *      *       0.0.0.0/0            0.0.0.0/0           mark match 0x0/0xff00 
 1078 91538 mwan3_rules  all  --  *      *       0.0.0.0/0            0.0.0.0/0           mark match 0x0/0xff00 

Chain mwan3_rules (1 references)
 pkts bytes target     prot opt in     out     source               destination         
 1058 89122 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0           mark match 0x0/0xff00 MARK xset 0x1300/0xff00 

Chain mwan3_wan (1 references)
 pkts bytes target     prot opt in     out     source               destination

Output of "ifconfig" : 

br-lan    Link encap:Ethernet  HWaddr 76:44:01:93:57:DB  
          inet addr:192.168.1.1  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::7444:1ff:fe93:57db/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:6912 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6468 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:1244701 (1.1 MiB)  TX bytes:3710324 (3.5 MiB)

eth0      Link encap:Ethernet  HWaddr 76:44:01:93:57:DB  
          inet6 addr: fe80::7444:1ff:fe93:57db/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:7066 errors:0 dropped:6 overruns:23 frame:0
          TX packets:6461 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:1385466 (1.3 MiB)  TX bytes:3735767 (3.5 MiB)
          Interrupt:4 

eth0.1    Link encap:Ethernet  HWaddr 76:44:01:93:57:DB  
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:7026 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6452 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:1254158 (1.1 MiB)  TX bytes:3708228 (3.5 MiB)

eth1      Link encap:Ethernet  HWaddr 74:44:01:93:57:DC  
          inet addr:10.0.77.3  Bcast:10.0.77.255  Mask:255.255.255.0
          inet6 addr: fe80::7644:1ff:fe93:57dc/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:9160 errors:0 dropped:0 overruns:0 frame:0
          TX packets:4931 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:3484445 (3.3 MiB)  TX bytes:986477 (963.3 KiB)
          Interrupt:5 

eth2      Link encap:Ethernet  HWaddr D0:57:85:74:9F:1E  
          inet addr:192.168.32.10  Bcast:192.168.32.255  Mask:255.255.255.0
          inet6 addr: 2600:1005:b10a:a5c6:d257:85ff:fe74:9f1e/64 Scope:Global
          inet6 addr: fe80::d257:85ff:fe74:9f1e/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1849 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1523 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:236730 (231.1 KiB)  TX bytes:157509 (153.8 KiB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:34 errors:0 dropped:0 overruns:0 frame:0
          TX packets:34 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:3469 (3.3 KiB)  TX bytes:3469 (3.3 KiB)

Output of "cat /etc/config/mwan3" : 

config interface 'wan1'
    option reliability '2'
    option count '1'
    option timeout '2'
    option interval '5'
    option down '3'
    option up '8'
    list track_ip '8.8.8.8'
    list track_ip '4.2.2.3'
    list track_ip '208.67.222.222'
    option reroute '1'
    option enabled '1'

config interface 'usbwan'
    option enabled '1'
    option count '1'
    option timeout '2'
    option interval '5'
    option down '3'
    option up '8'
    option reliability '2'
    option reroute '1'
    list track_ip '8.8.4.4'
    list track_ip '4.2.2.4'
    list track_ip '208.67.220.220'

config member 'wan1_m1_w3'
    option interface 'wan1'
    option metric '1'
    option weight '3'

config member 'wan1_m2_w3'
    option interface 'wan1'
    option metric '2'
    option weight '3'

config member 'wan2_m1_w2'
    option interface 'usbwan'
    option metric '1'
    option weight '2'

config member 'wan2_m2_w2'
    option interface 'usbwan'
    option metric '2'
    option weight '2'

config policy 'wan1_only'
    list use_member 'wan1_m1_w3'

config policy 'wan2_only'
    list use_member 'wan2_m1_w2'

config policy 'wan1_wan2_loadbalanced'
    list use_member 'wan1_m1_w3'
    list use_member 'wan2_m1_w2'

config policy 'wan1_pri_wan2_sec'
    list use_member 'wan1_m1_w3'
    list use_member 'wan2_m2_w2'

config policy 'wan2_pri_wan1_sec'
    list use_member 'wan1_m2_w3'
    list use_member 'wan2_m1_w2'

config rule 'rule6'
    option dest_ip '0.0.0.0/0'
    option proto 'all'
    option use_policy 'wan1_pri_wan2_sec'

Output of "cat /etc/config/network" : 

config interface 'loopback'
    option ifname 'lo'
    option proto 'static'
    option ipaddr '127.0.0.1'
    option netmask '255.0.0.0'

config interface 'lan'
    option ifname 'eth0.1'
    option type 'bridge'
    option proto 'static'
    option ipaddr '192.168.1.1'
    option netmask '255.255.255.0'

config interface 'wan1'
    option ifname 'eth1'
    option proto 'static'
    option ipaddr '10.0.77.3'
    option netmask '255.255.255.0'
    option gateway '10.0.77.1'
    option dns '8.8.8.8'
    option metric '10'

config switch
    option name 'rtl8366s'
    option reset '1'
    option enable_vlan '1'
    option blinkrate '2'

config switch_vlan
    option device 'rtl8366s'
    option vlan '1'
    option ports '0 1 2 3 5t'

config switch_port
    option device 'rtl8366s'
    option port '1'
    option led '6'

config switch_port
    option device 'rtl8366s'
    option port '2'
    option led '9'

config switch_port
    option device 'rtl8366s'
    option port '5'
    option led '2'

config interface 'usbwan'
    option ifname 'eth2'
    option _orig_ifname 'eth2'
    option _orig_bridge 'false'
    option proto 'static'
    option ipaddr '192.168.32.10'
    option netmask '255.255.255.0'
    option gateway '192.168.32.2'
    option dns '8.8.4.4'
    option accept_ra '1'

Post #484

Thomymaster

19 Sep 2013, 08:30

OK so if i get this correctly, then i just have to set reroute to 1 and the router generated traffic (i.e. dns and ddns updates) is routed via mwan3.

wan: 10.0.0.1
wwan: 10.0.1.1

So normally the source IP of the packets are 10.0.0.1 and when the wan line goes down, they are rewritten to 10.0.1.1 as they go out over the wwan interface. Or is the source IP still 10.0.0.1 and they are NATed via wwan?

Is that right?

Cheers Thomy

(Last edited by Thomymaster on 19 Sep 2013, 08:32)

Post #485

Adze

19 Sep 2013, 11:02

Thomymaster wrote:

OK so if i get this correctly, then i just have to set reroute to 1 and the router generated traffic (i.e. dns and ddns updates) is routed via mwan3.
wan: 10.0.0.1
wwan: 10.0.1.1
So normally the source IP of the packets are 10.0.0.1 and when the wan line goes down, they are rewritten to 10.0.1.1 as they go out over the wwan interface. Or is the source IP still 10.0.0.1 and they are NATed via wwan?
Is that right?

Cheers Thomy

The source IP is still 10.0.0.1 and they are indeed NATted via wwan.

(Last edited by Adze on 19 Sep 2013, 11:02)

Post #486

Adze

19 Sep 2013, 11:06

garretwcox wrote:

The ONLY issue I have is when I ping the mwan3 router from the deskop, I get no replies.
When I ping the desktop from the mwan3 router.. I get EXACTLY ONE reply, then no more. Crazy huh! I've never seen that before!

I will have to test this myself. It should work and your config looks ok-ish. You're only lacking a metric for your usbwan interface in your network config.

Post #487

garretwcox

20 Sep 2013, 02:55

Oh yeah, I had one, but I guess I lost it when I swapped usbwan from dchp to static while testing out different configs

Post #488

garretwcox

20 Sep 2013, 06:48

Okay, more details:
1. I was wrong about the laptop behind the mwan3 router being okay: I was just doing a single ping to test at first. It's actually exhibiting the same symptoms as pinging directly from the mwan3 router: 1 ping, then no more.

2. tracert from the desktop to the mwan3 router goes directly to 10.0.77.3
HOWEVER tracert from the laptop behind the mwan3 router (192.168.1.2) to the desktop (10.0.77.2) goes:
192.168.1.1 (okay, that makes sense, that's the mwan3 router)
10.0.77.1 (wtf!? why is it going to the gateway for an address in the same subnet as wan1?)
10.0.77.2

So...for shits and giggles, I got on the gateway (10.0.77.1) and set the LAN firewall zone to accept forward. Lo and behold...I can ping both ways again. This confirms that pings coming from the mwan3 router (10.0.77.3) think they have to hit the gateway (10.0.77.1) to get to the the 10.0.77.0/24 subnet. Everything else in 10.0.77.0/24 knows it can ping 10.0.77.3 directly, but they weren't getting the return pings since 10.0.77.3 was trying to route them through 10.0.77.1 and the firewall wasn't routing them.

Just as a third confirmation, all I have to do is to disable wan1 in the mwan3 config, and sure enough, a tracert from the laptop goes:
192.168.1.1
10.0.77.2

So....what in the world is causing the gateway to route this way? When I look at the routes under the mwan3 troubleshooting page or the status->routes page, everything looks correct: in fact, they look the same whether wan1 is enabled in mwan3 or not.

Post #489

dir2cas

24 Sep 2013, 15:11

Hello Adze, I had a view on your project for a long time and came out with my own customized multiwan solution taking some of the main features of mwan2/3. However, my solution is limited to tracking/operation only with 2 WAN interface and is based on my particular setup, which makes it quite uncompetitive and far behind mwan3 functionality.

However, one of the main advantages of my solution is the email notification ability and the correction of /tmp/resolv.conf.auto file (removing/adding the relevant DNS servers according to the WAN interfaces that are currently up and running).

I think, it will be nice to have an email notification working as an optional feature in mwan3.
Here is a sample configuration used in my script to send email notifications. It is based on ssmtp (very thin mail user agent). The main idea is that my code generates the whole configuration needed, does a backup of the original configuration and makes a custom conf. file, because ssmtp may operate with multiple configuration files at the same time. thus allowing to have multiple send/notification prifiles etc.

I am posting a short snippet of my code, including the variables, defined in the beginning, the email notify function and a sample call of the function.

# DEFINING VARIABLES
# Init start
START=99
SCRIPTNAME="mwan3"
LOG_MESSAGE="/tmp/${SCRIPTNAME}.msg"

# Email client settings (needed only if EMAIL_NOTIFY="1"):
HOSTNAME="hostname"
EMAIL_NOTIFY="1" # Enable (1) / Disable (0) email notification on failover actions
EMAIL_CLIENT="ssmtp" # The MUA (Mail user agent)
EMAIL_CLIENT_CONF="/etc/ssmtp/ssmtp_mwan3.conf" # Where the configuration of the mail clients resides in
EMAIL_USER="myuser@gmail.com" # The actual email account that is sending the email
EMAIL_PASSWD="mypassword"   # your email account password
SMTP_SERVER="smtp.gmail.com" # Outgoing mail server
DST_PORT="465" # Destiantion port used by the mail client (smtp port is 25 by default, without encryption)
EMAIL_DOMAIN="gmail.com" # Domain of the mail server
EMAIL_CLIENT_TLS="YES"     # "YES" or "NO" are valid options
EMAIL_SUBJECT="MY Network WAN status change" # Subject of the email message
# List of Email notification recipients
EMAIL_TO="user1@domain1.com user2@domain2.com"
EMAIL_SIGNATURE="Best Regards, OpenWrt"

########################################
# THE EMAIL FUNCTION ITSELF GENERATING THE CONFIGURATION OF THE EMAIL CLIENT AND SENDING THE ACTUAL MESSAGE USING
# THE PREDEFINED VARIABLES THAT COULD BE ALSO GET FROM AN EXTERNAL CONF FILE
EmailNotify() {
# Check if the MUA is installed
if [ ! -x "${EMAIL_CLIENT}" ]; then logger -s -t ${SCRIPTNAME} "Warning: Email client ${EMAIL_CLIENT} is not installed or configured on the system"; return 2; fi

# Checking if the Email client has already been configured for our account
sleep 1
#LOG_MESSAGE="${@}"
grep -e "${SCRIPTNAME}" ${EMAIL_CLIENT_CONF} &> /dev/null || {
    # Backing up if there is some old configuration first
    cp ${EMAIL_CLIENT_CONF} ${EMAIL_CLIENT_CONF}.backup > /dev/null 2>&1
    # Writing the Email Client configuration (SSMTP)
    echo -e "# ${EMAIL_CLIENT_CONF} -- a configuration file for sSMTP sendmail.\n###generated by ${SCRIPTNAME}\nroot=${EMAIL_USER}\nmailhub=${SMTP_SERVER}:${DST_PORT}\nrewriteDomain=${EMAIL_DOMAIN}\nhostname=${HOSTNAME}\nFromLineOverride=YES\nUseTLS=${EMAIL_CLIENT_TLS}\nAuthUser=${EMAIL_USER}\nAuthPass=${EMAIL_PASSWD}" > ${EMAIL_CLIENT_CONF}
    }

# Composing the Email message and pipe it to the smtp client in order to be sent
sleep 1
echo -e "From: <${EMAIL_USER}>\nSubject: ${EMAIL_SUBJECT}\n\n${HOSTNAME} alert notification message.\n$(cat "${LOG_MESSAGE}")\n\n${EMAIL_SIGNATURE}" | $(which $(echo "${EMAIL_CLIENT}")) -C ${EMAIL_CLIENT_CONF} ${EMAIL_TO}
sleep 1
logger -s -t ${SCRIPTNAME} "Sending email alert notification to $(echo "${EMAIL_TO}"|sed "s/ /,/g")"
unset EMAIL_PASSWD
unset LOG_MESSAGE
rm -f ${LOG_MESSAGE} &> /dev/null
}

########################################
# SAMPLE CALL OF EMAIL FUNCTION
if [ "${EMAIL_NOTIFY}" -eq "1" ]; then echo -e "Sending email notification to a list of predefined recipients"; sleep 10; EmailNotify "${LOG_MESSAGE}"; fi

Here is an example of how those variables could be placed in the mwan3 configuration file (/etc/config/mwan3). As I am not using the openwrt internal functions (like config_get, etc), I think that you will have no troubles if you like to integrate this function in your code.

config 'email_notify' 'gmail'
    option 'enabled' '1'  # Enable (1) / Disable (0) email notification on failover actions
    option 'hostname' 'myhostname' 
    option 'client' 'ssmtp'  # The MUA (Mail user agent)
    option 'client_conf' '/etc/ssmtp/ssmtp_mwan3.conf'  # Where the configuration of the mail clients resides in
    option 'user' 'myuser@gmail.com'   # The actual email account that is sending the email
    option 'passwd' 'mypassword'         # your email account password
    option 'smtp_server' 'smtp.gmail.com'  # Outgoing mail server
    option 'dst_port' '465' # Destiantion port used by the mail client (smtp port is 25 by default, without encryption)
    option 'email_domain' 'gmail.com'  # Domain of the mail server
    option 'use_tls' 'YES'  # "YES" or "NO" are valid options according to ssmpt configuration
    option 'email_subject' 'MY Network WAN status change' # Subject of the email message
    option 'email_signature' 'Best Regards, OpenWrt' # Optional Signature message
    list 'email_to' 'user1@domain1.com user2@domain2.com'  # List of Email notification recipients

The main idea is to have all the messages that you want to be sent, to be initially placed in a common variable (I am using a temp file in /tmp, RAM memory) after each mwan3track iteration (t.e track of all wan links), in order to have all the information after the relevant mwan3track provess.

ex.)
1) NO Trigger event (no link status change since the last track)
->no log messages -> NO email notification
2) Trigger event (one or more of the tracked links have changed its status)
-> relevant log messages to syslog and tmp message file -> email notification

Without exploring your code in details, probably the best places to call the email notification is in the mwan3track script file. If you find this useful, I will be glad to contribute to your project. If successful, luci integration of the new config section should not be a problem for the developers.

(Last edited by dir2cas on 24 Sep 2013, 15:14)

Post #490

Adze

24 Sep 2013, 15:24

dir2cas wrote:

Without exploring your code in details, probably the best places to call the email notification is in the mwan3track script file. If you find this useful, I will be glad to contribute to your project. If successful, luci integration of the new config section should not be a problem for the developers.

Thank you for the nice work. Mwan3 works with hotplug scripts. When an interface goes up or down a hotplug iface event is triggered. This means that all files in the /etc/hotplug.d/iface folder are run. You can place your custom email notification script there and it will start everytime an interface goes up or down... Imho there is no need to add it to mwan3.

Post #491

dir2cas

24 Sep 2013, 16:05

Yeah, You are absolutely right, I know that mwan3 is hotplug driven. The main idea behind integrating the email function in mwan3 is to parse some information that can be generated by mwan3 checks to the email function. It could be like e plug-in to mwan3 and could be enabled optionaly.

I think that the major of the work is to translate the configuration section in order to use the openwrt internal configuration functions. This will allow us to place all the configuration in /etc/config/mwan3 or another conf file.

The other part of the code should be easily integrated. I also could alter the code in order to replect the new variables names.

However, do you plan to add such functionality to mwan3?

Post #492

Thomymaster

25 Sep 2013, 12:48

This would be very nice

Post #493

Adze

27 Sep 2013, 14:23

garretwcox wrote:

Hi everyone, I'm having a very strange issue.
Load balancing, failover, all seems to work marvelously from within and behind the mwan3 test router. (AWESOME work by the way). However, the router cannot ping or be pinged by other clients in the wan subnet besides the gateway.

Sorry for the late reply. I now have a test setup and can reproduce your problem. I can ping hosts form the router directly connected to wan interfaces, but hosts can't ping the router. I will address this problem in the next version. Expect it within a couple of days...

Thnx

Post #494

Adze

27 Sep 2013, 21:15

garretwcox wrote:

Hi everyone, I'm having a very strange issue.
Load balancing, failover, all seems to work marvelously from within and behind the mwan3 test router. (AWESOME work by the way). However, the router cannot ping or be pinged by other clients in the wan subnet besides the gateway.

This problem is fixed in version 1.2-19. Thanks for noticing!

Post #495

Bluse-Blue

27 Sep 2013, 22:21

Hi all,

I do not get mwan3 to work as I expected and I would ask you for some troubleshooting help.

Goal: assign fixed wan interfaces to specific INTRANET source networks
(e.g. from 10.11.0.0/16 -> wan2, from 10.12.0.0/16 ->wan3)

-Openwrt trunk v 38237
-network setup:
______MWAN3 Router_______
| |
dsl-gw1 (10.13.1.102/29) <---> \ wan2 (10.13.1.101/29) |
\ |
dsl-gw2 (10.13.1.202/29) <----->| wan3 (10.13.1.201/29) |
| |
dsl-gw3 (10.13.1.122/29) <----->| wan4 (10.13.1.121/29) | ________plain Router________
/ | | |
dsl-gw4 (10.13.1.222/29) <--->/ wan5 (10.13.1.221/29) | | 10.10.10.20/8 WAN | <----> INTRANET
| | | | (10.0.0.0/8)
| (10.10.20.2/24) LAN |<------>| LAN (10.10.20.1/24) |
|_________________________| |_________________________|

network config:

root@mwan3_router:~# cat /etc/config/network 

config interface 'loopback'
    option ifname 'lo'
    option proto 'static'
    option ipaddr '127.0.0.1'
    option netmask '255.0.0.0'

config interface 'lan'
    option ifname 'eth0.1'
    option proto 'static'
    option ipaddr '10.10.20.2'
    option netmask '255.255.255.0'
    option dns    '8.8.8.8'
    option metric '1'

config interface 'wan2'
    option ifname 'eth0.2'
    option proto 'static'
    option ipaddr '10.13.1.101'
    option netmask '255.255.255.248'
    option gateway '10.13.1.102'
    option metric '2'
    
config interface 'wan3'
    option ifname 'eth0.3'
    option proto 'static'
    option ipaddr '10.13.1.201'
    option netmask '255.255.255.248'
    option gateway '10.13.1.202'
    option metric '3'

config interface 'wan4'
    option ifname 'eth0.4'
    option proto 'static'
    option ipaddr '10.13.1.121'
    option netmask '255.255.255.248'
    option gateway '10.13.1.122'
    option metric '4'

config interface 'wan5'
    option ifname 'eth0.5'
    option proto 'static'
    option ipaddr '10.13.1.221'
    option netmask '255.255.255.248'
    option gateway '10.13.1.222'
    option metric '5'

config switch
    option name 'switch0'
    option reset '1'
    option enable_vlan '1'

config switch_vlan
    option device 'switch0'
    option vlan '1'
    option ports '0t 1'

config switch_vlan
    option device 'switch0'
    option vlan '2'
    option ports '0t 2'

config switch_vlan
    option device 'switch0'
    option vlan '3'
    option ports '0t 3'

config switch_vlan
    option device 'switch0'
    option vlan '4'
    option ports '0t 4'

config switch_vlan
    option device 'switch0'
    option vlan '5'
    option ports '0t 5'

config route 'intranet'
    option interface 'lan'
    option target '10.0.0.0'
    option netmask '255.0.0.0'
    option gateway '10.10.20.1'

mwan3 configuration:

root@mwan3 router:~# cat /etc/config/mwan3 

config interface 'wan2'
    list track_ip '8.8.8.8'
    option reliability '1'
    option count '1'
    option timeout '2'
    option interval '5'
    option down '3'
    option reroute '0'
    option up '5'
    option enabled '0'

config interface 'wan3'
    list track_ip '8.8.4.4'
    option reliability '1'
    option count '1'
    option timeout '2'
    option interval '5'
    option down '3'
    option up '5'
    option reroute '0'
    option enabled '0'

config interface 'wan4'
    list track_ip '8.8.8.8'
    option reliability '1'
    option count '1'
    option timeout '2'
    option interval '5'
    option down '3'
    option up '5'
    option reroute '0'
    option enabled '0'

config interface 'wan5'
    list track_ip '8.8.4.4'
    option reliability '1'
    option count '1'
    option timeout '2'
    option interval '5'
    option down '3'
    option up '5'
    option reroute '0'
    option enabled '0'

config member 'wan2_m1'
    option interface 'wan2'
    option metric '1'
    option weight '1'

config member 'wan3_m1'
    option interface 'wan3'
    option metric '1'
    option weight '1'

config member 'wan4_m1'
    option interface 'wan4'
    option metric '1'
    option weight '1'

config member 'wan5_m1'
    option interface 'wan5'
    option metric '1'
    option weight '1'

config policy 'wan2_only'
    list use_member 'wan2_m1'

config policy 'wan3_only'
    list use_member 'wan3_m1'

config policy 'wan4_only'
    list use_member 'wan4_m1'

config policy 'wan5_only'
    list use_member 'wan5_m1'

config rule 'to_dsl-gw1'
    option src_ip '10.0.0.0/8'
    option dest_ip '10.13.1.102'
    option proto 'all'
    option use_policy 'wan2_only'

config rule 'to_dsl-gw2'
    option src_ip '10.0.0.0/8'
    option dest_ip '10.13.1.202'
    option proto 'all'
    option use_policy 'wan3_only'

config rule 'to_dsl-gw3'
    option src_ip '10.0.0.0/8'
    option dest_ip '10.13.1.122'
    option proto 'all'
    option use_policy 'wan4_only'

config rule 'to_dsl-gw4'
    option src_ip '10.0.0.0/8'
    option dest_ip '10.13.1.222'
    option proto 'all'
    option use_policy 'wan5_only'

config rule 'from_intranet_1'
    option src_ip '10.11.0.0/16'
    option proto 'all'
    option use_policy 'wan2_only'

config rule 'from_intranet_2'
    option src_ip '10.12.0.0/16'
    option proto 'all'
    option use_policy 'wan3_only'

config rule 'from_intranet_3'
    option src_ip '10.13.0.0/16'
    option proto 'all'
    option use_policy 'wan4_only'

config rule 'from_intranet_rest'
    option proto 'all'
    option use_policy 'wan5_only'
    option src_ip '10.0.0.0/8'

What is the problem ?
-hosts from INTRANET do not use their assigned WAN interface, but rather got loadbalanced round robin accross all WANs
-as long as wman3 is active, any ssh session from INTRANET to any dsl-gwXY is very unresponsive, like 3 sec delay, 3sec ok, 3sec delay....

troubleshooting so far:

from mwan3 router each wanXY can be used to ping google.de
dmesg shows is susspiciouse entry: kmod: dependency not loaded nf_conntrack_amanda
traffic generated on the mwan3 router uses always wan2
routing tables (all 8) look sufficient

What should I check to find the problem of the unwanted loadbalancing ?

Thx Bluse

(Last edited by Bluse-Blue on 27 Sep 2013, 22:22)

Post #496

Adze

27 Sep 2013, 23:17

Hi Bluse,

I see two mistakes in your config:

1. You have to set the option enabled to '1' on each interface. (but that should probably be on purpose..)
2. You need a rule for the intranet, telling the router to use the default routing table for intranet. Place this on top of all your other rules.

config rule 'to_intranet'
    option dest_ip '10.0.0.0/8'
    option proto 'all'
    option use_policy 'default'

Also, you can leave out all the "to_dsl-gw*" rules. They have no function. If it still doesn't work, please post your "iptables -L -t mangle -v -n" output.

Thnx

(Last edited by Adze on 27 Sep 2013, 23:25)

Post #497

Bluse-Blue

28 Sep 2013, 08:53

Hi Adze,

Thx for helping out here. I have added the intranet rule on top of all others.
But still the gateway assignment does not work as specified and ssh session from intranet to the gateway nodes are unrrsponsive.

here is the output of: "iptables -L -t mangle -v -n"

root@Kihei_gw:~# iptables -L -t mangle -v -n
Chain PREROUTING (policy ACCEPT 25807 packets, 12M bytes)
 pkts bytes target     prot opt in     out     source               destination         
31510   14M mwan3_pre  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
26089   12M fwmark     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain INPUT (policy ACCEPT 499 packets, 51974 bytes)
 pkts bytes target     prot opt in     out     source               destination         
  646 67748 mwan3_post  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain FORWARD (policy ACCEPT 25302 packets, 12M bytes)
 pkts bytes target     prot opt in     out     source               destination         
25569   12M mssfix     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT 509 packets, 88142 bytes)
 pkts bytes target     prot opt in     out     source               destination         
  657  111K mwan3_pre  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain POSTROUTING (policy ACCEPT 25811 packets, 12M bytes)
 pkts bytes target     prot opt in     out     source               destination         
31458   14M mwan3_post  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain fwmark (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain mssfix (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain mwan3_default (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       0.0.0.0/0            224.0.0.0/3          mark match 0x0/0xff00 MARK xset 0x7f00/0xff00
    1    60 MARK       all  --  *      *       0.0.0.0/0            127.0.0.0/8          mark match 0x0/0xff00 MARK xset 0x7f00/0xff00
  153 20545 MARK       all  --  *      *       0.0.0.0/0            10.0.0.0/8           mark match 0x0/0xff00 MARK xset 0x7f00/0xff00
    0     0 MARK       all  --  *      *       0.0.0.0/0            10.10.20.0/24        mark match 0x0/0xff00 MARK xset 0x7f00/0xff00
    0     0 MARK       all  --  *      *       0.0.0.0/0            10.13.1.96/29        mark match 0x0/0xff00 MARK xset 0x7f00/0xff00
    0     0 MARK       all  --  *      *       0.0.0.0/0            10.13.1.120/29       mark match 0x0/0xff00 MARK xset 0x7f00/0xff00
    0     0 MARK       all  --  *      *       0.0.0.0/0            10.13.1.200/29       mark match 0x0/0xff00 MARK xset 0x7f00/0xff00
    0     0 MARK       all  --  *      *       0.0.0.0/0            10.13.1.216/29       mark match 0x0/0xff00 MARK xset 0x7f00/0xff00

Chain mwan3_interfaces (1 references)
 pkts bytes target     prot opt in     out     source               destination         
  407 92367 mwan3_wan5  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
  251 17388 mwan3_wan4  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
  237 16445 mwan3_wan3  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
  196 13851 mwan3_wan2  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain mwan3_post (2 references)
 pkts bytes target     prot opt in     out     source               destination         
 2596  585K MARK       all  --  *      eth0.2  0.0.0.0/0            0.0.0.0/0            mark match ! 0x7f00/0xff00 MARK xset 0x100/0xff00
  894  179K MARK       all  --  *      eth0.3  0.0.0.0/0            0.0.0.0/0            mark match ! 0x7f00/0xff00 MARK xset 0x200/0xff00
  554 88902 MARK       all  --  *      eth0.4  0.0.0.0/0            0.0.0.0/0            mark match ! 0x7f00/0xff00 MARK xset 0x300/0xff00
12268 2022K MARK       all  --  *      eth0.5  0.0.0.0/0            0.0.0.0/0            mark match ! 0x7f00/0xff00 MARK xset 0x400/0xff00
14592   11M MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x8000/0x8000 MARK and 0xffff7fff
32104   14M CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0            CONNMARK save mask 0xff00

Chain mwan3_pre (2 references)
 pkts bytes target     prot opt in     out     source               destination         
32167   14M CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0            CONNMARK restore mask 0xff00
 1959  667K MARK       all  --  eth0.2 *       0.0.0.0/0            0.0.0.0/0            MARK xset 0x8100/0xff00
  714  259K MARK       all  --  eth0.3 *       0.0.0.0/0            0.0.0.0/0            MARK xset 0x8200/0xff00
  348 76139 MARK       all  --  eth0.4 *       0.0.0.0/0            0.0.0.0/0            MARK xset 0x8300/0xff00
11678   10M MARK       all  --  eth0.5 *       0.0.0.0/0            0.0.0.0/0            MARK xset 0x8400/0xff00
  692  149K mwan3_default  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xff00
  467  118K mwan3_interfaces  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xff00
  266  104K mwan3_rules  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xff00

Chain mwan3_rules (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       0.0.0.0/0            10.0.0.0/8           mark match 0x0/0xff00 MARK xset 0x7f00/0xff00
    0     0 MARK       all  --  *      *       10.11.0.0/19         0.0.0.0/0            mark match 0x0/0xff00 MARK xset 0x1100/0xff00
    1    40 MARK       all  --  *      *       10.11.0.0/16         0.0.0.0/0            mark match 0x0/0xff00 MARK xset 0x1200/0xff00
    0     0 MARK       all  --  *      *       10.12.0.0/16         0.0.0.0/0            mark match 0x0/0xff00 MARK xset 0x1300/0xff00
    0     0 MARK       all  --  *      *       10.0.0.0/8           0.0.0.0/0            mark match 0x0/0xff00 MARK xset 0x1000/0xff00

Chain mwan3_wan2 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
  152 10199 MARK       all  --  *      *       10.13.1.101          0.0.0.0/0            MARK xset 0x100/0xff00

Chain mwan3_wan3 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
   16  1344 MARK       all  --  *      *       10.13.1.201          0.0.0.0/0            MARK xset 0x200/0xff00

Chain mwan3_wan4 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
   16  1344 MARK       all  --  *      *       10.13.1.121          0.0.0.0/0            MARK xset 0x300/0xff00

Chain mwan3_wan5 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
   17  1428 MARK       all  --  *      *       10.13.1.221          0.0.0.0/0            MARK xset 0x400/0xff00

Chain qos_Default (0 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0            CONNMARK restore mask 0xff
    0     0 qos_Default_ct  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xff
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x1/0xff length 400:65535 MARK and 0xffffff00
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x2/0xff length 800:65535 MARK and 0xffffff00
    0     0 MARK       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xff length 0:500 MARK xset 0x2/0xff
    0     0 MARK       icmp --  *      *       0.0.0.0/0            0.0.0.0/0            MARK xset 0x1/0xff
    0     0 MARK       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xff tcp spts:1024:65535 dpts:1024:65535 MARK xset 0x4/0xff
    0     0 MARK       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xff udp spts:1024:65535 dpts:1024:65535 MARK xset 0x4/0xff
    0     0 MARK       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            length 0:128 mark match ! 0x4/0xff tcp flags:0x3F/0x02 MARK xset 0x1/0xff
    0     0 MARK       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            length 0:128 mark match ! 0x4/0xff tcp flags:0x3F/0x10 MARK xset 0x1/0xff

Chain qos_Default_ct (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xff tcp multiport ports 22,53 MARK xset 0x1/0xff
    0     0 MARK       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xff udp multiport ports 22,53 MARK xset 0x1/0xff
    0     0 MARK       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xff tcp multiport ports 20,21,25,80,110,443,993,995 MARK xset 0x3/0xff
    0     0 MARK       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xff tcp multiport ports 5190 MARK xset 0x2/0xff
    0     0 MARK       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xff udp multiport ports 5190 MARK xset 0x2/0xff
    0     0 CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0            CONNMARK save mask 0xff

I am not sure where it got mixed up.

Greetings Bluse

Post #498

Bluse-Blue

28 Sep 2013, 09:00

the updated mwan3 config is (I disabled the interfaces as it does not work till now):

config interface 'wan2'
    list track_ip '8.8.8.8'
    option reliability '1'
    option count '1'
    option timeout '2'
    option interval '5'
    option down '3'
    option reroute '0'
    option up '5'
    option enabled '0'

config interface 'wan3'
    list track_ip '8.8.4.4'
    option reliability '1'
    option count '1'
    option timeout '2'
    option interval '5'
    option down '3'
    option up '5'
    option reroute '0'
    option enabled '0'

config interface 'wan4'
    list track_ip '8.8.8.8'
    option reliability '1'
    option count '1'
    option timeout '2'
    option interval '5'
    option down '3'
    option up '5'
    option reroute '0'
    option enabled '0'

config interface 'wan5'
    list track_ip '8.8.4.4'
    option reliability '1'
    option count '1'
    option timeout '2'
    option interval '5'
    option down '3'
    option up '5'
    option reroute '0'
    option enabled '0'

config member 'wan2_m1'
    option interface 'wan2'
    option metric '1'
    option weight '1'

config member 'wan3_m1'
    option interface 'wan3'
    option metric '1'
    option weight '1'

config member 'wan4_m1'
    option interface 'wan4'
    option metric '1'
    option weight '1'

config member 'wan5_m1'
    option interface 'wan5'
    option metric '1'
    option weight '1'

config policy 'wan2_only'
    list use_member 'wan2_m1'

config policy 'wan3_only'
    list use_member 'wan3_m1'

config policy 'wan4_only'
    list use_member 'wan4_m1'

config policy 'wan5_only'
    list use_member 'wan5_m1'

config policy 'all_loadbalanced'
    list use_member 'wan2_m1'
    list use_member 'wan3_m1'
    list use_member 'wan4_m1'
    list use_member 'wan5_m1'

#rule to Freifunk intranet
config rule 'to_intranet'
    option dest_ip '10.0.0.0/8'
    option proto 'all'
    option use_policy 'default'
            
config rule 'from_urli_1'
    option src_ip '10.11.0.0/19'
    option proto 'all'
    option use_policy 'wan3_only'

config rule 'from_urli_2'
    option src_ip '10.11.0.0/16'
    option proto 'all'
    option use_policy 'wan4_only'

config rule 'from_klett'
    option src_ip '10.12.0.0/16'
    option proto 'all'
    option use_policy 'wan5_only'

config rule 'from_sundi'
    option proto 'all'
    option use_policy 'wan2_only'
    option src_ip '10.0.0.0/8'

Greetings Bluse

(Last edited by Bluse-Blue on 28 Sep 2013, 09:01)

Post #499

Adze

28 Sep 2013, 13:20

Hi Bluse,

Your config looks fine and so does the output of iptables. I need some more info for troubleshouting. While mwan3 is enabled and while you are trying to connect to a gateway, could you post the output of:

ip rule
ip route list table main
ip route list table 1001
ip route list table 1002
ip route list table 1003
ip route list table 1004
ip route list table 1016
ip route list table 1017
ip route list table 1018
ip route list table 1019
ip route list table 1020
cat /proc/net/nf_conntrack

Thnx

Post #500

Bluse-Blue

28 Sep 2013, 21:26

Hi Adzel,

Another test confirmed the unresponsive ssh session from intranet to one of the gatway machines through the mwan3 router. A ping from one of the gateway machines to an intranet machine shows a lot of packet losses:

[root@galaxy ~]# ping 10.10.10.4
PING 10.10.10.4 (10.10.10.4): 56 data bytes
64 bytes from 10.10.10.4: icmp_seq=2 ttl=62 time=2.349 ms
64 bytes from 10.10.10.4: icmp_seq=10 ttl=62 time=2.016 ms
64 bytes from 10.10.10.4: icmp_seq=19 ttl=62 time=1.813 ms
64 bytes from 10.10.10.4: icmp_seq=21 ttl=62 time=2.071 ms

But back to your requested command output to get the throubleshooting a step forward.

ip rule

root@Kirchheilingen_gw:~# ip rule
0:    from all lookup local 
1001:    from all fwmark 0x100/0xff00 lookup 1001 
1002:    from all fwmark 0x200/0xff00 lookup 1002 
1003:    from all fwmark 0x300/0xff00 lookup 1003 
1004:    from all fwmark 0x400/0xff00 lookup 1004 
1016:    from all fwmark 0x1000/0xff00 lookup 1016 
1017:    from all fwmark 0x1100/0xff00 lookup 1017 
1018:    from all fwmark 0x1200/0xff00 lookup 1018 
1019:    from all fwmark 0x1300/0xff00 lookup 1019 
32766:    from all lookup main 
32767:    from all lookup default

ip route list table main

root@Kirchheilingen_gw:~# ip route list table main
default via 10.13.1.102 dev eth0.2  proto static  metric 2 
default via 10.13.1.202 dev eth0.3  proto static  metric 3 
default via 10.13.1.122 dev eth0.4  proto static  metric 4 
default via 10.13.1.222 dev eth0.5  proto static  metric 5 
10.0.0.0/8 via 10.10.20.1 dev eth0.1  proto static  metric 1 
10.10.20.0/24 dev eth0.1  proto static  scope link  metric 1 
10.13.1.96/29 dev eth0.2  proto static  scope link  metric 2 
10.13.1.120/29 dev eth0.4  proto static  scope link  metric 4 
10.13.1.200/29 dev eth0.3  proto static  scope link  metric 3 
10.13.1.216/29 dev eth0.5  proto static  scope link  metric 5

ip route list table XY

root@Kirchheilingen_gw:~# ip route list table 1001
default via 10.13.1.102 dev eth0.2 
root@Kirchheilingen_gw:~# ip route list table 1002
default via 10.13.1.202 dev eth0.3 
root@Kirchheilingen_gw:~# ip route list table 1003
default via 10.13.1.122 dev eth0.4 
root@Kirchheilingen_gw:~# ip route list table 1004
default via 10.13.1.222 dev eth0.5 
root@Kirchheilingen_gw:~# ip route list table 1016
default via 10.13.1.102 dev eth0.2  metric 1 
root@Kirchheilingen_gw:~# ip route list table 1017
default via 10.13.1.202 dev eth0.3  metric 1 
root@Kirchheilingen_gw:~# ip route list table 1018
default via 10.13.1.122 dev eth0.4  metric 1 
root@Kirchheilingen_gw:~# ip route list table 1019
default via 10.13.1.222 dev eth0.5  metric 1

Table 1020 is not present.

cat /proc/net/nf_conntrack

root@Kirchheilingen_gw:~# cat /proc/net/nf_conntrack
ipv4     2 udp      17 25 src=10.13.1.101 dst=8.8.8.8 sport=64317 dport=53 packets=1 bytes=66 [UNREPLIED] src=8.8.8.8 dst=10.13.1.101 sport=53 dport=64317 packets=0 bytes=0 mark=256 use=2
ipv4     2 udp      17 36 src=10.13.1.101 dst=8.8.8.8 sport=51039 dport=53 packets=1 bytes=69 [UNREPLIED] src=8.8.8.8 dst=10.13.1.101 sport=53 dport=51039 packets=0 bytes=0 mark=256 use=2
ipv4     2 udp      17 23 src=10.10.20.2 dst=10.11.10.33 sport=53 dport=40405 packets=1 bytes=265 [UNREPLIED] src=10.11.10.33 dst=10.10.20.2 sport=40405 dport=53 packets=0 bytes=0 mark=32512 use=2
ipv4     2 udp      17 31 src=10.10.20.2 dst=10.10.10.39 sport=53 dport=53688 packets=1 bytes=142 [UNREPLIED] src=10.10.10.39 dst=10.10.20.2 sport=53688 dport=53 packets=0 bytes=0 mark=32512 use=2
ipv4     2 udp      17 22 src=10.13.1.101 dst=8.8.8.8 sport=21961 dport=53 packets=1 bytes=62 [UNREPLIED] src=8.8.8.8 dst=10.13.1.101 sport=53 dport=21961 packets=0 bytes=0 mark=256 use=2
ipv4     2 icmp     1 25 src=10.13.1.201 dst=8.8.4.4 type=8 code=0 id=5369 packets=1 bytes=84 [UNREPLIED] src=8.8.4.4 dst=10.13.1.201 type=0 code=0 id=5369 packets=0 bytes=0 mark=512 use=2
ipv4     2 tcp      6 2772 ESTABLISHED src=10.10.4.5 dst=10.13.1.222 sport=45866 dport=444 packets=4 bytes=588 src=10.13.1.222 dst=10.10.4.5 sport=444 dport=45866 packets=4 bytes=785 [ASSURED] mark=0 use=2
ipv4     2 udp      17 36 src=10.10.20.2 dst=10.100.10.12 sport=53 dport=27646 packets=1 bytes=85 [UNREPLIED] src=10.100.10.12 dst=10.10.20.2 sport=27646 dport=53 packets=0 bytes=0 mark=32512 use=2
ipv4     2 icmp     1 14 src=10.13.1.121 dst=8.8.8.8 type=8 code=0 id=5351 packets=1 bytes=84 [UNREPLIED] src=8.8.8.8 dst=10.13.1.121 type=0 code=0 id=5351 packets=0 bytes=0 mark=768 use=2
ipv4     2 icmp     1 6 src=10.13.1.101 dst=8.8.8.8 type=8 code=0 id=5339 packets=1 bytes=84 [UNREPLIED] src=8.8.8.8 dst=10.13.1.101 type=0 code=0 id=5339 packets=0 bytes=0 mark=256 use=2
ipv4     2 tcp      6 2801 ESTABLISHED src=10.12.6.117 dst=80.239.194.134 sport=51426 dport=80 packets=1 bytes=52 src=80.239.194.134 dst=10.12.6.117 sport=80 dport=51426 packets=6 bytes=8952 [ASSURED] mark=1024 use=2
ipv4     2 udp      17 38 src=10.10.20.2 dst=10.100.10.12 sport=53 dport=48841 packets=1 bytes=78 [UNREPLIED] src=10.100.10.12 dst=10.10.20.2 sport=48841 dport=53 packets=0 bytes=0 mark=32512 use=2
ipv4     2 tcp      6 2823 ESTABLISHED src=10.13.1.222 dst=10.10.4.5 sport=444 dport=45934 packets=3 bytes=725 src=10.10.4.5 dst=10.13.1.222 sport=45934 dport=444 packets=2 bytes=578 [ASSURED] mark=1024 use=2
ipv4     2 icmp     1 21 src=10.13.1.101 dst=8.8.8.8 type=8 code=0 id=5363 packets=1 bytes=84 [UNREPLIED] src=8.8.8.8 dst=10.13.1.101 type=0 code=0 id=5363 packets=0 bytes=0 mark=256 use=2
ipv4     2 tcp      6 2823 ESTABLISHED src=212.162.25.34 dst=10.12.6.117 sport=80 dport=51415 packets=2 bytes=2984 src=10.12.6.117 dst=212.162.25.34 sport=51415 dport=80 packets=2 bytes=120 [ASSURED] mark=1024 use=2
ipv4     2 icmp     1 18 src=10.13.1.221 dst=8.8.4.4 type=8 code=0 id=5357 packets=1 bytes=84 [UNREPLIED] src=8.8.4.4 dst=10.13.1.221 type=0 code=0 id=5357 packets=0 bytes=0 mark=1024 use=2
ipv4     2 tcp      6 2762 ESTABLISHED src=10.10.2.241 dst=111.221.77.154 sport=50420 dport=80 packets=3 bytes=167 src=111.221.77.154 dst=10.10.2.241 sport=80 dport=50420 packets=2 bytes=92 [ASSURED] mark=0 use=2
ipv4     2 udp      17 22 src=10.10.20.2 dst=10.11.10.33 sport=53 dport=7835 packets=1 bytes=260 [UNREPLIED] src=10.11.10.33 dst=10.10.20.2 sport=7835 dport=53 packets=0 bytes=0 mark=32512 use=2
ipv4     2 icmp     1 1 src=10.13.1.101 dst=8.8.8.8 type=8 code=0 id=5331 packets=1 bytes=84 [UNREPLIED] src=8.8.8.8 dst=10.13.1.101 type=0 code=0 id=5331 packets=0 bytes=0 mark=256 use=2
ipv4     2 udp      17 31 src=10.13.1.101 dst=8.8.8.8 sport=41822 dport=53 packets=1 bytes=62 [UNREPLIED] src=8.8.8.8 dst=10.13.1.101 sport=53 dport=41822 packets=0 bytes=0 mark=256 use=2
ipv4     2 udp      17 36 src=10.13.1.101 dst=8.8.8.8 sport=10217 dport=53 packets=1 bytes=60 [UNREPLIED] src=8.8.8.8 dst=10.13.1.101 sport=53 dport=10217 packets=0 bytes=0 mark=256 use=2
ipv4     2 udp      17 16 src=10.13.1.101 dst=8.8.8.8 sport=32652 dport=53 packets=1 bytes=63 [UNREPLIED] src=8.8.8.8 dst=10.13.1.101 sport=53 dport=32652 packets=0 bytes=0 mark=256 use=2
ipv4     2 tcp      6 2762 ESTABLISHED src=10.11.21.140 dst=77.234.40.66 sport=52939 dport=80 packets=3 bytes=359 src=77.234.40.66 dst=10.11.21.140 sport=80 dport=52939 packets=2 bytes=88 [ASSURED] mark=0 use=2
ipv4     2 udp      17 29 src=10.13.1.101 dst=8.8.8.8 sport=43028 dport=53 packets=1 bytes=70 [UNREPLIED] src=8.8.8.8 dst=10.13.1.101 sport=53 dport=43028 packets=0 bytes=0 mark=256 use=2
ipv4     2 udp      17 30 src=10.10.20.2 dst=10.10.10.39 sport=53 dport=57959 packets=1 bytes=139 [UNREPLIED] src=10.10.10.39 dst=10.10.20.2 sport=57959 dport=53 packets=0 bytes=0 mark=32512 use=2
ipv4     2 udp      17 30 src=10.10.20.2 dst=10.10.10.39 sport=53 dport=13516 packets=1 bytes=75 [UNREPLIED] src=10.10.10.39 dst=10.10.20.2 sport=13516 dport=53 packets=0 bytes=0 mark=32512 use=2
ipv4     2 icmp     1 5 src=10.13.1.201 dst=8.8.4.4 type=8 code=0 id=5337 packets=1 bytes=84 [UNREPLIED] src=8.8.4.4 dst=10.13.1.201 type=0 code=0 id=5337 packets=0 bytes=0 mark=512 use=2
ipv4     2 udp      17 16 src=10.10.20.2 dst=10.11.10.53 sport=53 dport=2512 packets=1 bytes=120 [UNREPLIED] src=10.11.10.53 dst=10.10.20.2 sport=2512 dport=53 packets=0 bytes=0 mark=32512 use=2
ipv4     2 tcp      6 2762 ESTABLISHED src=10.10.21.116 dst=77.234.45.55 sport=49676 dport=80 packets=3 bytes=387 src=77.234.45.55 dst=10.10.21.116 sport=80 dport=49676 packets=2 bytes=92 [ASSURED] mark=0 use=2
ipv4     2 udp      17 56 src=10.13.1.101 dst=8.8.8.8 sport=34909 dport=53 packets=1 bytes=74 [UNREPLIED] src=8.8.8.8 dst=10.13.1.101 sport=53 dport=34909 packets=0 bytes=0 mark=256 use=2
ipv4     2 udp      17 38 src=10.13.1.101 dst=8.8.8.8 sport=20152 dport=53 packets=1 bytes=62 [UNREPLIED] src=8.8.8.8 dst=10.13.1.101 sport=53 dport=20152 packets=0 bytes=0 mark=256 use=2
ipv4     2 udp      17 57 src=10.10.20.2 dst=10.10.10.39 sport=53 dport=60532 packets=1 bytes=264 [UNREPLIED] src=10.10.10.39 dst=10.10.20.2 sport=60532 dport=53 packets=0 bytes=0 mark=32512 use=2
ipv4     2 tcp      6 2814 ESTABLISHED src=157.56.192.175 dst=10.10.2.215 sport=443 dport=50731 packets=1 bytes=1267 src=10.10.2.215 dst=157.56.192.175 sport=50731 dport=443 packets=1 bytes=52 mark=1024 use=2
ipv4     2 icmp     1 26 src=10.13.1.101 dst=8.8.8.8 type=8 code=0 id=5371 packets=1 bytes=84 [UNREPLIED] src=8.8.8.8 dst=10.13.1.101 type=0 code=0 id=5371 packets=0 bytes=0 mark=256 use=2
ipv4     2 icmp     1 11 src=10.13.1.101 dst=8.8.8.8 type=8 code=0 id=5347 packets=1 bytes=84 [UNREPLIED] src=8.8.8.8 dst=10.13.1.101 type=0 code=0 id=5347 packets=0 bytes=0 mark=256 use=2
ipv4     2 icmp     1 15 src=10.13.1.201 dst=8.8.4.4 type=8 code=0 id=5353 packets=1 bytes=84 [UNREPLIED] src=8.8.4.4 dst=10.13.1.201 type=0 code=0 id=5353 packets=0 bytes=0 mark=512 use=2
ipv4     2 icmp     1 20 src=10.13.1.201 dst=8.8.4.4 type=8 code=0 id=5361 packets=1 bytes=84 [UNREPLIED] src=8.8.4.4 dst=10.13.1.201 type=0 code=0 id=5361 packets=0 bytes=0 mark=512 use=2
ipv4     2 icmp     1 29 src=10.13.1.121 dst=8.8.8.8 type=8 code=0 id=5377 packets=1 bytes=84 [UNREPLIED] src=8.8.8.8 dst=10.13.1.121 type=0 code=0 id=5377 packets=0 bytes=0 mark=768 use=2
ipv4     2 icmp     1 13 src=10.13.1.221 dst=8.8.4.4 type=8 code=0 id=5349 packets=1 bytes=84 [UNREPLIED] src=8.8.4.4 dst=10.13.1.221 type=0 code=0 id=5349 packets=0 bytes=0 mark=1024 use=2
ipv4     2 tcp      6 2762 ESTABLISHED src=10.10.4.5 dst=10.13.1.222 sport=45863 dport=444 packets=3 bytes=536 src=10.13.1.222 dst=10.10.4.5 sport=444 dport=45863 packets=2 bytes=112 [ASSURED] mark=0 use=2
ipv4     2 icmp     1 28 src=10.13.1.221 dst=8.8.4.4 type=8 code=0 id=5375 packets=1 bytes=84 [UNREPLIED] src=8.8.4.4 dst=10.13.1.221 type=0 code=0 id=5375 packets=0 bytes=0 mark=1024 use=2
ipv4     2 udp      17 39 src=10.13.1.101 dst=8.8.8.8 sport=60692 dport=53 packets=1 bytes=63 [UNREPLIED] src=8.8.8.8 dst=10.13.1.101 sport=53 dport=60692 packets=0 bytes=0 mark=256 use=2
ipv4     2 tcp      6 2762 ESTABLISHED src=10.12.6.117 dst=213.199.179.153 sport=51397 dport=40002 packets=4 bytes=238 src=213.199.179.153 dst=10.12.6.117 sport=40002 dport=51397 packets=3 bytes=231 [ASSURED] mark=0 use=2
ipv4     2 udp      17 25 src=10.13.1.101 dst=142.54.181.202 sport=44806 dport=123 packets=1 bytes=76 [UNREPLIED] src=142.54.181.202 dst=10.13.1.101 sport=123 dport=44806 packets=0 bytes=0 mark=256 use=2
ipv4     2 icmp     1 0 src=10.13.1.201 dst=8.8.4.4 type=8 code=0 id=5329 packets=1 bytes=84 [UNREPLIED] src=8.8.4.4 dst=10.13.1.201 type=0 code=0 id=5329 packets=0 bytes=0 mark=512 use=2
ipv4     2 udp      17 36 src=10.10.20.2 dst=10.100.10.12 sport=53 dport=63413 packets=1 bytes=76 [UNREPLIED] src=10.100.10.12 dst=10.10.20.2 sport=63413 dport=53 packets=0 bytes=0 mark=32512 use=2
ipv4     2 tcp      6 2824 ESTABLISHED src=10.13.1.202 dst=10.10.4.5 sport=444 dport=59872 packets=1 bytes=64 [UNREPLIED] src=10.10.4.5 dst=10.13.1.202 sport=59872 dport=444 packets=0 bytes=0 mark=512 use=2
ipv4     2 udp      17 57 src=10.13.1.101 dst=8.8.8.8 sport=5066 dport=53 packets=1 bytes=62 [UNREPLIED] src=8.8.8.8 dst=10.13.1.101 sport=53 dport=5066 packets=0 bytes=0 mark=256 use=2
ipv4     2 tcp      6 2762 ESTABLISHED src=10.10.100.253 dst=91.190.216.63 sport=37418 dport=12350 packets=3 bytes=260 src=91.190.216.63 dst=10.10.100.253 sport=12350 dport=37418 packets=1 bytes=60 [ASSURED] mark=0 use=2
ipv4     2 icmp     1 3 src=10.13.1.221 dst=8.8.4.4 type=8 code=0 id=5333 packets=1 bytes=84 [UNREPLIED] src=8.8.4.4 dst=10.13.1.221 type=0 code=0 id=5333 packets=0 bytes=0 mark=1024 use=2
ipv4     2 udp      17 19 src=10.13.1.101 dst=199.7.177.206 sport=32845 dport=123 packets=1 bytes=76 [UNREPLIED] src=199.7.177.206 dst=10.13.1.101 sport=123 dport=32845 packets=0 bytes=0 mark=256 use=2
ipv4     2 udp      17 57 src=10.10.20.2 dst=10.10.10.39 sport=53 dport=17455 packets=1 bytes=109 [UNREPLIED] src=10.10.10.39 dst=10.10.20.2 sport=17455 dport=53 packets=0 bytes=0 mark=32512 use=2
ipv4     2 tcp      6 2762 ESTABLISHED src=10.10.100.253 dst=91.190.216.63 sport=34080 dport=80 packets=3 bytes=169 src=91.190.216.63 dst=10.10.100.253 sport=80 dport=34080 packets=2 bytes=112 [ASSURED] mark=0 use=2
ipv4     2 udp      17 25 src=10.13.1.101 dst=91.206.8.36 sport=54045 dport=123 packets=1 bytes=76 [UNREPLIED] src=91.206.8.36 dst=10.13.1.101 sport=123 dport=54045 packets=0 bytes=0 mark=256 use=2
ipv4     2 udp      17 23 src=10.13.1.101 dst=8.8.8.8 sport=46901 dport=53 packets=1 bytes=65 [UNREPLIED] src=8.8.8.8 dst=10.13.1.101 sport=53 dport=46901 packets=0 bytes=0 mark=256 use=2
ipv4     2 icmp     1 24 src=10.13.1.121 dst=8.8.8.8 type=8 code=0 id=5367 packets=1 bytes=84 [UNREPLIED] src=8.8.8.8 dst=10.13.1.121 type=0 code=0 id=5367 packets=0 bytes=0 mark=768 use=2
ipv4     2 udp      17 39 src=10.13.1.101 dst=8.8.8.8 sport=30279 dport=53 packets=1 bytes=57 [UNREPLIED] src=8.8.8.8 dst=10.13.1.101 sport=53 dport=30279 packets=0 bytes=0 mark=256 use=2
ipv4     2 udp      17 39 src=10.13.1.101 dst=8.8.8.8 sport=53864 dport=53 packets=1 bytes=60 [UNREPLIED] src=8.8.8.8 dst=10.13.1.101 sport=53 dport=53864 packets=0 bytes=0 mark=256 use=2
ipv4     2 udp      17 25 src=10.13.1.101 dst=8.8.8.8 sport=41098 dport=53 packets=1 bytes=71 [UNREPLIED] src=8.8.8.8 dst=10.13.1.101 sport=53 dport=41098 packets=0 bytes=0 mark=256 use=2
ipv4     2 icmp     1 4 src=10.13.1.121 dst=8.8.8.8 type=8 code=0 id=5335 packets=1 bytes=84 [UNREPLIED] src=8.8.8.8 dst=10.13.1.121 type=0 code=0 id=5335 packets=0 bytes=0 mark=768 use=2
ipv4     2 udp      17 24 src=10.13.1.101 dst=8.8.8.8 sport=12528 dport=53 packets=1 bytes=60 [UNREPLIED] src=8.8.8.8 dst=10.13.1.101 sport=53 dport=12528 packets=0 bytes=0 mark=256 use=2
ipv4     2 udp      17 30 src=10.10.20.2 dst=10.100.10.12 sport=53 dport=51679 packets=1 bytes=178 [UNREPLIED] src=10.100.10.12 dst=10.10.20.2 sport=51679 dport=53 packets=0 bytes=0 mark=32512 use=2
ipv4     2 udp      17 30 src=10.13.1.101 dst=8.8.8.8 sport=15003 dport=53 packets=1 bytes=66 [UNREPLIED] src=8.8.8.8 dst=10.13.1.101 sport=53 dport=15003 packets=0 bytes=0 mark=256 use=2
ipv4     2 tcp      6 2814 ESTABLISHED src=10.11.21.140 dst=46.33.72.59 sport=52940 dport=80 packets=4 bytes=160 [UNREPLIED] src=46.33.72.59 dst=10.11.21.140 sport=80 dport=52940 packets=0 bytes=0 mark=512 use=2
ipv4     2 tcp      6 2780 ESTABLISHED src=173.194.69.94 dst=10.10.39.177 sport=80 dport=46477 packets=2 bytes=2293 src=10.10.39.177 dst=173.194.69.94 sport=46477 dport=80 packets=1 bytes=64 [ASSURED] mark=1024 use=2
ipv4     2 icmp     1 9 src=10.13.1.121 dst=8.8.8.8 type=8 code=0 id=5343 packets=1 bytes=84 [UNREPLIED] src=8.8.8.8 dst=10.13.1.121 type=0 code=0 id=5343 packets=0 bytes=0 mark=768 use=2
ipv4     2 tcp      6 3599 ESTABLISHED src=10.10.20.2 dst=10.10.4.229 sport=22 dport=59967 packets=282 bytes=68674 [UNREPLIED] src=10.10.4.229 dst=10.10.20.2 sport=59967 dport=22 packets=0 bytes=0 mark=32512 use=2
ipv4     2 icmp     1 23 src=10.13.1.221 dst=8.8.4.4 type=8 code=0 id=5365 packets=1 bytes=84 [UNREPLIED] src=8.8.4.4 dst=10.13.1.221 type=0 code=0 id=5365 packets=0 bytes=0 mark=1024 use=2
ipv4     2 tcp      6 2801 ESTABLISHED src=10.12.6.117 dst=77.67.60.154 sport=51423 dport=80 packets=5 bytes=212 src=77.67.60.154 dst=10.12.6.117 sport=80 dport=51423 packets=5 bytes=7460 [ASSURED] mark=1024 use=2
ipv4     2 icmp     1 16 src=10.13.1.101 dst=8.8.8.8 type=8 code=0 id=5355 packets=1 bytes=84 [UNREPLIED] src=8.8.8.8 dst=10.13.1.101 type=0 code=0 id=5355 packets=0 bytes=0 mark=256 use=2
ipv4     2 tcp      6 2803 ESTABLISHED src=10.12.6.117 dst=213.199.179.153 sport=51429 dport=40002 packets=1 bytes=107 [UNREPLIED] src=213.199.179.153 dst=10.12.6.117 sport=40002 dport=51429 packets=0 bytes=0 mark=1024 use=2
ipv4     2 udp      17 25 src=10.10.20.2 dst=10.10.10.39 sport=53 dport=35191 packets=1 bytes=117 [UNREPLIED] src=10.10.10.39 dst=10.10.20.2 sport=35191 dport=53 packets=0 bytes=0 mark=32512 use=2
ipv4     2 icmp     1 10 src=10.13.1.201 dst=8.8.4.4 type=8 code=0 id=5345 packets=1 bytes=84 [UNREPLIED] src=8.8.4.4 dst=10.13.1.201 type=0 code=0 id=5345 packets=0 bytes=0 mark=512 use=2
ipv4     2 udp      17 25 src=10.10.20.2 dst=10.10.10.39 sport=53 dport=20355 packets=1 bytes=236 [UNREPLIED] src=10.10.10.39 dst=10.10.20.2 sport=20355 dport=53 packets=0 bytes=0 mark=32512 use=2
ipv4     2 udp      17 39 src=10.10.20.2 dst=10.100.10.12 sport=53 dport=38356 packets=1 bytes=90 [UNREPLIED] src=10.100.10.12 dst=10.10.20.2 sport=38356 dport=53 packets=0 bytes=0 mark=32512 use=2
ipv4     2 udp      17 40 src=10.10.20.2 dst=10.100.10.12 sport=53 dport=9135 packets=1 bytes=157 [UNREPLIED] src=10.100.10.12 dst=10.10.20.2 sport=9135 dport=53 packets=0 bytes=0 mark=32512 use=2
ipv4     2 udp      17 30 src=10.13.1.101 dst=8.8.8.8 sport=20503 dport=53 packets=1 bytes=59 [UNREPLIED] src=8.8.8.8 dst=10.13.1.101 sport=53 dport=20503 packets=0 bytes=0 mark=256 use=2
ipv4     2 udp      17 39 src=10.10.20.2 dst=10.100.10.12 sport=53 dport=14443 packets=1 bytes=151 [UNREPLIED] src=10.100.10.12 dst=10.10.20.2 sport=14443 dport=53 packets=0 bytes=0 mark=32512 use=2
ipv4     2 udp      17 24 src=10.10.20.2 dst=10.11.10.24 sport=53 dport=10523 packets=1 bytes=76 [UNREPLIED] src=10.11.10.24 dst=10.10.20.2 sport=10523 dport=53 packets=0 bytes=0 mark=32512 use=2
ipv4     2 icmp     1 19 src=10.13.1.121 dst=8.8.8.8 type=8 code=0 id=5359 packets=1 bytes=84 [UNREPLIED] src=8.8.8.8 dst=10.13.1.121 type=0 code=0 id=5359 packets=0 bytes=0 mark=768 use=2
ipv4     2 udp      17 29 src=10.10.20.2 dst=10.10.10.39 sport=53 dport=58034 packets=1 bytes=86 [UNREPLIED] src=10.10.10.39 dst=10.10.20.2 sport=58034 dport=53 packets=0 bytes=0 mark=32512 use=2
ipv4     2 icmp     1 8 src=10.13.1.221 dst=8.8.4.4 type=8 code=0 id=5341 packets=1 bytes=84 [UNREPLIED] src=8.8.4.4 dst=10.13.1.221 type=0 code=0 id=5341 packets=0 bytes=0 mark=1024 use=2
ipv4     2 tcp      6 2824 ESTABLISHED src=10.13.1.122 dst=10.10.4.5 sport=444 dport=42459 packets=1 bytes=64 [UNREPLIED] src=10.10.4.5 dst=10.13.1.122 sport=42459 dport=444 packets=0 bytes=0 mark=768 use=2
ipv4     2 udp      17 30 src=10.13.1.101 dst=8.8.8.8 sport=31685 dport=53 packets=1 bytes=59 [UNREPLIED] src=8.8.8.8 dst=10.13.1.101 sport=53 dport=31685 packets=0 bytes=0 mark=256 use=2
ipv4     2 tcp      6 2795 ESTABLISHED src=10.10.4.5 dst=10.13.1.202 sport=59861 dport=444 packets=3 bytes=587 src=10.13.1.202 dst=10.10.4.5 sport=444 dport=59861 packets=3 bytes=301 [ASSURED] mark=512 use=2

Greetings Bluse

Sorry, posts 501 to 500 are missing from our archive.

Page 20 of 65