Error installing mwan3 and later no active user rules

Installing and Using OpenWrt Network and Wireless Configuration
1

Hi everyone,
I am new to this so please bear with me. I also searched of the best of my ability and could not find an answer so here is my question.

While installing mwan3, I got the following errors:

mwan3 error: ipv4: fib table does not exist. dump terminated 
error: ipv6: fib table does not exist. dump terminated

I have created the following test rule:

config rule 'test_rule'
        option dest_ip 'www.google.com'
        option proto 'all'
        option sticky '0'
        option use_policy 'balanced'

This still fails when the family is ipv4. If I remove the family, ignored it and carried on to have it appear an IPv6 user rule but not as an IPv4 one.
Apparently, from the little analysis I could do, none of the rules got set because of the above error (FIB table does not exist).

Please help.

2

I was going to post this as another question but then I thought they might be related.
My setup is a router with two WAN interfaces and an authoritative DHCP (using dnsmasq) serving my network on the LAN interface and WLAN bridged together. Before installing mwan3, I can ping -I eth1 google.com and ping -I eth2 google.com successfully and DHCP assigns addresses correctly in my LAN subnet 192.168.y.0/24 . Once I install mwan3, DHCP starts serving wrong addresses to some of my devices. The wrong addresses are in the subnet 192.168.0.0/24 or addresses like 169.254.151.52 with a mask 255.255.0.0
Note that some devices still get their correct static IP addresses from the DHCP server.

Could this be also related to the FIB problem above or is this a separate issue?

3

Please run the following commands (copy-paste the whole block) and paste the output here, using the "Preformatted text </> " button:
grafik
Remember to redact passwords, MAC addresses and any public IP addresses you may have

ubus call system board; \
uci export network; uci export dhcp; uci export firewall; \
head -n -0 /etc/firewall.user; uci export mwan3; \
ip -4 addr ; ip -4 ro li tab all ; ip -4 ru; \
ip -6 addr ; ip -6 ro li tab all ; ip -6 ru; \
ls -l  /etc/resolv.* /tmp/resolv.* /tmp/resolv.*/* ; head -n -0 /etc/resolv.* /tmp/resolv.* /tmp/resolv.*/*
4

Thanks for your response.

Here is the result from ubus call system board

{
        "kernel": "5.4.137",
        "hostname": "OpenWrt",
        "system": "ARMv8 Processor rev 3",
        "model": "Raspberry Pi 4 Model B Rev 1.4",
        "board_name": "raspberrypi,4-model-b",
        "release": {
                "distribution": "OpenWrt",
                "version": "21.02.0-rc4",
                "revision": "r16256-2d5ee43dc6",
                "target": "bcm27xx/bcm2711",
                "description": "OpenWrt 21.02.0-rc4 r16256-2d5ee43dc6"
        }
}

Network

package network

config interface 'loopback'
        option device 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ipv6 '0'

config device
        option name 'eth0'
        option ipv6 '0'

config device
        option name 'wlan0'
        option ipv6 '0'

config device
        option type 'bridge'
        option name 'br-lan'
        option ipv6 '0'
        list ports 'eth0'
        list ports 'wlan0'

config device
        option name 'eth1'
        option ipv6 '0'

config device
        option name 'eth2'
        option ipv6 '0'

config interface 'wlan0'
        option device 'wlan0'
        option proto 'static'
        option ipaddr '192.168.4.1'
        option netmask '255.255.255.0'
        option gateway '192.168.1.1'
        option broadcast '192.168.4.255'
        option delegate '0'
        list dns '208.67.222.123'
        list dns '208.67.220.123'

config interface 'lan'
        option device 'br-lan'
        option proto 'static'
        option ipaddr '192.168.4.1'
        option netmask '255.255.255.0'
        option gateway '192.168.1.1'
        option broadcast '192.168.4.255'
        option delegate '0'
        list dns '208.67.222.123'
        list dns '208.67.220.123'

config interface 'wan'
        option device 'eth1'
        option proto 'static'
        option ipaddr '192.168.1.4'
        option netmask '255.255.255.0'
        option gateway '192.168.1.1'
        option broadcast '192.168.1.255'
        option delegate '0'
        option force_link '1'
        option peerdns '0'
        list dns '208.67.222.123'
        list dns '208.67.220.123'
        option metric '30'

config interface 'vwan'
        option device 'eth2'
        option proto 'static'
        option ipaddr '192.168.6.11'
        option netmask '255.255.255.0'
        option gateway '192.168.6.2'
        option broadcast '192.168.6.255'
        option delegate '0'
        option force_link '1'
        option peerdns '0'
        list dns '208.67.222.123'
        list dns '208.67.220.123'
        option metric '20'

DHCP

config dnsmasq
        option localise_queries '1'
        option rebind_protection '1'
        option rebind_localhost '1'
        option local '/lan/'
        option domain 'lan'
        option expandhosts '1'
        option authoritative '1'
        option readethers '1'
        option leasefile '/tmp/dhcp.leases'
        option resolvfile '/tmp/resolv.conf.d/resolv.conf.auto'
        option localservice '1'
        option ednspacket_max '1232'
        option domainneeded '1'
        list addnhosts '/etc/safe-search/enabled'

config dhcp 'lan'
        option interface 'lan'
        option dhcpv6 'server'
        option ra 'server'
        list ra_flags 'managed-config'
        list ra_flags 'other-config'
        option ra_management '1'
        option start '230'
        option limit '250'
        option leasetime '15m'

config dhcp 'wan'
        option interface 'wan'
        option ignore '1'
        option start '100'
        option limit '150'
        option leasetime '12h'
        list ra_flags 'none'

config odhcpd 'odhcpd'
        option maindhcp '0'
        option leasefile '/tmp/hosts/odhcpd'
        option leasetrigger '/usr/sbin/odhcpd-update'
        option loglevel '4'

config dhcp 'wlan0'
        option interface 'wlan0'
        option start '250'
        option limit '250'
        option leasetime '15m'
        list ra_flags 'none'
        option dynamicdhcp '0'

config host
        option dns '1'
        option name 'AccessPoint1'
        list mac 'xx:xx:xx:xx:xx:xx'
        list mac 'yy:yy:yy:yy:yy:yy'
        option ip '192.168.4.30'
        option leasetime '15m'

It has many more devices getting static addresses on the same LAN subnet but that's about it.

Firewall

config defaults
        option synflood_protect '1'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'REJECT'

config zone 'lan'
        option name 'lan'
        list network 'lan'
        list network 'wlan0'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'
        option ip6assign '60'

config zone 'wan'
        option name 'wan'
        list network 'wan'
        list network 'vwan'
        option input 'REJECT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option masq '1'
        option mtu_fix '1'

config forwarding
        option src 'lan'
        option dest 'wan'

config rule
        option name 'Allow-DHCP-Renew'
        option src 'wan'
        option proto 'udp'
        option dest_port '68'
        option target 'ACCEPT'
        option family 'ipv4'

config rule
        option name 'Allow-Ping'
        option src 'wan'
        option proto 'icmp'
        option icmp_type 'echo-request'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-IGMP'
        option src 'wan'
        option proto 'igmp'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-DHCPv6'
        option src 'wan'
        option proto 'udp'
        option src_ip 'fc00::/6'
        option dest_ip 'fc00::/6'
        option dest_port '546'
        option family 'ipv6'
        option target 'ACCEPT'
        option enabled '0'

config rule
        option name 'Allow-MLD'
        option src 'wan'
        option proto 'icmp'
        option src_ip 'fe80::/10'
        list icmp_type '130/0'
        list icmp_type '131/0'
        list icmp_type '132/0'
        list icmp_type '143/0'
        option family 'ipv6'
        option target 'ACCEPT'
        option enabled '0'

config rule
        option name 'Allow-ICMPv6-Input'
        option src 'wan'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        list icmp_type 'router-solicitation'
        list icmp_type 'neighbour-solicitation'
        list icmp_type 'router-advertisement'
        list icmp_type 'neighbour-advertisement'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'
        option enabled '0'

config rule
        option name 'Allow-ICMPv6-Forward'
        option src 'wan'
        option dest '*'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'
        option enabled '0'

config rule
        option name 'Allow-IPSec-ESP'
        option src 'wan'
        option dest 'lan'
        option proto 'esp'
        option target 'ACCEPT'

config rule
        option name 'Allow-ISAKMP'
        option src 'wan'
        option dest 'lan'
        option dest_port '500'
        option proto 'udp'
        option target 'ACCEPT'

config rule
        option name 'Support-UDP-Traceroute'
        option src 'wan'
        option dest_port '33434:33689'
        option proto 'udp'
        option family 'ipv4'
        option target 'REJECT'
        option enabled 'false'

config include
        option path '/etc/firewall.user'

config rule 'samba_nsds'
        option name 'Allow-Samba/NS/DS'
        option src 'lan'
        option dest_port '137-138'
        option proto 'udp'
        option target 'ACCEPT'

config rule 'samba_ss'
        option name 'Allow-Samba/SS'
        option src 'lan'
        option dest_port '139'
        option proto 'tcp'
        option target 'ACCEPT'

config rule 'samba_smb'
        option name 'Allow-Samba/SMB'
        option src 'lan'
        option dest_port '445'
        option proto 'tcp'
        option target 'ACCEPT'

/etc/firewall.user seems empty

# This file is interpreted as shell script.
# Put your custom iptables rules here, they will
# be executed with each firewall (re-)start.

# Internal uci firewall chains are flushed and recreated on reload, so
# put custom rules into the root chains e.g. INPUT or FORWARD or into the
# special user chains, e.g. input_wan_rule or postrouting_lan_rule.

mwan3 is not installed now because it messes up my dhcp and my family complains that some devices are unable to connect to the network but when it is installed it has the following in the conf file

config globals 'globals'
        option local_source 'none'
        option mmx_mask '0x3F00'
        list rt_table_lookup '220'
        option logging '1'
        option loglevel 'debug'

config interface 'wan'
        option enabled '1'
        option family 'ipv4'
        option initial_state 'online'
        option track_method 'ping'
        option reliability '1'
        option size '56'
        option max_ttl '60'
        option check_quality '0'
        option failure_interval '5'
        option interval '5'
        option count '1'
        option timeout '2'
        option recovery_interval '5'
        option down '5'
        option up '2'

config interface 'vwan'
        option enabled '1'
        option family 'ipv4'
        option initial_state 'online'
        option track_method 'ping'
        option size '56'
        option max_ttl '60'
        option check_quality '0'
        option failure_interval '5'
        option down '5'
        option interval '5'
        option reliability '1'
        option count '1'
        option timeout '2'
        option recovery_interval '5'
        option up '2'

config member 'wan_m1_w3'
        option interface 'wan'
        option metric '1'
        option weight '3'

config member 'wan_m2_w3'
        option interface 'wan'
        option metric '2'
        option weight '3'

config member 'vwan_m1_w2'
        option interface 'vwan'
        option metric '1'
        option weight '2'

config member 'vwan_m2_w2'
        option interface 'vwan'
        option metric '2'
        option weight '2'

config policy 'wan_only'
        list use_member 'wan_m1_w3'

config policy 'vwan_only'
        list use_member 'vwan_m1_w2'

config policy 'balanced'
        list use_member 'wan_m1_w3'
        list use_member 'vwan_m1_w2'

config policy 'wan_vwan'
        list use_member 'wan_m1_w3'
        list use_member 'vwan_m2_w2'

config policy 'vwan_wan'
        list use_member 'wan_m2_w3'
        list use_member 'vwan_m1_w2'

config rule 'test_rule'
#       option family 'ipv4'
        option dest_ip 'www.google.com'
        option proto 'all'
        option sticky '0'
        option use_policy 'balanced'

config rule 'vwan users'
#       option family 'ipv4'
        option proto 'all'
        option use_policy 'vwan_only'
        option src_ip '192.168.4.86 192.168.4.87 192.168.4.88'
        option dest_ip '0.0.0.0/0'
        option sticky '1'
        option timeout '600000'
        option logging '1'

config rule 'default_rule_v4'
        option dest_ip '0.0.0.0/0'
#       option family 'ipv4'
        option proto 'all'
        option sticky '0'
        option use_policy 'wan_only'

ip 4 info

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    inet 192.168.1.4/24 brd 192.168.1.255 scope global eth1
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    inet 192.168.6.11/24 brd 192.168.6.255 scope global eth2
       valid_lft forever preferred_lft forever
5: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br-lan state UP group default qlen 1000
    inet 192.168.4.1/24 brd 192.168.4.255 scope global wlan0
       valid_lft forever preferred_lft forever
6: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 192.168.4.1/24 brd 192.168.4.255 scope global br-lan
       valid_lft forever preferred_lft forever
default via 192.168.6.2 dev eth2 proto static metric 20
default via 192.168.1.1 dev eth1 proto static metric 30
192.168.1.0/24 dev eth1 proto static scope link metric 30
192.168.4.0/24 dev br-lan proto kernel scope link src 192.168.4.1
192.168.4.0/24 dev wlan0 proto kernel scope link src 192.168.4.1
192.168.6.0/24 dev eth2 proto static scope link metric 20
broadcast 127.0.0.0 dev lo table local proto kernel scope link src 127.0.0.1
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1
broadcast 192.168.1.0 dev eth1 table local proto kernel scope link src 192.168.1.4
local 192.168.1.4 dev eth1 table local proto kernel scope host src 192.168.1.4
broadcast 192.168.1.255 dev eth1 table local proto kernel scope link src 192.168.1.4
broadcast 192.168.4.0 dev br-lan table local proto kernel scope link src 192.168.4.1
broadcast 192.168.4.0 dev wlan0 table local proto kernel scope link src 192.168.4.1
local 192.168.4.1 dev br-lan table local proto kernel scope host src 192.168.4.1
local 192.168.4.1 dev wlan0 table local proto kernel scope host src 192.168.4.1
broadcast 192.168.4.255 dev br-lan table local proto kernel scope link src 192.168.4.1
broadcast 192.168.4.255 dev wlan0 table local proto kernel scope link src 192.168.4.1
broadcast 192.168.6.0 dev eth2 table local proto kernel scope link src 192.168.6.11
local 192.168.6.11 dev eth2 table local proto kernel scope host src 192.168.6.11
broadcast 192.168.6.255 dev eth2 table local proto kernel scope link src 192.168.6.11
0:      from all lookup local
32766:  from all lookup main
32767:  from all lookup default

ip6 info

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 state UNKNOWN qlen 1000
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
5: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
    inet6 fe80::dea6:32ff:feb4:4d72/64 scope link
       valid_lft forever preferred_lft forever
fe80::/64 dev wlan0 proto kernel metric 256 pref medium
local ::1 dev lo table local proto kernel metric 0 pref medium
anycast fe80:: dev wlan0 table local proto kernel metric 0 pref medium
local fe80::dea6:32ff:feb4:4d72 dev wlan0 table local proto kernel metric 0 pref medium
multicast ff00::/8 dev wlan0 table local proto kernel metric 256 pref medium
0:      from all lookup local
32766:  from all lookup main
4200000001:     from all iif lo failed_policy
4200000003:     from all iif eth1 failed_policy
4200000004:     from all iif eth2 failed_policy
4200000005:     from all iif wlan0 failed_policy
4200000006:     from all iif br-lan failed_policy

Resolv info

lrwxrwxrwx    1 root     root            16 Jul 31 19:21 /etc/resolv.conf -> /tmp/resolv.conf
-rw-r--r--    1 root     root            47 Sep 17 05:29 /tmp/resolv.conf
-rw-r--r--    1 root     root           275 Sep 17 05:29 /tmp/resolv.conf.d/resolv.conf.auto

/tmp/resolv.conf.d:
-rw-r--r--    1 root     root           275 Sep 17 05:29 resolv.conf.auto
==> /etc/resolv.conf <==
search lan
nameserver 127.0.0.1
nameserver ::1

==> /tmp/resolv.conf <==
search lan
nameserver 127.0.0.1
nameserver ::1

==> /tmp/resolv.conf.d <==
head: /tmp/resolv.conf.d: I/O error

==> /tmp/resolv.conf.d/resolv.conf.auto <==
# Interface lan
nameserver 208.67.222.123
nameserver 208.67.220.123
# Interface wlan0
nameserver 208.67.222.123
nameserver 208.67.220.123
# Interface wanb
nameserver 208.67.222.123
nameserver 208.67.220.123
# Interface wan
nameserver 208.67.222.123
nameserver 208.67.220.123

Looking forward for your feedback.
Thanks.

5

Some mistakes I have spotted:

  1. IP conflict in lan and wlan0.
  2. Invalid gateway in wlan0 and lan.
  3. In dhcp-lan, the option limit '250' is the size of the pool, not the last IP. Change it to 21.
  4. The same in wlan0, change it to 1.
  5. ip6assign should not be under lan zone, rather than under lan interface.
  6. mwan3 looks correct.
  7. Keep the custom nameservers in wan and vwan only.
2 Likes
6

Thank you. Sounds like all good comments. I will implement them as soon as I go home tonight and let you know if they made a difference.

Do you know what the FIB error means and how to fix it?

mwan3 error: ipv4: fib table does not exist. dump terminated
error: ipv6: fib table does not exist. dump terminated
7

Also, DHCP is working perfectly as it is now. When I add mwan3 sometimes it does not affect DHCP for a whole day and sometimes it immediately starts messing up the addresses offered by DHCP (or may be blocking them since I get 192.168.0.x which is a subnet I am not using and sometime I get 169.254.x.x which sort of implies my machines does not see the DHCP server and thinks it does not exist). Can mwan3 has such an effect? Is it creating firewall rules that could be causing this?

8

Hi Trendy,

I tried out your suggestions as follows:
#1. Fixed wlan0 IP address
#2. The OpenWRT is routing from lan and wlan0 to wan and wanb. Wan traffic goes out through eth0 which is on subnet 192.168.1.0/24 and where the gateway 192.168.1.1 is. The route table in OpenWRT takes care of this. Does this make sense?
#3 and #4. ok. Fixed.
#5. Do you mean move it from firewall conf file to network conf file?
#7. Again, here you are referring to interfaces in network conf file, correct? What happens if I have them on lan interface? This is working so I just want to understand the side effect.

Regarding, my original mwan3 problems:
1. With the above modifications and another thing I did differently this time during the installation of mwan3 package which is allowing it to overwrite files from other packages -> The fib table does not exist issue during installation is solved, not sure due to which of the above.
2. In this trial, I did not have my mwan3 in /etc/config and so the package installed its default file and I just noticed that the example rules in the file get listed as active user rules.
So, I took that file and started modifying it section by section, restart mwan3 and get it to report active user rules until I found out the issue that causes the rules not to be reported by mwan3 which was that in one of the rules the IP address is written as a range. It seems this threw it off and I have to have a rule for each IP address.

Now, things looked good until I did a reboot. At which time, I lost my connection. For some reason, my OpenWRT is not able to route to either of the two boxes on the two WAN interfaces. I get no response when I ping them both. I also noticed that the LEDs on at least one of the WAN interfaces are off (ethernet cable going between my RPi4 running OpenWRT and another RPi4 running commercial VPN client on Raspbian OS). So, using the mwan3 diagnostic tools, I keep pinging this WAN interface and using mwan3 hotplug ifup repeatedly until the LEDs on the ethernet dongle come up and it resumes the internet connection. Any idea what is happening here? and how to get it to come up more reliably?

Note that when I remove the mwan3 package, my setup boots smoothly with no issues and establishes connection on both interfaces and I can do ping -I eth1 google.com and ping -I eth2 google.com successfully.

Thanks again.

9

Forgot to mention that when I reboot, mwan3 rules does not report any active user rules.
They show back after I do

$ mwan3 stop
$ mwan3 restart
Command failed: Not found
$ mwan3 restart
$ mwan3 rules

Right now, I am pinging and hotplugging ifup but not getting it up. Even with the ethernet LEDs flashing, can't ping beyond the WAN interfaces of the OpenWRT RPi4.

I did a reboot and the connection is back but my rules are not active!!! This to me indicates that somehow mwan3 causes the WAN interfaces not to be pingable.

10

I just noticed the following in OpenWRT system log, may be it helps explain this latest issue

Mon Sep 20 07:11:23 2021 daemon.notice netifd: Network device 'eth0' link is up
Mon Sep 20 07:11:23 2021 daemon.notice netifd: Network device 'lo' link is up
Mon Sep 20 07:11:23 2021 daemon.notice netifd: Interface 'loopback' has link connectivity
Mon Sep 20 07:11:23 2021 daemon.notice netifd: Network device 'eth1' link is up
Mon Sep 20 07:11:23 2021 daemon.notice netifd: Interface 'wan' has link connectivity
Mon Sep 20 07:11:23 2021 kern.info kernel: [   15.143672] r8152 1-1.2:1.0 eth1: carrier on
Mon Sep 20 07:11:23 2021 daemon.notice netifd: radio0 (1942): command failed: No error information (-524)
Mon Sep 20 07:11:23 2021 daemon.notice netifd: radio0 (1942): command failed: I/O error (-5)
Mon Sep 20 07:11:23 2021 user.notice mwan3-hotplug[1929]: mwan3 hotplug on lan not called because interface disabled
Mon Sep 20 07:11:23 2021 kern.info kernel: [   15.288215] device br-lan entered promiscuous mode
Mon Sep 20 07:11:23 2021 user.notice firewall: Reloading firewall due to ifup of lan (br-lan)
Mon Sep 20 07:11:23 2021 user.notice mwan3-hotplug[2615]: mwan3 hotplug on loopback not called because interface disabled
Mon Sep 20 07:11:24 2021 user.notice mwan3-hotplug[3048]: Execute ifup event on interface vwan (eth2)
Mon Sep 20 07:11:24 2021 user.warn mwan3-hotplug[3048]: failed to add 192.168.6.0/24 dev eth2 proto static scope link metric 20 to table 2
Mon Sep 20 07:11:24 2021 user.warn mwan3-hotplug[3048]: failed to add default via 192.168.6.2 dev eth2 proto static metric 20 to table 2
Mon Sep 20 07:11:24 2021 daemon.notice hostapd: Configuration file: /var/run/hostapd-phy0.conf (phy wlan0) --> new PHY
Mon Sep 20 07:11:24 2021 daemon.notice hostapd: ACS: Automatic channel selection started, this may take a bit
Mon Sep 20 07:11:24 2021 daemon.notice hostapd: wlan0: interface state UNINITIALIZED->ACS
Mon Sep 20 07:11:24 2021 daemon.notice hostapd: wlan0: ACS-STARTED
Mon Sep 20 07:11:24 2021 user.notice firewall: Reloading firewall due to ifup of vwan (eth2)
Mon Sep 20 07:11:24 2021 user.notice mwan3-hotplug[4204]: Execute ifup event on interface wan (eth1)
Mon Sep 20 07:11:25 2021 user.warn mwan3-hotplug[4204]: failed to add 192.168.1.0/24 dev eth1 proto static scope link metric 30 to table 1
Mon Sep 20 07:11:25 2021 user.warn mwan3-hotplug[4204]: failed to add default via 192.168.1.1 dev eth1 proto static metric 30 to table 1
:
:
Mon Sep 20 07:16:47 2021 user.notice mwan3-hotplug[8690]: Execute ifup event on interface wan (eth1)
Mon Sep 20 07:16:47 2021 user.notice mwan3-hotplug[8698]: Execute ifup event on interface vwan (eth2)
Mon Sep 20 07:16:48 2021 daemon.err dnsmasq[5679]: failed to send packet: Network unreachable
Mon Sep 20 07:16:48 2021 user.warn mwan3rtmon[9137]: failed to add 192.168.4.0/24 dev wlan0 proto kernel scope link src 192.168.4.2 to table 1 - error: RTNETLINK answers: File exists
Mon Sep 20 07:16:48 2021 user.warn mwan3rtmon[9137]: failed to add 192.168.4.0/24 dev wlan0 proto kernel scope link src 192.168.4.2 to table 2 - error: RTNETLINK answers: File exists
Mon Sep 20 07:16:50 2021 user.notice mwan3-hotplug[9515]: Execute ifup event on interface wan (eth1)
Mon Sep 20 07:16:50 2021 user.notice mwan3-hotplug[9523]: Execute ifup event on interface vwan (eth2)
Mon Sep 20 07:16:51 2021 user.warn mwan3rtmon[10142]: failed to add 192.168.4.0/24 dev wlan0 proto kernel scope link src 192.168.4.2 to table 1 - error: RTNETLINK answers: File exists
Mon Sep 20 07:16:51 2021 user.warn mwan3rtmon[10142]: failed to add 192.168.4.0/24 dev wlan0 proto kernel scope link src 192.168.4.2 to table 2 - error: RTNETLINK answers: File exists
11

mwan3 creates mangle firewall rules. Your problems with dhcp are connected to the dhcp pool you have misconfigured.

No, there shouldn't be any gateway in lan and wlan interfaces.

exactly

The principle is that when an interface is up, the nameservers configured under that interface are actively used. It will work, if all is up and fine. However if something is not working, there might be unexpected behaviour which may make troubleshooting difficult.
Post again the configs to see how is it now.

1 Like
12

network config

config interface 'loopback'
        option device 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
#       option ula_prefix 'fd55:a4b5:c2c1::/48'
        option ipv6 '0'

config device
        option name 'eth0'
        option ipv6 '0'

config device
        option name 'wlan0'
        option ipv6 '0'

config device
        option type 'bridge'
        option name 'br-lan'
        option ipv6 '0'
        list ports 'eth0'
        list ports 'wlan0'

config device
        option name 'eth1'
        option ipv6 '0'

config device
        option name 'eth2'
        option ipv6 '0'

config interface 'wlan0'
        option device 'wlan0'
        option proto 'static'
        option ipaddr '192.168.4.2'
        option netmask '255.255.255.0'
#       option gateway '192.168.1.1'
        option broadcast '192.168.4.255'
        option delegate '0'
#       list dns '208.67.222.123'
#       list dns '208.67.220.123'
        option ip6assign '60'

config interface 'lan'
        option device 'br-lan'
        option proto 'static'
        option ipaddr '192.168.4.1'
        option netmask '255.255.255.0'
#       option gateway '192.168.1.1'
        option broadcast '192.168.4.255'
        option delegate '0'
#       list dns '208.67.222.123'
#       list dns '208.67.220.123'
        option ip6assign '60'

config interface 'wan'
        option device 'eth1'
        #option proto 'dhcp'
        option proto 'static'
        option ipaddr '192.168.1.4'
        option netmask '255.255.255.0'
        option gateway '192.168.1.1'
        option broadcast '192.168.1.255'
        option delegate '0'
        option force_link '1'
        option peerdns '0'
        list dns '208.67.222.123'
        list dns '208.67.220.123'
        option metric '30'

config interface 'vwan'
        option device 'eth2'
        #option proto 'dhcp'
        option proto 'static'
        option ipaddr '192.168.6.11'
        option netmask '255.255.255.0'
        option gateway '192.168.6.2'
        option broadcast '192.168.6.255'
        option delegate '0'
        option force_link '1'
        option peerdns '0'
        list dns '208.67.222.123'
        list dns '208.67.220.123'
        option metric '20'

dhcp config

config dnsmasq
        option localise_queries '1'
        option rebind_protection '1'
        option rebind_localhost '1'
        option local '/lan/'
        option domain 'lan'
        option expandhosts '1'
        option authoritative '1'
        option readethers '1'
        option leasefile '/tmp/dhcp.leases'
        option resolvfile '/tmp/resolv.conf.d/resolv.conf.auto'
        option localservice '1'
        option ednspacket_max '1232'
        option domainneeded '1'
        list addnhosts '/etc/safe-search/enabled'
        option confdir '/tmp/dnsmasq.d'

config dhcp 'lan'
        option interface 'lan'
        option dhcpv6 'server'
        option ra 'server'
        list ra_flags 'managed-config'
        list ra_flags 'other-config'
        option ra_management '1'
        option start '230'
        option limit '20'
        option leasetime '15m'

config dhcp 'wan'
        option interface 'wan'
        option ignore '1'
        option start '100'
        option limit '150'
        option leasetime '12h'
        list ra_flags 'none'

config odhcpd 'odhcpd'
        option maindhcp '0'
        option leasefile '/tmp/hosts/odhcpd'
        option leasetrigger '/usr/sbin/odhcpd-update'
        option loglevel '4'

config dhcp 'wlan0'
        option interface 'wlan0'
        option start '250'
        option limit '1'
        option leasetime '15m'
        list ra_flags 'none'
        option dynamicdhcp '0'

config host
        option dns '1'
        option name 'AccessPoint1'
        list mac 'xx:xx:xx:xx:xx:xx'
        list mac 'yy:yy:yy:yy:yy:yy'
        option ip '192.168.4.30'
        option leasetime '15m'

# more static addresses after that

firewall config

config defaults
        option synflood_protect '1'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'REJECT'

config zone 'lan'
        option name 'lan'
        list network 'lan'
        list network 'wlan0'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'
#       option ip6assign '60'

config zone 'wan'
        option name 'wan'
        list network 'wan'
        list network 'vwan'
        option input 'REJECT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option masq '1'
        option mtu_fix '1'

config forwarding
        option src 'lan'
        option dest 'wan'

config rule
        option name 'Allow-DHCP-Renew'
        option src 'wan'
        option proto 'udp'
        option dest_port '68'
        option target 'ACCEPT'
        option family 'ipv4'

config rule
        option name 'Allow-Ping'
        option src 'wan'
        option proto 'icmp'
        option icmp_type 'echo-request'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-IGMP'
        option src 'wan'
        option proto 'igmp'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-DHCPv6'
        option src 'wan'
        option proto 'udp'
        option src_ip 'fc00::/6'
        option dest_ip 'fc00::/6'
        option dest_port '546'
        option family 'ipv6'
        option target 'ACCEPT'
        option enabled '0'

config rule
        option name 'Allow-MLD'
        option src 'wan'
        option proto 'icmp'
        option src_ip 'fe80::/10'
        list icmp_type '130/0'
        list icmp_type '131/0'
        list icmp_type '132/0'
        list icmp_type '143/0'
        option family 'ipv6'
        option target 'ACCEPT'
        option enabled '0'

config rule
        option name 'Allow-ICMPv6-Input'
        option src 'wan'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        list icmp_type 'router-solicitation'
        list icmp_type 'neighbour-solicitation'
        list icmp_type 'router-advertisement'
        list icmp_type 'neighbour-advertisement'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'
        option enabled '0'

config rule
        option name 'Allow-ICMPv6-Forward'
        option src 'wan'
        option dest '*'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'
        option enabled '0'

config rule
        option name 'Allow-IPSec-ESP'
        option src 'wan'
        option dest 'lan'
        option proto 'esp'
        option target 'ACCEPT'

config rule
        option name 'Allow-ISAKMP'
        option src 'wan'
        option dest 'lan'
        option dest_port '500'
        option proto 'udp'
        option target 'ACCEPT'

config rule
        option name 'Support-UDP-Traceroute'
        option src 'wan'
        option dest_port '33434:33689'
        option proto 'udp'
        option family 'ipv4'
        option target 'REJECT'
        option enabled 'false'

config include
        option path '/etc/firewall.user'

config redirect
        option target 'DNAT'
        option name 'Home Assistant'
        option src 'wan'
        option src_dport '8123'
        option dest 'lan'
        option dest_port '8123'
        option dest_ip '192.168.4.90'

config rule 'samba_nsds'
        option name 'Allow-Samba/NS/DS'
        option src 'lan'
        option dest_port '137-138'
        option proto 'udp'
        option target 'ACCEPT'

config rule 'samba_ss'
        option name 'Allow-Samba/SS'
        option src 'lan'
        option dest_port '139'
        option proto 'tcp'
        option target 'ACCEPT'

config rule 'samba_smb'
        option name 'Allow-Samba/SMB'
        option src 'lan'
        option dest_port '445'
        option proto 'tcp'
        option target 'ACCEPT'

mwan3 config

config globals 'globals'
        option mmx_mask '0x3F00'

config interface 'wan'
        option enabled '1'
        option family 'ipv4'
        option reliability '2'

config interface 'vwan'
        option enabled '1'
        option family 'ipv4'
        option reliability '1'

config member 'wan_m1_w3'
        option interface 'wan'
        option metric '1'
        option weight '3'

config member 'wan_m2_w3'
        option interface 'wan'
        option metric '2'
        option weight '3'

config member 'vwan_m1_w2'
        option interface 'vwan'
        option metric '1'
        option weight '2'

config member 'vwan_m2_w2'
        option interface 'vwan'
        option metric '2'
        option weight '2'

config policy 'wan_only'
        list use_member 'wan_m1_w3'

config policy 'vwan_only'
        list use_member 'vwan_m1_w2'

config policy 'balanced'
        list use_member 'wan_m1_w3'
        list use_member 'vwan_m1_w2'

config policy 'wan_vwan'
        list use_member 'wan_m1_w3'
        list use_member 'vwan_m2_w2'

config policy 'vwan_wan'
        list use_member 'wan_m2_w3'
        list use_member 'vwan_m1_w2'

config rule 'Mob1'
        option family 'ipv4'
        option use_policy 'vwan_only'
        option src_ip 'Mob1'
        option dest_ip '0.0.0.0/0'
        option proto 'all'
        option sticky '0'
        option logging '0'

config rule 'Mob2'
        option family 'ipv4'
        option use_policy 'wan_only'
        option src_ip 'Mob2'
        option dest_ip '0.0.0.0/0'
        option proto 'all'
        option sticky '0'
        option logging '0'

config rule 'PC0'
        option family 'ipv4'
        option src_ip 'PC0'
        option dest_ip '0.0.0.0/0'
        option proto 'all'
        option sticky '0'
        option use_policy 'wan_only'

config rule 'PC1'
        option family 'ipv4'
        option use_policy 'wan_only'
        option src_ip 'PC1'
        option dest_ip '0.0.0.0/0'
        option proto 'all'
        option sticky '0'
        option logging '0'

config rule 'PC2'
        option family 'ipv4'
        option use_policy 'wan_only'
        option src_ip 'PC2'
        option dest_ip '0.0.0.0/0'
        option proto 'all'
        option sticky '0'
        option logging '0'

config rule 'machine1'
        option family 'ipv4'
        option use_policy 'vwan_only'
        option src_ip 'machine1'
        option dest_ip '0.0.0.0/0'
        option proto 'all'
        option sticky '1'
        option logging '1'
        option timeout '600000'

config rule 'machine2'
        option family 'ipv4'
        option use_policy 'vwan_only'
        option src_ip 'machine2'
        option dest_ip '0.0.0.0/0'
        option proto 'all'
        option sticky '1'
        option logging '1'
        option timeout '600000'

config rule 'machine3'
        option family 'ipv4'
        option use_policy 'vwan_only'
        option src_ip 'machine3'
        option dest_ip '0.0.0.0/0'
        option proto 'all'
        option sticky '1'
        option logging '1'
        option timeout '600000'

config rule 'machine4'
        option family 'ipv4'
        option use_policy 'vwan_only'
        option src_ip 'machine4'
        option dest_ip '0.0.0.0/0'
        option proto 'all'
        option sticky '1'
        option logging '1'
        option timeout '600000'

config rule 'Display'
        option family 'ipv4'
        option use_policy 'wan_only'
        option src_ip 'Display'
        option dest_ip '0.0.0.0/0'

config rule 'Spkr'
        option family 'ipv4'
        option use_policy 'wan_only'
        option src_ip 'Spkr'
        option dest_ip '0.0.0.0/0'

config rule 'dev0'
        option family 'ipv4'
        option use_policy 'wan_only'
        option src_ip 'dev0'
        option dest_ip '0.0.0.0/0'

config rule 'dev1'
        option family 'ipv4'
        option use_policy 'wan_only'
        option src_ip 'dev1'
        option dest_ip '0.0.0.0/0'

config rule 'dev2'
        option family 'ipv4'
        option use_policy 'wan_only'
        option src_ip 'dev2'
        option dest_ip '0.0.0.0/0'

config rule 'default_rule_v4'
        option family 'ipv4'
        option use_policy 'vwan_only'
        option dest_ip '0.0.0.0/0'

Note that the names in the option src_ip lines do resolve correctly from dnsmasq static definitions. They are contiguous addresses though, so it is a shame that I have to create a rule for each one separately but if I do not do that the rules are never active no matter how much I stop and restart mwan3.

13

The situation right now is that DHCP is working fine and I don't see DHCP issues anymore.
Thank you very much for pointing out the issues there. I appreciate it.

However, I still have two problems on booting which are

  1. As I mentioned earlier, sometimes the vwan and ethernet LEDs do not come up during reboot and I have to hotplug the interface several times and even remove the cable and plug it in again to get it to work (As far as I can tell, this does not happen when I do not have mwan3 installed). So, to avoid losing internet in that situation, I currently set the default to wan which is not what I want by default.

  2. mwan3 rules are not properly shown as active on reboot, always, regardless of whether the above problem happens or not. To fix this, I do the following manually after reboot and everything works perfectly afterwards until the next reboot.

mwan3 stop
mwan3 restart
>> Command failed: Not found
[mwan3 restart]
mwan3 rules
>> … correctly reporting active rules

I can put this in /etc/rc.local and get it done at the end of the boot sequence but I'd rather have a proper fix than a workaround.

What could be the issues there?

14

Here you are bridging the eth0 with wlan0, but later on you treat them as different interfaces.

There shouldn't be interface wlan0, the lan covers both. The same in the lan firewall zone, only lan interface should be member.
In the mwan3 configuration there is no trach_method or track_ip.

1 Like
15

That's because even though my network is stable, I am not able to find the tracking parameters to keep it enabled. So, I opted for disabling tracking by removing the track_ip as described in the documentation. It is more important to me to assign devices to the two interfaces than fail-over for now.

16

I was following similar configurations. I will try to remove them as you suggest and report back.

17

Another observation I have. When I stop mwan3 and do ping -I eth1 google.com or ping -I eth2 google.com they both get all response packets starting from seq=0.
However, when I restart mwan3, the ping targeting the interface set in the default rule always loses the first 4 responses (starts from seq=4). Is this normal?

18

Indeed, I removed the wlan0 config section completely from /etc/config/network and the system still behaves normally. It did not solve the issue of not having any active user rules in mwan3 though.

19

Can you post the following?

uci export mwan3; mwan3 status; ip -4 addr; ip -4 ro list table all; ip -4 ru

Have you tried with an IP, say 8.8.8.8 ?

1 Like
20

Another observation as I test my setup more is that port forwarding that is working when mwan3 is stopped, gets blocked when mwan3 restarts!