Firewall fails on reboot

Installing and Using OpenWrt
1

No rules get loaded on reboot.

/etc/init.d/firewall restart do not get the firewall going.

Works again after manually running 'service firewall restart' or pressing reload firewall in luci.

How can I get the firewall to work on a reboot?
As 'service firewall restart' don't work from rc.local (before exit 0).

Raspberry pi, lxc container
OpenWrt 21.02.1, r16325-88151b8303

system log

with
cat /etc/rc.local

/etc/init.d/firewall restart
exit 0

Sat Nov 20 07:19:15 2021 user.notice dnsmasq: DNS rebinding protection is active, will discard upstream RFC1918 responses!
Sat Nov 20 07:19:15 2021 user.notice dnsmasq: Allowing 127.0.0.0/8 responses
Sat Nov 20 07:19:15 2021 daemon.info dnsmasq[500]: Connected to system UBus
Sat Nov 20 07:19:15 2021 daemon.info dnsmasq[500]: started, version 2.85 cachesize 150
Sat Nov 20 07:19:15 2021 daemon.info dnsmasq[500]: DNS service limited to local subnets
Sat Nov 20 07:19:15 2021 daemon.info dnsmasq[500]: compile time options: IPv6 GNU-getopt no-DBus UBus no-i18n no-IDN DHCP no-DHCPv6 no-Lua TFTP no-conntrack no-ipset no-auth no-cryptohash no-DNSSEC no-ID loop-detect inotify dumpfile
Sat Nov 20 07:19:15 2021 daemon.info dnsmasq[500]: UBus support enabled: connected to system bus
Sat Nov 20 07:19:15 2021 daemon.info dnsmasq[500]: using only locally-known addresses for domain test
Sat Nov 20 07:19:15 2021 daemon.info dnsmasq[500]: using only locally-known addresses for domain onion
Sat Nov 20 07:19:15 2021 daemon.info dnsmasq[500]: using only locally-known addresses for domain localhost
Sat Nov 20 07:19:15 2021 daemon.info dnsmasq[500]: using only locally-known addresses for domain local
Sat Nov 20 07:19:15 2021 daemon.info dnsmasq[500]: using only locally-known addresses for domain invalid
Sat Nov 20 07:19:15 2021 daemon.info dnsmasq[500]: using only locally-known addresses for domain bind
Sat Nov 20 07:19:15 2021 daemon.info dnsmasq[500]: using only locally-known addresses for domain lan
Sat Nov 20 07:19:15 2021 daemon.warn dnsmasq[500]: no servers found in /tmp/resolv.conf.d/resolv.conf.auto, will retry
Sat Nov 20 07:19:15 2021 daemon.info dnsmasq[500]: read /etc/hosts - 4 addresses
Sat Nov 20 07:19:15 2021 daemon.info dnsmasq[500]: read /tmp/hosts/dhcp.cfg01411c - 1 addresses
Sat Nov 20 07:19:15 2021 user.notice : Added device handler type: 8021ad
Sat Nov 20 07:19:15 2021 user.notice : Added device handler type: 8021q
Sat Nov 20 07:19:15 2021 user.notice : Added device handler type: macvlan
Sat Nov 20 07:19:15 2021 user.notice : Added device handler type: veth
Sat Nov 20 07:19:15 2021 user.notice : Added device handler type: bridge
Sat Nov 20 07:19:15 2021 user.notice : Added device handler type: Network device
Sat Nov 20 07:19:15 2021 user.notice : Added device handler type: tunnel
Sat Nov 20 07:19:15 2021 daemon.warn netifd: You have delegated IPv6-prefixes but haven't assigned them to any interface. Did you forget to set option ip6assign on your lan-interfaces?
Sat Nov 20 07:19:15 2021 daemon.notice netifd: Interface 'lan' is enabled
Sat Nov 20 07:19:15 2021 daemon.notice netifd: Interface 'lan' is setting up now
Sat Nov 20 07:19:15 2021 daemon.notice netifd: Interface 'lan' is now up
Sat Nov 20 07:19:15 2021 daemon.notice netifd: bridge 'br-lan' link is up
Sat Nov 20 07:19:15 2021 daemon.notice netifd: Interface 'lan' has link connectivity
Sat Nov 20 07:19:15 2021 daemon.notice netifd: Interface 'srv' is enabled
Sat Nov 20 07:19:15 2021 daemon.notice netifd: bridge 'br-lxc' link is up
Sat Nov 20 07:19:15 2021 daemon.notice netifd: Interface 'srv' has link connectivity
Sat Nov 20 07:19:15 2021 daemon.notice netifd: Interface 'srv' is setting up now
Sat Nov 20 07:19:15 2021 daemon.notice netifd: Interface 'srv' is now up
Sat Nov 20 07:19:15 2021 daemon.notice netifd: Interface 'wan' is enabled
Sat Nov 20 07:19:15 2021 daemon.notice netifd: bridge 'br-wan' link is up
Sat Nov 20 07:19:15 2021 daemon.notice netifd: Interface 'wan' has link connectivity
Sat Nov 20 07:19:15 2021 daemon.notice netifd: Interface 'wan' is setting up now
Sat Nov 20 07:19:15 2021 daemon.notice netifd: Interface 'loopback' is enabled
Sat Nov 20 07:19:15 2021 daemon.notice netifd: Interface 'loopback' is setting up now
Sat Nov 20 07:19:15 2021 daemon.notice netifd: Interface 'loopback' is now up
Sat Nov 20 07:19:15 2021 daemon.notice netifd: Network device 'eth2' link is up
Sat Nov 20 07:19:15 2021 daemon.notice netifd: Network device 'lo' link is up
Sat Nov 20 07:19:15 2021 daemon.notice netifd: Interface 'loopback' has link connectivity
Sat Nov 20 07:19:15 2021 daemon.notice netifd: Network device 'eth0' link is up
Sat Nov 20 07:19:15 2021 daemon.notice netifd: Network device 'eth1' link is up
Sat Nov 20 07:19:15 2021 daemon.notice netifd: wan (853): udhcpc: started, v1.33.1
Sat Nov 20 07:19:15 2021 user.notice firewall: Reloading firewall due to ifup of lan (br-lan)
Sat Nov 20 07:19:15 2021 daemon.notice netifd: wan (853): udhcpc: sending discover
Sat Nov 20 07:19:15 2021 cron.err crond[936]: crond (busybox 1.33.1) started, log level 5
Sat Nov 20 07:19:15 2021 daemon.notice netifd: wan (853): udhcpc: sending select for XXXXXXXX
Sat Nov 20 07:19:15 2021 user.notice firewall: Reloading firewall due to ifup of srv (br-lxc)
Sat Nov 20 07:19:16 2021 user.notice ucitrack: Setting up /etc/config/network reload dependency on /etc/config/dhcp
Sat Nov 20 07:19:16 2021 user.notice ucitrack: Setting up /etc/config/wireless reload dependency on /etc/config/network
Sat Nov 20 07:19:16 2021 user.notice ucitrack: Setting up /etc/config/firewall reload dependency on /etc/config/luci-splash
Sat Nov 20 07:19:16 2021 user.notice ucitrack: Setting up /etc/config/firewall reload dependency on /etc/config/qos
Sat Nov 20 07:19:16 2021 user.notice ucitrack: Setting up /etc/config/firewall reload dependency on /etc/config/miniupnpd
Sat Nov 20 07:19:16 2021 user.notice ucitrack: Setting up /etc/config/dhcp reload dependency on /etc/config/odhcpd
Sat Nov 20 07:19:16 2021 daemon.notice netifd: wan (853): udhcpc: lease of XXXXXXXX obtained, lease time 1200
Sat Nov 20 07:19:16 2021 daemon.notice netifd: Interface 'wan' is now up
Sat Nov 20 07:19:16 2021 daemon.info dnsmasq[500]: reading /tmp/resolv.conf.d/resolv.conf.auto
Sat Nov 20 07:19:16 2021 daemon.info dnsmasq[500]: using only locally-known addresses for domain test
Sat Nov 20 07:19:16 2021 daemon.info dnsmasq[500]: using only locally-known addresses for domain onion
Sat Nov 20 07:19:16 2021 daemon.info dnsmasq[500]: using only locally-known addresses for domain localhost
Sat Nov 20 07:19:16 2021 daemon.info dnsmasq[500]: using only locally-known addresses for domain local
Sat Nov 20 07:19:16 2021 daemon.info dnsmasq[500]: using only locally-known addresses for domain invalid
Sat Nov 20 07:19:16 2021 daemon.info dnsmasq[500]: using only locally-known addresses for domain bind
Sat Nov 20 07:19:16 2021 daemon.info dnsmasq[500]: using only locally-known addresses for domain lan
Sat Nov 20 07:19:16 2021 daemon.info dnsmasq[500]: using nameserver XXXXXXXX#53
Sat Nov 20 07:19:16 2021 daemon.info dnsmasq[500]: using nameserver XXXXXXXX#53
Sat Nov 20 07:19:16 2021 user.notice firewall: Reloading firewall due to ifup of wan (br-wan)
Sat Nov 20 07:19:16 2021 user.notice ucitrack: Setting up non-init /etc/config/fstab reload handler: /sbin/block mount
Sat Nov 20 07:19:16 2021 user.notice ucitrack: Setting up /etc/config/system reload dependency on /etc/config/luci_statistics
Sat Nov 20 07:19:16 2021 user.notice ucitrack: Setting up /etc/config/system reload dependency on /etc/config/dhcp
Sat Nov 20 07:19:16 2021 daemon.notice procd: /etc/rc.d/S95done: Warning: Unable to locate ipset utility, disabling ipset support
Sat Nov 20 07:19:16 2021 daemon.notice procd: /etc/rc.d/S95done: Warning: Section @defaults[0] requires unavailable target extension FLOWOFFLOAD, disabling
Sat Nov 20 07:19:16 2021 daemon.notice procd: /etc/rc.d/S95done: Warning: Section @defaults[0] requires unavailable target extension FLOWOFFLOAD, disabling
Sat Nov 20 07:19:16 2021 daemon.notice procd: /etc/rc.d/S95done:  * Set tcp_ecn to off
Sat Nov 20 07:19:16 2021 daemon.notice procd: /etc/rc.d/S95done:  * Set tcp_syncookies to on
Sat Nov 20 07:19:16 2021 daemon.notice procd: /etc/rc.d/S95done:  * Set tcp_window_scaling to on
Sat Nov 20 07:19:16 2021 daemon.notice procd: /etc/rc.d/S95done:  * Running script '/etc/firewall.user'
Sat Nov 20 07:19:16 2021 authpriv.info dropbear[1436]: Not backgrounding
Sat Nov 20 07:19:17 2021 daemon.info procd: - init complete -
Sat Nov 20 07:19:17 2021 daemon.info dnsmasq[500]: exiting on receipt of SIGTERM
Sat Nov 20 07:19:17 2021 daemon.info dnsmasq[1760]: Connected to system UBus
Sat Nov 20 07:19:17 2021 daemon.info dnsmasq[1760]: started, version 2.85 cachesize 150
Sat Nov 20 07:19:17 2021 daemon.info dnsmasq[1760]: DNS service limited to local subnets
Sat Nov 20 07:19:17 2021 daemon.info dnsmasq[1760]: compile time options: IPv6 GNU-getopt no-DBus UBus no-i18n no-IDN DHCP no-DHCPv6 no-Lua TFTP no-conntrack no-ipset no-auth no-cryptohash no-DNSSEC no-ID loop-detect inotify dumpfile
Sat Nov 20 07:19:17 2021 daemon.info dnsmasq[1760]: UBus support enabled: connected to system bus
Sat Nov 20 07:19:17 2021 daemon.info dnsmasq-dhcp[1760]: DHCP, IP range 10.10.10.100 -- 10.10.10.249, lease time 12h
Sat Nov 20 07:19:17 2021 daemon.info dnsmasq-dhcp[1760]: DHCP, IP range 192.168.1.100 -- 192.168.1.249, lease time 12h
Sat Nov 20 07:19:17 2021 daemon.info dnsmasq[1760]: using only locally-known addresses for domain test
Sat Nov 20 07:19:17 2021 daemon.info dnsmasq[1760]: using only locally-known addresses for domain onion
Sat Nov 20 07:19:17 2021 daemon.info dnsmasq[1760]: using only locally-known addresses for domain localhost
Sat Nov 20 07:19:17 2021 daemon.info dnsmasq[1760]: using only locally-known addresses for domain local
Sat Nov 20 07:19:17 2021 daemon.info dnsmasq[1760]: using only locally-known addresses for domain invalid
Sat Nov 20 07:19:17 2021 daemon.info dnsmasq[1760]: using only locally-known addresses for domain bind
Sat Nov 20 07:19:17 2021 daemon.info dnsmasq[1760]: using only locally-known addresses for domain lan
Sat Nov 20 07:19:17 2021 daemon.info dnsmasq[1760]: reading /tmp/resolv.conf.d/resolv.conf.auto
Sat Nov 20 07:19:17 2021 daemon.info dnsmasq[1760]: using only locally-known addresses for domain test
Sat Nov 20 07:19:17 2021 daemon.info dnsmasq[1760]: using only locally-known addresses for domain onion
Sat Nov 20 07:19:17 2021 daemon.info dnsmasq[1760]: using only locally-known addresses for domain localhost
Sat Nov 20 07:19:17 2021 daemon.info dnsmasq[1760]: using only locally-known addresses for domain local
Sat Nov 20 07:19:17 2021 daemon.info dnsmasq[1760]: using only locally-known addresses for domain invalid
Sat Nov 20 07:19:17 2021 daemon.info dnsmasq[1760]: using only locally-known addresses for domain bind
Sat Nov 20 07:19:17 2021 daemon.info dnsmasq[1760]: using only locally-known addresses for domain lan
Sat Nov 20 07:19:17 2021 daemon.info dnsmasq[1760]: using nameserver XXXXXX#53
Sat Nov 20 07:19:17 2021 daemon.info dnsmasq[1760]: using nameserver XXXXXX#53
Sat Nov 20 07:19:17 2021 daemon.info dnsmasq[1760]: read /etc/hosts - 4 addresses
Sat Nov 20 07:19:17 2021 daemon.info dnsmasq[1760]: read /tmp/hosts/dhcp.cfg01411c - 3 addresses
Sat Nov 20 07:19:17 2021 daemon.info dnsmasq-dhcp[1760]: read /etc/ethers - 0 addresses
Sat Nov 20 07:19:21 2021 daemon.err uhttpd[1042]: luci: accepted login on / for root from 192.168.1.140
Sat Nov 20 07:19:27 2021 authpriv.info dropbear[1826]: Child connection from 192.168.1.140:28239
Sat Nov 20 07:19:28 2021 authpriv.notice dropbear[1826]: Password auth succeeded for 'root' from 192.168.1.140:28239
Sat Nov 20 07:19:56 2021 daemon.info procd: Instance sysntpd::instance1 s in a crash loop 6 crashes, 3 seconds since last crash
Sat Nov 20 07:20:20 2021 daemon.info dnsmasq-dhcp[1760]: DHCPDISCOVER(br-lan) 24:df:a7:dc:0e:9b
Sat Nov 20 07:20:20 2021 daemon.info dnsmasq-dhcp[1760]: DHCPOFFER(br-lan) 192.168.1.100 24:df:a7:dc:0e:9b
Sat Nov 20 07:20:20 2021 daemon.info dnsmasq-dhcp[1760]: DHCPREQUEST(br-lan) 192.168.1.100 24:df:a7:dc:0e:9b
Sat Nov 20 07:20:20 2021 daemon.info dnsmasq-dhcp[1760]: DHCPACK(br-lan) 192.168.1.100 24:df:a7:dc:0e:9b Himalaya
Sat Nov 20 07:20:33 2021 daemon.info dnsmasq-dhcp[1760]: DHCPREQUEST(br-lan) 192.168.1.100 24:df:a7:dc:0e:9b
Sat Nov 20 07:20:33 2021 daemon.info dnsmasq-dhcp[1760]: DHCPACK(br-lan) 192.168.1.100 24:df:a7:dc:0e:9b Himalaya
Sat Nov 20 07:20:57 2021 daemon.info dnsmasq-dhcp[1760]: DHCPREQUEST(br-lan) 192.168.1.100 24:df:a7:dc:0e:9b
Sat Nov 20 07:20:57 2021 daemon.info dnsmasq-dhcp[1760]: DHCPACK(br-lan) 192.168.1.100 24:df:a7:dc:0e:9b Himalaya
3

Looks like you have special stuff in uci-defaults (or actually rc.local)
Scripts or config that use firewall related kmods that are not installed, e.g. ipset

Similarly, something related to flow offloads?

Based on that log extract, I guess that your firewall config contains stuff that is not fully supported by your hardware and/or installed modules. When you attempt to restart firewall from rc.local, the errors surface.

1 Like
4

I saw this behavior on my openwrt container. My container is running on a proxmox server and on every openwrt reboot the firewall wouldn’t reload even though the system log said it was reloading due to interfaces coming up.

My solution was to create a bash script that ran from rc.local to check if the firewall was loaded and reload it if needed. Then it does a couple more checks over a minute to make sure the firewall is still loaded and then the script would finish/terminate. After that a cronjob that runs every 30 minutes just checks to see if the firewall needs to be reloaded. The cronjob might be overkill but I haven’t had the firewall not loaded whenever I log in to luci to check.

1 Like
5

All setting except adding "/etc/init.d/firewall restart" in /etc/rc.local were the default settings for the image.

As I mentioned "/etc/init.d/firewall restart" do not fixes the problem. Yet "service firewall restart" do but I can not run it in /etc/rc.local. The errors you see are the same when I run "service firewall restart" so they are not the problem.

The problem seems to be that "service firewall restart" do something differently from "/etc/init.d/firewall restart" and on the container version of openwrt only "service firewall restart" do its job but in the startup script only "/etc/init.d/firewall restart" are used. So a bug

6

Tried follow your lead and added a bash script to restart the firewall.

But the log says it can't find the file:

daemon.notice procd: /etc/rc.d/S95done: /etc/rc.local: line 4: /firestart.sh: not found

Where did you put the script? I tried both "/root/" and "/" yet no luck....

7

I placed my bash script in the /root directory.

In the "System > Startup > Local Startup" options (in luci) my contents look like this:

# Put your custom commands here that should be executed once
# the system init finished. By default this file does nothing.

/root/checkfw.sh &

exit 0

Then I made sure the script was executable (chmod +x) and that was it.

I did test the script before hand to make sure openwrt's shell would read and run the commands properly.

8

Thanks for your input.

My problem was that I was using #!/bin/bash instead of #!/bin/ash.....

It did not fix my problem though as ash refused to recognise "service firewall restart" the command.

./firestart.sh
./firestart.sh: line 3: service: not found

Do anyone know what command is run when pressing "Restart Firewall" on the page "firewall" or from "startup" in the GUI?

Both successfully starts the firewall after boot.

9

/etc/init.d/firewall restart

10

/etc/init.d/firewall restart

1 Like
11

Normally that should just be /bin/sh
No need to specify ash, which is the defaults shell in busybox.

As bash is not installed by default, so a script with bash shebang will naturally fail.

1 Like
12

The command my script uses to restart the firewall is "fw3 restart"

13

Strange whatever bug that break the firewall seems to be after rc.local. As I did get the script working but it did not help.

I need to do a check like genuser1 did, that continue too check after startup.

14

Hi all,
I'm a bit late to this party as this thing has just hit me lately while migrating from 19.07 (which doesn't show the issue) to 21.02 on kernel 5.10, so had to abort migration for now. Tested on my laptop on a more recent kernel and both 19.07 and 21.02 fail to populate the firewall rules on boot.

What I noticed is that restarting from the Status->Firewall page in LuCI does work, while service firewall restart or /etc/init.d/firewall restart on the command line (and in rc.local) don't, and the output shows none of the defined rules being loaded.
After playing around a bit i spotted another pattern, i.e. the firewall restart commands work again after running (some) iptables commands, which may probably be why it works from the firewall status page, since it dumps the iptables state to show to the user.
So as a quick'n'dirty hack this oneliner seems to work:

sleep 30 && iptables -L > /dev/null && /etc/init.d/firewall restart

Sucks for crudeness and feels unreliable but looks deterministic enough to be considered as a temporary workaround. Hope someone could come up with a proper fix tho...

edit: apparently the sleep interval, which I added just to be sure and avoid any interference with the firewall too soon during its setup, can be safely removed...

15

Kernel 5.10 introduces new firewall semantics, in particular iptables tables are not instantiated until something "pokes" them (the iptables -L in your case does that).

Since OpenWrt 21.02 does not use kernel 5.10, the necessary changes to fw3 are not present in it. The fix is: https://git.openwrt.org/?p=project/firewall3.git;a=commitdiff;h=50979cc9c3805a72145440299b5c78e1be25c473

2 Likes
16

How is the status on older kernel versions?
Yesterday, I upgraded my LXC container to 21.02.3 running on Ubuntu 20.04 LTS with kernel 5.4.0-109-generic x86_64. Firewall rules are still partly missing, nat table is completely empty until I reload/restart the firewall manually via Luci.

17

LXC has similar issues. The /proc/net/ip_tables_names is not updated / iptables tables not instantiated in jails until something touches them. Usually an iptables -t filter -L; iptables -t nat -L; iptables -t mange -L suffices.

I think the problem happens in any environment where the kernel and kmods are not managed by OpenWrt. Traditional Linux distributions load the iptables table kmods on demand while OpenWrt always preloads them on boot. Combined with the fw3 logic of checking the presence of tables before attempting to use them, this leads to the observed issue of empty rulesets.

18

Will there be a fix for this problem? Should I create a service with higher priority than the firewall and run those iptables commands there?
Will firewall4 with nftables still have this problem? I will try 22.03.0-rc1 next, hopefully it's available for download via lxc-create.
I would be so fine, if I could convert my router LXC container running a standard Ubuntu LTS to OpenWrt, doing things via Luci is so much easier.

19

Ah so one needs to do it for every table? In fact I had also found this commit and was wondering...

20

A fix for this has been added to the firewall.git repo and will appear in OpenWrt once the pacakge is updated. The 22.03.0-rc1 release should contain it, backports to older versions will follow.

No, fw4 does not have this problem.

That's the right idea, the firewall might be started even earlier though, as consequence of an interface hotplug event, so you could add a custom event handler before 20-firewall:

# cat /etc/hotplug.d/iface/19-poke-iptables

#!/bin/sh

[ "$ACTION" = ifup ] || exit 0
fw3 -q network "$INTERFACE" >/dev/null || exit 0

for ipt in iptables ip6tables; do
  for tbl in filter nat mangle raw; do
    $ipt -t $tbl -L &>/dev/null
  done
done
2 Likes
21

Yes. The listing attempt for the given table will in turn cause the kernel to request the related module, which is then modprobed on demand, updating /proc/net/ip{,6}_tables_names

1 Like