NFT Command executed in /rc.local suddently disappear

Installing and Using OpenWrt
1

Hi,

I've a set of commands in /rc.local, they get executed but for some reason the last line seems reverted after few seconds after the boot.

# Put your custom commands here that should be executed once
# the system init finished. By default this file does nothing.

# Load nftables script, this replace iptables in openwrt21
nft -f /etc/nftables.conf

nft add set inet fw4 blackhole { type ipv4_addr \; timeout 24h \;}
nft insert rule inet fw4 forward_lan ip daddr @blackhole accept

#The blackhole set is updated by dnsmasq, configuration in /etc/dnsmasq.conf
exit 0

This behavior is not always reproducible, if I reboot OpenWrt multiple times I will get the
@blackhole filter sometimes there and sometimes not.

At the begin I was wondering that the last command was not always executed and failed for some reason, but seems instead executed.

So, I've done a reboot and start to grep the content of the nft table multiple times and found the below

BusyBox v1.35.0 (2022-12-28 12:41:04 UTC) built-in shell (ash)

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 -----------------------------------------------------
 OpenWrt SNAPSHOT, r21615-5863363493
 -----------------------------------------------------
=== WARNING! =====================================
There is no root password defined on this device!
Use the "passwd" command to set up a new password
in order to prevent unauthorized SSH logins.
--------------------------------------------------
root@OpenWrt:/# nft list table inet fw4 | grep @blackhole
                ip daddr @blackhole accept
root@OpenWrt:/# nft list table inet fw4 | grep @blackhole
                ip daddr @blackhole accept
root@OpenWrt:/# nft list table inet fw4 | grep @blackhole
                ip daddr @blackhole accept
root@OpenWrt:/# nft list table inet fw4 | grep @blackhole
                ip daddr @blackhole accept
root@OpenWrt:/# nft list table inet fw4 | grep @blackhole
                ip daddr @blackhole accept
root@OpenWrt:/# nft list table inet fw4 | grep @blackhole
root@OpenWrt:/# nft list table inet fw4 | grep @blackhole
root@OpenWrt:/# nft list table inet fw4 | grep @blackhole
root@OpenWrt:/# nft list table inet fw4 | grep @blackhole
root@OpenWrt:/#

So the rule is there for some time (few seconds, I've copy pasted the command in the shell multiple time) and then disappear.

I've no other nft commands included by me in other configuration files. A dirty fix is a sleep command and this works reliably.

Can someone help me to understand where to look to understand why this happen, so that a proper fix can be applied?

Thanks.

Regards,
Dario.

2

Race condition?

Tried adding an initial sleep?

3

Hi @frollic a sleep before the command does the trick, but my goal would be to understand why I get in this case.

4

My guess is, the firewall rule setup takes time, and your changes get applied too early in the process.

5

That is why I was suggesting uci based rules in the other thread.

Alternatively use nft include files.

4 Likes
6

@plinioseniore
I have encountered this behaviour in numerous scenarios.
This is what I found was happening:

Firewall4 does a restart on hotplug events such as a hardware interface coming up. As the system is booting, each hardware interface comes up in turn, and triggers Firewall4 to restart.
It will do this multiple times before the boot/startup process is completed.

When Firewall4 restarts, it flushes and deletes its own table inet/fw4.

Your rc.local script will run before all the network interfaces are up (most of the time), so the rules you add to inet/fw4 will be flushed by Firewall4.

To fix this in a clean way without using a "sleep" you can do one of the following:

  • Use an nft include file in the config of Firewall4 (as suggested by @jow )
  • Create your own table and put your rule in there, so Firewall4 will not touch it when it restarts.

To use your own table your script would look something like:

nft add table inet black_hole
nft add chain inet black_hole forward_lan
nft add set inet black_hole blackhole { type ipv4_addr \; timeout 24h \;}
nft insert rule inet black_hole forward_lan ip daddr @blackhole accept

To flush and delete your table, just do:
nft delete table inet black_hole

7

A separate table will likely not work for selectively accepting forwards as the subsequent fw4 will not honor the accept verdict

8

It depends upon what the OP's blackhole set does. A black hole set usually blocks everything with a match. Additional fine grained rules can then be added by firewall4.....

9

Hi @bluewavenet that's interesting. I've three lines and they refer to two different tables.

If the Firewall4 restart and rebuild the configuration multiple time while booting, should this affect also the below creation of the nftset?

nft add set inet fw4 blackhole { type ipv4_addr \; timeout 24h \;}

I will try to replicate, but from my understanding the blackhole set was always there rather the forward accept rule was missing.

The first command recall the /etc/nftables.conf file that is the below and refers to a different table, and if I understand this table will not be touched.

table ip nat {
        chain prerouting {
                type nat hook prerouting priority filter; policy accept;
                ip daddr x.y.0.0/16 tcp dport { 80, 443 } dnat to 192.168.56.2:48241
                ip daddr 0.0.0.0/0 udp dport 53 dnat to 192.168.56.2:53
        }
}
10

The set name is coming from some starting examples, but the way I'm using it is an allow list of IP addresses, so I'm not sure that this will work.

11

An allow list will work, but packets will be passed on to the firewall4 inet/fw4 chains for further processing. This can be useful, adding fine grained control, but can be bad by blocking what you just allowed....

12

No, I changed it to:

nft add set inet black_hole blackhole { type ipv4_addr \; timeout 24h \;}
13

@bluewavenet This is what happen if right after the boot I list the set and the table. There is no cut in this log out of the list of ip addresses in the set.

[    7.181028] br-lan: port 1(eth0) entered blocking state
[    7.181861] br-lan: port 1(eth0) entered disabled state
[    7.183042] device eth0 entered promiscuous mode
[    7.189472] 8021q: adding VLAN 0 to HW filter on device eth2
[    7.192199] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
[    7.193533] br-lan: port 1(eth0) entered blocking state
[    7.194227] br-lan: port 1(eth0) entered forwarding state
[    7.194929] IPv6: ADDRCONF(NETDEV_CHANGE): br-lan: link becomes ready
[    7.199920] 8021q: adding VLAN 0 to HW filter on device eth1
[    7.212227] e1000: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
[    7.213504] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[    9.243805] e1000: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
[    9.250938] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready



BusyBox v1.35.0 (2022-12-28 12:41:04 UTC) built-in shell (ash)

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 -----------------------------------------------------
 OpenWrt SNAPSHOT, r21615-5863363493
 -----------------------------------------------------
=== WARNING! =====================================
There is no root password defined on this device!
Use the "passwd" command to set up a new password
in order to prevent unauthorized SSH logins.
--------------------------------------------------
root@OpenWrt:/# nft list set inet fw4 blackhole
table inet fw4 {
        set blackhole {
                type ipv4_addr
                timeout 1d
                elements = { 5.200.6.34 expires 23h59m56s640ms, 20.54.37.64 expires 23h59m57s200ms,
							...
                             212.45.144.88 expires 23h59m56s660ms }
        }
}
root@OpenWrt:/# nft list table inet fw4 | grep blackhole
        set blackhole {
                ip daddr @blackhole accept
root@OpenWrt:/#
root@OpenWrt:/# nft list set inet fw4 blackhole
table inet fw4 {
        set blackhole {
                type ipv4_addr
                timeout 1d
                elements = { 5.200.6.34 expires 23h59m54s90ms, 20.54.37.64 expires 23h59m54s650ms,
							...
                            212.45.144.88 expires 23h59m54s110ms }
        }
}
root@OpenWrt:/# nft list table inet fw4 | grep blackhole
        set blackhole {
root@OpenWrt:/#
  • At first time of nft list set inet fw4 blackhole the set is shown
  • At first time of nft list table inet fw4 | grep blackhole the forward accept is shown
  • At second time of nft list set inet fw4 blackhole the set is shown
  • At second time of nft list table inet fw4 | grep blackhole the forward accept is not shown

If the issue I'm having is related to a rebuild of fw4 table, shouldn't also the blackhole set have been disappeared? At same time I see no print of a log that state interface up or down, that is what I would expect.

Do you feel that my issue is related to the behavior you listed before?

EDIT : Only now that I read what I wrote I figured out that nft list table inet fw4 | grep blackhole alone would have be enough as is showing the first line of the set and the forward accept.

14

First from your log:

This shows eth0 coming up. Firewall4 will restart.

Next from your log:

eth1 comes up. Firewall4 will restart again.

Third log entry:

eth2 comes up, Firewall4 restarts for the third time.

15

I see, but I cannot understand why the set is still there and the forward accept is not. Are both in fw4 table.

16

It will be a matter of timing.
If I'm correct, if you do service firewall restart, everything you added to inet/fw4 will be gone.

17

Yes, the restart erase both the set and the forward accept.

18

I've done an down/up cycle of eth1 interface and it confirms that this affect the firewall configuration, is not honestly clear to me why the effect is different from service firewall restart

root@OpenWrt:/# nft list table inet fw4 | grep blackhole
        set blackhole {
                ip daddr @blackhole accept
root@OpenWrt:/# ip link set eth1 down
root@OpenWrt:/# ip link set eth1 up
[  151.110589] 8021q: adding VLAN 0 to HW filter on device eth1
root@OpenWrt:/# nft list table inet fw4 | grep blackhole
        set blackhole {
                ip daddr @blackhole accept
root@OpenWrt:/# [  153.193011] e1000: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
[  153.202505] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready

root@OpenWrt:/# nft list table inet fw4 | grep blackhole
        set blackhole {
root@OpenWrt:/#

Thanks to all for supporting in this.

19

Last, the solution in my case to keep the configuration as nft commands in etc/rc.local and at same time ensure consistency during rebuild of fw4 while interfaces were flipping up/down is to use the relevant hotplug that triggers the fw4 rebuilds.

I've added as last line in /etc/hotplug.d/iface/20-firewall
sh /etc/rc.local

This will restart the rc.local every time that a change into an iterface trigger a rebuild of fw4. In my specific case, that was fine enough as etc/rc.local does only changes in the firewall, otherwise I should have added only the nft commands in that file.

Again thanks to all for the support.

1 Like
20

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.