Partial packet denials with netfilter when used in bridge mode (23.05.3)

Hello,

I am trying to block internet access to several devices using the netfilter bridge table. I have some simple rules to check this and what happens is that it only blocks a percentage of the packets.

My setup is a default install of openwrt 23.05.3 using a tp-link td-w8980 with openwrt firewall disabled:

[isp router]----------[openwrt bridge]---------[pc]

lan1: openwrt to isp router
lan2: pc to openwrt

default policy is accept and only the 1.1.1.1 is blocked:

nft flush ruleset
nft add table bridge br_filter
nft add chain bridge br_filter forward_chain '{type filter hook forward priority 0; policy accept;}'
nft add rule bridge br_filter forward_chain ip daddr 1.1.1.1 meta nftrace set 1 counter drop
nft add rule bridge br_filter forward_chain counter
# nft list ruleset
table bridge br_filter {
	chain forward_chain {
		type filter hook forward priority 0; policy accept;
		ip daddr 1.1.1.1 meta nftrace set 1 counter packets 11 bytes 924 drop
		counter packets 491154 bytes 501732600
	}
}

from the PC I ping the blocked IP:

mpd@sbc-01:~$ ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1): 56 data bytes
64 bytes from 1.1.1.1: icmp_seq=0 ttl=64 time=3.854 ms
[...]
64 bytes from 1.1.1.1: icmp_seq=33 ttl=64 time=23.336 ms
^C--- 1.1.1.1 ping statistics ---
35 packets transmitted, 23 packets received, 34% packet loss     <------- 34% packet loss 
round-trip min/avg/max/stddev = 3.854/14.601/26.780/6.214 ms

and only some packets are denied but not all, of course if the previous rules are deleted the success is 100%.

for the blocked packets the log seem correct:

# nft monitor trace
trace id 3446f4cc bridge br_filter forward_chain packet: iif "lan2" oif "lan1" ether saddr b0:f1:ec:c1:49:82 ether daddr 94:6a:b0:1d:a2:ed ip saddr 192.168.1.212 ip daddr 1.1.1.1 ip dscp cs0 ip ecn not-ect ip ttl 64 ip id 44204 ip protocol icmp ip length 84 icmp type echo-request icmp code net-unreachable icmp id 34963 icmp sequence 6 @th,64,96 0xcace286600000000f56d0e00 
trace id 3446f4cc bridge br_filter forward_chain rule ip daddr 1.1.1.1 meta nftrace set 1 counter packets 0 bytes 0 drop (verdict drop)
trace id 1ca1d464 bridge br_filter forward_chain packet: iif "lan2" oif "lan1" ether saddr b0:f1:ec:c1:49:82 ether daddr 94:6a:b0:1d:a2:ed ip saddr 192.168.1.212 ip daddr 1.1.1.1 ip dscp cs0 ip ecn not-ect ip ttl 64 ip id 45077 ip protocol icmp ip length 84 icmp type echo-request icmp code net-unreachable icmp id 34963 icmp sequence 13 @th,64,96 0xd1ce28660000000061790e00 
trace id 1ca1d464 bridge br_filter forward_chain rule ip daddr 1.1.1.1 meta nftrace set 1 counter packets 0 bytes 0 drop (verdict drop)
trace id 1ca1d464 bridge br_filter forward_chain packet: iif "lan2" oif "lan1" ether saddr b0:f1:ec:c1:49:82 ether daddr 94:6a:b0:1d:a2:ed ip saddr 192.168.1.212 ip daddr 1.1.1.1 ip dscp cs0 ip ecn not-ect ip ttl 64 ip id 45588 ip protocol icmp ip length 84 icmp type echo-request icmp code net-unreachable icmp id 34963 icmp sequence 16 @th,64,96 0xd4ce286600000000f9870e00 

Does anyone know why only some packets could be blocked and not all?

sysctl net.netfilter.nf_conntrack_icmp_timeout

You can use conntrack cli utility to drop all states. icmp state is like echo, timestamp etc, it persists if you keep pinging.

It seems to me that when using the bridge table you cannot use states, you can only use accept or drop because it is working at level 2:

https://wiki.nftables.org/wiki-nftables/index.php/Nftables_families#bridge
[..]
Tables of this family see traffic/packets traversing [bridges] No assumptions are made about L3 protocols.
[..]
Note that there is no nf_conntrack integration for the nftables bridge family.

The problem is that the blocking of the destination IP 1.1.1.1 should occur in all the packets that go to that address, but only a percentage goes and another percentage does not.

Yeah, no integration...
https://wiki.nftables.org/wiki-nftables/index.php/Bridge_filtering

and if you drop the first packet (ip daddr 1.1.1.1 drop) there shouldn't even be a state to trace.

So why do some packets pass and others don't?

Beenthere, some magically turn input or output, you need to filter pre/post routing on all ports...

I have added input and output interface to the drop rule: iif "lan2" oif "lan1" to be sure of the direction of the packets:

nft flush ruleset
nft add table bridge br_filter
nft add chain bridge br_filter forward_chain '{type filter hook forward priority 0; policy accept;}'
nft add rule bridge br_filter forward_chain iif "lan2" oif "lan1" ip daddr 1.1.1.1 meta nftrace set 1 counter drop
nft add rule bridge br_filter forward_chain counter
nft monitor trace

and the random blocking happens again: 35% packet loss

mpd@sbc-01:~$ ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1): 56 data bytes
64 bytes from 1.1.1.1: icmp_seq=0 ttl=64 time=3.322 ms
64 bytes from 1.1.1.1: icmp_seq=1 ttl=64 time=11.187 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=64 time=35.022 ms
64 bytes from 1.1.1.1: icmp_seq=3 ttl=64 time=6.034 ms
64 bytes from 1.1.1.1: icmp_seq=4 ttl=64 time=48.702 ms
64 bytes from 1.1.1.1: icmp_seq=5 ttl=64 time=29.531 ms
64 bytes from 1.1.1.1: icmp_seq=9 ttl=64 time=12.864 ms
64 bytes from 1.1.1.1: icmp_seq=10 ttl=64 time=8.841 ms
64 bytes from 1.1.1.1: icmp_seq=13 ttl=64 time=18.047 ms
^C--- 1.1.1.1 ping statistics ---
14 packets transmitted, 9 packets received, 35% packet loss
round-trip min/avg/max/stddev = 3.322/19.283/48.702/14.398 ms

network configuration is basic:

root@OpenWrt:~# brctl show
bridge name	bridge id		STP enabled	interfaces
br-lan		7fff.30b5c29dc47d	no		lan2
                                        lan3
                                        lan1
  
root@OpenWrt:~# ip -4 a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
8: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    inet 192.168.1.6/24 brd 192.168.1.255 scope global br-lan
       valid_lft forever preferred_lft forever
root@OpenWrt:~# 

brada4 you are right, I have tried disconnecting all my network devices and the blocking works. Now I have to find which one produces this problem.

Best is pre or postrouting

Using all three chains gives the same result. Only a third of ping packets are blocked and they are blocked in prerouting:

table bridge br_filter {
	chain prerouting_chain {
		type filter hook prerouting priority 0; policy accept;
		ip daddr 1.1.1.1 meta nftrace set 1 counter packets 87 bytes 7308 drop
		counter packets 269244 bytes 298561512
	}

	chain forward_chain {
		type filter hook forward priority 0; policy accept;
		ip daddr 1.1.1.1 meta nftrace set 1 counter packets 0 bytes 0 drop
		counter packets 264969 bytes 296124564
	}

	chain postrouting_chain {
		type filter hook postrouting priority 0; policy accept;
		ip daddr 1.1.1.1 meta nftrace set 1 counter packets 0 bytes 0 drop
		counter packets 266973 bytes 296320809
	}
}

Unfortunately the openwrt bridge does not have space to install tcpdump or the tools to make a root extension

It seems DSA bridge somehow does not get "to see" all the packets.

Yes, but I don't see where the problem could be. This switch is physically between the ISP router (lan1 port) and my lan switch (lan2 port). All packets necessarily have to go through that path.

I'll try to do more tests over the weekend.

In swconfig world you could assign each physical port to a tag on CPU port than bridge thoses completely on CPU, seems impossible with DSA, but formally each rule should apply to all passing traffic.

I had tried Receive Packet Steering (RPS) (https://openwrt.org/docs/guide-user/advanced/load_balancing_-_tuning_smp_irq#network_queues) but nothing changed.

Now that you mention it, I could also try downgrading to a version of openwrt that still uses swconfig.

You can set null route for IP(s) you want to totally block

Thanks but I need to filter by ports and protocols.

I installed a custom firmware so I could have space for tools like tcpdump.

If I run a continuous ping from a PC on the network, on the LAN port (lan3) I see about half of the switched packets that are blocked by nfttables and I do not see the other half that is not blocked.

root@OpenWrt:~# nft list ruleset
table bridge br_filter {
	chain prerouting_chain {
		type filter hook prerouting priority 0; policy accept;
		ip daddr 1.1.1.1 meta nftrace set 1 counter packets 1165 bytes 97860 drop
		counter packets 411776 bytes 210471130
	}
}
root@OpenWrt:~# brctl show
bridge name	bridge id		STP enabled	interfaces
br-lan		7fff.30b5c29dc47d	no		lan4
							lan2
							lan3
							lan1
root@OpenWrt:~# sysctl net.bridge
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-filter-pppoe-tagged = 0
net.bridge.bridge-nf-filter-vlan-tagged = 0
net.bridge.bridge-nf-pass-vlan-input-dev = 0
root@OpenWrt:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1508 qdisc fq_codel state UNKNOWN qlen 1000
    link/ether 30:b5:c2:9d:c4:7d brd ff:ff:ff:ff:ff:ff
    inet6 fe80::32b5:c2ff:fe9d:c47d/64 scope link 
       valid_lft forever preferred_lft forever
3: lan2@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master br-lan state LOWERLAYERDOWN qlen 1000
    link/ether 30:b5:c2:9d:c4:7d brd ff:ff:ff:ff:ff:ff
4: lan3@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-lan state UP qlen 1000
    link/ether 30:b5:c2:9d:c4:7d brd ff:ff:ff:ff:ff:ff
5: lan4@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-lan state UP qlen 1000
    link/ether 30:b5:c2:9d:c4:7d brd ff:ff:ff:ff:ff:ff
6: lan1@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master br-lan state LOWERLAYERDOWN qlen 1000
    link/ether 30:b5:c2:9d:c4:7d brd ff:ff:ff:ff:ff:ff
7: wlan0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 30:b5:c2:9d:c4:7f brd ff:ff:ff:ff:ff:ff
8: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 30:b5:c2:9d:c4:7d brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.6/24 brd 192.168.1.255 scope global br-lan
       valid_lft forever preferred_lft forever
    inet6 fd0b:4fbc:df1b::1/60 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::32b5:c2ff:fe9d:c47d/64 scope link 
       valid_lft forever preferred_lft forever

mpd@sbc-01:~$ ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1): 56 data bytes
64 bytes from 1.1.1.1: icmp_seq=0 ttl=64 time=7.206 ms
64 bytes from 1.1.1.1: icmp_seq=1 ttl=64 time=10.953 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=64 time=30.671 ms
64 bytes from 1.1.1.1: icmp_seq=3 ttl=64 time=30.635 ms
64 bytes from 1.1.1.1: icmp_seq=4 ttl=64 time=22.404 ms
64 bytes from 1.1.1.1: icmp_seq=8 ttl=64 time=5.114 ms
64 bytes from 1.1.1.1: icmp_seq=9 ttl=64 time=15.303 ms
64 bytes from 1.1.1.1: icmp_seq=10 ttl=64 time=18.202 ms
64 bytes from 1.1.1.1: icmp_seq=11 ttl=64 time=13.133 ms
64 bytes from 1.1.1.1: icmp_seq=14 ttl=64 time=10.191 ms
64 bytes from 1.1.1.1: icmp_seq=15 ttl=64 time=9.260 ms
64 bytes from 1.1.1.1: icmp_seq=21 ttl=64 time=20.840 ms
^C--- 1.1.1.1 ping statistics ---
23 packets transmitted, 12 packets received, 47% packet loss     <---- 47% packet loss
round-trip min/avg/max/stddev = 5.114/16.159/30.671/8.204 ms

For a few seconds the packages are seen continuously and then they are not seen continuously for another few seconds:

root@OpenWrt:~# tcpdump -nn -i lan3 host 1.1.1.1
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on lan3, link-type EN10MB (Ethernet), snapshot length 262144 bytes
12:38:23.204275 IP 192.168.1.212 > 1.1.1.1: ICMP echo request, id 25252, seq 5, length 64
12:38:24.200966 IP 192.168.1.212 > 1.1.1.1: ICMP echo request, id 25252, seq 6, length 64
12:38:25.201292 IP 192.168.1.212 > 1.1.1.1: ICMP echo request, id 25252, seq 7, length 64
12:38:30.210682 IP 192.168.1.212 > 1.1.1.1: ICMP echo request, id 25252, seq 12, length 64
12:38:31.220556 IP 192.168.1.212 > 1.1.1.1: ICMP echo request, id 25252, seq 13, length 64
12:38:34.214922 IP 192.168.1.212 > 1.1.1.1: ICMP echo request, id 25252, seq 16, length 64
12:38:35.214118 IP 192.168.1.212 > 1.1.1.1: ICMP echo request, id 25252, seq 17, length 64
12:38:36.215019 IP 192.168.1.212 > 1.1.1.1: ICMP echo request, id 25252, seq 18, length 64
12:38:37.215124 IP 192.168.1.212 > 1.1.1.1: ICMP echo request, id 25252, seq 19, length 64
12:38:38.216777 IP 192.168.1.212 > 1.1.1.1: ICMP echo request, id 25252, seq 20, length 64
12:38:40.246003 IP 192.168.1.212 > 1.1.1.1: ICMP echo request, id 25252, seq 22, length 64
^C
11 packets captured
11 packets received by filter
0 packets dropped by kernel

At the same time on the WAN (lan4) interface tcpdump does not see any of the switched packets:

root@OpenWrt:~# tcpdump -nn -i lan4 host 1.1.1.1
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on lan4, link-type EN10MB (Ethernet), snapshot length 262144 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel

It seems that somewhere filtering is not being managed well at the bridge level.

Enabling/disabling packet steering doesn't change anything.

1 Like

You can try "ifconfig br-lan promisc" and so on, it changes interface semantics, maybe makes it work (at least throw different error messages)

Thanks for all your help Brada4, but promiscuous mode doesn't work either.

I will try to find another solution because I can't get the firewall to work at the bridge level with nftables.

Only research I did (counter accept ; counter accept) the numbers in/out/forward match but some packets bypass firewall anyway.

Do you think I should open a case for this? the configuration seems correct, but it only blocks "some" and not all.