[SOLVED] Unable to mirror traffic based on ip address for monitoring using nft-dup

I'm using a Raspberry Pi 3B+ (192.168.1.6 / VLAN 1 / br0 on eth0) as a VPN gateway (OpenVPN tun0) for Client A (192.168.5.10 / VLAN 2 / eth0.2). I split the single Pi interface with a managed L2 switch and VLANs. I am trying to duplicate traffic destined for Client A to Monitoring PC on the VLAN 1 (192.168.1.107) I've installed the require kmod (kmod-nft-dup-inet) and confirmed with modprobe and lsmod that it is infact loaded.

However when trying to create the rule from the command line I get syntax errors, I've read through the forums for similar thread and also consulted the nft wiki (https://wiki.nftables.org/wiki-nftables/index.php/Duplicating_packets) but I am still not sure what I am doing wrong, I would be very grateful if anyone could point me in the correct direction.

I'm using the latest OpenWRT snapshot for the Pi 3B+

┌──(root@rpi-owrt)-[00:08:56]-[~]
└─# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br0 state UP qlen 1000
    link/ether b8:27:eb:27:4b:d5 brd ff:ff:ff:ff:ff:ff
3: teql0: <NOARP> mtu 1500 qdisc noop state DOWN qlen 100
    link/void
4: wlan0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether b8:27:eb:72:1e:80 brd ff:ff:ff:ff:ff:ff
5: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether b8:27:eb:27:4b:d5 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.6/24 brd 192.168.1.255 scope global br0
       valid_lft forever preferred_lft forever
    inet6 fe80::ba27:ebff:fe27:4bd5/64 scope link
       valid_lft forever preferred_lft forever
6: eth0.2@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc htb state UP qlen 1000
    link/ether b8:27:eb:27:4b:d5 brd ff:ff:ff:ff:ff:ff
    inet 192.168.5.1/24 brd 192.168.5.255 scope global eth0.2
       valid_lft forever preferred_lft forever
    inet6 fe80::ba27:ebff:fe27:4bd5/64 scope link
       valid_lft forever preferred_lft forever
7: eth0.3@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc htb state UP qlen 1000
    link/ether b8:27:eb:27:4b:d5 brd ff:ff:ff:ff:ff:ff
    inet 192.168.6.1/24 brd 192.168.6.255 scope global eth0.3
       valid_lft forever preferred_lft forever
    inet6 fe80::ba27:ebff:fe27:4bd5/64 scope link
       valid_lft forever preferred_lft forever
8: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN qlen 500
    link/[65534]
    inet 10.8.0.6/24 scope global tun0
       valid_lft forever preferred_lft forever
    inet6 fe80::a466:c6ac:c325:3bf1/64 scope link flags 800
       valid_lft forever preferred_lft forever
9: ifb0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc htb state UNKNOWN qlen 32
    link/ether 2e:2d:13:75:85:04 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::2c2d:13ff:fe75:8504/64 scope link
       valid_lft forever preferred_lft forever
10: ifb1: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc htb state UNKNOWN qlen 32
    link/ether 42:81:fb:59:4f:62 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::4081:fbff:fe59:4f62/64 scope link
       valid_lft forever preferred_lft forever
┌──(root@rpi-owrt)-[00:08:58]-[~]
└─# nft add rule inet fw4 mangle prerouting dup to ip saddr map { 192.168.5.10 : 192.168.1.107 }
Error: syntax error, unexpected dup
add rule inet fw4 mangle prerouting dup to ip saddr map { 192.168.5.10 : 192.168.1.107 }
                                    ^^^
┌──(root@rpi-owrt)-[00:09:10]-[~]
└─# nft add rule inet fw4 mangle prerouting dup to br0
Error: syntax error, unexpected dup
add rule inet fw4 mangle prerouting dup to br0
                                    ^^^
┌──(root@rpi-owrt)-[00:10:02]-[~]
└─# nft add rule mangle prerouting dup to br0
Error: Could not resolve hostname: Name does not resolve
add rule mangle prerouting dup to br0
                                  ^^^
┌──(root@rpi-owrt)-[00:10:32]-[~]
└─# nft add rule mangle prerouting dup to 192.168.1.107
Error: No such file or directory; did you mean table ‘fw4’ in family inet?
add rule mangle prerouting dup to 192.168.1.107
         ^^^^^^
┌──(root@rpi-owrt)-[00:10:45]-[~]
└─#

I think (not verified!) that you need to installed the nft-dup kmod to get that functionality:

$ opkg list | grep nft-dup
kmod-nft-dup-inet - 5.15.117-1 - Netfilter nf_tables dup in ip/ip6/inet family support

As I recall, this was added after 22.03 was released, so you'd need to be running either the 23.05 RC or SNAPSHOT to get it.

1 Like

Hi, thanks for your reply, I did install the kernel module and checked it with modprobe/lsmod, it does seem to be installed and available, and I am indeed running the latest snapshot.

It appears that you can only create rules with a dup action in a single family table. This is very odd, as everything else in nftables is dual-stack (with, of course, the caveat that you can't mix up the families in a given expression, set or map). Since this expression is "pure" IPv4, seems like it should work.

$ nft create table ip test4
$ nft create chain ip test4 input '{ type filter hook input priority 0; policy accept ; }'
$ nft add rule ip test4 input 'dup to ip saddr map { 192.168.5.10 : 192.168.1.107 }'
$ nft list table ip test4
table ip test4 {
        chain input {
                type filter hook input priority filter; policy accept;
                dup to ip saddr map { 192.168.5.10 : 192.168.1.107 }
        }
}

But, change the table to dual stack...

$ nft create table inet test46
$ nft create chain inet test46 input '{ type filter hook input priority 0; policy accept ; }'
$ nft add rule inet test46 input 'dup to ip saddr map { 192.168.5.10 : 192.168.1.107 }'
Error: unsupported family
add rule inet test46 input dup to ip saddr map { 192.168.5.10 : 192.168.1.107 }
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

So, add a pair of tables, one for each IP family?

1 Like

thank you very much for your time in testing that and the response, your reply is full of information, I just need to wrap my head around it :grin: will test it out and update this thread

Well, I tried some other stuff, too. Nothing appears to work in a dual-stack inet table.

  1. Tried named maps (https://wiki.nftables.org/wiki-nftables/index.php/Maps).
  2. Tried adding a redundant family selector on the rule meta nfproto ipv4, which is almost never necessary and should not be in this case, but I thought "Why not?"

Here's the script I used, in case it saves your some time:

#!/bin/sh -x

#$ nft add rule inet fw4 mangle_prerouting 'dup to ip saddr map { 192.168.5.10 : 192.168.1.107 }'
#Error: unsupported family
#add rule inet fw4 mangle_prerouting dup to ip saddr map { 192.168.5.10 : 192.168.1.107 }
#                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

if false ; then
    nft create table ip test4
    nft create chain ip test4 input '{ type filter hook input priority 0; policy accept ; }'
    nft add    rule  ip test4 input 'dup to ip saddr map { 192.168.5.10 : 192.168.1.107 }'
    nft list   table ip test4
    nft delete table ip test4
fi

if false ; then
    nft create table ip6 test6
    nft create chain ip6 test6 input '{ type filter hook input priority 0; policy accept ; }'
    nft add    rule  ip6 test6 input 'dup to ip6 saddr map { fe80::1:2:3:4 : fe80::4:3:2:1 }'
    nft list   table ip6 test6
    nft delete table ip6 test6
fi

if true ; then
    nft create table inet test46

    nft add map inet test46 dup_v4 '{ type ipv4_addr : ipv4_addr; elements = { 192.168.4.10   :  192.168.1.107 } }'
    nft add map inet test46 dup_v6 '{ type ipv6_addr : ipv6_addr; elements = { fe80::1:2:3:4  :  fe80::4:3:2:1 } }'

    nft create chain inet test46 input '{ type filter hook input priority 0; policy accept ; }'
    nft add    rule  inet test46 input 'meta nfproto ipv4  dup to ip  saddr map { 192.168.5.10 : 192.168.1.107 }'
    nft add    rule  inet test46 input 'meta nfproto ipv4  dup to ip  saddr map @dup_v4'
    nft add    rule  inet test46 input 'dup to ip6 saddr map { fe80::1:2:3:4 : fe80::4:3:2:1 }'
    nft list   table inet test46
    nft delete table inet test46
fi

Edit: Here's a quick ref for all those arcane expressions: https://wiki.nftables.org/wiki-nftables/index.php/Quick_reference-nftables_in_10_minutes

1 Like

In theory, you can also dup from a netdev table. Never tried it though.

1 Like

Thank you again for your detailed answer and for scripting the required commands, I'm still trying to wrap my head around nftables in general. I should also say that my raspberrry pi vpn box is ipv4 only. And the firewall zones have also been set to restrict to ipv4 only. My first area of confusion is that my error is not the same as yours mine says "Error: syntax error, unexpected dup" whereas yours says "Error: unsupported family" I did try your suggestion from your first answer, I created a new table then the chain and then the rule, however it didnt seem to actually do anything, opening wireshark on the monitoring computer didnt show any of the traffic.

Atleast following your steps from the first answer, I was able to create the table/chain and add the dup rule.

Do I understand correctly that since there are no hooks from the base table, that is why its not doing anything? Below is my entire ruleset.

table inet fw4 {
	chain input {
		type filter hook input priority filter; policy accept;
		iifname "lo" accept comment "!fw4: Accept traffic from loopback"
		ct state established,related accept comment "!fw4: Allow inbound established and related flows"
		tcp flags syn / fin,syn,rst,ack jump syn_flood comment "!fw4: Rate limit TCP syn packets"
		meta nfproto ipv4 iifname "br0" jump input_lan comment "!fw4: Handle lan IPv4 input traffic"
		meta nfproto ipv4 iifname "tun0" jump input_wan comment "!fw4: Handle wan IPv4 input traffic"
		meta nfproto ipv4 iifname "eth0.2" jump input_lan1 comment "!fw4: Handle lan1 IPv4 input traffic"
		meta nfproto ipv4 iifname "eth0.3" jump input_lan2 comment "!fw4: Handle lan2 IPv4 input traffic"
	}

	chain forward {
		type filter hook forward priority filter; policy drop;
		ct state established,related accept comment "!fw4: Allow forwarded established and related flows"
		meta nfproto ipv4 iifname "br0" jump forward_lan comment "!fw4: Handle lan IPv4 forward traffic"
		meta nfproto ipv4 iifname "tun0" jump forward_wan comment "!fw4: Handle wan IPv4 forward traffic"
		meta nfproto ipv4 iifname "eth0.2" jump forward_lan1 comment "!fw4: Handle lan1 IPv4 forward traffic"
		meta nfproto ipv4 iifname "eth0.3" jump forward_lan2 comment "!fw4: Handle lan2 IPv4 forward traffic"
		jump handle_reject
	}

	chain output {
		type filter hook output priority filter; policy accept;
		oifname "lo" accept comment "!fw4: Accept traffic towards loopback"
		ct state established,related accept comment "!fw4: Allow outbound established and related flows"
		meta nfproto ipv4 oifname "br0" jump output_lan comment "!fw4: Handle lan IPv4 output traffic"
		meta nfproto ipv4 oifname "tun0" jump output_wan comment "!fw4: Handle wan IPv4 output traffic"
		meta nfproto ipv4 oifname "eth0.2" jump output_lan1 comment "!fw4: Handle lan1 IPv4 output traffic"
		meta nfproto ipv4 oifname "eth0.3" jump output_lan2 comment "!fw4: Handle lan2 IPv4 output traffic"
	}

	chain prerouting {
		type filter hook prerouting priority filter; policy accept;
		meta nfproto ipv4 iifname "eth0.2" jump helper_lan1 comment "!fw4: Handle lan1 IPv4 helper assignment"
		meta nfproto ipv4 iifname "eth0.3" jump helper_lan2 comment "!fw4: Handle lan2 IPv4 helper assignment"
	}

	chain handle_reject {
		meta l4proto tcp reject with tcp reset comment "!fw4: Reject TCP traffic"
		reject comment "!fw4: Reject any other traffic"
	}

	chain syn_flood {
		limit rate 25/second burst 50 packets return comment "!fw4: Accept SYN packets below rate-limit"
		drop comment "!fw4: Drop excess packets"
	}

	chain input_lan {
		jump accept_from_lan
	}

	chain output_lan {
		jump accept_to_lan
	}

	chain forward_lan {
		jump accept_to_lan
	}

	chain accept_from_lan {
		meta nfproto ipv4 iifname "br0" counter packets 669 bytes 67699 accept comment "!fw4: accept lan IPv4 traffic"
	}

	chain accept_to_lan {
		meta nfproto ipv4 oifname "br0" ct state invalid counter packets 0 bytes 0 drop comment "!fw4: Prevent NAT leakage"
		meta nfproto ipv4 oifname "br0" counter packets 0 bytes 0 accept comment "!fw4: accept lan IPv4 traffic"
	}

	chain input_wan {
		jump reject_from_wan
	}

	chain output_wan {
		jump accept_to_wan
	}

	chain forward_wan {
		jump reject_to_wan
	}

	chain accept_to_wan {
		meta nfproto ipv4 oifname "tun0" ct state invalid counter packets 45 bytes 2340 drop comment "!fw4: Prevent NAT leakage"
		meta nfproto ipv4 oifname "tun0" counter packets 581 bytes 121648 accept comment "!fw4: accept wan IPv4 traffic"
	}

	chain reject_from_wan {
		meta nfproto ipv4 iifname "tun0" counter packets 0 bytes 0 jump handle_reject comment "!fw4: reject wan IPv4 traffic"
	}

	chain reject_to_wan {
		meta nfproto ipv4 oifname "tun0" counter packets 0 bytes 0 jump handle_reject comment "!fw4: reject wan IPv4 traffic"
	}

	chain input_lan1 {
		jump accept_from_lan1
	}

	chain output_lan1 {
		jump accept_to_lan1
	}

	chain forward_lan1 {
		meta nfproto ipv4 jump accept_to_wan comment "!fw4: Accept lan1 to wan IPv4 forwarding"
		jump accept_to_lan1
	}

	chain helper_lan1 {
	}

	chain accept_from_lan1 {
		meta nfproto ipv4 iifname "eth0.2" counter packets 0 bytes 0 accept comment "!fw4: accept lan1 IPv4 traffic"
	}

	chain accept_to_lan1 {
		meta nfproto ipv4 oifname "eth0.2" counter packets 0 bytes 0 accept comment "!fw4: accept lan1 IPv4 traffic"
	}

	chain input_lan2 {
		jump accept_from_lan2
	}

	chain output_lan2 {
		jump accept_to_lan2
	}

	chain forward_lan2 {
		meta nfproto ipv4 jump accept_to_wan comment "!fw4: Accept lan2 to wan IPv4 forwarding"
		jump accept_to_lan2
	}

	chain helper_lan2 {
	}

	chain accept_from_lan2 {
		meta nfproto ipv4 iifname "eth0.3" counter packets 0 bytes 0 accept comment "!fw4: accept lan2 IPv4 traffic"
	}

	chain accept_to_lan2 {
		meta nfproto ipv4 oifname "eth0.3" counter packets 0 bytes 0 accept comment "!fw4: accept lan2 IPv4 traffic"
	}

	chain dstnat {
		type nat hook prerouting priority dstnat; policy accept;
	}

	chain srcnat {
		type nat hook postrouting priority srcnat; policy accept;
		meta nfproto ipv4 oifname "br0" jump srcnat_lan comment "!fw4: Handle lan IPv4 srcnat traffic"
		meta nfproto ipv4 oifname "tun0" jump srcnat_wan comment "!fw4: Handle wan IPv4 srcnat traffic"
	}

	chain srcnat_lan {
		meta nfproto ipv4 masquerade comment "!fw4: Masquerade IPv4 lan traffic"
	}

	chain srcnat_wan {
		meta nfproto ipv4 masquerade comment "!fw4: Masquerade IPv4 wan traffic"
	}

	chain raw_prerouting {
		type filter hook prerouting priority raw; policy accept;
	}

	chain raw_output {
		type filter hook output priority raw; policy accept;
	}

	chain mangle_prerouting {
		type filter hook prerouting priority mangle; policy accept;
	}

	chain mangle_postrouting {
		type filter hook postrouting priority mangle; policy accept;
	}

	chain mangle_input {
		type filter hook input priority mangle; policy accept;
	}

	chain mangle_output {
		type route hook output priority mangle; policy accept;
	}

	chain mangle_forward {
		type filter hook forward priority mangle; policy accept;
		meta nfproto ipv4 iifname "br0" tcp flags syn tcp option maxseg size set rt mtu comment "!fw4: Zone lan IPv4 ingress MTU fixing"
		meta nfproto ipv4 oifname "br0" tcp flags syn tcp option maxseg size set rt mtu comment "!fw4: Zone lan IPv4 egress MTU fixing"
		meta nfproto ipv4 iifname "tun0" tcp flags syn tcp option maxseg size set rt mtu comment "!fw4: Zone wan IPv4 ingress MTU fixing"
		meta nfproto ipv4 oifname "tun0" tcp flags syn tcp option maxseg size set rt mtu comment "!fw4: Zone wan IPv4 egress MTU fixing"
	}
}
table ip test4 {
	chain input {
		type filter hook input priority filter; policy accept;
		dup to ip saddr map { 192.168.5.10 : 192.168.1.107 }
	}
}

You might want to try hook forward instead of hook input, or maybe hook prerouting

1 Like

thanks your suggestion, I will try that now.

beautiful! wireshark is now showing stuff!!

2 Likes

oh thats a bummer, i can only select one answer as the solution, thanks for your help too mate

1 Like

Yeah, my tests were non-functional, hence the input hook. I should have mentioned that (I wrote the script off the top of my head and just typed whatever came to mind first).

2 Likes

Good job! So, now you owe us the whole picture. :grin: What table and chain did you use, and what's the form of the final working rule?

1 Like

:laughing: yes ofcourse! this is what I'm using now and its doing exactly what I need

$ nft create chain ip duper input '{ type filter hook prerouting priority 0; policy accept ; }'
$ nft add rule ip duper input 'dup to ip saddr map { 192.168.5.10 : 192.168.1.107, 192.168.6.11 : 192.168.1.107 }'
$ nft add rule ip duper input 'dup to ip daddr map { 192.168.5.10 : 192.168.1.107, 192.168.6.11 : 192.168.1.107 }'
$ nft list table ip duper
table ip duper {
        chain input {
                type filter hook prerouting priority filter; policy accept;
                dup to ip saddr map { 192.168.5.10 : 192.168.1.107, 192.168.6.11 : 192.168.1.107 }
                dup to ip daddr map { 192.168.5.10 : 192.168.1.107, 192.168.6.11 : 192.168.1.107 }
        }
}

192.168.5.10 and 192.168.6.11 are the two clients that I am monitoring on wireshark located at 192.168.1.107

1 Like

Looks good!

One nitpick that has no bearing on functionality, just style: by convention, chain names are usually based on the hook to which they are assigned (and often suffixed with the priority, if it's not the "default 0" == filter).

chain input {
    ... hook input priority filter ...

chain prerouting_raw {
    ... hook prerouting priority raw ...
1 Like

that's good to know, I'll make a note, I'm also putting the ruleset into a nft file which will make things easier.

If you want to keep it in one file, here's a trick I use to "pretty format" the table, yet keep everything together (the -f - says "read stdin", which then is redirected from the "heredoc" delimited by TABLE).

#!/bin/sh
# First, delete the table if it exists.
nft list tables | grep -q 'inet snort' && nft -e delete table inet snort

# Now we rebuild it from scratch.
nft -f - <<TABLE
    table inet snort {
        chain prerouting_ips {
            type filter hook prerouting priority raw; policy accept;
            counter  queue flags bypass to 4-6
        }
    }
TABLE
2 Likes

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.