Mark firewall rules not working as expected

[UsernameHasBeenTaken]

Hello!

I need an advice why the firewall rules which set marks are not working as expected and how to fix/change them. I have two rules: the first one is marking traffic with 0x1 value, the second one is marking traffic with 0x2 value. They both can be applied to the same packet and I expect the firewall to match the first rule, apply the mark and do not process the second rule. But it seems that the second rule is also processed and as a result I don't know what mark do I get, but surely not the mark I need.

Some more details.
I have an nft set which then is filled by dnsmasq with two host IPs (ifconfig.io and myip.com):

/etc/config/firewall - nft set

config ipset
        option name 'no_vpn'
        option family 'ipv4'
        list match 'dest_net'

I also have two routing rules which redirects traffic either to the wan or to the wg_out interface:

/etc/config/network - routing rules

config rule
        option priority '100'
        option mark '0x1'
        option lookup 'main'

config rule
        option priority '110'
        option mark '0x2' 
        option lookup 'wg_out_table'

And I have two firewall rules to mark the traffic:

/etc/config/firewall - traffic rules

config rule
        option name 'Mark 0x1 ipconfig.io myip.com'
        option family 'ipv4'
        option src 'lan'
        option ipset 'no_vpn'
        option target 'MARK'
        option dest '*'
        list proto 'all'
        option set_mark '0x1'

config rule
        option name 'Mark 0x2 all other'
        option family 'ipv4'
        option src 'lan'
        option dest '*'
        option target 'MARK'
        option set_mark '0x2'
        list proto 'all'

Please note that I still have main and default routing tables which route traffic through wan.

The logic behind this is:

1. Mark traffic to the specific hosts (ifconfig.io, myip.com) with 0x1 and route them through wan.
2. Mark all other traffic with 0x2 and route it through wg_out.

Unfortunately if I go to ifconfig.io or myip.com I will see the wg_out IP address.
But If I rearrange these two rules and change their order (set mark 0x2 first and 0x1 second) then it works as it should. But I expect the opposite behavior.
I.e. I expect the firewall to match the first rule (before rearranging), mark traffic with 0x1 and stop processing (or at least do not process the second rule).

I am confused and do not understand what is happening. I will be happy if you help me sort it out.

[brada4]

Please connect to your OpenWrt device using ssh and copy the output of the following commands and post it here using the "Preformatted text </> " button:

Remember to redact passwords, MAC addresses and any public IP addresses you may have:

ubus call system board
cat /etc/config/network
cat /etc/config/wireless
cat /etc/config/dhcp
cat /etc/config/firewall

[UsernameHasBeenTaken]

Thanks!
Already did that in the initial post: please refer to the code hidden under spoilers. If you need additional configuration please tell me.

[dave14305]

The order that works is as expected. Mark rules do not “terminate” processing in the chain.

If you want the second rule to ignore the packets marked by the first rule, you could try adding option mark '!0x1' to the second rule to ignore packets already given 0x1.

Or you can use option ipset '!no_vpn'

[brada4]

@UsernameHasBeenTaken

But If I rearrange these two rules and change their order (set mark 0x2 first and 0x1 second) then it works as it should. But I expect the opposite behavior.

You need to make your own firewall that does opposite to nftables.

[UsernameHasBeenTaken]

@dave14305

The order that works is as expected. Mark rules do not “terminate” processing in the chain.

If you want the second rule to ignore the packets marked by the first rule, you could try adding option mark '!0x1' to the second rule to ignore packets already given 0x1.

Or you can use option ipset '!no_vpn'

I didn't know that mark rules do not terminate chain processing!
Thank you very much for the solution and education. That's exactly what I needed!

One more small question: how to write not 0x1 or 0x2 - if I will have more marks in the future?

I was looking at the simple accept/deny rules:
Accept ping -> Reject ping and Reject ping -> Accept ping. Thought that mark rules act the same...

@brada4

You need to make your own firewall that does opposite to nftables.

Oh, thank you for the suggestion! I definitely will try if I need to. But for now the question is resolved with a simple exclamation mark - Kudos to @dave14305

[dave14305]

@UsernameHasBeenTaken

One more small question: how to write not 0x1 or 0x2 - if I will have more marks in the future?

I haven’t given it much thought, but “not 1 or 2” might be represented by 0x0/0x3 (first and second bits are clear or zero).

[dave14305]

I think though, if you are going to have multiple rules, you can probably simplify by adding option mark '0x0' to each rule. Meaning, only apply this rule if no previous rule has already changed the mark.

[UsernameHasBeenTaken]

That seems to be even more elegant solution! And it covers the not 0x1 or not 0x2 case!

Thank you! I really appreciate your help!

[brada4]

It is a 32bit bitmask, and you can do all kinds of bitops on it.