Mwan3 dhcp ipset and firewall

My router has 2 WANs and VPNs. I need to create rules that route packets not just based on the source or destination IPs, but also FQDNs.
For the latter I use ipset.
This works only if I create ipsets manually (through a script). I tried to create ipsets via Luci but they appear to be ignored.

As a very simple example, I need to route packets destined to contatori.vodafone.it via the Wireless WAN (WWAN).

In /etc/config/dhcp I have:

config ipset 'contatori'
	list name 'contatori'
	list domain 'contatori.vodafone.it'
	list domain 'ricarica1click.vodafone.it'

/etc/config/mwan3:

config rule 'contatori_voda'
    option ipset 'contatori'
    option logging '1'
    option dest_port ''
    option proto ''
    option sticky ''
    option use_policy 'wwan_only'

/etc/config/firewall:

config ipset
    option name 'contatori'
    option family 'ipv4'
    option storage 'hash'
    list match 'ip'

The above configuration does not work under OpenWRT because no ipset is actually created (ipset -L does not include 'contatori') and mwan3 breaks at startup failing to create any rule, not even the rules that do not contain ipsets, because it does not find the ipset named in one of its rules.

To make it work I have to create the required ipset manually with:
> ipset create contatori hash:ip
and then restart mwan3

After the above the ipset can be seen with ipset -L, IPs get populated automatically into the ipset just by querying the DNS (dnsmasq).

Right now I resolved this issue with a simple script that I call from /etc/rc.local, however I was wondering what am I missing as I would expect this to work in OpenWRT out of the box.

Be aware that fw4 uses nftables as a backend with a built-in IP set implementation which is not compatible with the standalone IP sets used by fw3.

Meaning? I do not know what is the firewall version. I am using OpenWRT 22.03 with its default firewall. I assume OpenWRT 22.03 was put together with compatible packages that integrate together. It would not make any sense otherwise. I am asking how to make it work for my use case above.

Unfortunately, this is not the case since MWAN3 doesn't support nftables yet, and still depends on legacy iptables and standalone IP sets.

If none of your packages depend on fw4, try replacing it with fw3:

opkg update
opkg remove firewall4
opkg install firewall
/etc/init.d/firewall restart

Unfortunately an invalid assumption for any packages that use iptables.

It is complicated and a bit of a mess in my opinion.

  1. 22.03's dnsmasq-full does not support nftset but does support ipset.
  2. 22.03's fw4 cannot support nftset because dnsmasq-full does not support it.
  3. 22.03's fw4 cannot support ipset either, because ipset is designed to work with iptables.
  4. 22.03's fw4 does not create an ipset so mwan3 will not find an ipset.
  5. 23.05's dnsmsaq full does support nftset but does not support ipset.
  6. 23.05's fw4 does a weird ipset emulation using nftset, so mwan3 will not find the pseudo "ipset" that 23.05's fw4 creates.

As you have found out with your script workaround, there are ways to make it work.
Unfortunately the script will not work if you upgrade to 23.05 (see my list above re dnsmasq versions).

@vgaetera 's fw3 suggestion will work on 22.03, but fail on 23.05 (again, see my list above re dnsmasq versions) Unfortunately using fw3 may well break other packages that use nftables and depend on fw4....

1 Like

OK, thank you for the clarification regarding fw4. So, if there are so many components incompatible with fw4, why did they upgrade to fw4? What is it that fw4 provide that is worth breaking the integration with components that put all together define a "router"?

I think the OpenWRT project badly needs a "Product Manager". And an "Architect".

Migration to nftables is the way to go as iptables is deprecated.
Fw4 is fully nftables.
The issue is that some of the decisions made in developing fw4 broke some old packages that have not been migrated yet.
There are arguments for and against those decisions but as it stands now, OpenWrt's support of "sets" is different to every other Linux distribution.
Many packages are ported to OpenWrt from generic Linux, so if the package being ported is dependant on iptables with ipsets, it is going to be broken.

One argument is that this will encourage package developers to migrate to nftables, but many packages are ported to OpenWrt by OpenWrt enthusiasts and not the original developers, so it is a problem.

The top two packages effected by iptables/nftables non-migration are mwan3 and sql-scripts. There are several workarounds people have figured out for these, but more often than not the packages do not work "out of the box".

1 Like

Thanks to your clarification I replaced firewall4 with firewall and everything works now.
iptables is still being developed and supported.
I need a working router, not the latest packages.

There are two simple changes that will mitigate most of these problems, maintaining compatibility of non-nftables-migrated packages.

  1. Make iptables-nft the default package to load when iptables is used by a package (currently it is iptables-legacy, resulting in clashes and errors in Luci etc)
  2. Compile dnsmasq-full with support for BOTH ipset and nftset.

Good. But remember when you upgrade to 23.05 it will be broken again.

My main router is a Linksys WRT 32x (LOL) and 23.05 introduces further regressions (OpenWRT is into a degrading trend unfortunately). So I am not planning to upgrade to 23.05 any time soon. I have just moved to 22.03 from OpenWRT 19 that was the last best release.

I see that option for dnsmasq-full in 23.05, but mwan3 has no compile options. Does 23.05 mwan3 still look at ipset?

Unfortunately not. Any iptables development aimed at the iptables-nft version that real-time translates to nftables.
https://git.netfilter.org/iptables/

1 Like

Indeed, but it is not really "full", as ipset is disabled.

It has not been migrated to nftables.

Yes - because it has not been migrated.

You are mostly correct, but:

It's basically the same as firewalld preinstalled in Fedora, CentOS, RHEL, openSUSE, etc.

1 Like