23.05 dnsmasq, ipsets and mwan3 incompatibility?

[Axxel]

Ran a test upgrade to 23.05 last night (x86 host) and had to roll back. I'm not clear if all the pieces I use are still expected to work, several threads exist but the interop question seems unanswered.

On 23.05 dnsmasq will not start, because ipset has been removed and my dnsmasq config includes ipset directives. This appears to be an intended change, though not one announced in the known issues for the release.

I use ipsets in the manner describe in the mwan3 docs, specifically to move bulk traffic to multiple subdomains to an unmetered connection. I do not have a fixed list of IP addresses that I can hardcode, tying it to a DNS query makes it much more flexible.

I understand at the dnsmasq level the nftset command is the expected replacement. However, its not clear to me that nftsets are accessible to the iptables compatibility layer mwan3 is still using. Certainly I do not see the existing ipsets mirrored in 'nft list sets' on a working 22.03 installation. Its also not clear what table the dnsmasq-created nftset should be attached to (AIUI unlike ipsets, nftsets are not global).

Building a custom dnsmasq to restore ipset support isn't viable as a long term solution, the decision to remove dnsmasq support appears to be the defacto final policy in the last thread on the topic, but there's also no clear answer for what is supposed to replace this documented mwan3 feature.

ipset-translate exists, but doesn't provide continuous translation and only translates from iptables. A custom periodic translation is presumably possible, but wouldn't cover new entries immediately, which would limit its effectiveness in managing traffic to newly recognized hostnames in the set.

Is this expected to work?

[natevw]

Yes, this was the most recent post just as I logged on to ask something somewhat similar! In my case dnsmasq at least starts fine, but is still not populating any ipsets.

I have as part of "/etc/config/dhcp":

config ipset
	list name 'cameras'
	list domain 'speculor-diluvium.lan'
	list domain 'speculor-turriculam.lan'
	list domain 'speculor-inornatus.lan'
	list domain 'speculor-torqueo.lan'
	list domain 'speculor-opilio.lan'

and as part of my "/etc/config/firewall":

config ipset
	option family 'ipv4'
	list match 'ip'
	option name 'cameras'

but when I nft list ruleset | grep "set cameras" -A 3 it shows only:

	set cameras {
		type ipv4_addr
	}

I was under the impression from various threads and issues (https://github.com/openwrt/openwrt/pull/10820, Dnsmasq (full) and firewall4 - using ipset or nftables together - no out-of-the-box solution in OpenWrt 22.03.03, IP Sets with nftables in LuCi - #2 by dave14305, Ipset by domain using Luci - #7 by stangri) that the main hangup was waiting for a new version of dnsmasq > 2.86 and then the IP Sets tab in LuCI would be working again.

But now we are at "Dnsmasq version 2.89" and it still seems to not work? I do notice that dnsmasq lists:

Compile time options: IPv6 GNU-getopt no-DBus UBus no-i18n no-IDN DHCP no-DHCPv6 no-Lua TFTP no-conntrack no-ipset no-nftset no-auth no-cryptohash no-DNSSEC no-ID loop-detect inotify dumpfile

i.e. not only missing the old ipset (as Dnsmasq-full, ipset support removed (in 23.05 and master) was complaining about even in the -full version) but also compiled as no-nftset which seems to be mismatched — why would the default dnsmasq build still not support something that the default LuCI presents (without warning) as an available option?

UPDATE: I installed dnsmasq-full (via the tips at Install dnsmasq-full over dnsmasq - #5 by stangri) and rebooted (since it seemed wedged trying to renew DHCP…) and still nothing shown added from the DHCP set into the Firewall nft ruleset dump.

[vgaetera]

@natevw

still nothing shown added from the DHCP set into the Firewall nft ruleset dump

This part works for me, but the issue discussed in the OP is a different matter.

Example testing dnsmasq-full with nftables sets on OpenWrt 23.05.0

# Install packages
opkg update
opkg remove dnsmasq
opkg install dnsmasq-full

# Configure dnsmasq
uci -q delete dhcp.example
uci set dhcp.example="ipset"
uci add_list dhcp.example.name="example"
uci add_list dhcp.example.name="example6"
uci add_list dhcp.example.domain="example.com"
uci add_list dhcp.example.domain="example.net"
uci commit dhcp
service dnsmasq restart

# Configure firewall
for IPV in 4 6
do
uci -q delete firewall.example${IPV%4}
uci set firewall.example${IPV%4}="ipset"
uci set firewall.example${IPV%4}.name="example${IPV%4}"
uci set firewall.example${IPV%4}.family="ipv${IPV}"
uci set firewall.example${IPV%4}.match="ip"
done
uci commit firewall
service firewall restart

# Testing
> nslookup example.com localhost
Server:		localhost
Address:	[::1]:53

Non-authoritative answer:
Name:	example.com
Address: 93.184.216.34

Non-authoritative answer:
Name:	example.com
Address: 2606:2800:220:1:248:1893:25c8:1946

> nft list set inet fw4 example
table inet fw4 {
	set example {
		type ipv4_addr
		elements = { 93.184.216.34 }
	}
}

> nft list set inet fw4 example6
table inet fw4 {
	set example6 {
		type ipv6_addr
		elements = { 2606:2800:220:1:248:1893:25c8:1946 }
	}
}

> ubus call system board
{
	"kernel": "5.15.134",
	"hostname": "openwrt4",
	"system": "Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz",
	"model": "QEMU Standard PC (i440FX + PIIX, 1996)",
	"board_name": "qemu-standard-pc-i440fx-piix-1996",
	"rootfs_type": "ext4",
	"release": {
		"distribution": "OpenWrt",
		"version": "23.05.0",
		"revision": "r23497-6637af95aa",
		"target": "x86/64",
		"description": "OpenWrt 23.05.0 r23497-6637af95aa"
	}
}

[natevw]

@vgaetera

but the issue discussed in the OP is a different matter.

Okay, sorry @Axxel didn't mean to sidetrack your post so much, I assumed it was all related. Hopefully can still get your ipset-related discussion resolved too, and perhaps a moderator is able to split my post into its own topic?

[Axxel]

I assumed it was all related

Even if unrelated. Thanks for chiming in, several of the threads in your post were new to me. This post for example seems to show the periodic sync is possible, but will have the liveness issue I was concerned about.

While we wait for someone with more information to hopefully respond, I suppose its time to ask a process question... I'm reasonably sure I know the literal one-line Makefile change required to fix this, but my impression remains this is an intended change, despite the (unintended?) consequences. If the forum doesn't garner a response Is the appropriate next step a GitHub issue?

[jamesmacwhite]

Chiming in as a user of mwan3, but also help maintain the docs and keep an eye out on things.

The situation with mwan3 as I understand it currently is because it hasn't been fully converted to support nftables, so it relies on iptables compatibility related packages to allow for translation. Right now mwan3 essentially runs iptables through a nftables translation to generate valid firewall rules, rather than native.

The maintainer @feckert in fairness foresaw the ipset problem quite early on: Mwan3: nftables porting help - #3 by feckert. Under 22.03, the problem however wasn't as bad it looked as iptables-nft can help and dnsmasq supported ipset still in 22.03.

The issue now is the fact that the upstream dnsmasq-full package has now had it's default compile options updated to remove ipset support in favour of nftables in 23.05, given the firewall4 backend being nftables now established over two major releases. While this initially made some users annoyed because it's a major breaking change, the decision is based on the fact that only a few packages actually have any direct ipset dependencies now and the announcement of the iptables to nftables was made well in advance to prepare.

mwan3 nftset instead of ipset for rules

You can however appreciate that mwan3 being written originally for iptables and a firewall solution which OpenWrt has now dropped in core, now has a pretty major task of needing to be ported, but to ensure all the existing functionality works under the ported nftables version as well. Retesting it all for the variety of different network configurations and such is a major task.

For users of mwan3, that's all great, but what can users do? Here's some guides.

1. Wait until mwan3 natively supports nftables.
2. Use an alternative DNS resolver such as adguardhome, which has ipset support in a similar fashion to dnsmasq.
3. Go back to the 22.03 release where dnsmasq-full has ipset support still
4. Compile your own dnsmasq-full package with ipset enabled.

At the moment, for the least user inconvenience going back to 22.03 might be the easiest, 22.03 is still under active support and will be for a while.

As referenced there are tools to translate ipset to nftables but the issue will likely be the instant nature required of a DNS query returning one or more IP address being added into a set.

This is an important area however, the documentation needs to be updated to reflect the ipset issue, as it's a common feature used by many I'd imagine. I'll look at doing that.

[Axxel]

Thanks for your thorough summary, I appreciate the clear list of options. I'm personally going to freeze against 22.03 till the resolution becomes more clear.

That said, I have a question about this part of your post:

The issue now is the fact that the upstream dnsmasq-full package has now had it's default compile options updated to remove ipset support in favour of nftables in 23.05

From what I can tell the dnsmasq upstream still allows both options. Looking at the breaking change and discussions around it are two separate considerations at play here:

1. What "list of IPs" implementation should back the LuCI "IP set" implementation and /etc/config/dhcp, and how that should be configured in the generated dnsmasq.conf file.

2. Whether dnsmasq-full should be built with "-DHAVE_NFTSET", "-DHAVE_IPSET", or both.

The change in 1 seems consistent with the overall move to nftables, and probably requires the nftables update to mwan3. Perhaps there was an potential workaround for the LuCI code to manage both and ipset and nftset at the same time in the generated dnsmasq.conf, but I'm not qualified to judge that approach or its tradeoffs. If the LuCI IP set functionality switches to nftsets, then that aspect of the mwan3 documentation is broken.

However, the change in 2 appears to be unnecessary, and breaks the other documented way to add ipsets in mwan3, which is direct use of /etc/dnsmasq.conf. This is both how I already use the functionality and seems like an acceptable general workaround till mwan3 is updated. Since this method also already appears in the mwan3 documentation, its not a stretch to make it the officially supported mwan3 method on 23.05.

Assuming my summary is correct, from what I can tell this is a one line change to the OpenWRT Makefile of the dnsmasq-full, but there may be other considerations I'm missing.

[bluewavenet]

@Axxel

from what I can tell this is a one line change to the OpenWRT Makefile of the dnsmasq-full

Unfortunately there is more to it than that.
Fw4 does a horrible "ipset" emulation, based, I was informed, on the idea of preventing confusion of users with little experience but wanting to create a static "Set of IP addresses" like they could in fw3. This however required dnsmasq config to use a pseudo "ipset" / hybrid nftset mishmash unique to OpenWrt.
So now we have a version of dnsmasq that is fundamentally incompatible with the ipset utility, hence the reason for the removal of its support in dnsmasq-full.

[Axxel]

@bluewavenet

So now we have a version of dnsmasq that is fundamentally incompatible with the ipset utility, hence the reason for the removal of its support in dnsmasq-full.

I've seen you mention this other threads, I assume you are referring to this discussion? That seems to refer to point 1 in my more recent post, the desire to map uci "ipset" into nftables. But again, that's independent of the decision to remove ipset support from dnsmasq entirely.

From what I can tell, the change that turned off ipset support made "PACKAGE_dnsmasq_full_ipset" default "n" and "PACKAGE_dnsmasq_full_nftset" default "y". That seems to be the only limitation on dnsmasq-full, and is the one line change I was referring to.

[Axxel]

@bluewavenet

Unfortunately there is more to it than that.

Also, in a different thread you mentioned the ipset-dns package, have you had a chance to try it? That perhaps that should be another workaround mentioned in the mwan3 docs?

[bluewavenet]

@Axxel

That seems to be the only limitation on dnsmasq-full

No it's not the only change. the unique OpenWrt dnsmasq-full version uses an old style ipset config combined with a new style nftset config to write to an nftset created by fw4.

Enabling the dnsmasq-full ipsets option will break fw4's pseudo ipsets, while at the same time mwan3's ipset support will remain broken.

As an aside, the ipset package is required for iptables ipset support because iptables has no means of storing a database (of ip addresses in this case).

In the case of mwan3, a solution is:

1. compile dnsmasq-full with ipset support
2. patch the dnsmasq procd (aka init.d) script to make dnsmasq populate the ipset database.

A consequence of this would be the breaking of fw4's "set of ip's".

Another solution would be:

Fix fw4 to use the term nftset instead of pretending it is doing ipset, making it support nftsets transparently, letting mwan3 do its own thing in iptables-nft.

The best solution would be:

1. Fully migrate mwan3 to nftables (and nftsets)
2. That's it all done.... easy

[bluewavenet]

@Axxel

you mentioned the ipset-dns package, have you had a chance to try it?

No, because I have no need for it for my current requirements.
It has been unmaintained since 2017.....
Perhaps you would like to give it a try?

[dave14305]

@bluewavenet

Enabling the dnsmasq-full ipsets option will break fw4's pseudo ipsets

@bluewavenet

patch the dnsmasq procd (aka init.d) script to make dnsmasq populate the ipset database.

The current dnsmasq init script will already emit ipset= config lines if ipset support is compiled into dnsmasq. So if both ipset and nftset are enabled at compile time, it will emit both ipset= and nftset= lines. It can gernerate both or either from the same uci ipset config.

[bluewavenet]

@dave14305

So if both ipset and nftset are enabled at compile time, it will emit both ipset= and nftset= lines. It can gernerate both or either from the same uci ipset config.

From tests done a few months ago, it seemed to be an either/or situation. As far as I can remember, if both were enabled at compile time and fw4 configured an "ip set" then nothing worked and dnsmasq borked out and stopped.

Both could be enabled if fw4 did not play silly games for the sake of the lowest common denominator. In generic Linux, having both enabled is not a problem. OpenWrt should only be different with dnsmasq vs dnsmasq-full for the sake of saving space.

Nevertheless, if this had not happened, iptables dependent packages would probably linger on unchanged until iptables disappeared so it is probably a "good thing" from the long term perspective.

Having looked just now at the mwan3 code, the problem seems to be that the iptables access is very obfuscated, making migration an uphill and time consuming task... I could be wrong, but this might be the reason it has not yet been done.

[vgaetera]

@bluewavenet

Fw4 does a horrible "ipset" emulation

It works fine for me, what is your problem?

@bluewavenet

Perhaps you would like to give it a try?

It works fine as well:

Example testing ipset-dns on OpenWrt 23.05

# Install packages
opkg update
opkg install ipset ipset-dns

# Configure dnsmasq
uci -q delete dhcp.@dnsmasq[0].server
uci add_list dhcp.@dnsmasq[0].server="/example.com/127.0.0.1#53001"
uci add_list dhcp.@dnsmasq[0].server="/example.net/127.0.0.1#53001"
uci commit dhcp
service dnsmasq restart

# Configure ipset-dns
uci set ipset-dns.@ipset-dns[0].ipset="example"
uci set ipset-dns.@ipset-dns[0].ipset6="example6"
uci set ipset-dns.@ipset-dns[0].dns="8.8.8.8"
uci commit ipset-dns
service ipset-dns restart

# Configure startup
cat << "EOF" > /etc/rc.local
for IPV in 4 6
do ipset create example${IPV%4} hash:net family inet${IPV%4}
done
exit 0
EOF
sh /etc/rc.local

# Testing
> nslookup example.com localhost
Server:		localhost
Address:	[::1]:53

Non-authoritative answer:
Name:	example.com
Address: 93.184.216.34

Non-authoritative answer:
Name:	example.com
Address: 2606:2800:220:1:248:1893:25c8:1946

> ipset list
Name: example
Type: hash:net
Revision: 7
Header: family inet hashsize 1024 maxelem 65536 bucketsize 12 initval 0xc46384ed
Size in memory: 504
References: 0
Number of entries: 1
Members:
93.184.216.34

Name: example6
Type: hash:net
Revision: 7
Header: family inet6 hashsize 1024 maxelem 65536 bucketsize 12 initval 0x84b7ad22
Size in memory: 1312
References: 0
Number of entries: 1
Members:
2606:2800:220:1:248:1893:25c8:1946

> ubus call system board
{
	"kernel": "5.15.134",
	"hostname": "openwrt1",
	"system": "Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz",
	"model": "QEMU Standard PC (i440FX + PIIX, 1996)",
	"board_name": "qemu-standard-pc-i440fx-piix-1996",
	"rootfs_type": "ext4",
	"release": {
		"distribution": "OpenWrt",
		"version": "23.05.0",
		"revision": "r23497-6637af95aa",
		"target": "x86/64",
		"description": "OpenWrt 23.05.0 r23497-6637af95aa"
	}
}

---

Note that iptables, ipset and nftables are hosted by the same project, their development is coordinated, and ipset was developed in conjunction with iptables.
At this point, iptables are basically deprecated by nftables, which leaves no area of application for ipset and no reason to flog a dead horse.

[Axxel]

@bluewavenet

Enabling the dnsmasq-full ipsets option will break fw4's pseudo ipsets, while at the same time mwan3's ipset support will remain broken.

How would this cause a break? Can you provide a link to the relevant code?

I still believe we are talking about two separate problems. There is the ipset option within dnsmasq itself and available in /etc/dnsmasq.conf and then there is the mapping of uci ipsets performed by /etc/init.d/dnsmasq in the generated section of the dnsmasq config (/var/etc/dnsmasq.conf.cfg).

/etc/dnsmasq.conf is imported as a conf-file option in the generated /var/etc/dnsmasq.conf.cfg (code). This is documented.

I'm not proposing any compatibility with uci ipsets. I'm only suggesting that manually created ipset directives in /etc/dnsmasq.conf can be supported, and as far as I can tell it is simply a change in the build options.

[Axxel]

@dave14305

The current dnsmasq init script will already emit ipset= config lines if ipset support is compiled into dnsmasq. So if both ipset and nftset are enabled at compile time, it will emit both ipset= and nftset= lines. It can gernerate both or either from the same uci ipset config.

I saw this code in the init script, but it appeared at the same time the build option for ipsets was removed from the Makefile. Is it possible the dual-add approach caused other problems and that is why the Makefile was changed (instead of fixing the script, or removing the ipset options generated from uci)?

[bluewavenet]

@Axxel

How would this cause a break? Can you provide a link to the relevant code?

Fw4 adds config options to dnsmsaq, but they are not standard dnsmasq options.
This is translated by the dnsmasq procd script into nftset format.

If you compiled dnsmsaq to support both ipset and nftset, there is a high probability someone will configure an Fw4 "Ip Set" at the same time as using a legacy/iptables package that wants to do the dnsmasq standard (eg mwan3). The Fw4 syntax for one of its "Ip Sets" is not the same, and conflicts with, the standard syntax. At best, the Fw4 syntax will prevail, at worst, dnsmasq will crash on startup. (I tried this so have verified in practice).

From the dnsmasq man page ( https://thekelleys.org.uk/dnsmasq/docs/dnsmasq-man.html ):

--ipset=/<domain>[/<domain>...]/<ipset>[,<ipset>...]
    Places the resolved IP addresses of queries for one or more domains in the specified Netfilter IP set. If multiple setnames are given, then the addresses are placed in each of them, subject to the limitations of an IP set (IPv4 addresses cannot be stored in an IPv6 IP set and vice versa). Domains and subdomains are matched in the same way as --address. These IP sets must already exist. See ipset(8) for more details. 
--nftset=/<domain>[/<domain>...]/[(6|4)#[<family>#]<table>#<set>[,[(6|4)#[<family>#]<table>#<set>]...]
    Similar to the --ipset option, but accepts one or more nftables sets to add IP addresses into. These sets must already exist. See nft(8) for more details. The family, table and set are passed directly to the nft. If the spec starts with 4# or 6# then only A or AAAA records respectively are added to the set. Since an nftset can hold only IPv4 or IPv6 addresses, this avoids errors being logged for addresses of the wrong type.

The problem is that Fw4 puts in an "ipset like" option in the uci config for dnsmasq and this gets translated to a native dnsmasq --nftset config.
If a legacy package adds a uci ipset config, it will fail.

[bluewavenet]

@vgaetera

Example testing ipset-dns on OpenWrt 23.05

Excellent - this might be a functional workaround for mwan3.

[bluewavenet]

@vgaetera

It works fine for me, what is your problem?

For a simple "Set of IP addresses" it works fine.

If you have a legacy package that want to do the real "iptables ipset thing", you have a problem.

I don't have a problem as the projects I work on have fully migrated to nftables, but many people do have a problem and it comes up again and again on this forum.