Dnsmasq-full, ipset support removed (in 23.05 and master)

And BanIP has already left iptables is moving on to nftables right now.

There are others too, for example opennds and nodogsplash and possibly coovachilli, where populating ipsets is an option that provides additional functionality.

So is opennds and I heard mwan3 was being worked on.

The problem is that outside OpenWrt, iptables et al seems to be universally transparently supported, leaving little incentive for package developers to convert.

The issue I have is that dropping ipset support from dnsmasq-full was not announced in any way likely to be seen. I only found out after many hours of debugging trying to determine why ipsets had stopped working.

Looking at the commit linked above by @hnyman it seems that proper ipset support in dnsmasq was dropped as a consequence of keeping compatibility with legacy fw3 ipset config in the new fw4.
Now not withstanding that this is generally a "good thing" from the typical inexpert user point of view, it has the consequences we see here. In my opinion, the change should have been to introduce "nftsets" into the config, allowing ipset support to be left enabled in dnsmasq-full.
Now we are left with the confusing apparent compatibility of fw4 with genuine ipsets, where in fact it uses nftsets internally. But this is just my opinion - in the long term, it probably does not matter.

Perhaps we also need an interim dnsmasq-completely-full package version :wink:

There is always the ipset-dns package. It has fallen into the unmaintained category since the addition of ipsets into dnasmasq-full way back in ~2017, but I wonder if it still works.....

Not sure if they are in the same boat, pbr can operate in nft/nft set or iptables/ipset mode, so it can provide same functionality without ipset.

I argued for the same: https://github.com/openwrt/openwrt/pull/11073#issuecomment-1291196588, you can also see the response below.

Mmm. Not invented here syndrome?

I wrote another paragraph but thought better of it and deleted it again. :scream: :speak_no_evil:

For crowdsec-firewall-bouncer a new version is coming (PR just accepted) which does not use ipset any more.

1 Like

Help me understand this. Consider I'm a noob here, please

I am using OpenWRT with dnsmaq-full and MWAN3 in my setup. the purpose I use MWAN3 is that I have two WAN connections that I want to route traffic based on two basic rules. I am using IPSets

I have a 4G WAN connection and I only want to route social media traffic on this connection whilst I route everything else on my Fibre link.

this has been working fine for several months now. But two weeks back I updated OWRT to the latest version and after reading this thread, I am confused about whether MWAN3 is properly working or not. By the looks of it it does, Some traffic flows through the 4G connection. But I am worried if IPSETS isn't supported anymore, what actions should I take?

really appreciate your support, advice

on this

TL;DR If you were using ipsets populated by Dnsmasq:

  1. On OpenWrt, it will no longer work.
  2. If you are on any other flavour of Linux it will continue to work just fine.

It should work with nftables too.

1 Like

oh noo

so if i revert back to the previous version of OWRT should it work?

The point here is that the ability of dnasmasq to automatically populate an ipset has been removed from dnsmasq-full (only here in OpenWrt and not other flavours of Linux).

So unfortunately auto-populating an ipset by dnsmasq does not now work at all.
Any packages that use this functionality will have problems, possibly even causing them to fail.

This was done apparently to enable a simple "emulation" of the old ipset support from fw3 in fw4, allowing old fw3 ipset configs to be used unchanged in fw4, with no consideration for packages that use dnsmasq to populate their own ipsets.

There is an argument that all packages that use iptables should be immediately migrated to nftables or become defunct, but that is outside the scope of this thread.....

1 Like

Thank you for this thread. I've been working on an x64 setup of openwrt to replace pfsense for a week. This is the last piece to match feature for feature for my setup and I as banging my head against a wall on this.

I'm hoping this is resolved soon. I can't wait to move openwrt from the lab to production.

The problem here isn't so much that this has happened. The real problem is that this happens without any discussion with stakeholders. The change just happened, boom, no open discussion, no consultation as to whether this will hurt users. Decision made, change made, no announcement, just suddenly broken packages that users have to debug. This isn't the way an open source community is supposed to work.

This is beyond the fact that nftrables, yes ten years old that it may be, still isn't prime-time ready and the major distributions have every compatibility layer still in place for very, very good reasons.

1 Like

There is no problem with migrating from iptables to nftables.
All the compatability tools exist to help.

But the decision was made to deliberately break those tools by default, ie

  1. Introducing the nasty hack of iptables-zz-legacy as the default instead of iptables-nft
  2. Deliberately breaking ipset auto-population with the excuse that "the word ipset just means a set of ip addresses".
  3. There are other changes along the same lines but I will not mention them here for now.

These things were done apparently ignoring the certainty that some packages would be broken.

It could be argued that this would force effected packages to be updated and that could be a good thing, but I have not found any discussion of this anywhere.

The main casualty of this change is likely the mwan3 package, because of the ipset functionality. Under 22.03 the iptables-nft compatibility layer allowed things to still work, but the ipset support in dnsmasq-full was critical for it.

It is a bit of a catch 22 scenario because mwan3 does not currently natively support nftables, meaning it can't support nfset either: https://github.com/openwrt/packages/issues/20582.

Without turning this into a mwan3 specific thread, there's more discussion for mwan3 over here: 23.05 dnsmasq, ipsets and mwan3 incompatibility?

While the decision to update the default compile options has caused users issues, you can probably understand the rationale when firewall4 has been default in two major releases and the announcement of the move from iptables to nftables was made in advance. The fact that the quick scanning from @stangri suggests that only a couple of packages directly reference ipset dependencies is probably the reason why.

For the remaining packages left, the efforts would be to try and get them updated to support nftables natively as soon as possible. Easier said than done, but it is ultimately the solution now given the decision made, if you want to continue using dnsmasq in this way.

It would have perhaps been a smoother transition if either, the dnsmasq-full package has been compiled with both or a nftables and ipset version of dnsmasq-full was provided separately so packages could reference those dependencies, but I'd imagine the binary size, maintainer headache and potential conflicts around the latter probably are the reasons why it wasn't done.

1 Like

Ipset had to be left out of the compilation of dnsmasq because of the decision to do a pseudo emulation of ipset in fw4 to prevent "people being confused". The OpenWrt version of dnsmasq is now unique in that it uses "ipset" config to populate an nft set created by fw4.

It seems that during the development of fw4:

  1. Sight was lost of the fact that nftables is the firewall and fw4 (and its predecessors) is only a static firewall configuration tool.
  2. The user base was forced to the "lowest common denominator", to protect users with little knowledge from having to learn anything new.

Before a truly "full" version of dnsmasq could be provided, we are now in a situation where fw4 would have to be fixed.

It is possible to think up workarounds for the ipset problem, indeed this was done in the openNDS package but the other way round for 22.03 ie openNDS was migrated to nftables, but dnsmasq-full did not include nftables support.

But as far as mwan3 and any other packages still using iptables are concerned, either they will eventually be dropped as "not working/no longer maintained", or will be migrated by someone.

The mwan3 package is planned to be migrated to nftables, given the apparent ipset breakage from the recently released 23.05 series, this may likely speed up that timeline now. It had been using iptables-nft since 22.03. This has been highlighted in the documentation for now. The issue will be some of the very much iptables specific code that was never built with nftables in mind (going back years) that would need to be ported and then retested given the regression areas and the many different configurations and network interfaces used by users.

Porting an iptables package to native nftables is fairly straightforward once you have passed the initial nftables learning curve. Having ported a quite complex iptables package to nftables I feel qualified to make this statement :wink:

It all takes significant time though when you include all the necessary testing, but it is not such an onerous task......