23.05 dnsmasq, ipsets and mwan3 incompatibility?

Add to the wiki now.

Just wondered if you considered moving this into a gist? You can embed gists on the community forum platform along with the fact for easy installation, this can get used as a wget target to automatically download and place on an OpenWrt router?

Thanks for updating the Wiki. I see the update toward the top in reference to 23.05. It might be a good idea to update the text in the ipset section as well.

As already noted on the Github Issue there is now a Gist also available:

Installation with it is now as easy as:

wget -O/etc/init.d/nft2ipset https://gist.github.com/Kishi85/b7f379f9aa19f4878af28b8e1a8887ab/raw/
chmod +x /etc/init.d/nft2ipset
service nft2ipset enable
service nft2ipset start
1 Like

@jmccabe06 Thanks for spotting that. Also added in the warning banner as well alongside the gist update.

@Kishi Gist version now published on the wiki as well.

Thank you so much. Just implemented today.
It works for me, finally!!
Just noticed that it is necessary to let the ipset family to ipv4+ipv6 otherwise, if set as ipv4, the ipset is ignored. There is a note here from jmccabe06
It is also not super obvious ipset must be declared both in dnsmasq settings (in luci DHCP & DNS > General tab) and in ipset settings section (in luci DHCP & DNS > IP Sets tab), in addition to firewall advanced settings.
...and I have no idea why... :rofl:

For me it works by properly by:

  1. Define the nftables sets under "Firewall->IP Sets" first. Important: Use the correct family (IPv4 or IPv6) there, match on dest_ip and define separate sets for IPv6 and IPv4 if necessary. Adding a timeout (I personally use 600s) helps clearing old entries out of the ipsets automatically.
  2. Then add those sets to the DNSMASQ set resolving under "DHCP & DNS -> IP Sets" (Note: add both the IPv4 and the IPv6 set to the IP set option of the element as necessary. Multiple nftables sets are possible to be specified for each group.)
  3. Finally add them to mwan3 rules. I use specific, separate rules for IPv4/IPv6, but IPv4+IPv6 works as well because it'll match the family if the ipset anyway (which is IPv4 by default if not explicitly defined under "Firewall -> IPsets" as it is the default family for ipset).

Nothing else should be required (it's not for me at least). Most people probably forget to do step 1 and just let dnsmasq define nftables sets with defaults (IPv4 without timeout) hence things not working properly with IPv6.

I'll add this additional info to the Wiki as well for clarity.

As I'm upgrading from 21.02 to 23.05 and need to reconfigure everything, I noted that the IPset extra will create everything.
I added the "DHCP & DNS -> IP Sets" and executed "ipset setup". After that the IPsets also appeared in "Firewall -> IPsets". Everytime "ipset setup" is executed (like running as cronjob or with hotplug), it will overwrite the configuration (so changes in "Firewall -> IPsets" will be lost).

I had to change 1 line in the script (IPset extra is located under /etc/profile.d/ipset.sh) to get it work in my setup:

Changed line in /etc/profile.d/ipset.sh from:

set firewall.'${IPSET_NAME//-/_}'.match='net'

to:

set firewall.'${IPSET_NAME//-/_}'.match='dest_ip'

Another observation:
I had different IPset ending with something like "....ipset" (e.g. "TestOpenipset"). They will be visible in "Firewall -> IPsets" and looks rigth incl. the name, but they are not added/handled properly and name is cut after "ip":

#ipset list
...
Name: TestOpenip
Type: hash:ip
Revision: 5
Header: family inet hashsize 1024 maxelem 65536 bucketsize 12 initval 0xe96198ed
Size in memory: 112
References: 0
Number of entries: 0
Members:

Those also didn't appear in IPset under mwan3.
After adding the same list removing the word "ipset" in the name (e.g. "TestOpen") and executed "ipset setup", everything was fine.

Not sure if I'm getting something wrong here or if in the meanwhile there is a better way to populate the lists.
DNS is handled by an PiHole within the network and so DNS request will not reach the router (and it's not planned to change this :slight_smile: ).

Apologies for the call for assistance but I am missing how to successfully complete the use of mwan3 & the script.
I believe I followed the steps correct and not sure what else to try. I am starting to make a lot of changes and none seem to be the right one.

Configuration:

  • WAN & wg0_proton interfaces
  • LAN 192.168.10.0/24

My desired end state:

  • PBR of select clients & subnets over the vpn while allowing most of the lan devices to prefer the wan interface.
    Current state:
  • all clients utilizing the wan. Not being routed over the vpn.
  • vpn is working and from the router cli I can ping over the vpn as shown
root@OpenWrt:~# ping -I wg0_proton aa.com -c 3
PING aa.com (23.59.182.130): 56 data bytes
64 bytes from 23.59.182.130: seq=0 ttl=55 time=222.792 ms
64 bytes from 23.59.182.130: seq=1 ttl=55 time=226.408 ms
64 bytes from 23.59.182.130: seq=2 ttl=55 time=231.638 ms

--- aa.com ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 222.792/226.946/231.638 ms
root@OpenWrt:~# ping aa.com -c 3
PING aa.com (23.59.182.130): 56 data bytes
64 bytes from 23.59.182.130: seq=0 ttl=53 time=9.693 ms
64 bytes from 23.59.182.130: seq=1 ttl=53 time=10.013 ms
64 bytes from 23.59.182.130: seq=2 ttl=53 time=9.792 ms

I would greatly appreciate some guidance on what to try/validate if you guys have the time.
The part I am most fuzzy on is step #2 here. I do not understand what or why I need to put fqdn's for lan clients I want routed over the vpn for everything. But am open to clarity on step #1 as well if thats where my issue is.

Usage:

  1. Define the nftables sets in LuCI “Firewall → IP Sets” first. Use the correct family (IPv4 or IPv6) there, match on dest_ip and define separate sets for IPv4 and IPv6 if necessary. Adding a timeout helps clearing old entries out of the ipsets automatically.

  2. Then add those sets to dnsmasq resolving under “DHCP & DNS → IP Sets” (Note: add both the IPv4 and the IPv6 set to the IP set option of the element as necessary. (Multiple nftables sets are possible to be specified for each group).

  3. Finally add them to mwan3 rules. I use specific, separate rules for IPv4/IPv6, but IPv4+IPv6 works as well because it'll match the family if the ipset anyway (which is IPv4 by default if not explicitly defined under “Firewall → IPsets” as it is the default family for ipset).

Thank you in advance

Have you tried the other IP Sets under General Tab in Luci?

Hi and thank you for the response utop.
I eventually gave up on mwan3 and went with the functional PBR. It works for my current need even though I was wanting to get mwan3 going again after the code update.
Appreciate it. Maybe in the future there will be an update that allows it to work out of the box again but if not, I may take another stab in some future date.

1 Like

I couldn't download the gist script by WGET. I spent a lot of time going over it. Finally, I uninstalled Mwan3 so I could download it ::

Guys, I have been messing around with OpenWrt and networking for over 10 years and I straight up don't understand how we're supposed to configure MWAN properly here:

  1. Define the nftables sets in LuCI “Firewall → IP Sets” first. Use the correct family (IPv4 or IPv6) there, match on dest_ip and define separate sets for IPv4 and IPv6 if necessary. Adding a timeout helps clearing old entries out of the ipsets automatically.

Define what, exactly? There are two rules that pre-exist for me - MWAN-IPSetv4 and MWAN-IPSetv6. What am I entering here? This is my LAN subnet - I don't understand what this has to do with WAN or MWAN config.

https://i.imgur.com/hXWpfUf.jpeg

  1. Then add those sets to dnsmasq resolving under “DHCP & DNS → IP Sets” (Note: add both the IPv4 and the IPv6 set to the IP set option of the element as necessary. (Multiple nftables sets are possible to be specified for each group).

Once again I am confused. Are we asking OpenWrt to look at local subnets for DNS? Why? Also what is with FQDNs - I don't understand how domain names have anything to do with this. Am I supposed to be defining the local FQDN, which in my case is ".lan"?

Uploading: image.png…

  1. Finally add them to mwan3 rules. I use specific, separate rules for IPv4/IPv6, but IPv4+IPv6 works as well because it'll match the family if the ipset anyway (which is IPv4 by default if not explicitly defined under “Firewall → IPsets” as it is the default family for ipset).

I suppose this section makes more sense once we understand what we're doing with steps 1 and 2, but some sort of additional information with examples would be highly appreciated.