Dnsmasq (full) and firewall4 - using ipset or nftables together - no out-of-the-box solution in OpenWrt 22.03.03

Installing and Using OpenWrt Network and Wireless Configuration
21

Incremental IP set updates can provide some sort of benefit when combined with TTL per element, otherwise it is likely more performance efficient to put all elements to a file to load in bulk and make possible to restart the firewall service without losing IP sets, furthermore we cannot neglect IPv6 in general case.

On the other hand, there are services that can rely on dozens of IPv4 and hundreds of IPv6 addresses, and you can imagine how many times you need to run the script to exhaust such pools, when they return a limited set of random IPs on each request, so it is more reasonable to block such domains by ASN.

22

sure. but as per below :wink:

NOTE: these are high level, poorly tested (if at all) ideas. 
does not mean they are optimal, surely there are rough edges

adding ipv6 is easy, option timeout is also possible, and 5mins schedule can be customized of course too.

23

Yes, but simply processing IPv6 similar to IPv4 doubles the number of the resolveip and nft invocations on average, so more performance optimization would be necessary, and most users cannot optimize the code by themselves, possibly leading to negative user experience.

Then we will have the same problem as explained in a recent thread as we cannot guarantee that updating each N minutes is sufficient frequency, since upstream resolvers may return different IPs even more often and this opens a loophole.

And then how to determine that the pool of IPs is exhausted and we no longer need to flood the network with the same DNS traffic again and again every N minutes, and can finally decrease the frequency of DNS queries, specifically when using metered connections.

No offense, I just want to improve our common understanding of the problem. :slightly_smiling_face:

24

I managed to make a script to overcome that issue by translating nftsets into ipsets and flushing refilling then each time the firewall get restarted. That way if you define a static ipset in /etc/config/firewall it becomes also usable in /etc/config/mwan3

just create a file as /usr/local/sbin/nftset_to_ipset with the following content:

#!/bin/sh
for ipv in 4 6
do
    NFTSETS_LIST="$(nft -j list sets| jq -r '.nftables[].set | select(type=="object" and .type=="ipv'"${ipv}"'_addr") | .name')"
    if [ "$NFTSETS_LIST" != "" ]; then
        for one_nftset in "${NFTSETS_LIST}"
        do
            if ipset list --name | grep -q -E "^${one_nftset}\$"; then
                ipset flush ${one_nftset}
            else
                ipset create ${one_nftset} hash:net family inet${ipv/4/}
            fi
            for one_net in $(nft -j list ruleset | jq -c '.nftables[].set | select(.name=="'${one_nftset}'") | .elem[]' | sed 's#\({"prefix":{"addr":\)\?"\(.*
\)"\(,"len":\(.*\)}}\)\?#\2/\4#g' | sed 's#/$##')
            do
                ipset add ${one_nftset} ${one_net} 
            done
        done
    fi
done

then add the following section at the end of /etc/config/firewall

config include
	option	enabled		1
	option	type		script
	option	path		/usr/local/sbin/nftset_to_ipset

you can then add ipsets the way it is suppose to be, for example:

config ipset
	option name test1
	option enabled 1
	option family ipv4
	option match dest_net
        list   entry 133.15.80.160
        list   entry 117.15.80.248/29

config ipset
	option name test2
	option enabled 1
	option family ipv6
	option match dest_net
        list entry 2001:DB8:5002:AB41::801
        list entry fe80::/10

and use them in /etc/config/mwan3

config rule 'test_1'
    option ipset 'test1'
    option family 'ipv4'
    option use_policy 'wan1_wan2'

config rule 'test_2'
    option ipset 'test2'
    option family 'ipv6'
    option use_policy 'wan2_wan1'

note that the script covers only basic ipsets of ips and or nets and not the more complex case combining ips and ports for instance...

1 Like
25

that's not true. resolveip answers with both type of ip, it is matter of checking if an answered ip is ipv4 or ipv6 (includes ':' or '.') then use respective set.

totally true but how does link guarantee that? am afraid it is neither.

the only way to be real-time if you resolve the IPs via DNS real time then DNS resolver add to ipset. this piece is missing. and we are at square one again.

no offense is taken as i explicitly stated my "solution" is not an out-of-box package you just run, more of an idea -- which also explicitly stated not the "best and only" :slight_smile:

1 Like
26

I agree about resolveip, assuming that its output is processed appropriately.

Yes, and this is one of the reasons the script supports both DNS and ASN entries, which helps to efficiently process normal and transient domains, but the drawback is that the user needs to chose the correct method.

Correct, however pre-populating IP sets can also be important, e.g. for clients which use own hosts file to resolve domains, or can bypass the local DNS with some other method.

27

Hi @vgaetera ,
made some progress, can paste multi line commands, understand some more things .. but cant get it working currently.

step by step:

(First try, tried to follow fw3 document, found out later your code does it all in one:)

Following instructions of the fw 3 doc ( https://openwrt.org/docs/guide-user/firewall/fw3_configurations/dns_ipset#extras )
I have success to create the IP Sets and the firewall rule, like discribed there.
I can see the results in etc/config/dhcp and etc/config/firewall
Only thing I do not understand - IP Set in dhcp file are named "filter" and "filter6".
In firewall rule, the IP Sets are named "filter dest" and "filter 6 dest"

When I execute the specific code you postet, it runs through without any suspicious warnings / error.
What I see:

  • everthing around
/etc/hotplug.d/online/70-ipset-setup
/etc/hotplug.d/iface/90-online
/etc/hotplug.d/online/10-sleep
/etc/ nftables.d/ipset-setup.sh 
/etc/profile.d/ipset.sh ..

seems to work well .. as far I can read the code, understand what is going on, interpret (no) warnings / errors in the terminal

  • command "nft list ruleset" workz - but i cant find out if the IP sets are there, if there is anything in them
  • currently i have no idea how to check if resolveip is working, if it is populating IP Sets (nftables?), or to find out why the whole thing is not woking as disired @ current state.

(end 1st try)

(2nd try)

  • I tried to understand what work your code does, I did re-build the dns_ipset.sh script. nice feature / work although i still do not understand details. what i understand that I did not need to execute the commands in the fw3 doc, you do it in your code anyway, 3rd code download. as the ipset part and the hotplug part need resolveip installed ( i think ) wouldnt it be better to have the firwall part done first in dns-ipset.sh and then ipset / hotplug?

So I think I see that the major magic happens in the IPset script. There resolveip gets its instructions / IPs are colected, and stored in nftables. right?

Summary: I executed all comands, do understand better what might going on. currently only idea I have why it is not working are the differnt IP Set names in dhcp config file and firewall (rule) file.

what i really would LOVE - having the chance to see the IP Set / nftable populated with the IPs that resolveip found out talking to .. some dns and ripe.net (I think?) . command "nft list ruleset" is not readable for me

I think we are pretty close to the solution .. and I would love to document it for other noobs like me. With this example one does get so much more knowlege how things are managed in openWRT / the openWRT documetation etc etc.

Any help / hint very appreciated

28

You can pipe long output to a pager, list a specific set, and compare the output:

nft list ruleset | less
nft list sets
nft list set inet fw4 filter
nft list set inet fw4 filter6
resolveip example.org

We want all dependencies to be in the right place before running ipset setup.

IP sets are stored in /var/ipset-* and fw4 feeds them to nftables.

The word dest is not part of the IP set name, but a direction matching criterion to apply the filtering rules to the matching destinations.

1 Like
29

Fast reply on regarding the "order" of dependencies in your script dns-ipst.sh:

The script reports errors if resolveip is not installed before running the script
On the other hand, line 5 / downloaded code snippet 3 installs resolveip.
Therefore i asked if the order of the snippets should be different. There might be other dependencies I am not aware of.

Trying to work with your other infos / hints. thx 4 them !!

30

SUCCESS :slight_smile:
this one works !!!

All credits go to @vgaetera , 100 Kudos + thx

This documented demo solution is able to
filter / block traffic from "lan" to "wan"
based on domain names ( here: example.com and example.net )
in openWRT 22.03.03 standard installation
without use of non-released packages, and pretty easy to handle / achive results.

tl:dr

  • Install resolveip package
  • execute @vgaetera commands as one block in Terminal / Command line (e.g. Putty)
URL="https://openwrt.org/_export/code/docs/guide-user"
cat << EOF > dns-ipset.sh
$(uclient-fetch -O - "${URL}/advanced/ipset_extras?codeblock=0")
$(uclient-fetch -O - "${URL}/advanced/hotplug_extras?codeblock=0")
$(uclient-fetch -O - "${URL}/firewall/\
fw3_configurations/dns_ipset?codeblock=0")
EOF
sh dns-ipset.sh
  • reboot
  • www.example.com and www.example.net are blocked
  • block reason: fw4 rule, based on IP Sets stored in nftable with domain name input from \etc\config\dhcp

Motivation:
dnsmasq below ver 2.87 is incompatible with nftable used in fw4 and the iplist package. Approachs, that worked in earlier versions of openWRT do not work in openWRT 22.03.03 (which uses dnsmasq 2.86 )

Solution Description & Documentation:

  • Definition of domains to be used / filtered ( here: example.com , example.net can still be done in dnsmasq / LuCI / DNS&DHCP / IP Sets or in \etc\config\dhcp
  • Two standard fw4 rules are created to filter the traffic from lan to wan for domain "example.com" and "example.net". In this show-case (=code base), the rule names are "Filter-IPset-DNS-Forward" , one for IP4, one for IP6. The two rules are created automatically. I do edit them not in LuCi as LuCI does not show all lines of the rule. I use WinSCP to go to etc/config/firewall file and edit it there.
  • In the background, the @vgaetera command line code (see tl:dr) that produces and executes the script "dns-ipset.sh" (that is put together "live" from code snippets from three openWRT doc sites (real magic, see also ( here document)) does the following things:
    ** The used Ip Sets (named " filter" for IP4 and filter6 for IP6 in this use-case) act as an "envelope" for IPs. It seems the IPs are stored in files (see point 4 further down) and fetched by fw4 to store them in nftabels. Nftables are important base technolgy of fw4 and fw4 is able to work with IP Sets that are based on nftables.
    ** resolveip populates the IP Sets . So for noobs like me .. "example.com" is translated into one or many IPs like 1.2.3.4 , something a firewall can work with. Work is done in the code snippet Ipset_extras
    ** hotplug is used @ startup of router to run a script that starts the process of populating IPs into the IP Sets (with resolveip package).
    ** IP Set updates : The IP sets are populated at startup with Hotplug extras when the system becomes online and then updated on a schedule. Here crontag shows 0 */3 * * * . /etc/nftables.d/ipset-setup.sh - with my limited know how i think this might mean update of the IP Set all 20 minutes around the clock.

Tips, hints, deeper dives to understand better whats going on:

You can download / look at the three code snippets @vgaetera is using in his srcipt dns-ipset.sh here:

https://openwrt.org/_export/code/docs/guide-user/advanced/ipset_extras?codeblock=0
https://openwrt.org/_export/code/docs/guide-user/advanced/hotplug_extras?codeblock=0
https://openwrt.org/_export/code/docs/guide-user/firewall/fw3_configurations/dns_ipset?codeblock=0

If you - like me - are not able to paste multiple lines into a Terminal / Commandline window (me: Putty), try to put the commands in () like this and copy / paste the whole stuff in @ once:

(
URL="https://openwrt.org/_export/code/docs/guide-user"
cat << EOF > dns-ipset.sh
$(uclient-fetch -O - "${URL}/advanced/ipset_extras?codeblock=0")
$(uclient-fetch -O - "${URL}/advanced/hotplug_extras?codeblock=0")
$(uclient-fetch -O - "${URL}/firewall/\
fw3_configurations/dns_ipset?codeblock=0")
EOF
sh dns-ipset.sh
)

If you want to (cross)check if the IP Sets are existing and / or populated with IPs, try some of this commands:

nft list sets
nft list set inet fw4 filter
nft list set inet fw4 filter6
resolveip example.org

The used IP sets are viewable / stored in /var/ipset-* . fw4 takes them from here and feeds them to nftables. in this use case, the files are ipset-filter and ipset-filter6 and they are nicly filled with IPs :slight_smile:

As stated @ the very beginning of this post, this solution works for the real (!) domains (no placeholders !!) www.example.com and www.example.net . It also works for my use case i wanted to solve. I thought it did not work for a big austian news website derstandard.at, but learned that it redirects to www.derstandard.at - you need to block this one. :))

Pro Tip / Extra - ASN based filtering in fw4
If blocking or accepting traffic / content based on domain names like www.wikipedia.org does not block or accept the right (enough) IPs, the solution might be working with ASNs instead of domain names. Lots of Info regarding ASN can also be found here - ASN - more info. openWRT fw4 / @vgaetera 's solution can also handle ASN based filtering.
If you want to filter a specific AS, enter this commands in a Terminal (e.g. putty) window to add AS 2906 to the ipset "filter":

uci add_list dhcp.filter.asn="2906"
uci commit dhcp
ipset setup

The ASN is added / can be edited in the "dhcp" file in \etc\config and is added to the IP Sets that may already be there:

config ipset 'filter'
	list name 'filter'
	list name 'filter6'
        list domain 'example.com'
	list domain 'example.net'
        list asn '2906'

and is treated similar a domain name in the ipset 'filter' (the one we always use in this example / code base). Very easy, very consistent, great stuff :slight_smile:

You maybe find ASNs for your network / domain using a Linux Terminal Window using the methology described in the next code window, wikipedia example. Code Example also delivers IPs belonging to the AS as extra:

$ dig +short www.wikipedia.org. a
91.198.174.192
$ whois -h whois.radb.net 91.198.174.192 |grep ^origin
origin:         AS43821
$ whois -h whois.radb.net \!gAS43821
A31
185.15.56.0/22 91.198.174.0/24

WARNING: Blocking traffic based on ASNs can lead to unwanted results. You maybe block a CDN like Cloudflare or Microsoft with just blocking ONE ASN.

As stated before, I am just a noob, that can not write one line of code. Just doing some documentation of the stuff @vgaetera explained to me, putting all the stuff together in one Post in this thread. All credits to him and sorry if some things are not exact .. my fault / did not understand better ;).
So marking my own post as solution is not fair, but I have no better idea to compile all infos @ one place and link it as solution. One again, thx 2 @vgaetera for solution design and helping me to understand / implement.

1 Like
31

The browser is likely redirected to www.derstandard.at, which is served by Akamai, see:

nslookup www.derstandard.at

So, you need to block specifically the www subdomain, or make the browser stop using this redirect, which may be problematic if it's already visited/cached/bookmarked/etc.

Efficiently blocking CDNs such as this one requires to specify ASN and can affect other services.

1 Like
32

thx 4 clarification!

just wanted to deploy in production,
got an unexpected error during script run:
'/etc/ipset/test.txt' , file not there / can not be read etc.

I wonder if this has to do with this line in the fw part (3) of the script that is still part of dns-ipset.sh.

ipset setup

In my production, ipset package is not installed.
I assume this error can be ignored, just wanted to let u know

33

The code from the wiki does not use this path, thus it should be part of your local configuration.
If you have customized the code, make sure the changes are consistent across firewall configuration and all involved scripts.

34

strange. but
when i tested i took a "blank" 22.03.03, worked like charm

production .. i thougth i had never installed ipset / similar, but there must be some leftover from something. for sure no customizing of the script.

currently in production this happens:

root@OpenWrt:~# ipset
-ash: ipset: not found
root@OpenWrt:~# ipset setup
Unable to load file '/etc/ipset/test.txt' for set 'test': No such file or directory

no idea what is going on .. cant explain.

35 
grep -r -e test.txt /etc

grep(1) — Linux manual page

1 Like
36

root@OpenWrt:~# grep -r -e test.txt /etc
/etc/config/firewall: option loadfile '/etc/ipset/test.txt'
grep: /etc/localtime: No such file or directory
grep: /etc/ppp/resolv.conf: No such file or directory

found it
a leftover from a "config ipset" , when i tried to solve the problem.

thx again!

1 Like
37

Hello @vgaetera ,
I start to begin to understand ASN concept, thx 4 the link, + consequence if you go for it / block it. You say the script supports it, but the user needs to decide to use it or not.

Please could you give me a hint where this user decission needs to be set (in the script?) Is it det globaly for all domains in the ip set or for each domain?

In my case, where i 'accept' a domain and block all others, ASN could help. My target is latidoapp.at , looks like ASN

Thx!

38

You need to manually specify the ASN that you want to block:

uci add_list dhcp.filter.asn="2906"
uci add_list dhcp.filter.asn="40027"
uci commit dhcp
ipset setup

In theory, each IP can lead to a different ASN, but it can also be the same.
However, be aware that blocking Cloudflare is likely not a good idea.

39

Thx again @vgaetera .
I do not plan to edit more, I put all stuff together in this section.
Solution Summary

Final Edit was adding the ASN Section. If I got something wrong with the ASNs, please let me know.

I really learned so much & have the feeling this is a very professional solution. Would really like to send some beer / coffee as I appreciate your help really much - please give me a final hint how I can achive this final goal.

All the best, mopsza

1 Like
41

Thx 4 fredback! Did you succeed? Happy filtering based on domain names!