Adblock support thread

Mushoz · August 7, 2019, 4:53pm

I specified the path in Luci under the option Additional Hosts files. I was under the impression that this is the correct field for adding addn-hosts files, correct?

Edit:

It seems it is:

root@OpenWrt:~# cat /var/etc/dnsmasq.conf.cfg01411c
...
addn-hosts=/list.overall
...

shm0 · August 7, 2019, 5:00pm

I'm out of ideas now

//edit
Does it work if you put address=/example.com/ in /etc/dnsmasq.conf ?
//edit2
or
conf-file=/path/to/addn-hosts-file in /etc/dnsmasq.conf

Mushoz · August 7, 2019, 5:25pm

Both of these work Found the same syntax as well, and tried them

The good news:

It turns NXDOMAIN rather than localhost, which is good
It's faster than --server in my benchmarks

The bad news:

Rather than an average of 3 second queries, I am now seeing 1.5 second queries. An improvement, but still not usable. The only usable solution seems to be the host file approach via addn-hosts

Maybe I should dive into the sourcecode of Pi-hole and figure out if they use host files as well, or if they are doing something else. Hmmmm.

shm0 · August 7, 2019, 5:30pm

I wonder why there is such a perfomance difference between server, address and host file.
When you tested the host file method. Which IP did you use 127.0.0.1 or 0.0.0.0?

Mushoz · August 7, 2019, 5:31pm

I tried 0.0.0.0 (and :: for IPv6), but it resolved to 127.0.0.1 and ::1 instead of NXDOMAIN regardless.

shm0 · August 7, 2019, 5:34pm

What happens if use
address=/example.com/0.0.0.0 (or address=/example.com/127.0.0.1)
instead of address=/example.com/ ? Performance-wise?

Some offtopic..
I noticed openwrt's default configs of dnsmasq/unbound comes with edns packet size setting of 1280.
To prevent fragmentation, i guess?
Can someone explain to me why 1280?
Because 1280 is minimum allowed MTU of IPv6?
But then the value should actually be 1232. (1280 - 40 IPv6 Header - 8 UDP Header)
Shouldn't it be left at default 4096 as most resolvers/forwarders come with max udp packet size setting?
For example in the dnsmasq conf.h file
#define SAFE_PKTSZ 1280 /* "go anywhere" UDP packet size */
And unbound has:
max-udp-size:

Mushoz · August 7, 2019, 6:08pm

Not sure. Gedit was crashing when trying to edit the list via find & replace all. I think it won't make a difference, because I am finding multiple sources where they explicitly state that address entries are slow.

Mushoz · August 7, 2019, 6:43pm

From the install script of Pi-Hole (https://raw.githubusercontent.com/pi-hole/pi-hole/master/automated%20install/basic-install.sh):

# Gravity is a very important script as it aggregates all of the domains into a single HOSTS formatted list,
# which is what Pi-hole needs to begin blocking ads

And further in the script we receive confirmation that Pi-Hole does indeed use the host file for blocking domains:

local dnsmasq_pihole_id_string="addn-hosts=/etc/pihole/gravity.list"

So Pi-Hole does seem to be using a host file, and that also explains the massive performance difference between Pi-Hole and Adblock on the router. Furthermore, these seem to be the default files that are downloaded:

appendToListsFile() {
    case $1 in
        StevenBlack  )  echo "https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts" >> "${adlistFile}";;
        MalwareDom   )  echo "https://mirror1.malwaredomains.com/files/justdomains" >> "${adlistFile}";;
        Cameleon     )  echo "http://sysctl.org/cameleon/hosts" >> "${adlistFile}";;
        ZeusTracker  )  echo "https://zeustracker.abuse.ch/blocklist.php?download=domainblocklist" >> "${adlistFile}";;
        DisconTrack  )  echo "https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt" >> "${adlistFile}";;
        DisconAd     )  echo "https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt" >> "${adlistFile}";;
        HostsFile    )  echo "https://hosts-file.net/ad_servers.txt" >> "${adlistFile}";;
    esac
}

As far as I understand, the gravity script then uses these lists to populate the final host file. Source of the gravity script can be found here: https://github.com/pi-hole/pi-hole/blob/master/gravity.sh

dibdot · August 16, 2019, 4:32am

Hi,

I've just pushed a new adblock release 3.8.0 to master (full change log see first post). If you would like to try the new version, you have to update your adblock config as well. Please use the --force-maintainer opkg option for that. After the LuCI package update please reset you LuCI cache with rm -rf /tmp/luci-*

Two major changes are:

add support for 'DNS File Reset', where the final DNS blockfile will be purged after DNS backend loading (save storage space). A small background service will be started to trace/handle dns backend reloads/restarts

In LuCI you'll find this new extra option:

add support for the 'null' blocking variant in dnsmasq (via addn-hosts), which may provide better response times in dnsmasq

In LuCI you'll find this additional select box under the dns backend selection:

Have fun!
Dirk

shm0 · August 16, 2019, 7:37pm

Thanks for the update!
But why was the static blacklist removed from adblock conf?

dibdot · August 16, 2019, 7:47pm

The blacklist is now treated like the whitelist ... always active, there were too many users who unintentionally deactivated this list to wonder afterwards, that entries on the blacklist don't work.

shm0 · August 16, 2019, 9:28pm

Thank you for clearing that up!

ldir · August 17, 2019, 11:37am

I was confused why I couldn't get the DNS reporting to work. Looking in the script it turns out the 'reporting interface' is hard coded to 'br-lan'. Is there better way of changing this to 'eth1' other than hard coding the script?

My APU2 config doesn't use bridge interfaces for lan.

dibdot · August 17, 2019, 2:20pm

Nope, 'br-lan' is just the default ... simply set 'adb_repiface' accordingly ...

ldir · August 17, 2019, 9:16pm

ahh! additional parameters. ooops - thanks

RainGater · August 19, 2019, 12:03pm

Updated to 3.8.0 and looks much better than 3.6.5-2 as it seems to update/refresh the lists faster. Thanks a ton.

Btw, I picked all the available blacklist sources (overall domain count: 185723) and my router is Netgear R7800. Would it cause any performance hit because I chose all the sources? I am not seeing any hit now as the available memory is still 333 MB/466 MB. Not bad and load is not severe yet. I picked both your tweaks: DNS File Reset & 'null' blocking variant in dnsmasq.

dibdot · August 19, 2019, 1:47pm

Thanks for testing & your feedback!

I don't know. In my productive environments I'm using dnsmasq/null and unbound/nxdomain with round 50K domains and the new (still somewhat experimental "DNS File Reset") and see no downgrade performance wise.

WildByDesign · August 19, 2019, 2:46pm

Dirk, would you please be able to explain some more detail regarding this dnsmasq/null (I am assuming referring to addnhosts method) method?

I am not familiar with it, although I have read some of the recent discussions on how it may be faster as far as performance goes. However, I would really like to understand it better before I mess around with my config. I always have two instances of dnsmasq running with your adblock program and I don't want to do anything that would mess up my current setup.

Have you noticed better performance with dnsmasq/null?
How does the error messages in the browser differ between nxdomain and dnsmasq/null?

In my etc/config/dhcp file I have always had to manually add the following line:
option serversfile '/tmp/adb_list.overall'

That was specifically regarding my second dnsmasq instance only, of course.

If I were to switch to dnsmasq/null, for my second dnsmasq instance in etc/config/dhcp file, would I manually need to add a line for addnhosts method?

Thank you for your time, as always. It has been a pleasure testing your program over the past 3 years. I was actually looking through my archive of binaries and packages on my local storage and realized that I have stored every single build since adblock_1.0.0-1_all.ipk, including all of the various adblock2 testing files adblock_1.9.99-pre1-1_all.ipk all the way until these most recent build. Including, of course, all of the luci-app-adblock packages. It is crazy how time goes by so fast.

dibdot · August 20, 2019, 8:33am

The new blocking variant uses an unspecified IP address for blocking ("::" in IPv6 and "0.0.0.0" in IPv4). It must never be assigned to any node and indicates the absence of an address. If you dig for such address, you see this:

; <<>> DiG 9.11.5-P4-5.1+b1-Debian <<>> doubleclick.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 134
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;doubleclick.net.		IN	A

;; ANSWER SECTION:
doubleclick.net.	0	IN	A	0.0.0.0

Effectively the client does not even try to establish a connection for the requested website, so no need for pixel server etc. Both blocking variants are covered by the report engine. Selective whitelisting (you whitelist a subdomain, even if the top level domain is blocked) does not work with the 'null' variant.

No worries, you could simply switch between these dns varaints via LuCI.

I don't think you'll notice it when you surf. But I'm using only quite small block lists and my main router uses unbound and not dnsmasq. However the numbers of @Mushoz prove that the dnsmasq answers via "addn-hosts" are more performant ... just try it on your own.

Yep, just add for this additional dnsmasq instance something like ...

list addnhosts '/tmp/<LISTNAME>'

but there is no need to switch both instances, you can mix both variants (nxdomain/null).

Yep, that's really funny ... here's the first changelog from my private git ...

BTW, just pushed adblock 3.8.1 to master, you should start with that version cause I've fixed some 'null' related things with this update (see first post).

Thanks!

RainGater · August 20, 2019, 11:22am

I don't see 3.8.1 in the snapshot branch. I see only adblock_3.8.0-1_all, same as the previous one. Is this the correct one?

Also, when going from 3.8.0-1, do we still use --force-maintainer option or not needed?

opkg install adblock_3.8.0-1_all.ipk
opkg install luci-app-adblock_git-19.230.62046-276aa89-1_all.ipk

What about cache (rm -rf /tmp/luci-*), is it required?