Adblock-lean: set up adblock using dnsmasq blocklist

Community Builds, Projects & Packages
1163

Hi all, there have been a couple of additional updates to adblock-lean, with these main changes:

  • Consolidated processing of local and downloaded lists (this is mainly about improving code structure and readability. you may experience a very minor speedup but otherwise functionally everything is same as before)
  • Some improvements to console output (mainly more laconic messages)
  • Implemented session log which is now written to /var/log/abl_session.log, and update log which is written to /var/log/abl_update.log (update refers to application update. blocklist updates are logged in the session log). Previous session log is available at /var/log/abl_session.log.0
  • New command service adblock-lean print_log will retrieve the most recent session log if it's available

/var/log/ is mounted on the ramdisk, so as usual, during regular operation adblock-lean doesn't write anything to the flash storage.

Custom script users: note that now the most recent session log is passed in the first argument to the report_success, report_failure functions. Please update your custom scrpits with these changes in mind.

As of this time, the example custom script in the README has not been updated yet.
Edit: README has the updated example custom script now.

Cheers!

4 Likes
1164

I've recently installed my BananaPi R3 and was considering the different adblock options available for OpenWRT and it seems like adblock-lean is the most fitting for my usecase... except a couple of nits:

  1. Why is config stored under /root instead of /etc/ as would be expected by the FHS?
  2. Are there plans to "upstream" it as a package on OpenWrt so that it can be installed via LuCi under "Software" like the other Ad blocking options instead of the current custom install process?
1 Like
1165

Probably we will move config to /etc and set up as a package sometime. As a simple service script with no dependencies the package aspect is no biggie.

3 Likes
1166

Over in the adblock-lean optimisation thread we are busy wondering about whether to:

  • switch over from dnsmasq format blocklists to raw blocklists; or
  • keep support for both.

Hagezi supports both formats for the most part - e.g. for the PRO list:

https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/pro.txt
https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro.txt

I wonder what kinds of blocklists users are using predominantly and whether anyone would care if we switched over to raw blocklists and dropped support for dnsmasq format blocklists?

2 Likes
1167

Just to complement what @Lynx wrote above, the underlying reason for the switch is improved performance and lower memory consumption.

1 Like
1168

The only roadblock I can see is if anyone is using a Blocklist that has a Dnsmasq format, but not a corresponding raw list. But looking through several of the main list providers, I see the all offer both.

I vote just support raw lists and move to that personally. But more feedback from other users would be much appreciated here. Please keep in mind this is a simple change, only a URL update to point to raw lists. Nothing else required.

This is real, especially the speed increase. Tried and tested in one of @antonk branches.

2 Likes
1169

Although it is not important to the primary issue of blocklist format, it would be helpful if a brief description of " improved performance and lower memory consumption" is included here for those of us uninterested in reading through dozens of posts, or more, in the other thread. ( I gave up after a few minutes )

Is it performance of just startup/list processing time or does it improve name resolution times or both or something else?

Thanks.

1170
  1. Performance improvement applies to the time it takes to download+process lists, and possibly there will be a minor improvement to the time it takes to restart dnsmasq (applying previously downloaded and processed lists). DNS performance will be unaffected.
  2. Memory consumption will be reduced in all scenarios. This reduction may or may not be significant, depending on the memory capacity of the system. We also don't have exact numbers yet. It is possible that the transient memory use when processing and loading the lists will be reduced somewhat more than the subsequent reduction, which (stressing the if word) if true, may allow memory-constrained devices to load larger lists than was previously possible. Again, the exact numbers are yet TBD but there will be an improvement when switching to raw lists, so we are very likely to make this change.

And as @Wizballs noted, the change for the users should be pretty easy. We will also look into implementing automatic URL replacement for the commonly used lists.

Now if you would like to help, we are still interested in community input:

2 Likes
1171

Thanks for the detailed information. It is very clear to me now. I hope it helps anyone else that has a similar question.

I was reluctant to provide my take on blocklist usage as my router has some surplus ram and processing power making the use of adblock-lean non-essential. I am not currently using it but plan to return to it once this current burst of development quiets down.

I am using the oisd big list with my own simple script to fetch the list and restart dnsmasq. I run it manually a few times a month.

I am in favor of improving performance on marginal devices and making adblock-lean accessible to as many devices as possible. Looking at the raw vs dnsmasq format, it looks like most/any user provided lists in use now could be pre-processed by a user provided script or adapted to work with your proposed change.

With current information available to me now, I say move forward with the switch to raw format.

Thanks to all for the work put into this!

2 Likes
1172

This is still valuable info.

Also if you've been following the development a bit, you might have noticed that this project has been changing lately, so now the description 'super-simple' doesn't fit as much anymore. In the current state, I would describe it as super-efficient and very reliable. So in my view, it is now as good for the more powerful devices as for the less powerful ones. There are some features which this project doesn't have yet, like subdomain-level deduplication, but to my best knowledge this feature doesn't actually reduce the entries count significantly. We still may implement it if there is popular demand.

2 Likes
1173

Nice feedback. Mindful that adblock-lean is presently implemented as a simple service script, you might like to try the version on the master branch of the GitHub page. It has a lot of helpful features like checking lists are OK and dnsmasq is still functioning with new list, etc.

2 Likes
1174

Raw format for me..

2 Likes
1175

Do you mind sharing which lists you are using? The more info we get on this, the better we can implement a smooth transition.

1 Like
1176

This the lists I use, mostly Hagezi's.

Most recent session log:
[Aug 30 2024, 05:00:00] info: Started adblock-lean.
[Aug 30 2024, 05:00:00] info: gawk detected so using gawk for fast (sub)domain match removal.
[Aug 30 2024, 05:00:00] info: coreutils-sort detected so sort will be fast.
[Aug 30 2024, 05:00:00] info: Delaying adblock-lean by: 19 minutes (thundering herd prevention).
[Aug 30 2024, 05:19:00] info: Creating compressed backup of existing blocklist.
[Aug 30 2024, 05:19:00] info: No local allowlist identified.
[Aug 30 2024, 05:19:01] info: Not using any allowlist for blocklist processing.
[Aug 30 2024, 05:19:01] info: No local blocklist identified.
[Aug 30 2024, 05:19:01] info: Starting blocklist part(s) download.
[Aug 30 2024, 05:19:01] info: Downloading, checking and sanitizing blocklist part from: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/pro.txt.
[Aug 30 2024, 05:19:03] info: Successfully processed downloaded blocklist part (source file size: 4.3 MiB, sanitized line count: 162,950).
[Aug 30 2024, 05:19:03] info: Downloading, checking and sanitizing blocklist part from: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/tif.txt.
[Aug 30 2024, 05:19:16] info: Successfully processed downloaded blocklist part (source file size: 19.34 MiB, sanitized line count: 732,437).
[Aug 30 2024, 05:19:16] info: Downloading, checking and sanitizing blocklist part from: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/gambling.txt.
[Aug 30 2024, 05:19:22] info: Successfully processed downloaded blocklist part (source file size: 8.39 MiB, sanitized line count: 382,724).
[Aug 30 2024, 05:19:22] info: Downloading, checking and sanitizing blocklist part from: https://nsfw.oisd.nl/dnsmasq2.
[Aug 30 2024, 05:19:30] info: Successfully processed downloaded blocklist part (source file size: 9.66 MiB, sanitized line count: 377,687).
[Aug 30 2024, 05:19:30] info: Downloading, checking and sanitizing blocklist part from: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/native.amazon.txt.
[Aug 30 2024, 05:19:31] info: Successfully processed downloaded blocklist part (source file size: 12.94 KiB, sanitized line count: 351).
[Aug 30 2024, 05:19:31] info: Downloading, checking and sanitizing blocklist part from: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/native.apple.txt.
[Aug 30 2024, 05:19:31] info: Successfully processed downloaded blocklist part (source file size: 3.78 KiB, sanitized line count: 98).
[Aug 30 2024, 05:19:31] info: Downloading, checking and sanitizing blocklist part from: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/native.huawei.txt.
[Aug 30 2024, 05:19:32] info: Successfully processed downloaded blocklist part (source file size: 4.31 KiB, sanitized line count: 88).
[Aug 30 2024, 05:19:32] info: Downloading, checking and sanitizing blocklist part from: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/native.winoffice.txt.
[Aug 30 2024, 05:19:32] info: Successfully processed downloaded blocklist part (source file size: 13.55 KiB, sanitized line count: 338).
[Aug 30 2024, 05:19:32] info: Downloading, checking and sanitizing blocklist part from: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/native.tiktok.txt.
[Aug 30 2024, 05:19:33] info: Successfully processed downloaded blocklist part (source file size: 8.58 KiB, sanitized line count: 236).
[Aug 30 2024, 05:19:33] info: Downloading, checking and sanitizing blocklist part from: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/native.vivo.txt.
[Aug 30 2024, 05:19:33] info: Successfully processed downloaded blocklist part (source file size: 3.59 KiB, sanitized line count: 86).
[Aug 30 2024, 05:19:33] info: Downloading, checking and sanitizing blocklist part from: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/native.oppo-realme.txt.
[Aug 30 2024, 05:19:34] info: Successfully processed downloaded blocklist part (source file size: 11.07 KiB, sanitized line count: 298).
[Aug 30 2024, 05:19:34] info: Downloading, checking and sanitizing blocklist part from: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/native.xiaomi.txt.
[Aug 30 2024, 05:19:35] info: Successfully processed downloaded blocklist part (source file size: 12.74 KiB, sanitized line count: 382).
[Aug 30 2024, 05:19:35] info: Downloading, checking and sanitizing blocklist part from: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/hoster.txt.
[Aug 30 2024, 05:19:35] info: Successfully processed downloaded blocklist part (source file size: 35.85 KiB, sanitized line count: 1,833).
[Aug 30 2024, 05:19:35] info: Successfully generated preprocessed blocklist file with 1,659,508 lines.
[Aug 30 2024, 05:19:35] info: Sorting and merging the blocklist parts into a single blocklist file.
[Aug 30 2024, 05:19:43] info: Processed blocklist uncompressed file size: 40.81 MiB.
[Aug 30 2024, 05:19:43] info: New blocklist file check passed.
[Aug 30 2024, 05:19:43] info: Successfully imported new compressed blocklist file for use by dnsmasq with size: 9.49 MiB.
[Aug 30 2024, 05:19:43] info: Restarting dnsmasq.
[Aug 30 2024, 05:19:46] info: Waiting for dnsmasq initialization.
[Aug 30 2024, 05:19:59] info: Restart of dnsmasq completed.
[Aug 30 2024, 05:19:59] info: Processing time for blocklist generation and import: 0m:58s.
[Aug 30 2024, 05:19:59] info: Checking active blocklist.
[Aug 30 2024, 05:20:00] info: Active blocklist check passed with new blocklist file.
[Aug 30 2024, 05:20:00] info: New blocklist installed with good line count: 1,620,513.
[Aug 30 2024, 05:20:00] info: The locally installed adblock-lean is the latest version.


root@OpenWrtONE:~# free
              total        used        free      shared  buff/cache   available
Mem:         498544      200388      249832       11056       48324      243892
Swap:             0           0           0


root@OpenWrtONE:~# ubus call system board
{
        "kernel": "5.15.137",
        "hostname": "OpenWrtONE",
        "system": "ARMv8 Processor rev 4",
        "model": "Xiaomi Redmi Router AX6000 (OpenWrt U-Boot layout)",
        "board_name": "xiaomi,redmi-router-ax6000-ubootmod",
        "rootfs_type": "squashfs",
        "release": {
                "distribution": "OpenWrt",
                "version": "23.05.2",
                "revision": "r23630-842932a63d",
                "target": "mediatek/filogic",
                "description": "OpenWrt 23.05.2 r23630-842932a63d"
        }
}
2 Likes
1177

This is my second router.

root@OpenWrt:~# ubus call system board
{
        "kernel": "5.15.162",
        "hostname": "OpenWrt",
        "system": "MediaTek MT7621 ver:1 eco:3",
        "model": "Xiaomi Mi Router CR6608",
        "board_name": "xiaomi,mi-router-cr6608",
        "rootfs_type": "squashfs",
        "release": {
                "distribution": "OpenWrt",
                "version": "23.05.4",
                "revision": "r24012-d8dd03c46f",
                "target": "ramips/mt7621",
                "description": "OpenWrt 23.05.4 r24012-d8dd03c46f"
        }
}



Most recent session log:
[Aug 30 2024, 05:00:01] info: Started adblock-lean.
[Aug 30 2024, 05:00:01] info: gawk detected so using gawk for fast (sub)domain match removal.
[Aug 30 2024, 05:00:01] info: coreutils-sort detected so sort will be fast.
[Aug 30 2024, 05:00:01] info: Delaying adblock-lean by: 55 minutes (thundering herd prevention).
[Aug 30 2024, 05:55:01] info: Creating compressed backup of existing blocklist.
[Aug 30 2024, 05:55:01] info: No local allowlist identified.
[Aug 30 2024, 05:55:01] info: Not using any allowlist for blocklist processing.
[Aug 30 2024, 05:55:01] info: No local blocklist identified.
[Aug 30 2024, 05:55:01] info: Starting blocklist part(s) download.
[Aug 30 2024, 05:55:01] info: Downloading, checking and sanitizing blocklist part from: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/pro.txt.
[Aug 30 2024, 05:55:13] info: Successfully processed downloaded blocklist part (source file size: 4.3 MiB, sanitized line count: 162,950).
[Aug 30 2024, 05:55:13] info: Downloading, checking and sanitizing blocklist part from: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/tif.medium.txt.
[Aug 30 2024, 05:55:24] info: Successfully processed downloaded blocklist part (source file size: 3.82 MiB, sanitized line count: 156,696).
[Aug 30 2024, 05:55:24] info: Downloading, checking and sanitizing blocklist part from: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/hoster.txt.
[Aug 30 2024, 05:55:25] info: Successfully processed downloaded blocklist part (source file size: 35.85 KiB, sanitized line count: 1,833).
[Aug 30 2024, 05:55:25] info: Downloading, checking and sanitizing blocklist part from: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/native.xiaomi.txt.
[Aug 30 2024, 05:55:25] info: Successfully processed downloaded blocklist part (source file size: 12.74 KiB, sanitized line count: 382).
[Aug 30 2024, 05:55:25] info: Downloading, checking and sanitizing blocklist part from: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/native.tiktok.txt.
[Aug 30 2024, 05:55:26] info: Successfully processed downloaded blocklist part (source file size: 8.58 KiB, sanitized line count: 236).
[Aug 30 2024, 05:55:26] info: Downloading, checking and sanitizing blocklist part from: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/native.winoffice.txt.
[Aug 30 2024, 05:55:27] info: Successfully processed downloaded blocklist part (source file size: 13.55 KiB, sanitized line count: 338).
[Aug 30 2024, 05:55:27] info: Downloading, checking and sanitizing blocklist part from: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/native.huawei.txt.
[Aug 30 2024, 05:55:28] info: Successfully processed downloaded blocklist part (source file size: 4.31 KiB, sanitized line count: 88).
[Aug 30 2024, 05:55:28] info: Downloading, checking and sanitizing blocklist part from: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/native.apple.txt.
[Aug 30 2024, 05:55:28] info: Successfully processed downloaded blocklist part (source file size: 3.78 KiB, sanitized line count: 98).
[Aug 30 2024, 05:55:28] info: Successfully generated preprocessed blocklist file with 322,621 lines.
[Aug 30 2024, 05:55:28] info: Sorting and merging the blocklist parts into a single blocklist file.
[Aug 30 2024, 05:55:38] info: Processed blocklist uncompressed file size: 7.79 MiB.
[Aug 30 2024, 05:55:38] info: New blocklist file check passed.
[Aug 30 2024, 05:55:38] info: Successfully imported new compressed blocklist file for use by dnsmasq with size: 2.04 MiB.
[Aug 30 2024, 05:55:38] info: Restarting dnsmasq.
[Aug 30 2024, 05:55:42] info: Waiting for dnsmasq initialization.
[Aug 30 2024, 05:55:54] info: Restart of dnsmasq completed.
[Aug 30 2024, 05:55:54] info: Processing time for blocklist generation and import: 0m:52s.
[Aug 30 2024, 05:55:54] info: Checking active blocklist.
[Aug 30 2024, 05:55:54] info: Active blocklist check passed with new blocklist file.
[Aug 30 2024, 05:55:54] info: New blocklist installed with good line count: 306,757.
[Aug 30 2024, 05:55:55] info: The locally installed adblock-lean is the latest version.


free
              total        used        free      shared  buff/cache   available
Mem:         250312       83516      139132        3208       27664      126788
Swap:             0           0           0
2 Likes
1178

Now that my government is heading towards online censorship, I figured I would start enforcing DNS over TLS in my main cheap home router, and since many applications that do that in OpenWrt also serve as an adblocker, I figured I could do both.

Forgive me if this has been asked before (I've only skimmed through the thread searching for TLS and SSL), but does adblock-lean support DNS over TLS out of the box? If not, what is generally used to provide DNS over TLS by users running adblock-lean?

EDIT: A quick look at the repository shows that actual code consists of only a single shell script. Taking that into account, I assume the answer for DNS over TLS support is a no, unless I am missing something.

1179

Stubby (DOT)

1180

@Cthulhu88 I'm not an expert on DNS, and I did not design this project, but I helped coding some of its features. To my understanding, there is no direct connection between adblocking and DNS over TLS. adblock-lean, in particular, doesn't care how your DNS is communicated, as long as you are using dnsmasq. In a nutshell, it simply takes url's pointing to lists of domains you want to block (like Hagezi's lists and OISD) and optionally to allow, and optionally local blocklist or allowlist, compiles all that into a final blocklist and feeds that blocklist to dnsmasq. It will work with DNS over TLS or without.

(also AFAIK the other dns-based adblocking projects around do a similar thing, just not as efficiently :wink: - but with some extra features)

1181

Not quite what you're asking because it's DNS over HTTPS not DNS over TLS, but I use https-dns-proxy with the Force Router DNS, Canary Domains iCloud, and Canary Domains Mozilla all set to Force Router DNS to ensure clients are forced to go through my router, which ensures they go through my preferred DNS over HTTPS provider.

1182

adblock-lean works just fine with DNS over TLS using stubby. This is what I use myself (with the DNS over TLS version of Cloudflare’s family filter to block obscene material). Requests are ultimately still passed through dnsmasq for local caching (even if they are later on encrypted), and so adblock-lean, which leverages dnsmasq, works well in this scenario.