Optimization of adblock-lean

Hi, luci-adblock-lean is a separate project by @Ree9 . As you see, it was last updated about 3 months ago, and since then adblock-lean gained some new features and some config options were changed. Which is why luci-adblock-lean is incompatible with current version of adblock-lean. You can file an issue here and hopefully @Ree9 will respond to it.

This is a new one for me (having upgraded firmware):

root@OpenWrt:/etc/adblock-lean# sh /etc/init.d/adblock-lean update

Stopping adblock-lean.

Error: Missing file: '/usr/lib/adblock-lean/abl-lib.sh'.
Please run 'sh /etc/init.d/adblock-lean update -f' to fetch required files.

Cleaning up...
root@OpenWrt:/etc/adblock-lean# sh /etc/init.d/adblock-lean update -f
Failed to parse json data: unexpected end of data

Error: Failed to get GitHub download URL for commit '' (update channel: 'branch=support_adblocking_on_multiple_instances').
uclient-fetch output:
Downloading 'https://api.github.com/repos/lynxthecat/adblock-lean/commits?sha=support_adblocking_on_multiple_instances'
Connecting to 20.26.156.210:443
HTTP error 404

Error: Failed to update adblock-lean.

Cleaning up...

Specifying -v release fixed it.

That branch doesn't exist anymore, probably that's the reason.

The URL doesn't look right though. I'll look into this.

Edit: upon checking the API, the URL is correct. So the only issue was invalid update channel (branch=support_adblocking_on_multiple_instances) set in your adblock-lean init file.

Ah maybe a legacy issue from testing earlier versions? I wonder if where this happens we should suggest trying ā€˜-v release’, just so user has a way to proceed?

Yes, most likely.

This error is not specific to this particular situation. It might be printed under all sorts of fault conditions, including if there is no internet connection, or if GH is offline, or blocked by something (including by adblock-lean, eg when whitelist mode is on and the user does not allow connection to GH). So this specific advice will only cover this particular scenario, which I would think should never occur to people who are not specifically testing a specific branch, and those people will probably know that they can change the update channel. So I think a specific automated advice will be misleading in most cases and when it's not, probably wouldn't be needed.

Hey @Lynx and everyone else reading this :slight_smile:

I created a new branch various_improvements, with the commits listed here.

In particular, some bugs were fixed, support for hosts-format lists was added (along with supporting short IDs for stevenblack’s lists, eg stevenblack:base, stevenblack:social), and some (hopefully) improvements to console output messages were made.

Testing, comments, suggestions welcome.

To test:

service adblock-lean update -v branch=various_improvements

Note that I changed some config options names (what do you think about it @Lynx ?), so if you want to go back to stable version later, those options will be reset. So consider making a backup of your current config file before testing:

cp /etc/adblock-lean/config <backup_path>

I would be interested to hear opinions and improvement suggestions about the hosts-to-raw conversion code:

Wow - a lot of nice changes. The name changes all look good to me.

Are existing configs in need of manual adaptation?

root@OpenWrt:~# service adblock-lean update -v branch=various_improvements

Downloading adblock-lean, commit 'fedbf8b9b864cddedf3581a2956e44ad14efbac2' (update channel: 'branch=various_improvements').

Installing new files...
Copying file '/etc/init.d/adblock-lean'.
Copying file '/usr/lib/adblock-lean/abl-lib.sh'.
Copying file '/usr/lib/adblock-lean/abl-process.sh'.

adblock-lean has been updated to version 'fedbf8b9b864cddedf3581a2956e44ad14efbac2'.

Start adblock-lean now? (y|n)
y|n: y

Starting adblock-lean, version fedbf8b9b864cddedf3581a2956e44ad14efbac2.

gawk detected so using gawk for fast (sub)domain match removal and entries packing.
GNU sed detected so list processing will be fast.
coreutils-sort detected so sort will be fast.

Error: Failed to parse config. Errors: 
/etc/rc.common: eval: line 1187: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt: not found

Error: Failed to load config.

Cleaning up...

Generating new config didn’t seem to help:

root@OpenWrt:~# service adblock-lean gen_config

Based on the total usable memory of this device (488.76 MiB), the recommended preset is 'large':
Elements count: ~1,200,000
raw_block_lists="hagezi:pro hagezi:tif"
max_file_part_size_KB="24000"
max_blocklist_file_size_KB="29000"
min_good_line_count="340000"

[C]onfirm this preset or [p]ick another preset?
c|p: c
Selected preset 'large'.

Checking dnsmasq instances.
Detected only 1 dnsmasq instance - skipping manual instance selection.
Selected dnsmasq indexes: '0'.
Selected dnsmasq conf-dirs: /tmp/dnsmasq.cfg01411c.d

Cron job configuration:
A cron job can be created to enable automatic list updates.
The default schedule is '0 5 * * *': daily at 5am (5 o'clock at night)
The cron job will run with an added random number of minutes.

Create cron job with default schedule for automatic list updates? (y|n)
'n' will set the 'cron_schedule' setting to 'disable'. You can later create a cron job with a custom schedule as described in:
https://github.com/lynxthecat/adblock-lean/blob/master/README.md
y|n: y

Generating new default config for adblock-lean from preset 'large'.
This will overwrite existing config. Proceed? (y|n)
y|n: y

Saving new config file to '/etc/adblock-lean/config'.
root@OpenWrt:~# service adblock-lean start

Starting adblock-lean, version fedbf8b9b864cddedf3581a2956e44ad14efbac2.

gawk detected so using gawk for fast (sub)domain match removal and entries packing.
GNU sed detected so list processing will be fast.
coreutils-sort detected so sort will be fast.

Checking dnsmasq instances.

Creating compressed backup of existing blocklist.

Testing connectivity.

Downloading and processing blocklist parts (max parallel jobs: 2).

Not using any allowlist for blocklist processing.

Error: Failed to generate preprocessed blocklist file with at least one entry.

Error: Failed to generate new blocklist.

Restoring saved blocklist file.

Importing the blocklist file.

Restarting dnsmasq.

Waiting for dnsmasq initialization.
Restart of dnsmasq completed.
Successfully imported new compressed blocklist file for use by dnsmasq with size: 5.57 MiB

Checking the active blocklist.

Checking dnsmasq instances.

Using following nameservers for DNS resolution verification: 127.0.0.1, ::1
Testing adblocking.
Testing DNS resolution.
Previous blocklist restored and dnsmasq check passed.

Checking for adblock-lean updates.
The locally installed adblock-lean is the latest version.

Cleaning up...
root@OpenWrt:~#

Yep, there is a bug. Possibly 2 bugs. Looking into this.

@Lynx indeed 2 bugs found, and a third (minor) one while looking into the code :slight_smile:

Pushed the changes into the branch - should be working as intended now.

P.s. config options are automatically migrated but there was a bug in the code responsible for migration.

Edit: oops, forgot to merge the PR with the fixes. Now merged.

And indeed this one worked fine!

The output looks beautiful.

One minor idea. Might it not be an idea to use a bold colour to make the lists (e.g. here hagezi:pro and hagezi:tif) stand out like orange or something?

Do you mean in download start messages, success messages, or both?

I thought it made sense to use a different color in download start messages from the one in success messages because otherwise (especially now with vertical alignment) it’s harder to actually figure out which list started and which finished.

This is what it looks like now (I removed the Starting ... download and processing messages just now because I think they are not really needed):

And this is what it looks like if all list identifiers are colored the same:

You just get a same-colored block of text which IMO it’s harder to make sense of, whether it’s blue or orange.

Maybe it’s better to remove the green color of the success messages and only color the list identifiers in green, like so:

?

Ah, sorry I wasn’t clear - I meant the references to the lists themselves like hagezi:pro and hagezi:tif since these are now special terms, as it were. I wondered about orange or something like that that stands out more.

Just an idea - the coloured text output is already beautiful.

Did you see this by the way:

I was talking about them a well.

Actually GH didn't notify me. And yes, good catch. The double negative is not intended, of course.

I merged another PR into the various_improvements branch. The PR fixes the double-negative, changes colors in the progress messages a bit, reduces the number of messages adblock-lean sends to the system log and adds support for ${ABL_DEBUG_LOG} environment variable which, when set, forces all messages into the system log.

@Lynx if you would like to tweak the colors by yourself, I would be curious to see the result.

Otherwise, I think this can be merged into master?

Since it seemed like you are done tweaking code on this for the moment, I jumped in and tested the installation. I’ll check my router logs the next couple days to ensure the daily blocklist fetch still works and I will report here if I see any issues.

Here is the output of the update…

Update output:

root@R4S-wrt:~# cp /etc/adblock-lean/config /usb-stor/usb1p1/abl-config-$(appendTS)
root@R4S-wrt:~# ls -ltr /usb-stor/usb1p1/abl-config-2025* | tail -4
-rwxrwxrwx 1 root root 4781 Mar 24 2025 /usb-stor/usb1p1/abl-config-20250324T161455
-rwxrwxrwx 1 root root 4779 Mar 24 2025 /usb-stor/usb1p1/abl-config-20250324T180326
-rwxrwxrwx 1 root root 5056 Jun 5 13:43 /usb-stor/usb1p1/abl-config-20250605T134320
-rwxrwxrwx 1 root root 4566 Oct 10 16:57 /usb-stor/usb1p1/abl-config-20251010T165750
root@R4S-wrt:~#
root@R4S-wrt:~#
root@R4S-wrt:~# service adblock-lean update -v branch=various_improvements

Downloading adblock-lean, commit '0b2f7c5b58c70648655c340af50b34b672ffc184' (update channel: 'branch=various_improvements').

Installing new files...
Warning: File '/etc/init.d/adblock-lean' was manually modified - overwriting.
Saved a backup copy of manually modified file to /tmp/abl_old_modified_files/adblock-lean
Copying file '/etc/init.d/adblock-lean'.
Copying file '/usr/lib/adblock-lean/abl-lib.sh'.
Copying file '/usr/lib/adblock-lean/abl-process.sh'.

adblock-lean has been updated to version '0b2f7c5b58c70648655c340af50b34b672ffc184'.

Start adblock-lean now? (y|n)
y|n: y

Starting adblock-lean, version 0b2f7c5b58c70648655c340af50b34b672ffc184.

gawk detected so using gawk for fast (sub)domain match removal and entries packing.
GNU sed detected so list processing will be fast.
coreutils-sort detected so sort will be fast.

Following config options need to be migrated (option name has changed): 'blocklist_urls blocklist_ipv4_urls allowlist_urls dnsmasq_blocklist_urls dnsmasq_blocklist_ipv4_urls dnsmasq_allowlist_urls min_blocklist_ipv4_part_line_count'.
Corresponding config entries:
blocklist_urls="https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt "
blocklist_ipv4_urls=""
allowlist_urls=""
dnsmasq_blocklist_urls=""
dnsmasq_blocklist_ipv4_urls=""
dnsmasq_allowlist_urls=""
min_blocklist_ipv4_part_line_count="1"

Missing keys in config: 'hosts_allow_lists hosts_block_lists'.
Corresponding config entries:
hosts_allow_lists=""
hosts_block_lists=""

Perform following automatic changes? (y|n)

  1. Migrate config entries
  2. Re-add missing config entries with default values
    y|n: y

Old config file was saved as /tmp/adblock-lean_config.old.
This will overwrite existing config. Proceed? (y|n)
y|n: y

Saving new config file to '/etc/adblock-lean/config'.

Checking dnsmasq instances.

Creating uncompressed backup of existing blocklist.

Testing connectivity.

Downloading and processing blocklist parts (max parallel jobs: 4).
Not using any allowlist for blocklist processing.

Processing raw blocklist: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt
Processing raw blocklist: /etc/adblock-lean/blocklist
Processing raw blocklist: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt
Successfully processed list: /etc/adblock-lean/blocklist [ 414.00 B - 16 lines ]
Successfully processed list: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt [ 3.21 MiB - 181,801 lines ]
Successfully processed list: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt [ 13.61 MiB - 823,349 lines ]

Successfully generated preprocessed blocklist file with 1,005,166 entries.

Sorting and merging the blocklist parts into a single blocklist file.

Stopping dnsmasq.

Checking the resulting blocklist with 'dnsmasq --test'.
New blocklist file check passed.
Final list uncompressed file size: 15.74 MiB

Importing the blocklist file.

Restarting dnsmasq.

Waiting for dnsmasq initialization.
Restart of dnsmasq completed.
Successfully imported new blocklist file for use by dnsmasq with size: 15.74 MiB

Processing time for blocklist generation and import: 0m:31s.

Checking the active blocklist.

Checking dnsmasq instances.

Using following nameservers for DNS resolution verification: 127.0.0.1, ::1
Testing adblocking.
Testing DNS resolution.

Active blocklist check passed with the new blocklist.
New blocklist installed with entries count: 935,210

Checking for adblock-lean updates.
The locally installed adblock-lean is the latest version.
root@R4S-wrt:~#

@antonk The ā€œpatchā€ script that you helped me with still worked. It looks like the ā€œrandom_delay_mins=ā€¦ā€ line stayed the same. :slight_smile:

Looks good, thanks for testing!

@Lynx I sent a PR to the master branch, would you like to review it?

Hi @Lynx , did you notice my response in the PR?

Yep, sorry, all seems good to merge in now. Another great set of updates.

Do you think the name of adblock-lean is still good mindful that it could be used for something other than adblocking in the case of blocking offensive content?

I suppose the primary use case for adblock-lean will be filtering out ads, but the scope is clearly broader than just that, as brought home to me with your recent inclusion of NSFW related lists.

If you mean the oisd NSFW lists, haven’t they been usable since forever in adblock-lean? Similar Hagezi lists have been usable for a long time as well. I only added support for stevenblack lists recently.

Can’t every adblocker out there be used for similar purposes as well? So I definitely wouldn’t suggest to change the name for this reason. And regardless, IMO the name should stay since this is how people find the project. Imagine someone recommending adblock-lean but there is no more adblock-lean because the name was changed :slight_smile:

Yay! That said, I’m a little bit worried about the hosts-to-raw conversion. Theoretically, a hosts file could have an entry like so:


127.0.0.1 mydomain.mylan

While author’s intention would have been to associate mydomain.mylan with localhost, our conversion algorithm will simply add it to the blocklist. I’m not sure whether this scenario even makes sense for blocklists downloaded from the Internet, and we do not process local blocklists in hosts format. So I don’t know whether the concern is valid. What do you think?