Adblock support thread


#1

Hi,

in OpenWrt stable & snapshot package repo you'll find the adblock package (plus LuCI companion/configuration package):

stable OpenWrt version 18.06.: adblock 3.5.4 plus luci companion package

latest snapshot version: adblock 3.5.4 plus luci companion package

Link to the latest adblock documentation

Feel free to test, ask questions or make suggestions.


Changelog

---
release 3.5.4
backend:

  • add low priority mode (nice level 10), disabled by default (config option is called 'adb_nice' in the 'extra' config section, the range 0-19 is allowed)
  • enhance 'Force DNS' to redirect ports 53, 853 and 5353

frontend:

  • switch to dynamic XHR polling for runtime information and logfile
    viewing
  • add new 'Refresh' button to reload blocklists
  • various cleanups & small fixes
    ---
    release 3.5.3
  • enhance the whitelist function. Now sub-domains could be whitelisted
    (e.g. 'fakenews.facebook.com'), even if the correspondent tld is
    blacklisted (e.g. 'facebook.com') - this makes whitelisting
    much more flexible and predictable
  • rework the domain query function to adapt the whitelist changes
  • refine startup error checks/messages
  • small fixes
    ---
    release 3.5.2
  • add generic blocklist archive support
  • add support for blacklist archive from Toulouse 1 University Capitole
  • add support for urlhaus RPZ domains by abuse.ch
  • archive sub-categories (shalla & ut_capitole) are now configurable via LuCI CBI template
  • small bugfixes & enhancements
    ---
    release 3.5.1
  • maintenance update, just small bugfixes
    ---
    release 3.5.0
  • major performance boost: add a flexible 'Download Queue' to handle downloads & list processing in parallel, default queue size is '4', you can raise this e.g. to '8' or '16' to get it really fast
  • replace former 'whitelist mode': the new 'Jail' option builds an additional 'adb_list.jail' list in parallel to block access to all domains except those listed in the whitelist file, which can be used manually for guest wifi or kidsafe configurations
  • regex parser & query function now fully support IDN domains with non-ASCII characters
    add error handling in tld compression, to handle OOM conditions better
  • adblock.notify sends now html emails, to get a better look & feel, even on mobile devices
  • add czech regional blocklist maintained by turris omnia users
  • LuCI: Support new 'Download Queue' & 'Jail' options
  • LuCI: fix field width in "Runtime Information" section
    ---
    release 3.4.3 (current stable release)
  • add pidfile writing / check to prevent further race conditions
  • ease the download utility selection: uclient-fetch (default), wget, curl, aria2c, wget-nossl, busybox-wget are fully pre-configured available
  • add debug download logging in case of an error, e.g. wrong url
  • change 'malware' blocklist source url
  • add logfile information to email template
  • LuCI: add 'Download Utility' select box
  • LuCI: add new "running" status
    ---
    release 3.4.1
  • enable code to support Turris Omnia forthcoming upstream change
    (new kresd 'keep_cache' option) to preserve kresd DNS cache
  • fix a 'status' race condition while the adblock process is running in parallel
  • various small speed improvements
  • rework debug output
  • refine blacklist handling
  • enable the (empty) blacklist source in the default config
  • email notification supports mstmp, even without sendmail symlink
  • email notification writes minimal status to log (one-liner)
  • LuCI: refine logfile search term
  • LuCI: Textarea 'autoscroll down' in logfile view
  • LuCI: Left-align blocklist source table plus a more compact design
    ---
    release 3.4.0
  • preserve DNS cache after adblock processing,
  • email notification in case of an error or domain count < n (default 0, check readme)
  • removed securemecca from default config (service has been closed, read http://securemecca.blogspot.de/)
  • new separate functions for hash compare and list/overall count
  • add missing package dependencies
  • various clean-ups
  • update documentation
    ---
    release 3.1.1
  • new function to set/delete options in external uci config files
    • kresd: automated 'rpz_file' handling in /etc/config/resolver
    • firewall: automated 'force_dns' handling if you enable or disable adblock
  • support sha256sum (default) and md5sum for blocklist comparison & conditional dns restarts
  • cosmetics
    ---
    release 3.1.0
  • add 'whitelist mode', block access to all domains
    except those explicitly listed in the whitelist file
  • rework awk regex for all blocklist sources
    • include 'third-party' domains for all regional lists
    • change adguard url and refine filter ruleset
    • use POSIX character classes
    • fix regex for whitelist preparation
    • fix corner case parsing issues
  • fix enable/disable behavior
  • various other small fixes
  • documentation update
  • caution: config file update required!
    ---
    release 3.0.3
  • add new list source to default config to block browser-based crypto mining
    ---
    release 3.0.2
  • better system information
  • several kresd related documentation fixes
    ---
    release 3.0.1
  • fix startup issues with backends like dnscrypt-proxy or kresd
    which does not come up without an existing block list
  • fix a small 'chown' issue
    ---
    release 3.0.0
  • add kresd & turris omnia support
  • add dnscrypt-proxy support
  • change start priority to 30, to fix possible trigger issues on slow booting hardware
  • simplify suspend/resume handling (no longer use a hideout directory)
  • default config change (please update your config!), adblock is now disabled by default
  • enhanced LuCI frontend
  • many small changes & improvements
  • documentation update
    ---
    release 2.8.5
  • add preliminary kresd dns backend support for turris devices, see readme (untested!)
  • use tld compression for overall list, too
  • cosmetics
    ---
    release 2.8.3
  • refine manual/backup mode (exclude local blacklist processing)
  • cosmetics
    ---
    release 2.8.2
  • made DNS restart conditional (compare list hash values), to prevent needless restarts of the DNS backend
    ---
    release 2.8.1
  • revert 'wan6' interface trigger in default config due to current procd limitation (see issue: #4521 on github)
    ---
    release 2.8.0-2
  • add bind support (see readme)
  • export all blocked domains in one central file (adb_list.overall)
  • prerequisite for proper bind support
  • much faster sort operation with less memory consumption
  • backups are still handled per source separately, to be more flexible in adding/removing block list sources
  • add additional 'wan6' interface trigger in default configuration
  • various small fixes & optimizations
Ancient Releases

---

LEDE version 17.01.: adblock 3.4.3 plus luci companion package


Have fun!
Dirk


Xiaomi WiFi Router 3G
Adblock statistics
*FIXED* Adblock whitelist bit.ly does not work
Lede / adblock noob
#2

At the first glance, it seems to work nicely.


#3

logread -e "adblock" gives the following output:

Mon Dec 12 09:53:36 2016 daemon.err adblock.sh[944]: udhcpc: started, v1.25.1
Mon Dec 12 09:53:36 2016 daemon.err adblock.sh[944]: udhcpc: sending discover
Mon Dec 12 09:53:39 2016 daemon.err adblock.sh[944]: udhcpc: no lease, failing
Mon Dec 12 09:53:39 2016 daemon.err adblock.sh[944]: udhcpc: started, v1.25.1
Mon Dec 12 09:53:39 2016 daemon.err adblock.sh[944]: udhcpc: sending discover
Mon Dec 12 09:53:42 2016 daemon.err adblock.sh[944]: udhcpc: no lease, failing
Mon Dec 12 09:53:42 2016 user.notice adblock-[1.9.99-pre0] info: block lists with overall 132369 domains loaded (1.9.99-pre0, LEDE Reboot CURRENT r2437-854459a)

I removed the previous package and luci addon with luci.


#4

udhcpc is the DHCP client fetching a WAN address from your ISP for your WAN interface. Nothing to do with adblock (or DNS) by itself.

Does your router get WAN address when adblock is disabled?

But you seem to have a lot of blocklists loaded. You might test first with only a few lists. And you might reboot the router one more time so that there are surely no leftovers in dnsmasq tmp from the previous ablock version.


#5

My router gets a WAN address also when adblock is enabled.
As far as I can see adblock2 is working without a problem. I just don't understand the errormessages in the log.
My router has plenty of ram (512 MB), so I thought is is not a problem to enable almost every list.


#6

@htmt
Does this dhcp "error" always appear (i.e. during a normal /etc/init.d/adblock restart) or only during boot?
What happen if you change ...

procd_set_param stderr 1 to procd_set_param stderr 0

in /etc/init.d/adblock?

Thanks for testing!


#7

Yes the error appears always (reboot and restart).

Still the same errormessage.

I feel myself obliged to do so because adblock is why I bought this new router in the first place.
Thanks for making this great tool!


#8

OK, please make a "/etc/init.d/dnsmasq restart" for two times. First with enabled block lists and second time without adblock (after /etc/init.d/adblock stop). In both cases please check the log for dnsmasq/dhcpc errors like you've posted above.

Thanks again!


#9

Mon Dec 12 16:53:58 2016 daemon.err adblock.sh[1843]: udhcpc: started, v1.25.1
Mon Dec 12 16:53:58 2016 daemon.err adblock.sh[1843]: udhcpc: sending discover
Mon Dec 12 16:54:01 2016 daemon.err adblock.sh[1843]: udhcpc: no lease, failing
Mon Dec 12 16:54:01 2016 daemon.err adblock.sh[1843]: udhcpc: started, v1.25.1
Mon Dec 12 16:54:01 2016 daemon.err adblock.sh[1843]: udhcpc: sending discover
Mon Dec 12 16:54:04 2016 daemon.err adblock.sh[1843]: udhcpc: no lease, failing
Mon Dec 12 16:54:05 2016 user.notice adblock-[1.9.99-pre0] info: block lists with overall 132369 domains loaded (1.9.99-pre0, LEDE Reboot CURRENT r2437-854459a)
Mon Dec 12 17:05:48 2016 daemon.err adblock.sh[2657]: udhcpc: started, v1.25.1
Mon Dec 12 17:05:48 2016 daemon.err adblock.sh[2657]: udhcpc: sending discover
Mon Dec 12 17:05:51 2016 daemon.err adblock.sh[2657]: udhcpc: no lease, failing
Mon Dec 12 17:05:51 2016 daemon.err adblock.sh[2657]: udhcpc: started, v1.25.1
Mon Dec 12 17:05:51 2016 daemon.err adblock.sh[2657]: udhcpc: sending discover
Mon Dec 12 17:05:54 2016 daemon.err adblock.sh[2657]: udhcpc: no lease, failing
Mon Dec 12 17:05:54 2016 user.notice adblock-[1.9.99-pre0] info: block lists with overall 132369 domains loaded (1.9.99-pre0, LEDE Reboot CURRENT r2437-854459a)

I wouldn't be surprised when the problem is outside of Adblock.
My router is, as of now, not fully supported by LEDE (TP-Link Archer C2600).

Adblocking is functional with this release.

If a full log is usefull, let me know.


#10

So dns hijacking is no longer an option with adblock2?

I haven't mastered wrapping firewall rules in procd instance, but if you'd want to bring back the pixelserv, I found out uhttpd is really easy to include in PROCD instance.

On ipq806x LEDE Reboot CURRENT r2437+12 with uclient-fetch and ustream-mbedtls installed, adblock2 fails with error: fetch utility 'uclient-fetch' or 'wget' not found. Maybe the below will help:

root@EA8500:~# which /usr/bin/wget* root@EA8500:~# which uclient-fetch /bin/uclient-fetch root@EA8500:~# which /bin/wget* /bin/wget root@EA8500:~# ls -la $_ lrwxrwxrwx 1 root root 13 Dec 10 16:24 /bin/wget -> uclient-fetch


#11

:slight_smile:
Still it's all about dns spoofing/hijacking but we do not longer need all the firewall/pixel server stuff for that. The dns server itself returns a simple 'NXDOMAIN'. This is nothing but Non-eXistent Internet or Intranet domain name, if domain name is unable to resolve using the dns server, a condition called the 'NXDOMAIN' occurred.

On client side you'll see things like that:

root@x250:/home/dirk# nslookup doubleclick.net
Server: 192.168.1.1
Address: 192.168.1.1#53
** server can't find doubleclick.net: NXDOMAIN

or that (captured during a youtube session):

root@x250:/home/dirk# tcpdump -nt -i wlan0 udp port 53
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on wlan0, link-type EN10MB (Ethernet), capture size 262144 bytes
IP 192.168.1.103.40371 > 192.168.1.1.53: 44883+ A? www.google-analytics.com. (42)
IP 192.168.1.103.40371 > 192.168.1.1.53: 20204+ AAAA? www.google-analytics.com. (42)
IP 192.168.1.1.53 > 192.168.1.103.40371: 44883 NXDomain 0/0/0 (42)
IP 192.168.1.1.53 > 192.168.1.103.40371: 20204 NXDomain 0/0/0 (42)
[...]

simple & quite effective! :relaxed:

As I wrote in my initial post you have to set two options in adblock config, for uclient-fetch use the following:

option adb_fetch '/bin/uclient-fetch'
option adb_fetchparm '-q'


#12

By DNS hijacking I meant that if I have a client in the local network with hardcoded DNS (rather than using DNS server offered by DHCP) it will no longer be forced to use my router's dns server, will it now, without firewall rules?

Ah, sorry, missed that, I was hoping adblock2 would auto-discover what I have installed. :wink:


#13

adblock supports a wider range of router modes, incl. AP modes. Therefore it makes no sense to set firewall redirects like that. (same applies to adblock 1.x)

a guy who changed the default ssl-backend can add two config options, too. :slight_smile:
a positive side effect: you could use other tools like aria2 ...


#14

Ah, I see the reason. It'd still make sense to set the redirects for the router mode, which I'd expect to be like 90% of the cases. :wink:


#15

I've updated the download link in the first post for the new pre1 version, changes are:

  • procd fine tuning
  • many bugfixes

If nothing major comes up, I'll send a github PR before christmas ... :slight_smile:


#16

The initial adblock_1.9.99-pre0-1_all ran perfectly well for me. Now adblock_1.9.99-pre1-1_all is also running well perfectly for the past 12-14 hours or so. No issues to report on my end as far as functionality goes.

I must admit, I do wish that there still remained an option to keep the pixel server running to replace all of the DNS-related errors that come up now with this NXDOMAIN method. But anyway, I trust in your judgement and coding work 100% and I will continue testing all future releases as they are released.


#17

Hi Dave,

thanks for testing!
Do you have example urls with "broken links" where ads are not delivered by https server? For ad related content from https server you will receive broken links, even with adblock 1.x cause we cannot redirect secured connections.

br
dirk


#18

Just want to say thanks for this! I run your current version published in trunk (1.5.4-1 with the Luci counter part), and it's a great tool. Thanks for continuing to support it!


#19

Tested on my TP-Link Archer C2600 and my travel-router GLi AR150.
Works like a charm! I love it.
Especially the AR150 can handle this version much easier than the previous version.


#20

The LuCI app modified for the new adblock2 can be found from here:
https://www.dropbox.com/s/dobai5nfua5bifz/luci-app-adblock_git-16.352.65143-1741e5578-1_all.ipk?dl=0