OpenWrt Forum Archive

Topic: adblock package, release 2.x

The content of this topic has been archived between 22 Mar 2018 and 4 May 2018. Unfortunately there are posts – most likely complete pages – missing.

Would be nice if there was a more efficient  version than ran on low power hardware. My devices typically have < 1 MB of storage available and 64 MB RAM.

Thanks for the new version 0.22.0.

Memory consumption (during sorting of the downloaded blocklists) has decreased somewhat, although selecting at least 2 large lists in a device with 64 MB or less memory may cause risk of an out-of-memory error due to the apparent inefficiency of the default Busybox sort. Installing the GNU sort (coreutils-sort) helps.

And the automatic detection of most of the needed options is nice.

@hnyman: Thanks for testing & rapid deploying to trunk! ;-)

@roger_: my recommendation for low memory systems is to add an usb stick or any other storage device to supersize your /tmp directory with a swap partition (see openwrt wiki). After that you can resize and use your tmp partition without a breeze. Anyway, 0.22.0 should help a lot ... ;-)


Link to the latest adblock documentation

(Last edited by dibdot on 25 Nov 2015, 14:24)

small feedback for the next release:

I tested the "shallalist" source with default categories selection and to my surprise both Dropbox and Microsoft Onedrive got blocked. For some reason they are included in the "downloads" category, which is selected by default for shallalist.

You might consider either
* whitelisting the global cloud file sharing services (dropbox, onedrive), or
* removing the "downloads" from the shallalist defaults.

And I am not quite sure why bild.de should be in the default blacklist...
(you

Ps.
I am not quite sure about the need to delete shell variables CONFIG_* . Which variables? For what purpose? (I don't see the creation of them anywhere. If they are from an ancient version ages ago, just forget about them.)

(Last edited by hnyman on 27 Nov 2015, 14:25)

@hnyman

The CONFIG_* environment variables will be created during script runtime by uci configuration system. The vars set from parsing uci config should disappear once the adblock script quits, it's just a tweak to save space (10-20k) during runtime.

bild.de is just a personal "favorite", simply remove it if you like to read "news" with big pictures ... ;-))

For 0.22.2 I'll think about a dynamic whitelist to solve this shallalist issue with the "downloads" category. You find a short description of every category here.

Thanks!
Dirk

Hi, in my logs i see the following:
adblock[7638] error: failed to initialize new dynamic/volatile uhttpd instance (adblock, 192.168.6.1), rc: 1

what do i have to configure to get rid of it ?

uhttpd is just configured on 192.168.6.1:80 for luci

(Last edited by sok on 9 Dec 2015, 20:34)

@sok: the ip address of the local adblock interface/uhttpd instance needs to be a different subnet from the normal LAN, i.e. 192.168.5.1 in your case. See also /etc/adblock/README.md and /etc/adblock/samples/adblock.con.sample.

Thanks for the adblock solution!

Using TP-Link TL-WR1043ND_V1 I had to add a sleep line in /etc/rc.local to avoid error related to inexistent input network.

 /usr/bin/logger -t rc.local "start adblock script"
sleep 20
/usr/bin/adblock-update.sh >/dev/null 2>&1

Hi,

thanks for using adblock!
There is no need for such workaround in rc.local. Simply activate "wancheck" service in adblock config (see example in /etc/adblock/samples/adblock.conf.sample) and adblock will wait for your wan device to come up ... ;-))

dj_mcm2001 wrote:

Thanks for the adblock solution!

Using TP-Link TL-WR1043ND_V1 I had to add a sleep line in /etc/rc.local to avoid error related to inexistent input network.

 /usr/bin/logger -t rc.local "start adblock script"
sleep 20
/usr/bin/adblock-update.sh >/dev/null 2>&1

Hi, I am having the same issue as Sok.  Here is the output in the console when running the script

adblock[16592] info : domain adblock processing started (0.40.0, 15.05, 22.12.2015 16:13:49)
adblock[16592] info : backup/restore will be disabled
adblock[16592] info : dns query logging will be disabled
adblock[16592] info : debug logging will be disabled
adblock[16592] info : wan update check will be disabled
adblock[16592] info : get ntp time sync (0.openwrt.pool.ntp.org 1.openwrt.pool.ntp.org 2.openwrt.pool.ntp.org 3.openwrt.pool.ntp.org), after 0 loops
adblock[16592] error: failed to initialize new dynamic/volatile uhttpd instance (adblock, 192.168.9.1), rc: 1
adblock[16592] info : domain adblock processing finished (0.40.0, 15.05, 22.12.2015 16:14:05)

my Lan is 192.168.0.x so it is different in the config.  Any suggestions?

Please post your uhttpd config file.

jacqkeen wrote:

Hi, I am having the same issue as Sok.  Here is the output in the console when running the script

adblock[16592] info : domain adblock processing started (0.40.0, 15.05, 22.12.2015 16:13:49)
adblock[16592] info : backup/restore will be disabled
adblock[16592] info : dns query logging will be disabled
adblock[16592] info : debug logging will be disabled
adblock[16592] info : wan update check will be disabled
adblock[16592] info : get ntp time sync (0.openwrt.pool.ntp.org 1.openwrt.pool.ntp.org 2.openwrt.pool.ntp.org 3.openwrt.pool.ntp.org), after 0 loops
adblock[16592] error: failed to initialize new dynamic/volatile uhttpd instance (adblock, 192.168.9.1), rc: 1
adblock[16592] info : domain adblock processing finished (0.40.0, 15.05, 22.12.2015 16:14:05)

my Lan is 192.168.0.x so it is different in the config.  Any suggestions?

config uhttpd 'main'
        list listen_http '192.168.0.1:80'
        #list listen_http '[::]:80'
        list listen_https '192.168.0.1:443'
        #list listen_https '[::]:443'
        option home '/www'
        option rfc1918_filter '1'
        option max_requests '3'
        option max_connections '100'
        option cert '/etc/uhttpd.crt'
        option key '/etc/uhttpd.key'
        option cgi_prefix '/cgi-bin'
        option script_timeout '60'
        option network_timeout '30'
        option http_keepalive '20'
        option tcp_keepalive '1'
        option ubus_prefix '/ubus'

Gah! I restarted the uhttpd daemon and now i don't get the error! Sorry for the trouble.

IPset? Not, I didn't hear...

ipset create AD_NET4 hash:net family inet
ipset create AD_NET6 hash:net family inet6
ipset create AD_IP4 hash:ip family inet
ipset create AD_IP6 hash:ip family inet6
ipset create ADBLOCK  list:set
ipset add ADBLOCK AD_NET4
ipset add ADBLOCK AD_NET6
ipset add ADBLOCK AD_IP4
ipset add ADBLOCK AD_IP6
ipset add ADBLOCK 77.88.8.8
ipset add ADBLOCK 0.0.0.0/8
ipset add ****
etc

Option

#!/bin/bash

downloadList()
{
    while [ "$(curl http://.../blacklist.txt.gz -z blacklist.txt.gz -o blacklist.txt.gz -s -L -w %{http_code})" != "200" ]; do
        echo "Blocklist download failed! Trying again in 60 seconds..."
        sleep 60
    done
    echo "Blocklist download successful!"
}

applyList()
{
    local CHANGED=0
    for IP in $(zcat blacklist.txt.gz | sed -n '/^[0-9]/p'); do
        CHANGED=1
        ipset -q add ADBLOCK $IP -exist
    done
    if [ "$CHANGED" -eq 1 ]; then
        echo "New IP addresses added to blacklist!"
    else
        echo "No new IP addresses added to blacklist!"
    fi
}

# kill any existing blocklist downloader PIDs
pkill -f <nameOfYourScriptFileHere>
# download the blocklist
downloadList
# apply the blocklist
applyList

exit 0

Allso

downloadList()
{
    local RESULT
    while true; do
        RESULT="$(curl http://.../blacklist.txt.gz -z blacklist.txt.gz -o blacklist.txt.gz -s -L -w %{http_code})"
        case $RESULT in
            200)
                echo "Blocklist download successful!"
                break
                ;;
            404)
                echo "404 received! Trying again in 60 seconds..."
                sleep 60
                ;;
        esac
    done
}

Then

    iptables -N ADBLOCK
    iptables -F ADBLOCK
    iptables -I ADBLOCK -j DROP
    iptables -I ADBLOCK -j LOG --log-prefix "### AD DROPED ###: " --log-level 6
    iptables -I OUTPUT -o eth+ -m set --match-set ADBLOCK src  -j ADBLOCK
    
    ip6tables -N ADBLOCK
    ip6tables -F ADBLOCK
    ip6tables -I ADBLOCK -j DROP
    ip6tables -I ADBLOCK -j LOG --log-prefix "### AD DROPED ###: " --log-level 6
    ip6tables -I OUTPUT -o eth+ -m set --match-set ADBLOCK src  -j ADBLOCK

or only just like that

    iptables -I OUTPUT -o eth+ -m set --match-set ADBLOCK src  -j DROP
    ip6tables -I OUTPUT -o eth+ -m set --match-set ADBLOCK src  -j DROP
And stop forcing dnsmasq!!!111

(Last edited by Amarh on 26 Dec 2015, 17:17)

Amarh wrote:
And stop forcing dnsmasq!!!111

Since you seem to be very passionate about the matter can you elaborate on what's wrong with the dnsmasq adblocking?

stangri wrote:
Amarh wrote:
And stop forcing dnsmasq!!!111

Since you seem to be very passionate about the matter can you elaborate on what's wrong with the dnsmasq adblocking?

I hurried, I apologize. I forgot to deliver ")".
Briefly, use not to destination can make operation of service unstable. For block of such large number of addresses is better to use ipset, which for this purpose and was written. I anyway tried to use dnsmasq for blocking by blacklist, in which I had about 60000 IP, and in a week unfortunate dnsmasq fell two times. Besides, dns started working only after all unit a list is processed.
I am delighted with the operation done by the author, but I want to emphasize that for these purposes it is better to use those means which were specially developed.
You can independently try to create such IPset and to look at a difference. The single insignificant shortcoming - that IPset's in itself doesn't remain, and, after reset they shall be created again. But it is solved very simply. You can look how it is realized in packets bcp38 and dns-ipset. I use dynamic sets therefore added cron job for saving a set every minute, and when boot up, before or with  a firewall start, I do

ipset -exist restore < /where/the set/is located/ipset
To the author anyway many thanks for the work!

And

wink

When you say "For block of such large number of addresses is better to use ipset, which for this purpose and was written" -- can you again elaborate in which way is it better? Less RAM used? Faster operation?

I don't see the fact that you need to recreate rules after reboot as a big problem, I'd rather have the router re-download the blocklists on each start than have it store the list on the device.

My concern is that using your rules would seem to result in the web-pages with "broken" items, would it not? If this can be worked out and if there's a demonstrable benefit of using ipset, would be great if you could collaborate with the author of this package (and ideally also the script which is used in arokh's builds) to transition to ipset use (or offer an option of using either dnsmasq or ipset) to redirect the ad requests to the pixelserv.

I'm currently set up to use the dnsmasq and I'm willing to be a guinea pig for the transition. I'm fairly inexperienced with iptables tho and have some questions. Since I don't want to further thread-jack and OpenWrt forums do not support PMs, how can I reach out to you?

stangri wrote:

When you say "For block of such large number of addresses is better to use ipset, which for this purpose and was written" -- can you again elaborate in which way is it better? Less RAM used? Faster operation?

I don't see the fact that you need to recreate rules after reboot as a big problem, I'd rather have the router re-download the blocklists on each start than have it store the list on the device.

My concern is that using your rules would seem to result in the web-pages with "broken" items, would it not? If this can be worked out and if there's a demonstrable benefit of using ipset, would be great if you could collaborate with the author of this package (and ideally also the script which is used in arokh's builds) to transition to ipset use (or offer an option of using either dnsmasq or ipset) to redirect the ad requests to the pixelserv.

I'm currently set up to use the dnsmasq and I'm willing to be a guinea pig for the transition. I'm fairly inexperienced with iptables tho and have some questions. Since I don't want to further thread-jack and OpenWrt forums do not support PMs, how can I reach out to you?

It is unlikely I will be able to be very useful owing to congestion).
As for block of arrays of addresses, iptables works at the logic level of filtering, at the kernel level of system, and the ipset module was written especially for the purposes when it is necessary to process the big IP lists of addresses. Dnsmasq is small dhcp server with the dns resolver functions. It isn't intended for filtering, and on embeddable systems with limited resources it can reduce productivity noticeably. I suggest to use that tool which was created for such purposes. For readdressing on pixelserv it is possible to change the rule to redirect, etc. Something like this

iptables -t nat -A OUTPUT -m set --match-set ADBLOCK src -j DNAT --to-destination IP:PORT

In any case, I would do somehow so))
I, unfortunately, have now no time to experiment. Perhaps in January... And adblock in the browser is more flexible and functional;)

@stangri/amarh: this adblock package is a dns based adblock solution for openwrt. Feel free to invent another package based on ipset, iptables, privoxy or something else. As a starting point for ipset check firehol, they're already support openwrt ...

back to vacation! smile

re. uhttp bind: yes, see https://wiki.openwrt.org/doc/uci/uhttpd#securing_uhttpd for securing uhttpd ...

re. new adblock source: no problem, I'll add this source with the next release

re. your posted link: did you read the first paragraph? smile


stonecarver wrote:

I'm stuck at something "bind uhttpd to the standard LAN port". How do I do this?
I'm on latest trunk. Can I still access LuCi on my router over WiFi then?

Edit:
Can you please add http://www.malwaredomainlist.com/hostslist/hosts.txt

Edit2:
And how does this compare to this adblock solution? https://gist.github.com/teffalump/7227752
Pros, cons?

Hi,

currently I'm working on ipv6 support and I found a promising approach to combine the dns based solution with ipset/iptables.
I need some testers, ideally with ipv6 uplink. Please drop me a short mail (see package maintainer address) if you're interested in pre-testing - thanks.

Thanks again & a happy new year!
Dirk

@Amarh: ip based ad-/abuse-blocking does not work reliable in real world, i.e. youtube.com and google-analytics.com are using the same server infrastructure - see nslookup queries below. Therefore you can't block google-analytics without blocking harmless youtube videos...

Name:      www-google-analytics.l.google.com
Address 1: 173.194.116.133 fra02s28-in-f5.1e100.net
Address 2: 173.194.116.130 fra02s28-in-f2.1e100.net
Address 3: 173.194.116.142 fra02s28-in-f14.1e100.net
Address 4: 173.194.116.131 fra02s28-in-f3.1e100.net
Address 5: 173.194.116.128 fra02s28-in-f0.1e100.net
Address 6: 173.194.116.134 fra02s28-in-f6.1e100.net
Address 7: 173.194.116.132 fra02s28-in-f4.1e100.net
Address 8: 173.194.116.136 fra02s28-in-f8.1e100.net
Address 9: 173.194.116.135 fra02s28-in-f7.1e100.net
Address 10: 173.194.116.137 fra02s28-in-f9.1e100.net
Address 11: 173.194.116.129 fra02s28-in-f1.1e100.net
Address 12: 2a00:1450:4001:80d::1005 fra02s28-in-x05.1e100.net

Name:      youtube.com
Address 1: 173.194.116.131 fra02s28-in-f3.1e100.net
Address 2: 173.194.116.135 fra02s28-in-f7.1e100.net
Address 3: 173.194.116.142 fra02s28-in-f14.1e100.net
Address 4: 173.194.116.133 fra02s28-in-f5.1e100.net
Address 5: 173.194.116.129 fra02s28-in-f1.1e100.net
Address 6: 173.194.116.132 fra02s28-in-f4.1e100.net
Address 7: 173.194.116.128 fra02s28-in-f0.1e100.net
Address 8: 173.194.116.130 fra02s28-in-f2.1e100.net
Address 9: 173.194.116.134 fra02s28-in-f6.1e100.net
Address 10: 173.194.116.137 fra02s28-in-f9.1e100.net
Address 11: 173.194.116.136 fra02s28-in-f8.1e100.net
Address 12: 2a00:1450:4001:80d::1003 fra02s28-in-x03.1e100.net

Hello all

I have installed the adblocker on " Chaos Calmer 15.05"

In the beginning it did not download the "blacklists" from the internet.
After starting the script with "bash -x" I got the commands the script issues and I was able the pin point the command that went wrong.

Below part of the "bash -x" output
*****************************************
++ curl -q --insecure --silent --max-time 60 'http://pgl.yoyo.org/adservers/serverlis … =plaintext'
+ tmp_domains=
+ rc=16
+ '[' 16 -eq 0 ']'
+ '[' 16 -eq 0 ']'
+ f_log 'source download failed (yoyo)' 16
+ local 'log_msg=source download failed (yoyo)'
+ local log_rc=16
+ local 'class=info '
+ '[' -n 'source download failed (yoyo)' ']'
+ '[' 16 -ne 0 ']'
+ class=error
+ log_rc=', rc: 16'
+ log_msg='source download failed (yoyo), rc: 16'
+ /usr/bin/logger -s -t 'adblock[28797] error' 'source download failed (yoyo), rc: 16'
adblock[28797] error: source download failed (yoyo), rc: 16
*********************************************************************

After this I took the "curl" commandline and started it right from putty. Then I got a message stating the libmbedtls.so.9  library was missing.

After some googling I found out the library is part of ssl communication packages. So I additionaly installed "libpolarssl" which seems the incorporate the missing library and voila the addblocker is functioning without errors.

Next step is testing if it blocks everything I want it to block.