Adblock support thread

gt203 · December 10, 2018, 8:59pm

Hi dibdot!
I need your advice: adblock is counting overall twice.

My log:

Sun Dec  9 11:28:15 2018 user.info adblock-3.5.5-2[31809]: adblock instance started ::: action: reload, priority: 0, pid: 31809
Sun Dec  9 11:28:15 2018 user.debug adblock-3.5.5-2[31809]: f_jsnup::: status: running, mode: normal mode, cnt: 35742, notify: 0, notify_cnt: 0, notify_pid: -
Sun Dec  9 11:28:15 2018 user.debug adblock-3.5.5-2[31809]: f_uci  ::: config: dhcp, change:
Sun Dec  9 11:28:15 2018 user.debug adblock-3.5.5-2[31809]: f_uci  ::: config: firewall, change:
Sun Dec  9 11:28:15 2018 user.debug adblock-3.5.5-2[31809]: f_main ::: dns: dnsmasq, fetch_util: /bin/uclient-fetch (libustream-ssl), backup: 0, backup_mode: 0, dns_jail: 0, force_srt: 0, force_dns: 1, mem_total: 60, mem_free: 23, max_queue: 16
Sun Dec  9 11:28:15 2018 user.debug adblock-3.5.5-2[31809]: f_main ::: name: cameleon_pihole_def, enabled: 1
Sun Dec  9 11:28:15 2018 user.debug adblock-3.5.5-2[31809]: f_main ::: name: malware_pihole_def, enabled: 1
Sun Dec  9 11:28:15 2018 user.debug adblock-3.5.5-2[31809]: f_main ::: name: disconnect_malware, enabled: 1
Sun Dec  9 11:28:15 2018 user.debug adblock-3.5.5-2[31809]: f_main ::: name: disconnect_ad_pihole_def, enabled: 1
Sun Dec  9 11:28:15 2018 user.debug adblock-3.5.5-2[31809]: f_main ::: name: zeus_pihole_def, enabled: 1
Sun Dec  9 11:28:15 2018 user.debug adblock-3.5.5-2[31809]: f_main ::: name: blacklist, enabled: 1
Sun Dec  9 11:28:15 2018 user.debug adblock-3.5.5-2[31809]: f_list ::: name: blacklist, mode: download, cnt: 10, in_rc: 0, out_rc: 0
Sun Dec  9 11:28:16 2018 user.debug adblock-3.5.5-2[31809]: f_tld  ::: source: /tmp/tmp.kjDhnf/tmp.BdkPcG.blacklist, cnt: 10, cnt_srt: 10, cnt_tld: 10
Sun Dec  9 11:28:18 2018 user.debug adblock-3.5.5-2[31809]: f_tld  ::: source: /tmp/tmp.kjDhnf/tmp.BdkPcG.zeus_pihole_def, cnt: 381, cnt_srt: 381, cnt_tld: 380
Sun Dec  9 11:28:18 2018 user.debug adblock-3.5.5-2[31809]: f_list ::: name: disconnect_malware, mode: download, cnt: 2831, in_rc: 0, out_rc: 0
Sun Dec  9 11:28:18 2018 user.debug adblock-3.5.5-2[31809]: f_list ::: name: disconnect_ad_pihole_def, mode: download, cnt: 2703, in_rc: 0, out_rc: 0
Sun Dec  9 11:28:24 2018 user.debug adblock-3.5.5-2[31809]: f_tld  ::: source: /tmp/tmp.kjDhnf/tmp.BdkPcG.disconnect_ad_pihole_def, cnt: 2703, cnt_srt: 2703, cnt_tld: 2668
Sun Dec  9 11:28:24 2018 user.debug adblock-3.5.5-2[31809]: f_tld  ::: source: /tmp/tmp.kjDhnf/tmp.BdkPcG.disconnect_malware, cnt: 2831, cnt_srt: 2831, cnt_tld: 2790
Sun Dec  9 11:28:24 2018 user.debug adblock-3.5.5-2[31809]: f_list ::: name: cameleon_pihole_def, mode: download, cnt: 20567, in_rc: 0, out_rc: 0
Sun Dec  9 11:28:34 2018 user.debug adblock-3.5.5-2[31809]: f_tld  ::: source: /tmp/tmp.kjDhnf/tmp.BdkPcG.cameleon_pihole_def, cnt: 20567, cnt_srt: 20567, cnt_tld: 16409
Sun Dec  9 11:28:54 2018 user.debug adblock-3.5.5-2[31809]: f_list ::: name: malware_pihole_def, mode: download, cnt: 26850, in_rc: 0, out_rc: 0
Sun Dec  9 11:29:06 2018 user.debug adblock-3.5.5-2[31809]: f_tld  ::: source: /tmp/tmp.kjDhnf/tmp.BdkPcG.malware_pihole_def, cnt: 26850, cnt_srt: 26850, cnt_tld: 26362
Sun Dec  9 11:29:07 2018 user.debug adblock-3.5.5-2[31809]: f_list ::: name: overall, mode: merge, cnt: 97238, in_rc: 0, out_rc: 0
Sun Dec  9 11:29:07 2018 user.debug adblock-3.5.5-2[31809]: f_hash ::: hash_util: /usr/bin/sha256sum, hash: 68da62929ec9c976afb1add6172756da231c369e6a85446dd48dcd848c1163f5, out_rc: 1
Sun Dec  9 11:29:12 2018 user.debug adblock-3.5.5-2[31809]: f_list ::: name: overall, mode: final, cnt: 97238, in_rc: 0, out_rc: 0

Summary of all the "cnt_tld" is exactly the half of the "overall cnt"
How can that be?

Thank you!

gt203 · December 10, 2018, 9:09pm

Problem (?) nr. 2:

if i only load 1 list, it counts more at the end as it is in the list!
For example malware list :
entries: 27.043
adblock count: 52.724
See log below:

Mon Dec 10 22:02:12 2018 user.debug adblock-3.5.5-2[6478]: f_list ::: name: malware_pihole_def, mode: download, cnt: 26850, in_rc: 0, out_rc: 0
Mon Dec 10 22:02:24 2018 user.debug adblock-3.5.5-2[6478]: f_tld  ::: source: /tmp/tmp.PEafGi/tmp.fJEfbN.malware_pihole_def, cnt: 26850, cnt_srt: 26850, cnt_tld: 26362
Mon Dec 10 22:02:24 2018 user.debug adblock-3.5.5-2[6478]: f_list ::: name: overall, mode: merge, cnt: 52724, in_rc: 0, out_rc: 0
Mon Dec 10 22:02:24 2018 user.debug adblock-3.5.5-2[6478]: f_hash ::: hash_util: /usr/bin/sha256sum, hash: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855, out_rc: 1
Mon Dec 10 22:02:27 2018 user.debug adblock-3.5.5-2[6478]: f_list ::: name: overall, mode: final, cnt: 52724, in_rc: 0, out_rc: 0

Thank you!

dibdot · December 11, 2018, 8:29am

Clearly a bug in the f_tld() function, seen only on low memory devices - will be fixed in 3.6.0. In the meantime you could change it in /usr/bin/adblock.sh on your own:

old: line 577 ff.

[...]
			if [ ${?} -eq 0 ]
				then
					cnt_tld="$(wc -l 2>/dev/null < "${source}")"
				else
					mv -f "${temp}" > "${source}"
				fi
fi
[...]

new, added missing "rm" command :

[...]
			if [ ${?} -eq 0 ]
				then
					cnt_tld="$(wc -l 2>/dev/null < "${source}")"
                                        rm -f "${temp}"   # that's the bugfix!
				else
					mv -f "${temp}" > "${source}"
				fi
fi
[...]

Sorry for the mess!

gt203 · December 11, 2018, 2:08pm

Thanx dibdot!
It works , but it seems tld is not working anymore?
with bugfix: "cnt_tld: -"
without bugfix: "cnt_tld: 2668"

See logs:

with bugfix

Tue Dec 11 14:55:07 2018 user.debug adblock-3.5.5-2[6440]: f_tld  ::: source: /tmp/tmp.CADBOK/tmp.LAAJnc.disconnect_ad_pihole_def, cnt: 2703, cnt_srt: 2703, cnt_tld: -
Tue Dec 11 14:55:07 2018 user.debug adblock-3.5.5-2[6440]: f_tld  ::: source: /tmp/tmp.CADBOK/tmp.LAAJnc.disconnect_malware, cnt: 2828, cnt_srt: 2828, cnt_tld: -
Tue Dec 11 14:55:07 2018 user.debug adblock-3.5.5-2[6440]: f_list ::: name: overall, mode: merge, cnt: 5456, in_rc: 0, out_rc: 0
Tue Dec 11 14:55:07 2018 user.debug adblock-3.5.5-2[6440]: f_hash ::: hash_util: /usr/bin/sha256sum, hash: 01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b, out_rc: 1
Tue Dec 11 14:55:07 2018 user.debug adblock-3.5.5-2[6440]: f_list ::: name: overall, mode: final, cnt: 5456, in_rc: 0, out_rc: 0
Tue Dec 11 14:55:07 2018 user.debug adblock-3.5.5-2[6440]: f_hash ::: hash_util: /usr/bin/sha256sum, hash: e26d71fcb03cd82d6058a9ef1479665f685da05e0b358b27ec18ba780f61fdb0, out_rc: 1
Tue Dec 11 14:55:11 2018 user.debug adblock-3.5.5-2[6440]: f_dnsup::: cache_util: -, cache_rc: -, cache_flush: 1, cache_cnt: 0, rc: 0
Tue Dec 11 14:55:11 2018 user.debug adblock-3.5.5-2[6440]: f_jsnup::: status: enabled, mode: normal mode, cnt: 5456, notify: 0, notify_cnt: 0, notify_pid: -
Tue Dec 11 14:55:11 2018 user.info adblock-3.5.5-2[6440]: blocklist with overall 5456 domains loaded successfully (TP-Link TL-WR1043N/ND v3, OpenWrt 18.06.1 r7258-5eb055306f)

without bugfix

Tue Dec 11 15:02:00 2018 user.debug adblock-3.5.5-2[7230]: f_tld  ::: source: /tmp/tmp.FOEMBh/tmp.kpLFCK.disconnect_ad_pihole_def, cnt: 2703, cnt_srt: 2703, cnt_tld: 2668
Tue Dec 11 15:02:00 2018 user.debug adblock-3.5.5-2[7230]: f_tld  ::: source: /tmp/tmp.FOEMBh/tmp.kpLFCK.disconnect_malware, cnt: 2828, cnt_srt: 2828, cnt_tld: 2788
Tue Dec 11 15:02:00 2018 user.debug adblock-3.5.5-2[7230]: f_list ::: name: overall, mode: merge, cnt: 10912, in_rc: 0, out_rc: 0
Tue Dec 11 15:02:00 2018 user.debug adblock-3.5.5-2[7230]: f_hash ::: hash_util: /usr/bin/sha256sum, hash: 01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b, out_rc: 1
Tue Dec 11 15:02:01 2018 user.debug adblock-3.5.5-2[7230]: f_list ::: name: overall, mode: final, cnt: 10912, in_rc: 0, out_rc: 0
Tue Dec 11 15:02:01 2018 user.debug adblock-3.5.5-2[7230]: f_hash ::: hash_util: /usr/bin/sha256sum, hash: 059c2450db2bec061477fc79e3755923663bfcac6bf98b322774b9f189d1edc8, out_rc: 1
Tue Dec 11 15:02:04 2018 user.debug adblock-3.5.5-2[7230]: f_dnsup::: cache_util: -, cache_rc: -, cache_flush: 1, cache_cnt: 0, rc: 0
Tue Dec 11 15:02:05 2018 user.debug adblock-3.5.5-2[7230]: f_jsnup::: status: enabled, mode: normal mode, cnt: 10912, notify: 0, notify_cnt: 0, notify_pid: -
Tue Dec 11 15:02:05 2018 user.info adblock-3.5.5-2[7230]: blocklist with overall 10912 domains loaded successfully (TP-Link TL-WR1043N/ND v3, OpenWrt 18.06.1 r7258-5eb055306f)

Thank you!

dibdot · December 11, 2018, 3:05pm

No, the change is unrelated to that counter. Two options that comes in my mind:

you've messed up the tld function with a wrong placed insert
you've run in an (unrelated) OOM condition on your router (for whatever reason)

Please contact me directly via dev@brenken.org and I'll send you a fixed adblock version in return.

dibdot · December 20, 2018, 10:27am

Just in time before Christmas I've pushed adblock release 3.6.0 to master branch. Major change is the new "DNS Query Report", one of the most pihole like feature requests ...

In LuCI it looks like this:
dns_report_1
dns_report2

You'll get a new "Top10" reporting and the last 50 requested domains. Whithin the domain list you could directly blacklist or whitelist these domains.

This report is also available via console (only the Top10 reporting part!).

To make use of this new feature, please install package "tcpdump-mini" and start with a fresh adblock config (use the "--force-maintainer" option during opkg install). By default tcpdump will be started as a background task and write the pcap files to "/tmp" (in a loop max. 5 files in chunks of 1 MByte) ... of course, you could configure that to your needs.

BTW, please also reset your luci caches (rm -rf /tmp/luci-*)!

Merry Christmas!

oli · December 22, 2018, 3:24pm

Hello
@dibdo Congratulations for everything you do!
We have the possibility to exclude certain sites if we know we are interested in accessing them, but they are blocked?
When clicking the whitelist button, that site must arrive in the white list, right?
From what i see does not work.

dibdot · December 22, 2018, 5:45pm

Does the domain already exist in your whitelist? Only new/not existing domains will be added - just noticed that there was a bug in this function - fix is on the way with this PR (https://github.com/openwrt/luci/pull/2389).

oli · December 22, 2018, 6:53pm

Yes he appears to be in the whitelist, only that it continues to be blocked.

dibdot · December 22, 2018, 9:30pm

Please refresh/restart adblock processing after adding new domains to white- or blacklist.

leeandy · December 23, 2018, 8:18am

Hi @dibdot
I have multi vlan with difference subnet segment on same router. Can I set with multi report interface ?

shelterx · December 23, 2018, 8:19am

Interesting statistics, my chromecast seems to be very chatty, in just 1 day with no real usage (I haven't watched anything using Chromecast)
Count 1736 - ChromeCast

dibdot · December 23, 2018, 9:17am

yeah, it's a google product ...

dibdot · December 23, 2018, 9:28am

How many dns instances did you use? Currently tcpdump only listens on port 53 on "br-lan" interface - the latter one is configurable.

leeandy · December 23, 2018, 9:32am

Two dns instances. Dns service listen on eth0.1 & eth0.2, work on port 53. Tks.

vbarter · December 25, 2018, 9:50pm

Is this the proper way to add the snapshots source? I placed this in the Distribution feeds section.

src/gz openwrt_snapshot https://downloads.openwrt.org/snapshots/packages/x86_64/packages/Packages

oli · December 26, 2018, 8:56am

Everything works well, thanks for the answer!

herik · December 26, 2018, 10:56am

Hi please the youtube skip ad is still not removed or not blocked by adblock

Is there any way to block it?

oli · December 26, 2018, 11:00am

I also noticed this problem, on wifi side does not block the ads on YouTube.
I use the version compiled by Daniel Petre for wrt1900acs v2, https://superwrt.download/f/ Kernel Version 4.14.90 SNAPSHOT r8392+474-3dba852547 / LuCI Master (git-18.354.50772-49c558a) Adblock Version 3.6.0
adblock

leeandy · December 26, 2018, 1:54pm

@herik @oli
Hard to block 100% ads on youtube, because:
dibdot says:
Please remember that you block always on (sub-)domain level (and not on page element level!) - the same applies to other projects like pihole. For remaining ads on advertising platforms like youtube, facebook etc. it's likely that you need additional browser plugins which are hiding ads on DOM level, e.g. injecting JavaScript to skip the YouTube ads ... anyway you'll never get 100%.