Adblock support thread


#752

Ok... back now. That was silly... I guess I was pretty tired then...

root@OpenWrt:~# /etc/init.d/adblock status
::: adblock runtime information
  + adblock_status  : enabled
  + adblock_version : 3.6.5
  + overall_domains : 56964 (normal mode)
  + fetch_utility   : /bin/uclient-fetch (libustream-ssl)
  + dns_backend     : dnsmasq (/tmp)
  + last_rundate    : 11.02.2019 00:09:15
  + system_release  : NA ZBOX-CI327NANO-GS-01, OpenWrt 18.06.2 r7676-cddd7b4c77

And here's the json file:

root@OpenWrt:~# cat /tmp/adb_runtime.json
{ "data": { "adblock_status": "enabled", "adblock_version": "3.6.5", "overall_domains": "56964 (normal mode)", "fetch_utility": "\/bin\/uclient-fetch (libustream-ssl)", "dns_backend": "dnsmasq (\/tmp)", "last_rundate": "11.02.2019 00:09:15", "system_release": "NA ZBOX-CI327NANO-GS-01, OpenWrt 18.06.2 r7676-cddd7b4c77" } }

#753

OK, thank you. Your json file looks OK - as a last resort please reset/delete your LuCI cache (rm -rf /tmp/luci-*) and try again.


#754

Update and now work. Thanks.


#756
  • Does it apply on service adblock restart or only on reboot?
  • Does it apply to adb_trigger=timed?

#757

What's not working actually?
The discussed procd startup trigger affects only the boot behaviour. You can check the adblock service with this ubus call ...

ubus call service list '{"name":"adblock","verbose": true}'

... and you will see something like this ...

[...]
		"triggers": [
			[
				"interface.*.up",
				[
					"if",
					[
						"eq",
						"interface",
						"lan"
					],
					[
						"run_script",
						"\/etc\/init.d\/adblock",
						"start"
					]
				],
				30000
			],
[...]

That means: 30 seconds after the corresponding interface (e.g. 'lan') get's up, the service will be started.


#758

It's a 'hard' timeout during boot, mainly a workaround for TurrisOS 3.x which does not provide these startup triggers at all.


#759

Thank you, I've found it here:

# grep -A3 ' = "timed"' /usr/bin/adblock.sh 
	if [ "${adb_action}" = "start" ] && [ "${adb_trigger}" = "timed" ]
	then
		sleep ${adb_triggerdelay}
	fi

So the documentation states that default delay for adb_trigger=timed is (default 30 sec.), but it's actually adb_triggerdelay which is (int/default: '2').


#760

Well, if you look at the context ...

[...]
	if [ "${adb_action}" = "start" ] && [ "${adb_trigger}" = "timed" ]
	then
		sleep ${adb_triggerdelay}
	fi

	while [ ${cnt} -le 30 ]
	do
[...]

regardless of the used trigger setting, adblock waits 30 seconds for the dns backend ... plus 2 seconds by default if you've started with "timed" trigger. :wink:


#761

And, last resorts are sometimes where you have to go... Did the rm command, and got data coming back to me in the Runtime Information section... as well as the two buttons below it work. I can manually update the block lists now.


#762

Is it normal that unbound uses almost twice as much memory as dnsmasq?
I have added some more list from http://hosts-file.net/
Adblock file has around 500k entries.
dnsmasq uses 80mb of memory, while unbound uses 160mb.
I tried changed the local-zone entries from static to always_nxdomain but that didn't reduce memory usage.
Is it possible to have the ad_list_* files removed from tmp after the dnsbackend has started up to free up some memory?
Can you also add an option to modify the --parallel paramter of sort ?
Also is it possible to have all other tasks, like awk, run in parallel like shown here:
http://www.rankfocus.com/use-cpu-cores-linux-commands/
thank you.


#763

Which files did you mean? After adblock processing you should see only one big file (adb_list.overall) in your dns related directory. Of course, this file can't be removed ...

There is no such option in busybox sort. "parallel" is an external gnu utility which is not available in OpenWrt. Please note, that all download related tasks (incl. sort) already running in parallel (check/raise the 'adb_maxqueue' parameter) - only the final list preparation runs sequentially.


#764

Yes i mean the ad_list_* files.
Mine is around 20mb. Not that much. But keeping this in tmpfs is waste of ram...
Maybe an option to specify a path to keep the ad_list_* file on a permanent storage like an usb stick.
//edit
or maybe i should just move the entire unbound chroot to a permanent storage xS

I think i can be removed cause unbound (and dnsmasq) loads the entire file into ram at start up?
Maybe something like:

  • on adblock startup copy the adb_list_* files over to unbound chroot directory
  • after unbound has started, remove the adb_list_ files
    only problem here is when unbound gets restarted at runtime the file has to copied again

Also when backup mode is enabled and the adb_list_* files were stored on a permanent storage, can't the adb list sources processing be skipped everytime cause list sources haven't changed?

Yes, i know but i use a custom build with coreutils-sort, which does by default use all cores anyway.
But i was thinking about limiting the parallel processing maybe to numcores-1 or something like that...

Thats awk doing the preparation? On a large list it really takes some time.
(~500k entries on 1,3 Ghz Arm)

//edit2
Can you make adblock use the $UB_VARDIR from /usr/lib/unbound/defaults.sh pls?
To bad it isn't possible to specify the unbound chroot directory in /etc/config/unbound.
So i have to mod the Defaults.sh everytime. I think i have to open issue on github xS
Thank you.


#765

Please check the adblock readme (https://github.com/openwrt/packages/blob/master/net/adblock/files/README.md), there is already the option 'adb_dnsdir' ... :wink:

On boot you can configure it like that (enable the 'Backup mode').


#766

How could i miss that? xS
Thank you!

I have that enabled, it does skip the downloads, but adblock processes the entire lists again.
I mean, when the final adb file is stored on a permanent storage and backup mode is enabled, then the processing of the entire lists, actually can be skipped?


#767

Hmm, latest openwrt master "snapshot" made dnsmasq ran out of memory, not sure what caused it, haven't changed the configuration.
Adblock version 3.6.5

There's 512mb memory in the router, I usallly have 250mb free with adblock up and running, so OOM seems weird.

Anyone else noticed this?


#768

Hi,

Adblock is running well for me. I have a problem with DNS Query reports because although the router has been up for days, the "Start" time of the DNS Query report is usually only a few hours prior to the "End" time (i.e., current time). More specifically, given an uptime of a couple days shown on the Status Overview page:

Uptime 2d 0h 11m 0s

Then the first few lines of the report show the report start and end are only 4 hours apart:

:::
::: Adblock DNS-Query Report
:::
  + Start    ::: 2019-02-22, 12:17:14
  + End      ::: 2019-02-22, 16:31:54
  + Total    ::: 12342
  + Blocked  ::: 9228 (74.77 %)
:::

Whenever I come back and run a report, I get pretty much the same thing. The "End" time is the current time and the "Start" time is a few hours before it. Adblock is enabled as shown through /etc/init.d/adblock status. I'm using a Linksys WRT1900ACS running the most recent davidc502 build which has adblock 3.6.5 installed.

I've poked around to see if there's an option I'm setting incorrectly, but haven't seen anything. The /tmp filesystem has plenty of room available and shows only 5% use. There is a tcpdump process running in support of adblock. Any advice?

Thanks


#769

Yes, check the readme - online available here! :wink:

options

... and in LuCI here:

Please be aware that any tcpdump change needs a full stop/start cycle of adblock, cause the tcpdump background task must be started with new parameters ... said that, you can check these with the following ps command (you change the "-C" and "-W" parameters).


#770

Thanks! I did see the 'chunksize' in the readme, but the description didn't quite lead me to understanding that the parameters would control how much "effective time" of query that I'd get. :grinning: In fact I'm still not quite sure. Basically I only get a certain number of chunks (defined by repchunkcnt) that are a certain size (repchunksize)? Which should I change? Or does it effectively not matter because the important parameter is actually repchunkcnt * repchunksize?

Thanks for all your work!


#771

That depends on how chatty your LAN is ... by default the tcpdump background job is configured to use max. 5 chunks of 1 megabyte each to capture your network traffic in a "FIFO" loop. If you need long term statistics, change the report directory to a non volatile e.g. usb stick and raise the chunk size at least.


#772

Hi there! I am really enjoying adblocking on my wrt device. thank you so much for making this possible.

Now I am trying to add custom adblocking lists. Reading the github page this might not be super trivial as in most cases an awk-one-liner is required.

What would it take to add the poular list of firebog.net to the adblock backend in openwrt?
Would it be possible to have recursive lists like the following: https://v.firebog.net/hosts/lists.php?type=tick ?

Any help to further reading/studying material is much appreciated !

cheers