dibot,
Thanks for the advice. Moving to a usb stick worked well for having multi-day DNS query logs. Of course, what I learned primarily is that the rokus I have in my house are collectively making thousands and thousands of DNS requests that are the major things that adblock is taking care of. The top 5 blocked domains are roku and 90% of the blocked total.
1 Like
I built a new OpenWRT master build today, still getting OOM on dnsmasq, I guess I have to turn off adblock for now and see if it's dnsmasq+adblock that's causing issues.
Not sure where to report this really.
Hi Dirk,
I am also using backup mode and store the files on a USB drive but everytime I reboot the router, new sources are downloaded. Is this expected behavior? At least I didn't expect ;- )
I would like to skip the downloads if I use backup mode but more importent, I would like to skip processing time. Isn't this the purpose of the backup mode? Possible?
regards, Eric
Not without changing the script further - as by default it expects to run awk at that point with the expression in adb_src_rset.
An alternative optimization is to have a cron running on a vps etc. that downloads all lists you want and then processes them into a single file that you download from openwrt (you can even sort and so on offboard).
You can always add new sources to the config.
Like this:
config source 'adaway'
option adb_src 'https://adaway.org/hosts.txt'
option adb_src_rset '/^127\.0\.0\.1[[:space:]]+([[:alnum:]_-]+\.)+[[:alpha:]]+([[:space:]]|$)/{print tolower(\$2)}'
option adb_src_desc 'focus on mobile ads, infrequent updates, approx. 400 entries'
option enabled '1'
config source 'adguard'
option adb_src 'https://filters.adtidy.org/windows/filters/15.txt'
option adb_src_rset 'BEGIN{FS=\"[/|^|\r]\"}/^\|\|([[:alnum:]_-]+\.)+[[:alpha:]]+[\/\^\r]+$/{print tolower(\$3)}'
option adb_src_desc 'combined adguard dns filter list, frequent updates, approx. 17.000 entries'
option enabled '1'
config source 'blacklist'
option adb_src '/etc/adblock/adblock.blacklist'
option adb_src_rset '/^([[:alnum:]_-]+\.)+[[:alpha:]]+([[:space:]]|$)/{print tolower(\$1)}'
option adb_src_desc 'static local domain blacklist, always deny these domains'
option enabled '1'
config source 'coinblocker_1'
option adb_src 'https://raw.githubusercontent.com/hoshsadiq/adblock-nocoin-list/master/hosts.txt'
option adb_src_rset '/^0\.0\.0\.0[[:space:]]+([[:alnum:]_-]+\.)+[[:alpha:]]+([[:space:]]|$)/{print tolower(\$2)}'
option adb_src_desc 'focus on malicious bitcoin mining sites, infrequent updates, approx. 600 entries'
option enabled '1'
config source 'coinblocker_2'
option adb_src 'https://zerodot1.gitlab.io/CoinBlockerLists/hosts'
option adb_src_rset '/^0\.0\.0\.0[[:space:]]+([[:alnum:]_-]+\.)+[[:alpha:]]+([[:space:]]|$)/{print tolower(\$2)}'
option adb_src_desc 'focus on malicious bitcoin mining sites, infrequent updates, approx. 16.000 entries'
option enabled '1'
config source 'disconnect_ads'
option adb_src 'https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt'
option adb_src_rset '/^([[:alnum:]_-]+\.)+[[:alpha:]]+([[:space:]]|$)/{print tolower(\$1)}'
option adb_src_desc 'mozilla driven blocklist, focus on ads, numerous updates on the same day, approx. 2.700 entries'
option enabled '1'
config source 'disconnect_malvertising'
option adb_src 'https://s3.amazonaws.com/lists.disconnect.me/simple_malvertising.txt'
option adb_src_rset '/^([[:alnum:]_-]+\.)+[[:alpha:]]+([[:space:]]|$)/{print tolower(\$1)}'
option adb_src_desc 'mozilla driven blocklist, focus on malvertising, numerous updates on the same day, approx. 4.700 entries'
option enabled '1'
config source 'disconnect_malware'
option adb_src 'https://s3.amazonaws.com/lists.disconnect.me/simple_malware.txt'
option adb_src_rset '/^([[:alnum:]_-]+\.)+[[:alpha:]]+([[:space:]]|$)/{print tolower(\$1)}'
option adb_src_desc 'mozilla driven blocklist, focus on malware, numerous updates on the same day, approx. 50 entries'
option enabled '1'
config source 'dshield'
option adb_src 'https://www.dshield.org/feeds/suspiciousdomains_Medium.txt'
option adb_src_rset '/^([[:alnum:]_-]+\.)+[[:alpha:]]+([[:space:]]|$)/{print tolower(\$1)}'
option adb_src_desc 'generic blocklist, daily updates, approx. 2000 entries'
option enabled '1'
config source 'hphosts_ats'
option adb_src 'https://hosts-file.net/ad_servers.txt'
option adb_src_rset '/^127\.0\.0\.1[[:space:]]+([[:alnum:]_-]+\.)+[[:alpha:]]+([[:space:]]|\$)+/{print tolower(\$2)}'
option adb_src_desc 'ad/tracking servers, monthly updates, approx. 45.000 entries'
option enabled '1'
config source 'hphosts_emd'
option adb_src 'https://hosts-file.net/emd.txt'
option adb_src_rset '/^127\.0\.0\.1[[:space:]]+([[:alnum:]_-]+\.)+[[:alpha:]]+([[:space:]]|\$)+/{print tolower(\$2)}'
option adb_src_desc 'malware sites, monthly updates, approx. 250.000 entries'
option enabled '1'
config source 'hphosts_exp'
option adb_src 'https://hosts-file.net/exp.txt'
option adb_src_rset '/^127\.0\.0\.1[[:space:]]+([[:alnum:]_-]+\.)+[[:alpha:]]+([[:space:]]|\$)+/{print tolower(\$2)}'
option adb_src_desc 'exploit sites, monthly updates, approx. 1.000 entries'
option enabled '1'
config source 'hphosts_fsa'
option adb_src 'https://hosts-file.net/fsa.txt'
option adb_src_rset '/^127\.0\.0\.1[[:space:]]+([[:alnum:]_-]+\.)+[[:alpha:]]+([[:space:]]|\$)+/{print tolower(\$2)}'
option adb_src_desc 'fraud sites, monthly updates, approx. 190.000 entries'
option enabled '1'
config source 'hphosts_grm'
option adb_src 'https://hosts-file.net/grm.txt'
option adb_src_rset '/^127\.0\.0\.1[[:space:]]+([[:alnum:]_-]+\.)+[[:alpha:]]+([[:space:]]|\$)+/{print tolower(\$2)}'
option adb_src_desc 'sites involved in spam, monthly updates, approx. 500 entries'
option enabled '1'
config source 'hphosts_hjk'
option adb_src 'https://hosts-file.net/hjk.txt'
option adb_src_rset '/^127\.0\.0\.1[[:space:]]+([[:alnum:]_-]+\.)+[[:alpha:]]+([[:space:]]|\$)+/{print tolower(\$2)}'
option adb_src_desc 'hijack sites, monthly updates, approx. 5.000 entries'
option enabled '1'
config source 'hphosts_mmt'
option adb_src 'https://hosts-file.net/mmt.txt'
option adb_src_rset '/^127\.0\.0\.1[[:space:]]+([[:alnum:]_-]+\.)+[[:alpha:]]+([[:space:]]|\$)+/{print tolower(\$2)}'
option adb_src_desc 'misleading marketing, monthly updates, approx. 4.000 entries'
option enabled '1'
config source 'hphosts_pha'
option adb_src 'https://hosts-file.net/pha.txt'
option adb_src_rset '/^127\.0\.0\.1[[:space:]]+([[:alnum:]_-]+\.)+[[:alpha:]]+([[:space:]]|\$)+/{print tolower(\$2)}'
option adb_src_desc 'illegal pharmacy sites, monthly updates, approx. 15.000 entries'
option enabled '1'
config source 'hphosts_psh'
option adb_src 'https://hosts-file.net/psh.txt'
option adb_src_rset '/^127\.0\.0\.1[[:space:]]+([[:alnum:]_-]+\.)+[[:alpha:]]+([[:space:]]|\$)+/{print tolower(\$2)}'
option adb_src_desc 'phishing sites, monthly updates, approx. 170.000 entries'
option enabled '1'
config source 'hphosts_pup'
option adb_src 'https://hosts-file.net/pup.txt'
option adb_src_rset '/^127\.0\.0\.1[[:space:]]+([[:alnum:]_-]+\.)+[[:alpha:]]+([[:space:]]|\$)+/{print tolower(\$2)}'
option adb_src_desc 'pup sites, monthly updates, approx. 30.000 entries'
option enabled '1'
config source 'malware'
option adb_src 'https://mirror.espoch.edu.ec/malwaredomains/justdomains'
option adb_src_rset '/^([[:alnum:]_-]+\.)+[[:alpha:]]+([[:space:]]|$)/{print tolower(\$1)}'
option adb_src_desc 'broad blocklist, daily updates, approx. 18.300 entries'
option enabled '1'
config source 'malwarelist'
option adb_src 'http://www.malwaredomainlist.com/hostslist/hosts.txt'
option adb_src_rset '/^127\.0\.0\.1[[:space:]]+([[:alnum:]_-]+\.)+[[:alpha:]]+([[:space:]]|$)/{print tolower(\$2)}'
option adb_src_desc 'focus on malware, daily updates, approx. 1.200 entries'
option enabled '1'
config source 'openphish'
option adb_src 'https://openphish.com/feed.txt'
option adb_src_rset 'BEGIN{FS=\"/\"}/^http[s]?:\/\/([[:alnum:]_-]+\.)+[[:alpha:]]+(\/|$)/{print tolower(\$3)}'
option adb_src_desc 'focus on phishing, numerous updates on the same day, approx. 2.400 entries'
option enabled '0'
config source 'ransomware'
option adb_src 'https://ransomwaretracker.abuse.ch/downloads/RW_DOMBL.txt'
option adb_src_rset '/^([[:alnum:]_-]+\.)+[[:alpha:]]+([[:space:]]|$)/{print tolower(\$1)}'
option adb_src_desc 'focus on ransomware by abuse.ch, numerous updates on the same day, approx. 1900 entries'
option enabled '1'
config source 'reg_cn'
option adb_src 'https://easylist-downloads.adblockplus.org/easylistchina+easylist.txt'
option adb_src_rset 'BEGIN{FS=\"[|^]\"}/^\|\|([[:alnum:]_-]+\.)+[[:alpha:]]+\^("\\\$third-party")?$/{print tolower(\$3)}'
option adb_src_desc 'focus on chinese ads plus generic easylist additions, daily updates, approx. 11.700 entries'
option enabled '0'
config source 'reg_cz'
option adb_src 'https://raw.githubusercontent.com/qxstyles/turris-hole-czech-block-list/master/turris-hole-czech-block-list'
option adb_src_rset '/^([[:alnum:]_-]+\.)+[[:alpha:]]+([[:space:]]|$)/{print tolower(\$1)}'
option adb_src_desc 'focus on czech ads maintained by Turris Omnia Users, infrequent updates, approx. 100 entries'
option enabled '0'
config source 'reg_de'
option adb_src 'https://easylist-downloads.adblockplus.org/easylistgermany+easylist.txt'
option adb_src_rset 'BEGIN{FS=\"[|^]\"}/^\|\|([[:alnum:]_-]+\.)+[[:alpha:]]+\^("\\\$third-party")?$/{print tolower(\$3)}'
option adb_src_desc 'focus on german ads plus generic easylist additions, daily updates, approx. 9.200 entries'
option enabled '1'
config source 'reg_id'
option adb_src 'https://easylist-downloads.adblockplus.org/abpindo+easylist.txt'
option adb_src_rset 'BEGIN{FS=\"[|^]\"}/^\|\|([[:alnum:]_-]+\.)+[[:alpha:]]+\^("\\\$third-party")?$/{print tolower(\$3)}'
option adb_src_desc 'focus on indonesian ads plus generic easylist additions, weekly updates, approx. 9.600 entries'
option enabled '0'
config source 'reg_nl'
option adb_src 'https://easylist-downloads.adblockplus.org/easylistdutch+easylist.txt'
option adb_src_rset 'BEGIN{FS=\"[|^]\"}/^\|\|([[:alnum:]_-]+\.)+[[:alpha:]]+\^("\\\$third-party")?$/{print tolower(\$3)}'
option adb_src_desc 'focus on dutch ads plus generic easylist additions, weekly updates, approx. 9.400 entries'
option enabled '0'
config source 'reg_pl'
option adb_src 'http://adblocklist.org/adblock-pxf-polish.txt'
option adb_src_rset 'BEGIN{FS=\"[|^]\"}/^\|\|([[:alnum:]_-]+\.)+[[:alpha:]]+\^("\\\$third-party")?$/{print tolower(\$3)}'
option adb_src_desc 'focus on polish ads, daily updates, approx. 90 entries'
option enabled '0'
config source 'reg_ro'
option adb_src 'https://easylist-downloads.adblockplus.org/rolist+easylist.txt'
option adb_src_rset 'BEGIN{FS=\"[|^]\"}/^\|\|([[:alnum:]_-]+\.)+[[:alpha:]]+\^("\\\$third-party")?$/{print tolower(\$3)}'
option adb_src_desc 'focus on romanian ads plus generic easylist additions, weekly updates, approx. 9.400 entries'
option enabled '0'
config source 'reg_ru'
option adb_src 'https://easylist-downloads.adblockplus.org/ruadlist+easylist.txt'
option adb_src_rset 'BEGIN{FS=\"[|^]\"}/^\|\|([[:alnum:]_-]+\.)+[[:alpha:]]+\^("\\\$third-party")?$/{print tolower(\$3)}'
option adb_src_desc 'focus on russian ads plus generic easylist additions, weekly updates, approx. 14.500 entries'
option enabled '0'
config source 'shalla'
option adb_src 'http://www.shallalist.de/Downloads/shallalist.tar.gz'
option adb_src_rset '/^([[:alnum:]_-]+\.)+[[:alpha:]]+([[:space:]]|$)/{print tolower(\$1)}'
option adb_src_desc 'huge blocklist archive subdivided in different categories, daily updates. Check http://www.shallalist.de/categories.html for more categories'
list adb_src_cat 'adv'
list adb_src_cat 'costtraps'
list adb_src_cat 'spyware'
list adb_src_cat 'tracker'
option enabled '1'
config source 'spam404'
option adb_src 'https://raw.githubusercontent.com/Dawsey21/Lists/master/main-blacklist.txt'
option adb_src_rset '/^([[:alnum:]_-]+\.)+[[:alpha:]]+([[:space:]]|$)+/{print tolower(\$1)}'
option adb_src_desc 'generic blocklist, infrequent updates, approx. 6.000 entries'
option enabled '1'
config source 'sysctl'
option adb_src 'http://sysctl.org/cameleon/hosts'
option adb_src_rset '/^127\.0\.0\.1[[:space:]]+([[:alnum:]_-]+\.)+[[:alpha:]]+([[:space:]]|$)/{print tolower(\$2)}'
option adb_src_desc 'broad blocklist, weekly updates, approx. 16.500 entries'
option enabled '1'
config source 'urlhaus'
option adb_src 'https://urlhaus.abuse.ch/downloads/rpz'
option adb_src_rset '/^([[:alnum:]_-]+\.)+[[:alpha:]]+([[:space:]]|$)+/{print tolower(\$1)}'
option adb_src_desc 'urlhaus RPZ domains by abuse.ch, numerous updates on the same day, approx. 3.500 entries'
option enabled '1'
config source 'ut_capitole'
option adb_src 'https://dsi.ut-capitole.fr/blacklists/download/blacklists.tar.gz'
option adb_src_rset '/^([[:alnum:]_-]+\.)+[[:alpha:]]+([[:space:]]|$)/{print tolower(\$1)}'
option adb_src_desc 'huge blocklist archive subdivided in different categories, daily updates. Check https://dsi.ut-capitole.fr/blacklists/index_en.php for more categories'
option enabled '1'
list adb_src_cat 'cryptojacking'
list adb_src_cat 'malware'
list adb_src_cat 'phishing'
list adb_src_cat 'publicite'
config source 'whocares'
option adb_src 'http://someonewhocares.org/hosts/hosts'
option adb_src_rset '/^127\.0\.0\.1[[:space:]]+([[:alnum:]_-]+\.)+[[:alpha:]]+([[:space:]]|$)/{print tolower(\$2)}'
option adb_src_desc 'broad blocklist, weekly updates, approx. 10.000 entries'
option enabled '1'
config source 'winspy'
option adb_src 'https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/spy.txt'
option adb_src_rset '/^0\.0\.0\.0[[:space:]]+([[:alnum:]_-]+\.)+[[:alpha:]]+([[:space:]]|$)/{print tolower(\$2)}'
option adb_src_desc 'focus on windows spy & telemetry domains, infrequent updates, approx. 300 entries'
option enabled '1'
config source 'winhelp'
option adb_src 'http://winhelp2002.mvps.org/hosts.txt'
option adb_src_rset '/^0\.0\.0\.0[[:space:]]+([[:alnum:]_-]+\.)+[[:alpha:]]+([[:space:]]|$)/{print tolower(\$2)}'
option adb_src_desc 'broad blocklist, infrequent updates, approx. 13.000 entries'
option enabled '1'
config source 'youtube'
option adb_src 'https://api.hackertarget.com/hostsearch/?q=googlevideo.com'
option adb_src_rset 'BEGIN{FS=\",\"}/^(r[0-9]+\.){1,1}([[:alnum:]_-]+\.)+[[:alpha:]]+/{sub(/\./,\"---\");print tolower(\$1)}'
option adb_src_desc 'focus on youtube ad-related subdomains, dynamic request API, approx. 150 entries'
option enabled '0'
config source 'yoyo'
option adb_src 'https://pgl.yoyo.org/adservers/serverlist.php?hostformat=nohtml&showintro=0&mimetype=plaintext'
option adb_src_rset '/^([[:alnum:]_-]+\.)+[[:alpha:]]+([[:space:]]|$)/{print tolower(\$1)}'
option adb_src_desc 'focus on ad related domains, weekly updates, approx. 2.400 entries'
option enabled '1'
config source 'zeus'
option adb_src 'https://zeustracker.abuse.ch/blocklist.php?download=domainblocklist'
option adb_src_rset '/^([[:alnum:]_-]+\.)+[[:alpha:]]+([[:space:]]|$)/{print tolower(\$1)}'
option adb_src_desc 'focus on zeus botnet by abuse.ch, daily updates, approx. 400 entries'
option enabled '1'
I wouldn't use combined lists, as you can achieve the same result by using the individual sources.
Lists that update once a day or more often a day, are also suboptimal, i think.
Yep, that's one purpose of the backup mode ... the other is e.g. for (travel) router which are often temporary "offline" / without working uplink connection - to make sure that always "my" lists gets loaded even without repeated list downloads.
Please doublecheck, that at least these options are set in your extra config section of adblock:
config adblock 'extra'
option adb_backup '1'
option adb_backupdir '/mnt/data/adblock'
option adb_backup_mode '1'
If so, maybe your 'adb_backupdir' is not accessible during boot!?
1 Like
How did you determine the OOM condition in dnsmasq? If you are quite confident, that this comes from dnsmasq, then open a ticket at bugs.openwrt.org, since dnsmasq is an OpenWrt core component.
BTW, I don't see any problems with my "dnsmasq" routers, even after days/weeks. Maybe you use advanced stuff like stubby, dnscrypt-proxy or something like that - just a wild guess!?
That is not a list (from my perspective) - it's just a huge collection of other list sources ... you have to define every single list source.
It's not planned to support such recursive stuff - it's simply much to much for small devices like routers ... and IMHO not needed: Since years I use 8 blocklists plus personal blacklist with in sum round 30k domains - there is no need to block zillions of domains. 
Thank you for your quick reply!
Perhaps something like that. It is an USB stick with f2fs...with a samba and dlna share. At least the newly downloaded lists are available everytime ;- )
What do you think about that?
Sounds interesting - but not before next stable OpenWrt release 19.x ...
1 Like
What about update schedule setting per source list?
On some source list sites there is a notice to not waste traffic. (maybe your ip gets bannend?)
I have set my adblock to update once a week and try to use source lists that don't update that often.
However, if it would be possible to set a update schedule per source list, it would be possible to have some list update on daily basis and some on weekly basis?
1 Like
I don't know - yet ... what happens if first list is scheduled weekly and the second list daily, both list needs to be merged in adb_list.overall on a daily basis. Often the backup functionality is not used ... etc.
Probably I'll come up with a proposal in 3.7.x - we will see!
[239472.954813] Out of memory: Kill process 13197 (dnsmasq) score 258 or sacrifice child
[239472.963662] Killed process 13197 (dnsmasq) total-vm:124220kB, anon-rss:123144kB, file-rss:0kB, shmem-rss:0kB
[239473.050121] oom_reaper: reaped process 13197 (dnsmasq), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[240925.711809] kthreadd invoked oom-killer: gfp_mask=0x15080c0(GFP_KERNEL_ACCOUNT|__GFP_ZERO), nodemask=(null), order=1, oom_score_adj=0
I've disabled adblock, no OOM since then.
Then reduce the number of processed blocklist and go from there ...
Yes, I will, but the number of entries is the same as always, only thing i've changed is updating the openwrt master build. I have no fancy settings like dnscrypt etc.
EDIT: Seems stable with less blocklists, so perhaps something changed in dnsmasq's internals that causes it to handle things differently or it could be a memory related bug. I don't think it's a memory leak because overall system memory seems to be stable.
I have 764K entries in adblock, never had a OOM, never had a reboot. Internet is slow, though. Edit: I have a Netgear R7800.
Which version of OpenWRT? I have the R7800 too.
Currently SNAPSHOT r9419.
Interesting... I'm about on the same one. I built a new one today tho', I have to try and see if it still happens.
EDIT: I have more entries tho, but like I said. no issues in snapshot r9213 it's after that one somewhere the problems occured.
Overall Domains 773509 (normal mode)
180Mb memory free, seems a tad less than before as well.
EDIT2: Still dies because of OOM.
Hi,
Adblock working great!!!
Is there any way to "whitelist" one o more local IP's (local devices) to bypass Adblock?
Maybe could be done using VLAN's/changing DNS devices, but I prefer not to do it if Adblock gives me a "solution".
Many Thaks for your hard work and help!!!