Hey everyone happy new year! Honestly either way with this one. It will pick up an update the following day as you mentioned. But where it could be useful is on eg reboot where it can't fall back, or if worst case scenario oisd site was down for days/weeks (probably unlikely though).
Happy New Year! Hmm so seems like there's a case for leaving as is. I keep forgetting about it even though it seems to update every 5am just fine. We should all change the 5am update to some random minute to avoid simultaneously hammering the oisd server.
Same here I don't have to worry about it really now, check the logs every now and then to make sure it's all running fine.
There's an idea, build a random hour/minute into the script eg between 1am to 5am? Not sure if even possible with cron?
Great minds think alike since I was wondering about that too. We could have script take parameter from cron or otherwise so that it knows it was started from cron then delay by random X out of 30 mins?
Ah yep well that sounds good. Might as well go random 59 minutes tho, and spread across the whole hour?
Edit: And just a thought, people in different timezones will take care of spreading this across different hours of the day, so yeah, really only a randomiser for minutes is needed....
Sure. Seems environment variable option may be useful:
1 Like
Hmm challenge seems to be how to get random integer in POSIX shell (not bash where we can just use $RANDOM).
Oh yeah you are right no random in POSIX.
This seems to work ok. But it's actually not so random. Awk uses the time of day apparently, so maybe not a good option.
awk 'BEGIN{srand();print int(rand()*59)}'
This output could be manipulated somehow:
echo $(hexdump -n 1 -e '"%u"' </dev/urandom)
1 Like
Wouldn't a simple random sleep work here?
Also OISD is still down a day later. I did receive a response from the maintainer and he is aware of the issue.
1 Like
Code? I propose this:
start()
{
logger -t adblock-oisd "Started adblock-oisd."
if [[ "$RANDOM_DELAY" == "1" ]]; then
random_delay_mins=$(($(hexdump -n 1 -e '"%u"' </dev/urandom)%60))
logger -t adblock-oisd "Delaying download by $random_delay_mins minutes (thundering herd prevention)."
sleep "${random_delay_mins}m"
fi
@Wizballs see any issue with the above? I tested and seems to work fine - example output:
Mon Jan 2 13:52:20 2023 user.notice adblock-oisd: Started adblock-oisd.
Mon Jan 2 13:52:20 2023 user.notice adblock-oisd: Delaying download by 15 minutes (thundering herd prevention).
Mon Jan 2 14:07:30 2023 user.notice adblock-oisd: Download of new oisd.txt file suceeded.
Mon Jan 2 14:07:43 2023 user.notice adblock-oisd: oisd file check passed with new oisd.txt.
Mon Jan 2 14:08:02 2023 user.notice adblock-oisd: New oisd.txt installed with line count: 333389 and dnsmasq check passed.
1 Like
@Lynx I'm still learning nix shell scripting as I primarily write scripts in PowerShell
@Wizballs and @account4538, please see these commits:
In summary:
- adblock-oisd now checks for environment variable RANDOM_DELAY="1";
- if RANDOM_DELAY="1", adblock-oisd sleeps for a random number of minutes between 0 and 60 prior to downloading the new oisd.txt file; and
- the suggested Scheduled Task for adblock-oisd has been updated to set RANDOM_DELAY="1" prior to running adblock-oisd start().
Let me know if you see any issues or possible improvements.
@Wizballs would it be an idea to rename this thread to adblock-oisd, give basic description and have thread moved to community projects? That's kina how the original cake-autorate thread came about - original thread led to development of project,
2 Likes
No worries, I renamed to start with Adblock-oisd . Kept the rest of the description just so it's recognisable for everyone
- No need for double brackets.
- You may want to move the downloaded file outside of the dnsmasq run directory on stop and check for its existence/move it back on start to allow quick stop, use blocked domain, start scenarios.
Hmm I could add pause and resume functions to the service file with their own helper descriptions to provide that functionality.
OK I have added this functionality:
root@OpenWrt:/tmp# service adblock-oisd
Syntax: /etc/init.d/adblock-oisd [command]
Available commands:
start Start the service
stop Stop the service
restart Restart the service
reload Reload configuration files (or restart if service does not implement reload)
enable Enable service autostart
disable Disable service autostart
enabled Check if service is started on boot
adblock-oisd custom commands:
pause pause adblock-oisd
resume resume adblock-oisd
Tue Jan 3 12:47:06 2023 user.notice adblock-oisd: Received pause request.
Tue Jan 3 12:47:15 2023 user.notice adblock-oisd: /tmp/dnsmasq.d/oisd.txt temporarily saved as /tmp/oisd.txt.gz and dnsmasq restarted.
Tue Jan 3 12:47:23 2023 user.notice adblock-oisd: Received resume request.
Tue Jan 3 12:47:30 2023 user.notice adblock-oisd: /tmp/oisd.txt.gz uncompressed and written back to /tmp/dnsmasq.d/oisd.txt and dnsmasq restarted.
1 Like
I have also just added a status check with this commit:
root@OpenWrt:/etc/init.d# service adblock-oisd
Syntax: /etc/init.d/adblock-oisd [command]
Available commands:
start Start the service
stop Stop the service
restart Restart the service
reload Reload configuration files (or restart if service does not implement reload)
enable Enable service autostart
disable Disable service autostart
enabled Check if service is started on boot
adblock-oisd custom commands:
status check dnsmasq and good line count of existing oisd.txt
pause pause adblock-oisd
resume resume adblock-oisd
Wed Jan 4 12:17:49 2023 user.notice adblock-oisd: Presently installed oisd.txt has line count: 333316 and dnsmasq check passed.
You could get the current public IP address and simply take the last byte an treat that as the offset (round(60/${last_byte}) into the hour. Assuming that ISPs basically assign IPv4 addresses randomly that should spread out things sufficiently, and same for the last byte in IPv6 SLAAC addresses. IIUC you really just want to avoid a thundering herd coming in at exactly the same moment, so having each user using a deterministic offset would be okay?
1 Like
Still going strong here. Updates every 5am plus the random minute for thundering herd prevention. And just keeps doing its thing in the background. Happy with this little gadget!
@Wizballs should you perhaps add an intro in leading post linking to adblock-oisd on GitHub like:
UPDATE: an outcome of this thread has been the development of adblock-oisd - a super lean script written as an OpenWrt service file that implements adblocking via the oisd blocklist and incorporates several safety checks and useful features. adblock-oisd will download the latest oisd list and perform various safety checks associated with the file before and after restarting dnsmasq and act accordingly. It is designed to be run as a daily scheduled task.
adblock-oisd is available here:
2 Likes
You could cheat the system by just adding your own summary on post 212. That is the last post for me, and it happens to be your post.