Hi,
I am a Nextcloud news RSS client user that I tried to subscribe to the OpenWRT Release and security announcements feed. Unfortunately this has not been possible because it looks like the Nextcloud news user-agent is blacklisted.
curl -A "NextCloud-News/1.0" https://forum.openwrt.org/c/announcements/14.rss
Too many crawling requests. Error code: nextcloud-news_crawler_rate_limit.%
After asking in Nextcloud news, they pointed out that maybe one reason of being blacklisted by OpenWRT could be that they for a time were a too aggressive crawler, a bug that has been solved:
master ← wofferl:feat_limit_scraper
opened 06:12PM - 18 Mar 26 UTC
* Resolves: #3319
## Summary
This is the second of a series of three pull … requests to improve the full-text download feature.
The current implementation is unfair to content providers and gives the app a bad reputation as aggressive crawler.
Although the feed is only downloaded when necessary, depending on the settings, all referenced web pages are then downloaded again and again.
To prevent this, I have implemented a mechanism that ensures only new or updated content is downloaded, which also saves memory during the fetch.
This involves creating a list of known GUID hashes and their corresponding publication dates, which is used for comparison during the update job.
As some feeds do not have publication dates for the items and the current date is used in this case, these feed items are not updated and only downloaded once.
Here some example debug output showing one hourly fetch during my tests, the heise feed would currently fetch 160 websites per hour
> heise online News added: 3, skipped: 157, error: 0
> Items: 3 Memory used: 4 MB
> tagesschau.de - Die Nachrichten der ARD added: 3, skipped: 37, error: 0
> Items: 3 Memory used: 1 MB
> taz.de - Artikel aus der Onlineausgabe added: 10, skipped: 10, error: 0
> Items: 10 Memory used: 1 MB
Another way to avoid downloading unnecessary articles is coming with the third PR, which I am currently still testing. It allows individual articles to be updated on demand from the frontend via the backend controller and it advantages (readability, sanitize, etc)
## Checklist
- Code is [properly formatted](https://nextcloud.github.io/news/developer/#coding-style-guidelines)
- [Sign-off message](https://github.com/src-d/guide/blob/master/developer-community/fix-DCO.md) is added to all commits
- Changelog entry added for all important changes.
Then I would like to ask if it will be possible to whitelist Nextcloud news at least for news 28.2.0 and above.
Thanks
thess
April 24, 2026, 2:13am
2
They are not on the block list. However, they are rate-limited which is rather strict
I can remove them temporarily and if they (re-)appear on my radar, they will be put back on the naughty list.
Thanks, I've just check, for the time being I can be subscribed.
Tried the mailing list instead (or as an alternative)?
