We have a preliminary configuration of Anubis staged as a front-end to the Wiki to try to alleviate some of the continuous poundings on the site. There are still some tuning and exceptions necessary to be made over the next week or so - please bear with us. If this configuration does not work out, we will drop back to our previous filtering methods and look elsewhere for a solution.
When accessing the Wiki initially you will see a Proof-of-Work (PoW) challenge which hopefully you browser can pass. Otherwise you may be blocked from accessing the site.
I think Anubis is generally pretty bad for a number of reasons. The biggest problem I see is that you are breaking search indexing which means that less people will see OpenWRT resources. I also think AI scrapping can be good since it means that the AI will have some knowledge of OpenWRT. I have also found that it is very slow on mobile which a bit of a inconvenience.
With all of that being said, I'm not sure I see a better option. Arubis is pretty rudimentary and likely exploitable but the ease of setup is hard to beat. I would rather have a working wiki instead of a site that doesn't load. Maybe in the future there will be a better option.
@Darin755 - I have similar feelings with respect to Anubis client load. However, it is intentional and you should only have to pay the price occasionally (it's configurable). The current PoW settings are what I think a reasonable compromise.
Before Anubis we tried our best to block scrapers and LLM training bots. I think we will continue with that approach. Well behaved indexers were and will be let in with a degree of rate-limiting (Bingbots do not behave). My feeling towards LLM is that while useful for learning, there is just too much misinformation available that technically is hard to validate in any given field.
Thanks for the feedback - search indexing will not be broken.
The wiki IS our main website (besides this forum). Essentially, you will need to have a somewhat modern browser with Javascript and Cookies enabled to access the site. Could you visit: The Anubis User-Agent checker and post the output here with any location and/or IP info removed.
Curiously, the page clears noticeably faster on some browsers than others even when running on the same system.
Crawlers that we let in are either well-behaved or forcibly rate-limited. There are still rules in-play here for who gets "challenged", "allowed" or "denied" in addition to what filtering we do with Nginx.
As far as I know, they are working on a version with that option. Apparently there is a desire by the dev to commercialize the product and WASM may be a part of that effort. At the moment it is FOSS, a bit unpolished and perhaps moving towards services such as branding and special features for $$.
I am surprised it took this long for OpenWRT to follow the rest of the FOSS world and hide behind Anubis. I know it is not ideal, but even I had to add it to my website to protect against the AI scrapers. Currently there is not much else anyone can do unless a new solution is developed to this problem.
This thing consistently blocks me (using unadultered Google Chrome btw). I went into the Anubis repo and tried the links of others and was able to pass theirs. Something is wrong with your configuration or this is not the right tool for the OpenWRT website
Update:
I clicked on "Go back home" or something and passed that test, but I wasn't able to pass when I came from Google Search (i.e. by clicking an OpenWRT link result in Google Search)
I have seen this behavior while I was setting it up, experimenting with web (re-)configuration and tweaking rules. Many service restarts which can leave stale data behind on your browser were performed. When this state happens to me, I clear openwrt.org cookies and browser cache.
The server log usually shows token mismatches of stale data. Other than that, we are running Anubis out-of-the-box with some custom rules. There may be an IPv6 vs v4 issue lurking but I haven't seen any concrete evidence.
Edit: by manually navigating back to openwrt.org I was able to trigger the challenge again, and this time it passed. Though it is much slower than on other sites, taking maybe 10 seconds instead of the 2-3 seconds on other sites.
Sarcasm will not get you any attention - some details here would be more useful than 'me too' since a lot of the aforementioned issues have been addressed already.
Nice!