@dmei - I have unblocked your IP for now. Don't set off any alarms again please.
Just to add to this, the webservers are under constant attack from various bots (search engines (plural, including ones no one ever heard of), 'ai' companies, vulnerability scanners, ...) who do essentially exactly what you've been doing. It's happening around the clock and in a massively parallel fashion and pushes the server load a lot, mostly without actually helping a single human being. Especially wiki, ToH and gitweb resources are particularly costly in this regard, as their content is generated dynamically and present essentially unlimited content (past revisions and dynamic URLs) to stupid crawling scripts. The only way to fight this, is rate-limiting and blocking offenders, so if you set off the alarms, that's what needs to happen, because it really is costly and hurts everyone. Realistically, you aren't even going to read a single percent of the content you downloaded either.
Rate-limiting and blocking has been in effect for a couple of months now. I try to keep the forum and wiki bot lists in sync occasionally. There are different types of offenders on each site.
Blocking by IP only gets a small number of offenders. Using Agent ID strings is most effective but it becomes a game of whack-a-mole. Agent ID + IP is reserved for hard offenders who are suspected of scraping and get warnings for a reply.
@dmei I confirm your IP address got blocked because it was heavily abusing resources. It is simply not acceptable to scrape or mirror the wiki this way.
2 Likes
I don't want to belabor the point, and I'll assume positive intent that you really just wanted to have documentation available in case you are unable to get back online... but...
Out of the box, OpenWrt will often 'just work' for most users and situations with minimal additional configuration required (most of the time this is very easy and intuitive).
On x86, you may need to install driver packages for your ethernet and/or wifi cards, but aside from that it is generally fairly easy to get up and running. A good amount of the prep work can be done with you existing network still running normally, and then when things are configured appropriately (or nearly-so), you can swap your existing router for your new x86 device.
With that in mind, you absolutely don't want or need to have an entire offline copy of the entire OpenWrt site... Instead, ask us questions to get the system generally up and running and ready for the transition. You can always swap your old system back into place, and if you've got a cellular data connection, you can still read the main Openwrt.org site as well as the forums for help, even if your regular network is offline.
So... instead of the approach you have taken (which is considered abusive, even though I do believe that you didn't intend it that way), please setup a forum thread to help you get going and we'll help you along the way.
@ thess- thanks for unblocking
1 Like