[Solved] Problems accessing the forum (Thunderbird request flood triggers bot filter at server)

I'm regularly having issues accessing the site and I cannot understand why.

I spend more time hitting reload to try and get ANY page to load on here than actually reading or posting. Today alone I’m getting “Sorry, we couldn’t load that topic, possibly due to a connection problem. Please try again. If the problem persists, let us know.”, 404 and “429 Too Many Requests” (the latter is what most often happens).

I think it may very based on time of day as in the early hours it seems more likely to work without issues.

I'm using Firefox 57.0.4 on Fedora 27. I'm going to try using Chrome in the future to see if the problem still occurs but it seems odd if its a Firefox issue as it doesn't happen on any other site.

A 429 is the client (you) sending too many requests to the server (LEDE) within a given amount of time.

The server thinks you are spamming it with all the reload requests, so it cuts you off.

It's called "rate limiting".

The slow loading/no loading could be a DNS problem...

Try flushing the DNS cache.

Change DNS resolvers to any of the public servers from this list...

These issues were all happening one after the other, clearly not DNS as it would already have been cached by my router (which does full DNS lookups and caching) which I might add runs pfSense so arguably even more robust than LEDE.

If the site is getting too many requests then surely that is a coding glitch in the forum itself as Javascript can't do that of its own accord. Then again, as the site sometimes works fine this makes no sense, you'd think an actual bug would trigger ALL the time. The only thing I can think there is if its designed so if the AJAX request takes too long it tries again and is refused by the server?

Like I said though, if it happens again I will try switching to Chrome and doing mtr to see if there is anything obvious going on.

Incidentally I just noticed another issue in Firefox, the bottom of the edit/reply box is cut-off the bottom of the page.

It's not a coding glitch.

Look up the definition for an HTTP 429 error.

All of your refreshes and clicks are putting you over the rate limit.

If your DNS cache is stale, it could very well cause issues with slow loading/no loading.

I use Firefox 57.0.4 with no issues, so you might try disabling add-ons to see if it corrects the problem. If it does, re-enable add-ons one at a time until you find the one that is causing it.

Okay that makes sense, but I have gotten a "too many connections" on my very first load, no refreshes. The very first time I tried to visit the forum it did this.

Obviously I hoped this wouldn't happen again but my experience so far has not been great.

I have very few addons due to problems in the past, just greasemonkey, Twitter Media Downloader and Enable Right Click and Copy. Seems unlikely any of those would be an issue or the site would still be broken now, plus the last two I didn't even have installed the first time I had this issue. :wink:

So I would run Wireshark while accessing the site and see if anything else is sending requests to the forum.

It's possible that because you have had 429's in the past, the server now has your login flagged.

You'd have to ask @tmomas about that.

Fingers crossed it just sorts itself out. I haven't been using the forum enough to figure out any particular pattern, other than a guess its time related.

Your trust level is basic user at this time, so that, plus the 429's, might have something to do with it.

Doesn't hurt to ask @tmomas or @richb-hanover.

Both are site leaders.

Guess I need to make myself helpful then and raise that level. :stuck_out_tongue:

I'm sure you will.

Sorry for the inconvenience, I am looking into it as we speek. The 429 you're seeing are not related to server load issues or maintenance windows, you just appear to trigger nginx' rate limiting rules.

Update:
I can confirm that your IP frequently triggers 429 errors due to too many requests but not yet sure why. You seem to have uncommon browsing habits :slight_smile:

Update 2:
So on Jan 11th, at 20:14:39 UTC a useragent identifying itself as "Thunderbird/52.5.2" began issuing GET requests to /t/security-meltdown-and-spectre-vulnerabilities-in-arm/10283, making between 4 to 6 requests per second to the very same URL over and over again. This request flood stopped almost exactly one hour later at 21:14:31 UTC.

Excerpt from the log (IP redacted):

[11/Jan/2018:21:14:27 +0000] "forum.openwrt.org" 0.0.0.0 "GET /t/security-meltdown-and-spectre-vulnerabilities-in-arm/10283 HTTP/1.1" "Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.5.2" "-" 429 329 "-" - 0.000 "-"
[11/Jan/2018:21:14:27 +0000] "forum.openwrt.org" 0.0.0.0 "GET /t/security-meltdown-and-spectre-vulnerabilities-in-arm/10283 HTTP/1.1" "Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.5.2" "topics/show" 200 23893 "-" 0.008 0.008 "-"
[11/Jan/2018:21:14:28 +0000] "forum.openwrt.org" 0.0.0.0 "GET /t/security-meltdown-and-spectre-vulnerabilities-in-arm/10283 HTTP/1.1" "Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.5.2" "topics/show" 200 23893 "-" 0.013 0.013 "-"
[11/Jan/2018:21:14:28 +0000] "forum.openwrt.org" 0.0.0.0 "GET /t/security-meltdown-and-spectre-vulnerabilities-in-arm/10283 HTTP/1.1" "Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.5.2" "-" 429 329 "-" - 0.000 "-"
[11/Jan/2018:21:14:28 +0000] "forum.openwrt.org" 0.0.0.0 "GET /t/security-meltdown-and-spectre-vulnerabilities-in-arm/10283 HTTP/1.1" "Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.5.2" "topics/show" 200 23893 "-" 0.011 0.011 "-"
[11/Jan/2018:21:14:28 +0000] "forum.openwrt.org" 0.0.0.0 "GET /t/security-meltdown-and-spectre-vulnerabilities-in-arm/10283 HTTP/1.1" "Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.5.2" "-" 429 329 "-" - 0.000 "-"
[11/Jan/2018:21:14:28 +0000] "forum.openwrt.org" 0.0.0.0 "GET /t/security-meltdown-and-spectre-vulnerabilities-in-arm/10283 HTTP/1.1" "Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.5.2" "topics/show" 200 23893 "-" 0.007 0.007 "-"
[11/Jan/2018:21:14:28 +0000] "forum.openwrt.org" 0.0.0.0 "GET /t/security-meltdown-and-spectre-vulnerabilities-in-arm/10283 HTTP/1.1" "Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.5.2" "-" 429 329 "-" - 0.000 "-"

Overall, the Thunderbird client issued about 27000 requests over the cours of one hour, this caused the nginx reverse proxy to classify your IP as bot and return with HTTP 429 errors. The bot classification rule currently kicks in when exceeding 200 requests per minute.

Ah that makes sense, I knew something was screwy in Thunderbird but I had no idea it was actually doing something like this.

Apparently this is down to a problem I have been having, if I click a link in Thunderbird, it doesn't open, but the habit is to deeply ingrained I still keep doing it. What you have discovered is that it actually IS doing something, something very bad.

I don't understand why Thunderbird would even open links internally in the first place, surely it should just spawn the web browser specified and let THAT handle it?

I guess the good news is this forum is absolutely fine and if I actually remember NOT to click links, it shouldn't happen again.

Looks like you emailed a link to the topic [Security] Meltdown and Spectre vulnerabilities in ARM to yourself and then tried to open it in Thunderbird.

I use Thunderbird (as well as Firefox) and just did a test doing the same thing with Wireshark running, and did not see any request floods.

Firefox opened correctly and it took seconds to get to the site and display the topic.

This is on a 6 down/1 up DSL connection in the U.S.

So it looks like you created a click storm when the link wouldn't open right away.

As suggested before, try running MTR to see what the route to the LEDE forum and back looks like when you are having issues...on the first delayed click.

https://bitwizard.nl/mtr/

1 Like

My Thunderbird (58.0b2) also opens forum links nicely in Firefox (58.0b15).

The reason for your failure might be some failing add-on, proxy or something such in your setup. (failing https handshake at some stage?)

Running Wireshark would show that, and has been suggested to the OP.

A small thing in Firefox that can cause slow loading is Hardware Acceleration.

The OP might try unchecking that and see if it makes a difference.

Something is broken in Thunderbird, no clue what, it just suddenly stopped opening links and it has zero addons. But at least it means I know why certain sites have been hating me lately.

Time to save your Thunderbird profile and clean install Thunderbird.

Apparently I didn't need to, just had to switch on the option to request how to open links rather than let Thunderbird decide on its own.

Doesn't explain why Thunderbirds default method for opening links stopped working, but not really important so long as it works.