Mysterious reboots - could UPS battery backup solve it?

I'm really sorry if I'm posting this in the wrong place. I considered the hardware forum, but it seemed like this question might be seen by more people here.

I'm currently running OpenWrt 21.02.2 r16495-bf0c965af0 x86_64 on a PC-Engines APU2.

Regardless of version my router has rebooted itself every day or two since I built it, and I have no software explanation for it. I'm not doing anything remotely odd as far as I can tell with the configuration. Adding and removing features seems to have no effect.

However, I have noticed sometimes all the lights dim a bit at the same time in our flat. It makes me wonder if maybe the source of my problem is an instability in the power supply. This makes me want to look at investing in battery backup and regulation.

So I'd like to get an idea of the consensus here. Is it worth it for a small home network with a media server and a few clients?

I would set up some sort of persistent logs to see what is happening. How you do that is up to you, since there are plenty of ways, like e.g. just simply piping the output of logread -f into a file on a USB-stick or using rsyslog to send the logs in realtime to another device, or whatever you like best.

1 Like

I the light actually dim then I would start by checking the electrical system before anything worse happens.

4 Likes

There's a serial port, you could log the console, if you'd like.

Check if you have a replacement power adapter you could use, to rule out the one you got with the unit.

1 Like

'logread -f' to run on a permanently active terminal would be the simplest method to verify, whether some ununsual system activity might cause the reboot.
If thats not the case, considering a very small UPS might be a good idea. I also have an APU2 and an APU3, not seeing such a problem.

1 Like

I can’t tell you if that may be your issue, but your comments struck me as somewhat familiar.

I live in the middle of a 10+ hectare maple bush and power anomalies are frequent with low density and old infrastructure, it’s essentially if it breaks we’ll fix it mentality by the provider.

In any event, I run a small UPS just to give me bit of time to shutdown the network/devices gracefully. What I found over time with help of the apcupsd package was it’s ability alert me to Brownouts.

When I read your post I did a logread -e apc and *Voila! A transient Brownout or voltage below threshold alert yesterday.

Fri May 27 15:46:20 2022 daemon.alert apcupsd[2583]: Power failure on UPS. Running on batteries. 
Fri May 27 15:46:20 2022 daemon.alert apcupsd[2583]: Initiating system shutdown!
Fri May 27 15:46:20 2022 daemon.alert apcupsd[2583]: User logins prohibited
Fri May 27 15:46:25 2022 daemon.crit apcupsd[2583]: Running on UPS batteries.
Fri May 27 15:46:28 2022 daemon.crit apcupsd[2583]: Mains returned. No longer on UPS batteries.
Fri May 27 15:46:28 2022 daemon.alert apcupsd[2583]: Power is back. UPS running on mains.
Fri May 27 15:46:28 2022 daemon.alert apcupsd[2583]: Allowing logins

FWIW, I can’t recall a spontaneous reboot unless I caused it my self.

2 Likes

Chiming in here: In the 30 some odd years I've been with an internet connection; I've got working equipment older than any of the UPS's I've owned. They do their job and protected my equipment at the expense of multiple battery swaps and eventual UPS swaps.

The regard is what to keep up, ie what is so important that a utility interrupt has to be mitigated, and can I manage the audible alarm when AC power is down and DC power is established.

Over the years, as tech has improved, the items of absolute importance have decreased, yet the backbone of telecom backup is still absolute.

1 Like

It's a building with about 15-20 flats, dating to the 1950s. I presume that sometimes when somebody starts two laundry dryers at the same time, or whatever we get a lot of extra load.

This sounds like us.

We're in a well-populated part of a country with amazing infrastructure, but the building itself is from the 1950s and the property manager is new (took over from the owner on his retirement) and only starting to tackle technological debt issues.

I know they'll get to it, but I reckon your advice might get me over the hump. I'm going to check tonight.

I'm already shipping my logs to a syslog server elsewhere on the network, and I haven't found any kind of pattern, or rhyme or reason.

The logs just suddenly cut off without warning? Yeah, then it sounds like either a proper hard crash or brown-out. A basic UPS doesn't cost much and used ones can often be gotten really cheap (also, if you get a used one, don't believe when the manufacturer claims that only their first-party batteries are okay to use. 3rd-party batteries are perfectly fine as well)

1 Like

Especially when the panemic stopped the manufacturing of the battery you where supposed to use.

1 Like

Thanks!

I'm going forward. The only thing that's weird is that the media server doesn't have these issues, but it has a laptop-style power adaptor, which probably weighs more than the box itself (Intel NUC ~2014).

I'm in agreement with many of the other comments here that a UPS is a good idea in general. These devices protect your equipment from small surges, brown-outs, and also full blown blackouts, giving you time to ride it out or gracefully shut down the equipment. And since essentially all UPS units have some means of monitoring, you can typically get the status in your logs so you'll have some more data about your power conditions.

If it turns out that it doesn't help, I'd then look into the power supply for your actual device -- if it is always possible that the supply is inadequate (underspec or failing), and that could cause brown-out conditions on your device itself. Or it could be a thermal issue. You can always force this situation by stress-testing it (for example, running multiple concurrent speedtests and other high demand operations will increase the power consumption and thermal loading.

And failing those, maybe there is some random bug or conflict... hopefully your logs would reveal that, in time.

1 Like

The power-draw of the device itself and the capacity of and in how good condition any capacitors are in the power-supplies all matter. It's a combination of multiple things and your server probably just simply teeters close enough to the brink that it can't cope with transient voltage-drops on the line.

1 Like

Thanks again. This makes great sense.

I just sent a note to Pascal at PC-Engines to order another adaptor in case the original was messed up, and I'll order a UPS unit tomorrow morning.

Thanks again. I'll order the UPS tomorrow morning. I'm fully convinced.

Aircon starting up can dim the lights

I had an odd situation that plagued me for years...

tl;dr - I had to replace all surge protectors for all electronics (some protectors which were 30+ years old).

Over 7 years ago, I realized where I lived did not have "clean power"...mainly because the power flashed all the time (quick brown or even black outs). At one point in time, I replaced a motherboard twice in an x86_64 PC. With this machine on it's third motherboard, I bought a new surge protector (about $20 USD with a $15-20k warranty on equipment connected). I never lost another motherboard (I also bought a motherboard model with built-in surge protection)

In other scenarios, I had an issue where a Ethernet switch would randomly loose power...playing with the wall wart would get it running at times...another device...whenever I touched the power supply, power flash, etc., a Meraki MX60 would reboot...loosing all it's settings and sometimes, wiping its flash and booting into the recovery image. In the Meraki's case, the power supply was bad, and running about 4-5 Volts too "hot".

Both were resolved by replacing the power supplies (wall warts). So I would definitely start by checking checking surge protectors and power supplies.

See:

I now have a combination of UPS and solar/battery backups for networking equipment and sensitive electronics, etc. that I want running 24/7/365, for others, I use good surge protectors if I can function with those devices down during a power outage.

:bulb: You could also make an UPS yourself with: a powerful enough wall wart (13-14 V), solar charge controller, and a 12 V battery - given your device it 12 V, that is.

4 Likes

Oh, we're telling stories here now? Well, I may as well chime in, then!

My husband and I bought our house a couple of years ago and we noticed soon after that every now and then, randomly, the circuit breaker supplying power upstairs would trip. We also get brownouts here and there. Turns out that the previous owners have absolutely fscked up the wiring here and we'd basically have to redo the entire building to fix it all up.

Can't afford such a big task for years to come, so for now we've got all of our important stuff behind line-interactive UPSes. I've slowly been accumulating parts and equipment for a small solar-setup, but it's taking forever due to me having basically zero budget.

3 Likes