Apcupsd doesn't timely updates the measurements (for APC UPSs) since 19.07

As someone else mentioned, maybe you just have very stable power? In our location, the power is fairly dirty, plus I have a solar array on the roof to confound things even further. You can clearly see when the sun came out today at about 1100 and then clouds rolled back in at about 1300.

Along those lines, but a caveat from the v3.14.14-3 apcupsd.conf also says a UPSTYPE must be defined. Did not know if that exists in the 3.14.1 .conf.

FWIW, I’ve had my meter probes plugged into a spare receptacle on the UPS the last 9 hrs. and it has mimicked the reported line voltage from both apcaccess, and https://10.10.1.1/cgi-bin/apcupsd/upsstats.cgi.

My apcaccess - identical unit to your’s:

APC      : 001,034,0849
DATE     : 2023-03-30 13:53:02 -0400  
HOSTNAME : RuralRoots
VERSION  : 3.14.14 (31 May 2016) unknown
UPSNAME  : RuralRoots ES 550 UPS
CABLE    : USB Cable
DRIVER   : USB UPS Driver
UPSMODE  : Stand Alone
STARTTIME: 2023-03-30 13:53:01 -0400  
MODEL    : Back-UPS ES 550 
STATUS   : ONLINE 
LINEV    : 125.0 Volts
LOADPCT  : 23.0 Percent
BCHARGE  : 100.0 Percent
TIMELEFT : 7.4 Minutes
MBATTCHG : 5 Percent
MINTIMEL : 3 Minutes
MAXTIME  : 0 Seconds
SENSE    : Low
LOTRANS  : 88.0 Volts
HITRANS  : 142.0 Volts
ALARMDEL : 30 Seconds
BATTV    : 13.5 Volts
LASTXFER : Automatic or explicit self test
NUMXFERS : 0
TONBATT  : 0 Seconds
CUMONBATT: 0 Seconds
XOFFBATT : N/A
STATFLAG : 0x05000008
SERIALNO : 3B1106X22237  
BATTDATE : 2011-02-03
NOMINV   : 120 Volts
NOMBATTV : 12.0 Volts
FIRMWARE : 843.K2 .D USB FW:K2
END APC  : 2023-03-30 16:42:41 -0400 

How I would wish that. On the contrary, it's unstable and eventful, specially during the tropic's dry season (thankfully this year's "la niña" has easen the things a little bit for us so far).

Here I was measuring 121V back and forth to 124V, when the UPS was reporting 127V yesterday.


Me neither. But I got courious why the included apcupsd.conf refers to 3.14.1.


Anyways, the three of us are running the same config, I think.

Thanks for that. So if yours follows the line variations but mine doesn't, and we are running the same configuration, and there's no software errors, and time ago it worked fine when I had it connected to a linux PC with apcupsd/gapcmon. I don't know where else to look.

Maybe I will have to recreate that environment to discard a hardware failure.

For the sake of time and pinpointing the issue, I replicated the configuration on one vendor's OpenWrt implementation (over 18.06+propietary drivers). Now the things make more sense:


Therefore I pointed the official OpenWrt box to pull information from the other one:

Hence the issue is only when the UPS is connected to the original 22.03.3 router. But I haven't seen any USB errors or "UPS communication lost" errors, so it doesn't seems to be USB or hardware related. It has to be some library or installation/configuration issue.

I guess I'll begin by comparing libraries installed or not, and versions, but I'm not an OpenWrt/apcupsd/collectd expert.

Nice! I am still on 21.02.5 (mvebu broke on 23.03). I’ll build a new Snapshot tomorrow and let you know how it compares.

Thanks for taking the time for that. I'll continue investigating myself after some other stuff to take care of.

In the meantime, it's evident the difference when collectd of the original 22.03.3 router was gathering info from the local USB port (left), the time there was no UPS connected (no communication/data, center), and when gathering samples from the unofficial 18.06 fork (right, and credits to gl.iNet for getting this working):

I think the issue is at apcupsd level or lower. To rule-out collectd and statistics, I wrote an apcaccess datalogger script to follow the line Voltage (and variations):
/19:59, new value: 124 ----------------------------------------------------------------------------/21:18, new value: 125 ----------------------------
where "-", "/", or "\" represent the continuity or change for the last period (minute). During this run, it only recorded 1 variation for the ongoing 100minutes or so.

Because the release image for my mt76x8 didn't include USB drivers, I installed it guided by wiki and tutorials. i.e.

root@SVdP2:~# opkg list-installed | grep usb
kmod-usb-core - 5.10.161-1
kmod-usb-hid - 5.10.161-1
kmod-usb-ohci - 5.10.161-1
libusb-1.0-0 - 1.0.24-5
libusb-compat4 - 0.1.7-2
usbutils - 014-1
root@SVdP2:~# opkg list-installed | grep ups
apcupsd - 3.14.14-3
apcupsd-cgi - 3.14.14-3
collectd-mod-apcups - 5.12.0-33

Which as we know stablish the connection:

root@SVdP2:~#  ls -l /dev/usb
crw-------    1 root     root      180,  96 Mar 31 11:47 hiddev0
root@SVdP2:~# dmesg |grep usb
[   17.664485] usbcore: registered new interface driver usbfs
[   17.670153] usbcore: registered new interface driver hub
[   17.675741] usbcore: registered new device driver usb
[   18.054068] phy phy-10120000.usbphy.0: remote usb device wakeup disabled
[   18.060877] phy phy-10120000.usbphy.0: UTMI 16bit 30MHz
[   18.611262] usb 1-1: new low-speed USB device number 2 using ohci-platform
[   18.807087] usbcore: registered new interface driver usbhid
[   18.812754] usbhid: USB HID core driver
[   19.356641] hid-generic 0003:051D:0002.0001: hiddev96,hidraw0: USB HID v1.10 Device [APC Back-UPS ES 550 FW:843.K1 .D USB FW:K1 ] on usb-101c1000.ohci-1/input0
root@SVdP2:~# lsusb
Bus 001 Device 002: ID 051d:0002 APC Back-UPS ES 550 FW:843.K1 .D USB FW:K1
Bus 001 Device 001: ID 1d6b:0001 Linux 5.10.161 ohci_hcd Generic Platform OHCI controller
root@SVdP2:~# lsusb -t
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ohci-platform/1p, 12M
    |__ Port 1: Dev 2, If 0, Class=, Driver=usbhid, 1.5M
root@SVdP2:/# cat /sys/kernel/debug/usb/devices

T:  Bus=01 Lev=00 Prnt=00 Port=00 Cnt=00 Dev#=  1 Spd=12   MxCh= 1
B:  Alloc= 13/900 us ( 1%), #Int=  1, #Iso=  0
D:  Ver= 1.10 Cls=09(hub  ) Sub=00 Prot=00 MxPS=64 #Cfgs=  1
P:  Vendor=1d6b ProdID=0001 Rev= 5.10
S:  Manufacturer=Linux 5.10.161 ohci_hcd
S:  Product=Generic Platform OHCI controller
S:  SerialNumber=101c1000.ohci
C:* #Ifs= 1 Cfg#= 1 Atr=e0 MxPwr=  0mA
I:* If#= 0 Alt= 0 #EPs= 1 Cls=09(hub  ) Sub=00 Prot=00 Driver=hub
E:  Ad=81(I) Atr=03(Int.) MxPS=   2 Ivl=255ms

T:  Bus=01 Lev=01 Prnt=01 Port=00 Cnt=01 Dev#=  2 Spd=1.5  MxCh= 0
D:  Ver= 1.10 Cls=00(>ifc ) Sub=00 Prot=00 MxPS= 8 #Cfgs=  1
P:  Vendor=051d ProdID=0002 Rev= 1.06
S:  Manufacturer=APC
S:  Product=Back-UPS ES 550 FW:843.K1 .D USB FW:K1
S:  SerialNumber=3B1805X62100
C:* #Ifs= 1 Cfg#= 1 Atr=e0 MxPwr=  2mA
I:* If#= 0 Alt= 0 #EPs= 1 Cls=03(HID  ) Sub=00 Prot=00 Driver=usbhid
E:  Ad=81(I) Atr=03(Int.) MxPS=   6 Ivl=10ms
root@SVdP2:~# /etc/init.d/apcupsd stop
root@SVdP2:~# apctest


2023-03-31 21:02:41 apctest 3.14.14 (31 May 2016) unknown
Checking configuration ...
sharenet.type = Network & ShareUPS Disabled
cable.type = USB Cable
mode.type = USB UPS Driver
Setting up the port ...
Doing prep_device() ...

You are using a USB cable type, so I'm entering USB test mode
Hello, this is the apcupsd Cable Test program.
This part of apctest is for testing USB UPSes.

Getting UPS capabilities...SUCCESS

Please select the function you want to perform.

1)  Test kill UPS power
2)  Perform self-test
3)  Read last self-test result
4)  View/Change battery date
5)  View manufacturing date
6)  View/Change alarm behavior
7)  View/Change sensitivity
8)  View/Change low transfer voltage
9)  View/Change high transfer voltage
10) Perform battery calibration
11) Test alarm
12) View/Change self-test interval
 Q) Quit

Select function number: q

2023-03-31 21:03:40 End apctest.
root@SVdP2:~# /etc/init.d/apcupsd start

Just in case, I forced the DEVICE to /dev/usb/hiddev0, but it doesn't make difference.

As expected, if I unplug and plug the USB cable, the values are updated, but then seldom changes, unless there is an event, for which the UPS push the update.

I took the same USB-modded router with 22.03.3 and UPS, and tried nut instead of apcupsd. After some improvements made on rrtdtool's nut.js, got it working.

Therefore it's not hardware related. It's something about recent apcupsd and/or this UPS set of features combination, and therefore I kindly ask for the developers to take a look at the previous discussion.

Recently i had to swap my APC UPS due to a sudden death of the UPS.
It was all fine when i went to work, and the router and switch were offline when i came back. No chance to bring the UPS back to live. Well, it was approx. 18 years old...
On this old UPS i had the problem that USB-HID didn't work with my Fritz!Box 7520 and i had to use the serial connection (link).
My new UPS (4 years old) won't give me the same amount of data as the old one, using the smart-ups protocol with apcupsd. But i had a stable USB connection for several days.
To get more data i switched the UPS and apcupsd to the modbus protocol.
Now i got more data, but only for a few hours. After some time the data wasn't refreshed, i got the same old, invalid values for days until i restarted apcupsd.
But:
Just out of frustration i connected an old router (TP-Link TL-WDR4300, running OpenWrt 22.03.3) to my local LAN and ran the UPS and apcupsd on this old device.
Since a week or so, apcupsd is working fine and luci-statistics shows me some nice graphics.
Here's the big question: Why does it work on the old router, but not on the new one? The configuration is identical on both devices. Is the MIPS-build of apcupsd different to the ARM-V7 build or is it a hardware issue?
I don't like the idea to use an additional router, just so to run apcupsd...

This is basically all userspace communication, it 'should' be independent of the arch/ target/ device. Obviously there may be target (or even device-) specific bugs in the USB implementation, but that would 'usually' manifest itself on a more fundamental level.

1 Like

As far as apcupsd concerns, I haven't had log reports of commlost or disconnections in any case, what I've observed is the values are periodically updated for a 18.06-based SiFlower implementation, but not on the official 22.03.3 MediaTek target. Therefore I also think there's a bug somewhere, but not at MT USB level, because after I moved to nut on the same MT 22.03.3, the values of my reports started to be updated as it should. So, it's not the router/USB, it seems to be something that affects apcupsd or its dependencies, at least for some targets.

Does these new graphs show voltages variations or are flatliners? mine were moslty flatliners until I moved to nut.

i.e. ramips/mt76x8 22.03.3 apcupsd:

ramips/mt76x8 22.03.3 nut:

Note: I also replaced UPS this weekend. Had to reset the router to recognize it.

So hadn't i. That would've been an obvious hint :wink:

They show nice variations.


They were/are similar on my FB7520, but only for a few hours. After that, there's just a flat line.

Let's make a summary about the working and broken apcupsd implementations and usbhid units (please help me complete/fix it):

  • (SiFlower 18.06-fork: everything OK, at least for some hours)
  • ramips/mt76x8 22.03.3: communication seems OK, but mostly flat-line (no problems with nut)
  • ipq40xx/generic XX.XX.X?: communication seems OK, worked for a few hours, then flat-line
  • ath79/generic 22.03.3: everything OK

FWIW:

Dependencies for installing apcupsd 3.14.14-3 (mt76x8):
libpthread (already installed for nut, running fine)
libgcc1 (already installed for nut, running fine)
libusb-compat4 (already installed for nut, running fine)
libusb-1.0-0 (already installed for nut, running fine)
librt (already installed for nut, running fine)

An ath79/generic fork of 19.07.8 would install acpupsd 3.14.14-2, with the same dependencies.

  • lantiq/xrx200 (BT HH5A) 22.03.3: Both Modbus and Smart-UPS protocol: everything OK

The interesting thing is, that the old Smart-UPS protocol always worked fine on different routers. But the new Modbus communication protocol, which is (only?) available on newer Smart-UPS models does cause some issues. Whereas the 'newer' version of the Smart-UPS protocol works still fine, but features only a subset of data, compared to the 'older' version.

At my workplace, we're using different models of APC UPS. One of the newer models (which also has the new Modbus and the 'crippled' older Smart-UPS protocols) has got an AP9620 communication card installed, which delivers the 'complete' old Smart-UPS data. I got the permission to use this one at home for a few days to test with my 3 different router models. But i assume, that all will work fine with this card. I just swapped it with my UPS...when i'm back from work this evening, i'll see what my FB7520 has recorded...

My two Back-UPS ES are old (2008, with USB-RJ50 cable). The original and more limited one had this issues with apcupsd. I haven't tried the new one with it because: a)they seem to work fine with nut (except the events, I'm working on that), b)I could only downgrade to 22.03.0, and I suspect the issue was introduced with apcupsd 3.14.14-3. So if you or someone else could spare an USB router capable of running an older OpenWrt, we could try to pinpoint the change that affect us.

I found an old Netgear with USB2.0, so I connected my old Back-UPS, installed the minimum, i.e. kmod-usb-hid, apcupsd, collectd-mod-apcupsd, luci-app-statistics, and tested with 21.02.7 and 21.02.0 for ath79/generic, and 19.07.4 for ar71xx. The results were the same: flat lines under normal situation.


On the other hand, the other router with the other UPS showed more real measurements:

FWIW, the included in 19.07.4 is apcupsd - 3.14.14-2.

Finaly some progress. Went further back to 18.06.0 and the statistics started to make sense, so whatever collectd-mod-apcups does there (apcaccess I guess), it works:

Went a little forward, and 18.06.9 works the same, so whatever it went broke, it happened between 18.06.9 and 19.07.4. So can somebody please check what related change in that time period is affecting these measurements?

EDIT:
Tried to do the same with 19.07.2 and 19.07.0, but couldn't install luci-app-statistics because many "file is already provided" errors, so I collected some measurements myself:

root@Router:~# while true; do apcaccess |grep LINEV; sleep 30; done
LINEV    : 123.0 Volts
LINEV    : 123.0 Volts
LINEV    : 123.0 Volts
LINEV    : 123.0 Volts
LINEV    : 123.0 Volts
LINEV    : 123.0 Volts
LINEV    : 123.0 Volts
LINEV    : 123.0 Volts
LINEV    : 123.0 Volts
LINEV    : 123.0 Volts
LINEV    : 123.0 Volts
LINEV    : 123.0 Volts
LINEV    : 123.0 Volts
LINEV    : 123.0 Volts
LINEV    : 123.0 Volts
LINEV    : 123.0 Volts
LINEV    : 123.0 Volts
LINEV    : 123.0 Volts
LINEV    : 123.0 Volts
LINEV    : 123.0 Volts

So I guess the issue of the values not being refreshed unless there's an event or reconnection was introduced between 18.06.x and 19.07.

Here's a side-by-side comparison of one router with an APC UPS running apcupsd over 19.07.0 versus another router with another APC UPS running nut (over 22.03.5, and everything in the same room):

Quick reminder: this doesn´t happens with apcupsd over 18.06.x or older.

Edit:
I replaced the /usr/sbin binaries of apc from 19.07.0 over 18.06.9 and viceversa. No changes, so the issue should be something that mess with the measurements.

Which data protocol (apcsmart, usb, dumb) are you using on your UPS?

On my old Smart-UPS 750 (SUA750I, made approx. around 2006) i used the apcsmart protocol with both usb or serial connection on at least 3 different routers.
TP-Link TL-WDR4300 w/ OpenWrt since 15.05 up to 22.03, BT HomeHub 5A since 19.07 up to 22.03 and AVM Fritz!Box 7520 with current Master.
I had never any problems with the values not being refreshed using apcupsd 3.14.14.
But when the old UPS broke down and i got a newer one (SMT750I from 2018) the problems started. (link1)(link2)
Tl, dr;
I'm now using a AP9620 communication card in my SMT750I, with the old (complete) apcsmart/usb protocol on my FB7520 and had no problems for the last 3 weeks.
I still don't know why the MODBUS protocol made so many problems on the 7520 but worked fine on the older routers (w/ all of the above mentioned OpenWrt releases).
So the problems we both had are quite similar, but the cause seems to be quite different.

Hi there, thans for replying.

I have had one BE550G (older firmware/hardware apcupsd/ath79) and recenlty got a BE750G (newer firmware/hardware nut/mt76x8), both from 2008 and USB to RJ50 cable. (I also had a couple of modern Tripp·Lite's, as I've been trying to troubleshoot some things about apcupsd, nut, and RRDtool).

The majority of my tests have been done with the normal "usb" protocol, which used to run perfectly with several Tomato's and/or winapcupsd for several years. I haven't tried "smart" USB protocol until recently, when I started to pinpoint what's going on with OpenWrt's implementation. Anyways, "smart" doesn't make a difference for my testbed.

My findings so far:

  • 18.06.x => apcupsd 3.14.14-2 and the measurements shows logical variations
  • 19.07.0 => "a different binaries of" apcupds 3.14.14-2 and the flat liners start to occur (unless there's a reconnection or other kind of event)

To make this more curious, I swapped the /usr/sbin of one release into the other and it didn't change the observed behavior, so those binaries are not the root-cause in my case. I wonder: could it be the hid or usb-hid packages?