Domoticz library errors (error relocating, symbol not found) after upgrade to 23.05.4

Installing and Using OpenWrt
1

Hi!
I had a working Domoticz setup on a TPLink Archer C7 on 22.03.2.
Upgraded to 23.05.4, Domoticz still worked.
Did an opkg upgrade which also upgraded Domoticz, but it won't start now because of this:

# ldd /usr/bin/domoticz.23.05
	/lib/ld-musl-mips-sf.so.1 (0x77dc0000)
	libjsoncpp.so.25 => /usr/lib/libjsoncpp.so.25 (0x77d8a000)
	libsqlite3.so.0 => /usr/lib/libsqlite3.so.0 (0x77ca6000)
	libminizip.so.3 => /usr/lib/libminizip.so.3 (0x77c74000)
	libz.so.1 => /usr/lib/libz.so.1 (0x77c52000)
	libssl.so.3 => /usr/lib/libssl.so.3 (0x77bbe000)
	libcrypto.so.3 => /usr/lib/libcrypto.so.3 (0x778b6000)
	libboost_thread.so.1.82.0 => /usr/lib/libboost_thread.so.1.82.0 (0x77884000)
	libboost_system.so.1.82.0 => /usr/lib/libboost_system.so.1.82.0 (0x77862000)
	libopenzwave.so.1.6 => /usr/lib/libopenzwave.so.1.6 (0x77712000)
	libboost_chrono.so.1.82.0 => /usr/lib/libboost_chrono.so.1.82.0 (0x776f0000)
	libboost_atomic.so.1.82.0 => /usr/lib/libboost_atomic.so.1.82.0 (0x776ca000)
	libcurl.so.4 => /usr/lib/libcurl.so.4 (0x77666000)
	libmosquitto.so.1 => /usr/lib/libmosquitto.so.1 (0x77634000)
	liblua5.3.so.0.0.0 => /usr/lib/liblua5.3.so.0.0.0 (0x775fe000)
	libtelldus-core.so.2 => /usr/lib/libtelldus-core.so.2 (0x775dc000)
	libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x77406000)
	libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x773d4000)
	libc.so => /lib/ld-musl-mips-sf.so.1 (0x77dc0000)
	libatomic.so.1 => /lib/libatomic.so.1 (0x773b2000)
	libnghttp2.so.14 => /usr/lib/libnghttp2.so.14 (0x77380000)
	libmbedtls.so.14 => /usr/lib/libmbedtls.so.14 (0x7734e000)
	libmbedx509.so.1 => /usr/lib/libmbedx509.so.1 (0x7732c000)
	libmbedcrypto.so.7 => /usr/lib/libmbedcrypto.so.7 (0x772b8000)
	libcares.so.2 => /usr/lib/libcares.so.2 (0x77286000)
Error relocating /usr/bin/domoticz.23.05: _ZN4Json5ValueixERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE: symbol not found
Error relocating /usr/bin/domoticz.23.05: _ZN4Json5Value12removeMemberERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE: symbol not found
Error relocating /usr/bin/domoticz.23.05: _ZN9OpenZWave7Manager18GetNodeProductNameB5cxx11Ejh: symbol not found
Error relocating /usr/bin/domoticz.23.05: _ZNK4Json5Value14toStyledStringB5cxx11Ev: symbol not found
Error relocating /usr/bin/domoticz.23.05: _ZN9OpenZWave7Manager13GetValueLabelB5cxx11ERKNS_7ValueIDEi: symbol not found
Error relocating /usr/bin/domoticz.23.05: _ZN9OpenZWave7Manager23GetNodeDeviceTypeStringB5cxx11Ejh: symbol not found
Error relocating /usr/bin/domoticz.23.05: _ZN4Json19StreamWriterBuilderixERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE: symbol not found
Error relocating /usr/bin/domoticz.23.05: _ZN9OpenZWave7Manager21GetNodeManufacturerIdB5cxx11Ejh: symbol not found
Error relocating /usr/bin/domoticz.23.05: _ZN9OpenZWave7Manager13GetValueUnitsB5cxx11ERKNS_7ValueIDE: symbol not found
Error relocating /usr/bin/domoticz.23.05: _ZNK4Json5Value3getERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKS0_: symbol not found
Error relocating /usr/bin/domoticz.23.05: _ZNK4Json17ValueIteratorBase4nameB5cxx11Ev: symbol not found
Error relocating /usr/bin/domoticz.23.05: _ZN9OpenZWave7Manager18getVersionAsStringB5cxx11Ev: symbol not found
Error relocating /usr/bin/domoticz.23.05: _ZN4Json11writeStringB5cxx11ERKNS_12StreamWriter7FactoryERKNS_5ValueE: symbol not found
Error relocating /usr/bin/domoticz.23.05: _ZN9OpenZWave7Manager17GetNodeRoleStringB5cxx11Ejh: symbol not found
Error relocating /usr/bin/domoticz.23.05: _ZN9OpenZWave7Manager11GetNodeNameB5cxx11Ejh: symbol not found
Error relocating /usr/bin/domoticz.23.05: _ZN9OpenZWave7Manager8SetValueERKNS_7ValueIDERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE: symbol not found
Error relocating /usr/bin/domoticz.23.05: _ZN9OpenZWave7Manager16GetValueAsStringERKNS_7ValueIDEPNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE: symbol not found
Error relocating /usr/bin/domoticz.23.05: _ZN9OpenZWave7Manager13GetGroupLabelB5cxx11Ejhh: symbol not found
Error relocating /usr/bin/domoticz.23.05: _ZN9OpenZWave7Manager17GetNodeQueryStageB5cxx11Ejh: symbol not found
Error relocating /usr/bin/domoticz.23.05: _ZN9OpenZWave7Manager18GetNodeProductTypeB5cxx11Ejh: symbol not found
Error relocating /usr/bin/domoticz.23.05: _ZNK4Json5Value14getMemberNamesB5cxx11Ev: symbol not found
Error relocating /usr/bin/domoticz.23.05: _ZN9OpenZWave7Manager15GetNodeLocationB5cxx11Ejh: symbol not found
Error relocating /usr/bin/domoticz.23.05: _ZN9OpenZWave7Manager15SetNodeLocationEjhRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE: symbol not found
Error relocating /usr/bin/domoticz.23.05: _ZN9OpenZWave7Manager21GetValueListSelectionERKNS_7ValueIDEPNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE: symbol not found
Error relocating /usr/bin/domoticz.23.05: _ZNK4Json5ValueixERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE: symbol not found
Error relocating /usr/bin/domoticz.23.05: _ZN9OpenZWave7Manager16GetNodeProductIdB5cxx11Ejh: symbol not found
Error relocating /usr/bin/domoticz.23.05: _ZNK9OpenZWave12Notification11GetAsStringB5cxx11Ev: symbol not found
Error relocating /usr/bin/domoticz.23.05: _ZN9OpenZWave7Manager12GetValueHelpB5cxx11ERKNS_7ValueIDEi: symbol not found
Error relocating /usr/bin/domoticz.23.05: _ZN9OpenZWave7Manager11SetNodeNameEjhRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE: symbol not found
Error relocating /usr/bin/domoticz.23.05: _ZN4Json4PathC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_12PathArgumentESB_SB_SB_SB_: symbol not found
Error relocating /usr/bin/domoticz.23.05: _ZN9OpenZWave7Options6CreateERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_S8_: symbol not found
Error relocating /usr/bin/domoticz.23.05: _ZN9OpenZWave7Manager21SetValueListSelectionERKNS_7ValueIDERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE: symbol not found
Error relocating /usr/bin/domoticz.23.05: _ZN9OpenZWave7Manager22getVersionLongAsStringB5cxx11Ev: symbol not found
Error relocating /usr/bin/domoticz.23.05: _ZN9OpenZWave7Options13AddOptionBoolERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEb: symbol not found
Error relocating /usr/bin/domoticz.23.05: _ZN9OpenZWave7Manager23GetNodeManufacturerNameB5cxx11Ejh: symbol not found
Error relocating /usr/bin/domoticz.23.05: _ZNK4Json5Value8asStringB5cxx11Ev: symbol not found
Error relocating /usr/bin/domoticz.23.05: _ZN4Json15parseFromStreamERKNS_10CharReader7FactoryERSiPNS_5ValueEPNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE: symbol not found
Error relocating /usr/bin/domoticz.23.05: _ZN9OpenZWave7Options15AddOptionStringERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_b: symbol not found
Error relocating /usr/bin/domoticz.23.05: _ZN9OpenZWave7Manager21GetNodePlusTypeStringB5cxx11Ejh: symbol not found
Error relocating /usr/bin/domoticz.23.05: _ZN9OpenZWave7Manager17GetValueListItemsERKNS_7ValueIDEPSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaISA_EE: symbol not found
Error relocating /usr/bin/domoticz.23.05: _ZN9OpenZWave7Options12AddOptionIntERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEi: symbol not found
Error relocating /usr/bin/domoticz.23.05: _ZN9OpenZWave7Manager11GetNodeTypeB5cxx11Ejh: symbol not found
Error relocating /usr/bin/domoticz.23.05: _ZN9OpenZWave7Manager9AddDriverERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_6Driver19ControllerInterfaceE: symbol not found
Error relocating /usr/bin/domoticz.23.05: _ZN4Json5ValueC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE: symbol not found

This very much sounds like the respective libraries have not been updated correctly, but both libopenzwave and jsoncpp are up-to-date (or am I searching for the wrong packages? I don't get any packages in opkg list-upgradable except kernel modules which won't upgrade because of a kernel version mismatch)
I saw reports of people with other problems, but at least their binary seems to link successfully.
Could it be domoticz needs to be recompiled/linked against the current library versions for my architecture when it is fixed in others already?
Or do I miss something else?

I "downgraded" to the last version (i.e. unpacked all ipkgs of domoticzs and the four boost libs) and luckily it's working right now, but I'd need a proper fix.

Thanks in advance for any pointers!

2

Please connect to your OpenWrt device using ssh and copy the output of the following commands and post it here using the "Preformatted text </> " button:
grafik
Remember to redact passwords, MAC addresses and any public IP addresses you may have:

ubus call system board
opkg search /usr/bin/domoticz.23.05
opkg list-installed domoticz
3

I don't have access to the device at the moment, but I have the exact same hardware here.

# ubus call system board
{
	"kernel": "5.15.162",
	"hostname": "ap3",
	"system": "Qualcomm Atheros QCA9558 ver 1 rev 0",
	"model": "TP-Link Archer C7 v2",
	"board_name": "tplink,archer-c7-v2",
	"rootfs_type": "squashfs",
	"release": {
	        "distribution": "OpenWrt",
	        "version": "23.05.4",
	        "revision": "r24012-d8dd03c46f",
	        "target": "ath79/generic",
	        "description": "OpenWrt 23.05.4 r24012-d8dd03c46f"
	}
}

opkg search /usr/bin/domoticz.23.05

won't work, I renamed the binary on purpose. Normally it is part of the domoticz package I installed from the repo.

I'll provide the rest as soon as I have access to the device again.
And I'll probably try adding an extroot to my router and install it here as well to see whether it is an architecture problem or a setup problem.

Got me thinking, much appreciated!

4

You said that, do backups and install everything on top of clean openwrt.

5

Upgrading packages (via the CLI opkg upgrade command or the LuCI Upgrade... button) can result in major problems. It is generally highly discouraged, unless you know what you are doing or if there is specific instruction to do so.

6

I meant I just renamed it to have two versions in parallel to compare them :slight_smile: The unexpected name is not the actual error.

7

I understand upgrading is discoured, and I also understand why (for core packages!).
But I don't see how a package I installed to an extroot in some older version would ever get an upgrade without okpg upgrade?
I definitely had a lot of user-installed packages on past versions (I think I use openWRT/LEDE 20 years already :sweat_smile: ) I definitely had to upgrade, as the C lib had a different ABI after a sysupgrade and wouldn't start because of relocation errors.

I already tried removing and reinstalling domoticz and its deps to no avail, that didn't help.
But I'll try a clean reinstall just for good measure (I'm a Gentoo user, normally I never install things anew, just fix them in place :slight_smile: )

Will report back.

8

musl does not have compat stubs like glibc.

9

It wouldn't. You'd use a standard sysupgrade (which also necessitates rebuilding the extroot).

That should fix your issue.

It's a different story for OpenWrt (and likely similar OS's that target small embedded systems). That's the whole reason we wrote the wiki article referenced above.

10

I don't mean to be unteachable and I'll definitely go the route suggested above, so bear with me!
But just for my understanding (I am a seasoned linux sysadmin in my day job and this somehow trips me up):

How does rebuilding the extroot (i.e. wiping it and doing opkg install for the packages that were on it before, and copying back user data like the domoticz config+DB in this case, I guess?) differ technically from opkg upgrade (which should just overwrite whats there with the newly downloaded package, just like opkg install would, though there would be nothing to overwrite?)
I understand there might be files that don't get removed as opkg is not apt-get, but the binary I face issues with? How would that not get overwritten?
I even did a remove and reinstall (of domoticz only), which did not help.
Dependant libraries might be in LD_LIBRARY_PATH in wrong versions, but I double checked that as well.

The wikis says stuff goes wrong because of not matching ABI.
But a given release (say, 23.05.4) and its packages (that opkg uses) have to be ABI compatible, or an initial install (not upgrade) wouldn't work, either, right?
I need to understand things technically to memorize them and just can't wrap my head around what a reinstall would change. After all, opkgs are just tarballs with a little metadata around them.

It really feels like Windows if reinstalling magically fixes things and you don't have any clue what actually changed!

Thanks in advance!

PS: I feel the wiki should be reorganized. The part regarding upgrades affects everyone. It should not be after sections for LUKS setups, USB modeswitching directives, etc. which only few people will work with.

11

Part of the assumption here is that a sysupgrade specifically means an upgrade to the OpenWrt version, not simply a reset to defaults of the current version (which you can do without flashing).

So working from the above stated assumption, you're upgrading the entire OS, including the kernel. The kernel, upon which nearly everything depends, and other major parts of the system do not get upgraded if you use opkg upgrade.

Now, there are a few possible gottchas about package upgrades in general...

  1. there are lots of examples where the bulk upgrade simply consumes all the available space on the router, causing the upgrades to fail and leaving the system in a bad state
  2. there are indeed no ABI checks with opkg, so conditions may exist that could cause incompatibilities or other issues which could be related to dependencies and other things, and stuff could get out of sync and break.
  3. The general guidance is to only upgrade individual packages if and when there are bug fixes/security patches and/or feature additions that are actually necessary/relevant to your situation. Upgrading "just because" isn't really a good idea because of the above two points.

That said, when space is not an issue and everything is of a consistent OpenWrt version + kernel version, you should be right about the fact that upgrading packages should be no different than installing fresh. I cannot answer that question with certainty because I'm not aware of a specific set of controlled tests that can provide the exact insight into the failure mechanism. I have not personally run those tests (and for that matter, I've never used opkg upgrade either for all the reasons mentioned).

I think that the transition to the apk package manager (vs opkg) will help prevent some of these issues (space may still be a limiting factor in many devices), but I cannot say for sure.

12

working from the above stated assumption, you're upgrading the entire OS, including the kernel. The kernel, upon which nearly everything depends, and other major parts of the system do not get upgraded if you use opkg upgrade

I generally only ever assumed an opkg upgrade to be valid after a sysupgrade (not only reset to defaults).
And I would have thought if I do a sysupgrade AND an opkg upgrade both the kernel, libs and userspace shoud be in sync again, i.e. a sysupgrade to 23.05.4 and opkg upgrade would unconditionally upgrade to a compatible set of kernel, libs and userspace that have been built/tested against a 23.05.4 system.

there are lots of examples where the bulk upgrade simply consumes all the available space on the router, causing the upgrades to fail and leaving the system in a bad state

Space is definitely not an issue here, as I have to use an extroot for installing domoticz, anyway. But yes, definitely a reason for telling people to not try this at home!

The general guidance is to only upgrade individual packages if and when there are bug fixes/security patches and/or feature additions that are actually necessary/relevant to your situation. Upgrading "just because" isn't really a good idea because of the above two points.

The day-job sysadmin in me says upgrade everything whenever possible. I don't want to track every vulnerability of every package installed.
I understand the "never touch a running system" methodology here, but I'm willing to break stuff in maintenance windows if it makes for more current packages on internet-facing devices.
Doesn't apply to rookies, yes, and the advice is definitely sound.

there are indeed no ABI checks with opkg, so conditions may exist that could cause incompatibilities or other issues which could be related to dependencies and other things, and stuff could get out of sync and break.

And I think this is what it all boils down to: Some dependency does not seem to get upgraded together with all the packages that did. And yes, apk seems promising in handling this better as I guess Alpine will definitely have to deal with dependencies like any other full-blown linux distro.
The next upgrade based on apk will tell whether it fixed it (and yes, I am definitely willing to break things again. For Science!)

I did a test install on a fresh extroot on the very same hardware I have at home here and indeed, domoticz works. So the package is not really to blame linking-wise.

I'll do a diff to the extroot of the actual router and see what packages may have been left out.

Thanks for your time!

13

Everything you need is already included after a sysupgrade (and potentially opkg install). Upgrading packages is never necessary except for the rare case where a package needs to be upgraded due to a bug/security flaw or new feature addition that is actually needed.

It is very rare that the packages are found to have vulnerabilities that are material within the context of OpenWrt, but if/when those do come up, the information and remediation instructions are included in the announcements forum section.

The fix is simple as long as you are prepared... so yes, for science!

Anyitme!

If your problem is solved, please consider marking this topic as [Solved]. See How to mark a topic as [Solved] for a short how-to.
Thanks! :slight_smile: