I wish to run the following software on OpenWrt:
What is the best way to approach this? Arm builds appear to be available here:
Can I just lift those? Or is it better to compile from scratch? The source uses "autotools".
Are there any tutorials available to guide through this?
Thank you for the tip. I've made some progress:
include $(TOPDIR)/rules.mk
PKG_NAME:=corosync-qdevice
PKG_VERSION:=3.0.3
PKG_RELEASE:=1
PKG_SOURCE:=corosync-qdevice-$(PKG_VERSION).tar.gz
PKG_SOURCE_URL:=https://github.com/corosync/corosync-qdevice/releases/download/v$(PKG_VERSION)/
PKG_HASH:=0a4705abd17af795287ad3bb18c0abacf3c0027222e45f149cb9bebeb6056926
PKG_FIXUP:=autoreconf
PKG_BUILD_DEPENDS:=nss
include $(INCLUDE_DIR)/package.mk
define Package/corosync-qdevice
SECTION:=corosync
CATEGORY:=Corosync
TITLE:=Corosync Qdevice
endef
define Package/corosync-qdevice/install
$(INSTALL_DIR) $(1)/usr/sbin
endef
$(eval $(call BuildPackage,corosync-qdevice))
This fails on a missing corosync-common, so:
include $(TOPDIR)/rules.mk
PKG_NAME:=corosync
PKG_VERSION:=3.1.8
PKG_RELEASE:=1
PKG_SOURCE:=corosync-$(PKG_VERSION).tar.gz
PKG_SOURCE_URL:=https://github.com/corosync/corosync/releases/download/v$(PKG_VERSION)/
PKG_HASH:=7023544fa3bb36c00bbcabd9935b7269b41d896738a108ed32ea9b9c9b27ec3d
PKG_FIXUP:=autoreconf
PKG_BUILD_DEPENDS:=libqb
include $(INCLUDE_DIR)/package.mk
define Package/corosync
SECTION:=corosync
CATEGORY:=Corosync
TITLE:=Corosync
endef
define Package/corosync/install
$(INSTALL_DIR) $(1)/usr/sbin
endef
$(eval $(call BuildPackage,corosync))
This fails on a missing libqb, so:
include $(TOPDIR)/rules.mk
PKG_NAME:=libqb
PKG_VERSION:=2.0.8
PKG_RELEASE:=1
PKG_SOURCE:=libqb-$(PKG_VERSION).tar.gz
PKG_SOURCE_URL:=https://github.com/ClusterLabs/libqb/releases/download/v$(PKG_VERSION)/
PKG_HASH:=4832d0c1f12be38cbcf608585758d4a3124def253a3914b61afeebc68484311e
PKG_FIXUP:=autoreconf
PKG_BUILD_DEPENDS:=libxml2
include $(INCLUDE_DIR)/package.mk
define Package/libqb
SECTION:=corosync
CATEGORY:=Corosync
TITLE:=libqb
endef
define Package/libqb/install
echo "=================================="
find $(PKG_BUILD_DIR)
echo "=================================="
endef
$(eval $(call BuildPackage,libqb))
Which seems to build okay but:
./autogen.sh
./configure
make
make install
From the docs is it correct to think that these happen by default (hence no definitions needed in the package Makefile)?
Aha - make install is only called by default if PKG_INSTALL:=1. From here I will see what files to package (using InstallDev and Package/install resp)
Great job, It woult be great to use a openwrt device as a qdevice. Ill be happy to test it. 
I think I've managed to compile libqb, but making corosync keeps complaining that the libqb is not found, even though I have both PKG_BUILD_DEPENDS and DEPENDS set:
include $(TOPDIR)/rules.mk
PKG_NAME:=corosync
PKG_VERSION:=3.1.8
PKG_RELEASE:=1
PKG_SOURCE:=corosync-$(PKG_VERSION).tar.gz
PKG_SOURCE_URL:=https://github.com/corosync/corosync/releases/download/v$(PKG_VERSION)/
PKG_HASH:=7023544fa3bb36c00bbcabd9935b7269b41d896738a108ed32ea9b9c9b27ec3d
PKG_FIXUP:=autoreconf
PKG_INSTALL:=1
PKG_BUILD_DEPENDS:=libqb
include $(INCLUDE_DIR)/package.mk
define Package/corosync
SECTION:=corosync
CATEGORY:=Corosync
TITLE:=Corosync
DEPENDS:=+libqb
endef
define Package/corosync/install
echo "=========================="
find $(PKG_INSTALL_DIR)
echo "=========================="
endef
$(eval $(call BuildPackage,corosync))
Any tips?
The problem was a typo in my libqb makefile - the InstallDev was named as Install so wasn't including the headers.
I also needed kronosnet to be built and made good progress... until:
checking for netinet/sctp.h... (cached) no
configure: error: "missing required SCTP headers"
Unlike the other build dependences, adding lksctp-tools to the kronosnet Makefile didn't work:
include $(TOPDIR)/rules.mk
PKG_NAME:=kronosnet
PKG_VERSION:=1.28
PKG_RELEASE:=1
PKG_SOURCE:=kronosnet-$(PKG_VERSION).tar.gz
PKG_SOURCE_URL:=https://github.com/kronosnet/kronosnet/releases/download/v$(PKG_VERSION)/
PKG_HASH:=ea0d44fd7e891320378b419a38ca91c4edbcdc06e5b961aedabb4159d7e8150f
PKG_FIXUP:=autoreconf
PKG_INSTALL:=1
PKG_BUILD_DEPENDS:=libopenssl zstd liblz4 lzo xz bzip2 lksctp-tools
include $(INCLUDE_DIR)/package.mk
define Package/kronosnet
SECTION:=corosync
CATEGORY:=Corosync
TITLE:=kronosnet
endef
define Build/InstallDev
echo "============================="
find $(PKG_INSTALL_DIR)
echo "============================="
endef
define Package/kronosnet/install
endef
$(eval $(call BuildPackage,kronosnet))
It seems that lksctp-tools should have the correct libraries and when searching the filesystem seems to be available:
❯ find | grep netinet
./staging_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/usr/include/netinet
./staging_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/usr/include/netinet/sctp.h
Help please!
Kronosnet's Makefile has the following:
if test "x$enable_libknet_sctp" = xyes; then
AC_CHECK_HEADERS([netinet/sctp.h],, [AC_MSG_ERROR(["missing required SCTP headers"])])
fi
which seems to be failing. I suspect it's not set up to look in the right place.
I'll next look into disabling this feature (as I don't think it's needed in a qdevice) but it would be good to have the default feature switched on - does the makefile need patching or a location set to look in the right place?
I was able to make progress by disabling sctp in the kronosnet makefile (as well as a few other things). After some more dependency wrangling, patching and confusion with how to actually build an installable binary (apparently you don't get an IPK unless you select it in make menuconfig - go figure!), I managed to build an IPK for qdevice.
I'm currently testing it, but I don't think I'll be submitting any makefiles to openwrt anytime soon considering the haphazard journey it took to get here! 
I just started looking into building this possibly. Wanna share your stuff? No matter what's broken - I will at least try to help
Excuse the delay - I was rebuilding my cluster and wanted to actually try it before sharing. I'm glad to say that the build worked, albeit without TLS support. Here are the makefiles:
#libqb
include $(TOPDIR)/rules.mk
PKG_NAME:=libqb
PKG_VERSION:=2.0.8
PKG_RELEASE:=1
PKG_SOURCE:=libqb-$(PKG_VERSION).tar.gz
PKG_SOURCE_URL:=https://github.com/ClusterLabs/libqb/releases/download/v$(PKG_VERSION)/
PKG_HASH:=4832d0c1f12be38cbcf608585758d4a3124def253a3914b61afeebc68484311e
PKG_FIXUP:=autoreconf
PKG_INSTALL:=1
PKG_BUILD_DEPENDS:=libxml2
include $(INCLUDE_DIR)/package.mk
define Package/libqb
SECTION:=corosync
CATEGORY:=Corosync
TITLE:=libqb
DEPENDS:=+libxml2
endef
define Build/InstallDev
$(INSTALL_DIR) $(1)/usr/include/qb
$(INSTALL_DATA) $(PKG_INSTALL_DIR)/usr/include/qb/* $(1)/usr/include/qb
$(INSTALL_DIR) $(1)/usr/lib
$(INSTALL_DATA) $(PKG_INSTALL_DIR)/usr/lib/lib*.so* $(1)/usr/lib/
$(INSTALL_DIR) $(1)/usr/lib/pkgconfig
$(INSTALL_DATA) $(PKG_INSTALL_DIR)/usr/lib/pkgconfig/*.pc $(1)/usr/lib/pkgconfig
$(INSTALL_DIR) $(1)/usr/bin
$(INSTALL_BIN) $(PKG_INSTALL_DIR)/usr/bin/* $(1)/usr/bin
$(INSTALL_DIR) $(1)/usr/sbin
$(INSTALL_BIN) $(PKG_INSTALL_DIR)/usr/sbin/* $(1)/usr/sbin
endef
define Package/libqb/install
$(INSTALL_DIR) $(1)/usr/lib
$(INSTALL_DATA) $(PKG_INSTALL_DIR)/usr/lib/lib*.so.* $(1)/usr/lib/
$(INSTALL_DIR) $(1)/usr/bin
$(INSTALL_BIN) $(PKG_INSTALL_DIR)/usr/bin/* $(1)/usr/bin
$(INSTALL_DIR) $(1)/usr/sbin
$(INSTALL_BIN) $(PKG_INSTALL_DIR)/usr/sbin/* $(1)/usr/sbin
endef
$(eval $(call BuildPackage,libqb))
#kronosnet
include $(TOPDIR)/rules.mk
PKG_NAME:=kronosnet
PKG_VERSION:=1.28
PKG_RELEASE:=1
PKG_SOURCE:=kronosnet-$(PKG_VERSION).tar.gz
PKG_SOURCE_URL:=https://github.com/kronosnet/kronosnet/releases/download/v$(PKG_VERSION)/
PKG_HASH:=ea0d44fd7e891320378b419a38ca91c4edbcdc06e5b961aedabb4159d7e8150f
PKG_FIXUP:=autoreconf
PKG_INSTALL:=1
PKG_BUILD_DEPENDS:=openssl zstd liblz4 lzo xz bzip2 nss libqb libnl
include $(INCLUDE_DIR)/package.mk
define Package/kronosnet
SECTION:=corosync
CATEGORY:=Corosync
TITLE:=kronosnet
DEPENDS:=+libopenssl +liblz4 +liblzo +bzip2 +libnss +libnl +nspr +liblzma +libzstd
endef
define Build/Configure
$(call Build/Configure/Default, --disable-libknet-sctp --disable-man --disable-hardening)
endef
define Build/InstallDev
$(INSTALL_DIR) $(1)/usr/include/
$(INSTALL_DATA) $(PKG_INSTALL_DIR)/usr/include/* $(1)/usr/include/
$(INSTALL_DIR) $(1)/usr/lib
$(INSTALL_DATA) $(PKG_INSTALL_DIR)/usr/lib/lib*.so* $(1)/usr/lib/
$(INSTALL_DIR) $(1)/usr/lib/pkgconfig
$(INSTALL_DATA) $(PKG_INSTALL_DIR)/usr/lib/pkgconfig/*.pc $(1)/usr/lib/pkgconfig
$(INSTALL_DIR) $(1)/usr/lib/kronosnet
$(INSTALL_DATA) $(PKG_INSTALL_DIR)/usr/lib/kronosnet/*.so* $(1)/usr/lib/kronosnet
endef
define Package/kronosnet/install
$(INSTALL_DIR) $(1)/usr/lib
$(INSTALL_DATA) $(PKG_INSTALL_DIR)/usr/lib/lib*.so.* $(1)/usr/lib/
$(INSTALL_DIR) $(1)/usr/lib/kronosnet
$(INSTALL_DATA) $(PKG_INSTALL_DIR)/usr/lib/kronosnet/*.so* $(1)/usr/lib/kronosnet
endef
$(eval $(call BuildPackage,kronosnet))
Patch for Kronosnet
--- a/libknet/links.c
+++ b/libknet/links.c
@@ -896,7 +896,7 @@
if ((interval * 1000) < KNET_THREADS_TIMERES) {
log_warn(knet_h, KNET_SUB_LINK,
- "host: %u link: %u interval: %lu too small (%s). interval lower than thread_timer_res (%u ms) has no effect",
+ "host: %u link: %u interval: %llu too small (%s). interval lower than thread_timer_res (%u ms) has no effect",
host_id, link_id, interval, strerror(savederrno), (KNET_THREADS_TIMERES / 1000));
}
@@ -904,7 +904,7 @@
err = -1;
savederrno = EINVAL;
log_err(knet_h, KNET_SUB_LINK,
- "host: %u link: %u pong timeout: %lu too small (%s). timeout cannot be less than thread_timer_res (%u ms)",
+ "host: %u link: %u pong timeout: %llu too small (%s). timeout cannot be less than thread_timer_res (%u ms)",
host_id, link_id, timeout, strerror(savederrno), (KNET_THREADS_TIMERES / 1000));
goto exit_unlock;
}
#corosync
include $(TOPDIR)/rules.mk
PKG_NAME:=corosync
PKG_VERSION:=3.1.8
PKG_RELEASE:=1
PKG_SOURCE:=corosync-$(PKG_VERSION).tar.gz
PKG_SOURCE_URL:=https://github.com/corosync/corosync/releases/download/v$(PKG_VERSION)/
PKG_HASH:=7023544fa3bb36c00bbcabd9935b7269b41d896738a108ed32ea9b9c9b27ec3d
PKG_FIXUP:=autoreconf
PKG_INSTALL:=1
PKG_BUILD_DEPENDS:=libqb kronosnet
include $(INCLUDE_DIR)/package.mk
define Package/corosync
SECTION:=corosync
CATEGORY:=Corosync
TITLE:=Corosync
DEPENDS:=+kronosnet +libqb
endef
define Build/InstallDev
$(INSTALL_DIR) $(1)/usr/include/corosync
$(INSTALL_DATA) $(PKG_INSTALL_DIR)/usr/include/corosync/* $(1)/usr/include/corosync/
$(INSTALL_DIR) $(1)/usr/lib
$(INSTALL_DATA) $(PKG_INSTALL_DIR)/usr/lib/lib*.so* $(1)/usr/lib/
$(INSTALL_DIR) $(1)/usr/lib/pkgconfig
$(INSTALL_DATA) $(PKG_INSTALL_DIR)/usr/lib/pkgconfig/*.pc $(1)/usr/lib/pkgconfig
$(INSTALL_DIR) $(1)/usr/bin
$(INSTALL_BIN) $(PKG_INSTALL_DIR)/usr/bin/* $(1)/usr/bin
$(INSTALL_DIR) $(1)/usr/sbin
$(INSTALL_BIN) $(PKG_INSTALL_DIR)/usr/sbin/* $(1)/usr/sbin
endef
define Package/corosync/install
$(INSTALL_DIR) $(1)/usr/lib
$(INSTALL_DATA) $(PKG_INSTALL_DIR)/usr/lib/lib*.so* $(1)/usr/lib/
$(INSTALL_DIR) $(1)/usr/bin
$(INSTALL_BIN) $(PKG_INSTALL_DIR)/usr/bin/* $(1)/usr/bin
$(INSTALL_DIR) $(1)/usr/sbin
$(INSTALL_BIN) $(PKG_INSTALL_DIR)/usr/sbin/* $(1)/usr/sbin
endef
$(eval $(call BuildPackage,corosync))
#corosync-qdevice
include $(TOPDIR)/rules.mk
PKG_NAME:=corosync-qdevice
PKG_VERSION:=3.0.3
PKG_RELEASE:=1
PKG_SOURCE:=corosync-qdevice-$(PKG_VERSION).tar.gz
PKG_SOURCE_URL:=https://github.com/corosync/corosync-qdevice/releases/download/v$(PKG_VERSION)/
PKG_HASH:=0a4705abd17af795287ad3bb18c0abacf3c0027222e45f149cb9bebeb6056926
PKG_FIXUP:=autoreconf
PKG_INSTALL:=1
PKG_BUILD_DEPENDS:=nss corosync
include $(INCLUDE_DIR)/package.mk
define Package/corosync-qdevice
SECTION:=corosync
CATEGORY:=Corosync
TITLE:=Corosync Qdevice
DEPENDS:=+corosync +libnss
endef
define Build/InstallDev
$(INSTALL_DIR) $(1)/usr/share/pkgconfig
$(INSTALL_DATA) $(PKG_INSTALL_DIR)/usr/share/pkgconfig/corosync-qdevice.pc $(1)/usr/share/pkgconfig
$(INSTALL_DIR) $(1)/usr/bin
$(INSTALL_BIN) $(PKG_INSTALL_DIR)/usr/bin/* $(1)/usr/bin
$(INSTALL_DIR) $(1)/usr/sbin
$(INSTALL_BIN) $(PKG_INSTALL_DIR)/usr/sbin/* $(1)/usr/sbin
endef
define Package/corosync-qdevice/install
$(INSTALL_DIR) $(1)/usr/bin
$(INSTALL_BIN) $(PKG_INSTALL_DIR)/usr/bin/* $(1)/usr/bin
$(INSTALL_DIR) $(1)/usr/sbin
$(INSTALL_BIN) $(PKG_INSTALL_DIR)/usr/sbin/* $(1)/usr/sbin
endef
$(eval $(call BuildPackage,corosync-qdevice))
Have fun!
Thank you, @spammy, for the great work!
I just finished a successful build using your code. Do you have any intention to publish it anywhere? Maybe a github repository?
Do you have any updates? Any comments about how it is working for you in production?
P.S: I noticed the patch for kronosnet should not be applied for some archs (e.g. x86). We should probably update the Makefile for this.
Glad it worked! I've not had to think about it since deploying so it's working very well for me.
I don't plan to publish it so have it kept in a private repo for now. For all intents and purposes I consider it final (for my needs).
I'm happy for someone to take it forward to a stage where it could be included with openwrt. I don't think it's too far off.
Great news! Thank you for the feedback. It is nice to know it is working.
With your consent, I would like to use your code. I've already started a new repo, but I'm still learning the ropes.
Just for the sake of curiosity: in this private repo, do you have exactly the same code you posted above? Or is there any new fix/change?
The above code is the latest version. I did also create an init file, but manually and not in the package.
You're welcome to use the code (a credit would be nice!), and I strongly encourage you see if you can get it into the openwrt tree 
through a lot of experimenting, I was able to compile the 4 package files. However, when trying to add the openwrt router to the cluster, it complains about not having corosync-qnetd "ash: corosync-qnetd-certutil: not found".
Any pointers?
This rings a bell. Checking my init I use the following:
#!/bin/sh /etc/rc.common
START=99
STOP=99
USE_PROCD=1
SERVICE_DAEMONIZE=1
SERVICE_WRITE_PID=1
start_service() {
procd_open_instance
procd_set_param command /usr/bin/corosync-qnetd -s off
procd_set_param respawn
procd_close_instance
}
So try with -s off?
still get this when running pvecm qdevice setup OPENWRT_IP -f from the proxmox node
INFO: initializing qnetd server
ash: corosync-qnetd-certutil: not found
INFO: copying CA cert and initializing on all nodes
scp: /etc/corosync/qnetd/nssdb/qnetd-cacert.crt: No such file or directory
command 'scp -o 'BatchMode=yes' 'root@[OPENWRT_IP]:/etc/corosync/qnetd/nssdb/qnetd-cacert.crt' /etc/pve/qnetd-cacert.crt' failed: exit code 1
Yes, that's expected:
You can't use SSL - I'm not sure how you're setting up the cluster but the default pve tools default to SSL so you'll have to roll your own, or edit the script to set SSL to off.
do I need to disable SSL on each node, or re-create the cluster without SSL? Any pointers to where to start? I already deleted the cluster so I can start fresh.