Hi,
I am building customized image for the RaspberryPi 3, but I have no idea how to include e.g. the Python module paho-mqtt (https://pypi.org/project/paho-mqtt/) thats not listed in menuconfig.
Exists any further description or can somebody give some hints?
Thanks,
Nic
Very good question. If someone can help us with this topic I would be very grateful.
Pawel
Create a package for it and include in the build -- I'd use an existing Python 3 package Makefile as a template.
Thanks Jeff,
would that an example how to include a python module:
I tried this way, but as I remember it didnt build anything.
Nic
Same here. Build is complete but the libraries are not there.
I created a repo with a link to Virtualbox machine image that compiles clean image.
I was struggling to build my first image and decided to share it:
Ok, I found in 19.07 branch the package "/openwrt/feeds/packages/lang/python/python-paho-mqtt", it was already added but only for python3. If I take this one as a template for example, where do I have to put it?
Well, I copied it to /openwrt/package/python2-paho-mqtt changed the Makefile slightly according to python2:
include $(TOPDIR)/rules.mk
PKG_NAME:=python2-paho-mqtt
PKG_VERSION:=1.4.0
PKG_RELEASE:=1
PKG_LICENSE:=Eclipse Public License v1.0 / Eclipse Distribution License v1.0
PKG_MAINTAINER:=Josef Schlehofer <josef.schlehofer@nic.cz>, Alexandru Ardelean <ardeleanalex@gmail.com>
PKG_SOURCE:=paho-mqtt-$(PKG_VERSION).tar.gz
PKG_SOURCE_URL:=https://files.pythonhosted.org/packages/25/63/db25e62979c2a716a74950c9ed658dce431b5cb01fde29eb6cba9489a904
PKG_HASH:=e440a052b46d222e184be3be38676378722072fcd4dfd2c8f509fb861a7b0b79
PKG_BUILD_DIR:=$(BUILD_DIR)/paho-mqtt-$(PKG_VERSION)
PKG_BUILD_DEPENDS:=python
include $(INCLUDE_DIR)/package.mk
#include ../python-package.mk
$(call include_mk, python-package.mk)
define Package/python2-paho-mqtt
SECTION:=lang
CATEGORY:=Languages
SUBMENU:=Python
TITLE:=python2-paho-mqtt
URL:=http://eclipse.org/paho
DEPENDS:=+python-light
VARIANT:=python
endef
define Package/python2-paho-mqtt/description
MQTT version 3.1/3.1.1 client class
endef
$(eval $(call PyPackage,python2-paho-mqtt))
$(eval $(call BuildPackage,python2-paho-mqtt))
Found it in "make menuconfig" and selected it. No check errors:
$ make package/python2-paho-mqtt/check
make[1] package/python2-paho-mqtt/check
make[2] -C package/python2-paho-mqtt check
It ends without building the package (Nothing to be done for 'compile'.):
$ make package/python2-paho-mqtt/compile V=sc
...
echo "python-base" >> /home/nic/openwrt/staging_dir/target-aarch64_cortex-a53_musl/pkginfo/python.default.install
echo "python-light" >> /home/nic/openwrt/staging_dir/target-aarch64_cortex-a53_musl/pkginfo/python.default.install
echo "python" >> /home/nic/openwrt/staging_dir/target-aarch64_cortex-a53_musl/pkginfo/python.default.install
make[2]: Leaving directory '/home/nic/openwrt/feeds/packages/lang/python/python'
time: package/feeds/packages/python/compile#0.17#0.05#0.20
make[2]: Entering directory '/home/nic/openwrt/package/python2-paho-mqtt'
make[2]: Nothing to be done for 'compile'.
make[2]: Leaving directory '/home/nic/openwrt/package/python2-paho-mqtt'
time: package/python2-paho-mqtt/python/compile#0.07#0.00#0.07
make[1]: Leaving directory '/home/nic/openwrt'
So whats missing or wrong?
Best regards,
Nic
One thing to consider is that Python2 is EOL upstream, and OpenWrt is following that lead.
make[2]: Nothing to be done for 'compile'.
puzzles me.
The build system is pretty magical, except when it isn't
You might try
$(error $(call PyPackage,python2-paho-mqtt))
to see what is being evaluated as make rules.
Change the lines:
#$(eval $(call PyPackage,python2-paho-mqtt))
$(error $(call PyPackage,python2-paho-mqtt))
That returns currently:
make package/python2-paho-mqtt/compile V=sc
make[1]: Entering directory '/home/nic/openwrt'
make[2]: Entering directory '/home/nic/openwrt/package/python2-paho-mqtt'
Makefile:38: *** . Stop.
make[2]: Leaving directory '/home/nic/openwrt/package/python2-paho-mqtt'
time: package/python2-paho-mqtt/compile#0.01#0.00#0.03
package/Makefile:107: recipe for target 'package/python2-paho-mqtt/compile' failed
make[1]: *** [package/python2-paho-mqtt/compile] Error 2
make[1]: Leaving directory '/home/nic/openwrt'
/home/nic/openwrt/include/toplevel.mk:216: recipe for target 'package/python2-paho-mqtt/compile' failed
make: *** [package/python2-paho-mqtt/compile] Error 2
Just hit me, have you tried
make package/python2-paho-mqtt/install
Hmmm, I had expected that to dump the resulting string about to be evaluated then exit
Maybe...
$(info ===== PyPackage =====)
$(info $(call PyPackage,python2-paho-mqtt))
$(info ===== BuildPackage =====)
$(info $(call BuildPackage,python2-paho-mqtt))
$(info ===== Done =====)
$(eval $(call PyPackage,python2-paho-mqtt))
$(eval $(call BuildPackage,python2-paho-mqtt))
?What do you mean, I have to call
make package/python2-paho-mqtt/install after or before
make package/python2-paho-mqtt/compile
Normally I start with
make package/python2-paho-mqtt/clean then
make package/python2-paho-mqtt/compile
Based on the template, my makefile has no compile or install target:
...
include $(INCLUDE_DIR)/package.mk
#include ../python-package.mk
$(call include_mk, python-package.mk)
define Package/python2-paho-mqtt
SECTION:=lang
CATEGORY:=Languages
SUBMENU:=Python
TITLE:=python2-paho-mqtt
URL:=http://eclipse.org/paho
DEPENDS:=+python-light
VARIANT:=python
endef
define Package/python2-paho-mqtt/description
MQTT version 3.1/3.1.1 client class
endef
#$(eval $(call PyPackage,python2-paho-mqtt))
$(error $(call PyPackage,python2-paho-mqtt))
$(eval $(call BuildPackage,python2-paho-mqtt))
compile may do nothing, correctly
install may be needed to actually create the binaries as well as package them
Ok, but why has the template (/openwrt/feeds/packages/lang/python/python-paho-mqtt from the official feeds) then no install target:
include $(TOPDIR)/rules.mk
PKG_NAME:=python-paho-mqtt
PKG_VERSION:=1.4.0
PKG_RELEASE:=1
PKG_LICENSE:=Eclipse Public License v1.0 / Eclipse Distribution License v1.0
PKG_MAINTAINER:=Josef Schlehofer <josef.schlehofer@nic.cz>, Alexandru Ardelean <ardeleanalex@gmail.com>
PKG_SOURCE:=paho-mqtt-$(PKG_VERSION).tar.gz
PKG_SOURCE_URL:=https://files.pythonhosted.org/packages/25/63/db25e62979c2a716a74950c9ed658dce431b5cb01fde29eb6cba9489a904
PKG_HASH:=e440a052b46d222e184be3be38676378722072fcd4dfd2c8f509fb861a7b0b79
PKG_BUILD_DIR:=$(BUILD_DIR)/paho-mqtt-$(PKG_VERSION)
include $(INCLUDE_DIR)/package.mk
include ../python3-package.mk
define Package/python3-paho-mqtt
SECTION:=lang
CATEGORY:=Languages
SUBMENU:=Python
TITLE:=python3-paho-mqtt
URL:=http://eclipse.org/paho
DEPENDS:=+python3-light
VARIANT:=python3
endef
define Package/python3-paho-mqtt/description
MQTT version 3.1/3.1.1 client class
endef
$(eval $(call Py3Package,python3-paho-mqtt))
$(eval $(call BuildPackage,python3-paho-mqtt))
$(eval $(call BuildPackage,python3-paho-mqtt-src))
I do not have any idea how a install target could looks like?
PS: It was your idea to grab a python3 template 
Changed compile and install target:
...
define Build/Compile
$(call Build/Compile/PyMod,,install --prefix=/usr --root=$(PKG_INSTALL_DIR))
endef
define Package/python2-paho-mqtt/install
$(INSTALL_DIR) $(1)$(PYTHON_PKG_DIR)
$(CP) \
$(PKG_INSTALL_DIR)$(PYTHON_PKG_DIR)/* \
$(1)$(PYTHON3_PKG_DIR)
endef
#$(eval $(call PyPackage,python2-paho-mqtt))
#$(error $(call PyPackage,python2-paho-mqtt))
$(eval $(call BuildPackage,python2-paho-mqtt))
Compile it by
make package/python2-paho-mqtt/compile V=sc
..
make[2]: Entering directory '/home/nic/openwrt/package/python2-paho-mqtt'
touch /home/nic/openwrt/build_dir/target-aarch64_cortex-a53_musl/paho-mqtt-1.4.0/.prepared_7ec9b64cd1939e6f2b7f11b86f581ff7_6664517399ebbbc92a37c5bb081b5c53_check
. /home/nic/openwrt/include/shell.sh; gzip -dc /home/nic/openwrt/dl/paho-mqtt-1.4.0.tar.gz | tar -C /home/nic/openwrt/build_dir/target-aarch64_cortex-a53_musl/paho-mqtt-1.4.0/.. -xf -
[ ! -d ./src/ ] || cp -fpR ./src/. /home/nic/openwrt/build_dir/target-aarch64_cortex-a53_musl/paho-mqtt-1.4.0
touch /home/nic/openwrt/build_dir/target-aarch64_cortex-a53_musl/paho-mqtt-1.4.0/.prepared_7ec9b64cd1939e6f2b7f11b86f581ff7_6664517399ebbbc92a37c5bb081b5c53
rm -f /home/nic/openwrt/build_dir/target-aarch64_cortex-a53_musl/paho-mqtt-1.4.0/.configured_*
rm -f /home/nic/openwrt/staging_dir/target-aarch64_cortex-a53_musl/stamp/.python2-paho-mqtt.python_installed
(cd /home/nic/openwrt/build_dir/target-aarch64_cortex-a53_musl/paho-mqtt-1.4.0/./; if [ -x ./configure ]; then find /home/nic/openwrt/build_dir/target-aarch64_cortex-a53_musl/paho-mqtt-1.4.0/ -name config.guess | xargs -r chmod u+w; find /home/nic/openwrt/build_dir/target-aarch64_cortex-a53_musl/paho-mqtt-1.4.0/ -name config.guess | xargs -r -n1 cp --remove-destination /home/nic/openwrt/scripts/config.guess; find /home/nic/openwrt/build_dir/target-aarch64_cortex-a53_musl/paho-mqtt-1.4.0/ -name config.sub | xargs -r chmod u+w; find /home/nic/openwrt/build_dir/target-aarch64_cortex-a53_musl/paho-mqtt-1.4.0/ -name config.sub | xargs -r -n1 cp --remove-destination /home/nic/openwrt/scripts/config.sub; AR="aarch64-openwrt-linux-musl-gcc-ar" AS="aarch64-openwrt-linux-musl-gcc -c -Os -pipe -mcpu=cortex-a53 -fno-caller-saves -fno-plt -fhonour-copts -Wno-error=unused-but-set-variable -Wno-error=unused-result -iremap/home/nic/openwrt/build_dir/target-aarch64_cortex-a53_musl/paho-mqtt-1.4.0:paho-mqtt-1.4.0 -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro" LD=aarch64-openwrt-linux-musl-ld NM="aarch64-openwrt-linux-musl-gcc-nm" CC="aarch64-openwrt-linux-musl-gcc" GCC="aarch64-openwrt-linux-musl-gcc" CXX="aarch64-openwrt-linux-musl-g++" RANLIB="aarch64-openwrt-linux-musl-gcc-ranlib" STRIP=aarch64-openwrt-linux-musl-strip OBJCOPY=aarch64-openwrt-linux-musl-objcopy OBJDUMP=aarch64-openwrt-linux-musl-objdump SIZE=aarch64-openwrt-linux-musl-size CFLAGS="-Os -pipe -mcpu=cortex-a53 -fno-caller-saves -fno-plt -fhonour-copts -Wno-error=unused-but-set-variable -Wno-error=unused-result -iremap/home/nic/openwrt/build_dir/target-aarch64_cortex-a53_musl/paho-mqtt-1.4.0:paho-mqtt-1.4.0 -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro " CXXFLAGS="-Os -pipe -mcpu=cortex-a53 -fno-caller-saves -fno-plt -fhonour-copts -Wno-error=unused-but-set-variable -Wno-error=unused-result -iremap/home/nic/openwrt/build_dir/target-aarch64_cortex-a53_musl/paho-mqtt-1.4.0:paho-mqtt-1.4.0 -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro " CPPFLAGS="-I/home/nic/openwrt/staging_dir/target-aarch64_cortex-a53_musl/usr/include -I/home/nic/openwrt/staging_dir/target-aarch64_cortex-a53_musl/include -I/home/nic/openwrt/staging_dir/toolchain-aarch64_cortex-a53_gcc-7.4.0_musl/usr/include -I/home/nic/openwrt/staging_dir/toolchain-aarch64_cortex-a53_gcc-7.4.0_musl/include/fortify -I/home/nic/openwrt/staging_dir/toolchain-aarch64_cortex-a53_gcc-7.4.0_musl/include " LDFLAGS="-L/home/nic/openwrt/staging_dir/target-aarch64_cortex-a53_musl/usr/lib -L/home/nic/openwrt/staging_dir/target-aarch64_cortex-a53_musl/lib -L/home/nic/openwrt/staging_dir/toolchain-aarch64_cortex-a53_gcc-7.4.0_musl/usr/lib -L/home/nic/openwrt/staging_dir/toolchain-aarch64_cortex-a53_gcc-7.4.0_musl/lib -znow -zrelro " ./configure --target=aarch64-openwrt-linux --host=aarch64-openwrt-linux --build=x86_64-pc-linux-gnu --program-prefix="" --program-suffix="" --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --libexecdir=/usr/lib --sysconfdir=/etc --datadir=/usr/share --localstatedir=/var --mandir=/usr/man --infodir=/usr/info --disable-nls ; fi; )
touch /home/nic/openwrt/build_dir/target-aarch64_cortex-a53_musl/paho-mqtt-1.4.0/.configured_68b329da9893e34099c7d8ad5cb9c940
rm -f /home/nic/openwrt/build_dir/target-aarch64_cortex-a53_musl/paho-mqtt-1.4.0/.built
touch /home/nic/openwrt/build_dir/target-aarch64_cortex-a53_musl/paho-mqtt-1.4.0/.built_check
touch /home/nic/openwrt/build_dir/target-aarch64_cortex-a53_musl/paho-mqtt-1.4.0/.built
rm -rf /home/nic/openwrt/build_dir/target-aarch64_cortex-a53_musl/paho-mqtt-1.4.0/.pkgdir/python2-paho-mqtt.installed /home/nic/openwrt/build_dir/target-aarch64_cortex-a53_musl/paho-mqtt-1.4.0/.pkgdir/python2-paho-mqtt
mkdir -p /home/nic/openwrt/build_dir/target-aarch64_cortex-a53_musl/paho-mqtt-1.4.0/.pkgdir/python2-paho-mqtt
install -d -m0755 /home/nic/openwrt/build_dir/target-aarch64_cortex-a53_musl/paho-mqtt-1.4.0/.pkgdir/python2-paho-mqtt
cp -fpR /home/nic/openwrt/build_dir/target-aarch64_cortex-a53_musl/paho-mqtt-1.4.0/ipkg-install/* /home/nic/openwrt/build_dir/target-aarch64_cortex-a53_musl/paho-mqtt-1.4.0/.pkgdir/python2-paho-mqtt
cp: cannot stat '/home/nic/openwrt/build_dir/target-aarch64_cortex-a53_musl/paho-mqtt-1.4.0/ipkg-install/*': No such file or directory
Makefile:50: recipe for target '/home/nic/openwrt/build_dir/target-aarch64_cortex-a53_musl/paho-mqtt-1.4.0/.pkgdir/python2-paho-mqtt.installed' failed
make[2]: *** [/home/nic/openwrt/build_dir/target-aarch64_cortex-a53_musl/paho-mqtt-1.4.0/.pkgdir/python2-paho-mqtt.installed] Error 1
make[2]: Leaving directory '/home/nic/openwrt/package/python2-paho-mqtt'
time: package/python2-paho-mqtt/python/compile#0.09#0.01#0.10
package/Makefile:107: recipe for target 'package/python2-paho-mqtt/compile' failed
make[1]: *** [package/python2-paho-mqtt/compile] Error 2
make[1]: Leaving directory '/home/nic/openwrt'
/home/nic/openwrt/include/toplevel.mk:216: recipe for target 'package/python2-paho-mqtt/compile' failed
make: *** [package/python2-paho-mqtt/compile] Error 2
My build system is a Debian based x86_64 arch, but the build target is Pi3 (aarch64_cortex-a53_musl).
Is there any other settings needed between host and target system?
You still need the $(eval $(call PyPackage,python2-paho-mqtt)), I think.
My comment was that the compile step may be effectively a no-op. It may be the install step that actually does something.
Try
make package/python2-paho-mqtt/install
not
make package/python2-paho-mqtt/compile
make package/python2-paho-mqtt/install V=sc
make[1]: Entering directory '/home/nic/openwrt'
make[1]: *** No rule to make target 'package/python2-paho-mqtt/install'. Stop.
make[1]: Leaving directory '/home/nic/openwrt'
/home/nic/openwrt/include/toplevel.mk:216: recipe for target 'package/python2-paho-mqtt/install' failed
make: *** [package/python2-paho-mqtt/install] Error 2
I think the last 2 lines are the same as in my previous log.
I'm out of ideas, save for trying to find a legacy makefile for the Python 2 version and hope that it still works.
Edit: I just checked and the Python 3 version builds as expected
Thanks for y support.
That's not a direct answer, but when I was faced with the same problem, I've included the python-pip in the image and created the uci-defaults file which installed required pips.
@stangri,
I will check if I have incl. python-pip too.
But what do y mean by "created the uci-defaults file" where is that located?
Aha, ok found it: https://openwrt.org/docs/guide-user/base-system/uci#uci_defaults
I didnt know it, very interesting, thank you.
@jeff
Yes, compiling the python3 based package "python-paho-mqtt" (since 19.07) works on my build system fine, too. Thanks for the hint. Otherwise it makes no sense to focus on python2 stuff anymore.