How to include a Python (pip) module in OpenWrt image?

Hi,
I am building customized image for the RaspberryPi 3, but I have no idea how to include e.g. the Python module paho-mqtt (https://pypi.org/project/paho-mqtt/) thats not listed in menuconfig.
Exists any further description or can somebody give some hints?

Thanks,
Nic

1 Like

Very good question. If someone can help us with this topic I would be very grateful.
Pawel

Create a package for it and include in the build -- I'd use an existing Python 3 package Makefile as a template.

Thanks Jeff,
would that an example how to include a python module:

I tried this way, but as I remember it didnt build anything.

Nic

1 Like

Same here. Build is complete but the libraries are not there.
I created a repo with a link to Virtualbox machine image that compiles clean image.
I was struggling to build my first image and decided to share it:

Ok, I found in 19.07 branch the package "/openwrt/feeds/packages/lang/python/python-paho-mqtt", it was already added but only for python3. If I take this one as a template for example, where do I have to put it?
Well, I copied it to /openwrt/package/python2-paho-mqtt changed the Makefile slightly according to python2:

include $(TOPDIR)/rules.mk

PKG_NAME:=python2-paho-mqtt
PKG_VERSION:=1.4.0
PKG_RELEASE:=1
PKG_LICENSE:=Eclipse Public License v1.0 / Eclipse Distribution License v1.0
PKG_MAINTAINER:=Josef Schlehofer <josef.schlehofer@nic.cz>, Alexandru Ardelean <ardeleanalex@gmail.com>

PKG_SOURCE:=paho-mqtt-$(PKG_VERSION).tar.gz
PKG_SOURCE_URL:=https://files.pythonhosted.org/packages/25/63/db25e62979c2a716a74950c9ed658dce431b5cb01fde29eb6cba9489a904
PKG_HASH:=e440a052b46d222e184be3be38676378722072fcd4dfd2c8f509fb861a7b0b79
PKG_BUILD_DIR:=$(BUILD_DIR)/paho-mqtt-$(PKG_VERSION)
PKG_BUILD_DEPENDS:=python

include $(INCLUDE_DIR)/package.mk
#include ../python-package.mk
$(call include_mk, python-package.mk)

define Package/python2-paho-mqtt
  SECTION:=lang
  CATEGORY:=Languages
  SUBMENU:=Python
  TITLE:=python2-paho-mqtt
  URL:=http://eclipse.org/paho
  DEPENDS:=+python-light
  VARIANT:=python
endef

define Package/python2-paho-mqtt/description
  MQTT version 3.1/3.1.1 client class
endef

$(eval $(call PyPackage,python2-paho-mqtt))
$(eval $(call BuildPackage,python2-paho-mqtt))

Found it in "make menuconfig" and selected it. No check errors:

$ make package/python2-paho-mqtt/check
 make[1] package/python2-paho-mqtt/check
 make[2] -C package/python2-paho-mqtt check

It ends without building the package (Nothing to be done for 'compile'.):

$ make package/python2-paho-mqtt/compile V=sc
...
echo "python-base" >> /home/nic/openwrt/staging_dir/target-aarch64_cortex-a53_musl/pkginfo/python.default.install
echo "python-light" >> /home/nic/openwrt/staging_dir/target-aarch64_cortex-a53_musl/pkginfo/python.default.install
echo "python" >> /home/nic/openwrt/staging_dir/target-aarch64_cortex-a53_musl/pkginfo/python.default.install
make[2]: Leaving directory '/home/nic/openwrt/feeds/packages/lang/python/python'
time: package/feeds/packages/python/compile#0.17#0.05#0.20
make[2]: Entering directory '/home/nic/openwrt/package/python2-paho-mqtt'
make[2]: Nothing to be done for 'compile'.
make[2]: Leaving directory '/home/nic/openwrt/package/python2-paho-mqtt'
time: package/python2-paho-mqtt/python/compile#0.07#0.00#0.07
make[1]: Leaving directory '/home/nic/openwrt'

So whats missing or wrong?
Best regards,
Nic

One thing to consider is that Python2 is EOL upstream, and OpenWrt is following that lead.

make[2]: Nothing to be done for 'compile'.

puzzles me.

The build system is pretty magical, except when it isn't

You might try

$(error $(call PyPackage,python2-paho-mqtt))

to see what is being evaluated as make rules.

Change the lines:

#$(eval $(call PyPackage,python2-paho-mqtt))
$(error $(call PyPackage,python2-paho-mqtt))

That returns currently:

make package/python2-paho-mqtt/compile V=sc
make[1]: Entering directory '/home/nic/openwrt'
make[2]: Entering directory '/home/nic/openwrt/package/python2-paho-mqtt'
Makefile:38: *** .  Stop.
make[2]: Leaving directory '/home/nic/openwrt/package/python2-paho-mqtt'
time: package/python2-paho-mqtt/compile#0.01#0.00#0.03
package/Makefile:107: recipe for target 'package/python2-paho-mqtt/compile' failed
make[1]: *** [package/python2-paho-mqtt/compile] Error 2
make[1]: Leaving directory '/home/nic/openwrt'
/home/nic/openwrt/include/toplevel.mk:216: recipe for target 'package/python2-paho-mqtt/compile' failed
make: *** [package/python2-paho-mqtt/compile] Error 2

Just hit me, have you tried

make package/python2-paho-mqtt/install

Hmmm, I had expected that to dump the resulting string about to be evaluated then exit

https://www.gnu.org/software/make/manual/html_node/Make-Control-Functions.html#index-warnings_002c-printing

Maybe...

$(info ===== PyPackage =====)
$(info $(call PyPackage,python2-paho-mqtt))
$(info ===== BuildPackage =====)
$(info $(call BuildPackage,python2-paho-mqtt))
$(info ===== Done =====)
$(eval $(call PyPackage,python2-paho-mqtt))
$(eval $(call BuildPackage,python2-paho-mqtt))

?What do you mean, I have to call
make package/python2-paho-mqtt/install after or before
make package/python2-paho-mqtt/compile

Normally I start with
make package/python2-paho-mqtt/clean then
make package/python2-paho-mqtt/compile

Based on the template, my makefile has no compile or install target:

...
include $(INCLUDE_DIR)/package.mk
#include ../python-package.mk
$(call include_mk, python-package.mk)

define Package/python2-paho-mqtt
  SECTION:=lang
  CATEGORY:=Languages
  SUBMENU:=Python
  TITLE:=python2-paho-mqtt
  URL:=http://eclipse.org/paho
  DEPENDS:=+python-light
  VARIANT:=python
endef

define Package/python2-paho-mqtt/description
  MQTT version 3.1/3.1.1 client class
endef

#$(eval $(call PyPackage,python2-paho-mqtt))
$(error $(call PyPackage,python2-paho-mqtt))
$(eval $(call BuildPackage,python2-paho-mqtt))

compile may do nothing, correctly

install may be needed to actually create the binaries as well as package them

Ok, but why has the template (/openwrt/feeds/packages/lang/python/python-paho-mqtt from the official feeds) then no install target:

include $(TOPDIR)/rules.mk

PKG_NAME:=python-paho-mqtt
PKG_VERSION:=1.4.0
PKG_RELEASE:=1
PKG_LICENSE:=Eclipse Public License v1.0 / Eclipse Distribution License v1.0
PKG_MAINTAINER:=Josef Schlehofer <josef.schlehofer@nic.cz>, Alexandru Ardelean <ardeleanalex@gmail.com>

PKG_SOURCE:=paho-mqtt-$(PKG_VERSION).tar.gz
PKG_SOURCE_URL:=https://files.pythonhosted.org/packages/25/63/db25e62979c2a716a74950c9ed658dce431b5cb01fde29eb6cba9489a904
PKG_HASH:=e440a052b46d222e184be3be38676378722072fcd4dfd2c8f509fb861a7b0b79
PKG_BUILD_DIR:=$(BUILD_DIR)/paho-mqtt-$(PKG_VERSION)

include $(INCLUDE_DIR)/package.mk
include ../python3-package.mk

define Package/python3-paho-mqtt
  SECTION:=lang
  CATEGORY:=Languages
  SUBMENU:=Python
  TITLE:=python3-paho-mqtt
  URL:=http://eclipse.org/paho
  DEPENDS:=+python3-light
  VARIANT:=python3
endef

define Package/python3-paho-mqtt/description
  MQTT version 3.1/3.1.1 client class
endef

$(eval $(call Py3Package,python3-paho-mqtt))
$(eval $(call BuildPackage,python3-paho-mqtt))
$(eval $(call BuildPackage,python3-paho-mqtt-src))

I do not have any idea how a install target could looks like?

PS: It was your idea to grab a python3 template :wink:

Changed compile and install target:

...
define Build/Compile
        $(call Build/Compile/PyMod,,install --prefix=/usr --root=$(PKG_INSTALL_DIR))
endef

define Package/python2-paho-mqtt/install
        $(INSTALL_DIR) $(1)$(PYTHON_PKG_DIR)
        $(CP) \
            $(PKG_INSTALL_DIR)$(PYTHON_PKG_DIR)/* \
            $(1)$(PYTHON3_PKG_DIR)
endef

#$(eval $(call PyPackage,python2-paho-mqtt))
#$(error $(call PyPackage,python2-paho-mqtt))
$(eval $(call BuildPackage,python2-paho-mqtt))

Compile it by

make package/python2-paho-mqtt/compile V=sc
..
make[2]: Entering directory '/home/nic/openwrt/package/python2-paho-mqtt'
touch /home/nic/openwrt/build_dir/target-aarch64_cortex-a53_musl/paho-mqtt-1.4.0/.prepared_7ec9b64cd1939e6f2b7f11b86f581ff7_6664517399ebbbc92a37c5bb081b5c53_check
. /home/nic/openwrt/include/shell.sh; gzip -dc /home/nic/openwrt/dl/paho-mqtt-1.4.0.tar.gz | tar -C /home/nic/openwrt/build_dir/target-aarch64_cortex-a53_musl/paho-mqtt-1.4.0/.. -xf -
[ ! -d ./src/ ] || cp -fpR ./src/. /home/nic/openwrt/build_dir/target-aarch64_cortex-a53_musl/paho-mqtt-1.4.0
touch /home/nic/openwrt/build_dir/target-aarch64_cortex-a53_musl/paho-mqtt-1.4.0/.prepared_7ec9b64cd1939e6f2b7f11b86f581ff7_6664517399ebbbc92a37c5bb081b5c53
rm -f /home/nic/openwrt/build_dir/target-aarch64_cortex-a53_musl/paho-mqtt-1.4.0/.configured_*
rm -f /home/nic/openwrt/staging_dir/target-aarch64_cortex-a53_musl/stamp/.python2-paho-mqtt.python_installed
(cd /home/nic/openwrt/build_dir/target-aarch64_cortex-a53_musl/paho-mqtt-1.4.0/./; if [ -x ./configure ]; then find /home/nic/openwrt/build_dir/target-aarch64_cortex-a53_musl/paho-mqtt-1.4.0/ -name config.guess | xargs -r chmod u+w; find /home/nic/openwrt/build_dir/target-aarch64_cortex-a53_musl/paho-mqtt-1.4.0/ -name config.guess | xargs -r -n1 cp --remove-destination /home/nic/openwrt/scripts/config.guess; find /home/nic/openwrt/build_dir/target-aarch64_cortex-a53_musl/paho-mqtt-1.4.0/ -name config.sub | xargs -r chmod u+w; find /home/nic/openwrt/build_dir/target-aarch64_cortex-a53_musl/paho-mqtt-1.4.0/ -name config.sub | xargs -r -n1 cp --remove-destination /home/nic/openwrt/scripts/config.sub; AR="aarch64-openwrt-linux-musl-gcc-ar" AS="aarch64-openwrt-linux-musl-gcc -c -Os -pipe -mcpu=cortex-a53 -fno-caller-saves -fno-plt -fhonour-copts -Wno-error=unused-but-set-variable -Wno-error=unused-result -iremap/home/nic/openwrt/build_dir/target-aarch64_cortex-a53_musl/paho-mqtt-1.4.0:paho-mqtt-1.4.0 -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro" LD=aarch64-openwrt-linux-musl-ld NM="aarch64-openwrt-linux-musl-gcc-nm" CC="aarch64-openwrt-linux-musl-gcc" GCC="aarch64-openwrt-linux-musl-gcc" CXX="aarch64-openwrt-linux-musl-g++" RANLIB="aarch64-openwrt-linux-musl-gcc-ranlib" STRIP=aarch64-openwrt-linux-musl-strip OBJCOPY=aarch64-openwrt-linux-musl-objcopy OBJDUMP=aarch64-openwrt-linux-musl-objdump SIZE=aarch64-openwrt-linux-musl-size CFLAGS="-Os -pipe -mcpu=cortex-a53 -fno-caller-saves -fno-plt -fhonour-copts -Wno-error=unused-but-set-variable -Wno-error=unused-result -iremap/home/nic/openwrt/build_dir/target-aarch64_cortex-a53_musl/paho-mqtt-1.4.0:paho-mqtt-1.4.0 -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro " CXXFLAGS="-Os -pipe -mcpu=cortex-a53 -fno-caller-saves -fno-plt -fhonour-copts -Wno-error=unused-but-set-variable -Wno-error=unused-result -iremap/home/nic/openwrt/build_dir/target-aarch64_cortex-a53_musl/paho-mqtt-1.4.0:paho-mqtt-1.4.0 -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro " CPPFLAGS="-I/home/nic/openwrt/staging_dir/target-aarch64_cortex-a53_musl/usr/include -I/home/nic/openwrt/staging_dir/target-aarch64_cortex-a53_musl/include -I/home/nic/openwrt/staging_dir/toolchain-aarch64_cortex-a53_gcc-7.4.0_musl/usr/include -I/home/nic/openwrt/staging_dir/toolchain-aarch64_cortex-a53_gcc-7.4.0_musl/include/fortify -I/home/nic/openwrt/staging_dir/toolchain-aarch64_cortex-a53_gcc-7.4.0_musl/include " LDFLAGS="-L/home/nic/openwrt/staging_dir/target-aarch64_cortex-a53_musl/usr/lib -L/home/nic/openwrt/staging_dir/target-aarch64_cortex-a53_musl/lib -L/home/nic/openwrt/staging_dir/toolchain-aarch64_cortex-a53_gcc-7.4.0_musl/usr/lib -L/home/nic/openwrt/staging_dir/toolchain-aarch64_cortex-a53_gcc-7.4.0_musl/lib -znow -zrelro "   ./configure --target=aarch64-openwrt-linux --host=aarch64-openwrt-linux --build=x86_64-pc-linux-gnu --program-prefix="" --program-suffix="" --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --libexecdir=/usr/lib --sysconfdir=/etc --datadir=/usr/share --localstatedir=/var --mandir=/usr/man --infodir=/usr/info --disable-nls  ; fi; )
touch /home/nic/openwrt/build_dir/target-aarch64_cortex-a53_musl/paho-mqtt-1.4.0/.configured_68b329da9893e34099c7d8ad5cb9c940
rm -f /home/nic/openwrt/build_dir/target-aarch64_cortex-a53_musl/paho-mqtt-1.4.0/.built
touch /home/nic/openwrt/build_dir/target-aarch64_cortex-a53_musl/paho-mqtt-1.4.0/.built_check
touch /home/nic/openwrt/build_dir/target-aarch64_cortex-a53_musl/paho-mqtt-1.4.0/.built
rm -rf /home/nic/openwrt/build_dir/target-aarch64_cortex-a53_musl/paho-mqtt-1.4.0/.pkgdir/python2-paho-mqtt.installed /home/nic/openwrt/build_dir/target-aarch64_cortex-a53_musl/paho-mqtt-1.4.0/.pkgdir/python2-paho-mqtt
mkdir -p /home/nic/openwrt/build_dir/target-aarch64_cortex-a53_musl/paho-mqtt-1.4.0/.pkgdir/python2-paho-mqtt
install -d -m0755 /home/nic/openwrt/build_dir/target-aarch64_cortex-a53_musl/paho-mqtt-1.4.0/.pkgdir/python2-paho-mqtt
cp -fpR /home/nic/openwrt/build_dir/target-aarch64_cortex-a53_musl/paho-mqtt-1.4.0/ipkg-install/* /home/nic/openwrt/build_dir/target-aarch64_cortex-a53_musl/paho-mqtt-1.4.0/.pkgdir/python2-paho-mqtt
cp: cannot stat '/home/nic/openwrt/build_dir/target-aarch64_cortex-a53_musl/paho-mqtt-1.4.0/ipkg-install/*': No such file or directory
Makefile:50: recipe for target '/home/nic/openwrt/build_dir/target-aarch64_cortex-a53_musl/paho-mqtt-1.4.0/.pkgdir/python2-paho-mqtt.installed' failed
make[2]: *** [/home/nic/openwrt/build_dir/target-aarch64_cortex-a53_musl/paho-mqtt-1.4.0/.pkgdir/python2-paho-mqtt.installed] Error 1
make[2]: Leaving directory '/home/nic/openwrt/package/python2-paho-mqtt'
time: package/python2-paho-mqtt/python/compile#0.09#0.01#0.10
package/Makefile:107: recipe for target 'package/python2-paho-mqtt/compile' failed
make[1]: *** [package/python2-paho-mqtt/compile] Error 2
make[1]: Leaving directory '/home/nic/openwrt'
/home/nic/openwrt/include/toplevel.mk:216: recipe for target 'package/python2-paho-mqtt/compile' failed
make: *** [package/python2-paho-mqtt/compile] Error 2

My build system is a Debian based x86_64 arch, but the build target is Pi3 (aarch64_cortex-a53_musl).
Is there any other settings needed between host and target system?

You still need the $(eval $(call PyPackage,python2-paho-mqtt)), I think.

My comment was that the compile step may be effectively a no-op. It may be the install step that actually does something.

Try

make package/python2-paho-mqtt/install

not

make package/python2-paho-mqtt/compile
make package/python2-paho-mqtt/install V=sc

make[1]: Entering directory '/home/nic/openwrt'
make[1]: *** No rule to make target 'package/python2-paho-mqtt/install'.  Stop.
make[1]: Leaving directory '/home/nic/openwrt'
/home/nic/openwrt/include/toplevel.mk:216: recipe for target 'package/python2-paho-mqtt/install' failed
make: *** [package/python2-paho-mqtt/install] Error 2

I think the last 2 lines are the same as in my previous log.

I'm out of ideas, save for trying to find a legacy makefile for the Python 2 version and hope that it still works.

Edit: I just checked and the Python 3 version builds as expected

Thanks for y support.

That's not a direct answer, but when I was faced with the same problem, I've included the python-pip in the image and created the uci-defaults file which installed required pips.

@stangri,
I will check if I have incl. python-pip too.
But what do y mean by "created the uci-defaults file" where is that located?

Aha, ok found it: https://openwrt.org/docs/guide-user/base-system/uci#uci_defaults
I didnt know it, very interesting, thank you.

@jeff
Yes, compiling the python3 based package "python-paho-mqtt" (since 19.07) works on my build system fine, too. Thanks for the hint. Otherwise it makes no sense to focus on python2 stuff anymore.