Port Python library built on Ubuntu to OpenWrt

Hi,

I built a python package from the following repository on Ubuntu v.18. The real target for this was OpenWrt however OpenWrt only supports GCC 7.4 (mentioned here), and the build file has commands which are only available in GCC 8+, so I couldn't build it on OpenWrt and did it on Ubuntu. So, I think the easiest way would be to package the library I built on Ubuntu for PIP and upload it to PyPi to make it OS independent. How can this be done? Or is there a better way?

https://github.com/BPI-SINOVOIP/RPi.GPIO

The library I built is installed in the Ubuntu directory: /usr/local/lib/python3.8/dist-packages/RPi/*

Essentially, the goal is to "port" this library I built on Ubuntu to OpenWrt using PIP as an intermediary.

If you need gcc, it contains target specific object code, which in turn has (almost always) dependencies to the target runtime environment - starting with the elephant in the room, the libc (but usually quite a bit beyond 'just' that) and the libgcc runtime. While you can cross-compile on Ubuntu (using Ubuntu's toolchain), the result will inevitably depend on this glibc/ gcc-10-base based runtime environment (yes, static linking exists, but not really using glibc) - while OpenWrt is based on musl and (in master) gcc 8.4.0.

In order to submit a package to OpenWrt, there is no way around to build it from source with OpenWrt's buildsystem and on the buildbot infrastructure - taking a binary from somewhere isn't going to fly (neither functionally, because of the reasons laid out above, nor policy wise (there really needs to be a way to build it within OpenWrt, or it's effectively like proprietary code and prone to fail/ unfixable when it matters).

So if you 'just' need gcc >=8, that is available in master and openwrt-21.02 right now - master is likely to be bumped to gcc 10 right after 21.02.0 is out of the door (and gcc 7.5.0, gcc 9.3.0 and gcc 10.3.0 can already be selected in the advanced options for your own builds, although not buildbot images until it's switched centrally).

2 Likes

Hi,

Thank you for explaining, I appreciate your time.

My intention of porting was keeping it within the Python/PIP ecosystem and not submitting it to OpenWrt as a package as this is an obscure board specific library only used for GPIO programming and nothing else so I'm not sure if it even qualifies for submission :slight_smile:

My current OpenWrt image is v21.02.0-rc1, so now after your message I'm rebuilding it with gcc added, if this turns out to be v8+ it will resolve my issue as I'll be able to build this GPIO library natively.

While you may be able to get it into pip (I have no idea about the policies there), it won't actually run on OpenWrt that way <-- unsatisfiable runtime dependencies (libc, libgcc). pip works on a running OpenWrt system, but only for target agnostic python code - as soon as any of your python code (or its dependencies) depends on native target code, it will fail to install/ run (OpenWrt is not self-hosting, the runtime gcc packages are very, very limited (not able to build much more than hello_world.c) and lacks any kind of runtime -dev packages). pip can only be used to build the missing target binaries within OpenWrt cross-compiling/ buildroot environment (check for other example packages doing this, I'm not exactly a python specialist).

2 Likes

It does not qualify for OpenWrt core repository, but there are quite a few specific tools or libraries like that in Community Packages repository, https://github.com/openwrt/packages so if you think it can be useful to others with a Banana Pi or Raspberry Pi or whatever other similar board that is supported by OpenWrt, you can submit the package (and become its maintainer)

That said, the library you linked in the OP seems to be a C library with python bindings, so you probably need to compile it as a OpenWrt package (probably patching the parts that require gcc 8, at least temporarily until OpenWrt eventually switches to Gcc 8) if you want to use it in OpenWrt.

2 Likes

Thank you bobafetthotmail. I have never compiled an OpenWrt package suitable for submission. I'll be happy to take ownership and be a maintainer as this definately can be useful to anyone using BPI boards. I did go through some of the docs but was unable to completely follow along. For eg. packages/Makefile is just one file. The same file has to be used to create any number of packages? Or do we duplicate it for every package that needs to be created. How does it work?

imho... you should wait for a month for any movement in the above...

in the meantime... i'd suggest messing around in an alpine chroot and see how that goes...

1 Like

Hey wulfy23, I did read your post that you were in the same boat as mine on the gcc. Thanks for your input. I will try to patch the gcc v8 commands in the build file and see if it works, else will try again next month.

yes... i went through a similar experience several months ago... trying to add this to the community build...

python is not really my 'forte'... (gave up after several hours... fortunately ... there are not too many users interested in GPiO functionality ~ and there are a few non-python alternatives to most things... which to be honest... I kind of prefer unless it's an experimental~once off project)

I am still a little interested tho' so... if you make any progress... update this thread and I may mess around a little more if I find the time...

fwiw... tested chroot option;

apk add python3 gcc linux-headers musl-dev python3-dev
/usr/bin/python3.8 -m pip install --upgrade pip
pip3 install RPi.GPIO
1 Like

I didn't say everyone is rushing to adopt gcc8, I just said it will come eventually.

Imho for what this library is doing (remapping GPIO and giving Python bindings) trying to hack it a little to remove this gcc features requirement might be doable.

Depends from how related the packages are and if it makes sense to keep them as separate packages or not.

If a group of packages are all compiled by the same source code then they are usually defined in the same makefile

For example, this is the makefile for the Bash shell, and it is a single package https://github.com/openwrt/packages/blob/master/utils/bash/Makefile

This is the makefile of inotify-tools and it defines a couple tool packages and a library package since it's all built from the same source download https://github.com/openwrt/packages/blob/master/utils/inotify-tools/Makefile

This is the makefile of GNU coreutils and is using an automated internal function to auto-generate separate packages for each individual utility https://github.com/openwrt/packages/blob/master/utils/coreutils/Makefile

In general it's a good idea to look at similar package to what you want to add (in the sense of what dependencies and languages are used) and use its makefile as a starting point for your own package.

1 Like