Need to help to construct a Makefile for a new package

I am following the packages wiki article attempting to create package for vectorscan. My first issue is that the tag to the source code contains a forward slash and I cannot seem to get the syntax correct to allow the build system to get the tarball. Any suggestions?

Tag = vectorscan/5.4.9
And the path to source in the tarball once decompressed is vectorscan-vectorscan-5.4.9/

Here is what I started with:

include $(TOPDIR)/rules.mk

PKG_NAME:=vectorscan
PKG_VERSION:=5.4.9
PKG_RELEASE:=1

PKG_SOURCE:=$(PKG_VERSION).tar.gz
PKG_SOURCE_URL:=https://github.com/VectorCamp/vectorscan//archive/refs/tags/
PKG_HASH:=e61c78f26a9d04ccffab0df1159885c4503fc501172402c57f7357a2126ea3c6

Which results in:

% make package/$i/{clean,compile} package/index V=s
...
CMake Error: The source directory "/scratch/union/build_dir/target-aarch64_cortex-a72_musl/vectorscan-5.4.9" does not appear to contain CMakeLists.txt.
Specify --help for usage, or press the help button on the CMake GUI.
make[2]: *** [Makefile:52: /scratch/union/build_dir/target-aarch64_cortex-a72_musl/vectorscan-5.4.9/.configured_68b329da9893e34099c7d8ad5cb9c940] Error 1
make[2]: Leaving directory '/scratch/union/feeds/packages/utils/vectorscan'
time: package/feeds/packages/vectorscan/compile#0.34#0.04#0.38
    ERROR: package/feeds/packages/vectorscan failed to build.
make[1]: *** [package/Makefile:120: package/feeds/packages/vectorscan/compile] Error 1
make[1]: Leaving directory '/scratch/union'
make: *** [/scratch/union/include/toplevel.mk:231: package/vectorscan/compile] Error 2

Seem to give it away.

1 Like

Yes, I do not know how to modify the build system to have it look in /scratch/union/build_dir/target-aarch64_cortex-a72_musl/vectorscan-vectorscan-5.4.9 where the tarball unpacks the files.

EDIT: Just to test out things, I forked the source and used a tag without a forward slash.

My end game is to build vectorscan which has a requirement of ragel and neither of which are in the current packages repo.

I created a Makefile for ragel here and it builds OK. But when I try to build my Makefile for vectorscan, it is not pulling the ragel dependency. Why?

...
-- Found Python: /scratch/union/staging_dir/host/bin/python3 (found version "3.11.3") found components: Interpreter 
-- Build date: 
CMake Error at CMakeLists.txt:95 (message):
  Ragel state machine compiler not found


-- Configuring incomplete, errors occurred!
make[2]: *** [Makefile:57: /scratch/union/build_dir/target-aarch64_cortex-a72_musl/vectorscan-5.4.9/.configured_68b329da9893e34099c7d8ad5cb9c940] Error 1
make[2]: Leaving directory '/scratch/union/feeds/packages/utils/vectorscan'
time: package/feeds/packages/vectorscan/compile#0.78#0.20#1.81
    ERROR: package/feeds/packages/vectorscan failed to build.
make[1]: *** [package/Makefile:120: package/feeds/packages/vectorscan/compile] Error 1

@neheb ?

If the binary needs to be used and not headers, you need a hostpkg.

Btw, those depends look extremely fishy.

To create a host package does that mean I need to install it not to feeds/packages but to packages?

No that’s unrelated. You need to include host-build.mk. Probably make it host only if it’s not needed on the router.

I believe a host only package will be fine/it is not a dependency for the compiled package. Is the addition of that include $(INCLUDE_DIR)/host-build.mk and $(eval $(call HostBuild)) what is needed to create the host only package?

Uhhh PKG_HOST_ONLY or something like that. There are examples in the packages feed. git feel for it.

Man, this is really kicking my butt. I cannot get this to build as a host package. My latest attempt is here. I get no compilation out. Any thoughts are appreciated.

% make package/ragel/{clean,compile} package/index V=s
Collecting package info: done
make[2]: Entering directory '/scratch/union/scripts/config'
make[2]: 'conf' is up to date.
make[2]: Leaving directory '/scratch/union/scripts/config'
make[1]: Entering directory '/scratch/union'
make[2]: Entering directory '/scratch/union/feeds/packages/devel/ragel'
rm -rf /scratch/union/build_dir/target-aarch64_cortex-a72_musl/ragel-6.10
rm -f /scratch/union/staging_dir/target-aarch64_cortex-a72_musl/stamp/.ragel_installed
rm -f /scratch/union/staging_dir/target-aarch64_cortex-a72_musl/packages/ragel.list
make[2]: Leaving directory '/scratch/union/feeds/packages/devel/ragel'
time: package/feeds/packages/ragel/clean#0.24#0.02#0.26
make[1]: Leaving directory '/scratch/union'
make[2]: Entering directory '/scratch/union/scripts/config'
make[2]: 'conf' is up to date.
make[2]: Leaving directory '/scratch/union/scripts/config'
make[1]: Entering directory '/scratch/union'
make[2]: Entering directory '/scratch/union/feeds/packages/devel/ragel'
make[2]: Nothing to be done for 'compile'.
make[2]: Leaving directory '/scratch/union/feeds/packages/devel/ragel'
time: package/feeds/packages/ragel/compile#0.22#0.03#0.25
make[1]: Leaving directory '/scratch/union'
make[2]: Entering directory '/scratch/union/scripts/config'
make[2]: 'conf' is up to date.
make[2]: Leaving directory '/scratch/union/scripts/config'
make[1]: Entering directory '/scratch/union'
Generating package index...
Generating index for package ./libatomic1_12.3.0-4_aarch64_cortex-a72.ipk
Generating index for package ./libgcc1_12.3.0-4_aarch64_cortex-a72.ipk
Generating index for package ./libpthread_1.2.3-4_aarch64_cortex-a72.ipk
Generating index for package ./librt_1.2.3-4_aarch64_cortex-a72.ipk
Generating index for package ./libstdcpp6_12.3.0-4_aarch64_cortex-a72.ipk
Signing package index...
make[1]: Leaving directory '/scratch/union'

It's still a package so needs to be evaluated as one, the usual $(eval $(call BuildPackage,ragel)) is lacking.

@tmn505 - I added that call but nothing got compiled. See here.

Grateful for any insights you have to get this working. @Ansuel

include $(TOPDIR)/rules.mk

PKG_NAME:=ragel
PKG_VERSION:=6.10
PKG_RELEASE:=1

PKG_SOURCE:=$(PKG_NAME)-$(PKG_VERSION).tar.gz
PKG_SOURCE_URL:=https://www.colm.net/files/ragel/
PKG_HASH:=5f156edb65d20b856d638dd9ee2dfb43285914d9aa2b6ec779dac0270cd56c3f

PKG_MAINTAINER:=<REDACTED>
PKG_LICENSE:=GPL-2.0-only
PKG_LICENSE_FILES:=COPYING

HOST_BUILD_PARALLEL:=1
PKG_HOST_ONLY:=1

include $(INCLUDE_DIR)/host-build.mk
include $(INCLUDE_DIR)/package.mk

define Package/ragel
  SECTION:=devel
  CATEGORY:=Development
  TITLE:=Compiles finite state machines from regular languages into executable code
  URL:=https://www.colm.net/open-source/ragel/
  BUILDONLY:=1
endef

HOST_CONFIGURE_VARS += CXXFLAGS="$(HOST_CXXFLAGS) -std=gnu++98"

$(eval $(call HostBuild))
$(eval $(call BuildPackage,ragel))

To check it:
make package/ragel/host/{clean,compile} -j4 V=s

1 Like

Thank you very much, @1715173329 :heart_eyes: I see now that some of my attempts probably would have worked but I didn't realize that in order to build them, I had to include the literal /host/ in my make step.

Wrong:

make package/ragel/compile package/index V=s

Right:

make package/ragel/host/compile package/index V=s

Now time to move on to creating the package for vectorscan.

EDIT: I got a working one but it has a few issues that need to be worked out. I opened a draft PR. Grateful for any suggestions: https://github.com/openwrt/packages/pull/21140

Hello,

I am the lead developer of Vectorscan and we (VectorCamp) are very interested in officially supporting it for OpenWrt.
I was googling for any interest for Vectorscan in OpenWrt and found this thread so I guess I might as well reply here.

We intend to include it in our CI for Aarch64 only for now. Our main question is if we can do native builds for Aarch64 so that we can run the unit/functional tests or does the development environment assume cross-compilation? We can support at least 2 configurations, stable/development openwrt images. We would appreciate some pointers on which one to aim for.

I have personal experience with OpenWrt, but I have never developed for it.

Thanks

There are basically two approaches to accomplish this:

  • building a single package (and its dependency chain) using the prebuilt SDK
  • or building OpenWrt itself and everything else in there from source

For your final productive integration, you probably want to use the SDK route, but for development and more personal use (as in getting your feet wet), building from source tends to be much easier to get started. On contemporary desktop systems you should be able to build your complete (flashable) firmware image within 10-60 minutes (obviously much less for successive builds), just do your first tries for x86_64 or one of the SBC-like (RPi, sunxi, rockchip) targets (as these are unbrickable), before diving deeper into it.

and taking a look at what's being done for other, ideally roughly similar, packages should get you going with the packaging.

What about building on fast native Aarch64, for example Ampere, Graviton3, etc. These would also enable creating chroots and running any unit or other tests directly. That would be ideal for us. Is there support for native builds like that?

You can use arm64/ aarch as build host, but you always (need to) cross-compile (even on x86 for x86). However, the prebuilt SDK is only provided for x86_64, so you'd need to build from source.