Building a package: is it Ok to have subdirectories in the source tree?

Hi all (again)

So I'm trying to make an OpenWrt package out of my project. The project basically consists of a few main scripts (which go into /usr/bin), some library scripts (which go into /usr/lib), some config files (which go into /etc/geoip-shell) and an init script, which goes into the /etc/init.d directory.

Now the project works on any Linux system, so it has its own install and uninstall scripts. My idea for OpenWrt package is using the existing -install script in a way which generates all the files needed for OpenWrt and puts them in subdirectories, so for example files which go into /usr/bin are put in files/usr/bin. etc. Then another script goes through the resulting tree and makes a nice Makefile out of it. I have already implemented all this, and now the only question - is this allowed?

P.s. I prefer to automate this once so I don't have to manually change the Makefile in future releases, plus the project is made in a modular way, so there are a lot of small files (currently 23), and sometimes I add or remove files. This is why I prefer an automated solution.

The Makefile in the OpenWrt packages repo should be static. You only modify that via new version commits. But how you modify that Makefile in your own local repo is your own business.

One example of pretty similar package is e.g. adblock-fast from @stangri that has all files nicely in the files/ tree plus a normal Makefile that takes care of the file copying at installation.

If you have all files in files/ the installation command is pretty simple, as the copying handles subdirectories, so you can use wildcards.
But note that data files should have different permissions (INSTALL_DATA) than scripts (INSTALL_BIN)

Best might be to place data files in files/data (e.g. files/data/usr/lib) and scripts in files/scripts and use suitable wildcards.

2 Likes

Can you elaborate how this applies to my idea of generating the Makefile? I don't quite follow. Is there a problem to add or remove certain files from the package when updating to a new version?

Can you give an example of how this looks?
Currently my script just does something like this:

        $(INSTALL_DIR) $(1)/usr/bin
        $(INSTALL_BIN) ./files/usr/bin/geoip-shell-fetch.sh $(1)/usr/bin
        $(INSTALL_BIN) ./files/usr/bin/geoip-shell-fw-include.sh $(1)/usr/bin
        $(INSTALL_BIN) ./files/usr/bin/geoip-shell-backup.sh $(1)/usr/bin

etc etc

If you mean user's data then it gets populated dynamically after installation. The dedicated data directory I've currently settled on for OpenWrt is /etc/geoip-shell/data. The Makefile command is

$(INSTALL_DIR) $(1)/etc/geoip-shell/data

There are no files copied to that directory during the installation. During use, my app only stores in it compressed backups of the ip lists, firewall rules and config.
Is the location ok or should I move it to /usr/lib/geoip-shell? Also there seems to be only one command for directories, so what should I do about permissions and what the permissions should be for data files?

Also my app implements its own config management (currently not using UCI for this). The main config file is included in the installation with this command:

$(INSTALL_CONF) ./files/etc/geoip-shell/geoip-shell.conf $(1)/etc/geoip-shell

The content of the config file also gets populated dynamically. It may have some fairly sensitive information (like which ip lists for which countries are selected from which source, which exceptions for trusted ip's the user wants etc). Should I change it to INSTALL_DATA ?

There is another file which gets added to that directory later when fetching ip lists, which is the status file. It basically says for each ip list whether fetch was successful, the date of last fetch and how many elements were fetched. I can generate it with certain permissions or pre-install an empty file if that's important.

You can also put them all in a directory and just use $(CP) to copy it somewhere on the device within the Makefile: https://github.com/openwrt/packages/blob/master/net/pbr/Makefile#L98

3 Likes

Dynamically changing data should never be stored in the flash of a router as it will, sooner than you might expect, lead to failure of the flash chip.
Instead this should be either stored in the /tmp "ramdisk" or if you want it to survive a reboot, on some expendable and readily replaceable external storage like a usb stick.

1 Like

I'm aware of the flash storage wear and tear. For OpenWrt, the default update schedule (which is when backups are created) for my app is once a week to minimize it. Besides that, my app has an option to run without creating backups, then it will re-fetch the ip lists after every reboot (and only use /tmp, as you are suggesting). Config does need to be stored on permanent storage but it is a really small file, and this applies to all config for any app. I suppose adding an option to change backup directory path is also a good idea.

It does not matter how big the file is. Dynamic changes will still wear out the flash.

UCI supports dynamic changes and keeps them in memory, only writing to flash if committed. Even then, continual uci commits will still cause irreparable wear.

1 Like

I would say essential, with the default on /tmp with checks to make sure the /tmp "used space" does not grow too large as it takes memory from the free ram pool and if you take too much it can result in oom failures.

Let's say that I put the effort and implemented support for UCI config. To have the updated config stored, the commit command will come in at some point, right? So the config will still need to be stored on permanent storage. How is this effectively different from writing a text file then? Anyway, my app changes the config only when the user wants to change it, not regularly. With typical use, the config will be set a couple of times in a lifetime.

That is obviously fine for static or rarely changing data.

Dynamic stuff like regular backups is a big problem though.
A mountpoint such as /tmp/data will work well. If the external storage is not mounted it will end up on /tmp
As long as you do suitable checks when writing, you can avoid oom events/crashes.

The mounted storage can be anything, a remote sftp server (via sshfs), even a NAS, but the most obvious would be a usb stick, assuming the router has a usb port....

1 Like

Do you think it's a good idea for OpenWrt to have no permanent backup by default (which implies that after each reboot, ip lists will be re-fetched), and make backup opt-in?

Yes, the flash wear (and also limited flash space in smallest routers) is seen as a problem and avoiding flash writes is a design principle. Please follow that.

1 Like

Thanks. I also asked some questions in response to your comment above, hope you find them.

@hnyman This one has been answered but the other ones still have not.