I am working on firmware built via buildroot for OpenWrt 18.06.5. I'm trying to make flashing via sysupgrade as smooth as possible, but I keep running into problems with uci-defaults scripts, which my firmware makes use of.
I've found that running the sysupgrade command will reboot the system, as the docs say it should. Upon reboot, every script from the firmware's /etc/uci-defaults is installed there. Now, I'd expect that a sysupgrade would indeed put those scripts back, so I've written them to be idempotent. But I wouldn't expect them to still be there by the time I log in to check — I'd expect them to run. The part that really confuses me is that if I then do another reboot... they all run successfully and disappear!
Is sysupgrade somehow inhibiting the uci-defaults scripts from running on that first reboot? Or is it a side effect of something else? Do I need to put a reboot uci-defaults script in there to make it do that second reboot that runs all the scripts?
Related question: if a uci-defaults script fails, is it logged anywhere, or do I need to put logger calls in it myself?
No. They should be run quite normally on the boot after sysupograde. uci-defautls scripts are meant e.g. for config migration scripts, so naturally the scipts run after each boot (until they are removed)
There is something else.
One guess: There is something in your scripts that prevents their successful completion. E.g. the file system is not yet ready?
As documented (https://openwrt.org/docs/guide-developer/uci-defaults), the script is deleted if it exited/ended with code 0. If your script breaks and exits with some other error code, the script is left intact to wait for the next try at the next boot.
No. They should be run quite normally on the boot after sysupgrade.
Thanks, this is helpful to know.
One guess: There is something in your scripts that prevents their successful completion. E.g. the file system is not yet ready?
Hmm, this could be a good lead. If I use mtd to do a from-scratch flash, it's fine. If use sysupgrade then none of them run (from 00 to 99), including OpenWrt's own. I wonder if sysupgrade is somehow rebooting too early? Although since sysupgrade (I think) uses mtd at some stage under the hood... I don't that that makes sense.
Is there an overarching log for the part of the boot process that runs the uci-defaults scripts (S10-boot I think)? I can't find anything in logread, but I don't want to have to add logger calls to every script in there.
I put some logging statements in, and presto! The problem disappeared altogether. Thinking that maybe the logging statements had introduced some delay due to eg. buffering, I took them out again to see. Presto! It still works, now in exactly the same state as before. I can only conclude that my device has become sentient and is reading these forums.
I plan to continue testing to catch it, but I do worry that adding my own logging will introduce enough of a delay or dependency to hide it. I'll post back here if I find the problem.
Do look at the order the scripts have in /ROM/etc/uci-defaults/; they should be numbered to allow reliable and reproducible execution. The order might interfere (even some core services like odhcpd do not have a numbered script at this point).
UCI defaults scripts do get executed. However, of they return an error OpenWrt will not remove them from /etc/uci-defaults/ and keep running them at every boot until they finish successfully. That's what you might have been seeing. Setting an 'exit 1' somewhere in your script e.g. is enough you trigger this behaviour.
I've been trying to run them after booting (and SSH is available) to see if there are any major problems with them, but obviously that won't find any issues relating to early boot. Mostly this eliminates eg. syntax errors or non-POSIX-isms.
The thing is, most of them really are simply UCI config scripts. The first one that runs simply moves a file (/etc/banner_td to /etc/banner), and even it fails... sometimes.
I can get boot output on the serial console, but I can't actually see anything in there relating to these scripts. That's why I'm wondering if I can get some parent of the thing that calls the scripts (S10boot) to log to the kernel log (or whatever the name of the thing that appears on the serial output is).
Note that automatically run scripts pretty much run headless (with no display). If there is something that expects an interactive shell, the script may fail. That has been sometimes a reason for script failures.
I have a gut feeling that part of your difficulties from trying to invade the sysupgrade process too early. Filesystem is not ready, overlay has not been initialised or something like that.
Or a race condition like already mentioned.
There are a few recent threads about difficulties with sysupgrade / filesystem / scripts, and the underlying common things is generally "trying to make sysupgrade smooth via scripts". Especially with slower routers, the initialisation of the jffs2/ubi overlay make take some time.
Are you sure that you need to run your scripts from uci-defaults? Maybe they could be run from /etc/rc.local that is run late in the boot process.
Ps.
I also wonder why spend time to tweak scripts, as much easier might be to cook up a personal firmware with the needed packages built-in and your personal settings stored in the image. Using full buildroot or imagegenerator is rather simple. If you can tweak boot scripts, you can also easily build firmwares.
Have it in /files/etc in the buildroot. Then I have to version control it alongside the entire OpenWrt tree.
Have it in a package. I can't do that, because it collides with the same file in the base-files package and building the image fails.
Have it in a package as /etc/banner_td and a uci-defaults script to move it into place. Still not ideal, because I pay the cost of the file size twice, but if they're not big files it's not an issue.
They definitely run headless, they run successfully 100% of the time I flash using mtd.
I have a gut feeling that part of your difficulties from trying to invade the sysupgrade process too early.
I don't understand this. If it's in uci-defaults, why would it run during sysupgrade itself? Shouldn't it run on a reboot?
Are you sure that you need to run your scripts from uci-defaults? Maybe they could be run from /etc/rc.local that is run late in the boot process.
Would they not run every time then?
I also wonder why spend time to tweak scripts, as much easier might be to cook up a personal firmware with the needed packages built-in and your personal settings stored in the image. Using full buildroot or imagegenerator is rather simple. If you can tweak boot scripts, you can also easily build firmwares.
I am building images from the buildroot. That's how my uci-defaults scripts are there.
Here's what I don't understand: if the uci-defaults scripts run early enough that the filesystem might not be ready, why is it the official way to personalise UCI config? Shouldn't it be moved later in the boot process?
Oh, I just found this in the wiki. I'm not really sure how it uses a separate git repo though. The script just seems to version it in your own fork of OpenWrt.
I am building for multiple devices across a variety of architectures. ./scripts/env allows you to retain settings and switch between each of those targets.
You will see when you go inside env/, it has a .git subdir.
~/code/lede/master/env$ ls -al
total 284
drwxr-xr-x 4 user user 4096 sep 21 12:30 .
drwxr-xr-x 18 user user 4096 sep 22 12:41 ..
-rw-r--r-- 1 user user 271959 sep 21 12:30 .config
drwxr-xr-x 4 user user 4096 sep 20 23:17 files
drwxr-xr-x 8 user user 4096 sep 21 12:30 .git
OpenWrt, by default, will not keep track of anything you put in env/ unless you tell it to. Check git status after dropping files in env/, e.g. (in the top OpenWrt dir, mind you; not in env/ itself of course).
Normally files/ is excluded from the openwrt main sources git (in .gitignore), but if you want to version control that along the normal openwrt sources with the normal git, just remove the line from .gitignore.
And no, I wouldn't remove files/ (or env/files/, to which the former is just a symlink) from .gitignore. Better just use ./scripts/env. That's what it's for, if you want to keep track of changes to those files.
But yeah, one does not need to version control those files. But you'll have to if you if you build for multiple targets in the same tree (it would be a waste of space not to). And that's where the env script comes into play.
