Shell-scheduler: parallelization library for shell scripts

Community Builds, Projects & Packages
1

Hi all,

here is another side project of mine intended mainly for fellow shell coders. The title is fairly self-descriptive. This is a generalized version of the job scheduler I wrote for adblock-lean (and it's a part of adblock-lean code for a long time now). Now that I'm working on another adblock-lean feature that should benefit from parallelization, naturally I wanted to reuse the same code, and if I'm generalizing it already, I decided to package it as a separate library so other projects can benefit from it as well.

While the initial implementation was entirely coded by hand, I used a couple of clankers for this refactoring. That said, this is not vibe-coded. The clankers mainly helped me to find bugs, write tests, examples and the README. ~95% of the code is still written by hand and the other 5% went through a thorough review. The testing code and the README are not vibe-coded either: I held clanker's hand at every point and manually fixed what it messed up when writing the code and the text (which was a lot of work).

This library was specifically designed to work with Busybox ash but since Bash supports all required features, it works just as well on Bash.

If you use or want to use this library with your OpenWrt project, let me know. If at some point there are enough projects that integrate with it, perhaps it will make sense to make an apk package out of it and have it accepted into the official repo, so rather than every project keeping its own copy of parallelization code, we have a single, well-tested and reusable library.

Feature requests and/or bug reports are welcome as well.

2 Likes
2

Update:

  • fixed some bugs
  • added interface to easily and safely set and fetch job-specific parameters (README updated)
  • refactored the test suite for better readability
  • added many more tests (tests related to params setting and fetching are being worked on, should be available later today)
1 Like
3

Thanks Anton looks really useful :+1:

1 Like
4

Update:

  • Fixed an edge case (global/idle timeout during the initial job dispatch phase occurring almost simultaneously with job completion could prevent the scheduler from invoking the job completion callback for that job)
  • Added support for fetching job-specific parameters on-demand (rather than only have them automatically handed over as environment variables to each job as earlier).
  • Implemented built-in bookkeeping of job statuses - these are now provided via positional parameters to the scheduler termination callback. Each job is classified as either ok or failed or unfinished or undispatched. The implementation guarantees that each job ID passed to schedule_jobs() will appear exactly once in one of these four buckets (README has more info and examples).
  • Added many more tests, refactored and split the testing suite into several files (sorted by category) because the tests line count increased to ~4k lines and this was already unbearable (thanks, Claude).
  • Updated the readme.
  • Issued the first release. I assigned it the version 0.5 because on one hand, this is the first release, but on the other hand, this project is nearly feature-complete (at least I'm not planning any major new features for now) and the very substantial number and variety of tests is giving me reasonable confidence that the code is close to being production-ready. So it feels more like v0.8 but I decided to be conservative, in case someone runs it and it eats their cat.
1 Like
5

Update: released shell-scheduler v0.6 with following main changes:

  • Implement support for per-job timeouts (see the new REFERENCE.md and TIMEKEEPING.md files for details)
  • Improve the documentation and refactor it into 3 files: README.md, REFERENCE.md, TIMEKEEPING.md
  • Support assigning job param values to custom variables with job_get_params <job_id> <var_name>=<param_name>
  • Disallow duplicate job IDs
  • Improve validation of values received via environment variables
  • Significantly expand tests coverage
  • Various bugfixes
2 Likes
6

Thank you for your work

1 Like