Trying to troubleshooot unusual DHCP issues

I'm having some issues with DHCP being unreliable on a Raspberry Pi 3B device.

In the log, I see the following error:

Mon Jun  7 14:44:55 2021 daemon.debug dnsmasq-script[1]: /usr/lib/dnsmasq/dhcp-script.sh: .: line 5: can't open '/usr/share/libubox/jshn.sh': No such file or directory

But:

root@Automation-pibridge136:/usr/share/libubox# ls -alh
total 16K
drwxr-xr-x  2 root root 4.0K May 27 13:07 .
drwxr-xr-x 19 root root 4.0K May 27 13:07 ..
-rwxr-xr-x  1 root root 5.4K May 27 13:07 jshn.sh
root@Automation-pibridge136:/usr/share/libubox#

I've tried flagging the script as executable to see if it resolves the issue, but it makes no difference.

Why is dhcp-script unable to find a file that clearly exists?

Look at the first line, where the scripts informs which shell must execute it, and check that it exists.

Also, scripts edited on windows machines have different line endings, and that creates confusion on that first line, thus producing this error.

This wasn't edited at all by myself - the file is coming straight out of git.

jshn.sh doesn't even have a first line indicating which shell as it appears to be intended to always be sourced from another script.

The dependency appears to have been added in January with https://github.com/openwrt/openwrt/commit/aed95c4cb8d5a55a795910a759d08e1393f8d967#diff-941513bb8bba59bb7bad319f7fd3c2f355f66e27f5651dddf7ecea43c83831d8R5 - which adds the following line to dhcp-script.sh

. /usr/share/libubox/jshn.sh

I'm wondering if that commit was tested on a build that has a different default /bin/sh (the first line of dhcp-script merely references /bin/sh ) than what is in my builds - at least in the build I'm running, /bin/sh is provided by busybox - and it looks like that defaults to ash, and has not been changed - I'm assuming that patch would have been tested with a default shell configuration which is why I'm scratching my head as to what is going on here

jshn.sh has not had a first line (since it appears to be designed to be sourced from another script under all circumstances) for as long as it has existed - https://git.openwrt.org/?p=project/libubox.git;a=blob;f=sh/jshn.sh;h=923f2b36318136a51b4f66147ff6f7c4980267be;hb=fda6079b30a4f47daf95c8b93b9c86afee008f45

Strangely DHCP doesn't always fail - but it definitely seems to have started roughly around when that commit went in

Edit: Hmm, dnsmasq appears to be running in ujail on the problem system. Need to look at the full command line to see what the setup is, I wonder if that is preventing access to /usr/share/libubox

Edit 2: I'm guessing someone forgot to add /usr/share/libubox/jshn.sh to https://github.com/openwrt/openwrt/blob/master/package/network/services/dnsmasq/files/dnsmasq.init#L1114 - going to see if that fixes the issue

Edit 3: Now a new error is present:

Tue Jun  8 20:39:27 2021 daemon.debug dnsmasq-script[1]: /usr/lib/dnsmasq/dhcp-script.sh: line 188: jshn: not found
Tue Jun  8 20:39:27 2021 daemon.debug dnsmasq-script[1]: /usr/lib/dnsmasq/dhcp-script.sh: line 53: ubus: not found

It's almost as if the person who committed this was somehow testing in an environment where dnsmasq wasn't running under ujail - but that appears to be the default?

It isn't a path problem, PATH is set to:

/sbin:/usr/sbin:/bin:/usr/bin

and both ubus and jshn are within those paths - appears to be another ujail problem

Everything I'm seeing says that either the commit that added ubus/jshn dependencies was not tested at all, or since that patch some change to ujail has made it more restrictive and broken dhcp-script.sh

https://github.com/openwrt/openwrt/commit/6801ecd91b44ee1cd0dc0158657b427bdbb5f3cf#diff-405f5d9c928426fa54fcf3421b3fd191d1aaf731414e331f58866730e0af1e37 - "jail/seccomp: add support for aarch64" - hmm...

Another edit: Adding /bin/ubus and /usr/bin/jshn to the ujail setup on line 1114 of dnsmasq.init appears to fix the problem - but how the hell is this working on other architectures that had jail support prior to the patch above and/or how has no one noticed it's broken?

1 Like