# Master-nginx: Boot hangs at loading kernel after sysupgrade **URL:** \ **Category:** For Developers\ **Created:** [October 31, 2020, 8:21pm UTC](https://forum.openwrt.org/t/master-nginx-boot-hangs-at-loading-kernel-after-sysupgrade/78151 "2020-10-31T20:21:46Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![InkblotAdmirer](https://forum.openwrt.org/letter_avatar/inkblotadmirer/32/5_5575768a8748004e209b776fc1b2916d.png) [@InkblotAdmirer](https://forum.openwrt.org/u/InkblotAdmirer)\ **Post date:** [October 31, 2020, 8:21pm UTC](https://forum.openwrt.org/t/master-nginx-boot-hangs-at-loading-kernel-after-sysupgrade/78151/1 "2020-10-31T20:21:46Z")
I have multiple mvebu routers, various makes and models. After sysupgrade, several of these won't boot (or boot strangely). I hooked up serial to one and after a successful sysupgrade message I get this: ``` umount: can't unmount /dev: Resource busy umount: can't unmount /tmp: Resour ``` I'm guessing this last is cut short due to a reboot, so there may have been more messages coming. In any case, the restart proceeds until "loading kernel" and hangs there. The kernel info is correct, new build date, etc and checksums OK. I'm seeing this on WRT1900ACS and WRT3200ACM. Interestingly, not on WRT1200AC which I believe is same hardware as WRT1900ACS, just 2 antennas instead of 4. Any ideas on how to troubleshoot? I tried flashing both img and bin, same result. ---
**Author:** ![InkblotAdmirer](https://forum.openwrt.org/letter_avatar/inkblotadmirer/32/5_5575768a8748004e209b776fc1b2916d.png) [@InkblotAdmirer](https://forum.openwrt.org/u/InkblotAdmirer)\ **Post date:** [November 7, 2020, 5:57am UTC](https://forum.openwrt.org/t/master-nginx-boot-hangs-at-loading-kernel-after-sysupgrade/78151/2 "2020-11-07T05:57:07Z")
After some config trial and error I get a loading kernel -- see the diff below, I'm not sure which is the culprit or why I was messing with these. Either they were fat-fingered or they've been there a while and something changed making one of them matter. (I'm guessing the latter because I check diffconfigs after most upgrades and didn't notice these as changes.) However, the boot is now hanging at procd: ubus. I have added all new users (ntp, logd, ubus) to /etc/passwd, shadow and group. I also added the socket config to /etc/config/rpcd just in case (see [newest img upgrade failed](https://forum.openwrt.org/t/the-newest-openwrt-x86-64-generic-ext4-combined-img-upgrade-failed/78128)). Any other config changes that matter? For the record I sysupgraded without keeping changes and the boot is clean -- if it comes down to it I can reconfigure from scratch, I just figured this should have been a straightforward change. I still get the umount errors (see orig message) when sysupgrading -- not sure if this is related. Config diff, from non-booting to booting kernel: ``` < CONFIG_KERNEL_DEBUG_LL=y < CONFIG_KERNEL_DEBUG_LL_UART_NONE=y < # CONFIG_KERNEL_KALLSYMS is not set < # CONFIG_KERNEL_MEMCG_SWAP is not set > # CONFIG_KERNEL_SWAP is not set ``` ---
**Author:** ![InkblotAdmirer](https://forum.openwrt.org/letter_avatar/inkblotadmirer/32/5_5575768a8748004e209b776fc1b2916d.png) [@InkblotAdmirer](https://forum.openwrt.org/u/InkblotAdmirer)\ **Post date:** [November 7, 2020, 3:53pm UTC](https://forum.openwrt.org/t/master-nginx-boot-hangs-at-loading-kernel-after-sysupgrade/78151/3 "2020-11-07T15:53:09Z")
Increasing debug level for boot logging reveals this culprit: ``` [12.508718] procd: Finished hotplug exec instance, pid=1139 [12.514507] procd: Launched hotplug exec instance, pid=1144 sh: write error: No such device [12.525157] procd: Finished hotplug exec instance, pid=1144 [13.009273] procd: Coldplug complete [13.012878] procd: Change state 1 -> 2 [13.016665] procd: - watchdog - [13.019833] procd: Ping [13.022304] procd: Opened watchdog with timeout 30s [13.027267] procd: - ubus - [13.030172] procd: Create service ubus [13.034022] procd: Start instance ubus::instance1 [13.038907] procd: Started instance ubus::instance1[1149] [13.044412] procd: running /etc/init.d/ubus running [13.049384] procd: glob failed on /etc/init.d/ubus [13.054238] procd: Instance ubus::instance1 exit with error code 65280 after 0 seconds [13.079366] procd: Connection to ubus failed ``` The router from this point continually retries to start ubus, failing each time. I see no /etc/init.d/ubus on any of my routers, even the ones that boot past this point. I also don't see this file installed or created in the package/system/ubus/Makefile. Any idea what's going on here? ---
**Author:** ![anon50098793](https://forum.openwrt.org/letter_avatar/anon50098793/32/5_5575768a8748004e209b776fc1b2916d.png) [@anon50098793](https://forum.openwrt.org/u/anon50098793)\ **Post date:** [November 7, 2020, 5:28pm UTC](https://forum.openwrt.org/t/master-nginx-boot-hangs-at-loading-kernel-after-sysupgrade/78151/4 "2020-11-07T17:28:00Z")
in future, be a good sport and state clearly what versions you are working with... cheers... > [@The newest openwrt-x86-64-generic-ext4-combined.img upgrade failed](https://forum.openwrt.org/t/the-newest-openwrt-x86-64-generic-ext4-combined-img-upgrade-failed/78128): > > Upgrade from openwrt-x86-64-generic-ext4-combined.img(v5.4.71-r14718-4a0688ed71) to openwrt-x86-64-generic-ext4-combined.img(v5.4.72-r14818-237f708b3c) will timeout and ping no answer. But upgrade with openwrt-x86-64-generic-squashfs-combined.img is OK. And I find the reason is that [https://git.openwrt.org/?p=openwrt/openwrt.git;a=commitdiff;h=2dffadece9a7243a236ce7d91719787a671e23d4](https://git.openwrt.org/?p=openwrt/openwrt.git;a=commitdiff;h=2dffadece9a7243a236ce7d91719787a671e23d4) Move /var/run/ubus.sock to /var/run/ubus/ubus.sock in preparation for having ubusd run as non-root user. sy… > [@UAP-AC-PRO not reachable after first bootup](https://forum.openwrt.org/t/uap-ac-pro-not-reachable-after-first-bootup/78132/6): > > you should see something like these in your vanilla rootfs... ############!/bin/sh #["$(uci get rpcd.@rpcd[0].socket)" = "/var/run/ubus.sock" ] && { # uci set rpcd.@rpcd[0].socket='/var/run/ubus/ubus.sock' # uci commit rpcd # #} #["$(uci -q get uhttpd.main.ubus\_socket)" = "/var/run/ubus.sock"] && { # uci set uhttpd.main.ubus\_socket='/var/run/ubus/ubus.sock' # uci commit uhttpd # #} fgrep -r ubus build\_dir/\*/root\* | grep sock ---
**Author:** ![InkblotAdmirer](https://forum.openwrt.org/letter_avatar/inkblotadmirer/32/5_5575768a8748004e209b776fc1b2916d.png) [@InkblotAdmirer](https://forum.openwrt.org/u/InkblotAdmirer)\ **Post date:** [November 7, 2020, 7:05pm UTC](https://forum.openwrt.org/t/master-nginx-boot-hangs-at-loading-kernel-after-sysupgrade/78151/5 "2020-11-07T19:05:32Z")
Thanks for the response. I will update the original post to state I'm building from master and this seems related to the updates over the last few weeks to run logd, ubus, ntp as non-root. So I had linked to the post in your first response indicating I had made those changes. I use nginx not uhttpd so I believe the only one that matters is the rpcd config. In that case I just pre-updated the config as it doesn't seem to matter for builds prior to the non-root upgrades. With a `uci get rpcd.@rpcd[0].socket` I get a return of `/var/run/ubus/ubus.sock`. On the off-chance it matters I updated the uhttpd config to include the socket change, with no luck. I'm guessing there's something else legacy that matters in my config because I get a clean boot if I don't keep changes. ---
**Author:** ![anon50098793](https://forum.openwrt.org/letter_avatar/anon50098793/32/5_5575768a8748004e209b776fc1b2916d.png) [@anon50098793](https://forum.openwrt.org/u/anon50098793)\ **Post date:** [November 7, 2020, 7:09pm UTC](https://forum.openwrt.org/t/master-nginx-boot-hangs-at-loading-kernel-after-sysupgrade/78151/6 "2020-11-07T19:09:19Z")
``` fgrep -r sock /sbin/ubusd /etc/ /rom | grep 'run/ubus'; find /tmp/ | grep ubus.sock; grep BUILD_ID /etc/os-release ``` ---
**Author:** ![InkblotAdmirer](https://forum.openwrt.org/letter_avatar/inkblotadmirer/32/5_5575768a8748004e209b776fc1b2916d.png) [@InkblotAdmirer](https://forum.openwrt.org/u/InkblotAdmirer)\ **Post date:** [November 7, 2020, 10:11pm UTC](https://forum.openwrt.org/t/master-nginx-boot-hangs-at-loading-kernel-after-sysupgrade/78151/7 "2020-11-07T22:11:24Z")
Results -- this is from the version from which I am upgrading (I modified the /rom to /rom/etc as the command never finishes with just /rom): ``` fgrep -r sock /sbin/ubusd /etc/ /rom/etc/ | grep '/run/ubus'; find /tmp/ | grep ubus.sock; grep BUILD_ID /etc/os-release fgrep: /etc/localtime: No such file or directory /sbin/ubusd:/var/run/ubus.sock /etc/config/rpcd: option socket '/var/run/ubus/ubus.sock' /etc/config/uhttpd: option ubus_socket '/var/run/ubus/ubus.sock' fgrep: /etc/ppp/resolv.conf: No such file or directory /etc/nginx/conf.d/luci.locations: ubus_socket_path /var/run/ubus.sock; /etc/nginx/luci_uwsgi.conf: ubus_socket_path /var/run/ubus/ubus.sock; /rom/etc/config/rpcd: option socket /var/run/ubus.sock fgrep: /rom/etc/localtime: No such file or directory fgrep: /rom/etc/ppp/resolv.conf: No such file or directory /rom/etc/uci-defaults/60_nginx-luci-support: ubus_socket_path /var/run/ubus.sock; /tmp/run/ubus.sock BUILD_ID="r14721+18-4ff7bdfeeb" ``` The uci-defaults for nginx updates a file that I do not reference in my nginx config -- I reference the luci\_uwsgi.conf file in /etc/nginx. ---
**Author:** ![anon50098793](https://forum.openwrt.org/letter_avatar/anon50098793/32/5_5575768a8748004e209b776fc1b2916d.png) [@anon50098793](https://forum.openwrt.org/u/anon50098793)\ **Post date:** [November 7, 2020, 10:15pm UTC](https://forum.openwrt.org/t/master-nginx-boot-hangs-at-loading-kernel-after-sysupgrade/78151/8 "2020-11-07T22:15:54Z")
> [@anon50098793](#): > > what versions ok, so that's half of the picture... > [@InkblotAdmirer](#): > > ```plaintext > /sbin/ubusd:/var/run/ubus.sock > /etc/config/rpcd: option socket '/var/run/ubus/ubus.sock' > > ``` > [@InkblotAdmirer](#): > > I just pre-updated the config that's unchartered territory... > [@InkblotAdmirer](#): > > it doesn't seem to matter for builds prior to the non-root upgrades what is a non-root upgrade? and why exactly does it not matter? either way... please upgrade without keeping settings then manually re-apply what you need to... no point digging a hole to china when you can fly... ---
**Author:** ![InkblotAdmirer](https://forum.openwrt.org/letter_avatar/inkblotadmirer/32/5_5575768a8748004e209b776fc1b2916d.png) [@InkblotAdmirer](https://forum.openwrt.org/u/InkblotAdmirer)\ **Post date:** [November 7, 2020, 11:49pm UTC](https://forum.openwrt.org/t/master-nginx-boot-hangs-at-loading-kernel-after-sysupgrade/78151/9 "2020-11-07T23:49:27Z")
> [@anon50098793](#): > > what is a non-root upgrade? and why exactly does it not matter? I was referring to the series of commits that modify ntp, logd, ubusd to run as non-root. It doesn't seem to matter because even with that line in the config, prior to these commits the ubus socket is still at /var/run/ubus.sock. I may eventually bite the bullet and do what you're suggesting but I'm also inclined to understand what's going on here. Not being familiar with how procd, ubus etc interact the code is hard to walk through. I can't figure out why procd is reporting that ubus is trying to access /etc/init.d/ubus -- whether it does that all the time regardless and the failure to find it doesn't matter, etc. > [@InkblotAdmirer](#): > > ```plaintext > [13.034022] procd: Start instance ubus::instance1 > [13.038907] procd: Started instance ubus::instance1[1149] > [13.044412] procd: running /etc/init.d/ubus running > [13.049384] procd: glob failed on /etc/init.d/ubus > [13.054238] procd: Instance ubus::instance1 exit with error code 65280 after 0 seconds > > ``` Is it the fail on /etc/init.d/ubus that is the problem or is it something else that fails and emits the error code? I can find where the message is printed and what calls it, but I can't find the definitions for what 65280 means. ---
**Author:** ![InkblotAdmirer](https://forum.openwrt.org/letter_avatar/inkblotadmirer/32/5_5575768a8748004e209b776fc1b2916d.png) [@InkblotAdmirer](https://forum.openwrt.org/u/InkblotAdmirer)\ **Post date:** [November 8, 2020, 12:53am UTC](https://forum.openwrt.org/t/master-nginx-boot-hangs-at-loading-kernel-after-sysupgrade/78151/10 "2020-11-08T00:53:44Z")
Below is what happens during a "working" boot. It fails the /etc/init.d/ubus access but looks like there are follow-on items that include reading random numbers. ``` [12.374312] procd: - ubus - [12.377142] procd: Create service ubus [12.380939] procd: Start instance ubus::instance1 [12.385835] procd: Started instance ubus::instance1[1108] [12.391306] procd: running /etc/init.d/ubus running [12.398885] procd: glob failed on /etc/init.d/ubus [12.406008] random: ubusd: uninitialized urandom read (4 bytes read) [12.428580] random: ubusd: uninitialized urandom read (4 bytes read) [12.435054] random: ubusd: uninitialized urandom read (4 bytes read) [12.442179] procd: Connected to ubus, id=24685d4c [12.446939] procd: Change state 2 -> 3 ``` From ubus code: [ubusd\_id.c](https://git.openwrt.org/?p=project/ubus.git;a=blob;f=ubusd_id.c;h=8d9fede2dd10296db2d8abc189bc3075fb216150;hb=HEAD#l40) ``` 40 void ubus_init_id_tree(struct avl_tree *tree) 41 { 42 if (random_fd < 0) { 43 random_fd = open("/dev/urandom", O_RDONLY); 44 if (random_fd < 0) { 45 perror("open"); 46 exit(1); 47 } 48 } 49 50 avl_init(tree, ubus_cmp_id, false, NULL); 51 } ``` This appears to call above code, accouting for 2 of the randoms: [ubusd\_obj.c](https://git.openwrt.org/?p=project/ubus.git;a=blob;f=ubusd_obj.c;hb=ad0cd117db74934385d81605514e041b1a9cdda9) ``` 227 static void __constructor ubusd_obj_init(void) 228 { 229 ubus_init_id_tree(&objects); 230 ubus_init_id_tree(&obj_types); 231 ubus_init_string_tree(&path, false); 232 ubusd_event_init(); 233 ubusd_acl_init(); 234 ubusd_monitor_init(); 235 } ``` The third area that calls the init\_id routine doesn't appear to get specifically called by any code but is a constructor so gets called automatically when ubus is started I'm guessing. These are the only places /dev/random is read so either this fails or something before the random reads that doesn't emit any debug text. ---
**Author:** ![InkblotAdmirer](https://forum.openwrt.org/letter_avatar/inkblotadmirer/32/5_5575768a8748004e209b776fc1b2916d.png) [@InkblotAdmirer](https://forum.openwrt.org/u/InkblotAdmirer)\ **Post date:** [November 11, 2020, 1:02am UTC](https://forum.openwrt.org/t/master-nginx-boot-hangs-at-loading-kernel-after-sysupgrade/78151/11 "2020-11-11T01:02:10Z")
So I have figured out the files (possibly not all) that matter here. 1. Sysupgrading when /etc/shadow, /etc/passwd and /etc/group are included fails to boot, halting when procd starts ubus. 2. Sysupgrading without keeping changes, and replacing just passwd/shadow/group with those contained in the sysupgrade also fails to boot in the same manner. I replaced these one at a time subsequently: group didn't matter (booted fine after replacing), replacing passwd caused the boot to hang. I did not reflash and try shadow. 3. Sysupgrading without keeping changes, and replacing everything except these 3 files boots fine. The only real difference in passwd/shadow is an additional user I add for samba. Additional testing: editing the files to add the samba user I mentioned above works fine -- I add the user and the router reboots just fine. There seem to only be issues with overwriting passwd (and possibly shadow) through copy/move or tar. Strange? As mentioned above I'm flashing from `r14721+18-4ff7bdfeeb` to latest master, right now `r14887+17-a47279154e`. ``` ``` ---
**Author:** ![InkblotAdmirer](https://forum.openwrt.org/letter_avatar/inkblotadmirer/32/5_5575768a8748004e209b776fc1b2916d.png) [@InkblotAdmirer](https://forum.openwrt.org/u/InkblotAdmirer)\ **Post date:** [November 12, 2020, 7:20am UTC](https://forum.openwrt.org/t/master-nginx-boot-hangs-at-loading-kernel-after-sysupgrade/78151/12 "2020-11-12T07:20:59Z")
So I believe I have this resolved. I still don't understand which commit caused the issue but keeping files was broken at `r14721` and as of `r14898+17-c02096361c` it seems to no longer be an issue. To upgrade and get a booting router: - save off a config, external to router - sysupgrade -n (not keeping files) - ssh into router, transfer config over and extract to a temp location - in the temp location, move /etc/passwd,shadow,groups to a different temp location - cp all remaining config files recursively to their proper locations - edit passwd/shadow/groups manually to add info from the files reserved from the config This then boots and is once again sysupgradable keeping config. ---
**Author:** ![system](https://forum.openwrt.org/uploads/default/original/3X/7/4/740520c71f00039bbf001045c682f0ecb25e905a.png) [@system](https://forum.openwrt.org/u/system)\ **Post date:** [November 22, 2020, 7:21am UTC](https://forum.openwrt.org/t/master-nginx-boot-hangs-at-loading-kernel-after-sysupgrade/78151/13 "2020-11-22T07:21:05Z")
This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.