SoM in Custom Board Reboot Takes 2.5 Minutes

Hello all,

First off, I'm not a linux or embedded system expert, so I was hoping I might get some feedback from someone who is.

I've been using a SoM with an imx6 processor. If I use the SoM in the development board that came with it, the board will reboot normally (around 30 seconds) just using the reboot command. However, if I use the SoM in a custom board we made, the reboot will consistently take about 2.5 minutes. Due to my testing, I'm certain this is a problem in the shutting down phase rather than the booting phase. Note that reboot -f makes no difference.

That suggests to me that something is timing out; however, I am having trouble finding out what it is since I do not have access to a serial console on our custom board (I only have access to a USB on the device, which I use for ethernet to ssh).

I've tried changing /etc/config/system so that logs are preserved and all that, but the logs showed nothing exciting. (tells me the problem persists after log service terminates)

I've tried manually stopping as many services as I could to no avail.

The only thing that has gotten me anywhere is if I force the reboot by manually stopping the watchdog like described here. If I reboot this way, it will reboot in the normal amount of time.

It's my understanding that rebooting this way essentially directly reboots the cpu, bypassing whatever in the shutdown process that is causing the problem.

I would really appreciate any insight into troubleshooting this, as I've spent far too much time slogging through /sys/ and not really knowing what I was doing. The system appears to work identically in the dev board and our custom board except for this issue.

Thanks for any help you can give!

Are you using a console cable to monitor the boot/shutdown process? Or, do you have the ability to use one? This will show you what's going on, without any chance of interruption,even when it flips out.

1 Like

I faced the same problem when parasitic pulsations were received on one of the SoC pins configured like 1PPS. The kernel in this case handled interrupts and did not respond well to other commands.
Look to cat /proc/interrupts in this case

I am not. I only have access to 2 usb ports on my custom board, and in order to use them for anything like this it seems the drivers are either loaded too late or unloaded too early.

Thanks for the tip (never even heard of parasitic pulsations before).

As I said, I'm no expert, but nothing in /proc/interrupts really sticks out too much to me at this point. The only things I notice is the 12 interrupts on the serial and 4 on ethernet. I never removed them from the device tree, so their presence makes sense, but they don't have anything attached.

I suspected the serial line before, as the console is set to be on /dev/ttymxc0, but I have not tried changing or removing the console from the kernel boot args yet.

Here is the contents of /proc/interrupts after running for about 7 minutes. Maybe you can see something I do not.

cat /proc/interrupts
           CPU0
 17:      60616       GPC  55 Level     i.MX Timer Tick
 19:          0       GPC  94 Level     arm-pmu
 21:         12       GPC  26 Level     2020000.serial
 36:          4       GPC 120 Level     20b4000.ethernet
 37:          0       GPC 121 Level     20b4000.ethernet
 38:          0       GPC  80 Level     20bc000.wdog
 50:          0       GPC   2 Level     sdma
 51:       1858       GPC  43 Level     2184000.usb
 52:      47526       GPC  42 Level     2184200.usb
 55:      62599       GPC  22 Level     mmc0
 56:       4066       GPC  23 Level     mmc1
 58:          0       GPC  36 Level     21a0000.i2c
 59:          0       GPC  37 Level     21a4000.i2c
 63:          0       GPC  27 Level     21e8000.serial
 64:          0       GPC  28 Level     21ec000.serial
 68:          2       GPC   6 Level     2284000.rng
IPI0:          0  CPU wakeup interrupts
IPI1:          0  Timer broadcast interrupts
IPI2:          0  Rescheduling interrupts
IPI3:          0  Function call interrupts
IPI4:          0  CPU stop interrupts
IPI5:          0  IRQ work interrupts
IPI6:          0  completion interrupts
Err:          0

Thank you for taking an interest!

Does the board have provisions for a serial connection? USB Support is usually part of the bootloader, are you using u-boot? You'd need the console connection to use it, though.

the board does support serial console through UART; however, I do not have the necessary bits to convert the uart to usb (that component is built in on the dev board).

yeah, I am using u-boot. I was looking into a mod for u-boot to allow me to use USB-serial, but I'm not familiar enough with everything to know if having usb serial in u-boot would transfer and I would keep the serial console after booting.

Right now when I need a u-boot function (like putting it into UMS mode so that I can modify the openWRT installation), I just modify bootcmd with fw_setenv to do a one-time boot with the options I need.

The next version of our board will have JTAG, so hopefully I'll be able to get a serial console through there.

As long as you have a Serial 4-pin or RJ45, you can use a console cable AFAIK. USB UART handling is done via the cable, not the board..

Here is the 4-pin:

And a Rj45 version.
https://www.amazon.com/Console-Essential-Accesory-Ubiquity-Switches/dp/B01AFNBC3K

That's probably a good idea.

The only hold up might be that in the devboard that I've been using, there is a voltage translator between the uart-usb converter and the actual UART pads.

I don't know enough about this to say that's something we could just not have, nor could I tell if something similar was included in the cable you linked.

Thanks for your help!