Zyxel NR7101 - Bad start every other reboot?

I just noticed that whenever I reboot my Zyxel NR7101 every other reboot it goes into a bad state. No connection and I see stuff in log read like:

Fri Jul 14 23:29:07 2023 daemon.err ModemManager[1384]: hotplug: timed out waiting for the modem to get exported at /sys/devices/platform/1e1c0000.xhci/usb2/2-1
Fri Jul 14 23:29:08 2023 daemon.debug [2673]: transaction 0x25 aborted, but message is not abortable
Fri Jul 14 23:29:09 2023 daemon.debug [2673]: transaction 0x26 aborted, but message is not abortable
Fri Jul 14 23:29:10 2023 daemon.debug [2673]: transaction 0x27 aborted, but message is not abortable
Fri Jul 14 23:29:10 2023 daemon.err ModemManager[2300]: hotplug: modem not detected at sysfs path
Fri Jul 14 23:29:11 2023 daemon.debug [2673]: transaction 0x28 aborted, but message is not abortable
Fri Jul 14 23:29:12 2023 daemon.debug [2673]: transaction 0x29 aborted, but message is not abortable
Fri Jul 14 23:29:13 2023 daemon.debug [2673]: transaction 0x2a aborted, but message is not abortable
Fri Jul 14 23:29:14 2023 daemon.debug [2673]: transaction 0x2b aborted, but message is not abortable
Fri Jul 14 23:29:15 2023 daemon.debug [2673]: transaction 0x2c aborted, but message is not abortable
Fri Jul 14 23:29:15 2023 daemon.err ModemManager[2300]: hotplug: modem not detected at sysfs path
Fri Jul 14 23:29:16 2023 daemon.debug [2673]: transaction 0x2d aborted, but message is not abortable
Fri Jul 14 23:29:17 2023 daemon.debug [2673]: transaction 0x2e aborted, but message is not abortable
Fri Jul 14 23:29:17 2023 daemon.info [2673]: <info>  [device /sys/devices/platform/1e1c0000.xhci/usb2/2-1] creating modem with plugin 'quectel' and '5' ports
Fri Jul 14 23:29:17 2023 daemon.warn [2673]: <warn>  [plugin/quectel] could not grab port cdc-wdm0: Cannot add port 'usbmisc/cdc-wdm0', unhandled port type
Fri Jul 14 23:29:17 2023 daemon.info [2673]: <info>  [base-manager] modem for device '/sys/devices/platform/1e1c0000.xhci/usb2/2-1' successfully created
Fri Jul 14 23:29:20 2023 daemon.warn [2673]: <warn>  [modem0/sim0] couldn't load operator name: Failed to parse CRSM query result '+CRSM: 106,130,""'
Fri Jul 14 23:29:20 2023 daemon.err ModemManager[2300]: hotplug: timed out waiting for the modem to get exported at /sys/devices/platform/1e1c0000.xhci/usb2/2-1
Fri Jul 14 23:29:20 2023 daemon.warn [2673]: <warn>  [modem0/sim0] couldn't load list of emergency numbers: Failed to parse CRSM query result '+CRSM: 148,8,""'

dmesg: https://pastebin.com/wnUmWKUR

logread: https://pastebin.com/MUHjRhqP

Once booted OK it works just fine for months.

It is consistent - I reboot, bad state. Reboot again, good state. Reboot again, bad state, etc.

@bmork any clue what's going on here?

Not really. But I can't think of any other explanation than the modem not being properly reset on reboot. Maybe try toggling the 'lte_reset' gpio switch on system reboot?

There are also those "lte-pwrkey" and "lte-power" gpio-hogs, which I didn't see any reason to expose as switches. Maybe I missed the point? You could try to remove them from the device tree and export them to userspace instead, so makeing it possible to toggle the outputs.

Dumb question time: how do I do that?

There's probably some smarter way to do this since it is exposed in UCI, but you could do

echo 1  >/sys/class/gpio/gpio483/value
sleep 1
echo 0  >/sys/class/gpio/gpio483/value

Just don't do this while upgrading the modem firmware.

Yes this fixes the issue!

So does OpenWrt need to be updated in some way? This is with OpenWrt 22.03.5 r20134-5f15225c1e / LuCI openwrt-22.03 branch git-23.119.80898-65ef406

I upgraded to 23.05.2 and on first boot the modem didn't function and I needed to issue:

echo 1  >/sys/class/gpio/gpio483/value
sleep 1
echo 0  >/sys/class/gpio/gpio483/value

I'm thinking something needs to be updated in OpenWrt to prevent this problem of booting up into a bad state owing to improper modem reset?

@bmork any idea where/how this fix should be applied to OpenWrt? I don't even know where to raise a GitHub issue.

Not really. It's sort of on my todo-list to look at it. But don't expect that to happen any time soon...

The easiest fix is probably a shutdown script putting the modem into reset. It can simply do

echo 1  >/sys/class/gpio/gpio483/value

No need to disable the reset line again. The boot scripts will fix that.

But some care should be taken to avoid killing the modem on firmware upgrades. I just don't know how we should do that. Worst case is someone rebooting the NR7101 while the modem is writing to its internal flash.

Maybe we could do a simple AT or QMI command ping before resetting? And just skip the reset if we can't get a response from the modem? Skipping the reset on some unknown error should be harmless. Worst case is that you end up like now -having to boot a second time.

1 Like

Could this additionally or alternatively be viewed as a deficiency in ModemManager since ModemManager just sits there in an error state without attempting to reboot the modem?

I don't think so. This is related to the hardware design making it posible to reboot the host without resetting the modem. MM cannot be expected to know anything about that. And there is no generic reset it can use even if it realised that a reset is required. The GPIO reset is obviously specific to this system only. It's not for MM to touch, IMHO.

1 Like

This is to confirm that your proposal works.

The init.d mechanism already includes shutdown functionality as described here, and I noticed that the network init.d script at /etc/init.d/network already includes such a shutdown() call with:

shutdown() {
        ifdown -a
        sleep 1
}

and so I tried simply appending the line to set gpio483 to '1' to this call, as follows:

shutdown() {
        ifdown -a
        sleep 1
        echo 1  >/sys/class/gpio/gpio483/value
}

And indeed this ensures the modem is in a good state every reboot.

To make this conditional on some condition not associated with firmware upgrade, I'm not sure how to best query the modem: uqmi -d /dev/cdc-wdm0 --get-data-status does not go via proxy and gives a 'connection broken' error.