Support 4G/5G automatic reconnection using ModemManager

Any other ideas to try @vgaetera? Is it possible to issue mmcli connect and DHCP renew of sorts?

I just can't seem to get something that's faster than 'ifup $CFG' that works. I tried mmcli connect followed by renew, but that doesn't work.

It seems mmcli connect after disconnect is pretty much instantaneous and new IP details are reported in logread. But how to get those new details from modemmanager to netifd? Since otherwise 'ifstatus wan' shows bad details despite the proper reconnect.

Thinking about what is apparently missing from OpenWrt right now according to the modemmanager developer:

The way to solve this, after briefly talking with @jow- about this in IRC would be to have the netifd protocol handler launch a "watcher" process which brings up the connection, and is kept alive and running for as long as the network interface is assumed connected. If MM detects a network-initiated disconnection, the dispatcher script called by MM should kill that watcher process. At that point, netifd (if configured to autoconnect) will then kill the netdev, restart that process and await proto updates.

Anyone up to the task of writing this logic in the netfid modemmanager protocol handler?

perhaps that watcher process arrangement could give speedier reconnects somehow by dispensing with the need for the time consuming ifdown and ifup, whilst at the same time managing the IP change?

@mkrle any ideas?

It seems OpenWrt really needs a volunteer to implement the necessary changes to the netifd modemmanager protocol handler.

Can you try commenting out this line in /lib/netifd/proto/modemmanager.sh and checking if it sufficiently reduces the time taken by ifup:

mmcli --modem="${device}" --disable
2 Likes

Excellent call. That significantly reduces the reconnect time on my NR7101 from around five seconds to around three seconds. I wonder in what situations the temporally costly modem disable is actually needed?

Since we surely want the fastest possible reconnect, I wonder whether there might be any other optimisations available to try.

In my script:

root@OpenWrt:~# cat /usr/lib/ModemManager/connection.d/10-report-down-and-reconnect
#!/bin/sh

# Automatically report to netifd that the underlying modem
# is really disconnected and reconnect if interface was up

# require program name and at least 4 arguments
[ $# -lt 4 ] && exit 1

MODEM_PATH="$1"
BEARER_PATH="$2"
INTERFACE="$3"
STATE="$4"

[ "${STATE}" = "disconnected" ] || exit 0

. /usr/share/ModemManager/modemmanager.common
. /lib/netifd/netifd-proto.sh
INCLUDE_ONLY=1 . /lib/netifd/proto/modemmanager.sh

MODEM_STATUS=$(mmcli --modem="${MODEM_PATH}" --output-keyvalue)
[ -n "${MODEM_STATUS}" ] || exit 1

MODEM_DEVICE=$(modemmanager_get_field "${MODEM_STATUS}" "modem.generic.device")
[ -n "${MODEM_DEVICE}" ] || exit 2

CFG=$(mm_get_modem_config "${MODEM_DEVICE}")
[ -n "${CFG}" ] || exit 3

IFUP=$(ifstatus ${CFG} | jsonfilter -e '@.up')

logger -t "modemmanager" "interface ${CFG} (network device ${INTERFACE}) ${STATE}"
proto_init_update $INTERFACE 0
proto_send_update $CFG

[ "${IFUP}" = "true" ] && ifup ${CFG}

exit 0

what does:

proto_init_update $INTERFACE 0
proto_send_update $CFG

actually do before the [ "${IFUP}" = "true" ] && ifup ${CFG} call?

1 Like

My view of the protocol handlers is that they are designed to be simple - setup and teardown (+renew). When MM starts up (and before ifup), it puts the modem in initialized/disabled. state, so it makes sense for teardown to put it back in disabled state on ifdown.

What comes to my mind could be:

  • Add an uci setting to skip disable on teardown or
  • Trigger renew from the MM-down script, and implement renew as a minimal possible version of what the setup is already doing, e.g. just simple connect + IP setup

There seems to be some logic to this in my mind.

Any idea how this should change for that:

root@OpenWrt:~# cat /usr/lib/ModemManager/connection.d/10-report-down-and-reconnect
#!/bin/sh

# Automatically report to netifd that the underlying modem
# is really disconnected and reconnect if interface was up

# require program name and at least 4 arguments
[ $# -lt 4 ] && exit 1

MODEM_PATH="$1"
BEARER_PATH="$2"
INTERFACE="$3"
STATE="$4"

[ "${STATE}" = "disconnected" ] || exit 0

. /usr/share/ModemManager/modemmanager.common
. /lib/netifd/netifd-proto.sh
INCLUDE_ONLY=1 . /lib/netifd/proto/modemmanager.sh

MODEM_STATUS=$(mmcli --modem="${MODEM_PATH}" --output-keyvalue)
[ -n "${MODEM_STATUS}" ] || exit 1

MODEM_DEVICE=$(modemmanager_get_field "${MODEM_STATUS}" "modem.generic.device")
[ -n "${MODEM_DEVICE}" ] || exit 2

CFG=$(mm_get_modem_config "${MODEM_DEVICE}")
[ -n "${CFG}" ] || exit 3

IFUP=$(ifstatus ${CFG} | jsonfilter -e '@.up')

logger -t "modemmanager" "interface ${CFG} (network device ${INTERFACE}) ${STATE}"
proto_init_update $INTERFACE 0
proto_send_update $CFG

[ "${IFUP}" = "true" ] && ifup ${CFG}

exit 0
1 Like

A year later, in the latest openwrt 23.05 and main line tasks.
There is still no change in the Modemmanager, and I cannot reconnect myself after disconnecting the link.
You must manually click Reconnect on the network port interface to work properly.
Is anyone willing to think of a solution?

See post before yours. It’s a hack, but it works.

Well, it works with ModemManager only. ModemManager is quite big and thus unwelcome on smaller devices. When used with Huawei modems, it needlessly disables IPv6 (see my bug report from 5 years ago). Also, it is incompatible with luci-app-3ginfo-lite and luci-app-sms-tool-js from @IceG. So, it is not a universally acceptable solution, but, sadly, we don't have any other.

1 Like

The problem I see with uqmi and most other tools is that they need exclusive access to cdc-wdm device and/or USB serial (AT command) port. So a centralized daemon to manage/serialize requests and monitor/notify on events is probably need. This is more or less exactly what MM already does - heavy as it is I find it well architected, with an API/library, modem drivers etc.

It'd be great if we could get some feedback from the OpenWRT gurus on what the ideal architecture to manage the modem (re)connections would be, as I believe ifdown/ifup may be too heavy (slow) for this task. Otherwise we'll be stuck in the world of scripts and hacks.

3 Likes

For those interested, see this:

1 Like

@mkrle I've been looking into:

Right now the dispatcher script in modemmanager calls on disconnection:

logger -t "modemmanager" "interface ${CFG} (network device ${INTERFACE}) ${STATE}"                               
proto_init_update $INTERFACE 0
proto_send_update $CFG

This results in a lengthy 'ifdown' including a modem disable arising from /lib/netifd/proto/modemmanager.sh:

proto_modemmanager_teardown() {
        local interface="$1"

        local modemstatus bearerpath errorstring
        local bearermethod_ipv4 bearermethod_ipv6

        local device lowpower iptype
        json_get_vars device lowpower iptype

        echo "stopping network"

        # load connected bearer information, just the first one should be ok
        modemstatus=$(mmcli --modem="${device}" --output-keyvalue)
        bearerpath=$(modemmanager_get_field "${modemstatus}" "modem.generic.bearers.value\[1\]")
        [ -n "${bearerpath}" ] || {
                echo "couldn't load bearer path: disconnecting anyway"
                mmcli --modem="${device}" --simple-disconnect >/dev/null 2>&1
                return
        }

        # load bearer connection methods
        bearerstatus=$(mmcli --bearer "${bearerpath}" --output-keyvalue)
        bearermethod_ipv4=$(modemmanager_get_field "${bearerstatus}" "bearer.ipv4-config.method")
        [ -n "${bearermethod_ipv4}" ] &&
                echo "IPv4 connection teardown required in interface ${interface}: ${bearermethod_ipv4}"
        bearermethod_ipv6=$(modemmanager_get_field "${bearerstatus}" "bearer.ipv6-config.method")
        [ -n "${bearermethod_ipv6}" ] &&
                echo "IPv6 connection teardown required in interface ${interface}: ${bearermethod_ipv6}"

        # disconnection handling only requires special treatment in IPv4/PPP
        [ "${bearermethod_ipv4}" = "ppp" ] && modemmanager_disconnected_method_ppp_ipv4 "${interface}"
        modemmanager_disconnected_method_common "${interface}"

        # disconnect
        mmcli --modem="${device}" --simple-disconnect ||
                proto_notify_error "${interface}" DISCONNECT_FAILED

        # disable
        mmcli --modem="${device}" --disable

        # low power, only if requested
        [ "${lowpower:-0}" -lt 1 ] ||
                mmcli --modem="${device}" --set-power-state-low

        proto_init_update "*" 0
        proto_send_update "$interface"
}

What would be the correct netifd way to skip the 'ifdown' and hence the teardown call in /lib/netifd/proto/modemmanager.sh, and simply call the setup, which I think in /lib/netifd/proto/modemmanager.sh is:

proto_modemmanager_setup()

Looking at the way ppp protocol handler is implemented, it seems the teardown will receive exit code from the monitored protocol app. So I would suggest using return value from the mmcli tracking (from the new PR) to differentiate reconnect vs intentional ifdown, and then skip disable if the teardown is triggered by monitor process exiting.

And that's just a stab at it, I can imagine in some scenarios with some modems it may well be necessary to do a disable/enable, I only have limited experience with this.

I would say that the teardown should always be skipped. The modem is also useful for SMS, and disabling it would rob the user of this capability.

In other words: the modem should be enabled on detection, never disabled, and only the data connection (bearer) should be turned on/off when needed.

2 Likes

@mkrle and @patrakov what I can’t figure out is what are the netifd calls necessary for skipping teardown but reconnecting and updating IP?

Wrong question. The teardown does more than what is needed. Just remove the mmcli --disable call from the script, and you will save the time previously spent on re-registration with the operator.

1 Like

Where the connection is already disconnected (modemmanager dispatcher script on connection down), is teardown needed at all? So rather than just comment out the disable in teardown, can’t we just skip the teardown altogether and go straight to connect and update IP?

So I mean with my reconnect hack:

root@OpenWrt:~# cat /usr/lib/ModemManager/connection.d/10-report-down-and-reconnect
#!/bin/sh

# Automatically report to netifd that the underlying modem
# is really disconnected and reconnect if interface was up

# require program name and at least 4 arguments
[ $# -lt 4 ] && exit 1

MODEM_PATH="$1"
BEARER_PATH="$2"
INTERFACE="$3"
STATE="$4"

[ "${STATE}" = "disconnected" ] || exit 0

. /usr/share/ModemManager/modemmanager.common
. /lib/netifd/netifd-proto.sh
INCLUDE_ONLY=1 . /lib/netifd/proto/modemmanager.sh

MODEM_STATUS=$(mmcli --modem="${MODEM_PATH}" --output-keyvalue)
[ -n "${MODEM_STATUS}" ] || exit 1

MODEM_DEVICE=$(modemmanager_get_field "${MODEM_STATUS}" "modem.generic.device")
[ -n "${MODEM_DEVICE}" ] || exit 2

CFG=$(mm_get_modem_config "${MODEM_DEVICE}")
[ -n "${CFG}" ] || exit 3

IFUP=$(ifstatus ${CFG} | jsonfilter -e '@.up')

logger -t "modemmanager" "interface ${CFG} (network device ${INTERFACE}) ${STATE}"
proto_init_update $INTERFACE 0
proto_send_update $CFG

[ "${IFUP}" = "true" ] && ifup ${CFG}

exit 0

isn't there a way to get netifd to skip the teardown and go straight to setup and update IP? So a renew?

If I'm understanding correctly, this portion here:

logger -t "modemmanager" "interface ${CFG} (network device ${INTERFACE}) ${STATE}"
proto_init_update $INTERFACE 0
proto_send_update $CFG

[ "${IFUP}" = "true" ] && ifup ${CFG}

actually calls ifdown twice - once for the proto_init_update/send_update and a second time because ifup includes an ipdown call. But I'm hazy because I'm struggling to untangle the netifd stuff and I'm not seeing any helpful documentation (including here).

2 Likes

I know slightly off topic but can the interface also reboot based on low bandwidth or as such as I have the issue of the system locking in 3g mode and not Jumping back.
I did do a post on the cake autorate thread but it might be more appropriate here.
CAKE w/ Adaptive Bandwidth - Community Builds, Projects & Packages - OpenWrt Forum

@Lynx thank you for your script, which I have used to successfully reconnect for the last week.

However, last night, the folliowing happened:

Tue Jan 23 00:16:01 2024 daemon.info [26740]: <info>  [modem0/bearer3] connection #1 finished: duration 57599s, tx: 26476768614 bytes, rx: 11003710375 bytes
Tue Jan 23 00:16:01 2024 user.notice modemmanager: interface mwan3g2 (network device wwan0) disconnected, restart automatically with ifup

The restart worked, but only for 99s:

Tue Jan 23 00:17:43 2024 daemon.info [26740]: <info>  [modem0/bearer5] connection #1 finished: duration 99s, tx: 26477112354 bytes, rx: 530828 bytes
Tue Jan 23 00:17:43 2024 user.notice modemmanager: interface mwan3g2 (network device wwan0) disconnected, restart automatically with ifup

Ok, so restart again, but... then it went wrong:

Tue Jan 23 00:17:49 2024 daemon.warn [26740]: <warn>  [modem0] couldn't load operator code: Current operator MCC/MNC is still unknown
Tue Jan 23 00:17:49 2024 daemon.warn [26740]: <warn>  [modem0] couldn't load operator name: Current operator name is still unknown
Tue Jan 23 00:17:49 2024 daemon.info [26740]: <info>  [modem0] 3GPP registration state changed (idle -> searching)
Tue Jan 23 00:17:49 2024 daemon.info [26740]: <info>  [modem0] state changed (enabled -> searching)
Tue Jan 23 00:18:15 2024 daemon.warn [26740]: <warn>  [modem0/bearer6] connection attempt #1 failed: Operation was cancelled
Tue Jan 23 00:18:15 2024 daemon.info [26740]: <info>  [modem0] state changed (searching -> disconnecting)
Tue Jan 23 00:18:15 2024 daemon.info [26740]: <info>  [modem0] state changed (disconnecting -> searching)
Tue Jan 23 00:18:15 2024 daemon.info [26740]: <info>  [modem0/bearer6] connection #1 finished: duration 0s, tx: 0 bytes, rx: 0 bytes
Tue Jan 23 00:18:15 2024 daemon.notice netifd: mwan3g2 (28354): error: couldn't connect the modem: 'GDBus.Error:org.freedesktop.ModemManager1.Error.Core.Cancelled: Operation was cancelled'

The Operation Canceled does not trigger the modem manager report down script in

/usr/lib/ModemManager/connection.d/10-report-down

It does not trigger any new mwan3 hotplug events that I could use in

/etc/mwan3.user

Any ideas? Is there a way to configure ModemManager to perform multiple connection attempts?

Just to check you have only the one script and not two in that connection.d folder? And did you add the wording about automatic reconnect? Also, interaction with mwan3 may lead to issues and should be checked over.