Support 4G/5G automatic reconnection using ModemManager

Have you set these correctly:

yes belive so , il try an fresh install to morrow, to clear some files

Sat Jul 22 20:12:57 2023 daemon.notice netifd: 5G (4854): error: couldn't connect the modem: 'GDBus.Error:org.freedesktop.ModemManager1.Error.MobileEquipment.NetworkTimeout: Network ?

can't get modem to work, sim1 or 2 ?

Sim1 I think. I think you should really be using a different thread for this though.

+1

I also think an automatic reconnect of the interface is necessary, as soon as the netifd gets aware, that the underlying network connection goes down.

2 Likes

I consider this approach bloated, far too complicated. Simple script, pinging a well known target, i.g. 8.8.8.8, can do the job for a qmi-interface.

I disagree with you.

Firstly, I do not want to depend my uplink on an external target. If Google (hypothetically) decides tomorrow to not answer Pings anymore, this would interfere with my network uplink. Not good. Internetconnections should not rely on non-isp infrastructure.

Secondly, every other technology like DSL, Cable or Fiber is able to determine on it's own, if a link is working or not. And if not, the connection is automatically re-established. The current state of OpenWrt is, that packets are dropped, because the 2G/3G/4G/5G connection ist not being re-established by default, although it is easily possible, because of this commit.

1 Like

Yes. And more and more OpenWrt users are adopting 4G/LTE and so the importance of this is increasing every day.

Yes, and the necessary implementation is known. As the maintainer of ModemManager @aleksander0m wrote after detailed discussion with @jow:

The way to solve this, after briefly talking with @jow- about this in IRC would be to have the netifd protocol handler launch a "watcher" process which brings up the connection, and is kept alive and running for as long as the network interface is assumed connected. If MM detects a network-initiated disconnection, the dispatcher script called by MM should kill that watcher process. At that point, netifd (if configured to autoconnect) will then kill the netdev, restart that process and await proto updates.

Anyone up to the task of writing this logic in the netfid modemmanager protocol handler?

And most recently at:

But since now no netifd fix has been implemented in 22.03 since the release of 23.05, I suppose someone should raise the same issue against 23.05.

No. In case, modem hangs, or data volume exhausted, interface looks like being usable. But its not. Using ping, catches many more issues, compared to simply rely on interface state.
End-to-end connectivity is the important issue, usually. And can be more or less reliably checked using ping.

Look: netifd now already receives a report that the connection went down. It's entirely reasonable to expect netifd to bring that connection back up. The maintainer of ModemManager expects this. And I think it's a very legitimate expectation for users too.

My temporary workaround hack still works well to this day:

root@OpenWrt:~# cat /usr/lib/ModemManager/connection.d/10-report-down-and-reconnect
#!/bin/sh

# Automatically report to netifd that the underlying modem
# is really disconnected and reconnect if interface was up

# require program name and at least 4 arguments
[ $# -lt 4 ] && exit 1

MODEM_PATH="$1"
BEARER_PATH="$2"
INTERFACE="$3"
STATE="$4"

[ "${STATE}" = "disconnected" ] || exit 0

. /usr/share/ModemManager/modemmanager.common
. /lib/netifd/netifd-proto.sh
INCLUDE_ONLY=1 . /lib/netifd/proto/modemmanager.sh

MODEM_STATUS=$(mmcli --modem="${MODEM_PATH}" --output-keyvalue)
[ -n "${MODEM_STATUS}" ] || exit 1

MODEM_DEVICE=$(modemmanager_get_field "${MODEM_STATUS}" "modem.generic.device")
[ -n "${MODEM_DEVICE}" ] || exit 2

CFG=$(mm_get_modem_config "${MODEM_DEVICE}")
[ -n "${CFG}" ] || exit 3

IFUP=$(ifstatus ${CFG} | jsonfilter -e '@.up')

logger -t "modemmanager" "interface ${CFG} (network device ${INTERFACE}) ${STATE}"
proto_init_update $INTERFACE 0
proto_send_update $CFG

[ "${IFUP}" = "true" ] && ifup ${CFG}

exit 0

But this is not in keeping with the approach that @aleksander0m and @jow agreed should be implemented.

Well it looks reasonably clean and if it is respecting the user choice (e.g. no spurious reconnecting after a user issued a “ifdown wwan”) then I guess it could be committed as-is.

Just need to ensure that during such a reconnect event no settings are lost due to the proto update call missing options otherwise implemented by the proto handlers setup routine. Ideally the bringup logic and the reconnect update logic should share as much code as possible.

Well thanks. I mean I've used it for many months now and it for sure keeps the connection active on my NR7101. It does seem to me simpler than the approach that you discussed with @aleksander0m, but @aleksander0m seemed certain that it was not the responsibility of ModemManager to keep the connection up, and I thought you were convinced of that too before?

A question I have, and maybe @bmork could help with this, is whether the reconnect using ModemManager with modified dispatcher script (original introduced with this commit) below can be sped up at all? I think it takes about 2-4 seconds at the moment with this approach. I remember reconnections on my Huawei B818-263 were slightly quicker (maybe one second or so).

Can you see anything with this approach:

root@OpenWrt:~# cat /usr/lib/ModemManager/connection.d/10-report-down-and-reconnect
#!/bin/sh

# Automatically report to netifd that the underlying modem
# is really disconnected and reconnect if interface was up

# require program name and at least 4 arguments
[ $# -lt 4 ] && exit 1

MODEM_PATH="$1"
BEARER_PATH="$2"
INTERFACE="$3"
STATE="$4"

[ "${STATE}" = "disconnected" ] || exit 0

. /usr/share/ModemManager/modemmanager.common
. /lib/netifd/netifd-proto.sh
INCLUDE_ONLY=1 . /lib/netifd/proto/modemmanager.sh

MODEM_STATUS=$(mmcli --modem="${MODEM_PATH}" --output-keyvalue)
[ -n "${MODEM_STATUS}" ] || exit 1

MODEM_DEVICE=$(modemmanager_get_field "${MODEM_STATUS}" "modem.generic.device")
[ -n "${MODEM_DEVICE}" ] || exit 2

CFG=$(mm_get_modem_config "${MODEM_DEVICE}")
[ -n "${CFG}" ] || exit 3

IFUP=$(ifstatus ${CFG} | jsonfilter -e '@.up')

logger -t "modemmanager" "interface ${CFG} (network device ${INTERFACE}) ${STATE}"
proto_init_update $INTERFACE 0
proto_send_update $CFG

[ "${IFUP}" = "true" ] && ifup ${CFG}

exit 0

that might give an unnecessary delay?

Striving for speedy reconnection seems desirable to avoid unnecessary downtime.

1 Like

In case, modem hangs, or data volume exhausted, interface looks like being usable.

All the same could happen with other technologys like DSL, Cable or Fiber too. Still, the usual CPE including OpenWrt is perfectly able to manage these "challenges" without depending on external infrastructure.

Ping solutions like Watchcat are very usefull for Path Monitoring in case you use dynamic routing, for example through a VPN tunnel. But for the general internet uplink, i expect it to work reliably without external dependencies.

2 Likes

What do you think of this being implemented in ModemManager using modified dispatcher script as above vs the implementation @aleksander0m wants as discussed earlier:

The way to solve this, after briefly talking with @jow- about this in IRC would be to have the netifd protocol handler launch a "watcher" process which brings up the connection, and is kept alive and running for as long as the network interface is assumed connected. If MM detects a network-initiated disconnection, the dispatcher script called by MM should kill that watcher process. At that point, netifd (if configured to autoconnect) will then kill the netdev, restart that process and await proto updates.

Anyone up to the task of writing this logic in the netfid modemmanager protocol handler?

I agree with @aleksander0m, that MM is not in charge of any reconnection logic. Because of that, i prefer the solution that has been discussed by @jow and @aleksander0m earlier in IRC.

1 Like

It is also pertinent to consider that my 'ifup' hack gives a slow reconnect. See helpful analysis here:

It is desirable to figure out how to improve the reconnect time.

Any ideas?

Try experimenting, perhaps you can skip the teardown step with up+renew, or maybe down+up works faster than ifup:

for ACTION in down up
do ubus call network.interface \
"${ACTION}" "{'interface': '${CFG}'}"
done

You can also release/renew like this:

killall -SIGUSR2 udhcpc
killall -SIGUSR1 udhcpc

https://udhcp.busybox.net/README.udhcpc

1 Like
root@OpenWrt:~# ubus call network.interface up "{'interface': '${CFG}'}"
Command failed: Not found
root@OpenWrt:~# ubus call network.interface renew "{'interface': '${CFG}'}"
Command failed: Not found

OK yes understood, but trying now either in script or manual invocation with 'wan' doesn't work. Nothing in logread or console.

'ifup wan' works, but:

ubus call network.interface up "{'interface': 'wan'}"
ubus call network.interface renew "{'interface': 'wan'}"

seems to do nothing?

What are we missing?

Yes - setting 'ifdown wan' then running:

ubus call network.interface up "{'interface': 'wan'}"
ubus call network.interface renew "{'interface': 'wan'}"

works.

So what does this mean in terms of improving upon simply just 'ifup wan' (which is slow)?

Works but takes 5 seconds still. So no different to just 'ifup wan'. It seems down and then up is necessary for reasons I don't understand.

The mmcli bearer connect is quick but doesn't deal with the wan IP refresh.

mmcli --modem="${MODEM_PATH}" -b "${BEARER_PATH}" -c &&  exit 0
1 Like