4G Router randomly go Offline

Hello Everyone, I'm having an issue with a Raspberry Pi CM4 router (using Waveshare dual eth 4G/5G board) running the latest OpenWrt for CM4 (23.5.3) with a 4G LTE module (SIM7600G-H). These are used at remote locations for IoT as a 4G router.

The router works great most of the time; it seems stable and fast for our needs. However, it can randomly go offline for no apparent reason. Sometimes it will work for several months (2-3 months) and sometimes for only several days (10-15 days). We use a VPN for remote management and connection to our server, so we can see when it is offline (confirmed by the user at the site).

A reboot always fixes the issue right away. For this reason, we have set up WatchCat to restart the 4G interface if it fails to ping 8.8.8.8 for 30 minutes. While this helps, it can take several hours for the router to come back online (we might switch to a full ping reboot to be safe). I'm sure that a ping reboot is not the solution, but if it can help for now...

Has anyone had this issue or something similar and managed to find a solution?

Please post output of

ubus call system board

Here is the output:

{
        "kernel": "5.15.150",
        "hostname": " *custom name* ",
        "system": "ARMv8 Processor rev 3",
        "model": "Raspberry Pi Compute Module 4 Rev 1.1",
        "board_name": "raspberrypi,4-compute-module",
        "rootfs_type": "ext4",
        "release": {
                "distribution": "OpenWrt",
                "version": "23.05.3",
                "revision": "r23809-234f1a2efa",
                "target": "bcm27xx/bcm2711",
                "description": "OpenWrt 23.05.3 r23809-234f1a2efa"
        }
}

You should stop rebooting and collecting log events leading to "randomly offline"
And inquire operator regarding service windows of 30 minutes near location.

2 Likes

In my experience, 4G connections are not stable by default, and need some support by external tools. ModemManager is very good at recovering the connection when it goes down, but I also use watchcat for the cases when traffic stops flowing.

1 Like

Thanks for the advice. I have set a remote log server, but nothing really appeared to cause an issue. After checking the system log on the device (once rebooted) nothing unusual as well. Unfortunately i could not see the kernel log from the previous boot.

After reading on the forum i'm thinking it could be my provider terminating the 4G connection but i will have to check.

You need to wait for disconnect and collect offline logs, like on(remote)site.

I recently started using RM500Q-GL and found that random dropouts could be the norm for cellular services. ModemManager + 1.

We use ModemManager; it is quite stable, but it seems that auto-connect was not the norm until quite recently, according to this post:

We are using ModemManager 1.20.6-12, which seems to be the latest version available for OpenWrt 23.5.3 at the moment, and the auto-reconnect feature was merged in ModemManager 1.22.

It is not that easy to do, as it is in a remote location and the users do not have direct access to the router (physical and network).

I will try to install another 4G router as a WAN backup. If one fails, hopefully, the second one will work and allow me to check if it is indeed the 4G modem connection or something else.

Did you have any issue or déconnection? What version of ModemManager are you using?

if you have watchcat, can't you have it dump the logs to physical storage before it does the reboot ?

2 Likes

I have not thought of that, Thank you it is a good idea. I will create a script to dump the logs to emmc before the reboot.

1 Like

Your ISP could be dropping connections, when they feel that they are stale or too old.

2 Likes

Or when reconfiguring towers...

3 Likes

There is no need to reboot. Please see TL-MR6400 V4 - QMI Not working - #14 by AndrewZ

I wouldn't call it an issue since it seems to be provider-initiated disconnections. ModemManager is the latest 1.22.0-r16 in the snapshot.

Don't know if it's normal, but I get a lot of these:

Tue Jul  9 17:58:37 2024 daemon.warn [2723]: <wrn> [modem0] unexpected NR5G system info: domain valid yes, domain unknown, service status available
Tue Jul  9 17:58:37 2024 daemon.notice [2723]: <msg> [modem0] 3GPP registration state changed (home -> searching)
Tue Jul  9 17:58:37 2024 daemon.info [2723]: <inf> [modem0] consolidated registration state: cs 'searching', ps 'searching', eps 'searching', 5gs 'unknown' --> 'searching'
Tue Jul  9 17:58:37 2024 daemon.info [2723]: <inf> [modem0] access technology changed (5gnr -> unknown)
Tue Jul  9 17:58:37 2024 daemon.notice [2723]: <msg> [modem0] 3GPP registration state changed (searching -> registering)
Tue Jul  9 17:58:37 2024 daemon.info [2723]: <inf> [modem0] access technology changed (unknown -> 5gnr)
Tue Jul  9 17:58:37 2024 daemon.notice [2723]: <msg> [modem0] 3GPP registration state changed (registering -> home)
Tue Jul  9 17:58:37 2024 daemon.info [2723]: <inf> [modem0] consolidated registration state: cs 'searching', ps 'searching', eps 'searching', 5gs 'home' --> 'home'

The last disconnection happened like this:

Tue Jul  9 23:27:04 2024 daemon.warn [2723]: <wrn> [modem0] unexpected NR5G system info: domain valid yes, domain unknown, service status available
Tue Jul  9 23:27:04 2024 daemon.notice [2723]: <msg> [modem0] 3GPP registration state changed (home -> searching)
Tue Jul  9 23:27:04 2024 daemon.info [2723]: <inf> [modem0] consolidated registration state: cs 'searching', ps 'searching', eps 'searching', 5gs 'unknown' --> 'searching'
Tue Jul  9 23:27:04 2024 daemon.info [2723]: <inf> [modem0] access technology changed (5gnr -> unknown)
Tue Jul  9 23:27:04 2024 daemon.notice [2723]: <msg> [modem0] 3GPP registration state changed (searching -> registering)
Tue Jul  9 23:27:04 2024 daemon.info [2723]: <inf> [modem0] access technology changed (unknown -> 5gnr)
Tue Jul  9 23:27:04 2024 daemon.notice [2723]: <msg> [modem0] 3GPP registration state changed (registering -> home)
Tue Jul  9 23:27:04 2024 daemon.info [2723]: <inf> [modem0] consolidated registration state: cs 'searching', ps 'searching', eps 'searching', 5gs 'home' --> 'home'
Tue Jul  9 23:27:04 2024 daemon.warn [2723]: <wrn> [modem0] network reject indication received
Tue Jul  9 23:27:04 2024 daemon.warn [2723]: <wrn> [modem0]   service domain: ps
Tue Jul  9 23:27:04 2024 daemon.warn [2723]: <wrn> [modem0]   radio interface: 5gnr
Tue Jul  9 23:27:04 2024 daemon.warn [2723]: <wrn> [modem0]   reject cause: implicitly-detached
Tue Jul  9 23:27:04 2024 daemon.warn [2723]: <wrn> [modem0]   mcc: 460
Tue Jul  9 23:27:04 2024 daemon.warn [2723]: <wrn> [modem0]   mnc: 1
Tue Jul  9 23:27:04 2024 daemon.warn [2723]: <wrn> [modem0]   closed subscriber group: 1682425728
Tue Jul  9 23:27:04 2024 daemon.notice [2723]: <msg> [modem0/bearer0] verbose call end reason (3,1138): [cm] (null)
Tue Jul  9 23:27:04 2024 daemon.notice [2723]: <msg> [modem0] state changed (connected -> registered)
Tue Jul  9 23:27:04 2024 daemon.notice [2723]: <msg> [modem0/bearer0] connection #1 finished: duration 41308s, tx: 6949837 bytes, rx: 4276141 bytes
Tue Jul  9 23:27:04 2024 daemon.info ModemManager[10344]: hotplug: Reconnecting 'wwan' on 'disconnected' event
Tue Jul  9 23:27:04 2024 daemon.notice netifd: wwan (10380): stopping network
Tue Jul  9 23:27:04 2024 daemon.info [2723]: <inf> [modem0] processing user request to disconnect modem: all bearers
Tue Jul  9 23:27:04 2024 daemon.info [2723]: <inf> [modem0] all requested bearers disconnected
Tue Jul  9 23:27:04 2024 daemon.notice netifd: wwan (10380): successfully disconnected all bearers in the modem
Tue Jul  9 23:27:04 2024 daemon.info [2723]: <inf> [modem0] processing user request to disable modem...
Tue Jul  9 23:27:04 2024 daemon.notice [2723]: <msg> [modem0] state changed (registered -> disabling)
Tue Jul  9 23:27:04 2024 daemon.info [2723]: <inf> [modem0] access technology changed (5gnr -> unknown)
Tue Jul  9 23:27:04 2024 daemon.notice [2723]: <msg> [modem0] 3GPP packet service state changed (detached -> unknown)
Tue Jul  9 23:27:04 2024 daemon.notice [2723]: <msg> [modem0] 3GPP registration state changed (home -> unknown)
Tue Jul  9 23:27:04 2024 daemon.info [2723]: <inf> [modem0] consolidated registration state: cs 'searching', ps 'searching', eps 'searching', 5gs 'home' --> 'unknown'
Tue Jul  9 23:27:04 2024 daemon.notice [2723]: <msg> [modem0] 3GPP packet service state changed (unknown -> detached)
Tue Jul  9 23:27:04 2024 daemon.info [2723]: <inf> [modem0] disabled modem
Tue Jul  9 23:27:04 2024 daemon.notice netifd: wwan (10380): successfully disabled the modem
Tue Jul  9 23:27:04 2024 daemon.notice [2723]: <msg> [modem0] state changed (disabling -> disabled)
Tue Jul  9 23:27:04 2024 daemon.notice netifd: Interface 'wwan' is now down

Then ModemManager immediately redials.

As for reject cause: implicitly-detached, I find packet service state: detached from time to time, but the connection stayed up.