Inconsistent USB enumeration (PC Engines APU)

So, I am being hit by this exact bug

Last time the conversation petered out without getting very far, I'm not sure if other platforms might suffer the same problem but because of this a box intended as a stand-alone "wi-fi hotspot" randomly has no 4G WAN connection because every couple of boots the modem switches places in the USB tree. Because the path is hard coded in /etc/config/network the system can't find the modem when bringing up the interface.

ModemManager can find the modem and correctly list the /sys/devices path so presumably it should be possible to use mmcli at the right point in the startup and either edit /etc/config/network on the fly - or use uci to do the same thing.

But how/where would I go about doing that?

Please show the output of cat /sys/kernel/debug/usb/devices

Sure, though I only have time for a short reply at the moment.

The details are all in the previous thread anyway, the situation is identical and the problem is caused by asynchronous enumeration of the ehci and xhci interfaces which means that sometimes the modem shows up on USB bus 2 and sometimes on USB bus 4

The bug, arguably, is not the USB enumeration - there was a single change in kernel 6.6 which caused it BTW, again as described in the previous thread - but that we're putting a hard coded device path into /etc/config/network.

It's possible that just the PC engines APU is the only platform affected - this is a legacy platform (no new stock has been available for > 12 months owing to AMD discontinuing the SOC it is built around in 2023) so a "quick fix" is almost certainly appropriate.

OK, here you go.

Dumping /sys/kernel/debug/usb/devices gives this output on one boot and this output the next.

For this pair of reboots the modem shows up as either

T:  Bus=01 Lev=02 Prnt=02 Port=02 Cnt=01 Dev#=  3 Spd=480  MxCh= 0
D:  Ver= 2.00 Cls=ef(misc ) Sub=02 Prot=01 MxPS=64 #Cfgs=  1
P:  Vendor=2c7c ProdID=0306 Rev= 3.10
S:  Manufacturer=Quectel
S:  Product=EP06-E

The first time through

Or

T:  Bus=04 Lev=02 Prnt=02 Port=02 Cnt=01 Dev#=  3 Spd=480  MxCh= 0
D:  Ver= 2.00 Cls=ef(misc ) Sub=02 Prot=01 MxPS=64 #Cfgs=  1
P:  Vendor=2c7c ProdID=0306 Rev= 3.10
S:  Manufacturer=Quectel
S:  Product=EP06-E

The second time through.

So it gets a different path under /sys/devices each time - either

/sys/devices/pci0000:00/0000:00:13.0/usb1/1-1/1-1.3

Or

/sys/devices/pci0000:00/0000:00:13.0/usb4/4-1/4-1.3

Paths under usb2 and usb3 also crop up.

I think I have a few "quick fix" options:

  • I can revert to OpenWRT 23.05.4
  • I can do a custom build with the kernel change reverted
  • I can switch to using the QMI driver directly
  • I can add a script to fix the path in /etc/config/network during early boot.

The first option is not great as it would lock me to that version with no updates but is probably the one I am going for this weekend as my wife wants to take the box away to use as a personal hotspot - it has an unlimited SIM which the phones do not.

I don't mind doing custom OpenWRT builds but it gets to be a bit of a pain tracking new versions or adding packages.

If I can make it work using the QMI option is actually quite attractive because it even gave an IPv6 address. It only worked once though and did not bring the interface up on reboot so I need to figure out if that was simply something I did wrong or if it is broken.

All of which leaves the last option (if I can't get QMI working) - it's a really simple script:

uci set network.wwan0.device=$(mmcli -m 0 | grep device:|cut -c39-); uci commit

I just need to know where I can put it so that it is after the modem has been initialised and before the interface is brought up.

You don't need to worry about enumeration at all.

Get rid of ModemManager completely, switch the modem to MBIM and configure the interface in a standard way:

config interface 'lte'
        option proto 'mbim'
        option device '/dev/cdc-wdm0'
        option apn 'your APN'
        option auth 'none'
        option pdptype 'ipv4v6'

Make sure you have installed kmod-usb-serial-option and luci-proto-mbim with dependencies.

OK I'll give that a go when my wife is back from her trip. Or if I get time I have a spare system and an EC25.

Not particularly wedded to modem manager but I thought it was the recommended interface and it has been the most reliable for me generally. Also I'm a bit wary of fiddling with modem modes as I have a couple of small bricks from getting that wrong.

Exactly the same is applicable to EC25 as well. I recommend updating the modem firmware to the latest available version, if you're not in a sanctioned country, of course.

That usually requires a windows machine with the device installed though doesn't it. Plus actually obtaining the firmware.

I don't do WIndows unless I absolutely have to.

Please use search.

OK

To get the source you have to register with Quectel - I've tried that more than once and been roundly ignored each time.

There are some copies knocking around but only something like 1.4.17 vs 1.7 as the current version.

If I do look at the EC25 I'll see what firmware it's on and try asking on the Quectel forum - I can see some requests are answered positively but not all.

I definitely will look at MBIM (and QMI) as better and more sustainable ways to configure the modem but for now the idea of selecting the right device path in early boot works.

Not using mmcli though - ModemManager doesn't communicate with the modem nearly early enough in the boot process - I need something before the network is brought up.

Noting that the modem always shows under /sys/devices as /sys/devices/pci0000:00/0000:00:13.0/usbX/X-1/X-1.3 I simply added

uci set network.wwan0.device=$(echo /sys/devices/pci0000:00/0000:00:13.0/usb?/?-1/?-1.3)
uci commit

towards the end of the boot() function in /etc/init.d/boot

Yes, it's a complete hack and definitely not robust enough for production but appears to work well enough in practice (for me, YMMV of course).

Having now had time to try various configurations - yes this works and in some ways I should apologise for not marking this as the solution to my query.

But it is not unambiguously better than fixing the device path for ModemManager - at least not so far.

I lose the cell status page for one thing and can't find a good replacement (at least not one which is pre-packaged). Also using tools like mbimcli seems to be a one-shot deal. I can query various info such as the signal state but that kills the connection until I do an ifdown/ifup.

Also you didn't really answer my point about which modem interface we *should* be using - I guess there's a lot to be said "for the one that works" but (as I said) I thought mm was the preferred "modern" interface. Also I have an R11e-LTE6 (slightly bought by mistake) which only works with ModemManager (for a value of "works" which means ppp over the serial interface - I know there's a long thread on that device which I need to catch up on).

Finally the more I think about it the more I think this is going to bite any device which uses ModemManager and has both ehic and xhic if they are probed asynchronously post kernel-6.6.

I'm sort of happy, I have a couple of alternative ways of addressing the problem. None is perfect but at least I can set things up so the modem is found each time. I'll check that they work OK with the EP06 when I get it back from its field trip but I don't see why not. In any case I don't want to spend loads of time on 4G modems - I have some 5Gmodems that I want to try to get up and running.