FM350-GL T-Mobile (US) No CLAT/XLAT/IPv4 address provided

[cr08]

Much of my existing work and documentation has been done under the following thread: Fibocom FM350-GL Support - #195 by mrhaav just in case anyone wants to go back and look. But the CLAT/XLAT discussion will continue here.

After doing some further testing to rule things out, here's the gist of what I am running into:

I currently have a T-Mobile (US) Prepaid data only SIM and intend to use it with my FM350-GL. Using the available packages to interact with it both in PCI and USB mode I can get a working connection and routable V6 addresses all day long but under SA mode it won't provide a V4 address. If I set it to do NSA only, it's more reliable but in some cases I still don't get an address.

In comparison, I have a couple Quectel EC25-AF LTE only cards and those I get a V4 address out of the gate no issues. I do have an RM520N-GL coming in that I'm also going to test with to see if maybe this issue is specific to the FM350.

From the research I've done, it seems at least on consumer lines like mine with T-Mobile they do V6 only and rely on CLAT for V4. However adding the 464xlat package to facilitate this, it always automatically props up a virtual interface for this but in the logs it constantly cycles up/down with no progress.

I originally had a much longer wall of text below, but I've read more into how CLAT works and through some tinkering I think I've narrowed down the culprit. As @AndrewZ mentioned in the other thread, it seems to be DNS related.

By default it appears T-Mobile is providing the two following V6 DNS addresses:

nameserver fd00:976a::9
nameserver fd00:976a::10

I also see this on my Pixel 6a connected to 5G. However if I try to do a lookup from OWRT, it is unable to reach their DNS servers. My phone can, however.

root@OpenWrt:~# nslookup -debug -type=aaaa ipv4only.arpa fd00:976a::9
nslookup: can't connect to remote host: Network unreachable
root@OpenWrt:~# ping -6 fd00:976a::9
PING fd00:976a::9 (fd00:976a::9): 56 data bytes
ping: sendto: Network unreachable
root@OpenWrt:~# nslookup -debug -type=aaaa ipv4only.arpa fd00:976a::10
nslookup: can't connect to remote host: Network unreachable
root@OpenWrt:~# ping -6 fd00:976a::10
PING fd00:976a::10 (fd00:976a::10): 56 data bytes
ping: sendto: Network unreachable

But what I found as a workaround is that I'm able to provide Google's DNS64 addresses through the main interface and after that the automatic 464 virtual interface comes up properly and V4 connections are routable.

config interface 'test'
        option proto 'atc'
        option device '/dev/ttyUSB1'
        option apn 'fast.t-mobile.com'
        option auth '0'
        option pdp 'IPV4V6'
        option atc_debug '2'
        list dns '2001:4860:4860::64'
        list dns '2001:4860:4860::6464'

root@OpenWrt:~# nslookup -type=aaaa ipv4only.arpa '2001:4860:4860::64'
Server:         2001:4860:4860::64
Address:        [2001:4860:4860::64]:53

Non-authoritative answer:
Name:   ipv4only.arpa
Address: 64:ff9b::c000:ab
Name:   ipv4only.arpa
Address: 64:ff9b::c000:aa

root@OpenWrt:~# ping -4 google.com
PING google.com (142.250.190.142): 56 data bytes
64 bytes from 142.250.190.142: seq=0 ttl=52 time=41.228 ms
64 bytes from 142.250.190.142: seq=1 ttl=52 time=58.225 ms
^C
--- google.com ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 41.228/49.726/58.225 ms
root@OpenWrt:~# traceroute -4 google.com
traceroute to google.com (142.250.190.142), 30 hops max, 46 byte packets
 1  192.0.0.1 (192.0.0.1)  0.051 ms  0.052 ms  0.051 ms
 2  192.0.0.1 (192.0.0.1)  16.082 ms  24.253 ms  28.196 ms
 3  192.0.0.1 (192.0.0.1)  16.399 ms  17.223 ms  23.710 ms
 4  192.0.0.1 (192.0.0.1)  12.724 ms  24.277 ms  19.023 ms
 5  192.0.0.1 (192.0.0.1)  12.730 ms  35.986 ms  19.147 ms
 6  10.160.107.42 (10.160.107.42)  11.733 ms  17.317 ms  10.343 ms
 7  10.177.56.16 (10.177.56.16)  23.561 ms  13.178 ms  23.819 ms
 8  10.177.8.25 (10.177.8.25)  29.648 ms  42.738 ms  24.413 ms
 9  10.177.9.69 (10.177.9.69)  29.268 ms  27.878 ms  19.274 ms
10^C
root@OpenWrt:~#

I'm not sure if this will induce any performance penalties going through Google's gateway vs T-Mobile's. If there's any recommended tests or potential fixes to getting T's DNS to respond, that'd be awesome. My OWRT install I'm currently testing with here is basically a stock out of the box config with the WAN/LAN zones and default firewall rules already preconfigured.

[AndrewZ]

You're using just Google DNS, not a gateway, your traffic still goes to TM's NAT64 server. I don't see an issue with that.
Talking about TM's own DNS reachability - I think that could be related to the firewall and/or routing setup within the modem. I have reasons to believe that the same test on the modem itself will pass. Having adb installed on the router you can try adb shell nslookup ipv4only.arpa.
BTW, what is the firmware version on your modem?

You can partially solve the problem by setting Pref64 statically [in 464xlat interface configuration], but you will still need a working DNS64 server.

[cr08]

Here's the full firmware string it reports: 81600.0000.00.29.23.06_5000.0002.019.602.015_F09

I had previously updated the firmware on this card (the lack of a native V6/CLAT address was already occurring before that). Followed the instructions to do so via USB that has been documented on the 4PDA thread for this card: Fibocom FM350-GL – Discussion - 4PDA

This is a Dell variant DW5931e so I grabbed the latest driver I could find that they had to pull the firmware files from, this one dated March 2024: Dell Wireless 5931e and Intel FM350 Firmware and GNSS Driver | Driver Details | Dell US

The firmware update went successfully and other than power related issues I've been dealing with and this IPV4 issue, it has been operating fine barring the other already known quirks about this card.

As far as running ADB, that's where I'm currently running into an issue as it is showing it as being offline and unable to be interacted with. I even switched between the 40/41 USB modes just on a whim even though ADB is available on both.

root@OpenWrt:~# adb devices
List of devices attached
????????????    offline

Tried this in 3 different devices: The Unielec U7621-06 MT7621 board I'm playing with now running OpenWRT snapshots, an X86 laptop running 23.05.3, and my Windows laptop. All show the same. Weird because I have been able to use ADB previously just fine for this card.

As for the prefix and such, that's actually something I tried previously before I read up more on how the whole CLAT setup works and how it auto discovers the prefix on its own.

Originally I tried Cloudflare's V6 DNS addresses combined with this prefix I found on an ancient XDA thread: 2607:7700:0:a::/96 . It worked once or twice but was intermittent.

Ultimately it ended up with the workaround solution I posted above where all I needed was to supply Google's own DNS in the primary interface and the automatic 464xlat interface brings up a connection without needing to manually supply a prefix. That has been reliable so far, knock on wood. And as you said, since this is just DNS and not a gateway as I originally thought, it's not as big of a concern as I originally anticipated. And doing some mtr passes on some key addresses, latency/jitter doesn't seem to differ appreciably between V4 and V6.

I'm still up for doing some more diagnosing though to figure out why the card isn't doing this natively or isn't allowing routing to T-Mobile's own DNS. At the very least get some documentation out there. This ADB device offline issue is creating a bit of a block though.

[mk24]

Try setting sourcefilter '0' on wan6. By default (sourcefilter=1) it won't try to route ULAs to the Internet.

Also you could try using the modem's link-local as DNS, which if the modem is running a DNS service it will forward to the ISP.

[AndrewZ]

@cr08

As far as running ADB, that's where I'm currently running into an issue as it is showing it as being offline and unable to be interacted with.

Is your adb interface occupied by a serial driver by chance? See in the main thread:

Fibocom FM350-GL Support

Thank you for the information. I suggest using option driver, echo "0e8d 7127 ff" > /sys/bus/usb-serial/drivers/option1/new_id Your AT command port should be If#= 6, in the absence of other tty's it will be mapped to ttyUSB4, but with properly patched driver it should be ttyUSB3, as currently you have your ADB interface (If#= 5) occupied by the serial driver as well. Typically in such setup the gateway IP is not needed, as well as the correct netmask. You can use a device like wwan0 as your …

[cr08]

Yeah, I'm pretty sure that's the case. The current packages being used to manage this card add a hotplug file to bring up all the interfaces on this card under the serial driver (I think for @mrhaav's atc implementation it uses the option driver by default).

root@OpenWrt:~# lsusb -t
/:  Bus 001.Port 001: Dev 001, Class=root_hub, Driver=xhci-mtk/2p, 480M
    |__ Port 001: Dev 002, If 0, Class=[unknown], Driver=rt2800usb, 480M
    |__ Port 002: Dev 006, If 0, Class=[unknown], Driver=rndis_host, 480M
    |__ Port 002: Dev 006, If 1, Class=[unknown], Driver=rndis_host, 480M
    |__ Port 002: Dev 006, If 2, Class=[unknown], Driver=option, 480M
    |__ Port 002: Dev 006, If 3, Class=[unknown], Driver=option, 480M
    |__ Port 002: Dev 006, If 4, Class=[unknown], Driver=option, 480M
    |__ Port 002: Dev 006, If 5, Class=[unknown], Driver=usbfs, 480M
    |__ Port 002: Dev 006, If 6, Class=[unknown], Driver=option, 480M
    |__ Port 002: Dev 006, If 7, Class=[unknown], Driver=option, 480M
    |__ Port 002: Dev 006, If 8, Class=[unknown], Driver=option, 480M
    |__ Port 002: Dev 006, If 9, Class=[unknown], Driver=option, 480M

I've been looking into seeing how to get it to not do so for the adb interface. I'm not terribly familiar with the syntax.

[mrhaav]

Yes, I have a hotplug script that add echo '0e8d 7127' > /sys/bus/usb-serial/drivers/option1/new_id.
Even if I add echo '0e8d 7127 ff' > /sys/bus/usb-serial/drivers/option1/new_id, I get the same result.
But, how shall I do to avoid assign option-driver to the adb interface and which driver shall be used?

This is how my modem looks:

root@OpenWrt:~# lsusb
Bus 001 Device 001: ID 1d6b:0002 Linux 6.6.35 xhci-hcd xHCI Host Controller
Bus 002 Device 001: ID 1d6b:0003 Linux 6.6.35 xhci-hcd xHCI Host Controller
Bus 002 Device 002: ID 0e8d:7127 Fibocom Wireless Inc. FM350-GL
root@OpenWrt:~# lsusb -t
/:  Bus 001.Port 001: Dev 001, Class=root_hub, Driver=xhci-mtk/2p, 480M
/:  Bus 002.Port 001: Dev 001, Class=root_hub, Driver=xhci-mtk/1p, 20000M/x2
    |__ Port 001: Dev 002, If 0, Class=[unknown], Driver=rndis_host, 5000M/x2
    |__ Port 001: Dev 002, If 1, Class=[unknown], Driver=rndis_host, 5000M/x2
    |__ Port 001: Dev 002, If 2, Class=[unknown], Driver=option, 5000M/x2
    |__ Port 001: Dev 002, If 3, Class=[unknown], Driver=option, 5000M/x2
    |__ Port 001: Dev 002, If 4, Class=[unknown], Driver=option, 5000M/x2
    |__ Port 001: Dev 002, If 5, Class=[unknown], Driver=option, 5000M/x2
    |__ Port 001: Dev 002, If 6, Class=[unknown], Driver=option, 5000M/x2
    |__ Port 001: Dev 002, If 7, Class=[unknown], Driver=option, 5000M/x2
    |__ Port 001: Dev 002, If 8, Class=[unknown], Driver=option, 5000M/x2
    |__ Port 001: Dev 002, If 9, Class=[unknown], Driver=option, 5000M/x2


T:  Bus=02 Lev=01 Prnt=01 Port=00 Cnt=01 Dev#=  2 Spd=5000 MxCh= 0
D:  Ver= 3.00 Cls=ef(misc ) Sub=02 Prot=01 MxPS= 9 #Cfgs=  1
P:  Vendor=0e8d ProdID=7127 Rev= 0.01
S:  Manufacturer=Fibocom Wireless Inc.
S:  Product=FM350-GL
C:* #Ifs=10 Cfg#= 1 Atr=a0 MxPwr=896mA
A:  FirstIf#= 0 IfCount= 2 Cls=e0(wlcon) Sub=01 Prot=03
I:* If#= 0 Alt= 0 #EPs= 1 Cls=02(comm.) Sub=02 Prot=ff Driver=rndis_host
E:  Ad=82(I) Atr=03(Int.) MxPS=  64 Ivl=125us
I:* If#= 1 Alt= 0 #EPs= 2 Cls=0a(data ) Sub=00 Prot=00 Driver=rndis_host
E:  Ad=81(I) Atr=02(Bulk) MxPS=1024 Ivl=0ms
E:  Ad=01(O) Atr=02(Bulk) MxPS=1024 Ivl=0ms
I:* If#= 2 Alt= 0 #EPs= 2 Cls=ff(vend.) Sub=00 Prot=00 Driver=option
E:  Ad=83(I) Atr=02(Bulk) MxPS=1024 Ivl=0ms
E:  Ad=02(O) Atr=02(Bulk) MxPS=1024 Ivl=0ms
I:* If#= 3 Alt= 0 #EPs= 2 Cls=ff(vend.) Sub=00 Prot=00 Driver=option
E:  Ad=84(I) Atr=02(Bulk) MxPS=1024 Ivl=0ms
E:  Ad=03(O) Atr=02(Bulk) MxPS=1024 Ivl=0ms
I:* If#= 4 Alt= 0 #EPs= 2 Cls=ff(vend.) Sub=00 Prot=00 Driver=option
E:  Ad=85(I) Atr=02(Bulk) MxPS=1024 Ivl=0ms
E:  Ad=04(O) Atr=02(Bulk) MxPS=1024 Ivl=0ms
I:* If#= 5 Alt= 0 #EPs= 2 Cls=ff(vend.) Sub=42 Prot=01 Driver=option
E:  Ad=86(I) Atr=02(Bulk) MxPS=1024 Ivl=0ms
E:  Ad=05(O) Atr=02(Bulk) MxPS=1024 Ivl=0ms
I:* If#= 6 Alt= 0 #EPs= 2 Cls=ff(vend.) Sub=00 Prot=00 Driver=option
E:  Ad=87(I) Atr=02(Bulk) MxPS=1024 Ivl=0ms
E:  Ad=06(O) Atr=02(Bulk) MxPS=1024 Ivl=0ms
I:* If#= 7 Alt= 0 #EPs= 2 Cls=ff(vend.) Sub=00 Prot=00 Driver=option
E:  Ad=88(I) Atr=02(Bulk) MxPS=1024 Ivl=0ms
E:  Ad=07(O) Atr=02(Bulk) MxPS=1024 Ivl=0ms
I:* If#= 8 Alt= 0 #EPs= 2 Cls=ff(vend.) Sub=00 Prot=00 Driver=option
E:  Ad=89(I) Atr=02(Bulk) MxPS=1024 Ivl=0ms
E:  Ad=08(O) Atr=02(Bulk) MxPS=1024 Ivl=0ms
I:* If#= 9 Alt= 0 #EPs= 2 Cls=ff(vend.) Sub=00 Prot=00 Driver=option
E:  Ad=8a(I) Atr=02(Bulk) MxPS=1024 Ivl=0ms
E:  Ad=09(O) Atr=02(Bulk) MxPS=1024 Ivl=0ms

[AndrewZ]

@mrhaav

how shall I do to avoid assign option-driver to the adb interface and which driver shall be used?

We all need to use a patched option driver. The Linux patch is already submitted and it will appear in OpenWrt eventually. The workaround follows...

[AndrewZ]

It looks like your modem is in 0e8d:7127 configuration, so you have ADB interface as If#= 5.
To unbind the current driver and release the interface you will need to execute something like this:
echo 1-2:1.5 > /sys/bus/usb/drivers/option/unbind

[cr08]

Sorry for the late reply. Been a bit under the weather here the past few weeks so haven't had much time for tinkering.

So I did re-check on the USB side and I was mistaken. It looks like the ADB port is populated by the usbfs driver and not the option driver. Based on what I've dug up this seems like it is correct? But still no change on access.

@mrhaav actually DM'd me on the side regarding some of this and one thing I do want to sit down and work with is getting a tcpdump session going and see what that shows regarding v6 and RA. Seems like that might be very useful as well.

Crossing fingers I may have gotten a somewhat reasonable workaround to getting this card working in PCI mode reliably on my BPI-R64, I should have a better time testing going forward.

I also got a second FM350, same Dell variant. Likely has a different firmware. So I can also compare against that as well.

[AndrewZ]

@cr08

It looks like the ADB port is populated by the usbfs driver and not the option driver.

Correct, I looked into wrong example provided earlier.
If adb fails on the router you can try it on PC.
However, since TM uses the standard Pref64, any external DNS64 server can be used.

[cr08]

Yeah, I think at this point I'm going to stick with that and leave troubleshooting the automatic configuration for another time. A bunch of extra work has been added to the overall mess I've been working with so figuring out why it's not discovering everything automatically is the least of my worries. But if anyone else comes across this and wants to work on it, I'd be happy to give any help I can.

That said, there's a new 464xlat related issue I've run into lately running snapshots on the various devices I've been working with (x86-64 with a desktop Intel Haswell board and the BPI-R64) and that's the fact that I can't get the auto-configured interface to stop/disable. Per the wiki I should be able to add option iface_464xlat '0' to the primary interface to disable it but it keeps appearing. I think I've had luck with this once in the past but now not so much. I should add that the current failure of this option is occurring while using this card in PCI mode and using ModemManager. I haven't attempted USB mode yet.

I really need to get this disabled so I can add the interface manually with the proper prefix/DNS addresses. It's only a hunch but I've been having some weird issues with what seems like (unconfirmed) kernel panics/hard reboots on both devices not too long after a connection is established and the 464 auto-config interface starts its cycle with the T-Mobile SIM. But with my US Mobile/Verizon SIM, which provides a native V4 address, last time I tested it behaved fine and I basically had a speed test running on loop for about an hour with no issues.

Still have a bunch of different troubleshooting steps but getting this auto-config interface disabled is a prerequisite. So if there's any ideas on anything I can try, that'd be awesome. I can get config files and other data later this evening.

[cr08]

So I realized my previous post attempt with the logs/configs didn't make it here but I think it's all for naught at this stage. As a workaround I just removed the 464xlat package and tried testing purely with IPV6 and got the same reliability issues with this card so doesn't seem to be related. So my attention is going elsewhere to sort out stability.

And as previously noted, if I manually add Google's DNS64 addresses to the WAN interface, the automatic virtual 464 interface comes up properly so I'll leave it as that and mark this resolved. But if anyone else comes across this and has any info on disabling the automatic virtual interface (at least if its any different from the guidance/option listed in the wiki), it'd probably be useful to document.