Where do I begin to track down ICMP packets not making up the stack

This is what I've got. Two devices that can't ping each other, yet I clearly see the frames showing they make it to the wwan0 interface.

Like this:
Device .201 ping session to .203

Start ping from sysops .201 to airport .203
Sysops output of ping
root@AWC-EI:~# ping 107.90.50.203
PING 107.90.50.203 (107.90.50.203): 56 data bytes
--- 107.90.50.203 ping statistics ---
5 packets transmitted, 0 packets received, 100% packet loss

tcpdump from .201 - See the echo request and subsequent reply

root@AWC-EI:~# tcpdump -i wwan0 -U -s0 -v -n -l
tcpdump: listening on wwan0, link-type EN10MB (Ethernet), capture size 262144 bytes
11:48:27.635216 IP (tos 0x0, ttl 64, id 23354, offset 0, flags [DF], proto ICMP (1), length 84)
    107.90.50.201 > 107.90.50.203: ICMP echo request, id 21856, seq 0, length 64
11:48:28.235536 IP (tos 0x68, ttl 63, id 13556, offset 0, flags [none], proto ICMP (1), length 84)
    107.90.50.203 > 107.90.50.201: ICMP echo reply, id 21856, seq 0, length 64
11:48:28.643471 IP (tos 0x0, ttl 64, id 23421, offset 0, flags [DF], proto ICMP (1), length 84)
    107.90.50.201 > 107.90.50.203: ICMP echo request, id 21856, seq 1, length 64
11:48:29.273009 IP (tos 0x68, ttl 63, id 13570, offset 0, flags [none], proto ICMP (1), length 84)
    107.90.50.203 > 107.90.50.201: ICMP echo reply, id 21856, seq 1, length 64

tcpdump on .203 - see the echo request and reply

root@AWC-EI:~# tcpdump -i wwan0 -U -s0 -v -n -l
tcpdump: listening on wwan0, link-type EN10MB (Ethernet), capture size 262144 bytes
11:48:27.635216 IP (tos 0x0, ttl 64, id 23354, offset 0, flags [DF], proto ICMP (1), length 84)
    107.90.50.201 > 107.90.50.203: ICMP echo request, id 21856, seq 0, length 64
11:48:28.235536 IP (tos 0x68, ttl 63, id 13556, offset 0, flags [none], proto ICMP (1), length 84)
    107.90.50.203 > 107.90.50.201: ICMP echo reply, id 21856, seq 0, length 64
11:48:28.643471 IP (tos 0x0, ttl 64, id 23421, offset 0, flags [DF], proto ICMP (1), length 84)
    107.90.50.201 > 107.90.50.203: ICMP echo request, id 21856, seq 1, length 64
11:48:29.273009 IP (tos 0x68, ttl 63, id 13570, offset 0, flags [none], proto ICMP (1), length 84)
    107.90.50.203 > 107.90.50.201: ICMP echo reply, id 21856, seq 1, length 64

Where do I start to figure out why .201's ping reports 0 packets received? Same result in the other direction to. Yet I have no issue from either .201 or .203 pinging my two other devices.

Ping from .201 to .206 - works like a champ

root@AWC-EI:~# ping 107.90.50.206
PING 107.90.50.206 (107.90.50.206): 56 data bytes
64 bytes from 107.90.50.206: seq=0 ttl=63 time=439.704 ms
64 bytes from 107.90.50.206: seq=1 ttl=63 time=318.411 ms

Ping from .201 to .205 - also works

root@AWC-EI:~# ping 107.90.50.205
PING 107.90.50.205 (107.90.50.205): 56 data bytes
64 bytes from 107.90.50.205: seq=0 ttl=63 time=1187.012 ms
64 bytes from 107.90.50.205: seq=1 ttl=63 time=181.704 ms

Also no issue from .203 to .205 or .206. Just .201 to .203, neither can reach each other.

They're all bone stock, nothing but the APN was changed. What the heck am I battling?

Anything insightful in the logs?

Got any logs in mind? I can say I'm no Linux guru, just a EE that specializes in hardware design.
What your gut telling you?

Thanks again, Trendy!

Check in logread and dmesg for a start. The fact that you see them in tcpdump but not as a ping response means that something dropped them inside your system, either a firewall or something else.

Think I found it, this route on the .201 device. She sure as heck would catch .203 and not .205 and .206 which I can reach.

.201 SysOps
root@AWC-EI:~# ip r
	default via 107.90.50.202 dev wwan0  src 107.90.50.201 
	10.3.252.4/30 dev br-lan scope link  src 10.3.252.6 
	107.90.50.200/30 dev wwan0 scope link  src 107.90.50.201  -> this bad boy
Every other site does not

example Airport
root@AWC-EI:~# ip r
default via 107.90.50.204 dev wwan0 
107.90.50.200/29 dev wwan0 scope link  src 107.90.50.203 
192.168.113.0/24 dev br-lan scope link  src 192.168.113.1 

Now the question is where is it coming from?

I think it is connected to the issues you reported in the other topic. You have a /30 mask on some sites and a /29 on another. You are trying to ping something that is on another broadcast domain, but in the eyes of your router it is in the same.
May I suggest you to contact At&t and ask them to assign you IP addresses which are at least in different /29 subnets?
For example: .201 , .209, .217, .225, etc...

Last night I did ask them to the to move the .203 to .207, outside the reach of that route. I'll keep you posted. Not a solution, but a band aid until I can figure out where it's coming from.