Network (mtu?) issue with my LXC container

Hello everyone,

I wanted to install a basic alpine image in an LXC on my device and just try running a speedtest from Ookla with the prebuilt solftware for my armv7 CPU.

I am running openWRT 19.07 but I took the packages from the 21.02 repo.

Please, don't judge me :slight_smile:

The exe file works fine from the router itself, but it fails when running in the container.
From what I see, this is an internal networking issue (please bare in mind that the issue only occurs with my linux container, not when running these commands from the router itself).
When capturing either on the bridge or the WAN interface, I can see the TCP session establishing towards the speedtest server, but it seems that, as soon as the packets get larger, the packets are dropped somewhere. Let me show you the tcpdump on the WAN interface:

tcpdump -i eth4 port 443 -n -e
11:56:12.783874 80:69:1a:b2:bc:cf > a4:3e:51:8b:26:56, ethertype IPv4 (0x0800), length 74: 192.168.1.24.48294 > 104.17.106.111.443: Flags [S], seq 2406609767, win 65136, options [mss 472,sackOK,TS val 3349432948 ecr 0,nop,wscale 7], length 0
11:56:12.786246 a4:3e:51:8b:26:56 > 80:69:1a:b2:bc:cf, ethertype IPv4 (0x0800), length 74: 104.17.106.111.443 > 192.168.1.24.48294: Flags [S.], seq 405235817, ack 2406609768, win 65160, options [mss 1400,sackOK,TS val 1773357901 ecr 3349432948,nop,wscale 13], length 0
11:56:12.786573 80:69:1a:b2:bc:cf > a4:3e:51:8b:26:56, ethertype IPv4 (0x0800), length 66: 192.168.1.24.48294 > 104.17.106.111.443: Flags [.], ack 1, win 509, options [nop,nop,TS val 3349432951 ecr 1773357901], length 0
11:56:12.829812 80:69:1a:b2:bc:cf > a4:3e:51:8b:26:56, ethertype IPv4 (0x0800), length 212: 192.168.1.24.48294 > 104.17.106.111.443: Flags [P.], seq 1:147, ack 1, win 509, options [nop,nop,TS val 3349432994 ecr 1773357901], length 146

So in the above, we can see that the SYN from my container does reach the internet and we get a SYN-ACK. Then no news during 15seconds, and look at what is coming next:

11:56:27.789674 a4:3e:51:8b:26:56 > 80:69:1a:b2:bc:cf, ethertype IPv4 (0x0800), length 97: 104.17.106.111.443 > 192.168.1.24.48294: Flags [FP.], seq 4460:4491, ack 240, win 8, options [nop,nop,TS val 1773372904 ecr 3349446776], length 31
11:56:27.789984 80:69:1a:b2:bc:cf > a4:3e:51:8b:26:56, ethertype IPv4 (0x0800), length 78: 192.168.1.24.48294 > 104.17.106.111.443: Flags [.], ack 4409, win 509, options [nop,nop,TS val 3349447955 ecr 1773357951,nop,nop,sack 1 {4460:4492}], length 0
11:56:27.792249 a4:3e:51:8b:26:56 > 80:69:1a:b2:bc:cf, ethertype IPv4 (0x0800), length 117: 104.17.106.111.443 > 192.168.1.24.48294: Flags [P.], seq 4409:4460, ack 240, win 8, options [nop,nop,TS val 1773372907 ecr 3349447955], length 51
11:56:27.792378 80:69:1a:b2:bc:cf > a4:3e:51:8b:26:56, ethertype IPv4 (0x0800), length 66: 192.168.1.24.48294 > 104.17.106.111.443: Flags [.], ack 4492, win 509, options [nop,nop,TS val 3349447957 ecr 1773372907], length 0
11:56:27.792738 80:69:1a:b2:bc:cf > a4:3e:51:8b:26:56, ethertype IPv4 (0x0800), length 504: 192.168.1.24.48294 > 104.17.106.111.443: Flags [P.], seq 240:678, ack 4492, win 509, options [nop,nop,TS val 3349447957 ecr 1773372907], length 438
11:56:27.793762 80:69:1a:b2:bc:cf > a4:3e:51:8b:26:56, ethertype IPv4 (0x0800), length 66: 192.168.1.24.48294 > 104.17.106.111.443: Flags [F.], seq 678, ack 4492, win 509, options [nop,nop,TS val 3349447958 ecr 1773372907], length 0
11:56:27.795229 a4:3e:51:8b:26:56 > 80:69:1a:b2:bc:cf, ethertype IPv4 (0x0800), length 60: 104.17.106.111.443 > 192.168.1.24.48294: Flags [R], seq 405240309, win 0, length 0
11:56:27.796220 a4:3e:51:8b:26:56 > 80:69:1a:b2:bc:cf, ethertype IPv4 (0x0800), length 60: 104.17.106.111.443 > 192.168.1.24.48294: Flags [R], seq 405240309, win 0, length 0

OK, we can see that the relative sequence numbers back from the server directly jumped from 1 to 4460, so we had several packets which were discarded and which I cannot see on the WAN interface and the next one I see is 103 bytes large if you remove the ethernet header. So the largest IP packet I got was 103 bytes large.

I wanted to do that ping test from my container to see the mtu I was getting from there:

root@myLMS:~# ping -s 88 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 88 data bytes
76 bytes from 8.8.8.8: seq=0 ttl=116 time=2.277 ms
76 bytes from 8.8.8.8: seq=1 ttl=116 time=2.522 ms
^C
--- 8.8.8.8 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 2.277/2.399/2.522 ms
root@myLMS:~# ping -s 89 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 89 data bytes
^C
--- 8.8.8.8 ping statistics ---
5 packets transmitted, 0 packets received, 100% packet loss

So, in the end the mtu I get is 96 bytes. How, that is only true downstream! In the upstream, no problem. When you look at the capture if I increase even more the size of the request, you see that the cimp request still passes, but the reply is stuck to 110 (so 96 bytes):

root@OpenWrt:~# tcpdump -i eth4 icmp -n -e
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth4, link-type EN10MB (Ethernet), capture size 262144 bytes
12:34:09.061918 80:69:1a:b2:bc:cf > a4:3e:51:8b:26:56, ethertype IPv4 (0x0800), length 142: 192.168.1.24 > 8.8.8.8: ICMP echo request, id 441, seq 0, length 108
12:34:09.064119 a4:3e:51:8b:26:56 > 80:69:1a:b2:bc:cf, ethertype IPv4 (0x0800), length 110: 8.8.8.8 > 192.168.1.24: ICMP echo reply, id 441, seq 0, length 76
12:34:10.062117 80:69:1a:b2:bc:cf > a4:3e:51:8b:26:56, ethertype IPv4 (0x0800), length 142: 192.168.1.24 > 8.8.8.8: ICMP echo request, id 441, seq 1, length 108
12:34:10.064084 a4:3e:51:8b:26:56 > 80:69:1a:b2:bc:cf, ethertype IPv4 (0x0800), length 110: 8.8.8.8 > 192.168.1.24: ICMP echo reply, id 441, seq 1, length 76
12:34:11.062300 80:69:1a:b2:bc:cf > a4:3e:51:8b:26:56, ethertype IPv4 (0x0800), length 142: 192.168.1.24 > 8.8.8.8: ICMP echo request, id 441, seq 2, length 108
12:34:11.064578 a4:3e:51:8b:26:56 > 80:69:1a:b2:bc:cf, ethertype IPv4 (0x0800), length 110: 8.8.8.8 > 192.168.1.24: ICMP echo reply, id 441, seq 2, length 76
12:34:12.062471 80:69:1a:b2:bc:cf > a4:3e:51:8b:26:56, ethertype IPv4 (0x0800), length 142: 192.168.1.24 > 8.8.8.8: ICMP echo request, id 441, seq 3, length 108
12:34:12.064588 a4:3e:51:8b:26:56 > 80:69:1a:b2:bc:cf, ethertype IPv4 (0x0800), length 110: 8.8.8.8 > 192.168.1.24: ICMP echo reply, id 441, seq 3, length 76

When you look in details, you cannot see the fragmentation, you can only derive it occured since the DF bit is not set anymore.

The mtu set on all my interfaces is 1500.

The checkconfig gives the following:

root@OpenWrt:~# lxc-checkconfig
LXC version 4.0.10
--- Namespaces ---
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: enabled
Network namespace: enabled

--- Control groups ---
Cgroups: enabled
Cgroup namespace: enabled

Cgroup v1 mount points:
/sys/fs/cgroup
/sys/fs/cgroup/systemd

Cgroup v2 mount points:


Cgroup v1 freezer controller: missing
Cgroup v1 clone_children flag: enabled
Cgroup device: enabled
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: enabled
Cgroup cpuset: enabled

--- Misc ---
Veth pair device: enabled, loaded
Macvlan: enabled, loaded
Vlan: enabled, not loaded
Bridges: enabled, loaded
Advanced netfilter: enabled, loaded
CONFIG_NF_NAT_IPV4: missing
CONFIG_NF_NAT_IPV6: missing
CONFIG_IP_NF_TARGET_MASQUERADE: enabled, not loaded
CONFIG_IP6_NF_TARGET_MASQUERADE: enabled, not loaded
CONFIG_NETFILTER_XT_TARGET_CHECKSUM: missing
CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled, loaded
FUSE (for use with lxcfs): enabled, loaded

--- Checkpoint/Restore ---
checkpoint restore: missing
CONFIG_FHANDLE: missing
CONFIG_EVENTFD: enabled
CONFIG_EPOLL: enabled
CONFIG_UNIX_DIAG: missing
CONFIG_INET_DIAG: missing
CONFIG_PACKET_DIAG: missing
CONFIG_NETLINK_DIAG: missing
File capabilities:

And the confirmation that my container's got an ip:

root@OpenWrt:~# lxc-ls -f
NAME  STATE   AUTOSTART GROUPS IPV4          IPV6                                                                        UNPRIVILEGED
myLMS RUNNING 0         -      192.168.7.188 2a01:cb00:138e:a1f2:2ff:ddff:febb:cc01, fd6a:8384:5c8a:0:2ff:ddff:febb:cc01 false

Note, I did that on a fresh router, no fancy previous setup. If someone has some idea, I would really appreciate as I am getting stuck here.

Nice trip into archaeology.
Please post output of

ubus call system board

And lets check how we can upgrade to supported release (iptables firewall <= v21.02.99 is out of scope nowadays)

Hi there,

Yeah, well, I am using the QSDK, that's why I am using this outdated openWRT version. Here is my board output:

{
        "kernel": "5.4.213",
        "hostname": "OpenWrt",
        "system": "ARMv7 Processor rev 0 (v7l)",
        "model": "Qualcomm Technologies, Inc. IPQ9574/AP-AL02-C4",
        "board_name": "qcom,ipq9574-ap-al02-c4",
        "release": {
                "distribution": "OpenWrt",
                "version": "19.07-9",
                "revision": "",
                "target": "ipq95xx/ipq95xx_32",
                "description": "OpenWrt 19.07-9"
        }
}

I would understand that you don't want to spend time on that one. It really looks like there is some odd setup somewhere but I could not figure out where.

So ask Q who sold you SDK to provide you with working LXC. As much as OpenWRT is concerned you need to build and package newer packages for older vendor releases yourself, sometimes adding some history dust in makefiles.

That's fair. I will let you know if I find the solution in case it may be relevant for someone else.

This topic was automatically closed after 16 hours. New replies are no longer allowed.