ARM LXD Container not routing

I'm trying to migrate from a x86-64 LXD container router to a raspberry 4 based one (as my x86-64 machine died) and I have this little issue where the OpenWRT instance is able to go to Internet, but other computers in the network aren't, even the raspberry host.

Note that when I try to ping www.openwrt.org tcpdump running in the openwrt container is able to capture the ICMP packets:

18:09:10.930058 IP 192.168.0.3 > wiki-01.infra.openwrt.org: ICMP echo request, id 6, seq 1, length 64
18:09:11.906188 IP 192.168.0.3 > wiki-01.infra.openwrt.org: ICMP echo request, id 6, seq 2, length 64
18:09:12.930253 IP 192.168.0.3 > wiki-01.infra.openwrt.org: ICMP echo request, id 6, seq 3, length 64
18:09:13.956161 IP 192.168.0.3 > wiki-01.infra.openwrt.org: ICMP echo request, id 6, seq 4, length 64
18:09:14.978220 IP 192.168.0.3 > wiki-01.infra.openwrt.org: ICMP echo request, id 6, seq 5, length 64

If my memory serves well I had this very same problem back in the x86-64 when I tried to create a new container with 20.02 or a newer 19.07 and I'd get the very same problem.

Here's my /etc/config/network file:

config interface 'loopback'
	option ifname 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config interface 'lan'
	option ifname 'eth0'
	option proto 'static'
	option netmask '255.255.255.0'
	option ipaddr '192.168.0.101'
	option delegate '0'
	option type 'bridge'

config interface 'wan'
	option proto 'pppoe'
	option password ''
	option ipv6 '0'
	option mtu '1492'
	option delegate '0'
	option username ''
	option ifname 'eth1'

config interface 'wan6'
	option ifname 'eth1'
	option auto '0'
	option proto 'none'

Note that I tried to restore a configuration backup from the working OpenWRT container and still doesn't work.

What's going on?

So looks like your container is visible from the lan...

  • Did you mess with your firewall maybe?
  • How are your routes on the router?
  • What kind of virtual interfaces is your container attached to?
  • What network setup do you have on the host?
1 Like

Thanks for the reply.

Unfortunately, I didn't mess with the firewall. It's something else.

The raspberry host has a bridge network device attached to the ethernet port. It's address is set statically. Then there's a vlan device that is passed to the container.

The routes in the container look OK to me:

~ # ip route
default via 87.235.0.10 dev pppoe-wan
87.235.0.10 dev pppoe-wan scope link  src 46.27.180.242
192.168.0.0/24 dev eth0 scope link  src 192.168.0.101

Here's my container config:

architecture: aarch64
config:
  boot.autostart: "true"
  image.architecture: arm64
  image.description: Openwrt 19.07 arm64 (20211006_11:57)
  image.os: Openwrt
  image.release: "19.07"
  image.serial: "20211006_11:57"
  image.type: squashfs
  image.variant: default
  linux.kernel_modules: ip_tables,ip6_tables,netlink_diag,nf_nat,overlay,br_netfilter,slhc,ppp_generic,ppp_async,ppp_mppe,ip_gre,gre,pptp,
    pppoe, pppox
devices:
  ppp:
    path: /dev/ppp
    type: unix-char
ephemeral: false
profiles:
- router-iface
stateful: false
description: ""

And network profile:

config: {}
description: ""
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: br0
    type: nic
  eth1:
    name: eth1
    nictype: macvlan
    parent: ont
    type: nic
  root:
    path: /
    pool: default
    type: disk
name: router-iface

And this is the network config on the raspberry host. Note that my ISP requires me to use PPPoE with a tagged VLAN. Tagging inside the container does not work in the raspberry container while it did work in my x86-64 containers.

network:
    ethernets:
        eth0:
            dhcp4: no
            optional: true
    bridges:
        br0:
            dhcp4: no
            addresses:
             - 192.168.0.15/24
            gateway4: 192.168.0.101
            nameservers:
                search: [local]
                addresses: [1.1.1.1, 8.8.8.8]
            interfaces:
                - eth0
    vlans:
        ont:
            id: 100
            link: eth0
    version: 2
1 Like

I found something!

I tried to enable firewall logging and saw that there were no rules applied. iptables -L printed none.

However... if I click the Restart Firewall button in the Firewall Status page, or manually call /etc/init.d/firewall reload, look what happens:

~ # /etc/init.d/firewall reload
Warning: Unable to locate ipset utility, disabling ipset support
 * Clearing IPv4 filter table
 * Clearing IPv4 nat table
 * Clearing IPv4 mangle table
 * Clearing IPv4 raw table
 * Populating IPv4 filter table
Warning: fw3_ipt_rule_append(): Can't find target 'input_lan_rule'
Warning: fw3_ipt_rule_append(): Can't find target 'output_lan_rule'
Warning: fw3_ipt_rule_append(): Can't find target 'forwarding_lan_rule'
Warning: fw3_ipt_rule_append(): Can't find target 'input_wan_rule'
Warning: fw3_ipt_rule_append(): Can't find target 'output_wan_rule'
Warning: fw3_ipt_rule_append(): Can't find target 'forwarding_wan_rule'
Warning: fw3_ipt_rule_append(): Can't find target 'input_rule'
Warning: fw3_ipt_rule_append(): Can't find target 'output_rule'
Warning: fw3_ipt_rule_append(): Can't find target 'forwarding_rule'
   * Rule 'Allow-IPSec-ESP'
   * Rule 'Allow-ISAKMP'
   * Redirect 'uto'
   * Redirect 'transmission'
   * Redirect 'Skype-X64'
   * Redirect 'WireGuard'
   * Redirect 'transmission-rpi'
   * Redirect 'chia'
   * Forward 'lan' -> 'wan'
   * Forward 'wan' -> 'lan'
   * Zone 'lan'
   * Zone 'wan'
 * Populating IPv4 nat table
Warning: fw3_ipt_rule_append(): Can't find target 'prerouting_lan_rule'
Warning: fw3_ipt_rule_append(): Can't find target 'postrouting_lan_rule'
Warning: fw3_ipt_rule_append(): Can't find target 'prerouting_wan_rule'
Warning: fw3_ipt_rule_append(): Can't find target 'postrouting_wan_rule'
Warning: fw3_ipt_rule_append(): Can't find target 'prerouting_rule'
Warning: fw3_ipt_rule_append(): Can't find target 'postrouting_rule'
   * Redirect 'transmission'
   * Redirect 'Skype-X64'
   * Redirect 'WireGuard'
   * Zone 'lan'
   * Zone 'wan'
 * Populating IPv4 mangle table
   * Zone 'lan'
   * Zone 'wan'
 * Populating IPv4 raw table
   * Zone 'lan'
     - Using automatic conntrack helper attachment
   * Zone 'wan'
 * Clearing IPv6 filter table
 * Clearing IPv6 nat table
 * Clearing IPv6 mangle table
 * Clearing IPv6 raw table
 * Populating IPv6 filter table
Warning: fw3_ipt_rule_append(): Can't find target 'input_lan_rule'
Warning: fw3_ipt_rule_append(): Can't find target 'output_lan_rule'
Warning: fw3_ipt_rule_append(): Can't find target 'forwarding_lan_rule'
Warning: fw3_ipt_rule_append(): Can't find target 'input_wan_rule'
Warning: fw3_ipt_rule_append(): Can't find target 'output_wan_rule'
Warning: fw3_ipt_rule_append(): Can't find target 'forwarding_wan_rule'
Warning: fw3_ipt_rule_append(): Can't find target 'input_rule'
Warning: fw3_ipt_rule_append(): Can't find target 'output_rule'
Warning: fw3_ipt_rule_append(): Can't find target 'forwarding_rule'
   * Rule 'Allow-MLD'
   * Rule 'Allow-IPSec-ESP'
   * Rule 'Allow-ISAKMP'
   * Forward 'lan' -> 'wan'
   * Forward 'wan' -> 'lan'
   * Zone 'lan'
   * Zone 'wan'
 * Populating IPv6 nat table
Warning: fw3_ipt_rule_append(): Can't find target 'prerouting_lan_rule'
Warning: fw3_ipt_rule_append(): Can't find target 'postrouting_lan_rule'
Warning: fw3_ipt_rule_append(): Can't find target 'prerouting_wan_rule'
Warning: fw3_ipt_rule_append(): Can't find target 'postrouting_wan_rule'
Warning: fw3_ipt_rule_append(): Can't find target 'prerouting_rule'
Warning: fw3_ipt_rule_append(): Can't find target 'postrouting_rule'
   * Zone 'lan'
   * Zone 'wan'
 * Populating IPv6 mangle table
   * Zone 'lan'
   * Zone 'wan'
 * Populating IPv6 raw table
   * Zone 'lan'
     - Using automatic conntrack helper attachment
   * Zone 'wan'
 * Set tcp_ecn to off
 * Set tcp_syncookies to on
 * Set tcp_window_scaling to on

It seems that the firewall rules or the firewall service is not started during the boot process. And it looks exactly like this issue.

1 Like

Indeed, looks exactly like that... very weird yet interesting.
So, following those links it ultimately seems to depend on the host's environment, and not on fw3 itself. In my experience I haven't had such an issue on Debian 11 host and plain lxc v4 (and now that i recall on Debian 10 and lxc v3 neither...).

1 Like

I'm using Ubuntu 20.04 and LXC 4.0.7. Tried switching over to netfilter and the issue persists. In Debian 11-based DietPi I get the same behaviour but there's no iptables or netfilter.

What image are you using?

I'm using the one from linuxcontainers.org, which recently switched to the stock OpenWRT armvirt tarball. cvmiller's repo contains an init.sh script which reboots the firewall, and that makes me think some people (me included) did not experience this issue earlier because it was patched at the image level.

Needless to say, if I add cvmiller's firewall restart snippet inside a custom startup script, everything sort of works.

I'm using the OpenWrt lxc image on x86_64 available from the download template.

Perhaps that's an intentional workaround?