No IPV4 using Raspberry pi as router

Hi all,
I'm hoping to use a Raspberry pi 4 to improve WAN speeds with SQM now that I have gigabit fiber. I can't really take my internet down for long periods so I'm trying to do basic configuration by attaching it to my existing network - basically a double-NAT scenario. Also, I'm running openwrt virtually - so an RPi-4 running 32-bit Buster using nspawn to host the 32-bit, RPi-2 image of openwrt 19.07.

The basic topology is: Internet<->ONT<->OpenWRT router 1 (TP-1043nd)<->OpenWRT switch (TP-1043nd again - just used as a switch)<->Raspberry Pi<->back into the OpenWRT switch

I'm using 2 vlans for this. The TP-router connects vlan 11 (eth0.11 on the pi) to the internet. This is the Pi's WAN. The Pi should route this onto vlan 16 (eth0.16) and send it back out the same cable to the router which is also running an static interface on vlan 16 that is bridged to a wifi station.

In general most things seem to work - I was quite happy to see luci on my laptop's browser - first via a crossover cable and later via wifi on vlan16 once I had the topology described above.

The problem is my laptop connected to vlan16 can't see the IPv4 internet (The IPv6 internet, DHCP, and the local VLAN16 subnet work, but not the VLAN11 subnet, except for devices with interfaces in both). On my laptop, ping 8.8.8.8 fails. If I ssh into the pi-openwrt, then everything works: ping 8.8.8.8, ping test-ipv6.com as well as all the intranet devices I would expect to work.

I've been struggling with this for quite a few hours - comparing configuration files of the pi with those of my working router. About the only thing I can still think of is that raspberry os uses nftables and openwrt is using iptables. I'm not sure if that can cause any problems with firewall or NAT operation when running virtually - but it would still be odd that IPv4 works directly within openwrt.

Thanks for any suggestions!

My network, dhcp, and firewall config files are:


cat network

config interface 'loopback'
        option ifname 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'fd14:37d8:915e::/48'

config interface 'lan'
        option type 'bridge'
        option proto 'static'
        option netmask '255.255.255.0'
        option ip6assign '60'
        option ifname 'eth0.16'
        option ipaddr '192.168.16.1'

config interface 'WAN'
        option proto 'dhcp'
        option ifname 'eth0.11'

config interface 'WAN6'
        option ifname 'eth0.11'
        option proto 'dhcpv6'
        option reqaddress 'try'
        option reqprefix 'auto'


cat dhcp

config dnsmasq
        option domainneeded '1'
        option boguspriv '1'
        option filterwin2k '0'
        option localise_queries '1'
        option rebind_protection '1'
        option rebind_localhost '1'
        option local '/lan/'
        option domain 'lan'
        option expandhosts '1'
        option nonegcache '0'
        option authoritative '1'
        option readethers '1'
        option leasefile '/tmp/dhcp.leases'
        option resolvfile '/tmp/resolv.conf.auto'
        option nonwildcard '1'
        option localservice '1'

config dhcp 'lan'
        option interface 'lan'
        option leasetime '12h'
        option dhcpv6 'server'
        option ra 'server'
        option ra_management '1'
        option start '25'
        option limit '75'

config dhcp 'wan'
        option interface 'wan'
        option ignore '1'

config odhcpd 'odhcpd'
        option maindhcp '0'
        option leasefile '/tmp/hosts/odhcpd'
        option leasetrigger '/usr/sbin/odhcpd-update'
        option loglevel '4'



cat firewall

config defaults
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option synflood_protect '1'

config zone
        option name 'lan'
        list network 'lan'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'

config zone
        option name 'wan'
        list network 'wan'
        list network 'wan6'
        option input 'REJECT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option masq '1'
        option mtu_fix '1'

config forwarding
        option src 'lan'
        option dest 'wan'

config rule
        option name 'Allow-DHCP-Renew'
        option src 'wan'
        option proto 'udp'
        option dest_port '68'
        option target 'ACCEPT'
        option family 'ipv4'

config rule
        option name 'Allow-Ping'
        option src 'wan'
        option proto 'icmp'
        option icmp_type 'echo-request'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-IGMP'
        option src 'wan'
        option proto 'igmp'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-DHCPv6'
        option src 'wan'
        option proto 'udp'
        option src_ip 'fc00::/6'
        option dest_ip 'fc00::/6'
        option dest_port '546'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-MLD'
        option src 'wan'
        option proto 'icmp'
        option src_ip 'fe80::/10'
        list icmp_type '130/0'
        list icmp_type '131/0'
        list icmp_type '132/0'
        list icmp_type '143/0'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Input'
        option src 'wan'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        list icmp_type 'router-solicitation'
        list icmp_type 'neighbour-solicitation'
        list icmp_type 'router-advertisement'
        list icmp_type 'neighbour-advertisement'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Forward'
        option src 'wan'
        option dest '*'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-IPSec-ESP'
        option src 'wan'
        option dest 'lan'
        option proto 'esp'
        option target 'ACCEPT'

config rule
        option name 'Allow-ISAKMP'
        option src 'wan'
        option dest 'lan'
        option dest_port '500'
        option proto 'udp'
        option target 'ACCEPT'

config include
        option path '/etc/firewall.user'

config forwarding
        option dest 'lan'
        option src 'wan'

Linux is case sensitive OS. Currently the WAN and WAN6 are not assigned to the wan firewall zone.
Edit the /etc/config/network file and change the interface name to lower case letters.

1 Like

Thanks for that. I think luci tends to capitalize names. I went ahead and killed the VM, edited the network file placing those names into lower case and started up again but no change.

Only when it shows them.

Post once again the following uci export network; uci export firewall; iptables-save -c; ip -4 addr; ip -4 ro li tab all; ip -4 ru

1 Like

Thanks for the response.

Here is the output (none from iptables-save -c):

 -----------------------------------------------------
 OpenWrt 19.07.4, r11208-ce6496d796
 -----------------------------------------------------
root@openwrt:~# uci export network; uci export firewall; iptables-save -c; ip -4
 addr; ip -4 ro li tab all; ip -4 ru
package network

config interface 'loopback'
        option ifname 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'fd14:37d8:915e::/48'

config interface 'lan'
        option type 'bridge'
        option proto 'static'
        option netmask '255.255.255.0'
        option ip6assign '60'
        option ifname 'eth0.16'
        option ipaddr '192.168.16.1'

config interface 'wan'
        option proto 'dhcp'
        option ifname 'eth0.11'

config interface 'wan6'
        option ifname 'eth0.11'
        option proto 'dhcpv6'
        option reqaddress 'try'
        option reqprefix 'auto'

package firewall

config defaults
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option synflood_protect '1'

config zone
        option name 'lan'
        list network 'lan'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'

config zone
        option name 'wan'
        list network 'wan'
        list network 'wan6'
        option input 'REJECT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option masq '1'
        option mtu_fix '1'

config forwarding
        option src 'lan'
        option dest 'wan'

config rule
        option name 'Allow-DHCP-Renew'
        option src 'wan'
        option proto 'udp'
        option dest_port '68'
        option target 'ACCEPT'
        option family 'ipv4'

config rule
        option name 'Allow-Ping'
        option src 'wan'
        option proto 'icmp'
        option icmp_type 'echo-request'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-IGMP'
        option src 'wan'
        option proto 'igmp'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-DHCPv6'
        option src 'wan'
        option proto 'udp'
        option src_ip 'fc00::/6'
        option dest_ip 'fc00::/6'
        option dest_port '546'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-MLD'
        option src 'wan'
        option proto 'icmp'
        option src_ip 'fe80::/10'
        list icmp_type '130/0'
        list icmp_type '131/0'
        list icmp_type '132/0'
        list icmp_type '143/0'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Input'
        option src 'wan'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        list icmp_type 'router-solicitation'
        list icmp_type 'neighbour-solicitation'
        list icmp_type 'router-advertisement'
        list icmp_type 'neighbour-advertisement'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Forward'
        option src 'wan'
        option dest '*'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-IPSec-ESP'
        option src 'wan'
        option dest 'lan'
        option proto 'esp'
        option target 'ACCEPT'

config rule
        option name 'Allow-ISAKMP'
        option src 'wan'
        option dest 'lan'
        option dest_port '500'
        option proto 'udp'
        option target 'ACCEPT'

config include
        option path '/etc/firewall.user'

config forwarding
        option dest 'lan'
        option src 'wan'

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
4: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    inet 192.168.16.1/24 brd 192.168.16.255 scope global br-lan
       valid_lft forever preferred_lft forever
6: eth0.11@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    inet 192.168.11.79/24 brd 192.168.11.255 scope global eth0.11
       valid_lft forever preferred_lft forever
default via 192.168.11.1 dev eth0.11  src 192.168.11.79
192.168.11.0/24 dev eth0.11 scope link  src 192.168.11.79
192.168.16.0/24 dev br-lan scope link  src 192.168.16.1
broadcast 127.0.0.0 dev lo table local scope link  src 127.0.0.1
local 127.0.0.0/8 dev lo table local scope host  src 127.0.0.1
local 127.0.0.1 dev lo table local scope host  src 127.0.0.1
broadcast 127.255.255.255 dev lo table local scope link  src 127.0.0.1
broadcast 192.168.11.0 dev eth0.11 table local scope link  src 192.168.11.79
local 192.168.11.79 dev eth0.11 table local scope host  src 192.168.11.79
broadcast 192.168.11.255 dev eth0.11 table local scope link  src 192.168.11.79
broadcast 192.168.16.0 dev br-lan table local scope link  src 192.168.16.1
local 192.168.16.1 dev br-lan table local scope host  src 192.168.16.1
broadcast 192.168.16.255 dev br-lan table local scope link  src 192.168.16.1
0:      from all lookup local
32766:  from all lookup main
32767:  from all lookup default
root@openwrt:~#

Hmmm... I tried to nspawn openwrt in the past on x86 and had no luck... there was a thread about it: Using Systemd to run Openwrt on Raspberry Pi 4…welcome to join

are you sure OpenWrt is starting up fully? If so, what did you have to do to make it work? (feel free if you like to add to info in that other thread, it's not closed yet).

also, the nftables vs iptables stuff is definitely an issue when running in a container. It wouldn't be as much of an issue in a fully virtual situation, but that wouldn't perform as well either. One of the big issues is that nftables and iptables have totally different NAT modules.

Honestly, these days I run Raspbian as my router. I set the network up with systemd-networkd files in /etc/systemd/network (update to latest systemd) and do the firewall in nftables. nftables is FAR preferable to iptables. I put the log on a separate flash partition with jffs2

The only thing you don't get is luci, but I run netdata on the device to get a quick snapshot of how well its working, and do all my configuration in ssh.

I don't miss the lack of full set of modules, specialized tiny binaries, and etc.

1 Like

Thanks for the feedback.

I'm writing a post for the other thread on how I got this far. I wouldn't mind running everything in Buster without the overhead of the openwrt container, but I'm a bit apprehensive about getting started and struggling to duplicate all the configuration I originally struggled with on openwrt: multiple VLANs, a few php scripts to generate uci commands, mini-DLNA, SQM... - plus a simple, fast backup/restore (and auto-rollback on most failed configs), and I'm pretty sure no SD-card writes during normal operation. Other than some DLNA compatibility issues, it's been amazingly stable for a few years - so I'm definitely an openwrt fan. I really like the idea of having the same luci interface on nearly all my hardware.

I'll read up on nftables and iptables. Since nspawn hands eth0 to openwrt, I'm not sure what nftables would be doing on the host side. I haven't configured anything so I don't know what rules a bunch of packets on various unconfigured VLANs would trigger. I'll see if I can disable it for a specific device - perhaps at the same time as I start openwrt.

I suppose you are using nftables then? Still you'll have to verify that masquerade and forwardings configured in uci are somehow applied there, otherwise traffic cannot be passed from lan to wan and then be NATed.

1 Like

I've never directly dealt with iptables or nftables - so I need to learn more. What minimal firewall configuration I did was via luci. Ideally I'd like to keep it that way (using luci). I'm not sure if I can turn off nftables in the host and use iptables in openwrt.

Any idea why everything works when I'm ssh'ed into openwrt?

There are nftables packages for OpenWrt, but I have never used them and I don't know how good is the integration with Luci/UCI.

I told you already.

Nspawn is a container system not a virtual system. So there's just the one kernel and one set of modules. It's not possible to run the iptables modules and the nftables modules at the same time.

1 Like

Thanks. This is the first time I'm working with a container as opposed to a VM. It sure feels like a VM when you're connected to it so I'm still getting my head around what is responsible for what.

I understand there are iptables->nftables translation tools - I'm wondering if I could just monitor the iptables settings inside the container from the host and run the translation tool to update the host's nftables whenever it's updated. Not sure if translation is entirely straightforward (if any strings need to be substituted). I suppose it would only work in one direction so rule feedback in openwrt would still fail, but if it fixes my problems and still permits configuration within openwrt I may take it. Alternatively I may try to install nftables in openwrt and try my hand at directly writing rules and see if those propagate correctly between container and host. I guess I'll be chewing on this a bit yet.

It's possible to translate the iptables command lines into nftables rules so if you have scripts that run a bunch of iptables commands that's easy... however by default OpenWrt interacts directly with the kernel using an iptables library and it's not possible to intercept and translate those interactions.

You can write a secure nftables firewall that does basically everything the standard OpenWrt firewall does in about 20 lines of nftables script many of which are boilerplate... But if you go this route you lose all Luci based firewall editing etc. Getting nftables to work right on OpenWrt has been somewhat variable.

I think it's worth trying and would be able to write you a basic firewall you can start with if you want.

1 Like

Hmmm - my first tinkering seems to indicate openwrt uses ipset - which doesn't appear to have nftables translation support. If I type in fw3 I get:"Warning: Unable to locate ipset utility, disabling ipset support". In general these modules include both kernel and userland portions so I suspect it's not so trivial to follow this path. I'd still prefer to retain the openwrt side of things rather than some strange hybrid since it will be managing the network. Now thinking of making a custom kernel for Pi OS using iptables - just wondering what all that will break...

I think if you really want to do double duty on one machine you might be better off seeing if you can run OpenWrt from inside a KVM virtual machine rather than a container.

Thanks for all the suggestions.

I started to think along the same lines of a full VM after downloading the kernel compiler package for Raspberry OS. The number of configuration options is very impressive and that would just be to remove nftables (I suppose I could just try to uncheck netfilter quite high up), then there's how to add iptables and whatever dependencies it introduces and what are the chances of building a working OS - all for a one-off project...

On the VM side, something like Jailhouse seems interesting in that it looks like it can hard-partition the pi - assigning resources to a specific "cell" as it calls them. But getting it up and running looks challenging plus it has no virtual ethernet support so the RPi-OS side would probably have to get its internet via WiFi. With a typical host/guest scenario, I'd be in the same boat where the host system is still running nftables plus a bunch of virtual hardware to provide a virtual ethernet to openwrt. If anything, nspawn seems closer to "bare-metal" since openwrt captures eth0 and deals with the relevant kernel modules. The only problem is some of the modules are missing.

I'll have one more go at seeing what iptables modules I can install in RPi-OS with just apt and see if it makes any difference.

About your earlier suggestion regarding a simple nftables config on the host side - I can't seem to extract the firewall settings using the normal iptables tools inside openwrt since iptables is missing. Is all the information I'd need for nftables rules contained in the firewall (& other) configuration file(s) in openwrt? I'm wondering how hard it would be to write a script to translate configuration settings for the basic, common use cases. With that it should be possible to do basic configuration in luci and trigger a translation up to nftables on the host side. Assuming that is possible, will I run into further issues with SQM? That's about all I want to get running.

Honestly as much as I like OpenWrt and especially like the OpenWrt community, when it came time to set up a RPi4 as my router I just wrote a nftables firewall, and created a few VLAN interfaces using systemd network files, install dnsmasq and add a few lines of configs, voila I had a router.

If OpenWrt had an nftables firewall I'd probably consider running it in a container, since it's really quite great. But I'm so committed to nftables that I won't go back.

It's way more work to figure out how to run OpenWrt in a VM or container than it is to make your Pi running RaspberryPi OS be a secure wired router.

Tells you how to disable the legacy network stuff on RPi OS and enable systemd.

Then you write a few .netdev files to create some vlans, and a few .network files to set up ip addresses

Writing dnsmasq configs:

I suggest getting a keyboard and monitor so you can debug network things without getting disconnected. When it works, you can just plug it in and voila it's your router.

There's a baseline firewall in my previous attempt at doing nftables on openwrt: QoS and nftables … some findings to share

1 Like

Wow - thanks for all that. I actually came across your last post while tinkering (on QoS and nftables) - but I haven't read it all yet.

I'll be studying for awhile but just one quick question in advance - are you running a QoS package on your Pi? Is there SQM or something equivalent?

Thanks again.

I run something similar to the script in Help prioritizing games with alternative qdisc design

@moeller0 and I are trying to get one into the sqm-scripts package. When we get that debugged I'll post it as standalone as well

If you go this route, make sure you upgrade systemd to the latest version before proceeding.

I haven't had much time to look at this further but had a bit of inspiration a few days ago - why not look for Pi-OSes that use iptables. Ubuntu-MATE is one example. I still don't have things working - so IPv4 traffic isn't getting routed, but it seems tantalizingly close. Not sure if this question is valid - but is there an openwrt command that forces the firewall to configure itself - to generate all relevant calls to ipset and iptables - or any other means by which openwrt configures the firewall? I tried to edit & save the settings in luci and also to restart the service from ssh.

When starting nspawn, I still get no response from fw3 print or iptables-save in the container. I then tried the suggestions from this post: https://unix.stackexchange.com/questions/468015/how-to-make-systemd-nspawn-work-with-ufw-firewall - specifically:

On your host run the following as root(sudo):

systemctl disable network
systemctl disable networking
systemctl disable NetworkManager
systemctl enable systemd-networkd

Still no luck. I then just used iptables-save on my normal router, copied the result to the container, and used iptables-restore and it accepted it. It's not a valid configuration, but iptables-save and fw3 print returned results. The settings are lost if I restart the container. One interesting difference is ping 8.8.8.8 on a windows box connected to the router was returning "request timed out". After the iptables-restore operation it returns "destination port unreachable".

I'll try to mimic the settings I want as closely as possible on my working router and then feed the iptables-save results to the container and see if that works. But if anyone knows if I'm missing an easy step that would trigger a firewall configuration that would be much more desirable.