No IPV4 using Raspberry pi as router

Thanks. This is the first time I'm working with a container as opposed to a VM. It sure feels like a VM when you're connected to it so I'm still getting my head around what is responsible for what.

I understand there are iptables->nftables translation tools - I'm wondering if I could just monitor the iptables settings inside the container from the host and run the translation tool to update the host's nftables whenever it's updated. Not sure if translation is entirely straightforward (if any strings need to be substituted). I suppose it would only work in one direction so rule feedback in openwrt would still fail, but if it fixes my problems and still permits configuration within openwrt I may take it. Alternatively I may try to install nftables in openwrt and try my hand at directly writing rules and see if those propagate correctly between container and host. I guess I'll be chewing on this a bit yet.

It's possible to translate the iptables command lines into nftables rules so if you have scripts that run a bunch of iptables commands that's easy... however by default OpenWrt interacts directly with the kernel using an iptables library and it's not possible to intercept and translate those interactions.

You can write a secure nftables firewall that does basically everything the standard OpenWrt firewall does in about 20 lines of nftables script many of which are boilerplate... But if you go this route you lose all Luci based firewall editing etc. Getting nftables to work right on OpenWrt has been somewhat variable.

I think it's worth trying and would be able to write you a basic firewall you can start with if you want.

1 Like

Hmmm - my first tinkering seems to indicate openwrt uses ipset - which doesn't appear to have nftables translation support. If I type in fw3 I get:"Warning: Unable to locate ipset utility, disabling ipset support". In general these modules include both kernel and userland portions so I suspect it's not so trivial to follow this path. I'd still prefer to retain the openwrt side of things rather than some strange hybrid since it will be managing the network. Now thinking of making a custom kernel for Pi OS using iptables - just wondering what all that will break...

I think if you really want to do double duty on one machine you might be better off seeing if you can run OpenWrt from inside a KVM virtual machine rather than a container.

Thanks for all the suggestions.

I started to think along the same lines of a full VM after downloading the kernel compiler package for Raspberry OS. The number of configuration options is very impressive and that would just be to remove nftables (I suppose I could just try to uncheck netfilter quite high up), then there's how to add iptables and whatever dependencies it introduces and what are the chances of building a working OS - all for a one-off project...

On the VM side, something like Jailhouse seems interesting in that it looks like it can hard-partition the pi - assigning resources to a specific "cell" as it calls them. But getting it up and running looks challenging plus it has no virtual ethernet support so the RPi-OS side would probably have to get its internet via WiFi. With a typical host/guest scenario, I'd be in the same boat where the host system is still running nftables plus a bunch of virtual hardware to provide a virtual ethernet to openwrt. If anything, nspawn seems closer to "bare-metal" since openwrt captures eth0 and deals with the relevant kernel modules. The only problem is some of the modules are missing.

I'll have one more go at seeing what iptables modules I can install in RPi-OS with just apt and see if it makes any difference.

About your earlier suggestion regarding a simple nftables config on the host side - I can't seem to extract the firewall settings using the normal iptables tools inside openwrt since iptables is missing. Is all the information I'd need for nftables rules contained in the firewall (& other) configuration file(s) in openwrt? I'm wondering how hard it would be to write a script to translate configuration settings for the basic, common use cases. With that it should be possible to do basic configuration in luci and trigger a translation up to nftables on the host side. Assuming that is possible, will I run into further issues with SQM? That's about all I want to get running.

Honestly as much as I like OpenWrt and especially like the OpenWrt community, when it came time to set up a RPi4 as my router I just wrote a nftables firewall, and created a few VLAN interfaces using systemd network files, install dnsmasq and add a few lines of configs, voila I had a router.

If OpenWrt had an nftables firewall I'd probably consider running it in a container, since it's really quite great. But I'm so committed to nftables that I won't go back.

It's way more work to figure out how to run OpenWrt in a VM or container than it is to make your Pi running RaspberryPi OS be a secure wired router.

Tells you how to disable the legacy network stuff on RPi OS and enable systemd.

Then you write a few .netdev files to create some vlans, and a few .network files to set up ip addresses

Writing dnsmasq configs:

I suggest getting a keyboard and monitor so you can debug network things without getting disconnected. When it works, you can just plug it in and voila it's your router.

There's a baseline firewall in my previous attempt at doing nftables on openwrt: QoS and nftables … some findings to share

1 Like

Wow - thanks for all that. I actually came across your last post while tinkering (on QoS and nftables) - but I haven't read it all yet.

I'll be studying for awhile but just one quick question in advance - are you running a QoS package on your Pi? Is there SQM or something equivalent?

Thanks again.

I run something similar to the script in Help prioritizing games with alternative qdisc design

@moeller0 and I are trying to get one into the sqm-scripts package. When we get that debugged I'll post it as standalone as well

If you go this route, make sure you upgrade systemd to the latest version before proceeding.

I haven't had much time to look at this further but had a bit of inspiration a few days ago - why not look for Pi-OSes that use iptables. Ubuntu-MATE is one example. I still don't have things working - so IPv4 traffic isn't getting routed, but it seems tantalizingly close. Not sure if this question is valid - but is there an openwrt command that forces the firewall to configure itself - to generate all relevant calls to ipset and iptables - or any other means by which openwrt configures the firewall? I tried to edit & save the settings in luci and also to restart the service from ssh.

When starting nspawn, I still get no response from fw3 print or iptables-save in the container. I then tried the suggestions from this post: https://unix.stackexchange.com/questions/468015/how-to-make-systemd-nspawn-work-with-ufw-firewall - specifically:

On your host run the following as root(sudo):

systemctl disable network
systemctl disable networking
systemctl disable NetworkManager
systemctl enable systemd-networkd

Still no luck. I then just used iptables-save on my normal router, copied the result to the container, and used iptables-restore and it accepted it. It's not a valid configuration, but iptables-save and fw3 print returned results. The settings are lost if I restart the container. One interesting difference is ping 8.8.8.8 on a windows box connected to the router was returning "request timed out". After the iptables-restore operation it returns "destination port unreachable".

I'll try to mimic the settings I want as closely as possible on my working router and then feed the iptables-save results to the container and see if that works. But if anyone knows if I'm missing an easy step that would trigger a firewall configuration that would be much more desirable.

This is Debian based so it's using nftables by calling iptables command line which in modern version just translates the command to an nft command... Sorry :wink:

As far as I know all desktop distros use nftables these days most of them through the translation method

1 Like

In some cases, they use iptables and nftables simultaneously.
E.g. services like libvirt have not migrated to nftables yet.
So it may require to set up both iptables and nftables configs.

I'm pretty sure that this is not possible, at least with NAT. The way they use "both" is that the latest iptables commands are really not iptables commands, they're new nftables translator commands...

On Debian there is the default package: "iptables-nftables-compat" which provides a bunch of commands that translate. So when you run iptables .... it really converts that and runs "nft" on the conversion.

if you want to use real iptables you must install the iptables package which conflicts with iptables-nftables-compat... but then you also must disable all the nftables modules that conflict... it's a bunch of work to rip out nftables from modern desktops.

EDIT some of that wasn't right, I guess now iptables contains both the new style and the old style, and you have to fiddle around with Debian's alternatives system or something... in any case on a default Debian/Ubuntu install when you run "iptables" it is NOT running the traditional iptables commands, it's the nftables compatibility ones.

1 Like

Yep, you can see the rules added by iptables in the nftables runtime ruleset.
But in case of libvirt, those rules are in a separate table created dynamically.
It's problematic to manage with external tools like firewalld using nftables syntax.

Just to back up a minute - I also expected Ubuntu to use nftables since it is so close to Debian but I don't think it does - at least the 20.04 version I installed. For example: https://www.phoronix.com/scan.php?page=news_item&px=Ubuntu-20.10-Nftables. I don't have my pi handy at the moment. I'll try to double-check. Assuming it is iptables based - is there any simple command in openwrt that would trigger a sequence of configuration commands similar to what is generated by iptables-save? Should restarting the firewall service do this? I haven't been able to reboot inside the container - that tends to terminate nspawn.

1 Like

Nice! Your specific information definitely trumps my more general debian info. Looks like Ubuntu delayed their switch.

I'm hoping OpenWrt will switch to nftables soonish and we can have another long thread on tuning nftables to do good stuff.

should flush all the iptables and create them anew.

Hmmm - in that case I suspect something else is broken. I tried restarting the firewall in an ssh session ("service firewall restart") with no change - iptables-save was still blank. I'll play around a bit more later. Would still be nice to get it running.

It's always nice to have something to look forward to! :grimacing:

Thanks again.

+1 on that. Please let us know if you make it work.

Had a bit of time today and returned to this - and some progress! I still need to play around with various permutations to see which steps are required but the following worked for me:

  • On the host, run the systemctl commands to switch to systemd-networkd (as described in post 20 above).
  • open an ssh session on the lan side & enter fw3 print - this returns nothing
  • in a separate terminal on my laptop, ping 8.8.8.8 also fails.
  • In luci, go to status->firewall (not network->firewall) - the screen will show all empty chains although the counters are counting
  • now re-enter fw3 print - at this point a bunch of firewall settings are displayed, however, they don't appear to be active (the ping command is still failing)
  • at this point any firewall restart will activate the firewall: like fw3 restart
    or service firewall restart or clicking the restart firewall button on the luci interface. Ping 8.8.8.8 comes to life! Also the firewall status screen shows the chains although now counters on each heading remain at 0 packets/bytes.

Note that running in Ubuntu MATE, I can also restart the container from luci (this wasn't working in Raspberry OS) - at which point the firewall breaks again and I have to repeat the process. Also note before the firewall is up my laptop can connect to luci and ssh on the lan or wan side but after getting the firewall working I can only connect on the lan side - which I also take as a good indication of its operation.

Does anyone know what process is being triggered by clicking on the firewall status screen in luci and can I mimic it in a script? I tried uci commit firewall without any luck, nor do any of the simple restart commands work. fw3 print remains blank until I select that screen in luci.

Still, this is pretty amazing - it appears to be working! Just need to figure out how to automate these steps so it starts up correctly.

Thanks for any suggestions.

1 Like

i believe it *might* walk 'fw3 print network / zone" etc. etc.

fair chance the 'init' logic needs reworking to handle 'hostX' naming... ??? or similar... you might try renaming the internal psuedo interfaces and see if that helps...

I think I found a reasonable work-around - once I had the firewall up just use iptables-save > /usr/local/firewall-snapshot.txt. So now after starting the container I run cat /usr/local/firewall-snapshot.txt | iptables-restore and the firewall comes up - as best as I can tell. A couple extra steps when I alter the rules in luci, but not too big a deal. Still have some work to do but perhaps the most challenging part is over - fingers crossed. Thanks to everyone for your help so far!