Podman Networking Setup on OpenWrt

benjaminlevine · November 17, 2024, 9:53pm

I’ve been experimenting using Podman on my OpenWrt router to run a rootful Jellyfin container. My goal is to access the Jellyfin container via the router's LAN IP address (192.168.1.1) without needing to specify the container's internal IP address (e.g., 10.88.0.2).

What I’ve Done So Far

Set up Podman and Jellyfin

I followed the OpenWrt Docker Host Guide to configure Podman.
I referred to the Podman Networking Guide to understand the bridge network - the default network.

Expected Behavior

On my desktop PC running Linux Mint, rootful containers configured similarly (using the bridge network) are accessible via the host PC's IP address.
I expected the Jellyfin instance on the router to be accessible at http://192.168.1.1:8096.

Observed Behavior

On the router, I can only access Jellyfin using the container’s internal IP (e.g., http://10.88.0.2:8096).
The ports are not being forwarded/published to 192.168.1.1.
I've also got a lot of repeating messages in the system log indicating that the podman interface isn't coming up properly. Error at the end of the message

Steps I’ve Tried

I followed the suggestions in this GitHub issue, including adjusting firewall rules and ensuring proper port forwarding from the bridge network to the LAN. Unfortunately, this didn’t work for me.

Issue with Fixed IP

In an attempt to simplify access, I’ve tried assigning a fixed IP address to the container during creation. However, this has proven problematic:

Sometimes after rebooting the router or restarting the container, I encounter a Podman IPAM error stating that the IP address is already allocated to another container (the same container ID!).
This forces me to manually resolve the issue by modifying the container’s IP, which is not an ideal solution.

Can anyone help?

root@router1:~# ubus call system board
{
	"kernel": "6.6.60",
	"hostname": "router1",
	"system": "ARMv8 Processor rev 0",
	"model": "Bananapi BPI-R4",
	"board_name": "bananapi,bpi-r4",
	"rootfs_type": "squashfs",
	"release": {
		"distribution": "OpenWrt",
		"version": "SNAPSHOT",
		"revision": "r28072-b902284290",
		"target": "mediatek/filogic",
		"description": "OpenWrt SNAPSHOT r28072-b902284290",
		"builddate": "1731480827"
	}
}

Excerpt of network config:

config interface 'podman'
	option proto 'external'
	option device 'podman0'

Excerpt of firewall config:

config zone
	option name 'podman'
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'ACCEPT'
	option mtu_fix '1'
	list network 'podman'

config forwarding
	option src 'lan'
	option dest 'podman'

config forwarding
	option src 'podman'
	option dest 'wan'

config rule
	option name 'Podman DNS'
	option src 'podman'
	option dest_port '53'
	option target 'ACCEPT'

config rule
	option name 'Podman DHCP'
	list proto 'udp'
	option target 'ACCEPT'
	option src 'podman'

Error:

Sun Nov 17 21:30:06 2024 daemon.notice netifd: Interface 'podman' is now down
Sun Nov 17 21:30:06 2024 daemon.notice netifd: Interface 'podman' is setting up now
Sun Nov 17 21:30:06 2024 daemon.notice netifd: podman (7558): BusyBox v1.36.1 (2024-11-13 06:12:21 UTC) multi-call binary.
Sun Nov 17 21:30:06 2024 daemon.notice netifd: podman (7558):
Sun Nov 17 21:30:06 2024 daemon.notice netifd: podman (7558): Usage: ip [OPTIONS] address|route|link|neigh|rule [ARGS]
Sun Nov 17 21:30:06 2024 daemon.notice netifd: podman (7558):
Sun Nov 17 21:30:06 2024 daemon.notice netifd: podman (7558): OPTIONS := -f[amily] inet|inet6|link | -o[neline]
Sun Nov 17 21:30:06 2024 daemon.notice netifd: podman (7558):
Sun Nov 17 21:30:06 2024 daemon.notice netifd: podman (7558): ip addr add|del IFADDR dev IFACE | show|flush [dev IFACE] [to PREFIX]
Sun Nov 17 21:30:06 2024 daemon.notice netifd: podman (7558): ip route list|flush|add|del|change|append|replace|test ROUTE
Sun Nov 17 21:30:06 2024 daemon.notice netifd: podman (7558): ip link set IFACE [up|down] [arp on|off] [multicast on|off]
Sun Nov 17 21:30:06 2024 daemon.notice netifd: podman (7558): 	[promisc on|off] [mtu NUM] [name NAME] [qlen NUM] [address MAC]
Sun Nov 17 21:30:06 2024 daemon.notice netifd: podman (7558): 	[master IFACE | nomaster] [netns PID]
Sun Nov 17 21:30:06 2024 daemon.notice netifd: podman (7558): ip neigh show|flush [to PREFIX] [dev DEV] [nud STATE]
Sun Nov 17 21:30:06 2024 daemon.notice netifd: podman (7558): ip rule [list] | add|del SELECTOR ACTION
Sun Nov 17 21:30:06 2024 daemon.notice netifd: podman (7558): Failed to parse message data
Sun Nov 17 21:30:06 2024 daemon.notice netifd: podman (7558): BusyBox v1.36.1 (2024-11-13 06:12:21 UTC) multi-call binary.
Sun Nov 17 21:30:06 2024 daemon.notice netifd: podman (7558):
Sun Nov 17 21:30:06 2024 daemon.notice netifd: podman (7558): Usage: ip [OPTIONS] address|route|link|neigh|rule [ARGS]
Sun Nov 17 21:30:06 2024 daemon.notice netifd: podman (7558):
Sun Nov 17 21:30:06 2024 daemon.notice netifd: podman (7558): OPTIONS := -f[amily] inet|inet6|link | -o[neline]
Sun Nov 17 21:30:06 2024 daemon.notice netifd: podman (7558):
Sun Nov 17 21:30:06 2024 daemon.notice netifd: podman (7558): ip addr add|del IFADDR dev IFACE | show|flush [dev IFACE] [to PREFIX]
Sun Nov 17 21:30:06 2024 daemon.notice netifd: podman (7558): ip route list|flush|add|del|change|append|replace|test ROUTE
Sun Nov 17 21:30:06 2024 daemon.notice netifd: podman (7558): ip link set IFACE [up|down] [arp on|off] [multicast on|off]
Sun Nov 17 21:30:06 2024 daemon.notice netifd: podman (7558): 	[promisc on|off] [mtu NUM] [name NAME] [qlen NUM] [address MAC]
Sun Nov 17 21:30:06 2024 daemon.notice netifd: podman (7558): 	[master IFACE | nomaster] [netns PID]
Sun Nov 17 21:30:06 2024 daemon.notice netifd: podman (7558): ip neigh show|flush [to PREFIX] [dev DEV] [nud STATE]
Sun Nov 17 21:30:06 2024 daemon.notice netifd: podman (7558): ip rule [list] | add|del SELECTOR ACTION
Sun Nov 17 21:30:06 2024 daemon.notice netifd: podman (7558): Failed to parse message data
Sun Nov 17 21:30:06 2024 daemon.notice netifd: podman (7558): interface podman0 does not have ip address
Sun Nov 17 21:30:06 2024 daemon.notice netifd: Interface 'podman' is now down
Sun Nov 17 21:30:06 2024 daemon.notice netifd: Interface 'podman' is setting up now

xiaohei · November 18, 2024, 7:44am

Don't run Podman containers on the router for the time being. You can check out this issue

totkeks · November 26, 2024, 3:38pm

That issue is pretty old and from glancing over it, it seems someone managed to get it all running.

What alternatives are there then, if podman isn't working? docker? ujail?

benjaminlevine · November 26, 2024, 6:39pm

@totkeks your post was directed at @xiaohei but here's my current situation:

I do have podman working ok. With the caveat that I have to access to container from its internal IP address e.g. 10.88.0.2:8096 (rather than the router's IP address 192.168.1.1:8096).

I'm happy enough with this, I can setup a convenient name with dnsmasq e.g. list address '/jellyfin/jellyfin.lan/10.88.0.2'

Not ideal but good enough.

If my router loses power then I might end up with the IPAM error stating that the IP address is already allocated to another container. If so, I'll just have to assign a different IP address. Annoying.

In terms of alternatives, I did originally try docker (I liked the app luci-app-dockerman) but it caused a weird error of preventing lan clients pinging each other. See here Firewall Default Settings Blocking Device Communication in LAN Zone

In terms of ujail, I haven't tried,the setup looked very involved.

totkeks · November 26, 2024, 7:27pm

Thanks! I also played around with it the last hours.

Unfortunately, the network backend netavark from podman doesn't support nftables yet. It fails with iptables not found. So, I disabled it in the containers.conf via firewall_driver="none" in [network].

This results of course in the experience you are describing, that port forwards are not configured properly. The port on the router should be blocked by conmon, but there will be no forward between this port and the port on the container.

I'm currently looking on how to let dnsmasq on my router know about the containers. Or configure the firewall automatically. Though I'd rather use the dns method and don't use portforwards.

Besides that, we can just hope that netavark will be updated soon to support nftables on OpenWrt.

benjaminlevine · November 26, 2024, 7:42pm

This https://github.com/containers/netavark/issues/816#issuecomment-1913366128 suggests setting
firewall_driver = "nfttables" as working. I haven't tried this, just a random Google right now.

I guess that would be a firewall in addition to the openwrt one though.

I did also try setting up macvlan network in podman (rather than the default bridge network) to get an IP address via DHCP from openwrt directly. Then I could use static leases to set a fixed IP address and hostname. Didn't work out though, also due to netavark issues, there didn't seem to be a DHCP request received.

totkeks · November 26, 2024, 7:49pm

I tried that and it actually works. Thanks! Odd, that it isn't the default setting in the configuration. It sets the correct port forwards. Not sure if it interferes with fw4 / etc/config/firewall though.

I will try your idea now, as I'd like this more, even though letting containers access the router reduces overall security.

benjaminlevine · November 26, 2024, 8:05pm

Great! Working for me too!
I'm running podman 5.2.2 and netavark 1.12.2

For the macvlan option I did have to install the package kmod-macvlan. Let me know how you get on

totkeks · November 26, 2024, 9:02pm

Same versions here too.

I also tried ipvlan. Both works. Though I'm getting these strange messages in my log

eth0: renamed from mv-0ym3Xyer7L

when using either. I guess this is because of podman pretending to be eth0 or something.

Unfortunately, the dhcp option doesn't work:

     "ipam_options": {
          "driver": "dhcp"
     }

because the netavark-dhcp-proxy-client is not started automatically with podman or a network that uses dhcp. And I couldn't get it to work properly, when manually starting it.

Maybe someone else knows how to.

I'm going back to just updating my dns server solution.

benjaminlevine · November 26, 2024, 9:14pm

I had a similar experience - driver dhcp didn't work, even when I manually tried to start netavark-dhcp-proxy-client

totkeks · November 27, 2024, 6:08pm

I got it to work.

Disabling the firewall driver:

firewall_driver = "none"

Disabling dns and using the host-local driver for IPAM.

"dns_enabled": false,
     "ipam_options": {
          "driver": "host-local"
     }

And this script in /etc/hotplug.d/net/99-podman-dns:

#!/bin/sh

[[ $DEVICENAME == veth* ]] || exit 0

logger -t podman "Containers changed, rebuilding DNS hosts"

HOSTS_FILE="/tmp/hosts/podman"
echo "# Container DNS entries" > "$HOSTS_FILE"

podman ps -q| while read -r CONTAINER_ID; do
        NAME=$(podman inspect --format '{{.Name}}' "$CONTAINER_ID")
        IP=$(podman inspect --format '{{.NetworkSettings.IPAddress}}' "$CONTAINER_ID")

        if [ -n "$IP" ]; then
                echo "$IP    $NAME.<your dns domain>" >> "$HOSTS_FILE"
        fi
done

This uses the default config of dnsmasq, which makes use of the /tmp/hosts/ directory for additional hosts.

There is just one more change I had to make for this to work without having to reload dnsmasq (service dnsmasq reload).

Had to replace this code in /etc/init.d/dnsmasq:

if [ "$ignore_hosts_dir" = "1" ]; then
    xappend "--addn-hosts=$HOSTFILE"
    append EXTRA_MOUNT "$HOSTFILE"
else
    xappend "--addn-hosts=$HOSTFILE_DIR"
    append EXTRA_MOUNT "$HOSTFILE_DIR"
fi

because it uses the --addn-hosts dnsmasq flag to add the directory. There is another flag called --hostsdir that has the added benefit of watching the files for changes and update (add/remove) DNS entries automatically.

Just the lower part, where they actually use the directory in the code:

else
    xappend "--hostsdir=$HOSTFILE_DIR"
    append EXTRA_MOUNT "$HOSTFILE_DIR"
fi

After that, I got automated DNS entries for the containers, when I started them and they got automatically removed, when I stopped them.

The only question or option is, whether to use the container's name or hostname. Those can be different. Hostname and name are usually those random IDs, and you have to overwrite them separately via --name or --hostname. I'm going with the name for now.

system · December 7, 2024, 6:09pm

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.