No internet access from docker continers hosted on OpenWrt

Is this behaviour ok?

the output of nft list chain inet fw4 input; nft list chain inet fw4 forward; nft list chain inet fw4 output; nft list chain inet fw4 prerouting; nft list chain inet fw4 input_docker; nft list chain inet fw4 output_docker; nft list chain inet fw4 forward_docker; nft list chain inet fw4 helper_docker; nft list chain inet fw4 accept_from_docker; nft list chain inet fw4 accept_to_docker

before and after running `docker container run -it alpine /bin/ping 8.8.8.8`` on a rebooted system differs

root@OpenWrt:~# diff -ruN before_ping.txt after_ping.txt
--- before_ping.txt	2022-09-15 13:11:25.410131377 +0000
+++ after_ping.txt	2022-09-15 13:12:08.131727766 +0000
@@ -7,6 +7,7 @@
 		tcp flags syn / fin,syn,rst,ack jump syn_flood comment "!fw4: Rate limit TCP syn packets"
 		iifname "br-lan" jump input_lan comment "!fw4: Handle lan IPv4/IPv6 input traffic"
 		iifname "pppoe-wan6" jump input_wan comment "!fw4: Handle wan IPv4/IPv6 input traffic"
+		iifname "docker0" jump input_docker comment "!fw4: Handle docker IPv4/IPv6 input traffic"
 	}
 }
 table inet fw4 {
@@ -15,6 +16,7 @@
 		ct state established,related accept comment "!fw4: Allow forwarded established and related flows"
 		iifname "br-lan" jump forward_lan comment "!fw4: Handle lan IPv4/IPv6 forward traffic"
 		iifname "pppoe-wan6" jump forward_wan comment "!fw4: Handle wan IPv4/IPv6 forward traffic"
+		iifname "docker0" jump forward_docker comment "!fw4: Handle docker IPv4/IPv6 forward traffic"
 		jump handle_reject
 	}
 }
@@ -25,12 +27,14 @@
 		ct state established,related accept comment "!fw4: Allow outbound established and related flows"
 		oifname "br-lan" jump output_lan comment "!fw4: Handle lan IPv4/IPv6 output traffic"
 		oifname "pppoe-wan6" jump output_wan comment "!fw4: Handle wan IPv4/IPv6 output traffic"
+		oifname "docker0" jump output_docker comment "!fw4: Handle docker IPv4/IPv6 output traffic"
 	}
 }
 table inet fw4 {
 	chain prerouting {
 		type filter hook prerouting priority filter; policy accept;
 		iifname "br-lan" jump helper_lan comment "!fw4: Handle lan IPv4/IPv6 helper assignment"
+		iifname "docker0" jump helper_docker comment "!fw4: Handle docker IPv4/IPv6 helper assignment"
 	}
 }
 table inet fw4 {
@@ -55,9 +59,11 @@
 }
 table inet fw4 {
 	chain accept_from_docker {
+		iifname "docker0" counter packets 0 bytes 0 accept comment "!fw4: accept docker IPv4/IPv6 traffic"
 	}
 }
 table inet fw4 {
 	chain accept_to_docker {
+		oifname "docker0" counter packets 6 bytes 552 accept comment "!fw4: accept docker IPv4/IPv6 traffic"
 	}
 }

Before pinging there are no docker0 rules, after they are but its broken

do you use reload_config; ifup docker or the opposite order?

and no, ruleset should not change between ping. the correct order of things is: create bridge+interface, bring it up and notify fw4 about a new interface to insert the interface related rules. my guess ifup notifies fw4 via ubus call that interface is up hence i suggest to be that command as last. but uciadd which creates docker0 device+interface only runs at boot, so if you start/stop dockerd service it will not trigger network stuff changes. so if there is no lock somewhere this is not normal that ruleset changes when you ping from container. maybe you should wait network to settle down and only after do ping test, then if that solves the problem add some delay to init script.

the ruleset by the way is correct, this is the expected.

Hi @grrr2

does it correct ?

it solves my problem but is it safe to do this (forward docker to wan) ?
and should i leave all to accept (input, output, forward), or should i change one of them to another value ?
please answer me kindly, as i'm noob in networking

1 Like

hi,

it depends what you want / need.

X forward to Y is unidirectional, allowing traffic initiated in zone X to be forwarded to zone Y - but not vice versa! even reply traffic is prohibited by default, you'll need extra rule which usually is done using conntrack module (traffic marked as related+established, i.e. the reply) is allowed back from Y to X.

so if your docker needs to go directly to WAN then this rule is correct. if you set additional protection (which is commented out by default) in the docker config file, it will drop everything which is not related traffic. so recommend to enable that.

but. dockerd is using iptables and there is slow to no progress to make it nftables compatible, thus you might want to add fw4 rules just to be on the safe side and not purely rely on dockerd config.

and you can verify if docker->wan forward is correct or not by trying to connect to docker instance from wan. if without explicit allow rule you cannot connect then you are protected.

Hi @grrr2 @brun0 !

I have the same problem: network works only in the first docker container. After the first one, there is no network inside docker containers. I've applied all fixes and I can't seem to figure it out. Any ideas?

I'm running OpenWrt 22.03.3, it was working on 21.02.3.

So I've captured logs from logread from two runs of ping container:

docker container run -it alpine /bin/ping 8.8.8.8

First one is from working networking:

Second one is when networking is not working:

Honestly I can't seem to find what is wrong. I see a problem with resetting conntrack state, but I don't know how to solve it.

For now, a workaround was to run containers in host network.

Maybe I will comeback to it when I find time, but I'm no OpenWrt expert.

Thanks!

My solution was very simple as descibed in /etc/config/dockerd since I don't need fw4 features.

Docker doesn't work well out of the box with fw4. This is because Docker relies on a compatibility layer that
naively translates iptables rules. For the best compatibility replace the following dependencies:
firewall4 -> firewall
iptables-nft -> iptables-legacy
ip6tables-nft -> ip6tables-legacy

Runs perfect sinc nearly half a year by now

2 Likes

between 21.x and 22.x there was a big change, fw3 vs fw4. former is iptables based, latter is nftables. and docker is not (yet) nftables compatible. also docker run vs docker-compose works differently in terms of network usage, so it is not that convenient to use docker on owrt ... at least this is my experience. it is not owrt's fault, the docker package is just one package of the thousands which is more or less maintained and/or aligned with owrt network/firewall/uci architecture.
if docker is critical for you and want more comfort think about remaining on 21.x.
it is not impossible to use on 22.x though.

1 Like

You can use fw3 on 22.x by replaceing the pakages. Docker and the firewall luci part is working fine with 22.x and legacy firewall.

2 Likes

I don’t know if it helps in your case but have you tried disabling bridge firewalling?

1 Like

Hi @Hudra !

Unfortunately disabling bridge firewalling didn't help, but thanks!

I will try switching to iptables next.

Hi @brun0 !

Thanks for your reply, I've replaced firewall4 with firewall but I still have this problem, that only first container has internet access.

[19/02/23 | 21:05] ~ ➜  opkg list-installed | grep firewall
firewall - 2022-02-17-4cd7d4f3-3
luci-app-firewall - git-22.089.67563-7e3c1b4
[19/02/23 | 21:05] ~ ➜  


[19/02/23 | 21:05] ~ ➜  opkg list-installed | grep legacy  
ip6tables-zz-legacy - 1.8.7-7
iptables-zz-legacy - 1.8.7-7
xtables-legacy - 1.8.7-7
[19/02/23 | 21:05] ~ ➜  

[19/02/23 | 21:05] ~ ➜  service dockerd stop && service dockerd boot
[19/02/23 | 21:06] ~ ➜  docker container run -it alpine /bin/ping 8.8.8.8

PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=118 time=5.436 ms
64 bytes from 8.8.8.8: seq=1 ttl=118 time=5.883 ms
^C
--- 8.8.8.8 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 5.436/5.659/5.883 ms
Execution time: 0h:00m:03s sec                                                                                                                                                                                     
[19/02/23 | 21:06] ~ ➜  docker container run -it alpine /bin/ping 8.8.8.8

PING 8.8.8.8 (8.8.8.8): 56 data bytes
^C
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss
Execution time: 0h:00m:03s sec                                                                                                                                                                                     
[19/02/23 | 21:07] ~ ➜  


I don't have iptables-legacy I have iptables-zz-legacy. Same goes for ip6tables.

BTW did you do clean reinstall before switching from firewall to firewall?

Also do you have some conntrack package installed?

I have all the fixes from the thread applied.

Thanks!

Do you have instructions to do it? I just remove fw4 packages and install fw3 packages? Simple as that? Which packages exactly?

1 Like

brun0 wrote:

firewall4 -> firewall
iptables-nft -> iptables-legacy
ip6tables-nft -> ip6tables-legacy

just to write that I also have the same problem with docker. my2cents if can help on the analisys

I can add that if (for example) you install nextcloud as per composer below, the two containers within the same nexcloud docker network are unable to talk to each other. If you disable the firewall everything works (but it's not a solution).
To reach the nexcloud container (apache frontend) you must necessarily add it to the "bridge" network, otherwise even by correctly exposing the ports in the nexcloud docker network alone they will not be reached by the LAN.

I am following the continuation of the thread with interest

version: '2'

volumes:
  nextcloud:
  db:

services:
  db:
    image: mariadb:10.6
    restart: always
    command: --transaction-isolation=READ-COMMITTED --log-bin=binlog --binlog-format=ROW
    volumes:
      - db:/var/lib/mysql
    environment:
      - MYSQL_ROOT_PASSWORD=
      - MYSQL_PASSWORD=
      - MYSQL_DATABASE=nextcloud
      - MYSQL_USER=nextcloud
    networks:
      nextcloud:
        ipv4_address: 172.18.0.3

  app:
    image: nextcloud
    restart: always
    ports:
      - 8081:80
    links:
      - db
    volumes:
      - nextcloud:/var/www/html
    environment:
      - MYSQL_PASSWORD=
      - MYSQL_DATABASE=nextcloud
      - MYSQL_USER=nextcloud
      - MYSQL_HOST=db
    networks:
      nextcloud:
        ipv4_address: 172.18.0.2
        
networks:
  nextcloud:
    ipam:
      config:
        - subnet: 172.18.0.0/16

1 Like

Had the same problem with my docker containers not accessing internet, and also some misbehavior talking to each other.

My Nginx Proxy Manager container gave me some errors while downloading ip-ranges
(ERROR - Fetching https://ip-ranges.amazonaws.com/ip-ranges.json).
Also I was unable to create a proxy from NPM (Nginx Proxy Manager) using the name of the container plus port. Like: http://portainer:9000. I had to map the port and also use the host IP address to be able to properly connect. As expected, was unable to request SSL/TLS from LetsEncript.

These steps solved the problem:

1. Did a new image build changing the packets:
firewall4 -> firewall
iptables-nft -> iptables-zz-legacy
ip6tables-nft -> ip6tables-zz-legacy

2. Add firewall rule for interface docker for Input/Output/Foward with Accept to WAN.

Not sure if firewall (fw3) is safe as firewall4 (fw4), but everything is working as expected now.

Date: November 3, 2024
I recently tried OpenWrt version 23.05.5 on my PC. It took me a few days to find the right configuration, but I wanted to share my experience and thank everyone for the amazing updates to OpenWrt and its Docker hosting support. Here are the steps I took to resolve issues with Docker daemon connectivity:

  1. Modify the /etc/config/dockerd file (scroll to the bottom of the file):
### Retain default lines
config firewall 'firewall'
    option device 'docker0'
#   list blocked_interfaces 'wan'   # Comment this line to allow access from the WAN interface
    option extra_iptables_args '--match conntrack ! --ctstate RELATED,ESTABLISHED' # Allow outbound connections (this line is commented by default)
  1. Modify the /etc/config/firewall file :
### Retain default lines
config zone 'docker'
    option input 'ACCEPT'
    option output 'ACCEPT'
    option forward 'ACCEPT'
    option name 'docker'
    list network 'docker'

config forwarding # Allow docker0 to WAN
    option src 'docker'
    option dest 'wan'

config redirect # Expose Portainer to WAN
    option name 'Portainer'
    option src 'wan'
    option src_dport '9002'
    option dest 'lan'
    option dest_ip '172.17.0.1' # Change this IP to match your docker gateway IP
    option dest_port '9002'
    option proto 'tcp'

  1. Reboot the system.
  2. Test internet connectivity within the containers: You can use ping or curl to ensure the container can reach the internet.
docker run --rm -it yauritux/busybox-curl sh

I hope this helps anyone facing similar issues. Thank you again for the fantastic support and updates to OpenWrt!

1 Like