No internet access from docker continers hosted on OpenWrt

Installing and Using OpenWrt Network and Wireless Configuration
21

Is this behaviour ok?

the output of nft list chain inet fw4 input; nft list chain inet fw4 forward; nft list chain inet fw4 output; nft list chain inet fw4 prerouting; nft list chain inet fw4 input_docker; nft list chain inet fw4 output_docker; nft list chain inet fw4 forward_docker; nft list chain inet fw4 helper_docker; nft list chain inet fw4 accept_from_docker; nft list chain inet fw4 accept_to_docker

before and after running `docker container run -it alpine /bin/ping 8.8.8.8`` on a rebooted system differs

root@OpenWrt:~# diff -ruN before_ping.txt after_ping.txt
--- before_ping.txt	2022-09-15 13:11:25.410131377 +0000
+++ after_ping.txt	2022-09-15 13:12:08.131727766 +0000
@@ -7,6 +7,7 @@
 		tcp flags syn / fin,syn,rst,ack jump syn_flood comment "!fw4: Rate limit TCP syn packets"
 		iifname "br-lan" jump input_lan comment "!fw4: Handle lan IPv4/IPv6 input traffic"
 		iifname "pppoe-wan6" jump input_wan comment "!fw4: Handle wan IPv4/IPv6 input traffic"
+		iifname "docker0" jump input_docker comment "!fw4: Handle docker IPv4/IPv6 input traffic"
 	}
 }
 table inet fw4 {
@@ -15,6 +16,7 @@
 		ct state established,related accept comment "!fw4: Allow forwarded established and related flows"
 		iifname "br-lan" jump forward_lan comment "!fw4: Handle lan IPv4/IPv6 forward traffic"
 		iifname "pppoe-wan6" jump forward_wan comment "!fw4: Handle wan IPv4/IPv6 forward traffic"
+		iifname "docker0" jump forward_docker comment "!fw4: Handle docker IPv4/IPv6 forward traffic"
 		jump handle_reject
 	}
 }
@@ -25,12 +27,14 @@
 		ct state established,related accept comment "!fw4: Allow outbound established and related flows"
 		oifname "br-lan" jump output_lan comment "!fw4: Handle lan IPv4/IPv6 output traffic"
 		oifname "pppoe-wan6" jump output_wan comment "!fw4: Handle wan IPv4/IPv6 output traffic"
+		oifname "docker0" jump output_docker comment "!fw4: Handle docker IPv4/IPv6 output traffic"
 	}
 }
 table inet fw4 {
 	chain prerouting {
 		type filter hook prerouting priority filter; policy accept;
 		iifname "br-lan" jump helper_lan comment "!fw4: Handle lan IPv4/IPv6 helper assignment"
+		iifname "docker0" jump helper_docker comment "!fw4: Handle docker IPv4/IPv6 helper assignment"
 	}
 }
 table inet fw4 {
@@ -55,9 +59,11 @@
 }
 table inet fw4 {
 	chain accept_from_docker {
+		iifname "docker0" counter packets 0 bytes 0 accept comment "!fw4: accept docker IPv4/IPv6 traffic"
 	}
 }
 table inet fw4 {
 	chain accept_to_docker {
+		oifname "docker0" counter packets 6 bytes 552 accept comment "!fw4: accept docker IPv4/IPv6 traffic"
 	}
 }

Before pinging there are no docker0 rules, after they are but its broken

22

do you use reload_config; ifup docker or the opposite order?

and no, ruleset should not change between ping. the correct order of things is: create bridge+interface, bring it up and notify fw4 about a new interface to insert the interface related rules. my guess ifup notifies fw4 via ubus call that interface is up hence i suggest to be that command as last. but uciadd which creates docker0 device+interface only runs at boot, so if you start/stop dockerd service it will not trigger network stuff changes. so if there is no lock somewhere this is not normal that ruleset changes when you ping from container. maybe you should wait network to settle down and only after do ping test, then if that solves the problem add some delay to init script.

the ruleset by the way is correct, this is the expected.

23

Hi @grrr2

does it correct ?

image
image897×268 19 KB

it solves my problem but is it safe to do this (forward docker to wan) ?
and should i leave all to accept (input, output, forward), or should i change one of them to another value ?
please answer me kindly, as i'm noob in networking

1 Like
24

hi,

it depends what you want / need.

X forward to Y is unidirectional, allowing traffic initiated in zone X to be forwarded to zone Y - but not vice versa! even reply traffic is prohibited by default, you'll need extra rule which usually is done using conntrack module (traffic marked as related+established, i.e. the reply) is allowed back from Y to X.

so if your docker needs to go directly to WAN then this rule is correct. if you set additional protection (which is commented out by default) in the docker config file, it will drop everything which is not related traffic. so recommend to enable that.

but. dockerd is using iptables and there is slow to no progress to make it nftables compatible, thus you might want to add fw4 rules just to be on the safe side and not purely rely on dockerd config.

and you can verify if docker->wan forward is correct or not by trying to connect to docker instance from wan. if without explicit allow rule you cannot connect then you are protected.

25

Hi @grrr2 @brun0 !

I have the same problem: network works only in the first docker container. After the first one, there is no network inside docker containers. I've applied all fixes and I can't seem to figure it out. Any ideas?

I'm running OpenWrt 22.03.3, it was working on 21.02.3.

26

So I've captured logs from logread from two runs of ping container:

docker container run -it alpine /bin/ping 8.8.8.8

First one is from working networking:

Second one is when networking is not working:

Honestly I can't seem to find what is wrong. I see a problem with resetting conntrack state, but I don't know how to solve it.

For now, a workaround was to run containers in host network.

Maybe I will comeback to it when I find time, but I'm no OpenWrt expert.

Thanks!

27

My solution was very simple as descibed in /etc/config/dockerd since I don't need fw4 features.

Docker doesn't work well out of the box with fw4. This is because Docker relies on a compatibility layer that
naively translates iptables rules. For the best compatibility replace the following dependencies:
firewall4 -> firewall
iptables-nft -> iptables-legacy
ip6tables-nft -> ip6tables-legacy

Runs perfect sinc nearly half a year by now

2 Likes
28

between 21.x and 22.x there was a big change, fw3 vs fw4. former is iptables based, latter is nftables. and docker is not (yet) nftables compatible. also docker run vs docker-compose works differently in terms of network usage, so it is not that convenient to use docker on owrt ... at least this is my experience. it is not owrt's fault, the docker package is just one package of the thousands which is more or less maintained and/or aligned with owrt network/firewall/uci architecture.
if docker is critical for you and want more comfort think about remaining on 21.x.
it is not impossible to use on 22.x though.

1 Like
29

You can use fw3 on 22.x by replaceing the pakages. Docker and the firewall luci part is working fine with 22.x and legacy firewall.

2 Likes
30

I don’t know if it helps in your case but have you tried disabling bridge firewalling?

1 Like
31

Hi @Hudra !

Unfortunately disabling bridge firewalling didn't help, but thanks!

I will try switching to iptables next.

32

Hi @brun0 !

Thanks for your reply, I've replaced firewall4 with firewall but I still have this problem, that only first container has internet access.

[19/02/23 | 21:05] ~ ➜  opkg list-installed | grep firewall
firewall - 2022-02-17-4cd7d4f3-3
luci-app-firewall - git-22.089.67563-7e3c1b4
[19/02/23 | 21:05] ~ ➜  


[19/02/23 | 21:05] ~ ➜  opkg list-installed | grep legacy  
ip6tables-zz-legacy - 1.8.7-7
iptables-zz-legacy - 1.8.7-7
xtables-legacy - 1.8.7-7
[19/02/23 | 21:05] ~ ➜  

[19/02/23 | 21:05] ~ ➜  service dockerd stop && service dockerd boot
[19/02/23 | 21:06] ~ ➜  docker container run -it alpine /bin/ping 8.8.8.8

PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=118 time=5.436 ms
64 bytes from 8.8.8.8: seq=1 ttl=118 time=5.883 ms
^C
--- 8.8.8.8 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 5.436/5.659/5.883 ms
Execution time: 0h:00m:03s sec                                                                                                                                                                                     
[19/02/23 | 21:06] ~ ➜  docker container run -it alpine /bin/ping 8.8.8.8

PING 8.8.8.8 (8.8.8.8): 56 data bytes
^C
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss
Execution time: 0h:00m:03s sec                                                                                                                                                                                     
[19/02/23 | 21:07] ~ ➜

I don't have iptables-legacy I have iptables-zz-legacy. Same goes for ip6tables.

BTW did you do clean reinstall before switching from firewall to firewall?

Also do you have some conntrack package installed?

I have all the fixes from the thread applied.

Thanks!

33

Do you have instructions to do it? I just remove fw4 packages and install fw3 packages? Simple as that? Which packages exactly?

1 Like
34

brun0 wrote:

firewall4 -> firewall
iptables-nft -> iptables-legacy
ip6tables-nft -> ip6tables-legacy