Hi,
Not sure if dockerd package should support nftables (via iptables-legacy) or not even with wrapper. Also fw4 may have possible bugs which manifested while i am using docker but may not necessarily due to docker. (this is a long post, sorry)
Any insight is welcomed.
The environment
I am using openwrt 22.03-rc5 x86 with overlay filesystem (in VMware, probably that's not relevant though).
after installing dockerd, docker and docker-compose packages, and enable/start dockerd service everything looks ok. using package default dockerd config file, only enabling extra args in firewall section as below:
/etc/config/dockerd
config firewall 'firewall'
option device 'docker0'
list blocked_interfaces 'wan'
option extra_iptables_args '--match conntrack ! --ctstate RELATED,ESTABLISHED' # allow outbound connections
dockerd creates basic config including network and firewall related as below:
Summary
$ uci show | grep docker
dockerd.globals=globals
dockerd.globals.data_root='/opt/docker/'
dockerd.globals.log_level='info'
dockerd.globals.iptables='1'
dockerd.firewall=firewall
dockerd.firewall.device='docker0'
dockerd.firewall.blocked_interfaces='wan'
dockerd.firewall.extra_iptables_args='--match conntrack ! --ctstate RELATED,ESTABLISHED'
firewall.docker=zone
firewall.docker.input='ACCEPT'
firewall.docker.output='ACCEPT'
firewall.docker.forward='ACCEPT'
firewall.docker.name='docker'
firewall.docker.network='docker'
firewall.@forwarding[1].src='docker'
firewall.@forwarding[2].dest='docker'
network.docker=interface
network.docker.device='docker0'
network.docker.proto='none'
network.docker.auto='0'
network.@device[2].name='docker0'
$ uci show firewall | grep forwarding | grep '1\|2'
firewall.@forwarding[1]=forwarding
firewall.@forwarding[1].src='docker'
firewall.@forwarding[1].dest='wan'
firewall.@forwarding[2]=forwarding
firewall.@forwarding[2].src='lan'
firewall.@forwarding[2].dest='docker'
as docker using iptables as a dependency iptables-zz-legacy is intalled too:
Summary
$ opkg depends dockerd
dockerd depends on:
libc
ca-certificates
containerd
iptables
iptables-mod-extra
ip6tables
kmod-ipt-nat6
libseccomp
kmod-ipt-nat
kmod-ipt-physdev
kmod-nf-ipvs
kmod-veth
libnetwork
tini
uci-firewall
$ opkg list-installed | grep iptables
iptables-mod-extra - 1.8.7-7
iptables-zz-legacy - 1.8.7-7
I want to use adguard home docker container so it requires:
- to be available for lan clients as DNS resolver
- to access wan to reach upstream DNS servers
Now, the findings/problems/questions.
dockerd automatically creates via iptables docker related firewall rules:
Summary
$ iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N DOCKER
-N DOCKER-ISOLATION-STAGE-1
-N DOCKER-ISOLATION-STAGE-2
-N DOCKER-USER
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -i eth0 -o docker0 -m conntrack ! --ctstate RELATED,ESTABLISHED -j REJECT --reject-with icmp-port-unreachable
-A DOCKER-USER -j RETURN
using the approach am used to, on top of these rules, i allow docker -> wan forwarding in firewall zone config, but with the extra args in dockerd config it is limitied to related, established traffic. so I'd hope am good to go in a reasonable secure way.
But this is not working "out-of-box" as expected.
the problems:
- some rule are missing from fw4/nftables still, somehow there is a disconnect on the dcokerd-iptables-nftables path.
the system generated fw4 rules are:
$ nft list ruleset
table inet fw4 {
chain input {
type filter hook input priority filter; policy accept;
iifname "lo" accept comment "!fw4: Accept traffic from loopback"
ct state established,related accept comment "!fw4: Allow inbound established and related flows"
tcp flags syn / fin,syn,rst,ack jump syn_flood comment "!fw4: Rate limit TCP syn packets"
iifname "br-lan" jump input_lan comment "!fw4: Handle lan IPv4/IPv6 input traffic"
iifname "eth0" jump input_wan comment "!fw4: Handle wan IPv4/IPv6 input traffic"
iifname "eth4" jump input_console comment "!fw4: Handle console IPv4/IPv6 input traffic"
iifname "br-guest" jump input_guest comment "!fw4: Handle guest IPv4/IPv6 input traffic"
}
chain forward {
type filter hook forward priority filter; policy drop;
ct state established,related accept comment "!fw4: Allow forwarded established and related flows"
iifname "br-lan" jump forward_lan comment "!fw4: Handle lan IPv4/IPv6 forward traffic"
iifname "eth0" jump forward_wan comment "!fw4: Handle wan IPv4/IPv6 forward traffic"
iifname "eth4" jump forward_console comment "!fw4: Handle console IPv4/IPv6 forward traffic"
iifname "br-guest" jump forward_guest comment "!fw4: Handle guest IPv4/IPv6 forward traffic"
iifname "docker0" jump forward_docker comment "fix: handle docker forward traffic" # (1) <--- this rule is missing by default
jump handle_reject
}
chain output {
type filter hook output priority filter; policy accept;
oifname "lo" accept comment "!fw4: Accept traffic towards loopback"
ct state established,related accept comment "!fw4: Allow outbound established and related flows"
oifname "br-lan" jump output_lan comment "!fw4: Handle lan IPv4/IPv6 output traffic"
oifname "eth0" jump output_wan comment "!fw4: Handle wan IPv4/IPv6 output traffic"
oifname "eth4" jump output_console comment "!fw4: Handle console IPv4/IPv6 output traffic"
oifname "br-guest" jump output_guest comment "!fw4: Handle guest IPv4/IPv6 output traffic"
}
chain prerouting {
type filter hook prerouting priority filter; policy accept;
iifname "br-lan" jump helper_lan comment "!fw4: Handle lan IPv4/IPv6 helper assignment"
iifname "eth4" jump helper_console comment "!fw4: Handle console IPv4/IPv6 helper assignment"
iifname "br-guest" jump helper_guest comment "!fw4: Handle guest IPv4/IPv6 helper assignment"
}
chain handle_reject {
meta l4proto tcp reject with tcp reset comment "!fw4: Reject TCP traffic"
reject comment "!fw4: Reject any other traffic"
}
chain syn_flood {
limit rate 25/second burst 50 packets return comment "!fw4: Accept SYN packets below rate-limit"
drop comment "!fw4: Drop excess packets"
}
chain input_lan {
jump accept_from_lan
}
chain output_lan {
jump accept_to_lan
}
chain forward_lan {
jump accept_to_wan comment "!fw4: Accept lan to wan forwarding"
jump accept_to_docker comment "!fw4: Accept lan to docker forwarding"
jump accept_to_lan
}
chain helper_lan {
}
chain accept_from_lan {
iifname "br-lan" counter packets 1 bytes 48 accept comment "!fw4: accept lan IPv4/IPv6 traffic"
}
chain accept_to_lan {
oifname "br-lan" counter packets 0 bytes 0 accept comment "!fw4: accept lan IPv4/IPv6 traffic"
}
chain input_wan {
meta nfproto ipv4 udp dport 68 counter packets 0 bytes 0 accept comment "!fw4: Allow-DHCP-Renew"
icmp type echo-request counter packets 0 bytes 0 accept comment "!fw4: Allow-Ping"
meta nfproto ipv4 meta l4proto igmp counter packets 0 bytes 0 accept comment "!fw4: Allow-IGMP"
meta nfproto ipv6 udp dport 546 counter packets 0 bytes 0 accept comment "!fw4: Allow-DHCPv6"
ip6 saddr fe80::/10 icmpv6 type . icmpv6 code { mld-listener-query . no-route, mld-listener-report . no-route, mld-listener-done . no-route, mld2-listener-report . no-route } counter packets 0 bytes 0 accept comment "!fw4: Allow-MLD"
icmpv6 type { destination-unreachable, time-exceeded, echo-request, echo-reply, nd-router-solicit, nd-router-advert } limit rate 1000/second counter packets 0 bytes 0 accept comment "!fw4: Allow-ICMPv6-Input"
icmpv6 type . icmpv6 code { packet-too-big . no-route, parameter-problem . no-route, nd-neighbor-solicit . no-route, nd-neighbor-advert . no-route, parameter-problem . admin-prohibited } limit rate 1000/second counter packets 0 bytes 0 accept comment "!fw4: Allow-ICMPv6-Input"
jump reject_from_wan
}
chain output_wan {
jump accept_to_wan
}
chain forward_wan {
icmpv6 type { destination-unreachable, time-exceeded, echo-request, echo-reply } limit rate 1000/second counter packets 0 bytes 0 accept comment "!fw4: Allow-ICMPv6-Forward"
icmpv6 type . icmpv6 code { packet-too-big . no-route, parameter-problem . no-route, parameter-problem . admin-prohibited } limit rate 1000/second counter packets 0 bytes 0 accept comment "!fw4: Allow-ICMPv6-Forward"
meta l4proto esp counter packets 0 bytes 0 jump accept_to_lan comment "!fw4: Allow-IPSec-ESP"
udp dport 500 counter packets 0 bytes 0 jump accept_to_lan comment "!fw4: Allow-ISAKMP"
jump reject_to_wan
}
chain accept_to_wan {
oifname "eth0" counter packets 211 bytes 16020 accept comment "!fw4: accept wan IPv4/IPv6 traffic"
}
chain reject_from_wan {
iifname "eth0" counter packets 2732 bytes 748966 jump handle_reject comment "!fw4: reject wan IPv4/IPv6 traffic"
}
chain reject_to_wan {
oifname "eth0" counter packets 0 bytes 0 jump handle_reject comment "!fw4: reject wan IPv4/IPv6 traffic"
}
chain input_console {
ct status dnat accept comment "!fw4: Accept port redirections"
jump accept_from_console
}
chain output_console {
jump accept_to_console
}
chain forward_console {
ct status dnat accept comment "!fw4: Accept port forwards"
jump reject_to_console
}
chain helper_console {
}
chain accept_from_console {
iifname "eth4" counter packets 2735 bytes 749118 accept comment "!fw4: accept console IPv4/IPv6 traffic"
}
chain accept_to_console {
oifname "eth4" counter packets 6 bytes 1080 accept comment "!fw4: accept console IPv4/IPv6 traffic"
}
chain reject_to_console {
oifname "eth4" counter packets 0 bytes 0 jump handle_reject comment "!fw4: reject console IPv4/IPv6 traffic"
}
chain input_docker {
jump accept_from_docker
}
chain output_docker {
jump accept_to_docker
}
chain forward_docker {
jump accept_to_wan comment "!fw4: Accept docker to wan forwarding"
jump accept_to_docker
}
chain helper_docker {
}
chain accept_from_docker {
}
chain accept_to_docker { # (2) <--- this section is empty by default
oifname "docker0" accept comment "fix: accept traffic to docker"
}
chain dstnat {
type nat hook prerouting priority dstnat; policy accept;
iifname "eth4" jump dstnat_console comment "!fw4: Handle console IPv4/IPv6 dstnat traffic"
iifname "br-guest" jump dstnat_guest comment "!fw4: Handle guest IPv4/IPv6 dstnat traffic"
}
chain srcnat {
type nat hook postrouting priority srcnat; policy accept;
oifname "eth0" jump srcnat_wan comment "!fw4: Handle wan IPv4/IPv6 srcnat traffic"
}
chain srcnat_wan {
meta nfproto ipv4 masquerade comment "!fw4: Masquerade IPv4 wan traffic"
}
chain dstnat_console {
[ reduced ]
}
chain raw_prerouting {
type filter hook prerouting priority raw; policy accept;
}
chain raw_output {
type filter hook output priority raw; policy accept;
}
chain mangle_prerouting {
type filter hook prerouting priority mangle; policy accept;
}
chain mangle_postrouting {
type filter hook postrouting priority mangle; policy accept;
}
chain mangle_input {
type filter hook input priority mangle; policy accept;
}
chain mangle_output {
type route hook output priority mangle; policy accept;
}
chain mangle_forward {
type filter hook forward priority mangle; policy accept;
iifname "eth0" tcp flags syn tcp option maxseg size set rt mtu comment "!fw4: Zone wan IPv4/IPv6 ingress MTU fixing"
oifname "eth0" tcp flags syn tcp option maxseg size set rt mtu comment "!fw4: Zone wan IPv4/IPv6 egress MTU fixing"
}
chain dstnat_lan {
}
chain input_guest {
ct status dnat accept comment "!fw4: Accept port redirections"
jump accept_from_guest
}
chain output_guest {
jump accept_to_guest
}
chain forward_guest {
jump accept_to_wan comment "!fw4: Accept guest to wan forwarding"
ct status dnat accept comment "!fw4: Accept port forwards"
jump reject_to_guest
}
chain helper_guest {
}
chain accept_from_guest {
iifname "br-guest" counter packets 0 bytes 0 accept comment "!fw4: accept guest IPv4/IPv6 traffic"
}
chain accept_to_guest {
oifname "br-guest" counter packets 0 bytes 0 accept comment "!fw4: accept guest IPv4/IPv6 traffic"
}
chain reject_to_guest {
oifname "br-guest" counter packets 0 bytes 0 jump handle_reject comment "!fw4: reject guest IPv4/IPv6 traffic"
}
chain dstnat_guest {
[ reduced ]
}
}
the two missing rules are required to allow container to send traffic to wan and receive reply traffic.
- using
config includein firewall works partially.
config include
option type 'nftables'
option path '/etc/forward.nft'
option position 'chain-append'
option chain 'forward'
config include
option type 'nftables'
option path '/etc/accept_to_docker.nft'
option position 'chain-prepend'
option chain 'accept_to_docker'
$ cat /etc/forward.nft
iifname docker0 jump forward_docker comment "fix: handle docker forward traffic"
$ cat /etc/accept_to_docker.nft
oifname docker0 accept comment "fix: accept traffic to docker"
while the first include works and append the missing rule (1), the second for some reason does not although service firewall restart completes without error. i have to add manually the missing rule (2). is this a (known) bug or i miss something? setting position either direction (append or prepend) has no effect by the way.
- a strange fw4 thing: usually if i add comments to an
/etc/config/<service>file it remains even if reboot server and or restart service. this is not the case withfirewallservice: if i comment out a section and restart the service the whole commented section disappear from the file.
any thoughts please?
thanks