Wireguard not starting - docker

Hello! I'm currently troubleshooting an issue with WireGuard on my OpenWrt setup running inside a Docker container.

Just to clarify, the container was initially a working LXC container of openwrt. From there I exported and then turned into a Docker image.

My wireguard device is named mullvad.

/ # wg show
/ #

I am trying to diagnose why WireGuard isn't functioning as expected. Any guidance or pointers in the right direction would be greatly appreciated.

#Previously# I encountered the following error in the logs:

daemon.notice netifd:  RTNETLINK answers: Operation not permitted
daemon.notice netifd: Unable to modify interface: Operation not permitted

By adding "RUN chmod +s /sbin/tc" to the Dockerfile those errors went away.

#But now# the logs do not show that. Instead just says:

Thu Apr 11 22:45:08 2024 user.notice ucitrack: Setting up non-init /etc/config/fstab reload handler: /sbin/block mount
Thu Apr 11 22:45:08 2024 daemon.notice netifd: Interface 'mullvad' is now down 
Thu Apr 11 22:45:08 2024 daemon.info procd: - init complete -

Dockerfile

FROM scratch

ADD openwrt-lxc.tar /

RUN chmod +s /sbin/tc

CMD ["/sbin/init"]

Docker-Compose

version: '3.8'

services:
  openwrt-container:
    image: openwrt-lxc:latest
    container_name: openwrt-lxc
    volumes:
      - /home/ubuntu/openwrt/data/rootfs:/mnt/rootfs
      - /home/ubuntu/openwrt/copyback:/mnt/copyback
    restart: unless-stopped
    cap_add:
      - NET_ADMIN
    networks:
      - openwrt_network

networks:
  openwrt_network:
    driver: bridge

My network and firewall:

network.loopback=interface
network.loopback.device='lo'
network.loopback.proto='static'
network.loopback.ipaddr='127.0.0.1'
network.loopback.netmask='255.0.0.0'
network.@device[0]=device
network.@device[0].name='br-lan'
network.@device[0].type='bridge'
network.@device[0].ports='eth2' 'eth3'
network.lan=interface
network.lan.device='br-lan'
network.lan.proto='static'
network.lan.netmask='255.255.255.0'
network.lan.ipaddr=' 192.168.3.1'
network.wan=interface
network.wan.device='eth0'
network.wan.proto='dhcp'
network.mullvad=interface
network.mullvad.proto='wireguard'
network.mullvad.force_link='1'
network.mullvad.private_key='='
network.mullvad.addresses='...58/32'
network.@wireguard_mullvad[0]=wireguard_mullvad
network.@wireguard_mullvad[0].description='wg-'
network.@wireguard_mullvad[0].public_key='/='
network.@wireguard_mullvad[0].allowed_ips='0.0.0.0/0'
network.@wireguard_mullvad[0].endpoint_host='...2'
network.@wireguard_mullvad[0].endpoint_port=''
network.@wireguard_mullvad[0].persistent_keepalive=''
firewall.@defaults[0]=defaults
firewall.@defaults[0].syn_flood='1'
firewall.@defaults[0].input='ACCEPT'
firewall.@defaults[0].output='ACCEPT'
firewall.@defaults[0].forward='REJECT'
firewall.@defaults[0].disable_ipv6='1'
firewall.@zone[0]=zone
firewall.@zone[0].name='lan'
firewall.@zone[0].input='ACCEPT'
firewall.@zone[0].output='ACCEPT'
firewall.@zone[0].forward='ACCEPT'
firewall.@zone[0].network='lan'
firewall.@zone[1]=zone
firewall.@zone[1].name='wan'
firewall.@zone[1].network='wan'
firewall.@zone[1].input='REJECT'
firewall.@zone[1].output='ACCEPT'
firewall.@zone[1].forward='REJECT'
firewall.@zone[1].masq='1'
firewall.@zone[1].mtu_fix='1'
firewall.@rule[0]=rule
firewall.@rule[0].name='Allow-DHCP-Renew'
firewall.@rule[0].src='wan'
firewall.@rule[0].proto='udp'
firewall.@rule[0].dest_port='68'
firewall.@rule[0].target='ACCEPT'
firewall.@rule[0].family='ipv4'
firewall.@rule[1]=rule
firewall.@rule[1].name='Allow-Ping'
firewall.@rule[1].src='wan'
firewall.@rule[1].proto='icmp'
firewall.@rule[1].icmp_type='echo-request'
firewall.@rule[1].family='ipv4'
firewall.@rule[1].target='ACCEPT'
firewall.@rule[2]=rule
firewall.@rule[2].name='Block-IGMP'
firewall.@rule[2].src='wan'
firewall.@rule[2].proto='igmp'
firewall.@rule[2].family='ipv4'
firewall.@rule[2].target='DROP'
firewall.@rule[3]=rule
firewall.@rule[3].name='Allow-IPSec-ESP'
firewall.@rule[3].src='wan'
firewall.@rule[3].dest='lan'
firewall.@rule[3].proto='esp'
firewall.@rule[3].target='ACCEPT'
firewall.@rule[4]=rule
firewall.@rule[4].name='Allow-ISAKMP'
firewall.@rule[4].src='wan'
firewall.@rule[4].dest='lan'
firewall.@rule[4].dest_port='500'
firewall.@rule[4].proto='udp'
firewall.@rule[4].target='ACCEPT'
firewall.@rule[5]=rule
firewall.@rule[5].name='Restrict-Plex-Access'
firewall.@rule[5].src='lan'
firewall.@rule[5].dest='wan'
firewall.@rule[5].src_ip='...'
firewall.@rule[5].target='REJECT'
firewall.@zone[2]=zone
firewall.@zone[2].name='mullvad'
firewall.@zone[2].input='REJECT'
firewall.@zone[2].output='ACCEPT'
firewall.@zone[2].forward='REJECT'
firewall.@zone[2].masq='1'
firewall.@zone[2].mtu_fix='1'
firewall.@zone[2].network='mullvad'
firewall.@forwarding[0]=forwarding
firewall.@forwarding[0].src='lan'
firewall.@forwarding[0].dest='mullvad'
firewall.@rule[6]=rule
firewall.@rule[6].name='Allow-DHCP-Renew'
firewall.@rule[6].src='wan'
firewall.@rule[6].proto='udp'
firewall.@rule[6].dest_port='68'
firewall.@rule[6].target='ACCEPT'
firewall.@rule[6].family='ipv4'
firewall.@rule[7]=rule
firewall.@rule[7].name='Allow-WAN-to-Mullvad'
firewall.@rule[7].src='wan'
firewall.@rule[7].dest='mullvad'
firewall.@rule[7].target='ACCEPT'
firewall.@rule[8]=rule
firewall.@rule[8].name='Block-LAN-to-WAN'
firewall.@rule[8].src='lan'
firewall.@rule[8].dest='wan'
firewall.@rule[8].target='REJECT'
firewall.@rule[9]=rule
firewall.@rule[9].in='lan'
firewall.@rule[9].lookup='100'
firewall.@route[0]=route
firewall.@route[0].interface='vpn'
firewall.@route[0].target='0.0.0.0'
firewall.@route[0].netmask='0.0.0.0'
firewall.@route[0].table='100'

Thank you for any help or direction!

Your situation is slightly confusing...

Is OpenWrt running in a docker container on top of Ubunbu? What is the intent of this OpenWrt instance?

I made edits. I hope its easier to understand.
Openwrt inside docker - wireguard does not work. Thats really it, I just wanted to give all the information that may be relevant.

Such as, the docker openwrt image is created from my lxc's openwrt export.

Running OpenWrt inside a docker container is not generally recommended, especially if you are trying to setup an outbound VPN connection to a commercial VPN provider -- it is only useful if you can get all the traffic routed through the docker container which is not trivial.

that said, let's see the following from your openwrt config:

Please connect to your OpenWrt device using ssh and copy the output of the following commands and post it here using the "Preformatted text </> " button:
grafik
Remember to redact passwords, MAC addresses and any public IP addresses you may have:

ubus call system board
cat /etc/config/network
cat /etc/config/firewall

/ # ubus call system board

{
        "kernel": "6.5.0-1020-oracle",
        "hostname": "cc71286129ee",
        "system": "ARMv8 Processor rev 1",
        "release": {
                "distribution": "OpenWrt",
                "version": "21.02.0-rc3",
                "revision": "r16172-2aba3e9784",
                "target": "armvirt/64",
                "description": "OpenWrt 21.02.0-rc3 r16172-2aba3e9784"
        }
}

/ # cat /etc/config/network

config interface 'loopback'
        option device 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config device
        option name 'br-lan'
        option type 'bridge'
        list ports 'eth2'
        list ports 'eth3'

config interface 'lan'
        option device 'br-lan'
        option proto 'static'
        option netmask '255.255.255.0'
        option ipaddr ' 192.168.3.1'

config interface 'wan'
        option device 'eth0'
        option proto 'dhcp'

config interface 'mullvad'
        option proto 'wireguard'
        option force_link '1'
        option private_key '='
        list addresses '.../'


config wireguard_mullvad
        option description '---'
        option public_key '='
        list allowed_ips '0.0.0.0/0'
        option endpoint_host '...'
        option endpoint_port ''
        option persistent_keepalive ''

/ # cat /etc/config/firewall

config defaults
    option syn_flood        '1'
    option input            'ACCEPT'
    option output           'ACCEPT'
    option forward          'REJECT'
    option disable_ipv6     '1'  # Disable IPv6

config zone
    option name             'lan'
    option input            'ACCEPT'
    option output           'ACCEPT'
    option forward          'ACCEPT'
    list   network          'lan'

config zone
    option name             'wan'
    list   network          'wan'
    option input            'REJECT'
    option output           'ACCEPT'
    option forward          'REJECT'
    option masq             '1'
    option mtu_fix          '1'

config rule
    option name             'Allow-DHCP-Renew'
    option src              'wan'
    option proto            'udp'
    option dest_port        '68'
    option target           'ACCEPT'
    option family           'ipv4'


config rule
    option name             'Allow-Ping'
    option src              'wan'
    option proto            'icmp'
    option icmp_type        'echo-request'
    option family           'ipv4'
    option target           'ACCEPT'

# Block IGMP traffic
config rule
    option name             'Block-IGMP'
    option src              'wan'
    option proto            'igmp'
    option family           'ipv4'
    option target           'DROP'

config rule
    option name             'Allow-IPSec-ESP'
    option src              'wan'
    option dest             'lan'
    option proto            'esp'
    option target           'ACCEPT'

config rule
    option name             'Allow-ISAKMP'
    option src              'wan'
    option dest             'lan'
    option dest_port        '500'
    option proto            'udp'
    option target           'ACCEPT'

config rule
    option name             'Restrict-Plex-Access'
    option src              'lan'
    option dest             'wan'
    option src_ip           ''
    option target           'REJECT'

config zone
    option name         'mullvad'
    option input        'REJECT'
    option output       'ACCEPT'
    option forward      'REJECT'
    option masq         '1'
    option mtu_fix      '1'
    list network        'mullvad'

config forwarding
    option src          'lan'
    option dest         'mullvad'

config rule
    option name         'Allow-DHCP-Renew'
    option src          'wan'
    option proto        'udp'
    option dest_port    '68'
    option target       'ACCEPT'
    option family       'ipv4'

config rule
    option name         'Allow-WAN-to-Mullvad'
    option src          'wan'
    option dest         'mullvad'
    option target       'ACCEPT'

config rule
    option name         'Block-LAN-to-WAN'
    option src          'lan'
    option dest         'wan'
    option target       'REJECT'

config rule
    option in           'lan'
    option lookup       '100'

config route
    option interface    'vpn'
    option target       '0.0.0.0'
    option netmask      '0.0.0.0'
    option table        '100'

This is not from the official OpenWrt project. You'll need to ask the people who maintain the build you are using.

I took the image from

https://downloads.openwrt.org/releases/21.02.0-rc3/targets/armvirt/64/openwrt-21.02.0-rc3-armvirt-64-default-rootfs.tar.gz

That seems very strange...

OpenWrt 21.02 used kernel 5.4.
Even 23.05 (the latest) is only on 6.1. And it never says "oracle" in any of the normal builds. So if this really came from the downloads of the OpenWrt page, I cannot quite explain the linage of this.

But beyond that...

21.02 has been EOL for several years already and is unsupported. It ran through 21.02.7, but you should be running 23.05 now..

I don't see the target available on 23.05, but it is on 22.03, so you should at least be using that:
https://archive.openwrt.org/releases/22.03.5/targets/armvirt/

That seems very strange...

I am running on an oracle cloud server.

So if this really came from the downloads of the OpenWrt page, I cannot quite explain the linage of this.

lol I just copied https://forum.openwrt.org/t/can-access-openwrt-ssh-luci-ui-over-wan-in-lxc-container-on-ubuntu-raspberry-pi/102280
I dont always understand what versions are good or have problems so I just figured I would use what worked for the creator of this post.

I made an lxc container with it- which works and thats awesome. I wanted to filter docker-containers then just exported the lxc, took the rootfs, built an image with docker.

Networking wise, my plan was just to put the gateway ip of the docker containers set to the ip of the docker-openwrt.

Like I said, using a docker container for this purpose is likely to be non-trivial for the routing to work.

That said, there are some things that are wrong in your config.

I'd recommend that you start fresh with 22.03.6. Install wireguard (using the opkg package manager), then add your WG configuration stanza to the network config file, and add the WG network interface to the wan firewall zone. That's the most simple case and you need that to work first.

Ok awesome! Thank you.

Like I said, using a docker container for this purpose is likely to be non-trivial for the routing to work.

Ooo I have no doubt lol. Glutton for punishment I suppose.
I have had some interesting success with Dockenet though.
On docker, I did get ethernets on the openwrt container. I also got the containers to also have their ip/routes connected through that ethernet. They were talking and everything lol. But on that version readonly was a problem.

That said, there are some things that are wrong in your config.

hahahaha I tried lol

I'd recommend that you start fresh with 22.03.6. Install wireguard (using the opkg package manager), then add your WG configuration stanza to the network config file, and add the WG network interface to the wan firewall zone. That's the most simple case and you need that to work first.

Will do thanks again!

This is probably because the Oracle kernel does not contain the wireguard module (or not a compatible version), or lacks some related necessary module. Unlike a conventional VM where the OpenWrt kernel would be loaded and run, Docker uses namespaces to isolate kernel processes, but they all run in the "host" kernel.

2 Likes

This is probably because the Oracle kernel does not contain the wireguard module (or not a compatible version)

Thank you! Gives me a solid lead on where to start looking for a solution.

Pro-tip: docker is not a solution. You could consider a VM. But I'd recommend working bare-metal... ideally your main router, or it could be a device behind it (such as a Pi running OpenWrt directly).

Pro-tip: docker is not a solution

lol but you dont even know my use case. Seems like I offended you or something which was not my intent.
Its a remote server and putting everything under one lxc container will increase complexity with something I have been working on.
I did put it into a VM but that didnt fit the use case with networking I had hoped.

Making a reasonable guess based on the premise of this post -- you're trying to run a Wireguard VPN tunnel to Mullvad.

Not at all... sorry if what I said came across as annoyed or otherwise bothered. And likewise, I hope I didn't say anything that would have been seen as rude.

I'm referring specifically to Wireguard on OpenWrt in a docker container, which as @mk24 pointed out, will not be possible because of the lack of kernel support (this was something new I learned in this thread... I knew that containerized OpenWrt is non-standard, but now I have a new appreciation for why).

1 Like

No one is offended or hurt, but your experience is exactly why running OpenWrt in a container (such as docker) is not a supported configuration and can't ever be; running OpenWrt as a VM (full system virtualization} however is.

...and yes, it is slightly annoying that we have to repeat this like a broken record, as this comes up rather often (and replies in the form if "but it works for my limited use case" do not help with that).

The fact of the matter is, OpenWrt does expect (and hard-requires, e.g. for wireguard) to load and unload kernel modules as needed, it can do that on real hardware, as well as in a full system VM - it can't do that inside a container, as it has no authority over the host kernel.

No, this isn't the only problem with containerization here, OpenWrt also expects a bunch of (among others, netfilter-) kernel patches to be present, a certain set of sysctl configurations to be applied, as well as being able to apply nftables rule sets, routing tables and a lot more. A container isn't allowed by the host to do any of this and things will fail, some aspects more subtly than others, but the result can never fully work by design - and you will tear up a number of serious security issues by trying to force it.

So, yes, you can make OpenWrt's webinterface to make a demo round that way, but you can't make it do its job properly (even less in a cloud environment, where you have zero (out of band) influence over the host configuration).

kvm (qemu), virtualbox, hyper-v, parallels, vmware will work just fine.

docker, lxc, lxd, virtuozzo, OpenVZ, BSD jails, systemd namepaces and other container based approaches can never work, even less safely.

4 Likes