My fully automated OpenWrt router setup with Ansible

[diegobernardes]

I did a post on the r/openwrt, but I forgot to post it here.

---

I'm a network nerd who decided to give OpenWrt a shot on a NanoPi R6S. The main reasons for switching from MikroTik: DNS is pretty limited, and the hardware can't really handle SQM properly.

I won't lie. I massively underestimated how much time this would take. Lots of back and forth, but I'm really happy with where it landed. The whole router configures itself from scratch in a few minutes with zero manual steps.

The setup isn't super complex but it covers everything I need:

• A few VLANs
• Unbound with DNSSEC and DNS-over-TLS to Quad9
• adblock-fast feeding blocklists into Unbound. Per-device bypass
• CAKE SQM. Getting A+ on bufferbloat test
• Tailscale running as an exit node for all my portable devices
• Avahi for mDNS
• Chrony for NTP
• Watchcat for WAN monitoring

My current setup:

• Synolog DS923+
• Mini PC with a N100 running Talos Linux
• MikroTik RB5009 working as a switch
• NanoPi R6S
• 1gbps symmetrical WAN connection
• 3 TP-Link Omada EAP655 (I would love to run OpenWrt on these as well...)

Project source code: https://github.com/diegobernardes/openwrt

[dave14305]

I've never gotten too far in using Ansible or similar tools, but I can tell a lot of work went into this. Specific to OpenWrt, I would ask if this was really easier than just developing a uci-defaults script that runs automatically at boot and configures the router? I've been using such a script with Imagebuilder for a few years now with great success.

Does the default config file always get overwritten by these tasks and templates? How do you keep up with package config changes?

Not trying to rain on your parade. Just wanting to understand the pros and cons.

[diegobernardes]

@dave14305

I would ask if this was really easier than just developing a uci-defaults script that runs automatically at boot and configures the router?

Probably not on the short term, but it is simpler, for me, in the long run.

@dave14305

I've been using such a script with Imagebuilder for a few years now with great success.

What happens when you need to do a small change? You're always building a new image and flashing the device?

---

I’m a sucker for GitOps. When I was using MikroTik as a router, it was fully configured using Terraform, which is way better than Ansible. Unfortunately, there isn’t a good provider for OpenWrt.

It took some time to put everything together, especially with all the experimentation I was doing before, but now any change is quite simple. A single task apply deploys everything quickly and easily.

@dave14305

Does the default config file always get overwritten by these tasks and templates? How do you keep up with package config changes?

I’m using the default OpenWrt image. Yes, the roles always update the config, but when you use ansible.builtin.file or ansible.builtin.template, the file, and any dependent actions, only change if the content actually changes.

Basically, Ansible continuously adds or updates configuration. The only part that requires manual intervention is when files need to be deleted. That’s where Terraform does a much better job. But 99% of the time I’m just updating rather than deleting files. For example, firewall rules.

You can see the main execution flow here. It's also worth taking a look at the config file.

[dave14305]

@diegobernardes

What happens when you need to do a small change? You're always building a new image and flashing the device?

No, I just make the change on the router and mirror the change on my private git repo that has my image builder custom files. I only rebuild when a new release is available, or significant package updates have accumulated.

[russoz]

Hi @diegobernardes @dave14305 ,

Not sure if you are aware, there is now an official Ansible Community collection for OpenWrt - community.openwrt. You can check it out the Ansible Galaxy entry, the full docs, or directly at the git repo.

I see your playbooks use all the standard Ansible modules in Python, so they require that Python is installed in the router. The community.openwrt collection starts on the assumption that there will be no Python installed, which might be the case for a fair share of people.

Dave, neither Diego's code (as far as I looked into it) nor the collection touch on the problem of building OpenWrt images. We did have a suggestion about it, but it is yet something for a distant future. As to your question of being easier, it depends on a lot of things: obviously it will be harder on you if you never used Ansible before, but once you know how to work with it, it can be much easier. Ansible begun as configuration management system, so it can help you perform the changes, but also to detect config drift. Managing one router or 500 routers, if done right, can be as easy as running the same playbook again on a larger set of targets.

Another aspect of Ansible is that it aims to use a declarative idiom rather than imperative. It does not always work that way - it depends on the specific modules/plugins you are using, but many, if not most, will allow you to declare what state you want your target to be in, rather than just issue a command to do something. And there is where I will disagree a little bit with Diego - many modules in Ansible will allow you to delete things - if you declare your intent for them to be absent.

I would not recommend writing Ansible code using the exact same logic you would have in a shell script - it will feel like a bloated script. But if you delve into how it is supposed to work, your playbooks can be lean, easy to read, effective, repeatable, to mention few qualities.

My home OpenWrt setup is entirely written in Ansible (using c.openwrt of course), and whenever I had issues that required resetting the router, or when I upgrade it (and I always wipe the config clean when I do), I can have it running again with the exact same configs that I had before in 15min or less.

Hope I encouraged you to take a look at Ansible and the OpenWrt collection (rather than scare you :-P).

Cheers

[dekarl]

@diegobernardes

When I was using MikroTik as a router, it was fully configured using Terraform, which is way better than Ansible

Looking forward to an OpenWRT controller similar to Unifi's controller. I configure that with terraform.

OpenWISP has no "easy mode" and OpenSOHO is AP only ATM. Its close, but not there yet.

[diegobernardes]

@russoz

Not sure if you are aware, there is now an official Ansible Community collection for OpenWrt - community.openwrt.

Thanks for the heads up. I wanted to do everything from scratch because it has been quite some time since the last time I've used Ansible. But for sure a few parts could be simpler, specially apk.
I'll see what I can pick from the community.openrwt.

@russoz

I see your playbooks use all the standard Ansible modules in Python, so they require that Python is installed in the router. The community.openwrt collection starts on the assumption that there will be no Python installed, which might be the case for a fair share of people.

I can imagine setups where this could be problematic, but it's quite ok for me. I have plenty of resources available.

@russoz

And there is where I will disagree a little bit with Diego - many modules in Ansible will allow you to delete things - if you declare your intent for them to be absent.

True. Let me rephrase that. It's just simpler to "remove" configuration on Terraform than on Ansible. On Terraform you delete the code and the engine will detect the drift and update whatever needs to be updated. On Ansible you need to declare that you want to delete things, and after the first run, when things will be deleted, that line will still be there on your code. In the end they'll both do the same, it's just that, in my opinion, Terraform is a better model.

@russoz

My home OpenWrt setup is entirely written in Ansible (using c.openwrt of course), and whenever I had issues that required resetting the router, or when I upgrade it (and I always wipe the config clean when I do), I can have it running again with the exact same configs that I had before in 15min or less.

It's open source? If yes, do you mind sharing a link?

[russoz]

@diegobernardes

@russoz

My home OpenWrt setup is entirely written in Ansible (using c.openwrt of course), and whenever I had issues that required resetting the router, or when I upgrade it (and I always wipe the config clean when I do), I can have it running again with the exact same configs that I had before in 15min or less.

It's open source? If yes, do you mind sharing a link?

I developed that for personal use only, so it was not in a state that I could share, but I have been working on it, slowly but steadily, to separate the code from my configuration. And I got a new router (a Banana Pi ), so I can use the old one as a test bed for this. I reckon a couple of months more and I should have it presentable.

[russoz]

@diegobernardes

True. Let me rephrase that. It's just simpler to "remove" configuration on Terraform than on Ansible. On Terraform you delete the code and the engine will detect the drift and update whatever needs to be updated. On Ansible you need to declare that you want to delete things, and after the first run, when things will be deleted, that line will still be there on your code. In the end they'll both do the same, it's just that, in my opinion, Terraform is a better model.

I can certainly see what you mean. I just want to keep in mind that Terraform is meant to be an "Infrastructure as Code" tool, while Ansible is meant to be an automation tool. Some people use it for automating the provisioning of infrastructure, but I would agree that TF (or Opentofu, for that matter) is better doing that. OTOH, TF is not a good fit for automating generic things (not infrastructure). But, hey, that's probably a longer discussion that would be better served with a couple of beers

[rgnldo]

@diegobernardes

OpenWrt a shot on a NanoPi R6S

First, I commend your hardware investment, even though it's not a necessary or mandatory requirement for everyone.

If this serves for knowledge exchange, here's my opinion on the topics:

• Unbound with DNSSEC and DNS-over-TLS to Quad9 = I avoid using DNSSEC with DoT. For OpenWrt, I'm enjoying Dnsmasq + AdguardHome

• adblock-fast feeding blocklists into Unbound. Per-device bypass = Excellent separation for adblocking filters.

• CAKE SQM. Getting A+ on bufferbloat test = NanoPi R6S very nice

• Tailscale running as an exit node for all my portable devices = I use Netbird because it's WireGuard at the kernel level

• Avahi for mDNS = for privacy reasons, I typically disable this service.

• Chrony for NTP = Very good choice. I use OpenWrt's native NTP

• Watchcat for WAN monitoring = Prefer a script that verifies and reloads applications if necessary.

[diegobernardes]

@rgnldo

First, I commend your hardware investment, even though it's not a necessary or mandatory requirement for everyone.

That's a good call. For anyone reading this in the future you really don't need this kind of hardware as it's completely overkill for a OpenWrt deployment. The only part that indeed requires more CPU is CAKE. When I'm using all the bandwidth of my connection then my CPU goes to 70%. But most of the time this is the status of the router:

@rgnldo

I avoid using DNSSEC with DoT

Any specific reason?

@rgnldo

• adblock-fast feeding blocklists into Unbound. Per-device bypass = Excellent separation for adblocking filters.

I had way too many problems with my wife. Looks like she is the kind of person that click on ads. Many years of marriage, and I discovered this now.

@rgnldo

• Tailscale running as an exit node for all my portable devices = I use Netbird because it's WireGuard at the kernel level

I was not aware of this service. Nice to know! And they have enough proxies? Tailscale works really well for me. I can pretty much use my phone anywhere and I can barely feel the latency when using my home network.

@rgnldo

• Avahi for mDNS = for privacy reasons, I typically disable this service.

A compromise that I had to make to have things like Apple HomeKit working. But the VLANs are doing the heavy lifting in terms of protection.

[rgnldo]

@diegobernardes

Any specific reason?

I believe it's some local incompatibility with my setup. Some DNS upstreams aren't compatible.

"In some circumstances, DNSSEC may be configured for a zone but may not work properly during a complete query chain process as a result of something other than malicious actions. This is most often due to misconfiguration, or in some circumstances it is due to some intermediary device or network element either corrupting or refusing to pass DNSSEC messages appropriately. Sometimes the failure is the result of differing interpretations of IETF RFC standards between authoritative and recursive resolvers or between two differing versions or packages of either authoritative or recursive software. Any of these types of faults will end with the same result as though there was an attempt to subvert the domain maliciously; the recursive resolver (Quad9) will mark the lookup as failing the DNSSEC tests, the lookup itself will fail, and the client will receive an error." DNSSEC NTAs: No Good Compromises

[rgnldo]

@diegobernardes

I had way too many problems with my wife. Looks like she is the kind of person that click on ads. Many years of marriage, and I discovered this now.

I've been enjoying the workflow of isolating adblock filtering outside the DNS resolver. However, I'm really liking AdGuardHome, which handles this exceptionally well as both a DNS proxy and a DNSBL.

[rubenbe]

OpenSOHO half does the switches (if you consider it being an AP with zero Wifi radios )
In case you have very specific VLAN setups, you might need to fiddle a bit, since OpenSOHO has its own opinion on how to configure them (basically I want to make it unlikely to block the user out when they are configuring VLANs)

@diegobernardes

3 TP-Link Omada EAP655

I've switch all my APs to EAP655, and they work like a charm. (all maintained by OpenSOHO of course )

@diegobernardes