That's because it doesn't have a default route. @4rtz1z had the same problem. See his post above
1 Like
@slidermike check also that you don't have a malformed entry somewhere. Just looked at my config and I don't have this option defaultroute set. Can you post the relevant stanzas from your network config?
This is how mine looks. I can make it lose the default route, for example, by putting in something malformed like list allowed_ips '0.0.0.0/0, ::/0'
config interface 'wg1'
option proto 'wireguard'
option private_key '<KEY>'
option listen_port '<PORT>'
option metric '60'
option auto '0'
option multipath 'off'
list addresses '<IPV4>'
list addresses '<IPV6>'
config wireguard_wg1
option description '<PROVIDER>'
option public_key '<PUBKEY>'
option endpoint_port '<PORT>'
option endpoint_host '<PEER>'
option persistent_keepalive '0'
option route_allowed_ips '1'
list allowed_ips '0.0.0.0/0'
list allowed_ips '::/0'
Thanks for the key info @dl12345. I must have missed it when I thumbed through this thread.
@4rtz1z when you get a couple min would you mind sharing a screen shot of the section of the network config (including the command) for adding the default route to your WG interfaces please.
Greatly appreciate it. This is a sticky worthy "gotcha" since many of us run WG these days.
Do post your /etc/config/network stanzas for your wireguard interfaces. As i said above, I don't have this option explicitly defined for any of mine and they all correctly get a default route which you can see with ip route show.
If I deliberately make the malformed entry I showed above on one of the interfaces, then the default route for that interface vanishes and I see the same error as you, so I'm inclined to think there might be a syntax error in your config.
Nah your tip corrected my issue. What I was looking for was the exact location and syntax of the command he used to get the routing working. In his post he showed the effective null like this
"option route_allowed_ips '0'" which is correct for disabling. However lower in the comment he made a more general but not syntax command and referenced it goes into the specific interface config. However, I see two entries for each WG interface. I figured out which one (of each) it goes under.
All working good now.
Screen shots for the next person.
1 Like
Hi @dl12345
Firstly thank you for taking on the challenge of porting (or probably more accurate to say rewriting mwan3!). Given @feckert has had less time to work on mwan3 generally and has stated this previously, I would strongly consider opening a PR on the OpenWrt packages repo to discuss potentially a new version being available. Perhaps rather than using the mwan3 name, it might be easiest to rename your package to something else, unless @feckert gives permission to basically take over the existing mwan3 package, either way though there should be an option to get the package into official repositories.
I have 2x WAN connection which are IPv6 enabled, along with Wireguard clients. I have been on a very old OpenWrt build for sometime because of mwan3 and lack of nfset support, given newer OpenWrt versions basically disables ipset, compiling my own build seems a bit much. However your package looks like there's a path now and I'd be willing to test it out. I still have one ipset dependency which is Adguard Home, but I can put dnsmasq in the path before it temporarily, and hopefully the AdGuard Home team will implement native nfset support at some point.
I'd be willing to provide feedback on the IPv6 side as it was always a particular pain point, given multi-homing IPv6 is generally complex, the easiest and dirtiest solution has been IPv6 masquerading basically NAT6. I don't know how much you've tested IPv6 and multihoming at all?
Just wanted to thank you for the work and effort that's been put into this! I'll look at testing this soon. I'm several major release branches behind, I'm going to need to do a config backup and then basically copy over, so I need to plan things a bit!
5 Likes
That was exactly my motivation for porting it, since I was stuck on 19.07. I got tired of waiting for someone else to do it.
Indeed. It turned out to be much more involved than just a simple port. For all intents and purposes, it is a rewrite with substantial functionality and reliability upgrades, although doing it as a port definitely helped to keep the regressions to a minimum by using well-tested code as the basis and making iterative changes.
mwan3rtmon is completely new (it's now ucode) and there are several other component level additions, including mwan3ct (a selective conntrack flusher because conntrack can't do some things), and some other ucode scripts to improve operational reliability.
I did actually pose the question to @feckert on github although I'm not sure he's seen it yet.
I'd want the vote count at the top of this thread to hit 50+ (it's at 19 now) before I'd be comfortable with any kind of PR, be it new package or update to the existing one.
mwan3 is a race-condition magnet and at least one user in this thread managed to surface a race-condition that was affecting nobody else. There's merit in it being tested extensively, although it has been subjected to quite a bit of testing already by users in this thread.
I built this for my own use, so kept the name and didn't change the maintainer (after all, it's based on the original mwan3 code). I decided to share it as, well this is what we do in the open source community and it's clear many others were in a similar situation of being locked into older versions of openwrt. I have no vested interest in ensuring it's in the "official" repo, although concede that for the majority of users, it'd be much easier to find and upgrade if it were.
I'm conscious that the original author and maintainer is still the active maintainer. I had noticed some previous posts by @feckert where he stated that he's open to PRs and doesn't have the time to do it himself. My working assumption, however, is that he would not want to be listed as the maintainer due to the extensive changes (it's 5,000+ lines of code in mwan3 and another 3,500 in the luci app) and that there's a mwan4 that's in the initial phases of development for a couple of months now of which he's one of the contributors.
The iptables variant of mwan3 probably won't continue to function much longer in current versions of openwrt (if it even does still!), so there's a case for superseding it and keeping the same name. It could just as well be called "mwan3-ng" or something similar (thanks @slidermike for the name suggestion).
I think whichever way we go should be left to @feckert himself and the community. I don't want to tread on anyone's toes.
Some of the testers in this thread have dual stack IPv6 setups on both wans. It appears to be working for them if by "tested IPv6" you mean things don't break and can mwan3 mark and policy route IPv6. You should ask @woffko for his input as I can see in his diagnostic output that his IPv6 is being masqueraded (I assume via the fw4 toggle) and things look to be working smoothly for him.
What isn't in any released code yet is proper IPv6 failover and that is what I'm working on at the moment, testing two alternative mechanisms: one is a hybrid stateless 1:1 translation from the prefix of the failed wan to that of the failover one and the other is a pure prefix-deprecation mechanism which completely avoids any NAT but is more limited in its applicability in the sense that not every IPv6 deployment will be able to use it.
I've maintained 100% backwards compatibility, so it should just work. Although once migrated, and especially if you configure any of the new features, then the config will be altered such that you'll need to use your original config if you wish to revert.
However, caveat is that I'm not sure it will function on pre 25.12 openwrt releases. One issue raised on my github did point to that being the case, so you'd need to upgrade your distro to 25.12.4.
3 Likes
Awesome work, truly! There's a lot of OpenWrt users relying on mwan3, so your efforts are appreciated! I'll be testing your package soon and report back! I need to review the latest changes in 25.12 generally as I'm shamefully still on 21.02.
Potentially renaming your package to something like mwan4 might make sense in the long term, given that it nicely ties in well with fw4 and the nfset backend. I believe on OpenWrt mailing lists there was talk more recently about a "mwan4" package that would basically be the nfset port/rewrite. mwan3 itself took over from another package called multiwan back in the day, so it isn't unheard of to basically change the package name under major iterations. I would say given the work and effort, it has sufficently forked from the original codebase to warrant it being classified as new and different from the original package it was based off and that may end up being easier for users/documentation. The current mwan3 docs have a bit of legacy stuff hanging with it, being able to drop that is probably a good thing, starting from fresh. I'd be more than happy to help with updating/creating a new docs page for this project, as it is the likely successor.
From a community perspective, a different package name might be better generally to avoid confusion, because not everyone reads the community forum sadly. At the moment even the official documentation for mwan3 makes no reference to this project, so it could completely go unnoticed and people are still using the current ipset mwan3, because that's what the docs say or they have no idea this port exists. The docs can be updated of course, but I think to have the project clearly defined, labelling it as mwan4 could be the best way forward and given the work and effort put in, I think the OpenWrt package dev team would probably agree. Getting in touch with @feckert though is probably a good start point, but I see you have messaged on GitHub, but I don't think they would particularly have any issues. They sadly haven't been able to work on mwan3 for sometime but I totally get it, maintaining open source is usually thankless and life stuff happens. Appreciate you are waiting for more votes/testers, however releasing under a new package name might give you more room for handling any edge cases, rather it overtaking the original mwan3 and users getting a new package without realising.
I've added a note to the current mwan3 docs for more visibility about your port, pointing directly to this thread.
I look forward to testing soon though and will report any findings!
Thanks again for your work.
3 Likes
Yes, it works, with masquerading handled by fw4. I wouldn’t say everything is absolutely perfect; there are occasional glitches, but they are definitely not related to mwan3. There are known issues on the ISP side, and they are being worked on.
1 Like
I’d like to think a bit about the idea of a full Multi-WAN setup wizard.
Since dl12345 has essentially made a new version of mwan3(4)(ng), maybe it would make sense in the future to add some kind of automation through a wizard, or perhaps something simpler: automation for common use cases.
I’m not saying this needs to be done right now, just thinking about possible future improvements.
For example, the user could enter only minimal data, such as the desired percentage ratio between WAN links, and the wizard would automatically create the appropriate group and rules, including common sticky rules.
What do you think, dl12345?
Of course, I don’t think this would solve every possible problem or question, but it could at least eliminate some of them and make life easier for users.
1 Like
Running since 2 days without any issue.
Thanks for your work
1 Like
Same, running latest version for a few days and zero issues. Thank you dl12345 !
Err, that's been done for some time already. Perhaps you didn't notice because you didn't need to create a new policy? Interface members are created and managed automatically.
+1 for mwan4.
Note that, using a new name, we are avoiding regressions and bugs trying to maintain compatibility with mwan3 old configuration format.
2 Likes
I could easily have missed something — that would be typical of me. 
But do the sticky rules get created automatically as well?
The name mwan4 is being used by a early-stage intended successor to mwan3 which I believe @stangri is taking the lead on.
I'm not sure I understand the rest of your comment? Yes, 100% compatibility with the old mwan3 configuration format has always been a design imperative, although this hasn't really constrained the development of this version in any way. Regressions and bugs have come more from the fundamentally different underlying architecture of iptables vs nftables,
1 Like
No they don't. The only thing it makes sense to do is to perhaps put a few commonly used but inert (disabled) rules into the default config that would typically be used (for example a sticky for http/https)
1 Like
Go into the Policy tab and create a new policy and you'll see what I mean.
Hi, on latest snapshot (r34753-c82f2724f5), the 3.6.7 version got high cpu usage on mwan3track.
Mem: 245044K used, 1819484K free, 1352K shrd, 8128K buff, 95744K cached
CPU: 99% usr 0% sys 0% nic 0% idle 0% io 0% irq 0% sirq
Load average: 12.83 12.51 10.78 14/216 7235
PID PPID USER STAT RSS %RSS %CPU COMMAND
7865 1 root R 1260 0% 8% {mwan3track} /bin/sh /usr/sbin/mwan3track wan4
7867 1 root R 1260 0% 8% {mwan3track} /bin/sh /usr/sbin/mwan3track wana
7868 1 root R 1260 0% 8% {mwan3track} /bin/sh /usr/sbin/mwan3track wana_6
7866 1 root R 1260 0% 8% {mwan3track} /bin/sh /usr/sbin/mwan3track wan4_6
7870 1 root R 1260 0% 8% {mwan3track} /bin/sh /usr/sbin/mwan3track wanb_6
7863 1 root R 1260 0% 8% {mwan3track} /bin/sh /usr/sbin/mwan3track wan3
7860 1 root R 1260 0% 8% {mwan3track} /bin/sh /usr/sbin/mwan3track wan1_6
7862 1 root R 1260 0% 8% {mwan3track} /bin/sh /usr/sbin/mwan3track wan2_6
7869 1 root R 1260 0% 8% {mwan3track} /bin/sh /usr/sbin/mwan3track wanb
7859 1 root R 1260 0% 8% {mwan3track} /bin/sh /usr/sbin/mwan3track wan1
7861 1 root R 1260 0% 8% {mwan3track} /bin/sh /usr/sbin/mwan3track wan2
Feb 7 08:49:12 HOME-Router_Backup mwan3-hotplug[9750]: mwan3 hotplug on IoT not called because interface disabled
Feb 7 08:49:15 HOME-Router_Backup mwan3-hotplug[12206]: mwan3 hotplug on nokia not called because interface disabled
Feb 7 08:49:17 HOME-Router_Backup mwan3-hotplug[13035]: mwan3 hotplug on lan not called because interface disabled
Feb 7 08:49:19 HOME-Router_Backup mwan3-hotplug[14120]: mwan3 hotplug on loopback not called because interface disabled
Feb 7 08:49:20 HOME-Router_Backup mwan3-hotplug[14627]: mwan3 hotplug on vpn0 not called because interface disabled
Feb 7 08:49:22 HOME-Router_Backup mwan3-hotplug[14955]: mwan3 hotplug on vpn2 not called because interface disabled
Feb 7 08:49:30 HOME-Router_Backup mwan3-hotplug[15365]: mwan3 hotplug on tr069_1 not called because interface disabled
Jun 4 12:59:07 HOME-Router_Backup mwan3-hotplug[16913]: Execute ifup event on interface wan1 (pppoe-wan1)
Jun 4 12:59:07 HOME-Router_Backup mwan3-hotplug[16913]: create_iface_nft: mwan3_iface_in_wan1 added to mwan3_ifaces_in
Jun 4 12:59:11 HOME-Router_Backup mwan3track[7859]: Detect ifup event on interface wan1 ()
Jun 4 12:59:11 HOME-Router_Backup mwan3track[7859]: Register ifup event on interface wan1 ()
Jun 4 12:59:11 HOME-Router_Backup mwan3track[7859]: firstconnect: called on wan1/wan1 (pppoe-wan1). Status is online. SRC_IP is 218.93.xxx.xxx
Jun 4 12:59:27 HOME-Router_Backup mwan3-hotplug[19445]: Execute ifup event on interface wan2 (pppoe-wan2)
Jun 4 12:59:28 HOME-Router_Backup mwan3-hotplug[19445]: create_iface_nft: mwan3_iface_in_wan2 added to mwan3_ifaces_in
Jun 4 12:59:32 HOME-Router_Backup mwan3track[7861]: Detect ifup event on interface wan2 ()
Jun 4 12:59:32 HOME-Router_Backup mwan3track[7861]: Register ifup event on interface wan2 ()
Jun 4 12:59:32 HOME-Router_Backup mwan3track[7861]: firstconnect: called on wan2/wan2 (pppoe-wan2). Status is online. SRC_IP is 218.93.xxx.xxx
Jun 4 12:59:51 HOME-Router_Backup mwan3-hotplug[20942]: Execute ifup event on interface wan3 (pppoe-wan3)
Jun 4 12:59:51 HOME-Router_Backup mwan3-hotplug[20942]: create_iface_nft: mwan3_iface_in_wan3 added to mwan3_ifaces_in
Jun 4 12:59:59 HOME-Router_Backup mwan3track[7863]: Detect ifup event on interface wan3 ()
Jun 4 12:59:59 HOME-Router_Backup mwan3track[7863]: Register ifup event on interface wan3 ()
Jun 4 12:59:59 HOME-Router_Backup mwan3track[7863]: firstconnect: called on wan3/wan3 (pppoe-wan3). Status is online. SRC_IP is 100.89.xxx.xxx
Jun 4 13:00:02 HOME-Router_Backup mwan3-hotplug[24246]: Execute ifup event on interface wan4 (pppoe-wan4)
Jun 4 13:00:02 HOME-Router_Backup mwan3-hotplug[24246]: create_iface_nft: mwan3_iface_in_wan4 added to mwan3_ifaces_in
Jun 4 13:00:09 HOME-Router_Backup mwan3track[7865]: Detect ifup event on interface wan4 ()
Jun 4 13:00:09 HOME-Router_Backup mwan3track[7865]: Register ifup event on interface wan4 ()
Jun 4 13:00:09 HOME-Router_Backup mwan3track[7865]: firstconnect: called on wan4/wan4 (pppoe-wan4). Status is online. SRC_IP is 100.89.xxx.xxx
Jun 4 13:00:13 HOME-Router_Backup mwan3-hotplug[25309]: Execute ifup event on interface wan2_6 (pppoe-wan2)
Jun 4 13:00:13 HOME-Router_Backup mwan3-hotplug[25309]: create_iface_nft: mwan3_iface_in_wan2_6 added to mwan3_ifaces_in
Jun 4 13:00:23 HOME-Router_Backup mwan3track[7862]: Detect ifup event on interface wan2_6 ()
Jun 4 13:00:23 HOME-Router_Backup mwan3track[7862]: Register ifup event on interface wan2_6 ()
Jun 4 13:00:23 HOME-Router_Backup mwan3track[7862]: firstconnect: called on wan2_6/wan2_6 (pppoe-wan2). Status is online. SRC_IP is 240e:3a0:4
Jun 4 13:00:28 HOME-Router_Backup mwan3-hotplug[26480]: Execute ifup event on interface wan1_6 (pppoe-wan1)
Jun 4 13:00:28 HOME-Router_Backup mwan3-hotplug[26480]: create_iface_nft: mwan3_iface_in_wan1_6 added to mwan3_ifaces_in
Jun 4 13:00:39 HOME-Router_Backup mwan3track[7860]: Detect ifup event on interface wan1_6 ()
Jun 4 13:00:39 HOME-Router_Backup mwan3track[7860]: Register ifup event on interface wan1_6 ()
Jun 4 13:00:40 HOME-Router_Backup mwan3track[7860]: firstconnect: called on wan1_6/wan1_6 (pppoe-wan1). Status is online. SRC_IP is 240e:3a0:4
Jun 4 13:00:46 HOME-Router_Backup mwan3-hotplug[27813]: Execute ifup event on interface wan3_6 (pppoe-wan3)
Jun 4 13:00:47 HOME-Router_Backup mwan3-hotplug[27813]: create_iface_nft: mwan3_iface_in_wan3_6 added to mwan3_ifaces_in
Jun 4 13:01:01 HOME-Router_Backup mwan3track[7864]: Detect ifup event on interface wan3_6 ()
Jun 4 13:01:01 HOME-Router_Backup mwan3track[7864]: Register ifup event on interface wan3_6 ()
Jun 4 13:01:01 HOME-Router_Backup mwan3track[7864]: firstconnect: called on wan3_6/wan3_6 (pppoe-wan3). Status is online. SRC_IP is 2409:8a21:
Jun 4 13:01:08 HOME-Router_Backup mwan3-hotplug[29339]: Execute ifup event on interface wan4_6 (pppoe-wan4)
Jun 4 13:01:08 HOME-Router_Backup mwan3-hotplug[29339]: create_iface_nft: mwan3_iface_in_wan4_6 added to mwan3_ifaces_in
Jun 4 13:01:23 HOME-Router_Backup mwan3track[7866]: Detect ifup event on interface wan4_6 ()
Jun 4 13:01:23 HOME-Router_Backup mwan3track[7866]: Register ifup event on interface wan4_6 ()
Jun 4 13:01:24 HOME-Router_Backup mwan3track[7866]: firstconnect: called on wan4_6/wan4_6 (pppoe-wan4). Status is online. SRC_IP is 2409:8a21:
Jun 4 13:01:32 HOME-Router_Backup mwan3-hotplug[30602]: Execute ifup event on interface wana (wana)
Jun 4 13:01:33 HOME-Router_Backup mwan3-hotplug[30602]: create_iface_nft: mwan3_iface_in_wana added to mwan3_ifaces_in
Jun 4 13:01:50 HOME-Router_Backup mwan3track[7867]: Detect ifup event on interface wana ()
Jun 4 13:01:50 HOME-Router_Backup mwan3track[7867]: Register ifup event on interface wana ()
Jun 4 13:01:51 HOME-Router_Backup mwan3track[7867]: firstconnect: called on wana/wana (wana). Status is online. SRC_IP is 10.10.2.20
Jun 4 13:01:58 HOME-Router_Backup mwan3-hotplug[31836]: Execute ifup event on interface wana_6 (wana)
Jun 4 13:01:59 HOME-Router_Backup mwan3-hotplug[31836]: create_iface_nft: mwan3_iface_in_wana_6 added to mwan3_ifaces_in
Jun 4 13:02:19 HOME-Router_Backup mwan3track[7868]: Detect ifup event on interface wana_6 ()
Jun 4 13:02:19 HOME-Router_Backup mwan3track[7868]: Register ifup event on interface wana_6 ()
Jun 4 13:02:20 HOME-Router_Backup mwan3track[7868]: no src ipv6 address found from netifd for interface 'wana_6' dev 'wana' guessing d10:10:2:
Jun 4 13:02:20 HOME-Router_Backup mwan3track[7868]: firstconnect: called on wana_6/wana_6 (wana). Status is online. SRC_IP is d10:10:2::20
Jun 4 13:02:28 HOME-Router_Backup mwan3-hotplug[605]: Execute ifup event on interface wanb (wanb)
Jun 4 13:02:29 HOME-Router_Backup mwan3-hotplug[605]: create_iface_nft: mwan3_iface_in_wanb added to mwan3_ifaces_in
Jun 4 13:02:52 HOME-Router_Backup mwan3track[7869]: Detect ifup event on interface wanb ()
Jun 4 13:02:52 HOME-Router_Backup mwan3track[7869]: Register ifup event on interface wanb ()
Jun 4 13:02:53 HOME-Router_Backup mwan3track[7869]: firstconnect: called on wanb/wanb (wanb). Status is online. SRC_IP is 10.10.1.20
Jun 4 13:03:03 HOME-Router_Backup mwan3-hotplug[1868]: Execute ifup event on interface wanb_6 (wanb)
Jun 4 13:03:03 HOME-Router_Backup mwan3-hotplug[1868]: create_iface_nft: mwan3_iface_in_wanb_6 added to mwan3_ifaces_in
Jun 4 13:03:28 HOME-Router_Backup mwan3track[7870]: Detect ifup event on interface wanb_6 ()
Jun 4 13:03:28 HOME-Router_Backup mwan3track[7870]: Register ifup event on interface wanb_6 ()
Jun 4 13:03:29 HOME-Router_Backup mwan3track[7870]: no src ipv6 address found from netifd for interface 'wanb_6' dev 'wanb' guessing d10:10:1:
Jun 4 13:03:29 HOME-Router_Backup mwan3track[7870]: firstconnect: called on wanb_6/wanb_6 (wanb). Status is online. SRC_IP is d10:10:1::20
Seems the issue was come after commits:
rollback to busybox 1.37 fixed the issue temporarily.
I tested busybox 1.38 on my 25.12.4 and can't reproduce the problem. So it's probably related to your specific config combined with an interaction of the snapshot build you're running and busybox 1.38, since you said the problem goes away when you revert to busybox 1.37.
This, incidentally, is one of the reasons people are counselled not to run snapshot builds in production. They break things occasionally.
On one of the mwan3tracks with high cpu usage, please give the output of strace -p <pid> for around 10 seconds during the period of high cpu usage.
You might need to install strace if you don't have it installed.
Also give me the output of ubus call system board, if you have coreutils installed, the per-interface interval/count/track_method/reliability settings, and let me know if it's only the IPv6 trackers consuming cpu or if it's both IPv4 and IPv6.