Glad to hear
So you're using the non-NAT deprecation mechanism for failover and the 1:1 for whenever you're rule steering a foreign source out a WAN, right?
So you're using the non-NAT deprecation mechanism for failover and the 1:1 for whenever you're rule steering a foreign source out a WAN, right?
oh wait, I meant dual IPv6 with dual delegation on LAN interface for pure no-NAT deprecation for failover (not 1:1 NAT66). I wrote that wrong initially.
Sure - failover is deprecation. However, if you rule steer something with a prefix belonging to WANA out of WANB which cannot route that prefix, it will necessarily be NATed, otherwise rule steering won't work.
The no-NAT feature is confined to failover. Rule steering and load balancing will always require some form of NAT, be it a plain masquerade or a 1:1. If you don't define 1:1 mapping, it will simply use masquerade by default.
1 Like
I think AdGuard Home is taking care of DNS (53) so mwan3-nft could not use dnsmasq to resolve the ipset, the port for DNS of dnsmasq right now is 54:
Below are my configurations (The forum only allows me to upload one embed image, please click on the image to go to my Google Photos):
AdGuard Home has no support for nfset currently, so you'd be better off using dnsmasq and forwarding onto AdGuard Home, or switching back to dnsmasq. I used AdGuard Home for a while but dropped it when migrating to 25.12, in all honesty while it is nice to see client statistics, I ended up having to whitelist a lot of domains due to broken sites, it became cat and house and counter productive.
1 Like
Im looking for pointers if im doing this right. My end goal is to have load balanced ipv6. I have two ISPs both using pppoe auth. wan_6 offers me a dynamically allocated /64 and wanb_6 offers me a static /60. My current mwan3 nft config is this. I am running mwan3 nft pre release
config globals 'globals'
option mmx_mask '0x3F00'
option ipv6_routing 'on'
option ipv6_failover_type 'translate'
config interface 'wan'
option enabled '1'
list track_ip '1.0.0.1'
list track_ip '1.1.1.1'
list track_ip '208.67.222.222'
list track_ip '208.67.220.220'
option family 'ipv4'
option reliability '2'
option initial_state 'online'
option track_method 'ping'
option count '1'
option size '56'
option max_ttl '60'
option timeout '4'
option interval '10'
option failure_interval '5'
option recovery_interval '5'
option down '5'
option up '5'
config interface 'wan_6'
option enabled '1'
list track_ip '2606:4700:4700::1001'
list track_ip '2606:4700:4700::1111'
list track_ip '2620:0:ccd::2'
list track_ip '2620:0:ccc::2'
option family 'ipv6'
option reliability '2'
config interface 'wanb'
option enabled '1'
list track_ip '1.0.0.1'
list track_ip '1.1.1.1'
list track_ip '208.67.222.222'
list track_ip '208.67.220.220'
option family 'ipv4'
option reliability '1'
config interface 'wanb_6'
option enabled '1'
list track_ip '2606:4700:4700::1001'
list track_ip '2606:4700:4700::1111'
list track_ip '2620:0:ccd::2'
list track_ip '2620:0:ccc::2'
option family 'ipv6'
option reliability '1'
option ipv6_translate_pool 'auto'
list ipv6_translate_ula 'auto'
config member 'wan_m1_w3'
option interface 'wan'
option metric '1'
option weight '3'
config member 'wan_m2_w3'
option interface 'wan'
option metric '2'
option weight '3'
config member 'wanb_m1_w2'
option interface 'wanb'
option metric '1'
option weight '2'
config member 'wanb_m1_w3'
option interface 'wanb'
option metric '1'
option weight '3'
config member 'wanb_m2_w2'
option interface 'wanb'
option metric '2'
option weight '2'
config member 'wan6_m1_w3'
option interface 'wan_6'
option metric '1'
option weight '3'
config member 'wan6_m2_w3'
option interface 'wan_6'
option metric '2'
option weight '3'
config member 'wanb6_m1_w2'
option interface 'wanb_6'
option metric '1'
option weight '2'
config member 'wanb6_m1_w3'
option interface 'wanb_6'
option metric '1'
option weight '3'
config member 'wanb6_m2_w2'
option interface 'wanb_6'
option metric '2'
option weight '2'
config policy 'wan_only'
list use_member 'wan_m1_w3'
list use_member 'wan6_m1_w3'
config policy 'wanb_only'
list use_member 'wanb_m1_w2'
list use_member 'wanb6_m1_w2'
config policy 'balanced'
list use_member 'wan_m1_w3'
list use_member 'wanb_m1_w3'
list use_member 'wan6_m1_w3'
list use_member 'wanb6_m1_w3'
config policy 'wan_wanb'
list use_member 'wan_m1_w3'
list use_member 'wanb_m2_w2'
list use_member 'wan6_m1_w3'
list use_member 'wanb6_m2_w2'
config policy 'wanb_wan'
list use_member 'wan_m2_w3'
list use_member 'wanb_m1_w2'
list use_member 'wan6_m2_w3'
list use_member 'wanb6_m1_w2'
config rule 'https'
option sticky '1'
option dest_port '443'
option proto 'tcp'
option use_policy 'balanced'
config rule 'default_rule_v4'
option dest_ip '0.0.0.0/0'
option use_policy 'balanced'
option family 'ipv4'
config rule 'default_rule_v6'
option dest_ip '::/0'
option use_policy 'balanced'
option family 'ipv6'
The one bit of information you haven't given here is your LAN addressing. I am presuming, although this may be wrong, that you're using ULA addressing on your LAN due to the presence of the list ipv6_translate_ula 'auto' option.
As your configuration stands, wan_6 will masquerade all addresses with a foreign prefix and wanb_6 will 1:1 translate addresses with a foreign prefix, mapping them onto wanb_6's own /60 delegation.
For your described WANs, this seems like a sensible configuration if this is what you intend.
Some notes on your configuration
-
Sharing the same tracking IPs means that if one WAN goes down because the
track_ips all fail, the other will go down too. I would split them up,list track_ip '2606:4700:4700::1001'andlist track_ip '2620:0:ccd::2'on one of the WANs and a completely different set on the other WAN. -
Your reliability of
2onwan_6(and onwan) means at least two trackers must respond otherwise the link will be considered down. Do you want that? I find one is enough usually. -
Blindly load balancing everything is never a great idea IMHO. You should at the very least have sticky rules for http, https and QUIC if you're going to load balance. I have two symmetric links, both of 1Gbps and I don't load balance. Something, sometime is likely to break because of two separate connections in a logical session coming from two separate IPs. At the very least, you should make your https rule cover both ports 80 and 443, because otherwise 443 could end up pinned to one WAN and 80 to the other and if you have some logical session that is using a mix of both, the endpoints will see connections from two different IPs and that could break things. QUIC is a separate protocol and would need a separate rule and what could happen there is that a browser could fall back from QUIC to TCP 443 in a session and then the latter gets routed out of the other WAN with 50% probability. I would strongly discourage people from load balancing everything by default.
-
You're load balancing router-originated traffic too, but you have not set the snat6 option for the interfaces. Router-originated traffic is deliberately excluded from the translation unless explicitly configured, so half your router originated traffic will break, the half that goes out the WAN that cannot route the given prefix. Set
option snat6 '1'on each WAN. -
There's a bug in the current code that will be fixed next release. It relates to sticky
family=anyrules and its effect is that the IPv6 traffic that matches won't be stickied. You should separate the rule out into an IPv4 and an IPv6 one for now as a workaround. You'll need IPv4 and IPv6 variants for http and QUIC too.
This is my lan addressing right now. The whole /64 from wan_6 and /61 from wanb_6. I'm not sure if this is importan but since my ipv6 interfaces are virtual dynamic interfaces im unable to set metrics on them, so it shows up as "no metric set!"on the multiwan interface configuration.
I have taken all your considerations and updated the mwan3 configuration as followed
config globals 'globals'
option mmx_mask '0x3F00'
option ipv6_routing 'on'
option ipv6_failover_type 'translate'
config interface 'wan'
option enabled '1'
list track_ip '1.0.0.1'
list track_ip '1.1.1.1'
list track_ip '208.67.222.222'
list track_ip '208.67.220.220'
option family 'ipv4'
option reliability '1'
option initial_state 'online'
option track_method 'ping'
option count '1'
option size '56'
option max_ttl '60'
option timeout '4'
option interval '10'
option failure_interval '5'
option recovery_interval '5'
option down '5'
option up '5'
config interface 'wan_6'
option enabled '1'
list track_ip '2001:4860:4860::8888'
list track_ip '2606:4700:4700::1111'
option family 'ipv6'
option reliability '1'
option snat6 '1'
config interface 'wanb'
option enabled '1'
list track_ip '1.0.0.1'
list track_ip '1.1.1.1'
list track_ip '208.67.222.222'
list track_ip '208.67.220.220'
option family 'ipv4'
option reliability '1'
config interface 'wanb_6'
option enabled '1'
list track_ip '2620:0:ccd::2'
list track_ip '2606:4700:4700::1001'
option family 'ipv6'
option reliability '1'
option ipv6_translate_pool 'auto'
list ipv6_translate_ula 'auto'
option snat6 '1'
config member 'wan_m1_w3'
option interface 'wan'
option metric '1'
option weight '3'
config member 'wan_m2_w3'
option interface 'wan'
option metric '2'
option weight '3'
config member 'wanb_m1_w2'
option interface 'wanb'
option metric '1'
option weight '2'
config member 'wanb_m1_w3'
option interface 'wanb'
option metric '1'
option weight '3'
config member 'wanb_m2_w2'
option interface 'wanb'
option metric '2'
option weight '2'
config member 'wan6_m1_w3'
option interface 'wan_6'
option metric '1'
option weight '3'
config member 'wan6_m2_w3'
option interface 'wan_6'
option metric '2'
option weight '3'
config member 'wanb6_m1_w2'
option interface 'wanb_6'
option metric '1'
option weight '2'
config member 'wanb6_m1_w3'
option interface 'wanb_6'
option metric '1'
option weight '3'
config member 'wanb6_m2_w2'
option interface 'wanb_6'
option metric '2'
option weight '2'
config policy 'wan_only'
list use_member 'wan_m1_w3'
list use_member 'wan6_m1_w3'
config policy 'wanb_only'
list use_member 'wanb_m1_w2'
list use_member 'wanb6_m1_w2'
config policy 'balanced'
list use_member 'wan_m1_w3'
list use_member 'wanb_m1_w3'
list use_member 'wan6_m1_w3'
list use_member 'wanb6_m1_w3'
config policy 'wan_wanb'
list use_member 'wan_m1_w3'
list use_member 'wanb_m2_w2'
list use_member 'wan6_m1_w3'
list use_member 'wanb6_m2_w2'
config policy 'wanb_wan'
list use_member 'wan_m2_w3'
list use_member 'wanb_m1_w2'
list use_member 'wan6_m2_w3'
list use_member 'wanb6_m1_w2'
#config rule 'https'
# option sticky '1'
# option dest_port '443'
# option proto 'tcp'
# option use_policy 'balanced'
config rule 'default_rule_v4'
option dest_ip '0.0.0.0/0'
option use_policy 'balanced'
option family 'ipv4'
config rule 'default_rule_v6'
option dest_ip '::/0'
option use_policy 'balanced'
option family 'ipv6'
# --- Web Traffic (TCP 80, 443) ---
config rule 'sticky_web_v4'
option src_ip '0.0.0.0/0'
option proto 'tcp'
option dest_port '80,443'
option family 'ipv4'
option sticky '1'
option timeout '600'
option use_policy 'balanced'
config rule 'sticky_web_v6'
option src_ip '::/0'
option proto 'tcp'
option dest_port '80,443'
option family 'ipv6'
option sticky '1'
option timeout '600'
option use_policy 'balanced'
# --- QUIC Traffic (UDP 443) ---
config rule 'sticky_quic_v4'
option src_ip '0.0.0.0/0'
option proto 'udp'
option dest_port '443'
option family 'ipv4'
option sticky '1'
option timeout '600'
option use_policy 'balanced'
config rule 'sticky_quic_v6'
option src_ip '::/0'
option proto 'udp'
option dest_port '443'
option family 'ipv6'
option sticky '1'
option timeout '600'
option use_policy 'balanced'
One more thing i might add is i dont necessarily want load balanced equally. I just want wan be used more often and wanb be only used when the bandwidth requirement is there and multithreaded downloads can take advantage of it.
Hello. I discovered that my mwan3 OpenWRT in VM is only 100Mb speed traffic on LAN as well as WAN. Could this be a mwan3 problem?
I already tried configuration tips from some threads on the forum (virtio, queue, ...) ,no luck. I prefere to ask here before asking on the forum.
Your four sticky rules are dead, since in mwan3 rules render in config order and your default catch all rules come above them, so they're never reached.
Re the metric, no it does not matter for mwan3. It matters for main table default route ordering though, but that's not going to affect mwan3's functionality.
Worth noting that your sticky http/https rule and your desire for multithreaded downloads are in conflict. The sticky rule will pin all http/https traffic for a given host indefinitely while there is traffic and for 600 seconds after the traffic ceases, and so by definition it won't load balance.
But if you remove the sticky on it, then ecommerce transactions will probably break and you'll likely trigger anti-fraud mechanisms, as both ecommerce store and payment provider will see traffic from two different IPs. You have to choose between session integrity and aggregation and this isn't a mwan3 problem, this is just how internet traffic works.
Regarding the unequal balancing, mwan3 is not bandwidth aware. It can't balance based on bandwidth consumption. The weights determine a per-new-connection flow probability, not a load awareness.
The only way to get what you appear to be aiming at is to bond the two WANs and this requires functionality that is not available in Openwrt, coupled with a remote server to aggregate the separate flows coming from the two WAN interfaces into a single stream. And even that has substantial drawbacks depending on the characteristics of your WANs.
Why would mwan3 have anything to do with it? It's a bunch of ip and nft rules.
You're saying that the problem is both on LAN and on WAN. mwan3 only touches the routed path and so never sees bridged LAN to LAN connections.
Something common to both is limiting you, which points to the virtual NIC or to the host's physical NIC.
Post what ethtool <iface> reports for the interfaces.
If ethtool isn't reporting a fixed speed, then check that the host's physical NIC hasn't autonegotiated to 100Mbps for some reason (bad cable?) or is explicitly set to 100Mbps.
In any event, you can just stop mwan3 with /etc/init.d/mwan3 stop to check whether anything changes.
host physical interface bridged by interface bones for all VLANs
sudo ethtool eno8303
Settings for eno8303:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Link partner advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Link partner advertised pause frame use: No
Link partner advertised auto-negotiation: Yes
Link partner advertised FEC modes: Not reported
Speed: 1000Mb/s
Duplex: Full
Auto-negotiation: on
[...]
sudo ethtool bone
Settings for bone:
Supported ports: [ ]
Supported link modes: Not reported
Supported pause frame use: No
Supports auto-negotiation: No
Supported FEC modes: Not reported
Advertised link modes: Not reported
Advertised pause frame use: No
Advertised auto-negotiation: No
Advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Unknown! (255)
Auto-negotiation: off
[...]
host LAN interface
sudo ethtool lan
Settings for lan:
Supported ports: [ ]
Supported link modes: Not reported
Supported pause frame use: No
Supports auto-negotiation: No
Supported FEC modes: Not reported
Advertised link modes: Not reported
Advertised pause frame use: No
Advertised auto-negotiation: No
Advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Unknown! (255)
Auto-negotiation: off
[...]
Same from inside VM
ethtool bone
Settings for bone:
Supported ports: [ ]
Supported link modes: Not reported
Supported pause frame use: No
Supports auto-negotiation: No
Supported FEC modes: Not reported
Advertised link modes: Not reported
Advertised pause frame use: No
Advertised auto-negotiation: No
Advertised FEC modes: Not reported
Speed: Unknown!
Duplex: Unknown! (255)
Port: Other
PHYAD: 0
Transceiver: internal
Auto-negotiation: off
Link detected: yes
ethtool eth0
Settings for eth0:
Supported ports: [ ]
Supported link modes: Not reported
Supported pause frame use: No
Supports auto-negotiation: No
Supported FEC modes: Not reported
Advertised link modes: Not reported
Advertised pause frame use: No
Advertised auto-negotiation: No
Advertised FEC modes: Not reported
Speed: Unknown!
Duplex: Unknown! (255)
Port: Other
PHYAD: 0
Transceiver: internal
Auto-negotiation: off
Link detected: yes
ethtool eth1
Settings for eth1:
Supported ports: [ ]
Supported link modes: Not reported
Supported pause frame use: No
Supports auto-negotiation: No
Supported FEC modes: Not reported
Advertised link modes: Not reported
Advertised pause frame use: No
Advertised auto-negotiation: No
Advertised FEC modes: Not reported
Speed: Unknown!
Duplex: Unknown! (255)
Port: Other
PHYAD: 0
Transceiver: internal
Auto-negotiation: off
Link detected: yes
As shown above interfaces are configures as 1Gb or 10Gb. Inside the VM no values. With mwan3 stopped, no changes.
I understand that mwan3 should have no role on this matter.
Thanks for your help. I will ask on the forum in the mean time. If you have other clue, welcome.
Nothing pops out at me in that output. Hypervisor-side rate limiting, or less likely because LAN is affected too, QOS throttling?
I'm curious to know how many people are actually using the beta IPv6 code. I take no bug reports as a positive sign, but that could just mean low uptake.
If you're using beta IPv6 code, could you hit like on this message please? And otherwise, not. Thanks.
3 Likes
Sorry, I have a question. Is it possible to create an mwan3 policy so that only one device is routed through a specific IPv6 interface instead of applying it to all devices?
if by device you understand a specific IP address or some network subset then yes. i have such config.
1 Like
Yes, but a little differently to the way your question implies.
A policy simply says which interfaces get used, either in a load balancing or in a failover sequence or a combination of the two, and what to do if the policy members are not online (drop the traffic, send an ICMP unreachable or use the main table).
A rule will steer matched traffic towards a specific policy.
So you'd setup a policy with only that IPv6 interface as a member, then create a rule that matches the desired traffic and sends that traffic into the policy.
2 Likes
Got it: USB dock from notebook is dead at 100Mbs/Half/autoneg off. In the mean time, the Ethernet cable gave up. Easy to debug ...
Sorry to have taken your time, thanks for your support
I'd be interested in trying this on a TP-Link ER605.
It has an architecture of "MediaTek MT7621 ver:1 eco:3"
Is there a version that is suitable?
I am using as a daily-driver, and haven't really encountered any issues "yet". Although it was a "set-n-forget" type of setup and nobody complained when it switches from a to b and b to a.