PBR not working with mwan3

Please post your current pbr rules here, and I will try to formulate the equivalent ones for mwan3. So that you can uninstall pbr, and have only mwan3, without any loss of functionality.

Or even better, formulate what you are trying to achieve with mwan3 and pbr, so that I get the full picture and can work from that.

2 Likes

mwan3 distributes the load between 2 wan. I need to be able to assign my gateway to any IP in the network. In version 23.03, I did it using vpn-policy-routing and everything worked.

Maybe it is possible to send the right IP to the right gateway through mwan3?

I don't fully understand.

mwan3 distributes the load between 2 wan.

OK, I understand this - you are using load balancing, not fail-over. This is indeed a valid use case for mwan3.

I need to be able to assign my gateway to any IP in the network.

Please reword, I don't understand this at all. Or maybe provide examples.

In version 23.03, I did it using vpn-policy-routing and everything worked.

This was not supposed to work. Any combination of two packages that provide policy-based routing (and load balancing is a form of policy-based routing) is an unsupported setup.

P.S. It would help if you show your current /etc/config/mwan3 file in full, as we are going to modify it, and modifying it blindly is not a good idea. Also the old configuration from vpn-policy-routing would help.

My task is that I can indicate that IP 192.168.0.111 went only through wanb, and IP 192.168.0.112 only through the wg0 interface.

my mwan3 conf:

 config rule 'https'
  3         option sticky '1'
  4         option dest_port '443'
  5         option proto 'tcp'
  6         option use_policy 'balanced'
  7
  8 config rule 'default_rule_v4'
  9         option dest_ip '0.0.0.0/0'
 10         option use_policy 'balanced'
 11         option family 'ipv4'
 12
 13 config rule 'default_rule_v6'
 14         option dest_ip '::/0'
 15         option use_policy 'balanced'
 16         option family 'ipv6'
 17
 18 config globals 'globals'
 19         option mmx_mask '0x3F00'
 20         option logging '1'
 21         option loglevel 'notice'
 22         list rt_table_lookup '220'
 23
 24 config interface 'wan'
 25         option enabled '1'
 26         list track_ip '8.8.4.4'
 27         list track_ip '8.8.8.8'
 28         list track_ip '208.67.222.222'
 29         list track_ip '208.67.220.220'
 30         option family 'ipv4'
 31         option reliability '2'
 32
 33 config interface 'wan6'
 34         option enabled '0'
 35         list track_ip '2001:4860:4860::8844'
 36         list track_ip '2001:4860:4860::8888'
 37         list track_ip '2620:0:ccd::2'
 38         list track_ip '2620:0:ccc::2'
 39         option family 'ipv6'
 40         option reliability '2'
 41
 42 config interface 'wanb'
 43         list track_ip '8.8.4.4'
 44         list track_ip '8.8.8.8'
 45         list track_ip '208.67.222.222'
 46         list track_ip '208.67.220.220'
 47         option family 'ipv4'
 48         option reliability '1'
 49         option enabled '1'
 50         option initial_state 'online'
 51         option track_method 'ping'
 52         option count '1'
 53         option size '56'
 54         option max_ttl '60'
 55         option check_quality '0'
 56         option timeout '4'
 57         option interval '10'
 58         option failure_interval '5'
 59         option recovery_interval '5'
 60         option down '5'
 61         option up '5'
 62
 63 config interface 'wanb6'
 64         option enabled '0'
 65         list track_ip '2001:4860:4860::8844'
 66         list track_ip '2001:4860:4860::8888'
 67         list track_ip '2620:0:ccd::2'
 68         list track_ip '2620:0:ccc::2'
 69         option family 'ipv6'
 70         option reliability '1'
 71
 72 config member 'wan_m1_w3'
 73         option interface 'wan'
 74         option metric '1'
 75         option weight '3'
 76
 77 config member 'wan_m2_w3'
 78         option interface 'wan'
 79         option metric '2'
 80         option weight '3'
 81
 82 config member 'wanb_m1_w2'
 83         option interface 'wanb'
 84         option metric '1'
 85         option weight '2'
 86
 87 config member 'wanb_m2_w2'
 88         option interface 'wanb'
 89         option metric '2'
 90         option weight '2'
 91
 92 config member 'wan6_m1_w3'
 93         option interface 'wan6'
 94         option metric '1'
 95         option weight '3'
 96
 97 config member 'wan6_m2_w3'
 98         option interface 'wan6'
 99         option metric '2'
100         option weight '3'
101
102 config member 'wanb6_m1_w2'
103         option interface 'wanb6'
104         option metric '1'
105         option weight '2'
106
107 config member 'wanb6_m2_w2'
108         option interface 'wanb6'
109         option metric '2'
110         option weight '2'
111
112 config policy 'wan_only'
113         list use_member 'wan_m1_w3'
114         list use_member 'wan6_m1_w3'
115
116 config policy 'wanb_only'
117         list use_member 'wanb_m1_w2'
118         list use_member 'wanb6_m1_w2'
119
120 config policy 'balanced'
121         option last_resort 'unreachable'
122         list use_member 'wan_m1_w3'
123         list use_member 'wanb_m1_w2'
124         list use_member 'wan6_m1_w3'
125         list use_member 'wanb6_m1_w2'
126
127 config policy 'wan_wanb'
128         option last_resort 'unreachable'
129         list use_member 'wan_m1_w3'
130         list use_member 'wanb_m2_w2'
131         list use_member 'wan6_m1_w3'
132         list use_member 'wanb6_m2_w2'
133
134 config policy 'wanb_wan'
135         option last_resort 'unreachable'
136         list use_member 'wan_m2_w3'
137         list use_member 'wanb_m1_w2'
138         list use_member 'wan6_m2_w3'
139         list use_member 'wanb6_m1_w2'

OK, great, let me reword to confirm my understanding:

You have wan, wanb, their IPv6 counterparts, and wg0 (which is not in mwan3 configuration). You want 192.168.0.111 to use wanb, as if nothing else exists, 192.168.0.112 to go through wg0, and all other IP addresses to be balanced between wan and wanb. Good news - you don't need pbr for this, just some adjustments to mwan3 rules.

Is this correct?

Regarding IPv6 - do you actually use IPv6, or is that just a left-over from the default configuration?

Yes, that's right. I don't use ipv6. Now remove the ipv6 settings from the config

OK, so we need to add wg0 to mwan3 as an interface, and create policies and rules. I also think that we need to create /etc/firewall.user as a workaround for https://github.com/openwrt/packages/issues/19607.

I can either spend 60 minutes and write down a fully tested config for you, or we can have a video chat right now, and I will guide you through this.

The video chat link is: <deleted>

Thank you,

I removed the PBR policy and packages.
I have configured the MWN3 with the rule I need and it solved my problem

Right now I'll try to do it through mwan3. If I fail, I will turn to you for help.

Very strange, if i add interface wg0 in mwan3, he added, but after any time wg0 interface lost

I don't understand the word "lost".

Perhaps you need to create a rule that prevents packets created by wireguard from looping back. I.e. direct them, based on the destination IP and port, to wan or wanb.

But, I have already told you: with VPNs, there is a bug in mwan3, that needs to be worked around. Please save this as /etc/firewall.mwan3-fix and make executable:

#!/bin/sh
iptables -t mangle -D PREROUTING -m comment --comment "Do not inherit the mark of encrypted packets" -j MARK --set-xmark 0x0/0x3f00 || echo "Never mind"
iptables -t mangle -I PREROUTING 1 -m comment --comment "Do not inherit the mark of encrypted packets" -j MARK --set-xmark 0x0/0x3f00

And then add this to /etc/config/firewall:

config include
	option path '/etc/firewall.mwan3-fix'
	option type 'script'

And then /etc/init.d/firewall restart ; /etc/init.d/mwan3 restart.

Ok, try now

Another thing to check is that the metric is set on wg0, and that it is different from the metrics of wan and wanb.

Yes metric different

firewall.mwan3-fix added.
Add to maw3 needed rules, but no effect needed ip go to default wan.
My maw3 conf:

config rule 'https'
	option sticky '1'
	option dest_port '443'
	option proto 'tcp'
	option use_policy 'balanced'

config rule 'default_rule_v4'
	option dest_ip '0.0.0.0/0'
	option use_policy 'balanced'
	option family 'ipv4'

config globals 'globals'
	option mmx_mask '0x3F00'
	option logging '1'
	option loglevel 'notice'
	list rt_table_lookup '220'

config interface 'wan'
	option enabled '1'
	list track_ip '8.8.4.4'
	list track_ip '8.8.8.8'
	list track_ip '208.67.222.222'
	list track_ip '208.67.220.220'
	option family 'ipv4'
	option reliability '2'

config interface 'wanb'
	list track_ip '8.8.4.4'
	list track_ip '8.8.8.8'
	list track_ip '208.67.222.222'
	list track_ip '208.67.220.220'
	option family 'ipv4'
	option reliability '1'
	option enabled '1'
	option initial_state 'online'
	option track_method 'ping'
	option count '1'
	option size '56'
	option max_ttl '60'
	option check_quality '0'
	option timeout '4'
	option interval '10'
	option failure_interval '5'
	option recovery_interval '5'
	option down '5'
	option up '5'

config member 'wan_m1_w3'
	option interface 'wan'
	option metric '1'
	option weight '3'

config member 'wan_m2_w3'
	option interface 'wan'
	option metric '2'
	option weight '3'

config member 'wanb_m1_w2'
	option interface 'wanb'
	option metric '1'
	option weight '2'

config member 'wanb_m2_w2'
	option interface 'wanb'
	option metric '2'
	option weight '2'

config policy 'wan_only'
	list use_member 'wan_m1_w3'
	option last_resort 'unreachable'

config policy 'wanb_only'
	list use_member 'wanb_m1_w2'
	option last_resort 'unreachable'

config policy 'balanced'
	option last_resort 'unreachable'
	list use_member 'wan_m1_w3'
	list use_member 'wanb_m1_w2'

config policy 'wan_wanb'
	option last_resort 'unreachable'
	list use_member 'wan_m1_w3'
	list use_member 'wanb_m2_w2'

config policy 'wanb_wan'
	option last_resort 'unreachable'
	list use_member 'wan_m2_w3'
	list use_member 'wanb_m1_w2'

config interface 'wg0'
	option enabled '1'
	option initial_state 'online'
	option family 'ipv4'
	option track_method 'ping'
	option count '1'
	option size '56'
	option max_ttl '60'
	option timeout '4'
	option interval '10'
	option failure_interval '5'
	option recovery_interval '5'
	option down '5'
	option up '5'
	option interface 'wg0'
	option metric '3'
	option weight '4'
	list track_ip '8.8.8.8'
	list track_ip '0.0.0.0'
	option reliability '1'
	list use_member 'wg0'
	list use_member 'wg0'
	option last_resort 'unreachable'

config rule 'server2Win11'
	option proto 'all'
	option src_ip '192.168.0.193'
	option dest_ip '0.0.0.0/0'
	option sticky '0'
	option use_policy 'wg0_only'

config member 'wg0_m3_w4'
	option interface 'wg0'
	option metric '3'
	option weight '4'

config policy 'wg0_only'
	list use_member 'wg0_m3_w4'
	option last_resort 'unreachable'

Major finding from the video chat: even after uninstalling pbr, its iptables rules were still there. However the policy still doesn't work - debugging now.

Solved.

It didn't work because the default route through the WireGuard interface was not being created. The "default route" checkbox does not work, but after ticking "Route Allowed IPs" in the WireGuard peer settings everything started working.

1 Like

The problem is mwan3 is using iptables while pbr is configured to use nft.

With the appropriate config (README has details) pbr-iptables and mwan3 should work fine on 21.02 (as the pbr predecessor did), possibly pbr in iptables mode will work with mwan3 on 22.03 as well.

With mwan3 on 22.03 you neet to ensure you have iptables-nft installed and not iptables-zz-legacy. Unfortunately the legacy version is the default that you get when you install mwan3.

3 Likes