PBR not working with mwan3

I to have problem on version 23.03.3
I have 2 wan with mwan3.
PRB not work.
pbr give error on wan:
root@OpenWrt:~# /etc/init.d/pbr reload verbose=2
Activating traffic killswitch [✓]

Warning: iptables-legacy tables present, use iptables-legacy-save to see them

Setting up routing for 'wan/eth2/192.168.160.1' Error: argument "256
257" is wrong: invalid table ID

[✗]
Setting up routing for 'wanb/eth3/192.168.1.254' [✓]
Setting up routing for 'wg0/10.66.66.12' [✓]
Setting up routing for 'wg1/10.66.66.5' [✓]
Setting up routing for 'WG2/10.7.0.3' [✓]
Creating TOR redirects [✗]
Routing 'server2Win11' via wg0 [✗]
Deactivating traffic killswitch [✓]
pbr 1.0.1-16 monitoring interfaces: wan wanb wg0 wg1 WG2
Command failed: Invalid argument
pbr 1.0.1-16 (nft) started with gateways:
wanb/eth3/192.168.1.254
wg0/10.66.66.12
wg1/10.66.66.5
WG2/10.7.0.3
ERROR: Failed to set up 'wan/eth2/192.168.160.1'!
ERROR: Failed to set up 'tor/53->9053/80,443->9040'!
ERROR:
all
ERROR: Insertion failed for IPv4 for policy server2Win11
ERROR:
nft 'add rule inet fw4 pbr_prerouting ip saddr @pbr_wg0_4_src_ip_cfg066ff5 ip daddr {} tcp sport {0-65535} tcp dport {0-65535} goto pbr_mark_0x030000 comment "server2Win11"'

Don`t understand how fix it.

It would be better to start your own topic, rather than mixing it with existing ones.
@tmomas @psherman

First of all, please fix the formatting by using preformated text for the console output.
Just make sure you "sandwich" your text between two rows of backtick characters ` (which themselves will be invisible in the preview) looking in something like this in the editor:
```
Your Pasted Text as preformatted text with fixed width font
1
1111 (note with fixed-width fonts the numbers are right-aligned)
```
but looking like this in the rendered forum:

Your Pasted Text as preformatted text with fixed width font
   1
1111 (note with fixed-width fonts the numbers are right-aligned)

Then post the output of ubus call system board; opkg list-installed

The problem is that you use two packages with overlapping functionality. They are not even supposed to work together. Please convert your policy rules 100% to mwan3.

2 Likes

Thanks for you answer.

root@OpenWrt:~# opkg list-installed
base-files - 1498-r20028-43d71ad93e
blkid - 2.37.4-1
block-mount - 2022-06-02-93369be0-2
bnx2-firmware - 20220411-1
busybox - 1.35.0-5
ca-bundle - 20211016-1
cgi-io - 2022-08-10-901b0f04-21
coreutils - 9.0-2
coreutils-dirname - 9.0-2
curl - 7.86.0-2
dnsmasq-full - 2.86-15
dropbear - 2022.82-2
e2fsprogs - 1.46.5-2
fdisk - 2.37.4-1
firewall4 - 2022-10-18-7ae5e14b-1
frr - 8.2.0-1
frr-libfrr - 8.2.0-1
frr-pbrd - 8.2.0-1
fstools - 2022-06-02-93369be0-2
fwtool - 2019-11-12-8f7fe925-1
getrandom - 2021-08-03-205defb5-2
glib2 - 2.70.5-4
grub2 - 2.06-3
grub2-bios-setup - 2.06-3
grub2-efi - 2.06-3
iconv - 1.16-1
idn - 1.36-1
ip-full - 5.15.0-3
ip-tiny - 5.15.0-3
ip6tables-zz-legacy - 1.8.7-7
iperf3 - 3.11-1
ipset - 7.15-2
ipset-dns - 2017-10-08-ade2cf88-1
iptables-mod-conntrack-extra - 1.8.7-7
iptables-mod-ipopt - 1.8.7-7
iptables-nft - 1.8.7-7
iptables-zz-legacy - 1.8.7-7
jansson4 - 2.13.1-2
jshn - 2022-05-15-d2223ef9-1
jsonfilter - 2018-02-04-c7e938d6-1
kernel - 5.10.161-1-9309db0b721b9fb3fe826181264a78c4
kmod-amazon-ena - 5.10.161-1
kmod-amd-xgbe - 5.10.161-1
kmod-bnx2 - 5.10.161-1
kmod-button-hotplug - 5.10.161-3
kmod-crypto-crc32c - 5.10.161-1
kmod-crypto-hash - 5.10.161-1
kmod-crypto-kpp - 5.10.161-1
kmod-crypto-lib-chacha20 - 5.10.161-1
kmod-crypto-lib-chacha20poly1305 - 5.10.161-1
kmod-crypto-lib-curve25519 - 5.10.161-1
kmod-crypto-lib-poly1305 - 5.10.161-1
kmod-e1000 - 5.10.161-1
kmod-e1000e - 5.10.161-1
kmod-forcedeth - 5.10.161-1
kmod-fs-ext4 - 5.10.161-1
kmod-fs-vfat - 5.10.161-1
kmod-hwmon-core - 5.10.161-1
kmod-i2c-algo-bit - 5.10.161-1
kmod-i2c-core - 5.10.161-1
kmod-igb - 5.10.161-1
kmod-igc - 5.10.161-1
kmod-input-core - 5.10.161-1
kmod-ip6tables - 5.10.161-1
kmod-ipt-conntrack - 5.10.161-1
kmod-ipt-conntrack-extra - 5.10.161-1
kmod-ipt-core - 5.10.161-1
kmod-ipt-ipopt - 5.10.161-1
kmod-ipt-ipset - 5.10.161-1
kmod-ipt-nat - 5.10.161-1
kmod-ipt-nat-extra - 5.10.161-1
kmod-ipt-raw - 5.10.161-1
kmod-ixgbe - 5.10.161-1
kmod-lib-crc-ccitt - 5.10.161-1
kmod-lib-crc16 - 5.10.161-1
kmod-lib-crc32c - 5.10.161-1
kmod-libphy - 5.10.161-1
kmod-mdio - 5.10.161-1
kmod-mdio-devres - 5.10.161-1
kmod-mii - 5.10.161-1
kmod-nf-conntrack - 5.10.161-1
kmod-nf-conntrack-netlink - 5.10.161-1
kmod-nf-conntrack6 - 5.10.161-1
kmod-nf-flow - 5.10.161-1
kmod-nf-ipt - 5.10.161-1
kmod-nf-ipt6 - 5.10.161-1
kmod-nf-log - 5.10.161-1
kmod-nf-log6 - 5.10.161-1
kmod-nf-nat - 5.10.161-1
kmod-nf-reject - 5.10.161-1
kmod-nf-reject6 - 5.10.161-1
kmod-nfnetlink - 5.10.161-1
kmod-nft-compat - 5.10.161-1
kmod-nft-core - 5.10.161-1
kmod-nft-fib - 5.10.161-1
kmod-nft-nat - 5.10.161-1
kmod-nft-offload - 5.10.161-1
kmod-nls-base - 5.10.161-1
kmod-nls-cp437 - 5.10.161-1
kmod-nls-iso8859-1 - 5.10.161-1
kmod-nls-utf8 - 5.10.161-1
kmod-phy-realtek - 5.10.161-1
kmod-ppp - 5.10.161-1
kmod-pppoe - 5.10.161-1
kmod-pppox - 5.10.161-1
kmod-pps - 5.10.161-1
kmod-ptp - 5.10.161-1
kmod-r8169 - 5.10.161-1
kmod-scsi-core - 5.10.161-1
kmod-slhc - 5.10.161-1
kmod-tg3 - 5.10.161-1
kmod-udptunnel4 - 5.10.161-1
kmod-udptunnel6 - 5.10.161-1
kmod-usb-core - 5.10.161-1
kmod-usb-storage - 5.10.161-1
kmod-usb-storage-extras - 5.10.161-1
kmod-wireguard - 5.10.161-1
libatomic1 - 11.2.0-4
libattr - 2.5.1-1
libblkid1 - 2.37.4-1
libblobmsg-json20220515 - 2022-05-15-d2223ef9-1
libbpf20220308 - 2022-03-08-04c465fd-1
libc - 1.2.3-4
libcap - 2.63-1
libcharset1 - 1.16-1
libcomerr0 - 1.46.5-2
libcurl4 - 7.86.0-2
libelf1 - 0.186-1
libevent2-7 - 2.1.12-1
libext2fs2 - 1.46.5-2
libf2fs6 - 1.14.0-3
libfdisk1 - 2.37.4-1
libffi - 3.4.2-2
libgcc1 - 11.2.0-4
libgmp10 - 6.2.1-1
libiconv-full2 - 1.16-1
libidn - 1.36-1
libip4tc2 - 1.8.7-7
libip6tc2 - 1.8.7-7
libipset13 - 7.15-2
libiptext-nft0 - 1.8.7-7
libiptext0 - 1.8.7-7
libiptext6-0 - 1.8.7-7
libiwinfo-data - 2022-12-15-8d158096-1
libiwinfo-lua - 2022-12-15-8d158096-1
libiwinfo20210430 - 2022-12-15-8d158096-1
libjson-c5 - 0.15-2
libjson-script20220515 - 2022-05-15-d2223ef9-1
liblua5.1.5 - 5.1.5-10
liblucihttp-lua - 2022-07-08-6e68a106-1
liblucihttp0 - 2022-07-08-6e68a106-1
libmbedtls12 - 2.28.2-1
libmnl0 - 1.0.5-1
libmount1 - 2.37.4-1
libncurses6 - 6.3-2
libnetfilter-conntrack3 - 1.0.9-2
libnettle8 - 3.7.3-2
libnfnetlink0 - 1.0.2-1
libnftnl11 - 1.2.1-2
libnghttp2-14 - 1.44.0-1
libnl-tiny1 - 2021-11-21-8e0555fb-1
libopenssl1.1 - 1.1.1t-1
libowipcalc - 7
libpcre - 8.45-3
libpcre2 - 10.37-1
libpthread - 1.2.3-4
librt - 1.2.3-4
libsmartcols1 - 2.37.4-1
libss2 - 1.46.5-2
libssh2-1 - 1.9.0-2
libubox20220515 - 2022-05-15-d2223ef9-1
libubus-lua - 2022-06-01-2bebf93c-1
libubus20220601 - 2022-06-01-2bebf93c-1
libuci-lua - 2021-10-22-f84f49f0-6
libuci20130104 - 2021-10-22-f84f49f0-6
libuclient20201210 - 2021-05-14-6a6011df-1
libucode20220812 - 2022-12-02-46d93c9c-1
libustream-wolfssl20201210 - 2022-12-08-9217ab46-2
libuuid1 - 2.37.4-1
libwolfssl5.5.4.ee39414e - 5.5.4-stable-1
libxtables12 - 1.8.7-7
libyang - 2.0.112-1
logd - 2021-08-03-205defb5-2
lua - 5.1.5-10
luabitop - 1.0.2-1
luasec - 0.9-1
luasocket - 2019-04-21-733af884-1
luci - git-20.074.84698-ead5e81
luci-app-firewall - git-22.089.67563-7e3c1b4
luci-app-mwan3 - git-21.340.50573-2af8158
luci-app-opkg - git-22.273.29004-9f6876b
luci-app-pbr - 1.1.0-1
luci-app-ruantiblock - 0.9.7-0
luci-app-wireguard - git-23.018.72712-6d712c3
luci-base - git-23.039.29681-007c243
luci-i18n-base-ru - git-23.058.14209-1027acd
luci-i18n-mwan3-ru - git-23.058.14209-1027acd
luci-i18n-pbr-ru - git-23.058.14209-1027acd
luci-i18n-ruantiblock-ru - 0.9.7-0
luci-i18n-wireguard-ru - git-22.316.76227-771eb78
luci-lib-base - git-20.232.39649-1f6dc29
luci-lib-ip - git-20.250.76529-62505bd
luci-lib-jsonc - git-22.097.61921-7513345
luci-lib-nixio - git-20.234.06894-c4a4e43
luci-mod-admin-full - git-19.253.48496-3f93650
luci-mod-network - git-22.345.48602-4853e7b
luci-mod-status - git-23.038.33313-b256644
luci-mod-system - git-23.013.73113-588381e
luci-proto-ipv6 - git-21.148.48881-79947af
luci-proto-ppp - git-21.158.38888-88b9d84
luci-proto-wireguard - git-22.327.45657-14403fe
luci-ssl - git-20.244.36115-e10f954
luci-theme-bootstrap - git-22.288.45147-96ec0cd
mc - 4.8.27-3
mkf2fs - 1.14.0-3
mtd - 26
mwan3 - 2.11.4-1
nano-full - 7.2-1
netifd - 2022-08-25-76d2d41b-1
nftables-json - 1.0.2-2.1
odhcp6c - 2022-08-05-7d21e8d8-18
odhcpd-ipv6only - 2023-01-02-4a673e1c-2
openwrt-keyring - 2022-03-25-62471e69-3
opkg - 2022-02-24-d038e5b6-1
owipcalc - 7
partx-utils - 2.37.4-1
pbr - 1.0.1-16
ppp - 2.4.9.git-2021-01-04-3
ppp-mod-pppoe - 2.4.9.git-2021-01-04-3
procd - 2022-06-01-7a009685-2
procd-seccomp - 2022-06-01-7a009685-2
procd-ujail - 2022-06-01-7a009685-2
px5g-wolfssl - 6.2
r8169-firmware - 20220411-1
resolveip - 2
rpcd - 2022-12-15-7de4820c-1
rpcd-mod-file - 2022-12-15-7de4820c-1
rpcd-mod-iwinfo - 2022-12-15-7de4820c-1
rpcd-mod-luci - 20210614
rpcd-mod-rrdns - 20170710
ruantiblock - 0.9.7-0
ruantiblock-mod-lua - 0.9.7-0
terminfo - 6.3-2
tor-basic - 0.4.7.10-1
tor-geoip - 0.4.7.10-1
ubox - 2021-08-03-205defb5-2
ubus - 2022-06-01-2bebf93c-1
ubusd - 2022-06-01-2bebf93c-1
uci - 2021-10-22-f84f49f0-6
uclient-fetch - 2021-05-14-6a6011df-1
ucode - 2022-12-02-46d93c9c-1
ucode-mod-fs - 2022-12-02-46d93c9c-1
ucode-mod-ubus - 2022-12-02-46d93c9c-1
ucode-mod-uci - 2022-12-02-46d93c9c-1
uhttpd - 2022-10-31-23977554-1
uhttpd-mod-ubus - 2022-10-31-23977554-1
urandom-seed - 3
urngd - 2020-01-21-c7f7b6b6-1
usign - 2020-05-23-f1f65026-1
wg-installer-client - 28
wireguard-tools - 1.0.20210424-3
xtables-legacy - 1.8.7-7
xtables-nft - 1.8.7-7
zlib - 1.2.11-6

Sorry i dont now how its do.
How convert policy to mwan3?

Please post your current pbr rules here, and I will try to formulate the equivalent ones for mwan3. So that you can uninstall pbr, and have only mwan3, without any loss of functionality.

Or even better, formulate what you are trying to achieve with mwan3 and pbr, so that I get the full picture and can work from that.

2 Likes

mwan3 distributes the load between 2 wan. I need to be able to assign my gateway to any IP in the network. In version 23.03, I did it using vpn-policy-routing and everything worked.

Maybe it is possible to send the right IP to the right gateway through mwan3?

I don't fully understand.

mwan3 distributes the load between 2 wan.

OK, I understand this - you are using load balancing, not fail-over. This is indeed a valid use case for mwan3.

I need to be able to assign my gateway to any IP in the network.

Please reword, I don't understand this at all. Or maybe provide examples.

In version 23.03, I did it using vpn-policy-routing and everything worked.

This was not supposed to work. Any combination of two packages that provide policy-based routing (and load balancing is a form of policy-based routing) is an unsupported setup.

P.S. It would help if you show your current /etc/config/mwan3 file in full, as we are going to modify it, and modifying it blindly is not a good idea. Also the old configuration from vpn-policy-routing would help.

My task is that I can indicate that IP 192.168.0.111 went only through wanb, and IP 192.168.0.112 only through the wg0 interface.

my mwan3 conf:

 config rule 'https'
  3         option sticky '1'
  4         option dest_port '443'
  5         option proto 'tcp'
  6         option use_policy 'balanced'
  7
  8 config rule 'default_rule_v4'
  9         option dest_ip '0.0.0.0/0'
 10         option use_policy 'balanced'
 11         option family 'ipv4'
 12
 13 config rule 'default_rule_v6'
 14         option dest_ip '::/0'
 15         option use_policy 'balanced'
 16         option family 'ipv6'
 17
 18 config globals 'globals'
 19         option mmx_mask '0x3F00'
 20         option logging '1'
 21         option loglevel 'notice'
 22         list rt_table_lookup '220'
 23
 24 config interface 'wan'
 25         option enabled '1'
 26         list track_ip '8.8.4.4'
 27         list track_ip '8.8.8.8'
 28         list track_ip '208.67.222.222'
 29         list track_ip '208.67.220.220'
 30         option family 'ipv4'
 31         option reliability '2'
 32
 33 config interface 'wan6'
 34         option enabled '0'
 35         list track_ip '2001:4860:4860::8844'
 36         list track_ip '2001:4860:4860::8888'
 37         list track_ip '2620:0:ccd::2'
 38         list track_ip '2620:0:ccc::2'
 39         option family 'ipv6'
 40         option reliability '2'
 41
 42 config interface 'wanb'
 43         list track_ip '8.8.4.4'
 44         list track_ip '8.8.8.8'
 45         list track_ip '208.67.222.222'
 46         list track_ip '208.67.220.220'
 47         option family 'ipv4'
 48         option reliability '1'
 49         option enabled '1'
 50         option initial_state 'online'
 51         option track_method 'ping'
 52         option count '1'
 53         option size '56'
 54         option max_ttl '60'
 55         option check_quality '0'
 56         option timeout '4'
 57         option interval '10'
 58         option failure_interval '5'
 59         option recovery_interval '5'
 60         option down '5'
 61         option up '5'
 62
 63 config interface 'wanb6'
 64         option enabled '0'
 65         list track_ip '2001:4860:4860::8844'
 66         list track_ip '2001:4860:4860::8888'
 67         list track_ip '2620:0:ccd::2'
 68         list track_ip '2620:0:ccc::2'
 69         option family 'ipv6'
 70         option reliability '1'
 71
 72 config member 'wan_m1_w3'
 73         option interface 'wan'
 74         option metric '1'
 75         option weight '3'
 76
 77 config member 'wan_m2_w3'
 78         option interface 'wan'
 79         option metric '2'
 80         option weight '3'
 81
 82 config member 'wanb_m1_w2'
 83         option interface 'wanb'
 84         option metric '1'
 85         option weight '2'
 86
 87 config member 'wanb_m2_w2'
 88         option interface 'wanb'
 89         option metric '2'
 90         option weight '2'
 91
 92 config member 'wan6_m1_w3'
 93         option interface 'wan6'
 94         option metric '1'
 95         option weight '3'
 96
 97 config member 'wan6_m2_w3'
 98         option interface 'wan6'
 99         option metric '2'
100         option weight '3'
101
102 config member 'wanb6_m1_w2'
103         option interface 'wanb6'
104         option metric '1'
105         option weight '2'
106
107 config member 'wanb6_m2_w2'
108         option interface 'wanb6'
109         option metric '2'
110         option weight '2'
111
112 config policy 'wan_only'
113         list use_member 'wan_m1_w3'
114         list use_member 'wan6_m1_w3'
115
116 config policy 'wanb_only'
117         list use_member 'wanb_m1_w2'
118         list use_member 'wanb6_m1_w2'
119
120 config policy 'balanced'
121         option last_resort 'unreachable'
122         list use_member 'wan_m1_w3'
123         list use_member 'wanb_m1_w2'
124         list use_member 'wan6_m1_w3'
125         list use_member 'wanb6_m1_w2'
126
127 config policy 'wan_wanb'
128         option last_resort 'unreachable'
129         list use_member 'wan_m1_w3'
130         list use_member 'wanb_m2_w2'
131         list use_member 'wan6_m1_w3'
132         list use_member 'wanb6_m2_w2'
133
134 config policy 'wanb_wan'
135         option last_resort 'unreachable'
136         list use_member 'wan_m2_w3'
137         list use_member 'wanb_m1_w2'
138         list use_member 'wan6_m2_w3'
139         list use_member 'wanb6_m1_w2'

OK, great, let me reword to confirm my understanding:

You have wan, wanb, their IPv6 counterparts, and wg0 (which is not in mwan3 configuration). You want 192.168.0.111 to use wanb, as if nothing else exists, 192.168.0.112 to go through wg0, and all other IP addresses to be balanced between wan and wanb. Good news - you don't need pbr for this, just some adjustments to mwan3 rules.

Is this correct?

Regarding IPv6 - do you actually use IPv6, or is that just a left-over from the default configuration?

Yes, that's right. I don't use ipv6. Now remove the ipv6 settings from the config

OK, so we need to add wg0 to mwan3 as an interface, and create policies and rules. I also think that we need to create /etc/firewall.user as a workaround for https://github.com/openwrt/packages/issues/19607.

I can either spend 60 minutes and write down a fully tested config for you, or we can have a video chat right now, and I will guide you through this.

The video chat link is: <deleted>

Thank you,

I removed the PBR policy and packages.
I have configured the MWN3 with the rule I need and it solved my problem

Right now I'll try to do it through mwan3. If I fail, I will turn to you for help.

Very strange, if i add interface wg0 in mwan3, he added, but after any time wg0 interface lost

I don't understand the word "lost".

Perhaps you need to create a rule that prevents packets created by wireguard from looping back. I.e. direct them, based on the destination IP and port, to wan or wanb.

But, I have already told you: with VPNs, there is a bug in mwan3, that needs to be worked around. Please save this as /etc/firewall.mwan3-fix and make executable:

#!/bin/sh
iptables -t mangle -D PREROUTING -m comment --comment "Do not inherit the mark of encrypted packets" -j MARK --set-xmark 0x0/0x3f00 || echo "Never mind"
iptables -t mangle -I PREROUTING 1 -m comment --comment "Do not inherit the mark of encrypted packets" -j MARK --set-xmark 0x0/0x3f00

And then add this to /etc/config/firewall:

config include
	option path '/etc/firewall.mwan3-fix'
	option type 'script'

And then /etc/init.d/firewall restart ; /etc/init.d/mwan3 restart.

Ok, try now

Another thing to check is that the metric is set on wg0, and that it is different from the metrics of wan and wanb.