Mwan3 rule to specific interface

ambad4u · November 26, 2023, 1:29pm

hi & Good Day to All!,

i am trying something new, kindly refer to the image below

i have 3 instance(s) of OpenWRT x86 inside proxmox w/ its wan ports in 'pcie passthru mode'.

OpenWRT-A: exclusive access to ISP-A
OpenWRT-B: exclusive access to ISP-B
OpenWRT-X: MWAN3 in failover mode (will add loadbalance if i can)

my current use case is to have a choice on where to connect..., either on A, B or X

question is: how complex would be the configuration if it is possible to merge or use a single instance of OpenWRT?

lleachii · November 26, 2023, 1:40pm

Part 1 (which is extremely similar): Yet Another Dual/Multi ISP Question

It should be easy to configure 1 instead of 3 routers.

Especially given you have all the necessary configs already, it should be simple.

ambad4u · November 26, 2023, 1:43pm

i have the 3 working indeed, but i can't still wrap it in my head using 1
mwan3 is also giving me a hard time understanding it
i may need to step back a bit to get the whole/big picture.

lleachii · November 26, 2023, 1:56pm

I thought of 2 paths:

You haven't shown the final configs; but from the diagram and written description here, it seems simple enough to collapse Router_A and Router_B into a unified config on Router_X. It seems you merely need to configure WANs A and B (maybe setup LANs A and B as well), and lastly perhaps minor adjustments to mwan3.

Another option I just considered - setup a 4th router that you will configure. I'm not sure if this would be more or less confusing for you. Just a suggestion. It leaves the original 3 routers untouched; and depending on how you setup the interfaces on the Host, you can merely turn the other 3 routers off when done (i.e. not risk messing up the configs on Routers A, B and X).

Using the original 3 configs as your guide:

Setup LANs
Setup WANs
[Re]Configure mwan3

ambad4u · November 26, 2023, 2:16pm

i can indeed spawn a new vm in order to test. thank you for the suggestion!

plan would be:
after the creation of the 4th OpenWRT instance, i configure MWAN3 as-is/normally..., then i create additional tables (2) and link them the specific interfaces. so, the un-altered 'lan' interface is now (X) which provides failover/loadbalance..., while the 'special' interfaces linked to the 'custom' tables are the exclusive access to each ISP...

am i in the right direction?

ambad4u · December 4, 2023, 7:44am

its been approx a week and i am stuck...

what i did was somewhat follow my previous guide in: Yet Another Dual/Multi ISP Question

created custom 'wan/lan' interfaces, tables, routes, rules, firewall for each.

mwan3 is configured in the last part...
both ('wan_pldt' | "lan_pldt") && ('wan_globe' | "lan_globe") are working as expected.

as soon as i configure mwan3, internet is crippled...
devices on lan_pldt or lan_globe i can ping via ip but not dns
devices on lan_xmulti cannot ping the internet

there is an error on mwan3 status

Interface status:
 interface wan_pldt is error (16) and tracking is active
 interface wan_globe is error (16) and tracking is active

Current ipv4 policies:
distributed:
 wan_globe (50%)
 wan_pldt (50%)
only_globe:
 wan_globe (100%)
only_pldt:
 wan_pldt (100%)

My configs for reference:

/etc/iproute2/rt_tables

#
# reserved values
#
128     prelocal
255     local
254     main
253     default
0       unspec
#
# local
#
#1      inr.ruhep
333     oblivion
1000    table-pldt
2000    table-globe

/etc/config/network


config interface 'loopback'
	option device 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option ula_prefix 'fd9b:37bf:f49e::/48'
	option packet_steering '1'

config device
	option type 'bridge'
	option name 'core'
	list ports 'eth0'
	option bridge_empty '1'
	option ipv6 '0'

config bridge-vlan
	option device 'core'
	option vlan '404'
	list ports 'eth0:u*'

config interface 'core'
	option proto 'static'
	option device 'core.404'
	option ipaddr '10.10.10.254'
	option netmask '255.255.255.0'
	option defaultroute '0'
	option delegate '0'

config interface 'wan_pldt'
	option proto 'dhcp'
	option device 'eth1'
	option metric '10'
	option ip4table 'table-pldt'
	option delegate '0'
	option defaultroute '0'

config interface 'wan_globe'
	option proto 'dhcp'
	option device 'eth2'
	option metric '20'
	option ip4table 'table-globe'
	option delegate '0'
	option defaultroute '0'

config interface 'lan_pldt'
	option proto 'static'
	option device 'eth3'
	option ipaddr '172.31.10.254'
	option netmask '255.255.255.0'
	option defaultroute '0'
	option ip4table 'table-pldt'
	option delegate '0'

config interface 'lan_globe'
	option proto 'static'
	option device 'eth4'
	option ipaddr '172.31.20.254'
	option netmask '255.255.255.0'
	option defaultroute '0'
	option ip4table 'table-globe'
	option delegate '0'

config interface 'lan_xmulti'
	option proto 'static'
	option device 'eth5'
	option ipaddr '172.16.99.254'
	option netmask '255.255.255.0'

config route
	option target '0.0.0.0'
	option netmask '0.0.0.0'
	option gateway '10.20.30.254'
	option table 'table-pldt'
	option interface 'wan_pldt'

config route
	option target '0.0.0.0'
	option netmask '0.0.0.0'
	option gateway '10.20.70.254'
	option table 'table-globe'
	option interface 'wan_globe'

config rule
	option in 'lan_pldt'
	option priority '2'
	option lookup 'table-pldt'

config rule
	option in 'lan_globe'
	option priority '3'
	option lookup 'table-globe'

/etc/config/firewall


config defaults
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option synflood_protect '1'

config zone
	option name 'zlan_globe'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'REJECT'
	list network 'lan_globe'

config zone
	option name 'zlan_pldt'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'REJECT'
	list network 'lan_pldt'

config zone
	option name 'zlan_multi'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'REJECT'
	list network 'lan_xmulti'

config zone
	option name 'zglobe'
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option masq '1'
	option mtu_fix '1'
	list network 'wan_globe'

config zone
	option name 'zpldt'
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option masq '1'
	option mtu_fix '1'
	list network 'wan_pldt'

config zone
	option name 'zcore'
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'REJECT'
	list network 'core'

config rule
	option name 'Allow-DHCP-Renew'
	option src 'zpldt'
	option proto 'udp'
	option dest_port '68'
	option target 'ACCEPT'
	option family 'ipv4'

config rule
	option name 'Allow-Ping'
	option src 'zpldt'
	option proto 'icmp'
	option icmp_type 'echo-request'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-IGMP'
	option src 'zpldt'
	option proto 'igmp'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-DHCPv6'
	option src 'zpldt'
	option proto 'udp'
	option dest_port '546'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-MLD'
	option src 'zpldt'
	option proto 'icmp'
	option src_ip 'fe80::/10'
	list icmp_type '130/0'
	list icmp_type '131/0'
	list icmp_type '132/0'
	list icmp_type '143/0'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Input'
	option src 'zpldt'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	list icmp_type 'router-solicitation'
	list icmp_type 'neighbour-solicitation'
	list icmp_type 'router-advertisement'
	list icmp_type 'neighbour-advertisement'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Forward'
	option src 'zpldt'
	option dest '*'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-IPSec-ESP'
	option src 'zpldt'
	option dest 'lan'
	option proto 'esp'
	option target 'ACCEPT'

config rule
	option name 'Allow-ISAKMP'
	option src 'zpldt'
	option dest 'lan'
	option dest_port '500'
	option proto 'udp'
	option target 'ACCEPT'

config rule
	option name 'Allow-DHCP-Renew'
	option src 'zglobe'
	option proto 'udp'
	option dest_port '68'
	option target 'ACCEPT'
	option family 'ipv4'

config rule
	option name 'Allow-Ping'
	option src 'zglobe'
	option proto 'icmp'
	option icmp_type 'echo-request'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-IGMP'
	option src 'zglobe'
	option proto 'igmp'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-DHCPv6'
	option src 'zglobe'
	option proto 'udp'
	option dest_port '546'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-MLD'
	option src 'zglobe'
	option proto 'icmp'
	option src_ip 'fe80::/10'
	list icmp_type '130/0'
	list icmp_type '131/0'
	list icmp_type '132/0'
	list icmp_type '143/0'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Input'
	option src 'zglobe'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	list icmp_type 'router-solicitation'
	list icmp_type 'neighbour-solicitation'
	list icmp_type 'router-advertisement'
	list icmp_type 'neighbour-advertisement'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Forward'
	option src 'zglobe'
	option dest '*'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-IPSec-ESP'
	option src 'zglobe'
	option dest 'lan'
	option proto 'esp'
	option target 'ACCEPT'

config rule
	option name 'Allow-ISAKMP'
	option src 'zglobe'
	option dest 'lan'
	option dest_port '500'
	option proto 'udp'
	option target 'ACCEPT'

config rule
	option name 'Allow-Ports'
	option src 'zcore'
	option target 'ACCEPT'
	option family 'ipv4'
	list proto 'tcp'
	list proto 'udp'
	list proto 'icmp'
	option dest_port '22 67 80 443'

config rule
	option name 'Allow_only_AP_to_wan'
	option src 'zcore'
	option dest 'zglobe'
	option target 'ACCEPT'
	list proto 'all'
	list src_ip '10.10.10.252'

config forwarding
	option src 'zlan_pldt'
	option dest 'zpldt'

config forwarding
	option src 'zlan_globe'
	option dest 'zglobe'

config forwarding
	option src 'zlan_multi'
	option dest 'zglobe'

config forwarding
	option src 'zlan_multi'
	option dest 'zpldt'

/etc/config/mwan3


config globals 'globals'
	option mmx_mask '0x3F00'

config interface 'wan_pldt'
	option enabled '1'
	option initial_state 'offline'
	option family 'ipv4'
	list track_ip '1.1.1.1'
	list track_ip '8.8.8.8'
	list track_ip '9.9.9.9'
	option track_method 'ping'
	option reliability '1'
	option count '1'
	option size '56'
	option max_ttl '60'
	option timeout '4'
	option interval '10'
	option failure_interval '5'
	option recovery_interval '5'
	option down '5'
	option up '5'

config interface 'wan_globe'
	option enabled '1'
	option initial_state 'offline'
	option family 'ipv4'
	list track_ip '1.1.1.1'
	list track_ip '8.8.8.8'
	list track_ip '9.9.9.9'
	option track_method 'ping'
	option reliability '1'
	option count '1'
	option size '56'
	option max_ttl '60'
	option timeout '4'
	option interval '10'
	option failure_interval '5'
	option recovery_interval '5'
	option down '5'
	option up '5'

config member 'pldt'
	option interface 'wan_pldt'
	option metric '1'
	option weight '1'

config member 'globe'
	option interface 'wan_globe'
	option metric '1'
	option weight '1'

config policy 'only_pldt'
	list use_member 'pldt'
	option last_resort 'unreachable'

config policy 'only_globe'
	list use_member 'globe'
	option last_resort 'unreachable'

config policy 'distributed'
	list use_member 'pldt'
	list use_member 'globe'
	option last_resort 'unreachable'

config rule 'web'
	option proto 'tcp'
	option dest_port '443'
	option sticky '1'
	option use_policy 'distributed'

config rule 'v4'
	option family 'ipv4'
	option proto 'all'
	option sticky '0'
	option use_policy 'distributed'
	option dest_ip '0.0.0.0/0'

config rule 'v6'
	option family 'ipv6'
	option proto 'all'
	option dest_ip '::/0'
	option sticky '0'
	option use_policy 'distributed'

also...
as per observation...
if i 'blindly' follow creating mwan3 with default wan, wanb & lan (no custom routes/tables defined)..., i notice that every table defined has a gateway associated with it (i believe it is due to "Use default gateway" option)

i am certain that table '1' & '2' are created by mwan3 (see image below)
and i am more puzzled why after mwan3 is configured, the custom wan/lan almost does not work.

reference image of my ipv4 routes

i'll appreciate any insights/feedback in regards to this.
thanks in advance for your time reading my post.

lleachii · December 4, 2023, 11:22am

One doesn't need to use both route/rules and the PBR app.

You only need one. PBR is an overlying application that merely makes routes and rules from the high-level policies you configure.

You can see this calamity on the command line via:

ip -4 route

and

ip -4 rule

How to fix: You need to choose one or the other (manual routes/rules or PBR) and make your policies accordingly.

ambad4u · December 4, 2023, 1:17pm

hi lleachii,

so you mean if go to pbr "path"..., this is: 'pbr' and/or 'luci-app-pbr' packages respectively correct?

since i somewhat also need something like 'loadbalancing' and/or 'failover', does pbr cater this also?
OR
can mwan3 & pbr work together as well?

i am a bit in the dark now... and still confused.

edit:
when i delete all things in mwan3, my custom wan/lan works.
i'm unsure why when mwan3 is configured, the custom wan/lan does not work anymore... though they have a different table(s) and rule(s) in which i believe should not affect it.

ambad4u · December 6, 2023, 7:31am

i tried also by making mwan3 configured first (i have working mwan3 VM)
'added' table 100 as a custom table.
added route & rule for table 100

config route
        option target '0.0.0.0'
        option netmask '0.0.0.0'
        option gateway '10.110.220.2'
        option table '100'
        option interface 'wanb'

config rule
        option in 'xxx'
        option priority '5'
        option lookup '100'

mwan3 working... but, interface 'xxx' via table "100" does not work
any hints perhaps?

lleachii · December 6, 2023, 10:52am

Yes, as previously mentioned:

I'm not sure why you still wanna attempt configuring both mwan3/PBR and routes/rules. Is there some reason you haven't expressed?

Use one method to make routes and rules, do not combine methods.

The same applies for mwan3.

ambad4u · December 6, 2023, 1:43pm

is pbr == mwan3???
i am not using pbr , nor mentioned it
does mwan3 use pbr under the hood?
my apologies...

lleachii · December 6, 2023, 1:57pm

No; but mwan3 is also an overlying application that merely makes routes and rules from the high-level policies you configure.

ambad4u · December 6, 2023, 2:48pm

okay, i'm somewhat clear on that part...

if i go with mwan3..., how does one translate to make a specific 'lan' interface only travel to a specific 'wan' interface? any links or examples i can base upon?

anyways...
mwan3 rules might be hard to match?

say if i want interface 'xxx' go to 'wanb'...
source address is somewhat fine since we can define it (somewhat easy) (or is it?)
is destination address a wan address?

lleachii · December 6, 2023, 3:05pm

To be clear, are you asking how to:

translate a mwan3 policy into route/rules; or
translate route/rules into a mwan3 policy?

ambad4u · December 6, 2023, 3:22pm

into 'mwan3' (your 2nd bullet/option)

if i go to "manual", i don't know how to do a manual 'loadbalance'/'failover' by routes/rules (+ script i guess)..., but if there are any sample(s) or point me to an info about it, i'll appreciate it.

i thank you for your time helping me.

lleachii · December 6, 2023, 3:23pm

Gotcha - understood.

OK, I'm sure someone else with mwan3 experience can help you with that part.

ambad4u · December 6, 2023, 10:29pm

can we possibly rename the thread/post title then to something like: "mwan3 rule to specific interface"? or something more fit?

lleachii · December 6, 2023, 10:50pm

Title changed.

ambad4u · December 16, 2023, 2:23am

i think i got it...
we need to add a 'policy' & 'rule' in mwan3
the additional 'policy' i created (to_pldt & to_globe)

and the 'rule' i created (to_pldt_only & to_globe_only)
in the 'Source address' it should be the ip/cidr of the "SPECIFIC INTERFACE"
and the 'Policy assigned' should be the 'wan' you want to be used for the route/connection.
'Destination address' is something that i only blindly copied from "default_rule_v4"

some info on "Network" > "Interfaces"
normally created, static ip + dhcp + "Use default gateway" checked

here is my "Interface" & "Member" fields for reference

Interface

Member

i am not sure how to thoroughly test it as my test was only like speedtest and dnsleak.

speedtest tells me the isp and it seems the right ones...
but with dns leak test, it seems to use the 'primary' internet as what is configured via mwan3 (w/c i believe a normal behavior via mwan3 rules)