Yet Another Dual/Multi ISP Question

Installing and Using OpenWrt
1

Greetings to All!,

What I want/need to Achieve:
One OpenWRT 'box/instance' serving different ISP(s) [without using mwan3] instead of using individual OpenWRT boxes for each ISP.

Please take note: no loadbalancing, no failover (& NOT utilizing mwan3) just plain old manual routing/table

Here is a sort of visuals for "before"
openwrt-Page-1.drawio

Here is what I am after:
openwrt-Page-2.drawio

What I Have:
2 ISP(s), let's name them for the moment isp-A & isp-B
I am using the x86 variant of OpenWrt (23.05.0-rc2) as a VM inside Proxmox
All ethernet ports used for OpenWRT are all 'pcie passthrough' (working & no issues) [all intel btw...]
The "Interface Names" I use is pretty bad ATM, will revise sometime in the future.
I have 2 pi-hole VMs (non-docker/container) living on its own 'vmbrX' network
My OpenWRT is also using Unbound (via serial mode..., I think ^_^)
My 'lan/local network' is served with vlans as well... (served by another OpenWRT device in AP mode)

isp-A:
Device: eth2
Bridge: br-wan1
Interface: unconfigured
-has static ip via pppoe
-not used inside OpenWRT ATM

isp-B:
Device: eth1
Bridge: br-wan
Interface: wan
-currently used by OpenWRT (in bridged mode)

The Problem:
Everytime I add/create an interface for isp-A (eth2), internet does not work anymore.

Some Thoughts:
I still consider myself a noob specially with network/firewall but eager to learn.
As per research, I believe I need a custom route/table in order to make it work, but I am not sure where to begin as nftables/iptables examples confuses me.

I do not want mwan3 for now as I would like to learn more about how to manually add/create route/table as I think I need to better understand it as I will be doing some 'self-host' things in the future (hopefully).

I'll Appreciate any Insights/Directions for this.
Thank You for your Time Reading my Predicament.

PS: I am also very bad at doing/creating titles for posts.

Best Regards,
ambad4u

2

The keyword for doing that would be mwan3 (or doing what mwan3 does manually, which is not going to be a walk in the park) - so it might be relevant why you don't want mwan3.

That's to be expected, as the default routes are stomping over each other.

and

are kind of at odds to another, even if you do "want to learn", mwan3 could teach you what is needed.

Even if you'd know what you're doing, dual-wan of non-cooperating ISP connections is not trivial and rather limited in the sense of what you can achieve.

3

Hi slh,

Thank You for your reply!
Well..., I can really blindly install mwan3 and be done with it, as a matter of fact, I tried it once but I don't like how it behaves and it confuses me.

Also, I am aware before posting about adding "eth2" not having internet.

Anyways, with my current configuration and for example if i have an interface "lan44"... would creating a route/table and linking 'eth2' + 'lan44' give me an internet connection? (provided I set also its firewall things), or perhaps I think of it in the wrong way?

4

I assume you need to assign metrics to your wan interfaces (at least it worked for me)

immagine

you will use internet of the lowest metric, and you can vary it with a script.

or

1 Like
5

Hi ncompact,

Thanks for replying, but, I don't really need any kind of loadbalancing NOR failover. I can spun another OpenWRT VM instance and assign the other ISP to it and call it a day..., but as mentioned, I would like to combine both isp/wan inputs in one OpenWRT instance and its traffic should be depending on which table/route/interface I assign it.

I'll read the link you have provided...
I'll try to put some picture and edit/update my post.

6

if you need to combine traffic from multiple wans you will need the support of your internet providers.

former:
isp1 10Mbit
isp2 20Mbit
isp3 30Mbit

if you want a 60Mbit line you have to ask the internet providers.

if you want a failover that automatically chooses the 10 or 20 or 30 Mbit mwan3 line

or
chosen by you instant by instant script

1 Like
7

That's load balancing.

mwan3 or policy based routing (which can also be done with mwan3 or the PBR package) would work - you can also make IP Routes and IP Rules without additional software installation. Also BTW, you never described the output behavior.

Excellent advice.

1 Like
8

Hi lleachii,

The 'combine' I meant is something else, my apologies...
This "combine" thing is something like adding another 'wan' in my OpenWRT box but its traffic/route is directed to something else and does not have of any kind of load balancing.

9

OK, it's cool you want to use different terms or invent new ones - just know it will confuse others.

Since you feel that your use case should employ another term/definition, you may need to fully describe the desired behavior of your network and both WAN connections.

Such as describing this in detail.

e.g. - What does "traffic/route is directed to something else" mean?

EDIT:

Also, since you don't wish to load balance, I assume you desire one of the WAN connections to remain idle/unused at all times, correct?

10

both wan connections should be active..., basically, as you can see from the image/picture, its like combining 2 separate systems into 1.

I have updated the first post with some details.
I'll change/update my terms used here as much as I can, sorry for the fuzz/confusion.

1 Like
11

This looks like a case for policy based routing together with two DHCP server instances to assign IPs from different pools to the two separate internal networks. You need to decide wether devices behind eth3 should be able to talk to those behind eth4 and vice versa or not.

1 Like
12

Hi moeller0,

Hmmm, good one..., though it will be mostly just be like: 'talk to your group exclusively...', if I need to "cross-talk", I'll do my research.

Thanks!

13

what destination will a data packet starting from PCA have?
isp-A or isp-B

what destination will a data packet starting from PCB have?
isp-A or isp-B

ex:
PCA: ping -c 1 8.8.8.8
and
PCB: ping -c 1 8.8.8.8

immagine

1 Like
14

Also, are the WANs DHCP or statically assigned?

15

ideally...,
pc-A should go to isp-A
and
pc-B should go to isp-B

isp-A is static ip (pppoe)
isp-B is in bridged mode

16

ideally...,
pc-A should go to isp-A and pc-B should go to isp-B

ideally for you it means that sometimes it can vary

or will it really be that for me it always means ?

should the PCA be able to communicate with the PCB or not (they are separate networks) ?

17

I'll Appreciate any Insights/Directions for this.

Rewinding to the beginning of the thread, the "before" and "after" images look very nice but it can essentially be very much simplified by just getting two OpenWrt supported routers. This will free up the Proxmox box for something else while at the same time removing its single point of failure, removing the complexity of VMs, reducing power consumption etc etc. This will almost certainly be lower cost to implement and a common management port would be easy to implement in a similar way.
If you are in a "real" corporate environment and have a blade server or two in your server room, then adding a couple of blades can be very cost effective.
Just saying :wink:

2 Likes
18

For the sake of simplicity and to not complicate things.
lets put it that pc-A (or whatever network/computers placed on "eth3") does not need to communicate with pc-B.

19

ok and the other question ?

the connections are always
pc a -> isp a
pc b -> isp b

1 Like
20

correct.
but please note, as indicated on the image/picture, it is designated as 'devices'..., so I may add another device for 'network-A', so, whichever devices that are connected to eth3 can go to the internet via isp-A.

I'll try to experiment again this evening, I've added table 100 'custom' but cant get the gateway correct..., I'll update this evening (my time +8).
Thanks once again!