Yet Another Dual/Multi ISP Question

My apologies, my bad, did not clearly see until now.
below is the only 'code' placed on: "/etc/config/network"

config route
	option target '0.0.0.0'
	option netmask '0.0.0.0'
	option gateway '10.32.64.222'	
	option table '2'
	option interface 'globe'

config rule
	option src '10.123.123.0/24'
	option dest '0.0.0.0/0'
	option priority '2'
	option lookup '2'

the example was actually 2 rules and 2 route(s), I removed the others so to simplify troubleshooting, as if I can make this work, I can replicate this for other similar config (i think)

I used both, but I'll lean now to: "/etc/init.d/network reload".

Feel free to review my posts and provide updated configs. It seems like you're missing vital notes and inquires from my posts.

Also you can test your traffic by using the commamd:

ip route get xxx.xxx.xxx.xxx from <ip_of_interface>

You can do this in both directions.

Thank You, One Moment..., I'll post & explain it here...

Edit:

does: "xxx.xxx.xxx.xxx" is any virtually external ip???

Edit2:
I'll state again some info here...

  • I have plenty of interfaces, funny enough, i don't even use the default 'lan' network!
  • eth3 & eth4 mentioned earlier in the thread is to simplify the issue to those interface/networks.
  • I removed table '1' as it seems redundant, table '2' is enough for this problem (unless otherwise removing table '1' would equate an unusable OpenWRT device?)
  • interfaces/networks that used table 'main' is working fine. I mention before that I don't have internet, but just realized that the only interface that did not have internet was just the ones " e1v4 & xTEMP" (the ones that we are testing here)

contents of: "/etc/config/network"

root@OpenWrt-x86:~# cat /etc/config/network

config interface 'loopback'
        option device 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'fd5b:f0be:f6ba::/48'
        option packet_steering '1'

config interface 'wan6'
        option device 'eth1'
        option proto 'dhcpv6'

config device
        option type 'bridge'
        option name 'br-mgmt'
        list ports 'eth0'
        option ipv6 '0'

config interface 'mgmt'
        option proto 'static'
        option device 'br-mgmt'
        option ipaddr '10.10.10.254'
        option netmask '255.255.255.0'
        option defaultroute '0'
        option delegate '0'
        option broadcast '10.10.10.255'

config device
        option type 'bridge'
        option name 'br-lan'
        list ports 'eth10'

config interface 'lan'
        option proto 'static'
        option device 'br-lan'
        option ipaddr '10.123.234.254'
        option netmask '255.255.255.0'
        option broadcast '10.0.0.255'

config device
        option type 'bridge'
        option name 'br-pve'
        list ports 'eth5'
        option ipv6 '0'

config device
        option type 'bridge'
        option name 'br-wan'
        list ports 'eth1'
        list ports 'eth8'
        option macaddr '2c:53:4a:01:a0:f4'

config interface 'wan'
        option proto 'dhcp'
        option device 'br-wan'
        option peerdns '0'

config device
        option type 'bridge'
        option name 'br-DNS'
        list ports 'eth9'
        option ipv6 '0'

config interface 'DNS'
        option proto 'static'
        option device 'br-DNS'
        option ipaddr '172.27.72.254'
        option netmask '255.255.255.248'
        option broadcast '172.27.72.255'
        option delegate '0'

config interface 'pve'
        option proto 'static'
        option device 'br-pve'
        option ipaddr '192.168.134.254'
        option netmask '255.255.255.252'
        option broadcast '192.168.134.255'

config device
        option type 'bridge'
        option name 'br-e1v4'
        list ports 'eth6'
        option ipv6 '0'

config interface 'e1v4'
        option proto 'static'
        option device 'br-e1v4'
        option ipaddr '172.24.48.254'
        option netmask '255.255.255.0'
        option broadcast '172.24.48.255'

config device
        option type 'bridge'
        option name 'br-e2v4'
        list ports 'eth4'

config interface 'e2v4'
        option proto 'static'
        option device 'br-e2v4'
        option ipaddr '172.16.64.254'
        option netmask '255.255.255.248'
        option broadcast '172.16.64.255'
        option delegate '0'

config device
        option type 'bridge'
        option name 'br-e0v4'
        list ports 'eth3'
        list ports 'eth11'

config interface 'e0v4'
        option proto 'static'
        option device 'br-e0v4'
        option ipaddr '10.20.30.254'
        option netmask '255.255.255.0'
        option broadcast '10.20.30.255'
        option delegate '0'
        option defaultroute '0'

config device
        option type '8021q'
        option ifname 'br-e0v4'
        option vid '101'
        option name 'br-e0v4.101'
        option ipv6 '0'

config interface 'e0v4_101'
        option proto 'static'
        option device 'br-e0v4.101'
        option ipaddr '172.27.101.254'
        option netmask '255.255.255.0'
        option broadcast '172.27.101.255'

config device
        option type '8021q'
        option ifname 'br-e0v4'
        option vid '404'
        option name 'br-e0v4.404'
        option ipv6 '0'

config interface 'e0v4_404'
        option proto 'static'
        option device 'br-e0v4.404'
        option ipaddr '172.27.44.254'
        option netmask '255.255.255.0'
        option broadcast '172.27.44.255'
        option delegate '0'

config interface 'modemPLDT'
        option proto 'static'
        option device '@wan'
        option ipaddr '10.54.154.54'
        option netmask '255.255.255.0'

config device
        option type 'bridge'
        option name 'br-wan1'
        list ports 'eth2'

config device
        option type '8021q'
        option ifname 'br-e0v4'
        option vid '202'
        option name 'br-e0v4.202'
        option ipv6 '0'

config interface 'e0v4_202'
        option proto 'static'
        option device 'br-e0v4.202'
        option ipaddr '10.1.1.254'
        option netmask '255.255.255.0'
        option broadcast '10.1.1.255'

config interface 'globe'
        option proto 'static'
        option device 'br-wan1'
        option ipaddr '10.32.64.222'
        option netmask '255.255.255.0'
        option gateway '10.32.64.254'
        option broadcast '10.32.64.255'
        option defaultroute '0'

config device
        option type 'bridge'
        option name 'br-temp'
        list ports 'oldeth10'

config interface 'xTEMP'
        option proto 'static'
        option device 'br-temp'
        option ipaddr '10.123.123.254'
        option netmask '255.255.255.0'
        option broadcast '10.123.123.255'
        option delegate '0'
        option defaultroute '0'

config route
        option target '0.0.0.0'
        option netmask '0.0.0.0'
        option gateway '10.32.64.222'
        option table '2'
        option interface 'globe'

config rule
        option src '10.123.123.0/24'
        option dest '0.0.0.0/0'
        option priority '2'
        option lookup '2'

root@OpenWrt-x86:~#

contents of: "ip -4 route"

root@OpenWrt-x86:~# ip -4 route
default via 49.145.112.1 dev br-wan  src 49.145.116.152
10.1.1.0/24 dev br-e0v4.202 scope link  src 10.1.1.254
10.10.10.0/24 dev br-mgmt scope link  src 10.10.10.254
10.20.30.0/24 dev br-e0v4 scope link  src 10.20.30.254
10.32.64.0/24 dev br-wan1 scope link  src 10.32.64.222
10.54.154.0/24 dev br-wan scope link  src 10.54.154.54
10.123.123.0/24 dev br-temp scope link  src 10.123.123.254
10.123.234.0/24 dev br-lan scope link  src 10.123.234.254
49.145.112.0/20 dev br-wan scope link  src 49.145.116.152
172.16.64.248/29 dev br-e2v4 scope link  src 172.16.64.254
172.24.48.0/24 dev br-e1v4 scope link  src 172.24.48.254
172.27.44.0/24 dev br-e0v4.404 scope link  src 172.27.44.254
172.27.72.248/29 dev br-DNS scope link  src 172.27.72.254
172.27.101.0/24 dev br-e0v4.101 scope link  src 172.27.101.254
192.168.134.252/30 dev br-pve scope link  src 192.168.134.254
root@OpenWrt-x86:~#

contents of: "ip -4 rule"

root@OpenWrt-x86:~# ip -4 rule
0:      from all lookup local
2:      from 10.123.123.0/24 lookup 2
32766:  from all lookup main
32767:  from all lookup default
90026:  from all iif lo lookup 2
root@OpenWrt-x86:~#

other info(s): from OpenWRT

root@OpenWrt-x86:~# ip route get 139.59.209.225 from 10.123.123.254
139.59.209.225 from 10.123.123.254 dev br-wan1
root@OpenWrt-x86:~# ip route get 139.59.209.225 from 10.1.1.254
139.59.209.225 from 10.1.1.254 via 49.145.112.1 dev br-wan
root@OpenWrt-x86:~# ip route get 139.59.209.225 from 10.20.30.254
139.59.209.225 from 10.20.30.254 via 49.145.112.1 dev br-wan

from inside alpine linux(VM) using 'xTEMP' interface/network


Hope the info(s) suffice...

Many Thanks Once Again!

1 Like

Hi bluewavenet!

I do get your point...
My main reason is for learning..., & you are correct with having individual OpenWRT devices would make it simplier... (& I don't have to pester OpenWRT folks :slight_smile: )
I'm unsure about 'cost' effectiveness on what you mentioned, but that is subjective i guess..., ultimately, it will just boil down to what your system will serve and go from there.

Thank you for your inputs!

The response shoulda been more like:

139.59.209.225 from 10.123.123.254 via 10.32.64.254 dev br-wan1 table 2 uid 0

So something's odd.

  • :bulb: Why don't these match?
  • :confused: How is that also your interface IP
  • :spiral_notepad: You need to put the correct gateway in both locations
  • Also, I assume you enabled masquerade on the firewall zone containing br-wan1.

FYI - I don't see this config and I donno why it's there.

Not sure why you showed us that one, but looks ok - you got an IP via DHCP (assuming) it seems.

Thank You lleachii,

You have given me many good hints and info!
I'll review these, it may take me a bit & hopefully digest & ultimately solve this puzzle...

i believe i solved it (i think...)

below is the info for the 2nd internet (the 2nd wan interface)

config interface 'globe'
	option proto 'dhcp'
	option device 'br-wan1'
	option defaultroute '0'
	option ip4table '2'

below is the interface to be used for the 2nd internet connection

config interface 'xTEMP'
	option proto 'static'
	option device 'br-temp'
	option ipaddr '10.123.123.254'
	option netmask '255.255.255.0'
	option broadcast '10.123.123.255'
	option defaultroute '0'
	option ip4table '2'

the 'route'

config route 'alt_route'
	option target '0.0.0.0'
	option netmask '0.0.0.0'
	option gateway '10.32.64.254'
	option table '2'
	option interface 'globe'

the "new rule"

config rule 'xTEMP_globe'
	option in 'xTEMP'
	option priority '2'
	option lookup '2'

screenshot for the 'ipv4 routes'

screenshot for 'ipv4 rules' (i don't still understand... for now)

below screenshot is for the firewall:

i am unsure why the rule below was not working vs the above 'new rule'

config rule
        option src '10.123.123.0/24'
        option dest '0.0.0.0/0'
        option priority '2'
        option lookup '2'

there is something that is bothering me...
when i go to a speedtest, it is indeed telling me the right isp (globe)
openwrt-005

but when I go to DNS leak test, it is still correct:

however, the 'result' or the DNS it is using seems to be the other ISP that i have.

as per my understanding, since both interfaces ('globe' & 'xTEMP') are using a different 'routing table' (table: '2'), it should use the DNS of 'globe' since they are contained within 'table 2'??? or should this question be on another thread?

edit:
fixed my DNS issue by using 'option: 6' via DHCP (i.e. in my case: 6, 10.32.64.254)

This thread is very interesting to me.
I'm just wondering why it's so hard to configure instead of using OpenWRT as the lxc container in proxmox.
This offers very high performance at the kernel level and ease of configuration.
We just assign the interface to the container. I have no performance degradation caused by kvm virtualization.

Hi DienoX,

while there might be a good 'performance' via "lxc"..., what is keeping me from using it is that (and what i've heard)... if a "guest VM" produces 'panics'..., it will catastrophically affect the host as well!!!

'lxc' and similar things (like docker..., i guess) is best suited for Web Servers &/or Web Application, etc... but for OpenWRT..., I go straight FULL VM + ethernet passthru (w/c is what I am using).

i'm not saying that you are wrong..., there is a very good use case for using maybe OpenWRT in 'lxc' mode... like a low power x86 but have enough headroom to fit other extra/external services..., but ultimately, you should know what you are doing :slight_smile:

I understand your worries. There are several ways to remove them. Of course, everything has its compromise. I'm just wondering about the kernel panic issue. I worked with several organizations that used LTS kernels on their servers. For the past 5 years, I don't remember anyone mentioning the kernel panic issue on the lts kernel to me.

Have any of you come across an LTS kernel lady on production servers in the last 5 years? I ask because everyone has a different work environment.
*I mean kernel panic caused by kernel, not hardware failure.

i'll re-structure here or sort of summarize what i was trying to achieve & the solution i got from this wonderful community!!!

i have a working OpenWRT (virtual machine) w/ a typical & very common 1 'wan' interface & i want to 'ADD' another "wan" interface (a 2nd isp) using only the builtin things like: 'rule, routes & tables' & w/out utilizing 'MWAN3'. Please note that this example does not use any "load balancing" or "failover" techniques. (as i wanted to learn how to 'manually' add things before diving-in to some other advanced techniques like: MWAN3, etc...)

For visualization, there are 2 separate 'OpenWRT network/system' of w/c have 1 wan & 1 lan interface(s), & as you can see below, they both work independently.
openwrt-009

What i want to achieve is to have the same behavior as indicated above but instead of having 2 separate OpenWRT instance or box, only 1 is used.
(note: for simplicity..., lan-A cannot talk to lan-B & viceVERSA)
Also..., we will utilize 4 interfaces:
eth1 = wan-A (the default: wan)
eth3 = lan-A (the default: lan)
eth2 = wan-B (the 2nd isp, we'll name it: globe)
eth4 = lan-B (we'll name it: xTEMP)
The visuals should now look like below (please mind map accordingly :slight_smile: ).
openwrt-011

As you can see from the above image, there now includes from the dotted squares named: 'table: main' & 'table: 2'. For now, we'll settle with with these, you can rename 'table: 2' to any number or even a name! but please do diligent research before doing the change.

As mentioned, wan & lan are already working & we need to add the 2nd isp to our OpenWRT system.

Create a new 'interface' called "globe", this will serve as our 2nd isp & make sure that under "Advanced Settings":

  • "Use default gateway" is unchecked.
  • "Use DNS servers advertised by peer" is unchecked (optional)
  • "Override IPv4 routing table" is set to: "2"

If the Option "Use default gateway" is 'checked'..., there is a very high chance that you will lose internet connection as there will be 2 gateways now and will clash w/ each other.

The Option "Use DNS servers advertised by peer" is optional as one might like using any DNS they choose..., or perhaps they have their own DNS Servers like pi-hole/adguard. If you don't know this or don't care, leave this checked.

note: usual protocol for this is: "dhcp client", if you have like PPPoE or something else..., change accordingly...

Config under: "/etc/config/network" should now look like this

config interface 'globe'
	option proto 'dhcp'
	option device 'eth2'
	option defaultroute '0'
	option ip4table '2'
	option peerdns '0' #<- this is optional

Save, then Save & Apply...

Create another interface called: "xTEMP", this is "lan-B", this should almost mimic 'lan' (or lan-A), change: ip address, netmask, broadcast to your needs and also go in "Advanced Settings":

  • "Use default gateway" is unchecked.
  • "Override IPv4 routing table" is set to: "2"

Config under: "/etc/config/network" should now look like this

config interface 'xTEMP'
	option proto 'static'
	option device 'eth4'
	option ipaddr '10.123.123.254'
	option netmask '255.255.255.0'
	option broadcast '10.123.123.255'
	option defaultroute '0'
	option ip4table '2'

Save, then Save & Apply...

Then add the 'route' & 'rule' in: "/etc/config/network"

Note: "gateway" below is the 'gateway' of 'globe'

Note: setting the 'gateway' here is optional..., however, if not set, it seems to use the DNS of "isp-A"!!! (might not what you want!!!)

config route
	option target '0.0.0.0'
	option netmask '0.0.0.0'
	option gateway '10.32.64.254'
	option table '2'
	option interface 'globe'

config rule
	option in 'xTEMP'
	option priority '2'
	option lookup '2'

Save, then Save & Apply...

Create Firewall zones accordingly..., see reference below:
eth2 is under zWAN1
eth4 is under zTEMP
enable masquerade on zWAN1 (or mimic what you find on 'wan')

Save, then Save & Apply...

After 'Save & Apply', go to: Status -> Routing, you should see something similar below:

Connect your desired devices on your "2nd lan" (lan-B), you should have internet (using isp-B) by then.

i would like to thank the OpenWRT Community for helping me! & i would like to thank specially: ncompact & lleachii for their patience on guiding me on my predicament.

PS: can a solution be awarded more than once in a single thread/topic?

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.