https://techhub.hpe.com/eginfolib/networking/docs/switches/5500hi/5998-5335_hi-avail_cg/content/377975841.htm
The scenario I was trying to explain.
Each router is master in the local vlan. If one router dies, the other takes over.
1 Like
All the time, they talk about "external networks" and "access internet", and I don't see any connection or config between them, except VRRP. So... something is missing.
You can try and build 4 vms, 2 with openwrt and 2 with linux/windows, put both openwrt on vlan 1 and 2, put one linux/win on vlan 1 and the other on vlan 2. If one openwrt is the gateway for one linux/win, and the other is the gateway for the other linux/win, they wont ping each other. You need some other config to acomplish this.
And there is the fact that english is not my native language 
@luizluca can help? =P
Nothing is missing, it's just not tailor made to your scenario.
In the event that the VRRP instance cannot send anything to an interface where it has backup role, did you try what I told you with the dedicated cross link between the routers?
You can also provide the configurations that you have applied. So far we are talking in theory.
It isn't mine neither, no need to keep repeating that.
1 Like
Sorry, couldn't do tests today
Yes, I have one Vlan, not mentioned in first post, that connects all 3 routers (there is a third openwrt router, responsible for vpns)
Tomorrow I will try to check if that vlan is working
But, why a vlan to connect them?
Oh, wait... a "cross link"? A cable connecting only them?
A "cross link" (?) or a vlan that has only them... what it changes?
They already have 4 vlans (plus both internet vlans) to comunicate... no?
It can be a dedicated link, or a vlan in the trunk. It doesn't matter. You can use the vlan you have to connect the 3 routers. When one router receives a packet destined to the other vlan, for which it is backup, to send it to the other router over that vlan, since it cannot send it directly.
You mean...
Set a static route, on both routers?
Because as they are part of the all vlans, they already route things to them
So force a static route like...
Vlan 1 is 172.16.0.0/16
Vlan 2 is 172.17.0.0/16
Vlan 5 is 192.168.5.0/28 (Router 1 on 192.168.5.1 , Router 2 on 192.168.5.2)
Set a static route on router that is active on vrrp instance 1 .. "172.17.0.0 via 192.168.5.2 dev br-mgmt" and set a static route on router that is active on vrrp instance 2 .. "172.16.0.0 via 192.168.5.1 dev br-mgmt" ?
Essentially yes, but in case the static route doesn't work due to metric preference, I'd try with a set of rules/routes.
But... I have to do this, through keepalived?
Because... if router 2 is powered off, all vlans will use router 1 as gateway and the routes pointing to router 2 will fail ... right?
Ok. I'm at working, so my tests are limited.
Corrected Vlan 3 (first post I said "cameras", but vlan 3 is routers vlan), switch was not using it, so the router were not seeing each other through that vlan.
Using IP 172.168.19.100/16 (r1) and 172.168.19.98/16 (r2)
Reading the "rules/routes" link, I'm cofused about...
- Route doesn't have a gateway
- Rule ... did not understood that. "in" should be the interface where packets are coming in. "src" should be... lan ip address? (like 172.16.0.0/16) ?
What if I used masquerading? 
In case of a router failure, one router will have to handle everything for both apartments. So let's imagine that the router in apartment 2 is faulty, and configure the one in apartment 1 to do everything.
- The switches are configured so that both LANs and both WANs are all present as VLANs on the cable to router 1.
- Router 1 holds the .1 IP on both LANs and (optionally) offers a DHCP and DNS service where x.x.Y.1 is offered as the default route and DNS.
- Router 1 holds public IPs on both WANs. mwan3 is active to load balance and / or failover between the two WANs. So this network can tolerate failure of a WAN, though in a different way than failure of a router.
- Router 1 will NAT / route either LAN to the Internet.
Once that is working then you can set up router 2 to do exactly these same things, but be held in standby doing nothing unless router 1 fails.
This was already done.
What I want is both routers "working"
R1 gateway for apartment 1 (vlan 1)
R2 gateway for apartment 2 (vlan 2)
vlan 1 being able to talk to vlan 2
You would have to change routing with the keepalived events.
You can add one.
Yes, but src is not necessary.
I don't see what difference will it make when it failed already.
What failed? Never tried with NAT
I'm trying the rules you said.
I hope I don't get locked out of routers
Now using "real" info...
Router 1:
/etc/config/network (not whole file)
config interface 'villa'
option type 'bridge'
option ifname 'eth0.1'
option proto 'static'
option ipaddr '192.168.1.98'
option netmask '255.255.255.0'
option igmp_snooping '1'
option mtu '9000'
option dns '192.168.1.126'
config interface 'mgmt'
option ifname 'eth0.3'
option proto 'static'
option netmask '255.255.255.128'
option ipaddr '192.168.3.98'
option mtu '9000'
option type 'bridge'
config route
option metric '50'
option table '100'
option netmask '255.255.255.0'
option target '192.168.2.0'
option gateway '192.168.3.100'
option interface 'villa'
config rule
option in 'villa'
option lookup '100'
Router 2:
/etc/config/network (not whole file)
config interface 'renks'
option type 'bridge'
option ifname 'eth0.2'
option proto 'static'
option ipaddr '192.168.2.100'
option netmask '255.255.255.0'
option mtu '9000'
option igmp_snooping '1'
config interface 'mgmt'
option type 'bridge'
option ifname 'eth0.3'
option proto 'static'
option ipaddr '192.168.3.100'
option netmask '255.255.255.128'
option mtu '9000'
option igmp_snooping '1'
config route
option metric '50'
option table '100'
option netmask '255.255.255.0'
option target '192.168.1.0'
option gateway '192.168.3.98'
option interface 'renks'
config rule
option in 'renks'
option lookup '100'
not working.
I think I will stick to dhcp option 121
Oh, I was directly mentioned (but I still don't know why). Anyway, let's add my two cents.
When you draw a network, it is more important to inform who shares the same broadcast domain (network) with. The physical connections are less important when you have multiple vlans in a single cable.
I'm afraid you are losing too much doing double or even triple NATs. Each modem seems to be doing NAT (or else you'll not be able to connect two routers to each modem). If your ISP is already using CGNAT, it'll add another NAT layer. And, each router is again doing an additional NAT. That's a pretty expensive price to pay in user experience.
You already have some single point of failures like each switch. I believe you are really only trying to workaround a failed ISP connection, isn't it? I guess you still need a HW providing wifi on both sides so a dead router will not be fault-tolerant. Doing VRRP is just adding a bunch of complexity to the table.
If the faster router can handle both throughput, it would be much easier to simply connect both modems to a single router (as you are already doing) and use it as the router for both networks. You could even load balance. The second "router" can be converted into an AP. You could even configure the "modems" to a bridge mode and try to reduce a NAT layer.
If you really need to have each router working even if the trunk is down, you could let each router connect to its ISP and add a second WAN interface to the other router LAN. In that way, you'll only need to use the extra NAT layer when one ISP is down. Let mwan3 monitor wan and switch to wan2 if needed.
If both LANs need to talk to each other, as both routers would be directly connected to each LAN, you just need to make sure traffic between LAN1 and LAN2 does not get masqueraded and the connection is allowed (you can specify that in the firewall). There is no need for static routes. It'll be interesting as each router would only see half of the traffic but it should work. That might mess with the conntrack if you need to use a firewall between those two LANs.
It would be something like this (don't judge my artistic talent):
If you really want to use VRRP, you could (ab)use the wan2 connection for the VRRP. But I still believe that, although funny, it is not worth it. Normally a redundant router should synchronize all its state, including DHCP leases. This does not look like a scenario that requires 99.999% uptime. If one router dies, simply manually configure the other ISP and the LAN to the router that is still working.
If you are not intentionally making that scenario more complex than necessary (for academic reasons), I would suggest to try to simplify it. And I'm not even thinking about IPv6. Then you'll get a much more complex setup! But that seems to be off-topic for now.
3 Likes
I follow you on your site, that you talk about openwrt and saw your posts here, so I know you understand a lot and speak my language. Sorry about that.
Will try to reply your message, so I think you will receive notifications... sorry about that too.
Actually I didn't know what to draw, I just used something that was almost done
About the rest, let me see...
Maybe if I start over, with first question: Why two routers?
Mainly because I have difficulties in learning by reading, I like to learn by doing
And over the past 2 or 2 years and half, I had problems like:
- main router was in the other apartment, I was working on mine, and air conditioning technician shutdown the other apartment power, because he didn't know the correct circuit breaker. The nobreak could handle for sometime, but evently it powered off and I was like "what the hell is happening?" until I could discover the problem
- main router was in my apartment and one of my switches decided to block all the others, cutting internet from BOTH apartment
- main router in other apartment and the dog liked exposed cables...
Second question: Why two vlans (well... more, but mainly these two)? As you said about the drawing... "broadcast domain". For e.g. chromecasts in both apartments, Smart TVs and a multiroom sound system becomes a chaos when people (especially "nieces") start playing things on the wrong device
So...
Yes, my modems at this moment are not at bridge mode. I think I have NAT, but not CGNAT, because I receive public IPs (when in bridge mode).
I already used them with bridge mode and without bridge mode, using one router active AND using two routers active.
I was using them in bridge mode and the routers, acting only as routers without wifi, had keepalived configured. The problem with that is I couldn't manage to do failover right (maybe only when the active router was powered off), because the interface with the cable modem (dhcp) did not go down (ifdown wanc) and the interface with the fiber modem (pppoe) did go down, but it took several minutes to go up in stand by router. That's why I choosed to turn off bridge mode, turned on "zdm" on each modem pointing to a VIP (each modem pointing to a different VIP)
Right now my routers are configure to masq the connection to the modems, but it's just a matter of turning it off if I can make things work well, because if I cannot make things work I will just go back to bridge mode on and try new things to make keepalived work
About wifi... I have 3 APs in one apartment and 2 on the other, so the routers do not need to have wifi on.
The switch is "a single point of failure" as is the power itself, these are two apartment on the same residencial building, but depending on which switch (4 on one apartment, 2 on the other, just to avoid a lot of cables passing through), it takes down only part of the network. In short: yes, I want to workaround a failed ISP connection AND a isolated apartment problem... basically.
There are more complex parts, lol. Like... One internet is 500 Mbps, the other is 300 Mbps and I didn't upgraded to 600 Mbps, first because my connection to the router is 1 Gbps, second because I don't have any idea if this routers can handle 1.1 Gbps bandwidth and... that's another topic.
About that... I know this, from work, as "asymmetric traffic", when a connection goes through one path and comes back through another. The simplest example I have is: 192.168.1.2 asks to connect to 192.168.1.100, which is the IP address of a load balancer for 192.168.1.101 and 102. 192.168.1.101 receives the connection and responds directly to 192.168.1.2, as he is on the same network, which refuses the connection, because it asked to connect to 100. It's possible to make this work, but a little more configuration is needed (on the load balance and/or on the servers behind it)
Yes exactly. This starts out as "answer the question that was asked" but the system the original poster was asking about building really isn't the best approach to the problem.
If I were building this network, the key user-side features would be:
- The networks in the two apartments operate generally independently, but a wired link exists so that direct LAN-LAN traffic for file sharing, gaming, printing etc. is possible.
- Each apartment can automatically switch over to reaching the Internet via the other apartment should their local Internet modem or service fail.
Point 1 (independent operation) suggests that a power cut, disconnected cable, equipment failure etc. in one apartment should only affect that apartment, the other one will still have a working network with Internet. So that means a router in each apartment, routing to its ISP modem. These connections can be direct or with VLANs in a switch (in that apartment).
The inter-apartment link is possible by creating another interface on a router, physically linked it to the other apartment's LAN and giving it an IP on that network. The obvious way to run the physical link is to have two VLANs on the inter-apartment cable, one is each apartment's LAN. To do this symmetrically one would have to be careful not to create a loop route.
Now on to point 2. The usual way to do this is have a single router holding both WANs, a so called edge router. But this introduces a single point of failure where no one will have Internet if that router fails. Another way to do it would be to give the two independent routers a backup WAN via the other apartment's router. The router in the apartment with a failed modem would change its default route from itself to the other apartment's router. (note this route is via the other apartment's LAN, which already exists in the router, so there is no need for additional VLANs to support the WAN sharing) This would be controlled by a local instance of mwan3. In this case failure of one router will only affect that apartment.
Ok, looks like the common sense is to use the drawing from luiz, where each router has an interface on the other network and use it as backup, not load balancing wans, as all replies points to that. And if a router die, I "just have to reconfigure all", (or if I want to upgrade to Openwrt 21.02 and discover that it's better not to, as the internet speed will not pass 150 Mbps, so rollback to 19.07) so that apartment comes back online.. Sorry Trendy if I took too much of your time. Thanks trendy, luizluca and mk24.
1 Like
That diagram could do the trick, just advertise the network of the other apartment with a static route on each router and exclude the subnet from nat when it goes out wan2.