Basic (I think) routing problem with two routers

In my network I have two routers, both running LEDE stable. One is the main router, and the other is a secondary router, which has a separate interface to a few different (private) networks.

Only the main router provides DHCP and DNS to the hosts in my network. The hosts needs to be able to reach certain networks which sits behind the secondary router, but for regular internet traffic, the main router has the WAN connection, and is the default gateway.

What I have done to make this work, is to configure static routes to these secondary networks on both routers. As an example, the network 172.16.0.0/16 is reachable from behind the secondary router, so to make this work, on the main router I have configured a static route to this network with the gateway set to the secondary router LAN IP address.

On the secondary router I have the same static route configured, the difference being that on this router the route is set up to use the default gateway on the secondary interface where this network can be reached.

This setup works just fine, but it also means I have to reconfigure both routers every time I need to make changes to this routing. Not a big issue, but slightly cumbersome, and I am convinced there is an easier way. Ideally I'd like to just make the necessary changes on one router, I just don't know how.

a diagram/picture of the network is alway helpful.

Not sure if that would help, but I made a quick drawing:

The main router gets its WAN IP and gateway (and DNS) from the ISP over DHCP, and acts as the default gateway (and DHCP and DNS server) for the LAN.

The secondary router also gets it IP and gateway on the EXTIF by DHCP, in this case it has 192.168.10.10 as it's own IP, and the gateway is 192.168.10.1. Traffic to 172.16.0.0/16 is reachable via the gateway at 192.168.10.1.

When a host in my LAN needs to talk to for example 172.16.17.18, it will send the traffic towards the default gateway of 192.168.1.1. The router has a static route to this network via 192.168.1.2, so it forwards the traffic and also sends a ICMP redirect message to the host so that the host can continue sending directly to 192.168.1.2.

The secondary router also knows what to do with the traffic, because it has a static route to the requested network via the gateway 192.168.10.1.

All of this works fine, but it requires me to add the static routes to both routers, just with different gateways and interfaces. Is there an easier method, or is this in fact the easiest?

an idea, the main router could deploy the routing information via dhcp pakets. So the clients use the secondary router as next hop for the networks 172.16.0.0 and so on.

I don't think you can use that to deploy routing info for networks, only for hosts. And I'm not sure there's any guarantee that all hosts (I have a mix of hosts) actually reads and acts on such DHCP options.

That is one way to solve the problem

Another way it to add a vlan and put other router on that vlan and enable vlan on all devices, this is even harder.
Another way is to put another router between two routers and hosts... and put the two main gateways there and give to hosts a third network address space in order to work (this new router will redirect packet to corresponding router.
Another way is to make just one of the two routers dhcp server and try uci add_list dhcp.lan.dhcp_option="3,192.168.9.1" # another gateway but I don't know if you could specify two, but probably you could create on one router two networks, to be able to give the 2 addresses. I think that Windows, Linux , and probably OSX could have 2 different addreses on one same ethernet interface, but I don't know if it will work using DHCP.

The easiest and natural way it to put a Router in the middle. That is the work for a Router, route traffic to correct location.

Adding another router is not an option, really. I still don't get the DHCP suggestion. How would the clients know which gateway to use, as there's no specification which networks to reach on a specific gateway?

Sounds like I'll just have to stick with the static routes, after all.

Mh, I mean you should google "multi gateway DHCP" "multiple gateway dnsmasq" I don't know if it is possible but there are multiple people that was trying to do that...

You should use ONE of your two Routers to DHCP, just one... and set up additional vlan, and use one physical port of the Router1 to get connection to Router2 (Router1 will route traffic to wan or to the new vlan (Router2)...
Why you can't do this? you want some kind of redundancy in some way of two different service?

For example, if people lost connection to internet must be able still to connect to 172.16.0.0/16 network and viceversa, if they lost access to that network need to be able to use internet ?

I insist, the more natural and proper way is to do Routing in a Router (using their routing table), and avoid configuring each device nor do other experiments.

At least from your drawing, the "secondary" router is going to need a route to 172.16.0.0/16 as it isn't a directly connected network.

You could do something very ugly like having all hosts in 192.168.1.0/24 have a static route for 172.16.0.0/16 via 192.168.1.2, potentially assigning that through DHCP, if your clients support it.

Managing the a relatively simple, seemingly static topology over two routers doesn't seem worth any more complexity.

@braian87b: I only have one of the routers do DHCP and DNS. This has nothing to do with redundancy, the secondary router is there to provide multicast services, due to the main router R7800 not being able to cope with the traffic on it's own. That's also why I don't want to use a VLANs towards the secondary router, because that would send the traffic through that router. Which is what I wanted to avoid in the first place.

@jeff: That's correct. The secondary router has a gateway (192.168.10.1) which it can use to reach the other network. There are more than one network, though. I just used one of them as an illustration.

Relying on clients to support potentially ugly solutions is most likely not a good idea, so I think I'll just have to stick to the static routes option that I already have. There are no issues with it, except it feels a bit dumb to do the same setup on both routers, just with different interfaces and gateways.

I spoke too soon when I said the static routing works well.
There is an issue, but I don't fully understand it. Here's the issue:

The clients within the LAN (named hosts in the drawing) that needs to reach the servers in the 172.16.0.0/16 are mostly STBs for TV (Android-based). They have the default route to the main router, so when they try sending data to a server in the 172-network, the main router forwards the first packet and then sends an ICMP redirect for host back to the client. The client will then (usually) continue by sending directly to the redirected gateway. However, this doesn't always work, and I don't understand why. The clients is perhaps not caching the redirected routes, but it seems to me that the main router does not always send the redirects to the client either. I'm not sure why it doesn't, but the result is that the client works a bit erratically. Sometimes it ends up in some state where it hasn't been able to properly communicate with a server, and therefore doesn't work properly.

I don't understand why the main router stops sending redirects, but it may be that it presumes that clients should cache previous redirects (on a per server basis), and therefore just drops further communication that is directed to it?

To alleviate this issue, I did look more into @braian87b's suggestion regarding DHCP, and what I did was to assign the secondary router as the main gateway (through custom DHCP options) for these clients (and these clients only). Since these clients does almost all of their communication towards the 172-network, this avoids most of the redirects. However, when they need to communicate towards the public internet, they will receive a redirect from the secondary router instead.

However, I'm still not sure I'm fully in the clear. If my theory about the router stopping to send redirects is correct, then there may still be an issue for the internet-bound traffic from these clients.

So if anyone can shed more light on this issue, please do.

Ahh. I should have thought of this before writing the previous post. There are some settings for redirects that may affect me here. These are the defaults on my router(s)

net.ipv4.route.redirect_load = 2
net.ipv4.route.redirect_number = 9
net.ipv4.route.redirect_silence = 2048

If the units for redirect_silence is in seconds, that means I'll have a 34 minute timeout before the router again sends redirects after the limit has been reached. However, I haven't yet found an explantion for the other two parameters. Are they per host or per route? For the redirect_load parameter I found this explanation:

Factors which determine if more ICMP redirects should be sent to a specific host. No redirects will be sent once the load limit or the maximum number of redirects has been reached.

However, nothing about what kind of factors they are.

I tried digging some more into this, and it seems the redirect_interval is in jiffies, not seconds. The default HZ is 100 (as far as I can tell), so the silence value in seconds should then be just above 20 seconds. And that sounds a lot more reasonable.

And the load limit is the base value for the exponentially increasing delay between redirects. It starts out at the base value, and for each redirect (for the same peer) the delay value is doubled (by a binary left-shift). With a base value of 2, the first delay is then 2 jiffies, then 4, 8, 16, etc. When the number of redirects hits the value redirect_number, the redirect_silence timeout is started.

This explains my issue, and it also tells me that given the very low volume of traffic that will need redirection when the STBs use the secondary router as their default route, the issue is most likely solved. Even if relying on redirects is usually considered a bad idea.

If you try to bath yourself needing mid temperature water but using two hoses, one with very cold water and one with very hot water...

Do you understand now why this is must not be done like this?, you should put both hoses on a water pipe mixer with two taps one for cold and one for hot and delegate the job to the mixer.

The problem is that I need the traffic to NOT pass through my main router when it is destined (or rather, coming) for the 172-network.

This is the whole point of the secondary router. I can make the same setup work by giving my main router the upstream VLAN that allows traffic to this network (and just remove the secondary router), but then I get dropouts on TV when the internet connection is fully utilized (I have thoroughly tested this, but avoiding the dropouts was impossible). And that's not acceptable, even if it is the most sound alternative for the network architecture as such.

Having to add even one more router (or water pipe mixer as you called it :slight_smile: ) just cannot be right or necessary to solve this proper.