HH5a red WAN port confusion

hedzwillroll · November 2, 2024, 2:26pm

Hi,

I am currently using a BT Home Hub 5a (HH5a) with OpenWrt 23.05.
As I live in the UK, it's currently configured to use dsl0.101 in the wan interface and PPPoE as my ISP supports that method of authentication for my FTTC connection.

I would like to add a second WAN connection via the physical red port on the back of the router (it has 4 yellow 'LAN' ports and 1 red 'WAN' port, although if I understand correctly, OpenWrt doesn't make the distinction).

In Network>Devices, there is device called eth0 which shares the same MAC address as a device called wan, and a tunnel device called pppoe-wan. In Network>Interfaces, there is an interface called wan which has its device as pppoe-wan and an interface called wan6 which has its device as an alias of @wan. There is also Network>DSL which pertains to signalling for xDSL.

Is this 'red port' missing in OpenWrt? I initially though that eth0 would be this elusive 5th LAN port, but I now think that's not correct.

There does appear to be somewhat relevant information on the ToH page for this device, some of which involves editing /etc/board.json, though this information is only cosmetic as far as I can tell. With regards to the rest of the information on that page, it seems to be outdated and I didn't find a solution there (or one that I could understand anyway )..

Can anyone explain how I can add an upstream router to the 'red port' so OpenWrt will get a DHCP address from a LAN port on the upstream router (I will put OpenWrt in a DMZ on the upstream router)?

Thanks,
Stephen

EDIT: So originally, I created an Interface called ethwan and gave it the device eth0 with protocol DHCP. This did not work.
After posting this topic, I was digging around and noticed that eth0 is referred to as Ethernet switch in the device list and lan1, lan2, lan3 and lan4 are referred to as Switch ports. Then I noticed another Switch port called wan, so I created ethwan again, using device wan and protocol DHCP. This worked and ethwan got an IP address from the upstream router. However, it appers that this altered the default gateway and, as the upstream router doesn't have a public IP address yet (activation is due Monday), OpenWrt and my LAN had no internet access. Possibly due to ignorance, I had to remove ethwan and reboot to recover that.

So, I see I can get an IP from DHCP on the upstream router and, presumably, if that upstream router had internet access, eveything might have been OK and I could set to work understanding mwan3 for fail-over scenarios?

The confusion for me now is, there is a device called wan and also an interface called wan; presumably the wan interface name is arbitrary?

Also, what happened to the defaly gateway? (the upstream router is dishing out addresses on 192.168.1.0/24)

Closer to my goal, but still a knowledge gap.
Thanks for reading and any input if you can see my blindspot

psherman · November 2, 2024, 3:14pm

Without seeing the config and the resulting routes, we can only guess. However, that said -- you didn't mention installing mwan3, so I'm going to wager that the issue is related to the fact that you may not have installed and configured this package. This is how you setup the multiple WAN connections for load balancing, fail over, or other situations.

https://openwrt.org/docs/guide-user/network/wan/multiwan/mwan3

hedzwillroll · November 2, 2024, 4:13pm

Hi Peter,

Thanks for the reply.
I had not yet installed mwan3, you're correct. I was hoping to configure both WAN connections first and then disable one or the other to suit manually, before tackling mwan3.

That said, I have just installed luci-app-mwan3 (I had to do this from the CLI for some reason as it failed when trying with LUCI). Looking around in MultiWAN Manager, I can see that the default configuration suggests a second WAN called wanb.
I have set wan to have a metric of 1 (wan6 is not used, so I haven't set a metric for that). Presumably, I now configure wanb as my example ethwan above? i.e. add a new interface called wanb with protocol dhcp and device wan, giving this interface the metric 2?

Could you also clarify; Is the interface, wan and the device, wan connected or is the interface name arbitrary and therefore it coincidental that they're named the same?

The reason I ask, is that the device is in use and I don't want to keep interrupting internet access if I can avoid it.

Thanks again for prodding me in the direction of mwan3

Stephen

EDIT: Just to add, I'm reading the link you provided, specifically the section Creating additional WAN interfaces, subsection Routers using DSA. It says to take the LAN port out of br-lan so it can be used for a VLAN device, however this 'red port' on this device is not a member of br-lan and I can not identify with any certainty what this red port is actually called in OpenWrt.

EDIT2: Well I'm not having much luck with mwan3. It appears iptables-nft and ip6tables-nft are required for translation from iptables rules to nft rules which I'm not sure I really need as it's a routing problem not a firewall problem. Added to which, this itself seems to be broken as 23.05 compiles dnsmasq-full without ipset. It feels like I'm trying to clean a window with a hammer.

I would be happy if both WAN interfaces were configured and I had to raise and lower them manually making sure both were never up at the same time.

But....I can't work out how to employ this red port. What device in OpenWrt is this red port?

psherman · November 2, 2024, 6:38pm

The names can be confusing...

There is a device/port called wan -- this is not arbitrary as it is defined in the hardware definition that is used to build OpenWrt for your device.
That device/port is assigned for use in the wan interface. The name wan here is somewhat arbitrary. You can rename it, but the defaults assume it is named wan. If you change it, you also need to make sure that the interface name is updated in the wan firewall zone, otherwise it will break everything (in terms of routing) because the network won't be assigned to the appropriate zone.
The firewall zone called wan is also somewhat arbitrary. Again, it can be renamed, but if you do so, you need to make sure that all the places where the wan zone is used by name are updated with the new name.

Meanwhile, I haven't used mwan3 myself, nor have I used your device... so I can't help you with the details of your setup. Hopefully someone else can.

I do know that mwan3 does work properly on 23.05, but I cannot comment about the issues you've encountered because I don't use the package.

pavelgl · November 2, 2024, 9:16pm

mwan3 works just fine on 23.05, but first things first.

You can do that, or you can keep ethwan and just set its metric to 2. You can then use ethwan instead of wanb in the mwan3 configuration file.

As @psherman stated, the interface name is logical and can be arbitrary. The device name is a fixed physical (Linux kernel net_device) name and you should see it when you run ls -l /sys/class/net. The device name could be ethx.x, dslx.x, wan or whatever, and if the logical and physical interface names match, it's just a coincidance.

That's in case you need to create a second wan port and the wan device is already taken by the primary wan interface. You are using dsl0.101 for the primary wan interface, so you can use the "red" (wan) device to create the second wan interface.

Just set a higher metric for the second wan interface and there won't be any interruptions.

OpenWrt uses the same subnet for lan, so this will cause problems unless you change it.

hedzwillroll · November 2, 2024, 11:09pm

Thank you again, please forgive my frustration; I felt like I was going around in circles.
I'll reply to Pavel with some further observations.

hedzwillroll · November 3, 2024, 12:01am

Hi Pavel,

Thanks for replying. OpenWrt is not using 192.168.1.0/24, so no conflict with the upstream router there.

Firstly, can I check what exactly is actually required for mwan3 in this scenario (i.e. failover)?
If I understand correctly, I need mwan3, luci-app-mwan3, iptables-nft, ip6tables-nft (if using IPv6 presumably) and the script from Kishi? Do I actually need the firewall stuff if I only want failover?

Secondly, the result of ls -l /sys/class/net on my device, shows the following platform devices (among other virtual devices I have configured and the WiFi PHYs):

eth0 -> ../../devices/platform/1f400000.fpi/1e10b308.eth/net/eth0
lan1 -> ../../devices/platform/1f400000.fpi/1e108000.switch/net/lan1
lan2 -> ../../devices/platform/1f400000.fpi/1e108000.switch/net/lan2
lan3 -> ../../devices/platform/1f400000.fpi/1e108000.switch/net/lan3
lan4 -> ../../devices/platform/1f400000.fpi/1e108000.switch/net/lan4
wan -> ../../devices/platform/1f400000.fpi/1e108000.switch/net/wan

eth0 is classed as 'Ethernet Switch' in LUCI, whilst lan1-4 and wan are classed as 'Switch port'. What's confusing to me is that the eth0 and wan devices have the same MAC address.
It seems to me, that the wan device should be a member of br-lan as it appears to be the mystery red port, however it's never been a member of br-lan and doesn't have the same configuration options as lan1-4.

I followed the YouTube videos OpenWRT - Configure Multiwan Failover with mwan3 and OpenWRT 21.02 - Multi-WAN Load Balancing with mwan3 just to get a few pointers (I understand that they're referencing older versions of OpenWrt and that one of them pertains to load-balancing rather than failover).

I installed luci-app-mwan3 (and therefore mwan3) and both iptables-nft and ip6tables-nft (for good measure). I also installed the Kishi script just in case. I got lost around about where the instructions asked me to set up IP Sets and gave up (hence my question above about whther I actually need the nft stuff for failover).
The result was that I lost DNS resolution when I got an IP address from the upstream router, even though it had a lower metric than my existing WAN connection. If I stopped the second WAN interface, DNS resumed.

I should mention again, that the upstream router doesn't have a public IP yet, as the service has not yet been activated. I'm trying to get this working ahead of time.

Stephen

hedzwillroll · November 3, 2024, 12:36pm

For some additional clarity, see the attached images

I have some VLANs for 4 LAN subnets and a VLAN (br-lan.99) for a downstream Debian router that I'm building. I also have a site-to-site Wireguard VPN and a wwan leftover from when I was learning about USB tethering an old mobile phone for 4G backup. If it's relevant, lan1-2 are in br-lan and are trunk VLAN ports to two layer 2 switches. lan3-4 are in br-wifi along with the WifI radios.

As you can see, when I edit the configuration for the interface wan, its device is dsl0.101 because it's using the onboard xDSL modem (via Ethernet Adapter: "dsl0" . The device entry is Software VLAN: "dsl0.101" (wan). Is the wan in parentheses here referring to the interface rather than the device? Judging by the other entries, it would seem so. This would possibly mean that the device Switch port: "wan" further down is just another port that has the name wan, rather than anything related to the interface wan? If so, this is rather unfortunate naming. Perhaps this is a peculiarity to this device?

This other (crudely redacted) image shows which devices share a MAC address. As can be seen, eth0 and wan share an address which is definitely confusing me (I'm also fairly sure this MAC address changed at some point).

As is often the case with me, I find the barrier to entry with technology to be a lack of explanation of the basic fundamentals rather than the higher level concepts (or so it would seem to me anyway ). Now maybe it's true to say that it's encumbent upon me to understand the fundamentals before embarking on learning, but sometimes some information just isn't there.
Perhaps that's because anyone who is more than casually associated with a technology has assumed it's already understood, or have developed muscle memory for such things that they don't even realise they're omitting something fundamental to a beginner. This is why I love forums above all other forms of knowledge on the internet (excepting Wikipedia perhaps). A forum enables the sharing of knowledge and often, the steps through which knowledge was gained, which is invaluable for someone else who follows after! I must also add that the documentation for this project is, on the whole, excellent!

Thanks for reading. I have the house to myself today, so I will continue breaking this to try and make it work. If anyone can help with the wan thing in the mean-time, you have my sincere gratitude.
Stephen

pavelgl · November 3, 2024, 12:55pm

The mandatory packages are mwan3 and ip(6)tables-nft. You only need dnsmasq-full, ipset and the Kishi script if you intend to route by domain name specific destination addresses through a specific interface (which is not the case).

eth0 is an internal network interface between the CPU and the switch, so just ignore it.

No, it should stay out of the bridge.

Just to be clear, a lower metric value means a higher interface priority. When I suggested a value of 2, I assumed that the primary interface had a value of 1 (according to your post).

As for DNS, you can consider switching to public DNS resolvers. Otherwise, additional rules may be required as described here.

What you need is pretty straightforward and can be done with minimal editing of the default mwan3 configuration.

We will help you when you get the second wan interface up and running.

hedzwillroll · November 3, 2024, 2:23pm

Thank you, so this is if you have a scenario akin to PBR, for load-balancing right?

OK, I vaguely remember this is how things were done with VLAN on older versions using swconfig

Agreed (however it is technically just another port, correct?)

Yes, understood.

I realised I did not have any DNS servers configured in the ethwan interface when I configured it ). I have two pi-holes running on the LAN at the moment for DNS. I now have wan and ethwan interfaces running with ethwan receiving a 192.168.1/24 address from the upstream router and no DNS issues. wan has a metric of 10 and ethwan a metric of 20 (in the interface configuration, not mwan3)

I will try mwan3 again now I have the interfaces working correctly.

I thank you very much for sharing your time and knowledge with me Pavel
i shall now try mwan3 again with the knowledge that I only require mwan3 and the ip(6)tables-nft packages for my particular scenario (presumably not the ip6tables-nft if my ISP doesn't support IPv6?).

Could you confirm that the wan interface and the wan device are not inexorably linked?
i.e. the wan interface is a basic configuration in OpenWrt that defines communication with the outside and the wan device is in fact, the red port physically labelled WAN on this router?

Were this to be the case, I would rather see the wan device labelled wan1 which would be in keeping with the nomenclature for lan devices.

Thanks again,
Stephen

pavelgl · November 3, 2024, 4:04pm

Yes, it's just another port, but it should be separated from the other (LAN) ports at Layer 2.

With swconfig, the separation of the switch ports into lan and wan was done using VLANs.

In DSA, all switch ports are isolated by default, therefore you need a bridge to interconnect the four LAN ports.

You can remove lan4 from br-lan and use it as the device for the wan interface, or you can add wan (device) to br-lan and use it as an additional LAN port.

Sorry, I'm afraid I don't follow you.

Again, the interface name is logical and can be whatever you want. Device must be an available net_device. Probably in older versions when swconfig was used, the device name of your wan interface was eth0.2. Now with DSA it is wan .

Someone who has been using OpenWrt longer than me should confirm, but most likely wan has always been used as the interface name and this name can be found in multiple configuration files.

I don't have an answer to this, but the logic might be that there are multiple lan ports and only one wan, so it doesn't need to be numbered.

hedzwillroll · November 3, 2024, 4:52pm

OK, I think I understand now. As this router only has one WAN port, the device is called wan, however if a router had multiple WAN ports, would they not be called wan1, wan2, etc.?
This unfortunately caused me confusion because both were named the same, however I understand now.

I have re-installed mwan3. Incidentally, the package luci-app-mwan3 installs mwan3 and both iptables-nft and ip6tables-nft, so it's really only necessary to install luci-app-mwan3 if one is interested in using mwan3 with LUCI.

I have added wan and ethwan to the MultiWAN Manager Interface section, giving wan a metric of 1 and a weight of 1. ethwan gets a metric of 2 and a weight of 2 (I think the weight is not important here, as I'm not load-balancing).

In the Member section, I have added wan_m1_w1 and ethwan_m2_w2 with their appropriate interfaces.

In the Policy section I have added wan_fail_to_ethwan with both Members.

Finally in the Rule section, I have assigned the wan_fail_to_ethwan policy to the 3 default rules.

In Status>MultiWAN Manager, I see green and online for wan, and red and offline for ethwan which is to be expected as that doesn't have a public IP/route on that interface yet.

All seems to be ready, however I can't test until my service is live tomorrow.

One problem I have noticed, is that my Wireguard interface has defaulted to have a route via ethwan for some reason? This obviously means the tunnel is down, as it cannot find the other site via a tunnel that has no public IP.

Why has Wireguard done this and what should I do to rectify it? Does the tunnel need to be added to mwan3 and how do I ensure it changes its route depending on which WAN is active?

Thanks again for your help thus far
Stephen

pavelgl · November 3, 2024, 7:11pm

Sorry, I can't advise without seeing the mwan3 and network configurations, active ip rules, routing tables and so on...

hedzwillroll · November 4, 2024, 1:23pm

Fair enough, thanks.

mwan3 didn't work as expected this morning.
My xDSL connection via dsl0.101 ceased sometime early morning. I moved the modem lead from OpenWrt's DSL port to the DSL port on the back of the upstream router, but mwan3 did not failover.

I noticed in OpenWrt that the wan device had changed MAC address which meant it wasn't in the DMZ on the upstream router, as that upstream router was reserving an IP address for the MAC address that it had yesterday.
mwan3 also showed ethwan as no longer having a gateway metric (presumably because the MAC changed?). I changed ethwan to use the static address the upstream router was configured to reserve, rather than DHCP.

Yesterday, the result of ip route show confirmed that both wan and ethwan had entries for routing externally, however when I did ip route show this morning, neither wan or ethwan had an outside route, so I had to create a static route for 0.0.0.0/0 to ethwan in order to provide internet access.

So, I give up for now. It was worth trying, as I wouldn't mind being able to use mwan3, but it will have to wait. Perhaps I should try again with one of the lan ports instead of wan, but that seems to be a waste.

Does anyone have any idea why the device wan got a new MAC address? Remember, that device wan and eth0 on this device share the same MAC address (eth0 having been explained as an internal interface between the CPU and switch).
Why would that internal CPU<->Switch interface keep getting a different MAC? Is there any way to force a MAC for the eth0 (and wan) devices, as I can't reserve IP addresses for device wan otherwise.

hedzwillroll · November 16, 2024, 11:52am

So just in case this helps anyone else viewing this topic, I discovered this earlier in the system log:

Policy wan_fail_to_ethwan exceeds max of 15 chars. Not setting policy

which might explain why it didn't failover.

From the MultiWAN Manager - Policies section:

Policies are profiles grouping one or more members controlling how Mwan3 distributes traffic.Member interfaces with lower metrics are used first.Member interfaces with the same metric will be load-balanced.Load-balanced member interfaces distribute more traffic out those with higher weights.Names may contain characters A-Z, a-z, 0-9, _ and no spaces.Names must be 15 characters or less.Policies may not share the same name as configured interfaces, members or rules

So my bad!
Would be good if the Name textbox wouldn't allow >15 characters

It doesn't explain why my routes went missing or why the wan device MAC keeps changing though, but it does explain why it didn't even try.