Using mwan3 for a week or so now.
I've run into a few places where I log in, but my log in gets broken. I either have to keep logging back in, or I lose permissions. I assume because I'm sending packets from more than one ISP.
From simple sites where you just log in with a user name and password for access, to sites where you log in to stream content.
Is there a way to get mwan3 or a PCs browser to stick with a single ISP for connection to the same IP address you logged in to?
Welcome to the world of load-balancing, there is no real solution to this (even if you go to high-priced commercial business routers) - apart from using an external entity (ISP or server in a data centre) to stitch the WAN uplinks together again.
Approaches to mitigate this would involve distributing equal numbers of your clients to each of the WAN uplinks (resulting in a static and rather ineffective load-balancing, instead of a dynamic one). Everything else either gets complex very, very fast (with huge policy based routing rule sets) or just doesn't work.
tl;dr: fail-over is easy, load-balancing as a concept is broken by design.
You can create a rule for a specific source IP and destination 80 and 443 to use a single uplink policy. That will cover the scenario of the PC browser.
Could you be more specific, trendy?
This is starting to feel like a deal breaker.
But yeah, specific WANs to specific clients and failover seems like the only solution with actual load balancing being a joke.
In case you want to use the balancer policy, you can increase the stickyness timeout. From the mwan3 example:
config rule 'youtube'
option sticky '1'
option timeout '300'
option ipset 'youtube'
option dest_port '80,443'
option proto 'tcp'
option use_policy 'balanced'
300 is in seconds, so 5 minutes.
Otherwise, as mentioned before, make a rule for the tcp/80,443 for a specific lan host and assign it to a failover policy, like
config policy 'wan_wanb'
list use_member 'wan_m1_w3'
list use_member 'wanb_m2_w2'
list use_member 'wan6_m1_w3'
list use_member 'wanb6_m2_w2'
If the sticky isn't set, is every packet balanced with no WAN being held onto for any period of time?
If that's the case, could sticky be added to https, default_rule_v4 and default_rule_v6?
I know that would make things less balanced, but what I'm running into is issues occurring from a minute or two to seconds from logging in to a page then clicking the next link there.
Edit: I see https seems to have sticky on by default.
Edit 2: I guess it doesn't work that way. 
It's not about the individual packet, but for the connections. If individual packets were split to different uplinks, nothing would work.
Wouldn't it be a good idea to use the increased stickyness rule for all websites?
config rule 'fix_login'
option sticky '1'
option timeout '300'
option dest_port '80,443'
option proto 'tcp'
option use_policy 'balanced'
Wouldn't it be a good idea to use the increased stickyness rule for all websites or would that pretty much be about the same as setting a device to use only one WAN?
I mean if I do the rule below, will all websites go through the same WAN or would there still be balance with each website getting a different WAN?
config rule 'fix_login'
option sticky '1'
option timeout '1000000'
option dest_port '80,443'
option proto 'tcp'
option use_policy 'balanced'
EDIT: WAIT WAIT WAIT...
> option ipset 'youtube'
I didn't know this was a thing.
Name of IPset rule. Requires IPset rule in /etc/dnsmasq.conf (eg "ipset=/youtube.com/youtube")
Does that mean you have to use dnsmasq and create an ipset with all the ip addresses for a url? That seems... sloppy.
How would I set up kodi's forum which is:
Name: forum.kodi.tv
Addresses: 2606:4700:20::ac43:538f
2606:4700:20::6819:2447
2606:4700:20::6819:2347
172.67.83.143
104.25.36.71
104.25.35.71
FYI. Sticky rules should resolve this issue and mwan3 by default has a default rule for all https traffic which will "fix" the traffic from the same source for a period of time, this should normally resolve the issue where by a website does not like seeing different source IPs in the same session, but it has been found a regression in mwan3 2.11 has broken sticky rules, which is why you might be seeing problems now.
However for ISP specific related areas, you are better off created your own rules that will assign a policy that ensures the traffic goes through a specific WAN all the time.
I do see I'm running 2.11.4. I don't know if that issue is fixed or not.
Like I posted, having 6 ip addresses makes things like this a mess. What if they add or subtract IP address? I have to keep checking.
I really don't understand why OpenWRT doesn't allow you to set the domain name instead. Fresh Tomato has that as part of their load balancing. Seems like a good option.
Anyway, any links for how to properly set up an IPSet? That's new to me and guides I find seem to refer to firewall setups and aren't very helpful.
Sure mwan3 supports ipset, detailed in the docs: https://openwrt.org/docs/guide-user/network/wan/multiwan/mwan3#ipset_support
You need to create an ipset for one or more domains in dnsmasq and then add a mwan3 rule using a policy, enabling sticky support is optional. It would not be needed if you are assigning a policy which routes traffic through a single WAN.
You might need to check things though as 22.03 uses a different firewall using nftables rather than iptables, which as far as I know mwan3 needs iptables rules, but these have to be translated, so you might need to install iptables-nft and ip6tables-nft
Wow. I have no idea what that means. I need to stop updating OpenWrt.
The background to that is the firewall solution since 22.03 in OpenWrt (firewall4) now uses nftables, rather than iptables. Most packages have been updated to work with nftables, but mwan3 remains using iptables. However, there is a package iptables-nftables (IPv4) and ip6tables-nftables (IPv6) which acts as the translator, to essentially create firewall rules using iptables still but then gets converted to nftables rules so it works with firewall4.
I was just advising that you'll like need those packages installed for rules to work. ipset rules is what you need for your solution, the reason why you can't just specify a domain is because routing works on IP, but ipset allows you to handle sites that uses multiple IPs, CDNs etc, by dynamically adding any IP resolved into an ipset policy, so these get routed as needed.
The mwan3 docs explains ipset essentially, creating an ipset policy in dnsmasq, then creating your ipset rule in mwan3 to match the traffic, but if you are stuck I'm sure we can help. You don't need to modify the firewall directly, as mwan3 is adding all the required stuff for you via the config.
Updating OpenWrt is a good thing, you do have to pay attention to any major release, given major changes and potentially setup breaking changes between major versions. For example 21.02, changed the switch management from swconfig to DSA, 22.03, changed the upstream firewall to nftables. Minor releases for the same major branch generally don't cause issues, but always worth reading the release notes for any release along with anything specific to your routers target before upgrading.
I'm new to this level of network management. So an update that might say something like 'update works with nftables' is like saying 'herpy derpy do'.
I don't know if it's a major change or unimportant.
One of the biggest issues with OpenWRT is it assumes a very high level of understanding from every user. Even the online guides sound like they were written by someone for whom English is a second language (the first being a computer language).
I'm trying since there are features I need, but if an update is going to break things and require more research, it makes updating the software dangerous.