Ok, let's treat all of these things in order. This round robin one first as there's no point checking the other issues until we determine what's happening here.
I'll look at the speedtest and PBR issues after we've resolved this one.
Looking at the kernel implementation in nft_numgen.c, numgen uses an atomic CAS loop:
static u32 nft_ng_inc_gen(struct nft_ng_inc *priv)
{
u32 nval, oval;
do {
oval = atomic_read(priv->counter);
nval = (oval + 1 < priv->modulus) ? oval + 1 : 0;
} while (atomic_cmpxchg(priv->counter, oval, nval) != oval);
return nval + priv->offset;
}
It's a single global counter. Every CPU increments the same atomic value. So perfect round-robin is expected behaviour and is what I'm getting.
Possible (and likely) explanations why you're not seeing this behaviour:
The DNS record for ifconfig.me has both an IPv6 address and an IPv4 address. If you don't have IPv6 connectivity and it resolves only to a single IPv4 address for you then you should get perfect round robin. But you're not, so either the DNS is resolving to multiple IPv4 addresses for you or you have IPv6 connectivity or both.
1. ifconfig.me is resolving to multiple different IPv4 addresses for you
if DNS returns different IPs across queries and only 34.160.111.145 matches the dest_ip rule, unmatched connections fall through to default routing (one fixed wan egress). That would produce exactly this kind of clustered pattern. nslookup ifconfig.me will tell you this as it returns all A records in one shot.
Use an nft set / ip set in the rule instead of a destination ip address. Add an ipset to dnsmasq for ifconfig.me and filter on the nft set in the mwan3 rule instead. Just remember to restart dnsmasq and make sure whatever lan client you're doing the test from will resolve its ip from dnsmasq or just do a nslookup or dig on the openwrt host to force dnsmasq to resolve the query and populate the set.
You can check the set is populated on the openwrt host with
nft list set inet fw4 <setname>
Using an nftset will also resolve the potential IPv6 issue below, so changing the rule in this way is desirable. I've set my own host that I was testing from to prefer ipv4 because I get ipv6 through a tunnel, so it wasn't necessary in my case.
2. Your device has ipv6 connectivity and curl connects via ipv6 some of the time
curl's algorithm tries IPv6 first, only falling back to IPv4 after a 200ms timeout. ifconfig.me has an AAAA record and if you have IPv6 connectivity on one or both wans, connections where IPv6 completes in under 200ms bypass the mwan3 rule entirely (since it is IPv4-only) and exit via the IPv6 default route.
If IPv6 latency varies, for example one wan has consistently low IPv6 latency while the other is higher or has occasional spikes, then sometimes IPv6 wins the race and sometimes it doesn't, causing an unpredictable mix of IPv6-routed connections ignoring mwan3 and IPv4-routed connections load balanced by mwan3. The result would be the irregular pattern you see.
Again, fix it by changing the rule to use an nft set and add ifconfig.me to that nft set in the dnsmasq configuation.
You can also add a -4 parameter to the curl invocation just to be sure on an initial test. If you add the nft set and that produces round robin behaviour, then try again without the -4 parameter
while true; do echo $(curl -4 https://ifconfig.me 2> /dev/null); done