Can mwan3 be used with multiple wan interfaces to pick the interface that's currently providing the best bandwidth to a certain IP?
That would sure make my life easier. It seems that the best wan interface always changes and I'm experiencing pretty extreme variations in bandwidth. Like 100% at one time and 10% at another.
I'm not sure ping will really work. The IP that I need to reach is on the other side of the planet basically so depending on the circumstances in either location, the best wan interface can exhibit very different results.
Right now I got mwan3 working and the 3 wan interfaces all have the same priority so the connections are spread evenly among the three (inefficient).
Can you actually predict which of the three is the best or do you actual need to measure throughput over all three? And if a link is better than the others how long does that tyically last?
Wow. You guys write fast. Thanks for that first of all.
It could change at any time but every couple of hours would be a great start.
It would be great if I could predict it but I'm afraid I can't.
The values mwan3 uses to choose interface, can those be changed by other packages? Then it might just be a matter of time before enough people push for a solution.
@alcatraz if you have a variable capacity connection then you may well benefit from CAKE with autorate to resolve bufferbloat, i.e. stop the connection getting saturated resulting in buffers filling up and increased latency. CAKE-autorate adjusts the bandwidth based on measurements to various 'reflectors' like 8.8.8.8. With load bandwidth is increased but with RTT spikes the bandwidth is decreased. So with load, CAKE-autorate gives an indication of usable bandwidth because it oscillates around the usable bandwidth that the connection can manage without giving a significant RTT increase.
So in this sense the capacity estimation is passive. I like this the best because blasting the connection with speed tests every now and then will necessarily interfere with latency sensitive applications. Of course you could blast the connection manually with speed tests to give an up to date measurement with the passive approach already adopted in CAKE-autorate. Or just accept that connection switches will happen only under load when a given interface is being measured. Which feels like it might be OK.
I think there might be a way to leverage the measurements the CAKE-autorate script already does to help inform switching from one interface to the other. And this makes particular sense given that you probably want to use CAKE with autorate anyway to allow latency sensitive applications to work well during connection saturation notwithstanding variable capacity connections.
You can get data like this:
And I'm curious about which interface you would apply CAKE on. With mwan3 is there a 'global' wan and two child interfaces? Or is there a second 'wan'? With CAKE it can be helpful to have one interface through which all traffic is routed on which you can apply CAKE for download and upload. It could be that a solution here involves the use of an 'ifb' or 'veth' interface.
If you don't want/need CAKE then you could write a simple bash script by borrowing from this code on my GitHub:
monitor_achieved_rates()
{
# track rx and tx bytes transfered and divide by time since last update
# to determine achieved dl and ul transfer rates
local rx_bytes_path=$1
local tx_bytes_path=$2
local monitor_achieved_rates_interval_us=$3 # (microseconds)
compensated_monitor_achieved_rates_interval_us=$monitor_achieved_rates_interval_us
[[ -f $rx_bytes_path ]] && { read -r prev_rx_bytes < $rx_bytes_path; } 2> /dev/null || prev_rx_bytes=0
[[ -f $tx_bytes_path ]] && { read -r prev_tx_bytes < $tx_bytes_path; } 2> /dev/null || prev_tx_bytes=0
while true
do
t_start_us=${EPOCHREALTIME/./}
# If rx/tx bytes file exists, read it in, otherwise set to prev_bytes
# This addresses interfaces going down and back up
[[ -f $rx_bytes_path ]] && { read -r rx_bytes < $rx_bytes_path; } 2> /dev/null || rx_bytes=$prev_rx_bytes
[[ -f $tx_bytes_path ]] && { read -r tx_bytes < $tx_bytes_path; } 2> /dev/null || tx_bytes=$prev_tx_bytes
dl_achieved_rate_kbps=$(( ((8000*($rx_bytes - $prev_rx_bytes)) / $compensated_monitor_achieved_rates_interval_us ) ))
ul_achieved_rate_kbps=$(( ((8000*($tx_bytes - $prev_tx_bytes)) / $compensated_monitor_achieved_rates_interval_us ) ))
(($dl_achieved_rate_kbps<0)) && dl_achieved_rate_kbps=0
(($ul_achieved_rate_kbps<0)) && ul_achieved_rate_kbps=0
printf '%s' "$dl_achieved_rate_kbps" > /tmp/CAKE-autorate/dl_achieved_rate_kbps
printf '%s' "$ul_achieved_rate_kbps" > /tmp/CAKE-autorate/ul_achieved_rate_kbps
prev_rx_bytes=$rx_bytes
prev_tx_bytes=$tx_bytes
# read in the max_wire_packet_rtt_us
concurrent_read_positive_integer max_wire_packet_rtt_us /tmp/CAKE-autorate/max_wire_packet_rtt_us
compensated_monitor_achieved_rates_interval_us=$(( (($monitor_achieved_rates_interval_us>(10*$max_wire_packet_rtt_us) )) ? $monitor_achieved_rates_interval_us : $((10*$max_wire_packet_rtt_us)) ))
sleep_remaining_tick_time $t_start_us $compensated_monitor_achieved_rates_interval_us
done
}
This reads the rx_bytes and tx_bytes files to monitor the bandwidth used by an interface. And you could make simple decisions like if bandwidth < X on interface A then you switch to interface B, etc.
Clearly something along these lines would require you to have some bash knowledge. Or perhaps I can help. I think it would be worth thinking about whether you want/need CAKE and what sort of rules you would want to apply to decide to switch from one interface to the other.
haven't used mwan3 package myself, so I can't really say what it is, and isn't capable of, but wouldn't it be easier to test the throughput (or what ever you use for as a base for your measures) for all interfaces, and based on the results, alter the default gw to the fastest one ?
Thanks for the idea. Is cake-autorate a congestion control algorith I need to enable on the server? (It's running debian i386)
The client side (openwrt) only sees downloads so uploads don't need to be optimized.
It's one physical wan interface with several vpn connections to openwrt boxes I've placed in the same city but at different ISPs. One isp is always faster than the others, and it changes from time to time. It's all about the peering to the destination country and how that network handles congestion during peak hours.
I really appreciate your help. This is quite exciting.
Yep. Problem is it changes every once in a while. One gateway varies between 20-40% throughput, and another between 10-100%. Most of the time the latter is ideal but sometimes I'm stuck at 10% when 40% could be available on the other interface.
Imagine if I add a third interface, even better. But the constant manual switching would be annoying, because I don't know if it's going to be better.
I guuess you could use mwan3 for all three? uplinks and use metric parameter to change the active uplink on the fly. Then all you need to do is schedule repeated speedtests over all three links (independent on which is active via mwan3) to figure out which is the best one.
The catch is that repeated speedtests with sufficient temporal resolution will produce a ton of useless data transfers... (albeit short ones).