mwan3 with email notification and tracking verbosity enabled
/modifications applied/
Hello again,
In the process of testing the issue described in my previous posts, I did several modifications of mwan3 in order to increase the verbosity of the instrument. So, I decided to go a little further and add some hefty (according to me) features, using the great customize optional hotplug script of mwan3 together with some simple change in the mwan3 tracking engine - mwan3track. Tested and works OK on OpenWrt BB with mwan3 v1.5-10.
Probably this needs a separate topic, but here it is.
Brief description of the features added:
1) Email notification:
this feature is accomplished, modifying /etc/hotplug.d/iface/16-mwancustom
very thin email agent (mua) is used - ssmtp
the email agent is automatically configured for mwan3 usage by mwan3 itself
the only configuration needed is setting the relevant variables (in the relevant section) in /etc/hotplug.d/iface/16-mwancustom
the email content can be customized of course, either by changing the email variables for a basic change in the email structure, or by editing the code and alter the content that mwan3 pushes
the email notification is processes only on mwan3 enabled interfaces and only on interface status change
the email notification can be disabled by setting a preset variable (disables by default)
discovery and notification when the system has just been powered on / booted (up to 5 minutes after system boot) - in this case the message sent is simplified and mwan3 status reports and logs are not sent (can be disabled as well)
compatible with non-default logging targets (for example, if logging to /var/log/messages is configured), logread option is still available if no specific log file is configured
2) NAT table flushing on WAN switchover:
optional feature of mwan3 also added in /etc/hotplug.d/iface/16-mwancustom - now it flushes the NAT table, which is highly recommended for normal and optimal NAT table operation upon WAN links switchover. This is important for services that are exposed to Internet via DNAT by the router doing the multiwan. This feature will work only if you have conntrack-tools installed, conntrack-tools depends on libnetfilter-conntrack, which are relatively small packages.
ex.)
I am running a PBX behind the router with mwan3 and without NAT table flush, the sip connectivity does not work right, since the socket allocated from the primary wan somehow remains running on the other working one, after the primary wan failure and restoration as well. It is the same with openvpn instances that are terminated on an internal server (not the router itself, the router is doing simply DNAT). The main downside is that each connection from an internal host in the network (especially valid for TCP) has to be initiated again. Most services have daemons that are taking care of that, but if you are watching a video stream from your PC for example, you will have to reload it.
Here is the content of /etc/hotplug.d/iface/16-mwancustom
#!/bin/sh
. /lib/functions.sh
# to enable this script uncomment the case loop at the bottom
# to report mwan status on interface hotplug ifup/ifdown events modify the lines in the send_alert function
###################################
####### General variables: ########
###################################
# Define needed variables:
SCRIPTNAME="mwan3"
SYSTEM_LOG_FILE="/var/log/messages" # Leave it empty if using the default syslogd configuration with logread macro
LOG="$(which logger) -t ${SCRIPTNAME} -p"
###################################
#
###################################
## Email Notification variables: ##
###################################
#
# Email client settings (needed only if EMAIL_NOTIFY="1"):
EMAIL_NOTIFY="0" # Enable (1) / Disable (0) email notification on failover actions
EMAIL_CLIENT="$(which ssmtp)" # The MUA (Mail user agent)
EMAIL_CLIENT_CONF="/etc/ssmtp/ssmtp_${SCRIPTNAME}.conf" # Where the configuration of the MUA is located
EMAIL_USER="my.example@gmail.com" # The actual email account that is sending the email
EMAIL_PASSWD="MyEmailPassword" # Your email account password
SMTP_SERVER="smtp.gmail.com" # Outgoing mail server
DST_PORT="465" # Destiantion port used by the mail client (smtp port is 25 by default, without encryption)
EMAIL_DOMAIN="gmail.com" # Domain of the mail server
EMAIL_CLIENT_TLS="YES" # "YES" or "NO" are valid options
EMAIL_SUBJECT="Example, WAN status change detected" # Subject of the email message
# List of Email notification recipients
EMAIL_TO="example1@domain1 example2@domain2 example3@domain3"
EMAIL_SIGNATURE="powered by mwan3 & OpenWrt"
###################################
send_alert()
{
# variable "$1" stores the MWAN status information
# insert your code here to send the contents of "$1"
#echo "$1"
###################################
local HOST="$(uci get -p /var/state system.@system[0].hostname)"
# Check whether the email agent is installed
if [ "${EMAIL_NOTIFY}" -ne "1" ]; then ${LOG} notice "Email alert notification is not enabled"; return 2; fi
if [ ! -x "${EMAIL_CLIENT}" ]; then ${LOG} warning "Warning: Email client ${EMAIL_CLIENT} is not installed or configured on the system"; return 3; fi
LOG_MESSAGE="${@}"
sleep 1
# Checking whether the Email client has already been configured for our account and if not, write its configuration
grep -e "generated by ${SCRIPTNAME}" ${EMAIL_CLIENT_CONF} &> /dev/null || {
# Backing up if there is some old configuration first
cp ${EMAIL_CLIENT_CONF} ${EMAIL_CLIENT_CONF}.backup > /dev/null 2>&1
# Writing the Email Client configuration (SSMTP)
echo -e "# ${EMAIL_CLIENT_CONF} -- a configuration file for sSMTP sendmail.\n### generated by ${SCRIPTNAME}\nroot=${EMAIL_USER}\nmailhub=${SMTP_SERVER}:${DST_PORT}\nrewriteDomain=${EMAIL_DOMAIN}\nhostname=${HOST}\nFromLineOverride=YES\nUseTLS=${EMAIL_CLIENT_TLS}\nAuthUser=${EMAIL_USER}\nAuthPass=${EMAIL_PASSWD}" > ${EMAIL_CLIENT_CONF}
}
# Composing the Email message and pipe it to the smtp client in order to be sent
sleep 1
echo -e "From: <${EMAIL_USER}>\nSubject: ${EMAIL_SUBJECT}\n\n${HOST} alert notification message.\n\n$(echo -e "${LOG_MESSAGE}")\n\n\n${EMAIL_SIGNATURE}" | ${EMAIL_CLIENT} -C ${EMAIL_CLIENT_CONF} ${EMAIL_TO}
sleep 1
${LOG} notice "Sending email alert notification to $(echo "${EMAIL_TO}"|sed "s/ /,/g")"
unset EMAIL_PASSWD
unset LOG_MESSAGE
###################################
}
gather_event_info()
{
local device enabled
config_load mwan3
config_get enabled "${INTERFACE}" enabled 0
device=$(uci get -p /var/state network.${INTERFACE}.ifname) &> /dev/null
#echo -e "Interface is: ${INTERFACE}; Device is: ${device}|($DEVICE); enabled = ${enabled};"
if [ -n "$device" ] ; then
#ACTION=ifup INTERFACE=$1 DEVICE=$device /sbin/hotplug-call iface
# The interface is not enabled in mwan3 configuration, so no message alerting should be initiated
[ "${enabled}" -eq "1" ] || { echo "Warning: Interface ${INTERFACE} not enabled for message alerting"; exit 1; }
else
echo "Warning: Invalid interface call"; exit 1;
fi
# create event information message
local EVENT_INFO="Interface [ "$INTERFACE" ($DEVICE) ] on router [ "$(uci get -p /var/state system.@system[0].hostname)" ] has triggered a hotplug [ "$ACTION" ] event on "$(date +"%a %b %d %Y %T %Z")""
# get current interface, policy and rule status
local CURRENT_STATUS="$(/usr/sbin/mwan3 status)"
# get last 50 MWAN systemlog messages
if [ -e ${SYSTEM_LOG_FILE} ] ; then
local MWAN_LOG="$(echo -e "Last 50 MWAN systemlog entries. Newest entries sorted at the top:\n$($(which cat) ${SYSTEM_LOG_FILE} | grep mwan3 | tail -n 50 | sed 'x;1!H;$!d;x')")"
else
local MWAN_LOG="$(echo -e "Last 50 MWAN systemlog entries. Newest entries sorted at the top:\n$(logread | grep mwan3 | tail -n 50 | sed 'x;1!H;$!d;x')")"
fi
# get the uptime in minutes and send different notification if the system uptime is less than 5 mins
local UPTIME_MINUTES=$($(which uptime) | sed s/^.*up// | awk -F, '{ if ( $3 ~ /user/ ) { print $1 $2 } else { print $1 }}' | sed 's/^ *//' | awk '{ if ( $2=="min" ) { print $1 } }' | cut -d ":" -f1)
# pass event info to send_alert function
[ ! -z "${UPTIME_MINUTES}" -a "${UPTIME_MINUTES}" -le "5" ] && { send_alert "$(echo -e "The system has just been powered on and started ${UPTIME_MINUTES} min ago!\n\n$EVENT_INFO")"; exit 0; }
send_alert "$(echo -e "$EVENT_INFO\n\n$CURRENT_STATUS\n\n$MWAN_LOG")"
# Flushing the NAT table
[ -e $(which conntrack) ] && { ${LOG} notice "Flushing NAT Table"; sleep 1; $(which conntrack) -F &> /dev/null; sleep 1; };
}
case "$ACTION" in
ifup)
gather_event_info
;;
ifdown)
gather_event_info
;;
esac
exit 0
3) Tracking verbosity:
this feature is accomplished with small modification of /usr/sbin/mwan3track, which is not recommended at all, so a backup of the original file is obligatory
mwan3track now detects the failure/recovery based on packet loss values, comparing each result with a predefined packet loss threshold that we can set and consider crucial for the connectivity
if packet loss above the defined threshold is observed to any of the tracked hosts, a log message is produced in the system log
the above rule is valid while the interface is NOT marked as inactive/offline, in order to avoid the system log to be flooded with similar messages in the periods during which the connectivity via the particular interface is down
this feature helps to keep an eye on the link quality as well, because we can detect packet loss (even if it is not 100%) to the destinations tracked by mwan3. If we choose the monitored hosts in the right way, we can detect problems not only via a specific interface, but related to a global network issues, end destinations outages, etc
busybox ping applet is now hardcoded in order to avoid the issue described in my previous posts
Here is the content of the modified /usr/sbin/mwan3track
#!/bin/sh
[ -z "$9" ] && echo "Error: should not be started manually" && exit 0
if [ -e /var/run/mwan3track-$1.pid ] ; then
kill $(cat /var/run/mwan3track-$1.pid) &> /dev/null
rm /var/run/mwan3track-$1.pid &> /dev/null
fi
echo "$$" > /var/run/mwan3track-$1.pid
score=$(($7+$8))
track_ips=$(echo $* | cut -d ' ' -f 9-99)
host_up_count=0
lost=0
######
# Custom variables:
MUTE_WHILE_DOWN="0" #Do not edit - prevents logging of packet loss results while the connection vie the interface is down
PLOSS_DEFAULT="100" #Default Packet loss rate in %
PLOSS_THRESHOLD="5" #Packet loss rate in %, that triggers host condition change (up|down)
######
while true; do
for track_ip in $track_ips; do
######
#/bin/busybox ping -I $2 -c $4 -W $5 -s 4 -q $track_ip &> /dev/null
#/usr/bin/ping -I $2 -c $4 -W $5 -s 4 -q $track_ip &> /dev/null
PING_RESULT=$(/bin/busybox ping -I $2 -c $4 -W $5 -s 4 -q $track_ip | grep "packet loss") &> /dev/null
PLOSS=$(echo "${PING_RESULT}" | sed -e 's/^.*received, //' -e 's/^.*errors, //' -e 's/% packet loss.*//')
if [ -z "${PLOSS}" ]; then PLOSS=${PLOSS_DEFAULT}; fi
#if [ $? -eq 0 ]; then
if [ "${PLOSS}" -le "${PLOSS_THRESHOLD}" ]; then
######
let host_up_count++
else
let lost++
######
[ "${MUTE_WHILE_DOWN}" -eq "0" ] && logger -t mwan3track -p notice "Network issue detected on Interface $1 ($2): $track_ip -> ${PLOSS}% packet loss";
######
fi
done
if [ $host_up_count -lt $3 ]; then
let score--
if [ $score -lt $8 ]; then score=0 ; fi
if [ $score -eq $8 ]; then
logger -t mwan3track -p notice "Interface $1 ($2) is offline"
env -i ACTION=ifdown INTERFACE=$1 DEVICE=$2 /sbin/hotplug-call iface
score=0
######
MUTE_WHILE_DOWN="1"
######
fi
else
if [ $score -lt $(($7+$8)) ] && [ $lost -gt 0 ]; then
logger -t mwan3track -p info "Lost $(($lost*$4)) ping(s) on interface $1 ($2)"
fi
let score++
lost=0
if [ $score -gt $8 ]; then score=$(($7+$8)); fi
if [ $score -eq $8 ]; then
logger -t mwan3track -p notice "Interface $1 ($2) is online"
env -i ACTION=ifup INTERFACE=$1 DEVICE=$2 /sbin/hotplug-call iface
######
MUTE_WHILE_DOWN="0"
######
rm /var/run/mwan3track-$1.pid
exit 0
fi
fi
host_up_count=0
sleep $6
done
exit 1
And diff against the original file
[root@Core-Sf:~]# diff /usr/sbin/mwan3track /usr/sbin/mwan3track.original
--- /usr/sbin/mwan3track
+++ /usr/sbin/mwan3track.original
@@ -14,32 +14,14 @@
host_up_count=0
lost=0
-######
-# Custom variables:
-MUTE_WHILE_DOWN="0" #Do not edit - prevents logging of packet loss results while the connection vie the interface is down
-PLOSS_DEFAULT="100" #Default Packet loss rate in %
-PLOSS_THRESHOLD="5" #Packet loss rate in %, that triggers host condition change (up|down)
-######
-
while true; do
for track_ip in $track_ips; do
- ######
- #/bin/busybox ping -I $2 -c $4 -W $5 -s 4 -q $track_ip &> /dev/null
- #/usr/bin/ping -I $2 -c $4 -W $5 -s 4 -q $track_ip &> /dev/null
- PING_RESULT=$(/bin/busybox ping -I $2 -c $4 -W $5 -s 4 -q $track_ip | grep "packet loss") &> /dev/null
- PLOSS=$(echo "${PING_RESULT}" | sed -e 's/^.*received, //' -e 's/^.*errors, //' -e 's/% packet loss.*//')
- if [ -z "${PLOSS}" ]; then PLOSS=${PLOSS_DEFAULT}; fi
-
- #if [ $? -eq 0 ]; then
- if [ "${PLOSS}" -le "${PLOSS_THRESHOLD}" ]; then
- ######
+ ping -I $2 -c $4 -W $5 -s 4 -q $track_ip &> /dev/null
+ if [ $? -eq 0 ]; then
let host_up_count++
else
let lost++
- ######
- [ "${MUTE_WHILE_DOWN}" -eq "0" ] && logger -t mwan3track -p notice "Network issue detected on Interface $1 ($2): $track_ip -> ${PLOSS}% packet loss";
- ######
fi
done
@@ -52,9 +34,7 @@
logger -t mwan3track -p notice "Interface $1 ($2) is offline"
env -i ACTION=ifdown INTERFACE=$1 DEVICE=$2 /sbin/hotplug-call iface
score=0
- ######
- MUTE_WHILE_DOWN="1"
- ######
+
fi
else
@@ -73,9 +53,6 @@
logger -t mwan3track -p notice "Interface $1 ($2) is online"
env -i ACTION=ifup INTERFACE=$1 DEVICE=$2 /sbin/hotplug-call iface
- ######
- MUTE_WHILE_DOWN="0"
- ######
rm /var/run/mwan3track-$1.pid
exit 0
fi
The options and the variables are self-explanatory, but several comments were added as well.
Here is a snippet of the mwan3 logging, showing how it reacts on a network issue affecting one of the wan links. This is replicated, by disabling a phy link (connected to a manageable l2 switch) throughout the path to the ISP, but not physically connected to the mwan3 router itself, in order to avoid the generic netifd mechanisms to kick in, thus emulating a connectivity problem.
Aug 17 00:31:30 switch 786: Aug 17 00:31:27 EEST: %LINK-5-CHANGED: Interface FastEthernet0/2, changed state to administratively down
Aug 17 00:31:30 switch 787: Aug 17 00:31:28 EEST: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/2, changed state to down
Aug 17 00:31:34 Core-Sf mwan3track: Network issue detected on Interface wan2 (eth0.3): 208.67.220.220 -> 100% packet loss
Aug 17 00:31:41 Core-Sf mwan3track: Network issue detected on Interface wan2 (eth0.3): 208.67.222.222 -> 100% packet loss
Aug 17 00:31:47 Core-Sf mwan3track: Network issue detected on Interface wan2 (eth0.3): 8.8.8.8 -> 100% packet loss
Aug 17 00:31:53 Core-Sf mwan3track: Network issue detected on Interface wan2 (eth0.3): 87.120.130.66 -> 100% packet loss
Aug 17 00:31:59 Core-Sf mwan3track: Network issue detected on Interface wan2 (eth0.3): 46.249.80.1 -> 100% packet loss
Aug 17 00:31:59 Core-Sf mwan3track: Interface wan2 (eth0.3) is offline
Aug 17 00:31:59 Core-Sf mwan3: ifdown interface wan2 (eth0.3)
Aug 17 00:32:06 Core-Sf sSMTP[5675]: Creating SSL connection to host
Aug 17 00:32:07 Core-Sf sSMTP[5675]: SSL connection using ECDHE-RSA-AES128-GCM-SHA256
Aug 17 00:32:10 Core-Sf sSMTP[5675]: Sent mail for my.email@gmail.com (221 2.0.0 closing connection lk16sm13747142wic.6 - gsmtp) uid=0 username=root outbytes=8273
Aug 17 00:32:11 Core-Sf mwan3: Sending email alert notification to email1@domain1,email2@domain2,email3@domain3
Aug 17 00:32:11 Core-Sf mwan3: Flushing NAT Table
Aug 17 00:32:13 Core-Sf root: stopping ntpclient
########## bringing the switch port up after 3-4 minutes ##########
Aug 17 00:34:08 switch 788: Aug 17 00:34:07 EEST: %LINK-3-UPDOWN: Interface FastEthernet0/2, changed state to up
Aug 17 00:34:08 switch 789: Aug 17 00:34:08 EEST: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/2, changed state to up
Aug 17 00:35:22 Core-Sf mwan3track: Lost 95 ping(s) on interface wan2 (eth0.3)
Aug 17 00:36:42 Core-Sf mwan3track: Interface wan2 (eth0.3) is online
Aug 17 00:36:43 Core-Sf mwan3: ifup interface wan2 (eth0.3)
Aug 17 00:36:50 Core-Sf sSMTP[6647]: Creating SSL connection to host
Aug 17 00:36:50 Core-Sf sSMTP[6647]: SSL connection using ECDHE-RSA-AES128-GCM-SHA256
Aug 17 00:36:53 Core-Sf sSMTP[6647]: Sent mail for my.email@gmail.com (221 2.0.0 closing connection ir5sm14765639wjb.23 - gsmtp) uid=0 username=root outbytes=8678
Aug 17 00:36:54 Core-Sf mwan3: Sending email alert notification to email1@domain1,email2@domain2,email3@domain3
Aug 17 00:36:54 Core-Sf mwan3: Flushing NAT Table
Aug 17 00:36:57 Core-Sf firewall: Reloading firewall due to ifup of wan2 (eth0.3)
Aug 17 00:36:57 Core-Sf mwan3track: Network issue detected on Interface wan1 (eth0.2): 212.73.140.119 -> 100% packet loss
Aug 17 00:36:58 Core-Sf root: starting ntpclient
I believe that most of these modifications can be added to mwan3 project, especially the notification feature which can be imported in the mwan3 uci configuration file.
Download link for the the mwan3 files that were modified.
Hope, someone will find this useful.