Thanks a lot!!!!
This was the issue! My crl.pem file was configured to be only valid for 180 days. 
I edited the /etc/easy-rsa/vars file and raised the CRL publish time to 10 years.
set_var EASYRSA_CRL_DAYS 3650
Then ran easyrsa gen-crl in console, copied the new crl.pem file to my openvpn server config, restarted the server and now everything is working well again!
And thanks @vgaetera for pointing out how to force ipv4, so that my clients can now connect faster.