Assembler or C, a typical MIPS-based SoC is never going to be fast for VPN. Depending on the VPN bandwidth you need and security requirements, I’d put the end point back on the mini-server, in a VM, or on another device with appropriate hardware and crypto support.
I’d also evaluate your cipher list if you stay with IPSec