Software flow offloading, OpenVPN with deliberately slow pings

I'm using a VPN (running my own OpenVPN server in a cloud VPS). Some of the VPN clients are Android mobiles. In order to keep the VPN from turning on their radios too frequently and depleting the battery, I've configured the server to push a client config that has the client pinging the server every 60 seconds. The client is also configured to time out if 120 seconds go by without a server ping being received (ping-restart option).

The server pings the client also every 60 seconds. Note OpenVPN pings, unlike ICMP, are not echoed.

The router is Netgear R7800 running 19.07.7. Until today I had software flow offloading enabled. Sporadically, and more so in the past few days, I noticed that the mobile client was hitting the ping-restart timeout and restarting the VPN. I also saw evidence that packets heading to the client were dropped somewhere along the way (basing this on behavior and notifications delayed until I forced a VPN reconnect by waking up the mobile which causes outbound traffic - so not an "exact" observation).

Anyway, today after I caught this happening "live" I started monitoring /proc/net/nf_conntrack on the router. It looked like the entries for the mobile were going into [OFFLOAD] state before the cycle would repeat (client hits ping-restart timeout, restarts VPN).

After disabling software offloading on the router I no longer see this problem. So, while this appears to be a "fix", I was trying to see if it's possible to both have and eat the cake of "offloading" and came across this thread.

Looking at the code linked there, it does unfortunately look like (software? hardware too?) offload implies a non-tunable timeout for quietly dropping these conntrack entries.

Is there some way (other than patching) to work around this? Related question, what am I practically losing by disabling software offload?