I'm not exactly sure what to look for. I checked the system logs in OpenWrt but haven't found anything suspicious.
The issue seems similar to this one, but I'm using more recent modem Compal CH7465VF. I've tried to change the cable that is connected to WAN port on my router and also tried different port on the modem. No effect.
I didn't have this issue before I had the router (I was using the modem in router mode). I also think the problem wasn't present when I used the Asus router for about a week with AsusWrt. I've tried to go back to it from OpenWrt today to confirm, but wasn't able to (router firmware restoration failed). So it seems I'm stuck with the router running OpenWrt with this issue or I could go back to using ISP modem in router mode again.