It does not make any sense that packet loss affects only one protocol and not others. This would be a problem that is not on the network layer, but the physical layer. If the device uses RGMII as the ethernet interface it is likely that the clocks are not adjusted properly which occasionally causes the signals to be misread.
I could be wrong, but I believe packet loss in IPv4 but not IPv6 means there is a software-related problem in upstream devices, not the one with openwrt.
Now to the original problem
Even though you don't see CPU being maxed out, MT7621 is a dual-core device, so it probably will help to install the package irqbalance