Found the fix, their is a mismatch in mainline vs downstream uboot that causes its clock "fix up" based on the kernels DTB set the wrong clocks for (at least) GPLL and CPLL.
Fixing this the RGMII interface now works as intended.
I have also ported the rockchip loop-back tuning code to 6.1