Bringing up BCM3380 - extremely slow instruction execution?

Figured it out myself. Looks like I need to turn on caching for kseg0 in CP0 config resgister, as well as in a broadcom specific config register $22 (CP0_BCM_CFG_ICSHEN | CP0_BCM_CFG_DCSHEN).