Check out ./scripts/env and set a built environment for each target. You cannot chain two separate arch together, since the buildroot is tied to the arch, AFAIK.
You can, however, setup a wrapper script to switch between environments and build from the existing .config
You can edit the targets to include the specific packages. Since the packages aren't connected across targets/toolchains, it doesn't matter. You can do this by setting the packages you want in make menuconfig to default to be baked in/modules.
I'm sure there is probably a way to do this, but it is over-complicating things if you go the ./script/env route, since you can just call make -j$(`nproc` + 1) once you switch environments via the wrapper script.