There is no need to start from scratch. Usually the toolchain is quite usable.
I usually do "make clean" to clean the previous build. That leaves the build tools and toolchain intact. A "make" after that compiles the firmware itself from srcatch, but much faster as the toolchain compilation is the slow part. (Sure, you need to update the sources before the make, "git pull" for the main sources and feeds update for the feeds, but the toolchain is already there.)
When tools/ or the key toolchain components (like musl, gcc, etc.) change, you may need to use "make dirclean" to cleanup also tools & toolchain. I need to do that rarely, maybe once per 2-3 months.
One key part to long-term sanity is to make sure that your .config does not collect unnecessary garbage, as the dependencies of the core packages may change. I use a recipe approach. See example in