Compilation has slowed down due to Intel microcde patch for Spectre2 in Windows10 (Ubuntu in Virtualbox)

I have recently experienced severe slowdown in Openwrt compilation. I think that I have now found the culprit for my recent compiling slowness. I write this as warning for others.

I noticed that build time of my R7800 build (after make clean but using the existing toolchain, so just the plain firmware), grew from some 25 minutes (2 weeks ago) to some 80 minutes (now) in Ubuntu 18.04 in Virtualbox in Windows10 x64. In addition to Openwrt source commits, there have been both some minor updates to Ubuntu 18.04 and also the jump from VirtualBox 5.2.18 to 5.2.20. So I have wondered if this is just something caused by new Virtualbox version or some Ubuntu updates, or something in the Openwrt.

However, it is likely not Virtualbox or Ubuntu, but Windows 10 trying to be secure.

There is a new Windows 10 patch KB4100347 for Intel processor microcode to tackle the Spectre v2 vulnerability CVE 2017-5715 in Intel processors. Windows 10 installed that patch for me on 12.10.2018. The build jobs before that were quick, and the builds after that have been a nightmare.

Reverting that patch decreased the Openwrt compile time from 80+ minutes back to 25-30 minutes on Ubuntu 18.04 in Virtualbox 5.2.18.

Not sure if that is due to specifically Ubuntu-Virtualbox-Windows10 combination with my hardware, but the patch kills the compiling performance in the Ubuntu guest in my case. I will likely need to update the 4-year old Haswell motherboard & CPU to Spectre-proof new hardware versions in intermediate future (once Intel really gets the act together).

I am guessing, but likely the same microcode update will sooner or later hit the Linux distros, so this is likely not just a Windows 10 issue. (But of course the extreme slowness might be due to Windows 10 - Virtualbox - Ubuntu combination)

2 Likes

Virtualization is one of the areas hardest hit by the fixes, regardless of platform.

3 Likes

Help is on the way. The "retpoline" Windows 10 combined with the "import optimization" should help a bit.

... This is straight from the Windows/Azure Kernel Team development Manager:

1 Like

not that there wasnt enough warning before :wink:

Why virtual machine and not WSL?

Because I have been using this build method since 2010 or 2011, and WSL was not yet invented then...

I tested the bash in Windows in 2016 (see https://forum.archive.openwrt.org/viewtopic.php?id=67204&p=1#p336161), but I did not like it as at that time it forced to use the user profile dir in the main Windows drive for Openwrt. apparently that has changed a bit lately.

WSL will do in a pinch for small stuff. But is even slower than a VM.

See the performance number on Phoronix Tests from 2018-02-22:

https://www.phoronix.com/scan.php?page=article&item=wsl-february-2018&num=3

(There's a timed kernel compilation and WSL is getting wrecked by VirtualBox).

if you want more information, there's an awesome writeup over at Github from a dev:

WSL is faster if you disable Windows Defender while you're compiling. It was still slower than Virtualbox before the spectre patches, but maybe it's faster now.

True, Phoronix tested that in "Using W10Privacy To Boost Ubuntu WSL Performance On Windows 10" too, which disabled among other stuff the Windows Defender.

I'm curious, can you do some comparisons?

I'm a bit more curious why you'd use Virtualbox rather than Hyper-V or VMWare Player as from I can personally tell it's really slow and looking at various benchmarks Virtualbox pretty much always ends up last (worst performance).

Well, it works.
I have just started to use it years ago, and have never evaluated the others. I only use Linux for openwrt stuff, so it is not a major issue. A compilation time of 25 minutes (using 2 CPU cores) is quite reasonable as I can still use the PC for other stuff at the same time.

But slowdown to 80 minutes was too bad..

If you trust your VM, you can disable the mitigations in the VMs Linux kernel and in virtualbox to speed things up.

For linux see this from Ubuntu.
For virtualbox: VBoxManage modifyvm name --l1d-flush-on-sched off.

I re-tested using WSL Windows Subsystem for Linux in Windows 10 x64 1809 to build the current Openwrt master.

Some observations:

  • Ubuntu 18.04 LTS can be installed from the app store and works ok.
  • I disabled the recent Intel microcode update by the older April version of mcupdate_GenuineIntel.dll (to leave myself vulnerable to Spectre2, but be faster...). See e.g. discussion at https://www.techpowerup.com/forums/threads/after-a-windows-10-update-today-overclocking-is-lost-wtf-microsoft-and-intel.247595/page-2#post-3904080
  • In addition to the normal prerequisites, also "sudo apt-get install unzip" is needed. Just like in 2016 as explained in https://forum.archive.openwrt.org/viewtopic.php?id=67204&p=1#p336161
  • I tried using another drive (as /mnt/v ) for the build, but got prerequisite warnings about non-case-sensitive drive, so I used the main emulated Linux rootfs in the end.
  • It is nowadays possible to move the Linux rootfs from C: to another drive with 3rd-party tools (although by default the emulated Linux filesystem still gets installed into user's local data directory in C: drive). I used LxRunOffline 3.3.0 to move the already installed Linux rootfs from C: to my V: drive (another SSD that is less full...), and everything seems to work ok. Link to the tool: https://github.com/DDoSolitary/LxRunOffline
  • I needed to exclude V:/wsl directory from virus scanning to keep performance reasonable. Otherwise my F-Secure virus scanner scanned each compiler process separately (or something like that) and performance was quite weak.

I did not observe any improvement in the compilation speed compared to Ubuntu in VirtualBox, so I do not yet see much benefits in using WSL directly. (But I did not try disabling WindowsDefender like some have suggested)