It had to happen again. Anyone betting on new hardware vulnerabilities on Intel processors would have won. This time these are called the MDS vulnerabilities, which stands for Microarchitectural Data Sampling. The trouble is the ones who would have really made big money would have been those stating the new CPUs were on the same conditions or even worse as the older CPUs. And that is troubling. Luckily enough there is a good alternative on the AMD side, although only time will be the best judge, since it seems SMT is a difficult game to play.
To summarize the whole issue, these vulnerabilities are similar to Spectre and Meltdown, and as the OpenBSD guys suggested on August 2018 the only safe way is disabling Intel’s Hyper-Threading (Intel’s commercial name for SMT) altogether. This seems a radical approach but the only one to declare these issues are solved on your boxes. Something the world is not ready for, definitely.
If you find the articles in Adminbyaccident.com useful to you, please consider making a donation.
Use this link to get $200 credit at DigitalOcean and support Adminbyaccident.com costs.
Get $100 credit for free at Vultr using this link and support Adminbyaccident.com costs.
Mind Vultr supports FreeBSD on their VPS offer.
The vulnerabilities are very well explained in detail in the original papers (RIDL and Fallout). Basically, as it already happened with Spectre and Meltdown (my particular comment here), researches have found ways for a process in their control to read from the CPU caches, where information relative to other processes is temporarily stored by the CPU. Not only that, but modern CPUs do speculate, since many of their usual operations is repetitive (80%), and thus predictable, so some instructions are passed through the CPU pipeline even if they are not needed by the original process. If that calculation is not needed the CPU discards it but if it’s needed, this saves time. Modern CPU designs heavily rely on this to be fast and achieve the levels of parallelism we see.
The issue is not the technique itself but the fact there are not sufficient security measures on the Intel’s implementation of SMT, called Hyper-Threading. The information stored in those caches is not flushed out, it’s simple overwritten by the next processed instruction. The problem is what to look for, how to look for, how to measure, and critically where to look for. Something researchers have found out. Luckily for everyone the operating systems designers and Intel have coded mitigations that do not disable Hyper-Threading completely and do what the CPU should do by itself, which is flush the temporary stores and caches of the CPU at the right time.
Again, as the article on how to mitigate Spectre and Meltdown on FreeBSD or the more tedious how to patch Spectre and Meltdown the ROM way one, what is fundamentally needed is a microcode update. Some manufacturers may include some other bits if you choose to update the firmware of the motherboard (BIOS or UEFI).
Let’s go hands on this how to mitigate the MDS vulnerabilities of Intel processors in FreeBSD. Again, as it’s been already mentioned to solve the issue completely just disable Hyper-Threading in the BIOS/UEFI, but be aware of the big performance impact this may have.
Resources we’d need:
-
-
-
-
FreeBSD’s security advisory.
-
Intel’s microcode guide.
-
The guide you are reading.
-
-
-
First, we will update our system to the latest version. I am afraid you already know how to do this, but if not, the FreeBSD handbook will help in that regard in the 23.2.2 chapter. I’d also recommend to update by using beadm, which leverages the ZFS file system you are already probably using.
Second in line, is knowing what level of microcode we are in. To find this out we also need to know what specific CPU family and iteration we are running on. There is a nice package called x86info that will help us on that particular.
albert@HP:~ % sudo pkg install x86info
Updating FreeBSD repository catalog…
FreeBSD repository is up to date.
All repositories are up to date.
The following 1 package(s) will be affected (of 0 checked):
New packages to be INSTALLED:
x86info: 1.31.s02
Number of packages to be installed: 1
60 KiB to be downloaded.
Proceed with this action? [y/N]: y
[1/1] Fetching x86info-1.31.s02.txz: 100% 60 KiB 61.6kB/s 00:01
Checking integrity… done (0 conflicting)
[1/1] Installing x86info-1.31.s02…
[1/1] Extracting x86info-1.31.s02: 100%
Message from x86info-1.31.s02:
===> NOTICE:
The x86info port currently does not have a maintainer. As a result, it is
more likely to have unresolved issues, not be up-to-date, or even be removed in
the future. To volunteer to maintain this port, please create an issue at:
https://bugs.freebsd.org/bugzilla
More information about port maintainership is available at:
https://www.freebsd.org/doc/en/articles/contributing/ports-contributing.html#maintain-port
albert@HP:~ %
In order to use this tool we need to load a kernel module called cpuctl. We can do this temporarily or permanently. If we just want to get this on for the time being just issue the following command:
sudo kldload cpuctl
If you want to make this permanent you will need to edit the /boot/loader.conf file and add the following line:
cpuctl_enable=”YES”
To get this working if you’ve chosen the permanent way you will either have to reboot or better yet, use the temporarily method.
In order to check the module is correctly loaded we’ll just launch the following command:
sudo kldstat | grep cpuctl
We should get something similar to:
albert@HP:~ % sudo kldstat | grep cpuctl
32 1 0xffffffff82908000 31f7 cpuctl.ko
albert@HP:~ %
Now we know everything is in place we will use the x86info program to know what microcode we are using.
[albert@HP ~]$ sudo x86info -a | grep Microcode
Microcode version: 0x000000000000002d
[albert@HP ~]$
So, the microcode number this CPU is running on is 0x000000000000002d. Yours will be different. Write it down. Of course, at this point, it is wise to check if your CPU is Hyper-Threading capable or not. I’d patch anyway even if the CPU is not capable, since it may have the physical facilities but not operational because of the specific firmware of that specific CPU. Call me paranoid.
For this specific article I will be patching another hardware box, an Intel i5-4590 CPU with microcode as follows:
[albert@HP ~]$ sudo x86info -a | grep CPU
Found 4 identical CPUs
CPU Model (x86info’s best guess): Xeon E3 [Haswell]
Processor name string (BIOS programmed): Intel(R) Core(TM) i5-4590 CPU @ 3.30GHz
[albert@HP ~]$
[albert@HP ~]$ sudo x86info -a | grep Microcode
Microcode version: 0x0000000000000025
[albert@HP ~]$
This is now the best time to grab Intel’s microcode guide. I paste the link here again so you don’t need to scroll up. You must now look for your CPU and family. And on that line, you will see at the end, the column of pre-mitigation figure, and the last column where you will see the mitigation reference number.
If you didn’t permanently leave the configuration in place when you mitigated Spectre and Meltdown as this guide explained, you will see your microcode number of your CPU is on the pre-mitigation column. However, if you left it, it’s quite easy the latest microcode update is already in the CPU after a regular update.
If you didn’t and you are now patching, be aware though, at the time of writing this article, the devcpu-data package is in version 1.21, from January 2019, and if you need to patch the MDS vulnerabilities described in this article you may need to build de package yourself using ports, or just wait a few days, when the official package is build (usually just a few days after the latest source has been released).
We’ll be building the package here. So first, we’ll update our ports tree.
sudo portsnap fetch
Then extract the ports tree.
sudo portsnap extract
Because it’s easy you still have an old port installed remove it.
[albert@HP /usr/ports/sysutils/devcpu-data]$ sudo make deinstall
===> Deinstalling for devcpu-data
===> Deinstalling devcpu-data-1.20
Updating database digests format: 100%
Checking integrity… done (0 conflicting)
Deinstallation has been requested for the following 1 packages (of 0 packages in the universe):
Installed packages to be REMOVED:
devcpu-data-1.20
Number of packages to be removed: 1
The operation will free 3 MiB.
[1/1] Deinstalling devcpu-data-1.20…
[1/1] Deleting files for devcpu-data-1.20: 100%
[albert@HP ~]$
Once the old package has been removed we can build the new one.
[albert@HP /usr/ports/sysutils/devcpu-data]$ sudo make install clean BATCH=YES
===> Installing for devcpu-data-1.22
===> Checking if devcpu-data is already installed
===> Registering installation for devcpu-data-1.22
Installing devcpu-data-1.22…
Installing this port will allow host startup to update the CPU microcode on
a FreeBSD system automatically. There are two methods for updating CPU
microcode: the first methods loads and applies the update before the kernel
begins booting, and the second method loads and applies updates using an
rc script. The first method is preferred, but is currently only supported
on Intel i386 and amd64 processors running FreeBSD 12.0. It is safe to
enable both methods.
The first method ensures that any CPU features introduced by a microcode
update are visible to the kernel. In other words, the update is loaded
before the kernel performs CPU feature detection.
To enable updates using the first method, add the following lines to
the system’s /boot/loader.conf:
cpu_microcode_load=”YES”
cpu_microcode_name=”/boot/firmware/intel-ucode.bin”
This method will not load the microcode update until the system is
rebooted.
To enable updates using the second method, add the following line to
the system’s /etc/rc.conf:
microcode_update_enable=”YES”
Then, to ensure the update is applied, reboot the system or start the
microcode update service via:
# service microcode_update start
If the CPU requires a microcode update, a console message such as the following
will appear:
Updating CPU Microcode…
/usr/local/share/cpucontrol/m32306c3_00000022.fw: updating cpu /dev/cpuctl0 from rev 0x17 to rev 0x22… done.
/usr/local/share/cpucontrol/m32306c3_00000022.fw: updating cpu /dev/cpuctl2 from rev 0x17 to rev 0x22… done.
/usr/local/share/cpucontrol/m32306c3_00000022.fw: updating cpu /dev/cpuctl4 from rev 0x17 to rev 0x22… done.
/usr/local/share/cpucontrol/m32306c3_00000022.fw: updating cpu /dev/cpuctl6 from rev 0x17 to rev 0x22… done.
Done.
===> Cleaning for devcpu-data-1.22
[albert@HP /usr/ports/sysutils/devcpu-data]$
We’ve just built the package needed to updated the microcode in the CPU. As you can read in the installation instructions there is also a clear path of what to do. There are two methods though.
The first one tells us to add the following two lines into the /boot/loader.conf file which will load the microcode and the updates automatically after a reboot and before the kernel makes any CPU feature detection.
cpu_microcode_load=”YES”
cpu_microcode_name=”/boot/firmware/intel-ucode.bin”
The second method tell us to add the following line to the /etc/rc.conf file:
microcode_update_enable=”YES”
To apply the update we can either reboot the machine or start the service as follows:
[albert@HP ~]$ sudo service microcode_update start
Updating CPU Microcode…
Done.
[albert@HP ~]$
And now it is time to check if there’s been a microcode update. We do so by using again the x86info program.
[albert@HP ~]$ sudo x86info -a | grep Microcode
Microcode version: 0x0000000000000027
[albert@HP ~]$
And indeed, there’s been a change from version 0x0000000000000025 to the newest with the mitigations in place numbered 0x0000000000000027. Which happily coincides what corresponds to Haswell processors in the Intel’s guide.
Again, if you do not see any changes, make sure you are using the correct version of the devcpu-data package, which for mitigating the MDS vulnerabilities, should be 1.22 (May 14th 2019). To check it out for future releases, you can use Dan Langille’s Freshports page, the best source for the latest packages in FreeBSD land. The command giving you the actual version your system may be running is this:
[albert@HP ~]$ pkg info | grep devcpu-data
devcpu-data-1.22 Intel and AMD CPUs microcode updates
[albert@HP ~]$
This is all in this how to mitigate the MDS vulnerabilities of Intel processors in FreeBSD. However do not think AMD hasn’t got issues. They have claimed to be immune to the MDS vulnerabilities, but they had their issues with some specific of their own. The way those were presented to the world wasn’t the best, nor AMD seemed to have dealt with the issue with their best hand. That said both manufacturers, specially Intel, are now under good scrutiny and it is a pity Intel has failed in their new generation of CPU to solve the side-channel issues and are even worse now. Happily for consumers AMD has now a more than decent alternatives, and very competitive in not just price but performance too.
Disclaimer: The microcode numbers and CPU’s model may not coincide in some bits this article because I am not running FreeBSD on the HP ML110 G7 box anymore. I borrowed the old microcode from the past article for the first part. Later I’ve described the process for a desktop box I’ve worked with for this article.
If you find the articles in Adminbyaccident.com useful to you, please consider making a donation.
Use this link to get $200 credit at DigitalOcean and support Adminbyaccident.com costs.
Get $100 credit for free at Vultr using this link and support Adminbyaccident.com costs.
Mind Vultr supports FreeBSD on their VPS offer.