In a previous article I briefly, sort of, talked about the Spectre and Meltdown vulnerabilities. I have also written two guides to patch them from the OS side using a UNIX flavour from the BSD camp and a GNU/Linux distribution. Both actions resulted successful but there is a third way to patch this vulnerabilities. Regular OS patching should solve the issues these collection of vulnerabilities since OS vendors have released the microcode produced by Intel necessary to fix the hardware problems. However there is a third way to do this and it’s done by replacing the ROM embedded in the hardware in use. This is usually accomplished by updating the BIOS from the vendor which usually maintain and give updates for a variable amount of time.
So here it comes the how to patch Spectre and Meltdown the ROM way article. I am the owner of an HP Proliant ML110 G7 server equipped with an Intel Xeon E3-1220 v3 CPU affected by these two sets of vulnerabilities and still receiving support from HP. This line of servers is still in use and Proliants are used around the world by many different kinds of companies. Although I’d have preferred to build a Supermicro box I have to recognize HP does build very attractive boxes, well designed for a very reasonable price, hence the purchase back in the day.
For big deployments with dozens or hunders of boxes this is done through the PXE specification. Here you can read about one example. You basically create a new ROM image (well, the hardware vendor makes it) and distribute it using the PXE channel to the devices in your network/s. You boot the machines through the network interfaces, load the new ROM, execute the flashing of it and hopefully if everything went right you’re done. That is fancy and fantastic for big houses with manpower, experts and big pockets. But… what about you the aficionado or the pretender (as me)?
After several months of reading news about the issues here in place and having seen major disasters with machines rebooting on its own, and several other drawbacks, a new set of good patches and microcode releases have appeared. And since most of them seem to work well I thought it was good time to patch. Or good time to start patching because as promised in the beginning of the year more than a dozen vulnerabilities would be disclosed throughout the year. Since this machine is not facing the internet and it’s sitting on my internal network I have been “lazy” to patch it. However if this box would have been giving service to the internet and to real external users I’d have patched much earlier and even more cautiously.
The hands on, manual approach is needed in order to patch Spectre and Meltdown the ROM way if you happen to have one or less than a dozen boxes. There are several reasons to patch in this fashion instead of just updating the OS. But the main one is the new BIOS image has some improvements other than solving these vulnerabilities. It may contain new features or also patch older not so important issues. It may even contain a similar feature set but with improved stability which is of course always welcome.
To do this one needs some tools. First comes the necessary information. That is the support page from your vendor where they have put the patches on, a page you can try to reach from this vendor list, as well as Intel’s microcode release guide. If you open the guide you will see two columns on the right. One which explains what microcode version is still not patched and the one on the very right the the microcode version with the patches in place. This guide may not be that useful since the release notes from the vendor support page may suffice to acknowledge the fixes and the vulnerabilities solved in that particular BIOS release. However it can be helpful after having applied the patch Spectre and Meltdown the ROM way or the OS way too.
Second set of things is a currently supported device from a hardware vendor, that is your computer, server, whatever. And a USB stick. I would recommend a third device, a laptop, a second computer or even a smartphone in order to follow instructions, download the patch, write the new ROM into the USB stick and so on.
And thirdly, take a ton of patience and the latest version of the book of insults and swearing. I am just kidding, I know you already have those.
Before continuing. This guide is meant for illustrative purposes and your enjoyment. If you brick your device (this means malfunctioning behavior, damage, or total disaster) because you have followed the text and illustrations here appearing I will feel very sorry for you but it is not my responsibility. You are acting on your own and your own behalf. Shit happens as you will soon discover by reading the below writing.
For my device the needed support page is this. I have filtered out the results from the menu on the left to find out the BIOS download files. And I have downloaded the Rompaq package. Unfortunately enough there is an .exe file which is for Windows devices and another download ROM image for Linux devices with an .rpm file. This means that in HP heads they see the world in Microsoft Windows, SuSE and Red Hat, and nothing else. Which leaves the BSD users or the Debian based distributions users such as Ubuntu out of help. Yes, you can have a virtual machine running any of those operating systems and do the thing there but as you will see a bit later this is proposterous.
Since I have a couple of extra disks I installed Windows on the machine in order to run the hideous .exe file. Any machine you’re not using with a spare disk may do the work.
I always panic when I see this kind of message as a result of twenty something years of bad, awful, miserable Windows releases. I like it now though but I can’t resist myself to make fun of it.
The goal here is simple. Running the bloody downloaded package containing the new ROM. Little surprise when I doubled clicked on it and a dialog appeared to install the thing.
After installing a web page appeared so I could browse the different solutions in hand.
I chose to create a booting USB stick and I conveniently plugged in one USB into my device.
Then after clicking on the thing a number of files were placed into the USB and I was ready to reboot the server and boot from the recently created boot USB.
And it didn’t work. Oh! Surprise, surprise. Time to swear and pray to the Gods.
Why didn’t it work? The answer is simple. The solution is broken. Why? Did you expect to patch Spectre and Meltdown the ROM way without a hassle? Poor you (read me instead).
So there I was with an unfixed machine, an unusable boot stick, having lost a couple of hours and a frown in my face. A search engine query after (with the wording: “ROMPaq Firmware command or filename not recognized”) I discovered this is not hot news. This guy had the same issue six years ago and this other guy too. In a very big corporate fashion HP has not fixed the problem. So I followed these guys advice instead of allegedly reliable official support. When the executable installs it basically writes a series of files in your pc. There is a folder where the necessary files to make the bootable USB land. The problem comes because the tool just copies them partially forgetting a set of mandatory files including the precious new ROM image. Amazingly enough it also copies what you happened to have in your Downloads folder. Hurrah! Imagine you create a bootable USB disk with all the shit you’ve downloaded the latest months. Hilarious. Until you are one of the minions in charge of patching hundreds of boxes, you are doing the testing thing and this just takes a loooong time to burn the image on the USB. Then it doesn’t work in your test machine. You start wondering… what the heck?
Now. All this may look like Voodoo magic but is more simple than you imagine. What the tool writes into the USB stick is a series of files among them the new ROM image. In order to boot the computer and load this ROM there’s a need of a small OS. They have borrowed some booting files and utilities from FreeDOS. Yes. DOS. The free as in freedom version. If you look at the files you will find familiar names like command.com and autoexec.bat.
Here you can get a screenshot of what it took the first time I did the process (unnecessary files too).
My question is why on Earth do you need to pack all this, which is light, free, and simple into a binary like an .exe or .rpm file instead of grabbing them all and make a .zip, or .tar.gz file with some simple instructions like:
Step one. Insert the USB on your computer and format it to FAT32.
Second step. Uncompress the fixfilename.zip content into the USB by right-clicking onto the file.
Third step. Check these files have been copied into the USB (list of the files).
Fourth step. Safely detach the USB.
This would work across all platforms with browser support. Maybe there is a reason to distribute ROMs in this convoluted, unnecessary overworked way with .exe binaries, installing things and so on. Maybe I am missing something. But boy this is too much at my eyes. Get the file, decompress it. Grab the content and put into a USB stick and off you go looks much more simple and easy for literally everyone (even computer illiterates).
So, I did as as I say. I copied all the files into the stick (I also copied them into a folder in my main workstation, just in case I need them in the future and booted the HP Proliant server from the USB stick. A good old DOS prompt appeared and told me to wait a bit.
An HP splash screen warned me about what I was about to do (flashing the BIOS with a new ROM).
I proceeded and when it was finished I got the message to press the power button to powercycle the device. And so I did.
Here you can see the new BIOS. I blurred the serial and so on.
Lastly I checked from an Ubuntu install the microcode version and compared it with the Intel’s guide to see if I was on the good track. And indeed I was.
To check the microcode version you can launch the following command on a GNU/Linux box.
cat /proc/cpuinfo | grep microcode
For a UNIX like FreeBSD check out this guide of mine.
And finally as you can see I was successful in the patch Spectre and Meltdown the ROM way thing. A simple process for what should have been a short article filled with meaningless phrases from my sometimes convoluted way of writing has ended up being more complex and longer, much longer than necessary.
I hope you can patch Spectre and Meltdown the ROM way more easily than me.
I will also write to HP, and I invite you to do so if you experience the same issues, in order to inform them about their malfunctioning software and my/our discontent. This could mean I could buy a Supermicro box in the future instead of an HP. Could a big enterprise do the same because their staff is in the same position when distributing the new ROM through PXE?
The email will surely contain a list of issues found and good manners.
Limited download options. Debian and derivatives users are left out. All BSDs users are out. Illumos based distros users are out. Solaris is also out. I do not count Plan 9 or MINIX because they are intended to research not production use.
The program fails to copy the necessary files into the USB stick. This has happened for years and is still unfixed.
This leads to a failed boot and a frown face probably with a call to customer support where they have no idea what they are talking about.
The tool copies all the files from the Downloads folder, needed or not for the USB boot device creation. This could lead in an information disclosure, embarrassing faces and long writing times.
I hope you have enjoyed the reading and successfully patched all your systems.