Vertica Test Results for Operating System Patches for Meltdown and Spectre Security Flaws

[Deleted User]

Vertica engineers have run performance tests using the operating system patches for the Meltdown and Spectre security flaws. Based on the results, Vertica recommends that, for minimal performance impact and addressing most security issues, customers enable the PTI and IBPB features but not the IBRS feature.

Earlier this month, researchers announced two speculative execution security vulnerabilities in chips that are present in most modern processors.

• Meltdown: Meltdown allows multiple processes on a processor to access the contents of another process's memory. This vulnerability has only be verified on Intel processors: CVE-2017-5754.
• Spectre: Spectre affects processors that implement branch prediction and speculative execution. This vulnerability may allow processes to read and modify the data cache.
  Two variants of Spectre has been verified on several modern processors: Variant 1 (CVE-2017-5753) and Variant 2 (CVE-2017-5715). Variant 1 is fixed with a kernel patch. You cannot disable this patch and there is no measureable performance impact from this patch.

There are three optional patches that ship in current hotfixes of most Linux distributions. They install the following features:

• KPTI (Kernel Page Table Isolation) protects against Meltdown. Implemented in the kernel.
• IBRS (Indirect Branch Restricted Speculation) protects against Spectre Variant 2. Implemented in the kernel and CPU microcode.
• IBPB (Indirect Branch Prediction Barrier) protects against Spectre Variant 2. Implemented in the kernel and CPU microcode.

For more detailed information about these patches, see https://access.redhat.com/articles/3311301.

Vertica engineers ran a select few queries from the industry-standard TPC-H database benchmark using the following hardware configuration:

A cluster of 4 HP DL360 Gen9 servers configured with two Xeon E5-2698 v3 processors that use the Haswell microarchitecture. Each processor has 16 cores and 256GiB of memory, running the latest RHEL 6.9 kernel, version 2.6.32-696.18.7.el6.

Vertica engineers measured query performance of all three features enabled, and in all other configurations of the patches. The results are summarized as follows:

The following graph shows, for each TPC-H query, the detailed performance measurements. The graph shows the percentage increase in execution time compared to the execution time if all the features are disabled. The baseline for these tests is all three features disabled: PTI = 0, IBRS = 0, IBPB = 0.

Your results will vary depending on microarchitecture. Unfortunately, due to the way that IBRS impacts branch prediction, it is unlikely that Vertica will be able to make changes to the Vertica code base that can reduce the impact of these patches. Vertica engineers will continue to investigate all possibilities for minimizing the impact of these hardware vulnerabilities for our customers.

FAQs
Here are the answers to some questions you might have.

Q: Why didn’t you run more TPC-H queries?
A: Vertica engineers focused on testing more scenarios rather than testing more queries. We will continue internal testing, but we wanted to publish guidance as soon as possible so that our customers can start making decisions about how to deploy these security fixes.

Q: Which Intel CPUs have the worst performance degradations?
A: We don’t know for certain yet, and Intel isn’t saying. Our best understanding is that pre-Skylake is badly impacted by IBRS, post-Skylake somewhat less so, and 8th generation core processors the least.

Q: Did you run the test with the microcode installed?
A: Yes

Q: What configuration does Vertica recommend?
A: Vertica has found that PTI=1, IBRS=0, IBPB=1 is the setting with the least performance impact while still addressing some of the security issues. You can read about the security benefits of each setting on Red Hat’s site here: https://access.redhat.com/articles/3311301

Q: Can you characterize the types of operations that are most impacted?
A: Due to the nature of this change, there is not a single class of operations that can be avoided or that reduce the performance impact of the patches. We see a performance impact, in particular for IBRS, even in fundamental operations such as a simple loop or conditional statement. These are essential building blocks that cannot be optimized away.

Q: Do you plan to build Vertica using retpoline? (retpoline is a technique for compiling programs with built-in mitigations against Spectre; it’s a partial alternative to IBRS.)
A: Not in the immediate future. To implement this, every other program on your system will also need to be rebuilt.

Q: What does the performance look like on AMD?
A: Vertica engineers have not run any tests on AMD hardware yet.

ScottL

Does Intel's latest direction to not patch ("We recommend that OEMs, cloud service providers, system manufacturers, software vendors and end users stop deployment of current versions, as they may introduce higher than expected reboots and other unpredictable system behavior.") apply to the LInux patches that Vertica support has tested? I would assume we should not apply any patches until everything shakes out?

https://newsroom.intel.com/news/root-cause-of-reboot-issue-identified-updated-guidance-for-customers-and-partners/

mleuchtenburg

Hi Scott,

Yes, they're referring to the microcode needed to support IBRS and IBPB, which we tested. We've seen some otherwise unexplained instability on AWS systems which went away once Amazon changed how they were deploying this patch, due to the instability mentioned there. KPTI should be safe to deploy as it has no associated microcode change or instability.

Vertica will re-test once the new microcode is released.