PostgreSQL Performance Plummets in Linux 7.0: AWS Engineer Sounds Alarm

An Amazon/AWS engineer has raised serious concerns about PostgreSQL performance in the upcoming Linux 7.0 kernel, reporting that database throughput has dropped to approximately half of what it was in previous kernel versions. The discovery, made by Salvatore Dipietro of Amazon/AWS, reveals that Linux 7.0 in its near-final form delivers only about 0.51x the throughput on a Graviton4 server compared to prior kernels, with significantly more time being spent in user-space spinlocks.

The Root Cause: Preemption Mode Changes

The performance regression was traced back to a specific change in Linux 7.0 that restricts the available preemption modes for the kernel. This change, previously covered on Phoronix in "Linux 7.0 To Focus Just On Full & Lazy Preemption Models For Up-To-Date CPU Archs," was part of the kernel's scheduler updates and was upstreamed as part of the Linux 7.0 development cycle.

Attempted Fix and Pushback

In response to the severity of the reported regression, a patch was posted to the Linux kernel mailing list to restore PREEMPT_NONE as the default preemption model. However, this potential fix faces significant opposition from the kernel developers who originally implemented the preemption mode simplification.

Peter Zijlstra, who authored the original code simplifying the preemption modes, has pushed back against reverting the change. Instead, he suggests that the responsibility lies with PostgreSQL to adapt to the new kernel behavior. Zijlstra recommends that PostgreSQL make use of the Restartable Sequences (RSEQ) time slice extension, which was also upstreamed for Linux 7.0.

"The fix here is to make PostgreSQL make use of rseq slice extension: https://lkml.kernel.org/r/[email protected] That should limit the exposure to lock holder preemption (unless PostgreSQL is doing seriously egregious things)."

Implications for Production Systems

This standoff between kernel developers and database maintainers has significant implications for production systems. If the kernel developers' position stands and PostgreSQL is expected to adapt rather than the kernel reverting the change, Linux 7.0 stable could lead to a substantial performance drop for PostgreSQL in certain scenarios.

The timing is particularly concerning as Linux 7.0 stable is due for release in approximately two weeks. This kernel version will also power Ubuntu 26.04 LTS, scheduled for release later in April, meaning millions of production systems could be affected by this performance regression.

Technical Deep Dive: Understanding the Impact

The performance degradation stems from how the preemption mode changes affect user-space spinlocks. In the previous kernel versions, the broader range of preemption modes allowed for more flexible handling of these locks. The Linux 7.0 change, while simplifying the kernel's preemption model for modern CPU architectures, has inadvertently created a scenario where PostgreSQL's spinlock implementation becomes significantly less efficient.

PostgreSQL's current implementation appears to be incompatible with the more restrictive preemption model, causing threads to spend excessive time waiting in user-space spinlocks. This directly translates to the observed 50% throughput reduction and increased latency that AWS engineers have documented.

The RSEQ Solution: A Path Forward?

Zijlstra's suggestion to use Restartable Sequences (RSEQ) time slice extension represents a potential solution, but it places the burden on the PostgreSQL development team to implement these changes. RSEQ is designed to handle precisely these types of scenarios where user-space code needs to be aware of kernel preemption events.

However, implementing RSEQ support in PostgreSQL is not a trivial task. It requires significant code changes and thorough testing to ensure that the database server maintains its reliability and correctness while improving performance under the new kernel model.

Broader Context: Kernel Development Philosophy

This situation highlights an ongoing tension in kernel development between maintaining backward compatibility and pushing forward with architectural improvements. The Linux kernel team's decision to potentially leave the regression in place rather than reverting the preemption changes reflects a philosophy of encouraging user-space applications to adapt to kernel improvements rather than maintaining compatibility at all costs.

For database administrators and system architects planning deployments around Linux 7.0 and Ubuntu 26.04 LTS, this development serves as a crucial warning. Performance testing should be a top priority before any production deployment, particularly for PostgreSQL-heavy workloads.

Timeline and Next Steps

With Linux 7.0 stable release imminent and Ubuntu 26.04 LTS following shortly after, the PostgreSQL community faces a compressed timeline to address this issue. The database server's development team will need to evaluate the RSEQ approach and implement necessary changes if they agree with the kernel developers' assessment.

System administrators running PostgreSQL on test systems with Linux 7.0 should begin performance benchmarking immediately to understand the impact on their specific workloads. Those planning Ubuntu 26.04 LTS deployments should factor in potential PostgreSQL performance degradation and consider whether to delay upgrades or explore alternative database solutions temporarily.

This controversy underscores the complex interdependencies in modern software stacks and the challenges of maintaining performance across kernel and user-space boundaries. As Linux 7.0 approaches its stable release, all eyes will be on how the PostgreSQL community responds to this unexpected performance challenge.