Linux 7.0-rc4 Fixes Critical Scheduler Hangs and Performance Issues on Large Systems

This week's Linux 7.0-rc4 release candidate brings critical fixes for scheduler hangs and performance regressions that have been affecting large systems since the mm/cid rewrite was merged last November. The fixes address several serious issues that could cause kernel stalls, soft lockups, and RCU stalls during normal operations.

Scheduler Fixes Address Critical Hangs

The most significant changes come from the sched/urgent pull request, which addresses multiple issues in the scheduler code. These problems were introduced during the mm/cid rewrite and have been causing hangs and performance degradation on systems with many CPUs.

Thomas Gleixner led the investigation into kernel stalls that occurred when starting VSOCK listening sockets. The investigation revealed several race conditions and bugs in the mm/cid code that could lead to soft lockups, RCU stalls, and timeouts. The fixes include:

• CID hangs due to race between concurrent forks - A race condition that could cause the system to hang during process creation
• vfork()/CLONE_VM MMCID bug - A bug in the vfork implementation that could cause hangs
• Pointless preemption guard removal - Elimination of unnecessary code that was causing overhead
• Performance regression fix on large systems - A major optimization that removes slow counting logic using for_each_process_thread() in mm_cid_*fixup_tasks_to_cpus()

The performance fix is particularly important for large systems. The original code used a known-flaky and slow counting mechanism that became a bottleneck on systems with many CPUs. The new implementation uses a simple sched_mm_cid::node list instead, dramatically improving performance on large-scale systems.

x86 Suspend-to-RAM Hang Fix

A separate but equally important fix comes from the x86/urgent pull request. This addresses a suspend-to-RAM bug that could cause hangs when firmware unexpectedly re-enables x2apic hardware during resume.

The issue occurs when resuming from s2ram - firmware may re-enable x2apic mode, which the kernel had previously disabled during boot. This creates a mismatch where the kernel continues using the xapic interface while the hardware is in x2apic mode, resulting in hangs. The fix modifies lapic_resume() to disable x2apic if the kernel expects it to be disabled, specifically when x2apic_mode = 0.

Impact on Large-Scale Deployments

These fixes are particularly relevant for large-scale deployments and high-performance computing environments. The scheduler performance regression could have caused significant slowdowns on systems with many CPUs, while the hang fixes prevent potential system crashes during normal operations.

For system administrators and developers working with Linux on large systems, these fixes should be prioritized for testing and deployment. The combination of hang prevention and performance improvements makes Linux 7.0-rc4 a critical update for production environments.

Testing and Deployment

With Linux 7.0-rc4 expected to be released later today, system administrators should begin planning for testing these fixes in their environments. The fixes address both stability and performance concerns, making this release candidate particularly important for production systems.

Organizations running Linux on large-scale systems with many CPUs should pay special attention to the scheduler fixes, as they directly address performance bottlenecks that could be affecting their workloads. Similarly, anyone using suspend-to-RAM functionality on x86 systems should test the x2apic fix to ensure reliable system resume.

The fixes demonstrate the Linux kernel development community's responsiveness to issues discovered in recent changes, particularly the mm/cid rewrite that introduced these problems. The rapid turnaround on these fixes shows the effectiveness of the kernel's development and testing processes.