Linux Dropping SMC TCP ULP Support For Being "Fundamentally Broken"

The Linux kernel is set to remove its Shared Memory Communications (SMC) TCP Upper Layer Protocol (ULP) support in the upcoming Linux 6.20~7.0 kernel cycle, marking the end of a feature that was merged just four years ago.

The Broken Design

The TCP ULP support for SMC was originally introduced in early 2022 by an Alibaba engineer with the goal of allowing applications to transparently replace TCP with the SMC protocol. The idea was to enable in-place conversion of active TCP sockets into SMC sockets, providing a seamless migration path for legacy applications.

However, this seemingly elegant solution turned out to be "fundamentally broken." The implementation attempted to convert an active TCP socket into an SMC socket by modifying the underlying struct file, dentry, and inode in-place. This approach directly violated core VFS (Virtual File System) invariants that assume these structures are immutable for an open file.

Security and Stability Risks

The design flaw creates several serious issues:

• Use after free errors: Modifying immutable VFS structures while they're in use can lead to memory corruption
• System instability: The core file system assumptions being violated can cause unpredictable behavior
• Security vulnerabilities: Memory corruption issues often lead to exploitable security holes

Al Viro, one of Linux's most respected kernel maintainers, identified these critical issues, leading to the decision to remove the feature entirely rather than attempt to fix it.

Better Alternatives Exist

The commit message notes that cleaner alternatives already exist for achieving the same goal of application transparency:

• LD_PRELOAD: Allows intercepting and modifying library calls without changing the application
• BPF (Berkeley Packet Filter): Provides modern, safe ways to hook into and modify network behavior

These approaches don't violate core kernel invariants and provide the same or better functionality without the stability risks.

Context and Impact

SMC itself is designed for high-performance, low-latency communication between processes on the same system or within the same cluster, using shared memory as an optimization over traditional TCP. The TCP ULP support was meant to make this transition transparent for existing applications.

While the removal of this feature might disappoint some users who were planning to use it, the kernel community's decision to prioritize system stability and security over maintaining a broken implementation demonstrates the rigorous standards that govern Linux kernel development.

The revert will occur in the upcoming kernel cycle, meaning Linux 6.20 and later versions will no longer include this problematic TCP ULP support for SMC. Developers who need similar functionality will need to migrate to the recommended alternatives like BPF or LD_PRELOAD techniques.

The episode serves as a reminder that even features with good intentions and seemingly elegant designs must adhere to fundamental kernel architecture principles, and that the Linux kernel community remains committed to maintaining these critical invariants even when it means removing recently added features.