The Fil-C Optimized Calling Convention: Balancing Safety and Performance in Memory-Safe Systems

In the landscape of memory-safe programming systems, a fundamental tension exists between providing strong safety guarantees and achieving performance comparable to unsafe languages like C. The Fil-C Optimized Calling Convention represents an ambitious attempt to resolve this tension, creating a system that maintains robust safety even in the face of adversarial program behavior while delivering performance that approaches that of traditional calling conventions.

The Challenge of Memory-Safe Calling Conventions

Memory-safe systems face a significant challenge when implementing function calls. Unlike traditional languages where function calls are relatively simple operations, memory-safe systems must validate numerous aspects of each call to prevent memory unsafety. These validations include:

• Ensuring function pointers point to valid function capabilities
• Verifying that the correct number of arguments are passed
• Checking that arguments match the expected types
• Handling variadic arguments safely
• Managing return values properly

In the naive implementation, these checks can impose substantial overhead, potentially negating many of the performance benefits of memory safety. Fil-C addresses this challenge through a multi-layered approach that optimizes the common case while maintaining safety guarantees for edge cases.

The Fil-C Approach: Function Objects and Dual Entry Points

At the heart of Fil-C's calling convention is the concept of a "function object" – a rich structure that contains multiple entry points and metadata about the function. Each function object in Fil-C contains several key components:

1. fast_entrypoint: A raw pointer to an optimized entrypoint that uses a register-based calling convention
2. generic_entrypoint: A pointer to a fallback entrypoint using thread-local CC buffers
3. signature: A 64-bit arithmetic encoding of the function signature
4. data_ptr (for closures): A user-controlled flight pointer

This design allows Fil-C to have the best of both worlds: an optimized fast path for the common case where signatures match, and a fallback mechanism for when they don't.

The Register Calling Convention Optimization

The first major optimization Fil-C employs is the register calling convention, which allows arguments and return values to be passed in registers in the common case, similar to native calling conventions. This is achieved through several mechanisms:

Signature Matching and Thunks

When making a function call, Fil-C first checks if the caller's and callee's signatures match by comparing their arithmetic encodings. If they match, the call can proceed directly through the fast_entrypoint with arguments passed in registers. If they don't match, Fil-C employs a pair of thunks to translate between calling conventions:

• Caller entrypoint thunk: Converts from the fast calling convention to the generic calling convention
• Callee entrypoint thunk: Converts from the generic calling convention to the fast calling convention

These thunks are generated as weak symbols in ELF, ensuring that only one copy is linked in even if multiple modules define them.

Arithmetic Encoding of Signatures

A critical innovation in Fil-C is its arithmetic encoding of function signatures. This encoding allows any function signature (within reasonable limits) to be represented as a 64-bit integer. The encoding handles:

• Return values (0-2 values)
• Arguments (0-16 values)
• Various types (integers, floats, doubles, vectors, pointers)

The encoding is designed to be both compact and efficient to compute, allowing for quick signature comparisons at call sites. For example, the signature char* (*)(int, char*, double) is encoded as 60125.

This encoding is particularly clever in how it handles sequences of types, using a mathematical approach that allows representing both the sequence and its length efficiently.

Direct Call Optimizations

The second major optimization Fil-C employs is avoiding direct caller resolution through getter calls and capability checks. This optimization is more complex due to ELF linking and loading semantics, but it provides significant performance benefits for direct calls.

Symbol Mangling and Weak Symbols

Fil-C employs a sophisticated symbol mangling scheme that includes the function's signature in its exported symbols. When a function is defined, Fil-C exports two symbols:

• pizlonatedFI<signature>_functionname: The callsite interface
• pizlonatedFIP<signature>_functionname: The implementation

For strongly defined functions, Fil-C creates a strong alias from the interface symbol to the implementation symbol. This allows direct calls to bypass the getter and capability checks when signatures match.

Handling Weak and COMDAT Symbols

A significant challenge with this approach is handling weak symbols and COMDAT groups, particularly for C++ inline functions. Fil-C addresses this by:

1. Emitting null checks for direct calls to locally defined but weak symbols
2. Modifying LLVM to understand that locally defined symbols with COMDAT may be NULL
3. Using different relocation types for calls versus null checks to catch potential issues at link time

These modifications ensure that the optimization works correctly even with complex C++ linking scenarios while maintaining safety.

Performance Implications

The combination of these optimizations provides substantial performance benefits. According to the article, both the register calling convention and direct call optimizations each provide more than a 1% speed-up on PizBench9011. More importantly, they make the common case of function calls in Fil-C almost as efficient as calls in "Yolo-C" (presumably a traditional unsafe C implementation).

The key insight is that Fil-C achieves this performance without compromising its memory safety guarantees. Even in the optimized path, Fil-C maintains safety through:

• Capability checks on function pointers
• Signature validation
• Proper handling of variadic arguments
• Safe management of return values

Broader Implications for Memory-Safe Systems

The Fil-C calling convention represents several important advances in memory-safe systems:

1. Practical Performance: By optimizing the common case while maintaining safety guarantees, Fil-C demonstrates that memory safety doesn't have to come at a prohibitive performance cost.

2. Sophisticated ABI Design: The dual-entry-point approach and arithmetic encoding demonstrate the value of thoughtful ABI design in balancing safety and performance.

3. Integration with Existing Ecosystems: The careful handling of ELF linking and C++ integration shows that memory-safe systems can work within existing ecosystems rather than requiring entirely new toolchains.

4. Gradual Safety: The approach allows for different levels of optimization based on the specific use case, with more thorough safety checks when needed and optimized paths when the programmer is "behaving themselves."

Conclusion

The Fil-C Optimized Calling Convention represents a significant step forward in memory-safe systems programming. By combining sophisticated optimizations with rigorous safety guarantees, Fil-C demonstrates that it's possible to have the best of both worlds: the safety guarantees of memory-safe systems with the performance characteristics of traditional unsafe languages.

The innovations in Fil-C – particularly the arithmetic encoding of signatures and the dual-entry-point system – provide a blueprint for other memory-safe systems looking to improve performance. As systems programming continues to grapple with the challenges of memory safety, approaches like Fil-C's will become increasingly important in bridging the gap between safety and performance.

The work also highlights the importance of deep systems knowledge in designing memory-safe systems. The intricate handling of ELF linking, symbol resolution, and calling conventions demonstrates that effective memory safety requires understanding not just high-level abstractions, but also the low-level details of how programs are executed.

As we move toward a future where memory safety becomes increasingly important, the Fil-C Optimized Calling Convention offers a compelling vision of what's possible when safety and performance are treated as complementary rather than competing goals.