A New Architecture for High-Performance Network Servers

In the landscape of network server programming, certain patterns have become so entrenched they're considered canonical. The traditional approach involves a main loop waiting for events, then dispatching based on file descriptors and their associated states. While this pattern has served the industry well, Geo Carncross argues that a superior design is possible, particularly with modern system calls like epoll and kqueue.

The conventional server architecture has evolved from the once-popular approach of forking new processes for each connection to the more contemporary use of worker threads that rely on the kernel to schedule file descriptors among them. Despite the availability of high-performance event notification mechanisms, many developers still implement them through wrappers like libevent, which merely replicate the same slow designs that have persisted for over two decades.

Carncross proposes an alternative architecture built on two fundamental principles: dedicating one thread per core (with CPU affinity) and maintaining separate epoll/kqueue file descriptors for each thread. More significantly, the design assigns different state transitions—such as accept operations and reading requests—to specialized threads. When a client needs to transition between states, the file descriptor is passed to the appropriate thread's epoll/kqueue file descriptor.

This architectural approach eliminates decision points within the hot paths, employs simple blocking I/O calls, and enables the creation of compact, high-performance servers capable of processing 100,000 requests per second on contemporary hardware.

The implementation begins with creating a thread pool that matches the number of available CPU cores, though reserving some cores for system processes may be beneficial. Each thread is configured with system scope and detached state, then pinned to a specific processor core using CPU affinity settings. For Linux systems, this involves pthread_setaffinity_np(), while on macOS, the thread affinity policy must be set using thread_policy_set() with the appropriate Mach system calls.

Once threads are initialized, each creates its own epoll (on Linux) or kqueue (on BSD/macOS) instance. The listening socket configuration follows several critical optimizations: increasing the file descriptor limit to handle the expected connection volume, disabling lingering to prevent resource exhaustion, and enabling deferred accepts on Linux when the client speaks first (as in HTTP).

The accept loop operates independently of the event notification system, focusing solely on accepting new connections. Each accepted socket is immediately configured with appropriate options—such as receive timeouts and non-blocking mode—before being added to the appropriate worker thread's epoll set or kqueue. The scheduling of connections to workers can follow a simple round-robin approach, though Carncross notes that more sophisticated load balancing based on workload characteristics can improve throughput.

The request loop, where the actual processing occurs, waits for events using epoll_wait() or kevent(). Each file descriptor is exclusively handled by a single request in a single state, allowing for simplified buffer management. When handling a file descriptor, the implementation reads input as needed and employs write() or sendfile() for output. If multiple system calls are required, the task is scheduled with a specialized worker thread.

This architecture represents a significant departure from traditional event-driven models. Instead of a single thread multiplexing many connections through complex state machines, Carncross's approach leverages modern multi-core processors by dedicating resources per core and minimizing cross-thread communication. The explicit state transitions, while seemingly more complex, actually simplify the code by eliminating the need for intricate decision logic within the event loop.

The performance claims are substantial—100,000 requests per second on modern hardware—suggesting that this approach can outperform traditional designs, especially under high load conditions. The elimination of lock contention through careful thread assignment and the use of blocking I/O calls likely contributes to these impressive numbers.

However, this architecture is not without potential drawbacks. The increased complexity in managing state transitions between threads could make the codebase harder to maintain and debug. The approach also requires careful consideration of workload distribution, as imbalanced state transitions could lead to some threads being overutilized while others remain idle.

Additionally, while this design excels at handling high volumes of simple requests, it may not be as well-suited for applications requiring complex per-connection state management or those with highly variable processing requirements. The traditional event-driven model offers more flexibility in such scenarios.

The implications of this architecture extend beyond raw performance. By simplifying the I/O path and reducing complexity in the hot code paths, Carncross's approach could lead to more maintainable server codebases. The clear separation of concerns between different state handlers also facilitates modular development and testing.

For developers working on high-performance systems, particularly those handling simple request-response patterns, this architecture warrants serious consideration. The implementation details provided, while specific to Linux and macOS, offer a blueprint that could be adapted to other Unix-like systems.

As server hardware continues to evolve with more cores and specialized processing units, architectures like this one that can effectively leverage parallel resources will become increasingly valuable. Carncross's work represents a thoughtful reconsideration of fundamental server design principles in light of modern computing capabilities.

The article provides concrete implementation examples, from thread creation to socket configuration and event handling, making it accessible to developers with intermediate to advanced C programming skills. The code snippets, while not complete, illustrate the key concepts and could serve as a starting point for those interested in implementing this architecture.

In conclusion, while not a universal solution for all server development needs, Carncross's thread-per-core architecture with explicit state transitions offers a compelling alternative to traditional event-driven designs, particularly for high-throughput, low-latency applications where raw performance is paramount.