Latency: The Race to Zero...Are We There Yet? - InfoQ

Amir Langer discusses the evolution of latency reduction, from the Pony Express to modern hardware. He explains how separation of concerns - decoupling business logic from I/O - and tools like Aeron and the Disruptor achieve single-digit microsecond speeds. He shares insights into replicated state machines, consensus protocols like Raft, and the future of low-latency sequencer architectures.

Why Latency Matters

In the fintech industry, latency directly translates to profit and competitive advantage. When a trading system has lower latency than competitors, it can execute better deals and capture more favorable prices. The market makers with the lowest latency consistently achieve the smallest spreads because they can react faster to market changes.

Low latency also matters for recovery scenarios. When systems recover from failures, they become temporarily unresponsive - essentially unavailable. Minimizing recovery time is crucial for maintaining system availability and reliability.

Historical Context: Lessons from the Past

The evolution of latency reduction spans centuries. The Roman Empire's cursus publicus established an early system of roads and relay stations that dramatically reduced message delivery times compared to private couriers. This system of horses, wagons, and replacement points along routes enabled faster communication across the empire.

Samuel Morse's invention of the electrical telegraph was directly motivated by latency concerns. After missing a critical message about his wife's unexpected death while traveling, Morse dedicated himself to creating a communication system that would prevent others from experiencing similar tragedies.

The Pony Express represents a remarkable achievement in latency reduction. They reduced message delivery time from Missouri to California from 3-5 weeks down to just 10 days by implementing a system of horses and riders with replacement points along the route. However, this success story ended abruptly when the telegraph reached the west coast in 1861, rendering the Pony Express obsolete after only 18 months of operation.

Modern Challenges in Latency Reduction

Unlike previous decades when hardware upgrades provided immediate latency improvements, modern systems face more complex challenges. Processor designs have become increasingly sophisticated with multiple caching layers and shared memory architectures. While throughput continues to increase, latency has become the primary bottleneck.

The cloud introduces additional complexity with multiple layers of abstraction that add overhead. Distributed systems operate at unprecedented scales where communication between components, rather than CPU processing, becomes the limiting factor for latency.

Separation of Concerns: The Key to Low Latency

In 2010, the LMAX Disruptor project demonstrated how separation of concerns could achieve dramatic latency improvements. By decoupling different work streams - journaling messages separately from decoding and business logic processing - each thread could focus on a single task without interruptions or waiting.

This architectural approach allows threads to operate independently, eliminating the latency introduced by context switching and synchronization overhead. Each component handles its specific responsibility efficiently without being blocked by other operations.

Modern Tools for Low Latency

Aeron, open-sourced in 2014, provides efficient, low-latency, and reliable message transmission between processes. It supports UDP unicast and multicast, as well as inter-process communication (IPC) using shared memory. The project demonstrates that Java can achieve performance comparable to C implementations when properly optimized.

DPDK (Data Plane Development Kit) enables kernel bypass, allowing direct communication with network interface cards without going through operating system layers. This approach can achieve single-digit microsecond latencies by eliminating the overhead of socket abstractions and kernel transitions.

Replicated State Machines and Consensus

The fundamental building block for distributed low-latency systems is the replicated state machine. This simple model processes input events deterministically, modifying internal state and generating output events. The determinism ensures that all replicas processing the same input sequence will reach identical states.

Virtual synchrony and consensus protocols like Raft provide the foundation for fault-tolerant distributed systems. These approaches ensure that all replicas maintain consistent state while allowing for efficient message ordering and group management.

The Raft consensus protocol, designed for understandability, provides predictable latency by maintaining a strong leader until failure occurs. This predictability is crucial for low-latency applications where timing variations can be as problematic as absolute latency values.

Sequencer Architecture: The Future of Low Latency

The sequencer architecture represents the next evolution in low-latency distributed systems. Unlike traditional approaches where business logic runs within a cluster, the sequencer architecture separates the sequencing responsibility from application logic.

In this model, a fault-tolerant cluster runs only the sequencer component, which is responsible for ordering messages and assigning timestamps. State machines reside within applications, consuming the ordered log and processing business logic independently.

This architecture eliminates fan-out problems since applications only need to process the condensed log rather than multiple output messages. Recovery becomes simpler and faster because applications can resume processing from any checkpoint without requiring cluster coordination.

Practical Considerations and Trade-offs

While kernel bypass techniques can achieve impressive latency numbers, they come with significant trade-offs. These approaches require tight integration with specific hardware and may not be portable across different network interface cards or cloud providers.

Java's performance in low-latency scenarios often surprises developers. The JIT compiler's optimizations can make Java implementations comparable to C in many scenarios. However, developers must still be mindful of memory allocation patterns and garbage collection behavior to maintain predictable latency.

Looking Forward

The race to zero latency continues, but the focus has shifted from raw speed to architectural approaches that provide both low latency and other quality attributes like scalability and resilience. The key lies in understanding the fundamental principles of distributed systems and applying them appropriately to specific use cases.

Future developments may include more specialized messaging protocols designed specifically for low-latency scenarios, better integration between hardware and software layers, and architectural patterns that make low-latency design more accessible to mainstream applications.

As cloud providers continue to evolve their offerings, we may see more specialized services designed for low-latency requirements, potentially making these advanced techniques more widely available without requiring custom hardware or complex kernel-level programming.

The journey from the Pony Express to modern distributed systems demonstrates that while the fundamental challenge of latency remains constant, our approaches to solving it continue to evolve and improve.