Article illustration 1

At Airbnb, the key-value store known as Mussel sits at the heart of critical workflows like fraud detection, dynamic pricing, and personalization, serving as a bridge between offline analytics and online services. Since its inception, Mussel V1 handled billions of requests daily but strained under modern demands: scaling nodes required manual Chef scripts, hash partitioning caused latency spikes, and consistency controls were rudimentary. Faced with these bottlenecks, engineers undertook a bold rearchitecture—replacing the entire storage backend with a NewSQL system, now called Mussel V2. This new engine, running in production for over a year, merges Kubernetes-native operations with Kafka-driven streaming to deliver unprecedented scale and simplicity.

Why V1 Had to Evolve

Mussel V1’s limitations became stark as Airbnb’s data grew. Static hash partitioning led to hotspots that spiked read latency during traffic surges, while operational tasks like node replacement devoured hours of manual work. "Scaling or replacing nodes required multi-step Chef scripts on EC2," the team notes. Worse, V1 offered binary consistency—immediate or eventual—with no flexibility for varied SLA needs. As data swelled beyond 100TB per table, these issues threatened real-time use cases. The solution? A complete rebuild prioritizing automation, dynamic sharding, and granular control.

Inside Mussel V2’s Architecture

V2’s redesign centers on decoupling and cloud-native resilience:
- Stateless Dispatcher: Replacing V1’s monolithic design, this Kubernetes service handles API translation, dual-write migration logic, and dynamic throttling. Reads now support point lookups, range queries, and stale reads from local replicas to cap p99 latency at 25ms.
- Kafka as the Backbone: All writes first hit Kafka for durability, with a Replayer applying them asynchronously to the NewSQL backend. This absorbs traffic bursts and simplifies bootstrapping—critical for seamless migrations.
- Bulk Load Reinvented: Airflow pipelines transform warehouse data into S3-staged files, ingested via parallel workers (Kubernetes StatefulSets). Optimizations like delta merges and deduplication handle terabyte-scale replaces without downtime.
- TTL with Teeth: V1’s compaction-based expiration buckled under load. V2 shards expiration tasks range-wise, running parallel sweeps with minimal read impact—crucial for governance and cost control.

The Zero-Downtime Migration Challenge

Moving petabytes across thousands of tables demanded surgical precision. Airbnb adopted a blue/green strategy but hit a snag: V1 lacked native snapshots or CDC. Their solution? A custom pipeline:
1. Bootstrap with Sampling: Extract V1 data samples to presplit V2 tables, avoiding hotspots during ingestion.
2. Dual Writes & Shadowing: Kafka streams synced V1 and V2 during migration, while dispatchers routed reads to V1 but shadowed V2 for validation.
3. Reversible Cutovers: Each table shifted via stages—reverse traffic to V2, with fallback to V1 if errors spiked. Circuit breakers enabled instant rollbacks.

"We migrated over a petabyte with zero data loss or downtime," the team emphasizes. "Kafka’s p99 millisecond latency was our safety net."

Lessons from the Trenches

The migration unearthed hard truths:
- Consistency Isn’t Free: Moving to strong consistency required write deduplication and conflict resolution, trading storage cost for reliability.
- Presplitting is Non-Negotiable: Without accurate key-range sampling, bulk inserts overloaded shards. Range-based partitioning demands upfront data distribution analysis.
- Flexibility Wins: Letting services choose stale reads (from secondaries) vs. fresh reads (primaries) balanced cost and performance for diverse use cases.

Beyond the Migration

Today, Mussel V2 concurrently handles 100k+ writes/sec, terabyte bulk loads, and sub-25ms reads—all within a single platform. Engineers no longer jury-rig caches and queues; V2 provides this out-of-the-box. Future work includes fine-grained QoS controls and bulk load optimizations. For Airbnb, this isn’t just a storage upgrade—it’s an enabler for real-time innovation at global scale.

Source: Building a Next-Generation Key-Value Store at Airbnb