Article illustration 1

Deterministic Simulation Testing (DST) represents the gold standard for verifying distributed systems—where any failure can be reproduced exactly by replaying a random seed. When Polar Signals migrated their database from Go (FrostDB) to Rust, they seized the opportunity to fundamentally rearchitect their testing approach. Their journey reveals why state machines are becoming the cornerstone of next-generation database reliability.

The Four Pillars of Determinism

At DST's core lie four non-negotiable controls:
1. Concurrency: Precise orchestration of task execution order
2. Time: Virtualized clocks replacing system time
3. Randomness: Seed-driven reproducible decisions
4. Failure Injection: Systematic corruption of operations

"With FrostDB in Go, we wrestled control from the runtime scheduler through single-threaded execution," explains Alfonso Subiotto Marqués, Engineer at Polar Signals. "But failure injection remained cumbersome—every filesystem or network interaction required custom mock implementations."

The State Machine Revolution

Building their Rust database from scratch enabled a paradigm shift. Inspired by actor models and sled's simulation guide, Polar Signals architected their system as isolated state machines communicating solely via messages:

pub trait StateMachine {
    fn receive(&mut self, m: Message) -> Option<Vec<(Message, Destination)>>;

    fn tick(&mut self, curtime: Instant) -> Option<Vec<(Message, Destination)>>;
}

This design collapses system complexity into a single messaging channel—a technique called dimensionality reduction. illustrates how a central Message Bus (the Director) coordinates all interactions:

"By constraining all state transitions to receive and tick calls, we achieve compile-time guarantees against hidden concurrency. The message bus becomes the single source of truth for scheduling, time advancement, randomness, and failures," notes Subiotto Marqués.

Centralized Control Unleashed

  • Scheduling: The Director's event loop sequences state machine activations deterministically
  • Time: Virtual timestamps replace system clocks, enabling simulated hours in milliseconds
  • Randomness: A single seeded PRNG drives all decisions
  • Failure Injection: The bus intercepts any message to inject drops, delays, or errors—no per-component mocks needed

This architecture proved its value immediately, uncovering critical data loss and duplication bugs during early testing. The centralized failure model particularly shines—where traditional dependency injection requires endless boilerplate, the message bus corrupts all components uniformly.

Navigating the Tradeoffs

Despite its power, the approach demands careful navigation:
1. Cognitive Overhead: Developers must rigorously isolate state transitions, resisting the temptation to "leak" logic outside machines
2. Dependency Non-Determinism: Third-party libraries (e.g., UUID generators) require mocking or wrapping in state machines

"We mitigate this by minimizing dependencies and using block_on for async wrappers," explains Subiotto Marqués. "The mental model shift is significant—but forces explicit reasoning about state transitions that pays long-term dividends."

The Verdict: Rewrite vs. Retrofit

For existing systems, frameworks like Rust's madsim or Go's WASM-based runtimes offer practical DST pathways. But for greenfield projects? "State machines provide unparalleled determinism," asserts Subiotto Marqués. "We traded 'just write code' flexibility for the superpower of exhaustive failure simulation. When building infrastructure where correctness is non-negotiable, that's a trade worth making."

As databases increasingly underpin critical applications, Polar Signals' architectural gamble demonstrates how deep testing integration—not just afterthought harnesses—can elevate reliability. The curtain rises on a new era where deterministic state machines transform testing from a chore into a core design principle.