A Couple Million Lines of Haskell: Production Engineering at Mercury

The article by Ian Duncan from Mercury presents a compelling case study in large-scale Haskell production engineering, challenging conventional wisdom about the language's suitability for mission-critical systems. At its core, this is not merely a technical treatise but a philosophical exploration of what it means to build reliable systems in the face of constant change.

Reliability as Adaptive Capacity

Duncan introduces a refreshing perspective on system reliability, shifting from the traditional focus on failure prevention to what he terms "adaptive capacity." Rather than merely cataloging potential failure points, Mercury's approach emphasizes building systems that can absorb variation, degrade gracefully, and remain comprehensible to operators. This view aligns with resilience engineering principles, recognizing that reliability emerges from a system's ability to maintain functionality despite changing conditions.

This philosophy becomes particularly crucial in fast-growing organizations where, as Patrick McKenzie observed, half of your coworkers will always have less than a year of experience. In such environments, institutional knowledge evaporates unless deliberately preserved—a challenge Haskell's type system uniquely addresses.

Purity as Boundary, Not Property

One of the article's most insightful reframes is its treatment of purity. Duncan argues that purity is not an inherent property of Haskell but rather a boundary to be maintained through careful interface design. This perspective demystifies Haskell's theoretical purity by acknowledging the mutable operations occurring beneath abstractions like runST.

The key insight is that dangerous operations become tolerable when they are "fenced in, carefully exposed, and hard to misuse." This principle extends beyond purity to encompass retries, transactions, state machines, and distributed workflows. It represents a mature understanding that production engineering is about containment and control, not ideological purity.

Encoding Operational Knowledge in Types

The most significant production value Haskell offers, according to Duncan, is its ability to encode operational knowledge in types that survive the departure of the original engineers. This transforms the type system from a correctness proof into an operational aid that preserves institutional memory.

The article illustrates this beautifully with the transaction pattern example, where the only way to execute work is through a path that includes event publication. This makes the correct operational procedure "the path of least resistance," ensuring that critical invariants cannot be accidentally violated when engineers are under pressure or unfamiliar with the system.

Pragmatic Type System Usage

Duncan strikes an important balance between type system enthusiasm and practicality. He acknowledges that "every invariant you push into the type system is a constraint on every future engineer" and advocates for thoughtful tradeoffs:

• Encode invariants that protect against silent corruption
• Use runtime checks for invariants that fail loudly
• Resist over-modeling messy business domains in types
• Encapsulate complexity rather than leaking it across module boundaries

This nuanced approach recognizes that types serve the team as much as the compiler, and that operational effectiveness requires more than just type-level theorems.

Design for Introspection

A particularly valuable section addresses observability and introspection in Haskell's static typing context. Without runtime monkey-patching capabilities, Duncan argues that libraries must be designed with operational escape hatches from the start.

His recommendation of "records of functions" provides an elegant pattern for instrumentation, allowing cross-cutting concerns like tracing, timeouts, and fault injection to be composed orthogonally. The algebraic properties of such designs—particularly their Semigroup and Monoid instances—turn complex operational wiring into a "trivially correct by construction" problem.

The Reality of Production Haskell

The article refreshingly acknowledges that production Haskell is not without compromises. Duncan discusses unsafePerformIO usage in core libraries, the testing challenges of IO-heavy code, and the ecosystem's diffuse stewardship model. These honest acknowledgments ground the discussion in practical reality rather than theoretical ideals.

Why Haskell at Mercury?

Duncan addresses the inevitable question of Haskell's ROI in production. While acknowledging the initial learning curve and ecosystem limitations, he points to compelling long-term benefits:

• Mechanical refactoring that feels "genuinely complete rather than probably complete"
• New hires understanding module contracts through type signatures alone
• Reduced incidence of certain classes of production bugs

In financial services particularly, where the cost of bugs is measured in regulatory findings and other people's money, these benefits compound quickly.

Conclusion

What emerges from this article is a mature perspective on Haskell as a pragmatic engineering tool rather than an academic ideal. Mercury's success with a 2 million-line Haskell codebase demonstrates that the language's true production value lies not in its theoretical purity but in its ability to encode operational knowledge, contain dangerous operations, and provide mechanical leverage in complex, evolving systems.

The article's most important contribution may be its emphasis on Haskell as a tool for preserving institutional knowledge—making the "safe path the easy path" in environments where the people who first understood the system have moved on. In this sense, Haskell becomes not merely a programming language but an organizational mechanism for maintaining reliability through change.

For organizations considering Haskell for production, this article provides a realistic roadmap: not a silver bullet, but a genuinely powerful set of tools when wielded with pragmatism and operational awareness.