Beyond unwrap: Cloudflare Outage Reveals Error Handling as a Systemic Discipline

Cloudflare's detailed postmortem of its November 18 outage has ignited fervent discussions among developers on error handling practices, particularly the use of Rust's unwrap(). Described in the postmortem as a mechanism that returns a successful result or crashes the program—much like an assert—this single construct triggered a chain reaction that took services offline globally.

In a thoughtful analysis, infrastructure veteran Marc Brooker, in his blog post "What Now?", argues that the controversy misses the forest for the trees. Error handling isn't a isolated decision within a component but a "global property of the system, and the way it handles data." Drawing from experience at scale, Brooker uses an interactive "error handling game" to guide readers through nuanced scenarios, revealing when crashing enhances safety and when graceful continuation preserves availability.

The Outage's Rust Revelation

Cloudflare's incident began with an internal tool failure, leading to an unwrap() in Rust code that assumed a successful Result variant. Rust's Result enum enforces explicit error management, but unwrap() shortcuts this by panicking on Err—a pattern common for invariants but risky in production without systemic safeguards.

"If you’re not familiar with Rust, you need to know about Result, a kind of struct that can contain either a successful result, or an error. unwrap says basically 'return the successful results if there is one, otherwise crash the program.' You can think of it like an assert."

This sparked debates on asserts in production, but Brooker elevates the discourse: local choices like unwrap() must align with architecture-wide resilience.

Playing the Error Handling Game

Brooker's post poses five scenarios, prompting votes on crashing (✅) versus continuing (❌), followed by his rationale. These distill real-world trade-offs in distributed systems.

These aren't rote rules but probes into systemic context, favoring simplicity (crash uncorrelated faults) over degraded modes where correlations loom.

Three Principles for Systemic Resilience

Brooker's votes cohere around three pillars:

1. Correlated Failures: Uncorrelated issues (e.g., hardware faults) suit crashing for cleanliness. Correlated ones—from user exploits to config deploys—demand isolation and continuation to avoid cascade.

2. Higher-Layer Handling: Traditional autoscaling copes with low crash rates via health checks but buckles under floods. Fine-grained designs (serverless, actor models) thrive on crashes, restarting swiftly.

3. Meaningful Continuation: Feasible for config or non-invariant data via fallbacks, adding modes but boosting uptime. Impossible for stateful invariants (e.g., replication), where divergence trumps availability.

// Rust example: Explicit vs. unwrap
let result = risky_operation();
if let Ok(value) = result {
    // Proceed safely
} else {
    // Systemic handling: log, fallback, or propagate
    use_last_known_good();
}
// unwrap() here presumes invariant—system must guarantee it

Rust and Java shine over C/C++ by surfacing errors explicitly, avoiding "continue after segfault" pitfalls.

Architecting for the Inevitable

"Error handling in systems isn’t a local property," Brooker asserts. It permeates design, from language choice to ops. Blast radius techniques—cell architectures, shuffle sharding, regional independence—embody humility, capping mishandled errors' spread to slivers of traffic.

Cloudflare's misstep, while public, underscores a truth for all at scale: resilience isn't bolted on. By internalizing these global properties, developers craft systems that falter gracefully, turning outages from spectacles into footnotes in the log.