The Illusion of Scale, Part 4 – Latency Is a Design Decision, Not a Measurement

When I first walked into a stakeholder meeting with a neat spreadsheet that listed a 200 ms latency budget broken down by component, I felt like I had solved the problem. Auth service = 15 ms, business logic = 30 ms, DB query = 40 ms – the sum was comfortably under the limit. We shipped.

Two weeks later the auth call, which had been 15 ms in the lab, was regularly spiking to 200 ms in production. I scrambled for profiling tools, added new observability agents, and chased every obvious suspect: slow queries, cache misses, network timeouts. Nothing explained the jump.

The truth was embarrassingly simple: the auth service was shared with four other services that all peaked at the same time. No one had reserved capacity for us, so under load the service became a bottleneck. Our budget of 15 ms evaporated.

That moment forced a shift in mindset – from measure‑and‑optimize to design‑for‑latency.

---

Why Measuring Too Late Is a Trap

1. Architectural constraints are hidden deep – By the time you see a latency breach in production, the offending decisions are often three or four layers down (shared databases, synchronous fan‑out, global locks). Re‑architecting those pieces means rewriting code that other teams depend on, all while users are waiting.
2. Load tests are only as good as the traffic model – Synthetic load can’t anticipate the exact mix of spikes, contention, and usage patterns that a live system experiences. Real traffic brings shared dependencies and concurrent spikes that no test suite imagined.
3. Optimization becomes reconstruction – Tweaking a query or adding a cache helps marginally, but it doesn’t move the latency floor imposed by the system’s structure.

---

Where Latency Really Lives

Structural Pattern	Latency Floor	Why Optimization Won’t Help
Chattiness – many internal RPCs per request	Sum of all RPC latencies	Even perfect caching can’t reduce the number of round‑trips. Redesign the call graph.
Unbounded fan‑out – user‑controlled N records	O(N) processing time	A single power user can blow up the path; limits or pagination must be enforced up front.
Synchronous waits on async work – waiting for a write to propagate, a downstream confirmation, or a cache warm‑up	Fixed ceiling equal to the async operation’s latency	No amount of CPU tuning removes the wait; you must make the work truly async or decouple it.

These patterns are architectural rather than code‑level problems. The fix is to change the design, not to keep shaving milliseconds off a slow line of code.

---

A Pragmatic Approach: Latency Budgets as Design Artifacts

1. Define the target response time early – e.g., 200 ms for an end‑user request.
2. Allocate a budget to each component in the critical path – auth = 15 ms, business logic = 30 ms, DB = 40 ms, network = 20 ms, safety buffer = 30 ms, etc.
3. Document the budget in a place everyone sees – a shared Confluence page, a README, or a diagram with annotated numbers.
4. Identify shared downstream resources – If two services both call the same authentication endpoint, their budgets are not independent. The combined load must fit within the provisioned capacity.
5. Assign explicit owners – Every shared dependency gets a responsible team and a latency SLA. This forces the conversation about capacity, isolation, and fallback strategies before code is written.

When we performed this exercise on a later project, we discovered that the auth service was a single‑node deployment serving three independent front‑ends. The budget exercise made the lack of isolation obvious, and we provisioned a dedicated replica for the high‑priority service. The result: the latency budget held steady even under peak load.

---

The Real Cost of “10 ms”

At low traffic, a 10 ms improvement may feel negligible. At 100 k RPS, however, that 10 ms translates to 1 000 seconds of user wait time per second of operation – effectively a full minute of human attention wasted every second the system runs. That is a customer problem, not a pure engineering curiosity, which explains why high‑volume teams spend weeks polishing single‑digit‑millisecond paths.

---

A Real Incident: Missing Dependency Mapping

A stakeholder once asked why the system was “sometimes fast, sometimes slow, with no pattern.” The code was clean; the mystery lay in the infrastructure topology. Two services that appeared independent on the architecture diagram both wrote to the same MongoDB collection. Under concurrent load they contended for the same write lock, causing intermittent spikes.

The fix was simple – add a sharded collection and give each service its own shard key. But because we had no explicit diagram of that shared dependency, we spent days chasing dead‑ends in application code.

Since then, every critical path includes a dependency map with owners, capacity limits, and latency budgets. The map lives alongside the API contract, and any change to a shared resource triggers a budget re‑evaluation.

---

Takeaways

• Latency is a design problem – treat it like any other architectural trade‑off (e.g., consistency vs. availability).
• Budget early, document loudly – a written budget surfaces hidden shared resources before they become emergencies.
• Model contention, not just happy‑path latency – include worst‑case concurrent load in your capacity planning.
• Assign owners to shared dependencies – accountability prevents silent coupling.
• Remember the volume multiplier – a few milliseconds matter when you serve millions of requests per day.

The next time you hear “let’s just measure and tune,” ask whether the structure of the system already guarantees a lower bound higher than the target. If it does, no amount of profiling will bring you under budget – you need a redesign.

---

Next week’s post will explore why some systems survive the turnover of teams while others crumble, and what the survivors have in common.

---

Further reading

• Designing for Latency Budgets – Martin Kleppmann’s blog
• The Latency Numbers Every Engineer Should Know
• MongoDB Atlas – Multi‑cloud clusters for low‑latency global apps