The Bitter Lesson for Backend Engineering

The problem we faced

We were asked to support 500‑1000 RPS on a set of endpoints that didn’t even exist yet, while dozens of existing routes were stuck at single‑digit RPS. Adding more CPU or pods gave almost no lift. The obvious suspects – the ASGI loop, the ORM, the GIL, the database proxy – were all plausible, but none could be proved without a systematic search. Human intuition simply cannot hold the combinatorial space of query plans, connection pools, cache layers, and external calls in working memory.

The scalable approach

Instead of letting a single engineer chase one hypothesis after another, we built a goal‑oriented agent loop:

1. Define the objective – make the target traffic profile plausible without sacrificing latency budgets.
2. List plausible binding constraints – DB CPU, worker‑process saturation, middleware overhead, cache‑stampede, third‑party blocking.
3. Create a falsifier for each – a minimal load harness that isolates the factor and records a clear metric (CPU%, request‑time breakdown, query count, etc.).
4. Run agents in parallel – each agent owns a slice of the experiment space (instrumentation, harness generation, data collection, result synthesis).
5. Accept the evidence – if a hypothesis is falsified, retire it; if it survives, prioritize a treatment.

The human role was reduced to setting the risk boundary, approving live‑environment runs, and judging when a measured improvement is sufficient.

Trade‑offs

Aspect	Traditional intuition‑driven tuning	Agent‑driven systematic search
Speed	Weeks‑to‑months per bottleneck	Hours for multiple bottlenecks
Coverage	One story at a time, easy to miss interactions	Parallel exploration of many axes
Reliability	Prone to confirmation bias, hidden regressions	Measurements drive decisions, guardrails catch errors
Human effort	High cognitive load, context switching	Low – focus on goal definition and safety
Risk	Accidental overload of production during ad‑hoc tests	Requires strict isolation and stop‑conditions

The experiment that exposed the first axis

A gentle 3 RPS load on the paginated list endpoint showed database CPU at ~100 % while the proxy and connection pool were idle. A quick local sweep proved Python CPU scaled linearly, so the GIL was not the limiter. The agents recorded:

• DB CPU – primary bottleneck
• Connection fan‑out – not saturated at this load
• Event‑loop latency – negligible

With the axis identified, the agents rewrote the query patterns:

• Collapse N+1 fetches (179 → 15 queries)
• Replace COUNT(DISTINCT …) with a cheap pagination count (639 ms → 1.5 ms)
• Turn a costly GROUP BY into a correlated subquery (1.59 s → 0.174 s)

These changes alone cut the DB CPU load by more than 80 % on the test endpoint.

What the deeper search revealed

Running the same agents on a broader set of endpoints uncovered five distinct binding constraints:

1. DB‑CPU wall – many endpoints share the same heavy query shape.
2. Worker‑process ceiling – GIL‑bound gunicorn workers cap throughput at ~1 k RPS per instance.
3. Middleware floor – auth, DRF rendering, and observability add a fixed ~20 ms even on cache hits.
4. Cache‑stampede risk – misses funnel traffic back to the DB, negating surface fixes.
5. Third‑party blocking – synchronous external calls prevent linear scaling.

Each axis required a different treatment, and the agents produced a ranked backlog of PRs, complete with measurement scripts and payload‑equivalence checks.

Evidence of scale

In 36 hours the team plus three agents produced:

• A capacity model linking worker count, CPU‑per‑request, and target RPS.
• An inventory of hot queries derived from pg_stat_statements and EXPLAIN output.
• Four realistic load profiles (burst join, sustained livestream, etc.) with associated bottleneck maps.
• ~60 tracking tickets: hypothesis definitions, falsifiers, and execution tasks.
• Fifteen durable artifacts (playbooks, runbooks, methodology specs) stored in the repo.

A senior engineer working alone would typically need a quarter to achieve a fraction of this output, and many of the artifacts would never be written.

Guardrails that made autonomous work safe

Cheap parallel compute amplifies both value and error. The following safeguards prevented runaway experiments:

• Local‑first execution – all harnesses run against a cloned environment before any live traffic.
• Explicit live‑run gates – a human approves each production load window.
• Stop conditions – abort on error‑rate spikes, CPU saturation, or unexpected latency.
• Payload‑hash verification – ensure that performance gains do not alter response bodies.
• Append‑only coordination log – every action is recorded, making regression hunting trivial.

Without these, the agents could have DDoS‑ed the shared database or hidden a regression behind a stale cache.

When the method falls short

The approach shines when the system is observable and the goal is measurable. It is weaker for:

• Green‑field architecture decisions that lack telemetry.
• Product‑level trade‑offs (e.g., feature scope vs. latency) where human judgment dominates.
• Novel failure modes that have no existing instrumentation.

In those cases the human still leads the design; agents can assist by generating data once the observability gap is closed.

The meta‑lesson for backend teams

1. Elevate the human role – set clear, quantitative objectives; define falsifiers; enforce safety.
2. Invest in cheap compute – parallel agents that can read code, generate harnesses, and run benchmarks.
3. Make every hypothesis falsifiable – treat intuition as a hypothesis, not a solution.
4. Capture the process – store capacity models, hypothesis tickets, and verification scripts in version control.
5. Accept the evidence – if measurement kills a beloved story, celebrate the discovery.

The uncomfortable truth mirrors Richard Sutton’s “Bitter Lesson”: the most durable progress comes from methods that scale with computation, not from the cleverness of a single mind. Backend engineering is reaching the point where the cost of compute is low enough to let disciplined, parallel experimentation replace endless argument in war rooms.

---

Further reading

• Richard Sutton, The Bitter Lesson (2019) – https://www.incompleteideas.net/IncIdeas/BitterLesson.html
• Django caching strategies – https://docs.djangoproject.com/en/stable/topics/cache/
• Universal Scalability Law – https://www.usenix.org/legacy/events/usenix99/full_papers/vonluen/luen_html/

The agents described here are not a magic bullet; they are tools that amplify a well‑defined engineering process. When the process is sound, the compute does the heavy lifting, and senior engineers can focus on the higher‑level decisions that truly require human judgment.