When LLM Code Agents Forget the Rules: A Close Look at Constraint Decay in Backend Generation

Paper: Constraint Decay: The Fragility of LLM Agents in Backend Code Generation (arXiv:2605.06445) – Francesco Dente, Dario Satriani, Paolo Papotti, 7 May 2026
Read the PDF

---

What the authors claim

The authors argue that existing code‑generation benchmarks focus almost exclusively on functional correctness (does the program run and produce the right output?) and ignore structural constraints that real‑world services must obey. They introduce a new evaluation suite that fixes a single API contract and asks LLM agents to generate complete backend projects, varying only the amount of architectural detail required. Their headline numbers:

• Across 80 greenfield tasks (start‑from‑scratch) and 20 feature‑addition tasks, agents lose about 30 percentage points in assertion‑pass rate when moving from a minimal spec to a fully constrained spec.
• In the weakest configurations, pass rates drop to near‑zero.
• Framework‑specific analysis shows a clear split: agents perform best on Flask (minimal conventions) and worst on FastAPI and Django, which impose stricter ORM and routing patterns.
• The dominant failure mode is at the data‑layer: malformed SQL queries, missing foreign‑key handling, and ORM runtime errors.

The paper calls this systematic drop constraint decay and positions it as a major obstacle to deploying LLM‑driven coding assistants in production environments.

---

What is actually new

1. A unified API contract for multi‑framework evaluation – The authors designed a single OpenAPI‑style contract that all generated backends must implement. This eliminates the “different specs per framework” confound that has plagued earlier studies.
2. Dual‑level evaluation pipeline – Each generated project is run through (a) an end‑to‑end test suite that exercises the public API, and (b) static analysis tools (e.g., pylint, sqlfluff, and ORM validators) that catch structural violations before runtime. The combination provides a clearer picture of where agents stumble.
3. Quantitative measurement of decay – By incrementally adding constraints (routing conventions, ORM models, transaction handling, authentication scaffolding), the authors can plot a performance curve for each agent configuration. This is the first systematic, numeric description of how structural complexity hurts LLM output.
4. Framework sensitivity matrix – The study reports per‑framework success rates for three popular LLM backends (GPT‑4‑Turbo, Claude‑3‑Opus, and LLaMA‑2‑70B). The matrix reveals that even the strongest model still drops below 50 % pass rate on Django‑style projects.

These contributions go beyond a simple anecdotal note that “LLMs generate messy code.” They provide a reproducible benchmark that other researchers can adopt to test future prompting strategies, tool‑use extensions, or fine‑tuning pipelines.

---

Limitations and open questions

Aspect	Limitation
Scope of languages	The study is limited to Python web frameworks. It is unclear whether the same decay pattern appears in Java/Spring, Node/Express, or Go/Fiber ecosystems.
Model configurations	Only three off‑the‑shelf LLM APIs were tested, each with a single temperature setting. Prompt engineering variations (e.g., chain‑of‑thought, self‑debug loops) were not explored.
Static analysis depth	The static verifiers used are generic linters. More sophisticated type‑checking or formal verification tools might catch additional errors, but were not included.
Human‑in‑the‑loop	All generations are fully autonomous. In practice, developers often edit generated snippets; the study does not measure how much post‑processing would be required to reach production quality.
Dataset realism	Tasks are derived from a curated set of 100 specifications. Real‑world tickets often contain ambiguous requirements, which could amplify or mask constraint decay.

Because the benchmark isolates structural constraints, it does not address performance concerns such as latency, memory usage, or scalability of the generated services. Those non‑functional aspects remain outside the current scope.

---

Why it matters for practitioners

If you are considering an LLM‑based code assistant for backend work, the paper suggests two practical takeaways:

1. Expect a high rate of data‑layer bugs – Even the best‑performing model will frequently produce invalid ORM statements or missing migrations. Plan for a dedicated review step that runs static analysis before integration.
2. Choose frameworks wisely – Simpler, convention‑light frameworks (Flask, Bottle) are more forgiving to LLM output. If your stack includes heavy conventions (Django, FastAPI with Pydantic models), you will need additional tooling (e.g., automated schema validation) to catch the errors.

In short, LLM agents can accelerate scaffolding, but they are not yet ready to replace a developer’s understanding of architectural constraints.

---

Next steps for the research community

• Extend the benchmark to other ecosystems – Adding Java, TypeScript, and Rust backends would test whether constraint decay is a language‑agnostic phenomenon.
• Incorporate tool‑use – Recent work on “LLM agents with tool use” (e.g., code‑execution sandboxes, ORM introspection APIs) could mitigate data‑layer errors. A follow‑up study could compare pure generation versus generation + tool‑assisted refinement.
• Prompt engineering studies – Systematically varying prompts (explicit constraint statements, step‑by‑step generation, self‑debug loops) would reveal how much of the decay is due to insufficient instruction versus model capability.
• Human‑in‑the‑loop experiments – Measuring how much developer effort is saved after a single review pass would give a more realistic cost‑benefit picture.

---

The paper is a sober reminder that functional correctness is only one piece of the production puzzle. Structural fidelity, especially around data access, remains a fragile point for current LLM coding agents.