The pattern is universal across AI deployments: backend resources sit mostly idle, databases hum along efficiently, but large language models (LLMs) choke response times at 2+ seconds per request while users expect 200ms. Costs balloon, latency frustrates users, and no amount of configuration tweaking solves it. The core issue lies deeper—in the transformer architecture itself.

The Original Sin: Training Efficiency vs. Inference Reality

Transformers revolutionized AI with their 2017 "Attention Is All You Need" design, optimizing for parallel training by processing entire sequences simultaneously¹. But this strength became a weakness for inference. Unlike older recurrent models that generated tokens quickly with fixed-size state, transformers must run full forward passes sequentially for each new token—attending to an ever-growing context window with O(n²) computational complexity¹.

As AI engineer Kenneth Wolters observes: "The transformer solved the training problem brilliantly. It created the inference problem we’re still solving." Decoder-only models like GPT exacerbated this by making generation the entire task—every response requires hundreds of sequential passes through billions of parameters².

Anatomy of the Bottleneck: Dual Highways and Memory Walls

Two information flows create distinct constraints:
1. Vertical (Residual Stream): Serial processing through layers (32-96 deep). Linearly expensive but manageable.
2. Horizontal (K/V Stream): Attention over growing context. Quadratically expensive and memory-bound—the true bottleneck³.

During generation:
- Prefill Phase: Processes prompts in parallel (fast, compute-bound)
- Decode Phase: Generates tokens sequentially (slow, memory-bound)

"The GPU isn’t struggling with computation. It’s struggling with memory access," explains Wolters. Each new token requires loading the entire KV cache—stored Keys/Values from all prior tokens—from GPU memory. With context windows reaching 200K tokens, caches balloon to gigabytes per request⁴. Modern GPUs’ memory bandwidth simply can’t keep pace with the O(n²) memory access demands⁵.

Why Parameter Count Isn’t the Real Villain

Surprisingly, most LLM parameters (∼66%) reside in MLPs that scale linearly with sequence length. The attention mechanism—responsible for the quadratic bottleneck—holds fewer parameters but dominates latency due to memory access patterns. As Wolters notes: "The majority of parameters aren’t creating the majority of the bottleneck." This explains why quantizing MLPs often succeeds while compressing attention mechanisms degrades output quality⁶.

The Stateful Illusion

Transformers are mathematically stateless—identical inputs yield identical outputs. Yet generation requires maintaining ever-expanding context via KV caches. Unlike RNNs’ fixed-size state, these caches grow linearly with sequence length, consuming GPU memory that could serve more requests. A 2000-token context in a 32-layer model needs ∼1GB just for KV data at 16-bit precision⁷.

The Path Forward

This architectural reality frames the optimization landscape: solutions must address memory bandwidth constraints without sacrificing attention’s contextual power. Techniques like KV cache quantization, FlashAttention’s IO-aware algorithms⁸, and paged memory management become essential—not mere optimizations but necessary adaptations to the transformer’s inherent design trade-offs.

Engineers fighting LLM latency must first understand this core tension: the same architecture enabling unprecedented capability also dictates its performance limits. As we’ll explore next, effective optimization requires working with these constraints—not against them.

Source: Analysis based on technical deep dive by Kenneth Wolters. Reference citations map to source footnotes¹²³⁴⁵⁶⁷⁸.