How agentic AI strains modern memory hierarchies • The Register

Large language model inference is often stateless, with each query handled independently and no carryover from previous interactions. A request arrives, the model generates a response, and the computational state gets discarded. In such AI systems, memory grows linearly with sequence length and can become a bottleneck for long contexts.

Agentic AI refers to systems that maintain continuity across many steps. These AI agents don't answer a single question before resetting. They engage in extended workflows, remembering past instructions and building on intermediate results over time. In these multi-turn scenarios, the conversation context becomes a critical, persistent state rather than a transient input. This creates a memory residency requirement.

The inference engine cannot simply discard the state after generating a token. It must maintain the Key-Value (KV) cache, which is the intermediate representation of the attention mechanism, across multiple stages. In an agentic workflow, the time-to-live (TTL) of an inference context extends to minutes, hours, or even days in asynchronous workflows. So even though agentic algorithms might orchestrate multiple reasoning paths at the software level, the underlying inference process remains deterministic.

Managing this branching and extended KV cache across multiple steps therefore requires memory capable of rapid switching between different context states. In effect, memory becomes a record of the agent's reasoning process, where any prior node may be recalled to inform future decisions. As a result, the emergence of agentic AI systems is shifting the bottleneck from raw compute to memory capacity, bandwidth, and hierarchical design.

Existing memory hierarchies break down

AI infrastructure heavily relies on a hierarchy of memory and storage tiers (often labeled G1 through G4) to handle data. These range from GPUs' onboard high-bandwidth memory (HBM), the fastest, to network-attached storage, the largest and slowest. For typical workloads, this hierarchy balances speed and capacity, with the most performance-critical data in HBM and less critical data in CPU RAM or local SSD.

However, agentic long-context inference often pushes this hierarchy beyond the current state. There is no single tier that can optimally balance capacity and latency at the scale demanded by multi-turn, persistent-state agentic AI.

The obvious place to store a large context is the GPU's own memory (HBM), since that's where the model can access it fastest. If the entire context (and its KV cache) could reside in HBM, that would give the best performance. But the problem is memory capacity. GPU HBM is extremely fast but very limited and expensive. Even a state-of-the-art GPU has a fixed HBM capacity that cannot be expanded after manufacturing. Agentic contexts can require much more memory capacity if the user wants to accumulate millions of tokens or serve many concurrent long conversations.

There is a fundamental mismatch. HBM was optimized for access speed with nanosecond latency and high bandwidth, not for the capacity that agentic AI needs. Furthermore, the cost per gigabyte of HBM is extremely high, which makes it economically unfeasible to simply scale capacity by adding more GPUs. This limitation can result in underutilized compute resources, with GPUs spending cycles waiting for memory rather than performing inference.

Another common memory tier is the system DRAM, which offers high capacity at a much lower cost. However, the bandwidth gap relative to HBM is massive, with system DRAM bandwidth potentially an order of magnitude lower than GPU HBM, and transferring large KV cache datasets over buses like PCIe can introduce additional latency that stalls servers during tight token-generation loops.

Let's look at some architectural responses.

1. Near-compute memory tiers

One practical architectural solution is to introduce a new memory tier closer to the GPU that offers a better balance between speed and capacity for context storage. Instead of the huge gap between HBM and NVMe, we add an intermediate layer that can be accessed with latency and bandwidth much closer to memory than to disk.

Nvidia announced the Inference Context Memory Storage (ICMS) platform, which does this. It is an Ethernet-linked flash memory tier optimized for KV cache. The memory is integrated into the AI cluster and connected via a high-bandwidth fabric, allowing GPUs to retrieve context data from it with minimal jitter or delay. The company describes it as a new G3.5 tier that bridges the gap between the local fast tiers and remote storage.

The idea is that this tier is designed solely for serving inference context, so it can be tuned for that use case, such as large, streaming reads of cache data and high parallelism. Other vendors have also discussed extended memory fabrics. For instance, WEKA's Augmented Memory Grid extends GPU memory via a shared NVMe-backed fabric over remote direct memory access (RDMA). The common theme is bringing additional memory as physically close to the GPU as possible. This often involves specialized hardware or networking bridges to ensure that accessing this external memory is extremely fast and can be done in parallel with GPU computation.

The benefit of such near-compute memory tier is scale-out capacity with minimal performance loss. As reported by Nvidia, the net effect is that using such a tier can deliver multiple times higher effective token throughput and better power efficiency than relying on a traditional storage backend.

1. Compute Express Link (CXL) interconnect

Adding new memory tiers leads to disaggregated memory architectures. This means that the memory is not tied rigidly to each compute node, instead memory resources are pooled and shared across many processors via high-speed interconnects. The hardware enabler for this is the Compute Express Link (CXL), which allows attaching external memory to CPUs or accelerators via cache-coherent, low-latency links.

Researchers have proved that offloading the KV cache to CXL-attached memory can reduce GPU memory usage by up to 87 percent, while still meeting latency requirements. The key is that CXL can provide sub-100-nanosecond access latencies with high bandwidth. This makes external memory act like an extension of local RAM rather than a distant storage device.

From a system perspective, pooling memory across GPUs avoids context duplication and enables greater flexibility. For example, if Agent A's context is in the pool, and Agent B on another node needs to read some of it, that data doesn't have to be transferred via a slow path. Both agents can reference the same memory location via the fabric.

1. Memory management software

Regardless of hardware approach, whether it's new memory tiers or disaggregated pools, the system needs memory management software to fully capitalize on the hardware improvements. Simply adding terabytes of memory is not enough. We must intelligently manage which parts of the context reside in the fastest memory, which can be compressed or moved, and how to do all this without disrupting the model's operation.

Frameworks like Nvidia Dynamo, with the Nvidia Inferencing Extension Library, already attempt to orchestrate content movement in an optimal way. They have KV block managers that pre-fetch and pre-allocate chunks of context into the right tier before the model actually needs them.

What's next?

In the early days of deep learning, AI was all about bigger models and more compute. But with agentic AI systems, the question is "How effectively can AI remember?" The ability to retain and manage state is turning out to be the next critical bottleneck. In these systems, larger models are of little use if we cannot feed them the information they need from prior interactions or if the cost of using them in long contexts becomes prohibitive.

While Nvidia's involvement in this space will catalyze the ecosystem, it is important to note that the solution is one slice of the solution space. It addresses problems in large-scale deployments by introducing a new tier to handle overflow context, but it does not eliminate the inherent issue with fast, large memory. Nvidia's context memory platform is evidence that the bottleneck is real. Many companies are working on solving this problem, from CXL consortium efforts to alternative accelerator designs that incorporate large memory.

We have essentially shifted the hard part of the problem from pure computation to a memory- and data-management challenge. If we try to force the agentic AI system with existing hardware, we end up with GPUs waiting on data, recomputing context, and burning energy doing redundant work. All the signs point to memory architecture being the determinant of the scalability of next-generation AI.

The scalability of AI is also defined by memory and by how we design systems to remember more with less cost. Advancements in high-capacity near-memory, fast interconnects like CXL, caching algorithms, and software orchestration will collectively determine how far we can push AI capabilities in practice.