How to Architect Event-Driven Multi-Agent Systems for Production

Most teams build their first multi-agent system the obvious way. One agent calls another, that agent calls a tool, the tool returns a result, and the chain bubbles back up. It demos beautifully. Then it goes to production, traffic arrives in bursts, one model provider rate-limits you, an agent hangs waiting on a downstream call, and the whole graph of synchronous calls freezes at once. The pattern that felt clean in a notebook becomes a distributed systems problem nobody planned for.

Event-driven architecture is the answer the broader software world already settled on for this class of problem, and it maps onto agent orchestration almost directly. Instead of agents calling each other, they publish events to a message broker and subscribe to the events they care about. The coordination stops being a call stack and becomes a flow of messages.

Why direct calls break down

The trouble with chaining agents through direct invocation is that every link in the chain inherits the latency and failure modes of everything downstream. An agent waiting on a synchronous response is a thread or a coroutine doing nothing useful, holding memory, and counting down toward a timeout. When you have three or four agents deep in a reasoning loop, each potentially making LLM calls that take several seconds, the tail latency compounds. A single slow provider response stalls the entire pipeline.

Coupling is the deeper issue. When agent A calls agent B by reference, A needs to know B exists, where it lives, and what its interface looks like. Adding a fifth agent means touching the code of the agents that should talk to it. Scaling agent B to three instances means putting a load balancer in front of it and teaching A about that. The system resists the exact kind of incremental change that agent systems demand, because you are constantly adding capabilities.

The event-driven shape

In an event-driven design, agents become producers and consumers on a shared backbone. A planning agent receives a user request and publishes a task.created event. A research agent subscribes to that event type, picks up the work, does its job, and publishes research.completed. A writing agent waits for that, produces a draft, and emits draft.ready. None of these agents holds a reference to another. They agree only on the shape of the events.

This indirection buys several things at once. Agents can fail and restart without losing in-flight work, because the broker holds the messages until something consumes them. You can run ten instances of the research agent behind the same subscription and the broker distributes events across them. You can add a logging consumer or a guardrail consumer that reads the same event stream without any existing agent knowing it exists. The bus absorbs bursts, so a spike in requests queues up rather than knocking over downstream agents.

The backbone itself is usually a battle-tested broker rather than anything AI-specific. Apache Kafka gives you durable, replayable logs and high throughput, which matters when you want to reconstruct what an agent saw. NATS is lighter and lower-latency, a good fit when agents exchange many small messages. For teams already on a cloud provider, managed options like AWS EventBridge or Google Pub/Sub remove the operational burden of running the broker yourself.

Designing the events

The events are the contract, so they deserve more care than the agents. A useful event carries a stable type, a correlation ID that ties together every message belonging to one user request, a payload, and metadata about who produced it and when. The correlation ID is what lets you trace a single request as it fans out across a dozen agents and reassemble the story afterward.

Keep payloads self-contained where you can. If an event references a large artifact, like a document an agent produced, store the artifact in object storage and put a pointer in the event rather than the bytes themselves. This keeps the broker fast and your events small enough to log and inspect.

Version the schemas from day one. Agent systems evolve quickly, and the moment you have two services reading the same event type, a breaking change to that schema is a coordinated deploy. Additive changes, new optional fields, let producers and consumers move independently. A schema registry enforces this discipline so a malformed event gets rejected at publish time instead of crashing a consumer at 3am.

Orchestration versus choreography

There are two ways to arrange the flow, and real systems mix them. Choreography is the pure event-driven form: each agent reacts to events and emits new ones, and the overall behavior emerges from those local rules. It is maximally decoupled and easy to extend, but the flow lives implicitly across many subscriptions, which makes it harder to reason about and harder to see when something stalls midway.

Orchestration puts a coordinator in charge. A single orchestrator agent or workflow engine listens for events and decides what happens next, issuing commands that other agents execute. You lose some decoupling, since the orchestrator knows the steps, but you gain a single place to see and control the whole process. Durable workflow engines like Temporal are popular here because they persist the state of a long-running workflow, survive restarts, and handle retries and timeouts as first-class concerns. For agent systems where a single request might run for minutes across many steps, that durability matters more than it does for typical request-response services.

A common middle ground uses orchestration for the high-level business flow and choreography within each stage. The orchestrator decides that research must happen before writing, but the research stage itself is a swarm of agents reacting to each other's events.

The hard parts production exposes

Exactly-once processing is the first thing that bites. Brokers generally promise at-least-once delivery, which means an agent will occasionally see the same event twice. If that event triggers an LLM call that costs money or an action with side effects, duplicates are expensive or dangerous. The fix is idempotency: make consumers safe to run twice by keying side effects on the event's ID and discarding work already done.

Ordering is the second. A research agent might emit results out of order, or a retry might land an old event after a newer one. If your logic depends on sequence, you need partition keys that route related events to the same consumer in order, or you need consumers that tolerate reordering. Kafka's per-partition ordering is one tool here, but it constrains how you parallelize.

Observability is the part teams underestimate most. In a synchronous system, a stack trace tells you the path a request took. In an event-driven system that path is scattered across brokers and consumers, so you have to build the trace yourself. Propagate the correlation ID through every event, emit structured logs keyed on it, and adopt distributed tracing through something like OpenTelemetry so a single request's journey across agents shows up as one connected trace. Without this, debugging a failed multi-agent run is archaeology.

Then there is the dead letter queue, the place events go when a consumer fails to process them after retries. You want one, you want alerts on it, and you want a way to replay its contents once you have fixed the bug. Treat it as a feature, not an afterthought, because in a system making nondeterministic LLM calls, some events will fail in ways you did not anticipate.

What this costs and what it returns

None of this is free. An event-driven multi-agent system is genuinely harder to build and operate than a script that calls agents in sequence. You are running a broker, managing schemas, building observability that synchronous systems get for free, and reasoning about eventual consistency. For a prototype or a low-volume internal tool, that overhead is not worth it, and a simple orchestration library is the right call.

The architecture earns its keep when the system has to be reliable, has to scale to unpredictable load, and has to keep growing in capability without a rewrite each time. Those are precisely the conditions production imposes. The agents that looked like the interesting part turn out to be the easy part. The plumbing that moves events between them, durably and observably, is what separates a demo from a system you can put real traffic and real money through.

The encouraging part is that almost none of this is new. Event-driven architecture carried microservices through the same transition a decade ago, and the brokers, the patterns, and the operational playbooks are mature. Multi-agent systems get to inherit that work rather than reinvent it. The teams shipping reliable agent systems today are mostly the ones who recognized the problem was distributed systems wearing an AI costume.