Agora-1 Introduces Real‑Time Multi‑Agent World Modeling

![Featured image](){: .center}

The problem: world models are lonely

Traditional learned world models excel at generating high‑fidelity visual predictions for a single agent interacting with an environment. That architecture works well for single‑player games or solitary robot control, but it collapses when more than one participant needs to see and affect the same simulated reality. Existing multi‑agent attempts—such as Multiverse, Solaris, and MultiGen—either mash all agents into a single “split‑screen” tensor or stretch a diffusion transformer’s context to accommodate several agents. Those tricks quickly hit scaling limits and struggle to keep the world coherent when agents lose line‑of‑sight.

Agora-1’s answer: decouple simulation from rendering

Agora‑1 takes a different route. It learns two complementary functions:

1. State dynamics model – Trained on the internal game state of GoldenEye (positions, health, ammo, etc.), this module predicts how the shared world evolves after each action. Because it works on a structured, discrete representation, it can be updated quickly and remains agnostic to how many agents are watching.
2. DiT‑based renderer – Conditioned on the shared state, a diffusion‑in‑transformer (DiT) model synthesizes pixel‑perfect frames from any viewpoint. The renderer receives the exact same state vector for every participant, ensuring that each sees a consistent world even though their cameras differ.

The separation mirrors a classic game engine (physics vs. graphics) but with learned components instead of hand‑coded rules. The result is a learned game engine that can stream generated pixels to up to four participants in real time, keeping latency low enough for interactive play.

How it works in practice

• Matchmaking: Players are paired into a deathmatch lobby. Each client streams its control inputs (move, fire, jump) to the central Agora‑1 server.
• State update: The dynamics model integrates those inputs, updates the global state (e.g., player A’s health drops, bullet trajectories change), and stores the new state.
• View generation: For each participant, the renderer receives the updated state plus the participant’s camera parameters and produces a fresh frame. All frames are sent back simultaneously, giving the illusion of a shared, physics‑driven world.

Because the state is explicit, developers can also edit it directly—creating new levels, tweaking weapon stats, or inserting novel objects—while the renderer automatically visualizes the changes.

Scaling beyond four agents

Agora‑1’s current prototype supports four participants, but the architecture is not fundamentally limited to that number. The dynamics model scales with the number of agents linearly (it simply processes a larger action vector), and the renderer can be parallelized across GPUs to produce additional viewpoints. In the long term, the team envisions a foundation world model where the state representation can encode entire cities, complex physics, or multi‑modal sensor streams, enabling hundreds of agents to co‑exist.

A new playground for multi‑agent reinforcement learning

Single‑agent world models restrict the diversity of interactions an RL agent can experience. Agora‑1 expands the interaction space combinatorially:

• Collision and coordination: Agents can learn to avoid each other, form formations, or block opponents.
• Adversarial training: Using the PROWL framework, an RL adversary can probe the shared simulation for failure modes, generating fresh data that improves both the dynamics and rendering models.
• Imagined training: Policies trained entirely inside the generated world can later be transferred to real games or physical robots, because the underlying state dynamics are grounded in the original GoldenEye ruleset.

Beyond games: collaborative robotics and shared simulations

The same decoupled pipeline can power any domain where multiple actors need a common situational picture:

• Co‑botic assembly lines where several manipulators coordinate to move parts without collisions.
• Multi‑drone navigation in shared airspace, with each drone receiving a personalized visual feed derived from a shared physics model.
• Education platforms where students manipulate a shared virtual lab, seeing instant visual feedback from a learned physics engine.

Early results and next steps

• Latency: Benchmarks show sub‑50 ms round‑trip times for four participants on a single A100 GPU, sufficient for responsive gameplay.
• Visual fidelity: The DiT renderer produces 720p frames that are indistinguishable from the original GoldenEye graphics to casual observers.
• Open research: The team plans to release the trained dynamics model and a lightweight renderer for the community, along with a Python API to plug in custom agents.

What to watch

• State scaling: Moving from discrete game variables to continuous physics (e.g., torque, friction) will demand new architecture tricks.
• Generalization: Extending the approach to unseen environments without retraining the dynamics model remains an open challenge.
• Safety: Multi‑agent simulations can produce emergent behaviors; robust monitoring will be essential before deploying in safety‑critical robotics.

Try it yourself

Odyssey invites researchers to experiment with Agora‑1 via their public demo portal. The system can be run locally with a modest GPU, and the codebase (including the dynamics model and DiT renderer) will be open‑sourced later this year.

---

Agora‑1 marks a step toward shared, learned simulations where humans and AI can interact on equal footing. By treating world dynamics and visual output as separate, trainable modules, the platform sidesteps the scaling bottlenecks of earlier attempts and opens a fertile ground for multi‑agent research across games, robotics, and education.