Ursula – Distributed Event Streams Over HTTP, Backed by S3

Ursula is an open‑source server that lets developers treat append‑only event timelines as first‑class resources reachable via ordinary HTTP calls. It implements the Durable Streams Protocol (DSP) on top of a multi‑Raft architecture, persisting committed data in Amazon S3’s cold tier. The result is a system that preserves the four virtues most event‑stream platforms claim to offer—self‑hosting, low write latency, cheap storage, and quorum durability—while discarding the need for a dedicated broker network.

---

Core Argument: Why a New Kind of Stream Server?

Traditional streaming platforms such as Kafka or Pulsar were built around a broker‑centric model: a tightly coupled cluster of nodes that exchange data over a proprietary protocol, often requiring a VPN or private network. This model works well for on‑premises data centers but becomes fragile when the consumer base consists of browsers, mobile apps, or serverless functions that must reach the stream over the public internet.

Ursula’s thesis is that HTTP is the lingua franca of the modern web, and an event‑stream service should be reachable through the same stack that powers REST APIs and Server‑Sent Events (SSE). By exposing DSP over plain HTTP/SSE, Ursula eliminates the need for custom SDKs or private networking, allowing any HTTP‑capable client—curl, fetch, or a minimal Rust/TypeScript wrapper—to read, write, and tail streams.

---

Key Architectural Decisions

1. Multi‑Raft, Per‑Core Isolation

Ursula runs three (or five) identical processes that together form a single logical server. Each process spawns a thread per CPU core; each core owns a deterministic set of Raft groups. A stream’s name is hashed to a group ID, which maps to a specific core on every node. The group’s leader serialises writes, replicates them to the other replicas in that group, and acknowledges the client once a majority have persisted the entry.

Because groups are independent, there is no cross‑group transaction path. This design yields two benefits:

• Hot‑path simplicity – each core works with disjoint mutable state, avoiding lock contention across cores.
• Scalable durability – adding more cores or nodes simply adds more Raft groups, distributing load linearly.

2. HTTP Front Door with Axum

The HTTP layer is deliberately stateless. The axum framework parses incoming DSP requests, routes them to the appropriate core based on the stream hash, and returns responses. The mutable state (Raft log, in‑memory ring buffer, watchers) lives exclusively inside the group actor, keeping the front door lightweight and resilient to traffic spikes.

3. S3 as the Cold Store

Committed entries first reside in an in‑memory ring buffer and the Raft log. Background flushers periodically push older chunks to an S3 bucket. By leveraging S3’s standard storage class, Ursula inherits the durability guarantees of the service while keeping operating costs low—no need for a separate object store or a proprietary tiered storage system.

4. Stateless Clients via HTTP/SSE

Clients interact with streams using three simple verbs:

• PUT /<stream> – create a new stream.
• POST /<stream> – append raw bytes (Content‑Type: application/octet-stream).
• GET /<stream>?offset=-1&live=sse – tail the stream live via SSE.

Because the protocol is HTTP‑based, a browser can open an SSE connection and receive new events instantly, while a serverless function can append data with a single curl command.

---

Supporting Evidence: Benchmarks and Performance

Ursula was benchmarked on three c7g.4xlarge EC2 instances (each with 16 vCPU and 32 GiB RAM) forming a Raft quorum. The results are striking:

Metric	Ursula	Durable Streams (single node)	S2 Lite
Appends/sec (500 streams)	35.2 k	5.9 k	6.8 k
SSE fan‑out latency (p99, 1 000 subs)	6.1 ms	975 ms	110 ms

Ursula achieves a 5‑fold increase in throughput and a 160‑fold reduction in fan‑out latency compared to the reference Durable Streams implementation. The methodology matched hardware, network, and workload characteristics across all tests, ensuring an apples‑to‑apples comparison.

---

Implications for Developers and Operators

For Front‑End Engineers

The ability to tail a stream over SSE means collaborative editors, chat applications, and live dashboards can be built without a separate WebSocket gateway. A simple EventSource in the browser can subscribe to a stream and react to each new event as it arrives.

For Backend Teams

Because writes are acknowledged after a majority of Raft replicas have persisted, a single node failure does not jeopardise data integrity. At the same time, the use of S3 for long‑term storage eliminates the need to manage a separate log‑segment store, reducing operational overhead.

For Cost‑Conscious Operators

Storing the immutable log in S3’s standard tier avoids the per‑GB SaaS markup that many managed streaming services impose. The only additional cost is the transient memory used for the hot ring, which can be sized according to workload peaks.

---

Counter‑Perspectives and Open Questions

While Ursula’s design solves many pain points, a few considerations remain:

• Cold‑tier read latency – retrieving older events that have already been flushed to S3 incurs higher latency than reading from an in‑memory log. Applications that require frequent random access to historic data may need a caching layer.
• Dynamic membership – the current prototype assumes a static set of nodes. Adding or removing nodes requires a full cluster restart, which limits elasticity in cloud‑native environments. The roadmap lists dynamic Raft membership as a future feature.
• Conditional appends – the planned If-Match header for optimistic concurrency is not yet implemented, meaning concurrent writers must coordinate externally if they need strict ordering guarantees.

---

Roadmap Highlights

• If‑Match conditional appends – enable optimistic concurrency without external locks.
• Stateless WASM compute – attach deterministic WebAssembly modules to streams for on‑the‑fly compaction and state derivation.
• Dynamic Raft reconfiguration – support rolling upgrades and elastic scaling.
• Backup & restore tooling – provide a recovery path from the S3 cold tier in case of total cluster loss.
• Client SDKs – ergonomic Rust and TypeScript libraries that wrap the HTTP API for idiomatic usage.

---

Conclusion

Ursula demonstrates that a distributed, quorum‑replicated event stream can be built on top of familiar web primitives while still delivering the performance characteristics traditionally associated with specialized broker systems. By marrying Raft’s strong consistency guarantees with S3’s cheap durability and exposing the service via plain HTTP/SSE, Ursula offers a compelling alternative for teams that need reliable replayable timelines without the complexity of managing a Kafka‑style cluster.

The project is licensed under Apache 2.0 and welcomes contributions on GitHub. For a deeper dive, see the official documentation, the benchmark suite, and the source code at the tonbo‑io/ursula repository.