Eve: Shared-Memory Persistent Data Structures for ClojureScript and Clojure

Eve represents a significant advancement in shared-memory data structures for Clojure and ClojureScript ecosystems. This library provides persistent maps, vectors, sets, and lists backed by SharedArrayBuffer for in-process sharing or memory-mapped files for cross-process persistence, enabling multiple JVM, Node.js, and Babashka processes to share and atomically mutate Clojure data structures without traditional coordination mechanisms.

Revolutionary Architecture

The core innovation lies in Eve's ability to provide O(log32 N) persistent collections using 32-way branching HAMT (Hash Array Mapped Trie) maps, vectors, sets, and lists. This logarithmic complexity ensures that even a 1-billion-key map remains only 6 levels deep, providing exceptional performance characteristics. The swap latency remains constant regardless of atom size - JVM achieves p50 ~1 ms, Node.js p50 ~0.16 ms, and Babashka p50 ~1.4 ms, whether dealing with a 12 MB atom containing 1,800 keys or a 1 GB atom with 150,000 keys.

Cross-Process Coordination Without Locks

Traditional distributed systems often rely on coordination servers, locks, or leader election mechanisms to manage concurrent access to shared data. Eve eliminates these requirements through its lock-free Compare-And-Swap (CAS) operations on a single 32-bit root pointer. Multiple JVM, Node.js, and Babashka processes can simultaneously mutate the same atom without any coordination overhead. This approach enables true uncoordinated concurrency where processes operate independently while maintaining data consistency.

Memory-Mapped Persistence

Eve's memory-mapped file approach provides several advantages over traditional database systems. Data structures are stored directly in memory-mapped files that grow lazily from kilobytes to terabytes, supporting exabyte-scale durable atoms. The most significant benefit is that data survives process restarts without requiring export/import steps - the state persists automatically on disk. This eliminates the operational complexity typically associated with maintaining persistent state across application restarts.

In-Browser Shared Memory

For web applications, Eve leverages SharedArrayBuffer to enable web workers to share and atomically mutate persistent data structures without the overhead of postMessage serialization. This capability is particularly valuable for real-time collaborative applications and complex client-side state management where multiple workers need coordinated access to shared data.

Unified Platform Support

One of Eve's most compelling features is its unified approach across four platforms. Browser (ClojureScript), Node.js (ClojureScript), JVM (Clojure), and Babashka all use identical on-disk and in-memory layouts, hash functions, and CAS protocols. A domain created by one platform can be seamlessly joined by any other, providing true platform-agnostic data sharing. This consistency eliminates the impedance mismatch typically encountered when sharing data across different runtime environments.

Performance Characteristics

The performance profile of Eve is particularly impressive. Swap latency remains flat across atom sizes, as demonstrated by measurements on Linux with JDK 21, Node 18, and Babashka 1.x. A 11 MB atom with 2,812 keys achieves 0.71 ms swap latency on JVM, 0.09 ms on Node.js, and 0.38 ms on Babashka. Even at 1.1 GB with 62,012 keys, the latency increases only marginally to 0.82 ms, 0.07 ms, and 0.41 ms respectively. This flat performance curve is crucial for applications that need to scale their data structures without degrading operational latency.

Epoch-Based Garbage Collection

Eve implements a cooperative garbage collection system based on epochs, ensuring that old HAMT nodes are freed only after every reader has moved past them. This approach eliminates stop-the-world pauses that plague traditional garbage collection systems, maintaining application responsiveness even during cleanup operations. The epoch-based system provides a sophisticated balance between memory reclamation and read consistency.

Zero-Copy Reads

Another significant optimization is Eve's zero-copy read capability. The swap! operation walks mmap'd or SharedArrayBuffer memory directly without deserialization into intermediate heap objects. This direct memory access eliminates the overhead typically associated with reading persistent data structures, further enhancing performance for read-heavy workloads.

Practical Usage

Using Eve is straightforward across different platforms. For in-process atoms, no files on disk or native addons are required. The same API works across browser (SharedArrayBuffer), Node.js (SharedArrayBuffer), JVM (byte[] with sun.misc.Unsafe atomics), and Babashka (mmap files only). Named atoms using the same :id return the same atom across threads or workers, enabling consistent state management.

For cross-process persistent atoms, developers simply pass :persistent in the configuration map to store data on disk. The native addon is auto-loaded via node-gyp-build, making the setup process seamless. Process A can create a persistent atom, and Process B can join it using the same :id and path, automatically detecting and loading the existing atom's current value.

Building and Documentation

The project includes comprehensive documentation covering getting started guides, complete public API references, cross-process persistent atoms, data structures, specialized collections, typed shared objects, system architecture, internals, platform support, and testing. The build process is straightforward, with npm install handling dependency installation and native addon building automatically.

License and Availability

Eve is available under the MIT license, making it suitable for both open-source and commercial applications. The project is actively maintained on GitHub, with installation instructions provided for both deps.edn (for ClojureScript source) and package.json (for the native mmap addon).

Implications for Software Architecture

Eve represents a fundamental shift in how developers can approach shared state across processes and platforms. By eliminating the need for coordination servers, locks, and complex serialization protocols, it dramatically simplifies the architecture of distributed systems. Applications can now maintain a single source of truth that is accessible from multiple processes without the traditional complexity of distributed consensus algorithms.

The combination of persistent data structures, memory-mapped persistence, and cross-platform compatibility opens new possibilities for building scalable, fault-tolerant applications. From real-time collaborative editors to distributed databases, Eve provides the foundational building blocks for the next generation of shared-memory applications.

This technology is particularly relevant in the current landscape where applications increasingly need to operate across multiple processes, platforms, and devices while maintaining strong consistency guarantees. Eve's approach of using lock-free CAS operations and memory-mapped files provides a compelling alternative to traditional distributed systems architectures, potentially reducing both development complexity and operational overhead for many use cases.