ExtendDB Brings DynamoDB‑Compatible API to Pluggable Storage Backends, Starting with PostgreSQL

---

Technical announcement

AWS has released ExtendDB, an open‑source adapter that speaks the DynamoDB wire protocol while delegating actual data storage to a configurable backend. The initial release ships with a PostgreSQL backend, but the design is deliberately backend‑agnostic: any storage engine that implements a small Rust trait can be plugged in. ExtendDB runs as a single binary, requires no external runtime, and supports the full suite of AWS SDKs, CLI tools, and DynamoDB‑compatible libraries without code changes.

The project is hosted on GitHub under the Apache‑2.0 license (github.com/aws/extenddb). It targets Rust 1.85+ and PostgreSQL 14+, and is currently at version 0.1.0. While the feature set is not yet on par with DynamoDB Local, the roadmap promises broader API coverage and production‑grade performance.

---

Specification and architecture

Core components

Crate	Responsibility
extenddb-core	Types, validation, expression parsing – pure synchronous Rust.
extenddb-engine	Implements DynamoDB control‑plane (CreateTable, DeleteTable, etc.) and data‑plane semantics (PutItem, UpdateItem, Query, Scan).
extenddb-storage-postgres	PostgreSQL backend implementation built against the StorageBackend trait.
extenddb-server	HTTP server, management API, and optional web console.

The storage trait defines a minimal set of operations (read, write, conditional update, range scan, stream emit). Because the trait is synchronous, backends can be wrapped in async adapters if needed, but the core engine remains single‑threaded and deterministic.

Wire protocol handling

ExtendDB implements the DynamoDB HTTP/JSON protocol and SigV4 request signing. On first start it generates a self‑signed TLS certificate and a local credential store that mimics IAM, allowing existing SDK credential providers to work unchanged.

Deployment model

Scenario	Recommended setup
Local development / CI	Run the binary in a Docker container, mount a temporary PostgreSQL instance (e.g., postgres:15-alpine).
On‑prem production	Deploy ExtendDB as a systemd service behind an internal load balancer; configure PostgreSQL with high‑availability (Patroni or similar).
Hybrid cloud	Use ExtendDB in a VPC‑peered environment, connecting to an on‑prem PostgreSQL cluster while the application runs in AWS Lambda or ECS.

The binary has no external dependencies beyond the PostgreSQL client library, making it suitable for minimal container images (~30 MB). Configuration is supplied via environment variables (EXTENDDB_POSTGRES_URL, EXTENDDB_PORT, EXTENDDB_TLS_CERT, etc.).

---

Benchmarks and performance considerations

A community member posted early benchmark results on Reddit (concurrency 10 on a 16‑core MacBook):

• UpdateItem: 300 ms p90 latency
• GetItem: 20 ms p90 latency

These numbers reflect the current prototype implementation, which performs a full round‑trip through the Rust engine, SQL translation, and PostgreSQL transaction handling. The authors of a competing project, Rektifier, reported lower latencies (≈80 ms p90 for updates) by using prepared statements and connection pooling.

Bottleneck analysis

1. Synchronous engine – The core engine processes each request on a single thread. Scaling horizontally requires running multiple ExtendDB instances behind a load balancer.
2. SQL translation overhead – Complex DynamoDB expressions (e.g., ConditionExpression with nested functions) are parsed and then converted to equivalent SQL. The conversion path is not yet optimized for bulk operations.
3. PostgreSQL transaction cost – Each DynamoDB write translates to a PostgreSQL INSERT … ON CONFLICT … upsert. Without explicit batch support, high write rates incur per‑statement overhead.

Optimisation pathways

• Async engine: Refactor the engine crate to use tokio and async I/O, allowing multiple concurrent requests per process.
• Prepared statement cache: Cache generated SQL per table to avoid repeated parsing.
• Batch APIs: Implement BatchWriteItem and BatchGetItem by mapping to PostgreSQL COPY and SELECT … IN (…) patterns.
• Backend‑specific tuning: For PostgreSQL, enable max_parallel_workers_per_gather and tune shared_buffers to accommodate DynamoDB‑style hot‑spot access patterns.

---

Real‑world implications

Development velocity

Because ExtendDB mirrors the DynamoDB API, developers can run integration tests locally without provisioning an AWS account or paying for provisioned throughput. CI pipelines can spin up a lightweight PostgreSQL container, start ExtendDB, run the test suite, and tear down—all within a few seconds.

Compliance and data residency

Enterprises with strict data‑sovereignty requirements can keep the actual data on‑prem (or in a regulated cloud region) while still using DynamoDB‑compatible SDKs. This reduces the need to rewrite application code when moving workloads between AWS and private clouds.

Migration pathways

1. Lift‑and‑shift: Deploy ExtendDB with a PostgreSQL backend that mirrors the schema of an existing DynamoDB table. Export data from DynamoDB via aws dynamodb export-to-point-in-time and import into PostgreSQL using COPY.
2. Hybrid read‑through: Run ExtendDB in front of a read‑replica of DynamoDB (via DynamoDB Streams → PostgreSQL sync). Applications continue to read from ExtendDB while writes go directly to DynamoDB, enabling gradual migration.

Limitations to be aware of

• Feature parity: Advanced features like Global Tables, Transactions, and TTL are not yet supported.
• Performance ceiling: Current single‑process design caps throughput at roughly 1 k ops/s on commodity hardware; production workloads will need horizontal scaling.
• Operational complexity: Managing PostgreSQL tuning and ensuring data model compatibility adds operational overhead compared to a fully managed DynamoDB service.

---

Outlook

ExtendDB demonstrates a pragmatic approach to decoupling the DynamoDB API from its proprietary storage engine. By exposing a clean Rust trait for storage backends, the project invites community contributions for Cassandra, MySQL, or even distributed KV stores like TiKV. If the roadmap delivers higher parity and async scaling, ExtendDB could become a viable option for hybrid cloud strategies, on‑prem testing environments, and cost‑sensitive workloads that do not require the full scalability guarantees of native DynamoDB.

---

For a hands‑on start, clone the repository, run cargo build --release, and launch with ./target/release/extenddb --postgres-url postgresql://user:pass@localhost/extenddb. The built‑in web console is available at https://localhost:8080/console.