Postgres as Orchestrator: A New Approach to Durable Workflows

The durable workflow pattern has become increasingly popular in the developer community as a way to build reliable applications that can recover from failures. The traditional approach, implemented by systems like Temporal, Airflow, and AWS Step Functions, relies on external orchestration where a central server coordinates workflow execution across multiple workers. However, a growing chorus of developers and architects is questioning whether this complexity is necessary.

The fundamental concept of durable workflows is straightforward: checkpoint program state regularly to a database so that if the program crashes, it can resume from the last checkpoint. This is analogous to saving progress in a video game. Yet, the implementation has typically involved a complex external orchestrator that manages workflow state, dispatches tasks to workers, and handles recovery when workers fail.

The argument against external orchestration centers on a simple observation: if durable workflows are fundamentally about database persistence, then why introduce a separate orchestrator layer? This question has led to the emergence of database-backed workflow execution models, particularly those leveraging Postgres, which has become the de facto choice for many applications due to its reliability, scalability, and rich ecosystem.

The evidence for this shift is growing. Projects like DBOS are demonstrating that application servers can directly communicate with Postgres to execute workflows, bypassing the need for a central orchestrator. In this model, workflows are represented as entries in a Postgres table, and application servers cooperatively dequeue and execute these workflows using database locking mechanisms to ensure each workflow is processed by exactly one worker.

Proponents of this approach highlight several advantages. First, it simplifies the architecture by removing a separate orchestrator component. Second, it leverages decades of database research and engineering for scalability and availability. Postgres can handle tens of thousands of workflows per second on a single server and can be further scaled using distributed or sharded deployments. Third, observability becomes straightforward since workflow and step data is stored in regular tables and can be queried with standard SQL.

The security and reliability benefits are also compelling. With external orchestration, both the orchestrator and its data store become single points of failure. By contrast, a Postgres-backed system has only the database as a potential single point of failure, and since most applications already depend on Postgres, no new critical infrastructure is introduced.

However, this approach is not without its critics. Some argue that external orchestrators provide valuable abstractions that simplify workflow development, such as built-in support for retries, timeouts, and complex branching logic. Others question whether Postgres can handle the operational complexity of workflow orchestration at scale, particularly for organizations without deep Postgres expertise.

The community sentiment appears divided between those who appreciate the simplicity and operational benefits of database-backed workflows and those who value the higher-level abstractions provided by specialized orchestration systems. Adoption signals suggest that organizations with strong database expertise and existing Postgres investments are more likely to explore this approach, while those requiring sophisticated workflow features may prefer established external orchestrators.

Despite these counter-perspectives, the trend toward leveraging databases directly for workflow coordination seems to be gaining momentum, particularly as organizations seek to reduce operational complexity and leverage existing infrastructure investments. The success of projects like DBOS will likely depend on their ability to provide comprehensive workflow abstractions while maintaining the simplicity and operational benefits of the database-backed approach.

For developers interested in exploring this approach, resources like the DBOS quickstart, GitHub repository, and Discord community offer opportunities to learn more and contribute to this emerging pattern.