I Built a Distributed Transaction Coordinator and Realized I Was Solving the Wrong Problem

Shipping TransactCo felt like an achievement. For six months, I'd built a system that could coordinate transactions across dozens of microservices. Two-phase commit protocols, compensating transactions, distributed locks, saga orchestration - if it related to distributed transactions, TransactCo had it. The system worked exactly as designed. When services needed to coordinate work, they'd delegate to TransactCo, which would ensure either all operations completed successfully or none did. By any conventional measure, the project was complete. Yet as I watched it handle production traffic, a nagging question persisted: why were we still seeing data inconsistencies?

The Illusion of Control

Distributed transaction systems are seductive. They promise the simplicity of ACID transactions across services that fundamentally can't share the same ACID guarantees. TransactCo delivered on that promise - it could pause services, roll back partial work, and maintain consistency logs. It was technically impressive. But impressive doesn't mean correct.

The problem became apparent during a routine deployment. We updated an order service, which triggered a cascade of compensating transactions across inventory, payment, and shipping services. TransactCo dutifully coordinated the rollback when the new version failed to start. Everything appeared to be consistent. Yet three days later, a customer reported receiving an item they hadn't paid for. The payment had been rolled back, the inventory restored, but a shipping service had processed an order from an older version that hadn't been properly coordinated.

This wasn't a bug in TransactCo. It was a fundamental limitation. The system could coordinate transactions it knew about, but it couldn't coordinate the entire state of the business. There were always paths through the system that bypassed the coordinator - background jobs, external integrations, manual interventions - and these paths created inconsistencies that only surfaced later.

What Distributed Transactions Actually Solve (And What They Don't)

TransactCo excelled at managing transaction lifecycles. It could:

• Pause services during critical sections
• Roll back operations when failures occurred
• Maintain logs of all transaction-related operations
• Enforce ordering across service calls

What it couldn't do was manage the actual business invariants. The coordinator knew about the mechanics of transactions, but not the semantics of the business domain. It couldn't understand that canceling an order should prevent shipping, even if the shipping service received the message before the cancellation was processed. It couldn't reason about the fact that a payment refund should only happen after the returned item is received, not just when the return request is initiated.

These aren't technical problems. They're domain problems that require understanding business rules, not just transaction protocols. The coordinator was a tool without a purpose - it could manage transactions, but it couldn't ensure the business remained consistent.

From Coordinator to Saga Orchestration

The breakthrough came when I stopped thinking about transactions as technical mechanisms and started thinking about them as business processes. Instead of building a generic coordinator, I built a domain-specific saga orchestrator for our order management domain.

Unlike TransactCo, which treated all transactions similarly, the saga orchestrator understands the specific business rules:

• When an order is placed, reserve inventory and initiate payment
• If payment fails, release the reserved inventory
• When payment completes, prepare the order for fulfillment
• If fulfillment fails, cancel payment (with possible delays for processing)
• When the order ships, finalize the payment

This approach has several advantages:

1. Business awareness: The orchestrator understands domain rules, not just technical constraints
2. Resilience: Each step can have its own retry and compensation logic
3. Observability: Business stakeholders can trace the entire process, not just technical transactions
4. Flexibility: The system can handle exceptions according to business rules, not generic rollback patterns

The Trade-Offs

This approach isn't without costs:

Complexity shifts from infrastructure to application: Instead of a generic coordinator managing transactions, each service must implement proper compensation logic. The payment service needs to know how to reverse payments, the inventory service needs to know how to release reservations, and so on.

Eventual consistency becomes explicit: The system acknowledges that some inconsistencies may temporarily exist and must be resolved through business processes rather than technical prevention.

Testing becomes more challenging: Testing a saga requires understanding the entire business process, not just verifying that transactions can be rolled back.

What I Wish I'd Known Sooner

Looking back, I see three critical mistakes in my approach to distributed transactions:

1. Confusing coordination with consistency: I assumed that coordinating transactions would ensure consistency, but consistency is a property of the entire system, not just the transactions. A system can have perfectly coordinated transactions and still be inconsistent if there are paths that bypass the coordinator.

2. Underestimating the role of domain knowledge: Generic solutions work for technical problems, but business consistency requires domain-specific understanding. No coordinator can replace the knowledge that business stakeholders have about how their domain works.

3. Over-indexing on prevention: I focused too much on preventing inconsistencies through technical mechanisms, rather than accepting that inconsistencies will occur and building processes to detect and resolve them.

The Path Forward

The saga-based approach has transformed how we think about distributed consistency. Instead of trying to prevent all inconsistencies, we focus on:

• Detecting inconsistencies through business process monitoring
• Containing inconsistencies through domain boundaries
• Resolving inconsistencies through compensating business processes

This doesn't mean we've abandoned distributed transactions. We still use them for services that can share ACID guarantees. But for cross-service coordination, we've embraced the fact that distributed systems are inherently inconsistent and that the solution isn't better coordination, but better business processes.

TransactCo still runs in production, but its role has changed. It's no longer the centerpiece of our consistency strategy. Instead, it's a specialized tool for coordinating services that can share ACID guarantees - a small but important part of a much larger system.

The lesson is clear: in distributed systems, the technical solution follows from the business problem, not the other way around. Building a generic coordinator without understanding the domain it serves is an exercise in futility. The real challenge isn't coordinating transactions - it's understanding the business well enough to know what needs to be coordinated in the first place.