Building GoRelay: A Pragmatic Solution for Durable Task Queues in Go

The Problem: Silent Failures in Task Queues

Every Go developer has encountered the frustration of a task queue failing silently. A critical function like go sendEmail(user) might work initially but crash later, leaving tasks unprocessed and users without essential communications. This issue isn't just inconvenient—it can lead to lost data, damaged user trust, and costly downtime. Traditional solutions often require setting up Redis, configuring workers, and writing boilerplate code, which adds complexity and maintenance overhead.

The Solution: GoRelay

GoRelay is a durable task queue library designed to address these pain points. It allows Go developers to turn any function into a crash-resistant, retryable background job with minimal setup. The API is straightforward: register a function and enqueue tasks. For example, r.Register("email.send", SendEmail) and r.Enqueue("email.send", EmailPayload{To: "[email protected]"}) handle the rest.

What Makes It Different

GoRelay stands out through three key features:

1. Zero Infrastructure Setup: No need for Redis, PostgreSQL, or external services. A simple go get and you're ready to use.
2. Flexible Storage Backends: Supports SQLite (for zero-config development), PostgreSQL (for ACID compliance in production), and Redis (for high-throughput scenarios) with the same interface.
3. Built-in Dashboard: A real-time interface for monitoring task status, history, and errors, eliminating the need for third-party tools.

Retries are automatic with exponential backoff (1s, 2s, 4s, up to 1 hour) and random jitter to prevent thundering herd issues when services recover.

The Journey to v1.0

Building GoRelay as an open-source project taught the author valuable lessons. Starting with an API-first approach ensured the library met real-world needs. Abstracting storage was critical—users could start with SQLite and scale to PostgreSQL or Redis without code changes. The dashboard, initially considered optional, became a core feature, providing transparency into task processing.

How It Works Under the Hood

GoRelay's performance is driven by a lock-free ring buffer for task passing, eliminating mutex contention and goroutine blocking. Tasks are processed in priority order (High, Normal, Low), ensuring critical operations like payments aren't delayed by less urgent tasks. Automatic retries with exponential backoff and random jitter ensure reliability without manual intervention.

Real-World Use Cases

GoRelay is already being used for:

• Welcome emails after user signup
• PDF report generation
• SMS notifications
• Data processing pipelines
• Scheduled maintenance tasks

Lessons Learned

The author emphasizes the importance of documentation, testing, and iterative releases. Spending 40% of time on READMEs, examples, and godoc was worth it, as users judge a library by its documentation first. Tests caught unexpected bugs, and the go test -race command proved invaluable. Releasing early (v1.0) allowed for feedback-driven improvements.

Performance Considerations

GoRelay prioritizes efficiency:

• Lock-free ring buffer: Zero contention during normal operation.
• Zero-copy JSON: Efficient for small payloads.
• Batch writes: Reduce database round trips. Throughput depends on the storage backend: SQLite suits development, PostgreSQL offers production reliability, and Redis excels in high-volume scenarios. Memory usage typically stays under 50MB for most workloads.

Try GoRelay Today

Install with go get github.com/amitstephen-dev/[email protected]. A complete example is available in the documentation. Open the dashboard at http://localhost:8080 to monitor tasks in real-time.

Links

• GitHub
• Documentation
• Report Issues

Closing Thoughts

GoRelay demonstrates that simple tools can solve complex problems. By focusing on pragmatism—zero infrastructure, flexibility, and reliability—it offers a compelling alternative to traditional task queue systems. If GoRelay saves you time or solves a problem, consider starring the repo on GitHub to help others find it.

Built with love for the Go community