SQLite‑Based Job Queue Beats Cloud Brokers on an Android Phone

The problem: heavyweight brokers for tiny workloads

Independent developers and hobbyists often need a reliable way to hand off work from a cloud script to a local device – a Raspberry Pi, a Termux‑enabled phone, or any edge node behind a NAT. The usual recommendation is to stand up a full‑blown broker such as RabbitMQ, Redis, or Celery. Those systems consume megabytes of RAM, require persistent daemons, and usually involve a managed cloud plan that adds cost and vendor lock‑in. Opening inbound ports or maintaining reverse SSH tunnels is another option, but it introduces a large attack surface.

The solution: Intent Bus, a SQLite‑driven queue

Intent Bus implements a minimal Flask API that stores jobs in a single SQLite file. Workers poll the /jobs endpoint, acquire an exclusive lease, execute the payload, and report completion. The design relies on three SQLite features:

1. Write‑Ahead Logging (WAL) – allows concurrent readers while a single writer appends new rows.
2. Atomic row‑level locking – implemented with a lease_until timestamp column, preventing two workers from picking the same job.
3. Dead‑letter handling – jobs that exceed a retry count are moved to a separate table for later inspection.

All of this runs on a device that only needs outbound HTTP, so firewalls are no longer a problem.

Stress‑test methodology

We measured four load profiles across three environments:

Profile	Workers	Total jobs
Low	5	50
Medium	15	500
Heavy	40	2,000
Extreme	100+	5,000

The environments were:

• PythonAnywhere (free tier) – single‑threaded request handling.
• Render (free tier Docker) – multi‑threaded container.
• Android 12 phone (Termux) – native ARM CPU + flash storage.

Metrics collected: throughput (jobs / sec), P99 latency, success rate, and error counters.

Cloud baseline results

• PythonAnywhere collapsed at the Medium load; the single request thread became a bottleneck, producing 100 % network errors.
• Render handled Medium load with 99.5 % success, averaging 11.30 jobs/sec and a P99 latency of 0.517 s. At Heavy load the system sustained 14.18 jobs/sec with a P99 of 2.891 s. No jobs were lost, and the SQLite WAL kept read contention low.

These numbers already proved that a lightweight SQLite queue can survive realistic multi‑worker traffic when the host process is truly concurrent.

Android experiment – the edge wins

Running the same Flask server inside Termux on a standard Android 12 phone yielded surprising performance:

• Heavy load (40 workers, 2,000 jobs) produced 28.04 jobs/sec, more than double the Render container, with a P99 of 2.556 s.
• The phone’s flash storage avoided the hypervisor I/O throttling that free‑tier containers suffer from, allowing SQLite writes to complete in microseconds.

Pushing to the extreme

We launched the Extreme profile: 100+ workers continuously feeding 5,000 jobs over 267 seconds. The phone throttled under heat, and throughput settled at 18.52 jobs/sec. P99 latency rose to 9.002 s, but the success rate stayed at 98.89 %. Crucially, zero network drops, lease losses, or publish rejections were recorded. The locking logic held steady even when the device approached its thermal limits.

Trade‑offs and when to use this pattern

Aspect	SQLite‑WAL queue	Traditional broker
Resource footprint	< 20 MB RAM, single file on disk	Hundreds of MB RAM, multiple processes
Operational complexity	One HTTP endpoint, no daemon management	Cluster setup, monitoring, scaling
Throughput ceiling	Thousands of jobs/sec on modest hardware	Tens of thousands+ with sharding
Failure modes	Disk I/O contention, single writer lock	Network partitions, broker node failures
Consistency model	Strong ACID per job, linearizable reads/writes	Configurable (at‑least‑once, exactly‑once)

For indie projects, home automation, or occasional batch jobs, the SQLite approach gives predictable latency, zero cloud cost, and simple deployment. It is not a substitute for Kafka‑style pipelines that must handle millions of events per second across data centers.

Practical takeaways for developers

1. Configure WAL correctly – set journal_mode=WAL and synchronous=NORMAL to balance durability and speed.
2. Keep jobs small – store only identifiers and minimal payload; heavy data should be fetched from object storage.
3. Use exponential backoff on lease acquisition failures to avoid hammering the DB under contention.
4. Monitor file descriptor usage – on Android, the default limit can be reached with many workers; raise it via ulimit if needed.
5. Graceful shutdown – ensure workers release leases on SIGTERM to prevent orphaned jobs.

Where to find the code

• Server implementation – https://github.com/dsecurity49/Intent-Bus
• Python SDK – https://github.com/dsecurity49/Intent-Bus-sdk
• Product Hunt launch – https://www.producthunt.com/products/intent-bus

The repository includes a Dockerfile for cloud deployments and a Termux‑ready install.sh script for edge devices. CI pipelines run SQLite integrity checks on every push, guaranteeing that the database schema never drifts.

---

Bottom line

A well‑tuned SQLite WAL backend can act as a reliable, low‑cost job queue for thousands of concurrent tasks, even on an old Android phone. For developers who value simplicity and want to avoid the operational overhead of traditional brokers, Intent Bus demonstrates that “the cloud is not always the answer” – sometimes the edge device you already have in your pocket is more than enough.