Context
A product team needed to accept compute-heavy tasks without keeping web requests open or losing work when a process restarted. The key challenge was not raw throughput; it was making every transition observable and recoverable.
Constraints
- Individual jobs could run for several minutes.
- Clients needed a stable status model and safe retries.
- Workers had to be horizontally scalable.
- Operational complexity needed to stay modest for a small team.
Architecture
The API validates and records each job before publishing its identifier to a queue. Stateless workers claim jobs using a lease, write progress to PostgreSQL, and store short-lived coordination data in Redis. A bounded retry policy separates transient failures from terminal ones.
Technical decisions
- Used client-provided idempotency keys to prevent duplicate job creation.
- Modeled job status as an explicit state machine with guarded transitions.
- Added lease expiry so abandoned work can be recovered safely.
- Kept the database as the source of truth and the queue as delivery infrastructure.
Trade-offs
Exactly-once delivery was intentionally avoided. At-least-once processing plus idempotent handlers was simpler to reason about and easier to operate.
Outcome
The resulting design isolates long-running work from the request lifecycle and gives operators a clear path to inspect, retry, or cancel jobs.