QueueGuard
A worker supervisor that makes silent background-job failures loud.
2025 · Backend — supervisor process, retry semantics, and the dashboard.
The problem
Background jobs failed silently in production and nobody noticed until customers did.
What it does
A lightweight worker supervisor with retry budgets, dead-letter inspection, and a one-page dashboard.
Engineering decisions
Retry budgets instead of fixed attempt counts
A fixed maximum-attempts number is wrong in both directions: it gives up too early on a dependency having a bad thirty seconds, and it hammers one that is genuinely down. Budgets are defined per time window instead, so a job retries freely during a blip and backs off hard during an outage without anyone tuning a constant.
Dead letters are inspectable, not just logged
A failed job in a log line is an archaeology exercise. Each dead letter stores the full payload and the chain of failures that produced it, and can be replayed in one click once the underlying cause is fixed — which turned incident recovery from a manual re-run script into a button.
One page, deliberately
The temptation with job infrastructure is to build an analytics product. The dashboard answers three questions — what is failing, why, and can I replay it — and nothing else. Anything that did not serve those questions was left out, which is why it stayed maintainable as a side concern rather than a product.
Where it landed
- Silent failures surface within a minute instead of via customer reports
- Recovery is a replay rather than a hand-written re-run script
- Runs on the Redis already in the stack — no new infrastructure