The Lifecycle of a BullMQ Retry — A Deep Dive into Redis Internals

You configure attempts: 5 and backoff: { type: 'exponential', delay: 1000 }. Your job fails. BullMQ retries it. But what actually happens inside Redis?

Most developers configure retries without ever understanding the underlying mechanism. BullMQ's retry system is powered by Lua scripts executing atomically on Redis — no race conditions, no partial states, no lost jobs. It's an elegant state machine that operates entirely within Redis's single-threaded event loop, and understanding it is the key to debugging retry storms, optimizing backoff strategies, and using monitoring tools like QueueHub effectively.

In this post, we trace the exact Redis operations from job failure → retry decision → backoff delay → reprocessing. We'll cover all three retry paths — automatic, manual, and stalled-job recovery — with real Lua script references, Redis field names, and production debugging scenarios.

New to BullMQ retries? Read our companion guide BullMQ Job Retry Strategies first for the configuration side, then come back here for the deep dive.

---

The Redis Key Structure for Queue State

BullMQ stores queue state across roughly 15+ Redis keys per queue. Understanding these keys is essential to following the retry lifecycle. Every retry path reads from or writes to one or more of these keys atomically.

Queue Key Reference Table

Redis Key Pattern	Data Type	Purpose
bull:{queueName}:wait	LIST	Jobs waiting to be processed (FIFO order)
bull:{queueName}:active	LIST	Jobs currently being processed by a worker
bull:{queueName}:delayed	ZSET	Jobs scheduled for future processing (score = target timestamp)
bull:{queueName}:failed	ZSET	Permanently failed jobs (score = fail timestamp)
bull:{queueName}:completed	ZSET	Successful jobs (score = completion timestamp)
bull:{queueName}:prioritized	ZSET	Jobs with a numeric priority > 0
bull:{queueName}:stalled	SET	Jobs flagged as potentially stalled — checked by workers
bull:{queueName}:marker	LIST	Queue markers enabling efficient blocking pops
bull:{queueName}:events	STREAM	BullMQ event stream: waiting, active, completed, failed, retries-exhausted
bull:{queueName}:{jobId}	HASH	Individual job data: data, opts, atm, failedReason, stacktrace, etc.
bull:{queueName}:{jobId}:lock	STRING	Worker lock key — TTL enforced via PX

Inspecting Queue Keys Directly

You can inspect these keys without the BullMQ SDK using any Redis client. This is invaluable for debugging retry issues in production:

import IORedis from 'ioredis';

const redis = new IORedis({ host: 'localhost', port: 6379 });

// See the waiting job IDs (FIFO order)
const waitingJobs = await redis.lrange('bull:myqueue:wait', 0, -1);

// See all failed job timestamps and IDs
const failedJobs = await redis.zrange(
  'bull:myqueue:failed', 0, -1, 'WITHSCORES'
);

// Inspect a specific job's hash for retry state
const jobData = await redis.hgetall('bull:myqueue:12345');
console.log({
  attemptsMade: jobData.atm,       // attempt counter
  failedReason: jobData.failedReason,
  data: jobData.data,
});

Key insight: The failed set is a ZSET, not a LIST. This isn't an implementation quirk — it enables efficient time-range queries so the cleanJobsInSet Lua script can prune old failed jobs with a single ZREMRANGEBYSCORE call. The same design applies to delayed and completed ZSETs.

The Job Hash Fields That Drive Retries

Every job hash contains several fields directly relevant to the retry lifecycle:

• atm — attemptsMade (incremented on every retry)
• ats — attemptsStarted (incremented when a worker picks up the job)
• stc — stalledCount (incremented each time the job is flagged as stalled)
• opts — serialized job options including attempts, backoff, delay
• failedReason — the error message from the last failure
• stacktrace — accumulated error stack traces
• finishedOn — timestamp of last completion or failure

---

Path A: Automatic Retry on Failure (moveToFinished-14.lua)

The automatic retry path is triggered by the moveToFinished-14.lua script — called by every worker when a job's processor either completes successfully or throws an error. This is the most common retry path and the one you rely on when you configure attempts: 5 on a queue.

Step-by-Step Breakdown

When a job fails, moveToFinished-14.lua executes the following steps atomically inside Redis:

1. Lock released — The removeLock() Lua helper deletes the {jobId}:lock key and removes the jobId from the stalled SET
2. Job removed from active list — LREM bull:myqueue:active -1 {jobId}
3. Increment atm — HINCRBY bull:myqueue:{jobId} atm 1 (the attemptsMade counter)
4. Store failure data — HSET writes failedReason, finishedOn, stacktrace onto the job hash
5. Decision point — retry or permanent failure?

The Retry Decision Logic

The Lua pseudocode for the retry decision looks like this:

-- moveToFinished-14.lua decision logic (simplified)
local attemptsMade = rcall("HGET", jobKey, "atm")  -- attemptsMade
local attempts = rcall("HGET", jobKey, "opts").attempts  -- max attempts

if attemptsMade < attempts then
    -- RETRY PATH: calculate backoff, move to delayed or wait
    local delay = calculateBackoff(attemptsMade, backoffType, backoffDelay)
    if delay > 0 then
        moveToDelayed(jobKey, delay)  -- ZADD to delayed set
    else
        addToWaitList(jobKey)  -- LPUSH/RPUSH to wait list
    end
else
    -- PERMANENT FAILURE PATH
    rcall("ZADD", failedSetKey, timestamp, jobId)
    rcall("XADD", eventsStream, "*",
          "event", "retries-exhausted",
          "jobId", jobId)
end

In the actual BullMQ v5 source, the backoff calculation and retry-or-fail logic are embedded inside moveToFinished-14.lua as a single atomic transaction. This means a job cannot be lost between "failing" and "being retried" — it's all or nothing. If the Redis instance crashes in the middle, the entire transaction is rolled back.

State Machine Transitions

The flow for automatic retry follows this state machine:

waiting → active → failed → [delayed (if backoff > 0)] → waiting → ...
                       ↓ (no more attempts)
                   ZADD bull:myqueue:failed
                   (retries-exhausted event)

Tracing Retries with Job Events

You can observe every state transition using BullMQ's QueueEvents listener:

import { QueueEvents } from 'bullmq';

const queueEvents = new QueueEvents('myqueue', { connection });

queueEvents.on('failed', ({ jobId, failedReason, prev }) => {
  console.log(`[FAILED] Job ${jobId}: ${failedReason}`);
});

queueEvents.on('retries-exhausted', ({ jobId }) => {
  console.log(`[EXHAUSTED] Job ${jobId} — no more retries`);
});

queueEvents.on('waiting', ({ jobId, prev }) => {
  console.log(`[RETRY] Job ${jobId} moved from ${prev} → waiting`);
});

The prev field in the waiting event tells you the previous state — active for a fast retry with no delay, or delayed for a backoff-triggered retry.

---

Path B: Manual Retry via APIs (retryJob-11.lua and reprocessJob-8.lua)

Manual retries are triggered when you call job.retry(), use the QueueHub dashboard's "Retry" button, or run a bulk retry operation. These go through retryJob-11.lua or reprocessJob-8.lua, depending on whether you want to reset the attempt counter.

The retryJob-11.lua Script

Key inputs to the script:

Parameter	Purpose
KEYS[1]	Active list key
KEYS[2]	Wait list key
KEYS[3]	Paused list key
KEYS[4]	Job hash key
KEYS[11]	Stalled set key
ARGV[5]	Worker token (lock verification)

The KEYS[n] vs ARGV[n] distinction matters because Redis determines cluster hash slots from KEYS only — this ensures all operations land on the same cluster node.

What retryJob-11.lua does atomically:

1. Check job exists — EXISTS bull:myqueue:{jobId} → returns -1 if missing
2. Remove lock — deletes {jobId}:lock key; removes jobId from stalled SET; returns -2 if no lock
3. Remove from active — LREM bull:myqueue:active -1 {jobId}; returns -3 if not found in active
4. Re-add to target queue — uses the addJobInTargetList helper, which handles paused queues and priority sorting
5. Increment atm — HINCRBY bull:myqueue:{jobId} atm 1
6. Emit waiting event — XADD bull:myqueue:events * event waiting jobId {id} prev active

The reprocessJob-8.lua Script

When you retry with resetAttemptsMade: true, BullMQ calls reprocessJob-8.lua instead:

• Same steps as retryJob-11.lua, but additionally resets atm (attemptsMade) and ats (attemptsStarted) to 0
• Restores parent dependency links so parent jobs can re-collect their children
• Used by QueueHub's "Retry with fresh attempts" button — ideal when you've fixed the underlying bug and want a clean slate

import { Queue, Job } from 'bullmq';

const queue = new Queue('myqueue', { connection });

// Retry a failed job from QueueHub or custom dashboard
const job = await Job.fromId(queue, 'failed-job-42');

// Retry with attempt reset — gives the job a full retry budget
await job.retry('failed', { resetAttemptsMade: true });

// Verify the reset
const refreshedJob = await Job.fromId(queue, 'failed-job-42');
console.log(`Attempts made: ${refreshedJob.attemptsMade}`); // 0

---

Path C: Stalled Job Recovery (moveStalledJobsToWait-9.lua)

Stalled jobs are the silent failure mode of any queue system. A job is "stalled" when a worker grabs the lock and starts processing but never completes — the worker crashed, the event loop blocked longer than lockDuration (default 30s), or the worker never called job.updateProgress() or returned a result.

BullMQ handles this with the moveStalledJobsToWait-9.lua script, which runs periodically on every worker.

The Stall Detection Algorithm

1. Stalled check gate — checks if bull:{queue}:stalled-check key exists (acts as a mutex set with PX expiry to prevent double execution across workers)
2. Read current stalled set — SMEMBERS bull:{queue}:stalled returns all jobs flagged as potentially stalled
3. For each job: check if the lock key still exists → if no lock, it's truly stalled
4. Stall counter — HINCRBY {jobKey} stc 1 (increments the stalledCount field in the job hash)
5. If stc > maxStalledCount (default 1): the job is permanently failed with reason "job stalled more than allowable limit"
6. Otherwise: job is moved back to the wait list via the moveJobToWait helper

Handling Stalled Jobs in Your Worker

The best defense against false stalls is to emit progress regularly. This keeps the lock active and signals to BullMQ that the worker is alive:

const worker = new Worker(
  'myqueue',
  async job => {
    await job.updateProgress(10);
    // ... do work ...
    await job.updateProgress(50);
    // ... more work ...
    await job.updateProgress(90);
    return result;
  },
  {
    connection,
    stalledInterval: 30000, // how often to check for stalls (ms)
    maxStalledCount: 2,     // retry twice before permanent failure
  }
);
``\

When a job is repeatedly stalling, QueueHub surfaces it in the stalled jobs dashboard, showing the `stc` counter and the lock history so you can investigate the root cause.

---

## Backoff Mechanics: The Delayed ZSET

Backoff delays are where Redis's ZSET data structure truly shines. Every delayed retry is stored as a ZSET member with a score equal to the target Unix timestamp (in milliseconds).

### The Delayed ZSET Structure

- **Key:** `bull:{queue}:delayed`
- **Score:** Unix timestamp (ms) when the job should become available
- **Member:** Job ID

A separate scheduled job called `promoteDelayedJobs` runs periodically:

1. `ZRANGEBYSCORE bull:myqueue:delayed 0 {now}` — finds jobs whose scheduled time has arrived
2. Moves each job from the delayed ZSET back to the wait LIST
3. The worker picks it up on the next `BRPOPLPUSH` cycle

### Backoff Calculation Logic

The backoff calculation happens inside the Lua script, not in application code. Here's the TypeScript equivalent of the Lua logic:

```typescript
// Equivalent of BullMQ's backoff calculation (matching Lua logic)
function calculateBackoffDelay(
  attemptsMade: number,
  type?: 'fixed' | 'exponential' | 'custom',
  delay: number = 0,
  jitter: number = 0,
  customStrategy?: (attempts: number) => number
): number {
  let baseDelay = 0;

  if (type === 'fixed') {
    baseDelay = delay;
  } else if (type === 'exponential') {
    // 2^(attemptsMade - 1) * delay
    baseDelay = Math.pow(2, attemptsMade - 1) * delay;
  } else if (type === 'custom' && customStrategy) {
    baseDelay = customStrategy(attemptsMade);
  }

  // Jitter applied after base calculation
  if (jitter > 0) {
    const min = Math.min(baseDelay, baseDelay * jitter);
    const max = Math.max(baseDelay, baseDelay * jitter);
    baseDelay = min + Math.random() * (max - min);
  }

  return baseDelay;
}

With exponential backoff, a job's delay sequence looks like this:

Attempt	Delay (exponential, base=1000ms)	Redis Score
1	1000ms (2⁰ × 1000)	now + 1000
2	2000ms (2¹ × 1000)	now + 2000
3	4000ms (2² × 1000)	now + 4000
4	8000ms (2³ × 1000)	now + 8000
5	16000ms (2⁴ × 1000)	now + 16000

State Inspection with getStateV2-8.lua

The getStateV2-8.lua script can tell you exactly what state a job is in by checking which Redis key it belongs to. This is how QueueHub shows live retry state — it checks the job's presence in wait, active, delayed, completed, or failed keys in priority order and returns the first match.

---

How QueueHub Surfaces Retry State

Understanding the Redis internals gives you the knowledge to debug retries manually, but production observability tools like QueueHub automate this work by surfacing the exact same data through a clean UI.

What QueueHub Shows for Retrying Jobs

• Job detail view — attemptsMade vs maxAttempts progress bar showing how many retries are left
• State indicator — clearly distinguishes between "retrying" (job in delayed → waiting transition), "temporarily failed" (awaiting backoff), and "permanently failed"
• Retry timeline — visual timeline of each attempt with the backoff gap between them
• Bulk retry — calls retryJob-11.lua or reprocessJob-8.lua per job with optional resetAttemptsMade

How QueueHub Fetches Retry State

Here's a simplified version of QueueHub's retry state inspector:

// Simplified version of QueueHub's retry state inspector
async function getRetryState(queue: Queue, jobId: string) {
  const job = await Job.fromId(queue, jobId);
  if (!job) return null;

  const state = await job.getState();
  const remainingAttempts =
    (job.opts.attempts ?? 1) - job.attemptsMade;

  let nextRetryAt: Date | null = null;
  if (state === 'delayed' && job.delay) {
    // Job is in the delayed ZSET — waiting for backoff timer
    nextRetryAt = new Date(job.delay);
  }

  return {
    jobId,
    state, // 'failed' | 'delayed' | 'waiting' | 'active'
    attemptsMade: job.attemptsMade,
    maxAttempts: job.opts.attempts ?? 1,
    remainingAttempts,
    failedReason: job.failedReason,
    nextRetryAt, // null if permanently failed
    stacktrace: job.stacktrace,
  };
}

QueueHub's dashboard organizes this data into actionable views — failed job lists filtered by retries remaining, retry storm alerts based on delayed ZSET cardinality, and bulk retry workflows that leverage the atomic Lua scripts.

---

Debugging Retry Issues with Redis Internals

Knowing how the internals work helps you diagnose real-world retry problems before they escalate.

Scenario 1: Retry Storm (Thundering Herd)

Multiple jobs fail at once with exponential backoff. All of them come out of the delayed ZSET at roughly the same time, creating a cascade of simultaneous retries.

Detection: Check the delayed ZSET cardinality — ZCOUNT bull:queue:delayed and look for time-clustered entries with similar scores.

Fix: Add jitter to spread out the retry window:

{
  attempts: 5,
  backoff: { type: 'exponential', delay: 1000, jitter: 0.3 }
}

Scenario 2: Infinite Retry Loop

A bug in the processor causes a transient error pattern that never resolves. attemptsMade keeps incrementing but the job never reaches max.

Detection: Scan job hashes for high atm values:

// Find jobs with excessive retries — useful for QueueHub's "retry storm" alert
async function findRetryLoopCandidates(
  redis: IORedis,
  queueName: string
) {
  const failedJobs = await redis.zrange(
    `bull:${queueName}:failed`, 0, 99
  );
  const candidates = [];

  for (const jobId of failedJobs) {
    const job = await Job.fromId(new Queue(queueName), jobId);
    if (job && job.attemptsMade / (job.opts.attempts ?? 1) > 0.8) {
      candidates.push({
        jobId,
        attemptsMade: job.attemptsMade,
        maxAttempts: job.opts.attempts,
      });
    }
  }
  // Jobs that have used >80% of their retry budget
  return candidates;
}

Scenario 3: Lock Contention on Manual Retry

If retryJob-11.lua returns -2 (missing lock), it means the job is either still being processed or the lock was already cleaned up. This can happen when a worker is processing very slowly.

QueueHub surfaces these as "conflicted retry" events, helping you identify workers that are holding locks longer than expected.

---

Best Practices Informed by Redis Internals

Understanding the underlying Redis operations leads to concrete operational recommendations:

1. Set maxStalledCount conservatively — every stall detection cycle reads the entire active SET via SMEMBERS. A high count means more Redis memory churn on each check.

2. Use jitter on exponential backoff — prevents time-clustered delayed ZSET entries that cause cascading retry bursts under high failure rates.

3. Keep removeOnFail reasonable — the ZSET cleanup happens in Lua via ZREMRANGEBYSCORE, but it adds latency proportional to the set size. Setting it to true (immediate removal) avoids this scan entirely.

4. Monitor bull:{queue}:delayed ZSET cardinality — a sudden spike signals a retry storm. QueueHub tracks this as a "delayed job" metric and alerts you when it crosses thresholds.

5. Watch for retries-exhausted events — each one represents a permanently failed job that may need human intervention. In QueueHub's dashboard, these are marked for attention.

6. Understand the KEYS vs ARGV distinction — if you write custom Lua scripts that interact with BullMQ keys, remember that Redis cluster sharding uses KEYS[...] (not ARGV[...]) to determine the hash slot. All keys accessed in a script must belong to the same slot.

---

Conclusion

BullMQ's retry system is elegant but opaque without understanding the Redis internals. Three distinct paths move jobs through the retry lifecycle:

• Automatic retry via moveToFinished-14.lua — the standard path when a job fails within its attempt budget
• Manual retry via retryJob-11.lua and reprocessJob-8.lua — triggered by explicit API calls or dashboard actions
• Stalled job recovery via moveStalledJobsToWait-9.lua — the safety net for crashed workers

All three paths use atomic Lua scripts executing inside Redis's single-threaded event loop, ensuring no data loss between state transitions. The key data structures — failed ZSETs, delayed ZSETs with timestamp scores, stalled SETs, and job hashes with atm/ ats/ stc counters — form a complete state machine that's both reliable and inspectable.

Tools like QueueHub that understand these internals provide better observability into your queue's retry state — surfacing retry storms before they impact throughput, flagging jobs with near-exhausted retry budgets, and making manual retry operations a single-click affair backed by the same atomic Lua scripts.

See your queue's retry lifecycle in real time. Start a free QueueHub trial to monitor your BullMQ retries, visualize backoff patterns, and catch stalled jobs before they become production incidents.

---

Lua Scripts Referenced

Script	BullMQ Source
moveToFinished-14.lua	BullMQ GitHub
retryJob-11.lua	BullMQ GitHub
reprocessJob-8.lua	BullMQ GitHub
moveStalledJobsToWait-9.lua	BullMQ GitHub
getStateV2-8.lua	BullMQ GitHub
moveToDelayed-12.lua	BullMQ GitHub