Error-Type-Aware Retry Strategies for BullMQ External API Integration

Most BullMQ retry guides ask one question: how many attempts and what backoff? But when your queue talks to external APIs — Stripe, SendGrid, GitHub, OpenAI — that single-question framing misses the real challenge: different errors demand different retry strategies.

A 429 rate limit needs a different response than a 503 service outage. A 400 validation error should never be retried. A network ETIMEDOUT might warrant a longer wait than a Connection Refused. And retrying a POST /payments without an idempotency key is a receipt-disaster waiting to happen.

This post goes beyond attempts: 5 + exponential backoff. You'll learn how to classify API errors by type, apply conditional retry logic with BullMQ's custom backoff strategy, use idempotency keys for safe POST retries, implement a retry budget to protect downstream systems, add retry-specific OpenTelemetry observability, and route failed jobs to category-specific queues.

---

Why One-Size-Fits-All Retry Fails for External APIs

A single retry strategy treats all failures the same. In practice, API errors fall into distinct categories that demand different responses:

Error Category	HTTP / Network Signal	Retry Decision	Example
Transient (retryable)	429, 503, 502, 504, ECONNRESET, ETIMEDOUT	✅ Retry with intent	API is overloaded or briefly unreachable
Permanent (non-retryable)	400, 401, 403, 404, 422	❌ Never retry — fail fast	Invalid request or missing permissions
Ambiguous (unknown)	500, malformed response, SSL errors	⚠️ Retry cautiously	May be transient or permanent

A blind retry policy that treats all errors the same will:

• Retry 400s infinitely — wasting resources and delaying downstream jobs
• Retry 429s with too-aggressive backoff — burning attempts while the rate-limit window is still closed
• Fail to retry transient errors with enough delay — retrying before the API has recovered

Building an Error Classifier

The foundation of error-type-aware retry is a pure function that maps errors and HTTP responses to a classification. Keep it isolated from BullMQ — no side effects, no imports — so you can unit test it at millisecond speed.

// error-classifier.ts
type ErrorCategory = 'transient' | 'permanent' | 'unknown';

interface ClassificationResult {
  category: ErrorCategory;
  suggestedDelayMs?: number;
  reason: string;
}

function classifyApiError(
  error: Error,
  httpStatus?: number,
  headers?: Headers
): ClassificationResult {
  // Rate-limit detection
  if (httpStatus === 429) {
    const retryAfter = parseRetryAfter(headers?.get('retry-after'));
    return {
      category: 'transient',
      suggestedDelayMs: retryAfter ?? 5000,
      reason: `Rate limited (429). Retry-After: ${retryAfter ?? 'not-set'}ms`,
    };
  }

  // Server errors — retryable
  if (httpStatus && httpStatus >= 500) {
    return {
      category: 'transient',
      reason: `Server error (${httpStatus}). Retry.`,
    };
  }

  // Client errors — never retry
  if (httpStatus && httpStatus >= 400 && httpStatus < 500) {
    return {
      category: 'permanent',
      reason: `Client error (${httpStatus}). Non-retryable.`,
    };
  }

  // Network-level system errors
  const code = (error as NodeJS.ErrnoException).code;
  const transientCodes = new Set([
    'ECONNRESET', 'ETIMEDOUT', 'ECONNREFUSED', 'ENOTFOUND',
    'EAI_AGAIN', 'EPIPE', 'ENETUNREACH',
  ]);
  if (code && transientCodes.has(code)) {
    return {
      category: 'transient',
      suggestedDelayMs: code === 'ETIMEDOUT' ? 10_000 : undefined,
      reason: `Network error (${code}). Retry.`,
    };
  }

  // Fallback
  return {
    category: 'unknown',
    reason: `Unclassified: ${error.name}: ${error.message.slice(0, 100)}`,
  };
}

function parseRetryAfter(value: string | null | undefined): number | undefined {
  if (!value) return undefined;
  const seconds = parseInt(value, 10);
  return isNaN(seconds) ? undefined : seconds * 1000;
}

The classifier is pure — test it in 10ms with Vitest or Jest. This function becomes the single source of truth for every retry decision in your system.

Custom Backoff Strategy — Dynamic Delays Per Error Type

BullMQ's settings.backoffStrategy callback gives you access to the error object and attempt count on every retry evaluation. Combine it with the error classifier to dynamically tune delays.

The Problem with Static Backoff

A fixed delay of 5 seconds treats a 429 (where the API says "wait 30s") the same as a 503 (likely recovered in 2–5s). Exponential backoff with a 1-second base on a 429 will retry at 1s, 2s, then 4s — all while the rate-limit window is still closed. You burn all 5 attempts in under 30 seconds and the job goes to the dead letter queue, even though waiting 60 seconds would have succeeded.

Implementation — Error-Type-Aware Backoff Strategy

import { Worker, Job, UnrecoverableError } from 'bullmq';

const worker = new Worker(
  'api-queue',
  async (job: Job) => { /* processor logic */ },
  {
    settings: {
      backoffStrategy: (
        attemptsMade: number,
        type: string | undefined,
        err: Error | undefined,
        job: Job | undefined
      ): number => {
        const httpStatus = job?.data?._lastHttpStatus;
        const headers = job?.data?._lastResponseHeaders;

        const { category, suggestedDelayMs } = classifyApiError(
          err ?? new Error('Unknown'),
          httpStatus,
          headers ? new Headers(headers) : undefined,
        );

        // Fast-fail on permanent errors
        if (category === 'permanent') return -1;

        // Honor Retry-After header if present
        if (suggestedDelayMs) return suggestedDelayMs;

        // Per-category exponential backoff with jitter
        const baseDelay = httpStatus === 429 ? 10_000 : 2_000;
        const delay = Math.min(baseDelay * Math.pow(2, attemptsMade - 1), 60_000);
        const jitter = delay * 0.25 * (Math.random() * 2 - 1);
        return Math.round(delay + jitter);
      },
    },
  }
);

A return value of -1 tells BullMQ to move the job directly to the failed state — no retry attempt, no delay. This is how permanent errors (400, 401, 422) get fast-failed immediately without wasting a retry slot.

Saving Error Context on the Job

For the backoff strategy to work, the worker processor must store error context on the job before throwing:

async function apiProcessor(job: Job) {
  try {
    const response = await fetch(job.data.url, {
      method: job.data.method,
      headers: { 'Content-Type': 'application/json' },
      body: job.data.body ? JSON.stringify(job.data.body) : undefined,
    });

    if (!response.ok) {
      // Persist HTTP context for the backoff strategy
      await job.updateData({
        ...job.data,
        _lastHttpStatus: response.status,
        _lastResponseHeaders: Object.fromEntries(response.headers.entries()),
      });

      const { category } = classifyApiError(
        new Error(`HTTP ${response.status}`),
        response.status,
      );

      if (category === 'permanent') {
        throw new UnrecoverableError(
          `Non-retryable: HTTP ${response.status}`
        );
      }

      throw new Error(`API returned ${response.status}`);
    }

    return await response.json();
  } catch (error) {
    if (!(error instanceof UnrecoverableError)) {
      await job.updateData({
        ...job.data,
        _lastError: (error as Error).message,
      });
    }
    throw error;
  }
}

This pattern — save context, classify, then throw conditionally — is the core architecture for error-type-aware retry.

Idempotency Keys for Safe POST/PUT Retries

When BullMQ retries a job, it re-executes the entire processor. For GET requests, that's safe. For POST /payments, POST /orders, or PUT /invoices, a retry means sending the same mutation twice. Without idempotency, you get duplicate charges and double orders.

Generate Deterministic Idempotency Keys

The idempotency key must be deterministic — generated from the job's identity, not created fresh on each attempt. Store it in the job data at enqueue time:

import { Queue } from 'bullmq';
import { v5 as uuidv5 } from 'uuid';

const IDEMPOTENCY_NAMESPACE = '6ba7b810-9dad-11d1-80b4-00c04fd430c8';

function createIdempotencyKey(jobId: string, operation: string): string {
  return uuidv5(`${jobId}:${operation}`, IDEMPOTENCY_NAMESPACE);
}

// At enqueue time — key is stored once, reused on every retry
await queue.add('charge-payment', {
  paymentId: 'pm_123',
  amount: 2999,
  idempotencyKey: createIdempotencyKey('pm_123', 'charge'),
}, {
  attempts: 3,
  backoff: { type: 'exponential', delay: 2000 },
});

Send the Stored Key on Every Attempt

async function chargeProcessor(job: Job) {
  const { paymentId, amount, idempotencyKey } = job.data;

  const response = await fetch('https://api.stripe.com/v1/charges', {
    method: 'POST',
    headers: {
      Authorization: `Bearer ${process.env.STRIPE_SECRET_KEY}`,
      'Idempotency-Key': idempotencyKey,
      'Content-Type': 'application/x-www-form-urlencoded',
    },
    body: new URLSearchParams({
      amount: String(amount),
      currency: 'usd',
      payment: paymentId,
    }),
  });

  // Accept 409 (idempotency key already used) as success
  if (response.status === 200 || response.status === 409) {
    return response.json();
  }

  // Error handling from previous section
  // ...
}

The API's idempotency layer means retries are safe by default. The second, third, and fourth attempts don't create duplicate charges — they return the same result as the first successful attempt.

Retry Budget — Protecting Downstream APIs from Retry Storms

A retry budget caps the total retry effort your system dedicates to a specific external API within a sliding time window. When the budget is exhausted, new retries are deferred or the job is failed fast — preventing a cascading retry storm from overwhelming an already-struggling downstream service.

This is distinct from a circuit breaker: a circuit breaker opens when calls fail; a retry budget opens when retries consume too much quota.

Sliding-Window Budget with Redis

// retry-budget.ts
import { createClient } from 'redis';

interface RetryBudgetConfig {
  maxRetriesPerWindow: number;
  windowMs: number;
  endpointPrefix: string;
}

export class RetryBudget {
  private redis;
  private config: RetryBudgetConfig;

  constructor(redisUrl: string, config: RetryBudgetConfig) {
    this.redis = createClient({ url: redisUrl });
    this.config = config;
  }

  async consume(
    endpoint: string
  ): Promise<{ allowed: boolean; remaining: number }> {
    const key = `${this.config.endpointPrefix}:${endpoint}`;
    const now = Date.now();

    // Atomic sliding window using Redis sorted set
    const result = await this.redis
      .multi()
      .zRemRangeByScore(key, 0, now - this.config.windowMs)
      .zAdd(key, { score: now, value: `${now}-${Math.random()}` })
      .zCard(key)
      .expire(key, Math.ceil(this.config.windowMs / 1000) + 60)
      .exec();

    const count = (result?.[2] as number) ?? 0;
    const allowed = count <= this.config.maxRetriesPerWindow;
    const remaining = Math.max(0, this.config.maxRetriesPerWindow - count);

    return { allowed, remaining };
  }
}

Integrating the Budget with Your Backoff Strategy

const retryBudget = new RetryBudget(process.env.REDIS_URL!, {
  maxRetriesPerWindow: 50,
  windowMs: 60_000,
  endpointPrefix: 'retry-budget',
});

const worker = new Worker('api-queue', processor, {
  settings: {
    backoffStrategy: async (attemptsMade, type, err, job) => {
      const endpoint = job?.data?.url ?? 'unknown';
      const { allowed } = await retryBudget.consume(endpoint);

      if (!allowed) {
        job?.log(`Retry budget exhausted for ${endpoint}`);
        return -1; // move to failed state immediately
      }

      // Normal error-type-aware backoff logic
      // ...
    },
  },
});

The retry budget acts as an outer guard — it doesn't decide how long to wait, it decides whether to even attempt the retry. When the API is clearly overwhelmed, stop adding fuel to the fire.

Testing Error-Type-Aware Retry Logic

Testing retry behavior requires three layers, each validating different aspects of the system.

Layer 1: Unit Tests for the Classifier

The pure classifyApiError function is trivially testable:

import { describe, it, expect } from 'vitest';

describe('classifyApiError', () => {
  it('classifies 429 as transient with Retry-After delay', () => {
    const error = new Error('Too Many Requests');
    const headers = new Headers({ 'retry-after': '30' });
    const result = classifyApiError(error, 429, headers);

    expect(result.category).toBe('transient');
    expect(result.suggestedDelayMs).toBe(30_000);
  });

  it('classifies 400 as permanent', () => {
    const result = classifyApiError(new Error('Bad Request'), 400);
    expect(result.category).toBe('permanent');
  });

  it('classifies ECONNRESET as transient', () => {
    const error = new Error('read ECONNRESET');
    (error as NodeJS.ErrnoException).code = 'ECONNRESET';
    const result = classifyApiError(error);
    expect(result.category).toBe('transient');
  });

  it('classifies unknown errors as unknown', () => {
    const error = new Error('Unexpected token < in JSON');
    const result = classifyApiError(error);
    expect(result.category).toBe('unknown');
  });
});

Layer 2: Integration Tests with Real Redis

Use testcontainers or a local Redis instance to test the full retry lifecycle:

import { Worker, Queue, QueueEvents } from 'bullmq';
import IORedis from 'ioredis';
import http from 'http';

describe('API Queue Retry Integration', () => {
  let redis: IORedis;
  let queue: Queue;
  let mockApi: http.Server;
  let apiCallCount = 0;

  beforeAll(async () => {
    redis = new IORedis({ host: 'localhost', port: 6379 });

    // Mock API that fails twice then succeeds
    mockApi = http.createServer((req, res) => {
      apiCallCount++;
      if (apiCallCount <= 2) {
        res.writeHead(429, { 'Retry-After': '1' });
        res.end(JSON.stringify({ error: 'rate_limit_exceeded' }));
      } else {
        res.writeHead(200);
        res.end(JSON.stringify({ status: 'ok' }));
      }
    }).listen(0);

    queue = new Queue('test-api-queue', { connection: redis });
  });

  it('retries with Retry-After delay and succeeds on 3rd attempt', async () => {
    const port = (mockApi.address() as any).port;
    apiCallCount = 0;

    const worker = new Worker(
      'test-api-queue',
      async (job) => {
        const response = await fetch(`http://localhost:${port}/data`);
        if (!response.ok) {
          await job.updateData({
            ...job.data,
            _lastHttpStatus: response.status,
          });
          throw new Error(`API returned ${response.status}`);
        }
        return response.json();
      },
      {
        connection: redis,
        settings: {
          backoffStrategy: (attemptsMade, type, err, job) => {
            const status = job?.data?._lastHttpStatus;
            return status === 429 ? 1500 : 500;
          },
        },
      }
    );

    const events = new QueueEvents('test-api-queue', { connection: redis });
    const completed = new Promise<void>((resolve) => {
      events.on('completed', () => resolve());
    });

    await queue.add(
      'test-retry',
      { url: `http://localhost:${port}/data` },
      { attempts: 3 }
    );
    await completed;
    expect(apiCallCount).toBe(3);

    await worker.close();
    await events.close();
  });

  afterAll(async () => {
    await queue.close();
    await redis.quit();
    mockApi.close();
  });
});

Layer 3: Testing Idempotency Key Consistency

Assert that the same idempotency key is sent on all retry attempts:

it('sends the same idempotency key on all retry attempts', async () => {
  const receivedKeys: string[] = [];

  const mockApi = http.createServer((req, res) => {
    receivedKeys.push(req.headers['idempotency-key'] as string);
    res.writeHead(429);
    res.end();
  }).listen(0);

  // ... worker that sends idempotencyKey from job.data ...

  expect(new Set(receivedKeys).size).toBe(1); // all identical
  expect(receivedKeys.length).toBeGreaterThan(1); // at least one retry
  mockApi.close();
});

Retry-Specific Observability with OpenTelemetry

BullMQ's built-in bullmq-otel package covers overall job lifecycle spans. For debugging retry behavior, you need retry-specific dimensions:

• Which attempt is this?
• What error category was detected?
• What was the suggested delay?
• Did the retry budget reject it?
• Which external API endpoint was targeted?

Without these, a dashboard showing "12 failed jobs" doesn't tell you whether those are 12 unique failures or one job retrying 12 times against a dying API.

Adding Retry Attributes to Worker Spans

import { otel } from '@opentelemetry/api';
import { BullMQOtel } from 'bullmq-otel';

const tracer = otel.trace.getTracer('queuehub-retry-observer');

const worker = new Worker('api-queue', async (job) => {
  return tracer.startActiveSpan('api-queue.process', async (span) => {
    span.setAttribute('job.id', job.id!);
    span.setAttribute('job.attempt', job.attemptsMade + 1);
    span.setAttribute('job.maxAttempts', job.opts.attempts ?? 1);
    span.setAttribute('api.endpoint', job.data.url ?? 'unknown');
    span.setAttribute('api.method', job.data.method ?? 'GET');

    try {
      const result = await processJob(job);
      span.setStatus({ code: otel.SpanStatusCode.OK });
      return result;
    } catch (error) {
      const { category, reason } = classifyApiError(
        error as Error,
        job.data._lastHttpStatus,
      );

      span.setAttribute('retry.error_category', category);
      span.setAttribute('retry.error_reason', reason);
      span.setAttribute('retry.error_http_status', job.data._lastHttpStatus ?? 0);
      span.setAttribute('retry.attempts_made', job.attemptsMade);

      span.setStatus({
        code: otel.SpanStatusCode.ERROR,
        message: `[${category}] ${reason}`,
      });
      throw error;
    } finally {
      span.end();
    }
  });
}, {
  telemetry: new BullMQOtel('api-queue-service'),
});

Custom Retry Metrics

Export counters for graphing retry behavior over time:

import { metrics } from '@opentelemetry/api';

const retryMeter = metrics.getMeter('queuehub-retry-metrics');

const retryAttemptsCounter = retryMeter.createCounter(
  'bullmq.retry.attempts',
  { description: 'Retry attempts by error category' }
);

const retryBudgetExhausted = retryMeter.createCounter(
  'bullmq.retry.budget_exhausted',
  { description: 'Retries rejected by the retry budget' }
);

// In the error handler:
retryAttemptsCounter.add(1, {
  'error.category': category,
  'api.endpoint': job.data.url,
  'attempt': job.attemptsMade,
});

Structured Logging for Fast Debugging

logger.warn('API job failed — will retry', {
  jobId: job.id,
  queue: job.queueName,
  attempt: job.attemptsMade + 1,
  maxAttempts: job.opts.attempts,
  errorCategory: category,
  httpStatus: job.data._lastHttpStatus,
  apiEndpoint: job.data.url,
  idempotencyKey: job.data.idempotencyKey,
});

After a production incident, you should be able to query "show me all retry attempts for endpoint X in the last hour, grouped by error category" — and get the answer in seconds.

Failed Job Routing — Dedicated Queues by Error Category

The standard dead letter queue pattern sends all failed jobs to one sink. For API integration, a single sink is too coarse. Route to category-specific sinks:

• api-queue:failed-permanent — 400s, 401s, 404s (developer review needed)
• api-queue:failed-transient-exhausted — transient errors that exhausted retries (API may be down)
• api-queue:failed-unknown — unclassified errors (investigation needed)

const permanentDLQ = new Queue('api-queue:failed-permanent', { connection });
const exhaustedDLQ = new Queue(
  'api-queue:failed-transient-exhausted',
  { connection }
);
const unknownDLQ = new Queue('api-queue:failed-unknown', { connection });

worker.on('failed', async (job, error) => {
  if (!job) return;

  const httpStatus = job.data?._lastHttpStatus;
  const { category } = classifyApiError(error, httpStatus);

  const targetQueue = category === 'permanent'
    ? permanentDLQ
    : category === 'unknown'
    ? unknownDLQ
    : exhaustedDLQ;

  await targetQueue.add(job.name!, {
    ...job.data,
    _originalJobId: job.id,
    _failedAt: new Date().toISOString(),
    _failureReason: error.message,
  });
});

With category-specific queues, you set different alerting policies for each:

• failed-permanent queue depth > 0 → PagerDuty urgent (developer error or API contract change)
• failed-transient-exhausted queue growing fast → PagerDuty incident (API outage)
• failed-unknown queue depth > 5 in 1 hour → Slack notification (needs investigation)

Putting It All Together

Here's the full architecture:

• A BullMQ queue (api-queue) with a backoffStrategy that classifies errors and dynamically adjusts delays
• An error classifier that maps HTTP status codes and network errors to transient/permanent/unknown
• Idempotency keys generated deterministically and stored in job data at enqueue time
• A Redis-backed retry budget that caps retries per endpoint per time window
• Retry-specific OpenTelemetry spans and metrics for observability
• Category-specific failed queues with differentiated alerting

QueueHub surfaces all three failed queues in separate views, shows retry attempt counts per job, and lets you manually retry or dismiss based on error category. The structured logging and OTel spans flow directly into your existing observability stack, so you can investigate retry patterns without writing custom instrumentation.

Quick Reference

❌ What Most Guides Say	✅ What This Post Says
Set attempts: 5 on every job	Classify errors first — permanent errors throw UnrecoverableError immediately
Use backoff: { type: 'exponential' }	Use settings.backoffStrategy with dynamic delay per error category
Retry everything up to max attempts	Implement a retry budget — fail fast when downstream is overwhelmed
Generic DLQ for all failures	Route to category-specific DLQs — permanent, exhausted, unknown
Log "job failed"	Log structured retry context — category, attempt, delay, budget, idempotency key
Test "does it retry"	Test "does it retry with the correct delay for this error type"