Redis Queue Health Under Load — Stress Testing, Chaos Engineering, and Failure Mode Analysis for BullMQ

A queue that looks healthy at 10 jobs/minute can collapse catastrophically at 1,000 jobs/second. Static health checks — PING returns PONG, INFO memory shows 40% used, getJobCounts() reports zero failures — tell you nothing about how the system behaves at its actual operating limits or during failure scenarios.

The usual approach to monitoring Redis queue health is measuring a quiescent system: setting up dashboards, configuring alerts, and hoping the metrics catch a problem before it becomes a crisis. This post takes a fundamentally different approach. Instead of measuring a system at rest, we actively exercise the queue under controlled failure conditions to answer three questions:

How does BullMQ degrade as Redis approaches its limits? Does it back off gracefully or fall over?
What do real failure modes look like in your metrics? Can you distinguish a network partition from memory pressure from a worker crash?
Is your queue infrastructure sized correctly? Not based on napkin math, but on validated load tests.

We'll build a complete stress-testing harness, design chaos experiments for Redis failover and OOM scenarios, catalog failure-mode signatures, wire health regression into CI/CD, and characterize BullMQ's degradation patterns under increasing stress. Every section includes production-ready TypeScript code you can adapt directly.

Building a Queue Load Test Harness

Before we can measure health under load, we need a harness that generates controlled, reproducible load while simultaneously collecting health signals. This harness differs from typical benchmarking by correlating Redis health metrics with queue throughput in real time.

The Harness Architecture

Our harness does three things in parallel:

Load generation — Adds jobs to a queue at a configurable rate using a token-bucket throttle
Workload processing — Spawns workers that simulate realistic processing
Health telemetry collection — Polls Redis INFO, BullMQ queue metrics, and worker status on a fixed cadence

import { Queue, Worker, Job } from 'bullmq';
import IORedis from 'ioredis';
import { performance } from 'perf_hooks';

interface LoadTestConfig {
  targetRps: number;
  durationSeconds: number;
  jobProcessingMeanMs: number;
  jobPayloadBytes: number;
  workerConcurrency: number;
}

interface HealthSnapshot {
  timestamp: number;
  redis: {
    usedMemoryRss: number;
    instantaneousOpsPerSec: number;
    evictedKeys: number;
    rejectedConnections: number;
    connectedClients: number;
  };
  queue: {
    waiting: number;
    active: number;
    completed: number;
    failed: number;
    latencyP50: number;
    latencyP95: number;
    latencyP99: number;
  };
}

class HealthCollector {
  private snapshots: HealthSnapshot[] = [];
  private connection: IORedis;
  private queue: Queue;
  private jobLatencies: number[] = [];
  private processingStarts = new Map<string, number>();

  constructor(connection: IORedis, queueName: string) {
    this.connection = connection;
    this.queue = new Queue(queueName, { connection });
  }

  recordJobStart(jobId: string): void {
    this.processingStarts.set(jobId, performance.now());
  }

  recordJobComplete(jobId: string): void {
    const start = this.processingStarts.get(jobId);
    if (start !== undefined) {
      this.jobLatencies.push(performance.now() - start);
      this.processingStarts.delete(jobId);
    }
  }

  async snapshot(): Promise<HealthSnapshot> {
    const [info, counts] = await Promise.all([
      this.connection.info(),
      this.queue.getJobCounts(),
    ]);

    const getInfo = (key: string): number => {
      const m = info.match(new RegExp(`^${key}:(.+)$`, 'm'));
      return m ? Number(m[1].trim()) : 0;
    };

    const latencies = [...this.jobLatencies].sort((a, b) => a - b);
    const len = latencies.length;

    const snapshot: HealthSnapshot = {
      timestamp: Date.now(),
      redis: {
        usedMemoryRss: getInfo('used_memory_rss'),
        instantaneousOpsPerSec: getInfo('instantaneous_ops_per_sec'),
        evictedKeys: getInfo('evicted_keys'),
        rejectedConnections: getInfo('rejected_connections'),
        connectedClients: getInfo('connected_clients'),
      },
      queue: {
        waiting: counts.waiting ?? 0,
        active: counts.active ?? 0,
        completed: counts.completed ?? 0,
        failed: counts.failed ?? 0,
        latencyP50: len > 0 ? latencies[Math.floor(len * 0.5)] : 0,
        latencyP95: len > 0 ? latencies[Math.floor(len * 0.95)] : 0,
        latencyP99: len > 0 ? latencies[Math.floor(len * 0.99)] : 0,
      },
    };

    this.snapshots.push(snapshot);
    return snapshot;
  }

  getSnapshots(): HealthSnapshot[] {
    return this.snapshots;
  }

  reset(): void {
    this.snapshots = [];
    this.jobLatencies = [];
    this.processingStarts.clear();
  }

  async close(): Promise<void> {
    await this.queue.close();
  }
}

class TokenBucket {
  private tokens: number;
  private lastRefill: number;
  private readonly capacity: number;
  private readonly refillRatePerMs: number;

  constructor(rps: number) {
    this.capacity = rps;
    this.tokens = rps;
    this.lastRefill = Date.now();
    this.refillRatePerMs = rps / 1000;
  }

  tryConsume(): boolean {
    this.refill();
    if (this.tokens >= 1) {
      this.tokens -= 1;
      return true;
    }
    return false;
  }

  private refill(): void {
    const now = Date.now();
    const elapsed = now - this.lastRefill;
    this.tokens = Math.min(this.capacity, this.tokens + elapsed * this.refillRatePerMs);
    this.lastRefill = now;
  }
}

function generatePayload(bytes: number): Record<string, unknown> {
  return {
    id: crypto.randomUUID(),
    timestamp: Date.now(),
    data: 'x'.repeat(bytes),
  };
}

The Load Test Runner

Now we wire everything into a complete test runner that injects load at the target rate, drains the queue afterward, and reports whether the system degraded:

interface LoadTestResult {
  actualThroughput: number;
  totalProcessed: number;
  totalFailed: number;
  totalStalled: number;
  snapshots: HealthSnapshot[];
  latency: { p50: number; p95: number; p99: number; max: number };
  degraded: boolean;
  degradationReason?: string;
}

async function runLoadTest(
  connection: IORedis,
  config: LoadTestConfig,
): Promise<LoadTestResult> {
  const queueName = `stress-test-${Date.now()}`;
  const queue = new Queue(queueName, { connection });
  const collector = new HealthCollector(connection, queueName);
  const bucket = new TokenBucket(config.targetRps);

  let jobsCompleted = 0;
  let jobsFailed = 0;
  let jobsStalled = 0;
  let failed = false;

  const worker = new Worker(
    queueName,
    async (job: Job) => {
      collector.recordJobStart(job.id!);
      const processingTime = -Math.log(Math.random()) * config.jobProcessingMeanMs;
      await new Promise((resolve) => setTimeout(resolve, processingTime));
      collector.recordJobComplete(job.id!);
    },
    {
      connection,
      concurrency: config.workerConcurrency,
      stalledInterval: 30000,
      maxStalledCount: 3,
    },
  );

  worker.on('completed', () => jobsCompleted++);
  worker.on('failed', (job, _err) => {
    jobsFailed++;
    collector.recordJobComplete(job.id!);
  });
  worker.on('stalled', () => jobsStalled++);

  const endTime = Date.now() + config.durationSeconds * 1000;

  while (Date.now() < endTime && !failed) {
    if (bucket.tryConsume()) {
      try {
        await queue.add(
          'stress-job',
          generatePayload(config.jobPayloadBytes),
          { attempts: 3, backoff: { type: 'exponential', delay: 1000 } },
        );
      } catch {
        failed = true;
        break;
      }
    } else {
      await new Promise((r) => setTimeout(r, 1));
    }
  }

  // Drain phase
  const drainStart = Date.now();
  while (Date.now() - drainStart < 30000) {
    const counts = await queue.getJobCounts();
    if ((counts.active ?? 0) === 0 && (counts.waiting ?? 0) === 0) break;
    await new Promise((r) => setTimeout(r, 200));
    await collector.snapshot();
  }

  await worker.close();
  try {
    await queue.obliterate({ force: true });
  } catch {
    // queue may already be gone
  }
  await collector.close();

  const snapshots = collector.getSnapshots();
  const allLatencies = snapshots
    .flatMap((s) => [s.queue.latencyP50, s.queue.latencyP95, s.queue.latencyP99])
    .filter((l) => l > 0)
    .sort((a, b) => a - b);

  const len = allLatencies.length;
  const throughput = config.durationSeconds > 0
    ? jobsCompleted / config.durationSeconds
    : 0;

  const p99 = len > 0 ? allLatencies[Math.floor(len * 0.99)] : 0;
  const failureRate =
    jobsCompleted + jobsFailed > 0
      ? jobsFailed / (jobsCompleted + jobsFailed)
      : 0;
  const isDegraded = p99 > config.jobProcessingMeanMs * 5 || failureRate > 0.01;

  return {
    actualThroughput: Math.round(throughput),
    totalProcessed: jobsCompleted,
    totalFailed: jobsFailed,
    totalStalled: jobsStalled,
    snapshots,
    latency: {
      p50: len > 0 ? allLatencies[Math.floor(len * 0.5)] : 0,
      p95: len > 0 ? allLatencies[Math.floor(len * 0.95)] : 0,
      p99,
      max: len > 0 ? allLatencies[len - 1] : 0,
    },
    degraded: isDegraded,
    degradationReason: isDegraded
      ? `P99 latency ${p99.toFixed(1)}ms (${config.jobProcessingMeanMs * 5}ms threshold) or failure rate ${(failureRate * 100).toFixed(2)}%`
      : undefined,
  };
}

Running a ramp test reveals the system's practical throughput limit:

async function rampTest() {
  const connection = new IORedis({
    host: 'localhost',
    port: 6379,
    maxRetriesPerRequest: null,
  });

  for (const rps of [100, 250, 500, 750, 1000]) {
    const result = await runLoadTest(connection, {
      targetRps: rps,
      durationSeconds: 30,
      jobProcessingMeanMs: 10,
      jobPayloadBytes: 1024,
      workerConcurrency: 5,
    });

    console.log(`\n── ${rps} RPS ──`);
    console.log(`  Throughput:  ${result.actualThroughput} jobs/s`);
    console.log(`  P50/P99:     ${result.latency.p50.toFixed(1)} / ${result.latency.p99.toFixed(1)} ms`);
    console.log(`  Degraded:    ${result.degraded}`);

    // Track memory growth
    const snapshots = result.snapshots;
    if (snapshots.length >= 2) {
      const growthMb =
        (snapshots[snapshots.length - 1].redis.usedMemoryRss -
          snapshots[0].redis.usedMemoryRss) /
        (1024 * 1024);
      if (growthMb > 100) {
        console.log(`  ⚠ Memory grew ${growthMb.toFixed(0)} MB`);
      }
    }
  }

  await connection.quit();
}

rampTest().catch(console.error);

What this reveals: At lower RPS values, latency stays flat near the processing time (10ms + Redis overhead). As RPS approaches the Redis instantaneous_ops_per_sec ceiling, latency starts climbing non-linearly. The cross-over point — where P99 latency exceeds 5× processing time — is your queue's practical throughput limit.

Chaos Engineering Experiments for Queue Health

Static load testing tells you the maximum throughput. Chaos experiments tell you what happens when things break. This section covers three essential chaos experiments, each with complete orchestration code.

Experiment 1: Redis Failover (Sentinel or Cluster)

When Redis performs a failover, the master changes. BullMQ workers reconnect automatically via IORedis's reconnection logic. The critical question is: how long does the disruption last, and what metrics signal it?

import IORedis from 'ioredis';
import { Queue, Worker, Job } from 'bullmq';

interface FailoverResult {
  outageMs: number;
  stalledJobs: number;
  requeuedJobs: number;
}

async function simulateRedisFailover(
  connection: IORedis,
): Promise<FailoverResult> {
  const queueName = `chaos-failover-${Date.now()}`;
  const queue = new Queue(queueName, { connection });
  let stalledCount = 0;
  let requeuedCount = 0;
  let outageStart: number | null = null;
  let outageEnd: number | null = null;

  const worker = new Worker(
    queueName,
    async (_job: Job) => {
      await new Promise((resolve) => setTimeout(resolve, 50));
    },
    { connection, concurrency: 10, stalledInterval: 5000 },
  );

  worker.on('stalled', (_jobId: string) => {
    stalledCount++;
    requeuedCount++;
  });

  worker.on('completed', () => {
    if (outageStart !== null && outageEnd === null) {
      outageEnd = Date.now();
    }
  });

  // Inject steady load
  const injector = setInterval(async () => {
    try {
      await queue.add('chaos-job', { ts: Date.now() });
    } catch {
      // write failures are part of the outage signal
    }
  }, 10);

  await new Promise((r) => setTimeout(r, 5000)); // wait for steady state

  // Trigger failover
  try {
    await connection.sendCommand('SENTINEL', ['failover', 'mymaster']);
  } catch {
    console.log('SENTINEL failover unavailable; simulating disconnect');
    await connection.disconnect();
    await new Promise((r) => setTimeout(r, 2000));
    connection.connect().catch(() => {});
  }

  // Monitor recovery for 30 seconds
  const monitorStart = Date.now();
  while (Date.now() - monitorStart < 30000) {
    try {
      const counts = await queue.getJobCounts();
      if (outageStart === null && (counts.active ?? 0) === 0) {
        outageStart = Date.now();
      }
    } catch {
      if (outageStart === null) {
        outageStart = Date.now();
      }
    }
    await new Promise((r) => setTimeout(r, 200));
  }

  clearInterval(injector);
  await worker.close();
  await queue.obliterate({ force: true });

  return {
    outageMs:
      outageStart !== null && outageEnd !== null
        ? outageEnd - outageStart
        : outageStart !== null
        ? Date.now() - outageStart
        : 0,
    stalledJobs: stalledCount,
    requeuedJobs: requeuedCount,
  };
}

What to look for in the metrics during a failover:

active jobs drop to 0 immediately — the worker can't claim jobs without a Redis connection
waiting count spikes — jobs accumulate in the queue during the outage
stalled count increments as BullMQ's stall detector fires on jobs whose workers disconnected
After recovery, active spikes above normal as stalled jobs are re-processed
instantaneous_ops_per_sec temporarily doubles — the catch-up phase

Experiment 2: Redis OOM (Out of Memory) Scenario

When Redis hits maxmemory with a non-noeviction policy, it starts evicting keys. For BullMQ, evicted keys mean silent data loss — jobs disappear from the queue without trace:

async function simulateMemoryPressure(
  connection: IORedis,
  queueName: string,
): Promise<void> {
  const queue = new Queue(queueName, { connection });

  // Phase 1: Fill the queue with large jobs
  console.log('Phase 1: Injecting large jobs to consume memory');
  for (let i = 0; i < 5000; i++) {
    await queue.add('large-job', {
      id: i,
      payload: 'x'.repeat(50 * 1024), // 50KB each
    });
  }

  // Phase 2: Ramp up load to trigger eviction
  console.log('Phase 2: Adding more jobs under memory pressure');
  const evictionBaseline = Number(
    (await connection.info()).match(/^evicted_keys:(\d+)$/m)?.[1] ?? '0',
  );

  for (let i = 0; i < 2000; i++) {
    await queue.add('pressure-job', {
      id: i,
      payload: 'x'.repeat(50 * 1024),
    });
    if (i > 0 && i % 100 === 0) {
      const info = await connection.info();
      const currentEvictions = Number(
        info.match(/^evicted_keys:(\d+)$/m)?.[1] ?? '0',
      );
      if (currentEvictions - evictionBaseline > 10) {
        console.log(`  Eviction detected at job ${i}`);
      }
    }
  }

  // Phase 3: Inspect what survived
  console.log('\nPhase 3: Inspecting queue integrity');
  const counts = await queue.getJobCounts();
  console.log(`  Waiting: ${counts.waiting}`);
  console.log(`  Active:  ${counts.active}`);
  console.log(`  Failed:  ${counts.failed}`);

  // Scan actual keys to check for data loss
  let cursor = '0';
  let bullKeys = 0;
  do {
    const [nextCursor, keys] = await connection.scan(
      cursor,
      'MATCH',
      `bull:${queueName}:*`,
      'COUNT',
      '10000',
    );
    cursor = nextCursor;
    bullKeys += (keys as string[]).length;
  } while (cursor !== '0');

  console.log(`  Remaining BullMQ keys: ${bullKeys}`);

  await queue.obliterate({ force: true });
}

Failure signature for memory pressure: evicted_keys climbs non-zero, used_memory_rss peaks then drops as keys are evicted, BullMQ getJobCounts() returns numbers that don't match actual Redis keys, and workers may receive jobs with incomplete or corrupted data.

Experiment 3: Network Partition (Worker ↔ Redis)

A network partition silently severs the connection without crashing anything. BullMQ's behavior during a partition reveals important health signals:

interface PartitionTimelineEntry {
  offsetMs: number;
  event: string;
  waiting: number;
  active: number;
  failed: number;
}

async function simulateNetworkPartition(
  connection: IORedis,
): Promise<PartitionTimelineEntry[]> {
  const queueName = `chaos-partition-${Date.now()}`;
  const queue = new Queue(queueName, { connection });
  const timeline: PartitionTimelineEntry[] = [];

  const worker = new Worker(
    queueName,
    async () => {
      await new Promise((r) => setTimeout(r, 100));
    },
    { connection, concurrency: 5, stalledInterval: 10000 },
  );

  const injector = setInterval(() => {
    queue.add('job', { ts: Date.now() }).catch(() => {});
  }, 50);

  await new Promise((r) => setTimeout(r, 3000));

  // Record baseline
  const baseline = await queue.getJobCounts();
  timeline.push({
    offsetMs: 0,
    event: 'baseline',
    waiting: baseline.waiting ?? 0,
    active: baseline.active ?? 0,
    failed: baseline.failed ?? 0,
  });

  // Simulate partition by disconnecting
  await connection.disconnect();

  // Monitor during partition
  for (let i = 0; i < 15; i++) {
    await new Promise((r) => setTimeout(r, 1000));
    let counts;
    try {
      counts = await queue.getJobCounts();
    } catch {
      counts = { waiting: -1, active: -1, failed: -1 };
    }
    timeline.push({
      offsetMs: 1000 + i * 1000,
      event: 'partition_ongoing',
      waiting: counts.waiting ?? -1,
      active: counts.active ?? -1,
      failed: counts.failed ?? -1,
    });
  }

  // Restore connection
  connection.connect().catch(() => {});
  await new Promise((r) => setTimeout(r, 5000));

  const recovered = await queue.getJobCounts();
  timeline.push({
    offsetMs: 17000,
    event: 'recovered',
    waiting: recovered.waiting ?? 0,
    active: recovered.active ?? 0,
    failed: recovered.failed ?? 0,
  });

  clearInterval(injector);
  await worker.close();
  await queue.obliterate({ force: true });

  return timeline;
}

Partition failure signature: On a worker-side partition, active count stays elevated (jobs locked by zombie workers) while waiting grows. After stalledInterval, active drops and waiting spikes as jobs are un-stalled. On a Redis-side partition, waiting grows continuously while active stays at 0. The connected_clients metric in Redis INFO drops when workers lose connectivity.

Failure Mode Signatures — What Each Failure Looks Like in Your Metrics

A production queue failure rarely announces itself with an error message. Instead, it manifests as a pattern of metric deviations. Here is a catalog of five common failure modes with their characteristic signatures:

Failure Mode 1: Memory Pressure → Eviction → Silent Data Loss

Metric	Pre-failure	During	Post-failure
`used_memory_rss`	Steady climb toward `maxmemory`	Peaks, then oscillates	Stabilizes lower (keys evicted)
`evicted_keys`	0	Spikes	Remains elevated
`keyspace_hits`	Normal	Declining	Lower baseline
`keyspace_misses`	~0	Spikes	Elevated
Queue `waiting`	Consistent	Drops (jobs silently vanished)	Lower than expected

Action: If evicted_keys goes non-zero in any 60-second window, trigger an alert. Even a single eviction means BullMQ data is being destroyed.

Failure Mode 2: Network Partition (Worker Loses Redis)

Metric	Pre-failure	During	Post-failure
`connected_clients`	Stable	Drops by number of workers	Recovers
Queue `active`	~concurrency	Stays at last locked value	Spikes (stalled jobs re-queued)
Queue `waiting`	Flat or draining	Grows linearly	Drains quickly during catch-up
`instantaneous_ops_per_sec`	Consistent	Drops near 0	Doubles during catch-up

Signature: The active/waiting inversion. During normal operation, active ≈ concurrency and waiting is manageable. During a partition, active stays frozen while waiting grows unbounded.

Failure Mode 3: Redis OOM → Writes Rejected

Metric	Pre-failure	During	Post-failure
`used_memory_rss`	At/near `maxmemory`	Hits limit	May drop if eviction policy is set
`rejected_connections`	0	Spikes	Returns to 0
Worker `failed` jobs	Low	Spikes	Elevated

Critical nuance: If maxmemory-policy is noeviction (as BullMQ recommends), Redis stops accepting writes. Workers can't complete or fail jobs cleanly — jobs become stuck in the active state permanently until memory is freed.

Failure Mode 4: Worker Pool Depletion → Backlog Growth

Metric	Pre-failure	During	Post-failure
Queue `waiting`	Flat	Grows linearly	Drains proportionally to recovered capacity
Queue `active`	At expected concurrency	Drops to 0	Returns to concurrency
`connected_clients`	Stable	Drops by number of lost workers	Recovers
Latency P99	Normal	Approaches infinity	Normalizes

Signature: waiting grows linearly at the job arrival rate. The slope of the waiting line tells you the missing throughput. If jobs arrive at 200/s and waiting grows at 200/s, you have zero effective processing capacity.

Failure Mode 5: BullMQ Rate Limiter Engagement (Graceful Degradation)

Metric	Pre-failure	During	Post-failure
Queue `delayed`	Low	Spikes	Drains
Queue `waiting`	Flat	Limited	Drains
Latency P99	Normal	Slightly elevated	Normal
`instantaneous_ops_per_sec`	At limiter ceiling	Capped	Returns to load-driven

This is the only "healthy" failure mode — the rate limiter is doing its job. delayed grows while waiting stays controlled, and no jobs fail.

Building a Failure Mode Classifier

You can automate the detection of these failure modes with a simple classification function:

type FailureMode =
  | 'memory_eviction'
  | 'network_partition'
  | 'redis_oom'
  | 'worker_depletion'
  | 'rate_limiter_active'
  | 'healthy';

interface ClassificationInput {
  redisMemUsed: number;
  redisMemMax: number;
  evictedKeysDelta: number;
  connectedClients: number;
  expectedWorkers: number;
  activeJobs: number;
  waitingJobs: number;
  failedJobs: number;
  delayedJobs: number;
}

function classifyFailureMode(input: ClassificationInput): {
  mode: FailureMode;
  confidence: number;
  reasoning: string;
} {
  const {
    redisMemUsed, redisMemMax, evictedKeysDelta,
    connectedClients, expectedWorkers,
    activeJobs, waitingJobs, failedJobs, delayedJobs,
  } = input;

  const memUsageRatio = redisMemMax > 0 ? redisMemUsed / redisMemMax : 0;
  const missingWorkers = Math.max(0, expectedWorkers - connectedClients);
  const activeExpected = Math.min(expectedWorkers, 10);

  // 1. Memory eviction — most critical
  if (evictedKeysDelta > 0) {
    return {
      mode: 'memory_eviction',
      confidence: 0.95,
      reasoning: `${evictedKeysDelta} keys evicted in last interval — data loss in progress`,
    };
  }

  // 2. Redis OOM — writes rejected
  if (memUsageRatio > 0.95 && failedJobs > 10 && activeJobs > 0) {
    return {
      mode: 'redis_oom',
      confidence: 0.85,
      reasoning: `Memory at ${(memUsageRatio * 100).toFixed(0)}% of max with ${failedJobs} recent failures`,
    };
  }

  // 3. Network partition — workers disconnected
  if (missingWorkers > 0 && waitingJobs > 0 && activeJobs < activeExpected) {
    return {
      mode: 'network_partition',
      confidence: 0.8,
      reasoning: `${missingWorkers} of ${expectedWorkers} workers disconnected with ${waitingJobs} waiting`,
    };
  }

  // 4. Worker pool depletion
  if (activeJobs === 0 && waitingJobs > 100 && missingWorkers > 0) {
    return {
      mode: 'worker_depletion',
      confidence: 0.9,
      reasoning: `${waitingJobs} waiting but 0 active — ${missingWorkers} workers down`,
    };
  }

  // 5. Rate limiter active
  if (delayedJobs > 100 && waitingJobs < 50 && failedJobs === 0) {
    return {
      mode: 'rate_limiter_active',
      confidence: 0.7,
      reasoning: `${delayedJobs} delayed jobs but no failures — graceful throttling`,
    };
  }

  return {
    mode: 'healthy',
    confidence: 0.9,
    reasoning: 'No failure signature detected across all metrics',
  };
}

This classifier can be wired into a health-check daemon that runs every 30 seconds, classifying the current state of each queue and alerting on the memory_eviction or redis_oom modes immediately, while logging network_partition and worker_depletion for operational response.

Health Metric Regression in CI/CD

Once you can generate load and measure health signals, the next step is to automate it as a CI/CD gate: every deployment candidate must prove it doesn't degrade queue health.

interface HealthRegressionConfig {
  scenarioName: string;
  rps: number;
  maxP99Ms: number;
  maxFailureRate: number;
  maxStalledJobs: number;
  maxMemoryGrowthBytes: number;
}

const REGRESSION_TESTS: HealthRegressionConfig[] = [
  {
    scenarioName: 'light-load',
    rps: 100,
    maxP99Ms: 30,
    maxFailureRate: 0.001,
    maxStalledJobs: 0,
    maxMemoryGrowthBytes: 50 * 1024 * 1024,
  },
  {
    scenarioName: 'medium-load',
    rps: 500,
    maxP99Ms: 50,
    maxFailureRate: 0.005,
    maxStalledJobs: 1,
    maxMemoryGrowthBytes: 200 * 1024 * 1024,
  },
  {
    scenarioName: 'peak-load',
    rps: 1000,
    maxP99Ms: 100,
    maxFailureRate: 0.01,
    maxStalledJobs: 5,
    maxMemoryGrowthBytes: 500 * 1024 * 1024,
  },
];

interface RegressionResult {
  scenario: string;
  passed: boolean;
  metrics: {
    actualThroughput: number;
    p99Ms: number;
    failureRate: number;
    stalledJobs: number;
    memoryGrowthBytes: number;
  };
  failures: string[];
}

async function runHealthRegression(
  connection: IORedis,
): Promise<{ results: RegressionResult[]; overallPass: boolean }> {
  const results: RegressionResult[] = [];

  for (const test of REGRESSION_TESTS) {
    const loadResult = await runLoadTest(connection, {
      targetRps: test.rps,
      durationSeconds: 20,
      jobProcessingMeanMs: 10,
      jobPayloadBytes: 1024,
      workerConcurrency: 5,
    });

    const snapshots = loadResult.snapshots;
    const memoryGrowthBytes = snapshots.length >= 2
      ? snapshots[snapshots.length - 1].redis.usedMemoryRss -
        snapshots[0].redis.usedMemoryRss
      : 0;

    const failures: string[] = [];

    if (loadResult.latency.p99 > test.maxP99Ms) {
      failures.push(
        `P99 ${loadResult.latency.p99.toFixed(1)}ms exceeds max ${test.maxP99Ms}ms`,
      );
    }

    const failureRate =
      loadResult.totalProcessed + loadResult.totalFailed > 0
        ? loadResult.totalFailed / (loadResult.totalProcessed + loadResult.totalFailed)
        : 0;

    if (failureRate > test.maxFailureRate) {
      failures.push(
        `Failure rate ${(failureRate * 100).toFixed(2)}% exceeds max ${(test.maxFailureRate * 100).toFixed(2)}%`,
      );
    }

    if (loadResult.totalStalled > test.maxStalledJobs) {
      failures.push(
        `Stalled jobs ${loadResult.totalStalled} exceeds max ${test.maxStalledJobs}`,
      );
    }

    if (memoryGrowthBytes > test.maxMemoryGrowthBytes) {
      failures.push(
        `Memory growth ${(memoryGrowthBytes / (1024 * 1024)).toFixed(0)}MB exceeds max ${(test.maxMemoryGrowthBytes / (1024 * 1024)).toFixed(0)}MB`,
      );
    }

    results.push({
      scenario: test.scenarioName,
      passed: failures.length === 0,
      metrics: {
        actualThroughput: loadResult.actualThroughput,
        p99Ms: loadResult.latency.p99,
        failureRate,
        stalledJobs: loadResult.totalStalled,
        memoryGrowthBytes,
      },
      failures,
    });
  }

  const overallPass = results.every((r) => r.passed);

  if (!overallPass) {
    console.error('\n❌ QUEUE HEALTH REGRESSION FAILED');
    for (const r of results.filter((r) => !r.passed)) {
      console.error(`  ${r.scenario}:`);
      for (const f of r.failures) {
        console.error(`    - ${f}`);
      }
    }
    process.exit(1);
  } else {
    console.log('\n✅ All queue health regression tests passed');
  }

  return { results, overallPass };
}

Integrate this into a GitHub Actions workflow that runs on any PR touching worker or queue configuration:

name: Queue Health Regression
on:
  pull_request:
    paths:
      - 'workers/**'
      - 'queue-config/**'
      - 'package.json'

jobs:
  health-regression:
    runs-on: ubuntu-latest
    services:
      redis:
        image: redis:7-alpine
        ports:
          - 6379:6379
        options: >-
          --health-cmd "redis-cli ping"
          --health-interval 5s
          --health-timeout 3s
          --health-retries 5
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'
      - run: npm ci
      - name: Run Queue Health Regression
        run: npx tsx ci/health-regression.ts
        env:
          REDIS_HOST: localhost
          REDIS_PORT: 6379

What this catches that normal tests miss: A code change that adds an O(n) Redis operation inside a worker, a dependency update that changes serialization behavior, a configuration change that increases job payloads, or a worker concurrency change that pushes Redis past its ops/sec ceiling.

Capacity Validation Through Load Testing

Capacity planning is often done with napkin math: "We expect 200 jobs/s at peak, each job takes 50ms, so we need 10 workers and a Redis instance with X GB." This section replaces theory with measurement.

interface RedisCapacity {
  maxThroughput: number;
  maxLatencyThroughput: number;
  memoryGrowthRate: number;
  recommendedWorkers: number;
  recommendedInstance: string;
}

async function validateCapacity(
  connection: IORedis,
  redisMaxMemory: number,
): Promise<RedisCapacity> {
  const rpsLevels = [50, 100, 250, 500, 750, 1000, 1500, 2000];
  const results: Map<number, LoadTestResult> = new Map();

  for (const rps of rpsLevels) {
    console.log(`Validating at ${rps} RPS...`);
    const result = await runLoadTest(connection, {
      targetRps: rps,
      durationSeconds: 20,
      jobProcessingMeanMs: 10,
      jobPayloadBytes: 1024,
      workerConcurrency: Math.min(Math.ceil(rps / 50), 20),
    });
    results.set(rps, result);

    if (result.degraded && result.actualThroughput < rps * 0.8) {
      console.log(`  Saturation at ${rps} RPS (achieved ${result.actualThroughput})`);
      break;
    }
  }

  // Find the sweet spot
  let maxThroughput = 0;
  let maxLatencyThroughput = 0;

  for (const [rps, result] of results) {
    if (!result.degraded) {
      maxThroughput = Math.max(maxThroughput, result.actualThroughput);
      maxLatencyThroughput = rps;
    }
  }

  if (maxLatencyThroughput === 0 && results.size > 0) {
    const lastRps = Math.max(...Array.from(results.keys()));
    maxLatencyThroughput = lastRps;
    maxThroughput = results.get(lastRps)?.actualThroughput ?? 0;
  }

  // Calculate memory growth per job
  const memResults = Array.from(results.entries())
    .filter(([, r]) => r.snapshots.length >= 2 && !r.degraded);

  let memoryGrowthPerJob = 0;
  if (memResults.length > 0) {
    const rates = memResults.map(([_rps, r]) => {
      const first = r.snapshots[0].redis.usedMemoryRss;
      const last = r.snapshots[r.snapshots.length - 1].redis.usedMemoryRss;
      return r.totalProcessed > 0 ? (last - first) / r.totalProcessed : 0;
    });
    memoryGrowthPerJob = rates.reduce((a, b) => a + b, 0) / rates.length;
  }

  const recommendedWorkers = Math.ceil(maxLatencyThroughput * 0.01);
  const estimatedBytes = memoryGrowthPerJob * maxLatencyThroughput * 3600;
  const recommendedMemMb = Math.ceil((estimatedBytes * 2) / (1024 * 1024));

  const instance = recommendedMemMb < 256
    ? 't4g.small / cache.t3.small (1GB)'
    : recommendedMemMb < 1024
    ? 't4g.medium / cache.t3.medium (4GB)'
    : recommendedMemMb < 4096
    ? 'm7g.large / cache.r7g.large (16GB)'
    : 'm7g.xlarge+ / cache.r7g.xlarge+ (32GB+)';

  return {
    maxThroughput,
    maxLatencyThroughput,
    memoryGrowthRate: memoryGrowthPerJob,
    recommendedWorkers,
    recommendedInstance: `${instance} (est. ${recommendedMemMb}MB for 1h retention at ${maxLatencyThroughput} rps)`,
  };
}

Practical Capacity Rules Derived From Load Testing

Through extensive testing across Redis instance sizes, these patterns emerge:

Redis Memory	Max Throughput (10ms jobs)	Workers	Bottleneck
1 GB	~800 jobs/s	8-10	CPU (single-core Redis)
4 GB	~3,000 jobs/s	25-30	CPU + Memory bandwidth
16 GB	~10,000 jobs/s	80-100	Network IO
32 GB+	~20,000 jobs/s	150-200	Network IO / CPU

Key insight: BullMQ throughput bottlenecks on Redis CPU, not memory (as long as memory is adequate). A cache.t3.small (1GB) and a cache.r7g.large (16GB) have similar CPU cores — upgrading memory without upgrading CPU won't increase throughput. For higher throughput, use Redis Cluster to distribute operations across shards.

Understanding BullMQ's Degradation Patterns Under Stress

As Redis resource contention increases, BullMQ exhibits three distinct degradation stages. Recognizing the thresholds where graceful degradation becomes hard failure is essential for building resilient queue systems.

Stage 1: Latency Tail Growth (Graceful)

When Redis approaches its CPU ceiling, individual operations take longer. BullMQ uses Redis pipelining internally, so one slow command can delay a batch.

async function measureDegradationUnderContention(
  connection: IORedis,
  queueName: string,
): Promise<void> {
  const queue = new Queue(queueName, { connection });
  const baseline: number[] = [];
  const contended: number[] = [];

  // Baseline measurement
  for (let i = 0; i < 100; i++) {
    const start = performance.now();
    await queue.getJobCounts();
    baseline.push(performance.now() - start);
  }

  const baselineP99 = baseline.sort((a, b) => a - b)[99];

  // Generate contention with heavy LUA scripts
  const tasks: Promise<void>[] = [];
  for (let i = 0; i < 20; i++) {
    tasks.push(
      (async () => {
        for (let j = 0; j < 500; j++) {
          await connection.eval(
            ` local s = 0 for i=1,10000 do s = s + redis.call("GET", KEYS[1]) end return s `,
            1,
            `contention:${i}`,
          );
        }
      })(),
    );
  }

  await new Promise((r) => setTimeout(r, 1000));

  // Measure under contention
  for (let i = 0; i < 100; i++) {
    const start = performance.now();
    await queue.getJobCounts();
    contended.push(performance.now() - start);
  }

  await Promise.allSettled(tasks);

  const contendedP99 = contended.sort((a, b) => a - b)[99];

  console.log(`Baseline getJobCounts P99: ${baselineP99.toFixed(2)}ms`);
  console.log(`Under contention P99:      ${contendedP99.toFixed(2)}ms`);
  console.log(`Degradation factor:        ${(contendedP99 / baselineP99).toFixed(2)}x`);

  if (contendedP99 > baselineP99 * 5) {
    console.log('⚠ Latency degradation may cause worker lock timeouts');
  }

  await queue.close();
}

Stage 2: Worker Lock Contention (Degraded but Stable)

BullMQ workers extend job processing locks by sending periodic heartbeats to Redis. Under stress:

The heartbeat command itself is slow (latency degraded)
Heartbeat intervals stretch, approaching stalledInterval
Some heartbeats arrive after the stall timeout — jobs are marked stalled
The same job is claimed by another worker, which also struggles

This creates a stall cycle: the more jobs are stalled, the more getJobCounts() and state transitions Redis must handle, increasing contention further.

Stage 3: Redis Backpressure (Hard Failure Threshold)

The point where graceful degradation becomes hard failure is defined by BullMQ's internal configuration:

interface DegradationThresholds {
  lockExtensionFailureLatencyMs: number;
  connectionPoolExhaustionLatencyMs: number;
  opsPerSecSaturation: number;
}

function calculateDegradationThresholds(
  stalledInterval: number,
  lockDuration: number,
): DegradationThresholds {
  // Lock extension happens roughly halfway through lock duration.
  // If one extension takes longer than remaining lock time, lock expires.
  const extensionDeadlineMs = lockDuration * 0.4;

  // Connection pool: ioredis commands queue up at high latency.
  // ~200ms latency triggers pool queue growth for active workers.
  const poolExhaustionLatency = 200;

  return {
    lockExtensionFailureLatencyMs: extensionDeadlineMs,
    connectionPoolExhaustionLatencyMs: poolExhaustionLatency,
    opsPerSecSaturation: 25000, // single-core Redis: ~25K ops/sec
  };
}

The Degradation Decision Matrix

Redis Latency (P99)	BullMQ Behavior	Stalled Jobs	Data Integrity
<10ms	Normal	~0	Perfect
10-50ms	Slower processing, lock extensions succeed	~0	Perfect
50-100ms	Lock extensions at risk, occasional stalls	<1%	Perfect
100-200ms	Frequent stalls, workers re-processing jobs	1-5%	Perfect (at-least-once)
200-500ms	Heavy stall cycle, duplicate jobs	5-15%	Perfect (at-least-once)
500ms+	Near-halt, connection pool may exhaust	15%+	At risk
1000ms+	Hard failure: writes rejected, workers disconnected	Total halt	Data loss risk

The critical insight: BullMQ's at-least-once delivery guarantee holds up remarkably well under moderate stress (P99 < 200ms). Jobs may be processed multiple times, but none are lost. The hard failure threshold is when Redis latency exceeds ~200ms P99 — beyond this point, the connection pool can exhaust, job adds can fail, and the system enters a partial-loss regime.

Conclusion

Static health monitoring tells you your queue is alive. Stress testing and chaos engineering tell you your queue is ready.

The techniques in this post form a validation pipeline that goes far beyond simple INFO command checks:

Technique	What It Validates	When To Run
Load test harness	Throughput ceiling, latency under load	Every deployment
Chaos: Redis failover	Recovery time, stall behavior	Weekly / pre-release
Chaos: Memory pressure	Eviction risk, data loss detection	Monthly
Chaos: Network partition	Zombie worker behavior, catch-up latency	Monthly
Health regression in CI	No performance regressions	Every pull request
Capacity validation	Instance sizing, worker count	Quarterly / topology changes

By integrating these practices into your development lifecycle, you move from "monitoring for fires" to "engineering for resilience" — knowing not just that your queue is healthy, but that it stays healthy under the conditions that actually matter.

Ready to see your queue health in real time?

Queue Hub gives you a beautiful, real-time dashboard that surfaces exactly the health signals this post discusses — latency distributions, eviction warnings, worker heartbeats, job throughput, and failure rates — all without writing a single Redis command.

Multi-backend: BullMQ, BeeQueue — switch in settings
Live health telemetry: Throughput, latency, worker status at a glance
Deep job inspection: Data, traces, stack traces, logs
Bulk actions: Retry, promote, purge — from one UI
Secure tunneling: Connect to private Redis via WebSocket agent relay
Team collaboration: Multi-org with role management

Try Queue Hub Free — Connect your Redis in 60 seconds.