·QueueHub Team·16 min read

Proactive Redis Queue Health: Anomaly Detection and Capacity Planning for BullMQ

BullMQRedisAnomaly DetectionCapacity PlanningPredictive MonitoringTypeScriptSRE

Your dashboards are live. Your alerts fire when Redis hits 90% memory or latency spikes. You saw the problem coming three hours before the alert — the memory growth curve was linear, predictable, and completely invisible to your threshold-based monitoring tools.

This is the fundamental gap in queue observability today. Threshold-based monitoring tells you when something breaks. Anomaly detection and trend analysis tell you when something will break. The difference is the gap between being woken up at 3 AM to resize your ElastiCache instance versus doing it during business hours while sipping coffee.

In this post, we'll build a proactive monitoring pipeline for BullMQ queues that uses statistical anomaly detection, trend-based capacity forecasting, and a unified health scoring system. Every code example is production-ready TypeScript.

Statistical Anomaly Detection for Queue Metrics

Traditional threshold-based alerts — "alert if memory > 80%" or "alert if failed jobs > 10" — are brittle. They miss gradual degradation and trigger false alarms on predictable spikes. Statistical methods adapt to your queue's normal patterns automatically.

Z-Score Anomaly Detection for Job Failure Rates

A z-score measures how many standard deviations a data point is from the historical mean. Values with an absolute z-score greater than 3 are statistically unusual — likely anomalous.

For BullMQ, instead of hard-coding "alert on failed jobs > 10," compute the z-score of the failure rate over a rolling window. A burst of 20 failures might be normal during peak hours but anomalous at 3 AM.

import { Queue } from 'bullmq';
import IORedis from 'ioredis';

interface AnomalyResult {
  value: number;
  zScore: number;
  isAnomaly: boolean;
  mean: number;
  stdDev: number;
}

class ZScoreAnomalyDetector {
  private windows = new Map<string, number[]>();

  constructor(
    private readonly windowSize = 60,
    private readonly threshold = 3
  ) {}

  feed(key: string, value: number): AnomalyResult {
    let window = this.windows.get(key);
    if (!window) {
      window = [];
      this.windows.set(key, window);
    }

    window.push(value);
    if (window.length > this.windowSize) window.shift();

    if (window.length < 10) {
      return { value, zScore: 0, isAnomaly: false, mean: value, stdDev: 0 };
    }

    const mean = window.reduce((a, b) => a + b, 0) / window.length;
    const variance = window.reduce((sum, v) => sum + (v - mean) ** 2, 0) / window.length;
    const stdDev = Math.sqrt(variance) || 1;
    const zScore = (value - mean) / stdDev;

    return { value, zScore, isAnomaly: Math.abs(zScore) > this.threshold, mean, stdDev };
  }
}

// Usage with BullMQ
async function monitorFailureAnomalies(queue: Queue, detector: ZScoreAnomalyDetector) {
  const counts = await queue.getJobCounts('failed', 'completed');
  const total = counts.completed + counts.failed;
  const failureRate = total > 0 ? (counts.failed / total) * 100 : 0;

  const result = detector.feed(queue.name, failureRate);
  console.log(`[${queue.name}] Failure rate: ${failureRate.toFixed(2)}%`);
  console.log(`  Z-score: ${result.zScore.toFixed(2)} — ${result.isAnomaly ? '🚨 ANOMALY' : '✅ Normal'}`);
  return result;
}

const connection = new IORedis({ host: 'localhost', port: 6379, maxRetriesPerRequest: null });
const queue = new Queue('notifications', { connection });
const detector = new ZScoreAnomalyDetector(60, 3);

EWMA for Detecting Memory Acceleration

The Z-score detects point anomalies, but the most dangerous queue problems are gradual. Memory leaks, connection pool exhaustion, and worker degradation all happen slowly. The Exponentially Weighted Moving Average (EWMA) gives more weight to recent observations, making it sensitive to trend changes.

class EWMAMemoryDetector {
  private ema: number | null = null;
  private emv = 0;
  private dataPoints = 0;

  constructor(private readonly alpha = 0.15) {}

  feed(memoryPercent: number) {
    this.dataPoints++;

    if (this.ema === null) {
      this.ema = memoryPercent;
      return { smoothed: memoryPercent, raw: memoryPercent, deviation: 0, isAccelerating: false };
    }

    const prevEma = this.ema;
    this.ema = this.alpha * memoryPercent + (1 - this.alpha) * this.ema;

    const diff = memoryPercent - this.ema;
    this.emv = (1 - this.alpha) * (this.emv + this.alpha * diff * diff);

    const stdDev = Math.sqrt(this.emv);
    const deviation = (memoryPercent - this.ema) / (stdDev || 1);

    return {
      smoothed: this.ema,
      raw: memoryPercent,
      deviation,
      isAccelerating: deviation > 2.0 && this.dataPoints > 10,
    };
  }
}

async function pollRedisMemory(connection: IORedis.Redis, maxMemoryMB: number, detector: EWMAMemoryDetector) {
  const info = await connection.info('memory');
  const match = info.match(/used_memory:(\d+)/);
  if (!match) throw new Error('Could not parse used_memory');

  const usedPercent = (parseInt(match[1], 10) / (1024 * 1024)) / maxMemoryMB * 100;
  const result = detector.feed(usedPercent);

  console.log(`[Memory] ${result.isAccelerating ? '⚠️ ACCELERATING' : '✅ Stable'} — ${usedPercent.toFixed(1)}% (smoothed: ${result.smoothed.toFixed(1)}%, deviation: ${result.deviation.toFixed(2)}σ)`);
  return result;
}

Pattern Recognition: Spike vs. Drift vs. Cycle

Not all anomalies are the same. The shape of the anomaly tells you what's happening:

Pattern Z-Score Signature Likely Cause
Spike failure |z| > 4, single point API outage, network blip
Gradual drift |z| climbs 1→2→3 over hours Memory leak, connection drain
Daily oscillation |z| cycles at 24h intervals Peak-hour load, global cron
Step change |z| jumps and stays Config change, new deploy
Random bursts |z| spikes then recovers Retry storms, network jitter

A combined approach — Z-score for point anomalies and EWMA for trend shifts — gives you coverage for both acute and chronic problems.

Trend Analysis and Predictive Capacity Planning

Reactive scaling (adding Redis memory after OOM) is expensive and dangerous. Predictive capacity planning uses historical trends to forecast when you'll hit limits.

Linear Regression for Memory Growth Forecasting

Fit a line to historical memory usage data and project forward. The slope tells you your daily growth rate. The intersection with your maxmemory threshold tells you your "day of reckoning."

interface DataPoint { timestamp: number; value: number; }

interface ForecastResult {
  slopeMBPerDay: number;
  rSquared: number;
  forecast: (daysFromNow: number) => number;
  daysUntilLimit: number | null;
}

class MemoryForecaster {
  forecast(data: DataPoint[], maxMemoryMB: number): ForecastResult {
    const t0 = data[0].timestamp;
    const days = data.map(d => (d.timestamp - t0) / 86400000);
    const values = data.map(d => d.value);
    const n = data.length;

    const sumX = days.reduce((a, b) => a + b, 0);
    const sumY = values.reduce((a, b) => a + b, 0);
    const sumXY = days.reduce((s, x, i) => s + x * values[i], 0);
    const sumX2 = days.reduce((s, x) => s + x * x, 0);

    const slope = (n * sumXY - sumX * sumY) / (n * sumX2 - sumX * sumX);
    const intercept = (sumY - slope * sumX) / n;

    // R-squared
    const meanY = sumY / n;
    const ssRes = values.reduce((s, y, i) => s + (y - (intercept + slope * days[i])) ** 2, 0);
    const ssTot = values.reduce((s, y) => s + (y - meanY) ** 2, 0);
    const rSquared = 1 - ssRes / ssTot;

    const lastDay = days[days.length - 1];
    let daysUntilLimit: number | null = null;
    if (slope > 0) {
      daysUntilLimit = Math.max(0, (maxMemoryMB - intercept) / slope - lastDay);
    }

    return {
      slopeMBPerDay: slope * 86400000,
      rSquared,
      forecast: (d: number) => intercept + slope * (lastDay + d),
      daysUntilLimit,
    };
  }
}

async function generateCapacityReport(connection: IORedis.Redis, maxMemoryMB: number, history: DataPoint[]) {
  const forecaster = new MemoryForecaster();
  const result = forecaster.forecast(history, maxMemoryMB);
  const current = history[history.length - 1].value;

  console.log('=== Redis Capacity Forecast ===');
  console.log(`Current: ${current.toFixed(0)} MB / ${maxMemoryMB} MB (${(current / maxMemoryMB * 100).toFixed(1)}%)`);
  console.log(`Growth rate: ${result.slopeMBPerDay.toFixed(0)} MB/day`);
  console.log(`R² fit: ${result.rSquared.toFixed(3)}`);

  if (result.daysUntilLimit !== null) {
    if (result.daysUntilLimit <= 7) {
      console.log(`🚨 Will hit maxmemory in ${result.daysUntilLimit.toFixed(1)} days — URGENT`);
    } else if (result.daysUntilLimit <= 30) {
      console.log(`⚠️ Will hit maxmemory in ${result.daysUntilLimit.toFixed(1)} days — plan resize`);
    } else {
      console.log(`✅ Projected to hit maxmemory in ${result.daysUntilLimit.toFixed(1)} days — adequate`);
    }
    console.log(`  30-day forecast: ${result.forecast(30).toFixed(0)} MB`);
    console.log(`  90-day forecast: ${result.forecast(90).toFixed(0)} MB`);
  }
}

Backlog Trend Analysis

A steadily growing backlog means you're losing ground. The formula for "days to drain" tells you whether your current worker pool is sustainable:

daysToDrain = backlogSize / (throughputPerDay - arrivalRatePerDay)

If the arrival rate exceeds throughput, the backlog grows forever — you need more workers.

interface BacklogAnalysis {
  currentBacklog: number;
  arrivalRatePerHour: number;
  throughputPerHour: number;
  backlogTrend: 'growing' | 'shrinking' | 'stable';
  daysToDrain: number | null;
  workersNeeded: number;
}

class BacklogAnalyzer {
  analyze(snapshots: { timestamp: number; waiting: number; completed: number; failed: number }[]): BacklogAnalysis {
    const first = snapshots[0];
    const last = snapshots[snapshots.length - 1];
    const hours = (last.timestamp - first.timestamp) / 3600000;

    const processed = (last.completed - first.completed) + (last.failed - first.failed);
    const throughput = processed / hours;
    const arrived = (last.waiting - first.waiting) + processed;
    const arrivalRate = arrived / hours;

    const netRate = throughput - arrivalRate;
    let trend: 'growing' | 'shrinking' | 'stable';
    let daysToDrain: number | null = null;
    let workersNeeded = 0;

    if (Math.abs(netRate) < arrivalRate * 0.05) {
      trend = 'stable';
    } else if (netRate < 0) {
      trend = 'growing';
      workersNeeded = Math.ceil(Math.abs(netRate) / throughput);
    } else {
      trend = 'shrinking';
      daysToDrain = last.waiting / (netRate * 24);
    }

    return {
      currentBacklog: last.waiting,
      arrivalRatePerHour: Math.round(arrivalRate * 100) / 100,
      throughputPerHour: Math.round(throughput * 100) / 100,
      backlogTrend: trend,
      daysToDrain: daysToDrain !== null ? Math.round(daysToDrain * 10) / 10 : null,
      workersNeeded,
    };
  }
}

Getting the Raw Data from BullMQ

BullMQ's built-in getMetrics() method provides per-minute job completion and failure counts. This is the foundation data for all trend analysis:

import { MetricsTime } from 'bullmq';

async function collectMetrics(queue: Queue) {
  const completed = await queue.getMetrics('completed', 0, MetricsTime.ONE_WEEK * 2);
  const failed = await queue.getMetrics('failed', 0, MetricsTime.ONE_WEEK * 2);

  console.log(`Data points: ${completed.data.length} (completed), ${failed.data.length} (failed)`);
  console.log(`Last minute: ${completed.data[completed.data.length - 1]} completed`);
  return { completed, failed };
}

Building a Queue Health Scoring System

Individual metrics are noise. A health score aggregates multiple signals into a single number (0–100) that tells you at a glance whether your queue infrastructure needs attention.

Factor Weight What It Measures
Memory pressure 30% Used / maxmemory ratio
Backlog health 25% Trend direction + size
Failure rate 20% Z-score deviation from baseline
Processing latency 15% EWMA-smoothed completion time
Worker coverage 10% Active vs. required workers 
interface HealthScoreInput {
  memoryUsagePercent: number;
  backlogGrowing: boolean;
  backlogSize: number;
  failureRate: number;
  failureZScore: number;
  avgProcessingTimeMs: number;
  expectedProcessingTimeMs: number;
  activeWorkers: number;
  requiredWorkers: number;
}

interface HealthScoreResult {
  score: number;
  grade: 'A' | 'B' | 'C' | 'D' | 'F';
  recommendations: string[];
}

class QueueHealthScorer {
  score(inputs: HealthScoreInput): HealthScoreResult {
    const recs: string[] = [];

    const memoryScore = Math.max(0, 100 - inputs.memoryUsagePercent * 1.1);
    if (inputs.memoryUsagePercent > 80) recs.push('Redis memory above 80%. Review retention policies.');

    let backlogScore: number;
    if (inputs.backlogGrowing) {
      backlogScore = Math.max(0, 100 - Math.log2(inputs.backlogSize + 1) * 15);
      recs.push(`Backlog growing (${inputs.backlogSize} waiting). Consider adding workers.`);
    } else {
      backlogScore = inputs.backlogSize > 1000 ? 70 : 95;
    }

    let failureScore: number;
    if (inputs.failureZScore > 3) {
      failureScore = Math.max(0, 100 - (inputs.failureZScore - 3) * 25);
      recs.push(`Abnormal failure rate (z=${inputs.failureZScore.toFixed(1)}).`);
    } else if (inputs.failureRate > 5) {
      failureScore = 60;
    } else {
      failureScore = 95;
    }

    const ratio = inputs.avgProcessingTimeMs / (inputs.expectedProcessingTimeMs || 1);
    const latencyScore = ratio <= 1 ? 100 : ratio <= 2 ? 80 : ratio <= 5 ? 50 : Math.max(0, 100 - ratio * 10);
    if (ratio > 2) recs.push(`Processing times ${ratio.toFixed(1)}x expected. Investigate workers.`);

    const workerRatio = inputs.activeWorkers / (inputs.requiredWorkers || 1);
    const workerScore = workerRatio >= 1 ? 100 : workerRatio >= 0.75 ? 80 : workerRatio >= 0.5 ? 60 : 30;
    if (workerRatio < 0.75) recs.push(`Worker shortage (${inputs.activeWorkers}/${inputs.requiredWorkers}).`);

    const score = Math.round(
      memoryScore * 0.30 + backlogScore * 0.25 + failureScore * 0.20 + latencyScore * 0.15 + workerScore * 0.10
    );

    const grade: HealthScoreResult['grade'] = score >= 90 ? 'A' : score >= 70 ? 'B' : score >= 50 ? 'C' : score >= 30 ? 'D' : 'F';

    return { score, grade, recommendations: recs };
  }
}

// Usage
async function reportHealth(queue: Queue, connection: IORedis.Redis) {
  const counts = await queue.getJobCounts('waiting', 'active', 'completed', 'failed');
  const info = await connection.info('memory');
  const used = parseInt(info.match(/used_memory:(\d+)/)?.[1] ?? '0', 10);
  const max = parseInt(info.match(/maxmemory:(\d+)/)?.[1] ?? '0', 10);

  const scorer = new QueueHealthScorer();
  const report = scorer.score({
    memoryUsagePercent: max > 0 ? (used / max) * 100 : 50,
    backlogGrowing: false,
    backlogSize: counts.waiting,
    failureRate: counts.completed > 0 ? (counts.failed / (counts.completed + counts.failed)) * 100 : 0,
    failureZScore: 1.2,
    avgProcessingTimeMs: 450,
    expectedProcessingTimeMs: 300,
    activeWorkers: 3,
    requiredWorkers: 5,
  });

  console.log(`\n=== Health Score: ${queue.name} ===`);
  console.log(`Overall: ${report.score}/100 (Grade ${report.grade})`);
  report.recommendations.forEach(r => console.log(`  → ${r}`));
}

SLO Tracking with Burn-Rate Alerting

Queue health is a means to an end — delivering jobs within SLO. Instead of alerting after a breach, use burn-rate alerting to catch violations before they happen.

The idea: Track your error budget consumption rate. A burn rate of 1.0 means you're consuming budget at exactly the rate that exhausts it at the end of the window. A burn rate of 2.0 means you'll exhaust it twice as fast.

SLO Definition Typical Target
Processing SLO Jobs completed within time threshold 99% < 30s
Freshness SLO Jobs picked up within delay 99.9% < 5s
Completion SLO Jobs that eventually succeed 99.5%
Throughput SLO Minimum jobs per minute Varies 
interface SLOConfig {
  target: number;       // e.g., 0.995 for 99.5%
  burnRateThresholds: { critical: number; warning: number };
}

class SLOBurnRateTracker {
  constructor(private config: SLOConfig) {}

  evaluate(totalJobs: number, failedJobs: number): { attainment: number; budgetRemaining: number; burnRate: number; status: string } {
    const ok = totalJobs - failedJobs;
    const attainment = totalJobs > 0 ? ok / totalJobs : 1;
    const allowedErrors = Math.floor(totalJobs * (1 - this.config.target));
    const budgetRemaining = Math.max(0, allowedErrors - failedJobs);
    const allowedErrorRate = 1 - this.config.target;
    const burnRate = allowedErrorRate > 0 ? (failedJobs / totalJobs) / allowedErrorRate : 0;

    let status: string;
    if (burnRate >= this.config.burnRateThresholds.critical) status = '🚨 CRITICAL';
    else if (burnRate >= this.config.burnRateThresholds.warning) status = '⚠️ WARNING';
    else status = '✅ HEALTHY';

    return {
      attainment: Math.round(attainment * 10000) / 100,
      budgetRemaining: Math.round(budgetRemaining * 100) / 100,
      burnRate: Math.round(burnRate * 100) / 100,
      status,
    };
  }
}

async function checkSLO(queue: Queue) {
  const metrics = await queue.getMetrics('completed', 0, MetricsTime.ONE_WEEK);
  const failedMetrics = await queue.getMetrics('failed', 0, MetricsTime.ONE_WEEK);

  const totalCompleted = metrics.data.reduce((a, b) => a + b, 0);
  const totalFailed = failedMetrics.data.reduce((a, b) => a + b, 0);

  const tracker = new SLOBurnRateTracker({
    target: 0.995,
    burnRateThresholds: { critical: 3, warning: 1.5 },
  });

  const result = tracker.evaluate(totalCompleted + totalFailed, totalFailed);
  console.log(`SLO: ${result.attainment}% | Budget: ${result.budgetRemaining}% | Burn rate: ${result.burnRate}x | ${result.status}`);
}

Automated Health Reports and Cost Optimization

Real-time dashboards are great for firefighting. Period-over-period reports reveal slow-burn trends that dashboards flatten.

Weekly Comparison Reports

Track week-over-week changes for all key metrics. A 5% week-over-week memory increase sustained for a month is a 22% monthly growth — invisible on any single day's dashboard but obvious in a trend view.

interface WeeklyComparison {
  metric: string;
  current: number;
  previous: number;
  changePercent: number;
  trend: 'good' | 'bad' | 'neutral';
}

function compareWeek(current: any, previous: any): WeeklyComparison[] {
  const compute = (metric: string, cur: number, prev: number, lowerBetter: boolean) => {
    const change = prev > 0 ? ((cur - prev) / prev) * 100 : 0;
    const isBad = lowerBetter ? change > 5 : change < -5;
    return { metric, current: cur, previous: prev, changePercent: Math.round(change * 10) / 10, trend: isBad ? 'bad' : Math.abs(change) > 5 ? 'good' : 'neutral' as const };
  };

  return [
    compute('Memory %', current.memoryPct, previous.memoryPct, true),
    compute('Failure Rate %', current.failureRate, previous.failureRate, true),
    compute('Backlog', current.backlog, previous.backlog, true),
    compute('Throughput', current.throughput, previous.throughput, false),
    compute('Health Score', current.healthScore, previous.healthScore, false),
  ];
}

// Schedule as a BullMQ repeatable job
const reportQueue = new Queue('queue-health-reports', { connection });
await reportQueue.upsertJobScheduler('weekly-report', { pattern: '0 9 * * 1' }, {
  name: 'generate-report',
  data: { type: 'weekly', channels: ['slack:#ops'] },
  opts: { removeOnComplete: { age: 86400 * 30 } },
});

Data-Driven Redis Right-Sizing

Most teams either over-provision (wasting money) or under-provision (risking OOM). Trend data lets you find the sweet spot.

Use your linear regression forecast to determine the optimal Redis instance size, then add 30% headroom for spikes:

required = max(currentUsage, forecast90day) × 1.3

Round up to the nearest instance tier. The cost savings from downsizing one over-provisioned database can easily pay for your entire monitoring stack.

Integrating with Queue Hub

Queue Hub surfaces these predictive signals alongside real-time queue data, giving operators a single pane of glass for both reactive and proactive monitoring:

  • Health score timeline — plotted health score over days and weeks, annotated with anomalous events
  • Capacity forecast cards — "Forecast to hit 90% memory in 14 days" on your queue overview
  • Anomaly feed — z-score events logged and correlated with job types and deployment markers
  • SLO burn-rate alerts — integrated notifications when burn rates exceed thresholds
  • Weekly health digests — auto-generated reports comparing this week to last

Pushing anomaly events to Queue Hub is as simple as a webhook call:

async function reportAnomaly(event: { queueName: string; metric: string; value: number; zScore: number }) {
  await fetch('https://api.queuehub.tech/v1/events', {
    method: 'POST',
    headers: { 'Authorization': 'Bearer YOUR_API_KEY', 'Content-Type': 'application/json' },
    body: JSON.stringify({ type: 'anomaly_detected', source: 'proactive-monitor', ...event }),
  });
}

Putting It All Together

The full proactive monitoring pipeline looks like this:

Data Collection → Analysis Layer → Action Layer → Output
  Redis INFO       Z-Score          Health score    Queue Hub UI
  BullMQ metrics   EWMA             Forecast        Slack / email
  Job counts       Regression       SLO tracker     PagerDuty
                   Trend analysis   Reports

Each component runs as a scheduled job (or a BullMQ repeatable job itself) that feeds into the next stage. The end result is a system that doesn't just tell you when Redis is at 95% memory — it tells you when it will reach 95% and what to do about it.

Try It with Queue Hub

Ready to move beyond reactive monitoring? Queue Hub gives you the health scores, capacity forecasts, and anomaly detection covered in this post — alongside full queue management, job inspection, and live worker views. Try Queue Hub today and see what your Redis queues have been trying to tell you.

← All Articles
Share on XShare on LinkedIn

Related Articles

2026-07-12

Best BullMQ Dashboard Alternatives in 2026: A Comprehensive Comparison

Comparing every BullMQ UI option side by side: Bull Board, Arena, Taskforce, QueueHub, and raw redis-cli. Feature matrices, pricing, pros and cons, and recommendations for every team size.

2026-07-10

QueueHub vs pg-boss: Redis vs PostgreSQL as a Job Queue Backend

pg-boss uses PostgreSQL as a Node.js job queue, while BullMQ (monitored by QueueHub) uses Redis. We compare the two approaches across throughput, persistence, transactional queues, deployment, and when transactional enqueueing matters.

2026-07-09

QueueHub vs Temporal: Job Queues vs Workflow Orchestration

Temporal is a workflow orchestration platform — a different category from Redis-backed job queues. We compare Temporal.io against BullMQ + QueueHub, covering complexity, use cases, durability, observability, and when to choose each.