Beyond Green and Red — Building an Intelligent Queue Health Score System for BullMQ

Introduction — Beyond Red/Green: Why Your Queue Needs a Number

You run six queues. The first has 12,000 waiting jobs — but it's a batch analytics queue designed to hold millions. The second has just 3 waiting jobs — but every worker is stalled and nothing has processed in 22 minutes. The third shows a 2% failure rate — a passing grade until you realize those failures all happened in the last 90 seconds and the downstream API is on fire.

A red light / green light dashboard cannot express this nuance. Green tells you nothing useful when a silent degradation is already underway. Red tells you too late — the incident has already started. What you need is a single number that captures the full dimensional state of every queue and answers one question instantly: "On a scale of 0 to 100, how healthy is this queue right now, and is it getting better or worse?"

This is the third post in our Queue Monitoring Best Practices series. The first post covered the four-layer observability stack (events, metrics, dashboards, alerting). The second covered alerting playbooks, worker health checks, and Redis introspection. This post takes a different approach entirely: instead of collecting and reacting to individual metrics, we combine them into a composite health score — a single, interpretable number that distills multi-dimensional queue health into actionable insight.

You will learn:

• How to design a weighted composite health score across five dimensions
• Time-decayed scoring — making recent failures count more than old ones
• RESTful health score API endpoints for integration with PagerDuty, Opsgenie, and Slack
• Automatic tier escalation with hysteresis to prevent alert flapping
• Trend analysis using linear regression on score history
• Alert fatigue reduction by gating on the composite score instead of individual metrics
• How Queue Hub visualizes health scores in its dashboard

Section 1: Composite Health Score Design — The Anatomy of a 0–100 Score

A good health score has four properties: interpretable (72 means something concrete), sensitive to degradation (a rising backlog moves the needle immediately), stable against noise (a single transient failure doesn't tank the score), and decomposable (you can answer "why is this queue at 47?" by examining the sub-scores).

The Five Dimensions

We score every queue along five axes, each normalized to a 0–100 float, then combine them with configurable weights:

Dimension	Raw Metric	100 = Perfect	0 = Critical
Depth Saturation	waiting / maxExpectedDepth	Empty queue or below expected ceiling	Backlog exceeds 3× expected depth
Processing Latency Ratio	currentP50 / baselineP50	At or below baseline	>5× baseline
Failure Rate	failedLastWindow / totalLastWindow	0% failures	>15% failure rate
Worker Utilization	activeWorkers / expectedWorkers	Full expected worker count	Zero workers processing
Redis Memory Pressure	usedMemory / maxMemory	<60% used	>90% used

The Weighted Score Function

import { Queue, JobCounts } from 'bullmq';
import IORedis from 'ioredis';

export interface HealthDimensions {
  depthSaturation: number;
  processingLatency: number;
  failureRate: number;
  workerUtilization: number;
  redisMemoryPressure: number;
}

export interface HealthScoreConfig {
  weights: {
    depthSaturation: number;
    processingLatency: number;
    failureRate: number;
    workerUtilization: number;
    redisMemoryPressure: number;
  };
  maxExpectedDepth: number;
  expectedWorkers: number;
  latencyBaselineMs: number;
  maxMemoryBytes: number;
  failureWindowMs: number;
}

const DEFAULT_CONFIG: HealthScoreConfig = {
  weights: {
    depthSaturation: 0.30,
    processingLatency: 0.25,
    failureRate: 0.20,
    workerUtilization: 0.15,
    redisMemoryPressure: 0.10,
  },
  maxExpectedDepth: 10000,
  expectedWorkers: 3,
  latencyBaselineMs: 500,
  maxMemoryBytes: 4 * 1024 * 1024 * 1024,
  failureWindowMs: 5 * 60 * 1000,
};

function scoreDepthSaturation(
  jobCounts: JobCounts,
  config: HealthScoreConfig
): number {
  const ratio = jobCounts.waiting / config.maxExpectedDepth;
  if (ratio <= 1) return 100;
  if (ratio >= 3) return 0;
  return Math.round(100 - ((ratio - 1) / 2) * 100);
}

function scoreProcessingLatency(
  p50Ms: number,
  config: HealthScoreConfig
): number {
  const ratio = p50Ms / config.latencyBaselineMs;
  if (ratio <= 1) return 100;
  if (ratio >= 5) return 0;
  return Math.round(100 - ((ratio - 1) / 4) * 100);
}

function scoreFailureRate(
  failuresInWindow: number,
  totalInWindow: number
): number {
  if (totalInWindow === 0) return 100;
  const rate = failuresInWindow / totalInWindow;
  if (rate <= 0.01) return 100;
  if (rate >= 0.15) return 0;
  return Math.round(100 - ((rate - 0.01) / 0.14) * 100);
}

function scoreWorkerUtilization(
  activeWorkers: number,
  expectedWorkers: number
): number {
  const ratio = activeWorkers / expectedWorkers;
  if (ratio >= 1) return 100;
  if (ratio <= 0) return 0;
  return Math.round(ratio * 100);
}

function scoreRedisMemoryPressure(
  usedMemoryBytes: number,
  maxMemoryBytes: number
): number {
  const ratio = usedMemoryBytes / maxMemoryBytes;
  if (ratio <= 0.6) return 100;
  if (ratio >= 0.9) return 0;
  return Math.round(100 - ((ratio - 0.6) / 0.3) * 100);
}

export function computeHealthScore(
  dimensions: HealthDimensions,
  config: HealthScoreConfig = DEFAULT_CONFIG
): number {
  const { weights } = config;
  const raw =
    dimensions.depthSaturation * weights.depthSaturation +
    dimensions.processingLatency * weights.processingLatency +
    dimensions.failureRate * weights.failureRate +
    dimensions.workerUtilization * weights.workerUtilization +
    dimensions.redisMemoryPressure * weights.redisMemoryPressure;
  return Math.round(Math.max(0, Math.min(100, raw)));
}

export function computeDimensions(
  jobCounts: JobCounts,
  p50Ms: number,
  failuresInWindow: number,
  totalInWindow: number,
  activeWorkers: number,
  usedMemoryBytes: number,
  config: HealthScoreConfig = DEFAULT_CONFIG
): HealthDimensions {
  return {
    depthSaturation: scoreDepthSaturation(jobCounts, config),
    processingLatency: scoreProcessingLatency(p50Ms, config),
    failureRate: scoreFailureRate(failuresInWindow, totalInWindow),
    workerUtilization: scoreWorkerUtilization(
      activeWorkers,
      config.expectedWorkers
    ),
    redisMemoryPressure: scoreRedisMemoryPressure(
      usedMemoryBytes,
      config.maxMemoryBytes
    ),
  };
}

Each sub-score uses a piecewise linear function with three zones:

• Green zone (score = 100): The metric is comfortably within acceptable bounds
• Yellow zone (score = 100 → 0, linearly): The metric is degrading
• Red zone (score = 0): The metric is critically unhealthy

The weights reflect a judgment call: depth saturation and latency are the most operationally urgent signals, while Redis memory pressure — important but slower-moving — receives the lowest weight. You should tune these for your workload.

Section 2: Time-Decayed Scoring — Why the Last 60 Seconds Matter More Than Last Week

A queue that failed 100 jobs yesterday and zero today is not the same as a queue that failed 100 jobs in the last minute. Raw failure counts flatten this distinction. Time-decayed scoring solves it by applying an exponential decay factor to historical observations so that recent failures carry exponentially more weight.

Exponential Decay Function

Every failure is recorded with a timestamp. When computing the failure sub-score, we weight each observation by e^(-λ * age), where λ (lambda) controls the half-life — how quickly old failures fade.

export interface DecayConfig {
  halfLifeMs: number;
  now?: number;
}

const DEFAULT_DECAY: DecayConfig = {
  halfLifeMs: 120_000, // 2 minutes
};

function decayWeight(
  timestampMs: number,
  config: DecayConfig = DEFAULT_DECAY
): number {
  const currentTime = config.now ?? Date.now();
  const ageMs = currentTime - timestampMs;
  if (ageMs < 0) return 0;
  const lambda = Math.LN2 / config.halfLifeMs;
  return Math.exp(-lambda * ageMs);
}

export interface DecayedFailureCounts {
  decayedFailures: number;
  decayedTotal: number;
  effectiveSampleSize: number;
}

export function computeDecayedFailureRate(
  failureTimestamps: number[],
  totalTimestamps: number[],
  config: DecayConfig = DEFAULT_DECAY
): DecayedFailureCounts {
  const decayedFailures = failureTimestamps.reduce(
    (sum, ts) => sum + decayWeight(ts, config),
    0
  );
  const decayedTotal = totalTimestamps.reduce(
    (sum, ts) => sum + decayWeight(ts, config),
    0
  );
  return {
    decayedFailures,
    decayedTotal,
    effectiveSampleSize: decayedTotal,
  };
}

With a half-life of 2 minutes:

• A failure 1 minute ago gets weight ~0.71
• A failure 2 minutes ago gets weight ~0.50
• A failure 10 minutes ago gets weight ~0.03 (effectively zero)

This means a burst of failures that stops will normalize your health score within about 10 minutes, preventing alert fatigue from stale data while maintaining sensitivity to active problems.

Storing Timestamps for Decay

You don't need a time-series database. A Redis sorted set per queue stores each job outcome with its timestamp, trimmed periodically to keep only recent data:

const TIMESTAMP_KEY_PREFIX = 'qh:decay';

function decayKey(queueName: string): string {
  return `${TIMESTAMP_KEY_PREFIX}:${queueName}`;
}

export async function recordJobOutcome(
  redis: IORedis,
  queueName: string,
  jobId: string,
  outcome: 'completed' | 'failed',
  timestampMs: number = Date.now()
): Promise<void> {
  const key = decayKey(queueName);
  const member = `${outcome}:${jobId}`;
  await redis.zadd(key, timestampMs, member);
  await redis.zremrangebyscore(key, 0, timestampMs - 3_600_000);
}

export async function getDecayedFailureTimestamps(
  redis: IORedis,
  queueName: string,
  windowMs: number = 300_000
): Promise<{ failures: number[]; total: number[] }> {
  const key = decayKey(queueName);
  const cutoff = Date.now() - windowMs;
  const members = await redis.zrangebyscore(key, cutoff, '+inf');
  const failures: number[] = [];
  const total: number[] = [];
  for (const member of members) {
    const [outcome] = member.split(':');
    const score = await redis.zscore(key, member);
    if (score !== null) {
      total.push(Number(score));
      if (outcome === 'failed') {
        failures.push(Number(score));
      }
    }
  }
  return { failures, total };
}

Section 3: Health Score API Endpoints — Exposing Scores for Integration

The health score is most valuable when it's consumable by incident management tools (PagerDuty, Opsgenie), custom dashboards, CI/CD pipelines, and Slack bots. The natural interface is a REST API.

Express Router with Tier Classification

import express, { Request, Response, Router } from 'express';
import { Queue, JobCounts } from 'bullmq';
import IORedis from 'ioredis';

type HealthTier = 'green' | 'yellow' | 'orange' | 'red';

const TIER_THRESHOLDS = { green: 80, yellow: 60, orange: 40, red: 0 };

interface QueueHealthResponse {
  queueName: string;
  score: number;
  tier: HealthTier;
  dimensions: HealthDimensions;
  config: Partial<HealthScoreConfig>;
  timestamp: number;
}

interface AllHealthResponse {
  overall: number;
  queues: QueueHealthResponse[];
  timestamp: number;
}

export function classifyTier(score: number): HealthTier {
  if (score >= TIER_THRESHOLDS.green) return 'green';
  if (score >= TIER_THRESHOLDS.yellow) return 'yellow';
  if (score >= TIER_THRESHOLDS.orange) return 'orange';
  return 'red';
}

export function createHealthRouter(
  redis: IORedis,
  queues: Map<string, Queue>,
  config: HealthScoreConfig = DEFAULT_CONFIG
): Router {
  const router = Router();

  async function evaluateSingleQueue(
    queueName: string,
    queue: Queue
  ): Promise<QueueHealthResponse> {
    const jobCounts = await queue.getJobCounts(
      'waiting', 'active', 'completed', 'failed', 'delayed'
    );
    const metrics = await queue.getMetrics('completed');
    const p50Ms = metrics?.data?.[0] ?? config.latencyBaselineMs;
    const workers = await queue.getWorkers();
    const activeWorkers = workers.filter((w) => !w.stalled).length;
    const info = await redis.info('memory');
    const usedBytes = parseInt(
      info.match(/used_memory:(\d+)/)?.[1] ?? '0', 10
    );
    const { failures, total } = await getDecayedFailureTimestamps(
      redis, queueName, config.failureWindowMs
    );
    const dimensions: HealthDimensions = {
      depthSaturation: scoreDepthSaturation(jobCounts, config),
      processingLatency: scoreProcessingLatency(p50Ms, config),
      failureRate: scoreFailureRate(failures.length, total.length),
      workerUtilization: scoreWorkerUtilization(
        activeWorkers, config.expectedWorkers
      ),
      redisMemoryPressure: scoreRedisMemoryPressure(
        usedBytes, config.maxMemoryBytes
      ),
    };
    const score = computeHealthScore(dimensions, config);
    return {
      queueName, score, tier: classifyTier(score),
      dimensions,
      config: { maxExpectedDepth: config.maxExpectedDepth, expectedWorkers: config.expectedWorkers },
      timestamp: Date.now(),
    };
  }

  router.get('/health/queues', async (_req: Request, res: Response) => {
    try {
      const entries = Array.from(queues.entries());
      const results = await Promise.all(
        entries.map(([name, q]) => evaluateSingleQueue(name, q))
      );
      const overall = Math.round(
        results.reduce((sum, r) => sum + r.score, 0) / results.length
      );
      const payload: AllHealthResponse = { overall, queues: results, timestamp: Date.now() };
      res.json(payload);
    } catch (error) {
      const message = error instanceof Error ? error.message : 'Unknown error';
      res.status(500).json({ error: message });
    }
  });

  return router;
}

Example API Response

{
  "queueName": "email-notifications",
  "score": 64,
  "tier": "yellow",
  "dimensions": {
    "depthSaturation": 100,
    "processingLatency": 82,
    "failureRate": 12,
    "workerUtilization": 67,
    "redisMemoryPressure": 85
  },
  "timestamp": 1788356412345
}

An operator reading this knows instantly: the overall score is 64 (yellow tier), and the breakdown points to the failure rate sub-score at 12 — that's the dimension to investigate.

Bootstrapping the Express Server

async function main(): Promise<void> {
  const redis = new IORedis({
    host: process.env.REDIS_HOST ?? 'localhost',
    port: Number(process.env.REDIS_PORT ?? 6379),
    maxRetriesPerRequest: null,
  });

  const queueNames = (process.env.MONITORED_QUEUES ?? 'email,reports,payments')
    .split(',').map((s) => s.trim());

  const queues = new Map<string, Queue>();
  for (const name of queueNames) {
    const queue = new Queue(name, { connection: redis });
    queues.set(name, queue);
  }

  const app = express();
  const healthRouter = createHealthRouter(redis, queues);
  app.use('/api/v1', healthRouter);

  const port = Number(process.env.HEALTH_PORT ?? 9730);
  app.listen(port, () => {
    console.log(`Queue health API listening on port ${port}`);
  });
}

main().catch((err) => {
  console.error('Failed to start health API:', err);
  process.exit(1);
});

PagerDuty / Opsgenie Webhook Integration

For incident management tools, expose a webhook payload that includes the score, tier, and dimension breakdown:

export interface HealthWebhookPayload {
  source: string;
  queue: string;
  currentScore: number;
  previousScore: number;
  tier: HealthTier;
  previousTier: HealthTier;
  dimensions: HealthDimensions;
  timestamp: number;
}

export async function sendHealthWebhook(
  webhookUrl: string,
  payload: HealthWebhookPayload,
  signal?: AbortSignal
): Promise<Response> {
  const body = {
    event_type: 'queue_health_change',
    payload: {
      summary: `Queue "${payload.queue}" changed to tier ${payload.tier} (score: ${payload.currentScore})`,
      severity:
        payload.tier === 'red' ? 'critical'
        : payload.tier === 'orange' ? 'warning'
        : payload.tier === 'yellow' ? 'info'
        : 'ok',
      source: payload.source,
      custom_details: payload,
    },
  };
  return fetch(webhookUrl, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify(body),
    signal,
  });
}

Section 4: Automatic Tier Escalation — Green to Red with Hysteresis

A static score is useful. A score with tier transitions — gated by duration and hysteresis — is operationally powerful. The idea: a queue does not escalate from green to red instantly. It must spend sustained time below each threshold, and the thresholds have hysteresis to prevent flapping.

Tier Transition Engine

export interface TierState {
  queueName: string;
  currentTier: HealthTier;
  currentScore: number;
  enteredCurrentTierAt: number;
  lastTransitionAt: number;
  scoreHistory: number[];
}

export interface TierConfig {
  thresholds: Record<HealthTier, number>;
  escalationDelayMs: number;
  deescalationDelayMs: number;
  hysteresisGap: number;
}

const DEFAULT_TIER_CONFIG: TierConfig = {
  thresholds: { green: 80, yellow: 60, orange: 40, red: 0 },
  escalationDelayMs: 30_000,
  deescalationDelayMs: 60_000,
  hysteresisGap: 5,
};

const TIER_ORDER: HealthTier[] = ['green', 'yellow', 'orange', 'red'];

function tierIndex(tier: HealthTier): number {
  return TIER_ORDER.indexOf(tier);
}

export class TierTransitionEngine {
  private states = new Map<string, TierState>();

  constructor(private config: TierConfig = DEFAULT_TIER_CONFIG) {}

  evaluate(
    queueName: string,
    score: number,
    now: number = Date.now()
  ): {
    state: TierState;
    transitioned: boolean;
    previousTier: HealthTier | null;
  } {
    const threshold = this.config.thresholds;
    const gap = this.config.hysteresisGap;

    let rawTier: HealthTier;
    if (score >= threshold.green + gap) {
      rawTier = 'green';
    } else if (score >= threshold.yellow + gap / 2) {
      rawTier = 'yellow';
    } else if (score >= threshold.orange + gap / 2) {
      rawTier = 'orange';
    } else {
      rawTier = 'red';
    }

    let existing = this.states.get(queueName);
    if (!existing) {
      const initial: TierState = {
        queueName, currentTier: rawTier, currentScore: score,
        enteredCurrentTierAt: now, lastTransitionAt: now, scoreHistory: [score],
      };
      this.states.set(queueName, initial);
      return { state: initial, transitioned: true, previousTier: null };
    }

    existing.scoreHistory.push(score);
    if (existing.scoreHistory.length > 100) existing.scoreHistory.shift();
    existing.currentScore = score;

    const currentIdx = tierIndex(existing.currentTier);
    const rawIdx = tierIndex(rawTier);
    // Escalation: dropping multiple tiers at once
    // De-escalation: rising one or more tiers
    const shouldTransition = rawIdx < currentIdx || rawIdx > currentIdx + 1;

    if (shouldTransition) {
      const delay = rawIdx < currentIdx
        ? this.config.escalationDelayMs
        : this.config.deescalationDelayMs;

      if (now - existing.enteredCurrentTierAt >= delay) {
        const previousTier = existing.currentTier;
        existing.currentTier = rawTier;
        existing.enteredCurrentTierAt = now;
        existing.lastTransitionAt = now;
        return { state: existing, transitioned: true, previousTier };
      }
    }

    return { state: existing, transitioned: false, previousTier: null };
  }
}

Why hysteresis? Consider a queue bouncing between score 79 and 81. Without hysteresis, every poll cycle triggers a green → yellow → green transition. With a hysteresis gap of 5:

• To escalate from green → yellow: score must drop below 80 - 5 = 75
• To de-escalate from yellow → green: score must rise above 80 + 5 = 85

This creates a deadband that prevents flapping. Combined with the escalation delay (30 seconds sustained below threshold), transient blips never trigger tier changes.

Section 5: Trend Analysis — Detecting Degradation Before It Crosses Thresholds

A health score of 72 tells you the current state. But a score that dropped from 88 to 72 over 30 minutes tells you something more important: the queue is degrading. Trend analysis detects these trajectories using linear regression.

Linear Regression on Score History

export interface TrendResult {
  slope: number;
  intercept: number;
  currentScore: number;
  projectedScore30min: number;
  projectedScore60min: number;
  isDegrading: boolean;
  isImproving: boolean;
}

function linearRegression(
  xValues: number[],
  yValues: number[]
): { slope: number; intercept: number; rSquared: number } {
  const n = xValues.length;
  if (n < 2) return { slope: 0, intercept: yValues[0] ?? 0, rSquared: 0 };

  const sumX = xValues.reduce((a, b) => a + b, 0);
  const sumY = yValues.reduce((a, b) => a + b, 0);
  const meanX = sumX / n;
  const meanY = sumY / n;

  let numerator = 0;
  let denomX = 0;
  let denomY = 0;

  for (let i = 0; i < n; i++) {
    const dx = xValues[i] - meanX;
    const dy = yValues[i] - meanY;
    numerator += dx * dy;
    denomX += dx * dx;
    denomY += dy * dy;
  }

  const slope = denomX !== 0 ? numerator / denomX : 0;
  const intercept = meanY - slope * meanX;
  const rSquared = denomX !== 0 && denomY !== 0
    ? (numerator * numerator) / (denomX * denomY)
    : 0;

  return { slope, intercept, rSquared };
}

export function analyzeTrend(
  scoreHistory: number[],
  tickIntervalMs: number = 15_000
): TrendResult {
  const xValues = scoreHistory.map(
    (_, i) => (i - scoreHistory.length + 1) * tickIntervalMs
  );
  const { slope, intercept } = linearRegression(xValues, scoreHistory);
  const currentScore = scoreHistory[scoreHistory.length - 1];
  const projection30min = intercept + slope * (
    xValues.length * tickIntervalMs + 1_800_000
  );
  const projection60min = intercept + slope * (
    xValues.length * tickIntervalMs + 3_600_000
  );

  return {
    slope, intercept, currentScore,
    projectedScore30min: Math.round(Math.max(0, Math.min(100, projection30min))),
    projectedScore60min: Math.round(Math.max(0, Math.min(100, projection60min))),
    isDegrading: slope < -0.001,
    isImproving: slope > 0.001,
  };
}

Trend-Aware Alerting Decision Matrix

Current Score	Trend Slope	Action
≥ 80	Flat or improving	No action
≥ 80	Degrading (slope < -0.01)	Log observation, no alert yet
60–79	Degrading	Yellow: investigate — send warning to Slack
40–59	Degrading sharply	Orange: escalate — page on-call engineer
< 40	Any	Red: critical — page immediately
Any	Strongly improving	De-escalate previous alert

export interface TrendAlert {
  queueName: string;
  score: number;
  tier: HealthTier;
  trend: TrendResult;
  action: 'none' | 'log' | 'warn' | 'escalate' | 'critical';
  message: string;
}

export function evaluateTrendAlert(
  queueName: string,
  scoreHistory: number[],
  tickIntervalMs: number = 15_000
): TrendAlert {
  const trend = analyzeTrend(scoreHistory, tickIntervalMs);
  const currentScore = trend.currentScore;
  const tier = classifyTier(currentScore);

  let action: TrendAlert['action'] = 'none';
  let message = '';

  if (currentScore >= 80) {
    if (trend.isDegrading && trend.slope < -0.01) {
      action = 'log';
      message = `Queue "${queueName}" is still green (${currentScore}) but degrading at ${trend.slope.toFixed(4)} pts/ms. Projected 60min: ${trend.projectedScore60min}.`;
    }
  } else if (currentScore >= 60) {
    if (trend.isDegrading) {
      action = 'warn';
      message = `Queue "${queueName}" is yellow (${currentScore}) and degrading. Projected 30min: ${trend.projectedScore30min}. Investigate soon.`;
    }
  } else if (currentScore >= 40) {
    action = trend.slope < -0.005 ? 'escalate' : 'warn';
    message = trend.slope < -0.005
      ? `Queue "${queueName}" is orange (${currentScore}) and degrading sharply. Paging on-call.`
      : `Queue "${queueName}" is orange (${currentScore}). Needs investigation.`;
  } else {
    action = 'critical';
    message = `Queue "${queueName}" is critical (${currentScore}). Immediate intervention required.`;
  }

  return { queueName, score: currentScore, tier, trend, action, message };
}

Section 6: Alert Fatigue Reduction — The Composite Score as a Gate

The single biggest operational win of a composite health score is alert fatigue reduction. Without it, a brief Redis resize operation that spikes latency, stalls a worker reconnect, and causes a 3-second depth spike could fire three separate alerts for one event. Operators learn to ignore these, and real incidents get buried.

The Gating Pattern

With a composite score, you invert the logic: every raw metric is collected and stored, but only the composite score triggers tier transitions and alerts. Individual metrics are surfaced in the breakdown for root cause analysis, not for alerting.

export interface GatedAlertConfig {
  minScoreForAlert: number;
  minDegradationRate: number;
  cooldownMs: number;
  maxAlertsPerHour: number;
}

const DEFAULT_GATE_CONFIG: GatedAlertConfig = {
  minScoreForAlert: 60,
  minDegradationRate: -0.002,
  cooldownMs: 300_000,
  maxAlertsPerHour: 6,
};

export class GatedAlertManager {
  private alertTimestamps: string[] = [];
  private lastAlertByQueue = new Map<string, number>();

  constructor(private config: GatedAlertConfig = DEFAULT_GATE_CONFIG) {}

  shouldFire(
    queueName: string,
    score: number,
    trend: TrendResult,
    now: number = Date.now()
  ): { fire: boolean; reason: string } {
    if (score > this.config.minScoreForAlert &&
        trend.slope > this.config.minDegradationRate) {
      return {
        fire: false,
        reason: `Score ${score} above threshold and degradation rate within bounds.`,
      };
    }

    const lastFire = this.lastAlertByQueue.get(queueName) ?? 0;
    if (now - lastFire < this.config.cooldownMs) {
      return { fire: false, reason: `In cooldown (${now - lastFire}ms since last alert).` };
    }

    const lastHour = now - 3_600_000;
    const recentAlerts = this.alertTimestamps.filter(
      (ts) => Number(ts) > lastHour
    ).length;
    if (recentAlerts >= this.config.maxAlertsPerHour) {
      return { fire: false, reason: `Rate limited: ${recentAlerts} alerts/hour.` };
    }

    this.alertTimestamps.push(String(now));
    this.lastAlertByQueue.set(queueName, now);
    return { fire: true, reason: `Score ${score} below ${this.config.minScoreForAlert}.` };
  }
}

What This Prevents

Without the gated alert manager, a 30-second burst of 15 failed jobs triggers your "high failure rate" alert. With it:

1. The failure rate sub-score drops from 100 to 60
2. The composite score drops from 92 to 82 — still green
3. No alert fires
4. 90 seconds later, failures stop; the decay function restores the sub-score
5. The composite score returns to 90+ without anyone being paged

The operator never gets paged for transient problems. Only sustained degradation that pushes the composite score below the gate threshold generates an alert.

Slack Notification Format

When an alert does fire, the notification should contain the score, tier, trend, and dimension breakdown:

export function formatSlackMessage(alert: TrendAlert): Record<string, unknown> {
  const emoji = alert.action === 'critical' ? ':red_circle:'
    : alert.action === 'escalate' ? ':orange_circle:'
    : alert.action === 'warn' ? ':large_yellow_circle:'
    : ':information_source:';

  return {
    text: `${emoji} Queue Health Alert: ${alert.queueName}`,
    attachments: [{
      color: alert.action === 'critical' ? 'danger'
        : alert.action === 'escalate' ? 'warning' : 'good',
      fields: [
        { title: 'Queue', value: alert.queueName, short: true },
        { title: 'Score', value: String(alert.score), short: true },
        { title: 'Tier', value: alert.tier.toUpperCase(), short: true },
        { title: 'Trend', value: alert.trend.isDegrading ? ':arrow_down:' : ':arrow_up:', short: true },
        { title: '30m Projection', value: String(alert.trend.projectedScore30min), short: true },
        { title: '60m Projection', value: String(alert.trend.projectedScore60min), short: true },
        { title: 'Message', value: alert.message, short: false },
      ],
      ts: Math.floor(Date.now() / 1000),
    }],
  };
}

Section 7: Queue Hub Integration — Visualizing the Health Score

The health score system is most valuable when it's visible at a glance. Queue Hub integrates it directly into the overview dashboard and per-queue views.

Dashboard Health Score Card

On the Queue Hub overview page, each queue card displays:

• A circular gauge (0–100) color-coded by tier: green (≥80), yellow (60–79), orange (40–59), red (<40)
• A sparkline of the last 100 score data points with a trend arrow indicator
• The five dimension sub-scores as small bar indicators beneath the gauge
• A tier badge — GREEN, YELLOW, ORANGE, or RED — with estimated time remaining in current tier

Per-Queue Detail View

Clicking into a queue reveals:

• Score history chart — time-series line chart over the last hour, day, or week
• Dimension breakdown — five overlaid lines showing how each dimension contributed over time
• Tier timeline — horizontal Gantt-style visualization of tier transitions
• Trend projection — dashed extension showing projected score 30 and 60 minutes ahead
• Top contributing factors — natural language summary: "The score dropped 18 points in the last 15 minutes, driven by failure rate (12→4) and depth saturation (100→67)."

WebSocket-Powered Live Updates

Queue Hub uses WebSocket push from the health score daemon so the dashboard updates every 15 seconds without polling:

import { WebSocketServer, WebSocket } from 'ws';

export function broadcastHealthScores(
  wss: WebSocketServer,
  scores: AllHealthResponse
): void {
  const payload = JSON.stringify({
    type: 'health_update',
    data: scores,
  });
  wss.clients.forEach((client: WebSocket) => {
    if (client.readyState === WebSocket.OPEN) {
      client.send(payload);
    }
  });
}

Multi-Backend Health Aggregation

For Queue Hub users managing multiple Redis backends (local, TLS, Valkey, AWS ElastiCache), the health score is computed per-backend with a cross-backend rollup. The overall platform health score is the weighted average of all tracked queues across all connected backends, making it possible to answer "is my entire queue infrastructure healthy?" from a single number.

Putting It All Together: The Health Score Daemon

Here is the complete daemon that ties everything together — collecting metrics, computing scores, tracking trends, managing tier transitions, and gating alerts:

import express from 'express';
import { Queue } from 'bullmq';
import IORedis from 'ioredis';
import { WebSocketServer } from 'ws';

const redis = new IORedis({
  host: process.env.REDIS_HOST ?? 'localhost',
  port: Number(process.env.REDIS_PORT ?? 6379),
  maxRetriesPerRequest: null,
});

const queueNames = (process.env.MONITORED_QUEUES ?? 'email,reports,payments')
  .split(',').map((s) => s.trim());

const queues = new Map<string, Queue>();
for (const name of queueNames) {
  queues.set(name, new Queue(name, { connection: redis }));
}

const healthConfig: HealthScoreConfig = {
  weights: {
    depthSaturation: 0.30, processingLatency: 0.25,
    failureRate: 0.20, workerUtilization: 0.15,
    redisMemoryPressure: 0.10,
  },
  maxExpectedDepth: Number(process.env.MAX_EXPECTED_DEPTH ?? 10000),
  expectedWorkers: Number(process.env.EXPECTED_WORKERS ?? 3),
  latencyBaselineMs: Number(process.env.LATENCY_BASELINE_MS ?? 500),
  maxMemoryBytes: Number(process.env.MAX_MEMORY_BYTES ?? 4 * 1024 * 1024 * 1024),
  failureWindowMs: 300_000,
};

const tierConfig: TierConfig = {
  thresholds: { green: 80, yellow: 60, orange: 40, red: 0 },
  escalationDelayMs: 30_000,
  deescalationDelayMs: 60_000,
  hysteresisGap: 5,
};

const app = express();
const healthRouter = createHealthRouter(redis, queues, healthConfig);
app.use('/api/v1', healthRouter);

const server = app.listen(Number(process.env.HEALTH_PORT ?? 9730));
const wss = new WebSocketServer({ server });

const engine = new TierTransitionEngine(tierConfig);
const gateManager = new GatedAlertManager();
const scoreHistoryMap = new Map<string, number[]>();

setInterval(async () => {
  for (const [name, queue] of queues.entries()) {
    try {
      const jobCounts = await queue.getJobCounts(
        'waiting', 'active', 'completed', 'failed', 'delayed'
      );
      const metrics = await queue.getMetrics('completed');
      const p50Ms = metrics?.data?.[0] ?? healthConfig.latencyBaselineMs;
      const workers = await queue.getWorkers();
      const activeWorkers = workers.filter((w) => !w.stalled).length;
      const info = await redis.info('memory');
      const usedBytes = parseInt(
        info.match(/used_memory:(\d+)/)?.[1] ?? '0', 10
      );
      const { failures, total } = await getDecayedFailureTimestamps(
        redis, name, healthConfig.failureWindowMs
      );

      const dimensions = computeDimensions(
        jobCounts, p50Ms, failures.length, total.length,
        activeWorkers, usedBytes, healthConfig
      );
      const score = computeHealthScore(dimensions, healthConfig);

      let history = scoreHistoryMap.get(name) ?? [];
      history.push(score);
      if (history.length > 100) history = history.slice(-100);
      scoreHistoryMap.set(name, history);

      const { transitioned, previousTier } = engine.evaluate(name, score);
      if (transitioned && previousTier !== null) {
        console.log(
          `[TIER] ${name}: ${previousTier} -> ${classifyTier(score)} (score: ${score})`
        );
      }

      const trend = analyzeTrend(history);
      const alert = evaluateTrendAlert(name, history);
      const { fire } = gateManager.shouldFire(name, score, trend);

      if (fire) {
        console.log(`[ALERT] ${alert.action.toUpperCase()}: ${alert.message}`);
      }
    } catch (error) {
      console.error(`[ERROR] Health loop for "${name}":`, error);
    }
  }

  const allResults: AllHealthResponse = {
    overall: Math.round(
      Array.from(scoreHistoryMap.entries())
        .map(([_, hist]) => hist[hist.length - 1] ?? 0)
        .reduce((a, b) => a + b, 0) / Math.max(1, scoreHistoryMap.size)
    ),
    queues: [],
    timestamp: Date.now(),
  };

  broadcastHealthScores(wss, allResults);
}, 15_000);

console.log(
  'Health score daemon running on port',
  Number(process.env.HEALTH_PORT ?? 9730)
);

Conclusion — From Raw Metrics to a Single Number You Can Trust

A composite health score transforms queue observability from a fire hose of individual metrics into a single, interpretable signal. The five-dimensional weighted score gives you nuance. Time decay gives you recency. Trend analysis gives you foresight. Tier transitions with hysteresis give you stability. And the gated alert manager gives you peace of mind — no more 3 AM pages for transient blips.

The code in this post is ready to deploy. Wire it into your Express server, point a WebSocket at Queue Hub, configure your PagerDuty webhook, and you will know — in one number — exactly how healthy every queue in your system is, right now and for the next hour.

Ready to put an end to alert fatigue and start monitoring with a single, reliable health score? Try Queue Hub today — it comes with built-in health score visualization, WebSocket-powered live updates, and multi-backend aggregation across all your BullMQ and BeeQueue deployments.