Beyond Static Priority — Building a Dynamic Priority Escalation Engine with BullMQ
You set priority: 1 on your onboarding emails and priority: 100 on nightly analytics reports. Everything works — for a week. Then a surge of customer-facing jobs hits the queue, your high-priority onboarding jobs are now competing with other high-priority jobs, and the ones added first are still stuck behind newer ones of equal priority.
This is the fundamental limitation of static priority: it's a snapshot of importance at insertion time. It cannot account for:
- Jobs that grow more urgent the longer they wait
- Shifting business conditions (weekend vs. weekday, post-deploy vs. steady state)
- Customer tier that wasn't known at enqueue time
- Queue depth that changes dynamically
The solution? Make priority dynamic — automatically escalating waiting jobs based on real-time configurable rules. In this post, we'll build three production-ready patterns with complete TypeScript code:
- Dynamic Priority Escalation Engine — a dedicated worker that periodically re-evaluates waiting jobs and escalates their priority based on configurable rules
- Multi-Factor Priority Scoring — a scoring function that combines job criticality, customer tier, wait time, and queue depth
- Priority-Weighted Worker Pool Routing — separate "fast lane" and "bulk" worker pools for isolated processing
Pattern 1: Dynamic Priority Escalation Engine
The core idea is simple: a dedicated worker periodically inspects waiting jobs and calls changePriority() on those that have been waiting too long. The rules are configurable per job type.
Configuration Schema
// config/escalation-rules.ts
export interface EscalationRule {
name: string;
jobNamePattern: string;
predicate?: (jobData: Record<string, unknown>) => boolean;
schedule: Array<{
waitThresholdMs: number;
escalateToPriority: number;
}>;
maxPriority: number;
allowDowngrade: boolean;
}
export interface EscalationConfig {
checkIntervalMs: number;
batchSize: number;
urgentPriorityThreshold: number;
rules: EscalationRule[];
}
A sample configuration demonstrates the pattern:
export const defaultEscalationConfig: EscalationConfig = {
checkIntervalMs: 15_000,
batchSize: 500,
urgentPriorityThreshold: 5,
rules: [
{
name: 'Stale critical onboarding',
jobNamePattern: 'onboarding-email',
schedule: [
{ waitThresholdMs: 60_000, escalateToPriority: 10 },
{ waitThresholdMs: 300_000, escalateToPriority: 5 },
{ waitThresholdMs: 900_000, escalateToPriority: 1 },
],
maxPriority: 1,
allowDowngrade: false,
},
{
name: 'Tier-1 customer jobs',
jobNamePattern: '*',
predicate: (data) => (data as any).customerTier === 'tier-1',
schedule: [
{ waitThresholdMs: 30_000, escalateToPriority: 3 },
],
maxPriority: 3,
allowDowngrade: false,
},
{
name: 'Background report jobs',
jobNamePattern: 'generate-report',
schedule: [
{ waitThresholdMs: 600_000, escalateToPriority: 50 },
],
maxPriority: 50,
allowDowngrade: false,
},
],
};
The Escalation Worker
The escalation worker runs on a dedicated internal queue, using BullMQ's own scheduling to manage its lifecycle. Each "tick" is a job that evaluates waiting jobs in the target queue.
// workers/escalation-worker.ts
import { Queue, Job, Worker } from 'bullmq';
import { EscalationConfig, EscalationRule } from '../config/escalation-rules';
interface EscalationResult {
evaluated: number;
escalated: number;
skipped: number;
errors: number;
details: Array<{
jobId: string;
name: string;
oldPriority: number;
newPriority: number;
ruleName: string;
}>;
}
export class PriorityEscalationWorker {
private readonly queue: Queue;
private readonly config: EscalationConfig;
private worker: Worker | null = null;
private running = false;
private cycleCount = 0;
constructor(queue: Queue, config: EscalationConfig) {
this.queue = queue;
this.config = config;
}
async start(): Promise<void> {
if (this.running) return;
this.running = true;
const internalQueue = new Queue(
`__escalation:${this.queue.name}`,
{ connection: this.queue.opts.connection },
);
await internalQueue.add(
'escalation-tick',
{ cycle: 0 },
{ removeOnComplete: true, removeOnFail: false },
);
this.worker = new Worker(
internalQueue.name,
async () => {
const result = await this.runEscalationCycle();
this.cycleCount++;
await internalQueue.add(
'escalation-tick',
{ cycle: this.cycleCount },
{
delay: this.config.checkIntervalMs,
removeOnComplete: true,
removeOnFail: false,
},
);
return result;
},
{
connection: this.queue.opts.connection,
concurrency: 1,
},
);
this.worker.on('completed', (job, result: EscalationResult) => {
console.log(
`[Escalation] ${result.evaluated} evaluated, ` +
`${result.escalated} escalated, ${result.errors} errors`,
);
});
this.worker.on('failed', (job, err) => {
console.error(`[Escalation] Cycle failed: ${err.message}`);
});
}
async stop(): Promise<void> {
this.running = false;
await this.worker?.close();
}
private async runEscalationCycle(): Promise<EscalationResult> {
const result: EscalationResult = {
evaluated: 0,
escalated: 0,
skipped: 0,
errors: 0,
details: [],
};
try {
const [prioritizedJobs, waitingJobs] = await Promise.all([
this.queue.getJobs(['prioritized'], 0, this.config.batchSize),
this.queue.getJobs(['waiting'], 0, this.config.batchSize),
]);
const jobsToEvaluate = [...prioritizedJobs, ...waitingJobs];
result.evaluated = jobsToEvaluate.length;
for (const job of jobsToEvaluate) {
if (!job) continue;
try {
const now = Date.now();
const waitTime = job.timestamp ? now - job.timestamp : 0;
const matched = this.findMatchingRule(job, waitTime);
if (!matched) {
result.skipped++;
continue;
}
const targetPriority = this.computeTargetPriority(job, matched);
if (targetPriority !== null && targetPriority !== job.opts.priority) {
await job.changePriority({ priority: targetPriority });
result.escalated++;
result.details.push({
jobId: job.id!,
name: job.name,
oldPriority: job.opts.priority ?? 0,
newPriority: targetPriority,
ruleName: matched.rule.name,
});
} else {
result.skipped++;
}
} catch (err) {
result.errors++;
console.error(`[Escalation] Error processing job ${job.id}: ${err}`);
}
}
} catch (err) {
console.error(`[Escalation] Cycle error: ${err}`);
throw err;
}
return result;
}
private findMatchingRule(
job: Job,
waitTime: number,
): { rule: EscalationRule; stage: EscalationRule['schedule'][number] } | null {
for (const rule of this.config.rules) {
if (!this.matchesGlob(job.name, rule.jobNamePattern)) continue;
if (rule.predicate && !rule.predicate(job.data)) continue;
const applicableStages = rule.schedule
.filter((s) => waitTime >= s.waitThresholdMs)
.sort((a, b) => b.escalateToPriority - a.escalateToPriority);
if (applicableStages.length === 0) continue;
const mostAggressive = applicableStages.reduce((best, curr) =>
curr.escalateToPriority < best.escalateToPriority ? curr : best,
);
return { rule, stage: mostAggressive };
}
return null;
}
private computeTargetPriority(
job: Job,
matched: { rule: EscalationRule; stage: EscalationRule['schedule'][number] },
): number | null {
const currentPriority = job.opts.priority ?? 0;
const candidate = matched.stage.escalateToPriority;
const clamped = Math.max(candidate, matched.rule.maxPriority);
if (!matched.rule.allowDowngrade && clamped > currentPriority) {
return null;
}
if (clamped === currentPriority) return null;
return clamped;
}
private matchesGlob(value: string, pattern: string): boolean {
if (pattern === '*') return true;
const regexStr = pattern
.replace(/[.+^${}()|[\]\\]/g, '\\$&')
.replace(/\*/g, '.*')
.replace(/\?/g, '.');
return new RegExp(`^${regexStr}$`).test(value);
}
}
Usage
// main.ts
import { Queue } from 'bullmq';
import { PriorityEscalationWorker } from './workers/escalation-worker';
import { defaultEscalationConfig } from './config/escalation-rules';
const connection = { host: 'localhost', port: 6379 };
const mainQueue = new Queue('my-app-queue', {
connection,
defaultJobOptions: { priority: 100 },
});
const escalationWorker = new PriorityEscalationWorker(
mainQueue,
defaultEscalationConfig,
);
await escalationWorker.start();
process.on('SIGTERM', async () => {
await escalationWorker.stop();
await mainQueue.close();
process.exit(0);
});
Anti-Patterns and Pitfalls
| Pitfall | Consequence | Solution |
|---|---|---|
| Escalating too frequently | Redis hammering, O(n) sorted set updates | Use checkIntervalMs >= 15_000 and configurable batchSize |
| Escalating every job to priority 1 | Priority inversion — all jobs become equal | Enforce maxPriority per rule and use hysteresis |
| Escalation worker competing with processors | Thundering herd on same Redis keys | Set concurrency: 1 on the escalation worker |
Escalating jobs in active state |
changePriority() throws an error |
Only fetch waiting and prioritized jobs |
| Escalating already-escalated jobs | Wasted Redis calls | Set allowDowngrade: false and check current priority |
Pattern 2: Multi-Factor Priority Scoring
Sometimes a simple threshold schedule isn't enough. You need a priority score that combines multiple signals — job type, customer tier, wait time, and queue depth — into a single numeric priority.
The Scoring Function
// scoring/multi-factor-priority.ts
export interface PriorityFactors {
criticality: number;
tierMultiplier: number;
waitSeconds: number;
queueDepth: number;
maxPriority: number;
}
export function computeMultiFactorPriority(factors: PriorityFactors): number {
const {
criticality,
tierMultiplier,
waitSeconds,
queueDepth,
maxPriority = 2_097_152,
} = factors;
const AGE_FACTOR = 0.5;
const MAX_AGE_PENALTY = 40;
const DEPTH_FACTOR = 0.1;
const baseUrgency = Math.min(criticality * tierMultiplier, 100);
const agePenalty = Math.min(AGE_FACTOR * waitSeconds, MAX_AGE_PENALTY);
const urgency = Math.min(baseUrgency + agePenalty, 100);
const depthBonus = Math.min(queueDepth * DEPTH_FACTOR, 50);
const rawScore = maxPriority - Math.round(urgency * (maxPriority / 100)) +
Math.round(depthBonus * (maxPriority / 100));
return Math.max(1, Math.min(rawScore, maxPriority));
}
The formula works like this:
- Base urgency =
criticality × tierMultiplier— a password-reset email (criticality 85, tier-1 multiplier 1.5) starts at urgency 100 - Age penalty grows at 0.5 per second, capped at 40 — jobs gain urgency the longer they wait
- Depth bonus penalizes non-critical jobs in busy queues — when 500 jobs are waiting, a background report gets pushed further down
- Inversion: BullMQ uses lower numbers = higher priority, so we subtract urgency from
maxPriority
Scoring Config
// scoring/scoring-config.ts
export interface JobTypeProfile {
pattern: string;
criticality: number;
}
export interface ScoringConfig {
jobTypes: JobTypeProfile[];
tierMultiplier: Record<string, number>;
defaultCriticality: number;
defaultTierMultiplier: number;
rescoreIntervalMs: number;
}
export const defaultScoringConfig: ScoringConfig = {
jobTypes: [
{ pattern: 'onboarding-email', criticality: 90 },
{ pattern: 'password-reset', criticality: 85 },
{ pattern: 'invoice-generation', criticality: 60 },
{ pattern: 'generate-report', criticality: 20 },
{ pattern: 'data-export', criticality: 15 },
{ pattern: 'cleanup-task', criticality: 5 },
],
tierMultiplier: {
'tier-1': 1.5,
'tier-2': 1.2,
'tier-3': 1.0,
'tier-4': 0.7,
},
defaultCriticality: 30,
defaultTierMultiplier: 1.0,
rescoreIntervalMs: 30_000,
};
Scoring Worker
// workers/scoring-worker.ts
import { Queue, Job, Worker } from 'bullmq';
import { computeMultiFactorPriority } from '../scoring/multi-factor-priority';
import { ScoringConfig } from '../scoring/scoring-config';
interface ScoringResult {
evaluated: number;
updated: number;
errors: number;
priorityDistribution: Record<number, number>;
}
export class PriorityScoringWorker {
private readonly queue: Queue;
private readonly config: ScoringConfig;
private worker: Worker | null = null;
constructor(queue: Queue, config: ScoringConfig) {
this.queue = queue;
this.config = config;
}
async start(): Promise<void> {
const internalQueue = new Queue(
`__scoring:${this.queue.name}`,
{ connection: this.queue.opts.connection },
);
await internalQueue.add(
'scoring-tick',
{},
{ removeOnComplete: true, removeOnFail: false },
);
this.worker = new Worker(
internalQueue.name,
async () => {
const result = await this.runScoringCycle();
await internalQueue.add(
'scoring-tick',
{},
{
delay: this.config.rescoreIntervalMs,
removeOnComplete: true,
removeOnFail: false,
},
);
return result;
},
{
connection: this.queue.opts.connection,
concurrency: 1,
},
);
}
async stop(): Promise<void> {
await this.worker?.close();
}
private async runScoringCycle(): Promise<ScoringResult> {
const result: ScoringResult = {
evaluated: 0,
updated: 0,
errors: 0,
priorityDistribution: {},
};
try {
const [prioritizedJobs, waitingJobs] = await Promise.all([
this.queue.getJobs(['prioritized'], 0, 500),
this.queue.getJobs(['waiting'], 0, 500),
]);
const allJobs = [...prioritizedJobs, ...waitingJobs];
result.evaluated = allJobs.length;
const counts = await this.queue.getJobCounts(
'wait', 'prioritized', 'active', 'delayed',
);
const queueDepth = (counts.wait ?? 0) + (counts.prioritized ?? 0);
for (const job of allJobs) {
if (!job || !job.id) continue;
try {
const profile = this.findProfile(job.name);
const criticality = profile?.criticality ?? this.config.defaultCriticality;
const tier = (job.data as any).customerTier as string | undefined;
const tierMultiplier = tier
? (this.config.tierMultiplier[tier] ?? this.config.defaultTierMultiplier)
: this.config.defaultTierMultiplier;
const waitSeconds = job.timestamp
? Math.floor((Date.now() - job.timestamp) / 1000)
: 0;
const newPriority = computeMultiFactorPriority({
criticality,
tierMultiplier,
waitSeconds,
queueDepth,
maxPriority: 2_097_152,
});
const currentPriority = job.opts.priority ?? 0;
if (newPriority !== currentPriority) {
await job.changePriority({ priority: newPriority });
result.updated++;
result.priorityDistribution[newPriority] =
(result.priorityDistribution[newPriority] ?? 0) + 1;
}
} catch (err) {
result.errors++;
}
}
} catch (err) {
console.error(`[Scoring] Cycle error: ${err}`);
throw err;
}
return result;
}
private findProfile(jobName: string): JobTypeProfile | undefined {
return this.config.jobTypes.find((p) => {
if (p.pattern === '*') return true;
const regex = new RegExp(
`^${p.pattern.replace(/[.+^${}()|[\]\\]/g, '\\$&').replace(/\*/g, '.*')}$`,
);
return regex.test(jobName);
});
}
}
Scoring Walkthrough
| Job | Criticality | Tier | Wait Time | Queue Depth | Computed Priority |
|---|---|---|---|---|---|
onboarding-email |
90 | tier-1 (x1.5) | 60s | 200 | ~419,430 (high) |
password-reset |
85 | tier-2 (x1.2) | 10s | 200 | ~628,490 (medium-high) |
generate-report |
20 | tier-3 (x1.0) | 300s | 200 | ~1,782,579 (low) |
cleanup-task |
5 | tier-4 (x0.7) | 600s | 200 | ~2,054,109 (very low) |
As the onboarding-email job waits longer, its age penalty grows, pushing its priority higher. Meanwhile the cleanup-task never competes with customer-facing work.
Pattern 3: Priority-Weighted Worker Pool Routing
When high-priority jobs arrive in a busy queue, they still compete for worker attention. A dedicated fast-lane worker pool ensures urgent jobs get processed immediately by workers with low concurrency and fast processors, while bulk work goes to a high-concurrency pool.
The architecture uses three queues: a main entry queue, a fast-lane queue, and a bulk queue. A router worker classifies each job and forwards it to the appropriate sub-queue.
Router Worker
// workers/pool-router.ts
import { Queue, Job, Worker } from 'bullmq';
interface PoolRoutingConfig {
fastQueue: Queue;
bulkQueue: Queue;
fastLaneThreshold: number;
routerQueueName: string;
redisConnection: { host: string; port: number };
}
export class PriorityRouter {
private readonly config: PoolRoutingConfig;
private worker: Worker | null = null;
private readonly routerQueue: Queue;
constructor(config: PoolRoutingConfig) {
this.config = config;
this.routerQueue = new Queue(config.routerQueueName, {
connection: config.redisConnection,
});
}
async start(): Promise<void> {
this.worker = new Worker(
this.config.routerQueueName,
async (job: Job) => {
const { originalQueueName, originalJobId, jobName, jobData, jobOpts } =
job.data as {
originalQueueName: string;
originalJobId: string;
jobName: string;
jobData: Record<string, unknown>;
jobOpts: { priority: number; [key: string]: unknown };
};
const priority = jobOpts.priority ?? 100;
const targetQueue =
priority <= this.config.fastLaneThreshold
? this.config.fastQueue
: this.config.bulkQueue;
await targetQueue.add(jobName, jobData, {
...jobOpts,
jobId: originalJobId,
});
return {
routedTo: targetQueue.name,
priority,
originalJobId,
};
},
{
connection: this.config.redisConnection,
concurrency: 10,
},
);
}
async stop(): Promise<void> {
await this.worker?.close();
await this.routerQueue.close();
}
}
Pool Workers
The fast-lane worker runs with concurrency: 1 to minimize context switching and lock contention for critical jobs. The bulk worker runs with concurrency: 20 for maximum throughput.
// workers/pool-workers.ts
import { Worker, Job } from 'bullmq';
export function createFastLaneWorker(
queueName: string,
processor: (job: Job) => Promise<unknown>,
connection: { host: string; port: number },
): Worker {
return new Worker(
queueName,
async (job: Job) => {
console.log(`[FastLane] Processing ${job.id} (priority ${job.opts.priority})`);
return processor(job);
},
{
connection,
concurrency: 1,
lockDuration: 30_000,
stalledInterval: 15_000,
},
);
}
export function createBulkWorker(
queueName: string,
processor: (job: Job) => Promise<unknown>,
connection: { host: string; port: number },
): Worker {
return new Worker(
queueName,
async (job: Job) => {
console.log(`[Bulk] Processing ${job.id} (priority ${job.opts.priority})`);
return processor(job);
},
{
connection,
concurrency: 20,
lockDuration: 60_000,
stalledInterval: 30_000,
},
);
}
Wiring It Together
// main-pool.ts
import { Queue, Job } from 'bullmq';
import { PriorityRouter } from './workers/pool-router';
import { createFastLaneWorker, createBulkWorker } from './workers/pool-workers';
const connection = { host: 'localhost', port: 6379 };
async function main() {
const fastQueue = new Queue('app-fast-lane', { connection });
const bulkQueue = new Queue('app-bulk', { connection });
const mainQueue = new Queue('app-main', { connection });
const router = new PriorityRouter({
fastQueue,
bulkQueue,
fastLaneThreshold: 10,
routerQueueName: 'app-router',
redisConnection: connection,
});
await router.start();
const fastWorker = createFastLaneWorker(
'app-fast-lane',
async (job: Job) => {
await sendEmail(job.data);
return { processed: 'fast', jobId: job.id };
},
connection,
);
const bulkWorker = createBulkWorker(
'app-bulk',
async (job: Job) => {
await generateReport(job.data);
return { processed: 'bulk', jobId: job.id };
},
connection,
);
await mainQueue.add('onboarding-email', { to: 'user@example.com' }, { priority: 1 });
await mainQueue.add('generate-report', { month: '2026-06' }, { priority: 100 });
process.on('SIGTERM', async () => {
await router.stop();
await fastWorker.close();
await bulkWorker.close();
await fastQueue.close();
await bulkQueue.close();
await mainQueue.close();
});
}
main();
Monitoring Dynamic Priority with QueueHub
All three patterns generate rich observability data. Here's how QueueHub helps you monitor and verify your dynamic priority system.
Priority Distribution Panel
QueueHub's Queue Overview dashboard shows a Priority Distribution histogram with real-time counts from queue.getCountsPerPriority():
const counts = await queue.getCountsPerPriority([0, 1, 5, 10, 25, 50, 100, 500]);
// Returns: { '0': 12, '1': 3, '5': 7, '10': 22, '25': 15, '50': 8, '100': 31, '500': 44 }
Buckets are color-coded: red for urgent (priority 1-10), yellow for normal (11-100), green for background (101+). This gives operators an instant visual check that the escalation engine is working as expected.
Escalation Activity Feed
QueueHub's Job Activity timeline surfaces every changePriority event with timestamps, job IDs, old and new priority values, and the triggering rule. QueueHub captures this via BullMQ's global events:
queueEvents.on('changed', ({ jobId, data }) => {
// QueueHub records: { jobId, oldPriority, newPriority, timestamp }
});
Starvation Detection
QueueHub can alert when any waiting job has a wait time exceeding its escalation schedule's last threshold but hasn't been escalated. Configured in Settings → Alerts:
Condition: ANY(waiting_jobs WHERE wait_time > escalation_schedule[-1].threshold)
AND priority != expected_escalated_priority
Action: Slack webhook notification
Worker Pool Health
QueueHub's Live Worker View shows both fast-lane and bulk pools side by side — concurrency, active jobs, and processing rate. The cross-pool ratio (fast vs. bulk job completion rates) is a leading indicator of whether your threshold is correctly tuned.
Production Considerations
Escalation Worker Sizing
| Queue Size | Check Interval | Batch Size | Recommended Setup |
|---|---|---|---|
| < 1,000 | 30s | 200 | Single escalation worker |
| 1,000 - 10,000 | 15s | 500 | One worker per physical core |
| 10,000 - 100,000 | 10s | 1000 | Partition by shard; one worker per shard |
| > 100,000 | 5s | 2000 | Consider BullMQ Pro Groups with per-group priority |
Hysteresis
Without hysteresis, a job could bounce between priorities if the escalation worker and scoring worker disagree:
// utils/hysteresis.ts
export function applyHysteresis(
currentPriority: number,
candidatePriority: number,
hysteresisThreshold: number = 5,
): number {
const diff = Math.abs(currentPriority - candidatePriority);
if (diff < hysteresisThreshold) return currentPriority;
return candidatePriority;
}
Redis Connection Management
Reuse the same Redis connection across all workers and queues to avoid exhausting connections:
import { Redis } from 'ioredis';
const redisConnection = new Redis({
host: 'localhost',
port: 6379,
maxRetriesPerRequest: null,
enableReadyCheck: false,
});
const queue = new Queue('my-queue', { connection: redisConnection });
const escalationWorker = new PriorityEscalationWorker(queue, config);
Which Pattern to Use?
| Pattern | Best For | Complexity | Redis Ops/Cycle |
|---|---|---|---|
| Dynamic Escalation | Fixed job types with known SLA tiers | Low | O(batchSize x log n) |
| Multi-Factor Scoring | Heterogeneous jobs with multiple importance dimensions | Medium | O(batchSize x log n) |
| Weighted Pool Routing | Isolating critical jobs from noisy neighbors | Medium-High | O(1) per job + worker overhead |
Conclusion
Static priority is the console.log of job queue design — it works for demos but breaks under real production load. By making priority dynamic — escalating based on wait time, scoring from multiple signals, or routing to dedicated worker pools — you build queues that automatically adapt to changing conditions without human intervention.
QueueHub gives you the visibility to trust your dynamic priority system:
- Watch priority distributions shift in real time on the Queue Overview dashboard
- See every escalation event in the Job Activity feed
- Get alerted when a job's wait exceeds its escalation threshold
- Compare worker pool performance side by side in the Live Worker View
Ready to take control of your BullMQ queues? Try QueueHub for free — connect your Redis instance, see your priority distributions in seconds, and never let a critical job languish again.
Related Articles
Testing and Development Workflows for BullMQ vs BeeQueue
A practical guide to testing job queue code with BullMQ and BeeQueue — test fixtures, producer and worker testing patterns, local development workflows, and CI integration.
BeeQueue Production Patterns and Redis Internals: Beyond the Basics
Go beyond basic BeeQueue setup. Master Redis key internals, Prometheus health checks, dead letter queues, graceful shutdown, batch processing, and benchmarking methodology.
BullMQ vs BeeQueue: Operational Realities in Production
A no-fluff comparison of BullMQ and BeeQueue through the lens of production operations — deployment patterns, monitoring stacks, debugging real failures, metrics export, upgrade strategies, and the DevOps workflows each library supports (or doesn't).