Automating Queue Cleanup: Job Retention, Sanitize & TTL Patterns for BullMQ

The Hidden Cost of Unchecked Queue Growth

Your BullMQ queue is silently consuming Redis memory right now. Every completed job, every failed attempt, every event log entry occupies a Redis key that lives in RAM. Unlike a disk-backed database, Redis holds all this data in memory — and when memory runs out, something has to give.

The default behavior in BullMQ stores completed and failed jobs forever. No TTL. No auto-cleanup. No safety net. Out of the box, every job you add stays in Redis until you explicitly remove it. This is by design — BullMQ prioritizes data safety over automatic deletion — but it's also the single most common cause of Redis memory saturation in production.

Consider this real-world scenario: A team deploys a notification service using BullMQ with no removal configuration. Traffic is moderate — 500,000 jobs per day. Each job carries a small payload of roughly 500 bytes. After one month, that's 15 million jobs occupying approximately 7.5 GB of Redis memory. After three months, you're looking at 22+ GB. Redis crashes under the weight, and without careful monitoring, the team doesn't even realize jobs are being lost until users complain about missing notifications.

The fix isn't complicated, but it must be deliberate. Queue cleanup falls into three categories:

Automatic on-job-completion — configured per-job or per-worker using BullMQ's built-in removal options
Proactive scheduled cleanup — cron-driven cleanup workers that run on a timer, independent of job flow
Manual intervention — on-demand operations via code or a dashboard like QueueHub

This post covers all three approaches in depth, along with retention strategies, sanitization techniques, and Redis memory management best practices. If you're running BullMQ in production and haven't given cleanup deliberate thought, start here.

BullMQ's Built-in Cleanup APIs

removeOnComplete and removeOnFail — Your First Line of Defense

The simplest and most important cleanup mechanism is the removeOnComplete and removeOnFail options. These tell BullMQ to automatically remove jobs from Redis once they reach a finalized state.

These options work lazily — removal happens only when a new job completes or fails, not on a timer. If your queue goes idle for hours, jobs that reached terminal states before the idle period won't be cleaned until activity resumes.

You have three strategy options:

Strategy 1 — Remove all finalized jobs immediately:

import { Queue } from 'bullmq';

const queue = new Queue('notifications', { connection });

await queue.add('send-email', { to: 'user@example.com' }, {
  removeOnComplete: true,
  removeOnFail: true,
});

Best for fire-and-forget patterns like one-time notifications, ephemeral events, or webhook deliveries. No debugging history is retained.

Strategy 2 — Keep a fixed count of jobs:

await queue.add('send-email', { to: 'user@example.com' }, {
  removeOnComplete: { count: 1000 },
  removeOnFail: { count: 5000 },
});

Retains the most recent N completed and M failed jobs. A good default for most queues — keep enough history for debugging (especially failed jobs) while capping unbounded growth.

Strategy 3 — Age-based retention with the KeepJobs object:

await queue.add('send-email', { to: 'user@example.com' }, {
  removeOnComplete: {
    age: 3600,     // keep completed jobs for 1 hour
    count: 1000,   // max 1000 kept
    limit: 100,    // remove at most 100 per iteration
  },
  removeOnFail: {
    age: 86400,    // keep failed jobs for 24 hours
    limit: 50,     // remove at most 50 per iteration
  },
});

Combines count and age thresholds. The limit parameter controls how many jobs are removed in a single cleanup iteration, which prevents Redis CPU spikes.

You can set these options on individual jobs via JobsOptions or as defaults on the Worker:

import { Worker } from 'bullmq';

const worker = new Worker('notifications', async (job) => {
  // process the job
}, {
  connection,
  removeOnComplete: { count: 1000 },
  removeOnFail: { count: 5000 },
});

Per-job options override worker defaults, giving you fine-grained control when certain jobs need different retention rules.

Queue.clean() — Proactive Time-Based Cleanup

The queue.clean() method is your proactive cleanup tool. Unlike lazy auto-removal, clean() runs on demand and removes jobs older than a specified grace period.

Signature: queue.clean(grace: number, limit: number, type?: JobState): Promise<string[]>

grace — milliseconds. Jobs younger than this are kept.
limit — maximum number of jobs to remove in one call. Start with 1000 and tune based on your Redis CPU profile.
type — the job state to clean: 'completed', 'failed', 'waiting', 'active', 'delayed', 'paused', 'prioritized', or 'wait'.

import { Queue } from 'bullmq';

const queue = new Queue('email', { connection });

// Remove completed jobs older than 1 hour, max 1000 at a time
const deleted = await queue.clean(60 * 60 * 1000, 1000, 'completed');
console.log(`Cleaned ${deleted.length} completed jobs`);

// Remove failed jobs older than 24 hours
const failedDeleted = await queue.clean(24 * 60 * 60 * 1000, 500, 'failed');
console.log(`Cleaned ${failedDeleted.length} failed jobs`);

The method returns an array of deleted job IDs — useful for logging and auditing. When queue.clean() runs, it fires a 'cleaned' event. Note that this event is not fired when jobs are removed via removeOnComplete or removeOnFail — only when you call clean() explicitly.

Queue.drain() — Emergency Reset

queue.drain() clears all waiting and delayed jobs without affecting active, completed, or failed jobs. Think of it as a controlled reset — useful when you need to clear a backlog after a downstream outage without losing jobs currently being processed.

// Drain waiting jobs only
await queue.drain();

// Also drain delayed jobs
await queue.drain(true);

Use cases include: clearing accumulated jobs during a deployment, stopping a burst of scheduled maintenance jobs, or resetting a development queue.

Queue.obliterate() — The Nuclear Option

When you need to completely destroy a queue and all its contents — waiting, active, completed, failed, delayed, events, metrics, everything — obliterate() is the tool.

// Safe obliterate — refuses if any active jobs exist
await queue.obliterate();

// Force obliterate — removes everything regardless of active jobs
await queue.obliterate({ force: true });

By default, obliterate() refuses if any active jobs exist. Pass { force: true } only when you're certain — this can disrupt in-flight processing.

Performance note: obliterate() iterates over all stored job keys. For queues with millions of jobs, this can take a couple of minutes. GitHub discussion #2733 reports approximately 2 minutes for 4 million jobs — which is still significantly faster than draining one job at a time.

Queue.trimEvents() — Event Stream Hygiene

BullMQ's QueueEvents stream auto-trims to approximately 10,000 events by default. While sufficient for most use cases, high-throughput queues can generate millions of events during bursts, and the default cap may still accumulate.

// Keep only the last 100 events
await queue.trimEvents(100);

Call trimEvents() in your cleanup worker to keep event memory in check, especially for queues processing thousands of jobs per minute.

Queue.removeOrphanedJobs() — Data Integrity Cleanup

Introduced in BullMQ v5 as a migration helper, removeOrphanedJobs() finds and removes job hash keys that exist in Redis but aren't referenced in any queue state set. This is a safety net for bugs that can leave orphaned data behind.

const removed = await queue.removeOrphanedJobs(1000, 0);
console.log(`Removed ${removed} orphaned job hashes`);

The first parameter (count) controls the SCAN iteration batch size (default 1000). The second (limit) caps total removals — pass 0 for unlimited.

Building an Automated Cleanup Worker

Why Auto-Removal Isn't Enough

Relying solely on removeOnComplete and removeOnFail creates a coverage gap. Here's why:

Lazy removal only fires when new jobs complete. During low-traffic periods (overnight, weekends, holidays), completed and failed jobs accumulate indefinitely.
Event streams are auto-trimmed but can still grow. The default 10,000 event cap may be insufficient for bursty queues.
Stalled jobs in the active detection path may never trigger cleanup. If a worker crashes and its jobs stall, the cleanup code in the completion handler never runs.
Failed jobs in certain edge cases (e.g., jobs that fail validation before entering a worker) may not trigger the automatic removal path.

The solution is a dedicated cleanup worker that runs on a cron schedule, independent of your application's job flow.

The Cleanup Worker Pattern

BullMQ's upsertJobScheduler API lets you register repeatable jobs on a cron schedule. Here's a complete cleanup worker architecture:

import { Queue, Worker } from 'bullmq';
import IORedis from 'ioredis';

const connection = new IORedis({ host: 'localhost', port: 6379 });
const cleanupQueue = new Queue('queue-cleanup', { connection });

// Register a cleanup job every hour
await cleanupQueue.upsertJobScheduler(
  'hourly-cleanup',
  { pattern: '0 * * * *' },  // every hour
  {
    name: 'cleanup',
    data: {
      queues: ['email', 'notifications', 'reports'],
      gracePeriods: {
        completed: 3600,   // 1 hour in seconds
        failed: 86400,     // 24 hours in seconds
      },
      batchSize: 1000,
    },
    opts: { removeOnComplete: true, removeOnFail: { count: 10 } },
  }
);

The cleanup worker processor iterates over each queue and calls clean() for completed and failed states:

const worker = new Worker(
  'queue-cleanup',
  async (job) => {
    const { queues, gracePeriods, batchSize } = job.data;
    const results = [];

    for (const queueName of queues) {
      const queue = new Queue(queueName, { connection });
      try {
        const completedRemoved = await queue.clean(
          gracePeriods.completed * 1000,
          batchSize,
          'completed'
        );

        const failedRemoved = await queue.clean(
          gracePeriods.failed * 1000,
          batchSize,
          'failed'
        );

        await queue.trimEvents(1000);

        results.push({
          queue: queueName,
          completedRemoved: completedRemoved.length,
          failedRemoved: failedRemoved.length,
        });
      } finally {
        await queue.close();
      }
    }

    return results;
  },
  { connection }
);

worker.on('completed', (job, result) => {
  console.log('Cleanup complete:', JSON.stringify(result));
});

Rate-Limited Batch Cleanup

Calling queue.clean() on a queue with millions of jobs can spike Redis CPU. Limit the impact with two techniques:

Rate-limited worker using BullMQ's built-in limiter:

const cleanupWorker = new Worker(
  'queue-cleanup',
  async (job) => { /* cleanup processor */ },
  {
    connection,
    limiter: {
      max: 5,        // only 5 cleanup iterations per
      duration: 1000, // second
    },
  }
);

Staggered cleanup across multiple runs:

Instead of cleaning all states in one run, spread the work:

Run 1: clean 'completed' jobs only
Run 2: clean 'failed' jobs only
Run 3: trim events

This spreads the Redis load across multiple cron intervals, reducing the risk of blocking commands on a busy Redis instance.

Job Retention Policies

Defining a Retention Strategy

Not all jobs are created equal. Your retention strategy should balance business requirements, debugging needs, and memory budget. Here are the factors to weigh:

Business requirements: How long does your compliance policy require job history? Financial systems may need 90-day retention; a notification service may need only hours.
Debugging needs: Failed jobs are invaluable for root cause analysis. Keep them significantly longer than completed jobs. A failed job at 3AM on a Saturday is the difference between a quick fix and an all-hands incident.
Memory budget: Each job's memory footprint in Redis is approximately key overhead (80-120 bytes) + metadata (200-400 bytes) + payload size. A 1 KB payload job uses roughly 1.5 KB in Redis. At 1 million jobs per day, that's about 1.5 GB per day of memory.
Throughput: High-throughput queues processing millions of jobs daily need aggressive cleanup. Low-throughput queues handling critical business workflows can afford longer retention.

Suggested default retention tiers:

Job Type	Retention	Rationale
Completed (transient)	Remove immediately or keep 100	Fire-and-forget; no history needed
Completed (auditable)	Keep 7-30 days	Compliance and audit trail
Failed	Keep 30 days	Sufficient debugging window
Failed (critical)	Keep 90 days or offload	Long-term analysis and compliance

TTL Patterns in BullMQ

BullMQ does not use Redis TTL on job keys. This is intentional — BullMQ manages job lifecycle explicitly through sorted sets and Lua scripts. Applying Redis TTL directly to job keys would break queue integrity by removing job metadata while leaving dangling references in the queue's sorted sets.

What works instead:

removeOnComplete / removeOnFail with the age parameter (the KeepJobs pattern shown earlier)
queue.clean() for proactive time-based removal
Redis maxmemory-policy noeviction combined with explicit cleanup (never Redis eviction)

Common mistake: Setting maxmemory-policy: allkeys-lru or volatile-ttl thinking it provides cleanup. These policies cause Redis to arbitrarily delete BullMQ keys, breaking queue integrity, producing stalled workers, and corrupting job state. Always use noeviction and let cleanup operations fail loudly when memory is exhausted.

Offloading Job Data to External Storage

For long-term retention without Redis bloat, archive job data to an external database:

import { Worker } from 'bullmq';

const worker = new Worker(
  'email',
  async (job) => {
    const result = await processEmail(job.data);

    // Archive to external DB for long-term retention
    await db.jobHistory.create({
      jobId: job.id,
      queue: 'email',
      data: job.data,
      result,
      completedAt: new Date(),
    });

    // Remove from Redis immediately
    return result;
  },
  {
    connection,
    removeOnComplete: { count: 0 },  // remove immediately
    removeOnFail: { age: 3600 },      // keep failed jobs 1 hour
  }
);

External storage options include PostgreSQL, MongoDB, S3-compatible object storage, or ClickHouse for analytics. BullMQ maintainers recommend this approach in GitHub Discussion #2001 — offload to a separate database where storage is cheap and query patterns are optimized for historical access.

Queue Sanitization Strategies

Stale Job Detection and Removal

Stale jobs can accumulate in several ways:

Jobs stuck in "active" state from dead workers — BullMQ's stall detection handles these in approximately 30 seconds, but this only moves them back to "waiting" or "failed" status. They still occupy memory.
Orphaned job hashes — job keys in Redis not referenced by any queue state set. BullMQ v5's removeOrphanedJobs() addresses this.
Jobs in "waiting-children" with no pending children — stale flow graph nodes in complex job chains.
Repeatable job scheduler entries for deleted queues — scheduler metadata that outlives its queue.

Detection approaches:

// Check for orphaned job hashes
const orphanedCount = await queue.removeOrphanedJobs();
console.log(`Removed ${orphanedCount} orphaned job hashes`);

// Check for stalled active jobs
const counts = await queue.getJobCounts();
if (counts.active > 0 && workersActive === 0) {
  console.warn('Active jobs with no active workers — possible stale jobs');
}

Job Data Purging

Job payloads often contain large objects — serialized API responses, file contents, or database results. This data persists in Redis even after processing. Two patterns can help:

Strip payload after processing:

// In your worker, after successful processing:
await job.updateData({ _purged: true, originalSize: job.data.length });

Store payload reference instead of full data:

// Producer stores payload in S3, adds reference in job data
await queue.add('process-file', {
  s3Key: 'jobs/payloads/abc-123.json',
  // Instead of embedding a 10 MB payload, just reference it
});

Deduplication and Cleanup Interactions

BullMQ's deduplication prevents adding jobs with a duplicate jobId by tracking the existing job key. An important detail: once a job is removed by cleanup, a new job with the same jobId will be accepted again — the dedup guard is gone.

If you rely on idempotent job execution, consider maintaining an external deduplication store (a database unique constraint or a Redis set with a longer TTL) to prevent duplicate processing after cleanup removes the BullMQ-level guard.

Redis Memory Bloat — Root Causes and Prevention

What Eats Redis Memory in BullMQ

Component	Redis Key Pattern	Memory per Item
Job data (hash)	`bull:queue:job:<id>`	~500 bytes + payload
Completed set (sorted set)	`bull:queue:completed`	~40 bytes per reference
Failed set (sorted set)	`bull:queue:failed`	~40 bytes per reference
Event stream	`bull:queue:events`	~200 bytes per event
Metrics	`bull:queue:metrics:*`	Minute/hourly counters
Worker heartbeats	`bull:queue:worker:<id>`	~100 bytes (TTL-managed)
Delayed set	`bull:queue:delayed`	~40 bytes per reference

The job data hashes dominate memory consumption. A GitHub issue (bullmq #2734) describes a team whose queue reached 10 GB of Redis memory because they had no removeOnComplete configured and their job payloads were large. The root cause was entirely preventable.

Memory Monitoring Checklist

Track used_memory and used_memory_rss from Redis INFO output
Set memory alert thresholds at 75%, 85%, and 95% of maxmemory
Monitor evicted_keys — if this counter is rising with noeviction, Redis is refusing writes, and jobs are being silently dropped
Use a dashboard like QueueHub to see per-queue job counts and trends at a glance
Log cleanup job metrics: jobs removed per cycle, total Redis memory freed

Preventing Bloat Before It Starts

The most effective cleanup strategy is prevention. Follow these guidelines at every stage:

Design-time: Keep job payloads small. Reference external data instead of embedding it. A job ID referencing a database row is far cheaper than the full serialized row.
Add-time: Set removeOnComplete / removeOnFail on every job. Make it a linting rule or code review checklist item.
Worker-time: Clean up sensitive or large data from job payloads in the completed event handler.
Infrastructure-time: Configure Redis maxmemory-policy noeviction. Let cleanup failures crash loudly instead of silently losing data.

Conclusion

Queue cleanup is not an afterthought — it's a core operational practice for any production BullMQ deployment. Here are the key takeaways:

Default behavior is dangerous. BullMQ keeps every completed and failed job forever unless you configure removal. The "remove nothing" default is a trap.
Three-tier defense wins. Configure per-job auto-removal (removeOnComplete/ removeOnFail) for day-to-day management. Add cron-based proactive cleanup (queue.clean()) for coverage during low-traffic periods. Keep manual tools (drain(), obliterate()) for emergencies.
Rate limit your cleanup. Aggressive cleanup can spike Redis CPU just as badly as a burst of incoming jobs. Use the limit parameter and rate-limit your cleanup worker.
Keep failed jobs longer than completed ones. Failed jobs are debugging gold — they contain the context needed to fix root causes. Completed jobs (especially successful ones) have far less diagnostic value.
Never use Redis eviction as a cleanup strategy. Always set maxmemory-policy noeviction. Let cleanup fail loudly rather than letting Redis silently evict critical job data.
Monitor your cleanup effectiveness. Track how many jobs are removed per cycle, what your Redis memory trend looks like, and whether your cleanup worker is keeping up with volume.

Ready to take control of your queue health? Try Queue Hub by QueueHub to visualize your queues, spot memory issues before they escalate, and manage cleanup across all your queues from a single dashboard. With per-queue job counts, Redis memory gauges, and one-click clean/drain/obliterate actions, QueueHub gives you the observability you need to keep your BullMQ queues running smoothly.