Redis Memory Management for BullMQ Queues: maxmemory, Eviction Policies & Job Retention

Redis is the beating heart of BullMQ — it holds every job, every state transition, every metric, and every piece of queue metadata. All of this data lives exclusively in RAM. Without proper memory management, jobs disappear silently, workers stall, and debugging becomes a nightmare.

This guide covers everything you need to know about Redis memory management for BullMQ queues: from maxmemory configuration and eviction policies to job retention strategies and production monitoring. By the end, you'll have a practical playbook for keeping your queues running smoothly at any scale.

---

Why Redis Memory Matters for BullMQ

BullMQ is built on top of Redis, and virtually all of its state is stored in memory. Every enqueued job, completed job, failed job, worker heartbeat, queue metric, and scheduling instruction is a Redis key. Unlike a traditional database that flushes to disk on every write, Redis is primarily an in-memory store — persistence (RDB/AOF) is a safety net, not a working set.

When Redis runs out of memory, one of two things happens:

• noeviction (the only safe policy for BullMQ): writes fail with an OOM error. Jobs can't be added, workers can't acknowledge completion, and the entire queue system grinds to a halt.
• An eviction policy (anything else): Redis starts deleting keys arbitrarily to free memory. This means your jobs vanish — completed, failed, waiting, active — without warning.

Neither scenario is acceptable in production. The solution is proactive memory management: understanding what consumes memory, setting appropriate limits, and purging data you no longer need.

---

Redis maxmemory Configuration & Sizing

Setting maxmemory

You can configure maxmemory in redis.conf:

# redis.conf
maxmemory 2gb
maxmemory-policy noeviction

Or at runtime using redis-cli:

redis-cli CONFIG SET maxmemory 2gb
redis-cli CONFIG SET maxmemory-policy noeviction

Sizing Formula

A good starting point for sizing your Redis instance for BullMQ uses this formula:

maxmemory = (jobs per day × retention days × average job size) × safety multiplier

Where:

• Safety multiplier = 1.5 to 2.0 (accounts for metadata, worker keys, logs, and growth spikes)
• Average job size includes the job data payload plus BullMQ's internal metadata (~500 bytes per job baseline)

Example: 100,000 jobs/day × 7 days retention × 2 KB average job size × 1.5 safety = ~2.1 GB

Rule of Thumb

• Use 60–70% of available RAM as your maxmemory setting
• Set monitoring alerts at 75% and 85% utilization
• Reserve headroom for Redis background saves (fork uses copy-on-write) and traffic spikes

Measuring Actual Usage

Use the MEMORY USAGE command to inspect specific keys:

redis-cli MEMORY USAGE bull:myqueue:completed
redis-cli MEMORY USAGE bull:myqueue:id

And INFO memory for the full picture:

redis-cli INFO memory | grep -E "used_memory_human|maxmemory|used_memory_peak_human"

---

Eviction Policies: Why noeviction Is Required for BullMQ

Redis offers 8 eviction policies that control what happens when memory reaches maxmemory. Here's how each one behaves with BullMQ:

Policy	Behavior	Safe for BullMQ?
noeviction	Returns errors on writes when memory is full	✅ Required
allkeys-lru	Evicts least-recently-used keys from all keys	❌ Will delete jobs
allkeys-lfu	Evicts least-frequently-used keys from all keys	❌ Will delete jobs
allkeys-random	Evicts random keys from all keys	❌ Will delete jobs
volatile-lru	Evicts LRU keys among those with TTL set	❌ Unsafe — BullMQ keys lack TTL
volatile-lfu	Evicts LFU keys among those with TTL set	❌ Unsafe
volatile-random	Evicts random keys among those with TTL set	❌ Unsafe
volatile-ttl	Evicts keys with shortest TTL	❌ Unsafe

The BullMQ documentation is unequivocal: BullMQ cannot work properly if Redis evicts keys arbitrarily. When Redis evicts a BullMQ key — say, a list of completed jobs or a queue metadata key — the result is data corruption, phantom jobs, and inexplicable worker failures.

The Shared Redis Counterargument

If you're sharing a Redis instance across multiple services and can't set noeviction, some teams attempt volatile-lru as a compromise. This is fragile and dangerous because BullMQ's core queue keys do not have TTLs — they live indefinitely until explicitly cleaned. volatile-lru will either evict other services' keys (if they use TTL) or fall back to error behavior identical to noeviction when no volatile keys exist.

The right answer: use a dedicated Redis instance for BullMQ. It doesn't need to be large — a t3.micro with 0.5 GB RAM handles moderate workloads — but it must be exclusively yours with noeviction.

---

How BullMQ Uses Redis Memory

Key Namespace Anatomy

Each BullMQ queue creates 20+ Redis keys in the bull:{queueName}: namespace:

bull:myqueue:meta              # Queue metadata (version, status)
bull:myqueue:id                # Auto-incrementing job ID counter
bull:myqueue:wait              # List of waiting jobs
bull:myqueue:active            # List of currently active jobs
bull:myqueue:completed         # Sorted set of completed jobs (score = timestamp)
bull:myqueue:failed            # Sorted set of failed jobs (score = timestamp)
bull:myqueue:delayed           # Sorted set of delayed jobs (score = timestamp)
bull:myqueue:paused            # List of paused jobs
bull:myqueue:repeat            # Set of repeatable job configurations
bull:myqueue:workers           # Set of active worker identifiers
bull:myqueue:stalled-check     # String tracking stall detection
bull:myqueue:events            # Pub/sub channels for queue events
bull:myqueue:{jobId}           # Individual job data (one per job)
...and more

Biggest Memory Consumers

1. Completed jobs (#1) — By default, BullMQ keeps every completed job forever. On a busy queue, this is the fastest-growing memory sink.
2. Failed jobs (#2) — Similarly, failed jobs accumulate indefinitely unless pruned.
3. Job data blobs (#3) — Large payloads (e.g., serialized objects, base64-encoded data, full API responses) dramatically increase per-job memory.

Hidden Traps

• Stalled jobs leaving ghost entries: When a worker crashes without proper cleanup, the stalled job remains in the active list and its individual job key persists until the stall detector times out and moves it back to wait. In extreme cases, multiple stalled detection cycles can leave duplicate job entries.
• Large job data payloads: Storing entire database records or file contents in job data. A single job with a 500 KB payload occupies 500 KB × (number of states: waiting + active + completed/failed + delayed backups).

// ❌ Anti-pattern: storing full objects in job data
await myQueue.add("process-order", {
  orderId: 42,
  customer: { /* huge nested object */ },
  items: [/* many items */],
  fullInvoicePdf: "base64-encoded-megabytes...",
});

// ✅ Best practice: store only identifiers
await myQueue.add("process-order", {
  orderId: 42,
});
// Worker fetches full data from the primary database

• Unbounded job logs: Using job.log() extensively without limits. Each log entry is a separate Redis list push that lives forever.
• Long queue names: Queue names in the key prefix are stored repeatedly. bull:my-very-long-queue-name-that-describes-everything: adds unnecessary bytes to every key.

---

Monitoring Redis Memory for BullMQ

Essential Commands

Command	What It Tells You
INFO memory	Used memory, peak memory, fragmentation ratio
MEMORY STATS	Per-type memory breakdown
MEMORY USAGE <key>	Exact memory for a specific key
INFO keyspace	Key counts per database
SLOWLOG GET 10	Slow commands (often memory-intensive operations)

Eviction Monitoring

# Check if evictions are happening
redis-cli INFO stats | grep evicted_keys

# Monitor in real-time
redis-cli --stat

If evicted_keys is non-zero and your policy is not noeviction, BullMQ keys are being destroyed.

Redis Big Keys Scanning

redis-cli --bigkeys

This scans for the largest keys in your Redis instance. For BullMQ, completed and failed sorted sets are typically the biggest offenders.

How Memory Issues Manifest in Queue Hub

When running low on memory, you'll see:

• Dropped job counts — the completed/failed counters show fewer entries than expected
• Null job data — clicking on a job returns empty or partial data
• Stalled workers — workers show as active but never process jobs
• Enqueue failures — queue.add() throws OutOfMemory errors

Queue Hub's dashboard surfaces these symptoms immediately, showing you memory pressure before it becomes critical.

---

The clean() Method: Your First Line of Defense

BullMQ provides the clean() method to programmatically remove completed or failed jobs. This is your primary tool for keeping memory under control.

API

import { Queue } from "bullmq";

const myQueue = new Queue("my-queue", {
  connection: { host: "localhost", port: 6379 },
});

// Remove completed jobs older than 1 hour (3600 seconds)
const removedCount = await myQueue.clean(3600, 1000, "completed");
console.log(`Removed ${removedCount} completed jobs`);

// Remove failed jobs older than 2 hours
const failedRemoved = await myQueue.clean(7200, 500, "failed");

Parameters:

Parameter	Type	Description
grace	number	Age in seconds — jobs older than this are removed
limit	number	Max number of jobs to remove per call
type	`'completed'	'failed'

### Scheduled Cron Cleanup

In production, run cleanup on a schedule:

```typescript
// Run cleanup every hour
setInterval(async () => {
  try {
    await myQueue.clean(3600, 1000, "completed");
    await myQueue.clean(3600, 500, "failed");
    console.log(`[cleanup] Purged old jobs at ${new Date().toISOString()}`);
  } catch (err) {
    console.error("[cleanup] Error:", err);
  }
}, 60 * 60 * 1000);

Or use a cron job:

# crontab — run every 30 minutes
*/30 * * * * cd /app && node scripts/clean-queue.mjs >> /var/log/queue-clean.log 2>&1

clean() vs drain() vs obliterate()

Method	What It Does	When to Use
clean()	Removes jobs older than a grace period by type	Routine maintenance — scheduled cleanup
drain()	Removes all waiting jobs (preserves completed/failed)	Draining backlog before maintenance
obliterate()	Destroys the entire queue — all keys, all jobs, all metadata	Decommissioning a queue completely. Use with force: true to bypass active checks.

// Drain waiting jobs only
await myQueue.drain();

// ⚠️ Completely obliterate a queue (irreversible!)
await myQueue.obliterate({ force: true });

---

Job Retention Strategies: removeOnComplete / removeOnFail

The cleanest approach is to prevent memory bloat at the source using BullMQ's built-in retention options.

Worker-Level Defaults

Set default retention at the worker level so every job inherits the policy:

import { Worker } from "bullmq";

const worker = new Worker("my-queue", async (job) => {
  // process the job
}, {
  connection: { host: "localhost", port: 6379 },
  removeOnComplete: { count: 100, age: 3600 * 24 }, // keep 100, max 1 day
  removeOnFail: { count: 50, age: 3600 * 24 * 7 },  // keep 50, max 7 days
});

Per-Job Overrides

Override retention for specific jobs:

// Job that should be kept longer (e.g., audit trail)
await myQueue.add("audit-event", { eventId: 42 }, {
  removeOnComplete: { age: 3600 * 24 * 90 },  // 90 days
  removeOnFail: { age: 3600 * 24 * 365 },      // 1 year
});

// High-throughput job that should disappear immediately
await myQueue.add("analytics-ping", { metric: "pageview" }, {
  removeOnComplete: true,   // remove immediately on completion
  removeOnFail: { count: 0 }, // don't keep failures either
});

Retention Decision Matrix

Use Case	removeOnComplete	removeOnFail	Strategy
High-throughput (logs, analytics, pings)	true or {count: 0}	{count: 0}	Zero retention — jobs are fire-and-forget
Development	{count: 500}	{count: 500}	Keep recent jobs for debugging
Production with error tracking	{count: 1000, age: 86400} (1 day)	{count: 5000, age: 604800} (7 days)	Keep more failures for root cause analysis
Compliance / audit	{age: 7776000} (90 days)	{age: 7776000} (90 days)	Long retention mandated by policy
Analytics pipeline	{count: 100}	{count: 100}	Keep samples for trend analysis

The "No Retention" Anti-Pattern

Setting removeOnComplete: true and removeOnFail: { count: 0 } on a worker that processes essential business operations (order fulfillment, payment processing, email delivery) means you lose all visibility into past job outcomes. When something goes wrong, you'll have no history to investigate.

A better approach: keep a modest number of recent jobs for debugging, and use a scheduled clean() for bulk removal of older entries. Or use Queue Hub, which gives you a live view even with limited retention.

---

Best Practices: Do's and Don'ts

Quick Reference Table

✅ Do	❌ Don't
Set maxmemory-policy noeviction	Use allkeys-lru or any eviction policy
Schedule regular clean() calls	Let completed/failed jobs accumulate indefinitely
Use removeOnComplete / removeOnFail options	Rely on drain() for routine maintenance
Store IDs in job data, fetch full data in workers	Bloat job payloads with full objects
Set memory usage alerts at 75% and 85%	Run Redis without monitoring
Use a dedicated Redis for BullMQ	Share BullMQ's Redis with cache workloads
Test cleanup logic in staging	Add retention without testing first
Use short, descriptive queue names	Use verbose queue names

Job Data Optimization

Keep job data minimal:

// ❌ Don't: embed everything
await queue.add("send-email", {
  to: "user@example.com",
  subject: "Welcome!",
  html: "<html>...megabytes of template...</html>",
  attachments: [{ filename: "report.pdf", content: "base64-data..." }],
});

// ✅ Do: reference external data
await queue.add("send-email", {
  emailId: "e_abc123", // Worker fetches from DB
  templateName: "welcome",
  userId: 42,
});

Redis Instance Sizing Guide

Workload	Jobs/Day	Retention	Recommended Instance	RAM
Development / Single app	< 10K	1 day	t3.micro	0.5 GB
Small production	10K–100K	7 days	t3.small	2 GB
Medium production	100K–1M	3 days	t3.medium	4 GB
High throughput	1M–10M	1 day	r6g.large	8 GB
Enterprise / Data-heavy	10M+	Custom	r6g.xlarge+	16 GB+

Valkey Considerations

If you're using Valkey (the Redis fork) instead of Redis, the same principles apply. Valkey supports all the same eviction policies, MEMORY USAGE, and maxmemory configuration. As of Valkey 7.2+, the behavior is identical to Redis 7.x for BullMQ use cases.

---

Emergency OOM Playbook

When Redis runs out of memory and you need to recover immediately:

1. Identify the culprit: Run redis-cli --bigkeys to find the largest keys.
2. Check eviction policy: redis-cli CONFIG GET maxmemory-policy. If not noeviction, change it.
3. Increase maxmemory (temporarily): redis-cli CONFIG SET maxmemory 4gb if RAM is available.
4. Clean completed/failed jobs: redis-cli ZREMRANGEBYRANK bull:myqueue:completed 0 -1000 (removes all but newest 1000).
5. Restart workers: After freeing memory, restart workers to clear stale connections.
6. Delete orphaned keys: Check for BullMQ keys that belong to deleted queues: redis-cli KEYS "bull:deleted-queue:*" and DEL them.
7. Scale up: Provision a larger Redis instance. For ElastiCache/MemoryDB, this means a vertical scaling operation.

---

How Queue Hub Helps

Managing Redis memory across multiple queues and environments is a constant operational challenge. Queue Hub (at queuehub.tech) gives you the tools to stay on top of it:

• Overview dashboard — See memory usage, job throughput, and queue health at a glance
• Job filtering — Filter by state, age, and name to find problematic jobs without scanning Redis directly
• Live worker view — Monitor active workers, stalled detectors, and processing lag in real time
• Agent tunneling — Connect to Redis instances behind VPCs or firewalls without exposing them to the internet
• Multi-backend support — Manage BullMQ and BeeQueue queues from a single interface

Instead of SSHing into production to run MEMORY USAGE commands, you can see your biggest consumers and clean them with a single click.

---

Summary Checklist

Pre-Production (10 Items)

• maxmemory-policy set to noeviction
• maxmemory sized using the workload formula (60–70% of available RAM)
• removeOnComplete configured on all workers
• removeOnFail configured on all workers
• Scheduled clean() cron job in place
• Memory alerts configured at 75% and 85% utilization
• Job payloads optimized — store IDs, not objects
• Queue names are short and descriptive
• Dedicated Redis instance for BullMQ (not shared with caches)
• Cleanup scripts tested in staging environment

Monthly Audit (6 Items)

• Review INFO memory — check used, peak, and fragmentation ratio
• Run redis-cli --bigkeys — identify growth trends
• Check evicted_keys — confirm zero since last audit
• Review completed/failed job counts — adjust retention if growing
• Scan for orphaned queue keys from decommissioned queues
• Validate Queue Hub dashboard accuracy and alert thresholds

---

Put Your Redis Memory on Autopilot

You don't have to manage all of this manually. Queue Hub gives you real-time visibility, one-click cleanup, and live monitoring for all your queues. Stop wrestling with redis-cli commands in production — see exactly what's consuming memory and fix it instantly.

👉 Try Queue Hub at queuehub.tech — your queues deserve a proper dashboard.