BullMQ Flows: Parent-Child Jobs and When to Use Them
Most BullMQ applications treat a job as a single unit of work, and that is fine for most workloads. But a surprising share of real queue traffic is batch-shaped: send 10,000 transactional emails, generate thumbnails for 500 images, call a dozen third-party APIs and merge the results. Model that with individual jobs and you end up writing your own coordination layer — tracking which sub-tasks finished, waiting for stragglers, and gluing results back together. BullMQ flows are the built-in answer: a parent job that stays incomplete until its child jobs finish, with results collected automatically. This post explains how flow trees work under the hood, when they beat hand-rolled job chaining, and the pitfalls that make flows stall in production.
What Are BullMQ Flows?
A BullMQ flow is a tree of jobs with exactly two levels: one parent job and any number of child jobs. You create the whole tree in a single call using the FlowProducer class, and children can live in different queues from the parent — which is the whole point.
import { FlowProducer } from "bullmq";
const flow = new FlowProducer({
connection: { host: "127.0.0.1", port: 6379 },
});
await flow.add({
name: "send-batch",
queueName: "email-batches",
data: { campaignId: 42 },
children: [
{ name: "send-email", queueName: "emails", data: { userId: 1, campaignId: 42 } },
{ name: "send-email", queueName: "emails", data: { userId: 2, campaignId: 42 } },
{ name: "send-email", queueName: "emails", data: { userId: 3, campaignId: 42 } },
],
});
The parent (send-batch) is added to email-batches; the three children are added to emails. Workers on each queue pick up their own jobs. The parent sits in a waiting-children state until every child completes — only then does it move to waiting and get processed by a worker on its own queue.
How Flow Trees Work Under the Hood
- Each child job carries a reference to its parent (parent queue name and job id), so workers and dashboards can walk the tree.
- The parent's record in Redis keeps a count of expected children. As children complete, their return values are stored and attached to the parent.
- When the last child finishes, the parent is released from waiting-children and becomes eligible for processing.
- When a worker picks up the parent, it can read the aggregated child results with
job.waitChildren():
const worker = new Worker(
"email-batches",
async (job) => {
const results = await job.waitChildren();
// Every send-email child has completed — results is keyed by child job name
await sendDigest(job.data.campaignId, results);
},
{ connection: { host: "127.0.0.1", port: 6379 } }
);
- Trees are limited to two levels: children cannot have children. If you need a deeper pipeline, compose two flows or use sequential steps.
Because children live in their own queues, each child queue gets its own concurrency, its own workers, and its own monitoring. That is the feature's real superpower: the fan-out is processed by whatever capacity you attach to the child queue, and the parent simply waits.
When to Reach for Flows
- Fan-out/fan-in batches — process a list of items in parallel, then run one aggregation step when all of them are done (the pattern above).
- Parallel API calls — fetch from several providers simultaneously and merge the responses in the parent.
- Cross-queue pipelines — children that need different worker resources (a CPU-heavy queue and an I/O-heavy queue) are naturally expressed as flows.
- Retryable sub-tasks — a failed child retries on its own without re-running the siblings.
When Not to Use Flows
- Simple sequential work. If step B always runs after step A and needs nothing else, a single job that does both is simpler.
- Very wide fan-out. Every child is a Redis write plus a result to collect; trees with thousands of children add real Redis pressure. Chunk the batch instead.
- No aggregate result needed. If the "parent" has no work to do, you are paying the coordination cost for nothing — just add the jobs.
- Strict ordering. Children run concurrently and finish in whatever order workers get to them. If order matters, flows are the wrong tool.
Pitfalls That Stall Flows
- Flows are not atomic.
flow.add()performs multiple Redis writes; if it fails partway through, you can end up with orphan children whose parent never materialized. Re-run the add — BullMQ deduplicates — but still watch for children with no parent. - A stalled child blocks the parent. The parent stays in waiting-children until every child completes, so one child that stalls (lock expired, worker died) parks the whole tree. Child queues need the same stalled-job hygiene as any other queue: a sensible
lockDuration, healthy workers, and alerting on stalled counts. If you are not sure what makes a job stall, read our guide on BullMQ stalled jobs. - Fan-out without a rate budget. If children call an external API, N children hitting it concurrently is N times the request rate — a limiter on the child queue is usually the difference between a successful batch and a 429 storm. Our guide on BullMQ rate limiting covers exactly this.
- Retention options change the deal. If you set
removeOnCompleteon child jobs, completed children (and their return values) may be cleaned up before the parent reads them. If the parent needs the aggregated results, keep completed children around until the tree finishes. - Priorities inside a tree. Children in the same queue are still subject to that queue's priority ordering, which can interleave unrelated jobs between your children — usually fine, but worth remembering when you tune priorities. See how priority() works before mixing it with flows.
Summary
BullMQ flows turn fan-out/fan-in batches into a first-class queue primitive: one parent job, many parallel children, automatic result aggregation, and a single completion point. Use them when a unit of work is really a tree of parallel sub-tasks with an aggregation step, keep fan-out bounded, and treat child queues with the same monitoring discipline as any other queue — stalled jobs, rate limits, and retention settings all apply. With a queue dashboard that renders the parent-child tree, batch pipelines stop being a mystery and become just another queue to observe.
Related Articles
BullMQ Worker Concurrency: How to Choose the Right Value
BullMQ worker concurrency decides how many jobs one worker runs at once — and the right value depends entirely on whether your jobs are I/O-bound or CPU-bound. Learn how the semaphore works, how to size it, and how to change it at runtime.
BullMQ Job Retention: removeOnComplete, removeOnFail, and Cleaning Up Redis
Without retention settings BullMQ keeps completed and failed jobs in Redis forever. Learn how job history is stored, how removeOnComplete and removeOnFail work, choosing retention per job class, and cleaning accumulated backlogs safely.
SQS Visibility Timeout: How Message Redelivery Works and How to Tune It
The SQS visibility timeout decides whether a slow consumer means the message waits or gets processed twice. Learn how the lease works under the hood, how to tune it against your real processing time, and how to spot redelivery failure modes before they become incidents.