Skip to main content

BullMQ in Production: Retries, Dead-Letter Queues and Monitoring for Node.js Jobs

Run Node.js background jobs with BullMQ in production: idempotent jobs, exponential retries, dead-letter queues, schedulers and graceful shutdown.

Amit Verma4 min readUpdated
Install Node.js using NVM

Last winter we woke up to a stuck invoice queue because a cron job and a manual retry script fought over the same Redis connection. Half the invoices were generated twice, the other half not at all. That morning pushed me to formalise how we run background jobs in Node.js.

BullMQ became the backbone. It is TypeScript-first, built on Redis, and gives you visibility into every stage of a job's life. But a queue library only gets you halfway — the rest is the set of habits around it. This post is that set of habits.

The mental model

BullMQ has three moving parts you actually touch:

  • Queue — where producers add jobs. Cheap to create, safe to use from your API handlers.
  • Worker — a long-running process that pulls jobs and runs your handler.
  • QueueEvents — a listener for global job events (completed, failed, stalled) across all workers.

If you learned BullMQ from older tutorials you may have seen a QueueScheduler. It was removed in BullMQ 2.0 — delayed jobs and stalled-job recovery are now handled by the workers themselves. If your code still creates one, delete it.

One shared connection config

Workers use blocking Redis commands, so the ioredis connection they use must set maxRetriesPerRequest: null. Put that in one place so nobody forgets it:

1// queues/connection.ts
2import type { ConnectionOptions } from 'bullmq'
3
4export const connection: ConnectionOptions = {
5  host: process.env.REDIS_HOST ?? '127.0.0.1',
6  port: Number(process.env.REDIS_PORT ?? 6379),
7  password: process.env.REDIS_PASSWORD,
8  maxRetriesPerRequest: null,
9}

Producing jobs: small payloads, stable IDs

Two rules removed most of our production incidents:

  1. Pass identifiers, not blobs. A job payload should be { invoiceId }, not the whole invoice. Small payloads keep Redis lean and mean a replayed job always reads the current state from the database.
  2. Give jobs a deterministic jobId. BullMQ will not add a second job with an ID that already exists in the queue. That single option is what would have saved us from the double-invoice morning.
1// queues/invoice.ts
2import { Queue } from 'bullmq'
3import { connection } from './connection'
4
5export type InvoiceJob = { invoiceId: string }
6
7export const invoiceQueue = new Queue<InvoiceJob>('invoice', {
8  connection,
9  defaultJobOptions: {
10    attempts: 5,
11    backoff: { type: 'exponential', delay: 5_000 }, // 5s, 10s, 20s, 40s…
12    removeOnComplete: { age: 24 * 3600, count: 1_000 },
13    removeOnFail: { age: 7 * 24 * 3600 },
14  },
15})
16
17export async function enqueueInvoice(invoiceId: string) {
18  await invoiceQueue.add('generate', { invoiceId }, { jobId: `invoice-${invoiceId}` })
19}

removeOnComplete and removeOnFail matter more than they look. Without them, every finished job stays in Redis forever and your memory graph turns into a ramp.

Consuming jobs: idempotent handlers

Retries mean your handler will run more than once for the same job — after a crash, a timeout, or a deploy. Design for it:

1// workers/invoice.worker.ts
2import { Worker, UnrecoverableError, type Job } from 'bullmq'
3import { connection } from '../queues/connection'
4import type { InvoiceJob } from '../queues/invoice'
5import { db } from '../db'
6import { renderAndEmailInvoice } from '../invoices'
7
8async function handle(job: Job<InvoiceJob>) {
9  const invoice = await db.invoice.findUnique({ where: { id: job.data.invoiceId } })
10
11  // Retrying will never fix a missing record — fail fast, skip remaining attempts.
12  if (!invoice) throw new UnrecoverableError(`Invoice ${job.data.invoiceId} not found`)
13
14  // Idempotency guard: a previous attempt may have finished the work.
15  if (invoice.sentAt) return { skipped: true }
16
17  await job.updateProgress(50)
18  await renderAndEmailInvoice(invoice)
19  await db.invoice.update({ where: { id: invoice.id }, data: { sentAt: new Date() } })
20
21  return { sent: true }
22}
23
24export const invoiceWorker = new Worker<InvoiceJob>('invoice', handle, {
25  connection,
26  concurrency: 10,
27  limiter: { max: 50, duration: 1_000 }, // respect the email provider's rate limit
28})

Notice the split between errors. A normal throw means "try again later" and uses the backoff. UnrecoverableError means "this will never work" and moves the job straight to failed. Getting that distinction right is the difference between a queue that heals itself and one that burns retries on garbage.

A dead-letter queue in 20 lines

BullMQ doesn't ship a dead-letter queue, but building one is simple. When a job has used all its attempts (or failed unrecoverably), copy it to a separate queue that humans look at:

1// workers/dead-letter.ts
2import { Queue } from 'bullmq'
3import { connection } from '../queues/connection'
4import { invoiceWorker } from './invoice.worker'
5
6const deadLetters = new Queue('dead-letter', { connection })
7
8invoiceWorker.on('failed', async (job, err) => {
9  if (!job) return
10  const outOfAttempts = job.attemptsMade >= (job.opts.attempts ?? 1)
11  if (!outOfAttempts && err.name !== 'UnrecoverableError') return
12
13  await deadLetters.add('invoice', {
14    queue: 'invoice',
15    jobId: job.id,
16    data: job.data,
17    reason: err.message,
18    failedAt: new Date().toISOString(),
19  })
20})

Now "what failed overnight?" is one dashboard view instead of a log search, and replaying a fixed job is a one-liner.

Scheduled jobs without cron fights

Our original bug was a system cron and the app both touching the same queue. BullMQ's job schedulers replace that. upsertJobScheduler is idempotent, so every instance of your app can call it on boot and you still get exactly one schedule:

1await invoiceQueue.upsertJobScheduler(
2  'nightly-reconcile',
3  { pattern: '0 2 * * *' }, // 02:00 every night
4  { name: 'reconcile', data: { invoiceId: '*' } },
5)

Graceful shutdown

On every deploy your platform sends SIGTERM. If the worker just dies, active jobs become "stalled" and get picked up again later — safe if your handlers are idempotent, but slow and noisy. Close the worker so it finishes what it's doing first:

1async function shutdown() {
2  await invoiceWorker.close() // waits for active jobs to finish
3  process.exit(0)
4}
5process.on('SIGTERM', shutdown)
6process.on('SIGINT', shutdown)

Health checks that actually page you

We poll queue depth every minute and alert on trends, not single values:

1const counts = await invoiceQueue.getJobCounts('waiting', 'active', 'delayed', 'failed')
2metrics.gauge('queue.invoice.waiting', counts.waiting)
3metrics.gauge('queue.invoice.failed', counts.failed)

Three alerts cover most problems:

  • Waiting keeps growing for 10+ minutes — workers are down or too slow. Scale up, or route traffic to a read-only mode.
  • Failed jumps suddenly — a bad deploy or an upstream outage. Check the dead-letter queue.
  • Active stuck at zero while waiting is above zero — workers lost their Redis connection.

For a visual view, Bull Board mounts as a small admin UI inside Express, Fastify or Next.js, so you can browse and retry jobs.

Checklist

  • Payloads hold IDs, not documents.
  • Every job that must run once has a deterministic jobId.
  • Handlers are idempotent; permanent errors throw UnrecoverableError.
  • removeOnComplete / removeOnFail are set.
  • Failed jobs land in a dead-letter queue.
  • Schedules use upsertJobScheduler, not OS cron.
  • Workers close gracefully on SIGTERM.
  • Queue depth is graphed and alerted on.

None of this is clever. It's a small amount of ceremony — and it has saved us from the late-night panic that used to follow every seasonal traffic spike.

Share this article

Found it helpful? Share it with your network.