VISEN GROUP
Technology & Engineering

Engineering Reliable Background Jobs: Mastering Worker Queues, Idempotency, and Exponential Backoff

November 10, 20262 min readUncategorized

Not all tasks belong in the synchronous HTTP request-response cycle. Heavy operations—such as generating PDF invoices, processing video transcodes, or dispatching webhook batches—must be offloaded to asynchronous background worker queues (BullMQ, Celery, RabbitMQ).

However, asynchronous queues introduce failure modes: network dropouts, third-party API rate limits, and poisoned messages. This runbook covers robust background job orchestration.

1. The Anatomy of Reliable Job Processing

A resilient background worker architecture guarantees three core properties:

  1. At-Least-Once Execution: Jobs are persisted in a durable data store (Redis / PostgreSQL) so they survive worker container crashes.
  2. Exponential Backoff with Jitter: When an external API fails, retrying immediately floods the failing service. Backoff algorithms introduce exponentially increasing delays between retries.
  3. Dead Letter Queues (DLQ): After max retry attempts fail, jobs are quarantined in a DLQ for manual inspection rather than looping forever.

2. Production BullMQ Worker Implementation in TypeScript

Here is a robust background worker setup using BullMQ (powered by Redis) featuring automatic retries and exponential backoff:

TypeScript

import { Worker, Job, Queue } from "bullmq";

const connection = { host: "localhost", port: 6379 };

// 1. Initialize Notification Queue
export const notificationQueue = new Queue("notification-processing", { connection });

// 2. Define Resilient Worker
const notificationWorker = new Worker(
  "notification-processing",
  async (job: Job) => {
    const { userId, templateId, payload } = job.data;
    console.log(`Processing job ${job.id} for user ${userId} (Attempt ${job.attemptsMade + 1})`);

    // Simulate external third-party email provider call that might fail
    await sendEmailViaProvider(userId, templateId, payload);
  },
  {
    connection,
    concurrency: 10, // Process up to 10 jobs concurrently per worker instance
    limiter: {
      max: 50,       // Max 50 jobs per duration to respect third-party rate limits
      duration: 1000, // Per 1 second
    },
  }
);

// Event listeners for monitoring lifecycle
notificationWorker.on("completed", (job) => {
  console.log(`Job ${job.id} completed successfully.`);
});

notificationWorker.on("failed", (job, err) => {
  console.error(`Job ${job?.id} failed with error: ${err.message}. Attempts made: ${job?.attemptsMade}`);
});

3. Enqueuing Jobs with Exponential Backoff Strategy

TypeScript

export async function dispatchEmailJob(userId: string, templateId: string, payload: any) {
  await notificationQueue.add(
    "send-email",
    { userId, templateId, payload },
    {
      attempts: 5, // Retry up to 5 times before failing permanently
      backoff: {
        type: "exponential",
        delay: 5000, // Starts at 5s, then 10s, 20s, 40s...
      },
      removeOnComplete: true, // Clean up Redis memory on success
      removeOnFail: false,    // Retain failed jobs in DLQ for inspection
    }
  );
}

Leave a Reply

Your email address will not be published. Required fields are marked *