VISEN GROUP
Technology & Engineering

Mastering Apache Kafka: Consumer Groups, Offset Commit Strategies, and Rebalance Management

November 10, 20263 min readUncategorized

In high-throughput event-driven systems, Apache Kafka acts as the foundational message backbone. However, improper configuration of consumer groups, partition assignments, and commit strategies leads to severe operational bottlenecks: duplicate event processing, message loss, or infinite rebalance storms.

This engineering runbook explores advanced Kafka consumer architecture and reliable offset management.

1. The Anatomy of Kafka Consumer Groups

Kafka topics are split into multiple Partitions, allowing horizontal scaling of message consumers. A Consumer Group is a set of independent worker instances cooperating to consume data from a topic.

  • The Core Rule: Each partition within a topic is consumed by exactly one consumer within a given consumer group at any given time.
  • Scaling Limit: If your topic has 8 partitions, scaling your consumer group beyond 8 active worker instances yields zero benefit; the extra instances remain idle (“standby”).

2. Offset Commit Strategies: At-Least-Once vs At-Most-Once

Choosing when and how to commit message offsets dictates your data integrity guarantees:

  1. Auto-Commit (At-Most-Once): The consumer client automatically commits offsets periodically in the background. If your worker crashes after committing the offset but before finishing business logic processing, the event is permanently lost. Avoid for financial or critical data.
  2. Manual Asynchronous/Synchronous Commit (At-Least-Once): The worker processes the business logic completely, updates the database, and only then explicitly commits the offset. If the worker crashes before commit, Kafka redelivers the message to another consumer upon restart.

3. Production Consumer Implementation in Node.js (KafkaJS)

TypeScript

import { Kafka, Consumer, EachMessagePayload } from "kafkajs";

const kafka = new Kafka({
  clientId: "order-processing-service",
  brokers: ["kafka-broker-1:9092", "kafka-broker-2:9092"],
});

const consumer: Consumer = kafka.consumer({ 
  groupId: "order-fulfillment-group",
  sessionTimeout: 30000,
  heartbeatInterval: 3000,
});

export async function runOrderConsumer() {
  await consumer.connect();
  await consumer.subscribe({ topic: "orders.created", fromBeginning: false });

  await consumer.run({
    // Disable auto-commit to guarantee At-Least-Once delivery
    autoCommit: false,
    eachMessage: async ({ topic, partition, message }: EachMessagePayload) => {
      const messageValue = message.value?.toString();
      if (!messageValue) return;

      const orderEvent = JSON.parse(messageValue);

      try {
        // 1. Execute Core Business Logic
        await processFulfillmentWorkflow(orderEvent);

        // 2. Manually commit offset ONLY after successful processing
        await consumer.commitOffsets([
          {
            topic,
            partition,
            offset: (BigInt(message.offset) + 1n).toString(),
          },
        ]);
      } catch (error) {
        console.error(`Failed to process message offset ${message.offset}:`, error);
        // Implement Dead Letter Queue (DLQ) routing or exponential retry backoff here
      }
    },
  });
}

4. Handling Rebalance Storms

A Rebalance occurs when a consumer joins or leaves the group, triggering Kafka to reassign partitions across active workers. During rebalancing, message consumption pauses temporarily.

Best Practices to Prevent Rebalance Storms:

  • Tune session.timeout.ms and max.poll.interval.ms: If your database batch processing takes longer than max.poll.interval.ms, Kafka assumes the worker is dead and triggers a rebalance. Set this generously high for heavy workloads.
  • Cooperative Sticky Assignor: Configure your consumer groups to use the CooperativeStickyAssignor protocol (partitionAssignor: [PartitionAssignors.cooperativeSticky]). This minimizes partition movement, allowing stable consumers to continue processing unaffected partitions during rebalancing.

Leave a Reply

Your email address will not be published. Required fields are marked *