VISEN GROUP
Technology & Engineering

Building Resilient Microservices: Implementing Circuit Breakers to Prevent Cascading Failures

November 10, 20263 min readUncategorized

In a complex microservices topology, a localized database slowdown or third-party API outage can easily cascade across your entire infrastructure. When upstream services keep firing requests to an unresponsive downstream service, their connection pools saturate, memory spikes, and threads block—turning a single component failure into a total system outage.

This engineering runbook explores how to implement the Circuit Breaker Pattern to isolate failing dependencies, fail fast, and gracefully recover.

1. The Three States of a Circuit Breaker

A robust circuit breaker operates similarly to an electrical circuit breaker, cycling through three distinct states:

          [ Success Threshold Reached ]
          ┌───────────────────────────┐
          │                           │
          ▼                           │
    ┌───────────┐               ┌───────────┐
    │  CLOSED   │ ──(Failures)─►│   OPEN    │
    │ (Healthy) │               │ (Failing) │
    └───────────┘               └───────────┘
          ▲                           │
          │                           ▼
          │                   [ Timeout Expires ]
          │                           │
          │                      ┌───────────┐
          └───────────────────── │ HALF-OPEN │
           [ Test Probe Succeeds ] (Testing) │
                                 └───────────┘
  1. Closed (Healthy State): All requests pass through to the downstream service. The breaker monitors error rates (e.g., HTTP 5xx or connection timeouts). If failures exceed a configured threshold (e.g., 50% failure over 10 requests), the circuit trips.
  2. Open (Failing State): All incoming requests fail instantly without touching the external dependency (Fast-Fail), returning a default fallback response or cached data. This gives the downstream service time to recover.
  3. Half-Open (Testing State): After a cooldown period (e.g., 30 seconds), the breaker allows a single test request through. If successful, the circuit resets to Closed. If it fails, it returns to Open.

2. Production Implementation in TypeScript

Here is a robust, production-grade Circuit Breaker class implementing failure counting, state transitions, and fallback handling:

TypeScript

export enum CircuitState {
  CLOSED,
  OPEN,
  HALF_OPEN,
}

export interface CircuitBreakerOptions {
  failureThreshold: number; // Number of failures before tripping
  successThreshold: number; // Successes required to close from half-open
  cooldownPeriodMs: number; // Time to stay open before trying half-open
}

export class CircuitBreaker {
  private state: CircuitState = CircuitState.CLOSED;
  private failureCount: number = 0;
  private successCount: number = 0;
  private nextAttemptTimestamp: number = Date.now();

  constructor(
    private readonly actionName: string,
    private readonly options: CircuitBreakerOptions
  ) {}

  async execute<T>(action: () => Promise<T>, fallback: () => Promise<T>): Promise<T> {
    if (this.state === CircuitState.OPEN) {
      if (Date.now() >= this.nextAttemptTimestamp) {
        this.transitionTo(CircuitState.HALF_OPEN);
      } else {
        console.warn(`[CircuitBreaker] '${this.actionName}' is OPEN. Executing fallback.`);
        return await fallback();
      }
    }

    try {
      const result = await action();
      this.onSuccess();
      return result;
    } catch (error) {
      this.onFailure();
      console.error(`[CircuitBreaker] Action '${this.actionName}' failed. Falling back.`);
      return await fallback();
    }
  }

  private onSuccess() {
    this.failureCount = 0;
    if (this.state === CircuitState.HALF_OPEN) {
      this.successCount++;
      if (this.successCount >= this.options.successThreshold) {
        this.transitionTo(CircuitState.CLOSED);
      }
    }
  }

  private onFailure() {
    this.failureCount++;
    if (this.state === CircuitState.HALF_OPEN || this.failureCount >= this.options.failureThreshold) {
      this.transitionTo(CircuitState.OPEN);
      this.nextAttemptTimestamp = Date.now() + this.options.cooldownPeriodMs;
    }
  }

  private transitionTo(newState: CircuitState) {
    console.log(`[CircuitBreaker] State transition for '${this.actionName}': ${CircuitState[this.state]} ──► ${CircuitState[newState]}`);
    this.state = newState;
    if (newState === CircuitState.CLOSED) {
      this.failureCount = 0;
      this.successCount = 0;
    }
  }
}

3. Usage Example: Calling an External Payment Gateway

TypeScript

const paymentCircuit = new CircuitBreaker("StripePaymentAPI", {
  failureThreshold: 5,
  successThreshold: 2,
  cooldownPeriodMs: 15000, // 15 seconds
});

async function processCheckout(orderId: string, amount: number) {
  return await paymentCircuit.execute(
    // Primary Action
    async () => {
      return await stripeClient.charges.create({ amount, currency: "usd" });
    },
    // Fallback Action
    async () => {
      // Log failed external attempt and queue for asynchronous offline processing
      await queueOfflinePayment(orderId, amount);
      return { status: "QUEUED_FOR_OFFLINE_PROCESSING", orderId };
    }
  );
}

Leave a Reply

Your email address will not be published. Required fields are marked *