As backend architectures scale from a single monolith to distributed microservices or clustered Node.js instances behind a load balancer, race conditions become one of the hardest production bugs to diagnose. When two concurrent requests attempt to modify the same resource simultaneously—such as purchasing the last item in stock or processing duplicate wallet withdrawals—standard database transactions alone may fail if your services run across separate database sessions or independent server nodes.
This engineering runbook explores how to implement a robust Distributed Locking Mechanism using Redis, ensuring exclusive access to critical sections across distributed systems.
1. The Anatomy of a Distributed Race Condition
Consider an inventory check for a flash sale item with a stock count of 1:
Request A (Instance 1) ──► SELECT stock FROM items WHERE id = 5; (Returns 1) ──► UPDATE stock = 0 ──► SUCCESS
Request B (Instance 2) ──► SELECT stock FROM items WHERE id = 5; (Returns 1) ──► UPDATE stock = 0 ──► SUCCESS
Because both requests executed their read operations concurrently before either completed its write, both instances believed stock was available, resulting in an inventory oversell.
In distributed environments, you need a synchronization primitive that spans across all server nodes: a Distributed Lock.
2. The Mechanics of a Safe Redis Lock
Implementing a distributed lock using Redis requires satisfying three strict criteria:
- Mutual Exclusion: At any given time, only one client can hold a lock.
- Deadlock Free: A lock must eventually release even if the locking client crashes or loses connectivity (achieved via TTL).
- Safety on Release: A client must never release a lock held by another client.
Why Simple SETNX Fails Without Safety Tokens
If you only run SETNX lock:item 1 and your process hangs for longer than the lock TTL, the lock will expire automatically. Another process will acquire it, and when your original process finally resumes, it will delete the other process’s lock.
To prevent this, every lock acquisition must generate a unique random token (UUID), and release operations must use a Redis Lua script to verify ownership before deletion.
3. Production Implementation in TypeScript
Here is a robust, production-grade distributed lock manager using Redis and ioredis:
TypeScript
import Redis from "ioredis";
import { randomBytes } from "crypto";
export class DistributedLock {
constructor(private readonly redis: Redis) {}
/**
* Attempts to acquire a distributed lock.
* @param resourceKey Unique identifier for the locked resource
* @param ttlSeconds Lock expiration time to prevent deadlocks
* @returns Unique lock token if successful, or null if locked
*/
async acquire(resourceKey: string, ttlSeconds: number = 10): Promise<string | null> {
const lockKey = `lock:${resourceKey}`;
const token = randomBytes(16).toString("hex");
// SET command with NX (Not Exists) and EX (Expiration in seconds)
const result = await this.redis.set(lockKey, token, "EX", ttlSeconds, "NX");
return result === "OK" ? token : null;
}
/**
* Safely releases a distributed lock using an atomic Lua script.
*/
async release(resourceKey: string, token: string): Promise<boolean> {
const lockKey = `lock:${resourceKey}`;
// Lua script checks if the stored token matches before deleting
const luaScript = `
if redis.call("get", KEYS[1]) == ARGV[1] then
return redis.call("del", KEYS[1])
else
return 0
end
`;
const result = await this.redis.eval(luaScript, 1, lockKey, token);
return result === 1;
}
}
4. Executing Critical Sections Safely
Integrating the distributed lock into your business logic workflow requires handling acquisition failures gracefully—either by returning a retry response or queuing the request:
TypeScript
const redisClient = new Redis(process.env.REDIS_URL);
const locker = new DistributedLock(redisClient);
async function processOrderCheckout(userId: string, productId: string) {
const resourceKey = `product:checkout:${productId}`;
const token = await locker.acquire(resourceKey, 5); // 5-second lock TTL
if (!token) {
throw new Error("Resource is currently locked by another transaction. Please retry.");
}
try {
// === CRITICAL SECTION START ===
const stock = await db.query("SELECT stock FROM products WHERE id = $1", [productId]);
if (stock.rows[0].stock <= 0) {
throw new Error("Out of stock");
}
await db.query("UPDATE products SET stock = stock - 1 WHERE id = $1", [productId]);
await db.query("INSERT INTO orders (user_id, product_id) VALUES ($1, $2)", [userId, productId]);
// === CRITICAL SECTION END ===
} finally {
// Always release the lock in a finally block to prevent deadlocks during exceptions
await locker.release(resourceKey, token);
}
}
5. Architectural Gotchas and Operational Warnings
- Clock Drift and TTL Tuning: If your critical section takes longer to execute than your lock’s TTL, the lock will expire prematurely, breaking mutual exclusion. Always size your TTL generously relative to your p99 processing time.
- Single Redis Node vs. Redlock: For standard architectures, a primary-replica Redis setup with failover is sufficient. If your system requires extreme multi-node quorum consensus across independent Redis masters, consider implementing the official Redlock algorithm.
- Avoid Busy-Waiting (Spinlocks): Do not write loops that hammer Redis with
while(!(await locker.acquire(...))). Instead, combine distributed locking with a Message Queue or exponential backoff retry policy.
Implementing distributed locks eliminates race conditions across clustered microservices, safeguarding your core data integrity under heavy concurrent traffic.