Break the System: Failure Injection Lab
Inject failures into a live model of a production e-commerce system, watch one broken part spread to customers, then add resilience until it is healthy again.
Failure simulations
- Consistent Hashing Ring — Place keys on a hash ring, add or remove servers, toggle virtual nodes, and compare key movement with modulo hashing.
- Distributed Transaction: Two-Phase Commit — Coordinate Order, Payment and Inventory with 2PC; crash participants or the coordinator at each phase.
- Saga: Compensating a Failed Checkout — Run a checkout saga as orchestration or choreography; fail any step and watch compensations unwind it.
- Circuit Breaker States — Degrade a dependency and watch the breaker trip from closed to open to half-open, with and without fallbacks.
- Retry Storm & Backoff with Jitter — Slow a service down and watch naive retries multiply traffic; add backoff, jitter and retry budgets to recover.
- Load Balancing Algorithms — Send traffic through round robin, least connections, weighted and consistent-hash balancers; make one server slow.
- Microservice Failure Cascade — Kill Payment, Inventory or Notification in a checkout chain and add timeouts, retries, breakers, async events and sagas.
- Idempotency Keys & Duplicate Payments — Click “Pay” on a flaky network, retry, and see duplicate charges disappear once idempotency keys are enabled.
Patterns that keep the system alive
- Timeout — Never wait forever for a remote call.
- Retry with Backoff & Jitter — Retry transient failures with growing, randomised delays.
- Circuit Breaker — Stop calling a failing dependency and fail fast.
- Bulkhead — Isolate resources so one failure cannot sink everything.
- Rate Limiter — Cap how many requests a client or dependency can make.
- Graceful Degradation & Fallback — Serve a reduced experience instead of an error.
Go deeper
- Microservices Lab — Failures, retries, circuit breakers, sagas and idempotency.
- Why Microservices Exist — and When Not to Use Them — Level 9 · Microservices
- Synchronous vs Asynchronous Communication — Level 9 · Microservices
- Timeouts, Retries & Retry Storms — Level 9 · Microservices
- Circuit Breakers & Bulkheads — Level 9 · Microservices
- Idempotency: Making Retries Safe — Level 9 · Microservices
- Distributed Transactions: 2PC, Sagas & the Outbox — Level 9 · Microservices