Chaos Engineering — Resilience Game Days
Prepared and ran a chaos engineering game day for the merchant-promotion service — dependency fault injection with toxiproxy across staging and production, keeping success rate near 99% with MTTA 1m / MTTR 12m, and turning one non-standardized error message into a cross-team follow-up with product and UI/UX.
- Role
- Backend Engineer / Reliability
- Timeline
- 2022
- Team
- 3
- Domain
- E-Commerce / Reliability & Chaos Engineering
- Stack
- 9 tech
- Status
- Live
Overview
The Chaos Engineering — Resilience Game Days project prepared and executed a chaos engineering game day for the merchant-promotion service, focused on how the service behaves when its external dependencies misbehave. As backend engineer on the reliability track, I owned the game-day plan, the fault-injection setup with toxiproxy, and the follow-up with product and UI/UX. The game day was executed in December 2022 with no blocking flow failed.
Gallery
A backend has no screens, so its "UI" is the operational surface: the game-day runbook, the toxiproxy fault-injection configuration, and the reliability dashboards watched during the run.
Problem & Solution
The merchant-promotion service was already operating at roughly a 99% success rate with 100% monitoring coverage and low MTTA (1m) / MTTR (12m), but its resilience to dependency failure — latency spikes, timeouts, and disconnects on downstream RPC calls — had not been proven end-to-end. Without a deliberate chaos exercise, unknown failure modes would only surface during real incidents, where the blast radius and recovery cost are much higher.
I designed the game day around toxiproxy as the fault-injection proxy sitting in front of each external dependency, so latency, timeout, and disconnect scenarios could be applied per dependency and reverted on demand. The scenarios were rehearsed in staging first, then executed in production under a runbook with explicit rollback triggers and on-call hand-off. Success rate, latency, and MTTA / MTTR were watched live on Prometheus / Grafana. The result: no significant issues, no blocking flow failed, and the service stayed up throughout the exercise. The only failing test case was caused by a non-standardized error message in one flow; instead of hot-patching, I raised it with product and UI/UX and scheduled the fix into the next sprint.
Workflow
Three actors interact with the service; the diagrams are written in English to stay consistent across both languages.
Architecture
The service under test is the merchant-promotion service (Go, RPC framework) with a Managed RDS and Redis behind it, and multiple external dependencies reached over RPC — a catalog service, a loyalty/points service, and a shop-info service. During the game day, calls to those dependencies were routed through toxiproxy, which applied controlled latency, timeout, and disconnect faults. Prometheus / Grafana captured success rate, latency, and MTTA / MTTR signals in real time so the reliability team could compare behavior against the pre-run baseline and stop the run on drift.
Features
- Game-day plan and success criteria for the merchant-promotion service
- Dependency mapping across upstream and downstream services
- Runbook with rollback triggers and on-call hand-off
- toxiproxy setup in front of external dependencies
- Latency, timeout, and disconnect scenarios per dependency
- Fault-injection coverage across staging and production
- Live monitoring against MTTA 1m / MTTR 12m targets
- Post-run review of failing test cases and gaps
- Cross-functional follow-up with product and UI/UX
Tech Stack
Engineering Challenges
External RPC calls needed to be exercised under latency, timeout, and disconnect conditions without corrupting real state or leaking chaos beyond the target service. Fix: put toxiproxy in front of each dependency, script the fault matrix per dependency, and gate every scenario behind explicit runbook approval so faults were reversible on demand.
The service had to keep serving traffic near a 99% success rate while faults were being injected in production. Fix: rehearse the full scenario set in staging first, run production with tight guardrails and pre-agreed abort criteria, and watch success rate, latency, and MTTA/MTTR live on Prometheus / Grafana so the run could be stopped on the first meaningful drift.
The game day found one failing test case caused by an error message that was not standardized across the flow — a bug real traffic would rarely expose. Fix: capture the finding as a concrete action item, align with product and UI/UX on the standard error contract, and schedule the fix into the next sprint instead of hot-patching it during the run.
Outcomes
Lessons Learned
A well-scoped chaos exercise is worth more than a long incident review after the fact: running the game day against real downstream dependencies with toxiproxy produced 0 blocking failures while keeping MTTA 1m / MTTR 12m intact across staging + production. The most useful finding was not a crash but a small correctness gap — a non-standardized error message — which is exactly the class of issue chaos engineering is meant to surface, and exactly the kind that is cheaper to fix through a normal sprint follow-up than under incident pressure.
Related Santekno Tutorials
Interested in working together?
Order, payment, or high-traffic event pipelines — let's talk architecture and trade-offs.