Graceful Handling for External Calls — Dependency Resilience
Implemented graceful handling — timeouts, fallbacks, and graceful degradation — for every external call the merchant-promotion service depends on, and rolled out a Redis caching layer across repo functions that grew coverage from 97 to 195 functions, reducing external RPS cost and keeping the service healthy when dependencies degrade.
- Role
- Backend Engineer
- Timeline
- 2022
- Team
- 3
- Domain
- E-Commerce / Reliability & Dependency Resilience
- Stack
- 8 tech
- Status
- Live
Overview
The Graceful Handling for External Calls initiative made the merchant-promotion service resilient to failures in every external dependency it talks to — a catalog service, a loyalty/points service, a shop-info service, and other internal RPC dependencies. As backend engineer on the reliability track, I owned the resilient RPC-client contract, the fallback and graceful-degradation rules, and the Redis caching layer that reduced external RPS cost across the repo.
Gallery
A backend has no screens, so its "UI" is the operational surface: the resilient RPC-client contract, the caching/timeout/fallback configuration, and the reliability dashboards that show reduced external RPS and stable success rate.
Problem & Solution
The merchant-promotion service depended on several external services over RPC, and its behavior under dependency stress was uneven: slow or failing dependencies could inflate latency, drive up the service's error rate, and raise external RPS cost through repeated calls for slow-changing data. Without a uniform resilience contract, each call site had its own timeout, retry, and error-handling shape — making it hard to reason about how the service should degrade when one dependency misbehaves.
I rolled out a resilient RPC-client contract used everywhere the merchant-promotion service calls an external dependency: per-call timeouts, a fallback / graceful-degradation path per dependency, and circuit-breaker-style guarding so a repeatedly failing dependency stops dragging the whole request path down. In front of external reads I added a Redis caching layer, growing coverage from 97 to 195 repo functions with per-key TTLs tuned to hot vs. cold reads. Metrics for external RPS, error rate, and cache hit ratio were wired into Prometheus / Grafana so the rollout could be watched live, and unit test coverage on the touched paths was kept near 91%.
Workflow
Three actors interact with the service; the diagrams are written in English to stay consistent across both languages.
Architecture
The service under change is the merchant-promotion service (Go, RPC framework) that fans out to several external dependencies via RPC — a catalog service, a loyalty/points service, a shop-info service, and other internal RPC dependencies. Each dependency is now reached through a resilient client wrapper that enforces a timeout, checks a Redis caching layer first, and, on error or timeout, executes a fallback that returns a degraded value instead of failing the request. Prometheus / Grafana surface external RPS, error rate, and cache hit ratio so both the RPS reduction and the graceful-degradation behavior are directly observable.
Features
- Timeouts on every external RPC call
- Fallback paths with graceful degradation per dependency
- Circuit-breaker-style guarding for repeatedly failing dependencies
- Redis caching layer in front of external reads
- Coverage grown from 97 to 195 repo functions
- Per-key TTLs tuned to reduce external RPS cost
- Unified resilient RPC-client contract across the codebase
- Metrics for RPS, error rate, and cache hit ratio on Prometheus / Grafana
- Sustained roughly 91% unit test coverage on the resilient paths
Tech Stack
Engineering Challenges
The merchant-promotion service called several external dependencies over RPC — a catalog service, a loyalty/points service, a shop-info service, and other internal RPC dependencies — and any of them going slow could drag the whole request path down. Fix: standardize a resilient RPC-client wrapper that enforces per-call timeouts, applies a fallback / graceful-degradation path on error, and prevents a single degraded dependency from cascading into the service.
Repeated calls to the same external dependencies were driving up RPS cost with little benefit for data that changes slowly. Fix: introduce a Redis caching layer in front of external reads, grow coverage from 97 to 195 repo functions, and tune per-key TTLs so hot reads land on cache while writes and short-TTL keys keep freshness where it matters.
The change touched a large surface of the merchant-promotion service and had to land without regressions on live traffic. Fix: introduce the resilient client and caching hooks incrementally, keep unit test coverage near 91% on the touched paths, and watch Prometheus / Grafana for external RPS, error rate, and cache hit ratio during rollout so any regression could be caught early.
Outcomes
Lessons Learned
Graceful handling is more valuable as a shared contract than as a set of one-off patches: unifying timeout, fallback, and cache behavior across every external call made it far easier to keep the service healthy when a dependency degrades — and the Redis caching layer alone, grown from 97 → 195 covered functions, cut external RPS↓ meaningfully while leaving the service resilient to dependency failure. The biggest lesson is that the win comes from consistency: the same resilience shape everywhere is what lets you reason about degradation, and Prometheus / Grafana turn that consistency into a signal you can actually watch during rollout.
Related Santekno Tutorials
Interested in working together?
Order, payment, or high-traffic event pipelines — let's talk architecture and trade-offs.