Monitoring & Alerting — New Relic + PagerDuty
Rolled out monitoring and alerting for the merchant-promotion service on New Relic and integrated PagerDuty for on-call paging — reaching 100% monitoring coverage with MTTA 1m / MTTR 12m and emergency on-call access, supporting a 99% service success rate.
- Role
- Backend Engineer / Reliability
- Timeline
- 2022
- Team
- 3
- Domain
- E-Commerce / Observability & On-Call
- Stack
- 10 tech
- Status
- Live
Overview
The Monitoring & Alerting — New Relic + PagerDuty initiative rolled out an end-to-end observability and on-call stack for the merchant-promotion service. As backend engineer on the reliability track, I owned the New Relic instrumentation, alert policies, and PagerDuty integration, plus the emergency on-call access story used during maintenance and incidents. The result was 100% monitoring coverage on the service, with MTTA 1m / MTTR 12m and a 99% success rate sustained through the H1–H2 2022 window.
Gallery
A backend has no screens, so its "UI" is the operational surface: New Relic dashboards and alert policies, the PagerDuty on-call rotation, and the runbook engineers reach for at 3 a.m.
Problem & Solution
The merchant-promotion service was already serving high traffic across critical promotion and voucher flows, but its monitoring and paging story was uneven: dashboards existed in pieces, alerts did not consistently reach a human, and there was no standard, safe path for on-call engineers to intervene during maintenance. Without unified monitoring and a reliable paging channel, real degradations risked being detected late — and every extra minute of MTTA/MTTR would eat directly into the service's success rate.
I wired the merchant-promotion service into New Relic for APM and custom metrics, then defined alert policies around the signals that actually predict user impact — error rate, latency, and overall success rate — instead of one-metric thresholds. New Relic incidents were forwarded into PagerDuty, which owned the on-call rotation and escalation paths and delivered pages by push, call, and SMS so the first responder was reached quickly enough to keep MTTA around 1 minute. For maintenance and mitigation, I standardized on emergency on-call access driven by runbooks so on-call engineers could act fast without loosening baseline access rules. Together this brought monitoring coverage to 100% with MTTR ~ 12 minutes, backing the service's 99% success rate.
Workflow
Three actors interact with the service; the diagrams are written in English to stay consistent across both languages.
Architecture
The merchant-promotion service (Go, RPC framework) exports APM traces and custom metrics to New Relic, which owns alert policy evaluation, and continues to surface secondary dashboards on Prometheus / Grafana. When a policy breaches, New Relic raises an incident to PagerDuty, which pages the correct rotation, handles escalation timers, and records the acknowledge / resolve timeline that feeds MTTA / MTTR. On-call engineers use emergency on-call access to reach the service during maintenance or mitigation, following the alert-specific runbook rather than an ad-hoc path. Managed RDS and Redis sit behind the service and inherit the same monitoring surface indirectly through the service's own health signals.
Features
- New Relic APM wired into the merchant-promotion service
- Custom metrics for error rate, latency, and success rate
- Alert policies aligned with MTTA / MTTR targets
- PagerDuty integrated as the paging channel from New Relic
- On-call rotation and escalation paths for every alert policy
- Push, call, and SMS notifications routed through PagerDuty
- Emergency on-call access for maintenance windows
- Runbook per alert policy with clear mitigation steps
- Post-incident review loop feeding dashboards and thresholds
Tech Stack
Engineering Challenges
Every critical path in the merchant-promotion service needed a signal in New Relic, but adding raw thresholds everywhere would have drowned the on-call rotation in noise. Fix: mapped critical endpoints and dependencies first, then defined alert policies that combined error rate, latency, and success rate so each page corresponded to a real, actionable degradation instead of a single-metric blip.
A one-minute MTTA target is only credible if the path from signal to on-call engineer is short and deterministic. Fix: forwarded New Relic incidents straight into PagerDuty, tuned incident severities and escalation timers, and enabled push, call, and SMS channels so the first responder was reached quickly and pages could not sit unacknowledged.
Incident response and planned maintenance both needed a fast, controlled way for on-call engineers to reach the service without weakening day-to-day access rules. Fix: standardized on emergency on-call access with runbook-driven usage, scoped to on-call responders during active incidents or maintenance, so mitigation stayed quick without becoming the default path for regular work.
Outcomes
Lessons Learned
Good observability is less about capturing every metric and more about paging the right person, fast, when a real degradation appears. Pairing New Relic alert policies with PagerDuty on-call paging — and standardizing on emergency on-call access for maintenance — was what actually made 100% monitoring coverage useful: it kept MTTA around 1 minute and MTTR around 12 minutes, and it protected the service's 99% success rate through H1–H2 2022. The single biggest lesson: alerts should be defined by the incident they represent, not by the metric they read.
Related Santekno Tutorials
Interested in working together?
Order, payment, or high-traffic event pipelines — let's talk architecture and trade-offs.