Data-Comparison Script Repair
Repaired a multi-source data-comparison script, cutting test-seller noise with SDK-based filtering for reliable, scalable, low-maintenance results.
- Role
- Backend Engineer
- Timeline
- 2025
- Team
- 2
- Domain
- E-Commerce / Data Reconciliation
- Stack
- 6 tech
- Status
- Live
Overview
The Data-Comparison Script Repair improved a multi-source data comparison solution, focusing on accuracy, noise reduction, and operational stability. I owned the end-to-end technical design and planning, then delivered and validated the solution in production.
Gallery
A backend has no screens, so its "UI" is the operational surface: the comparison self-tests, the noise-filtered report, and post-release stability monitoring.
Problem & Solution
The existing comparison mechanism suffered from data noise — especially test-seller data — which undermined result reliability and downstream confidence. Any fix also had to remain scalable and low-maintenance as test data grew.
I analyzed the existing mechanism, identified test-seller data as the core noise source, and proposed multiple options with their long-term cost. I recommended adopting the test-seller SDK over config-based filtering, enabling automatic filtering without frequent rule updates. I implemented the comparison logic on the data-comparison platform using its Golang scripting model, reducing noise at the source rather than in post-processing.
Workflow
Three actors interact with the service; the diagrams are written in English to stay consistent across both languages.
Architecture
The data-comparison script fetches records from multiple sources and calls the test-seller SDK to filter test sellers before comparison, so inconsistencies reflect real data. Prometheus / Grafana provided post-release monitoring that confirmed no regressions or comparison anomalies.
Features
- Multi-source comparison design
- Noise reduction at the data source
- test-seller SDK filtering
- Self-testing across scenarios
- Post-release monitoring
Tech Stack
Engineering Challenges
test-seller data was the primary factor reducing comparison reliability. Fix: filter it at the source level rather than post-processing, so comparison logic stays simple and results stay trustworthy.
Config-based filtering would become expensive and error-prone as test data scaled. Fix: recommend and adopt the test-seller SDK for automatic filtering without frequent rule updates.
A fix for today could break as data grew. Fix: evaluate multiple options for long-term cost and maintenance, choosing an SDK-based approach that stays scalable as volumes rise.
Outcomes
Lessons Learned
Reducing noise at the source beats post-processing, and choosing a solution based on expected future scale — not just today's state — prevents rework. Evaluating trade-offs before committing kept the design scalable and low-maintenance.
Related Santekno Tutorials
Interested in working together?
Order, payment, or high-traffic event pipelines — let's talk architecture and trade-offs.