Santekno
Projects/Data-Comparison Script Repair
ByteDance logoPart of ByteDance · Software Engineer, Promotion Platform Team
Backend / APILive

Data-Comparison Script Repair

Repaired a multi-source data-comparison script, cutting test-seller noise with SDK-based filtering for reliable, scalable, low-maintenance results.

2025-dataeyes-compare-reconciliation · main
# E-Commerce / Data Reconciliation
$ go -X POST /2025-dataeyes-compare-reconciliation/v1/sync
→ 201 Created
{ "status": "ok" }
publish 2025-dataeyes-compare-reconciliation.synced
Role
Backend Engineer
Timeline
2025
Team
2
Domain
E-Commerce / Data Reconciliation
Stack
6 tech
Status
Live
01

Overview

The Data-Comparison Script Repair improved a multi-source data comparison solution, focusing on accuracy, noise reduction, and operational stability. I owned the end-to-end technical design and planning, then delivered and validated the solution in production.

A backend has no screens, so its "UI" is the operational surface: the comparison self-tests, the noise-filtered report, and post-release stability monitoring.

Test view — comparison self-test results
Comparison report — noise-filtered output
Ops view — post-release stability monitoring
03

Problem & Solution

Problem

The existing comparison mechanism suffered from data noise — especially test-seller data — which undermined result reliability and downstream confidence. Any fix also had to remain scalable and low-maintenance as test data grew.

Solution

I analyzed the existing mechanism, identified test-seller data as the core noise source, and proposed multiple options with their long-term cost. I recommended adopting the test-seller SDK over config-based filtering, enabling automatic filtering without frequent rule updates. I implemented the comparison logic on the data-comparison platform using its Golang scripting model, reducing noise at the source rather than in post-processing.

04

Workflow

Three actors interact with the service; the diagrams are written in English to stay consistent across both languages.

flowchart diagram

records

records

filter test data

Data Source A

Data-Comparison Script

Data Source B

test-seller SDK

Comparison Result

Use case interaction
sequence diagram
test-seller SDKComparison ScriptSource BSource Atest-seller SDKComparison ScriptSource BSource Afetch recordsfetch recordsis test seller?filter decisioncompare & report
Sequence flow
05

Architecture

flowchart diagram

metrics

Data Source A

Comparison Script · Go

Data Source B

test-seller SDK

Comparison Report

Prometheus / Grafana

System architecture
Architecture notes

The data-comparison script fetches records from multiple sources and calls the test-seller SDK to filter test sellers before comparison, so inconsistencies reflect real data. Prometheus / Grafana provided post-release monitoring that confirmed no regressions or comparison anomalies.

state diagram

SDK removes test data

cross-source compare

self-test

fetched

filtered

compared

validated

Comparison run lifecycle
06

Features

Comparison
  • Multi-source comparison design
  • Noise reduction at the data source
  • test-seller SDK filtering
Reliability
  • Self-testing across scenarios
  • Post-release monitoring
07

Tech Stack

language
Go
tooling
data-comparison platform
sdk
test-seller SDK
database
Managed RDS (relational database)
observability
PrometheusGrafana
08

Engineering Challenges

01
Data noise from test-seller data

test-seller data was the primary factor reducing comparison reliability. Fix: filter it at the source level rather than post-processing, so comparison logic stays simple and results stay trustworthy.

02
Avoiding costly config-based filtering

Config-based filtering would become expensive and error-prone as test data scaled. Fix: recommend and adopt the test-seller SDK for automatic filtering without frequent rule updates.

03
Designing for future data scale

A fix for today could break as data grew. Fix: evaluate multiple options for long-term cost and maintenance, choosing an SDK-based approach that stays scalable as volumes rise.

09

Outcomes

0
post-release comparison anomalies
2
solution options evaluated
SDK-based
filtering mechanism adopted
live
release status
10

Lessons Learned

Reducing noise at the source beats post-processing, and choosing a solution based on expected future scale — not just today's state — prevents rework. Evaluating trade-offs before committing kept the design scalable and low-maintenance.

11

Related Santekno Tutorials

Interested in working together?

Order, payment, or high-traffic event pipelines — let's talk architecture and trade-offs.