Abstract
The PC Tracker Backend is the collection and persistence layer for the PartsRadar price-tracking platform. It runs scheduled crawlers against retailer listing pages, normalizes the results into canonical product records, and persists them in SQL with Redis for fast runtime state. The core engineering problem is keeping repeated refreshes idempotent so price history accumulates rather than duplicating.
1. What This Is
PC Tracker Backend is the closed-source backend component of the 2026 PartsRadar project set. It sits at the front of the pipeline: it owns the "get data from retailers" half, while a separate classification and analytics layer handles the rest.
The implementation is not publicly available. No matching repository was found in the reviewed GitHub account, so the architecture described here is drawn from the portfolio notes rather than source code.
2. How It Works
A scheduler triggers a refresh cycle for each configured retailer. The crawler fetches and parses listing pages, the normalizer maps raw fields into a consistent backend representation, and the results flow into SQL for durable storage and Redis for fast runtime access. Downstream, PartsRadar search, charts, deals, and alerting read from the refreshed data.
| # | Stage | Input | Tool | Output |
|---|---|---|---|---|
| 01 | Scheduled trigger | Configured retailer list | Scheduling layer | Refresh job per retailer |
| 02 | Fetch and parse | Retailer listing pages | Python crawler | Raw listing records |
| 03 | Normalize | Raw records | Pipeline normalizer | Canonical product / price / availability fields |
| 04 | Classify | Normalized naming data | Entity-resolution layer | Resolved product entities |
| 05 | Persist | Canonical records | SQL + Redis | Durable history + runtime state |
| 06 | Expose | Persisted data | PartsRadar API | Search, charts, deals, alerts |
3. Implementation Notes
3.1 Idempotent refresh
Because the crawler runs on a schedule, the same product can be scraped many times. The normalization step must produce a stable identity so a refresh updates the existing record and appends a price-history row rather than inserting an unrelated duplicate. This is the main correctness concern for the whole pipeline.
3.2 Crawling decoupled from presentation
Retailer-specific extraction logic is isolated from the classification and UI layers. If a retailer changes its page layout, only the per-retailer parser needs updating; the downstream SQL schema and the PartsRadar frontend are unaffected.
3.3 Redis for runtime state
Redis handles fast cache and state access around the data backend — likely crawl cursors, rate-limit counters, or short-lived session data — keeping the SQL layer focused on durable product and price-history records.
4. Constraints
-
Closed source, no public repo
The implementation is not publicly available. Architecture details are inferred from portfolio notes, so specific framework choices, schema layouts, and error-handling strategies are unverified.
-
No visible resilience layer
The notes do not mention retry logic, backoff, or rate-limiting for retailer endpoints. A single failing retailer could stall or corrupt a refresh cycle without a documented recovery path.
-
No test or CI coverage noted
There is no mention of unit tests, integration tests, or a CI pipeline. The idempotent-refresh guarantee rests on the normalization logic alone, with no automated verification described.
-
Single-deployment scope
The architecture implies a single backend instance per deployment. There is no mention of horizontal scaling, queue-based fan-out, or multi-region concerns.
5. Next
- a. Add per-retailer retry with exponential backoff and a circuit breaker so one failing source does not block the full refresh cycle.
- b. Introduce an explicit deduplication key (e.g., normalized SKU or hash of name + vendor) checked before SQL insert to enforce the idempotent-refresh guarantee at the database level.
- c. Add integration tests for the fetch → normalize → persist path using fixture HTML, so parser regressions are caught before deployment.
— end of report —