Abstract
PartsRadar is a full-stack PC-component price-intelligence platform for the Indian market. It scrapes listings from 15+ retailers, resolves noisy product names into canonical entities via embedding-based classification, and serves search, price history, and vendor analytics through a Flask dashboard. The portfolio snapshot captured 21,440 products across 50,938 listings with a seven-hour refresh cadence.
1. What This Is
A closed-source, full-stack data project. The platform collects PC-part listings from multiple Indian retailers, normalizes inconsistent product titles into comparable entities, persists price observations, and exposes a public search-and-comparison interface with deal signals and category/vendor analytics.
I treated the data pipeline and the frontend as two sides of one system: analytics are generated from normalized data rather than raw scraped HTML being rendered directly. The architecture graph is Retailers → Scrapers → Clean Names → Embedding Classifier → SQL + Redis → Charts + Alerts → Dashboard.
2. How It Works
The pipeline is structured as six sequential stages, each independently testable:
| # | Stage | Input | Tool | Output |
|---|---|---|---|---|
| 01 | Collect | Retailer product pages | Scrapy / Requests | Raw listing records |
| 02 | Clean | Raw records | Python (Pandas) | Normalized titles, prices, vendor fields |
| 03 | Classify | Normalized titles | Embedding model | Canonical product entities |
| 04 | Persist | Entities + listings + price observations | MySQL + Redis | Durable store + fast cache |
| 05 | Analyze | Stored history | Pandas + Plotly | Category/vendor stats, deal & trend signals |
| 06 | Serve | Analytics + entity index | Flask + Tailwind CSS | Search, product cards, vendor browsing, build comparison |
3. Implementation Notes
3.1 Entity resolution
The hardest data problem in the project. Retailers describe the same hardware with inconsistent spelling, capacity labels, brand prefixes, and marketing copy. Exact-name matching is not sufficient, so the embedding classifier maps noisy retailer titles to a canonical entity. This stage is the linchpin: if it mis-groups two products, every downstream price-history and deal-signal calculation is corrupted.
3.2 Data layer split
MySQL holds the durable product, listing, and price-history tables. Redis handles faster application-level access and transient state (e.g., recent-refresh timestamps, hot product lookups). Keeping the two stores separate means the analytics queries do not compete with the scraping write path.
3.3 Analytics surface
Pandas computes category mix, vendor coverage, price momentum, volatility, and trending-product rankings. Plotly renders the interactive charts. The Flask dashboard layers search, product cards, vendor browsing, deal discovery, and build-oriented comparison on top of these pre-computed signals rather than querying raw rows at request time.
4. Constraints
-
Closed-source codebase
No public repository. Evidence is the live site and the portfolio snapshot, so an external reviewer cannot audit the scraper logic or classifier thresholds directly.
-
Scraper fragility
Retailer DOM changes or anti-bot measures can silently break collection flows. No mention of selector regression tests or headless-browser fallback in the notes.
-
Probabilistic entity resolution
Embedding-based matching is approximate. A mis-classified entity silently merges two distinct products, corrupting price history and deal signals with no obvious error at the UI layer.
-
Snapshot vs. live data
The 21,440-product / 50,938-listing figure is a point-in-time capture. The live dataset shifts independently, so any metric derived from the snapshot may not reflect current coverage.
5. Next
- a. Add automated regression tests for scraper selectors and a golden-set evaluation for entity-resolution accuracy, so regressions surface before they hit production data.
- b. Introduce a user-facing feedback loop ("not the same product") so mis-classified entities can be corrected and fed back into the classifier training set.
- c. Ship price-drop alerts (email or webhook) triggered when a tracked product falls below a user-set threshold, turning the analytics layer into an actionable signal.
— end of report —