Abstract
A FastAPI service that aggregates public HTTP and SOCKS proxy lists from configured GitHub sources, validates candidates concurrently, and serves the results through password-protected endpoints. The design separates the validation pipeline from the request path via a recurring background refresh loop, keeping API latency predictable regardless of dataset size.
1. What This Is
I built this as a data and API engineering experiment. The service pulls raw proxy records from public GitHub repositories, normalizes them by protocol type, runs concurrent reachability checks, and exposes the validated subset through a small REST API. The goal was to turn volatile, unstructured public data into a consumable endpoint without coupling the expensive validation work to individual API calls.
Any downstream networking use of the served data must remain lawful and authorized; the project itself is a pipeline and API-design exercise, not a proxy distribution tool.
2. How It Works
The pipeline runs as a repeating background job. Each cycle fetches, validates, and stores a fresh dataset; the API simply reads the latest snapshot.
| # | Stage | Input | Tool | Output |
|---|---|---|---|---|
| 01 | Fetch sources | Configured GitHub raw URLs | httpx | Raw proxy text files |
| 02 | Parse & normalize | Raw text lines | Python parsing | HTTP / SOCKS candidate lists |
| 03 | Concurrent validation | Candidate lists | ThreadPoolExecutor + httpx / PySocks | Reachable proxy records |
| 04 | Persist snapshot | Validated records | In-memory / flat-file store | Latest dataset |
| 05 | Serve via API | Latest dataset | FastAPI + password auth | JSON response to client |
3. Implementation Notes
3.1 Concurrent validation
Each candidate is checked for reachability inside a ThreadPoolExecutor worker. HTTP proxies are probed with httpx; SOCKS candidates go through PySocks. tqdm prints a progress bar in the terminal during batch runs, which is useful when iterating locally but irrelevant in production.
3.2 Background refresh loop
The collect-validate-store cycle runs on a fixed interval independent of incoming requests. A client hitting the API reads the most recent snapshot rather than triggering a fresh validation pass, so response time stays flat even when the source dataset grows from hundreds to tens of thousands of entries.
3.3 Protocol separation
HTTP and SOCKS candidates are split before validation because the transport libraries differ. This keeps the worker logic simple: each worker knows which client to instantiate, and failures in one protocol batch do not block the other.
4. Constraints
-
Volatile source data
Public proxy lists on GitHub change or vanish without notice. A missing or reformatted source file can silently shrink or empty the dataset between refresh cycles.
-
Basic password auth only
Endpoints are guarded by a shared password loaded via python-dotenv. There is no token rotation, per-client identity, or rate limiting, so a leaked credential exposes the full dataset.
-
Single-process background loop
The refresh job lives in one process. If it crashes or the host reboots, no new data is produced until the service restarts. There is no supervisor, retry, or distributed-worker fallback.
-
No structured persistence
Validated results are held in memory or a flat file. There is no audit trail, no historical comparison, and no way to answer "how many proxies were alive last cycle" without re-running the pipeline.
5. Next
- a. Replace the shared password with JWT-based auth and add per-endpoint rate limiting.
- b. Introduce a lightweight persistence layer (SQLite) so each refresh cycle is logged and historical trends can be queried.
- c. Containerize the service with Docker, add a /health endpoint, and run the refresh loop under a process supervisor for crash recovery.
— end of report —