Abstract
A scheduled automation pipeline that collects candidate public proxy endpoints, validates them, and publishes the surviving entries as protocol- and anonymity-specific files on a roughly 20-minute cycle. The interesting engineering problem is not list size but keeping a fast-decaying dataset fresh enough to be useful.
1. What This Is
This repository is the published data-output side of a set of public-proxy experiments. It maintains separate lists for HTTP/HTTPS, SOCKS4, and SOCKS5, and splits entries by anonymity class (elite, anonymous, transparent) where upstream sources provide that label.
The design bet is that freshness is a better quality signal than raw list size. Public endpoints die quickly, so a large static list is worse than a smaller one that was validated minutes ago. The repo is versioned in Git so the automated pipeline is observable over time and the output can be consumed by related validation or API tooling.
2. How It Works
The pipeline runs on a fixed schedule and repeats the same five stages each cycle:
| # | Stage | Input | Tool | Output |
|---|---|---|---|---|
| 01 | Collect | Upstream public-proxy sources | Scheduled job | Raw candidate endpoints |
| 02 | Normalize & Classify | Raw candidates | Parsing / tagging | Entries tagged by protocol and anonymity |
| 03 | Validate | Classified candidates | Reachability / proxy test | Passing endpoints |
| 04 | Write | Passing endpoints | File writer | Protocol/anonymity-specific list files |
| 05 | Publish | Changed files | Git commit / push | Versioned repo state |
3. Implementation Notes
3.1 Protocol-specific file layout
Outputs are split into predictable files per protocol (HTTP/HTTPS, SOCKS4, SOCKS5) so downstream consumers can pull only the format they need without parsing a monolithic list. Anonymity sub-categories are layered on top where the upstream source provides them.
3.2 Versioned publication
Each refresh cycle produces a Git commit. This gives a free audit trail of how the endpoint population shifts over time and makes the pipeline observable without a separate metrics system. The trade-off is that the repo accumulates history for data that is intentionally ephemeral.
4. Constraints
-
Short endpoint lifetimes
A 20-minute cadence still leaves a window where many listed proxies are dead. Consumers polling between refreshes will hit stale entries; the list is a snapshot, not a live guarantee.
-
Self-reported anonymity labels
Elite / anonymous / transparent tags come from upstream sources and are not independently verified by the validation stage. A proxy labelled "elite" may still log traffic.
-
No access control
The published data is a public Git repo with no authentication. Anyone can read the lists, and the pipeline has no mechanism to revoke or gate access to a specific consumer.
-
Single scheduled job
The refresh is one recurring workflow. There is no visible backpressure, retry, or alerting if a cycle fails, so a silent outage would go unnoticed until the data goes stale.
5. Next
- a. Expose a lightweight HTTP endpoint (or S3 bucket) for programmatic consumption so downstream tools stop polling raw Git files.
- b. Track per-endpoint lifetime statistics across cycles to quantify median time-to-death and feed that back into the validation timeout.
- c. Add a simple health-check alert (e.g., if a cycle produces zero new entries or the file set is unchanged for N cycles) so silent pipeline failures surface.
— end of report —