proxy-scraper-cli 1.10.0__tar.gz → 1.12.0__tar.gz

{proxy_scraper_cli-1.10.0 → proxy_scraper_cli-1.12.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: proxy-scraper-cli
-Version: 1.10.0
+Version: 1.12.0
 Summary: Finds free HTTP, SOCKS4 and SOCKS5 proxies that actually work: 700+ sources, real checks, honeypot filtering and a rotating local proxy server.
 Author: Maximilian Feix
 License-Expression: MIT
@@ -205,7 +205,7 @@ Every protocol and country also has its own page with a plain download, e.g. [SO
 | HTTP · SOCKS4 · SOCKS5 | `1.2.3.4:8080` | [http.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/http.txt) · [socks4.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/socks4.txt) · [socks5.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/socks5.txt) |
 | HTTPS-capable only | `type://ip:port` | [https.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/https.txt) |
 | Elite only | `type://ip:port` | [elite.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/elite.txt) |
-| Gets through to Google · Reddit · Amazon (no captcha, no 403 in the last run) | `type://ip:port` | [google.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/works-with/google.txt) · [reddit.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/works-with/reddit.txt) · [amazon.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/works-with/amazon.txt) |
+| Gets through to Google · Reddit · Amazon · Instagram · TikTok (no captcha, no block in the last run) | `type://ip:port` | [google.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/works-with/google.txt) · [reddit.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/works-with/reddit.txt) · [amazon.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/works-with/amazon.txt) · [instagram.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/works-with/instagram.txt) · [tiktok.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/works-with/tiktok.txt) |
 | Stable, on the list in 90 %+ of this week's runs | `type://ip:port` | [stable.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/stable.txt) |
 | With all details | latency, country, HTTPS, anonymity, exit IP, uptime, sites | [proxies.json](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/proxies.json) · [proxies.csv](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/proxies.csv) |
 
@@ -219,6 +219,8 @@ Every protocol and country also has its own page with a plain download, e.g. [SO
 curl -s https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/socks5.txt | head
 ```
 
+**Over time:** every day the first run's `proxies.json` is kept for good as a gzipped asset of the release of that year, like [snapshots-2026](../../releases/tag/snapshots-2026) – `proxies-2026-09-28.json.gz` and so on, for anyone who wants to look at free proxies over weeks and months.
+
 Every file is also on GitHub Pages, which sits behind a CDN, isn't rate limited like raw.githubusercontent and sends CORS headers, so it works straight from the browser: `https://maximilianfeix.github.io/proxy-scraper/socks5.txt`. [jsDelivr](https://cdn.jsdelivr.net/gh/maximilianfeix/proxy-scraper@proxy-list/) works too, but can lag behind by a few hours.
 
 Or let the tool start from it: `proxy-scraper --recheck live` downloads the list and checks it again from **your** network – about 30 seconds instead of a full scan (517 of 1,169 worked from here). With `--serve` you have a rotating proxy in under a minute.
@@ -440,7 +442,7 @@ pool = itertools.cycle(p.url for p in proxies)
 r = requests.get("https://api.ipify.org", proxies={"https": next(pool)}, timeout=15)  # pip install "requests[socks]"
 ```
 
-Same filters as `find_proxies`, plus `min_uptime`, `works_on` (`["google"]`, `"reddit"`, `"amazon"`) and `limit`. Every result also has `uptime_24h`, `uptime_7d`, `first_seen`, `up_for_hours` and `sites`.
+Same filters as `find_proxies`, plus `min_uptime`, `works_on` (`["google"]`, `"reddit"`, `"amazon"`, `"instagram"`, `"tiktok"`) and `limit`. Every result also has `uptime_24h`, `uptime_7d`, `first_seen`, `up_for_hours` and `sites`.
 
 <a id="proxy-server"></a>
 

{proxy_scraper_cli-1.10.0 → proxy_scraper_cli-1.12.0}/README.md

@@ -164,7 +164,7 @@ Every protocol and country also has its own page with a plain download, e.g. [SO
 | HTTP · SOCKS4 · SOCKS5 | `1.2.3.4:8080` | [http.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/http.txt) · [socks4.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/socks4.txt) · [socks5.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/socks5.txt) |
 | HTTPS-capable only | `type://ip:port` | [https.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/https.txt) |
 | Elite only | `type://ip:port` | [elite.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/elite.txt) |
-| Gets through to Google · Reddit · Amazon (no captcha, no 403 in the last run) | `type://ip:port` | [google.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/works-with/google.txt) · [reddit.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/works-with/reddit.txt) · [amazon.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/works-with/amazon.txt) |
+| Gets through to Google · Reddit · Amazon · Instagram · TikTok (no captcha, no block in the last run) | `type://ip:port` | [google.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/works-with/google.txt) · [reddit.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/works-with/reddit.txt) · [amazon.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/works-with/amazon.txt) · [instagram.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/works-with/instagram.txt) · [tiktok.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/works-with/tiktok.txt) |
 | Stable, on the list in 90 %+ of this week's runs | `type://ip:port` | [stable.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/stable.txt) |
 | With all details | latency, country, HTTPS, anonymity, exit IP, uptime, sites | [proxies.json](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/proxies.json) · [proxies.csv](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/proxies.csv) |
 
@@ -178,6 +178,8 @@ Every protocol and country also has its own page with a plain download, e.g. [SO
 curl -s https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/socks5.txt | head
 ```
 
+**Over time:** every day the first run's `proxies.json` is kept for good as a gzipped asset of the release of that year, like [snapshots-2026](../../releases/tag/snapshots-2026) – `proxies-2026-09-28.json.gz` and so on, for anyone who wants to look at free proxies over weeks and months.
+
 Every file is also on GitHub Pages, which sits behind a CDN, isn't rate limited like raw.githubusercontent and sends CORS headers, so it works straight from the browser: `https://maximilianfeix.github.io/proxy-scraper/socks5.txt`. [jsDelivr](https://cdn.jsdelivr.net/gh/maximilianfeix/proxy-scraper@proxy-list/) works too, but can lag behind by a few hours.
 
 Or let the tool start from it: `proxy-scraper --recheck live` downloads the list and checks it again from **your** network – about 30 seconds instead of a full scan (517 of 1,169 worked from here). With `--serve` you have a rotating proxy in under a minute.
@@ -399,7 +401,7 @@ pool = itertools.cycle(p.url for p in proxies)
 r = requests.get("https://api.ipify.org", proxies={"https": next(pool)}, timeout=15)  # pip install "requests[socks]"
 ```
 
-Same filters as `find_proxies`, plus `min_uptime`, `works_on` (`["google"]`, `"reddit"`, `"amazon"`) and `limit`. Every result also has `uptime_24h`, `uptime_7d`, `first_seen`, `up_for_hours` and `sites`.
+Same filters as `find_proxies`, plus `min_uptime`, `works_on` (`["google"]`, `"reddit"`, `"amazon"`, `"instagram"`, `"tiktok"`) and `limit`. Every result also has `uptime_24h`, `uptime_7d`, `first_seen`, `up_for_hours` and `sites`.
 
 <a id="proxy-server"></a>
 

{proxy_scraper_cli-1.10.0 → proxy_scraper_cli-1.12.0}/proxy_scraper_cli.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: proxy-scraper-cli
-Version: 1.10.0
+Version: 1.12.0
 Summary: Finds free HTTP, SOCKS4 and SOCKS5 proxies that actually work: 700+ sources, real checks, honeypot filtering and a rotating local proxy server.
 Author: Maximilian Feix
 License-Expression: MIT
@@ -205,7 +205,7 @@ Every protocol and country also has its own page with a plain download, e.g. [SO
 | HTTP · SOCKS4 · SOCKS5 | `1.2.3.4:8080` | [http.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/http.txt) · [socks4.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/socks4.txt) · [socks5.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/socks5.txt) |
 | HTTPS-capable only | `type://ip:port` | [https.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/https.txt) |
 | Elite only | `type://ip:port` | [elite.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/elite.txt) |
-| Gets through to Google · Reddit · Amazon (no captcha, no 403 in the last run) | `type://ip:port` | [google.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/works-with/google.txt) · [reddit.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/works-with/reddit.txt) · [amazon.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/works-with/amazon.txt) |
+| Gets through to Google · Reddit · Amazon · Instagram · TikTok (no captcha, no block in the last run) | `type://ip:port` | [google.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/works-with/google.txt) · [reddit.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/works-with/reddit.txt) · [amazon.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/works-with/amazon.txt) · [instagram.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/works-with/instagram.txt) · [tiktok.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/works-with/tiktok.txt) |
 | Stable, on the list in 90 %+ of this week's runs | `type://ip:port` | [stable.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/stable.txt) |
 | With all details | latency, country, HTTPS, anonymity, exit IP, uptime, sites | [proxies.json](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/proxies.json) · [proxies.csv](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/proxies.csv) |
 
@@ -219,6 +219,8 @@ Every protocol and country also has its own page with a plain download, e.g. [SO
 curl -s https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/socks5.txt | head
 ```
 
+**Over time:** every day the first run's `proxies.json` is kept for good as a gzipped asset of the release of that year, like [snapshots-2026](../../releases/tag/snapshots-2026) – `proxies-2026-09-28.json.gz` and so on, for anyone who wants to look at free proxies over weeks and months.
+
 Every file is also on GitHub Pages, which sits behind a CDN, isn't rate limited like raw.githubusercontent and sends CORS headers, so it works straight from the browser: `https://maximilianfeix.github.io/proxy-scraper/socks5.txt`. [jsDelivr](https://cdn.jsdelivr.net/gh/maximilianfeix/proxy-scraper@proxy-list/) works too, but can lag behind by a few hours.
 
 Or let the tool start from it: `proxy-scraper --recheck live` downloads the list and checks it again from **your** network – about 30 seconds instead of a full scan (517 of 1,169 worked from here). With `--serve` you have a rotating proxy in under a minute.
@@ -440,7 +442,7 @@ pool = itertools.cycle(p.url for p in proxies)
 r = requests.get("https://api.ipify.org", proxies={"https": next(pool)}, timeout=15)  # pip install "requests[socks]"
 ```
 
-Same filters as `find_proxies`, plus `min_uptime`, `works_on` (`["google"]`, `"reddit"`, `"amazon"`) and `limit`. Every result also has `uptime_24h`, `uptime_7d`, `first_seen`, `up_for_hours` and `sites`.
+Same filters as `find_proxies`, plus `min_uptime`, `works_on` (`["google"]`, `"reddit"`, `"amazon"`, `"instagram"`, `"tiktok"`) and `limit`. Every result also has `uptime_24h`, `uptime_7d`, `first_seen`, `up_for_hours` and `sites`.
 
 <a id="proxy-server"></a>
 

{proxy_scraper_cli-1.10.0 → proxy_scraper_cli-1.12.0}/proxy_scraper_cli.egg-info/SOURCES.txt

@@ -40,9 +40,11 @@ proxyscraper/preferences.py
 proxyscraper/proxypages.py
 proxyscraper/publish.py
 proxyscraper/report.py
+proxyscraper/runsteps.py
 proxyscraper/sites.py
 proxyscraper/sources.json
 proxyscraper/sources.py
+proxyscraper/speed.py
 proxyscraper/targets.py
 proxyscraper/server/__init__.py
 proxyscraper/server/core.py
@@ -104,6 +106,7 @@ tests/test_server_closed_writer.py
 tests/test_server_fixes.py
 tests/test_sites.py
 tests/test_sources_json.py
+tests/test_speed.py
 tests/test_tamper.py
 tests/test_targets.py
 tests/test_ui.py

{proxy_scraper_cli-1.10.0 → proxy_scraper_cli-1.12.0}/proxyscraper/__init__.py

@@ -1,7 +1,7 @@
 """proxy-scraper: collects public HTTP/SOCKS4/SOCKS5 proxies from hundreds of sources, checks them
 in parallel with its own protocol handshakes and learns which sources deliver good proxies."""
 
-__version__ = "1.10.0"
+__version__ = "1.12.0"
 
 
 def __getattr__(name):

{proxy_scraper_cli-1.10.0 → proxy_scraper_cli-1.12.0}/proxyscraper/agent.py

@@ -179,17 +179,18 @@ def normalize_countries(countries: Iterable[str]) -> List[str]:
 def select(rows: Iterable[dict], protocol: str = "any", countries: Iterable[str] = (), https_only: bool = False,
            elite_only: bool = False, exclude_datacenter: bool = False, exclude_blocklisted: bool = False,
            stable_only: bool = False, max_latency_ms: int = 0, run_hours: int = 1, min_uptime: int = 0,
-           works_on: Iterable[str] = ()) -> List[dict]:
+           works_on: Iterable[str] = (), min_speed_kbps: int = 0) -> List[dict]:
     """The rows that pass every filter, fastest first – the same filters as on the website."""
     return sorted(matching(rows, protocol, countries, https_only, elite_only, exclude_datacenter,
-                           exclude_blocklisted, stable_only, max_latency_ms, run_hours, min_uptime, works_on),
+                           exclude_blocklisted, stable_only, max_latency_ms, run_hours, min_uptime, works_on,
+                           min_speed_kbps),
                   key=lambda r: r.get("latency") or 0)
 
 
 def matching(rows: Iterable[dict], protocol: str = "any", countries: Iterable[str] = (), https_only: bool = False,
              elite_only: bool = False, exclude_datacenter: bool = False, exclude_blocklisted: bool = False,
              stable_only: bool = False, max_latency_ms: int = 0, run_hours: int = 1,
-             min_uptime: int = 0, works_on: Iterable[str] = ()) -> Iterator[dict]:
+             min_uptime: int = 0, works_on: Iterable[str] = (), min_speed_kbps: int = 0) -> Iterator[dict]:
     """The rows that pass every filter, in list order – checks the filters before the first row is asked for."""
     protocol = _check_protocol(protocol)
     wanted = set(normalize_countries(countries))
@@ -205,7 +206,8 @@ def matching(rows: Iterable[dict], protocol: str = "any", countries: Iterable[st
            and (not stable_only or (r.get("streak") or 0) >= stable_runs)
            and (not max_latency_ms or (r.get("latency") or 0) <= max_latency_ms)
            and (not min_uptime or (r.get("uptime_7d") or 0) >= min_uptime)
-           and all((r.get("sites") or {}).get(s) for s in sites))
+           and all((r.get("sites") or {}).get(s) for s in sites)
+           and (not min_speed_kbps or (r.get("speed_kbps") or 0) >= min_speed_kbps))
 
 
 def describe(item: Union[dict, CheckResult], run_hours: Optional[int] = None) -> dict:
@@ -229,6 +231,7 @@ def describe(item: Union[dict, CheckResult], run_hours: Optional[int] = None) ->
         "up_for_hours": streak * run_hours if streak and run_hours else None,
         "uptime_7d_percent": item.get("uptime_7d"),
         "works_on": [name for name, ok in (item.get("sites") or {}).items() if ok],
+        "speed_kbps": item.get("speed_kbps"),
     }
 
 

{proxy_scraper_cli-1.10.0 → proxy_scraper_cli-1.12.0}/proxyscraper/api.py

@@ -166,7 +166,8 @@ class LiveProxy(CheckResult):
     uptime_7d: Optional[int] = None   # the same over the week
     first_seen: str = ""              # ISO time of the first run it was listed in
     up_for_hours: int = 0             # listed without a gap for this long
-    sites: Dict[str, bool] = field(default_factory=dict)  # "google", "reddit", "amazon" -> got through last run?
+    sites: Dict[str, bool] = field(default_factory=dict)  # "google", "reddit", … -> got through in the last run?
+    speed_kbps: Optional[int] = None  # download speed in KiB/s in the last run (HTTPS-capable ones only)
 
 
 def _live_fetch(url: str, timeout: float = 20, headers=None):
@@ -184,19 +185,22 @@ def _live_proxy(row: dict, run_hours: int) -> LiveProxy:
         org=row.get("org") or "", hosting=row.get("hosting"), blocklisted=row.get("blocklisted"),
         uptime_24h=row.get("uptime_24h"), uptime_7d=row.get("uptime_7d"), first_seen=row.get("first_seen") or "",
         up_for_hours=streak * run_hours,
-        sites={k: v for k, v in (row.get("sites") or {}).items() if isinstance(v, bool)})
+        sites={k: v for k, v in (row.get("sites") or {}).items() if isinstance(v, bool)},
+        speed_kbps=row.get("speed_kbps") if type(row.get("speed_kbps")) is int else None)
 
 
 async def live_proxies_async(*, types: Iterable[str] = PROXY_TYPES, countries: Iterable[str] = (), https: bool = False,
                              anonymity: str = "", max_latency: int = 0, no_datacenter: bool = False,
                              no_blocklisted: bool = False, min_uptime: int = 0, works_on: Iterable[str] = (),
-                             limit: int = 0) -> List[LiveProxy]:
+                             min_speed: int = 0, limit: int = 0) -> List[LiveProxy]:
     """The proxies from the hourly list that pass the filters, fastest first. Nothing is checked here: they
     worked from GitHub's servers in the last run (at most an hour ago). find_proxies checks from your network.
 
     Same filters as find_proxies, plus
     min_uptime  only proxies listed in at least this share (percent) of the week's runs, e.g. 90
-    works_on    only proxies that got through to these sites in the last run: "google", "reddit", "amazon"
+    works_on    only proxies that got through to these sites in the last run: google, reddit, amazon, instagram,
+                tiktok
+    min_speed   only proxies that downloaded at least this many KiB/s in the last run
     limit       at most this many (0 = all)
     """
     from .agent import AgentError, LiveSource
@@ -226,7 +230,7 @@ async def live_proxies_async(*, types: Iterable[str] = PROXY_TYPES, countries: I
                     if r.get("ptype") in types and isinstance(r.get("proxy"), str)),
                    key=lambda p: p.latency)
     found = [p for p in found if opts.filters.accepts(p) and (not min_uptime or (p.uptime_7d or 0) >= min_uptime)
-             and all(p.sites.get(s) for s in works_on)]
+             and all(p.sites.get(s) for s in works_on) and (not min_speed or (p.speed_kbps or 0) >= min_speed)]
     return found[:limit] if limit else found
 
 

{proxy_scraper_cli-1.10.0 → proxy_scraper_cli-1.12.0}/proxyscraper/checker.py

@@ -334,6 +334,10 @@ class Checker:
             writer.close()
         return 200 <= status < 400
 
+    async def tls_tunnel(self, ptype: str, proxy: str, host: str, ip: str, port: int = 443):
+        """Public side of _tls_tunnel for the steps after a run (sites, speed): ip as text."""
+        return await self._tls_tunnel(ptype, proxy, host, socket.inet_aton(ip), port)
+
     async def _tls_tunnel(self, ptype: str, proxy: str, host: str, ip_bytes: bytes, port: int):
         """Tunnel through the proxy to host:port with verified TLS inside. None = tunnel refused or MITM."""
         loop = asyncio.get_running_loop()

{proxy_scraper_cli-1.10.0 → proxy_scraper_cli-1.12.0}/proxyscraper/mcp_server.py

@@ -17,6 +17,7 @@ from mcp.types import ToolAnnotations
 from pydantic import BaseModel, Field
 
 from . import __version__, agent
+from .sites import SITES
 
 REPO = "https://github.com/maximilianfeix/proxy-scraper"
 
@@ -33,6 +34,7 @@ Free proxies are run by strangers: never send passwords, cookies, API keys or pe
 and expect some of them to stop working at any time."""
 
 Protocol = Literal["any", "http", "socks4", "socks5"]
+SiteName = Literal[tuple(s.name for s in SITES)]  # the sites the hourly check tries (sites.py)
 Countries = Annotated[List[str], Field(description="Two-letter country codes of the exit IP, e.g. ['DE', 'NL']. "
                                                    "Empty = any country.")]
 
@@ -51,7 +53,10 @@ class Proxy(BaseModel):
     blocklisted: Optional[bool] = Field(description="Exit IP on the SpamCop blocklist – expect captchas")
     up_for_hours: Optional[int] = Field(description="How long it has worked without a gap (live list only)")
     works_on: List[str] = Field(default_factory=list, description="Big sites it got through to in the last hourly "
-                                                                  "check: google, reddit, amazon (live list only)")
+                                                                  f"check: {', '.join(s.name for s in SITES)} "
+                                                                  "(live list only)")
+    speed_kbps: Optional[int] = Field(default=None, description="Download speed in KiB/s in the last hourly check "
+                                                                "(HTTPS-capable proxies, live list only)")
     uptime_7d_percent: Optional[int] = Field(default=None, description="Share of the last 7 days' hourly checks it "
                                                                        "passed (live list only)")
 
@@ -121,9 +126,12 @@ def build_server(live: Optional[agent.LiveSource] = None, fetcher: Optional[agen
         min_uptime_percent: Annotated[int, Field(ge=0, le=100, description="Only proxies that passed at least this "
                                                                            "share of the last 7 days' hourly checks, "
                                                                            "e.g. 90 for the most reliable")] = 0,
-        works_on: Annotated[List[Literal["google", "reddit", "amazon"]],
+        works_on: Annotated[List[SiteName],
                             Field(description="Only proxies that got through to all of these sites in the last "
                                               "hourly check (most free proxies get a captcha or a 403 there)")] = [],  # noqa: B006
+        min_speed_kbps: Annotated[int, Field(ge=0, description="Only proxies that downloaded at least this many KiB/s "
+                                                                "in the last hourly check, e.g. 300 for loading "
+                                                                "real pages")] = 0,
         limit: Annotated[int, Field(ge=1, le=100, description="How many to return (fastest first)")] = 10,
     ) -> ProxyList:
         """Get working free proxies right now, fastest first.
@@ -135,7 +143,7 @@ def build_server(live: Optional[agent.LiveSource] = None, fetcher: Optional[agen
             data = await live.get()
             rows = agent.select(data.rows, protocol, countries, https_only, elite_only, exclude_datacenter,
                                 exclude_blocklisted, stable_only, max_latency_ms, data.run_hours, min_uptime_percent,
-                                works_on)
+                                works_on, min_speed_kbps)
         except agent.AgentError as e:
             _fail(e)
         return ProxyList(proxies=[Proxy(**agent.describe(r, data.run_hours)) for r in rows[:limit]],

{proxy_scraper_cli-1.10.0 → proxy_scraper_cli-1.12.0}/proxyscraper/proxypages.py

@@ -14,9 +14,10 @@ from pathlib import Path
 from typing import Dict, List, Optional, Sequence, Tuple
 
 from .pages import COUNTRIES, CSS, SITE_URL, _short_name
+from .sites import SITES
 
 INDEX_MIN_UPTIME = 50  # percent of the week's runs
-SITE_NAMES = {"google": "Google", "reddit": "Reddit", "amazon": "Amazon"}
+SITE_NAMES = {s.name: s.title for s in SITES}
 
 EXTRA_CSS = """
 .wrap > * { min-width: 0; }
@@ -97,8 +98,10 @@ def _summary(entries: List[dict]) -> dict:
     https = [r.get("https") for r in entries]
     rank = {"elite": 3, "anonymous": 2, "transparent": 1}
     sites = {k for r in entries for k, ok in (r.get("sites") or {}).items() if ok}
+    speeds = [r["speed_kbps"] for r in entries if r.get("speed_kbps")]
     return {
         "uptime": max(uptimes) if uptimes else None,
+        "speed": max(speeds) if speeds else None,
         "latency": min(r["latency"] for r in entries) if all("latency" in r for r in entries) else None,
         "https": True if True in https else False if False in https else None,
         "anonymity": max((r.get("anonymity") or "" for r in entries), key=lambda a: rank.get(a, 0)) or None,
@@ -128,6 +131,7 @@ def _render(proxy: str, entries: List[dict], updated: datetime, bits: Optional[i
             ("Country", _country(cc)),
             ("Provider", f"{info['org'] or 'unknown'}" + (f" (AS{info['asn']})" if info["asn"] else "")),
             ("Latency", f"{info['latency']:,} ms" if info["latency"] is not None else "unknown"),
+            ("Download speed", f"{info['speed']:,} KiB/s" if info["speed"] else "not measured"),
             ("HTTPS", _yes_no(info["https"], "yes, verified TLS", "no")),
             ("Anonymity", info["anonymity"] or "unknown"),
         ]

{proxy_scraper_cli-1.10.0 → proxy_scraper_cli-1.12.0}/proxyscraper/publish.py

@@ -74,6 +74,9 @@ def stats_for(rows: List[dict], now: datetime) -> dict:
         "run_hours": RUN_HOURS,
         "countries": dict(Counter(r["country"] for r in rows if r.get("country")).most_common(15)),
         "median_latency": round(statistics.median(r["latency"] for r in rows)) if rows else 0,
+        # download speed of the HTTPS-capable ones (speed.py), None when it wasn't measured
+        "median_speed_kbps": round(statistics.median(speeds)) if (speeds := [r["speed_kbps"] for r in rows
+                                                                               if r.get("speed_kbps")]) else None,
     }
 
 
@@ -244,7 +247,8 @@ def load_streaks(path: Optional[Path]) -> Dict[str, int]:
 
 
 def publish(run_dir: Path, out: Path, minimum: int = 20, now: Optional[datetime] = None,
-            history: Optional[Path] = None, streaks: Optional[Path] = None, seen: Optional[Path] = None) -> int:
+            history: Optional[Path] = None, streaks: Optional[Path] = None, seen: Optional[Path] = None,
+            feed: Optional[Path] = None) -> int:
     rows = load_rows(run_dir)
     if len(rows) < minimum:
         print(f"Only {len(rows)} hits (< {minimum}) – the old list stays online.")
@@ -270,7 +274,7 @@ def publish(run_dir: Path, out: Path, minimum: int = 20, now: Optional[datetime]
     with (run_dir / "proxies.csv").open(newline="", encoding="utf-8") as src, \
             (out / "proxies.csv").open("w", newline="", encoding="utf-8") as dst:
         reader = csv.DictReader(src)
-        extra = ("first_seen", "uptime_24h", "uptime_7d", "works_with")
+        extra = ("first_seen", "uptime_24h", "uptime_7d", "works_with", "speed_kbps")
         base = list(reader.fieldnames or ["proxy"])
         writer = csv.DictWriter(dst, fieldnames=[*base, *(k for k in extra if k not in base)])
         writer.writeheader()
@@ -309,7 +313,7 @@ def publish(run_dir: Path, out: Path, minimum: int = 20, now: Optional[datetime]
     write_pages(rows, out, now, extra=proxy_pages)
     write_json(out / "history.json", runs)
     # the weekly report: lifetimes, sites, countries – a page, markdown and a short post (report.py)
-    write_report(weekly_facts(rows, runs, {url: e["bits"] for url, e in listed.items()}, now), out)
+    write_report(weekly_facts(rows, runs, {url: e["bits"] for url, e in listed.items()}, now), out, previous_feed=feed)
     for theme in THEMES:  # charts for the README, which embeds them from GitHub Pages
         (out / f"chart-{theme}.svg").write_text(trend_svg(runs, now, theme), encoding="utf-8")
         (out / f"countries-{theme}.svg").write_text(countries_svg(stats["countries"], theme), encoding="utf-8")
@@ -335,8 +339,10 @@ def main(argv: Optional[List[str]] = None) -> int:
     p.add_argument("--streaks", type=Path, help="streaks.json from last time (how long each proxy has been listed)")
     p.add_argument("--seen", type=Path,
                    help="seen.json from last time (which runs each proxy was listed in, for the uptime)")
+    p.add_argument("--feed", type=Path, help="report/feed.xml from last time (the weekly entries it keeps)")
     args = p.parse_args(argv)
-    return publish(args.run_dir, args.out_dir, args.min, history=args.history, streaks=args.streaks, seen=args.seen)
+    return publish(args.run_dir, args.out_dir, args.min, history=args.history, streaks=args.streaks, seen=args.seen,
+                   feed=args.feed)
 
 
 if __name__ == "__main__":

{proxy_scraper_cli-1.10.0 → proxy_scraper_cli-1.12.0}/proxyscraper/report.py

@@ -5,6 +5,7 @@ get through to the big sites, where they are. Rebuilt with every run from the sa
 
 from __future__ import annotations
 
+import xml.etree.ElementTree as ET
 from collections import Counter
 from datetime import datetime, timedelta
 from html import escape
@@ -13,11 +14,12 @@ from typing import Dict, List, Optional, Sequence
 
 from .charts import FONT, THEMES, WIDTH
 from .pages import COUNTRIES, CSS, SITE_URL, _short_name
+from .sites import SITES
 
 DAYS = 7
 RUN_HOURS = 1
 REPORT_URL = f"{SITE_URL}report/"
-SITE_NAMES = {"google": "Google", "reddit": "Reddit", "amazon": "Amazon"}
+SITE_NAMES = {s.name: s.title for s in SITES}
 # checks in a row a proxy was on the list for -> label (the runs are roughly hourly, but not always)
 BUCKETS = [(1, 1, "1 check"), (2, 5, "2–5 checks"), (6, 23, "6–23 checks"), (24, 71, "24–71 checks"),
            (72, 10 ** 6, "72 checks +")]
@@ -102,6 +104,7 @@ def weekly_facts(rows: List[dict], runs: Sequence[dict], seen: Dict[str, int], n
         if any(r.get("hosting") is not None for r in rows) else None,
         "countries": Counter(r["country"] for r in rows if r.get("country")).most_common(5),
         "latency_now": window[-1][1].get("median_latency") if window else None,
+        "speeds": sorted(r["speed_kbps"] for r in rows if r.get("speed_kbps")),
         "sites": sites,
     }
 
@@ -141,6 +144,11 @@ def _lines(f: dict) -> List[str]:
     line = (f"Between {f['low']:,} and {f['peak']:,} proxies worked at any one time. Right now: {f['total']:,}, "
             f"{f['https']} % of them tunnel HTTPS")
     out.append(line + (f", {f['datacenter']} % exit from a datacenter." if f["datacenter"] is not None else "."))
+    speeds = f.get("speeds") or []
+    if speeds:
+        median, top = speeds[len(speeds) // 2], speeds[9 * len(speeds) // 10]
+        out.append(f"Downloading 100 KB through them: half manage {median:,} KiB/s or more, "
+                   f"the fastest tenth over {top:,} KiB/s.")
     if f["countries"]:
         out.append("Most of them sit in " + ", ".join(f"{_country(cc)} ({n:,})" for cc, n in f["countries"]) + ".")
     return out
@@ -215,6 +223,7 @@ def page(f: dict) -> str:
 <title>The week in free proxies ({escape(_period(f))}) | proxy-scraper</title>
 <meta name="description" content="{escape(_lines(f)[0])}">
 <link rel="canonical" href="{REPORT_URL}">
+<link rel="alternate" type="application/atom+xml" title="The week in free proxies" href="{REPORT_URL}feed.xml">
 <meta property="og:title" content="The week in free proxies">
 <meta property="og:description" content="{escape(_lines(f)[0])}">
 <meta property="og:image" content="{SITE_URL}og.png">
@@ -249,6 +258,7 @@ picture img {{ width: 100%; height: auto; border-radius: 16px; }}
     <img src="{root}chart-dark.svg" alt="Working proxies over the week, by protocol">
   </picture>
   <footer><span>Want to share it? <a href="report.md">Markdown</a> · <a href="post.txt">a short post</a> ·
+  <a href="feed.xml">RSS/Atom feed</a> ·
   <a href="https://github.com/maximilianfeix/proxy-scraper">the code</a></span></footer>
 </div>
 </body>
@@ -256,11 +266,63 @@ picture img {{ width: 100%; height: auto; border-radius: 16px; }}
 """
 
 
-def write_report(f: dict, out: Path) -> None:
+FEED_WEEKS = 12
+_ATOM = "{http://www.w3.org/2005/Atom}"
+
+
+def feed(f: dict, previous: Optional[str]) -> str:
+    """Atom feed with one entry per finished ISO week. The first run of a week adds the entry for the week that just
+    ended; every later run that week hands back the feed unchanged – feed readers and Slack/Discord bridges see one
+    new entry a week, not an update every hour. `previous`: the feed from the last run (None or garbage: start over)."""
+    now = f["now"]
+    monday = (now - timedelta(days=now.weekday())).replace(hour=0, minute=0, second=0, microsecond=0)
+    year, week, _ = (monday - timedelta(days=1)).isocalendar()
+    entry_id = f"{REPORT_URL}{year}-W{week:02d}"
+    kept: List[str] = []
+    if previous:
+        try:
+            root = ET.fromstring(previous)
+            old = root.findall(f"{_ATOM}entry")
+        except ET.ParseError:
+            old = []
+        if any((e.findtext(f"{_ATOM}id") or "") == entry_id for e in old):
+            return previous  # this week's entry is out already – leave it exactly as it is
+        ET.register_namespace("", _ATOM[1:-1])
+        kept = [ET.tostring(e, encoding="unicode") for e in old][:FEED_WEEKS - 1]
+    stamp = monday.strftime("%Y-%m-%dT%H:%M:%SZ")
+    text = "\n\n".join(_lines(f))
+    new = f"""<entry>
+    <title>The week in free proxies: {escape(_period(f))}</title>
+    <link rel="alternate" href="{REPORT_URL}"/>
+    <id>{entry_id}</id>
+    <updated>{stamp}</updated>
+    <content type="text">{escape(text)}</content>
+  </entry>"""
+    body = "\n  ".join([new, *kept])
+    return f"""<?xml version="1.0" encoding="utf-8"?>
+<feed xmlns="http://www.w3.org/2005/Atom">
+  <title>The week in free proxies</title>
+  <subtitle>How long free proxies last and which sites let them through – from hourly checks by proxy-scraper</subtitle>
+  <link rel="self" href="{REPORT_URL}feed.xml"/>
+  <link rel="alternate" href="{REPORT_URL}"/>
+  <id>{REPORT_URL}</id>
+  <updated>{stamp}</updated>
+  <author><name>proxy-scraper</name></author>
+  {body}
+</feed>
+"""
+
+
+def write_report(f: dict, out: Path, previous_feed: Optional[Path] = None) -> None:
     folder = out / "report"
     folder.mkdir(parents=True, exist_ok=True)
     (folder / "index.html").write_text(page(f), encoding="utf-8")
     (folder / "report.md").write_text(markdown(f), encoding="utf-8")
     (folder / "post.txt").write_text(post(f) + "\n", encoding="utf-8")
+    try:
+        previous = previous_feed.read_text(encoding="utf-8") if previous_feed else None
+    except OSError:
+        previous = None
+    (folder / "feed.xml").write_text(feed(f, previous), encoding="utf-8")
     for theme in THEMES:
         (folder / f"lifetimes-{theme}.svg").write_text(lifetimes_svg(f, theme), encoding="utf-8")

proxy_scraper_cli-1.12.0/proxyscraper/runsteps.py

@@ -0,0 +1,74 @@
+"""Shared plumbing for the extra steps that run on a finished run's proxies.json in the hourly workflow –
+which sites let a proxy through (sites.py), how fast it downloads (speed.py).
+
+Each step loads the run's rows, tries the HTTPS-capable proxies through a verified TLS tunnel with a limit on how
+many at once, and writes the file back in one go: if the workflow kills the step at its timeout, the publish step
+still finds the old file intact.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import json
+import socket
+from pathlib import Path
+from typing import Awaitable, Callable, Iterable, List, Tuple
+
+from .paths import atomic_write
+
+# what a probe through a free proxy can run into – all of it just means "no answer from this one"
+PROBE_ERRORS = (OSError, asyncio.TimeoutError, asyncio.IncompleteReadError, asyncio.LimitOverrunError, ValueError)
+
+
+def load_rows(run_dir: Path) -> Tuple[Path, List[dict]]:
+    path = run_dir / "proxies.json"
+    return path, json.loads(path.read_text(encoding="utf-8"))
+
+
+def save_rows(path: Path, rows: List[dict]) -> None:
+    atomic_write(path, json.dumps(rows, indent=1, ensure_ascii=False))
+
+
+def https_rows(rows: Iterable[dict]) -> List[dict]:
+    """Only these can tunnel to an https:// site – the others failed the HTTPS test in the run."""
+    return [r for r in rows if r.get("https") is True]
+
+
+def resolve_ipv4(host: str, port: int = 443) -> str:
+    """Looked up once per step: SOCKS4 needs the IP, and a thousand lookups of the same host help nobody."""
+    return socket.getaddrinfo(host, port, family=socket.AF_INET)[0][4][0]
+
+
+async def run_limited(jobs: Iterable[Callable[[], Awaitable[None]]], concurrency: int) -> None:
+    """Run the jobs with at most `concurrency` at a time."""
+    slots = asyncio.Semaphore(concurrency)
+
+    async def one(job: Callable[[], Awaitable[None]]) -> None:
+        async with slots:
+            await job()
+
+    await asyncio.gather(*(one(job) for job in jobs))
+
+
+class Tunnels:
+    """Verified TLS tunnels to a host through a proxy – the same handshake code the checks use."""
+
+    def __init__(self, timeout: float):
+        from .checker import Checker  # only here: the checker pulls in more than these steps need
+
+        self.checker = Checker(judge_ip="127.0.0.1", own_ips=(), timeout=timeout, connect_timeout=timeout / 2)
+
+    async def open(self, row: dict, host: str, ip: str, port: int = 443):
+        """(reader, writer) of the TLS connection, or None if the proxy refused the tunnel or broke up the TLS."""
+        return await self.checker.tls_tunnel(row["ptype"], row["proxy"], host, ip, port)
+
+
+def browser_get(host: str, path: str, *extra: str) -> bytes:
+    """A plain GET the way a browser would send it (so sites answer as they would to a person)."""
+    from .netio import USER_AGENT
+
+    lines = [f"GET {path} HTTP/1.1", f"Host: {host}", f"User-Agent: {USER_AGENT}",
+             "Accept: text/html,application/xhtml+xml,*/*;q=0.8", "Accept-Language: en-US,en;q=0.8", *extra,
+             "Connection: close"]
+    return ("\r\n".join(lines) + "\r\n\r\n").encode()
+