proxy-scraper-cli 1.16.0__tar.gz → 1.18.0__tar.gz

{proxy_scraper_cli-1.16.0 → proxy_scraper_cli-1.18.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: proxy-scraper-cli
-Version: 1.16.0
+Version: 1.18.0
 Summary: Finds free HTTP, SOCKS4 and SOCKS5 proxies that actually work: 700+ sources, real checks, honeypot filtering and a rotating local proxy server.
 Author: Maximilian Feix
 License-Expression: MIT
@@ -704,6 +704,7 @@ pip download requests   # git, pip, npm & co. now go through the pool
     types: socks5
     https: true
     min-uptime: 90        # on the list for 90 %+ of the week
+    min-speed: 100        # optional: downloaded 100+ KB/s in the last check
     works-on: google      # optional: google, reddit, amazon
     recheck: true         # optional: check them again from the runner
 - run: curl -x "${{ steps.proxies.outputs.proxy }}" https://api.ipify.org
@@ -716,8 +717,12 @@ pip download requests   # git, pip, npm & co. now go through the pool
 ```bash
 curl -x "$(proxy-scraper --pick --https-only --min-uptime 90)" https://api.ipify.org
 proxy-scraper --pick 10 --country DE --works-on google > de.txt
+proxy-scraper --pick 5 --min-speed 200     # only ones that downloaded 200+ KB/s
 ```
 
+"Fastest first" means the first answer plus the download speed measured in the last check – in a test with real
+pages, the 25 fastest by download loaded four times as many pages as the 25 quickest to answer a tiny request.
+
 **Without installing anything** – straight from the live list
 
 ```bash
@@ -762,7 +767,7 @@ curl.exe -x (Get-Content "$run\all.txt" -TotalCount 1) http://api.ipify.org
 | `--discover` | search GitHub for new sources right now |
 | `--no-cache` | download every list again (unchanged ones are normally skipped via ETag) |
 | `--list-sources [N]` | show the source ranking (add `--json` for scripts: url, status, hit rate, checks, last change) |
-| `--pick [N]` | print N proxies (default 1) from the hourly checked list and exit – no scan, same filters, plus `--min-uptime PERCENT` and `--works-on google,reddit,…` |
+| `--pick [N]` | print N proxies (default 1) from the hourly checked list and exit – no scan, same filters, plus `--min-uptime PERCENT`, `--min-speed KBPS` and `--works-on google,reddit,…` |
 | `--serve-host ADDR` | where the proxy server listens (default `127.0.0.1`; `0.0.0.0` for Docker, with a warning) |
 | `--serve-password SECRET` | clients must send this password in the proxy login; better set `PROXY_SCRAPER_SERVE_PASSWORD` |
 | `--rotate STRATEGY` · `--sticky SEC` | how the proxy server picks proxies, see [above](#proxy-server) |

{proxy_scraper_cli-1.16.0 → proxy_scraper_cli-1.18.0}/README.md

@@ -663,6 +663,7 @@ pip download requests   # git, pip, npm & co. now go through the pool
     types: socks5
     https: true
     min-uptime: 90        # on the list for 90 %+ of the week
+    min-speed: 100        # optional: downloaded 100+ KB/s in the last check
     works-on: google      # optional: google, reddit, amazon
     recheck: true         # optional: check them again from the runner
 - run: curl -x "${{ steps.proxies.outputs.proxy }}" https://api.ipify.org
@@ -675,8 +676,12 @@ pip download requests   # git, pip, npm & co. now go through the pool
 ```bash
 curl -x "$(proxy-scraper --pick --https-only --min-uptime 90)" https://api.ipify.org
 proxy-scraper --pick 10 --country DE --works-on google > de.txt
+proxy-scraper --pick 5 --min-speed 200     # only ones that downloaded 200+ KB/s
 ```
 
+"Fastest first" means the first answer plus the download speed measured in the last check – in a test with real
+pages, the 25 fastest by download loaded four times as many pages as the 25 quickest to answer a tiny request.
+
 **Without installing anything** – straight from the live list
 
 ```bash
@@ -721,7 +726,7 @@ curl.exe -x (Get-Content "$run\all.txt" -TotalCount 1) http://api.ipify.org
 | `--discover` | search GitHub for new sources right now |
 | `--no-cache` | download every list again (unchanged ones are normally skipped via ETag) |
 | `--list-sources [N]` | show the source ranking (add `--json` for scripts: url, status, hit rate, checks, last change) |
-| `--pick [N]` | print N proxies (default 1) from the hourly checked list and exit – no scan, same filters, plus `--min-uptime PERCENT` and `--works-on google,reddit,…` |
+| `--pick [N]` | print N proxies (default 1) from the hourly checked list and exit – no scan, same filters, plus `--min-uptime PERCENT`, `--min-speed KBPS` and `--works-on google,reddit,…` |
 | `--serve-host ADDR` | where the proxy server listens (default `127.0.0.1`; `0.0.0.0` for Docker, with a warning) |
 | `--serve-password SECRET` | clients must send this password in the proxy login; better set `PROXY_SCRAPER_SERVE_PASSWORD` |
 | `--rotate STRATEGY` · `--sticky SEC` | how the proxy server picks proxies, see [above](#proxy-server) |

{proxy_scraper_cli-1.16.0 → proxy_scraper_cli-1.18.0}/proxy_scraper_cli.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: proxy-scraper-cli
-Version: 1.16.0
+Version: 1.18.0
 Summary: Finds free HTTP, SOCKS4 and SOCKS5 proxies that actually work: 700+ sources, real checks, honeypot filtering and a rotating local proxy server.
 Author: Maximilian Feix
 License-Expression: MIT
@@ -704,6 +704,7 @@ pip download requests   # git, pip, npm & co. now go through the pool
     types: socks5
     https: true
     min-uptime: 90        # on the list for 90 %+ of the week
+    min-speed: 100        # optional: downloaded 100+ KB/s in the last check
     works-on: google      # optional: google, reddit, amazon
     recheck: true         # optional: check them again from the runner
 - run: curl -x "${{ steps.proxies.outputs.proxy }}" https://api.ipify.org
@@ -716,8 +717,12 @@ pip download requests   # git, pip, npm & co. now go through the pool
 ```bash
 curl -x "$(proxy-scraper --pick --https-only --min-uptime 90)" https://api.ipify.org
 proxy-scraper --pick 10 --country DE --works-on google > de.txt
+proxy-scraper --pick 5 --min-speed 200     # only ones that downloaded 200+ KB/s
 ```
 
+"Fastest first" means the first answer plus the download speed measured in the last check – in a test with real
+pages, the 25 fastest by download loaded four times as many pages as the 25 quickest to answer a tiny request.
+
 **Without installing anything** – straight from the live list
 
 ```bash
@@ -762,7 +767,7 @@ curl.exe -x (Get-Content "$run\all.txt" -TotalCount 1) http://api.ipify.org
 | `--discover` | search GitHub for new sources right now |
 | `--no-cache` | download every list again (unchanged ones are normally skipped via ETag) |
 | `--list-sources [N]` | show the source ranking (add `--json` for scripts: url, status, hit rate, checks, last change) |
-| `--pick [N]` | print N proxies (default 1) from the hourly checked list and exit – no scan, same filters, plus `--min-uptime PERCENT` and `--works-on google,reddit,…` |
+| `--pick [N]` | print N proxies (default 1) from the hourly checked list and exit – no scan, same filters, plus `--min-uptime PERCENT`, `--min-speed KBPS` and `--works-on google,reddit,…` |
 | `--serve-host ADDR` | where the proxy server listens (default `127.0.0.1`; `0.0.0.0` for Docker, with a warning) |
 | `--serve-password SECRET` | clients must send this password in the proxy login; better set `PROXY_SCRAPER_SERVE_PASSWORD` |
 | `--rotate STRATEGY` · `--sticky SEC` | how the proxy server picks proxies, see [above](#proxy-server) |

{proxy_scraper_cli-1.16.0 → proxy_scraper_cli-1.18.0}/proxy_scraper_cli.egg-info/SOURCES.txt

@@ -109,6 +109,7 @@ tests/test_server_fixes.py
 tests/test_sites.py
 tests/test_sources_json.py
 tests/test_speed.py
+tests/test_speed_ranking.py
 tests/test_tamper.py
 tests/test_targets.py
 tests/test_ui.py

{proxy_scraper_cli-1.16.0 → proxy_scraper_cli-1.18.0}/proxyscraper/__init__.py

@@ -1,7 +1,7 @@
 """proxy-scraper: collects public HTTP/SOCKS4/SOCKS5 proxies from hundreds of sources, checks them
 in parallel with its own protocol handshakes and learns which sources deliver good proxies."""
 
-__version__ = "1.16.0"
+__version__ = "1.18.0"
 
 
 def __getattr__(name):

{proxy_scraper_cli-1.16.0 → proxy_scraper_cli-1.18.0}/proxyscraper/action.py

@@ -63,7 +63,7 @@ def main() -> int:
         filters = {"types": _list("PS_TYPES") or ["http", "socks4", "socks5"],
                    "countries": os.environ.get("PS_COUNTRIES", "").strip(),
                    "https": _flag("PS_HTTPS"), "min_uptime": _number("PS_MIN_UPTIME", 0),
-                   "works_on": _list("PS_WORKS_ON")}
+                   "min_speed": _number("PS_MIN_SPEED", 0), "works_on": _list("PS_WORKS_ON")}
         # a recheck starts from all of them – the limit applies to what still works from here
         found = live_proxies(**filters, limit=0 if recheck else limit)
     except (InputError, ValueError) as e:
@@ -75,7 +75,9 @@ def main() -> int:
     urls = [p.url for p in found]
     if recheck and urls:
         print(f"Checking {len(urls)} proxies again from this runner …")
-        urls = [r.url for r in sorted(check_proxies(urls, https=filters["https"]), key=lambda r: r.latency)]
+        # in the live list's order: it's ranked by page load, the recheck only knows the first answer
+        working = {r.url for r in check_proxies(urls, https=filters["https"])}
+        urls = [u for u in urls if u in working]
     if limit:
         urls = urls[:limit]
     # a file of its own per step: a job may use the action twice (socks5 in one step, http in the next)

{proxy_scraper_cli-1.16.0 → proxy_scraper_cli-1.18.0}/proxyscraper/agent.py

@@ -429,7 +429,8 @@ def _as_result(r: dict) -> Optional[CheckResult]:
         return CheckResult(make_key(r["ptype"], r["proxy"]), r["ptype"], r["proxy"], int(r.get("latency") or 0),
                            str(r.get("exit_ip") or ""), https=r.get("https"), anonymity=r.get("anonymity") or "",
                            country=r.get("country") or "", asn=int(r.get("asn") or 0), org=r.get("org") or "",
-                           hosting=r.get("hosting"), blocklisted=r.get("blocklisted"))
+                           hosting=r.get("hosting"), blocklisted=r.get("blocklisted"),
+                           speed_kbps=r["speed_kbps"] if type(r.get("speed_kbps")) is int else None)
     except (KeyError, TypeError, ValueError):
         return None
 

{proxy_scraper_cli-1.16.0 → proxy_scraper_cli-1.18.0}/proxyscraper/api.py

@@ -204,8 +204,9 @@ async def live_proxies_async(*, types: Iterable[str] = PROXY_TYPES, countries: I
                              anonymity: str = "", max_latency: int = 0, no_datacenter: bool = False,
                              no_blocklisted: bool = False, min_uptime: int = 0, works_on: Iterable[str] = (),
                              min_speed: int = 0, limit: int = 0) -> List[LiveProxy]:
-    """The proxies from the hourly list that pass the filters, fastest first. Nothing is checked here: they
-    worked from GitHub's servers in the last run (at most an hour ago). find_proxies checks from your network.
+    """The proxies from the hourly list that pass the filters, fastest first – by first answer plus measured
+    download speed. Nothing is checked here: they worked from GitHub's servers in the last run (at most an hour
+    ago). find_proxies checks from your network.
 
     Same filters as find_proxies, plus
     min_uptime  only proxies listed in at least this share (percent) of the week's runs, e.g. 90
@@ -215,6 +216,7 @@ async def live_proxies_async(*, types: Iterable[str] = PROXY_TYPES, countries: I
     limit       at most this many (0 = all)
     """
     from .agent import AgentError, LiveSource
+    from .server.pool import page_ms, typical_speed
     from .sites import SITE
 
     types = _types(types)
@@ -237,9 +239,11 @@ async def live_proxies_async(*, types: Iterable[str] = PROXY_TYPES, countries: I
         data = await LiveSource(fetch=_live_fetch).get()
     except AgentError as e:
         raise ConnectionError(str(e)) from None
-    found = sorted((_live_proxy(r, data.run_hours) for r in data.rows
-                    if r.get("ptype") in types and isinstance(r.get("proxy"), str)),
-                   key=lambda p: p.latency)
+    found = [_live_proxy(r, data.run_hours) for r in data.rows
+             if r.get("ptype") in types and isinstance(r.get("proxy"), str)]
+    # fastest first – by how quickly a page comes through when the list has download speeds (#186)
+    typical = typical_speed(found)
+    found.sort(key=lambda p: page_ms(p, typical))
     found = [p for p in found if opts.filters.accepts(p) and (not min_uptime or (p.uptime_7d or 0) >= min_uptime)
              and all(p.sites.get(s) for s in works_on) and (not min_speed or (p.speed_kbps or 0) >= min_speed)]
     return found[:limit] if limit else found

{proxy_scraper_cli-1.16.0 → proxy_scraper_cli-1.18.0}/proxyscraper/checker.py

@@ -93,6 +93,7 @@ class CheckResult:
     org: str = ""
     hosting: Optional[bool] = None  # exit probably in a datacenter? None = unknown
     blocklisted: Optional[bool] = None  # exit IP on the SpamCop blocklist? None = unknown
+    speed_kbps: Optional[int] = None  # download speed from the live list (KiB/s), None = not measured
 
     @property
     def url(self) -> str:

{proxy_scraper_cli-1.16.0 → proxy_scraper_cli-1.18.0}/proxyscraper/cli.py

@@ -71,7 +71,8 @@ def pick(args) -> int:
         found = live_proxies(types=args.types, countries=args.country or "", https=args.https_only,
                              anonymity=args.anonymity or "", max_latency=args.max_latency,
                              no_datacenter=args.no_datacenter, no_blocklisted=args.no_blocklisted,
-                             min_uptime=args.min_uptime, works_on=args.works_on, limit=args.pick)
+                             min_uptime=args.min_uptime, works_on=args.works_on, min_speed=args.min_speed,
+                             limit=args.pick)
     except (ConnectionError, ValueError) as e:
         print(f"proxy-scraper: {e}", file=sys.stderr)
         return 1
@@ -204,6 +205,8 @@ def parse_args(argv: Optional[List[str]] = None) -> argparse.Namespace:
                            "one URL per line, and exit – e.g. curl -x \"$(proxy-scraper --pick --https-only)\" …")
     live.add_argument("--min-uptime", type=percent, default=0, metavar="PERCENT",
                       help="with --pick: only proxies on the list in at least this share of the week's checks")
+    live.add_argument("--min-speed", type=positive_int, default=0, metavar="KBPS",
+                      help="with --pick: only proxies that downloaded at least this many KB/s in the last check")
     live.add_argument("--works-on", type=site_list, default=[], metavar="SITES",
                       help=f"with --pick: only proxies that got through to these sites in the last check "
                            f"({', '.join(s.name for s in SITES)})")
@@ -251,8 +254,8 @@ def parse_args(argv: Optional[List[str]] = None) -> argparse.Namespace:
     args = p.parse_args(argv)
     if args.json and args.list_sources is None:
         p.error("--json only works together with --list-sources")
-    if (args.min_uptime or args.works_on) and args.pick is None:
-        p.error("--min-uptime and --works-on only work together with --pick")
+    if (args.min_uptime or args.works_on or args.min_speed) and args.pick is None:
+        p.error("--min-uptime, --min-speed and --works-on only work together with --pick")
     if args.pick is not None:  # --pick exits right away: options for a scan or the server would be silently ignored
         scan_only = [flag for dest, flag in (("target", "--target"), ("serve", "--serve"), ("recheck", "--recheck"),
                                              ("want", "--want"), ("output", "-o/--output"), ("export", "--export"))

{proxy_scraper_cli-1.16.0 → proxy_scraper_cli-1.18.0}/proxyscraper/pages.py

@@ -7,6 +7,8 @@ that answers exactly that, and every page links back to the full, filterable lis
 
 from __future__ import annotations
 
+import json
+import statistics
 from datetime import datetime
 from html import escape
 from pathlib import Path
@@ -100,6 +102,22 @@ footer { color: var(--muted); font-size: 14px; display: grid; gap: 6px; }
 """
 
 
+PAGE_CSS = """
+.wrap > * { min-width: 0; }  /* grid items default to their content's width: the wide table would push the page */
+.facts { display: grid; grid-template-columns: repeat(auto-fit, minmax(min(100%, 150px), 1fr)); gap: 1px; margin: 0;
+         background: var(--line); border: 1px solid var(--line); border-radius: 16px; overflow: hidden; }
+.facts div { background: var(--panel); padding: 16px 18px; }
+.facts dt { font-size: 26px; font-weight: 600; letter-spacing: -.02em; font-variant-numeric: tabular-nums; }
+.facts dd { margin: 2px 0 0; color: var(--muted); font-size: 14px; }
+h2 { font-size: 24px; letter-spacing: -.02em; margin: 0 0 12px; font-weight: 600; }
+.use pre { background: var(--panel); border: 1px solid var(--line); border-radius: 12px; padding: 14px 16px;
+           overflow-x: auto; font-size: 14px; margin: 0 0 10px; }
+.note { color: var(--muted); font-size: 14px; }
+.faq h3 { font-size: 18px; font-weight: 500; margin: 22px 0 6px; }
+.faq p { margin: 0; color: var(--muted); max-width: 70ch; }
+"""
+
+
 def _list_query(path: str) -> str:
     """The same filter on the full list: the website reads its filters from the address."""
     kind = path.strip("/")
@@ -109,6 +127,61 @@ def _list_query(path: str) -> str:
             "elite": "?elite=1"}.get(kind, "")
 
 
+SITE_NAMES = {"tiktok": "TikTok"}
+
+
+def _pick_flags(path: str) -> str:
+    """The --pick filters that give the same selection as this page."""
+    kind = path.strip("/")
+    if kind.startswith("country/"):
+        return f"--country {kind.split('/')[1].upper()}"
+    return {"http": "--types http", "socks4": "--types socks4", "socks5": "--types socks5", "https": "--https-only",
+            "elite": "--anonymity elite"}.get(kind, "")
+
+
+def _short(path: str) -> str:
+    """How a page's proxies are called in a question: "proxies in Germany", "SOCKS5 proxies"."""
+    kind = path.strip("/")
+    if kind.startswith("country/"):
+        return f"proxies in {_short_name(COUNTRIES.get(kind.split('/')[1].upper(), kind.split('/')[1].upper()))}"
+    return {"https": "HTTPS proxies", "elite": "elite proxies"}.get(kind, f"{kind.upper()} proxies")
+
+
+def _facts(rows: List[dict]) -> Dict[str, object]:
+    speeds = [r["speed_kbps"] for r in rows if r.get("speed_kbps")]
+    return {"total": len(rows), "https": sum(1 for r in rows if r.get("https")),
+            "stable": sum(1 for r in rows if (r.get("uptime_7d") or 0) >= 90),
+            # None when none of them was tried: the site check only runs on HTTPS-capable ones, and not at all
+            # when it couldn't resolve the site – "0 get through" would blame proxies for a check that didn't run
+            "google": (sum(1 for r in rows if (r.get("sites") or {}).get("google"))
+                       if any("google" in (r.get("sites") or {}) for r in rows) else None),
+            "speed": round(statistics.median(speeds)) if speeds else None}
+
+
+def _faq(path: str, facts: Dict[str, object], when: str) -> List[Tuple[str, str]]:
+    short, flags = _short(path), _pick_flags(path)
+    if facts["google"] is None:
+        got = "These proxies weren't part of the last site check – it only tries the ones that tunnel HTTPS."
+    elif facts["google"]:
+        got = f"In the last check, {facts['google']:,} of them got through to Google search without a captcha."
+    else:
+        got = "In the last check, none of them got through to Google search without a captcha."
+    return [
+        (f"How many free {short} work right now?",
+         f"{facts['total']:,} passed every check in the last run ({when}): a real handshake, a honeypot check on two "
+         f"sites and a content check. {facts['https']:,} of them tunnel HTTPS with verified TLS, and "
+         f"{facts['stable']:,} were on the list in 90 % or more of this week's hourly checks. The list is checked "
+         "again every hour."),
+        (f"Which free {short} get through to Google?",
+         f"{got} The site check also tries Reddit, Amazon, Instagram and TikTok – the full list "
+         "on the website can be filtered by it."),
+        (f"How do I get working {short} in code or the terminal?",
+         f"Without installing anything, download proxies.txt from this page. With the tool: pipx install "
+         f"proxy-scraper-cli, then proxy-scraper --pick 5 {flags} prints five from the hourly list. From Python, "
+         "live_proxies() takes the same filters."),
+    ]
+
+
 def _render(path: str, title: str, what: str, rows: List[dict], updated: datetime, nav: str) -> str:
     depth = path.count("/")
     root = "../" * depth
@@ -120,19 +193,49 @@ def _render(path: str, title: str, what: str, rows: List[dict], updated: datetim
     else:
         lede = (f"No {what} passed every check in the last run ({when}). The list is checked again every hour, "
                 "so look again later or try the full list.")
-    body_rows = "\n".join(
-        f"<tr><td class=\"mono\"><a href=\"{root}proxy/{escape(r['proxy'].replace(':', '-'))}/\">"
-        f"{escape(r['proxy'])}</a></td><td>{escape(r['ptype'].upper())}</td>"
-        f"<td>{escape(r.get('country') or '–')}</td><td>{r['latency']:,} ms</td>"
-        f"<td>{'yes' if r.get('https') else 'no'}</td><td>{escape(r.get('anonymity') or '–')}</td>"
-        f"<td>{escape((r.get('org') or '–')[:40])}</td></tr>"
-        for r in rows[:ROWS_PER_PAGE])
+    def cells(r: dict) -> str:
+        through = ", ".join(SITE_NAMES.get(n, n.title()) for n, ok in (r.get("sites") or {}).items() if ok) or "–"
+        return (f"<tr><td class=\"mono\"><a href=\"{root}proxy/{escape(r['proxy'].replace(':', '-'))}/\">"
+                f"{escape(r['proxy'])}</a></td><td>{escape(r['ptype'].upper())}</td>"
+                f"<td>{escape(r.get('country') or '–')}</td><td>{r['latency']:,} ms</td>"
+                f"<td>{(str(r['speed_kbps']) + ' KB/s') if r.get('speed_kbps') else '–'}</td>"
+                f"<td>{(str(r['uptime_7d']) + ' %') if r.get('uptime_7d') is not None else '–'}</td>"
+                f"<td>{'yes' if r.get('https') else 'no'}</td><td>{escape(through)}</td>"
+                f"<td>{escape((r.get('org') or '–')[:40])}</td></tr>")
+    body_rows = "\n".join(cells(r) for r in rows[:ROWS_PER_PAGE])
     table = (f"<div class=\"table\"><table><caption>The {min(total, ROWS_PER_PAGE):,} fastest, as ip:port. "
              f"The download has all {total:,} as type://ip:port.</caption>"
              "<thead><tr><th scope=\"col\">Proxy</th><th scope=\"col\">Type</th><th scope=\"col\">Country</th>"
-             "<th scope=\"col\">Latency</th><th scope=\"col\">HTTPS</th><th scope=\"col\">Anonymity</th>"
+             "<th scope=\"col\">Latency</th><th scope=\"col\">Speed</th><th scope=\"col\">Uptime</th>"
+             "<th scope=\"col\">HTTPS</th><th scope=\"col\">Gets through</th>"
              f"<th scope=\"col\">Provider</th></tr></thead><tbody>{body_rows}</tbody></table></div>"
              if total else "")
+    facts = _facts(rows)
+    facts_html = "" if not total else (
+        "<dl class=\"facts\">"
+        f"<div><dt>{facts['total']:,}</dt><dd>working right now</dd></div>"
+        f"<div><dt>{facts['https']:,}</dt><dd>tunnel HTTPS</dd></div>"
+        f"<div><dt>{facts['stable']:,}</dt><dd>on the list 90 %+ of the week</dd></div>"
+        + (f"<div><dt>{facts['google']:,}</dt><dd>get through to Google</dd></div>"
+           if facts["google"] is not None else "")
+        + (f"<div><dt>{facts['speed']:,} KB/s</dt><dd>median download speed</dd></div>" if facts["speed"] else "")
+        + "</dl>")
+    flags = _pick_flags(path)
+    # curl goes to an https:// address, so only proxies that can tunnel it
+    curl_flags = flags if "--https-only" in flags else f"{flags} --https-only".strip()
+    commands = (
+        "<section class=\"use\"><h2>In the terminal or in code</h2>"
+        f"<pre class=\"mono\">pipx install proxy-scraper-cli\nproxy-scraper --pick 5 {escape(flags)}\n"
+        f"curl -x \"$(proxy-scraper --pick {escape(curl_flags)})\" https://api.ipify.org</pre>"
+        "<p class=\"note\">--pick takes proxies from the same hourly list in about half a second. "
+        f"From Python: <code>live_proxies()</code>, see the <a href=\"{REPO_URL}#from-python\">README</a>."
+        "</p></section>")
+    faq = _faq(path, facts, when)
+    faq_html = "<section class=\"faq\"><h2>Questions</h2>" + "".join(
+        f"<h3>{escape(q)}</h3><p>{escape(a)}</p>" for q, a in faq) + "</section>"
+    faq_ld = json.dumps({"@context": "https://schema.org", "@type": "FAQPage", "mainEntity": [
+        {"@type": "Question", "name": q, "acceptedAnswer": {"@type": "Answer", "text": a}} for q, a in faq]},
+        ensure_ascii=False).replace("</", "<\\/")
     description = (f"{total:,} free {what}, checked every hour: real handshake, honeypot and content check. "
                    "Download as text or filter the full list.")
     return f"""<!doctype html>
@@ -154,7 +257,8 @@ def _render(path: str, title: str, what: str, rows: List[dict], updated: datetim
 <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
 <link href="https://fonts.googleapis.com/css2?family=Geist:wght@400..700&family=Geist+Mono:wght@400;500&display=swap"
       rel="stylesheet">
-<style>{CSS}</style>
+<style>{CSS}{PAGE_CSS}</style>
+<script type="application/ld+json">{faq_ld}</script>
 </head>
 <body>
 <div class="wrap">
@@ -171,7 +275,10 @@ def _render(path: str, title: str, what: str, rows: List[dict], updated: datetim
       <a class="btn" href="{REPO_URL}">Check them from your network</a>
     </div>
   </main>
+  {facts_html}
   {table}
+  {commands}
+  {faq_html}
   {nav}
   <footer>
     <span>Free proxies are run by strangers. Never send passwords or personal data through them.</span>

{proxy_scraper_cli-1.16.0 → proxy_scraper_cli-1.18.0}/proxyscraper/server/pool.py

@@ -4,12 +4,16 @@ Selection per connection in three steps:
   1. filters from the request (Selection): country, type – via the user name, e.g. "country-de-type-socks5"
   2. sticky: the same session (or with --sticky the same target site) keeps the same proxy for a while
   3. strategy for everything else: weighted (default), random, round-robin, fastest
+
+"Fast" means how quickly a page comes through, not just the first answer: when the proxies come from the live list,
+which measures a download per proxy every hour, that download counts too (#186).
 """
 
 from __future__ import annotations
 
 import random
 import re
+import statistics
 import time
 from dataclasses import dataclass
 from typing import Callable, Dict, List, Optional, Set, Tuple
@@ -20,6 +24,25 @@ DISABLE_AFTER = 3           # this many failures in a row -> out of the rotation
 STRATEGIES = ("weighted", "random", "round-robin", "fastest")
 SESSION_SECONDS = 600       # how long a session (session-…) keeps its proxy when --sticky isn't set
 _TOKEN_RE = re.compile(r"(country|type|session)[-_]([A-Za-z0-9]+)")
+PAGE_KB = 100               # roughly one page – the download the speed step measures
+UNMEASURED_KBPS = 20        # tried in the speed step, no download got through: counted as slow (it passed the checks)
+
+
+def page_ms(result, typical_kbps: Optional[float]) -> float:
+    """Roughly how long a page takes through this proxy: the first answer plus the download.
+
+    typical_kbps is the median measured speed of the set, None when it has no speeds at all (an own scan) – then
+    latency decides alone. The speed step only tries HTTPS-capable proxies: one of those without a speed didn't
+    get the download through, the others were never tried and count as typical."""
+    if not typical_kbps:
+        return result.latency
+    kbps = result.speed_kbps or (UNMEASURED_KBPS if result.https else typical_kbps)
+    return result.latency + PAGE_KB * 1000 / kbps
+
+
+def typical_speed(results) -> Optional[float]:
+    speeds = [r.speed_kbps for r in results if r.speed_kbps]
+    return statistics.median(speeds) if speeds else None
 
 
 @dataclass(frozen=True)
@@ -57,12 +80,17 @@ class PoolEntry:
     fail_streak: int = 0
     active: int = 0
     disabled: bool = False
+    typical_kbps: Optional[float] = None  # set by the pool: median measured speed of its proxies
+
+    @property
+    def cost(self) -> float:
+        return page_ms(self.result, self.typical_kbps)
 
     @property
     def weight(self) -> float:
         # prefer fast and proven ones, but give everyone a chance
         reliability = (self.ok + 1) / (self.ok + self.fail + 2)
-        return reliability / (self.result.latency + 300)
+        return reliability / (self.cost + 300)
 
 
 class ProxyPool:
@@ -71,7 +99,8 @@ class ProxyPool:
                  clock: Callable[[], float] = time.monotonic, strict_tls: bool = False):
         if strategy not in STRATEGIES:
             raise ValueError(f"unknown strategy {strategy!r} (possible: {', '.join(STRATEGIES)})")
-        self.entries = [PoolEntry(r) for r in sorted(results, key=lambda r: r.latency)]
+        typical = typical_speed(results)
+        self.entries = sorted((PoolEntry(r, typical_kbps=typical) for r in results), key=lambda e: e.cost)
         self.rng = rng or random.Random()
         self.strategy = strategy
         self.sticky_seconds = sticky_seconds
@@ -135,8 +164,8 @@ class ProxyPool:
             # the fastest free one; if all are busy, the least busy one (then the faster one)
             idle = [e for e in candidates if not e.active]
             if idle:
-                return min(idle, key=lambda e: (e.result.latency, -e.weight))
-            return min(candidates, key=lambda e: (e.active, e.result.latency))
+                return min(idle, key=lambda e: (e.cost, -e.weight))
+            return min(candidates, key=lambda e: (e.active, e.cost))
         if self.strategy == "round-robin":
             ordered = sorted(candidates, key=lambda e: e.result.key)
             entry = ordered[self._next % len(ordered)]
@@ -186,7 +215,10 @@ class ProxyPool:
         for r in fresh.values():
             kept.append(PoolEntry(r))
             added += 1
-        self.entries = sorted(kept, key=lambda e: e.result.latency)
+        typical = typical_speed([e.result for e in kept])
+        for entry in kept:
+            entry.typical_kbps = typical
+        self.entries = sorted(kept, key=lambda e: e.cost)
         return added
 
     def revive(self, entry: PoolEntry) -> None:

{proxy_scraper_cli-1.16.0 → proxy_scraper_cli-1.18.0}/tests/test_action.py

@@ -11,7 +11,7 @@ def fake(n, **extra):
             for i in range(n)]
 
 
-def run(monkeypatch, tmp_path, env, found=3, rechecked=None):
+def run(monkeypatch, tmp_path, env, found=3, rechecked=None, recheck=None):
     calls = {}
 
     def live(**kwargs):
@@ -20,7 +20,7 @@ def run(monkeypatch, tmp_path, env, found=3, rechecked=None):
 
     def check(urls, **kwargs):
         calls["check"] = list(urls)
-        return fake(found)[:rechecked]
+        return recheck(found) if recheck else fake(found)[:rechecked]
     monkeypatch.setattr(action, "live_proxies", live)
     monkeypatch.setattr(action, "check_proxies", check)
     out = tmp_path / "out.txt"
@@ -36,10 +36,10 @@ def run(monkeypatch, tmp_path, env, found=3, rechecked=None):
 def test_filters_reach_live_proxies_and_outputs_are_set(monkeypatch, tmp_path):
     code, out, calls = run(monkeypatch, tmp_path, {
         "PS_TYPES": "socks5, http", "PS_COUNTRIES": "de,nl", "PS_HTTPS": "true", "PS_MIN_UPTIME": "90",
-        "PS_WORKS_ON": "google", "PS_LIMIT": "2"})
+        "PS_MIN_SPEED": "100", "PS_WORKS_ON": "google", "PS_LIMIT": "2"})
     assert code == 0
     assert calls["live"] == {"types": ["socks5", "http"], "countries": "de,nl", "https": True, "min_uptime": 90,
-                             "works_on": ["google"], "limit": 2}
+                             "min_speed": 100, "works_on": ["google"], "limit": 2}
     assert out["proxy"] == "socks5://1.1.1.0:1080" and out["count"] == "2"  # limit
     file = out["file"]
     assert file.startswith(str(tmp_path)) and Path(file).read_text().splitlines()[0] == "socks5://1.1.1.0:1080"
@@ -67,3 +67,12 @@ def test_nothing_found_fails_the_step_unless_allowed(monkeypatch, tmp_path, caps
 def test_bad_inputs_are_explained(monkeypatch, tmp_path, capsys):
     code, _, _ = run(monkeypatch, tmp_path, {"PS_LIMIT": "lots"})
     assert code == 1 and "limit" in capsys.readouterr().out
+
+
+
+def test_a_recheck_keeps_the_live_lists_order(monkeypatch, tmp_path):
+    # the live list comes ranked by page load; the recheck's latencies say nothing about downloads
+    def slower_first(n):  # from the runner, the list's first proxy happens to answer slowest
+        return [CheckResult(r.key, r.ptype, r.proxy, 500 - r.latency, r.exit_ip, True) for r in fake(n)]
+    _, out, _ = run(monkeypatch, tmp_path, {"PS_RECHECK": "true"}, recheck=slower_first)
+    assert out["proxy"] == "socks5://1.1.1.0:1080"

{proxy_scraper_cli-1.16.0 → proxy_scraper_cli-1.18.0}/tests/test_pages.py

@@ -1,5 +1,6 @@
 """Static pages per protocol and country for search engines, plus the sitemap (#121)."""
 
+import html
 from datetime import datetime, timezone
 
 from proxyscraper import pages
@@ -58,3 +59,44 @@ def test_filter_button_opens_the_list_with_the_same_filter(tmp_path):
     for path, query in expected.items():
         html = (tmp_path / path / "index.html").read_text(encoding="utf-8")
         assert f'href="{"../" * (path.count("/") + 1)}{query}#list">Filter the full list' in html
+
+
+def test_a_country_page_has_the_numbers_commands_and_answers(tmp_path):
+    import json
+    import re
+    rows = [row(1, "socks5", https=True, uptime_7d=95, speed_kbps=400, sites={"google": True}),
+            row(2, "http", https=False, uptime_7d=20),
+            row(3, "http", country="US")]
+    pages.write_pages(rows, tmp_path, NOW)
+    de = (tmp_path / "country" / "de" / "index.html").read_text(encoding="utf-8")
+    facts = re.sub(r"\s+", " ", re.sub(r"<[^>]+>", " ", de))
+    assert "2 working right now" in facts and "1 tunnel HTTPS" in facts and "1 on the list 90 %+ of the week" in facts
+    assert "1 get through to Google" in facts and "400 KB/s" in facts
+    assert "proxy-scraper --pick 5 --country DE" in de and 'curl -x "$(proxy-scraper --pick --country DE' in de
+    assert ">Speed<" in de and ">Uptime<" in de
+    ld = [json.loads(b) for b in re.findall(r'<script type="application/ld\+json">(.*?)</script>', de, re.S)]
+    faq = next(d for d in ld if d["@type"] == "FAQPage")
+    assert any("Germany" in q["name"] for q in faq["mainEntity"])
+
+
+def test_protocol_pages_pick_by_protocol(tmp_path):
+    pages.write_pages([row(1, "socks5", https=True)], tmp_path, NOW)
+    assert "proxy-scraper --pick 5 --types socks5" in (tmp_path / "socks5" / "index.html").read_text()
+    assert "proxy-scraper --pick 5 --https-only" in (tmp_path / "https" / "index.html").read_text()
+    assert "proxy-scraper --pick 5 --anonymity elite" in (tmp_path / "elite" / "index.html").read_text()
+
+
+def test_an_empty_page_still_renders(tmp_path):
+    pages.write_pages([], tmp_path, NOW)
+    assert "No proxies in Germany" in (tmp_path / "country" / "de" / "index.html").read_text()
+
+
+def test_the_median_speed_is_the_real_median():
+    assert pages._facts([row(1, speed_kbps=100), row(2, speed_kbps=400)])["speed"] == 250
+
+
+def test_pages_dont_claim_google_blocked_proxies_that_were_never_checked(tmp_path):
+    pages.write_pages([row(1, "socks4"), row(2, "socks4")], tmp_path, NOW)
+    page = html.unescape((tmp_path / "socks4" / "index.html").read_text(encoding="utf-8"))
+    assert "none of them got through" not in page and "<dd>get through to Google</dd>" not in page
+    assert "weren't part of the last site check" in page

{proxy_scraper_cli-1.16.0 → proxy_scraper_cli-1.18.0}/tests/test_pick.py

@@ -42,3 +42,10 @@ def test_options_for_a_scan_are_an_error_not_ignored(extra, capsys):
     with pytest.raises(SystemExit):
         cli.run(["--pick", *extra])
     assert "--pick" in capsys.readouterr().err
+
+
+def test_min_speed_keeps_only_proxies_that_downloaded_that_fast(monkeypatch, capsys):
+    rows = [dict(ROWS[0], speed_kbps=300), dict(ROWS[2], speed_kbps=40), ROWS[1]]
+    monkeypatch.setattr(api, "_live_fetch", fetch_from({"proxies.json": rows, "stats.json": STATS}))
+    assert cli.run(["--pick", "5", "--min-speed", "100"]) == 0
+    assert capsys.readouterr().out == "socks5://1.1.1.1:1080\n"

proxy_scraper_cli-1.18.0/tests/test_speed_ranking.py

@@ -0,0 +1,58 @@
+"""Download speed in the choice of proxy (#186): a quick answer to one tiny request says little about loading a
+page – in a test on the live list, the 25 proxies with the best latency loaded 8 of 50 pages, the 25 that downloaded
+fastest in the last run 33 of 50."""
+
+import random
+
+from proxyscraper import api, live_proxies
+from proxyscraper.agent import _as_result
+from proxyscraper.checker import CheckResult
+from proxyscraper.server import ProxyPool
+from proxyscraper.server.pool import page_ms
+
+from .test_live_api import ROWS, STATS, fetch_from
+
+
+def result(n, latency, speed=None, https=True):
+    proxy = f"10.0.0.{n}:80"
+    return CheckResult(f"http {proxy}", "http", proxy, latency, "9.9.9.9", https=https, speed_kbps=speed)
+
+
+def test_a_proxy_that_downloads_fast_beats_one_that_only_answers_fast():
+    quick_ping, fast_download = result(1, 100), result(2, 400, speed=500)
+    assert page_ms(fast_download, typical_kbps=100) < page_ms(quick_ping, typical_kbps=100)
+    pool = ProxyPool([quick_ping, fast_download], strategy="fastest")
+    assert pool.pick(set()).result is fast_download
+
+
+def test_the_weighted_choice_leans_to_the_fast_downloader():
+    pool = ProxyPool([result(1, 100), result(2, 400, speed=500)], rng=random.Random(1))
+    picks = [pool.pick(set()).result.proxy for _ in range(1000)]
+    assert picks.count("10.0.0.2:80") > 700
+
+
+def test_without_any_speed_data_latency_decides_as_before():
+    pool = ProxyPool([result(1, 400), result(2, 100)], strategy="fastest")
+    assert pool.pick(set()).result.latency == 100
+
+
+def test_the_live_list_brings_its_speed_into_the_pool():
+    row = {"ptype": "http", "proxy": "1.2.3.4:80", "latency": 100, "https": True, "speed_kbps": 250}
+    assert _as_result(row).speed_kbps == 250
+    assert _as_result({**row, "speed_kbps": "fast"}).speed_kbps is None
+
+
+def test_live_proxies_put_the_measured_fast_ones_first(monkeypatch):
+    rows = [dict(r) for r in ROWS]
+    rows[0]["speed_kbps"] = 400  # socks5, latency 300 – the slowest to answer, the only one that downloaded
+    monkeypatch.setattr(api, "_live_fetch", fetch_from({"proxies.json": rows, "stats.json": STATS}))
+    # the socks4 one answers faster but got no download through in the speed step
+    assert [p.url for p in live_proxies(https=True)] == ["socks5://1.1.1.1:1080", "socks4://3.3.3.3:4145"]
+
+
+def test_proxies_the_speed_step_never_tried_count_as_typical_not_slow():
+    # it only measures HTTPS-capable ones – a plain-HTTP-only proxy wasn't slow, it just wasn't measured
+    plain_only, measured = result(1, 50, https=False), result(2, 900, speed=100)
+    pool = ProxyPool([plain_only, measured], strategy="fastest")
+    assert pool.pick(set()).result is plain_only
+    assert pool.pick(set(), tls=True).result is measured