proxy-scraper-cli 1.7.1__tar.gz → 1.8.0__tar.gz

{proxy_scraper_cli-1.7.1 → proxy_scraper_cli-1.8.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: proxy-scraper-cli
-Version: 1.7.1
+Version: 1.8.0
 Summary: Finds free HTTP, SOCKS4 and SOCKS5 proxies that actually work: 700+ sources, real checks, honeypot filtering and a rotating local proxy server.
 Author: Maximilian Feix
 License-Expression: MIT
@@ -175,7 +175,13 @@ Keys: <kbd>↑</kbd><kbd>↓</kbd> select · <kbd>Space</kbd> toggle · <kbd>1</
 
 Don't want to scan yourself? Every hour **GitHub Actions** runs the tool and publishes the hits to the [`proxy-list`](../../tree/proxy-list) branch – every entry worked in the last run, fastest first.
 
-**→ [Browse it on the website](https://maximilianfeix.github.io/proxy-scraper/)** – search, filter by type, country, HTTPS, provider and latency, see how long each proxy has been up, copy or download exactly the proxies you need. `streaks.json` on the branch has the number of runs in a row for every proxy.
+<a href="https://maximilianfeix.github.io/proxy-scraper/"><picture>
+  <source media="(prefers-color-scheme: dark)" srcset="https://maximilianfeix.github.io/proxy-scraper/chart-dark.svg">
+  <source media="(prefers-color-scheme: light)" srcset="https://maximilianfeix.github.io/proxy-scraper/chart-light.svg">
+  <img src="https://maximilianfeix.github.io/proxy-scraper/chart-dark.svg" alt="Working proxies over the last days, stacked by protocol – redrawn with every run" width="100%">
+</picture></a>
+
+**→ [Browse it on the website](https://maximilianfeix.github.io/proxy-scraper/)** – search, filter by type, country, HTTPS, provider and latency, see how long each proxy has been up and how often it was on the list this week, copy or download exactly the proxies you need.
 
 Every protocol and country also has its own page with a plain download, e.g. [SOCKS5](https://maximilianfeix.github.io/proxy-scraper/socks5/) or [Germany](https://maximilianfeix.github.io/proxy-scraper/country/de/) (`country/de/proxies.txt`).
 
@@ -197,12 +203,22 @@ Every protocol and country also has its own page with a plain download, e.g. [SO
 | HTTP · SOCKS4 · SOCKS5 | `1.2.3.4:8080` | [http.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/http.txt) · [socks4.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/socks4.txt) · [socks5.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/socks5.txt) |
 | HTTPS-capable only | `type://ip:port` | [https.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/https.txt) |
 | Elite only | `type://ip:port` | [elite.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/elite.txt) |
-| With all details | latency, country, HTTPS, anonymity, exit IP | [proxies.json](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/proxies.json) · [proxies.csv](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/proxies.csv) |
+| Gets through to Google · Reddit · Amazon (no captcha, no 403 in the last run) | `type://ip:port` | [google.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/works-with/google.txt) · [reddit.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/works-with/reddit.txt) · [amazon.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/works-with/amazon.txt) |
+| Stable, on the list in 90 %+ of this week's runs | `type://ip:port` | [stable.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/stable.txt) |
+| With all details | latency, country, HTTPS, anonymity, exit IP, uptime, sites | [proxies.json](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/proxies.json) · [proxies.csv](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/proxies.csv) |
+
+<a href="https://maximilianfeix.github.io/proxy-scraper/"><picture>
+  <source media="(prefers-color-scheme: dark)" srcset="https://maximilianfeix.github.io/proxy-scraper/countries-dark.svg">
+  <source media="(prefers-color-scheme: light)" srcset="https://maximilianfeix.github.io/proxy-scraper/countries-light.svg">
+  <img src="https://maximilianfeix.github.io/proxy-scraper/countries-dark.svg" alt="Countries with the most working proxies right now" width="100%">
+</picture></a>
 
 ```bash
 curl -s https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/socks5.txt | head
 ```
 
+Every file is also on GitHub Pages, which sits behind a CDN, isn't rate limited like raw.githubusercontent and sends CORS headers, so it works straight from the browser: `https://maximilianfeix.github.io/proxy-scraper/socks5.txt`. [jsDelivr](https://cdn.jsdelivr.net/gh/maximilianfeix/proxy-scraper@proxy-list/) works too, but can lag behind by a few hours.
+
 Or let the tool start from it: `proxy-scraper --recheck live` downloads the list and checks it again from **your** network – about 30 seconds instead of a full scan (517 of 1,169 worked from here). With `--serve` you have a rotating proxy in under a minute.
 
 <a id="mcp"></a>
@@ -215,7 +231,7 @@ Or let the tool start from it: `proxy-scraper --recheck live` downloads the list
 
 | Tool | What it does |
 |---|---|
-| `get_proxies` | working proxies right now, from the hourly list – filter by protocol, country, HTTPS, elite, no datacenter, not blocklisted, stable, latency |
+| `get_proxies` | working proxies right now, from the hourly list – filter by protocol, country, HTTPS, elite, no datacenter, not blocklisted, stable, uptime, latency, gets through to Google/Reddit/Amazon |
 | `check_proxies` | checks proxies from your own network, so they work from where your code runs (30–90 s, reports progress) |
 | `fetch_url` | loads a page through a verified proxy, switches proxies by itself when one fails, returns readable text – HTTPS only through proxies with verified TLS |
 
@@ -411,6 +427,19 @@ if __name__ == "__main__":  # needed on macOS/Windows, the parser uses a process
 
 Same run as the command line – sources, learning, every check, result files – just without terminal output. Each result has `url`, `latency`, `exit_ip`, `https`, `anonymity`, `country`, `asn`, `org` and `hosting`. There's an async version of both (`find_proxies_async`, `check_proxies_async`).
 
+Don't need a fresh scan? `live_proxies` takes the [live list](#live-list) instead – no checks, one download, done in about a second:
+
+```python
+import itertools, requests
+from proxyscraper import live_proxies
+
+proxies = live_proxies(types=["socks5"], https=True, min_uptime=90)  # the reliable ones this week
+pool = itertools.cycle(p.url for p in proxies)
+r = requests.get("https://api.ipify.org", proxies={"https": next(pool)}, timeout=15)  # pip install "requests[socks]"
+```
+
+Same filters as `find_proxies`, plus `min_uptime`, `works_on` (`["google"]`, `"reddit"`, `"amazon"`) and `limit`. Every result also has `uptime_24h`, `uptime_7d`, `first_seen`, `up_for_hours` and `sites`.
+
 <a id="proxy-server"></a>
 
 ## Rotating proxy server

{proxy_scraper_cli-1.7.1 → proxy_scraper_cli-1.8.0}/README.md

@@ -135,7 +135,13 @@ Keys: <kbd>↑</kbd><kbd>↓</kbd> select · <kbd>Space</kbd> toggle · <kbd>1</
 
 Don't want to scan yourself? Every hour **GitHub Actions** runs the tool and publishes the hits to the [`proxy-list`](../../tree/proxy-list) branch – every entry worked in the last run, fastest first.
 
-**→ [Browse it on the website](https://maximilianfeix.github.io/proxy-scraper/)** – search, filter by type, country, HTTPS, provider and latency, see how long each proxy has been up, copy or download exactly the proxies you need. `streaks.json` on the branch has the number of runs in a row for every proxy.
+<a href="https://maximilianfeix.github.io/proxy-scraper/"><picture>
+  <source media="(prefers-color-scheme: dark)" srcset="https://maximilianfeix.github.io/proxy-scraper/chart-dark.svg">
+  <source media="(prefers-color-scheme: light)" srcset="https://maximilianfeix.github.io/proxy-scraper/chart-light.svg">
+  <img src="https://maximilianfeix.github.io/proxy-scraper/chart-dark.svg" alt="Working proxies over the last days, stacked by protocol – redrawn with every run" width="100%">
+</picture></a>
+
+**→ [Browse it on the website](https://maximilianfeix.github.io/proxy-scraper/)** – search, filter by type, country, HTTPS, provider and latency, see how long each proxy has been up and how often it was on the list this week, copy or download exactly the proxies you need.
 
 Every protocol and country also has its own page with a plain download, e.g. [SOCKS5](https://maximilianfeix.github.io/proxy-scraper/socks5/) or [Germany](https://maximilianfeix.github.io/proxy-scraper/country/de/) (`country/de/proxies.txt`).
 
@@ -157,12 +163,22 @@ Every protocol and country also has its own page with a plain download, e.g. [SO
 | HTTP · SOCKS4 · SOCKS5 | `1.2.3.4:8080` | [http.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/http.txt) · [socks4.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/socks4.txt) · [socks5.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/socks5.txt) |
 | HTTPS-capable only | `type://ip:port` | [https.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/https.txt) |
 | Elite only | `type://ip:port` | [elite.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/elite.txt) |
-| With all details | latency, country, HTTPS, anonymity, exit IP | [proxies.json](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/proxies.json) · [proxies.csv](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/proxies.csv) |
+| Gets through to Google · Reddit · Amazon (no captcha, no 403 in the last run) | `type://ip:port` | [google.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/works-with/google.txt) · [reddit.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/works-with/reddit.txt) · [amazon.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/works-with/amazon.txt) |
+| Stable, on the list in 90 %+ of this week's runs | `type://ip:port` | [stable.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/stable.txt) |
+| With all details | latency, country, HTTPS, anonymity, exit IP, uptime, sites | [proxies.json](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/proxies.json) · [proxies.csv](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/proxies.csv) |
+
+<a href="https://maximilianfeix.github.io/proxy-scraper/"><picture>
+  <source media="(prefers-color-scheme: dark)" srcset="https://maximilianfeix.github.io/proxy-scraper/countries-dark.svg">
+  <source media="(prefers-color-scheme: light)" srcset="https://maximilianfeix.github.io/proxy-scraper/countries-light.svg">
+  <img src="https://maximilianfeix.github.io/proxy-scraper/countries-dark.svg" alt="Countries with the most working proxies right now" width="100%">
+</picture></a>
 
 ```bash
 curl -s https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/socks5.txt | head
 ```
 
+Every file is also on GitHub Pages, which sits behind a CDN, isn't rate limited like raw.githubusercontent and sends CORS headers, so it works straight from the browser: `https://maximilianfeix.github.io/proxy-scraper/socks5.txt`. [jsDelivr](https://cdn.jsdelivr.net/gh/maximilianfeix/proxy-scraper@proxy-list/) works too, but can lag behind by a few hours.
+
 Or let the tool start from it: `proxy-scraper --recheck live` downloads the list and checks it again from **your** network – about 30 seconds instead of a full scan (517 of 1,169 worked from here). With `--serve` you have a rotating proxy in under a minute.
 
 <a id="mcp"></a>
@@ -175,7 +191,7 @@ Or let the tool start from it: `proxy-scraper --recheck live` downloads the list
 
 | Tool | What it does |
 |---|---|
-| `get_proxies` | working proxies right now, from the hourly list – filter by protocol, country, HTTPS, elite, no datacenter, not blocklisted, stable, latency |
+| `get_proxies` | working proxies right now, from the hourly list – filter by protocol, country, HTTPS, elite, no datacenter, not blocklisted, stable, uptime, latency, gets through to Google/Reddit/Amazon |
 | `check_proxies` | checks proxies from your own network, so they work from where your code runs (30–90 s, reports progress) |
 | `fetch_url` | loads a page through a verified proxy, switches proxies by itself when one fails, returns readable text – HTTPS only through proxies with verified TLS |
 
@@ -371,6 +387,19 @@ if __name__ == "__main__":  # needed on macOS/Windows, the parser uses a process
 
 Same run as the command line – sources, learning, every check, result files – just without terminal output. Each result has `url`, `latency`, `exit_ip`, `https`, `anonymity`, `country`, `asn`, `org` and `hosting`. There's an async version of both (`find_proxies_async`, `check_proxies_async`).
 
+Don't need a fresh scan? `live_proxies` takes the [live list](#live-list) instead – no checks, one download, done in about a second:
+
+```python
+import itertools, requests
+from proxyscraper import live_proxies
+
+proxies = live_proxies(types=["socks5"], https=True, min_uptime=90)  # the reliable ones this week
+pool = itertools.cycle(p.url for p in proxies)
+r = requests.get("https://api.ipify.org", proxies={"https": next(pool)}, timeout=15)  # pip install "requests[socks]"
+```
+
+Same filters as `find_proxies`, plus `min_uptime`, `works_on` (`["google"]`, `"reddit"`, `"amazon"`) and `limit`. Every result also has `uptime_24h`, `uptime_7d`, `first_seen`, `up_for_hours` and `sites`.
+
 <a id="proxy-server"></a>
 
 ## Rotating proxy server

{proxy_scraper_cli-1.7.1 → proxy_scraper_cli-1.8.0}/proxy_scraper_cli.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: proxy-scraper-cli
-Version: 1.7.1
+Version: 1.8.0
 Summary: Finds free HTTP, SOCKS4 and SOCKS5 proxies that actually work: 700+ sources, real checks, honeypot filtering and a rotating local proxy server.
 Author: Maximilian Feix
 License-Expression: MIT
@@ -175,7 +175,13 @@ Keys: <kbd>↑</kbd><kbd>↓</kbd> select · <kbd>Space</kbd> toggle · <kbd>1</
 
 Don't want to scan yourself? Every hour **GitHub Actions** runs the tool and publishes the hits to the [`proxy-list`](../../tree/proxy-list) branch – every entry worked in the last run, fastest first.
 
-**→ [Browse it on the website](https://maximilianfeix.github.io/proxy-scraper/)** – search, filter by type, country, HTTPS, provider and latency, see how long each proxy has been up, copy or download exactly the proxies you need. `streaks.json` on the branch has the number of runs in a row for every proxy.
+<a href="https://maximilianfeix.github.io/proxy-scraper/"><picture>
+  <source media="(prefers-color-scheme: dark)" srcset="https://maximilianfeix.github.io/proxy-scraper/chart-dark.svg">
+  <source media="(prefers-color-scheme: light)" srcset="https://maximilianfeix.github.io/proxy-scraper/chart-light.svg">
+  <img src="https://maximilianfeix.github.io/proxy-scraper/chart-dark.svg" alt="Working proxies over the last days, stacked by protocol – redrawn with every run" width="100%">
+</picture></a>
+
+**→ [Browse it on the website](https://maximilianfeix.github.io/proxy-scraper/)** – search, filter by type, country, HTTPS, provider and latency, see how long each proxy has been up and how often it was on the list this week, copy or download exactly the proxies you need.
 
 Every protocol and country also has its own page with a plain download, e.g. [SOCKS5](https://maximilianfeix.github.io/proxy-scraper/socks5/) or [Germany](https://maximilianfeix.github.io/proxy-scraper/country/de/) (`country/de/proxies.txt`).
 
@@ -197,12 +203,22 @@ Every protocol and country also has its own page with a plain download, e.g. [SO
 | HTTP · SOCKS4 · SOCKS5 | `1.2.3.4:8080` | [http.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/http.txt) · [socks4.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/socks4.txt) · [socks5.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/socks5.txt) |
 | HTTPS-capable only | `type://ip:port` | [https.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/https.txt) |
 | Elite only | `type://ip:port` | [elite.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/elite.txt) |
-| With all details | latency, country, HTTPS, anonymity, exit IP | [proxies.json](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/proxies.json) · [proxies.csv](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/proxies.csv) |
+| Gets through to Google · Reddit · Amazon (no captcha, no 403 in the last run) | `type://ip:port` | [google.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/works-with/google.txt) · [reddit.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/works-with/reddit.txt) · [amazon.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/works-with/amazon.txt) |
+| Stable, on the list in 90 %+ of this week's runs | `type://ip:port` | [stable.txt](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/stable.txt) |
+| With all details | latency, country, HTTPS, anonymity, exit IP, uptime, sites | [proxies.json](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/proxies.json) · [proxies.csv](https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/proxies.csv) |
+
+<a href="https://maximilianfeix.github.io/proxy-scraper/"><picture>
+  <source media="(prefers-color-scheme: dark)" srcset="https://maximilianfeix.github.io/proxy-scraper/countries-dark.svg">
+  <source media="(prefers-color-scheme: light)" srcset="https://maximilianfeix.github.io/proxy-scraper/countries-light.svg">
+  <img src="https://maximilianfeix.github.io/proxy-scraper/countries-dark.svg" alt="Countries with the most working proxies right now" width="100%">
+</picture></a>
 
 ```bash
 curl -s https://raw.githubusercontent.com/maximilianfeix/proxy-scraper/proxy-list/socks5.txt | head
 ```
 
+Every file is also on GitHub Pages, which sits behind a CDN, isn't rate limited like raw.githubusercontent and sends CORS headers, so it works straight from the browser: `https://maximilianfeix.github.io/proxy-scraper/socks5.txt`. [jsDelivr](https://cdn.jsdelivr.net/gh/maximilianfeix/proxy-scraper@proxy-list/) works too, but can lag behind by a few hours.
+
 Or let the tool start from it: `proxy-scraper --recheck live` downloads the list and checks it again from **your** network – about 30 seconds instead of a full scan (517 of 1,169 worked from here). With `--serve` you have a rotating proxy in under a minute.
 
 <a id="mcp"></a>
@@ -215,7 +231,7 @@ Or let the tool start from it: `proxy-scraper --recheck live` downloads the list
 
 | Tool | What it does |
 |---|---|
-| `get_proxies` | working proxies right now, from the hourly list – filter by protocol, country, HTTPS, elite, no datacenter, not blocklisted, stable, latency |
+| `get_proxies` | working proxies right now, from the hourly list – filter by protocol, country, HTTPS, elite, no datacenter, not blocklisted, stable, uptime, latency, gets through to Google/Reddit/Amazon |
 | `check_proxies` | checks proxies from your own network, so they work from where your code runs (30–90 s, reports progress) |
 | `fetch_url` | loads a page through a verified proxy, switches proxies by itself when one fails, returns readable text – HTTPS only through proxies with verified TLS |
 
@@ -411,6 +427,19 @@ if __name__ == "__main__":  # needed on macOS/Windows, the parser uses a process
 
 Same run as the command line – sources, learning, every check, result files – just without terminal output. Each result has `url`, `latency`, `exit_ip`, `https`, `anonymity`, `country`, `asn`, `org` and `hosting`. There's an async version of both (`find_proxies_async`, `check_proxies_async`).
 
+Don't need a fresh scan? `live_proxies` takes the [live list](#live-list) instead – no checks, one download, done in about a second:
+
+```python
+import itertools, requests
+from proxyscraper import live_proxies
+
+proxies = live_proxies(types=["socks5"], https=True, min_uptime=90)  # the reliable ones this week
+pool = itertools.cycle(p.url for p in proxies)
+r = requests.get("https://api.ipify.org", proxies={"https": next(pool)}, timeout=15)  # pip install "requests[socks]"
+```
+
+Same filters as `find_proxies`, plus `min_uptime`, `works_on` (`["google"]`, `"reddit"`, `"amazon"`) and `limit`. Every result also has `uptime_24h`, `uptime_7d`, `first_seen`, `up_for_hours` and `sites`.
+
 <a id="proxy-server"></a>
 
 ## Rotating proxy server

{proxy_scraper_cli-1.7.1 → proxy_scraper_cli-1.8.0}/proxy_scraper_cli.egg-info/SOURCES.txt

@@ -14,6 +14,7 @@ proxyscraper/api.py
 proxyscraper/app.py
 proxyscraper/asndb.py
 proxyscraper/blocklist.py
+proxyscraper/charts.py
 proxyscraper/checker.py
 proxyscraper/cli.py
 proxyscraper/compat.py
@@ -36,6 +37,7 @@ proxyscraper/paths.py
 proxyscraper/pipeline.py
 proxyscraper/preferences.py
 proxyscraper/publish.py
+proxyscraper/sites.py
 proxyscraper/sources.json
 proxyscraper/sources.py
 proxyscraper/targets.py
@@ -64,6 +66,7 @@ tests/test_app.py
 tests/test_asndb.py
 tests/test_auth.py
 tests/test_blocklist.py
+tests/test_charts.py
 tests/test_checker.py
 tests/test_compat.py
 tests/test_completion.py
@@ -73,6 +76,7 @@ tests/test_fetchcache.py
 tests/test_geodb.py
 tests/test_judges.py
 tests/test_learning.py
+tests/test_live_api.py
 tests/test_mcp_server.py
 tests/test_options.py
 tests/test_output_stdout.py
@@ -89,7 +93,9 @@ tests/test_serve_refill.py
 tests/test_serve_v2.py
 tests/test_server.py
 tests/test_server_fixes.py
+tests/test_sites.py
 tests/test_tamper.py
 tests/test_targets.py
 tests/test_ui.py
+tests/test_uptime.py
 tests/test_wizard.py

{proxy_scraper_cli-1.7.1 → proxy_scraper_cli-1.8.0}/proxyscraper/__init__.py

@@ -1,12 +1,13 @@
 """proxy-scraper: collects public HTTP/SOCKS4/SOCKS5 proxies from hundreds of sources, checks them
 in parallel with its own protocol handshakes and learns which sources deliver good proxies."""
 
-__version__ = "1.7.1"
+__version__ = "1.8.0"
 
 
 def __getattr__(name):
     # load the Python API only when needed – "import proxyscraper" should stay light (the CLI doesn't need it)
-    if name in ("find_proxies", "find_proxies_async", "check_proxies", "check_proxies_async"):
+    if name in ("find_proxies", "find_proxies_async", "check_proxies", "check_proxies_async", "live_proxies",
+                "live_proxies_async", "LiveProxy"):
         from . import api
         return getattr(api, name)
     raise AttributeError(f"module 'proxyscraper' has no attribute {name!r}")

{proxy_scraper_cli-1.7.1 → proxy_scraper_cli-1.8.0}/proxyscraper/agent.py

@@ -35,6 +35,7 @@ from .pages import SITE_URL
 from .parsing import PROXY_TYPES, make_key
 from .publish import RAW_BASE
 from .server import ProxyPool, RotatingServer
+from .sites import SITES
 
 LIVE_BASES = (SITE_URL.rstrip("/"), RAW_BASE)  # GitHub Pages first, the raw branch as the mirror
 CACHE_SECONDS = 300.0  # the list changes once an hour – no need to load 1 MB for every question
@@ -149,6 +150,15 @@ class LiveSource:
 
 # --------------------------------------------------------------------------- filtering
 
+def _check_sites(names: Iterable[str]) -> List[str]:
+    known = [s.name for s in SITES]
+    names = [str(n).lower() for n in names]
+    for n in names:
+        if n not in known:
+            raise AgentError(f"works_on takes {', '.join(known)}, not {n!r}")
+    return names
+
+
 def _check_protocol(protocol: str) -> str:
     protocol = (protocol or "any").lower()
     if protocol not in PROTOCOLS:
@@ -168,20 +178,23 @@ def normalize_countries(countries: Iterable[str]) -> List[str]:
 
 def select(rows: Iterable[dict], protocol: str = "any", countries: Iterable[str] = (), https_only: bool = False,
            elite_only: bool = False, exclude_datacenter: bool = False, exclude_blocklisted: bool = False,
-           stable_only: bool = False, max_latency_ms: int = 0, run_hours: int = 1) -> List[dict]:
+           stable_only: bool = False, max_latency_ms: int = 0, run_hours: int = 1, min_uptime: int = 0,
+           works_on: Iterable[str] = ()) -> List[dict]:
     """The rows that pass every filter, fastest first – the same filters as on the website."""
     return sorted(matching(rows, protocol, countries, https_only, elite_only, exclude_datacenter,
-                           exclude_blocklisted, stable_only, max_latency_ms, run_hours),
+                           exclude_blocklisted, stable_only, max_latency_ms, run_hours, min_uptime, works_on),
                   key=lambda r: r.get("latency") or 0)
 
 
 def matching(rows: Iterable[dict], protocol: str = "any", countries: Iterable[str] = (), https_only: bool = False,
              elite_only: bool = False, exclude_datacenter: bool = False, exclude_blocklisted: bool = False,
-             stable_only: bool = False, max_latency_ms: int = 0, run_hours: int = 1) -> Iterator[dict]:
+             stable_only: bool = False, max_latency_ms: int = 0, run_hours: int = 1,
+             min_uptime: int = 0, works_on: Iterable[str] = ()) -> Iterator[dict]:
     """The rows that pass every filter, in list order – checks the filters before the first row is asked for."""
     protocol = _check_protocol(protocol)
     wanted = set(normalize_countries(countries))
     stable_runs = -(-24 // max(run_hours, 1))  # runs in a row that make a day
+    sites = _check_sites(works_on)
     return (r for r in rows
             if (protocol == "any" or r.get("ptype") == protocol)
            and (not wanted or r.get("country") in wanted)
@@ -190,7 +203,9 @@ def matching(rows: Iterable[dict], protocol: str = "any", countries: Iterable[st
            and (not exclude_datacenter or not r.get("hosting"))
            and (not exclude_blocklisted or not r.get("blocklisted"))
            and (not stable_only or (r.get("streak") or 0) >= stable_runs)
-           and (not max_latency_ms or (r.get("latency") or 0) <= max_latency_ms))
+           and (not max_latency_ms or (r.get("latency") or 0) <= max_latency_ms)
+           and (not min_uptime or (r.get("uptime_7d") or 0) >= min_uptime)
+           and all((r.get("sites") or {}).get(s) for s in sites))
 
 
 def describe(item: Union[dict, CheckResult], run_hours: Optional[int] = None) -> dict:
@@ -212,6 +227,8 @@ def describe(item: Union[dict, CheckResult], run_hours: Optional[int] = None) ->
         "datacenter": item.get("hosting"),
         "blocklisted": item.get("blocklisted"),
         "up_for_hours": streak * run_hours if streak and run_hours else None,
+        "uptime_7d_percent": item.get("uptime_7d"),
+        "works_on": [name for name, ok in (item.get("sites") or {}).items() if ok],
     }
 
 

{proxy_scraper_cli-1.7.1 → proxy_scraper_cli-1.8.0}/proxyscraper/api.py

@@ -6,7 +6,14 @@
         for p in find_proxies(want=20, https=True, countries=["DE", "NL"]):
             print(p.url, p.latency, p.country)
 
-Behind it runs exactly the same as on the command line (sources, learning, honeypot and
+Or skip the checking and take the list that GitHub Actions checks every hour:
+
+    from proxyscraper import live_proxies
+
+    for p in live_proxies(types=["socks5"], countries="DE", https=True, min_uptime=90):
+        print(p.url, p.latency, p.uptime_7d)
+
+Behind find_proxies runs exactly the same as on the command line (sources, learning, honeypot and
 tampering checks, result files under results/), just without output in the terminal.
 
 Large lists are parsed in a process pool. On macOS and Windows it starts the worker processes
@@ -20,9 +27,11 @@ import asyncio
 import contextlib
 import io
 import os
+import re
 import tempfile
 import threading
-from typing import Iterable, List, Optional
+from dataclasses import dataclass, field
+from typing import Dict, Iterable, List, Optional
 
 from rich.console import Console
 
@@ -32,13 +41,22 @@ from .parsing import PROXY_TYPES
 from .targets import parse_target
 from .ui import widgets
 
-__all__ = ["CheckResult", "check_proxies", "check_proxies_async", "find_proxies", "find_proxies_async"]
+__all__ = ["CheckResult", "LiveProxy", "check_proxies", "check_proxies_async", "find_proxies", "find_proxies_async",
+           "live_proxies", "live_proxies_async"]
+
+
+def _types(types: Iterable[str]) -> List[str]:
+    """["socks5"], "socks5" or "socks5,http" – a plain string shouldn't turn into its letters."""
+    if isinstance(types, str):
+        return [t.strip().lower() for t in types.split(",") if t.strip()]
+    return list(types)
 
 
 def _options(types: Iterable[str], want: int, limit: int, https: bool, countries: Iterable[str], anonymity: str,
              max_latency: int, targets: Iterable[str], no_datacenter: bool, no_blocklisted: bool, timeout: float,
              concurrency: int,
              recheck: Optional[str]) -> RunOptions:
+    types = _types(types)
     if isinstance(countries, str):
         countries = parse_countries(countries)
     if anonymity not in ("", "anonymous", "elite"):
@@ -137,3 +155,81 @@ async def check_proxies_async(proxies: Iterable[str], **kwargs) -> List[CheckRes
 
 def check_proxies(proxies: Iterable[str], **kwargs) -> List[CheckResult]:
     return asyncio.run(check_proxies_async(proxies, **kwargs))
+
+
+# --------------------------------------------------------------------------- the hourly list
+
+@dataclass
+class LiveProxy(CheckResult):
+    """A proxy from the hourly list: the same fields as a CheckResult, plus how reliable it has been."""
+    uptime_24h: Optional[int] = None  # share of today's hourly runs it was listed in, in percent
+    uptime_7d: Optional[int] = None   # the same over the week
+    first_seen: str = ""              # ISO time of the first run it was listed in
+    up_for_hours: int = 0             # listed without a gap for this long
+    sites: Dict[str, bool] = field(default_factory=dict)  # "google", "reddit", "amazon" -> got through last run?
+
+
+def _live_fetch(url: str, timeout: float = 20, headers=None):
+    from .netio import http_get
+    return http_get(url, timeout=timeout, headers=headers)
+
+
+def _live_proxy(row: dict, run_hours: int) -> LiveProxy:
+    streak = row.get("streak") if type(row.get("streak")) is int else 0
+    return LiveProxy(
+        key=f"{row['ptype']} {row['proxy']}", ptype=row["ptype"], proxy=row["proxy"],
+        latency=int(row.get("latency") or 0),
+        exit_ip=row.get("exit_ip") or "", https=row.get("https"), anonymity=row.get("anonymity") or "",
+        country=row.get("country") or "", targets=dict(row.get("targets") or {}), asn=row.get("asn") or 0,
+        org=row.get("org") or "", hosting=row.get("hosting"), blocklisted=row.get("blocklisted"),
+        uptime_24h=row.get("uptime_24h"), uptime_7d=row.get("uptime_7d"), first_seen=row.get("first_seen") or "",
+        up_for_hours=streak * run_hours,
+        sites={k: v for k, v in (row.get("sites") or {}).items() if isinstance(v, bool)})
+
+
+async def live_proxies_async(*, types: Iterable[str] = PROXY_TYPES, countries: Iterable[str] = (), https: bool = False,
+                             anonymity: str = "", max_latency: int = 0, no_datacenter: bool = False,
+                             no_blocklisted: bool = False, min_uptime: int = 0, works_on: Iterable[str] = (),
+                             limit: int = 0) -> List[LiveProxy]:
+    """The proxies from the hourly list that pass the filters, fastest first. Nothing is checked here: they
+    worked from GitHub's servers in the last run (at most an hour ago). find_proxies checks from your network.
+
+    Same filters as find_proxies, plus
+    min_uptime  only proxies listed in at least this share (percent) of the week's runs, e.g. 90
+    works_on    only proxies that got through to these sites in the last run: "google", "reddit", "amazon"
+    limit       at most this many (0 = all)
+    """
+    from .agent import AgentError, LiveSource
+    from .sites import SITE
+
+    types = _types(types)
+    unknown = [t for t in types if t not in PROXY_TYPES]
+    if unknown:
+        raise ValueError(f"unknown proxy type {unknown[0]!r}, use {', '.join(PROXY_TYPES)}")
+    if isinstance(countries, str):
+        countries = parse_countries(countries)
+    countries = {c.strip().upper() for c in countries}
+    bad = [c for c in countries if not re.fullmatch(r"[A-Z]{2}", c)]
+    if bad:
+        raise ValueError(f"countries are two-letter codes like DE or US, not {bad[0]!r}")
+    works_on = [str(s).lower() for s in ([works_on] if isinstance(works_on, str) else works_on)]
+    unknown_sites = [s for s in works_on if s not in SITE]
+    if unknown_sites:
+        raise ValueError(f"works_on takes {', '.join(SITE)}, not {unknown_sites[0]!r}")
+    opts = _options(types, 0, 0, https, countries, anonymity, max_latency, (), no_datacenter, no_blocklisted, 8.0, 1,
+                    None)
+    try:
+        data = await LiveSource(fetch=_live_fetch).get()
+    except AgentError as e:
+        raise ConnectionError(str(e)) from None
+    found = sorted((_live_proxy(r, data.run_hours) for r in data.rows
+                    if r.get("ptype") in types and isinstance(r.get("proxy"), str)),
+                   key=lambda p: p.latency)
+    found = [p for p in found if opts.filters.accepts(p) and (not min_uptime or (p.uptime_7d or 0) >= min_uptime)
+             and all(p.sites.get(s) for s in works_on)]
+    return found[:limit] if limit else found
+
+
+def live_proxies(**kwargs) -> List[LiveProxy]:
+    """Like live_proxies_async, just synchronous (starts its own event loop)."""
+    return asyncio.run(live_proxies_async(**kwargs))

proxy_scraper_cli-1.8.0/proxyscraper/charts.py

@@ -0,0 +1,135 @@
+"""SVG charts of the live list for the README: the working proxies over the last week, stacked by protocol,
+and where they are. Written with every run next to the list, so the README shows today's numbers.
+
+Plain SVG without scripts or web fonts – GitHub shows README images through a proxy that allows neither.
+"""
+
+from __future__ import annotations
+
+from datetime import datetime, timedelta
+from typing import Dict, List, Optional, Sequence, Tuple
+from xml.sax.saxutils import escape
+
+from .pages import COUNTRIES
+from .parsing import PROXY_TYPES
+
+THEMES = {
+    "dark": {"bg": "#121113", "panel": "#1A191C", "line": "#2B292F", "text": "#EDEBE6", "muted": "#9A97A0",
+             "http": "#8DB8FF", "socks4": "#C7A6FF", "socks5": "#D4F77A"},
+    "light": {"bg": "#FFFFFF", "panel": "#F6F6F3", "line": "#DAD9D4", "text": "#121113", "muted": "#5F5C66",
+              "http": "#1F5FD1", "socks4": "#7442D6", "socks5": "#3F5A00"},
+}
+FONT = "-apple-system, BlinkMacSystemFont, 'Segoe UI', Helvetica, Arial, sans-serif"
+WIDTH, HEIGHT = 880, 300
+DAYS = 7
+
+
+def _when(run: dict) -> Optional[datetime]:
+    try:
+        return datetime.fromisoformat(run["updated"])
+    except (KeyError, TypeError, ValueError):
+        return None
+
+
+def recent_runs(runs: Sequence[dict], now: datetime, days: int = DAYS) -> List[Tuple[datetime, dict]]:
+    """The runs of the last `days`, oldest first. Runs less than 20 minutes apart (a manual run right after the
+    hourly one) count once, the later one wins – otherwise the chart gets spikes that mean nothing."""
+    since = now - timedelta(days=days)
+    timed = sorted(((t, r) for r in runs if (t := _when(r)) and t > since and isinstance(r.get("by_type"), dict)),
+                   key=lambda x: x[0])
+    kept: List[Tuple[datetime, dict]] = []
+    for t, r in timed:
+        if kept and t - kept[-1][0] < timedelta(minutes=20):
+            kept[-1] = (t, r)
+        else:
+            kept.append((t, r))
+    return kept
+
+
+def _num(n: int) -> str:
+    return f"{n:,}"
+
+
+def trend_svg(runs: Sequence[dict], now: datetime, theme: str = "dark") -> str:
+    """Working proxies per run over the last week, stacked by protocol, with today's total as the headline."""
+    c = THEMES[theme]
+    points = recent_runs(runs, now)
+    left, right, top, bottom = 250, WIDTH - 24, 36, HEIGHT - 44
+    parts = [f'<svg xmlns="http://www.w3.org/2000/svg" width="{WIDTH}" height="{HEIGHT}" '
+             f'viewBox="0 0 {WIDTH} {HEIGHT}" '
+             f'font-family="{FONT}" role="img" aria-label="Working proxies over the last {DAYS} days">',
+             f'<rect width="{WIDTH}" height="{HEIGHT}" rx="16" fill="{c["bg"]}"/>']
+    last = points[-1][1] if points else {"total": 0, "by_type": {}}
+    total = last.get("total", 0)
+    parts += [f'<text x="28" y="58" fill="{c["muted"]}" font-size="14">Working right now</text>',
+              f'<text x="26" y="112" fill="{c["text"]}" font-size="52" font-weight="600" letter-spacing="-1.5">'
+              f'{_num(total)}</text>']
+    for i, t in enumerate(PROXY_TYPES):
+        y = 158 + i * 28
+        n = last.get("by_type", {}).get(t, 0)
+        parts += [f'<rect x="28" y="{y - 10}" width="10" height="10" rx="2" fill="{c[t]}"/>',
+                  f'<text x="48" y="{y}" fill="{c["muted"]}" font-size="14">{t.upper()}</text>',
+                  f'<text x="200" y="{y}" fill="{c["text"]}" font-size="14" text-anchor="end">{_num(n)}</text>']
+    # the x axis covers the week, or less while the history is younger than that
+    start = max(now - timedelta(days=DAYS), points[0][0]) if len(points) >= 2 else now - timedelta(days=DAYS)
+    start = min(start, now - timedelta(hours=6))
+    span = (now - start).total_seconds()
+
+    def x_of(t: datetime) -> float:
+        return left + (right - left) * (t - start).total_seconds() / span
+
+    parts.append(f'<text x="28" y="{HEIGHT - 28}" fill="{c["muted"]}" font-size="12">checked every hour'
+                 f'{f", last {DAYS} days" if span >= timedelta(days=DAYS).total_seconds() - 3600 else ""}</text>')
+    parts.append(f'<line x1="{left}" y1="{bottom}" x2="{right}" y2="{bottom}" stroke="{c["line"]}" stroke-width="1"/>')
+    midnight = start.replace(hour=0, minute=0, second=0, microsecond=0) + timedelta(days=1)
+    while midnight < now:  # a line at the start of every day, labelled with the day
+        x = x_of(midnight)
+        parts.append(f'<line x1="{x:.1f}" y1="{top}" x2="{x:.1f}" y2="{bottom}" stroke="{c["line"]}"/>')
+        if right - x > 40:
+            parts.append(f'<text x="{x + 6:.1f}" y="{bottom + 22}" fill="{c["muted"]}" font-size="12">'
+                         f'{escape(midnight.strftime("%a %d"))}</text>')
+        midnight += timedelta(days=1)
+    if len(points) >= 2:
+        peak = max(r.get("total", 0) for _, r in points) or 1
+
+        def y_of(v: float) -> float:
+            return bottom - (bottom - top) * v / (peak * 1.08)
+
+        below = [0.0] * len(points)
+        for t in PROXY_TYPES:  # stacked: each protocol on top of the ones before it
+            above = [b + (r.get("by_type", {}).get(t, 0) or 0) for b, (_, r) in zip(below, points)]
+            upper = " ".join(f"{x_of(when):.1f},{y_of(v):.1f}" for (when, _), v in zip(points, above))
+            lower = " ".join(f"{x_of(when):.1f},{y_of(v):.1f}" for (when, _), v in reversed(list(zip(points, below))))
+            parts.append(f'<polygon points="{upper} {lower}" fill="{c[t]}" fill-opacity="0.85"/>')
+            below = above
+        parts.append(f'<text x="{right}" y="{top - 12}" fill="{c["muted"]}" font-size="12" text-anchor="end">'
+                     f'peak {_num(peak)}</text>')
+    else:
+        parts.append(f'<text x="{(left + right) / 2:.0f}" y="{(top + bottom) / 2:.0f}" fill="{c["muted"]}" '
+                     f'font-size="14" text-anchor="middle">The chart fills up with the next runs</text>')
+    parts.append("</svg>")
+    return "\n".join(parts) + "\n"
+
+
+def countries_svg(countries: Dict[str, int], theme: str = "dark", limit: int = 10) -> str:
+    """Where the working proxies are: the top countries as bars."""
+    c = THEMES[theme]
+    top = sorted(countries.items(), key=lambda kv: -kv[1])[:limit]
+    row, pad = 24, 28
+    height = pad * 2 + 22 + row * max(len(top), 1)
+    most = max((n for _, n in top), default=1) or 1
+    parts = [f'<svg xmlns="http://www.w3.org/2000/svg" width="{WIDTH}" height="{height}" '
+             f'viewBox="0 0 {WIDTH} {height}" '
+             f'font-family="{FONT}" role="img" aria-label="Countries with the most working proxies">',
+             f'<rect width="{WIDTH}" height="{height}" rx="16" fill="{c["bg"]}"/>',
+             f'<text x="28" y="{pad + 10}" fill="{c["muted"]}" font-size="14">Where they are</text>']
+    for i, (cc, n) in enumerate(top):
+        y = pad + 36 + i * row
+        width = (WIDTH - 250 - 90) * n / most
+        name = COUNTRIES.get(cc, cc)
+        name = name[4:] if name.startswith("the ") else name
+        parts += [f'<text x="28" y="{y + 11}" fill="{c["text"]}" font-size="13">{escape(name)}</text>',
+                  f'<rect x="250" y="{y}" width="{max(width, 2):.1f}" height="14" rx="3" fill="{c["socks5"]}"/>',
+                  f'<text x="{250 + width + 10:.1f}" y="{y + 11}" fill="{c["muted"]}" font-size="12">{_num(n)}</text>']
+    parts.append("</svg>")
+    return "\n".join(parts) + "\n"