proxy-scraper-cli 1.8.1__tar.gz → 1.10.0__tar.gz

{proxy_scraper_cli-1.8.1 → proxy_scraper_cli-1.10.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: proxy-scraper-cli
-Version: 1.8.1
+Version: 1.10.0
 Summary: Finds free HTTP, SOCKS4 and SOCKS5 proxies that actually work: 700+ sources, real checks, honeypot filtering and a rotating local proxy server.
 Author: Maximilian Feix
 License-Expression: MIT
@@ -35,6 +35,7 @@ Provides-Extra: mcp
 Requires-Dist: mcp<3,>=2.2; python_version >= "3.10" and extra == "mcp"
 Provides-Extra: dev
 Requires-Dist: pytest; extra == "dev"
+Requires-Dist: hypothesis; extra == "dev"
 Requires-Dist: ruff==0.16.8; extra == "dev"
 Dynamic: license-file
 
@@ -138,11 +139,12 @@ python3 proxy_scraper.py
 
 Every [release](https://github.com/maximilianfeix/proxy-scraper/releases/latest) also ships a wheel you can install with `pip install <file>.whl`, and a multi-arch image (amd64/arm64) on `ghcr.io`. In the container the wizard never shows up, it runs straight away. For the proxy server use `--serve --serve-host 0.0.0.0` with `-p 127.0.0.1:8899:8899`, so the port is only open on your own machine.
 
-Tab completion for bash, zsh and fish:
+Tab completion for bash, zsh, fish and PowerShell:
 
 ```bash
 eval "$(proxy-scraper --completion zsh)"     # in ~/.zshrc (after compinit), same for bash in ~/.bashrc
 proxy-scraper --completion fish > ~/.config/fish/completions/proxy-scraper.fish
+proxy-scraper --completion powershell | Out-String | Invoke-Expression   # in $PROFILE
 ```
 
 Installed, the learned state lives in your user data folder (`~/Library/Application Support/proxy-scraper`, `%LOCALAPPDATA%\proxy-scraper` or `~/.local/share/proxy-scraper`; override with `PROXY_SCRAPER_HOME`) and results go to `./results`. Run from a clone, both stay inside the project.
@@ -578,14 +580,92 @@ for p in proxies:
         continue  # free proxies come and go – just take the next one
 ```
 
-**proxychains, Clash / Mihomo** – ready-made configs with `--export`
+**httpx** (`pip install "httpx[socks]"`) – straight from the hourly list, no scan
+
+```python
+import httpx
+from proxyscraper import live_proxies
+
+for p in live_proxies(types=["http", "socks5"], https=True, min_uptime=90, limit=10):  # httpx can't do SOCKS4
+    try:
+        with httpx.Client(proxy=p.url, timeout=10) as client:
+            print(p.url, "→", client.get("https://api.ipify.org").text)
+        break
+    except httpx.HTTPError:
+        continue  # next one
+```
+
+**aiohttp** (`pip install aiohttp aiohttp-socks` – aiohttp alone can't do SOCKS)
+
+```python
+import asyncio
+
+import aiohttp
+from aiohttp_socks import ProxyConnector, ProxyError
+from proxyscraper import live_proxies_async
+
+
+async def main():
+    for p in await live_proxies_async(https=True, min_uptime=90, limit=10):
+        try:
+            async with aiohttp.ClientSession(connector=ProxyConnector.from_url(p.url)) as session:
+                async with session.get("https://api.ipify.org", timeout=aiohttp.ClientTimeout(total=10)) as r:
+                    print(p.url, "→", await r.text())
+                    return
+        except (aiohttp.ClientError, ProxyError, asyncio.TimeoutError, OSError):
+            continue  # free proxies come and go – just take the next one
+
+asyncio.run(main())
+```
+
+**Scrapy** – a downloader middleware that sends every request, retries included, through the next proxy. Scrapy only speaks HTTP proxies, `https=True` picks the ones that can tunnel `https://` pages
+
+```python
+import itertools
+
+import scrapy
+from scrapy.crawler import CrawlerProcess
+from proxyscraper import live_proxies
+
+PROXIES = [p.url for p in live_proxies(types="http", https=True, min_uptime=50)]
+if not PROXIES:
+    raise SystemExit("No proxy matches right now – loosen the filters (e.g. min_uptime)")
+POOL = itertools.cycle(PROXIES)
+
+
+class RotatingProxy:
+    def process_request(self, request, spider=None):  # newer Scrapy leaves out spider
+        request.meta["proxy"] = next(POOL)
+
+
+class IpSpider(scrapy.Spider):
+    name = "ip"
+    start_urls = [f"https://api.ipify.org/?n={i}" for i in range(3)]
+    custom_settings = {
+        "DOWNLOADER_MIDDLEWARES": {f"{__name__}.RotatingProxy": 350},
+        "RETRY_TIMES": 5, "DOWNLOAD_TIMEOUT": 15,
+    }
+
+    def parse(self, response):
+        print(response.meta["proxy"], "→", response.text)
+
+
+if __name__ == "__main__":
+    process = CrawlerProcess()
+    process.crawl(IpSpider)
+    process.start()
+```
+
+In a Scrapy project, put `RotatingProxy` in `middlewares.py` and add it to `DOWNLOADER_MIDDLEWARES` in `settings.py`. For long crawls, reload the pool now and then – the list changes every hour.
+
+**proxychains, Clash / Mihomo, sing-box** – ready-made configs with `--export`
 
 ```bash
-proxy-scraper --want 30 -y --export proxychains,clash
+proxy-scraper --want 30 -y --export proxychains,clash,singbox
 proxychains4 -f results/latest/proxychains.conf curl https://api.ipify.org
 ```
 
-`clash.yaml` has all HTTP and SOCKS5 proxies plus a `url-test` group that always picks the fastest. Both configs leave out HTTP proxies that can't tunnel (`CONNECT`), because these tools tunnel everything.
+`clash.yaml` has all HTTP and SOCKS5 proxies plus a `url-test` group that always picks the fastest. `singbox.json` does the same for sing-box, SOCKS4 included, and opens a local proxy: `sing-box run -c results/latest/singbox.json`, then use `127.0.0.1:2080` as HTTP or SOCKS5 proxy. All three leave out HTTP proxies that can't tunnel (`CONNECT`), because these tools tunnel everything.
 
 **In a pipe** – `-o -` prints the hits to stdout, the interface moves to stderr
 
@@ -601,6 +681,22 @@ export HTTPS_PROXY=http://127.0.0.1:8899 HTTP_PROXY=http://127.0.0.1:8899
 pip download requests   # git, pip, npm & co. now go through the pool
 ```
 
+**In a GitHub workflow** – the action picks working proxies for the next steps
+
+```yaml
+- id: proxies
+  uses: maximilianfeix/proxy-scraper@v1
+  with:
+    types: socks5
+    https: true
+    min-uptime: 90        # on the list for 90 %+ of the week
+    works-on: google      # optional: google, reddit, amazon
+    recheck: true         # optional: check them again from the runner
+- run: curl -x "${{ steps.proxies.outputs.proxy }}" https://api.ipify.org
+```
+
+`proxy` is the fastest one, `file` a text file with all of them (up to `limit`, default 20), `count` how many. Without a match the step fails, unless `fail-if-empty: false`.
+
 **Without installing anything** – straight from the live list
 
 ```bash
@@ -644,16 +740,16 @@ curl.exe -x (Get-Content "$run\all.txt" -TotalCount 1) http://api.ipify.org
 | `-t`, `--timeout S` | timeout per proxy (default: 8 s) |
 | `--discover` | search GitHub for new sources right now |
 | `--no-cache` | download every list again (unchanged ones are normally skipped via ETag) |
-| `--list-sources [N]` | show the source ranking |
+| `--list-sources [N]` | show the source ranking (add `--json` for scripts: url, status, hit rate, checks, last change) |
 | `--serve-host ADDR` | where the proxy server listens (default `127.0.0.1`; `0.0.0.0` for Docker, with a warning) |
 | `--serve-password SECRET` | clients must send this password in the proxy login; better set `PROXY_SCRAPER_SERVE_PASSWORD` |
 | `--rotate STRATEGY` · `--sticky SEC` | how the proxy server picks proxies, see [above](#proxy-server) |
 | `--serve-refill HOURS` | while serving, check fresh proxies every HOURS and add the hits to the pool |
 | `--serve [PORT]` | afterwards serve as a rotating proxy on `127.0.0.1:PORT` (default: 8899) |
 | `-o FILE` | also write all hits to this file; `-o -` prints them to stdout |
-| `--export FORMATS` | extra files for other tools: `proxychains`, `clash`, `curl` or `all` |
+| `--export FORMATS` | extra files for other tools: `proxychains`, `clash`, `singbox`, `curl` or `all` |
 | `-V`, `--version` | print the version |
-| `--completion SHELL` | print the tab completion script for bash, zsh or fish |
+| `--completion SHELL` | print the tab completion script for bash, zsh, fish or PowerShell |
 
 Everything else: `proxy-scraper --help`
 
@@ -735,10 +831,12 @@ Only for things that don't matter. Public proxies are run by strangers who can r
 
 ## Roadmap
 
-What's next is tracked in the milestone [**v1.8**](../../milestone/8) – ideas and wishes are welcome as an [issue](../../issues/new/choose). During Hacktoberfest there are [beginner-friendly issues](../../issues?q=is%3Aopen+label%3Ahacktoberfest) with pointers on where to start.
+What's next is in the [open issues](../../issues) – ideas and wishes are welcome as an [issue](../../issues/new/choose). During Hacktoberfest there are [beginner-friendly issues](../../issues?q=is%3Aopen+label%3Ahacktoberfest) with pointers on where to start.
+
+- [ ] [Measure throughput, not just latency](../../issues/140)
+- [ ] [Daily snapshots as a dataset](../../issues/143)
 
-- [ ] [Protocol detection on the same connection](../../issues/44)
-- [ ] [sing-box export](../../issues/113), [PowerShell completion](../../issues/114), [shareable filters on the website](../../issues/116)
+Shipped in v1.8 and v1.9: uptime per proxy and `stable.txt`, which proxies get through to Google, Reddit and Amazon, `live_proxies()` in Python, a page per proxy with its week of checks, a weekly report, live charts in this README, shareable filters on the website, sources mined from other scrapers' lists, untyped lists tried as HTTP and SOCKS5, `--export singbox`, PowerShell completion and recipes for httpx, aiohttp and Scrapy.
 
 Shipped in [v1.7](../../milestone/7): an MCP server so AI agents get working proxies and can load pages through them, 28 new sources and a GitHub search that runs daily and keeps what it found, and a cleaner website.
 
@@ -783,7 +881,7 @@ proxyscraper/
 ├── geodb.py            DB-IP country database (monthly, binary search)
 ├── targets.py          target sites for --target
 ├── output.py           result files
-├── exporters.py        proxychains, Clash and curl formats (--export)
+├── exporters.py        proxychains, Clash, sing-box and curl formats (--export)
 ├── server/             rotating proxy server (--serve): pool · http · upstream · socks · status · core
 ├── api.py              find_proxies() / check_proxies() for Python
 ├── agent.py            MCP tools without the SDK: live list, filters, fetch through proxies

{proxy_scraper_cli-1.8.1 → proxy_scraper_cli-1.10.0}/README.md

@@ -98,11 +98,12 @@ python3 proxy_scraper.py
 
 Every [release](https://github.com/maximilianfeix/proxy-scraper/releases/latest) also ships a wheel you can install with `pip install <file>.whl`, and a multi-arch image (amd64/arm64) on `ghcr.io`. In the container the wizard never shows up, it runs straight away. For the proxy server use `--serve --serve-host 0.0.0.0` with `-p 127.0.0.1:8899:8899`, so the port is only open on your own machine.
 
-Tab completion for bash, zsh and fish:
+Tab completion for bash, zsh, fish and PowerShell:
 
 ```bash
 eval "$(proxy-scraper --completion zsh)"     # in ~/.zshrc (after compinit), same for bash in ~/.bashrc
 proxy-scraper --completion fish > ~/.config/fish/completions/proxy-scraper.fish
+proxy-scraper --completion powershell | Out-String | Invoke-Expression   # in $PROFILE
 ```
 
 Installed, the learned state lives in your user data folder (`~/Library/Application Support/proxy-scraper`, `%LOCALAPPDATA%\proxy-scraper` or `~/.local/share/proxy-scraper`; override with `PROXY_SCRAPER_HOME`) and results go to `./results`. Run from a clone, both stay inside the project.
@@ -538,14 +539,92 @@ for p in proxies:
         continue  # free proxies come and go – just take the next one
 ```
 
-**proxychains, Clash / Mihomo** – ready-made configs with `--export`
+**httpx** (`pip install "httpx[socks]"`) – straight from the hourly list, no scan
+
+```python
+import httpx
+from proxyscraper import live_proxies
+
+for p in live_proxies(types=["http", "socks5"], https=True, min_uptime=90, limit=10):  # httpx can't do SOCKS4
+    try:
+        with httpx.Client(proxy=p.url, timeout=10) as client:
+            print(p.url, "→", client.get("https://api.ipify.org").text)
+        break
+    except httpx.HTTPError:
+        continue  # next one
+```
+
+**aiohttp** (`pip install aiohttp aiohttp-socks` – aiohttp alone can't do SOCKS)
+
+```python
+import asyncio
+
+import aiohttp
+from aiohttp_socks import ProxyConnector, ProxyError
+from proxyscraper import live_proxies_async
+
+
+async def main():
+    for p in await live_proxies_async(https=True, min_uptime=90, limit=10):
+        try:
+            async with aiohttp.ClientSession(connector=ProxyConnector.from_url(p.url)) as session:
+                async with session.get("https://api.ipify.org", timeout=aiohttp.ClientTimeout(total=10)) as r:
+                    print(p.url, "→", await r.text())
+                    return
+        except (aiohttp.ClientError, ProxyError, asyncio.TimeoutError, OSError):
+            continue  # free proxies come and go – just take the next one
+
+asyncio.run(main())
+```
+
+**Scrapy** – a downloader middleware that sends every request, retries included, through the next proxy. Scrapy only speaks HTTP proxies, `https=True` picks the ones that can tunnel `https://` pages
+
+```python
+import itertools
+
+import scrapy
+from scrapy.crawler import CrawlerProcess
+from proxyscraper import live_proxies
+
+PROXIES = [p.url for p in live_proxies(types="http", https=True, min_uptime=50)]
+if not PROXIES:
+    raise SystemExit("No proxy matches right now – loosen the filters (e.g. min_uptime)")
+POOL = itertools.cycle(PROXIES)
+
+
+class RotatingProxy:
+    def process_request(self, request, spider=None):  # newer Scrapy leaves out spider
+        request.meta["proxy"] = next(POOL)
+
+
+class IpSpider(scrapy.Spider):
+    name = "ip"
+    start_urls = [f"https://api.ipify.org/?n={i}" for i in range(3)]
+    custom_settings = {
+        "DOWNLOADER_MIDDLEWARES": {f"{__name__}.RotatingProxy": 350},
+        "RETRY_TIMES": 5, "DOWNLOAD_TIMEOUT": 15,
+    }
+
+    def parse(self, response):
+        print(response.meta["proxy"], "→", response.text)
+
+
+if __name__ == "__main__":
+    process = CrawlerProcess()
+    process.crawl(IpSpider)
+    process.start()
+```
+
+In a Scrapy project, put `RotatingProxy` in `middlewares.py` and add it to `DOWNLOADER_MIDDLEWARES` in `settings.py`. For long crawls, reload the pool now and then – the list changes every hour.
+
+**proxychains, Clash / Mihomo, sing-box** – ready-made configs with `--export`
 
 ```bash
-proxy-scraper --want 30 -y --export proxychains,clash
+proxy-scraper --want 30 -y --export proxychains,clash,singbox
 proxychains4 -f results/latest/proxychains.conf curl https://api.ipify.org
 ```
 
-`clash.yaml` has all HTTP and SOCKS5 proxies plus a `url-test` group that always picks the fastest. Both configs leave out HTTP proxies that can't tunnel (`CONNECT`), because these tools tunnel everything.
+`clash.yaml` has all HTTP and SOCKS5 proxies plus a `url-test` group that always picks the fastest. `singbox.json` does the same for sing-box, SOCKS4 included, and opens a local proxy: `sing-box run -c results/latest/singbox.json`, then use `127.0.0.1:2080` as HTTP or SOCKS5 proxy. All three leave out HTTP proxies that can't tunnel (`CONNECT`), because these tools tunnel everything.
 
 **In a pipe** – `-o -` prints the hits to stdout, the interface moves to stderr
 
@@ -561,6 +640,22 @@ export HTTPS_PROXY=http://127.0.0.1:8899 HTTP_PROXY=http://127.0.0.1:8899
 pip download requests   # git, pip, npm & co. now go through the pool
 ```
 
+**In a GitHub workflow** – the action picks working proxies for the next steps
+
+```yaml
+- id: proxies
+  uses: maximilianfeix/proxy-scraper@v1
+  with:
+    types: socks5
+    https: true
+    min-uptime: 90        # on the list for 90 %+ of the week
+    works-on: google      # optional: google, reddit, amazon
+    recheck: true         # optional: check them again from the runner
+- run: curl -x "${{ steps.proxies.outputs.proxy }}" https://api.ipify.org
+```
+
+`proxy` is the fastest one, `file` a text file with all of them (up to `limit`, default 20), `count` how many. Without a match the step fails, unless `fail-if-empty: false`.
+
 **Without installing anything** – straight from the live list
 
 ```bash
@@ -604,16 +699,16 @@ curl.exe -x (Get-Content "$run\all.txt" -TotalCount 1) http://api.ipify.org
 | `-t`, `--timeout S` | timeout per proxy (default: 8 s) |
 | `--discover` | search GitHub for new sources right now |
 | `--no-cache` | download every list again (unchanged ones are normally skipped via ETag) |
-| `--list-sources [N]` | show the source ranking |
+| `--list-sources [N]` | show the source ranking (add `--json` for scripts: url, status, hit rate, checks, last change) |
 | `--serve-host ADDR` | where the proxy server listens (default `127.0.0.1`; `0.0.0.0` for Docker, with a warning) |
 | `--serve-password SECRET` | clients must send this password in the proxy login; better set `PROXY_SCRAPER_SERVE_PASSWORD` |
 | `--rotate STRATEGY` · `--sticky SEC` | how the proxy server picks proxies, see [above](#proxy-server) |
 | `--serve-refill HOURS` | while serving, check fresh proxies every HOURS and add the hits to the pool |
 | `--serve [PORT]` | afterwards serve as a rotating proxy on `127.0.0.1:PORT` (default: 8899) |
 | `-o FILE` | also write all hits to this file; `-o -` prints them to stdout |
-| `--export FORMATS` | extra files for other tools: `proxychains`, `clash`, `curl` or `all` |
+| `--export FORMATS` | extra files for other tools: `proxychains`, `clash`, `singbox`, `curl` or `all` |
 | `-V`, `--version` | print the version |
-| `--completion SHELL` | print the tab completion script for bash, zsh or fish |
+| `--completion SHELL` | print the tab completion script for bash, zsh, fish or PowerShell |
 
 Everything else: `proxy-scraper --help`
 
@@ -695,10 +790,12 @@ Only for things that don't matter. Public proxies are run by strangers who can r
 
 ## Roadmap
 
-What's next is tracked in the milestone [**v1.8**](../../milestone/8) – ideas and wishes are welcome as an [issue](../../issues/new/choose). During Hacktoberfest there are [beginner-friendly issues](../../issues?q=is%3Aopen+label%3Ahacktoberfest) with pointers on where to start.
+What's next is in the [open issues](../../issues) – ideas and wishes are welcome as an [issue](../../issues/new/choose). During Hacktoberfest there are [beginner-friendly issues](../../issues?q=is%3Aopen+label%3Ahacktoberfest) with pointers on where to start.
+
+- [ ] [Measure throughput, not just latency](../../issues/140)
+- [ ] [Daily snapshots as a dataset](../../issues/143)
 
-- [ ] [Protocol detection on the same connection](../../issues/44)
-- [ ] [sing-box export](../../issues/113), [PowerShell completion](../../issues/114), [shareable filters on the website](../../issues/116)
+Shipped in v1.8 and v1.9: uptime per proxy and `stable.txt`, which proxies get through to Google, Reddit and Amazon, `live_proxies()` in Python, a page per proxy with its week of checks, a weekly report, live charts in this README, shareable filters on the website, sources mined from other scrapers' lists, untyped lists tried as HTTP and SOCKS5, `--export singbox`, PowerShell completion and recipes for httpx, aiohttp and Scrapy.
 
 Shipped in [v1.7](../../milestone/7): an MCP server so AI agents get working proxies and can load pages through them, 28 new sources and a GitHub search that runs daily and keeps what it found, and a cleaner website.
 
@@ -743,7 +840,7 @@ proxyscraper/
 ├── geodb.py            DB-IP country database (monthly, binary search)
 ├── targets.py          target sites for --target
 ├── output.py           result files
-├── exporters.py        proxychains, Clash and curl formats (--export)
+├── exporters.py        proxychains, Clash, sing-box and curl formats (--export)
 ├── server/             rotating proxy server (--serve): pool · http · upstream · socks · status · core
 ├── api.py              find_proxies() / check_proxies() for Python
 ├── agent.py            MCP tools without the SDK: live list, filters, fetch through proxies

{proxy_scraper_cli-1.8.1 → proxy_scraper_cli-1.10.0}/proxy_scraper_cli.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: proxy-scraper-cli
-Version: 1.8.1
+Version: 1.10.0
 Summary: Finds free HTTP, SOCKS4 and SOCKS5 proxies that actually work: 700+ sources, real checks, honeypot filtering and a rotating local proxy server.
 Author: Maximilian Feix
 License-Expression: MIT
@@ -35,6 +35,7 @@ Provides-Extra: mcp
 Requires-Dist: mcp<3,>=2.2; python_version >= "3.10" and extra == "mcp"
 Provides-Extra: dev
 Requires-Dist: pytest; extra == "dev"
+Requires-Dist: hypothesis; extra == "dev"
 Requires-Dist: ruff==0.16.8; extra == "dev"
 Dynamic: license-file
 
@@ -138,11 +139,12 @@ python3 proxy_scraper.py
 
 Every [release](https://github.com/maximilianfeix/proxy-scraper/releases/latest) also ships a wheel you can install with `pip install <file>.whl`, and a multi-arch image (amd64/arm64) on `ghcr.io`. In the container the wizard never shows up, it runs straight away. For the proxy server use `--serve --serve-host 0.0.0.0` with `-p 127.0.0.1:8899:8899`, so the port is only open on your own machine.
 
-Tab completion for bash, zsh and fish:
+Tab completion for bash, zsh, fish and PowerShell:
 
 ```bash
 eval "$(proxy-scraper --completion zsh)"     # in ~/.zshrc (after compinit), same for bash in ~/.bashrc
 proxy-scraper --completion fish > ~/.config/fish/completions/proxy-scraper.fish
+proxy-scraper --completion powershell | Out-String | Invoke-Expression   # in $PROFILE
 ```
 
 Installed, the learned state lives in your user data folder (`~/Library/Application Support/proxy-scraper`, `%LOCALAPPDATA%\proxy-scraper` or `~/.local/share/proxy-scraper`; override with `PROXY_SCRAPER_HOME`) and results go to `./results`. Run from a clone, both stay inside the project.
@@ -578,14 +580,92 @@ for p in proxies:
         continue  # free proxies come and go – just take the next one
 ```
 
-**proxychains, Clash / Mihomo** – ready-made configs with `--export`
+**httpx** (`pip install "httpx[socks]"`) – straight from the hourly list, no scan
+
+```python
+import httpx
+from proxyscraper import live_proxies
+
+for p in live_proxies(types=["http", "socks5"], https=True, min_uptime=90, limit=10):  # httpx can't do SOCKS4
+    try:
+        with httpx.Client(proxy=p.url, timeout=10) as client:
+            print(p.url, "→", client.get("https://api.ipify.org").text)
+        break
+    except httpx.HTTPError:
+        continue  # next one
+```
+
+**aiohttp** (`pip install aiohttp aiohttp-socks` – aiohttp alone can't do SOCKS)
+
+```python
+import asyncio
+
+import aiohttp
+from aiohttp_socks import ProxyConnector, ProxyError
+from proxyscraper import live_proxies_async
+
+
+async def main():
+    for p in await live_proxies_async(https=True, min_uptime=90, limit=10):
+        try:
+            async with aiohttp.ClientSession(connector=ProxyConnector.from_url(p.url)) as session:
+                async with session.get("https://api.ipify.org", timeout=aiohttp.ClientTimeout(total=10)) as r:
+                    print(p.url, "→", await r.text())
+                    return
+        except (aiohttp.ClientError, ProxyError, asyncio.TimeoutError, OSError):
+            continue  # free proxies come and go – just take the next one
+
+asyncio.run(main())
+```
+
+**Scrapy** – a downloader middleware that sends every request, retries included, through the next proxy. Scrapy only speaks HTTP proxies, `https=True` picks the ones that can tunnel `https://` pages
+
+```python
+import itertools
+
+import scrapy
+from scrapy.crawler import CrawlerProcess
+from proxyscraper import live_proxies
+
+PROXIES = [p.url for p in live_proxies(types="http", https=True, min_uptime=50)]
+if not PROXIES:
+    raise SystemExit("No proxy matches right now – loosen the filters (e.g. min_uptime)")
+POOL = itertools.cycle(PROXIES)
+
+
+class RotatingProxy:
+    def process_request(self, request, spider=None):  # newer Scrapy leaves out spider
+        request.meta["proxy"] = next(POOL)
+
+
+class IpSpider(scrapy.Spider):
+    name = "ip"
+    start_urls = [f"https://api.ipify.org/?n={i}" for i in range(3)]
+    custom_settings = {
+        "DOWNLOADER_MIDDLEWARES": {f"{__name__}.RotatingProxy": 350},
+        "RETRY_TIMES": 5, "DOWNLOAD_TIMEOUT": 15,
+    }
+
+    def parse(self, response):
+        print(response.meta["proxy"], "→", response.text)
+
+
+if __name__ == "__main__":
+    process = CrawlerProcess()
+    process.crawl(IpSpider)
+    process.start()
+```
+
+In a Scrapy project, put `RotatingProxy` in `middlewares.py` and add it to `DOWNLOADER_MIDDLEWARES` in `settings.py`. For long crawls, reload the pool now and then – the list changes every hour.
+
+**proxychains, Clash / Mihomo, sing-box** – ready-made configs with `--export`
 
 ```bash
-proxy-scraper --want 30 -y --export proxychains,clash
+proxy-scraper --want 30 -y --export proxychains,clash,singbox
 proxychains4 -f results/latest/proxychains.conf curl https://api.ipify.org
 ```
 
-`clash.yaml` has all HTTP and SOCKS5 proxies plus a `url-test` group that always picks the fastest. Both configs leave out HTTP proxies that can't tunnel (`CONNECT`), because these tools tunnel everything.
+`clash.yaml` has all HTTP and SOCKS5 proxies plus a `url-test` group that always picks the fastest. `singbox.json` does the same for sing-box, SOCKS4 included, and opens a local proxy: `sing-box run -c results/latest/singbox.json`, then use `127.0.0.1:2080` as HTTP or SOCKS5 proxy. All three leave out HTTP proxies that can't tunnel (`CONNECT`), because these tools tunnel everything.
 
 **In a pipe** – `-o -` prints the hits to stdout, the interface moves to stderr
 
@@ -601,6 +681,22 @@ export HTTPS_PROXY=http://127.0.0.1:8899 HTTP_PROXY=http://127.0.0.1:8899
 pip download requests   # git, pip, npm & co. now go through the pool
 ```
 
+**In a GitHub workflow** – the action picks working proxies for the next steps
+
+```yaml
+- id: proxies
+  uses: maximilianfeix/proxy-scraper@v1
+  with:
+    types: socks5
+    https: true
+    min-uptime: 90        # on the list for 90 %+ of the week
+    works-on: google      # optional: google, reddit, amazon
+    recheck: true         # optional: check them again from the runner
+- run: curl -x "${{ steps.proxies.outputs.proxy }}" https://api.ipify.org
+```
+
+`proxy` is the fastest one, `file` a text file with all of them (up to `limit`, default 20), `count` how many. Without a match the step fails, unless `fail-if-empty: false`.
+
 **Without installing anything** – straight from the live list
 
 ```bash
@@ -644,16 +740,16 @@ curl.exe -x (Get-Content "$run\all.txt" -TotalCount 1) http://api.ipify.org
 | `-t`, `--timeout S` | timeout per proxy (default: 8 s) |
 | `--discover` | search GitHub for new sources right now |
 | `--no-cache` | download every list again (unchanged ones are normally skipped via ETag) |
-| `--list-sources [N]` | show the source ranking |
+| `--list-sources [N]` | show the source ranking (add `--json` for scripts: url, status, hit rate, checks, last change) |
 | `--serve-host ADDR` | where the proxy server listens (default `127.0.0.1`; `0.0.0.0` for Docker, with a warning) |
 | `--serve-password SECRET` | clients must send this password in the proxy login; better set `PROXY_SCRAPER_SERVE_PASSWORD` |
 | `--rotate STRATEGY` · `--sticky SEC` | how the proxy server picks proxies, see [above](#proxy-server) |
 | `--serve-refill HOURS` | while serving, check fresh proxies every HOURS and add the hits to the pool |
 | `--serve [PORT]` | afterwards serve as a rotating proxy on `127.0.0.1:PORT` (default: 8899) |
 | `-o FILE` | also write all hits to this file; `-o -` prints them to stdout |
-| `--export FORMATS` | extra files for other tools: `proxychains`, `clash`, `curl` or `all` |
+| `--export FORMATS` | extra files for other tools: `proxychains`, `clash`, `singbox`, `curl` or `all` |
 | `-V`, `--version` | print the version |
-| `--completion SHELL` | print the tab completion script for bash, zsh or fish |
+| `--completion SHELL` | print the tab completion script for bash, zsh, fish or PowerShell |
 
 Everything else: `proxy-scraper --help`
 
@@ -735,10 +831,12 @@ Only for things that don't matter. Public proxies are run by strangers who can r
 
 ## Roadmap
 
-What's next is tracked in the milestone [**v1.8**](../../milestone/8) – ideas and wishes are welcome as an [issue](../../issues/new/choose). During Hacktoberfest there are [beginner-friendly issues](../../issues?q=is%3Aopen+label%3Ahacktoberfest) with pointers on where to start.
+What's next is in the [open issues](../../issues) – ideas and wishes are welcome as an [issue](../../issues/new/choose). During Hacktoberfest there are [beginner-friendly issues](../../issues?q=is%3Aopen+label%3Ahacktoberfest) with pointers on where to start.
+
+- [ ] [Measure throughput, not just latency](../../issues/140)
+- [ ] [Daily snapshots as a dataset](../../issues/143)
 
-- [ ] [Protocol detection on the same connection](../../issues/44)
-- [ ] [sing-box export](../../issues/113), [PowerShell completion](../../issues/114), [shareable filters on the website](../../issues/116)
+Shipped in v1.8 and v1.9: uptime per proxy and `stable.txt`, which proxies get through to Google, Reddit and Amazon, `live_proxies()` in Python, a page per proxy with its week of checks, a weekly report, live charts in this README, shareable filters on the website, sources mined from other scrapers' lists, untyped lists tried as HTTP and SOCKS5, `--export singbox`, PowerShell completion and recipes for httpx, aiohttp and Scrapy.
 
 Shipped in [v1.7](../../milestone/7): an MCP server so AI agents get working proxies and can load pages through them, 28 new sources and a GitHub search that runs daily and keeps what it found, and a cleaner website.
 
@@ -783,7 +881,7 @@ proxyscraper/
 ├── geodb.py            DB-IP country database (monthly, binary search)
 ├── targets.py          target sites for --target
 ├── output.py           result files
-├── exporters.py        proxychains, Clash and curl formats (--export)
+├── exporters.py        proxychains, Clash, sing-box and curl formats (--export)
 ├── server/             rotating proxy server (--serve): pool · http · upstream · socks · status · core
 ├── api.py              find_proxies() / check_proxies() for Python
 ├── agent.py            MCP tools without the SDK: live list, filters, fetch through proxies

{proxy_scraper_cli-1.8.1 → proxy_scraper_cli-1.10.0}/proxy_scraper_cli.egg-info/SOURCES.txt

@@ -9,6 +9,7 @@ proxy_scraper_cli.egg-info/requires.txt
 proxy_scraper_cli.egg-info/top_level.txt
 proxyscraper/__init__.py
 proxyscraper/__main__.py
+proxyscraper/action.py
 proxyscraper/agent.py
 proxyscraper/api.py
 proxyscraper/app.py
@@ -36,7 +37,9 @@ proxyscraper/parsing.py
 proxyscraper/paths.py
 proxyscraper/pipeline.py
 proxyscraper/preferences.py
+proxyscraper/proxypages.py
 proxyscraper/publish.py
+proxyscraper/report.py
 proxyscraper/sites.py
 proxyscraper/sources.json
 proxyscraper/sources.py
@@ -60,6 +63,7 @@ proxyscraper/ui/report.py
 proxyscraper/ui/serve.py
 proxyscraper/ui/widgets.py
 proxyscraper/ui/wizard.py
+tests/test_action.py
 tests/test_agent.py
 tests/test_api.py
 tests/test_app.py
@@ -84,17 +88,22 @@ tests/test_output_stdout.py
 tests/test_packaging.py
 tests/test_pages.py
 tests/test_parsing.py
+tests/test_parsing_fuzz.py
 tests/test_paths.py
 tests/test_pipeline.py
+tests/test_proxy_pages.py
 tests/test_proxy_sources.py
 tests/test_publish.py
+tests/test_report.py
 tests/test_robustness.py
 tests/test_serve_password.py
 tests/test_serve_refill.py
 tests/test_serve_v2.py
 tests/test_server.py
+tests/test_server_closed_writer.py
 tests/test_server_fixes.py
 tests/test_sites.py
+tests/test_sources_json.py
 tests/test_tamper.py
 tests/test_targets.py
 tests/test_ui.py

{proxy_scraper_cli-1.8.1 → proxy_scraper_cli-1.10.0}/proxy_scraper_cli.egg-info/requires.txt

@@ -3,6 +3,7 @@ certifi
 
 [dev]
 pytest
+hypothesis
 ruff==0.16.8
 
 [fast]

{proxy_scraper_cli-1.8.1 → proxy_scraper_cli-1.10.0}/proxyscraper/__init__.py

@@ -1,7 +1,7 @@
 """proxy-scraper: collects public HTTP/SOCKS4/SOCKS5 proxies from hundreds of sources, checks them
 in parallel with its own protocol handshakes and learns which sources deliver good proxies."""
 
-__version__ = "1.8.1"
+__version__ = "1.10.0"
 
 
 def __getattr__(name):