proxy-scraper-cli 1.19.0__tar.gz → 1.21.0__tar.gz

{proxy_scraper_cli-1.19.0 → proxy_scraper_cli-1.21.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: proxy-scraper-cli
-Version: 1.19.0
+Version: 1.21.0
 Summary: Finds free HTTP, SOCKS4 and SOCKS5 proxies that actually work: 700+ sources, real checks, honeypot filtering and a rotating local proxy server.
 Author: Maximilian Feix
 License-Expression: MIT
@@ -52,6 +52,7 @@ Dynamic: license-file
 [![Live proxies](https://img.shields.io/endpoint?url=https%3A%2F%2Fraw.githubusercontent.com%2Fmaximilianfeix%2Fproxy-scraper%2Fproxy-list%2Fbadges%2Ftotal.json&style=flat-square&labelColor=121113)](#live-list)
 [![PyPI](https://img.shields.io/pypi/v/proxy-scraper-cli?style=flat-square&color=D4F77A&labelColor=121113&label=pypi)](https://pypi.org/project/proxy-scraper-cli/)
 [![Python](https://img.shields.io/badge/python-3.9–3.13-D4F77A?style=flat-square&labelColor=121113)](pyproject.toml)
+[![Stars](https://img.shields.io/github/stars/maximilianfeix/proxy-scraper?style=flat-square&color=D4F77A&labelColor=121113)](https://github.com/maximilianfeix/proxy-scraper/stargazers)
 [![License](https://img.shields.io/badge/license-MIT-D4F77A?style=flat-square&labelColor=121113)](LICENSE)
 
 <a href="https://maximilianfeix.github.io/proxy-scraper/"><img src="https://img.shields.io/badge/Browse_the_live_list-D4F77A?style=for-the-badge&labelColor=121113" alt="Browse the live list"></a>
@@ -94,7 +95,7 @@ Most free proxy lists are 95 % dead, and a good part of the rest are honeypots o
 - [Options](#options)
 - [GitHub Actions](#github-actions)
 - [FAQ](#faq)
-- [Roadmap](#roadmap) · [Contributing](#contributing) · [Acknowledgements](#acknowledgements)
+- [Roadmap](#roadmap) · [Contributing](#contributing) · [Community](#community) · [Acknowledgements](#acknowledgements)
 
 </details>
 
@@ -384,6 +385,7 @@ macOS, Linux and Windows, Python 3.9 to 3.13. Only two dependencies: `rich` and
 | Only proxies that reach *your* target site | ❌ | ✅ `--target` |
 | Learns which sources are worth it | ❌ | ✅ |
 | Usable as a single rotating proxy | ❌ | ✅ `--serve` |
+| API to fetch a proxy (proxy_pool compatible) | ❌ | ✅ [`/get`](#pool-api) |
 | Ready-made list without running anything | ✅ | ✅ [live list](#live-list) |
 
 <a id="examples"></a>
@@ -456,6 +458,21 @@ with ProxyRotator(country="DE") as rotator:
 
 Usually a few seconds; when many proxies in a row are down it can take a minute (`timeout=` is per attempt). There's `aget()` with `async with` for asyncio code.
 
+Already using requests, httpx, Playwright or Scrapy? `proxy_url()` gives you one local proxy address that rotates behind the scenes – every connection goes out through another proxy from the list, best first, dead ones skipped:
+
+```python
+import requests
+from proxyscraper import ProxyRotator
+
+with ProxyRotator(country="DE") as rotator:
+    proxy = rotator.proxy_url()          # http://country-de:…@127.0.0.1:…
+    requests.get("https://api.ipify.org", proxies={"http": proxy, "https": proxy})
+    # httpx.Client(proxy=proxy) · scrapy: meta={"proxy": proxy} · curl -x "$proxy"
+    # Playwright wants the login apart: chromium.launch(proxy=rotator.playwright_proxy())
+```
+
+It runs on 127.0.0.1 with a random password, takes over each new hourly list by itself and stops with the `with` block.
+
 <a id="proxy-server"></a>
 
 ## Rotating proxy server
@@ -504,6 +521,38 @@ Without a password, `--serve-host` means **anyone who reaches the port can use i
 
 In testing: 20 of 20 HTTPS requests succeeded, over 15 different exit IPs. In the wizard this is **Proxy server right away**.
 
+<a id="pool-api"></a>
+
+### Proxy pool API
+
+Some programs want a proxy *address*, not a proxy – to hand it to a browser, a worker or a queue. The same port answers plain HTTP requests with one:
+
+```bash
+curl http://127.0.0.1:8899/get                          # one proxy as JSON, chosen like a connection would be
+curl "http://127.0.0.1:8899/get?country=DE&https=1&format=txt"   # → http://203.0.113.7:8080
+curl "http://127.0.0.1:8899/all?protocol=socks5&limit=20"        # the 20 best SOCKS5 proxies
+curl "http://127.0.0.1:8899/report?proxy=203.0.113.7:8080&ok=0"  # it failed you – three times and it's out
+```
+
+| Endpoint | What it does |
+|---|---|
+| `/get` | one proxy, picked by the `--rotate` strategy |
+| `/pop` | like `/get`, and the proxy leaves the pool |
+| `/all` | every usable proxy, best first (`limit=N`) |
+| `/count` | totals by type and country |
+| `/delete?proxy=IP:PORT` | take a proxy out |
+| `/report?proxy=IP:PORT&ok=0` | feedback from your own requests |
+
+Filters work on `/get`, `/pop` and `/all`: `country=DE,AT`, `protocol=socks5`, `https=1`, `anonymity=elite`, `max_latency=1500`, and `format=txt` for plain URLs. With `--serve-password` the API wants it as Basic auth, like the status page.
+
+**Coming from [jhao104/proxy_pool](https://github.com/jhao104/proxy_pool)?** The endpoints, `type=https` and the JSON fields (`proxy`, `https`, `region`, `anonymous`, `check_count`, `fail_count`, …) are the same, so point your code at port 8899 and it keeps working – without Redis, and with proxies that passed the honeypot, tampering and TLS checks:
+
+```python
+import requests
+proxy = requests.get("http://127.0.0.1:8899/get?type=https").json()["url"]   # e.g. socks5://…, type included
+requests.get("https://example.com", proxies={"http": proxy, "https": proxy})
+```
+
 <a id="discord-bot"></a>
 
 ## Discord and Telegram bot
@@ -543,6 +592,15 @@ flowchart LR
 </div>
 </details>
 
+### Your own lists
+
+```bash
+proxy-scraper --source https://example.com/my-list.txt --source socks5=./socks.txt   # on top of the 700+ sources
+proxy-scraper --only-sources --source bought.txt --want 50                            # only yours
+```
+
+Any text with `ip:port` works; lines like `socks5://user:pass@host:port` keep their type, bare ones are tried as HTTP and SOCKS5 unless you write `http=…`.
+
 <a id="output"></a>
 
 ## Output
@@ -909,7 +967,7 @@ proxyscraper/
 ├── targets.py          target sites for --target
 ├── output.py           result files
 ├── exporters.py        proxychains, Clash, sing-box and curl formats (--export)
-├── server/             rotating proxy server (--serve): pool · http · upstream · socks · status · core
+├── server/             rotating proxy server (--serve): pool · http · upstream · socks · status · api · core
 ├── api.py              find_proxies() / check_proxies() for Python
 ├── agent.py            MCP tools without the SDK: live list, filters, fetch through proxies
 ├── mcp_server.py       MCP server (proxy-scraper-mcp) for AI agents
@@ -922,6 +980,20 @@ proxyscraper/
 
 </details>
 
+<a id="community"></a>
+
+## Community
+
+Questions, ideas and things you built with it go to [Discussions](https://github.com/maximilianfeix/proxy-scraper/discussions) – bugs to the [issues](../../issues). If proxy-scraper saves you time, a ⭐ helps others find it.
+
+<a href="https://star-history.com/#maximilianfeix/proxy-scraper&Date">
+  <picture>
+    <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=maximilianfeix/proxy-scraper&type=Date&theme=dark">
+    <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=maximilianfeix/proxy-scraper&type=Date">
+    <img alt="Star history of proxy-scraper" src="https://api.star-history.com/svg?repos=maximilianfeix/proxy-scraper&type=Date" width="600">
+  </picture>
+</a>
+
 <a id="acknowledgements"></a>
 
 ## Acknowledgements

{proxy_scraper_cli-1.19.0 → proxy_scraper_cli-1.21.0}/README.md

@@ -11,6 +11,7 @@
 [![Live proxies](https://img.shields.io/endpoint?url=https%3A%2F%2Fraw.githubusercontent.com%2Fmaximilianfeix%2Fproxy-scraper%2Fproxy-list%2Fbadges%2Ftotal.json&style=flat-square&labelColor=121113)](#live-list)
 [![PyPI](https://img.shields.io/pypi/v/proxy-scraper-cli?style=flat-square&color=D4F77A&labelColor=121113&label=pypi)](https://pypi.org/project/proxy-scraper-cli/)
 [![Python](https://img.shields.io/badge/python-3.9–3.13-D4F77A?style=flat-square&labelColor=121113)](pyproject.toml)
+[![Stars](https://img.shields.io/github/stars/maximilianfeix/proxy-scraper?style=flat-square&color=D4F77A&labelColor=121113)](https://github.com/maximilianfeix/proxy-scraper/stargazers)
 [![License](https://img.shields.io/badge/license-MIT-D4F77A?style=flat-square&labelColor=121113)](LICENSE)
 
 <a href="https://maximilianfeix.github.io/proxy-scraper/"><img src="https://img.shields.io/badge/Browse_the_live_list-D4F77A?style=for-the-badge&labelColor=121113" alt="Browse the live list"></a>
@@ -53,7 +54,7 @@ Most free proxy lists are 95 % dead, and a good part of the rest are honeypots o
 - [Options](#options)
 - [GitHub Actions](#github-actions)
 - [FAQ](#faq)
-- [Roadmap](#roadmap) · [Contributing](#contributing) · [Acknowledgements](#acknowledgements)
+- [Roadmap](#roadmap) · [Contributing](#contributing) · [Community](#community) · [Acknowledgements](#acknowledgements)
 
 </details>
 
@@ -343,6 +344,7 @@ macOS, Linux and Windows, Python 3.9 to 3.13. Only two dependencies: `rich` and
 | Only proxies that reach *your* target site | ❌ | ✅ `--target` |
 | Learns which sources are worth it | ❌ | ✅ |
 | Usable as a single rotating proxy | ❌ | ✅ `--serve` |
+| API to fetch a proxy (proxy_pool compatible) | ❌ | ✅ [`/get`](#pool-api) |
 | Ready-made list without running anything | ✅ | ✅ [live list](#live-list) |
 
 <a id="examples"></a>
@@ -415,6 +417,21 @@ with ProxyRotator(country="DE") as rotator:
 
 Usually a few seconds; when many proxies in a row are down it can take a minute (`timeout=` is per attempt). There's `aget()` with `async with` for asyncio code.
 
+Already using requests, httpx, Playwright or Scrapy? `proxy_url()` gives you one local proxy address that rotates behind the scenes – every connection goes out through another proxy from the list, best first, dead ones skipped:
+
+```python
+import requests
+from proxyscraper import ProxyRotator
+
+with ProxyRotator(country="DE") as rotator:
+    proxy = rotator.proxy_url()          # http://country-de:…@127.0.0.1:…
+    requests.get("https://api.ipify.org", proxies={"http": proxy, "https": proxy})
+    # httpx.Client(proxy=proxy) · scrapy: meta={"proxy": proxy} · curl -x "$proxy"
+    # Playwright wants the login apart: chromium.launch(proxy=rotator.playwright_proxy())
+```
+
+It runs on 127.0.0.1 with a random password, takes over each new hourly list by itself and stops with the `with` block.
+
 <a id="proxy-server"></a>
 
 ## Rotating proxy server
@@ -463,6 +480,38 @@ Without a password, `--serve-host` means **anyone who reaches the port can use i
 
 In testing: 20 of 20 HTTPS requests succeeded, over 15 different exit IPs. In the wizard this is **Proxy server right away**.
 
+<a id="pool-api"></a>
+
+### Proxy pool API
+
+Some programs want a proxy *address*, not a proxy – to hand it to a browser, a worker or a queue. The same port answers plain HTTP requests with one:
+
+```bash
+curl http://127.0.0.1:8899/get                          # one proxy as JSON, chosen like a connection would be
+curl "http://127.0.0.1:8899/get?country=DE&https=1&format=txt"   # → http://203.0.113.7:8080
+curl "http://127.0.0.1:8899/all?protocol=socks5&limit=20"        # the 20 best SOCKS5 proxies
+curl "http://127.0.0.1:8899/report?proxy=203.0.113.7:8080&ok=0"  # it failed you – three times and it's out
+```
+
+| Endpoint | What it does |
+|---|---|
+| `/get` | one proxy, picked by the `--rotate` strategy |
+| `/pop` | like `/get`, and the proxy leaves the pool |
+| `/all` | every usable proxy, best first (`limit=N`) |
+| `/count` | totals by type and country |
+| `/delete?proxy=IP:PORT` | take a proxy out |
+| `/report?proxy=IP:PORT&ok=0` | feedback from your own requests |
+
+Filters work on `/get`, `/pop` and `/all`: `country=DE,AT`, `protocol=socks5`, `https=1`, `anonymity=elite`, `max_latency=1500`, and `format=txt` for plain URLs. With `--serve-password` the API wants it as Basic auth, like the status page.
+
+**Coming from [jhao104/proxy_pool](https://github.com/jhao104/proxy_pool)?** The endpoints, `type=https` and the JSON fields (`proxy`, `https`, `region`, `anonymous`, `check_count`, `fail_count`, …) are the same, so point your code at port 8899 and it keeps working – without Redis, and with proxies that passed the honeypot, tampering and TLS checks:
+
+```python
+import requests
+proxy = requests.get("http://127.0.0.1:8899/get?type=https").json()["url"]   # e.g. socks5://…, type included
+requests.get("https://example.com", proxies={"http": proxy, "https": proxy})
+```
+
 <a id="discord-bot"></a>
 
 ## Discord and Telegram bot
@@ -502,6 +551,15 @@ flowchart LR
 </div>
 </details>
 
+### Your own lists
+
+```bash
+proxy-scraper --source https://example.com/my-list.txt --source socks5=./socks.txt   # on top of the 700+ sources
+proxy-scraper --only-sources --source bought.txt --want 50                            # only yours
+```
+
+Any text with `ip:port` works; lines like `socks5://user:pass@host:port` keep their type, bare ones are tried as HTTP and SOCKS5 unless you write `http=…`.
+
 <a id="output"></a>
 
 ## Output
@@ -868,7 +926,7 @@ proxyscraper/
 ├── targets.py          target sites for --target
 ├── output.py           result files
 ├── exporters.py        proxychains, Clash, sing-box and curl formats (--export)
-├── server/             rotating proxy server (--serve): pool · http · upstream · socks · status · core
+├── server/             rotating proxy server (--serve): pool · http · upstream · socks · status · api · core
 ├── api.py              find_proxies() / check_proxies() for Python
 ├── agent.py            MCP tools without the SDK: live list, filters, fetch through proxies
 ├── mcp_server.py       MCP server (proxy-scraper-mcp) for AI agents
@@ -881,6 +939,20 @@ proxyscraper/
 
 </details>
 
+<a id="community"></a>
+
+## Community
+
+Questions, ideas and things you built with it go to [Discussions](https://github.com/maximilianfeix/proxy-scraper/discussions) – bugs to the [issues](../../issues). If proxy-scraper saves you time, a ⭐ helps others find it.
+
+<a href="https://star-history.com/#maximilianfeix/proxy-scraper&Date">
+  <picture>
+    <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=maximilianfeix/proxy-scraper&type=Date&theme=dark">
+    <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=maximilianfeix/proxy-scraper&type=Date">
+    <img alt="Star history of proxy-scraper" src="https://api.star-history.com/svg?repos=maximilianfeix/proxy-scraper&type=Date" width="600">
+  </picture>
+</a>
+
 <a id="acknowledgements"></a>
 
 ## Acknowledgements

{proxy_scraper_cli-1.19.0 → proxy_scraper_cli-1.21.0}/proxy_scraper_cli.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: proxy-scraper-cli
-Version: 1.19.0
+Version: 1.21.0
 Summary: Finds free HTTP, SOCKS4 and SOCKS5 proxies that actually work: 700+ sources, real checks, honeypot filtering and a rotating local proxy server.
 Author: Maximilian Feix
 License-Expression: MIT
@@ -52,6 +52,7 @@ Dynamic: license-file
 [![Live proxies](https://img.shields.io/endpoint?url=https%3A%2F%2Fraw.githubusercontent.com%2Fmaximilianfeix%2Fproxy-scraper%2Fproxy-list%2Fbadges%2Ftotal.json&style=flat-square&labelColor=121113)](#live-list)
 [![PyPI](https://img.shields.io/pypi/v/proxy-scraper-cli?style=flat-square&color=D4F77A&labelColor=121113&label=pypi)](https://pypi.org/project/proxy-scraper-cli/)
 [![Python](https://img.shields.io/badge/python-3.9–3.13-D4F77A?style=flat-square&labelColor=121113)](pyproject.toml)
+[![Stars](https://img.shields.io/github/stars/maximilianfeix/proxy-scraper?style=flat-square&color=D4F77A&labelColor=121113)](https://github.com/maximilianfeix/proxy-scraper/stargazers)
 [![License](https://img.shields.io/badge/license-MIT-D4F77A?style=flat-square&labelColor=121113)](LICENSE)
 
 <a href="https://maximilianfeix.github.io/proxy-scraper/"><img src="https://img.shields.io/badge/Browse_the_live_list-D4F77A?style=for-the-badge&labelColor=121113" alt="Browse the live list"></a>
@@ -94,7 +95,7 @@ Most free proxy lists are 95 % dead, and a good part of the rest are honeypots o
 - [Options](#options)
 - [GitHub Actions](#github-actions)
 - [FAQ](#faq)
-- [Roadmap](#roadmap) · [Contributing](#contributing) · [Acknowledgements](#acknowledgements)
+- [Roadmap](#roadmap) · [Contributing](#contributing) · [Community](#community) · [Acknowledgements](#acknowledgements)
 
 </details>
 
@@ -384,6 +385,7 @@ macOS, Linux and Windows, Python 3.9 to 3.13. Only two dependencies: `rich` and
 | Only proxies that reach *your* target site | ❌ | ✅ `--target` |
 | Learns which sources are worth it | ❌ | ✅ |
 | Usable as a single rotating proxy | ❌ | ✅ `--serve` |
+| API to fetch a proxy (proxy_pool compatible) | ❌ | ✅ [`/get`](#pool-api) |
 | Ready-made list without running anything | ✅ | ✅ [live list](#live-list) |
 
 <a id="examples"></a>
@@ -456,6 +458,21 @@ with ProxyRotator(country="DE") as rotator:
 
 Usually a few seconds; when many proxies in a row are down it can take a minute (`timeout=` is per attempt). There's `aget()` with `async with` for asyncio code.
 
+Already using requests, httpx, Playwright or Scrapy? `proxy_url()` gives you one local proxy address that rotates behind the scenes – every connection goes out through another proxy from the list, best first, dead ones skipped:
+
+```python
+import requests
+from proxyscraper import ProxyRotator
+
+with ProxyRotator(country="DE") as rotator:
+    proxy = rotator.proxy_url()          # http://country-de:…@127.0.0.1:…
+    requests.get("https://api.ipify.org", proxies={"http": proxy, "https": proxy})
+    # httpx.Client(proxy=proxy) · scrapy: meta={"proxy": proxy} · curl -x "$proxy"
+    # Playwright wants the login apart: chromium.launch(proxy=rotator.playwright_proxy())
+```
+
+It runs on 127.0.0.1 with a random password, takes over each new hourly list by itself and stops with the `with` block.
+
 <a id="proxy-server"></a>
 
 ## Rotating proxy server
@@ -504,6 +521,38 @@ Without a password, `--serve-host` means **anyone who reaches the port can use i
 
 In testing: 20 of 20 HTTPS requests succeeded, over 15 different exit IPs. In the wizard this is **Proxy server right away**.
 
+<a id="pool-api"></a>
+
+### Proxy pool API
+
+Some programs want a proxy *address*, not a proxy – to hand it to a browser, a worker or a queue. The same port answers plain HTTP requests with one:
+
+```bash
+curl http://127.0.0.1:8899/get                          # one proxy as JSON, chosen like a connection would be
+curl "http://127.0.0.1:8899/get?country=DE&https=1&format=txt"   # → http://203.0.113.7:8080
+curl "http://127.0.0.1:8899/all?protocol=socks5&limit=20"        # the 20 best SOCKS5 proxies
+curl "http://127.0.0.1:8899/report?proxy=203.0.113.7:8080&ok=0"  # it failed you – three times and it's out
+```
+
+| Endpoint | What it does |
+|---|---|
+| `/get` | one proxy, picked by the `--rotate` strategy |
+| `/pop` | like `/get`, and the proxy leaves the pool |
+| `/all` | every usable proxy, best first (`limit=N`) |
+| `/count` | totals by type and country |
+| `/delete?proxy=IP:PORT` | take a proxy out |
+| `/report?proxy=IP:PORT&ok=0` | feedback from your own requests |
+
+Filters work on `/get`, `/pop` and `/all`: `country=DE,AT`, `protocol=socks5`, `https=1`, `anonymity=elite`, `max_latency=1500`, and `format=txt` for plain URLs. With `--serve-password` the API wants it as Basic auth, like the status page.
+
+**Coming from [jhao104/proxy_pool](https://github.com/jhao104/proxy_pool)?** The endpoints, `type=https` and the JSON fields (`proxy`, `https`, `region`, `anonymous`, `check_count`, `fail_count`, …) are the same, so point your code at port 8899 and it keeps working – without Redis, and with proxies that passed the honeypot, tampering and TLS checks:
+
+```python
+import requests
+proxy = requests.get("http://127.0.0.1:8899/get?type=https").json()["url"]   # e.g. socks5://…, type included
+requests.get("https://example.com", proxies={"http": proxy, "https": proxy})
+```
+
 <a id="discord-bot"></a>
 
 ## Discord and Telegram bot
@@ -543,6 +592,15 @@ flowchart LR
 </div>
 </details>
 
+### Your own lists
+
+```bash
+proxy-scraper --source https://example.com/my-list.txt --source socks5=./socks.txt   # on top of the 700+ sources
+proxy-scraper --only-sources --source bought.txt --want 50                            # only yours
+```
+
+Any text with `ip:port` works; lines like `socks5://user:pass@host:port` keep their type, bare ones are tried as HTTP and SOCKS5 unless you write `http=…`.
+
 <a id="output"></a>
 
 ## Output
@@ -909,7 +967,7 @@ proxyscraper/
 ├── targets.py          target sites for --target
 ├── output.py           result files
 ├── exporters.py        proxychains, Clash, sing-box and curl formats (--export)
-├── server/             rotating proxy server (--serve): pool · http · upstream · socks · status · core
+├── server/             rotating proxy server (--serve): pool · http · upstream · socks · status · api · core
 ├── api.py              find_proxies() / check_proxies() for Python
 ├── agent.py            MCP tools without the SDK: live list, filters, fetch through proxies
 ├── mcp_server.py       MCP server (proxy-scraper-mcp) for AI agents
@@ -922,6 +980,20 @@ proxyscraper/
 
 </details>
 
+<a id="community"></a>
+
+## Community
+
+Questions, ideas and things you built with it go to [Discussions](https://github.com/maximilianfeix/proxy-scraper/discussions) – bugs to the [issues](../../issues). If proxy-scraper saves you time, a ⭐ helps others find it.
+
+<a href="https://star-history.com/#maximilianfeix/proxy-scraper&Date">
+  <picture>
+    <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=maximilianfeix/proxy-scraper&type=Date&theme=dark">
+    <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=maximilianfeix/proxy-scraper&type=Date">
+    <img alt="Star history of proxy-scraper" src="https://api.star-history.com/svg?repos=maximilianfeix/proxy-scraper&type=Date" width="600">
+  </picture>
+</a>
+
 <a id="acknowledgements"></a>
 
 ## Acknowledgements

{proxy_scraper_cli-1.19.0 → proxy_scraper_cli-1.21.0}/proxy_scraper_cli.egg-info/SOURCES.txt

@@ -48,6 +48,7 @@ proxyscraper/sources.py
 proxyscraper/speed.py
 proxyscraper/targets.py
 proxyscraper/server/__init__.py
+proxyscraper/server/api.py
 proxyscraper/server/core.py
 proxyscraper/server/http.py
 proxyscraper/server/pool.py
@@ -88,6 +89,7 @@ tests/test_mcp_server.py
 tests/test_mine_sources.py
 tests/test_options.py
 tests/test_output_stdout.py
+tests/test_own_sources.py
 tests/test_packaging.py
 tests/test_pages.py
 tests/test_parsing.py
@@ -101,6 +103,7 @@ tests/test_publish.py
 tests/test_report.py
 tests/test_robustness.py
 tests/test_rotator.py
+tests/test_serve_api.py
 tests/test_serve_password.py
 tests/test_serve_refill.py
 tests/test_serve_v2.py

{proxy_scraper_cli-1.19.0 → proxy_scraper_cli-1.21.0}/proxyscraper/__init__.py

@@ -1,7 +1,7 @@
 """proxy-scraper: collects public HTTP/SOCKS4/SOCKS5 proxies from hundreds of sources, checks them
 in parallel with its own protocol handshakes and learns which sources deliver good proxies."""
 
-__version__ = "1.19.0"
+__version__ = "1.21.0"
 
 
 def __getattr__(name):

{proxy_scraper_cli-1.19.0 → proxy_scraper_cli-1.21.0}/proxyscraper/agent.py

@@ -458,6 +458,22 @@ def _to_text(body: bytes, headers, content_type: str, raw_html: bool) -> str:
     return text if raw_html or "html" not in content_type.lower() else html_to_text(text)
 
 
+def _require_match(live: LiveList, protocol: str, countries: List[str], tls: bool) -> None:
+    if not any(True for _ in matching(live.rows, protocol, countries, https_only=tls, run_hours=live.run_hours)):
+        what = " ".join(x for x in (countries[0] if countries else "", "" if protocol == "any" else protocol,
+                                    "HTTPS-capable" if tls else "") if x)
+        raise AgentError(f"No {what} proxy in the live list right now."
+                         + (shortage_note(live.rows, 1, 0, countries) or ""))
+
+
+def _wishes(protocol: str, countries: List[str]) -> List[str]:
+    """What the rotating server reads from the proxy login: country-de, type-socks5."""
+    wishes = [f"country-{countries[0].lower()}"] if countries else []
+    if protocol != "any":
+        wishes.append(f"type-{protocol}")
+    return wishes
+
+
 class PageFetcher:
     """Loads pages through the rotating server of this package, running on 127.0.0.1 with a random password:
     the same failover as `--serve`, and HTTPS only through proxies that passed the verified-TLS test.
@@ -503,6 +519,19 @@ class PageFetcher:
                 self._list_id = list_id
         return live
 
+    async def local_proxy(self, protocol: str = "any", country: str = "") -> str:
+        """The URL of the rotating server behind this fetcher, with the login, for any HTTP client:
+        http://country-de:<password>@127.0.0.1:<port>. Every connection through it gets a proxy of its own."""
+        protocol = _check_protocol(protocol)
+        countries = normalize_countries([country] if country else [])
+        _require_match(await self._ready(), protocol, countries, tls=False)
+        user = "-".join(_wishes(protocol, countries)) or "any"
+        return f"http://{user}:{self._password}@127.0.0.1:{self.server.port}"
+
+    async def refresh(self) -> None:
+        """Take over a new run's list, if there is one (fetch does that by itself)."""
+        await self._ready()
+
     async def fetch(self, url: str, protocol: str = "any", country: str = "", max_chars: int = 20000,
                     raw_html: bool = False, progress: Optional[Progress] = None, tick: float = 5.0,
                     with_body: bool = False) -> dict:
@@ -511,15 +540,9 @@ class PageFetcher:
         countries = normalize_countries([country] if country else [])
         live = await self._ready()
         tls = urlsplit(url).scheme == "https"
-        if not any(True for _ in matching(live.rows, protocol, countries, https_only=tls, run_hours=live.run_hours)):
-            what = " ".join(x for x in (countries[0] if countries else "", "" if protocol == "any" else protocol,
-                                        "HTTPS-capable" if tls else "") if x)
-            raise AgentError(f"No {what} proxy in the live list right now."
-                             + (shortage_note(live.rows, 1, 0, countries) or ""))
+        _require_match(live, protocol, countries, tls)
         session = secrets.token_hex(6)
-        wishes = [f"country-{countries[0].lower()}"] if countries else []
-        if protocol != "any":
-            wishes.append(f"type-{protocol}")
+        wishes = _wishes(protocol, countries)
         pool, port = self.server.pool, self.server.port
         started = time.monotonic()
         result, faults, suspects = None, [], []

{proxy_scraper_cli-1.19.0 → proxy_scraper_cli-1.21.0}/proxyscraper/api.py

@@ -27,12 +27,14 @@ import asyncio
 import contextlib
 import email.message
 import io
+import logging
 import os
 import re
 import tempfile
 import threading
 from dataclasses import dataclass, field
 from typing import Dict, Iterable, List, Optional, Tuple
+from urllib.parse import urlsplit
 
 from rich.console import Console
 
@@ -42,6 +44,8 @@ from .parsing import PROXY_TYPES
 from .targets import parse_target
 from .ui import widgets
 
+log = logging.getLogger(__name__)
+
 __all__ = [
     "CheckResult",
     "LiveProxy",
@@ -309,6 +313,15 @@ class ProxyResponse:
             raise ConnectionError(f"the body from {self.final_url} was cut off at 2 MB")
 
 
+REFRESH_SECONDS = 300  # proxy_url(): how often the pool behind it looks for a new run's list
+
+
+def _playwright(url: str) -> Dict[str, str]:
+    parts = urlsplit(url)
+    return {"server": f"{parts.scheme}://{parts.hostname}:{parts.port}", "username": parts.username or "",
+            "password": parts.password or ""}
+
+
 class ProxyRotator:
     """Loads URLs through proxies from the hourly list and switches to the next one when a proxy fails –
     the loop every scraper writes by hand. HTTPS only through proxies with verified TLS.
@@ -330,6 +343,7 @@ class ProxyRotator:
         self._loop: Optional[asyncio.AbstractEventLoop] = None  # the sync API's own loop, in a thread
         self._thread: Optional[threading.Thread] = None
         self._start_lock = threading.Lock()
+        self._refresher: Optional[asyncio.Task] = None
 
     def _use(self, mode: str) -> None:
         if self._mode is None:
@@ -367,7 +381,43 @@ class ProxyRotator:
                              proxy_country=page["proxy_country"], elapsed_ms=page["elapsed_ms"],
                              truncated=page["body_truncated"])
 
+    async def aproxy_url(self) -> str:
+        self._use("async")
+        return await self._proxy_url()
+
+    async def _proxy_url(self) -> str:
+        from .agent import AgentError
+
+        if self._fetcher is None:
+            self._fetcher = self._make_fetcher()
+        try:
+            url = await self._fetcher.local_proxy(self.protocol, self.country)
+        except AgentError as e:
+            raise ConnectionError(str(e)) from None
+        if self._refresher is None:  # nobody calls get() on this path, so the pool refreshes by itself
+            self._refresher = asyncio.ensure_future(self._keep_fresh())
+        return url
+
+    async def aplaywright_proxy(self) -> Dict[str, str]:
+        self._use("async")
+        return _playwright(await self._proxy_url())
+
+    async def _keep_fresh(self) -> None:
+        from .agent import AgentError
+
+        while True:
+            await asyncio.sleep(REFRESH_SECONDS)
+            try:
+                await self._fetcher.refresh()
+            except AgentError:
+                pass  # GitHub not reachable right now: the pool keeps what it has, next try later
+            except Exception as e:  # a broken list must not end the refreshing for good
+                log.warning("proxy-scraper: could not refresh the proxy list: %s", e)
+
     async def aclose(self) -> None:
+        if self._refresher is not None:
+            self._refresher.cancel()
+            self._refresher = None
         if self._fetcher is not None:
             await self._fetcher.close()
             self._fetcher = None
@@ -380,13 +430,37 @@ class ProxyRotator:
 
     # sync: one event loop in a background thread keeps the proxy server alive between calls
     def get(self, url: str) -> ProxyResponse:
-        with self._start_lock:  # several threads calling get() at once still share one loop
-            self._use("sync")
+        return self._run_sync(self._aget(url))
+
+    def _run_sync(self, coro):
+        with self._start_lock:  # several threads calling at once still share one loop
+            try:
+                self._use("sync")
+            except RuntimeError:
+                coro.close()
+                raise
             if self._loop is None:
                 self._loop = asyncio.new_event_loop()
                 self._thread = threading.Thread(target=self._loop.run_forever, name="proxy-rotator", daemon=True)
                 self._thread.start()
-        return asyncio.run_coroutine_threadsafe(self._aget(url), self._loop).result()
+        return asyncio.run_coroutine_threadsafe(coro, self._loop).result()
+
+    def proxy_url(self) -> str:
+        """A local proxy URL for any library – requests, httpx, aiohttp, Playwright, Scrapy, curl:
+
+            with ProxyRotator(country="DE") as rotator:
+                proxy = rotator.proxy_url()
+                requests.get(url, proxies={"http": proxy, "https": proxy})
+
+        Every connection through it gets a proxy from the live list (best first, with failover), HTTPS only
+        through verified-TLS ones. It lives as long as the rotator and takes over each new run's list."""
+        return self._run_sync(self._proxy_url())
+
+    def playwright_proxy(self) -> Dict[str, str]:
+        """proxy_url() the way Playwright wants it – it ignores a login inside the server address:
+
+            browser = p.chromium.launch(proxy=rotator.playwright_proxy())"""
+        return _playwright(self.proxy_url())
 
     def close(self) -> None:
         with self._start_lock:

{proxy_scraper_cli-1.19.0 → proxy_scraper_cli-1.21.0}/proxyscraper/app.py

@@ -327,8 +327,10 @@ class Run:
         plan = await collect_sources(self.opts, self.quality)
         info("Sources", Text.assemble(
             (fmt(len(plan.sources)), f"bold {ACCENT}"), " active  ",
-            (f"({fmt(plan.n_curated)} curated · {fmt(plan.n_meta)} from {plan.meta_ok}/{plan.meta_total} "
-             f"meta lists · {fmt(plan.n_discovered)} discovered)", MUTED),
+            (f"({fmt(plan.n_own)} own)" if self.opts.only_sources else
+             f"({fmt(plan.n_curated)} curated · {fmt(plan.n_meta)} from {plan.meta_ok}/{plan.meta_total} "
+             f"meta lists · {fmt(plan.n_discovered)} discovered"
+             + (f" · {fmt(plan.n_own)} own)" if plan.n_own else ")"), MUTED),
         ))
         if plan.discovery_ran and not plan.discovery_token:
             note("GitHub discovery is limited without a token – run `gh auth login` or set GITHUB_TOKEN.",
@@ -355,7 +357,7 @@ class Run:
             hosts = sorted(INSECURE_HOSTS)
             note(f"Certificate could not be verified (TLS inspection on this network?), loaded anyway: "
                  f"{', '.join(hosts[:4])}{' …' if len(hosts) > 4 else ''}", MUTED, "ℹ")
-        return prioritize(res, self.quality, self.history, self.opts.types)
+        return prioritize(res, self.quality, self.history, self.opts.types, listed_only=self.opts.only_sources)
 
     # ------------------------------------------------------------------ phase 3+4: check, learn, report