proxy-scraper-cli 1.20.0__tar.gz → 1.22.0__tar.gz

{proxy_scraper_cli-1.20.0 → proxy_scraper_cli-1.22.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: proxy-scraper-cli
-Version: 1.20.0
+Version: 1.22.0
 Summary: Finds free HTTP, SOCKS4 and SOCKS5 proxies that actually work: 700+ sources, real checks, honeypot filtering and a rotating local proxy server.
 Author: Maximilian Feix
 License-Expression: MIT
@@ -52,6 +52,7 @@ Dynamic: license-file
 [![Live proxies](https://img.shields.io/endpoint?url=https%3A%2F%2Fraw.githubusercontent.com%2Fmaximilianfeix%2Fproxy-scraper%2Fproxy-list%2Fbadges%2Ftotal.json&style=flat-square&labelColor=121113)](#live-list)
 [![PyPI](https://img.shields.io/pypi/v/proxy-scraper-cli?style=flat-square&color=D4F77A&labelColor=121113&label=pypi)](https://pypi.org/project/proxy-scraper-cli/)
 [![Python](https://img.shields.io/badge/python-3.9–3.13-D4F77A?style=flat-square&labelColor=121113)](pyproject.toml)
+[![Stars](https://img.shields.io/github/stars/maximilianfeix/proxy-scraper?style=flat-square&color=D4F77A&labelColor=121113)](https://github.com/maximilianfeix/proxy-scraper/stargazers)
 [![License](https://img.shields.io/badge/license-MIT-D4F77A?style=flat-square&labelColor=121113)](LICENSE)
 
 <a href="https://maximilianfeix.github.io/proxy-scraper/"><img src="https://img.shields.io/badge/Browse_the_live_list-D4F77A?style=for-the-badge&labelColor=121113" alt="Browse the live list"></a>
@@ -68,6 +69,8 @@ Dynamic: license-file
 
 Most free proxy lists are 95 % dead, and a good part of the rest are honeypots or proxies that inject scripts into your pages. **proxy-scraper** collects public HTTP, SOCKS4 and SOCKS5 proxies from 700+ sources and keeps only the ones that pass every check. It learns with every run which sources are worth it, and it can turn the result into a single rotating proxy.
 
+**In numbers:** in one day it checked 3.6 million free proxies – 1.1 % worked, and more than half of the ones that answered failed a second request. [What's actually out there →](docs/free-proxies-in-numbers.md)
+
 <picture>
   <source media="(prefers-color-scheme: dark)" srcset="docs/checks-dark.svg">
   <source media="(prefers-color-scheme: light)" srcset="docs/checks-light.svg">
@@ -94,7 +97,7 @@ Most free proxy lists are 95 % dead, and a good part of the rest are honeypots o
 - [Options](#options)
 - [GitHub Actions](#github-actions)
 - [FAQ](#faq)
-- [Roadmap](#roadmap) · [Contributing](#contributing) · [Acknowledgements](#acknowledgements)
+- [Roadmap](#roadmap) · [Contributing](#contributing) · [Community](#community) · [Acknowledgements](#acknowledgements)
 
 </details>
 
@@ -147,7 +150,7 @@ proxy-scraper --completion fish > ~/.config/fish/completions/proxy-scraper.fish
 proxy-scraper --completion powershell | Out-String | Invoke-Expression   # in $PROFILE
 ```
 
-Installed, the learned state lives in your user data folder (`~/Library/Application Support/proxy-scraper`, `%LOCALAPPDATA%\proxy-scraper` or `~/.local/share/proxy-scraper`; override with `PROXY_SCRAPER_HOME`) and results go to `./results`. Run from a clone, both stay inside the project.
+Installed, the learned state lives in your user data folder (`~/Library/Application Support/proxy-scraper`, `%LOCALAPPDATA%\proxy-scraper` or `~/.local/share/proxy-scraper`; override with `PROXY_SCRAPER_HOME`) and results go to `./results`. Run from a clone, both stay inside the project. After the second run that finds proxies it asks once for a GitHub star; `PROXY_SCRAPER_NO_STAR_HINT=1` turns that off (it never shows in CI).
 
 </details>
 
@@ -384,6 +387,7 @@ macOS, Linux and Windows, Python 3.9 to 3.13. Only two dependencies: `rich` and
 | Only proxies that reach *your* target site | ❌ | ✅ `--target` |
 | Learns which sources are worth it | ❌ | ✅ |
 | Usable as a single rotating proxy | ❌ | ✅ `--serve` |
+| API to fetch a proxy (proxy_pool compatible) | ❌ | ✅ [`/get`](#pool-api) |
 | Ready-made list without running anything | ✅ | ✅ [live list](#live-list) |
 
 <a id="examples"></a>
@@ -519,6 +523,38 @@ Without a password, `--serve-host` means **anyone who reaches the port can use i
 
 In testing: 20 of 20 HTTPS requests succeeded, over 15 different exit IPs. In the wizard this is **Proxy server right away**.
 
+<a id="pool-api"></a>
+
+### Proxy pool API
+
+Some programs want a proxy *address*, not a proxy – to hand it to a browser, a worker or a queue. The same port answers plain HTTP requests with one:
+
+```bash
+curl http://127.0.0.1:8899/get                          # one proxy as JSON, chosen like a connection would be
+curl "http://127.0.0.1:8899/get?country=DE&https=1&format=txt"   # → http://203.0.113.7:8080
+curl "http://127.0.0.1:8899/all?protocol=socks5&limit=20"        # the 20 best SOCKS5 proxies
+curl "http://127.0.0.1:8899/report?proxy=203.0.113.7:8080&ok=0"  # it failed you – three times and it's out
+```
+
+| Endpoint | What it does |
+|---|---|
+| `/get` | one proxy, picked by the `--rotate` strategy |
+| `/pop` | like `/get`, and the proxy leaves the pool |
+| `/all` | every usable proxy, best first (`limit=N`) |
+| `/count` | totals by type and country |
+| `/delete?proxy=IP:PORT` | take a proxy out |
+| `/report?proxy=IP:PORT&ok=0` | feedback from your own requests |
+
+Filters work on `/get`, `/pop` and `/all`: `country=DE,AT`, `protocol=socks5`, `https=1`, `anonymity=elite`, `max_latency=1500`, and `format=txt` for plain URLs. With `--serve-password` the API wants it as Basic auth, like the status page.
+
+**Coming from [jhao104/proxy_pool](https://github.com/jhao104/proxy_pool)?** The endpoints, `type=https` and the JSON fields (`proxy`, `https`, `region`, `anonymous`, `check_count`, `fail_count`, …) are the same, so point your code at port 8899 and it keeps working – without Redis, and with proxies that passed the honeypot, tampering and TLS checks:
+
+```python
+import requests
+proxy = requests.get("http://127.0.0.1:8899/get?type=https").json()["url"]   # e.g. socks5://…, type included
+requests.get("https://example.com", proxies={"http": proxy, "https": proxy})
+```
+
 <a id="discord-bot"></a>
 
 ## Discord and Telegram bot
@@ -558,6 +594,15 @@ flowchart LR
 </div>
 </details>
 
+### Your own lists
+
+```bash
+proxy-scraper --source https://example.com/my-list.txt --source socks5=./socks.txt   # on top of the 700+ sources
+proxy-scraper --only-sources --source bought.txt --want 50                            # only yours
+```
+
+Any text with `ip:port` works; lines like `socks5://user:pass@host:port` keep their type, bare ones are tried as HTTP and SOCKS5 unless you write `http=…`.
+
 <a id="output"></a>
 
 ## Output
@@ -875,8 +920,9 @@ Only for things that don't matter. Public proxies are run by strangers who can r
 
 What's next is in the [open issues](../../issues) – ideas and wishes are welcome as an [issue](../../issues/new/choose). During Hacktoberfest there are [beginner-friendly issues](../../issues?q=is%3Aopen+label%3Ahacktoberfest) with pointers on where to start.
 
-- [ ] [Measure throughput, not just latency](../../issues/140)
-- [ ] [Daily snapshots as a dataset](../../issues/143)
+- [ ] [Mirror the daily snapshots to a Hugging Face dataset](../../issues/188)
+
+Shipped in v1.10 to v1.21: download speed per proxy and "best first" everywhere, daily snapshots, a rotating proxy URL for requests, httpx and Playwright, `--pick` for working proxies without a scan, a proxy pool API compatible with jhao104/proxy_pool, your own lists with `--source`, a GitHub Action, a Telegram bot and an Atom feed for the weekly report.
 
 Shipped in v1.8 and v1.9: uptime per proxy and `stable.txt`, which proxies get through to Google, Reddit and Amazon, `live_proxies()` in Python, a page per proxy with its week of checks, a weekly report, live charts in this README, shareable filters on the website, sources mined from other scrapers' lists, untyped lists tried as HTTP and SOCKS5, `--export singbox`, PowerShell completion and recipes for httpx, aiohttp and Scrapy.
 
@@ -924,7 +970,7 @@ proxyscraper/
 ├── targets.py          target sites for --target
 ├── output.py           result files
 ├── exporters.py        proxychains, Clash, sing-box and curl formats (--export)
-├── server/             rotating proxy server (--serve): pool · http · upstream · socks · status · core
+├── server/             rotating proxy server (--serve): pool · http · upstream · socks · status · api · core
 ├── api.py              find_proxies() / check_proxies() for Python
 ├── agent.py            MCP tools without the SDK: live list, filters, fetch through proxies
 ├── mcp_server.py       MCP server (proxy-scraper-mcp) for AI agents
@@ -937,6 +983,20 @@ proxyscraper/
 
 </details>
 
+<a id="community"></a>
+
+## Community
+
+Questions, ideas and things you built with it go to [Discussions](https://github.com/maximilianfeix/proxy-scraper/discussions) – bugs to the [issues](../../issues). If proxy-scraper saves you time, a ⭐ helps others find it.
+
+<a href="https://star-history.com/#maximilianfeix/proxy-scraper&Date">
+  <picture>
+    <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=maximilianfeix/proxy-scraper&type=Date&theme=dark">
+    <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=maximilianfeix/proxy-scraper&type=Date">
+    <img alt="Star history of proxy-scraper" src="https://api.star-history.com/svg?repos=maximilianfeix/proxy-scraper&type=Date" width="600">
+  </picture>
+</a>
+
 <a id="acknowledgements"></a>
 
 ## Acknowledgements

{proxy_scraper_cli-1.20.0 → proxy_scraper_cli-1.22.0}/README.md

@@ -11,6 +11,7 @@
 [![Live proxies](https://img.shields.io/endpoint?url=https%3A%2F%2Fraw.githubusercontent.com%2Fmaximilianfeix%2Fproxy-scraper%2Fproxy-list%2Fbadges%2Ftotal.json&style=flat-square&labelColor=121113)](#live-list)
 [![PyPI](https://img.shields.io/pypi/v/proxy-scraper-cli?style=flat-square&color=D4F77A&labelColor=121113&label=pypi)](https://pypi.org/project/proxy-scraper-cli/)
 [![Python](https://img.shields.io/badge/python-3.9–3.13-D4F77A?style=flat-square&labelColor=121113)](pyproject.toml)
+[![Stars](https://img.shields.io/github/stars/maximilianfeix/proxy-scraper?style=flat-square&color=D4F77A&labelColor=121113)](https://github.com/maximilianfeix/proxy-scraper/stargazers)
 [![License](https://img.shields.io/badge/license-MIT-D4F77A?style=flat-square&labelColor=121113)](LICENSE)
 
 <a href="https://maximilianfeix.github.io/proxy-scraper/"><img src="https://img.shields.io/badge/Browse_the_live_list-D4F77A?style=for-the-badge&labelColor=121113" alt="Browse the live list"></a>
@@ -27,6 +28,8 @@
 
 Most free proxy lists are 95 % dead, and a good part of the rest are honeypots or proxies that inject scripts into your pages. **proxy-scraper** collects public HTTP, SOCKS4 and SOCKS5 proxies from 700+ sources and keeps only the ones that pass every check. It learns with every run which sources are worth it, and it can turn the result into a single rotating proxy.
 
+**In numbers:** in one day it checked 3.6 million free proxies – 1.1 % worked, and more than half of the ones that answered failed a second request. [What's actually out there →](docs/free-proxies-in-numbers.md)
+
 <picture>
   <source media="(prefers-color-scheme: dark)" srcset="docs/checks-dark.svg">
   <source media="(prefers-color-scheme: light)" srcset="docs/checks-light.svg">
@@ -53,7 +56,7 @@ Most free proxy lists are 95 % dead, and a good part of the rest are honeypots o
 - [Options](#options)
 - [GitHub Actions](#github-actions)
 - [FAQ](#faq)
-- [Roadmap](#roadmap) · [Contributing](#contributing) · [Acknowledgements](#acknowledgements)
+- [Roadmap](#roadmap) · [Contributing](#contributing) · [Community](#community) · [Acknowledgements](#acknowledgements)
 
 </details>
 
@@ -106,7 +109,7 @@ proxy-scraper --completion fish > ~/.config/fish/completions/proxy-scraper.fish
 proxy-scraper --completion powershell | Out-String | Invoke-Expression   # in $PROFILE
 ```
 
-Installed, the learned state lives in your user data folder (`~/Library/Application Support/proxy-scraper`, `%LOCALAPPDATA%\proxy-scraper` or `~/.local/share/proxy-scraper`; override with `PROXY_SCRAPER_HOME`) and results go to `./results`. Run from a clone, both stay inside the project.
+Installed, the learned state lives in your user data folder (`~/Library/Application Support/proxy-scraper`, `%LOCALAPPDATA%\proxy-scraper` or `~/.local/share/proxy-scraper`; override with `PROXY_SCRAPER_HOME`) and results go to `./results`. Run from a clone, both stay inside the project. After the second run that finds proxies it asks once for a GitHub star; `PROXY_SCRAPER_NO_STAR_HINT=1` turns that off (it never shows in CI).
 
 </details>
 
@@ -343,6 +346,7 @@ macOS, Linux and Windows, Python 3.9 to 3.13. Only two dependencies: `rich` and
 | Only proxies that reach *your* target site | ❌ | ✅ `--target` |
 | Learns which sources are worth it | ❌ | ✅ |
 | Usable as a single rotating proxy | ❌ | ✅ `--serve` |
+| API to fetch a proxy (proxy_pool compatible) | ❌ | ✅ [`/get`](#pool-api) |
 | Ready-made list without running anything | ✅ | ✅ [live list](#live-list) |
 
 <a id="examples"></a>
@@ -478,6 +482,38 @@ Without a password, `--serve-host` means **anyone who reaches the port can use i
 
 In testing: 20 of 20 HTTPS requests succeeded, over 15 different exit IPs. In the wizard this is **Proxy server right away**.
 
+<a id="pool-api"></a>
+
+### Proxy pool API
+
+Some programs want a proxy *address*, not a proxy – to hand it to a browser, a worker or a queue. The same port answers plain HTTP requests with one:
+
+```bash
+curl http://127.0.0.1:8899/get                          # one proxy as JSON, chosen like a connection would be
+curl "http://127.0.0.1:8899/get?country=DE&https=1&format=txt"   # → http://203.0.113.7:8080
+curl "http://127.0.0.1:8899/all?protocol=socks5&limit=20"        # the 20 best SOCKS5 proxies
+curl "http://127.0.0.1:8899/report?proxy=203.0.113.7:8080&ok=0"  # it failed you – three times and it's out
+```
+
+| Endpoint | What it does |
+|---|---|
+| `/get` | one proxy, picked by the `--rotate` strategy |
+| `/pop` | like `/get`, and the proxy leaves the pool |
+| `/all` | every usable proxy, best first (`limit=N`) |
+| `/count` | totals by type and country |
+| `/delete?proxy=IP:PORT` | take a proxy out |
+| `/report?proxy=IP:PORT&ok=0` | feedback from your own requests |
+
+Filters work on `/get`, `/pop` and `/all`: `country=DE,AT`, `protocol=socks5`, `https=1`, `anonymity=elite`, `max_latency=1500`, and `format=txt` for plain URLs. With `--serve-password` the API wants it as Basic auth, like the status page.
+
+**Coming from [jhao104/proxy_pool](https://github.com/jhao104/proxy_pool)?** The endpoints, `type=https` and the JSON fields (`proxy`, `https`, `region`, `anonymous`, `check_count`, `fail_count`, …) are the same, so point your code at port 8899 and it keeps working – without Redis, and with proxies that passed the honeypot, tampering and TLS checks:
+
+```python
+import requests
+proxy = requests.get("http://127.0.0.1:8899/get?type=https").json()["url"]   # e.g. socks5://…, type included
+requests.get("https://example.com", proxies={"http": proxy, "https": proxy})
+```
+
 <a id="discord-bot"></a>
 
 ## Discord and Telegram bot
@@ -517,6 +553,15 @@ flowchart LR
 </div>
 </details>
 
+### Your own lists
+
+```bash
+proxy-scraper --source https://example.com/my-list.txt --source socks5=./socks.txt   # on top of the 700+ sources
+proxy-scraper --only-sources --source bought.txt --want 50                            # only yours
+```
+
+Any text with `ip:port` works; lines like `socks5://user:pass@host:port` keep their type, bare ones are tried as HTTP and SOCKS5 unless you write `http=…`.
+
 <a id="output"></a>
 
 ## Output
@@ -834,8 +879,9 @@ Only for things that don't matter. Public proxies are run by strangers who can r
 
 What's next is in the [open issues](../../issues) – ideas and wishes are welcome as an [issue](../../issues/new/choose). During Hacktoberfest there are [beginner-friendly issues](../../issues?q=is%3Aopen+label%3Ahacktoberfest) with pointers on where to start.
 
-- [ ] [Measure throughput, not just latency](../../issues/140)
-- [ ] [Daily snapshots as a dataset](../../issues/143)
+- [ ] [Mirror the daily snapshots to a Hugging Face dataset](../../issues/188)
+
+Shipped in v1.10 to v1.21: download speed per proxy and "best first" everywhere, daily snapshots, a rotating proxy URL for requests, httpx and Playwright, `--pick` for working proxies without a scan, a proxy pool API compatible with jhao104/proxy_pool, your own lists with `--source`, a GitHub Action, a Telegram bot and an Atom feed for the weekly report.
 
 Shipped in v1.8 and v1.9: uptime per proxy and `stable.txt`, which proxies get through to Google, Reddit and Amazon, `live_proxies()` in Python, a page per proxy with its week of checks, a weekly report, live charts in this README, shareable filters on the website, sources mined from other scrapers' lists, untyped lists tried as HTTP and SOCKS5, `--export singbox`, PowerShell completion and recipes for httpx, aiohttp and Scrapy.
 
@@ -883,7 +929,7 @@ proxyscraper/
 ├── targets.py          target sites for --target
 ├── output.py           result files
 ├── exporters.py        proxychains, Clash, sing-box and curl formats (--export)
-├── server/             rotating proxy server (--serve): pool · http · upstream · socks · status · core
+├── server/             rotating proxy server (--serve): pool · http · upstream · socks · status · api · core
 ├── api.py              find_proxies() / check_proxies() for Python
 ├── agent.py            MCP tools without the SDK: live list, filters, fetch through proxies
 ├── mcp_server.py       MCP server (proxy-scraper-mcp) for AI agents
@@ -896,6 +942,20 @@ proxyscraper/
 
 </details>
 
+<a id="community"></a>
+
+## Community
+
+Questions, ideas and things you built with it go to [Discussions](https://github.com/maximilianfeix/proxy-scraper/discussions) – bugs to the [issues](../../issues). If proxy-scraper saves you time, a ⭐ helps others find it.
+
+<a href="https://star-history.com/#maximilianfeix/proxy-scraper&Date">
+  <picture>
+    <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=maximilianfeix/proxy-scraper&type=Date&theme=dark">
+    <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=maximilianfeix/proxy-scraper&type=Date">
+    <img alt="Star history of proxy-scraper" src="https://api.star-history.com/svg?repos=maximilianfeix/proxy-scraper&type=Date" width="600">
+  </picture>
+</a>
+
 <a id="acknowledgements"></a>
 
 ## Acknowledgements

{proxy_scraper_cli-1.20.0 → proxy_scraper_cli-1.22.0}/proxy_scraper_cli.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: proxy-scraper-cli
-Version: 1.20.0
+Version: 1.22.0
 Summary: Finds free HTTP, SOCKS4 and SOCKS5 proxies that actually work: 700+ sources, real checks, honeypot filtering and a rotating local proxy server.
 Author: Maximilian Feix
 License-Expression: MIT
@@ -52,6 +52,7 @@ Dynamic: license-file
 [![Live proxies](https://img.shields.io/endpoint?url=https%3A%2F%2Fraw.githubusercontent.com%2Fmaximilianfeix%2Fproxy-scraper%2Fproxy-list%2Fbadges%2Ftotal.json&style=flat-square&labelColor=121113)](#live-list)
 [![PyPI](https://img.shields.io/pypi/v/proxy-scraper-cli?style=flat-square&color=D4F77A&labelColor=121113&label=pypi)](https://pypi.org/project/proxy-scraper-cli/)
 [![Python](https://img.shields.io/badge/python-3.9–3.13-D4F77A?style=flat-square&labelColor=121113)](pyproject.toml)
+[![Stars](https://img.shields.io/github/stars/maximilianfeix/proxy-scraper?style=flat-square&color=D4F77A&labelColor=121113)](https://github.com/maximilianfeix/proxy-scraper/stargazers)
 [![License](https://img.shields.io/badge/license-MIT-D4F77A?style=flat-square&labelColor=121113)](LICENSE)
 
 <a href="https://maximilianfeix.github.io/proxy-scraper/"><img src="https://img.shields.io/badge/Browse_the_live_list-D4F77A?style=for-the-badge&labelColor=121113" alt="Browse the live list"></a>
@@ -68,6 +69,8 @@ Dynamic: license-file
 
 Most free proxy lists are 95 % dead, and a good part of the rest are honeypots or proxies that inject scripts into your pages. **proxy-scraper** collects public HTTP, SOCKS4 and SOCKS5 proxies from 700+ sources and keeps only the ones that pass every check. It learns with every run which sources are worth it, and it can turn the result into a single rotating proxy.
 
+**In numbers:** in one day it checked 3.6 million free proxies – 1.1 % worked, and more than half of the ones that answered failed a second request. [What's actually out there →](docs/free-proxies-in-numbers.md)
+
 <picture>
   <source media="(prefers-color-scheme: dark)" srcset="docs/checks-dark.svg">
   <source media="(prefers-color-scheme: light)" srcset="docs/checks-light.svg">
@@ -94,7 +97,7 @@ Most free proxy lists are 95 % dead, and a good part of the rest are honeypots o
 - [Options](#options)
 - [GitHub Actions](#github-actions)
 - [FAQ](#faq)
-- [Roadmap](#roadmap) · [Contributing](#contributing) · [Acknowledgements](#acknowledgements)
+- [Roadmap](#roadmap) · [Contributing](#contributing) · [Community](#community) · [Acknowledgements](#acknowledgements)
 
 </details>
 
@@ -147,7 +150,7 @@ proxy-scraper --completion fish > ~/.config/fish/completions/proxy-scraper.fish
 proxy-scraper --completion powershell | Out-String | Invoke-Expression   # in $PROFILE
 ```
 
-Installed, the learned state lives in your user data folder (`~/Library/Application Support/proxy-scraper`, `%LOCALAPPDATA%\proxy-scraper` or `~/.local/share/proxy-scraper`; override with `PROXY_SCRAPER_HOME`) and results go to `./results`. Run from a clone, both stay inside the project.
+Installed, the learned state lives in your user data folder (`~/Library/Application Support/proxy-scraper`, `%LOCALAPPDATA%\proxy-scraper` or `~/.local/share/proxy-scraper`; override with `PROXY_SCRAPER_HOME`) and results go to `./results`. Run from a clone, both stay inside the project. After the second run that finds proxies it asks once for a GitHub star; `PROXY_SCRAPER_NO_STAR_HINT=1` turns that off (it never shows in CI).
 
 </details>
 
@@ -384,6 +387,7 @@ macOS, Linux and Windows, Python 3.9 to 3.13. Only two dependencies: `rich` and
 | Only proxies that reach *your* target site | ❌ | ✅ `--target` |
 | Learns which sources are worth it | ❌ | ✅ |
 | Usable as a single rotating proxy | ❌ | ✅ `--serve` |
+| API to fetch a proxy (proxy_pool compatible) | ❌ | ✅ [`/get`](#pool-api) |
 | Ready-made list without running anything | ✅ | ✅ [live list](#live-list) |
 
 <a id="examples"></a>
@@ -519,6 +523,38 @@ Without a password, `--serve-host` means **anyone who reaches the port can use i
 
 In testing: 20 of 20 HTTPS requests succeeded, over 15 different exit IPs. In the wizard this is **Proxy server right away**.
 
+<a id="pool-api"></a>
+
+### Proxy pool API
+
+Some programs want a proxy *address*, not a proxy – to hand it to a browser, a worker or a queue. The same port answers plain HTTP requests with one:
+
+```bash
+curl http://127.0.0.1:8899/get                          # one proxy as JSON, chosen like a connection would be
+curl "http://127.0.0.1:8899/get?country=DE&https=1&format=txt"   # → http://203.0.113.7:8080
+curl "http://127.0.0.1:8899/all?protocol=socks5&limit=20"        # the 20 best SOCKS5 proxies
+curl "http://127.0.0.1:8899/report?proxy=203.0.113.7:8080&ok=0"  # it failed you – three times and it's out
+```
+
+| Endpoint | What it does |
+|---|---|
+| `/get` | one proxy, picked by the `--rotate` strategy |
+| `/pop` | like `/get`, and the proxy leaves the pool |
+| `/all` | every usable proxy, best first (`limit=N`) |
+| `/count` | totals by type and country |
+| `/delete?proxy=IP:PORT` | take a proxy out |
+| `/report?proxy=IP:PORT&ok=0` | feedback from your own requests |
+
+Filters work on `/get`, `/pop` and `/all`: `country=DE,AT`, `protocol=socks5`, `https=1`, `anonymity=elite`, `max_latency=1500`, and `format=txt` for plain URLs. With `--serve-password` the API wants it as Basic auth, like the status page.
+
+**Coming from [jhao104/proxy_pool](https://github.com/jhao104/proxy_pool)?** The endpoints, `type=https` and the JSON fields (`proxy`, `https`, `region`, `anonymous`, `check_count`, `fail_count`, …) are the same, so point your code at port 8899 and it keeps working – without Redis, and with proxies that passed the honeypot, tampering and TLS checks:
+
+```python
+import requests
+proxy = requests.get("http://127.0.0.1:8899/get?type=https").json()["url"]   # e.g. socks5://…, type included
+requests.get("https://example.com", proxies={"http": proxy, "https": proxy})
+```
+
 <a id="discord-bot"></a>
 
 ## Discord and Telegram bot
@@ -558,6 +594,15 @@ flowchart LR
 </div>
 </details>
 
+### Your own lists
+
+```bash
+proxy-scraper --source https://example.com/my-list.txt --source socks5=./socks.txt   # on top of the 700+ sources
+proxy-scraper --only-sources --source bought.txt --want 50                            # only yours
+```
+
+Any text with `ip:port` works; lines like `socks5://user:pass@host:port` keep their type, bare ones are tried as HTTP and SOCKS5 unless you write `http=…`.
+
 <a id="output"></a>
 
 ## Output
@@ -875,8 +920,9 @@ Only for things that don't matter. Public proxies are run by strangers who can r
 
 What's next is in the [open issues](../../issues) – ideas and wishes are welcome as an [issue](../../issues/new/choose). During Hacktoberfest there are [beginner-friendly issues](../../issues?q=is%3Aopen+label%3Ahacktoberfest) with pointers on where to start.
 
-- [ ] [Measure throughput, not just latency](../../issues/140)
-- [ ] [Daily snapshots as a dataset](../../issues/143)
+- [ ] [Mirror the daily snapshots to a Hugging Face dataset](../../issues/188)
+
+Shipped in v1.10 to v1.21: download speed per proxy and "best first" everywhere, daily snapshots, a rotating proxy URL for requests, httpx and Playwright, `--pick` for working proxies without a scan, a proxy pool API compatible with jhao104/proxy_pool, your own lists with `--source`, a GitHub Action, a Telegram bot and an Atom feed for the weekly report.
 
 Shipped in v1.8 and v1.9: uptime per proxy and `stable.txt`, which proxies get through to Google, Reddit and Amazon, `live_proxies()` in Python, a page per proxy with its week of checks, a weekly report, live charts in this README, shareable filters on the website, sources mined from other scrapers' lists, untyped lists tried as HTTP and SOCKS5, `--export singbox`, PowerShell completion and recipes for httpx, aiohttp and Scrapy.
 
@@ -924,7 +970,7 @@ proxyscraper/
 ├── targets.py          target sites for --target
 ├── output.py           result files
 ├── exporters.py        proxychains, Clash, sing-box and curl formats (--export)
-├── server/             rotating proxy server (--serve): pool · http · upstream · socks · status · core
+├── server/             rotating proxy server (--serve): pool · http · upstream · socks · status · api · core
 ├── api.py              find_proxies() / check_proxies() for Python
 ├── agent.py            MCP tools without the SDK: live list, filters, fetch through proxies
 ├── mcp_server.py       MCP server (proxy-scraper-mcp) for AI agents
@@ -937,6 +983,20 @@ proxyscraper/
 
 </details>
 
+<a id="community"></a>
+
+## Community
+
+Questions, ideas and things you built with it go to [Discussions](https://github.com/maximilianfeix/proxy-scraper/discussions) – bugs to the [issues](../../issues). If proxy-scraper saves you time, a ⭐ helps others find it.
+
+<a href="https://star-history.com/#maximilianfeix/proxy-scraper&Date">
+  <picture>
+    <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=maximilianfeix/proxy-scraper&type=Date&theme=dark">
+    <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=maximilianfeix/proxy-scraper&type=Date">
+    <img alt="Star history of proxy-scraper" src="https://api.star-history.com/svg?repos=maximilianfeix/proxy-scraper&type=Date" width="600">
+  </picture>
+</a>
+
 <a id="acknowledgements"></a>
 
 ## Acknowledgements

{proxy_scraper_cli-1.20.0 → proxy_scraper_cli-1.22.0}/proxy_scraper_cli.egg-info/SOURCES.txt

@@ -48,6 +48,7 @@ proxyscraper/sources.py
 proxyscraper/speed.py
 proxyscraper/targets.py
 proxyscraper/server/__init__.py
+proxyscraper/server/api.py
 proxyscraper/server/core.py
 proxyscraper/server/http.py
 proxyscraper/server/pool.py
@@ -88,6 +89,7 @@ tests/test_mcp_server.py
 tests/test_mine_sources.py
 tests/test_options.py
 tests/test_output_stdout.py
+tests/test_own_sources.py
 tests/test_packaging.py
 tests/test_pages.py
 tests/test_parsing.py
@@ -95,12 +97,14 @@ tests/test_parsing_fuzz.py
 tests/test_paths.py
 tests/test_pick.py
 tests/test_pipeline.py
+tests/test_preferences.py
 tests/test_proxy_pages.py
 tests/test_proxy_sources.py
 tests/test_publish.py
 tests/test_report.py
 tests/test_robustness.py
 tests/test_rotator.py
+tests/test_serve_api.py
 tests/test_serve_password.py
 tests/test_serve_refill.py
 tests/test_serve_v2.py

{proxy_scraper_cli-1.20.0 → proxy_scraper_cli-1.22.0}/proxyscraper/__init__.py

@@ -1,7 +1,7 @@
 """proxy-scraper: collects public HTTP/SOCKS4/SOCKS5 proxies from hundreds of sources, checks them
 in parallel with its own protocol handshakes and learns which sources deliver good proxies."""
 
-__version__ = "1.20.0"
+__version__ = "1.22.0"
 
 
 def __getattr__(name):

{proxy_scraper_cli-1.20.0 → proxy_scraper_cli-1.22.0}/proxyscraper/app.py

@@ -41,6 +41,7 @@ from .judges import JudgeProbe, JudgeWatch, rank_judges
 from .netio import INSECURE_HOSTS, http_get
 from .options import STDOUT, RunOptions
 from .output import ResultWriter, latest_results
+from .pages import REPO_URL
 from .parsing import PROXY_TYPES, parse_keys, split_key
 from .paths import is_checkout
 from .pipeline import (
@@ -53,6 +54,7 @@ from .pipeline import (
     run_checks,
     scrape,
 )
+from .preferences import star_hint_due
 from .publish import RAW_BASE
 from .server import ProxyPool, RotatingServer
 from .targets import Target, parse_target
@@ -327,8 +329,10 @@ class Run:
         plan = await collect_sources(self.opts, self.quality)
         info("Sources", Text.assemble(
             (fmt(len(plan.sources)), f"bold {ACCENT}"), " active  ",
-            (f"({fmt(plan.n_curated)} curated · {fmt(plan.n_meta)} from {plan.meta_ok}/{plan.meta_total} "
-             f"meta lists · {fmt(plan.n_discovered)} discovered)", MUTED),
+            (f"({fmt(plan.n_own)} own)" if self.opts.only_sources else
+             f"({fmt(plan.n_curated)} curated · {fmt(plan.n_meta)} from {plan.meta_ok}/{plan.meta_total} "
+             f"meta lists · {fmt(plan.n_discovered)} discovered"
+             + (f" · {fmt(plan.n_own)} own)" if plan.n_own else ")"), MUTED),
         ))
         if plan.discovery_ran and not plan.discovery_token:
             note("GitHub discovery is limited without a token – run `gh auth login` or set GITHUB_TOKEN.",
@@ -355,7 +359,7 @@ class Run:
             hosts = sorted(INSECURE_HOSTS)
             note(f"Certificate could not be verified (TLS inspection on this network?), loaded anyway: "
                  f"{', '.join(hosts[:4])}{' …' if len(hosts) > 4 else ''}", MUTED, "ℹ")
-        return prioritize(res, self.quality, self.history, self.opts.types)
+        return prioritize(res, self.quality, self.history, self.opts.types, listed_only=self.opts.only_sources)
 
     # ------------------------------------------------------------------ phase 3+4: check, learn, report
 
@@ -589,6 +593,9 @@ class Run:
             note(f"{fmt(stats.blocklisted)} of {fmt(len(run.results))} hits ({share}) "
                  "exit from an IP on the SpamCop blocklist – sites that use it show captchas or block them. "
                  "Only the others: --no-blocklisted", MUTED, "ℹ")
+        if kept and opts.output != STDOUT and widgets.console.is_terminal and star_hint_due():
+            note("Found what you needed? A star on GitHub is how others find this: "
+                 f"[link={REPO_URL}]{REPO_URL}[/link] [{MUTED}](shown once)[/]", ACCENT, "★")
         if run.reached_goal:
             note(f"Goal of {fmt(opts.want)} hits reached – stopped early.", GOOD, "✔")
         elif run.interrupted:

{proxy_scraper_cli-1.20.0 → proxy_scraper_cli-1.22.0}/proxyscraper/cli.py

@@ -246,6 +246,12 @@ def parse_args(argv: Optional[List[str]] = None) -> argparse.Namespace:
     s.add_argument("--discover-repos", type=non_negative_int, default=DEFAULT_DISCOVER_REPOS,
                    help=f"max. repos during discovery (default: {DEFAULT_DISCOVER_REPOS}, 40 without a token)")
     s.add_argument("--all-sources", action="store_true", help="also load dead, outdated and unreachable sources")
+    s.add_argument("--source", action="append", metavar="URL|FILE",
+                   help="also load this proxy list (repeatable): a URL or a local file, one proxy per line or "
+                        "any text with ip:port in it; TYPE=… fixes the type, e.g. socks5=https://…/list.txt – "
+                        "without it, lines with a scheme keep theirs and bare ones are tried as HTTP and SOCKS5")
+    s.add_argument("--only-sources", action="store_true",
+                   help="only the lists given with --source – no built-in, meta or discovered sources")
     s.add_argument("--no-cache", action="store_true",
                    help="reload every list completely (otherwise unchanged ones are skipped via ETag)")
     s.add_argument("--list-sources", nargs="?", const=50, type=positive_int, metavar="N",

{proxy_scraper_cli-1.20.0 → proxy_scraper_cli-1.22.0}/proxyscraper/options.py

@@ -115,6 +115,8 @@ class RunOptions:
     no_discover: bool = False
     discover_repos: int = DEFAULT_DISCOVER_REPOS
     all_sources: bool = False
+    sources: List[str] = field(default_factory=list)  # own lists (--source): URLs, files, 'socks5=…'
+    only_sources: bool = False  # only the own lists – no curated, meta or discovered ones
     no_cache: bool = False  # reload every list instead of taking unchanged ones from the cache
     no_dnsbl: bool = False  # skip the blocklist lookup of the exit IPs
     serve: int = 0  # port of the rotating proxy server after the run, 0 = off
@@ -149,6 +151,8 @@ class RunOptions:
             raise ValueError("--sticky must not be negative")
         if self.serve_refill < 0:
             raise ValueError("--serve-refill must not be negative")
+        if self.only_sources and not self.sources:
+            raise ValueError("--only-sources needs at least one --source")
         if not 0 <= self.serve <= 65535:
             raise ValueError("port must be between 1 and 65535")
 
@@ -200,6 +204,8 @@ class RunOptions:
             no_discover=args.no_discover,
             discover_repos=args.discover_repos,
             all_sources=args.all_sources,
+            sources=list(dict.fromkeys(args.source or [])),
+            only_sources=args.only_sources,
             no_cache=args.no_cache,
             no_dnsbl=args.no_dnsbl,
             serve=args.serve,
@@ -245,6 +251,9 @@ class RunOptions:
         _flag(argv, "--no-discover", self.no_discover)
         _opt(argv, "--discover-repos", self.discover_repos, DEFAULT_DISCOVER_REPOS)
         _flag(argv, "--all-sources", self.all_sources)
+        for spec in self.sources:
+            argv += ["--source", spec]
+        _flag(argv, "--only-sources", self.only_sources)
         _flag(argv, "--no-cache", self.no_cache)
         _flag(argv, "--no-dnsbl", self.no_dnsbl)
         if self.serve: