proxy-scraper-cli 1.20.0__tar.gz → 1.21.0__tar.gz

{proxy_scraper_cli-1.20.0 → proxy_scraper_cli-1.21.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: proxy-scraper-cli
-Version: 1.20.0
+Version: 1.21.0
 Summary: Finds free HTTP, SOCKS4 and SOCKS5 proxies that actually work: 700+ sources, real checks, honeypot filtering and a rotating local proxy server.
 Author: Maximilian Feix
 License-Expression: MIT
@@ -52,6 +52,7 @@ Dynamic: license-file
 [![Live proxies](https://img.shields.io/endpoint?url=https%3A%2F%2Fraw.githubusercontent.com%2Fmaximilianfeix%2Fproxy-scraper%2Fproxy-list%2Fbadges%2Ftotal.json&style=flat-square&labelColor=121113)](#live-list)
 [![PyPI](https://img.shields.io/pypi/v/proxy-scraper-cli?style=flat-square&color=D4F77A&labelColor=121113&label=pypi)](https://pypi.org/project/proxy-scraper-cli/)
 [![Python](https://img.shields.io/badge/python-3.9–3.13-D4F77A?style=flat-square&labelColor=121113)](pyproject.toml)
+[![Stars](https://img.shields.io/github/stars/maximilianfeix/proxy-scraper?style=flat-square&color=D4F77A&labelColor=121113)](https://github.com/maximilianfeix/proxy-scraper/stargazers)
 [![License](https://img.shields.io/badge/license-MIT-D4F77A?style=flat-square&labelColor=121113)](LICENSE)
 
 <a href="https://maximilianfeix.github.io/proxy-scraper/"><img src="https://img.shields.io/badge/Browse_the_live_list-D4F77A?style=for-the-badge&labelColor=121113" alt="Browse the live list"></a>
@@ -94,7 +95,7 @@ Most free proxy lists are 95 % dead, and a good part of the rest are honeypots o
 - [Options](#options)
 - [GitHub Actions](#github-actions)
 - [FAQ](#faq)
-- [Roadmap](#roadmap) · [Contributing](#contributing) · [Acknowledgements](#acknowledgements)
+- [Roadmap](#roadmap) · [Contributing](#contributing) · [Community](#community) · [Acknowledgements](#acknowledgements)
 
 </details>
 
@@ -384,6 +385,7 @@ macOS, Linux and Windows, Python 3.9 to 3.13. Only two dependencies: `rich` and
 | Only proxies that reach *your* target site | ❌ | ✅ `--target` |
 | Learns which sources are worth it | ❌ | ✅ |
 | Usable as a single rotating proxy | ❌ | ✅ `--serve` |
+| API to fetch a proxy (proxy_pool compatible) | ❌ | ✅ [`/get`](#pool-api) |
 | Ready-made list without running anything | ✅ | ✅ [live list](#live-list) |
 
 <a id="examples"></a>
@@ -519,6 +521,38 @@ Without a password, `--serve-host` means **anyone who reaches the port can use i
 
 In testing: 20 of 20 HTTPS requests succeeded, over 15 different exit IPs. In the wizard this is **Proxy server right away**.
 
+<a id="pool-api"></a>
+
+### Proxy pool API
+
+Some programs want a proxy *address*, not a proxy – to hand it to a browser, a worker or a queue. The same port answers plain HTTP requests with one:
+
+```bash
+curl http://127.0.0.1:8899/get                          # one proxy as JSON, chosen like a connection would be
+curl "http://127.0.0.1:8899/get?country=DE&https=1&format=txt"   # → http://203.0.113.7:8080
+curl "http://127.0.0.1:8899/all?protocol=socks5&limit=20"        # the 20 best SOCKS5 proxies
+curl "http://127.0.0.1:8899/report?proxy=203.0.113.7:8080&ok=0"  # it failed you – three times and it's out
+```
+
+| Endpoint | What it does |
+|---|---|
+| `/get` | one proxy, picked by the `--rotate` strategy |
+| `/pop` | like `/get`, and the proxy leaves the pool |
+| `/all` | every usable proxy, best first (`limit=N`) |
+| `/count` | totals by type and country |
+| `/delete?proxy=IP:PORT` | take a proxy out |
+| `/report?proxy=IP:PORT&ok=0` | feedback from your own requests |
+
+Filters work on `/get`, `/pop` and `/all`: `country=DE,AT`, `protocol=socks5`, `https=1`, `anonymity=elite`, `max_latency=1500`, and `format=txt` for plain URLs. With `--serve-password` the API wants it as Basic auth, like the status page.
+
+**Coming from [jhao104/proxy_pool](https://github.com/jhao104/proxy_pool)?** The endpoints, `type=https` and the JSON fields (`proxy`, `https`, `region`, `anonymous`, `check_count`, `fail_count`, …) are the same, so point your code at port 8899 and it keeps working – without Redis, and with proxies that passed the honeypot, tampering and TLS checks:
+
+```python
+import requests
+proxy = requests.get("http://127.0.0.1:8899/get?type=https").json()["url"]   # e.g. socks5://…, type included
+requests.get("https://example.com", proxies={"http": proxy, "https": proxy})
+```
+
 <a id="discord-bot"></a>
 
 ## Discord and Telegram bot
@@ -558,6 +592,15 @@ flowchart LR
 </div>
 </details>
 
+### Your own lists
+
+```bash
+proxy-scraper --source https://example.com/my-list.txt --source socks5=./socks.txt   # on top of the 700+ sources
+proxy-scraper --only-sources --source bought.txt --want 50                            # only yours
+```
+
+Any text with `ip:port` works; lines like `socks5://user:pass@host:port` keep their type, bare ones are tried as HTTP and SOCKS5 unless you write `http=…`.
+
 <a id="output"></a>
 
 ## Output
@@ -924,7 +967,7 @@ proxyscraper/
 ├── targets.py          target sites for --target
 ├── output.py           result files
 ├── exporters.py        proxychains, Clash, sing-box and curl formats (--export)
-├── server/             rotating proxy server (--serve): pool · http · upstream · socks · status · core
+├── server/             rotating proxy server (--serve): pool · http · upstream · socks · status · api · core
 ├── api.py              find_proxies() / check_proxies() for Python
 ├── agent.py            MCP tools without the SDK: live list, filters, fetch through proxies
 ├── mcp_server.py       MCP server (proxy-scraper-mcp) for AI agents
@@ -937,6 +980,20 @@ proxyscraper/
 
 </details>
 
+<a id="community"></a>
+
+## Community
+
+Questions, ideas and things you built with it go to [Discussions](https://github.com/maximilianfeix/proxy-scraper/discussions) – bugs to the [issues](../../issues). If proxy-scraper saves you time, a ⭐ helps others find it.
+
+<a href="https://star-history.com/#maximilianfeix/proxy-scraper&Date">
+  <picture>
+    <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=maximilianfeix/proxy-scraper&type=Date&theme=dark">
+    <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=maximilianfeix/proxy-scraper&type=Date">
+    <img alt="Star history of proxy-scraper" src="https://api.star-history.com/svg?repos=maximilianfeix/proxy-scraper&type=Date" width="600">
+  </picture>
+</a>
+
 <a id="acknowledgements"></a>
 
 ## Acknowledgements

{proxy_scraper_cli-1.20.0 → proxy_scraper_cli-1.21.0}/README.md

@@ -11,6 +11,7 @@
 [![Live proxies](https://img.shields.io/endpoint?url=https%3A%2F%2Fraw.githubusercontent.com%2Fmaximilianfeix%2Fproxy-scraper%2Fproxy-list%2Fbadges%2Ftotal.json&style=flat-square&labelColor=121113)](#live-list)
 [![PyPI](https://img.shields.io/pypi/v/proxy-scraper-cli?style=flat-square&color=D4F77A&labelColor=121113&label=pypi)](https://pypi.org/project/proxy-scraper-cli/)
 [![Python](https://img.shields.io/badge/python-3.9–3.13-D4F77A?style=flat-square&labelColor=121113)](pyproject.toml)
+[![Stars](https://img.shields.io/github/stars/maximilianfeix/proxy-scraper?style=flat-square&color=D4F77A&labelColor=121113)](https://github.com/maximilianfeix/proxy-scraper/stargazers)
 [![License](https://img.shields.io/badge/license-MIT-D4F77A?style=flat-square&labelColor=121113)](LICENSE)
 
 <a href="https://maximilianfeix.github.io/proxy-scraper/"><img src="https://img.shields.io/badge/Browse_the_live_list-D4F77A?style=for-the-badge&labelColor=121113" alt="Browse the live list"></a>
@@ -53,7 +54,7 @@ Most free proxy lists are 95 % dead, and a good part of the rest are honeypots o
 - [Options](#options)
 - [GitHub Actions](#github-actions)
 - [FAQ](#faq)
-- [Roadmap](#roadmap) · [Contributing](#contributing) · [Acknowledgements](#acknowledgements)
+- [Roadmap](#roadmap) · [Contributing](#contributing) · [Community](#community) · [Acknowledgements](#acknowledgements)
 
 </details>
 
@@ -343,6 +344,7 @@ macOS, Linux and Windows, Python 3.9 to 3.13. Only two dependencies: `rich` and
 | Only proxies that reach *your* target site | ❌ | ✅ `--target` |
 | Learns which sources are worth it | ❌ | ✅ |
 | Usable as a single rotating proxy | ❌ | ✅ `--serve` |
+| API to fetch a proxy (proxy_pool compatible) | ❌ | ✅ [`/get`](#pool-api) |
 | Ready-made list without running anything | ✅ | ✅ [live list](#live-list) |
 
 <a id="examples"></a>
@@ -478,6 +480,38 @@ Without a password, `--serve-host` means **anyone who reaches the port can use i
 
 In testing: 20 of 20 HTTPS requests succeeded, over 15 different exit IPs. In the wizard this is **Proxy server right away**.
 
+<a id="pool-api"></a>
+
+### Proxy pool API
+
+Some programs want a proxy *address*, not a proxy – to hand it to a browser, a worker or a queue. The same port answers plain HTTP requests with one:
+
+```bash
+curl http://127.0.0.1:8899/get                          # one proxy as JSON, chosen like a connection would be
+curl "http://127.0.0.1:8899/get?country=DE&https=1&format=txt"   # → http://203.0.113.7:8080
+curl "http://127.0.0.1:8899/all?protocol=socks5&limit=20"        # the 20 best SOCKS5 proxies
+curl "http://127.0.0.1:8899/report?proxy=203.0.113.7:8080&ok=0"  # it failed you – three times and it's out
+```
+
+| Endpoint | What it does |
+|---|---|
+| `/get` | one proxy, picked by the `--rotate` strategy |
+| `/pop` | like `/get`, and the proxy leaves the pool |
+| `/all` | every usable proxy, best first (`limit=N`) |
+| `/count` | totals by type and country |
+| `/delete?proxy=IP:PORT` | take a proxy out |
+| `/report?proxy=IP:PORT&ok=0` | feedback from your own requests |
+
+Filters work on `/get`, `/pop` and `/all`: `country=DE,AT`, `protocol=socks5`, `https=1`, `anonymity=elite`, `max_latency=1500`, and `format=txt` for plain URLs. With `--serve-password` the API wants it as Basic auth, like the status page.
+
+**Coming from [jhao104/proxy_pool](https://github.com/jhao104/proxy_pool)?** The endpoints, `type=https` and the JSON fields (`proxy`, `https`, `region`, `anonymous`, `check_count`, `fail_count`, …) are the same, so point your code at port 8899 and it keeps working – without Redis, and with proxies that passed the honeypot, tampering and TLS checks:
+
+```python
+import requests
+proxy = requests.get("http://127.0.0.1:8899/get?type=https").json()["url"]   # e.g. socks5://…, type included
+requests.get("https://example.com", proxies={"http": proxy, "https": proxy})
+```
+
 <a id="discord-bot"></a>
 
 ## Discord and Telegram bot
@@ -517,6 +551,15 @@ flowchart LR
 </div>
 </details>
 
+### Your own lists
+
+```bash
+proxy-scraper --source https://example.com/my-list.txt --source socks5=./socks.txt   # on top of the 700+ sources
+proxy-scraper --only-sources --source bought.txt --want 50                            # only yours
+```
+
+Any text with `ip:port` works; lines like `socks5://user:pass@host:port` keep their type, bare ones are tried as HTTP and SOCKS5 unless you write `http=…`.
+
 <a id="output"></a>
 
 ## Output
@@ -883,7 +926,7 @@ proxyscraper/
 ├── targets.py          target sites for --target
 ├── output.py           result files
 ├── exporters.py        proxychains, Clash, sing-box and curl formats (--export)
-├── server/             rotating proxy server (--serve): pool · http · upstream · socks · status · core
+├── server/             rotating proxy server (--serve): pool · http · upstream · socks · status · api · core
 ├── api.py              find_proxies() / check_proxies() for Python
 ├── agent.py            MCP tools without the SDK: live list, filters, fetch through proxies
 ├── mcp_server.py       MCP server (proxy-scraper-mcp) for AI agents
@@ -896,6 +939,20 @@ proxyscraper/
 
 </details>
 
+<a id="community"></a>
+
+## Community
+
+Questions, ideas and things you built with it go to [Discussions](https://github.com/maximilianfeix/proxy-scraper/discussions) – bugs to the [issues](../../issues). If proxy-scraper saves you time, a ⭐ helps others find it.
+
+<a href="https://star-history.com/#maximilianfeix/proxy-scraper&Date">
+  <picture>
+    <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=maximilianfeix/proxy-scraper&type=Date&theme=dark">
+    <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=maximilianfeix/proxy-scraper&type=Date">
+    <img alt="Star history of proxy-scraper" src="https://api.star-history.com/svg?repos=maximilianfeix/proxy-scraper&type=Date" width="600">
+  </picture>
+</a>
+
 <a id="acknowledgements"></a>
 
 ## Acknowledgements

{proxy_scraper_cli-1.20.0 → proxy_scraper_cli-1.21.0}/proxy_scraper_cli.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: proxy-scraper-cli
-Version: 1.20.0
+Version: 1.21.0
 Summary: Finds free HTTP, SOCKS4 and SOCKS5 proxies that actually work: 700+ sources, real checks, honeypot filtering and a rotating local proxy server.
 Author: Maximilian Feix
 License-Expression: MIT
@@ -52,6 +52,7 @@ Dynamic: license-file
 [![Live proxies](https://img.shields.io/endpoint?url=https%3A%2F%2Fraw.githubusercontent.com%2Fmaximilianfeix%2Fproxy-scraper%2Fproxy-list%2Fbadges%2Ftotal.json&style=flat-square&labelColor=121113)](#live-list)
 [![PyPI](https://img.shields.io/pypi/v/proxy-scraper-cli?style=flat-square&color=D4F77A&labelColor=121113&label=pypi)](https://pypi.org/project/proxy-scraper-cli/)
 [![Python](https://img.shields.io/badge/python-3.9–3.13-D4F77A?style=flat-square&labelColor=121113)](pyproject.toml)
+[![Stars](https://img.shields.io/github/stars/maximilianfeix/proxy-scraper?style=flat-square&color=D4F77A&labelColor=121113)](https://github.com/maximilianfeix/proxy-scraper/stargazers)
 [![License](https://img.shields.io/badge/license-MIT-D4F77A?style=flat-square&labelColor=121113)](LICENSE)
 
 <a href="https://maximilianfeix.github.io/proxy-scraper/"><img src="https://img.shields.io/badge/Browse_the_live_list-D4F77A?style=for-the-badge&labelColor=121113" alt="Browse the live list"></a>
@@ -94,7 +95,7 @@ Most free proxy lists are 95 % dead, and a good part of the rest are honeypots o
 - [Options](#options)
 - [GitHub Actions](#github-actions)
 - [FAQ](#faq)
-- [Roadmap](#roadmap) · [Contributing](#contributing) · [Acknowledgements](#acknowledgements)
+- [Roadmap](#roadmap) · [Contributing](#contributing) · [Community](#community) · [Acknowledgements](#acknowledgements)
 
 </details>
 
@@ -384,6 +385,7 @@ macOS, Linux and Windows, Python 3.9 to 3.13. Only two dependencies: `rich` and
 | Only proxies that reach *your* target site | ❌ | ✅ `--target` |
 | Learns which sources are worth it | ❌ | ✅ |
 | Usable as a single rotating proxy | ❌ | ✅ `--serve` |
+| API to fetch a proxy (proxy_pool compatible) | ❌ | ✅ [`/get`](#pool-api) |
 | Ready-made list without running anything | ✅ | ✅ [live list](#live-list) |
 
 <a id="examples"></a>
@@ -519,6 +521,38 @@ Without a password, `--serve-host` means **anyone who reaches the port can use i
 
 In testing: 20 of 20 HTTPS requests succeeded, over 15 different exit IPs. In the wizard this is **Proxy server right away**.
 
+<a id="pool-api"></a>
+
+### Proxy pool API
+
+Some programs want a proxy *address*, not a proxy – to hand it to a browser, a worker or a queue. The same port answers plain HTTP requests with one:
+
+```bash
+curl http://127.0.0.1:8899/get                          # one proxy as JSON, chosen like a connection would be
+curl "http://127.0.0.1:8899/get?country=DE&https=1&format=txt"   # → http://203.0.113.7:8080
+curl "http://127.0.0.1:8899/all?protocol=socks5&limit=20"        # the 20 best SOCKS5 proxies
+curl "http://127.0.0.1:8899/report?proxy=203.0.113.7:8080&ok=0"  # it failed you – three times and it's out
+```
+
+| Endpoint | What it does |
+|---|---|
+| `/get` | one proxy, picked by the `--rotate` strategy |
+| `/pop` | like `/get`, and the proxy leaves the pool |
+| `/all` | every usable proxy, best first (`limit=N`) |
+| `/count` | totals by type and country |
+| `/delete?proxy=IP:PORT` | take a proxy out |
+| `/report?proxy=IP:PORT&ok=0` | feedback from your own requests |
+
+Filters work on `/get`, `/pop` and `/all`: `country=DE,AT`, `protocol=socks5`, `https=1`, `anonymity=elite`, `max_latency=1500`, and `format=txt` for plain URLs. With `--serve-password` the API wants it as Basic auth, like the status page.
+
+**Coming from [jhao104/proxy_pool](https://github.com/jhao104/proxy_pool)?** The endpoints, `type=https` and the JSON fields (`proxy`, `https`, `region`, `anonymous`, `check_count`, `fail_count`, …) are the same, so point your code at port 8899 and it keeps working – without Redis, and with proxies that passed the honeypot, tampering and TLS checks:
+
+```python
+import requests
+proxy = requests.get("http://127.0.0.1:8899/get?type=https").json()["url"]   # e.g. socks5://…, type included
+requests.get("https://example.com", proxies={"http": proxy, "https": proxy})
+```
+
 <a id="discord-bot"></a>
 
 ## Discord and Telegram bot
@@ -558,6 +592,15 @@ flowchart LR
 </div>
 </details>
 
+### Your own lists
+
+```bash
+proxy-scraper --source https://example.com/my-list.txt --source socks5=./socks.txt   # on top of the 700+ sources
+proxy-scraper --only-sources --source bought.txt --want 50                            # only yours
+```
+
+Any text with `ip:port` works; lines like `socks5://user:pass@host:port` keep their type, bare ones are tried as HTTP and SOCKS5 unless you write `http=…`.
+
 <a id="output"></a>
 
 ## Output
@@ -924,7 +967,7 @@ proxyscraper/
 ├── targets.py          target sites for --target
 ├── output.py           result files
 ├── exporters.py        proxychains, Clash, sing-box and curl formats (--export)
-├── server/             rotating proxy server (--serve): pool · http · upstream · socks · status · core
+├── server/             rotating proxy server (--serve): pool · http · upstream · socks · status · api · core
 ├── api.py              find_proxies() / check_proxies() for Python
 ├── agent.py            MCP tools without the SDK: live list, filters, fetch through proxies
 ├── mcp_server.py       MCP server (proxy-scraper-mcp) for AI agents
@@ -937,6 +980,20 @@ proxyscraper/
 
 </details>
 
+<a id="community"></a>
+
+## Community
+
+Questions, ideas and things you built with it go to [Discussions](https://github.com/maximilianfeix/proxy-scraper/discussions) – bugs to the [issues](../../issues). If proxy-scraper saves you time, a ⭐ helps others find it.
+
+<a href="https://star-history.com/#maximilianfeix/proxy-scraper&Date">
+  <picture>
+    <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=maximilianfeix/proxy-scraper&type=Date&theme=dark">
+    <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=maximilianfeix/proxy-scraper&type=Date">
+    <img alt="Star history of proxy-scraper" src="https://api.star-history.com/svg?repos=maximilianfeix/proxy-scraper&type=Date" width="600">
+  </picture>
+</a>
+
 <a id="acknowledgements"></a>
 
 ## Acknowledgements

{proxy_scraper_cli-1.20.0 → proxy_scraper_cli-1.21.0}/proxy_scraper_cli.egg-info/SOURCES.txt

@@ -48,6 +48,7 @@ proxyscraper/sources.py
 proxyscraper/speed.py
 proxyscraper/targets.py
 proxyscraper/server/__init__.py
+proxyscraper/server/api.py
 proxyscraper/server/core.py
 proxyscraper/server/http.py
 proxyscraper/server/pool.py
@@ -88,6 +89,7 @@ tests/test_mcp_server.py
 tests/test_mine_sources.py
 tests/test_options.py
 tests/test_output_stdout.py
+tests/test_own_sources.py
 tests/test_packaging.py
 tests/test_pages.py
 tests/test_parsing.py
@@ -101,6 +103,7 @@ tests/test_publish.py
 tests/test_report.py
 tests/test_robustness.py
 tests/test_rotator.py
+tests/test_serve_api.py
 tests/test_serve_password.py
 tests/test_serve_refill.py
 tests/test_serve_v2.py

{proxy_scraper_cli-1.20.0 → proxy_scraper_cli-1.21.0}/proxyscraper/__init__.py

@@ -1,7 +1,7 @@
 """proxy-scraper: collects public HTTP/SOCKS4/SOCKS5 proxies from hundreds of sources, checks them
 in parallel with its own protocol handshakes and learns which sources deliver good proxies."""
 
-__version__ = "1.20.0"
+__version__ = "1.21.0"
 
 
 def __getattr__(name):

{proxy_scraper_cli-1.20.0 → proxy_scraper_cli-1.21.0}/proxyscraper/app.py

@@ -327,8 +327,10 @@ class Run:
         plan = await collect_sources(self.opts, self.quality)
         info("Sources", Text.assemble(
             (fmt(len(plan.sources)), f"bold {ACCENT}"), " active  ",
-            (f"({fmt(plan.n_curated)} curated · {fmt(plan.n_meta)} from {plan.meta_ok}/{plan.meta_total} "
-             f"meta lists · {fmt(plan.n_discovered)} discovered)", MUTED),
+            (f"({fmt(plan.n_own)} own)" if self.opts.only_sources else
+             f"({fmt(plan.n_curated)} curated · {fmt(plan.n_meta)} from {plan.meta_ok}/{plan.meta_total} "
+             f"meta lists · {fmt(plan.n_discovered)} discovered"
+             + (f" · {fmt(plan.n_own)} own)" if plan.n_own else ")"), MUTED),
         ))
         if plan.discovery_ran and not plan.discovery_token:
             note("GitHub discovery is limited without a token – run `gh auth login` or set GITHUB_TOKEN.",
@@ -355,7 +357,7 @@ class Run:
             hosts = sorted(INSECURE_HOSTS)
             note(f"Certificate could not be verified (TLS inspection on this network?), loaded anyway: "
                  f"{', '.join(hosts[:4])}{' …' if len(hosts) > 4 else ''}", MUTED, "ℹ")
-        return prioritize(res, self.quality, self.history, self.opts.types)
+        return prioritize(res, self.quality, self.history, self.opts.types, listed_only=self.opts.only_sources)
 
     # ------------------------------------------------------------------ phase 3+4: check, learn, report
 

{proxy_scraper_cli-1.20.0 → proxy_scraper_cli-1.21.0}/proxyscraper/cli.py

@@ -246,6 +246,12 @@ def parse_args(argv: Optional[List[str]] = None) -> argparse.Namespace:
     s.add_argument("--discover-repos", type=non_negative_int, default=DEFAULT_DISCOVER_REPOS,
                    help=f"max. repos during discovery (default: {DEFAULT_DISCOVER_REPOS}, 40 without a token)")
     s.add_argument("--all-sources", action="store_true", help="also load dead, outdated and unreachable sources")
+    s.add_argument("--source", action="append", metavar="URL|FILE",
+                   help="also load this proxy list (repeatable): a URL or a local file, one proxy per line or "
+                        "any text with ip:port in it; TYPE=… fixes the type, e.g. socks5=https://…/list.txt – "
+                        "without it, lines with a scheme keep theirs and bare ones are tried as HTTP and SOCKS5")
+    s.add_argument("--only-sources", action="store_true",
+                   help="only the lists given with --source – no built-in, meta or discovered sources")
     s.add_argument("--no-cache", action="store_true",
                    help="reload every list completely (otherwise unchanged ones are skipped via ETag)")
     s.add_argument("--list-sources", nargs="?", const=50, type=positive_int, metavar="N",

{proxy_scraper_cli-1.20.0 → proxy_scraper_cli-1.21.0}/proxyscraper/options.py

@@ -115,6 +115,8 @@ class RunOptions:
     no_discover: bool = False
     discover_repos: int = DEFAULT_DISCOVER_REPOS
     all_sources: bool = False
+    sources: List[str] = field(default_factory=list)  # own lists (--source): URLs, files, 'socks5=…'
+    only_sources: bool = False  # only the own lists – no curated, meta or discovered ones
     no_cache: bool = False  # reload every list instead of taking unchanged ones from the cache
     no_dnsbl: bool = False  # skip the blocklist lookup of the exit IPs
     serve: int = 0  # port of the rotating proxy server after the run, 0 = off
@@ -149,6 +151,8 @@ class RunOptions:
             raise ValueError("--sticky must not be negative")
         if self.serve_refill < 0:
             raise ValueError("--serve-refill must not be negative")
+        if self.only_sources and not self.sources:
+            raise ValueError("--only-sources needs at least one --source")
         if not 0 <= self.serve <= 65535:
             raise ValueError("port must be between 1 and 65535")
 
@@ -200,6 +204,8 @@ class RunOptions:
             no_discover=args.no_discover,
             discover_repos=args.discover_repos,
             all_sources=args.all_sources,
+            sources=list(dict.fromkeys(args.source or [])),
+            only_sources=args.only_sources,
             no_cache=args.no_cache,
             no_dnsbl=args.no_dnsbl,
             serve=args.serve,
@@ -245,6 +251,9 @@ class RunOptions:
         _flag(argv, "--no-discover", self.no_discover)
         _opt(argv, "--discover-repos", self.discover_repos, DEFAULT_DISCOVER_REPOS)
         _flag(argv, "--all-sources", self.all_sources)
+        for spec in self.sources:
+            argv += ["--source", spec]
+        _flag(argv, "--only-sources", self.only_sources)
         _flag(argv, "--no-cache", self.no_cache)
         _flag(argv, "--no-dnsbl", self.no_dnsbl)
         if self.serve:

{proxy_scraper_cli-1.20.0 → proxy_scraper_cli-1.21.0}/proxyscraper/pipeline.py

@@ -9,7 +9,10 @@ import random
 from collections import Counter
 from concurrent.futures import ProcessPoolExecutor
 from dataclasses import dataclass, field
+from pathlib import Path
 from typing import Callable, Dict, Iterable, List, Optional, Set, Tuple
+from urllib.parse import urlsplit
+from urllib.request import url2pathname
 
 from . import sources as srcs
 from .asndb import ProviderLookup
@@ -45,12 +48,18 @@ class SourcePlan:
     meta_ok: int
     meta_total: int
     n_discovered: int
+    n_own: int = 0
     discovery_ran: bool = False
     discovery_token: bool = False
 
 
 async def collect_sources(opts: RunOptions, quality: srcs.SourceStats) -> SourcePlan:
-    """Curated + meta + discovered sources, minus the ones learned to be bad."""
+    """Curated + meta + discovered sources, minus the ones learned to be bad, plus the own ones (--source)."""
+    own = dict(srcs.parse_source_spec(spec) for spec in opts.sources)
+    wanted = set(opts.types)
+    own = {u: t for u, t in own.items() if t == "auto" or t in wanted}
+    if opts.only_sources:
+        return SourcePlan(own, Counter(), 0, 0, 0, 0, 0, n_own=len(own))
     sources, meta = srcs.load_source_file()
     n_curated = len(sources)
 
@@ -77,7 +86,6 @@ async def collect_sources(opts: RunOptions, quality: srcs.SourceStats) -> Source
         for url, ptype in extra.items():
             sources.setdefault(url, ptype)
 
-    wanted = set(opts.types)
     sources = {u: t for u, t in sources.items() if t == "auto" or t in wanted}
     skipped: Counter = Counter()
     if not opts.all_sources:
@@ -89,8 +97,9 @@ async def collect_sources(opts: RunOptions, quality: srcs.SourceStats) -> Source
             else:
                 active[url] = ptype
         sources = active
+    sources.update(own)  # asked for by name – never skipped, and the given type wins
     return SourcePlan(
-        sources, skipped, n_curated, len(meta_found), meta_ok, len(meta), len(discovered),
+        sources, skipped, n_curated, len(meta_found), meta_ok, len(meta), len(discovered), n_own=len(own),
         discovery_ran=run_discovery, discovery_token=token is not None,
     )
 
@@ -123,7 +132,9 @@ async def scrape(sources: Dict[str, str], types, quality: srcs.SourceStats, view
     with ProcessPoolExecutor(max_workers=max(1, (os.cpu_count() or 2) - 1)) as pool:
 
         async def download(url: str):
-            """-> (data, keys from the cache). Exactly one of the two is set."""
+            """-> (data, headers, keys from the cache). Exactly one of data and keys is set."""
+            if url.startswith("file://"):  # own list from --source
+                return await asyncio.to_thread(Path(url2pathname(urlsplit(url).path)).read_bytes), {}, None
             conditional = cache.conditional_headers(url, sources[url])
             async with sem:
                 status, headers, body = await http_request(url, timeout=30, headers=conditional or None)
@@ -152,7 +163,8 @@ async def scrape(sources: Dict[str, str], types, quality: srcs.SourceStats, view
                         keys = parse_blob(data, sources[url], parse_types)
                     else:
                         keys = await loop.run_in_executor(pool, parse_blob, data, sources[url], parse_types)
-                    cache.store(url, headers, keys, sources[url])
+                    if not url.startswith("file://"):
+                        cache.store(url, headers, keys, sources[url])
             except Exception:  # source unreachable/broken – counts as a failure in the statistics
                 pass
             n = parsed = 0
@@ -180,13 +192,15 @@ async def scrape(sources: Dict[str, str], types, quality: srcs.SourceStats, view
     return res
 
 
-def prioritize(res: ScrapeResult, quality: srcs.SourceStats, history: ProxyHistory, types) -> List[str]:
+def prioritize(res: ScrapeResult, quality: srcs.SourceStats, history: ProxyHistory, types,
+               listed_only: bool = False) -> List[str]:
     """Order of checking: known working proxies first, then by quality of the sources.
 
-    Within the same source quality, proxies that appear in more lists win.
+    Within the same source quality, proxies that appear in more lists win. listed_only (--only-sources):
+    the history only moves proxies of these lists to the front, it doesn't add others.
     """
     wanted = set(types)
-    known = [k for k in history.ranked_keys() if split_key(k)[0] in wanted]
+    known = [k for k in history.ranked_keys() if split_key(k)[0] in wanted and (not listed_only or k in res.index)]
     known_set = set(known)
 
     scores = [quality.score(u) for u in res.urls]