reqstorm 2.2.0__tar.gz → 2.3.0__tar.gz

{reqstorm-2.2.0/reqstorm.egg-info → reqstorm-2.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: reqstorm
-Version: 2.2.0
+Version: 2.3.0
 Summary: Send large numbers of HTTP requests concurrently with asyncio, with retries, timeouts and per-request results.
 Author-email: Melih Colpan <colpanmelih@gmail.com>
 License-Expression: MIT
@@ -28,15 +28,19 @@ License-File: LICENSE
 Requires-Dist: aiohttp<4,>=3.9
 Provides-Extra: pydantic
 Requires-Dist: pydantic>=2; extra == "pydantic"
+Provides-Extra: socks
+Requires-Dist: aiohttp-socks>=0.10; extra == "socks"
 Provides-Extra: test
 Requires-Dist: pytest>=8; extra == "test"
 Requires-Dist: pytest-asyncio>=0.23; extra == "test"
 Requires-Dist: trustme>=1.1; extra == "test"
 Requires-Dist: pydantic>=2; extra == "test"
+Requires-Dist: aiohttp-socks>=0.10; extra == "test"
 Provides-Extra: lint
 Requires-Dist: ruff>=0.6; extra == "lint"
 Requires-Dist: mypy>=1.10; extra == "lint"
 Requires-Dist: pydantic>=2; extra == "lint"
+Requires-Dist: aiohttp-socks>=0.10; extra == "lint"
 Dynamic: license-file
 
 # reqstorm
@@ -123,11 +127,11 @@ reqstorm does all of that for you, with one call.
 | One result per request | Failures are recorded, never raised; every attempt is kept |
 | Rate limits | Per host, in any unit (`"100/min"`), or `"auto"` from the server's 429s and headers |
 | Concurrency | Overall and per host |
-| Retries | Immediate, with backoff and `Retry-After`; plus end-of-run rounds |
+| Retries | Exponential backoff with a cap and jitter, `Retry-After`; plus end-of-run rounds |
 | Reports | Failures by reason, p50/p95/p99 response times, per-host figures |
 | Pagination | Next links, `Link` headers, cursors and page numbers |
 | Requests from data | URL templates over CSV rows or SQL query results |
-| Tokens and proxies | Refresh an expired token on 401; rotate through proxies |
+| Tokens and proxies | Refresh an expired token on 401; rotate through HTTP and SOCKS proxies |
 | Caching | ETag / `If-None-Match`: unchanged resources cost a 304 |
 | Output | JSONL, CSV, SQLite, PostgreSQL, MySQL; ordered or as completed |
 | Typed columns | JSON fields to checked columns, nested paths, arrays to rows |
@@ -142,7 +146,7 @@ reqstorm does all of that for you, with one call.
 $ python -m pip install reqstorm
 ```
 
-reqstorm supports Python 3.9 to 3.13 on Linux, macOS and Windows. Its only dependency is [aiohttp](https://docs.aiohttp.org). For PostgreSQL or MySQL output, install the driver you already use (`psycopg`, `psycopg2`, `pymysql`, `mysqlclient` or `mysql-connector-python`). To use Pydantic models as schemas, install `reqstorm[pydantic]`.
+reqstorm supports Python 3.9 to 3.13 on Linux, macOS and Windows. Its only dependency is [aiohttp](https://docs.aiohttp.org). For PostgreSQL or MySQL output, install the driver you already use (`psycopg`, `psycopg2`, `pymysql`, `mysqlclient` or `mysql-connector-python`). To use Pydantic models as schemas, install `reqstorm[pydantic]`; for SOCKS proxies, `reqstorm[socks]`.
 
 ## Quick start
 
@@ -306,13 +310,16 @@ results = reqstorm.fetch_all_sync(
     timeout=10,            # seconds per attempt
     retries=3,             # retry right away...
     backoff=0.5,           # ...0.5 s, 1 s, 2 s apart
+    max_backoff=30,        # never wait longer
     retry_rounds=2,        # then resend what still
     retry_round_delay=30,  # failed, 30 s later
 )
 ```
 
 - **`timeout`** is the time allowed for one attempt, including reading the body (default 30 s; `None` disables it). A slow server only fails its own requests.
-- **`retries`** retries a request right away. The delay starts at `backoff` and doubles; a `Retry-After` header (up to 60 s) takes precedence.
+- **`retries`** retries a request right away, also when no response came back at all (timeouts, dropped connections). The wait starts at `backoff` and doubles each time (0.5, 1, 2, 4, ... s), up to **`max_backoff`** (30 s by default), so many retries never wait minutes.
+- **Jitter** (on by default) waits a random time between half and all of that delay, so thousands of requests that failed together do not retry at the same moment. `jitter=False` waits exactly.
+- A **`Retry-After`** header from the server (up to 60 s) is followed as given instead.
 - **`retry_rounds`** holds back requests that still failed for a retryable reason and sends them again after the rest of the batch, for temporary outages. Each request still produces exactly one result, with all its attempts in `history`.
 
 | Retried | Not retried |
@@ -484,15 +491,17 @@ auth = reqstorm.BearerAuth(refresh=get_token)
 reqstorm.fetch_all_sync(urls, auth=auth)
 ```
 
-**Proxies.** One proxy, a pool used in turn, or one per `Request`:
+**Proxies.** One proxy, a pool used in turn, or one per `Request`. HTTP and SOCKS proxies can be mixed:
 
 ```python
 reqstorm.fetch_all_sync(urls, proxy=[
     "http://proxy-1.example.com:8080",
-    "http://proxy-2.example.com:8080",
+    "socks5h://user:pass@proxy-2.example.com:1080",
 ])
 ```
 
+SOCKS4 and SOCKS5 (`socks5h://` and `socks4a://` let the proxy resolve host names, as with Tor) need `pip install "reqstorm[socks]"`.
+
 **Caching.** `Cache` keeps responses in an SQLite file. The next run asks the server with `If-None-Match`; an unchanged resource comes back as a `304` with no body, and the stored response is used. With `ttl`, recent responses skip the network entirely:
 
 ```python

{reqstorm-2.2.0 → reqstorm-2.3.0}/README.md

@@ -82,11 +82,11 @@ reqstorm does all of that for you, with one call.
 | One result per request | Failures are recorded, never raised; every attempt is kept |
 | Rate limits | Per host, in any unit (`"100/min"`), or `"auto"` from the server's 429s and headers |
 | Concurrency | Overall and per host |
-| Retries | Immediate, with backoff and `Retry-After`; plus end-of-run rounds |
+| Retries | Exponential backoff with a cap and jitter, `Retry-After`; plus end-of-run rounds |
 | Reports | Failures by reason, p50/p95/p99 response times, per-host figures |
 | Pagination | Next links, `Link` headers, cursors and page numbers |
 | Requests from data | URL templates over CSV rows or SQL query results |
-| Tokens and proxies | Refresh an expired token on 401; rotate through proxies |
+| Tokens and proxies | Refresh an expired token on 401; rotate through HTTP and SOCKS proxies |
 | Caching | ETag / `If-None-Match`: unchanged resources cost a 304 |
 | Output | JSONL, CSV, SQLite, PostgreSQL, MySQL; ordered or as completed |
 | Typed columns | JSON fields to checked columns, nested paths, arrays to rows |
@@ -101,7 +101,7 @@ reqstorm does all of that for you, with one call.
 $ python -m pip install reqstorm
 ```
 
-reqstorm supports Python 3.9 to 3.13 on Linux, macOS and Windows. Its only dependency is [aiohttp](https://docs.aiohttp.org). For PostgreSQL or MySQL output, install the driver you already use (`psycopg`, `psycopg2`, `pymysql`, `mysqlclient` or `mysql-connector-python`). To use Pydantic models as schemas, install `reqstorm[pydantic]`.
+reqstorm supports Python 3.9 to 3.13 on Linux, macOS and Windows. Its only dependency is [aiohttp](https://docs.aiohttp.org). For PostgreSQL or MySQL output, install the driver you already use (`psycopg`, `psycopg2`, `pymysql`, `mysqlclient` or `mysql-connector-python`). To use Pydantic models as schemas, install `reqstorm[pydantic]`; for SOCKS proxies, `reqstorm[socks]`.
 
 ## Quick start
 
@@ -265,13 +265,16 @@ results = reqstorm.fetch_all_sync(
     timeout=10,            # seconds per attempt
     retries=3,             # retry right away...
     backoff=0.5,           # ...0.5 s, 1 s, 2 s apart
+    max_backoff=30,        # never wait longer
     retry_rounds=2,        # then resend what still
     retry_round_delay=30,  # failed, 30 s later
 )
 ```
 
 - **`timeout`** is the time allowed for one attempt, including reading the body (default 30 s; `None` disables it). A slow server only fails its own requests.
-- **`retries`** retries a request right away. The delay starts at `backoff` and doubles; a `Retry-After` header (up to 60 s) takes precedence.
+- **`retries`** retries a request right away, also when no response came back at all (timeouts, dropped connections). The wait starts at `backoff` and doubles each time (0.5, 1, 2, 4, ... s), up to **`max_backoff`** (30 s by default), so many retries never wait minutes.
+- **Jitter** (on by default) waits a random time between half and all of that delay, so thousands of requests that failed together do not retry at the same moment. `jitter=False` waits exactly.
+- A **`Retry-After`** header from the server (up to 60 s) is followed as given instead.
 - **`retry_rounds`** holds back requests that still failed for a retryable reason and sends them again after the rest of the batch, for temporary outages. Each request still produces exactly one result, with all its attempts in `history`.
 
 | Retried | Not retried |
@@ -443,15 +446,17 @@ auth = reqstorm.BearerAuth(refresh=get_token)
 reqstorm.fetch_all_sync(urls, auth=auth)
 ```
 
-**Proxies.** One proxy, a pool used in turn, or one per `Request`:
+**Proxies.** One proxy, a pool used in turn, or one per `Request`. HTTP and SOCKS proxies can be mixed:
 
 ```python
 reqstorm.fetch_all_sync(urls, proxy=[
     "http://proxy-1.example.com:8080",
-    "http://proxy-2.example.com:8080",
+    "socks5h://user:pass@proxy-2.example.com:1080",
 ])
 ```
 
+SOCKS4 and SOCKS5 (`socks5h://` and `socks4a://` let the proxy resolve host names, as with Tor) need `pip install "reqstorm[socks]"`.
+
 **Caching.** `Cache` keeps responses in an SQLite file. The next run asks the server with `If-None-Match`; an unchanged resource comes back as a `304` with no body, and the stored response is used. With `ttl`, recent responses skip the network entirely:
 
 ```python

{reqstorm-2.2.0 → reqstorm-2.3.0}/pyproject.toml

@@ -31,8 +31,9 @@ classifiers = [
 
 [project.optional-dependencies]
 pydantic = ["pydantic>=2"]
-test = ["pytest>=8", "pytest-asyncio>=0.23", "trustme>=1.1", "pydantic>=2"]
-lint = ["ruff>=0.6", "mypy>=1.10", "pydantic>=2"]
+socks = ["aiohttp-socks>=0.10"]
+test = ["pytest>=8", "pytest-asyncio>=0.23", "trustme>=1.1", "pydantic>=2", "aiohttp-socks>=0.10"]
+lint = ["ruff>=0.6", "mypy>=1.10", "pydantic>=2", "aiohttp-socks>=0.10"]
 
 [project.scripts]
 reqstorm = "reqstorm._cli:main"

{reqstorm-2.2.0 → reqstorm-2.3.0}/reqstorm/__init__.py

@@ -32,7 +32,7 @@ from ._schema import Field, Schema, SchemaError, extract, infer_schema
 from ._sync import fetch_all_sync, fetch_to_db_sync, fetch_to_file_sync, stream_sync
 from ._template import from_template, read_csv, read_sql
 
-__version__ = "2.2.0"
+__version__ = "2.3.0"
 
 __all__ = [
     "Attempt",

{reqstorm-2.2.0 → reqstorm-2.3.0}/reqstorm/_cli.py

@@ -57,7 +57,11 @@ def _parser() -> argparse.ArgumentParser:
     request.add_argument("--bearer", metavar="TOKEN", help="send 'Authorization: Bearer TOKEN' "
                          "(or set REQSTORM_TOKEN)")  # fmt: skip
     request.add_argument(
-        "--proxy", action="append", default=[], metavar="URL", help="proxy URL; repeat to rotate"
+        "--proxy",
+        action="append",
+        default=[],
+        metavar="URL",
+        help="http://, socks5://, socks5h:// or socks4:// proxy URL; repeat to rotate",
     )
     request.add_argument("--no-verify", action="store_true", help="do not verify TLS certificates")
 
@@ -67,7 +71,13 @@ def _parser() -> argparse.ArgumentParser:
     pace.add_argument("--per-host", type=int, default=0, metavar="N", help="max requests in flight per host")
     pace.add_argument("--timeout", type=float, default=30.0, help="seconds per attempt (default 30)")
     pace.add_argument("--retries", type=int, default=0)
-    pace.add_argument("--backoff", type=float, default=0.5)
+    pace.add_argument(
+        "--backoff", type=float, default=0.5, help="seconds before the first retry, doubled after"
+    )
+    pace.add_argument(
+        "--max-backoff", type=float, default=30.0, help="longest wait between retries (default 30)"
+    )
+    pace.add_argument("--no-jitter", action="store_true", help="wait exactly, not a random part of the delay")
     pace.add_argument("--retry-rounds", type=int, default=0)
     pace.add_argument("--retry-round-delay", type=float, default=5.0)
     pace.add_argument("--paginate", metavar="STRATEGY",
@@ -204,7 +214,7 @@ def main(argv: Optional[Sequence[str]] = None) -> int:
     args = parser.parse_args(argv)
     try:
         return _run(args)
-    except (ValueError, KeyError, SchemaError, OSError, json.JSONDecodeError) as error:
+    except (ValueError, KeyError, SchemaError, OSError, ImportError, json.JSONDecodeError) as error:
         message = error.args[0] if isinstance(error, KeyError) and error.args else error
         print(f"reqstorm: error: {message}", file=sys.stderr)
         return 2
@@ -260,6 +270,8 @@ def _batch(args: argparse.Namespace, cache: Optional[Cache]) -> int:
         timeout=args.timeout,
         retries=args.retries,
         backoff=args.backoff,
+        max_backoff=args.max_backoff,
+        jitter=not args.no_jitter,
         verify_ssl=not args.no_verify,
         rate_limit=rate,
         auth=BearerAuth(token) if token else None,

{reqstorm-2.2.0 → reqstorm-2.3.0}/reqstorm/_client.py

@@ -7,6 +7,7 @@ import base64
 import collections
 import itertools
 import json as jsonlib
+import random
 import ssl as ssllib
 import time
 from dataclasses import dataclass, field
@@ -34,6 +35,7 @@ from multidict import CIMultiDict, CIMultiDictProxy
 from ._adaptive import AdaptiveRateLimiter
 from ._limits import HostRateLimiter, RateLimit
 from ._progress import Progress, ProgressTarget
+from ._socks import SocksSessions, check_available, is_socks
 
 __all__ = ["Attempt", "HTTPStatusError", "Request", "Result", "Results", "fetch_all", "stream"]
 
@@ -245,6 +247,18 @@ class _Options:
     auth: Any = None
     cache: Any = None
     proxies: Optional[Iterator[str]] = None
+    socks: Optional[SocksSessions] = None
+    max_backoff: float = 30.0
+    jitter: bool = True
+
+    def backoff_delay(self, retry: int) -> float:
+        """Seconds to wait before retry number ``retry`` (1 for the first)."""
+        delay = min(self.backoff * (2 ** (retry - 1)), self.max_backoff)
+        if self.jitter:
+            # "Equal jitter": somewhere between half the delay and the full delay, so
+            # requests that failed together do not all retry at the same moment
+            delay = delay / 2 + random.uniform(0, delay / 2)
+        return delay
 
     def next_proxy(self) -> Optional[str]:
         return next(self.proxies) if self.proxies is not None else None
@@ -327,7 +341,13 @@ async def _send(session: aiohttp.ClientSession, request: Request, index: int, op
             if limiter is not None:
                 await limiter.wait(request.url)
             attempt_started = time.monotonic()
-            async with session.request(
+            proxy = request.proxy or options.next_proxy()
+            sender = session
+            if proxy is not None and is_socks(proxy):
+                if options.socks is None:
+                    raise ValueError("a SOCKS proxy needs a session created by reqstorm, not session=")
+                sender, proxy = options.socks.session(proxy), None
+            async with sender.request(
                 request.method or "GET",
                 request.url,
                 headers=headers or None,
@@ -336,7 +356,7 @@ async def _send(session: aiohttp.ClientSession, request: Request, index: int, op
                 data=request.data,
                 timeout=options.timeout,
                 ssl=options.ssl,
-                proxy=request.proxy or options.next_proxy(),
+                proxy=proxy,
             ) as response:
                 body = await response.read()
                 status, response_headers, final_url = response.status, response.headers, str(response.url)
@@ -361,7 +381,7 @@ async def _send(session: aiohttp.ClientSession, request: Request, index: int, op
             if status not in options.retry_statuses or retries_used >= options.retries:
                 break
             retries_used += 1
-            delay = _retry_after(response_headers) or options.backoff * (2 ** (retries_used - 1))
+            delay = _retry_after(response_headers) or options.backoff_delay(retries_used)
         except asyncio.CancelledError:
             raise
         except Exception as error:  # one failing request must not stop the others
@@ -371,7 +391,7 @@ async def _send(session: aiohttp.ClientSession, request: Request, index: int, op
             if not _is_retryable_error(error) or retries_used >= options.retries:
                 break
             retries_used += 1
-            delay = options.backoff * (2 ** (retries_used - 1))
+            delay = options.backoff_delay(retries_used)
         if delay:
             await asyncio.sleep(delay)
     if cache_key and result.ok and result.status == 200 and not result.from_cache:
@@ -392,6 +412,8 @@ async def stream(
     timeout: Optional[float] = 30.0,
     retries: int = 0,
     backoff: float = 0.5,
+    max_backoff: float = 30.0,
+    jitter: bool = True,
     retry_statuses: Sequence[int] = DEFAULT_RETRY_STATUSES,
     verify_ssl: bool = True,
     ssl: Optional[ssllib.SSLContext] = None,
@@ -416,6 +438,8 @@ async def stream(
         raise ValueError("retries must not be negative")
     if concurrency_per_host < 0:
         raise ValueError("concurrency_per_host must not be negative")
+    if backoff < 0 or max_backoff < 0:
+        raise ValueError("backoff and max_backoff must not be negative")
     if rate_limit == "auto":
         limiter: Any = AdaptiveRateLimiter()
     elif rate_limit is not None:
@@ -423,6 +447,10 @@ async def stream(
     else:
         limiter = None
     proxies = [proxy] if isinstance(proxy, str) else list(proxy or [])
+    if any(is_socks(item) for item in proxies):
+        check_available()  # fail before sending anything, not on every request
+        if session is not None:
+            raise ValueError("SOCKS proxies cannot be combined with session=; reqstorm opens their sessions")
     options = _Options(
         method=method,
         headers=headers,
@@ -432,12 +460,15 @@ async def stream(
         timeout=aiohttp.ClientTimeout(total=timeout),
         retries=retries,
         backoff=backoff,
+        max_backoff=max_backoff,
+        jitter=jitter,
         retry_statuses=frozenset(retry_statuses),
         ssl=ssl if ssl is not None else verify_ssl,
         rate_limiter=limiter,
         auth=auth,
         cache=cache,
         proxies=itertools.cycle(proxies) if proxies else None,
+        socks=SocksSessions(concurrency, concurrency_per_host) if session is None else None,
     )
 
     owns_session = session is None
@@ -527,6 +558,8 @@ async def stream(
             pass
         if owns_session:
             await session.close()
+        if options.socks is not None:
+            await options.socks.close()
 
 
 async def _execute(
@@ -590,6 +623,8 @@ async def fetch_all(
     timeout: Optional[float] = ...,
     retries: int = ...,
     backoff: float = ...,
+    max_backoff: float = ...,
+    jitter: bool = ...,
     retry_statuses: Sequence[int] = ...,
     retry_rounds: int = ...,
     retry_round_delay: float = ...,
@@ -631,6 +666,8 @@ async def fetch_all(
     timeout: Optional[float] = 30.0,
     retries: int = 0,
     backoff: float = 0.5,
+    max_backoff: float = 30.0,
+    jitter: bool = True,
     retry_statuses: Sequence[int] = DEFAULT_RETRY_STATUSES,
     retry_rounds: int = 0,
     retry_round_delay: float = 5.0,
@@ -660,7 +697,13 @@ async def fetch_all(
         timeout: Seconds allowed per attempt, including reading the body. ``None`` disables it.
         retries: How many times to retry a request right away after a connection error,
             a timeout or a status in ``retry_statuses``. Invalid URLs are not retried.
-        backoff: Delay before the first retry in seconds, doubled for each further retry.
+        backoff: Delay before the first retry in seconds, doubled for each further retry
+            (0.5, 1, 2, 4, ... with the default).
+        max_backoff: Upper limit for that delay in seconds (default 30), so many retries
+            never wait minutes. A ``Retry-After`` from the server is followed instead, up
+            to 60 seconds.
+        jitter: Wait a random time between half and all of the delay, so requests that
+            failed together do not retry at the same moment. ``False`` waits exactly.
             A ``Retry-After`` header (in seconds, up to 60) takes precedence.
         retry_rounds: After all requests have finished, send the ones that still failed
             for a retryable reason again, up to this many more rounds.
@@ -677,8 +720,10 @@ async def fetch_all(
         concurrency_per_host: Maximum requests in flight to each host; 0 means no per-host limit.
         auth: ``reqstorm.BearerAuth``: sends a token and refreshes it once on a 401.
         cache: ``reqstorm.Cache``: reuses earlier GET responses, revalidating with ETag.
-        proxy: A proxy URL (``"http://user:pass@host:port"``), or a list of them used in
-            turn. ``Request(proxy=...)`` sets one per request.
+        proxy: A proxy URL, or a list of them used in turn, attempt by attempt:
+            ``"http://user:pass@host:port"``, or ``socks5://``, ``socks5h://`` (the proxy
+            resolves host names), ``socks4://`` and ``socks4a://`` with
+            ``pip install 'reqstorm[socks]'``. ``Request(proxy=...)`` sets one per request.
         paginate: ``reqstorm.NextLink``, ``LinkHeader``, ``Cursor`` or ``PageNumber``: each
             starting URL is followed through its pages. Results carry ``page`` and
             ``seed_index``. Cannot be combined with ``retry_rounds``.
@@ -708,6 +753,8 @@ async def fetch_all(
         timeout=timeout,
         retries=retries,
         backoff=backoff,
+        max_backoff=max_backoff,
+        jitter=jitter,
         retry_statuses=retry_statuses,
         verify_ssl=verify_ssl,
         ssl=ssl,

reqstorm-2.3.0/reqstorm/_socks.py

@@ -0,0 +1,84 @@
+"""SOCKS4 and SOCKS5 proxies, through the optional aiohttp-socks package."""
+
+from __future__ import annotations
+
+from typing import Any, Dict, Optional, Tuple
+from urllib.parse import unquote, urlsplit
+
+import aiohttp
+
+SCHEMES = ("socks4", "socks4a", "socks5", "socks5h")
+_MISSING = "SOCKS proxies need the aiohttp-socks package: pip install 'reqstorm[socks]'"
+
+
+def is_socks(proxy: Optional[str]) -> bool:
+    return proxy is not None and urlsplit(proxy).scheme.lower() in SCHEMES
+
+
+def check_available() -> None:
+    try:
+        import aiohttp_socks  # noqa: F401
+    except ImportError:
+        raise ImportError(_MISSING) from None
+
+
+def parse(proxy: str) -> Tuple[str, str, int, Optional[str], Optional[str], bool]:
+    """(version, host, port, username, password, remote_dns) of a SOCKS proxy URL.
+
+    As with curl, ``socks5h://`` and ``socks4a://`` let the proxy resolve host names, while
+    ``socks5://`` and ``socks4://`` resolve them locally.
+    """
+    parts = urlsplit(proxy)
+    scheme = parts.scheme.lower()
+    if scheme not in SCHEMES:
+        raise ValueError(f"not a SOCKS proxy URL: {proxy!r}")
+    if not parts.hostname:
+        raise ValueError(f"SOCKS proxy URL without a host: {proxy!r}")
+    try:
+        port = parts.port or 1080
+    except ValueError:
+        raise ValueError(f"invalid port in SOCKS proxy URL {proxy!r}") from None
+    username = unquote(parts.username) if parts.username is not None else None
+    password = unquote(parts.password) if parts.password is not None else None
+    version = "socks5" if scheme.startswith("socks5") else "socks4"
+    return version, parts.hostname, port, username, password, scheme in ("socks5h", "socks4a")
+
+
+class SocksSessions:
+    """One aiohttp session per SOCKS proxy, created on first use and closed together.
+
+    A SOCKS proxy is a property of the connection, not of the request, so each proxy needs
+    its own connector. The connection limits match those of the main session.
+    """
+
+    def __init__(self, limit: int, limit_per_host: int) -> None:
+        self._limit = limit
+        self._limit_per_host = limit_per_host
+        self._sessions: Dict[str, aiohttp.ClientSession] = {}
+
+    def session(self, proxy: str) -> aiohttp.ClientSession:
+        existing = self._sessions.get(proxy)
+        if existing is not None:
+            return existing
+        check_available()
+        from aiohttp_socks import ProxyConnector, ProxyType
+
+        version, host, port, username, password, remote_dns = parse(proxy)
+        connector: Any = ProxyConnector(
+            proxy_type=ProxyType.SOCKS5 if version == "socks5" else ProxyType.SOCKS4,
+            host=host,
+            port=port,
+            username=username,
+            password=password,
+            rdns=remote_dns,
+            limit=self._limit,
+            limit_per_host=self._limit_per_host,
+        )
+        created = aiohttp.ClientSession(connector=connector)
+        self._sessions[proxy] = created
+        return created
+
+    async def close(self) -> None:
+        for session in self._sessions.values():
+            await session.close()
+        self._sessions.clear()

{reqstorm-2.2.0 → reqstorm-2.3.0/reqstorm.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: reqstorm
-Version: 2.2.0
+Version: 2.3.0
 Summary: Send large numbers of HTTP requests concurrently with asyncio, with retries, timeouts and per-request results.
 Author-email: Melih Colpan <colpanmelih@gmail.com>
 License-Expression: MIT
@@ -28,15 +28,19 @@ License-File: LICENSE
 Requires-Dist: aiohttp<4,>=3.9
 Provides-Extra: pydantic
 Requires-Dist: pydantic>=2; extra == "pydantic"
+Provides-Extra: socks
+Requires-Dist: aiohttp-socks>=0.10; extra == "socks"
 Provides-Extra: test
 Requires-Dist: pytest>=8; extra == "test"
 Requires-Dist: pytest-asyncio>=0.23; extra == "test"
 Requires-Dist: trustme>=1.1; extra == "test"
 Requires-Dist: pydantic>=2; extra == "test"
+Requires-Dist: aiohttp-socks>=0.10; extra == "test"
 Provides-Extra: lint
 Requires-Dist: ruff>=0.6; extra == "lint"
 Requires-Dist: mypy>=1.10; extra == "lint"
 Requires-Dist: pydantic>=2; extra == "lint"
+Requires-Dist: aiohttp-socks>=0.10; extra == "lint"
 Dynamic: license-file
 
 # reqstorm
@@ -123,11 +127,11 @@ reqstorm does all of that for you, with one call.
 | One result per request | Failures are recorded, never raised; every attempt is kept |
 | Rate limits | Per host, in any unit (`"100/min"`), or `"auto"` from the server's 429s and headers |
 | Concurrency | Overall and per host |
-| Retries | Immediate, with backoff and `Retry-After`; plus end-of-run rounds |
+| Retries | Exponential backoff with a cap and jitter, `Retry-After`; plus end-of-run rounds |
 | Reports | Failures by reason, p50/p95/p99 response times, per-host figures |
 | Pagination | Next links, `Link` headers, cursors and page numbers |
 | Requests from data | URL templates over CSV rows or SQL query results |
-| Tokens and proxies | Refresh an expired token on 401; rotate through proxies |
+| Tokens and proxies | Refresh an expired token on 401; rotate through HTTP and SOCKS proxies |
 | Caching | ETag / `If-None-Match`: unchanged resources cost a 304 |
 | Output | JSONL, CSV, SQLite, PostgreSQL, MySQL; ordered or as completed |
 | Typed columns | JSON fields to checked columns, nested paths, arrays to rows |
@@ -142,7 +146,7 @@ reqstorm does all of that for you, with one call.
 $ python -m pip install reqstorm
 ```
 
-reqstorm supports Python 3.9 to 3.13 on Linux, macOS and Windows. Its only dependency is [aiohttp](https://docs.aiohttp.org). For PostgreSQL or MySQL output, install the driver you already use (`psycopg`, `psycopg2`, `pymysql`, `mysqlclient` or `mysql-connector-python`). To use Pydantic models as schemas, install `reqstorm[pydantic]`.
+reqstorm supports Python 3.9 to 3.13 on Linux, macOS and Windows. Its only dependency is [aiohttp](https://docs.aiohttp.org). For PostgreSQL or MySQL output, install the driver you already use (`psycopg`, `psycopg2`, `pymysql`, `mysqlclient` or `mysql-connector-python`). To use Pydantic models as schemas, install `reqstorm[pydantic]`; for SOCKS proxies, `reqstorm[socks]`.
 
 ## Quick start
 
@@ -306,13 +310,16 @@ results = reqstorm.fetch_all_sync(
     timeout=10,            # seconds per attempt
     retries=3,             # retry right away...
     backoff=0.5,           # ...0.5 s, 1 s, 2 s apart
+    max_backoff=30,        # never wait longer
     retry_rounds=2,        # then resend what still
     retry_round_delay=30,  # failed, 30 s later
 )
 ```
 
 - **`timeout`** is the time allowed for one attempt, including reading the body (default 30 s; `None` disables it). A slow server only fails its own requests.
-- **`retries`** retries a request right away. The delay starts at `backoff` and doubles; a `Retry-After` header (up to 60 s) takes precedence.
+- **`retries`** retries a request right away, also when no response came back at all (timeouts, dropped connections). The wait starts at `backoff` and doubles each time (0.5, 1, 2, 4, ... s), up to **`max_backoff`** (30 s by default), so many retries never wait minutes.
+- **Jitter** (on by default) waits a random time between half and all of that delay, so thousands of requests that failed together do not retry at the same moment. `jitter=False` waits exactly.
+- A **`Retry-After`** header from the server (up to 60 s) is followed as given instead.
 - **`retry_rounds`** holds back requests that still failed for a retryable reason and sends them again after the rest of the batch, for temporary outages. Each request still produces exactly one result, with all its attempts in `history`.
 
 | Retried | Not retried |
@@ -484,15 +491,17 @@ auth = reqstorm.BearerAuth(refresh=get_token)
 reqstorm.fetch_all_sync(urls, auth=auth)
 ```
 
-**Proxies.** One proxy, a pool used in turn, or one per `Request`:
+**Proxies.** One proxy, a pool used in turn, or one per `Request`. HTTP and SOCKS proxies can be mixed:
 
 ```python
 reqstorm.fetch_all_sync(urls, proxy=[
     "http://proxy-1.example.com:8080",
-    "http://proxy-2.example.com:8080",
+    "socks5h://user:pass@proxy-2.example.com:1080",
 ])
 ```
 
+SOCKS4 and SOCKS5 (`socks5h://` and `socks4a://` let the proxy resolve host names, as with Tor) need `pip install "reqstorm[socks]"`.
+
 **Caching.** `Cache` keeps responses in an SQLite file. The next run asks the server with `If-None-Match`; an unchanged resource comes back as a `304` with no body, and the stored response is used. With `ttl`, recent responses skip the network entirely:
 
 ```python

{reqstorm-2.2.0 → reqstorm-2.3.0}/reqstorm.egg-info/SOURCES.txt

@@ -18,6 +18,7 @@ reqstorm/_pydantic.py
 reqstorm/_report.py
 reqstorm/_schema.py
 reqstorm/_schema_sinks.py
+reqstorm/_socks.py
 reqstorm/_sync.py
 reqstorm/_template.py
 reqstorm/py.typed
@@ -42,6 +43,7 @@ tests/test_report.py
 tests/test_run_report.py
 tests/test_schema.py
 tests/test_schema_db.py
+tests/test_socks_backoff.py
 tests/test_stream.py
 tests/test_sync.py
 tests/test_template.py

{reqstorm-2.2.0 → reqstorm-2.3.0}/reqstorm.egg-info/requires.txt

@@ -4,12 +4,17 @@ aiohttp<4,>=3.9
 ruff>=0.6
 mypy>=1.10
 pydantic>=2
+aiohttp-socks>=0.10
 
 [pydantic]
 pydantic>=2
 
+[socks]
+aiohttp-socks>=0.10
+
 [test]
 pytest>=8
 pytest-asyncio>=0.23
 trustme>=1.1
 pydantic>=2
+aiohttp-socks>=0.10

reqstorm-2.3.0/tests/test_socks_backoff.py

@@ -0,0 +1,144 @@
+import time
+
+import pytest
+
+import reqstorm
+from reqstorm._client import _Options
+from reqstorm._socks import is_socks, parse
+
+pytest.importorskip("aiohttp_socks")
+
+
+def _port(server):
+    return int(server.rsplit(":", 1)[1])
+
+
+async def test_socks5_connects_through_the_proxy(server, socks_proxy):
+    results = await reqstorm.fetch_all([server + "/ok"], proxy=socks_proxy.url("socks5"))
+    assert results[0].text() == "ok"
+    assert socks_proxy.requests == [(5, "127.0.0.1", _port(server))]
+
+
+async def test_socks5h_lets_the_proxy_resolve_names(server, socks_proxy):
+    url = f"http://api.example.test:{_port(server)}/ok"
+    results = await reqstorm.fetch_all([url], proxy=socks_proxy.url("socks5h"))
+    assert results[0].text() == "ok"
+    assert socks_proxy.requests == [(5, "api.example.test", _port(server))]
+
+
+async def test_socks5_resolves_names_locally(server, socks_proxy):
+    url = f"http://api.example.test:{_port(server)}/ok"
+    results = await reqstorm.fetch_all([url], proxy=socks_proxy.url("socks5"), timeout=5)
+    assert results[0].error is not None  # the made-up name does not resolve here
+    assert socks_proxy.requests == []
+
+
+async def test_socks5_with_username_and_password(server, socks_proxy_auth):
+    proxy = socks_proxy_auth.url("socks5", ("ayse", "p%40ss%3Aword"))  # percent-encoded in the URL
+    results = await reqstorm.fetch_all([server + "/ok"], proxy=proxy)
+    assert results[0].text() == "ok"
+    wrong = await reqstorm.fetch_all([server + "/ok"], proxy=socks_proxy_auth.url("socks5", ("ayse", "x")))
+    assert wrong[0].error is not None and not wrong[0].ok
+
+
+@pytest.mark.parametrize("scheme, requested", [("socks4", "127.0.0.1"), ("socks4a", "api.example.test")])
+async def test_socks4_and_4a(server, socks_proxy, scheme, requested):
+    host = "127.0.0.1" if scheme == "socks4" else "api.example.test"
+    results = await reqstorm.fetch_all([f"http://{host}:{_port(server)}/ok"], proxy=socks_proxy.url(scheme))
+    assert results[0].text() == "ok"
+    assert socks_proxy.requests == [(4, requested, _port(server))]
+
+
+async def test_https_through_socks(tls_server, trusted_context, socks_proxy):
+    results = await reqstorm.fetch_all([tls_server + "/ok"], proxy=socks_proxy.url(), ssl=trusted_context)
+    assert results[0].text() == "ok" and len(socks_proxy.requests) == 1
+
+
+async def test_pool_mixes_socks_and_http_proxies(server, second_server, socks_proxy):
+    # second_server stands in for an HTTP proxy and answers itself; the SOCKS proxy
+    # forwards to the real server. /proxy-check reports which port answered.
+    url = f"http://api.example.test:{_port(server)}/proxy-check"
+    proxies = [second_server, socks_proxy.url("socks5h")]
+    results = await reqstorm.fetch_all([url] * 4, proxy=proxies, concurrency=1)
+    ports = [result.json()["port"] for result in results]
+    assert ports == [_port(second_server), _port(server)] * 2
+    assert len(socks_proxy.requests) == 1  # the SOCKS connection is reused
+
+
+async def test_per_request_socks_proxy_and_connection_reuse(server, socks_proxy):
+    requests = [reqstorm.Request(server + "/ok", proxy=socks_proxy.url()) for _ in range(5)]
+    results = await reqstorm.fetch_all(requests, concurrency=1)
+    assert all(result.ok for result in results)
+    assert len(socks_proxy.requests) == 1  # one connection, kept alive
+
+
+async def test_socks_cannot_share_a_user_session(server, socks_proxy):
+    import aiohttp
+
+    async with aiohttp.ClientSession() as session:
+        with pytest.raises(ValueError, match="session"):
+            await reqstorm.fetch_all([server + "/ok"], proxy=socks_proxy.url(), session=session)
+
+
+async def test_missing_aiohttp_socks_is_explained(server, monkeypatch):
+    import builtins
+
+    real_import = builtins.__import__
+
+    def without_socks(name, *args, **kwargs):
+        if name.startswith("aiohttp_socks"):
+            raise ImportError(name)
+        return real_import(name, *args, **kwargs)
+
+    monkeypatch.setattr(builtins, "__import__", without_socks)
+    with pytest.raises(ImportError, match=r"reqstorm\[socks\]"):
+        await reqstorm.fetch_all([server + "/ok"], proxy="socks5://127.0.0.1:1080")
+
+
+def test_proxy_url_parsing():
+    assert parse("socks5h://user:p%40ss@proxy.example.com") == (
+        "socks5",
+        "proxy.example.com",
+        1080,
+        "user",
+        "p@ss",
+        True,
+    )
+    assert parse("SOCKS4://10.0.0.1:9050")[:3] == ("socks4", "10.0.0.1", 9050)
+    assert is_socks("socks5://x:1") and not is_socks("http://x:1") and not is_socks(None)
+    with pytest.raises(ValueError):
+        parse("socks5://")
+
+
+def _options(**overrides):
+    import aiohttp
+
+    defaults = dict(method="GET", headers=None, params=None, json=None, data=None,
+                    timeout=aiohttp.ClientTimeout(total=1), retries=10, backoff=0.5,
+                    retry_statuses=frozenset(), ssl=True, rate_limiter=None)  # fmt: skip
+    return _Options(**{**defaults, **overrides})
+
+
+def test_backoff_doubles_and_stops_at_max_backoff():
+    options = _options(jitter=False, max_backoff=4)
+    assert [options.backoff_delay(n) for n in range(1, 7)] == [0.5, 1, 2, 4, 4, 4]
+
+
+def test_jitter_stays_between_half_and_the_full_delay():
+    options = _options(max_backoff=30)
+    delays = [options.backoff_delay(4) for _ in range(500)]  # 4 seconds without jitter
+    assert all(2 <= delay <= 4 for delay in delays)
+    assert max(delays) - min(delays) > 1  # actually spread out
+
+
+async def test_max_backoff_limits_the_real_wait(server):
+    started = time.monotonic()
+    results = await reqstorm.fetch_all(
+        [server + "/flaky?key=cap&fail=3"], retries=3, backoff=10, max_backoff=0.1
+    )
+    assert results[0].ok and time.monotonic() - started < 1.5
+
+
+async def test_negative_backoff_is_rejected(server):
+    with pytest.raises(ValueError, match="backoff"):
+        await reqstorm.fetch_all([server + "/ok"], max_backoff=-1)