reqstorm 2.1.0__tar.gz → 2.2.0__tar.gz

{reqstorm-2.1.0/reqstorm.egg-info → reqstorm-2.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: reqstorm
-Version: 2.1.0
+Version: 2.2.0
 Summary: Send large numbers of HTTP requests concurrently with asyncio, with retries, timeouts and per-request results.
 Author-email: Melih Colpan <colpanmelih@gmail.com>
 License-Expression: MIT
@@ -26,13 +26,17 @@ Requires-Python: >=3.9
 Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: aiohttp<4,>=3.9
+Provides-Extra: pydantic
+Requires-Dist: pydantic>=2; extra == "pydantic"
 Provides-Extra: test
 Requires-Dist: pytest>=8; extra == "test"
 Requires-Dist: pytest-asyncio>=0.23; extra == "test"
 Requires-Dist: trustme>=1.1; extra == "test"
+Requires-Dist: pydantic>=2; extra == "test"
 Provides-Extra: lint
 Requires-Dist: ruff>=0.6; extra == "lint"
 Requires-Dist: mypy>=1.10; extra == "lint"
+Requires-Dist: pydantic>=2; extra == "lint"
 Dynamic: license-file
 
 # reqstorm
@@ -72,7 +76,8 @@ results = reqstorm.fetch_all_sync(
 
 print(results.summary())
 # {'total': 7000, 'ok': 6987, 'failed': 13,
-#  'failures': {'HTTP 404': 9, 'TimeoutError': 4}}
+#  'failures': {'HTTP 404': 9, 'TimeoutError': 4},
+#  'latency': {'p50': 0.21, 'p95': 0.73, ...}, ...}
 
 for error in results.errors():
     print(error["url"], error["error"] or error["status"])
@@ -89,9 +94,13 @@ for error in results.errors():
 - [Writing results to a file](#writing-results-to-a-file)
 - [Writing results to a database](#writing-results-to-a-database)
 - [Structured data: JSON to typed columns](#structured-data-json-to-typed-columns)
+- [Pagination](#pagination)
+- [Requests from a CSV file or a table](#requests-from-a-csv-file-or-a-table)
+- [Tokens, proxies and caching](#tokens-proxies-and-caching)
 - [Requests, headers and bodies](#requests-headers-and-bodies)
 - [Streaming results](#streaming-results)
 - [TLS and sessions](#tls-and-sessions)
+- [Command line](#command-line)
 - [When to use something else](#when-to-use-something-else)
 - [Coming from reqt](#coming-from-reqt)
 - [Development](#development)
@@ -112,15 +121,19 @@ reqstorm does all of that for you, with one call.
 | Feature | What you get |
 |---|---|
 | One result per request | Failures are recorded, never raised; every attempt is kept |
-| Rate limits | Per host, in any unit: `"10/s"`, `"100/min"`, `"1000/h"` |
+| Rate limits | Per host, in any unit (`"100/min"`), or `"auto"` from the server's 429s and headers |
 | Concurrency | Overall and per host |
 | Retries | Immediate, with backoff and `Retry-After`; plus end-of-run rounds |
-| Reports | `summary()`, `errors()`, `to_dicts()` |
+| Reports | Failures by reason, p50/p95/p99 response times, per-host figures |
+| Pagination | Next links, `Link` headers, cursors and page numbers |
+| Requests from data | URL templates over CSV rows or SQL query results |
+| Tokens and proxies | Refresh an expired token on 401; rotate through proxies |
+| Caching | ETag / `If-None-Match`: unchanged resources cost a 304 |
 | Output | JSONL, CSV, SQLite, PostgreSQL, MySQL; ordered or as completed |
 | Typed columns | JSON fields to checked columns, nested paths, arrays to rows |
 | Resume | Skip what already succeeded after an interruption |
 | Planning | `estimate()` before you start, progress with ETA while running |
-| API | Blocking (scripts, Jupyter) and asyncio |
+| API | Blocking (scripts, Jupyter), asyncio and a `reqstorm` command |
 | Safety | TLS verified, timeouts on, bounded concurrency, fully typed |
 
 ## Installation
@@ -129,7 +142,7 @@ reqstorm does all of that for you, with one call.
 $ python -m pip install reqstorm
 ```
 
-reqstorm supports Python 3.9 to 3.13 on Linux, macOS and Windows. Its only dependency is [aiohttp](https://docs.aiohttp.org). For PostgreSQL or MySQL output, install the driver you already use (`psycopg`, `psycopg2`, `pymysql`, `mysqlclient` or `mysql-connector-python`).
+reqstorm supports Python 3.9 to 3.13 on Linux, macOS and Windows. Its only dependency is [aiohttp](https://docs.aiohttp.org). For PostgreSQL or MySQL output, install the driver you already use (`psycopg`, `psycopg2`, `pymysql`, `mysqlclient` or `mysql-connector-python`). To use Pydantic models as schemas, install `reqstorm[pydantic]`.
 
 ## Quick start
 
@@ -205,6 +218,17 @@ results.failed       # failed results
 results.summary()    # counts, failures grouped by reason
 results.errors()     # failed requests as plain dicts
 results.to_dicts()   # every result as a dict
+results.report()     # response times, statuses, hosts
+```
+
+```python
+>>> results.report()["latency"]
+{'min': 0.081, 'p50': 0.214, 'p90': 0.502,
+ 'p95': 0.733, 'p99': 1.902, 'max': 10.004,
+ 'mean': 0.297}
+>>> results.report()["hosts"]["api.example.com:443"]
+{'requests': 7000, 'ok': 6987, 'failed': 13,
+ 'latency': {...}}
 ```
 
 `errors()` returns plain data, ready for a log file or a DataFrame:
@@ -255,6 +279,8 @@ results = reqstorm.fetch_all_sync(
 
 The limit applies to each host (`host:port`) separately and counts retries too, so requests to different APIs never slow each other down. Requests to one host are spaced evenly.
 
+**Don't know the limit?** `rate_limit="auto"` learns it from the server. A `429 Too Many Requests` pauses the host for `Retry-After` and slows it down; `X-RateLimit-Remaining` and `X-RateLimit-Reset` spread the remaining requests over the window; the rate recovers when the server stops pushing back. 429 responses are retried without using up `retries`.
+
 **Plan before you send.** `estimate` predicts the duration and names the bottleneck, without sending anything:
 
 ```python
@@ -395,8 +421,85 @@ print(summary.rows, summary.rejected)
 
 The same schema works for `.db`, `.jsonl` and `.csv` files, and `reqstorm.extract(results, schema)` returns the rows as Python lists. `reqstorm.infer_schema(samples)` drafts a schema from a few responses for you to review.
 
+**Already have a Pydantic model?** Pass it as the schema; its fields become columns and Pydantic validates each record:
+
+```python
+class Product(BaseModel):
+    id: int = Field(
+        json_schema_extra={"key": True})
+    name: str
+    price: Optional[float] = Field(
+        None,
+        json_schema_extra={"path": "pricing.amount"})
+
+reqstorm.fetch_to_file_sync(
+    urls, "shop.db", schema=Product, explode="items"
+)
+```
+
 More in the [structured data guide](https://reqstorm.github.io/guide/structured-data/).
 
+## Pagination
+
+`paginate=` follows each starting URL through all of its pages, concurrently with the other URLs and under the same rate limits:
+
+```python
+results = reqstorm.fetch_all_sync(
+    ["https://api.example.com/products"],
+    paginate=reqstorm.NextLink("links.next"),
+)
+```
+
+| Strategy | Next page comes from |
+|---|---|
+| `NextLink("links.next")` | a URL in the JSON body |
+| `LinkHeader()` | the `Link` header (GitHub style) |
+| `Cursor("meta.next", param="cursor")` | a cursor in the body |
+| `PageNumber("page", items="data")` | `?page=2, 3, ...` until empty |
+
+Each result has `page` and `seed_index` (its starting URL). Combined with a schema and `explode`, every page of a catalogue becomes typed rows in one call. `max_pages` (1000 by default) stops an API that never ends.
+
+## Requests from a CSV file or a table
+
+`from_template` makes one request per row. Values are percent-encoded, and rows are read lazily:
+
+```python
+rows = reqstorm.read_csv("users.csv")
+requests = reqstorm.from_template(
+    "https://api.example.com/users/{id}",
+    rows,
+    params={"country": "{country}"},
+)
+reqstorm.fetch_to_file_sync(requests, "users.jsonl")
+```
+
+`read_sql(connection, query)` reads rows from any database connection instead, and `json=` builds a request body per row.
+
+## Tokens, proxies and caching
+
+**Tokens that expire.** `BearerAuth` gets a new token when a response is 401 and sends the request again. Concurrent 401s share one refresh:
+
+```python
+auth = reqstorm.BearerAuth(refresh=get_token)
+reqstorm.fetch_all_sync(urls, auth=auth)
+```
+
+**Proxies.** One proxy, a pool used in turn, or one per `Request`:
+
+```python
+reqstorm.fetch_all_sync(urls, proxy=[
+    "http://proxy-1.example.com:8080",
+    "http://proxy-2.example.com:8080",
+])
+```
+
+**Caching.** `Cache` keeps responses in an SQLite file. The next run asks the server with `If-None-Match`; an unchanged resource comes back as a `304` with no body, and the stored response is used. With `ttl`, recent responses skip the network entirely:
+
+```python
+with reqstorm.Cache("responses.sqlite") as cache:
+    reqstorm.fetch_all_sync(urls, cache=cache)
+```
+
 ## Requests, headers and bodies
 
 Options given to `fetch_all` apply to every request; `reqstorm.Request` varies them per request:
@@ -452,6 +555,24 @@ results = reqstorm.fetch_all_sync(urls, ssl=context)
 
 In async code, `session=` takes an existing `aiohttp.ClientSession` to share cookies, connection pools or proxy settings. reqstorm does not close it.
 
+## Command line
+
+The `reqstorm` command runs a batch without any Python code:
+
+```console
+$ reqstorm urls.txt -o results.jsonl \
+    --rate 100/min --retries 2
+$ reqstorm urls.txt --estimate --rate 100/min
+$ cat urls.txt | reqstorm --rate auto -q > out.jsonl
+$ reqstorm users.csv -o users.db \
+    --template "https://api.example.com/users/{id}"
+$ reqstorm urls.txt -o shop.db --report \
+    --paginate next:links.next \
+    --schema products.json --explode items
+```
+
+`reqstorm --help` lists every option, and the [command line guide](https://reqstorm.github.io/guide/cli/) explains them.
+
 ## When to use something else
 
 reqstorm is built for batches. For other jobs, these are better fits:

{reqstorm-2.1.0 → reqstorm-2.2.0}/README.md

@@ -35,7 +35,8 @@ results = reqstorm.fetch_all_sync(
 
 print(results.summary())
 # {'total': 7000, 'ok': 6987, 'failed': 13,
-#  'failures': {'HTTP 404': 9, 'TimeoutError': 4}}
+#  'failures': {'HTTP 404': 9, 'TimeoutError': 4},
+#  'latency': {'p50': 0.21, 'p95': 0.73, ...}, ...}
 
 for error in results.errors():
     print(error["url"], error["error"] or error["status"])
@@ -52,9 +53,13 @@ for error in results.errors():
 - [Writing results to a file](#writing-results-to-a-file)
 - [Writing results to a database](#writing-results-to-a-database)
 - [Structured data: JSON to typed columns](#structured-data-json-to-typed-columns)
+- [Pagination](#pagination)
+- [Requests from a CSV file or a table](#requests-from-a-csv-file-or-a-table)
+- [Tokens, proxies and caching](#tokens-proxies-and-caching)
 - [Requests, headers and bodies](#requests-headers-and-bodies)
 - [Streaming results](#streaming-results)
 - [TLS and sessions](#tls-and-sessions)
+- [Command line](#command-line)
 - [When to use something else](#when-to-use-something-else)
 - [Coming from reqt](#coming-from-reqt)
 - [Development](#development)
@@ -75,15 +80,19 @@ reqstorm does all of that for you, with one call.
 | Feature | What you get |
 |---|---|
 | One result per request | Failures are recorded, never raised; every attempt is kept |
-| Rate limits | Per host, in any unit: `"10/s"`, `"100/min"`, `"1000/h"` |
+| Rate limits | Per host, in any unit (`"100/min"`), or `"auto"` from the server's 429s and headers |
 | Concurrency | Overall and per host |
 | Retries | Immediate, with backoff and `Retry-After`; plus end-of-run rounds |
-| Reports | `summary()`, `errors()`, `to_dicts()` |
+| Reports | Failures by reason, p50/p95/p99 response times, per-host figures |
+| Pagination | Next links, `Link` headers, cursors and page numbers |
+| Requests from data | URL templates over CSV rows or SQL query results |
+| Tokens and proxies | Refresh an expired token on 401; rotate through proxies |
+| Caching | ETag / `If-None-Match`: unchanged resources cost a 304 |
 | Output | JSONL, CSV, SQLite, PostgreSQL, MySQL; ordered or as completed |
 | Typed columns | JSON fields to checked columns, nested paths, arrays to rows |
 | Resume | Skip what already succeeded after an interruption |
 | Planning | `estimate()` before you start, progress with ETA while running |
-| API | Blocking (scripts, Jupyter) and asyncio |
+| API | Blocking (scripts, Jupyter), asyncio and a `reqstorm` command |
 | Safety | TLS verified, timeouts on, bounded concurrency, fully typed |
 
 ## Installation
@@ -92,7 +101,7 @@ reqstorm does all of that for you, with one call.
 $ python -m pip install reqstorm
 ```
 
-reqstorm supports Python 3.9 to 3.13 on Linux, macOS and Windows. Its only dependency is [aiohttp](https://docs.aiohttp.org). For PostgreSQL or MySQL output, install the driver you already use (`psycopg`, `psycopg2`, `pymysql`, `mysqlclient` or `mysql-connector-python`).
+reqstorm supports Python 3.9 to 3.13 on Linux, macOS and Windows. Its only dependency is [aiohttp](https://docs.aiohttp.org). For PostgreSQL or MySQL output, install the driver you already use (`psycopg`, `psycopg2`, `pymysql`, `mysqlclient` or `mysql-connector-python`). To use Pydantic models as schemas, install `reqstorm[pydantic]`.
 
 ## Quick start
 
@@ -168,6 +177,17 @@ results.failed       # failed results
 results.summary()    # counts, failures grouped by reason
 results.errors()     # failed requests as plain dicts
 results.to_dicts()   # every result as a dict
+results.report()     # response times, statuses, hosts
+```
+
+```python
+>>> results.report()["latency"]
+{'min': 0.081, 'p50': 0.214, 'p90': 0.502,
+ 'p95': 0.733, 'p99': 1.902, 'max': 10.004,
+ 'mean': 0.297}
+>>> results.report()["hosts"]["api.example.com:443"]
+{'requests': 7000, 'ok': 6987, 'failed': 13,
+ 'latency': {...}}
 ```
 
 `errors()` returns plain data, ready for a log file or a DataFrame:
@@ -218,6 +238,8 @@ results = reqstorm.fetch_all_sync(
 
 The limit applies to each host (`host:port`) separately and counts retries too, so requests to different APIs never slow each other down. Requests to one host are spaced evenly.
 
+**Don't know the limit?** `rate_limit="auto"` learns it from the server. A `429 Too Many Requests` pauses the host for `Retry-After` and slows it down; `X-RateLimit-Remaining` and `X-RateLimit-Reset` spread the remaining requests over the window; the rate recovers when the server stops pushing back. 429 responses are retried without using up `retries`.
+
 **Plan before you send.** `estimate` predicts the duration and names the bottleneck, without sending anything:
 
 ```python
@@ -358,8 +380,85 @@ print(summary.rows, summary.rejected)
 
 The same schema works for `.db`, `.jsonl` and `.csv` files, and `reqstorm.extract(results, schema)` returns the rows as Python lists. `reqstorm.infer_schema(samples)` drafts a schema from a few responses for you to review.
 
+**Already have a Pydantic model?** Pass it as the schema; its fields become columns and Pydantic validates each record:
+
+```python
+class Product(BaseModel):
+    id: int = Field(
+        json_schema_extra={"key": True})
+    name: str
+    price: Optional[float] = Field(
+        None,
+        json_schema_extra={"path": "pricing.amount"})
+
+reqstorm.fetch_to_file_sync(
+    urls, "shop.db", schema=Product, explode="items"
+)
+```
+
 More in the [structured data guide](https://reqstorm.github.io/guide/structured-data/).
 
+## Pagination
+
+`paginate=` follows each starting URL through all of its pages, concurrently with the other URLs and under the same rate limits:
+
+```python
+results = reqstorm.fetch_all_sync(
+    ["https://api.example.com/products"],
+    paginate=reqstorm.NextLink("links.next"),
+)
+```
+
+| Strategy | Next page comes from |
+|---|---|
+| `NextLink("links.next")` | a URL in the JSON body |
+| `LinkHeader()` | the `Link` header (GitHub style) |
+| `Cursor("meta.next", param="cursor")` | a cursor in the body |
+| `PageNumber("page", items="data")` | `?page=2, 3, ...` until empty |
+
+Each result has `page` and `seed_index` (its starting URL). Combined with a schema and `explode`, every page of a catalogue becomes typed rows in one call. `max_pages` (1000 by default) stops an API that never ends.
+
+## Requests from a CSV file or a table
+
+`from_template` makes one request per row. Values are percent-encoded, and rows are read lazily:
+
+```python
+rows = reqstorm.read_csv("users.csv")
+requests = reqstorm.from_template(
+    "https://api.example.com/users/{id}",
+    rows,
+    params={"country": "{country}"},
+)
+reqstorm.fetch_to_file_sync(requests, "users.jsonl")
+```
+
+`read_sql(connection, query)` reads rows from any database connection instead, and `json=` builds a request body per row.
+
+## Tokens, proxies and caching
+
+**Tokens that expire.** `BearerAuth` gets a new token when a response is 401 and sends the request again. Concurrent 401s share one refresh:
+
+```python
+auth = reqstorm.BearerAuth(refresh=get_token)
+reqstorm.fetch_all_sync(urls, auth=auth)
+```
+
+**Proxies.** One proxy, a pool used in turn, or one per `Request`:
+
+```python
+reqstorm.fetch_all_sync(urls, proxy=[
+    "http://proxy-1.example.com:8080",
+    "http://proxy-2.example.com:8080",
+])
+```
+
+**Caching.** `Cache` keeps responses in an SQLite file. The next run asks the server with `If-None-Match`; an unchanged resource comes back as a `304` with no body, and the stored response is used. With `ttl`, recent responses skip the network entirely:
+
+```python
+with reqstorm.Cache("responses.sqlite") as cache:
+    reqstorm.fetch_all_sync(urls, cache=cache)
+```
+
 ## Requests, headers and bodies
 
 Options given to `fetch_all` apply to every request; `reqstorm.Request` varies them per request:
@@ -415,6 +514,24 @@ results = reqstorm.fetch_all_sync(urls, ssl=context)
 
 In async code, `session=` takes an existing `aiohttp.ClientSession` to share cookies, connection pools or proxy settings. reqstorm does not close it.
 
+## Command line
+
+The `reqstorm` command runs a batch without any Python code:
+
+```console
+$ reqstorm urls.txt -o results.jsonl \
+    --rate 100/min --retries 2
+$ reqstorm urls.txt --estimate --rate 100/min
+$ cat urls.txt | reqstorm --rate auto -q > out.jsonl
+$ reqstorm users.csv -o users.db \
+    --template "https://api.example.com/users/{id}"
+$ reqstorm urls.txt -o shop.db --report \
+    --paginate next:links.next \
+    --schema products.json --explode items
+```
+
+`reqstorm --help` lists every option, and the [command line guide](https://reqstorm.github.io/guide/cli/) explains them.
+
 ## When to use something else
 
 reqstorm is built for batches. For other jobs, these are better fits:

{reqstorm-2.1.0 → reqstorm-2.2.0}/pyproject.toml

@@ -30,8 +30,12 @@ classifiers = [
 ]
 
 [project.optional-dependencies]
-test = ["pytest>=8", "pytest-asyncio>=0.23", "trustme>=1.1"]
-lint = ["ruff>=0.6", "mypy>=1.10"]
+pydantic = ["pydantic>=2"]
+test = ["pytest>=8", "pytest-asyncio>=0.23", "trustme>=1.1", "pydantic>=2"]
+lint = ["ruff>=0.6", "mypy>=1.10", "pydantic>=2"]
+
+[project.scripts]
+reqstorm = "reqstorm._cli:main"
 
 [project.urls]
 Homepage = "https://reqstorm.github.io"

{reqstorm-2.1.0 → reqstorm-2.2.0}/reqstorm/__init__.py

@@ -11,6 +11,8 @@ async def main():
 asyncio.run(main())
 """
 
+from ._auth import BearerAuth
+from ._cache import Cache
 from ._client import (
     DEFAULT_RETRY_STATUSES,
     Attempt,
@@ -24,20 +26,29 @@ from ._client import (
 from ._files import Summary, fetch_to_db, fetch_to_file
 from ._legacy import Reqt
 from ._limits import parse_rate
+from ._paginate import Cursor, LinkHeader, NextLink, PageNumber, Paginator
 from ._plan import Estimate, estimate
 from ._schema import Field, Schema, SchemaError, extract, infer_schema
 from ._sync import fetch_all_sync, fetch_to_db_sync, fetch_to_file_sync, stream_sync
+from ._template import from_template, read_csv, read_sql
 
-__version__ = "2.1.0"
+__version__ = "2.2.0"
 
 __all__ = [
-    "DEFAULT_RETRY_STATUSES",
     "Attempt",
+    "BearerAuth",
+    "Cache",
+    "Cursor",
+    "DEFAULT_RETRY_STATUSES",
     "Estimate",
     "Field",
     "HTTPStatusError",
-    "Request",
+    "LinkHeader",
+    "NextLink",
+    "PageNumber",
+    "Paginator",
     "Reqt",
+    "Request",
     "Result",
     "Results",
     "Schema",
@@ -51,9 +62,12 @@ __all__ = [
     "fetch_to_db_sync",
     "fetch_to_file",
     "fetch_to_file_sync",
+    "from_template",
     "infer_schema",
-    "stream",
     "parse_rate",
+    "read_csv",
+    "read_sql",
+    "stream",
     "stream_sync",
     "__version__",
 ]

reqstorm-2.2.0/reqstorm/__main__.py

@@ -0,0 +1,5 @@
+import sys
+
+from ._cli import main
+
+sys.exit(main())

reqstorm-2.2.0/reqstorm/_adaptive.py

@@ -0,0 +1,118 @@
+"""A per-host rate limiter that learns the rate from the server's responses."""
+
+from __future__ import annotations
+
+import asyncio
+import time
+from dataclasses import dataclass
+from email.utils import parsedate_to_datetime
+from typing import Dict, Mapping, Optional
+
+from ._limits import host_key
+
+# Header names, most specific first. "RateLimit-*" is the IETF draft; "X-RateLimit-*" is the
+# common convention (GitHub, Twitter/X, many others).
+_REMAINING = ("RateLimit-Remaining", "X-RateLimit-Remaining", "X-Rate-Limit-Remaining")
+_RESET = ("RateLimit-Reset", "X-RateLimit-Reset", "X-Rate-Limit-Reset")
+_MAX_PAUSE = 300.0
+
+
+def _header(headers: Mapping[str, str], names: tuple) -> Optional[str]:
+    for name in names:
+        value = headers.get(name)
+        if value is not None:
+            return value
+    return None
+
+
+def seconds_until_reset(headers: Mapping[str, str], now: Optional[float] = None) -> Optional[float]:
+    """Seconds until the server's rate-limit window resets, from ``*-Reset`` or ``Retry-After``.
+
+    Reset values above 10^9 are Unix timestamps (GitHub style); smaller ones are seconds
+    (IETF draft style). ``Retry-After`` may also be an HTTP date.
+    """
+    now = time.time() if now is None else now
+    for value in (_header(headers, _RESET), headers.get("Retry-After")):
+        if value is None:
+            continue
+        value = value.strip()
+        try:
+            number = float(value)
+        except ValueError:
+            try:
+                return max(parsedate_to_datetime(value).timestamp() - now, 0.0)
+            except (TypeError, ValueError, IndexError, OverflowError):
+                continue
+        seconds = number - now if number > 1e9 else number
+        return min(max(seconds, 0.0), _MAX_PAUSE)
+    return None
+
+
+@dataclass
+class _HostState:
+    interval: float = 0.0  # seconds between requests; 0 means no spacing
+    next_slot: float = 0.0
+    paused_until: float = 0.0
+
+
+class AdaptiveRateLimiter:
+    """``rate_limit="auto"``: no limit until the server signals one, then follow it.
+
+    - A 429 response doubles the spacing between requests to that host and pauses it for
+      ``Retry-After`` (or until the rate-limit window resets).
+    - ``*-RateLimit-Remaining`` / ``*-RateLimit-Reset`` headers spread the remaining requests
+      evenly over the rest of the window, and pause the host when none are left.
+    - Every successful response without such headers shortens the spacing by 10 %, so the
+      rate recovers once the server stops pushing back.
+    """
+
+    adaptive = True
+
+    def __init__(self) -> None:
+        self._hosts: Dict[str, _HostState] = {}
+
+    def _state(self, url: str) -> Optional[_HostState]:
+        host = host_key(url)
+        if host is None:
+            return None
+        return self._hosts.setdefault(host, _HostState())
+
+    def interval(self, url: str) -> float:
+        state = self._state(url)
+        return state.interval if state else 0.0
+
+    async def wait(self, url: str) -> None:
+        state = self._state(url)
+        if state is None:
+            return
+        now = time.monotonic()
+        slot = max(now, state.next_slot, state.paused_until)
+        state.next_slot = slot + state.interval
+        if slot > now:
+            await asyncio.sleep(slot - now)
+
+    def record(self, url: str, status: Optional[int], headers: Mapping[str, str]) -> None:
+        state = self._state(url)
+        if state is None or status is None:
+            return
+        now = time.monotonic()
+        reset = seconds_until_reset(headers)
+        if status == 429:
+            state.interval = min(max(state.interval * 2, 0.1), 60.0)
+            pause = reset if reset is not None else max(state.interval, 1.0)
+            state.paused_until = max(state.paused_until, now + pause)
+            return
+        remaining_text = _header(headers, _REMAINING)
+        if remaining_text is not None and reset is not None:
+            try:
+                remaining = float(remaining_text)
+            except ValueError:
+                remaining = None
+            if remaining is not None:
+                if remaining <= 0:
+                    state.paused_until = max(state.paused_until, now + reset)
+                else:
+                    state.interval = min(reset / remaining, 60.0)
+                return
+        if 200 <= status < 400 and state.interval:
+            state.interval = state.interval * 0.9 if state.interval > 0.01 else 0.0

reqstorm-2.2.0/reqstorm/_auth.py

@@ -0,0 +1,79 @@
+"""Authentication that refreshes an expired token during a long run."""
+
+from __future__ import annotations
+
+import asyncio
+import inspect
+from typing import Any, Awaitable, Callable, Dict, Optional, Union
+
+__all__ = ["BearerAuth"]
+
+TokenSource = Callable[[], Union[str, Awaitable[str]]]
+
+
+class BearerAuth:
+    """Send ``Authorization: Bearer <token>``, and get a new token when the server says it expired.
+
+        auth = reqstorm.BearerAuth(refresh=get_token)
+        reqstorm.fetch_all_sync(urls, auth=auth)
+
+    Args:
+        token: The current token. Leave it out to fetch one with ``refresh`` before the
+            first request.
+        refresh: A function or coroutine function that returns a new token. It is called
+            when a response is 401 Unauthorized; the request is then sent again once with
+            the new token. Concurrent 401s share a single refresh.
+        header: Header to send the token in (default ``Authorization``).
+        scheme: Prefix before the token (default ``Bearer``); ``""`` sends the token alone.
+    """
+
+    def __init__(
+        self,
+        token: Optional[str] = None,
+        *,
+        refresh: Optional[TokenSource] = None,
+        header: str = "Authorization",
+        scheme: str = "Bearer",
+    ) -> None:
+        if token is None and refresh is None:
+            raise ValueError("BearerAuth needs a token, a refresh function, or both")
+        self.token = token
+        self._refresh = refresh
+        self.header = header
+        self.scheme = scheme
+        self.refreshes = 0
+        self._generation = 0
+        self._lock: Optional[asyncio.Lock] = None
+
+    @property
+    def can_refresh(self) -> bool:
+        return self._refresh is not None
+
+    async def headers(self) -> Dict[str, str]:
+        if self.token is None:
+            await self.refresh(self._generation)
+        value = f"{self.scheme} {self.token}" if self.scheme else str(self.token)
+        return {self.header: value}
+
+    @property
+    def generation(self) -> int:
+        """Changes each time the token is refreshed."""
+        return self._generation
+
+    async def refresh(self, seen_generation: int) -> None:
+        """Get a new token, unless another request already did since ``seen_generation``."""
+        if self._refresh is None:
+            return
+        if self._lock is None:
+            self._lock = asyncio.Lock()
+        async with self._lock:
+            if self._generation != seen_generation and self.token is not None:
+                return  # refreshed by a concurrent request while we waited
+            token: Any = self._refresh()
+            if inspect.isawaitable(token):
+                token = await token
+            if not isinstance(token, str) or not token:
+                raise ValueError("the refresh function must return a non-empty token string")
+            self.token = token
+            self._generation += 1
+            self.refreshes += 1