reqstorm 2.3.0__tar.gz → 2.4.1__tar.gz

{reqstorm-2.3.0/reqstorm.egg-info → reqstorm-2.4.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: reqstorm
-Version: 2.3.0
+Version: 2.4.1
 Summary: Send large numbers of HTTP requests concurrently with asyncio, with retries, timeouts and per-request results.
 Author-email: Melih Colpan <colpanmelih@gmail.com>
 License-Expression: MIT
@@ -94,6 +94,7 @@ for error in results.errors():
 - [Quick start](#quick-start)
 - [Results and reports](#results-and-reports)
 - [Rate limits and concurrency](#rate-limits-and-concurrency)
+- [Progress and logging](#progress-and-logging)
 - [Timeouts and retries](#timeouts-and-retries)
 - [Writing results to a file](#writing-results-to-a-file)
 - [Writing results to a database](#writing-results-to-a-database)
@@ -137,6 +138,7 @@ reqstorm does all of that for you, with one call.
 | Typed columns | JSON fields to checked columns, nested paths, arrays to rows |
 | Resume | Skip what already succeeded after an interruption |
 | Planning | `estimate()` before you start, progress with ETA while running |
+| Logging | Retries, pauses and failures; per-run level, file and JSON, independent of the app |
 | API | Blocking (scripts, Jupyter), asyncio and a `reqstorm` command |
 | Safety | TLS verified, timeouts on, bounded concurrency, fully typed |
 
@@ -302,6 +304,41 @@ reqstorm: 3500/7000 (50%)  ok 3493  failed 7
           1.7 req/s  ETA 35m 00s
 ```
 
+## Progress and logging
+
+`progress=True` shows a line on stderr that keeps moving even while every request is waiting, so you can tell a long run is alive:
+
+```text
+reqstorm: 3500/7000 (50%)  ok 3493
+  failed 7  active 12  retries 41
+  1.7 req/s  ETA 34m 10s
+```
+
+It shows requests in flight, retries, the rate over the last minute, the retry round, hosts paused after a 429, and the time left. `progress=` also takes a function, which gets a `ProgressInfo` every two seconds. For a generator, pass `total=` to get a percentage.
+
+**Logs.** reqstorm logs each retry and its reason, pauses, token refreshes and final failures. Turn them on for one run, independent of your application's logging setup:
+
+```python
+reqstorm.fetch_to_file_sync(
+    urls, "out.jsonl",
+    log_level="INFO",    # or DEBUG: every attempt
+    log_file="logs/",    # a new file per run
+    log_format="json",   # optional
+)
+```
+
+```text
+14:32:41 WARNING reqstorm: GET .../items/412
+  failed (HTTP 503) on attempt 1 of 3;
+  retrying in 0.4s
+15:42:18 ERROR   reqstorm: GET .../items/913
+  failed after 3 attempts: HTTP 404
+```
+
+Log files are never overwritten: a directory or `{time}` in the name gives a time-stamped file per run, and an existing `run.log` becomes `run-2.log`. Without `log_level`, reqstorm logs to the standard `"reqstorm"` logger and follows your application's configuration.
+
+**Read results while they are written.** Output files are written at least once a second (`flush_interval`), even while nothing finishes, and never with half a line; SQLite files are opened in WAL mode, so other programs can read them during the run.
+
 ## Timeouts and retries
 
 ```python
@@ -580,7 +617,7 @@ $ reqstorm urls.txt -o shop.db --report \
     --schema products.json --explode items
 ```
 
-`reqstorm --help` lists every option, and the [command line guide](https://reqstorm.github.io/guide/cli/) explains them.
+`-v` / `-vv` log more, `-q` logs only errors, and `--log-file logs/` writes a new log file per run. `reqstorm --help` lists every option, and the [command line guide](https://reqstorm.github.io/guide/cli/) explains them.
 
 ## When to use something else
 

{reqstorm-2.3.0 → reqstorm-2.4.1}/README.md

@@ -49,6 +49,7 @@ for error in results.errors():
 - [Quick start](#quick-start)
 - [Results and reports](#results-and-reports)
 - [Rate limits and concurrency](#rate-limits-and-concurrency)
+- [Progress and logging](#progress-and-logging)
 - [Timeouts and retries](#timeouts-and-retries)
 - [Writing results to a file](#writing-results-to-a-file)
 - [Writing results to a database](#writing-results-to-a-database)
@@ -92,6 +93,7 @@ reqstorm does all of that for you, with one call.
 | Typed columns | JSON fields to checked columns, nested paths, arrays to rows |
 | Resume | Skip what already succeeded after an interruption |
 | Planning | `estimate()` before you start, progress with ETA while running |
+| Logging | Retries, pauses and failures; per-run level, file and JSON, independent of the app |
 | API | Blocking (scripts, Jupyter), asyncio and a `reqstorm` command |
 | Safety | TLS verified, timeouts on, bounded concurrency, fully typed |
 
@@ -257,6 +259,41 @@ reqstorm: 3500/7000 (50%)  ok 3493  failed 7
           1.7 req/s  ETA 35m 00s
 ```
 
+## Progress and logging
+
+`progress=True` shows a line on stderr that keeps moving even while every request is waiting, so you can tell a long run is alive:
+
+```text
+reqstorm: 3500/7000 (50%)  ok 3493
+  failed 7  active 12  retries 41
+  1.7 req/s  ETA 34m 10s
+```
+
+It shows requests in flight, retries, the rate over the last minute, the retry round, hosts paused after a 429, and the time left. `progress=` also takes a function, which gets a `ProgressInfo` every two seconds. For a generator, pass `total=` to get a percentage.
+
+**Logs.** reqstorm logs each retry and its reason, pauses, token refreshes and final failures. Turn them on for one run, independent of your application's logging setup:
+
+```python
+reqstorm.fetch_to_file_sync(
+    urls, "out.jsonl",
+    log_level="INFO",    # or DEBUG: every attempt
+    log_file="logs/",    # a new file per run
+    log_format="json",   # optional
+)
+```
+
+```text
+14:32:41 WARNING reqstorm: GET .../items/412
+  failed (HTTP 503) on attempt 1 of 3;
+  retrying in 0.4s
+15:42:18 ERROR   reqstorm: GET .../items/913
+  failed after 3 attempts: HTTP 404
+```
+
+Log files are never overwritten: a directory or `{time}` in the name gives a time-stamped file per run, and an existing `run.log` becomes `run-2.log`. Without `log_level`, reqstorm logs to the standard `"reqstorm"` logger and follows your application's configuration.
+
+**Read results while they are written.** Output files are written at least once a second (`flush_interval`), even while nothing finishes, and never with half a line; SQLite files are opened in WAL mode, so other programs can read them during the run.
+
 ## Timeouts and retries
 
 ```python
@@ -535,7 +572,7 @@ $ reqstorm urls.txt -o shop.db --report \
     --schema products.json --explode items
 ```
 
-`reqstorm --help` lists every option, and the [command line guide](https://reqstorm.github.io/guide/cli/) explains them.
+`-v` / `-vv` log more, `-q` logs only errors, and `--log-file logs/` writes a new log file per run. `reqstorm --help` lists every option, and the [command line guide](https://reqstorm.github.io/guide/cli/) explains them.
 
 ## When to use something else
 

{reqstorm-2.3.0 → reqstorm-2.4.1}/reqstorm/__init__.py

@@ -26,13 +26,14 @@ from ._client import (
 from ._files import Summary, fetch_to_db, fetch_to_file
 from ._legacy import Reqt
 from ._limits import parse_rate
+from ._observe import ProgressInfo
 from ._paginate import Cursor, LinkHeader, NextLink, PageNumber, Paginator
 from ._plan import Estimate, estimate
 from ._schema import Field, Schema, SchemaError, extract, infer_schema
 from ._sync import fetch_all_sync, fetch_to_db_sync, fetch_to_file_sync, stream_sync
 from ._template import from_template, read_csv, read_sql
 
-__version__ = "2.3.0"
+__version__ = "2.4.1"
 
 __all__ = [
     "Attempt",
@@ -46,6 +47,7 @@ __all__ = [
     "LinkHeader",
     "NextLink",
     "PageNumber",
+    "ProgressInfo",
     "Paginator",
     "Reqt",
     "Request",

{reqstorm-2.3.0 → reqstorm-2.4.1}/reqstorm/_adaptive.py

@@ -6,10 +6,12 @@ import asyncio
 import time
 from dataclasses import dataclass
 from email.utils import parsedate_to_datetime
-from typing import Dict, Mapping, Optional
+from typing import Dict, Mapping, Optional, Tuple
 
 from ._limits import host_key
 
+__all__ = ["AdaptiveRateLimiter", "seconds_until_reset"]
+
 # Header names, most specific first. "RateLimit-*" is the IETF draft; "X-RateLimit-*" is the
 # common convention (GitHub, Twitter/X, many others).
 _REMAINING = ("RateLimit-Remaining", "X-RateLimit-Remaining", "X-Rate-Limit-Remaining")
@@ -91,17 +93,28 @@ class AdaptiveRateLimiter:
         if slot > now:
             await asyncio.sleep(slot - now)
 
-    def record(self, url: str, status: Optional[int], headers: Mapping[str, str]) -> None:
+    def paused(self) -> Dict[str, float]:
+        """Hosts that are paused right now, with the seconds left."""
+        now = time.monotonic()
+        return {
+            host: state.paused_until - now for host, state in self._hosts.items() if state.paused_until > now
+        }
+
+    def record(
+        self, url: str, status: Optional[int], headers: Mapping[str, str]
+    ) -> Optional[Tuple[float, str]]:
+        """Learn from a response. Returns ``(seconds, reason)`` when the host is paused."""
         state = self._state(url)
         if state is None or status is None:
-            return
+            return None
         now = time.monotonic()
         reset = seconds_until_reset(headers)
         if status == 429:
             state.interval = min(max(state.interval * 2, 0.1), 60.0)
             pause = reset if reset is not None else max(state.interval, 1.0)
             state.paused_until = max(state.paused_until, now + pause)
-            return
+            source = "Retry-After" if reset is not None else "no Retry-After"
+            return pause, f"429 Too Many Requests ({source}), spacing requests {state.interval:.2f}s apart"
         remaining_text = _header(headers, _REMAINING)
         if remaining_text is not None and reset is not None:
             try:
@@ -111,8 +124,9 @@ class AdaptiveRateLimiter:
             if remaining is not None:
                 if remaining <= 0:
                     state.paused_until = max(state.paused_until, now + reset)
-                else:
-                    state.interval = min(reset / remaining, 60.0)
-                return
+                    return (reset, "rate limit used up until the window resets") if reset > 0 else None
+                state.interval = min(reset / remaining, 60.0)
+                return None
         if 200 <= status < 400 and state.interval:
             state.interval = state.interval * 0.9 if state.interval > 0.01 else 0.0
+        return None

{reqstorm-2.3.0 → reqstorm-2.4.1}/reqstorm/_cli.py

@@ -19,100 +19,142 @@ from ._schema import Field, Schema, SchemaError, infer_schema
 from ._sync import fetch_all_sync, fetch_to_file_sync, stream_sync
 from ._template import from_template, read_csv
 
-EXAMPLES = """\
+EPILOG = """\
 examples:
   reqstorm urls.txt -o results.jsonl --rate 100/min --retries 2
   reqstorm urls.txt --estimate --rate 100/min
   cat urls.txt | reqstorm --rate auto > results.jsonl
-  reqstorm users.csv --template "https://api.example.com/users/{id}" -o users.db
+  reqstorm urls.txt -o results.db -v --log-file logs/
+  reqstorm users.csv --template "https://api.example.com/users/{id}" -o users.jsonl
   reqstorm urls.txt --paginate next:links.next --schema products.json --explode items -o shop.db
   reqstorm urls.txt --infer-schema 5 --explode items > products.json
+  reqstorm urls.txt --proxy socks5h://127.0.0.1:9050 -o out.jsonl
 
-exit status: 0 when every request succeeded, 1 when some failed or records were rejected,
-2 for invalid arguments.
+rate limits:
+  5 or 0.5 (per second), 10/s, 100/min, 30/5min, 1000/h, 2/day, per host; or auto to
+  follow the server's 429 responses and X-RateLimit headers.
+
+environment:
+  REQSTORM_TOKEN   bearer token, instead of --bearer (keeps it out of shell history)
+
+exit status:
+  0 every request succeeded   1 some failed or records were rejected
+  2 invalid arguments         130 interrupted
+
+documentation: https://reqstorm.github.io/guide/cli/
 """
 
 
+# fmt: off
 def _parser() -> argparse.ArgumentParser:
     parser = argparse.ArgumentParser(
         prog="reqstorm",
         description="Send many HTTP requests with rate limits, retries and progress, and save the results.",
-        epilog=EXAMPLES,
-        formatter_class=argparse.RawDescriptionHelpFormatter,
+        epilog=EPILOG,
+        formatter_class=lambda prog: argparse.RawDescriptionHelpFormatter(prog, max_help_position=32),
     )
-    parser.add_argument("input", nargs="?", default="-",
-                        help="file with one URL per line, or a CSV with --template "
-                        "(default: standard input)")  # fmt: skip
-    parser.add_argument("-o", "--output", help="write results to a .jsonl, .csv or .db/.sqlite file "
-                        "(default: JSON lines on standard output)")  # fmt: skip
-    parser.add_argument("--version", action="version", version=f"reqstorm {__version__}")
+    add = parser.add_argument
+    add("input", nargs="?", default="-",
+        help="file with one URL per line (blank lines and # comments are skipped), or a CSV file "
+        "with --template; default: standard input")
+    add("-o", "--output", metavar="FILE",
+        help="write results to FILE: .jsonl, .csv, or .db/.sqlite for SQLite; "
+        "default: JSON lines on standard output")
+    add("--version", action="version", version=f"reqstorm {__version__}")
 
     request = parser.add_argument_group("requests")
-    request.add_argument("-X", "--method", default="GET")
-    request.add_argument("-H", "--header", action="append", default=[], metavar="'NAME: VALUE'")
-    request.add_argument("--json", help="JSON body sent with every request")
-    request.add_argument("--data", help="raw body sent with every request")
-    request.add_argument("--template", metavar="URL", help="URL template filled from each CSV row, "
-                         "e.g. 'https://api.example.com/users/{id}'")  # fmt: skip
-    request.add_argument("--bearer", metavar="TOKEN", help="send 'Authorization: Bearer TOKEN' "
-                         "(or set REQSTORM_TOKEN)")  # fmt: skip
-    request.add_argument(
-        "--proxy",
-        action="append",
-        default=[],
-        metavar="URL",
-        help="http://, socks5://, socks5h:// or socks4:// proxy URL; repeat to rotate",
-    )
-    request.add_argument("--no-verify", action="store_true", help="do not verify TLS certificates")
-
-    pace = parser.add_argument_group("pace and retries")
-    pace.add_argument("--rate", help="per-host rate limit: 5, 10/s, 100/min, 1000/h, or auto")
-    pace.add_argument("-c", "--concurrency", type=int, default=100)
-    pace.add_argument("--per-host", type=int, default=0, metavar="N", help="max requests in flight per host")
-    pace.add_argument("--timeout", type=float, default=30.0, help="seconds per attempt (default 30)")
-    pace.add_argument("--retries", type=int, default=0)
-    pace.add_argument(
-        "--backoff", type=float, default=0.5, help="seconds before the first retry, doubled after"
-    )
-    pace.add_argument(
-        "--max-backoff", type=float, default=30.0, help="longest wait between retries (default 30)"
-    )
-    pace.add_argument("--no-jitter", action="store_true", help="wait exactly, not a random part of the delay")
-    pace.add_argument("--retry-rounds", type=int, default=0)
-    pace.add_argument("--retry-round-delay", type=float, default=5.0)
-    pace.add_argument("--paginate", metavar="STRATEGY",
-                      help="next:PATH, link, cursor:PATH[:PARAM], or page[:PARAM[:ITEMS_PATH]]")  # fmt: skip
-    pace.add_argument("--max-pages", type=int, default=1000)
-    pace.add_argument(
-        "--cache", metavar="FILE", help="reuse responses from this cache file (revalidated with ETag)"
-    )
-    pace.add_argument(
-        "--cache-ttl", type=float, help="seconds a cached response is used without asking the server"
-    )
+    add = request.add_argument
+    add("-X", "--method", default="GET", help="HTTP method for every request (default: GET)")
+    add("-H", "--header", action="append", default=[], metavar="'NAME: VALUE'",
+        help="header sent with every request; repeat for more")
+    add("--json", metavar="JSON", help="JSON body sent with every request")
+    add("--data", metavar="TEXT", help="raw body sent with every request")
+    add("--template", metavar="URL",
+        help="treat the input as CSV and build one URL per row from {column} placeholders, "
+        "e.g. 'https://api.example.com/users/{id}'")
+    add("--bearer", metavar="TOKEN", help="send 'Authorization: Bearer TOKEN' (or set REQSTORM_TOKEN)")
+    add("--proxy", action="append", default=[], metavar="URL",
+        help="send through a proxy: http://, socks5://, socks5h://, socks4:// or socks4a://; "
+        "repeat to rotate through several")
+    add("--no-verify", action="store_true",
+        help="do not verify TLS certificates (only for hosts you control)")
+
+    pace = parser.add_argument_group("pace")
+    add = pace.add_argument
+    add("--rate", metavar="LIMIT",
+        help="per-host rate limit: 5, 10/s, 100/min, 1000/h, or auto (default: none)")
+    add("-c", "--concurrency", type=int, default=100, metavar="N",
+        help="requests in flight at once (default: 100)")
+    add("--per-host", type=int, default=0, metavar="N",
+        help="requests in flight per host (default: no limit)")
+    add("--timeout", type=float, default=30.0, metavar="SECONDS",
+        help="time allowed for one attempt, including the body (default: 30)")
+
+    retry = parser.add_argument_group("retries")
+    add = retry.add_argument
+    add("--retries", type=int, default=0, metavar="N",
+        help="retry timeouts, connection errors and 429/5xx up to N times (default: 0)")
+    add("--backoff", type=float, default=0.5, metavar="SECONDS",
+        help="wait before the first retry, doubled after each (default: 0.5)")
+    add("--max-backoff", type=float, default=30.0, metavar="SECONDS",
+        help="longest wait between retries (default: 30)")
+    add("--no-jitter", action="store_true", help="wait exactly, instead of a random half to all of the delay")
+    add("--retry-rounds", type=int, default=0, metavar="N",
+        help="send requests that still failed again after all others, up to N rounds (needs -o)")
+    add("--retry-round-delay", type=float, default=5.0, metavar="SECONDS",
+        help="wait before each retry round (default: 5)")
+
+    more = parser.add_argument_group("pagination and caching")
+    add = more.add_argument
+    add("--paginate", metavar="STRATEGY",
+        help="follow pages: next:PATH (URL in the JSON), link (Link header), cursor:PATH[:PARAM], "
+        "or page[:PARAM[:ITEMS_PATH]]")
+    add("--max-pages", type=int, default=1000, metavar="N",
+        help="pages per starting URL at most (default: 1000)")
+    add("--cache", metavar="FILE", help="keep responses in this SQLite file and revalidate them with ETag")
+    add("--cache-ttl", type=float, metavar="SECONDS",
+        help="use a cached response without asking the server for this long (default: always ask)")
 
     output = parser.add_argument_group("output")
-    output.add_argument(
-        "--resume", action="store_true", help="skip requests already saved successfully in --output"
-    )
-    output.add_argument("--ordered", action="store_true", help="write results in input order")
-    output.add_argument("--body", choices=["text", "base64", "none"], default="text")
-    output.add_argument("--include-headers", action="store_true")
-    output.add_argument("--schema", metavar="FILE", help="JSON file mapping columns to fields: "
-                        '{"price": {"path": "pricing.amount", "type": "float"}}')  # fmt: skip
-    output.add_argument("--explode", metavar="PATH", help="with --schema: one row per element of this array")
-    output.add_argument("--table", default="reqstorm_results", help="table name for .db output")
-    output.add_argument("--rejects-table", help="with --schema and .db output: table for rejected records")
-    output.add_argument("--infer-schema", type=int, metavar="N",
-                        help="fetch the first N inputs, print a draft --schema file and exit")  # fmt: skip
-
-    info = parser.add_argument_group("information")
-    info.add_argument("--estimate", action="store_true", help="print how long the batch would take and exit")
-    info.add_argument("--latency", type=float, default=0.5, help="typical response time for --estimate")
-    info.add_argument("-q", "--quiet", action="store_true", help="no progress line")
-    info.add_argument(
-        "--report", action="store_true", help="print response times, statuses and hosts at the end"
-    )
+    add = output.add_argument
+    add("--body", choices=["text", "base64", "none"], default="text",
+        help="how to store response bodies (default: text)")
+    add("--include-headers", action="store_true", help="store response headers too")
+    add("--resume", action="store_true",
+        help="skip requests already saved successfully in -o, append the rest")
+    add("--ordered", action="store_true", help="write results in input order instead of as they finish")
+    add("--table", default="reqstorm_results", metavar="NAME",
+        help="table for .db output (default: reqstorm_results)")
+    add("--flush-interval", type=float, default=1.0, metavar="SECONDS",
+        help="write waiting results to -o at least this often (default: 1)")
+
+    data = parser.add_argument_group("typed columns")
+    add = data.add_argument
+    add("--schema", metavar="FILE",
+        help="JSON file mapping columns to fields, e.g. {\"price\": {\"path\": \"pricing.amount\", "
+        "\"type\": \"float\"}}; writes typed rows instead of raw responses (needs -o)")
+    add("--explode", metavar="PATH", help="with --schema: one row per element of the array at PATH")
+    add("--rejects-table", metavar="NAME",
+        help="with --schema and .db output: also store rejected records here")
+    add("--infer-schema", type=int, metavar="N",
+        help="fetch the first N inputs, print a draft --schema file, exit")
+
+    info = parser.add_argument_group("progress and logging")
+    add = info.add_argument
+    add("-v", "--verbose", action="count", default=0,
+        help="log more: -v run events (INFO), -vv every attempt (DEBUG); "
+        "warnings and errors are always logged")
+    add("-q", "--quiet", action="store_true", help="no progress line, log errors only")
+    add("--log-file", metavar="PATH",
+        help="log to a file instead of stderr; a directory or a name with {time} gives a time-stamped "
+        "file per run, and an existing file is never overwritten")
+    add("--log-json", action="store_true", help="log one JSON object per line")
+    add("--report", action="store_true", help="print response times, statuses and hosts at the end")
+    add("--estimate", action="store_true", help="print how long the batch would take, send nothing, exit")
+    add("--latency", type=float, default=0.5, metavar="SECONDS",
+        help="typical response time assumed by --estimate (default: 0.5)")
     return parser
+# fmt: on
 
 
 def _paginator(text: Optional[str], max_pages: int) -> Optional[Paginator]:
@@ -293,17 +335,23 @@ def _batch(args: argparse.Namespace, cache: Optional[Cache]) -> int:
         return 0
 
     progress: Any = False if args.quiet else True
+    options["total"] = _count(args) if not args.paginate else None
+    options["log_level"] = "ERROR" if args.quiet else ("WARNING", "INFO", "DEBUG")[min(args.verbose, 2)]
+    options["log_file"] = args.log_file
+    options["log_format"] = "json" if args.log_json else "text"
     if args.output:
         summary = fetch_to_file_sync(
             _inputs(args), args.output, body=args.body, include_headers=args.include_headers,
             resume=args.resume, ordered=args.ordered, progress=progress, retry_rounds=args.retry_rounds,
             retry_round_delay=args.retry_round_delay, table=args.table, schema=schema,
-            rejects_table=args.rejects_table, **options,
+            rejects_table=args.rejects_table, flush_interval=args.flush_interval, **options,
         )  # fmt: skip
         message = f"reqstorm: {summary.ok} ok, {summary.failed} failed, {summary.skipped} skipped"
         if schema is not None:
             message += f", {summary.rows} rows, {summary.rejected} rejected"
         print(message + f" -> {args.output}", file=sys.stderr)
+        if summary.log_file:
+            print(f"reqstorm: log written to {summary.log_file}", file=sys.stderr)
         if args.report:
             print(json.dumps(summary.report, indent=2), file=sys.stderr)
         return 0 if summary.failed == 0 and summary.rejected == 0 else 1
@@ -314,7 +362,7 @@ def _batch(args: argparse.Namespace, cache: Optional[Cache]) -> int:
 
     report = Report()
     failed = 0
-    for result in stream_sync(_inputs(args), **options):
+    for result in stream_sync(_inputs(args), progress=progress, **options):
         report.add(result)
         failed += 0 if result.ok else 1
         sys.stdout.write(json.dumps(result.to_dict(body=args.body, include_headers=args.include_headers),