goodput-http 0.1.0__py3-none-any.whl

goodput/CLAUDE.md

@@ -0,0 +1,115 @@
+# CLAUDE.md — src/goodput
+
+Module-level guidance for the library source. Read the root `CLAUDE.md` first for
+the invariants and toolchain; this file maps the modules and their contracts.
+
+## Dependency direction
+
+Keep imports flowing one way to avoid cycles. Rough layers (low → high):
+
+```
+exceptions, clock, redaction, stats, models, control_mode   (leaf primitives)
+        │
+retry_after, classify, ratelimit, circuit, retry, scope, events
+        │
+routing/ (route → health → selector), control/ (base → aimd/gradient)
+        │
+transport, sources, report, sinks/, checkpoint, plugins
+        │
+engine  ──►  api  ──►  __init__ (public surface),  cli
+        │
+sim  (test/simulation transport; depends on transport+routing, used by tests)
+```
+
+`engine.py` is the only module that wires everything together. Do not import
+`engine` from lower layers. `config.py` sits beside the leaf layer (it depends on
+`control_mode`, `circuit`, `retry`) and is imported widely.
+
+## Module responsibilities (contracts)
+
+- **`control_mode.py`** — the authority primitive `Controlled[T]` and the
+  `fixed/adaptive/bounded_adaptive/disabled/inherit` constructors. **This is the
+  most important file in the project.** `Controlled` is a frozen dataclass;
+  "changing" a value calls `.propose()` which returns `(new, accepted, reason)`
+  and *refuses* to mutate FIXED/DISABLED/INHERIT. `.as_float()` widens int knobs
+  for controllers. Never add a mutation path that bypasses `.propose()`.
+- **`models.py`** — `Request` (immutable template), `LogicalRequest`,
+  `AttemptRecord`, `LogicalResult`, `Outcome`, `ErrorClass`, `ThrottleSignal`,
+  `Classification`. `LogicalResult.retry_count == attempt_count - 1`. Idempotency
+  is resolved by `Request.is_idempotent()` (explicit override, else method).
+- **`config.py`** — `EngineConfig` (pydantic v2, `extra="forbid"`),
+  `Objective`/`Mode`/`ThrottleScopeMode` enums, `SafetyCeilings`,
+  `AuthorizationContext`. `validate_constraints()` returns a `ValidationPlan`
+  with pre-execution conflict detection and a `render()` dry-run report. Add new
+  contradiction checks here, not in the engine.
+- **`classify.py`** — `Classifiers` bundle (success/throttle/error) over a
+  transport-agnostic `ResponseView`. Defaults: 2xx success, 429/Retry-After/
+  exhausted-RateLimit throttle, 5xx retryable, 4xx not. Custom classifiers plug
+  in via `Classifiers.build(...)`.
+- **`retry.py` / `retry_after.py`** — `decide_retry()` is pure and takes injected
+  `rng` (no internal randomness — the engine owns the RNG for determinism).
+  `RetryBudget` bounds amplification. `retry_after.py` parsers are total (never
+  raise; return `None` on malformed input).
+- **`ratelimit.py`** — `TokenBucket` (clock-injected, weighted costs) and
+  `ScopedRateLimiter` keyed by scope string. Rate ≠ concurrency: this is pacing
+  only. Isolation is achieved purely by giving distinct keys.
+- **`circuit.py`** — `CircuitBreaker` (CLOSED/OPEN/HALF_OPEN) + `CircuitRegistry`.
+  Throttles count toward opening by default; auth errors do not.
+- **`routing/`** — see below.
+- **`control/`** — see below.
+- **`transport.py`** — `Transport` protocol + `HttpxTransport` (one persistent
+  `AsyncClient` **per route**, never per request). `_map_httpx_error()` maps
+  exceptions to `ErrorClass`. `AttemptOutcome` carries a `ResponseView`.
+- **`scope.py`** — `make_scope_resolver(mode)` returns `(Request, Route) -> str`.
+  ISOLATED_FOR_TEST keys by route+credential+IP; SHARED uses one key; DECLARED
+  keys per origin. `_host_of()` is reused by the engine's authorization check.
+- **`engine.py`** — the orchestrator: bounded pipeline, permit accounting,
+  control loop, watchdog, checkpoint integration. Structured-concurrency: an
+  inner task group runs producer+workers; when it drains, the outer group's
+  background loops are cancelled.
+- **`report.py`** — `MetricsAccumulator` (incremental, bounded) → `Report`.
+  Strictly separates logical-request counts from attempt counts.
+- **`stats.py`** — bounded-memory `LogHistogram`, `EwmaRate`, `RollingWindow`,
+  and `ExactQuantiles` (small workloads only).
+- **`events.py`** — failure-isolated synchronous `EventBus`. Subscribers can't
+  crash the engine. Event `data` must already be redaction-safe.
+- **`redaction.py`** — the single `Redactor`. All serialization paths use it.
+- **`sinks/`**, **`checkpoint.py`**, **`plugins.py`**, **`cli.py`**, **`api.py`**,
+  **`sim.py`** — see their docstrings; `sim.py` is the deterministic test server.
+
+## routing/
+
+- **`route.py`** — `Route` (id, kind, proxy_url, local_address, weight, per-route
+  `Controlled` concurrency) and `RoutePool` (with `active_count` authority). A
+  FIXED `active_count` means "keep exactly N active whenever possible."
+- **`health.py`** — `RouteHealthTracker`. `_is_route_attributable()` decides
+  whether a failure can quarantine a route (transport-level: yes; HTTP/app: no).
+  This distinction is a hard requirement — do not blur it.
+- **`selector.py`** — selectors take an injected `rng` (0..1) for determinism and
+  only choose among `health.is_available()` routes. They never change the route
+  set or active count.
+
+## control/
+
+- **`base.py`** — `Sample` (observations), `Decision` (auditable record),
+  `Controller` protocol, and `make_decision()` which routes a proposal through
+  the knob's `.propose()` so authority/bounds are always enforced — even a buggy
+  controller cannot exceed bounds or move a FIXED knob.
+- **`aimd.py`** — additive-increase/multiplicative-decrease with a Vegas-style
+  min-latency baseline and post-decrease cooldown (anti-oscillation).
+- **`gradient.py`** — smoothed latency-gradient controller.
+
+When adding a controller: implement `decide(knob, sample) -> (knob, Decision)`,
+always go through `make_decision()`, register it under the `goodput.controllers`
+entry point in `pyproject.toml`, and add a simulation test.
+
+## Style
+
+- `from __future__ import annotations` at the top of every module; prefer
+  `X | None` over `Optional[X]` (ruff enforces).
+- Full type hints; `mypy --strict` must pass on `src`. Avoid `# type: ignore`
+  except the few documented spots (pydantic field variance, optional `tomli`).
+- Match the existing docstring density: each module opens with a docstring citing
+  the relevant brief section(s); public classes/functions are documented.
+- No new required dependencies. Substantial integrations go behind extras in
+  `pyproject.toml` and import lazily inside functions.

goodput/__init__.py

@@ -0,0 +1,139 @@
+"""goodput — an adaptive HTTP execution engine that maximizes successful
+logical-request goodput.
+
+Public API (brief §32). The most common entry points::
+
+    import goodput as gp
+
+    result = gp.run_sync(
+        gp.repeat(gp.Request("GET", "https://example.com/health"), times=1000),
+        config=gp.EngineConfig(
+            global_concurrency=gp.bounded_adaptive(minimum=1, maximum=200, initial=10),
+        ),
+    )
+    print(result.report.summary())
+
+The library configures no logging and performs no networking at import time
+(brief §4).
+"""
+
+from __future__ import annotations
+
+from .api import run, run_sync
+from .classify import Classifiers, ResponseView
+from .config import (
+    AuthorizationContext,
+    EngineConfig,
+    Mode,
+    Objective,
+    SafetyCeilings,
+    ThrottleScopeMode,
+    ValidationPlan,
+    load_config,
+)
+from .control_mode import (
+    Controlled,
+    ControlMode,
+    FixedValueError,
+    adaptive,
+    bounded_adaptive,
+    disabled,
+    fixed,
+    inherit,
+)
+from .engine import Engine, RunResult
+from .exceptions import (
+    AuthorizationError,
+    ConfigError,
+    ConstraintViolation,
+    EmergencyStop,
+    GoodputError,
+    PluginError,
+    SinkError,
+)
+from .models import (
+    AttemptRecord,
+    Classification,
+    ErrorClass,
+    LogicalRequest,
+    LogicalResult,
+    Outcome,
+    Request,
+    ThrottleSignal,
+)
+from .report import Report
+from .retry import RetryBudget, RetryPolicy
+from .routing import Route, RouteKind, RoutePool
+from .sources import (
+    from_async_iterable,
+    from_factory,
+    from_iterable,
+    from_jsonl,
+    parameter_matrix,
+    repeat,
+)
+
+__version__ = "0.1.0"
+
+__all__ = [
+    "__version__",
+    # high-level
+    "run",
+    "run_sync",
+    "Engine",
+    "RunResult",
+    # config
+    "EngineConfig",
+    "Mode",
+    "Objective",
+    "ThrottleScopeMode",
+    "SafetyCeilings",
+    "AuthorizationContext",
+    "ValidationPlan",
+    "load_config",
+    # control authority
+    "ControlMode",
+    "Controlled",
+    "FixedValueError",
+    "fixed",
+    "adaptive",
+    "bounded_adaptive",
+    "disabled",
+    "inherit",
+    # models
+    "Request",
+    "LogicalRequest",
+    "LogicalResult",
+    "AttemptRecord",
+    "Classification",
+    "ThrottleSignal",
+    "Outcome",
+    "ErrorClass",
+    # classify
+    "Classifiers",
+    "ResponseView",
+    # retry
+    "RetryPolicy",
+    "RetryBudget",
+    # routing
+    "Route",
+    "RouteKind",
+    "RoutePool",
+    # sources
+    "repeat",
+    "from_iterable",
+    "from_async_iterable",
+    "from_factory",
+    "from_jsonl",
+    "parameter_matrix",
+    # report
+    "Report",
+    # exceptions
+    "GoodputError",
+    "ConfigError",
+    "ConstraintViolation",
+    "AuthorizationError",
+    "EmergencyStop",
+    "SinkError",
+    "PluginError",
+]

goodput/api.py

@@ -0,0 +1,33 @@
+"""High-level convenience API (brief §32).
+
+Small, intuitive entry points that wrap the :class:`~goodput.engine.Engine`:
+
+* :func:`run` — execute an async request source and return a :class:`RunResult`.
+* :func:`run_sync` — a synchronous wrapper for scripts/notebooks.
+
+Power users instantiate :class:`~goodput.engine.Engine` directly for full control
+over transport, routes, selectors, controllers, and sinks.
+"""
+
+from __future__ import annotations
+
+import anyio
+
+from .config import EngineConfig
+from .engine import Engine, RunResult
+from .sources import RequestSource
+
+
+async def run(source: RequestSource, *, config: EngineConfig | None = None, **kwargs: object) -> RunResult:
+    """Execute ``source`` with a fresh engine and return the run result.
+
+    Extra keyword arguments are forwarded to :class:`~goodput.engine.Engine`
+    (e.g. ``routes=``, ``selector=``, ``controller=``, ``sinks=``, ``seed=``).
+    """
+    async with Engine(config, **kwargs) as engine:  # type: ignore[arg-type]
+        return await engine.run(source)
+
+
+def run_sync(source: RequestSource, *, config: EngineConfig | None = None, **kwargs: object) -> RunResult:
+    """Synchronous wrapper around :func:`run` (brief §32 convenience wrapper)."""
+    return anyio.run(lambda: run(source, config=config, **kwargs))

goodput/checkpoint.py

@@ -0,0 +1,88 @@
+"""Checkpointing and resume (brief §29).
+
+A checkpoint records which logical-request IDs have reached a terminal state so a
+resumed run can skip them. We deliberately make a narrow, honest guarantee
+(brief §29): resume avoids *re-executing terminal requests we recorded*; it does
+**not** provide network-side exactly-once semantics — that requires server-side
+idempotency or equivalent application guarantees.
+
+The default store is a JSONL append log (crash-safe-ish: each terminal ID is one
+line, flushed periodically). A config hash + schema version guard against
+resuming with an incompatible configuration.
+"""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from typing import TextIO
+
+SCHEMA_VERSION = 1
+
+
+class CheckpointStore:
+    """Append-only checkpoint of completed logical-request IDs."""
+
+    def __init__(self, path: str | Path, *, config_hash: str = "", flush_every: int = 50) -> None:
+        self.path = Path(path)
+        self._config_hash = config_hash
+        self._flush_every = flush_every
+        self._fh: TextIO | None = None
+        self._since_flush = 0
+        self._completed: set[str] = set()
+
+    def load(self) -> set[str]:
+        """Load previously-completed IDs (validating schema + config hash)."""
+        if not self.path.exists():
+            return set()
+        with open(self.path, encoding="utf-8") as fh:
+            first = fh.readline().strip()
+            if first:
+                header = json.loads(first)
+                if header.get("schema") != SCHEMA_VERSION:
+                    raise ValueError(
+                        f"checkpoint schema {header.get('schema')} != {SCHEMA_VERSION}"
+                    )
+                if self._config_hash and header.get("config_hash") != self._config_hash:
+                    raise ValueError(
+                        "checkpoint config_hash mismatch: refusing to resume with a "
+                        "different configuration"
+                    )
+            for line in fh:
+                line = line.strip()
+                if not line:
+                    continue
+                obj = json.loads(line)
+                if "id" in obj:
+                    self._completed.add(obj["id"])
+        return set(self._completed)
+
+    def open(self) -> None:
+        new_file = not self.path.exists()
+        self.path.parent.mkdir(parents=True, exist_ok=True)
+        self._fh = open(self.path, "a", encoding="utf-8")
+        if new_file:
+            self._fh.write(
+                json.dumps({"schema": SCHEMA_VERSION, "config_hash": self._config_hash}) + "\n"
+            )
+            self._fh.flush()
+
+    def is_completed(self, logical_id: str) -> bool:
+        return logical_id in self._completed
+
+    def record(self, logical_id: str) -> None:
+        if self._fh is None:
+            self.open()
+        assert self._fh is not None
+        self._completed.add(logical_id)
+        self._fh.write(json.dumps({"id": logical_id}) + "\n")
+        self._since_flush += 1
+        if self._since_flush >= self._flush_every:
+            self._fh.flush()
+            self._since_flush = 0
+
+    def close(self) -> None:
+        if self._fh is not None:
+            self._fh.flush()
+            self._fh.close()
+            self._fh = None

goodput/circuit.py

@@ -0,0 +1,139 @@
+"""Circuit breakers (brief §21).
+
+A circuit breaker protects a scope (origin/route/IP/credential/...) from
+hammering a failing dependency. States: CLOSED → OPEN (on failure threshold) →
+HALF_OPEN (after reset delay; admits limited probes) → CLOSED (on probe success)
+or back to OPEN (on probe failure).
+
+Throttle, overload, auth, and transport errors can be weighted differently
+(brief §21): by default throttles count toward opening (they indicate the server
+wants less traffic) but auth failures do *not* (they will not be fixed by
+waiting).
+"""
+
+from __future__ import annotations
+
+import enum
+from dataclasses import dataclass, field
+
+from .clock import Clock, MonotonicClock
+from .models import ErrorClass
+
+
+class CircuitState(str, enum.Enum):
+    CLOSED = "closed"
+    OPEN = "open"
+    HALF_OPEN = "half_open"
+
+
+@dataclass
+class CircuitConfig:
+    failure_threshold: int = 5
+    """Consecutive (weighted) failures before opening."""
+    error_ratio_threshold: float = 0.5
+    min_samples: int = 20
+    """Minimum samples in the window before ratio can trip the breaker."""
+    reset_seconds: float = 5.0
+    half_open_max_probes: int = 3
+    window_size: int = 50
+    count_throttle_as_failure: bool = True
+    count_auth_as_failure: bool = False
+
+
+@dataclass
+class CircuitBreaker:
+    """A single circuit breaker for one scope key."""
+
+    config: CircuitConfig = field(default_factory=CircuitConfig)
+    clock: Clock = field(default_factory=MonotonicClock)
+    state: CircuitState = CircuitState.CLOSED
+    _consecutive_failures: int = 0
+    _opened_at: float | None = None
+    _half_open_probes: int = 0
+    _window: list[bool] = field(default_factory=list)  # True = failure
+
+    def _counts_as_failure(self, error_class: ErrorClass) -> bool:
+        if error_class is ErrorClass.NONE:
+            return False
+        if error_class is ErrorClass.THROTTLE and not self.config.count_throttle_as_failure:
+            return False
+        if error_class is ErrorClass.AUTH and not self.config.count_auth_as_failure:
+            return False
+        return True
+
+    def allow(self) -> bool:
+        """Whether a new attempt may proceed under the current state."""
+        if self.state is CircuitState.CLOSED:
+            return True
+        if self.state is CircuitState.OPEN:
+            assert self._opened_at is not None
+            if self.clock.now() - self._opened_at >= self.config.reset_seconds:
+                self.state = CircuitState.HALF_OPEN
+                self._half_open_probes = 0
+                return True
+            return False
+        # HALF_OPEN: admit a bounded number of probes.
+        if self._half_open_probes < self.config.half_open_max_probes:
+            self._half_open_probes += 1
+            return True
+        return False
+
+    def record(self, *, success: bool, error_class: ErrorClass = ErrorClass.NONE) -> CircuitState:
+        """Record an attempt outcome and return the resulting state."""
+        is_failure = (not success) and self._counts_as_failure(error_class)
+
+        self._window.append(is_failure)
+        if len(self._window) > self.config.window_size:
+            self._window.pop(0)
+
+        if self.state is CircuitState.HALF_OPEN:
+            if is_failure:
+                self._trip()
+            elif success:
+                self._close()
+            return self.state
+
+        if is_failure:
+            self._consecutive_failures += 1
+        elif success:
+            self._consecutive_failures = 0
+
+        if self._should_trip():
+            self._trip()
+        return self.state
+
+    def _should_trip(self) -> bool:
+        if self._consecutive_failures >= self.config.failure_threshold:
+            return True
+        if len(self._window) >= self.config.min_samples:
+            ratio = sum(self._window) / len(self._window)
+            if ratio >= self.config.error_ratio_threshold:
+                return True
+        return False
+
+    def _trip(self) -> None:
+        self.state = CircuitState.OPEN
+        self._opened_at = self.clock.now()
+        self._half_open_probes = 0
+
+    def _close(self) -> None:
+        self.state = CircuitState.CLOSED
+        self._consecutive_failures = 0
+        self._opened_at = None
+        self._window.clear()
+
+
+class CircuitRegistry:
+    """Circuit breakers keyed by scope (brief §21 scopes)."""
+
+    def __init__(self, config: CircuitConfig | None = None, clock: Clock | None = None) -> None:
+        self._config = config or CircuitConfig()
+        self._clock = clock or MonotonicClock()
+        self._breakers: dict[str, CircuitBreaker] = {}
+
+    def breaker(self, key: str) -> CircuitBreaker:
+        b = self._breakers.get(key)
+        if b is None:
+            b = CircuitBreaker(config=self._config, clock=self._clock)
+            self._breakers[key] = b
+        return b