aimai-kit 1.0.0__tar.gz

aimai_kit-1.0.0/.gitignore

@@ -0,0 +1,16 @@
+.env
+.venv/
+__pycache__/
+*.py[cod]
+.pytest_cache/
+.ruff_cache/
+.coverage
+htmlcov/
+dist/
+build/
+*.egg-info/
+evals/**/local-*.json
+.DS_Store
+
+# Local artifacts from the example tools and experiments
+.aimai-*.sqlite3

aimai_kit-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 fport
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

aimai_kit-1.0.0/PKG-INFO

@@ -0,0 +1,238 @@
+Metadata-Version: 2.5
+Name: aimai-kit
+Version: 1.0.0
+Summary: A framework-free LLM engineering toolkit: provider adapters, prompt and context engineering, structured outputs, tools, and a bounded agent loop
+Project-URL: Homepage, https://github.com/fport/aimai-kit
+Project-URL: Repository, https://github.com/fport/aimai-kit
+Project-URL: Documentation, https://github.com/fport/aimai-kit/tree/main/docs
+Project-URL: Issues, https://github.com/fport/aimai-kit/issues
+Author: fport
+License-Expression: MIT
+License-File: LICENSE
+Keywords: agents,anthropic,context-engineering,gemini,llm,openai,prompt-engineering,structured-outputs,tool-calling
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Typing :: Typed
+Requires-Python: >=3.12
+Requires-Dist: jinja2>=3.1
+Requires-Dist: pydantic>=2.9
+Requires-Dist: python-dotenv>=1.0
+Requires-Dist: tiktoken>=0.14
+Provides-Extra: providers
+Requires-Dist: anthropic<2,>=1.0; extra == 'providers'
+Requires-Dist: google-genai>=2.0; extra == 'providers'
+Requires-Dist: openai>=3.0; extra == 'providers'
+Description-Content-Type: text/markdown
+
+# aimai-kit
+
+[![CI](https://github.com/fport/aimai-kit/actions/workflows/ci.yml/badge.svg)](https://github.com/fport/aimai-kit/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/aimai-kit.svg)](https://pypi.org/project/aimai-kit/)
+[![Python](https://img.shields.io/pypi/pyversions/aimai-kit.svg)](https://pypi.org/project/aimai-kit/)
+[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
+
+A framework-free LLM engineering toolkit in one Python package: provider
+adapters, prompt and context engineering, structured outputs, a tool layer,
+a bounded agent loop, and a harness for long-running work.
+
+Five layers, each built on the one below it, each with its own measurements.
+
+| Layer | Package | What it adds |
+|---|---|---|
+| 1 | `provider/` | Token/cost/latency measurement, four adapters, retry and fallback |
+| 2 | `prompts/` | Versioned prompts, context budget, structured output, a repair loop |
+| 3 | `tools/` | Schemas from signatures, three provider exports, a five-gate executor |
+| 4 | `agent/` | A bounded loop, four budgets, loop detection, checkpoints |
+| 5 | `harness/` | Segments, spill, compaction, a memory store, sub-agents |
+
+**343 tests**, no vendor SDK outside `provider/adapters/`, and every layer
+runnable without an API key.
+
+---
+
+## Install
+
+```bash
+pip install "aimai-kit[providers]"
+```
+
+From source:
+
+```bash
+uv sync --all-extras --group dev
+cp .env.example .env     # add your keys
+uv run pytest            # 343 tests, live ones excluded
+```
+
+## Quick start
+
+```python
+from aimai_kit.provider.adapters import make_adapter
+from aimai_kit.provider.types import ChatRequest, Message, Role
+
+client = make_adapter("anthropic:claude-opus-5")
+result = client.complete(
+    ChatRequest(
+        messages=[Message(role=Role.USER, content="Say hello.")],
+        system="Be brief.",
+        max_output_tokens=64,
+    )
+)
+print(result.text, result.usage.input_tokens, result.usage.output_tokens)
+```
+
+Structured extraction with citation verification:
+
+```python
+from aimai_kit.prompts import PromptRegistry, build_request, generate_structured
+from aimai_kit.prompts.grounding import verify_citations
+from aimai_kit.prompts.schemas import ContractSummary
+
+built = build_request(
+    PromptRegistry("prompts"), "extract_contract@v2", document, schema=ContractSummary
+)
+repaired = generate_structured(client, built.req, ContractSummary)
+summary, grounding = verify_citations(repaired.value, document)
+print(summary.amount_minor, grounding.ratio)   # ungrounded fields are dropped
+```
+
+An agent with tools:
+
+```python
+from aimai_kit.agent import Agent, Budgets, Thread
+from aimai_kit.tools import CallContext, ToolExecutor
+from aimai_kit.tools.examples.orders import build_registry, seed_database
+
+seed_database()
+executor = ToolExecutor(build_registry())
+agent = Agent(client, executor, budgets=Budgets(max_steps=8, max_seconds=60))
+run = agent.run(Thread(), "What is the status of order 1002?",
+                ctx=CallContext(user_id="u-1", tenant_id="t-1"))
+print(run.stop_reason, run.answer)
+```
+
+## Command line
+
+```bash
+# Compare models: TTFT from streaming, real usage from one complete call
+uv run model-probe --prompt evals/probe/sample-prompt.txt \
+  --models anthropic:claude-opus-5 anthropic:claude-haiku-4-5 -n 5
+
+# Extraction quality against a golden set (no API key needed)
+uv run prompt-lab eval --prompt extract_contract@v2 --schema v2 \
+  --pricing config/pricing.toml --pricing-model claude-opus-5
+
+# With a real model
+uv run prompt-lab --model anthropic:claude-opus-5 eval
+```
+
+---
+
+## Why each layer looks the way it does
+
+Short version below; the full reasoning, including what was rejected and why,
+is in `docs/`.
+
+**[Provider](docs/01-provider.md)** — one internal message format and five
+error classes. Retry decisions are driven by the error *class*, never by
+matching message text. Cached tokens are normalized to a single rule across
+providers, because Anthropic reports them outside `input_tokens` and every
+cost calculation downstream depends on which convention you picked.
+
+**[Prompts and context](docs/02-prompts.md)** — prompts are versioned files
+identified as `name@vN+fingerprint`. The document goes into a user message,
+never the system block, so the cache prefix stays byte-identical across
+requests. Trimming is an explicit decision that produces a report line, and
+the non-trimmable sections raise rather than shrink.
+
+**[Tools](docs/03-tools.md)** — the schema is derived from the function
+signature, so the two cannot drift apart. Five gates run before anything
+executes, each producing a message the model can act on. Server context
+(`tenant_id`) is injected from the call and is absent from the schema, so a
+model cannot claim to be another tenant.
+
+**[Agent loop](docs/04-agent.md)** — four budgets, one stop reason, and a
+final tool-free turn so a stopped run still answers. Every tool call gets a
+result, including refused ones. Repetition is warned about before it is
+stopped, because a warned model usually recovers.
+
+**[Harness](docs/05-harness.md)** — the atomic unit of context is a segment,
+not a message, so trimming can never separate a tool call from its result.
+Large output spills to disk with a reference the agent can follow. Compaction
+converts old turns instead of dropping them, with a versioned prompt that
+names what must survive.
+
+---
+
+## Measurements
+
+Every number below is reproducible from this repository with no credentials.
+Full tables and the honest caveats are in **[docs/measurements.md](docs/measurements.md)**.
+
+| Experiment | Finding |
+|---|---|
+| Schema v1 vs v2 | Grounding 0% → 100%, at 0.06 more attempts and 16% more cost per document |
+| Grounding attribution | v1's `start_date` reads 0% with grounding on and 88.9% with it off — the drop is the missing citation field, not extraction |
+| Tool descriptions | Cutting descriptions to one line leaves selection accuracy unchanged but raises forbidden-tool calls from 0% to 4.5% |
+| Loop detection | p95 steps 7 → 3, at the cost of completion 100% → 75% on runs that would have recovered on their own |
+| Harness configurations | Naive trim 101k tokens and no answer; compaction 69.5k and no answer; compaction plus a sub-agent 12k and the answer survives |
+
+The models behind these numbers are deterministic stubs, not providers. That
+is deliberate: the point is that the measurement harness works and the
+comparisons are reproducible. Run the same commands with `--model
+anthropic:claude-opus-5` for numbers about a model.
+
+---
+
+## Testing
+
+CI runs the suite on Python 3.12 and 3.13, and a second job re-runs every
+measurement script and fails if a committed result changed. A table in the
+docs that no longer matches the code is a broken build, not a reader's
+problem.
+
+```bash
+uv run pytest                 # 343 tests
+uv run pytest -m live         # calls real APIs, needs keys
+uv run ruff check src/ tests/
+```
+
+A few tests are worth calling out because of what they protect:
+
+| Test | Guards against |
+|---|---|
+| `test_no_vendor_leak.py` | An SDK import escaping `adapters/` (AST-based, with an inverse check) |
+| `test_prefix_stable.py` | A variable field leaking into the cache prefix and silently multiplying the bill |
+| `test_schema_regression.py` | A schema changing without anyone noticing that old eval results are now incomparable |
+| `test_segments.py` | A trim separating a tool call from its result |
+| `test_compaction.py` | Compaction dropping a planted fact — three needles, both prompt versions |
+| `test_eval_harness.py` | The eval harness silently returning "correct" for everything |
+
+---
+
+## Known limits
+
+- **The golden sets are synthetic.** `scripts/generate_golden_set.py` writes 36
+  contracts, 10 of them deliberate edge cases. Replace them with your own
+  documents for a real evaluation; the `EDGE_CASES` map is the guide for what
+  to look for.
+- **The stubs are not models.** They perform real extraction and real
+  selection, but they know the shape of the synthetic data, so their accuracy
+  is optimistic.
+- **The pricing catalog only ships Anthropic rows.** OpenAI and Gemini are
+  commented out in `config/pricing.toml`; fill them from your own billing
+  page. A missing model produces a visible "window not in catalog" warning
+  rather than a silent zero.
+- **Synchronous only.** Async adds no teaching value at this size.
+- **Isolation is three layers and only two are in this repository.** The
+  in-process path jail and the cleaned subprocess environment are here;
+  closing the network belongs to deployment, and it is the layer that matters
+  most.
+
+## License
+
+MIT.

aimai_kit-1.0.0/README.md

@@ -0,0 +1,208 @@
+# aimai-kit
+
+[![CI](https://github.com/fport/aimai-kit/actions/workflows/ci.yml/badge.svg)](https://github.com/fport/aimai-kit/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/aimai-kit.svg)](https://pypi.org/project/aimai-kit/)
+[![Python](https://img.shields.io/pypi/pyversions/aimai-kit.svg)](https://pypi.org/project/aimai-kit/)
+[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
+
+A framework-free LLM engineering toolkit in one Python package: provider
+adapters, prompt and context engineering, structured outputs, a tool layer,
+a bounded agent loop, and a harness for long-running work.
+
+Five layers, each built on the one below it, each with its own measurements.
+
+| Layer | Package | What it adds |
+|---|---|---|
+| 1 | `provider/` | Token/cost/latency measurement, four adapters, retry and fallback |
+| 2 | `prompts/` | Versioned prompts, context budget, structured output, a repair loop |
+| 3 | `tools/` | Schemas from signatures, three provider exports, a five-gate executor |
+| 4 | `agent/` | A bounded loop, four budgets, loop detection, checkpoints |
+| 5 | `harness/` | Segments, spill, compaction, a memory store, sub-agents |
+
+**343 tests**, no vendor SDK outside `provider/adapters/`, and every layer
+runnable without an API key.
+
+---
+
+## Install
+
+```bash
+pip install "aimai-kit[providers]"
+```
+
+From source:
+
+```bash
+uv sync --all-extras --group dev
+cp .env.example .env     # add your keys
+uv run pytest            # 343 tests, live ones excluded
+```
+
+## Quick start
+
+```python
+from aimai_kit.provider.adapters import make_adapter
+from aimai_kit.provider.types import ChatRequest, Message, Role
+
+client = make_adapter("anthropic:claude-opus-5")
+result = client.complete(
+    ChatRequest(
+        messages=[Message(role=Role.USER, content="Say hello.")],
+        system="Be brief.",
+        max_output_tokens=64,
+    )
+)
+print(result.text, result.usage.input_tokens, result.usage.output_tokens)
+```
+
+Structured extraction with citation verification:
+
+```python
+from aimai_kit.prompts import PromptRegistry, build_request, generate_structured
+from aimai_kit.prompts.grounding import verify_citations
+from aimai_kit.prompts.schemas import ContractSummary
+
+built = build_request(
+    PromptRegistry("prompts"), "extract_contract@v2", document, schema=ContractSummary
+)
+repaired = generate_structured(client, built.req, ContractSummary)
+summary, grounding = verify_citations(repaired.value, document)
+print(summary.amount_minor, grounding.ratio)   # ungrounded fields are dropped
+```
+
+An agent with tools:
+
+```python
+from aimai_kit.agent import Agent, Budgets, Thread
+from aimai_kit.tools import CallContext, ToolExecutor
+from aimai_kit.tools.examples.orders import build_registry, seed_database
+
+seed_database()
+executor = ToolExecutor(build_registry())
+agent = Agent(client, executor, budgets=Budgets(max_steps=8, max_seconds=60))
+run = agent.run(Thread(), "What is the status of order 1002?",
+                ctx=CallContext(user_id="u-1", tenant_id="t-1"))
+print(run.stop_reason, run.answer)
+```
+
+## Command line
+
+```bash
+# Compare models: TTFT from streaming, real usage from one complete call
+uv run model-probe --prompt evals/probe/sample-prompt.txt \
+  --models anthropic:claude-opus-5 anthropic:claude-haiku-4-5 -n 5
+
+# Extraction quality against a golden set (no API key needed)
+uv run prompt-lab eval --prompt extract_contract@v2 --schema v2 \
+  --pricing config/pricing.toml --pricing-model claude-opus-5
+
+# With a real model
+uv run prompt-lab --model anthropic:claude-opus-5 eval
+```
+
+---
+
+## Why each layer looks the way it does
+
+Short version below; the full reasoning, including what was rejected and why,
+is in `docs/`.
+
+**[Provider](docs/01-provider.md)** — one internal message format and five
+error classes. Retry decisions are driven by the error *class*, never by
+matching message text. Cached tokens are normalized to a single rule across
+providers, because Anthropic reports them outside `input_tokens` and every
+cost calculation downstream depends on which convention you picked.
+
+**[Prompts and context](docs/02-prompts.md)** — prompts are versioned files
+identified as `name@vN+fingerprint`. The document goes into a user message,
+never the system block, so the cache prefix stays byte-identical across
+requests. Trimming is an explicit decision that produces a report line, and
+the non-trimmable sections raise rather than shrink.
+
+**[Tools](docs/03-tools.md)** — the schema is derived from the function
+signature, so the two cannot drift apart. Five gates run before anything
+executes, each producing a message the model can act on. Server context
+(`tenant_id`) is injected from the call and is absent from the schema, so a
+model cannot claim to be another tenant.
+
+**[Agent loop](docs/04-agent.md)** — four budgets, one stop reason, and a
+final tool-free turn so a stopped run still answers. Every tool call gets a
+result, including refused ones. Repetition is warned about before it is
+stopped, because a warned model usually recovers.
+
+**[Harness](docs/05-harness.md)** — the atomic unit of context is a segment,
+not a message, so trimming can never separate a tool call from its result.
+Large output spills to disk with a reference the agent can follow. Compaction
+converts old turns instead of dropping them, with a versioned prompt that
+names what must survive.
+
+---
+
+## Measurements
+
+Every number below is reproducible from this repository with no credentials.
+Full tables and the honest caveats are in **[docs/measurements.md](docs/measurements.md)**.
+
+| Experiment | Finding |
+|---|---|
+| Schema v1 vs v2 | Grounding 0% → 100%, at 0.06 more attempts and 16% more cost per document |
+| Grounding attribution | v1's `start_date` reads 0% with grounding on and 88.9% with it off — the drop is the missing citation field, not extraction |
+| Tool descriptions | Cutting descriptions to one line leaves selection accuracy unchanged but raises forbidden-tool calls from 0% to 4.5% |
+| Loop detection | p95 steps 7 → 3, at the cost of completion 100% → 75% on runs that would have recovered on their own |
+| Harness configurations | Naive trim 101k tokens and no answer; compaction 69.5k and no answer; compaction plus a sub-agent 12k and the answer survives |
+
+The models behind these numbers are deterministic stubs, not providers. That
+is deliberate: the point is that the measurement harness works and the
+comparisons are reproducible. Run the same commands with `--model
+anthropic:claude-opus-5` for numbers about a model.
+
+---
+
+## Testing
+
+CI runs the suite on Python 3.12 and 3.13, and a second job re-runs every
+measurement script and fails if a committed result changed. A table in the
+docs that no longer matches the code is a broken build, not a reader's
+problem.
+
+```bash
+uv run pytest                 # 343 tests
+uv run pytest -m live         # calls real APIs, needs keys
+uv run ruff check src/ tests/
+```
+
+A few tests are worth calling out because of what they protect:
+
+| Test | Guards against |
+|---|---|
+| `test_no_vendor_leak.py` | An SDK import escaping `adapters/` (AST-based, with an inverse check) |
+| `test_prefix_stable.py` | A variable field leaking into the cache prefix and silently multiplying the bill |
+| `test_schema_regression.py` | A schema changing without anyone noticing that old eval results are now incomparable |
+| `test_segments.py` | A trim separating a tool call from its result |
+| `test_compaction.py` | Compaction dropping a planted fact — three needles, both prompt versions |
+| `test_eval_harness.py` | The eval harness silently returning "correct" for everything |
+
+---
+
+## Known limits
+
+- **The golden sets are synthetic.** `scripts/generate_golden_set.py` writes 36
+  contracts, 10 of them deliberate edge cases. Replace them with your own
+  documents for a real evaluation; the `EDGE_CASES` map is the guide for what
+  to look for.
+- **The stubs are not models.** They perform real extraction and real
+  selection, but they know the shape of the synthetic data, so their accuracy
+  is optimistic.
+- **The pricing catalog only ships Anthropic rows.** OpenAI and Gemini are
+  commented out in `config/pricing.toml`; fill them from your own billing
+  page. A missing model produces a visible "window not in catalog" warning
+  rather than a silent zero.
+- **Synchronous only.** Async adds no teaching value at this size.
+- **Isolation is three layers and only two are in this repository.** The
+  in-process path jail and the cleaned subprocess environment are here;
+  closing the network belongs to deployment, and it is the layer that matters
+  most.
+
+## License
+
+MIT.

aimai_kit-1.0.0/config/pricing.toml

@@ -0,0 +1,44 @@
+# Pricing catalog — NOT hardcoded, read from here.
+# Unit: USD per 1,000,000 tokens. Prices are written IN QUOTES; a float is
+# rejected outright by pricing.py (the Decimal(float) trap).
+#
+# cached_input_per_mtok: the prompt-cache READ price. On Anthropic that is
+# roughly 10% of the input price. Delete the line to disable the discount.
+#
+# VERIFY: these figures are Anthropic list prices as of 2026-09. Fill in the
+# OpenAI and Gemini rows from your own billing page; leaving a row out beats
+# guessing — for a model missing from the catalog, model-probe prints
+# "window not in catalog" and reports zero cost.
+
+[models."claude-opus-5"]
+input_per_mtok = "5.00"
+output_per_mtok = "25.00"
+cached_input_per_mtok = "0.50"
+context_window = 1000000
+max_output_tokens = 128000
+
+[models."claude-sonnet-5"]
+input_per_mtok = "2.00"
+output_per_mtok = "10.00"
+cached_input_per_mtok = "0.20"
+context_window = 1000000
+max_output_tokens = 128000
+
+[models."claude-haiku-4-5"]
+input_per_mtok = "1.00"
+output_per_mtok = "5.00"
+cached_input_per_mtok = "0.10"
+context_window = 200000
+max_output_tokens = 64000
+
+# --- fill in from your own billing page ---------------------------------
+# [models."gpt-5.5"]
+# input_per_mtok = "0.00"
+# output_per_mtok = "0.00"
+# cached_input_per_mtok = "0.00"
+# context_window = 0
+#
+# [models."gemini-3-pro"]
+# input_per_mtok = "0.00"
+# output_per_mtok = "0.00"
+# context_window = 0

aimai_kit-1.0.0/docs/01-provider.md

@@ -0,0 +1,129 @@
+# Provider layer — why it looks like this
+
+The job of this layer is to make every provider look the same to everything
+above it, without flattening away the differences that matter. Those are two
+opposing pressures, and most of the decisions here are about where to put the
+line between them.
+
+## The internal message format
+
+`ChatRequest` / `ChatResult` / `Usage` / `Message` are the only vocabulary the
+rest of the package speaks. Three choices in them are worth explaining.
+
+**`system` is a top-level field, not an entry in the message list.** Anthropic
+and Gemini already expect it that way. OpenAI wants it as a message, and
+converting one into the other is a single line in the adapter. Normalizing in
+the other direction — burying the system prompt inside the message list and
+digging it out again in two of three adapters — would mean string surgery on
+every call, in the place least suited to it.
+
+**Unused fields exist and stay empty.** `ChatRequest` carried `json_schema`
+and `tools` from the beginning, filled by later layers. Adding a field is
+backward compatible; renaming one is not. The whole point of a stable
+protocol is that the layer above can grow without the layer below changing,
+and the way to get that is to leave room rather than to guess correctly.
+
+**Provider-specific settings go in `extra`.** Gemini's `thinking_config` and
+OpenAI's `reasoning_effort` have no equivalent in the other providers. Putting
+them in the internal model would pollute it; leaving them unreachable would
+force callers around the abstraction. A namespaced escape hatch keeps both
+properties.
+
+## The trap that costs money: usage field names
+
+This is the single most expensive detail in the layer.
+
+OpenAI reports cached tokens *inside* `input_tokens`, with a detail object
+breaking out the cached portion. Anthropic reports `cache_read_input_tokens`
+*separately*, not included in `input_tokens`. Both are reasonable; they are
+not the same.
+
+Pick one convention and normalize to it in the adapters. This package treats
+cached tokens as a **subset of input tokens** — OpenAI's convention — and the
+Anthropic adapter folds cache reads and cache writes into `input_tokens` on
+the way in.
+
+Skip that normalization and Anthropic costs come out systematically low. Not
+by a rounding error: on a cache-heavy workload the cached portion is most of
+the input. The failure is silent, appears only in a billing report, and
+looks like a pricing bug rather than an accounting one.
+
+`ModelPricing.cost()` now raises when `cached_input_tokens > input_tokens`,
+which is the shape an unnormalized adapter produces.
+
+## Error classification
+
+Five classes, and the classification carries the retry decision:
+
+| Class | Retry? | Because |
+|---|---|---|
+| `RateLimited` | yes | the provider said to wait, and told you how long |
+| `TransientError` | yes | 5xx and connection failures are worth one more try |
+| `InvalidRequest` | no | the same 400 comes back |
+| `AuthError` | no | the key is still wrong |
+| `AllProvidersFailed` | no | every link in the chain is down |
+
+`retryable` lives on the class, so the retry policy reads it from one place.
+The alternative — inspecting error messages for words like "rate limit" —
+breaks silently the day a provider rewrites its copy, and the breakage looks
+like an outage.
+
+## Retry and fallback
+
+**The SDK's own retry is disabled** (`max_retries=0`). Two layers of retry
+multiply: three attempts over two layers is six calls, and a deadline
+computed for three is meaningless. Exactly one layer decides.
+
+**There is a total deadline.** Three attempts times a 60-second `Retry-After`
+is a request that hangs for three minutes. The user's client gave up long
+before. Without a deadline, retrying becomes an outage of its own.
+
+**Jitter is not decoration.** Fifty clients that all receive a 429 and all
+wait exactly two seconds come back at the same instant and hit the same wall.
+Random spread is what turns a thundering herd back into a queue.
+
+**No retry on streaming.** Once the first chunk reached the user, retrying
+means erasing half a sentence on screen. That is a UI decision and belongs to
+the caller; the library does not hide it.
+
+**Falling back is an event, not a quiet rescue.** `llm_fallbacks_total` is
+exported with labels. A service that silently degrades looks healthy on every
+dashboard even while the primary provider is completely down — users are
+still getting answers, just slower, more expensive, and from a weaker model.
+The fallback rate needs an alert on it precisely because the error rate will
+not show anything.
+
+## Measuring: why TTFT and duration are separate
+
+TTFT is what a user feels. Total duration is what a capacity plan needs. A
+run with 200 ms TTFT and 9 s duration and one with 4 s TTFT and 5 s duration
+have similar averages and completely different user experiences.
+
+`model-probe` measures them in the only way that actually works: N streaming
+runs give TTFT and duration, then one `complete` call gives the real `usage`,
+which streaming may not report. The cost is N+1 calls. The payoff is the
+drift column — the local tiktoken estimate next to the provider's reported
+count. tiktoken is the OpenAI vocabulary, and on Anthropic and Gemini that
+drift reaches 10-20%, which is why the context budget carries a safety margin
+rather than filling to the limit.
+
+## Percentiles, not means
+
+Latency in an LLM service is skewed, so the reports return p50/p95/p99.
+
+One caveat is built into the tests: nearest-rank at small N swallows outliers.
+With twenty runs, p95 is the nineteenth value — a single ten-second run does
+not appear until p99. "p95 is fine" does not mean "there are no bad runs".
+Write N next to the SLO.
+
+Failed calls are excluded from latency (a 429 returning in 40 ms means no work
+happened) but included in cost (the input tokens may still have been billed).
+
+## The boundary that is tested
+
+Vendor SDKs are imported only under `provider/adapters/`, and
+`test_no_vendor_leak.py` enforces it with an AST walk rather than a regex, so
+comments and string literals do not raise false alarms. It also runs the
+inverse check — that the adapters really do import an SDK — because otherwise
+deleting the SDK calls by accident would leave the suite green and the
+guarantee empty.