aid-runtime-python 0.1.39__py3-none-any.whl → 0.1.48__py3-none-any.whl

aid_runtime/__init__.py

@@ -268,7 +268,7 @@ from aid_runtime.types import (
 )
 
 #: Kept in sync with ``pyproject.toml`` by a test, so the two cannot drift.
-__version__ = "0.1.39"
+__version__ = "0.1.48"
 
 __all__ = [
     "ATTRIBUTES",

aid_runtime/retrieval/gate.py

@@ -65,7 +65,7 @@ class RetrievalGate:
         # from these, which is the part of the protocol a retrieval metric shares.
         self._gate = BaselineGate(
             thresholds,
-            metrics=cast("Sequence[Metric]", self._metrics),
+            metrics=cast(Sequence[Metric], self._metrics),
         )
 
     @property

aid_runtime_python-0.1.48.dist-info/METADATA

@@ -0,0 +1,276 @@
+Metadata-Version: 2.4
+Name: aid-runtime-python
+Version: 0.1.48
+Summary: The AID Framework Python runtime: model gateway with pinning and cost budgets, prompt registry, tool executor, OTel GenAI tracing, plain-async pipeline, and an eval harness that gates on statistical regression against a stored baseline.
+Author: The AID Framework authors
+License-Expression: Apache-2.0
+Project-URL: Homepage, https://github.com/aid-framework/aid-framework
+Project-URL: Documentation, https://github.com/aid-framework/aid-framework/blob/main/docs/design.md
+Project-URL: Issues, https://github.com/aid-framework/aid-framework/issues
+Keywords: ai,codegen,llm,evals,opentelemetry,pydantic
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Typing :: Typed
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+Requires-Dist: httpx<1,>=0.27
+Requires-Dist: opentelemetry-api<2,>=1.24
+Requires-Dist: pydantic<3,>=2.7
+Requires-Dist: pydantic-ai-slim<3,>=2
+Provides-Extra: pgvector
+Requires-Dist: psycopg[binary]<4,>=3.1; extra == "pgvector"
+Provides-Extra: deepagents
+Requires-Dist: deepagents==0.7.32; extra == "deepagents"
+Provides-Extra: dev
+Requires-Dist: mypy<3,>=1.11; extra == "dev"
+Requires-Dist: opentelemetry-sdk<2,>=1.24; extra == "dev"
+Requires-Dist: pytest<10,>=8; extra == "dev"
+Requires-Dist: pytest-asyncio<2,>=0.24; extra == "dev"
+Requires-Dist: ruff<1,>=0.6; extra == "dev"
+
+# `aid-runtime-python`
+
+The Python runtime for [AID Framework](https://github.com/aid-framework/aid-framework): the behavior layer that
+apps generated by the [`py-fastapi`](https://github.com/aid-framework/aid-framework/tree/main/packages/generators/py-fastapi)
+target import. Generated code is thin glue; this is where the behavior lives.
+
+Distribution name **`aid-runtime-python`**, import package **`aid_runtime`**. The two differ deliberately: a
+distribution name is conventionally hyphenated, and a Python import package cannot be. That asymmetry is asserted
+by the test suite so it cannot drift.
+
+## Install
+
+```sh
+pip install aid-runtime-python
+```
+
+Requires Python 3.11 or newer. Dependencies are `httpx`, `pydantic`, `pydantic-ai-slim` and
+`opentelemetry-api`.
+
+Two optional extras:
+
+```sh
+pip install "aid-runtime-python[pgvector]"   # the pgvector store adapter
+pip install "aid-runtime-python[deepagents]" # the deepagents agent engine, pinned
+```
+
+## Quick start
+
+```python
+from aid_runtime import (
+    Budget,
+    CompletionRequest,
+    CostProfile,
+    ModelGateway,
+    ModelSpec,
+    OpenAICompatibleProvider,
+    PinnedModel,
+)
+
+spec = ModelSpec(
+    name="gpt-4o-2024-08-06",
+    provider="openai-compatible",
+    cost=CostProfile(input_per_1k_usd=0.0025, output_per_1k_usd=0.01),
+    context_window=128_000,
+    max_output_tokens=4_096,
+)
+
+gateway = ModelGateway(
+    providers={"openai-compatible": OpenAICompatibleProvider(api_key=...)},
+    aliases={"small": PinnedModel(alias="small", spec=spec)},
+    budget=Budget(max_usd=5.0, scope="nightly"),
+)
+
+result = await gateway.generate("small", CompletionRequest.prompt("hello"))
+
+result.text, result.usage, result.cost_usd, result.alias, result.attempts, result.fell_back
+```
+
+## What it provides
+
+| Module | Provides |
+| --- | --- |
+| `gateway.py` | `ModelGateway` — alias resolution, capability negotiation, ordered fallback, cost accounting, GenAI spans |
+| `providers/` | The `Provider` protocol, one HTTP adapter (`OpenAICompatibleProvider`), one deterministic fake (`FakeProvider`) |
+| `prompts.py` | `PromptTemplate` / `PromptRegistry` — versioned prompts, typed rendering, few-shot assembly |
+| `structured.py` | `build_agent` — structured output via Pydantic AI only |
+| `tools.py` | `Tool` / `ToolExecutor` — declared definitions, argument validation, structured results |
+| `pipeline.py` | `Pipeline` / `PipelineContext` — plain-async named steps over a frozen context |
+| `agents/` | `AgentRuntime` — the loop, transfer vs. delegate, step/token/cost budgets with bounded degrade, memory, checkpointed human-in-the-loop |
+| `retrieval/` | Connectors, chunkers, the `Embedder` / `Reranker` / `VectorStore` protocols, hybrid fusion, citations, retrieval metrics |
+| `evals/` | Datasets, metrics, and the statistical baseline gate |
+| `tracing.py` | OTel GenAI spans: `llm_span`, `agent_span`, `record_usage`, `record_cost`, `record_attributes`, `record_hitl`, `record_exception` |
+| `cost.py` | `Budget`, `CostLedger`, `estimate_cost_usd`, `estimate_tokens` |
+| `types.py` | The request/response/usage/spec models everything else is typed against |
+
+194 names are exported. The layering is acyclic and one-directional: `errors` and `types` import nothing else in
+the package, `gateway` is imported only by the package `__init__` and the eval runner, and no core module imports
+a higher layer back.
+
+## Model pinning
+
+An **alias resolves to an exact provider model id**, including a pinned version. That is not cosmetic: a baseline
+recorded against `gpt-4o` means nothing if the alias floats to `gpt-4o-2026-…` on the next run, so an **unknown
+alias is an error** (`UnknownModelAliasError`) rather than a silent fall-through to a provider default.
+
+- `resolve(alias)` returns the `PinnedModel` (alias + `spec`); `provider_for(alias)` its provider.
+- `require(alias, capabilities)` asserts the capabilities *and* returns the pinned model, checking **both** sides —
+  a model that supports tool calling behind an adapter that does not serialise tool definitions cannot tool-call —
+  and raises `CapabilityMismatchError` naming what is missing.
+- `generate(alias, request, *, fallbacks=(), budget=None)` tries the alias then each fallback **in the order
+  given**, and reports the aliases actually attempted. Fallbacks are ordered, not conditional.
+- Every charged call opens a span and updates the ledger. `CostLedger` tracks per-call `LedgerEntry` records and
+  yields a `SpendSummary`; `Budget` (`max_usd`, `max_total_tokens`, `scope`) is enforced *before* the call, and
+  `estimate_usage` / `estimate_cost_usd` price a request from a model's `CostProfile` when a caller wants to check
+  before spending.
+
+## The eval gate measures statistics, not equality
+
+A model is **expected** to vary between runs, so a metric that compares two runs for equality is either always
+red or measuring nothing. Non-determinism is not a bug to be fixed here; it is the property the gate is built to
+be robust against.
+
+- `Dataset` — versioned cases with inputs, expectations, tags and metadata; `from_records` / `from_json` /
+  `from_path`, `to_json`, and lookups by id (`case`) and by tag (`by_tag`).
+- Metrics — `ExactMatch`, `NormalizedContains`, `JsonSchemaValid`, and `LlmJudge`. `metric_names` and
+  `default_metrics` cover the common set.
+- `run_evaluation(dataset, runner, metrics, runs=5)` samples the runner **N times** and produces an `EvalRun` of
+  per-run metric means, so a single unlucky run does not decide the verdict. `evaluate_once` is the single-pass
+  primitive; a `DatasetRun` gives per-case outcomes and a rendered `summary()`.
+- `EvalBaseline.from_run(run, model=..., note=...)` stores a baseline as JSON: per metric the sample count, mean
+  and spread. A committed example ships at `runtimes/python/tests/fixtures/capitals-baseline.json`.
+- `BaselineGate(thresholds).evaluate(run, baseline)` compares the current sample against the baseline using a
+  one-sided normal test. Each `Threshold` carries a `metric`, an optional absolute `min_mean`, a
+  `max_regression` allowance, and an `alpha` significance level. A metric passes by clearing `min_mean`, by being
+  within `max_regression` of the baseline, or by regressing only within sampling noise.
+
+Two deliberate limits:
+
+- **`LlmJudge` is implemented and never CI-required.** A judge is itself a model call, and a gate that depends on
+  one non-deterministic call inside the gate you are trying to trust is the problem this design avoids.
+  `gateway_judge(gateway, alias)` builds the judge call when a caller wants it.
+- **`GateConfigurationError` refuses a threshold on a model-derived metric**, because an N-run sample of a
+  probabilistic judge cannot carry a fixed threshold honestly.
+
+`cost` is the enforced budget. A latency p95 budget belongs with dashboards, not here.
+
+## Retrieval
+
+`retrieval/` is an IR-driven layer: `binding_from_spec(retriever, embedding=..., store=...)` folds the IR's
+`retrievers`, `embeddings` and `stores` entries into one `RetrieverBinding`, which a `Retriever` then ingests and
+searches through. The runtime consumes the names the IR emits, field for field.
+
+- Three protocols, in the same style as `providers.Provider`: `Embedder`, `Reranker`, `VectorStore`.
+- `FakeDeterministicEmbedder` maps text to a stable unit vector with a hash — keyless and offline, which is what
+  lets a retrieval gate run in CI. `OpenAICompatibleEmbedder` is the one hosted adapter: a single POST to
+  `/embeddings` over `httpx`, with the retry, timeout and error-chaining behavior of the HTTP provider.
+- `LexicalReranker` is the only reranker. It is deterministic and lexical; there is no hosted rerank adapter.
+- Chunk ids are **content hashes**, so re-ingesting the same input produces the same ids and a no-op diff. Ingest
+  is held to idempotence rather than to the byte-identical codegen guarantee.
+- **Pinning.** `embed_model` names an IR `embeddings` entry and resolves at codegen time to an exact `model_id`
+  plus its recorded width; the resolved id, width and store `kind` travel into the generated code. An alias is
+  never resolved at runtime, because a baseline recorded against a floating model means nothing.
+- **Cost.** Embedding calls charge the existing `CostLedger` through the existing `Budget`, using the catalog's
+  `CostProfile` for the pinned model. There is no retrieval-specific budgeting machinery.
+
+`RetrievalMetric` (`RecallAtK`, `MeanReciprocalRank`, `CitationGrounding`) is deliberately a **separate protocol**
+from the shipped `Metric`, which scores a single string prediction. Retrieval scores a *retrieved set*:
+recall@k and MRR need the ranked ids, and citation grounding needs the passages behind the citations. Retrieval is
+therefore sampled through `RetrievalGate.sample(...)`, producing a standard `EvalRun`, and then judged by the
+**unchanged** `BaselineGate`. Retrieval metrics are deterministic, so they pass the gate's own eligibility check.
+
+### pgvector
+
+`PgVectorStore` is a thin adapter over `psycopg` and the `vector` extension.
+
+- It is **locally provisioned and not covered by CI.** No CI job starts a database; the in-memory adapter carries
+  the store tests, and both adapters share the same `VectorStore` protocol and `similarity` semantics.
+  `compose.pgvector.yml` in the repository brings up a database for local use.
+- The `[pgvector]` extra is **import-safe when absent**: importing `aid_runtime` or `aid_runtime.retrieval` never
+  imports the driver, and loading the adapter raises a `ConfigurationError` naming the extra.
+
+## Control flow is plain async
+
+`Pipeline` is a list of named async (or plain) steps over a `PipelineContext` that is **frozen**, so a step cannot
+mutate what an earlier step observed — it returns a new context, which keeps a failed run's state inspectable. A
+step may return a context, a mapping, or `None`. Each step runs in its own span, a step raising anything aborts
+the run with the original exception chained (`PipelineError.__cause__`), and `StopPipeline` is the supported early
+exit.
+
+**Pydantic AI is used for structured output and nothing else** (`structured.build_agent`). The orchestration
+library stays out of the control-flow path deliberately, so it remains swappable, and **no graph or orchestration
+dependency is taken at all**: the agent loop in `agents/` is this package's own.
+
+Tracing follows the OTel GenAI semantic conventions: `gen_ai.*` attributes for the operation, model, provider,
+request parameters and usage, with AID-specific values namespaced `aid.*` so the two never collide. A failing call
+records its exception exactly once.
+
+## The agent runtime
+
+`AgentRuntime.from_ir(document, providers=..., ...)` reads the emitted IR and runs an agent to a stop condition:
+model call, tool calls, observations, until a terminal answer, `max_steps`, or a budget breach. Model calls go
+through `ModelGateway`, so fallback, cost accounting and tracing apply without a second implementation.
+
+**Transfer and delegation are different, and the API says which.** `_transfer` (`handoffs`) hands the conversation
+to the target and never resumes the caller, so `RunResult.agent_id` is the target and `path` grows. `_delegate`
+(`delegates`) runs the target in an isolated `[system, user]` context and returns its answer as a single tool
+observation, so `RunResult.agent_id` is the caller and `path` is unchanged. Both refuse to re-enter an agent
+already on the path.
+
+**A budget breach fails hard.** `BudgetExceededError` stops the run; the gateway preflight and the ledger both
+enforce the ceiling. An agent declaring `on_budget_exhausted: "degrade"` (spec v0.2) instead steps exactly one rung
+down its declared alias ladder — no retry loop, and the build-time gate rejects the declaration when no strictly
+cheaper alias exists. Each degraded step is recorded in `StepRecord.degraded_from`, and a checkpoint carries its
+current alias so a resumed run re-routes from where it stopped.
+
+**Memory** is in-process plus one store-backed adapter. `thread` scope is the checkpointer unit: trimmed to its
+`token_budget`, never written to a store. `user` and `global` scopes read and write the store and render as one
+extra system message, so injection is visible in the request rather than hidden.
+
+**Human-in-the-loop** pauses by returning a checkpoint (completed steps, pending call, current alias), and
+`resume(checkpoint, decision)` continues it. The pause is recorded as an `aid.hitl` span event, not an attribute,
+because a run can pause more than once.
+
+## Limits, stated plainly
+
+These are refused explicitly rather than approximated — each raises `FeatureDeferredError` or a
+`ConfigurationError` naming the reason:
+
+- `ModelGateway.stream` — the transport is request/response only, and no provider advertises
+  `Capability.STREAMING`.
+- Retrieval: non-file sources, `semantic` chunking, `incremental` refresh, retriever `filters`, a hosted
+  reranker, and the `qdrant` / `pinecone` / `weaviate` / `custom` store kinds.
+- Escalation routing and third-party orchestration frameworks.
+
+## Development
+
+From a checkout of the repository:
+
+```sh
+pip install -e ".[dev]"
+pytest
+ruff check .
+ruff format --check .
+mypy
+```
+
+`mypy` runs strict over `src` **and** `tests`, so the tests are type-checked too.
+
+**Non-responsibility:** no app-specific prompts, tools, or rubrics. Those belong to the generated app's
+`business/` tier — the seam this runtime is called from.
+
+## See also
+
+- [`@aid-framework/runtime`](https://www.npmjs.com/package/@aid-framework/runtime) — the same runtime in
+  TypeScript, module for module
+- [`@aid-framework/generator-py-fastapi`](https://www.npmjs.com/package/@aid-framework/generator-py-fastapi) — the
+  generator that emits apps against this runtime
+
+## License
+
+[Apache-2.0](https://github.com/aid-framework/aid-framework/blob/main/LICENSE). Code generated by AID Framework is
+not subject to this license.

{aid_runtime_python-0.1.39.dist-info → aid_runtime_python-0.1.48.dist-info}/RECORD

@@ -1,4 +1,4 @@
-aid_runtime/__init__.py,sha256=E8pW04ODgOND66tyve5B8CYcSjCjZeMuX93Mey_o0I0,11009
+aid_runtime/__init__.py,sha256=yA0v2rlJEFNknADLofgxAgVW1i78O5gAdkkUa6ZsoDc,11009
 aid_runtime/cache.py,sha256=jhw_EBd9dfWEUzYeibOvIkVBqqiLUjqF5WKEPz9GkhE,6836
 aid_runtime/concurrency.py,sha256=z0tFQHw35ttFLKfazGBjCMPOLQOnUotE_OPpvR1piH4,2829
 aid_runtime/control.py,sha256=u4U97QFdYSSjOPY3D6f1-GZhkGhygEdR-apdgw_jJok,13797
@@ -38,7 +38,7 @@ aid_runtime/retrieval/chunking.py,sha256=GMJvY1GoIua-FkpCH1S5qNmUrWSaSbl6sBJPCrS
 aid_runtime/retrieval/citations.py,sha256=UL9fsxLTzzO0n8tEW3X2O5erJ0hrk08NYMkZ9Pk8Jdg,4287
 aid_runtime/retrieval/connectors.py,sha256=JNEwq6S8zKX1GdlWe4Gn2Rdon87DWXlG_WNQfZTlN34,4744
 aid_runtime/retrieval/embedding.py,sha256=ZPjALe_nEzYpa8N5COExPGrojjZFX8awTz6vzJYfx_4,12374
-aid_runtime/retrieval/gate.py,sha256=fxrI7Og9d3J-NKUcn3QI-2N4K3o4-LQehXxCK19qzCM,4842
+aid_runtime/retrieval/gate.py,sha256=3avH1Gwp7iv7mMOskmNbk70Yq85tXwJrKVrQAKKI4Js,4840
 aid_runtime/retrieval/hybrid.py,sha256=-GIBeI3njT5uaEvSeq2t151Zlv2yOPHOPje4TJLNVyU,5804
 aid_runtime/retrieval/metrics.py,sha256=LN36BIXzpbhkG9grzE0q5wncgXt_Y85-b5jGJ038-3Y,7904
 aid_runtime/retrieval/pgvector.py,sha256=kdE6Gdk9JhR6-QxN353CmuNYMLyr5eBvnQl4krLzLew,10040
@@ -46,7 +46,7 @@ aid_runtime/retrieval/rerank.py,sha256=L4ICBi9YHDzhnKVtImKh5Wzre1HDzS8at-AKDkXBJ
 aid_runtime/retrieval/retriever.py,sha256=hVLP0xOLKioYsa-aS1akyJl5p-WbpVjm8OfCw26m4CE,13941
 aid_runtime/retrieval/stores.py,sha256=VEi68AaMHcQjdAbWOHflp3jCl1_1db71rIzZjTouLlo,5857
 aid_runtime/retrieval/types.py,sha256=8vKP21rmE9LglnB5gtv0dLBIkXzaPqdG3-GS98K0gpc,13498
-aid_runtime_python-0.1.39.dist-info/METADATA,sha256=NqIxXTImMqSvUWC2zPniEyZoopmh3ByGXVD2ICZVOGI,16364
-aid_runtime_python-0.1.39.dist-info/WHEEL,sha256=mOdXR9C3wMVov7cSY8PY-XCzo3SmmWCHvGQr5csJoao,91
-aid_runtime_python-0.1.39.dist-info/top_level.txt,sha256=QvODgNVRDDMmLcTnoIQ4jYnhqf-N4b9sHWEmaUUxp44,12
-aid_runtime_python-0.1.39.dist-info/RECORD,,
+aid_runtime_python-0.1.48.dist-info/METADATA,sha256=lu5Id8N5V5c5J9iJfPNGIIAmd5sVr6ZCk_0SiOgrJ9E,15559
+aid_runtime_python-0.1.48.dist-info/WHEEL,sha256=mOdXR9C3wMVov7cSY8PY-XCzo3SmmWCHvGQr5csJoao,91
+aid_runtime_python-0.1.48.dist-info/top_level.txt,sha256=QvODgNVRDDMmLcTnoIQ4jYnhqf-N4b9sHWEmaUUxp44,12
+aid_runtime_python-0.1.48.dist-info/RECORD,,

aid_runtime_python-0.1.39.dist-info/METADATA

@@ -1,268 +0,0 @@
-Metadata-Version: 2.4
-Name: aid-runtime-python
-Version: 0.1.39
-Summary: The AID Framework Python runtime: model gateway with pinning and cost budgets, prompt registry, tool executor, OTel GenAI tracing, plain-async pipeline, and an eval harness that gates on statistical regression against a stored baseline.
-Author: The AID Framework authors
-License-Expression: Apache-2.0
-Project-URL: Homepage, https://github.com/aid-framework/aid-framework
-Project-URL: Documentation, https://github.com/aid-framework/aid-framework/blob/main/docs/design.md
-Project-URL: Issues, https://github.com/aid-framework/aid-framework/issues
-Keywords: ai,codegen,llm,evals,opentelemetry,pydantic
-Classifier: Development Status :: 3 - Alpha
-Classifier: Intended Audience :: Developers
-Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.11
-Classifier: Programming Language :: Python :: 3.12
-Classifier: Programming Language :: Python :: 3.13
-Classifier: Typing :: Typed
-Requires-Python: >=3.11
-Description-Content-Type: text/markdown
-Requires-Dist: httpx<1,>=0.27
-Requires-Dist: opentelemetry-api<2,>=1.24
-Requires-Dist: pydantic<3,>=2.7
-Requires-Dist: pydantic-ai-slim<3,>=2
-Provides-Extra: pgvector
-Requires-Dist: psycopg[binary]<4,>=3.1; extra == "pgvector"
-Provides-Extra: deepagents
-Requires-Dist: deepagents==0.7.32; extra == "deepagents"
-Provides-Extra: dev
-Requires-Dist: mypy<3,>=1.11; extra == "dev"
-Requires-Dist: opentelemetry-sdk<2,>=1.24; extra == "dev"
-Requires-Dist: pytest<10,>=8; extra == "dev"
-Requires-Dist: pytest-asyncio<2,>=0.24; extra == "dev"
-Requires-Dist: ruff<1,>=0.6; extra == "dev"
-
-# AID runtime — Python
-
-The Python runtime library: the behavior layer imported by apps generated by the
-[`py-fastapi`](../../packages/generators/py-fastapi/) generator. Generated code is thin glue; this is where
-the behavior lives ([`docs/design.md` §4](../../docs/design.md), principle 1).
-
-Distribution name **`aid-runtime-python`**, import package **`aid_runtime`**. The two differ deliberately: a
-distribution name is conventionally hyphenated, and a Python import package cannot be. That asymmetry is
-asserted in `tests/test_package.py` so it cannot drift.
-
-Requires Python 3.11+.
-
-```sh
-pip install -e ".[dev]"        # the runtime plus its test/lint/type tooling
-pytest -q
-pip install -e ".[pgvector]"   # only if you want the pgvector store adapter
-```
-
-## What it provides
-
-| Module | Provides |
-| --- | --- |
-| `gateway.py` | `ModelGateway` — alias resolution, capability negotiation, ordered fallback, cost accounting, GenAI spans |
-| `providers/` | The `Provider` protocol, one HTTP adapter (`OpenAICompatibleProvider`), and one deterministic fake (`FakeProvider`) |
-| `prompts.py` | `PromptTemplate` / `PromptRegistry` — versioned prompts, typed rendering, few-shot assembly |
-| `structured.py` | `build_agent` — structured output via Pydantic AI only |
-| `tools.py` | `Tool` / `ToolExecutor` — declared definitions, argument validation, structured results |
-| `pipeline.py` | `Pipeline` / `PipelineContext` — plain-async named steps over a frozen context |
-| `agents/` | The native agent runtime: `AgentRuntime` — the loop, transfer vs. delegate, step/token/cost budgets with bounded degrade, memory, and a checkpointed human-in-the-loop pause |
-| `retrieval/` | Retrieval: connectors, chunkers, the `Embedder` / `Reranker` / `VectorStore` protocols, hybrid fusion, citations, and retrieval metrics |
-| `evals/` | Datasets, metrics, and the statistical baseline gate |
-| `tracing.py` | OTel GenAI spans: `llm_span`, `agent_span`, `record_usage`, `record_cost`, `record_attributes`, `record_hitl`, `record_exception` |
-| `cost.py` | `Budget`, `CostLedger`, `estimate_cost_usd`, `estimate_tokens` |
-| `types.py` | The request/response/usage/spec models everything else is typed against |
-
-194 names are exported from `aid_runtime`. The layering is acyclic and one-directional: `errors` and `types` import
-nothing else in the package, `gateway` is imported only by the package `__init__` and the eval runner, and no
-core module imports a higher layer back.
-
-## Model pinning
-
-An **alias resolves to an exact provider model id**, including a pinned version. That is not cosmetic: a
-baseline recorded against `gpt-4o` means nothing if the alias floats to `gpt-4o-2026-…` on the next run, so an
-**unknown alias is an error** (`UnknownModelAliasError`) rather than a silent fall-through to a provider
-default.
-
-```python
-from aid_runtime import (
-    Budget,
-    CompletionRequest,
-    CostProfile,
-    ModelGateway,
-    ModelSpec,
-    OpenAICompatibleProvider,
-    PinnedModel,
-)
-
-spec = ModelSpec(
-    name="gpt-4o-2024-08-06",
-    provider="openai-compatible",
-    cost=CostProfile(input_per_1k_usd=0.0025, output_per_1k_usd=0.01),
-    context_window=128_000,
-    max_output_tokens=4096,
-)
-
-gateway = ModelGateway(
-    providers={"openai-compatible": OpenAICompatibleProvider(api_key=...)},
-    aliases={"small": PinnedModel(alias="small", spec=spec)},
-    budget=Budget(max_usd=5.0, scope="nightly"),
-)
-
-result = await gateway.generate("small", CompletionRequest.prompt("hello"))
-result.text, result.usage, result.cost_usd, result.alias, result.attempts, result.fell_back
-```
-
-- `resolve(alias)` returns the `PinnedModel` (alias + `spec`); `provider_for(alias)` its provider. `require(alias,
-  capabilities)` asserts the capabilities *and* returns the pinned model, checking **both** sides — a model that
-  supports tool calling behind an adapter that does not serialise tool definitions cannot tool-call — and raises
-  `CapabilityMismatchError` naming what is missing.
-- `generate(alias, request, *, fallbacks=(), budget=None)` tries the alias then each fallback **in the order
-  given**, and reports the aliases actually attempted on the result. Fallbacks are ordered, not conditional.
-- Every charged call opens a span and updates the ledger. `CostLedger` tracks per-call `LedgerEntry` records
-  and yields a `SpendSummary`; `Budget` (`max_usd`, `max_total_tokens`, `scope`) is enforced before the call,
-  and `estimate_usage` / `estimate_cost_usd` price a request from a model's `CostProfile` when a caller wants
-  to check before spending.
-
-## The eval harness gates on statistics, not equality
-
-Phase 0 ratifies sampling-based gating: a model is **expected** to vary between runs, so a metric that
-compares two runs for equality is either always red or measuring nothing. Non-determinism here is not a bug to
-be fixed; it is the property the gate has to be robust against.
-
-- `Dataset` — versioned cases with inputs, expectations, tags and metadata; `from_records` / `from_json` /
-  `from_path`, `to_json`, and lookups by id (`case`) and by tag (`by_tag`).
-- Metrics — `ExactMatch`, `NormalizedContains`, `JsonSchemaValid`, and `LlmJudge`. `metric_names` and
-  `default_metrics` cover the common set.
-- `run_evaluation(dataset, runner, metrics, runs=5)` samples the runner **N times** and produces an `EvalRun`
-  of per-run metric means, so a single unlucky run does not decide the verdict. `evaluate_once` is the
-  single-pass primitive; a `DatasetRun` gives per-case outcomes and a rendered `summary()`.
-- `EvalBaseline.from_run(run, model=..., note=...)` stores a baseline as JSON: per metric the sample count,
-  mean and spread. `runtimes/python/tests/fixtures/capitals-baseline.json` is a committed example.
-- `BaselineGate(thresholds).evaluate(run, baseline)` compares the current sample against the baseline using a
-  one-sided normal test, with each `Threshold` carrying `metric`, an optional absolute `min_mean`, a
-  `max_regression` allowance, and an `alpha` significance level. A metric passes by clearing `min_mean`, by
-  being within `max_regression` of the baseline, or by regressing only within sampling noise.
-
-Two deliberate limits:
-
-- **`LlmJudge` is implemented and never CI-required.** A judge is itself a model call, and a gate that
-  depends on one non-deterministic call inside the gate you are trying to trust is the problem this design
-  avoids. `gateway_judge(gateway, alias)` builds the judge call when a caller wants it.
-- **`GateConfigurationError` refuses a threshold on a model-derived metric**, because an N-run sample of a
-  probabilistic judge cannot carry a fixed threshold honestly.
-
-`cost` is the enforced budget; a latency p95 budget belongs with Phase 2 dashboards, not here.
-
-## Retrieval
-
-`retrieval/` is an IR-driven layer: `binding_from_spec(retriever, embedding=..., store=...)` folds the IR's
-`retrievers`, `embeddings` and `stores` entries into one `RetrieverBinding`, which a `Retriever` then ingests
-and searches through. The runtime consumes the names the IR emits, field for field.
-
-- Three protocols, in the same style as `providers.Provider`: `Embedder`, `Reranker`, `VectorStore`. Phase 1
-  ships the implementations listed below and no others.
-- `FakeDeterministicEmbedder` maps text to a stable unit vector with a hash — keyless and offline, which is
-  what lets a retrieval gate run in CI. `OpenAICompatibleEmbedder` is the one hosted adapter: a single POST to
-  `/embeddings` over `httpx`, with the retry, timeout and error-chaining behaviour of the HTTP provider.
-- `LexicalReranker` is the only reranker. It is deterministic and lexical; there is no hosted rerank adapter.
-- Chunk ids are **content hashes**, so re-ingesting the same input produces the same ids and a no-op diff.
-  Ingest is therefore held to idempotence rather than to the byte-identical codegen guarantee.
-- **Pinning.** `embedModel` names an IR `embeddings` entry and resolves at codegen time to an exact `modelId`
-  plus its recorded width; the resolved id, width and store `kind` travel into the generated code. An alias is
-  never resolved at runtime, because a baseline recorded against a floating model means nothing.
-- **Cost.** Embedding calls charge the existing `CostLedger` through the existing `Budget`, using the catalog's
-  `CostProfile` for the pinned model. There is no retrieval-specific budgeting machinery.
-
-`RetrievalMetric` (`RecallAtK`, `MeanReciprocalRank`, `CitationGrounding`) is deliberately a **separate
-protocol** from the shipped `Metric`, which scores a single string prediction. Retrieval scores a *retrieved
-set*: recall@k and MRR need the ranked ids, and citation-grounding needs the passages behind the citations.
-Widening `Metric` would have changed an interface the existing gate already threshold-checks, for every
-existing implementation — so retrieval is sampled through `RetrievalGate.sample(...)` instead, producing a
-standard `EvalRun`, and then judged by the **unchanged** `BaselineGate`. Retrieval metrics are deterministic,
-so they pass the gate's own eligibility check.
-
-### pgvector
-
-`PgVectorStore` is a thin adapter over `psycopg` and the `vector` extension. Two consequences worth stating:
-
-- It is **locally provisioned and not covered by any gate.** No CI job starts a database, and no required
-  check touches one; the in-memory adapter carries the store gates, and the two adapters share the same
-  `VectorStore` protocol and the same `similarity` semantics. `compose.pgvector.yml` beside this README brings
-  up a database for local use.
-- The `[pgvector]` extra is **import-safe when absent**: importing `aid_runtime` or `aid_runtime.retrieval`
-  never imports the driver, and the adapter loading it raises a `ConfigurationError` naming the extra.
-
-## Control flow is plain async
-
-`Pipeline` is a list of named async (or plain) steps over a `PipelineContext` that is **frozen**, so a step
-cannot mutate what an earlier step observed — it returns a new context, which keeps a failed run's state
-inspectable. A step may return a context, a mapping, or `None`. Each step runs in its own span, a step raising
-anything aborts the run with the original exception chained (`PipelineError.__cause__`), and `StopPipeline` is
-the supported early exit.
-
-**Pydantic AI is used for structured output and nothing else** (`structured.build_agent`). The orchestration
-library stays out of the control-flow path deliberately, so it remains swappable, and **no graph/orchestration
-dependency is taken at all**: the agent loop in `agents/` is this package's own, and escalation routing and
-third-party orchestration frameworks remain out of scope ([`docs/design.md` §14](../../docs/design.md)).
-
-Tracing follows the OTel GenAI semantic conventions: `gen_ai.*` attributes for the operation, model, provider,
-request parameters and usage, with AID-specific values namespaced `aid.*` so the two never collide. A failing
-call records its exception exactly once.
-
-## The agent runtime
-
-`AgentRuntime.from_ir(document, providers=..., ...)` reads the emitted IR and runs an agent to a stop
-condition: model call, tool calls, observations, until a terminal answer, `maxSteps`, or a budget breach.
-Model calls go through `ModelGateway`, so fallback, cost accounting and tracing apply without a second
-implementation.
-
-**Transfer and delegation are different, and the API says which.** `_transfer` (`handoffs`) hands the
-conversation to the target and never resumes the caller, so `RunResult.agent_id` is the target and `path`
-grows. `_delegate` (`delegates`) runs the target in an isolated `[system, user]` context and returns its answer
-as a single tool observation, so `RunResult.agent_id` is the caller and `path` is unchanged. Both refuse to
-re-enter an agent already on the path.
-
-**A budget breach fails hard.** `BudgetExceededError` stops the run; the gateway preflight and the ledger both
-enforce the ceiling. An agent declaring `onBudgetExhausted: 'degrade'` (v0.2) instead steps exactly one rung
-down its declared alias ladder — no retry loop, and the build-time gate rejects the declaration when no
-strictly cheaper alias exists. Each degraded step is recorded in `StepRecord.degraded_from`, and a checkpoint
-carries its current alias so a resumed run re-routes from where it stopped.
-
-**Memory** is in-process plus one store-backed adapter. `thread` scope is the checkpointer unit: trimmed to its
-`tokenBudget`, never written to a store. `user` and `global` scopes read and write the store and render as one
-extra system message, so injection is visible in the request rather than hidden.
-
-**Human-in-the-loop** pauses by returning a checkpoint (completed steps, pending call, current alias), and
-`resume(checkpoint, decision)` continues it. The pause is recorded as an `aid.hitl` span event, not an
-attribute, because a run can pause more than once.
-
-## Development
-
-Four gates, all run by the `python` job in [`.github/workflows/ci.yml`](../../.github/workflows/ci.yml) over
-real content:
-
-```sh
-pytest
-ruff check .
-ruff format --check .
-mypy
-```
-
-`mypy` runs strict over `src` **and** `tests`, so the tests are type-checked too. The `python` job is a real
-gate but is **advisory**: the `main` ruleset requires the `node`, `workflows` and `licenses` contexts only.
-
-`tests/fixtures/capitals-baseline.json` is regenerated from `FakeProvider(accuracy=0.6, seed=0)` over 10 runs
-— the fake is seeded and deterministic per call, so the committed baseline reproduces exactly while a fresh
-sample of it still varies, which is exactly the property the gate has to tolerate.
-
-**Non-responsibility:** no app-specific prompts, tools, or rubrics. Those belong to the generated app's
-`business/` tier ([`docs/design.md` §8.4](../../docs/design.md), the seam).
-
-**Status:** Phase 0 scope implemented, plus the Phase 1 retrieval runtime. One thing deferred and
-undeclared, on purpose: `ModelGateway.stream` raises `FeatureDeferredError`, and no provider advertises
-`Capability.STREAMING` or target declares the capability, because the transport is request/response-only.
-Retrieval likewise fails loudly rather than
-approximating: non-file sources, `semantic` chunking, `incremental` refresh, retriever `filters`, a hosted
-reranker, and the `qdrant` / `pinecone` / `weaviate` / `custom` store kinds all raise `FeatureDeferredError`.
-Also deferred (§14): escalation routing, memory, guardrails, prompt/semantic caching, rate limiting, token
-buckets, and adapters beyond the one HTTP provider and the fake.
-
-Emitting retrieval into a generated app and indexing it from the CLI are later layers, so this package adds no
-entry point for either.
-
-Reference: [`docs/design.md` §9, §12, §14, §17 #2](../../docs/design.md).