evalrouter 0.3.0__py3-none-any.whl

evalrouter-0.3.0.dist-info/METADATA

@@ -0,0 +1,267 @@
+Metadata-Version: 2.4
+Name: evalrouter
+Version: 0.3.0
+Summary: Typed Kimpton evaluation client and command-line interface
+License-Expression: LicenseRef-Kimpton-EvalRouter-SDK
+License-File: LICENSE
+Requires-Python: <3.14,>=3.12
+Requires-Dist: httpx==0.28.1
+Requires-Dist: rich==15.0.0
+Provides-Extra: benchmark
+Requires-Dist: pydantic==2.13.5; extra == 'benchmark'
+Provides-Extra: keyring
+Requires-Dist: keyring==25.7.0; extra == 'keyring'
+Description-Content-Type: text/markdown
+
+# EvalRouter CLI
+
+Evaluate models or qualified repository agents from your terminal: discover
+supported targets, review a spending cap, run an evaluation, and export results. Requires Python 3.12
+or 3.13. Distributed under the proprietary [Kimpton EvalRouter SDK License](LICENSE).
+
+## Install and create your account
+
+We recommend [uv](https://docs.astral.sh/uv/getting-started/installation/) to install the CLI in its own Python environment:
+
+```sh
+uv tool install --python 3.12 evalrouter
+evalrouter --help
+```
+
+With pip, use `pip install evalrouter`. uv supplies Python 3.12 if needed. If
+your shell cannot find `evalrouter`, run `uv tool update-shell` and restart your
+terminal. Upgrade an existing installation with `uv tool upgrade evalrouter`.
+
+This package was previously published as `kimpton-evalrouter-sdk`. The command,
+the `kimpton_evalrouter` import and every API are unchanged. To switch, run
+`uv tool uninstall kimpton-evalrouter-sdk` (or
+`pip uninstall -y kimpton-evalrouter-sdk evalrouter`) and install `evalrouter`.
+
+The package supplies the `evalrouter` command; no repository checkout is needed.
+[Create an EvalRouter account](https://evalrouter.ai/register), verify your email,
+and create a workspace key in [API key settings](https://evalrouter.ai/settings/api-keys).
+Package installation does not grant account access or evaluation credit.
+
+Set `EVALROUTER_API_KEY` and `EVALROUTER_WORKSPACE_ID` through your secret manager
+or private environment. Use `https://api.evalrouter.ai` for `EVALROUTER_BASE_URL`,
+without `/v1`. Never put credentials in arguments, URLs, request files or logs.
+The CLI uses an existing workspace key; browser sign-up is a separate step.
+
+## Run an evaluation in one command
+
+```sh
+evalrouter run gpqa-diamond --model gpt-4o-mini
+```
+
+`run` matches the benchmark and model by slug, name or ID, suggesting the
+closest names after a typo. It creates a free quote, shows the coverage,
+conservative estimate, spending cap and concurrency, and asks `Run it? [Y/n]`.
+Only a yes starts paid work. It then follows progress and prints the scores,
+the change from your previous completed run of the same benchmark and model,
+and the report link. In a terminal, missing choices are asked for; run
+`evalrouter run` alone to pick everything. `--sample N` or `--full` sets
+coverage, `--max-cost USD` the cap and `--dry-run` stops after the quote.
+
+Without a terminal (CI, agents, `--json` or `--no-input`) `run` never prompts
+and refuses unless both `--yes` and `--max-cost` are given:
+
+```sh
+evalrouter run gpqa-diamond --model gpt-4o-mini --max-cost 5 --yes --json
+```
+
+The idempotency key of each accepted quote is saved locally before the run is
+submitted (`~/.local/state/evalrouter`, or `EVALROUTER_STATE_DIR`; identifiers
+only, never credentials). Repeating the same command after a dropped connection
+resumes that run instead of starting another; `--new` starts a separate one.
+Afterwards, `results`, `export --format html,json`, `wait` and `open` default to
+your last run and accept a unique run ID prefix. Watching a run survives brief
+API outages such as a 503 during a deployment.
+
+## Lists
+
+`catalog`, `catalog --models`, `status`, `connections list` and `workspace list`
+print a compact table sized to your terminal with one suggested next command;
+`catalog` hides benchmarks in review or unavailable unless you pass `--all`, and
+`catalog -i` browses them interactively. Every list accepts `--json`,
+`--jq '.data[].id'` (a small jq subset), `-w/--wide`, `--columns a,b`,
+`--sort COL` (`--sort=-COL` descending), `--filter KEY=VALUE` and `--limit N`.
+Piped output stays JSON. Colour follows `NO_COLOR`, `CI` and `TERM=dumb`.
+
+`run` shows the quote (expected cost from your previous run when known, worst
+case, and the smallest workable cap) before asking for the cap, refuses a cap
+that cannot fit one task, and asks before repeating a run that just finished
+(scripts pass `--new`). `--all-metrics` lists every metric and comparison.
+
+## Discover, quote, then run
+
+```sh
+evalrouter catalog --status active --json
+evalrouter catalog --models --json
+evalrouter catalog --slug BENCHMARK_SLUG --json
+```
+
+Choose a profile whose `quote_availability.status` is `ready_for_quote` and a
+compatible managed model route from those responses. An active catalog entry
+alone does not mean it is ready to run.
+Save this as `quote.json`, replacing both uppercase identifiers:
+
+```json
+{
+  "model": {"kind": "managed", "route_id": "MODEL_ROUTE_ID"},
+  "selection": {"profile_ids": ["BENCHMARK_PROFILE_ID"]},
+  "coverage": {"mode": "sample", "sample_count": 3, "seed": 42},
+  "max_charge_microusd": "1000000"
+}
+```
+
+```sh
+evalrouter quote --config quote.json --json
+```
+
+Review compatibility, coverage, cost components, warnings and expiry. A quote
+starts no paid work. The example's $1 platform cap is not a price guarantee or
+promise that a particular evaluation fits. Paid runs require available credit.
+
+Persist the returned quote ID and an operation key before submitting:
+
+```sh
+evalrouter run --quote REVIEWED_QUOTE_ID --idempotency-key SAVED_OPERATION_KEY --json
+evalrouter wait RUN_ID --wait-timeout 3600 --json
+evalrouter results RUN_ID --json
+evalrouter export RUN_ID --format json --output result.json
+evalrouter export RUN_ID --format csv --output result.csv
+evalrouter export RUN_ID --format html --output result.html
+```
+
+Replace `RUN_ID` with the returned ID. A lost submission response is recovered
+with the same quote and operation key; a new key may start separate work.
+Inspect terminal status, coverage, errors and billing alongside scores. A sample
+is not a full-benchmark score. Exports refuse overwrite by default; use the
+result's integer `--version` for repeatable reports.
+
+## Supported commands
+
+`catalog` (`--models`, `--slug` or `--environment`), `connections list/create/check/update/disable`,
+`quote`, `run`, `status`, `wait`, `cancel`, `results`, `export` and `open`.
+Repository qualification uses `agent-builds options/preview/create/list/status/cancel`.
+Authoring your own benchmark package uses `benchmark init/validate/bundle`.
+Run each command with `--help` for exact flags. A connected model's credential
+is never a command argument: `connections create` asks with hidden input in a
+terminal, and scripts use `--key-stdin` or `--key-env NAME`
+(`connections update --rotate-key` replaces it). Connection checks can send
+a small model request, and your model provider may bill separately.
+
+For `wait` and `run --wait`, `--progress auto` shows elapsed time and status in
+an interactive terminal, with a percentage when validated processed and planned
+counts are available. Use `--progress plain` for structured progress or
+`--progress off` to suppress it. Redirected stderr and `--json` retain structured
+progress. A progress percentage counts processed samples; it is not a score.
+
+Use `--config -` for JSON on stdin. `--json` writes one result on stdout and
+progress on stderr. Exit codes: 0 success; 1 API/transport or failed run;
+2 invalid input or rejected request (including a refused unattended `run`);
+4 partial/cancelled waited run; 5 confirmation declined, nothing started. Interrupting
+a local wait does not cancel server work. Use `evalrouter cancel RUN_ID`
+explicitly when cancellation is intended.
+
+See the [CLI documentation](https://evalrouter.ai/developers/docs) for account
+setup, model selection, spending semantics, recovery and versioned exports.
+The CLI uses the [model API](https://evalrouter.ai/developers/docs/model-api),
+which is also documented for direct HTTP integrations.
+
+## Author your own benchmark
+
+`benchmark init`, `benchmark validate` and `benchmark bundle` run entirely on
+your machine: no account, no network request, and nothing from your package is
+imported or executed. The normative validator ships inside `evalrouter`, and
+its one dependency (pydantic) is an optional extra so the base installation
+stays thin (httpx and rich only). With pip, use `pip install 'evalrouter[benchmark]'`:
+
+```sh
+uv tool install --python 3.12 "evalrouter[benchmark]"
+evalrouter benchmark init ./my-benchmark --namespace example --name my-benchmark
+evalrouter benchmark validate ./my-benchmark
+evalrouter benchmark bundle ./my-benchmark --output my-benchmark.zip
+```
+
+Without the extra these three commands stop before reading your package and
+report `benchmark_extra_required` with the exact install string; `--help` and
+every network command keep working. `init` writes an explicitly synthetic draft
+and claims no license: replace the tasks, rights, references and limits before
+submitting. Local validation is not platform admission — EvalRouter revalidates
+every submission on its own servers.
+
+## Repository agents
+
+Repository commands require version 0.2.0 or later and a service that has enabled
+repository qualification for the workspace. Installing the package does not
+enable that service. Check its current availability and supported targets first:
+
+```sh
+evalrouter agent-builds options --json
+evalrouter catalog --environment ENVIRONMENT_FAMILY_PATH --json
+```
+
+The catalog lookup takes the family path before `@`; qualification inputs still
+require the exact versioned reference returned by options. Stop if qualification
+is disabled or the desired exact target/model is absent.
+The supported runtime is locked Node/npm with an `evalrouter-agent.json` manifest;
+custom images and Python agent runtimes are not supported. Save `build.json`:
+
+```json
+{
+  "repository_url": "YOUR_PUBLIC_GITHUB_REPOSITORY_URL",
+  "ref": "main",
+  "manifest_path": "evalrouter-agent.json",
+  "qualification": {
+    "environment_ref": "EXACT_ENVIRONMENT_FROM_OPTIONS",
+    "model": {"kind": "managed", "route_id": "MODEL_FROM_OPTIONS"}
+  },
+  "max_cost_microusd": "1000000"
+}
+```
+
+For a private repository, connect GitHub in the web application and explicitly
+select that repository for this workspace. Add
+`source: {"github_connection_id": "CONNECTION_ID", "github_repository_id": 123}`
+using the two actual IDs from that connection. These IDs select a grant; they
+are not credentials. The CLI does not accept GitHub tokens or authorize access.
+
+```sh
+evalrouter agent-builds preview --config build.json --json
+```
+
+Preview resolves an immutable commit and returns the qualification scope, costs,
+cap and expiry. It creates no job or credit hold and starts no sandbox/model
+work. Review it, then save the quote ID and a durable operation key before:
+
+```sh
+evalrouter agent-builds create --quote REVIEWED_BUILD_QUOTE --idempotency-key SAVED_BUILD_KEY --json
+evalrouter agent-builds status BUILD_JOB_ID --json
+evalrouter agent-builds list --quote REVIEWED_BUILD_QUOTE --json
+```
+
+Creation reserves the quoted cap and may incur its stated costs. Recover an
+unknown submission with the same quote and key or the quote-filtered list;
+a new key may create new work. Cancel explicitly with
+`evalrouter agent-builds cancel BUILD_JOB_ID`. A `cancelling` or `reconciling`
+status remains unresolved. Wait for terminal cleanup and zero held funds.
+
+Ready requires `status: ready`, `cleanup_confirmed: true`, zero held funds and
+an `agent.ref`. It establishes only the returned scope, not benchmark quality.
+Use that record's exact `qualification.environment_ref`, `qualification.split`
+and `qualification.evaluation_coverage` in a separate evaluation quote:
+
+```json
+{
+  "agent": "READY_AGENT_REF",
+  "selection": {"environment": "RETURNED_ENVIRONMENT_REF", "split": "RETURNED_SPLIT"},
+  "coverage": "REPLACE_WITH_RETURNED_EVALUATION_COVERAGE_OBJECT",
+  "max_charge_microusd": "1000000"
+}
+```
+
+Replace the coverage placeholder with the entire returned object; do not guess
+tasks, counts or seeds. Use the existing `quote`, `run`, `wait`, `results` and
+`export` commands above. Qualification and evaluation have separate caps and
+charges. A qualification pass does not submit an evaluation automatically.

evalrouter-0.3.0.dist-info/RECORD

@@ -0,0 +1,33 @@
+evalrouter_contracts/__init__.py,sha256=QEe5ymjfZc8zmGw1hV3kHeciL0QRTYUSQz03eDJ7Ngk,568
+evalrouter_contracts/bundle.py,sha256=3yW73EtwEY_dIa8sIGIfPGXkxbJvcEZzk7pocESRUdU,2109
+evalrouter_contracts/components.py,sha256=FyqOf_nOgczd4H4UxFBffaax8tXkZX4M_uHD6-A8vxU,6495
+evalrouter_contracts/models.py,sha256=PLNSvlVG_-mSSUDXUqA6QeLeEEeExCukDpNu844tGHs,10673
+evalrouter_contracts/protocol.py,sha256=iBDMqlK7JkH0xPVr7mpKIoGDyUfoxVU9x9OboxeOcb0,8941
+evalrouter_contracts/py.typed,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+evalrouter_contracts/templates.py,sha256=Jn2aCPuy_LY3BV5PmRipqifse4KuFr8oQlmak9uszi0,5390
+evalrouter_contracts/validation.py,sha256=TzoAbtSETNdCgS_TvEjkAFVPJl3wv9eg-aD7IPKVrck,13085
+kimpton_evalrouter/__init__.py,sha256=aoBQS7zNMjMnqx0QIJLG1zPlI9dEWAahxOXxVmvoA3E,3351
+kimpton_evalrouter/__main__.py,sha256=k1ocEWawweo1qCJWNFAAvyxz3tcY13dzvCenHszij30,48
+kimpton_evalrouter/_account.py,sha256=RfCLuPsebYrsUNMqMHdexYyz-xxOPM5C9eKtWDtvaiY,55036
+kimpton_evalrouter/_boundary.py,sha256=VcHrxpjzWZzOjKdt-kb2uF5KcJMfnKzDLUTXtp_IW_U,7273
+kimpton_evalrouter/_guided.py,sha256=Txz-6FuhJc0SD_O4a1AzdRBI3e7NS3wy_xNqVEOr_ZA,72581
+kimpton_evalrouter/_helpers.py,sha256=LC3_lTqkxzM3qsO_lJ3lDx20tssuXBKr1UBxriz2ACY,3513
+kimpton_evalrouter/_listing.py,sha256=gdyczD_cdGZk_Zs0ioPCJ8RNt5rBZlD8v43TW8_T8x0,40495
+kimpton_evalrouter/_resources.py,sha256=agOtIrPjFaTOCPNV10OL9JX9lhm53-AWCkdsiwMWYcE,16919
+kimpton_evalrouter/_transport.py,sha256=zvXixX2PIWWz67u_PfeksrzRafKU2KIJdVkT6H6oR_I,16033
+kimpton_evalrouter/cli.py,sha256=ptw7wYOPVe6-hHayGaKZwbmdeQOTQJ6-BnFWjVdrD90,70185
+kimpton_evalrouter/py.typed,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+kimpton_evalrouter/types.py,sha256=5VZr73u2FzCxlCP3ZxehMSBIUuCyeC90TbDKlCT0EvU,29716
+kimpton_evalrouter/render/__init__.py,sha256=80_zJZa19y9kQC9ZY7gGZQnDj36MBJNfGU8s2W-E81E,3765
+kimpton_evalrouter/render/_console.py,sha256=4SN0ZBHc5rWoo0bSkDUEiBU_OKWgLifR0mFp4kqF1s8,3078
+kimpton_evalrouter/render/_theme.py,sha256=sDu6JvztJkSfaSxnKxfhgQ8niww5tsYA27QFdTSRoVE,2238
+kimpton_evalrouter/render/_widgets.py,sha256=aTRwFCI-aZOYSU6cpNSRinlfwfjr3ih9wVPYWK5RnzQ,4739
+kimpton_evalrouter/render/live.py,sha256=6ZCDptTzpMAVqtICiljKlCU00tq0sJNLrqvtv78r7Gs,21431
+kimpton_evalrouter/render/picker.py,sha256=OzaPIBfQUKSfa9x2u6u2EdbVjmrwePuN11aIC2ZlXRE,7625
+kimpton_evalrouter/render/screens.py,sha256=vzo_g8V7aG6qO-ns_JBQej8Wh9mp5eKbngm8xXntUeA,32045
+kimpton_evalrouter/render/tables.py,sha256=1s4g6yeBQ_CjBe9nu66rmb97NYDehPJMkFYEJU50FlE,14832
+evalrouter-0.3.0.dist-info/METADATA,sha256=3Ie5EsgZ3W3kuuWxyoS9_qN_B33mQ0znkIDnWh7p8kY,12360
+evalrouter-0.3.0.dist-info/WHEEL,sha256=qtCwoSJWgHk21S1Kb4ihdzI2rlJ1ZKaIurTj_ngOhyQ,87
+evalrouter-0.3.0.dist-info/entry_points.txt,sha256=KHK6JH7M7ucqxHMqB3d2Rns9HJ3z-s4Yo_wlf90aHrU,59
+evalrouter-0.3.0.dist-info/licenses/LICENSE,sha256=WsVi5Xlfp5XKnJGvoSNy722NnnWyv0Zp0p55gCZDgsU,1608
+evalrouter-0.3.0.dist-info/RECORD,,

evalrouter-0.3.0.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: hatchling 1.27.0
+Root-Is-Purelib: true
+Tag: py3-none-any

evalrouter-0.3.0.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+evalrouter = kimpton_evalrouter.cli:main

evalrouter-0.3.0.dist-info/licenses/LICENSE

@@ -0,0 +1,29 @@
+Kimpton EvalRouter SDK License
+
+Copyright (c) 2026 Kimpton. All rights reserved.
+
+Permission is granted to download, install, execute, and make internal copies of
+this SDK and its documentation solely to access the EvalRouter service and to
+integrate that access into applications you own or control, subject to your
+applicable EvalRouter service agreement and authorization.
+
+This is proprietary software, not an open-source license. Except for the limited
+permission above or as required by applicable law, no permission is granted to
+modify, redistribute, sublicense, sell, or use this software to provide a
+competing service. Preserve this license and all copyright notices in permitted
+copies. All rights not expressly granted are reserved.
+
+This license covers only the distributed SDK, CLI, and accompanying
+documentation. It grants no rights to EvalRouter backend or service source,
+benchmarks, datasets, models, credentials, trademarks, or other excluded material.
+Third-party dependencies remain subject to their own licenses.
+
+Downloading this software does not create an account, grant service access,
+provide credits, or waive service charges. Access remains subject to separate
+account, workspace, authorization, and billing requirements.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR
+PURPOSE, AND NONINFRINGEMENT. TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW,
+THE COPYRIGHT HOLDERS SHALL NOT BE LIABLE FOR ANY CLAIM, DAMAGES, OR OTHER
+LIABILITY ARISING FROM THE SOFTWARE OR ITS USE.

evalrouter_contracts/__init__.py

@@ -0,0 +1,11 @@
+"""Normative package-v1 contracts: models, canonicalization and offline validation.
+
+This package is the one authoritative validator. The public `evalrouter`
+distribution bundles it (its `[benchmark]` extra supplies pydantic), and the
+internal `kimpton-evalrouter-contracts` workspace package serves the private
+`kimpton-evalrouter-authoring` tools and the EvalRouter API. It reads bytes,
+computes digests and refuses invalid packages; it never imports contributed
+code, opens a network connection, signs anything or carries operator state.
+"""
+
+__version__ = "0.3.0"

evalrouter_contracts/bundle.py

@@ -0,0 +1,46 @@
+"""Deterministic ZIPs of declared files only; exclusive publication after validation."""
+
+import os
+import tempfile
+import zipfile
+from pathlib import Path
+
+from .validation import DirectoryReader, PackageError, canonical, manifest, validate
+
+
+def bundle(source: Path, destination: Path) -> dict:
+    reader = DirectoryReader(source)
+    package = manifest(reader)
+    destination.parent.mkdir(parents=True, exist_ok=True)
+    fd, temporary = tempfile.mkstemp(prefix=".evalrouter-bundle-", dir=destination.parent)
+    os.close(fd)
+    temp = Path(temporary)
+    try:
+        with zipfile.ZipFile(temp, "w", compression=zipfile.ZIP_STORED) as archive:
+            for name in sorted(["evalrouter.json", *(file.path for file in package.files)]):
+                entry = zipfile.ZipInfo(name, date_time=(1980, 1, 1, 0, 0, 0))
+                entry.create_system = 3
+                entry.external_attr = 0o100644 << 16
+                with archive.open(entry, "w") as output:
+                    if name == "evalrouter.json":
+                        output.write(canonical(package.model_dump()))
+                    else:
+                        limit = next(f.size_bytes for f in package.files if f.path == name)
+                        with reader.open(name) as stream:
+                            copied = 0
+                            while chunk := stream.read(min(1024 * 1024, limit + 1 - copied)):
+                                copied += len(chunk)
+                                if copied > limit:
+                                    raise PackageError(
+                                        f"{name}: file grew beyond its declared size"
+                                    )
+                                output.write(chunk)
+        # Validate the actual immutable snapshot, not merely the source folder.
+        report = validate(temp)
+        try:
+            os.link(temp, destination)
+        except FileExistsError as exc:
+            raise PackageError("Bundle destination already exists; choose a new path") from exc
+        return report
+    finally:
+        temp.unlink(missing_ok=True)

evalrouter_contracts/components.py

@@ -0,0 +1,181 @@
+"""`evalrouter.native-components.v1`: the typed component graph a native package points at.
+
+Native packages only. `parameters` exists solely on `NativeProtocol`, so a
+declarative `text.v1` package and a `koliseum` original package have nowhere to
+carry the pointer and are out of scope for this contract.
+
+The pointer rides `protocol.parameters` rather than a new envelope field because
+`package_sha256` is `sha256(canonical(package.model_dump()))` with no
+`exclude_defaults`: adding even an optional defaulted field to
+`EvaluationPackage` would move the digest of every historical package.
+"""
+
+import hashlib
+import re
+from typing import Annotated, Literal
+
+from pydantic import AfterValidator, Field, model_validator
+
+from .models import Contract, Digest, PackagePath, Slug, safe_path
+
+CONTRACT = "evalrouter.native-components.v1"
+POINTER_KEYS = ("component_contract", "component_path", "component_sha256")
+INDEX_LIMIT = 256 * 1024
+
+#: Critical extensions this build understands. An unrecognized critical key is a
+#: refusal before preparation, quote, or execution; unknown optional keys
+#: round-trip and contribute nothing to compiled semantics.
+RECOGNIZED_CRITICAL_EXTENSIONS: frozenset[str] = frozenset()
+
+#: Model operations a component may request, mapped onto declared capabilities.
+OPERATION_CAPABILITY = {
+    "generate": "text_generation",
+    "continuation_loglikelihood": "continuation_loglikelihood",
+}
+Operation = Literal["generate", "continuation_loglikelihood"]
+
+
+def entrypoint_target(value: str) -> str:
+    """Split `<declared package path>:<symbol>` and return the path part."""
+    path, separator, symbol = value.rpartition(":")
+    if not separator or not path or not re.fullmatch(r"[A-Za-z_][A-Za-z0-9_]*", symbol):
+        raise ValueError("Entrypoints are <declared code path>:<symbol>")
+    return safe_path(path)
+
+
+def checked_entrypoint(value: str) -> str:
+    entrypoint_target(value)
+    return value
+
+
+Entrypoint = Annotated[
+    str,
+    Field(pattern=r"^[a-zA-Z0-9_./:-]+$", max_length=200),
+    AfterValidator(checked_entrypoint),
+]
+ExtensionName = Annotated[
+    str, Field(pattern=r"^[a-z][a-z0-9]*(?:[.-][a-z0-9]+)*\.v[1-9][0-9]*$", max_length=120)
+]
+ExtensionKey = Annotated[str, Field(pattern=r"^[a-z][a-z0-9_]*$", max_length=80)]
+#: An extension body is arbitrary JSON so that an unknown *optional* extension
+#: written by a later build round-trips byte-exactly through this one instead of
+#: refusing the whole package. Object keys stay snake_case at every level, which
+#: is the only shape restriction; pydantic-core's own recursion guard bounds
+#: depth, and `INDEX_LIMIT` bounds total size.
+type ExtensionValue = (
+    str | int | float | bool | None | list[ExtensionValue] | dict[ExtensionKey, ExtensionValue]
+)
+ExtensionBody = dict[ExtensionKey, ExtensionValue]
+
+
+def canonical_bytes(value: dict) -> bytes:
+    """Canonical JSON bytes.
+
+    `canonical` is imported lazily because `validation` imports this module and
+    `packages/koliseum-benchmarks` imports `canonical` from `validation`, where
+    it must keep living.
+    """
+    from .validation import canonical
+
+    return canonical(value)
+
+
+def definition_digest(definition: dict) -> str:
+    """SHA-256 over the canonical bytes of a component's `definition` only."""
+    return hashlib.sha256(canonical_bytes(definition)).hexdigest()
+
+
+class TasksDefinition(Contract):
+    #: `Literal[False]` rather than `bool`: `binding` is `Literal["package-dataset"]`
+    #: in v1, so no `true` value can ever compile. Typing it here puts the refusal
+    #: at the component-document seam that `validate()` reports, instead of letting
+    #: `validate()` bless a package the compiler always rejects. The material-bound
+    #: case arrives with `MaterialReceipt` in a later contract version.
+    binding: Literal["package-dataset"]
+    material_required: Literal[False]
+    selection: Literal["all"]
+    task_id_field: str = Field(min_length=1, max_length=120)
+
+
+class RunnerDefinition(Contract):
+    dependency_lock: PackagePath
+    entrypoint: Entrypoint
+    framework: str = Field(min_length=1, max_length=80)
+    framework_version: str = Field(min_length=1, max_length=40)
+    kind: Literal["inspect", "lm-eval", "sandbox"]
+
+
+class ModelInterfaceDefinition(Contract):
+    operations: list[Operation] = Field(min_length=1, max_length=8)
+
+    @model_validator(mode="after")
+    def unique(self):
+        if len(set(self.operations)) != len(self.operations):
+            raise ValueError("Model interface operations must be unique")
+        return self
+
+
+class GraderDefinition(Contract):
+    aggregation: Literal["mean", "native"]
+    entrypoint: Entrypoint
+    metric: Slug
+
+
+class Component(Contract):
+    """Wrapper/definition split: the digest covers the definition, never itself."""
+
+    sha256: Digest
+
+    @model_validator(mode="after")
+    def digest(self):
+        if definition_digest(self.definition.model_dump()) != self.sha256:
+            raise ValueError("Component sha256 must equal the digest of its definition")
+        return self
+
+
+class TasksComponent(Component):
+    definition: TasksDefinition
+    schema_version: Literal["evalrouter.component.tasks.v1"]
+
+
+class RunnerComponent(Component):
+    definition: RunnerDefinition
+    schema_version: Literal["evalrouter.component.runner.v1"]
+
+
+class ModelInterfaceComponent(Component):
+    definition: ModelInterfaceDefinition
+    schema_version: Literal["evalrouter.component.model-interface.v1"]
+
+
+class GraderComponent(Component):
+    definition: GraderDefinition
+    schema_version: Literal["evalrouter.component.grader.v1"]
+
+
+class ComponentSet(Contract):
+    """Keyed by family name, so duplicates are structurally impossible."""
+
+    grader: GraderComponent
+    model_interface: ModelInterfaceComponent
+    runner: RunnerComponent
+    tasks: TasksComponent
+
+
+class Extensions(Contract):
+    """Both maps are required: an omitted section is not the same as an empty one."""
+
+    critical: dict[ExtensionName, ExtensionBody] = Field(max_length=50)
+    optional: dict[ExtensionName, ExtensionBody] = Field(max_length=50)
+
+
+class NativeComponentsIndex(Contract):
+    components: ComponentSet
+    extensions: Extensions
+    schema_version: Literal["evalrouter.native-components.v1"]
+
+    def digest(self) -> str:
+        return hashlib.sha256(canonical_bytes(self.model_dump())).hexdigest()
+
+    def unrecognized_critical(self) -> list[str]:
+        return sorted(set(self.extensions.critical) - RECOGNIZED_CRITICAL_EXTENSIONS)