kimpton-evalrouter-sdk 0.1.0__tar.gz

kimpton_evalrouter_sdk-0.1.0/LICENSE

@@ -0,0 +1,29 @@
+Kimpton EvalRouter SDK License
+
+Copyright (c) 2026 Kimpton. All rights reserved.
+
+Permission is granted to download, install, execute, and make internal copies of
+this SDK and its documentation solely to access the EvalRouter service and to
+integrate that access into applications you own or control, subject to your
+applicable EvalRouter service agreement and authorization.
+
+This is proprietary software, not an open-source license. Except for the limited
+permission above or as required by applicable law, no permission is granted to
+modify, redistribute, sublicense, sell, or use this software to provide a
+competing service. Preserve this license and all copyright notices in permitted
+copies. All rights not expressly granted are reserved.
+
+This license covers only the distributed SDK, CLI, and accompanying
+documentation. It grants no rights to EvalRouter backend or service source,
+benchmarks, datasets, models, credentials, trademarks, or other excluded material.
+Third-party dependencies remain subject to their own licenses.
+
+Downloading this software does not create an account, grant service access,
+provide credits, or waive service charges. Access remains subject to separate
+account, workspace, authorization, and billing requirements.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR
+PURPOSE, AND NONINFRINGEMENT. TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW,
+THE COPYRIGHT HOLDERS SHALL NOT BE LIABLE FOR ANY CLAIM, DAMAGES, OR OTHER
+LIABILITY ARISING FROM THE SOFTWARE OR ITS USE.

kimpton_evalrouter_sdk-0.1.0/PKG-INFO

@@ -0,0 +1,78 @@
+Metadata-Version: 2.4
+Name: kimpton-evalrouter-sdk
+Version: 0.1.0
+Summary: Typed Kimpton evaluation client and command-line interface
+License-Expression: LicenseRef-Kimpton-EvalRouter-SDK
+License-File: LICENSE
+Requires-Python: <3.14,>=3.12
+Requires-Dist: httpx==0.28.1
+Description-Content-Type: text/markdown
+
+# EvalRouter Python SDK and CLI
+
+Python SDK and CLI for Python 3.12–3.13, distributed under the proprietary
+[Kimpton EvalRouter SDK License](LICENSE). Discover evaluations and models,
+submit bounded evaluation runs, inspect results, and export reports.
+
+Install with `python -m pip install kimpton-evalrouter-sdk==0.1.0`. The same
+installation supplies `kimpton_evalrouter` and the `evalrouter` console command.
+No repository checkout or editable installation is required.
+
+Set `EVALROUTER_BASE_URL` to the configured API origin, `EVALROUTER_API_KEY` to a
+workspace API key, and `EVALROUTER_WORKSPACE_ID` to its workspace. The public API
+origin is `https://api.evalrouter.ai`. Package download does not grant service
+access. While website access is restricted, obtain an invitation from your
+EvalRouter operator, then create and fund the account in the web application.
+Never paste credentials into command arguments, source files, logs, or browser
+code. HTTPS is required outside explicit loopback development (`--allow-http`).
+
+```python
+from kimpton_evalrouter import Client
+
+with Client() as client:
+    available = client.evals.list()
+    # Choose an admitted exact eval reference and compatible model from discovery.
+    run = client.run(
+        model="YOUR_MANAGED_MODEL",
+        eval="YOUR_EXACT_EVAL_REFERENCE",
+        budget_usd="1.00",
+        idempotency_key="my-durable-evaluation-request",
+    )
+    result = client.runs.results(run["id"])
+    client.runs.export_to_file(run["id"], "result.json", params={"format": "json"})
+```
+
+The budget is a cap, not a price guarantee or promise that a particular evaluation
+fits. Check catalog availability and coverage. Managed routes and checked
+`connection:<id>` endpoints have distinct billing bases. A quote does not execute
+work. Reuse an idempotency key for retries of the same authorized submission.
+
+For explicit quote review, create a documented `NewQuote` JSON object in
+`quote.json`, then use the returned quote ID:
+
+```sh
+evalrouter catalog --json
+evalrouter catalog --models --json
+evalrouter quote --config quote.json --json
+evalrouter run --quote YOUR_QUOTE_ID --idempotency-key my-durable-request --wait --json
+evalrouter results YOUR_RUN_ID --json
+evalrouter export YOUR_RUN_ID --format json --output result.json
+```
+
+The supported CLI commands are `catalog`, `quote`, `run`, `status`, `wait`,
+`cancel`, `results`, `export`, and `connections list/create/check/update/disable`.
+Run a command with `--help` for flags. Connection credentials are read through
+`--key-env`, never a credential argument. JSON input can use `--config -` for
+stdin. `--json` prints one result on stdout; progress is on stderr. Exit status
+0 means success, 1 an API/transport or failed-run error, 2 invalid input or a
+rejected request, and 4 a partial/cancelled waited run. Interrupting a local wait
+does not cancel server work. Use `cancel` explicitly.
+
+The Python API exposes `Client` (`EvalRouter` alias), `catalog`, `evals`,
+`connections`, `quotes`, `evaluations`, `runs`, their request/response types, and
+bounded wait/export helpers. `catalog.evals`/`catalog.eval` retain the discovery
+aliases. Errors are `ClientError`, `APIError`, `RequestTimeout` and `WaitCancelled`.
+Use `RequestOptions` for request deadlines and local cancellation. JSON/CSV/HTML
+exports require an explicit destination and refuse overwrite unless requested.
+Account setup and key creation are separate web steps; the CLI uses an existing
+workspace API key.

kimpton_evalrouter_sdk-0.1.0/README.md

@@ -0,0 +1,68 @@
+# EvalRouter Python SDK and CLI
+
+Python SDK and CLI for Python 3.12–3.13, distributed under the proprietary
+[Kimpton EvalRouter SDK License](LICENSE). Discover evaluations and models,
+submit bounded evaluation runs, inspect results, and export reports.
+
+Install with `python -m pip install kimpton-evalrouter-sdk==0.1.0`. The same
+installation supplies `kimpton_evalrouter` and the `evalrouter` console command.
+No repository checkout or editable installation is required.
+
+Set `EVALROUTER_BASE_URL` to the configured API origin, `EVALROUTER_API_KEY` to a
+workspace API key, and `EVALROUTER_WORKSPACE_ID` to its workspace. The public API
+origin is `https://api.evalrouter.ai`. Package download does not grant service
+access. While website access is restricted, obtain an invitation from your
+EvalRouter operator, then create and fund the account in the web application.
+Never paste credentials into command arguments, source files, logs, or browser
+code. HTTPS is required outside explicit loopback development (`--allow-http`).
+
+```python
+from kimpton_evalrouter import Client
+
+with Client() as client:
+    available = client.evals.list()
+    # Choose an admitted exact eval reference and compatible model from discovery.
+    run = client.run(
+        model="YOUR_MANAGED_MODEL",
+        eval="YOUR_EXACT_EVAL_REFERENCE",
+        budget_usd="1.00",
+        idempotency_key="my-durable-evaluation-request",
+    )
+    result = client.runs.results(run["id"])
+    client.runs.export_to_file(run["id"], "result.json", params={"format": "json"})
+```
+
+The budget is a cap, not a price guarantee or promise that a particular evaluation
+fits. Check catalog availability and coverage. Managed routes and checked
+`connection:<id>` endpoints have distinct billing bases. A quote does not execute
+work. Reuse an idempotency key for retries of the same authorized submission.
+
+For explicit quote review, create a documented `NewQuote` JSON object in
+`quote.json`, then use the returned quote ID:
+
+```sh
+evalrouter catalog --json
+evalrouter catalog --models --json
+evalrouter quote --config quote.json --json
+evalrouter run --quote YOUR_QUOTE_ID --idempotency-key my-durable-request --wait --json
+evalrouter results YOUR_RUN_ID --json
+evalrouter export YOUR_RUN_ID --format json --output result.json
+```
+
+The supported CLI commands are `catalog`, `quote`, `run`, `status`, `wait`,
+`cancel`, `results`, `export`, and `connections list/create/check/update/disable`.
+Run a command with `--help` for flags. Connection credentials are read through
+`--key-env`, never a credential argument. JSON input can use `--config -` for
+stdin. `--json` prints one result on stdout; progress is on stderr. Exit status
+0 means success, 1 an API/transport or failed-run error, 2 invalid input or a
+rejected request, and 4 a partial/cancelled waited run. Interrupting a local wait
+does not cancel server work. Use `cancel` explicitly.
+
+The Python API exposes `Client` (`EvalRouter` alias), `catalog`, `evals`,
+`connections`, `quotes`, `evaluations`, `runs`, their request/response types, and
+bounded wait/export helpers. `catalog.evals`/`catalog.eval` retain the discovery
+aliases. Errors are `ClientError`, `APIError`, `RequestTimeout` and `WaitCancelled`.
+Use `RequestOptions` for request deadlines and local cancellation. JSON/CSV/HTML
+exports require an explicit destination and refuse overwrite unless requested.
+Account setup and key creation are separate web steps; the CLI uses an existing
+workspace API key.

kimpton_evalrouter_sdk-0.1.0/pyproject.toml

@@ -0,0 +1,19 @@
+[build-system]
+requires = ["hatchling==1.27.0"]
+build-backend = "hatchling.build"
+
+[project]
+name = "kimpton-evalrouter-sdk"
+version = "0.1.0"
+description = "Typed Kimpton evaluation client and command-line interface"
+readme = "README.md"
+license = "LicenseRef-Kimpton-EvalRouter-SDK"
+license-files = ["LICENSE"]
+requires-python = ">=3.12,<3.14"
+dependencies = ["httpx==0.28.1"]
+
+[project.scripts]
+evalrouter = "kimpton_evalrouter.cli:main"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/kimpton_evalrouter"]

kimpton_evalrouter_sdk-0.1.0/src/kimpton_evalrouter/__init__.py

@@ -0,0 +1,92 @@
+"""Typed API client. This package is independent of the API and native runners."""
+
+from functools import cached_property as _cached_property
+
+from . import types
+from ._resources import Catalog as _Catalog
+from ._resources import Connections as _Connections
+from ._resources import Evals as _Evals
+from ._resources import Evaluations as _Evaluations
+from ._resources import Quotes as _Quotes
+from ._resources import Runs as _Runs
+from ._transport import APIError, ClientError, RequestOptions, RequestTimeout, WaitCancelled
+from ._transport import Transport as _Transport
+
+__all__ = [
+    "Client",
+    "EvalRouter",
+    "APIError",
+    "ClientError",
+    "RequestOptions",
+    "RequestTimeout",
+    "WaitCancelled",
+    "types",
+]
+__version__ = "0.1.0"
+
+
+class Client(_Transport):
+    @_cached_property
+    def catalog(self) -> _Catalog:
+        return _Catalog(self)
+
+    @_cached_property
+    def connections(self) -> _Connections:
+        return _Connections(self)
+
+    @_cached_property
+    def quotes(self) -> _Quotes:
+        return _Quotes(self)
+
+    @_cached_property
+    def runs(self) -> _Runs:
+        return _Runs(self)
+
+    @_cached_property
+    def evals(self) -> _Evals:
+        return _Evals(self)
+
+    @_cached_property
+    def evaluations(self) -> _Evaluations:
+        return _Evaluations(self)
+
+    def run(
+        self,
+        *,
+        model: str,
+        eval: str,
+        budget_usd: str | float | None = None,
+        max_charge_microusd: str | None = None,
+        provider: str | None = None,
+        coverage: dict | None = None,
+        name: str = "Evaluation",
+        metadata: dict[str, str] | None = None,
+        idempotency_key: str | None = None,
+        wait: bool = True,
+        timeout: float = 3600,
+        poll_interval: float = 2,
+    ) -> "types.RunRecord":
+        """The canonical call: route `eval` to whichever provider owns it and run it.
+
+        `model` is a managed route id or `connection:<uuid>`;
+        `eval` is `eval://provider/name/version` or `name@version`. Returns the
+        terminal run when `wait` is true, otherwise the admitted run.
+        """
+        from uuid import uuid4
+
+        body: dict = {"model": model, "eval": eval, "name": name, "metadata": metadata or {}}
+        if provider:
+            body["provider"] = provider
+        if coverage:
+            body["coverage"] = coverage
+        if max_charge_microusd is not None:
+            body["max_charge_microusd"] = max_charge_microusd
+        else:
+            body["budget_usd"] = str(budget_usd)
+        run = self.evaluations.create(body, idempotency_key=idempotency_key or f"run-{uuid4()}")
+        if not wait:
+            return run
+        return self.runs.wait(str(run["id"]), timeout=timeout, poll_interval=poll_interval)
+
+
+EvalRouter = Client

kimpton_evalrouter_sdk-0.1.0/src/kimpton_evalrouter/__main__.py

@@ -0,0 +1,3 @@
+from .cli import main
+
+raise SystemExit(main())

kimpton_evalrouter_sdk-0.1.0/src/kimpton_evalrouter/_boundary.py

@@ -0,0 +1,50 @@
+"""Validate supported evaluation inputs before issuing a request."""
+
+from ._transport import ClientError
+
+
+def validate_submission(resource, body):
+    allowed = (
+        {"selection", "model", "coverage", "max_charge_microusd"}
+        if resource == "quotes"
+        else {
+            "model",
+            "eval",
+            "budget_usd",
+            "max_charge_microusd",
+            "provider",
+            "coverage",
+            "name",
+            "metadata",
+            "split",
+        }
+    )
+    if not isinstance(body, dict) or set(body) - allowed:
+        raise ClientError(
+            "unsupported_feature", "This submission contains unsupported input fields."
+        )
+    model = body.get("model")
+    if isinstance(model, str) and resource == "evaluations":
+        valid = bool(model) and not model.startswith(("checkpoint:", "agent:", "artifact:"))
+    elif isinstance(model, dict):
+        keys = {"managed": {"kind", "route_id"}, "connection": {"kind", "connection_id"}}
+        kind = model.get("kind")
+        valid = isinstance(kind, str) and kind in keys and set(model) == keys[kind]
+    else:
+        valid = False
+    if resource == "quotes":
+        selection = body.get("selection")
+        valid = (
+            valid
+            and isinstance(selection, dict)
+            and (
+                set(selection) == {"profile_ids"}
+                or "eval" in selection
+                and not set(selection) - {"eval", "provider", "split"}
+            )
+        )
+    if not valid:
+        raise ClientError(
+            "unsupported_feature",
+            "Choose an admitted eval and a managed model or checked connection.",
+        )

kimpton_evalrouter_sdk-0.1.0/src/kimpton_evalrouter/_helpers.py

@@ -0,0 +1,101 @@
+"""Local waiting and explicit export destinations; neither starts evaluations."""
+
+from __future__ import annotations
+
+import os
+import tempfile
+import threading
+import time
+from collections.abc import Callable
+from pathlib import Path
+
+from . import types
+from ._transport import ClientError, RequestOptions, Resource, checkpoint, pause, positive
+
+TERMINAL = {"completed", "partial", "failed", "cancelled"}
+STATES = TERMINAL | {"queued", "running", "cancelling", "finalizing"}
+
+
+class RunHelpers(Resource):
+    def get(self, run_id: str, *, options: RequestOptions | None = None) -> types.RunRecord:
+        raise NotImplementedError
+
+    def export(
+        self,
+        run_id: str,
+        *,
+        params: types.RunsExportParams | None = None,
+        options: RequestOptions | None = None,
+    ) -> bytes:
+        raise NotImplementedError
+
+    def wait(
+        self,
+        run_id: str,
+        *,
+        timeout: float = 3600,
+        poll_interval: float = 2,
+        cancel_event: threading.Event | None = None,
+        on_progress: Callable[[types.RunRecord], None] | None = None,
+    ) -> types.RunRecord:
+        deadline = time.monotonic() + positive(timeout, "Wait timeout")
+        interval = positive(poll_interval, "Poll interval")
+        while True:
+            remaining = checkpoint(deadline, cancel_event)
+            run = self.get(
+                run_id, options=RequestOptions(timeout=remaining, cancel_event=cancel_event)
+            )
+            checkpoint(deadline, cancel_event)
+            if run.get("status") not in STATES:
+                raise ClientError("invalid_response", "API returned an unknown evaluation state.")
+            if on_progress:
+                on_progress(run)
+            checkpoint(deadline, cancel_event)
+            if run["status"] in TERMINAL:
+                return run
+            pause(interval, deadline, cancel_event)
+
+    def export_to_file(
+        self,
+        run_id: str,
+        destination: str | Path,
+        *,
+        params: types.RunsExportParams | None = None,
+        options: RequestOptions | None = None,
+        overwrite: bool = False,
+    ) -> Path:
+        path = Path(destination)
+        if not overwrite and path.exists():
+            raise ClientError("destination_exists", "Export destination already exists.")
+        data = self.export(run_id, params=params, options=options)
+        return save_export(data, path, overwrite=overwrite)
+
+
+def save_export(data: bytes, destination: str | Path, *, overwrite: bool = False) -> Path:
+    path = Path(destination)
+    if not overwrite and path.exists():
+        raise ClientError("destination_exists", "Export destination already exists.")
+    temporary = None
+    try:
+        with tempfile.NamedTemporaryFile(
+            dir=path.parent, prefix=".kimpton-export-", delete=False
+        ) as stream:
+            temporary = Path(stream.name)
+            stream.write(data)
+            stream.flush()
+            os.fsync(stream.fileno())
+        if overwrite:
+            os.replace(temporary, path)
+        else:
+            os.link(temporary, path)
+            temporary.unlink()
+        return path
+    except FileExistsError:
+        raise ClientError("destination_exists", "Export destination already exists.") from None
+    except OSError:
+        raise ClientError(
+            "export_write_failed", "Could not write the export destination."
+        ) from None
+    finally:
+        if temporary is not None:
+            temporary.unlink(missing_ok=True)