evalkit-py 0.2.0.dev12__py3-none-any.whl

evalkit/bundled/skills/evalkit-deployment/references/config-and-deploy.md

@@ -0,0 +1,333 @@
+# Config and deployment
+
+## Trying evalkit first
+
+To see evalkit work before deploying it, run the try-it stack in `../try/`,
+following [Try locally](../../evalkit-docs/try-locally.md). It uses the same
+image and your own LLM, and runs a sample agent through an SDK run with traces.
+
+## Deployment shape
+
+One prebuilt docker image: server + built UI + skills + cost-map snapshot. This skill ships a [`docker-compose.yml`](../docker-compose.yml)
+that runs `server` + `postgres`; it pulls the image from
+`EVALKIT_IMAGE` (set it to the image you were given — registries differ per
+client, so there is no default). Postgres holds everything evalkit stores
+except bytes: users, workspaces, agents, cases, experiments and scores.
+External services are first-class:
+
+- An existing Postgres: set `db.url` and drop the bundled postgres service.
+- Blob storage: `file://` (mount a volume) or `s3://`-style URL.
+
+The **UI is part of the same server image** and served at the server root —
+there is no separate UI deployment. At runtime the FastAPI app serves
+`/assets` and returns `index.html` for every non-`/api` path (so client-side
+routes survive a refresh); deploying the image deploys the UI. The PyPI
+wheel (`evalkit-py`) bundles the UI too, so `pip install "evalkit-py[server]"`
+and `evalkit serve` also serve it.
+
+## Setup sequence (exact commands)
+
+```bash
+# Copy this skill's deployment files into your deploy dir, then edit them:
+cp <this-skill-dir>/config.example.yaml config.yaml   # fill in: blob, encryption_key
+cp <this-skill-dir>/docker-compose.yml .
+# Env referenced by config.example.yaml + compose: EVALKIT_IMAGE (the server
+# image you were given), POSTGRES_PASSWORD, EVALKIT_ENCRYPTION_KEY.
+# API keys are not in config: the commands below store them in the database.
+# First start: EVALKIT_ADMIN_PASSWORD (8+ characters, at most 72 bytes)
+# creates the admin user; a server with no admin and no
+# EVALKIT_ADMIN_PASSWORD refuses to start. Put these in a .env
+# file next to docker-compose.yml.
+# With db.migrate_on_start: true in config.yaml (see "Migrations on start"),
+# this one command starts everything and migrates:
+docker compose up -d
+# Without it, migrate before starting the server — it verifies the schema and
+# refuses to boot when it is missing or outdated:
+#   docker compose up -d postgres
+#   docker compose run --rm --no-deps server evalkit db upgrade --config /etc/evalkit/config.yaml
+#   docker compose up -d
+uv tool install --reinstall "evalkit-py==$(curl -fsSL http://localhost:8020/api/cli | jq -r .version)"   # the CLI, from PyPI at the server's version
+evalkit login --url http://localhost:8020   # admin / $EVALKIT_ADMIN_PASSWORD
+evalkit users add <teammate>                    # optional, admin only
+evalkit workspaces create dev                    # admin only; then prod
+evalkit workspaces members add dev <teammate> --role writer   # or --role reviewer
+evalkit llm-providers create openai --base-url https://api.openai.com/v1 --api-key-env OPENAI_API_KEY   # admin only
+evalkit judge-default set --provider openai --model gpt-5   # admin only
+evalkit agents create <name> --workspace dev --harness-url https://agent-dev.internal/evalkit/v1 --api-key-env AGENT_DEV_KEY
+evalkit sync <cases-dir>/<suite> --workspace dev   # your case YAML; uploads its fixtures too
+evalkit harness check <name>@dev                 # all declared capabilities must pass
+evalkit run <suite>@dev --agent <name>@dev                   # first experiment
+```
+
+The UI serves at the server root (http://localhost:8020); the server's own
+API reference is at `/docs` (Swagger, generated by FastAPI). Config flow: the
+compose file bind-mounts your `config.yaml` to `/etc/evalkit/config.yaml` and
+passes secrets as environment variables, which `env:VAR` values resolve at
+load.
+
+**Health check:** the server exposes `GET /api/health` → `{"ok": true}` (200
+once the app is up). Use it for load-balancer / Kubernetes liveness and
+readiness probes rather than `/` (which returns the SPA `index.html`).
+
+## Migrations on start
+
+`db.migrate_on_start: true` makes the server run `evalkit db upgrade` itself
+before it starts serving. It holds a Postgres advisory lock while it
+migrates, so replicas that start together migrate one at a time and the rest
+find the schema current.
+
+- Turn it on for a single-node compose deployment: `docker compose up -d` is
+  then the whole start, and an upgrade is pull the new image and `up -d`
+  again.
+- Leave it off when a separate step runs migrations, for example a Helm
+  pre-upgrade hook, or when you want to back up and run a migration by hand.
+  A release whose notes call for a maintenance window needs that window whatever this setting says.
+- There is no downgrade. Back up the database before you pull a new image
+  with this on, the same as before a manual `db upgrade`.
+
+## Self-managed Postgres
+
+To use an existing Postgres instead of the bundled `postgres` service, create
+a dedicated role and database, then point `db.url` at it and drop the
+`postgres` service (and its `depends_on`) from `docker-compose.yml`. evalkit
+keeps all its records in this database; back it up like any production
+database.
+
+As a Postgres superuser (e.g. `psql -U postgres`):
+
+```sql
+CREATE ROLE evalkit WITH LOGIN PASSWORD '<password>';
+-- Postgres 16+ (and managed Postgres like RDS/Aurora/Cloud SQL, where the
+-- admin is not a true superuser) requires you to be a member of a role to
+-- create objects owned by it. Grant it to yourself first:
+GRANT evalkit TO CURRENT_USER;
+CREATE DATABASE evalkit OWNER evalkit;
+-- `evalkit db upgrade` creates tables in the public schema. On Postgres 15+
+-- non-owners cannot create in `public` by default, so make evalkit own it:
+\connect evalkit
+ALTER SCHEMA public OWNER TO evalkit;
+```
+
+`env:` resolves a whole value only (it can't be embedded mid-string), so keep
+the password out of `config.yaml` by putting the entire URL in an env var —
+exactly as the bundled compose does for the built-in postgres:
+
+```yaml
+# config.yaml
+db:
+  url: env:EVALKIT_DATABASE_URL
+```
+
+```bash
+# .env (async psycopg driver, host reachable from the server container)
+EVALKIT_DATABASE_URL=postgresql+psycopg://evalkit:<password>@<host>:5432/evalkit
+```
+
+Run `evalkit db upgrade` (via `docker compose run ... server evalkit db upgrade`
+or the CLI) to create the schema, exactly as in the setup sequence above.
+
+## config.yaml
+
+Start from `config.example.yaml`. Any string value supports `env:VAR`
+indirection, resolved at load time — keep secrets out of the file.
+
+| Section | What it holds |
+|---|---|
+| `server` | `base_url` (used in links the CLI prints), optional `accent_color` (any CSS colour, e.g. `'#dc2626'`; tints the web UI so each environment is told apart at a glance) |
+| `db` | Postgres URL (`postgresql+psycopg://…`); optional `migrate_on_start` (default `false`, see below) |
+| `blob` | one blob URL: `fixtures/sha256/<sha256>` (case input files), `trajectories/`, `artifacts/` |
+| `encryption_key` | Fernet key that encrypts the stored API keys of agents, remote evaluators, LLM providers and trace sources. The server refuses to start without it once any exist. `evalkit db rekey --config config.yaml --old-key-env OLD --new-key-env NEW` re-encrypts all of them after a change. |
+| `allow_private_endpoints` | default `false`. The server refuses (400) a harness URL, evaluator endpoint or LLM provider base URL that resolves to a loopback, private, link-local, unspecified, multicast or reserved address, or does not resolve. Set `true` when they run in the same cluster or on `host.docker.internal`. |
+| `sentry` | optional: `dsn`, `environment`, `traces_sample_rate` (default `0.0`, errors only), `max_request_body_size` (default `never`; `small` up to 1 KB, `medium` up to 10 KB, `always`; bodies can hold client case and trajectory data). With it, the server reports unhandled errors, 500 responses and error logs to Sentry, tagged with release `evalkit@<version>`. IPs, cookies and sensitive headers are never sent. |
+| `cost_map` | `bundled` (snapshot in the image) or a path/URL override |
+| `auth` | `token_ttl_hours` for UI/CLI tokens |
+| `mcp` | optional `oauth`: `issuer`, `jwks_uri`, optional `audience` (default `<server.base_url>/mcp`), `algorithm` (default `RS256`), `scope` (default `evalkit:read`: the scope name your identity provider grants for read access). With it, `/mcp` also accepts that identity provider's JWTs carrying `scope`; evalkit login tokens work either way |
+| `otel` | collector limits, below |
+| `limits` | request body caps, below |
+
+### Request body limits
+
+The server reads request bodies as a stream and stops at the cap. A request
+over a cap gets `413` before the server reads past it, and the server logs
+the route, the cap and the size received.
+Fixture uploads and trajectory or artifact writes stream straight to blob
+storage, so a large upload does not sit in server memory. Downloads of
+fixtures, experiment blobs and stored run files stream too. Sending a
+fixture to a harness as a multipart upload first copies it to a temporary
+file in the server's temp directory (`/tmp` in the image). Give it room for
+the fixtures of the runs in flight: each agent's `max_concurrent_runs`,
+summed over the agents running at once, times the largest case's fixtures.
+In Kubernetes, `/tmp` on the container counts against the pod's ephemeral
+storage, and running out gets the pod evicted: mount an `emptyDir` at `/tmp`
+and set `ephemeral-storage` requests and limits to cover it. Run workspace
+files and folder zips stream too, from blob storage or the harness, one
+chunk at a time.
+
+On S3 or another object store, add a bucket lifecycle rule that aborts
+incomplete multipart uploads after a day and expires objects under
+`fixtures/staging/` after a day. A client that disconnects, or a server that
+stops, in the middle of an upload can leave either behind, and evalkit does
+not clean them up.
+
+| Key | Default | Applies to |
+|---|---|---|
+| `limits.fixture_upload_bytes` | 500 MB (`524288000`) | one fixture file uploaded by `evalkit sync` |
+| `limits.experiment_blob_bytes` | 500 MB (`524288000`) | one trajectory or artifact a local or SDK run writes |
+| `limits.case_upload_bytes` | 50 MB (`52428800`) | the case YAML files of one `evalkit sync` or `evalkit lint` |
+| `limits.request_bytes` | 10 MB (`10485760`) | every other request body (JSON) |
+| `otel.max_batch_bytes` | 16 MB | one collector export, see below |
+
+The `413` detail names the cap and its key, and the CLI prints it, e.g.
+`fixture upload is larger than the 500 MB limit (server config limits.fixture_upload_bytes)`.
+Raise the key in `config.yaml` and restart the server to allow larger
+bodies. A proxy in front of the server may have its own, smaller limit.
+
+`credentials:` and `judge_default:` are no longer accepted. A config that
+still has them stops the server at start, and the error prints the commands
+that recreate them in the database.
+
+Agents, remote evaluators, their API keys, LLM providers, the default judge,
+trace sources, workspaces and their reviewers are not in `config.yaml`. They live in the database and are managed with
+the CLI or the UI:
+
+| Thing | Create with | Notes |
+|---|---|---|
+| Agent | `evalkit agents create <name> --workspace dev --harness-url … --api-key-env VAR` | admin or writer; the key is stored encrypted and never shown (`agents show` prints `api key  set\|not set`); omit `--harness-url` for a code-driven agent; `--trajectory-source otel` needs `--mapping`; run settings such as `--max-concurrent-runs` (min'd with the harness's own limit), `--run-timeout`, `--stop-grace` are flags; `evalkit agents edit` changes them |
+| Remote evaluator | `evalkit evaluators create <name> --workspace dev --endpoint … --kind score --api-key-env VAR` | admin or writer; `--kind gauge` for measurements; runs use the evaluators in the agent's workspace |
+| LLM provider | `evalkit llm-providers create <name> --base-url … --api-key-env VAR` | admin only; server runs may only use a judge whose base URL is an active provider |
+| Default judge | `evalkit judge-default set --provider <name> --model M [--effort E] [--max-tokens N] [--concurrency N]` | admin only; `evalkit judge-default show` prints it. Use a reproducibility-tested model |
+| Trace source | `evalkit trace-sources create <name> --url … --api-key-env VAR` | admin only to create; any user can import from it (see below) |
+| Shared workspace | `evalkit workspaces create <name>` | admin only |
+| Writer / reviewer | `evalkit workspaces members add <ws> <user> --role writer\|reviewer` | admin only |
+
+Boundary rule — config vs capabilities: config answers "where and how do I
+reach you"; capabilities answer "what can you do once reached". Where both
+express a limit (concurrency), evalkit takes the minimum.
+
+## Outbound URLs and network egress
+
+- The server calls these URLs:
+  - a shared agent's `harness_url` (set by an admin or a writer)
+  - shared remote evaluator endpoints (set by an admin or a writer)
+  - LLM provider base URLs, used by judges (set by an admin)
+  - trace source URLs (set by an admin)
+- Without `allow_private_endpoints: true`, the server refuses agent, evaluator
+  and LLM provider URLs that resolve to internal addresses: localhost, private
+  ranges, link-local (including `169.254.169.254`), or that don't resolve.
+- The server never calls URLs from personal workspaces.
+- A server run whose agent, evaluator or LLM provider has no stored key calls
+  it without a key header. The create, resume or evaluate response carries
+  `warnings`, also logged.
+- The server does not follow redirects on these calls, so a harness or
+  evaluator cannot bounce it to another address.
+- One more outbound call: the LiteLLM cost map. The server image bundles it,
+  so the default setup makes no call. A server run from source without it
+  fetches it from `raw.githubusercontent.com`, and a `cost_map.source` set to
+  a URL is fetched at start. Allow that host if you restrict egress, or set
+  `cost_map.source` to a file path.
+- If this matters on your network, restrict the server's egress there, e.g.
+  block `169.254.169.254` and any internal ranges the server does not need.
+
+## Upgrades
+
+Pull the new image → back up the database → with `migrate_on_start`, start
+the new server (it migrates first); without it, stop or hold the new server →
+`evalkit db upgrade` (same compose-run pattern as setup) → start the server → developers update the CLI with the install command
+or `evalkit update` (every CLI call nudges them until they do), which also moves a project's `evalkit-py` pin and
+refreshes the skills in client repos that vendor them. Deploy only builds that are on PyPI: every `develop`
+(`X.Y.Z.devN`) and `main` (`X.Y.Z`) build of the publish workflow is.
+
+## CLI for the deployer
+
+The server image already contains the CLI — `docker compose run --rm server
+evalkit …` works with no local install. Everyone else installs `evalkit-py`
+from PyPI at the server's version: `GET /api/cli` (unauthenticated) returns
+`{"version": …, "package": "evalkit-py"}`, and the `/docs/install` page in the
+UI shows the commands with the version filled in. A proxy in front of the
+server must pass `/api/cli` without auth. For what to hand every other developer on the team, follow the
+**evalkit-usage** skill.
+Client suites that need input files register them per that skill's
+`references/suite-fixtures.md`.
+
+
+## OpenTelemetry collector and trace sources
+
+The server accepts OTLP/HTTP spans at `POST /api/otel/v1/traces` (protobuf
+or JSON, gzip accepted) with a user token or an agent token, and converts a
+session's spans to ATIF on demand. The optional `otel` block:
+
+```yaml
+otel:
+  settle_seconds: 10        # a session is settled when its newest span arrived longer ago than this
+  max_batch_bytes: 16777216 # one export request; 413 beyond it
+  max_session_bytes: 402653184 # span attributes one session read loads; 413 beyond it
+```
+
+An agent whose trajectories come from spans is created with
+`--trajectory-source otel --mapping mapping.json`. The mapping's `answer`
+(`root`, a span name, or `<span name>.<output key>`) and `exclude_spans`
+say where the answer comes from and which spans to leave out. The agent's
+`--trajectory-wait` (default 60 seconds) is how long a run waits for its
+session to settle, and `--provides key=true` sets capability flags evalkit
+cannot infer from spans.
+
+`settle_seconds` and `trajectory_wait_seconds` answer two different
+questions. `settle_seconds` is the server's rule for "this session is
+complete": no span has arrived for that long, so the batch exporter (which
+ships every 5 s by default) has nothing left. `trajectory_wait_seconds` is
+the runner's patience: how long one run keeps polling for that settled
+state before recording the trajectory as `partial` (root span seen) or
+`unavailable` (nothing seen) and moving on. Raise the first when spans
+arrive in bursts with long gaps; raise the second when the agent keeps
+emitting for a while after it answers.
+
+The two size limits protect the server, not the database. An export
+request over `max_batch_bytes` (16 MiB, after gzip) gets `413` and its spans
+are dropped. Building one session's trajectory (for a run, `evalkit traces
+check` or the Mappings tab) needs memory in proportion to the span
+attributes it loads, so a session over `max_session_bytes` (384 MiB) gets
+`413` with its size and the limit instead of being loaded. LLM spans carry
+the full input of each call, so a long agent session can store hundreds of
+MB. A build loads that input only for the first LLM call of each agent and
+subagent, and the limit counts only what a build loads, so the stored size
+of a session can be far above the limit.
+
+The server's memory limit is what these defaults are sized for. They keep
+two session reads at once (the most the server runs together) inside a
+2 GiB container: one read of the largest session allowed peaks at about
+0.6 GB above the idle server (about 200 MB). With less memory, lower
+`max_session_bytes`; raise it only with more memory. The server builds
+sessions in its own process, so a server killed for memory drops every
+request it is serving until it restarts; with a single replica, every route
+returns 502 until then.
+
+Agents with large spans should export in small batches with a longer
+timeout: `OTEL_BSP_MAX_EXPORT_BATCH_SIZE=16` and
+`OTEL_EXPORTER_OTLP_TIMEOUT=60` (seconds in Python; other SDKs read
+milliseconds). The server's Tracing docs page and the connect-traces skill
+give the agent's developers the same settings.
+
+Spans land in the `spans` table; mappings in `span_mappings`; agent tokens
+are rows in `auth_tokens` with `kind = agent`. History comes from a trace
+source registered with `evalkit trace-sources create`:
+
+- `evalkit traces import <agent> <session id> --source <trace source> --project <name>`
+  pulls one session.
+- `evalkit traces import <agent> --source <trace source> --project <name> --since <time>
+  [--until <time>]` pulls every trace that started in the range as a job on the
+  server, `otel.import_page_traces` traces per request to the source. Jobs are rows
+  in `trace_imports`; `evalkit traces imports <agent>` lists them, and a failed or
+  stopped job continues with `--continue`. A range import needs an active mapping
+  with a `session_id` rule.
+- The agent page's Import tab does both.
+
+Trace sources are server-wide. Any user can import from any trace source
+into an agent they can write, so developers can mine cases from traces on
+their own. This means the source's API key is shared with every user in
+effect: anyone can import any project that key can read. Give each trace
+source a key scoped to only the projects you are happy to share with every
+user. Imported traces are readable by every user too, since every user can
+read every workspace.
+
+Retention for `spans` is not built yet; size the database for the traffic you
+point at the collector and the history you import.

evalkit/bundled/skills/evalkit-deployment/references/harness-api.md

@@ -0,0 +1,139 @@
+# Implementing the Harness API (v1)
+
+The HTTP contract evalkit uses to drive the client's agent. Base URL is the
+agent's `--harness-url` (convention: `https://<host>/evalkit/v1`). Every
+request from evalkit carries the agent's key in an `x-api-key: <key>` header
+(the server sends the shared agent's stored key; a local run sends
+`credentials.agents` from the developer's `evalkit.yaml`) — reject anything else with 401. Exact request/response schemas: run
+`evalkit contracts export --target <dir>` to write `harness-api-v1.json`
+(OpenAPI, generated from the source-of-truth pydantic models) plus the other
+contract files, matching the installed evalkit version.
+
+## Endpoints
+
+| Endpoint | Purpose |
+|---|---|
+| `GET /capabilities` | Self-declaration: `contract_version`, `agent`, `provides` (trajectory, tool_calls, thinking, token_usage, artifacts, workspace, run_listing), `accepts` (files, file_urls, model_override, stop), `limits.max_concurrent_runs`, optional `input_schema` and `check_input` (below). Snapshotted onto every experiment. |
+| `POST /runs` | Start one agent run in a fresh session. Body: `{input, model?, metadata?, files?}`. `input` is the case input exactly as written; evalkit never rewrites it. When the case has fixture files and you declare `accepts.file_urls`, the JSON body carries `files: [{name, url, context}]` and you fetch each url yourself (the url is evalkit's blob store url plus the object key, e.g. `s3://bucket/fixtures/sha256/<sha256>`; your agent harness system must have access). If you declare only `accepts.files`, evalkit instead sends `multipart/form-data` with a `payload` JSON part (carrying `files: [{name, context}]`, no urls) plus one part per file. `context` is the file's description from the suite's fixture registry; show it to your agent however suits it (for example, an "Attached files" block ahead of the question). URLs win when both are declared; a harness declaring neither never receives fixture cases. Return `201 {run_id}` — your own id, typically the session id. For an agent whose trajectory source is `otel` this `run_id` is the session evalkit polls; a queue-backed harness whose worker opens its own session returns that session id here, or stamps this `run_id` on the job's spans in the attribute its mapping's `session_id` rule names. |
+| `GET /runs/{id}` | Poll status: `queued | running | completed | failed | cancelled`, plus `answer` and/or `output` (a completed run carries at least one; below), optional `artifacts` list, optional `error` (a completed run may carry a non-fatal error, e.g. a partial answer). |
+| `POST /runs/{id}/stop` | Cancel a run. Called on timeout when `accepts.stop`. Return as soon as the stop is requested. Report a terminal status from `GET /runs/{id}` once `GET /runs/{id}/trajectory` returns the run's full trajectory. evalkit keeps polling for up to `stop_grace_seconds` (agent config, default 120), then fetches the trajectory. |
+| `GET /runs/{id}/trajectory` | The run's trajectory as an ATIF document (below). Only meaningful after a terminal status; evalkit fetches it once and persists it. |
+| `GET /runs?query=&user_id=&client_id=&limit=&offset=&since=&until=` | Optional (`run_listing`): list runs/sessions, newest first, for the UI browser and `evalkit runs list` / `runs pull`. `{runs: [{run_id, title, user_id, client_id, created_at}], total}`. `since` / `until` (ISO 8601) keep runs created at or after `since` and before `until`; evalkit drops runs outside the window itself when a harness ignores them, and `evalkit harness check` warns. A harness without multi-tenancy returns fixed dummy user/client ids. Returned ids must work with the other `/runs/{id}` endpoints. A harness that keeps several of its own runs per session (for example one per chat message) and answers each with the whole session sets `session_id` on every item: evalkit's run is the session, so it lists items sharing a `session_id` once, as the newest. |
+| `GET /runs/{id}/files` and `GET /runs/{id}/files/{path}` | Optional (`workspace`): workspace listing `{files: [{path, size, content_type}]}` and raw file bytes. Powers the UI workspace viewer and artifact collection. |
+
+Notes:
+
+- `metadata` on `POST /runs` carries `{experiment_id, case_id, repetition}` so
+  the harness can stamp its own sessions/logs and keep eval traffic
+  distinguishable from app traffic (runs outside any experiment). Ignoring it
+  is allowed.
+- Timeouts, polling intervals, and retries are evalkit-side config, not
+  contract fields.
+
+## Typed agents: `output`, `input_schema`, `check_input`
+
+For agents that take typed inputs and return typed results (classifiers,
+taggers, extractors), rather than answering a question in prose:
+
+- `output` on `GET /runs/{id}`: the agent's structured result as a JSON
+  object, for example `{"label": "EVENT"}`. Send it instead of JSON-encoding
+  the result into `answer`. A completed run may carry `output` only, `answer`
+  only, or both. evalkit stores `output` as-is and passes it parsed to custom
+  evaluators as `run.output`; text-based evaluators (rubric judges) and the
+  Custom Evaluator API's `run.answer` get `output` as canonical JSON (sorted
+  keys) when `answer` is absent. An evalkit older than 0.2.0 still requires
+  `answer`.
+- `input_schema` in capabilities: a JSON Schema (draft 2020-12 unless its own
+  `$schema` says otherwise) for the `input` you accept. It never covers
+  `files`, which evalkit sends separately. Without one, evalkit assumes the
+  default question-and-answer schema, `{"type": "object", "required":
+  ["question"], "properties": {"question": {"type": "string"}}}`, so every
+  agent is checked the same way. Before an experiment starts,
+  evalkit checks every selected case's input against it and refuses the
+  whole experiment, listing each failing case, if any fails. Case authors can
+  run the same check on demand with `evalkit lint <dir> --agent <name>`. An
+  invalid schema makes the capabilities response invalid.
+- `check_input` in capabilities: the input `evalkit harness check` sends in
+  place of its default probe `{"question": "Hi. Reply with one word."}`.
+  Declare it whenever that default doesn't fit your `input_schema` or your
+  agent. It must pass `input_schema` (or the default schema), or the
+  capabilities response is invalid.
+
+## The trajectory: ATIF
+
+evalkit accepts ATIF (Agent Trajectory Interchange Format). The spec is the
+Harbor RFC, and it is the source of truth; evalkit doesn't copy it or publish a
+schema of its own. Pinned version:
+https://github.com/harbor-framework/harbor/blob/74cc6312018c349c6bd2400c89a0ac4983ac1085/rfcs/0001-trajectory-format.md.
+Accepted `schema_version` values ship in `atif-accepted.json` (from
+`evalkit contracts export`; currently `ATIF-v1.8`).
+
+Required subset (validation fails without these):
+
+- `schema_version`
+- `steps[]`, each with `step_id` (sequential int from 1), `source`
+  (`user` / `agent` / `system`), and `message` (string or content-part list)
+- tool interactions as `tool_calls[]` (`tool_call_id`, `function_name`,
+  `arguments`) paired to `observation.results[]` entries by `source_call_id`
+
+Everything else is optional and capability-declared: `reasoning_content`
+(thinking), per-step `metrics` (`prompt_tokens`, `completion_tokens`,
+`cached_tokens`, `cost_usd`), `model_name`, `timestamp`, `final_metrics`,
+`subagent_trajectories` (a list of embedded, independently valid trajectories),
+`extra`.
+
+Follow these RFC rules; evalkit's metrics depend on them:
+
+- Set `prompt_tokens` to every input token the model processed, cached ones
+  included, and `cached_tokens` to the cached part of it. evalkit prices
+  `prompt_tokens - cached_tokens` at the input rate and `cached_tokens` at the
+  cache-read rate. Put provider-specific counts such as cache writes in
+  `metrics.extra`.
+- Make the root `final_metrics` token totals cover the whole run, every
+  embedded subagent included. evalkit's token gauges read the root totals.
+- Give every embedded subagent a `trajectory_id`, unique within
+  `subagent_trajectories`, and set `subagent_trajectory_ref:
+  [{"trajectory_id": ...}]` on the result of the tool call that started it.
+  Link by `trajectory_id`; `session_id` may be shared across a run.
+  `evalkit harness check` warns when ids are missing, and a later release will
+  require them.
+- Set a subagent's `agent.name` to its type (e.g. `researcher`). evalkit
+  records it next to each subagent's metrics.
+- Set `agent.name` and `agent.version` on every trajectory.
+
+### evalkit `extra` conventions
+
+ATIF leaves room for producer metadata in `extra`. These are the only fields
+evalkit reads from it. All are optional; anything else in `extra` is kept and
+ignored.
+
+| Field | On | Type | Meaning | evalkit uses it for |
+|---|---|---|---|---|
+| `is_error` | `ObservationResult.extra` | bool | The tool call failed. Keep the error message in `content`. | `errored_tool_calls` gauge; judges and diagnosis |
+| `timestamp` | `ObservationResult.extra` | ISO 8601 string | When the result came back. | Tool time: this minus the calling step's `timestamp` |
+| `error_type` | `ObservationResult.extra` | string | Your own class for an errored result (e.g. `mcp_error`, `exit_code`). Free-form. | Grouping errors without parsing message text |
+| `full_output_path` | `ObservationResult.extra` | string | When `content` is a shortened version of the real output, a path your `GET /runs/{id}/files/{path}` serves with the full output. Keep `content` as what the agent saw. | `evalkit runs files <dir> <run_id> --path <it>` fetches it |
+| `is_error` | `Step.extra` | bool | The turn this step belongs to failed (timed out, the model call errored, the agent crashed). Put it on the step that records the failure, often a `system` step. | `failed_turns` in the MCP `get_runs` and `get_run_stats` |
+| `error` | `Step.extra` | string | The failure message, when the step's `message` is not it. | The turn's `error` in MCP `get_runs` |
+
+Conventions:
+
+- A turn is one user step and every step after it until the next user step.
+  A multi-turn chat session is one run with several user steps.
+- The final answer is NOT read from the trajectory — it comes from the run
+  status response. The trajectory endpoint returns a pure ATIF document.
+- Convert from your native session format at read time (a converter from your
+  session log to ATIF), or record ATIF natively. Keep the converter total: a
+  session your converter can't handle should fail loudly, not emit a truncated
+  document.
+
+## Verification
+
+`evalkit harness check <name>@dev` exercises: capabilities shape → the
+probe input against `input_schema` (or the default schema) → a trivial run (with
+`check_input`, else "Hi. Reply with one word.") to a terminal status with an
+`answer` or `output`, carrying one
+small input file by url when `accepts.file_urls` or as multipart when only
+`accepts.files` → trajectory ATIF validation → workspace files, run listing,
+and stop where declared. All declared capabilities must pass before running
+real experiments.

evalkit/bundled/skills/evalkit-deployment/try/.env.example

@@ -0,0 +1,11 @@
+# Copy to .env and fill in the LLM. docker compose and the evalkit CLI and SDK
+# (next to evalkit.yaml) all read .env.
+#
+# Any OpenAI-compatible endpoint that supports tool calls and structured
+# output: OpenAI, a LiteLLM proxy, vLLM, Ollama, ...
+LLM_BASE_URL=https://api.openai.com/v1
+LLM_API_KEY=
+LLM_MODEL=gpt-5-mini
+
+# The evalkit admin password on this machine.
+EVALKIT_ADMIN_PASSWORD=evalkit-admin

evalkit/bundled/skills/evalkit-deployment/try/agent.py

@@ -0,0 +1,131 @@
+import json
+import os
+from typing import Any
+
+import httpx
+from opentelemetry import trace
+
+NAME = "analyst"
+MAX_TURNS = 6
+FACTS = {
+    "revenue_fy2025": "48.2 million",
+    "revenue_fy2024": "41.5 million",
+    "gross_profit_fy2025": "18.5 million",
+    "debt_fy2025": "20.0 million",
+    "cash_fy2025": "12.7 million",
+}
+INSTRUCTIONS = (
+    "You are a financial analyst answering questions about Acme Widgets. "
+    "Look up every figure with lookup_financials; never guess one. "
+    "State figures in millions, and percentages to one decimal place."
+)
+TOOLS = [
+    {
+        "type": "function",
+        "function": {
+            "name": "lookup_financials",
+            "description": "One reported figure for Acme Widgets.",
+            "parameters": {
+                "type": "object",
+                "properties": {"metric": {"type": "string", "enum": sorted(FACTS)}},
+                "required": ["metric"],
+            },
+        },
+    }
+]
+
+tracer = trace.get_tracer("acme-analyst")
+
+
+def lookup_financials(metric: str) -> str:
+    return FACTS.get(metric, f"unknown metric {metric}")
+
+
+def gen_ai_message(message: dict[str, Any]) -> dict[str, Any]:
+    if message["role"] == "tool":
+        parts = [
+            {
+                "type": "tool_call_response",
+                "id": message["tool_call_id"],
+                "response": message["content"],
+            }
+        ]
+    else:
+        parts = [{"type": "text", "content": message["content"]}] if message.get("content") else []
+        parts += [
+            {
+                "type": "tool_call",
+                "id": call["id"],
+                "name": call["function"]["name"],
+                "arguments": json.loads(call["function"]["arguments"] or "{}"),
+            }
+            for call in message.get("tool_calls") or []
+        ]
+    return {"role": message["role"], "parts": parts}
+
+
+class Analyst:
+    def __init__(self, client: httpx.Client | None = None):
+        self.model = os.environ["LLM_MODEL"]
+        self.client = client or httpx.Client(
+            base_url=os.environ["LLM_BASE_URL"],
+            headers={"Authorization": f"Bearer {os.environ['LLM_API_KEY']}"},
+            timeout=120,
+        )
+
+    def chat(self, messages: list[dict[str, Any]]) -> dict[str, Any]:
+        with tracer.start_as_current_span(f"chat {self.model}") as span:
+            span.set_attribute("gen_ai.operation.name", "chat")
+            span.set_attribute("gen_ai.request.model", self.model)
+            span.set_attribute(
+                "gen_ai.system_instructions",
+                json.dumps([{"type": "text", "content": INSTRUCTIONS}]),
+            )
+            span.set_attribute(
+                "gen_ai.input.messages", json.dumps([gen_ai_message(m) for m in messages])
+            )
+            response = self.client.post(
+                "/chat/completions",
+                json={
+                    "model": self.model,
+                    "messages": [{"role": "system", "content": INSTRUCTIONS}, *messages],
+                    "tools": TOOLS,
+                },
+            )
+            response.raise_for_status()
+            body = response.json()
+            reply = body["choices"][0]["message"]
+            span.set_attribute("gen_ai.output.messages", json.dumps([gen_ai_message(reply)]))
+            usage = body.get("usage") or {}
+            span.set_attribute("gen_ai.usage.input_tokens", usage.get("prompt_tokens") or 0)
+            span.set_attribute("gen_ai.usage.output_tokens", usage.get("completion_tokens") or 0)
+            return reply
+
+    def call_tool(self, call: dict[str, Any]) -> str:
+        name = call["function"]["name"]
+        with tracer.start_as_current_span(f"execute_tool {name}") as span:
+            span.set_attribute("gen_ai.operation.name", "execute_tool")
+            span.set_attribute("gen_ai.tool.name", name)
+            span.set_attribute("gen_ai.tool.call.id", call["id"])
+            span.set_attribute("gen_ai.tool.call.arguments", call["function"]["arguments"])
+            arguments = json.loads(call["function"]["arguments"] or "{}")
+            result = lookup_financials(**arguments)
+            span.set_attribute("gen_ai.tool.call.result", result)
+            return result
+
+    def ask(self, question: str) -> str:
+        with tracer.start_as_current_span(f"invoke_agent {NAME}") as span:
+            span.set_attribute("gen_ai.operation.name", "invoke_agent")
+            span.set_attribute("gen_ai.agent.name", NAME)
+            messages: list[dict[str, Any]] = [{"role": "user", "content": question}]
+            for _ in range(MAX_TURNS):
+                reply = self.chat(messages)
+                messages.append(
+                    {k: v for k, v in reply.items() if k in ("role", "content", "tool_calls")}
+                )
+                if not reply.get("tool_calls"):
+                    return reply.get("content") or ""
+                for call in reply["tool_calls"]:
+                    result = self.call_tool(call)
+                    messages.append({"role": "tool", "tool_call_id": call["id"], "content": result})
+            raise RuntimeError(f"no answer after {MAX_TURNS} model calls")

evalkit/bundled/skills/evalkit-deployment/try/cases/acme/balance-sheet/cases.yaml

@@ -0,0 +1,25 @@
+cases:
+- id: acme-net-debt
+  input:
+    question: What was Acme Widgets' net debt at FY2025 year-end?
+  evaluators:
+    output:
+      answer_rubrics:
+        rubrics:
+        - rubric: States debt of 20.0 million and cash of 12.7 million.
+          type: factual_retrieval
+        - rubric: States net debt of 7.3 million.
+          type: derived_computation
+    trajectory:
+      required_tools_called:
+        tools: [lookup_financials]
+
+- id: acme-operating-margin
+  input:
+    question: What was Acme Widgets' operating margin in FY2025?
+  evaluators:
+    output:
+      answer_rubrics:
+        rubrics:
+        - rubric: Says the operating margin cannot be determined because operating income is not available, and does not state an operating margin figure.
+          type: analysis_interpretation