evalkit-py 0.2.0.dev12__py3-none-any.whl

evalkit/bundled/docs/mcp.md

@@ -0,0 +1,44 @@
+# MCP
+
+The server is also an MCP server at `{{origin}}/mcp` (streamable HTTP). An agent connected to it can read every agent's runs and trajectories, and every experiment's scores. It cannot change anything.
+
+## Connect
+
+With an evalkit login token (`evalkit login` stores it in `~/.evalkit/config.toml`):
+
+```bash
+claude mcp add --transport http evalkit {{origin}}/mcp --header "Authorization: Bearer $EVALKIT_TOKEN"
+```
+
+Only login tokens work here: agent tokens (`evalkit tokens create`) and download tickets are refused.
+
+When the server's config has an `mcp.oauth` block, MCP clients that speak OAuth (Claude, Cursor, Codex) can connect with the URL alone: the server points them to the identity provider, and a token issued for `{{origin}}/mcp` with the configured scope (`mcp.oauth.scope`, default `evalkit:read`) is accepted. Evalkit login tokens keep working next to it.
+
+```yaml
+mcp:
+  oauth:
+    issuer: https://id.example.com
+    jwks_uri: https://id.example.com/.well-known/jwks.json
+    # audience: defaults to <server.base_url>/mcp
+    # scope: your identity provider's name for read access; defaults to evalkit:read
+```
+
+An OAuth token is not an evalkit user. Whoever the identity provider gives that scope reads everything an evalkit login reads, which today is every workspace, and archiving someone in evalkit does not cut off their OAuth access: revoke it at the identity provider.
+
+## Tools
+
+| Tool | What it returns |
+|---|---|
+| `find_agents` | Agents and the id (`name@workspace`) the run tools take |
+| `find_runs` | An agent's runs in a time window, newest first: id, title, time, `user_id`, `client_id`, app or eval. `include` adds each run's `metrics`, `turns` or `tool_calls` (what it called, with which arguments). `failed_only` keeps runs with a failure; `tool_error` keeps runs where that tool errored. `timezone` reads date-only bounds as `get_run_stats` does. Up to 5,000 rows when listing alone, 200 with `include` or a filter |
+| `get_runs` | Up to 200 runs by id: turns, failed turns, failures, duration, cost and where it came from, tokens, cost by model, tool errors by tool, opening question, final answer; `include` adds `turns` and `tool_calls` |
+| `get_run_stats` | Totals over a time window, or one row per day, user, source, or tool that errored; `group_by` takes several at once and reads the window once |
+| `get_trajectory` | One run's ATIF trajectory, paged by step, long text clipped |
+| `get_run_files`, `download_run_file` | Files the run produced |
+| `find_experiments`, `get_experiments`, `get_experiment_runs` | Experiments, their scorecard, and per-case scores with evaluator explanations |
+
+Every tool answers `{data, meta}`, with `data` a list of rows. Search tools page with `limit` and `offset` (`meta.next_offset` is the next page); tools that take ids answer one row per id in input order, with `note: {"code": "not_found"}` for an unknown one. Dates are `start_date` / `end_date` (`YYYY-MM-DD`, inclusive, or an ISO 8601 timestamp). Money is `{"value", "unit": "currency", "currency": "USD"}`. Errors read `<code>: <detail>`. Tools that read trajectories have a time budget for the whole call, listing included: a run not read in time comes back as a row with `error` and `detail` instead of metrics, counted in `meta.failed` by `get_runs`; ask for it again with `get_runs`. Optional fields with no value are left out of rows. When listing alone uses the budget, the call fails with `harness_timeout`; narrow the window. With `failed_only` or `tool_error`, `find_runs` reads the window again on every call, so take one page and fetch the runs not read with `get_runs` rather than paging on.
+
+A turn is one user message and everything the agent did until the next one. `cost_source` is `reported` when the harness priced the run, `estimated` when evalkit priced it from token counts, `mixed` for both. A turn counts as failed when the harness marks a step in it with `extra.is_error` (see [Harness API](harness-api.md)).
+
+`user_id` and `client_id` are what the harness recorded. Resolve them to people and firms with your identity service.

evalkit/bundled/docs/quickstart.md

@@ -0,0 +1,142 @@
+# Quickstart
+
+From nothing to a scored experiment on an evalkit server your team already runs. Each step links to its full page; [Concepts](concepts.md) explains the terms.
+
+No server yet? [Try locally](try-locally.md) runs one on your machine with a sample agent and your own LLM, and ends with how to deploy one for your team.
+
+## 1. Pick how evalkit reaches your agent
+
+| Path                 | What you build                                                                                    | Pick it when                                                                                                     |
+| -------------------- | ------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
+| SDK run + traces     | A Python script that calls your agent once per case, and an OpenTelemetry exporter to this server | The default: start here. Runs start from your machine or CI                                                     |
+| Harness API          | An HTTP service: start a run, report its status, serve its ATIF trajectory                        | A deployed agent the server should drive (shared agents, runs started from the UI), or you need the Workspace tab |
+| Harness API + traces | The same service minus the trajectory endpoint; the trajectory comes from your spans              | You want the Harness API but your agent already emits OpenTelemetry spans                                        |
+
+The Workspace tab on a run lists the files the agent wrote. evalkit reads them through the Harness API files endpoints, so only the two Harness API paths have it. See [Harness API](harness-api.md) for what each endpoint enables.
+
+## 2. Install and log in
+
+```bash
+uv tool install --reinstall "evalkit-py=={{version}}"
+evalkit login --url {{origin}}
+```
+
+More in [Install](install.md) and [Login](login.md).
+
+## 3. Write `evalkit.yaml` and one case
+
+At the root of the repository that holds your eval cases:
+
+```yaml
+cases: eval/cases
+fixtures: eval/fixtures
+
+credentials:
+  agents:
+    myagent: env:MYAGENT_API_KEY # only for a Harness API agent that needs a key
+  llm_providers:
+    https://litellm.example.com: env:LITELLM_API_KEY # the server's judge base URL
+```
+
+Put the keys in a `.env` next to `evalkit.yaml`. Then create your agent on the server once:
+
+```bash
+# SDK path: a code-driven agent whose trajectory comes from its traces
+evalkit agents create myagent --trajectory-source otel --mapping openinference.json
+# Harness API path
+evalkit agents create myagent --harness-url http://localhost:8080/evalkit/v1
+```
+
+`openinference.json` (or `gen_ai.json`) comes from the `connect-traces` skill.
+
+Each directory under `cases` is a dataset. Add `eval/cases/smoke/cases.yaml`:
+
+```yaml
+cases:
+  - id: store-count
+    input:
+      question: "How many company-operated stores did Starbucks have at FY2021 year-end?"
+    evaluators:
+      output:
+        answer_rubrics:
+          rubrics:
+            - rubric: "States 17,133 company-operated stores at FY2021 year-end."
+              type: factual_retrieval
+```
+
+More in [Config](config.md).
+
+## 4. Run it
+
+SDK path: install the SDK into your project (see [Install](install.md)), then write `eval/run.py`:
+
+```python
+from evalkit.sdk import run_experiment, tracing
+from myagent import ask
+
+tracing.register(agent="myagent")
+
+def task(input: dict) -> str:
+    return ask(input["question"])
+
+if __name__ == "__main__":
+    run_experiment(dataset="smoke", agent="myagent", task=task, dry_run=True)
+```
+
+`dry_run=True` runs one case without creating an experiment and prints what the server made of the spans. When it looks right, drop `dry_run`; `run_experiment` then returns a result whose `url` is the experiment page. More in [SDK runs](sdk-runs.md) and [Tracing](tracing.md).
+
+Harness API path:
+
+```bash
+evalkit harness check myagent    # exercises every capability the harness declares
+evalkit run smoke --agent myagent
+```
+
+A run executes every case it selects, whether the dataset is in your own workspace (`smoke`) or a shared one (`smoke@dev`). Review state never filters a run.
+
+## 5. Read the results
+
+`evalkit run` prints the scorecard when it finishes. On this server, open the dataset under Datasets and pick the experiment. The experiment page has the scorecard; each run opens its trajectory, scores and, on the Harness API paths, its workspace files. If converted spans look wrong (missing messages, wrong span kinds), check one session with `evalkit traces check myagent <session id>` and see [Mappings](mappings.md).
+
+## 6. Attach a file to a case
+
+Drop the file in the `fixtures` directory from your `evalkit.yaml`, register it once in `fixtures.yaml` at the root of your `cases` directory, and name it in the case. The `context` line is what the agent is told about the file.
+
+```bash
+cp ~/Downloads/meridian-balance-sheet.csv <fixtures>/
+shasum -a 256 <fixtures>/meridian-balance-sheet.csv       # sha256 for the registry
+```
+
+```yaml
+# <cases>/fixtures.yaml
+fixtures:
+  - key: meridian-balance-sheet.csv
+    sha256: 652d32…eece7
+    context: Meridian's FY2025 balance sheet, USD thousands.
+```
+
+```yaml
+# in the case
+- id: net-debt-fy2025
+  fixtures: [meridian-balance-sheet.csv]
+  input:
+    question: Using only the attached file, what was net debt at FY2025 year-end?
+```
+
+A run or sync uploads any fixture whose sha256 the server doesn't hold yet, and the server keeps one copy per sha256. So you only need the files you touch, and if the bytes on disk don't match the registry the run stops and shows both hashes.
+
+## 7. Iterate and promote
+
+```bash
+evalkit evaluate <experiment-id>        # re-score the same runs after an evaluator change; no agent calls
+evalkit runs pull --experiment <experiment-id>   # download every run (trajectory, case, scores) to diagnose failures
+```
+
+Cases reach a shared dataset through a PR. On merge, CI syncs them into a shared workspace such as `dev`, where they start Unreviewed; a reviewer of that workspace approves them, and if `dev` is linked to `prod`, approval promotes them there.
+
+```bash
+evalkit review request finance@dev --case-id net-debt-fy2025 --reviewer <user>
+evalkit review approve finance@dev net-debt-fy2025   # reviewers only
+```
+
+If an admin linked `dev` to a promotion target (`evalkit workspaces promote-to dev prod`), that approval also lands the case in `finance@prod`, approved. `prod` takes cases only that way: no sync, no separate review. The case page in either workspace shows the other's version and links to it.

evalkit/bundled/docs/sdk-runs.md

@@ -0,0 +1,171 @@
+# SDK runs
+
+`run_experiment` runs an experiment from your own code: it calls your task once per case and repetition, stores the runs on this server, and scores them with the judge, the remote evaluators of the agent's workspace and any Python evaluators you pass. The agent is a code-driven agent on the server (no harness URL), created once with `evalkit agents create`. Invoking the agent, timeouts and stopping are your code.
+
+```python
+# eval/run.py — uv run python eval/run.py
+from evalkit.sdk import run_experiment, RunResult, evaluator, tracing
+from myagent import MyAgent, setup_tracing
+
+setup_tracing()                          # your own OpenTelemetry provider and instrumentors
+tracing.register(agent="myagent-sdk")   # add this server as an export destination
+agent = MyAgent()
+
+def task(input: dict, files: list) -> str:
+    return agent.ask(input["question"], attachments=[f.path for f in files])
+
+@evaluator(name="citation_check")
+def citation_check(case, run) -> float:
+    cited = {c["url"] for c in (run.output or {}).get("citations", [])}
+    return 1.0 if cited >= set(case.config["required"]) else 0.0
+
+if __name__ == "__main__":
+    result = run_experiment(dataset="finance", agent="myagent-sdk", task=task,
+                            evaluators=[citation_check], concurrency=4)
+    print(result.url)
+```
+
+## The task
+
+A sync or async callable. Parameters bind **by name** from this set, so their order does not matter; any other required parameter is an error before the first call, and an optional one keeps its default.
+
+| Parameter  | Value                                                                 |
+| ---------- | --------------------------------------------------------------------- |
+| `input`    | the case input, verbatim, minus `files`                               |
+| `files`    | `list[Fixture]`, the case's fixture files, downloaded before the call |
+| `model`    | the `model` argument of `run_experiment`, or `None`                   |
+| `metadata` | a dict with the keys below                                            |
+
+A `Fixture` has:
+
+- `name`: the fixture key the case names.
+- `path`: where the file sits on this machine.
+- `context`: the file's description from the fixture registry, or `None`. A Harness API agent gets the same text with each file; show it to your agent the same way (for example an "Attached files" block ahead of the question).
+
+`metadata` holds:
+
+- `session_id`: the SDK's session id for this run, the `session.id` of its `evalkit.run` span.
+- `experiment_id`: the experiment, or `None` on a dry run.
+- `case_id`: the case.
+- `repetition`: 1 for the first repetition, 2 for the second, and so on.
+- `tags`: the case's tags.
+
+`model` is the only way `run_experiment(model=...)` reaches your agent. The experiment records the requested model either way, so a task without a `model` parameter would silently run the agent's default; `run_experiment` prints a warning before the first call when that happens.
+
+A task with exactly one parameter whose name is not in the table (`def task(q)`) receives the input. Return a `str` (the answer), a `dict` (the structured output) or a `RunResult`:
+
+```python
+from evalkit.sdk import RunResult
+
+RunResult(answer="…", output={"citations": []}, model="gpt-5",
+          session_id="…",          # override the session the SDK polls
+          trajectory={...})        # push an ATIF document instead of tracing
+```
+
+A task that raises records the run as `failed` with the exception and the experiment continues.
+
+## An agent in another process
+
+The SDK stamps `session.id` and the `evalkit.*` attributes only on spans made in this Python process. An agent behind HTTP or a job queue, or written in another language, joins the run in one of two ways. Pick one, not both.
+
+- **Continue the trace.** Inject the W3C trace context into the request and have the agent start its spans from it. Every span in a trace belongs to the session of the trace's root, the SDK's `evalkit.run` span, even if the agent stamps its own `session.id` or `gen_ai.conversation.id`.
+
+  ```python
+  from opentelemetry import propagate
+
+  def task(input: dict) -> str:
+      headers: dict[str, str] = {}
+      propagate.inject(headers)            # adds traceparent
+      return httpx.post(AGENT_URL, json=input, headers=headers).json()["answer"]
+  ```
+
+- **Return the agent's session.** The agent puts its own session id on its root span as `session.id`, and the task returns it. Use this when the trace context cannot reach the agent, for example through a queue. The SDK moves its `evalkit.run` span into that session, so no empty session is left behind, and the trajectory is built from the agent's trace.
+
+  ```python
+  def task(input: dict) -> RunResult:
+      job = agent_api.start(input["question"])           # your API returns the session it opened
+      return RunResult(answer=job.wait(), session_id=job.session_id)
+  ```
+
+  If your API cannot tell you the session id, send it one instead: pass `metadata["session_id"]` along with the job and have the agent use it as its session id (or wrap the job in `tracing.eval_run(metadata)`, see [Tracing](tracing.md)). Its spans then join the SDK's session and the task returns only the answer.
+
+  ```python
+  def task(input: dict, metadata: dict) -> str:
+      return queue.submit(input["question"], session_id=metadata["session_id"]).wait()
+  ```
+
+Either way the Runs page links the session to its experiment and case.
+
+The agent's exporter needs its own token. Mint one with `evalkit tokens create --agent <name>`; it never expires. Your login token expires, and an exporter that holds it then drops every span with nothing but a line in its own log.
+
+## Datasets and workspaces
+
+- `dataset="finance"` runs your own workspace's copy; if `finance` is a directory under `cases` it is synced first.
+- `dataset="finance@dev"` or `dataset="finance", workspace="dev"` runs the shared cases, every review state included.
+- `case_ids=[...]`, `split="..."`, `repetitions`, `model` and `resume=<experiment id>` mean what the same options mean on `evalkit run`.
+- `dry_run=True` runs one case (`dry_run=N` runs N) without creating an experiment and prints the answer and what the server made of the session. If the server cannot read a session, that case prints a `trajectory read failed` line and the dry run goes on. Use it while wiring up tracing; the session shows on the Runs page with its case question and case id.
+
+## Notebooks
+
+`run_experiment` refuses to run inside a running event loop. In a notebook use the async form:
+
+```python
+from evalkit.sdk import arun_experiment
+
+result = await arun_experiment(dataset="finance", agent="myagent-sdk", task=task)
+```
+
+## The agent and the judge
+
+`agent="myagent-sdk"` names an agent on the server; a bare name means your own workspace. Create it once:
+
+```bash
+evalkit agents create myagent-sdk --trajectory-source otel --mapping openinference.json \
+  --max-concurrent-runs 4 --provides thinking=true
+```
+
+Its `answer` rule and `exclude_spans` live in its span mapping. `evalkit.yaml` is still required; it holds the cases, fixtures and credentials.
+
+The judge is the server's default unless you pass one:
+
+```python
+from evalkit.sdk import Judge, run_experiment
+
+run_experiment(dataset="finance", agent="myagent-sdk", task=task,
+               judge=Judge(base_url="https://litellm.example.com", model="gpt-5"))
+```
+
+An admin sets the default judge with `evalkit judge-default set`. Either way, the SDK reads the judge's key from `credentials.llm_providers` in `evalkit.yaml`, keyed by the judge's base URL.
+
+## Capabilities
+
+A code-driven agent has no `GET /capabilities`, so evalkit writes its capabilities onto each experiment itself: `trajectory`, `tool_calls`, `token_usage` and `run_listing` are true; `thinking`, `artifacts` and `workspace` are false. Evaluators gated on a capability skip when it is false. evalkit does not read capabilities from the spans. Set `provides` on the agent (`evalkit agents create --provides`, or `evalkit agents edit`) to declare what it really does, for example `thinking: true` when its spans carry the model's reasoning. Declare only what works: a false `true` produces failing scores blamed on the agent.
+
+## Python evaluators
+
+`@evaluator(name=..., kind="evaluator" | "gauge", requires_trajectory=False)` turns a function into an evaluator. It receives the same `EvalCase` and `EvalRun` the HTTP Custom Evaluator API sends: the case id, input, expected block and this evaluator's config from the case YAML; the answer, structured output and the ATIF trajectory as a dict. Return a float, a `ScoreItem`, or a list of them. A case selects it by name under `evaluators.custom`, exactly like an HTTP evaluator.
+
+## Bring your own benchmark
+
+A published benchmark is a suite plus a Python evaluator:
+
+1. Write a script that turns the benchmark's file into case YAML. Each case gets the question as `input`. Its grading data (rubrics, reference answer) goes under `evaluators.custom.<grader name>`.
+2. `evalkit sync <dir>` puts it in a dataset like any suite.
+3. Write the grader as an `@evaluator` with that name. It reads the grading data from `case.config`, and calls its own judge models if it needs any. Pass it to `run_experiment`.
+
+`evalkit lint` and `evalkit sync` warn with `W-CUSTOM-UNKNOWN` for the grader name, because the server can't see a Python evaluator. That warning is expected.
+
+## Re-scoring
+
+Python evaluators live in your script, so a re-score that needs them starts from the script:
+
+```python
+from evalkit.sdk import evaluate_experiment
+
+evaluate_experiment(experiment_id="…", evaluators=[citation_check],
+                    refetch_trajectories=True)
+```
+
+`refetch_trajectories` polls the collector again for runs whose trajectory was `partial` or `unavailable` and re-scores them. `evalkit evaluate` refuses experiments that used Python evaluators and points here.
+
+Next: [Tracing](tracing.md).

evalkit/bundled/docs/tracing.md

@@ -0,0 +1,171 @@
+# Tracing
+
+The collector is an OTLP/HTTP endpoint on this server:
+
+```
+POST {{origin}}/api/otel/v1/traces
+```
+
+It accepts `application/x-protobuf` and `application/json`, gzip or not, from any OpenTelemetry SDK. Spans are stored as received and converted to ATIF with the agent's mapping when a run or the Runs page asks for them.
+
+## From Python with the SDK
+
+```python
+from evalkit.sdk import tracing
+
+tracing.register(agent="myagent-sdk")
+```
+
+`register` adds an exporter for this server to your global tracer provider, creating one if none is set. Your existing exporters keep running. The exporter carries an `X-Evalkit-Agent` header, which names the agent on the server; the `evalkit.agent` resource attribute is only set when `register` creates the provider itself. It reads the server URL from `EVALKIT_SERVER_URL` or your `evalkit login`, and the token from `EVALKIT_OTEL_TOKEN` (an agent token), then `EVALKIT_TOKEN`, then your `evalkit login`. It needs the `otel` extra.
+
+### An existing span pipeline
+
+`register` changes the global provider. When you build your own pipeline, take evalkit's parts and put them where you need them:
+
+```python
+from evalkit.sdk import tracing
+
+exporter = tracing.exporter(agent="myagent-sdk")    # OTLPSpanExporter: URL, token and X-Evalkit-Agent set
+processor = tracing.processor(agent="myagent-sdk")  # batch processor around that exporter
+provider.add_span_processor(processor)              # what register does for you
+headers = tracing.headers(agent="myagent-sdk")      # the same headers, for an exporter you build yourself
+```
+
+All three resolve the server and token the same way `register` does. `processor` also stamps the run's `session.id` and `evalkit.*` attributes on spans that start under a task call (or under `tracing.eval_run`, below).
+
+Spans handed straight to a processor or exporter, for example by a bridge that converts another tool's spans, never pass through the provider, so `register` cannot see them. Hand them to `tracing.processor(...)` too: give the bridge that processor, or one processor that fans out to all of yours and evalkit's (OpenTelemetry's `SynchronousMultiSpanProcessor`). Those spans are already finished, so `processor` cannot stamp them: the bridge copies `session.id` onto each span it converts. How eval-only export treats them is below.
+
+### Choosing what a shared worker exports
+
+A worker that serves both app runs (production, staging or manual use) and eval runs exports every span it makes, so every app turn also becomes a session on this server, which you can browse on the Runs page and pull with `evalkit runs pull <agent> --source app`. Whether to send app traffic is your choice. `eval_run` marks eval runs either way, and to send only those, filter on the mark:
+
+```python
+from evalkit.sdk import tracing
+
+provider.add_span_processor(tracing.processor(agent="myagent", only=tracing.is_eval_span))
+
+def handle_job(job):
+    if job.evalkit:                           # the task's metadata, sent along with the job
+        with tracing.eval_run(job.evalkit):
+            return run_agent(job)
+    return run_agent(job)
+```
+
+The task passes its `metadata` argument along with the job (see [SDK runs](sdk-runs.md)). `eval_run` stamps `session.id` (the SDK's session id), `evalkit.case_id`, `evalkit.experiment_id` and `evalkit.repetition` on every span started inside it, so the worker's spans join the run's session and the task returns only the answer. `is_eval_span` is true for a span that carries `evalkit.case_id`. The stamp lives in a context variable: spans started in another thread need that thread to run inside `eval_run` too.
+
+A harness agent learns that a run is an eval run from the `metadata` of `POST /runs`: `experiment_id`, `case_id` and `repetition`. That is what it passes to `eval_run`, with the `run_id` it returns as the session id: `tracing.eval_run({**metadata, "session_id": run_id})`.
+
+`only` decides by trace, not by span. It is any function that takes a span and returns a bool, and it runs on each span as the span starts and again as it ends. Once it is true for one span, the processor keeps every span of that span's trace that reaches it: spans started inside `eval_run`, the job's own spans started before `eval_run` in the same trace, and finished spans a bridge hands straight to the processor. Spans of other traces are dropped. A custom `only` picks whole traces the same way.
+
+The rule for order: a span is kept when its trace is already marked by the time the span ends. The first span started inside `eval_run` marks its trace as it starts, before anything under it ends, so spans that end before it and spans a bridge hands over later are both kept. A span that ends before any span of its trace was marked is dropped. A bridge that may hand spans over that early can copy the `evalkit.*` attributes onto them, so each one marks its own trace. A bridge should copy `session.id` either way: the server takes a trace's session from its root span when the root reaches it, and from each span's own `session.id` otherwise.
+
+The processor remembers the 10,000 most recently used eval traces and forgets the oldest. Each kept span refreshes its trace, so a trace is only forgotten once 10,000 other eval traces have been used since its last span.
+
+Spans from another process in the same trace, such as an MCP gateway pod the agent calls, never reach this processor. They go through that process's own exporter, so that process makes its own choice: either it continues the trace, wraps its work in `eval_run` with the run's metadata sent along with the request, and filters with `only=tracing.is_eval_span` too, or it exports everything.
+
+Do not use a sampler for this: a sampler drops spans for every exporter, your own backend included.
+
+Around every task call the SDK opens a root span named `evalkit.run` with `session.id` set to the run id, and stamps `session.id`, `evalkit.experiment_id`, `evalkit.case_id` and `evalkit.repetition` onto every span your code starts underneath it. That is what groups a trace into one session. When the task returns `RunResult(session_id=...)`, the root span takes that session id instead, so it joins the agent's session rather than sitting alone; see [SDK runs](sdk-runs.md) for agents in another process.
+
+## From any OpenTelemetry SDK
+
+Point the exporter at the collector and name the agent, either with a header or an attribute:
+
+```bash
+export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT={{origin}}/api/otel/v1/traces
+export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer <token>,X-Evalkit-Agent=myagent"
+export OTEL_RESOURCE_ATTRIBUTES="evalkit.agent=myagent"   # alternative to the header
+```
+
+The collector picks the agent per span: the span's `evalkit.agent` attribute first, then the resource's, then the `X-Evalkit-Agent` header. One process that serves several agents can therefore share one exporter and set `evalkit.agent` on each agent's spans. The value is an agent reference (`myagent` or `myagent@workspace`). Every agent a batch names is checked as if it came in the header: an agent token writes only its own agent, a login token only agents in workspaces you can write to (your own, or a shared one where you are a writer). If any span names an agent the token may not write, the whole batch is refused.
+
+Put the session id on the root span of each trace, in the attribute your mapping's `session_id` rule names (`session.id` in `openinference.json`, `gen_ai.conversation.id` in `gen_ai.json`). Every span in a trace belongs to the session of the trace's root, even one that carries a session attribute of its own. Only when the root has no session do child spans keep theirs.
+
+## Deployed agents
+
+A service cannot carry a person's token. Mint an agent token, bound to one agent and valid only for the collector:
+
+```bash
+evalkit tokens create --agent ouragent@dev     # needs writer on dev
+evalkit tokens list
+evalkit tokens revoke <id>
+```
+
+Set it on the deployment as `EVALKIT_OTEL_TOKEN` next to `EVALKIT_SERVER_URL`; `tracing.register`, `tracing.exporter` and `tracing.processor` pick both up. For a non-Python exporter put it in `OTEL_EXPORTER_OTLP_HEADERS`. Use an agent token for any exporter outside the process that calls `run_experiment` too, even on your laptop: the login token expires under a long-running worker.
+
+The collector logs every refused export as `otel export rejected` with the status, the reason (`expired token`, `invalid token`, `unknown agent`, `forbidden`, `too large`, `malformed`) and the agent. OpenTelemetry exporters only log a failed export on their side, so a silent agent is worth checking in both logs.
+
+## Size limits and large spans
+
+LLM spans from OpenInference and GenAI instrumentations carry the full input of each call: the messages so far, the system prompt and the tool schemas. Every turn repeats the whole context, so a long agent session grows with roughly the square of its turns. A 14-minute chat with subagents can store hundreds of MB of span attributes.
+
+The server has two limits, both set in its config:
+
+| Limit                    | Default | What happens past it                                                                                                                                                                                                                      |
+| ------------------------ | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `otel.max_batch_bytes`   | 16 MiB  | One export request (after gzip is undone) gets `413` and its spans are dropped. Exporters do not split a refused batch, so keep batches small.                                                                                            |
+| `otel.max_session_bytes` | 384 MiB | Reading the session (a run's trajectory, `evalkit traces check`, the Mappings tab) gets `413` with the session's size and the limit. A run records the trajectory as `unavailable` and logs the message. `--dump spans.json` still works. |
+
+`otel.max_session_bytes` counts the span attributes a trajectory build loads, not everything stored. evalkit reads the input messages (and system prompt) of an LLM span only for the first LLM call of each agent and subagent; later calls add only their output, tool calls and tokens to the trajectory. So the build leaves the input of every later LLM call in the database and does not count it. The limit keeps one session from using up the server's memory, so raise it only together with the server's memory limit.
+
+For agents with large spans, send fewer spans per export and give each export more time. For the Python SDK, `tracing.register` included:
+
+```bash
+export OTEL_BSP_MAX_EXPORT_BATCH_SIZE=16   # spans per export; default 512
+export OTEL_BSP_MAX_QUEUE_SIZE=2048        # spans waiting for export; raise it if the exporter logs "Queue full"
+export OTEL_EXPORTER_OTLP_TIMEOUT=60       # seconds per export in Python; default 10
+```
+
+Other SDKs follow the OpenTelemetry spec and read `OTEL_EXPORTER_OTLP_TIMEOUT` in milliseconds (`60000`). A 10 MB span can take more than 10 s to upload, longer than the default timeout. In an OpenTelemetry Collector, set `send_batch_max_size` on the `batch` processor and `timeout` on the `otlphttp` exporter instead.
+
+## Exporting to evalkit and another backend
+
+The standard OTLP variables name one endpoint. To keep exporting to your tracing backend and send the same spans here, either:
+
+- add a second `otlphttp` exporter to the OpenTelemetry Collector the agent already exports to (recommended for deployed agents), or
+- add a second span processor and exporter in code. `tracing.register` does this for Python.
+
+```yaml
+exporters:
+  otlphttp/evalkit:
+    traces_endpoint: {{origin}}/api/otel/v1/traces
+    headers:
+      Authorization: Bearer ${env:EVALKIT_OTEL_TOKEN}
+      X-Evalkit-Agent: ouragent@dev
+service:
+  pipelines:
+    traces:
+      exporters: [otlp/existing, otlphttp/evalkit]
+```
+
+The collector speaks OTLP over HTTP only, not gRPC. Bodies may be chunked with no `Content-Length` (the JavaScript exporter sends them that way); a proxy in front of the collector must allow that.
+
+## The main agent and subagents
+
+Mark the agent's top-level span as an AGENT span (`openinference.span.kind=AGENT`, or `gen_ai.operation.name=invoke_agent`). evalkit builds the main trajectory from it, and every AGENT span inside it becomes a subagent trajectory named by the span's `agent.name` (`gen_ai.agent.name`), or by the span name when that is missing. When the root span is not an AGENT span, as with the SDK's `evalkit.run` span, evalkit takes the earliest AGENT span that is not inside another one as the main agent. If that span is a tool call that starts a subagent partway through the run, the subagent's steps land in the main trajectory and the call has no result, so mark the top-level span as an AGENT span rather than relying on this.
+
+## Harness agents with traces
+
+A harness agent created with `--trajectory-source otel` keeps the whole Harness API flow and skips `GET /runs/{id}/trajectory`: it puts the run id it returned on its spans, in the attribute its mapping's `session_id` rule names, and evalkit polls the session until it settles. The harness declares `provides.trajectory: false`; evalkit still runs the trajectory evaluators because the trajectory comes from the collector.
+
+The session evalkit polls is the `run_id` that `POST /runs` returns, so there is no separate session field. A queue-backed agent whose worker picks its own session id returns that id as `run_id`, or puts the `run_id` it returned on its spans in that session attribute. To export only eval runs from a harness that also serves production, see Choosing what a shared worker exports above.
+
+## Importing history from a trace source
+
+An admin registers a trace source (a Phoenix server) once. After that, any
+user can import traces from it into an OTel agent they can write, with
+`evalkit traces import` or the agent page's Import tab.
+
+- The trace source's API key is shared with every user in effect: anyone can
+  import any project that key can read.
+- Imported traces land in the agent's workspace. Every user can read every
+  workspace, so traces imported into your personal workspace are visible to
+  everyone on the server. See [Where runs happen](workspaces.md).
+
+## Checking a session
+
+```bash
+evalkit traces check myagent-sdk <session id>
+```
+
+prints the settle state, span kinds, tokens, the attributes the mapping did not read and the ATIF summary. `--dump spans.json` writes the raw spans. `--dump esf.json` leaves out the input messages of every LLM call after each agent's first, as the trajectory does not use them; `spans.json` has the raw values. A session over `otel.max_session_bytes` fails the check with its size and the limit (see Size limits and large spans above). If the kinds or messages look wrong, the agent needs a mapping: see [Mappings](mappings.md).