evalkit-py 0.2.0.dev12__py3-none-any.whl

evalkit/bundled/skills/connect-traces/SKILL.md

@@ -0,0 +1,296 @@
+---
+name: connect-traces
+description: Use when wiring a client's OpenTelemetry traces into evalkit so run_experiment or a harness run gets trajectories, when creating an OTel agent (it needs a span mapping), or when `evalkit traces check` shows wrong span kinds, empty messages or missing tokens.
+---
+
+# connect-traces
+
+## Overview
+
+evalkit scores an agent on its **trajectory**: the LLM calls, messages, tool
+calls and token counts of one run. When the agent emits OpenTelemetry spans,
+evalkit builds that trajectory from the spans itself: the agent exports every
+span to the evalkit server (the **collector**), the server groups the spans of
+one run into an **agent session** by the `session.id` attribute, reads each
+span into the evalkit span format (**ESF**) with a **mapping**, and converts
+the session to ATIF. Nothing in the client's code converts anything.
+
+Your job ends when `evalkit traces check` shows the right span kinds,
+messages, tool calls and tokens for one session. Scoring, scorecards and the
+ATIF conversion are evalkit's side and need nothing from you.
+
+Traces replace the trajectory only. The files a run wrote (the run's
+Workspace tab in evalkit) come from the Harness API files endpoints, which
+spans do not carry. If the developer wants that tab, the agent needs a
+Harness API with those endpoints (see the evalkit-usage skill).
+
+Every OTel agent has its own mapping, set when the agent is created. Two
+starting points ship in `references/`: `openinference.json` for spans with
+`openinference.span.kind`, and `gen_ai.json` for spans with
+`gen_ai.operation.name`. Copy the one that
+fits and change it if the agent's attributes differ. The mapping also holds
+two run-level settings:
+
+- `answer`: where the run's answer comes from. `root` (the usual value), a
+  span name, or `<span name>.<output key>`.
+- `exclude_spans`: span names left out of the trajectory.
+
+A span the agent's mapping cannot read is not an error: it comes out as kind
+`other` with every attribute listed as unmapped, and the trajectory simply
+lacks it. That is what a wrong mapping looks like in the check.
+
+## Steps
+
+1. **Identify the instrumentation.** Grep the repository for `openinference`
+   and `gen_ai.`. Note which one applies, or neither. Then make sure the
+   instrumentation records message content: without it evalkit gets token
+   counts but no messages, tool arguments or results, and the trajectory is
+   close to empty. The GenAI conventions turn content capture off by
+   default; the usual switches are:
+
+   | Instrumentation | Switch |
+   | -- | -- |
+   | OpenTelemetry GenAI instrumentations (Python `opentelemetry-instrumentation-openai-v2`, Google GenAI, Vertex AI, …) | `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true` |
+   | OpenInference | on by default; check `OPENINFERENCE_HIDE_INPUTS` / `OPENINFERENCE_HIDE_OUTPUTS` (and the `_MESSAGES` variants) are not `true` |
+   | OpenLLMetry (Traceloop) | on by default; check `TRACELOOP_TRACE_CONTENT` is not `false` |
+   | Vercel AI SDK | `experimental_telemetry: { isEnabled: true }`; `recordInputs` / `recordOutputs` default to on |
+
+   Turn it on wherever spans are exported to evalkit: the process that runs
+   the eval, and any deployed agent or worker those runs call. Whether a
+   deployed agent also exports its app traffic (runs outside any experiment:
+   production, staging, or by hand) is the developer's choice; ask them.
+   evalkit shows app sessions on the Runs page too, and `evalkit runs pull
+   <agent> --source app` downloads them. Exporting app traffic is also what
+   lets the eval-driven-development skill mine new cases from real runs
+   (its Loop 2).
+   To export only eval runs, see "Choosing what a shared worker exports"
+   below.
+2. **Create the agent with a mapping.** Copy the starting point from
+   step 1 into the repo and create the agent with it. Add `--harness-url`
+   when the agent also has a Harness API; leave it out for a code-driven
+   agent.
+
+   ```bash
+   cp <this skill>/references/openinference.json eval/span-mapping.json
+   evalkit agents create <agent> --trajectory-source otel --mapping eval/span-mapping.json
+   ```
+
+   The agent must exist before any span arrives: the collector refuses spans
+   for an unknown agent, and neither a token nor the first export creates
+   one.
+3. **Register the exporter.** *After* the client's own tracer provider is set
+   up, add
+
+   ```python
+   from evalkit.sdk import tracing
+
+   tracing.register(agent="<agent name>")
+   ```
+
+   `register` adds an exporter to the provider that exists; placed before the
+   client's setup it creates a second provider and the agent's spans never
+   reach it. It needs the `otel` extra (`uv add "evalkit-py[otel]==<server version>"`).
+   When the client builds its own pipeline and should not touch the global
+   provider, use the parts `register` is made of:
+   `tracing.processor(agent=...)` returns evalkit's batch processor (it also
+   stamps the run's `session.id` and `evalkit.*` attributes),
+   `tracing.exporter(agent=...)` the OTLP HTTP exporter inside it, and
+   `tracing.headers(agent=...)` its headers. All three read the server URL
+   and token like `register` does, `EVALKIT_OTEL_TOKEN` first.
+   `register` only sees spans that pass through the provider's span
+   processors. Code that hands spans straight to one processor or exporter
+   (a bridge that converts another tool's spans, for example) bypasses it;
+   give that code `tracing.processor(...)` as well, or one
+   `SynchronousMultiSpanProcessor` that holds every processor, evalkit's
+   included. Such spans are already finished, so the processor cannot stamp
+   them: the bridge copies `session.id` onto each span it converts.
+   A non-Python agent, or any exporter outside the process that calls
+   `run_experiment`, sets
+   `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=<server>/api/otel/v1/traces` and
+   `OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer <token>,X-Evalkit-Agent=<agent>"`
+   instead, with an agent token from `evalkit tokens create --agent <agent>`.
+   The login token expires and a long-running worker then loses every span
+   without an error of its own. One process that serves several agents can
+   name the agent per span or per resource with the `evalkit.agent`
+   attribute; the header is the default for spans that carry none.
+   LLM spans that carry the full input of every call (OpenInference and
+   GenAI instrumentations do) are large, so also set
+   `OTEL_BSP_MAX_EXPORT_BATCH_SIZE=16` and `OTEL_EXPORTER_OTLP_TIMEOUT=60`
+   on every process that exports (seconds in Python; other SDKs read
+   milliseconds, `60000`). The server refuses a batch over
+   `otel.max_batch_bytes` (16 MiB by default) with `413` and the spans are
+   lost, and the default 10 s timeout is too short for multi-MB spans.
+4. **Run one session.** `run_experiment(..., dry_run=True)` runs one case,
+   creates no experiment, and prints the session id (`session <id>: N spans
+   ...`). The agent's own entry point works too; the Runs page lists the
+   session either way.
+5. **Check it.**
+
+   ```bash
+   evalkit traces check <agent> <session id> --dump spans.json
+   ```
+
+   Read the lines against this table. `spans.json` holds the raw attributes.
+
+   | Line | What it tells you |
+   | -- | -- |
+   | `session` | span count, trace count, settled or not |
+   | `root` | the root span's name and kind; missing root means the session id is not on the root span |
+   | `mapping` | the active version, `vN` |
+   | `llm` | spans read as `llm`, the model, token totals; `no llm spans` means the kind rule does not match; `without messages` means those spans carry no messages, usually because message content capture is off |
+   | `tools` | spans read as `tool`, and how many carry a `tool_call_id` that matches a call the LLM spans emitted; a gap means tool spans exist but their call ids are not read, so results only match by time and order; `in no step` names tool spans that started before any LLM span and so are not in the trajectory |
+   | `unmapped` | attributes no rule read, apart from the mapping's `known_attributes` |
+   | `trajectory` | ATIF steps, tool calls, subagents |
+
+   Then check the agent's top-level span is an AGENT span
+   (`openinference.span.kind=AGENT` or `gen_ai.operation.name=invoke_agent`).
+   evalkit builds the main trajectory from it, and every AGENT span inside it
+   becomes a subagent named by its `agent.name` (`gen_ai.agent.name`). When
+   the root span is not an AGENT span, evalkit takes the earliest AGENT span
+   that is not inside another one as the main agent. If that span is a tool
+   call that starts a subagent partway through the run, the subagent's steps
+   end up in the main trajectory and its call has no result. Fix that in the
+   agent's instrumentation: mark its top-level span as an AGENT span.
+
+   If the check fails with `413` and names `otel.max_session_bytes`, the
+   session holds more span attributes than the server will build into a
+   trajectory (384 MiB by default). The limit counts what a build loads: the
+   input messages of each agent's first LLM call, and the output, tool calls
+   and tokens of every LLM call, not the input of later calls. Check a shorter session, and tell the
+   developer: the deployment's admin can raise the limit together with the
+   server's memory.
+
+   When the lines look right, stop.
+6. **Fix the mapping** when they do not. Edit `eval/span-mapping.json` (it is
+   JSON, one rule per ESF field) and change the attribute keys to the ones in
+   `spans.json`.
+   `references/span-mapping-schema.md` explains every rule shape and the
+   three message formats. Push it as a draft and check with it:
+
+   ```bash
+   evalkit traces mapping push <agent> eval/span-mapping.json
+   evalkit traces check <agent> <session id> --mapping draft
+   ```
+
+   Repeat until the unmapped list holds nothing that matters and the counts
+   are right. A push replaces the previous draft and gets the next version.
+7. **Activation needs the developer's approval.** If the developer has
+   approved the document in this conversation, run
+   `evalkit traces mapping push <agent> eval/span-mapping.json --activate`.
+   Otherwise tell them to review it on the agent's Mappings tab (Agents →
+   the agent → Mappings in the web UI) and activate it there. Green check
+   output is not approval; a mapping applies to every future run of the agent.
+
+## An agent in another process
+
+`tracing.register` stamps `session.id` and the `evalkit.*` attributes only on
+spans made in the process that calls `run_experiment`. An agent behind HTTP
+or a job queue, or written in another language, joins the run in one of two
+ways. Pick one, not both.
+
+- **Continue the trace.** The task injects the W3C trace context into the
+  request (`opentelemetry.propagate.inject(headers)`) and the agent starts its
+  spans from it (`traceparent`). Every span in a trace belongs to the session
+  of the trace's root, the SDK's `evalkit.run` span, even when the agent
+  stamps its own `session.id` or `gen_ai.conversation.id`. The task returns
+  only the answer.
+- **Return the agent's session.** The agent puts its own session id on its
+  root span as `session.id`, and the task returns
+  `RunResult(answer=..., session_id=<that id>)`. Use this when the trace
+  context cannot reach the agent, for example through a queue. The SDK moves
+  its `evalkit.run` span into that session so no empty session is left
+  behind, and the server builds the trajectory from the agent's trace.
+
+Either way the run's row on the Runs page links the session to its case.
+
+A queue-backed agent usually opens its session in the worker, after the API
+call has returned. Two ways to get the session id to evalkit, no new field
+needed on either path:
+
+- **Code-driven (`run_experiment`).** The task returns
+  `RunResult(session_id=...)` with whatever id the agent's own API returns.
+  If the API cannot return one, the task passes `metadata["session_id"]`
+  (the SDK's own session id) along with the job, and the agent uses it as
+  its session id; the task then returns only the answer.
+- **Harness API with an `otel` trajectory source.** The session evalkit polls is the `run_id`
+  that `POST /runs` returns. Return the agent's own session id as `run_id`,
+  or stamp the returned `run_id` on the job's spans as `session.id`.
+
+## Choosing what a shared worker exports
+
+A worker that serves app and eval runs exports every span it makes, so each
+app turn also becomes a session on the evalkit server, where
+the developer can browse it on the Runs page. That is a valid setup; ask the
+developer whether they want it. evalkit provides the stamping to tell eval
+runs apart either way. To export only the eval runs:
+
+```python
+from evalkit.sdk import tracing
+
+provider.add_span_processor(tracing.processor(agent="<agent>", only=tracing.is_eval_span))
+
+
+def handle_job(job):
+    if job.evalkit:  # the task's metadata, sent with the job
+        with tracing.eval_run(job.evalkit):
+            return run_agent(job)
+    return run_agent(job)
+```
+
+`eval_run` stamps `session.id` (the SDK's session id) and the `evalkit.*`
+attributes on every span started inside it, so those spans also join the
+run's session. `is_eval_span` is true for a span that carries
+`evalkit.case_id`. Spans started in another thread need that thread inside
+`eval_run` too. Do not use a sampler: it drops spans for every exporter, the
+client's own backend included.
+
+A harness agent learns that a run is an eval run from the `metadata` of
+`POST /runs` (`experiment_id`, `case_id`, `repetition`). That is what it
+passes to `eval_run`, with the `run_id` it returns as the session id:
+`tracing.eval_run({**metadata, "session_id": run_id})`.
+
+`only` decides by trace. It runs on each span as it starts and as it ends;
+once it is true for one span, the processor keeps every span of that trace
+that reaches it, including finished spans a bridge hands straight to
+`processor.on_end` and the job's spans started before `eval_run`. Other
+traces are dropped. A custom `only` picks whole traces the same way.
+
+- **Order.** A span is kept when its trace is already marked as the span
+  ends. The first span started inside `eval_run` marks the trace as it
+  starts, so spans that end before it and spans bridged later are kept. A
+  span that ends before any span of its trace was marked is dropped; a
+  bridge that may hand spans over that early can copy the `evalkit.*`
+  attributes onto them so each marks its own trace.
+- **Bridges.** Still copy `session.id` onto every converted span: the server
+  takes a trace's session from its root span when the root reaches it, and
+  from each span's own `session.id` otherwise.
+- **Memory.** The processor remembers the 10,000 most recently used eval
+  traces and forgets the oldest.
+- **Other processes.** Spans from another process in the same trace (an MCP
+  gateway pod, for example) never reach this processor; they go through that
+  process's own exporter. That process makes its own choice: continue the
+  trace, wrap its work in `eval_run` with the run's metadata sent along with
+  the request and filter with `only=tracing.is_eval_span`, or export
+  everything. Ask the developer which.
+
+A worker in another language does the same with its own processor: forward
+a span to the evalkit exporter only when its trace has a span carrying an
+attribute the worker set on eval jobs.
+
+## Capabilities of a code-driven agent
+
+A code-driven agent has no capabilities endpoint, so evalkit records
+`thinking: false` (and `artifacts`, `workspace` false) for it and never reads
+capabilities from spans. When the agent's spans carry the model's reasoning,
+set it on the agent:
+
+```bash
+evalkit agents edit <agent> --provides thinking=true
+```
+
+Declare only what works; evaluators gated on a flag trust it.
+
+If the agent's spans cannot be expressed with the rule shapes in the schema
+reference, open an evalkit issue rather than working around it.
+
+For bugs, feature requests and feedback on evalkit, open an issue at https://github.com/inferedge-ai/evalkit/issues; questions go to https://github.com/inferedge-ai/evalkit/discussions.

evalkit/bundled/skills/connect-traces/references/gen_ai.json

@@ -0,0 +1,42 @@
+{
+  "schema_version": "1",
+  "kind": {
+    "attribute": "gen_ai.operation.name",
+    "values": {
+      "chat": "llm", "text_completion": "llm", "generate_content": "llm",
+      "embeddings": "other", "execute_tool": "tool", "invoke_agent": "agent",
+      "create_agent": "other"
+    }
+  },
+  "session_id": {"attribute": "gen_ai.conversation.id"},
+  "model": {"attribute": "gen_ai.request.model"},
+  "system_instructions": {"attribute": "gen_ai.system_instructions", "json": true},
+  "input_messages": {"attribute": "gen_ai.input.messages", "message_format": "gen_ai"},
+  "output_messages": {"attribute": "gen_ai.output.messages", "message_format": "gen_ai"},
+  "tokens": {
+    "prompt": {"attribute": "gen_ai.usage.input_tokens"},
+    "completion": {"attribute": "gen_ai.usage.output_tokens"},
+    "cached": {"attribute": "gen_ai.usage.cache_read.input_tokens"},
+    "cache_write": {"attribute": "gen_ai.usage.cache_creation.input_tokens"}
+  },
+  "cost_usd": null,
+  "tool": {
+    "name": {"attribute": "gen_ai.tool.name"},
+    "call_id": {"attribute": "gen_ai.tool.call.id"},
+    "arguments": {"attribute": "gen_ai.tool.call.arguments", "json": true},
+    "result": {"attribute": "gen_ai.tool.call.result"}
+  },
+  "agent_name": {"attribute": "gen_ai.agent.name"},
+  "input": {"attribute": "gen_ai.input.messages", "json": true},
+  "output": {"attribute": "gen_ai.output.messages", "json": true},
+  "known_attributes": [
+    "gen_ai.system", "gen_ai.provider.name", "gen_ai.agent.*", "gen_ai.main_agent.*",
+    "gen_ai.conversation.id", "gen_ai.conversation.compacted", "gen_ai.data_source.id",
+    "gen_ai.operation.name", "gen_ai.output.type", "gen_ai.prompt.*", "gen_ai.request.*",
+    "gen_ai.response.*", "gen_ai.skill.*", "gen_ai.system_instructions",
+    "gen_ai.token.modality", "gen_ai.tool.definitions", "gen_ai.tool.description",
+    "gen_ai.tool.type", "gen_ai.usage.reasoning.output_tokens", "gen_ai.usage.audio.*",
+    "gen_ai.usage.image.*", "gen_ai.usage.text.*", "gen_ai.workflow.name",
+    "server.address", "server.port", "error.type"
+  ]
+}

evalkit/bundled/skills/connect-traces/references/openinference.json

@@ -0,0 +1,35 @@
+{
+  "schema_version": "1",
+  "kind": {
+    "attribute": "openinference.span.kind",
+    "values": {"LLM": "llm", "TOOL": "tool", "AGENT": "agent", "CHAIN": "chain", "RETRIEVER": "retriever"}
+  },
+  "session_id": {"attribute": "session.id"},
+  "model": {"attribute": "llm.model_name"},
+  "input_messages": {"attribute": "llm.input_messages", "message_format": "openinference"},
+  "output_messages": {"attribute": "llm.output_messages", "message_format": "openinference"},
+  "tokens": {
+    "prompt": {"attribute": "llm.token_count.prompt"},
+    "completion": {"attribute": "llm.token_count.completion"},
+    "cached": {"attribute": "llm.token_count.prompt_details.cache_read"},
+    "cache_write": {"attribute": "llm.token_count.prompt_details.cache_write"}
+  },
+  "cost_usd": {"attribute": "llm.cost.total"},
+  "tool": {
+    "name": {"attribute": "tool.name"},
+    "call_id": {"attribute": "tool.id"},
+    "arguments": {"attribute": "input.value", "json": true},
+    "result": {"attribute": "output.value"}
+  },
+  "agent_name": {"attribute": "agent.name"},
+  "input": {"attribute": "input.value", "json": true},
+  "output": {"attribute": "output.value", "json": true},
+  "known_attributes": [
+    "openinference.span.kind", "session.id", "user.id", "metadata", "tag.tags",
+    "input.mime_type", "output.mime_type", "llm.provider", "llm.system",
+    "llm.invocation_parameters", "llm.finish_reason", "llm.tools.*",
+    "llm.token_count.total", "llm.token_count.prompt_details.*",
+    "llm.token_count.completion_details.*", "llm.cost.*", "tool.description",
+    "tool.parameters", "tool.json_schema", "agent.name", "graph.node.*"
+  ]
+}

evalkit/bundled/skills/connect-traces/references/span-mapping-schema.md

@@ -0,0 +1,88 @@
+# Span mapping schema (version 1)
+
+A mapping is one JSON document with one rule per ESF field. Every mapping is
+complete on its own; there is no inheritance. The JSON Schema ships as
+`span-mapping-v1.json` with `evalkit contracts export`.
+
+```json
+{
+  "schema_version": "1",
+  "kind": {"attribute": "...", "values": {"<raw>": "llm|tool|agent|chain|retriever|other"}},
+  "session_id": {"attribute": "..."},
+  "model": {"attribute": "..."},
+  "system_instructions": {"attribute": "...", "json": true},
+  "input_messages": {"attribute": "...", "message_format": "openinference|gen_ai|openai"},
+  "output_messages": {"attribute": "...", "message_format": "openinference|gen_ai|openai"},
+  "tokens": {
+    "prompt": {"attribute": "..."},
+    "completion": {"attribute": "..."},
+    "cached": {"attribute": "..."},
+    "cache_write": {"attribute": "..."}
+  },
+  "cost_usd": {"attribute": "..."},
+  "tool": {
+    "name": {"attribute": "..."},
+    "call_id": {"attribute": "..."},
+    "arguments": {"attribute": "...", "json": true},
+    "result": {"attribute": "..."}
+  },
+  "agent_name": {"attribute": "..."},
+  "input": {"attribute": "...", "json": true},
+  "output": {"attribute": "...", "json": true},
+  "known_attributes": ["...", "prefix.*"],
+  "answer": "root",
+  "exclude_spans": ["..."]
+}
+```
+
+Two fields apply to the whole run, not to one span:
+
+- `answer` (default `root`): where the run's answer comes from. `root`, a
+  span name, or `<span name>.<output key>`.
+- `exclude_spans` (default empty): span names left out of the trajectory.
+
+## Rule shapes
+
+| Shape | What evalkit does |
+| -- | -- |
+| `{"attribute": K}` | reads attribute `K`; null when absent |
+| `{"attribute": K, "json": true}` | reads `K` and parses it with `json.loads` when it is a string; a parse failure keeps the string |
+| `{"attribute": K, "values": {...}}` | reads `K` and translates the value through the table; for `kind`, an unlisted value becomes `other` |
+| `{"attribute": K, "message_format": F}` | collects the messages laid out per format `F` under `K` (see below) |
+| `null` | the field is never read |
+
+Which fields are read depends on the kind: `system_instructions`,
+`input_messages`, `output_messages` and the token rules on `llm` spans; the `tool` rules on
+`tool` spans, and `tool.call_id` on `agent` spans too (the call that started
+a subagent); `agent_name` on `agent` spans (it names the subagent's
+trajectory; the span name is used without it); `input` and `output` on everything that is not an llm span.
+Attributes no rule read are reported as unmapped by `evalkit traces check`,
+except those matching an entry of `known_attributes` (exact keys or glob
+patterns such as `llm.tools.*`): list there the attributes you know about and
+do not need.
+
+`system_instructions` holds a system prompt kept outside the input messages,
+as a string or a list of `{"type": "text", "content"}` parts; it becomes the
+first input message, with role `system`. `tokens.cache_write` counts input
+tokens written to the prompt cache; it lands in the step's
+`metrics.extra.cache_creation_input_tokens`.
+
+## Message formats
+
+- `openinference`: flattened indexed keys under the attribute prefix `K`:
+  `K.{i}.message.role`, `K.{i}.message.content`,
+  `K.{i}.message.tool_calls.{j}.tool_call.id`,
+  `K.{i}.message.tool_calls.{j}.tool_call.function.name`,
+  `K.{i}.message.tool_calls.{j}.tool_call.function.arguments`,
+  `K.{i}.message.tool_call_id`, and content parts
+  `K.{i}.message.contents.{j}.message_content.type` / `.text`, where type
+  `text` is message text (when `message.content` is absent) and type
+  `reasoning` is the model's reasoning.
+- `gen_ai`: attribute `K` holds a JSON string: a list of messages with `role`
+  and `parts`, each part typed `text` (`content`), `reasoning` (`content`),
+  `tool_call` (`id`, `name`, `arguments`) or `tool_call_response` (`id`,
+  `response`, or `result` when `response` is absent).
+- `openai`: attribute `K` holds a JSON string: a list of OpenAI chat messages
+  with `role`, `content` (a string or a list of `{"type": "text", "text"}`
+  parts), `tool_calls` (`id`, `function.name`, `function.arguments` as a JSON
+  string) and `tool_call_id`.