@arizeai/phoenix-client 6.5.4 → 6.5.5

package/docs/annotations.mdx

@@ -0,0 +1,83 @@
+---
+title: "Annotations"
+description: "Attach structured feedback to spans, documents, and sessions with @arizeai/phoenix-client"
+---
+
+Annotations are structured feedback records — labels, scores, and explanations — attached to observability artifacts in Phoenix. They are the primary mechanism for recording quality signals, whether those signals come from end-users, LLM judges, or programmatic checks.
+
+<section className="hidden" data-agent-context="relevant-source-files" aria-label="Relevant source files">
+  <h2>Relevant Source Files</h2>
+  <ul>
+    <li><code>src/types/annotations.ts</code> for the shared <code>Annotation</code> base interface</li>
+    <li><code>src/spans/types.ts</code> for <code>SpanAnnotation</code> and <code>DocumentAnnotation</code></li>
+    <li><code>src/sessions/types.ts</code> for <code>SessionAnnotation</code></li>
+  </ul>
+</section>
+
+## Why Annotations Matter
+
+Annotations close the feedback loop on your LLM application:
+
+- **Human feedback** — Thumbs-up/down from end-users, QA reviews from teammates, or labeling tasks for dataset curation.
+- **LLM-as-judge evaluations** — Automated quality scoring using a second LLM (groundedness, helpfulness, safety).
+- **Code-based metrics** — Programmatic checks like regex validation, threshold comparisons, or retrieval precision calculations.
+
+Once attached, annotations appear in the Phoenix UI alongside traces and can be used to filter spans, build datasets, and track improvements during experimentation.
+
+## Annotation Targets
+
+Phoenix supports three annotation targets, each focused on a different level of your application:
+
+- **[Span Annotations](./span-annotations)** — Feedback on individual traced operations: an LLM call, a tool invocation, a retrieval step. The most common annotation target.
+- **[Document Annotations](./document-annotations)** — Feedback on specific retrieved documents within a retriever span, indexed by position. Essential for evaluating RAG pipeline quality.
+- **[Session Annotations](./session-annotations)** — Feedback on multi-turn conversations or threads as a whole. Use for conversation-level quality signals like resolution rate or customer satisfaction.
+
+## Annotator Kinds
+
+Every annotation records _who or what_ produced the feedback:
+
+| Kind | Default | Use case |
+|------|---------|----------|
+| `"HUMAN"` | Yes | Manual review, end-user thumbs-up/down, labeling tasks |
+| `"LLM"` | — | LLM-as-judge evaluations, automated quality scoring |
+| `"CODE"` | — | Programmatic rules, regex checks, threshold-based metrics |
+
+## Shared Annotation Shape
+
+All annotation types share this base interface:
+
+```ts
+interface Annotation {
+  name: string;                        // What is being measured (e.g. "groundedness")
+  label?: string;                      // Categorical result (e.g. "grounded")
+  score?: number;                      // Numeric result (e.g. 0.95)
+  explanation?: string;                // Free-text justification
+  identifier?: string;                 // For idempotent upserts
+  metadata?: Record<string, unknown>;  // Arbitrary context
+}
+```
+
+At least one of `label`, `score`, or `explanation` must be provided.
+
+Each target adds its own identifier field — `spanId` for spans, `spanId` + `documentPosition` for documents, and `sessionId` for sessions.
+
+## Sync vs. Async
+
+All annotation write functions accept an optional `sync` parameter:
+
+- **`sync: false`** (default) — The server acknowledges receipt and processes the annotation asynchronously. Higher throughput, but the response does not include the annotation ID.
+- **`sync: true`** — The server processes the annotation synchronously and returns its ID. Useful in tests or workflows that need to read the annotation back immediately.
+
+<section className="hidden" data-agent-context="source-map" aria-label="Source map">
+  <h2>Source Map</h2>
+  <ul>
+    <li><code>src/types/annotations.ts</code></li>
+    <li><code>src/spans/addSpanAnnotation.ts</code></li>
+    <li><code>src/spans/logSpanAnnotations.ts</code></li>
+    <li><code>src/spans/addDocumentAnnotation.ts</code></li>
+    <li><code>src/spans/logDocumentAnnotations.ts</code></li>
+    <li><code>src/spans/getSpanAnnotations.ts</code></li>
+    <li><code>src/sessions/addSessionAnnotation.ts</code></li>
+    <li><code>src/sessions/logSessionAnnotations.ts</code></li>
+  </ul>
+</section>

package/docs/datasets.mdx

@@ -0,0 +1,77 @@
+---
+title: "Datasets"
+description: "Create and inspect datasets with @arizeai/phoenix-client"
+---
+
+Datasets are the foundation for experiment runs. The dataset helpers cover creation, idempotent creation, record inspection, and example appends.
+
+<section className="hidden" data-agent-context="relevant-source-files" aria-label="Relevant source files">
+  <h2>Relevant Source Files</h2>
+  <ul>
+    <li>
+      <code>src/datasets/createOrGetDataset.ts</code> for the exact return
+      shape of the idempotent helper
+    </li>
+  </ul>
+</section>
+
+## Create A Dataset
+
+```ts
+import { createDataset } from "@arizeai/phoenix-client/datasets";
+
+const { datasetId } = await createDataset({
+  name: "support-eval",
+  description: "Support questions with expected answers",
+  examples: [
+    {
+      input: { question: "Where is my order?" },
+      output: { answer: "Use the tracking page in your account." },
+      metadata: { channel: "chat" },
+    },
+  ],
+});
+```
+
+## Reuse Or Append
+
+```ts
+import {
+  appendDatasetExamples,
+  createOrGetDataset,
+} from "@arizeai/phoenix-client/datasets";
+
+const dataset = await createOrGetDataset({
+  name: "support-eval",
+  description: "Support questions with expected answers",
+  examples: [],
+});
+
+await appendDatasetExamples({
+  dataset,
+  examples: [
+    {
+      input: { question: "How do I reset my password?" },
+      output: { answer: "Use the forgot password flow." },
+    },
+  ],
+});
+```
+
+`createOrGetDataset()` returns `{ datasetId }`, so you can pass that object directly as the dataset selector for append or experiment calls.
+
+## Read Back Dataset State
+
+Use `getDataset`, `getDatasetExamples`, and `getDatasetInfo` to inspect datasets after creation.
+
+<section className="hidden" data-agent-context="source-map" aria-label="Source map">
+  <h2>Source Map</h2>
+  <ul>
+    <li><code>src/datasets/createDataset.ts</code></li>
+    <li><code>src/datasets/createOrGetDataset.ts</code></li>
+    <li><code>src/datasets/appendDatasetExamples.ts</code></li>
+    <li><code>src/datasets/getDataset.ts</code></li>
+    <li><code>src/datasets/getDatasetExamples.ts</code></li>
+    <li><code>src/datasets/getDatasetInfo.ts</code></li>
+  </ul>
+</section>

package/docs/document-annotations.mdx

@@ -0,0 +1,208 @@
+---
+title: "Document Annotations"
+description: "Log document-level annotations for RAG evaluation with @arizeai/phoenix-client"
+---
+
+Document annotations tag individual retrieved documents as relevant or irrelevant within a retriever span. They are the building block for RAG evaluation — once you annotate documents with relevance scores, Phoenix automatically computes retrieval metrics like **nDCG**, **Precision@K**, **MRR**, and **Hit Rate** across your project.
+
+All functions are imported from `@arizeai/phoenix-client/spans`. See [Annotations](./annotations) for the shared annotation model and concepts.
+
+<section className="hidden" data-agent-context="relevant-source-files" aria-label="Relevant source files">
+  <h2>Relevant Source Files</h2>
+  <ul>
+    <li><code>src/spans/addDocumentAnnotation.ts</code> for the single-annotation API</li>
+    <li><code>src/spans/logDocumentAnnotations.ts</code> for batch logging</li>
+    <li><code>src/spans/types.ts</code> for the <code>DocumentAnnotation</code> interface</li>
+  </ul>
+</section>
+
+## Why Document Annotations
+
+When a retriever returns a ranked list of documents, you need to know:
+
+- **Were the right documents retrieved?** (relevance)
+- **Were they ranked in the right order?** (nDCG, MRR)
+- **Was at least one relevant document returned?** (hit rate)
+- **How many of the top-K were relevant?** (Precision@K)
+
+Document annotations let you label each retrieved document with a relevance score. Phoenix then aggregates those scores into standard retrieval metrics — both per-span and across your entire project.
+
+## How Document Annotations Work
+
+Each document annotation targets a specific document by its **position** in the retriever span's output. The `documentPosition` is a 0-based index: if a retriever returns 5 documents, positions `0` through `4` are valid targets.
+
+Document annotations share the same fields as span annotations (`spanId`, `name`, `annotatorKind`, `label`, `score`, `explanation`, `metadata`). The `documentPosition` tells Phoenix _which_ retrieved document the feedback applies to.
+
+### Automatic Retrieval Metrics
+
+<Note>
+  Phoenix automatically computes **nDCG**, **Precision@K**, **MRR**, and **Hit Rate** from document annotations that have `annotatorKind: "LLM"` and a numeric `score`. Annotations with `annotatorKind: "HUMAN"` or `"CODE"` are stored but do not feed into the auto-computed retrieval metrics.
+</Note>
+
+If you want Phoenix to compute retrieval metrics for you, use `annotatorKind: "LLM"` when logging relevance scores. This is the typical pattern when running an LLM-as-judge relevance evaluator over your retrieval results.
+
+## Score All Documents In A Retrieval
+
+The most common pattern: after a retriever returns N documents, score each one for relevance. Use `logDocumentAnnotations` to send them in a single batch:
+
+```ts
+import { logDocumentAnnotations } from "@arizeai/phoenix-client/spans";
+
+// retrievedDocs comes from your evaluator — each has a relevanceScore
+const annotations = retrievedDocs.map((doc, position) => ({
+  spanId: retrieverSpanId,
+  documentPosition: position,
+  name: "relevance",
+  annotatorKind: "LLM" as const,
+  score: doc.relevanceScore,
+  label: doc.relevanceScore > 0.7 ? "relevant" : "not-relevant",
+}));
+
+await logDocumentAnnotations({ documentAnnotations: annotations });
+// Phoenix now auto-computes nDCG, Precision@K, MRR, and Hit Rate
+// for this retriever span in the UI.
+```
+
+## Binary Relevance Labeling
+
+The simplest relevance scheme: each document is either relevant (1) or not (0). This is the most common input for hit rate and nDCG:
+
+```ts
+import { logDocumentAnnotations } from "@arizeai/phoenix-client/spans";
+
+const annotations = retrievedDocs.map((doc, position) => ({
+  spanId: retrieverSpanId,
+  documentPosition: position,
+  name: "relevance",
+  annotatorKind: "LLM" as const,
+  score: isRelevant(doc, userQuery) ? 1 : 0,
+  label: isRelevant(doc, userQuery) ? "relevant" : "irrelevant",
+}));
+
+await logDocumentAnnotations({ documentAnnotations: annotations });
+```
+
+With binary scores:
+- **Hit Rate** = 1 if any document has score 1, else 0
+- **Precision@K** = fraction of top-K documents with score 1
+- **MRR** = 1 / (rank of first document with score 1)
+- **nDCG** = normalized discounted cumulative gain across the ranked list
+
+## Graded Relevance
+
+For finer-grained evaluation, use continuous scores (e.g. 0–1) instead of binary. This gives nDCG more signal about _how_ relevant each document is, not just whether it's relevant at all:
+
+```ts
+import { logDocumentAnnotations } from "@arizeai/phoenix-client/spans";
+
+// LLM judge returns a 0-1 relevance score per document
+const annotations = retrievedDocs.map((doc, position) => ({
+  spanId: retrieverSpanId,
+  documentPosition: position,
+  name: "relevance",
+  annotatorKind: "LLM" as const,
+  score: doc.relevanceScore, // e.g. 0.0, 0.3, 0.7, 1.0
+  explanation: doc.relevanceReasoning,
+  metadata: { model: "gpt-4o-mini" },
+}));
+
+await logDocumentAnnotations({ documentAnnotations: annotations });
+```
+
+## Add A Single Document Annotation
+
+For one-off annotations — e.g. a human reviewer flagging a specific document:
+
+```ts
+import { addDocumentAnnotation } from "@arizeai/phoenix-client/spans";
+
+await addDocumentAnnotation({
+  documentAnnotation: {
+    spanId: "retriever-span-id",
+    documentPosition: 0,
+    name: "relevance",
+    annotatorKind: "LLM",
+    score: 0.95,
+    label: "relevant",
+    explanation: "Document directly answers the user question.",
+  },
+});
+```
+
+## Multi-Dimensional Document Scoring
+
+Score the same documents on multiple axes by using different annotation names. Each name creates a separate annotation series in the Phoenix UI:
+
+```ts
+import { logDocumentAnnotations } from "@arizeai/phoenix-client/spans";
+
+const relevanceAnnotations = docs.map((doc, position) => ({
+  spanId: retrieverSpanId,
+  documentPosition: position,
+  name: "relevance",
+  annotatorKind: "LLM" as const,
+  score: doc.relevanceScore,
+}));
+
+const recencyAnnotations = docs.map((doc, position) => ({
+  spanId: retrieverSpanId,
+  documentPosition: position,
+  name: "recency",
+  annotatorKind: "CODE" as const,
+  score: isRecent(doc.publishDate) ? 1 : 0,
+}));
+
+await logDocumentAnnotations({
+  documentAnnotations: [...relevanceAnnotations, ...recencyAnnotations],
+});
+```
+
+## Re-Ranking Evaluation
+
+Document annotations are useful for evaluating re-rankers. Annotate the same retriever span before and after re-ranking to compare the quality of the original vs. re-ranked order:
+
+```ts
+import { logDocumentAnnotations } from "@arizeai/phoenix-client/spans";
+
+// Score documents in the re-ranker's output order
+const annotations = rerankedDocs.map((doc, position) => ({
+  spanId: rerankerSpanId,
+  documentPosition: position,
+  name: "relevance",
+  annotatorKind: "LLM" as const,
+  score: doc.relevanceScore,
+}));
+
+await logDocumentAnnotations({ documentAnnotations: annotations });
+// Compare nDCG between the retriever span and re-ranker span
+// in the Phoenix UI to measure re-ranking effectiveness.
+```
+
+## Parameter Reference
+
+### `DocumentAnnotation`
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `spanId` | `string` | Yes | The retriever span's OpenTelemetry ID |
+| `documentPosition` | `number` | Yes | 0-based index of the document in retrieval results |
+| `name` | `string` | Yes | Annotation name (e.g. `"relevance"`) |
+| `annotatorKind` | `"HUMAN" \| "LLM" \| "CODE"` | No | Defaults to `"HUMAN"`. Use `"LLM"` for auto-computed retrieval metrics. |
+| `label` | `string` | No* | Categorical label (e.g. `"relevant"`, `"irrelevant"`) |
+| `score` | `number` | No* | Numeric relevance score (e.g. 0 or 1 for binary, 0–1 for graded) |
+| `explanation` | `string` | No* | Free-text explanation |
+| `metadata` | `Record<string, unknown>` | No | Arbitrary metadata |
+
+\*At least one of `label`, `score`, or `explanation` is required.
+
+Document annotations are unique by `(name, spanId, documentPosition)`. Unlike span annotations, the `identifier` field is not supported for document annotations.
+
+<section className="hidden" data-agent-context="source-map" aria-label="Source map">
+  <h2>Source Map</h2>
+  <ul>
+    <li><code>src/spans/addDocumentAnnotation.ts</code></li>
+    <li><code>src/spans/logDocumentAnnotations.ts</code></li>
+    <li><code>src/spans/types.ts</code></li>
+    <li><code>src/types/annotations.ts</code></li>
+  </ul>
+</section>

package/docs/experiments.mdx

@@ -0,0 +1,271 @@
+---
+title: "Experiments"
+description: "Run experiments with @arizeai/phoenix-client"
+---
+
+The experiments module runs tasks over dataset examples, records experiment runs in Phoenix, and can evaluate each run with either plain experiment evaluators or `@arizeai/phoenix-evals` evaluators.
+
+<section className="hidden" data-agent-context="relevant-source-files" aria-label="Relevant source files">
+  <h2>Relevant Source Files</h2>
+  <ul>
+    <li><code>src/experiments/runExperiment.ts</code> for the task execution flow and return shape</li>
+    <li><code>src/experiments/helpers/getExperimentEvaluators.ts</code> for evaluator normalization</li>
+    <li><code>src/experiments/helpers/fromPhoenixLLMEvaluator.ts</code> for the phoenix-evals bridge</li>
+    <li><code>src/experiments/getExperimentRuns.ts</code> for reading runs back after execution</li>
+  </ul>
+</section>
+
+## Two Common Patterns
+
+Use `asExperimentEvaluator()` when your evaluation logic is plain TypeScript.
+
+Use `@arizeai/phoenix-evals` evaluators directly when you want model-backed judging.
+
+## Code-Based Example
+
+If you just want to compare task output against a reference answer or apply deterministic checks, use `asExperimentEvaluator()`:
+
+```ts
+/* eslint-disable no-console */
+import { createDataset } from "@arizeai/phoenix-client/datasets";
+import {
+  asExperimentEvaluator,
+  runExperiment,
+} from "@arizeai/phoenix-client/experiments";
+
+async function main() {
+  const { datasetId } = await createDataset({
+    name: `simple-dataset-${Date.now()}`,
+    description: "a simple dataset",
+    examples: [
+      {
+        input: { name: "John" },
+        output: { text: "Hello, John!" },
+        metadata: {},
+      },
+      {
+        input: { name: "Jane" },
+        output: { text: "Hello, Jane!" },
+        metadata: {},
+      },
+      {
+        input: { name: "Bill" },
+        output: { text: "Hello, Bill!" },
+        metadata: {},
+      },
+    ],
+  });
+
+  const experiment = await runExperiment({
+    dataset: { datasetId },
+    task: async (example) => `hello ${example.input.name}`,
+    evaluators: [
+      asExperimentEvaluator({
+        name: "matches",
+        kind: "CODE",
+        evaluate: async ({ output, expected }) => {
+          const matches = output === expected?.text;
+          return {
+            label: matches ? "matches" : "does not match",
+            score: matches ? 1 : 0,
+            explanation: matches
+              ? "output matches expected"
+              : "output does not match expected",
+            metadata: {},
+          };
+        },
+      }),
+      asExperimentEvaluator({
+        name: "contains-hello",
+        kind: "CODE",
+        evaluate: async ({ output }) => {
+          const matches =
+            typeof output === "string" && output.includes("hello");
+          return {
+            label: matches ? "contains hello" : "does not contain hello",
+            score: matches ? 1 : 0,
+            explanation: matches
+              ? "output contains hello"
+              : "output does not contain hello",
+            metadata: {},
+          };
+        },
+      }),
+    ],
+  });
+
+  console.table(experiment.runs);
+  console.table(experiment.evaluationRuns);
+}
+
+main().catch(console.error);
+```
+
+This pattern is useful when:
+
+- you already know the exact correctness rule
+- you want fast, deterministic evaluation
+- you do not want to call another model during evaluation
+
+## Model-Backed Example
+
+If you want a model-backed experiment with automatic tracing and an LLM-as-a-judge evaluator, this is the core pattern:
+
+```ts
+import { openai } from "@ai-sdk/openai";
+import { createOrGetDataset } from "@arizeai/phoenix-client/datasets";
+import { runExperiment } from "@arizeai/phoenix-client/experiments";
+import type { ExperimentTask } from "@arizeai/phoenix-client/types/experiments";
+import { createClassificationEvaluator } from "@arizeai/phoenix-evals";
+import { generateText } from "ai";
+
+const model = openai("gpt-4o-mini");
+
+const main = async () => {
+  const answersQuestion = createClassificationEvaluator({
+    name: "answersQuestion",
+    model,
+    promptTemplate:
+      "Does the following answer the user's question: <question>{{input.question}}</question><answer>{{output}}</answer>",
+    choices: {
+      correct: 1,
+      incorrect: 0,
+    },
+  });
+
+  const dataset = await createOrGetDataset({
+    name: "correctness-eval",
+    description: "Evaluate the correctness of the model",
+    examples: [
+      {
+        input: {
+          question: "Is ArizeAI Phoenix Open-Source?",
+          context: "ArizeAI Phoenix is Open-Source.",
+        },
+      },
+      // ... more examples
+    ],
+  });
+
+  const task: ExperimentTask = async (example) => {
+    if (typeof example.input.question !== "string") {
+      throw new Error("Invalid input: question must be a string");
+    }
+    if (typeof example.input.context !== "string") {
+      throw new Error("Invalid input: context must be a string");
+    }
+
+    return generateText({
+      model,
+      experimental_telemetry: {
+        isEnabled: true,
+      },
+      prompt: [
+        {
+          role: "system",
+          content: `You answer questions based on this context: ${example.input.context}`,
+        },
+        {
+          role: "user",
+          content: example.input.question,
+        },
+      ],
+    }).then((response) => {
+      if (response.text) {
+        return response.text;
+      }
+      throw new Error("Invalid response: text is required");
+    });
+  };
+
+  const experiment = await runExperiment({
+    experimentName: "answers-question-eval",
+    experimentDescription:
+      "Evaluate the ability of the model to answer questions based on the context",
+    dataset,
+    task,
+    evaluators: [answersQuestion],
+    repetitions: 3,
+  });
+
+  console.log(experiment.id);
+  console.log(Object.values(experiment.runs).length);
+  console.log(experiment.evaluationRuns?.length ?? 0);
+};
+
+main().catch(console.error);
+```
+
+## What This Example Shows
+
+- `createOrGetDataset()` creates or reuses the dataset the experiment will run against
+- `task` receives the full dataset example object
+- `generateText()` emits traces that Phoenix can attach to the experiment when telemetry is enabled
+- `createClassificationEvaluator()` from `@arizeai/phoenix-evals` can be passed directly to `runExperiment()`
+- `runExperiment()` records both task runs and evaluation runs in Phoenix
+
+## Task Inputs
+
+`runExperiment()` calls your task with the full dataset example, not just `example.input`.
+
+That means your task should usually read:
+
+- `example.input` for the task inputs
+- `example.output` for any reference answer
+- `example.metadata` for additional context
+
+In the example above, the task validates `example.input.question` and `example.input.context` before generating a response.
+
+## Evaluator Inputs
+
+When an evaluator runs, it receives a normalized object with these fields:
+
+| Field | Description |
+|--------|-------------|
+| `input` | The dataset example's `input` object |
+| `output` | The task output for that run |
+| `expected` | The dataset example's `output` object |
+| `metadata` | The dataset example's `metadata` object |
+
+This is why the `createClassificationEvaluator()` prompt can reference `{{input.question}}` and `{{output}}`.
+
+For code-based evaluators created with `asExperimentEvaluator()`, those same fields are available inside `evaluate({ input, output, expected, metadata })`.
+
+## What `runExperiment()` Returns
+
+The returned object includes the experiment metadata plus the task and evaluation results from the run.
+
+- `experiment.id` is the experiment ID in Phoenix
+- `experiment.projectName` is the Phoenix project that received the task traces
+- `experiment.runs` is a map of run IDs to task run objects
+- `experiment.evaluationRuns` contains evaluator results when evaluators are provided
+
+## Follow-Up Helpers
+
+Use these exports for follow-up workflows:
+
+- `createExperiment`
+- `getExperiment`
+- `getExperimentInfo`
+- `getExperimentRuns`
+- `listExperiments`
+- `resumeExperiment`
+- `resumeEvaluation`
+- `deleteExperiment`
+
+## Tracing Behavior
+
+`runExperiment()` can register a tracer provider for the task run so that task spans and evaluator spans show up in Phoenix during the experiment. This is why tasks that call the AI SDK can still emit traces to Phoenix when global tracing is enabled.
+
+<section className="hidden" data-agent-context="source-map" aria-label="Source map">
+  <h2>Source Map</h2>
+  <ul>
+    <li><code>src/experiments/runExperiment.ts</code></li>
+    <li><code>src/experiments/createExperiment.ts</code></li>
+    <li><code>src/experiments/getExperiment.ts</code></li>
+    <li><code>src/experiments/getExperimentRuns.ts</code></li>
+    <li><code>src/experiments/helpers/getExperimentEvaluators.ts</code></li>
+    <li><code>src/experiments/helpers/fromPhoenixLLMEvaluator.ts</code></li>
+    <li><code>src/experiments/helpers/asExperimentEvaluator.ts</code></li>
+  </ul>
+</section>