@arizeai/phoenix-client 6.10.0 → 6.11.0

package/docs/ci-evals-annotations.mdx

@@ -0,0 +1,190 @@
+---
+title: "CI Eval Test Annotations"
+description: "Record annotations and evaluator results on Phoenix experiment runs"
+---
+
+CI eval test annotations are how the `@arizeai/phoenix-client/vitest` and
+`@arizeai/phoenix-client/jest` submodules record scored, labeled, or
+explained results on an experiment run. They map directly to Phoenix's
+`experiment_evaluations` REST surface — each annotation becomes one
+`ExperimentEvaluation` with an `ExperimentEvaluationResult` body.
+
+## The `Annotation` Shape
+
+```ts
+interface Annotation {
+  /** Phoenix evaluation name. Required. */
+  name: string;
+  /** Numeric or boolean score. Booleans are stored as 0 / 1. */
+  score?: number | boolean | null;
+  /** Categorical label. */
+  label?: string | null;
+  /** Free-form explanation, shown in the Phoenix UI. */
+  explanation?: string | null;
+  /** Custom metadata. */
+  metadata?: Record<string, unknown>;
+  /** Source of the annotation. Defaults to "CODE". */
+  annotatorKind?: "LLM" | "CODE" | "HUMAN";
+}
+```
+
+The fields line up with Phoenix's
+`ExperimentEvaluationResult` plus the `name` and `annotator_kind` carried
+on the surrounding evaluation body.
+
+## `logAnnotation(annotation)`
+
+Records a single annotation against the current run. Must be called
+inside a `test()` body.
+
+```ts
+import * as px from "@arizeai/phoenix-client/vitest";
+
+px.describe("demo", () => {
+  px.test(
+    "manual annotation",
+    { input: { x: 1 }, expected: { y: 2 } },
+    async ({ input, expected }) => {
+      const result = myApp(input.x);
+      px.logOutput({ y: result });
+
+      px.logAnnotation({
+        name: "harmfulness",
+        score: 0.2,
+        explanation: "no PII detected",
+        annotatorKind: "CODE",
+      });
+    },
+  );
+});
+```
+
+## `evaluate(evaluator, params?)`
+
+Runs an evaluator object and records its result as an annotation on the
+current run. An evaluator is any object with a `name` and an `evaluate`
+function, including evaluators created with
+`@arizeai/phoenix-evals.createEvaluator()` and
+`@arizeai/phoenix-client/experiments.asExperimentEvaluator()`.
+The evaluator call is traced as an OpenInference `EVALUATOR` span, and the
+annotation is linked back to that evaluator trace.
+
+If `params` is omitted, Phoenix supplies the current test's `input`,
+recorded `output`, `expected`, `metadata`, and task `traceId`. If `params`
+is supplied, it is merged on top of those defaults.
+
+```ts
+import * as px from "@arizeai/phoenix-client/vitest";
+import { createEvaluator } from "@arizeai/phoenix-evals";
+
+const correctness = createEvaluator(
+  async ({ output, expected }: {
+    output: { sql: string };
+    expected: { sql: string };
+  }) => {
+    const grade = await llmAsJudge(output.sql, expected.sql);
+    return {
+      score: grade.score,
+      label: grade.passed ? "correct" : "incorrect",
+      explanation: grade.rationale,
+    };
+  },
+  { name: "correctness", kind: "LLM" },
+);
+
+px.describe("generate sql", () => {
+  px.test(
+    "select all",
+    {
+      input: { userQuery: "Get all users from the customers table" },
+      expected: { sql: "SELECT * FROM customers;" },
+    },
+    async ({ input, expected }) => {
+      const sql = await myApp(input.userQuery);
+      px.logOutput({ sql });
+      await px.evaluate(correctness, {
+        output: { sql },
+        expected: expected ?? { sql: "" },
+      });
+    },
+  );
+});
+```
+
+The annotation name comes from `evaluator.name`. The evaluator result can be
+a number, boolean, string label, `null`, or an object with `score`, `label`,
+`explanation`, and `metadata`.
+
+## Using `@arizeai/phoenix-evals`
+
+`createEvaluator()` gives you a reusable evaluator object. `px.evaluate()`
+runs that evaluator in the test context, traces the call, and records its
+result on the experiment run. If the evaluator itself uses OpenInference
+telemetry, those implementation spans appear under the evaluator trace:
+
+```ts
+import * as px from "@arizeai/phoenix-client/vitest";
+import { createEvaluator } from "@arizeai/phoenix-evals";
+
+async function judgeHallucination(answer: string) {
+  return llmAsJudge({ answer });
+}
+
+const hallucination = createEvaluator(
+  async ({ output }: { output: { answer: string } }) => {
+    const result = await judgeHallucination(output.answer);
+    return {
+      score: result.score,
+      label: result.label,
+      explanation: result.explanation,
+    };
+  },
+  { name: "hallucination", kind: "LLM" },
+);
+```
+
+If you prefer to hand-write a plain evaluator, use the same shape:
+
+```ts
+const exactMatch = {
+  name: "exact_match",
+  kind: "CODE" as const,
+  evaluate: ({ output, expected }: {
+    output: { sql: string };
+    expected: { sql: string };
+  }) => ({
+    score: output.sql === expected.sql,
+  }),
+};
+```
+
+Use `createEvaluator()` or OpenInference decorators from
+`@arizeai/phoenix-otel` when the evaluator implementation should emit child
+spans of its own. The older `traceEvaluator(fn)` helper remains available for
+raw function wrapping, but evaluator objects are the preferred interface.
+
+## Built-In `pass` Annotation
+
+Every test automatically records a `pass` boolean annotation based on
+whether the test body threw. You don't have to log it yourself; it's
+included in the reporter summary and on the run in Phoenix.
+
+## Aggregating Annotations In CI
+
+Suite `acceptanceCriteria` aggregate annotation scores after all cases run.
+Use them when a metric should clear a threshold across the dataset — either an
+`average` bar (e.g. mean `correctness >= 0.8`) or a `passRate` rule requiring a
+minimum fraction of runs to satisfy a per-run `passFn` predicate (e.g. 100% of
+runs must have `valid_sql === true`). See
+[CI Eval Tests: Vitest](./ci-evals-vitest#acceptance-criteria) for the full
+configuration shape.
+
+<section className="hidden" data-agent-context="source-map" aria-label="Source map">
+  <h2>Source Map</h2>
+  <ul>
+    <li><code>src/testing/helpers.ts</code></li>
+    <li><code>src/testing/acceptance.ts</code></li>
+    <li><code>src/testing/phoenix-test-tracking.ts</code></li>
+    <li><code>src/testing/types.ts</code></li>
+  </ul>
+</section>

package/docs/ci-evals-jest.mdx

@@ -0,0 +1,78 @@
+---
+title: "CI Eval Tests: Jest"
+description: "Wire @arizeai/phoenix-client/jest into a Jest project"
+---
+
+`@arizeai/phoenix-client/jest` exposes the same API as the Vitest
+entrypoint but binds to Jest's globals (`describe`, `test`, `it`,
+`beforeAll`, `afterAll`).
+
+## Setup
+
+Create a separate `phoenix.jest.config.cjs`:
+
+```js
+module.exports = {
+  testMatch: ["**/*.eval.?(c|m)[jt]s"],
+  reporters: ["default", "@arizeai/phoenix-client/jest/reporter"],
+  setupFiles: ["dotenv/config"],
+  testTimeout: 30000,
+};
+```
+
+- `testMatch` keeps eval suites separate from regular tests.
+- `reporters` keeps Jest's default reporter and adds the Phoenix
+  summary block at the end of the run.
+- `setupFiles: ["dotenv/config"]` loads `PHOENIX_HOST`, `PHOENIX_API_KEY`,
+  and other env vars from `.env`.
+- `testTimeout` is bumped because LLM calls can be slow.
+
+<Note>
+The `jsdom` test environment is not supported. Either omit
+`testEnvironment` or set it to `"node"`. For TypeScript or ESM use
+`ts-jest` / `@swc/jest` per Jest's docs.
+</Note>
+
+Add a script to `package.json`:
+
+```json
+{
+  "scripts": {
+    "eval": "jest --config phoenix.jest.config.cjs"
+  }
+}
+```
+
+## API
+
+```ts
+import * as px from "@arizeai/phoenix-client/jest";
+```
+
+The exported names match the Vitest entrypoint exactly:
+
+- `describe`, `describe.only`, `describe.skip`
+- `test`, `test.only`, `test.skip`, `test.each`
+- `it` (alias for `test`)
+- `logOutput`, `logAnnotation`, `evaluate`
+
+See [CI Eval Tests: Vitest](./ci-evals-vitest) for the full API reference — the surface is
+identical. The only difference is the import path and the underlying
+runner.
+
+## Notes
+
+- This module reads Jest globals off `globalThis` at suite-declaration
+  time, so importing it outside of a Jest test run will throw with a
+  helpful error message.
+- Jest's `--bail` flag will short-circuit the experiment; the Phoenix
+  reporter still prints a summary for the suites that ran.
+
+<section className="hidden" data-agent-context="source-map" aria-label="Source map">
+  <h2>Source Map</h2>
+  <ul>
+    <li><code>src/jest/index.ts</code></li>
+    <li><code>src/jest/reporter.ts</code></li>
+    <li><code>src/testing/runner.ts</code></li>
+  </ul>
+</section>

package/docs/ci-evals-vitest.mdx

@@ -0,0 +1,240 @@
+---
+title: "CI Eval Tests: Vitest"
+description: "Wire @arizeai/phoenix-client/vitest into a Vitest project"
+---
+
+`@arizeai/phoenix-client/vitest` ships a Vitest entrypoint plus an optional
+reporter that prints a Phoenix-flavored summary at the end of the run.
+
+## Setup
+
+Create a separate `phoenix.vitest.config.ts` so eval files don't get
+swept into your normal unit-test config:
+
+```ts
+import { defineConfig } from "vitest/config";
+
+export default defineConfig({
+  test: {
+    include: ["**/*.eval.?(c|m)[jt]s"],
+    reporters: ["default", "@arizeai/phoenix-client/vitest/reporter"],
+    setupFiles: ["dotenv/config"],
+    testTimeout: 30000,
+  },
+});
+```
+
+- `include` keeps eval suites separate from unit tests by matching the
+  `*.eval.ts` convention.
+- `reporters` keeps Vitest's default diagnostics and enables the Phoenix
+  summary block.
+- `setupFiles: ["dotenv/config"]` loads `PHOENIX_HOST`, `PHOENIX_API_KEY`,
+  and any other env vars from `.env`.
+- `testTimeout` is bumped because LLM calls can be slow.
+
+<Note>
+The `jsdom` test environment is not supported. Either omit `environment`
+or set it to `"node"`.
+</Note>
+
+Add a script to `package.json`:
+
+```json
+{
+  "scripts": {
+    "eval": "vitest run --config phoenix.vitest.config.ts"
+  }
+}
+```
+
+The script intentionally uses `vitest run` rather than watch mode —
+many evaluators include longer-running LLM calls.
+
+## API
+
+```ts
+import * as px from "@arizeai/phoenix-client/vitest";
+```
+
+### `describe(name, fn, config?)`
+
+Declares a Phoenix test suite. The suite name is the dataset and
+experiment name on the Phoenix server. `describe.only` and
+`describe.skip` work like Vitest's variants.
+
+```ts
+px.describe("my suite", () => { ... }, {
+  datasetName: "override-dataset-and-experiment-name",
+  description: "what this suite is for",
+  metadata: { model: "gpt-4o-mini" },
+  client: myCustomPhoenixClient, // overrides createClient()
+  repetitions: 3,                // run each test in this suite 3x
+  acceptanceCriteria: [
+    { annotationName: "token_f1", metric: "average", threshold: 0.8 },
+    {
+      annotationName: "token_f1",
+      metric: "passRate",
+      passFn: (a) => typeof a.score === "number" && a.score >= 0.5,
+      minPassRate: 0.9,
+    },
+  ],
+  dryRun: true,                  // run this suite locally; sync nothing
+});
+```
+
+| Field | Type | Description |
+|--------|------|-------------|
+| `datasetName` | `string` | Override the dataset / experiment name (defaults to the suite name). |
+| `description` | `string` | Description recorded on the dataset and experiment. |
+| `metadata` | `Record<string, unknown>` | Suite-level metadata applied to the experiment. |
+| `client` | `PhoenixClient` | Pre-configured `@arizeai/phoenix-client` instance. |
+| `repetitions` | `number` | Run each test in the suite this many times (default `1`; `PHOENIX_TEST_REPETITIONS` overrides the default). Per-test `repetitions` wins. |
+| `acceptanceCriteria` | `AcceptanceCriterion[]` | Aggregate annotation thresholds that fail the suite after all tests run. |
+| `dryRun` | `boolean` | Run the whole suite locally — no dataset, experiment, runs, or annotations are created in Phoenix. Same effect as `PHOENIX_TEST_TRACING=false`, scoped to this suite. |
+
+### `test(name, params, fn, timeout?)`
+
+Declares a single Phoenix test case. The `params` object carries the
+Phoenix `Example` fields. `test.only`, `test.skip`, and `test.each`
+mirror Vitest semantics. `it` is a re-export of `test`.
+
+```ts
+px.test(
+  "a case",
+  {
+    input: { userQuery: "..." },
+    expected: { sql: "..." },
+    metadata: { hard: true },
+    id: "stable-example-id",
+  },
+  async ({ input, expected, metadata }) => {
+    const sql = await myApp(input.userQuery);
+    px.logOutput({ sql });
+    expect(sql).toEqual(expected?.sql);
+  },
+);
+```
+
+| Param field | Maps to |
+|--------|---------|
+| `input` | `Example.input` |
+| `expected` | `Example.output` (reference) |
+| `metadata` | `Example.metadata` |
+| `id` | `Example.id` (stable upsert id) |
+| `repetitions` | Number of runs against this example (overrides the suite value). Reported as `"<name> [rep i/N]"`. |
+| `dryRun` | When `true`, this case runs as an ordinary local test — no dataset example, no run, nothing uploaded — even if the suite syncs. |
+
+### `test.each(table)(name, fn, timeout?)`
+
+Run the same test body across many examples.
+
+```ts
+const DATASET = [
+  { input: { userQuery: "whats up" }, expected: { sql: "n/a" } },
+  { input: { userQuery: "how are you?" }, expected: { sql: "n/a" } },
+];
+
+px.describe("offtopic inputs", () => {
+  px.test.each(DATASET)("offtopic input", async ({ input, expected }) => {
+    const sql = await myApp(input.userQuery);
+    px.logOutput({ sql });
+  });
+});
+```
+
+The name template supports `%i`, `%s`, and `%j` for parity with Vitest's
+`test.each`. Without a placeholder the row index is appended.
+
+### Logging
+
+- `px.logOutput(value)` records the actual `output` for the run.
+- `px.logAnnotation({ name, score, ... })` records an annotation.
+- `px.evaluate(evaluator, params?)` runs an evaluator object and records its
+  result as an annotation linked to the evaluator trace. Evaluators can come
+  from `@arizeai/phoenix-evals.createEvaluator()`, `asExperimentEvaluator()`,
+  or any plain `{ name, evaluate }` object.
+
+See [CI Eval Test Annotations](./ci-evals-annotations) for the full annotation shape.
+
+### Acceptance Criteria
+
+Use `acceptanceCriteria` to gate the suite on aggregate annotation scores in
+CI. Criteria run *after* the suite finishes, so all cases still execute and the
+reporter shows the full scorecard before failing. Each criterion aggregates one
+annotation (by `annotationName`) with one `metric`:
+
+- `metric: "average"` — gate on overall quality: the mean score across all runs
+  must clear `threshold` (compared in `direction`).
+- `metric: "passRate"` — gate on consistency: each run passes when its `passFn`
+  predicate returns `true`, and the suite passes when the fraction of passing
+  runs is at least `minPassRate` (e.g. `minPassRate: 0.9` ⇒ 90% must pass).
+
+`passFn` receives the run's annotation and returns a boolean, so it can express
+any pass rule — a score bar, a range, a label match, a metadata check, etc.
+
+```ts
+px.describe("text-to-sql scorecard", () => {
+  // tests log token_f1 (0–1), valid_sql (boolean), and latency_ms annotations
+}, {
+  acceptanceCriteria: [
+    // overall quality: the mean token_f1 must be >= 0.8
+    { annotationName: "token_f1", metric: "average", threshold: 0.8 },
+    // consistency: at least 90% of runs must score >= 0.7 on token_f1
+    {
+      annotationName: "token_f1",
+      metric: "passRate",
+      passFn: (a) => typeof a.score === "number" && a.score >= 0.7,
+      minPassRate: 0.9,
+    },
+    // hard floor: every run must produce valid SQL (boolean must be true)
+    {
+      annotationName: "valid_sql",
+      metric: "passRate",
+      passFn: (a) => a.score === true,
+      minPassRate: 1,
+    },
+    // budget: lower is better, so the mean latency must stay <= 800ms
+    {
+      annotationName: "latency_ms",
+      metric: "average",
+      threshold: 800,
+      direction: "minimize",
+    },
+  ],
+});
+```
+
+| Field | Description |
+|-------|-------------|
+| `annotationName` | Annotation to aggregate. If a run logs the same annotation more than once, the last one counts. |
+| `metric` | `"average"` checks the mean of all numeric/boolean scores against `threshold` (tolerates weak runs if the mean holds); `"passRate"` counts runs whose `passFn` returns `true` and requires that **fraction** to reach `minPassRate`. |
+| `threshold` | **`average` only.** The bar the mean must clear (in `direction`). |
+| `direction` | **`average` only.** `"maximize"` (default; higher mean is better, clears `threshold` when `>=`) or `"minimize"` (lower is better, clears when `<=`). Use `"minimize"` for cost, latency, or error rates. |
+| `passFn` | **`passRate` only.** `(annotation) => boolean` predicate deciding whether a single run passes, given its last annotation for `annotationName` (`score`, `label`, `explanation`, `metadata`, …). |
+| `minPassRate` | **`passRate` only.** Minimum fraction of runs (`0`–`1`) that must pass for the suite to pass (`1` = all). The suite passes when `passRate >= minPassRate`. |
+
+**Edge cases.** An `average` criterion with no numeric/boolean scores — or a
+`passRate` criterion whose annotation was never logged — fails rather than
+passing vacuously. Skipped tests are excluded from the aggregate; dry-run tests
+are included because they still execute locally.
+
+In the reporter's `Acceptance Criteria` block the reported value is the **mean**
+for `average` and the **fraction of runs that passed** for `passRate` (so a
+fully-passing `passRate` criterion reads `1.000`).
+
+## Reporter Output
+
+When `@arizeai/phoenix-client/vitest/reporter` is loaded, the runner prints
+a per-suite block at the end of the run with pass/fail counts,
+annotation aggregates, acceptance criteria, and links to the Phoenix dataset and experiment.
+The default Vitest reporter still runs alongside it.
+
+<section className="hidden" data-agent-context="source-map" aria-label="Source map">
+  <h2>Source Map</h2>
+  <ul>
+    <li><code>src/vitest/index.ts</code></li>
+    <li><code>src/vitest/reporter.ts</code></li>
+    <li><code>src/testing/runner.ts</code></li>
+    <li><code>src/testing/acceptance.ts</code></li>
+  </ul>
+</section>