@databricks/appkit 0.73.0 → 0.74.1

package/docs/api/appkit/Interface.EvalDefinition.md

@@ -52,6 +52,28 @@ optional description: string;
 
 Short human description, shown in reports.
 
+***
+
+### tags?[​](#tags "Direct link to tags?")
+
+```ts
+optional tags: string[];
+
+```
+
+Free-form tags for filtering (see the runner's `tags` / `--tag` option).
+
+***
+
+### timeoutMs?[​](#timeoutms "Direct link to timeoutMs?")
+
+```ts
+optional timeoutMs: number;
+
+```
+
+Per-eval timeout (ms): `runEval` races the test against it and records a non-passing result instead of hanging. Overrides the runner/CLI default.
+
 ## Methods[​](#methods "Direct link to Methods")
 
 ### test()[​](#test "Direct link to test()")

package/docs/api/appkit/Interface.EvalDriver.md

@@ -22,15 +22,21 @@ Drop the current conversation so the next `send` starts a fresh thread. Optional
 ### send()[​](#send "Direct link to send()")
 
 ```ts
-send(message: string): Promise<DriveResult>;
+send(message: string, options?: {
+  signal?: AbortSignal;
+}): Promise<DriveResult>;
 
 ```
 
+Drive one turn. `options.signal`, when provided, aborts the in-flight turn: the runner passes its per-eval timeout signal so a timed-out eval cancels the request instead of leaking a live stream.
+
 #### Parameters[​](#parameters "Direct link to Parameters")
 
-| Parameter | Type     |
-| --------- | -------- |
-| `message` | `string` |
+| Parameter         | Type                          |
+| ----------------- | ----------------------------- |
+| `message`         | `string`                      |
+| `options?`        | { `signal?`: `AbortSignal`; } |
+| `options.signal?` | `AbortSignal`                 |
 
 #### Returns[​](#returns-1 "Direct link to Returns")
 

package/docs/api/appkit/Interface.EvalResult.md

@@ -42,6 +42,17 @@ id: string;
 
 ***
 
+### infraFailure?[​](#infrafailure "Direct link to infraFailure?")
+
+```ts
+optional infraFailure: boolean;
+
+```
+
+A turn failed at the transport/agent level (`succeeded: false`), not on an assertion — a retryable infra flake, distinct from `error`.
+
+***
+
 ### passed[​](#passed "Direct link to passed")
 
 ```ts

package/docs/api/appkit/Interface.EvalSummary.md

@@ -31,6 +31,17 @@ passed: number;
 
 ***
 
+### passRate[​](#passrate "Direct link to passRate")
+
+```ts
+passRate: number;
+
+```
+
+Fraction of scored (non-skipped) evals that passed, 0..1 (1 when none scored).
+
+***
+
 ### skipped[​](#skipped "Direct link to skipped")
 
 ```ts

package/docs/api/appkit/Interface.RunEvalOptions.md

@@ -43,3 +43,14 @@ optional strict: boolean;
 ```
 
 When true, soft assertion failures also fail the eval.
+
+***
+
+### timeoutMs?[​](#timeoutms "Direct link to timeoutMs?")
+
+```ts
+optional timeoutMs: number;
+
+```
+
+Runner-level default per-eval timeout (ms). `def.timeoutMs` wins over this; when both are unset the eval runs unbounded (current behavior).

package/docs/api/appkit/Interface.RunEvalsOptions.md

@@ -160,6 +160,17 @@ Progress callback, invoked as evals are discovered, started, and finished.
 
 ***
 
+### retries?[​](#retries "Direct link to retries?")
+
+```ts
+optional retries: number;
+
+```
+
+Re-run an eval up to this many extra times when it fails on infrastructure — a thrown error/timeout (`result.error`) or a transport/agent turn failure (`result.infraFailure`). Assertion failures are never retried. Defaults to `0`.
+
+***
+
 ### rootDir?[​](#rootdir "Direct link to rootDir?")
 
 ```ts
@@ -182,6 +193,17 @@ Soft assertion failures also fail the eval.
 
 ***
 
+### tags?[​](#tags "Direct link to tags?")
+
+```ts
+optional tags: string[];
+
+```
+
+Only run evals whose `tags` intersect this list. Empty/undefined runs all. Tags live on the eval def, so filtering happens after each file is loaded.
+
+***
+
 ### timeoutMs?[​](#timeoutms "Direct link to timeoutMs?")
 
 ```ts
@@ -189,7 +211,7 @@ optional timeoutMs: number;
 
 ```
 
-Per-turn wall-clock timeout (ms) before a turn is failed. Defaults to 120s.
+Default per-eval timeout (ms): `runEval` races the whole test against it and it also caps each driver turn. A per-eval `def.timeoutMs` overrides it, and it wins over an agent's `evals.config.ts` `timeoutMs`. Unbounded when unset.
 
 ***
 

package/docs/api/appkit/Interface.TestContext.md

@@ -152,6 +152,28 @@ Assert a tool was called during the run (gate by default).
 
 ***
 
+### calledToolWith()[​](#calledtoolwith "Direct link to calledToolWith()")
+
+```ts
+calledToolWith(name: string, expected: Record<string, unknown>): AssertionHandle;
+
+```
+
+Assert a tool was called with arguments that deep-contain `expected`: every key in `expected` must equal the actual argument (recursively for nested objects; arrays match element-for-element), so extra arguments are ignored. Gate by default.
+
+#### Parameters[​](#parameters-4 "Direct link to Parameters")
+
+| Parameter  | Type                          |
+| ---------- | ----------------------------- |
+| `name`     | `string`                      |
+| `expected` | `Record`<`string`, `unknown`> |
+
+#### Returns[​](#returns-4 "Direct link to Returns")
+
+[`AssertionHandle`](./docs/api/appkit/Interface.AssertionHandle.md)
+
+***
+
 ### check()[​](#check "Direct link to check()")
 
 ```ts
@@ -161,14 +183,14 @@ check(value: string, matcher: Matcher): AssertionHandle;
 
 Assert a value against a matcher, e.g. `t.check(t.reply, includes("Sunny"))`.
 
-#### Parameters[​](#parameters-4 "Direct link to Parameters")
+#### Parameters[​](#parameters-5 "Direct link to Parameters")
 
 | Parameter | Type                                                      |
 | --------- | --------------------------------------------------------- |
 | `value`   | `string`                                                  |
 | `matcher` | [`Matcher`](./docs/api/appkit/TypeAlias.Matcher.md) |
 
-#### Returns[​](#returns-4 "Direct link to Returns")
+#### Returns[​](#returns-5 "Direct link to Returns")
 
 [`AssertionHandle`](./docs/api/appkit/Interface.AssertionHandle.md)
 
@@ -183,7 +205,7 @@ reset(): void;
 
 Start a fresh conversation: the next `send` opens a new thread with no history. Use to run several independent one-shot checks in one test. Consecutive `send`s (without a `reset`) stay in one multi-turn conversation.
 
-#### Returns[​](#returns-5 "Direct link to Returns")
+#### Returns[​](#returns-6 "Direct link to Returns")
 
 `void`
 
@@ -198,13 +220,13 @@ send(message: string): Promise<void>;
 
 Send a user message to the agent and capture its response.
 
-#### Parameters[​](#parameters-5 "Direct link to Parameters")
+#### Parameters[​](#parameters-6 "Direct link to Parameters")
 
 | Parameter | Type     |
 | --------- | -------- |
 | `message` | `string` |
 
-#### Returns[​](#returns-6 "Direct link to Returns")
+#### Returns[​](#returns-7 "Direct link to Returns")
 
 `Promise`<`void`>
 
@@ -219,13 +241,13 @@ skip(reason?: string): never;
 
 Skip this eval with an optional reason.
 
-#### Parameters[​](#parameters-6 "Direct link to Parameters")
+#### Parameters[​](#parameters-7 "Direct link to Parameters")
 
 | Parameter | Type     |
 | --------- | -------- |
 | `reason?` | `string` |
 
-#### Returns[​](#returns-7 "Direct link to Returns")
+#### Returns[​](#returns-8 "Direct link to Returns")
 
 `never`
 
@@ -240,6 +262,6 @@ succeeded(): AssertionHandle;
 
 Assert the last turn completed successfully (gate by default).
 
-#### Returns[​](#returns-8 "Direct link to Returns")
+#### Returns[​](#returns-9 "Direct link to Returns")
 
 [`AssertionHandle`](./docs/api/appkit/Interface.AssertionHandle.md)

package/docs/api/appkit.md

@@ -54,6 +54,7 @@ Documentation merge entry for Typedoc — combines the stable `@databricks/appki
 | [DatabricksAuth](./docs/api/appkit/Interface.DatabricksAuth.md)                                       | Resolved Databricks host + bearer token for the eval runner's REST calls.                                                                                                                                                                                                                                                                                                                                                                                     |
 | [DatasetRow](./docs/api/appkit/Interface.DatasetRow.md)                                               | One row of a managed evaluation dataset. `inputs` are the kwargs passed to the agent for the turn; `expectations` (when present) is the row's ground truth / guidelines. Mirrors the `{inputs, expectations}` shape of `mlflow.genai` datasets and of the Unity Catalog table backing a managed eval dataset.                                                                                                                                                 |
 | [DiscoveredEval](./docs/api/appkit/Interface.DiscoveredEval.md)                                       | An eval file found under `server/agents/<agent>/evals/`.                                                                                                                                                                                                                                                                                                                                                                                                      |
+| [DiscoveredEvalConfig](./docs/api/appkit/Interface.DiscoveredEvalConfig.md)                           | A per-agent `evals.config.ts` found under `server/agents/<agent>/evals/`.                                                                                                                                                                                                                                                                                                                                                                                     |
 | [DriveResult](./docs/api/appkit/Interface.DriveResult.md)                                             | What a driver returns for a single `t.send`.                                                                                                                                                                                                                                                                                                                                                                                                                  |
 | [EndpointConfig](./docs/api/appkit/Interface.EndpointConfig.md)                                       | -                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
 | [EntityMutationHooks](./docs/api/appkit/Interface.EntityMutationHooks.md)                             | Mutation lifecycle for one entity. A before hook may return a replacement payload, which is revalidated against the trusted schema before it is persisted. Every hook, the mutation, and any write a hook issues through `ctx.app.database` share one transaction, so a rejection anywhere rolls all of them back. Throw `DatabaseValidationError` to answer a generated route with `422`; any other failure stays an opaque server error.                    |
@@ -198,9 +199,11 @@ Documentation merge entry for Typedoc — combines the stable `@databricks/appki
 | [createWorkspaceClient](./docs/api/appkit/Function.createWorkspaceClient.md)           | Construct an AppKit workspace client.                                                                                                                                                                                                                                                                                                                                                                      |
 | [database](./docs/api/appkit/Function.database.md)                                     | Create a typed database plugin registration for a finalized schema.                                                                                                                                                                                                                                                                                                                                        |
 | [defineEval](./docs/api/appkit/Function.defineEval.md)                                 | Define an agent eval. Default-export the result from a `server/agents/<id>/evals/*.eval.ts` file.                                                                                                                                                                                                                                                                                                          |
+| [defineEvalConfig](./docs/api/appkit/Function.defineEvalConfig.md)                     | Define per-directory eval config. Default-export from `evals.config.ts`.                                                                                                                                                                                                                                                                                                                                   |
 | [defineManifest](./docs/api/appkit/Function.defineManifest.md)                         | Validates a raw manifest (typically a `manifest.json` import) against the canonical Zod schema and returns it as a strict [PluginManifest](./docs/api/appkit/Interface.PluginManifest.md).                                                                                                                                                                                                           |
 | [defineSchema](./docs/api/appkit/Function.defineSchema.md)                             | Compile one declared schema. The returned type keeps the table names the builder returned, so `api.tables` and `hooks` can name only real tables.                                                                                                                                                                                                                                                          |
 | [defineTool](./docs/api/appkit/Function.defineTool.md)                                 | Defines a single tool entry for a plugin's internal registry.                                                                                                                                                                                                                                                                                                                                              |
+| [discoverEvalConfigs](./docs/api/appkit/Function.discoverEvalConfigs.md)               | Discover the per-agent `evals.config.ts` (from [defineEvalConfig](./docs/api/appkit/Function.defineEvalConfig.md)) at `<rootDir>/server/agents/<agent>/evals/evals.config.ts`. Config is per-agent: each agent's config applies only to that agent's evals. Agents without a config file are omitted. Returns a stable, sorted list.                                                                 |
 | [discoverEvalFiles](./docs/api/appkit/Function.discoverEvalFiles.md)                   | Discover evals under `<rootDir>/server/agents/<agent>/evals/` — co-located with each agent's `agent.{md,ts}` (same folder-per-agent layout the agents plugin discovers). The agent id is the folder name; the eval id is the file path relative to that evals dir with `.eval.ts` stripped. Sorted + stable.                                                                                               |
 | [enumColumn](./docs/api/appkit/Function.enumColumn.md)                                 | -                                                                                                                                                                                                                                                                                                                                                                                                          |
 | [equals](./docs/api/appkit/Function.equals.md)                                         | Passes when the value equals `expected` exactly.                                                                                                                                                                                                                                                                                                                                                           |
@@ -212,6 +215,8 @@ Documentation merge entry for Typedoc — combines the stable `@databricks/appki
 | [formatEvalDetail](./docs/api/appkit/Function.formatEvalDetail.md)                     | Indented detail lines for a failing eval (error + failing assertions).                                                                                                                                                                                                                                                                                                                                     |
 | [formatEvalHeadline](./docs/api/appkit/Function.formatEvalHeadline.md)                 | The one-line header for a single eval result (no failure detail).                                                                                                                                                                                                                                                                                                                                          |
 | [formatEvalResults](./docs/api/appkit/Function.formatEvalResults.md)                   | Render all results as a human-readable console report (non-streaming).                                                                                                                                                                                                                                                                                                                                     |
+| [formatResultsJson](./docs/api/appkit/Function.formatResultsJson.md)                   | Render results as a machine-readable JSON report (2-space indented): `{ summary: EvalSummary, results: EvalResult[] }`. Faithful to the types — every field present on a result round-trips.                                                                                                                                                                                                               |
+| [formatResultsJUnit](./docs/api/appkit/Function.formatResultsJUnit.md)                 | Render results as JUnit XML for standard CI test reporters: a single `<testsuite name="appkit-agent-evals">` with one `<testcase>` per result. Failures carry a `<failure>` (error or failing-gate summary); skips a `<skipped>`. All attribute/text values are XML-escaped.                                                                                                                               |
 | [formatSummaryLine](./docs/api/appkit/Function.formatSummaryLine.md)                   | The final PASS/FAIL summary line.                                                                                                                                                                                                                                                                                                                                                                          |
 | [fromSupervisorApi](./docs/api/appkit/Function.fromSupervisorApi.md)                   | Creates an [AgentAdapter](./docs/api/appkit/Interface.AgentAdapter.md) backed by the Databricks AI Gateway Responses API (`/ai-gateway/mlflow/v1/responses`).                                                                                                                                                                                                                                        |
 | [functionToolToDefinition](./docs/api/appkit/Function.functionToolToDefinition.md)     | -                                                                                                                                                                                                                                                                                                                                                                                                          |
@@ -247,10 +252,12 @@ Documentation merge entry for Typedoc — combines the stable `@databricks/appki
 | [runAgent](./docs/api/appkit/Function.runAgent.md)                                     | Standalone agent execution without `createApp`. Resolves the adapter, binds inline tools, and drives the adapter's `run()` loop to completion.                                                                                                                                                                                                                                                             |
 | [runEval](./docs/api/appkit/Function.runEval.md)                                       | Run a single eval against a driver. Never throws for assertion or agent failures — those become a non-passing [EvalResult](./docs/api/appkit/Interface.EvalResult.md). Only a malformed eval definition surfaces as `result.error`.                                                                                                                                                                  |
 | [runEvalsInDir](./docs/api/appkit/Function.runEvalsInDir.md)                           | Discover, load, and run every eval under each agent's `evals/` dir, driving the agents on a running app. Never throws for an individual eval — load/run failures become non-passing [EvalResult](./docs/api/appkit/Interface.EvalResult.md)s.                                                                                                                                                        |
+| [runWithRetries](./docs/api/appkit/Function.runWithRetries.md)                         | Run `attempt` up to `1 + retries` times, stopping as soon as it returns a result that is neither a thrown error / per-eval timeout (`error`) nor a transport/agent turn failure (`infraFailure`). Assertion failures set neither, so a failed-but-completed eval is returned on the first try and never retried. Returns the last result when every attempt failed on infra.                               |
 | [summarize](./docs/api/appkit/Function.summarize.md)                                   | -                                                                                                                                                                                                                                                                                                                                                                                                          |
 | [text](./docs/api/appkit/Function.text.md)                                             | -                                                                                                                                                                                                                                                                                                                                                                                                          |
 | [timestamp](./docs/api/appkit/Function.timestamp.md)                                   | -                                                                                                                                                                                                                                                                                                                                                                                                          |
 | [tool](./docs/api/appkit/Function.tool.md)                                             | Factory for defining function tools with Zod schemas.                                                                                                                                                                                                                                                                                                                                                      |
 | [toolsFromRegistry](./docs/api/appkit/Function.toolsFromRegistry.md)                   | Produces the `AgentToolDefinition[]` a ToolProvider exposes to the LLM, deriving `parameters` JSON Schema from each entry's Zod schema.                                                                                                                                                                                                                                                                    |
+| [userTurns](./docs/api/appkit/Function.userTurns.md)                                   | Extract every user-message content, in order, from an MLflow `{"messages":[{"role":"user","content":"..."}]}` input. A dataset row can carry a full multi-turn conversation; replaying these against one thread (one `t.send` per returned string) lets the agent see the accumulating history.                                                                                                            |
 | [uuid](./docs/api/appkit/Function.uuid.md)                                             | -                                                                                                                                                                                                                                                                                                                                                                                                          |
 | [varchar](./docs/api/appkit/Function.varchar.md)                                       | -                                                                                                                                                                                                                                                                                                                                                                                                          |

package/docs/plugins/agents.md

@@ -73,7 +73,7 @@ Migrating from `config/agents/`
 
 Earlier versions kept markdown agents under `config/agents/<id>/agent.md`. That location is still read as a deprecated fallback (one-time warning on boot); move each folder to `server/agents/<id>/agent.md` so every agent — markdown and code — lives in one place.
 
-Requests land at `POST /invocations` (or its alias `POST /responses`) with an OpenAI Responses-compatible body. These endpoints run the agent to completion and return a single JSON response — no SSE. Streaming clients should use `POST /chat`. Every tool call runs through `asUser(req)` so SQL executes as the requesting user, file access respects Unity Catalog ACLs, and telemetry spans are created automatically.
+Requests land at `POST /invocations` (or its alias `POST /responses`) with an OpenAI Responses-compatible body. These endpoints run the agent to completion and return a single JSON response — no SSE. Streaming clients should use `POST /chat`. Every tool call is traced automatically. Plugin-toolkit tool calls (the `plugin:<name>` entries / `plugins.<name>.toolkit()`) additionally run through `asUser(req)`, so their SQL executes as the requesting user and file access respects Unity Catalog ACLs. A hand-rolled `tool({ execute })` is **not** wrapped: its `execute` receives only the validated tool arguments (no `req`), so it runs with the app's service-principal identity and cannot opt into OBO. If a tool must act as the requesting user, expose it as a plugin tool rather than a hand-rolled `execute`. See [Execution context](./docs/plugins/execution-context.md).
 
 No HITL on `/invocations` and `/responses`
 
@@ -218,7 +218,7 @@ Skills are on-demand instruction packs — the same `SKILL.md` format Claude Cod
 A skill is a directory with a `SKILL.md` plus any bundled reference files:
 
 ```text
-config/agents/
+server/agents/
   skills/                    # shared pool — any agent can opt in
     pdf-forms/
       SKILL.md
@@ -248,8 +248,8 @@ To fill a PDF form:
 
 ### Visibility[​](#visibility "Direct link to Visibility")
 
-* **Per-agent skills** (`config/agents/<id>/skills/`) are always visible to that agent.
-* **Global skills** (`config/agents/skills/`, and catalog-volume skills) are **opt-in**: list them in the agent's frontmatter, `skills: [pdf-forms]`. Set `autoInheritSkills: true` (or `{ file, code }`) on the plugin to make every global skill visible without listing — off by default so each agent's always-on catalog stays lean.
+* **Per-agent skills** (`server/agents/<id>/skills/`) are always visible to that agent.
+* **Global skills** (`server/agents/skills/`, and catalog-volume skills) are **opt-in**: list them in the agent's frontmatter, `skills: [pdf-forms]`. Set `autoInheritSkills: true` (or `{ file, code }`) on the plugin to make every global skill visible without listing — off by default so each agent's always-on catalog stays lean.
 
 ### How the agent uses a skill[​](#how-the-agent-uses-a-skill "Direct link to How the agent uses a skill")
 
@@ -687,6 +687,257 @@ appkit.agents.getThreads(userId);   // list user's threads
 
 ```
 
+## Evaluating agents[​](#evaluating-agents "Direct link to Evaluating agents")
+
+AppKit ships an eval framework for the agents you build here. You author evals in TypeScript with `defineEval`, drive the agent by sending it messages, and assert on its reply and tool usage with deterministic matchers or LLM judges. Evals run against a **running app** over HTTP (`--url`), and — with Databricks creds and an experiment — report to MLflow as native "Evaluation runs" with per-assertion and per-judge feedback attached to each turn's trace. The eval API is part of the beta surface: import it from `@databricks/appkit/beta`.
+
+Evals live beside each agent: `server/agents/<agent-id>/evals/*.eval.ts`. Each file default-exports one `defineEval({ test })`. The agent under test defaults to the parent `<agent-id>` directory; set `agent:` to target a different one.
+
+### A first eval[​](#a-first-eval "Direct link to A first eval")
+
+```ts
+// server/agents/query/evals/smoke.eval.ts
+import { defineEval } from "@databricks/appkit/beta";
+
+export default defineEval({
+  description: "Query agent responds to a greeting",
+  async test(t) {
+    await t.send("Hi there!");
+    t.succeeded(); // gate: the turn completed without an agent/stream error
+  },
+});
+
+```
+
+Start the app, then run the evals against it:
+
+```bash
+# the app must be running and reachable at --url
+appkit agent eval --url http://localhost:3000
+
+# scope to one agent/eval by substring, and point at a project root
+appkit agent eval query --root apps/dev-playground --url http://localhost:3000
+
+```
+
+The positional `[filter]` matches evals whose `<agent>/<id>` contains the substring (or an exact agent id). The command discovers every `*.eval.ts` under `server/agents/*/evals/`, drives each against the running app, and exits non-zero if any gate fails.
+
+### Assertions[​](#assertions "Direct link to Assertions")
+
+Every assertion returns a chainable handle. Assertions are **gates by default** — a failure fails the eval (non-zero exit). Chain `.soft()` to demote to a tracked-only metric, `.gate()` to promote a soft assertion back, or `.atLeast(n)` to set the pass threshold on a scored assertion.
+
+| Assertion                          | Passes when                                                                                                                          |
+| ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
+| `t.succeeded()`                    | The last turn completed without an agent/stream error.                                                                               |
+| `t.calledTool(name)`               | The agent called `name` during the run.                                                                                              |
+| `t.calledToolWith(name, expected)` | `name` was called with arguments that deep-contain `expected` (every key in `expected` matches recursively; extra args are ignored). |
+| `t.check(value, matcher)`          | `value` satisfies the matcher — `includes(substring)`, `equals(expected)`, or `matches(pattern)`.                                    |
+
+```ts
+import { defineEval, includes } from "@databricks/appkit/beta";
+
+export default defineEval({
+  description: "Helper agent answers a math question",
+  agent: "helper",
+  async test(t) {
+    await t.send("What is 2 + 2?");
+    t.succeeded();                          // gate
+    t.check(t.reply, includes("4")).soft(); // tracked metric, won't fail the gate
+  },
+});
+
+```
+
+```ts
+// deep-partial tool-arg check
+await t.send("What's the weather in Brooklyn?");
+t.calledTool("get_weather");
+t.calledToolWith("get_weather", { city: "Brooklyn" });
+
+```
+
+Call `t.skip("reason")` to skip an eval, and read `t.reply`, `t.toolCalls`, and `t.sessionId` to inspect the last turn.
+
+### LLM-as-judge[​](#llm-as-judge "Direct link to LLM-as-judge")
+
+`t.judge.*` scores the last reply with an LLM judge (via `autoevals` pointed at a Databricks serving endpoint). Each judge returns a scored assertion (0..1) that **gates by default** — a miss fails the eval. Chain `.atLeast(n)` to set the pass threshold, or `.soft()` to track it only. Judges require a judge model: pass `--judge-model <endpoint>` (or set `APPKIT_JUDGE_MODEL`) plus Databricks auth; without one, `t.judge.*` throws with a clear message.
+
+```ts
+async test(t) {
+  await t.send("What's the weather in Brooklyn?");
+  t.succeeded();
+
+  // closedQA needs no ground truth — it judges the reply against a question.
+  (await t.judge.closedQA(
+    "Does the response describe weather conditions for Brooklyn?",
+  )).atLeast(0.5);
+}
+
+```
+
+* `t.judge.factuality(expected)` — score the reply against an expected reference answer.
+* `t.judge.closedQA(criteria)` — score whether the reply answers the question, per `criteria`.
+* `t.judge.custom(spec)` — a prompt-template judge (`{ name, promptTemplate, choiceScores }`), the TS analog of MLflow's `@scorer`.
+
+Guard judge calls with `isJudgeConfigured()` when an eval should still exercise the drive path without a judge model configured:
+
+```ts
+import { defineEval, isJudgeConfigured } from "@databricks/appkit/beta";
+// ...
+if (isJudgeConfigured()) {
+  (await t.judge.closedQA(guideline)).atLeast(0.5);
+}
+
+```
+
+### Conversations[​](#conversations "Direct link to Conversations")
+
+Each `t.send` is one user turn. How you sequence them controls the thread:
+
+```ts
+// One-shot: a single turn.
+await t.send("Summarize Q3 revenue.");
+t.succeeded();
+
+// Multi-turn: consecutive sends share one thread, so the agent sees history.
+await t.send("Show me the orders table.");
+await t.send("Now filter it to last week.");
+t.succeeded();
+
+// t.reset() drops the conversation: the next send opens a fresh thread with
+// no history. Use it to run several independent one-shot checks in one test.
+await t.send("What's 2 + 2?");
+t.check(t.reply, includes("4"));
+t.reset();
+await t.send("What's the capital of France?");
+t.check(t.reply, includes("Paris"));
+
+```
+
+### Datasets[​](#datasets "Direct link to Datasets")
+
+Add `dataset: { table }` to sweep a Databricks **managed evaluation dataset** — a Unity Catalog `catalog.schema.table` with `inputs`/`expectations` columns. The eval runs once per row; the runner binds each row's `inputs` to `t.input` and `expectations` to `t.expected`. Reading the dataset requires a workspace client and warehouse (`--warehouse-id` + auth).
+
+```ts
+import { defineEval, isJudgeConfigured, userTurns } from "@databricks/appkit/beta";
+
+export default defineEval({
+  description: "Query agent satisfies each dataset row's guidelines",
+  dataset: { table: "main.mario.appkit_eval_dataset" }, // optional `limit?`
+  async test(t) {
+    // Replay every user turn in the row against one thread, so the agent sees
+    // the accumulating conversation. A single-user-turn row sends once.
+    for (const turn of userTurns(t.input)) {
+      await t.send(turn);
+    }
+    t.succeeded();
+
+    if (isJudgeConfigured()) {
+      for (const guideline of guidelines(t.expected)) {
+        (await t.judge.closedQA(guideline)).atLeast(0.5);
+      }
+    }
+  },
+});
+
+```
+
+The row shapes match the MLflow managed-dataset UI:
+
+```text
+inputs        {"messages":[{"role":"user","content":"..."}]}
+expectations  {"guidelines":{"value":["...","..."]}}   (optional)
+
+```
+
+`userTurns(t.input)` extracts every `role: "user"` message content in order from the `{messages:[...]}` input — a row can carry a full multi-turn conversation, and replaying each user turn against one thread lets the agent build up history (interleaved assistant/system turns are ignored; the agent generates its own). Read `expectations.guidelines.value` yourself; the UI wraps the array as `{value: [...]}`:
+
+```ts
+function guidelines(expected: Record<string, unknown> | undefined): string[] {
+  const g = (expected?.guidelines as { value?: unknown } | undefined)?.value;
+  return Array.isArray(g) ? g.map(String) : [];
+}
+
+```
+
+Run a dataset eval:
+
+```bash
+appkit agent eval dataset --root apps/dev-playground --url http://localhost:3000 \
+  --profile <profile> --warehouse-id <warehouse-id> --judge-model <endpoint>
+
+```
+
+### Running evals & CI[​](#running-evals--ci "Direct link to Running evals & CI")
+
+`appkit agent eval [filter]` — run agent evals (`server/agents/<id>/evals/*.eval.ts`) against a running app.
+
+| Flag                         | Description                                                                                                     |
+| ---------------------------- | --------------------------------------------------------------------------------------------------------------- |
+| `[filter]`                   | Only run evals whose `<agent>/<id>` contains this substring (or an exact agent id)                              |
+| `--url <url>`                | Base URL of the running app (default `http://localhost:3000`)                                                   |
+| `--strict`                   | Fail on soft-assertion misses too                                                                               |
+| `--root <dir>`               | Project root containing `server/agents/` (default: cwd)                                                         |
+| `--header <header...>`       | Extra request header as `'Key: value'` (repeatable)                                                             |
+| `--tag <tag...>`             | Only run evals tagged with one of these tags (repeatable)                                                       |
+| `--profile <name>`           | Databricks CLI profile to authenticate with via OAuth (default: `DATABRICKS_CONFIG_PROFILE`)                    |
+| `--databricks-host <host>`   | Databricks host for writing MLflow assessments (default: `DATABRICKS_HOST`)                                     |
+| `--databricks-token <token>` | Databricks token for writing MLflow assessments (default: `DATABRICKS_TOKEN`)                                   |
+| `--experiment <id>`          | MLflow experiment id for the evaluation run (default: `MLFLOW_EXPERIMENT_ID`)                                   |
+| `--warehouse-id <id>`        | SQL warehouse id for reading managed evaluation datasets (default: `DATABRICKS_WAREHOUSE_ID`)                   |
+| `--judge-model <endpoint>`   | Databricks serving endpoint to use as the LLM judge for `t.judge.*` (default: `APPKIT_JUDGE_MODEL`)             |
+| `--concurrency <n>`          | Max evals/dataset rows to drive concurrently (default: 4)                                                       |
+| `--timeout <ms>`             | Default per-eval timeout in ms (a per-eval `timeoutMs` overrides it)                                            |
+| `--retries <n>`              | Re-run an eval up to N times when it fails on an infra error (turn/timeout); assertion failures are not retried |
+| `--min-pass-rate <rate>`     | Gate on aggregate pass rate (0..1) instead of requiring every eval to pass; exit 1 when below                   |
+| `--reporter <format>`        | Report format: `text` (live console), `json` (dashboards), or `junit` (CI test reporters)                       |
+| `--output <file>`            | Write the json/junit report to this file instead of stdout (ignored for text)                                   |
+
+Notes:
+
+* **Auth is OAuth-first.** `--profile <name>` mints an OAuth token from your Databricks CLI profile — no PAT needed. An explicit `--databricks-host`/`--databricks-token` (or the `DATABRICKS_*` env vars) wins over the profile.
+* **`--retries` only absorbs infra flakiness.** A retry fires only when an eval throws or times out (`result.error` set); a wrong reply is real signal and is never retried. Each attempt gets a fresh driver.
+* **Gating.** By default the run exits non-zero if any eval fails. `--min-pass-rate 0.9` switches to threshold mode: it exits non-zero only when the aggregate pass rate drops below the threshold. `--strict` additionally treats soft-assertion misses as failures.
+* **CI reports.** `--reporter junit --output results.xml` writes a JUnit file for CI test reporters; `--reporter json` emits machine-readable results. In machine reporters, human-facing lines go to stderr so stdout stays clean for the report.
+
+```bash
+# CI: gate on 90% pass rate, emit JUnit, authenticate + report to MLflow
+appkit agent eval --url "$APP_URL" \
+  --profile ci \
+  --experiment "$MLFLOW_EXPERIMENT_ID" \
+  --concurrency 4 --retries 1 \
+  --min-pass-rate 0.9 \
+  --reporter junit --output eval-results.xml
+
+```
+
+### Per-directory config[​](#per-directory-config "Direct link to Per-directory config")
+
+Drop an `evals.config.ts` beside an agent's evals to set defaults for that agent's runs:
+
+```ts
+// server/agents/query/evals/evals.config.ts
+import { defineEvalConfig } from "@databricks/appkit/beta";
+
+export default defineEvalConfig({
+  maxConcurrency: 4,   // run up to 4 evals/rows concurrently
+  timeoutMs: 30_000,   // default per-eval timeout
+});
+
+```
+
+Precedence: a **CLI flag** wins over the **`evals.config.ts`** value, which wins over the **built-in default** (concurrency `4`, no timeout). A per-eval `def.timeoutMs` overrides both for that eval.
+
+### MLflow reporting[​](#mlflow-reporting "Direct link to MLflow reporting")
+
+When `--experiment <id>` (or `MLFLOW_EXPERIMENT_ID`) is set together with Databricks auth, the runner creates a native MLflow **Evaluation run** up front. As each eval runs against the app, its turn trace is linked to the run, and every assertion and judge score is written back as feedback on that trace:
+
+* Each deterministic assertion becomes a `CODE`-sourced Feedback with a boolean value.
+* Each judge assertion becomes an `LLM_JUDGE`-sourced Feedback with its numeric 0..1 score and rationale.
+* An overall `appkit_eval` pass/fail Feedback is attached per eval, and aggregate metrics are logged when the run finishes.
+
+Skip `--experiment` (and `MLFLOW_EXPERIMENT_ID`) to run evals purely locally with no MLflow side effects — the CLI prints a reminder that the evaluation run was skipped.
+
 ## Frontmatter schema[​](#frontmatter-schema "Direct link to Frontmatter schema")
 
 | Key                | Type            | Notes                                                                                                                                                                                                                                                                                           |