@databricks/appkit 0.73.0 → 0.74.0

package/docs/api/appkit/Function.discoverEvalConfigs.md

@@ -0,0 +1,18 @@
+# Function: discoverEvalConfigs()
+
+```ts
+function discoverEvalConfigs(rootDir: string): DiscoveredEvalConfig[];
+
+```
+
+Discover the per-agent `evals.config.ts` (from [defineEvalConfig](./docs/api/appkit/Function.defineEvalConfig.md)) at `<rootDir>/server/agents/<agent>/evals/evals.config.ts`. Config is per-agent: each agent's config applies only to that agent's evals. Agents without a config file are omitted. Returns a stable, sorted list.
+
+## Parameters[​](#parameters "Direct link to Parameters")
+
+| Parameter | Type     |
+| --------- | -------- |
+| `rootDir` | `string` |
+
+## Returns[​](#returns "Direct link to Returns")
+
+[`DiscoveredEvalConfig`](./docs/api/appkit/Interface.DiscoveredEvalConfig.md)\[]

package/docs/api/appkit/Function.formatResultsJUnit.md

@@ -0,0 +1,18 @@
+# Function: formatResultsJUnit()
+
+```ts
+function formatResultsJUnit(results: EvalResult[]): string;
+
+```
+
+Render results as JUnit XML for standard CI test reporters: a single `<testsuite name="appkit-agent-evals">` with one `<testcase>` per result. Failures carry a `<failure>` (error or failing-gate summary); skips a `<skipped>`. All attribute/text values are XML-escaped.
+
+## Parameters[​](#parameters "Direct link to Parameters")
+
+| Parameter | Type                                                               |
+| --------- | ------------------------------------------------------------------ |
+| `results` | [`EvalResult`](./docs/api/appkit/Interface.EvalResult.md)\[] |
+
+## Returns[​](#returns "Direct link to Returns")
+
+`string`

package/docs/api/appkit/Function.formatResultsJson.md

@@ -0,0 +1,18 @@
+# Function: formatResultsJson()
+
+```ts
+function formatResultsJson(results: EvalResult[]): string;
+
+```
+
+Render results as a machine-readable JSON report (2-space indented): `{ summary: EvalSummary, results: EvalResult[] }`. Faithful to the types — every field present on a result round-trips.
+
+## Parameters[​](#parameters "Direct link to Parameters")
+
+| Parameter | Type                                                               |
+| --------- | ------------------------------------------------------------------ |
+| `results` | [`EvalResult`](./docs/api/appkit/Interface.EvalResult.md)\[] |
+
+## Returns[​](#returns "Direct link to Returns")
+
+`string`

package/docs/api/appkit/Function.runWithRetries.md

@@ -0,0 +1,28 @@
+# Function: runWithRetries()
+
+```ts
+function runWithRetries(
+   retries: number, 
+   attempt: (attemptNumber: number) => Promise<EvalResult>, 
+   options: {
+  baseDelayMs?: number;
+}): Promise<EvalResult>;
+
+```
+
+Run `attempt` up to `1 + retries` times, stopping as soon as it returns a result that is neither a thrown error / per-eval timeout (`error`) nor a transport/agent turn failure (`infraFailure`). Assertion failures set neither, so a failed-but-completed eval is returned on the first try and never retried. Returns the last result when every attempt failed on infra.
+
+Between attempts it waits a full-jittered exponential backoff (infra flakes are overload-correlated). `retries` is coerced to a finite non-negative integer; `baseDelayMs: 0` disables the wait (tests).
+
+## Parameters[​](#parameters "Direct link to Parameters")
+
+| Parameter              | Type                                                                                                      |
+| ---------------------- | --------------------------------------------------------------------------------------------------------- |
+| `retries`              | `number`                                                                                                  |
+| `attempt`              | (`attemptNumber`: `number`) => `Promise`<[`EvalResult`](./docs/api/appkit/Interface.EvalResult.md)> |
+| `options`              | { `baseDelayMs?`: `number`; }                                                                             |
+| `options.baseDelayMs?` | `number`                                                                                                  |
+
+## Returns[​](#returns "Direct link to Returns")
+
+`Promise`<[`EvalResult`](./docs/api/appkit/Interface.EvalResult.md)>

package/docs/api/appkit/Function.userTurns.md

@@ -0,0 +1,20 @@
+# Function: userTurns()
+
+```ts
+function userTurns(input: Record<string, unknown>): string[];
+
+```
+
+Extract every user-message content, in order, from an MLflow `{"messages":[{"role":"user","content":"..."}]}` input. A dataset row can carry a full multi-turn conversation; replaying these against one thread (one `t.send` per returned string) lets the agent see the accumulating history.
+
+Only `role === "user"` turns are returned — any interleaved `assistant`/ `system` messages in the row are ignored, since the agent generates its own responses; you never inject the dataset's assistant turns. A single-user-turn row yields a one-element array (backward compatible); a row with no `messages` yields `[]`.
+
+## Parameters[​](#parameters "Direct link to Parameters")
+
+| Parameter | Type                          |
+| --------- | ----------------------------- |
+| `input`   | `Record`<`string`, `unknown`> |
+
+## Returns[​](#returns "Direct link to Returns")
+
+`string`\[]

package/docs/api/appkit/Interface.DiscoveredEvalConfig.md

@@ -0,0 +1,25 @@
+# Interface: DiscoveredEvalConfig
+
+A per-agent `evals.config.ts` found under `server/agents/<agent>/evals/`.
+
+## Properties[​](#properties "Direct link to Properties")
+
+### agent[​](#agent "Direct link to agent")
+
+```ts
+agent: string;
+
+```
+
+The agent id whose evals this config applies to.
+
+***
+
+### file[​](#file "Direct link to file")
+
+```ts
+file: string;
+
+```
+
+Absolute path to the `evals.config.ts` file.

package/docs/api/appkit/Interface.DriveResult.md

@@ -37,6 +37,34 @@ Whether the turn completed without an agent/stream error.
 
 ***
 
+### toolCallDetails[​](#toolcalldetails "Direct link to toolCallDetails")
+
+```ts
+toolCallDetails: {
+  args: Record<string, unknown>;
+  name: string;
+}[];
+
+```
+
+Tool calls with their parsed arguments, in call order.
+
+#### args[​](#args "Direct link to args")
+
+```ts
+args: Record<string, unknown>;
+
+```
+
+#### name[​](#name "Direct link to name")
+
+```ts
+name: string;
+
+```
+
+***
+
 ### toolCalls[​](#toolcalls "Direct link to toolCalls")
 
 ```ts

package/docs/api/appkit/Interface.EvalDefinition.md

@@ -52,6 +52,28 @@ optional description: string;
 
 Short human description, shown in reports.
 
+***
+
+### tags?[​](#tags "Direct link to tags?")
+
+```ts
+optional tags: string[];
+
+```
+
+Free-form tags for filtering (see the runner's `tags` / `--tag` option).
+
+***
+
+### timeoutMs?[​](#timeoutms "Direct link to timeoutMs?")
+
+```ts
+optional timeoutMs: number;
+
+```
+
+Per-eval timeout (ms): `runEval` races the test against it and records a non-passing result instead of hanging. Overrides the runner/CLI default.
+
 ## Methods[​](#methods "Direct link to Methods")
 
 ### test()[​](#test "Direct link to test()")

package/docs/api/appkit/Interface.EvalDriver.md

@@ -22,15 +22,21 @@ Drop the current conversation so the next `send` starts a fresh thread. Optional
 ### send()[​](#send "Direct link to send()")
 
 ```ts
-send(message: string): Promise<DriveResult>;
+send(message: string, options?: {
+  signal?: AbortSignal;
+}): Promise<DriveResult>;
 
 ```
 
+Drive one turn. `options.signal`, when provided, aborts the in-flight turn: the runner passes its per-eval timeout signal so a timed-out eval cancels the request instead of leaking a live stream.
+
 #### Parameters[​](#parameters "Direct link to Parameters")
 
-| Parameter | Type     |
-| --------- | -------- |
-| `message` | `string` |
+| Parameter         | Type                          |
+| ----------------- | ----------------------------- |
+| `message`         | `string`                      |
+| `options?`        | { `signal?`: `AbortSignal`; } |
+| `options.signal?` | `AbortSignal`                 |
 
 #### Returns[​](#returns-1 "Direct link to Returns")
 

package/docs/api/appkit/Interface.EvalResult.md

@@ -42,6 +42,17 @@ id: string;
 
 ***
 
+### infraFailure?[​](#infrafailure "Direct link to infraFailure?")
+
+```ts
+optional infraFailure: boolean;
+
+```
+
+A turn failed at the transport/agent level (`succeeded: false`), not on an assertion — a retryable infra flake, distinct from `error`.
+
+***
+
 ### passed[​](#passed "Direct link to passed")
 
 ```ts

package/docs/api/appkit/Interface.EvalSummary.md

@@ -31,6 +31,17 @@ passed: number;
 
 ***
 
+### passRate[​](#passrate "Direct link to passRate")
+
+```ts
+passRate: number;
+
+```
+
+Fraction of scored (non-skipped) evals that passed, 0..1 (1 when none scored).
+
+***
+
 ### skipped[​](#skipped "Direct link to skipped")
 
 ```ts

package/docs/api/appkit/Interface.RunEvalOptions.md

@@ -43,3 +43,14 @@ optional strict: boolean;
 ```
 
 When true, soft assertion failures also fail the eval.
+
+***
+
+### timeoutMs?[​](#timeoutms "Direct link to timeoutMs?")
+
+```ts
+optional timeoutMs: number;
+
+```
+
+Runner-level default per-eval timeout (ms). `def.timeoutMs` wins over this; when both are unset the eval runs unbounded (current behavior).

package/docs/api/appkit/Interface.RunEvalsOptions.md

@@ -160,6 +160,17 @@ Progress callback, invoked as evals are discovered, started, and finished.
 
 ***
 
+### retries?[​](#retries "Direct link to retries?")
+
+```ts
+optional retries: number;
+
+```
+
+Re-run an eval up to this many extra times when it fails on infrastructure — a thrown error/timeout (`result.error`) or a transport/agent turn failure (`result.infraFailure`). Assertion failures are never retried. Defaults to `0`.
+
+***
+
 ### rootDir?[​](#rootdir "Direct link to rootDir?")
 
 ```ts
@@ -182,6 +193,17 @@ Soft assertion failures also fail the eval.
 
 ***
 
+### tags?[​](#tags "Direct link to tags?")
+
+```ts
+optional tags: string[];
+
+```
+
+Only run evals whose `tags` intersect this list. Empty/undefined runs all. Tags live on the eval def, so filtering happens after each file is loaded.
+
+***
+
 ### timeoutMs?[​](#timeoutms "Direct link to timeoutMs?")
 
 ```ts
@@ -189,7 +211,7 @@ optional timeoutMs: number;
 
 ```
 
-Per-turn wall-clock timeout (ms) before a turn is failed. Defaults to 120s.
+Default per-eval timeout (ms): `runEval` races the whole test against it and it also caps each driver turn. A per-eval `def.timeoutMs` overrides it, and it wins over an agent's `evals.config.ts` `timeoutMs`. Unbounded when unset.
 
 ***
 

package/docs/api/appkit/Interface.TestContext.md

@@ -152,6 +152,28 @@ Assert a tool was called during the run (gate by default).
 
 ***
 
+### calledToolWith()[​](#calledtoolwith "Direct link to calledToolWith()")
+
+```ts
+calledToolWith(name: string, expected: Record<string, unknown>): AssertionHandle;
+
+```
+
+Assert a tool was called with arguments that deep-contain `expected`: every key in `expected` must equal the actual argument (recursively for nested objects; arrays match element-for-element), so extra arguments are ignored. Gate by default.
+
+#### Parameters[​](#parameters-4 "Direct link to Parameters")
+
+| Parameter  | Type                          |
+| ---------- | ----------------------------- |
+| `name`     | `string`                      |
+| `expected` | `Record`<`string`, `unknown`> |
+
+#### Returns[​](#returns-4 "Direct link to Returns")
+
+[`AssertionHandle`](./docs/api/appkit/Interface.AssertionHandle.md)
+
+***
+
 ### check()[​](#check "Direct link to check()")
 
 ```ts
@@ -161,14 +183,14 @@ check(value: string, matcher: Matcher): AssertionHandle;
 
 Assert a value against a matcher, e.g. `t.check(t.reply, includes("Sunny"))`.
 
-#### Parameters[​](#parameters-4 "Direct link to Parameters")
+#### Parameters[​](#parameters-5 "Direct link to Parameters")
 
 | Parameter | Type                                                      |
 | --------- | --------------------------------------------------------- |
 | `value`   | `string`                                                  |
 | `matcher` | [`Matcher`](./docs/api/appkit/TypeAlias.Matcher.md) |
 
-#### Returns[​](#returns-4 "Direct link to Returns")
+#### Returns[​](#returns-5 "Direct link to Returns")
 
 [`AssertionHandle`](./docs/api/appkit/Interface.AssertionHandle.md)
 
@@ -183,7 +205,7 @@ reset(): void;
 
 Start a fresh conversation: the next `send` opens a new thread with no history. Use to run several independent one-shot checks in one test. Consecutive `send`s (without a `reset`) stay in one multi-turn conversation.
 
-#### Returns[​](#returns-5 "Direct link to Returns")
+#### Returns[​](#returns-6 "Direct link to Returns")
 
 `void`
 
@@ -198,13 +220,13 @@ send(message: string): Promise<void>;
 
 Send a user message to the agent and capture its response.
 
-#### Parameters[​](#parameters-5 "Direct link to Parameters")
+#### Parameters[​](#parameters-6 "Direct link to Parameters")
 
 | Parameter | Type     |
 | --------- | -------- |
 | `message` | `string` |
 
-#### Returns[​](#returns-6 "Direct link to Returns")
+#### Returns[​](#returns-7 "Direct link to Returns")
 
 `Promise`<`void`>
 
@@ -219,13 +241,13 @@ skip(reason?: string): never;
 
 Skip this eval with an optional reason.
 
-#### Parameters[​](#parameters-6 "Direct link to Parameters")
+#### Parameters[​](#parameters-7 "Direct link to Parameters")
 
 | Parameter | Type     |
 | --------- | -------- |
 | `reason?` | `string` |
 
-#### Returns[​](#returns-7 "Direct link to Returns")
+#### Returns[​](#returns-8 "Direct link to Returns")
 
 `never`
 
@@ -240,6 +262,6 @@ succeeded(): AssertionHandle;
 
 Assert the last turn completed successfully (gate by default).
 
-#### Returns[​](#returns-8 "Direct link to Returns")
+#### Returns[​](#returns-9 "Direct link to Returns")
 
 [`AssertionHandle`](./docs/api/appkit/Interface.AssertionHandle.md)

package/docs/api/appkit.md

@@ -54,6 +54,7 @@ Documentation merge entry for Typedoc — combines the stable `@databricks/appki
 | [DatabricksAuth](./docs/api/appkit/Interface.DatabricksAuth.md)                                       | Resolved Databricks host + bearer token for the eval runner's REST calls.                                                                                                                                                                                                                                                                                                                                                                                     |
 | [DatasetRow](./docs/api/appkit/Interface.DatasetRow.md)                                               | One row of a managed evaluation dataset. `inputs` are the kwargs passed to the agent for the turn; `expectations` (when present) is the row's ground truth / guidelines. Mirrors the `{inputs, expectations}` shape of `mlflow.genai` datasets and of the Unity Catalog table backing a managed eval dataset.                                                                                                                                                 |
 | [DiscoveredEval](./docs/api/appkit/Interface.DiscoveredEval.md)                                       | An eval file found under `server/agents/<agent>/evals/`.                                                                                                                                                                                                                                                                                                                                                                                                      |
+| [DiscoveredEvalConfig](./docs/api/appkit/Interface.DiscoveredEvalConfig.md)                           | A per-agent `evals.config.ts` found under `server/agents/<agent>/evals/`.                                                                                                                                                                                                                                                                                                                                                                                     |
 | [DriveResult](./docs/api/appkit/Interface.DriveResult.md)                                             | What a driver returns for a single `t.send`.                                                                                                                                                                                                                                                                                                                                                                                                                  |
 | [EndpointConfig](./docs/api/appkit/Interface.EndpointConfig.md)                                       | -                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
 | [EntityMutationHooks](./docs/api/appkit/Interface.EntityMutationHooks.md)                             | Mutation lifecycle for one entity. A before hook may return a replacement payload, which is revalidated against the trusted schema before it is persisted. Every hook, the mutation, and any write a hook issues through `ctx.app.database` share one transaction, so a rejection anywhere rolls all of them back. Throw `DatabaseValidationError` to answer a generated route with `422`; any other failure stays an opaque server error.                    |
@@ -198,9 +199,11 @@ Documentation merge entry for Typedoc — combines the stable `@databricks/appki
 | [createWorkspaceClient](./docs/api/appkit/Function.createWorkspaceClient.md)           | Construct an AppKit workspace client.                                                                                                                                                                                                                                                                                                                                                                      |
 | [database](./docs/api/appkit/Function.database.md)                                     | Create a typed database plugin registration for a finalized schema.                                                                                                                                                                                                                                                                                                                                        |
 | [defineEval](./docs/api/appkit/Function.defineEval.md)                                 | Define an agent eval. Default-export the result from a `server/agents/<id>/evals/*.eval.ts` file.                                                                                                                                                                                                                                                                                                          |
+| [defineEvalConfig](./docs/api/appkit/Function.defineEvalConfig.md)                     | Define per-directory eval config. Default-export from `evals.config.ts`.                                                                                                                                                                                                                                                                                                                                   |
 | [defineManifest](./docs/api/appkit/Function.defineManifest.md)                         | Validates a raw manifest (typically a `manifest.json` import) against the canonical Zod schema and returns it as a strict [PluginManifest](./docs/api/appkit/Interface.PluginManifest.md).                                                                                                                                                                                                           |
 | [defineSchema](./docs/api/appkit/Function.defineSchema.md)                             | Compile one declared schema. The returned type keeps the table names the builder returned, so `api.tables` and `hooks` can name only real tables.                                                                                                                                                                                                                                                          |
 | [defineTool](./docs/api/appkit/Function.defineTool.md)                                 | Defines a single tool entry for a plugin's internal registry.                                                                                                                                                                                                                                                                                                                                              |
+| [discoverEvalConfigs](./docs/api/appkit/Function.discoverEvalConfigs.md)               | Discover the per-agent `evals.config.ts` (from [defineEvalConfig](./docs/api/appkit/Function.defineEvalConfig.md)) at `<rootDir>/server/agents/<agent>/evals/evals.config.ts`. Config is per-agent: each agent's config applies only to that agent's evals. Agents without a config file are omitted. Returns a stable, sorted list.                                                                 |
 | [discoverEvalFiles](./docs/api/appkit/Function.discoverEvalFiles.md)                   | Discover evals under `<rootDir>/server/agents/<agent>/evals/` — co-located with each agent's `agent.{md,ts}` (same folder-per-agent layout the agents plugin discovers). The agent id is the folder name; the eval id is the file path relative to that evals dir with `.eval.ts` stripped. Sorted + stable.                                                                                               |
 | [enumColumn](./docs/api/appkit/Function.enumColumn.md)                                 | -                                                                                                                                                                                                                                                                                                                                                                                                          |
 | [equals](./docs/api/appkit/Function.equals.md)                                         | Passes when the value equals `expected` exactly.                                                                                                                                                                                                                                                                                                                                                           |
@@ -212,6 +215,8 @@ Documentation merge entry for Typedoc — combines the stable `@databricks/appki
 | [formatEvalDetail](./docs/api/appkit/Function.formatEvalDetail.md)                     | Indented detail lines for a failing eval (error + failing assertions).                                                                                                                                                                                                                                                                                                                                     |
 | [formatEvalHeadline](./docs/api/appkit/Function.formatEvalHeadline.md)                 | The one-line header for a single eval result (no failure detail).                                                                                                                                                                                                                                                                                                                                          |
 | [formatEvalResults](./docs/api/appkit/Function.formatEvalResults.md)                   | Render all results as a human-readable console report (non-streaming).                                                                                                                                                                                                                                                                                                                                     |
+| [formatResultsJson](./docs/api/appkit/Function.formatResultsJson.md)                   | Render results as a machine-readable JSON report (2-space indented): `{ summary: EvalSummary, results: EvalResult[] }`. Faithful to the types — every field present on a result round-trips.                                                                                                                                                                                                               |
+| [formatResultsJUnit](./docs/api/appkit/Function.formatResultsJUnit.md)                 | Render results as JUnit XML for standard CI test reporters: a single `<testsuite name="appkit-agent-evals">` with one `<testcase>` per result. Failures carry a `<failure>` (error or failing-gate summary); skips a `<skipped>`. All attribute/text values are XML-escaped.                                                                                                                               |
 | [formatSummaryLine](./docs/api/appkit/Function.formatSummaryLine.md)                   | The final PASS/FAIL summary line.                                                                                                                                                                                                                                                                                                                                                                          |
 | [fromSupervisorApi](./docs/api/appkit/Function.fromSupervisorApi.md)                   | Creates an [AgentAdapter](./docs/api/appkit/Interface.AgentAdapter.md) backed by the Databricks AI Gateway Responses API (`/ai-gateway/mlflow/v1/responses`).                                                                                                                                                                                                                                        |
 | [functionToolToDefinition](./docs/api/appkit/Function.functionToolToDefinition.md)     | -                                                                                                                                                                                                                                                                                                                                                                                                          |
@@ -247,10 +252,12 @@ Documentation merge entry for Typedoc — combines the stable `@databricks/appki
 | [runAgent](./docs/api/appkit/Function.runAgent.md)                                     | Standalone agent execution without `createApp`. Resolves the adapter, binds inline tools, and drives the adapter's `run()` loop to completion.                                                                                                                                                                                                                                                             |
 | [runEval](./docs/api/appkit/Function.runEval.md)                                       | Run a single eval against a driver. Never throws for assertion or agent failures — those become a non-passing [EvalResult](./docs/api/appkit/Interface.EvalResult.md). Only a malformed eval definition surfaces as `result.error`.                                                                                                                                                                  |
 | [runEvalsInDir](./docs/api/appkit/Function.runEvalsInDir.md)                           | Discover, load, and run every eval under each agent's `evals/` dir, driving the agents on a running app. Never throws for an individual eval — load/run failures become non-passing [EvalResult](./docs/api/appkit/Interface.EvalResult.md)s.                                                                                                                                                        |
+| [runWithRetries](./docs/api/appkit/Function.runWithRetries.md)                         | Run `attempt` up to `1 + retries` times, stopping as soon as it returns a result that is neither a thrown error / per-eval timeout (`error`) nor a transport/agent turn failure (`infraFailure`). Assertion failures set neither, so a failed-but-completed eval is returned on the first try and never retried. Returns the last result when every attempt failed on infra.                               |
 | [summarize](./docs/api/appkit/Function.summarize.md)                                   | -                                                                                                                                                                                                                                                                                                                                                                                                          |
 | [text](./docs/api/appkit/Function.text.md)                                             | -                                                                                                                                                                                                                                                                                                                                                                                                          |
 | [timestamp](./docs/api/appkit/Function.timestamp.md)                                   | -                                                                                                                                                                                                                                                                                                                                                                                                          |
 | [tool](./docs/api/appkit/Function.tool.md)                                             | Factory for defining function tools with Zod schemas.                                                                                                                                                                                                                                                                                                                                                      |
 | [toolsFromRegistry](./docs/api/appkit/Function.toolsFromRegistry.md)                   | Produces the `AgentToolDefinition[]` a ToolProvider exposes to the LLM, deriving `parameters` JSON Schema from each entry's Zod schema.                                                                                                                                                                                                                                                                    |
+| [userTurns](./docs/api/appkit/Function.userTurns.md)                                   | Extract every user-message content, in order, from an MLflow `{"messages":[{"role":"user","content":"..."}]}` input. A dataset row can carry a full multi-turn conversation; replaying these against one thread (one `t.send` per returned string) lets the agent see the accumulating history.                                                                                                            |
 | [uuid](./docs/api/appkit/Function.uuid.md)                                             | -                                                                                                                                                                                                                                                                                                                                                                                                          |
 | [varchar](./docs/api/appkit/Function.varchar.md)                                       | -                                                                                                                                                                                                                                                                                                                                                                                                          |

package/llms.txt

@@ -99,9 +99,11 @@ npx @databricks/appkit docs <query>
 - [Function: createWorkspaceClient()](./docs/api/appkit/Function.createWorkspaceClient.md): Construct an AppKit workspace client.
 - [Function: database()](./docs/api/appkit/Function.database.md): Create a typed database plugin registration for a finalized schema.
 - [Function: defineEval()](./docs/api/appkit/Function.defineEval.md): Define an agent eval. Default-export the result from a
+- [Function: defineEvalConfig()](./docs/api/appkit/Function.defineEvalConfig.md): Define per-directory eval config. Default-export from evals.config.ts.
 - [Function: defineManifest()](./docs/api/appkit/Function.defineManifest.md): Validates a raw manifest (typically a manifest.json import) against the
 - [Function: defineSchema()](./docs/api/appkit/Function.defineSchema.md): Compile one declared schema. The returned type keeps the table names the
 - [Function: defineTool()](./docs/api/appkit/Function.defineTool.md): Defines a single tool entry for a plugin's internal registry.
+- [Function: discoverEvalConfigs()](./docs/api/appkit/Function.discoverEvalConfigs.md): Discover the per-agent evals.config.ts (from defineEvalConfig) at
 - [Function: discoverEvalFiles()](./docs/api/appkit/Function.discoverEvalFiles.md): Discover evals under /server/agents//evals/ — co-located
 - [Function: enumColumn()](./docs/api/appkit/Function.enumColumn.md): Parameters
 - [Function: equals()](./docs/api/appkit/Function.equals.md): Passes when the value equals expected exactly.
@@ -113,6 +115,8 @@ npx @databricks/appkit docs <query>
 - [Function: formatEvalDetail()](./docs/api/appkit/Function.formatEvalDetail.md): Indented detail lines for a failing eval (error + failing assertions).
 - [Function: formatEvalHeadline()](./docs/api/appkit/Function.formatEvalHeadline.md): The one-line header for a single eval result (no failure detail).
 - [Function: formatEvalResults()](./docs/api/appkit/Function.formatEvalResults.md): Render all results as a human-readable console report (non-streaming).
+- [Function: formatResultsJson()](./docs/api/appkit/Function.formatResultsJson.md): Render results as a machine-readable JSON report (2-space indented):
+- [Function: formatResultsJUnit()](./docs/api/appkit/Function.formatResultsJUnit.md): Render results as JUnit XML for standard CI test reporters: a single
 - [Function: formatSummaryLine()](./docs/api/appkit/Function.formatSummaryLine.md): The final PASS/FAIL summary line.
 - [Function: fromSupervisorApi()](./docs/api/appkit/Function.fromSupervisorApi.md): Creates an AgentAdapter backed by the Databricks AI Gateway
 - [Function: functionToolToDefinition()](./docs/api/appkit/Function.functionToolToDefinition.md): Parameters
@@ -148,11 +152,13 @@ npx @databricks/appkit docs <query>
 - [Function: runAgent()](./docs/api/appkit/Function.runAgent.md): Standalone agent execution without createApp. Resolves the adapter, binds
 - [Function: runEval()](./docs/api/appkit/Function.runEval.md): Run a single eval against a driver. Never throws for assertion or agent
 - [Function: runEvalsInDir()](./docs/api/appkit/Function.runEvalsInDir.md): Discover, load, and run every eval under each agent's evals/ dir, driving
+- [Function: runWithRetries()](./docs/api/appkit/Function.runWithRetries.md): Run attempt up to 1 + retries times, stopping as soon as it returns a
 - [Function: summarize()](./docs/api/appkit/Function.summarize.md): Parameters
 - [Function: text()](./docs/api/appkit/Function.text.md): Returns
 - [Function: timestamp()](./docs/api/appkit/Function.timestamp.md): Parameters
 - [Function: tool()](./docs/api/appkit/Function.tool.md): Factory for defining function tools with Zod schemas.
 - [Function: toolsFromRegistry()](./docs/api/appkit/Function.toolsFromRegistry.md): Produces the AgentToolDefinition[] a ToolProvider exposes to the LLM,
+- [Function: userTurns()](./docs/api/appkit/Function.userTurns.md): Extract every user-message content, in order, from an MLflow
 - [Function: uuid()](./docs/api/appkit/Function.uuid.md): Returns
 - [Function: varchar()](./docs/api/appkit/Function.varchar.md): Parameters
 - [Interface: AgentAdapter](./docs/api/appkit/Interface.AgentAdapter.md): Properties
@@ -174,6 +180,7 @@ npx @databricks/appkit docs <query>
 - [Interface: DatabricksAuth](./docs/api/appkit/Interface.DatabricksAuth.md): Resolved Databricks host + bearer token for the eval runner's REST calls.
 - [Interface: DatasetRow](./docs/api/appkit/Interface.DatasetRow.md): One row of a managed evaluation dataset. inputs are the kwargs passed to the
 - [Interface: DiscoveredEval](./docs/api/appkit/Interface.DiscoveredEval.md): An eval file found under server/agents//evals/.
+- [Interface: DiscoveredEvalConfig](./docs/api/appkit/Interface.DiscoveredEvalConfig.md): A per-agent evals.config.ts found under server/agents//evals/.
 - [Interface: DriveResult](./docs/api/appkit/Interface.DriveResult.md): What a driver returns for a single t.send.
 - [Interface: EndpointConfig](./docs/api/appkit/Interface.EndpointConfig.md): Properties
 - [Interface: EntityMutationHooks<TTable>](./docs/api/appkit/Interface.EntityMutationHooks.md): Mutation lifecycle for one entity. A before hook may return a replacement

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@databricks/appkit",
   "type": "module",
-  "version": "0.73.0",
+  "version": "0.74.0",
   "main": "./dist/index.js",
   "types": "./dist/index.d.ts",
   "bin": {