@databricks/appkit-ui 0.72.0 → 0.74.0

package/docs/api/appkit/Function.discoverEvalConfigs.md

@@ -0,0 +1,18 @@
+# Function: discoverEvalConfigs()
+
+```ts
+function discoverEvalConfigs(rootDir: string): DiscoveredEvalConfig[];
+
+```
+
+Discover the per-agent `evals.config.ts` (from [defineEvalConfig](./docs/api/appkit/Function.defineEvalConfig.md)) at `<rootDir>/server/agents/<agent>/evals/evals.config.ts`. Config is per-agent: each agent's config applies only to that agent's evals. Agents without a config file are omitted. Returns a stable, sorted list.
+
+## Parameters[​](#parameters "Direct link to Parameters")
+
+| Parameter | Type     |
+| --------- | -------- |
+| `rootDir` | `string` |
+
+## Returns[​](#returns "Direct link to Returns")
+
+[`DiscoveredEvalConfig`](./docs/api/appkit/Interface.DiscoveredEvalConfig.md)\[]

package/docs/api/appkit/Function.formatResultsJUnit.md

@@ -0,0 +1,18 @@
+# Function: formatResultsJUnit()
+
+```ts
+function formatResultsJUnit(results: EvalResult[]): string;
+
+```
+
+Render results as JUnit XML for standard CI test reporters: a single `<testsuite name="appkit-agent-evals">` with one `<testcase>` per result. Failures carry a `<failure>` (error or failing-gate summary); skips a `<skipped>`. All attribute/text values are XML-escaped.
+
+## Parameters[​](#parameters "Direct link to Parameters")
+
+| Parameter | Type                                                               |
+| --------- | ------------------------------------------------------------------ |
+| `results` | [`EvalResult`](./docs/api/appkit/Interface.EvalResult.md)\[] |
+
+## Returns[​](#returns "Direct link to Returns")
+
+`string`

package/docs/api/appkit/Function.formatResultsJson.md

@@ -0,0 +1,18 @@
+# Function: formatResultsJson()
+
+```ts
+function formatResultsJson(results: EvalResult[]): string;
+
+```
+
+Render results as a machine-readable JSON report (2-space indented): `{ summary: EvalSummary, results: EvalResult[] }`. Faithful to the types — every field present on a result round-trips.
+
+## Parameters[​](#parameters "Direct link to Parameters")
+
+| Parameter | Type                                                               |
+| --------- | ------------------------------------------------------------------ |
+| `results` | [`EvalResult`](./docs/api/appkit/Interface.EvalResult.md)\[] |
+
+## Returns[​](#returns "Direct link to Returns")
+
+`string`

package/docs/api/appkit/Function.readEvalDataset.md

@@ -0,0 +1,21 @@
+# Function: readEvalDataset()
+
+```ts
+function readEvalDataset(client: WorkspaceClient, options: ReadEvalDatasetOptions): Promise<DatasetRow[]>;
+
+```
+
+Read a Databricks managed evaluation dataset (a Unity Catalog table with `inputs`/`expectations` columns) into rows, over the public SQL Statement Execution API. Reuses SQLWarehouseConnector for submit/poll/transform — its result transform already JSON-parses string columns into objects, so `inputs`/`expectations` come back as records whether the table stores them as JSON strings or structs.
+
+The Python `mlflow.genai.datasets` API needs a Spark session (no TS equivalent), so we read the backing table directly.
+
+## Parameters[​](#parameters "Direct link to Parameters")
+
+| Parameter | Type                                                                                    |
+| --------- | --------------------------------------------------------------------------------------- |
+| `client`  | [`WorkspaceClient`](./docs/api/appkit/Interface.WorkspaceClient.md)               |
+| `options` | [`ReadEvalDatasetOptions`](./docs/api/appkit/Interface.ReadEvalDatasetOptions.md) |
+
+## Returns[​](#returns "Direct link to Returns")
+
+`Promise`<[`DatasetRow`](./docs/api/appkit/Interface.DatasetRow.md)\[]>

package/docs/api/appkit/Function.resolveWorkspaceClient.md

@@ -0,0 +1,18 @@
+# Function: resolveWorkspaceClient()
+
+```ts
+function resolveWorkspaceClient(options: ResolveDatabricksAuthOptions): WorkspaceClient | undefined;
+
+```
+
+Construct a Databricks `WorkspaceClient` for the eval runner — the object the SDK-backed connectors (e.g. `SQLWarehouseConnector`) take. An explicit host+token builds a PAT client; otherwise the profile (or ambient config) is used and the SDK resolves credentials, minting OAuth as needed. Returns `undefined` if construction throws (missing/invalid config).
+
+## Parameters[​](#parameters "Direct link to Parameters")
+
+| Parameter | Type                                                                                                |
+| --------- | --------------------------------------------------------------------------------------------------- |
+| `options` | [`ResolveDatabricksAuthOptions`](./docs/api/appkit/Interface.ResolveDatabricksAuthOptions.md) |
+
+## Returns[​](#returns "Direct link to Returns")
+
+[`WorkspaceClient`](./docs/api/appkit/Interface.WorkspaceClient.md) | `undefined`

package/docs/api/appkit/Function.runWithRetries.md

@@ -0,0 +1,28 @@
+# Function: runWithRetries()
+
+```ts
+function runWithRetries(
+   retries: number, 
+   attempt: (attemptNumber: number) => Promise<EvalResult>, 
+   options: {
+  baseDelayMs?: number;
+}): Promise<EvalResult>;
+
+```
+
+Run `attempt` up to `1 + retries` times, stopping as soon as it returns a result that is neither a thrown error / per-eval timeout (`error`) nor a transport/agent turn failure (`infraFailure`). Assertion failures set neither, so a failed-but-completed eval is returned on the first try and never retried. Returns the last result when every attempt failed on infra.
+
+Between attempts it waits a full-jittered exponential backoff (infra flakes are overload-correlated). `retries` is coerced to a finite non-negative integer; `baseDelayMs: 0` disables the wait (tests).
+
+## Parameters[​](#parameters "Direct link to Parameters")
+
+| Parameter              | Type                                                                                                      |
+| ---------------------- | --------------------------------------------------------------------------------------------------------- |
+| `retries`              | `number`                                                                                                  |
+| `attempt`              | (`attemptNumber`: `number`) => `Promise`<[`EvalResult`](./docs/api/appkit/Interface.EvalResult.md)> |
+| `options`              | { `baseDelayMs?`: `number`; }                                                                             |
+| `options.baseDelayMs?` | `number`                                                                                                  |
+
+## Returns[​](#returns "Direct link to Returns")
+
+`Promise`<[`EvalResult`](./docs/api/appkit/Interface.EvalResult.md)>

package/docs/api/appkit/Function.userTurns.md

@@ -0,0 +1,20 @@
+# Function: userTurns()
+
+```ts
+function userTurns(input: Record<string, unknown>): string[];
+
+```
+
+Extract every user-message content, in order, from an MLflow `{"messages":[{"role":"user","content":"..."}]}` input. A dataset row can carry a full multi-turn conversation; replaying these against one thread (one `t.send` per returned string) lets the agent see the accumulating history.
+
+Only `role === "user"` turns are returned — any interleaved `assistant`/ `system` messages in the row are ignored, since the agent generates its own responses; you never inject the dataset's assistant turns. A single-user-turn row yields a one-element array (backward compatible); a row with no `messages` yields `[]`.
+
+## Parameters[​](#parameters "Direct link to Parameters")
+
+| Parameter | Type                          |
+| --------- | ----------------------------- |
+| `input`   | `Record`<`string`, `unknown`> |
+
+## Returns[​](#returns "Direct link to Returns")
+
+`string`\[]

package/docs/api/appkit/Interface.AssertionHandle.md

@@ -11,7 +11,7 @@ atLeast(threshold: number): AssertionHandle;
 
 ```
 
-Soft assertion that passes only when the score is at least `threshold`.
+Set the pass threshold for a scored assertion: it passes only when the score is at least `threshold`. Keeps the current severity (gate unless also chained with `.soft()`).
 
 #### Parameters[​](#parameters "Direct link to Parameters")
 

package/docs/api/appkit/Interface.DatabaseValidationIssue.md

@@ -0,0 +1,21 @@
+# Interface: DatabaseValidationIssue
+
+One rejected field; `path` names public columns, never their values.
+
+## Properties[​](#properties "Direct link to Properties")
+
+### message[​](#message "Direct link to message")
+
+```ts
+readonly message: string;
+
+```
+
+***
+
+### path[​](#path "Direct link to path")
+
+```ts
+readonly path: readonly string[];
+
+```

package/docs/api/appkit/Interface.DatasetRow.md

@@ -0,0 +1,21 @@
+# Interface: DatasetRow
+
+One row of a managed evaluation dataset. `inputs` are the kwargs passed to the agent for the turn; `expectations` (when present) is the row's ground truth / guidelines. Mirrors the `{inputs, expectations}` shape of `mlflow.genai` datasets and of the Unity Catalog table backing a managed eval dataset.
+
+## Properties[​](#properties "Direct link to Properties")
+
+### expectations?[​](#expectations "Direct link to expectations?")
+
+```ts
+optional expectations: Record<string, unknown>;
+
+```
+
+***
+
+### inputs[​](#inputs "Direct link to inputs")
+
+```ts
+inputs: Record<string, unknown>;
+
+```

package/docs/api/appkit/Interface.DiscoveredEvalConfig.md

@@ -0,0 +1,25 @@
+# Interface: DiscoveredEvalConfig
+
+A per-agent `evals.config.ts` found under `server/agents/<agent>/evals/`.
+
+## Properties[​](#properties "Direct link to Properties")
+
+### agent[​](#agent "Direct link to agent")
+
+```ts
+agent: string;
+
+```
+
+The agent id whose evals this config applies to.
+
+***
+
+### file[​](#file "Direct link to file")
+
+```ts
+file: string;
+
+```
+
+Absolute path to the `evals.config.ts` file.

package/docs/api/appkit/Interface.DriveResult.md

@@ -37,6 +37,34 @@ Whether the turn completed without an agent/stream error.
 
 ***
 
+### toolCallDetails[​](#toolcalldetails "Direct link to toolCallDetails")
+
+```ts
+toolCallDetails: {
+  args: Record<string, unknown>;
+  name: string;
+}[];
+
+```
+
+Tool calls with their parsed arguments, in call order.
+
+#### args[​](#args "Direct link to args")
+
+```ts
+args: Record<string, unknown>;
+
+```
+
+#### name[​](#name "Direct link to name")
+
+```ts
+name: string;
+
+```
+
+***
+
 ### toolCalls[​](#toolcalls "Direct link to toolCalls")
 
 ```ts

package/docs/api/appkit/Interface.EntityMutationHooks.md

@@ -0,0 +1,173 @@
+# Interface: EntityMutationHooks\<TTable>
+
+Mutation lifecycle for one entity. A before hook may return a replacement payload, which is revalidated against the trusted schema before it is persisted. Every hook, the mutation, and any write a hook issues through `ctx.app.database` share one transaction, so a rejection anywhere rolls all of them back. Throw `DatabaseValidationError` to answer a generated route with `422`; any other failure stays an opaque server error.
+
+## Type Parameters[​](#type-parameters "Direct link to Type Parameters")
+
+| Type Parameter              | Default type |
+| --------------------------- | ------------ |
+| `TTable` *extends* `string` | `string`     |
+
+## Methods[​](#methods "Direct link to Methods")
+
+### afterCreate()?[​](#aftercreate "Direct link to afterCreate()?")
+
+```ts
+optional afterCreate(row: FacetOf<TTable, "row">, context: HookContext): MaybePromise<void>;
+
+```
+
+#### Parameters[​](#parameters "Direct link to Parameters")
+
+| Parameter | Type                                                              |
+| --------- | ----------------------------------------------------------------- |
+| `row`     | `FacetOf`<`TTable`, `"row"`>                                      |
+| `context` | [`HookContext`](./docs/api/appkit/Interface.HookContext.md) |
+
+#### Returns[​](#returns "Direct link to Returns")
+
+`MaybePromise`<`void`>
+
+***
+
+### afterDelete()?[​](#afterdelete "Direct link to afterDelete()?")
+
+```ts
+optional afterDelete(id: IdValue, context: HookContext): MaybePromise<void>;
+
+```
+
+#### Parameters[​](#parameters-1 "Direct link to Parameters")
+
+| Parameter | Type                                                              |
+| --------- | ----------------------------------------------------------------- |
+| `id`      | `IdValue`                                                         |
+| `context` | [`HookContext`](./docs/api/appkit/Interface.HookContext.md) |
+
+#### Returns[​](#returns-1 "Direct link to Returns")
+
+`MaybePromise`<`void`>
+
+***
+
+### afterUpdate()?[​](#afterupdate "Direct link to afterUpdate()?")
+
+```ts
+optional afterUpdate(row: FacetOf<TTable, "row">, context: HookContext): MaybePromise<void>;
+
+```
+
+#### Parameters[​](#parameters-2 "Direct link to Parameters")
+
+| Parameter | Type                                                              |
+| --------- | ----------------------------------------------------------------- |
+| `row`     | `FacetOf`<`TTable`, `"row"`>                                      |
+| `context` | [`HookContext`](./docs/api/appkit/Interface.HookContext.md) |
+
+#### Returns[​](#returns-2 "Direct link to Returns")
+
+`MaybePromise`<`void`>
+
+***
+
+### afterUpsert()?[​](#afterupsert "Direct link to afterUpsert()?")
+
+```ts
+optional afterUpsert(row: FacetOf<TTable, "row">, context: HookContext): MaybePromise<void>;
+
+```
+
+#### Parameters[​](#parameters-3 "Direct link to Parameters")
+
+| Parameter | Type                                                              |
+| --------- | ----------------------------------------------------------------- |
+| `row`     | `FacetOf`<`TTable`, `"row"`>                                      |
+| `context` | [`HookContext`](./docs/api/appkit/Interface.HookContext.md) |
+
+#### Returns[​](#returns-3 "Direct link to Returns")
+
+`MaybePromise`<`void`>
+
+***
+
+### beforeCreate()?[​](#beforecreate "Direct link to beforeCreate()?")
+
+```ts
+optional beforeCreate(values: FacetOf<TTable, "insert">, context: HookContext): MaybePromise<void | FacetOf<TTable, "insert">>;
+
+```
+
+#### Parameters[​](#parameters-4 "Direct link to Parameters")
+
+| Parameter | Type                                                              |
+| --------- | ----------------------------------------------------------------- |
+| `values`  | `FacetOf`<`TTable`, `"insert"`>                                   |
+| `context` | [`HookContext`](./docs/api/appkit/Interface.HookContext.md) |
+
+#### Returns[​](#returns-4 "Direct link to Returns")
+
+`MaybePromise`<`void` | `FacetOf`<`TTable`, `"insert"`>>
+
+***
+
+### beforeDelete()?[​](#beforedelete "Direct link to beforeDelete()?")
+
+```ts
+optional beforeDelete(id: IdValue, context: HookContext): MaybePromise<void>;
+
+```
+
+#### Parameters[​](#parameters-5 "Direct link to Parameters")
+
+| Parameter | Type                                                              |
+| --------- | ----------------------------------------------------------------- |
+| `id`      | `IdValue`                                                         |
+| `context` | [`HookContext`](./docs/api/appkit/Interface.HookContext.md) |
+
+#### Returns[​](#returns-5 "Direct link to Returns")
+
+`MaybePromise`<`void`>
+
+***
+
+### beforeUpdate()?[​](#beforeupdate "Direct link to beforeUpdate()?")
+
+```ts
+optional beforeUpdate(
+   id: IdValue, 
+   values: FacetOf<TTable, "update">, 
+context: HookContext): MaybePromise<void | FacetOf<TTable, "update">>;
+
+```
+
+#### Parameters[​](#parameters-6 "Direct link to Parameters")
+
+| Parameter | Type                                                              |
+| --------- | ----------------------------------------------------------------- |
+| `id`      | `IdValue`                                                         |
+| `values`  | `FacetOf`<`TTable`, `"update"`>                                   |
+| `context` | [`HookContext`](./docs/api/appkit/Interface.HookContext.md) |
+
+#### Returns[​](#returns-6 "Direct link to Returns")
+
+`MaybePromise`<`void` | `FacetOf`<`TTable`, `"update"`>>
+
+***
+
+### beforeUpsert()?[​](#beforeupsert "Direct link to beforeUpsert()?")
+
+```ts
+optional beforeUpsert(values: FacetOf<TTable, "insert">, context: HookContext): MaybePromise<void | FacetOf<TTable, "insert">>;
+
+```
+
+#### Parameters[​](#parameters-7 "Direct link to Parameters")
+
+| Parameter | Type                                                              |
+| --------- | ----------------------------------------------------------------- |
+| `values`  | `FacetOf`<`TTable`, `"insert"`>                                   |
+| `context` | [`HookContext`](./docs/api/appkit/Interface.HookContext.md) |
+
+#### Returns[​](#returns-7 "Direct link to Returns")
+
+`MaybePromise`<`void` | `FacetOf`<`TTable`, `"insert"`>>

package/docs/api/appkit/Interface.EvalDefinition.md

@@ -15,6 +15,34 @@ Target agent id. Defaults to the eval's parent `server/agents/<id>` dir.
 
 ***
 
+### dataset?[​](#dataset "Direct link to dataset?")
+
+```ts
+optional dataset: {
+  limit?: number;
+  table: string;
+};
+
+```
+
+Run this eval once per row of a Databricks managed evaluation dataset (a Unity Catalog `catalog.schema.table` with `inputs`/`expectations` columns). Each row is bound to `t.input`/`t.expected`. Requires the runner to have a workspace client + warehouse (`--warehouse-id`). Omit for a single-run eval.
+
+#### limit?[​](#limit "Direct link to limit?")
+
+```ts
+optional limit: number;
+
+```
+
+#### table[​](#table "Direct link to table")
+
+```ts
+table: string;
+
+```
+
+***
+
 ### description?[​](#description "Direct link to description?")
 
 ```ts
@@ -24,6 +52,28 @@ optional description: string;
 
 Short human description, shown in reports.
 
+***
+
+### tags?[​](#tags "Direct link to tags?")
+
+```ts
+optional tags: string[];
+
+```
+
+Free-form tags for filtering (see the runner's `tags` / `--tag` option).
+
+***
+
+### timeoutMs?[​](#timeoutms "Direct link to timeoutMs?")
+
+```ts
+optional timeoutMs: number;
+
+```
+
+Per-eval timeout (ms): `runEval` races the test against it and records a non-passing result instead of hanging. Overrides the runner/CLI default.
+
 ## Methods[​](#methods "Direct link to Methods")
 
 ### test()[​](#test "Direct link to test()")

package/docs/api/appkit/Interface.EvalDriver.md

@@ -4,19 +4,40 @@ Abstraction over how the agent is driven. The HTTP driver posts to a running app
 
 ## Methods[​](#methods "Direct link to Methods")
 
+### reset()?[​](#reset "Direct link to reset()?")
+
+```ts
+optional reset(): void;
+
+```
+
+Drop the current conversation so the next `send` starts a fresh thread. Optional: drivers without a session concept omit it.
+
+#### Returns[​](#returns "Direct link to Returns")
+
+`void`
+
+***
+
 ### send()[​](#send "Direct link to send()")
 
 ```ts
-send(message: string): Promise<DriveResult>;
+send(message: string, options?: {
+  signal?: AbortSignal;
+}): Promise<DriveResult>;
 
 ```
 
+Drive one turn. `options.signal`, when provided, aborts the in-flight turn: the runner passes its per-eval timeout signal so a timed-out eval cancels the request instead of leaking a live stream.
+
 #### Parameters[​](#parameters "Direct link to Parameters")
 
-| Parameter | Type     |
-| --------- | -------- |
-| `message` | `string` |
+| Parameter         | Type                          |
+| ----------------- | ----------------------------- |
+| `message`         | `string`                      |
+| `options?`        | { `signal?`: `AbortSignal`; } |
+| `options.signal?` | `AbortSignal`                 |
 
-#### Returns[​](#returns "Direct link to Returns")
+#### Returns[​](#returns-1 "Direct link to Returns")
 
 `Promise`<[`DriveResult`](./docs/api/appkit/Interface.DriveResult.md)>

package/docs/api/appkit/Interface.EvalResult.md

@@ -42,6 +42,17 @@ id: string;
 
 ***
 
+### infraFailure?[​](#infrafailure "Direct link to infraFailure?")
+
+```ts
+optional infraFailure: boolean;
+
+```
+
+A turn failed at the transport/agent level (`succeeded: false`), not on an assertion — a retryable infra flake, distinct from `error`.
+
+***
+
 ### passed[​](#passed "Direct link to passed")
 
 ```ts

package/docs/api/appkit/Interface.EvalSummary.md

@@ -31,6 +31,17 @@ passed: number;
 
 ***
 
+### passRate[​](#passrate "Direct link to passRate")
+
+```ts
+passRate: number;
+
+```
+
+Fraction of scored (non-skipped) evals that passed, 0..1 (1 when none scored).
+
+***
+
 ### skipped[​](#skipped "Direct link to skipped")
 
 ```ts

package/docs/api/appkit/Interface.HookApp.md

@@ -0,0 +1,12 @@
+# Interface: HookApp
+
+The only capability a hook receives: entities bound to its transaction.
+
+## Properties[​](#properties "Direct link to Properties")
+
+### database[​](#database "Direct link to database")
+
+```ts
+readonly database: TransactionClient;
+
+```

package/docs/api/appkit/Interface.HookContext.md

@@ -0,0 +1,21 @@
+# Interface: HookContext
+
+Which entity is being mutated, and the surface a hook may write through.
+
+## Properties[​](#properties "Direct link to Properties")
+
+### app[​](#app "Direct link to app")
+
+```ts
+readonly app: HookApp;
+
+```
+
+***
+
+### entity[​](#entity "Direct link to entity")
+
+```ts
+readonly entity: string;
+
+```

package/docs/api/appkit/Interface.ReadEvalDatasetOptions.md

@@ -0,0 +1,34 @@
+# Interface: ReadEvalDatasetOptions
+
+## Properties[​](#properties "Direct link to Properties")
+
+### limit?[​](#limit "Direct link to limit?")
+
+```ts
+optional limit: number;
+
+```
+
+Optional row cap.
+
+***
+
+### table[​](#table "Direct link to table")
+
+```ts
+table: string;
+
+```
+
+Fully-qualified UC table: `catalog.schema.table`.
+
+***
+
+### warehouseId[​](#warehouseid "Direct link to warehouseId")
+
+```ts
+warehouseId: string;
+
+```
+
+SQL warehouse id to run the read against.

package/docs/api/appkit/Interface.ReadSerializerContext.md

@@ -0,0 +1,21 @@
+# Interface: ReadSerializerContext
+
+Which entity and generated operation produced the row being shaped.
+
+## Properties[​](#properties "Direct link to Properties")
+
+### entity[​](#entity "Direct link to entity")
+
+```ts
+readonly entity: string;
+
+```
+
+***
+
+### operation[​](#operation "Direct link to operation")
+
+```ts
+readonly operation: "detail" | "list";
+
+```