@databricks/appkit 0.72.0 → 0.74.0

package/docs/api/appkit/Interface.EntityMutationHooks.md

@@ -0,0 +1,173 @@
+# Interface: EntityMutationHooks\<TTable>
+
+Mutation lifecycle for one entity. A before hook may return a replacement payload, which is revalidated against the trusted schema before it is persisted. Every hook, the mutation, and any write a hook issues through `ctx.app.database` share one transaction, so a rejection anywhere rolls all of them back. Throw `DatabaseValidationError` to answer a generated route with `422`; any other failure stays an opaque server error.
+
+## Type Parameters[​](#type-parameters "Direct link to Type Parameters")
+
+| Type Parameter              | Default type |
+| --------------------------- | ------------ |
+| `TTable` *extends* `string` | `string`     |
+
+## Methods[​](#methods "Direct link to Methods")
+
+### afterCreate()?[​](#aftercreate "Direct link to afterCreate()?")
+
+```ts
+optional afterCreate(row: FacetOf<TTable, "row">, context: HookContext): MaybePromise<void>;
+
+```
+
+#### Parameters[​](#parameters "Direct link to Parameters")
+
+| Parameter | Type                                                              |
+| --------- | ----------------------------------------------------------------- |
+| `row`     | `FacetOf`<`TTable`, `"row"`>                                      |
+| `context` | [`HookContext`](./docs/api/appkit/Interface.HookContext.md) |
+
+#### Returns[​](#returns "Direct link to Returns")
+
+`MaybePromise`<`void`>
+
+***
+
+### afterDelete()?[​](#afterdelete "Direct link to afterDelete()?")
+
+```ts
+optional afterDelete(id: IdValue, context: HookContext): MaybePromise<void>;
+
+```
+
+#### Parameters[​](#parameters-1 "Direct link to Parameters")
+
+| Parameter | Type                                                              |
+| --------- | ----------------------------------------------------------------- |
+| `id`      | `IdValue`                                                         |
+| `context` | [`HookContext`](./docs/api/appkit/Interface.HookContext.md) |
+
+#### Returns[​](#returns-1 "Direct link to Returns")
+
+`MaybePromise`<`void`>
+
+***
+
+### afterUpdate()?[​](#afterupdate "Direct link to afterUpdate()?")
+
+```ts
+optional afterUpdate(row: FacetOf<TTable, "row">, context: HookContext): MaybePromise<void>;
+
+```
+
+#### Parameters[​](#parameters-2 "Direct link to Parameters")
+
+| Parameter | Type                                                              |
+| --------- | ----------------------------------------------------------------- |
+| `row`     | `FacetOf`<`TTable`, `"row"`>                                      |
+| `context` | [`HookContext`](./docs/api/appkit/Interface.HookContext.md) |
+
+#### Returns[​](#returns-2 "Direct link to Returns")
+
+`MaybePromise`<`void`>
+
+***
+
+### afterUpsert()?[​](#afterupsert "Direct link to afterUpsert()?")
+
+```ts
+optional afterUpsert(row: FacetOf<TTable, "row">, context: HookContext): MaybePromise<void>;
+
+```
+
+#### Parameters[​](#parameters-3 "Direct link to Parameters")
+
+| Parameter | Type                                                              |
+| --------- | ----------------------------------------------------------------- |
+| `row`     | `FacetOf`<`TTable`, `"row"`>                                      |
+| `context` | [`HookContext`](./docs/api/appkit/Interface.HookContext.md) |
+
+#### Returns[​](#returns-3 "Direct link to Returns")
+
+`MaybePromise`<`void`>
+
+***
+
+### beforeCreate()?[​](#beforecreate "Direct link to beforeCreate()?")
+
+```ts
+optional beforeCreate(values: FacetOf<TTable, "insert">, context: HookContext): MaybePromise<void | FacetOf<TTable, "insert">>;
+
+```
+
+#### Parameters[​](#parameters-4 "Direct link to Parameters")
+
+| Parameter | Type                                                              |
+| --------- | ----------------------------------------------------------------- |
+| `values`  | `FacetOf`<`TTable`, `"insert"`>                                   |
+| `context` | [`HookContext`](./docs/api/appkit/Interface.HookContext.md) |
+
+#### Returns[​](#returns-4 "Direct link to Returns")
+
+`MaybePromise`<`void` | `FacetOf`<`TTable`, `"insert"`>>
+
+***
+
+### beforeDelete()?[​](#beforedelete "Direct link to beforeDelete()?")
+
+```ts
+optional beforeDelete(id: IdValue, context: HookContext): MaybePromise<void>;
+
+```
+
+#### Parameters[​](#parameters-5 "Direct link to Parameters")
+
+| Parameter | Type                                                              |
+| --------- | ----------------------------------------------------------------- |
+| `id`      | `IdValue`                                                         |
+| `context` | [`HookContext`](./docs/api/appkit/Interface.HookContext.md) |
+
+#### Returns[​](#returns-5 "Direct link to Returns")
+
+`MaybePromise`<`void`>
+
+***
+
+### beforeUpdate()?[​](#beforeupdate "Direct link to beforeUpdate()?")
+
+```ts
+optional beforeUpdate(
+   id: IdValue, 
+   values: FacetOf<TTable, "update">, 
+context: HookContext): MaybePromise<void | FacetOf<TTable, "update">>;
+
+```
+
+#### Parameters[​](#parameters-6 "Direct link to Parameters")
+
+| Parameter | Type                                                              |
+| --------- | ----------------------------------------------------------------- |
+| `id`      | `IdValue`                                                         |
+| `values`  | `FacetOf`<`TTable`, `"update"`>                                   |
+| `context` | [`HookContext`](./docs/api/appkit/Interface.HookContext.md) |
+
+#### Returns[​](#returns-6 "Direct link to Returns")
+
+`MaybePromise`<`void` | `FacetOf`<`TTable`, `"update"`>>
+
+***
+
+### beforeUpsert()?[​](#beforeupsert "Direct link to beforeUpsert()?")
+
+```ts
+optional beforeUpsert(values: FacetOf<TTable, "insert">, context: HookContext): MaybePromise<void | FacetOf<TTable, "insert">>;
+
+```
+
+#### Parameters[​](#parameters-7 "Direct link to Parameters")
+
+| Parameter | Type                                                              |
+| --------- | ----------------------------------------------------------------- |
+| `values`  | `FacetOf`<`TTable`, `"insert"`>                                   |
+| `context` | [`HookContext`](./docs/api/appkit/Interface.HookContext.md) |
+
+#### Returns[​](#returns-7 "Direct link to Returns")
+
+`MaybePromise`<`void` | `FacetOf`<`TTable`, `"insert"`>>

package/docs/api/appkit/Interface.EvalDefinition.md

@@ -15,6 +15,34 @@ Target agent id. Defaults to the eval's parent `server/agents/<id>` dir.
 
 ***
 
+### dataset?[​](#dataset "Direct link to dataset?")
+
+```ts
+optional dataset: {
+  limit?: number;
+  table: string;
+};
+
+```
+
+Run this eval once per row of a Databricks managed evaluation dataset (a Unity Catalog `catalog.schema.table` with `inputs`/`expectations` columns). Each row is bound to `t.input`/`t.expected`. Requires the runner to have a workspace client + warehouse (`--warehouse-id`). Omit for a single-run eval.
+
+#### limit?[​](#limit "Direct link to limit?")
+
+```ts
+optional limit: number;
+
+```
+
+#### table[​](#table "Direct link to table")
+
+```ts
+table: string;
+
+```
+
+***
+
 ### description?[​](#description "Direct link to description?")
 
 ```ts
@@ -24,6 +52,28 @@ optional description: string;
 
 Short human description, shown in reports.
 
+***
+
+### tags?[​](#tags "Direct link to tags?")
+
+```ts
+optional tags: string[];
+
+```
+
+Free-form tags for filtering (see the runner's `tags` / `--tag` option).
+
+***
+
+### timeoutMs?[​](#timeoutms "Direct link to timeoutMs?")
+
+```ts
+optional timeoutMs: number;
+
+```
+
+Per-eval timeout (ms): `runEval` races the test against it and records a non-passing result instead of hanging. Overrides the runner/CLI default.
+
 ## Methods[​](#methods "Direct link to Methods")
 
 ### test()[​](#test "Direct link to test()")

package/docs/api/appkit/Interface.EvalDriver.md

@@ -4,19 +4,40 @@ Abstraction over how the agent is driven. The HTTP driver posts to a running app
 
 ## Methods[​](#methods "Direct link to Methods")
 
+### reset()?[​](#reset "Direct link to reset()?")
+
+```ts
+optional reset(): void;
+
+```
+
+Drop the current conversation so the next `send` starts a fresh thread. Optional: drivers without a session concept omit it.
+
+#### Returns[​](#returns "Direct link to Returns")
+
+`void`
+
+***
+
 ### send()[​](#send "Direct link to send()")
 
 ```ts
-send(message: string): Promise<DriveResult>;
+send(message: string, options?: {
+  signal?: AbortSignal;
+}): Promise<DriveResult>;
 
 ```
 
+Drive one turn. `options.signal`, when provided, aborts the in-flight turn: the runner passes its per-eval timeout signal so a timed-out eval cancels the request instead of leaking a live stream.
+
 #### Parameters[​](#parameters "Direct link to Parameters")
 
-| Parameter | Type     |
-| --------- | -------- |
-| `message` | `string` |
+| Parameter         | Type                          |
+| ----------------- | ----------------------------- |
+| `message`         | `string`                      |
+| `options?`        | { `signal?`: `AbortSignal`; } |
+| `options.signal?` | `AbortSignal`                 |
 
-#### Returns[​](#returns "Direct link to Returns")
+#### Returns[​](#returns-1 "Direct link to Returns")
 
 `Promise`<[`DriveResult`](./docs/api/appkit/Interface.DriveResult.md)>

package/docs/api/appkit/Interface.EvalResult.md

@@ -42,6 +42,17 @@ id: string;
 
 ***
 
+### infraFailure?[​](#infrafailure "Direct link to infraFailure?")
+
+```ts
+optional infraFailure: boolean;
+
+```
+
+A turn failed at the transport/agent level (`succeeded: false`), not on an assertion — a retryable infra flake, distinct from `error`.
+
+***
+
 ### passed[​](#passed "Direct link to passed")
 
 ```ts

package/docs/api/appkit/Interface.EvalSummary.md

@@ -31,6 +31,17 @@ passed: number;
 
 ***
 
+### passRate[​](#passrate "Direct link to passRate")
+
+```ts
+passRate: number;
+
+```
+
+Fraction of scored (non-skipped) evals that passed, 0..1 (1 when none scored).
+
+***
+
 ### skipped[​](#skipped "Direct link to skipped")
 
 ```ts

package/docs/api/appkit/Interface.HookApp.md

@@ -0,0 +1,12 @@
+# Interface: HookApp
+
+The only capability a hook receives: entities bound to its transaction.
+
+## Properties[​](#properties "Direct link to Properties")
+
+### database[​](#database "Direct link to database")
+
+```ts
+readonly database: TransactionClient;
+
+```

package/docs/api/appkit/Interface.HookContext.md

@@ -0,0 +1,21 @@
+# Interface: HookContext
+
+Which entity is being mutated, and the surface a hook may write through.
+
+## Properties[​](#properties "Direct link to Properties")
+
+### app[​](#app "Direct link to app")
+
+```ts
+readonly app: HookApp;
+
+```
+
+***
+
+### entity[​](#entity "Direct link to entity")
+
+```ts
+readonly entity: string;
+
+```

package/docs/api/appkit/Interface.ReadEvalDatasetOptions.md

@@ -0,0 +1,34 @@
+# Interface: ReadEvalDatasetOptions
+
+## Properties[​](#properties "Direct link to Properties")
+
+### limit?[​](#limit "Direct link to limit?")
+
+```ts
+optional limit: number;
+
+```
+
+Optional row cap.
+
+***
+
+### table[​](#table "Direct link to table")
+
+```ts
+table: string;
+
+```
+
+Fully-qualified UC table: `catalog.schema.table`.
+
+***
+
+### warehouseId[​](#warehouseid "Direct link to warehouseId")
+
+```ts
+warehouseId: string;
+
+```
+
+SQL warehouse id to run the read against.

package/docs/api/appkit/Interface.ReadSerializerContext.md

@@ -0,0 +1,21 @@
+# Interface: ReadSerializerContext
+
+Which entity and generated operation produced the row being shaped.
+
+## Properties[​](#properties "Direct link to Properties")
+
+### entity[​](#entity "Direct link to entity")
+
+```ts
+readonly entity: string;
+
+```
+
+***
+
+### operation[​](#operation "Direct link to operation")
+
+```ts
+readonly operation: "detail" | "list";
+
+```

package/docs/api/appkit/Interface.RunEvalOptions.md

@@ -24,6 +24,17 @@ Stable id for the eval (e.g. its file path relative to the evals dir).
 
 ***
 
+### row?[​](#row "Direct link to row?")
+
+```ts
+optional row: DatasetRow;
+
+```
+
+Dataset row bound to `t.input`/`t.expected` for dataset-driven evals.
+
+***
+
 ### strict?[​](#strict "Direct link to strict?")
 
 ```ts
@@ -32,3 +43,14 @@ optional strict: boolean;
 ```
 
 When true, soft assertion failures also fail the eval.
+
+***
+
+### timeoutMs?[​](#timeoutms "Direct link to timeoutMs?")
+
+```ts
+optional timeoutMs: number;
+
+```
+
+Runner-level default per-eval timeout (ms). `def.timeoutMs` wins over this; when both are unset the eval runs unbounded (current behavior).

package/docs/api/appkit/Interface.RunEvalsOptions.md

@@ -160,6 +160,17 @@ Progress callback, invoked as evals are discovered, started, and finished.
 
 ***
 
+### retries?[​](#retries "Direct link to retries?")
+
+```ts
+optional retries: number;
+
+```
+
+Re-run an eval up to this many extra times when it fails on infrastructure — a thrown error/timeout (`result.error`) or a transport/agent turn failure (`result.infraFailure`). Assertion failures are never retried. Defaults to `0`.
+
+***
+
 ### rootDir?[​](#rootdir "Direct link to rootDir?")
 
 ```ts
@@ -182,6 +193,17 @@ Soft assertion failures also fail the eval.
 
 ***
 
+### tags?[​](#tags "Direct link to tags?")
+
+```ts
+optional tags: string[];
+
+```
+
+Only run evals whose `tags` intersect this list. Empty/undefined runs all. Tags live on the eval def, so filtering happens after each file is loaded.
+
+***
+
 ### timeoutMs?[​](#timeoutms "Direct link to timeoutMs?")
 
 ```ts
@@ -189,4 +211,26 @@ optional timeoutMs: number;
 
 ```
 
-Per-turn wall-clock timeout (ms) before a turn is failed. Defaults to 120s.
+Default per-eval timeout (ms): `runEval` races the whole test against it and it also caps each driver turn. A per-eval `def.timeoutMs` overrides it, and it wins over an agent's `evals.config.ts` `timeoutMs`. Unbounded when unset.
+
+***
+
+### warehouseId?[​](#warehouseid "Direct link to warehouseId?")
+
+```ts
+optional warehouseId: string;
+
+```
+
+SQL warehouse id used to read managed evaluation datasets.
+
+***
+
+### workspaceClient?[​](#workspaceclient "Direct link to workspaceClient?")
+
+```ts
+optional workspaceClient: WorkspaceClient;
+
+```
+
+Workspace client used to read managed evaluation datasets (for evals that declare `dataset`). Required alongside [warehouseId](#warehouseid) for those evals.

package/docs/api/appkit/Interface.TestContext.md

@@ -4,6 +4,28 @@ The `t` context passed to an eval's `test` function.
 
 ## Properties[​](#properties "Direct link to Properties")
 
+### expected[​](#expected "Direct link to expected")
+
+```ts
+readonly expected: Record<string, unknown> | undefined;
+
+```
+
+The current dataset row's `expectations` (ground truth / guidelines), or `undefined` when the row has none or the eval isn't dataset-driven.
+
+***
+
+### input[​](#input "Direct link to input")
+
+```ts
+readonly input: Record<string, unknown>;
+
+```
+
+The current dataset row's `inputs` when the eval is dataset-driven (see [EvalDefinition.dataset](./docs/api/appkit/Interface.EvalDefinition.md#dataset)); `{}` for a plain single-run eval.
+
+***
+
 ### judge[​](#judge "Direct link to judge")
 
 ```ts
@@ -15,7 +37,7 @@ judge: {
 
 ```
 
-LLM-as-judge scoring of the last reply (via autoevals → a Databricks judge model). Each returns a scored, soft-by-default assertion; chain `.atLeast(n)` to set the pass threshold or `.gate()` to make it a hard gate. Requires the judge to be configured (`--judge-model`).
+LLM-as-judge scoring of the last reply (via autoevals → a Databricks judge model). Each returns a scored assertion that gates by default (a miss fails the eval); chain `.atLeast(n)` to change the pass threshold or `.soft()` to demote to a tracked-only metric. Requires the judge to be configured (`--judge-model`).
 
 #### closedQA()[​](#closedqa "Direct link to closedQA()")
 
@@ -130,6 +152,28 @@ Assert a tool was called during the run (gate by default).
 
 ***
 
+### calledToolWith()[​](#calledtoolwith "Direct link to calledToolWith()")
+
+```ts
+calledToolWith(name: string, expected: Record<string, unknown>): AssertionHandle;
+
+```
+
+Assert a tool was called with arguments that deep-contain `expected`: every key in `expected` must equal the actual argument (recursively for nested objects; arrays match element-for-element), so extra arguments are ignored. Gate by default.
+
+#### Parameters[​](#parameters-4 "Direct link to Parameters")
+
+| Parameter  | Type                          |
+| ---------- | ----------------------------- |
+| `name`     | `string`                      |
+| `expected` | `Record`<`string`, `unknown`> |
+
+#### Returns[​](#returns-4 "Direct link to Returns")
+
+[`AssertionHandle`](./docs/api/appkit/Interface.AssertionHandle.md)
+
+***
+
 ### check()[​](#check "Direct link to check()")
 
 ```ts
@@ -139,19 +183,34 @@ check(value: string, matcher: Matcher): AssertionHandle;
 
 Assert a value against a matcher, e.g. `t.check(t.reply, includes("Sunny"))`.
 
-#### Parameters[​](#parameters-4 "Direct link to Parameters")
+#### Parameters[​](#parameters-5 "Direct link to Parameters")
 
 | Parameter | Type                                                      |
 | --------- | --------------------------------------------------------- |
 | `value`   | `string`                                                  |
 | `matcher` | [`Matcher`](./docs/api/appkit/TypeAlias.Matcher.md) |
 
-#### Returns[​](#returns-4 "Direct link to Returns")
+#### Returns[​](#returns-5 "Direct link to Returns")
 
 [`AssertionHandle`](./docs/api/appkit/Interface.AssertionHandle.md)
 
 ***
 
+### reset()[​](#reset "Direct link to reset()")
+
+```ts
+reset(): void;
+
+```
+
+Start a fresh conversation: the next `send` opens a new thread with no history. Use to run several independent one-shot checks in one test. Consecutive `send`s (without a `reset`) stay in one multi-turn conversation.
+
+#### Returns[​](#returns-6 "Direct link to Returns")
+
+`void`
+
+***
+
 ### send()[​](#send "Direct link to send()")
 
 ```ts
@@ -161,13 +220,13 @@ send(message: string): Promise<void>;
 
 Send a user message to the agent and capture its response.
 
-#### Parameters[​](#parameters-5 "Direct link to Parameters")
+#### Parameters[​](#parameters-6 "Direct link to Parameters")
 
 | Parameter | Type     |
 | --------- | -------- |
 | `message` | `string` |
 
-#### Returns[​](#returns-5 "Direct link to Returns")
+#### Returns[​](#returns-7 "Direct link to Returns")
 
 `Promise`<`void`>
 
@@ -182,13 +241,13 @@ skip(reason?: string): never;
 
 Skip this eval with an optional reason.
 
-#### Parameters[​](#parameters-6 "Direct link to Parameters")
+#### Parameters[​](#parameters-7 "Direct link to Parameters")
 
 | Parameter | Type     |
 | --------- | -------- |
 | `reason?` | `string` |
 
-#### Returns[​](#returns-6 "Direct link to Returns")
+#### Returns[​](#returns-8 "Direct link to Returns")
 
 `never`
 
@@ -203,6 +262,6 @@ succeeded(): AssertionHandle;
 
 Assert the last turn completed successfully (gate by default).
 
-#### Returns[​](#returns-7 "Direct link to Returns")
+#### Returns[​](#returns-9 "Direct link to Returns")
 
 [`AssertionHandle`](./docs/api/appkit/Interface.AssertionHandle.md)