@databricks/appkit 0.71.0 → 0.73.0

package/docs/api/appkit/Interface.ReadEvalDatasetOptions.md

@@ -0,0 +1,34 @@
+# Interface: ReadEvalDatasetOptions
+
+## Properties[​](#properties "Direct link to Properties")
+
+### limit?[​](#limit "Direct link to limit?")
+
+```ts
+optional limit: number;
+
+```
+
+Optional row cap.
+
+***
+
+### table[​](#table "Direct link to table")
+
+```ts
+table: string;
+
+```
+
+Fully-qualified UC table: `catalog.schema.table`.
+
+***
+
+### warehouseId[​](#warehouseid "Direct link to warehouseId")
+
+```ts
+warehouseId: string;
+
+```
+
+SQL warehouse id to run the read against.

package/docs/api/appkit/Interface.ReadSerializerContext.md

@@ -0,0 +1,21 @@
+# Interface: ReadSerializerContext
+
+Which entity and generated operation produced the row being shaped.
+
+## Properties[​](#properties "Direct link to Properties")
+
+### entity[​](#entity "Direct link to entity")
+
+```ts
+readonly entity: string;
+
+```
+
+***
+
+### operation[​](#operation "Direct link to operation")
+
+```ts
+readonly operation: "detail" | "list";
+
+```

package/docs/api/appkit/Interface.ReportOutcome.md

@@ -0,0 +1,53 @@
+# Interface: ReportOutcome
+
+## Properties[​](#properties "Direct link to Properties")
+
+### failures[​](#failures "Direct link to failures")
+
+```ts
+failures: {
+  error?: string;
+  status?: number;
+  traceId: string;
+}[];
+
+```
+
+#### error?[​](#error "Direct link to error?")
+
+```ts
+optional error: string;
+
+```
+
+#### status?[​](#status "Direct link to status?")
+
+```ts
+optional status: number;
+
+```
+
+#### traceId[​](#traceid "Direct link to traceId")
+
+```ts
+traceId: string;
+
+```
+
+***
+
+### skipped[​](#skipped "Direct link to skipped")
+
+```ts
+skipped: number;
+
+```
+
+***
+
+### written[​](#written "Direct link to written")
+
+```ts
+written: number;
+
+```

package/docs/api/appkit/Interface.ResolveDatabricksAuthOptions.md

@@ -0,0 +1,34 @@
+# Interface: ResolveDatabricksAuthOptions
+
+## Properties[​](#properties "Direct link to Properties")
+
+### host?[​](#host "Direct link to host?")
+
+```ts
+optional host: string;
+
+```
+
+Explicit host; wins over the profile/SDK-resolved host when set.
+
+***
+
+### profile?[​](#profile "Direct link to profile?")
+
+```ts
+optional profile: string;
+
+```
+
+`~/.databrickscfg` profile to authenticate with (e.g. `dogfood`).
+
+***
+
+### token?[​](#token "Direct link to token?")
+
+```ts
+optional token: string;
+
+```
+
+Explicit bearer token; when set, no OAuth is minted (PAT/CI path).

package/docs/api/appkit/Interface.RunEvalOptions.md

@@ -0,0 +1,45 @@
+# Interface: RunEvalOptions
+
+## Properties[​](#properties "Direct link to Properties")
+
+### driver[​](#driver "Direct link to driver")
+
+```ts
+driver: EvalDriver;
+
+```
+
+Drives the agent and returns reply/tool-calls/success per `send`.
+
+***
+
+### id[​](#id "Direct link to id")
+
+```ts
+id: string;
+
+```
+
+Stable id for the eval (e.g. its file path relative to the evals dir).
+
+***
+
+### row?[​](#row "Direct link to row?")
+
+```ts
+optional row: DatasetRow;
+
+```
+
+Dataset row bound to `t.input`/`t.expected` for dataset-driven evals.
+
+***
+
+### strict?[​](#strict "Direct link to strict?")
+
+```ts
+optional strict: boolean;
+
+```
+
+When true, soft assertion failures also fail the eval.

package/docs/api/appkit/Interface.RunEvalsOptions.md

@@ -0,0 +1,214 @@
+# Interface: RunEvalsOptions
+
+## Properties[​](#properties "Direct link to Properties")
+
+### baseUrl[​](#baseurl "Direct link to baseUrl")
+
+```ts
+baseUrl: string;
+
+```
+
+Base URL of the running app to drive, e.g. `http://localhost:3000`.
+
+***
+
+### concurrency?[​](#concurrency "Direct link to concurrency?")
+
+```ts
+optional concurrency: number;
+
+```
+
+Max evals to drive concurrently. Each eval opens one stream to the app as the same user, so keep this at or below the app's `maxConcurrentStreamsPerUser` (default 5) or the surplus streams hit the 429 guard. Defaults to 4; clamped to `[1, total]`.
+
+***
+
+### filter?[​](#filter "Direct link to filter?")
+
+```ts
+optional filter: string;
+
+```
+
+Substring filter on `<agent>/<id>` (or an exact agent id).
+
+***
+
+### headers?[​](#headers "Direct link to headers?")
+
+```ts
+optional headers: Record<string, string>;
+
+```
+
+Extra request headers for the driver (e.g. auth for a deployed app).
+
+***
+
+### judge?[​](#judge "Direct link to judge?")
+
+```ts
+optional judge: {
+  host: string;
+  model: string;
+  token: string;
+};
+
+```
+
+When set, enable `t.judge.*` LLM-as-judge scoring via autoevals against a Databricks serving endpoint (`model`).
+
+#### host[​](#host "Direct link to host")
+
+```ts
+host: string;
+
+```
+
+#### model[​](#model "Direct link to model")
+
+```ts
+model: string;
+
+```
+
+#### token[​](#token "Direct link to token")
+
+```ts
+token: string;
+
+```
+
+***
+
+### mlflow?[​](#mlflow "Direct link to mlflow?")
+
+```ts
+optional mlflow: {
+  experimentId: string;
+  host: string;
+  sqlWarehouseId?: string;
+  token: string;
+};
+
+```
+
+When set, create a native MLflow "Evaluation run": each eval's trace is linked to the run, pass/fail is written as feedback, and aggregate metrics are logged. Requires Databricks creds + the target experiment.
+
+#### experimentId[​](#experimentid "Direct link to experimentId")
+
+```ts
+experimentId: string;
+
+```
+
+#### host[​](#host-1 "Direct link to host")
+
+```ts
+host: string;
+
+```
+
+#### sqlWarehouseId?[​](#sqlwarehouseid "Direct link to sqlWarehouseId?")
+
+```ts
+optional sqlWarehouseId: string;
+
+```
+
+SQL warehouse id for writing assessments to UC-backed (V4) traces.
+
+#### token[​](#token-1 "Direct link to token")
+
+```ts
+token: string;
+
+```
+
+***
+
+### now?[​](#now "Direct link to now?")
+
+```ts
+optional now: number;
+
+```
+
+Wall-clock timestamp (ms) for run create/finish — pass `Date.now()`.
+
+***
+
+### onEvent()?[​](#onevent "Direct link to onEvent()?")
+
+```ts
+optional onEvent: (event: EvalProgress) => void;
+
+```
+
+Progress callback, invoked as evals are discovered, started, and finished.
+
+#### Parameters[​](#parameters "Direct link to Parameters")
+
+| Parameter | Type                                                                |
+| --------- | ------------------------------------------------------------------- |
+| `event`   | [`EvalProgress`](./docs/api/appkit/TypeAlias.EvalProgress.md) |
+
+#### Returns[​](#returns "Direct link to Returns")
+
+`void`
+
+***
+
+### rootDir?[​](#rootdir "Direct link to rootDir?")
+
+```ts
+optional rootDir: string;
+
+```
+
+Project root containing `server/agents/`. Defaults to `process.cwd()`.
+
+***
+
+### strict?[​](#strict "Direct link to strict?")
+
+```ts
+optional strict: boolean;
+
+```
+
+Soft assertion failures also fail the eval.
+
+***
+
+### timeoutMs?[​](#timeoutms "Direct link to timeoutMs?")
+
+```ts
+optional timeoutMs: number;
+
+```
+
+Per-turn wall-clock timeout (ms) before a turn is failed. Defaults to 120s.
+
+***
+
+### warehouseId?[​](#warehouseid "Direct link to warehouseId?")
+
+```ts
+optional warehouseId: string;
+
+```
+
+SQL warehouse id used to read managed evaluation datasets.
+
+***
+
+### workspaceClient?[​](#workspaceclient "Direct link to workspaceClient?")
+
+```ts
+optional workspaceClient: WorkspaceClient;
+
+```
+
+Workspace client used to read managed evaluation datasets (for evals that declare `dataset`). Required alongside [warehouseId](#warehouseid) for those evals.

package/docs/api/appkit/Interface.TestContext.md

@@ -0,0 +1,245 @@
+# Interface: TestContext
+
+The `t` context passed to an eval's `test` function.
+
+## Properties[​](#properties "Direct link to Properties")
+
+### expected[​](#expected "Direct link to expected")
+
+```ts
+readonly expected: Record<string, unknown> | undefined;
+
+```
+
+The current dataset row's `expectations` (ground truth / guidelines), or `undefined` when the row has none or the eval isn't dataset-driven.
+
+***
+
+### input[​](#input "Direct link to input")
+
+```ts
+readonly input: Record<string, unknown>;
+
+```
+
+The current dataset row's `inputs` when the eval is dataset-driven (see [EvalDefinition.dataset](./docs/api/appkit/Interface.EvalDefinition.md#dataset)); `{}` for a plain single-run eval.
+
+***
+
+### judge[​](#judge "Direct link to judge")
+
+```ts
+judge: {
+  closedQA: Promise<AssertionHandle>;
+  custom: Promise<AssertionHandle>;
+  factuality: Promise<AssertionHandle>;
+};
+
+```
+
+LLM-as-judge scoring of the last reply (via autoevals → a Databricks judge model). Each returns a scored assertion that gates by default (a miss fails the eval); chain `.atLeast(n)` to change the pass threshold or `.soft()` to demote to a tracked-only metric. Requires the judge to be configured (`--judge-model`).
+
+#### closedQA()[​](#closedqa "Direct link to closedQA()")
+
+```ts
+closedQA(criteria: string): Promise<AssertionHandle>;
+
+```
+
+Score whether the reply answers the question, per optional `criteria`.
+
+##### Parameters[​](#parameters "Direct link to Parameters")
+
+| Parameter  | Type     |
+| ---------- | -------- |
+| `criteria` | `string` |
+
+##### Returns[​](#returns "Direct link to Returns")
+
+`Promise`<[`AssertionHandle`](./docs/api/appkit/Interface.AssertionHandle.md)>
+
+#### custom()[​](#custom "Direct link to custom()")
+
+```ts
+custom(spec: CustomJudgeSpec): Promise<AssertionHandle>;
+
+```
+
+A custom prompt-template judge (the TS analog of MLflow's `@scorer`).
+
+##### Parameters[​](#parameters-1 "Direct link to Parameters")
+
+| Parameter | Type                                                                      |
+| --------- | ------------------------------------------------------------------------- |
+| `spec`    | [`CustomJudgeSpec`](./docs/api/appkit/Interface.CustomJudgeSpec.md) |
+
+##### Returns[​](#returns-1 "Direct link to Returns")
+
+`Promise`<[`AssertionHandle`](./docs/api/appkit/Interface.AssertionHandle.md)>
+
+#### factuality()[​](#factuality "Direct link to factuality()")
+
+```ts
+factuality(expected: string): Promise<AssertionHandle>;
+
+```
+
+Score factuality of the reply against an expected reference.
+
+##### Parameters[​](#parameters-2 "Direct link to Parameters")
+
+| Parameter  | Type     |
+| ---------- | -------- |
+| `expected` | `string` |
+
+##### Returns[​](#returns-2 "Direct link to Returns")
+
+`Promise`<[`AssertionHandle`](./docs/api/appkit/Interface.AssertionHandle.md)>
+
+***
+
+### reply[​](#reply "Direct link to reply")
+
+```ts
+readonly reply: string;
+
+```
+
+The last assistant reply.
+
+***
+
+### sessionId[​](#sessionid "Direct link to sessionId")
+
+```ts
+readonly sessionId: string | undefined;
+
+```
+
+The current session/thread id, if any.
+
+***
+
+### toolCalls[​](#toolcalls "Direct link to toolCalls")
+
+```ts
+readonly toolCalls: string[];
+
+```
+
+Tools called during the last turn.
+
+## Methods[​](#methods "Direct link to Methods")
+
+### calledTool()[​](#calledtool "Direct link to calledTool()")
+
+```ts
+calledTool(name: string): AssertionHandle;
+
+```
+
+Assert a tool was called during the run (gate by default).
+
+#### Parameters[​](#parameters-3 "Direct link to Parameters")
+
+| Parameter | Type     |
+| --------- | -------- |
+| `name`    | `string` |
+
+#### Returns[​](#returns-3 "Direct link to Returns")
+
+[`AssertionHandle`](./docs/api/appkit/Interface.AssertionHandle.md)
+
+***
+
+### check()[​](#check "Direct link to check()")
+
+```ts
+check(value: string, matcher: Matcher): AssertionHandle;
+
+```
+
+Assert a value against a matcher, e.g. `t.check(t.reply, includes("Sunny"))`.
+
+#### Parameters[​](#parameters-4 "Direct link to Parameters")
+
+| Parameter | Type                                                      |
+| --------- | --------------------------------------------------------- |
+| `value`   | `string`                                                  |
+| `matcher` | [`Matcher`](./docs/api/appkit/TypeAlias.Matcher.md) |
+
+#### Returns[​](#returns-4 "Direct link to Returns")
+
+[`AssertionHandle`](./docs/api/appkit/Interface.AssertionHandle.md)
+
+***
+
+### reset()[​](#reset "Direct link to reset()")
+
+```ts
+reset(): void;
+
+```
+
+Start a fresh conversation: the next `send` opens a new thread with no history. Use to run several independent one-shot checks in one test. Consecutive `send`s (without a `reset`) stay in one multi-turn conversation.
+
+#### Returns[​](#returns-5 "Direct link to Returns")
+
+`void`
+
+***
+
+### send()[​](#send "Direct link to send()")
+
+```ts
+send(message: string): Promise<void>;
+
+```
+
+Send a user message to the agent and capture its response.
+
+#### Parameters[​](#parameters-5 "Direct link to Parameters")
+
+| Parameter | Type     |
+| --------- | -------- |
+| `message` | `string` |
+
+#### Returns[​](#returns-6 "Direct link to Returns")
+
+`Promise`<`void`>
+
+***
+
+### skip()[​](#skip "Direct link to skip()")
+
+```ts
+skip(reason?: string): never;
+
+```
+
+Skip this eval with an optional reason.
+
+#### Parameters[​](#parameters-6 "Direct link to Parameters")
+
+| Parameter | Type     |
+| --------- | -------- |
+| `reason?` | `string` |
+
+#### Returns[​](#returns-7 "Direct link to Returns")
+
+`never`
+
+***
+
+### succeeded()[​](#succeeded "Direct link to succeeded()")
+
+```ts
+succeeded(): AssertionHandle;
+
+```
+
+Assert the last turn completed successfully (gate by default).
+
+#### Returns[​](#returns-8 "Direct link to Returns")
+
+[`AssertionHandle`](./docs/api/appkit/Interface.AssertionHandle.md)

package/docs/api/appkit/TypeAlias.DatabaseApiConfig.md

@@ -0,0 +1,53 @@
+# Type Alias: DatabaseApiConfig\<TSchema>
+
+```ts
+type DatabaseApiConfig<TSchema> = 
+  | boolean
+  | {
+  tables?: readonly SchemaTableName<TSchema>[];
+  writes?: DatabaseApiWritesConfig<TSchema>;
+};
+
+```
+
+Full generated CRUD for every declared table by default. Set false to disable all generated routes, or use an object to restrict tables and writes. Keyed routes require a public primary key; upsert stays programmatic. Route names must start with a letter, contain only letters, digits, `_`, or `-`, be at most 64 characters, and be unique ignoring case. Invalid names fail setup; exclude internal tables with `api.tables` or use `api: false`.
+
+Routes run as the app's service principal and apply no per-user filter, so anyone the app admits receives every enabled operation. An exposed table also becomes includable from its neighbours; a relation whose target stays off cannot be included.
+
+Text filters accept caller-supplied `like`/`ilike` patterns; a server-side `statement_timeout` cancels a pattern that would otherwise hold its pooled connection to completion.
+
+## Type Parameters[​](#type-parameters "Direct link to Type Parameters")
+
+| Type Parameter                                                              |
+| --------------------------------------------------------------------------- |
+| `TSchema` *extends* [`Schema`](./docs/api/appkit/Interface.Schema.md) |
+
+## Type Declaration[​](#type-declaration "Direct link to Type Declaration")
+
+`boolean`
+
+```ts
+{
+  tables?: readonly SchemaTableName<TSchema>[];
+  writes?: DatabaseApiWritesConfig<TSchema>;
+}
+
+```
+
+### tables?[​](#tables "Direct link to tables?")
+
+```ts
+readonly optional tables: readonly SchemaTableName<TSchema>[];
+
+```
+
+Tables that remain exposed. Defaults to every declared table.
+
+### writes?[​](#writes "Direct link to writes?")
+
+```ts
+readonly optional writes: DatabaseApiWritesConfig<TSchema>;
+
+```
+
+All writes by default. Set false for read-only routes.

package/docs/api/appkit/TypeAlias.DatabaseApiWriteOperation.md

@@ -0,0 +1,8 @@
+# Type Alias: DatabaseApiWriteOperation
+
+```ts
+type DatabaseApiWriteOperation = "create" | "update" | "delete";
+
+```
+
+Generated HTTP write operations.