@databricks/appkit 0.70.0 → 0.72.0

package/docs/api/appkit/Interface.TestContext.md

@@ -0,0 +1,208 @@
+# Interface: TestContext
+
+The `t` context passed to an eval's `test` function.
+
+## Properties[​](#properties "Direct link to Properties")
+
+### judge[​](#judge "Direct link to judge")
+
+```ts
+judge: {
+  closedQA: Promise<AssertionHandle>;
+  custom: Promise<AssertionHandle>;
+  factuality: Promise<AssertionHandle>;
+};
+
+```
+
+LLM-as-judge scoring of the last reply (via autoevals → a Databricks judge model). Each returns a scored, soft-by-default assertion; chain `.atLeast(n)` to set the pass threshold or `.gate()` to make it a hard gate. Requires the judge to be configured (`--judge-model`).
+
+#### closedQA()[​](#closedqa "Direct link to closedQA()")
+
+```ts
+closedQA(criteria: string): Promise<AssertionHandle>;
+
+```
+
+Score whether the reply answers the question, per optional `criteria`.
+
+##### Parameters[​](#parameters "Direct link to Parameters")
+
+| Parameter  | Type     |
+| ---------- | -------- |
+| `criteria` | `string` |
+
+##### Returns[​](#returns "Direct link to Returns")
+
+`Promise`<[`AssertionHandle`](./docs/api/appkit/Interface.AssertionHandle.md)>
+
+#### custom()[​](#custom "Direct link to custom()")
+
+```ts
+custom(spec: CustomJudgeSpec): Promise<AssertionHandle>;
+
+```
+
+A custom prompt-template judge (the TS analog of MLflow's `@scorer`).
+
+##### Parameters[​](#parameters-1 "Direct link to Parameters")
+
+| Parameter | Type                                                                      |
+| --------- | ------------------------------------------------------------------------- |
+| `spec`    | [`CustomJudgeSpec`](./docs/api/appkit/Interface.CustomJudgeSpec.md) |
+
+##### Returns[​](#returns-1 "Direct link to Returns")
+
+`Promise`<[`AssertionHandle`](./docs/api/appkit/Interface.AssertionHandle.md)>
+
+#### factuality()[​](#factuality "Direct link to factuality()")
+
+```ts
+factuality(expected: string): Promise<AssertionHandle>;
+
+```
+
+Score factuality of the reply against an expected reference.
+
+##### Parameters[​](#parameters-2 "Direct link to Parameters")
+
+| Parameter  | Type     |
+| ---------- | -------- |
+| `expected` | `string` |
+
+##### Returns[​](#returns-2 "Direct link to Returns")
+
+`Promise`<[`AssertionHandle`](./docs/api/appkit/Interface.AssertionHandle.md)>
+
+***
+
+### reply[​](#reply "Direct link to reply")
+
+```ts
+readonly reply: string;
+
+```
+
+The last assistant reply.
+
+***
+
+### sessionId[​](#sessionid "Direct link to sessionId")
+
+```ts
+readonly sessionId: string | undefined;
+
+```
+
+The current session/thread id, if any.
+
+***
+
+### toolCalls[​](#toolcalls "Direct link to toolCalls")
+
+```ts
+readonly toolCalls: string[];
+
+```
+
+Tools called during the last turn.
+
+## Methods[​](#methods "Direct link to Methods")
+
+### calledTool()[​](#calledtool "Direct link to calledTool()")
+
+```ts
+calledTool(name: string): AssertionHandle;
+
+```
+
+Assert a tool was called during the run (gate by default).
+
+#### Parameters[​](#parameters-3 "Direct link to Parameters")
+
+| Parameter | Type     |
+| --------- | -------- |
+| `name`    | `string` |
+
+#### Returns[​](#returns-3 "Direct link to Returns")
+
+[`AssertionHandle`](./docs/api/appkit/Interface.AssertionHandle.md)
+
+***
+
+### check()[​](#check "Direct link to check()")
+
+```ts
+check(value: string, matcher: Matcher): AssertionHandle;
+
+```
+
+Assert a value against a matcher, e.g. `t.check(t.reply, includes("Sunny"))`.
+
+#### Parameters[​](#parameters-4 "Direct link to Parameters")
+
+| Parameter | Type                                                      |
+| --------- | --------------------------------------------------------- |
+| `value`   | `string`                                                  |
+| `matcher` | [`Matcher`](./docs/api/appkit/TypeAlias.Matcher.md) |
+
+#### Returns[​](#returns-4 "Direct link to Returns")
+
+[`AssertionHandle`](./docs/api/appkit/Interface.AssertionHandle.md)
+
+***
+
+### send()[​](#send "Direct link to send()")
+
+```ts
+send(message: string): Promise<void>;
+
+```
+
+Send a user message to the agent and capture its response.
+
+#### Parameters[​](#parameters-5 "Direct link to Parameters")
+
+| Parameter | Type     |
+| --------- | -------- |
+| `message` | `string` |
+
+#### Returns[​](#returns-5 "Direct link to Returns")
+
+`Promise`<`void`>
+
+***
+
+### skip()[​](#skip "Direct link to skip()")
+
+```ts
+skip(reason?: string): never;
+
+```
+
+Skip this eval with an optional reason.
+
+#### Parameters[​](#parameters-6 "Direct link to Parameters")
+
+| Parameter | Type     |
+| --------- | -------- |
+| `reason?` | `string` |
+
+#### Returns[​](#returns-6 "Direct link to Returns")
+
+`never`
+
+***
+
+### succeeded()[​](#succeeded "Direct link to succeeded()")
+
+```ts
+succeeded(): AssertionHandle;
+
+```
+
+Assert the last turn completed successfully (gate by default).
+
+#### Returns[​](#returns-7 "Direct link to Returns")
+
+[`AssertionHandle`](./docs/api/appkit/Interface.AssertionHandle.md)

package/docs/api/appkit/TypeAlias.EvalProgress.md

@@ -0,0 +1,26 @@
+# Type Alias: EvalProgress
+
+```ts
+type EvalProgress = 
+  | {
+  total: number;
+  type: "discovered";
+}
+  | {
+  runId: string;
+  type: "run-created";
+}
+  | {
+  id: string;
+  index: number;
+  total: number;
+  type: "start";
+}
+  | {
+  index: number;
+  result: EvalResult;
+  total: number;
+  type: "result";
+};
+
+```

package/docs/api/appkit/TypeAlias.Matcher.md

@@ -0,0 +1,18 @@
+# Type Alias: Matcher()
+
+```ts
+type Matcher = (value: string) => MatchResult;
+
+```
+
+A deterministic matcher: inspects a string value and returns a result.
+
+## Parameters[​](#parameters "Direct link to Parameters")
+
+| Parameter | Type     |
+| --------- | -------- |
+| `value`   | `string` |
+
+## Returns[​](#returns "Direct link to Returns")
+
+[`MatchResult`](./docs/api/appkit/Interface.MatchResult.md)

package/docs/api/appkit/TypeAlias.Severity.md

@@ -0,0 +1,8 @@
+# Type Alias: Severity
+
+```ts
+type Severity = "gate" | "soft";
+
+```
+
+Whether an assertion fails the eval (`gate`) or is tracked only (`soft`).

package/docs/api/appkit.md

@@ -11,23 +11,24 @@ Documentation merge entry for Typedoc — combines the stable `@databricks/appki
 
 ## Classes[​](#classes "Direct link to Classes")
 
-| Class                                                                         | Description                                                                                                                                    |
-| ----------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
-| [AppKitError](./docs/api/appkit/Class.AppKitError.md)                   | Base error class for all AppKit errors. Provides a consistent structure for error handling across the framework.                               |
-| [AppKitMcpClient](./docs/api/appkit/Class.AppKitMcpClient.md)           | Lightweight MCP client for Databricks-hosted MCP servers.                                                                                      |
-| [AuthenticationError](./docs/api/appkit/Class.AuthenticationError.md)   | Error thrown when authentication fails. Use for missing tokens, invalid credentials, or authorization failures.                                |
-| [ConfigurationError](./docs/api/appkit/Class.ConfigurationError.md)     | Error thrown when configuration is missing or invalid. Use for missing environment variables, invalid settings, or setup issues.               |
-| [ConnectionError](./docs/api/appkit/Class.ConnectionError.md)           | Error thrown when a connection or network operation fails. Use for database pool errors, API failures, timeouts, etc.                          |
-| [DatabricksAdapter](./docs/api/appkit/Class.DatabricksAdapter.md)       | Adapter that talks directly to Databricks Model Serving `/invocations` endpoint.                                                               |
-| [ExecutionError](./docs/api/appkit/Class.ExecutionError.md)             | Error thrown when an operation execution fails. Use for statement failures, canceled operations, or unexpected states.                         |
-| [InitializationError](./docs/api/appkit/Class.InitializationError.md)   | Error thrown when a service or component is not properly initialized. Use when accessing services before they are ready.                       |
-| [Plugin](./docs/api/appkit/Class.Plugin.md)                             | Base abstract class for creating AppKit plugins.                                                                                               |
-| [PolicyDeniedError](./docs/api/appkit/Class.PolicyDeniedError.md)       | Thrown when a policy denies an action.                                                                                                         |
-| [ResourceRegistry](./docs/api/appkit/Class.ResourceRegistry.md)         | Central registry for tracking plugin resource requirements. Deduplication uses type + resourceKey (machine-stable); alias is for display only. |
-| [ServerError](./docs/api/appkit/Class.ServerError.md)                   | Error thrown when server lifecycle operations fail. Use for server start/stop issues, configuration conflicts, etc.                            |
-| [SupervisorApiAdapter](./docs/api/appkit/Class.SupervisorApiAdapter.md) | Adapter that calls the Databricks AI Gateway Responses API (`/ai-gateway/mlflow/v1/responses`).                                                |
-| [TunnelError](./docs/api/appkit/Class.TunnelError.md)                   | Error thrown when remote tunnel operations fail. Use for tunnel connection issues, message parsing failures, etc.                              |
-| [ValidationError](./docs/api/appkit/Class.ValidationError.md)           | Error thrown when input validation fails. Use for invalid parameters, missing required fields, or type mismatches.                             |
+| Class                                                                         | Description                                                                                                                                                                                                                                                    |
+| ----------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [AppKitError](./docs/api/appkit/Class.AppKitError.md)                   | Base error class for all AppKit errors. Provides a consistent structure for error handling across the framework.                                                                                                                                               |
+| [AppKitMcpClient](./docs/api/appkit/Class.AppKitMcpClient.md)           | Lightweight MCP client for Databricks-hosted MCP servers.                                                                                                                                                                                                      |
+| [AuthenticationError](./docs/api/appkit/Class.AuthenticationError.md)   | Error thrown when authentication fails. Use for missing tokens, invalid credentials, or authorization failures.                                                                                                                                                |
+| [ConfigurationError](./docs/api/appkit/Class.ConfigurationError.md)     | Error thrown when configuration is missing or invalid. Use for missing environment variables, invalid settings, or setup issues.                                                                                                                               |
+| [ConnectionError](./docs/api/appkit/Class.ConnectionError.md)           | Error thrown when a connection or network operation fails. Use for database pool errors, API failures, timeouts, etc.                                                                                                                                          |
+| [DatabricksAdapter](./docs/api/appkit/Class.DatabricksAdapter.md)       | Adapter that talks directly to Databricks Model Serving `/invocations` endpoint.                                                                                                                                                                               |
+| [ExecutionError](./docs/api/appkit/Class.ExecutionError.md)             | Error thrown when an operation execution fails. Use for statement failures, canceled operations, or unexpected states.                                                                                                                                         |
+| [InitializationError](./docs/api/appkit/Class.InitializationError.md)   | Error thrown when a service or component is not properly initialized. Use when accessing services before they are ready.                                                                                                                                       |
+| [MlflowClient](./docs/api/appkit/Class.MlflowClient.md)                 | A thin client over the Databricks workspace REST API, owning the host + bearer token so callers (eval-run creation, assessment writes, the judge's serving endpoint) don't each re-derive URLs or re-attach auth. The host is normalized once at construction. |
+| [Plugin](./docs/api/appkit/Class.Plugin.md)                             | Base abstract class for creating AppKit plugins.                                                                                                                                                                                                               |
+| [PolicyDeniedError](./docs/api/appkit/Class.PolicyDeniedError.md)       | Thrown when a policy denies an action.                                                                                                                                                                                                                         |
+| [ResourceRegistry](./docs/api/appkit/Class.ResourceRegistry.md)         | Central registry for tracking plugin resource requirements. Deduplication uses type + resourceKey (machine-stable); alias is for display only.                                                                                                                 |
+| [ServerError](./docs/api/appkit/Class.ServerError.md)                   | Error thrown when server lifecycle operations fail. Use for server start/stop issues, configuration conflicts, etc.                                                                                                                                            |
+| [SupervisorApiAdapter](./docs/api/appkit/Class.SupervisorApiAdapter.md) | Adapter that calls the Databricks AI Gateway Responses API (`/ai-gateway/mlflow/v1/responses`).                                                                                                                                                                |
+| [TunnelError](./docs/api/appkit/Class.TunnelError.md)                   | Error thrown when remote tunnel operations fail. Use for tunnel connection issues, message parsing failures, etc.                                                                                                                                              |
+| [ValidationError](./docs/api/appkit/Class.ValidationError.md)           | Error thrown when input validation fails. Use for invalid parameters, missing required fields, or type mismatches.                                                                                                                                             |
 
 ## Interfaces[​](#interfaces "Direct link to Interfaces")
 
@@ -39,18 +40,31 @@ Documentation merge entry for Typedoc — combines the stable `@databricks/appki
 | [AgentRunContext](./docs/api/appkit/Interface.AgentRunContext.md)                                     | -                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
 | [AgentsPluginConfig](./docs/api/appkit/Interface.AgentsPluginConfig.md)                               | Base configuration interface for AppKit plugins                                                                                                                                                                                                                                                                                                                                                                                                               |
 | [AgentToolDefinition](./docs/api/appkit/Interface.AgentToolDefinition.md)                             | -                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
+| [AssertionHandle](./docs/api/appkit/Interface.AssertionHandle.md)                                     | Chainable handle returned by every assertion to control its severity. Mirrors eve: assertions are gates by default; `.soft()` demotes to a tracked metric; `.atLeast(n)` is a soft, score-thresholded assertion.                                                                                                                                                                                                                                              |
+| [AssertionResult](./docs/api/appkit/Interface.AssertionResult.md)                                     | A single recorded assertion outcome.                                                                                                                                                                                                                                                                                                                                                                                                                          |
+| [Assessment](./docs/api/appkit/Interface.Assessment.md)                                               | A Feedback assessment in the MLflow REST proto-JSON shape.                                                                                                                                                                                                                                                                                                                                                                                                    |
 | [AutoInheritToolsConfig](./docs/api/appkit/Interface.AutoInheritToolsConfig.md)                       | Auto-inherit configuration. When enabled for a given agent origin, agents with no explicit `tools:` declaration receive every registered ToolProvider plugin tool whose author marked `autoInheritable: true`. Tools without that flag — destructive, state-mutating, or privilege-sensitive — never spread automatically and must be wired via `tools:` (object or function form in code, `plugin:NAME` entries in markdown frontmatter).                    |
 | [BasePluginConfig](./docs/api/appkit/Interface.BasePluginConfig.md)                                   | Base configuration interface for AppKit plugins                                                                                                                                                                                                                                                                                                                                                                                                               |
 | [CacheConfig](./docs/api/appkit/Interface.CacheConfig.md)                                             | Configuration for the CacheInterceptor. Controls TTL, size limits, storage backend, and probabilistic cleanup.                                                                                                                                                                                                                                                                                                                                                |
+| [CustomJudgeSpec](./docs/api/appkit/Interface.CustomJudgeSpec.md)                                     | A custom LLM-judge definition: a prompt template and choice→score mapping.                                                                                                                                                                                                                                                                                                                                                                                    |
 | [DatabaseCredential](./docs/api/appkit/Interface.DatabaseCredential.md)                               | Database credentials with OAuth token for Postgres connection                                                                                                                                                                                                                                                                                                                                                                                                 |
 | [DatabaseRegistry](./docs/api/appkit/Interface.DatabaseRegistry.md)                                   | CANONICAL augmentation target. Empty by default; the generated `database.d.ts` augments it via `declare module "@databricks/appkit" { interface DatabaseRegistry { ... } }`.                                                                                                                                                                                                                                                                                  |
+| [DatabricksAuth](./docs/api/appkit/Interface.DatabricksAuth.md)                                       | Resolved Databricks host + bearer token for the eval runner's REST calls.                                                                                                                                                                                                                                                                                                                                                                                     |
+| [DiscoveredEval](./docs/api/appkit/Interface.DiscoveredEval.md)                                       | An eval file found under `server/agents/<agent>/evals/`.                                                                                                                                                                                                                                                                                                                                                                                                      |
+| [DriveResult](./docs/api/appkit/Interface.DriveResult.md)                                             | What a driver returns for a single `t.send`.                                                                                                                                                                                                                                                                                                                                                                                                                  |
 | [EndpointConfig](./docs/api/appkit/Interface.EndpointConfig.md)                                       | -                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
+| [EvalDefinition](./docs/api/appkit/Interface.EvalDefinition.md)                                       | A single eval, default-exported from a `*.eval.ts` file.                                                                                                                                                                                                                                                                                                                                                                                                      |
+| [EvalDriver](./docs/api/appkit/Interface.EvalDriver.md)                                               | Abstraction over how the agent is driven. The HTTP driver posts to a running app's agents endpoint; future drivers (in-process) implement the same shape.                                                                                                                                                                                                                                                                                                     |
+| [EvalResult](./docs/api/appkit/Interface.EvalResult.md)                                               | The outcome of running one eval.                                                                                                                                                                                                                                                                                                                                                                                                                              |
+| [EvalRunSummary](./docs/api/appkit/Interface.EvalRunSummary.md)                                       | -                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
+| [EvalSummary](./docs/api/appkit/Interface.EvalSummary.md)                                             | -                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
 | [FilePolicyUser](./docs/api/appkit/Interface.FilePolicyUser.md)                                       | Minimal user identity passed to the policy function.                                                                                                                                                                                                                                                                                                                                                                                                          |
 | [FileResource](./docs/api/appkit/Interface.FileResource.md)                                           | Describes the file or directory being acted upon.                                                                                                                                                                                                                                                                                                                                                                                                             |
 | [FunctionTool](./docs/api/appkit/Interface.FunctionTool.md)                                           | -                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
 | [GenerateDatabaseCredentialRequest](./docs/api/appkit/Interface.GenerateDatabaseCredentialRequest.md) | Request parameters for generating database OAuth credentials                                                                                                                                                                                                                                                                                                                                                                                                  |
 | [GenerationParams](./docs/api/appkit/Interface.GenerationParams.md)                                   | Optional generation parameters forwarded to the OpenAI-compatible serving request body. Names match the serving API wire keys. Only keys that are set are sent — undefined values are omitted so the endpoint applies its own defaults. Ranges are not validated here; the serving endpoint validates.                                                                                                                                                        |
 | [HostedSupervisorTool](./docs/api/appkit/Interface.HostedSupervisorTool.md)                           | Tagged record returned by every [supervisorTools](./docs/api/appkit/Variable.supervisorTools.md) factory. The `__kind` discriminator lets the agents plugin (and standalone `runAgent`) classify these tools without a structural match against the wire format — keeps the SA wire shape free to evolve and avoids namespace collisions with MCP hosted tools (which use `type: "genie-space"` hyphenated, vs SA's `type: "genie_space"` underscored). |
+| [HttpDriverOptions](./docs/api/appkit/Interface.HttpDriverOptions.md)                                 | -                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
 | [IAiSearchConfig](./docs/api/appkit/Interface.IAiSearchConfig.md)                                     | Base configuration interface for AppKit plugins                                                                                                                                                                                                                                                                                                                                                                                                               |
 | [IJobsConfig](./docs/api/appkit/Interface.IJobsConfig.md)                                             | Configuration for the Jobs plugin.                                                                                                                                                                                                                                                                                                                                                                                                                            |
 | [IndexConfig](./docs/api/appkit/Interface.IndexConfig.md)                                             | -                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
@@ -58,22 +72,30 @@ Documentation merge entry for Typedoc — combines the stable `@databricks/appki
 | [JobAPI](./docs/api/appkit/Interface.JobAPI.md)                                                       | User-facing API for a single configured job.                                                                                                                                                                                                                                                                                                                                                                                                                  |
 | [JobConfig](./docs/api/appkit/Interface.JobConfig.md)                                                 | Per-job configuration options.                                                                                                                                                                                                                                                                                                                                                                                                                                |
 | [JobsConnectorConfig](./docs/api/appkit/Interface.JobsConnectorConfig.md)                             | -                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
+| [JudgeConfig](./docs/api/appkit/Interface.JudgeConfig.md)                                             | -                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
+| [JudgeScore](./docs/api/appkit/Interface.JudgeScore.md)                                               | A normalized judge result. `score` is 0..1.                                                                                                                                                                                                                                                                                                                                                                                                                   |
 | [LakebasePool](./docs/api/appkit/Interface.LakebasePool.md)                                           | Subset of `pg.Pool` exposed by the Lakebase plugin.                                                                                                                                                                                                                                                                                                                                                                                                           |
 | [LakebasePoolConfig](./docs/api/appkit/Interface.LakebasePoolConfig.md)                               | Configuration for creating a Lakebase connection pool                                                                                                                                                                                                                                                                                                                                                                                                         |
 | [LakebasePoolManager](./docs/api/appkit/Interface.LakebasePoolManager.md)                             | Manages multiple Lakebase connection pools keyed by an identifier (e.g. userId).                                                                                                                                                                                                                                                                                                                                                                              |
+| [MatchResult](./docs/api/appkit/Interface.MatchResult.md)                                             | Result of a deterministic matcher run against a value.                                                                                                                                                                                                                                                                                                                                                                                                        |
 | [McpConnectAllResult](./docs/api/appkit/Interface.McpConnectAllResult.md)                             | Per-endpoint outcome of [AppKitMcpClient.connectAll](./docs/api/appkit/Class.AppKitMcpClient.md#connectall). Callers (the agents plugin in particular) use the split to warn at startup when some MCP servers are unreachable without aborting boot for the rest.                                                                                                                                                                                       |
 | [Message](./docs/api/appkit/Interface.Message.md)                                                     | -                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
 | [PluginManifest](./docs/api/appkit/Interface.PluginManifest.md)                                       | Plugin manifest that declares metadata and resource requirements. Attached to plugin classes as a static property. Extends the shared PluginManifest with strict resource types.                                                                                                                                                                                                                                                                              |
 | [PluginToolkitProvider](./docs/api/appkit/Interface.PluginToolkitProvider.md)                         | Minimum shape every entry in the [Plugins](./docs/api/appkit/TypeAlias.Plugins.md) map must expose. Core plugins (analytics, files, genie, lakebase) implement this directly via their `.toolkit()` method. The agents plugin and standalone `runAgent` synthesize this shape for any registered plugin that doesn't implement `.toolkit()` directly (falling back to `getAgentTools()` walking).                                                       |
+| [PostResult](./docs/api/appkit/Interface.PostResult.md)                                               | Structured result for a best-effort POST that must not throw.                                                                                                                                                                                                                                                                                                                                                                                                 |
 | [PromptContext](./docs/api/appkit/Interface.PromptContext.md)                                         | Context passed to `baseSystemPrompt` callbacks.                                                                                                                                                                                                                                                                                                                                                                                                               |
 | [RegisteredAgent](./docs/api/appkit/Interface.RegisteredAgent.md)                                     | -                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
+| [ReportOutcome](./docs/api/appkit/Interface.ReportOutcome.md)                                         | -                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
 | [RequestedClaims](./docs/api/appkit/Interface.RequestedClaims.md)                                     | Optional claims for fine-grained Unity Catalog table permissions When specified, the returned token will be scoped to only the requested tables                                                                                                                                                                                                                                                                                                               |
 | [RequestedResource](./docs/api/appkit/Interface.RequestedResource.md)                                 | Resource to request permissions for in Unity Catalog                                                                                                                                                                                                                                                                                                                                                                                                          |
 | [RerankerConfig](./docs/api/appkit/Interface.RerankerConfig.md)                                       | -                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
+| [ResolveDatabricksAuthOptions](./docs/api/appkit/Interface.ResolveDatabricksAuthOptions.md)           | -                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
 | [ResourceEntry](./docs/api/appkit/Interface.ResourceEntry.md)                                         | Internal representation of a resource in the registry. Extends ResourceRequirement with resolution state and plugin ownership.                                                                                                                                                                                                                                                                                                                                |
 | [ResourceRequirement](./docs/api/appkit/Interface.ResourceRequirement.md)                             | Declares a resource requirement for a plugin. Can be defined statically in a manifest or dynamically via getResourceRequirements().                                                                                                                                                                                                                                                                                                                           |
 | [RunAgentInput](./docs/api/appkit/Interface.RunAgentInput.md)                                         | -                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
 | [RunAgentResult](./docs/api/appkit/Interface.RunAgentResult.md)                                       | -                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
+| [RunEvalOptions](./docs/api/appkit/Interface.RunEvalOptions.md)                                       | -                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
+| [RunEvalsOptions](./docs/api/appkit/Interface.RunEvalsOptions.md)                                     | -                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
 | [Schema](./docs/api/appkit/Interface.Schema.md)                                                       | One finalized schema. `TTableName` keeps the declared names in the type, so configuration that addresses a table by name is checked against the schema it was written for. Code that accepts any schema uses the default.                                                                                                                                                                                                                                     |
 | [SearchRequest](./docs/api/appkit/Interface.SearchRequest.md)                                         | -                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
 | [SearchResponse](./docs/api/appkit/Interface.SearchResponse.md)                                       | -                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
@@ -84,6 +106,7 @@ Documentation merge entry for Typedoc — combines the stable `@databricks/appki
 | [SupervisorApiAdapterOptions](./docs/api/appkit/Interface.SupervisorApiAdapterOptions.md)             | -                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
 | [SupervisorExtension](./docs/api/appkit/Interface.SupervisorExtension.md)                             | Shape of the value at `AgentInput.extensions[SUPERVISOR_EXTENSION_KEY]`. The agents plugin / `runAgent` build this from the tool index; advanced callers invoking `adapter.run(...)` directly populate it themselves.                                                                                                                                                                                                                                         |
 | [TelemetryConfig](./docs/api/appkit/Interface.TelemetryConfig.md)                                     | OpenTelemetry configuration for AppKit applications                                                                                                                                                                                                                                                                                                                                                                                                           |
+| [TestContext](./docs/api/appkit/Interface.TestContext.md)                                             | The `t` context passed to an eval's `test` function.                                                                                                                                                                                                                                                                                                                                                                                                          |
 | [Thread](./docs/api/appkit/Interface.Thread.md)                                                       | -                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
 | [ThreadStore](./docs/api/appkit/Interface.ThreadStore.md)                                             | -                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
 | [ToolAnnotations](./docs/api/appkit/Interface.ToolAnnotations.md)                                     | -                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
@@ -108,6 +131,7 @@ Documentation merge entry for Typedoc — combines the stable `@databricks/appki
 | [BaseSystemPromptOption](./docs/api/appkit/TypeAlias.BaseSystemPromptOption.md) | -                                                                                                                                                                                                                                                                                                                      |
 | [ConfigSchema](./docs/api/appkit/TypeAlias.ConfigSchema.md)                     | Configuration schema definition for plugin config. Re-exported from the standard JSON Schema Draft 7 types.                                                                                                                                                                                                            |
 | [DatabaseExports](./docs/api/appkit/TypeAlias.DatabaseExports.md)               | Typed database API published by the plugin.                                                                                                                                                                                                                                                                            |
+| [EvalProgress](./docs/api/appkit/TypeAlias.EvalProgress.md)                     | -                                                                                                                                                                                                                                                                                                                      |
 | [ExecutionResult](./docs/api/appkit/TypeAlias.ExecutionResult.md)               | Discriminated union for plugin execution results.                                                                                                                                                                                                                                                                      |
 | [FileAction](./docs/api/appkit/TypeAlias.FileAction.md)                         | Every action the files plugin can perform.                                                                                                                                                                                                                                                                             |
 | [FilePolicy](./docs/api/appkit/TypeAlias.FilePolicy.md)                         | A policy function that decides whether `user` may perform `action` on `resource`. Return `true` to allow, `false` to deny.                                                                                                                                                                                             |
@@ -115,6 +139,7 @@ Documentation merge entry for Typedoc — combines the stable `@databricks/appki
 | [IAppRouter](./docs/api/appkit/TypeAlias.IAppRouter.md)                         | Express router type for plugin route registration                                                                                                                                                                                                                                                                      |
 | [IDatabaseConfig](./docs/api/appkit/TypeAlias.IDatabaseConfig.md)               | Configuration for one schema-bound DatabasePlugin instance.                                                                                                                                                                                                                                                            |
 | [JobsExport](./docs/api/appkit/TypeAlias.JobsExport.md)                         | Public API shape of the jobs plugin. Callable to select a job by key.                                                                                                                                                                                                                                                  |
+| [Matcher](./docs/api/appkit/TypeAlias.Matcher.md)                               | A deterministic matcher: inspects a string value and returns a result.                                                                                                                                                                                                                                                 |
 | [PluginData](./docs/api/appkit/TypeAlias.PluginData.md)                         | Tuple of plugin class, config, and name. Created by `toPlugin()` and passed to `createApp()`.                                                                                                                                                                                                                          |
 | [Plugins](./docs/api/appkit/TypeAlias.Plugins.md)                               | Plugin map passed to the function form of [AgentDefinition.tools](./docs/api/appkit/Interface.AgentDefinition.md#tools). Each entry exposes a `.toolkit(opts?)` method that returns a record of [ToolkitEntry](./docs/api/appkit/Interface.ToolkitEntry.md) markers ready to be spread into a tool record. |
 | [ResolvedToolEntry](./docs/api/appkit/TypeAlias.ResolvedToolEntry.md)           | Internal tool-index entry after a tool record has been resolved to a dispatchable form.                                                                                                                                                                                                                                |
@@ -122,6 +147,7 @@ Documentation merge entry for Typedoc — combines the stable `@databricks/appki
 | [ResourcePermission](./docs/api/appkit/TypeAlias.ResourcePermission.md)         | Union of all possible permission levels across all resource types.                                                                                                                                                                                                                                                     |
 | [SearchFilters](./docs/api/appkit/TypeAlias.SearchFilters.md)                   | -                                                                                                                                                                                                                                                                                                                      |
 | [ServingFactory](./docs/api/appkit/TypeAlias.ServingFactory.md)                 | Factory function returned by `AppKit.serving`.                                                                                                                                                                                                                                                                         |
+| [Severity](./docs/api/appkit/TypeAlias.Severity.md)                             | Whether an assertion fails the eval (`gate`) or is tracked only (`soft`).                                                                                                                                                                                                                                              |
 | [SupervisorTool](./docs/api/appkit/TypeAlias.SupervisorTool.md)                 | Tools supported by the Databricks AI Gateway Responses API. The shapes match the wire format the endpoint expects, so the adapter passes the array straight into the request body.                                                                                                                                     |
 | [ToolRegistry](./docs/api/appkit/TypeAlias.ToolRegistry.md)                     | -                                                                                                                                                                                                                                                                                                                      |
 | [ToPlugin](./docs/api/appkit/TypeAlias.ToPlugin.md)                             | Factory function type returned by `toPlugin()`. Accepts optional config and returns a PluginData tuple.                                                                                                                                                                                                                |
@@ -148,20 +174,31 @@ Documentation merge entry for Typedoc — combines the stable `@databricks/appki
 | [bigid](./docs/api/appkit/Function.bigid.md)                                           | -                                                                                                                                                                                                                                                                                                                            |
 | [bigint](./docs/api/appkit/Function.bigint.md)                                         | -                                                                                                                                                                                                                                                                                                                            |
 | [boolean](./docs/api/appkit/Function.boolean.md)                                       | -                                                                                                                                                                                                                                                                                                                            |
+| [buildAssessments](./docs/api/appkit/Function.buildAssessments.md)                     | -                                                                                                                                                                                                                                                                                                                            |
+| [configureJudge](./docs/api/appkit/Function.configureJudge.md)                         | Configure the judge once. Sets the OpenAI-compatible client env autoevals reads and the default judge model. No-op-safe: on failure, judging stays disabled and [isJudgeConfigured](./docs/api/appkit/Function.isJudgeConfigured.md) returns false.                                                                    |
 | [createAgent](./docs/api/appkit/Function.createAgent.md)                               | Pure factory for agent definitions: cycle-detects the sub-agent graph and returns the same object, stamped with a non-enumerable AGENT\_BRAND so discovery recognizes it. Safe at module top-level; no adapter is built. Don't `Object.freeze` the definition before passing it in — the brand is written onto the argument. |
 | [createApp](./docs/api/appkit/Function.createApp.md)                                   | Bootstraps AppKit with the provided configuration.                                                                                                                                                                                                                                                                           |
+| [createHttpDriver](./docs/api/appkit/Function.createHttpDriver.md)                     | Drives an agent by POSTing to a running app's chat endpoint and parsing the SSE response. Keeps the thread id across `send`s so multi-turn evals share a conversation. Agent/stream errors surface as `succeeded: false` rather than throwing, so `t.succeeded()` can assert on them.                                        |
 | [createLakebasePool](./docs/api/appkit/Function.createLakebasePool.md)                 | Create a Lakebase pool with appkit's logger integration. Telemetry automatically uses appkit's OpenTelemetry configuration via global registry.                                                                                                                                                                              |
 | [createLakebasePoolManager](./docs/api/appkit/Function.createLakebasePoolManager.md)   | Create a pool manager that maintains per-key Lakebase connection pools.                                                                                                                                                                                                                                                      |
 | [createWorkspaceClient](./docs/api/appkit/Function.createWorkspaceClient.md)           | Construct an AppKit workspace client.                                                                                                                                                                                                                                                                                        |
 | [database](./docs/api/appkit/Function.database.md)                                     | Create a typed database plugin registration for a finalized schema.                                                                                                                                                                                                                                                          |
+| [defineEval](./docs/api/appkit/Function.defineEval.md)                                 | Define an agent eval. Default-export the result from a `server/agents/<id>/evals/*.eval.ts` file.                                                                                                                                                                                                                            |
 | [defineManifest](./docs/api/appkit/Function.defineManifest.md)                         | Validates a raw manifest (typically a `manifest.json` import) against the canonical Zod schema and returns it as a strict [PluginManifest](./docs/api/appkit/Interface.PluginManifest.md).                                                                                                                             |
 | [defineSchema](./docs/api/appkit/Function.defineSchema.md)                             | Compile one declared schema. The returned type keeps the table names the builder returned, so `crudRoutes` and `hooks` can name only real tables.                                                                                                                                                                            |
 | [defineTool](./docs/api/appkit/Function.defineTool.md)                                 | Defines a single tool entry for a plugin's internal registry.                                                                                                                                                                                                                                                                |
+| [discoverEvalFiles](./docs/api/appkit/Function.discoverEvalFiles.md)                   | Discover evals under `<rootDir>/server/agents/<agent>/evals/` — co-located with each agent's `agent.{md,ts}` (same folder-per-agent layout the agents plugin discovers). The agent id is the folder name; the eval id is the file path relative to that evals dir with `.eval.ts` stripped. Sorted + stable.                 |
 | [enumColumn](./docs/api/appkit/Function.enumColumn.md)                                 | -                                                                                                                                                                                                                                                                                                                            |
+| [equals](./docs/api/appkit/Function.equals.md)                                         | Passes when the value equals `expected` exactly.                                                                                                                                                                                                                                                                             |
+| [evalGlyph](./docs/api/appkit/Function.evalGlyph.md)                                   | Status glyph for a single eval result.                                                                                                                                                                                                                                                                                       |
 | [executeFromRegistry](./docs/api/appkit/Function.executeFromRegistry.md)               | Validates tool-call arguments against the entry's schema and invokes its handler. On validation failure, returns an LLM-friendly error string (matching the behavior of `tool()`) rather than throwing, so the model can self-correct on its next turn.                                                                      |
 | [extractServingEndpoints](./docs/api/appkit/Function.extractServingEndpoints.md)       | Extract serving endpoint config from a server file by AST-parsing it. Looks for `serving({ endpoints: { alias: { env: "..." }, ... } })` calls and extracts the endpoint alias names and their environment variable mappings.                                                                                                |
 | [findServerFile](./docs/api/appkit/Function.findServerFile.md)                         | Find the server entry file by checking candidate paths in order.                                                                                                                                                                                                                                                             |
 | [fk](./docs/api/appkit/Function.fk.md)                                                 | Declare foreign-key to another column.                                                                                                                                                                                                                                                                                       |
+| [formatEvalDetail](./docs/api/appkit/Function.formatEvalDetail.md)                     | Indented detail lines for a failing eval (error + failing assertions).                                                                                                                                                                                                                                                       |
+| [formatEvalHeadline](./docs/api/appkit/Function.formatEvalHeadline.md)                 | The one-line header for a single eval result (no failure detail).                                                                                                                                                                                                                                                            |
+| [formatEvalResults](./docs/api/appkit/Function.formatEvalResults.md)                   | Render all results as a human-readable console report (non-streaming).                                                                                                                                                                                                                                                       |
+| [formatSummaryLine](./docs/api/appkit/Function.formatSummaryLine.md)                   | The final PASS/FAIL summary line.                                                                                                                                                                                                                                                                                            |
 | [fromSupervisorApi](./docs/api/appkit/Function.fromSupervisorApi.md)                   | Creates an [AgentAdapter](./docs/api/appkit/Interface.AgentAdapter.md) backed by the Databricks AI Gateway Responses API (`/ai-gateway/mlflow/v1/responses`).                                                                                                                                                          |
 | [functionToolToDefinition](./docs/api/appkit/Function.functionToolToDefinition.md)     | -                                                                                                                                                                                                                                                                                                                            |
 | [generateDatabaseCredential](./docs/api/appkit/Function.generateDatabaseCredential.md) | Generate OAuth credentials for Postgres database connection using the proper Postgres API.                                                                                                                                                                                                                                   |
@@ -173,19 +210,28 @@ Documentation merge entry for Typedoc — combines the stable `@databricks/appki
 | [getUsernameWithApiLookup](./docs/api/appkit/Function.getUsernameWithApiLookup.md)     | Resolves the PostgreSQL username for a Lakebase connection.                                                                                                                                                                                                                                                                  |
 | [getWorkspaceClient](./docs/api/appkit/Function.getWorkspaceClient.md)                 | Get workspace client from config or SDK default auth chain                                                                                                                                                                                                                                                                   |
 | [id](./docs/api/appkit/Function.id.md)                                                 | -                                                                                                                                                                                                                                                                                                                            |
+| [includes](./docs/api/appkit/Function.includes.md)                                     | Passes when the value contains `substring`.                                                                                                                                                                                                                                                                                  |
 | [integer](./docs/api/appkit/Function.integer.md)                                       | -                                                                                                                                                                                                                                                                                                                            |
 | [isFunctionTool](./docs/api/appkit/Function.isFunctionTool.md)                         | -                                                                                                                                                                                                                                                                                                                            |
 | [isHostedTool](./docs/api/appkit/Function.isHostedTool.md)                             | -                                                                                                                                                                                                                                                                                                                            |
+| [isJudgeConfigured](./docs/api/appkit/Function.isJudgeConfigured.md)                   | -                                                                                                                                                                                                                                                                                                                            |
 | [isSQLTypeMarker](./docs/api/appkit/Function.isSQLTypeMarker.md)                       | Type guard to check if a value is a SQL type marker                                                                                                                                                                                                                                                                          |
 | [isSupervisorTool](./docs/api/appkit/Function.isSupervisorTool.md)                     | Type guard for [HostedSupervisorTool](./docs/api/appkit/Interface.HostedSupervisorTool.md). Used by the agents plugin (`buildToolIndex`) and standalone `runAgent` (`classifyTool`) to route supervisor-hosted tools to the extensions payload rather than the adapter's `tools` array.                                |
 | [isToolkitEntry](./docs/api/appkit/Function.isToolkitEntry.md)                         | Type guard for `ToolkitEntry` — used by the agents plugin to differentiate toolkit references from inline tools in a mixed `tools` record.                                                                                                                                                                                   |
 | [jsonb](./docs/api/appkit/Function.jsonb.md)                                           | -                                                                                                                                                                                                                                                                                                                            |
 | [loadAgentFromFile](./docs/api/appkit/Function.loadAgentFromFile.md)                   | Loads a single markdown agent file and resolves its frontmatter against registered plugin toolkits + ambient tool library.                                                                                                                                                                                                   |
 | [loadAgentsFromDir](./docs/api/appkit/Function.loadAgentsFromDir.md)                   | Scans a directory for one subdirectory per agent, each containing `agent.md` (frontmatter + body). Produces an `AgentDefinition` record keyed by agent id (folder name). Throws on frontmatter errors or unresolved references. Returns an empty map if the directory does not exist.                                        |
+| [matches](./docs/api/appkit/Function.matches.md)                                       | Passes when the value matches `pattern`.                                                                                                                                                                                                                                                                                     |
 | [mcpServer](./docs/api/appkit/Function.mcpServer.md)                                   | Factory for declaring a custom MCP server tool.                                                                                                                                                                                                                                                                              |
+| [normalizeHost](./docs/api/appkit/Function.normalizeHost.md)                           | Ensure the host has a scheme (Databricks env often lacks `https://`).                                                                                                                                                                                                                                                        |
 | [parseTextToolCalls](./docs/api/appkit/Function.parseTextToolCalls.md)                 | Parses text-based tool calls from model output.                                                                                                                                                                                                                                                                              |
+| [reportToMlflow](./docs/api/appkit/Function.reportToMlflow.md)                         | Write one pass/fail assessment per eval result to the Databricks MLflow REST API. Never throws — failures are collected so the run still reports.                                                                                                                                                                            |
+| [resolveDatabricksAuth](./docs/api/appkit/Function.resolveDatabricksAuth.md)           | -                                                                                                                                                                                                                                                                                                                            |
 | [resolveHostedTools](./docs/api/appkit/Function.resolveHostedTools.md)                 | -                                                                                                                                                                                                                                                                                                                            |
 | [runAgent](./docs/api/appkit/Function.runAgent.md)                                     | Standalone agent execution without `createApp`. Resolves the adapter, binds inline tools, and drives the adapter's `run()` loop to completion.                                                                                                                                                                               |
+| [runEval](./docs/api/appkit/Function.runEval.md)                                       | Run a single eval against a driver. Never throws for assertion or agent failures — those become a non-passing [EvalResult](./docs/api/appkit/Interface.EvalResult.md). Only a malformed eval definition surfaces as `result.error`.                                                                                    |
+| [runEvalsInDir](./docs/api/appkit/Function.runEvalsInDir.md)                           | Discover, load, and run every eval under each agent's `evals/` dir, driving the agents on a running app. Never throws for an individual eval — load/run failures become non-passing [EvalResult](./docs/api/appkit/Interface.EvalResult.md)s.                                                                          |
+| [summarize](./docs/api/appkit/Function.summarize.md)                                   | -                                                                                                                                                                                                                                                                                                                            |
 | [text](./docs/api/appkit/Function.text.md)                                             | -                                                                                                                                                                                                                                                                                                                            |
 | [timestamp](./docs/api/appkit/Function.timestamp.md)                                   | -                                                                                                                                                                                                                                                                                                                            |
 | [tool](./docs/api/appkit/Function.tool.md)                                             | Factory for defining function tools with Zod schemas.                                                                                                                                                                                                                                                                        |