@databricks/appkit 0.71.0 → 0.72.0

package/docs/api/appkit/Function.evalGlyph.md

@@ -0,0 +1,18 @@
+# Function: evalGlyph()
+
+```ts
+function evalGlyph(result: EvalResult): string;
+
+```
+
+Status glyph for a single eval result.
+
+## Parameters[​](#parameters "Direct link to Parameters")
+
+| Parameter | Type                                                            |
+| --------- | --------------------------------------------------------------- |
+| `result`  | [`EvalResult`](./docs/api/appkit/Interface.EvalResult.md) |
+
+## Returns[​](#returns "Direct link to Returns")
+
+`string`

package/docs/api/appkit/Function.formatEvalDetail.md

@@ -0,0 +1,18 @@
+# Function: formatEvalDetail()
+
+```ts
+function formatEvalDetail(result: EvalResult): string[];
+
+```
+
+Indented detail lines for a failing eval (error + failing assertions).
+
+## Parameters[​](#parameters "Direct link to Parameters")
+
+| Parameter | Type                                                            |
+| --------- | --------------------------------------------------------------- |
+| `result`  | [`EvalResult`](./docs/api/appkit/Interface.EvalResult.md) |
+
+## Returns[​](#returns "Direct link to Returns")
+
+`string`\[]

package/docs/api/appkit/Function.formatEvalHeadline.md

@@ -0,0 +1,18 @@
+# Function: formatEvalHeadline()
+
+```ts
+function formatEvalHeadline(result: EvalResult): string;
+
+```
+
+The one-line header for a single eval result (no failure detail).
+
+## Parameters[​](#parameters "Direct link to Parameters")
+
+| Parameter | Type                                                            |
+| --------- | --------------------------------------------------------------- |
+| `result`  | [`EvalResult`](./docs/api/appkit/Interface.EvalResult.md) |
+
+## Returns[​](#returns "Direct link to Returns")
+
+`string`

package/docs/api/appkit/Function.formatEvalResults.md

@@ -0,0 +1,18 @@
+# Function: formatEvalResults()
+
+```ts
+function formatEvalResults(results: EvalResult[]): string;
+
+```
+
+Render all results as a human-readable console report (non-streaming).
+
+## Parameters[​](#parameters "Direct link to Parameters")
+
+| Parameter | Type                                                               |
+| --------- | ------------------------------------------------------------------ |
+| `results` | [`EvalResult`](./docs/api/appkit/Interface.EvalResult.md)\[] |
+
+## Returns[​](#returns "Direct link to Returns")
+
+`string`

package/docs/api/appkit/Function.formatSummaryLine.md

@@ -0,0 +1,18 @@
+# Function: formatSummaryLine()
+
+```ts
+function formatSummaryLine(results: EvalResult[]): string;
+
+```
+
+The final PASS/FAIL summary line.
+
+## Parameters[​](#parameters "Direct link to Parameters")
+
+| Parameter | Type                                                               |
+| --------- | ------------------------------------------------------------------ |
+| `results` | [`EvalResult`](./docs/api/appkit/Interface.EvalResult.md)\[] |
+
+## Returns[​](#returns "Direct link to Returns")
+
+`string`

package/docs/api/appkit/Function.includes.md

@@ -0,0 +1,18 @@
+# Function: includes()
+
+```ts
+function includes(substring: string): Matcher;
+
+```
+
+Passes when the value contains `substring`.
+
+## Parameters[​](#parameters "Direct link to Parameters")
+
+| Parameter   | Type     |
+| ----------- | -------- |
+| `substring` | `string` |
+
+## Returns[​](#returns "Direct link to Returns")
+
+[`Matcher`](./docs/api/appkit/TypeAlias.Matcher.md)

package/docs/api/appkit/Function.isJudgeConfigured.md

@@ -0,0 +1,10 @@
+# Function: isJudgeConfigured()
+
+```ts
+function isJudgeConfigured(): boolean;
+
+```
+
+## Returns[​](#returns "Direct link to Returns")
+
+`boolean`

package/docs/api/appkit/Function.matches.md

@@ -0,0 +1,18 @@
+# Function: matches()
+
+```ts
+function matches(pattern: RegExp): Matcher;
+
+```
+
+Passes when the value matches `pattern`.
+
+## Parameters[​](#parameters "Direct link to Parameters")
+
+| Parameter | Type     |
+| --------- | -------- |
+| `pattern` | `RegExp` |
+
+## Returns[​](#returns "Direct link to Returns")
+
+[`Matcher`](./docs/api/appkit/TypeAlias.Matcher.md)

package/docs/api/appkit/Function.normalizeHost.md

@@ -0,0 +1,18 @@
+# Function: normalizeHost()
+
+```ts
+function normalizeHost(raw: string): string;
+
+```
+
+Ensure the host has a scheme (Databricks env often lacks `https://`).
+
+## Parameters[​](#parameters "Direct link to Parameters")
+
+| Parameter | Type     |
+| --------- | -------- |
+| `raw`     | `string` |
+
+## Returns[​](#returns "Direct link to Returns")
+
+`string`

package/docs/api/appkit/Function.reportToMlflow.md

@@ -0,0 +1,23 @@
+# Function: reportToMlflow()
+
+```ts
+function reportToMlflow(
+   client: MlflowClient, 
+   results: EvalResult[], 
+sqlWarehouseId?: string): Promise<ReportOutcome>;
+
+```
+
+Write one pass/fail assessment per eval result to the Databricks MLflow REST API. Never throws — failures are collected so the run still reports.
+
+## Parameters[​](#parameters "Direct link to Parameters")
+
+| Parameter         | Type                                                               |
+| ----------------- | ------------------------------------------------------------------ |
+| `client`          | [`MlflowClient`](./docs/api/appkit/Class.MlflowClient.md)    |
+| `results`         | [`EvalResult`](./docs/api/appkit/Interface.EvalResult.md)\[] |
+| `sqlWarehouseId?` | `string`                                                           |
+
+## Returns[​](#returns "Direct link to Returns")
+
+`Promise`<[`ReportOutcome`](./docs/api/appkit/Interface.ReportOutcome.md)>

package/docs/api/appkit/Function.resolveDatabricksAuth.md

@@ -0,0 +1,16 @@
+# Function: resolveDatabricksAuth()
+
+```ts
+function resolveDatabricksAuth(options: ResolveDatabricksAuthOptions): Promise<DatabricksAuth | undefined>;
+
+```
+
+## Parameters[​](#parameters "Direct link to Parameters")
+
+| Parameter | Type                                                                                                |
+| --------- | --------------------------------------------------------------------------------------------------- |
+| `options` | [`ResolveDatabricksAuthOptions`](./docs/api/appkit/Interface.ResolveDatabricksAuthOptions.md) |
+
+## Returns[​](#returns "Direct link to Returns")
+
+`Promise`<[`DatabricksAuth`](./docs/api/appkit/Interface.DatabricksAuth.md) | `undefined`>

package/docs/api/appkit/Function.runEval.md

@@ -0,0 +1,19 @@
+# Function: runEval()
+
+```ts
+function runEval(def: EvalDefinition, options: RunEvalOptions): Promise<EvalResult>;
+
+```
+
+Run a single eval against a driver. Never throws for assertion or agent failures — those become a non-passing [EvalResult](./docs/api/appkit/Interface.EvalResult.md). Only a malformed eval definition surfaces as `result.error`.
+
+## Parameters[​](#parameters "Direct link to Parameters")
+
+| Parameter | Type                                                                    |
+| --------- | ----------------------------------------------------------------------- |
+| `def`     | [`EvalDefinition`](./docs/api/appkit/Interface.EvalDefinition.md) |
+| `options` | [`RunEvalOptions`](./docs/api/appkit/Interface.RunEvalOptions.md) |
+
+## Returns[​](#returns "Direct link to Returns")
+
+`Promise`<[`EvalResult`](./docs/api/appkit/Interface.EvalResult.md)>

package/docs/api/appkit/Function.runEvalsInDir.md

@@ -0,0 +1,18 @@
+# Function: runEvalsInDir()
+
+```ts
+function runEvalsInDir(options: RunEvalsOptions): Promise<EvalRunSummary>;
+
+```
+
+Discover, load, and run every eval under each agent's `evals/` dir, driving the agents on a running app. Never throws for an individual eval — load/run failures become non-passing [EvalResult](./docs/api/appkit/Interface.EvalResult.md)s.
+
+## Parameters[​](#parameters "Direct link to Parameters")
+
+| Parameter | Type                                                                      |
+| --------- | ------------------------------------------------------------------------- |
+| `options` | [`RunEvalsOptions`](./docs/api/appkit/Interface.RunEvalsOptions.md) |
+
+## Returns[​](#returns "Direct link to Returns")
+
+`Promise`<[`EvalRunSummary`](./docs/api/appkit/Interface.EvalRunSummary.md)>

package/docs/api/appkit/Function.summarize.md

@@ -0,0 +1,16 @@
+# Function: summarize()
+
+```ts
+function summarize(results: EvalResult[]): EvalSummary;
+
+```
+
+## Parameters[​](#parameters "Direct link to Parameters")
+
+| Parameter | Type                                                               |
+| --------- | ------------------------------------------------------------------ |
+| `results` | [`EvalResult`](./docs/api/appkit/Interface.EvalResult.md)\[] |
+
+## Returns[​](#returns "Direct link to Returns")
+
+[`EvalSummary`](./docs/api/appkit/Interface.EvalSummary.md)

package/docs/api/appkit/Interface.AssertionHandle.md

@@ -0,0 +1,54 @@
+# Interface: AssertionHandle
+
+Chainable handle returned by every assertion to control its severity. Mirrors eve: assertions are gates by default; `.soft()` demotes to a tracked metric; `.atLeast(n)` is a soft, score-thresholded assertion.
+
+## Methods[​](#methods "Direct link to Methods")
+
+### atLeast()[​](#atleast "Direct link to atLeast()")
+
+```ts
+atLeast(threshold: number): AssertionHandle;
+
+```
+
+Soft assertion that passes only when the score is at least `threshold`.
+
+#### Parameters[​](#parameters "Direct link to Parameters")
+
+| Parameter   | Type     |
+| ----------- | -------- |
+| `threshold` | `number` |
+
+#### Returns[​](#returns "Direct link to Returns")
+
+`AssertionHandle`
+
+***
+
+### gate()[​](#gate "Direct link to gate()")
+
+```ts
+gate(): AssertionHandle;
+
+```
+
+Promote to a hard gate — failure fails the eval (non-zero exit).
+
+#### Returns[​](#returns-1 "Direct link to Returns")
+
+`AssertionHandle`
+
+***
+
+### soft()[​](#soft "Direct link to soft()")
+
+```ts
+soft(): AssertionHandle;
+
+```
+
+Demote to a tracked metric — doesn't fail unless running with `strict`.
+
+#### Returns[​](#returns-2 "Direct link to Returns")
+
+`AssertionHandle`

package/docs/api/appkit/Interface.AssertionResult.md

@@ -0,0 +1,48 @@
+# Interface: AssertionResult
+
+A single recorded assertion outcome.
+
+## Properties[​](#properties "Direct link to Properties")
+
+### detail?[​](#detail "Direct link to detail?")
+
+```ts
+optional detail: string;
+
+```
+
+***
+
+### label[​](#label "Direct link to label")
+
+```ts
+label: string;
+
+```
+
+***
+
+### pass[​](#pass "Direct link to pass")
+
+```ts
+pass: boolean;
+
+```
+
+***
+
+### score?[​](#score "Direct link to score?")
+
+```ts
+optional score: number;
+
+```
+
+***
+
+### severity[​](#severity "Direct link to severity")
+
+```ts
+severity: Severity;
+
+```

package/docs/api/appkit/Interface.Assessment.md

@@ -0,0 +1,83 @@
+# Interface: Assessment
+
+A Feedback assessment in the MLflow REST proto-JSON shape.
+
+## Properties[​](#properties "Direct link to Properties")
+
+### assessment\_name[​](#assessment_name "Direct link to assessment_name")
+
+```ts
+assessment_name: string;
+
+```
+
+***
+
+### feedback[​](#feedback "Direct link to feedback")
+
+```ts
+feedback: {
+  value: unknown;
+};
+
+```
+
+#### value[​](#value "Direct link to value")
+
+```ts
+value: unknown;
+
+```
+
+***
+
+### metadata?[​](#metadata "Direct link to metadata?")
+
+```ts
+optional metadata: Record<string, string>;
+
+```
+
+***
+
+### rationale?[​](#rationale "Direct link to rationale?")
+
+```ts
+optional rationale: string;
+
+```
+
+***
+
+### source[​](#source "Direct link to source")
+
+```ts
+source: {
+  source_id: string;
+  source_type: "CODE" | "HUMAN" | "LLM_JUDGE";
+};
+
+```
+
+#### source\_id[​](#source_id "Direct link to source_id")
+
+```ts
+source_id: string;
+
+```
+
+#### source\_type[​](#source_type "Direct link to source_type")
+
+```ts
+source_type: "CODE" | "HUMAN" | "LLM_JUDGE";
+
+```
+
+***
+
+### trace\_id[​](#trace_id "Direct link to trace_id")
+
+```ts
+trace_id: string;
+
+```

package/docs/api/appkit/Interface.CustomJudgeSpec.md

@@ -0,0 +1,30 @@
+# Interface: CustomJudgeSpec
+
+A custom LLM-judge definition: a prompt template and choice→score mapping.
+
+## Properties[​](#properties "Direct link to Properties")
+
+### choiceScores[​](#choicescores "Direct link to choiceScores")
+
+```ts
+choiceScores: Record<string, number>;
+
+```
+
+***
+
+### name[​](#name "Direct link to name")
+
+```ts
+name: string;
+
+```
+
+***
+
+### promptTemplate[​](#prompttemplate "Direct link to promptTemplate")
+
+```ts
+promptTemplate: string;
+
+```

package/docs/api/appkit/Interface.DatabricksAuth.md

@@ -0,0 +1,21 @@
+# Interface: DatabricksAuth
+
+Resolved Databricks host + bearer token for the eval runner's REST calls.
+
+## Properties[​](#properties "Direct link to Properties")
+
+### host[​](#host "Direct link to host")
+
+```ts
+host: string;
+
+```
+
+***
+
+### token[​](#token "Direct link to token")
+
+```ts
+token: string;
+
+```

package/docs/api/appkit/Interface.DiscoveredEval.md

@@ -0,0 +1,36 @@
+# Interface: DiscoveredEval
+
+An eval file found under `server/agents/<agent>/evals/`.
+
+## Properties[​](#properties "Direct link to Properties")
+
+### agent[​](#agent "Direct link to agent")
+
+```ts
+agent: string;
+
+```
+
+The agent id (the `server/agents/<agent>` directory name).
+
+***
+
+### file[​](#file "Direct link to file")
+
+```ts
+file: string;
+
+```
+
+Absolute path to the `*.eval.ts` file.
+
+***
+
+### id[​](#id "Direct link to id")
+
+```ts
+id: string;
+
+```
+
+Id relative to the agent's evals dir, without `.eval.ts` (e.g. `weather/basic`).

package/docs/api/appkit/Interface.DriveResult.md

@@ -0,0 +1,58 @@
+# Interface: DriveResult
+
+What a driver returns for a single `t.send`.
+
+## Properties[​](#properties "Direct link to Properties")
+
+### reply[​](#reply "Direct link to reply")
+
+```ts
+reply: string;
+
+```
+
+The final assistant message text.
+
+***
+
+### sessionId?[​](#sessionid "Direct link to sessionId?")
+
+```ts
+optional sessionId: string;
+
+```
+
+Thread/session id, when the driver exposes one.
+
+***
+
+### succeeded[​](#succeeded "Direct link to succeeded")
+
+```ts
+succeeded: boolean;
+
+```
+
+Whether the turn completed without an agent/stream error.
+
+***
+
+### toolCalls[​](#toolcalls "Direct link to toolCalls")
+
+```ts
+toolCalls: string[];
+
+```
+
+Names of tools the agent called during the turn.
+
+***
+
+### traceId?[​](#traceid "Direct link to traceId?")
+
+```ts
+optional traceId: string;
+
+```
+
+MLflow trace id for the turn, when tracing is enabled on the app.

package/docs/api/appkit/Interface.EvalDefinition.md

@@ -0,0 +1,46 @@
+# Interface: EvalDefinition
+
+A single eval, default-exported from a `*.eval.ts` file.
+
+## Properties[​](#properties "Direct link to Properties")
+
+### agent?[​](#agent "Direct link to agent?")
+
+```ts
+optional agent: string;
+
+```
+
+Target agent id. Defaults to the eval's parent `server/agents/<id>` dir.
+
+***
+
+### description?[​](#description "Direct link to description?")
+
+```ts
+optional description: string;
+
+```
+
+Short human description, shown in reports.
+
+## Methods[​](#methods "Direct link to Methods")
+
+### test()[​](#test "Direct link to test()")
+
+```ts
+test(t: TestContext): void | Promise<void>;
+
+```
+
+The eval body: drive the agent and assert on its behavior.
+
+#### Parameters[​](#parameters "Direct link to Parameters")
+
+| Parameter | Type                                                              |
+| --------- | ----------------------------------------------------------------- |
+| `t`       | [`TestContext`](./docs/api/appkit/Interface.TestContext.md) |
+
+#### Returns[​](#returns "Direct link to Returns")
+
+`void` | `Promise`<`void`>

package/docs/api/appkit/Interface.EvalDriver.md

@@ -0,0 +1,22 @@
+# Interface: EvalDriver
+
+Abstraction over how the agent is driven. The HTTP driver posts to a running app's agents endpoint; future drivers (in-process) implement the same shape.
+
+## Methods[​](#methods "Direct link to Methods")
+
+### send()[​](#send "Direct link to send()")
+
+```ts
+send(message: string): Promise<DriveResult>;
+
+```
+
+#### Parameters[​](#parameters "Direct link to Parameters")
+
+| Parameter | Type     |
+| --------- | -------- |
+| `message` | `string` |
+
+#### Returns[​](#returns "Direct link to Returns")
+
+`Promise`<[`DriveResult`](./docs/api/appkit/Interface.DriveResult.md)>