@databricks/appkit 0.70.0 → 0.72.0

package/docs/api/appkit/Interface.EvalResult.md

@@ -0,0 +1,83 @@
+# Interface: EvalResult
+
+The outcome of running one eval.
+
+## Properties[​](#properties "Direct link to Properties")
+
+### assertions[​](#assertions "Direct link to assertions")
+
+```ts
+assertions: AssertionResult[];
+
+```
+
+***
+
+### description?[​](#description "Direct link to description?")
+
+```ts
+optional description: string;
+
+```
+
+***
+
+### error?[​](#error "Direct link to error?")
+
+```ts
+optional error: string;
+
+```
+
+Set when the eval threw before completing.
+
+***
+
+### id[​](#id "Direct link to id")
+
+```ts
+id: string;
+
+```
+
+***
+
+### passed[​](#passed "Direct link to passed")
+
+```ts
+passed: boolean;
+
+```
+
+True when all gates passed (and, under strict, all soft assertions too).
+
+***
+
+### skipped?[​](#skipped "Direct link to skipped?")
+
+```ts
+optional skipped: {
+  reason?: string;
+};
+
+```
+
+Set when the eval called `t.skip`.
+
+#### reason?[​](#reason "Direct link to reason?")
+
+```ts
+optional reason: string;
+
+```
+
+***
+
+### traceId?[​](#traceid "Direct link to traceId?")
+
+```ts
+optional traceId: string;
+
+```
+
+MLflow trace id of the eval's last turn, for attaching assessments.

package/docs/api/appkit/Interface.EvalRunSummary.md

@@ -0,0 +1,46 @@
+# Interface: EvalRunSummary
+
+## Properties[​](#properties "Direct link to Properties")
+
+### mlflow?[​](#mlflow "Direct link to mlflow?")
+
+```ts
+optional mlflow: {
+  finish: FinishOutcome;
+  report: ReportOutcome;
+  runId: string;
+};
+
+```
+
+Present when an MLflow evaluation run was created.
+
+#### finish[​](#finish "Direct link to finish")
+
+```ts
+finish: FinishOutcome;
+
+```
+
+#### report[​](#report "Direct link to report")
+
+```ts
+report: ReportOutcome;
+
+```
+
+#### runId[​](#runid "Direct link to runId")
+
+```ts
+runId: string;
+
+```
+
+***
+
+### results[​](#results "Direct link to results")
+
+```ts
+results: EvalResult[];
+
+```

package/docs/api/appkit/Interface.EvalSummary.md

@@ -0,0 +1,48 @@
+# Interface: EvalSummary
+
+## Properties[​](#properties "Direct link to Properties")
+
+### allPassed[​](#allpassed "Direct link to allPassed")
+
+```ts
+allPassed: boolean;
+
+```
+
+True when no eval failed (skips don't count as failures).
+
+***
+
+### failed[​](#failed "Direct link to failed")
+
+```ts
+failed: number;
+
+```
+
+***
+
+### passed[​](#passed "Direct link to passed")
+
+```ts
+passed: number;
+
+```
+
+***
+
+### skipped[​](#skipped "Direct link to skipped")
+
+```ts
+skipped: number;
+
+```
+
+***
+
+### total[​](#total "Direct link to total")
+
+```ts
+total: number;
+
+```

package/docs/api/appkit/Interface.HttpDriverOptions.md

@@ -0,0 +1,67 @@
+# Interface: HttpDriverOptions
+
+## Properties[​](#properties "Direct link to Properties")
+
+### agent?[​](#agent "Direct link to agent?")
+
+```ts
+optional agent: string;
+
+```
+
+Agent alias to target. Omit to use the app's default agent.
+
+***
+
+### baseUrl[​](#baseurl "Direct link to baseUrl")
+
+```ts
+baseUrl: string;
+
+```
+
+Base URL of the running app, e.g. `http://localhost:3000`.
+
+***
+
+### headers?[​](#headers "Direct link to headers?")
+
+```ts
+optional headers: Record<string, string>;
+
+```
+
+Extra request headers (e.g. auth for a deployed app).
+
+***
+
+### mlflowRunId?[​](#mlflowrunid "Direct link to mlflowRunId?")
+
+```ts
+optional mlflowRunId: string;
+
+```
+
+MLflow run id to link each turn's trace to (for evaluation runs).
+
+***
+
+### path?[​](#path "Direct link to path?")
+
+```ts
+optional path: string;
+
+```
+
+Chat endpoint path. Defaults to `/api/agents/chat`.
+
+***
+
+### timeoutMs?[​](#timeoutms "Direct link to timeoutMs?")
+
+```ts
+optional timeoutMs: number;
+
+```
+
+Max wall-clock time for a single turn before it is abandoned as a failed turn (`succeeded: false`). Without this a hung agent — a blocked tool, a stalled model — never ends the SSE stream (heartbeats keep it alive), so the read loop spins forever and wedges the whole sequential suite. Defaults to 120s.

package/docs/api/appkit/Interface.JudgeConfig.md

@@ -0,0 +1,34 @@
+# Interface: JudgeConfig
+
+## Properties[​](#properties "Direct link to Properties")
+
+### client[​](#client "Direct link to client")
+
+```ts
+client: MlflowClient;
+
+```
+
+Client for the workspace hosting the judge serving endpoint.
+
+***
+
+### model[​](#model "Direct link to model")
+
+```ts
+model: string;
+
+```
+
+Serving endpoint name used as the judge model.
+
+***
+
+### token[​](#token "Direct link to token")
+
+```ts
+token: string;
+
+```
+
+Bearer token for the serving endpoint.

package/docs/api/appkit/Interface.JudgeScore.md

@@ -0,0 +1,21 @@
+# Interface: JudgeScore
+
+A normalized judge result. `score` is 0..1.
+
+## Properties[​](#properties "Direct link to Properties")
+
+### rationale?[​](#rationale "Direct link to rationale?")
+
+```ts
+optional rationale: string;
+
+```
+
+***
+
+### score[​](#score "Direct link to score")
+
+```ts
+score: number;
+
+```

package/docs/api/appkit/Interface.MatchResult.md

@@ -0,0 +1,34 @@
+# Interface: MatchResult
+
+Result of a deterministic matcher run against a value.
+
+## Properties[​](#properties "Direct link to Properties")
+
+### detail?[​](#detail "Direct link to detail?")
+
+```ts
+optional detail: string;
+
+```
+
+Human-readable explanation, shown on failure.
+
+***
+
+### pass[​](#pass "Direct link to pass")
+
+```ts
+pass: boolean;
+
+```
+
+***
+
+### score?[​](#score "Direct link to score?")
+
+```ts
+optional score: number;
+
+```
+
+Optional 0..1 score for scored matchers (similarity, judges).

package/docs/api/appkit/Interface.PostResult.md

@@ -0,0 +1,30 @@
+# Interface: PostResult
+
+Structured result for a best-effort POST that must not throw.
+
+## Properties[​](#properties "Direct link to Properties")
+
+### error?[​](#error "Direct link to error?")
+
+```ts
+optional error: string;
+
+```
+
+***
+
+### ok[​](#ok "Direct link to ok")
+
+```ts
+ok: boolean;
+
+```
+
+***
+
+### status?[​](#status "Direct link to status?")
+
+```ts
+optional status: number;
+
+```

package/docs/api/appkit/Interface.ReportOutcome.md

@@ -0,0 +1,53 @@
+# Interface: ReportOutcome
+
+## Properties[​](#properties "Direct link to Properties")
+
+### failures[​](#failures "Direct link to failures")
+
+```ts
+failures: {
+  error?: string;
+  status?: number;
+  traceId: string;
+}[];
+
+```
+
+#### error?[​](#error "Direct link to error?")
+
+```ts
+optional error: string;
+
+```
+
+#### status?[​](#status "Direct link to status?")
+
+```ts
+optional status: number;
+
+```
+
+#### traceId[​](#traceid "Direct link to traceId")
+
+```ts
+traceId: string;
+
+```
+
+***
+
+### skipped[​](#skipped "Direct link to skipped")
+
+```ts
+skipped: number;
+
+```
+
+***
+
+### written[​](#written "Direct link to written")
+
+```ts
+written: number;
+
+```

package/docs/api/appkit/Interface.ResolveDatabricksAuthOptions.md

@@ -0,0 +1,34 @@
+# Interface: ResolveDatabricksAuthOptions
+
+## Properties[​](#properties "Direct link to Properties")
+
+### host?[​](#host "Direct link to host?")
+
+```ts
+optional host: string;
+
+```
+
+Explicit host; wins over the profile/SDK-resolved host when set.
+
+***
+
+### profile?[​](#profile "Direct link to profile?")
+
+```ts
+optional profile: string;
+
+```
+
+`~/.databrickscfg` profile to authenticate with (e.g. `dogfood`).
+
+***
+
+### token?[​](#token "Direct link to token?")
+
+```ts
+optional token: string;
+
+```
+
+Explicit bearer token; when set, no OAuth is minted (PAT/CI path).

package/docs/api/appkit/Interface.RunEvalOptions.md

@@ -0,0 +1,34 @@
+# Interface: RunEvalOptions
+
+## Properties[​](#properties "Direct link to Properties")
+
+### driver[​](#driver "Direct link to driver")
+
+```ts
+driver: EvalDriver;
+
+```
+
+Drives the agent and returns reply/tool-calls/success per `send`.
+
+***
+
+### id[​](#id "Direct link to id")
+
+```ts
+id: string;
+
+```
+
+Stable id for the eval (e.g. its file path relative to the evals dir).
+
+***
+
+### strict?[​](#strict "Direct link to strict?")
+
+```ts
+optional strict: boolean;
+
+```
+
+When true, soft assertion failures also fail the eval.

package/docs/api/appkit/Interface.RunEvalsOptions.md

@@ -0,0 +1,192 @@
+# Interface: RunEvalsOptions
+
+## Properties[​](#properties "Direct link to Properties")
+
+### baseUrl[​](#baseurl "Direct link to baseUrl")
+
+```ts
+baseUrl: string;
+
+```
+
+Base URL of the running app to drive, e.g. `http://localhost:3000`.
+
+***
+
+### concurrency?[​](#concurrency "Direct link to concurrency?")
+
+```ts
+optional concurrency: number;
+
+```
+
+Max evals to drive concurrently. Each eval opens one stream to the app as the same user, so keep this at or below the app's `maxConcurrentStreamsPerUser` (default 5) or the surplus streams hit the 429 guard. Defaults to 4; clamped to `[1, total]`.
+
+***
+
+### filter?[​](#filter "Direct link to filter?")
+
+```ts
+optional filter: string;
+
+```
+
+Substring filter on `<agent>/<id>` (or an exact agent id).
+
+***
+
+### headers?[​](#headers "Direct link to headers?")
+
+```ts
+optional headers: Record<string, string>;
+
+```
+
+Extra request headers for the driver (e.g. auth for a deployed app).
+
+***
+
+### judge?[​](#judge "Direct link to judge?")
+
+```ts
+optional judge: {
+  host: string;
+  model: string;
+  token: string;
+};
+
+```
+
+When set, enable `t.judge.*` LLM-as-judge scoring via autoevals against a Databricks serving endpoint (`model`).
+
+#### host[​](#host "Direct link to host")
+
+```ts
+host: string;
+
+```
+
+#### model[​](#model "Direct link to model")
+
+```ts
+model: string;
+
+```
+
+#### token[​](#token "Direct link to token")
+
+```ts
+token: string;
+
+```
+
+***
+
+### mlflow?[​](#mlflow "Direct link to mlflow?")
+
+```ts
+optional mlflow: {
+  experimentId: string;
+  host: string;
+  sqlWarehouseId?: string;
+  token: string;
+};
+
+```
+
+When set, create a native MLflow "Evaluation run": each eval's trace is linked to the run, pass/fail is written as feedback, and aggregate metrics are logged. Requires Databricks creds + the target experiment.
+
+#### experimentId[​](#experimentid "Direct link to experimentId")
+
+```ts
+experimentId: string;
+
+```
+
+#### host[​](#host-1 "Direct link to host")
+
+```ts
+host: string;
+
+```
+
+#### sqlWarehouseId?[​](#sqlwarehouseid "Direct link to sqlWarehouseId?")
+
+```ts
+optional sqlWarehouseId: string;
+
+```
+
+SQL warehouse id for writing assessments to UC-backed (V4) traces.
+
+#### token[​](#token-1 "Direct link to token")
+
+```ts
+token: string;
+
+```
+
+***
+
+### now?[​](#now "Direct link to now?")
+
+```ts
+optional now: number;
+
+```
+
+Wall-clock timestamp (ms) for run create/finish — pass `Date.now()`.
+
+***
+
+### onEvent()?[​](#onevent "Direct link to onEvent()?")
+
+```ts
+optional onEvent: (event: EvalProgress) => void;
+
+```
+
+Progress callback, invoked as evals are discovered, started, and finished.
+
+#### Parameters[​](#parameters "Direct link to Parameters")
+
+| Parameter | Type                                                                |
+| --------- | ------------------------------------------------------------------- |
+| `event`   | [`EvalProgress`](./docs/api/appkit/TypeAlias.EvalProgress.md) |
+
+#### Returns[​](#returns "Direct link to Returns")
+
+`void`
+
+***
+
+### rootDir?[​](#rootdir "Direct link to rootDir?")
+
+```ts
+optional rootDir: string;
+
+```
+
+Project root containing `server/agents/`. Defaults to `process.cwd()`.
+
+***
+
+### strict?[​](#strict "Direct link to strict?")
+
+```ts
+optional strict: boolean;
+
+```
+
+Soft assertion failures also fail the eval.
+
+***
+
+### timeoutMs?[​](#timeoutms "Direct link to timeoutMs?")
+
+```ts
+optional timeoutMs: number;
+
+```
+
+Per-turn wall-clock timeout (ms) before a turn is failed. Defaults to 120s.