@databricks/appkit 0.71.0 → 0.72.0

package/llms.txt

@@ -71,6 +71,7 @@ npx @databricks/appkit docs <query>
 - [Class: DatabricksAdapter](./docs/api/appkit/Class.DatabricksAdapter.md): Adapter that talks directly to Databricks Model Serving /invocations endpoint.
 - [Class: ExecutionError](./docs/api/appkit/Class.ExecutionError.md): Error thrown when an operation execution fails.
 - [Class: InitializationError](./docs/api/appkit/Class.InitializationError.md): Error thrown when a service or component is not properly initialized.
+- [Class: MlflowClient](./docs/api/appkit/Class.MlflowClient.md): A thin client over the Databricks workspace REST API, owning the host + bearer
 - [Abstract Class: Plugin<TConfig>](./docs/api/appkit/Class.Plugin.md): Base abstract class for creating AppKit plugins.
 - [Class: PolicyDeniedError](./docs/api/appkit/Class.PolicyDeniedError.md): Thrown when a policy denies an action.
 - [Class: ResourceRegistry](./docs/api/appkit/Class.ResourceRegistry.md): Central registry for tracking plugin resource requirements.
@@ -86,20 +87,31 @@ npx @databricks/appkit docs <query>
 - [Function: bigid()](./docs/api/appkit/Function.bigid.md): Returns
 - [Function: bigint()](./docs/api/appkit/Function.bigint.md): Returns
 - [Function: boolean()](./docs/api/appkit/Function.boolean.md): Returns
+- [Function: buildAssessments()](./docs/api/appkit/Function.buildAssessments.md): Parameters
+- [Function: configureJudge()](./docs/api/appkit/Function.configureJudge.md): Configure the judge once. Sets the OpenAI-compatible client env autoevals
 - [Function: createAgent()](./docs/api/appkit/Function.createAgent.md): Pure factory for agent definitions: cycle-detects the sub-agent graph and
 - [Function: createApp()](./docs/api/appkit/Function.createApp.md): Bootstraps AppKit with the provided configuration.
+- [Function: createHttpDriver()](./docs/api/appkit/Function.createHttpDriver.md): Drives an agent by POSTing to a running app's chat endpoint and parsing the
 - [Function: createLakebasePool()](./docs/api/appkit/Function.createLakebasePool.md): Create a Lakebase pool with appkit's logger integration.
 - [Function: createLakebasePoolManager()](./docs/api/appkit/Function.createLakebasePoolManager.md): Create a pool manager that maintains per-key Lakebase connection pools.
 - [Function: createWorkspaceClient()](./docs/api/appkit/Function.createWorkspaceClient.md): Construct an AppKit workspace client.
 - [Function: database()](./docs/api/appkit/Function.database.md): Create a typed database plugin registration for a finalized schema.
+- [Function: defineEval()](./docs/api/appkit/Function.defineEval.md): Define an agent eval. Default-export the result from a
 - [Function: defineManifest()](./docs/api/appkit/Function.defineManifest.md): Validates a raw manifest (typically a manifest.json import) against the
 - [Function: defineSchema()](./docs/api/appkit/Function.defineSchema.md): Compile one declared schema. The returned type keeps the table names the
 - [Function: defineTool()](./docs/api/appkit/Function.defineTool.md): Defines a single tool entry for a plugin's internal registry.
+- [Function: discoverEvalFiles()](./docs/api/appkit/Function.discoverEvalFiles.md): Discover evals under /server/agents//evals/ — co-located
 - [Function: enumColumn()](./docs/api/appkit/Function.enumColumn.md): Parameters
+- [Function: equals()](./docs/api/appkit/Function.equals.md): Passes when the value equals expected exactly.
+- [Function: evalGlyph()](./docs/api/appkit/Function.evalGlyph.md): Status glyph for a single eval result.
 - [Function: executeFromRegistry()](./docs/api/appkit/Function.executeFromRegistry.md): Validates tool-call arguments against the entry's schema and invokes its
 - [Function: extractServingEndpoints()](./docs/api/appkit/Function.extractServingEndpoints.md): Extract serving endpoint config from a server file by AST-parsing it.
 - [Function: findServerFile()](./docs/api/appkit/Function.findServerFile.md): Find the server entry file by checking candidate paths in order.
 - [Function: fk()](./docs/api/appkit/Function.fk.md): Declare foreign-key to another column.
+- [Function: formatEvalDetail()](./docs/api/appkit/Function.formatEvalDetail.md): Indented detail lines for a failing eval (error + failing assertions).
+- [Function: formatEvalHeadline()](./docs/api/appkit/Function.formatEvalHeadline.md): The one-line header for a single eval result (no failure detail).
+- [Function: formatEvalResults()](./docs/api/appkit/Function.formatEvalResults.md): Render all results as a human-readable console report (non-streaming).
+- [Function: formatSummaryLine()](./docs/api/appkit/Function.formatSummaryLine.md): The final PASS/FAIL summary line.
 - [Function: fromSupervisorApi()](./docs/api/appkit/Function.fromSupervisorApi.md): Creates an AgentAdapter backed by the Databricks AI Gateway
 - [Function: functionToolToDefinition()](./docs/api/appkit/Function.functionToolToDefinition.md): Parameters
 - [Function: generateDatabaseCredential()](./docs/api/appkit/Function.generateDatabaseCredential.md): Generate OAuth credentials for Postgres database connection using the proper Postgres API.
@@ -111,19 +123,28 @@ npx @databricks/appkit docs <query>
 - [Function: getUsernameWithApiLookup()](./docs/api/appkit/Function.getUsernameWithApiLookup.md): Resolves the PostgreSQL username for a Lakebase connection.
 - [Function: getWorkspaceClient()](./docs/api/appkit/Function.getWorkspaceClient.md): Get workspace client from config or SDK default auth chain
 - [Function: id()](./docs/api/appkit/Function.id.md): Returns
+- [Function: includes()](./docs/api/appkit/Function.includes.md): Passes when the value contains substring.
 - [Function: integer()](./docs/api/appkit/Function.integer.md): Returns
 - [Function: isFunctionTool()](./docs/api/appkit/Function.isFunctionTool.md): Parameters
 - [Function: isHostedTool()](./docs/api/appkit/Function.isHostedTool.md): Parameters
+- [Function: isJudgeConfigured()](./docs/api/appkit/Function.isJudgeConfigured.md): Returns
 - [Function: isSQLTypeMarker()](./docs/api/appkit/Function.isSQLTypeMarker.md): Type guard to check if a value is a SQL type marker
 - [Function: isSupervisorTool()](./docs/api/appkit/Function.isSupervisorTool.md): Type guard for HostedSupervisorTool. Used by the agents plugin
 - [Function: isToolkitEntry()](./docs/api/appkit/Function.isToolkitEntry.md): Type guard for ToolkitEntry — used by the agents plugin to differentiate
 - [Function: jsonb()](./docs/api/appkit/Function.jsonb.md): Returns
 - [Function: loadAgentFromFile()](./docs/api/appkit/Function.loadAgentFromFile.md): Loads a single markdown agent file and resolves its frontmatter against
 - [Function: loadAgentsFromDir()](./docs/api/appkit/Function.loadAgentsFromDir.md): Scans a directory for one subdirectory per agent, each containing
+- [Function: matches()](./docs/api/appkit/Function.matches.md): Passes when the value matches pattern.
 - [Function: mcpServer()](./docs/api/appkit/Function.mcpServer.md): Factory for declaring a custom MCP server tool.
+- [Function: normalizeHost()](./docs/api/appkit/Function.normalizeHost.md): Ensure the host has a scheme (Databricks env often lacks https://).
 - [Function: parseTextToolCalls()](./docs/api/appkit/Function.parseTextToolCalls.md): Parses text-based tool calls from model output.
+- [Function: reportToMlflow()](./docs/api/appkit/Function.reportToMlflow.md): Write one pass/fail assessment per eval result to the Databricks MLflow REST
+- [Function: resolveDatabricksAuth()](./docs/api/appkit/Function.resolveDatabricksAuth.md): Parameters
 - [Function: resolveHostedTools()](./docs/api/appkit/Function.resolveHostedTools.md): Parameters
 - [Function: runAgent()](./docs/api/appkit/Function.runAgent.md): Standalone agent execution without createApp. Resolves the adapter, binds
+- [Function: runEval()](./docs/api/appkit/Function.runEval.md): Run a single eval against a driver. Never throws for assertion or agent
+- [Function: runEvalsInDir()](./docs/api/appkit/Function.runEvalsInDir.md): Discover, load, and run every eval under each agent's evals/ dir, driving
+- [Function: summarize()](./docs/api/appkit/Function.summarize.md): Parameters
 - [Function: text()](./docs/api/appkit/Function.text.md): Returns
 - [Function: timestamp()](./docs/api/appkit/Function.timestamp.md): Parameters
 - [Function: tool()](./docs/api/appkit/Function.tool.md): Factory for defining function tools with Zod schemas.
@@ -136,18 +157,31 @@ npx @databricks/appkit docs <query>
 - [Interface: AgentRunContext](./docs/api/appkit/Interface.AgentRunContext.md): Properties
 - [Interface: AgentsPluginConfig](./docs/api/appkit/Interface.AgentsPluginConfig.md): Base configuration interface for AppKit plugins
 - [Interface: AgentToolDefinition](./docs/api/appkit/Interface.AgentToolDefinition.md): Properties
+- [Interface: AssertionHandle](./docs/api/appkit/Interface.AssertionHandle.md): Chainable handle returned by every assertion to control its severity.
+- [Interface: AssertionResult](./docs/api/appkit/Interface.AssertionResult.md): A single recorded assertion outcome.
+- [Interface: Assessment](./docs/api/appkit/Interface.Assessment.md): A Feedback assessment in the MLflow REST proto-JSON shape.
 - [Interface: AutoInheritToolsConfig](./docs/api/appkit/Interface.AutoInheritToolsConfig.md): Auto-inherit configuration. When enabled for a given agent origin, agents
 - [Interface: BasePluginConfig](./docs/api/appkit/Interface.BasePluginConfig.md): Base configuration interface for AppKit plugins
 - [Interface: CacheConfig](./docs/api/appkit/Interface.CacheConfig.md): Configuration for the CacheInterceptor. Controls TTL, size limits, storage backend, and probabilistic cleanup.
+- [Interface: CustomJudgeSpec](./docs/api/appkit/Interface.CustomJudgeSpec.md): A custom LLM-judge definition: a prompt template and choice→score mapping.
 - [Interface: DatabaseCredential](./docs/api/appkit/Interface.DatabaseCredential.md): Database credentials with OAuth token for Postgres connection
 - [Interface: DatabaseRegistry](./docs/api/appkit/Interface.DatabaseRegistry.md): CANONICAL augmentation target. Empty by default; the generated database.d.ts
+- [Interface: DatabricksAuth](./docs/api/appkit/Interface.DatabricksAuth.md): Resolved Databricks host + bearer token for the eval runner's REST calls.
+- [Interface: DiscoveredEval](./docs/api/appkit/Interface.DiscoveredEval.md): An eval file found under server/agents//evals/.
+- [Interface: DriveResult](./docs/api/appkit/Interface.DriveResult.md): What a driver returns for a single t.send.
 - [Interface: EndpointConfig](./docs/api/appkit/Interface.EndpointConfig.md): Properties
+- [Interface: EvalDefinition](./docs/api/appkit/Interface.EvalDefinition.md): A single eval, default-exported from a *.eval.ts file.
+- [Interface: EvalDriver](./docs/api/appkit/Interface.EvalDriver.md): Abstraction over how the agent is driven. The HTTP driver posts to a running
+- [Interface: EvalResult](./docs/api/appkit/Interface.EvalResult.md): The outcome of running one eval.
+- [Interface: EvalRunSummary](./docs/api/appkit/Interface.EvalRunSummary.md): Properties
+- [Interface: EvalSummary](./docs/api/appkit/Interface.EvalSummary.md): Properties
 - [Interface: FilePolicyUser](./docs/api/appkit/Interface.FilePolicyUser.md): Minimal user identity passed to the policy function.
 - [Interface: FileResource](./docs/api/appkit/Interface.FileResource.md): Describes the file or directory being acted upon.
 - [Interface: FunctionTool](./docs/api/appkit/Interface.FunctionTool.md): Properties
 - [Interface: GenerateDatabaseCredentialRequest](./docs/api/appkit/Interface.GenerateDatabaseCredentialRequest.md): Request parameters for generating database OAuth credentials
 - [Interface: GenerationParams](./docs/api/appkit/Interface.GenerationParams.md): Optional generation parameters forwarded to the OpenAI-compatible serving
 - [Interface: HostedSupervisorTool](./docs/api/appkit/Interface.HostedSupervisorTool.md): Tagged record returned by every supervisorTools factory. The
+- [Interface: HttpDriverOptions](./docs/api/appkit/Interface.HttpDriverOptions.md): Properties
 - [Interface: IAiSearchConfig](./docs/api/appkit/Interface.IAiSearchConfig.md): Base configuration interface for AppKit plugins
 - [Interface: IJobsConfig](./docs/api/appkit/Interface.IJobsConfig.md): Configuration for the Jobs plugin.
 - [Interface: IndexConfig](./docs/api/appkit/Interface.IndexConfig.md): Properties
@@ -155,22 +189,30 @@ npx @databricks/appkit docs <query>
 - [Interface: JobAPI](./docs/api/appkit/Interface.JobAPI.md): User-facing API for a single configured job.
 - [Interface: JobConfig](./docs/api/appkit/Interface.JobConfig.md): Per-job configuration options.
 - [Interface: JobsConnectorConfig](./docs/api/appkit/Interface.JobsConnectorConfig.md): Properties
+- [Interface: JudgeConfig](./docs/api/appkit/Interface.JudgeConfig.md): Properties
+- [Interface: JudgeScore](./docs/api/appkit/Interface.JudgeScore.md): A normalized judge result. score is 0..1.
 - [Interface: LakebasePool](./docs/api/appkit/Interface.LakebasePool.md): Subset of pg.Pool exposed by the Lakebase plugin.
 - [Interface: LakebasePoolConfig](./docs/api/appkit/Interface.LakebasePoolConfig.md): Configuration for creating a Lakebase connection pool
 - [Interface: LakebasePoolManager](./docs/api/appkit/Interface.LakebasePoolManager.md): Manages multiple Lakebase connection pools keyed by an identifier (e.g. userId).
+- [Interface: MatchResult](./docs/api/appkit/Interface.MatchResult.md): Result of a deterministic matcher run against a value.
 - [Interface: McpConnectAllResult](./docs/api/appkit/Interface.McpConnectAllResult.md): Per-endpoint outcome of AppKitMcpClient.connectAll. Callers (the
 - [Interface: Message](./docs/api/appkit/Interface.Message.md): Properties
 - [Interface: PluginManifest<TName>](./docs/api/appkit/Interface.PluginManifest.md): Plugin manifest that declares metadata and resource requirements.
 - [Interface: PluginToolkitProvider](./docs/api/appkit/Interface.PluginToolkitProvider.md): Minimum shape every entry in the Plugins map must expose. Core
+- [Interface: PostResult](./docs/api/appkit/Interface.PostResult.md): Structured result for a best-effort POST that must not throw.
 - [Interface: PromptContext](./docs/api/appkit/Interface.PromptContext.md): Context passed to baseSystemPrompt callbacks.
 - [Interface: RegisteredAgent](./docs/api/appkit/Interface.RegisteredAgent.md): Properties
+- [Interface: ReportOutcome](./docs/api/appkit/Interface.ReportOutcome.md): Properties
 - [Interface: RequestedClaims](./docs/api/appkit/Interface.RequestedClaims.md): Optional claims for fine-grained Unity Catalog table permissions
 - [Interface: RequestedResource](./docs/api/appkit/Interface.RequestedResource.md): Resource to request permissions for in Unity Catalog
 - [Interface: RerankerConfig](./docs/api/appkit/Interface.RerankerConfig.md): Properties
+- [Interface: ResolveDatabricksAuthOptions](./docs/api/appkit/Interface.ResolveDatabricksAuthOptions.md): Properties
 - [Interface: ResourceEntry](./docs/api/appkit/Interface.ResourceEntry.md): Internal representation of a resource in the registry.
 - [Interface: ResourceRequirement](./docs/api/appkit/Interface.ResourceRequirement.md): Declares a resource requirement for a plugin.
 - [Interface: RunAgentInput](./docs/api/appkit/Interface.RunAgentInput.md): Properties
 - [Interface: RunAgentResult](./docs/api/appkit/Interface.RunAgentResult.md): Properties
+- [Interface: RunEvalOptions](./docs/api/appkit/Interface.RunEvalOptions.md): Properties
+- [Interface: RunEvalsOptions](./docs/api/appkit/Interface.RunEvalsOptions.md): Properties
 - [Interface: Schema<TTableName>](./docs/api/appkit/Interface.Schema.md): One finalized schema. TTableName keeps the declared names in the type, so
 - [Interface: SearchRequest](./docs/api/appkit/Interface.SearchRequest.md): Properties
 - [Interface: SearchResponse<T>](./docs/api/appkit/Interface.SearchResponse.md): Type Parameters
@@ -181,6 +223,7 @@ npx @databricks/appkit docs <query>
 - [Interface: SupervisorApiAdapterOptions](./docs/api/appkit/Interface.SupervisorApiAdapterOptions.md): Properties
 - [Interface: SupervisorExtension](./docs/api/appkit/Interface.SupervisorExtension.md): Shape of the value at AgentInput.extensions[SUPERVISOREXTENSIONKEY].
 - [Interface: TelemetryConfig](./docs/api/appkit/Interface.TelemetryConfig.md): OpenTelemetry configuration for AppKit applications
+- [Interface: TestContext](./docs/api/appkit/Interface.TestContext.md): The t context passed to an eval's test function.
 - [Interface: Thread](./docs/api/appkit/Interface.Thread.md): Properties
 - [Interface: ThreadStore](./docs/api/appkit/Interface.ThreadStore.md): Methods
 - [Interface: ToolAnnotations](./docs/api/appkit/Interface.ToolAnnotations.md): Properties
@@ -200,6 +243,7 @@ npx @databricks/appkit docs <query>
 - [Type Alias: BaseSystemPromptOption](./docs/api/appkit/TypeAlias.BaseSystemPromptOption.md)
 - [Type Alias: ConfigSchema](./docs/api/appkit/TypeAlias.ConfigSchema.md): Configuration schema definition for plugin config.
 - [Type Alias: DatabaseExports](./docs/api/appkit/TypeAlias.DatabaseExports.md): Typed database API published by the plugin.
+- [Type Alias: EvalProgress](./docs/api/appkit/TypeAlias.EvalProgress.md)
 - [Type Alias: ExecutionResult<T>](./docs/api/appkit/TypeAlias.ExecutionResult.md): Discriminated union for plugin execution results.
 - [Type Alias: FileAction](./docs/api/appkit/TypeAlias.FileAction.md): Every action the files plugin can perform.
 - [Type Alias: FilePolicy()](./docs/api/appkit/TypeAlias.FilePolicy.md): A policy function that decides whether user may perform action on
@@ -207,6 +251,7 @@ npx @databricks/appkit docs <query>
 - [Type Alias: IAppRouter](./docs/api/appkit/TypeAlias.IAppRouter.md): Express router type for plugin route registration
 - [Type Alias: IDatabaseConfig<TSchema>](./docs/api/appkit/TypeAlias.IDatabaseConfig.md): Configuration for one schema-bound DatabasePlugin instance.
 - [Type Alias: JobsExport()](./docs/api/appkit/TypeAlias.JobsExport.md): Public API shape of the jobs plugin.
+- [Type Alias: Matcher()](./docs/api/appkit/TypeAlias.Matcher.md): A deterministic matcher: inspects a string value and returns a result.
 - [Type Alias: PluginData<T, U, N>](./docs/api/appkit/TypeAlias.PluginData.md): Tuple of plugin class, config, and name. Created by toPlugin() and passed to createApp().
 - [Type Alias: Plugins](./docs/api/appkit/TypeAlias.Plugins.md): Plugin map passed to the function form of AgentDefinition.tools.
 - [Type Alias: ResolvedToolEntry](./docs/api/appkit/TypeAlias.ResolvedToolEntry.md): Internal tool-index entry after a tool record has been resolved to a dispatchable form.
@@ -214,6 +259,7 @@ npx @databricks/appkit docs <query>
 - [Type Alias: ResourcePermission](./docs/api/appkit/TypeAlias.ResourcePermission.md): Union of all possible permission levels across all resource types.
 - [Type Alias: SearchFilters](./docs/api/appkit/TypeAlias.SearchFilters.md)
 - [Type Alias: ServingFactory](./docs/api/appkit/TypeAlias.ServingFactory.md): Factory function returned by AppKit.serving.
+- [Type Alias: Severity](./docs/api/appkit/TypeAlias.Severity.md): Whether an assertion fails the eval (gate) or is tracked only (soft).
 - [Type Alias: SupervisorTool](./docs/api/appkit/TypeAlias.SupervisorTool.md): Tools supported by the Databricks AI Gateway Responses API. The shapes match
 - [Type Alias: ToolRegistry](./docs/api/appkit/TypeAlias.ToolRegistry.md)
 - [Type Alias: ToPlugin()<T, U, N>](./docs/api/appkit/TypeAlias.ToPlugin.md): Factory function type returned by toPlugin(). Accepts optional config and returns a PluginData tuple.

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@databricks/appkit",
   "type": "module",
-  "version": "0.71.0",
+  "version": "0.72.0",
   "main": "./dist/index.js",
   "types": "./dist/index.d.ts",
   "bin": {
@@ -76,6 +76,7 @@
     "@opentelemetry/semantic-conventions": "1.38.0",
     "@types/semver": "7.7.1",
     "apache-arrow": "21.1.0",
+    "autoevals": "0.3.0",
     "dotenv": "16.6.1",
     "drizzle-orm": "0.45.2",
     "express": "4.22.2",