anpord 0.1.24 → 0.1.25

package/README.md

@@ -1,6 +1,6 @@
 # anpord
 
-TypeScript SDK for [Anpord](https://anpord.com). Define coding-agent evals, run them across agents and sandboxes, and manage versioned prompts.
+TypeScript SDK and CLI for [Anpord](https://anpord.com). Define coding-agent evals, run them on different harnesses, models and sandboxes, and manage versioned prompts.
 
 ## Install
 
@@ -8,32 +8,44 @@ TypeScript SDK for [Anpord](https://anpord.com). Define coding-agent evals, run
 npm install anpord
 ```
 
-## Define an eval
+## Define a suite
+
+A suite groups cases that share a prompt and setup. Suites and cases need an `id`, a lowercase handle such as `writes-hello`; a `name` is optional and defaults to the id. Each variant is a harness, model and sandbox to run every case on, with an optional profile.
 
 ```ts
-import { suite, empty } from "anpord";
+import { command, suite, empty } from "anpord";
 
 export default suite({
-  name: "greeting",
+  id: "greeting",
   source: empty,
   prompt: "{{task}}",
   cases: [
     {
-      name: "writes hello",
+      id: "writes-hello",
       variables: { task: "Create hello.txt containing exactly hello" },
-      verify: 'test "$(cat hello.txt)" = hello',
+      validate: command('test "$(cat hello.txt)" = hello'),
+    },
+  ],
+  variants: [
+    { harness: "codex", model: "gpt-5.6-sol" },
+    { harness: "claude", model: "sonnet", sandbox: "daytona" },
+    {
+      harness: "opencode",
+      model: "anthropic/claude-sonnet-4.6",
+      profile: { dir: "./profiles/strict", name: "strict" },
     },
   ],
-  tasks: [{ harness: "codex", model: "model-id", provider: "daytona" }],
   trials: 3,
 });
 ```
 
+A case has one `validate`: a shell `command(...)` whose exit code decides the trial, a validator function, a model judge, or an array of validators and judges that must all pass. A command decides a case on its own and cannot go in an array.
+
 ```sh
 npx anpord eval ./greeting.eval.ts
 ```
 
-With no file, the CLI discovers every `**/*.eval.ts` suite.
+Each file starts a batch: one run per case per variant. Each run makes `trials` attempts, each in its own sandbox. With no file, the CLI finds every `*.eval.ts` file under the current directory.
 
 ## Mock MCP and CLI
 
@@ -82,7 +94,7 @@ export const usersCli = cli({
 });
 ```
 
-Attach definitions with `mcp: [usersMcp]` or `cli: [usersCli]`. Handlers use plain TypeScript. Schemas infer handler types and validate calls at runtime.
+Attach them to a suite with `mcp: [usersMcp]` or `cli: [usersCli]`. Schemas type the handlers and validate calls at runtime. The `command` from `anpord/cli` declares a mock CLI command, unlike the `command` validator from `anpord`, so alias one of them in a file that imports both.
 
 ## Model judges
 
@@ -99,28 +111,45 @@ const correctness = judge({
 });
 ```
 
-Set `validate: correctness`, or combine it with code: `validate: [checkToolCalls, correctness]`. Every check must pass. Judges use the organization's harness connection, including Codex subscriptions. For direct OpenAI calls, use `provider: "openai"` and configure an OpenAI key in the environment connection.
+Set `validate: correctness` on a case, or combine it with code: `validate: [checkToolCalls, correctness]`. Every check must pass. A judge with `harness` uses that harness connection, including a ChatGPT sign-in for Codex. To call OpenAI directly, use `provider: "openai"` and add an OpenAI model connection.
 
-See [model judges](https://docs.anpord.com/evals/judges) for isolation, authentication, and unscored failures.
+See [model judges](https://docs.anpord.com/evals/judges) for isolation, authentication and unscored failures.
 
-## Use the API
+## Start a batch from code
 
 ```ts
 import { Anpord } from "anpord";
-import { compileEval } from "anpord/eval";
+import greeting from "./greeting.eval";
 
 const anpord = new Anpord();
-const run = await anpord.evals.startAndWait(
-  await compileEval("./greeting.eval.ts")
-);
+const batch = await anpord.evals.batches.startAndWait(greeting);
 
-console.log(run.status, run.cells);
+console.log(batch.status, batch.runs);
 await anpord.dispose();
 ```
 
-The client reads `ANPORD_API_KEY`.
+The client reads `ANPORD_API_KEY`. `batch.runs` holds one run per case and variant, each with its trials and pass rate. `evals.batches.start` returns as soon as the batch is started, and `evals.batches.wait({ id })` waits on one started earlier.
+
+## Read and rerun from code
+
+```ts
+const { cases } = await anpord.evals.cases.list({ suite: "greeting" });
+const detail = await anpord.evals.cases.get({ id: "writes-hello" });
+
+const rerun = await anpord.evals.cases.run({
+  id: "writes-hello",
+  trials: 3,
+  variants: detail.variants.map(({ variant }) => variant.id),
+});
+
+const { runs } = await anpord.evals.runs.list({ caseId: "writes-hello" });
+const run = await anpord.evals.runs.get({ id: rerun.runs[0].id });
+const models = await anpord.evals.models.list({ harness: "codex" });
+```
+
+`evals.batches`, `evals.cases`, `evals.runs` and `evals.models` follow the data model: a suite holds cases, a case runs on variants, and each run of a case on a variant holds its trials. A batch is the runs started together.
 
-See the [documentation](https://docs.anpord.com) for cases, validators, profiles, mock interfaces, prompt releases, and the API reference.
+See the [documentation](https://docs.anpord.com) for cases, validators, profiles, mocks, prompt releases and the API reference.
 
 ## License
 

package/dist/api-context.cjs

@@ -1,5 +1,5 @@
 Object.defineProperty(exports, Symbol.toStringTag, { value: "Module" });
-const require_api_mocks = require("./api-mocks-CgiC7T9U.cjs");
+const require_api_mocks = require("./api-mocks-BEorMdN5.cjs");
 let effect = require("effect");
 let node_fs_promises = require("node:fs/promises");
 //#region src/mock-api/context.ts

package/dist/api-context.d.cts

@@ -1,4 +1,4 @@
-import { t as ApiCall } from "./api-mocks-ofqP55P_.cjs";
+import { t as ApiCall } from "./api-mocks-bVa7eYtc.cjs";
 //#region src/mock-api/context.d.ts
 declare const apiContext: {
   url: (name: string) => Promise<string>;

package/dist/api-context.d.mts

@@ -1,4 +1,4 @@
-import { t as ApiCall } from "./api-mocks-ofqP55P_.mjs";
+import { t as ApiCall } from "./api-mocks-bVa7eYtc.mjs";
 //#region src/mock-api/context.d.ts
 declare const apiContext: {
   url: (name: string) => Promise<string>;

package/dist/api-context.mjs

@@ -1,4 +1,4 @@
-import { a as ApiCall, n as API_MANIFEST, o as ApiManifest, t as API_JOURNAL } from "./api-mocks-BYWvJp9v.mjs";
+import { a as ApiCall, n as API_MANIFEST, o as ApiManifest, t as API_JOURNAL } from "./api-mocks-RCkffSgG.mjs";
 import { Schema } from "effect";
 import { readFile } from "node:fs/promises";
 //#region src/mock-api/context.ts

package/dist/{eval-validations-CZm7vVtQ.cjs → api-mocks-BEorMdN5.cjs}

@@ -110,6 +110,78 @@ const validationExecution = (identity, startedAt) => ({
 	logs: []
 });
 //#endregion
+//#region ../schema/src/domain/api-mocks.ts
+const API_PROGRAM = "workspace/.anpord/api/program.json";
+const API_MANIFEST = ".anpord/api/manifest.json";
+const API_JOURNAL = ".anpord/api/calls.jsonl";
+const API_READY = "ANPORD_API_READY=";
+const ApiProgram = effect.Schema.Struct({ entry: effect.Schema.Literal(".anpord/api/server.mjs") });
+const ApiManifest = effect.Schema.Array(effect.Schema.Struct({
+	name: effect.Schema.String,
+	url: effect.Schema.String,
+	endpoints: effect.Schema.Array(effect.Schema.Struct({
+		method: effect.Schema.String,
+		path: effect.Schema.String,
+		description: effect.Schema.optional(effect.Schema.String)
+	}))
+}));
+var ApiCall = class extends effect.Schema.Class("ApiCall")({
+	api: effect.Schema.String,
+	index: effect.Schema.NonNegativeInt,
+	method: effect.Schema.String,
+	path: effect.Schema.String,
+	matched: effect.Schema.Boolean,
+	startedAt: effect.Schema.Number,
+	durationMs: effect.Schema.NonNegativeInt,
+	input: ValidationValue,
+	output: ValidationValue,
+	status: effect.Schema.Int,
+	error: effect.Schema.NullOr(effect.Schema.String.pipe(effect.Schema.maxLength(2e3))),
+	logs: effect.Schema.Array(ValidationValue).pipe(effect.Schema.maxItems(64))
+}) {};
+//#endregion
+Object.defineProperty(exports, "API_JOURNAL", {
+	enumerable: true,
+	get: function() {
+		return API_JOURNAL;
+	}
+});
+Object.defineProperty(exports, "API_MANIFEST", {
+	enumerable: true,
+	get: function() {
+		return API_MANIFEST;
+	}
+});
+Object.defineProperty(exports, "API_PROGRAM", {
+	enumerable: true,
+	get: function() {
+		return API_PROGRAM;
+	}
+});
+Object.defineProperty(exports, "API_READY", {
+	enumerable: true,
+	get: function() {
+		return API_READY;
+	}
+});
+Object.defineProperty(exports, "ApiCall", {
+	enumerable: true,
+	get: function() {
+		return ApiCall;
+	}
+});
+Object.defineProperty(exports, "ApiManifest", {
+	enumerable: true,
+	get: function() {
+		return ApiManifest;
+	}
+});
+Object.defineProperty(exports, "ApiProgram", {
+	enumerable: true,
+	get: function() {
+		return ApiProgram;
+	}
+});
 Object.defineProperty(exports, "EvalValidation", {
 	enumerable: true,
 	get: function() {
@@ -134,12 +206,6 @@ Object.defineProperty(exports, "VALIDATION_TEXT_LIMIT", {
 		return VALIDATION_TEXT_LIMIT;
 	}
 });
-Object.defineProperty(exports, "ValidationValue", {
-	enumerable: true,
-	get: function() {
-		return ValidationValue;
-	}
-});
 Object.defineProperty(exports, "unavailableValue", {
 	enumerable: true,
 	get: function() {

package/dist/{eval-validations-DNO-qwkU.mjs → api-mocks-RCkffSgG.mjs}

@@ -110,4 +110,34 @@ const validationExecution = (identity, startedAt) => ({
 	logs: []
 });
 //#endregion
-export { ValidationValue as a, validationExecution as c, VALIDATION_TEXT_LIMIT as i, validationSnapshot as l, EvalValidations as n, unavailableValue as o, VALIDATION_FRAME as r, validationCapture as s, EvalValidation as t };
+//#region ../schema/src/domain/api-mocks.ts
+const API_PROGRAM = "workspace/.anpord/api/program.json";
+const API_MANIFEST = ".anpord/api/manifest.json";
+const API_JOURNAL = ".anpord/api/calls.jsonl";
+const API_READY = "ANPORD_API_READY=";
+const ApiProgram = Schema.Struct({ entry: Schema.Literal(".anpord/api/server.mjs") });
+const ApiManifest = Schema.Array(Schema.Struct({
+	name: Schema.String,
+	url: Schema.String,
+	endpoints: Schema.Array(Schema.Struct({
+		method: Schema.String,
+		path: Schema.String,
+		description: Schema.optional(Schema.String)
+	}))
+}));
+var ApiCall = class extends Schema.Class("ApiCall")({
+	api: Schema.String,
+	index: Schema.NonNegativeInt,
+	method: Schema.String,
+	path: Schema.String,
+	matched: Schema.Boolean,
+	startedAt: Schema.Number,
+	durationMs: Schema.NonNegativeInt,
+	input: ValidationValue,
+	output: ValidationValue,
+	status: Schema.Int,
+	error: Schema.NullOr(Schema.String.pipe(Schema.maxLength(2e3))),
+	logs: Schema.Array(ValidationValue).pipe(Schema.maxItems(64))
+}) {};
+//#endregion
+export { ApiCall as a, EvalValidation as c, VALIDATION_TEXT_LIMIT as d, unavailableValue as f, validationSnapshot as h, API_READY as i, EvalValidations as l, validationExecution as m, API_MANIFEST as n, ApiManifest as o, validationCapture as p, API_PROGRAM as r, ApiProgram as s, API_JOURNAL as t, VALIDATION_FRAME as u };

package/dist/{api-mocks-ofqP55P_.d.cts → api-mocks-bVa7eYtc.d.cts}

@@ -58,43 +58,43 @@ declare const ApiCall_base: Schema.Class<ApiCall, {
   }>>>;
 }>, never, {
   readonly api: string;
-} & {
-  readonly status: number;
-} & {
-  readonly startedAt: number;
-} & {
-  readonly path: string;
-} & {
-  readonly index: number;
 } & {
   readonly output: {
     readonly text: string;
-    readonly truncated: boolean;
     readonly format: "text" | "json";
     readonly state: "captured" | "disabled" | "unavailable";
+    readonly truncated: boolean;
   };
 } & {
   readonly input: {
     readonly text: string;
-    readonly truncated: boolean;
     readonly format: "text" | "json";
     readonly state: "captured" | "disabled" | "unavailable";
+    readonly truncated: boolean;
   };
 } & {
-  readonly error: string | null;
+  readonly path: string;
+} & {
+  readonly method: string;
+} & {
+  readonly index: number;
+} & {
+  readonly matched: boolean;
+} & {
+  readonly startedAt: number;
 } & {
   readonly durationMs: number;
 } & {
-  readonly method: string;
+  readonly status: number;
+} & {
+  readonly error: string | null;
 } & {
   readonly logs: readonly {
     readonly text: string;
-    readonly truncated: boolean;
     readonly format: "text" | "json";
     readonly state: "captured" | "disabled" | "unavailable";
+    readonly truncated: boolean;
   }[];
-} & {
-  readonly matched: boolean;
 }, {}, {}>;
 declare class ApiCall extends ApiCall_base {}
 //#endregion

package/dist/{api-mocks-ofqP55P_.d.mts → api-mocks-bVa7eYtc.d.mts}

@@ -58,43 +58,43 @@ declare const ApiCall_base: Schema.Class<ApiCall, {
   }>>>;
 }>, never, {
   readonly api: string;
-} & {
-  readonly status: number;
-} & {
-  readonly startedAt: number;
-} & {
-  readonly path: string;
-} & {
-  readonly index: number;
 } & {
   readonly output: {
     readonly text: string;
-    readonly truncated: boolean;
     readonly format: "text" | "json";
     readonly state: "captured" | "disabled" | "unavailable";
+    readonly truncated: boolean;
   };
 } & {
   readonly input: {
     readonly text: string;
-    readonly truncated: boolean;
     readonly format: "text" | "json";
     readonly state: "captured" | "disabled" | "unavailable";
+    readonly truncated: boolean;
   };
 } & {
-  readonly error: string | null;
+  readonly path: string;
+} & {
+  readonly method: string;
+} & {
+  readonly index: number;
+} & {
+  readonly matched: boolean;
+} & {
+  readonly startedAt: number;
 } & {
   readonly durationMs: number;
 } & {
-  readonly method: string;
+  readonly status: number;
+} & {
+  readonly error: string | null;
 } & {
   readonly logs: readonly {
     readonly text: string;
-    readonly truncated: boolean;
     readonly format: "text" | "json";
     readonly state: "captured" | "disabled" | "unavailable";
+    readonly truncated: boolean;
   }[];
-} & {
-  readonly matched: boolean;
 }, {}, {}>;
 declare class ApiCall extends ApiCall_base {}
 //#endregion

package/dist/api-runtime.cjs

@@ -1,4 +1,4 @@
 Object.defineProperty(exports, Symbol.toStringTag, { value: "Module" });
-const require_runtime = require("./runtime-Bfs2j0jW.cjs");
+const require_runtime = require("./runtime-DX_9jN8J.cjs");
 exports.runApiServers = require_runtime.runApiServers;
 exports.withApi = require_runtime.withApi;

package/dist/api-runtime.d.cts

@@ -1,4 +1,4 @@
-import { t as ApiCall } from "./api-mocks-ofqP55P_.cjs";
+import { t as ApiCall } from "./api-mocks-bVa7eYtc.cjs";
 import { t as ApiDefinition } from "./define-NZvmQuIv.cjs";
 //#region src/mock-api/runtime.d.ts
 declare const withApi: <A>({ api, run }: {

package/dist/api-runtime.d.mts

@@ -1,4 +1,4 @@
-import { t as ApiCall } from "./api-mocks-ofqP55P_.mjs";
+import { t as ApiCall } from "./api-mocks-bVa7eYtc.mjs";
 import { t as ApiDefinition } from "./define-NZvmQuIv.mjs";
 //#region src/mock-api/runtime.d.ts
 declare const withApi: <A>({ api, run }: {

package/dist/api-runtime.mjs

@@ -1,2 +1,2 @@
-import { n as withApi, t as runApiServers } from "./runtime-CYr8RbsW.mjs";
+import { n as withApi, t as runApiServers } from "./runtime-B58_CBI5.mjs";
 export { runApiServers, withApi };

package/dist/api.cjs

@@ -1,5 +1,5 @@
 Object.defineProperty(exports, Symbol.toStringTag, { value: "Module" });
-const require_runtime = require("./runtime-Bfs2j0jW.cjs");
+const require_runtime = require("./runtime-DX_9jN8J.cjs");
 exports.api = require_runtime.api;
 exports.endpoint = require_runtime.endpoint;
 exports.withApi = require_runtime.withApi;

package/dist/api.d.cts

@@ -1,4 +1,4 @@
-import { t as ApiCall } from "./api-mocks-ofqP55P_.cjs";
+import { t as ApiCall } from "./api-mocks-bVa7eYtc.cjs";
 import { a as endpoint, i as api, n as ApiHandlerContext, r as EndpointDefinition, t as ApiDefinition } from "./define-NZvmQuIv.cjs";
 import { withApi } from "./api-runtime.cjs";
 export { type ApiCall, type ApiDefinition, type ApiHandlerContext, type EndpointDefinition, api, endpoint, withApi };

package/dist/api.d.mts

@@ -1,4 +1,4 @@
-import { t as ApiCall } from "./api-mocks-ofqP55P_.mjs";
+import { t as ApiCall } from "./api-mocks-bVa7eYtc.mjs";
 import { a as endpoint, i as api, n as ApiHandlerContext, r as EndpointDefinition, t as ApiDefinition } from "./define-NZvmQuIv.mjs";
 import { withApi } from "./api-runtime.mjs";
 export { type ApiCall, type ApiDefinition, type ApiHandlerContext, type EndpointDefinition, api, endpoint, withApi };

package/dist/api.mjs

@@ -1,2 +1,2 @@
-import { i as endpoint, n as withApi, r as api } from "./runtime-CYr8RbsW.mjs";
+import { i as endpoint, n as withApi, r as api } from "./runtime-B58_CBI5.mjs";
 export { api, endpoint, withApi };