anpord 0.1.11 → 0.1.12

package/README.md

@@ -1,6 +1,6 @@
 # anpord
 
-TypeScript SDK for [Anpord](https://anpord.com). Run coding agent evals across harnesses, models, and sandboxes, and manage versioned prompts from the same client.
+TypeScript SDK for [Anpord](https://anpord.com). Define coding-agent evals, run them across agents and sandboxes, and manage versioned prompts.
 
 ## Install
 
@@ -8,116 +8,71 @@ TypeScript SDK for [Anpord](https://anpord.com). Run coding agent evals across h
 npm install anpord
 ```
 
-## Run an eval
+## Define an eval
 
 ```ts
-import { Anpord } from "anpord";
-
-const anpord = new Anpord({ apiKey: process.env.ANPORD_API_KEY });
+import { defineEval, empty } from "anpord";
 
-const run = await anpord.evals.startAndWait({
+export default defineEval({
+  name: "greeting",
+  source: empty,
+  prompt: "{{task}}",
   cases: [
     {
-      name: "writes the requested file",
+      name: "writes hello",
       variables: { task: "Create hello.txt containing exactly hello" },
-      verify: "test \"$(cat hello.txt)\" = hello",
+      verify: 'test "$(cat hello.txt)" = hello',
     },
   ],
-  prompt: "{{task}}",
-  tasks: [{ harness: "codex", model: "gpt-5.6-sol", provider: "daytona" }],
+  tasks: [{ harness: "codex", model: "model-id", provider: "daytona" }],
   trials: 3,
 });
-
-for (const cell of run.cells) {
-  console.log(cell.caseName, cell.distribution?.passRate);
-}
 ```
 
-For a TypeScript validator, export the function from its own file:
-
-```ts
-import type { Validator } from "anpord";
-
-export const hasGreeting: Validator = async ({ readText }) =>
-  (await readText("hello.txt")) === "hello";
+```sh
+npx anpord eval ./greeting.eval.ts
 ```
 
-Reference it directly from `greeting.eval.ts`:
+With no file, the CLI discovers every `**/*.eval.ts` suite.
 
-```ts
-import { defineEval } from "anpord";
-import { hasGreeting } from "./validators/greeting";
+## Mock MCP and CLI
 
-export default defineEval({
-  name: "greeting",
-  cases: [
-    {
-      name: "greeting",
-      variables: { task: "Write hello.txt" },
-      validate: hasGreeting,
-    },
-  ],
-  prompt: "{{task}}",
-  tasks: [{ harness: "codex", model: "gpt-5.6-sol", provider: "daytona" }],
-  trials: 3,
-});
-```
+Install Zod for the examples:
 
 ```sh
-npx anpord eval
+npm install zod
 ```
 
-With no paths, the CLI discovers every `**/*.eval.ts` file and starts the
-suites concurrently. Pass files or directories to run a smaller set.
-
-## Mock MCP servers
-
-Define local MCP dependencies once and Anpord gives every built-in agent trial
-a fresh stdio server:
-
 ```ts
 import { z } from "zod";
-import { defineEval } from "anpord";
 import { server, tool } from "anpord/mcp";
 
-const api = server({
-  name: "example",
+const User = z.object({ id: z.string(), name: z.string() });
+
+export const usersMcp = server({
+  name: "users",
   version: "1.0.0",
   tools: [
     tool({
       name: "users_get",
       inputSchema: z.object({ id: z.string() }),
-      outputSchema: z.object({ id: z.string(), name: z.string() }),
+      outputSchema: User,
       handler: ({ id }) => ({ id, name: "Ada" }),
     }),
   ],
 });
-
-export default defineEval({
-  mcp: [api],
-  // cases, prompt, tasks, and trials
-});
 ```
 
-Handlers return plain values or promises. Validators can inspect exact calls
-with `await context.mcp.calls("example")`. No service credentials are needed.
-
-## Mock CLIs
-
-Define a local executable with the same Standard Schema fixtures and attach it
-to any eval:
-
 ```ts
 import { z } from "zod";
-import { defineEval } from "anpord";
 import { cli, command } from "anpord/cli";
 
-const client = cli({
-  path: "example",
+export const usersCli = cli({
+  path: "users",
   version: "1.0.0",
   commands: [
     command({
-      path: ["users", "get"],
+      path: ["get"],
       inputSchema: z.object({ id: z.string() }),
       outputSchema: z.object({ id: z.string(), name: z.string() }),
       options: { id: { type: "string" } },
@@ -125,24 +80,47 @@ const client = cli({
     }),
   ],
 });
+```
 
-export default defineEval({
-  cli: [client],
-  // cases, prompt, tasks, and trials
+Attach definitions with `mcp: [usersMcp]` or `cli: [usersCli]`. Handlers use plain TypeScript. Schemas infer handler types and validate calls at runtime.
+
+## Model judges
+
+```ts
+import { judge } from "anpord/validators";
+
+const correctness = judge({
+  name: "correctness",
+  harness: "codex",
+  model: "gpt-5.6-sol",
+  rubric: "The answer matches the reference without inventing facts.",
+  expected: "Ada",
+  choices: { correct: 1, incorrect: 0 },
 });
 ```
 
-Every built-in agent receives the executable on `PATH`. Validators can inspect
-calls with `await context.cli.calls("example")`.
+Set `validate: correctness`, or combine it with code: `validate: [checkToolCalls, correctness]`. Every check must pass. Judges use the organization's harness connection, including Codex subscriptions. For direct OpenAI calls, use `provider: "openai"` and configure an OpenAI key in the environment connection.
 
-## Resolve a prompt
+See [model judges](https://docs.anpord.com/evals/judges) for isolation, authentication, and unscored failures.
+
+## Use the API
 
 ```ts
-const prompt = await anpord.prompts.get({ id: "support-reply" });
-console.log(prompt.content, prompt.version);
+import { Anpord } from "anpord";
+import { compileEval } from "anpord/eval";
+
+const anpord = new Anpord();
+const run = await anpord.evals.startAndWait(
+  await compileEval("./greeting.eval.ts")
+);
+
+console.log(run.status, run.cells);
+await anpord.dispose();
 ```
 
-See the [documentation](https://docs.anpord.com) for eval concepts, prompt releases, SDK methods, and the API reference.
+The client reads `ANPORD_API_KEY`.
+
+See the [documentation](https://docs.anpord.com) for cases, validators, profiles, mock interfaces, prompt releases, and the API reference.
 
 ## License
 

package/dist/bin.cjs

@@ -1,23 +1,19 @@
 #!/usr/bin/env node
-const require_client = require("./client-Ci0woWLW.cjs");
-const require_errors = require("./errors-Boum34f5.cjs");
+const require_client = require("./client-BnimnYiJ.cjs");
+const require_errors = require("./errors-BX1wry8K.cjs");
 const require_config = require("./config.cjs");
-const require_compiler = require("./compiler-Ap8DM-JU.cjs");
+const require_compiler = require("./compiler-BFxukiJy.cjs");
 let _effect_cli = require("@effect/cli");
 let _effect_platform = require("@effect/platform");
 let _effect_platform_node = require("@effect/platform-node");
 let effect = require("effect");
 let yaml = require("yaml");
 //#region package.json
-var version$1 = "0.1.11";
+var version$1 = "0.1.12";
 //#endregion
 //#region ../template/src/extract.ts
-/**
-* Distinct variable names in source order, so callers can count or list them.
-*
-* Escaped braces are read here too, so a name the renderer will treat as
-* literal text is never reported as a variable the editor should draw.
-*/
+/** Reads escapes too, so text the renderer leaves literal is never reported as
+* a variable. */
 function extractVariables(template) {
 	const names = [];
 	for (const [, open, close, name] of template.matchAll(require_errors.tokenMatcher())) if (open === void 0 && close === void 0 && name !== void 0) names.push(name);
@@ -25,13 +21,6 @@ function extractVariables(template) {
 }
 //#endregion
 //#region src/cli/declarations.ts
-/**
-* The generated file has to open by importing the module it augments.
-*
-* Without it `declare module` shadows the module rather than adding to it, and
-* every import of the SDK stops resolving with an error naming the missing
-* export rather than the file that hid it.
-*/
 const PREAMBLE = [
 	"// Generated by `anpord generate`. Do not edit.",
 	"// Run it again after changing which variables a prompt uses.",
@@ -40,7 +29,6 @@ const PREAMBLE = [
 	"declare module \"anpord\" {",
 	"  interface AnpordPromptVariables {"
 ].join("\n");
-/** Quoted, because an id may carry characters an identifier cannot. */
 const entry = (id, names) => {
 	return `    "${id}": ${names.length === 0 ? "Record<string, never>" : `{ ${names.map((name) => `"${name}": string`).join("; ")} }`};`;
 };
@@ -108,8 +96,6 @@ const promptContent = (prompt) => effect.Effect.sync(() => {
 	process.stdout.write(prompt.content);
 	if (!prompt.content.endsWith("\n")) process.stdout.write("\n");
 });
-/** A result the caller may pipe into another tool, so it belongs on stdout
-* alongside the prompt content rather than beside the status messages. */
 const row = (line) => effect.Effect.sync(() => {
 	process.stdout.write(`${line}\n`);
 });
@@ -192,8 +178,6 @@ var CaseFileNotJson = class extends effect.Data.TaggedError("CaseFileNotJson") {
 		return `${this.path} is not JSON: ${this.reason}`;
 	}
 };
-/** Names the case rather than the byte offset, because the author fixing it
-* reads the file by case, not by position. */
 var CaseFileNotEvalsJson = class extends effect.Data.TaggedError("CaseFileNotEvalsJson") {
 	get message() {
 		return `${this.path} is not an evals-json file: ${this.reason}`;
@@ -206,29 +190,17 @@ var CaseFileEmpty = class extends effect.Data.TaggedError("CaseFileEmpty") {
 };
 //#endregion
 //#region src/imports/typescript-literal.ts
-/** A backslash-escaped double-quoted literal. Backslash first, so the escapes
-* added after it are not themselves escaped. */
 const quoted = (value) => `"${value.replaceAll("\\", "\\\\").replaceAll("\"", "\\\"").replaceAll("\n", "\\n").replaceAll("\r", "\\r").replaceAll("	", "\\t")}"`;
-/** A template literal, which keeps a multi-line prompt readable in the
-* generated file. A backtick ends the literal and `${` opens a substitution,
-* so both are escaped; a lone `$` is left alone because only the pair means
-* anything. A trailing backslash would escape the closing backtick. */
 const templated = (value) => `\`${value.replaceAll("\\", "\\\\").replaceAll("`", "\\`").replaceAll("${", "\\${")}\``;
-/** A comment cannot carry the sequence that closes it, and a newline would end
-* a line comment and let the rest of the text run as code. */
 const commentSafe = (value) => value.replaceAll("*/", "*\\/").replaceAll(/\r?\n/g, " ");
 //#endregion
 //#region src/imports/unwritten-check.ts
 const PLACEHOLDER = "unwritten";
-/** The author's own words, kept whole, because they are the specification for
-* the check that replaces the line beneath them. */
 const proseLine = (text) => [
 	"          /* Write this check, then delete the line under it: */",
 	`          /* ${commentSafe(text)} */`,
 	`          ${PLACEHOLDER}(${quoted(text)}),`
 ].join("\n");
-/** Local rather than imported, so the generated file carries its own proof
-* that an unconverted assertion fails. Deleting the last call deletes it. */
 const placeholderBlock = [
 	"/* A check nobody has written yet. It is false, so the case stays red until",
 	"   the sentence above it becomes a real check. The argument is that",
@@ -259,8 +231,6 @@ const nameOf = (subject) => `${slug$1(subject.name ?? "case", "case")}-${subject
 const scorerLine = (assertion) => [`          /* ${commentSafe(assertion.text)} */`, `          ${SCORER_OF[assertion.kind]}(answer, [${assertion.needles.map(quoted).join(", ")}]),`].join("\n");
 const scorerLines = (subject) => subject.assertions.map((assertion) => isStructured(assertion) ? scorerLine(assertion) : proseLine(assertion)).join("\n");
 const expectationComment = (subject) => subject.expected_output === void 0 ? [] : [`      /* The file's expected output: ${commentSafe(subject.expected_output)} */`];
-/** The JSON names fixture directories by convention and carries none of their
-* contents, so the author supplies the files the name stood for. */
 const sourceComment = (subject) => subject.files.length === 0 ? "      /* This case named no fixture directory. Add the files it starts from. */" : `      /* This case named ${subject.files.map(commentSafe).join(", ")}. The JSON carries the directory names, not their contents, so add the files here. */`;
 const caseBlock$1 = (subject) => [
 	"    {",
@@ -319,16 +289,12 @@ const renderEvalSuite = (file) => {
 };
 //#endregion
 //#region src/imports/evals-json-schema.ts
-/** The three checks that convert mechanically, because each is a needle list
-* and a rule over it rather than a description of intent. */
 const AssertionKind = effect.Schema.Literal("content_contains_any", "content_contains_all", "content_contains_none");
 const StructuredAssertion = effect.Schema.Struct({
 	kind: AssertionKind,
 	needles: effect.Schema.Array(effect.Schema.String),
 	text: effect.Schema.String
 });
-/** Both dialects occur, sometimes in sibling files: a bare string is prose one
-* person wrote for another, an object is a mechanical check. */
 const Assertion = effect.Schema.Union(effect.Schema.String, StructuredAssertion);
 const EvalsJsonCase = effect.Schema.Struct({
 	assertions: effect.Schema.Array(Assertion),
@@ -356,8 +322,6 @@ const idAt = (parsed, index) => {
 	const found = parsed?.evals?.[index];
 	return found?.id === void 0 ? `${index}` : String(found.id);
 };
-/** The case as its author sees it. A path into the decoded value tells them
-* nothing; the id is what they search the file for. */
 const located = (parsed, path) => {
 	const [head, index, ...rest] = path;
 	if (head !== "evals" || typeof index !== "number") return path.map(String).join(".");
@@ -393,8 +357,6 @@ var CaseFileNotYaml = class extends effect.Data.TaggedError("CaseFileNotYaml") {
 		return `${this.path} is not YAML: ${this.reason}`;
 	}
 };
-/** Names the file rather than a path into the decoded value, because one case
-* is one file and the file name is what the author searches for. */
 var CaseFileNotYamlCase = class extends effect.Data.TaggedError("CaseFileNotYamlCase") {
 	get message() {
 		return `${this.path} is not a yaml case: ${this.reason}`;
@@ -411,20 +373,12 @@ const slug = (value, fallback) => {
 	const cleaned = value.toLowerCase().replaceAll(/[^a-z0-9]+/g, "-").replaceAll(/^-+|-+$/g, "");
 	return cleaned === "" ? fallback : cleaned;
 };
-/** Every line is prose a person wrote for a model judge, so none of it
-* converts and the whole list is what a human still owes. A case with no
-* lines owes one too: something has to say what a good answer is. */
 const tallyOf = (files) => ({
 	cases: files.length,
 	converted: 0,
 	needsAuthor: files.reduce((total, file) => total + Math.max(file.subject.judge_context.length, 1), 0)
 });
-/** Anpord has no step budget, so the number is kept as a note rather than
-* dropped: the suite it came from ran under it, and a case that needed ten
-* steps is a different case from one that needed a hundred. */
 const budgetComment = (subject) => `      /* The file allowed ${subject.max_steps} steps. Anpord does not cap steps, so this is a note, not a limit. */`;
-/** A file with no judge context says nothing about what a good answer is, so
-* it is owed a check like any other rather than passing by default. */
 const UNJUDGED = "This case named no judge context. Write what a good answer is.";
 const judgeLines = (subject) => (subject.judge_context.length === 0 ? [UNJUDGED] : subject.judge_context).map((line) => proseLine(line)).join("\n");
 const caseBlock = (file) => [
@@ -446,8 +400,6 @@ const caseBlock = (file) => [
 	"      },",
 	"    },"
 ].join("\n");
-/** A directory has no name of its own in the files, so a suite built from
-* several is named generically and the author renames it. */
 const suiteName = (files) => files.length === 1 ? slug(files[0]?.subject.name ?? "", "imported-suite") : "imported-suite";
 const renderYamlSuite = (files) => `${[
 	"import { defineEval, files } from \"anpord\";",
@@ -469,12 +421,7 @@ const renderYamlSuite = (files) => `${[
 ].join("\n")}\n`;
 //#endregion
 //#region src/imports/yaml-cases-schema.ts
-/** The runner's own default when a file omits the field, kept here so an
-* imported case carries the budget it actually ran under. */
 const DEFAULT_MAX_STEPS = 15;
-/** A missing `judge_context` is a case whose author wrote nothing about how to
-* judge it, which the runner reads as one generic line. It is not an error, so
-* it decodes to an empty list and the generated case says so. */
 const YamlCase = effect.Schema.Struct({
 	judge_context: effect.Schema.optionalWith(effect.Schema.Array(effect.Schema.String), { default: () => [] }),
 	max_steps: effect.Schema.optionalWith(effect.Schema.Int, { default: () => DEFAULT_MAX_STEPS }),
@@ -484,10 +431,6 @@ const YamlCase = effect.Schema.Struct({
 const decodeYamlCase = effect.Schema.decodeUnknown(YamlCase, { errors: "all" });
 //#endregion
 //#region src/imports/yaml-document.ts
-/** A YAML document can parse into a value while still carrying errors the
-* author needs to see, so the errors are read rather than the throw relied on.
-* The value is returned as `unknown` because the schema, not the parser,
-* decides what shape it has. */
 const parseYamlDocument = (body) => effect.Effect.try({
 	catch: (cause) => cause instanceof Error ? cause.message : String(cause),
 	try: () => {
@@ -519,8 +462,6 @@ const readCase = (path) => effect.Effect.gen(function* () {
 		subject: yield* decode(path, parsed)
 	};
 });
-/** Sorted, so the same directory always produces the same suite and a diff of
-* two imports shows what changed rather than what moved. */
 const caseFilesIn = (directory) => effect.Effect.gen(function* () {
 	return (yield* (yield* _effect_platform.FileSystem.FileSystem).readDirectory(directory).pipe(effect.Effect.mapError((cause) => new CaseDirectoryUnreadable({
 		cause,
@@ -547,20 +488,13 @@ const importYamlCases = (path) => effect.Effect.gen(function* () {
 }).pipe(effect.Effect.withSpan("Imports.yamlCases"));
 //#endregion
 //#region src/cli/import-formats.ts
-/** A format is an entry here, so adding one adds an entry rather than editing
-* the command. Each name carries the importer it names, so the parsed option
-* is the importer itself and no lookup can miss. */
 const FORMATS = [["evals-json", importEvalsJson], ["yaml", importYamlCases]];
 //#endregion
 //#region src/cli/eval-import.ts
-/** A path rather than a file, because a format may keep one case per file and
-* import a directory of them as one suite. */
 const caseFile = _effect_cli.Args.path({ name: "path" }).pipe(_effect_cli.Args.withDescription("The case file, or a directory of them, to read"));
 const format = _effect_cli.Options.choiceWithValue("format", FORMATS).pipe(_effect_cli.Options.withDescription("The case file's format"));
 const out$1 = _effect_cli.Options.file("out").pipe(_effect_cli.Options.withDescription("Where to write the suite; it goes to stdout when omitted"), _effect_cli.Options.optional);
 const plural = (count, one) => `${count} ${count === 1 ? one : `${one}s`}`;
-/** The count of unwritten checks leads, because a suite that reports a pass
-* for a check nobody wrote is the failure this product exists to prevent. */
 const summaryOf = (tally) => {
 	const read = `Read ${plural(tally.cases, "case")}, converted ${plural(tally.converted, "assertion")}.`;
 	return tally.needsAuthor === 0 ? read : `${read} ${plural(tally.needsAuthor, "assertion")} could not be converted and ${tally.needsAuthor === 1 ? "needs" : "need"} a human: each is written as prose, and the suite fails until you write the check it describes.`;
@@ -835,12 +769,6 @@ const promote = _effect_cli.Command.make("promote", {
 	} });
 	return yield* note(`${id} v${pin} is now ${to}`);
 })).pipe(_effect_cli.Command.withDescription("Point a channel at a version"));
-/**
-* Read from the stream rather than by opening `/dev/stdin` as a file. The path
-* only names the pipe, so reading it races whoever is writing: a body arriving
-* in more than one chunk, which is what a pipe does under load, was read as
-* whatever had landed by then.
-*/
 const readStdin = effect.Effect.async((resume) => {
 	let body = "";
 	process.stdin.setEncoding("utf8");
@@ -865,8 +793,6 @@ const push = _effect_cli.Command.make("push", {
 	return yield* note(`${id} is now v${prompt.version}`);
 })).pipe(_effect_cli.Command.withDescription("Add a version to a prompt"));
 const out = _effect_cli.Options.file("out").pipe(_effect_cli.Options.withDescription("Where to write the declarations"), _effect_cli.Options.withDefault("anpord-env.d.ts"));
-/** Reading one prompt at a time because the list carries no content, bounded
-* so a large organisation does not open a connection per prompt. */
 const READ_AT_ONCE = 8;
 const writeDeclarations = ({ out: path }) => effect.Effect.gen(function* () {
 	const api = yield* require_client.AnpordApi;
@@ -878,8 +804,6 @@ const writeDeclarations = ({ out: path }) => effect.Effect.gen(function* () {
 });
 const DESCRIPTION = "Write TypeScript declarations for prompt variables";
 const generate = _effect_cli.Command.make("generate", { out }, writeDeclarations).pipe(_effect_cli.Command.withDescription(DESCRIPTION));
-/** A second command rather than an alias, because a command carries one name
-* and the shorter one is what anybody types twice. */
 const gen = _effect_cli.Command.make("gen", { out }, writeDeclarations).pipe(_effect_cli.Command.withDescription(DESCRIPTION));
 const withClient = (command) => _effect_cli.Command.provide(command, require_config.ClientLayer);
 const commands = [