niceeval 0.4.4 → 0.4.6

package/docs-site/reference/define-eval.mdx

@@ -0,0 +1,160 @@
+---
+title: "defineEval: declare, configure, and run NiceEval evals"
+sidebarTitle: "defineEval"
+description: "Reference for defineEval options, the test context t, Turn return values, sandbox helpers, and dataset array exports."
+---
+
+`defineEval` is the main entry point for authoring evals. Each eval file normally calls it once and default-exports the result.
+
+```ts
+import { defineEval } from "niceeval";
+
+export default defineEval({
+  description: "Brooklyn weather query",
+  async test(t) {
+    await t.send("What's the weather like in Brooklyn today?");
+    t.succeeded();
+  },
+});
+```
+
+<Note>
+  Do not provide `id` or `name`. NiceEval derives the eval ID from the file path.
+</Note>
+
+## `defineEval` options
+
+<ParamField body="description" type="string">
+  Human-readable description shown in console output and reports.
+</ParamField>
+
+<ParamField body="agent" type="string">
+  Agent name for this eval. Usually omitted so the experiment supplies the agent.
+</ParamField>
+
+<ParamField body="tags" type="string[]">
+  Tags for filtering or organizing evals.
+</ParamField>
+
+<ParamField body="judge" type="JudgeConfig">
+  Judge model override for this eval.
+</ParamField>
+
+<ParamField body="reporters" type="Reporter[]">
+  Eval-specific reporters. The instance only observes evals that reference it. If multiple evals share the same instance, their results land in the same destination. See [Reporters](../guides/reporters).
+</ParamField>
+
+<ParamField body="timeoutMs" type="number">
+  Timeout override.
+</ParamField>
+
+<ParamField body="metadata" type="Record<string, unknown>">
+  Extra metadata for reports or external tools.
+</ParamField>
+
+<ParamField body="test" type="(t: TestContext) => Promise<void>" required>
+  Eval body. Drive the agent and declare assertions here.
+</ParamField>
+
+## Test context: `t`
+
+### Always available
+
+<ParamField body="t.send(text)" type="(text: string, opts?) => Promise<Turn>">
+  Sends a message to the agent and returns the current turn.
+</ParamField>
+
+<ParamField body="t.sendFile(path, text?, opts?)" type="function">
+  Sends a file plus optional text.
+</ParamField>
+
+<ParamField body="t.check(value, assertion)" type="(value: unknown, assertion: Assertion) => void">
+  Records a value assertion. Failure does not immediately abort the test.
+</ParamField>
+
+<ParamField body="t.require(value, assertion)" type="(value: unknown, assertion: Assertion) => void">
+  Records a prerequisite assertion. Failure aborts the test immediately.
+</ParamField>
+
+<ParamField body="t.skip(reason)" type="(reason: string) => void">
+  Skips the eval.
+</ParamField>
+
+### Conversation
+
+<ParamField body="t.reply" type="string">
+  Last assistant text message in the current session.
+</ParamField>
+
+<ParamField body="t.newSession()" type="() => TestContext">
+  Starts an independent session line.
+</ParamField>
+
+### Tool observability
+
+<ParamField body="t.calledTool(name, opts?)" type="function">
+  Asserts that a tool was called.
+</ParamField>
+
+<ParamField body="t.notCalledTool(name, opts?)" type="function">
+  Asserts that a tool was not called. This is only trustworthy when the event stream is complete.
+</ParamField>
+
+<ParamField body="t.event(type, opts?)" type="function">
+  Low-level event stream assertion.
+</ParamField>
+
+### Workspace / sandbox
+
+<ParamField body="t.sandbox.fileChanged(path)" type="function">
+  Asserts that a workspace file changed.
+</ParamField>
+
+<ParamField body="t.sandbox.runCommand(command, args?)" type="function">
+  Runs a verification command in the sandbox.
+</ParamField>
+
+## Judge assertions
+
+```ts
+t.judge.autoevals.factuality(expected, { on: t.reply }).atLeast(0.8);
+t.judge.autoevals.closedQA(question, { on: t.reply }).atLeast(0.7);
+t.judge.autoevals.closedQA(rubric, { on: t.reply }).atLeast(0.75);
+```
+
+## `Turn`
+
+<ResponseField name="events" type="StreamEvent[]">
+  Standard event stream for this turn.
+</ResponseField>
+
+<ResponseField name="data" type="unknown | undefined">
+  Structured output, present only when declared by the send call.
+</ResponseField>
+
+<ResponseField name="status" type='"completed" | "failed" | "waiting"'>
+  Current turn status.
+</ResponseField>
+
+<ResponseField name="message" type="string">
+  Assistant text for this turn.
+</ResponseField>
+
+<ResponseField name="toolCalls" type="ToolCall[]">
+  Tool calls for this turn.
+</ResponseField>
+
+## Dataset export
+
+```ts
+export default rows.map((row) =>
+  defineEval({
+    description: row.task,
+    async test(t) {
+      await t.send(row.prompt);
+    },
+  }),
+);
+```
+
+Array exports generate stable IDs such as `file/0000`, `file/0001`, and so on.

package/docs-site/reference/events.mdx

@@ -0,0 +1,131 @@
+---
+title: "Standard event stream reference"
+sidebarTitle: "Events"
+description: "The nine StreamEvent variants: what each one means, when adapters emit it, and which assertions consume it. Producing this stream is the adapter's core job."
+---
+
+An adapter `send` returns a `Turn`, and its `events: StreamEvent[]` is the **only data source for assertions**: `t.calledTool()`, `t.reply`, `toolOrder`, `noFailedActions`, and more all read from this stream. Once your adapter translates "what the agent did this turn" into this stream, the whole assertion surface becomes available.
+
+## Turn: the return value of `send`
+
+```ts
+interface Turn {
+  events: StreamEvent[];                           // events for this turn, in real order
+  data?: unknown;                                 // structured output -> outputEquals / outputMatches
+  status: "completed" | "failed" | "waiting";     // waiting = paused for human input
+  usage?: { inputTokens: number; outputTokens: number }; // -> maxTokens / maxCost / cost reports
+}
+```
+
+`data` means "the structured product of this turn". Only fill it when the application answer itself is structured, such as extraction, classification, or form filling. `outputEquals` / `outputMatches` read from it. If the app only returns text, leave it unset. Do not copy raw response bodies or message text into `data`, and do not serialize structured output back into `events`. Include `usage` only when you have it. Do not invent numbers.
+
+## Event table
+
+| Event | Meaning | Consumed by |
+|---|---|---|
+| `message` | the agent (or user) said something | `t.reply`, `messageIncludes`, judge inputs |
+| `action.called` | a tool call started | `calledTool`, `toolOrder`, `maxToolCalls`, `notCalledTool` |
+| `action.result` | the result of that call | `calledTool` status matching, `noFailedActions` |
+| `subagent.called` | delegated to a subagent | `calledSubagent` |
+| `subagent.completed` | the subagent returned | `calledSubagent` status, `noFailedActions` |
+| `input.requested` | paused and waiting for human input | `t.parked()`, `t.requireInputRequest()` |
+| `thinking` | reasoning text | `event("thinking")`, viewer display |
+| `compaction` | context got compacted | `event("compaction")` |
+| `error` | the turn hit an error | `event("error")`, viewer display |
+
+Any event can also be checked with `event(type)`, `notEvent(type)`, `eventOrder(types)`, or `eventsSatisfy(label, predicate)`.
+
+## Event details
+
+### `message` - what was said
+
+```ts
+{ type: "message", role: "assistant" | "user", text: string }
+```
+
+Emit one `message` with `role: "assistant"` for each chunk of assistant text. **Tool results are not assistant messages.** Do not wrap tool output as `message`, or `t.reply` will read the wrong thing. User `message` events are usually recorded by NiceEval automatically.
+
+### `action.called` / `action.result` - which tool was used, and what happened
+
+```ts
+{ type: "action.called", callId: string, name: string, input: JsonValue }
+{ type: "action.result", callId: string, output?: JsonValue,
+  status: "completed" | "failed" | "rejected" }
+```
+
+- Pair each `action.called` with an `action.result` that has the same `callId`, so concurrent calls do not get mixed up. Use the explicit ID from your source when it exists.
+- Fill `status` truthfully: actual tool execution failure is `"failed"`. Human denial is `"rejected"`.
+- Keep `name` as the tool's original name.
+
+### `subagent.called` / `subagent.completed` - which subagent was delegated to
+
+```ts
+{ type: "subagent.called", callId: string, name: string, remoteUrl?: string }
+{ type: "subagent.completed", callId: string, output?: JsonValue,
+  status: "completed" | "failed" }
+```
+
+Emit this pair when the subject under test delegates work to a subagent and waits for it to return.
+
+### `input.requested` - paused for human input
+
+```ts
+{ type: "input.requested", request: {
+    id?: string,
+    action?: string,
+    input?: JsonValue,
+    prompt?: string,
+    options?: { id: string, label?: string }[],
+} }
+```
+
+When the agent pauses for a person, emit one event per pending question and return overall turn `status: "waiting"`. `t.requireInputRequest(filter)` matches this `request` structure field by field, so populate as many fields as you can. See the HITL part of [Connect Your Agent](/guides/connect-your-agent).
+
+### `thinking` / `compaction` / `error`
+
+```ts
+{ type: "thinking", text: string }
+{ type: "compaction", reason?: string }
+{ type: "error", message: string }
+```
+
+Emit them when your source gives them to you. Do not fabricate them. `compaction` mostly appears in coding-agent CLIs when context is compressed automatically.
+
+## Three mapping rules
+
+1. **Order is fact.** Keep events in real execution order. `toolOrder` and `eventOrder` depend on that sequence.
+2. **Match `callId`s.** Every `called` event should have the corresponding `result` with the same ID.
+3. **Be honest about completeness.** If you declare `toolObservability: true`, all tool calls must be in the stream. If you omit some of them, negative assertions can silently pass. If your manual mapping cannot be complete, [do not claim that capability](/reference/capabilities).
+
+## A complete mapping example
+
+When the agent response contains step records, the mapping is usually a small loop:
+
+```ts
+import type { StreamEvent } from "niceeval";
+
+function toStreamEvents(body: MyBotResponse): StreamEvent[] {
+  const events: StreamEvent[] = [];
+  for (const step of body.steps) {
+    if (step.type === "tool_call") {
+      events.push({ type: "action.called", callId: step.id, name: step.tool, input: step.args });
+      events.push({
+        type: "action.result",
+        callId: step.id,
+        output: step.result,
+        status: step.error ? "failed" : "completed",
+      });
+    }
+    if (step.type === "text") {
+      events.push({ type: "message", role: "assistant", text: step.text });
+    }
+  }
+  return events;
+}
+```
+
+## Related reading
+
+- [Connect Your Agent](/guides/connect-your-agent) - The from-scratch integration tutorial.
+- [Capabilities](/reference/capabilities) - What it means to claim "the event stream is complete".
+- [Authoring Evals](/guides/authoring) - The assertions that consume this stream.

package/docs-site/reference/expect.mdx

@@ -0,0 +1,112 @@
+---
+title: "niceeval/expect matchers and custom assertion reference"
+sidebarTitle: "Assertion Matchers"
+description: "Reference for niceeval/expect: includes, equals, matches, similarity, satisfies. Chain `.gate()` or `.atLeast(0.7)`, or build custom matchers with `makeAssertion`."
+---
+
+`niceeval/expect` provides a set of composable matchers for `t.check()` and `t.require()`. A matcher returns an `Assertion` with a default severity, either `gate` or `soft`.
+
+```ts
+import { includes, equals, matches, similarity, satisfies } from "niceeval/expect";
+```
+
+## How matchers are used
+
+```ts
+t.check(t.reply, includes("confirmed"));
+t.check(turn.data, equals({ intent: "refund" }));
+t.require(turn.status, equals("completed"));
+```
+
+| Method | On failure | Best for |
+|---|---|---|
+| `t.check` | record result, keep executing | most assertions |
+| `t.require` | throw immediately, abort the test | preconditions where continuing makes no sense |
+
+## Matchers
+
+### `includes`
+
+```ts
+includes(substring: string | RegExp): Assertion
+```
+
+Assert that a string contains the given substring, or matches a regex.
+
+```ts
+t.check(t.reply, includes("Paris"));
+t.check(t.reply, includes(/order #\d+/i));
+```
+
+### `equals`
+
+```ts
+equals(expected: unknown): Assertion
+```
+
+Assert deep structural equality, useful for JSON, arrays, and objects.
+
+```ts
+t.check(turn.data, equals({ intent: "refund" }));
+t.check(turn.data, equals(["a", "b", "c"]));
+```
+
+### `matches`
+
+```ts
+matches(pattern: RegExp): Assertion
+```
+
+Assert that a string matches a regex.
+
+```ts
+ t.check(t.reply, matches(/#[A-Z0-9]{8}/));
+```
+
+### `similarity`
+
+```ts
+similarity(expected: string): Assertion
+```
+
+Use this for semantic or textual similarity checks, usually together with a threshold.
+
+```ts
+t.check(t.reply, similarity("The capital of France is Paris.").atLeast(0.8));
+```
+
+### `satisfies`
+
+```ts
+satisfies(predicate: (value: unknown) => boolean): Assertion
+```
+
+Check a value with a custom predicate.
+
+```ts
+t.check(turn.data, satisfies((value) => Array.isArray(value)));
+```
+
+## Gate and soft
+
+```ts
+t.check(t.reply, includes("required").gate());
+t.check(t.reply, includes("nice to have").atLeast(0.7));
+```
+
+- `gate()`: failure makes the eval fail.
+- `soft()`: contributes to scoring without necessarily failing the eval.
+
+## Custom matchers
+
+```ts
+import { makeAssertion } from "niceeval/expect";
+
+const validEmail = makeAssertion("valid email", (value) => {
+  return typeof value === "string" && value.includes("@") ? 1 : 0;
+});
+
+t.check(t.reply, validEmail());
+```
+
+Custom matchers should return a score between 0 and 1 and use a clear name.

package/docs-site/tracker.js

@@ -0,0 +1,8 @@
+(function () {
+  var script = document.createElement("script");
+  script.src = "https://api.goshipfast.com/tracker.js";
+  script.async = true;
+  script.setAttribute("data-project", "cmr24oe2n006qlj10lr9t38n8");
+  script.setAttribute("data-endpoint", "https://api.goshipfast.com");
+  document.head.appendChild(script);
+})();