raindrop-ai 0.4.0 → 0.5.0

package/README.md

@@ -65,6 +65,174 @@ On runtimes that lack a usable `AsyncLocalStorage.enterWith` (notably Cloudflare
 - `withSpan`/`withTool` invoke your callback as `fn.apply(thisArg, args)` — the callback receives exactly the arguments you pass (an earlier version prepended a spurious leading `undefined`). If you had code compensating for that shifted argument, remove the workaround.
 - With per-client interaction registries, `resumeInteraction(eventId)` returns only interactions that **this** client began; a second client resuming the same event id gets its own fresh interaction rather than the first client's.
 
+## Eval mode
+
+A replay calls your agent once per row and needs every span and event that call
+emits to land on that row. `withEvalScope` binds one opaque **correlation id**
+to the execution context, so it does, with no change to the agent:
+
+```ts
+import { withEvalScope } from "raindrop-ai";
+
+for (const row of rows) {
+  await withEvalScope({ correlationId: row.correlationId }, () => supportAgent(row.input));
+}
+```
+
+Everything started inside the callback carries the id: spans you create with
+`withSpan`/`withTool`, auto-instrumented LLM spans from libraries you never call
+directly, and events from `trackAi`. The agent itself is unchanged and never
+sees a correlation id.
+
+`currentEvalScope()` reads the scope if you need it, and returns `undefined`
+outside one. Nothing changes for a caller who never enters a scope. Spans and
+events carry exactly the attributes they carried before.
+
+**Concurrency.** Rows run concurrently and each keeps its own id. The scope is
+bound in `AsyncLocalStorage`, so two agents in flight at once never see each
+other's row.
+
+**Nesting replaces, it does not merge.** A correlation id names exactly one
+row, and a span belongs to exactly one row, so a nested `withEvalScope` wins for
+its own duration and the outer scope is restored when it returns. Merging two
+ids would produce a third identity that is neither.
+
+### Known limits
+
+**Work whose async context predates the scope carries no id.** A continuation
+attached before the row started, a background flush loop, or a pooled
+connection's pending callback runs outside the scope even when the row triggers
+it. Those spans land in normal traffic.
+
+**Work detached inside the scope keeps the id and can outlive the row.** This
+is the opposite of what it sounds like. `AsyncLocalStorage` propagates into
+timers and unawaited promises, so a detached task started inside the callback
+still carries the correlation id when it emits, possibly after the row has been
+recorded. Narrow it by awaiting everything the row starts before the callback
+returns.
+
+Both limits are demonstrated by tests in `packages/core/src/eval-scope.test.ts`.
+
+### Replaying a dataset
+
+`replay` is the loop that uses the scope. It asks Raindrop for the dataset's
+rows, calls your agent once per row inside that row's scope, says which rows
+failed, and waits for the traces to land:
+
+```ts
+import Raindrop, { replay } from "raindrop-ai";
+
+const raindrop = new Raindrop({ apiKey: process.env.RAINDROP_QUERY_API_KEY });
+
+const result = await replay(raindrop, {
+  dataset: "support-cases",
+  run: (row) => supportAgent(row.input),
+  evaluators: ["concise", "task-completed"],
+});
+
+for (const row of result.rows) {
+  console.log(row.id, row.status);
+  for (const verdict of row.verdicts) {
+    console.log(verdict.evaluator, verdict);
+  }
+}
+```
+
+After the traces land, `replay` starts each requested evaluator against the
+replay and waits for its result. Each item in `result.rows` contains the
+original dataset row, its trace status, and one verdict per evaluator in
+`row.verdicts`. A verdict identifies its `evaluator` and contains either the
+grade (`pass`, `score`, or `value`), an evaluator error, or the reason it was
+not graded.
+
+`result.evaluators` contains aggregate status, summary, error, and URL for each
+evaluator. A failed evaluator is returned there instead of making the whole
+replay throw.
+
+An evaluator can run beside the agent when its verdict depends on state that is
+not in the trace. Keep the evaluator saved in Raindrop so its name and output
+type appear consistently, then provide its local `judge`:
+
+```ts
+const result = await replay(raindrop, {
+  dataset: "support-cases",
+  run: (row) => supportAgent(row.input),
+  evaluators: [
+    {
+      evaluator: "ticket-closed",
+      judge: async ({ row }) => {
+        const ticket = await testEnvironment.tickets.get(row.properties.ticketId);
+        return {
+          pass: ticket.status === "closed",
+          note: `final status: ${ticket.status}`,
+        };
+      },
+    },
+  ],
+});
+```
+
+The callback receives the dataset `row` and the value returned by `run`. It
+runs immediately after the row finishes, while the local environment is still
+available. Return `{ pass, note? }`, `{ score, note? }`, or `{ value, note? }`
+to match the saved evaluator's output type. A thrown error is stored as that
+row's errored verdict. Raindrop marks the resulting run as `local` in the UI.
+
+If rows mutate one shared test environment, set `concurrency: 1`. The default
+of four is appropriate when each row has isolated state.
+
+The row counts are computed on the server, so they match what the web screen
+shows. By default, evaluators are checked every two seconds for up to ten
+minutes. Use `evalPollIntervalMs` and `evalWaitMs` to change those waits.
+
+**A row that fails is `missing`, never dropped.** A callback that throws is
+reported with its error, and so is a row whose agent returned but whose spans
+never shipped, after `traceWaitMs` (30 seconds by default). A replay in which
+every row failed still finishes, with `missing` equal to the row count. A
+replay that quietly dropped rows would report a better number than it earned.
+
+**Concurrency** defaults to four rows at once. Each row keeps its own
+correlation id, so a span never lands on the wrong row.
+
+`createReplay` and `runReplay` are the trace-only halves if you want them
+separately. Call `evaluateReplay(raindrop, replayId, { evaluators })` after
+`runReplay` to apply the evaluators and wait for their aggregate results.
+
+A newly created replay also returns `ingestUrl`, the dedicated replay endpoint.
+`runReplay` binds it around each agent call automatically. Spans created by
+those calls use the client's Query SDK `apiKey` and carry the replay id and project in
+the `X-Raindrop-Replay-Id` and `X-Raindrop-Project-Id` headers. Dataset and eval control
+calls use that same organization API key. A `writeKey` is needed only if the client
+also sends ordinary production telemetry. Existing eval callers must migrate from
+`writeKey` to `apiKey`; write keys no longer authorize evals or datasets.
+
+Deploy backend support for SDK-key replay ingest before upgrading eval consumers.
+Older SDK eval clients must migrate when the backend changes. Production telemetry
+continues to use `writeKey`; eval requests do not fall back to it.
+
+**Running a replay somebody started in the dashboard.** A replay created there
+is queued with no runner, because only your process can call your agent.
+`claimReplay` picks one up and returns the same handle `createReplay` does, so
+it goes straight to `runReplay`:
+
+```ts
+import Raindrop, { claimReplay, runReplay } from "raindrop-ai";
+
+const handle = await claimReplay(raindrop, replayId);
+const result = await runReplay(raindrop, handle, {
+  run: (row) => supportAgent(row.input),
+});
+```
+
+The process that acquires the runner lease gets `ingestUrl`. A converged create
+retry or later claim gets `ingestUrl: null`, so a second process cannot run the
+same replay concurrently. `runReplay` refuses that handle before calling the
+agent.
+
+**One limit worth knowing.** In `useExternalOtel` mode you own the tracer
+provider, so `replay`'s own flush cannot reach your span processor. Flush your
+provider yourself, or keep its batch delay well under `traceWaitMs`.
+
 ## Detached sub-agents
 
 A **detached** sub-agent runs in another process (a queue worker, a container,