raindrop-ai 0.7.1 → 0.8.0

package/README.md

@@ -127,6 +127,74 @@ The Query SDK organization `apiKey` authenticates evals, dataset publication/rea
 
 Use the returned `runId` with `readEvalRun`, `evaluateEvalRun`, or `compareEvalRuns`. To execute a queued run, pass `{ runId }` to `runEvalSuite`. Use `traceTools.getOutput(trace)` and `traceTools.getToolCalls(trace, name)` in local evaluators.
 
+### Run eval agents on your server
+
+Protect the route with your existing authentication before calling `handleEval`. The handler does not authenticate incoming requests. Run it on a server that can await the entire execution. Do not return early from a serverless request.
+
+The simplest agent is a slug and a `run` function:
+
+```ts
+import { createEvalHandler, defineAgent } from "raindrop-ai";
+
+const support = defineAgent({
+	slug: "support",
+	run: (row) => supportAgent(row.input),
+});
+
+const handleEval = createEvalHandler(client, { agents: [support] });
+
+export async function route(request: Request): Promise<Response> {
+	if (!(await isAuthorized(request))) return new Response("Unauthorized", { status: 401 });
+	return handleEval(request);
+}
+```
+
+Declare `parameters` to vary the agent per run without redeploying. Each key is a `z.string()`, `z.number()`, `z.boolean()`, or `z.enum()` schema, optionally with `.default()`, `.optional()`, and `.describe()`. Raindrop reads the schema from `GET`, renders one form control per key, and saves each edit as a new version of a named parameter set, so runs pin an exact version. The handler validates incoming values and applies defaults before any row runs; a mismatch returns 400 with the schema error instead of claiming the run. `context.parameters` is typed from the declaration. Agents without `parameters` accept only `{}`.
+
+```ts
+import { z } from "zod";
+
+const support = defineAgent({
+	slug: "support",
+	parameters: {
+		model: z.enum(["gpt-4o", "gpt-4o-mini"]).default("gpt-4o"),
+		temperature: z.number().min(0).max(2).default(0.2),
+		systemPrompt: z.string().max(4000).describe("Prepended to every conversation"),
+	},
+	run: (row, { parameters }) => supportAgent(row.input, parameters),
+});
+```
+
+Add `setup` when rows need shared infrastructure. It runs once per execution with the run's parameters, every row receives its result as `environment`, and `cleanup` receives the same value once the rows settle, including after failures. `cleanup` does not run when `setup` itself throws. Give temporary infrastructure an expiry because a terminated process cannot execute cleanup. Concurrent rows share the environment, so isolate row state or use an agent that supports concurrent requests.
+
+```ts
+const sandboxed = defineAgent({
+	slug: "support-sandboxed",
+	name: "Support agent in a sandbox",
+	parameters: { image: z.string().default("support:latest") },
+	setup: (parameters) => createSandbox(parameters.image),
+	run: (row, { environment }) => supportAgent(row.input, environment),
+	cleanup: (environment) => environment.dispose(),
+});
+```
+
+The same agent runs locally through a suite. Set either `agent` or `run`, not both; `runEvalSuite` and `createEvalSuiteRun` accept `parameters` and start one execution per run.
+
+```ts
+const suite = defineEvalSuite({
+	name: "Support regression",
+	dataset: "support-examples",
+	agent: support,
+	evaluators: [{ evaluator: "task-completed" }],
+});
+
+await runEvalSuite(client, suite, { parameters: { model: "gpt-4o-mini", systemPrompt: "Be brief" } });
+```
+
+GET returns registered agent metadata and parameter fields. POST accepts `{ runId, agent, parameters }`. Raindrop creates the queued run and pins its dataset before dispatch. The handler claims the run with your client's Query SDK credentials, executes the agent over the claimed rows, and uploads its traces. Raindrop grades them separately with the selected hosted evaluators.
+
+The handler returns `{ runId, status, counts, traceEvidence }` after execution. Repeated requests for terminal runs return their existing status and counts without rerunning. A run claimed by another process returns 409. Execution failures return 500 with the thrown error's message in `error`, which Raindrop shows on the run. `queryUrl` and `replayIngestUrl` are server configuration, never request fields.
+
 <details>
 <summary>Legacy replay APIs and tracing internals</summary>
 
@@ -254,7 +322,7 @@ minutes. Use `evalPollIntervalMs` and `evalWaitMs` to change those waits.
 
 **A row that fails is `missing`, never dropped.** A callback that throws is
 reported with its error, and so is a row whose agent returned but whose spans
-never shipped, after `traceWaitMs` (30 seconds by default). A replay in which
+never shipped, after `traceWaitMs` (60 seconds by default). A replay in which
 every row failed still finishes, with `missing` equal to the row count. A
 replay that quietly dropped rows would report a better number than it earned.