raindrop-ai 0.6.0 → 0.7.0-otelv2

package/README.md

@@ -81,25 +81,32 @@ const client = new Raindrop({
 
 const suite = defineEvalSuite({
   name: "Support responses",
-  dataset: "support-cases",
+  dataset: "support-examples",
   run: ({ input }) => supportAgent(input ?? ""),
-  evaluators: [{ evaluator: "task-completed", threshold: { equals: true } }],
+  evaluators: [{ evaluator: "task-completed" }],
 });
 
 try {
-  const result = await runEvalSuite(client, suite, {
-    destination: { kind: "raindrop" },
-  });
+  const result = await runEvalSuite(client, suite);
   console.log(result);
 } finally {
   await client.close();
 }
 ```
 
-The dataset string names a published, project-scoped eval dataset, not an ordinary custom-upload dataset ID. Publish its immutable version with `publishEvalDataset`. Cases can contain only input and scenario properties; an optional reference must be a complete captured trace. Local evaluators can declare `requiresReference: true`, and hosted programs can export `const requiresReference = true`, to reject missing references before agent execution. A local `defineDataset` object does not publish itself and is not accepted by the hosted runner. See the [Vitest guide](../vitest/README.md) for publication, evaluator requirements, and credential setup.
+The dataset string names a published, project-scoped eval dataset, not an ordinary custom-upload dataset ID. You can also pass a local `defineDataset` object with rows containing just `id` and `input`. `runEvalSuite` and `evalTests` upload local dataset rows and resolve existing hosted evaluators automatically. Create hosted evaluators in Raindrop and reference their slugs; evaluator source is not uploaded. Dataset versions and evaluator scopes are inferred; Raindrop is the default destination. Existing version pins and update-conflict protections remain in effect.
+
+Use `publishEvalDataset` when supplying captured reference traces. Local evaluators can declare `requiresReference: true`, and hosted programs can export `const requiresReference = true`, to reject missing references before agent execution. See the [author/run/compare flow](./EVALS.md) and [Vitest guide](../vitest/README.md).
 
 The Query SDK organization `apiKey` authenticates evals, dataset publication/reads, and replay trace uploads. Browser login and production `writeKey` credentials do not replace it. The agent's model credentials and test environment are separate. Running locally describes where your callback executes, not where results are stored.
 
+Use the returned `runId` with `readEvalRun`, `evaluateEvalRun`, or `compareEvalRuns`. To execute a queued run, pass `{ runId }` to `runEvalSuite`. Use `traceTools.getOutput(trace)` and `traceTools.getToolCalls(trace, name)` in local evaluators.
+
+<details>
+<summary>Legacy replay APIs and tracing internals</summary>
+
+New integrations should use the suite functions above. The following lower-level APIs remain available for existing integrations and custom workers.
+
 ### Trace correlation
 
 A replay calls your agent once per row and needs every span and event that call
@@ -160,7 +167,7 @@ import Raindrop, { replay } from "raindrop-ai";
 const raindrop = new Raindrop({ apiKey: process.env.RAINDROP_QUERY_API_KEY });
 
 const result = await replay(raindrop, {
-  dataset: "support-cases",
+  dataset: "support-examples",
   run: (row) => supportAgent(row.input),
   evaluators: ["concise", "task-completed"],
 });
@@ -190,7 +197,7 @@ type appear consistently, then provide its local `judge`:
 
 ```ts
 const result = await replay(raindrop, {
-  dataset: "support-cases",
+  dataset: "support-examples",
   run: (row) => supportAgent(row.input),
   evaluators: [
     {
@@ -268,6 +275,8 @@ agent.
 provider, so `replay`'s own flush cannot reach your span processor. Flush your
 provider yourself, or keep its batch delay well under `traceWaitMs`.
 
+</details>
+
 ## Detached sub-agents
 
 A **detached** sub-agent runs in another process (a queue worker, a container,