@tangle-network/sandbox 0.0.0-develop.20260519174708.e008250 → 0.0.0-develop.20260530024439.0bf13d0

package/README.md

@@ -43,6 +43,77 @@ console.log(task.response);
 await box.delete();
 ```
 
+## Stream durability is platform-managed — do not build your own
+
+> **If you are about to add a Cloudflare Durable Object, a KV bucket, an in-Worker
+> ring buffer, or any other state store to "buffer agent stream events so the user
+> survives a reload" — stop.** The Tangle orchestrator already buffers every event
+> for every session to a Redis sorted-set keyed by `sessionId` with TTL, and the
+> SDK ships a browser/Worker-safe client that reconnects, replays missed events,
+> and persists `lastEventId` across tab reloads. The work is done. Use it.
+
+The decision tree:
+
+| You need… | Use |
+|---|---|
+| Fire-and-forget streaming from a server you control (CLI, cron, batch worker) | `box.streamPrompt()` / `box.streamTask()` — internal auto-reconnect handles transient drops within the same call |
+| Survive **client process death** (Worker isolate eviction, laptop crash, deploy) and resume later | `box.dispatchPrompt(msg, { sessionId })` then `box.session(sessionId).events({ since })` / `.result()` from a fresh process |
+| Survive **browser** disconnects (wifi flap, tab reload, mobile background) with `Last-Event-ID` replay | `SessionGatewayClient` from `@tangle-network/sandbox/session-gateway` |
+| Retry a payment-triggered or webhook-triggered run safely | `box.dispatchPrompt(msg, { sessionId: deterministicKeyFromRequest })` — same `sessionId` is idempotent: a duplicate dispatch returns the in-flight or completed session, never re-executes |
+| Inspect what happened to a turn that died mid-stream | `box.session(id).status()` (terminal state), `box.session(id).events({ since })` (replay buffered events) |
+
+### Dispatch + reconnect (the "Worker restart" pattern)
+
+```typescript
+import { Sandbox } from "@tangle-network/sandbox";
+
+const client = new Sandbox({ apiKey: process.env.TANGLE_API_KEY! });
+
+// In your /chat handler:
+const sessionId = req.headers.get("x-turn-id") ?? crypto.randomUUID();
+const { sessionId: id, alreadyExisted } = await box.dispatchPrompt(prompt, {
+  sessionId, // idempotent: a retry with the same id is a lookup, not a re-execute
+});
+
+// Stream to the browser; if the client comes back later with the same sessionId
+// and a Last-Event-ID, hand them the replay path below.
+for await (const event of box.session(id).events({
+  since: req.headers.get("last-event-id") ?? undefined,
+})) {
+  res.write(`id: ${event.id}\ndata: ${JSON.stringify(event)}\n\n`);
+}
+```
+
+### Browser-direct streaming (without proxying tokens through your server)
+
+```typescript
+import { SessionGatewayClient } from "@tangle-network/sandbox/session-gateway";
+
+const client = new SessionGatewayClient({
+  url: "wss://your-sandbox-api.example.com/session",
+  token: await fetchScopedToken(),   // mint via box.mintScopedToken({ scope: 'session', sessionId })
+  sessionId,
+  autoReconnect: true,
+  enableReplayPersistence: true,     // remembers lastEventId across reloads
+  replayStorage: { /* localStorage adapter, see session-gateway docs */ },
+  handlers: {
+    onMessage: (event) => render(event),
+    onReplayStart: ({ since }) => showSpinner(`replaying from ${since}`),
+    onReplayComplete: () => hideSpinner(),
+    onBackpressureWarning: ({ droppedCount, suggestReplay }) =>
+      suggestReplay && client.replay(client.stats.replay.lastEventId),
+  },
+});
+client.connect();
+```
+
+`SessionGatewayClient` handles auto-reconnect with exponential backoff, sequence-gap
+detection, replay-on-reconnect, and `lastEventId` persistence. None of this requires a
+Durable Object.
+
+See `examples/cf-worker-chat.ts`, `examples/browser-streaming-resume.ts`, and
+`examples/reconnect-from-last-event-id.ts` for end-to-end patterns.
+
 ## Features
 
 - **Sandbox Management** - Create, list, stop, resume, and delete sandboxes
@@ -50,10 +121,11 @@ await box.delete();
 - **AI Agent Tasks** - Multi-turn agent execution with automatic tool use
 - **Snapshots** - Save and restore sandbox state
 - **BYOS3** - Bring your own S3 storage for snapshots
-- **Batch Execution** - Run tasks across multiple sandboxes in parallel
+- **Fleets** - Coordinated multi-machine workloads with policy, workspace snapshots, and parallel dispatch
+- **Intelligence Reports** - Deterministic or agentic post-hoc analysis of sandbox or fleet evidence (fleet reports can refine to a single dispatch via `subject.dispatchId`)
 - **Event Streaming** - Real-time SSE streams for agent events
 - **Collaboration Foundations** - Token issuance and document identity helpers for collaborative editing
-- **Trace Intelligence** - Export raw sandbox and fleet traces, derived insights, timing metrics, and OTEL JSON to customer-owned observability systems
+- **Trace Intelligence** - Export raw sandbox and fleet traces, embedded intelligence envelope, timing metrics, and OTEL JSON to customer-owned observability systems
 
 ## Collaboration Foundations
 
@@ -113,31 +185,41 @@ The bridge and client are SDK-side primitives. Product/backend endpoints and CRD
 
 ## Trace Intelligence
 
-Sandboxes and fleet runs return raw trace evidence by default. The derived intelligence envelope is opt-in unless you call the explicit intelligence helper. The raw trace is the source of truth; the intelligence envelope is the platform's current analysis over that evidence.
+The platform exposes two distinct intelligence primitives. Use the right one for the job.
+
+| Primitive | What you get | Billable | API call |
+|---|---|---|---|
+| **Embedded envelope** | Inline summary in a trace/dispatch response: signals, recommended actions, fanout timings, dispatch failure classes | No (`billing.billable: false`) | `trace({ includeIntelligence: true })`, `intelligence()`, dispatch responses |
+| **Intelligence Report** | A pollable job over sandbox or fleet evidence; fleet reports can refine to a single dispatch via `subject.dispatchId`. Runs in `deterministic` or `agentic` mode against a budget. | `deterministic`: free. `agentic`: billed against `budget.maxUsd` | `intelligence.createReport(...)`, see [Intelligence Reports](#intelligence-reports) |
+
+### Embedded envelope (free)
+
+The embedded envelope is opt-in on `trace()` calls (`includeIntelligence: true`) and always included on dispatch responses (because dispatch already pays the analysis cost as part of producing the result).
 
 ```typescript
-const boxBundle = await box.trace();              // { trace }
-const boxInsights = await box.intelligence();     // intelligence only
+const boxBundle = await box.trace();                              // { trace } only
+const boxInsights = await box.intelligence();                     // envelope only
 const boxBundleWithInsights = await box.trace({ includeIntelligence: true });
 
-const run = await fleet.dispatchExecWithInsights("pytest -q", {
+const run = await fleet.dispatchExecDetailed("pytest -q", {
   machines: ["worker-1", "worker-2"],
 });
-
-console.log(run.trace.events);
-console.log(run.intelligence.signals);
+console.log(run.results);
+console.log(run.intelligence.signals);            // always present on dispatch
 console.log(run.intelligence.recommendedActions);
 
-const bundle = await fleet.trace();           // { trace }
-const insights = await fleet.intelligence();  // intelligence only
-const bundleWithInsights = await fleet.trace({ includeIntelligence: true });
+const fleetBundle = await fleet.trace();                          // { trace } only
+const fleetInsights = await fleet.intelligence();                 // envelope only
+const fleetBundleWithInsights = await fleet.trace({ includeIntelligence: true });
 ```
 
-Sandbox traces cover lifecycle, runtime, usage, timing, and current health snapshots. Fleet traces add machine lifecycle, workspace state, dispatch results, fanout timings, and critical path. The intelligence envelope tells you what to inspect next: reliability, parallelism efficiency for fleets, dispatch failure classes, resource attribution, timing bottlenecks, and recommended actions.
+Sandbox traces cover lifecycle, runtime, usage, timing, and current health snapshots. Fleet traces add machine lifecycle, workspace state, dispatch results, fanout timings, and critical path. The embedded envelope tells you what to inspect next: reliability, parallelism efficiency for fleets, dispatch failure classes, resource attribution, timing bottlenecks, and recommended actions.
 
-The built-in intelligence in this SDK is deterministic platform analysis. It reports `billing.billable: false` and `billing.costUsd: 0`; customers are not charged for generating these envelopes. Products opt into generated intelligence per trace request with `includeIntelligence: true` or by calling `intelligence()`.
+The embedded envelope is deterministic platform analysis. It reports `billing.billable: false` and `billing.costUsd: 0`; customers are not charged for generating these envelopes.
 
-Customers can keep the complete raw payload or forward it to their own observability stack. Use `format: "tangle"` to preserve the native envelope, or `format: "otel-json"` for OpenTelemetry-style collectors and platforms that accept OTLP JSON.
+### Exporting traces to your observability stack
+
+Use `format: "tangle"` to preserve the native envelope, or `format: "otel-json"` for OpenTelemetry-style collectors and platforms that accept OTLP JSON.
 
 ```typescript
 await box.exportTrace({
@@ -278,7 +360,7 @@ console.log(`Compute: ${usage.computeMinutes} minutes`);
 
 #### `client.runBatch(tasks, options?)`
 
-Run tasks across multiple sandboxes in parallel.
+Run ad-hoc tasks across freshly-provisioned sandboxes in parallel. Use this when the work is one-shot and the sandboxes do not need to share a workspace or be addressable by stable `machineId`. For coordinated multi-machine workloads with policy enforcement, workspace sharing, dispatch buffering, and intelligence reports, see [Fleets](#fleets).
 
 ```typescript
 const result = await client.runBatch([
@@ -293,6 +375,14 @@ const result = await client.runBatch([
 console.log(`Success rate: ${result.successRate}%`);
 ```
 
+#### `client.fleets`
+
+Fleet client — see [Fleets](#fleets) for the full surface (`create`, `createWithCoordinator`, `list`, `delete`, `estimateCost`, `capabilities`, `operations`, `reconcile`, `reapExpired`).
+
+#### `client.intelligence`
+
+Intelligence report client — see [Intelligence Reports](#intelligence-reports) for the full surface (`createReport`, `createAgenticReport`, `getReport`, `listReports`, `waitForReport`).
+
 ### Sandbox Instance
 
 After creating or retrieving a sandbox, you get a `SandboxInstance` with these methods:
@@ -677,6 +767,317 @@ box.expiresAt       // Date | undefined
 box.error           // Error message if failed
 ```
 
+## Fleets
+
+A **fleet** is a coordinated group of sandboxes that runs one logical workload across many machines. Fleets are the canonical primitive for parallel work, distributed simulation, multi-agent experiments, or any task that needs more than one sandbox under one lifecycle.
+
+Fleets give you:
+
+- A single id (`fleetId`) plus a stable machine id (`machineId`) per member that you choose
+- Policy enforcement (CPU / memory / storage / accelerator caps, allowed drivers / images / templates, max spend, max concurrent creates) — checked client-side **before** sandboxes are provisioned
+- Per-dispatch parallelism, retries, timeouts, idempotency, cancellation, and result buffering
+- Shared workspace modes (`isolated`, `shared`) with cross-machine snapshot, restore, and reconcile
+- Fleet-scoped telemetry: usage, cost estimate, trace bundle, embedded intelligence envelope, and full intelligence reports
+- Scoped tokens for handing a fleet to a downstream service without leaking the parent API key
+
+### Create a fleet
+
+There are two creation shapes. Use `create` when every machine is symmetric, and `createWithCoordinator` when one machine acts as orchestrator over a pool of workers.
+
+```typescript
+// Symmetric fleet
+const fleet = await client.fleets.create({
+  defaults: {
+    image: "python:3.12",
+    maxLifetimeSeconds: 60 * 60,
+  },
+  policy: {
+    maxMachines: 4,
+    maxConcurrentCreates: 2,
+    maxTotalCpu: 8,
+    maxTotalMemoryMb: 16_384,
+    maxSpendUsd: 5,
+    allowAccelerators: false,
+  },
+  machines: [
+    { machineId: "worker-1", resources: { cpuCores: 2, memoryMB: 4096 } },
+    { machineId: "worker-2", resources: { cpuCores: 2, memoryMB: 4096 } },
+  ],
+  workspace: { mode: "isolated" },
+  metadata: { experiment: "react-19-bump" },
+  idempotencyKey: "exp-react-19-2025-05-18",
+});
+
+// Coordinator + workers
+const cluster = await client.fleets.createWithCoordinator({
+  defaults: { image: "python:3.12" },
+  coordinator: { resources: { cpuCores: 1, memoryMB: 1024 } },
+  workers: [
+    { machineId: "worker-1", resources: { cpuCores: 2, memoryMB: 4096 } },
+    { machineId: "worker-2", resources: { cpuCores: 2, memoryMB: 4096 } },
+  ],
+});
+```
+
+`createWithCoordinator` is sugar over `create`: it injects a `coordinator` machine with `role: "coordinator"` and tags the workers `role: "worker"` in metadata. After creation both shapes return a `SandboxFleet` instance.
+
+### Dispatch across machines
+
+```typescript
+// Fire and collect — returns FleetExecDispatchResult[]
+const results = await fleet.dispatchExec("pytest -q", {
+  machines: fleet.ids,           // default: every machine
+  maxConcurrent: 2,
+  timeoutMs: 60_000,
+  retry: { attempts: 2, backoffMs: 1_000 },
+  dispatchId: "pytest-run-1",    // idempotent — same id replays the same dispatch
+});
+
+// Detailed — returns the full response including dispatchId, durationMs,
+// trace, and the embedded intelligence envelope (always present on dispatch).
+const detailed = await fleet.dispatchExecDetailed("pytest -q");
+console.log(detailed.intelligence.signals);
+
+// Prompt variant — runs an agent prompt on each selected machine
+const promptResults = await fleet.dispatchPrompt(
+  "Summarize what changed in this branch and why.",
+  { machines: ["worker-1"], model: "anthropic/claude-sonnet-4-20250514" },
+);
+
+// Stream events as they happen
+for await (const event of fleet.dispatchExecStream("npm test", {
+  machines: fleet.ids,
+})) {
+  if (event.type === "result") console.log(event.data);
+}
+
+// Read previously-buffered results by dispatchId
+const buffered = await fleet.dispatchResults("pytest-run-1", {
+  limit: 100,
+  machines: ["worker-1", "worker-2"],
+});
+
+// Cancel an in-flight dispatch
+await fleet.cancelDispatch("pytest-run-1", "abandoning experiment");
+```
+
+### Single-machine helpers
+
+When you want to talk to one machine in the fleet directly:
+
+```typescript
+const result = await fleet.exec("worker-1", "echo hello");
+const reply  = await fleet.prompt("worker-1", "What is the structure?");
+const box    = await fleet.sandbox("worker-1");      // full SandboxInstance
+```
+
+### Workspace snapshots (shared workspace mode)
+
+```typescript
+const snap = await fleet.createWorkspaceSnapshot();
+await fleet.restoreWorkspaceSnapshot(snap.snapshotId);
+await fleet.reconcileWorkspace();
+```
+
+### Dynamic topology
+
+```typescript
+await fleet.attachMachine({
+  machineId: "worker-3",
+  sandboxId: "sbx_abc",   // existing sandbox to bind into the fleet
+  role: "worker",
+});
+await fleet.detachMachine("worker-3");
+```
+
+### Artifacts, usage, cost, tokens
+
+```typescript
+const artifacts = await fleet.collectArtifacts([
+  { machineId: "worker-1", path: "/workspace/report.json", maxBytes: 1_000_000 },
+]);
+
+const usage    = await fleet.usage();             // current usage rollup
+const estimate = await fleet.cost();              // cost estimate
+const manifest = await fleet.manifest();          // machine manifest as persisted
+
+// Scoped token — hand to a downstream service without leaking the parent key
+const token = await fleet.createToken({ expiresInSeconds: 3600 });
+```
+
+### Pre-flight cost estimate (without creating)
+
+```typescript
+const preEstimate = await client.fleets.estimateCost({
+  defaults: { image: "python:3.12" },
+  policy: { maxMachines: 4, maxSpendUsd: 5 },
+  machines: [
+    { machineId: "worker-1", resources: { cpuCores: 2, memoryMB: 4096 } },
+    { machineId: "worker-2", resources: { cpuCores: 2, memoryMB: 4096 } },
+  ],
+});
+```
+
+### Operations
+
+```typescript
+const caps  = await client.fleets.capabilities();   // which drivers / templates / images
+const ops   = await client.fleets.operations();     // operations summary
+const recon = await client.fleets.reconcile();      // reconcile drift
+const reaped = await client.fleets.reapExpired();   // sweep expired fleets
+```
+
+### Lookup and delete
+
+```typescript
+const found = await client.fleets.list({ fleetId: "fleet_abc" });
+await client.fleets.delete("fleet_abc", { continueOnError: true });
+```
+
+### Single-shot batches without fleets
+
+For one-shot, ad-hoc parallel work that does not need fleet-level policy / workspaces / dispatch buffering / intelligence reports, `client.runBatch(tasks, options)` is the simpler primitive. New code that needs more than one sandbox under one logical lifecycle should reach for fleets.
+
+## Intelligence Reports
+
+The **Intelligence Reports** API generates structured post-hoc analyses over two subject types: a single **sandbox** or a single **fleet**. A fleet subject can optionally be narrowed to one dispatch within the fleet via `subject.dispatchId`. Two modes:
+
+- **`deterministic`** (default) — platform-side rule-based analysis. Free. Returns immediately or near-immediately. Surfaces lifecycle, runtime, plan-headroom, and resource-density signals derived directly from your trace evidence.
+- **`agentic`** — runs the **Tangle Trace Analyst**, an LLM-driven reasoning loop, over your OTLP trace evidence. Returns findings with evidence references, recommended actions, and a validation plan. **Billed** against `budget.maxUsd`; the platform never spends past the budget you set. Async (returns a job; poll for terminal state).
+
+The dedicated client lives on `client.intelligence`:
+
+```typescript
+import { Sandbox } from "@tangle-network/sandbox";
+import type { IntelligenceClient } from "@tangle-network/sandbox";
+
+const client = new Sandbox({ apiKey, baseUrl });
+const intelligence: IntelligenceClient = client.intelligence;
+```
+
+### Create a report
+
+```typescript
+// Deterministic — over a single sandbox
+const det = await client.intelligence.createReport({
+  subject: { type: "sandbox", id: box.id },
+});
+
+// Deterministic — over a fleet
+await client.intelligence.createReport({
+  subject: { type: "fleet", id: fleet.fleetId },
+});
+
+// Deterministic — narrowed to one dispatch within a fleet
+await client.intelligence.createReport({
+  subject: {
+    type: "fleet",
+    id: fleet.fleetId,
+    dispatchId: "pytest-run-1",
+  },
+});
+
+// Agentic — billed against the budget
+const agentic = await client.intelligence.createReport({
+  subject: { type: "fleet", id: fleet.fleetId },
+  mode: "agentic",
+  budget: { billTo: "customer", maxUsd: 5 },
+  acknowledgeCost: true,
+  metadata: { experiment: "react-19-bump" },
+});
+
+// Shorthand for the agentic + budget pattern
+const shortcut = await client.intelligence.createAgenticReport({
+  subject: { type: "sandbox", id: box.id },
+  maxUsd: 2,
+});
+```
+
+### Poll a report to completion
+
+Agentic reports return `status: "pending"` immediately. Either poll yourself with `getReport`, or use the built-in `waitForReport`:
+
+```typescript
+const job = await client.intelligence.createAgenticReport({
+  subject: { type: "fleet", id: fleet.fleetId },
+  maxUsd: 5,
+});
+
+const completed = await client.intelligence.waitForReport(job.jobId, {
+  timeoutMs: 5 * 60 * 1000,
+  pollMs: 2_000,
+});
+
+if (completed.status === "completed") {
+  console.log(completed.findings);
+  console.log(completed.recommendedActions);
+}
+```
+
+### List existing reports
+
+```typescript
+const recent = await client.intelligence.listReports({
+  subjectType: "fleet",
+  subjectId: fleet.fleetId,
+  limit: 20,
+});
+```
+
+### Per-subject shortcuts
+
+`SandboxInstance` and `SandboxFleet` expose convenience wrappers so you don't have to thread `subject` manually:
+
+```typescript
+await box.createIntelligenceReport({ mode: "deterministic" });
+await box.createAgenticIntelligenceReport({ maxUsd: 2 });
+
+await fleet.createIntelligenceReport({ mode: "deterministic" });
+await fleet.createAgenticIntelligenceReport({ maxUsd: 5 });
+
+// Fleet helpers accept the v2 refinement fields directly:
+await fleet.createIntelligenceReport({
+  mode: "deterministic",
+  dispatchId: "pytest-run-1",
+});
+```
+
+Both wrappers post to `POST /v1/intelligence/reports` with the right `subject` filled in.
+
+### Time windows and baselines
+
+Every report can be narrowed by a time window and compared against a same-type baseline. The analyzer rejects mixed-type comparisons because the delta would be meaningless.
+
+```typescript
+// Bound the analysis to a one-hour window.
+await fleet.createIntelligenceReport({
+  window: { since: Date.now() - 60 * 60 * 1000 },
+});
+
+// Compare two dispatches of the same fleet against each other.
+await fleet.createIntelligenceReport({
+  dispatchId: "run-after",
+  compareTo: { type: "fleet", id: fleet.fleetId, dispatchId: "run-before" },
+});
+
+// Sandbox baseline.
+await box.createIntelligenceReport({
+  compareTo: { type: "sandbox", id: previousBox.id },
+});
+```
+
+### Cost before commit
+
+Estimate cost without creating a report. Subject ownership is verified the same way as `createReport`, so the endpoint never becomes an existence oracle for foreign subjects.
+
+```typescript
+const estimate = await client.intelligence.estimateReport({
+  subject: { type: "fleet", id: fleet.fleetId, dispatchId: "pytest-run-1" },
+  mode: "agentic",
+});
+console.log(`Would cost ${estimate.costUsd} USD (${estimate.reason})`);
+```
+
 ## Error Handling
 
 ```typescript
@@ -728,6 +1129,38 @@ import type {
   BatchResult,
   BatchOptions,
   UsageInfo,
+  // Fleets
+  CreateSandboxFleetOptions,
+  CreateSandboxFleetWithCoordinatorOptions,
+  SandboxFleetMachineSpec,
+  SandboxFleetInfo,
+  SandboxFleetManifest,
+  SandboxFleetUsage,
+  SandboxFleetCostEstimate,
+  SandboxFleetToken,
+  SandboxFleetTraceBundle,
+  SandboxFleetTraceOptions,
+  SandboxFleetDispatchResponse,
+  FleetExecDispatchOptions,
+  FleetExecDispatchResult,
+  FleetPromptDispatchOptions,
+  FleetPromptDispatchResult,
+  FleetDispatchResultBuffer,
+  FleetDispatchResultBufferOptions,
+  FleetDispatchStreamOptions,
+  FleetDispatchCancelResult,
+  FleetMachineId,
+  // Intelligence Reports
+  IntelligenceReport,
+  IntelligenceReportBudget,
+  CreateIntelligenceReportOptions,
+} from "@tangle-network/sandbox";
+
+// Concrete classes — useful when you need to reference the type itself
+import {
+  IntelligenceClient,
+  SandboxFleet,
+  SandboxFleetClient,
 } from "@tangle-network/sandbox";
 ```