la-machina-engine 0.7.0 → 0.7.2

package/README.md

@@ -694,7 +694,9 @@ Shape:
 
 ### Webhooks
 
-Pass a `webhook` object to `start()` / `resumeAsync()` and the engine will POST the final `EngineResponse` to your URL on the configured events.
+Async runs deliver status changes to a URL you own. Pass a `webhook`
+object to `start()` / `resumeAsync()` and the engine POSTs the final
+`EngineResponse` whenever the run reaches a terminal or pause state.
 
 ```ts
 await engine.start({
@@ -705,46 +707,285 @@ await engine.start({
     url: 'https://your-app.com/hooks/la-machina',
     secret: 'shared-hmac-secret',           // optional — enables X-LaMachina-Signature
     events: ['paused', 'done', 'failed'],  // default: all three
-    headers: { 'X-Tenant': 'acme' },       // optional — passed through
+    headers: { 'X-Tenant': 'acme' },       // optional — passed through per request
   },
 })
 ```
 
-**Request headers:**
+#### Events — what fires and when
 
-| Header | Value |
-|---|---|
-| `Content-Type` | `application/json` |
-| `X-LaMachina-Event` | `status.paused` \| `status.done` \| `status.failed` |
-| `X-LaMachina-RunId` | Run ID from your `start()` call |
-| `X-LaMachina-Delivery` | Unique UUID per delivery attempt |
-| `X-LaMachina-Timestamp` | Unix ms (used in HMAC input) |
-| `X-LaMachina-Signature` | `sha256=<hex>` — HMAC over `${timestamp}.${body}` (only if `secret` set) |
+Three events, mapped 1:1 from `EngineResponse.status`:
+
+| Event | Fires when | `data` field | `meta.pauseReason` |
+|---|---|---|---|
+| `done` | Model reached `end_turn` cleanly | Output string (or parsed JSON in structured-output mode) | — |
+| `paused` | Gate callback returned `{ allow: false }`, OR run needs runner handoff | `null` | `gate_required` \| `handoff_to_runner` |
+| `failed` | Anything threw (API 5xx after retries, max turns, timeout, cancel, runner unreachable) | `null` | — (`errors[0]` has the cause) |
+
+`queued`, `running`, and `not_found` **never fire webhooks** — they're
+only observable via `getStatus()` polling. Webhooks are terminal /
+pausal only.
+
+#### When webhooks do vs don't fire
+
+| API call | Webhooks? | Why |
+|---|---|---|
+| `engine.start({webhook})` | ✓ fires on terminal/pause | |
+| `engine.resumeAsync({webhook})` | ✓ fires on terminal/pause | |
+| `engine.run()` | **never** | Caller already has the response in hand |
+| `engine.resume()` | **never** | Same — synchronous, caller holds the result |
+| `engine.cancelRun(runId)` | in-flight run aborts and fires `failed` | Cancellation is a normal failure path |
+
+Webhooks are for the async surface exclusively. Anything running
+synchronously returns its response directly.
+
+#### Request shape
+
+`POST {webhook.url}` with body = `JSON.stringify(EngineResponse)` and:
+
+| Header | Value | Notes |
+|---|---|---|
+| `Content-Type` | `application/json` | |
+| `X-LaMachina-Event` | `status.done` \| `status.paused` \| `status.failed` | Event-type routing on the receiver |
+| `X-LaMachina-RunId` | Run ID from your `start()` call | Correlate with client-side state |
+| `X-LaMachina-Delivery` | Fresh UUID per attempt | **Use this for idempotency** — same delivery ID = retry of same logical event |
+| `X-LaMachina-Timestamp` | Unix ms | Covered by the HMAC — lets receivers reject replays |
+| `X-LaMachina-Signature` | `sha256=<hex>` | Only when `secret` is set; see "Verifying the signature" below |
+| _(user headers)_ | whatever you passed in `webhook.headers` | Merged last, cannot override engine headers |
+
+Request timeout is 30 s by default. The engine aborts slower receivers
+and treats them as a retryable network failure.
+
+#### Payload — one schema for every event
+
+The body is always an `EngineResponse` (the same shape `engine.run()`
+returns). The event type determines which fields are meaningful:
+
+**`done` payload:**
+
+```jsonc
+{
+  "runId": "run_abc",
+  "status": "done",
+  "data": "The analysis is complete. Revenue grew 15% YoY.",
+  "meta": {
+    "nodeId": "analyze",
+    "turns": 5,
+    "tokensUsed": { "input": 12500, "output": 3200, "cacheReadInput": 8000 },
+    "durationMs": 8500,
+    "output": "The analysis is complete. Revenue grew 15% YoY.",
+    "transcript": { "path": "projects/run_abc/nodes/analyze", "lastShardIndex": 2 }
+  },
+  "errors": [],
+  "timestamp": 1712966400000
+}
+```
+
+**`paused` payload:**
+
+```jsonc
+{
+  "runId": "run_abc",
+  "status": "paused",
+  "data": null,
+  "meta": {
+    "nodeId": "publish",
+    "pauseReason": "gate_required",
+    "turns": 3,
+    "tokensUsed": { "input": 8200, "output": 1900 },
+    "pendingToolCall": {
+      "toolName": "Publish",
+      "toolUseId": "toolu_01abc",
+      "input": { "post": { "title": "...", "body": "..." } }
+    },
+    "transcript": { "path": "projects/run_abc/nodes/publish", "lastShardIndex": 1 }
+  },
+  "errors": [],
+  "timestamp": 1712966400000
+}
+```
+
+Use `pendingToolCall.input` to render an approval UI, then call
+`engine.resumeAsync({ runId, gateAnswer: 'approve', webhook: {...} })`
+to continue. A separate `done` (or `failed`) webhook will fire for the
+resumed run.
+
+**`failed` payload:**
+
+```jsonc
+{
+  "runId": "run_abc",
+  "status": "failed",
+  "data": null,
+  "meta": {
+    "nodeId": "n1",
+    "cancelled": true                    // present when the failure was engine.cancelRun()
+  },
+  "errors": [
+    { "code": "CANCELLED", "message": "Run was cancelled by client" }
+    // Other codes: RUN_FAILED, RESUME_FAILED, ERR_RUNNER_UNREACHABLE, ERR_MAX_TURNS, ORPHANED, …
+  ],
+  "timestamp": 1712966400000
+}
+```
+
+The `errors[]` array holds `{code, message}` pairs — use `errors[0].code`
+for programmatic routing, `message` for display.
+
+#### Verifying the signature
+
+When `webhook.secret` is set, the engine signs
+`${X-LaMachina-Timestamp}.${body}` with HMAC-SHA256 and sets
+`X-LaMachina-Signature: sha256=<hex>`. Verify in Node:
+
+```ts
+import { createHmac, timingSafeEqual } from 'node:crypto'
+
+function verifyLaMachinaWebhook(req: Request, rawBody: string, secret: string): boolean {
+  const ts = req.headers.get('x-lamachina-timestamp')
+  const sig = req.headers.get('x-lamachina-signature')
+  if (!ts || !sig) return false
+
+  // Reject replays older than 5 minutes
+  if (Math.abs(Date.now() - Number(ts)) > 5 * 60_000) return false
+
+  const expected =
+    'sha256=' +
+    createHmac('sha256', secret).update(`${ts}.${rawBody}`).digest('hex')
+
+  // Constant-time comparison
+  const a = Buffer.from(sig)
+  const b = Buffer.from(expected)
+  return a.length === b.length && timingSafeEqual(a, b)
+}
+```
+
+On Cloudflare Workers (Web Crypto, no `node:crypto`):
+
+```ts
+async function verifyLaMachinaWebhook(req: Request, rawBody: string, secret: string) {
+  const ts = req.headers.get('x-lamachina-timestamp')
+  const sig = req.headers.get('x-lamachina-signature')
+  if (!ts || !sig) return false
+  if (Math.abs(Date.now() - Number(ts)) > 5 * 60_000) return false
+
+  const key = await crypto.subtle.importKey(
+    'raw',
+    new TextEncoder().encode(secret),
+    { name: 'HMAC', hash: 'SHA-256' },
+    false,
+    ['sign'],
+  )
+  const buf = await crypto.subtle.sign('HMAC', key, new TextEncoder().encode(`${ts}.${rawBody}`))
+  const expected =
+    'sha256=' +
+    Array.from(new Uint8Array(buf))
+      .map((b) => b.toString(16).padStart(2, '0'))
+      .join('')
+  return expected === sig
+}
+```
+
+**Always verify against the raw bytes** you read from the request.
+Re-serializing the parsed JSON will produce different bytes and the
+signature won't match.
+
+#### Idempotency — receivers MUST handle duplicates
+
+`X-LaMachina-Delivery` is unique per attempt, but retries of the same
+logical event may send the same payload to your endpoint multiple
+times (network flaps, receiver returns 5xx, etc.). De-duplicate on:
+
+- `X-LaMachina-Delivery` — reject second delivery with the same ID
+- OR `runId + status + timestamp` — simpler, event-level dedup
+
+Pattern: insert the delivery ID into a short-TTL cache (Redis, R2, DB
+unique constraint); on collision return 200 without reprocessing.
 
-**Retry schedule** (exponential-ish):
+#### Retry schedule
+
+Fixed schedule per delivery attempt:
 
 ```
 attempt 1: immediate
-attempt 2: +10s
-attempt 3: +60s
-attempt 4: +5min
-attempt 5: +30min
-then give up
+attempt 2: +10 s  (after the previous attempt's failure)
+attempt 3: +60 s
+attempt 4: +5 min
+attempt 5: +30 min
+→ give up
 ```
 
 Retry decisions:
 
-| HTTP | Retry? |
+| Receiver response | Retry? |
 |---|---|
-| 2xx | No (delivered) |
+| 2xx | No — delivered |
 | 408 Request Timeout | Yes |
 | 429 Rate Limited | Yes |
-| 5xx | Yes |
-| 410 Gone | **No** (permanent — resource removed) |
-| Other 4xx | No (client bug — don't retry) |
+| 5xx (500–599) | Yes |
+| **410 Gone** | **No — give up immediately** (resource intentionally removed) |
+| Other 4xx (400/401/403/404/…) | No — payload/auth bug; retrying won't help |
 | Network error / timeout | Yes |
 
-Every attempt is appended to `state.webhook.deliveries[]` for audit.
+Every attempt — success or failure — is appended to
+`state.webhook.deliveries[]` in `state.json`, including the HTTP status,
+error message, delivery ID, timestamps, and attempt number. Inspect
+via `engine.getStatus(runId)` or read `state.json` directly from R2.
+
+#### Manual replay
+
+If the receiver was down and the engine has already given up (5
+attempts exhausted, or 4xx stopped retries), replay any past delivery:
+
+```ts
+const status = await engine.getStatus(runId)
+const missed = status.meta.webhook?.deliveries.find((d) => d.status === 'failed')
+if (missed) {
+  await engine.retryWebhook(runId, missed.id)
+}
+```
+
+`retryWebhook` fires a fresh POST with a **new** delivery ID (so
+receivers that already processed the original ID won't reject it as a
+dup — this is a deliberate re-issuance, not a network retry) and
+continues the retry schedule from attempt 1.
+
+#### Correlated pause → resume
+
+When a run emits `paused`, the client typically gathers a decision and
+calls `resumeAsync({runId, gateAnswer, webhook})`. The resumed run
+will emit **another** webhook on completion — usually `done`, sometimes
+`paused` again if the model hits a second gate, or `failed` if resume
+fails. Receivers should track `runId` state across events:
+
+| State sequence | Meaning |
+|---|---|
+| `paused` → `done` | Happy-path HITL — approved and completed |
+| `paused` → `paused` → `done` | Multi-step approval — each gate wake fires its own event |
+| `paused` → `failed` (`CANCELLED`) | User rejected at the gate and cancelled the run |
+| `paused` → (no follow-up) | Orphaned — caller never called `resumeAsync` |
+
+Use `runId` as your correlation key across all events for a run.
+
+#### What's NOT a webhook event (deliberate omissions)
+
+These are intentionally out of scope:
+
+- **Per-turn progress** — too chatty. Poll `getStatus(runId)` for
+  live turn / token / activity updates (the heartbeat writes
+  `state.json` every ~500 ms when activity changes).
+- **Per-tool dispatch** — that's what `preToolCall` / `postToolCall`
+  hooks are for (in-process, synchronous).
+- **Subagent lifecycle** — the parent's terminal/pause state is what
+  fires; child runs are opaque to external receivers.
+- **Resume started / resume completed** — `resumeAsync()` returns
+  immediately with `{runId, nodeId, status: 'running'}`; the next
+  webhook you'll see is the resumed run's terminal state.
+
+If you need finer-grained updates, use `getStatus()` polling — it
+reads the heartbeat-updated `state.json` and gives you
+`turns / tokensUsed / currentActivity / lastTool` in real time
+without any webhook-driven traffic.
 
 ### Node.js example — sync HITL and async HITL together
 
@@ -1181,6 +1422,15 @@ await writeKnowledgeIndex({ adapter: k, base: 'hr-policies' })
 await writeKnowledgeIndex({ adapter: k, base: 'sales-playbook' })
 ```
 
+**Forgot to build the index?** Both tools fall back to an in-memory
+build on first call when `_index.json` is missing or corrupted. The
+fallback caches for the rest of the run, so subsequent searches are
+free. This makes the index a performance optimisation (skip the walk
+on every fresh run), not a setup requirement — drop files into the
+folder and the agent can discover them immediately. Pre-build with
+`writeKnowledgeIndex()` for production-scale corpora where the
+first-call cost matters.
+
 **Configure the engine** to enable the tools (off by default):
 
 ```ts