@pipelex/sdk 0.58.0 → 0.60.0

package/docs/method-copy.md

@@ -1,6 +1,6 @@
 # Copying a method into another organization
 
-A stored method belongs to the organization that holds it, and access to it comes from membership in that organization. `copyMethod` makes a new method in another organization the caller may change methods in, holding the original's draft, its form inputs and its saved examples, and leaves the original as it was. It is how a method reaches a team from the personal workspace that holds it, how it moves from a team into the caller's personal workspace, and, with the method's own organization as the destination, how a method is duplicated. `listMethodCopies` lists the copies made of a method, so a surface can say where they went. The client reaches them on the hosted API's `POST /v1/methods/{method_id}/copy` and `GET /v1/methods/{method_id}/copies`, and the types are in `src/product-models.ts`.
+A stored method belongs to the organization that holds it, and access to it comes from membership in that organization. `copyMethod` makes a new method in another organization the caller may change methods in, holding the original's draft, its form inputs and its saved examples, and leaves the original as it was. It is how a method reaches a team from the personal workspace that holds it, how it moves from a team into the caller's personal workspace, and, with the method's own organization as the destination, how a method is duplicated. The client reaches it on the hosted API's `POST /v1/methods/{method_id}/copy`, and the types are in `src/product-models.ts`.
 
 ```ts
 import { PipelexApiClient } from "@pipelex/sdk";
@@ -19,18 +19,17 @@ if (result.outcome === "refused") {
 }
 ```
 
-## The methods
+## The method
 
 | Method | Route | Answer |
 |---|---|---|
 | `copyMethod(methodId, { destination_org_id, name?, expected_draft_updated_at? }, { idempotencyKey? })` | `POST /v1/methods/{id}/copy` | A `MethodCopyResult`, discriminated on `outcome`: `copied` or `refused`. |
-| `listMethodCopies(methodId, { limit?, cursor? })` | `GET /v1/methods/{id}/copies` | One `MethodCopyPage`, `{ items, nextCursor }`, of `MethodCopyMarker` rows. `limit` is 1 to 100, 50 by default. |
 
-Both take a bare catalog id, as every method route does, and refuse a suffixed one (`mt_…@3`) with a `RequestArgumentError` before anything is sent.
+It takes a bare catalog id, as every method route does, and refuses a suffixed one (`mt_…@3`) with a `RequestArgumentError` before anything is sent.
 
 **The copy's body is closed.** The client sends `destination_org_id` and, when given, `name` and `expected_draft_updated_at`, and nothing else; a `null` reads as not given. `destination_org_id` is the canonical id of the destination, as a membership's `org_id` gives it; `name` is the copy's name, the original's when it is left out; `expected_draft_updated_at` is the original's draft token the caller last saw (`MethodData.updated_at`). A `destination_org_id` that is not a non-empty string, and a `name` or a token given that is not a string, are refused with a `RequestArgumentError` before any request.
 
-**Only a web session may copy.** A Pipelex API key belongs to one organization and cannot speak for another, so the platform refuses a copy sent with one `403` `auth.forbidden`, whatever its role. The listing is a read, open to every member of the method's organization and to every credential.
+**Only a web session may copy.** A Pipelex API key belongs to one organization and cannot speak for another, so the platform refuses a copy sent with one `403` `auth.forbidden`, whatever its role.
 
 ## What a copy holds
 
@@ -83,7 +82,7 @@ A non-2xx answer means no verdict could be produced, and none of them leaves any
 | `502` | `runtime.unavailable` | The platform could not read the caller's memberships. Its `retry` is `yes`. |
 | `500` | `pipelex.storage.not_configured` | The platform has no usable credentials for storage. Its `retry` is `no`. |
 | `503` | `pipelex.storage.unavailable` | Storage failed a read, a copy or a write. Its `retry` is `yes`. |
-| `503` | `runtime.unavailable` | The copy did not finish inside its deadline, a stored file it copied was written over meanwhile, or DynamoDB failed a read or a write. Its `retry` is `yes`. When DynamoDB failed both the write of the copy's row and the read that would confirm it, the `detail` names the copy's method id and says that a retry may make a second copy: the copy may stand in the destination, without the original's marker of it and without the destination's records of its files, so look for that id in the destination before copying again. |
+| `503` | `runtime.unavailable` | The copy did not finish inside its deadline, a stored file it copied was written over meanwhile, or DynamoDB failed a read or a write. Its `retry` is `yes`. When DynamoDB failed both the write of the copy's row and the read that would confirm it, the `detail` names the copy's method id and says that a retry may make a second copy: the copy may stand in the destination, without the destination's records of its files, so look for that id in the destination before copying again. |
 
 `MethodErrorCode` names `pipelex.org.not_found` and `pipelex.catalog.copy_too_large` beside the other method codes ([errors.md](./errors.md#the-method-codes)).
 
@@ -115,14 +114,6 @@ try {
 
 Without a key, a copy whose answer was lost cannot be told from one that never ran, and sending it again may make a second copy. Even with one, two rare windows can make a second copy on a retry. A platform process can die after writing the copy and before storing its answer, a window the creation of a method shares. And the `503` whose `detail` names the copy released the key while the copy may stand, which is why it names the copy: look for that method id in the destination before copying again.
 
-## The copies of a method
-
-The original keeps a marker of each copy made of it, and `listMethodCopies` pages them. A `MethodCopyMarker` holds `copy_method_id`, `destination_org_id`, `copied_by_user_id` and `copied_at`. The destination is named by its id alone, never by its name, so the listing tells no reader the name of an organization they do not belong to: a surface shows a destination's name from the reader's own memberships (`listMemberships`), and a copy in an organization the reader does not belong to stays an id. The rows come in the order of the copies' ids, which says nothing of when each was made; sort by `copied_at` to show them by date.
-
-Pass the page's `nextCursor` back as `cursor` to continue; `null` is the last page. A method never copied lists an empty page, as does a copy, which names its original through `copied_from` and has no copies of its own. A marker stays when its copy is erased, and the copy then reads as not found. The marker is written once the copy exists, and a failure to write it does not fail the copy, so a copy can, rarely, be missing from the list. The original's erasure and its discard delete its markers with it.
-
-The listing answers `404` `pipelex.catalog.not_found` for an unknown method, `409` `pipelex.catalog.being_deleted` while its erasure runs, and `422` `request.invalid` for a cursor it did not issue for that method or a `limit` outside 1 to 100. A page whose `items` is not an array or whose `next_cursor` is neither a string nor `null` is an answer the SDK cannot read.
-
 ## `copied_from` on a method
 
 Every read of a method, `getMethod` and the `method` of every write, carries `copied_from`: `{ org_id, method_id, draft_digest }` of the original on a copy, `null` on a method created afresh, and absent from a platform that predates copies. It is written once, when the copy is made, and never rewritten, so it names the draft the copy started from even after the original has changed or been erased; its `draft_digest` is `null` for a draft that held no tree. It holds opaque ids only, never a name, and the original's organization may be one the reader does not belong to. The list rows (`MethodSummary`) do not carry it.

package/docs/webhooks.md

@@ -0,0 +1,130 @@
+# Caller webhooks
+
+A run can call webhooks of its caller once it has ended, so the caller learns of the end without polling. The run registers the URLs when it starts, in `callback_urls`; when it ends, the runtime sends each URL one signed `POST` carrying a `run.finished` event; the caller's receiver verifies the call, then fetches the results by the run's id. The contract is the spec `conformance/specs/pipelex-caller-webhooks.md`, in the `conformance` repository, which both the hosted plane and a self-hosted runner honour. This page covers the four parts the SDK takes: registering the URLs, verifying a call (`verifyWebhook`), reading the hosted delivery log, and reading or rotating the organization's signing secret. The Python SDK's twin is [`python/docs/webhooks.md`](https://github.com/Pipelex/pipelex-sdk/blob/main/python/docs/webhooks.md).
+
+## Registering the webhooks — `callback_urls`
+
+`execute`, `start` and `startAndWaitForResult` take `callback_urls`, an array of URLs:
+
+```ts
+const ack = await client.start({
+  method_id: "mt_abc123",
+  inputs,
+  callback_urls: ["https://hooks.example.com/pipelex"],
+});
+ack.callback_urls; // ["https://hooks.example.com/***"]
+```
+
+It is a Pipelex API extension (`PipelexApiRunExtensions`), beside `method_ref`: the hosted API and a self-hosted runner both take it, and `startAndWaitForResult` keeps it on its blocking fallback, so a bare runner calls the same webhooks once its blocking run has ended. Each server holds the URLs to its own rules, which the client does not restate. The hosted API takes at most 5 HTTPS URLs of at most 2,048 characters, none repeated, and refuses a URL whose host resolves to a private, loopback, link-local or metadata address with a `422` `request.invalid` whose `field` names it as `callback_urls.<index>`; a deployment that does not deliver caller webhooks refuses the list with a `422` `request.unsupported`, and a self-hosted runner whose operator configured no signing secret with a `422` `pipelex.webhook.secret_not_configured`.
+
+The client refuses, before anything is sent, a value that is not an array (`client.argument_invalid`, reason `not_a_list`) or an array holding anything but strings (`not_a_string`), and does not send an empty array, which the platform treats as absent. `callback_urls` is no run source: a run naming its webhooks and nothing to run is refused as any run with nothing to run is (`nothing_to_run`). It is reserved on `extra`, as `method_ref` is, so it reaches the body by its named option alone.
+
+A URL may carry a credential of the receiver's in its path or query string. Every read of the run lists the URLs redacted to their scheme, host and a port other than the scheme's default, with `/***` in place of the rest: the start's acknowledgement (`PipelexRunResultStart.callback_urls`), the status read (`RunRead.callback_urls`) and the run record (`RunDetail.callback_urls`). For a run that registered none the acknowledgement leaves the member out, and the two reads answer `null`.
+
+## Receiving a call — `verifyWebhook`
+
+A call is a Standard Webhooks v1 `POST`. Its `webhook-id` header names the delivery, the same on every attempt and replay; `webhook-timestamp` is when the attempt was sent, in Unix seconds; and `webhook-signature` lists `v1,` and the base64 HMAC-SHA256 of `{webhook-id}.{webhook-timestamp}.{body}` under each secret live when it was sent. `verifyWebhook(body, headers, secret, options?)` checks all three and returns the event:
+
+```ts
+import {
+  WebhookEventUnreadableError,
+  WebhookVerificationError,
+  isRunFinishedEvent,
+  verifyWebhook,
+} from "@pipelex/sdk";
+
+export async function POST(request: Request): Promise<Response> {
+  const body = await request.text(); // the raw body, never a parsed and re-serialized one
+  let event;
+  try {
+    event = await verifyWebhook(body, request.headers, process.env.PIPELEX_WEBHOOK_SECRET!);
+  } catch (err) {
+    if (err instanceof WebhookVerificationError) return new Response(null, { status: 401 });
+    if (err instanceof WebhookEventUnreadableError) {
+      // Verified, so sent by the runtime, but no event this SDK reads: every retry carries the same body.
+      console.warn("Unreadable webhook", request.headers.get("webhook-id"), err.problem);
+      return new Response(null, { status: 204 });
+    }
+    throw err;
+  }
+  if (!isRunFinishedEvent(event)) return new Response(null, { status: 204 }); // a type this receiver does not read
+  if (await alreadyHandled(request.headers.get("webhook-id")!)) return new Response(null, { status: 204 });
+  const results = await client.getRunResult(event.data.pipeline_run_id);
+  // ...
+  return new Response(null, { status: 204 });
+}
+```
+
+It follows the contract's verification procedure, in its order:
+
+1. **The headers.** A call missing `webhook-id`, `webhook-timestamp` or `webhook-signature`, or carrying one empty, is refused with the refusal `missing_header`. Header names are matched whatever their case; `headers` is a fetch `Headers` or a plain record such as Node's `IncomingHttpHeaders`, a header a record gives as several values, or names under several cases, being read as those values joined by spaces.
+2. **The timestamp.** A `webhook-timestamp` that is not a whole number of Unix seconds, is past `Number.MAX_SAFE_INTEGER`, or is more than five minutes (`WEBHOOK_TIMESTAMP_TOLERANCE_SECONDS`, 300) from the receiver's clock either way, is refused with `timestamp`; exactly five minutes is accepted. `options.now` sets the clock, for a test or a call received earlier.
+3. **The signature.** The signature is computed from the raw body under each secret given and compared in constant time with every `v1` entry of `webhook-signature`, an entry of another version being ignored. Entries are separated by spaces, and a comma ending one is dropped, since a header sent more than once reaches a receiver joined by a comma and a space, as fetch's `Headers` and Node join it. No match is the refusal `signature`.
+
+Only then is the body read. The body must be the bytes received, as a `string`, `Uint8Array` or `ArrayBuffer`: the same JSON written with other bytes, re-serialized or with its non-ASCII characters escaped, signs differently and is refused. In a framework that parses JSON bodies by itself, read the raw body instead, such as `express.raw({ type: "application/json" })` in Express.
+
+The secret is the `whsec_` value the runtime signs with, given whole: the organization's on the hosted plane ([below](#the-organizations-signing-secret)), or the one a self-hosted runner's operator configured. During a rotation, give both the new and the previous secret as an array: the call verifies under either. A secret that is not `whsec_` and a key in standard base64 with its padding is refused with a `RequestArgumentError` (`client.argument_invalid` naming `secret`) before the call is looked at, and no refusal quotes it.
+
+`verifyWebhook` uses Web Crypto (`globalThis.crypto.subtle`), so it runs in Node, Deno, Bun, an edge runtime or a worker, and stays out of the Node host like the rest of the barrel.
+
+### What a refusal carries
+
+| Error | When | `code` |
+| --- | --- | --- |
+| `WebhookVerificationError`, `refusal` `missing_header` | a header is absent or empty | `auth.missing_credential` |
+| `WebhookVerificationError`, `refusal` `timestamp` | the timestamp is malformed or out of tolerance | `auth.invalid_credential` |
+| `WebhookVerificationError`, `refusal` `signature` | no signature matches under any secret given | `auth.invalid_credential` |
+| `WebhookEventUnreadableError` | a verified body that is not UTF-8 or not JSON | `request.malformed`, `reason` `not_utf8` or `not_json` |
+| `WebhookEventUnreadableError` | a verified body that is no event this SDK reads | `request.invalid`, `field` the member at fault (`type`, `timestamp`, `attempt`, `data`, `data.pipeline_run_id`, `data.status`) |
+| `RequestArgumentError` | a secret, a body or a clock of the wrong kind | `client.argument_invalid` naming `secret`, `body` or `now` |
+
+Answer a `WebhookVerificationError` with a status outside `2xx`, such as `401`: the sender counts it as a failed attempt and retries, and a replay later sends the delivery again. Answer a `WebhookEventUnreadableError` with a `2xx` instead, and log it, as the example does: the call was verified, so the runtime sent it, and every retry and replay of the delivery carries the same body, so failing it would only spend the delivery's attempts on a body this SDK will never read. Both errors are `ProblemError`s, never a `PipelexRequestError`, since nothing was requested.
+
+### The event
+
+`verifyWebhook` returns a `RunFinishedEvent` (`isRunFinishedEvent`), or an `UnknownWebhookEvent` for a `type` this SDK does not read, which a receiver acknowledges with a `2xx` and ignores. A `run.finished` event holds:
+
+- `type`, `run.finished` for every ending, the outcome being in `data.status`;
+- `timestamp`, when the run ended (its `finished_at`), not when the call was sent;
+- `attempt`, which attempt of the delivery this call is, from 1 to `Number.MAX_SAFE_INTEGER`. It is informational: it increases with each retry and a replay continues it, but a runtime that runs its delivery step again may skip numbers or start again at 1 under the same `webhook-id`. Deduplicate on the `webhook-id` header, never on `attempt` or the body;
+- `data`, the run as its caller's status read served it when the delivery was created, the same on every attempt and replay.
+
+The event never carries the run's results: fetch them with `getRunResult(event.data.pipeline_run_id)`.
+
+**Two shapes of `data` reach a receiver today, and `RunFinishedEventData` accepts both.** A self-hosted runner sends the MTHDS protocol's status read: `pipeline_run_id`, a lowercase `status` (`completed` or `failed`, a cancelled or timed-out run reading `failed` with a code that says why), and for a failed run its problem as `error`, a member with no value left out. The hosted plane sends the body of its own `GET /v1/runs/{id}/status`, this SDK's `RunRead`, without `callback_urls`: an uppercase `status` (`COMPLETED`, `FAILED`, `CANCELLED`, `TERMINATED` or `TIMED_OUT`), a member with no value written as `null`, and the platform's own members, such as `finished_at`, `method_id` and `request_id`. That hosted shape does not validate against the contract's JSON Schema, which describes the protocol's read; the hosted event follows its status read onto the protocol's shape when that read moves, with no change of its own. Until then, tell a success from a failure under both spellings with `event.data.status.toLowerCase() === "completed"`. A failed run's `error` is kept when it is a problem document and read as `null` otherwise, as the status read keeps it.
+
+## The hosted delivery log
+
+The hosted plane records each delivery, one per URL a run registered, with every attempt. Any member of the organization reads them, by web session or API key:
+
+- `listRunWebhookDeliveries(runId)` — `GET /v1/runs/{id}/webhook-deliveries`, a `WebhookDeliveryList` whose `deliveries` are `WebhookDeliverySummary` rows in the order of the run's URLs, none for a run that registered none or has not ended. Each has its `delivery_id` (the receiver's `webhook-id`), its `url_index` and redacted `url`, its `state` (`pending` on its retry schedule, `delivered` once a receiver answered `2xx`, `failed` past its last attempt) and its `attempts`, each with when it was sent, the HTTP `status_code` the receiver answered, or the `failure` that kept an answer from coming (`timeout`, `connection`, `tls`, `refused_address`), and its `duration_ms`. No response body is kept.
+- `getRunWebhookDelivery(runId, deliveryId)` — one `WebhookDeliveryDetail`, the summary and its `event`: the body its latest attempt sent, or, before any attempt, the body its first attempt will send. Every attempt and replay of the delivery sends that body, only its `attempt` changing, so the `event` matches what a receiver logged only once an attempt was made: it then equals, as JSON, the body that attempt brought the receiver. Compare the two as parsed JSON values, never as text, since the read promises the same value, not the same bytes.
+- `replayRunWebhookDelivery(runId, deliveryId, { idempotencyKey? })` — sends a settled delivery again: one new attempt under the same `webhook-id`, with the same `data`, the next `attempt`, a fresh timestamp and the signatures of the secrets live then. It answers `202` with a `WebhookReplayAccepted` naming the attempt; a replay asked again before it is sent is that same attempt. A delivery still `pending` is refused `409` `pipelex.org.not_manageable` with the reason `webhook_delivery_pending`, and one holding the most attempts a delivery holds with `webhook_delivery_attempts_exhausted`; past the organization's replay rate the platform answers `429` `account.rate_limited` with its `retry_after`.
+
+### Retrying a replay — `idempotencyKey`
+
+`replayRunWebhookDelivery`'s last argument takes an `idempotencyKey`, sent as the `Idempotency-Key` header of the replay and not sent at all when absent or `null`. The platform stores the replay's answer under the key, so the same replay sent again under the same key gets the stored answer, naming the same attempt, and the delivery is not sent twice: a replay whose answer was lost, as on a client timeout, can be sent again safely. Generate one key per replay you mean to ask for (a `crypto.randomUUID()` is a good one), and keep it across that replay's retries:
+
+```ts
+const idempotencyKey = crypto.randomUUID();
+const accepted = await client.replayRunWebhookDelivery(runId, deliveryId, { idempotencyKey });
+```
+
+A key reused for another request is refused `409` `request.idempotency_conflict`, whose `retry` is `no`, and a key whose earlier request is still being handled `409` `request.idempotency_key_in_use`, whose `retry` is `yes`: send it again under the same key after its `retryAfter`. A key is 1 to 255 visible ASCII characters, `!` to `~`; anything else, an empty string included, is a `RequestArgumentError` naming `idempotency_key` before anything is sent, the rule a start's key is held to ([errors.md](./errors.md#a-starts-idempotency-key--idempotencykey)).
+
+The states and failures are read open: one this SDK does not name arrives as its raw string.
+
+## The organization's signing secret
+
+On the hosted plane every caller webhook of an organization is signed with the organization's secret. Its admins read it, from a web session only, since the hosted gateway refuses an API key on these routes:
+
+- `getWebhookSecret(orgId)` — `GET /v1/organizations/{org_id}/webhook-secret`, a `WebhookSecret`: the `secret`, when it was created, when it was last rotated, and until when the secret the last rotation replaced still signs (`previous_secret_expires_at`). The platform generates the first secret on this first read. Treat the answer as a credential: never log or cache it.
+- `rotateWebhookSecret(orgId, { idempotencyKey? })` — `POST /v1/organizations/{org_id}/webhook-secret/rotate`. The new secret is current at once and the previous one keeps signing beside it for 24 hours, so a receiver configured with both, as `verifyWebhook` takes them, misses nothing. The answer carries a secret, so the platform never replays it: the same `Idempotency-Key` sent again is refused `409` `request.already_executed` and rotates nothing more, and the secret is then read with `getWebhookSecret`.
+
+Both refuse another role with `403` `auth.forbidden` whose `params.requirement` is `admin`, an organization other than the caller's active one with `403` `pipelex.org.mismatch`, and one with no record with `404` `pipelex.org.not_found`.
+
+## How the verification is held to the contract
+
+The contract's known-answer vectors, `tests/caller_webhooks/fixtures/signature-vectors.json` in the `conformance` repository, are vendored byte for byte at `tests/fixtures/caller-webhooks/signature-vectors.json`, with `tests/fixtures/caller-webhooks/vendored.json` recording the commit it was copied at and its SHA-256. `tests/webhooks.test.ts` fails when the copy no longer has that digest, verifies every verification case at its `receiver_clock` and compares its verdict and refusal, and verifies every signing case under each secret that signed it. The shared case file `tests/fixtures/webhook-events.json` adds what the vectors leave to each SDK: the problem of each refusal, what each verified body reads as (the hosted and self-hosted shapes, an unknown type, the unreadable bodies) and the secrets refused.
+
+All three files are shared with the Python SDK the way every case file is ([`docs/layout.md`](https://github.com/Pipelex/pipelex-sdk/blob/main/docs/layout.md) at the repository's root): the source is here, `scripts/shared-files.mjs` declares the copies under `python/tests/fixtures/`, and `make check-shared-files` fails on a stale copy. To refresh the vectors, copy the file from the conformance repository at the commit that last changed it, update the commit and the digest in `vendored.json`, and run `make shared-files` at the repository's root.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@pipelex/sdk",
-  "version": "0.58.0",
+  "version": "0.60.0",
   "description": "TypeScript SDK for the Pipelex hosted API — execute methods, manage runs, and call the product surface.",
   "license": "MIT",
   "repository": {