@anyformat/sdk 0.2.2 → 1.0.0-rc.2

package/README.md

@@ -1,13 +1,21 @@
 # @anyformat/sdk
 
-TypeScript SDK for the AnyFormat API. Mirrors the Python SDK's fluent flow.
+TypeScript SDK for the AnyFormat API (v3). Mirrors the Python SDK's fluent flow.
+
+```bash
+npm install @anyformat/sdk@rc
+```
+
+> **Release candidate.** v3 support ships as `1.0.0-rc.1` on the npm `rc`
+> dist-tag. Plain `npm install @anyformat/sdk` keeps resolving to the stable
+> 0.2.x (v2) line until 1.0.0 final ships after the RC bake period.
 
 ```ts
 import { Anyformat, Schema } from "@anyformat/sdk";
 
 const af = new Anyformat({ apiKey: process.env.ANYFORMAT_API_KEY! });
 
-const result = await af
+const workflow = await af
   .workflow("Invoice")
   .parse()
   .extract([
@@ -18,21 +26,240 @@ const result = await af
       Schema.option("EUR", "Euro."),
     ]),
   ])
-  .run(file)
-  .wait();
+  .create();
+
+const run = await workflow.run(new File([bytes], "invoice.pdf"));
+const result = await run.wait();
 
 result.field("vendor")?.value; // -> "Acme Corp"
 ```
 
-## Managing workflows
+`workflow.run(files)` uploads 1..100 files as one **document packet** (the unit
+runs execute on) and enqueues a **run**. `run.wait()` polls
+`GET /v3/runs/{run_id}/` until the status is terminal: `processed` resolves
+with the results inline; `error` throws `ExtractionFailedError` and `cancelled`
+throws `ExtractionCancelledError` (both subclass `RunFailedError`). A transient
+429/5xx *during* polling is retried with backoff (honouring `Retry-After`), so a
+short spike doesn't abort a long wait.
+
+`result.field("line_items")` returns the array of row records for a
+list/object field (and the scalar `ExtractedField` for a scalar field); only a
+genuinely missing field returns `undefined`.
+
+## Upload now, run later
+
+Uploading spends no credits — trigger extraction when you're ready, and re-run
+the same packet after editing the workflow (each trigger creates a new run):
+
+```ts
+const upload = await workflow.upload([fileA, fileB]); // one packet, two files
+console.log(upload.documentPacketId, upload.files);
+
+const run = await upload.run();
+const result = await run.wait();
+```
+
+Or import server-side from HTTPS URLs:
+
+```ts
+const upload = await workflow.uploadFromUrls(["https://example.com/invoices/april.pdf"]);
+```
+
+Packets stay addressable later:
+
+```ts
+const packet = await af.getDocumentPacket(upload.documentPacketId);
+packet.status;        // not_started | queued | in_progress | processed | error | cancelled
+packet.latest_run_id; // null until the packet has been run
+
+await af.runDocumentPacket(packet.id);   // re-run on the latest workflow version
+await af.deleteDocumentPacket(packet.id); // irreversible
+```
+
+## Dataset uploads (ground truth)
+
+Upload a document into a workflow's **dataset** as one
+document packet — 1..100 files treated as one document, plus optional `groundTruth`
+(the expected extraction). Atomic: any rejected file or a ground-truth failure
+fails the whole call and stores nothing. Filenames are unique within a
+workflow, so by default a colliding name fails with `409`; pass
+`onConflict: "rename"` to auto-rename instead (a rename is never silent):
+
+```ts
+const entry = await workflow.uploadToDataset(new File([bytes], "invoice.pdf"), {
+  groundTruth: { vendor: "ACME", total: "42.00", line_items: [{ sku: "A1", qty: "2" }] },
+});
+console.log(entry.documentPacketId, entry.groundTruthSaved); // "069d…", true
+```
+
+`groundTruth` keys are the workflow schema's field identifiers (`persistent_id`);
+a scalar maps to `string | null`, a table/object field to an array of row
+objects. Bulk ingestion loops one atomic call per document — there is no batch
+endpoint, so each call is bounded and independently retryable:
+
+```ts
+await workflow.uploadDocumentsToDataset([
+  { files: new File([a], "a.pdf"), groundTruth: { vendor: "ACME" } },
+  { files: [new File([b1], "b1.pdf"), new File([b2], "b2.pdf")] }, // one 2-file document
+]);
+```
+
+Both are also available off the client: `af.uploadToDataset(workflowId, files, opts)`
+and `af.uploadDocumentsToDataset(workflowId, documents)`.
+
+## Idempotency keys
+
+Upload and run triggers accept an `Idempotency-Key` (Stripe convention) —
+retrying with the same key replays the original packet/run instead of
+creating (and billing) a duplicate:
+
+```ts
+const run = await workflow.run(file, { idempotencyKey: "invoice-2026-07-0042" });
+```
+
+## Pagination
+
+Every list is keyset-paginated: pass `cursor` from the previous page's
+`nextCursor` until it is `null`. There are no totals or page numbers.
+
+```ts
+let cursor: string | undefined;
+do {
+  const page = await af.listRuns(workflow.id, { limit: 100, cursor });
+  for (const run of page.items) console.log(run.id, run.status);
+  cursor = page.nextCursor ?? undefined;
+} while (cursor);
+```
+
+`listWorkflows`, `listDocumentPackets(workflowId)` and `listRuns(workflowId)`
+all follow this shape.
+
+To walk **every** row without cursor bookkeeping, use the auto-paging async
+iterators — they follow `nextCursor` for you and fetch each page lazily as you
+consume it:
 
 ```ts
-const workflows = await af.listWorkflows({ pageSize: 50 }); // -> Workflow[]
-for (const wf of workflows) console.log(wf.id, wf.name);
+for await (const run of af.iterRuns(workflow.id)) console.log(run.id, run.status);
 
-await af.deleteWorkflow(workflows[0].id); // -> the deleted id
+const all = [];
+for await (const wf of af.iterWorkflows()) all.push(wf);
 ```
 
+`iterWorkflows`, `iterDocumentPackets(workflowId)` and `iterRuns(workflowId)`
+mirror their `list*` counterparts.
+
+## Editing a workflow — the `persistent_id` round-trip
+
+`GET /v3/workflows/{id}/` returns the full typed graph; PATCH accepts the same
+shape back. Every extraction field carries a server-assigned `persistent_id` —
+echo it unchanged (including across renames) and the field keeps its identity,
+so analytics, ground truth and quality metrics stay attached:
+
+```ts
+const detail = await af.getWorkflow(workflow.id);
+
+for (const node of detail.nodes) {
+  if (node.type !== "extract") continue;
+  for (const field of node.extraction_schema.fields) {
+    if (field.name === "total") field.name = "grand_total"; // rename, persistent_id untouched
+  }
+}
+
+await af.updateWorkflow(workflow.id, detail); // PATCH echoes GET's shape
+```
+
+When authoring from scratch with a known identity, set it on the factory:
+
+```ts
+Schema.float("grand_total", "Grand total amount.", {
+  persistentId: "0686bb97-8c30-70f0-8000-97669e00aaaa",
+});
+```
+
+Omit `persistentId` for new fields — an omitted id falls back to name-matching
+against the prior version.
+
+## Quick parse: one call → markdown
+
+`af.parse()` runs the platform's atomic parse operation (a fast lite parse)
+and returns a run id. Poll `getParseMarkdown(runId)` for the markdown —
+no workflow to author or manage.
+
+```ts
+const runId = await af.parse(file); // File | Blob
+let markdown = await af.getParseMarkdown(runId);
+while (markdown === null) {
+  await new Promise((r) => setTimeout(r, 3000));
+  markdown = await af.getParseMarkdown(runId); // null until processed
+}
+console.log(markdown);
+```
+
+## Smart lookup
+
+Resolve a field by matching the document against a reference file (CSV/catalog) instead of
+reading it off the page — flag the field with `{ source: "smart_lookup" }` and pass the file
+via `lookupFiles`. A smart-lookup field is overwritten by the lookup; use
+`{ source: "lookup_if_missing" }` to extract it from the document too and let the lookup
+fill only the slots extraction
+left empty:
+
+```ts
+const wf = await af
+  .workflow("Invoice + vendor lookup")
+  .parse()
+  .extract(
+    [
+      Schema.string("vendor_name", "Vendor as printed on the invoice."),
+      Schema.string("vendor_id", "Canonical vendor code from the catalog, joined on vendor_name.", { source: "smart_lookup" }),
+    ],
+    { lookupFiles: ["vendor_catalog.csv"], lookupSuggestion: "Join vendor_name → vendor_id." },
+  )
+  .create();
+const result = await (await wf.run(file)).wait();
+```
+
+Full runnable example: [`examples/smart-lookup.mjs`](examples/smart-lookup.mjs).
+
+## Errors
+
+API failures throw a typed subclass of `APIError` carrying the v3 error
+envelope: `status`, `errorCode`, `retryable`, `requestId`, `detail`,
+and the raw `body`. `err.message` is the envelope's `error` summary alone —
+`detail` is developer-facing and stays on `err.detail`, never in `message`
+(ADR platform/0053 §2). Dynamic values (e.g. the missing scope on a scope
+denial) are named in the error message, not a separate structured field.
+
+Most classes are chosen by HTTP status:
+
+| status | class | `error_code` | retry? |
+| --- | --- | --- | --- |
+| 400 | `BadRequestError` | `VALIDATION_ERROR` / `TOPOLOGY_INVALID` | no |
+| 401 | `UnauthorizedError` | `AUTH_FAILED` | no |
+| 402 | `PaymentRequiredError` | `PAYMENT_REQUIRED` | no — top up credit first |
+| 403 | `ForbiddenError` | `ACCESS_DENIED` | no |
+| 404 | `NotFoundError` | `NOT_FOUND` | no |
+| 422 | `APIError` | `VALIDATION_ERROR` | no |
+| 429 | `RateLimitedError` | `RATE_LIMITED` | yes, after `err.retryAfter` seconds (`Retry-After`) |
+| 5xx | `ServerError` | `INTERNAL_ERROR` / `GATEWAY_TIMEOUT` | yes |
+
+A few `error_code` values name a condition a caller would plausibly handle on
+its own, so they get a more specific class than their status alone. Each still
+subclasses its status class, so an `instanceof ForbiddenError` catch keeps
+working:
+
+| `error_code` | class | status |
+| --- | --- | --- |
+| `insufficient_api_key_scope` | `InsufficientScopeError` | 403 (`ForbiddenError`) |
+| `terms_acceptance_required` | `TermsAcceptanceRequiredError` | 403 (`ForbiddenError`) |
+| `WORKFLOW_NAME_CONFLICT` | `WorkflowNameConflictError` | 409 (`APIError`) |
+
+`Run.wait()` additionally throws `ExtractionFailedError` (status `error`) or
+`ExtractionCancelledError` (status `cancelled`) — both subclass `RunFailedError`,
+so `catch (e) { if (e instanceof RunFailedError) … }` still catches either — and
+`SDKTimeoutError` when the wait budget elapses. It transparently retries a
+transient 429/5xx encountered while polling rather than surfacing it.
+
 ## How it's built
 
 The wire types and the low-level HTTP client are **generated** from the API's
@@ -43,22 +270,44 @@ ergonomic layer (`Schema`, `WorkflowBuilder`, `Result`, `Anyformat`) wraps it.
 ## Regenerating (and the drift guard)
 
 ```bash
-npm run gen          # dump the v2 OpenAPI offline + regenerate src/sdk
-npm run gen:check    # regenerate and fail if the committed output drifted
+bun run gen          # dump both OpenAPI artifacts offline + regenerate src/sdk
+bun run gen:check    # regenerate and fail if the committed output drifted
 ```
 
-`gen:check` runs in CI and as a pre-commit hook (scoped to changes under
-`anyformat/services/api` and `anyformat/workflow`) so the committed SDK can
-never fall behind the backend. The SDK's own `tsc` then fails to compile if a
-backend model change alters a type the hand-written layer relies on — drift is
-caught as a type error, not just a diff.
+Two committed artifacts: `openapi/v3.json` feeds the generated client;
+`openapi/v2.json` stays as the frozen v2 surface guard (the dump script's
+`--surface v2` output must remain byte-identical). `gen:check` runs in CI and
+as a pre-commit hook (scoped to changes under `anyformat/services/api` and
+`anyformat/workflow`) so the committed SDK can never fall behind the backend.
+The SDK's own `tsc` then fails to compile if a backend model change alters a
+type the hand-written layer relies on — drift is caught as a type error, not
+just a diff.
 
 ## Scripts
 
+Install deps with `bun install --frozen-lockfile` (matches CI + the frontend
+toolchain). Then:
+
 | script | what |
 | --- | --- |
-| `npm run gen` | dump spec + regenerate `src/sdk` |
-| `npm run gen:check` | regenerate + assert no drift |
-| `npm run build` | bundle to `dist/` (ESM + CJS + types) via tsup |
-| `npm run typecheck` | `tsc --noEmit` |
-| `npm test` | vitest |
+| `bun run gen` | dump specs + regenerate `src/sdk` |
+| `bun run gen:check` | regenerate + assert no drift |
+| `bun run build` | bundle to `dist/` (ESM + CJS + types) via tsup |
+| `bun run typecheck` | `tsc --noEmit` |
+| `bun run test` | vitest |
+
+## Releasing
+
+Cut on demand — a maintainer bumps the version in a PR, then pushes a `sdk-js-vX.Y.Z` tag on the release commit. Full flow + one-time setup: [`../RELEASING.md`](../RELEASING.md). Break-glass: `./publish.sh [--build-only|--dry-run]`.
+
+## Node configuration knobs
+
+Every knob is an option on `.parse({...})` / `.extract(fields, {...})`. `parse` options are a
+`mode`-discriminated union, so a wrong-mode knob is a compile-time error.
+
+**`.parse({...})`** — `mode` (`"standard"`/`"agentic"`/`"lite"`/`"flash"`), `promptHint`, `figureEnhancement`,
+`cache`; **agentic-only** `effort` (`"low"`/`"mid"`/`"accurate"`); **lite**: no knobs of its own — one
+OCR-only recipe that never reads `promptHint`; **flash-only** `scannedPages` (`"skip"`/`"fail"`) — no model call, no OCR.
+
+**`.extract(fields, {...})`** — `mode` (`"standard"`/`"agentic"`/`"max"`/`"lite"`), `useImages`,
+`lookupSuggestion`, `lookupReasoningEffort` (`"minimal"`/`"low"`/`"medium"`/`"high"`), `lookupFiles`.