@pipelex/sdk 0.18.1 → 0.18.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +14 -0
- package/dist/index.d.ts +1 -1
- package/dist/index.js +1 -1
- package/docs/architecture.md +128 -0
- package/docs/artifact-download.md +165 -0
- package/docs/build-routes.md +101 -0
- package/docs/crate-routes.md +197 -0
- package/docs/input-preparation.md +186 -0
- package/docs/run-results.md +159 -0
- package/docs/run-usage.md +155 -0
- package/package.json +3 -2
|
@@ -0,0 +1,197 @@
|
|
|
1
|
+
# Crate routes (`/v1/resolve`, `/v1/codegen`)
|
|
2
|
+
|
|
3
|
+
Two routes project a **closure** of MTHDS files into the artifacts downstream tooling actually consumes: `resolve` emits the **normalized library crate**, and `codegen` projects that crate into **stamped typed artifacts** plus their lock. Like the [build routes](./build-routes.md), they are Pipelex API extensions rather than MTHDS Protocol operations — but note the ownership split: the _crate_ is standard-owned (the MTHDS Library Crate Format), while the HTTP surface serving it, and every type projection on top of it, are ours.
|
|
4
|
+
|
|
5
|
+
> **Both crate routes are served on every hosted origin, and by any `pipelex-api` runner.** On the hosted plane a route is reachable only when the gateway's API-key allowlist and the platform's tooling proxy both list its path — each enumerates routes explicitly, and an unlisted path answers a gateway `403 {"message":"Forbidden"}`, refused before any service sees the request, so not even an RFC 7807 problem body. `resolve` and `codegen` are listed by both.
|
|
6
|
+
>
|
|
7
|
+
> Measured 2026-08-23 against `api.pipelex.com` (`pipelex-hosted@0.10.1`) with a real API key: an empty body to either route comes back as an RFC 7807 `422` naming the missing fields — a request-shape verdict only a route that reached the service can produce — and a real closure comes back as a `200` carrying the artifacts. `api-dev.pipelex.com` has served them since 2026-08-13. Use a key when re-measuring: unauthenticated, every path answers `401` whether or not it is allowlisted, so a keyless probe cannot tell the two states apart.
|
|
8
|
+
>
|
|
9
|
+
> **`lint` / `format` are the exception and answer `403` on both origins.** That blocks nothing, because linting and formatting `.mthds` are toolchain capabilities rather than hosted ones: `plxt` carries both, and the post-edit hook this repo builds (`npm run build:hook`, vendored into `pipelex-plugins`) runs them offline through `@pipelex/tools-wasm` with no credentials — from the published package, `client.lint` / `client.format` are the documented fallback, against a runner. Exposing them on the hosted origins is a known, non-critical item on the platform's list, tracked in [`wip/hosted-exposure-crate-and-tools-routes.md`](https://github.com/Pipelex/pipelex-sdk-js/blob/main/wip/hosted-exposure-crate-and-tools-routes.md).
|
|
10
|
+
|
|
11
|
+
## The shared envelope
|
|
12
|
+
|
|
13
|
+
Both take the same closure selector — inline `files`, a `method_ref`, or a hosted `method_id`, exactly one (the strict tooling XOR):
|
|
14
|
+
|
|
15
|
+
```ts
|
|
16
|
+
interface CrateRequestBase {
|
|
17
|
+
files?: MthdsFileItem[]; // [{ content, source? }] — `source` is the provenance label
|
|
18
|
+
method_ref?: string; // address form server-resolved (pipelex-api >= 0.21.0); registry form reserved → 501
|
|
19
|
+
}
|
|
20
|
+
|
|
21
|
+
interface PipelexHostedToolingExtensions {
|
|
22
|
+
method_id?: string; // hosted-only — the platform resolves the stored method server-side
|
|
23
|
+
}
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
This is the same `MthdsFileItem` and the same `source` semantics the build routes use, so the [notes there](./build-routes.md#the-shared-envelope) apply verbatim: pass a filename per file and, when the engine can attribute a diagnostic to one, it comes back as `source` on the corresponding `validation_errors[]` item.
|
|
27
|
+
|
|
28
|
+
An **address-form** `method_ref` (`github.com/<owner>/<repo>[/<selector>][@<tag>]`) is resolved by the server through the same fetch path as a `method_ref` run: the repository fetched at the tag, the package located by manifest identity, the package's real relative paths feeding the per-file `source` labels. Any non-address reference stays reserved and answers `501`.
|
|
29
|
+
|
|
30
|
+
`method_id` is the hosted platform's selector, and it is a **pass-through**: nothing is expanded client-side — the platform resolves the id against the org's catalog and injects the stored source before the runner sees the request. It is meaningless against a bare runner (no catalog), and on `api.pipelex.com` its availability follows the platform deploy that adds the tooling-route transform. An unknown or foreign-org id is a `404` (indistinguishable by design); a stored method with no MTHDS source is a `422`.
|
|
31
|
+
|
|
32
|
+
Supplying **no** selector or **more than one** is a request-shape `422` — the tooling routes are stateless, so there is no linkage exception; a second selector could only be ignored, which is the worst contract of the three. The SDK does not model the XOR in the type system — the union would force the overwhelmingly common `{ files }` call site to pick a branch for no gain, and the server's answer is a typed `ApiResponseError` either way.
|
|
33
|
+
|
|
34
|
+
The old advice to expand a stored method client-side (`resolve({ files: await client.getMethodClosure(methodId) })`) remains valid — `getMethodClosure` stays public as the local expansion utility, and it is what a caller uses against a bare runner or on the routes with no by-id form (`/v1/build/*`, `prepareInputs`).
|
|
35
|
+
|
|
36
|
+
## `resolve` — the normalized crate
|
|
37
|
+
|
|
38
|
+
Resolution is a first-class language operation alongside validation: the closure is loaded and statically validated, then emitted with fully qualified refs, refinement flattened, natives materialized, and a fingerprint set.
|
|
39
|
+
|
|
40
|
+
```ts
|
|
41
|
+
const result = await client.resolve({ files: [{ content: src, source: "method.mthds" }] });
|
|
42
|
+
|
|
43
|
+
if (!result.is_valid) {
|
|
44
|
+
for (const err of result.validation_errors) console.error(`${err.source ?? "?"}: ${err.message}`);
|
|
45
|
+
return;
|
|
46
|
+
}
|
|
47
|
+
|
|
48
|
+
console.log(result.crate.fingerprint); // rides INSIDE the crate, not beside it
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
Two things to internalize:
|
|
52
|
+
|
|
53
|
+
- **`fingerprint` and `mthds_version` are crate members**, not siblings of `crate`.
|
|
54
|
+
- **Compare `fingerprint` values; never hash the crate yourself.** The fingerprint is a property of the _logical_ crate, not of an encoding: the server hashes `{concepts, pipes, domains}` with provenance `source` stripped, excluding `source_map`, `mthds_version` and `fingerprint` itself. And these are not the bytes `pipelex resolve --format json` prints — the CLI pretty-prints, the route answers compact JSON. Same logical crate, different serialization. Hashing `JSON.stringify(result.crate)` and comparing it against `result.crate.fingerprint` will mismatch on every call, and will look like tampering.
|
|
55
|
+
- **`crate` is typed as opaque transport** (`Record<string, unknown>`). Its schema belongs to the MTHDS standard; restating it here would be a second source of truth, free to drift from the one the server emits.
|
|
56
|
+
|
|
57
|
+
`resolve` runs **no dry-run sweep**. A valid verdict says the library resolves, never that it runs — runnability is `validate`'s vocabulary.
|
|
58
|
+
|
|
59
|
+
## `codegen` — stamped artifacts and the trust chain
|
|
60
|
+
|
|
61
|
+
`codegen` resolves the same way, then projects the crate through **two explicit axes**:
|
|
62
|
+
|
|
63
|
+
| Axis | Values | Notes |
|
|
64
|
+
| -------- | ------------------------------------------------------- | -------------------------------------------------------------- |
|
|
65
|
+
| `kind` | `types` | The crate's whole concept set. Per-pipe kinds are future work. |
|
|
66
|
+
| `target` | `ts-zod`, `python-pydantic`, `python-structures` | `ts-zod` is the natural one for TypeScript consumers. |
|
|
67
|
+
|
|
68
|
+
```ts
|
|
69
|
+
const result = await client.codegen({ files, kind: "types", target: "ts-zod" });
|
|
70
|
+
if (!result.is_valid) return;
|
|
71
|
+
|
|
72
|
+
for (const artifact of result.artifacts) await writeFile(artifact.path, artifact.content);
|
|
73
|
+
await writeFile(result.lock_filename, result.lock); // "codegen.lock", beside the artifacts
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
**The trust chain is the reason the lock rides along.** Write every artifact at its `path` and the `lock` content as `lock_filename`, both **verbatim**, and the resulting tree is byte-identical to what a local `pipelex codegen types` run produces — same stamps, same lock — so the offline `pipelex codegen check` passes on it. Reformatting an artifact, or re-serializing the lock through your own TOML writer, breaks that chain. There is deliberately **no** server-side check route: the check is offline by design.
|
|
77
|
+
|
|
78
|
+
`crate_fingerprint` and `engine_version` say what the artifacts were generated _from_ — the crate `resolve` would have returned for the same closure, and the pipelex engine that emitted them.
|
|
79
|
+
|
|
80
|
+
The SDK stays transport-only: it hands you the artifacts and does not write files for you. Verifying a committed tree later is the other half of the chain, and that half needs no server at all — see [the offline check](#the-offline-check--runcodegencheck) below.
|
|
81
|
+
|
|
82
|
+
### `pipe_ref` is rejected, not ignored
|
|
83
|
+
|
|
84
|
+
`kind: "types"` is concept-set-wide, so passing `pipe_ref` alongside it is a request-shape **`422`**. Silently ignoring the selector would mislead a caller into believing the artifacts were narrowed to one pipe. The field exists on the request for the future per-pipe kinds.
|
|
85
|
+
|
|
86
|
+
## The response is a verdict, not a payload
|
|
87
|
+
|
|
88
|
+
Both routes return a **discriminated 200**, the same discipline as `validate` and the build routes: an unresolvable closure is the _successful product_ of the call (the request was well-formed; the library was not), so it rides a 200 with `is_valid: false` and the shared `CrateInvalidReport` — the very same invalid arm the build routes return, carrying the same structured `validation_errors[]`.
|
|
89
|
+
|
|
90
|
+
**Branch on `is_valid` before reading the arm.** A consumer that only catches throws will render a success over an unusable result, because nothing threw.
|
|
91
|
+
|
|
92
|
+
## What throws
|
|
93
|
+
|
|
94
|
+
Only a **no-verdict** condition, as the typed `ApiResponseError` — branch on its `status`, never on its message:
|
|
95
|
+
|
|
96
|
+
| Status | Cause |
|
|
97
|
+
| ------------- | --------------------------------------------------------------------------------------------------------- |
|
|
98
|
+
| `422` | Request shape: no closure selector or more than one, an unknown `kind`/`target`, a `pipe_ref` on `kind: "types"`, a stored method with no MTHDS source. |
|
|
99
|
+
| `404` | Unknown or foreign-org `method_id` (indistinguishable by design). |
|
|
100
|
+
| `501` | Registry-form `method_ref` — reserved, not implemented (the address form resolves). |
|
|
101
|
+
| `401` / `403` | Auth. |
|
|
102
|
+
| `5xx` | Server fault. |
|
|
103
|
+
|
|
104
|
+
Note the split, same as the build routes: a bad **closure** is a 200 verdict; a bad **request** throws.
|
|
105
|
+
|
|
106
|
+
(The offline check below is not a route and throws its own `CodegenLockError` instead — it never reaches the network.)
|
|
107
|
+
|
|
108
|
+
## The offline check — `runCodegenCheck`
|
|
109
|
+
|
|
110
|
+
A consumer that commits the generated tree needs a CI gate over it. That gate is `runCodegenCheck`, and it is **pure**: no filesystem, no network, no API key, no `PipelexApiClient`. You walk your own tree and hand in the text; the SDK owns the verdict.
|
|
111
|
+
|
|
112
|
+
The reason it is a separate, engine-free step is the same reason there is no server-side check route. Regeneration is a **dev action** — it needs the engine, so it is `codegen` above. The check is a **CI action** — it needs only a hash function. Keeping them apart is what stops an upstream template improvement from reddening a consumer's CI on a tree nobody touched.
|
|
113
|
+
|
|
114
|
+
```ts
|
|
115
|
+
import { isStampableArtifactPath, runCodegenCheck } from "@pipelex/sdk";
|
|
116
|
+
|
|
117
|
+
const report = await runCodegenCheck({
|
|
118
|
+
lockContent: await readFile("src/generated/echo/codegen.lock", "utf8"),
|
|
119
|
+
// Every file under the lock's directory, recursively, with paths relative to it.
|
|
120
|
+
files: await readTree("src/generated/echo", isStampableArtifactPath),
|
|
121
|
+
});
|
|
122
|
+
|
|
123
|
+
if (!report.isCurrent) {
|
|
124
|
+
for (const drift of report.drifts) {
|
|
125
|
+
console.error(`${drift.category}: ${drift.path} — ${drift.detail}`);
|
|
126
|
+
}
|
|
127
|
+
process.exit(1);
|
|
128
|
+
}
|
|
129
|
+
```
|
|
130
|
+
|
|
131
|
+
A `codegen()` response feeds straight in with no mapping — `GeneratedArtifact` and `CodegenTreeFile` are structurally identical on purpose, so `runCodegenCheck({ lockContent: result.lock, files: result.artifacts })` type-checks and reports `isCurrent`. That is worth doing right after a regeneration, before writing anything to disk.
|
|
132
|
+
|
|
133
|
+
The function is `async` because it hashes through **WebCrypto** (`crypto.subtle`) rather than `node:crypto`, so the check adds no Node builtin to the barrel's import graph. (That is not the same as the barrel being browser-bundleable today: `upload.ts` still names `node:fs/promises`, so a browser-targeting bundler must mark `node:*` external.) One caveat that never bites a CI script but should not surprise anyone: `crypto.subtle` is secure-context-only, so a browser page must be served over https (or localhost).
|
|
134
|
+
|
|
135
|
+
### The algorithm
|
|
136
|
+
|
|
137
|
+
Each locked artifact is located in the supplied tree and the hash of the body **below its stamp** is recomputed. Absent is a drift; a body that no longer hashes to what the stamp or the lock records is a drift. Then, in the other direction, a stamped file the lock does not track is a drift — the stale-artifact class a per-file stamp can never catch on its own, and the reason the lock rides along with the artifacts at all.
|
|
138
|
+
|
|
139
|
+
The verdict is a structured report, never an exit code alone: `drifts` enumerates the drifting artifacts by category, and `isCurrent` is exactly `drifts.length === 0`. Drift order is deterministic — locked-artifact drifts first, then orphans, each sorted by path — so a consumer can print it, snapshot it, or diff two runs.
|
|
140
|
+
|
|
141
|
+
The report also carries `crateFingerprint` and `engineVersion`, read off the lock header. The check itself never compares them against anything (that would need the engine), but surfacing them is what lets a caller ask the question the check deliberately does not: _is this committed tree even from the crate my method resolves to today?_ — a live `codegen()` response's `crate_fingerprint` is the value to compare against.
|
|
142
|
+
|
|
143
|
+
### The drift taxonomy
|
|
144
|
+
|
|
145
|
+
| Category | What it means |
|
|
146
|
+
| ------------- | ----------------------------------------------------------------------------------------------------- |
|
|
147
|
+
| `missing` | Listed in the lock, absent from the tree you handed in. |
|
|
148
|
+
| `modified` | Present, stamp self-consistent, but the body no longer matches the **locked** hash — regenerate. |
|
|
149
|
+
| `hand-edited` | Present, but its stamp is gone, unparseable, or disagrees with the body below it — someone edited it. |
|
|
150
|
+
| `orphan` | A stamped file the lock does not track — yesterday's artifact, left behind. Remove or regenerate. |
|
|
151
|
+
|
|
152
|
+
Two properties of that table are load-bearing and are pinned by tests:
|
|
153
|
+
|
|
154
|
+
- **At most one drift per locked path, and `hand-edited` outranks `modified`.** A hand edit trips both conditions; reporting it twice would print the same file under two contradictory categories.
|
|
155
|
+
- **An orphan only has to _look_ stamped.** A stray whose stamp is corrupt below the begin-marker line still counts — using the stricter parse there would silently ignore exactly the stale file the lock exists to catch.
|
|
156
|
+
|
|
157
|
+
The category values are the canonical strings `pipelex codegen check` uses, and so are the `detail` sentences, verbatim. A consumer switching between the CLI and this helper reads the same report.
|
|
158
|
+
|
|
159
|
+
### Where it knowingly differs from the CLI
|
|
160
|
+
|
|
161
|
+
Verdict parity is the design constraint, and the stamp-header text rules mirror Python's exactly — the line-boundary set, the strip set, the drive-prefix rule, and the comment-prefix gate over the header region are all matched deliberately, each pinned by a test. These differences remain:
|
|
162
|
+
|
|
163
|
+
- **The projection line is shape-checked, not vocabulary-checked.** pipelex resolves `kind` and `target` against its own enums, which cannot lag its own emitter; an SDK copy can. So a stamp reading `projection: types / rust-serde` from a newer engine verifies here and would be `hand-edited` there. Tightening it would report every artifact of such a tree as a hand edit — the failure mode is worse than the gap, and it is pinned in both directions.
|
|
164
|
+
- **The "not valid UTF-8" branch cannot arise.** `content` reaches the check already decoded, so the caller owns that verdict — see the decode obligation above.
|
|
165
|
+
- **A TOML float is accepted as a lock version, where pipelex calls it malformed.** `lock_version = 1.0` is a float upstream and not an `int`, so the CLI refuses it; the TOML parser here decodes it to the same `1` as the integer, so this reader cannot tell them apart and reads version 1. Not deliberate so much as unavoidable, and unreachable with any emitter — the divergence is a no-verdict there against a verdict here, on a lock nothing writes.
|
|
166
|
+
|
|
167
|
+
### The lock format version
|
|
168
|
+
|
|
169
|
+
`codegen.lock` opens with `lock_version`, and this build reads version 1. The field exists because the reader is deliberately strict *within* a version — an unknown key is a malformed lock, not a field to ignore — and that strictness is unworkable *across* versions without it.
|
|
170
|
+
|
|
171
|
+
Four rules, mirroring the CLI exactly:
|
|
172
|
+
|
|
173
|
+
- **A lock with no `lock_version` key is version 1.** The field was introduced with version 1, so every lock written before it existed is already conformant and nothing on disk needs migrating.
|
|
174
|
+
- **The version is read _before_ the key set is validated.** This is the load-bearing ordering. The other way round, a lock from a newer codegen is rejected over whichever new key it happens to carry — an opaque shape complaint about a key the writer was entitled to add, instead of a verdict naming the version and saying what to upgrade.
|
|
175
|
+
- **A version this build does not know is refused**, with a message that names the version found and which side to upgrade. A version *greater* than the one read means the lock came from a newer codegen; anything else (`0`, negative, non-integer) is malformed rather than futuristic.
|
|
176
|
+
- **Strictness within a known version stays.** An unexpected key in a version-1 lock is still a no-verdict.
|
|
177
|
+
|
|
178
|
+
So the upgrade path is: publish an SDK that tolerates the new version *before* the pipelex release that starts writing it. A reader that has not learned a version yet fails loudly and actionably rather than silently guessing at a shape it was not written for.
|
|
179
|
+
|
|
180
|
+
### The caller's obligations
|
|
181
|
+
|
|
182
|
+
The pure input moves the tree-reading obligations onto you. Each one, unmet, produces a **wrong verdict** rather than an error — so none of them can be left implicit:
|
|
183
|
+
|
|
184
|
+
- **Pass each file's text as read, without reformatting it.** The hash is over exact UTF-8 bytes, and the stamp parser requires the text to _start with_ the begin-marker line. Re-encoding, inserting a BOM, or running Prettier over a generated artifact all report `hand-edited`.
|
|
185
|
+
- **Line endings are the one exception, and they are handled for you.** `\r\n` and lone `\r` are normalized to `\n` before anything is parsed or hashed, mirroring the universal-newline translation pipelex's own reader applies. A Windows checkout under `core.autocrlf=true` is therefore **not** a false hand-edit here, exactly as it is not one for the CLI. Committing generated artifacts with a `.gitattributes` entry (`src/generated/** -text`) is still worth doing for diff hygiene; it is not load-bearing for the verdict.
|
|
186
|
+
- **Walk the whole tree, recursively, from the lock's directory**, and pass paths relative to it. An incomplete list yields a wrong verdict in either direction: an omitted _locked_ file is reported `missing` though it sits on disk, and an omitted _orphan_ is never seen at all — `isCurrent: true`, a false negative on precisely the drift class orphan detection exists for. Filter your walk with `isStampableArtifactPath` (or `STAMPABLE_ARTIFACT_SUFFIXES`) so it picks up exactly what the check considers; a file of any other type is skipped rather than rejected, which is what lets you park a sidecar such as `sources.json` beside the lock. Pruning vendor and VCS directories and skipping symlinks is walk policy and stays with you — moot for a per-method generated directory, which holds nothing else.
|
|
187
|
+
- **Decode the bytes yourself.** `content` is already a `string`, so pipelex's "not valid UTF-8, therefore not generated output" branch cannot arise here. A file you could not decode is not generated output: report it yourself, or leave it out and accept the `missing` drift.
|
|
188
|
+
|
|
189
|
+
### What it deliberately does not verify
|
|
190
|
+
|
|
191
|
+
It never compares a stamp's `crate_fingerprint` against the lock's, and it never re-resolves the crate. Both need the engine, which is the whole point of the offline split — and both are questions the caller can ask itself with `crateFingerprint` in hand.
|
|
192
|
+
|
|
193
|
+
### What the check throws
|
|
194
|
+
|
|
195
|
+
`CodegenLockError`, and only for a **no-verdict** condition: a malformed lock, a lock format version this build cannot read, an unknown key in it, or a path — in the lock or in your `files` — that is not a safe canonical artifact path (absolute, drive-prefixed, backslashed, control-charactered, `..`-bearing, empty, duplicated, or, for a locked path, of a type codegen does not stamp). None of these is a drift: the check could not produce a verdict at all, which is a distinct outcome and typically a distinct exit code. It deliberately does not derive from `PipelineRequestError` — nothing was requested over the wire.
|
|
196
|
+
|
|
197
|
+
A `"./types.ts"` spelling is worth calling out, because it is the one a hand-rolled walk produces by accident: left to resolve as-is it would report both a `missing` and an `orphan` for the same file, so it throws instead.
|
|
@@ -0,0 +1,186 @@
|
|
|
1
|
+
# Input preparation (`uploadFile` / `prepareInputs`)
|
|
2
|
+
|
|
3
|
+
> **Status: implemented** (`src/upload.ts`, `src/prepare-inputs.ts`). This document records the contract, and is the public account of it. The raw `upload()` primitive described in [architecture.md](./architecture.md) is the wire call `uploadFile` and `prepareInputs` build on.
|
|
4
|
+
>
|
|
5
|
+
> **Current scope.** `prepareInputs` names the method the same three ways every other method-taking operation does — inline `files`, a `method_ref` address, or a stored `method_id` — exactly one per call, all three resolved server-side by the one `validate` call it composes. One piece remains deliberately deferred and additive (it does not change this contract): the opt-in ingest of `http(s)` URLs into storage — for now an `http(s)` URL at a file position always passes through unchanged.
|
|
6
|
+
|
|
7
|
+
## Why this exists
|
|
8
|
+
|
|
9
|
+
A hosted run cannot see the caller's filesystem or a browser's selected bytes. Turning caller-local assets into run-ready inputs is therefore the SDK's job, not the runner's — the SDK process is the only component that can read the local file or hold the browser bytes. Today that work is re-implemented by every consumer (read file → base64 → `POST /v1/upload` → rewrite the input to the returned URI). `prepareInputs` makes it one reusable, explicit operation.
|
|
10
|
+
|
|
11
|
+
Preparation is **explicit and separate from running.** `execute` / `start` never silently upload local files. The payoff: file-access errors happen *before* a run exists, prepared inputs are inspectable and reusable across a model sweep or retries without re-uploading, and `start` keeps a deterministic JSON-input contract.
|
|
12
|
+
|
|
13
|
+
## The two operations
|
|
14
|
+
|
|
15
|
+
### `uploadFile` — single-asset convenience
|
|
16
|
+
|
|
17
|
+
Uploads one asset and returns its upload record. It is the language-native convenience over the raw `upload()` wire call (base64 JSON body), assembling the record client-side.
|
|
18
|
+
|
|
19
|
+
- **Accepted sources (every runtime — browser, Node, edge):** `Blob`, `File`, `ArrayBuffer`, `Uint8Array`.
|
|
20
|
+
- **Path strings are Node-only.** A filesystem path string is read from disk, which only a process that owns a filesystem can do. In a non-Node runtime a path string fails instructively rather than being misread — the SDK does not silently treat it as text or as a URL.
|
|
21
|
+
- A string that is an **HTTP(S) URL** or an existing **`pipelex-storage://` URI** is not a local asset and passes through in any runtime (see pass-through rules below).
|
|
22
|
+
- Open file objects and streams are **deferred** — they can be added later without removing anything.
|
|
23
|
+
|
|
24
|
+
> **Runtime acceptance vs. bundling.** These source types are accepted at *runtime* in every environment, but at the *packaging* level `@pipelex/sdk` is **Node-first**: it statically references `node:fs/promises` (to read path strings), so bundling the SDK for a browser or edge target currently requires marking `node:*` external. This is a deliberate stance — every consumer runs the SDK server-side in Node (Next.js Server Actions, the MCP server, the plugins hook) — revisited only if the SDK is ever imported into a browser client bundle or an edge route.
|
|
25
|
+
|
|
26
|
+
The returned **upload record** guarantees, beyond the source identity:
|
|
27
|
+
|
|
28
|
+
| Field | Guarantee |
|
|
29
|
+
| --- | --- |
|
|
30
|
+
| `uri` | The `pipelex-storage://` reference for the uploaded asset. |
|
|
31
|
+
| `contentType` (MIME) | Known client-side at upload time. |
|
|
32
|
+
| `size` (bytes) | Known client-side at upload time. |
|
|
33
|
+
| `filename` | Already in the wire model. |
|
|
34
|
+
| checksum | **Not present.** Within-preparation dedup relies on source identity, not hashing; cross-preparation dedup is a hosted storage-policy concern (Phase 5). |
|
|
35
|
+
|
|
36
|
+
The MIME type and size are known client-side, so the record is assembled without extending the `/v1/upload` response.
|
|
37
|
+
|
|
38
|
+
### `prepareInputs` — signature-driven input preparation
|
|
39
|
+
|
|
40
|
+
```
|
|
41
|
+
client.prepareInputs({ files, pipe_ref?, inputs }) → PreparedInputs
|
|
42
|
+
client.prepareInputs({ method_ref, pipe_ref?, inputs }) → PreparedInputs
|
|
43
|
+
client.prepareInputs({ method_id, pipe_ref?, inputs }) → PreparedInputs
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
Takes the **method** as exactly one of three selectors, the optional target **pipe**, and the caller's `inputs`; resolves the pipe's declared input signature; interprets the inputs top-down against it; uploads the file-bearing values; and returns `PreparedInputs`. Per input, the caller may submit **either** the compact value **or** the explicit `{ concept, content }` envelope — see "[Compact or explicit-envelope inputs](#compact-or-explicit-envelope-inputs)" below:
|
|
47
|
+
|
|
48
|
+
- `inputs` — a **copy** of the caller's inputs with each asset reference replaced by the canonical content shape carrying `pipelex-storage://` in its `url` field (see "Rewritten-input shape" below). Copy-on-write: the caller's original object is never mutated.
|
|
49
|
+
- `uploads` — one upload record per prepared asset (the `uploadFile` record shape), exposing `uri` so callers can log which source became which reference without reverse-engineering the rewritten object.
|
|
50
|
+
|
|
51
|
+
The prepared `inputs` are passed to the existing run lifecycle unchanged.
|
|
52
|
+
|
|
53
|
+
#### The three selectors
|
|
54
|
+
|
|
55
|
+
Exactly one per call. The type pins the other two to `never`, so a second selector is a compile error; an untyped caller gets an `InputPreparationError` naming the three forms. **Empty is absent** — `files: []`, `method_ref: ""`, `method_id: " "` — mirroring the run options' rule, so an empty selector may sit beside a real one without tripping the XOR.
|
|
56
|
+
|
|
57
|
+
| Selector | What it is | Who resolves it |
|
|
58
|
+
| --- | --- | --- |
|
|
59
|
+
| `files` | the inline MTHDS closure (`{content, source?}` entries) | nobody — inline |
|
|
60
|
+
| `method_ref` | a published method's address, `github.com/<owner>/<repo>[/<selector>][@<tag>]` | the runner, server-side (pipelex-api >= 0.21.0 fetches the repository at the tag) |
|
|
61
|
+
| `method_id` | a stored method's catalog id (`mt_…`) | the hosted platform, which injects the stored source before the runner sees the request |
|
|
62
|
+
|
|
63
|
+
Nothing is expanded client-side: `method_id` here is a **pass-through**, the same rule every other id-taking operation follows. `getMethodClosure` stays available for callers that want the files in hand, but it is no longer a step on the way to preparing inputs.
|
|
64
|
+
|
|
65
|
+
#### Where the signature comes from
|
|
66
|
+
|
|
67
|
+
One `POST /v1/validate` per call, whatever the selector, asking for the **input-form descriptor**:
|
|
68
|
+
|
|
69
|
+
```
|
|
70
|
+
validate(<selector or contents>, allowSignatures = true, …, views = ["input_form"])
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
`allowSignatures: true` is deliberate. Preparation needs a pipe's *declared* inputs, and a bundle mid-authoring with an unresolved signature somewhere else must not be refused inputs for a pipe whose inputs are declared — whether the bundle runs is the run's verdict, not preparation's. An `is_valid: false` verdict still means the closure does not load, which is a preparation failure.
|
|
74
|
+
|
|
75
|
+
A `method_ref` makes the server clone a repository first; `validate` needs no special budget for it, because the route already defaults to the 20-minute execute ceiling.
|
|
76
|
+
|
|
77
|
+
#### Pipe selection
|
|
78
|
+
|
|
79
|
+
`validate` has no pipe selector — its report describes every pipe, keyed by qualified `pipe_ref` — so the helper picks one, in this order:
|
|
80
|
+
|
|
81
|
+
1. **`pipe_ref` when given.** Qualified-only: `domain.pipe_code`. A bare code, or a ref the method does not declare, is an `InputPreparationError` listing the qualified refs — one step to fix. The helper never grows a searched `pipe_code`: search is a run-route affordance, and the descriptor is keyed by qualified refs. An `alias->domain.pipe_code` ref is refused for a different reason: the descriptor describes the method's *own* pipes, and an alias names one belonging to a dependency package, which the report never carries a descriptor for — the run route takes such a ref, preparation cannot.
|
|
82
|
+
2. **The report's typed resolved default** (`default_pipe_ref`), once the runner serves it: the ref a caller gets by omitting the selector on the run and build routes, manifest-aware for a fetched package. Read when present; a server that predates it sends nothing.
|
|
83
|
+
3. **The bundle's declared `main_pipe`**, read defensively from the opaque `bundle_blueprint` and qualified by its `domain`.
|
|
84
|
+
4. **The single pipe**, when the method declares exactly one.
|
|
85
|
+
5. Otherwise an `InputPreparationError` naming the candidates and asking for `pipe_ref`.
|
|
86
|
+
|
|
87
|
+
> **The manifest-only `main_pipe` gap.** A published package may name its entry pipe in `METHODS.toml` alone — `github.com/Pipelex/methods/documents` and `.../image_generation` do — and the validate report never carries a manifest. Until step 2's field ships, such a package needs an explicit `pipe_ref`; the error lists the candidates, so the fix is one line.
|
|
88
|
+
|
|
89
|
+
## `getMethodClosure` — the explicit expansion utility
|
|
90
|
+
|
|
91
|
+
```
|
|
92
|
+
client.getMethodClosure(methodId) → MthdsFileItem[]
|
|
93
|
+
```
|
|
94
|
+
|
|
95
|
+
`getMethodClosure` is the public **local expansion utility** — a client-side semantic layer over `getMethod` (the platform has no route that returns a parsed closure). It fetches the method, parses its polymorphic `mthds` source with [`methodSourceToContents`](#methodsourcetocontents--the-canonical-source-parser), and labels each resulting file with the `method_id` as its `source` provenance.
|
|
96
|
+
|
|
97
|
+
Reach for it whenever you want the files **in hand** — to edit, to diff, or to feed a route with no by-id form:
|
|
98
|
+
|
|
99
|
+
```
|
|
100
|
+
client.buildInputs({ files: await client.getMethodClosure("mt_…"), pipe_ref?, format?, explicit? })
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
`/v1/build/*` is the last such family, and it is being retired. Every other method-taking operation — `execute` / `start`, `validate` / `resolve` / `codegen`, and now `prepareInputs` — takes a `method_id` **natively**, as a server pass-through: the platform resolves it and injects the stored source before the runner sees the request (see [architecture.md → hosted run extensions](./architecture.md#hosted-run-extensions-method_id)). That is the uniform rule — *an id option is always a server pass-through, and any client-side expansion is the caller's own explicit call*. An untyped caller still passing `method_id` to `buildInputs` gets a teaching error naming this migration (`PipelineRequestError`); for typed callers the field is a compile error.
|
|
104
|
+
|
|
105
|
+
- **Requires an API key.** The methods catalog is org-scoped to the key's org, so resolution only works with an authenticated Pipelex-product key.
|
|
106
|
+
- **Unknown or foreign-org id → the `getMethod` `404`** (`ApiResponseError`, `code: "not_found"`), propagated unchanged. An id from another org is indistinguishable from a nonexistent one — both 404.
|
|
107
|
+
- **A real, in-org method whose source parses to nothing → `EmptyMethodSourceError`** — the row exists but has no runnable MTHDS source yet. This is a distinct outcome from the 404 (see "[Error and capability behavior](#error-and-capability-behavior)"), and `getMethodClosure` is now its only raiser: `prepareInputs` passes the id to the server, which answers a sourceless method with a `422`.
|
|
108
|
+
|
|
109
|
+
### `methodSourceToContents` — the canonical source parser
|
|
110
|
+
|
|
111
|
+
```
|
|
112
|
+
methodSourceToContents(mthds: string) → string[]
|
|
113
|
+
```
|
|
114
|
+
|
|
115
|
+
A stored method's `mthds` field is **polymorphic**: it is either the raw single-bundle source, or a JSON array of `{ name, content }` file entries (a multi-file closure). `methodSourceToContents` is the one canonical parser that turns either shape into a flat list of file contents — a verbatim port of the platform's `_method_source_to_contents`, so the SDK and the runtime read a stored source identically. It is exported from the barrel for consumers (e.g. `pipelex-mcp`) that need the parse without the fetch. Blank entries are dropped; a source that parses to no non-blank content yields `[]`, which is what `getMethodClosure` turns into `EmptyMethodSourceError`.
|
|
116
|
+
|
|
117
|
+
## Compact or explicit-envelope inputs
|
|
118
|
+
|
|
119
|
+
Each input may be submitted in **either** of two shapes, and preparation treats them equivalently:
|
|
120
|
+
|
|
121
|
+
- **Compact** — the bare value: a source string / bytes / canonical `{ url }` content (e.g. `photo: "https://…/p.png"`).
|
|
122
|
+
- **Explicit envelope** — the `{ concept, content }` shape that `buildInputs({ explicit: true })` returns per input (e.g. `photo: { concept: "native.Image", content: { url: "https://…/p.png" } }`). This is the default template the hosted console and MCP hand agents to fill, so an agent that fills the template can hand it straight back.
|
|
123
|
+
|
|
124
|
+
When a value is an envelope (a plain object whose keys are **exactly** `concept` and `content` — matching the runtime's `_is_explicit` in `input_shaper.py`), preparation unwraps `content`, interprets it exactly as the compact value would be, and **re-wraps** the result — so the concept annotation rides through to the run. The runtime accepts the envelope as a run input (`input_shaper.py` `_is_explicit` / `_shape_explicit`), and file-reference resolution (`input_normalizer.py`) runs over the already-shaped content, so a re-wrapped `{ concept, content: { url: "pipelex-storage://…" } }` resolves identically to the compact `{ url }`. The envelope's `content` may itself be a scalar, a canonical file content, a list, or a structured object nesting file fields — the same top-down walk applies underneath.
|
|
125
|
+
|
|
126
|
+
## Signature-driven asset identification
|
|
127
|
+
|
|
128
|
+
The SDK **must not** guess that every string resembling a path is an asset — that would make ordinary text inputs environment-dependent and could upload unintended files. Interpretation comes from the method's **declared signature**, never from a value's shape alone. This mirrors the runtime's own top-down interpretation (`InputShaper`) combined with the file-reference resolution of `input_normalizer` in `pipelex`, so local and hosted execution read the same inputs the same way. (A caller value may be compact or the explicit `{ concept, content }` envelope — see "[Compact or explicit-envelope inputs](#compact-or-explicit-envelope-inputs)"; the walk below is over the compact value, i.e. the envelope's `content`.)
|
|
129
|
+
|
|
130
|
+
The declared signature is the **input-form descriptor** — the MTHDS standard's `input_form` artifact, carried on the validate report under `views: ["input_form"]`. It states the kind of every input at every depth, so a file position is a *fact of the method*, never an inference from the value in front of it.
|
|
131
|
+
|
|
132
|
+
The walk is discriminated on the node's `kind`:
|
|
133
|
+
|
|
134
|
+
| Node kind | What the walk does |
|
|
135
|
+
| --- | --- |
|
|
136
|
+
| `document`, `image` | A **file position.** The caller's value there is resolved: a local path, `data:` URL, or bytes is uploaded and rewritten to canonical content carrying `pipelex-storage://` in `url`; an `http(s)` URL or an existing storage URI passes through; a canonical `{ url }` content dict has its `url` resolved and its other keys preserved. |
|
|
137
|
+
| `object` | Walks the declared `fields` by name against a plain-object value. Keys the descriptor does not name are copied through untouched. **Optional fields (`required: false`) are walked when present.** |
|
|
138
|
+
| `list` | Walks `item` against each element of an array value. |
|
|
139
|
+
| everything else (`text`, `prose`, `date`, `number`, `boolean`, `enum`, `unknown`) | Passes through at any depth. |
|
|
140
|
+
|
|
141
|
+
Two consequences worth stating outright, because they are what the descriptor buys:
|
|
142
|
+
|
|
143
|
+
- **Whether a concept is native or refines `Document` / `Image` is the descriptor's statement** (`refines`), never a shape test. A concept refining `native.Document` arrives as `kind: "document"` and is prepared as one.
|
|
144
|
+
- **`unknown` is not interpreted.** It is the standard's escape hatch for a `Dynamic` / `Composite` input, and the walk does not enter it — including a canonical file dict nested inside one. The signature genuinely declares no file there, and uploading on the strength of a `url` key is precisely the guess this helper stopped making. A caller with such an input uploads with `uploadFile` first and passes the storage URI.
|
|
145
|
+
|
|
146
|
+
A repeated reference to the **same source** within one preparation is uploaded once and rewritten consistently (within-preparation dedup by source identity). A caller value whose shape disagrees with the node — a scalar at an `object`, a non-array at a `list` — passes through for the run to reject; preparation never second-guesses the signature.
|
|
147
|
+
|
|
148
|
+
> **Why the descriptor, and why not the template.** Preparation used to read the *explicit inputs template* (`buildInputs` with `explicit: true`), whose file signal was a rendered `{"url": …}` dict. That signal is a side effect of a field being **named** `url` or `*_url`, not of the field being an Image or a Document — so a text field named `url` was read off the caller's disk and uploaded, and an **optional** nested file field, which the required-only template never rendered, was invisible and travelled to the runner as a literal path. Both were live misclassifications. The descriptor states the kind instead of implying it, and states it at every depth, so both edges fall out correctly rather than needing special cases. It costs the same one round trip, it is the standard's own artifact rather than a presentation of it, and it is what `@pipelex/mthds-form`, `pipelex-app`, `mthds-ui` and `pipelex-mcp` already derive from. The template remains what it always was — a fill-in scaffold for a person or an agent — and is projected from this same descriptor.
|
|
149
|
+
|
|
150
|
+
### Pass-through rules
|
|
151
|
+
|
|
152
|
+
| Source at a file-bearing input | Action |
|
|
153
|
+
| --- | --- |
|
|
154
|
+
| Local path (Node) / data URL / bytes | Upload → rewrite to `pipelex-storage://` |
|
|
155
|
+
| Existing `pipelex-storage://` URI | Already prepared — pass through unchanged |
|
|
156
|
+
| HTTP(S) URL | Pass through unchanged, **unless** the caller explicitly asks to ingest it into Pipelex storage |
|
|
157
|
+
|
|
158
|
+
## Rewritten-input shape: `url` carries the URI
|
|
159
|
+
|
|
160
|
+
The runtime's canonical image/document content stores its reference in a **`url`** field. Preparation emits inputs the runtime interprets natively, so a rewritten input keeps the canonical content shape with `url` holding the `pipelex-storage://` value — exactly what the runtime's `input_normalizer` writes.
|
|
161
|
+
|
|
162
|
+
The "uploaded reference is named `uri`" decision applies to the **upload surface**: the raw upload result and each upload record expose the storage reference as `uri`. Preparation must **not** invent a `uri` field inside rewritten inputs — that would produce inputs the runtime does not recognize.
|
|
163
|
+
|
|
164
|
+
## Error and capability behavior
|
|
165
|
+
|
|
166
|
+
Upload is a **hosted Pipelex-product capability**, even though the SDK can be pointed at other base URLs. A deployment that does not support upload must produce a specific, actionable error — preparation must never silently leave a local path in place and let a later run fail obscurely.
|
|
167
|
+
|
|
168
|
+
The contract distinguishes these semantic outcomes, each a typed subclass of `InputPreparationError` (catch the base to handle any preparation failure, a subclass to branch on category):
|
|
169
|
+
|
|
170
|
+
- **an unresolvable signature** (`InputPreparationError`) — the closure did not validate (`is_valid: false`, carrying the first error's message), the report carried no `input_form` descriptor, or no pipe could be chosen (an unknown or bare `pipe_ref`, or no default). A *no-verdict* condition from `/v1/validate` — a malformed selector, an unknown or foreign-org `method_id`, no package at the address, auth, a server fault — stays an `ApiResponseError` and propagates unchanged;
|
|
171
|
+
- **empty method source** (`EmptyMethodSourceError`, carries `methodId`) — `getMethodClosure` found the stored method but its `mthds` source parses to nothing (the row exists, no runnable source yet). Distinct from the `getMethod` `404` for an unknown/foreign id, which stays an `ApiResponseError`. `prepareInputs` never raises it: it hands the id to the server, which answers a sourceless method with a `422`;
|
|
172
|
+
- **invalid local source** (`InvalidLocalSourceError`) — missing, unreadable, or a path string outside Node;
|
|
173
|
+
- **rejected asset** (`RejectedAssetError`) — the server refused it (e.g. a `413` past the service-defined size cap — see "Storage policy" — surfaced as a clear rejection, not a raw transport error);
|
|
174
|
+
- **unsupported server capability** (`UnsupportedUploadCapabilityError`) — the configured deployment has no upload route;
|
|
175
|
+
- **authentication / authorization failure** (`UploadAuthenticationError`) — `401` / `403`;
|
|
176
|
+
- **transport failure** (`UploadTransportError`) — a network or server fault, a malformed data URL payload, or any other unexpected upload failure.
|
|
177
|
+
|
|
178
|
+
All preparation failures are raised **before any run is created**.
|
|
179
|
+
|
|
180
|
+
## Storage policy (inherited, Phase 1)
|
|
181
|
+
|
|
182
|
+
The SDK ships against **today's route behavior**: a service-defined size cap (hosted default 50 MiB via `MAX_UPLOAD_MIB`, rejected with `413`), auth required, per-user keys, and nothing else — no MIME validation, retention, quotas, dedup, or cleanup. The SDK documents limits as **service-defined** and surfaces server rejections as clear "rejected asset" errors; it does **not** hardcode a client-side cap. Real storage policy (retention, quotas, org scoping, cleanup) is a later hosted-owner deliverable.
|
|
183
|
+
|
|
184
|
+
## Stability across the future endpoint move
|
|
185
|
+
|
|
186
|
+
The public abstraction sits deliberately **above** the HTTP route. Callers depend on `uploadFile` / `prepareInputs`, the `uri` result field, and the `pipelex-storage://` scheme — never on which backend service owns the route. The current transport is `POST /v1/upload` on `pipelex-api`; when hosted storage upload later moves to `pipelex-platform` (together with its paired resolution route, as one storage domain), the public path and wire shape are kept compatible so released SDK versions keep working, and any wire-protocol change is absorbed inside the SDK's upload transport. `uploadFile`, `prepareInputs`, and the prepared run-input shape stay stable across that move.
|
|
@@ -0,0 +1,159 @@
|
|
|
1
|
+
# Reading a run's results
|
|
2
|
+
|
|
3
|
+
A completed run hands back one object, `RunResults` (`src/runs.ts`), and every field of it is described here. The same object comes back from `startAndWaitForResult`, from `waitForResult(runId)`, and from the `completed` arm of `getRunResult(runId)` — one accessor set whichever path ran, which is the point of the type.
|
|
4
|
+
|
|
5
|
+
Two paths produce it. Against the hosted API the SDK starts a durable run and polls `GET /v1/runs/{id}/results`, where the platform relays the run's S3 artifacts verbatim. Against a bare `pipelex-api` runner, which has no run store, the SDK falls back to the blocking `POST /v1/execute` and maps the runner's native `pipe_output` onto the same shape, lifting the artifacts that ride it onto their own fields. `startAndWaitForResult` picks between the two from the `GET /v1/version` handshake, so a consumer does not choose.
|
|
6
|
+
|
|
7
|
+
| field | type | hosted (durable) path | bare-runner (blocking) path |
|
|
8
|
+
|---|---|---|---|
|
|
9
|
+
| `pipeline_run_id` | `string` | the run store's id | the runner's own id for the call |
|
|
10
|
+
| `main_stuff` | `unknown` | the `main_stuff.json` artifact | resolved out of the returned working memory |
|
|
11
|
+
| `working_memory` | `DictWorkingMemory \| null` | the `working_memory.json` artifact | lifted off `pipe_output` |
|
|
12
|
+
| `graph_spec` | `unknown` | the `graphspec.json` artifact | lifted off `pipe_output` |
|
|
13
|
+
| `graph_assembly_error` | `string \| null \| undefined` | absent until the platform relays it | lifted off `pipe_output` |
|
|
14
|
+
| `tokens_usages` | `TokensUsageRecord[] \| null` | the `tokens_usages.json` artifact | lifted off `pipe_output` |
|
|
15
|
+
| `usage_assembly_error` | `string \| null` | relayed | lifted off `pipe_output` |
|
|
16
|
+
| `pipe_output` | `DictPipeOutput \| null \| undefined` | absent | the runner's whole native output |
|
|
17
|
+
|
|
18
|
+
## `pipeline_run_id` — the durable handle
|
|
19
|
+
|
|
20
|
+
The run id is what makes a run readable after the process that started it has gone. `start` returns it in its acknowledgement before the run finishes, and every lifecycle read takes it: `getRunStatus(runId)` for the status row, `getRunResult(runId)` for a single result lookup, `waitForResult(runId)` to resume polling a run an earlier session started. It is also what a `RunTimeoutError` leaves you with — the run keeps executing server-side, so the timeout is a reason to re-poll by id, not a reason to run the method again.
|
|
21
|
+
|
|
22
|
+
```ts
|
|
23
|
+
const ack = await client.start({ pipe_code: "my_domain.summarize", inputs: { text: "..." } });
|
|
24
|
+
console.log(ack.pipeline_run_id); // persist this — it outlives the process
|
|
25
|
+
|
|
26
|
+
// …later, in another process
|
|
27
|
+
const results = await client.waitForResult(ack.pipeline_run_id);
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
Against a bare runner the id identifies the call the runner just answered, but there is no run store behind it: the lifecycle routes are absent, so re-reading it raises `RunLifecycleUnavailableError`. Durable resumption is a hosted capability.
|
|
31
|
+
|
|
32
|
+
## `main_stuff` — the output
|
|
33
|
+
|
|
34
|
+
`main_stuff` is the resolved content of the run's main output and is always present for a completed run. On the hosted path it is the `main_stuff.json` artifact; on the blocking path the SDK resolves it out of the returned working memory through the response's `main_stuff_name`. Both deliver the same content shape, so there is no shape-guessing and no path-dependent branch to write. A completed run that cannot deliver one throws `MissingMainStuffError` rather than handing back a half-filled result.
|
|
35
|
+
|
|
36
|
+
It is typed `unknown` because the content is polymorphic: a structured output arrives as an object of the concept's fields, and a multiple output as the envelope `{ items: [...] }` that the runtime's `ListContent` serialises to. Every content type serialises to an object, natives included — a text output is `{ "text": "…" }` and a number `{ "number": 0 }` — so a guard written for a bare `""` or `0` never fires, and an empty multiple output is `{ "items": [] }` rather than `[]`. Narrow it where you read it, ideally through the types generated for the method rather than a hand-written cast.
|
|
37
|
+
|
|
38
|
+
For a multiple output that means reading `items` off the object and mapping the generated per-concept parser over its members, because codegen emits a parser per concept and no wrapper type for the envelope. **Do not rely on the parser to catch the mistake for you.** The generated schemas are not strict, so a concept with a required field rejects the envelope loudly, while one whose fields are all optional parses it to `{}` and discards the output in silence.
|
|
39
|
+
|
|
40
|
+
```ts
|
|
41
|
+
const state = await client.getRunResult(runId);
|
|
42
|
+
|
|
43
|
+
switch (state.state) {
|
|
44
|
+
case "completed": {
|
|
45
|
+
// `state.result` is the RunResults; `main_stuff` is the output content.
|
|
46
|
+
const summary = state.result.main_stuff as { title: string; bullets: string[] };
|
|
47
|
+
console.log(summary.title, summary.bullets.length);
|
|
48
|
+
break;
|
|
49
|
+
}
|
|
50
|
+
case "running":
|
|
51
|
+
console.log(`not finished — poll again in ${state.retry_after_seconds ?? 2}s`);
|
|
52
|
+
break;
|
|
53
|
+
case "failed":
|
|
54
|
+
console.error(`run ended as ${state.status}: ${state.message}`);
|
|
55
|
+
break;
|
|
56
|
+
}
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
`getRunResult` is the single-shot lookup and returns that discriminated state. `waitForResult(runId)` drives the same lookup in a loop, honouring the server's `Retry-After`, and returns the `RunResults` directly — throwing `RunFailedError` on a terminal non-completed status and `RunTimeoutError` when the budget runs out.
|
|
60
|
+
|
|
61
|
+
## `working_memory` — every named stuff of the run
|
|
62
|
+
|
|
63
|
+
`working_memory` is the run's whole working memory: every named stuff the run held when it finished, as `{ root, aliases }`. `root` maps each stuff's name to the stuff itself — its concept ref, a string such as `native.Text`, and its content — and `aliases` maps further names onto root keys. It holds the inputs the run was given and the intermediates it produced as well as the main output, which is what a consumer needs to show what went into a run beside what came out, or to repopulate the inputs of a run it restores.
|
|
64
|
+
|
|
65
|
+
It reads the same on both paths. On the hosted path it is the `working_memory.json` artifact, relayed verbatim; on the blocking path the SDK lifts it off `pipe_output`. On the hosted path it is `null` when the artifact had not been written when the results were delivered. On the blocking path it is always there, because the SDK resolves `main_stuff` out of it and a response without one throws `MissingMainStuffError` first.
|
|
66
|
+
|
|
67
|
+
`main_stuff` is one of its entries — the content of the stuff the run names as its main output — already resolved. Read `main_stuff` for the output and `working_memory` for everything else; there is no need to find the main output in the working memory by hand. The type, `DictWorkingMemory`, mirrors the MTHDS standard's `DictWorkingMemory` field for field.
|
|
68
|
+
|
|
69
|
+
```ts
|
|
70
|
+
for (const [name, stuff] of Object.entries(results.working_memory?.root ?? {})) {
|
|
71
|
+
console.log(name, stuff.concept); // e.g. "text native.Text"
|
|
72
|
+
}
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
**Older runs carry the concept as an object.** A runtime older than pipelex 0.60.0 wrote each stuff's `concept` as the full concept object rather than its ref string: every hosted run whose `working_memory.json` was written before the hosted plane moved to that release, and every run of a bare runner still pinned below it. The artifact is relayed as it was written and never migrated. The type deliberately stays the standard's `concept: string`, which is what every run from 0.60.0 on carries, rather than widening to describe relics — so a consumer listing runs that may predate the release reads `concept` as `unknown` and narrows it itself:
|
|
76
|
+
|
|
77
|
+
```ts
|
|
78
|
+
const concept: unknown = stuff.concept;
|
|
79
|
+
const ref = typeof concept === "string" ? concept : undefined; // undefined: a pre-0.60.0 concept object
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
## `graph_spec` — the executed graph
|
|
83
|
+
|
|
84
|
+
`graph_spec` is the graph the run actually executed: `meta.mode` is `"live"`, and there is one node per pipe with its execution status, its start and end timestamps, its inputs and outputs, and the inference models and cost attributed to it. It is the same document a local `pipelex` run writes as `graphspec.json`, so anything that reads one of those files reads this value unchanged.
|
|
85
|
+
|
|
86
|
+
The field is typed `unknown` by a standing ruling ([`architecture.md`](./architecture.md#standard-artifacts-on-the-validate-report)): the canonical declaration is `GraphSpec` in `@pipelex/mthds-ui`, which carries a React peer dependency that a server-side SDK has no business taking, and the MTHDS standard declares nothing this SDK could import instead. The value is relayed verbatim either way — the typing says where the schema lives, not that the content is uncertain.
|
|
87
|
+
|
|
88
|
+
**Rendering it.** `@pipelex/mthds-ui` ships the viewer that consumes it, and its `GraphSpec` type is the cast to use on the consumer side. The viewer must be loaded client-side only, because ReactFlow touches browser globals at module evaluation, so the import goes through `next/dynamic` with `ssr: false` rather than a static one. It also fills its parent, so that parent needs `position: relative` and a height of its own or the graph renders at zero height:
|
|
89
|
+
|
|
90
|
+
```tsx
|
|
91
|
+
"use client";
|
|
92
|
+
|
|
93
|
+
import dynamic from "next/dynamic";
|
|
94
|
+
import type { RunResults } from "@pipelex/sdk";
|
|
95
|
+
import type { GraphSpec } from "@pipelex/mthds-ui/graph";
|
|
96
|
+
|
|
97
|
+
const GraphViewer = dynamic(
|
|
98
|
+
() => import("@pipelex/mthds-ui/graph/react").then((m) => m.GraphViewer),
|
|
99
|
+
{ ssr: false },
|
|
100
|
+
);
|
|
101
|
+
|
|
102
|
+
export function RunGraph({ results }: { results: RunResults }) {
|
|
103
|
+
if (!results.graph_spec) return null;
|
|
104
|
+
return (
|
|
105
|
+
<div style={{ position: "relative", height: "600px" }}>
|
|
106
|
+
<GraphViewer graphspec={results.graph_spec as GraphSpec} />
|
|
107
|
+
</div>
|
|
108
|
+
);
|
|
109
|
+
}
|
|
110
|
+
```
|
|
111
|
+
|
|
112
|
+
**Keeping it.** The value is plain JSON, so persisting it is a write; there is no SDK helper and none is needed. Keeping it is worth doing for anything you may have to explain later, because it is the only record of what the run did pipe by pipe:
|
|
113
|
+
|
|
114
|
+
```ts
|
|
115
|
+
import { writeFile } from "node:fs/promises";
|
|
116
|
+
|
|
117
|
+
await writeFile("graphspec.json", JSON.stringify(results.graph_spec, null, 2));
|
|
118
|
+
```
|
|
119
|
+
|
|
120
|
+
**When it is null.** On the hosted path, the artifact may not have been written when the results were delivered. On either path, the runner may have assembled no graph at all — which is what the next field is for.
|
|
121
|
+
|
|
122
|
+
## `graph_assembly_error` — why there is no graph
|
|
123
|
+
|
|
124
|
+
`graph_assembly_error` is the graph's twin of `usage_assembly_error`, and it exists for the same reason: a null `graph_spec` alone cannot say whether graph assembly was off, broke, or simply had not finished writing. When the runner's assembly failed, this field carries the runner's message.
|
|
125
|
+
|
|
126
|
+
On the blocking path the SDK lifts it off `pipe_output`, beside the graph itself. **On the hosted path the key is absent, so the field reads `undefined` rather than `null`**: the platform's results body relays no such key and the SDK parses that body as it arrives, so the failure the bare runner reports is not yet observable through the hosted API. The field is declared ahead of that relay so consumers have one accessor to write against and nothing breaks the day the wire gains the key — the value appears on its own, with no SDK change. Until then, treat `undefined` on the hosted path as "no information", not as "assembly succeeded", and compare with `!= null` rather than `=== null` so that the same branch keeps working once the key arrives.
|
|
127
|
+
|
|
128
|
+
```ts
|
|
129
|
+
if (results.graph_assembly_error != null) {
|
|
130
|
+
console.warn("graph assembly failed for this run:", results.graph_assembly_error);
|
|
131
|
+
} else if (results.graph_spec == null) {
|
|
132
|
+
// no graph: assembly was off, the artifact was not written, or (hosted) the error is not relayed
|
|
133
|
+
}
|
|
134
|
+
```
|
|
135
|
+
|
|
136
|
+
## `tokens_usages` and `usage_assembly_error` — what the run consumed
|
|
137
|
+
|
|
138
|
+
The usage pair reports what each inference call consumed and cost — one `TokensUsageRecord` per call, in completion order — and reads identically on both paths. The null-versus-empty semantics, the cost rules (`null` is unrated, `0` is priced at zero), the non-additive token categories and the pre-contract artifacts that still type-check all have their own page: [`run-usage.md`](./run-usage.md).
|
|
139
|
+
|
|
140
|
+
For the run's totals, call `summarizeUsage(results)` rather than adding the records up by hand: it returns the total cost, the input and output token totals and a per-pipe rollup, with a `state` that tells a run with records from one that did no inference and from one whose usage is unavailable. See [Summarizing a run](./run-usage.md#summarizing-a-run--summarizeusage).
|
|
141
|
+
|
|
142
|
+
## `pipe_output` — the runner's native output
|
|
143
|
+
|
|
144
|
+
`pipe_output` is the bare runner's whole native output, and it is present on the blocking path only — the hosted results body carries no such key, so on that path it reads `undefined`. It is supplementary: `main_stuff`, `working_memory`, the graph and the usage pair are all lifted out of it onto fields that read the same on both paths, so a consumer that wants every named stuff of the run reads `working_memory`, not `pipe_output.working_memory`, and its code keeps working against the hosted API. What `pipe_output` adds is the runner's output exactly as it arrived. It is typed `DictPipeOutput`, which is extension-open — the runner's Pipelex extension fields are reachable through the index signature without casting the whole value away.
|
|
145
|
+
|
|
146
|
+
## Produced files
|
|
147
|
+
|
|
148
|
+
A run that produces an image, a PDF or a document does not embed the bytes. The content inside `main_stuff` carries the file's durable reference — a `pipelex-storage://` URI, in the content's `url` — beside a `public_url` the storage provider signed when the run wrote the file. **That signed link is short-lived and must not be stored**: it expires on the provider's own schedule, so a link persisted in a database or rendered into a cached page stops working without warning, while the `pipelex-storage://` reference beside it is permanent and is what belongs in your records.
|
|
149
|
+
|
|
150
|
+
To read the bytes, mint a fresh link from the reference:
|
|
151
|
+
|
|
152
|
+
```ts
|
|
153
|
+
const resolved = await client.resolveStorageUrl({ uri: "pipelex-storage://..." });
|
|
154
|
+
// { url, expires_at, content_type } — fetch `url` now; re-resolve for the next reader.
|
|
155
|
+
```
|
|
156
|
+
|
|
157
|
+
The same rule holds in a browser: resolve on the server, hand the client a link it uses immediately, and never let a presigned URL outlive the request it was minted for.
|
|
158
|
+
|
|
159
|
+
Bringing the files down is the SDK's job, not something to re-implement over `resolveStorageUrl`: `collectArtifacts` lists a result's references without touching the network, `resolveArtifacts` mints fresh links for a whole list in one bulk call, `fetchArtifact` streams one file within safe bounds, and `downloadArtifacts` saves everything a run produced under a directory, by run id even days later. All four are on their own page, [`artifact-download.md`](./artifact-download.md). The upload direction — turning local files into `pipelex-storage://` references before a run — is the mirror of this and has its own page too, [`input-preparation.md`](./input-preparation.md).
|