@pipelex/sdk 0.47.0 → 0.48.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +8 -6
- package/dist/cli/script.d.ts +1 -1
- package/dist/cli/source.d.ts +8 -7
- package/dist/cli/source.js +11 -5
- package/dist/cli/source.js.map +1 -1
- package/dist/client.d.ts +321 -210
- package/dist/client.js +819 -514
- package/dist/client.js.map +1 -1
- package/dist/errors.d.ts +5 -2
- package/dist/errors.js +5 -2
- package/dist/errors.js.map +1 -1
- package/dist/hooks/bundle-gather.d.ts +28 -40
- package/dist/hooks/bundle-gather.js +55 -230
- package/dist/hooks/bundle-gather.js.map +1 -1
- package/dist/hooks/claude-mthds-check.d.ts +7 -5
- package/dist/hooks/claude-mthds-check.js +8 -53
- package/dist/hooks/claude-mthds-check.js.map +1 -1
- package/dist/hooks/validate-stage.d.ts +27 -0
- package/dist/hooks/validate-stage.js +57 -0
- package/dist/hooks/validate-stage.js.map +1 -0
- package/dist/index.d.ts +3 -4
- package/dist/index.js +4 -3
- package/dist/index.js.map +1 -1
- package/dist/method-trees.d.ts +208 -73
- package/dist/method-trees.js +221 -34
- package/dist/method-trees.js.map +1 -1
- package/dist/models.d.ts +162 -59
- package/dist/prepare-inputs.d.ts +46 -27
- package/dist/prepare-inputs.js +123 -42
- package/dist/prepare-inputs.js.map +1 -1
- package/dist/product-models.d.ts +14 -16
- package/dist/version.d.ts +1 -1
- package/dist/version.js +1 -1
- package/docs/architecture.md +17 -18
- package/docs/cli.md +1 -1
- package/docs/crate-routes.md +42 -20
- package/docs/errors.md +5 -5
- package/docs/hook-bundle.md +1 -1
- package/docs/input-preparation.md +11 -11
- package/docs/method-trees.md +119 -21
- package/docs/run-results.md +5 -2
- package/package.json +2 -2
- package/dist/bundle-resources.d.ts +0 -62
- package/dist/bundle-resources.js +0 -165
- package/dist/bundle-resources.js.map +0 -1
- package/docs/bundle-resources.md +0 -107
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
> **Status: implemented** (`src/upload.ts`, `src/upload-grant.ts`, `src/prepare-inputs.ts`). This document records the contract, and is the public account of it. The raw `upload()` primitive described in [architecture.md](./architecture.md) is the wire call `uploadFile` and `prepareInputs` build on.
|
|
4
4
|
>
|
|
5
|
-
> **Current scope.** `prepareInputs` names the method the same three ways every other method-taking operation does —
|
|
5
|
+
> **Current scope.** `prepareInputs` names the method the same three ways every other method-taking operation does — the method's tree, a `method_ref` address, or a stored `method_id` — exactly one per call, all three resolved server-side by the one `POST /v1/pipe-io` call it composes. One piece remains deliberately deferred and additive (it does not change this contract): the opt-in ingest of `http(s)` URLs into storage — for now an `http(s)` URL at a file position always passes through unchanged.
|
|
6
6
|
|
|
7
7
|
## Why this exists
|
|
8
8
|
|
|
@@ -93,7 +93,7 @@ It sends the file as the raw body of a `PUT` to `grant.url`, with `grant.headers
|
|
|
93
93
|
### `prepareInputs` — signature-driven input preparation
|
|
94
94
|
|
|
95
95
|
```
|
|
96
|
-
client.prepareInputs({ files,
|
|
96
|
+
client.prepareInputs({ directory | files | tree | contents, dependencies?, pipe_ref?, inputs }) → PreparedInputs
|
|
97
97
|
client.prepareInputs({ method_ref, pipe_ref?, inputs }) → PreparedInputs
|
|
98
98
|
client.prepareInputs({ method_id, pipe_ref?, inputs }) → PreparedInputs
|
|
99
99
|
```
|
|
@@ -105,19 +105,19 @@ Takes the **method** as exactly one of three selectors, the optional target **pi
|
|
|
105
105
|
|
|
106
106
|
The prepared `inputs` are passed to the existing run lifecycle unchanged.
|
|
107
107
|
|
|
108
|
-
**Stopping it.** An optional `signal` stops the preparation between its steps: once it has aborted, neither the pipe I/O request nor any upload starts, and `prepareInputs` throws the abort, as the run wait does. It is read again once each file's bytes are in hand, right before its upload request, so a file whose read outlasted the caller's patience is not sent; `uploadFile` takes the same `signal` in its options for that check. A request already sent runs to its end, so an upload under way finishes, but the file after it is not sent. The signal is not sent to the server.
|
|
108
|
+
**Stopping it.** An optional `signal` stops the preparation between its steps: once it has aborted, neither the pipe I/O request nor any upload starts, and `prepareInputs` throws the abort, as the run wait does. It is read again once each file's bytes are in hand, right before its upload request, so a file whose read outlasted the caller's patience is not sent; `uploadFile` takes the same `signal` in its options for that check. A request already sent runs to its end, so an upload under way finishes, but the file after it is not sent. A `directory`, `tree` or `contents` adds steps before the pipe I/O request — the version handshake, the directory's listing, and for a tree sent by hash the blob route and each read and upload of a missing blob — and the signal is read before each of them too and handed to each, through the route's `CrateRouteOptions`, so an abort stops the one under way, a blob's upload included: one during the handshake stops it and leaves the blob route, the uploads and the pipe I/O request unasked. The signal is not sent to the server.
|
|
109
109
|
|
|
110
110
|
#### The three selectors
|
|
111
111
|
|
|
112
|
-
Exactly one per call. The type pins the other two to `never`, so a second selector is a compile error; an untyped caller gets an `InputPreparationError` naming the three forms. **Empty is absent** — `files:
|
|
112
|
+
Exactly one per call. The type pins the other two to `never`, so a second selector is a compile error; an untyped caller gets an `InputPreparationError` naming the three forms. **Empty is absent** — `files: {}`, `contents: {}`, `directory: ""`, `method_ref: ""`, `method_id: " "` — mirroring the run options' rule, so an empty selector may sit beside a real one without tripping the XOR. A `files` that is not a map, such as the retired list of `{ content, source }` items, is an `InputPreparationError` rather than an absent selector, and so, before any request, is a `directory` that is not a string, a `contents` that is not a map of text or bytes, a `dependencies` that is not a map of file maps, and a `tree` that is not a listed tree (its `files`, its `digest`, a `readFile`) or holds no file.
|
|
113
113
|
|
|
114
114
|
| Selector | What it is | Who resolves it |
|
|
115
115
|
| --- | --- | --- |
|
|
116
|
-
| `files`
|
|
116
|
+
| the tree | the method's tree, in exactly one of the forms the crate routes take: `files`, the protocol's map from tree path to file entry; `directory`, the method's directory on disk, listed by the `mthds` package's lister (Node only); `tree`, a listed `MethodTreeSource`; or `contents`, each file's bytes or text by tree path ([crate-routes.md](./crate-routes.md#the-shared-envelope)) | nobody — sent with the request, a `directory`, `tree` or `contents` inline when small and by hash otherwise |
|
|
117
117
|
| `method_ref` | a published method's address, `github.com/<owner>/<repo>[/<selector>][@<tag>]` | the runner, server-side (pipelex-api >= 0.21.0 fetches the repository at the tag) |
|
|
118
|
-
| `method_id` | a stored method's catalog id (`mt_…`, `mt_…@<n>` or `mt_…@draft`) | the hosted platform, which injects the
|
|
118
|
+
| `method_id` | a stored method's catalog id (`mt_…`, `mt_…@<n>` or `mt_…@draft`) | the hosted platform, which injects the tree of the version it names before the runner sees the request |
|
|
119
119
|
|
|
120
|
-
|
|
120
|
+
The tree carries every file the method's steps read, such as a `PipeDocGen` step's `template_file`, at the tree path the step names ([method-trees.md](./method-trees.md#the-files-a-step-reads)), and it may bring `dependencies`, the method packages it calls by address, each a `files` map keyed by its address. They ride the one pipe I/O request, which loads the closure and fails to load it, as a `MethodLoadError`, when a step's template file is not in the tree or a call reaches a package nothing shipped. A published or stored method's packages are its own, so `dependencies` beside `method_ref` or `method_id` is an `InputPreparationError` before any request, and so is the tree given in two forms.
|
|
121
121
|
|
|
122
122
|
Nothing is expanded client-side: `method_id` here is a **pass-through**, the same rule every other id-taking operation follows (see "[A method's files in hand](#a-methods-files-in-hand)").
|
|
123
123
|
|
|
@@ -126,12 +126,12 @@ Nothing is expanded client-side: `method_id` here is a **pass-through**, the sam
|
|
|
126
126
|
One `POST /v1/pipe-io` per call, whatever the selector, through the client's own `pipeIo` (see [crate-routes.md](./crate-routes.md#pipeio--a-methods-io-artifacts-without-a-validation)):
|
|
127
127
|
|
|
128
128
|
```
|
|
129
|
-
pipeIo({ <files [+
|
|
129
|
+
pipeIo({ <directory | files | tree | contents [+ dependencies] | method_ref | method_id>, pipe_ref? })
|
|
130
130
|
```
|
|
131
131
|
|
|
132
132
|
The route resolves the closure, **selects the pipe**, and answers with that pipe's **input-form descriptor** under the qualified ref it resolved. It runs no dry run, so preparing inputs costs one static load rather than a validation of the whole method. It also refuses nothing for a pending signature elsewhere in the method: preparation needs a pipe's *declared* inputs, and whether the method runs is the run's verdict, not preparation's. An `is_valid: false` verdict still means the closure does not load, which is a preparation failure.
|
|
133
133
|
|
|
134
|
-
A `
|
|
134
|
+
A `directory`, `tree` or `contents` too large to send inline is uploaded first, by hash, and a blob that fails to upload throws `MethodFileTransferError` before the route is asked. A `method_ref` makes the server clone a repository first, so the call gets the crate routes' fetch-sized budget. On the hosted API the gateway caps a request at about 30 seconds whatever the client allows, so a cold clone can answer a `502` `ApiResponseError`; a retry clears it once the runner has cached the clone.
|
|
135
135
|
|
|
136
136
|
`prepareInputs` needs an API that serves `POST /v1/pipe-io`. Against one that does not, the route's `404` (a runner) or `403` (a hosted origin whose gateway does not list it yet) propagates as an `ApiResponseError`. It also needs the route's pipe-selection refusals typed, as step 3 of "Pipe selection" below describes, which `pipelex-api` does from v0.33.1: v0.33.0, the release that added the route, typed them as the generic `ValidationError`, so against it a refused selection reaches the caller as that `ApiResponseError` rather than an `InputPreparationError`.
|
|
137
137
|
|
|
@@ -139,8 +139,8 @@ A `method_ref` makes the server clone a repository first, so the call gets the c
|
|
|
139
139
|
|
|
140
140
|
The route selects the pipe, and the ref it answers with is the one prepared — the helper derives no default of its own:
|
|
141
141
|
|
|
142
|
-
1. **`pipe_ref` when given.** Qualified-only: `domain.pipe_code`, normalized (trimmed, and blank is absent) and checked **before any request**. A bare code is refused with an `InputPreparationError` asking for the qualified form, since a request names a pipe by its qualified ref and search is a run-route affordance this helper never grows. An `alias->domain.pipe_code` ref is refused for a different reason: the alias names a dependency package's pipe, and preparation covers the method's *own* pipes
|
|
143
|
-
2. **Otherwise the method's own entry pipe**, chosen by the route:
|
|
142
|
+
1. **`pipe_ref` when given.** Qualified-only: `domain.pipe_code`, normalized (trimmed, and blank is absent) and checked **before any request**. A bare code is refused with an `InputPreparationError` asking for the qualified form, since a request names a pipe by its qualified ref and search is a run-route affordance this helper never grows. An `alias->domain.pipe_code` ref is refused for a different reason: the alias names a dependency package's pipe, and preparation covers the method's *own* pipes, whose input form the pipe I/O route describes. The run route takes such a ref; preparation cannot.
|
|
143
|
+
2. **Otherwise the method's own entry pipe**, chosen by the route: the `main_pipe` of the `METHODS.toml` at the tree's root, whether the tree was sent or fetched with a package, else the closure's `main_pipe` when exactly one domain declares one. A method that names its entry pipe in `METHODS.toml` alone — the published `github.com/Pipelex/methods/documents` does — therefore prepares with no `pipe_ref`.
|
|
144
144
|
3. **A selection the route refuses** — a `pipe_ref` that names no pipe of the method, a method with no entry pipe, one whose domains declare several, a manifest `main_pipe` the closure does not declare or declares in several domains — comes back as a `422` whose `errorType` is `EntryPipeNotFoundError` or `EntryPipeAmbiguousError`. The helper turns exactly those into an `InputPreparationError` carrying the server's detail, which names the candidates where there are some, with the `ApiResponseError` as its `cause`. The run route refuses a selector-less run in the same cases, so preparation never succeeds for a pipe the run would not take.
|
|
145
145
|
|
|
146
146
|
There is no single-pipe fallback: a method that declares one pipe and no `main_pipe` is refused a selector-less run, and a selector-less preparation with it.
|
package/docs/method-trees.md
CHANGED
|
@@ -1,28 +1,119 @@
|
|
|
1
|
-
# A
|
|
1
|
+
# A method's tree
|
|
2
2
|
|
|
3
|
-
A
|
|
3
|
+
A method is a **method tree**: the files it is made of, each at its tree path — its bundles, its manifest when it has one, and every file a step reads or that travels with the method, such as a `PipeDocGen` step's template. A tree is named by its **tree digest**, `sha256:` and the SHA-256 of its listing, so two methods holding the same files hold the same digest. Both are defined by the MTHDS standard (`mthds/docs/spec/method-tree.md`), and on the wire a tree is the protocol's `files` map, each key a tree path and each value the file's text, `{ base64 }` for bytes that are not text, or `{ sha256, size }` for bytes the receiver already holds.
|
|
4
4
|
|
|
5
|
-
The platform keeps each file once per organization, as a blob named by its SHA-256, so a
|
|
5
|
+
Every call that takes a method takes its tree the same way: the runs (`execute`, `start`, `startAndWaitForResult`), `validate`, the crate routes (`resolve`, `codegen`, `pipeIo`), `prepareInputs`, and the saves (`createMethod`, `writeDraft`). A saved method holds one too: the draft holds one, and each published version holds the one it was published from. The platform keeps each file once per organization, as a blob named by its SHA-256, so a request sends only the files the organization does not hold yet, and a read lists the files with a short-lived link to each one's bytes.
|
|
6
6
|
|
|
7
|
-
##
|
|
7
|
+
## The four forms of a tree
|
|
8
8
|
|
|
9
|
-
|
|
9
|
+
A call takes the tree in exactly one of four forms (`MethodTreeForms`):
|
|
10
10
|
|
|
11
11
|
| Form | What it is | How it is sent |
|
|
12
12
|
|---|---|---|
|
|
13
|
-
| `
|
|
14
|
-
| `
|
|
15
|
-
| `
|
|
13
|
+
| `directory` | The method's directory on disk. **Node only.** | Listed, then inline when small and by hash otherwise (below). |
|
|
14
|
+
| `contents` | Each file's bytes (`Uint8Array`) or its text, by tree path, a map held in memory. | Inline when small, by hash otherwise. |
|
|
15
|
+
| `tree` | A `MethodTreeSource`: a tree's `files` (each with its `path`, `size` and `sha256`), its `digest`, and `readFile(path)` giving each file's bytes. The `MethodTree` the `mthds` package's `listMethodTree` returns is one. | Inline when small, by hash otherwise. |
|
|
16
|
+
| `files` | The protocol's map of file entries (`MethodFiles` from `mthds/protocol`). | As given. |
|
|
17
|
+
|
|
18
|
+
```ts
|
|
19
|
+
// A run of the method in a directory: the client lists it, and sends what the API lacks.
|
|
20
|
+
const results = await client.startAndWaitForResult({
|
|
21
|
+
directory: "methods/receipts",
|
|
22
|
+
inputs: { receipt: "https://example.com/receipt.png" },
|
|
23
|
+
});
|
|
24
|
+
|
|
25
|
+
// The same method validated from memory, its template among its files.
|
|
26
|
+
const verdict = await client.validate({
|
|
27
|
+
contents: {
|
|
28
|
+
"receipts.mthds": source,
|
|
29
|
+
"templates/receipt.docx": await readFile("methods/receipts/templates/receipt.docx"),
|
|
30
|
+
},
|
|
31
|
+
});
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
**A directory is listed by the standard's lister.** The client loads the `mthds` package's `listMethodTree` when a call gives a `directory`, and lists it under the MTHDS standard's membership rule and Manifest Discovery: the tree is rooted at the nearest `METHODS.toml` at or above the directory, else at the directory itself, and holds every file the rule keeps. Hidden entries, `node_modules`, `__pycache__`, what a `.gitignore` or the manifest's `exclude` keeps out, and **a run's inputs file** (`inputs.json`, `inputs.toml`) never travel with the method. A tree holding a symbolic link or a special file is refused rather than followed. The directory is listed under the limits the deployment announces (below), so a tree the deployment would refuse for its size is refused while it is listed, before any file is hashed. Every refusal is a `RequestArgumentError` naming the call and the directory, whose `cause` is `mthds`'s `MethodTreeError`, with its `code` (`not_found`, `symbolic_link`, `limit_exceeded`, `empty_tree` and the rest). Listing needs Node: a browser gives `contents`, `tree` or `files`.
|
|
35
|
+
|
|
36
|
+
**A map in memory.** A text in `contents` is encoded in UTF-8, and a text a file cannot hold, one carrying a lone surrogate, is refused, as is a value that is neither a string nor a `Uint8Array`. The keys are sent as they are, and the platform refuses one that is not a tree path with the `422` `invalid_method_files` naming it.
|
|
37
|
+
|
|
38
|
+
**One form, and never an empty tree.** Two forms at once are refused with a `RequestArgumentError` before anything is read or sent. A `files` map, `contents` or a `tree` holding no file is refused too on the run routes and on `validate`, as the standard refuses it: a tree holds at least one file. On the crate routes and in `prepareInputs`, an empty map is absent instead, so it never reaches the wire as a second selector, and in a save `files: null` empties the draft. An empty `directory` string is absent everywhere.
|
|
39
|
+
|
|
40
|
+
## How a tree travels
|
|
41
|
+
|
|
42
|
+
A tree given as a `directory`, `contents` or `tree` is sent by the rule the `mthds` package's `MthdsApiClient` follows, so the tree a save sends is the tree a run would send. The client reads the API's version handshake first (`GET /v1/version`) and keeps its answer for five minutes, so every call in that time reads the same answer and a deployment that changes is read afresh soon after ([below](#the-deployments-limits)):
|
|
43
|
+
|
|
44
|
+
- **When the API keeps a blob store** (`blob_store: true`), a tree whose inline map weighs no more than the inline threshold travels **inline**, with no further round trip. The map is weighed as the request spends it, its JSON with no whitespace, in UTF-8, and the threshold is the lower of `INLINE_TREE_MAX_BYTES` (256 KiB) and the `max_inline_bytes` the handshake announces. Inline, a file whose bytes are UTF-8 travels as its text and any other as `{ base64 }`, so a binary file weighs its Base64. A heavier tree travels **by hash**: the client asks `POST /v1/blobs/missing` which files the receiver lacks, naming the tree's digest, uploads each one it lacks once, at most eight at a time, straight to the upload target the platform names, under the client's request time limit (`requestTimeoutMs`, twenty minutes unless the client was built with another), and sends every file as `{ sha256, size }`. A file uploaded once is never uploaded again, by this method or any other of the organization. A `404` from the blob route says the kept handshake is stale, the API having lost its blob store since it was read: the client forgets the handshake and prepares the tree once more on a fresh one.
|
|
45
|
+
- **When the API keeps no blob store**, which a bare runner usually does, the whole tree travels inline whatever its weight, held to the `max_inline_bytes` the handshake announces: a heavier tree is refused with a `RequestArgumentError` before anything is sent, since the API would refuse it. The files' listed sizes are checked first, since the inline map weighs at least that much, so a tree whose files alone pass the limit is refused before any of them is read; the message says which was measured, the files alone or the inline map, with its weight and the limit.
|
|
16
46
|
|
|
17
|
-
A
|
|
47
|
+
A `files` map is sent as given, with no handshake. A run's `start` carrying a tree, a run that names a `method_ref`, and a start whose `extra` carries a method bundle (`mthds_contents` or `bundle_b64`, as the `pipelex-sdk` command sends one) get the client's request ceiling rather than the 30-second start budget, since the body can be large. On the crate routes, a request whose `files` carry more than `INLINE_TREE_MAX_BYTES` inline, as a tree sent to an API keeping no blob store or a `files` map given as is may, gets the request ceiling too rather than the 30-second budget of a static route ([crate-routes.md](./crate-routes.md)). A save carrying a tree (`createMethod`, `writeDraft`), inline or by hash, gets the request ceiling as well, rather than the 30-second budget of a save sending none, since a tree sent inline to an API keeping no blob store can be large and its whole upload is charged against the budget.
|
|
18
48
|
|
|
19
|
-
A
|
|
49
|
+
**A refusal of the tree's size is sent once more.** When the API refuses a tree the client prepared, a `directory`, `contents` or `tree`, as past one of its limits (a `422` coded `method_tree_too_large`, or `method.package_invalid` naming a limit, or of type `MethodTreeTooLarge`), the handshake that announced the limits may be stale: the client forgets it, prepares the tree again on a fresh one and sends the request once more, as `mthds`'s client does. A `files` map given as is was not prepared from the handshake, so it is not sent again, and neither is a `start` under an idempotency key, since the platform binds the key to the refusal it stores.
|
|
50
|
+
|
|
51
|
+
**Stopping a call.** A call that takes a `signal` (`validate`, `startAndWaitForResult`, the crate routes, `prepareInputs`) reads it before each step a `directory`, `contents` or `tree` costs and hands it to the step's request, so an abort stops the version handshake, the directory's listing, the blob route, a file's read and an upload already under way, and nothing further starts.
|
|
52
|
+
|
|
53
|
+
A `MethodTree` reads a file again whenever its bytes are sent, inline or as an upload, and checks it against the hash its listing recorded, so a file changed since the listing is refused by `mthds` rather than sent as another tree than the one listed. Apart from listing a directory, sending a tree reads no Node module: hashing is Web Crypto's, and Base64 is written by the SDK.
|
|
54
|
+
|
|
55
|
+
**What is uploaded is what the listing names.** Before each upload the client hashes the bytes a tree's `readFile` gives and refuses them, with a `RequestArgumentError`, unless they have the size and the SHA-256 the tree lists for them: a blob is stored under its hash for the whole organization, so a hand-built `MethodTreeSource` whose `readFile` disagrees with its listing never stores other bytes under that hash. An answer of `POST /v1/blobs/missing` asking for a blob the tree does not hold is the platform's fault, not the caller's: it is an `ApiResponseError` of an answer the SDK cannot read, verdict `runtime` and not retryable, thrown before any upload. An upload that fails throws `MethodFileTransferError` with `operation` `upload`, and the request is not sent or the method not saved ([below](#when-a-transfer-fails)).
|
|
56
|
+
|
|
57
|
+
**A blob that went missing.** A request whose hash entries name a blob the API does not hold is refused with a `422` `missing_blobs` listing them all, which also releases a start's idempotency key. A request built from a `directory`, `contents` or `tree` uploads every blob it names first, so it meets this only if a blob was removed between the upload and the request. The client answers it as `mthds`'s client does: a `422` problem coded `request.blobs_missing`, whose `params.missing_blobs` names blobs the tree holds, has those blobs asked of the blob route again, uploaded and the request sent once more, while the platform's `missing_blobs` code is not read as one and is thrown, so sending the same call again asks the blob route afresh and uploads what is missing.
|
|
58
|
+
|
|
59
|
+
On the crate routes, `read_scope` is the scope the blob route is asked under, so the blobs are uploaded into the store the request's hash entries are read from; the hosted platform reads the caller's organization whatever the request says ([crate-routes.md](./crate-routes.md#the-shared-envelope)).
|
|
60
|
+
|
|
61
|
+
## The files a step reads
|
|
62
|
+
|
|
63
|
+
A file a step reads, such as a `PipeDocGen` step's `template_file` (the Word, Excel or PowerPoint template it fills, or the HTML template it prints as a PDF), is a file of the tree like any other, at the tree path the step names from the root. A `directory` sends it with the bundles; a `contents` or `files` map carries it under its tree path. A step that names a file the tree does not hold cannot load: on `validate`, `pipeIo`, `resolve` and `codegen` that is an `is_valid: false` verdict whose validation error says the file is missing, `prepareInputs` throws it as a `MethodLoadError`, and on `execute` and `start` it is a `422` `ValidateBundleError`, whose `validationErrors` say the same.
|
|
64
|
+
|
|
65
|
+
### The template check's warnings
|
|
66
|
+
|
|
67
|
+
A dry run hands an office template file of the tree to the document engine's template check. An error finding fails the step's dry run, and so the verdict, as a `dry_run` validation error on the step. A warning finding, such as a field of the template that matches no input, rides the valid verdict's `warnings` as a `template_check_warning` item:
|
|
68
|
+
|
|
69
|
+
| Member | Value |
|
|
70
|
+
| --- | --- |
|
|
71
|
+
| `category` | `dry_run` |
|
|
72
|
+
| `error_type` | `template_check_warning` |
|
|
73
|
+
| `pipe_code`, `domain_code` | The `PipeDocGen` step |
|
|
74
|
+
| `template_file` | The template file as the step names it; `null` for a step of the server's own library, whose path is never shown |
|
|
75
|
+
| `template_location` | The place in the file the finding names, in the engine's words (a defined name, a table, a cell, a shape, a tag); `null` when it names none |
|
|
76
|
+
| `message` | The finding in words |
|
|
77
|
+
|
|
78
|
+
The warnings come in families, always in this order: `optional_force_redundant`, `input_presence_vacuous`, the `hint_*` lints, then `template_check_warning`. None of them flips `is_valid`.
|
|
20
79
|
|
|
21
80
|
```ts
|
|
22
|
-
|
|
81
|
+
if (verdict.is_valid) {
|
|
82
|
+
for (const warning of verdict.warnings) {
|
|
83
|
+
if (warning.error_type === "template_check_warning") {
|
|
84
|
+
console.warn(`${warning.template_file ?? "?"} ${warning.template_location ?? ""}: ${warning.message}`);
|
|
85
|
+
}
|
|
86
|
+
}
|
|
87
|
+
}
|
|
88
|
+
```
|
|
23
89
|
|
|
24
|
-
|
|
25
|
-
|
|
90
|
+
### Which step reads which file
|
|
91
|
+
|
|
92
|
+
Whatever form the method was sent in, its tree in any of the four forms, a `method_ref` or a `method_id`, the valid verdict names the files of the method's tree each step reads. `tree_files_read` is keyed by the same namespaced `pipe_ref` as `pipe_io_contracts`, each entry a list of `TreeFileRead` items, and a step that reads no file, a controller included, has an empty list. `tree_file_readers` states the same facts the other way: keyed by tree path in sorted order, it lists for each file the refs of the steps that read it, each once, and a file of the tree with no entry, such as a bundle itself or a README, is read by no step.
|
|
93
|
+
|
|
94
|
+
| Member | Value |
|
|
95
|
+
| --- | --- |
|
|
96
|
+
| `tree_path` | The file's path from the root of the method's tree, with forward slashes, never a path on the server |
|
|
97
|
+
| `use` | `template` for a `PipeDocGen` step's `template_file`; `template_asset` for an image, a stylesheet or a font its HTML template names by a literal relative URL, or one a stylesheet it names refers to; `python_module` for the module holding a `PipeFunc` step's function, when that module is a file of the tree |
|
|
98
|
+
|
|
99
|
+
The paths come from what each step already names, so a client can show which step uses a file, or warn before one a step reads is renamed or removed, without parsing the method.
|
|
100
|
+
|
|
101
|
+
```ts
|
|
102
|
+
if (verdict.is_valid && verdict.tree_file_readers) {
|
|
103
|
+
for (const [treePath, pipeRefs] of Object.entries(verdict.tree_file_readers)) {
|
|
104
|
+
console.log(`${treePath} is read by ${pipeRefs.join(", ")}`);
|
|
105
|
+
}
|
|
106
|
+
}
|
|
107
|
+
```
|
|
108
|
+
|
|
109
|
+
`use` is typed `TreeFileUse | (string & {})`: a runner of a later version may add a use as another kind of step comes to read files of its tree, so a consumer switching on it says what a use it does not know does. Both fields are optional. A `pipelex-api` runner reports them from the first pipelex release after 0.90.0, and the hosted API once it runs such a release; an older runner sends neither, which reads as `undefined`, where an empty list says that the step reads no file.
|
|
110
|
+
|
|
111
|
+
## Saving a tree
|
|
112
|
+
|
|
113
|
+
`createMethod(input)` and `writeDraft(methodId, input)` take the tree in the four forms (`MethodTreeInput`), and send it by the rule above.
|
|
114
|
+
|
|
115
|
+
```ts
|
|
116
|
+
const method = await client.createMethod({ name: "Receipts", directory: "methods/receipts" });
|
|
26
117
|
|
|
27
118
|
const draft = await client.writeDraft(method.method_id, {
|
|
28
119
|
contents: { "main.mthds": source, "helpers/fx.py": helper },
|
|
@@ -30,9 +121,7 @@ const draft = await client.writeDraft(method.method_id, {
|
|
|
30
121
|
});
|
|
31
122
|
```
|
|
32
123
|
|
|
33
|
-
**A draft write replaces the whole tree.** `writeDraft` refuses a draft that gives no tree, and `files: null` empties the draft. `createMethod` with no tree creates a method with an empty draft.
|
|
34
|
-
|
|
35
|
-
**What is uploaded is what the listing names.** Before each upload the client hashes the bytes a tree's `readFile` gives and refuses them, with a `RequestArgumentError`, unless they have the size and the SHA-256 the tree lists for them: a blob is stored under its hash for the whole organization, so a hand-built `MethodTreeSource` whose `readFile` disagrees with its listing never stores other bytes under that hash. An answer of `POST /v1/blobs/missing` asking for a blob the tree does not hold is the platform's fault, not the caller's: it is an `ApiResponseError` of an answer the SDK cannot read, verdict `runtime` and not retryable, thrown before any upload.
|
|
124
|
+
**A draft write replaces the whole tree.** `writeDraft` refuses a draft that gives no tree, and `files: null` empties the draft. `createMethod` with no tree creates a method with an empty draft. A `contents` or `tree` holding no file is refused before anything is sent.
|
|
36
125
|
|
|
37
126
|
The save is refused, with the `ApiResponseError` of its problem, when the platform will not take the tree:
|
|
38
127
|
|
|
@@ -41,14 +130,21 @@ The save is refused, with the `ApiResponseError` of its problem, when the platfo
|
|
|
41
130
|
| `422` `invalid_method_files` | The map breaks the standard's rules, a path that is not a tree path or two paths that collide among them; the problem's `tree_refusal` names the rule. |
|
|
42
131
|
| `422` `method_tree_too_large` | The tree passes one of the deployment's limits, which the problem's `limit` names. |
|
|
43
132
|
| `422` `blob_mismatch` | A hash entry states another size than the blob the organization holds under that hash. |
|
|
44
|
-
| `422` `missing_blobs` | A hash entry names bytes the organization does not hold
|
|
133
|
+
| `422` `missing_blobs` | A hash entry names bytes the organization does not hold (above). |
|
|
45
134
|
| `413` `payload_too_large` | The method would leave no room for the summary a publish adds. |
|
|
46
135
|
|
|
47
|
-
An upload that fails throws `MethodFileTransferError` with `operation` `upload`, and the method is not saved (below).
|
|
48
|
-
|
|
49
136
|
## The deployment's limits
|
|
50
137
|
|
|
51
|
-
|
|
138
|
+
The version handshake (`GET /v1/version`) announces the limits the deployment holds a tree to under `limits.method_tree`: `max_files`, `max_file_bytes`, `max_total_bytes` and `max_inline_bytes`. A limit the server does not announce is absent, a value that is not a positive integer is left out rather than trusted, and none is ever assumed, so a bare runner, which announces none, yields no limit. The client keeps the handshake's answer for five minutes (`HANDSHAKE_TTL_MS` in `src/method-trees.ts`), so every call in that time reads the same answer, and asks again once it is older; a handshake that fails is asked again on the next call. A `404` from the blob route and a refusal of the tree's size each drop the kept answer at once, since each says it may be stale ([above](#how-a-tree-travels)).
|
|
139
|
+
|
|
140
|
+
`methodTreeLimits()` returns the three listing limits, `{ maxFileCount, maxFileSize, maxTotalSize }`, in the shape `listMethodTree` takes as its `limits` option, for a caller listing a tree of its own; a `directory` given to a call is listed under them already. Listed under those limits, a tree the platform would refuse for its size is refused while it is listed, with `mthds`'s `TreeLimitExceededError` naming the limit, before any file is hashed. The inline limit, `max_inline_bytes`, is no listing limit: the client applies it when it chooses how a tree travels.
|
|
141
|
+
|
|
142
|
+
```ts
|
|
143
|
+
import { listMethodTree } from "mthds";
|
|
144
|
+
|
|
145
|
+
const tree = await listMethodTree("methods/receipts", { limits: await client.methodTreeLimits() });
|
|
146
|
+
const method = await client.createMethod({ name: "Receipts", tree });
|
|
147
|
+
```
|
|
52
148
|
|
|
53
149
|
## Reading a tree
|
|
54
150
|
|
|
@@ -83,7 +179,7 @@ Each file is downloaded through its link, at most eight at a time, with no crede
|
|
|
83
179
|
|
|
84
180
|
## When a transfer fails
|
|
85
181
|
|
|
86
|
-
`MethodFileTransferError` is thrown when a file could not be moved between this machine and storage. `operation` says which way (`upload` for a save, `download` for `writeMethodTree`), `code` why, `sha256` which bytes, `path` the tree path when one is known, and `status` storage's HTTP status when it answered. Neither its message nor any member carries the link, nor storage's error body, which can echo the signed request, and it keeps no `cause`: the failure `fetch` throws for a request that got no answer can quote the URL, so only its network code (`ECONNREFUSED`, `ABORT_TIMEOUT`) reaches the message.
|
|
182
|
+
`MethodFileTransferError` is thrown when a file could not be moved between this machine and storage. `operation` says which way (`upload` for a request or a save that sends a tree by hash, `download` for `writeMethodTree`), `code` why, `sha256` which bytes, `path` the tree path when one is known, and `status` storage's HTTP status when it answered. Neither its message nor any member carries the link, nor storage's error body, which can echo the signed request, and it keeps no `cause`: the failure `fetch` throws for a request that got no answer can quote the URL, so only its network code (`ECONNREFUSED`, `ABORT_TIMEOUT`) reaches the message.
|
|
87
183
|
|
|
88
184
|
| `code` | When | Retryable |
|
|
89
185
|
|---|---|---|
|
|
@@ -96,6 +192,8 @@ Its `errorDomain` is always `runtime`. An upload to a target that asks for a cre
|
|
|
96
192
|
|
|
97
193
|
## What changed from the source fields
|
|
98
194
|
|
|
195
|
+
A run and a validation used to take a method as inline `.mthds` texts (`mthds_contents`, with `mthds_sources` naming them on `validate`), a zipped bundle (`bundle_b64`), or a `files` map of texts, with the files a step reads sent beside it as `resources`. They take its tree now, as every other call does, and `validateFiles`, `MthdsFile`, `ValidateFilesOptions`, `BundleResources` and the SDK's own copy of the runner's path rule for `resources` are gone, with no alias kept. A field the client no longer names still passes through `extra` to a runner that reads it.
|
|
196
|
+
|
|
99
197
|
A method used to hold its source as two strings, `mthds` (a bundle, or the JSON text of a file array) and `python` (the JSON text of the custom PipeFunc files), and a version carried `source_digest`. Both fields are gone from every read and every write, with no alias kept: a method is a tree, a version's digest is `tree_digest`, and the publish state is derived from the two digests, the draft unchanged when `draft_digest` is not `null` and equals `latest_published.tree_digest`. `methodSourceToContents`, `getMethodClosure` and `EmptyMethodSourceError` went with them.
|
|
100
198
|
|
|
101
199
|
A run's `source_digest` is something else, and stays: the SHA-256 of the sources a run sent as text, 64 hex digits with no prefix. `RunDetail` also carries the run's `tree_digest` and `crate_fingerprint`, each `null` for a run that recorded none.
|
package/docs/run-results.md
CHANGED
|
@@ -9,7 +9,10 @@ Two paths produce it. Against the hosted API the SDK starts a durable run and po
|
|
|
9
9
|
```ts
|
|
10
10
|
import { resultsFromExecute, summarizeUsage } from "@pipelex/sdk";
|
|
11
11
|
|
|
12
|
-
const executeResult = await client.execute({
|
|
12
|
+
const executeResult = await client.execute({
|
|
13
|
+
pipe_code: "my_domain.my_pipe",
|
|
14
|
+
contents: { "my_domain.mthds": source },
|
|
15
|
+
});
|
|
13
16
|
const results = resultsFromExecute(executeResult);
|
|
14
17
|
console.log(results.main_stuff, summarizeUsage(results).total_cost_usd);
|
|
15
18
|
```
|
|
@@ -72,7 +75,7 @@ const results = await client.startAndWaitForResult(
|
|
|
72
75
|
|
|
73
76
|
It is never called on the blocking path, a bare runner's `POST /v1/execute` or the fallback to it, which has no run id to give before it answers. The acknowledgement is handed over whole, a `method_ref` run's `method_provenance` and a `method_id` run's `method_version` (the version the run runs, a number or `"draft"`) included. The callback runs synchronously and its return value is ignored; an exception it throws propagates out of `startAndWaitForResult` before anything is polled, and the run it was told about keeps going.
|
|
74
77
|
|
|
75
|
-
**An abort before the run exists creates none.** The `signal` is read before the run is started as well as during the wait: a caller that aborts while the `GET /v1/version` handshake is in flight gets its abort, and neither `POST /v1/start` nor the blocking `POST /v1/execute` is sent; nor is the execute when the abort lands while a runner that looked hosted refuses the start. Once the start or the execute is sent, the abort stops only the wait, and a started run goes on, by its id.
|
|
78
|
+
**An abort before the run exists creates none.** The `signal` is read before the run is started as well as during the wait: a caller that aborts while the `GET /v1/version` handshake is in flight gets its abort, and neither `POST /v1/start` nor the blocking `POST /v1/execute` is sent; nor is the execute when the abort lands while a runner that looked hosted refuses the start. The same holds while the method's tree is prepared: the signal is handed to the directory's listing, each file's read, the blob route and each upload of a missing blob, so an abort stops the one under way and no run is started ([method-trees.md](./method-trees.md#how-a-tree-travels)). The tree is prepared once, before the start, so a fallback to the blocking execute sends the files the start sent, reading and uploading none of them again. Once the start or the execute is sent, the abort stops only the wait, and a started run goes on, by its id.
|
|
76
79
|
|
|
77
80
|
**Knowing whether a run may exist.** `onStarting`, the other callback `StartAndWaitForResultOptions` takes, is called right before each request that may create a run is sent: `POST /v1/start`, and the blocking `POST /v1/execute` of a bare runner or of the fallback to it, so it is called twice when a runner that looked hosted refuses the start. Until it is called no run exists, and once the `signal` has aborted it is not called and no such request is sent, so a caller that stops waiting before then can say that no run was started; from then until `onStarted`, a run may exist that the API has not yet named. It runs synchronously and its return value is ignored; an exception it throws propagates before the request is sent. The `pipelex-sdk` command words its interrupt message by it ([`cli.md`](cli.md)).
|
|
78
81
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@pipelex/sdk",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.48.0",
|
|
4
4
|
"description": "TypeScript SDK for the Pipelex hosted API — execute methods, manage runs, and call the product surface.",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"repository": {
|
|
@@ -74,7 +74,7 @@
|
|
|
74
74
|
"vitest": "^4.0.18"
|
|
75
75
|
},
|
|
76
76
|
"dependencies": {
|
|
77
|
-
"mthds": "^0.
|
|
77
|
+
"mthds": "^0.33.0",
|
|
78
78
|
"smol-toml": "^1.6.0",
|
|
79
79
|
"undici": "^7.29.1"
|
|
80
80
|
}
|
|
@@ -1,62 +0,0 @@
|
|
|
1
|
-
/**
|
|
2
|
-
* The files a bundle's steps read, sent with the bundle as the request's `resources`.
|
|
3
|
-
*
|
|
4
|
-
* A bundle sent as content has no directory, so a file one of its steps reads from beside the
|
|
5
|
-
* bundle, such as a `PipeDocGen` step's `template_file`, has to travel with it. The runner takes
|
|
6
|
-
* such files as `resources`: a JSON object mapping each file's path, exactly as the step names it,
|
|
7
|
-
* relative to the bundle and written with forward slashes, to the file's bytes in standard padded
|
|
8
|
-
* base64. It takes them beside `mthds_contents` (or a method bundle) on `/v1/validate`,
|
|
9
|
-
* `/v1/execute` and `/v1/start`, and beside inline `files[]` on `/v1/pipe-io`, `/v1/resolve` and
|
|
10
|
-
* `/v1/codegen`. A caller hands this SDK the bytes, keyed by that path, and the SDK encodes them.
|
|
11
|
-
*
|
|
12
|
-
* The SDK refuses before any request what the runner refuses for certain about the keys, with the
|
|
13
|
-
* runner's own rule (`resource_path_refusal` in pipelex's `bundle_resource.py`): an empty path, a
|
|
14
|
-
* backslash, a colon, a NUL character, an absolute path, a `..` segment, a path naming no file,
|
|
15
|
-
* and two keys naming one file once `.` segments and repeated slashes collapse. Each is a
|
|
16
|
-
* `RequestArgumentError` naming the key, where the runner answers a `422` `InvalidResourcePath`.
|
|
17
|
-
* The sizes are left to the server, since a deployment configures them: 32 files, 5 MiB each and
|
|
18
|
-
* 10 MiB in all by default, a `413` `PayloadTooLarge` beyond, and 512 KiB in all for the `.html`
|
|
19
|
-
* template files, a `413` `BundleResourcesTooLargeError`. So is every rule that needs the parsed
|
|
20
|
-
* bundle: a file no step names is a `422` `BundleResourceNotNamedError`.
|
|
21
|
-
*/
|
|
22
|
-
/** The bytes of one file a bundle's steps read: the byte forms `uploadFile` takes. */
|
|
23
|
-
export type BundleResourceBytes = Uint8Array | ArrayBuffer | Blob;
|
|
24
|
-
/**
|
|
25
|
-
* The files a bundle's steps read, such as a `PipeDocGen` step's template file, keyed by the path
|
|
26
|
-
* the step names (`templates/invoice.docx`), each as its bytes. `./templates/invoice.docx` names
|
|
27
|
-
* the same file, so only one of the two spellings may be given.
|
|
28
|
-
*/
|
|
29
|
-
export type BundleResources = Readonly<Record<string, BundleResourceBytes>>;
|
|
30
|
-
/**
|
|
31
|
-
* Whether `resources` is a plain object, the one shape a map of paths to bytes takes: its
|
|
32
|
-
* prototype is `Object.prototype` or `null`. A `Map`, an array or a class instance is not one,
|
|
33
|
-
* and the encoder refuses it rather than reading it as a map with no file in it.
|
|
34
|
-
*/
|
|
35
|
-
export declare function isResourceMap(resources: unknown): resources is Record<string, unknown>;
|
|
36
|
-
/**
|
|
37
|
-
* Whether `resources` carries at least one file. An absent, `null` or empty map carries none, so
|
|
38
|
-
* it is not sent, as an empty bundle encoding is not. A value that is not a plain object carries
|
|
39
|
-
* none either; the encoder refuses it.
|
|
40
|
-
*/
|
|
41
|
-
export declare function hasBundleResources(resources: unknown): boolean;
|
|
42
|
-
/**
|
|
43
|
-
* Why `path` cannot name a file inside a bundle, or `undefined` when it can — the runner's rule,
|
|
44
|
-
* held by the key of a file sent with a bundle and by the path a step names.
|
|
45
|
-
*/
|
|
46
|
-
export declare function bundleResourcePathRefusal(path: string): string | undefined;
|
|
47
|
-
/**
|
|
48
|
-
* The one spelling of a bundle-relative path the runner keys a file by: `.` segments and repeated
|
|
49
|
-
* slashes collapse, so `./templates//invoice.docx` is `templates/invoice.docx`. Only meaningful
|
|
50
|
-
* for a path {@link bundleResourcePathRefusal} accepts.
|
|
51
|
-
*/
|
|
52
|
-
export declare function canonicalBundleResourcePath(path: string): string;
|
|
53
|
-
/**
|
|
54
|
-
* Encode `resources` for the wire: each key as given, each value as its bytes in standard padded
|
|
55
|
-
* base64. Returns `undefined` when there is nothing to send, so the key stays off the wire.
|
|
56
|
-
*
|
|
57
|
-
* Every key is checked before any value is read, so a refused path costs no read of a `Blob`.
|
|
58
|
-
*
|
|
59
|
-
* @throws {RequestArgumentError} `resources` is not a map of paths to bytes, a key cannot name a
|
|
60
|
-
* file inside the bundle, two keys name one file, or a value is not bytes.
|
|
61
|
-
*/
|
|
62
|
-
export declare function encodeBundleResources(resources: BundleResources | null | undefined): Promise<Record<string, string> | undefined>;
|
package/dist/bundle-resources.js
DELETED
|
@@ -1,165 +0,0 @@
|
|
|
1
|
-
/**
|
|
2
|
-
* The files a bundle's steps read, sent with the bundle as the request's `resources`.
|
|
3
|
-
*
|
|
4
|
-
* A bundle sent as content has no directory, so a file one of its steps reads from beside the
|
|
5
|
-
* bundle, such as a `PipeDocGen` step's `template_file`, has to travel with it. The runner takes
|
|
6
|
-
* such files as `resources`: a JSON object mapping each file's path, exactly as the step names it,
|
|
7
|
-
* relative to the bundle and written with forward slashes, to the file's bytes in standard padded
|
|
8
|
-
* base64. It takes them beside `mthds_contents` (or a method bundle) on `/v1/validate`,
|
|
9
|
-
* `/v1/execute` and `/v1/start`, and beside inline `files[]` on `/v1/pipe-io`, `/v1/resolve` and
|
|
10
|
-
* `/v1/codegen`. A caller hands this SDK the bytes, keyed by that path, and the SDK encodes them.
|
|
11
|
-
*
|
|
12
|
-
* The SDK refuses before any request what the runner refuses for certain about the keys, with the
|
|
13
|
-
* runner's own rule (`resource_path_refusal` in pipelex's `bundle_resource.py`): an empty path, a
|
|
14
|
-
* backslash, a colon, a NUL character, an absolute path, a `..` segment, a path naming no file,
|
|
15
|
-
* and two keys naming one file once `.` segments and repeated slashes collapse. Each is a
|
|
16
|
-
* `RequestArgumentError` naming the key, where the runner answers a `422` `InvalidResourcePath`.
|
|
17
|
-
* The sizes are left to the server, since a deployment configures them: 32 files, 5 MiB each and
|
|
18
|
-
* 10 MiB in all by default, a `413` `PayloadTooLarge` beyond, and 512 KiB in all for the `.html`
|
|
19
|
-
* template files, a `413` `BundleResourcesTooLargeError`. So is every rule that needs the parsed
|
|
20
|
-
* bundle: a file no step names is a `422` `BundleResourceNotNamedError`.
|
|
21
|
-
*/
|
|
22
|
-
import { RequestArgumentError } from "./errors.js";
|
|
23
|
-
import { bytesToBase64 } from "./upload.js";
|
|
24
|
-
/**
|
|
25
|
-
* Whether `resources` is a plain object, the one shape a map of paths to bytes takes: its
|
|
26
|
-
* prototype is `Object.prototype` or `null`. A `Map`, an array or a class instance is not one,
|
|
27
|
-
* and the encoder refuses it rather than reading it as a map with no file in it.
|
|
28
|
-
*/
|
|
29
|
-
export function isResourceMap(resources) {
|
|
30
|
-
if (typeof resources !== "object" || resources === null)
|
|
31
|
-
return false;
|
|
32
|
-
const prototype = Object.getPrototypeOf(resources);
|
|
33
|
-
return prototype === Object.prototype || prototype === null;
|
|
34
|
-
}
|
|
35
|
-
/**
|
|
36
|
-
* Whether `resources` carries at least one file. An absent, `null` or empty map carries none, so
|
|
37
|
-
* it is not sent, as an empty bundle encoding is not. A value that is not a plain object carries
|
|
38
|
-
* none either; the encoder refuses it.
|
|
39
|
-
*/
|
|
40
|
-
export function hasBundleResources(resources) {
|
|
41
|
-
return isResourceMap(resources) && Object.keys(resources).length > 0;
|
|
42
|
-
}
|
|
43
|
-
/**
|
|
44
|
-
* Python's whitespace, the set `str.strip()` removes and `str.isspace()` accepts, which the
|
|
45
|
-
* runner's rule tests a path against. It differs from JavaScript's `trim()`: it holds the
|
|
46
|
-
* separators U+001C to U+001F and U+0085, and not the byte-order mark U+FEFF.
|
|
47
|
-
*/
|
|
48
|
-
const PYTHON_WHITESPACE = new Set([
|
|
49
|
-
0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x1c, 0x1d, 0x1e, 0x1f, 0x20, 0x85, 0xa0, 0x1680, 0x2000, 0x2001,
|
|
50
|
-
0x2002, 0x2003, 0x2004, 0x2005, 0x2006, 0x2007, 0x2008, 0x2009, 0x200a, 0x2028, 0x2029, 0x202f,
|
|
51
|
-
0x205f, 0x3000,
|
|
52
|
-
]);
|
|
53
|
-
/** Whether `path` holds nothing but Python's whitespace: `not path.strip()`, the runner's test. */
|
|
54
|
-
function isBlankForPython(path) {
|
|
55
|
-
for (const character of path) {
|
|
56
|
-
if (!PYTHON_WHITESPACE.has(character.codePointAt(0)))
|
|
57
|
-
return false;
|
|
58
|
-
}
|
|
59
|
-
return true;
|
|
60
|
-
}
|
|
61
|
-
/**
|
|
62
|
-
* Why `path` cannot name a file inside a bundle, or `undefined` when it can — the runner's rule,
|
|
63
|
-
* held by the key of a file sent with a bundle and by the path a step names.
|
|
64
|
-
*/
|
|
65
|
-
export function bundleResourcePathRefusal(path) {
|
|
66
|
-
if (isBlankForPython(path))
|
|
67
|
-
return "it is empty";
|
|
68
|
-
if (path.includes("\\"))
|
|
69
|
-
return "it holds a backslash; write the path with forward slashes";
|
|
70
|
-
if (path.includes(":"))
|
|
71
|
-
return "it holds a colon";
|
|
72
|
-
if (path.includes("\0"))
|
|
73
|
-
return "it holds a NUL character";
|
|
74
|
-
if (path.startsWith("/"))
|
|
75
|
-
return "it is absolute; write it relative to the bundle";
|
|
76
|
-
const segments = pathSegments(path);
|
|
77
|
-
if (segments.includes(".."))
|
|
78
|
-
return "it leaves the bundle through a '..' segment";
|
|
79
|
-
if (segments.length === 0)
|
|
80
|
-
return "it names no file";
|
|
81
|
-
return undefined;
|
|
82
|
-
}
|
|
83
|
-
/**
|
|
84
|
-
* The one spelling of a bundle-relative path the runner keys a file by: `.` segments and repeated
|
|
85
|
-
* slashes collapse, so `./templates//invoice.docx` is `templates/invoice.docx`. Only meaningful
|
|
86
|
-
* for a path {@link bundleResourcePathRefusal} accepts.
|
|
87
|
-
*/
|
|
88
|
-
export function canonicalBundleResourcePath(path) {
|
|
89
|
-
return pathSegments(path).join("/");
|
|
90
|
-
}
|
|
91
|
-
function pathSegments(path) {
|
|
92
|
-
return path.split("/").filter((segment) => segment !== "" && segment !== ".");
|
|
93
|
-
}
|
|
94
|
-
/**
|
|
95
|
-
* Encode `resources` for the wire: each key as given, each value as its bytes in standard padded
|
|
96
|
-
* base64. Returns `undefined` when there is nothing to send, so the key stays off the wire.
|
|
97
|
-
*
|
|
98
|
-
* Every key is checked before any value is read, so a refused path costs no read of a `Blob`.
|
|
99
|
-
*
|
|
100
|
-
* @throws {RequestArgumentError} `resources` is not a map of paths to bytes, a key cannot name a
|
|
101
|
-
* file inside the bundle, two keys name one file, or a value is not bytes.
|
|
102
|
-
*/
|
|
103
|
-
export async function encodeBundleResources(resources) {
|
|
104
|
-
if (resources === undefined || resources === null)
|
|
105
|
-
return undefined;
|
|
106
|
-
if (!isResourceMap(resources)) {
|
|
107
|
-
throw new RequestArgumentError("resources must be a plain object mapping each file's path, as the step names it, to the " +
|
|
108
|
-
`file's bytes; got ${describeValue(resources)}.`);
|
|
109
|
-
}
|
|
110
|
-
// Read once: a second read of a proxy or a getter could differ from the one checked.
|
|
111
|
-
const entries = Object.entries(resources);
|
|
112
|
-
if (entries.length === 0)
|
|
113
|
-
return undefined;
|
|
114
|
-
const pathByCanonical = new Map();
|
|
115
|
-
for (const [path] of entries) {
|
|
116
|
-
const refusal = bundleResourcePathRefusal(path);
|
|
117
|
-
if (refusal !== undefined) {
|
|
118
|
-
throw new RequestArgumentError(`resources key ${JSON.stringify(path)} cannot name a file inside the bundle: ${refusal}.`);
|
|
119
|
-
}
|
|
120
|
-
const canonical = canonicalBundleResourcePath(path);
|
|
121
|
-
const first = pathByCanonical.get(canonical);
|
|
122
|
-
if (first !== undefined) {
|
|
123
|
-
throw new RequestArgumentError(`resources keys ${JSON.stringify(first)} and ${JSON.stringify(path)} name the same file ` +
|
|
124
|
-
`"${canonical}": send it once.`);
|
|
125
|
-
}
|
|
126
|
-
pathByCanonical.set(canonical, path);
|
|
127
|
-
}
|
|
128
|
-
// A null prototype, so a key such as `__proto__` is an own property that reaches the wire.
|
|
129
|
-
const wire = Object.create(null);
|
|
130
|
-
for (const [path, value] of entries) {
|
|
131
|
-
wire[path] = bytesToBase64(await resourceBytes(path, value));
|
|
132
|
-
}
|
|
133
|
-
return wire;
|
|
134
|
-
}
|
|
135
|
-
async function resourceBytes(path, value) {
|
|
136
|
-
if (value instanceof Uint8Array)
|
|
137
|
-
return value;
|
|
138
|
-
if (value instanceof ArrayBuffer)
|
|
139
|
-
return new Uint8Array(value);
|
|
140
|
-
if (typeof Blob !== "undefined" && value instanceof Blob) {
|
|
141
|
-
return new Uint8Array(await value.arrayBuffer());
|
|
142
|
-
}
|
|
143
|
-
throw new RequestArgumentError(`resources[${JSON.stringify(path)}] must be the file's bytes (a Uint8Array, an ArrayBuffer ` +
|
|
144
|
-
`or a Blob); got ${describeValue(value)}.` +
|
|
145
|
-
(typeof value === "string"
|
|
146
|
-
? " A string is read neither as a path nor as base64: read the file and pass its bytes."
|
|
147
|
-
: ""));
|
|
148
|
-
}
|
|
149
|
-
function describeValue(value) {
|
|
150
|
-
if (value === null)
|
|
151
|
-
return "null";
|
|
152
|
-
if (Array.isArray(value))
|
|
153
|
-
return "an array";
|
|
154
|
-
if (value instanceof Map)
|
|
155
|
-
return "a Map; pass Object.fromEntries(map)";
|
|
156
|
-
if (typeof value === "string")
|
|
157
|
-
return "a string";
|
|
158
|
-
if (typeof value === "object") {
|
|
159
|
-
const name = Object.getPrototypeOf(value)
|
|
160
|
-
?.constructor?.name;
|
|
161
|
-
return typeof name === "string" && name !== "" ? `an instance of ${name}` : "an object";
|
|
162
|
-
}
|
|
163
|
-
return `a value of type ${typeof value}`;
|
|
164
|
-
}
|
|
165
|
-
//# sourceMappingURL=bundle-resources.js.map
|
|
@@ -1 +0,0 @@
|
|
|
1
|
-
{"version":3,"file":"bundle-resources.js","sourceRoot":"","sources":["../src/bundle-resources.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;GAoBG;AAEH,OAAO,EAAE,oBAAoB,EAAE,MAAM,aAAa,CAAC;AACnD,OAAO,EAAE,aAAa,EAAE,MAAM,aAAa,CAAC;AAY5C;;;;GAIG;AACH,MAAM,UAAU,aAAa,CAAC,SAAkB;IAC9C,IAAI,OAAO,SAAS,KAAK,QAAQ,IAAI,SAAS,KAAK,IAAI;QAAE,OAAO,KAAK,CAAC;IACtE,MAAM,SAAS,GAAY,MAAM,CAAC,cAAc,CAAC,SAAS,CAAC,CAAC;IAC5D,OAAO,SAAS,KAAK,MAAM,CAAC,SAAS,IAAI,SAAS,KAAK,IAAI,CAAC;AAC9D,CAAC;AAED;;;;GAIG;AACH,MAAM,UAAU,kBAAkB,CAAC,SAAkB;IACnD,OAAO,aAAa,CAAC,SAAS,CAAC,IAAI,MAAM,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC,MAAM,GAAG,CAAC,CAAC;AACvE,CAAC;AAED;;;;GAIG;AACH,MAAM,iBAAiB,GAAwB,IAAI,GAAG,CAAC;IACrD,IAAI,EAAE,IAAI,EAAE,IAAI,EAAE,IAAI,EAAE,IAAI,EAAE,IAAI,EAAE,IAAI,EAAE,IAAI,EAAE,IAAI,EAAE,IAAI,EAAE,IAAI,EAAE,IAAI,EAAE,MAAM,EAAE,MAAM,EAAE,MAAM;IAC9F,MAAM,EAAE,MAAM,EAAE,MAAM,EAAE,MAAM,EAAE,MAAM,EAAE,MAAM,EAAE,MAAM,EAAE,MAAM,EAAE,MAAM,EAAE,MAAM,EAAE,MAAM,EAAE,MAAM;IAC9F,MAAM,EAAE,MAAM;CACf,CAAC,CAAC;AAEH,mGAAmG;AACnG,SAAS,gBAAgB,CAAC,IAAY;IACpC,KAAK,MAAM,SAAS,IAAI,IAAI,EAAE,CAAC;QAC7B,IAAI,CAAC,iBAAiB,CAAC,GAAG,CAAC,SAAS,CAAC,WAAW,CAAC,CAAC,CAAE,CAAC;YAAE,OAAO,KAAK,CAAC;IACtE,CAAC;IACD,OAAO,IAAI,CAAC;AACd,CAAC;AAED;;;GAGG;AACH,MAAM,UAAU,yBAAyB,CAAC,IAAY;IACpD,IAAI,gBAAgB,CAAC,IAAI,CAAC;QAAE,OAAO,aAAa,CAAC;IACjD,IAAI,IAAI,CAAC,QAAQ,CAAC,IAAI,CAAC;QAAE,OAAO,2DAA2D,CAAC;IAC5F,IAAI,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC;QAAE,OAAO,kBAAkB,CAAC;IAClD,IAAI,IAAI,CAAC,QAAQ,CAAC,IAAI,CAAC;QAAE,OAAO,0BAA0B,CAAC;IAC3D,IAAI,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC;QAAE,OAAO,iDAAiD,CAAC;IACnF,MAAM,QAAQ,GAAG,YAAY,CAAC,IAAI,CAAC,CAAC;IACpC,IAAI,QAAQ,CAAC,QAAQ,CAAC,IAAI,CAAC;QAAE,OAAO,6CAA6C,CAAC;IAClF,IAAI,QAAQ,CAAC,MAAM,KAAK,CAAC;QAAE,OAAO,kBAAkB,CAAC;IACrD,OAAO,SAAS,CAAC;AACnB,CAAC;AAED;;;;GAIG;AACH,MAAM,UAAU,2BAA2B,CAAC,IAAY;IACtD,OAAO,YAAY,CAAC,IAAI,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;AACtC,CAAC;AAED,SAAS,YAAY,CAAC,IAAY;IAChC,OAAO,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,MAAM,CAAC,CAAC,OAAO,EAAE,EAAE,CAAC,OAAO,KAAK,EAAE,IAAI,OAAO,KAAK,GAAG,CAAC,CAAC;AAChF,CAAC;AAED;;;;;;;;GAQG;AACH,MAAM,CAAC,KAAK,UAAU,qBAAqB,CACzC,SAA6C;IAE7C,IAAI,SAAS,KAAK,SAAS,IAAI,SAAS,KAAK,IAAI;QAAE,OAAO,SAAS,CAAC;IACpE,IAAI,CAAC,aAAa,CAAC,SAAS,CAAC,EAAE,CAAC;QAC9B,MAAM,IAAI,oBAAoB,CAC5B,0FAA0F;YACxF,qBAAqB,aAAa,CAAC,SAAS,CAAC,GAAG,CACnD,CAAC;IACJ,CAAC;IACD,qFAAqF;IACrF,MAAM,OAAO,GAAG,MAAM,CAAC,OAAO,CAAC,SAAS,CAAC,CAAC;IAC1C,IAAI,OAAO,CAAC,MAAM,KAAK,CAAC;QAAE,OAAO,SAAS,CAAC;IAE3C,MAAM,eAAe,GAAG,IAAI,GAAG,EAAkB,CAAC;IAClD,KAAK,MAAM,CAAC,IAAI,CAAC,IAAI,OAAO,EAAE,CAAC;QAC7B,MAAM,OAAO,GAAG,yBAAyB,CAAC,IAAI,CAAC,CAAC;QAChD,IAAI,OAAO,KAAK,SAAS,EAAE,CAAC;YAC1B,MAAM,IAAI,oBAAoB,CAC5B,iBAAiB,IAAI,CAAC,SAAS,CAAC,IAAI,CAAC,0CAA0C,OAAO,GAAG,CAC1F,CAAC;QACJ,CAAC;QACD,MAAM,SAAS,GAAG,2BAA2B,CAAC,IAAI,CAAC,CAAC;QACpD,MAAM,KAAK,GAAG,eAAe,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC;QAC7C,IAAI,KAAK,KAAK,SAAS,EAAE,CAAC;YACxB,MAAM,IAAI,oBAAoB,CAC5B,kBAAkB,IAAI,CAAC,SAAS,CAAC,KAAK,CAAC,QAAQ,IAAI,CAAC,SAAS,CAAC,IAAI,CAAC,sBAAsB;gBACvF,IAAI,SAAS,kBAAkB,CAClC,CAAC;QACJ,CAAC;QACD,eAAe,CAAC,GAAG,CAAC,SAAS,EAAE,IAAI,CAAC,CAAC;IACvC,CAAC;IAED,2FAA2F;IAC3F,MAAM,IAAI,GAAG,MAAM,CAAC,MAAM,CAAC,IAAI,CAA2B,CAAC;IAC3D,KAAK,MAAM,CAAC,IAAI,EAAE,KAAK,CAAC,IAAI,OAAO,EAAE,CAAC;QACpC,IAAI,CAAC,IAAI,CAAC,GAAG,aAAa,CAAC,MAAM,aAAa,CAAC,IAAI,EAAE,KAAK,CAAC,CAAC,CAAC;IAC/D,CAAC;IACD,OAAO,IAAI,CAAC;AACd,CAAC;AAED,KAAK,UAAU,aAAa,CAAC,IAAY,EAAE,KAAc;IACvD,IAAI,KAAK,YAAY,UAAU;QAAE,OAAO,KAAK,CAAC;IAC9C,IAAI,KAAK,YAAY,WAAW;QAAE,OAAO,IAAI,UAAU,CAAC,KAAK,CAAC,CAAC;IAC/D,IAAI,OAAO,IAAI,KAAK,WAAW,IAAI,KAAK,YAAY,IAAI,EAAE,CAAC;QACzD,OAAO,IAAI,UAAU,CAAC,MAAM,KAAK,CAAC,WAAW,EAAE,CAAC,CAAC;IACnD,CAAC;IACD,MAAM,IAAI,oBAAoB,CAC5B,aAAa,IAAI,CAAC,SAAS,CAAC,IAAI,CAAC,2DAA2D;QAC1F,mBAAmB,aAAa,CAAC,KAAK,CAAC,GAAG;QAC1C,CAAC,OAAO,KAAK,KAAK,QAAQ;YACxB,CAAC,CAAC,sFAAsF;YACxF,CAAC,CAAC,EAAE,CAAC,CACV,CAAC;AACJ,CAAC;AAED,SAAS,aAAa,CAAC,KAAc;IACnC,IAAI,KAAK,KAAK,IAAI;QAAE,OAAO,MAAM,CAAC;IAClC,IAAI,KAAK,CAAC,OAAO,CAAC,KAAK,CAAC;QAAE,OAAO,UAAU,CAAC;IAC5C,IAAI,KAAK,YAAY,GAAG;QAAE,OAAO,qCAAqC,CAAC;IACvE,IAAI,OAAO,KAAK,KAAK,QAAQ;QAAE,OAAO,UAAU,CAAC;IACjD,IAAI,OAAO,KAAK,KAAK,QAAQ,EAAE,CAAC;QAC9B,MAAM,IAAI,GAAI,MAAM,CAAC,cAAc,CAAC,KAAK,CAAiD;YACxF,EAAE,WAAW,EAAE,IAAI,CAAC;QACtB,OAAO,OAAO,IAAI,KAAK,QAAQ,IAAI,IAAI,KAAK,EAAE,CAAC,CAAC,CAAC,kBAAkB,IAAI,EAAE,CAAC,CAAC,CAAC,WAAW,CAAC;IAC1F,CAAC;IACD,OAAO,mBAAmB,OAAO,KAAK,EAAE,CAAC;AAC3C,CAAC"}
|