@statewalker/webrun-files-rpc 0.1.0 → 0.1.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +43 -13
- package/dist/cjs/index.cjs +147 -26
- package/dist/client.d.ts +5 -0
- package/dist/client.d.ts.map +1 -1
- package/dist/errors.d.ts.map +1 -1
- package/dist/esm/index.js +147 -26
- package/dist/index.d.ts +1 -1
- package/dist/index.d.ts.map +1 -1
- package/dist/server.d.ts +2 -0
- package/dist/server.d.ts.map +1 -1
- package/dist/types.d.ts +35 -7
- package/dist/types.d.ts.map +1 -1
- package/package.json +4 -4
- package/src/client.ts +31 -6
- package/src/errors.ts +13 -4
- package/src/index.ts +3 -0
- package/src/server.ts +158 -13
- package/src/types.ts +38 -7
package/README.md
CHANGED
|
@@ -7,7 +7,7 @@ A `FilesApi` (from `@statewalker/webrun-files`) over any RPC, with streamed read
|
|
|
7
7
|
- **Server** (`newFilesRpcServer({ fs })`): serves any `FilesApi` as plain method calls.
|
|
8
8
|
- **Client** (`newFilesRpcClient({ rpc })`): a `FilesApi` whose every call goes through those
|
|
9
9
|
methods: a Durable Object or Workers RPC stub, comlink, `@statewalker/webrun-rpc`, or the server
|
|
10
|
-
object itself.
|
|
10
|
+
object itself. It also appends (`append(path, content)`), which `FilesApi` does not.
|
|
11
11
|
|
|
12
12
|
Content moves as **chunks keyed by `(streamId, chunkId)`**. Both sides keep chunks by their key
|
|
13
13
|
and wait for the ones still on their way:
|
|
@@ -37,10 +37,13 @@ Keys make the calls safe on real transports:
|
|
|
37
37
|
- a chunk read twice is answered twice;
|
|
38
38
|
- an open with the same `streamId` opens one stream;
|
|
39
39
|
- a `closeWrite` delivered again answers as the first did.
|
|
40
|
-
-
|
|
41
|
-
|
|
42
|
-
`
|
|
43
|
-
|
|
40
|
+
- a metadata call (`mkdir`, `remove`, `move`, `copy`) with a `requestId` runs once: delivered
|
|
41
|
+
again, concurrently or later, it gets the first answer (or failure). The server keeps answers
|
|
42
|
+
for `idleTimeout`, at most `maxRequests` of them. The client sends a fresh `requestId` with
|
|
43
|
+
every such call.
|
|
44
|
+
- Without a `requestId`, a metadata call delivered twice runs twice: the end state is the same
|
|
45
|
+
when nothing else writes in between, but the second may report `false` though the first did
|
|
46
|
+
it, and a late `copy` retry copies again.
|
|
44
47
|
|
|
45
48
|
## How to use
|
|
46
49
|
|
|
@@ -58,8 +61,8 @@ Exports:
|
|
|
58
61
|
- `newFilesRpcServer`, `newFilesRpcClient`, `FilesRpcClient`;
|
|
59
62
|
- `FilesRpcError` and `errorCode`;
|
|
60
63
|
- types `FilesRpc`, `FilesRpcServer`, `FilesRpcServerOptions`, `FilesRpcClientOptions`,
|
|
61
|
-
`ReadResult`, `ListPage`, `ListPageOptions`, `OpenOptions`,
|
|
62
|
-
`FilesRpcErrorCode`.
|
|
64
|
+
`AppendableFilesApi`, `ReadResult`, `ListPage`, `ListPageOptions`, `OpenOptions`,
|
|
65
|
+
`OpenReadOptions`, `OpenWriteOptions`, `RequestOptions`, `FilesRpcErrorCode`.
|
|
63
66
|
|
|
64
67
|
### `newFilesRpcServer(options)`
|
|
65
68
|
|
|
@@ -75,6 +78,7 @@ Exports:
|
|
|
75
78
|
| `idleTimeout` | 2 min | A stream not used for this long is dropped (a read closed, a write aborted). |
|
|
76
79
|
| `maxStreams` | 256 | Open streams at most (`TOO_MANY_STREAMS` beyond). |
|
|
77
80
|
| `maxPageSize` | 256 | Largest listing page. |
|
|
81
|
+
| `maxRequests` | 1024 | Answers of metadata calls kept by `requestId` (oldest dropped beyond). |
|
|
78
82
|
| `sweepInterval` | none | Sweep idle streams every this many ms. Without it, opening a stream sweeps; a server that stops receiving opens needs this (or calls to `sweep()`). |
|
|
79
83
|
| `now` | `Date.now` | The clock (tests). |
|
|
80
84
|
|
|
@@ -94,6 +98,9 @@ The server is a `FilesRpc` plus:
|
|
|
94
98
|
| `writeAhead` | 4 | `writeChunk` calls not yet answered while writing. |
|
|
95
99
|
| `pageSize` | 256 | Listing page size. |
|
|
96
100
|
|
|
101
|
+
Besides the `FilesApi` methods, the client has `append(path, content)`: it adds `content` to the
|
|
102
|
+
end of `path` (creating it if it does not exist), streamed exactly as `write` streams.
|
|
103
|
+
|
|
97
104
|
### The protocol (`FilesRpc`)
|
|
98
105
|
|
|
99
106
|
Every argument and result is a plain value: strings, numbers, booleans, plain objects and
|
|
@@ -104,12 +111,13 @@ Every argument and result is a plain value: strings, numbers, booleans, plain ob
|
|
|
104
111
|
| `openRead(path, {start?, length?, streamId?})` | `{streamId}`; a file that does not exist reads as nothing |
|
|
105
112
|
| `readChunk(streamId, chunkId)` | `{data, done}` once chunk `chunkId` exists; past the end `{data: empty, done: true}` |
|
|
106
113
|
| `closeRead(streamId)` | ends the stream (unknown or ended: nothing happens) |
|
|
107
|
-
| `openWrite(path, {streamId?})` | `{streamId}`; the `FilesApi`'s `write` starts consuming chunks in order |
|
|
114
|
+
| `openWrite(path, {streamId?, append?})` | `{streamId}`; the `FilesApi`'s `write` (or, with `append`, the append) starts consuming chunks in order |
|
|
108
115
|
| `writeChunk(streamId, chunkId, data)` | resolves once the chunk is kept (within `writeWindow` of the next to write: at once) |
|
|
109
116
|
| `closeWrite(streamId, chunks)` | resolves once all `chunks` chunks are written |
|
|
110
117
|
| `abortWrite(streamId, reason?)` | cancels the write (unknown or ended: nothing happens) |
|
|
111
118
|
| `list(path, {recursive?, after?, limit?})` | `{entries, more}`; ask again with `after` = the last path |
|
|
112
|
-
| `stats`, `exists
|
|
119
|
+
| `stats`, `exists` | as the `FilesApi` |
|
|
120
|
+
| `mkdir(path, {requestId?})`, `remove(path, {requestId?})`, `move(source, target, {requestId?})`, `copy(source, target, {requestId?})` | as the `FilesApi`; with a `requestId`, once (a call delivered again gets the first answer) |
|
|
113
121
|
|
|
114
122
|
- **Stream ids:** `streamId` may be chosen by the caller: 16 to 128 characters of `[A-Za-z0-9_-]`,
|
|
115
123
|
random, such as `crypto.randomUUID()` without dashes. Then an open delivered twice opens one
|
|
@@ -216,6 +224,27 @@ await server.closeRead(r);
|
|
|
216
224
|
keeps what it had.
|
|
217
225
|
- `write()` starts at `openWrite`, not at the first chunk. A FilesApi that truncates when a
|
|
218
226
|
write starts has emptied the file already if the write is then aborted.
|
|
227
|
+
- **Appends:** `openWrite(path, { append: true })` streams like a write, and the chunks go to the
|
|
228
|
+
end of the file.
|
|
229
|
+
- A served FilesApi with an `append(path, content)` method (`AppendableFilesApi`) appends
|
|
230
|
+
through it, with whatever atomicity it gives.
|
|
231
|
+
- Any other FilesApi is appended to with `read`, `write` and `remove` only, in two steps:
|
|
232
|
+
1. the file's content and the chunks are written to a temporary sibling,
|
|
233
|
+
`.<name>.append-<id>` (the name cut to 64 characters, the stream id to 16);
|
|
234
|
+
2. the sibling is written back over the file, then removed.
|
|
235
|
+
- It never reads and writes one path at once, so a FilesApi that empties a file when a write
|
|
236
|
+
starts is safe. Only a directory is refused (`INVALID`); `stats` saying the file is missing is
|
|
237
|
+
not trusted, since reading a missing file yields nothing and the file is created all the same.
|
|
238
|
+
- A failure or an abort during step 1 leaves the file as it was and removes the sibling. An
|
|
239
|
+
abort counts until step 2 starts, even after every chunk was sent. A failure during step 2
|
|
240
|
+
may leave the file incomplete, so the sibling, the only complete copy, is kept and its path
|
|
241
|
+
is in the error.
|
|
242
|
+
- Appends to one file (by its normal path) through one server run one after the other. Other
|
|
243
|
+
writers are not excluded. The emulation copies the file twice and shows the sibling in
|
|
244
|
+
listings while it runs.
|
|
245
|
+
- `close()` and expiry leave a write whose outcome is the FilesApi's alone (every chunk handed
|
|
246
|
+
over, or an emulated append writing back), so a `closeWrite` waiting on it gets the real
|
|
247
|
+
outcome.
|
|
219
248
|
- **Ended streams:** ids that ended are remembered for `idleTimeout`. Within that time, an open
|
|
220
249
|
delivered after its stream ended does not reopen it, and a `closeWrite` delivered again gets the
|
|
221
250
|
first answer.
|
|
@@ -235,8 +264,8 @@ await server.closeRead(r);
|
|
|
235
264
|
- **The client:**
|
|
236
265
|
- **Read:** it keeps `readAhead` `readChunk` calls in flight and yields the answers in key
|
|
237
266
|
order. Breaking out of the loop, or aborting through `signal`, closes the stream.
|
|
238
|
-
- **Write:** it cuts any sequence of byte arrays into chunks of exactly `chunkSize`
|
|
239
|
-
`writeAhead` unanswered. Any failure (of the source, or of a call) aborts the write.
|
|
267
|
+
- **Write and append:** it cuts any sequence of byte arrays into chunks of exactly `chunkSize`
|
|
268
|
+
and keeps `writeAhead` unanswered. Any failure (of the source, or of a call) aborts the write.
|
|
240
269
|
- **List:** it asks page after page with `after`. The server also applies `after` itself, in
|
|
241
270
|
code-point order, so a FilesApi that ignores it cannot make pages repeat.
|
|
242
271
|
- **Not covered:**
|
|
@@ -257,9 +286,10 @@ encoding come from `webrun-files-tests`.
|
|
|
257
286
|
| `server-read.test.ts` | numbered chunks with `done`, out-of-order and concurrent requests, ranges, missing files, retries, release, bounded read-ahead, closing, failing sources |
|
|
258
287
|
| `server-write.test.ts` | in-order and out-of-order chunks, duplicates, backpressure, `closeWrite` waiting for a gap or refusing a wrong count, empty files, abort, failing writes, byte types |
|
|
259
288
|
| `server-lifecycle.test.ts` | idle expiry, the stream limit, `close()`, unguessable ids, paged listing, metadata |
|
|
260
|
-
| `protocol.test.ts` | caller-chosen keys (opens and `closeWrite` delivered twice, after the end), what is refused, waiting requests failed by a close, error codes across a structured clone, many streams at once, release never dropping an undelivered chunk, metadata calls delivered twice |
|
|
289
|
+
| `protocol.test.ts` | caller-chosen keys (opens and `closeWrite` delivered twice, after the end), what is refused, waiting requests failed by a close, error codes across a structured clone, many streams at once, release never dropping an undelivered chunk, metadata calls delivered twice (with a `requestId`: once, the same answer, failures too, kept for `idleTimeout` and `maxRequests`; without: the same end state) |
|
|
261
290
|
| `robustness.test.ts` | the window (huge chunk ids refused at once, bounded pulls), busy streams not swept, writes too far ahead refused, writes waiting for the holder swept, the sweep timer, a failing write's error reaching the client, a late abort not undoing a commit, a source yielding non-bytes, waiters past the count, aborting a hanging read, `close()` not hanging, listing with a FilesApi that ignores `after` |
|
|
262
291
|
| `client.test.ts` | the `FilesApi` round trip, iterables, ranges, pipelining depth, early stop and abort, failing sources, write depth, listing; delayed, reordered and duplicated calls (five seeds) |
|
|
263
|
-
| `
|
|
292
|
+
| `append.test.ts` | appends through the FilesApi's own `append` and through the emulation, also on a FilesApi that writes as it goes: existing and missing files, large files, aborts (also after every chunk was sent) and failing sources leaving the file unchanged and no temporary file, a failed write-back keeping the temporary copy, `close()` during the write-back, two appends at once, `stats` not trusted, odd paths and long names, directories refused; over the transport simulator; an open delivered twice |
|
|
293
|
+
| `conformance.test.ts` | the repository's `FilesApi` suite (`webrun-files-tests`) on the client over the server: directly, and through the transport simulator (cloned, reordered, every call delivered twice at random) |
|
|
264
294
|
| `big-files.test.ts` | the repository's big-file suite (64 MiB) on the client over the server, every call structured-cloned |
|
|
265
295
|
| `readme.test.ts` | the examples above |
|
package/dist/cjs/index.cjs
CHANGED
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
Object.defineProperty(exports, Symbol.toStringTag, { value: "Module" });
|
|
2
|
-
|
|
2
|
+
// src/client.ts
|
|
3
3
|
const toBytes$1 = (v) => v instanceof Uint8Array ? v : ArrayBuffer.isView(v) ? new Uint8Array(v.buffer, v.byteOffset, v.byteLength) : new Uint8Array(v);
|
|
4
4
|
const newStreamId$1 = () => crypto.randomUUID().replace(/-/g, "");
|
|
5
5
|
/** Cuts any sequence of byte arrays into chunks of exactly `size` bytes (the last may be shorter). */
|
|
@@ -56,8 +56,9 @@ var FilesRpcClient = class {
|
|
|
56
56
|
streamId: newStreamId$1()
|
|
57
57
|
});
|
|
58
58
|
const pending = /* @__PURE__ */ new Map();
|
|
59
|
+
let fail;
|
|
59
60
|
const aborted = signal ? new Promise((_, reject) => {
|
|
60
|
-
|
|
61
|
+
fail = () => reject(signal.reason ?? /* @__PURE__ */ new Error("aborted"));
|
|
61
62
|
if (signal.aborted) fail();
|
|
62
63
|
else signal.addEventListener("abort", fail, { once: true });
|
|
63
64
|
}) : void 0;
|
|
@@ -82,6 +83,7 @@ var FilesRpcClient = class {
|
|
|
82
83
|
signal?.throwIfAborted();
|
|
83
84
|
}
|
|
84
85
|
} finally {
|
|
86
|
+
if (fail) signal?.removeEventListener("abort", fail);
|
|
85
87
|
await this.#rpc.closeRead(streamId).catch(() => {});
|
|
86
88
|
}
|
|
87
89
|
}
|
|
@@ -90,7 +92,20 @@ var FilesRpcClient = class {
|
|
|
90
92
|
* then `closeWrite` with their count; any failure aborts the write.
|
|
91
93
|
*/
|
|
92
94
|
async write(path, content) {
|
|
93
|
-
|
|
95
|
+
return this.#send(path, content, false);
|
|
96
|
+
}
|
|
97
|
+
/**
|
|
98
|
+
* Adds `content` to the end of `path` (creating it if it does not exist), streamed as `write`
|
|
99
|
+
* streams. The server appends through the served FilesApi's own `append`, or emulates it.
|
|
100
|
+
*/
|
|
101
|
+
async append(path, content) {
|
|
102
|
+
return this.#send(path, content, true);
|
|
103
|
+
}
|
|
104
|
+
async #send(path, content, append) {
|
|
105
|
+
const { streamId } = await this.#rpc.openWrite(path, {
|
|
106
|
+
streamId: newStreamId$1(),
|
|
107
|
+
...append ? { append: true } : {}
|
|
108
|
+
});
|
|
94
109
|
const inFlight = /* @__PURE__ */ new Set();
|
|
95
110
|
let failure;
|
|
96
111
|
let count = 0;
|
|
@@ -115,7 +130,7 @@ var FilesRpcClient = class {
|
|
|
115
130
|
}
|
|
116
131
|
}
|
|
117
132
|
async mkdir(path) {
|
|
118
|
-
await this.#rpc.mkdir(path);
|
|
133
|
+
await this.#rpc.mkdir(path, { requestId: newStreamId$1() });
|
|
119
134
|
}
|
|
120
135
|
/** Pages of `pageSize`, each asked after the last path of the one before. */
|
|
121
136
|
async *list(path, options = {}) {
|
|
@@ -138,21 +153,20 @@ var FilesRpcClient = class {
|
|
|
138
153
|
return this.#rpc.exists(path);
|
|
139
154
|
}
|
|
140
155
|
remove(path) {
|
|
141
|
-
return this.#rpc.remove(path);
|
|
156
|
+
return this.#rpc.remove(path, { requestId: newStreamId$1() });
|
|
142
157
|
}
|
|
143
158
|
move(source, target) {
|
|
144
|
-
return this.#rpc.move(source, target);
|
|
159
|
+
return this.#rpc.move(source, target, { requestId: newStreamId$1() });
|
|
145
160
|
}
|
|
146
161
|
copy(source, target) {
|
|
147
|
-
return this.#rpc.copy(source, target);
|
|
162
|
+
return this.#rpc.copy(source, target, { requestId: newStreamId$1() });
|
|
148
163
|
}
|
|
149
164
|
};
|
|
150
165
|
/** A FilesApi over a FilesRpc. */
|
|
151
166
|
function newFilesRpcClient(options) {
|
|
152
167
|
return new FilesRpcClient(options);
|
|
153
168
|
}
|
|
154
|
-
|
|
155
|
-
//#region src/errors.ts
|
|
169
|
+
// src/errors.ts
|
|
156
170
|
/**
|
|
157
171
|
* An error of the protocol itself. The code also leads the message (`UNKNOWN_STREAM: …`), since
|
|
158
172
|
* a structured clone keeps an error's name and message but not its other properties:
|
|
@@ -169,11 +183,19 @@ var FilesRpcError = class extends Error {
|
|
|
169
183
|
/** The FilesRpcError code of an error, also after it crossed a transport; undefined otherwise. */
|
|
170
184
|
function errorCode(e) {
|
|
171
185
|
const code = e?.code;
|
|
172
|
-
if (typeof code === "string") return code;
|
|
173
|
-
return
|
|
186
|
+
if (typeof code === "string" && CODES.has(code)) return code;
|
|
187
|
+
return LEADING_CODE.exec(String(e?.message ?? ""))?.[1];
|
|
174
188
|
}
|
|
175
|
-
|
|
176
|
-
|
|
189
|
+
const CODES = /* @__PURE__ */ new Set([
|
|
190
|
+
"UNKNOWN_STREAM",
|
|
191
|
+
"CHUNK_RELEASED",
|
|
192
|
+
"TOO_MANY_STREAMS",
|
|
193
|
+
"ABORTED",
|
|
194
|
+
"INVALID"
|
|
195
|
+
]);
|
|
196
|
+
/** A message led by a code, as FilesRpcError writes it (`CODE: …`). */
|
|
197
|
+
const LEADING_CODE = new RegExp(`^(${[...CODES].join("|")}):`);
|
|
198
|
+
// src/keyed.ts
|
|
177
199
|
/**
|
|
178
200
|
* Values kept by key, and waiters for keys that have no value yet: what both ends of a stream
|
|
179
201
|
* use to keep chunks by chunkId and wait for the ones still on their way.
|
|
@@ -236,8 +258,23 @@ var Keyed = class {
|
|
|
236
258
|
this.#waiters.clear();
|
|
237
259
|
}
|
|
238
260
|
};
|
|
239
|
-
|
|
240
|
-
|
|
261
|
+
// src/server.ts
|
|
262
|
+
/**
|
|
263
|
+
* Whether a write's outcome is the FilesApi's alone: every chunk was handed to it (an emulated
|
|
264
|
+
* append, until its write-back starts, can still be cancelled).
|
|
265
|
+
*/
|
|
266
|
+
const committed = (s) => s.total !== void 0 && s.next >= s.total && (!s.emulated || s.committing === true);
|
|
267
|
+
/** `path` in normal form: one leading slash, no empty, "." or ".." segments, no trailing slash. */
|
|
268
|
+
function normalPath(path) {
|
|
269
|
+
const out = [];
|
|
270
|
+
for (const seg of String(path).split("/")) {
|
|
271
|
+
if (!seg || seg === ".") continue;
|
|
272
|
+
if (seg === "..") out.pop();
|
|
273
|
+
else out.push(seg);
|
|
274
|
+
}
|
|
275
|
+
return `/${out.join("/")}`;
|
|
276
|
+
}
|
|
277
|
+
const messageOf = (e) => String(e?.message ?? e);
|
|
241
278
|
const EMPTY = /* @__PURE__ */ new Uint8Array();
|
|
242
279
|
const STREAM_ID = /^[A-Za-z0-9_-]{16,128}$/;
|
|
243
280
|
function newStreamId() {
|
|
@@ -300,6 +337,40 @@ function newFilesRpcServer(options) {
|
|
|
300
337
|
* open it anew, and a closeWrite delivered again answers as the first one did.
|
|
301
338
|
*/
|
|
302
339
|
const ended = /* @__PURE__ */ new Map();
|
|
340
|
+
/** Answers of metadata calls, by method and requestId, oldest first. */
|
|
341
|
+
const requests = /* @__PURE__ */ new Map();
|
|
342
|
+
const maxRequests = Math.max(1, options.maxRequests ?? 1024);
|
|
343
|
+
/** Runs `call` once per requestId (within idleTimeout): a call delivered again gets its answer. */
|
|
344
|
+
const once = (method, opts, call) => {
|
|
345
|
+
const id = opts?.requestId;
|
|
346
|
+
if (id === void 0) return call();
|
|
347
|
+
if (typeof id !== "string" || !STREAM_ID.test(id)) return Promise.reject(new FilesRpcError("INVALID", "a requestId is 16 to 128 characters of [A-Za-z0-9_-]"));
|
|
348
|
+
const key = `${method} ${id}`;
|
|
349
|
+
const kept = requests.get(key);
|
|
350
|
+
if (kept) return kept.answer;
|
|
351
|
+
const answer = call();
|
|
352
|
+
answer.catch(() => {});
|
|
353
|
+
const entry = {
|
|
354
|
+
at: now(),
|
|
355
|
+
settled: false,
|
|
356
|
+
answer
|
|
357
|
+
};
|
|
358
|
+
requests.set(key, entry);
|
|
359
|
+
const settle = () => {
|
|
360
|
+
entry.at = now();
|
|
361
|
+
entry.settled = true;
|
|
362
|
+
};
|
|
363
|
+
answer.then(settle, settle);
|
|
364
|
+
while (requests.size > maxRequests) {
|
|
365
|
+
let victim;
|
|
366
|
+
for (const [k, r] of requests) if (r.settled) {
|
|
367
|
+
victim = k;
|
|
368
|
+
break;
|
|
369
|
+
}
|
|
370
|
+
requests.delete(victim ?? requests.keys().next().value);
|
|
371
|
+
}
|
|
372
|
+
return answer;
|
|
373
|
+
};
|
|
303
374
|
const finish = (id, failure) => {
|
|
304
375
|
streams.delete(id);
|
|
305
376
|
ended.set(id, {
|
|
@@ -442,6 +513,52 @@ function newFilesRpcServer(options) {
|
|
|
442
513
|
s.wake = void 0;
|
|
443
514
|
}
|
|
444
515
|
}
|
|
516
|
+
/** Emulated appends in progress, by normal path: one at a time per file. */
|
|
517
|
+
const appending = /* @__PURE__ */ new Map();
|
|
518
|
+
/**
|
|
519
|
+
* Appends `content` to `path`: through the FilesApi's own `append`, or with read, write and
|
|
520
|
+
* remove only. The emulation never reads and writes one path at once (a FilesApi may empty a
|
|
521
|
+
* file when a write starts):
|
|
522
|
+
* 1. the file's content and the chunks go to a temporary sibling;
|
|
523
|
+
* 2. that sibling is written back over the file, then removed.
|
|
524
|
+
* A failure or an abort in step 1 leaves the file as it was and removes the sibling. A failure
|
|
525
|
+
* in step 2 keeps the sibling (the only complete copy) and names it in the error. Appends to
|
|
526
|
+
* one file through this server run one after the other; other writers are not excluded.
|
|
527
|
+
*/
|
|
528
|
+
const append = async (path, content, id, s) => {
|
|
529
|
+
const own = fs.append;
|
|
530
|
+
if (typeof own === "function") return own.call(fs, path, content);
|
|
531
|
+
s.emulated = true;
|
|
532
|
+
const file = normalPath(path);
|
|
533
|
+
const run = (appending.get(file) ?? Promise.resolve()).then(async () => {
|
|
534
|
+
if ((await fs.stats(file))?.kind === "directory") throw new FilesRpcError("INVALID", `cannot append to a directory: ${file}`);
|
|
535
|
+
const slash = file.lastIndexOf("/");
|
|
536
|
+
const temp = `${file.slice(0, slash + 1)}.${file.slice(slash + 1, slash + 65)}.append-${id.slice(0, 16)}`;
|
|
537
|
+
try {
|
|
538
|
+
await fs.write(temp, (async function* () {
|
|
539
|
+
yield* fs.read(file);
|
|
540
|
+
yield* content;
|
|
541
|
+
})());
|
|
542
|
+
if (s.failure !== void 0) throw s.failure;
|
|
543
|
+
} catch (e) {
|
|
544
|
+
await fs.remove(temp).catch(() => {});
|
|
545
|
+
throw e;
|
|
546
|
+
}
|
|
547
|
+
s.committing = true;
|
|
548
|
+
try {
|
|
549
|
+
await fs.write(file, fs.read(temp));
|
|
550
|
+
} catch (e) {
|
|
551
|
+
throw Object.assign(/* @__PURE__ */ new Error(`${messageOf(e)} (the append failed while writing ${file} back: it may be incomplete; its content and the appended chunks are kept in ${temp})`), { cause: e });
|
|
552
|
+
}
|
|
553
|
+
await fs.remove(temp).catch(() => {});
|
|
554
|
+
});
|
|
555
|
+
const settled = run.catch(() => {});
|
|
556
|
+
appending.set(file, settled);
|
|
557
|
+
settled.then(() => {
|
|
558
|
+
if (appending.get(file) === settled) appending.delete(file);
|
|
559
|
+
});
|
|
560
|
+
return run;
|
|
561
|
+
};
|
|
445
562
|
const failWrite = (s, failure) => {
|
|
446
563
|
s.failure ??= failure;
|
|
447
564
|
for (const r of s.room.splice(0)) r.reject(s.failure);
|
|
@@ -454,17 +571,17 @@ function newFilesRpcServer(options) {
|
|
|
454
571
|
async exists(path) {
|
|
455
572
|
return fs.exists(path);
|
|
456
573
|
},
|
|
457
|
-
async mkdir(path) {
|
|
458
|
-
await fs.mkdir(path);
|
|
574
|
+
async mkdir(path, opts) {
|
|
575
|
+
await once("mkdir", opts, () => fs.mkdir(path));
|
|
459
576
|
},
|
|
460
|
-
async remove(path) {
|
|
461
|
-
return fs.remove(path);
|
|
577
|
+
async remove(path, opts) {
|
|
578
|
+
return once("remove", opts, () => fs.remove(path));
|
|
462
579
|
},
|
|
463
|
-
async move(source, target) {
|
|
464
|
-
return fs.move(source, target);
|
|
580
|
+
async move(source, target, opts) {
|
|
581
|
+
return once("move", opts, () => fs.move(source, target));
|
|
465
582
|
},
|
|
466
|
-
async copy(source, target) {
|
|
467
|
-
return fs.copy(source, target);
|
|
583
|
+
async copy(source, target, opts) {
|
|
584
|
+
return once("copy", opts, () => fs.copy(source, target));
|
|
468
585
|
},
|
|
469
586
|
async list(path, opts = {}) {
|
|
470
587
|
const limit = Math.min(Math.max(1, Math.floor(Number(opts.limit) || maxPageSize)), maxPageSize);
|
|
@@ -557,7 +674,7 @@ function newFilesRpcServer(options) {
|
|
|
557
674
|
room: [],
|
|
558
675
|
done: Promise.resolve()
|
|
559
676
|
};
|
|
560
|
-
s.done = fs.write(path, ordered(s)).then(() => {
|
|
677
|
+
s.done = (opts.append ? append(path, ordered(s), id, s) : fs.write(path, ordered(s))).then(() => {
|
|
561
678
|
if (s.failure !== void 0) throw s.failure;
|
|
562
679
|
}, (e) => {
|
|
563
680
|
failWrite(s, e);
|
|
@@ -621,7 +738,7 @@ function newFilesRpcServer(options) {
|
|
|
621
738
|
const id = String(streamId);
|
|
622
739
|
const s = streams.get(id);
|
|
623
740
|
if (s?.kind !== "write") return;
|
|
624
|
-
if (s
|
|
741
|
+
if (committed(s)) return;
|
|
625
742
|
const error = new FilesRpcError("ABORTED", reason ? String(reason) : "the write was aborted");
|
|
626
743
|
finish(id, error);
|
|
627
744
|
failWrite(s, error);
|
|
@@ -640,6 +757,7 @@ function newFilesRpcServer(options) {
|
|
|
640
757
|
else failWrite(s, error);
|
|
641
758
|
}
|
|
642
759
|
for (const [id, e] of [...ended]) if (t - e.at > idleTimeout) ended.delete(id);
|
|
760
|
+
for (const [key, r] of [...requests]) if (r.settled && t - r.at > idleTimeout) requests.delete(key);
|
|
643
761
|
return dropped;
|
|
644
762
|
},
|
|
645
763
|
openStreams() {
|
|
@@ -649,6 +767,10 @@ function newFilesRpcServer(options) {
|
|
|
649
767
|
if (timer) clearInterval(timer);
|
|
650
768
|
const sources = [];
|
|
651
769
|
for (const [id, s] of [...streams]) {
|
|
770
|
+
if (s.kind === "write" && committed(s)) {
|
|
771
|
+
streams.delete(id);
|
|
772
|
+
continue;
|
|
773
|
+
}
|
|
652
774
|
const error = new FilesRpcError("ABORTED", "the server closed");
|
|
653
775
|
finish(id, error);
|
|
654
776
|
if (s.kind === "read") sources.push(closeSource(s));
|
|
@@ -663,7 +785,6 @@ function newFilesRpcServer(options) {
|
|
|
663
785
|
timer?.unref?.();
|
|
664
786
|
return server;
|
|
665
787
|
}
|
|
666
|
-
//#endregion
|
|
667
788
|
exports.FilesRpcClient = FilesRpcClient;
|
|
668
789
|
exports.FilesRpcError = FilesRpcError;
|
|
669
790
|
exports.errorCode = errorCode;
|
package/dist/client.d.ts
CHANGED
|
@@ -26,6 +26,11 @@ export declare class FilesRpcClient implements FilesApi {
|
|
|
26
26
|
* then `closeWrite` with their count; any failure aborts the write.
|
|
27
27
|
*/
|
|
28
28
|
write(path: string, content: Iterable<Uint8Array> | AsyncIterable<Uint8Array>): Promise<void>;
|
|
29
|
+
/**
|
|
30
|
+
* Adds `content` to the end of `path` (creating it if it does not exist), streamed as `write`
|
|
31
|
+
* streams. The server appends through the served FilesApi's own `append`, or emulates it.
|
|
32
|
+
*/
|
|
33
|
+
append(path: string, content: Iterable<Uint8Array> | AsyncIterable<Uint8Array>): Promise<void>;
|
|
29
34
|
mkdir(path: string): Promise<void>;
|
|
30
35
|
/** Pages of `pageSize`, each asked after the last path of the one before. */
|
|
31
36
|
list(path: string, options?: ListOptions): AsyncIterable<FileInfo>;
|
package/dist/client.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"client.d.ts","sourceRoot":"","sources":["../src/client.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EACV,QAAQ,EACR,SAAS,EACT,QAAQ,EACR,WAAW,EACX,WAAW,EACZ,MAAM,2BAA2B,CAAC;AACnC,OAAO,KAAK,EAAE,QAAQ,EAAc,MAAM,YAAY,CAAC;AAEvD,MAAM,WAAW,qBAAqB;IACpC,wFAAwF;IACxF,GAAG,EAAE,QAAQ,CAAC;IACd,8FAA8F;IAC9F,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,4FAA4F;IAC5F,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,qEAAqE;IACrE,UAAU,CAAC,EAAE,MAAM,CAAC;IACpB,sCAAsC;IACtC,QAAQ,CAAC,EAAE,MAAM,CAAC;CACnB;AAyCD,0FAA0F;AAC1F,qBAAa,cAAe,YAAW,QAAQ;;IAO7C,YAAY,OAAO,EAAE,qBAAqB,EAMzC;IAED;;;OAGG;IACI,IAAI,CAAC,IAAI,EAAE,MAAM,EAAE,OAAO,GAAE,WAAgB,GAAG,aAAa,CAAC,UAAU,CAAC,
|
|
1
|
+
{"version":3,"file":"client.d.ts","sourceRoot":"","sources":["../src/client.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EACV,QAAQ,EACR,SAAS,EACT,QAAQ,EACR,WAAW,EACX,WAAW,EACZ,MAAM,2BAA2B,CAAC;AACnC,OAAO,KAAK,EAAE,QAAQ,EAAc,MAAM,YAAY,CAAC;AAEvD,MAAM,WAAW,qBAAqB;IACpC,wFAAwF;IACxF,GAAG,EAAE,QAAQ,CAAC;IACd,8FAA8F;IAC9F,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,4FAA4F;IAC5F,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,qEAAqE;IACrE,UAAU,CAAC,EAAE,MAAM,CAAC;IACpB,sCAAsC;IACtC,QAAQ,CAAC,EAAE,MAAM,CAAC;CACnB;AAyCD,0FAA0F;AAC1F,qBAAa,cAAe,YAAW,QAAQ;;IAO7C,YAAY,OAAO,EAAE,qBAAqB,EAMzC;IAED;;;OAGG;IACI,IAAI,CAAC,IAAI,EAAE,MAAM,EAAE,OAAO,GAAE,WAAgB,GAAG,aAAa,CAAC,UAAU,CAAC,CA4C9E;IAED;;;OAGG;IACG,KAAK,CACT,IAAI,EAAE,MAAM,EACZ,OAAO,EAAE,QAAQ,CAAC,UAAU,CAAC,GAAG,aAAa,CAAC,UAAU,CAAC,GACxD,OAAO,CAAC,IAAI,CAAC,CAEf;IAED;;;OAGG;IACG,MAAM,CACV,IAAI,EAAE,MAAM,EACZ,OAAO,EAAE,QAAQ,CAAC,UAAU,CAAC,GAAG,aAAa,CAAC,UAAU,CAAC,GACxD,OAAO,CAAC,IAAI,CAAC,CAEf;IAsCK,KAAK,CAAC,IAAI,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC,CAEvC;IAED,6EAA6E;IACtE,IAAI,CAAC,IAAI,EAAE,MAAM,EAAE,OAAO,GAAE,WAAgB,GAAG,aAAa,CAAC,QAAQ,CAAC,CAY5E;IAED,KAAK,CAAC,IAAI,EAAE,MAAM,GAAG,OAAO,CAAC,SAAS,GAAG,SAAS,CAAC,CAElD;IACD,MAAM,CAAC,IAAI,EAAE,MAAM,GAAG,OAAO,CAAC,OAAO,CAAC,CAErC;IACD,MAAM,CAAC,IAAI,EAAE,MAAM,GAAG,OAAO,CAAC,OAAO,CAAC,CAErC;IACD,IAAI,CAAC,MAAM,EAAE,MAAM,EAAE,MAAM,EAAE,MAAM,GAAG,OAAO,CAAC,OAAO,CAAC,CAErD;IACD,IAAI,CAAC,MAAM,EAAE,MAAM,EAAE,MAAM,EAAE,MAAM,GAAG,OAAO,CAAC,OAAO,CAAC,CAErD;CACF;AAED,kCAAkC;AAClC,wBAAgB,iBAAiB,CAAC,OAAO,EAAE,qBAAqB,GAAG,cAAc,CAEhF"}
|
package/dist/errors.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"errors.d.ts","sourceRoot":"","sources":["../src/errors.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,iBAAiB,EAAE,MAAM,YAAY,CAAC;AAEpD;;;;GAIG;AACH,qBAAa,aAAc,SAAQ,KAAK;IACtC,QAAQ,CAAC,IAAI,EAAE,iBAAiB,CAAC;IACjC,YAAY,IAAI,EAAE,iBAAiB,EAAE,OAAO,EAAE,MAAM,EAInD;CACF;AAED,kGAAkG;AAClG,wBAAgB,SAAS,CAAC,CAAC,EAAE,OAAO,GAAG,iBAAiB,GAAG,SAAS,
|
|
1
|
+
{"version":3,"file":"errors.d.ts","sourceRoot":"","sources":["../src/errors.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,iBAAiB,EAAE,MAAM,YAAY,CAAC;AAEpD;;;;GAIG;AACH,qBAAa,aAAc,SAAQ,KAAK;IACtC,QAAQ,CAAC,IAAI,EAAE,iBAAiB,CAAC;IACjC,YAAY,IAAI,EAAE,iBAAiB,EAAE,OAAO,EAAE,MAAM,EAInD;CACF;AAED,kGAAkG;AAClG,wBAAgB,SAAS,CAAC,CAAC,EAAE,OAAO,GAAG,iBAAiB,GAAG,SAAS,CAMnE"}
|