@statewalker/webrun-files-rpc 0.0.0-stage → 0.1.1

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2022-2026 statewalker
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,3 +1,295 @@
-# Temporary Holding Version
+# @statewalker/webrun-files-rpc
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+## What it is
+
+A `FilesApi` (from `@statewalker/webrun-files`) over any RPC, with streamed reads and writes.
+
+- **Server** (`newFilesRpcServer({ fs })`): serves any `FilesApi` as plain method calls.
+- **Client** (`newFilesRpcClient({ rpc })`): a `FilesApi` whose every call goes through those
+  methods: a Durable Object or Workers RPC stub, comlink, `@statewalker/webrun-rpc`, or the server
+  object itself. It also appends (`append(path, content)`), which `FilesApi` does not.
+
+Content moves as **chunks keyed by `(streamId, chunkId)`**. Both sides keep chunks by their key
+and wait for the ones still on their way:
+- the server answers `readChunk(streamId, k)` once chunk `k` exists;
+- it writes chunk `k` once chunks `0..k-1` are written, whatever order they arrived in.
+
+So the client can keep several chunks in flight, the transport may deliver calls out of order or
+twice, and every byte still lands in order.
+
+## Why it exists
+
+A `FilesApi` streams: `read()` yields chunks and `write()` takes them. An RPC usually doesn't:
+- Durable Object RPC carries a `ReadableStream` only on workerd; other Workers-compatible runtimes
+  refuse it ("RPC streams cannot cross isolate boundaries").
+- JSON and message-based transports have no streams at all.
+
+`@statewalker/webrun-files-http` solves this for HTTP, where request and response bodies stream.
+This package solves it for everything else: a stream becomes a sequence of ordinary calls, each
+carrying one chunk, and any transport that can make a call can carry a file of any size.
+
+Keys make the calls safe on real transports:
+- **Pipelining:** several `readChunk` or `writeChunk` calls may be in flight; the answers are put
+  back in order by `chunkId`.
+- **Reordering:** a chunk that arrives early is kept until the ones before it are written.
+- **Retries:**
+  - a chunk written twice is written once;
+  - a chunk read twice is answered twice;
+  - an open with the same `streamId` opens one stream;
+  - a `closeWrite` delivered again answers as the first did.
+  - a metadata call (`mkdir`, `remove`, `move`, `copy`) with a `requestId` runs once: delivered
+    again, concurrently or later, it gets the first answer (or failure). The server keeps answers
+    for `idleTimeout`, at most `maxRequests` of them. The client sends a fresh `requestId` with
+    every such call.
+  - Without a `requestId`, a metadata call delivered twice runs twice: the end state is the same
+    when nothing else writes in between, but the second may report `false` though the first did
+    it, and a late `copy` retry copies again.
+
+## How to use
+
+```bash
+pnpm add @statewalker/webrun-files-rpc @statewalker/webrun-files
+```
+
+Inside this repository, depend on it with `"@statewalker/webrun-files-rpc": "workspace:^"`.
+
+One entry point: ESM (`dist/esm/index.js`), CommonJS (`dist/cjs/index.cjs`), types
+(`dist/index.d.ts`); sources in `src/`. The bundle has no imports: `@statewalker/webrun-files` is
+used for its types only.
+
+Exports:
+- `newFilesRpcServer`, `newFilesRpcClient`, `FilesRpcClient`;
+- `FilesRpcError` and `errorCode`;
+- types `FilesRpc`, `FilesRpcServer`, `FilesRpcServerOptions`, `FilesRpcClientOptions`,
+  `AppendableFilesApi`, `ReadResult`, `ListPage`, `ListPageOptions`, `OpenOptions`,
+  `OpenReadOptions`, `OpenWriteOptions`, `RequestOptions`, `FilesRpcErrorCode`.
+
+### `newFilesRpcServer(options)`
+
+| Option | Default | |
+|---|---|---|
+| `fs` | (required) | The `FilesApi` served. |
+| `chunkSize` | 256 KiB | Largest chunk a read answers. |
+| `readAhead` | 4 | Chunks produced ahead, counting from the highest one asked. |
+| `retain` | 16 | A delivered chunk is dropped once the reader asked for one `retain` chunks past it (until then it can be asked again). |
+| `window` | 64 | A read request may name a chunk at most `window` past the lowest chunk not yet delivered (`INVALID` beyond): what one stream makes the server pull and hold stays within `window + readAhead` chunks. |
+| `writeWindow` | 16 | Chunks of a write kept ahead of the next one to write; a chunk beyond it waits (backpressure), one beyond two windows is refused. |
+| `maxChunkSize` | 4 MiB | Largest chunk a write takes. |
+| `idleTimeout` | 2 min | A stream not used for this long is dropped (a read closed, a write aborted). |
+| `maxStreams` | 256 | Open streams at most (`TOO_MANY_STREAMS` beyond). |
+| `maxPageSize` | 256 | Largest listing page. |
+| `maxRequests` | 1024 | Answers of metadata calls kept by `requestId` (oldest dropped beyond). |
+| `sweepInterval` | none | Sweep idle streams every this many ms. Without it, opening a stream sweeps; a server that stops receiving opens needs this (or calls to `sweep()`). |
+| `now` | `Date.now` | The clock (tests). |
+
+The server is a `FilesRpc` plus:
+- `sweep()`: drops idle streams and returns how many (opening a stream also sweeps);
+- `openStreams()`: how many streams are open;
+- `close()`: ends every stream and stops the sweep timer. It doesn't wait for the FilesApi's
+  writes to settle, and waits at most a second for sources to end.
+
+### `newFilesRpcClient(options)`
+
+| Option | Default | |
+|---|---|---|
+| `rpc` | (required) | The server's methods, through any transport. |
+| `chunkSize` | 256 KiB | Bytes per chunk written (at most the server's `maxChunkSize`). |
+| `readAhead` | 4 | `readChunk` calls in flight while reading (below the server's `window`). |
+| `writeAhead` | 4 | `writeChunk` calls not yet answered while writing. |
+| `pageSize` | 256 | Listing page size. |
+
+Besides the `FilesApi` methods, the client has `append(path, content)`: it adds `content` to the
+end of `path` (creating it if it does not exist), streamed exactly as `write` streams.
+
+### The protocol (`FilesRpc`)
+
+Every argument and result is a plain value: strings, numbers, booleans, plain objects and
+`Uint8Array`.
+
+| Call | Answer |
+|---|---|
+| `openRead(path, {start?, length?, streamId?})` | `{streamId}`; a file that does not exist reads as nothing |
+| `readChunk(streamId, chunkId)` | `{data, done}` once chunk `chunkId` exists; past the end `{data: empty, done: true}` |
+| `closeRead(streamId)` | ends the stream (unknown or ended: nothing happens) |
+| `openWrite(path, {streamId?, append?})` | `{streamId}`; the `FilesApi`'s `write` (or, with `append`, the append) starts consuming chunks in order |
+| `writeChunk(streamId, chunkId, data)` | resolves once the chunk is kept (within `writeWindow` of the next to write: at once) |
+| `closeWrite(streamId, chunks)` | resolves once all `chunks` chunks are written |
+| `abortWrite(streamId, reason?)` | cancels the write (unknown or ended: nothing happens) |
+| `list(path, {recursive?, after?, limit?})` | `{entries, more}`; ask again with `after` = the last path |
+| `stats`, `exists` | as the `FilesApi` |
+| `mkdir(path, {requestId?})`, `remove(path, {requestId?})`, `move(source, target, {requestId?})`, `copy(source, target, {requestId?})` | as the `FilesApi`; with a `requestId`, once (a call delivered again gets the first answer) |
+
+- **Stream ids:** `streamId` may be chosen by the caller: 16 to 128 characters of `[A-Za-z0-9_-]`,
+  random, such as `crypto.randomUUID()` without dashes. Then an open delivered twice opens one
+  stream; the client always does this.
+- **Errors:** protocol errors are `FilesRpcError` with a `code`:
+  - `UNKNOWN_STREAM`, `CHUNK_RELEASED`, `TOO_MANY_STREAMS`, `ABORTED`, `INVALID`.
+  - The code also leads the message (`UNKNOWN_STREAM: …`), because a structured clone keeps an
+    error's message but not its other properties. `errorCode(e)` reads it on either side.
+
+## Examples
+
+### Both ends in one process
+
+```ts
+import { MemFilesApi } from "@statewalker/webrun-files-mem";
+import { newFilesRpcClient, newFilesRpcServer } from "@statewalker/webrun-files-rpc";
+
+const server = newFilesRpcServer({ fs: new MemFilesApi() });
+const files = newFilesRpcClient({ rpc: server }); // any transport carrying these calls works
+
+await files.write("/notes/today.md", [new TextEncoder().encode("# Today\n")]);
+for await (const chunk of files.read("/notes/today.md")) console.log(new TextDecoder().decode(chunk));
+```
+
+### Served by a Durable Object, called through its stub
+
+```ts
+import { DurableObject } from "cloudflare:workers";
+import { newFilesRpcClient, newFilesRpcServer, type FilesRpcServer } from "@statewalker/webrun-files-rpc";
+
+export class Files extends DurableObject {
+  #files: FilesRpcServer = newFilesRpcServer({ fs: makeFilesApi(this.ctx) }); // any FilesApi
+  openRead(path: string, o?: object) { return this.#files.openRead(path, o); }
+  readChunk(id: string, k: number) { return this.#files.readChunk(id, k); }
+  closeRead(id: string) { return this.#files.closeRead(id); }
+  openWrite(path: string, o?: object) { return this.#files.openWrite(path, o); }
+  writeChunk(id: string, k: number, d: Uint8Array) { return this.#files.writeChunk(id, k, d); }
+  closeWrite(id: string, n: number) { return this.#files.closeWrite(id, n); }
+  abortWrite(id: string, r?: string) { return this.#files.abortWrite(id, r); }
+  list(path: string, o?: object) { return this.#files.list(path, o); }
+  stats(path: string) { return this.#files.stats(path); }
+  exists(path: string) { return this.#files.exists(path); }
+  mkdir(path: string) { return this.#files.mkdir(path); }
+  remove(path: string) { return this.#files.remove(path); }
+  move(a: string, b: string) { return this.#files.move(a, b); }
+  copy(a: string, b: string) { return this.#files.copy(a, b); }
+}
+
+// In a Worker: the stub has the same methods, so it is the client's `rpc`.
+const files = newFilesRpcClient({ rpc: env.FILES.get(env.FILES.idFromName("main")) as never });
+```
+
+A stream lives in the server's memory: keep one stream's calls on one Durable Object (they are,
+through one stub).
+
+### The protocol by hand
+
+```ts
+const server = newFilesRpcServer({ fs: new MemFilesApi(), chunkSize: 4 });
+const streamId = crypto.randomUUID().replace(/-/g, "");
+await server.openWrite("/a.txt", { streamId });
+await Promise.all([
+  server.writeChunk(streamId, 1, new TextEncoder().encode("world")), // out of order: kept
+  server.writeChunk(streamId, 0, new TextEncoder().encode("hello ")),
+]);
+await server.closeWrite(streamId, 2); // resolves once both are written, in order
+
+const { streamId: r } = await server.openRead("/a.txt");
+for (let chunkId = 0; ; chunkId++) {
+  const { data, done } = await server.readChunk(r, chunkId); // "hell", "o wo", "rld"
+  if (done) break;
+}
+await server.closeRead(r);
+```
+
+`tests/readme.test.ts` runs the first and the last example as written.
+
+## Internals
+
+- **Reads:** `openRead` opens the `FilesApi`'s `read()` iterator; nothing is pulled yet.
+  - `readChunk(k)` raises the highest id asked. A pump pulls the source until `readAhead` chunks
+    from it exist, cutting the bytes into chunks of exactly `chunkSize`.
+  - A chunk is cut only once more bytes follow it or the source has ended, so the last chunk
+    carries `done: true` and no empty chunk is needed.
+  - Chunks wait in a keyed store (`src/keyed.ts`); a request for a chunk not produced yet waits
+    there.
+  - If the source fails, the bytes before the failure are still served, and later chunks fail
+    with its error.
+- **Release:** a delivered chunk is dropped once the reader asked for one `retain` chunks past it.
+  - A chunk below that line that nobody has received yet (asked out of order, or still to be
+    produced) is kept until its first delivery. So a client asking further ahead than the server
+    retains still gets every chunk.
+  - A chunk delivered and dropped is `CHUNK_RELEASED`.
+- **Writes:** `openWrite` starts the `FilesApi`'s `write()` at once, on a generator that yields
+  the stream's chunks in key order and waits for the next one.
+  - `writeChunk` keeps a copy of its chunk; a view of a larger buffer is cut out.
+  - A chunk at or beyond `next + writeWindow` waits until the window moves; one at or beyond
+    `next + 2 × writeWindow` is refused (`INVALID`). So a writer makes the server hold at most two
+    windows of chunks.
+  - `closeWrite(n)` tells the generator where to stop and waits for `write()` to finish. If more
+    than `n` chunks arrived, the write fails with `INVALID`.
+  - `abortWrite`, expiry and `close()` make the generator throw, so `write()` fails. What then
+    remains of the file is up to the `FilesApi`: `MemFilesApi` writes only at the end, so the file
+    keeps what it had.
+  - `write()` starts at `openWrite`, not at the first chunk. A FilesApi that truncates when a
+    write starts has emptied the file already if the write is then aborted.
+- **Appends:** `openWrite(path, { append: true })` streams like a write, and the chunks go to the
+  end of the file.
+  - A served FilesApi with an `append(path, content)` method (`AppendableFilesApi`) appends
+    through it, with whatever atomicity it gives.
+  - Any other FilesApi is appended to with `read`, `write` and `remove` only, in two steps:
+    1. the file's content and the chunks are written to a temporary sibling,
+       `.<name>.append-<id>` (the name cut to 64 characters, the stream id to 16);
+    2. the sibling is written back over the file, then removed.
+  - It never reads and writes one path at once, so a FilesApi that empties a file when a write
+    starts is safe. Only a directory is refused (`INVALID`); `stats` saying the file is missing is
+    not trusted, since reading a missing file yields nothing and the file is created all the same.
+  - A failure or an abort during step 1 leaves the file as it was and removes the sibling. An
+    abort counts until step 2 starts, even after every chunk was sent. A failure during step 2
+    may leave the file incomplete, so the sibling, the only complete copy, is kept and its path
+    is in the error.
+  - Appends to one file (by its normal path) through one server run one after the other. Other
+    writers are not excluded. The emulation copies the file twice and shows the sibling in
+    listings while it runs.
+  - `close()` and expiry leave a write whose outcome is the FilesApi's alone (every chunk handed
+    over, or an emulated append writing back), so a `closeWrite` waiting on it gets the real
+    outcome.
+- **Ended streams:** ids that ended are remembered for `idleTimeout`. Within that time, an open
+  delivered after its stream ended does not reopen it, and a `closeWrite` delivered again gets the
+  first answer.
+- **Idle and busy:** a stream is idle when nothing happened on it for `idleTimeout`: no call
+  started or ended, and (writes) the FilesApi took no chunk.
+  - A `readChunk` waiting on a slow source keeps its stream alive, and so does a write whose next
+    chunk is there for the FilesApi to take.
+  - A write waiting only for the holder (`writeChunk` waiting for room, `closeWrite` waiting for a
+    missing chunk) is swept once idle; its waiting calls fail with `ABORTED`.
+  - A write whose chunks were all handed to the FilesApi is never swept. An `abortWrite` arriving
+    then is too late and does nothing (the commit stands).
+- **Failures:** a write the FilesApi fails keeps its stream until it is closed, aborted or swept,
+  so the next `writeChunk` or `closeWrite` reports that failure, not `UNKNOWN_STREAM`. A source
+  yielding something that is not bytes fails the stream with `INVALID`.
+- **Bounds:** a read request beyond `window` is refused at once, so asking for chunk
+  1,000,000,000 neither loops nor loads anything.
+- **The client:**
+  - **Read:** it keeps `readAhead` `readChunk` calls in flight and yields the answers in key
+    order. Breaking out of the loop, or aborting through `signal`, closes the stream.
+  - **Write and append:** it cuts any sequence of byte arrays into chunks of exactly `chunkSize`
+    and keeps `writeAhead` unanswered. Any failure (of the source, or of a call) aborts the write.
+  - **List:** it asks page after page with `after`. The server also applies `after` itself, in
+    code-point order, so a FilesApi that ignores it cannot make pages repeat.
+- **Not covered:**
+  - A stream lives in one server's memory; a server restart loses open streams (`UNKNOWN_STREAM`).
+  - Serializing `Uint8Array` for a JSON-only transport (base64, msgpack) is the transport's job.
+  - Per-caller authorization is the `FilesApi`'s or the transport's job: serve a guarded
+    `FilesApi`.
+
+## Tests
+
+`pnpm --filter @statewalker/webrun-files-rpc test` (Vitest; build the workspace first, the
+shared suites are imported from `webrun-files-tests`' `dist/`). The protocol tests use their own
+pseudo-random bytes and the transport simulator in `tests/helpers.ts`; byte collection and text
+encoding come from `webrun-files-tests`.
+
+| File | Covers |
+|---|---|
+| `server-read.test.ts` | numbered chunks with `done`, out-of-order and concurrent requests, ranges, missing files, retries, release, bounded read-ahead, closing, failing sources |
+| `server-write.test.ts` | in-order and out-of-order chunks, duplicates, backpressure, `closeWrite` waiting for a gap or refusing a wrong count, empty files, abort, failing writes, byte types |
+| `server-lifecycle.test.ts` | idle expiry, the stream limit, `close()`, unguessable ids, paged listing, metadata |
+| `protocol.test.ts` | caller-chosen keys (opens and `closeWrite` delivered twice, after the end), what is refused, waiting requests failed by a close, error codes across a structured clone, many streams at once, release never dropping an undelivered chunk, metadata calls delivered twice (with a `requestId`: once, the same answer, failures too, kept for `idleTimeout` and `maxRequests`; without: the same end state) |
+| `robustness.test.ts` | the window (huge chunk ids refused at once, bounded pulls), busy streams not swept, writes too far ahead refused, writes waiting for the holder swept, the sweep timer, a failing write's error reaching the client, a late abort not undoing a commit, a source yielding non-bytes, waiters past the count, aborting a hanging read, `close()` not hanging, listing with a FilesApi that ignores `after` |
+| `client.test.ts` | the `FilesApi` round trip, iterables, ranges, pipelining depth, early stop and abort, failing sources, write depth, listing; delayed, reordered and duplicated calls (five seeds) |
+| `append.test.ts` | appends through the FilesApi's own `append` and through the emulation, also on a FilesApi that writes as it goes: existing and missing files, large files, aborts (also after every chunk was sent) and failing sources leaving the file unchanged and no temporary file, a failed write-back keeping the temporary copy, `close()` during the write-back, two appends at once, `stats` not trusted, odd paths and long names, directories refused; over the transport simulator; an open delivered twice |
+| `conformance.test.ts` | the repository's `FilesApi` suite (`webrun-files-tests`) on the client over the server: directly, and through the transport simulator (cloned, reordered, every call delivered twice at random) |
+| `big-files.test.ts` | the repository's big-file suite (64 MiB) on the client over the server, every call structured-cloned |
+| `readme.test.ts` | the examples above |