@openclaw/fs-safe 0.1.1 → 0.2.0

package/docs/reading.md

@@ -0,0 +1,174 @@
+# Reading
+
+The `Root` handle exposes five read shapes. Pick the narrowest one that gives you what you need — narrower shapes do less work and surface fewer footguns.
+
+```ts
+const result = await fs.read("notes/today.txt");        // { buffer, realPath, stat }
+const text   = await fs.readText("notes/today.txt");    // string
+const bytes  = await fs.readBytes("image.png");         // Buffer
+const json   = await fs.readJson<Config>("config.json"); // T
+const opened = await fs.open("large.log");               // FileHandle for streaming
+```
+
+## What every read does
+
+Regardless of shape, every read goes through the same boundary checks:
+
+1. Resolve the relative path against the canonical real root.
+2. Reject anything that escapes the root (`outside-workspace`).
+3. Reject `..` segments and absolute inputs (unless via `readAbsolute` with an in-root absolute path).
+4. Open with `O_NOFOLLOW` where available. A symlink in the path triggers `symlink` unless the call's `symlinks` policy is `follow-within-root`.
+5. Stat the open fd and compare to the resolved path's identity (`sameFileIdentity`). A swap mid-call triggers `path-mismatch`.
+6. If `hardlinks: "reject"`, refuse files with `nlink > 1` (`hardlink`).
+7. If `maxBytes` is set, refuse reads larger than the cap (`too-large`).
+
+## Read shapes
+
+### `fs.read(rel, options?)`
+
+The full result. Use it when you need both the bytes and the verified `realPath` or `stat`:
+
+```ts
+const { buffer, realPath, stat } = await fs.read("notes/today.txt");
+console.log(`${stat.size} bytes at ${realPath}`);
+```
+
+### `fs.readText(rel, options?)`
+
+`buffer.toString(encoding)`. Defaults to `defaults.encoding ?? "utf8"`. Pass `encoding` per call to override:
+
+```ts
+const utf16 = await fs.readText("doc.txt", { encoding: "utf16le" });
+```
+
+### `fs.readBytes(rel, options?)`
+
+The buffer alone. Useful when you don't care about the realPath or stat:
+
+```ts
+const png = await fs.readBytes("image.png");
+```
+
+### `fs.readJson<T>(rel, options?)`
+
+`readText` + `JSON.parse`. The generic is a *cast*, not a validator — validate the parsed value at your application boundary if it came from a less-trusted source.
+
+```ts
+type Config = { tokens: string[] };
+const config = await fs.readJson<Config>("config.json");
+```
+
+For tighter control over malformed-or-missing JSON, use the standalone helpers in [`@openclaw/fs-safe/json`](json.md): `tryReadJson` (returns `null` on missing/invalid) vs `readJson` (throws).
+
+### `fs.open(rel, options?)`
+
+Returns a `FileHandle` plus the verified `realPath` and `stat`. Use this for streaming or partial reads, and **always close the handle**:
+
+```ts
+const opened = await fs.open("large.log");
+try {
+  const stream = opened.handle.createReadStream();
+  for await (const chunk of stream) {
+    process.stdout.write(chunk);
+  }
+} finally {
+  await opened.handle.close();
+}
+```
+
+## Read options
+
+```ts
+type RootReadOptions = {
+  hardlinks?: "reject" | "allow";   // override defaults.hardlinks
+  maxBytes?: number;                // refuse reads larger than this many bytes
+  nonBlockingRead?: boolean;        // schedule the read off the main loop
+  symlinks?: "reject" | "follow-within-root"; // override defaults.symlinks
+};
+```
+
+`maxBytes` is enforced eagerly: the library reads up to `maxBytes + 1` and throws `too-large` if there is more, so a hostile target cannot silently exhaust memory.
+
+`nonBlockingRead` is a scheduling hint. It does not affect safety — it lets you keep the event loop responsive when reading large files.
+
+## `readAbsolute()` and `reader()`
+
+Some APIs hand you an absolute path that the caller has already produced. Going back to a relative form just to call `read()` is awkward, so the library exposes:
+
+```ts
+fs.readAbsolute(absPath, options?)   // ReadResult, abs path must be inside the root
+fs.reader(options?)              // (path) => Promise<Buffer>
+```
+
+`readAbsolute` accepts absolute paths. Anything outside the root throws `outside-workspace`.
+
+`reader()` returns a closure that takes either a relative or an absolute path and returns a Buffer. Useful for plugging `fs-safe` into framework loader hooks:
+
+```ts
+const load = fs.reader({ maxBytes: 4 * 1024 * 1024 });
+await someLibrary.parseTemplate({ load });
+```
+
+## Inspection vs reading
+
+`fs.exists`, `fs.stat`, and `fs.list` are advisory. They are safe to call to drive UI or decisions, but they do **not** pin the file:
+
+```ts
+if (await fs.exists("notes/today.txt")) {
+  // the file existed when stat() ran — it may not now
+  const text = await fs.readText("notes/today.txt"); // this is the call that pins
+}
+```
+
+A symlink swap between `exists` and `readText` is caught by the read; the boundary is per-call.
+
+## Streaming patterns
+
+### Read into a writable stream
+
+```ts
+import { pipeline } from "node:stream/promises";
+
+const opened = await fs.open("large.log");
+try {
+  await pipeline(opened.handle.createReadStream(), process.stdout);
+} finally {
+  await opened.handle.close();
+}
+```
+
+### Read in chunks
+
+```ts
+const opened = await fs.open("large.bin");
+try {
+  const buf = Buffer.alloc(64 * 1024);
+  let off = 0;
+  while (true) {
+    const { bytesRead } = await opened.handle.read(buf, 0, buf.length, off);
+    if (bytesRead === 0) break;
+    consume(buf.subarray(0, bytesRead));
+    off += bytesRead;
+  }
+} finally {
+  await opened.handle.close();
+}
+```
+
+## Common errors
+
+- **`outside-workspace`** — relative path escaped the root, or `readAbsolute` got an absolute path outside.
+- **`not-found`** — the file is gone.
+- **`not-file`** — you read a directory or a non-regular file (FIFO, socket, …).
+- **`symlink`** — a path component is a symlink and the policy is `reject`.
+- **`path-mismatch`** — opened fd identity did not match the resolved path. Almost always a TOCTOU swap by something else.
+- **`hardlink`** — `hardlinks: "reject"` saw `nlink > 1`.
+- **`too-large`** — read exceeded `maxBytes`.
+
+See [Errors](errors.md) for the full list.
+
+## See also
+
+- [Writing](writing.md) — companion verbs for produce-side I/O.
+- [JSON files](json.md) — standalone strict/lenient JSON helpers.
+- [Secure file reads](secure-file.md) — pinned absolute file reads with permission checks.

package/docs/regular-file.md

@@ -0,0 +1,175 @@
+# Regular file helpers
+
+The advanced `regular-file` helpers provide direct read/append/stat helpers for absolute file paths, with an explicit "regular file or nothing" contract. Useful when you have a trusted absolute path and want a thin layer on top of `fs` that:
+
+- refuses non-regular files (directories, FIFOs, sockets, symlinks)
+- enforces a `maxBytes` read cap
+- separates "missing" from "io-error" in the result type
+
+```ts
+import {
+  readRegularFile,
+  readRegularFileSync,
+  appendRegularFile,
+  appendRegularFileSync,
+  resolveRegularFileAppendFlags,
+  statRegularFile,
+  statRegularFileSync,
+  type AppendRegularFileOptions,
+  type RegularFileStatResult,
+} from "@openclaw/fs-safe/advanced";
+```
+
+## Stat
+
+### `statRegularFile(filePath)`
+
+Async. Returns:
+
+```ts
+type RegularFileStatResult =
+  | { missing: true }
+  | { missing: false; stat: Stats };
+```
+
+A non-regular file (directory, FIFO, …) returns `{ missing: false }` with a `stat` whose `isFile()` is false — the helper does not throw, you decide what to do.
+
+```ts
+import { statRegularFile } from "@openclaw/fs-safe/advanced";
+
+const r = await statRegularFile("/var/log/app.log");
+if (r.missing) return;
+if (!r.stat.isFile()) throw new Error("expected a regular file");
+console.log(`size=${r.stat.size}`);
+```
+
+### `statRegularFileSync(filePath)`
+
+Synchronous variant. Same shape.
+
+## Read
+
+### `readRegularFile(params)`
+
+Async. Reads the entire file into a Buffer if it is a regular file, with `maxBytes` enforcement.
+
+```ts
+import { readRegularFile } from "@openclaw/fs-safe/advanced";
+
+const result = await readRegularFile({
+  filePath: "/var/log/app.log",
+  maxBytes: 4 * 1024 * 1024,
+});
+if (result.missing) return null;
+if (!result.regular) throw new Error("not a regular file");
+processLog(result.buffer);
+```
+
+Result shape:
+
+```ts
+type Result =
+  | { missing: true }
+  | { missing: false; regular: false; stat: Stats }
+  | { missing: false; regular: true; stat: Stats; buffer: Buffer };
+```
+
+Throws `FsSafeError` with code `too-large` if the file exceeds `maxBytes`. Other I/O errors propagate as `NodeJS.ErrnoException`.
+
+### `readRegularFileSync(params)`
+
+Synchronous variant. Same shape; the only required field is `filePath`. `maxBytes` is optional.
+
+## Append
+
+### `appendRegularFile(options)`
+
+Async. Opens the file in append mode, writes data, closes. Refuses non-regular targets:
+
+```ts
+import { appendRegularFile } from "@openclaw/fs-safe/advanced";
+
+await appendRegularFile({
+  filePath: "/var/log/app.log",
+  data: `[${new Date().toISOString()}] ${line}\n`,
+  encoding: "utf8",
+  prependNewlineIfNeeded: true,
+});
+```
+
+### Options
+
+```ts
+type AppendRegularFileOptions = {
+  filePath: string;
+  data: string | Buffer;
+  encoding?: BufferEncoding;             // default utf8 when data is string
+  prependNewlineIfNeeded?: boolean;      // insert "\n" if file does not end with one
+  flags?: number;                         // raw open flags; default O_WRONLY | O_APPEND
+  mode?: number;                          // default 0o644 if file is created
+};
+```
+
+`prependNewlineIfNeeded` reads the trailing byte of the existing file and prepends a `\n` to your data if it isn't already present. Useful for log appenders that want to preserve line boundaries even when callers forget the newline.
+
+### `appendRegularFileSync(options)`
+
+Synchronous. Same options.
+
+### `resolveRegularFileAppendFlags(append, truncateExisting)`
+
+Helper that returns the right open-flag bitmask for combinations of "append" / "truncate". Use it when you're building your own open path and want to match the append helpers' behavior:
+
+```ts
+import { resolveRegularFileAppendFlags } from "@openclaw/fs-safe/advanced";
+
+const flags = resolveRegularFileAppendFlags(true, false); // O_WRONLY | O_APPEND | O_CREAT
+```
+
+## Difference from `Root` methods
+
+| `regular-file` | `Root` |
+|---|---|
+| Absolute paths only. | Relative to the root. |
+| No identity check post-open. | Identity check on every read/write. |
+| Caller must be confident the path is trusted. | Boundary check is automatic. |
+| Returns explicit `{missing, regular}` shape. | Throws `FsSafeError` with `code`. |
+
+If your call site already trusts the path (it came from your own config, not a caller), `regular-file` is a thinner, faster surface. If the path is caller-influenced, prefer `root()` or wrap in [`pathScope()`](path-scope.md).
+
+## Common patterns
+
+### Read a config file if it's there, else seed
+
+```ts
+const r = await readRegularFile({ filePath: "/etc/app/config.json", maxBytes: 64 * 1024 });
+if (r.missing) {
+  await writeJson("/etc/app/config.json", defaultConfig);
+} else if (r.regular) {
+  applyConfig(JSON.parse(r.buffer.toString("utf8")));
+} else {
+  throw new Error("/etc/app/config.json is not a regular file");
+}
+```
+
+### Cheap "exists and is a file" check
+
+```ts
+const r = await statRegularFile(p);
+if (r.missing || !r.stat.isFile()) return false;
+return true;
+```
+
+### Bounded log tail
+
+```ts
+const r = await readRegularFile({ filePath: logPath, maxBytes: 1 * 1024 * 1024 });
+if (r.missing || !r.regular) return [];
+return r.buffer.toString("utf8").split("\n").slice(-100);
+```
+
+## See also
+
+- [Reading](reading.md) — `Root` reads with boundary checks.
+- [Atomic writes](atomic.md) — for atomic write semantics, prefer `replaceFileAtomic`.
+- [`fs.appendFile`](https://nodejs.org/api/fs.html#fsappendfilepath-data-options-callback) — Node's stock append, without regular-file gating.

package/docs/root.md

@@ -0,0 +1,205 @@
+# root()
+
+`root()` is the primary entry point. It takes a trusted directory and returns a capability-style `Root` handle whose methods accept relative paths and refuse to escape the directory.
+
+```ts
+import { root } from "@openclaw/fs-safe";
+
+const fs = await root("/srv/workspace", {
+  hardlinks: "reject",
+  symlinks: "reject",
+  mkdir: true,
+});
+```
+
+## Signature
+
+```ts
+function root(rootDir: string, defaults?: RootDefaults): Promise<Root>;
+
+type RootDefaults = {
+  hardlinks?: "reject" | "allow";  // refuse files with nlink > 1 on read; defaults to "reject"
+  maxBytes?: number;               // refuse reads larger than this many bytes; defaults to 16 MiB
+  mkdir?: boolean;                 // create missing parent dirs on write/openWritable/append
+  mode?: number;                   // file mode applied to new writes; per-call override available
+  nonBlockingRead?: boolean;       // schedule reads on a worker; useful for large files
+  symlinks?: "reject" | "follow-within-root"; // policy when a path component is a symlink
+};
+```
+
+`root()` resolves the directory through the real filesystem. A symlinked input becomes the canonical path; a non-existent root throws `FsSafeError` with code `not-found`, and malformed or non-directory roots throw `invalid-path`.
+
+`defaults` apply to every method on the returned handle. Per-call options on individual methods override the defaults for that call only.
+
+## The `Root` interface
+
+Every method on the returned handle accepts paths relative to the root and rejects anything that would escape it.
+
+### Reads
+
+```ts
+fs.read(rel, options?)         // { buffer, realPath, stat }
+fs.readBytes(rel, options?)    // Buffer
+fs.readText(rel, options?)     // string
+fs.readJson<T>(rel, options?)  // parsed T
+fs.open(rel, options?)         // { handle, realPath, stat, [Symbol.asyncDispose] }
+fs.readAbsolute(absPath, options?) // ReadResult; absPath must already be inside the root
+fs.reader(options?)            // (path) => Promise<Buffer>; useful for loader APIs
+```
+
+`open()` returns a Node `FileHandle` for streaming. Prefer `await using` for cleanup:
+
+```ts
+await using opened = await fs.open("large.log");
+{
+  for await (const chunk of opened.handle.createReadStream()) {
+    process.stdout.write(chunk);
+  }
+}
+```
+
+### Writes
+
+```ts
+fs.write(rel, data, options?)            // overwrite-ok atomic write
+fs.create(rel, data, options?)           // throws "already-exists" if target exists
+fs.writeJson(rel, value, options?)       // JSON.stringify + atomic write
+fs.createJson(rel, value, options?)      // create() variant of writeJson
+fs.append(rel, data, options?)           // append text/buffer; respects mkdir default
+fs.copyIn(rel, sourceAbsPath, options?)  // copy from outside the root, atomically, with size cap
+fs.openWritable(rel, options?)           // FileHandle for streaming writes; supports await using
+fs.move(from, to, options?)              // rename within the root; defaults to no clobber
+fs.remove(rel)                           // unlink file or rmdir empty directory
+fs.mkdir(rel)                            // mkdir -p (creates missing parents)
+fs.ensureRoot()                          // accepts "" / "." as the root itself
+```
+
+`write`, `create`, `append`, `writeJson`, and `createJson` accept `mode?: number`; use `0o600` for credentials and other private state. `writeJson` also accepts the same options as `JSON.stringify` plus `trailingNewline?: boolean` (defaults `true` so the file ends in `\n`).
+
+`copyIn` is a one-shot ingest from a trusted absolute source path: it streams the source through the boundary, atomically renames into the root, and respects `maxBytes`.
+
+`openWritable` opens a writable file with options `mode?: number` and `writeMode?: "replace" | "append" | "update"`. `replace` truncates existing files and is the default; `update` keeps existing contents. Use it for streaming output. Prefer `await using` for cleanup.
+
+### Inspection (advisory)
+
+```ts
+fs.exists(rel)                   // boolean
+fs.stat(rel)                     // PathStat
+fs.list(rel)                     // string[]
+fs.list(rel, { withFileTypes })  // DirEntry[]
+fs.resolve(rel)                  // absolute path inside the root, after canonicalization
+```
+
+These do not pin a later operation. They are safe to expose to UIs and decision points; for the actual read or write, use the verb methods so the operation pins identity at the point of use.
+
+## Python helper mode
+
+On POSIX, mutation and inspection methods that need fd-relative directory
+operations go through one persistent Python helper process. This avoids a
+spawn-per-call cost while still using `openat`/`renameat`/`unlinkat`-style
+operations that Node's `fs` API does not expose ergonomically.
+
+```ts
+import { configureFsSafePython } from "@openclaw/fs-safe/config";
+
+configureFsSafePython({ mode: "off" });     // Node-only fallback path
+configureFsSafePython({ mode: "require" }); // fail if fd-relative helper unavailable
+```
+
+`auto` is the default. Configure the mode before creating roots. Without the
+helper, root methods still run, but same-UID races that swap parent directories
+between validation and mutation are harder to close completely. Use `require`
+when that downgrade should be treated as a deployment failure. See
+[Python helper policy](python-helper.md) for deployment guidance.
+
+### Properties
+
+```ts
+fs.rootDir       // the directory you passed in
+fs.rootReal      // its canonical real path (after symlink resolution)
+fs.rootWithSep   // rootReal with a trailing separator, for prefix comparisons
+fs.defaults      // the RootDefaults you passed
+```
+
+## Failure semantics
+
+Every method throws `FsSafeError` with a `code`. Branch on `err.code`, not message text. Common codes:
+
+| Code | When it fires |
+|---|---|
+| `invalid-path` | The input path is malformed, including embedded NUL bytes. |
+| `outside-workspace` | The input resolves outside the root, or contains a `..` segment that would escape it. |
+| `not-found` | The target does not exist (or its parent does not, with `mkdir: false`). |
+| `not-file` | A read or copy targeted a non-regular file (directory, FIFO, socket, …). |
+| `already-exists` | `create()` or `move()` without `overwrite` hit an existing target. |
+| `symlink` | A path component is a symlink, and the call's `symlinks` policy is `reject`. |
+| `hardlink` | The target's `nlink > 1` and `hardlinks` policy is `reject`. |
+| `path-mismatch` | Post-open identity check failed — the opened fd does not match the resolved path. |
+| `too-large` | Read exceeded `maxBytes`. |
+
+Full list in the [Errors](errors.md) reference.
+
+## Defaults vs per-call options
+
+Defaults reduce repetition; per-call options handle exceptions:
+
+```ts
+const fs = await root("/srv/workspace", {
+  symlinks: "reject",
+  hardlinks: "reject",
+  mkdir: true,
+});
+
+// Default: symlinks rejected.
+await fs.readText("config.toml");
+
+// One specific path needs to follow a symlink that lands inside the root.
+await fs.readText("links/current.log", { symlinks: "follow-within-root" });
+```
+
+Text helpers default to UTF-8. Pass `encoding` per call to `readText`, `readJson`, `write`, `create`, or `append` when you need another encoding.
+
+## Common patterns
+
+### Read-only loader
+
+```ts
+const fs = await root("/srv/workspace", { symlinks: "reject", hardlinks: "reject" });
+const load = fs.reader();
+const a = await load("notes/today.txt");        // relative
+const b = await load("/srv/workspace/state.bin"); // absolute, but inside the root
+```
+
+`fs.reader()` returns a `(path) => Promise<Buffer>` callback. Useful when wiring `fs-safe` into APIs that accept a generic loader function. Absolute paths outside the root are rejected with `outside-workspace`.
+
+### "Touch only if missing" seeding
+
+```ts
+try {
+  await fs.create("config/seed.json", initialJson);
+} catch (err) {
+  if (err instanceof FsSafeError && err.code === "already-exists") {
+    // existing config wins
+  } else {
+    throw err;
+  }
+}
+```
+
+### Replace + verify
+
+```ts
+await fs.write("state.json", JSON.stringify(state, null, 2));
+const echoed = await fs.readJson<State>("state.json");
+assertDeepEqual(echoed, state);
+```
+
+`write` is atomic, so the file is either old or new — never half-written. Re-reading lets you detect a parallel writer, if one exists.
+
+## See also
+
+- [Reading](reading.md) — read variants in depth, plus stream patterns.
+- [Writing](writing.md) — write/create/move/remove in depth.
+- [pathScope()](path-scope.md) — the same boundary semantics over an absolute path you already trust.
+- [Atomic writes](atomic.md) — the lower-level helpers used by `fs.write`.
+- [Errors](errors.md) — the closed code union you'll be catching.