@openclaw/fs-safe 0.1.1 → 0.2.0

package/docs/install-path.md

@@ -0,0 +1,167 @@
+# Install paths
+
+Helpers for code that creates per-name install directories under a trusted base — typical for plugins, packages, snapshots, anywhere you want `<base>/<safe-name>/`. The combination of [`resolveSafeInstallDir`](#resolvesafeinstalldir) and [`assertCanonicalPathWithinBase`](#assertcanonicalpathwithinbase) gives you "compute the install path safely, then re-verify after creation."
+
+```ts
+import {
+  assertCanonicalPathWithinBase,
+  resolveSafeInstallDir,
+  safeDirName,
+  safePathSegmentHashed,
+} from "@openclaw/fs-safe/advanced";
+```
+
+## `resolveSafeInstallDir`
+
+```ts
+function resolveSafeInstallDir(params: {
+  baseDir: string;
+  id: string;
+  invalidNameMessage: string;
+  nameEncoder?: (id: string) => string;   // default safeDirName
+}): { ok: true; path: string } | { ok: false; error: string };
+```
+
+Computes the absolute install directory for `id` under `baseDir`, after running `id` through `nameEncoder` (`safeDirName` by default). Verifies the result stays inside `baseDir` — anything that would escape returns `{ ok: false, error: invalidNameMessage }`.
+
+```ts
+const r = resolveSafeInstallDir({
+  baseDir: "/srv/plugins",
+  id: "@scope/my-plugin",
+  invalidNameMessage: "invalid plugin name",
+});
+if (!r.ok) return reply(400, r.error);
+
+await fs.mkdir(r.path, { recursive: true });
+```
+
+For ids whose default-sanitized form might collide (e.g. `"foo/bar"` and `"foo\\bar"` both map to `"foo__bar"`), pass `nameEncoder: safePathSegmentHashed` to append a content hash:
+
+```ts
+const r = resolveSafeInstallDir({
+  baseDir: "/srv/plugins",
+  id: untrustedId,
+  invalidNameMessage: "invalid plugin name",
+  nameEncoder: safePathSegmentHashed,
+});
+```
+
+The helper does **not** create the directory — it returns the path. Pair with `fs.mkdir`, [`Root.mkdir`](root.md), or `assertCanonicalPathWithinBase` before/after creation as needed.
+
+## `assertCanonicalPathWithinBase`
+
+Async. Verifies that a candidate absolute path's canonical real path stays inside the base. Useful as a post-`mkdir` check, or when you have an existing path you didn't compute yourself.
+
+```ts
+function assertCanonicalPathWithinBase(params: {
+  baseDir: string;
+  candidatePath: string;
+  boundaryLabel: string;
+}): Promise<void>;
+```
+
+Throws if the candidate resolves outside `baseDir` after `realpath`. The `boundaryLabel` is included in the error message ("Invalid path: must stay within {boundaryLabel}").
+
+```ts
+await assertCanonicalPathWithinBase({
+  baseDir: "/srv/plugins",
+  candidatePath: "/srv/plugins/my-plugin",
+  boundaryLabel: "plugin install dir",
+});
+```
+
+If the candidate does not exist, the helper validates the parent directory instead — useful for "the directory I'm about to create" semantics.
+
+## Segment sanitizers
+
+### `safeDirName`
+
+Returns a directory-safe segment derived from `input` by replacing `/` and `\` with `__`. Trims whitespace; returns an empty string if the input was only whitespace.
+
+```ts
+safeDirName("@scope/my-plugin");       // "@scope__my-plugin"
+safeDirName("../../etc");              // "..__..__etc"
+safeDirName("plugin-v1");              // "plugin-v1"
+safeDirName("");                       // ""
+```
+
+`safeDirName` does *not* try to be exhaustive about Windows-reserved names or special characters. It is purely a separator-stripping pass — `resolveSafeInstallDir` adds the boundary check on top so an `"../../etc"` input cannot escape `baseDir`.
+
+For stricter sanitization, use `safePathSegmentHashed`.
+
+### `safePathSegmentHashed`
+
+Returns a directory-safe segment **plus** a short content hash when sanitization changed the input or when the safe form is too long. Use this when input collisions matter:
+
+```ts
+safePathSegmentHashed("plugin-v1");                  // "plugin-v1"  (unchanged short input)
+safePathSegmentHashed("plugin/v1");                  // "plugin-v1-3f2a..."
+safePathSegmentHashed("plugin\\v1");                 // "plugin-v1-91c4..."  (different hash; same safe form)
+safePathSegmentHashed("Über@");                       // "ber-9aae..."
+safePathSegmentHashed("");                           // "skill"  (empty fallback)
+safePathSegmentHashed(".");                          // "skill"
+```
+
+The sanitization is more aggressive than `safeDirName`: any character not in `[A-Za-z0-9._-]` becomes `-`, runs of `-` collapse, leading and trailing `-` are stripped, the empty/`.`/`..` fallback is `"skill"`. Long results are truncated to 50 chars before the hash is appended.
+
+The hash is the first 10 hex chars of `sha256(originalInput)`. It guarantees that two distinct inputs which sanitize to the same string yield distinct outputs.
+
+## Common patterns
+
+### Install a plugin
+
+```ts
+import { resolveSafeInstallDir, assertCanonicalPathWithinBase, safePathSegmentHashed } from "@openclaw/fs-safe/advanced";
+import { extractArchive } from "@openclaw/fs-safe/archive";
+import fs from "node:fs/promises";
+
+const r = resolveSafeInstallDir({
+  baseDir: "/srv/plugins",
+  id: untrustedName,
+  invalidNameMessage: "invalid plugin name",
+  nameEncoder: safePathSegmentHashed,
+});
+if (!r.ok) return reply(400, r.error);
+
+await fs.mkdir(r.path, { recursive: true, mode: 0o755 });
+await assertCanonicalPathWithinBase({
+  baseDir: "/srv/plugins",
+  candidatePath: r.path,
+  boundaryLabel: "plugin install dir",
+});
+
+await extractArchive({
+  archivePath: pluginZip,
+  destDir: r.path,
+  kind: "zip",
+  timeoutMs: 30_000,
+});
+```
+
+### Per-version snapshot directories
+
+```ts
+const snap = resolveSafeInstallDir({
+  baseDir: "/srv/snapshots",
+  id: `${runId}-${version}`,
+  invalidNameMessage: "invalid snapshot id",
+});
+if (!snap.ok) throw new Error(snap.error);
+await fs.mkdir(snap.path, { recursive: true });
+```
+
+### Reject and log on bad input
+
+```ts
+const r = resolveSafeInstallDir({ baseDir, id, invalidNameMessage: "bad name" });
+if (!r.ok) {
+  logger.warn({ id, base: baseDir, error: r.error }, "rejected install attempt");
+  return reply(400, r.error);
+}
+```
+
+## See also
+
+- [`root()`](root.md) — when the install dir becomes a root for further writes.
+- [Filenames](filename.md) — `sanitizeUntrustedFileName` for file-name (not directory-name) sanitization.
+- [Archive extraction](archive.md) — extract into the install dir computed by these helpers.

package/docs/install.md

@@ -0,0 +1,150 @@
+# Install
+
+`fs-safe` is published to npm as `@openclaw/fs-safe`. It targets Node 20.11 or newer, ships ESM only, and works on macOS, Linux, and Windows.
+
+## Package managers
+
+```bash
+pnpm add @openclaw/fs-safe
+```
+
+```bash
+npm install @openclaw/fs-safe
+```
+
+```bash
+yarn add @openclaw/fs-safe
+```
+
+```bash
+bun add @openclaw/fs-safe
+```
+
+## Node version
+
+Minimum **Node 20.11**. The package uses `fs.promises`, `fs.constants.O_NOFOLLOW` where available, and `node:stream/promises`. Earlier Node releases will fail at import time.
+
+Verify the runtime:
+
+```bash
+node --version
+# v20.11.0 or newer
+```
+
+## TypeScript
+
+Types ship with the package — no `@types/openclaw__fs-safe` needed. The `exports` map in `package.json` provides typed entries for every subpath:
+
+```ts
+import { root, FsSafeError } from "@openclaw/fs-safe";
+import { writeJson } from "@openclaw/fs-safe/json";
+import { extractArchive } from "@openclaw/fs-safe/archive";
+```
+
+A working `tsconfig.json` for consumers:
+
+```jsonc
+{
+  "compilerOptions": {
+    "target": "es2022",
+    "module": "node18",
+    "moduleResolution": "node16",
+    "strict": true,
+    "esModuleInterop": true,
+    "skipLibCheck": true
+  }
+}
+```
+
+## Subpath exports
+
+Use the main entry for the common surface, or the focused subpaths when you want a leaner import or to depend on a narrower contract:
+
+| Subpath | Contents |
+|---|---|
+| `@openclaw/fs-safe` | Small common surface: `root`, root types, and errors. |
+| `@openclaw/fs-safe/root` | `root()`, `Root`, `RootDefaults`, related types. |
+| `@openclaw/fs-safe/config` | Process-global Python helper configuration. |
+| `@openclaw/fs-safe/path` | `isPathInside`, `safeRealpathSync`, `isWithinDir`, error helpers. |
+| `@openclaw/fs-safe/json` | `tryReadJson`, `readJson`, `readJsonIfExists`, `writeJson`, sync variants. |
+| `@openclaw/fs-safe/store` | `fileStore()`, `fileStoreSync()`, and `jsonStore<T>()`. |
+| `@openclaw/fs-safe/secret` | Secret file read/write helpers. |
+| `@openclaw/fs-safe/atomic` | `replaceFileAtomic`, `writeTextAtomic`, `replaceDirectoryAtomic`, `movePathWithCopyFallback`. |
+| `@openclaw/fs-safe/temp` | `tempWorkspace`, `withTempWorkspace`, sync variants, `resolveSecureTempRoot`. |
+| `@openclaw/fs-safe/secure-file` | `readSecureFile` for pinned absolute file reads with permissions checks. |
+| `@openclaw/fs-safe/file-lock` | `acquireFileLock`, `withFileLock`, `createFileLockManager`, and related lock types. |
+| `@openclaw/fs-safe/permissions` | POSIX mode and Windows ACL inspection/remediation helpers. |
+| `@openclaw/fs-safe/walk` | `walkDirectory`, `walkDirectorySync`, related types. Budget-bounded, not root-bounded. |
+| `@openclaw/fs-safe/archive` | `extractArchive`, `resolveArchiveKind`, limits, preflight helpers. |
+| `@openclaw/fs-safe/advanced` | Lower-level composition helpers: path scopes, root-file open, install paths, local-root readers, temp-file targets, sibling-temp writes, regular-file helpers, `pathExists`, `withTimeout`, and related advanced types. This surface is less stable than the focused public subpaths. |
+| `@openclaw/fs-safe/errors` | `FsSafeError`, `FsSafeErrorCode`. |
+| `@openclaw/fs-safe/types` | Shared types: `DirEntry`, `PathStat`, `BasePathOptions`, … |
+| `@openclaw/fs-safe/test-hooks` | Test-only hooks for injecting races. Active under `NODE_ENV=test`. |
+
+## Runtime dependencies
+
+`@openclaw/fs-safe` lists `jszip` and `tar` as optional dependencies for [archive extraction](archive.md). They are loaded lazily and only required when ZIP/TAR helpers run. Installs that omit optional dependencies can still import and use every non-archive subpath; archive calls fail with a clear missing-optional-dependency message.
+
+There are no peer dependencies and no native build step.
+
+## Python helper policy
+
+On POSIX, `root()` uses one persistent Python helper process for the
+fd-relative operations Node does not expose cleanly. The default is `auto`: use
+the helper when it starts, fall back to Node-only behavior when it is disabled
+or unavailable.
+
+```ts
+import { configureFsSafePython } from "@openclaw/fs-safe/config";
+
+configureFsSafePython({ mode: "auto" });    // default
+configureFsSafePython({ mode: "off" });     // never spawn Python
+configureFsSafePython({ mode: "require" }); // fail closed if unavailable
+```
+
+Environment variables are read at runtime:
+
+```bash
+FS_SAFE_PYTHON_MODE=off      # auto | off | require
+FS_SAFE_PYTHON=/usr/bin/python3
+```
+
+OpenClaw compatibility aliases are also accepted:
+`OPENCLAW_FS_SAFE_PYTHON_MODE`, `OPENCLAW_FS_SAFE_PYTHON`,
+`OPENCLAW_PINNED_PYTHON`, and `OPENCLAW_PINNED_WRITE_PYTHON`.
+
+Disabling Python keeps the public API working, but downgrades POSIX mutation
+hardening from fd-relative syscalls to Node path operations guarded by lexical
+and canonical checks plus identity verification. Use `require` for
+security-sensitive deployments where that downgrade should be a startup/runtime
+failure instead of a fallback. The full tradeoff is documented in
+[Python helper policy](python-helper.md).
+
+## Verify the install
+
+```ts
+import { root, FsSafeError } from "@openclaw/fs-safe";
+import os from "node:os";
+import path from "node:path";
+
+const dir = path.join(os.tmpdir(), "fs-safe-smoke");
+await import("node:fs/promises").then((fs) => fs.mkdir(dir, { recursive: true }));
+
+const fs = await root(dir);
+await fs.write("hello.txt", "ok\n");
+console.log(await fs.readText("hello.txt"));
+
+try {
+  await fs.write("../escape.txt", "x");
+} catch (err) {
+  if (err instanceof FsSafeError) console.log("blocked:", err.code);
+}
+```
+
+If the script prints `ok` followed by `blocked: outside-workspace`, your install is healthy.
+
+## Next
+
+- [Quickstart](quickstart.md) — write, read, atomic, temp.
+- [Security model](security-model.md) — what the boundary defends against.
+- [Errors](errors.md) — the closed code union you'll be catching.

package/docs/json-store.md

@@ -0,0 +1,204 @@
+# JSON store
+
+`jsonStore` is exported from `@openclaw/fs-safe/store`. It is the absolute-path
+convenience wrapper for `fileStore(...).json(...)`: a small read-modify-write
+handle around a single JSON file. It bakes in atomic writes, explicit fallback
+reads, and optional cross-process locking via
+[`acquireFileLock`](sidecar-lock.md).
+
+```ts
+import { jsonStore } from "@openclaw/fs-safe/store";
+
+const settings = jsonStore<{ theme: "light" | "dark"; volume: number }>({
+  filePath: "/var/lib/app/settings.json",
+});
+
+const current = await settings.readOr({ theme: "dark", volume: 0.7 });
+await settings.write({ ...current, volume: 1 });
+await settings.updateOr({ theme: "dark", volume: 0.7 }, (prev) => ({ ...prev, theme: "light" }));
+```
+
+If you already have a store/root context, prefer binding the JSON file from that
+store:
+
+```ts
+import { fileStore } from "@openclaw/fs-safe/store";
+
+const files = fileStore({ rootDir: "/var/lib/app", private: true });
+const settings = files.json<Settings>("settings.json", { lock: true });
+```
+
+## When to reach for it
+
+- You have a single JSON state file and want `read / readOr / readRequired / write / update` semantics.
+- You want every write atomic at file mode `0o600` and parents at `0o700` by default.
+- You want optional cross-process locking with one boolean.
+
+For ad-hoc read/write of multiple JSON files, use the standalone helpers in
+[`json`](json.md). For object-style storage of many files at known modes, use
+[`fileStore`](file-store.md) and bind JSON files with `store.json(rel)`.
+
+## Factory: `jsonStore<T>(options)`
+
+```ts
+type JsonStoreOptions<T> = {
+  filePath: string;
+  dirMode?: number;                                // default 0o700
+  mode?: number;                                   // default 0o600
+  trailingNewline?: boolean;                       // default true
+  lock?: boolean | JsonStoreLockOptions;           // false / undefined = no lock
+};
+
+type JsonStoreLockOptions = {
+  staleMs?: number;     // default 30_000
+  timeoutMs?: number;   // default 30_000
+  retry?: FileLockRetryOptions;
+  staleRecovery?: "fail-closed";
+  managerKey?: string;  // default `fs-safe.json-store:<filePath>`
+};
+
+type JsonStore<T> = {
+  readonly filePath: string;
+  read(): Promise<T | undefined>;
+  readOr(fallback: T): Promise<T>;
+  readRequired(): Promise<T>;
+  write(value: T): Promise<void>;
+  update(run: (current: T | undefined) => T | Promise<T>): Promise<T>;
+  updateOr(fallback: T, run: (current: T) => T | Promise<T>): Promise<T>;
+};
+```
+
+`jsonStore({ filePath })` resolves `rootDir = dirname(filePath)` and calls
+`fileStore({ rootDir, private: true }).json(basename(filePath), options)`.
+
+The store does **not** validate the parsed value against `T` at runtime — the cast is unchecked. Wrap with a schema (zod/valibot) if the file might be hand-edited or written by another process you don't control.
+
+## `read()`
+
+Returns the parsed contents, or `undefined` if the file does not exist. Invalid JSON throws (via [`readJsonIfExists`](json.md)).
+
+```ts
+const state = await store.read();
+```
+
+## `readOr(fallback)`
+
+Returns the parsed contents or the per-call fallback. Object fallbacks are cloned so callers can safely mutate the returned value:
+
+```ts
+const state = await store.readOr(defaultState);
+```
+
+## `readRequired()`
+
+Strict disk read. Throws when the file is missing or invalid:
+
+```ts
+const state = await store.readRequired();
+```
+
+## `write(value)`
+
+Atomic JSON write at `mode` (default `0o600`), creating parent dirs at `dirMode` (default `0o700`) if needed. When `lock: true` is set, takes the sidecar lock for the duration of the write.
+
+```ts
+await store.write({ ...state, lastSeen: Date.now() });
+```
+
+## `update(run)`
+
+Read, transform, write — under the lock if locking is enabled. Returns the new value:
+
+```ts
+const next = await store.update((prev) => ({ count: (prev?.count ?? 0) + 1 }));
+```
+
+`run` is async-friendly. The whole `read → run → write` sequence runs inside one `withLock` call, so concurrent updaters from different processes serialize cleanly.
+
+Use `update(run)` when missing state is part of your model. Use `updateOr(fallback, run)` when the missing-file case should start from a concrete value and you want to merge into defaults:
+
+```ts
+const next = await store.updateOr({ count: 0 }, (prev) => ({ count: prev.count + 1 }));
+```
+
+## Locking
+
+Set `lock: true` for default behavior, or pass an options object to tune:
+
+```ts
+const counter = jsonStore<{ count: number }>({
+  filePath: "/var/lib/app/counter.json",
+  lock: {
+    staleMs: 60_000,
+    timeoutMs: 10_000,
+    staleRecovery: "fail-closed",
+    retry: { retries: 30, minTimeout: 100, maxTimeout: 5_000, randomize: true },
+  },
+});
+```
+
+When `lock` is falsy, `read` / `write` / `update` are unlocked. The `update` shape is still useful — it gives you a single function for the read-modify-write pattern — but it offers no concurrency guarantees if other processes also write to the file.
+
+Process-wide lock defaults from `configureFsSafeLocks()` apply only after locking is explicitly enabled. They do not make JSON stores lock by default.
+
+The default `managerKey` namespaces the in-process `FileLockManager` per absolute file path, so two `jsonStore` calls on the same file share lock state automatically.
+
+## Common patterns
+
+### Per-feature settings file
+
+```ts
+type Settings = { theme: "light" | "dark"; muted: boolean };
+
+const settings = jsonStore<Settings>({
+  filePath: path.join(homedir(), ".myapp/settings.json"),
+});
+
+// Read on boot
+applySettings(await settings.readOr({ theme: "dark", muted: false }));
+
+// Toggle on UI action
+await settings.update((prev) => {
+  const current = prev ?? { theme: "dark", muted: false };
+  return { ...current, muted: !current.muted };
+});
+```
+
+### Cross-process counter
+
+```ts
+const counter = jsonStore<{ count: number }>({
+  filePath: "/var/lib/app/counter.json",
+  lock: true,
+});
+
+const { count } = await counter.updateOr({ count: 0 }, (prev) => ({ count: prev.count + 1 }));
+console.log("now at", count);
+```
+
+### Migration on boot
+
+```ts
+const config = jsonStore<Config>({ filePath });
+const current = await config.readOr(defaultConfig);
+if (current.version !== CURRENT_VERSION) {
+  await config.write(migrate(current));
+}
+```
+
+## Difference from raw `writeJson` / `readJsonIfExists`
+
+| `jsonStore` | Raw helpers |
+|---|---|
+| Read-modify-write in one call (`update`). | Compose `readJsonIfExists` + `writeJson` yourself. |
+| Optional cross-process lock with one flag. | Manage `withFileLock` yourself. |
+| Explicit `readOr` / `updateOr` fallbacks. | Caller handles `null` and clones. |
+| Mode/dirMode locked per store. | Per-call. |
+
+`jsonStore` is the right shape when one file owns one piece of state and many call sites read or update it. For one-off writes, the raw helpers are leaner.
+
+## See also
+
+- [JSON files](json.md) — the standalone helpers `jsonStore` is built on.
+- [File lock](sidecar-lock.md) — the cross-process lock used when `lock: true`.
+- [File store](file-store.md) — the multi-file equivalent of this surface.