@openclaw/fs-safe 0.1.1 → 0.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (207) hide show
  1. package/CHANGELOG.md +58 -0
  2. package/README.md +58 -2
  3. package/dist/absolute-path.d.ts +14 -0
  4. package/dist/absolute-path.d.ts.map +1 -1
  5. package/dist/absolute-path.js +203 -2
  6. package/dist/advanced.d.ts +2 -1
  7. package/dist/advanced.d.ts.map +1 -1
  8. package/dist/advanced.js +2 -1
  9. package/dist/archive-staging.d.ts.map +1 -1
  10. package/dist/archive-staging.js +81 -7
  11. package/dist/archive.d.ts.map +1 -1
  12. package/dist/archive.js +25 -42
  13. package/dist/bounded-read-stream.d.ts +8 -0
  14. package/dist/bounded-read-stream.d.ts.map +1 -0
  15. package/dist/bounded-read-stream.js +20 -0
  16. package/dist/config.d.ts +1 -0
  17. package/dist/config.d.ts.map +1 -1
  18. package/dist/config.js +1 -0
  19. package/dist/directory-guard.d.ts +18 -0
  20. package/dist/directory-guard.d.ts.map +1 -0
  21. package/dist/directory-guard.js +70 -0
  22. package/dist/file-lock.d.ts +4 -2
  23. package/dist/file-lock.d.ts.map +1 -1
  24. package/dist/file-lock.js +13 -2
  25. package/dist/file-store-boundary.d.ts +25 -0
  26. package/dist/file-store-boundary.d.ts.map +1 -0
  27. package/dist/file-store-boundary.js +168 -0
  28. package/dist/file-store-prune.d.ts +12 -0
  29. package/dist/file-store-prune.d.ts.map +1 -0
  30. package/dist/file-store-prune.js +86 -0
  31. package/dist/file-store-source.d.ts +5 -0
  32. package/dist/file-store-source.d.ts.map +1 -0
  33. package/dist/file-store-source.js +30 -0
  34. package/dist/file-store.d.ts +2 -6
  35. package/dist/file-store.d.ts.map +1 -1
  36. package/dist/file-store.js +122 -125
  37. package/dist/guarded-mkdir.d.ts +6 -0
  38. package/dist/guarded-mkdir.d.ts.map +1 -0
  39. package/dist/guarded-mkdir.js +45 -0
  40. package/dist/guarded-mutation.d.ts +33 -0
  41. package/dist/guarded-mutation.d.ts.map +1 -0
  42. package/dist/guarded-mutation.js +76 -0
  43. package/dist/index.d.ts +2 -0
  44. package/dist/index.d.ts.map +1 -1
  45. package/dist/index.js +2 -0
  46. package/dist/json-document-store.d.ts +2 -0
  47. package/dist/json-document-store.d.ts.map +1 -1
  48. package/dist/json-document-store.js +7 -3
  49. package/dist/json-durable-queue.d.ts +45 -0
  50. package/dist/json-durable-queue.d.ts.map +1 -0
  51. package/dist/json-durable-queue.js +200 -0
  52. package/dist/json.d.ts +35 -5
  53. package/dist/json.d.ts.map +1 -1
  54. package/dist/json.js +69 -2
  55. package/dist/local-roots.d.ts.map +1 -1
  56. package/dist/local-roots.js +17 -3
  57. package/dist/lock-config.d.ts +10 -0
  58. package/dist/lock-config.d.ts.map +1 -0
  59. package/dist/lock-config.js +12 -0
  60. package/dist/move-path.d.ts +1 -0
  61. package/dist/move-path.d.ts.map +1 -1
  62. package/dist/move-path.js +195 -7
  63. package/dist/output.d.ts +13 -0
  64. package/dist/output.d.ts.map +1 -0
  65. package/dist/output.js +69 -0
  66. package/dist/path-stat.d.ts +4 -0
  67. package/dist/path-stat.d.ts.map +1 -0
  68. package/dist/path-stat.js +15 -0
  69. package/dist/path.d.ts +1 -0
  70. package/dist/path.d.ts.map +1 -1
  71. package/dist/path.js +6 -3
  72. package/dist/pinned-write.d.ts.map +1 -1
  73. package/dist/pinned-write.js +44 -19
  74. package/dist/private-file-store.d.ts +5 -7
  75. package/dist/private-file-store.d.ts.map +1 -1
  76. package/dist/private-file-store.js +21 -34
  77. package/dist/private-temp-workspace.d.ts.map +1 -1
  78. package/dist/private-temp-workspace.js +28 -15
  79. package/dist/regular-file.d.ts.map +1 -1
  80. package/dist/regular-file.js +52 -8
  81. package/dist/replace-directory.d.ts.map +1 -1
  82. package/dist/replace-directory.js +5 -4
  83. package/dist/replace-file.d.ts +1 -1
  84. package/dist/replace-file.d.ts.map +1 -1
  85. package/dist/replace-file.js +93 -31
  86. package/dist/root-context.d.ts +23 -0
  87. package/dist/root-context.d.ts.map +1 -0
  88. package/dist/root-context.js +62 -0
  89. package/dist/root-errors.d.ts +4 -0
  90. package/dist/root-errors.d.ts.map +1 -0
  91. package/dist/root-errors.js +21 -0
  92. package/dist/root-impl.d.ts +108 -0
  93. package/dist/root-impl.d.ts.map +1 -0
  94. package/dist/{safe-root.js → root-impl.js} +484 -265
  95. package/dist/root.d.ts +1 -107
  96. package/dist/root.d.ts.map +1 -1
  97. package/dist/root.js +1 -1332
  98. package/dist/safe-path-segment.d.ts +9 -0
  99. package/dist/safe-path-segment.d.ts.map +1 -0
  100. package/dist/safe-path-segment.js +51 -0
  101. package/dist/secret-file.d.ts.map +1 -1
  102. package/dist/secret-file.js +6 -1
  103. package/dist/sibling-temp.d.ts +1 -0
  104. package/dist/sibling-temp.d.ts.map +1 -1
  105. package/dist/sibling-temp.js +32 -7
  106. package/dist/sidecar-lock.d.ts +2 -0
  107. package/dist/sidecar-lock.d.ts.map +1 -1
  108. package/dist/sidecar-lock.js +93 -13
  109. package/dist/store.d.ts +1 -0
  110. package/dist/store.d.ts.map +1 -1
  111. package/dist/store.js +1 -0
  112. package/dist/temp-target.d.ts.map +1 -1
  113. package/dist/temp-target.js +7 -4
  114. package/dist/test-hooks.d.ts +6 -0
  115. package/dist/test-hooks.d.ts.map +1 -1
  116. package/dist/text-atomic.d.ts +7 -0
  117. package/dist/text-atomic.d.ts.map +1 -1
  118. package/dist/text-atomic.js +3 -2
  119. package/dist/trash.d.ts.map +1 -1
  120. package/dist/trash.js +48 -13
  121. package/docs/advanced.md +110 -0
  122. package/docs/archive.md +199 -0
  123. package/docs/atomic.md +191 -0
  124. package/docs/config.md +94 -0
  125. package/docs/contributing.md +80 -0
  126. package/docs/errors.md +154 -0
  127. package/docs/file-store.md +213 -0
  128. package/docs/filename.md +0 -0
  129. package/docs/index.md +77 -0
  130. package/docs/install-path.md +167 -0
  131. package/docs/install.md +150 -0
  132. package/docs/json-store.md +204 -0
  133. package/docs/json.md +192 -0
  134. package/docs/local-roots.md +164 -0
  135. package/docs/output.md +92 -0
  136. package/docs/path-scope.md +130 -0
  137. package/docs/path.md +191 -0
  138. package/docs/permissions.md +95 -0
  139. package/docs/private-file-store.md +45 -0
  140. package/docs/python-helper.md +103 -0
  141. package/docs/quickstart.md +147 -0
  142. package/docs/reading.md +174 -0
  143. package/docs/regular-file.md +175 -0
  144. package/docs/root.md +205 -0
  145. package/docs/secret-file.md +154 -0
  146. package/docs/secure-file.md +82 -0
  147. package/docs/security-model.md +99 -0
  148. package/docs/sidecar-lock.md +230 -0
  149. package/docs/store.md +77 -0
  150. package/docs/temp.md +284 -0
  151. package/docs/test-hooks.md +74 -0
  152. package/docs/testing.md +178 -0
  153. package/docs/timing.md +121 -0
  154. package/docs/types.md +155 -0
  155. package/docs/walk.md +63 -0
  156. package/docs/writing.md +222 -0
  157. package/package.json +10 -2
  158. package/dist/archive-utils.d.ts +0 -3
  159. package/dist/archive-utils.d.ts.map +0 -1
  160. package/dist/archive-utils.js +0 -26
  161. package/dist/boundary-file-read.d.ts +0 -44
  162. package/dist/boundary-file-read.d.ts.map +0 -1
  163. package/dist/boundary-file-read.js +0 -129
  164. package/dist/boundary-path.d.ts +0 -39
  165. package/dist/boundary-path.d.ts.map +0 -1
  166. package/dist/boundary-path.js +0 -598
  167. package/dist/fs-pinned-path-helper.d.ts +0 -7
  168. package/dist/fs-pinned-path-helper.d.ts.map +0 -1
  169. package/dist/fs-pinned-path-helper.js +0 -182
  170. package/dist/fs-pinned-write-helper.d.ts +0 -21
  171. package/dist/fs-pinned-write-helper.d.ts.map +0 -1
  172. package/dist/fs-pinned-write-helper.js +0 -263
  173. package/dist/hardlink-guards.d.ts +0 -7
  174. package/dist/hardlink-guards.d.ts.map +0 -1
  175. package/dist/hardlink-guards.js +0 -30
  176. package/dist/install-safe-path.d.ts +0 -20
  177. package/dist/install-safe-path.d.ts.map +0 -1
  178. package/dist/install-safe-path.js +0 -94
  179. package/dist/json-file.d.ts +0 -3
  180. package/dist/json-file.d.ts.map +0 -1
  181. package/dist/json-file.js +0 -123
  182. package/dist/json-files.d.ts +0 -20
  183. package/dist/json-files.d.ts.map +0 -1
  184. package/dist/json-files.js +0 -153
  185. package/dist/path-alias-guards.d.ts +0 -19
  186. package/dist/path-alias-guards.d.ts.map +0 -1
  187. package/dist/path-alias-guards.js +0 -21
  188. package/dist/path-guards.d.ts +0 -7
  189. package/dist/path-guards.d.ts.map +0 -1
  190. package/dist/path-guards.js +0 -49
  191. package/dist/path-safety.d.ts +0 -12
  192. package/dist/path-safety.d.ts.map +0 -1
  193. package/dist/path-safety.js +0 -50
  194. package/dist/safe-open-sync.d.ts +0 -24
  195. package/dist/safe-open-sync.d.ts.map +0 -1
  196. package/dist/safe-open-sync.js +0 -71
  197. package/dist/safe-root.d.ts +0 -123
  198. package/dist/safe-root.d.ts.map +0 -1
  199. package/dist/secure-temp-workspace.d.ts +0 -25
  200. package/dist/secure-temp-workspace.d.ts.map +0 -1
  201. package/dist/secure-temp-workspace.js +0 -136
  202. package/dist/sibling-temp-file.d.ts +0 -16
  203. package/dist/sibling-temp-file.d.ts.map +0 -1
  204. package/dist/sibling-temp-file.js +0 -73
  205. package/dist/sibling-temp-write.d.ts +0 -8
  206. package/dist/sibling-temp-write.d.ts.map +0 -1
  207. package/dist/sibling-temp-write.js +0 -40
@@ -0,0 +1,154 @@
1
+ # Secret files
2
+
3
+ Helpers for reading and writing credentials. Files are written at mode `0o600`, dirs at `0o700`, with a maximum read size to avoid OOM on bogus input.
4
+
5
+ ```ts
6
+ import {
7
+ readSecretFileSync,
8
+ tryReadSecretFileSync,
9
+ writeSecretFileAtomic,
10
+ DEFAULT_SECRET_FILE_MAX_BYTES,
11
+ PRIVATE_SECRET_DIR_MODE,
12
+ PRIVATE_SECRET_FILE_MODE,
13
+ } from "@openclaw/fs-safe/secret";
14
+ ```
15
+
16
+ ## When to use these vs `writeJson`
17
+
18
+ | Use these when | Use `writeJson` when |
19
+ |---|---|
20
+ | The file is a credential (token, key, password). | The file is application state. |
21
+ | You want the parent directory created at `0o700` if missing. | You don't care about the parent directory mode. |
22
+ | You want a hard size cap on reads (to defend against bogus input). | You're reading bounded JSON state. |
23
+ | Mode `0o600` is mandatory, not just nice. | Mode is whatever umask gives you. |
24
+
25
+ ## Constants
26
+
27
+ ```ts
28
+ DEFAULT_SECRET_FILE_MAX_BYTES = 16 * 1024; // 16 KiB
29
+ PRIVATE_SECRET_DIR_MODE = 0o700;
30
+ PRIVATE_SECRET_FILE_MODE = 0o600;
31
+ ```
32
+
33
+ The 16 KiB cap is intentionally aggressive — credentials should be small. If you need bigger, pass `maxBytes` explicitly.
34
+
35
+ ## Reading
36
+
37
+ ### `tryReadSecretFileSync(filePath, label, options?)`
38
+
39
+ The lenient reader. Returns the trimmed secret string, or `undefined` when the path is missing, empty, unreadable, too large, or rejected by the validation checks.
40
+
41
+ ```ts
42
+ import { tryReadSecretFileSync } from "@openclaw/fs-safe/secret";
43
+
44
+ const token = tryReadSecretFileSync("/var/lib/app/auth.token", "auth token");
45
+ if (token) {
46
+ useToken(token);
47
+ } else {
48
+ await reauthenticate();
49
+ }
50
+ ```
51
+
52
+ ### `readSecretFileSync(filePath, label, options?)`
53
+
54
+ Strict reader. Throws `FsSafeError` when the file is missing, too large, empty, unreadable, or rejected by the validation checks. Use when failing loudly is the right call:
55
+
56
+ ```ts
57
+ const token = readSecretFileSync("/var/lib/app/auth.token");
58
+ ```
59
+
60
+ ### Read options
61
+
62
+ ```ts
63
+ type SecretFileReadOptions = {
64
+ maxBytes?: number; // default DEFAULT_SECRET_FILE_MAX_BYTES (16 KiB)
65
+ rejectSymlink?: boolean;
66
+ };
67
+ ```
68
+
69
+ The reader trims the file content and rejects empty results. `rejectSymlink` blocks a symlink path before the pinned read.
70
+
71
+ ## Writing
72
+
73
+ ### `writeSecretFileAtomic(params)`
74
+
75
+ Async. Creates the parent directory at `dirMode` (default `0o700`) if missing, writes content to a sibling temp file at `mode` (default `0o600`), atomically renames over the destination, and re-asserts the file mode after rename.
76
+
77
+ ```ts
78
+ import { writeSecretFileAtomic } from "@openclaw/fs-safe/secret";
79
+
80
+ await writeSecretFileAtomic({
81
+ rootDir: "/var/lib/app",
82
+ filePath: "/var/lib/app/auth.token",
83
+ content: token,
84
+ });
85
+ ```
86
+
87
+ ### Parameters
88
+
89
+ ```ts
90
+ type WriteSecretFileParams = {
91
+ rootDir: string; // trusted root directory (created at dirMode if missing)
92
+ filePath: string; // absolute path; must be inside rootDir
93
+ content: string | Uint8Array;
94
+ mode?: number; // file mode for the new file (default PRIVATE_SECRET_FILE_MODE = 0o600)
95
+ dirMode?: number; // mode for the root and intermediate dirs (default PRIVATE_SECRET_DIR_MODE = 0o700)
96
+ };
97
+ ```
98
+
99
+ The directory mode is asserted on each component along the path: `rootDir`, then any intermediate dirs, then the parent. The helper enforces that every component matches `dirMode` — wider permissions on an existing directory cause the write to fail. Audit and tighten existing secret directories yourself.
100
+
101
+ For more permissive credentials, override `mode`:
102
+
103
+ ```ts
104
+ await writeSecretFileAtomic({
105
+ rootDir: "/var/lib/app",
106
+ filePath: "/var/lib/app/readonly.token",
107
+ content: token,
108
+ mode: 0o400, // tighter than the default
109
+ });
110
+ ```
111
+
112
+ ## Common patterns
113
+
114
+ ### Load on boot, reauthenticate on miss
115
+
116
+ ```ts
117
+ const token = tryReadSecretFileSync("/var/lib/app/auth.token", "auth token");
118
+ if (!token) await runOauthFlow();
119
+ ```
120
+
121
+ ### Refresh and persist a token
122
+
123
+ ```ts
124
+ const fresh = await refreshToken(currentRefresh);
125
+ await writeSecretFileAtomic({
126
+ rootDir: "/var/lib/app",
127
+ filePath: "/var/lib/app/auth.token",
128
+ content: JSON.stringify(fresh),
129
+ });
130
+ ```
131
+
132
+ ### Compose with `withTimeout`
133
+
134
+ ```ts
135
+ import { withTimeout } from "@openclaw/fs-safe/advanced";
136
+
137
+ await withTimeout(
138
+ writeSecretFileAtomic({ rootDir, filePath, content }),
139
+ 5_000,
140
+ "persist auth token",
141
+ );
142
+ ```
143
+
144
+ ## Threat model notes
145
+
146
+ - These helpers protect the secret file from **other processes with the same UID** that respect filesystem permissions. They do not defend against root or against attackers who can read process memory.
147
+ - Validation failures are tripwires, not authorization. Investigate before clearing a rejected credential file.
148
+ - If the destination directory is on a tmpfs that does not honor mode bits, the helpers will set the mode bits but the OS may ignore them. Audit your platform.
149
+
150
+ ## See also
151
+
152
+ - [JSON files](json.md) — `writeJson` accepts `mode: 0o600` for non-secret JSON state.
153
+ - [Atomic writes](atomic.md) — the lower-level `replaceFileAtomic` used by these helpers.
154
+ - [Private file-store mode](private-file-store.md) — root-bounded JSON+text stores using secret-file write policy.
@@ -0,0 +1,82 @@
1
+ # Secure file reads
2
+
3
+ `readSecureFile()` is for absolute file paths that should be treated like credentials or other sensitive local inputs. It is stricter than `fs.readFile()` and different from `root().read()`: the file path is absolute, but the read is still fd-pinned and permission-checked before bytes are returned.
4
+
5
+ ```ts
6
+ import { readSecureFile } from "@openclaw/fs-safe/secure-file";
7
+
8
+ const { buffer, realPath, permissions } = await readSecureFile({
9
+ filePath: "/var/lib/app/auth.token",
10
+ label: "auth token",
11
+ trust: { trustedDirs: ["/var/lib/app"] },
12
+ io: { maxBytes: 16 * 1024, timeoutMs: 5_000 },
13
+ });
14
+ ```
15
+
16
+ ## Checks
17
+
18
+ The helper:
19
+
20
+ - requires a local absolute path and rejects UNC/network paths by default
21
+ - rejects directories and, by default, symlink paths
22
+ - opens the file before reading and verifies the opened fd still matches the path and realpath
23
+ - optionally requires the real path to live under one of `trust.trustedDirs`
24
+ - rejects hard-to-verify or unsafe permissions unless `permissions.allowInsecure` is set
25
+ - rejects files owned by another POSIX uid
26
+ - enforces `maxBytes` before and after reading
27
+ - closes the handle on success, error, and timeout
28
+
29
+ On POSIX, unsafe permissions mean group/world writable, and group/world readable unless `permissions.allowReadableByOthers` is true. On Windows, the helper uses the ACL inspection helpers from [`permissions`](permissions.md) and refuses the read if ACLs cannot be verified.
30
+
31
+ ## Options
32
+
33
+ ```ts
34
+ type SecureFileReadOptions = {
35
+ filePath: string;
36
+ label?: string;
37
+ trust?: {
38
+ trustedDirs?: string[];
39
+ allowSymlink?: boolean;
40
+ allowNetworkPath?: boolean;
41
+ };
42
+ permissions?: {
43
+ allowInsecure?: boolean;
44
+ allowReadableByOthers?: boolean;
45
+ };
46
+ inject?: {
47
+ platform?: NodeJS.Platform;
48
+ env?: NodeJS.ProcessEnv;
49
+ exec?: PermissionExec;
50
+ };
51
+ io?: {
52
+ maxBytes?: number;
53
+ timeoutMs?: number;
54
+ };
55
+ };
56
+ ```
57
+
58
+ `permissions.allowInsecure` is a migration escape hatch. Prefer fixing permissions and using [`formatPermissionRemediation`](permissions.md) to show the user what to run. `trust.allowNetworkPath` is off by default because UNC paths are remote authority, not local filesystem input. `inject` is for tests and platform adapters; production callers usually leave it unset.
59
+
60
+ ## Errors
61
+
62
+ `readSecureFile()` throws `FsSafeError` with codes such as:
63
+
64
+ | Code | Meaning |
65
+ |---|---|
66
+ | `invalid-path` | `filePath` was not a local absolute path. |
67
+ | `not-found` | The path could not be stat'd before open. |
68
+ | `not-file` | The opened target is not a regular file. |
69
+ | `symlink` | The path is a symlink and `trust.allowSymlink` is false. |
70
+ | `path-mismatch` | The path or realpath changed between open and verification. |
71
+ | `outside-workspace` | `realPath` is outside `trust.trustedDirs`. |
72
+ | `permission-unverified` | Required mode/ACL checks could not be completed. |
73
+ | `insecure-permissions` | Mode bits or ACLs grant broader access than allowed. |
74
+ | `not-owned` | POSIX owner uid is not the current process uid. |
75
+ | `too-large` | File size or bytes read exceeded `maxBytes`. |
76
+ | `timeout` | `timeoutMs` elapsed while reading. |
77
+
78
+ ## See also
79
+
80
+ - [Permissions](permissions.md) — standalone POSIX mode and Windows ACL checks.
81
+ - [Secret files](secret-file.md) — mode-0600 credential read/write helpers.
82
+ - [Reading](reading.md) — root-bounded relative reads.
@@ -0,0 +1,99 @@
1
+ # Security model
2
+
3
+ `fs-safe` is a library-level guardrail: a capability-style root handle for Node.js code that handles untrusted relative paths. It assumes the calling process already has whatever filesystem permissions it needs and aims to stop trivial path tricks from broadening that authority. It is not a sandbox and does not replace operating-system isolation.
4
+
5
+ The same shape exists in other languages: Go's [`os.Root` / `OpenInRoot`](https://go.dev/blog/osroot) and Rust's [`cap-std`](https://github.com/bytecodealliance/cap-std) both expose a root handle whose operations refuse to escape it. `fs-safe` is the Node-side equivalent: a single `root()` capability that carries the boundary across every read, write, move, and remove, instead of leaving each call site to redo `path.resolve(...).startsWith(...)` and hope.
6
+
7
+ ## Threat model
8
+
9
+ You hand a `root()` boundary to a piece of code that takes caller-controlled relative paths. The library defends against a caller that:
10
+
11
+ - supplies `..` traversal segments to escape the boundary
12
+ - supplies an absolute path where a relative one is expected
13
+ - replaces a path component with a symlink between check and use (TOCTOU)
14
+ - replaces the destination directory with a symlink right before a write
15
+ - creates a hardlink that aliases an out-of-tree inode and asks you to read or replace it
16
+ - triggers a partial write that leaves a half-written file at the destination
17
+ - ships an archive with `..` paths, absolute paths, or symlinks pointing outside the destination
18
+
19
+ It does **not** defend against:
20
+
21
+ - a process running with permissions to write anywhere on the filesystem and choosing to ignore the library
22
+ - another process with the same UID racing to mutate the same directory between two separate `fs-safe` calls — the boundary is per-call, not per-session
23
+ - traversal across filesystem boundaries, bind mounts, device files, `/proc`-style virtual filesystems, or any other path your process can normally access from inside the root
24
+ - container escape, TOCTOU between fork and exec of helpers, or kernel-level vulnerabilities
25
+ - semantic content checks: file types, archive payload schemas, signature verification
26
+
27
+ If you need full sandboxing, run the worker under reduced privileges (uid, container, seccomp, chroot, jail) and use `fs-safe` inside the sandbox to keep the worker honest about its own workspace.
28
+
29
+ ## Defenses, by failure mode
30
+
31
+ ### Path traversal and absolute paths
32
+
33
+ Every relative path is resolved against the canonicalized real path of the root, then checked with `isPathInside`. Inputs containing `..`, leading `/` (without `pathScope` opt-in), or that resolve outside the root throw `outside-workspace`.
34
+
35
+ ### Symlinks (read side)
36
+
37
+ `open()` and `read()` use `fs.open` with `O_NOFOLLOW` on POSIX where available. The library then `fstat`s the open fd and `realpath`s the original input, asserting the two refer to the same inode (`sameFileIdentity`). A symlink swap that happens between resolve and open will fail at the identity check rather than silently following the new target.
38
+
39
+ Per-call `symlinks: "follow-within-root"` allows symlinks whose final target is still inside the root. The default is `"reject"`.
40
+
41
+ ### Symlinks (write side)
42
+
43
+ Writes use a sibling-temp + rename helper that opens the parent directory by fd, then performs the rename `at` the parent fd. Replacing the parent directory with a symlink between the parent-fd open and the rename does not divert the write.
44
+
45
+ ### Hardlink aliasing
46
+
47
+ When `hardlinks: "reject"` is set, reads stat the target and refuse if `nlink > 1`, on the conservative assumption that a hardlinked file might alias an out-of-tree inode. This is defense-in-depth: the link count check is best-effort and platform-dependent. Treat it as a tripwire, not authorization.
48
+
49
+ ### TOCTOU between resolve and use
50
+
51
+ `resolve()`, `exists()`, `stat()`, and `list()` are explicitly **not** race-resistant — they answer a question and return. To act on a path with race resistance, use `read()`, `open()`, `write()`, `create()`, `copyIn()`, `move()`, or `remove()`. They re-pin the path identity at the point of use.
52
+
53
+ ### Atomic writes
54
+
55
+ `replaceFileAtomic` writes to a sibling temp file in the destination directory, optionally `fsync`s it, optionally `fsync`s the parent directory after rename, and atomically renames over the destination. On failure mid-write, the destination is either the old contents (rename never happened) or the new contents (rename succeeded). There is no half-written intermediate state visible at the destination path.
56
+
57
+ Within one process, async writes to the same target are queued so their temp-write/rename phases do not overlap. Cross-process writers still need an external protocol such as the sidecar lock helpers.
58
+
59
+ ### Archive extraction
60
+
61
+ `extractArchive` first stages into a private temp directory (mode 0700) outside the destination, validates each entry path against `..` and absolute prefixes, refuses link-type entries by default, enforces entry count and byte budgets, and only then merges the staged tree into the destination through the same boundary checks used by direct writes.
62
+
63
+ ## What "library-level" means
64
+
65
+ A library cannot revoke its own caller's authority. If your code chooses to bypass `fs-safe` and call `fs.writeFile` directly with the same path, you bypass the defenses too. The contract `fs-safe` enforces is: *every filesystem operation that touches caller-controlled input goes through the boundary*. That contract is yours to keep.
66
+
67
+ The library does not modify or constrain the global Node.js `fs` namespace, and it does not patch the runtime. Other code in the same process retains its normal filesystem authority.
68
+
69
+ ## Platform notes
70
+
71
+ - **POSIX (Linux, macOS):** Best-defended path. Uses `O_NOFOLLOW`, fd identity checks, and one persistent Python helper process for fd-relative `unlinkat` / `mkdirat` / `renameat` / parent-fd write operations. Configure `FS_SAFE_PYTHON_MODE=require` when helper startup must fail closed, or `off` when you need a no-Python runtime. See [Python helper policy](python-helper.md).
72
+ - **Windows:** Falls back to the safest Node-level behavior available. `O_NOFOLLOW` is not honored. Some fd-relative POSIX hardening is unavailable. The library does the path canonicalization, identity, and atomic-rename checks it can.
73
+
74
+ The library does not advertise different security guarantees per platform — it advertises the same surface and relies on the strongest mechanism the platform offers.
75
+
76
+ ## Limitations to keep in mind
77
+
78
+ | Limitation | What it means |
79
+ |---|---|
80
+ | Not ambient authority removal | Code that can import `node:fs` can still bypass the handle. Keep caller-controlled path operations behind `root()` by convention, review, and tests. |
81
+ | Absolute paths are escape hatches | APIs that accept or return absolute paths exist for audit, ingest, and advanced composition. Prefer root-relative names in normal application flow. |
82
+ | Not a mount/device boundary | `root()` keeps path traversal inside the directory tree; it does not make device files, bind mounts, or virtual filesystems safe to expose. |
83
+ | Per-call, not per-session | Another process with the same privileges can still mutate the tree between two separate calls. Use one verb method for the operation you need to make race-resistant. |
84
+ | Hardlink rejection is best-effort | Link-count checks depend on platform metadata. Treat `hardlinks: "reject"` as a tripwire, not an authorization primitive. |
85
+ | Mode bits are not a full policy engine | `replaceFileAtomic` and secret-file helpers set requested modes, but you should still set umask and inspect modes when policy requires it. |
86
+ | Archive extraction is path safety, not content safety | Unsafe entry paths and links are rejected; malicious payload contents remain your application layer's problem. |
87
+ | Helper failures degrade fd-relative hardening | `helper-unavailable` falls back in `auto` mode and fails closed in `require` mode. Atomicity and identity checks remain, but parent-directory swaps between validation and mutation are less tightly pinned without the helper. |
88
+
89
+ ## Recommended deployment shape
90
+
91
+ - Run worker code under a dedicated UID with the smallest filesystem privileges that still allow the worker to do its job.
92
+ - Mount the workspace directory writable; mount everything else read-only or not at all.
93
+ - Use `fs-safe`'s `root()` for that workspace.
94
+ - For credentials, use [secret files](secret-file.md) (mode 0600 in mode-0700 dirs) rather than the workspace.
95
+ - For scratch space, use a [private temp workspace](temp.md) — don't reuse the workspace root.
96
+
97
+ ## Reporting issues
98
+
99
+ Suspected security issues belong in private disclosure first. See [`SECURITY.md`](https://github.com/openclaw/fs-safe/blob/main/SECURITY.md) in the repo for the current contact path.
@@ -0,0 +1,230 @@
1
+ # File lock
2
+
3
+ `acquireFileLock()` and `withFileLock()` provide a cross-process file lock with retry and process-exit cleanup. The lock is implemented as a sidecar file (e.g. `state.json` ↔ `state.json.lock`) — only one acquirer can create the sidecar with `O_CREAT | O_EXCL` at a time.
4
+
5
+ ```ts
6
+ import { acquireFileLock } from "@openclaw/fs-safe/file-lock";
7
+
8
+ const handle = await acquireFileLock("/var/lib/app/state.json", {
9
+ managerKey: "snapshot",
10
+ staleMs: 5 * 60_000,
11
+ payload: async () => ({ pid: process.pid, host: os.hostname() }),
12
+ });
13
+ try {
14
+ // ...exclusive work on /var/lib/app/state.json...
15
+ } finally {
16
+ await handle.release();
17
+ }
18
+ ```
19
+
20
+ ## Why sidecar?
21
+
22
+ The lock file sits next to the protected resource. If a process crashes mid-lock, the next acquirer notices the held entry, inspects its payload (PID, host, acquired-at timestamp), and decides — via `shouldReclaim` (defaulting to "is the lock older than `staleMs`?") — whether it should keep waiting or fail.
23
+
24
+ The library installs a `process.on("exit")` handler that releases all currently-held locks synchronously, so well-behaved exits leave no stale sidecars. Crashed holders leave their sidecar behind; remove those through an application-owned recovery path after you have proved the holder cannot still be writing.
25
+
26
+ ## API
27
+
28
+ ```ts
29
+ function acquireFileLock<TPayload>(
30
+ targetPath: string,
31
+ options: FileLockAcquireOptions<TPayload>,
32
+ ): Promise<FileLockHandle>;
33
+
34
+ function withFileLock<T, TPayload>(
35
+ targetPath: string,
36
+ options: FileLockAcquireOptions<TPayload>,
37
+ fn: () => Promise<T>,
38
+ ): Promise<T>;
39
+
40
+ function createFileLockManager(key: string): FileLockManager;
41
+ ```
42
+
43
+ `managerKey` is an optional identifier used to keep state isolated across multiple lock domains in the same process. Use distinct keys for distinct domains (`"snapshot"`, `"compact"`, `"build"`). If omitted, fs-safe derives one from the target path.
44
+
45
+ ## Acquire options
46
+
47
+ ```ts
48
+ type FileLockAcquireOptions<TPayload extends Record<string, unknown>> = {
49
+ managerKey?: string; // optional in-process manager namespace
50
+ lockPath?: string; // override; defaults to `${targetPath}.lock`
51
+ staleMs?: number; // default 30_000
52
+ timeoutMs?: number; // overall acquire deadline; default unbounded
53
+ retry?: FileLockRetryOptions;
54
+ staleRecovery?: "fail-closed"; // default
55
+ allowReentrant?: boolean; // if this process already holds it, increment a count instead of failing
56
+ payload: () => TPayload | Promise<TPayload>;
57
+ shouldReclaim?: (params: {
58
+ lockPath: string;
59
+ normalizedTargetPath: string;
60
+ payload: Record<string, unknown> | null;
61
+ staleMs: number;
62
+ nowMs: number;
63
+ heldByThisProcess: boolean;
64
+ }) => boolean | Promise<boolean>;
65
+ metadata?: Record<string, unknown>; // attached to heldEntries() output for diagnostics
66
+ };
67
+
68
+ type FileLockRetryOptions = {
69
+ retries?: number; // number of retry attempts after the first failure
70
+ factor?: number; // exponential backoff factor (default 2)
71
+ minTimeout?: number; // initial delay (ms)
72
+ maxTimeout?: number; // delay cap (ms)
73
+ randomize?: boolean; // jitter
74
+ };
75
+ ```
76
+
77
+ `payload` is a function so you can re-evaluate it on each retry (e.g. timestamp, PID).
78
+
79
+ ## Release handle
80
+
81
+ ```ts
82
+ type FileLockHandle = {
83
+ lockPath: string;
84
+ normalizedTargetPath: string;
85
+ release: () => Promise<void>;
86
+ [Symbol.asyncDispose](): Promise<void>;
87
+ };
88
+ ```
89
+
90
+ Always release in a `finally`:
91
+
92
+ ```ts
93
+ const handle = await acquireFileLock(targetPath, {
94
+ staleMs: 60_000,
95
+ payload: () => ({ pid: process.pid }),
96
+ });
97
+ try {
98
+ await doExclusiveWork();
99
+ } finally {
100
+ await handle.release();
101
+ }
102
+ ```
103
+
104
+ If your process dies before `release()` runs and skips the exit handler, the sidecar remains. Once `staleMs` elapses (or your `shouldReclaim` returns true), acquisition fails closed instead of deleting by path, because Node cannot atomically bind that deletion to the file that was inspected.
105
+
106
+ ## `withFileLock` — common shape made one-liner
107
+
108
+ ```ts
109
+ const result = await withFileLock(
110
+ "/var/lib/app/state.json",
111
+ {
112
+ managerKey: "compact",
113
+ staleMs: 30_000,
114
+ payload: () => ({ pid: process.pid, what: "compact" }),
115
+ },
116
+ async () => {
117
+ return await runCompaction();
118
+ },
119
+ );
120
+ ```
121
+
122
+ Acquires, runs `fn`, releases regardless of success/failure. Returns the result of `fn`.
123
+
124
+ ## Long-lived managers
125
+
126
+ Most callers should use `acquireFileLock()` or `withFileLock()`. Use `createFileLockManager(key)` only when a long-lived service needs diagnostics or lifecycle control over locks it currently holds:
127
+
128
+ ```ts
129
+ const locks = createFileLockManager("session-writes");
130
+ const handle = await locks.acquire(sessionPath, {
131
+ staleMs: 60_000,
132
+ payload: () => ({ pid: process.pid }),
133
+ });
134
+
135
+ for (const held of locks.heldEntries()) {
136
+ console.log(held.lockPath, held.acquiredAt);
137
+ }
138
+
139
+ await handle.release();
140
+ await locks.drain();
141
+ ```
142
+
143
+ ## Stale policy: `shouldReclaim`
144
+
145
+ The default policy treats locks whose `createdAt` is older than `staleMs` as stale. Pass a custom callback when you want a richer notion of "is the holder still alive":
146
+
147
+ ```ts
148
+ import { kill } from "node:process";
149
+
150
+ const handle = await acquireFileLock(targetPath, {
151
+ staleMs: 60_000,
152
+ payload: () => ({ pid: process.pid }),
153
+ shouldReclaim: ({ payload, nowMs, staleMs }) => {
154
+ if (!payload) return true;
155
+ const pid = Number(payload.pid);
156
+ if (!Number.isFinite(pid)) return true;
157
+ try {
158
+ kill(pid, 0);
159
+ return false; // process still alive — keep waiting
160
+ } catch {
161
+ return true; // process gone — fail closed for recovery
162
+ }
163
+ },
164
+ });
165
+ ```
166
+
167
+ `heldByThisProcess` is true when this manager already holds the lock (relevant for the reentrant case). A `true` result does not delete the sidecar; it lets the acquire loop stop waiting once the retry/timeout policy says to give up.
168
+
169
+ ## What sidecar locks defend against
170
+
171
+ - **Two processes writing the same file at once.** `acquire` serializes the critical section.
172
+ - **Accidentally deleting a fresh lock during stale recovery.** Stale third-party locks fail closed because safe compare-and-unlink is not available through Node's path APIs.
173
+ - **Race between simultaneous acquire attempts.** `O_CREAT | O_EXCL` ensures one wins.
174
+
175
+ ## What they do **not** defend against
176
+
177
+ - **Misbehaving holders that ignore the lock.** Locks are advisory — only callers that go through `acquire` are bound.
178
+ - **Automatic stale lock deletion.** If a process crashes, use the payload and your own supervisor/process table to decide when to remove the sidecar.
179
+ - **Multi-host coordination over network filesystems.** Behavior depends on the underlying filesystem's `O_EXCL` semantics; treat as best-effort.
180
+
181
+ ## Common patterns
182
+
183
+ ### Compact under lock
184
+
185
+ ```ts
186
+ await withFileLock(
187
+ "/var/lib/app/db.sqlite",
188
+ {
189
+ staleMs: 30_000,
190
+ payload: () => ({ pid: process.pid, what: "compact" }),
191
+ },
192
+ async () => {
193
+ await runCompaction();
194
+ },
195
+ );
196
+ ```
197
+
198
+ ### Try once, give up if held
199
+
200
+ ```ts
201
+ try {
202
+ await withFileLock(
203
+ targetPath,
204
+ { staleMs: 30_000, retry: { retries: 0 }, payload: () => ({ pid: process.pid }) },
205
+ async () => await work(),
206
+ );
207
+ } catch (err) {
208
+ console.log("another process is doing this; skipping");
209
+ }
210
+ ```
211
+
212
+ ### Wait politely with backoff
213
+
214
+ ```ts
215
+ await withFileLock(
216
+ targetPath,
217
+ {
218
+ staleMs: 60_000,
219
+ timeoutMs: 30_000,
220
+ retry: { retries: 30, minTimeout: 100, maxTimeout: 5_000, factor: 1.7, randomize: true },
221
+ payload: () => ({ pid: process.pid }),
222
+ },
223
+ async () => await work(),
224
+ );
225
+ ```
226
+
227
+ ## See also
228
+
229
+ - [Atomic writes](atomic.md) — single-writer atomicity that often replaces the need for a lock entirely.
230
+ - `createAsyncLock` from `@openclaw/fs-safe/advanced` — in-process serialization for a single Node process.
package/docs/store.md ADDED
@@ -0,0 +1,77 @@
1
+ ---
2
+ title: Store
3
+ description: "Overview of @openclaw/fs-safe/store: fileStore, fileStoreSync, and jsonStore."
4
+ ---
5
+
6
+ # `@openclaw/fs-safe/store`
7
+
8
+ The `store` subpath bundles two managed wrappers around the same safe-write primitives `root()` uses:
9
+
10
+ ```ts
11
+ import {
12
+ ensureJsonDurableQueueDirs,
13
+ fileStore,
14
+ fileStoreSync,
15
+ jsonStore,
16
+ loadPendingJsonDurableQueueEntries,
17
+ resolveJsonDurableQueueEntryPaths,
18
+ writeJsonDurableQueueEntry,
19
+ type FileStore,
20
+ type FileStoreOptions,
21
+ type FileStoreSync,
22
+ type JsonStore,
23
+ type JsonStoreOptions,
24
+ } from "@openclaw/fs-safe/store";
25
+ ```
26
+
27
+ | Helper | Use it for |
28
+ |---|---|
29
+ | [`fileStore()`](file-store.md) | Multi-file directories with safe relative paths, size limits, atomic replacement, stream writes, copy-in, and TTL-based pruning. |
30
+ | `fileStoreSync()` | Synchronous variant of `fileStore()` for places that genuinely cannot await. |
31
+ | [`jsonStore()`](json-store.md) | A single keyed JSON state file with explicit fallback, atomic writes, and optional sidecar locking around read-modify-write updates. |
32
+ | Durable JSON queue helpers | Append/load/ack JSON entry files using atomic writes and delivered markers. |
33
+ | [Private file-store mode](private-file-store.md) | `fileStore({ private: true })` for credentials, tokens, and per-agent state at `0600` files under `0700` directories. |
34
+
35
+ `fileStore().json("rel.json")` and `jsonStore({ filePath })` are intentionally separate primitives. Use `fileStore().json(...)` when JSON state lives alongside other files in the same managed directory; use `jsonStore({ filePath })` when you have a single absolute path and want the keyed JSON shape directly.
36
+
37
+ ## Picking a shape
38
+
39
+ - **Multi-file directory under one root** — reach for `fileStore()`. It exposes `write`, `writeJson`, `writeText`, `writeStream`, `read*`, `open`, `copyIn`, `remove`, and `pruneExpired` against safe relative paths.
40
+ - **One JSON state file** — reach for `jsonStore({ filePath })`. Its `update()` and `updateOr()` methods cover the merge-into-defaults and read-modify-write cases.
41
+ - **Credentials or tokens** — pass `private: true` to `fileStore()`. Same store shape; writes route through the secret-file atomic path with `0600`/`0700` permissions.
42
+ - **Durable work queues** — use the durable JSON queue helpers when each work item is a standalone JSON file and acknowledgement is represented by moving it through a short-lived `.delivered` marker.
43
+
44
+ ## Durable JSON queues
45
+
46
+ The durable queue helpers are intentionally low-level. They do not decide retry,
47
+ dedupe, or recovery policy; they just provide the filesystem mechanics that
48
+ several queue implementations otherwise rewrite by hand.
49
+
50
+ ```ts
51
+ await ensureJsonDurableQueueDirs({ queueDir, failedDir });
52
+
53
+ const paths = resolveJsonDurableQueueEntryPaths(queueDir, id);
54
+ await writeJsonDurableQueueEntry({
55
+ filePath: paths.jsonPath,
56
+ entry,
57
+ tempPrefix: "queue",
58
+ });
59
+
60
+ const pending = await loadPendingJsonDurableQueueEntries({ queueDir, tempPrefix: "queue" });
61
+ ```
62
+
63
+ `id` must be a single safe path segment: non-empty, not dot-prefixed, and made
64
+ from letters, numbers, `_`, `-`, and `.`. Slashes, backslashes, NUL bytes, `.`,
65
+ and `..` are rejected.
66
+
67
+ Use `ackJsonDurableQueueEntry()` after durable processing succeeds and
68
+ `moveJsonDurableQueueEntryToFailed()` when the caller wants to quarantine an
69
+ entry for inspection.
70
+
71
+ ## Related pages
72
+
73
+ - [`fileStore`](file-store.md) — full API for the multi-file store.
74
+ - [`jsonStore`](json-store.md) — single-file JSON store with locking.
75
+ - [Private file-store mode](private-file-store.md) — credential-shaped variant.
76
+ - [JSON files](json.md) — lower-level `readJson` / `writeJson` helpers.
77
+ - [Atomic writes](atomic.md) — what `fileStore` and `jsonStore` use under the hood.