@openclaw/fs-safe 0.21.0 → 0.21.1

package/docs/testing.md

@@ -1,6 +1,29 @@
 # Testing
 
-## Manual watch stress campaign
+## Coverage gates
+
+The coverage workflow measures `src/**/*.ts` with V8 on Linux, macOS, and
+Windows, then merges their counters before enforcing the thresholds in
+`vitest.config.ts`: lines 94%, statements 92%, functions 95%, and branches 89%.
+`pnpm test:coverage:collect` disables per-OS thresholds so platform-only paths
+are credited by their own OS; `pnpm test:coverage:merge` requires all three
+reports and enforces the combined gate. The DACL batch child entrypoint also
+runs in-process in a unit test so its protocol and budget handling are measured.
+
+The Linux Rust job uses pinned `cargo-llvm-cov` and a pinned nightly compiler
+for line and branch instrumentation. It combines native crate unit tests with
+TS native suites running the instrumented addon, and exports LCOV plus separate
+unit-only and combined summaries. The Rust line threshold is 71%; addon execution
+must also increase covered Rust lines beyond the unit tests. Only native crate sources compiled on Linux are measured;
+macOS/Windows Rust implementations and the archive WASM crate are outside this
+report. Run `pnpm build` followed by `bash scripts/native-coverage.sh` on Linux
+after installing the toolchain versions specified in `coverage.yml`.
+
+Percentages complement behavioral gates: mutation-policy proof, nightly watch
+stress, and platform lanes are equally important. High coverage cannot establish
+root confinement, race safety, event delivery, or bounded resource retirement.
+
+## Watch stress campaign
 
 Build from the exact revision being qualified with `pnpm install --frozen-lockfile`,
 `pnpm native:build`, and `pnpm build`, then run on a disposable machine:
@@ -13,7 +36,8 @@ Individual scenario names are `scale`, `fanout`, `churn`, `lifecycle`,
 `adversarial`, `limits`, `idle`, and `soak`. The runner uses plain Node and no
 additional dependencies. Each scenario prints one JSON result line; progress
 goes to stderr. `all` isolates scenarios in child processes and stops at the
-first failure. This suite is manual and is not part of per-PR CI.
+first failure. The full qualification remains manual. A smaller nightly campaign runs outside
+per-PR CI; it can also be dispatched with `watch-stress.yml`.
 
 Run `node scripts/watch-stress.mjs --scenario oracle-selftest` first to check
 that a poisoned cache fails comparison and can recover only after invalidation.
@@ -33,14 +57,18 @@ differences across 10,000-mutation batches and isolated edit latency measurement
 Lifecycle performs 10,000 ready/close cycles plus admission
 cancellation, close-during-ready, 1,000 scope replacements, and callback-close.
 Adversarial cases exercise Root swaps, outside symlinks, recursive deletion and
-10,000 same-name create/delete pairs. Soak runs 30 minutes, checking the oracle
+10,000 same-name create/delete pairs. Soak runs 60 minutes, checking the oracle
 each minute. Linux needs passwordless `sudo` for the limits scenario; it lowers
 `fs.inotify.max_user_watches` to 1 in a child shell with a restoration trap and
 verifies restoration. Never run that scenario on a shared production host.
 
 RSS limits are fixed before execution: 512 MiB peak for churn/soak, at most
-64 MiB growth after warmup, and at most 32 MiB lifecycle growth after 3,000
-cycles. Reports include samples and fitted slopes. Idle runs for ten minutes
+64 MiB churn growth after warmup, and at most 32 MiB lifecycle growth after 3,000
+cycles. Soak collects garbage twice at every checkpoint, limits collected-heap
+and external-memory growth to 8 MiB after minute five, and requires the fitted
+RSS slope over minutes 31–60 to stay at or below 1 MiB/minute. The runner launches
+soak with `--expose-gc` automatically, including through `--scenario all`.
+Reports include samples and fitted slopes. Idle runs for ten minutes
 with 16 subscriptions and a one-hour reconciliation interval to isolate native
 hub wakeups, requiring less than 1% of one CPU and, on Linux, at most 30 hub
 context switches. macOS captures `ps -M`; Windows captures PowerShell thread
@@ -53,6 +81,38 @@ records native overflow and recovery, but the shared batch does not distinguish
 RDCW kernel-buffer loss from bounded native queue loss; a Windows qualification
 must retain that limitation rather than call it proved kernel overflow.
 
+### Nightly workload
+
+The nightly workflow runs on `ubuntu-latest` (x64), `ubuntu-24.04-arm`,
+`macos-15` (arm64), `macos-15-intel`, and the Windows latest 16-core runner,
+using Node 24 and freshly built native bindings. Each job has a 25-minute
+budget, uploads one JSON result per scenario (including failures), and fails
+if any scenario fails. Remaining scenarios still run after a failure. Only
+Linux runs the kernel-limits scenario automatically.
+
+The same harness accepts these environment overrides; defaults remain the
+full qualification workload. Child processes inherit the settings, which are
+recorded in every JSON result.
+
+| Environment variable | Default | Nightly |
+| --- | ---: | ---: |
+| `FS_SAFE_STRESS_SCALE_DIRECTORIES` (25 files each) | 2,000 | 400 (10,000 files) |
+| `FS_SAFE_STRESS_FANOUT` (maximum subscriptions) | 256 | 64 |
+| `FS_SAFE_STRESS_CHURN_SECONDS` | 300 | 60 |
+| `FS_SAFE_STRESS_LIFECYCLE_CYCLES` | 10,000 | 1,000 |
+| `FS_SAFE_STRESS_IDLE_SECONDS` | 600 | 120 |
+| `FS_SAFE_STRESS_SOAK_MINUTES` (minimum 10) | 60 | 10 |
+
+Soaks shorter than 30 minutes keep the hard peak-RSS, collected-heap, and external-memory
+growth gates, but report the second-half RSS slope and its limit without using it
+to pass or fail (`memory.rssSlopeGated: false`): [#701](https://github.com/openclaw/fs-safe/pull/701)
+found that V8 capacity expansion and allocator retention can raise RSS while live memory stays flat.
+Runs of 30 minutes or longer enforce the same second-half slope limit; only runs of
+at least 60 minutes report `memory.qualification: true`.
+Lifecycle memory is sampled across ten intervals, with
+the first two excluded as warm-up; idle's Linux wakeup bound scales with the
+requested duration (three per minute). Adversarial cases remain unchanged.
+
 ## Linux openat2 fallback
 
 Build the host addon and package first. The test hook is cached with the native
@@ -62,7 +122,7 @@ between tests in one process:
 ```bash
 pnpm native:build
 pnpm build
-FS_SAFE_TEST_NO_OPENAT2=1 FS_SAFE_NATIVE_MODE=require pnpm test test/linux-openat2-fallback.test.ts test/root-move-noreplace.test.ts test/root-move-native-integration.test.ts test/native-write-containment.test.ts
+FS_SAFE_TEST_NO_OPENAT2=1 FS_SAFE_NATIVE_MODE=require pnpm test test/linux-openat2-parity.test.ts test/linux-openat2-fallback.test.ts test/root-move-noreplace.test.ts test/root-move-native-integration.test.ts test/native-write-containment.test.ts
 ```
 
 On Linux, the seccomp harness also exercises the real syscall failure without
@@ -73,13 +133,21 @@ unprivileged seccomp filter; it affects only its child process:
 cc test/fixtures/deny-openat2.c -o /tmp/fs-safe-deny-openat2
 /tmp/fs-safe-deny-openat2 ENOSYS node test/fixtures/linux-openat2-fallback.mjs "$PWD/native/fs-safe-native.linux-x64-gnu.node"
 /tmp/fs-safe-deny-openat2 EPERM node test/fixtures/linux-openat2-fallback.mjs "$PWD/native/fs-safe-native.linux-x64-gnu.node"
+FS_SAFE_TEST_OPENAT2_FILTER=/tmp/fs-safe-deny-openat2 FS_SAFE_NATIVE_MODE=require pnpm test test/linux-openat2-parity.test.ts
 ```
 
 Use the matching native artifact filename on other Linux architectures/libcs.
-The fixture proves nested moves, collisions, read/write, traversal, symlink and
-hardlink rejection, cached selection, and `helper-unavailable` for strict
+The fixtures prove in-root alias operations and policy parity, nested moves,
+collisions, read/write, traversal and escaping-link rejection, hardlink rejection,
+cached selection, and `helper-unavailable` for strict
 bounded cleanup. Bounded-cleanup success tests require real `openat2`; run the
-full suite with the environment hook unset.
+full suite with the environment hook unset. PR CI's `Native check
+(linux-x64-no-openat2)` runs the native Node suites and watch proofs with the
+hook set. It replaces the four bounded-cleanup success suites and quarantine
+success proof with the explicit refusal fixture above. XFS tree-clone proof
+requires the same openat2/NO_XDEV primitive and runs in the ordinary Linux
+lanes. Bun's full compatibility
+suite remains in the normal native lanes, because it includes bounded cleanup.
 
 `@openclaw/fs-safe/test-hooks` exposes test-only injection points. Registration
 is allowed only when `process.env.NODE_ENV === "test"` or
@@ -244,7 +312,8 @@ regression with `MallocStackLogging=1 node --expose-gc scripts/watch-cleanup-lea
 It compares `leaks` results before and after 100 and 1,000 subscription cycles,
 requiring zero growth in leaked allocations. Allocation stacks are saved under
 `.artifacts/watch-cleanup-leaks`. The native macOS CI lane runs this short proof;
-the full stress campaign remains manual.
+the full stress qualification remains manual, with the smaller nightly
+campaign above.
 
 Run the full local gate before handoff:
 
@@ -270,6 +339,59 @@ workspace reads that bypass pinned file descriptors.
 
 `pnpm check` also runs `pnpm lint:file-size`. New source and test files should stay under 500 lines. Existing larger files have explicit budgets in `scripts/check-file-size.mjs`; do not increase those budgets as part of unrelated work.
 
+## Watch memory diagnosis
+
+The original soak rule rejected more than 64 MiB RSS growth after minute five.
+That outcome remains in `memory.legacyRss`, with its original limit and pass/fail
+value; it is no longer the soak pass criterion. Static guarded poll scans alone
+reproduced growth from 72 to 181 MiB RSS while collected heap stayed near 7–8 MiB:
+V8 expanded its almost-empty young generation to 128 MiB, with 103 MiB physically
+committed. A diagnostic run limiting that space ended at 85 MiB RSS with the
+same live heap. The normal harness keeps Node's default nursery sizing.
+
+The 60-minute qualification separates this capacity warm-up from the later RSS
+trend. The measured Linux event run had under 1 MiB collected-heap drift and a
+0.55 MiB/minute second-half RSS slope; its final ten-minute RSS range was 1.60 MiB.
+The 8 MiB live-memory allowance and 1 MiB/minute RSS slope leave measurement
+margin while rejecting retained growth and continued rapid RSS growth. The
+512 MiB peak ceiling is unchanged. Native allocation leaks need independent
+accounting/profiling too: the investigation found a much smaller cleanup-hook
+context leak even when registrations, pending sets, and TSFN counters retired.
+
+Build the native addon and package from the same revision, then run each control
+in a fresh Node process on a disposable machine:
+
+```sh
+node --expose-gc scripts/watch-memory.mjs --arm events --minutes 60 --output .artifacts/events
+node --expose-gc scripts/watch-memory.mjs --arm none --minutes 30 --output .artifacts/none
+node --expose-gc scripts/watch-memory.mjs --arm poll --minutes 30 --output .artifacts/poll
+node --expose-gc scripts/watch-memory.mjs --arm lifecycle --minutes 30 --output .artifacts/lifecycle
+node --expose-gc scripts/watch-memory.mjs --arm steady --minutes 30 --output .artifacts/steady
+```
+
+`events` and `poll` combine low-rate edits, a 10,000-operation burst every fifth
+minute, and subscription cycling. `steady` omits cycling; `lifecycle` omits
+writes. `none` drives the same writer and consumer cache using explicit synthetic
+invalidations, without constructing subscriptions. That arm is an allocation
+control, not evidence of watcher correctness. Every arm checks its consumer
+cache against an independent filesystem walk at each checkpoint.
+
+The diagnostic runner emits JSONL to stdout and `memory.jsonl` in its output
+directory. Each minute includes all five `process.memoryUsage()` fields before
+and after collection, V8 heap-space capacity, Linux `smaps_rollup`, operation and
+guarded-read counts, and live/created/destroyed native watch allocations.
+`--gc none` measures the same workload without forced collection. `--snapshots`
+writes V8 snapshots at minutes 5 and 30; use a separate run because snapshots
+perturb memory usage. On macOS, `--tools` saves `vmmap --summary` and `leaks`
+reports at minutes 5, 30, and 60. `--smoke` is a short calibration, explicitly
+marked in output; it is not a duration-qualified soak.
+
+The native allocation getter is internal and requires `NODE_ENV=test` or
+`VITEST=true`; the runner sets the former. After close, it requires no live
+registrations, pending sets, callback payloads, or thread-safe functions.
+Diagnostic success establishes correctness and retirement; it does not classify
+an RSS curve as a leak or automatically approve a memory budget.
+
 ## See also
 
 - [Security model](security-model.md) — what the boundary is supposed to defend; design tests around the same threats.

package/docs/watch.md

@@ -56,11 +56,20 @@ means invalidate **every configured scope**. Initial admission and every
 successful `setScopes` publish one undetailed `reconcile` invalidation. A rename
 or identity replacement is structural; metadata changes to the same ordinary
 file may be content changes. Neither means the file is settled or readable.
+An admitted native hint can invalidate an entry even when its before/after scan
+metadata is identical: the path may have changed and been restored between scans
+(ABA), or content may have changed without a distinguishable metadata change.
+Structural hints remain conservative in that case.
+This does not promise delivery for a differently spelled alias that appears and
+disappears entirely between scans: without an observed identity, it cannot be
+admitted as the selected path. Observation is not a complete transient history.
 Raw event names remain private: detail comes from guarded scans, prior guarded
-snapshots, or explicitly configured targets. Unknown names and overflow lose
-detail. Exclusion callbacks are synchronous; excluded directories are not
-scanned. Exclusions are a scan policy, not a promise that overflow cannot wake
-the application.
+snapshots, or explicitly configured targets. Unclassifiable hints inside selected,
+non-excluded territory lose detail. Hints for excluded or unselected paths are
+ignored. Exclusion callbacks are synchronous; excluded directories are recorded
+without descent. Bounded exclusion records recognize late deletion hints.
+Genuine backend event loss and detail-budget exhaustion still invalidate every
+scope, including when an excluded subtree caused the underlying event pressure.
 
 ## Transport and mode
 
@@ -96,6 +105,14 @@ pending detail and queued batches are bounded. The last removal stops and joins
 the native thread. No Worker threads, eval programs, or JS `fs.watch` are used.
 
 macOS uses FileEvents, NoDefer and WatchRoot with a 30 ms FSEvents latency.
+After guarded admission, streams use selected tree anchors and entry parents,
+falling back to the nearest admitted ancestor for missing paths. Nested anchors
+are deduplicated, with at most 128 paths. The eight shallowest non-overlapping
+excluded directories are also passed to `FSEventStreamSetExclusionPaths`.
+When the stream paths change, the old stream is stopped, invalidated and released
+on its dispatch queue, then its replacement starts before another guarded pass
+covers the handover. Native exclusions reduce traffic but cannot eliminate real
+FSEvents drops, including during recursive deletion.
 Absolute hints are reduced lexically against the admitted canonical Root;
 outside paths never become detail. Dropped/wrapped streams, RootChanged and
 Unmount trigger guarded reconciliation. Pathname hints can reflect activity
@@ -107,11 +124,18 @@ READ/WRITE/DELETE sharing, backup semantics and overlapped I/O. Each handle
 observes the entire subtree; guarded scans filter hints to configured scopes.
 No descendant watch handles are retained, so directories inside the Root can be
 renamed while watching, including directories containing selected scopes.
-Root-wide noise can coalesce into whole-scope invalidation. Completed 64 KiB buffers are copied and the
+Completed buffers are copied and the
 read re-armed before names are examined. Zero-byte / enumeration-loss completions
-invalidate every scope. Cancellation waits for IOCP completion before closing
+invalidate every scope. Buffers are 1 MiB on confirmed local volumes, and 64 KiB
+on network/UNC or unclassified volumes. Cancellation waits for IOCP completion before closing
 the handle or freeing its buffer; there are no detached retirement waits.
 
+On Windows, an open file handle—including fs-safe's pinned reads, editors, and
+antivirus—blocks renaming that file's ancestor directories, even with delete
+sharing. The watch backend retains only the Root's `ReadDirectoryChangesW` handle,
+which permits renames beneath the Root. Callers renaming directories concurrently
+with reads should use bounded retries, as Windows tools do.
+
 Windows prevents ordinary renames of the Root's own ancestors while its directory
 handle is open, even with DELETE sharing. Use `mode: "poll"` when callers must not
 retain that handle. Renaming or replacing the Root still fails guarded observation;
@@ -125,6 +149,7 @@ handles and delivery queues, and closing one does not retire another's observati
 | `scopes` | At most 128 literal scopes |
 | `persistent` | `true`; `false` lets Node exit with the subscription still open |
 | `intervalMs` | 30000 with events; 1000 with poll; minimum 20 ms |
+| `pollIntervalMs` | Optional polling override; minimum 20 ms, maximum 2147483647 ms (same as `intervalMs`) |
 | `maxDirectories` | 4096 observed directories, including scope ancestors |
 | `maxEntries` | 100000 examined entries per pass, including excluded entries |
 | `maxPendingPaths` | 256; maximum 4096 |
@@ -132,6 +157,12 @@ handles and delivery queues, and closing one does not retire another's observati
 
 Periodic guarded reconciliation runs without needing an event. It catches
 missed events and works on filesystems where native hints are incomplete.
+When polling is selected, the interval is `pollIntervalMs`, then `intervalMs`,
+then 1000 ms, in that order. This applies to explicit `mode: "poll"`, `auto`
+selecting polling, and `auto` falling back after an unsupported event backend.
+`pollIntervalMs` does not change the events reconciliation interval, which
+remains `intervalMs` or 30000 ms. For example, `mode: "auto", pollIntervalMs: 25`
+uses 25 ms polling when needed and retains the 30-second events reconciliation.
 Scans are metadata comparisons: content changes preserving all compared
 metadata may be missed in polling mode. No mode promises transactional
 snapshots, complete history, or hard real-time delivery.
@@ -143,12 +174,20 @@ registration/listing identity is retried up to three times per directory; furthe
 churn invalidates that subtree. Poll mode starts with its first scan and detects
 changes during the crawl on the next comparison. Neither mode waits for two
 agreeing scans.
+`ready` and `reconcile()` do not drain the operating system's event queue.
+For example, FSEvents can deliver coalesced setup creation activity after `ready`,
+even with its stream starting at the current event ID. Consumers must tolerate
+these advisory invalidations; tests measuring a quiet interval should first
+observe a selected sentinel edit and drain its trailing events.
 
 Each later pass compares with the previous snapshot, publishes bounded differences,
 and adopts its result as the next snapshot. Vanishing entries, kind changes, and
 transient descendant scan errors produce structural invalidations, preserving the
-Root identity checks. Events during a pass coalesce into one pending pass; detail
-overflow or catching up beyond the 25 ms coalescing window emits `overflow` without
+Root identity checks. Events during a pass coalesce into one pending pass and
+retain bounded detail regardless of how long the scan takes. The 25 ms hint
+coalescing window does not impose a scan deadline. A full native callback queue
+retains its bounded pending batch for retry. Genuine backend loss, exhausted
+detail capacity, or an unclassifiable selected hint emits `overflow` without
 detail. Sustained writes cannot exhaust a pass budget or disable observation.
 
 `reconcile()` resolves after a complete pass that **started after the call**. Calls
@@ -174,11 +213,39 @@ or inaccessible), fatal native backend/registration failures such as `watch-limi
 deterministic size limits (`too-large`, operation `scan`), or callback contract
 violations. Invalid scope admission, including symbolic parents, still rejects;
 it never grants authority through a link. Transient descendant churn does not
-make an admitted subscription unavailable. Callbacks may synchronously retire the owner; returning a thenable from
-`onInvalidate`, `onHealth`, or `exclude` rejects observation. Application async
+make an admitted subscription unavailable. Callbacks may synchronously retire
+the owner. Returning a thenable or a synchronous or asynchronous generator object
+from `onInvalidate`, `onHealth`, or `exclude` rejects observation without advancing
+generators. Application async
 work remains application-owned and is not joined by the subscription.
 
 `close()` is terminal, idempotent, and joined. Observation failures remain in
 health but do not make successful retirement reject. Retirement failures do
 reject, with a `SuppressedError` retaining an earlier observation failure when
 both exist. No new generation or callback can be admitted after close.
+
+The stress harness also provides `node scripts/watch-stress.mjs --scenario soak-short`,
+a two-minute lifecycle and collected-memory smoke. It is separate from the
+full one-hour `soak` scenario and its stronger memory-growth qualification.
+
+## Seeded consumer-cache stress test
+
+After building the package and host binding, run the model against real temporary
+Roots in both modes:
+
+```sh
+node scripts/watch-stress/model-runner.mjs --seeds 2000 --mode both --output watch-model-results.json
+```
+
+Each seed generates file and directory edits, renames, replacements, deep trees,
+symlink retargets, bursts, scope changes, and subscription retirement. A consumer
+queues refreshes only from `onInvalidate`; checkpoints drain those requests after
+quiescence and `reconcile()`, then compare its cache with a guarded Root walk and
+an independent mutation model. Native hints never authorize consumer reads.
+
+`--seed`, `--steps`, `--settle` (milliseconds), and `--concurrency` control a run.
+Failures are shrunk by fast-check and saved beside the report as `.<mode>-<seed>.failure.json`;
+replay one with `--replay <failure.json>`. Event mode requires a working native
+binding and never silently falls back to polling. The small
+`test/watch-model.test.ts` corpus runs in ordinary CI; native-event cases also run
+when `FS_SAFE_TEST_WATCH_EVENTS=1`. Keep fixtures on normal `os.tmpdir()` storage.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@openclaw/fs-safe",
-  "version": "0.21.0",
+  "version": "0.21.1",
   "description": "Capability-style filesystem roots for Node.js apps that handle untrusted relative paths.",
   "keywords": [
     "filesystem",
@@ -173,13 +173,13 @@
     "archive:producer-smoke": "node scripts/archive-producer-smoke.mjs"
   },
   "optionalDependencies": {
-    "@openclaw/fs-safe-darwin-arm64": "0.21.0",
-    "@openclaw/fs-safe-darwin-x64": "0.21.0",
-    "@openclaw/fs-safe-linux-arm64-gnu": "0.21.0",
-    "@openclaw/fs-safe-linux-arm64-musl": "0.21.0",
-    "@openclaw/fs-safe-linux-x64-gnu": "0.21.0",
-    "@openclaw/fs-safe-linux-x64-musl": "0.21.0",
-    "@openclaw/fs-safe-win32-x64-msvc": "0.21.0",
+    "@openclaw/fs-safe-darwin-arm64": "0.21.1",
+    "@openclaw/fs-safe-darwin-x64": "0.21.1",
+    "@openclaw/fs-safe-linux-arm64-gnu": "0.21.1",
+    "@openclaw/fs-safe-linux-arm64-musl": "0.21.1",
+    "@openclaw/fs-safe-linux-x64-gnu": "0.21.1",
+    "@openclaw/fs-safe-linux-x64-musl": "0.21.1",
+    "@openclaw/fs-safe-win32-x64-msvc": "0.21.1",
     "jszip": "^3.10.2"
   },
   "devDependencies": {