disk-space-saviour 0.15.2 → 0.17.0

package/CHANGELOG.md

@@ -1,5 +1,25 @@
 # Changelog
 
+## 0.17.0
+
+### Minor Changes
+
+- a518d7b: Reuse recent authenticated scans, stream safe global-cache cleanup before project discovery, and stop emergency scans as soon as the free-space goal is met. Add bounded discovery, persistent directory and size indexes, separate candidate/sizing progress counters, and macOS swap diagnostics while retaining fresh deletion checks.
+
+## 0.16.0
+
+### Minor Changes
+
+- dddd5f6: Batch remote filesystem operations, add scan budgets and partial reports on
+  cancellation, discover containerd upperdirs, batch safe container deletions with
+  allocated-byte accounting, and retry process inspection by owner while reporting
+  blocked sizes and environment hints.
+
+  Find version-manager download caches and idle semver baseline builds, honor
+  configured cache roots, and watch host task health within its mount namespace.
+  Preserve scan and interrupted cleanup pre-scan reports when audit storage is
+  unwritable, and avoid root-owned default state under another user's home.
+
 ## 0.15.2
 
 ### Patch Changes

package/README.md

@@ -76,7 +76,14 @@ Run `dss --help` for every option. The most useful ones:
 - `--scanner NAME` restricts scanning to `projects`, `global`, `versions`,
   `agents` or `system`.
 - `--report FILE` limits cleaning to items in a saved `dss scan --json`
-  report. The current rules rescan those items before deletion.
+  report. `--from FILE` also accepts an audit JSON containing the full report.
+  Recent reports authenticated by this installation are reused; older, unsigned
+  or edited reports require discovery again. Changing discovery or action options
+  also rebuilds the plan. Every deletion still gets fresh checks.
+- `clean` automatically reuses a matching scan from the last 30 minutes.
+  `--fresh` bypasses this plan and the discovery/size index; `--cache-dir DIR`
+  chooses the state location (default `$XDG_STATE_HOME/disk-space-saviour`, or
+  `~/.local/state/disk-space-saviour`).
 
 ## Tiers
 
@@ -118,6 +125,7 @@ is needed. Agent scratch cleanup requires inactivity and Git safety checks.
   macOS) are skipped. So are projects with a running `cargo`, `gradle`,
   `npm`, `node` or similar tool, and anything modified within
   `--older-than` (1h).
+  Container paths are checked individually before each bounded deletion batch.
 - **Rust builds are pruned in two parts.** `cargo-superseded` removes
   older hashes of a unit (`.rlib`, `.rmeta`, `.so`/`.dylib`/`.a`,
   `build/*`, `.fingerprint`) that a newer build replaced, but never while
@@ -135,6 +143,9 @@ is needed. Agent scratch cleanup requires inactivity and Git safety checks.
   environment stops at once, and the audit log and summary say which
   process was lost. A build that simply finishes during the cleanup also
   stops it: an exit cannot be told apart from a crash.
+  Host cleanup watches processes in its own mount namespace; each container
+  watches its own tasks. Processes with unknown namespace membership remain
+  watched.
 - **Git awareness.** Whole projects and containers are protected while they
   have unsaved changes, unpushed commits or stashes. Ignored dependency and
   build outputs can be cleaned despite unrelated source changes; tracked or
@@ -152,6 +163,9 @@ is needed. Agent scratch cleanup requires inactivity and Git safety checks.
   and left per environment, and disk usage before and after. Scan audits
   include every candidate and its blockers; blocked candidates do not
   contribute to reclaimable tier totals.
+  If a read-only scan cannot save its audit, it warns and still prints the
+  report. Root scans with another user's `HOME` use a private temporary audit
+  directory. Set `--audit-dir` to choose a persistent writable location.
 
 Long-lived `npm exec`/`npx` servers allow idle npm download and node-gyp
 header caches to be cleaned. Active npx hashes remain protected through
@@ -165,6 +179,53 @@ and produces one environment hint. Old isolation logs and sanitized upload stagi
 are moderate-tier candidates. Opam download cleanup removes its cache
 directory directly and does not invoke `opam clean` or migrate the root.
 
+Scans have a two-minute budget per filesystem environment, configurable with
+`--scan-budget 5m` (library: `scanBudget`). Progress goes to stderr, leaving
+JSON stdout clean. SIGINT and SIGTERM stop new work, cancel subprocesses and
+print completed findings marked `aborted`; incomplete findings remain blocked.
+The CLI exits 130 or 143 respectively. A second signal exits immediately.
+Library callers can pass an `AbortSignal` and an `onScanProgress` callback.
+
+Global caches run first, prioritizing rules that can use native sizing without
+recursive activity timestamps, and emit findings as each rule finishes. Non-interactive
+`clean` plans or deletes completed safe host caches while the remaining discovery
+continues. `emergency` does the same, checks actual free space after each deletion,
+and skips the remaining scan once the goal is met. Interactive runs show a plan
+before asking for deletion consent. A later phase hitting its budget blocks its
+partial findings while completed independent findings stay usable.
+
+Project discovery uses eight directory workers, prunes matched generated trees,
+Git objects, home media/Photos libraries, macOS Library and mounted volumes.
+Source activity is checked separately without sizing entire project trees again.
+Spotlight (`mdfind`) or `plocate`/`locate` supplies verified early hints when
+available; the walk also runs because an index can miss files. Version-pin discovery
+also prunes dependency/build trees.
+
+The persistent `index/` stores directory listings and candidate measurements.
+Each descendant directory's own mtime/ctime and identity is checked before reusing
+its listing, so a nested addition is visible even when an ancestor did not change.
+Unchanged directories avoid another listing; discovery still checks their metadata.
+Sizing uses bounded workers and `du -skx` when no recursive activity timestamp is
+needed. Sizes cached for up to an hour are labeled **estimated**, because changing
+an existing descendant need not update the candidate's directory mtime. Cleanup
+always bypasses that cache and rechecks processes, open files, Git, path boundaries
+and sizes. Safe cleanup also rechecks source activity. Explicit moderate cleanup
+preserves the ability to remove ignored dependencies when only source changed.
+Library callers opt into state with `cacheDir`.
+
+On macOS, reports include `sysctl vm.swapusage` and the largest resident/compressed
+memory consumers from `top`. These are memory diagnostics, not verified per-process
+swap ownership. When Docker Desktop consumes memory, saving work and restarting it
+or lowering its VM memory limit can help. dss never deletes swap files.
+
+Cache discovery honors tool configuration before default paths, including
+`GOMODCACHE`, `GOPATH`, `GOCACHE`, `CARGO_HOME`, `RUSTUP_HOME`,
+`GRADLE_USER_HOME`, `DENO_DIR`, `BUN_INSTALL_CACHE_DIR` and `XDG_CACHE_HOME`.
+Go, npm, pip, Yarn and pnpm cache queries are bounded and cached per environment.
+Safe rules also cover perlbrew build/download trees, SDKMAN tmp/archive contents,
+nvm downloads, idle `target/semver-checks`, and downloaded Ruby `.gem` files.
+SDKMAN parent directories and installed gems are preserved.
+
 ## Docker
 
 With `--docker`, or automatically when a daemon is reachable, `dss` handles
@@ -188,6 +249,11 @@ three kinds of Docker data:
   this is denied or incomplete, cleanup stays blocked and names the
   unreadable processes. This also applies when dss runs inside a container
   with access to its Docker daemon.
+  Root probes also retry unreadable processes as their owning UID/GID with
+  `setpriv` when available. Readable evidence is retained after failed retries;
+  unresolved inspection produces one environment hint. Blocked totals show
+  measured bytes and separately identify visible bytes whose freeable size is
+  unknown.
 - **Stopped containers** are listed with their size, image, command, owner
   session (`HIVE_MIND_PARENT_SESSION_ID` and other session labels or
   environment variables, else a session id in the container name), task
@@ -223,6 +289,13 @@ memory headroom, and skips excessive candidate counts. Fresh Git checks before
 stopped-container removal still inspect new work. Verbose mode reports storage
 measurement progress. Host scans prune runtime, nested overlay, read-only and
 unselected mount points; explicitly selected ordinary mount roots are allowed.
+For containerd-backed running containers, the daemon-side init process's
+`/proc/<pid>/mountinfo` supplies the overlay upperdir when `GraphDriver` is absent.
+JSON environments and the text summary show the storage source. Paths are
+measured on the daemon host and removed inside the container so overlay
+whiteouts are respected; cleanup reports the measured change in allocated
+upperdir blocks when available. Remote filesystem operations use bounded
+POSIX-shell batches and do not require Node inside the target container.
 
 Docker discovery uses a size-free container listing and measures each
 container independently. Failed size queries are retried once; objects

package/bin/dss.js

@@ -3,14 +3,31 @@
 import { runCli } from '../src/cli.js';
 
 const controller = new AbortController();
-const abort = () => controller.abort();
-process.once('SIGINT', abort);
-process.once('SIGTERM', abort);
+let received = null,
+  fallback;
+const abort = (signal) => {
+  const code = signal === 'SIGTERM' ? 143 : 130;
+  if (received) {
+    process.exit(code);
+  }
+  received = code;
+  controller.abort(new Error(`interrupted by ${signal}`));
+  fallback = setTimeout(() => process.exit(code), 5000);
+  fallback.unref();
+};
+const interrupt = () => abort('SIGINT'),
+  terminate = () => abort('SIGTERM');
+process.on('SIGINT', interrupt);
+process.on('SIGTERM', terminate);
 try {
   process.exitCode = await runCli(process.argv.slice(2), {
     signal: controller.signal,
   });
+  if (received) {
+    process.exitCode = received;
+  }
 } finally {
-  process.removeListener('SIGINT', abort);
-  process.removeListener('SIGTERM', abort);
+  clearTimeout(fallback);
+  process.removeListener('SIGINT', interrupt);
+  process.removeListener('SIGTERM', terminate);
 }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "disk-space-saviour",
-  "version": "0.15.2",
+  "version": "0.17.0",
   "description": "Safely reclaims disk space: caches and build outputs of 20+ ecosystems, forgotten projects, and Docker (running, stopped and nested containers), with an emergency mode",
   "type": "module",
   "main": "src/index.js",

package/src/audit.js

@@ -12,6 +12,7 @@ import os from 'node:os';
 import path from 'node:path';
 
 export const AUDIT_SCHEMA = 1;
+let privateAuditDirectory;
 
 export function throwIfAborted(options) {
   if (options.signal?.aborted) {
@@ -74,8 +75,24 @@ export function startAudit(command, fields = {}) {
  * @returns {Promise<string>}
  */
 export async function writeAudit(audit, options = {}) {
-  const dir = auditDirectory(options);
-  await fsp.mkdir(dir, { recursive: true });
+  let dir = auditDirectory(options);
+  // sudo/docker exec may retain another user's HOME. Do not leave root-owned
+  // default state there; explicit audit locations remain the caller's choice.
+  if (
+    !options.auditDir &&
+    !process.env.DSS_AUDIT_DIR &&
+    process.getuid?.() === 0
+  ) {
+    const home = os.homedir();
+    const owner = await fsp.stat(home).catch(() => null);
+    if (owner && owner.uid !== 0 && dir.startsWith(`${home}${path.sep}`)) {
+      privateAuditDirectory ??= fsp.mkdtemp(
+        path.join(os.tmpdir(), 'dss-audit-0-')
+      );
+      dir = await privateAuditDirectory;
+    }
+  }
+  await fsp.mkdir(dir, { recursive: true, mode: 0o700 });
   if (!options.inProgress) {
     audit.finishedAt ??= new Date().toISOString();
   }

package/src/clean.js

@@ -31,6 +31,9 @@ import {
 import { imageNamed } from './docker/images.js';
 import { resolveEnvironment } from './env/resolve.js';
 import { configureScanPolicy } from './env/policy.js';
+import { configureCacheRoots } from './env/cache-roots.js';
+import { environmentScope, signalExecutor } from './env/scope.js';
+import { batches } from './env/batches.js';
 import { GitInspector } from './git.js';
 import { HealthWatch } from './health.js';
 import { consentFlag, dropNested, selectByTier, tierRank } from './items.js';
@@ -41,6 +44,7 @@ import { OTHER_RULES } from './rules/other.js';
 import { compareVersions, VERSION_RULES } from './rules/versions.js';
 import { scanVersions } from './scanners/versions.js';
 import { analyzeRustProfile } from './scanners/rust.js';
+import { currentProjectTier } from './scanners/projects.js';
 import {
   filterItems,
   revalidateReport,
@@ -158,7 +162,7 @@ export class Cleaner {
     this.report = report;
     this.options = options;
     this.contexts = new Map();
-    this.health = new HealthWatch();
+    this.health = new HealthWatch(options.priorHealth);
     this.descriptors = new Map(
       (report.environments ?? []).map((env) => [env.id, env])
     );
@@ -219,6 +223,18 @@ export class Cleaner {
       return;
     }
     const ctx = this.context(item.env);
+    const executor = ctx.executor;
+    ctx.executor = signalExecutor(executor, this.options.signal);
+    try {
+      return await environmentScope(ctx.env, this.options.signal, () =>
+        this.#processScoped(ctx, item, entry)
+      );
+    } finally {
+      ctx.executor = executor;
+    }
+  }
+
+  async #processScoped(ctx, item, entry) {
     const fresh = await this.#recheck(ctx, item);
     if (fresh.reason) {
       entry.reason = fresh.reason;
@@ -356,7 +372,20 @@ export class Cleaner {
   }
 
   async #staticBlocker(ctx, item) {
+    if (item.project && this.options.tier === 'safe') {
+      const tier = await currentProjectTier(
+        { ...ctx, options: this.options, now: this.options.now() },
+        item
+      );
+      if (tierRank(tier) > tierRank(this.options.tier)) {
+        return `project recently active; requires the ${tier} tier`;
+      }
+    }
     if (item.recheck?.type === 'version') {
+      if (!ctx.cacheRootsConfigured) {
+        await configureCacheRoots(ctx.env, await ctx.env.homeDirs());
+        ctx.cacheRootsConfigured = true;
+      }
       const rule = VERSION_RULES.find(
         (candidate) => candidate.id === item.rule
       );
@@ -366,6 +395,7 @@ export class Cleaner {
           options: {
             ...this.options,
             homes: [item.recheck.home],
+            freshSizes: true,
             roots: this.report.options.roots ?? [],
           },
           now: this.options.now(),
@@ -410,9 +440,18 @@ export class Cleaner {
     if (
       ctx.env.kind === 'host' &&
       type === 'remove' &&
-      this.report.docker?.daemons?.some((daemon) => daemon.env === ctx.env.id)
+      (this.report.docker?.daemons?.some(
+        (daemon) => daemon.env === ctx.env.id
+      ) ||
+        this.options.streaming)
     ) {
-      const mounts = await runningBindMounts(new DockerCli(ctx.executor));
+      const docker = new DockerCli(ctx.executor);
+      if (this.options.streaming && !(await docker.info())) {
+        return this.options.docker === true
+          ? 'cannot verify running Docker bind mounts'
+          : null;
+      }
+      const mounts = await runningBindMounts(docker);
       for (const mount of mounts) {
         if (
           item.paths.some(
@@ -469,6 +508,9 @@ export class Cleaner {
    * item stops; an item checked path by path skips just the busy path.
    */
   async #removePaths(ctx, item, entry) {
+    if (ctx.env.kind === 'container' && item.checks?.perPath) {
+      return this.#removeBatches(ctx, item, entry);
+    }
     for (const target of item.action.paths) {
       if (this.options.signal?.aborted) {
         throw new Error('cleanup aborted');
@@ -490,18 +532,114 @@ export class Cleaner {
         entry.reason = boundary;
         return;
       }
-      await ctx.env.remove([target]);
-      ctx.env.writableLayer?.forget?.(target, ctx.env.path);
-      entry.status = 'removed';
-      entry.deletedPaths.push(target);
-      entry.freedBytes += usage?.bytes ?? 0;
-      await this.options.onProgress?.(entry);
+      await this.#deletePaths(ctx, entry, [target], usage?.bytes ?? 0);
     }
     if (entry.deletedPaths.length === 0 && entry.skippedPaths.length > 0) {
       entry.reason = 'every path was in use or recently written';
     }
   }
 
+  async #removeBatches(ctx, item, entry) {
+    for (const batch of batches(item.action.paths)) {
+      if (!(await this.#removeBatch(ctx, item, entry, batch))) {
+        return;
+      }
+    }
+  }
+
+  /** One short remote deletion per batch, after fresh health/liveness/boundary checks. */
+  async #removeBatch(ctx, item, entry, batch) {
+    await configureScanPolicy(ctx.env, this.options);
+    await ctx.liveness.refresh(true);
+    const stopped = this.health.verify(item, ctx.liveness.processList());
+    if (stopped) {
+      entry.reason = stopped;
+      return false;
+    }
+    await ctx.liveness.resolve(batch);
+    const usages = await ctx.env.usageMany(batch);
+    const { selected, reason } = this.#batchSelection(
+      ctx,
+      item,
+      entry,
+      batch,
+      usages
+    );
+    if (reason) {
+      entry.reason = reason;
+      return false;
+    }
+    if (!selected.length) {
+      return true;
+    }
+    await this.#deletePaths(ctx, entry, selected, sumBytes(usages, selected));
+    const lost = await this.health.check(ctx.env, item);
+    if (lost) {
+      entry.reason = lost;
+    }
+    return !lost;
+  }
+
+  #batchSelection(ctx, item, entry, batch, usages) {
+    const selected = [];
+    for (const target of batch) {
+      const blocker =
+        ctx.env.scanPolicy.removalReason(target) ??
+        ctx.liveness.busyReason({ ...item, paths: [target] });
+      if (blocker) {
+        return { selected: [], reason: blocker };
+      }
+      const usage = usages.get(target);
+      const busy = this.#pathActivity(ctx, item, target, usage);
+      const unavailable = this.#unavailableUsage(usage);
+      if (busy || unavailable) {
+        entry.skippedPaths.push({ path: target, reason: busy ?? unavailable });
+      } else {
+        selected.push(target);
+      }
+    }
+    return { selected };
+  }
+
+  #unavailableUsage(usage) {
+    if (!usage) {
+      return 'path disappeared';
+    }
+    if (usage.sizeUnknown) {
+      return 'writable-layer size unknown';
+    }
+    return usage.bytes === 0 ? 'no writable bytes remain' : null;
+  }
+
+  async #deletePaths(ctx, entry, selected, estimate) {
+    const before = await ctx.env.writableLayer?.allocatedBytes?.(
+      selected,
+      ctx.env.path
+    );
+    await ctx.env.remove(selected);
+    const after = await ctx.env.writableLayer?.allocatedBytes?.(
+      selected,
+      ctx.env.path
+    );
+    for (const target of selected) {
+      ctx.env.writableLayer?.forget?.(target, ctx.env.path);
+    }
+    entry.status = 'removed';
+    entry.deletedPaths.push(...selected);
+    entry.freedBytes += this.#freedBytes(ctx, entry, before, after, estimate);
+    await this.options.onProgress?.(entry);
+  }
+
+  #freedBytes(ctx, entry, before, after, fallback) {
+    if (typeof before === 'number' && typeof after === 'number') {
+      return Math.max(0, before - after);
+    }
+    if (ctx.env.writableLayer) {
+      entry.freedEstimated = true;
+    }
+    return fallback;
+  }
+
   #pathActivity(ctx, item, target, usage) {
     return (
       item.checks?.perPath &&
@@ -789,7 +927,8 @@ export async function clean(report, input = {}) {
   const audit = startAudit(input.command ?? 'clean', {
     dryRun: Boolean(options.dryRun),
     tier: options.tier,
-    report: { createdAt: report.createdAt, totals: report.totals },
+    report,
+    entries: [...(input.priorEntries ?? [])],
   });
   const persistProgress = async (entry) => {
     if (!audit.entries.includes(entry)) {
@@ -809,7 +948,10 @@ export async function clean(report, input = {}) {
     await writeAudit(audit, { ...options, inProgress: true });
   }
   try {
-    for (const item of planItems(report, options)) {
+    const done = new Set(audit.entries.map((entry) => entry.id));
+    for (const item of planItems(report, options).filter(
+      (item) => !done.has(item.id)
+    )) {
       if (options.signal?.aborted) {
         throw new Error('cleanup aborted');
       }