disk-space-saviour 0.15.1 → 0.16.0

package/CHANGELOG.md

@@ -1,5 +1,25 @@
 # Changelog
 
+## 0.16.0
+
+### Minor Changes
+
+- dddd5f6: Batch remote filesystem operations, add scan budgets and partial reports on
+  cancellation, discover containerd upperdirs, batch safe container deletions with
+  allocated-byte accounting, and retry process inspection by owner while reporting
+  blocked sizes and environment hints.
+
+  Find version-manager download caches and idle semver baseline builds, honor
+  configured cache roots, and watch host task health within its mount namespace.
+  Preserve scan and interrupted cleanup pre-scan reports when audit storage is
+  unwritable, and avoid root-owned default state under another user's home.
+
+## 0.15.2
+
+### Patch Changes
+
+- 841a999: Fix the remaining CI/CD false positives, false negatives and warnings: tests that cannot run on a runner are reported as skipped with the reason instead of passing vacuously or vanishing from the summary, the Deno leg runs with `-A --parallel`, the pedantic zizmor pass runs online and triggers on every file it reads, jscpd 5 receives the comment-skipping setting it understands, the example app reviews its locked install scripts, and release logs no longer warn about `deno.lock` or print the changeset publish output twice.
+
 ## 0.15.1
 
 ### Patch Changes

package/README.md

@@ -118,6 +118,7 @@ is needed. Agent scratch cleanup requires inactivity and Git safety checks.
   macOS) are skipped. So are projects with a running `cargo`, `gradle`,
   `npm`, `node` or similar tool, and anything modified within
   `--older-than` (1h).
+  Container paths are checked individually before each bounded deletion batch.
 - **Rust builds are pruned in two parts.** `cargo-superseded` removes
   older hashes of a unit (`.rlib`, `.rmeta`, `.so`/`.dylib`/`.a`,
   `build/*`, `.fingerprint`) that a newer build replaced, but never while
@@ -135,6 +136,9 @@ is needed. Agent scratch cleanup requires inactivity and Git safety checks.
   environment stops at once, and the audit log and summary say which
   process was lost. A build that simply finishes during the cleanup also
   stops it: an exit cannot be told apart from a crash.
+  Host cleanup watches processes in its own mount namespace; each container
+  watches its own tasks. Processes with unknown namespace membership remain
+  watched.
 - **Git awareness.** Whole projects and containers are protected while they
   have unsaved changes, unpushed commits or stashes. Ignored dependency and
   build outputs can be cleaned despite unrelated source changes; tracked or
@@ -152,6 +156,9 @@ is needed. Agent scratch cleanup requires inactivity and Git safety checks.
   and left per environment, and disk usage before and after. Scan audits
   include every candidate and its blockers; blocked candidates do not
   contribute to reclaimable tier totals.
+  If a read-only scan cannot save its audit, it warns and still prints the
+  report. Root scans with another user's `HOME` use a private temporary audit
+  directory. Set `--audit-dir` to choose a persistent writable location.
 
 Long-lived `npm exec`/`npx` servers allow idle npm download and node-gyp
 header caches to be cleaned. Active npx hashes remain protected through
@@ -165,6 +172,21 @@ and produces one environment hint. Old isolation logs and sanitized upload stagi
 are moderate-tier candidates. Opam download cleanup removes its cache
 directory directly and does not invoke `opam clean` or migrate the root.
 
+Scans have a two-minute budget per filesystem environment, configurable with
+`--scan-budget 5m` (library: `scanBudget`). Progress goes to stderr, leaving
+JSON stdout clean. SIGINT and SIGTERM stop new work, cancel subprocesses and
+print completed findings marked `aborted`; incomplete findings remain blocked.
+The CLI exits 130 or 143 respectively. A second signal exits immediately.
+Library callers can pass an `AbortSignal` and an `onScanProgress` callback.
+
+Cache discovery honors tool configuration before default paths, including
+`GOMODCACHE`, `GOPATH`, `GOCACHE`, `CARGO_HOME`, `RUSTUP_HOME`,
+`GRADLE_USER_HOME`, `DENO_DIR`, `BUN_INSTALL_CACHE_DIR` and `XDG_CACHE_HOME`.
+Go, npm, pip, Yarn and pnpm cache queries are bounded and cached per environment.
+Safe rules also cover perlbrew build/download trees, SDKMAN tmp/archive contents,
+nvm downloads, idle `target/semver-checks`, and downloaded Ruby `.gem` files.
+SDKMAN parent directories and installed gems are preserved.
+
 ## Docker
 
 With `--docker`, or automatically when a daemon is reachable, `dss` handles
@@ -188,6 +210,11 @@ three kinds of Docker data:
   this is denied or incomplete, cleanup stays blocked and names the
   unreadable processes. This also applies when dss runs inside a container
   with access to its Docker daemon.
+  Root probes also retry unreadable processes as their owning UID/GID with
+  `setpriv` when available. Readable evidence is retained after failed retries;
+  unresolved inspection produces one environment hint. Blocked totals show
+  measured bytes and separately identify visible bytes whose freeable size is
+  unknown.
 - **Stopped containers** are listed with their size, image, command, owner
   session (`HIVE_MIND_PARENT_SESSION_ID` and other session labels or
   environment variables, else a session id in the container name), task
@@ -223,6 +250,13 @@ memory headroom, and skips excessive candidate counts. Fresh Git checks before
 stopped-container removal still inspect new work. Verbose mode reports storage
 measurement progress. Host scans prune runtime, nested overlay, read-only and
 unselected mount points; explicitly selected ordinary mount roots are allowed.
+For containerd-backed running containers, the daemon-side init process's
+`/proc/<pid>/mountinfo` supplies the overlay upperdir when `GraphDriver` is absent.
+JSON environments and the text summary show the storage source. Paths are
+measured on the daemon host and removed inside the container so overlay
+whiteouts are respected; cleanup reports the measured change in allocated
+upperdir blocks when available. Remote filesystem operations use bounded
+POSIX-shell batches and do not require Node inside the target container.
 
 Docker discovery uses a size-free container listing and measures each
 container independently. Failed size queries are retried once; objects
@@ -343,7 +377,7 @@ Configured in `.prettierrc`:
 npm install
 npm test                  # Node.js, 30s per test
 bun test --timeout 30000  # Bun
-deno test --allow-read    # Deno (read-only tests)
+deno test -A --parallel   # Deno (fixtures need -A)
 npm run check             # lint + format + duplication
 npm run test:dind         # Docker-in-Docker integration (needs --privileged)
 node bin/dss.js scan --verbose

package/bin/dss.js

@@ -3,14 +3,31 @@
 import { runCli } from '../src/cli.js';
 
 const controller = new AbortController();
-const abort = () => controller.abort();
-process.once('SIGINT', abort);
-process.once('SIGTERM', abort);
+let received = null,
+  fallback;
+const abort = (signal) => {
+  const code = signal === 'SIGTERM' ? 143 : 130;
+  if (received) {
+    process.exit(code);
+  }
+  received = code;
+  controller.abort(new Error(`interrupted by ${signal}`));
+  fallback = setTimeout(() => process.exit(code), 5000);
+  fallback.unref();
+};
+const interrupt = () => abort('SIGINT'),
+  terminate = () => abort('SIGTERM');
+process.on('SIGINT', interrupt);
+process.on('SIGTERM', terminate);
 try {
   process.exitCode = await runCli(process.argv.slice(2), {
     signal: controller.signal,
   });
+  if (received) {
+    process.exitCode = received;
+  }
 } finally {
-  process.removeListener('SIGINT', abort);
-  process.removeListener('SIGTERM', abort);
+  clearTimeout(fallback);
+  process.removeListener('SIGINT', interrupt);
+  process.removeListener('SIGTERM', terminate);
 }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "disk-space-saviour",
-  "version": "0.15.1",
+  "version": "0.16.0",
   "description": "Safely reclaims disk space: caches and build outputs of 20+ ecosystems, forgotten projects, and Docker (running, stopped and nested containers), with an emergency mode",
   "type": "module",
   "main": "src/index.js",

package/src/audit.js

@@ -12,6 +12,7 @@ import os from 'node:os';
 import path from 'node:path';
 
 export const AUDIT_SCHEMA = 1;
+let privateAuditDirectory;
 
 export function throwIfAborted(options) {
   if (options.signal?.aborted) {
@@ -74,8 +75,24 @@ export function startAudit(command, fields = {}) {
  * @returns {Promise<string>}
  */
 export async function writeAudit(audit, options = {}) {
-  const dir = auditDirectory(options);
-  await fsp.mkdir(dir, { recursive: true });
+  let dir = auditDirectory(options);
+  // sudo/docker exec may retain another user's HOME. Do not leave root-owned
+  // default state there; explicit audit locations remain the caller's choice.
+  if (
+    !options.auditDir &&
+    !process.env.DSS_AUDIT_DIR &&
+    process.getuid?.() === 0
+  ) {
+    const home = os.homedir();
+    const owner = await fsp.stat(home).catch(() => null);
+    if (owner && owner.uid !== 0 && dir.startsWith(`${home}${path.sep}`)) {
+      privateAuditDirectory ??= fsp.mkdtemp(
+        path.join(os.tmpdir(), 'dss-audit-0-')
+      );
+      dir = await privateAuditDirectory;
+    }
+  }
+  await fsp.mkdir(dir, { recursive: true, mode: 0o700 });
   if (!options.inProgress) {
     audit.finishedAt ??= new Date().toISOString();
   }

package/src/clean.js

@@ -31,6 +31,9 @@ import {
 import { imageNamed } from './docker/images.js';
 import { resolveEnvironment } from './env/resolve.js';
 import { configureScanPolicy } from './env/policy.js';
+import { configureCacheRoots } from './env/cache-roots.js';
+import { environmentScope, signalExecutor } from './env/scope.js';
+import { batches } from './env/batches.js';
 import { GitInspector } from './git.js';
 import { HealthWatch } from './health.js';
 import { consentFlag, dropNested, selectByTier, tierRank } from './items.js';
@@ -219,6 +222,18 @@ export class Cleaner {
       return;
     }
     const ctx = this.context(item.env);
+    const executor = ctx.executor;
+    ctx.executor = signalExecutor(executor, this.options.signal);
+    try {
+      return await environmentScope(ctx.env, this.options.signal, () =>
+        this.#processScoped(ctx, item, entry)
+      );
+    } finally {
+      ctx.executor = executor;
+    }
+  }
+
+  async #processScoped(ctx, item, entry) {
     const fresh = await this.#recheck(ctx, item);
     if (fresh.reason) {
       entry.reason = fresh.reason;
@@ -357,6 +372,10 @@ export class Cleaner {
 
   async #staticBlocker(ctx, item) {
     if (item.recheck?.type === 'version') {
+      if (!ctx.cacheRootsConfigured) {
+        await configureCacheRoots(ctx.env, await ctx.env.homeDirs());
+        ctx.cacheRootsConfigured = true;
+      }
       const rule = VERSION_RULES.find(
         (candidate) => candidate.id === item.rule
       );
@@ -469,6 +488,9 @@ export class Cleaner {
    * item stops; an item checked path by path skips just the busy path.
    */
   async #removePaths(ctx, item, entry) {
+    if (ctx.env.kind === 'container' && item.checks?.perPath) {
+      return this.#removeBatches(ctx, item, entry);
+    }
     for (const target of item.action.paths) {
       if (this.options.signal?.aborted) {
         throw new Error('cleanup aborted');
@@ -490,18 +512,114 @@ export class Cleaner {
         entry.reason = boundary;
         return;
       }
-      await ctx.env.remove([target]);
-      ctx.env.writableLayer?.forget?.(target, ctx.env.path);
-      entry.status = 'removed';
-      entry.deletedPaths.push(target);
-      entry.freedBytes += usage?.bytes ?? 0;
-      await this.options.onProgress?.(entry);
+      await this.#deletePaths(ctx, entry, [target], usage?.bytes ?? 0);
     }
     if (entry.deletedPaths.length === 0 && entry.skippedPaths.length > 0) {
       entry.reason = 'every path was in use or recently written';
     }
   }
 
+  async #removeBatches(ctx, item, entry) {
+    for (const batch of batches(item.action.paths)) {
+      if (!(await this.#removeBatch(ctx, item, entry, batch))) {
+        return;
+      }
+    }
+  }
+
+  /** One short remote deletion per batch, after fresh health/liveness/boundary checks. */
+  async #removeBatch(ctx, item, entry, batch) {
+    await configureScanPolicy(ctx.env, this.options);
+    await ctx.liveness.refresh(true);
+    const stopped = this.health.verify(item, ctx.liveness.processList());
+    if (stopped) {
+      entry.reason = stopped;
+      return false;
+    }
+    await ctx.liveness.resolve(batch);
+    const usages = await ctx.env.usageMany(batch);
+    const { selected, reason } = this.#batchSelection(
+      ctx,
+      item,
+      entry,
+      batch,
+      usages
+    );
+    if (reason) {
+      entry.reason = reason;
+      return false;
+    }
+    if (!selected.length) {
+      return true;
+    }
+    await this.#deletePaths(ctx, entry, selected, sumBytes(usages, selected));
+    const lost = await this.health.check(ctx.env, item);
+    if (lost) {
+      entry.reason = lost;
+    }
+    return !lost;
+  }
+
+  #batchSelection(ctx, item, entry, batch, usages) {
+    const selected = [];
+    for (const target of batch) {
+      const blocker =
+        ctx.env.scanPolicy.removalReason(target) ??
+        ctx.liveness.busyReason({ ...item, paths: [target] });
+      if (blocker) {
+        return { selected: [], reason: blocker };
+      }
+      const usage = usages.get(target);
+      const busy = this.#pathActivity(ctx, item, target, usage);
+      const unavailable = this.#unavailableUsage(usage);
+      if (busy || unavailable) {
+        entry.skippedPaths.push({ path: target, reason: busy ?? unavailable });
+      } else {
+        selected.push(target);
+      }
+    }
+    return { selected };
+  }
+
+  #unavailableUsage(usage) {
+    if (!usage) {
+      return 'path disappeared';
+    }
+    if (usage.sizeUnknown) {
+      return 'writable-layer size unknown';
+    }
+    return usage.bytes === 0 ? 'no writable bytes remain' : null;
+  }
+
+  async #deletePaths(ctx, entry, selected, estimate) {
+    const before = await ctx.env.writableLayer?.allocatedBytes?.(
+      selected,
+      ctx.env.path
+    );
+    await ctx.env.remove(selected);
+    const after = await ctx.env.writableLayer?.allocatedBytes?.(
+      selected,
+      ctx.env.path
+    );
+    for (const target of selected) {
+      ctx.env.writableLayer?.forget?.(target, ctx.env.path);
+    }
+    entry.status = 'removed';
+    entry.deletedPaths.push(...selected);
+    entry.freedBytes += this.#freedBytes(ctx, entry, before, after, estimate);
+    await this.options.onProgress?.(entry);
+  }
+
+  #freedBytes(ctx, entry, before, after, fallback) {
+    if (typeof before === 'number' && typeof after === 'number') {
+      return Math.max(0, before - after);
+    }
+    if (ctx.env.writableLayer) {
+      entry.freedEstimated = true;
+    }
+    return fallback;
+  }
+
   #pathActivity(ctx, item, target, usage) {
     return (
       item.checks?.perPath &&

package/src/cli.js

@@ -37,6 +37,7 @@ Scope:
   --recursive                docker: recurse into nested daemons
   --container ID|NAME        limit Docker work to these containers (repeat)
   --max-depth N              directory depth for project search (default 6)
+  --scan-budget AGE          time budget per filesystem environment (2m)
   --only NAME                only these ecosystems, rules or kinds (repeat)
   --exclude PATH|GLOB        never touch matching paths (repeat)
   --scanner NAME             projects, global, versions, agents, system
@@ -80,6 +81,7 @@ const OPTIONS = {
   recursive: { type: 'boolean' },
   container: { type: 'string', multiple: true },
   'max-depth': { type: 'string' },
+  'scan-budget': { type: 'string' },
   only: { type: 'string', multiple: true },
   exclude: { type: 'string', multiple: true },
   scanner: { type: 'string', multiple: true },
@@ -158,6 +160,7 @@ export function toOptions(values, paths = []) {
     dockerDepth: integer(values.depth, '--depth'),
     containers: values.container,
     maxDepth: integer(values['max-depth'], '--max-depth'),
+    scanBudget: values['scan-budget'],
     only: values.only,
     exclude: values.exclude,
     scanners: values.scanner,
@@ -294,9 +297,18 @@ class Cli {
 
   async scan(options, values) {
     const report = await this.api.scan(options);
+    return this.printScan(report, options, values);
+  }
+
+  async printScan(report, options, values) {
     const audit = scanAudit(report);
     if (options.audit !== false) {
-      await writeAudit(audit, options);
+      try {
+        await writeAudit(audit, options);
+      } catch (error) {
+        report.audit = { file: null, error: error.message, code: error.code };
+        this.io.stderr(`Audit log: not written (${error.message})`);
+      }
     }
     this.print(values, report, () =>
       [
@@ -305,7 +317,7 @@ class Cli {
         'Run `dss clean --tier safe` to see the plan, add --yes to delete.',
       ].join('\n')
     );
-    return report.errors.length > 0 ? 1 : 0;
+    return report.aborted ? 130 : report.errors.length > 0 ? 1 : 0;
   }
 
   async loadReport(options, values) {
@@ -360,6 +372,9 @@ class Cli {
 
   async clean(options, values) {
     const report = await this.loadReport(options, values);
+    if (report.aborted) {
+      return this.printScan(report, options, values);
+    }
     const audit = await this.consent(values, (extra) =>
       this.api.clean(report, { ...options, ...extra })
     );
@@ -379,6 +394,9 @@ class Cli {
       throw new UsageError(error.message);
     }
     const report = await this.loadReport(options, values);
+    if (report.aborted) {
+      return this.printScan(report, options, values);
+    }
     const audit = await this.consent(values, (extra) =>
       this.api.emergency({ ...options, report, ...extra })
     );
@@ -441,6 +459,10 @@ export async function runCli(argv, deps = {}) {
     if (deps.signal) {
       options.signal = deps.signal;
     }
+    options.onScanProgress = (progress) =>
+      io.stderr(
+        `[scan] ${progress.env}: ${progress.scanner}, ${Math.round(progress.elapsedMs / 1000)}s, ${progress.items} items (${progress.status})`
+      );
     return await dispatch(
       new Cli(io, deps.api ?? API),
       command,

package/src/docker/cli.js

@@ -306,12 +306,16 @@ export class DockerCli {
     const stream = this.executor.spawn(
       this.argv(['cp', `${id}:${source}`, '-'])
     );
-    return pipeInto(stream, [
-      'tar',
-      '-x',
-      '-C',
-      destination,
-      ...excludes.map((pattern) => `--exclude=${pattern}`),
-    ]);
+    return pipeInto(
+      stream,
+      [
+        'tar',
+        '-x',
+        '-C',
+        destination,
+        ...excludes.map((pattern) => `--exclude=${pattern}`),
+      ],
+      { signal: this.executor.signal }
+    );
   }
 }

package/src/docker/scan.js

@@ -22,6 +22,7 @@
 
 import { backupDirectory } from '../audit.js';
 import { ShellEnv } from '../env/shell.js';
+import { checkSignal, signalExecutor } from '../env/scope.js';
 import { containerExecutor, trace } from '../exec.js';
 import { block, makeItem } from '../items.js';
 import { DockerCli, parseDockerSize } from './cli.js';
@@ -380,6 +381,7 @@ class DockerScan {
   }
 
   async daemon(record, depth, inherited) {
+    checkSignal(this.options.signal);
     const docker = new DockerCli(record.executor);
     const info = await docker.info();
     if (!info) {
@@ -443,6 +445,7 @@ class DockerScan {
       })),
     };
     for (const ps of all) {
+      checkSignal(this.options.signal);
       ps.sizes = await this.bestEffort(
         record.env.id,
         () => docker.containerSize(ps.ID),
@@ -544,6 +547,7 @@ class DockerScan {
       }))
     );
     for (const ps of all) {
+      checkSignal(this.options.signal);
       const selected = selection.ids.has(ps.ID);
       if (!selected) {
         continue;
@@ -597,7 +601,10 @@ class DockerScan {
 
   async running(record, depth, ps, entry, inspected) {
     const label = `${record.env.label}/${entry.name}`;
-    const executor = containerExecutor(record.executor, ps.ID, { label });
+    const executor = signalExecutor(
+      containerExecutor(record.executor, ps.ID, { label }),
+      this.options.signal
+    );
     const env = new ShellEnv(executor, {
       id: `${record.env.id}/${entry.name}`,
       label,
@@ -614,14 +621,20 @@ class DockerScan {
     entry.env = env.id;
     entry.scanned = true;
     const chain = [...record.chain, { containerId: ps.ID, name: entry.name }];
-    this.result.environments.push({
+    const descriptor = {
       id: env.id,
       label,
       kind: 'container',
       depth,
       chain,
-    });
+    };
+    this.result.environments.push(descriptor);
     this.result.items.push(...(await this.scanEnvironment(env)));
+    Object.assign(descriptor, {
+      scanStatus: env.scanStatus,
+      hint: env.probeHint ?? null,
+      layerSource: env.writableLayer?.source ?? null,
+    });
     if (await env.which('docker')) {
       try {
         await this.daemon({ env, executor, chain }, depth, true);
@@ -650,6 +663,14 @@ export async function scanDocker(hostRecord, options, scanEnvironment) {
   if (!(await hostRecord.env.which('docker'))) {
     return scan.result;
   }
-  await scan.daemon(hostRecord, 0, false);
+  try {
+    await scan.daemon(hostRecord, 0, false);
+  } catch (error) {
+    scan.result.errors.push({
+      env: hostRecord.env.id,
+      scanner: 'docker',
+      message: error.message,
+    });
+  }
   return scan.result;
 }

package/src/docker/upperdir.js

@@ -0,0 +1,44 @@
+import { batches } from '../env/batches.js';
+
+/** Root overlay from the running container's mount namespace, read by its daemon. */
+export function rootUpperdir(mountinfo) {
+  for (const line of mountinfo.split('\n')) {
+    const [mount, filesystem] = line.split(' - ');
+    if (mount.split(' ')[4] !== '/' || !filesystem?.startsWith('overlay ')) {
+      continue;
+    }
+    const options = filesystem.split(' ').slice(2).join(' ');
+    const upper = /(?:^|,)upperdir=([^,]+)/.exec(options)?.[1];
+    if (upper) {
+      return upper.replace(/\\([0-7]{3})/g, (_, code) =>
+        String.fromCharCode(Number.parseInt(code, 8))
+      );
+    }
+  }
+  return null;
+}
+
+/** One transport call per bounded batch; null means inaccessible, never guessed. */
+export async function upperdirBytes(executor, paths) {
+  const measured = new Map();
+  const script = `for t do
+    printf '\\001%s\\n' "$t"
+    value=$(LC_ALL=C du -skx -- "$t" 2>&1)
+    code=$?
+    if [ "$code" = 0 ]; then printf '%s\\n' "$value" | cut -f1
+    else case "$value" in *'No such file or directory'*) echo 0 ;; *) echo unknown ;; esac; fi
+  done`;
+  for (const batch of batches(paths)) {
+    const result = await executor.run(['sh', '-c', script, 'dss', ...batch]);
+    for (const record of result.stdout.split('\u0001').slice(1)) {
+      const newline = record.indexOf('\n'),
+        target = record.slice(0, newline),
+        value = record.slice(newline + 1).trim();
+      measured.set(
+        target,
+        result.code === 0 && /^\d+$/.test(value) ? Number(value) * 1024 : null
+      );
+    }
+  }
+  return measured;
+}