@link-assistant/hive-mind 2.13.5 → 2.14.0

package/CHANGELOG.md

@@ -1,5 +1,15 @@
 # @link-assistant/hive-mind
 
+## 2.14.0
+
+### Minor Changes
+
+- e796988: Add the experimental `--use-router` option, which stops handing a Docker-isolated task the operator's real subscription. With the flag, `~/.claude`, `~/.claude.json`, `~/.codex` and `~/.agents` are mounted only into a reference-counted `hive-mind-router` sidecar (pinned to `ghcr.io/link-assistant/router:0.119.0`) on an internal Docker network; each task is joined to that network and given its own `la_sk_…` token scoped to the one repository it works on, so every model request lands in that token's own redacted log. The sidecar runs while at least one task holds a lease and is stopped when the last one is released, with the Telegram bot reconciling leases against Docker so a killed bot cannot leave it running forever. At end of task the agent's session data is drained into the router's data volume for audit, and that volume is never removed by any code path.
+
+  GitHub is routed too: the sidecar terminates TLS itself with a certificate covering `api.github.com`, and the task container resolves that name to the router, so an unmodified `gh` is mediated without holding a GitHub credential of its own. `git` pushes through the router's `/git/*` proxy, which refuses remote-ref deletions and, from router `0.110.0`, non-fast-forward updates as well — it asks GitHub's compare API whether the proposed tip is ahead and fails closed when it cannot tell. A read-only `pre-push` hook in the task covers the same ground locally and for non-routed remotes. `--model formal-ai` is served through the router as a stored OpenAI-compatible provider, so it is audited like every other model.
+
+  Nothing changes without the flag. The remaining limits are documented and printed before every routed run: destructive GitHub API calls are blocked by HTTP method rather than by effect, so `PUT` branch protection, `PUT` rulesets, `POST` transfer and `PATCH` visibility/archived/default_branch are still forwarded (upstream `link-assistant/router#329`) — which puts branch protection itself within reach of a routed token; the Formal AI sidecar's own upstream calls are not routed; and the router resolves exact model ids only, not aliases, so the tier-shaped surface (`--plan`, `--escalate`, fallback chains) needs dated ids on a routed run. New docs: `docs/ROUTER.md`, `docs/COLLECTING-LOGS.md`, `docs/case-studies/issue-2164/` and `examples/collect-logs.mjs`.
+
 ## 2.13.5
 
 ### Patch Changes

package/README.hi.md

@@ -88,6 +88,8 @@ Hive Mind में औसत प्रोग्रामर से अलग
 
 Hive Mind अपने प्रबंधित terminal, log, telemetry, upload और GitHub publication paths में पहचाने गए credentials को sanitize करता है। Scanner विफल होने पर publication रोक दिया जाता है, लेकिन यह किसी autonomous tool को किसी असंबंधित exfiltration path का उपयोग करने से नहीं रोकता। Masking contract, supported formats, maintenance process और incident-response guidance के लिए [Credential sanitization](./docs/CREDENTIAL-SANITIZATION.hi.md) देखें।
 
+डिफ़ॉल्ट रूप से Docker में आइसोलेट किए गए कार्यों को ऑपरेटर के Claude/Codex क्रेडेंशियल मिल जाते हैं। `--use-router` उन्हें रोक देता है और मॉडल ट्रैफ़िक को `hive-mind-router` sidecar से भेजता है, जिससे हर कार्य को अपना सीमित दायरे वाला टोकन और अपना request लॉग मिलता है — देखें [राउटर आइसोलेशन](./docs/ROUTER.hi.md) (प्रयोगात्मक) और [लॉग एकत्र करना](./docs/COLLECTING-LOGS.hi.md)।
+
 `hive.mjs` चलाने के लिए न्यूनतम सिस्टम आवश्यकताएँ:
 
 ```

package/README.md

@@ -88,6 +88,8 @@ We strongly recommend:
 
 Hive Mind sanitizes recognized credentials in its maintained terminal, log, telemetry, upload, and GitHub publication paths. Publication is blocked if the scanner fails, but this does not prevent an autonomous tool from using an unrelated exfiltration path. See [Credential sanitization](./docs/CREDENTIAL-SANITIZATION.md) for the masking contract, supported formats, maintenance process, and incident-response guidance.
 
+Docker-isolated tasks receive the operator's Claude/Codex credentials by default. `--use-router` withholds them and routes model traffic through a `hive-mind-router` sidecar instead, giving each task its own scoped token and its own request log — see [Router isolation](./docs/ROUTER.md) (experimental) and [Collecting logs](./docs/COLLECTING-LOGS.md).
+
 Minimum system requirements to run `hive.mjs`:
 
 ```

package/README.ru.md

@@ -88,6 +88,8 @@ Hive Mind обладает высоким уровнем творчества,
 
 Hive Mind скрывает распознанные учётные данные в обслуживаемых им путях вывода терминала, журналов, телеметрии, загрузок и публикаций GitHub. При сбое сканера публикация блокируется, однако это не мешает автономному инструменту использовать посторонний канал утечки. Соглашение о маскировании, поддерживаемые форматы, процесс сопровождения и рекомендации по реагированию на инциденты приведены в разделе [Санитизация учётных данных](./docs/CREDENTIAL-SANITIZATION.ru.md).
 
+По умолчанию задачи, изолированные в Docker, получают учётные данные Claude/Codex оператора. `--use-router` их не передаёт и вместо этого направляет трафик модели через sidecar `hive-mind-router`, выдавая каждой задаче собственный ограниченный токен и собственный журнал запросов — см. [Изоляция через роутер](./docs/ROUTER.ru.md) (экспериментально) и [Сбор журналов](./docs/COLLECTING-LOGS.ru.md).
+
 Минимальные системные требования для запуска `hive.mjs`:
 
 ```

package/README.zh.md

@@ -88,6 +88,8 @@ Hive Mind 具备与普通程序员无异的高度创造力。当需求不明确
 
 Hive Mind 会在其维护的终端、日志、遥测、上传和 GitHub 发布路径中清理可识别的凭证。如果扫描器失败，发布会被阻止；但这无法阻止自主工具通过不相关的路径泄露数据。有关掩码约定、支持的格式、维护流程和事件响应指南，请参阅[凭证清理](./docs/CREDENTIAL-SANITIZATION.zh.md)。
 
+默认情况下，Docker 隔离的任务会拿到操作者的 Claude/Codex 凭据。`--use-router` 不再传入这些凭据，而是让模型流量经由 `hive-mind-router` sidecar 转发，使每个任务获得仅限自身的令牌和自己的请求日志——参见[路由器隔离](./docs/ROUTER.zh.md)（实验性）与[收集日志](./docs/COLLECTING-LOGS.zh.md)。
+
 运行 `hive.mjs` 的最低系统要求：
 
 ```

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@link-assistant/hive-mind",
-  "version": "2.13.5",
+  "version": "2.14.0",
   "description": "AI-powered issue solver and hive mind for collaborative problem solving",
   "main": "src/hive.mjs",
   "type": "module",

package/src/docker-sidecar.lib.mjs

@@ -0,0 +1,276 @@
+/**
+ * Mechanics shared by every on-demand Docker sidecar Hive Mind runs.
+ *
+ * The Formal AI sidecar (issue #2146) and the router sidecar (issue #2164) are
+ * different services with different reasons to exist, but their *mechanics* are
+ * the same problem solved twice: one container shared by concurrent tasks, a
+ * lease per task, a durable JSON record that is only ever a cache of what
+ * Docker actually reports, and an exclusive lock so a launch and an update can
+ * never interleave.
+ *
+ * Those mechanics live here so the two lifecycles stay in step — a fix to lease
+ * reconciliation or to the network guard applies to both — and so each sidecar
+ * module is left holding only the part that is genuinely its own: which image,
+ * which mounts, which readiness check, and what the task is handed.
+ *
+ * @see https://github.com/link-assistant/hive-mind/issues/2146
+ * @see https://github.com/link-assistant/hive-mind/issues/2164
+ */
+
+import { execFile } from 'node:child_process';
+import fs from 'node:fs';
+import path from 'node:path';
+import { promisify } from 'node:util';
+
+import { resolveBotStateDir } from './session-store.lib.mjs';
+
+const execFileAsync = promisify(execFile);
+
+export const DEFAULT_DOCKER_TIMEOUT_MS = 120_000;
+// Pulling a sidecar image is the one Docker call that legitimately takes many
+// minutes, so it gets its own budget instead of the general command timeout.
+export const DEFAULT_IMAGE_TIMEOUT_MS = 600_000;
+
+export const sleep = ms => new Promise(resolve => setTimeout(resolve, ms));
+
+/** Run `docker …` and return trimmed stdout. Throws on a non-zero exit. */
+export const dockerText = async (run, args, { timeoutMs = DEFAULT_DOCKER_TIMEOUT_MS } = {}) => {
+  const result = await run('docker', args, { encoding: 'utf8', timeout: timeoutMs });
+  return String(result?.stdout ?? '').trim();
+};
+
+/** Run `docker …` for its effect only, reporting success as a boolean. */
+export const dockerOk = async (run, args, options) => {
+  try {
+    await dockerText(run, args, options);
+    return true;
+  } catch {
+    return false;
+  }
+};
+
+/** The message a failed `docker` invocation should be reported with. */
+export const dockerErrorMessage = error => error?.stderr?.toString?.().trim() || error?.message || String(error);
+
+/**
+ * Inspect a container without treating "absent" as an error.
+ *
+ * @returns {Promise<{exists: boolean, running: boolean, image: string|null, imageDigest: string|null}>}
+ */
+export const inspectDockerContainer = async (name, { run = execFileAsync, timeoutMs } = {}) => {
+  try {
+    const raw = await dockerText(run, ['inspect', name, '--format', '{{.State.Running}}|{{.Config.Image}}|{{.Image}}'], { timeoutMs });
+    const [running, image, imageDigest] = raw.split('|');
+    return { exists: true, running: running === 'true', image: image || null, imageDigest: imageDigest || null };
+  } catch {
+    return { exists: false, running: false, image: null, imageDigest: null };
+  }
+};
+
+/** Resolve the local content digest of an image reference, or null when it is absent. */
+export const readDockerImageDigest = async (image, { run = execFileAsync, timeoutMs } = {}) => {
+  try {
+    return (await dockerText(run, ['image', 'inspect', image, '--format', '{{.Id}}'], { timeoutMs })) || null;
+  } catch {
+    return null;
+  }
+};
+
+/** A container's IPv4 address on one named network, or null when it has none. */
+export const readDockerContainerAddress = async (containerName, network, { run = execFileAsync, timeoutMs } = {}) => {
+  try {
+    return (await dockerText(run, ['inspect', containerName, '--format', `{{with index .NetworkSettings.Networks "${network}"}}{{.IPAddress}}{{end}}`], { timeoutMs })) || null;
+  } catch {
+    return null;
+  }
+};
+
+/**
+ * Create the private network a sidecar and its tasks share.
+ *
+ * `--internal` is the security requirement: the sidecar endpoint must not be
+ * published to the host and must not be reachable from any other network. An
+ * existing network that is *not* internal is a stale artifact from an older
+ * deployment and is replaced — but only while nothing is attached to it, since
+ * removing a network out from under a running container would break it.
+ */
+export const ensureInternalDockerNetwork = async ({ name, label, run = execFileAsync, timeoutMs, log = null, verbose = false, logPrefix = 'docker-sidecar' } = {}) => {
+  // `null` means "absent", which is different from "present but not internal".
+  let internal = null;
+  let containers = 0;
+  try {
+    const raw = await dockerText(run, ['network', 'inspect', name, '--format', '{{.Internal}}|{{len .Containers}}'], { timeoutMs });
+    const [internalFlag, containerCount] = raw.split('|');
+    internal = internalFlag === 'true';
+    containers = Number(containerCount) || 0;
+  } catch {
+    // Absent; fall through to creation.
+  }
+
+  if (internal === true) return { created: false, internal: true };
+
+  if (internal === false) {
+    if (containers > 0) {
+      if (log) await log(`⚠️ Network '${name}' is not internal but still has ${containers} attached container(s); leaving it in place`);
+      return { created: false, internal: false };
+    }
+    if (verbose && log) await log(`[VERBOSE] ${logPrefix}: replacing non-internal network '${name}'`);
+    await dockerOk(run, ['network', 'rm', name], { timeoutMs });
+  }
+
+  await dockerText(run, ['network', 'create', '--internal', '--label', `${label}=network`, name], { timeoutMs });
+  if (verbose && log) await log(`[VERBOSE] ${logPrefix}: created internal network '${name}'`);
+  return { created: true, internal: true };
+};
+
+/**
+ * Add a network to an already-created container.
+ *
+ * A single `docker run --network` *replaces* the container's default bridge, so
+ * an `--internal` network passed that way would also cut the container off from
+ * GitHub and the package registries. Attaching afterwards is additive, which is
+ * what both sidecars need — for the task container, and for the router sidecar
+ * itself, which must keep its outbound route to the vendor APIs.
+ */
+export const attachDockerNetwork = async ({ network, container, alias = null, run = execFileAsync, timeoutMs, log = null, verbose = false, logPrefix = 'docker-sidecar' } = {}) => {
+  if (!container) return { attached: false, error: 'no container' };
+  const args = ['network', 'connect'];
+  if (alias) args.push('--alias', alias);
+  args.push(network, container);
+  try {
+    await dockerText(run, args, { timeoutMs });
+    if (verbose && log) await log(`[VERBOSE] ${logPrefix}: attached '${container}' to '${network}'${alias ? ` as '${alias}'` : ''}`);
+    return { attached: true, error: null };
+  } catch (error) {
+    const message = dockerErrorMessage(error);
+    // Docker reports an already-attached container as an error; that is success.
+    if (/already exists in network/i.test(message)) return { attached: true, error: null };
+    if (log) await log(`⚠️ Could not attach '${container}' to network '${network}': ${message}`);
+    return { attached: false, error: message };
+  }
+};
+
+/** Create a named volume if it is missing. Never removed by any caller: it holds the data the sidecar exists to keep. */
+export const ensureDockerVolume = async ({ name, label, role = 'data', run = execFileAsync, timeoutMs, log = null, verbose = false, logPrefix = 'docker-sidecar' } = {}) => {
+  if (await dockerOk(run, ['volume', 'inspect', name], { timeoutMs })) return { created: false };
+  await dockerText(run, ['volume', 'create', '--label', `${label}=${role}`, name], { timeoutMs });
+  if (verbose && log) await log(`[VERBOSE] ${logPrefix}: created volume '${name}'`);
+  return { created: true };
+};
+
+/** Path of a sidecar's durable record inside the bot state directory. */
+export const resolveSidecarStatePath = (fileName, env = process.env) => path.join(resolveBotStateDir(env), fileName);
+
+/** Read a durable sidecar record. A missing or corrupt file is an empty record, never a throw. */
+export const readSidecarState = ({ fileName, emptyState, env = process.env, fsImpl = fs } = {}) => {
+  try {
+    const parsed = JSON.parse(fsImpl.readFileSync(resolveSidecarStatePath(fileName, env), 'utf8'));
+    return { ...emptyState, ...parsed, leases: Array.isArray(parsed?.leases) ? parsed.leases : [] };
+  } catch {
+    return { ...emptyState, leases: [] };
+  }
+};
+
+/**
+ * Persist a sidecar record atomically so a crash mid-write cannot corrupt it.
+ *
+ * `mode` exists because the router's record holds its JWT signing secret, which
+ * mints subscription access: that file must not be world-readable (issue #2164).
+ */
+export const writeSidecarState = (state, { fileName, env = process.env, fsImpl = fs, mode = 0o600 } = {}) => {
+  const target = resolveSidecarStatePath(fileName, env);
+  fsImpl.mkdirSync(path.dirname(target), { recursive: true });
+  const temporary = `${target}.tmp`;
+  fsImpl.writeFileSync(temporary, `${JSON.stringify(state, null, 2)}\n`, { encoding: 'utf8', mode });
+  fsImpl.renameSync(temporary, target);
+  return state;
+};
+
+/**
+ * How long a lease whose container has never been seen running is kept.
+ *
+ * A lease is taken *before* start-command creates the task container, because
+ * the endpoint and the token have to be known when the task's environment is
+ * built. During that window the container legitimately does not exist yet — and
+ * creating it can take a long time when the isolation image still has to be
+ * pulled.
+ */
+export const LEASE_START_GRACE_MS = 60 * 60 * 1000;
+
+/**
+ * Drop leases whose task container no longer runs, so a crashed run cannot pin
+ * a sidecar up forever.
+ *
+ * Liveness is re-derived from Docker on every call rather than trusted from the
+ * store, which is what lets a restarted bot converge instead of orphaning.
+ *
+ * @param {Array<object>} leases
+ * @param {{onDropped?: (lease: object) => Promise<void>|void}} options
+ *   `onDropped` lets a sidecar clean up what the lease owned — the router
+ *   revokes the task's token there.
+ */
+export const reconcileSidecarLeases = async (leases, { run = execFileAsync, timeoutMs, log = null, verbose = false, now = () => Date.now(), onDropped = null, logPrefix = 'docker-sidecar' } = {}) => {
+  const live = [];
+  for (const lease of leases) {
+    if (!lease?.sessionId) continue;
+    const container = await inspectDockerContainer(lease.sessionId, { run, timeoutMs });
+    if (container.exists && container.running) {
+      live.push(lease.containerSeen ? lease : { ...lease, containerSeen: true });
+      continue;
+    }
+    if (!lease.containerSeen) {
+      const age = now() - (Date.parse(lease.acquiredAt ?? '') || 0);
+      if (age < LEASE_START_GRACE_MS) {
+        if (verbose && log) await log(`[VERBOSE] ${logPrefix}: keeping lease '${lease.sessionId}' whose container has not appeared yet (${Math.round(age / 1000)}s into the ${Math.round(LEASE_START_GRACE_MS / 1000)}s launch grace)`);
+        live.push(lease);
+        continue;
+      }
+    }
+    if (verbose && log) await log(`[VERBOSE] ${logPrefix}: dropping stale lease '${lease.sessionId}' (container exists=${container.exists} running=${container.running})`);
+    if (onDropped) await onDropped(lease);
+  }
+  return live;
+};
+
+/**
+ * Periodic best-effort maintenance timer shared by the sidecar lifecycles.
+ *
+ * Both the Formal AI and the router sidecar need the same thing: a tick that
+ * runs immediately, repeats on an interval, never keeps the process alive on
+ * shutdown, and never rejects into the bot's event loop. Keeping one
+ * implementation means a fix to that shape applies to both.
+ *
+ * @returns {{stop: () => void}}
+ */
+export const startSidecarMaintenance = ({ runTick, logPrefix = 'sidecar-maintenance', env = process.env, log = null, verbose = false, intervalMs, setIntervalImpl = setInterval, clearIntervalImpl = clearInterval } = {}) => {
+  const tick = () => {
+    runTick({ env, log, verbose }).catch(error => {
+      console.error(`[${logPrefix}] tick failed: ${error?.message || error}`);
+    });
+  };
+
+  const timer = setIntervalImpl(tick, intervalMs);
+  timer?.unref?.();
+  tick();
+  return {
+    stop: () => clearIntervalImpl(timer),
+  };
+};
+
+export default {
+  attachDockerNetwork,
+  dockerErrorMessage,
+  dockerOk,
+  dockerText,
+  ensureDockerVolume,
+  ensureInternalDockerNetwork,
+  inspectDockerContainer,
+  readDockerContainerAddress,
+  readDockerImageDigest,
+  readSidecarState,
+  reconcileSidecarLeases,
+  resolveSidecarStatePath,
+  sleep,
+  startSidecarMaintenance,
+  writeSidecarState,
+};

package/src/formal-ai-maintenance.lib.mjs

@@ -21,6 +21,7 @@
  */
 
 import { updateAgenticClisWhenIdle } from './agentic-cli-updater.lib.mjs';
+import { startSidecarMaintenance } from './docker-sidecar.lib.mjs';
 import { reconcileFormalAiSidecar, stopFormalAiSidecar, withFormalAiSidecarLock } from './formal-ai-sidecar.lib.mjs';
 import { updateFormalAiSidecarWhenIdle } from './formal-ai-updater.lib.mjs';
 
@@ -88,19 +89,6 @@ export const runFormalAiMaintenanceTick = async ({ env = process.env, run, log =
  *
  * @returns {{stop: () => void}}
  */
-export const startFormalAiMaintenance = ({ env = process.env, log = null, verbose = false, intervalMs = DEFAULT_FORMAL_AI_MAINTENANCE_INTERVAL_MS, setIntervalImpl = setInterval, clearIntervalImpl = clearInterval, runTick = runFormalAiMaintenanceTick } = {}) => {
-  const tick = () => {
-    runTick({ env, log, verbose }).catch(error => {
-      console.error(`[formal-ai-maintenance] tick failed: ${error?.message || error}`);
-    });
-  };
-
-  const timer = setIntervalImpl(tick, intervalMs);
-  timer?.unref?.();
-  tick();
-  return {
-    stop: () => clearIntervalImpl(timer),
-  };
-};
+export const startFormalAiMaintenance = ({ env = process.env, log = null, verbose = false, intervalMs = DEFAULT_FORMAL_AI_MAINTENANCE_INTERVAL_MS, setIntervalImpl = setInterval, clearIntervalImpl = clearInterval, runTick = runFormalAiMaintenanceTick } = {}) => startSidecarMaintenance({ runTick, logPrefix: 'formal-ai-maintenance', env, log, verbose, intervalMs, setIntervalImpl, clearIntervalImpl });
 
 export default { DEFAULT_FORMAL_AI_MAINTENANCE_INTERVAL_MS, runFormalAiMaintenanceTick, startFormalAiMaintenance, stopIdleFormalAiSidecar };

package/src/formal-ai-sidecar.lib.mjs

@@ -43,18 +43,23 @@
 
 import { execFile } from 'node:child_process';
 import fs from 'node:fs';
-import path from 'node:path';
 import { promisify } from 'node:util';
 
 import { FORMAL_AI_MINIMUM_VERSION, isFormalAiVersionAtLeast } from './formal-ai-version.lib.mjs';
 import { ensureFormalAiSidecarImage, resolveFormalAiSidecarImage } from './formal-ai-image.lib.mjs';
 import { isFormalAiModel } from './formal-ai-model.lib.mjs';
 import { getModelFromArgs } from './model-args.lib.mjs';
-import { resolveBotStateDir } from './session-store.lib.mjs';
 import { withStateLock } from './state-lock.lib.mjs';
+import { attachDockerNetwork, DEFAULT_IMAGE_TIMEOUT_MS, dockerOk, dockerText, ensureDockerVolume, ensureInternalDockerNetwork, inspectDockerContainer, readDockerContainerAddress, readDockerImageDigest, readSidecarState, reconcileSidecarLeases, resolveSidecarStatePath, sleep, writeSidecarState } from './docker-sidecar.lib.mjs';
 
 const execFileAsync = promisify(execFile);
 
+// Re-exported because callers of this module have always imported them from
+// here; the implementations now live in the shared sidecar module.
+export { inspectDockerContainer, readDockerImageDigest };
+
+const LOG_PREFIX = 'formal-ai-sidecar';
+
 /** Container, network, volume and alias names. Stable so reconciliation works across restarts. */
 export const FORMAL_AI_SIDECAR_CONTAINER_NAME = 'hive-mind-formal-ai';
 export const FORMAL_AI_SIDECAR_NETWORK_NAME = 'hive-mind-formal-ai';
@@ -92,17 +97,13 @@ export const FORMAL_AI_SIDECAR_LABEL = 'com.link-assistant.hive-mind.formal-ai';
 
 const STATE_FILE_NAME = 'formal-ai-sidecar.json';
 const SIDECAR_LOCK_NAME = 'formal-ai-sidecar';
-const DEFAULT_DOCKER_TIMEOUT_MS = 120_000;
 // Pulling a sidecar image is the one Docker call that legitimately takes many
 // minutes, so it gets its own budget instead of the general command timeout.
-const DEFAULT_IMAGE_TIMEOUT_MS = 600_000;
 const DEFAULT_HEALTH_ATTEMPTS = 60;
 const DEFAULT_HEALTH_DELAY_MS = 1000;
 
 const EMPTY_STATE = Object.freeze({ version: 1, image: null, imageDigest: null, startedAt: null, leases: [], lastUpdate: null });
 
-const sleep = ms => new Promise(resolve => setTimeout(resolve, ms));
-
 /**
  * True when a task will be driven by Formal AI.
  *
@@ -149,27 +150,13 @@ export const isFormalAiSidecarEnabled = (env = process.env) => {
   return !['0', 'false', 'no', 'off'].includes(raw);
 };
 
-export const resolveFormalAiSidecarStatePath = (env = process.env) => path.join(resolveBotStateDir(env), STATE_FILE_NAME);
+export const resolveFormalAiSidecarStatePath = (env = process.env) => resolveSidecarStatePath(STATE_FILE_NAME, env);
 
 /** Read the durable sidecar record. A missing or corrupt file is an empty record, never a throw. */
-export const readFormalAiSidecarState = ({ env = process.env, fsImpl = fs } = {}) => {
-  try {
-    const parsed = JSON.parse(fsImpl.readFileSync(resolveFormalAiSidecarStatePath(env), 'utf8'));
-    return { ...EMPTY_STATE, ...parsed, leases: Array.isArray(parsed?.leases) ? parsed.leases : [] };
-  } catch {
-    return { ...EMPTY_STATE, leases: [] };
-  }
-};
+export const readFormalAiSidecarState = ({ env = process.env, fsImpl = fs } = {}) => readSidecarState({ fileName: STATE_FILE_NAME, emptyState: EMPTY_STATE, env, fsImpl });
 
 /** Persist the sidecar record atomically so a crash mid-write cannot corrupt it. */
-export const writeFormalAiSidecarState = (state, { env = process.env, fsImpl = fs } = {}) => {
-  const target = resolveFormalAiSidecarStatePath(env);
-  fsImpl.mkdirSync(path.dirname(target), { recursive: true });
-  const temporary = `${target}.tmp`;
-  fsImpl.writeFileSync(temporary, `${JSON.stringify(state, null, 2)}\n`, 'utf8');
-  fsImpl.renameSync(temporary, target);
-  return state;
-};
+export const writeFormalAiSidecarState = (state, { env = process.env, fsImpl = fs } = {}) => writeSidecarState(state, { fileName: STATE_FILE_NAME, env, fsImpl });
 
 /**
  * Serialize every sidecar mutation — task launches, task releases and image
@@ -180,35 +167,6 @@ export const writeFormalAiSidecarState = (state, { env = process.env, fsImpl = f
  */
 export const withFormalAiSidecarLock = (fn, options = {}) => withStateLock(SIDECAR_LOCK_NAME, fn, options);
 
-const dockerText = async (run, args, { timeoutMs = DEFAULT_DOCKER_TIMEOUT_MS } = {}) => {
-  const result = await run('docker', args, { encoding: 'utf8', timeout: timeoutMs });
-  return String(result?.stdout ?? '').trim();
-};
-
-const dockerOk = async (run, args, options) => {
-  try {
-    await dockerText(run, args, options);
-    return true;
-  } catch {
-    return false;
-  }
-};
-
-/**
- * Inspect a container without treating "absent" as an error.
- *
- * @returns {Promise<{exists: boolean, running: boolean, image: string|null, imageDigest: string|null}>}
- */
-export const inspectDockerContainer = async (name, { run = execFileAsync, timeoutMs } = {}) => {
-  try {
-    const raw = await dockerText(run, ['inspect', name, '--format', '{{.State.Running}}|{{.Config.Image}}|{{.Image}}'], { timeoutMs });
-    const [running, image, imageDigest] = raw.split('|');
-    return { exists: true, running: running === 'true', image: image || null, imageDigest: imageDigest || null };
-  } catch {
-    return { exists: false, running: false, image: null, imageDigest: null };
-  }
-};
-
 /**
  * The sidecar's IPv4 address on the internal network.
  *
@@ -218,22 +176,7 @@ export const inspectDockerContainer = async (name, { run = execFileAsync, timeou
  * post-attachment would be a needless gamble; the address cannot change during
  * a lease, because an image replacement requires zero leases.
  */
-export const readFormalAiSidecarAddress = async ({ containerName = FORMAL_AI_SIDECAR_CONTAINER_NAME, network = FORMAL_AI_SIDECAR_NETWORK_NAME, run = execFileAsync, timeoutMs } = {}) => {
-  try {
-    return (await dockerText(run, ['inspect', containerName, '--format', `{{with index .NetworkSettings.Networks "${network}"}}{{.IPAddress}}{{end}}`], { timeoutMs })) || null;
-  } catch {
-    return null;
-  }
-};
-
-/** Resolve the local content digest of an image reference, or null when it is absent. */
-export const readDockerImageDigest = async (image, { run = execFileAsync, timeoutMs } = {}) => {
-  try {
-    return (await dockerText(run, ['image', 'inspect', image, '--format', '{{.Id}}'], { timeoutMs })) || null;
-  } catch {
-    return null;
-  }
-};
+export const readFormalAiSidecarAddress = async ({ containerName = FORMAL_AI_SIDECAR_CONTAINER_NAME, network = FORMAL_AI_SIDECAR_NETWORK_NAME, run = execFileAsync, timeoutMs } = {}) => readDockerContainerAddress(containerName, network, { run, timeoutMs });
 
 /**
  * Create the private network the sidecar and its tasks share.
@@ -244,34 +187,7 @@ export const readDockerImageDigest = async (image, { run = execFileAsync, timeou
  * artifact from the Compose deployment and is replaced when nothing is
  * attached to it.
  */
-export const ensureFormalAiNetwork = async ({ run = execFileAsync, timeoutMs, log = null, verbose = false } = {}) => {
-  // `null` means "absent", which is different from "present but not internal".
-  let internal = null;
-  let containers = 0;
-  try {
-    const raw = await dockerText(run, ['network', 'inspect', FORMAL_AI_SIDECAR_NETWORK_NAME, '--format', '{{.Internal}}|{{len .Containers}}'], { timeoutMs });
-    const [internalFlag, containerCount] = raw.split('|');
-    internal = internalFlag === 'true';
-    containers = Number(containerCount) || 0;
-  } catch {
-    // Absent; fall through to creation.
-  }
-
-  if (internal === true) return { created: false, internal: true };
-
-  if (internal === false) {
-    if (containers > 0) {
-      if (log) await log(`⚠️ Formal AI network '${FORMAL_AI_SIDECAR_NETWORK_NAME}' is not internal but still has ${containers} attached container(s); leaving it in place`);
-      return { created: false, internal: false };
-    }
-    if (verbose && log) await log(`[VERBOSE] formal-ai-sidecar: replacing non-internal network '${FORMAL_AI_SIDECAR_NETWORK_NAME}'`);
-    await dockerOk(run, ['network', 'rm', FORMAL_AI_SIDECAR_NETWORK_NAME], { timeoutMs });
-  }
-
-  await dockerText(run, ['network', 'create', '--internal', '--label', `${FORMAL_AI_SIDECAR_LABEL}=network`, FORMAL_AI_SIDECAR_NETWORK_NAME], { timeoutMs });
-  if (verbose && log) await log(`[VERBOSE] formal-ai-sidecar: created internal network '${FORMAL_AI_SIDECAR_NETWORK_NAME}'`);
-  return { created: true, internal: true };
-};
+export const ensureFormalAiNetwork = async ({ run = execFileAsync, timeoutMs, log = null, verbose = false } = {}) => ensureInternalDockerNetwork({ name: FORMAL_AI_SIDECAR_NETWORK_NAME, label: FORMAL_AI_SIDECAR_LABEL, run, timeoutMs, log, verbose, logPrefix: LOG_PREFIX });
 
 /**
  * Create the persisted-memory volume if it is missing and hand it to the
@@ -281,16 +197,12 @@ export const ensureFormalAiNetwork = async ({ run = execFileAsync, timeoutMs, lo
  * task boundaries, sidecar stops, image replacement and rollback.
  */
 export const ensureFormalAiMemoryVolume = async ({ image, run = execFileAsync, timeoutMs, log = null, verbose = false } = {}) => {
-  if (await dockerOk(run, ['volume', 'inspect', FORMAL_AI_MEMORY_VOLUME_NAME], { timeoutMs })) {
-    return { created: false };
-  }
-
-  await dockerText(run, ['volume', 'create', '--label', `${FORMAL_AI_SIDECAR_LABEL}=memory`, FORMAL_AI_MEMORY_VOLUME_NAME], { timeoutMs });
+  const result = await ensureDockerVolume({ name: FORMAL_AI_MEMORY_VOLUME_NAME, label: FORMAL_AI_SIDECAR_LABEL, role: 'memory', run, timeoutMs, log, verbose, logPrefix: LOG_PREFIX });
+  if (!result.created) return result;
   // A fresh named volume is root-owned; the image runs application commands as
   // `box`, so seed the ownership exactly as upstream's own upgrade fixture does.
   await dockerOk(run, ['run', '--rm', '--volume', `${FORMAL_AI_MEMORY_VOLUME_NAME}:${FORMAL_AI_MEMORY_MOUNT}`, '--entrypoint', 'chown', image, '-R', 'box:box', FORMAL_AI_MEMORY_MOUNT], { timeoutMs });
-  if (verbose && log) await log(`[VERBOSE] formal-ai-sidecar: created memory volume '${FORMAL_AI_MEMORY_VOLUME_NAME}'`);
-  return { created: true };
+  return result;
 };
 
 /** Build the `docker run` argv for the sidecar. Exported so tests can assert the contract. */
@@ -369,29 +281,7 @@ export const waitForFormalAiSidecarHealth = async ({ containerName = FORMAL_AI_S
  * `LEASE_START_GRACE_MS` elapses; afterwards, and always once the container has
  * been observed, liveness is Docker's answer alone.
  */
-const LEASE_START_GRACE_MS = 60 * 60 * 1000;
-
-const reconcileLeases = async (leases, { run, timeoutMs, log, verbose, now = () => Date.now() }) => {
-  const live = [];
-  for (const lease of leases) {
-    if (!lease?.sessionId) continue;
-    const container = await inspectDockerContainer(lease.sessionId, { run, timeoutMs });
-    if (container.exists && container.running) {
-      live.push(lease.containerSeen ? lease : { ...lease, containerSeen: true });
-      continue;
-    }
-    if (!lease.containerSeen) {
-      const age = now() - (Date.parse(lease.acquiredAt ?? '') || 0);
-      if (age < LEASE_START_GRACE_MS) {
-        if (verbose && log) await log(`[VERBOSE] formal-ai-sidecar: keeping lease '${lease.sessionId}' whose container has not appeared yet (${Math.round(age / 1000)}s into the ${Math.round(LEASE_START_GRACE_MS / 1000)}s launch grace)`);
-        live.push(lease);
-        continue;
-      }
-    }
-    if (verbose && log) await log(`[VERBOSE] formal-ai-sidecar: dropping stale lease '${lease.sessionId}' (container exists=${container.exists} running=${container.running})`);
-  }
-  return live;
-};
+const reconcileLeases = async (leases, options) => reconcileSidecarLeases(leases, { ...options, logPrefix: LOG_PREFIX });
 
 /**
  * Re-derive the sidecar record from Docker.
@@ -522,17 +412,7 @@ export const acquireFormalAiSidecar = async ({ sessionId, tool = null, model = n
  */
 export const attachTaskToFormalAiNetwork = async ({ sessionId, run = execFileAsync, timeoutMs, log = null, verbose = false } = {}) => {
   if (!sessionId) return { attached: false, error: 'no sessionId' };
-  try {
-    await dockerText(run, ['network', 'connect', FORMAL_AI_SIDECAR_NETWORK_NAME, sessionId], { timeoutMs });
-    if (verbose && log) await log(`[VERBOSE] formal-ai-sidecar: attached task container '${sessionId}' to '${FORMAL_AI_SIDECAR_NETWORK_NAME}'`);
-    return { attached: true, error: null };
-  } catch (error) {
-    const message = error?.stderr?.toString?.().trim() || error?.message || String(error);
-    // Docker reports an already-attached container as an error; that is success.
-    if (/already exists in network/i.test(message)) return { attached: true, error: null };
-    if (log) await log(`⚠️ Could not attach task container '${sessionId}' to the Formal AI network: ${message}`);
-    return { attached: false, error: message };
-  }
+  return attachDockerNetwork({ network: FORMAL_AI_SIDECAR_NETWORK_NAME, container: sessionId, run, timeoutMs, log, verbose, logPrefix: LOG_PREFIX });
 };
 
 /**