@link-assistant/hive-mind 1.78.10 → 1.78.12

package/CHANGELOG.md

@@ -1,5 +1,96 @@
 # @link-assistant/hive-mind
 
+## 1.78.12
+
+### Patch Changes
+
+- 5f60c04: fix(isolation): default nested Docker daemon to fuse-overlayfs so multi-GB images fit on disk + add storage-driver/disk preflight diagnostics (#1914)
+
+  `--isolation docker` was reopened after PR #1915: native Docker isolation and
+  host-image passthrough now work, but the first isolated task on the >30 GB
+  `konard/hive-mind-dind` image still died with:
+
+  ```
+  failed to register layer: no space left on device
+  ```
+
+  even though most layers reported `Already exists` (the daemon was correctly
+  seeded — passthrough is working). The failure was during layer **registration**,
+  not download.
+
+  **Root cause (in this repo).** `Dockerfile.dind` baked `ENV
+DIND_STORAGE_DRIVER="vfs"` (commit 44d2c29e). `vfs` performs **no copy-on-write**:
+  it materializes a full, independent copy of the entire filesystem for _every_
+  layer, so a multi-GB image's on-disk footprint becomes the _sum_ of all
+  cumulative layer sizes — many times the image size — and overflows the disk.
+  Worse, pinning the env var **defeated box-dind's storage-driver auto-detection**
+  (`overlay2 → fuse-overlayfs → vfs`, with graceful fallback): box would otherwise
+  have picked a copy-on-write driver here. `/dev/fuse` is present (the dind
+  container runs `--privileged`), the `fuse-overlayfs` binary ships in box-dind,
+  and `overlay` is in `/proc/filesystems` — so copy-on-write was available the
+  whole time but was being bypassed by the `vfs` pin.
+
+  **Fix.** `Dockerfile.dind` now pins `ENV DIND_STORAGE_DRIVER="fuse-overlayfs"` — a
+  copy-on-write driver that also works overlay-on-overlay (the compatibility reason
+  `vfs` was originally chosen; `overlay2` can fail on the overlay-backed hosts our
+  deploys run on). Under `fuse-overlayfs`, registering a 498 MB top layer on a
+  ~30 GB base costs ~498 MB instead of ~30 GB, so the image fits. Empirically
+  verified in the box-dind environment (`docs/case-studies/issue-1914/data/fuse-overlayfs-capability-proof.log`).
+
+  **Self-diagnosing preflight.** `src/isolation-runner.lib.mjs` gained two probes —
+  `checkDockerStorageDriver()` and `checkDockerDiskSpace()` — wired into
+  `preflightDockerIsolation()`. Before running an isolated task it now warns, with
+  an actionable remedy, when the nested daemon is on `vfs` (even if the image is
+  already present) or when free space at the Docker data root is below 40 GiB, so
+  the next operator hitting this gets a clear breadcrumb instead of a cryptic
+  `no space left on device`. Both probes are best-effort and never throw.
+
+  Added `tests/test-issue-1914-storage-driver-diagnostics.mjs` (34 assertions),
+  extended `tests/test-issue-1914-preflight-passthrough.mjs` and
+  `tests/test-docker-dind-variant.mjs`, refreshed `docs/DOCKER*.md`, and expanded
+  the `docs/case-studies/issue-1914` case study with the reopen timeline, refined
+  root-cause analysis, captured evidence, and an upstream observability request
+  (link-foundation/box#104: warn when the nested daemon lands on `vfs`).
+
+## 1.78.11
+
+### Patch Changes
+
+- 24fb17e: fix(retry): auto-resume on server-side 429 "Server is temporarily limiting requests" rate-limit errors (#1924)
+
+  A long-running solve session (177 turns, ~72 min) was thrown away when the Claude
+  CLI surfaced a **server-side temporary rate limit**:
+
+  ```
+  API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited
+  ```
+
+  The CLI reports this as a `result` event with `is_error: true` and
+  `api_error_status: 429`, and the HTTP response carries `x-should-retry: true`.
+  This is a transient throttle that clears on its own — distinct from an account
+  usage/quota limit (the message literally says "not your usage limit", and there
+  is no reset time to wait for).
+
+  Root cause: the error matched neither `classifyRetryableError` (no pattern for
+  the 429 throttle wording) nor `isUsageLimitError` (correctly, since it is not a
+  quota limit), so it fell through to a hard failure with exit code 1 and **no
+  auto-resume**, unlike every other transient class (overload 500/529, 503,
+  internal server error, request timeout, socket drops).
+
+  Fix: `classifyRetryableError` (in `src/tool-retry.lib.mjs`, the shared classifier
+  used by every tool wrapper — claude, codex, gemini, opencode, qwen, agent) now
+  recognises this throttle and marks it retryable (`isCapacity: false`, so no model
+  switch), so it retries with the session preserved (`--resume`) after a backoff.
+  `src/claude.lib.mjs` additionally detects the structured `api_error_status === 429`
+  directly (robust to wording changes) and logs a verbose diagnostic with the
+  `request_id`. The matcher is narrow so genuine account usage limits stay on the
+  usage-limit reset-time path.
+
+  Added `tests/test-issue-1924-rate-limit-retry.mjs` (18 assertions) and a full
+  case study with timeline, root-cause analysis, upstream references
+  (anthropics/claude-code#53915, #53922), and the captured logs under
+  `docs/case-studies/issue-1924`.
+
 ## 1.78.10
 
 ### Patch Changes

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@link-assistant/hive-mind",
-  "version": "1.78.10",
+  "version": "1.78.12",
   "description": "AI-powered issue solver and hive mind for collaborative problem solving",
   "main": "src/hive.mjs",
   "type": "module",

package/src/claude.lib.mjs

@@ -648,6 +648,7 @@ export const executeClaudeCommand = async params => {
     let is503Error = false;
     let isInternalServerError = false;
     let isRequestTimeout = false;
+    let isRateLimitError = false; // Issue #1924: server-side 429 temporary rate limiting
     let apiMarkedNotRetryable = false;
     let resultNumTurns = 0;
     let stderrErrors = [];
@@ -977,6 +978,14 @@ export const executeClaudeCommand = async params => {
                     isRequestTimeout = true;
                     await log('⏱️ Detected request timeout from Claude CLI (will retry with --resume)', { verbose: true });
                   }
+                  // Issue #1924: Server-side temporary rate limiting (HTTP 429) — a transient
+                  // throttle, not an account usage limit ("...not your usage limit..."), so retry
+                  // with --resume. The message text is handled by classifyRetryableError; this also
+                  // catches the structured api_error_status if the wording ever changes.
+                  if (data.api_error_status === 429) {
+                    isRateLimitError = true;
+                    await log(`⚠️ Detected server-side rate limiting (429) from Claude CLI (will retry with --resume). request_id=${data.request_id || 'unknown'}`, { verbose: true });
+                  }
                   // Issue #1834: Detect corrupted extended-thinking-block 400 (un-resumable session).
                   // Capture diagnostics (request id, content path) to aid debugging and upstream reports.
                   if ((lastMessage.includes('thinking') || lastMessage.includes('redacted_thinking')) && lastMessage.includes('cannot be modified')) {
@@ -1174,7 +1183,7 @@ export const executeClaudeCommand = async params => {
         return await executeWithRetry();
       }
       // Issues #1331, #1353, #1472/#1475: Unified transient error retry (exponential backoff, session preservation)
-      const isTransientError = isStartupTimeout || isActivityTimeout || isOverloadError || isInternalServerError || is503Error || isRequestTimeout || retryableLastError.isRetryable || (lastMessage.includes('API Error: 500') && (lastMessage.includes('Overloaded') || lastMessage.includes('Internal server error'))) || (lastMessage.includes('API Error: 529') && (lastMessage.includes('overloaded_error') || lastMessage.includes('Overloaded'))) || (lastMessage.includes('api_error') && lastMessage.includes('Overloaded')) || (lastMessage.includes('overloaded_error') && lastMessage.includes('Overloaded')) || lastMessage.includes('API Error: 503') || (lastMessage.includes('503') && (lastMessage.includes('upstream connect error') || lastMessage.includes('remote connection failure'))) || lastMessage === 'Request timed out' || lastMessage.includes('Request timed out');
+      const isTransientError = isStartupTimeout || isActivityTimeout || isOverloadError || isInternalServerError || is503Error || isRequestTimeout || isRateLimitError || retryableLastError.isRetryable || (lastMessage.includes('API Error: 500') && (lastMessage.includes('Overloaded') || lastMessage.includes('Internal server error'))) || (lastMessage.includes('API Error: 529') && (lastMessage.includes('overloaded_error') || lastMessage.includes('Overloaded'))) || (lastMessage.includes('api_error') && lastMessage.includes('Overloaded')) || (lastMessage.includes('overloaded_error') && lastMessage.includes('Overloaded')) || lastMessage.includes('API Error: 503') || (lastMessage.includes('503') && (lastMessage.includes('upstream connect error') || lastMessage.includes('remote connection failure'))) || lastMessage === 'Request timed out' || lastMessage.includes('Request timed out');
       if ((commandFailed || isTransientError) && isTransientError) {
         // Issue #1472/#1475: Startup/activity timeout → 30s–2min backoff; #1353: Request timeout → 5min–1hr; general → 2min–30min
         const isTimeoutRetry = isStartupTimeout || isActivityTimeout;
@@ -1208,7 +1217,7 @@ export const executeClaudeCommand = async params => {
         }
         if (retryCount < maxRetries) {
           const delay = Math.min(initialDelay * Math.pow(retryLimits.retryBackoffMultiplier, retryCount), maxDelay);
-          const errorLabel = isStartupTimeout ? 'Stream startup timeout (Issue #1472/#1475)' : isActivityTimeout ? 'Stream activity timeout (Issue #1472)' : isRequestTimeout ? 'Request timeout' : retryableLastError.label || (isOverloadError || (lastMessage.includes('API Error: 500') && lastMessage.includes('Overloaded')) || (lastMessage.includes('API Error: 529') && lastMessage.includes('Overloaded')) ? `API overload (${lastMessage.includes('529') ? '529' : '500'})` : isInternalServerError || lastMessage.includes('Internal server error') ? 'Internal server error (500)' : '503 network error');
+          const errorLabel = isStartupTimeout ? 'Stream startup timeout (Issue #1472/#1475)' : isActivityTimeout ? 'Stream activity timeout (Issue #1472)' : isRequestTimeout ? 'Request timeout' : retryableLastError.label || (isOverloadError || (lastMessage.includes('API Error: 500') && lastMessage.includes('Overloaded')) || (lastMessage.includes('API Error: 529') && lastMessage.includes('Overloaded')) ? `API overload (${lastMessage.includes('529') ? '529' : '500'})` : isInternalServerError || lastMessage.includes('Internal server error') ? 'Internal server error (500)' : isRateLimitError ? 'Server rate limited (429)' : '503 network error');
           const notRetryableHint = apiMarkedNotRetryable ? ' (API says not retryable — will stop early if no progress)' : '';
           const delayLabel = delay >= 60000 ? `${Math.round(delay / 60000)} min` : `${Math.round(delay / 1000)}s`;
           const retryMode = isStartupTimeout ? ' (fresh start)' : ' (session preserved)';

package/src/isolation-runner.lib.mjs

@@ -47,6 +47,12 @@ const DEFAULT_HOST_DOCKER_SOCK = '/var/run/host-docker.sock';
 // throwaway container — booting the dind image's dockerd entrypoint — purely to
 // check whether bash exists. See issue #1914.
 const DOCKER_ISOLATION_SHELL = 'sh';
+// Free-space floor (GiB) below which the preflight warns that an impending
+// isolation-image pull may fail with `no space left on device`. The Hive Mind
+// isolation images are well over 30 GB extracted, so a host/nested daemon with
+// less headroom than this cannot safely pull one. Diagnostic only — never
+// blocks startup. See issue #1914.
+const DOCKER_ISOLATION_LOW_DISK_GIB = 40;
 
 function normalizeProcessIds(value) {
   if (!value || typeof value !== 'object') return {};
@@ -87,12 +93,13 @@ function maybeAddMount(mounts, source, target, existsSync) {
 /**
  * Resolve the tag used for the Docker isolation image.
  *
- * Defaults to `latest`, but operators can pin it (e.g. to the exact version
- * already present on the host) via `HIVE_MIND_DOCKER_ISOLATION_IMAGE_TAG`.
- * Pinning matters for Docker-in-Docker deployments: the nested daemon starts
- * with an empty image store, so an unpinned `:latest` whose registry digest has
- * drifted from the host copy forces a fresh multi-gigabyte pull on every task.
- * A pinned tag lets a pre-seeded image be reused instead. See issue #1879.
+ * Release Docker images bake this env var from `HIVE_MIND_VERSION`, so a parent
+ * container started via `:latest` still launches child isolation containers from
+ * the same immutable release tag. Local/PR builds fall back to `latest`, and
+ * operators can override the tag explicitly when using custom images. Pinning
+ * matters for Docker-in-Docker deployments: the nested daemon starts with an
+ * empty image store, so a `:latest` digest drift from the host copy forces a
+ * fresh multi-gigabyte pull. See issue #1879.
  */
 export function resolveDockerIsolationImageTag({ env = process.env } = {}) {
   const explicit = String(env.HIVE_MIND_DOCKER_ISOLATION_IMAGE_TAG || '').trim();
@@ -618,6 +625,80 @@ export async function checkDockerImagePresent(image, verbose = false) {
   }
 }
 
+/**
+ * Report the storage driver the (nested) Docker daemon is using.
+ *
+ * `vfs` performs NO copy-on-write — it stores a full copy of every image layer
+ * — so the multi-gigabyte Hive Mind images consume many times their real size
+ * on disk and the first isolated `docker run`/pull dies with
+ * `failed to register layer: no space left on device` (issue #1914 reopen).
+ * The preflight uses this to warn loudly when the daemon is on `vfs` instead of
+ * letting the disk silently overflow mid-task.
+ *
+ * Never throws: returns the lowercased driver name, or `null` when docker is
+ * unavailable / the daemon is unreachable.
+ *
+ * @param {boolean} [verbose] - Enable verbose logging
+ * @returns {Promise<string|null>} e.g. 'fuse-overlayfs', 'overlay2', 'vfs', or null
+ */
+export async function checkDockerStorageDriver(verbose = false) {
+  try {
+    const result = await $({ mirror: false })`docker info --format ${'{{.Driver}}'}`;
+    const driver = (result.stdout?.toString() || '').trim().toLowerCase() || null;
+    if (verbose) console.log(`[VERBOSE] isolation-runner: docker storage driver: ${driver || '(unknown)'}`);
+    return driver;
+  } catch {
+    if (verbose) console.log('[VERBOSE] isolation-runner: docker info unavailable; storage driver unknown');
+    return null;
+  }
+}
+
+/**
+ * Report the free space (in GiB) on the Docker daemon's data root.
+ *
+ * The Hive Mind isolation images are multiple gigabytes; when the nested daemon
+ * has to pull one, it needs room for the extracted layers. This lets the
+ * preflight predict a `no space left on device` failure (issue #1914) instead
+ * of discovering it mid-pull. Resolves the daemon's real data root via
+ * `docker info` and falls back to `/var/lib/docker`, then reads `df -Pk`.
+ *
+ * Never throws: returns `{ availableGiB, dataRoot }`, or `null` when the
+ * information cannot be determined (no docker, no df, unparseable output).
+ *
+ * @param {boolean} [verbose] - Enable verbose logging
+ * @returns {Promise<{availableGiB: number, dataRoot: string}|null>}
+ */
+export async function checkDockerDiskSpace(verbose = false) {
+  try {
+    let dataRoot = '/var/lib/docker';
+    try {
+      const info = await $({ mirror: false })`docker info --format ${'{{.DockerRootDir}}'}`;
+      const root = (info.stdout?.toString() || '').trim();
+      if (root) dataRoot = root;
+    } catch {
+      // Daemon unreachable: fall back to the conventional data root. If df then
+      // fails on it (e.g. the path does not exist) we return null below.
+    }
+
+    const df = await $({ mirror: false })`df -Pk ${dataRoot}`;
+    // `df -P` guarantees one logical line per filesystem (no wrapping). The last
+    // line is the data row: Filesystem 1024-blocks Used Available Capacity Mount
+    const lines = (df.stdout?.toString() || '').trim().split('\n');
+    const cols = (lines[lines.length - 1] || '').trim().split(/\s+/);
+    const availableKb = Number(cols[3]);
+    if (!Number.isFinite(availableKb)) {
+      if (verbose) console.log('[VERBOSE] isolation-runner: could not parse df output for Docker disk space');
+      return null;
+    }
+    const availableGiB = availableKb / (1024 * 1024);
+    if (verbose) console.log(`[VERBOSE] isolation-runner: Docker data root '${dataRoot}' has ${availableGiB.toFixed(1)} GiB free`);
+    return { availableGiB, dataRoot };
+  } catch {
+    if (verbose) console.log('[VERBOSE] isolation-runner: df unavailable; Docker disk space unknown');
+    return null;
+  }
+}
+
 /**
  * Startup preflight for `--isolation docker`.
  *
@@ -637,42 +718,78 @@ export async function checkDockerImagePresent(image, verbose = false) {
  * blocks startup — a misconfigured passthrough should degrade to a slow first
  * task, not a dead bot.
  *
+ * It also surfaces the two root causes of the issue #1914 reopen
+ * (`failed to register layer: no space left on device`): a non-copy-on-write
+ * storage driver (`vfs`, which copies every layer in full) and a Docker data
+ * root with too little free space to hold the >30 GB image. Both are reported
+ * as loud, actionable warnings so the disk overflow is self-diagnosing at
+ * startup instead of surfacing mid-task.
+ *
  * @param {Object} [options]
  * @param {Object} [options.env] - Environment (defaults to process.env)
  * @param {Function} [options.existsSync] - fs.existsSync (injectable for tests)
  * @param {boolean} [options.verbose] - Enable verbose logging
  * @param {Object} [options.logger] - Logger with .log/.warn (defaults to console)
  * @param {Function} [options.checkImagePresent] - Image-presence probe (injectable for tests)
- * @returns {Promise<{image: string, sock: string, socketMounted: boolean, imagePresent: boolean, isDind: boolean, ok: boolean, warnings: string[]}>}
+ * @param {Function} [options.checkStorageDriver] - Storage-driver probe (injectable for tests)
+ * @param {Function} [options.checkDiskSpace] - Disk-space probe (injectable for tests)
+ * @returns {Promise<{image: string, sock: string, socketMounted: boolean, imagePresent: boolean, isDind: boolean, storageDriver: (string|null), storageDriverOk: boolean, diskAvailableGiB: (number|null), ok: boolean, warnings: string[]}>}
  */
 export async function preflightDockerIsolation(options = {}) {
-  const { env = process.env, existsSync = fs.existsSync, verbose = false, logger = console, checkImagePresent = checkDockerImagePresent } = options;
+  const { env = process.env, existsSync = fs.existsSync, verbose = false, logger = console, checkImagePresent = checkDockerImagePresent, checkStorageDriver = checkDockerStorageDriver, checkDiskSpace = checkDockerDiskSpace } = options;
 
   const image = getDockerIsolationImage({ env });
   const sock = resolveHostDockerSock({ env });
   const isDind = shouldRunPrivilegedDockerIsolation(image, env);
   const socketMounted = Boolean(existsSync(sock));
   const imagePresent = Boolean(await checkImagePresent(image, verbose));
-
-  const result = { image, sock, socketMounted, imagePresent, isDind, ok: imagePresent, warnings: [] };
+  const storageDriver = await checkStorageDriver(verbose);
+  const disk = await checkDiskSpace(verbose);
+  const diskAvailableGiB = disk && Number.isFinite(disk.availableGiB) ? disk.availableGiB : null;
+  // Unknown driver (probe returned null) is treated as ok — we only flag the
+  // one driver known to overflow the disk, never block on missing information.
+  const storageDriverOk = storageDriver !== 'vfs';
+
+  const result = { image, sock, socketMounted, imagePresent, isDind, storageDriver, storageDriverOk, diskAvailableGiB, ok: imagePresent, warnings: [] };
   const info = typeof logger.log === 'function' ? logger.log.bind(logger) : () => {};
   const warn = typeof logger.warn === 'function' ? logger.warn.bind(logger) : info;
 
-  if (imagePresent) {
-    info(`✅ Docker isolation image '${image}' is already present locally — isolated tasks reuse it (no multi-GB pull). See issue #1914.`);
-    return result;
+  const preload = `node scripts/preload-dind-isolation-image.mjs --image ${image}`;
+
+  // Root Cause A of the issue #1914 reopen: a non-copy-on-write storage driver.
+  // `vfs` stores a full copy of every image layer, so the multi-GB images
+  // consume many times their size on disk and any layer write (pull, run,
+  // commit) can fail with `failed to register layer: no space left on device`.
+  // This is dangerous even when the image is already present — a task that
+  // commits or pulls more layers still overflows — so we warn independent of
+  // image presence.
+  if (storageDriver === 'vfs') {
+    result.warnings.push(`The Docker daemon backing '--isolation docker' is using the 'vfs' storage driver, which performs NO copy-on-write: ` + `it stores a full copy of every image layer, so the multi-GB Hive Mind images consume many times their size on disk and isolated tasks can fail with 'failed to register layer: no space left on device' (issue #1914). ` + `Switch to a copy-on-write driver: rebuild/redeploy with the current Dockerfile.dind (it defaults to 'fuse-overlayfs'), or for an already-running container add '-e DIND_STORAGE_DRIVER=fuse-overlayfs' to the bot container's 'docker run' and recreate it.`);
   }
 
-  // Image absent: the first isolated task will pull the full image. Explain the
-  // most likely cause and the exact fix instead of letting the operator first
-  // discover it as a surprise multi-gigabyte download mid-task.
-  const preload = `node scripts/preload-dind-isolation-image.mjs --image ${image}`;
-  if (isDind && !socketMounted) {
-    result.warnings.push(`Docker isolation image '${image}' is NOT in the nested Docker daemon and the host Docker socket is not mounted at ${sock}. ` + `box host-image passthrough cannot seed the nested daemon, so the FIRST isolated task will pull the full image (the Hive Mind images are multiple GB). ` + `Fix the deployment: add '-v /var/run/docker.sock:${sock}:ro' and '-e DIND_HOST_PASSTHROUGH_IMAGES="konard/hive-mind konard/hive-mind-dind"' to the bot container's 'docker run', or seed it now with: ${preload}`);
-  } else if (isDind && socketMounted) {
-    result.warnings.push(`Docker isolation image '${image}' is NOT in the nested Docker daemon even though the host Docker socket is mounted at ${sock}. ` + `box host-image passthrough may have skipped it (check DIND_HOST_PASSTHROUGH mode, the DIND_HOST_PASSTHROUGH_IMAGES allowlist, and that the host actually has '${image}' with a registry digest). ` + `The first isolated task will pull the full image. Seed it now with: ${preload}`);
-  } else {
-    result.warnings.push(`Docker isolation image '${image}' is not present locally; the first isolated task will pull it. ` + `If this host already has it under a different tag, pin HIVE_MIND_DOCKER_ISOLATION_IMAGE_TAG, or seed it with: ${preload}`);
+  if (!imagePresent) {
+    // Image absent: the first isolated task will pull the full image. Explain
+    // the most likely cause and the exact fix instead of letting the operator
+    // first discover it as a surprise multi-gigabyte download mid-task.
+    if (isDind && !socketMounted) {
+      result.warnings.push(`Docker isolation image '${image}' is NOT in the nested Docker daemon and the host Docker socket is not mounted at ${sock}. ` + `box host-image passthrough cannot seed the nested daemon, so the FIRST isolated task will pull the full image (the Hive Mind images are multiple GB). ` + `Fix the deployment: add '-v /var/run/docker.sock:${sock}:ro' and '-e DIND_HOST_PASSTHROUGH_IMAGES="konard/hive-mind konard/hive-mind-dind"' to the bot container's 'docker run', or seed it now with: ${preload}`);
+    } else if (isDind && socketMounted) {
+      result.warnings.push(`Docker isolation image '${image}' is NOT in the nested Docker daemon even though the host Docker socket is mounted at ${sock}. ` + `box host-image passthrough may have skipped it (check DIND_HOST_PASSTHROUGH mode, the DIND_HOST_PASSTHROUGH_IMAGES allowlist, and that the host actually has '${image}' with a registry digest). ` + `The first isolated task will pull the full image. Seed it now with: ${preload}`);
+    } else {
+      result.warnings.push(`Docker isolation image '${image}' is not present locally; the first isolated task will pull it. ` + `If this host already has it under a different tag, pin HIVE_MIND_DOCKER_ISOLATION_IMAGE_TAG, or seed it with: ${preload}`);
+    }
+
+    // Root Cause B of the issue #1914 reopen: too little disk for the pull. The
+    // image is well over 30 GB extracted; predict the `no space left on device`
+    // failure here rather than hitting it mid-pull.
+    if (diskAvailableGiB != null && diskAvailableGiB < DOCKER_ISOLATION_LOW_DISK_GIB) {
+      const root = disk?.dataRoot || 'the Docker data root';
+      result.warnings.push(`Only ~${diskAvailableGiB.toFixed(0)} GiB free on ${root} and the isolation image '${image}' is not present yet. ` + `The Hive Mind isolation image is well over 30 GB extracted, so the first isolated task's pull may fail with 'no space left on device' (issue #1914). ` + `Seed it via host passthrough (mount the host docker socket) or with '${preload}', and free space on the Docker data root.`);
+    }
+  }
+
+  if (imagePresent) {
+    info(`✅ Docker isolation image '${image}' is already present locally — isolated tasks reuse it (no multi-GB pull). See issue #1914.`);
   }
   for (const w of result.warnings) warn(`⚠️ ${w}`);
   return result;

package/src/tool-retry.lib.mjs

@@ -72,6 +72,22 @@ export const classifyRetryableError = value => {
     return { message, isRetryable: false, isCapacity: false, requiresFreshSession: true, label: 'Corrupted thinking blocks (un-resumable session)' };
   }
 
+  // Issue #1924: Server-side temporary rate limiting (HTTP 429), distinct from an
+  // account usage/quota limit. The Claude CLI surfaces this as a synthetic
+  // assistant/result message and an api_error_status of 429:
+  //   "API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited"
+  // The response carries `x-should-retry: true` and the stream emits a
+  // `rate_limit_event` with `status: "rejected"`. Because the message explicitly
+  // says "not your usage limit", it is NOT a usage-limit reset-time situation and
+  // must NOT be routed through detectUsageLimit() (there is no reset time to wait
+  // for). It is a transient throttle that clears on its own, so it is safe to
+  // retry with the session preserved (--resume) after a backoff. Switching models
+  // does not help (the throttle is request-rate, not model capacity), so
+  // isCapacity is false.
+  if (lower.includes('temporarily limiting requests') || (lower.includes('rate limited') && lower.includes('not your usage limit')) || (lower.includes('rate_limit') && lower.includes('429'))) {
+    return { message, isRetryable: true, isCapacity: false, label: 'Server rate limited (429)' };
+  }
+
   if (lower.includes('api error: 503') || (lower.includes('503') && (lower.includes('upstream connect error') || lower.includes('remote connection failure')))) {
     return { message, isRetryable: true, isCapacity: false, label: '503 network error' };
   }