@link-assistant/hive-mind 2.0.10 → 2.0.12

package/CHANGELOG.md

@@ -1,5 +1,81 @@
 # @link-assistant/hive-mind
 
+## 2.0.12
+
+### Patch Changes
+
+- f17bac4: fix(clone): detect interrupted clones that exit 0, retry them, and explain the failure instead of the bare "Failed to get current branch" (#1957)
+
+  A `/solve` run crashed with the unactionable message `Failed to get current branch`.
+  Root cause: `gh repo clone` (and the `git clone` it wraps) **exited 0 even though the
+  transfer was interrupted** mid-stream (`fetch-pack: unexpected disconnect while reading
+sideband packet`), leaving **no `.git` directory** (`size=0 B`). The solver trusted the
+  exit code, logged `✅ Cloned to:`, then every subsequent git command failed with
+  `fatal: not a git repository`; the first to propagate it (`git branch --show-current` in
+  `verifyDefaultBranchAndStatus`) threw the bare error with no clue about what went wrong
+  or how to recover.
+
+  Defense-in-depth fix applied across the codebase:
+  - `cloneRepository()` (`src/solve.repository.lib.mjs`) no longer trusts the exit code:
+    after `gh repo clone` it validates the result with `git rev-parse --is-inside-work-tree`
+    and only treats the clone as successful when the exit code is 0 **and** a real working
+    tree exists. In `--verbose` mode it logs why a 0-exit clone was rejected.
+  - New exported helper `cleanPartialClone()` empties the target directory before each
+    retry so a partial clone does not make `gh repo clone <dir>` fail with "directory
+    exists and is not empty".
+  - `classifyCloneError()` now classifies the interrupted-transfer vocabulary
+    (`unexpected disconnect`, `sideband`, `early eof`, `the remote end hung up`,
+    `rpc failed`, `fetch-pack`, `index-pack failed`, `transfer closed`) as a retryable
+    `NETWORK` error, so the existing 3× exponential-backoff retry loop recovers from it.
+    404 / ENOSPC / auth failures stay non-retryable.
+  - `isTransientNetworkError()` (`src/lib.mjs`, shared by many gh/git retry call sites)
+    gains the same vocabulary, so the fix propagates everywhere — not just the clone path.
+  - Both failure points now print concrete, root-cause-obvious guidance: the clone-failure
+    path adds NETWORK causes/fixes, and `verifyDefaultBranchAndStatus()`
+    (`src/solve.repo-setup.lib.mjs`) detects `not a git repository` and logs an
+    `INCOMPLETE CLONE DETECTED` block (What happened / Error details / How to fix:
+    check network·VPN·proxy, re-run — clones auto-retry, verify access, check GitHub
+    status) instead of the bare message.
+
+  Adds `tests/test-issue-1957-incomplete-clone.mjs` (26 assertions) and a deep case study
+  in `docs/case-studies/issue-1957/`.
+
+## 2.0.11
+
+### Patch Changes
+
+- a29902a: fix(codex): don't fail a completed turn on echoed fixture content; expand transient network auto-retry (#1955)
+
+  A `--tool codex` run failed with `❌ Codex emitted error event: Network lookup
+skipped in fixture` even though the codex session **succeeded** (`turn.completed=1`,
+  `turn.failed=0`, working tree clean, full pricing produced). The phrase was not a
+  real error: while building an unrelated NDJSON adapter, the codex agent printed a
+  **test fixture** to its terminal. In verbose mode (`RUST_LOG=debug`) the codex CLI
+  renders OTEL telemetry (`codex_otel.log_only`, `event.name="codex.tool_result"`)
+  to stderr, including a raw `Output:` dump of each command's stdout. Our line-by-line
+  parser — which consumes stderr as well as stdout — `JSON.parse`d the fixture line
+  `{"type":"error","message":"Network lookup skipped in fixture"}` and bucketed it as
+  a genuine codex stream error.
+  - `getCodexErrorEventSummary()` (`src/codex.lib.mjs`) now treats any stray
+    **non-`turn`** error event as non-fatal whenever the turn completed successfully
+    (a `turn.completed` with no `turn.failed`). `turn.failed` remains the authoritative
+    failure signal and is never suppressed; suppressed strays are still recorded in
+    `ignoredEvents` (and logged per-event in verbose mode) for observability. This is
+    transport-agnostic — it fixes the false positive regardless of how the echo
+    arrived.
+  - `classifyRetryableError()` (`src/tool-retry.lib.mjs`, shared by
+    claude/codex/gemini/qwen/opencode) now classifies the full set of genuinely
+    transient network faults as retryable (`isCapacity:false`): DNS failures
+    (`ENOTFOUND`, `EAI_AGAIN`, "temporary failure in name resolution"), connection
+    faults (`ETIMEDOUT`, `ECONNREFUSED`, `EHOSTUNREACH`, `ENETUNREACH`, `EPIPE`,
+    "no route to host", "network is unreachable"), and gateway errors (502/504 and
+    Cloudflare `52x`); the 503 branch was broadened to "service unavailable". Aligns
+    with AWS retry guidance, RFC 9110 §15.6, and the getaddrinfo(3) man-page. The
+    fixture phrase itself is explicitly guarded to stay non-retryable.
+
+  Adds `tests/test-issue-1955-codex-fixture-false-positive.mjs` (23 tests) and a deep
+  case study in `docs/case-studies/issue-1955/`.
+
 ## 2.0.10
 
 ### Patch Changes

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@link-assistant/hive-mind",
-  "version": "2.0.10",
+  "version": "2.0.12",
   "description": "AI-powered issue solver and hive mind for collaborative problem solving",
   "main": "src/hive.mjs",
   "type": "module",

package/src/codex.lib.mjs

@@ -358,6 +358,30 @@ const isNonFatalCodexItemErrorMessage = message => /^in-process app-server event
 export const getCodexErrorEventSummary = codexJsonState => {
   const events = [];
   const ignoredEvents = [];
+
+  // Issue #1955: When the codex turn genuinely completed (a `turn.completed`
+  // event was observed) and codex never emitted a `turn.failed`, the session
+  // SUCCEEDED. Any stray top-level `error` (stream) or nested item `error` event
+  // in that case is non-fatal and must not fail the run. Two things produce such
+  // strays:
+  //   1. A transient error codex itself retried/recovered from before completing
+  //      the turn (e.g. a momentary stream blip).
+  //   2. Echoed content that merely *looks* like a codex protocol event. The
+  //      codex CLI prints OTEL telemetry (`codex_otel.log_only`,
+  //      event.name="codex.tool_result") containing a raw `Output:` dump of each
+  //      command's stdout. When a command prints a line shaped like a protocol
+  //      event — e.g. a printed NDJSON fixture line
+  //      `{"type":"error","message":"Network lookup skipped in fixture"}` — our
+  //      line-by-line parser misreads it as a genuine codex stream error and
+  //      fails an otherwise-successful run. This was the exact false positive in
+  //      issue #1955 (codex finished, working tree clean, CI passed, yet the run
+  //      was reported failed).
+  // `turn.failed` is the authoritative failure signal, so it is NEVER suppressed
+  // here; only non-`turn` error events are gated on turn completion.
+  const turnCompleted = (codexJsonState?.eventCounts?.['turn.completed'] || 0) > 0;
+  const turnFailed = (codexJsonState?.turnFailures?.length || 0) > 0;
+  const sessionSucceeded = turnCompleted && !turnFailed;
+
   const addEvents = (type, items = []) => {
     for (const item of items) {
       const message = unwrapCodexErrorMessage(item?.message);
@@ -369,6 +393,13 @@ export const getCodexErrorEventSummary = codexJsonState => {
         });
         continue;
       }
+      if (type !== 'turn' && sessionSucceeded) {
+        ignoredEvents.push({
+          ...event,
+          reason: 'Codex turn completed successfully with no turn.failed; stray non-turn error event is non-fatal (Issue #1955)',
+        });
+        continue;
+      }
       events.push(event);
     }
   };
@@ -1146,7 +1177,13 @@ export const executeCodexCommand = async params => {
       const codexErrorSummary = getCodexErrorEventSummary(codexJsonState);
       if (codexErrorSummary.ignoredEvents.length > 0) {
         const ignoredMessages = [...new Set(codexErrorSummary.ignoredEvents.map(event => event.message))].join('; ');
-        await log(`⚠️ Ignoring non-fatal Codex item error event(s): ${ignoredMessages}`, { level: 'warning', verbose: true });
+        await log(`⚠️ Ignoring non-fatal Codex error event(s): ${ignoredMessages}`, { level: 'warning', verbose: true });
+        // Issue #1955: trace why each stray error event was treated as non-fatal so a
+        // future regression (e.g. a real error wrongly suppressed) is diagnosable from
+        // the verbose log without re-deriving the turn.completed/turn.failed state.
+        for (const ignored of codexErrorSummary.ignoredEvents) {
+          await log(`   ↳ [${ignored.type}] "${ignored.message}" — ${ignored.reason}`, { verbose: true });
+        }
       }
       if (codexErrorSummary.hasError) {
         const limitSource = codexErrorSummary.message || lastMessage;

package/src/lib.mjs

@@ -471,7 +471,10 @@ export const isTransientNetworkError = error => {
   const combined = msg + ' ' + output;
 
   // Issue #1536: added 'unexpected eof' — seen in gh CLI when connection drops mid-response
-  const transientPatterns = ['i/o timeout', 'dial tcp', 'connection refused', 'connection reset', 'econnreset', 'etimedout', 'enotfound', 'ehostunreach', 'enetunreach', 'network is unreachable', 'temporary failure', 'http 502', 'http 503', 'http 504', 'bad gateway', 'service unavailable', 'gateway timeout', 'tls handshake timeout', 'ssl_error', 'socket hang up', 'unexpected eof'];
+  // Issue #1957: added git fetch-pack/sideband disconnect patterns — seen when a
+  // `gh repo clone` / `git clone` connection drops mid-transfer, leaving an incomplete
+  // (or missing) working tree even though the wrapper can exit 0.
+  const transientPatterns = ['i/o timeout', 'dial tcp', 'connection refused', 'connection reset', 'econnreset', 'etimedout', 'enotfound', 'ehostunreach', 'enetunreach', 'network is unreachable', 'temporary failure', 'http 502', 'http 503', 'http 504', 'bad gateway', 'service unavailable', 'gateway timeout', 'tls handshake timeout', 'ssl_error', 'socket hang up', 'unexpected eof', 'unexpected disconnect', 'sideband', 'early eof', 'the remote end hung up', 'rpc failed', 'fetch-pack', 'index-pack failed', 'remote end hung up unexpectedly', 'transfer closed'];
 
   return transientPatterns.some(pattern => combined.includes(pattern));
 };

package/src/solve.repo-setup.lib.mjs

@@ -65,9 +65,44 @@ export async function verifyDefaultBranchAndStatus({ tempDir, log, formatAligned
   const defaultBranchResult = await $({ cwd: tempDir })`git branch --show-current`;
 
   if (defaultBranchResult.code !== 0) {
+    // Issue #1957: the most common cause of this failure is an interrupted/incomplete
+    // clone — `git` reports "fatal: not a git repository" because there is no `.git`
+    // directory. The previous message ("Failed to get current branch") gave the user
+    // no idea what happened or what to do. Detect the incomplete-clone case and give
+    // concrete, actionable instructions instead.
+    const branchErr = ((defaultBranchResult.stderr?.toString() || '') + (defaultBranchResult.stdout?.toString() || '')).trim();
+    const isNotAGitRepo = /not a git repository/i.test(branchErr);
+
+    await log('');
+    await log(`${formatAligned('❌', isNotAGitRepo ? 'INCOMPLETE CLONE DETECTED' : 'FAILED TO READ CURRENT BRANCH', '')}`, { level: 'error' });
+    await log('');
+    await log('  🔍 What happened:');
+    if (isNotAGitRepo) {
+      await log(`     The working directory is not a valid git repository: ${tempDir}`);
+      await log('     This almost always means the clone was interrupted before it finished');
+      await log('     (e.g. "fetch-pack: unexpected disconnect while reading sideband packet"),');
+      await log('     so no .git directory was created.');
+    } else {
+      await log(`     Could not determine the current branch in: ${tempDir}`);
+    }
+    await log('');
+    await log('  📦 Error details:');
+    for (const line of (branchErr || 'Unknown error').split('\n')) {
+      if (line.trim()) await log(`     ${line}`);
+    }
+    await log('');
+    await log('  🔧 How to fix:');
+    await log('     1. Check your network connection / VPN / proxy (interrupted transfers are usually transient)');
+    await log('     2. Re-run the command — the solver retries clones automatically and a fresh run usually succeeds');
+    if (owner && repo) await log(`     3. Verify access to the repository: gh repo view ${owner}/${repo}`);
+    await log(`     ${owner && repo ? '4' : '3'}. Check GitHub status if it keeps failing: https://www.githubstatus.com`);
+    await log('');
+    await log('  ℹ️  If this is reproducible, ask a Hive Mind administrator to inspect the solver terminal log for the clone error.');
+    await log('');
+
     await log('Error: Failed to get current branch');
-    await log(defaultBranchResult.stderr ? defaultBranchResult.stderr.toString() : 'Unknown error');
-    throw new Error('Failed to get current branch');
+    if (branchErr) await log(branchErr);
+    throw new Error(isNotAGitRepo ? 'Failed to get current branch - working directory is not a git repository (incomplete clone)' : 'Failed to get current branch');
   }
 
   let defaultBranch = defaultBranchResult.stdout.toString().trim();

package/src/solve.repository.lib.mjs

@@ -922,8 +922,12 @@ export const classifyCloneError = errorOutput => {
   }
 
   // Network-related errors - typically retryable
-  if (output.includes('connection refused') || output.includes('connection timed out') || output.includes('connection reset') || output.includes('unable to connect') || output.includes('network is unreachable') || output.includes('ssl error')) {
-    return { type: 'NETWORK', retryable: true, description: 'Network connectivity issue' };
+  // Issue #1957: git fetch-pack/sideband disconnects (e.g.
+  // "fetch-pack: unexpected disconnect while reading sideband packet",
+  // "early EOF", "the remote end hung up unexpectedly", "RPC failed",
+  // "index-pack failed") leave an incomplete or missing clone but are transient.
+  if (output.includes('connection refused') || output.includes('connection timed out') || output.includes('connection reset') || output.includes('unable to connect') || output.includes('network is unreachable') || output.includes('ssl error') || output.includes('unexpected disconnect') || output.includes('sideband') || output.includes('early eof') || output.includes('remote end hung up') || output.includes('rpc failed') || output.includes('fetch-pack') || output.includes('index-pack failed') || output.includes('transfer closed')) {
+    return { type: 'NETWORK', retryable: true, description: 'Network connectivity issue (interrupted transfer)' };
   }
 
   // Authentication/permission errors - not retryable
@@ -945,6 +949,24 @@ export const classifyCloneError = errorOutput => {
   return { type: 'UNKNOWN', retryable: true, description: 'Unknown error' };
 };
 
+// Issue #1957: remove leftovers from an interrupted clone so a retry can start clean.
+// We empty the directory in place (rather than removing it) because the path was
+// created up-front by setupTempDirectory and may be the configured working directory.
+export const cleanPartialClone = async tempDir => {
+  try {
+    const entries = await fs.readdir(tempDir);
+    for (const entry of entries) {
+      await fs.rm(path.join(tempDir, entry), { recursive: true, force: true });
+    }
+  } catch (error) {
+    // Directory may not exist yet, or be unreadable — non-fatal; the retry/clone
+    // will surface any real problem with a clearer message.
+    if (error?.code !== 'ENOENT') {
+      reportError(error, { context: 'clean_partial_clone', tempDir, operation: 'empty_directory' });
+    }
+  }
+};
+
 // Clone repository and set up remotes with retry mechanism
 export const cloneRepository = async (repoToClone, tempDir, argv, owner, repo) => {
   const maxRetries = 3;
@@ -959,9 +981,25 @@ export const cloneRepository = async (repoToClone, tempDir, argv, owner, repo) =
 
     // Use 2>&1 to capture all output and filter "Cloning into" message
     const cloneResult = await $`gh repo clone ${repoToClone} ${tempDir} 2>&1`;
-
-    // Verify clone was successful
+    const cloneOutput = (cloneResult.stdout || cloneResult.stderr || '').toString().trim();
+
+    // Issue #1957: `gh repo clone` (and the `git clone` it wraps) can exit 0 even when
+    // the underlying transfer was interrupted — e.g. "fetch-pack: unexpected disconnect
+    // while reading sideband packet" — leaving an incomplete or completely missing
+    // working tree (no `.git`). Trusting the exit code alone made the solver report
+    // "✅ Cloned to:" and then crash much later with the unhelpful "Failed to get
+    // current branch". Verify the clone actually produced a usable git repository.
+    let repoIsValid = false;
     if (cloneResult.code === 0) {
+      const validityCheck = await $({ cwd: tempDir })`git rev-parse --is-inside-work-tree 2>&1`;
+      repoIsValid = validityCheck.code === 0 && validityCheck.stdout.toString().trim() === 'true';
+      if (!repoIsValid && argv.verbose) {
+        await log(`${formatAligned('🔧', 'Clone validation:', `git rev-parse failed despite exit 0 — ${(validityCheck.stdout || validityCheck.stderr || '').toString().trim().split('\n')[0]}`)}`);
+      }
+    }
+
+    // Verify clone was successful (exit code 0 AND a valid working tree exists)
+    if (cloneResult.code === 0 && repoIsValid) {
       await log(`${formatAligned('✅', 'Cloned to:', tempDir)}`);
 
       // Verify and fix remote configuration
@@ -975,7 +1013,15 @@ export const cloneRepository = async (repoToClone, tempDir, argv, owner, repo) =
     }
 
     // Clone failed - analyze error and determine if retry is appropriate
-    const errorOutput = (cloneResult.stderr || cloneResult.stdout || 'Unknown error').toString().trim();
+    // Issue #1957: when the wrapper exited 0 but left no valid repo, surface the
+    // interrupted-transfer output (which carries the real "unexpected disconnect"
+    // reason) so it is classified as a retryable network error rather than UNKNOWN.
+    const errorOutput = cloneResult.code === 0 && !repoIsValid ? cloneOutput || 'Clone exited 0 but no valid git repository was created (interrupted transfer / incomplete clone)' : (cloneResult.stderr || cloneResult.stdout || 'Unknown error').toString().trim();
+
+    // Issue #1957: a partial clone can leave stray files behind that make a retry of
+    // `gh repo clone <dir>` fail with "directory exists and is not empty". Clean the
+    // target directory before classifying/retrying so the next attempt starts fresh.
+    await cleanPartialClone(tempDir);
 
     const errorClassification = classifyCloneError(errorOutput);
 
@@ -1015,6 +1061,9 @@ export const cloneRepository = async (repoToClone, tempDir, argv, owner, repo) =
         await log('     • Network connectivity issues');
         if (errorClassification.type === 'TRANSIENT') await log('     • GitHub server issues (temporary)');
         if (errorClassification.type === 'RATE_LIMIT') await log('     • API rate limiting exceeded');
+        // Issue #1957: the transfer started but was interrupted (e.g. the connection
+        // dropped while reading the pack). The retries above were already exhausted.
+        if (errorClassification.type === 'NETWORK') await log('     • Connection dropped mid-transfer (the clone was interrupted before completing)');
         if (argv.fork) await log('     • Fork not ready yet (try again in a moment)');
         await log('');
         await log('  🔧 How to fix:');
@@ -1024,6 +1073,11 @@ export const cloneRepository = async (repoToClone, tempDir, argv, owner, repo) =
         if (argv.fork) await log(`     4. Check fork: gh repo view ${repoToClone}`);
         if (errorClassification.type === 'TRANSIENT') await log('     5. Wait and retry / check: https://www.githubstatus.com');
         if (errorClassification.type === 'RATE_LIMIT') await log('     5. Wait for rate limit to reset or use --token with different token');
+        if (errorClassification.type === 'NETWORK') {
+          await log('     5. Check your network connection / VPN / proxy, then re-run the command');
+          await log('     6. On slow or unstable links, a shallower history transfers faster and is less');
+          await log('        likely to be interrupted; ask a Hive Mind administrator if clone tuning is available');
+        }
         await log('');
       }
       await safeExit(1, 'Repository setup failed');

package/src/tool-retry.lib.mjs

@@ -89,6 +89,44 @@ export const classifyRetryableError = value => {
     return { message, isRetryable: true, isCapacity: false, label: 'Socket/connection closed unexpectedly' };
   }
 
+  // Issue #1955: Transient DNS resolution failures. When the local resolver, the
+  // upstream DNS, or the network briefly drops, Node's undici/fetch (and the Codex
+  // CLI's reqwest stack) surface the failure with one of these signatures:
+  //   getaddrinfo ENOTFOUND api.openai.com / getaddrinfo EAI_AGAIN api.github.com /
+  //   "Temporary failure in name resolution" / "dns error" / "failed to lookup
+  //   address information". These are 100% temporary — the host is not gone, name
+  //   resolution simply failed for a moment — so the same request is safe to retry
+  //   after a backoff. Switching models does not help (it is a network-layer fault),
+  //   so isCapacity is false.
+  // NOTE: deliberately scoped to real resolver error tokens so it never matches
+  // unrelated text that merely contains the word "lookup" (e.g. the echoed fixture
+  // line "Network lookup skipped in fixture" from issue #1955, which is not an error
+  // at all).
+  if (lower.includes('enotfound') || lower.includes('eai_again') || lower.includes('temporary failure in name resolution') || lower.includes('getaddrinfo') || lower.includes('dns error') || lower.includes('failed to lookup address information') || lower.includes('name or service not known')) {
+    return { message, isRetryable: true, isCapacity: false, label: 'DNS resolution failure' };
+  }
+
+  // Issue #1955: Transient connection-level network failures from the OS/socket
+  // layer — the peer is unreachable or refused the connection for a moment, or a
+  // connect/read timed out. These are temporary (load balancer rotating, a node
+  // briefly down, a VPN/proxy hiccup, a flaky link) and the identical request
+  // typically succeeds on retry. Covers Node libuv error codes and their textual
+  // equivalents. ETIMEDOUT/"timed out" here is the connection/socket timeout
+  // (distinct from the API-level "request timed out" handled above).
+  if (lower.includes('etimedout') || lower.includes('connection timed out') || lower.includes('econnrefused') || lower.includes('connection refused') || lower.includes('ehostunreach') || lower.includes('no route to host') || lower.includes('enetunreach') || lower.includes('network is unreachable') || lower.includes('epipe') || lower.includes('eai_fail')) {
+    return { message, isRetryable: true, isCapacity: false, label: 'Transient network connection failure' };
+  }
+
+  // Issue #1955: Transient HTTP gateway / proxy errors (502 Bad Gateway, 504 Gateway
+  // Timeout) and Cloudflare's edge family (520 Unknown Error, 521 Web Server Is Down,
+  // 522 Connection Timed Out, 523 Origin Is Unreachable, 524 A Timeout Occurred).
+  // These come from an intermediary (CDN/proxy/load balancer), not from a request the
+  // client got wrong, and clear on their own — OpenAI/Anthropic/GitHub all front their
+  // APIs with such proxies. Safe to retry the same request after a backoff.
+  if (lower.includes('502 bad gateway') || lower.includes('bad gateway') || lower.includes('504 gateway timeout') || lower.includes('gateway time-out') || lower.includes('gateway timeout') || lower.includes('api error: 502') || lower.includes('api error: 504') || /\b52[0-4]\b/.test(lower)) {
+    return { message, isRetryable: true, isCapacity: false, label: 'Gateway error (502/504/52x)' };
+  }
+
   // Issue #1834: Corrupted extended-thinking blocks. When extended thinking is combined with tool
   // use, Claude Code can persist a thinking block to the session transcript with the `thinking`
   // text emptied to "" while retaining the original `signature`. On resume/continue the block is
@@ -120,7 +158,10 @@ export const classifyRetryableError = value => {
     return { message, isRetryable: true, isCapacity: false, label: 'Server rate limited (429)' };
   }
 
-  if (lower.includes('api error: 503') || (lower.includes('503') && (lower.includes('upstream connect error') || lower.includes('remote connection failure')))) {
+  // Issue #1955: broadened to also catch the bare "503 Service Unavailable" that
+  // GitHub/OpenAI/Anthropic return when a backend is briefly saturated — a
+  // transient, self-clearing condition, safe to retry with the same request.
+  if (lower.includes('api error: 503') || lower.includes('503 service unavailable') || lower.includes('service unavailable') || (lower.includes('503') && (lower.includes('upstream connect error') || lower.includes('remote connection failure')))) {
     return { message, isRetryable: true, isCapacity: false, label: '503 network error' };
   }