@link-assistant/hive-mind 2.0.10 → 2.0.11

package/CHANGELOG.md

@@ -1,5 +1,41 @@
 # @link-assistant/hive-mind
 
+## 2.0.11
+
+### Patch Changes
+
+- a29902a: fix(codex): don't fail a completed turn on echoed fixture content; expand transient network auto-retry (#1955)
+
+  A `--tool codex` run failed with `❌ Codex emitted error event: Network lookup
+skipped in fixture` even though the codex session **succeeded** (`turn.completed=1`,
+  `turn.failed=0`, working tree clean, full pricing produced). The phrase was not a
+  real error: while building an unrelated NDJSON adapter, the codex agent printed a
+  **test fixture** to its terminal. In verbose mode (`RUST_LOG=debug`) the codex CLI
+  renders OTEL telemetry (`codex_otel.log_only`, `event.name="codex.tool_result"`)
+  to stderr, including a raw `Output:` dump of each command's stdout. Our line-by-line
+  parser — which consumes stderr as well as stdout — `JSON.parse`d the fixture line
+  `{"type":"error","message":"Network lookup skipped in fixture"}` and bucketed it as
+  a genuine codex stream error.
+  - `getCodexErrorEventSummary()` (`src/codex.lib.mjs`) now treats any stray
+    **non-`turn`** error event as non-fatal whenever the turn completed successfully
+    (a `turn.completed` with no `turn.failed`). `turn.failed` remains the authoritative
+    failure signal and is never suppressed; suppressed strays are still recorded in
+    `ignoredEvents` (and logged per-event in verbose mode) for observability. This is
+    transport-agnostic — it fixes the false positive regardless of how the echo
+    arrived.
+  - `classifyRetryableError()` (`src/tool-retry.lib.mjs`, shared by
+    claude/codex/gemini/qwen/opencode) now classifies the full set of genuinely
+    transient network faults as retryable (`isCapacity:false`): DNS failures
+    (`ENOTFOUND`, `EAI_AGAIN`, "temporary failure in name resolution"), connection
+    faults (`ETIMEDOUT`, `ECONNREFUSED`, `EHOSTUNREACH`, `ENETUNREACH`, `EPIPE`,
+    "no route to host", "network is unreachable"), and gateway errors (502/504 and
+    Cloudflare `52x`); the 503 branch was broadened to "service unavailable". Aligns
+    with AWS retry guidance, RFC 9110 §15.6, and the getaddrinfo(3) man-page. The
+    fixture phrase itself is explicitly guarded to stay non-retryable.
+
+  Adds `tests/test-issue-1955-codex-fixture-false-positive.mjs` (23 tests) and a deep
+  case study in `docs/case-studies/issue-1955/`.
+
 ## 2.0.10
 
 ### Patch Changes

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@link-assistant/hive-mind",
-  "version": "2.0.10",
+  "version": "2.0.11",
   "description": "AI-powered issue solver and hive mind for collaborative problem solving",
   "main": "src/hive.mjs",
   "type": "module",

package/src/codex.lib.mjs

@@ -358,6 +358,30 @@ const isNonFatalCodexItemErrorMessage = message => /^in-process app-server event
 export const getCodexErrorEventSummary = codexJsonState => {
   const events = [];
   const ignoredEvents = [];
+
+  // Issue #1955: When the codex turn genuinely completed (a `turn.completed`
+  // event was observed) and codex never emitted a `turn.failed`, the session
+  // SUCCEEDED. Any stray top-level `error` (stream) or nested item `error` event
+  // in that case is non-fatal and must not fail the run. Two things produce such
+  // strays:
+  //   1. A transient error codex itself retried/recovered from before completing
+  //      the turn (e.g. a momentary stream blip).
+  //   2. Echoed content that merely *looks* like a codex protocol event. The
+  //      codex CLI prints OTEL telemetry (`codex_otel.log_only`,
+  //      event.name="codex.tool_result") containing a raw `Output:` dump of each
+  //      command's stdout. When a command prints a line shaped like a protocol
+  //      event — e.g. a printed NDJSON fixture line
+  //      `{"type":"error","message":"Network lookup skipped in fixture"}` — our
+  //      line-by-line parser misreads it as a genuine codex stream error and
+  //      fails an otherwise-successful run. This was the exact false positive in
+  //      issue #1955 (codex finished, working tree clean, CI passed, yet the run
+  //      was reported failed).
+  // `turn.failed` is the authoritative failure signal, so it is NEVER suppressed
+  // here; only non-`turn` error events are gated on turn completion.
+  const turnCompleted = (codexJsonState?.eventCounts?.['turn.completed'] || 0) > 0;
+  const turnFailed = (codexJsonState?.turnFailures?.length || 0) > 0;
+  const sessionSucceeded = turnCompleted && !turnFailed;
+
   const addEvents = (type, items = []) => {
     for (const item of items) {
       const message = unwrapCodexErrorMessage(item?.message);
@@ -369,6 +393,13 @@ export const getCodexErrorEventSummary = codexJsonState => {
         });
         continue;
       }
+      if (type !== 'turn' && sessionSucceeded) {
+        ignoredEvents.push({
+          ...event,
+          reason: 'Codex turn completed successfully with no turn.failed; stray non-turn error event is non-fatal (Issue #1955)',
+        });
+        continue;
+      }
       events.push(event);
     }
   };
@@ -1146,7 +1177,13 @@ export const executeCodexCommand = async params => {
       const codexErrorSummary = getCodexErrorEventSummary(codexJsonState);
       if (codexErrorSummary.ignoredEvents.length > 0) {
         const ignoredMessages = [...new Set(codexErrorSummary.ignoredEvents.map(event => event.message))].join('; ');
-        await log(`⚠️ Ignoring non-fatal Codex item error event(s): ${ignoredMessages}`, { level: 'warning', verbose: true });
+        await log(`⚠️ Ignoring non-fatal Codex error event(s): ${ignoredMessages}`, { level: 'warning', verbose: true });
+        // Issue #1955: trace why each stray error event was treated as non-fatal so a
+        // future regression (e.g. a real error wrongly suppressed) is diagnosable from
+        // the verbose log without re-deriving the turn.completed/turn.failed state.
+        for (const ignored of codexErrorSummary.ignoredEvents) {
+          await log(`   ↳ [${ignored.type}] "${ignored.message}" — ${ignored.reason}`, { verbose: true });
+        }
       }
       if (codexErrorSummary.hasError) {
         const limitSource = codexErrorSummary.message || lastMessage;

package/src/tool-retry.lib.mjs

@@ -89,6 +89,44 @@ export const classifyRetryableError = value => {
     return { message, isRetryable: true, isCapacity: false, label: 'Socket/connection closed unexpectedly' };
   }
 
+  // Issue #1955: Transient DNS resolution failures. When the local resolver, the
+  // upstream DNS, or the network briefly drops, Node's undici/fetch (and the Codex
+  // CLI's reqwest stack) surface the failure with one of these signatures:
+  //   getaddrinfo ENOTFOUND api.openai.com / getaddrinfo EAI_AGAIN api.github.com /
+  //   "Temporary failure in name resolution" / "dns error" / "failed to lookup
+  //   address information". These are 100% temporary — the host is not gone, name
+  //   resolution simply failed for a moment — so the same request is safe to retry
+  //   after a backoff. Switching models does not help (it is a network-layer fault),
+  //   so isCapacity is false.
+  // NOTE: deliberately scoped to real resolver error tokens so it never matches
+  // unrelated text that merely contains the word "lookup" (e.g. the echoed fixture
+  // line "Network lookup skipped in fixture" from issue #1955, which is not an error
+  // at all).
+  if (lower.includes('enotfound') || lower.includes('eai_again') || lower.includes('temporary failure in name resolution') || lower.includes('getaddrinfo') || lower.includes('dns error') || lower.includes('failed to lookup address information') || lower.includes('name or service not known')) {
+    return { message, isRetryable: true, isCapacity: false, label: 'DNS resolution failure' };
+  }
+
+  // Issue #1955: Transient connection-level network failures from the OS/socket
+  // layer — the peer is unreachable or refused the connection for a moment, or a
+  // connect/read timed out. These are temporary (load balancer rotating, a node
+  // briefly down, a VPN/proxy hiccup, a flaky link) and the identical request
+  // typically succeeds on retry. Covers Node libuv error codes and their textual
+  // equivalents. ETIMEDOUT/"timed out" here is the connection/socket timeout
+  // (distinct from the API-level "request timed out" handled above).
+  if (lower.includes('etimedout') || lower.includes('connection timed out') || lower.includes('econnrefused') || lower.includes('connection refused') || lower.includes('ehostunreach') || lower.includes('no route to host') || lower.includes('enetunreach') || lower.includes('network is unreachable') || lower.includes('epipe') || lower.includes('eai_fail')) {
+    return { message, isRetryable: true, isCapacity: false, label: 'Transient network connection failure' };
+  }
+
+  // Issue #1955: Transient HTTP gateway / proxy errors (502 Bad Gateway, 504 Gateway
+  // Timeout) and Cloudflare's edge family (520 Unknown Error, 521 Web Server Is Down,
+  // 522 Connection Timed Out, 523 Origin Is Unreachable, 524 A Timeout Occurred).
+  // These come from an intermediary (CDN/proxy/load balancer), not from a request the
+  // client got wrong, and clear on their own — OpenAI/Anthropic/GitHub all front their
+  // APIs with such proxies. Safe to retry the same request after a backoff.
+  if (lower.includes('502 bad gateway') || lower.includes('bad gateway') || lower.includes('504 gateway timeout') || lower.includes('gateway time-out') || lower.includes('gateway timeout') || lower.includes('api error: 502') || lower.includes('api error: 504') || /\b52[0-4]\b/.test(lower)) {
+    return { message, isRetryable: true, isCapacity: false, label: 'Gateway error (502/504/52x)' };
+  }
+
   // Issue #1834: Corrupted extended-thinking blocks. When extended thinking is combined with tool
   // use, Claude Code can persist a thinking block to the session transcript with the `thinking`
   // text emptied to "" while retaining the original `signature`. On resume/continue the block is
@@ -120,7 +158,10 @@ export const classifyRetryableError = value => {
     return { message, isRetryable: true, isCapacity: false, label: 'Server rate limited (429)' };
   }
 
-  if (lower.includes('api error: 503') || (lower.includes('503') && (lower.includes('upstream connect error') || lower.includes('remote connection failure')))) {
+  // Issue #1955: broadened to also catch the bare "503 Service Unavailable" that
+  // GitHub/OpenAI/Anthropic return when a backend is briefly saturated — a
+  // transient, self-clearing condition, safe to retry with the same request.
+  if (lower.includes('api error: 503') || lower.includes('503 service unavailable') || lower.includes('service unavailable') || (lower.includes('503') && (lower.includes('upstream connect error') || lower.includes('remote connection failure')))) {
     return { message, isRetryable: true, isCapacity: false, label: '503 network error' };
   }