@hybridlabor-api/aos 4.2.0-beta.0 → 4.3.0

package/skills/global_config/bdbresilience/SKILL.md

@@ -0,0 +1,216 @@
+---
+name: bdbresilience
+description: Autonomous CI/CD error recovery, distributed file-based locking, diagnostic triage, and two-phase GO gate resilience engine for BDB Agent OS multi-agent pipelines.
+category: bdb-core
+triggers:
+  - error recovery
+  - transient error
+  - rate limit 429
+  - exponential backoff
+  - distributed lock
+  - lockfile
+  - concurrency control
+  - state race
+  - ci/cd triage
+  - log parser
+  - flaky test
+  - self-healing
+  - verification gate
+  - go gate
+  - two-phase gate
+capabilities:
+  - monitoring
+  - security
+  - context-management
+  - testing
+  - automation
+disable-model-invocation: false
+user-invocable: true
+---
+
+# 🛡️ BDB Resilience & Concurrency Engine (`/bdbresilience`)
+
+The `/bdbresilience` master skill suite provides deterministic runtime fault tolerance, concurrency coordination, diagnostic log triage, and pre-tool deployment gating for autonomous multi-agent pipelines operating across the BDB Agent OS ecosystem.
+
+---
+
+## 1. Core Tenets
+
+1. **Fail Deterministically, Recover Gracefully**: Never swallow errors silently or crash abruptly. Intercept all tool, network, and runtime faults through a 4-tier taxonomy (`transient`, `tool_level_fault`, `auth_credential`, `unrecoverable`). Execute structured recovery before attempting escalation.
+2. **Zero-Race Concurrency**: Shared mutable state—including `production_artifacts/state.json`, git branches, and worktrees—must never suffer lost updates or dirty reads. Enforce mutual exclusion through atomic POSIX/APFS file locking (`O_CREAT | O_EXCL`) or fragment isolation (`state.d/<nodeId>.json`).
+3. **No Hallucinated Triage**: Parse raw CI/CD logs directly (Vitest, Jest, tsc, ESLint, GitHub Actions). Extract verified file coordinates, line numbers, failure diffs, and exact root causes into structured JSON reports. Never synthesize speculative fixes without log verification.
+4. **Strict Two-Phase Gate Precedence**: High-consequence actions (`git push`, `npm publish`, `npm version`, recursive `rm`) require an uncompromised human verification gate. Enforce the BDB Pre-Tool Gate protocol: lock execution in strict read-only mode until a literal, isolated human token `"GO"` is validated.
+
+---
+
+## 2. When to Use vs. When to Exclude
+
+### ✅ When to Use
+- **External API & Network Flakiness**: Handling HTTP 429 rate limits, socket timeouts (`ETIMEDOUT`), connection resets (`ECONNRESET`), and temporary provider dropouts.
+- **Tool Failures with Fallbacks**: Automatically rerouting failed tool invocations to secondary providers (e.g., primary MCP down → secondary CLI fallback) with immutable diversion audit trails.
+- **Concurrent Agent Execution**: Protecting shared files (`state.json`), databases, or worktrees during parallel build node execution (Engineering, UI/UX, Media).
+- **CI/CD Pipeline Failures**: Ingesting build and test runner failure logs, distinguishing transient infrastructure errors from code regressions, and generating actionable repair proposals.
+- **Release Verification & Deployment**: Enforcing the two-phase approval gate prior to running destructive or outward-facing operations.
+
+### ❌ When to Exclude
+- **Single-Agent Read-Only Probes**: Reading static documentation, viewing local files, or querying local git status where no concurrency or network calls occur.
+- **Internal Synchronous Transforms**: Pure CPU operations, in-memory string formatting, or deterministic array manipulations without side effects.
+- **Explicit User-Directed Interrupts**: Direct manual cancellation or kill signals received from the operator.
+
+---
+
+## 3. Resilience Architecture & Operational Playbooks
+
+```
++---------------------------------------------------------------------------------------+
+|                               /bdbresilience Skill Suite                              |
+|                             Master Operational Architecture                           |
++-------------------------------------------+-------------------------------------------+
+                                            |
+        +-----------------------------------+-----------------------------------+
+        |                                                                       |
+        v                                                                       v
++---------------------------------------+     +-----------------------------------------+
+|      Pattern 1: Error Recovery        |     |     Pattern 2: Distributed Locking      |
+|  - 4-Tier Failure Classification      |     |  - POSIX Atomic Creation (O_CREAT|EXCL) |
+|  - Full Jitter Exponential Backoff    |     |  - Monotonic Fencing Tokens             |
+|  - Secondary Tool Router & Diversion  |     |  - Background Heartbeat Renewal         |
+|  - Fail-Closed Human Escalation       |     |  - Atomic Stale / Orphan Eviction       |
+|  [references/error-recovery.md]       |     |  [references/distributed-locking.md]    |
++---------------------------------------+     +-----------------------------------------+
+        |                                                                       |
+        +-----------------------------------+-----------------------------------+
+                                            |
+        +-----------------------------------+-----------------------------------+
+        |                                                                       |
+        v                                                                       v
++---------------------------------------+     +-----------------------------------------+
+|       Pattern 3: CI/CD Triage         |     |       Pattern 4: Two-Phase GO Gate      |
+|  - Multi-Runner ANSI Normalization    |     |  - Phase 1: Strict Read-Only Planning   |
+|  - Flaky vs Deterministic Classifier  |     |  - Phase 2: Literal Token Execution     |
+|  - JSON Diagnostic Report Schema      |     |  - Fail-Closed Transcript Scanner       |
+|  - isCleanRun() Pass Predicate        |     |  - Caller-Written approvals Ledger      |
+|  [references/cicd-triage.md]          |     |  [references/two-phase-go-gate.md]      |
++---------------------------------------+     +-----------------------------------------+
+```
+
+### Specialized Reference Guides
+- 📘 **[Error Recovery & Retry Routing](references/error-recovery.md)**: Taxonomy classification, Full Jitter backoff formula, diversion logging, and zero-hallucination escalation payloads.
+- 📘 **[Distributed Locking & Concurrency](references/distributed-locking.md)**: Atomic lockfile creation, lease TTLs, heartbeat renewal, atomic rename break eviction, and `state.json` protection.
+- 📘 **[CI/CD Self-Healing Triage](references/cicd-triage.md)**: Multi-runner log parsers, flaky vs. deterministic classifier, and structured JSON diagnostics.
+- 📘 **[Two-Phase Pre-Tool GO Gate](references/two-phase-go-gate.md)**: PreToolUse hook specifications, transcript token verification, and fail-closed gate mechanics.
+- 📘 **[Node Integration Contracts](contracts/nodes-integration.md)**: **PROPOSAL, not applied.** How Reviewer and Shipping *would* be equipped in `nodes.json`, plus hooks for `/startcycle-graph` and `/bdbrainstorm`. No AOS node carries `bdbresilience` today.
+
+---
+
+## 4. Operational Commands & Procedures
+
+### `/bdbresilience recover`
+Invokes the automated error classification and recovery pipeline for a failed operation.
+```typescript
+import { classifyError, withRetry, executeWithFallback } from 'bdb-cicd-resilience/recovery/index.js';
+
+// 1. Classify error
+const classification = classifyError(error, { toolName: 'api_fetch', attempt: 1 });
+
+// 2. Retry with full jitter if transient
+//    (withRetry classifies internally too, and rethrows at once on canRetry: false)
+if (classification.category === 'transient') {
+  const result = await withRetry(
+    () => callApi(),
+    { maxRetries: 3, baseDelayMs: 500, maxDelayMs: 10000 }
+  );
+}
+
+// 3. Fallback route if tool fault. The fallback is a tool NAME; your execute
+//    function does the dispatch, and a canRetry:false classification suppresses
+//    the diversion instead of re-sending the call to a second provider.
+if (classification.category === 'tool_level_fault') {
+  const fallbackResult = await executeWithFallback({
+    originalTool: 'mcp_primary',
+    fallbackTool: 'cli_fallback',
+    execute: (toolName) => invokeTool(toolName, args),
+    auditFilePath: 'production_artifacts/diversions.jsonl',
+  });
+}
+```
+
+### `/bdbresilience lock`
+Coordinates exclusive access to a shared resource using deterministic atomic locking.
+```typescript
+import { withStateLock, withDistributedLock } from 'bdb-cicd-resilience/locking/index.js';
+
+// Guard state.json mutations
+await withStateLock('production_artifacts/state.json', (state) => {
+  state.artifacts.backend = 'production_artifacts/02_backend_schema.md';
+  state.findings.push({ id: 'F-ENG-01', status: 'fixed' });
+  return state;
+}, { ttlMs: 15000, acquireTimeoutMs: 10000 });
+```
+
+### `/bdbresilience triage`
+Parses build or test logs to isolate failures and propose fixes.
+```typescript
+import { generateDiagnosticReport, isCleanRun } from 'bdb-cicd-resilience/triage/index.js';
+
+const report = generateDiagnosticReport(rawBuildOutput, 'vitest');
+console.log(`Failures: ${report.summary.totalFailures} (Transient: ${report.summary.transientCount})`);
+if (isCleanRun(report)) {
+  // The ONLY sanctioned "it passed". `totalFailures === 0` alone is also true
+  // for an empty or unrecognised log (summary.parseStatus 'empty' / 'unparsed').
+} else if (report.summary.canAutoRetry) {
+  // Safe infrastructure retry
+} else {
+  // Escalate exact deterministic failure coordinates to engineer
+}
+```
+
+### `/bdbresilience gate`
+Evaluates pre-tool execution authorization against the conversation transcript.
+```typescript
+import { verifyGoGate, checkPreToolGate } from 'bdb-cicd-resilience/triage/index.js';
+
+const gate = checkPreToolGate('git push origin main', transcriptPath);
+if (!gate.allowed) {
+  throw new Error(`Gate Closed: ${gate.reason}. Reply with GO to unlock.`);
+}
+```
+
+---
+
+## 5. Common Rationalizations vs. Reality
+
+| Rationalization | Engineering Reality | Resilience Protocol |
+|-----------------|---------------------|---------------------|
+| *"A simple `setTimeout(1000)` retry is fine without jitter."* | Synchronous fixed backoffs cause thundering herds on rate-limited endpoints. | **Mandatory Full Jitter**: Sleep for $T = \text{random}(0, \min(T_{\max}, T_{\text{base}} \cdot 2^{\text{attempt}}))$. |
+| *"State file collisions won't happen because agents finish quickly."* | Parallel agent fan-out runs in separate processes; uncoordinated writes cause lost updates. | **Atomic POSIX Lock**: Use `open(..., 'wx')` with fencing tokens or isolated `state.d/<nodeId>.json` fragments. |
+| *"The test failed due to an intermittent CI glitch, let's ignore it."* | Masking real assertion failures as "flaky" leads to shipping broken code to production. | **Deterministic Classifier**: Only classify network/OOM/timeout as flaky; assertion and type errors must never be bypassed. |
+| *"The user said 'starte jetzt', so that counts as approval."* | Compound action verbs violate the two-phase safety protocol; user intent may be unconfirmed. | **Strict Literal Token**: Gate unlocks ONLY if the single trimmed word is `"GO"`. Fail-closed on everything else. |
+
+---
+
+## 6. Red Flags Checklist
+
+- [ ] **Hardcoded Delays**: Retrying without random jitter or exponential backoff.
+- [ ] **Missing Finally Block**: Acquiring a distributed lock without releasing in a `finally` block or relying on process exit.
+- [ ] **Unlink Without Verification**: Deleting a stale lockfile without checking process liveness (`process.kill(pid, 0)`) or using the atomic rename break protocol.
+- [ ] **Synthetic Error Hallucination**: Fabricating an error root cause when log parsing failed or was inconclusive.
+- [ ] **Soft Gate Bypass**: Proceeding with deployment when gate evaluation returned `allowed: false` or when transcript was missing.
+- [ ] **Sidechain Approval**: Permitting an automated subagent message (`isSidechain: true`) to satisfy human approval.
+
+---
+
+## 7. Verification & Testing Protocol
+
+To verify the `/bdbresilience` suite and its underlying TypeScript engine:
+
+```bash
+# 1. Typecheck TypeScript implementation
+npm run typecheck
+
+# 2. Compile to dist/ distribution
+npm run build
+
+# 3. Execute full unit, integration, and E2E verification
+npm test
+```

package/skills/global_config/bdbresilience/contracts/nodes-integration.md

@@ -0,0 +1,225 @@
+# 🤝 Node & Pipeline Integration Contracts
+
+> **Status: partially applied. Read the split below before relying on anything here.**
+>
+> **Applied (2026-09-09).** This skill ships inside AOS at
+> `skills/global_config/bdbresilience/`, and `.agents/nodes.json` lists
+> `bdbresilience` in the `skills` array of two nodes: **`shipping`** (it runs
+> lint/typecheck/tests, which is exactly what the triage parsers read) and
+> **`engineering`** (the error taxonomy and backoff guidance). The dispatcher
+> injects each node's `skills` list into its prompt, so those two nodes are told
+> to reach for this skill. That is the whole of the integration: **guidance, not
+> code.** AOS does not depend on the `bdb-cicd-resilience` package — it has two
+> runtime dependencies and neither is this one.
+>
+> **Not applied, and not currently possible.** Every code-level integration below.
+> `startcycle-dispatch.mjs` contains no `withStateLock` call and cannot contain one:
+> the Workflow runtime gives the dispatcher script **no filesystem access**, which is
+> precisely why AOS solves its own concurrent-write race with single-writer fragments
+> (`production_artifacts/state.d/<node>.json`) plus a merge step instead of a lock.
+> A lock needs a filesystem; the dispatcher does not have one. Do not wire one in.
+>
+> **Where a lock would genuinely fit**, if this is revisited: `installer.js`, which
+> does a real read-modify-write on `~/.agents/.bdb-install-manifest.json` and can be
+> run twice concurrently. That is the only place in AOS with both filesystem access
+> and real contention. Before doing it, note that this library's Windows paths
+> (`EPERM`/`EBUSY` handling, close-before-unlink ordering) are **written but never
+> executed** — two tests skip on non-Windows — and AOS ships to Windows users.
+>
+> §6 documents the `withStateLock()` signature as it actually exists in
+> `src/locking/resource-lock.ts`.
+
+**Deliverable**: Requirement R4 Integration Contracts (proposal)  
+**Target Architecture**: BDB Agent OS (`.agents/nodes.json`, `/startcycle-graph`, `/bdbrainstorm`)  
+**Specification Reference**: `survey_miner_1/survey_report.md`
+
+---
+
+## 1. Proposed Architectural Role
+
+The proposal is to wire the `/bdbresilience` skill suite into the BDB Agent OS multi-agent dispatcher graph. Equipping the **Reviewer** and **Shipping** nodes with resilience capabilities would give the ecosystem automated adversarial auditing of network and concurrency boundaries, and deterministic self-healing quality gates.
+
+Resilience principles would additionally be seeded upstream in **`/bdbrainstorm`** during concept ideation, so that downstream implementation nodes (Engineering, UI/UX, Media) receive explicit reliability requirements.
+
+Applying this proposal is out of scope for the current cycle. It belongs to an AOS integration cycle, which is also where several deferred design questions get their answers — fencing-token verification, the default `ttlMs`, and retry idempotency.
+
+---
+
+## 2. Not adopted: the Reviewer node
+
+> **This proposal was considered and declined on 2026-09-09.** The live
+> `nodes.json` does **not** carry `bdbresilience` in `reviewer.skills`, and that is
+> deliberate, not an oversight.
+>
+> Reviewer reads build artifacts and the plan's contract and argues about
+> correctness. It does not run CI, does not read tool logs, and does not classify
+> retryable failures — the three things this skill is for. A skill allowlist that
+> lists everything guides nothing, so the registration went to `shipping` (which
+> runs the gates whose output the triage parsers read) and `engineering` (which
+> owns the error taxonomy) and stopped there.
+>
+> The section is kept because the reasoning is worth having on record, and because
+> the shape of the entry is a useful template if a future node genuinely needs it.
+
+The **Reviewer** node executes adversarial verification of build-node outputs against the execution plan contract using the *doubt-driven development* discipline. The JSON below is the entry that **would** be added, had this been adopted.
+
+### Declarative Configuration (`.agents/nodes.json`)
+```json
+{
+  "reviewer": {
+    "label": "Reviewer",
+    "agentType": "reviewer",
+    "personaFile": ".claude/agents/reviewer.md",
+    "model": "sonnet",
+    "role": "review",
+    "artifactKey": "review",
+    "writes": null,
+    "optional": false,
+    "skills": [
+      "ui-review",
+      "ux-audit",
+      "architect-review",
+      "systematic-debugging",
+      "bdbresilience"
+    ],
+    "instructions": null
+  }
+}
+```
+
+### Proposed Reviewer Resilience Audit Discipline
+Once equipped with `bdbresilience`, Reviewer would evaluate build outputs (`state.artifacts.{frontend,backend,media}`) against strict resilience checks:
+
+1. **Network & Tool Call Audits**:
+   - Verifies that all external HTTP, API, database, and MCP calls implement structured error classification and Full Jitter exponential backoff.
+   - Flags bare `catch (e) {}` blocks, unhandled Promise rejections, and silent error swallows as:
+     `{ "id": "F-RES-NET-01", "severity": "blocking", "node": "engineering", "status": "open" }`.
+2. **Concurrency & Resource Access Audits**:
+   - Verifies that parallel operations accessing shared files, worktrees, or databases either write isolated fragments (`production_artifacts/state.d/<nodeId>.json`) or acquire a distributed file lock.
+   - Flags missing `finally { await handle.release(); }` blocks as `severity: "blocking"`.
+3. **Ownership Attribution & Stable ID Generation**:
+   - Findings must be attributed strictly to the owning build node (`engineering`, `ui_ux`, or `media`).
+   - Finding IDs must remain stable across cycles (e.g. `F-ENG-01`) so the dispatcher's **No-Progress Guard** can detect stalled repair loops and escalate immediately to human review.
+
+---
+
+## 3. Applied: the Shipping node (`.agents/nodes.json`)
+
+> **Adopted 2026-09-09.** `shipping.skills` now ends with `"bdbresilience"`, and
+> `engineering.skills` likewise. Those two are the whole of the applied
+> integration — see the status block at the top of this file.
+
+The **Shipping** node acts as the release gatekeeper, running mechanical verification gates (lint, typecheck, tests, a11y, seo) after Reviewer findings are cleared. It is the natural home for this skill: the triage parsers read exactly the tsc, ESLint and Jest/Vitest output those gates produce, and `isCleanRun()` is the predicate that decides whether an unrecognised log counts as passing (it does not).
+
+### Declarative Configuration (`.agents/nodes.json`)
+```json
+{
+  "shipping": {
+    "label": "Godmode_Shipping",
+    "agentType": "godmode-shipping",
+    "personaFile": ".claude/agents/godmode-shipping.md",
+    "model": "sonnet",
+    "role": "gate",
+    "artifactKey": "report",
+    "writes": null,
+    "optional": false,
+    "skills": [
+      "godmode-shipping",
+      "webapp-testing",
+      "seo-audit",
+      "wcag-audit-patterns",
+      "github-repo",
+      "clean-code",
+      "bdbresilience"
+    ],
+    "instructions": null
+  }
+}
+```
+
+### Proposed Shipping Quality Gate & Verification Discipline
+Once equipped with `bdbresilience`, Shipping would execute automated diagnostic triage and gating:
+
+1. **Automated Diagnostic Triage Execution**:
+   - Runs mechanical verification commands: `npm run typecheck`, `npm run build`, `npm test`, `npm run lint`.
+   - If a command fails, Shipping pipes stderr/stdout directly into `generateDiagnosticReport(log, runner)`:
+     - A run counts as clean **only if `isCleanRun(report)` is true**, never on `totalFailures === 0` alone — an empty or unparseable log also has zero diagnostics, and reading that as a pass is a fail-open gate.
+     - If all failures are classified as `transient_infra` (`canAutoRetry: true`): Shipping automatically retries the command up to 2 times with Full Jitter before recording a failure.
+     - If any failure is classified as `deterministic_code_regression`: Shipping generates a structured failure section in `production_artifacts/04_release_report.md` detailing the exact failed assertion, target file, line coordinate, and remediation command.
+2. **Gate Population & Node Attribution**:
+   - Updates `state.gate`:
+     ```json
+     {
+       "lint": "pass",
+       "typecheck": "fail",
+       "tests": "pass",
+       "a11y": "skip",
+       "seo": "skip",
+       "blockingNodes": ["engineering"]
+     }
+     ```
+   - Instructs the dispatcher to re-invoke only the failing node (`engineering`) rather than re-running all build nodes.
+3. **Pre-Tool GO Gate Release Protocol**:
+   - When all checks pass, Shipping sets `state.phase = "ready_to_ship"`.
+   - Concludes turn by outputting the Release Report and instructing the operator:
+     > *"All quality gates passed. Reply with the literal word GO to authorize release deployment."*
+   - Strictly prohibits calling `git push` or `npm publish` without human approval verified via `verifyGoGate()`.
+
+---
+
+## 4. Proposed: Integration with `/startcycle-graph`
+
+`~/.claude/workflows/startcycle-dispatch.mjs` and `~/.claude/hooks/graph-gate.mjs` both exist, but neither references this library. The proposal is that `bdbresilience` would power core graph infrastructure:
+
+1. **Parallel Build Fan-Out Isolation**:
+   - When `ui_ux`, `engineering`, and `media` execute concurrently, each node would write exclusively to:
+     `production_artifacts/state.d/<nodeId>.json`
+   - A dedicated barrier folding agent would consolidate fragments into `production_artifacts/state.json` under `withStateLock()`. **This call does not exist in the dispatcher today.**
+2. **Loop Retention Hook (`graph-gate.mjs`)**:
+   - Would intercept turn completion if `state.gate` contains failing checks and `iteration < max_iterations`, blocking session turn-end with exit code 2 to force repair execution. Whether the existing hook already does this independently of `bdbresilience` was not verified for this document.
+3. **No-Progress Guard**:
+   - If Reviewer reports the exact same set of blocking finding IDs across successive iterations, the dispatcher halts immediately (`state.phase = "escalated"`, `state.needs_human = true`). This guard is part of the AOS graph contract, not of this library.
+
+---
+
+## 5. Proposed: Integration with `/bdbrainstorm`
+
+The proposal is that `~/.agents/skills/bdbrainstorm/SKILL.md` would embed resilience requirements into concept planning. Its current content was not inspected for this document:
+
+1. **Pillar 2: `/grill-me` Interactive Inquiries**:
+   - Grills the user on SLA thresholds, API rate limits, failure blast radius, and recovery procedures:
+     - *"What are the rate limits and fallback providers for external dependency X?"*
+     - *"How will concurrent writes to shared database entities be serialized?"*
+2. **Pillar 4: Engineering Godmode (DDD & Clean Architecture)**:
+   - Mandates that domain models represent failure states explicitly as Discriminated Unions (e.g. `Result<T, ClassifiedError>`).
+   - Requires Architecture Decision Records (ADRs) for locking mechanisms and retry backoff strategies.
+3. **Pillar 6: Shipping Godmode & Pipeline Hand-off**:
+   - Packages resilience specifications directly into `state.goal` so downstream Architect and TechLead nodes include them in `production_artifacts/00_execution_plan.md`.
+
+---
+
+## 6. `withStateLock()` — the one part of this document that is real
+
+This function exists today in `src/locking/resource-lock.ts` and behaves as described. It is what a barrier folding agent in §4 would call.
+
+```typescript
+import { withStateLock } from 'bdb-cicd-resilience/locking/index.js';
+
+const merged = await withStateLock(
+  'production_artifacts/state.json',
+  async (state) => {
+    state.findings.push({ id: 'F-ENG-01', status: 'fixed' });
+    return state;
+  },
+  { ttlMs: 15000, acquireTimeoutMs: 10000 }
+);
+```
+
+- Signature: `withStateLock<T>(statePath, updater, options?) => Promise<T>`.
+- The lock resource key is `state:<resolved absolute path>`, so two callers passing different relative paths to the same file still serialize.
+- `acquireTimeoutMs` defaults to `10000` here (not `0` as in `acquireLock`), so callers queue rather than fail on first contention.
+- A missing state file is read as `{}`; any other read error propagates.
+- The updater's return value is written; returning `undefined` writes the state object as mutated.
+- **The write is refused if the lease was lost while the updater ran** — `handle.isExpired()` is checked after the updater returns and before the temp+rename write, and throws rather than clobber a successor's state file.
+- The state file itself *is* written temp+rename. That is unrelated to the lockfile, which is never renamed into place (see `references/distributed-locking.md` §2).
+- The lock is released in a `finally`, and release errors are swallowed.

package/skills/global_config/bdbresilience/references/cicd-triage.md

@@ -0,0 +1,179 @@
+# 🩺 Reference Guide: CI/CD Self-Healing Triage & Diagnostic Reporting
+
+**Pattern**: Pattern 3 — Build, Test & Lint Failure Triage  
+**Module**: `bdb-cicd-resilience/triage`  
+**Authoritative Source**: BDB Agent OS CI/CD Triage Specification
+
+---
+
+## 1. Overview & Problem Statement
+
+When automated CI/CD pipelines fail (during `npm test`, `tsc --noEmit`, `eslint`, or GitHub Actions runs), agents frequently hallucinate root causes by reading unstructured terminal output or trying random edits without identifying the real failure. Alternatively, agents waste execution turns retrying deterministic code bugs, or conversely halt prematurely on transient infrastructure hiccups (such as an npm registry timeout or port bind collision).
+
+The CI/CD Triage Engine ingests raw terminal logs, normalizes ANSI codes and runner annotations, parses failure coordinates across 5 major runners, classifies failures into transient infrastructure versus deterministic regressions, and produces verified JSON diagnostic summaries with exact remediation commands.
+
+---
+
+## 2. Multi-Runner Log Ingestion & Normalization
+
+Raw build logs contain noisy ANSI color sequences, carriage return overwrites (`\r\n`), and runner-specific group wrappers (`##[group]`, `::error::`).
+
+### ANSI Normalization Pipeline
+1. **Control Sequence Stripping**: Removes terminal escapes (`\x1b[[0-9;]*[mGKF]`).
+2. **CRLF Normalization**: Converts Windows line endings to standard Unix newlines (`\r\n` → `\n`).
+3. **CI Annotation Scrubbing**: Normalizes GitHub Actions workflow commands:
+   - `##[group]...##[endgroup]`
+   - `##[error]...`
+   - `::error file={name},line={line}::{message}`
+
+---
+
+## 3. Runner-Specific Log Parsers
+
+The engine incorporates dedicated parsers for each standard ecosystem tool:
+
+| Runner | Detected Signatures | Extracted Data Coordinates | Remediation Command Generated |
+|--------|---------------------|----------------------------|-------------------------------|
+| **Vitest** | `FAIL tests/...`<br>`AssertionError:`<br>`expected ... to be ...` | Test file path, line, column, assertion diff snippet | `npx vitest run <file> -t "<test>"` |
+| **Jest** | `● <describe> › <test>`<br>`Expected: ... Received: ...`<br>`at ... (<file>:<line>:<col>)` | Test spec file, stack frame line/col (prioritizing user code over `node_modules`) | `npx jest <file> -t "<test>"` |
+| **TypeScript (`tsc`)** | `src/auth.ts(42,15): error TS2322`<br>`src/auth.ts:42:15 - error TS2322` | Source file, line, column, TS error code, type mismatch explanation | `npx tsc --noEmit` |
+| **ESLint** | `<file>:<line>:<col>: <msg> [<rule>]`<br>Tabular/Stylish reporter lines | Target source file, line, col, rule ID, severity | `npx eslint --fix <file>` |
+| **GitHub Actions** | `##[error]Process completed with exit code 137`<br>`The operation was canceled`<br>`Request timeout after 30000ms` | Runner step, exit code, memory or timeout fault | Automated infrastructure retry with jitter |
+
+---
+
+## 4. Flaky Infrastructure vs. Deterministic Regression Classifier
+
+The core intelligence distinguishes between errors that can be safely retried and code bugs requiring source modification:
+
+```
+                            [ Parsed Diagnostic Error ]
+                                         │
+                                         ▼
+                 ┌───────────────────────────────────────────────┐
+                 │     classifyDiagnostic(error, context)        │
+                 └───────────────────────┬───────────────────────┘
+                                         │
+                 ┌───────────────────────┴───────────────────────┐
+                 ▼                                               ▼
+     [ TRANSIENT INFRASTRUCTURE ]                   [ DETERMINISTIC REGRESSION ]
+     - HTTP 429 Throttling                          - AssertionError (Expected X, got Y)
+     - Socket ETIMEDOUT / ECONNRESET                - TypeScript Type Error (TS2322)
+     - Runner Exit Code 137 (OOM)                   - SyntaxError / Parsing Failure
+     - EADDRINUSE (Port collision)                  - ESLint Rule Violation
+     - Registry 503 / DNS Blip                      - Missing Export / ReferenceError
+                 │                                               │
+                 ▼                                               ▼
+       canAutoRetry: true                              canAutoRetry: false
+   (Retry step with Full Jitter)                  (Requires source code repair)
+```
+
+### Classification Heuristics
+- **Transient Infrastructure (`transient_infra`)**:
+  - `canAutoRetry: true`
+  - Does NOT count against agent repair loop limits.
+  - Automatically triggered up to 2 times before escalating.
+- **Deterministic Code Regression (`deterministic_code_regression`)**:
+  - `canAutoRetry: false`
+  - Attributed to the responsible build node (Engineering, UI/UX).
+  - Feeds into `production_artifacts/review_findings.md` or `state.findings`.
+- **Unrecognised text** falls to `deterministic_code_regression`, i.e. this classifier **fails closed**. That is the opposite of `classifyError`'s unknown default in the recovery module, and deliberately so — see `error-recovery.md` §7 for why neither should be changed to match the other.
+
+---
+
+## 5. Was Anything Understood? — `parseStatus` and `isCleanRun`
+
+`totalFailures === 0` is **not** the answer to "did this run pass". A log that was empty, and a log that no parser recognised, both produce zero diagnostics. Reading either as a clean gate is a fail-open defect, so the report carries a third axis:
+
+| `parseStatus` | When | `totalFailures` | `canAutoRetry` | `requiresHumanIntervention` |
+|---|---|---|---|---|
+| `parsed` | a runner signature matched, or a parser extracted ≥1 diagnostic | as extracted | `n > 0 && transient === n` | `deterministic > 0` |
+| `empty` | the log is empty or pure ANSI after stripping | `0` | `false` | `true` |
+| `unparsed` | nothing matched — e.g. a Go, pytest or Maven failure | `0` | `false` | `true` |
+
+`empty` is deliberately **not** clean: a build step that produced no output is not evidence that it passed. Neither status ever carries a synthetic diagnostic — inventing one would inflate `totalFailures` and lie to every consumer counting failures.
+
+The predicate is exported so no consumer has to reconstruct the rule:
+
+```typescript
+import { isCleanRun } from 'bdb-cicd-resilience/triage/index.js';
+
+if (isCleanRun(report)) { /* the only sanctioned "it passed" */ }
+```
+
+`isCleanRun(report) === (report.summary.parseStatus === 'parsed' && report.summary.totalFailures === 0)`. Any AOS integration must call it rather than reading `totalFailures` directly.
+
+---
+
+## 6. Structured JSON Diagnostic Schema
+
+`dominantCategory` is `undefined` whenever `totalFailures === 0`; it is not defaulted to a classification.
+
+```json
+{
+  "timestamp": "2026-09-05T14:45:00.000Z",
+  "runner": "vitest",
+  "summary": {
+    "totalFailures": 2,
+    "transientCount": 1,
+    "deterministicCount": 1,
+    "parseStatus": "parsed",
+    "dominantCategory": "deterministic_code_regression",
+    "canAutoRetry": false,
+    "requiresHumanIntervention": true
+  },
+  "diagnostics": [
+    {
+      "file": "tests/unit/auth.test.ts",
+      "line": 42,
+      "column": 14,
+      "assertionSnippet": "expect(user.isAuthenticated).toBe(true)",
+      "classification": "deterministic_code_regression",
+      "rootCause": "AssertionError: expected false to be true // Received user object without auth token",
+      "remediationCommand": "npx vitest run tests/unit/auth.test.ts -t \"verifies authenticated user\"",
+      "canAutoRetry": false
+    },
+    {
+      "file": "src/services/api.ts",
+      "classification": "transient_infra",
+      "rootCause": "FetchError: request to https://registry.npmjs.org timed out after 30000ms (ETIMEDOUT)",
+      "remediationCommand": "npm cache clean --force && npm install",
+      "canAutoRetry": true
+    }
+  ]
+}
+```
+
+---
+
+## 7. TypeScript API Usage Example
+
+```typescript
+import {
+  generateDiagnosticReport,
+  classifyDiagnostic,
+  isCleanRun,
+  stripAnsi
+} from 'bdb-cicd-resilience/triage/index.js';
+
+// Clean and parse raw output
+const cleanLog = stripAnsi(rawStderr);
+const report = generateDiagnosticReport(cleanLog);
+
+if (isCleanRun(report)) {
+  console.log('✅ All checks passed clean.');
+} else if (report.summary.parseStatus !== 'parsed') {
+  console.error(`❌ Log was ${report.summary.parseStatus} — nothing was understood, not a pass.`);
+} else if (report.summary.canAutoRetry && report.summary.deterministicCount === 0) {
+  console.log('⚠️ Transient infrastructure failure detected. Executing auto-retry...');
+  await executeRetryCommand();
+} else {
+  console.error('❌ Deterministic code regression detected:');
+  for (const diag of report.diagnostics) {
+    if (diag.classification === 'deterministic_code_regression') {
+      console.error(`  - ${diag.file}:${diag.line} -> ${diag.rootCause}`);
+      console.error(`    Suggested fix command: ${diag.remediationCommand}`);
+    }
+  }
+}
+```