agent-bober 0.11.6 → 0.15.0

package/skills/bober.deploy/SKILL.md

@@ -0,0 +1,262 @@
+---
+name: bober-deploy
+description: Use when executing a remediation action — classifies by blast radius, gates risky actions via Tier 2 checkpoint, records a ChangeEntry with inverse BEFORE execution. The execution-level discipline that runbook steps delegate to.
+---
+
+# Remediation Execution Discipline
+
+## Overview
+
+The deploy skill governs **how** a remediation action is executed — the precondition check, the risky-action gate, the execution itself, the ChangeEntry write, and the postcondition verification. It is the execution substrate that `bober.runbook` steps delegate to and that the `bober-deployer` agent implements.
+
+The spirit of this discipline: **every change must be auditable, reversible, and gated by proportional human oversight**. Risky changes that cannot be reversed without human judgment must always pass through a checkpoint. This is not bureaucracy — it is the minimum viable safety net for a system that executes shell commands against production infrastructure.
+
+## The Iron Law
+
+```
+NO RISKY ACTION WITHOUT CHECKPOINT APPROVAL; NO ACTION WITHOUT RECORDED INVERSE
+```
+
+Both clauses are unconditional. They do not have exceptions for urgency, familiarity, or pipeline mode.
+
+## When to Use
+
+Use this skill whenever:
+- Executing a remediation action proposed by the `bober-diagnoser` agent
+- Executing a runbook step with `blastRadius: 'risky'`
+- Running any shell command that modifies system state (cluster, database, secrets, filesystem)
+- Recording a deployment, configuration change, or rollback to the incident changelog
+
+Do NOT use this skill for:
+- Read-only investigations (use `bober.diagnose`)
+- Runbook authoring (see `bober.runbook` for the step format)
+- Postmortem writing (see `bober.postmortem`)
+
+## Action Classification
+
+### SAFE Actions
+
+A safe action is one where: (a) it is read-only, (b) it can be reversed by simply re-running it with different parameters (idempotent redo), or (c) it flips a feature flag back to its default state.
+
+| Category | Examples |
+|----------|---------|
+| Read-only cluster queries | `kubectl get`, `kubectl describe`, `kubectl logs`, `kubectl top` |
+| Read-only container queries | `docker ps`, `docker logs`, `docker inspect` |
+| Read-only file operations | `cat`, `head`, `tail`, `grep`, `find`, `jq` |
+| Read-only HTTP probes | `curl -I`, `curl -X GET` |
+| Read-only git operations | `git log`, `git diff`, `git status`, `git show` |
+| System state reads | `ps`, `df`, `lsof`, `netstat`, `uptime` |
+| Observability queries | All `obs__*__*` tools |
+| Feature flag to default | `ff --set my.flag=false` when `false` is the declared default |
+
+### RISKY Actions
+
+A risky action is one that is stateful, destructive, or externally observable — i.e., a failure could affect users, require manual recovery, or leave the system in an indeterminate state.
+
+| Category | Examples |
+|----------|---------|
+| Kubernetes mutations | `kubectl scale`, `kubectl rollout restart`, `kubectl delete`, `kubectl apply`, `kubectl patch`, `kubectl edit` |
+| Infrastructure mutations | `terraform apply`, `terraform destroy`, `helm install/upgrade/uninstall/rollback` |
+| Database migrations | `alembic upgrade`, `rake db:migrate`, `flyway migrate`, `knex migrate`, `liquibase update` |
+| Secret rotation | `vault write/rotate/delete`, `aws secretsmanager rotate-secret/put-secret-value` |
+| DNS changes | `aws route53 change-resource-record-sets`, `gcloud dns record-sets create` |
+| Load balancer config | `aws elbv2 modify-listener`, `aws elbv2 modify-target-group-attributes` |
+| Process control | `systemctl start/stop/restart`, `service ... restart`, `kill`, `pkill`, `killall` |
+| Package installation | `npm install`, `pip install`, `apt install`, `brew install`, `yarn add` |
+| Privilege escalation | Any command prefixed with `sudo` |
+| State-mutating HTTP | `curl -X POST/PUT/PATCH/DELETE` |
+| File mutations | `rm`, `rmdir`, `mv` (overwrite), `cp` (overwrite), shell redirects `>`, `>>` |
+| Feature flag from default | Any flag change that moves away from the declared default state |
+
+### Classification Rule
+
+**WHEN IN DOUBT: classify risky.**
+
+The cost of a false-risky classification is one extra checkpoint approval. The cost of a false-safe classification is an unreviewed mutation to production infrastructure.
+
+The classifier (`classifyCommand()` in `src/orchestrator/deploy/classify.ts`) evaluates the **entire command string** — not just the leading verb. A multi-command Bash invocation such as `echo 'safe' && kubectl scale deployment api --replicas=6` is **risky** because `kubectl scale` appears in the command string. Wrapping a risky verb inside a safe-looking command does not change the blast radius.
+
+## Execution Loop
+
+Execute each proposed action in this exact sequence. Do not skip steps. Do not reorder steps.
+
+```
+FOR each ProposedAction (id, description, classification, reasoning, command, inverse):
+
+  1. VALIDATE: assert inverse.description is non-empty. If empty → ABORT (reason: missing_inverse).
+     No ChangeEntry is written for an aborted action.
+
+  2. CLASSIFY: re-run classifyCommand(action.command). If the executor's classification is
+     'risky' even though action.classification is 'safe', treat the action as risky.
+     (The executor is the authoritative classifier — the agent's field is a hint.)
+
+  3. LOG: append ActionEntry to actions.jsonl for the audit trail.
+
+  4. PRECONDITION CHECK: if action.preconditionCheck is defined, run it.
+     If precondition fails → ABORT (reason: precondition_failed). No ChangeEntry written.
+
+  5. GATE (risky actions only):
+     a. Resolve mechanism via resolveRiskyActionMechanismName(config, isRisky=true).
+     b. IF allowAutopilotRiskyActions=false (default): invoke mech.request() with the
+        action description, classification reasoning, command, and inverse.
+     c. IF outcome.approved=false → ABORT (reason: checkpoint_rejected). Append timeline event.
+        Do NOT execute. Do NOT write ChangeEntry.
+     d. IF outcome.edit=true → re-classify the modified command before executing.
+     e. IF allowAutopilotRiskyActions=true → skip interactive approval, log STERN WARNING to
+        stderr, proceed to execution. ChangeEntry IS STILL WRITTEN (audit trail preserved).
+
+  6. WRITE ChangeEntry status='pending' to changelog.jsonl BEFORE execution.
+     (This ensures the ChangeEntry exists on disk even if the process crashes mid-execution.)
+
+  7. EXECUTE via executor seam (defaultExecutor in production; injected seam in tests).
+
+  8. WRITE ChangeEntry status='executed' | 'failed' to changelog.jsonl AFTER execution.
+     (Both 'pending' and terminal entries are present — operational tooling sees the transition.)
+
+  9. POSTCONDITION CHECK: if action.postconditionCheck is defined, run it.
+     If postcondition fails → invoke Abort Discipline (see below).
+
+ 10. RECORD result in DeployResult (executed or aborted array).
+```
+
+## Hard Gate — Risky Actions
+
+Any action classified as risky MUST invoke the Tier 2 checkpoint mechanism before execution. This is UNCONDITIONAL:
+
+- **`pipeline.mode='autopilot'` does NOT bypass risky-action approval.** Autopilot trades human-in-the-loop for speed on SAFE actions; the risky-action gate is the production safety floor and does not move.
+- **`pipeline.checkpointMechanism='noop'` does NOT apply to risky actions.** When the configured mechanism is `noop` but the action is risky, the executor uses the default `disk` fallback. The gate cannot be configured away.
+- **Multi-command Bash invocations do NOT slip through the gate.** An action that wraps `kubectl scale` inside `echo 'safe' && kubectl scale ...` is classified by COMMAND CONTENT, not by step authorship. The classifier checks for state-mutating verbs in the entire command string.
+
+The gate receives the action description, the classification reasoning, the proposed command, and the declared inverse. The operator can approve, reject, or modify. A modification is re-classified before execution.
+
+<EXTREMELY-IMPORTANT>
+Risky actions invoke the Tier 2 checkpoint mechanism regardless of pipeline.mode. Autopilot mode does NOT bypass risky-action approval. If `pipeline.mode='autopilot'` and `pipeline.checkpointMechanism='noop'`, the executor STILL invokes a non-noop mechanism (default 'disk' fallback) for any action classified as risky. This is the production safety guarantee — bypassing it forfeits the guarantee.
+</EXTREMELY-IMPORTANT>
+
+## allowAutopilotRiskyActions Escape Hatch
+
+`pipeline.allowAutopilotRiskyActions=true` is available for **fully-automated environments** (CI pipelines, batch remediation jobs) where no human is available to approve a checkpoint. Default: `false`.
+
+When `true`:
+- Interactive approval is skipped.
+- A STERN WARNING is logged to stderr: `[bober deploy] WARN allowAutopilotRiskyActions=true — auto-approved risky action <id>: <description>. Inverse recorded: "<inverse.description>".`
+- The ChangeEntry **IS STILL WRITTEN** with the required `inverse` field. The audit trail is ALWAYS preserved.
+- This is **"skip the interactive approval"** — NOT **"skip the audit trail"**.
+
+<EXTREMELY-IMPORTANT>
+`pipeline.allowAutopilotRiskyActions=true` is a footgun. Setting it to `true` in a non-automated environment (i.e., a human-supervised incident response) removes the human checkpoint that catches misclassifications, operator errors, and cascade failures. Default `false` is the SAFE default. Set `true` ONLY when no human is available AND the risk of delayed remediation exceeds the risk of unreviewed execution. Document the justification in the incident postmortem.
+</EXTREMELY-IMPORTANT>
+
+## ChangeEntry Write-then-Update
+
+Every executed action writes TWO ChangeEntries to `changelog.jsonl`:
+
+1. **Before execution** — `status: 'pending'`. Written BEFORE the executor seam is called.
+   Purpose: if the process crashes mid-execution, the entry exists on disk. Operational tooling
+   can detect 'pending' entries that never transitioned to 'executed' or 'failed' and flag them
+   for manual review.
+
+2. **After execution** — `status: 'executed'` or `status: 'failed'`. Written AFTER the executor
+   returns (or throws). Both entries share the same `id` field; readers correlate by `id`.
+
+The `inverse` field is REQUIRED on BOTH entries. Sprint 21 rollback awareness reads `inverse` from
+changelog entries to reconstruct the rollback plan. An entry without `inverse` is a schema violation
+(Zod will throw at write time).
+
+```jsonl
+{"id":"act-1","type":"risky-action","executedAt":"2026-05-25T12:00:00Z","description":"scale api to 6","inverse":{"description":"scale back to 3","command":"kubectl scale deployment api --replicas=3"},"status":"pending"}
+{"id":"act-1","type":"risky-action","executedAt":"2026-05-25T12:00:02Z","description":"scale api to 6","inverse":{"description":"scale back to 3","command":"kubectl scale deployment api --replicas=3"},"status":"executed"}
+```
+
+## Abort Discipline
+
+When a postcondition check fails after execution, follow this three-step cascade:
+
+**Step 1 — Execute the declared inverse.**
+The inverse is the rollback command declared in `action.inverse.command`. Run it via the executor seam. The inverse itself is classified by `classifyCommand()` — if it is risky, it requires checkpoint approval too.
+
+**Step 2 — Verify the inverse's effect.**
+After executing the inverse, run the original precondition (or the action's postcondition with inverted expected state) to confirm the rollback held. If the inverse also fails, proceed to Step 3.
+
+**Step 3 — Escalate via checkpoint and STOP.**
+Even if Step 1 failed, escalate via the Tier 2 checkpoint mechanism with the full context: the failed action, the postcondition result, the inverse attempt result, and the current observable state. STOP — do not proceed to subsequent actions. Their preconditions may assume this action's postcondition held, which it did not.
+
+<EXTREMELY-IMPORTANT>
+If a postcondition fails AND the declared inverse also fails (or no inverse was declared), the incident state is now indeterminate. The executor MUST escalate via checkpoint — do not silently proceed, do not retry the failed action. Only a human (or the configured escalation handler) can decide the next move from an indeterminate state.
+</EXTREMELY-IMPORTANT>
+
+## Worked Example — Scaling API Deployment
+
+**Context:** Diagnoser hypothesizes replica exhaustion. Next action: `kubectl scale deployment api --replicas=6`.
+
+**ProposedAction:**
+```json
+{
+  "id": "act-scale-1",
+  "description": "Scale api deployment to 6 replicas to relieve replica pressure",
+  "classification": "risky",
+  "reasoning": "kubectl scale is stateful and externally observable — changes live traffic routing",
+  "command": "kubectl scale deployment api --replicas=6 -n prod",
+  "inverse": {
+    "description": "Scale api deployment back to 3 replicas",
+    "command": "kubectl scale deployment api --replicas=3 -n prod"
+  },
+  "preconditionCheck": "kubectl get deployment api -n prod -o jsonpath='{.status.readyReplicas}'",
+  "postconditionCheck": "kubectl get deployment api -n prod -o jsonpath='{.status.readyReplicas}' | grep -q '^6$'"
+}
+```
+
+**Execution trace:**
+1. `inverse.description` is non-empty — validation passes.
+2. `classifyCommand("kubectl scale deployment api --replicas=6 -n prod")` → `'risky'` (kubectl scale verb).
+3. ActionEntry written to `actions.jsonl`.
+4. Precondition check: `kubectl get deployment api ...` → returns `3` (replicas currently 3) — passes.
+5. Gate: mechanism resolves to `disk` (floor applies; checkpointMechanism=noop but action is risky). Operator approves via `.bober/approvals/` file.
+6. ChangeEntry `{id: "act-scale-1", status: "pending", inverse: {...}}` written to `changelog.jsonl`.
+7. Executor: `kubectl scale deployment api --replicas=6 -n prod` → exit code 0.
+8. ChangeEntry `{id: "act-scale-1", status: "executed", inverse: {...}}` written to `changelog.jsonl`.
+9. Postcondition check: `kubectl get deployment api ... | grep -q '^6$'` → passes.
+10. DeployResult: `executed: [{actionId: "act-scale-1", status: "executed", durationMs: 1240}]`.
+
+## Red Flags — STOP
+
+- About to execute without an `inverse.description` on the ProposedAction — stop, you have no exit strategy.
+- About to classify `echo 'safe' && kubectl scale ...` as safe — the classifier reads the entire string. `kubectl scale` makes it risky.
+- About to skip the checkpoint because the pipeline is in autopilot mode — Iron Law: risky actions always gate.
+- About to skip the ChangeEntry write because "the action is small" — the audit trail is the safety net for the next operator. Every change is recorded.
+- About to skip the precondition check because "the incident confirms the bad state" — the precondition is also a guard against executing the wrong remediation on the wrong environment.
+- About to continue to the next action after a postcondition failure — this is the most common failure mode. Stop. Execute the inverse. Escalate. Let the operator decide.
+- About to set `allowAutopilotRiskyActions=true` in a human-supervised context — this flag is for unattended automation. In a live incident with a human in the loop, leave it `false`.
+- About to skip the stern warning when `allowAutopilotRiskyActions=true` auto-approves — the warning is the audit signal that human approval was bypassed.
+
+## Common Rationalizations
+
+| Rationalization | Reality |
+|-----------------|---------|
+| "The pipeline is in autopilot, so risky actions auto-approve" | Iron Law: risky actions always gate, regardless of pipeline.mode. Autopilot only skips approval for SAFE actions. |
+| "kubectl scale is a minor operation — it's basically safe" | kubectl scale is stateful and externally observable. It is in the RISKY list explicitly. Classify it risky. |
+| "I'll add the inverse after I see what the execution does" | The inverse must be declared BEFORE execution. Discovering it post-hoc means you cannot roll back if the execution crashes. |
+| "allowAutopilotRiskyActions=true means skip all safety" | It means skip interactive approval. ChangeEntry IS still written. Audit trail IS still preserved. Warning IS still logged. |
+| "The diagnoser recommended this — it's pre-approved" | Recommendation is not approval. Every risky action needs a checkpoint approval regardless of its source. |
+| "The precondition passed last time — I won't check again" | System state changes. The precondition check is run immediately before execution, every time. |
+| "Different words so the rule doesn't apply" | Spirit over letter. When in doubt, classify risky, require approval, record inverse. |
+| "I can bundle two mutations into one command to save time" | Bundled mutations have bundled inverses. A failure mid-bundle leaves the system in a half-mutated state. Split them. |
+
+## Quick Reference
+
+| Question | Answer |
+|----------|--------|
+| Is `kubectl get pods` safe? | Yes — read-only. |
+| Is `kubectl scale` safe? | No — risky, requires checkpoint. |
+| Is `echo 'ok' && kubectl delete pod x` safe? | No — `kubectl delete` is risky; entire string is risky. |
+| Can autopilot mode bypass risky-action checkpoint? | No — Iron Law applies unconditionally. |
+| What does `allowAutopilotRiskyActions=true` skip? | Interactive approval only. ChangeEntry is still written. Warning is still logged. |
+| What happens if inverse is missing? | executeAction throws BEFORE execution. No ChangeEntry is written. |
+| What happens if postcondition fails? | Execute inverse → escalate via checkpoint → STOP. |
+| What happens if the executor crashes mid-execution? | ChangeEntry with status='pending' exists on disk. Final status may be absent or 'failed'. Operational tooling detects the 'pending' state. |
+
+## Related Skills
+
+- **`bober.runbook`** (`skills/bober.runbook/SKILL.md`) — multi-step runbook execution. Runbook steps delegate to this skill's execution discipline for each step.
+- **`bober.diagnose`** (`skills/bober.diagnose/SKILL.md`) — the investigation skill that produces `nextActions`. This skill executes what the diagnoser recommends.
+- **`bober-deployer` agent** (`agents/bober-deployer.md`) — the agent that uses this skill. The agent prompt implements the discipline described here.

package/skills/bober.diagnose/SKILL.md

@@ -0,0 +1,254 @@
+---
+name: bober-diagnose
+description: Use when investigating a production incident or system-level failure — gather evidence at component boundaries, hypothesize-and-disprove, verify resolution against pre-defined criteria
+---
+
+> Adapted from [obra/superpowers](https://github.com/obra/superpowers) — MIT License.
+> Structural source: skills/systematic-debugging/SKILL.md (four-phase discipline).
+> Adaptations: system-level (incident) context; boundary enumeration as Phase 2; pre-defined resolution-verification criteria as Phase 4.
+
+# Systematic Incident Diagnosis
+
+## Overview
+
+Random restarts mask root causes. Symptom-fixes destroy the ability to verify resolution. Reactive remediation without confirmed root cause prolongs incidents and introduces new failures.
+
+**Core principle:** ALWAYS verify root cause at two independent boundaries before remediation.
+
+**Violating the letter of this process is violating the spirit of incident response.**
+
+## The Iron Law
+
+```
+NO REMEDIATION WITHOUT VERIFIED ROOT CAUSE AT TWO INDEPENDENT BOUNDARIES
+```
+
+If your evidence comes from a single component, you have a candidate hypothesis — not a verified root cause. Continue gathering at independent boundaries before proposing any remediation.
+
+## When to Use
+
+Use for ANY production incident:
+- Error rate or latency regression
+- Partial or total service outage
+- Capacity event or unexpected resource exhaustion
+- Data inconsistency or corruption report
+- Security alert or anomalous traffic pattern
+- Cascading failures across components
+
+**Use this ESPECIALLY when:**
+- Under time pressure (a page just fired — emergencies make guessing tempting)
+- A restart "seems like it might help"
+- The dashboard looks bad and everyone is watching
+- You have one promising hypothesis and want to skip straight to fixing
+- You've already tried one change and it didn't work
+
+**Don't skip when:**
+- Incident seems obvious ("it was the 14:00 deploy") — correlation must become causation via independent evidence
+- Symptom looks simple (one endpoint, one user) — scope must be confirmed, not assumed
+- Executive is watching — systematic diagnosis is FASTER than repeated rollbacks that don't fix the root cause
+
+## The Four Phases
+
+You MUST complete each phase before proceeding to the next. The gates between phases are not advisory — they are the process.
+
+### Phase 1: Reproduce and Confirm
+
+**BEFORE gathering evidence at component boundaries:**
+
+1. **Confirm Symptom Is Current (not stale)**
+   - Do NOT rely on the initial user report timestamp — that is when it was NOTICED, not necessarily when it is happening NOW.
+   - Query an observability MCP at this moment: `obs__datadog__query_metric`, `obs__loki__query_logs`, or equivalent configured in `bober.config.json` → `observability.providers`.
+   - If the symptom is no longer observable, record that — the incident may be self-resolved or intermittent. Do NOT proceed to Phase 2 on a stale symptom without confirming current status.
+
+2. **Confirm Scope**
+   - Is it one user, one customer, one region, one endpoint, or all of them?
+   - Scope determines which component boundaries matter in Phase 2. A tenant-isolated failure points to tenant-specific data paths; a global failure points to shared infrastructure.
+   - Record scope explicitly — "assumed global" is not confirmation.
+
+3. **Confirm Timing**
+   - When exactly did it start? (First alert timestamp ≠ first occurrence)
+   - Has severity changed since it started — increasing, plateau, or decreasing?
+   - Was there a deploy, config change, or infrastructure event within the relevant window? (Record as a potential correlation — not yet a cause.)
+
+4. **Record Initial State to `observations.jsonl`**
+   - Append each confirmed observation to `.bober/incidents/<id>/observations.jsonl` with the shape: `{timestamp, phase, observation, source, verified}`.
+   - Do NOT record assumptions as observations. `"verified": true` means you queried a source and got the data; `"verified": false` means a user-reported claim not yet confirmed.
+
+   **Worked example:**
+   ```jsonl
+   {"timestamp": "2026-05-24T14:05:00Z", "phase": 1, "observation": "Symptom: 500 errors on /api/checkout from 14:00 UTC; ~12% error rate; all regions", "source": "user-report", "verified": false}
+   {"timestamp": "2026-05-24T14:06:00Z", "phase": 1, "observation": "Confirmed current via fresh metric query — error rate still 11.8% at 14:06", "source": "obs__datadog__query_metric", "verified": true}
+   {"timestamp": "2026-05-24T14:07:00Z", "phase": 1, "observation": "Scope: all regions, all customers — global incident, not tenant-isolated", "source": "obs__datadog__query_metric", "verified": true}
+   ```
+
+<EXTREMELY-IMPORTANT>
+BEFORE proceeding to Phase 2, you MUST have completed Phase 1 in writing — symptom confirmed current, scope confirmed, timing confirmed, observations.jsonl appended. If any of these is incomplete, return to Phase 1. Skipping Phase 1 makes Phase 2 evidence-gathering ungrounded.
+</EXTREMELY-IMPORTANT>
+
+### Phase 2: Gather Evidence at Boundaries
+
+**BEFORE forming hypotheses, gather at every relevant boundary:**
+
+1. **Enumerate Component Boundaries**
+   - Draw the request/data path end to end: `client → CDN → load balancer → API gateway → service → cache → database → storage`.
+   - Each arrow between components is a boundary — a place where data crosses a trust or technology boundary and where failure can manifest differently on each side.
+   - The scope confirmed in Phase 1 determines which boundaries are relevant. A global failure requires querying all boundaries. A single-region failure may skip globally-shared boundaries.
+
+2. **Query at Each Boundary via Observability MCPs**
+   - Use the `obs__<provider>__<tool>` namespace configured in `bober.config.json` → `observability.providers`.
+   - Provider kinds: `logs | metrics | traces | errors | custom`.
+   - Query examples per boundary layer:
+     - CDN/network layer: `obs__cloudflare__query_analytics` — cache-hit rate, 5xx responses at edge
+     - App-layer logs: `obs__loki__query_logs` — structured error logs, trace IDs
+     - Infra metrics: `obs__datadog__query_metric` — CPU, memory, connection pool saturation
+     - Distributed traces: `obs__tempo__query_traces` — end-to-end latency, where spans break
+     - Error tracking: `obs__sentry__query_events` — exception types, frequency, first seen
+   - Record each query result to `observations.jsonl` with `phase: 2` and the `obs__<provider>__<tool>` value as `source`.
+
+3. **Correlate Timestamps with `changelog.jsonl`**
+   - Read `.bober/incidents/<id>/changelog.jsonl` for recent deploys, config changes, and infrastructure events.
+   - Cross-reference incident-start timestamp (confirmed in Phase 1) with deploy and change timestamps.
+   - **CRITICAL:** Temporal correlation is not causation. A deploy 5 minutes before symptom onset is a hypothesis input, not a conclusion. Record it as a candidate, not a confirmed cause.
+
+4. **Multi-Boundary Iron-Law Check**
+   - Before any hypothesis becomes remediation-eligible, you MUST have evidence from at least TWO independent boundaries.
+   - "Two log entries from the same service" is NOT two independent boundaries — that is one boundary queried twice.
+   - Two independent boundaries means two different components in the system path, each providing telemetry that either supports or contradicts the same hypothesis.
+
+<EXTREMELY-IMPORTANT>
+BEFORE proceeding to Phase 3, you MUST have queried at least two independent boundaries and recorded their findings as observations. If only one boundary has data, return to Phase 2 — Phase 3 hypothesis formation on single-boundary evidence violates the Iron Law.
+</EXTREMELY-IMPORTANT>
+
+### Phase 3: Hypothesize and Disprove
+
+**Scientific method under pressure:**
+
+1. **Formulate Falsifiable Hypotheses**
+   - Each hypothesis is a single falsifiable claim: "The database connection pool is exhausted, causing checkout service to queue and timeout, explaining the 500 errors on /api/checkout."
+   - Rank hypotheses by count of supporting evidence entries across independent boundaries — more independent boundary support = higher initial rank.
+   - Drop hypotheses with zero evidence from Phase 2. Ungrounded hypotheses are guesses, not hypotheses.
+
+2. **Pattern-Match Against the Anti-Pattern Catalog**
+   - Before listing a hypothesis as a candidate, check `.bober/anti-patterns/README.md` for pattern matches.
+   - Two anti-patterns are especially common in incidents: **Symptom-Fix Instead of Root-Cause** (see `root-cause-tracing.md`) and **Single-Layer Validation** (see `defense-in-depth.md`).
+   - If your hypothesis matches one, cite the anti-pattern by name in your hypothesis record.
+
+3. **ACTIVELY DISPROVE the Top Hypothesis** *(REQUIRED — not advisory)*
+   - Try to find evidence that DISPROVES your top hypothesis. A hypothesis you cannot disprove is not strongly tested.
+   - For each top hypothesis, ask: what would NOT be true if this hypothesis were correct? Go look for that evidence.
+   - Example: if your hypothesis is "the 14:00 deploy caused the error spike," actively look for contradicting evidence — was the error rate elevated BEFORE 14:00? Is the same endpoint failing in a region that did NOT receive the 14:00 deploy? Are the error signatures different from what a rollout regression would produce?
+   - Record the disproof attempt in `hypotheses.md` under the hypothesis. If you found contradicting evidence, demote the hypothesis. If you looked and found no contradicting evidence, the hypothesis survived the attempt — that is meaningful.
+   - You MUST document what you checked and what you found (or did not find). "I tried to disprove it" without a recorded attempt is not a disproof attempt.
+
+4. **Promote or Demote Confidence**
+   - Promote a hypothesis to `confidence: 'high'` ONLY if: (a) evidence from ≥2 independent boundaries AND (b) survived an active disproof attempt with no contradicting evidence found.
+   - Promote to `confidence: 'medium'` if: multi-boundary evidence but disproof attempt inconclusive (no contradicting evidence found, but also no strong confirming evidence).
+   - Assign `confidence: 'low'` if: single-boundary evidence OR disproof attempt found partially contradicting evidence.
+   - Only `confidence: 'medium'` or `confidence: 'high'` hypotheses are remediation-eligible. `confidence: 'low'` → return to Phase 2 for more evidence.
+   - `blastRadius: 'risky'` remediation actions require `requiresApproval: true` — set this in the `nextActions` entry regardless of confidence level.
+
+<EXTREMELY-IMPORTANT>
+BEFORE proceeding to Phase 4, you MUST have actively attempted to disprove your top hypothesis and recorded the attempt. A hypothesis that has NOT survived a disproof attempt is NOT remediation-eligible. If you have not yet tried to disprove it, return to Phase 3.
+</EXTREMELY-IMPORTANT>
+
+### Phase 4: Verify Resolution Against Pre-Defined Criteria
+
+**Resolution criteria MUST be defined BEFORE remediation — retrofitted criteria are meaningless.**
+
+1. **Pre-Define Resolution-Verification Criteria**
+   - Before ANY state-mutating action, write the resolution criteria. All five fields are REQUIRED: metric, threshold, window, comparison baseline, verification source. Without all five, the criterion is not actionable.
+
+   **Worked example:**
+   ```
+   Resolution criteria for inc-20260524-500-errors-on:
+     - Metric: api.checkout.error_rate
+     - Threshold: < 0.1%
+     - Window: 10 minutes sustained
+     - Comparison baseline: 7-day rolling average
+     - Verification source: obs__datadog__query_metric
+   ```
+
+   - Record these criteria in `actions.jsonl` BEFORE the remediation action that follows.
+   - Criteria written after the fact are retrofitted to the outcome and provide no verification value.
+
+2. **Apply Remediation via bober-deployer**
+   - NEVER run state-mutating commands from the diagnoser directly. The diagnoser emits a `nextActions` entry; the orchestrator routes it to `agents/bober-deployer.md` (Sprint 20 — `skills/bober.deploy/SKILL.md` once that sprint lands).
+   - Actions classified `blastRadius: 'risky'` MUST include `requiresApproval: true` and will trigger a Tier 2 checkpoint gate before execution.
+   - The deployer records each action in `changelog.jsonl` with a required `inverse` field — the rollback instruction for every state change.
+   - See `skills/bober.deploy/SKILL.md` (Sprint 20) for the full remediation execution discipline.
+
+3. **Monitor Against Criteria via Observability MCPs**
+   - After remediation completes, begin monitoring the named metric using the named verification source.
+   - Query at a cadence appropriate to the window (e.g., every 2 minutes for a 10-minute window).
+   - Record each observation to `observations.jsonl` with `phase: 4`.
+   - "The dashboard looks better" is NOT resolution. The criterion requires the named metric to meet the named threshold for the FULL named window.
+
+4. **Mark Resolved Only When Criteria Met for the Named Window**
+   - When the metric meets the threshold for the complete window duration, append resolution to `actions.jsonl` and update `incident.json` with `status: 'resolved'` and `resolvedAt`.
+   - If criteria are not met within a reasonable monitoring period, or if the symptom returns, return to Phase 1 — the remediation was symptomatic, not root cause. The incident is not closed.
+   - Explicitly forbidden: marking an incident resolved because the dashboard looks better, because a restart seemed to help, or because the page quieted down. These are not resolution criteria — they are noise.
+
+## Red Flags - STOP and Follow Process
+
+If you catch yourself thinking:
+- "The dashboard looks better, mark it resolved"
+- "Just restart the service and see if it helps"
+- "It's obviously the database, skip the cache layer"
+- "The deploy at 14:00 must be it, ship the rollback"
+- "One log line is enough — I can see the error right there"
+- "No time for hypothesis-disproof, the page is loud"
+- "The metric is back to baseline, declare resolved"
+- "Stale alert — the customer probably just refreshed"
+- "I've seen this before, I know what it is"
+- Proposing remediation before confirming evidence at two independent boundaries
+- **"Just one restart to stabilize, then we'll investigate"**
+
+**ALL of these mean: STOP. Return to Phase 1.**
+
+## Common Rationalizations
+
+| Excuse | Reality |
+|--------|---------|
+| "The dashboard looks green, we're resolved" | Resolution criteria require a NAMED metric meeting a NAMED threshold for a NAMED window. Eyeballing a dashboard is not verification. |
+| "The deploy at 14:00 caused this — roll it back" | Correlation is not causation. Verify the deploy is the cause via independent telemetry before remediating. Rolling back the wrong thing extends the incident. |
+| "Logs are unambiguous, one source is enough" | Iron Law: two independent boundaries. One source = continue gathering. Do not remediate on single-boundary evidence. |
+| "No time to disprove the hypothesis, the page is loud" | Confirmation bias under pressure is the dominant incident-response failure mode. The disproof step exists EXACTLY for these moments. Skip it and you risk a second incident. |
+| "Stale alert — the customer probably just refreshed" | Phase 1 requires CONFIRMING the symptom is current. "Probably refreshed" is not confirmation — query current state before proceeding. |
+| "We'll set the resolution criteria after the fix lands" | Criteria set after the fix are retrofitted to the outcome. They MUST be pre-defined to be meaningful. Post-hoc criteria always pass. |
+| "I've seen this before, skip to Phase 3" | Pattern memory is a hypothesis, not evidence. Phase 1 (confirm) and Phase 2 (boundaries) still produce the multi-source evidence required by the Iron Law. |
+| "The MCP is slow, I'll just go from logs" | If a primary observability source is degraded, that is itself a diagnostic signal. Do NOT invent values for missing telemetry — low-confidence gaps are hypotheses, not evidence. |
+
+## Quick Reference
+
+| Phase | Key Activities | Success Criteria |
+|-------|---------------|------------------|
+| **1. Reproduce + Confirm** | Confirm symptom current, confirm scope, confirm timing, write `observations.jsonl` | Symptom is real, scoped, and timed |
+| **2. Gather at Boundaries** | Enumerate components; query `obs__<provider>__<tool>` at each boundary; correlate `changelog.jsonl` | Evidence at ≥2 independent boundaries |
+| **3. Hypothesize + Disprove** | Rank hypotheses by evidence; pattern-match anti-catalog; actively seek contradicting evidence | Top hypothesis survived a recorded disproof attempt |
+| **4. Verify Resolution** | Pre-define metric+threshold+window+baseline+source; remediate via bober-deployer; monitor; mark resolved | Criteria met for the full named window |
+
+## When Process Reveals "No Root Cause"
+
+If systematic investigation across all relevant boundaries turns up no root cause:
+
+1. You have still completed the process — this is a valid outcome, not a process failure.
+2. Document every boundary queried and every hypothesis disproved in `hypotheses.md`.
+3. Implement appropriate defensive handling (circuit breaker, retry with backoff, graceful degradation).
+4. Add monitoring for early detection of recurrence.
+
+**But:** 95% of "no root cause" cases are incomplete boundary coverage. Check that Phase 2 covered every component in the request path before declaring no root cause.
+
+## Related Skills
+
+- **`bober.debug`** (`skills/bober.debug/SKILL.md`) — Use when the incident root cause turns out to be code-level (a test reproduces it, single component, deterministic). `bober.diagnose` handles "is the system broken?"; `bober.debug` handles "is the code wrong?". They are siblings, not parent-child. Used by `agents/bober-diagnoser.md`.
+- **`bober.runbook`** (`skills/bober.runbook/SKILL.md`, Sprint 18) — When the diagnoser's next action is "follow runbook X", use `bober.runbook` for execution discipline (precondition → execute → postcondition for every step).
+- **`bober.deploy`** (`skills/bober.deploy/SKILL.md`, Sprint 20) — Remediation execution via `agents/bober-deployer.md`. Required for any `blastRadius: 'risky'` action. Never run state-mutating commands from the diagnoser — always route through bober-deployer.
+- **`.bober/anti-patterns/`** — Pattern catalog. Phase 3 hypothesis formation must check **Symptom-Fix Instead of Root-Cause** (`root-cause-tracing.md`) and **Single-Layer Validation** (`defense-in-depth.md`) for matches.
+
+## Real-World Impact
+
+From incident response patterns:
+- Systematic diagnosis with disproof discipline: median time-to-resolution 20–40 minutes
+- Restart-and-see approach: 2–4 hours of repeated remediation attempts, each buying 15 minutes of relief
+- Retrofitted resolution criteria: re-opening rate ~60% within 24 hours
+- Pre-defined resolution criteria with named window: re-opening rate ~8%

package/skills/bober.graph/SKILL.md

@@ -0,0 +1,85 @@
+---
+name: bober.graph
+description: Manage the code graph index — init, sync, and check status. Requires graph.enabled=true in bober.config.json and tokensave installed.
+handoffs:
+  - label: "Generate Onboarding Docs"
+    command: /bober-onboard
+    prompt: "Generate onboarding docs from the code graph"
+  - label: "Analyse Impact"
+    command: /bober-impact
+    prompt: "Analyse the impact radius of a symbol"
+---
+
+# bober.graph — Code Graph Management Skill
+
+You are running the **bober.graph** skill. Your job is to initialise, sync, or check the status of the code graph for this project.
+
+The code graph is powered by `tokensave`. Once indexed, it enables semantic search, impact analysis, and onboarding document generation.
+
+## Prerequisites
+
+- `tokensave` >= 6.0.0-beta.1 must be installed
+- `graph.enabled: true` must be set in `bober.config.json`
+
+Install tokensave:
+- macOS: `brew install aovestdipaperino/tap/tokensave`
+- Linux: `cargo install tokensave`
+- Windows: `scoop bucket add tokensave https://github.com/aovestdipaperino/scoop-bucket && scoop install tokensave`
+
+## Available Subcommands
+
+### Check prerequisites
+```bash
+agent-bober graph check-prereq
+```
+Reports whether tokensave is installed and compatible. Outputs JSON.
+
+### Initialise the graph
+```bash
+agent-bober graph init
+```
+Runs `tokensave init --tier <languageTier>` and writes an initial manifest to `.bober/graph/manifest.json`. Run this once in a fresh checkout.
+
+### Sync the graph
+```bash
+agent-bober graph sync
+```
+Re-indexes changed files. Use `--force` for a full re-index regardless of changes.
+
+### Check graph status
+```bash
+agent-bober graph status
+```
+Prints whether the graph is ready, how many files are indexed, the tokensave version, and the last synced HEAD SHA.
+
+Use `--json` for machine-readable output:
+```bash
+agent-bober graph status --json
+```
+
+## Step-by-Step: Fresh Setup
+
+1. Verify prerequisites: `agent-bober graph check-prereq`
+2. Enable the graph in `bober.config.json`:
+   ```json
+   { "graph": { "enabled": true } }
+   ```
+3. Initialise: `agent-bober graph init`
+4. Check status: `agent-bober graph status`
+5. After committing new code, sync: `agent-bober graph sync`
+
+## Alternatively: Use MCP Tools Directly
+
+If Claude Code is connected to the agent-bober MCP server with `exposeOnExternalMcp: true`, you can use the MCP tools:
+- `semantic_search_nodes` — semantic search over the graph
+- `query_graph` — trace callers, callees, imports, or test coverage
+- `get_impact_radius` — find what a symbol affects
+- `get_architecture_overview` — high-level structural summary
+- `detect_changes` — risk-scored analysis of recent changes
+- `get_review_context` — token-efficient source snippets for review
+
+## Error Handling
+
+- If graph.enabled=false: the commands exit 1 with a message explaining how to enable it
+- If tokensave is missing: `graph init` exits 2 with a platform-aware install hint
+- If the graph is stale (new commits since last sync): `graph sync` to bring it up to date