@matt82198/aesop 0.1.0-beta.3 → 0.1.0-beta.4

package/docs/CHECKPOINTING.md

@@ -0,0 +1,100 @@
+# Durable Checkpointing & State Management
+
+**TL;DR**: Maintain STATE.md (intent, phase, NEXT STEPS) + BUILDLOG.md (append-only progress log). On resume after context loss, read both, verify from disk, then proceed. Survives session wipes, enables rapid context handoff, and keeps orchestrator cheap (no re-reading full history).
+
+---
+
+## STATE.md: Authoritative intent & decisions
+
+**Purpose**: Single source of truth for current intent, locked decisions, data model/contracts, key file paths, current phase, and explicit NEXT STEPS.
+
+**Structure**:
+- What we're trying to build (intent, not mechanism)
+- Locked decisions (technology choices, data models, architecture trade-offs)
+- Current phase (e.g., "Phase 2: API integration")
+- Explicit NEXT STEPS (enumerated, actionable, owner assigned)
+- Key paths (file locations, service boundaries, data contracts)
+
+**Discipline**: Hand-update STATE.md at phase boundaries. Never leave it stale; always update BEFORE moving to the next phase. An out-of-date STATE.md is worse than no STATE.md (it's actively misleading).
+
+**Recovery use case**: If context is lost or a session is interrupted, STATE.md is the first file to read. It answers "where were we?" and "what's next?" without re-reading logs or transcripts.
+
+## BUILDLOG.md: Append-only progress snapshots
+
+**Purpose**: Timestamped, immutable record of progress. Never overwrite or edit earlier entries (append-only discipline).
+
+**Pattern**: Each entry contains:
+- Timestamp (ISO 8601 or human-readable)
+- Work completed (features, bugs, integrations, tests)
+- Status (green / yellow / blocked / pending decision)
+- Next step or blocker (for orchestrator reference)
+
+**Example**:
+```
+[2026-07-11 13:00] Phase 1 complete: scaffolding + unit tests passing
+[2026-07-11 14:30] API integration: 2 endpoints done, 1 blocked on schema review
+[2026-07-11 15:45] BLOCKED: waiting for design review on request body format
+[2026-07-11 16:20] Design approved; resume API endpoint 3 implementation
+```
+
+**Append-only rule**: Agents and loops append one line per work unit. The orchestrator never edits earlier entries. This prevents accidental data loss and keeps the log auditable.
+
+## Recovery workflow: On context loss or resume
+
+When you resume work (context wipe, new session, task restart), follow this protocol:
+
+1. **Read STATE.md**: Understand intent, locked decisions, and current phase
+2. **Skim BUILDLOG.md**: Review recent progress and current blockers
+3. **Run git log**: Verify actual file state and recent commits vs. STATE.md
+4. **Sync**: If STATE.md is stale, update it before proceeding; if reality diverged from STATE.md, note the delta and surface it
+
+**Never invent state.** Verify everything from disk (git log, file timestamps, handoff files) before trusting in-memory recollection. Disk is the source of truth.
+
+## Dated archive strategy: Log rotation
+
+When BUILDLOG.md grows beyond **~200 lines or ~20 KB**:
+
+1. Identify the cut point (e.g., last entry from yesterday or the last "green" state marker)
+2. Move older entries to `BUILDLOG-YYYY-MM.md` (dated archive)
+3. Keep only recent entries in the live `BUILDLOG.md`
+4. Commit both files
+5. Update any orchestrator references to point to the live file
+
+**Why rotation?** The orchestrator reads the live BUILDLOG.md frequently; a 50-line log is faster and cheaper than 500 lines. Full history remains available in archives for auditing and debugging.
+
+**Example**:
+- Live file: `BUILDLOG.md` (entries from 2026-07-08 onward)
+- Archive: `BUILDLOG-2026-07.md` (entries through 2026-07-07)
+
+## Handoff metadata across team boundaries
+
+When handing off work to another team member or agent:
+
+- Include the git commit hash (anchors history)
+- Snapshot key file paths and their state
+- Document any external dependencies or waiting states
+- List any environment assumptions (database schema versions, external service versions, etc.)
+
+Example:
+```
+[2026-07-12 09:00] Handing off to feature-team: commit abc1234
+  - Database schema: v3.2 (see db/migrations/)
+  - API contracts frozen in openapi.yaml
+  - Waiting for design review on component layout
+  - Next: implement sign-up form based on design feedback
+```
+
+## Multi-session coordination
+
+For work spanning multiple people or sessions:
+
+1. **Orchestrator updates STATE.md** after each major decision or phase change
+2. **Subagents append to BUILDLOG.md** after each work unit
+3. **On session resume**, orchestrator syncs from disk and briefs the incoming worker
+4. **Hand-updates to STATE.md** happen only at phase boundaries (not every commit)
+
+This keeps overhead low (one STATE.md update per phase, one BUILDLOG line per work unit) while preserving a clear audit trail.
+
+---
+
+**Why STATE.md + BUILDLOG.md?** They ensure work survives context loss, enables rapid resume without re-reading history, and provides an audit trail for debugging. Together with git history and the heartbeat protocol, they form the **checkpointing core** that makes orchestration systems reliable and recoverable.

package/docs/DISPATCH-MODEL.md

@@ -1,5 +1,9 @@
 # Aesop Dispatch Model — Cost & Orchestration Patterns
 
+**TL;DR**: Spawn 5–8 cheap Haiku subagents in parallel (not serial Opus). Result: **lower cost (Haiku ~1/3 Sonnet) plus a parallel speedup** via parallelism. Rule: subagents ALWAYS Haiku unless scoped work genuinely exceeds its capability (rare).
+
+---
+
 ## The fable conceit
 
 In Aesop's Fables, Aesop (the narrator) directs the characters (tortoise, hare, fox, etc.) through moral tales. Here, **Aesop (the orchestrator)** directs a fleet of **Fables (the subagents)** toward reliable task completion.
@@ -21,7 +25,7 @@ The tortoise represents deliberate, resourceful thinking; the hare, quick speed.
 1 orchestrator (Opus, 5 min thinking)  = 5 × 60 = 300 tokens
 5 subagents (Haiku, 2 min each)        = 5 × 2 × 20 = 200 tokens
 Total                                   = 500 tokens/session
-= **83% savings** vs all-Opus
+= a large saving vs all-Opus (Haiku is 1/5 the per-token cost of Opus)
 ```
 
 ### Scaling to 10 parallel domains
@@ -30,14 +34,14 @@ Total                                   = 500 tokens/session
 1 orchestrator (Opus, 8 min)       = 480 tokens
 10 subagents (Haiku, 3 min each)   = 600 tokens
 Total                              = 1,080 tokens
-= **90% savings** vs 10 Opus tasks
+= a large saving vs all-Opus (most domains run on Haiku at 1/5 Opus cost)
 ```
 
 ## MAIN RULE: Subagents are ALWAYS the cheap tier
 
 This is **the single most important cost lever** in the entire system. Every subagent spawned must default to Haiku, never Sonnet or Opus. This rule, more than any other, multiplies your savings at scale.
 
-**Why**: Haiku is 1/3 the cost of Sonnet with sufficient capability for scoped domain work. Scaling from 1 subagent to 6–8 in parallel multiplies the per-domain savings — what would cost 10× Opus now costs <1×. Violating this rule (spawning Sonnet/Opus subagents) erases the economic advantage of the entire dispatch model.
+**Why**: Haiku is 1/3 the cost of Sonnet with sufficient capability for scoped domain work. Scaling from 1 subagent to 6–8 in parallel multiplies the per-domain savings — most of the work runs on Haiku at 1/5 the per-token cost of Opus. Violating this rule (spawning Sonnet/Opus subagents) erases the economic advantage of the entire dispatch model.
 
 **The exception**: Only use Sonnet/Opus for a subagent if:
 1. The task genuinely exceeds Haiku capability (rare for scoped domains), **AND**
@@ -79,7 +83,7 @@ Orchestrator → Phase 1
         └─ Haiku-4: Integration test
         
 Orchestrator reassembles + QA
-Total cost: still 1 Opus + 4 Haiku = **90% savings**
+Total cost: 1 Opus orchestrator + 4 Haiku implementers — the four implementers run at 1/5 the per-token cost of Opus
 ```
 
 ### Pattern 3: Hierarchical (supervisor + workers)
@@ -96,7 +100,7 @@ Orchestrator
 │  └─ Haiku-5: config module
 └─ (Sonnet synthesizes reviews into final report)
 
-Cost: 1 Opus + 1 Sonnet + 5 Haiku = ~$0.015 (still 75% savings)
+Cost: 1 Opus + 1 Sonnet + 5 Haiku = ~1/3 the cost of running every domain on Opus
 ```
 
 ## Decision flowchart
@@ -118,7 +122,7 @@ Is this task scoped to <5 min reasoning?
 
 ### ❌ All-Opus fleet
 
-**Cost**: 10× baseline. **Why**: throws CPU power at problems that are scoped.
+**Cost**: higher — Sonnet/Opus per-token rates. **Why**: throws CPU power at problems that are scoped.
 
 **Fix**: Decompose into Haiku-sized tasks.
 
@@ -204,4 +208,4 @@ Surfacing to the user after 3 retries ensures visibility and prevents silent cos
 
 ---
 
-**Summary**: Aesop's dispatch model trades a small amount of orchestration complexity for **10× cost savings** and **4× speed** (via parallelism). It works by keeping subagents small, orchestrator lean, and feedback constant.
+**Summary**: Aesop's dispatch model trades a small amount of orchestration complexity for **lower cost** (Haiku ~1/3 Sonnet) and **parallel speedup** (via parallelism). It works by keeping subagents small, orchestrator lean, and feedback constant.

package/docs/FORENSICS.md

@@ -0,0 +1,204 @@
+# Forensics: Making Agent Failures Git-Bisectable
+
+## The Pillar
+
+Agent behavior is source code. When an agent misbehaves, you can reconstruct exactly what rules, memory, and policies were active at the time of failure — then bisect the codebase to find the commit that broke it. This turns vague "the agent was weird yesterday" into a reproducible root cause.
+
+**Key principle**: Rules (CARDINAL-RULES.md, CLAUDE.md), checkpoints (STATE.md, BUILDLOG.md), and policies (hooks/, monitor/CHARTER.md) are the agent's executable spec. Versioning them in git means agent failures become debuggable like code regressions.
+
+## Tools
+
+### `tools/agent-forensics.sh`
+
+Read-only incident-forensics tool. Reconstructs agent behavior from git history using only git plumbing (no checkout, no working-tree mutation).
+
+**Usage:**
+
+```bash
+# Snapshot agent behavior at a commit
+bash tools/agent-forensics.sh <commit>
+
+# Show what behavior changed between two commits
+bash tools/agent-forensics.sh --diff <commitA> <commitB>
+```
+
+**Output:**
+- Commit header (hash, date, subject)
+- First ~40 lines of rules (docs/CARDINAL-RULES.md)
+- CLAUDE.md snapshot (if present)
+- STATE.md snapshot (if present)
+- Last 30 lines of BUILDLOG.md (if tracked)
+- For `--diff`: side-by-side changes in CLAUDE.md, STATE.md, docs/, hooks/, monitor/CHARTER.md
+
+**Exit codes:**
+- `0` on success
+- `1` on error (unknown commit, I/O failure); clean error message printed to stderr, never raw stack traces
+
+**Examples:**
+
+```bash
+# Forensics at HEAD
+bash tools/agent-forensics.sh HEAD
+
+# Forensics at a specific commit
+bash tools/agent-forensics.sh 858ac08
+
+# What changed in behavior between two releases?
+bash tools/agent-forensics.sh --diff v0.1.0-beta.2 v0.1.0-beta.3
+```
+
+## Recipe: Bisect an Agent Regression
+
+When an agent's behavior regressed (e.g., "rules started being ignored on July 10"):
+
+1. **Identify the failure time window.** (e.g., "agent started proposing without filtering on 2026-07-10 18:00 UTC")
+
+2. **Run forensics at the last known-good commit** and the first known-bad commit:
+   ```bash
+   bash tools/agent-forensics.sh <good-commit>
+   bash tools/agent-forensics.sh <bad-commit>
+   ```
+   Compare the output manually. Often you'll spot the culprit immediately (e.g., a rule was deleted, CLAUDE.md lost a policy).
+
+3. **If manual inspection doesn't find it, bisect:**
+   ```bash
+   git bisect start
+   git bisect bad <bad-commit>
+   git bisect good <good-commit>
+   ```
+
+4. **Write a behavior predicate** (a test that reproduces the failure):
+   ```bash
+   # Example: check that docs/CARDINAL-RULES.md mentions "Haiku" at this commit
+   git bisect run bash -c '
+     git show HEAD:docs/CARDINAL-RULES.md | grep -q "Haiku" && exit 0 || exit 1
+   '
+   ```
+   Bisect will pin-point the exact commit that broke the behavior.
+
+5. **Review the offending commit** using `--diff`:
+   ```bash
+   bash tools/agent-forensics.sh --diff <parent-of-bad> <bad-commit>
+   ```
+
+6. **Fix the issue** (restore the rule, reinstate the policy, etc.) in a new commit on a feature branch.
+
+## What to Check When an Incident Happens
+
+### Incident Discovery
+1. Capture the **current behavior** (what went wrong, when, which agent):
+   - Run `bash tools/agent-forensics.sh HEAD` to see rules + state at time of report.
+   - Check logs in `state/FLEET-BACKUP.log`, `state/ACTIONS.log`, or agent transcript.
+   - Note the **exact timestamp** of the first wrong behavior.
+
+2. **Cross-reference the timestamp** with `git log --since="<time>"` to find commits near that time.
+
+### Root Cause Narrowing
+3. **Did a rule change?**
+   ```bash
+   git log -p --since="<before-incident>" -- docs/CARDINAL-RULES.md CLAUDE.md | head -100
+   ```
+   Look for deletions, edits to retry logic, dispatch budgets, or policy changes.
+
+4. **Did state get corrupted?**
+   ```bash
+   bash tools/agent-forensics.sh <commit-before-incident>
+   bash tools/agent-forensics.sh <commit-at-incident>
+   # Compare STATE.md and BUILDLOG.md for anomalies (e.g., PHASE stuck, cost spike)
+   ```
+
+5. **Did a hook fail silently?**
+   ```bash
+   git log -p --since="<before-incident>" -- hooks/ monitor/CHARTER.md
+   cat state/SECURITY-AUDIT.log | tail -50
+   ```
+
+### Confirmation via Bisect
+6. Once you have a suspect commit or time range, use the bisect recipe above to narrow it down.
+
+7. **Write the fix** in a feature branch. Update CLAUDE.md if the rule itself was wrong; add an explanatory comment referencing the incident date.
+
+8. **Add a regression test:** if the incident was "agent ignored retry cap," add a check in pre-push-policy.sh or a future test-harness to catch that rule drifting again.
+
+## Durable Artifacts to Check
+
+| File | Purpose | Checked via |
+|------|---------|-------------|
+| **docs/CARDINAL-RULES.md** | Dispatch, cost, retry, TDD policy | `bash tools/agent-forensics.sh <commit>` |
+| **CLAUDE.md** | Project-level rules, branch discipline, commit gates | manual forensics; git log |
+| **STATE.md** | Current phase, locked decisions, next steps | forensics snapshot |
+| **BUILDLOG.md** | Append-only cycle log (cost, actions, phase changes) | forensics (last 5 entries) |
+| **hooks/** | Pre-push policy, secret-scan gate, lint checks | git show or forensics --diff |
+| **monitor/CHARTER.md** | Orchestration monitor config (AUTO/PROPOSE thresholds) | forensics --diff |
+
+## Example: Debugging a Cost Spike
+
+**Symptom**: Agent token spend jumped from 50k to 200k tokens on 2026-07-10.
+
+1. Run forensics at the spike time:
+   ```bash
+   bash tools/agent-forensics.sh <commit-on-2026-07-10>
+   ```
+   Check STATE.md and BUILDLOG.md. Look for:
+   - Did `parallelism` increase? (more agents spawned → more tokens)
+   - Did `retries` increase? (retry cap lowered → more respawns)
+   - Did `model` change to Opus? (rule 1 violation)
+
+2. Narrow the window:
+   ```bash
+   git log --oneline --since="2026-07-09" --until="2026-07-11"
+   bash tools/agent-forensics.sh --diff <2026-07-09-commit> <2026-07-10-spike-commit>
+   ```
+
+3. If a rule changed (e.g., retries bumped from 3 to 6), fix it:
+   ```bash
+   git log -p --since="2026-07-09" -- docs/CARDINAL-RULES.md | grep -A5 -B5 retries
+   ```
+
+4. Restore or correct the rule, commit, and monitor spend in the next cycle.
+
+## Integration with the Refinement Loop
+
+Forensic replay strengthens the orchestration refinement cycle by enabling rapid root-cause analysis when the fleet exhibits unexpected behavior changes.
+
+**When to invoke forensics in refinement:**
+
+1. **Suspected behavior regression** — If PROPOSALS.md or monitoring signals suggest rules changed unexpectedly:
+   ```bash
+   # Find the suspect commit range
+   git log --oneline --all | head -20
+   
+   # Compare behavior-controlling files between two commits
+   bash tools/agent-forensics.sh --diff <last-known-good> <suspect-bad>
+   ```
+   This immediately shows what changed in CLAUDE.md, STATE.md, docs/, hooks/, and monitor/CHARTER.md.
+
+2. **Cost or performance spike** — If SIGNALS.json reports elevated token spend or cost anomalies:
+   ```bash
+   # Snapshot state at the anomaly time
+   bash tools/agent-forensics.sh <commit-at-spike-time>
+   ```
+   Check the BUILDLOG.md excerpt for cost-related entries, parallelism changes, or dispatch budget adjustments.
+
+3. **After merging a PR** — Before rolling to production, verify no unintended behavior changes were introduced:
+   ```bash
+   # Compare main...feature-branch
+   bash tools/agent-forensics.sh --diff main <feature-branch-head>
+   ```
+   If behavior-controlling files show unexpected changes, halt the merge and investigate.
+
+**Output interpretation:**
+
+- **CLAUDE.md or STATE.md changed** → review what rules or phase-state was altered
+- **docs/CARDINAL-RULES.md modified** → check if dispatch budgets, retry caps, or cost levers were adjusted
+- **hooks/ or monitor/CHARTER.md changed** → verify policy enforcement or signal thresholds are intentional
+- **No changes in behavior-controlling files** → regression is likely in agent logic (non-behavior files) or external state
+
+**Integration point:** The monitor/collect-signals.mjs can emit a PROPOSAL when it detects rule drift or state anomalies, prompting operators to run forensics on the linked commits to verify the change was intentional.
+
+## See Also
+
+- **docs/CARDINAL-RULES.md** — The executable spec for agent behavior.
+- **docs/DISPATCH-MODEL.md** — Cost levers, parallelism, watchdog stall detection.
+- **docs/GOVERNANCE.md** — Memory/state single-writer guarantees, proposal flow.
+- **README.md** — Quick start; includes examples of running forensics on a live incident.

package/docs/GOVERNANCE.md

@@ -1,6 +1,8 @@
 # Governance & Control
 
-This guide covers the operational patterns that keep orchestration systems coherent, scalable, and auditable at production scale: single-instance loops, single-writer files, asynchronous inbox coordination, log rotation, branching discipline, and security gates.
+**TL;DR**: Keep the system coherent: one instance of each loop (heartbeat protocol), single writer per control file (MEMORY.md, STATE.md), append-only logs, secret-scan gate on every push, feature branches only (never direct-to-main). Prevents races, data loss, credential leaks, and makes work auditable.
+
+---
 
 ## Single-instance loops
 
@@ -41,7 +43,7 @@ Some files must have exactly one writer to prevent race conditions and data loss
 
 ## Inbox pattern for coordination
 
-Loops that need to request memory updates or signal decisions use an append-only inbox. The orchestrator reviews inbox entries on every `/power` cycle.
+Loops that need to request memory updates or signal decisions use an append-only inbox. The orchestrator reviews inbox entries on every `/power` cycle. See [skills/power/SKILL.md](../skills/power/SKILL.md) for orchestrator priming details.
 
 ### Inbox format