@matt82198/aesop 0.1.0-beta.4 → 0.1.0-rc.1

package/docs/reproduce.md

@@ -0,0 +1,121 @@
+# Reproducing Aesop
+
+This document explains how to reproduce the project's claims from a clean checkout.
+
+## Overview
+
+The Aesop project includes:
+1. **Full test suites** (Node.js + Python) that verify core functionality
+2. **An offline benchmark scorer** that proves the benchmark scoring logic works correctly
+
+Both can be reproduced entirely offline, without API keys or external dependencies, from any clean clone of the repository.
+
+## Reproducing from a Clean Clone
+
+### Prerequisites
+
+- **Node.js**: 18 or later
+- **Python**: 3.12 or later
+- **Git**: any recent version
+
+### Steps
+
+1. Clone the repository (no special flags needed):
+   ```bash
+   git clone https://github.com/matt82198/aesop.git
+   cd aesop
+   ```
+
+2. Verify Node.js and Python are installed:
+   ```bash
+   node --version
+   python --version
+   ```
+
+3. Install Node.js dependencies:
+   ```bash
+   npm ci
+   ```
+
+4. Run the full test suite (Python + Node.js):
+   ```bash
+   # Run Node.js tests
+   npm run test:node
+   
+   # Run Python tests
+   python -m unittest discover -s tests
+   ```
+
+5. Run the benchmark scorer offline:
+   ```bash
+   # Run the benchmark unit tests (syntax, ground truth format, etc.)
+   python -m unittest tests.test_bench_runner -v
+   
+   # Run the offline mock benchmark
+   python tools/bench_runner.py --runner mock
+   ```
+
+## What the `.github/workflows/reproduce.yml` Job Does
+
+The `reproduce` workflow in GitHub Actions automates the above steps:
+
+1. **Fresh checkout**: Uses a clean clone (no state reuse from other CI jobs)
+2. **Syntax checks**: Validates all Node.js (.mjs) and shell scripts (.sh)
+3. **Node.js tests**: Runs the full test suite with `npm run test:node`
+4. **Python tests**: Runs the full test suite with `python -m unittest discover`
+5. **Benchmark scorer**: Proves the offline mock benchmark reproduces (no external API calls)
+
+The workflow is triggered:
+- **Manually**: Via GitHub Actions "Run workflow" button (workflow_dispatch)
+- **Weekly**: Every Sunday at 2:00 AM UTC (schedule)
+
+## What Is and Isn't Reproduced
+
+### Reproduced (offline, no API keys needed)
+
+- ✓ All committed test suites (correctness, integration, governance)
+- ✓ The benchmark **scorer logic** (matching, scoring, reporting)
+- ✓ The benchmark **mock runner** (deterministic offline accuracy score)
+- ✓ The committed benchmark files (`bench/tasks.jsonl`, `bench/ground_truth.jsonl`)
+
+### Not Reproduced (requires external APIs)
+
+- ✗ A real multi-model benchmark (calling Claude API, OpenAI, etc.)
+- ✗ The dashboard browser tests (Playwright, requires UI setup)
+- ✗ Performance/timing benchmarks
+- ✗ The full CI job's additional linting and drift checks
+
+The benchmark mock runner always returns the same accuracy score because it uses a hardcoded heuristic (plain string/regex matching), not a real model. This is intentional — the tests prove the *scorer* works correctly, not that any model achieves a specific accuracy.
+
+## Interpreting Results
+
+### Success
+
+```
+YAML is valid
+All Node.js tests pass
+All Python tests pass
+Benchmark reproduction successful
+```
+
+### Failure
+
+If any step fails, the workflow exits with a non-zero exit code and prints the failure details. Common issues:
+
+1. **Node.js test failure**: Check `tests/*.test.mjs` for the specific failing test.
+2. **Python test failure**: Check the error message; most often a missing test fixture or a test isolation bug.
+3. **Benchmark reproduction failure**: Run `python tools/bench_runner.py --runner mock` manually to see the exact error.
+
+## For Project Maintainers
+
+When you add new features or tests:
+
+1. Ensure all tests pass locally: `npm run test:all` (or individually per suite)
+2. The `reproduce` workflow will catch any regressions automatically
+3. If reproduction fails, it indicates the committed test fixtures or scorer are inconsistent
+
+## References
+
+- Test discovery: `python -m unittest discover --help`
+- Benchmark scorer: `python tools/bench_runner.py --help`
+- GitHub Actions workflow syntax: https://docs.github.com/en/actions

package/docs/self-stats-data.json

@@ -0,0 +1,11 @@
+{
+  "_source": "Populated by orchestrator at wave close via fleet telemetry aggregation",
+  "_updated": "Awaiting orchestrator population",
+  "total_sessions": null,
+  "total_turns": null,
+  "total_user_prompts": null,
+  "max_tokens_single_turn": null,
+  "cumulative_agent_runs": null,
+  "cumulative_tokens": null,
+  "total_coding_hours": null
+}

package/docs/templates/FLEET-OPS-ANALYSIS.example.md

@@ -0,0 +1,27 @@
+# Fleet-Ops Analysis Summary — Cycle N
+
+**Generated:** YYYY-MM-DDTHH:MM:SSZ
+
+## Top Failure Modes (cumulative this session)
+
+| Category | Count | Severity | Notes |
+|----------|-------|----------|-------|
+| [FAILURE_MODE_NAME] | 0 | [HIGH/MED/LOW] | Brief description of failure pattern, root cause hypothesis, and reference to any related recommendations or PRs. |
+| | | | |
+| | | | |
+
+## Token Spend Summary
+
+Brief summary of session token spend patterns:
+- [Agent/component type]: ~XXXk tokens — brief observation
+- Heavy [category]: XXXk — specific tasks or drivers
+- Typical [operation type]: XXk-XXXk; [context or pattern]
+
+Include observations about:
+- Which domains consumed the most tokens
+- Whether spend is aligned with task scope
+- Any unexpected patterns or efficiency issues
+
+## Notes
+
+(Optional section for additional context, anomalies, or deferred investigations)

package/docs/templates/FLEET-OPS-RECOMMENDATIONS.example.md

@@ -0,0 +1,24 @@
+# Fleet-Ops Recommendations — machinery fixes for wave open
+
+Rolling, append-only list maintained by the orchestration/fleet-ops monitor.
+The buildsystem skill reads this at every wave open (Phase 0.3). Mark items
+ADOPTED/REJECTED/IN-PROGRESS with wave number when actioned; do not delete rows.
+
+| # | Date | Source | Severity | Recommendation | Status |
+|---|------|--------|----------|----------------|--------|
+| 1 | YYYY-MM-DD | [source/agent/session phase] | [HIGH/MED/LOW] | Clear, actionable recommendation describing the problem, suggested fix, and rationale. Include specific file paths or component names where applicable. | OPEN |
+| 2 | YYYY-MM-DD | [source/agent/session phase] | [HIGH/MED/LOW] | Second example recommendation with similar detail level. Link to related findings if applicable. | OPEN |
+| | | | | | |
+
+## How to Use This Template
+
+1. **Record findings at wave close:** The orchestration monitor appends new rows for every machinery issue or systemic improvement identified.
+2. **Mark status transitions:** When a recommendation is adopted (wave-N or PR reference), update Status to `ADOPTED wave-N`. For rejected items, mark `REJECTED wave-N` with brief rationale.
+3. **Track implementation:** Use Status column to indicate OPEN (unactioned), IN-PROGRESS (wave-X, assigned to agent/owner), ADOPTED (implemented, wave-Y), or REJECTED (not pursued, wave-Y).
+4. **Preserve history:** Never delete rows — the log is append-only. This creates an auditable trail for machinery decisions.
+
+## Notes
+
+- **Source field:** Where did this insight come from? (session coordinator, CI failure, agent forensics, monitor signal, PR review, etc.)
+- **Severity:** HIGH = blocks reliable operation or impacts multiple waves; MED = worth addressing in near term; LOW = nice-to-have optimization.
+- **Implementation scope:** Recommendations should be specific enough for a developer to act on, but can span multiple waves if complex.

package/docs/templates/PROPOSALS-LOG.example.md

@@ -0,0 +1,64 @@
+# Orchestration Refinement Proposals — Action Log
+
+Actioned proposals moved from PROPOSALS.md (grounded in signals from YYYY-MM-DD). Per standing order, move decided entries here with date and outcome.
+
+---
+
+## ACCEPTED-IMPLEMENTED — YYYY-MM-DD
+
+### 1. [Proposal Title]
+
+**Outcome:** [Brief description of what was implemented, where code/docs changed, wave number if applicable, and verification of completion.]
+
+---
+
+## IN-PROGRESS — YYYY-MM-DD
+
+### 1. [Proposal Title]
+
+**Status:** IN-PROGRESS (wave-N, [assigned-agent or owner])  
+**Rationale:** [Why this proposal was accepted and what timeline/dependencies exist.]
+
+---
+
+## DEFERRED — YYYY-MM-DD
+
+### 1. [Proposal Title]
+
+**Status:** DEFERRED ([reason: blocked-by/low-priority/waiting-for-data])  
+**Rationale:** [Why this proposal was deferred, under what conditions it might be reconsidered.]
+
+---
+
+## REJECTED — YYYY-MM-DD
+
+### 1. [Proposal Title]
+
+**Status:** REJECTED (wave-N, [reason])  
+**Rationale:** [Explanation of rejection, any alternatives considered, and future implications.]
+
+---
+
+## No further action required (MONITORING ONGOING)
+
+### 1. [Proposal Title]
+
+**Status:** MONITORING  
+[Brief note on what is being tracked and how the team will know if action becomes necessary.]
+
+---
+
+## How to Use This Template
+
+1. **Capture proposals:** When orchestration/monitor/coordinator identifies a potential process or machinery improvement, record it in a staging PROPOSALS.md file with date, source, and rationale.
+2. **Move to log on decision:** Once a proposal is decided (ACCEPTED/REJECTED/DEFERRED), move the entry here with date and outcome.
+3. **Track implementation:** Use status sections (ACCEPTED-IMPLEMENTED, IN-PROGRESS, DEFERRED, REJECTED) to organize proposals by lifecycle stage.
+4. **Preserve history:** Keep all entries, do not delete. This creates an auditable trail of organizational decisions and learning.
+5. **Link to execution:** When implemented, reference the wave number or PR. When in-progress, name the owner.
+
+## Key Fields
+
+- **Proposal Title:** Concise statement of the proposed change or investigation.
+- **Outcome/Status:** What happened as a result (implemented, rejected, deferred, monitoring).
+- **Rationale:** Why this decision was made, dependencies, or conditions for change.
+- **Wave/PR/Agent:** Link to the work that carried out the decision (if applicable).

package/hooks/CLAUDE.md

@@ -30,6 +30,57 @@ Runs on `git push` via `.git/hooks/pre-push` symlink or copy.
 - See `docs/HOOK-INSTALL.md` for symlink (Linux/macOS/Git Bash) and copy (Windows) methods
 - Test with `bash hooks/pre-push-policy.sh --test` before org distribution
 
+## Hook: pre-commit-waveguard.sh
+
+Prevents accidental commits to the PRIMARY aesop tree during a wave cycle. Runs on `git commit` via `.git/hooks/pre-commit`.
+
+**Purpose**: During orchestrated waves, the orchestrator sets a marker file (`state/.wave-in-flight`) in the PRIMARY tree only. Sibling worktrees do not inherit this marker (separate working trees), so fleet agents commit freely in worktrees while stray commits to the primary tree are rejected.
+
+**Mechanism**:
+1. **Marker Contract**: Orchestrator writes `state/.wave-in-flight` in the PRIMARY tree before dispatching wave work. The marker is git-ignored, so sibling worktrees checked out during the wave do NOT carry it.
+2. **Pre-Commit Check**: Hook resolves the marker relative to the CURRENT working tree via `git rev-parse --show-toplevel` (NOT a hardcoded `$HOME/aesop` — that resolved to the primary tree from every worktree and blocked the whole fleet mid-wave, the wave-24 incident). Primary tree (has marker during a wave) → exit 1 (reject); sibling worktree (no marker) → exit 0 (allow).
+3. **Override**: User or orchestrator may delete `state/.wave-in-flight` to manually allow commits to primary tree.
+
+**Exit Contract**:
+- Exit 0: Marker absent, commit allowed (normal operation)
+- Exit 1: Marker present, commit blocked with clear error message
+
+**Error Message**:
+```
+Error: Wave in flight. Commit from a sibling worktree, or clear <marker_path> to override.
+```
+
+**Installation**:
+- Run `bash hooks/install-waveguard.sh` to idempotently install into `.git/hooks/pre-commit`.
+- If a pre-commit hook already exists, installer backs it up (`.git/hooks/pre-commit.waveguard-backup`) and wraps both (waveguard first, then existing hook if present).
+
+**Idempotency**:
+- Installer checks if hook already calls waveguard; skips if already installed.
+- Safe to re-run multiple times.
+
+## Hook: hooks/claude/force-model-policy.mjs
+
+Claude Code hook enforcing subagent Haiku dispatch (cost optimization).
+
+**Trigger**: On skill invocation or task delegation from main orchestrator thread. Examines Claude API request and enforces model constraint.
+
+**Policy**:
+- **Main orchestrator** (Fable/Opus on primary): no override (uses native model)
+- **Subagent dispatch** (fleet workers): **enforce Haiku** (claude-haiku-4-5-*). Exit 1 on non-Haiku model request.
+- **Specialists** (typed dispatches): Pin model to Haiku in the dispatch call; hook validates + blocks if violated.
+
+**Logging**:
+- On policy violation: log to `state/SECURITY-ALERTS.log` with timestamp, model-name, worker-id, and reason.
+- No alerts on compliant requests.
+
+**Self-Test**:
+- `node hooks/claude/force-model-policy.mjs --test` validates:
+  1. Haiku model allowed on subagents
+  2. Non-Haiku (e.g., Opus) blocked on subagents
+  3. Orchestrator not subject to policy
+  4. JSON logging format is valid
+- Exit 0 = all pass; exit 1 = any fail
+
 ## Invariants
 
 - POSIX sh compatible, CRLF-safe (no line continuations)

package/hooks/install-waveguard.sh

@@ -0,0 +1,68 @@
+#!/usr/bin/env bash
+set -uo pipefail
+
+main() {
+  local repo_root
+  repo_root=$(git rev-parse --show-toplevel 2>/dev/null)
+  if [ -z "$repo_root" ]; then
+    printf 'Error: Not in a git repository\n' >&2
+    return 1
+  fi
+
+  local hooks_dir="$repo_root/.git/hooks"
+  local waveguard_src="$repo_root/hooks/pre-commit-waveguard.sh"
+  local pre_commit_dest="$hooks_dir/pre-commit"
+  local pre_commit_orig="$hooks_dir/pre-commit.waveguard-backup"
+
+  if [ ! -f "$waveguard_src" ]; then
+    printf 'Error: Source hook not found at %s\n' "$waveguard_src" >&2
+    return 1
+  fi
+
+  mkdir -p "$hooks_dir"
+
+  if [ -f "$pre_commit_dest" ]; then
+    if grep -q 'pre-commit-waveguard' "$pre_commit_dest" 2>/dev/null; then
+      printf 'Info: pre-commit hook already has waveguard; skipping.\n'
+      return 0
+    fi
+
+    cp "$pre_commit_dest" "$pre_commit_orig"
+    printf 'Info: Existing pre-commit hook backed up to %s\n' "$pre_commit_orig"
+  fi
+
+  cat > "$pre_commit_dest" <<'HOOK_WRAPPER'
+#!/usr/bin/env bash
+set -uo pipefail
+
+repo_root=$(git rev-parse --show-toplevel 2>/dev/null || echo "")
+if [ -z "$repo_root" ]; then
+  exit 1
+fi
+
+waveguard_hook="$repo_root/hooks/pre-commit-waveguard.sh"
+if [ -f "$waveguard_hook" ]; then
+  bash "$waveguard_hook"
+  waveguard_exit=$?
+  if [ $waveguard_exit -ne 0 ]; then
+    exit $waveguard_exit
+  fi
+fi
+
+backup_hook="$repo_root/.git/hooks/pre-commit.waveguard-backup"
+if [ -f "$backup_hook" ] && [ -x "$backup_hook" ]; then
+  bash "$backup_hook"
+  exit $?
+fi
+
+exit 0
+HOOK_WRAPPER
+
+  chmod +x "$pre_commit_dest"
+  printf 'Info: Installed waveguard pre-commit hook to %s\n' "$pre_commit_dest"
+  return 0
+}
+
+if [ "${BASH_SOURCE[0]}" = "${0}" ]; then
+  main "$@"
+fi

package/hooks/pre-commit-waveguard.sh

@@ -0,0 +1,26 @@
+#!/usr/bin/env bash
+set -uo pipefail
+
+main() {
+  # Resolve the marker relative to the CURRENT working tree, NOT a hardcoded primary path.
+  # This is load-bearing: the marker (state/.wave-in-flight) is git-ignored, so a sibling
+  # worktree checked out during a wave does NOT carry it — only the PRIMARY tree (where the
+  # orchestrator sets it) does. Checking the current tree's own toplevel means primary-tree
+  # commits are blocked during a wave while legitimate worktree-agent commits pass. A prior
+  # version hardcoded ${AESOP_ROOT:-$HOME/aesop}, which resolved to the primary tree from every
+  # worktree and thus blocked the entire fleet mid-wave (wave-24 incident). Do not reintroduce.
+  local toplevel
+  toplevel=$(git rev-parse --show-toplevel 2>/dev/null) || exit 0
+  local marker_file="$toplevel/state/.wave-in-flight"
+
+  if [ -f "$marker_file" ]; then
+    printf 'Error: Wave in flight in this tree (%s). Commit from a sibling worktree, or clear the marker to override.\n' "$marker_file" >&2
+    exit 1
+  fi
+
+  exit 0
+}
+
+if [ "${BASH_SOURCE[0]}" = "${0}" ]; then
+  main "$@"
+fi