@matt82198/aesop 0.1.0-beta.4 → 0.1.0-beta.5

package/monitor/CHARTER.md

@@ -1,137 +1,103 @@
-# Aesop Orchestration Monitor — Charter
+# Orchestration Refinement Monitor — Charter
 
-A standing background monitor (Haiku subagent) that watches how the multi-agent system operates and acts on points of refinement. It improves the *machinery*, never the *mission*.
+A standing background monitor (Haiku) that watches how the multi-agent system operates and
+acts on points of refinement. It improves the *machinery*, never the *mission*.
 
 ## FIXED GOAL (never changes)
+> Ship and maintain your projects correctly, cheaply, and autonomously under the cardinal rules
+> (`~/.claude/CLAUDE.md`). The monitor's job is to make that machine run better — it must NOT
+> redirect, re-scope, or invent new project goals. If it ever thinks the goal should change, it
+> writes a note in `PROPOSALS.md` and stops there.
+
+## What it watches (via `collect-signals.mjs`, refreshed each cycle)
+1. **Junk-script sprawl** — one-off Python/JS scripts Claude Code writes to temp/scratch/cloud and
+   never reuses. Repeated throwaway `.py` to remote agents is a known cost/clutter sink.
+2. **Memory gaps** — projects/decisions/gotchas with no durable memory file; stale or duplicate memories.
+3. **Rule friction** — cardinal-rule violations in recent activity (non-Haiku subagents, main-thread
+   big reads, unpushed green work) and rules that are ambiguous, contradicted, or repeatedly ignored.
+4. **Orchestration health** — hung/stale agents, drift, repeated failures, git state across repos.
 
-> Maintain your projects correctly, cheaply, and autonomously under your cardinal rules.
-> The monitor's job is to make that machine run better — it must NOT redirect, re-scope,
-> or invent new project goals. If it ever thinks the goal should change, it writes a note
-> in `PROPOSALS.md` and stops there.
+## Action tiers
+- **AUTO (apply immediately, then log to `ACTIONS.log`):**
+  - Persist/refresh project memory files (one fact each) + update `MEMORY.md` index.
+  - Quarantine confirmed-dead one-off scripts: move junk temp `.py`/`.mjs` (older than 24h, not
+    referenced by any repo, not the active session's scratchpad) into `orchestration-monitor/quarantine/`
+    with a manifest — never delete outright, never touch a live session scratchpad or repo `src/`.
+  - Clarifying, non-behavioral edits to a memory/log the monitor owns.
+- **PROPOSE (write to `PROPOSALS.md`, do NOT apply — needs user OK):**
+  - Any change to `~/.claude/CLAUDE.md`, `~/.claude/skills/**`, or a project `CLAUDE.md`.
+  - Anything that alters agent behavior, model choice, push targets, or goal scope.
+  - Deleting anything outside the monitor's own quarantine.
 
-## Standing checks (deterministic signal collection)
+## Hard guardrails (inviolable)
+- Preserve the FIXED GOAL. Improve rules; never rewrite intent.
+- Subagents are ALWAYS Haiku (this monitor included). Opus orchestrates on the main thread only.
+- NEVER work around a classifier denial (cross-repo private-source push = exfil; reusing a
+  credential meant for another service = credential exploration). If seen, log + propose, don't act.
+- NEVER print secret values. NEVER push to a remote or run a destructive command as an AUTO action.
+- Idempotent + additive: safe to run repeatedly; prefer append/quarantine over overwrite/delete.
+- Stay cheap: read the signal brief, not raw code/logs. One tight cycle, then sleep.
 
-Each cycle, `collect-signals.mjs` runs these checks and emits `BRIEF.md` + `SIGNALS.json`:
+## End-of-day wipe-survival sweep
 
-1. **Heartbeat liveness check** — read `.heartbeats/*` (epoch on line 1) + legacy beat files
-   (`.monitor-heartbeat`, `.watchdog-heartbeat`); flag any older than threshold (watchdog 300s,
-   monitor 3600s, others 1800s default) as stale loops; append warning to BRIEF.md.
+The monitor may run `python ~/scripts/eod_sweep.py` in its final cycle of the day.
+This verifies all known git repos are safe (no data loss risk):
+- working trees clean OR dirty-file list logged
+- current branch pushed to origin (ahead-count 0)
+- untracked files not silently .gitignore'd
+- SECURITY-ALERTS.log has no unreviewed HIGH/MED items
+- memory INBOX consumed (0 pending items)
+- no heartbeat file claims a dead loop (>1h stale)
 
-2. **Git state across repos** — for each repo in config: branch, last commit, uncommitted files,
-   commits ahead of remote. Identifies unpushed work and dirty state.
+Output: single verdict line (`EOD-SWEEP: SAFE` or `EOD-SWEEP: AT-RISK — <n> findings`) + one finding per line;
+appends to handoff log. Exit code 0 only if SAFE.
 
-3. **Memory freshness** — scan project memory files (excluding index/inbox); flag any >30 days old;
-   append summary to BRIEF.md ("N memories stale — keeper should re-verify").
+If findings detected, the monitor stages an alert: `EOD-SWEEP AT-RISK — <n> findings` (each finding on bullet).
 
-4. **Log rotation** — check append-only logs against max-lines/max-kb thresholds (configurable,
-   default 500 lines / 40 KB). If a log exceeds threshold, emit rotation signal (AUTO tier can
-   invoke rotate_logs.py if available); append rotation summary to BRIEF.md.
+## Standing checks (trimmed for solo operation)
 
-5. **Junk-script sprawl detection** *(extended, opt-in via `monitor.extended_signals`)* — scan temp/scratch roots for throwaway `.py`/.mjs`/`.js` files
-   older than 24 hours, not in a live session directory (avoid false-positives during active work).
-   Count total, estimate quarantinable, list oldest. AUTO tier quarantines confirmed junk into
-   `monitor/quarantine/` with manifest.
+**KEPT (durable value):**
 
-6. **Stray scripts in repo roots** *(extended, opt-in via `monitor.extended_signals`)* — scan recent commits (7d) to detect one-off `.py`/`.mjs`/`.sql`
-   scripts committed directly to repo root (not under proper src/scripts paths). Flag for cleanup.
+1. **Scripts outside `~/scripts`** — flag `.py`/`.mjs`/`.sh` accumulating outside `~/scripts`
+   (scratchpads, temp, repo roots). Quarantine per existing tiers; when one looks genuinely reusable,
+   suggest promotion to `~/scripts` (indexed in `~/scripts/CLAUDE.md`).
 
-7. **Security alert review loop** — tail SECURITY-ALERTS.log for new HIGH/MED entries (skip suppressed);
-   each new alert noted in BRIEF.md. Monitor agent does semantic review (REAL vs FP) each cycle.
+2. **Remote Python execution = HIGH-tier finding** — any scheduled/cloud agent or workflow that would
+   execute Python remotely violates Rule 5 (local-only execution). Flag immediately; never create one.
 
-8. **Respawn watch (Rule 6 retry cap)** *(extended, opt-in via `monitor.extended_signals`)* — parse agent spawn records; normalize descriptions to
-   prompt-signatures (first ~40 chars, lowercased); count occurrences in recent window (last 50 rows).
-   Flag any signature appearing >3 times as probable hung-agent loop → rule 6 cap breached.
-   Limitation: heuristic may have false positives for intentional similar-task fan-outs.
+3. **Memory freshness** — scan `~/.claude/projects/*/memory/*.md` (excluding INBOX.md, MEMORY.md); emit `staleMemories` signal in SIGNALS.json listing any file >30d old; append to BRIEF.md: "N memories >30d — keeper should re-verify: <names>".
 
-9. **Cost cadence tracking** — every 3rd cycle, harvest agent spawn ledger and append cost tick
-   to COST-LOG.md (cycle count, timestamp, model distribution). Flags non-haiku specializations.
+4. **SECURITY ALERT REVIEW LOOP** — on each /power cycle, arm a persistent Monitor tailing
+   SECURITY-ALERTS.log filtered to new 'HIGH'/'MED' lines (exclude SUPPRESSED-FP). Each new alert spawns a read-only fleet-auditor agent to classify REAL vs FALSE-POSITIVE (mask any secret: first 4 chars + ***), verdict appended to SECURITY-ALERTS.log as a REVIEW note and, if REAL, surfaced to the user immediately. Rationale: the append-only alert log is only useful if alerts are triaged; unreviewed HIGHs rot into ignored noise.
 
-10. **Unreviewed agent prompts** *(extended, opt-in via `monitor.extended_signals`)* — run fleet-prompt-extractor.py (if available) to collect NEW
-    spawns since last review; emit count in SIGNALS.json + note in BRIEF.md.
+5. **Fleet ledger harvest** — each cycle, run `python ~/scripts/fleet_ledger.py harvest` to scan session task outputs (JSONL format) and append missing agent outcomes to OUTCOMES-LEDGER.md. Tracks last-harvest state in .fleet-ledger-harvest.json sidecar. Deterministic, local-only, append-only; feeds cost analysis and agent respawn detection. Then invoke `python ~/scripts/fleet_ledger.py rotate` to archive old rows if ledger exceeds 200 lines.
 
-## Extended signals (opt-in)
+6. **Log rotation** — after ledger operations, invoke `python ~/scripts/rotate_logs.py <file> --max-lines 500 --max-kb 40` over: hook-activity.log, SECURITY-ALERTS.log, FLEET-BACKUP.log (BUILDLOG.md at 1000-line threshold — historical record). Emit `logRotations` list in SIGNALS.json and summary line in BRIEF.md.
 
-Checks 5, 6, 8, and 10 are marked as "extended" — they are disabled by default but can be
-enabled via configuration:
+7. **Heartbeat liveness check** — read state/.heartbeats/* (epoch on line 1) + legacy beat files (.monitor-heartbeat, .watchdog-heartbeat, etc.); flag any older than threshold (watchdog 300s, monitor 3600s, others 1800s default) as `staleLoops` signal in SIGNALS.json; append warning to BRIEF.md. Continuous watchdog complement to /power selftest.
 
-- **Config key:** `monitor.extended_signals` (boolean, default: `false`) in aesop.config.json
-- **Env override:** `AESOP_EXTENDED_SIGNALS` (string `'true'` or `'1'` to enable)
-- **Precedence:** env var > config file > default (false)
+8. **Respawn watch (Rule 6 machine-enforcement)** — parse OUTCOMES-LEDGER.md for agent descriptions; normalize each to prompt-signature (first ~40 chars, lowercased, trimmed); count occurrences per signature within recent window (last 50 rows OR since marker in .signal-state.json). Any signature appearing >3 times = probable hung-agent respawn loop → add to `respawnWatch` in SIGNALS.json + warning line in BRIEF.md. *Limitation:* heuristic may have false positives for intentional similar-task fan-outs (e.g., per-file processing with distinct file names in description); BRIEF includes note to manually review flagged signatures. Pure node parsing (no LLM).
 
-When disabled, extended checks emit `{"skipped": true}` in SIGNALS.json and BRIEF.md notes them
-as "extended (off)". AUTO junk-quarantine action only runs when the junk check is enabled.
-PROPOSE-tier signals for extended checks (respawn-watch-breach, stray-repo-scripts) are only
-emitted when extended_signals is ON.
+9. **Semantic prompt review** — each cycle, collect-signals.mjs runs `fleet_prompt_extractor.py` to identify NEW unreviewed agent-spawn prompts (tracks seen-hash state in `.fleet-prompts-seen.json`); count is emitted as `unreviewedPrompts` in SIGNALS.json + noted in BRIEF.md. If count > 0, the monitor agent does semantic review (read each prompt, classify: OK / DRIFT / SUSPICIOUS) and runs `drift_escalator.py` to write verdicts to DRIFT-REVIEW.log + escalate any DRIFT/SUSPICIOUS findings to SECURITY-ALERTS.log + DRIFT-ESCALATIONS.md. This closes the gap: regex fleet-scan is NOT semantic review; LLM judgment gates malformed/off-goal/injection prompts. Completes before any flagged prompt's agent runs.
 
-## Action tiers
+10. **Mirror refresh cadence (throttled)** — each monitor cycle mirrors scripts and docs to your fleet-ops directory and regenerates asset indexes. If either produces diffs: check `.last-mirror-pr-time` in state/. Only open a PR if >1 hour has passed since last PR; otherwise log silently as no-op to ACTIONS.log with timestamp. Ensures mirrors stay in sync and asset lists reflect tier-1 changes, but prevents PR spam. Pure Python automation; runs in monitor context only.
 
-- **AUTO (apply immediately, then log to `ACTIONS.log`):**
-  - Heartbeat freshness checks (read-only).
-  - Log rotation (invoke rotate_logs.py if available; fail-open if not).
-  - Persist heartbeat write (`.monitor-heartbeat` update after cycle).
-  - Quarantine confirmed-junk scripts: move old temp `.py`/`.mjs` (not live session) into
-    `monitor/quarantine/` with manifest — never delete outright (only when extended_signals is ON).
-
-- **PROPOSE (write to `PROPOSALS.md`, do NOT apply — needs user approval):**
-  - Changes to cardinal rules or agent configuration.
-  - Alterations to agent behavior, model choice, or push targets.
-  - Deletions of anything outside monitor's own quarantine.
-  - Any change to orchestration policy.
+11. **UI inbox drain (session-independent work queue)** — each cycle, run `python ~/scripts/inbox_drain.py pending` to surface queued submissions that arrived while no session was running. If items exist, write a one-line `QUEUED WORK PENDING` flag to BRIEF.md so an active monitor also surfaces it. This complements /power's inbox drain (step 5); the monitor ensures work doesn't rot in the queue even between sessions.
 
-## Hard guardrails (inviolable)
+**CUT or DEMOTED (no durable value for solo operator):**
 
-- Preserve the FIXED GOAL. Improve rules; never rewrite intent.
-- **Subagents are ALWAYS Haiku** (cost optimization). Orchestrator on main thread only.
-- NEVER push destructively or to restricted remotes as an AUTO action.
-- NEVER edit cardinal-rules files (CLAUDE.md, skills/, agent configs) — only PROPOSE changes.
-- Idempotent + additive: safe to run repeatedly; prefer append over overwrite.
-- Stay cheap: read the signal brief, not raw logs. One tight cycle, then sleep.
-- Robust to missing files: treat missing dirs/logs as empty, never crash.
+- **Suppression audit** — (CUT) A solo operator watching the dashboard notices suppression count manually; meta-audits add ceremony without signal.
+- **Cost cadence** — (DEMOTED; blocked) Ledger parsing incomplete; once cost meter is proven, restore. For now, flag as blocked in PROPOSALS.md if touched.
+
+**Rationale & changes:**
+
+The original 12-check charter was ceremony-heavy for a one-person system. Cuts preserve the critical signals (liveness, security triage, hung loops, prompt safety, cost data acquisition) and demote/cut only checks whose value is marginal or already visible to a solo operator. Check #10 was throttled instead of cut because the mirror sync is critical, but PR-per-cycle wastes bandwidth — now it only PRs when changes exist AND at least 1 hour has passed.
 
-## Outputs
+Total standing checks: **11** (reflecting genericization; some checks are environment-specific).
 
-- `BRIEF.md` — human-readable status snapshot (overwrite each cycle).
-- `SIGNALS.json` — machine-readable metrics and findings (overwrite each cycle).
+## Outputs (all under orchestration-monitor/)
+- `BRIEF.md` / `SIGNALS.json` — the deterministic snapshot (written by the collector).
 - `ACTIONS.log` — append-only record of AUTO actions taken (timestamped).
-- `PROPOSALS.md` — staged changes awaiting user approval.
-- `quarantine/` — parked junk scripts + `MANIFEST.tsv` (append-only).
-- `.monitor-heartbeat` — epoch timestamp (line 1) for liveness.
-- `.signal-state.json` — sidecar state (cycle count, seen prompts, etc.).
-
-## Operating it
-
-1. **Deploy the monitor** as a continuous background task (or invoke manually each cycle).
-2. **Customize via config**: populate `aesop.config.json` with your repo paths, thresholds.
-3. **Review PROPOSALS.md** periodically; approve changes or reject them.
-4. **Check ACTIONS.log** to see what was automated.
-5. **Monitor BRIEF.md** for drift signals; respond to findings.
-
-## Customization
-
-Edit `collect-signals.mjs` to:
-- Add/remove checks based on your project needs.
-- Configure paths via environment variables (AESOP_ROOT, BRAIN_ROOT, SCRIPTS_ROOT, TEMP_ROOT)
-  or load from aesop.config.json.
-- Set heartbeat thresholds appropriate to your workflow.
-- Integrate custom signal collectors (e.g., compliance checks, custom linters).
-
-See comments in `collect-signals.mjs` for extension points. **Keep it CRLF-safe** (no line
-continuations; maintain Windows+POSIX compatibility).
-
-## Single-instance guard
-
-Before running, check `.monitor-heartbeat` — if <300s old, skip cycle (another monitor is running).
-After cycle completes, update `.monitor-heartbeat` with current epoch. 
-Override with `AESOP_MONITOR_FORCE=true` or `AESOP_MONITOR_FORCE=1` for manual runs or tests (explicit comparison, not truthiness: strings like `'0'` or `'false'` do NOT bypass the guard).
-
-## Single-writer discipline with atomic operations
-
-- Only the monitor edits `BRIEF.md`, `SIGNALS.json`, `ACTIONS.log`, `.monitor-heartbeat`, `.signal-state.json`.
-  - Writes are atomic (temp file + rename) to prevent mid-write corruption if a kill occurs.
-  - Rename is retried on EPERM/EBUSY (Windows readers holding file) with exponential backoff; .tmp is cleaned up on final failure; prior file is preserved (degrade gracefully).
-- `PROPOSALS.md` uses atomic operations (mkdir-style lockfile) to prevent race conditions between concurrent appends (emitProposal) and full rewrites (accept/reject).
-  - **Both emitProposal and moveProposal (accept/reject) acquire the PROPOSALS.md.lock before read-check-append/write**.
-  - Lock contains pid+timestamp for staleness detection (>60s stale locks are reclaimed).
-  - Lock is acquired, held during read-modify-write, released after write completes.
-  - **Fail-closed (P0 wave-8 fix)**: If lock acquisition fails after timeout (default 30s, configurable via AESOP_LOCK_TIMEOUT_MS env var or config), the operation throws an error instead of proceeding. In emitProposal (monitor context), lock failures skip proposal emission for that cycle and retry next cycle. In proposals.mjs CLI (accept/reject), lock failures exit with error code 1 (caller should retry).
-- Quarantine manifest is append-only; never edit entries, only add new ones.
+- `PROPOSALS.md` — staged rule/behavior changes awaiting user approval.
+- `quarantine/` — parked junk scripts + `MANIFEST.tsv`.
+- `.monitor-heartbeat` — liveness for any dashboard.

package/monitor/collect-signals.mjs

@@ -455,6 +455,70 @@ function checkUnreviewedPrompts() {
   return 0;
 }
 
+// 11) Isolation violation detection (rec #5: git-tracked dirs don't flag)
+// Detects worktrees created INSIDE repo root (violation) vs sanctioned ../sibling-wt-* (OK)
+// FP fix: git-tracked source directories (new modules) do NOT flag as violations
+function detectIsolationViolations() {
+  if (!fs.existsSync(AESOP_ROOT)) return { violations: [], count: 0 };
+
+  const violations = [];
+  const trackedFiles = new Set();
+
+  // Get list of git-tracked files to exclude legitimate source dirs
+  try {
+    const tracked = sh('git ls-files', AESOP_ROOT);
+    tracked.split('\n').forEach(f => {
+      if (f) {
+        const dir = f.split('/')[0];
+        trackedFiles.add(dir);
+      }
+    });
+  } catch {
+    // If git ls-files fails, proceed without tracked set (less precise but fail-open)
+  }
+
+  // Walk AESOP_ROOT (shallow) looking for dirs with nested .git
+  try {
+    const entries = fs.readdirSync(AESOP_ROOT, { withFileTypes: true });
+    for (const entry of entries) {
+      if (!entry.isDirectory()) continue;
+      if (entry.name.startsWith('.')) continue;
+      if (entry.name === 'node_modules') continue;
+
+      const dirPath = path.join(AESOP_ROOT, entry.name);
+      const gitPath = path.join(dirPath, '.git');
+
+      // Check if this directory has a .git file or .git dir (worktree indicator)
+      try {
+        const gitStat = fs.statSync(gitPath);
+        // It has .git — check if it's a worktree (contains "gitdir:" or is a symlink)
+        const isWorktree = gitStat.isFile() || gitStat.isSymbolicLink();
+
+        if (isWorktree) {
+          // This looks like a worktree. Check if it's git-tracked (FP fix)
+          const isTracked = trackedFiles.has(entry.name);
+
+          if (!isTracked) {
+            // Untracked worktree inside repo root = VIOLATION
+            violations.push({
+              path: entry.name,
+              type: 'untracked-worktree-in-root',
+              message: `Worktree '${entry.name}' found inside repo root (untracked). Should use sanctioned ../aesop-wt-* sibling instead.`,
+            });
+          }
+          // If tracked, it's a legitimate source directory; don't flag
+        }
+      } catch (e) {
+        // Entry doesn't have .git or is inaccessible; skip
+      }
+    }
+  } catch (e) {
+    // Directory read failed; fail-open with no violations
+  }
+
+  return { violations, count: violations.length };
+}
+
 // === AUTO Actions ===
 // Log rotation: invoke rotate_logs.py if available and log needs rotation
 function performAutoLogRotation(logFiles, actionsLogPath) {
@@ -661,6 +725,7 @@ const staleLoops = checkHeartbeats();
 const gitState = checkGitState();
 const memory = checkMemoryFreshness();
 const logFiles = checkLogFiles();
+const isolationViolations = detectIsolationViolations();
 
 // Extended signal checks (5, 6, 8, 10) — skipped if extended_signals is OFF
 const junk = extendedSignals ? detectJunkScripts() : { skipped: true };
@@ -709,6 +774,7 @@ const signals = {
   respawnWatch,
   costTick,
   unreviewedPrompts,
+  isolationViolations,
 };
 
 const brief = [];
@@ -756,6 +822,17 @@ if (needsRotation.length === 0) {
 }
 brief.push('');
 
+brief.push('## Isolation violations');
+if (isolationViolations.count === 0) {
+  brief.push('✓ No worktrees detected inside repo root (sanctioned siblings OK).');
+} else {
+  brief.push(`🚨 **${isolationViolations.count} isolation violation(s)** detected (worktrees inside repo root):`);
+  for (const v of isolationViolations.violations) {
+    brief.push(`  - ${v.path}: ${v.message}`);
+  }
+}
+brief.push('');
+
 // Extended signals section (if disabled, just note they're off; if enabled, show details)
 if (extendedSignals) {
   brief.push('## Junk-script sprawl (temp/scratch)');
@@ -839,6 +916,14 @@ if (extendedSignals) {
 }
 
 // Core proposals (always emitted)
+if (isolationViolations.count > 0) {
+  emitProposal(
+    'isolation-violation-detected',
+    `${isolationViolations.count} worktree(s) detected inside repo root. Agents should create worktrees as sanctioned siblings (../aesop-wt-*), not inside the repo root to maintain isolation.`,
+    `Review the violation(s): ${isolationViolations.violations.map(v => v.path).join(', ')}. Move worktrees to ../aesop-wt-<name> location outside repo root. Add pre-commit hook or monitoring to prevent future in-root worktrees.`
+  );
+}
+
 if (alerts.highMedCount > 0) {
   emitProposal(
     'security-alerts-high-med',

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@matt82198/aesop",
-  "version": "0.1.0-beta.4",
+  "version": "0.1.0-beta.5",
   "description": "Multi-agent orchestration for AI coding agents, designed to survive failure \u00e2\u20ac\u201d a plain-file brain, git as the only durable layer, cheap subagent fleets, and guardrails enforced in code.",
   "bin": {
     "aesop": "bin/cli.js"
@@ -11,11 +11,14 @@
     "daemons/",
     "dash/",
     "monitor/",
+    "mcp/",
     "tools/",
     "ui/",
     "docs/",
     "hooks/",
     "skills/",
+    "state_store/",
+    "scan/",
     "aesop.config.example.json",
     "CLAUDE-TEMPLATE.md",
     "README.md",