@matt82198/aesop 0.2.0 → 0.3.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +28 -0
- package/README.md +50 -260
- package/bin/cli.js +7 -3
- package/docs/HOOK-INSTALL.md +15 -56
- package/docs/INSTALL.md +4 -3
- package/docs/PORTING.md +166 -0
- package/docs/TEAM-STATE.md +16 -184
- package/docs/reproduce.md +33 -3
- package/driver/CLAUDE.md +50 -48
- package/driver/codex_driver.py +59 -12
- package/driver/openai_transport.py +33 -0
- package/driver/wave_bridge.py +9 -1
- package/driver/wave_loop.py +437 -70
- package/driver/wave_scheduler.py +890 -0
- package/hooks/pre-push-policy.sh +22 -5
- package/monitor/collect-signals.mjs +69 -0
- package/package.json +4 -4
- package/state_store/__init__.py +2 -1
- package/state_store/projections.py +63 -0
- package/state_store/read_api.py +156 -0
- package/state_store/write_api.py +462 -0
- package/tools/cost_ceiling.py +106 -15
- package/tools/cost_projection.py +559 -0
- package/tools/crossos_drift.py +394 -0
- package/tools/eod_sweep.py +137 -33
- package/tools/mutation_test.py +130 -8
- package/tools/proposals.mjs +47 -2
- package/tools/reproduce.js +405 -0
- package/tools/secret_scan.py +73 -18
- package/tools/stall_check.py +247 -16
- package/tools/stateapi_lint.py +325 -0
- package/tools/test_battery.py +173 -0
- package/tools/verify_cost_panel.py +345 -0
- package/tools/wave_preflight.py +296 -29
- package/tools/wave_templates.py +170 -45
- package/ui/collectors.py +81 -0
- package/ui/handler.py +230 -85
- package/ui/sse.py +3 -3
- package/ui/wave_telemetry.py +24 -18
- package/ui/web/dist/assets/{index-DqZLgwNg.css → index-CNQxaiOW.css} +1 -1
- package/ui/web/dist/assets/index-CP68RIh3.js +9 -0
- package/ui/web/dist/index.html +2 -2
- package/docs/QUICKSTART.md +0 -80
- package/ui/web/dist/assets/index-BGbgw2Nh.js +0 -9
package/docs/PORTING.md
ADDED
|
@@ -0,0 +1,166 @@
|
|
|
1
|
+
# Porting Aesop to Your Repository
|
|
2
|
+
|
|
3
|
+
Guide for adopters porting the orchestration harness to a foreign repo. Step-by-step with prerequisites, scaffold, config, and the 10 likeliest failure modes from real deployments.
|
|
4
|
+
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
## Prerequisites
|
|
8
|
+
|
|
9
|
+
Ensure you have:
|
|
10
|
+
- **Node.js** ≥18 (from `package.json`, `.nvmrc`: v18.20.5)
|
|
11
|
+
- **Python** ≥3.10 (for daemons, health checks, secret-scan)
|
|
12
|
+
- **Git** ≥2.40 (worktree + pre-push hooks)
|
|
13
|
+
- **Bash** v4+ (or Git Bash on Windows)
|
|
14
|
+
- **Optional**: Playwright (for UI verification testing); jq (for dashboard JSON)
|
|
15
|
+
|
|
16
|
+
Check versions:
|
|
17
|
+
```bash
|
|
18
|
+
node --version && python3 --version && git --version && bash --version
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
### OS Notes
|
|
22
|
+
- **Windows**: Use Git Bash for all shell commands; paths use `/c/Users/...` POSIX style
|
|
23
|
+
- **Linux/macOS**: Standard Bash; ensure `/usr/bin/bash` or equivalent exists
|
|
24
|
+
|
|
25
|
+
---
|
|
26
|
+
|
|
27
|
+
## 1. Scaffold & Install
|
|
28
|
+
|
|
29
|
+
### Step 1: Create your harness
|
|
30
|
+
```bash
|
|
31
|
+
npx @matt82198/aesop my-fleet \
|
|
32
|
+
--name "my-project" \
|
|
33
|
+
--repos "/path/to/repo1,/path/to/repo2"
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
Creates `my-fleet/` with daemons, skills, config, and UI.
|
|
37
|
+
|
|
38
|
+
### Step 2: Copy skills to Claude Code home
|
|
39
|
+
```bash
|
|
40
|
+
cp -r my-fleet/skills/power ~/.claude/skills/power
|
|
41
|
+
cp -r my-fleet/skills/buildsystem ~/.claude/skills/buildsystem
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
### Step 3: Configure & test
|
|
45
|
+
Edit `my-fleet/aesop.config.json`:
|
|
46
|
+
```json
|
|
47
|
+
{
|
|
48
|
+
"backend": "claude",
|
|
49
|
+
"aesop_root": "/path/to/my-fleet",
|
|
50
|
+
"brain_root": "~/.claude",
|
|
51
|
+
"repos": [
|
|
52
|
+
{ "path": "/path/to/repo1", "name": "my-project" }
|
|
53
|
+
],
|
|
54
|
+
"dashboardPort": 8770
|
|
55
|
+
}
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
Test dashboard backend (Ollama example):
|
|
59
|
+
```json
|
|
60
|
+
{
|
|
61
|
+
"backend": "openai-compatible",
|
|
62
|
+
"base_url": "http://localhost:11434/v1",
|
|
63
|
+
"model": "mistral",
|
|
64
|
+
"is_local": true
|
|
65
|
+
}
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
### Step 4: Install pre-push hook
|
|
69
|
+
```bash
|
|
70
|
+
mkdir -p my-fleet/.git/hooks
|
|
71
|
+
cp my-fleet/hooks/pre-push-policy.sh my-fleet/.git/hooks/pre-push
|
|
72
|
+
chmod +x my-fleet/.git/hooks/pre-push
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
### Step 5: Verify with health check
|
|
76
|
+
```bash
|
|
77
|
+
cd my-fleet
|
|
78
|
+
python tools/health_score.py
|
|
79
|
+
```
|
|
80
|
+
Expected output: All checks ✓; port 8770 available; git hooks installed.
|
|
81
|
+
|
|
82
|
+
---
|
|
83
|
+
|
|
84
|
+
## 2. First `/power`, First Wave
|
|
85
|
+
|
|
86
|
+
Run the orchestrator once to prime the brain:
|
|
87
|
+
```bash
|
|
88
|
+
cd my-fleet
|
|
89
|
+
/power
|
|
90
|
+
```
|
|
91
|
+
Expected output: State created (STATE.md, BUILDLOG.md, .watchdog-heartbeat).
|
|
92
|
+
|
|
93
|
+
Run one complete wave:
|
|
94
|
+
```bash
|
|
95
|
+
/buildsystem --one-turn
|
|
96
|
+
```
|
|
97
|
+
Expected verdicts: All items "PASS" or "REVIEW"; zero code defects. Health score should report ≥85/100.
|
|
98
|
+
|
|
99
|
+
---
|
|
100
|
+
|
|
101
|
+
## 3. The 10 Likeliest Failure Modes
|
|
102
|
+
|
|
103
|
+
Each: symptom → cause → fix.
|
|
104
|
+
|
|
105
|
+
### 1. Secret-scan blocks legit push
|
|
106
|
+
- **Symptom**: `git push` fails: "secret detected" but code has no real secret
|
|
107
|
+
- **Cause**: Test fixtures with dummy secrets (credit card, API key patterns) are parsed as real
|
|
108
|
+
- **Fix**: Assemble secrets at runtime: `"key" + "_" + "123"` not `"key_123"`; use `---` delimiters in comments. Secret-scan only scans staged files, not .git.
|
|
109
|
+
|
|
110
|
+
### 2. Worktree isolation violated
|
|
111
|
+
- **Symptom**: Editing feature branch X; changes also appear in primary tree
|
|
112
|
+
- **Cause**: Agents not using isolated worktrees; sharing same `.git/index`
|
|
113
|
+
- **Fix**: Each agent task must use `git worktree add ../wt-item-slug -b feature/item-slug origin/main`; verify with `git worktree list`
|
|
114
|
+
|
|
115
|
+
### 3. Heartbeat stale/missing
|
|
116
|
+
- **Symptom**: Watchdog won't start; error: "unreadable state dir" or "stale heartbeat"
|
|
117
|
+
- **Cause**: `state/.watchdog-heartbeat` missing/corrupted or directory unreadable (permissions)
|
|
118
|
+
- **Fix**: `rm -f state/.watchdog-heartbeat && bash daemons/run-watchdog.sh --once` (re-creates it)
|
|
119
|
+
|
|
120
|
+
### 4. Port 8770 conflict
|
|
121
|
+
- **Symptom**: `python ui/serve.py` fails: "Address already in use"
|
|
122
|
+
- **Cause**: Old dashboard process still bound, or another service using port 8770
|
|
123
|
+
- **Fix**: `lsof -i :8770 | grep -v PID | awk '{print $2}' | xargs kill -9` (Mac/Linux); Windows: `netstat -ano | grep 8770`, kill the PID. Or change `dashboardPort` in config.
|
|
124
|
+
|
|
125
|
+
### 5. Git identity placeholder
|
|
126
|
+
- **Symptom**: Pre-push hook fails: "user identity invalid"; commits fail
|
|
127
|
+
- **Cause**: Git config has template defaults (e.g., "John Doe") instead of your name
|
|
128
|
+
- **Fix**: `git config user.name "Your Name" && git config user.email "you@example.com"`
|
|
129
|
+
|
|
130
|
+
### 6. CRLF line endings corrupt scripts
|
|
131
|
+
- **Symptom**: Bash script fails: "command not found" at line 50 (but line 50 exists)
|
|
132
|
+
- **Cause**: Editor/Windows converted LF to CRLF; bash reads `\r` as part of command
|
|
133
|
+
- **Fix**: `git config core.autocrlf false` locally; convert file: `dos2unix daemons/run-watchdog.sh`
|
|
134
|
+
|
|
135
|
+
### 7. Test count drift in CI
|
|
136
|
+
- **Symptom**: Health-check fails: "expected 127 tests, got 128"
|
|
137
|
+
- **Cause**: Added new test file but didn't update baseline in tools/health_score.py or CI config
|
|
138
|
+
- **Fix**: Run `python tools/health_score.py --json` to get true count; update baseline in CI gate
|
|
139
|
+
|
|
140
|
+
### 8. UTF-8 encoding on Windows
|
|
141
|
+
- **Symptom**: Pre-publish gate fails with encoding error in secret_scan.py
|
|
142
|
+
- **Cause**: Python opens files in system encoding (cp1252) instead of UTF-8
|
|
143
|
+
- **Fix**: Aesop tools now force UTF-8 internally; ensure `PYTHONIOENCODING=utf-8` if running external Python scripts
|
|
144
|
+
|
|
145
|
+
### 9. Cost ceiling never triggers
|
|
146
|
+
- **Symptom**: Cost ceilings configured; spending uncapped; halt never fires
|
|
147
|
+
- **Cause**: Claude Code driver returns `get_tokens_spent()=None`; coerces to 0; ledger fallback skipped
|
|
148
|
+
- **Fix**: For Claude Code, cost tracking integrates live API; for non-Claude backends, ledger fallback logs at end of wave
|
|
149
|
+
|
|
150
|
+
### 10. Hook TTY behavior blocks CI pushes
|
|
151
|
+
- **Symptom**: `git push` fails in CI/cron: "cannot read stdin"; interactive prompt hangs
|
|
152
|
+
- **Cause**: Hook tries to detect TTY when stdin is `/dev/null` (non-interactive context)
|
|
153
|
+
- **Fix**: Hook now handles empty stdin (rc=2 for delete-only, skip scan + log); use `hooks/pre-push-policy.sh --test` in CI (no-op validation)
|
|
154
|
+
|
|
155
|
+
---
|
|
156
|
+
|
|
157
|
+
## Next Steps
|
|
158
|
+
|
|
159
|
+
1. **Read [INSTALL.md](./INSTALL.md)** — Full setup and environment variables
|
|
160
|
+
2. **Run [health_score.py](../tools/health_score.py)** — Continuous readiness checks
|
|
161
|
+
3. **Explore the [README](../README.md)** — Demo walkthrough and proof numbers
|
|
162
|
+
4. **For troubleshooting**, check [GOVERNANCE.md](./GOVERNANCE.md) or open an issue on GitHub
|
|
163
|
+
|
|
164
|
+
---
|
|
165
|
+
|
|
166
|
+
**License**: PolyForm Strict 1.0.0 (source-available, noncommercial). See [LICENSE](../LICENSE) for details.
|
package/docs/TEAM-STATE.md
CHANGED
|
@@ -1,203 +1,35 @@
|
|
|
1
|
-
# Team-Shared State:
|
|
1
|
+
# Team-Shared State: Current & Future Architecture
|
|
2
2
|
|
|
3
|
-
**Status**:
|
|
3
|
+
**Status (0.1.0)**: Single-instance with durable git checkpointing. Event-sourced SQLite module (`state_store/`) is production-ready but not yet integrated into orchestrator reader/writer paths.
|
|
4
4
|
|
|
5
5
|
---
|
|
6
6
|
|
|
7
|
-
##
|
|
7
|
+
## Current State Model (Git-as-Checkpoint)
|
|
8
8
|
|
|
9
9
|
Aesop currently stores orchestration state in **two git-tracked files**:
|
|
10
|
-
- `STATE.md` — intent, decisions, phase, NEXT STEPS (single-writer orchestrator)
|
|
11
|
-
- `BUILDLOG.md` — append-only progress snapshots (agents append one line per work unit)
|
|
12
10
|
|
|
13
|
-
|
|
11
|
+
- **STATE.md** — Orchestrator intent, decisions, phase, and NEXT STEPS (single-writer)
|
|
12
|
+
- **BUILDLOG.md** — Append-only progress snapshots (agents append one line per work unit)
|
|
14
13
|
|
|
15
|
-
|
|
16
|
-
2. **No transactions/concurrency** — Git has no concept of atomic multi-file updates or per-agent locks; coordination is manual and error-prone.
|
|
17
|
-
3. **No real-time status** — Readers must `git fetch` to see the latest state; no subscription/push model for live updates.
|
|
18
|
-
4. **Single-writer bottleneck** — The orchestrator is the only writer to STATE.md; other workers cannot safely report their own state.
|
|
14
|
+
**Why it works**: Single-instance is proven. State survives machine wipes, resync on session restart, human-diffable for review and forensics.
|
|
19
15
|
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
## Current Truth: Git-as-State (Single-Instance)
|
|
23
|
-
|
|
24
|
-
### Data model
|
|
25
|
-
|
|
26
|
-
State lives as **human-readable JSON** in two git-tracked files:
|
|
27
|
-
|
|
28
|
-
```
|
|
29
|
-
orchestrator (Fable/Opus) ←→ STATE.md (single-writer, locked decisions + phase + NEXT STEPS)
|
|
30
|
-
←→ BUILDLOG.md (append-only, agents append one line per work unit)
|
|
31
|
-
|
|
32
|
-
↓ (read-only consumers)
|
|
33
|
-
|
|
34
|
-
dashboards, monitoring, forensics
|
|
35
|
-
```
|
|
36
|
-
|
|
37
|
-
**Invariants**:
|
|
38
|
-
- Only the orchestrator writes to STATE.md (enforced by discipline, not by code)
|
|
39
|
-
- BUILDLOG.md is append-only; orchestrator never rewrites earlier entries
|
|
40
|
-
- All state is human-diffable (intentional — enables review, forensics, inheritance)
|
|
41
|
-
- Durability via git: push after each update, recover from `git log` on session restart
|
|
42
|
-
- Git is the **source of truth** for current state
|
|
43
|
-
|
|
44
|
-
### Limitations for team scale
|
|
45
|
-
|
|
46
|
-
- **No concurrent writers** — Two orchestrators or agents trying to write simultaneously causes merge conflicts
|
|
47
|
-
- **No transactions** — A phase update + next-steps change is two separate commits, not atomic
|
|
48
|
-
- **High latency** — State changes require `git push`; observability is on a push-frequency, not real-time
|
|
49
|
-
- **No locking** — State can be read/modified while orchestrator is deciding; no optimistic concurrency
|
|
50
|
-
- **No history/events** — Only final state (current file contents) is stored; intermediate states are lost
|
|
16
|
+
**Limitation**: Does not scale to teams—no concurrent writers, no transactions, high latency (requires `git push`).
|
|
51
17
|
|
|
52
18
|
---
|
|
53
19
|
|
|
54
|
-
##
|
|
55
|
-
|
|
56
|
-
### Proposed data model
|
|
20
|
+
## Future: SQLite Event Log + Git Export
|
|
57
21
|
|
|
58
|
-
|
|
59
|
-
orchestrator/agents write events ─→ SQLite WAL event log (LIVE substrate, fast, transactional)
|
|
60
|
-
↓
|
|
61
|
-
state_store/ EventStore
|
|
62
|
-
(append() atomic across processes/threads)
|
|
63
|
-
↓
|
|
64
|
-
[snapshots for O(n) tail-replay]
|
|
65
|
-
↓
|
|
66
|
-
projections/StateAPI (project tracker state)
|
|
67
|
-
↓
|
|
68
|
-
export_tracker() ─────────→ STATE.md / BUILDLOG.md (git-tracked EXPORT, human-diffable, immutable)
|
|
69
|
-
↓
|
|
70
|
-
[dashboards, monitoring, forensics read git export]
|
|
71
|
-
```
|
|
72
|
-
|
|
73
|
-
### Key design principles
|
|
74
|
-
|
|
75
|
-
1. **Event sourcing**: State is a fold of immutable events, not mutable files. Each write is a single `append(stream, event_type, payload, actor)` call — atomic, ordered, versionable.
|
|
76
|
-
|
|
77
|
-
2. **Append-only**: No deletes or mutations. History is preserved forever. Replayability + auditing for free.
|
|
78
|
-
|
|
79
|
-
3. **Per-stream versioning**: Each "stream" (e.g., `"tracker"`, `"orchestrator_status"`) has its own 1-based, gapless version. Enables deterministic snapshots and tail-replay (see `state_store/store.py` line 81–92 and `state_store/projections.py` line 117–127).
|
|
80
|
-
|
|
81
|
-
4. **Concurrency-safe writes**: SQLite WAL mode + `BEGIN IMMEDIATE` transactions guarantee that two writers on the same file never collide or duplicate a version (see `state_store/store.py` line 30–94). Tested in `tests/test_state_store.py::EventStoreTest::test_concurrent_appends_have_no_dupes_or_gaps`.
|
|
82
|
-
|
|
83
|
-
5. **Snapshots for O(n) tail-replay**: Instead of replaying all events on every read, save materialized state snapshots at key versions; future replays fold only newer events (see `state_store/store.py` line 130–196 and `state_store/projections.py` line 84–114).
|
|
84
|
-
|
|
85
|
-
6. **Git as export, not source**: The SQLite store is the system-of-record. `export_tracker(api, out_path)` renders the current projection back to human-readable JSON for git durability and review (see `state_store/export.py`). Git is now **read-only for state** (except for manual edits, which become new events on next `ingest_tracker_json`).
|
|
86
|
-
|
|
87
|
-
7. **Projection-based readers**: Callers do not query the raw event log; instead, they call `api.project(view)` to get the current state as a dict. New projections (e.g., `project_orchestrator_status`) can be registered in `state_store/api.py` without changing the event store (see line 12–35).
|
|
88
|
-
|
|
89
|
-
---
|
|
22
|
+
**Vision**: Event-sourced SQLite WAL becomes the system-of-record. Git exports are rendered from current projections for durability and review.
|
|
90
23
|
|
|
91
|
-
|
|
24
|
+
**In the repo now**:
|
|
25
|
+
- `state_store/store.py` — EventStore with atomic `append()`, concurrency-safe writes
|
|
26
|
+
- `state_store/projections.py` — Fold events into tracker state
|
|
27
|
+
- `state_store/api.py` — Facade for backend swaps (SQLite today, Postgres later)
|
|
28
|
+
- `tests/test_state_store.py` — Concurrent-write proofs, projection tests
|
|
92
29
|
|
|
93
|
-
|
|
30
|
+
**Next steps**: Wire the orchestrator and agent writers to use `StateAPI` instead of git direct writes. This unblocks team-scale coordination without breaking single-instance durability (git exports remain).
|
|
94
31
|
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
**Class**: `EventStore(db_path)`
|
|
98
|
-
|
|
99
|
-
**Public API**:
|
|
100
|
-
- `append(stream: str, event_type: str, payload: dict, actor: str = "system") -> int` — Append one event, return its per-stream version. Atomic: multiple writers see no dupes or gaps (tests line 90–108).
|
|
101
|
-
- `read(stream: str) -> list` — All events in a stream, ascending by version. Returns list of dicts with `{"id", "ts", "actor", "stream", "type", "payload", "version"}`.
|
|
102
|
-
- `read_all() -> list` — All events across all streams, ascending by global id.
|
|
103
|
-
- `save_snapshot(stream: str, event_version: int, projection: dict) -> None` — Persist a materialized projection at a specific version (line 130–153).
|
|
104
|
-
- `read_snapshot(stream: str) -> tuple | None` — Read the most recent snapshot; returns `(event_version, projection_dict, checksum)` or None (line 155–197).
|
|
105
|
-
|
|
106
|
-
**Concurrency model**:
|
|
107
|
-
- Multiple writers: `BEGIN IMMEDIATE` locks the write lock up front, so read-max-version-then-insert is atomic (line 80).
|
|
108
|
-
- Multiple readers: SQLite WAL mode allows readers to run concurrently with writers on separate .db-wal shadow files (no blocking).
|
|
109
|
-
- Process/thread-safe: each `append()` call opens its own connection; `busy_timeout=5000` (5 seconds) waits if the DB is locked (line 32–94).
|
|
110
|
-
- Tested with two threads, each with its own EventStore instance, appending 100 events concurrently with no dupes (tests line 90–108).
|
|
111
|
-
|
|
112
|
-
### 2. Projections (Fold Events into State)
|
|
113
|
-
|
|
114
|
-
**File**: `state_store/projections.py`
|
|
115
|
-
|
|
116
|
-
**Functions**:
|
|
117
|
-
- `_fold_events(events, order=None, items=None) -> (order, items)` — Fold events into an accumulated state tuple. Mutates order + items in place. Helper used by both full and snapshot-aware projections (line 25–71).
|
|
118
|
-
|
|
119
|
-
- `project_tracker(events: list) -> dict` — Original full-replay projection: fold all events → `{"version": 1, "items": [items in first-seen order]}`. Defines tracker event schema:
|
|
120
|
-
- `item_created` (payload = full item dict): establishes item, kept in first-seen order
|
|
121
|
-
- `item_updated` (payload = `{"id": ..., ...partial fields}`): merges partial fields onto existing item
|
|
122
|
-
- `item_archived` (payload = `{"id": ...}`): sets `status` to `"archived"`, optionally merges `completed_at`
|
|
123
|
-
- Unknown event types: ignored (line 74–81).
|
|
124
|
-
|
|
125
|
-
- `project_tracker_with_snapshot(store, stream: str, events: list) -> dict` — Snapshot-aware projection: load latest snapshot; fold only tail events (after snapshot version) for O(n) tail-replay. Falls back to full replay if snapshot is missing or corrupt (line 84–114).
|
|
126
|
-
|
|
127
|
-
- `save_snapshot(store, stream: str, event_version: int, projection: dict) -> None` — Wrapper over `store.save_snapshot()` (line 117–126).
|
|
128
|
-
|
|
129
|
-
**Event schema for tracker**:
|
|
130
|
-
```json
|
|
131
|
-
{
|
|
132
|
-
"type": "item_created",
|
|
133
|
-
"payload": {"id": "abc", "title": "...", "lane": "proposed", "status": "todo", ...}
|
|
134
|
-
}
|
|
135
|
-
{
|
|
136
|
-
"type": "item_updated",
|
|
137
|
-
"payload": {"id": "abc", "lane": "in-progress"}
|
|
138
|
-
}
|
|
139
|
-
{
|
|
140
|
-
"type": "item_archived",
|
|
141
|
-
"payload": {"id": "abc", "completed_at": "2026-07-17T...Z"}
|
|
142
|
-
}
|
|
143
|
-
```
|
|
144
|
-
|
|
145
|
-
Tolerance: unknown ids on update/archive are ignored (no error); unknown event types are skipped (line 62–70).
|
|
146
|
-
|
|
147
|
-
### 3. Facade API (Single Seam for Backend Swaps)
|
|
148
|
-
|
|
149
|
-
**File**: `state_store/api.py`
|
|
150
|
-
|
|
151
|
-
**Class**: `StateAPI(db_path)`
|
|
152
|
-
|
|
153
|
-
**Public API**:
|
|
154
|
-
- `append(stream: str, event_type: str, payload: dict, actor: str = "system") -> int` — Append event, return version.
|
|
155
|
-
- `get(stream: str) -> list` — Read all events in a stream.
|
|
156
|
-
- `project(view: str) -> dict` — Fold the named stream through its projector into current state. Raises `ValueError` if projector not registered (line 15–35).
|
|
157
|
-
|
|
158
|
-
**Backend swaps**: The backend (SQLite now, Postgres later) is abstracted here. Callers **never** touch `EventStore` directly; they use `StateAPI` only. On cutover to Postgres, only `api.py` changes; call sites stay the same.
|
|
159
|
-
|
|
160
|
-
---
|
|
161
|
-
|
|
162
|
-
## New State Streams (Beyond Tracker)
|
|
163
|
-
|
|
164
|
-
The tracker stream is the first proof of concept. The event log supports **arbitrary streams**; new streams are registered by adding a projector to `_PROJECTORS` in `state_store/api.py` (line 12).
|
|
165
|
-
|
|
166
|
-
### Proposed: `orchestrator_status` stream
|
|
167
|
-
|
|
168
|
-
**Purpose**: Record orchestrator phase, NEXT STEPS, decisions, and audit results in real-time without git writes.
|
|
169
|
-
|
|
170
|
-
**Events**:
|
|
171
|
-
```json
|
|
172
|
-
{
|
|
173
|
-
"type": "phase_changed",
|
|
174
|
-
"payload": {"phase": "wave-rc.3", "reason": "stable release shipped"}
|
|
175
|
-
}
|
|
176
|
-
{
|
|
177
|
-
"type": "next_steps_updated",
|
|
178
|
-
"payload": {"steps": ["reconcile state_store SI", "add orchestrator_status stream", ...]}
|
|
179
|
-
}
|
|
180
|
-
{
|
|
181
|
-
"type": "decision_locked",
|
|
182
|
-
"payload": {"key": "model_dispatch_core", "value": "out-of-repo (structural)", "justification": "..."}
|
|
183
|
-
}
|
|
184
|
-
{
|
|
185
|
-
"type": "audit_findings_recorded",
|
|
186
|
-
"payload": {"level": "wave-rc.2", "findings_count": 3, "summary": "..."}
|
|
187
|
-
}
|
|
188
|
-
```
|
|
189
|
-
|
|
190
|
-
**Projection** (pseudo-code):
|
|
191
|
-
```python
|
|
192
|
-
def project_orchestrator_status(events):
|
|
193
|
-
status = {
|
|
194
|
-
"phase": None,
|
|
195
|
-
"decisions": {},
|
|
196
|
-
"next_steps": [],
|
|
197
|
-
"latest_audit": None,
|
|
198
|
-
}
|
|
199
|
-
for ev in events:
|
|
200
|
-
if ev["type"] == "phase_changed":
|
|
32
|
+
For implementation details, see `state_store/` source code and inline documentation.
|
|
201
33
|
status["phase"] = ev["payload"]["phase"]
|
|
202
34
|
elif ev["type"] == "decision_locked":
|
|
203
35
|
key = ev["payload"]["key"]
|
package/docs/reproduce.md
CHANGED
|
@@ -1,6 +1,21 @@
|
|
|
1
1
|
# Reproducing Aesop
|
|
2
2
|
|
|
3
|
-
This document explains how to reproduce the project's claims from a clean checkout.
|
|
3
|
+
This document explains how to reproduce the project's claims from a clean checkout or installed package.
|
|
4
|
+
|
|
5
|
+
## Quick Start: One-Command Verification
|
|
6
|
+
|
|
7
|
+
Run the verification suite with automatic context detection:
|
|
8
|
+
|
|
9
|
+
```bash
|
|
10
|
+
aesop reproduce
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
This command automatically detects your context (repository checkout or installed package) and runs the appropriate test suite:
|
|
14
|
+
|
|
15
|
+
- **Repository checkout**: Runs the full test suite (Node syntax, shell syntax, Node tests, Python tests, benchmark, browser proofs)
|
|
16
|
+
- **Installed package**: Runs shipped self-checks (preflight checks, health score, secret-scan selftest, packaging assertions)
|
|
17
|
+
|
|
18
|
+
Output includes per-step timing and a clear pass/fail summary. Exit code 0 = all checks passed.
|
|
4
19
|
|
|
5
20
|
## Overview
|
|
6
21
|
|
|
@@ -55,20 +70,35 @@ Both can be reproduced entirely offline, without API keys or external dependenci
|
|
|
55
70
|
python tools/bench_runner.py --runner mock
|
|
56
71
|
```
|
|
57
72
|
|
|
58
|
-
##
|
|
73
|
+
## CI Equivalence: aesop reproduce vs. GitHub Actions
|
|
59
74
|
|
|
60
|
-
The `reproduce`
|
|
75
|
+
The `aesop reproduce` command mirrors `.github/workflows/reproduce.yml` for local execution.
|
|
76
|
+
|
|
77
|
+
### GitHub Actions Workflow (`reproduce.yml`)
|
|
78
|
+
|
|
79
|
+
The `reproduce` workflow in GitHub Actions automates testing:
|
|
61
80
|
|
|
62
81
|
1. **Fresh checkout**: Uses a clean clone (no state reuse from other CI jobs)
|
|
63
82
|
2. **Syntax checks**: Validates all Node.js (.mjs) and shell scripts (.sh)
|
|
64
83
|
3. **Node.js tests**: Runs the full test suite with `npm run test:node`
|
|
65
84
|
4. **Python tests**: Runs the full test suite with `python -m unittest discover`
|
|
66
85
|
5. **Benchmark scorer**: Proves the offline mock benchmark reproduces (no external API calls)
|
|
86
|
+
6. **Browser proofs**: Validates UI components with Playwright
|
|
67
87
|
|
|
68
88
|
The workflow is triggered:
|
|
69
89
|
- **Manually**: Via GitHub Actions "Run workflow" button (workflow_dispatch)
|
|
70
90
|
- **Weekly**: Every Sunday at 2:00 AM UTC (schedule)
|
|
71
91
|
|
|
92
|
+
### Local Reproduction
|
|
93
|
+
|
|
94
|
+
To run the same tests locally from a clean clone:
|
|
95
|
+
|
|
96
|
+
```bash
|
|
97
|
+
aesop reproduce
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
This produces the same verification steps and provides immediate feedback before pushing.
|
|
101
|
+
|
|
72
102
|
## What Is and Isn't Reproduced
|
|
73
103
|
|
|
74
104
|
### Reproduced (offline, no API keys needed)
|
package/driver/CLAUDE.md
CHANGED
|
@@ -1,11 +1,8 @@
|
|
|
1
1
|
# driver/ — AgentDriver backend-portability seam
|
|
2
2
|
|
|
3
|
-
**What**:
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
Grounded in a multi-model portability design spike.
|
|
8
|
-
**Phase 1 (interface + reference adapter, shipped)** + **Phase 2 (Codex OpenAI Chat Completions, shipped)** + **Phase 3 (wave-manifest bridge, shipped)**.
|
|
3
|
+
**What**: the domain contract letting aesop's wave loop run on non-Claude backends
|
|
4
|
+
(Codex, open models); the loop dispatches only through the `AgentDriver` interface.
|
|
5
|
+
**Phases 1-3 shipped** (interface + reference adapter; Codex Chat Completions; wave bridge).
|
|
9
6
|
|
|
10
7
|
## Files
|
|
11
8
|
|
|
@@ -24,6 +21,8 @@ Grounded in a multi-model portability design spike.
|
|
|
24
21
|
- **openai_compatible_driver.py** — OpenAI-compatible backend (Ollama, OpenRouter, etc.).
|
|
25
22
|
- **verification_policy.py** — Maps verification tier -> orchestrator tuning (validate_all_json,
|
|
26
23
|
spot_check_frac, repair_cap, require_adversarial_review).
|
|
24
|
+
- **wave_loop.py** — the wave ENGINE: preflight ownership guard, parallel build,
|
|
25
|
+
bounded repair, adversarial review, per-repo batched git ship, recovery journal.
|
|
27
26
|
- **wave_bridge.py** — Phase 3: bridges AgentDriver backends to wave manifest items.
|
|
28
27
|
build_manifest_item() enriches with verificationTier + model; dispatch_item() routes
|
|
29
28
|
by capability and decides green ONLY from test exit code (not model's say-so).
|
|
@@ -33,8 +32,7 @@ Grounded in a multi-model portability design spike.
|
|
|
33
32
|
- **../tests/test_codex_driver_e2e.py** — Phase 2 end-to-end offline tests
|
|
34
33
|
(FakeTransport, red-to-green verification, retry logic, ownership enforcement)
|
|
35
34
|
+ gated live test (AESOP_CODEX_LIVE env var).
|
|
36
|
-
- **../tests/test_wave_bridge.py** — Phase 3
|
|
37
|
-
non-Claude backend drives a real RED stub to verified-GREEN offline (no API key,
|
|
35
|
+
- **../tests/test_wave_bridge.py** — Phase 3 offline e2e (honest green: exit 0 only).
|
|
38
36
|
no network). Tests: manifest building, routing, fail-safe, ownership enforcement,
|
|
39
37
|
headline test (red stub + FakeTransport fix + test pass -> ok=True).
|
|
40
38
|
|
|
@@ -85,47 +83,18 @@ Optional (non-abstract): `get_tokens_spent()`.
|
|
|
85
83
|
|
|
86
84
|
## Phase 2 Codex Implementation Details
|
|
87
85
|
|
|
88
|
-
|
|
89
|
-
end-to-end through the AgentDriver and produce orchestrator-verified results,
|
|
90
|
-
entirely offline (no API key, no network in CI).
|
|
91
|
-
|
|
92
|
-
**dispatch_worker**: Orchestrator-managed worker (Tier 2):
|
|
93
|
-
- Injects owned-file contents into the prompt (worker has no filesystem access).
|
|
94
|
-
- Calls OpenAI Chat Completions API via injectable transport (default=urllib).
|
|
95
|
-
- Requests strict JSON schema output (full-file replacements).
|
|
96
|
-
- Validates ALL JSON with bounded in-turn retry (<=2 attempts).
|
|
97
|
-
- Enforces ownership: rejects out-of-scope paths wholesale.
|
|
98
|
-
- Pre-dispatch max_owned_bytes guard: fails safe on oversized files (no
|
|
99
|
-
truncation).
|
|
100
|
-
- Writes full-file replacements to disk; never applies diffs.
|
|
101
|
-
- Records usage.total_tokens for cost tracking.
|
|
102
|
-
- CRITICAL: Green is NOT decided by the model's done:true; it is decided by
|
|
103
|
-
the orchestrator running run_command and getting exit 0 (center verification).
|
|
104
|
-
|
|
105
|
-
**Transport seam**: The injectable transport callable keeps CI offline. Tests
|
|
106
|
-
pass FakeTransport with canned responses; production code reads OPENAI_API_KEY
|
|
107
|
-
from environment and uses default_openai_transport (stdlib urllib, hard timeout).
|
|
108
|
-
|
|
109
|
-
**Verification policy**: verification_policy(caps) maps tier 2 -> {
|
|
110
|
-
validate_all_json: True, spot_check_frac: 0.50, repair_cap: 2,
|
|
111
|
-
require_adversarial_review: True }. Feeds the wave's integration verifier.
|
|
86
|
+
Codex driver (Tier 2): injects file contents into prompt, calls OpenAI Chat Completions via injectable transport, validates JSON with bounded retry, enforces ownership. CRITICAL: Green = exit 0 only. Verification policy: tier 2 -> {validate_all_json:True, spot_check_frac:0.50, repair_cap:2, require_adversarial_review:True}. **P1 Security**: Default model map uses gpt-4o-mini (worker, supports json_schema); init-time guard rejects models lacking json_schema support unless `allow_unverified_models=True` (P1 gate: prevent gpt-3.5-turbo silent failures).
|
|
112
87
|
|
|
113
88
|
## Phase 3 Bridge Implementation Details
|
|
114
89
|
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
**build_manifest_item(driver, item) -> dict**:
|
|
119
|
-
|
|
120
|
-
|
|
121
|
-
|
|
122
|
-
|
|
123
|
-
validateAllJson from verification_policy(caps)).
|
|
124
|
-
- Preserves all input fields; adds model, tier, and all four policy fields.
|
|
125
|
-
- The policy is RESOLVED ONCE in Python and carried as literal manifest fields, so
|
|
126
|
-
the template cannot recompute/drift. Tier-1/Claude path with no verificationTier
|
|
127
|
-
maintains byte-identical behavior (tier-1 defaults: repairCap=1, requireAdversarialReview=false,
|
|
128
|
-
spotCheckFrac=0.10, validateAllJson=false).
|
|
90
|
+
Connects AgentDriver backends to wave-flat-dispatch manifest items; verification
|
|
91
|
+
tier is driven by backend capability, not config.
|
|
92
|
+
|
|
93
|
+
**build_manifest_item(driver, item) -> dict**: enriches a backlog item with model
|
|
94
|
+
(driver.resolve_model), verificationTier (probe), and the four policy knobs from
|
|
95
|
+
verification_policy(caps) — RESOLVED ONCE, carried as literal manifest fields so the
|
|
96
|
+
template cannot recompute/drift; tier-1/Claude path stays byte-identical (repairCap=1,
|
|
97
|
+
requireAdversarialReview=false, spotCheckFrac=0.10, validateAllJson=false).
|
|
129
98
|
|
|
130
99
|
**dispatch_item(driver, item) -> dict**:
|
|
131
100
|
- Routes by worker_filesystem_access: True→{route:'harness'}; False→orchestrator-managed
|
|
@@ -138,11 +107,44 @@ items, making verification tier driven by backend capability, not config.
|
|
|
138
107
|
stub, applies a fix, runs the test, and returns ok=True ONLY because the test passed
|
|
139
108
|
(exit 0). All offline, no API key, no network.
|
|
140
109
|
|
|
110
|
+
## Wave Scheduler (WS3a Pilot) + GATE-1 Handoff Kit
|
|
111
|
+
|
|
112
|
+
**wave_scheduler.py** — single-cycle backlog orchestration: intake up to N file-disjoint todo items from tracker.json (empty/missing ownsFiles REJECTED; paths normalized posix+casefold-on-Windows before overlap checks; required fields pre-validated) -> manifest via build_manifest_item (model + verificationTier from driver.probe) -> HALT + cost-ceiling gates (fail-CLOSED: module import failure or check exception = abort with honest Report, phase=gate_unavailable) -> run_wave (recovery journal + git ship) -> STOP before merge; Report JSON with per-item observability (GATE-1). After ship, selected items atomically marked in_progress in tracker (temp+os.replace; dry-run never mutates) so a second run cannot double-dispatch.
|
|
113
|
+
|
|
114
|
+
**CLI** (GATE-1): `python driver/wave_scheduler.py --tracker <path> --max-items N --dry-run|--execute --driver claude|codex` (default: claude). For codex+execute, requires OPENAI_API_KEY env var; dry-run works without it.
|
|
115
|
+
|
|
116
|
+
**Tests** (35+): disjoint/normalization/rejection, gate fail-closed, dry-run, GATE-1 per-item/driver/ceiling/codex tests; module-tmpdir hygiene; all TestCase.
|
|
117
|
+
|
|
118
|
+
**Invariants**: stdlib-only, ASCII, Windows+Linux safe (list-form subprocess); manifest items carry resolved policy knobs from verification_policy (no recompute drift); merge stays manual in the pilot.
|
|
119
|
+
|
|
120
|
+
### REPORT-CONTRACT (GATE-1 Orchestrator Handoff)
|
|
121
|
+
|
|
122
|
+
The scheduler emits a Report JSON consumed by the orchestrator to decide merge eligibility:
|
|
123
|
+
```
|
|
124
|
+
{
|
|
125
|
+
"phase": "dispatch"|"intake"|"halt"|"ceiling"|"gate_unavailable"|"manifest",
|
|
126
|
+
"wave_id": "<uuid>",
|
|
127
|
+
"items_selected": ["id1", "id2"], // IDs selected for this wave
|
|
128
|
+
"items_shipped": [
|
|
129
|
+
{ "slug": "feat/x", "backend": "claude-code|codex", "tier": 1|2|3|4, "verified": true|false (false = NOT PROVEN, not necessarily failed; tier null = no build record), "testExit": 0|1|null },
|
|
130
|
+
...
|
|
131
|
+
],
|
|
132
|
+
"merged": false, "tracker_update_attempted": true|absent, "tracker_unmapped_slugs": [..]|absent (LOUD -> success false), // pilot always false (manual merge)
|
|
133
|
+
"success": true|false,
|
|
134
|
+
"timestamp": "<ISO8601>",
|
|
135
|
+
"branch": "<branch_name>", // optional; set when ship succeeds
|
|
136
|
+
"sha": "<git_sha>", // optional; set when ship succeeds
|
|
137
|
+
"halt_reason": "...", // optional; set if halted
|
|
138
|
+
"ceiling_reason": "...", // optional; ceiling exceeded before dispatch (abort-before-dispatch only)
|
|
139
|
+
"error": "..." // optional; unexpected error
|
|
140
|
+
}
|
|
141
|
+
```
|
|
142
|
+
Ceiling semantics: checked BEFORE run_wave dispatch (phase=ceiling, ceiling_reason populated). Mid-wave ceiling trips are run_wave's responsibility. items_shipped[] carries per-item observability: slug, backend (driver name), tier (verificationTier), verified (from test exit 0), testExit (test command exit code or null if not run).
|
|
143
|
+
|
|
141
144
|
## Status
|
|
142
145
|
|
|
143
146
|
- **Phase 1**: shipped. Interface + Claude reference adapter + contract tests.
|
|
144
147
|
- **Phase 2**: shipped. Codex OpenAI Chat Completions implementation. Offline tests GREEN.
|
|
145
148
|
- **Phase 3**: shipped. Wave bridge: driver → manifest, orchestrator-side dispatch.
|
|
146
149
|
Proves non-Claude backends drive items end-to-end with honest green (test exit 0 only).
|
|
147
|
-
- **
|
|
148
|
-
- **Future**: Open-model adapter (Tier-4 backend).
|
|
150
|
+
- **Wave Scheduler (WS3a) + GATE-1**: shipped. Single-cycle orchestration: intake → manifest → dispatch → report (manual merge). Per-item observability, driver injection (--driver claude|codex), ceiling semantics documented.
|