@matt82198/aesop 0.3.2 → 0.4.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +47 -0
- package/README.md +24 -9
- package/aesop.config.example.json +16 -0
- package/bin/cli.js +107 -34
- package/daemons/run-watchdog.sh +6 -2
- package/docs/CONFIGURE.md +31 -20
- package/docs/FIRST-WAVE.md +29 -9
- package/docs/INSTALL.md +181 -26
- package/docs/MICROKERNEL.md +452 -0
- package/docs/PORTING.md +4 -0
- package/docs/README.md +7 -3
- package/docs/THE-AESOP-HYPOTHESIS.md +156 -0
- package/driver/CLAUDE.md +123 -123
- package/driver/README.md +40 -2
- package/driver/adjudication_gate.py +367 -0
- package/driver/aesop.config.example.json +20 -4
- package/driver/backend_config.py +603 -12
- package/driver/claude_code_driver.py +9 -17
- package/driver/codex_driver.py +80 -25
- package/driver/context_pack.py +454 -0
- package/driver/openai_compatible_driver.py +53 -11
- package/driver/openai_transport.py +31 -10
- package/driver/orchestrator_backend.py +332 -0
- package/driver/orchestrator_driver.py +589 -0
- package/driver/proc_util.py +134 -0
- package/driver/wave_loop.py +801 -59
- package/driver/wave_scheduler.py +361 -37
- package/monitor/collect-signals.mjs +83 -0
- package/package.json +1 -1
- package/state_store/coordination.py +65 -5
- package/tools/ci_merge_wait.py +88 -42
- package/tools/ci_shard_runner.py +128 -0
- package/tools/doctor.js +124 -12
- package/tools/reproduce.js +11 -2
- package/tools/seated_shadow_adjudication.py +920 -0
- package/tools/self_stats.py +61 -6
- package/tools/shadow_adjudication.py +1024 -0
- package/tools/verify_test_suite_count.py +250 -0
- package/tools/verify_ui_trio_redaction_proof.py +292 -0
- package/tools/wave_manifest_lint.py +485 -0
- package/tools/wave_scorecard.py +430 -0
- package/ui/config.py +1 -1
- package/ui/cost.py +176 -3
- package/ui/web/dist/assets/{index-CP68RIh3.js → index-4rp6B_ID.js} +1 -1
- package/ui/web/dist/assets/{index-CNQxaiOW.css → index-B2lTtpsH.css} +1 -1
- package/ui/web/dist/index.html +2 -2
|
@@ -0,0 +1,452 @@
|
|
|
1
|
+
# Aesop as an AI Micro-Kernel: Two Swappable Seats
|
|
2
|
+
|
|
3
|
+
**Status**: HS-1 (unified two-seat config) + HS-2 (live orchestrator-seat swap)
|
|
4
|
+
shipped. This doc is the conceptual centerpiece for both — what the seam
|
|
5
|
+
actually is, what is proven about it today, and how to swap a seat's model
|
|
6
|
+
in about a minute.
|
|
7
|
+
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
## What it is
|
|
11
|
+
|
|
12
|
+
Aesop's wave loop has exactly two places where a large language model makes
|
|
13
|
+
a decision:
|
|
14
|
+
|
|
15
|
+
1. **The worker seat** — does the work: writes files, runs commands, reports
|
|
16
|
+
a structured result. Interface: `AgentDriver`.
|
|
17
|
+
2. **The orchestrator seat** — judges the work: decides whether a
|
|
18
|
+
test-verified item is safe to ship. Interface: `OrchestratorBackend`
|
|
19
|
+
(called through `OrchestratorDriver.decide()`).
|
|
20
|
+
|
|
21
|
+
Both seats are **swappable parts**, not the engine itself. The engine —
|
|
22
|
+
`driver/wave_loop.py`'s `run_wave()` — never talks to a model directly; it
|
|
23
|
+
calls exactly two abstract interfaces and does not care what sits behind
|
|
24
|
+
either one. This is the micro-kernel idea applied to an agent harness: keep
|
|
25
|
+
the kernel (the loop, the state contract, the human-facing report) tiny and
|
|
26
|
+
stable, and make every model a replaceable driver plugged into a fixed seam.
|
|
27
|
+
|
|
28
|
+
**Identity lives in files, not memory.** Aesop is crash-only: there is no
|
|
29
|
+
in-process state that a restart loses. Recovery is by reading `STATE.md`,
|
|
30
|
+
`tracker.json`, the recovery journal, and the Report JSON off disk — the same
|
|
31
|
+
files a *different* model in either seat would read and write. That is what
|
|
32
|
+
makes a seat swap safe: the model is not the thing being recovered, the
|
|
33
|
+
files are.
|
|
34
|
+
|
|
35
|
+
**What is invariant across a seat swap** — the two things a human or a
|
|
36
|
+
downstream tool actually depends on:
|
|
37
|
+
|
|
38
|
+
- **The Report JSON** — the wave scheduler's output contract (`phase`,
|
|
39
|
+
`wave_id`, `items_selected`/`items_shipped`, `blocked`,
|
|
40
|
+
`orchestrator_gate`, …, documented in full in `driver/wave_scheduler.py`'s
|
|
41
|
+
module docstring). Swapping either seat adds *at most* two well-known
|
|
42
|
+
optional keys (`orchestrator_review`, per-item `final_catch`) and changes
|
|
43
|
+
no existing key's shape.
|
|
44
|
+
- **The state layer** — `STATE.md`, `tracker.json`, receipts, and the
|
|
45
|
+
recovery journal. Same file names, same key sets, regardless of which
|
|
46
|
+
model is deciding.
|
|
47
|
+
|
|
48
|
+
What is **not** invariant, and isn't meant to be: which model made a given
|
|
49
|
+
decision, and (for the orchestrator seat) whether a decision was even routed
|
|
50
|
+
through an API model at all versus made by the live harness. That is exactly
|
|
51
|
+
the part that's supposed to change when you swap a seat.
|
|
52
|
+
|
|
53
|
+
---
|
|
54
|
+
|
|
55
|
+
## The two seats
|
|
56
|
+
|
|
57
|
+
### Worker seat — `AgentDriver`
|
|
58
|
+
|
|
59
|
+
Selected by `seats.worker` in `aesop.config.json`. Concrete backends today:
|
|
60
|
+
|
|
61
|
+
| `backend` | What it is | Notes |
|
|
62
|
+
|---|---|---|
|
|
63
|
+
| `claude` (default) | Claude Code CLI harness | No API key; two ops run as concrete Python, three are serviced by the harness itself |
|
|
64
|
+
| `codex` | OpenAI Chat Completions | Requires a `json_schema`-capable model (`gpt-4o`, `gpt-4o-mini`, `gpt-4-turbo`; `gpt-3.5-turbo` is rejected at construction unless you pass `allow_unverified_models=True`) |
|
|
65
|
+
| `openai-compatible` | Any OpenAI-compatible HTTP endpoint | Ollama, OpenRouter, Together, etc. — **requires `base_url`** |
|
|
66
|
+
|
|
67
|
+
**Live wiring**: `driver/wave_scheduler.py`'s `resolve_worker_driver()` reads
|
|
68
|
+
`seats.worker` from `aesop.config.json` and calls `build_driver()` on it
|
|
69
|
+
(`driver/backend_config.py`). `--driver claude|codex` on the CLI overrides
|
|
70
|
+
the config. No `seats` block, or a bare legacy flat `{"backend": ...}` block
|
|
71
|
+
with no `seats` wrapper, keeps behavior byte-identical to a pre-0.4.0
|
|
72
|
+
install: `ClaudeCodeDriver`, no key needed. (The legacy flat block still
|
|
73
|
+
*parses*, and direct `build_driver()` callers still honor it — it is only
|
|
74
|
+
*inert* in the scheduler's default dispatch path, so migrate it into
|
|
75
|
+
`seats.worker` to actually activate it.)
|
|
76
|
+
|
|
77
|
+
### Orchestrator seat — `OrchestratorBackend`
|
|
78
|
+
|
|
79
|
+
Selected by `seats.orchestrator`. This is the decision seat: the thing that
|
|
80
|
+
calls `OrchestratorDriver.decide()` to produce a structured verdict.
|
|
81
|
+
|
|
82
|
+
| `backend` | What it is |
|
|
83
|
+
|---|---|
|
|
84
|
+
| `harness` (default, also accepts `"claude"`) | The **null** `HarnessOrchestratorBackend`. `decide_call()` raises on purpose — there is no Python code path that "calls" the harness. This is the honest way of saying: the live Claude Code session driving this loop IS the orchestrator seat, and no swapped backend exists. |
|
|
85
|
+
| `openai-compatible` | `OpenAICompatibleOrchestratorBackend` — a real OpenAI-compatible HTTP call, same `model`/`base_url`/`api_key_env`/`is_local` shape as the worker seat, plus `timeout_s`. `base_url` is optional here (defaults to the hosted OpenAI endpoint) — unlike the worker seat, which requires it. |
|
|
86
|
+
|
|
87
|
+
**Live wiring**: `driver/wave_loop.py`'s `run_wave(..., orchestrator_backend=...)`
|
|
88
|
+
Phase 6 — the pre-ship gate — routes **one `final_catch` decision per
|
|
89
|
+
test-verified item** through the configured backend when one is live;
|
|
90
|
+
`driver/wave_scheduler.py`'s `resolve_orchestrator_backend()` is what builds
|
|
91
|
+
that backend from `seats.orchestrator` and passes it through. With no
|
|
92
|
+
`seats.orchestrator` block (or `backend: "harness"`/`"claude"`),
|
|
93
|
+
`resolve_orchestrator_backend()` returns `None` and Phase 6 stays exactly
|
|
94
|
+
what it was pre-HS-2: `adversarial_review = "deferred"`, no
|
|
95
|
+
`orchestrator_review` key, no OpenAI backend constructed, no key required.
|
|
96
|
+
|
|
97
|
+
Be precise about scope here, because it's easy to overstate: Phase 6's
|
|
98
|
+
`final_catch` is the **only** decision point HS-2 wired to the seat. Backlog
|
|
99
|
+
ranking, in-session adjudication, and PR merges are still made by the live
|
|
100
|
+
harness directly — see [What's proven vs bounded](#whats-proven-vs-bounded).
|
|
101
|
+
|
|
102
|
+
---
|
|
103
|
+
|
|
104
|
+
## The invariant boundary
|
|
105
|
+
|
|
106
|
+
The claim "the Report JSON and state layer are unchanged across a seat
|
|
107
|
+
swap" is not just asserted in this doc — it's a committed, offline,
|
|
108
|
+
automated proof: `tests/test_hs2_swap_proof.py`. It drives the **same task**
|
|
109
|
+
through the public scheduler path twice — once with the default harness
|
|
110
|
+
orchestrator seat, once with a swapped `FakeOrchestratorBackend` — both on a
|
|
111
|
+
non-Claude fake worker seat, and asserts:
|
|
112
|
+
|
|
113
|
+
- Identical Report JSON key sets, top-level and per-item.
|
|
114
|
+
- Identical values for `slug`/`backend`/`tier`/`verified`/`testExit`.
|
|
115
|
+
- Identical tracker terminal state (`in_progress`) and structure.
|
|
116
|
+
- Identical journal file names and entry key sets.
|
|
117
|
+
- The swapped backend demonstrably decided (`call_count == 1`).
|
|
118
|
+
|
|
119
|
+
A companion **bounded live run** — `bench/results/hs2-swap-proof-2026-07-25.md`
|
|
120
|
+
/ `.json` — drove one real task (fix a broken `multiply`) through `run_wave`
|
|
121
|
+
twice against a live codex (gpt-4o-mini) worker seat: arm A the default
|
|
122
|
+
harness orchestrator seat, arm B `seats.orchestrator` = openai-compatible
|
|
123
|
+
gpt-4o-mini. Both arms dispatched, test-exit 0, `verified: True`; arm B's
|
|
124
|
+
gpt-4o-mini seat returned a schema-valid `final_catch` verdict (`merge`,
|
|
125
|
+
with evidence + confidence) on the first attempt. Result shape was
|
|
126
|
+
invariant modulo exactly the two documented opt-in keys
|
|
127
|
+
(`orchestrator_review`, `final_catch`). Total spend: 3 gpt-4o-mini calls
|
|
128
|
+
(~1.1k worker tokens + one decision call), well under the run's US$2 cap.
|
|
129
|
+
`git=None` — the live proof never shipped anything.
|
|
130
|
+
|
|
131
|
+
That is the whole evidentiary basis for "the seam is real and the boundary
|
|
132
|
+
holds": one offline proof covering the general mechanism plus one small,
|
|
133
|
+
real, bounded live run. See [Bounds](#whats-proven-vs-bounded) for what that
|
|
134
|
+
does and does not establish.
|
|
135
|
+
|
|
136
|
+
---
|
|
137
|
+
|
|
138
|
+
## How to swap a seat
|
|
139
|
+
|
|
140
|
+
Everything lives in one namespaced block, `seats`, in `aesop.config.json`:
|
|
141
|
+
|
|
142
|
+
```json
|
|
143
|
+
{
|
|
144
|
+
"seats": {
|
|
145
|
+
"worker": { "backend": "claude" },
|
|
146
|
+
"orchestrator": { "backend": "harness" }
|
|
147
|
+
}
|
|
148
|
+
}
|
|
149
|
+
```
|
|
150
|
+
|
|
151
|
+
That's the *default* — writing it explicitly changes nothing, and deleting
|
|
152
|
+
it changes nothing either. To swap a seat, replace its block. See the
|
|
153
|
+
[quickstart](#swap-a-seats-model-in-60-seconds) below for copy-paste
|
|
154
|
+
examples.
|
|
155
|
+
|
|
156
|
+
Fields, common to both seats' `openai-compatible` backend:
|
|
157
|
+
|
|
158
|
+
| Field | Required? | Meaning |
|
|
159
|
+
|---|---|---|
|
|
160
|
+
| `backend` | yes | `"claude"` / `"codex"` / `"openai-compatible"` (worker); `"harness"` / `"claude"` / `"openai-compatible"` (orchestrator) |
|
|
161
|
+
| `model` | required for `codex` and `openai-compatible` | Model id |
|
|
162
|
+
| `base_url` | **required** for worker `openai-compatible`; optional for orchestrator (defaults to `https://api.openai.com/v1`) | The HTTP endpoint |
|
|
163
|
+
| `api_key_env` | optional | Env var name holding the API key — read at **call time**, never stored in the config |
|
|
164
|
+
| `is_local` | optional | See below |
|
|
165
|
+
|
|
166
|
+
### SECURITY notes (read these before pointing a seat at a real endpoint)
|
|
167
|
+
|
|
168
|
+
- **`api_key_env` is a heuristic allowlist, not a guarantee.** Known
|
|
169
|
+
LLM-provider names (`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`,
|
|
170
|
+
`OPENROUTER_API_KEY`, `TOGETHER_API_KEY`, `GROQ_API_KEY`,
|
|
171
|
+
`MISTRAL_API_KEY`, `DEEPSEEK_API_KEY`, `FIREWORKS_API_KEY`,
|
|
172
|
+
`OLLAMA_API_KEY`, `AZURE_OPENAI_API_KEY`, `GOOGLE_API_KEY`) pass
|
|
173
|
+
**silently**. A name that doesn't look like a key env var, or that
|
|
174
|
+
contains an obvious non-LLM secret fragment (`SECRET`, `TOKEN`,
|
|
175
|
+
`PASSWORD`, `PASSWD`, `CREDENTIAL`, `PRIVATE`, `ACCESS`, `SESSION`,
|
|
176
|
+
`COOKIE`, `SIGNING`), is **hard-rejected**. Any other key-shaped name
|
|
177
|
+
(a custom LLM gateway) is **allowed but prints a loud `NOTICE` to
|
|
178
|
+
stderr** naming the risk: its value *will* be sent as a Bearer token
|
|
179
|
+
to your configured `base_url`. This is deliberately best-effort — no
|
|
180
|
+
name check can prove an env var actually holds an LLM key. The NOTICE
|
|
181
|
+
is the real signal; review it whenever a non-provider name appears.
|
|
182
|
+
- **`is_local: true` requires a loopback `base_url`** (`localhost`,
|
|
183
|
+
`127.0.0.1`, or `::1`) — construction rejects it otherwise. `is_local`
|
|
184
|
+
waives the API-key requirement (a dummy `local-only` Bearer is sent
|
|
185
|
+
instead), so it must be pinned to loopback: `is_local` plus a remote
|
|
186
|
+
`base_url` would ship your prompt content to an arbitrary host with no
|
|
187
|
+
key needed at all.
|
|
188
|
+
- **`base_url` is SSRF-validated** (`driver/backend_config.py`'s
|
|
189
|
+
`validate_base_url`): scheme must be `http`/`https`, no embedded
|
|
190
|
+
credentials, and both IP literals *and* DNS-resolved hostnames are
|
|
191
|
+
checked against private/loopback/link-local/reserved ranges (including
|
|
192
|
+
IPv4-mapped IPv6 forms like `::ffff:169.254.169.254`, which would
|
|
193
|
+
otherwise bypass the IPv4 checks, and the `169.254.169.254` cloud
|
|
194
|
+
metadata address). **Residual, documented, not closed**: this is a
|
|
195
|
+
load/construct-time check with a bounded (5s) DNS resolution. A
|
|
196
|
+
TTL-0 DNS-rebinding attacker can pass validation and then re-point the
|
|
197
|
+
name at a private address before the actual HTTP call; closing that
|
|
198
|
+
fully requires connection-time address pinning in the transport, which
|
|
199
|
+
this does not do. An unresolvable hostname is allowed through (offline
|
|
200
|
+
config loading must not fail) and the eventual connection just fails
|
|
201
|
+
on its own.
|
|
202
|
+
|
|
203
|
+
---
|
|
204
|
+
|
|
205
|
+
## What's PROVEN vs BOUNDED
|
|
206
|
+
|
|
207
|
+
Scrupulously, so nothing here gets over-read:
|
|
208
|
+
|
|
209
|
+
### PROVEN
|
|
210
|
+
|
|
211
|
+
- **Worker swap is live.** `resolve_worker_driver()` builds a real,
|
|
212
|
+
configured `AgentDriver` from `seats.worker` and the scheduler dispatches
|
|
213
|
+
through it; codex and openai-compatible backends have offline test
|
|
214
|
+
coverage (`tests/test_codex_driver_e2e.py`, `tests/test_seats_config.py`).
|
|
215
|
+
- **Orchestrator swap is live** (HS-2). `run_wave`'s Phase 6 routes a real
|
|
216
|
+
`final_catch` decision through a configured `OrchestratorBackend` when one
|
|
217
|
+
is present; verdict has real effect (`block` stops the ship and
|
|
218
|
+
quarantines files; `merge` approves).
|
|
219
|
+
- **Swap transparency is proven** two ways: an offline test suite
|
|
220
|
+
(`tests/test_hs2_swap_proof.py`) establishing the no-op invariant and the
|
|
221
|
+
end-to-end Report/state shape invariance, *and* one bounded live run
|
|
222
|
+
(both seats real: worker + orchestrator gpt-4o-mini) where both arms went
|
|
223
|
+
green with an invariant result shape. **Small N** — see below.
|
|
224
|
+
|
|
225
|
+
### BOUNDED / NOT yet claimed
|
|
226
|
+
|
|
227
|
+
- **The proofs are small-N.** The offline swap-transparency test is one
|
|
228
|
+
synthetic task on a fake backend; the live run is one real task, one
|
|
229
|
+
model (gpt-4o-mini), one repeat. This proves the *plumbing* — config to
|
|
230
|
+
seat to real API call to schema-valid verdict to recorded effect — not
|
|
231
|
+
decision **quality**, and not anything at scale.
|
|
232
|
+
- **Seat decision quality is a separate, ongoing question**, studied
|
|
233
|
+
in the shadow-adjudication bench line (`bench/README.md`,
|
|
234
|
+
`tools/seated_shadow_adjudication.py`). The headline finding there —
|
|
235
|
+
**"Context at the Seam"**: decontextualized adjudication (facts only, no
|
|
236
|
+
file brain) leaves both a frontier model (gpt-5.6-sol) and a cheaper
|
|
237
|
+
model (gpt-5.5) abstaining (`undetermined`, ~80–100% of runs) on the
|
|
238
|
+
hardest synthesis-heavy corpus item (item 9, a whitelist-gate-weakening
|
|
239
|
+
false positive that requires chaining "health check is top-level only" +
|
|
240
|
+
"secret_scan.py scans recursively" into "no real coverage gap"). Giving
|
|
241
|
+
the *same* models **real seated context** (actual STATE.md/tracker.json/
|
|
242
|
+
BUILDLOG.md plus the real cited code) flips both models to the correct
|
|
243
|
+
`false_positive` verdict, **stable across repeated runs and across two
|
|
244
|
+
independently-run corpus variants** (`bench/results/SEATED-AB-2026-07-24.md`,
|
|
245
|
+
`bench/results/seated-redo-2026-07-24-gpt-5_6-sol_repeat3.md`,
|
|
246
|
+
`bench/results/seated-redo-2026-07-24--neutral-seated-sol_repeat3.md`).
|
|
247
|
+
This result is **robust in the sense that matters for this doc**: it
|
|
248
|
+
reproduced after an earlier round of this same bench line caught and fixed
|
|
249
|
+
a context-leak/seam confound (a dropped-prompt / schema-mismatch bug where
|
|
250
|
+
"seated" runs weren't actually receiving the real context they claimed
|
|
251
|
+
to) — the cited results here are the **re-verified**, post-fix runs, not
|
|
252
|
+
the confounded ones. Bound: N=3 per model, one item, two OpenAI-family
|
|
253
|
+
models, not a full-corpus or cross-lab claim.
|
|
254
|
+
- **Some orchestrator decisions are not routed through the seat at all.**
|
|
255
|
+
Backlog ranking, PR merges, and in-session adjudication by the live
|
|
256
|
+
harness are made outside `run_wave`'s Phase 6 and are unaffected by HS-2.
|
|
257
|
+
The seat swap covers exactly one gate: the pre-ship `final_catch` check.
|
|
258
|
+
- **`wave_loop.py`'s standalone `--manifest` CLI still hardcodes
|
|
259
|
+
`ClaudeCodeDriver`** and does not read `aesop.config.json` at all. The
|
|
260
|
+
config-driven entry point is `driver/wave_scheduler.py` — that is where
|
|
261
|
+
`seats` actually takes effect. If you invoke `wave_loop.py` directly by
|
|
262
|
+
its manifest CLI, no seat swap applies.
|
|
263
|
+
- **Repair stays mechanical on both seats by design.** Swapping who
|
|
264
|
+
decides `final_catch` never changes the bounded-retry repair semantics.
|
|
265
|
+
|
|
266
|
+
---
|
|
267
|
+
|
|
268
|
+
## The block gate
|
|
269
|
+
|
|
270
|
+
`run_wave`'s Phase 6 (`_orchestrator_final_catch` in `driver/wave_loop.py`)
|
|
271
|
+
reviews every **test-verified** item — a failed item never reaches the
|
|
272
|
+
seat at all — and acts on the verdict:
|
|
273
|
+
|
|
274
|
+
| Verdict | Effect |
|
|
275
|
+
|---|---|
|
|
276
|
+
| `merge` | Approved; ships exactly as it would with the default harness seat. |
|
|
277
|
+
| `block` | `verified` is flipped to `False`; the item does **not** ship; the recovery journal is rewritten so a resume can't skip-and-ship it; and the item is marked **terminal** (tracker status `blocked`, never re-selected) and visible in the Report (`Report.blocked: [{slug, reason, quarantine}]`). |
|
|
278
|
+
| `escalate` / `undetermined` / `DECISION_FAILED` | Degrades to today's default behavior: ships to branch (merge stays manual downstream) with an honest per-item record. **A seat outage never fabricates a verdict and never blocks a test-proven item** — this is crash-only degradation, not silent failure. |
|
|
279
|
+
|
|
280
|
+
**Quarantine on block**: a blocked item's already-written files are
|
|
281
|
+
restored to their pre-build state — `git checkout --` for tracked files,
|
|
282
|
+
delete for untracked ones — so refused code doesn't linger in the working
|
|
283
|
+
tree for a later `git add -A` to accidentally ship. This acts on **file
|
|
284
|
+
paths only**: empty strings, `.`/`..`, and directory entries are rejected
|
|
285
|
+
with a per-file error record rather than acted on, specifically because a
|
|
286
|
+
directory or dot pathspec would revert *other* items' uncommitted verified
|
|
287
|
+
work, not just the blocked item's. An ambiguous untracked/tracked
|
|
288
|
+
determination (e.g. a git index lock) never deletes — fail-safe, not
|
|
289
|
+
fail-delete.
|
|
290
|
+
|
|
291
|
+
**Gate visibility**: if every single decision on a wave came back
|
|
292
|
+
`DECISION_FAILED`, the wave-level `orchestrator_review.gate_status` is
|
|
293
|
+
`"degraded"` (not `"active"`) — a 100%-failing seat is not allowed to look
|
|
294
|
+
like an approving one, even though ship semantics for already-verified
|
|
295
|
+
items are unaffected.
|
|
296
|
+
|
|
297
|
+
---
|
|
298
|
+
|
|
299
|
+
## Honest engineering note
|
|
300
|
+
|
|
301
|
+
This seam did not work correctly on the first attempt at wiring it. Early
|
|
302
|
+
rounds of the shadow-adjudication bench line ran through a shim that
|
|
303
|
+
dropped the prompt before it reached the model, and a schema/prompt
|
|
304
|
+
mismatch meant "seated" runs weren't actually seeing the context they
|
|
305
|
+
claimed to have — green test output, wrong conclusion (**green is not
|
|
306
|
+
correct**). Each seat — worker and orchestrator — cleared **two rounds of
|
|
307
|
+
adversarial audit** before the swap-transparency claims in this doc were
|
|
308
|
+
considered solid enough to write down. The context-leak confound described
|
|
309
|
+
above under [Bounds](#whats-proven-vs-bounded) is exactly this class of
|
|
310
|
+
bug, caught, fixed, and re-verified before being cited here.
|
|
311
|
+
|
|
312
|
+
---
|
|
313
|
+
|
|
314
|
+
## Swap a seat's model in 60 seconds
|
|
315
|
+
|
|
316
|
+
Every config block below was verified offline against this repo's actual
|
|
317
|
+
loader — `load_backend_config()` → `build_driver()` /
|
|
318
|
+
`build_orchestrator_backend()` — with no network call and no API key
|
|
319
|
+
(construction is always offline-safe; keys are read at call time only).
|
|
320
|
+
Copy, paste, done.
|
|
321
|
+
|
|
322
|
+
### Worker seat
|
|
323
|
+
|
|
324
|
+
**Ollama, local, no key:**
|
|
325
|
+
|
|
326
|
+
```bash
|
|
327
|
+
ollama serve # in one terminal
|
|
328
|
+
ollama pull mistral # in another
|
|
329
|
+
```
|
|
330
|
+
|
|
331
|
+
```json
|
|
332
|
+
{
|
|
333
|
+
"seats": {
|
|
334
|
+
"worker": {
|
|
335
|
+
"backend": "openai-compatible",
|
|
336
|
+
"base_url": "http://localhost:11434/v1",
|
|
337
|
+
"model": "mistral",
|
|
338
|
+
"is_local": true
|
|
339
|
+
}
|
|
340
|
+
}
|
|
341
|
+
}
|
|
342
|
+
```
|
|
343
|
+
|
|
344
|
+
No env var needed — `is_local: true` sends a dummy `local-only` Bearer.
|
|
345
|
+
Verification tier: 3 (heaviest — small local models get the most checking).
|
|
346
|
+
|
|
347
|
+
**OpenRouter, hosted:**
|
|
348
|
+
|
|
349
|
+
```bash
|
|
350
|
+
export OPENROUTER_API_KEY=sk-or-...
|
|
351
|
+
```
|
|
352
|
+
|
|
353
|
+
```json
|
|
354
|
+
{
|
|
355
|
+
"seats": {
|
|
356
|
+
"worker": {
|
|
357
|
+
"backend": "openai-compatible",
|
|
358
|
+
"base_url": "https://openrouter.ai/api/v1",
|
|
359
|
+
"model": "openai/gpt-4-turbo",
|
|
360
|
+
"api_key_env": "OPENROUTER_API_KEY"
|
|
361
|
+
}
|
|
362
|
+
}
|
|
363
|
+
}
|
|
364
|
+
```
|
|
365
|
+
|
|
366
|
+
Verification tier: 2 (hosted, ~0.92 accuracy assumption).
|
|
367
|
+
|
|
368
|
+
**Codex (OpenAI Chat Completions), hosted:**
|
|
369
|
+
|
|
370
|
+
```bash
|
|
371
|
+
export OPENAI_API_KEY=sk-...
|
|
372
|
+
```
|
|
373
|
+
|
|
374
|
+
```json
|
|
375
|
+
{
|
|
376
|
+
"seats": {
|
|
377
|
+
"worker": {
|
|
378
|
+
"backend": "codex",
|
|
379
|
+
"model": "gpt-4o-mini"
|
|
380
|
+
}
|
|
381
|
+
}
|
|
382
|
+
}
|
|
383
|
+
```
|
|
384
|
+
|
|
385
|
+
`model` must support `response_format: json_schema` — `gpt-4o`,
|
|
386
|
+
`gpt-4o-mini`, and `gpt-4-turbo` are the known-capable set; anything else
|
|
387
|
+
(including `gpt-3.5-turbo`) is rejected at construction unless you pass
|
|
388
|
+
`allow_unverified_models=True` yourself. Verification tier: 2.
|
|
389
|
+
|
|
390
|
+
### Orchestrator seat
|
|
391
|
+
|
|
392
|
+
**OpenAI hosted (`gpt-4o-mini`) as the decision seat:**
|
|
393
|
+
|
|
394
|
+
```bash
|
|
395
|
+
export OPENAI_API_KEY=sk-...
|
|
396
|
+
```
|
|
397
|
+
|
|
398
|
+
```json
|
|
399
|
+
{
|
|
400
|
+
"seats": {
|
|
401
|
+
"orchestrator": {
|
|
402
|
+
"backend": "openai-compatible",
|
|
403
|
+
"model": "gpt-4o-mini",
|
|
404
|
+
"api_key_env": "OPENAI_API_KEY"
|
|
405
|
+
}
|
|
406
|
+
}
|
|
407
|
+
}
|
|
408
|
+
```
|
|
409
|
+
|
|
410
|
+
Note `base_url` is omitted — the orchestrator seat defaults to
|
|
411
|
+
`https://api.openai.com/v1` (the worker seat has no such default; it
|
|
412
|
+
requires `base_url` explicitly).
|
|
413
|
+
|
|
414
|
+
**Ollama, local, no key, as the decision seat:**
|
|
415
|
+
|
|
416
|
+
```bash
|
|
417
|
+
ollama serve
|
|
418
|
+
ollama pull mistral
|
|
419
|
+
```
|
|
420
|
+
|
|
421
|
+
```json
|
|
422
|
+
{
|
|
423
|
+
"seats": {
|
|
424
|
+
"orchestrator": {
|
|
425
|
+
"backend": "openai-compatible",
|
|
426
|
+
"base_url": "http://localhost:11434/v1",
|
|
427
|
+
"model": "mistral",
|
|
428
|
+
"is_local": true
|
|
429
|
+
}
|
|
430
|
+
}
|
|
431
|
+
}
|
|
432
|
+
```
|
|
433
|
+
|
|
434
|
+
**Combine any worker + any orchestrator block** under one `seats` key —
|
|
435
|
+
they're independent. Running `driver/wave_scheduler.py --execute` against a
|
|
436
|
+
hosted (non-`is_local`) seat requires that seat's `api_key_env` to be set;
|
|
437
|
+
`--dry-run` never needs a key, because building a driver or backend is
|
|
438
|
+
always offline-safe.
|
|
439
|
+
|
|
440
|
+
---
|
|
441
|
+
|
|
442
|
+
## See also
|
|
443
|
+
|
|
444
|
+
- [docs/INSTALL.md](INSTALL.md) — "Using Non-Claude Backends" section: setup
|
|
445
|
+
prerequisites, verification-tier table, troubleshooting.
|
|
446
|
+
- `driver/CLAUDE.md` — the full technical contract for both seats.
|
|
447
|
+
- `bench/README.md` — the held-out benchmark measuring *quality*, separate
|
|
448
|
+
from the plumbing this doc covers.
|
|
449
|
+
- `bench/results/hs2-swap-proof-2026-07-25.md` — the bounded live
|
|
450
|
+
swap-transparency run.
|
|
451
|
+
- `bench/results/SEATED-AB-2026-07-24.md` — the seated-context adjudication
|
|
452
|
+
finding.
|
package/docs/PORTING.md
CHANGED
|
@@ -2,6 +2,10 @@
|
|
|
2
2
|
|
|
3
3
|
Guide for adopters porting the orchestration harness to a foreign repo. Step-by-step with prerequisites, scaffold, config, and the 10 likeliest failure modes from real deployments.
|
|
4
4
|
|
|
5
|
+
**Quick decision**:
|
|
6
|
+
- **New harness directory?** See [INSTALL.md](INSTALL.md) for the quick 5-minute scaffold.
|
|
7
|
+
- **Existing codebase?** You're in the right place. This guide covers integrating Aesop into your project with common pitfalls.
|
|
8
|
+
|
|
5
9
|
---
|
|
6
10
|
|
|
7
11
|
## Prerequisites
|
package/docs/README.md
CHANGED
|
@@ -77,7 +77,7 @@ The `/buildsystem` skill runs **one complete iteration of the autonomous deliver
|
|
|
77
77
|
This is the repeatable loop that runs your delivery cycle indefinitely, with each wave learning from the prior audit. You can run `/buildsystem` once per wave (typically 30 min–2 hours depending on backlog size). See [HOW-THE-LOOP-WORKS.md](HOW-THE-LOOP-WORKS.md) for a concrete walkthrough.
|
|
78
78
|
|
|
79
79
|
### Team State & Multi-Instance Design
|
|
80
|
-
**Current Status (0.
|
|
80
|
+
**Current Status (0.4.0)**: Single-instance proven, with swappable worker + orchestrator model backends (Claude, Codex, OpenAI-compatible). See [MICROKERNEL.md](MICROKERNEL.md) for the two-seat architecture + a 60-second model-swap quickstart. State is durably checkpointed in git (STATE.md, BUILDLOG.md, tracker.json exports).
|
|
81
81
|
|
|
82
82
|
**In Design**: Multi-instance coordination via the state_store substrate. The event-sourced SQLite layer is production-ready but currently opt-in. A future release will enable multiple Aesop instances (e.g., per-team subgroups or geographic regions) to coordinate around a single source of truth—a Postgres-backed event log, with git as a diffable export. See [TEAM-STATE.md](TEAM-STATE.md) (design in progress) for the vision and current architecture decisions.
|
|
83
83
|
|
|
@@ -89,6 +89,7 @@ Once you've completed the adopter journey, use these for operational reference:
|
|
|
89
89
|
|
|
90
90
|
### Wave Cycle & Orchestration
|
|
91
91
|
- **[HOW-THE-LOOP-WORKS.md](HOW-THE-LOOP-WORKS.md)** — Concrete walkthrough of one complete `/buildsystem` wave cycle (rank → fan-out → verify → merge → close)
|
|
92
|
+
- **[MICROKERNEL.md](MICROKERNEL.md)** — The two swappable seats (worker + orchestrator): what's invariant across a model swap, what's proven vs. bounded, and a 60-second quickstart
|
|
92
93
|
|
|
93
94
|
### Dispatch & Cost
|
|
94
95
|
- **[DISPATCH-MODEL.md](DISPATCH-MODEL.md)** — Haiku-first subagent dispatch, cost analysis, patterns (fan-out, sequential, hierarchical)
|
|
@@ -121,7 +122,7 @@ Once you've completed the adopter journey, use these for operational reference:
|
|
|
121
122
|
- **[av-resilience.md](av-resilience.md)** — Antivirus and behavioral-engine resilience patterns for reliable agent execution
|
|
122
123
|
|
|
123
124
|
### Lessons & Case Studies
|
|
124
|
-
- **[autonomous-swe.md](autonomous-swe.md)** —
|
|
125
|
+
- **[autonomous-swe.md](autonomous-swe.md)** — What "autonomous SWE" means here (a fleet running the wave loop under a human who owns the outward gates): the committed evidence behind each claim (held-out benchmark, verified audit, proven kill-switch, reproducible package), the 0.1.0-rc.1 baseline, 0.4.0 evolution (multi-model support), and the limits the project owns
|
|
125
126
|
- **[case-study-portfolio.md](case-study-portfolio.md)** — How Aesop built its own portfolio site; full audit trail and cost breakdown
|
|
126
127
|
- **[SCRIPTS-POLICY.md](SCRIPTS-POLICY.md)** — Local-only execution, shared script library (`~/scripts`), task-local vs. reusable heuristics
|
|
127
128
|
|
|
@@ -135,7 +136,7 @@ Once you've completed the adopter journey, use these for operational reference:
|
|
|
135
136
|
**I want to understand the cost model**
|
|
136
137
|
→ [DISPATCH-MODEL.md](DISPATCH-MODEL.md) or [HOW-THE-LOOP-WORKS.md](HOW-THE-LOOP-WORKS.md#why-its-fast--cheap)
|
|
137
138
|
|
|
138
|
-
**I want to know what's actually proven vs. claimed
|
|
139
|
+
**I want to know what's actually proven vs. claimed**
|
|
139
140
|
→ [autonomous-swe.md](autonomous-swe.md)
|
|
140
141
|
|
|
141
142
|
**I need to understand how state survives a crash**
|
|
@@ -144,6 +145,9 @@ Once you've completed the adopter journey, use these for operational reference:
|
|
|
144
145
|
**I want to understand multi-instance coordination**
|
|
145
146
|
→ [TEAM-STATE.md](TEAM-STATE.md)
|
|
146
147
|
|
|
148
|
+
**I want to swap the worker or orchestrator model (Ollama, OpenRouter, OpenAI...)**
|
|
149
|
+
→ [MICROKERNEL.md](MICROKERNEL.md)
|
|
150
|
+
|
|
147
151
|
**I'm reviewing a PR that changes orchestration behavior**
|
|
148
152
|
→ [BEHAVIORAL-PR-REVIEW.md](BEHAVIORAL-PR-REVIEW.md)
|
|
149
153
|
|