cool-workflow 0.1.81 → 0.1.83
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/plugin.json +2 -2
- package/.codex-plugin/plugin.json +4 -4
- package/README.md +125 -121
- package/apps/architecture-review/app.json +1 -1
- package/apps/architecture-review-fast/app.json +1 -1
- package/apps/end-to-end-golden-path/app.json +1 -1
- package/apps/pr-review-fix-ci/app.json +1 -1
- package/apps/release-cut/app.json +1 -1
- package/apps/research-synthesis/app.json +1 -1
- package/dist/candidate-scoring.js +20 -26
- package/dist/capability-core.js +80 -93
- package/dist/capability-registry.js +30 -3
- package/dist/cli.js +12 -1
- package/dist/commit.js +217 -204
- package/dist/coordinator/util.js +6 -9
- package/dist/dispatch.js +11 -3
- package/dist/doctor.js +153 -0
- package/dist/evidence-reasoning.js +4 -1
- package/dist/execution-backend/agent.js +11 -48
- package/dist/execution-backend.js +11 -31
- package/dist/gates.js +48 -0
- package/dist/mcp-server.js +11 -0
- package/dist/multi-agent/helpers.js +6 -10
- package/dist/multi-agent/ids.js +20 -0
- package/dist/multi-agent-eval.js +27 -1
- package/dist/multi-agent-host.js +53 -21
- package/dist/multi-agent-operator-ux.js +2 -1
- package/dist/multi-agent-trust.js +5 -5
- package/dist/node-projection.js +59 -0
- package/dist/node-snapshot.js +8 -18
- package/dist/orchestrator/lifecycle-operations.js +22 -1
- package/dist/orchestrator.js +29 -2
- package/dist/reclamation/hash.js +72 -0
- package/dist/reclamation.js +31 -112
- package/dist/run-registry/queue.js +6 -7
- package/dist/run-registry.js +35 -24
- package/dist/scheduler.js +112 -57
- package/dist/topology.js +25 -4
- package/dist/trust-audit.js +70 -38
- package/dist/validation.js +328 -0
- package/dist/version.js +1 -1
- package/dist/worker-isolation.js +163 -58
- package/docs/agent-delegation-drive.7.md +90 -85
- package/docs/agent-framework.md +33 -32
- package/docs/candidate-scoring.7.md +26 -24
- package/docs/canonical-workflow-apps.7.md +40 -40
- package/docs/capability-topology-registry.7.md +24 -24
- package/docs/cli-mcp-parity.7.md +227 -154
- package/docs/contract-migration-tooling.7.md +49 -41
- package/docs/control-plane-scheduling.7.md +45 -40
- package/docs/coordinator-blackboard.7.md +30 -30
- package/docs/dogfood-one-real-repo.7.md +44 -44
- package/docs/durable-state-and-locking.7.md +35 -30
- package/docs/end-to-end-golden-path.7.md +29 -29
- package/docs/error-feedback.7.md +27 -27
- package/docs/evidence-adoption-reasoning-chain.7.md +63 -58
- package/docs/execution-backends.7.md +84 -79
- package/docs/getting-started.md +35 -18
- package/docs/index.md +3 -3
- package/docs/mcp-app-surface.7.md +64 -64
- package/docs/multi-agent-cli-mcp-surface.7.md +83 -77
- package/docs/multi-agent-eval-replay-harness.7.md +59 -54
- package/docs/multi-agent-operator-ux.7.md +70 -65
- package/docs/multi-agent-runtime-core.7.md +39 -39
- package/docs/multi-agent-topologies.7.md +24 -24
- package/docs/multi-agent-trust-policy-audit.7.md +38 -38
- package/docs/node-snapshot-diff-replay.7.md +26 -21
- package/docs/observability-cost-accounting.7.md +50 -45
- package/docs/operator-ux.7.md +30 -30
- package/docs/pipeline-runner.7.md +31 -31
- package/docs/project-index.md +15 -6
- package/docs/real-execution-backends.7.md +47 -42
- package/docs/release-and-migration.7.md +43 -38
- package/docs/release-tooling.7.md +74 -39
- package/docs/routines.md +16 -16
- package/docs/run-registry-control-plane.7.md +120 -115
- package/docs/run-retention-reclamation.7.md +46 -41
- package/docs/sandbox-profiles.7.md +32 -32
- package/docs/scheduled-tasks.md +14 -14
- package/docs/security-trust-hardening.7.md +29 -29
- package/docs/source-context-profiles.7.md +28 -28
- package/docs/state-explosion-management.7.md +64 -59
- package/docs/state-node.7.md +8 -8
- package/docs/team-collaboration.7.md +63 -58
- package/docs/trust-model.md +126 -126
- package/docs/unix-principles.md +80 -80
- package/docs/vendor-manifest-loadability.7.md +20 -20
- package/docs/verifier-gated-commit.7.md +16 -16
- package/docs/web-desktop-workbench.7.md +70 -65
- package/docs/worker-isolation.7.md +34 -37
- package/docs/workflow-app-framework.7.md +38 -38
- package/manifest/plugin.manifest.json +4 -4
- package/manifest/source-context-profiles.json +1 -1
- package/package.json +3 -2
- package/scripts/canonical-apps.js +4 -4
- package/scripts/children/batch-delegate-child.js +58 -0
- package/scripts/children/http-delegate-child.js +39 -0
- package/scripts/dogfood-release.js +1 -1
- package/scripts/gen-parity-doc.js +106 -0
- package/scripts/golden-path.js +4 -4
- package/scripts/release-flow.js +181 -5
- package/dist/verifier-registry.js +0 -46
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Multi-Agent CLI + MCP Surface
|
|
2
2
|
|
|
3
|
-
CW v0.1.20 adds the
|
|
3
|
+
CW v0.1.20 adds the best host-facing control loop for multi-agent work:
|
|
4
4
|
|
|
5
5
|
```text
|
|
6
6
|
multi-agent run -> status -> step -> blackboard -> score -> select
|
|
@@ -11,48 +11,48 @@ CW v0.1.25 extends this surface with State Explosion Management commands:
|
|
|
11
11
|
`multi-agent summarize`, and `multi-agent graph --view <view>` (with optional
|
|
12
12
|
`--focus <id>` and `--depth <n>`). Matching MCP tools are `cw_summary_refresh`,
|
|
13
13
|
`cw_summary_show`, `cw_blackboard_summarize`, `cw_multi_agent_summarize`, and
|
|
14
|
-
`cw_multi_agent_graph_compact`. All responses keep source refs and
|
|
15
|
-
|
|
14
|
+
`cw_multi_agent_graph_compact`. All responses keep source refs and hints for how
|
|
15
|
+
to open them up. See [state-explosion-management.7.md](state-explosion-management.7.md).
|
|
16
16
|
|
|
17
17
|
CW v0.1.26 adds `multi-agent reasoning <run-id> [--evidence <id>] [--refresh]`
|
|
18
|
-
(MCP: `cw_evidence_reasoning`, `cw_evidence_reasoning_refresh`), which
|
|
19
|
-
*why* each evidence item was
|
|
18
|
+
(MCP: `cw_evidence_reasoning`, `cw_evidence_reasoning_refresh`), which makes clear
|
|
19
|
+
*why* each evidence item was taken in, and an added `rationaleStatus` field on
|
|
20
20
|
`multi-agent evidence` rows. See
|
|
21
21
|
[evidence-adoption-reasoning-chain.7.md](evidence-adoption-reasoning-chain.7.md).
|
|
22
22
|
|
|
23
|
-
This is userland over the
|
|
24
|
-
multi-agent, blackboard, candidate, audit, and commit primitives
|
|
25
|
-
|
|
26
|
-
run.
|
|
23
|
+
This is userland over the kernel records that are already there. The low-level
|
|
24
|
+
topology, multi-agent, blackboard, candidate, audit, and commit primitives are
|
|
25
|
+
still there to use, but agent hosts should use this high-level surface when they
|
|
26
|
+
drive a run.
|
|
27
27
|
|
|
28
28
|
## CLI Loop
|
|
29
29
|
|
|
30
|
-
|
|
30
|
+
Make or join a topology-backed run without starting up workers:
|
|
31
31
|
|
|
32
32
|
```bash
|
|
33
33
|
node scripts/cw.js multi-agent run <run-id> --topology judge-panel --task <task-id>
|
|
34
34
|
node scripts/cw.js multi-agent run --app architecture-review --repo /path/to/repo --question "Review this" --topology map-reduce
|
|
35
35
|
```
|
|
36
36
|
|
|
37
|
-
Read the
|
|
37
|
+
Read the joined host status:
|
|
38
38
|
|
|
39
39
|
```bash
|
|
40
40
|
node scripts/cw.js multi-agent status <run-id>
|
|
41
41
|
node scripts/cw.js multi-agent status <run-id> --json
|
|
42
42
|
```
|
|
43
43
|
|
|
44
|
-
|
|
44
|
+
Do one deterministic step at a time:
|
|
45
45
|
|
|
46
46
|
```bash
|
|
47
47
|
node scripts/cw.js multi-agent step <run-id> --sandbox readonly
|
|
48
48
|
```
|
|
49
49
|
|
|
50
|
-
`step` may
|
|
51
|
-
register a candidate, score a candidate with
|
|
52
|
-
a scored candidate, or
|
|
53
|
-
|
|
50
|
+
`step` may make a dispatch manifest, get fanin, snapshot the blackboard,
|
|
51
|
+
register a candidate, score a candidate with verifier evidence that is already
|
|
52
|
+
there, select a scored candidate, or put forward the verifier-gated commit
|
|
53
|
+
command. It never starts up agents on its own.
|
|
54
54
|
|
|
55
|
-
Work with the active blackboard when it is
|
|
55
|
+
Work with the active blackboard when it is clear which one it is:
|
|
56
56
|
|
|
57
57
|
```bash
|
|
58
58
|
node scripts/cw.js multi-agent blackboard <run-id> summary
|
|
@@ -62,7 +62,7 @@ node scripts/cw.js multi-agent blackboard <run-id> add-artifact --topic <topic-i
|
|
|
62
62
|
node scripts/cw.js multi-agent blackboard <run-id> snapshot
|
|
63
63
|
```
|
|
64
64
|
|
|
65
|
-
Score and select
|
|
65
|
+
Score and select in a clear way:
|
|
66
66
|
|
|
67
67
|
```bash
|
|
68
68
|
node scripts/cw.js multi-agent score <run-id> <candidate-id> --criterion correctness=1 --criterion evidence=1 --evidence <ref>
|
|
@@ -72,7 +72,7 @@ node scripts/cw.js commit <run-id> --selection <selection-id> --reason "verified
|
|
|
72
72
|
|
|
73
73
|
## Operator Inspection
|
|
74
74
|
|
|
75
|
-
v0.1.21
|
|
75
|
+
v0.1.21 adds to the host loop these pointed operator commands:
|
|
76
76
|
|
|
77
77
|
```bash
|
|
78
78
|
node scripts/cw.js multi-agent graph <run-id>
|
|
@@ -81,13 +81,13 @@ node scripts/cw.js multi-agent failures <run-id>
|
|
|
81
81
|
node scripts/cw.js multi-agent evidence <run-id>
|
|
82
82
|
```
|
|
83
83
|
|
|
84
|
-
The human output is
|
|
85
|
-
or blocked agents,
|
|
84
|
+
The human output is short and ready to use: agent graph, dependencies, failed
|
|
85
|
+
or blocked agents, evidence taken in, missing evidence, and the next action.
|
|
86
86
|
Use `--json` or `--format json` for deterministic script output.
|
|
87
87
|
|
|
88
88
|
## MCP Tools
|
|
89
89
|
|
|
90
|
-
MCP hosts should
|
|
90
|
+
MCP hosts should pick:
|
|
91
91
|
|
|
92
92
|
- `cw_multi_agent_run`
|
|
93
93
|
- `cw_multi_agent_status`
|
|
@@ -101,11 +101,11 @@ MCP hosts should prefer:
|
|
|
101
101
|
- `cw_multi_agent_evidence`
|
|
102
102
|
|
|
103
103
|
The older `cw_multi_agent_*`, `cw_topology_*`, `cw_blackboard_*`, and
|
|
104
|
-
`cw_candidate_*` tools
|
|
104
|
+
`cw_candidate_*` tools are still there as deep primitives.
|
|
105
105
|
|
|
106
106
|
## Stable Responses
|
|
107
107
|
|
|
108
|
-
Every high-level response is JSON and
|
|
108
|
+
Every high-level response is JSON and has in it:
|
|
109
109
|
|
|
110
110
|
- `runId`
|
|
111
111
|
- active topology and multi-agent ids
|
|
@@ -114,28 +114,28 @@ Every high-level response is JSON and includes:
|
|
|
114
114
|
- `state`, `performed`, `nextAction`, and `nextActions`
|
|
115
115
|
- `blockedReasons`, `requiredHostAction`, and `evidenceRequirements`
|
|
116
116
|
- state, report, blackboard, audit, ranking, worker manifest, and result paths
|
|
117
|
-
-
|
|
117
|
+
- joined topology, multi-agent, multi-agent operator, blackboard, worker,
|
|
118
118
|
candidate, feedback, commit, and audit summaries
|
|
119
119
|
|
|
120
120
|
## Fail-Closed Rules
|
|
121
121
|
|
|
122
122
|
The host surface fails closed when:
|
|
123
123
|
|
|
124
|
-
- active topology or blackboard state
|
|
125
|
-
- a fanout
|
|
124
|
+
- it is not clear which active topology or blackboard state to use
|
|
125
|
+
- a fanout does not cover all the roles
|
|
126
126
|
- worker output has not been recorded
|
|
127
|
-
- fanin
|
|
127
|
+
- fanin is missing required evidence or blackboard links
|
|
128
128
|
- score evidence is missing
|
|
129
|
-
- selection
|
|
129
|
+
- selection is missing a score or verifier readiness
|
|
130
130
|
- a verifier-gated commit is not ready
|
|
131
131
|
|
|
132
132
|
## Smoke Coverage
|
|
133
133
|
|
|
134
134
|
`test/multi-agent-cli-mcp-surface-smoke.js` covers the full host loop over the
|
|
135
|
-
official `judge-panel` topology, CLI and MCP parity,
|
|
136
|
-
|
|
137
|
-
artifact/message linkage, audit provenance, and Operator UX
|
|
138
|
-
|
|
135
|
+
official `judge-panel` topology, CLI and MCP parity, the failure when the
|
|
136
|
+
topology is not clear, the failure when evidence is missing, score/select that
|
|
137
|
+
works, blackboard artifact/message linkage, audit provenance, and Operator UX
|
|
138
|
+
next actions. It is part of `npm test` and `npm run release:check`.
|
|
139
139
|
|
|
140
140
|
`test/multi-agent-operator-ux-smoke.js` covers the v0.1.21 graph,
|
|
141
141
|
dependencies, failures, evidence adoption, report output, and MCP parity.
|
|
@@ -145,51 +145,52 @@ role-policy, blackboard-write, message-provenance, judge-rationale,
|
|
|
145
145
|
policy-violation, report, audit provenance, and MCP parity surface.
|
|
146
146
|
|
|
147
147
|
`test/multi-agent-eval-replay-harness-smoke.js` covers the v0.1.24 eval/replay
|
|
148
|
-
commands and MCP tools: snapshot, replay, compare, score, gate, report, and
|
|
149
|
-
|
|
148
|
+
commands and MCP tools: snapshot, replay, compare, score, gate, report, and the
|
|
149
|
+
way it finds regressions under control.
|
|
150
150
|
## CLI ↔ MCP Parity (v0.1.27)
|
|
151
151
|
|
|
152
|
-
Every command and tool
|
|
153
|
-
registry (`src/capability-registry.ts`) and
|
|
154
|
-
so `cw <cmd> --json` and the matching `cw_<tool>` result
|
|
152
|
+
Every command and tool named above is declared in the v0.1.27 capability
|
|
153
|
+
registry (`src/capability-registry.ts`) and checked by `npm run parity:check`,
|
|
154
|
+
so `cw <cmd> --json` and the matching `cw_<tool>` result show one data source.
|
|
155
155
|
See [cli-mcp-parity.7.md](cli-mcp-parity.7.md).
|
|
156
156
|
|
|
157
157
|
## Run Registry / Control Plane (v0.1.28)
|
|
158
158
|
|
|
159
|
-
The runs
|
|
160
|
-
|
|
161
|
-
a fingerprinted, fail-closed index over the same per-run
|
|
162
|
-
source of truth. See [run-registry-control-plane.7.md](run-registry-control-plane.7.md).
|
|
159
|
+
The runs talked about here can be indexed, searched, taken up again, put in an
|
|
160
|
+
archive, and run again across repos by the v0.1.28 Run Registry / Control Plane,
|
|
161
|
+
which builds a fingerprinted, fail-closed index over the same per-run
|
|
162
|
+
`.cw/runs/<id>/state.json` source of truth. See [run-registry-control-plane.7.md](run-registry-control-plane.7.md).
|
|
163
163
|
|
|
164
164
|
## Execution Backends (v0.1.29)
|
|
165
165
|
|
|
166
|
-
v0.1.29
|
|
167
|
-
contract with
|
|
168
|
-
drivers,
|
|
169
|
-
`backend list|show|probe`. The result/evidence envelope
|
|
170
|
-
backends; the backend id + sandbox attestation are recorded as
|
|
171
|
-
surface is
|
|
166
|
+
v0.1.29 takes execution up into a pluggable driver layer: one narrow
|
|
167
|
+
`ExecutionBackend` contract with `node`/`bun`/`shell`/`container`/`remote`/`ci`
|
|
168
|
+
drivers that you can swap, picked by `--backend` (in parallel to `--sandbox`) and
|
|
169
|
+
looked at through `backend list|show|probe`. The result/evidence envelope has the
|
|
170
|
+
same schema across backends; the backend id + sandbox attestation are recorded as
|
|
171
|
+
provenance, so this surface is the same no matter which backend ran a run. See
|
|
172
172
|
[execution-backends.7.md](execution-backends.7.md).
|
|
173
173
|
## Web / Desktop Workbench (v0.1.30)
|
|
174
174
|
|
|
175
175
|
v0.1.30 adds the Web / Desktop Workbench: a read-only, localhost-only human
|
|
176
|
-
console that
|
|
176
|
+
console that shows this surface (and the other four operator panels — run
|
|
177
177
|
graph, blackboard, worker logs, candidate compare, audit timeline) for any run,
|
|
178
|
-
reading the SAME capability `--json` payloads. It is a THIRD FRONT DOOR
|
|
179
|
-
the CLI and MCP that holds no authoritative state and forks no schema:
|
|
180
|
-
|
|
181
|
-
|
|
178
|
+
reading the SAME capability `--json` payloads. It is a THIRD FRONT DOOR by the
|
|
179
|
+
side of the CLI and MCP that holds no authoritative state and forks no schema:
|
|
180
|
+
each panel is the same as its `cw <cmd> --json` payload byte-for-byte
|
|
181
|
+
(parity-gated), and a refresh builds everything again from disk. See
|
|
182
182
|
[web-desktop-workbench.7.md](web-desktop-workbench.7.md).
|
|
183
183
|
|
|
184
184
|
## Observability + Cost Accounting (v0.1.31)
|
|
185
185
|
|
|
186
186
|
v0.1.31 adds Observability + Cost Accounting: `metrics show`/`metrics summary`
|
|
187
|
-
|
|
188
|
-
fail-closed `n/a`), and host-attested token/cost from
|
|
189
|
-
— no metrics database, no collector daemon, no hidden counter.
|
|
190
|
-
and
|
|
191
|
-
× a recorded pricing policy) or clearly `estimated`,
|
|
192
|
-
verbs are parity-gated and
|
|
187
|
+
work out durations, failure/verifier/acceptance rates (with sample counts and
|
|
188
|
+
fail-closed `n/a`), and host-attested token/cost from durable run state that is
|
|
189
|
+
already there — no metrics database, no collector daemon, no hidden counter.
|
|
190
|
+
Usage is added on and not required (absent ⇒ `unreported`, never 0); cost is
|
|
191
|
+
`attested` (attested usage × a recorded pricing policy) or clearly `estimated`,
|
|
192
|
+
with pricing as policy. Both verbs are parity-gated and show read-only in the
|
|
193
|
+
v0.1.30 Workbench. See
|
|
193
194
|
[observability-cost-accounting.7.md](observability-cost-accounting.7.md).
|
|
194
195
|
|
|
195
196
|
|
|
@@ -198,24 +199,24 @@ verbs are parity-gated and render read-only in the v0.1.30 Workbench. See
|
|
|
198
199
|
v0.1.32 adds Team Collaboration: a host-attested actor and append-only
|
|
199
200
|
approvals/rejections/comments/handoffs provenance-linked to a durable target,
|
|
200
201
|
plus a review gate that STACKS ON the verifier gate — required approvals from
|
|
201
|
-
authorized roles,
|
|
202
|
-
and never
|
|
203
|
-
recording who
|
|
204
|
-
authorized roles, self-approval) is data,
|
|
205
|
-
|
|
206
|
-
Workbench. See [Team Collaboration](team-collaboration.7.md).
|
|
202
|
+
authorized roles, made to happen inside `resolveCommitGate` AFTER the verifier
|
|
203
|
+
checks and never in place of them, failing closed on quorum/authority/self-approval
|
|
204
|
+
and recording who gave approval to the very artifact that shipped. Policy
|
|
205
|
+
(required approvals, authorized roles, self-approval) is data, off by default
|
|
206
|
+
(pre-v0.1.32 behavior the same). The verbs are parity-gated and show read-only in
|
|
207
|
+
the v0.1.30 Workbench. See [Team Collaboration](team-collaboration.7.md).
|
|
207
208
|
|
|
208
209
|
## Release Tooling (v0.1.33)
|
|
209
210
|
|
|
210
|
-
the per-tag mechanical surfaces (version bump across 17 surfaces, feature scaffold, and the forward-reference docs) become deterministic scripts, with a
|
|
211
|
+
the per-tag mechanical surfaces (version bump across 17 surfaces, feature scaffold, and the forward-reference docs) become deterministic scripts, with a release gate that has no repeats. See release-tooling(7).
|
|
211
212
|
|
|
212
213
|
## Real Execution Backend Integrations (v0.1.34)
|
|
213
214
|
|
|
214
|
-
container/remote/ci backends
|
|
215
|
+
container/remote/ci backends truly run (docker/podman run, remote/CI POST-and-poll) under the sandbox contract, with byte-stable evidence vs node and fail-closed refusal when a runtime/endpoint is not there to use. See real-execution-backends(7).
|
|
215
216
|
|
|
216
217
|
## Node Snapshot / Diff / Replay (v0.1.35)
|
|
217
218
|
|
|
218
|
-
per-node snapshot, structural diff, and
|
|
219
|
+
per-node snapshot, structural diff, and on-its-own deterministic replay over StateNode, using the v0.1.23 eval harness again; fail-closed on source drift (valid|stale|absent). See node-snapshot-diff-replay(7).
|
|
219
220
|
|
|
220
221
|
## Contract Migration Tooling (v0.1.36)
|
|
221
222
|
|
|
@@ -227,31 +228,31 @@ priority + concurrency limits + lease lifecycle + retry/backoff + fail-closed pa
|
|
|
227
228
|
|
|
228
229
|
## Agent Delegation Drive (v0.1.38)
|
|
229
230
|
|
|
230
|
-
|
|
231
|
+
start up an outside agent process per worker, take in result.md + attestation, auto-drive plan->dispatch->fulfill->accept->commit
|
|
231
232
|
|
|
232
233
|
## Run Retention & Provable Reclamation (v0.1.39)
|
|
233
234
|
|
|
234
|
-
tiered, append-only, cryptographically-verifiable run reclamation: seal the audit skeleton, free the
|
|
235
|
+
tiered, append-only, cryptographically-verifiable run reclamation: seal the audit skeleton, free the bulk that can be built again, prove it
|
|
235
236
|
|
|
236
237
|
## Durable State & Locking (v0.1.40)
|
|
237
238
|
|
|
238
|
-
atomic temp->rename writes + fsync-durability for authoritative stores; portable stale-stealing file lock
|
|
239
|
+
atomic temp->rename writes + fsync-durability for authoritative stores; portable stale-stealing file lock that puts in order the cross-process read-modify-write stores
|
|
239
240
|
|
|
240
241
|
## Self-Audit Hardening & Pure-Router Decomposition (v0.1.41)
|
|
241
242
|
|
|
242
|
-
evidence grounding + durable audit append + symlink-hardened containment + deterministic worker ids + recursive redaction; BackendRegistry self-describing drivers (no per-id switches); orchestrator god-object
|
|
243
|
+
evidence grounding + durable audit append + symlink-hardened containment + deterministic worker ids + recursive redaction; BackendRegistry self-describing drivers (no per-id switches); orchestrator god-object broken up into per-domain operation modules (pure loadRun->delegate router)
|
|
243
244
|
|
|
244
245
|
## Robust Result Ingest (v0.1.42)
|
|
245
246
|
|
|
246
|
-
|
|
247
|
+
take in findings/evidence from any agent shape that makes sense (alt keys + prose), CW works out grounded evidence by itself, give a warning on empty capture — closes the v0.1.41 live-drive 'accepted with 0 captured' failure
|
|
247
248
|
|
|
248
249
|
## No-False-Green Gate & Launch Prep (v0.1.43)
|
|
249
250
|
|
|
250
|
-
Hard gate
|
|
251
|
+
Hard gate that stops empty-capture verifier-gated commits, plus quickstart and launch-prep docs.
|
|
251
252
|
|
|
252
253
|
## Release-Gate Determinism & Agents Vendor (v0.1.44)
|
|
253
254
|
|
|
254
|
-
Release-readiness checks now
|
|
255
|
+
Release-readiness checks now check the committed blob (`git show HEAD:<path>`) in place of the mutable working tree — getting rid of false-red/false-green from working-tree writes at the same time (iCloud/Spotlight/editor). Adds the `agents` vendor manifest target: a generated `.agents/plugins/cool-workflow/` adapter that gives any non-Claude AI agent one common interface to CW.
|
|
255
256
|
|
|
256
257
|
## P1-P2 Fixes & CI Content Surfaces (v0.1.49)
|
|
257
258
|
|
|
@@ -268,8 +269,13 @@ Migration DAG with reversible edges (v0.1.45), capability auto-discovery (v0.1.4
|
|
|
268
269
|
|
|
269
270
|
## Fast Architecture Review (v0.1.80)
|
|
270
271
|
|
|
271
|
-
Adds the opt-in fast architecture-review lane: scoped JSONL source contexts, diff-aware exports,
|
|
272
|
+
Adds the opt-in fast architecture-review lane: scoped JSONL source contexts, diff-aware exports, Map and Assess results you can use again, wrapper metrics you can measure, a background full-review handoff you can act on, and userland model policy flags for routing fast/strong workers without changing the full review contract.
|
|
272
273
|
|
|
273
274
|
## New Both-Surface Verbs (v0.1.81)
|
|
274
275
|
|
|
275
|
-
v0.1.81 adds `audit verify` (`cw_audit_verify`) and `run inspect-archive` (`cw_run_inspect_archive`) — both declared once in the capability registry and
|
|
276
|
+
v0.1.81 adds `audit verify` (`cw_audit_verify`) and `run inspect-archive` (`cw_run_inspect_archive`) — both declared once in the capability registry and shown the same way on the CLI and MCP, fail-closed (non-zero exit / `ok:false` on a chain that is not verified or an archive that has been tampered with).
|
|
277
|
+
_No changes in v0.1.82._
|
|
278
|
+
|
|
279
|
+
## Hardening and Onboarding (v0.1.83)
|
|
280
|
+
|
|
281
|
+
Loaders fail closed on corrupt state; store writes are made safe under more than one writer; a new cw doctor checks your setup; help lists every command; and the docs are put into Basic English.
|
|
@@ -1,16 +1,16 @@
|
|
|
1
1
|
# Multi-Agent Eval & Replay Harness
|
|
2
2
|
|
|
3
3
|
CW v0.1.23 added a deterministic replay harness for topology-backed
|
|
4
|
-
multi-agent runs. It turns a
|
|
5
|
-
|
|
6
|
-
as a release gate.
|
|
4
|
+
multi-agent runs. It turns a finished run into plain JSON evidence. You can
|
|
5
|
+
replay this evidence without live agents, compare it with normalized rules,
|
|
6
|
+
score it, and use it as a release gate.
|
|
7
7
|
|
|
8
|
-
CW v0.1.25
|
|
9
|
-
derived summary layer is regression-gated
|
|
8
|
+
CW v0.1.25 adds State Explosion Management metrics to the harness, so the
|
|
9
|
+
derived summary layer is regression-gated next to the raw run:
|
|
10
10
|
`summary_freshness`, `compact_graph_parity`, `blackboard_digest_parity`,
|
|
11
11
|
`critical_path_parity`, `evidence_digest_parity`, and `expansion_ref_integrity`.
|
|
12
|
-
Pre-0.1.25 snapshots load with empty summary sections, so old fixtures
|
|
13
|
-
|
|
12
|
+
Pre-0.1.25 snapshots load with empty summary sections, so old fixtures keep
|
|
13
|
+
working. See
|
|
14
14
|
[state-explosion-management.7.md](state-explosion-management.7.md).
|
|
15
15
|
|
|
16
16
|
CW v0.1.26 adds Evidence Adoption Reasoning Chain metrics: `reasoning_freshness`,
|
|
@@ -18,15 +18,15 @@ CW v0.1.26 adds Evidence Adoption Reasoning Chain metrics: `reasoning_freshness`
|
|
|
18
18
|
snapshots load with empty reasoning sections. See
|
|
19
19
|
[evidence-adoption-reasoning-chain.7.md](evidence-adoption-reasoning-chain.7.md).
|
|
20
20
|
|
|
21
|
-
The harness is
|
|
21
|
+
The harness is file-first by design:
|
|
22
22
|
|
|
23
23
|
- snapshots, replay runs, comparisons, scores, findings, gates, and reports are
|
|
24
24
|
stored under `.cw/evals/<suite-id>/`
|
|
25
|
-
- the baseline run is not
|
|
26
|
-
- replay output is written to
|
|
27
|
-
- every CLI command
|
|
25
|
+
- the baseline run is not changed during replay
|
|
26
|
+
- replay output is written to a separate `replay/` directory
|
|
27
|
+
- every CLI command can give deterministic JSON with `--json` or
|
|
28
28
|
`--format json`
|
|
29
|
-
- MCP tools return JSON only and include generated
|
|
29
|
+
- MCP tools return JSON only and include the paths of generated artifacts
|
|
30
30
|
|
|
31
31
|
## Commands
|
|
32
32
|
|
|
@@ -55,10 +55,10 @@ node scripts/cw.js eval gate .cw/evals/<suite-id>
|
|
|
55
55
|
node scripts/cw.js eval report .cw/evals/<suite-id>/replay-run.json
|
|
56
56
|
```
|
|
57
57
|
|
|
58
|
-
`npm run eval:replay` runs the deterministic smoke suite
|
|
58
|
+
`npm run eval:replay` runs the deterministic smoke suite. It is part of
|
|
59
59
|
`npm test` and `npm run release:check`.
|
|
60
60
|
|
|
61
|
-
Human output uses
|
|
61
|
+
Human output uses fixed panels:
|
|
62
62
|
|
|
63
63
|
```text
|
|
64
64
|
Eval Suite
|
|
@@ -75,7 +75,7 @@ Next Action
|
|
|
75
75
|
|
|
76
76
|
## Artifacts
|
|
77
77
|
|
|
78
|
-
Each suite writes
|
|
78
|
+
Each suite writes these fixed files:
|
|
79
79
|
|
|
80
80
|
- `suite.json`
|
|
81
81
|
- `snapshot.json`
|
|
@@ -86,7 +86,7 @@ Each suite writes predictable files:
|
|
|
86
86
|
- `gate.json`
|
|
87
87
|
- `report.md`
|
|
88
88
|
|
|
89
|
-
The snapshot
|
|
89
|
+
The snapshot keeps workflow app identity, inputs, topology shape, roles,
|
|
90
90
|
groups, memberships, fanout/fanin state, blackboard records, worker outputs,
|
|
91
91
|
candidate scores, selection rationale, verifier-gated commit inputs,
|
|
92
92
|
trust/policy/audit records, expected operator summaries, evidence adoption, and
|
|
@@ -94,7 +94,7 @@ report sections.
|
|
|
94
94
|
|
|
95
95
|
## Comparison Rules
|
|
96
96
|
|
|
97
|
-
The comparison checks:
|
|
97
|
+
The comparison checks these:
|
|
98
98
|
|
|
99
99
|
- topology id and topology run shape
|
|
100
100
|
- roles, groups, memberships, fanout, and fanin records
|
|
@@ -107,7 +107,7 @@ The comparison checks:
|
|
|
107
107
|
- verifier-gated commit readiness
|
|
108
108
|
- report sections
|
|
109
109
|
|
|
110
|
-
Normalization
|
|
110
|
+
Normalization takes out unstable paths, timestamps, generated temp roots, and
|
|
111
111
|
machine-local directories. It does not hide changed evidence, policy,
|
|
112
112
|
selection, scoring, or commit-gate behavior.
|
|
113
113
|
|
|
@@ -138,7 +138,7 @@ Scores are deterministic metrics:
|
|
|
138
138
|
- `verifier_commit_gate_parity`
|
|
139
139
|
- `report_parity`
|
|
140
140
|
|
|
141
|
-
Each metric
|
|
141
|
+
Each metric gives back `id`, `status`, `score`, `maxScore`, `reason`, evidence
|
|
142
142
|
refs, baseline refs, and replay refs.
|
|
143
143
|
|
|
144
144
|
## Gate
|
|
@@ -149,13 +149,13 @@ selected candidate, changed evidence adoption, changed policy violations,
|
|
|
149
149
|
missing provenance, lost verifier-gated commit readiness, or graph/dependency
|
|
150
150
|
loss.
|
|
151
151
|
|
|
152
|
-
|
|
153
|
-
must be
|
|
154
|
-
release gate can
|
|
152
|
+
You can show improvements as changed findings in later suites, but they
|
|
153
|
+
must be seen in `score.json`, `findings.json`, and `report.md` before a
|
|
154
|
+
release gate can take them.
|
|
155
155
|
|
|
156
156
|
## MCP Parity
|
|
157
157
|
|
|
158
|
-
The MCP surface
|
|
158
|
+
The MCP surface matches the CLI:
|
|
159
159
|
|
|
160
160
|
- `cw_eval_snapshot`
|
|
161
161
|
- `cw_eval_replay`
|
|
@@ -168,7 +168,7 @@ MCP responses are deterministic JSON and include artifact paths.
|
|
|
168
168
|
|
|
169
169
|
## Release Use
|
|
170
170
|
|
|
171
|
-
Use this harness after a topology-backed run
|
|
171
|
+
Use this harness after a topology-backed run gets a score, a selection, and a
|
|
172
172
|
verifier-gated commit:
|
|
173
173
|
|
|
174
174
|
```bash
|
|
@@ -180,41 +180,41 @@ node scripts/cw.js eval gate .cw/evals/release-replay
|
|
|
180
180
|
node scripts/cw.js eval report .cw/evals/release-replay/replay-run.json
|
|
181
181
|
```
|
|
182
182
|
|
|
183
|
-
The gate proves the replay
|
|
184
|
-
evidence adoption stayed traceable, trust/policy/audit records
|
|
185
|
-
explainable, judge rationale is
|
|
183
|
+
The gate proves the replay finished, graph/dependencies stayed stable,
|
|
184
|
+
evidence adoption stayed traceable, trust/policy/audit records stayed
|
|
185
|
+
explainable, judge rationale is there, scoring/selection did not regress, and
|
|
186
186
|
verifier-gated commit readiness still holds.
|
|
187
187
|
## CLI ↔ MCP Parity (v0.1.27)
|
|
188
188
|
|
|
189
|
-
Every command and tool
|
|
190
|
-
registry (`src/capability-registry.ts`) and
|
|
191
|
-
so `cw <cmd> --json` and the matching `cw_<tool>` result
|
|
189
|
+
Every command and tool named above is declared in the v0.1.27 capability
|
|
190
|
+
registry (`src/capability-registry.ts`) and checked by `npm run parity:check`,
|
|
191
|
+
so `cw <cmd> --json` and the matching `cw_<tool>` result show one data source.
|
|
192
192
|
See [cli-mcp-parity.7.md](cli-mcp-parity.7.md).
|
|
193
193
|
|
|
194
194
|
## Run Registry / Control Plane (v0.1.28)
|
|
195
195
|
|
|
196
196
|
The runs described here are indexed, searchable, resumable, archivable, and
|
|
197
|
-
rerunnable across repos by the v0.1.28 Run Registry / Control Plane
|
|
197
|
+
rerunnable across repos by the v0.1.28 Run Registry / Control Plane. It derives
|
|
198
198
|
a fingerprinted, fail-closed index over the same per-run `.cw/runs/<id>/state.json`
|
|
199
199
|
source of truth. See [run-registry-control-plane.7.md](run-registry-control-plane.7.md).
|
|
200
200
|
|
|
201
201
|
## Execution Backends (v0.1.29)
|
|
202
202
|
|
|
203
|
-
v0.1.29
|
|
203
|
+
v0.1.29 moves execution into a pluggable driver layer: one narrow `ExecutionBackend`
|
|
204
204
|
contract with interchangeable `node`/`bun`/`shell`/`container`/`remote`/`ci`
|
|
205
|
-
drivers,
|
|
206
|
-
`backend list|show|probe`. The result/evidence envelope
|
|
207
|
-
backends; the backend id + sandbox attestation are
|
|
208
|
-
surface
|
|
205
|
+
drivers, chosen by `--backend` (next to `--sandbox`) and looked at with
|
|
206
|
+
`backend list|show|probe`. The result/evidence envelope has the same schema across
|
|
207
|
+
backends; the backend id + sandbox attestation are kept as provenance, so this
|
|
208
|
+
surface stays the same no matter which backend ran a run. See
|
|
209
209
|
[execution-backends.7.md](execution-backends.7.md).
|
|
210
210
|
## Web / Desktop Workbench (v0.1.30)
|
|
211
211
|
|
|
212
212
|
v0.1.30 adds the Web / Desktop Workbench: a read-only, localhost-only human
|
|
213
|
-
console that
|
|
213
|
+
console that shows this surface (and the other four operator panels — run
|
|
214
214
|
graph, blackboard, worker logs, candidate compare, audit timeline) for any run,
|
|
215
|
-
reading the SAME capability `--json` payloads. It is a THIRD FRONT DOOR
|
|
215
|
+
reading the SAME capability `--json` payloads. It is a THIRD FRONT DOOR next to
|
|
216
216
|
the CLI and MCP that holds no authoritative state and forks no schema: each panel
|
|
217
|
-
equals its `cw <cmd> --json` payload byte-for-byte (parity-gated), and refresh
|
|
217
|
+
equals its `cw <cmd> --json` payload byte-for-byte (parity-gated), and a refresh
|
|
218
218
|
re-derives everything from disk. See
|
|
219
219
|
[web-desktop-workbench.7.md](web-desktop-workbench.7.md).
|
|
220
220
|
|
|
@@ -222,11 +222,11 @@ re-derives everything from disk. See
|
|
|
222
222
|
|
|
223
223
|
v0.1.31 adds Observability + Cost Accounting: `metrics show`/`metrics summary`
|
|
224
224
|
derive durations, failure/verifier/acceptance rates (with sample counts and
|
|
225
|
-
fail-closed `n/a`), and host-attested token/cost from
|
|
226
|
-
— no metrics database, no collector daemon, no hidden counter. Usage is
|
|
227
|
-
and optional (absent ⇒ `unreported`, never 0); cost is `attested` (attested usage
|
|
225
|
+
fail-closed `n/a`), and host-attested token/cost from run state that is already
|
|
226
|
+
durable — no metrics database, no collector daemon, no hidden counter. Usage is
|
|
227
|
+
additive and optional (absent ⇒ `unreported`, never 0); cost is `attested` (attested usage
|
|
228
228
|
× a recorded pricing policy) or clearly `estimated`, with pricing as policy. Both
|
|
229
|
-
verbs are parity-gated and
|
|
229
|
+
verbs are parity-gated and show read-only in the v0.1.30 Workbench. See
|
|
230
230
|
[observability-cost-accounting.7.md](observability-cost-accounting.7.md).
|
|
231
231
|
|
|
232
232
|
|
|
@@ -236,10 +236,10 @@ v0.1.32 adds Team Collaboration: a host-attested actor and append-only
|
|
|
236
236
|
approvals/rejections/comments/handoffs provenance-linked to a durable target,
|
|
237
237
|
plus a review gate that STACKS ON the verifier gate — required approvals from
|
|
238
238
|
authorized roles, enforced inside `resolveCommitGate` AFTER the verifier checks
|
|
239
|
-
and never
|
|
240
|
-
recording who approved the
|
|
241
|
-
authorized roles, self-approval) is data, default off (pre-v0.1.32 behavior
|
|
242
|
-
|
|
239
|
+
and never in place of them, failing closed on quorum/authority/self-approval and
|
|
240
|
+
recording who approved the same artifact that shipped. Policy (required approvals,
|
|
241
|
+
authorized roles, self-approval) is data, default off (pre-v0.1.32 behavior is
|
|
242
|
+
the same). The verbs are parity-gated and show read-only in the v0.1.30
|
|
243
243
|
Workbench. See [Team Collaboration](team-collaboration.7.md).
|
|
244
244
|
|
|
245
245
|
## Release Tooling (v0.1.33)
|
|
@@ -248,11 +248,11 @@ the per-tag mechanical surfaces (version bump across 17 surfaces, feature scaffo
|
|
|
248
248
|
|
|
249
249
|
## Real Execution Backend Integrations (v0.1.34)
|
|
250
250
|
|
|
251
|
-
container/remote/ci backends really execute (docker/podman run, remote/CI POST-and-poll) under the sandbox contract, with byte-stable evidence vs node and fail-closed refusal when a runtime/endpoint is
|
|
251
|
+
container/remote/ci backends really execute (docker/podman run, remote/CI POST-and-poll) under the sandbox contract, with byte-stable evidence vs node and fail-closed refusal when a runtime/endpoint is not there. See real-execution-backends(7).
|
|
252
252
|
|
|
253
253
|
## Node Snapshot / Diff / Replay (v0.1.35)
|
|
254
254
|
|
|
255
|
-
per-node snapshot, structural diff, and isolated deterministic replay over StateNode,
|
|
255
|
+
per-node snapshot, structural diff, and isolated deterministic replay over StateNode, using the v0.1.23 eval harness again; fail-closed on source drift (valid|stale|absent). See node-snapshot-diff-replay(7).
|
|
256
256
|
|
|
257
257
|
## Contract Migration Tooling (v0.1.36)
|
|
258
258
|
|
|
@@ -280,15 +280,15 @@ evidence grounding + durable audit append + symlink-hardened containment + deter
|
|
|
280
280
|
|
|
281
281
|
## Robust Result Ingest (v0.1.42)
|
|
282
282
|
|
|
283
|
-
capture findings/evidence from any reasonable agent shape (alt keys + prose), CW derives grounded evidence itself, warn on empty capture —
|
|
283
|
+
capture findings/evidence from any reasonable agent shape (alt keys + prose), CW derives grounded evidence itself, warn on empty capture — fixes the v0.1.41 live-drive 'accepted with 0 captured' failure
|
|
284
284
|
|
|
285
285
|
## No-False-Green Gate & Launch Prep (v0.1.43)
|
|
286
286
|
|
|
287
|
-
Hard gate
|
|
287
|
+
Hard gate that blocks empty-capture verifier-gated commits, plus quickstart and launch-prep docs.
|
|
288
288
|
|
|
289
289
|
## Release-Gate Determinism & Agents Vendor (v0.1.44)
|
|
290
290
|
|
|
291
|
-
Release-readiness checks now
|
|
291
|
+
Release-readiness checks now check the committed blob (`git show HEAD:<path>`) instead of the mutable working tree — this removes false-red/false-green from concurrent working-tree writes (iCloud/Spotlight/editor). Adds the `agents` vendor manifest target: a generated `.agents/plugins/cool-workflow/` adapter that gives any non-Claude AI agent one common interface to CW.
|
|
292
292
|
|
|
293
293
|
## P1-P2 Fixes & CI Content Surfaces (v0.1.49)
|
|
294
294
|
|
|
@@ -305,6 +305,11 @@ Migration DAG with reversible edges (v0.1.45), capability auto-discovery (v0.1.4
|
|
|
305
305
|
|
|
306
306
|
## Fast Architecture Review (v0.1.80)
|
|
307
307
|
|
|
308
|
-
Adds the opt-in fast architecture-review lane: scoped JSONL source contexts, diff-aware exports, reusable Map and Assess results, measurable wrapper metrics,
|
|
308
|
+
Adds the opt-in fast architecture-review lane: scoped JSONL source contexts, diff-aware exports, reusable Map and Assess results, measurable wrapper metrics, useful background full-review handoff, and userland model policy flags for routing fast/strong workers without changing the full review contract.
|
|
309
309
|
|
|
310
|
-
_No changes to the multi-agent eval/replay harness in v0.1.81 (the multi-agent-eval module was
|
|
310
|
+
_No changes to the multi-agent eval/replay harness in v0.1.81 (the multi-agent-eval module was split into behavior-preserving siblings; replay output is byte-identical)._
|
|
311
|
+
_v0.1.82 — replay now RE-DERIVES the projection from the raw captured state instead of copying the baseline, so a nondeterministic projection is caught instead of passing quietly; a new regression smoke (including an intrinsic-nondeterminism case) proves the moat has teeth._
|
|
312
|
+
|
|
313
|
+
## Hardening and Onboarding (v0.1.83)
|
|
314
|
+
|
|
315
|
+
Loaders fail closed on corrupt state; store writes are made safe under more than one writer; a new cw doctor checks your setup; help lists every command; and the docs are put into Basic English.
|