dsh-continual-evolve 0.3.0 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +85 -409
- package/README.zh.md +86 -272
- package/lib/apply.js +7 -1
- package/lib/auto.d.ts +17 -0
- package/lib/auto.js +5 -2
- package/lib/benchmark-command.js +2 -0
- package/lib/command.d.ts +3 -0
- package/lib/command.js +66 -2
- package/lib/consolidate.d.ts +65 -0
- package/lib/consolidate.js +113 -0
- package/lib/fate.d.ts +2 -1
- package/lib/fate.js +5 -4
- package/lib/index.d.ts +22 -0
- package/lib/index.js +19 -1
- package/lib/inject.d.ts +26 -17
- package/lib/inject.js +82 -49
- package/lib/promotion.d.ts +83 -0
- package/lib/promotion.js +127 -0
- package/lib/search.d.ts +83 -0
- package/lib/search.js +136 -0
- package/lib/service.js +41 -1
- package/lib/skill-render.d.ts +9 -1
- package/lib/skill-render.js +40 -2
- package/lib/state.js +6 -1
- package/lib/tool.js +8 -1
- package/lib/types.d.ts +8 -0
- package/lib/types.js +8 -0
- package/lib/usage.d.ts +17 -4
- package/lib/usage.js +41 -10
- package/lib/wrapup-command.d.ts +2 -1
- package/lib/wrapup-command.js +4 -3
- package/lib/wrapup.d.ts +15 -6
- package/lib/wrapup.js +45 -6
- package/package.json +9 -7
package/README.md
CHANGED
|
@@ -7,452 +7,128 @@
|
|
|
7
7
|
[](https://github.com/ZK-Andy/dsh-continual-evolve/actions/workflows/ci.yml)
|
|
8
8
|
[](LICENSE)
|
|
9
9
|
[](package.json)
|
|
10
|
-
[]()
|
|
10
|
+
[]()
|
|
12
11
|
|
|
13
|
-
Continual self-evolution for [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness): a versioned, auditable, rollback-safe
|
|
12
|
+
Continual self-evolution for [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness): a versioned, auditable, rollback-safe harness state layer — prompt notes, memories, skills, subagent specs — refined from session trajectories.
|
|
14
13
|
|
|
15
|
-
|
|
16
|
-
> shipped the full evolution loop: the pure-core engine, model tools and
|
|
17
|
-
> the `/evolve` command, the automatic review gate (turn-interval +
|
|
18
|
-
> compaction checkpoints, human approval for global edits), real
|
|
19
|
-
> system-prompt injection (prompt notes + delegation specs, zero token
|
|
20
|
-
> cost when empty), and the benchmark-driven validation loop (code-owned
|
|
21
|
-
> scoring, non-regressive acceptance, rubric ACL). Since then the plugin
|
|
22
|
-
> keeps growing with usage-driven enhancements — the memory layer (ranked
|
|
23
|
-
> injection, trajectory citations, archive), per-installation rubric keys,
|
|
24
|
-
> plugin-owned file logging, the session wrap-up (`/evolve wrapup`), and
|
|
25
|
-
> the gate's automatic local-fate dimension (local entries get a promoted
|
|
26
|
-
> or archived exit on the gate's own cadence — consulted first, never
|
|
27
|
-
> written silently). See the Roadmap for the full shipped and candidate
|
|
28
|
-
> lists.
|
|
29
|
-
|
|
30
|
-
## Background
|
|
31
|
-
|
|
32
|
-
This project started as a research question: *can a harness improve itself,
|
|
33
|
-
and what would a production-grade version look like?* Three lines of evidence
|
|
34
|
-
shaped the answer:
|
|
35
|
-
|
|
36
|
-
- **penguin-harness** demonstrated the concept (benchmark → evaluate →
|
|
37
|
-
optimize → accept/rollback) but with **zero code-level enforcement** — every
|
|
38
|
-
guarantee was a prompt contract. Its report (`docs/research/`) became the
|
|
39
|
-
hardening checklist this project implements.
|
|
40
|
-
- **prime-agent `/refine`** proved the engineering shape: versioned harness
|
|
41
|
-
entries, atomic persistence, optimistic concurrency, inverse-op rollback.
|
|
42
|
-
This package is an original implementation of that shape on the DSH plugin
|
|
43
|
-
surface.
|
|
44
|
-
- Academic work (Self-Harness, AHE, HarnessOpt-Bench) supplied the discipline:
|
|
45
|
-
frozen evaluation runtime, code-owned aggregation, non-regressive
|
|
46
|
-
acceptance.
|
|
47
|
-
|
|
48
|
-
The result: **the model proposes, the code guarantees.** Every mechanical
|
|
49
|
-
safety property (schema validation, snapshots, versioning, audit trail,
|
|
50
|
-
acceptance decisions) is enforced in code — never by asking the model to
|
|
51
|
-
behave.
|
|
14
|
+
**The model proposes, the code guarantees.** Every mechanical safety property — schema validation, atomic writes, snapshots, versioning, audit trail, acceptance decisions — is enforced in code, never by prompt discipline.
|
|
52
15
|
|
|
53
16
|
## Why
|
|
54
17
|
|
|
55
|
-
Agents accumulate reusable experience
|
|
56
|
-
|
|
57
|
-
- **Versioned entries** keyed by kind (`prompt` / `memory` / `skill` / `subagent`), each with a recorded provenance and version
|
|
58
|
-
- **Evidence trail**: every refinement appends an event carrying `trigger / changes / evidence / outcome`
|
|
59
|
-
- **Deterministic rollback**: inverse edits are generated from applied results — no LLM re-guessing
|
|
60
|
-
- **Code-enforced safety**, not prompt discipline: schema validation, atomic writes, corrupt-file degrade, optimistic concurrency, immutable base system prompt
|
|
61
|
-
- **Local (session) and global (cross-session) scopes** with merge semantics
|
|
62
|
-
|
|
63
|
-
## Design provenance
|
|
64
|
-
|
|
65
|
-
Inspired by three bodies of work (see [`docs/design.md`](docs/design.md)):
|
|
18
|
+
Agents accumulate reusable experience (repeated failures, durable facts, reusable procedures) and forget it next session. This plugin turns that experience into first-class state:
|
|
66
19
|
|
|
67
|
-
- **
|
|
68
|
-
- **
|
|
69
|
-
-
|
|
20
|
+
- **Local scope** per session; **global scope** across sessions with merge semantics — plus mechanical promotion guards so only portable, substantial, non-duplicate knowledge reaches global
|
|
21
|
+
- **Deterministic rollback**: inverse edits generated from applied results — no LLM re-guessing
|
|
22
|
+
- **Benchmark loop**: candidate refinements are evaluated against frozen cases by a separate scorer before acceptance (rubric encrypted at rest)
|
|
23
|
+
- **Store hygiene**: `/evolve consolidate` turns write-time conflict hints and zero-use staleness into one approved, fully reversible batch of archives
|
|
70
24
|
|
|
71
|
-
##
|
|
25
|
+
## How it works
|
|
72
26
|
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
| Package manager | pnpm (DSH ecosystem standard) |
|
|
79
|
-
| Build | `tsc` → `lib/` (main `lib/index.js`, types `lib/index.d.ts`) |
|
|
80
|
-
| Tests | Vitest |
|
|
81
|
-
| Lint | oxlint (DSH official repo convention) |
|
|
82
|
-
| License | MIT |
|
|
83
|
-
|
|
84
|
-
## Project layout
|
|
85
|
-
|
|
86
|
-
```
|
|
87
|
-
dsh-continual-evolve/
|
|
88
|
-
├── package.json # exports / files / engines / scripts + dsh.bundle manifest
|
|
89
|
-
├── cordis.patch.yml # bundle patch (dsh plugin add activates on install)
|
|
90
|
-
├── tsconfig.json / .oxlintrc.json / .editorconfig / .gitignore
|
|
91
|
-
├── LICENSE / README.md / README.zh.md
|
|
92
|
-
├── docs/
|
|
93
|
-
│ ├── design.md # full design doc (incl. hardening matrix)
|
|
94
|
-
│ └── research/ # penguin-harness report + prime-agent reference source
|
|
95
|
-
├── src/
|
|
96
|
-
│ ├── index.ts # cordis plugin entry (service mount + wiring)
|
|
97
|
-
│ ├── types.ts # HarnessState / entry / edit / result types
|
|
98
|
-
│ ├── state.ts # atomic persistence, corrupt degrade, merge, concurrency
|
|
99
|
-
│ ├── validate.ts # code-enforced edit validation
|
|
100
|
-
│ ├── apply.ts # per-edit apply pass with optimistic locking
|
|
101
|
-
│ ├── rollback.ts # deterministic inverse-op rollback
|
|
102
|
-
│ ├── plan.ts # proposal JSON parsing (truncation-aware)
|
|
103
|
-
│ ├── tool.ts # evolve_* model-facing tools (5)
|
|
104
|
-
│ ├── command.ts # /evolve command dispatcher + shared utilities
|
|
105
|
-
│ ├── goal-command.ts # /evolve goal subcommand handler
|
|
106
|
-
│ ├── mount-command.ts # /evolve mount + unmount subcommand handlers
|
|
107
|
-
│ ├── benchmark-command.ts # /evolve benchmark subcommand handler
|
|
108
|
-
│ ├── wrapup-command.ts # /evolve wrapup subcommand handler
|
|
109
|
-
│ ├── planner.ts # ctx.llm planner
|
|
110
|
-
│ ├── llm-text.ts # unified streaming-text helper (BlockAssembler + finish check)
|
|
111
|
-
│ ├── render.ts # bounded prompt rendering
|
|
112
|
-
│ ├── inject.ts # dynamic system-prompt section (prompt notes + delegation specs, ranked injection)
|
|
113
|
-
│ ├── source.ts # trajectory citations (sessionId + event seqs of distilled entries)
|
|
114
|
-
│ ├── auto.ts # auto-review gate (turn/compaction triggers + audit, global-aware view, local-fate phase)
|
|
115
|
-
│ ├── fate.ts # gate local-fate dimension — auto promote/archive of local entries (consulted first, cooldown)
|
|
116
|
-
│ ├── notify.ts # gate visibility — follow-up notice after an approved auto-refine
|
|
117
|
-
│ ├── goal.ts # goal-driven evolution rounds (/evolve goal)
|
|
118
|
-
│ ├── review.ts # gate LLM judgment (declines local duplicates of globally covered topics)
|
|
119
|
-
│ ├── approval.ts # human approval for global edits
|
|
120
|
-
│ ├── skill.ts # skill materialization ($DSH_HOME/skills/)
|
|
121
|
-
│ ├── skill-render.ts # shared skill rendering (skillNameOf + renderSkillMarkdown, breaks circular dependency)
|
|
122
|
-
│ ├── skillquality.ts # skill standard in the loop (skill-creator template reading + frontmatter code checks)
|
|
123
|
-
│ ├── mount.ts # hot-mounted skill plugins (loader.create + boot restore)
|
|
124
|
-
│ ├── benchmark.ts # benchmark store + CellScore types (with runtime evidence fields)
|
|
125
|
-
│ ├── rubric.ts # rubric ACL (AES-256-GCM envelopes, auto-generated local key)
|
|
126
|
-
│ ├── logfile.ts # plugin-owned file logging (JSONL exporter + rotation)
|
|
127
|
-
│ ├── score.ts # code-owned aggregation + acceptance rule
|
|
128
|
-
│ ├── evaluate.ts # two-stage evaluation runner (executor evidence → independent reviewer) + failure-cell protocol + runtime verification
|
|
129
|
-
│ ├── pool.ts # bounded-concurrency worker pool for evaluation runs
|
|
130
|
-
│ ├── store.ts # store layout + snapshots + result history
|
|
131
|
-
│ ├── service.ts # evolution engine (onApplied hook)
|
|
132
|
-
│ ├── usage.ts # entry injection usage tracking (durable counts, staleness detection)
|
|
133
|
-
│ ├── failures.ts # failure-signature aggregation (gate + benchmark failures by class, /evolve failures)
|
|
134
|
-
│ └── wrapup.ts # session wrap-up lifecycle (promote / split-promote → global, guarded archive; shared proposal builders; staleness signal)
|
|
135
|
-
└── test/ # 28 files, 401 tests
|
|
136
|
-
```
|
|
27
|
+
1. **Sediment** — the model creates entries via `evolve_add`, or the automatic review gate proposes them from the session trajectory (turn-interval + compaction checkpoints).
|
|
28
|
+
2. **Guard** — code-enforced validation: edit schema, blast-radius/scope coherence, and the promotion policy (project-scoped markers, thin content, near-duplicate detection keep the global store clean). Global creates that near-duplicate an existing entry are rejected at write time (≥0.8 similarity); moderate overlaps carry a `conflictHint` for later consolidation.
|
|
29
|
+
3. **Approve** — global writes require explicit human approval; local-fate proposals are consulted before they land.
|
|
30
|
+
4. **Apply & inject** — atomic apply with snapshot + audit event. Prompt notes and delegation specs inject into the system prompt (capped, relevance-ranked, zero tokens when empty); memories/skills appear as a capped directory index.
|
|
31
|
+
5. **Validate & roll back** — benchmarks score candidates against frozen cases; rejected candidates roll back deterministically.
|
|
137
32
|
|
|
138
33
|
## Install
|
|
139
34
|
|
|
140
35
|
```bash
|
|
141
36
|
# from npm (installs and activates — ships its own bundle patch)
|
|
142
|
-
dsh plugin
|
|
143
|
-
|
|
144
|
-
# or from source (first GitHub installs require approving the allowBuilds build step)
|
|
145
|
-
dsh plugin --profile web add github:ZK-Andy/dsh-continual-evolve
|
|
146
|
-
```
|
|
147
|
-
|
|
148
|
-
Swap `web` for your profile name (`headless`, or a custom profile).
|
|
149
|
-
|
|
150
|
-
## In-session usage (after restart)
|
|
151
|
-
|
|
152
|
-
```
|
|
153
|
-
/evolve help + current local store
|
|
154
|
-
/evolve list [global] list entries
|
|
155
|
-
/evolve history applied refinements (ids for rollback)
|
|
156
|
-
/evolve rollback <id> deterministically revert a refinement
|
|
157
|
-
/evolve plan [msg] LLM planner against the current store
|
|
158
|
-
/evolve wrapup assess this session's local entries: promote reusable ones to the
|
|
159
|
-
global store (approval required), archive session-specific ones
|
|
160
|
-
/evolve archive <id> hide an entry from injection (data kept, restorable)
|
|
161
|
-
/evolve unarchive <id> restore an archived entry
|
|
162
|
-
/evolve log [tail N] [session <id>] show the recent plugin log (default 50 lines; optional per-session filter)
|
|
163
|
-
/evolve failures aggregated failure counts (review-gate + benchmark, by class — D1 observation layer)
|
|
164
|
-
/evolve export <path> backup the local store to JSON
|
|
165
|
-
/evolve import <path> restore a store from an export file
|
|
166
|
-
/evolve mount <skillId> hot-mount a skill entry as a live cordis plugin (tool: skill_<name>)
|
|
167
|
-
/evolve mount list list hot-mounted plugins (restored on boot)
|
|
168
|
-
/evolve unmount <id> remove a hot-mounted plugin
|
|
169
|
-
/evolve goal show the evolution goal (round-driven auto-review)
|
|
170
|
-
/evolve goal <objective> create/update the evolution goal — while active, the review gate runs EVERY round
|
|
171
|
-
/evolve goal done complete the evolution goal
|
|
172
|
-
```
|
|
173
|
-
|
|
174
|
-
Model-facing tools: `evolve_list`, `evolve_add`, `evolve_update`, `evolve_delete`, `evolve_rollback`.
|
|
175
|
-
|
|
176
|
-
## Memory layer
|
|
177
|
-
|
|
178
|
-
Beyond the persisted store itself, four features keep injected memory
|
|
179
|
-
"understanding you" as entries grow (gap analysis vs. Mem0 / Letta / Zep /
|
|
180
|
-
LangMem; no external services — everything is pure functions):
|
|
181
|
-
|
|
182
|
-
- **Ranked injection** — when a kind holds more than the 6-entry cap, the
|
|
183
|
-
injected block no longer shows the fixed first six: entries are scored by
|
|
184
|
-
relevance to the agent's most recent direct user messages (keyword/BM25
|
|
185
|
-
level: title hits weigh 2×) and then by recency (`updated_at`, 30-day
|
|
186
|
-
half-life), so the freshest *and most relevant* entries fill the cap. The
|
|
187
|
-
empty-store zero-token behavior is unchanged.
|
|
188
|
-
- **Trajectory citations** — every newly created entry records
|
|
189
|
-
`metadata.sourceSession` + `metadata.sourceSeqs` pointing at the direct
|
|
190
|
-
user messages it was distilled from (DSH sessions are event-sourced with
|
|
191
|
-
contiguous seqs, so the citation expands back into the durable session
|
|
192
|
-
log). Listings show `src=<sessionId>:<seqs>`; old entries are not migrated
|
|
193
|
-
and never error.
|
|
194
|
-
- **Archive** — `/evolve archive <id>` hides an entry from injection
|
|
195
|
-
(`metadata.archivedAt`, data kept, rollback-compatible) and
|
|
196
|
-
`/evolve unarchive <id>` restores it. Archived entries are marked
|
|
197
|
-
`[archived]` in `evolve_list` and skipped by injection; the overflow count
|
|
198
|
-
excludes them.
|
|
199
|
-
- **Session wrap-up** — a session's local entries otherwise become orphans when
|
|
200
|
-
it ends (later sessions never see them). `/evolve wrapup` gives them an exit.
|
|
201
|
-
Each entry is audited mechanically — global-coverage is judged by **title
|
|
202
|
-
similarity only** (a bare id collision with a different title is intentionally
|
|
203
|
-
NOT coverage; the actual matching global titles are shown to the assessor so
|
|
204
|
-
it judges against real content) — then classified as `promote` / `archive` /
|
|
205
|
-
`keep`. Promotions move reusable entries into the global store **through the
|
|
206
|
-
human approval gate**, keeping their trajectory citation and adding a
|
|
207
|
-
`sourcedFromLocal=<session>:<id>` back-link; the local copy is stamped
|
|
208
|
-
`promotedTo` and retired from injection so it is never offered again.
|
|
209
|
-
**Split promotion** (A-form): a mixed entry (durable facts + session snapshot)
|
|
210
|
-
can be archived while carrying a cleaned `promote` sub-object — only the
|
|
211
|
-
durable part lands globally, the snapshot stays in the archive. A **symmetric
|
|
212
|
-
archive guard** requires user confirmation before an archive that is NOT
|
|
213
|
-
globally covered AND was distilled from real user messages hides that content
|
|
214
|
-
from future sessions (over-archiving gets the same protection as
|
|
215
|
-
over-writing); operational entries still archive silently. Everything stays
|
|
216
|
-
snapshot/versioned/rollbackable.
|
|
217
|
-
- **Gate local-fate (automatic wrap-up)** — the same wrap-up machinery now
|
|
218
|
-
runs inside the auto-review gate on its own cadence (`fateIntervalTurns`),
|
|
219
|
-
so local entries get their exit while the session is still running instead
|
|
220
|
-
of waiting for a manual `/evolve wrapup`. On each due gate run the audited
|
|
221
|
-
candidates are classified by the assessor and partitioned by the same
|
|
222
|
-
deterministic guards; the user is consulted FIRST before anything governed
|
|
223
|
-
lands (one dialog covering promotes, split promotions and review-required
|
|
224
|
-
archives — the consultSkillEdits pattern, with a decline cooldown). Covered
|
|
225
|
-
or operational entries still archive silently, and at compaction the gate
|
|
226
|
-
never opens a dialog: only silent archives apply, governed actions are
|
|
227
|
-
deferred with an audit record pointing at `/evolve wrapup`. Every fate
|
|
228
|
-
decision lands in `reviews.jsonl` (`approved` / `declined` / `deferred` /
|
|
229
|
-
`assessed` / `failed`) and applied actions are visible via a follow-up
|
|
230
|
-
notice. Apply writes are byte-identical to the wrap-up command (shared
|
|
231
|
-
proposal builders).
|
|
232
|
-
- **Global-aware gate** — the auto-review gate and planner judge the merged
|
|
233
|
-
global + local state with every entry's real scope labeled, so a topic
|
|
234
|
-
already covered by a global entry is declined instead of being re-sedimented
|
|
235
|
-
as a local duplicate.
|
|
236
|
-
|
|
237
|
-
## Skill standard in the loop
|
|
37
|
+
dsh plugin add dsh-continual-evolve
|
|
238
38
|
|
|
239
|
-
|
|
240
|
-
|
|
241
|
-
tool. To keep self-evolved skills on the quality bar, the plugin references
|
|
242
|
-
the **skill-creator** / **skill-audit** skills (user-level skills distilled
|
|
243
|
-
by the author from the official deepseek-harness 11 skills; template facts
|
|
244
|
-
verified against deepseek-harness `47f9438`) at runtime — they stay the
|
|
245
|
-
single source of truth on disk, nothing is copied:
|
|
246
|
-
|
|
247
|
-
- Every planning call receives a `<skill_quality_standard>` block: the
|
|
248
|
-
`skill-creator/references/template.md` facts when those skills are
|
|
249
|
-
installed (`<dshHome>/skills/`), or a builtin distilled guide otherwise
|
|
250
|
-
(~1KB, low-frequency calls). The planner must ground skill proposals in a
|
|
251
|
-
REAL trigger scenario from the trajectory, must not duplicate the
|
|
252
|
-
official 11 skills or existing entries, and self-checks every proposed
|
|
253
|
-
skill against the 7 structural features.
|
|
254
|
-
- The gate judges skill-related trajectories against the skill-audit
|
|
255
|
-
dimensions (frontmatter routing, structural features, paragraph skeleton,
|
|
256
|
-
duplication) and declines proposals that would not meet the standard.
|
|
257
|
-
- The mechanical frontmatter rules of `validate-frontmatter.mjs` are
|
|
258
|
-
code-enforced at apply time: skill bodies must not open with a second
|
|
259
|
-
`---` block (it would shadow the generated frontmatter), and resource
|
|
260
|
-
references may not escape the skill directory. After materialization the
|
|
261
|
-
rendered SKILL.md is re-checked and dangling `references/`/`scripts/`
|
|
262
|
-
links are logged as warnings.
|
|
263
|
-
- **Two skill forms** — `executable` skills keep the python reference
|
|
264
|
-
contract (hot-mountable as tools); `guidance` skills are SKILL.md
|
|
265
|
-
documents with no reference, the form for recurring multi-step workflows
|
|
266
|
-
(session start/end routines, handoff procedures). Code enforces the
|
|
267
|
-
split: a guidance skill must NOT carry a reference or arguments contract.
|
|
268
|
-
- **User-governed skill creation** — the gate never writes a skill
|
|
269
|
-
silently: when the planner proposes skill edits, the user is asked
|
|
270
|
-
(固化/不固化) before they land; a rejected candidate is not offered
|
|
271
|
-
again within a cooldown window. The rest of a proposal proceeds
|
|
272
|
-
regardless, so memory/prompt distillation is never blocked by a skill
|
|
273
|
-
decision.
|
|
274
|
-
|
|
275
|
-
## Logging
|
|
276
|
-
|
|
277
|
-
Plugin-owned file logging: every cordis log message (from this plugin or any
|
|
278
|
-
other) is appended to `<dshHome>/evolve/plugin.log` as JSONL (0600, rotated to
|
|
279
|
-
`plugin.log.1` past `logMaxBytes`). It works no matter how `dsh web` is
|
|
280
|
-
launched — no extra component to install, no startup-script dependency.
|
|
281
|
-
View the tail with `/evolve log [tail N]`, or read the file directly:
|
|
282
|
-
|
|
283
|
-
```bash
|
|
284
|
-
tail -f ~/.dsh/evolve/plugin.log # live
|
|
285
|
-
/evolve log 100 # last 100 lines in the chat
|
|
39
|
+
# or from source (first GitHub installs require approving the allowBuilds step)
|
|
40
|
+
dsh plugin add ZK-Andy/dsh-continual-evolve
|
|
286
41
|
```
|
|
287
42
|
|
|
288
|
-
|
|
289
|
-
`@deepseek-ai/cordis-plugin-logger-console` plugin can be added to the
|
|
290
|
-
profile (optional; the file log remains the baseline that always exists).
|
|
291
|
-
|
|
292
|
-
## Benchmark-driven validation (Phase 3)
|
|
293
|
-
|
|
294
|
-
```
|
|
295
|
-
/evolve benchmark new <title> [runs] create a benchmark (runs = repeats per case, default 1)
|
|
296
|
-
/evolve benchmark add-case <bid> <title> <statement> <rubric>
|
|
297
|
-
/evolve benchmark list list benchmarks
|
|
298
|
-
/evolve benchmark reset <bid> clear the scoreboard (re-run reference)
|
|
299
|
-
/evolve benchmark status <bid> scoreboard + decisions
|
|
300
|
-
/evolve benchmark run <bid> evaluate current state → reference
|
|
301
|
-
/evolve benchmark run <bid> candidate <refinementId> evaluate post-refinement state → decide
|
|
302
|
-
/evolve benchmark casecheck <bid> quality-gate check all cases
|
|
303
|
-
/evolve benchmark pilot <bid> <cid> single pilot run for calibration
|
|
304
|
-
/evolve benchmark freeze <bid> <cid> freeze a case as formal baseline
|
|
305
|
-
/evolve benchmark meta <bid> <cid> <field> <value> set case metadata (capability/distinguisher/shortcuts)
|
|
306
|
-
```
|
|
307
|
-
|
|
308
|
-
The loop: freeze a reference score → evolve a candidate (`/evolve plan`) →
|
|
309
|
-
run the same case × run matrix against the post-refinement state → the
|
|
310
|
-
**code-owned** acceptance rule keeps the candidate only if the overall mean
|
|
311
|
-
strictly improves with no case regressing (Self-Harness style).
|
|
312
|
-
|
|
313
|
-
**Evaluator/scorer separation (two-stage, gap A1)** — each case × run unit
|
|
314
|
-
is a PAIR of fresh subagents:
|
|
315
|
-
|
|
316
|
-
1. the **executor** performs the task with its tools and records **concrete
|
|
317
|
-
evidence** of what it did and found — it NEVER sees the rubric, so the
|
|
318
|
-
agent under test cannot optimize toward or self-grade against the grading
|
|
319
|
-
criteria;
|
|
320
|
-
2. an **independent reviewer** grades that evidence strictly against the
|
|
321
|
-
rubric (the only branch that receives the decrypted rubric), eliminating
|
|
322
|
-
the "self-produced and self-scored" bias.
|
|
323
|
-
|
|
324
|
-
Each cell records the executor's session id, so a score can be drilled back
|
|
325
|
-
to the exact transcript that produced it (trace evidence pointer, gap A4).
|
|
43
|
+
Restart `dsh web` after installing or updating.
|
|
326
44
|
|
|
327
|
-
|
|
328
|
-
(rubric decrypt error, executor/reviewer crash, protocol error) is recorded
|
|
329
|
-
as a **failed** cell, NEVER a zero: aggregation excludes failed cells from
|
|
330
|
-
every mean and counts them (`/evolve benchmark status` shows `(N failed)`),
|
|
331
|
-
and the acceptance rule rejects a round with more failed cells than
|
|
332
|
-
`maxFailedCells` (0 by default) instead of silently averaging a 0 into the
|
|
333
|
-
mean.
|
|
45
|
+
## Usage
|
|
334
46
|
|
|
335
|
-
|
|
336
|
-
construction (the planner never sees rubric files, and the executor branch
|
|
337
|
-
never decrypts); a rejection is recorded in the scoreboard and the
|
|
338
|
-
refinement is rolled back automatically (`autoRollbackOnReject`, on by
|
|
339
|
-
default).
|
|
47
|
+
Commands (in-session):
|
|
340
48
|
|
|
341
|
-
|
|
342
|
-
|
|
343
|
-
|
|
344
|
-
|
|
345
|
-
|
|
346
|
-
|
|
347
|
-
|
|
348
|
-
|
|
349
|
-
|
|
|
350
|
-
|
|
351
|
-
|
|
|
352
|
-
|
|
|
353
|
-
|
|
|
354
|
-
|
|
|
355
|
-
|
|
356
|
-
|
|
357
|
-
|
|
358
|
-
|
|
359
|
-
moves the score. Earlier runs in the same session
|
|
360
|
-
produced honest `REJECTED` decisions (0 → 0 stub cases, and 100 → 100
|
|
361
|
-
where the baseline was already perfect).
|
|
49
|
+
| Command | Effect |
|
|
50
|
+
|---|---|
|
|
51
|
+
| `/evolve` | help + current local store |
|
|
52
|
+
| `/evolve list · history · rollback <id>` | inspect and revert (add `global` for the cross-session store) |
|
|
53
|
+
| `/evolve plan [msg]` | run the LLM planner against the store |
|
|
54
|
+
| `/evolve wrapup` | assess this session's local entries: promote / archive / keep |
|
|
55
|
+
| `/evolve archive · unarchive · demote <id>` | hide from injection (data kept, restorable) — `demote` targets global noise |
|
|
56
|
+
| `/evolve consolidate [apply]` | report (or apply) one batch archive of conflict-hinted + stale zero-use global entries |
|
|
57
|
+
| `/evolve failures` | aggregated failure classes (gate + benchmark) |
|
|
58
|
+
| `/evolve log [tail N] [session <id>]` | plugin log |
|
|
59
|
+
| `/evolve export · import <path>` | backup / restore a store |
|
|
60
|
+
| `/evolve mount · unmount <skillId>` | hot-mount an executable skill as a live plugin |
|
|
61
|
+
| `/evolve goal [objective · done · block]` | round-driven auto-review goal |
|
|
62
|
+
| `/evolve benchmark …` | case lifecycle, runs, acceptance |
|
|
63
|
+
|
|
64
|
+
Model tools: `evolve_list / add / update / delete / rollback`.
|
|
65
|
+
|
|
66
|
+
Injection shape: prompt notes and delegation specs inject with content (≤6/kind × 180 chars, relevance-ranked). Memories and skills appear as a directory index (`[kind:id] title`, capped at 15 lines with a fold counter) — full text via `evolve_list`. Empty store = zero injected tokens.
|
|
362
67
|
|
|
363
68
|
## Configuration
|
|
364
69
|
|
|
365
70
|
| Key | Default | Meaning |
|
|
366
71
|
|---|---|---|
|
|
367
72
|
| `baseDir` | resolved DSH home | root for the `evolve/` stores |
|
|
368
|
-
| `
|
|
369
|
-
| `
|
|
370
|
-
| `
|
|
371
|
-
| `
|
|
372
|
-
| `
|
|
373
|
-
| `
|
|
374
|
-
| `
|
|
375
|
-
| `
|
|
376
|
-
| `
|
|
377
|
-
| `
|
|
378
|
-
| `
|
|
379
|
-
| `
|
|
380
|
-
| `
|
|
381
|
-
| `
|
|
382
|
-
| `
|
|
383
|
-
| `
|
|
384
|
-
| `
|
|
385
|
-
|
|
386
|
-
|
|
73
|
+
| `autoReview` | `false` | enable the automatic review gate |
|
|
74
|
+
| `reviewIntervalTurns` | `6` | gate cadence on the turn-interval path |
|
|
75
|
+
| `maxReviewInputChars` | `40000` | trajectory slice handed to the gate |
|
|
76
|
+
| `reviewBudgetTokens` | `4096` | output budget for the gate call |
|
|
77
|
+
| `notifyOnAutoReview` | `true` | visible follow-up notice after an applied gate run |
|
|
78
|
+
| `requireGlobalApproval` | `true` | global edits ask for explicit approval |
|
|
79
|
+
| `localFate` | `true` | gate audits local entries and proposes promote/archive (consulted, never silent) |
|
|
80
|
+
| `fateIntervalTurns` | follows `reviewIntervalTurns` | minimum turns between fate assessments |
|
|
81
|
+
| `goalBlockedWrapupTurns` | `3` | consecutive blocked-goal gate runs trigger one fate assessment (`0` disables) |
|
|
82
|
+
| `promotionBlockPatterns` | POSIX paths, session ids, `~/.dsh` | content matching these is project-scoped and never promoted to global |
|
|
83
|
+
| `promotionMinChars` | `100` | whole promotions below this length stay local |
|
|
84
|
+
| `injectionDirectoryLines` | `15` | entry-directory lines per build before folding into a counter |
|
|
85
|
+
| `sectionOrder` | `118` | system-prompt section order |
|
|
86
|
+
| `skillsDir` | `<dshHome>/skills` | where skill entries materialize as SKILL.md bundles |
|
|
87
|
+
| `rubricKey` | auto-generated key file | AES-256-GCM passphrase for benchmark rubrics (`DSH_EVOLVE_RUBRIC_KEY` overrides) |
|
|
88
|
+
| `logToFile` / `logLevel` / `logMaxBytes` | `true` / `1` / 5 MiB | plugin-owned JSONL file log with rotation |
|
|
89
|
+
| `autoRollbackOnReject` | `true` | deterministic rollback after a benchmark rejection |
|
|
90
|
+
| `reviewModel` | agent's own | optional cheaper model for the gate (`"provider/model"`) |
|
|
91
|
+
|
|
92
|
+
Example profile patch:
|
|
387
93
|
|
|
388
94
|
```yaml
|
|
389
|
-
-
|
|
390
|
-
|
|
391
|
-
|
|
392
|
-
|
|
393
|
-
autoReview: true
|
|
394
|
-
reviewIntervalTurns: 6
|
|
95
|
+
- id: continual-evolve
|
|
96
|
+
config:
|
|
97
|
+
autoReview: true
|
|
98
|
+
reviewIntervalTurns: 6
|
|
395
99
|
```
|
|
396
100
|
|
|
397
101
|
## Development
|
|
398
102
|
|
|
399
103
|
```bash
|
|
400
|
-
pnpm install
|
|
401
|
-
pnpm
|
|
402
|
-
pnpm
|
|
403
|
-
pnpm
|
|
404
|
-
pnpm lint # oxlint src test
|
|
104
|
+
pnpm install && pnpm build # deps + tsc -> lib/
|
|
105
|
+
pnpm test # vitest (543 tests)
|
|
106
|
+
pnpm test:coverage # v8 coverage, thresholds enforced in CI
|
|
107
|
+
pnpm lint # oxlint src test
|
|
405
108
|
```
|
|
406
109
|
|
|
407
|
-
|
|
408
|
-
|
|
409
|
-
Where we still lag behind prime-agent `/refine` and penguin-harness — and what to build next: [`docs/gap-analysis.md`](docs/gap-analysis.md) (P0+P1+P2+P3 shipped: evaluator/scorer separation, failure-cell protocol, runtime provenance verification + material-drift detection, usage statistics, auto-decay, case lifecycle + quality gate, entry directory view, review model separation, blast-radius annotations, duration tracking, evolve_complete events, seed benchmark; D1 observation layer + D3 goal-blocked trigger shipped; remaining: cross-process sync on demand + D1/D2 full engineering pending experiment data).
|
|
410
|
-
|
|
411
|
-
## Roadmap
|
|
110
|
+
Project layout:
|
|
412
111
|
|
|
413
|
-
|
|
112
|
+
```
|
|
113
|
+
├── src/ # engine, tools, commands, gate, fate, benchmark, usage…
|
|
114
|
+
├── test/ # vitest suites (33 files)
|
|
115
|
+
├── lib/ # build output (tsc)
|
|
116
|
+
├── docs/
|
|
117
|
+
│ ├── design.md # full design doc (hardening matrix)
|
|
118
|
+
│ ├── FAQ.md # real failure/fix records
|
|
119
|
+
│ ├── gap-analysis.md # vs prime-agent /refine + penguin-harness
|
|
120
|
+
│ ├── experiment-bootstrap.md
|
|
121
|
+
│ ├── archive/ # closed point-in-time reports
|
|
122
|
+
│ └── research/ # penguin report + prime-agent annotated source
|
|
123
|
+
├── examples/README.md # seed benchmark cases
|
|
124
|
+
└── .agents/ # AI collaboration layer (AGENTS.md, skills, ADR notes)
|
|
125
|
+
```
|
|
414
126
|
|
|
415
|
-
|
|
416
|
-
- **2026-08 maintenance wave (done)**:
|
|
417
|
-
- **memory layer** — ranked injection (relevance + recency scoring fills the per-kind cap), trajectory citations (`metadata.sourceSession` + `sourceSeqs`, shown as `src=session:seqs`), archive/unarchive (`/evolve archive <id>`, injection skips archived entries), global-aware gate (declines local duplicates of globally covered topics)
|
|
418
|
-
- **per-installation rubric key** — auto-generated local key file (`<dshHome>/evolve/rubric.key`, 0600); no more publicly known dev key
|
|
419
|
-
- **plugin-owned file logging** — every cordis log message lands in `<dshHome>/evolve/plugin.log` (JSONL, 0600, rotated), viewable via `/evolve log`; works with any launch method, no extra component to install
|
|
420
|
-
- **trajectory-grounded planning** — `/evolve plan` (and every planner call, including the gate's refine step) now reads the session trajectory: the caller's recent direct user messages are extracted from the session log and fed to the planner as a `<session_trajectory>` block, so proposals are grounded in what the user actually said (explicit `trajectory` overrides; empty trajectory is omitted at zero cost)
|
|
421
|
-
- **gate-proposed archiving** — stale entries are a first-class refine target: the planner can emit `action: "archive"` (kind + id only), which stamps `metadata.archivedAt` through the normal apply path — snapshot, version bump, audit event, and a deterministic rollback inverse that restores the pre-archive state. Archive hides from injection but never deletes; re-archiving an archived entry is rejected, and the base system prompt stays immutable
|
|
422
|
-
- **automatic rollback on benchmark rejection** — the acceptance loop is closed: when the code-owned decision rejects a candidate, the refinement is reverted automatically through the same engine path as `/evolve rollback` (deterministic inverse edits, snapshotted and audited; configurable via `autoRollbackOnReject`, on by default). Failures report the manual fallback instead of throwing
|
|
423
|
-
- **per-session log filtering** — `/evolve log [tail N] [session <id>]` keeps only the lines mentioning a given session id (exact token match, drawn from the rendered message and raw args); gate records now carry the session id in their log line
|
|
424
|
-
- **skill standard in the loop** — the planner and gate now author and judge skill entries against the skill-creator/skill-audit standard (author-distilled from the official deepseek-harness 11 skills): every plan call injects the `template.md` facts (builtin distilled guide as fallback) as `<skill_quality_standard>`; apply code-enforces the frontmatter mechanics (no shadowing `---`, no escaping resource refs); materialized SKILL.md files are re-checked and dangling resource references are logged;
|
|
425
|
-
- **guidance skills + user-governed creation** — a second skill form (SKILL.md documents without a python reference) lets recurring workflows be proposed as skills; the gate offers every auto-created skill to the user (固化/不固化) before it lands, with a rejection cooldown — skills grow under governance, never silently
|
|
426
|
-
- **2026-08-17 wrap-up wave (done)**:
|
|
427
|
-
- **`/evolve wrapup`** — a session's local entries get a real exit at session end: mechanical audit (local candidates + global-coverage detection; coverage judges **title similarity only** — a bare id collision with a different title is deliberately NOT coverage, and the real matching global titles are shown to the assessor) → LLM classification (`promote` / `archive` / `keep` + A-form split promotion: archive a mixed entry while promoting a cleaned durable sub-object) → deterministic guards re-checked at apply time (promote can never write a global duplicate; the symmetric archive guard requires user confirmation before an uncovered, user-sourced archive hides content; splits that duplicate a global topic drop to plain archive) → one human approval gate for every global create
|
|
428
|
-
- **gate local-fate dimension** — the wrap-up machinery now runs inside the auto-review gate on its own cadence (`fateIntervalTurns`, compaction unconditional): local entries are audited, classified and partitioned while the session is still running; governed actions are consulted first (one dialog, decline cooldown), covered/operational entries archive silently, compaction applies only silent archives and defers governed actions with an audit record; every decision lands in `reviews.jsonl` and applied actions get a follow-up notice. Apply writes are shared with the wrap-up command (byte-identical proposals)
|
|
429
|
-
- **2026-08-19 research-wave precursors (done)**:
|
|
430
|
-
- **goal-blocked wrap-up (D3)** — a goal stuck in `blocked` for `goalBlockedWrapupTurns` consecutive gate runs (default 3) triggers one local-fate assessment, so the blocked encounter is distilled before the session moves on; the streak resets on any non-blocked run and after each assessment, and declined proposals follow the normal fate cooldown (never nagged). Disable with `goalBlockedWrapupTurns: 0`
|
|
431
|
-
- **failure-signature aggregation (D1 observation layer)** — `/evolve failures` counts every failed review-gate record and benchmark failed cell by deterministic failure class (`rubric-decrypt` / `executor` / `reviewer` / `material-drift` / `gate` / `max-tokens` / …), the data layer a future failure-signature Refiner would route on
|
|
432
|
-
- **bootstrap-update experiment scaffold (D2)** — [`docs/experiment-bootstrap.md`](docs/experiment-bootstrap.md) designs a ≤3-round controlled experiment (fixed reference → evolve harness → candidate) to test whether a refined harness accelerates the next run; `scripts/benchmark-trend.sh` extracts the per-run trend table (overall / totalDurationMs / failed / case-hash consistency) from scoreboards
|
|
433
|
-
- **2026-08-17 gap P0 (done)**:
|
|
434
|
-
- **evaluator/scorer separation** — benchmark evaluation is now two-stage (gap A1): the executor performs the task and records concrete evidence without ever seeing the rubric; an independent reviewer grades that evidence against the rubric (the only branch that decrypts it). The assessed agent can no longer optimize toward or self-grade against the criteria.
|
|
435
|
-
- **failure-cell protocol** — cells carry `status: ok|failed` (gap A2): failed units are excluded from every mean and counted, and the acceptance rule rejects rounds with failures beyond `maxFailedCells` (0 default) instead of averaging a zero into the mean. Scoreboard status/run surfaces failed counts and per-cell reasons.
|
|
436
|
-
- **trace evidence pointer** — each cell records the executor's session id (gap A4), so a score drills back to the exact transcript that earned it
|
|
437
|
-
- **2026-08-18 gap P1 (done)**:
|
|
438
|
-
- **runtime evidence verification (A3)** — cells now record actual `provider`, `model`, and `caseHash` (SHA-256 prefix of statement + rubric) written by the host, not the model; material changes between reference and candidate runs are detected and re-mark the affected candidate cells as failed (version_changed semantics, `score.flagMaterialDrift`), so a drifted round can never be accepted
|
|
439
|
-
- **entry usage statistics (B1)** — injection counts are durably tracked per entry in `<baseDir>/evolve/usage.json`; `evolve_list` shows usage counts; `zeroUsageEntries()` surfaces never-injected local entries as archive candidates
|
|
440
|
-
- **automatic staleness detection (B2)** — entries with zero injection usage AND old recency are flagged `stale` in wrap-up candidates; the LLM assessor is instructed to prefer "archive" for stale entries
|
|
441
|
-
- **2026-08-18 gap P2 (done)**:
|
|
442
|
-
- **case lifecycle + quality gate (A5)** — cases follow a `draft → calibrating → frozen` state machine; `casecheck` runs mechanical quality validation (capability contract, distinguisher, shortcuts); `pilot` performs a single-run calibration; `freeze` locks a case as a formal baseline (requires quality gate pass); `meta` sets case metadata fields
|
|
443
|
-
- **entry directory view (B3)** — the injection block now includes a lightweight directory of ALL non-archived entries (id + title, one line each) when entries exceed the curated cap, giving the model a zero-cost overview
|
|
444
|
-
- **review model separation (C1)** — `reviewModel` config option lets the review gate use a cheaper model than the main agent
|
|
445
|
-
- **blast-radius annotations (C2)** — every edit now carries a `blastRadius` field (`general` / `project` / `session`); the planner is instructed to annotate it and the parser validates values
|
|
446
|
-
- **duration tracking (C3)** — each evaluation cell records `durationMs` (wall-clock time); aggregate totals and decision reports show timing comparison
|
|
447
|
-
- **2026-08-18 code refactoring (done)**:
|
|
448
|
-
- **circular dependency break (P1-1)** — extracted `skill-render.ts` to decouple `skill.ts ↔ skillquality.ts`
|
|
449
|
-
- **LLM call deduplication (P1-2)** — extracted `llm-text.ts` with shared `streamText()` (~107 lines removed from review/planner/wrapup)
|
|
450
|
-
- **config type derivation (P2-1)** — `EvolveConfig` now derived from schemastery schema via `Schemastery.TypeT` (eliminated 20-line handwritten interface)
|
|
451
|
-
- **command.ts split (P2-2)** — 860-line god file split into `goal-command.ts`, `mount-command.ts`, `benchmark-command.ts`, `wrapup-command.ts`
|
|
452
|
-
- **P3 cleanups** — `questionServiceOf()` cast dedup (4 sites), dead exports removed, contradictory comments fixed
|
|
127
|
+
## Docs & provenance
|
|
453
128
|
|
|
454
|
-
|
|
129
|
+
- Design: [`docs/design.md`](docs/design.md) · Pitfalls: [`docs/FAQ.md`](docs/FAQ.md) · Gap analysis: [`docs/gap-analysis.md`](docs/gap-analysis.md) · D2 experiment: [`docs/experiment-bootstrap.md`](docs/experiment-bootstrap.md)
|
|
130
|
+
- Lineage: **penguin-harness** (concept; Apache-2.0) — report in [`docs/research/penguin-harness-self-evolution.md`](docs/research/penguin-harness-self-evolution.md); **prime-agent `/refine`** (engineering shape; MIT) — annotated reference source in [`docs/research/prime-agent-refinement.ts`](docs/research/prime-agent-refinement.ts). This package is an original implementation on the DSH plugin surface.
|
|
455
131
|
|
|
456
132
|
## License
|
|
457
133
|
|
|
458
|
-
MIT
|
|
134
|
+
[MIT](LICENSE)
|