dsh-continual-evolve 0.3.0 → 0.5.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -7,452 +7,128 @@
7
7
  [![CI](https://github.com/ZK-Andy/dsh-continual-evolve/actions/workflows/ci.yml/badge.svg)](https://github.com/ZK-Andy/dsh-continual-evolve/actions/workflows/ci.yml)
8
8
  [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
9
9
  [![Node](https://img.shields.io/badge/node-%5E22.19%20%7C%7C%20%3E%3D24-339933)](package.json)
10
- [![Tests](https://img.shields.io/badge/tests-401%20passing-brightgreen)]()
11
- [![Status](https://img.shields.io/badge/status-all%20phases%20complete%20%C2%B7%20maintenance-ff69b4)]()
10
+ [![Tests](https://img.shields.io/badge/tests-543%20passing-brightgreen)]()
12
11
 
13
- Continual self-evolution for [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness): a versioned, auditable, rollback-safe layer of harness state — prompt notes, memories, skills, and subagent specs — refined from session trajectories.
12
+ Continual self-evolution for [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness): a versioned, auditable, rollback-safe harness state layer — prompt notes, memories, skills, subagent specs — refined from session trajectories.
14
13
 
15
- > **Status: all phases complete; in long-term maintenance.** Phases 1–3
16
- > shipped the full evolution loop: the pure-core engine, model tools and
17
- > the `/evolve` command, the automatic review gate (turn-interval +
18
- > compaction checkpoints, human approval for global edits), real
19
- > system-prompt injection (prompt notes + delegation specs, zero token
20
- > cost when empty), and the benchmark-driven validation loop (code-owned
21
- > scoring, non-regressive acceptance, rubric ACL). Since then the plugin
22
- > keeps growing with usage-driven enhancements — the memory layer (ranked
23
- > injection, trajectory citations, archive), per-installation rubric keys,
24
- > plugin-owned file logging, the session wrap-up (`/evolve wrapup`), and
25
- > the gate's automatic local-fate dimension (local entries get a promoted
26
- > or archived exit on the gate's own cadence — consulted first, never
27
- > written silently). See the Roadmap for the full shipped and candidate
28
- > lists.
29
-
30
- ## Background
31
-
32
- This project started as a research question: *can a harness improve itself,
33
- and what would a production-grade version look like?* Three lines of evidence
34
- shaped the answer:
35
-
36
- - **penguin-harness** demonstrated the concept (benchmark → evaluate →
37
- optimize → accept/rollback) but with **zero code-level enforcement** — every
38
- guarantee was a prompt contract. Its report (`docs/research/`) became the
39
- hardening checklist this project implements.
40
- - **prime-agent `/refine`** proved the engineering shape: versioned harness
41
- entries, atomic persistence, optimistic concurrency, inverse-op rollback.
42
- This package is an original implementation of that shape on the DSH plugin
43
- surface.
44
- - Academic work (Self-Harness, AHE, HarnessOpt-Bench) supplied the discipline:
45
- frozen evaluation runtime, code-owned aggregation, non-regressive
46
- acceptance.
47
-
48
- The result: **the model proposes, the code guarantees.** Every mechanical
49
- safety property (schema validation, snapshots, versioning, audit trail,
50
- acceptance decisions) is enforced in code — never by asking the model to
51
- behave.
14
+ **The model proposes, the code guarantees.** Every mechanical safety property — schema validation, atomic writes, snapshots, versioning, audit trail, acceptance decisions — is enforced in code, never by prompt discipline.
52
15
 
53
16
  ## Why
54
17
 
55
- Agents accumulate reusable experience in every session — repeated failures, durable facts, reusable procedures and then forget it at the next turn or session. This plugin makes that experience first-class persistent state:
56
-
57
- - **Versioned entries** keyed by kind (`prompt` / `memory` / `skill` / `subagent`), each with a recorded provenance and version
58
- - **Evidence trail**: every refinement appends an event carrying `trigger / changes / evidence / outcome`
59
- - **Deterministic rollback**: inverse edits are generated from applied results — no LLM re-guessing
60
- - **Code-enforced safety**, not prompt discipline: schema validation, atomic writes, corrupt-file degrade, optimistic concurrency, immutable base system prompt
61
- - **Local (session) and global (cross-session) scopes** with merge semantics
62
-
63
- ## Design provenance
64
-
65
- Inspired by three bodies of work (see [`docs/design.md`](docs/design.md)):
18
+ Agents accumulate reusable experience (repeated failures, durable facts, reusable procedures) and forget it next session. This plugin turns that experience into first-class state:
66
19
 
67
- - **prime-agent `/refine`** (MIT): the state model, atomic persistence, optimistic concurrency, per-edit validation, and inverse-op rollback this package implements annotated reference source in [`docs/research/prime-agent-refinement.ts`](docs/research/prime-agent-refinement.ts). The code here is an original implementation, written for the DSH plugin surface.
68
- - **penguin-harness** (Apache-2.0): the benchmark-driven evolution loop research report in [`docs/research/penguin-harness-self-evolution.md`](docs/research/penguin-harness-self-evolution.md); its prompt-only contracts are the anti-pattern this package hardens.
69
- - Academic: Self-Harness (arXiv 2606.09498), AHE (arXiv 2604.25850), HarnessOpt-Bench (arXiv 2608.06301).
20
+ - **Local scope** per session; **global scope** across sessions with merge semanticsplus mechanical promotion guards so only portable, substantial, non-duplicate knowledge reaches global
21
+ - **Deterministic rollback**: inverse edits generated from applied results no LLM re-guessing
22
+ - **Benchmark loop**: candidate refinements are evaluated against frozen cases by a separate scorer before acceptance (rubric encrypted at rest)
23
+ - **Store hygiene**: `/evolve consolidate` turns write-time conflict hints and zero-use staleness into one approved, fully reversible batch of archives
70
24
 
71
- ## Tech stack
25
+ ## How it works
72
26
 
73
- | Layer | Choice |
74
- |---|---|
75
- | Language | TypeScript (strict, ES2024, ESM) |
76
- | Runtime | Node `^22.19.0 \|\| >=24.0.0` (matches DSH) |
77
- | Plugin seam | `@deepseek-ai/cordis` (`name` / `apply` / `inject` entry) |
78
- | Package manager | pnpm (DSH ecosystem standard) |
79
- | Build | `tsc` → `lib/` (main `lib/index.js`, types `lib/index.d.ts`) |
80
- | Tests | Vitest |
81
- | Lint | oxlint (DSH official repo convention) |
82
- | License | MIT |
83
-
84
- ## Project layout
85
-
86
- ```
87
- dsh-continual-evolve/
88
- ├── package.json # exports / files / engines / scripts + dsh.bundle manifest
89
- ├── cordis.patch.yml # bundle patch (dsh plugin add activates on install)
90
- ├── tsconfig.json / .oxlintrc.json / .editorconfig / .gitignore
91
- ├── LICENSE / README.md / README.zh.md
92
- ├── docs/
93
- │ ├── design.md # full design doc (incl. hardening matrix)
94
- │ └── research/ # penguin-harness report + prime-agent reference source
95
- ├── src/
96
- │ ├── index.ts # cordis plugin entry (service mount + wiring)
97
- │ ├── types.ts # HarnessState / entry / edit / result types
98
- │ ├── state.ts # atomic persistence, corrupt degrade, merge, concurrency
99
- │ ├── validate.ts # code-enforced edit validation
100
- │ ├── apply.ts # per-edit apply pass with optimistic locking
101
- │ ├── rollback.ts # deterministic inverse-op rollback
102
- │ ├── plan.ts # proposal JSON parsing (truncation-aware)
103
- │ ├── tool.ts # evolve_* model-facing tools (5)
104
- │ ├── command.ts # /evolve command dispatcher + shared utilities
105
- │ ├── goal-command.ts # /evolve goal subcommand handler
106
- │ ├── mount-command.ts # /evolve mount + unmount subcommand handlers
107
- │ ├── benchmark-command.ts # /evolve benchmark subcommand handler
108
- │ ├── wrapup-command.ts # /evolve wrapup subcommand handler
109
- │ ├── planner.ts # ctx.llm planner
110
- │ ├── llm-text.ts # unified streaming-text helper (BlockAssembler + finish check)
111
- │ ├── render.ts # bounded prompt rendering
112
- │ ├── inject.ts # dynamic system-prompt section (prompt notes + delegation specs, ranked injection)
113
- │ ├── source.ts # trajectory citations (sessionId + event seqs of distilled entries)
114
- │ ├── auto.ts # auto-review gate (turn/compaction triggers + audit, global-aware view, local-fate phase)
115
- │ ├── fate.ts # gate local-fate dimension — auto promote/archive of local entries (consulted first, cooldown)
116
- │ ├── notify.ts # gate visibility — follow-up notice after an approved auto-refine
117
- │ ├── goal.ts # goal-driven evolution rounds (/evolve goal)
118
- │ ├── review.ts # gate LLM judgment (declines local duplicates of globally covered topics)
119
- │ ├── approval.ts # human approval for global edits
120
- │ ├── skill.ts # skill materialization ($DSH_HOME/skills/)
121
- │ ├── skill-render.ts # shared skill rendering (skillNameOf + renderSkillMarkdown, breaks circular dependency)
122
- │ ├── skillquality.ts # skill standard in the loop (skill-creator template reading + frontmatter code checks)
123
- │ ├── mount.ts # hot-mounted skill plugins (loader.create + boot restore)
124
- │ ├── benchmark.ts # benchmark store + CellScore types (with runtime evidence fields)
125
- │ ├── rubric.ts # rubric ACL (AES-256-GCM envelopes, auto-generated local key)
126
- │ ├── logfile.ts # plugin-owned file logging (JSONL exporter + rotation)
127
- │ ├── score.ts # code-owned aggregation + acceptance rule
128
- │ ├── evaluate.ts # two-stage evaluation runner (executor evidence → independent reviewer) + failure-cell protocol + runtime verification
129
- │ ├── pool.ts # bounded-concurrency worker pool for evaluation runs
130
- │ ├── store.ts # store layout + snapshots + result history
131
- │ ├── service.ts # evolution engine (onApplied hook)
132
- │ ├── usage.ts # entry injection usage tracking (durable counts, staleness detection)
133
- │ ├── failures.ts # failure-signature aggregation (gate + benchmark failures by class, /evolve failures)
134
- │ └── wrapup.ts # session wrap-up lifecycle (promote / split-promote → global, guarded archive; shared proposal builders; staleness signal)
135
- └── test/ # 28 files, 401 tests
136
- ```
27
+ 1. **Sediment** the model creates entries via `evolve_add`, or the automatic review gate proposes them from the session trajectory (turn-interval + compaction checkpoints).
28
+ 2. **Guard** — code-enforced validation: edit schema, blast-radius/scope coherence, and the promotion policy (project-scoped markers, thin content, near-duplicate detection keep the global store clean). Global creates that near-duplicate an existing entry are rejected at write time (≥0.8 similarity); moderate overlaps carry a `conflictHint` for later consolidation.
29
+ 3. **Approve** global writes require explicit human approval; local-fate proposals are consulted before they land.
30
+ 4. **Apply & inject** atomic apply with snapshot + audit event. Prompt notes and delegation specs inject into the system prompt (capped, relevance-ranked, zero tokens when empty); memories/skills appear as a capped directory index.
31
+ 5. **Validate & roll back** benchmarks score candidates against frozen cases; rejected candidates roll back deterministically.
137
32
 
138
33
  ## Install
139
34
 
140
35
  ```bash
141
36
  # from npm (installs and activates — ships its own bundle patch)
142
- dsh plugin --profile web add dsh-continual-evolve
143
-
144
- # or from source (first GitHub installs require approving the allowBuilds build step)
145
- dsh plugin --profile web add github:ZK-Andy/dsh-continual-evolve
146
- ```
147
-
148
- Swap `web` for your profile name (`headless`, or a custom profile).
149
-
150
- ## In-session usage (after restart)
151
-
152
- ```
153
- /evolve help + current local store
154
- /evolve list [global] list entries
155
- /evolve history applied refinements (ids for rollback)
156
- /evolve rollback <id> deterministically revert a refinement
157
- /evolve plan [msg] LLM planner against the current store
158
- /evolve wrapup assess this session's local entries: promote reusable ones to the
159
- global store (approval required), archive session-specific ones
160
- /evolve archive <id> hide an entry from injection (data kept, restorable)
161
- /evolve unarchive <id> restore an archived entry
162
- /evolve log [tail N] [session <id>] show the recent plugin log (default 50 lines; optional per-session filter)
163
- /evolve failures aggregated failure counts (review-gate + benchmark, by class — D1 observation layer)
164
- /evolve export <path> backup the local store to JSON
165
- /evolve import <path> restore a store from an export file
166
- /evolve mount <skillId> hot-mount a skill entry as a live cordis plugin (tool: skill_<name>)
167
- /evolve mount list list hot-mounted plugins (restored on boot)
168
- /evolve unmount <id> remove a hot-mounted plugin
169
- /evolve goal show the evolution goal (round-driven auto-review)
170
- /evolve goal <objective> create/update the evolution goal — while active, the review gate runs EVERY round
171
- /evolve goal done complete the evolution goal
172
- ```
173
-
174
- Model-facing tools: `evolve_list`, `evolve_add`, `evolve_update`, `evolve_delete`, `evolve_rollback`.
175
-
176
- ## Memory layer
177
-
178
- Beyond the persisted store itself, four features keep injected memory
179
- "understanding you" as entries grow (gap analysis vs. Mem0 / Letta / Zep /
180
- LangMem; no external services — everything is pure functions):
181
-
182
- - **Ranked injection** — when a kind holds more than the 6-entry cap, the
183
- injected block no longer shows the fixed first six: entries are scored by
184
- relevance to the agent's most recent direct user messages (keyword/BM25
185
- level: title hits weigh 2×) and then by recency (`updated_at`, 30-day
186
- half-life), so the freshest *and most relevant* entries fill the cap. The
187
- empty-store zero-token behavior is unchanged.
188
- - **Trajectory citations** — every newly created entry records
189
- `metadata.sourceSession` + `metadata.sourceSeqs` pointing at the direct
190
- user messages it was distilled from (DSH sessions are event-sourced with
191
- contiguous seqs, so the citation expands back into the durable session
192
- log). Listings show `src=<sessionId>:<seqs>`; old entries are not migrated
193
- and never error.
194
- - **Archive** — `/evolve archive <id>` hides an entry from injection
195
- (`metadata.archivedAt`, data kept, rollback-compatible) and
196
- `/evolve unarchive <id>` restores it. Archived entries are marked
197
- `[archived]` in `evolve_list` and skipped by injection; the overflow count
198
- excludes them.
199
- - **Session wrap-up** — a session's local entries otherwise become orphans when
200
- it ends (later sessions never see them). `/evolve wrapup` gives them an exit.
201
- Each entry is audited mechanically — global-coverage is judged by **title
202
- similarity only** (a bare id collision with a different title is intentionally
203
- NOT coverage; the actual matching global titles are shown to the assessor so
204
- it judges against real content) — then classified as `promote` / `archive` /
205
- `keep`. Promotions move reusable entries into the global store **through the
206
- human approval gate**, keeping their trajectory citation and adding a
207
- `sourcedFromLocal=<session>:<id>` back-link; the local copy is stamped
208
- `promotedTo` and retired from injection so it is never offered again.
209
- **Split promotion** (A-form): a mixed entry (durable facts + session snapshot)
210
- can be archived while carrying a cleaned `promote` sub-object — only the
211
- durable part lands globally, the snapshot stays in the archive. A **symmetric
212
- archive guard** requires user confirmation before an archive that is NOT
213
- globally covered AND was distilled from real user messages hides that content
214
- from future sessions (over-archiving gets the same protection as
215
- over-writing); operational entries still archive silently. Everything stays
216
- snapshot/versioned/rollbackable.
217
- - **Gate local-fate (automatic wrap-up)** — the same wrap-up machinery now
218
- runs inside the auto-review gate on its own cadence (`fateIntervalTurns`),
219
- so local entries get their exit while the session is still running instead
220
- of waiting for a manual `/evolve wrapup`. On each due gate run the audited
221
- candidates are classified by the assessor and partitioned by the same
222
- deterministic guards; the user is consulted FIRST before anything governed
223
- lands (one dialog covering promotes, split promotions and review-required
224
- archives — the consultSkillEdits pattern, with a decline cooldown). Covered
225
- or operational entries still archive silently, and at compaction the gate
226
- never opens a dialog: only silent archives apply, governed actions are
227
- deferred with an audit record pointing at `/evolve wrapup`. Every fate
228
- decision lands in `reviews.jsonl` (`approved` / `declined` / `deferred` /
229
- `assessed` / `failed`) and applied actions are visible via a follow-up
230
- notice. Apply writes are byte-identical to the wrap-up command (shared
231
- proposal builders).
232
- - **Global-aware gate** — the auto-review gate and planner judge the merged
233
- global + local state with every entry's real scope labeled, so a topic
234
- already covered by a global entry is declined instead of being re-sedimented
235
- as a local duplicate.
236
-
237
- ## Skill standard in the loop
37
+ dsh plugin add dsh-continual-evolve
238
38
 
239
- The planner and the auto-review gate are raw `ctx.llm` calls they do not
240
- live in an agent session, so they cannot load skills through the `skill`
241
- tool. To keep self-evolved skills on the quality bar, the plugin references
242
- the **skill-creator** / **skill-audit** skills (user-level skills distilled
243
- by the author from the official deepseek-harness 11 skills; template facts
244
- verified against deepseek-harness `47f9438`) at runtime — they stay the
245
- single source of truth on disk, nothing is copied:
246
-
247
- - Every planning call receives a `<skill_quality_standard>` block: the
248
- `skill-creator/references/template.md` facts when those skills are
249
- installed (`<dshHome>/skills/`), or a builtin distilled guide otherwise
250
- (~1KB, low-frequency calls). The planner must ground skill proposals in a
251
- REAL trigger scenario from the trajectory, must not duplicate the
252
- official 11 skills or existing entries, and self-checks every proposed
253
- skill against the 7 structural features.
254
- - The gate judges skill-related trajectories against the skill-audit
255
- dimensions (frontmatter routing, structural features, paragraph skeleton,
256
- duplication) and declines proposals that would not meet the standard.
257
- - The mechanical frontmatter rules of `validate-frontmatter.mjs` are
258
- code-enforced at apply time: skill bodies must not open with a second
259
- `---` block (it would shadow the generated frontmatter), and resource
260
- references may not escape the skill directory. After materialization the
261
- rendered SKILL.md is re-checked and dangling `references/`/`scripts/`
262
- links are logged as warnings.
263
- - **Two skill forms** — `executable` skills keep the python reference
264
- contract (hot-mountable as tools); `guidance` skills are SKILL.md
265
- documents with no reference, the form for recurring multi-step workflows
266
- (session start/end routines, handoff procedures). Code enforces the
267
- split: a guidance skill must NOT carry a reference or arguments contract.
268
- - **User-governed skill creation** — the gate never writes a skill
269
- silently: when the planner proposes skill edits, the user is asked
270
- (固化/不固化) before they land; a rejected candidate is not offered
271
- again within a cooldown window. The rest of a proposal proceeds
272
- regardless, so memory/prompt distillation is never blocked by a skill
273
- decision.
274
-
275
- ## Logging
276
-
277
- Plugin-owned file logging: every cordis log message (from this plugin or any
278
- other) is appended to `<dshHome>/evolve/plugin.log` as JSONL (0600, rotated to
279
- `plugin.log.1` past `logMaxBytes`). It works no matter how `dsh web` is
280
- launched — no extra component to install, no startup-script dependency.
281
- View the tail with `/evolve log [tail N]`, or read the file directly:
282
-
283
- ```bash
284
- tail -f ~/.dsh/evolve/plugin.log # live
285
- /evolve log 100 # last 100 lines in the chat
39
+ # or from source (first GitHub installs require approving the allowBuilds step)
40
+ dsh plugin add ZK-Andy/dsh-continual-evolve
286
41
  ```
287
42
 
288
- For live output in a foreground terminal, the official
289
- `@deepseek-ai/cordis-plugin-logger-console` plugin can be added to the
290
- profile (optional; the file log remains the baseline that always exists).
291
-
292
- ## Benchmark-driven validation (Phase 3)
293
-
294
- ```
295
- /evolve benchmark new <title> [runs] create a benchmark (runs = repeats per case, default 1)
296
- /evolve benchmark add-case <bid> <title> <statement> <rubric>
297
- /evolve benchmark list list benchmarks
298
- /evolve benchmark reset <bid> clear the scoreboard (re-run reference)
299
- /evolve benchmark status <bid> scoreboard + decisions
300
- /evolve benchmark run <bid> evaluate current state → reference
301
- /evolve benchmark run <bid> candidate <refinementId> evaluate post-refinement state → decide
302
- /evolve benchmark casecheck <bid> quality-gate check all cases
303
- /evolve benchmark pilot <bid> <cid> single pilot run for calibration
304
- /evolve benchmark freeze <bid> <cid> freeze a case as formal baseline
305
- /evolve benchmark meta <bid> <cid> <field> <value> set case metadata (capability/distinguisher/shortcuts)
306
- ```
307
-
308
- The loop: freeze a reference score → evolve a candidate (`/evolve plan`) →
309
- run the same case × run matrix against the post-refinement state → the
310
- **code-owned** acceptance rule keeps the candidate only if the overall mean
311
- strictly improves with no case regressing (Self-Harness style).
312
-
313
- **Evaluator/scorer separation (two-stage, gap A1)** — each case × run unit
314
- is a PAIR of fresh subagents:
315
-
316
- 1. the **executor** performs the task with its tools and records **concrete
317
- evidence** of what it did and found — it NEVER sees the rubric, so the
318
- agent under test cannot optimize toward or self-grade against the grading
319
- criteria;
320
- 2. an **independent reviewer** grades that evidence strictly against the
321
- rubric (the only branch that receives the decrypted rubric), eliminating
322
- the "self-produced and self-scored" bias.
323
-
324
- Each cell records the executor's session id, so a score can be drilled back
325
- to the exact transcript that produced it (trace evidence pointer, gap A4).
43
+ Restart `dsh web` after installing or updating.
326
44
 
327
- **Failure-cell protocol (gap A2)** — a unit that cannot produce a score
328
- (rubric decrypt error, executor/reviewer crash, protocol error) is recorded
329
- as a **failed** cell, NEVER a zero: aggregation excludes failed cells from
330
- every mean and counts them (`/evolve benchmark status` shows `(N failed)`),
331
- and the acceptance rule rejects a round with more failed cells than
332
- `maxFailedCells` (0 by default) instead of silently averaging a 0 into the
333
- mean.
45
+ ## Usage
334
46
 
335
- Aggregation and decisions live in `src/score.ts`. Rubric isolation is by
336
- construction (the planner never sees rubric files, and the executor branch
337
- never decrypts); a rejection is recorded in the scoreboard and the
338
- refinement is rolled back automatically (`autoRollbackOnReject`, on by
339
- default).
47
+ Commands (in-session):
340
48
 
341
- Ready-to-use seed cases are in [`examples/`](examples/) — copy-paste the
342
- statement and rubric to get started in under a minute.
343
-
344
- ### Real recorded run (ACCEPT)
345
-
346
- A live `dsh web` session, one case, one candidate the first genuine
347
- acceptance:
348
-
349
- | Step | Command | Outcome |
350
- |---|---|---|
351
- | reference | `/evolve benchmark run lint_convention` | **90** the evaluator agent actually grepped the harness store and reported *"lint/ruff/eslint/mypy appear in zero entries"* |
352
- | candidate | `/evolve plan 记住:写代码前必须先运行适用的 lint 检查` | creates `memory:convention_lint_before_code` |
353
- | re-evaluate | `/evolve benchmark run lint_convention candidate <id>` | **100** — evaluator ran `evolve_list`, hit the memory, quoted it verbatim |
354
- | decision | | `overall: 90 100` · `lint_knowledge: 90 → 100` · **DECISION: ACCEPTED** |
355
-
356
- The executor does not grade model common sense it inspects the actual
357
- harness state under test (grep, `evolve_list`) and records what it found;
358
- the independent reviewer grades that record. A harness change measurably
359
- moves the score. Earlier runs in the same session
360
- produced honest `REJECTED` decisions (0 → 0 stub cases, and 100 → 100
361
- where the baseline was already perfect).
49
+ | Command | Effect |
50
+ |---|---|
51
+ | `/evolve` | help + current local store |
52
+ | `/evolve list · history · rollback <id>` | inspect and revert (add `global` for the cross-session store) |
53
+ | `/evolve plan [msg]` | run the LLM planner against the store |
54
+ | `/evolve wrapup` | assess this session's local entries: promote / archive / keep |
55
+ | `/evolve archive · unarchive · demote <id>` | hide from injection (data kept, restorable) — `demote` targets global noise |
56
+ | `/evolve consolidate [apply]` | report (or apply) one batch archive of conflict-hinted + stale zero-use global entries |
57
+ | `/evolve failures` | aggregated failure classes (gate + benchmark) |
58
+ | `/evolve log [tail N] [session <id>]` | plugin log |
59
+ | `/evolve export · import <path>` | backup / restore a store |
60
+ | `/evolve mount · unmount <skillId>` | hot-mount an executable skill as a live plugin |
61
+ | `/evolve goal [objective · done · block]` | round-driven auto-review goal |
62
+ | `/evolve benchmark …` | case lifecycle, runs, acceptance |
63
+
64
+ Model tools: `evolve_list / add / update / delete / rollback`.
65
+
66
+ Injection shape: prompt notes and delegation specs inject with content (≤6/kind × 180 chars, relevance-ranked). Memories and skills appear as a directory index (`[kind:id] title`, capped at 15 lines with a fold counter) — full text via `evolve_list`. Empty store = zero injected tokens.
362
67
 
363
68
  ## Configuration
364
69
 
365
70
  | Key | Default | Meaning |
366
71
  |---|---|---|
367
72
  | `baseDir` | resolved DSH home | root for the `evolve/` stores |
368
- | `sectionOrder` | 118 | system-prompt section order |
369
- | `autoReview` | `false` | enable the automatic review gate (costs a cheap model call per interval) |
370
- | `reviewIntervalTurns` | 6 | gate runs when this many turns passed since the last review |
371
- | `maxReviewInputChars` | 40000 | trajectory slice handed to the gate |
372
- | `reviewBudgetTokens` | 4096 | output budget for the gate call |
373
- | `notifyOnAutoReview` | `true` | after an approved gate run that applied edits, queue a visible follow-up notice in the session (persisted entries + rollback command) |
374
- | `requireGlobalApproval` | `true` | cross-session (global) edits ask the user for "批准" before applying |
375
- | `skillsDir` | `<dshHome>/skills` | root where skill entries materialize as SKILL.md bundles |
376
- | `rubricKey` | auto-generated local key file (`<dshHome>/evolve/rubric.key`, 0600) → dev fallback | passphrase for AES-256-GCM rubric encryption (benchmark rubrics never touch the disk in plaintext). When unset, the plugin generates a random per-installation key file on first use — every install gets its own key, no setup needed; `DSH_EVOLVE_RUBRIC_KEY` is the environment-variable override |
377
- | `logToFile` | `true` | write all cordis log messages to `<dshHome>/evolve/plugin.log` (JSONL, 0600) — plugin-owned logging works with any launch method, no extra component to install |
378
- | `logLevel` | `1` | file log level: 0=error, 1=info, 2=warn, 3=debug |
379
- | `logMaxBytes` | 5 MiB | rotate the log to `plugin.log.1` when it exceeds this size |
380
- | `autoRollbackOnReject` | `true` | after a benchmark decision rejects a candidate, roll the refinement back automatically (same engine path as `/evolve rollback` — deterministic, snapshotted, audited) |
381
- | `localFate` | `true` | gate local-fate dimension: the gate audits the session's local entries on its own cadence and proposes promote/archive — consulted first, never written silently (only meaningful with `autoReview`) |
382
- | `fateIntervalTurns` | follows `reviewIntervalTurns` | minimum turns between local-fate assessments on the turn-interval path (compaction is unconditional) |
383
- | `goalBlockedWrapupTurns` | `3` | D3: after this many consecutive gate runs observing the goal phase `blocked`, run one local-fate assessment (`0` disables) |
384
- | `reviewModel` | (agent's own) | optional model override for the review gate (cheaper model); format: `"provider/model"` or just `"model"` |
385
-
386
- Example (profile `cordis.patch.yml`):
73
+ | `autoReview` | `false` | enable the automatic review gate |
74
+ | `reviewIntervalTurns` | `6` | gate cadence on the turn-interval path |
75
+ | `maxReviewInputChars` | `40000` | trajectory slice handed to the gate |
76
+ | `reviewBudgetTokens` | `4096` | output budget for the gate call |
77
+ | `notifyOnAutoReview` | `true` | visible follow-up notice after an applied gate run |
78
+ | `requireGlobalApproval` | `true` | global edits ask for explicit approval |
79
+ | `localFate` | `true` | gate audits local entries and proposes promote/archive (consulted, never silent) |
80
+ | `fateIntervalTurns` | follows `reviewIntervalTurns` | minimum turns between fate assessments |
81
+ | `goalBlockedWrapupTurns` | `3` | consecutive blocked-goal gate runs trigger one fate assessment (`0` disables) |
82
+ | `promotionBlockPatterns` | POSIX paths, session ids, `~/.dsh` | content matching these is project-scoped and never promoted to global |
83
+ | `promotionMinChars` | `100` | whole promotions below this length stay local |
84
+ | `injectionDirectoryLines` | `15` | entry-directory lines per build before folding into a counter |
85
+ | `sectionOrder` | `118` | system-prompt section order |
86
+ | `skillsDir` | `<dshHome>/skills` | where skill entries materialize as SKILL.md bundles |
87
+ | `rubricKey` | auto-generated key file | AES-256-GCM passphrase for benchmark rubrics (`DSH_EVOLVE_RUBRIC_KEY` overrides) |
88
+ | `logToFile` / `logLevel` / `logMaxBytes` | `true` / `1` / 5 MiB | plugin-owned JSONL file log with rotation |
89
+ | `autoRollbackOnReject` | `true` | deterministic rollback after a benchmark rejection |
90
+ | `reviewModel` | agent's own | optional cheaper model for the gate (`"provider/model"`) |
91
+
92
+ Example profile patch:
387
93
 
388
94
  ```yaml
389
- - insert:
390
- - id: continual-evolve
391
- name: 'dsh-continual-evolve'
392
- config:
393
- autoReview: true
394
- reviewIntervalTurns: 6
95
+ - id: continual-evolve
96
+ config:
97
+ autoReview: true
98
+ reviewIntervalTurns: 6
395
99
  ```
396
100
 
397
101
  ## Development
398
102
 
399
103
  ```bash
400
- pnpm install # install dev deps
401
- pnpm dev # tsc --watch
402
- pnpm build # tsc -> lib/
403
- pnpm test # vitest run
404
- pnpm lint # oxlint src test
104
+ pnpm install && pnpm build # deps + tsc -> lib/
105
+ pnpm test # vitest (543 tests)
106
+ pnpm test:coverage # v8 coverage, thresholds enforced in CI
107
+ pnpm lint # oxlint src test
405
108
  ```
406
109
 
407
- Hit a wall? See [`docs/FAQ.md`](docs/FAQ.md) — real failure/fix records (service planes, schema DSL, structured output, gate counting, verifying prompt injection).
408
-
409
- Where we still lag behind prime-agent `/refine` and penguin-harness — and what to build next: [`docs/gap-analysis.md`](docs/gap-analysis.md) (P0+P1+P2+P3 shipped: evaluator/scorer separation, failure-cell protocol, runtime provenance verification + material-drift detection, usage statistics, auto-decay, case lifecycle + quality gate, entry directory view, review model separation, blast-radius annotations, duration tracking, evolve_complete events, seed benchmark; D1 observation layer + D3 goal-blocked trigger shipped; remaining: cross-process sync on demand + D1/D2 full engineering pending experiment data).
410
-
411
- ## Roadmap
110
+ Project layout:
412
111
 
413
- **Shipped**
112
+ ```
113
+ ├── src/ # engine, tools, commands, gate, fate, benchmark, usage…
114
+ ├── test/ # vitest suites (33 files)
115
+ ├── lib/ # build output (tsc)
116
+ ├── docs/
117
+ │ ├── design.md # full design doc (hardening matrix)
118
+ │ ├── FAQ.md # real failure/fix records
119
+ │ ├── gap-analysis.md # vs prime-agent /refine + penguin-harness
120
+ │ ├── experiment-bootstrap.md
121
+ │ ├── archive/ # closed point-in-time reports
122
+ │ └── research/ # penguin report + prime-agent annotated source
123
+ ├── examples/README.md # seed benchmark cases
124
+ └── .agents/ # AI collaboration layer (AGENTS.md, skills, ADR notes)
125
+ ```
414
126
 
415
- - **Phases 1–3 (done)**: pure-core engine (state model, validation, apply, rollback, proposal parsing) → `evolve_*` tools + `/evolve` command + `ctx.llm` planner → auto-refine review gate (turn-interval + compaction checkpoints, visible follow-up notices), global-scope human approval, executable skills, real system-prompt injection (prompt notes + delegation specs, inherited by subagents), benchmark-driven validation loop (code-owned scoreboard, non-regressive acceptance, rubric isolation by construction), hot-mounted skill plugins, goal-driven evolution rounds.
416
- - **2026-08 maintenance wave (done)**:
417
- - **memory layer** — ranked injection (relevance + recency scoring fills the per-kind cap), trajectory citations (`metadata.sourceSession` + `sourceSeqs`, shown as `src=session:seqs`), archive/unarchive (`/evolve archive <id>`, injection skips archived entries), global-aware gate (declines local duplicates of globally covered topics)
418
- - **per-installation rubric key** — auto-generated local key file (`<dshHome>/evolve/rubric.key`, 0600); no more publicly known dev key
419
- - **plugin-owned file logging** — every cordis log message lands in `<dshHome>/evolve/plugin.log` (JSONL, 0600, rotated), viewable via `/evolve log`; works with any launch method, no extra component to install
420
- - **trajectory-grounded planning** — `/evolve plan` (and every planner call, including the gate's refine step) now reads the session trajectory: the caller's recent direct user messages are extracted from the session log and fed to the planner as a `<session_trajectory>` block, so proposals are grounded in what the user actually said (explicit `trajectory` overrides; empty trajectory is omitted at zero cost)
421
- - **gate-proposed archiving** — stale entries are a first-class refine target: the planner can emit `action: "archive"` (kind + id only), which stamps `metadata.archivedAt` through the normal apply path — snapshot, version bump, audit event, and a deterministic rollback inverse that restores the pre-archive state. Archive hides from injection but never deletes; re-archiving an archived entry is rejected, and the base system prompt stays immutable
422
- - **automatic rollback on benchmark rejection** — the acceptance loop is closed: when the code-owned decision rejects a candidate, the refinement is reverted automatically through the same engine path as `/evolve rollback` (deterministic inverse edits, snapshotted and audited; configurable via `autoRollbackOnReject`, on by default). Failures report the manual fallback instead of throwing
423
- - **per-session log filtering** — `/evolve log [tail N] [session <id>]` keeps only the lines mentioning a given session id (exact token match, drawn from the rendered message and raw args); gate records now carry the session id in their log line
424
- - **skill standard in the loop** — the planner and gate now author and judge skill entries against the skill-creator/skill-audit standard (author-distilled from the official deepseek-harness 11 skills): every plan call injects the `template.md` facts (builtin distilled guide as fallback) as `<skill_quality_standard>`; apply code-enforces the frontmatter mechanics (no shadowing `---`, no escaping resource refs); materialized SKILL.md files are re-checked and dangling resource references are logged;
425
- - **guidance skills + user-governed creation** — a second skill form (SKILL.md documents without a python reference) lets recurring workflows be proposed as skills; the gate offers every auto-created skill to the user (固化/不固化) before it lands, with a rejection cooldown — skills grow under governance, never silently
426
- - **2026-08-17 wrap-up wave (done)**:
427
- - **`/evolve wrapup`** — a session's local entries get a real exit at session end: mechanical audit (local candidates + global-coverage detection; coverage judges **title similarity only** — a bare id collision with a different title is deliberately NOT coverage, and the real matching global titles are shown to the assessor) → LLM classification (`promote` / `archive` / `keep` + A-form split promotion: archive a mixed entry while promoting a cleaned durable sub-object) → deterministic guards re-checked at apply time (promote can never write a global duplicate; the symmetric archive guard requires user confirmation before an uncovered, user-sourced archive hides content; splits that duplicate a global topic drop to plain archive) → one human approval gate for every global create
428
- - **gate local-fate dimension** — the wrap-up machinery now runs inside the auto-review gate on its own cadence (`fateIntervalTurns`, compaction unconditional): local entries are audited, classified and partitioned while the session is still running; governed actions are consulted first (one dialog, decline cooldown), covered/operational entries archive silently, compaction applies only silent archives and defers governed actions with an audit record; every decision lands in `reviews.jsonl` and applied actions get a follow-up notice. Apply writes are shared with the wrap-up command (byte-identical proposals)
429
- - **2026-08-19 research-wave precursors (done)**:
430
- - **goal-blocked wrap-up (D3)** — a goal stuck in `blocked` for `goalBlockedWrapupTurns` consecutive gate runs (default 3) triggers one local-fate assessment, so the blocked encounter is distilled before the session moves on; the streak resets on any non-blocked run and after each assessment, and declined proposals follow the normal fate cooldown (never nagged). Disable with `goalBlockedWrapupTurns: 0`
431
- - **failure-signature aggregation (D1 observation layer)** — `/evolve failures` counts every failed review-gate record and benchmark failed cell by deterministic failure class (`rubric-decrypt` / `executor` / `reviewer` / `material-drift` / `gate` / `max-tokens` / …), the data layer a future failure-signature Refiner would route on
432
- - **bootstrap-update experiment scaffold (D2)** — [`docs/experiment-bootstrap.md`](docs/experiment-bootstrap.md) designs a ≤3-round controlled experiment (fixed reference → evolve harness → candidate) to test whether a refined harness accelerates the next run; `scripts/benchmark-trend.sh` extracts the per-run trend table (overall / totalDurationMs / failed / case-hash consistency) from scoreboards
433
- - **2026-08-17 gap P0 (done)**:
434
- - **evaluator/scorer separation** — benchmark evaluation is now two-stage (gap A1): the executor performs the task and records concrete evidence without ever seeing the rubric; an independent reviewer grades that evidence against the rubric (the only branch that decrypts it). The assessed agent can no longer optimize toward or self-grade against the criteria.
435
- - **failure-cell protocol** — cells carry `status: ok|failed` (gap A2): failed units are excluded from every mean and counted, and the acceptance rule rejects rounds with failures beyond `maxFailedCells` (0 default) instead of averaging a zero into the mean. Scoreboard status/run surfaces failed counts and per-cell reasons.
436
- - **trace evidence pointer** — each cell records the executor's session id (gap A4), so a score drills back to the exact transcript that earned it
437
- - **2026-08-18 gap P1 (done)**:
438
- - **runtime evidence verification (A3)** — cells now record actual `provider`, `model`, and `caseHash` (SHA-256 prefix of statement + rubric) written by the host, not the model; material changes between reference and candidate runs are detected and re-mark the affected candidate cells as failed (version_changed semantics, `score.flagMaterialDrift`), so a drifted round can never be accepted
439
- - **entry usage statistics (B1)** — injection counts are durably tracked per entry in `<baseDir>/evolve/usage.json`; `evolve_list` shows usage counts; `zeroUsageEntries()` surfaces never-injected local entries as archive candidates
440
- - **automatic staleness detection (B2)** — entries with zero injection usage AND old recency are flagged `stale` in wrap-up candidates; the LLM assessor is instructed to prefer "archive" for stale entries
441
- - **2026-08-18 gap P2 (done)**:
442
- - **case lifecycle + quality gate (A5)** — cases follow a `draft → calibrating → frozen` state machine; `casecheck` runs mechanical quality validation (capability contract, distinguisher, shortcuts); `pilot` performs a single-run calibration; `freeze` locks a case as a formal baseline (requires quality gate pass); `meta` sets case metadata fields
443
- - **entry directory view (B3)** — the injection block now includes a lightweight directory of ALL non-archived entries (id + title, one line each) when entries exceed the curated cap, giving the model a zero-cost overview
444
- - **review model separation (C1)** — `reviewModel` config option lets the review gate use a cheaper model than the main agent
445
- - **blast-radius annotations (C2)** — every edit now carries a `blastRadius` field (`general` / `project` / `session`); the planner is instructed to annotate it and the parser validates values
446
- - **duration tracking (C3)** — each evaluation cell records `durationMs` (wall-clock time); aggregate totals and decision reports show timing comparison
447
- - **2026-08-18 code refactoring (done)**:
448
- - **circular dependency break (P1-1)** — extracted `skill-render.ts` to decouple `skill.ts ↔ skillquality.ts`
449
- - **LLM call deduplication (P1-2)** — extracted `llm-text.ts` with shared `streamText()` (~107 lines removed from review/planner/wrapup)
450
- - **config type derivation (P2-1)** — `EvolveConfig` now derived from schemastery schema via `Schemastery.TypeT` (eliminated 20-line handwritten interface)
451
- - **command.ts split (P2-2)** — 860-line god file split into `goal-command.ts`, `mount-command.ts`, `benchmark-command.ts`, `wrapup-command.ts`
452
- - **P3 cleanups** — `questionServiceOf()` cast dedup (4 sites), dead exports removed, contradictory comments fixed
127
+ ## Docs & provenance
453
128
 
454
- The upcoming/candidates list is empty for now future work is driven by real usage.
129
+ - Design: [`docs/design.md`](docs/design.md) · Pitfalls: [`docs/FAQ.md`](docs/FAQ.md) · Gap analysis: [`docs/gap-analysis.md`](docs/gap-analysis.md) · D2 experiment: [`docs/experiment-bootstrap.md`](docs/experiment-bootstrap.md)
130
+ - Lineage: **penguin-harness** (concept; Apache-2.0) — report in [`docs/research/penguin-harness-self-evolution.md`](docs/research/penguin-harness-self-evolution.md); **prime-agent `/refine`** (engineering shape; MIT) — annotated reference source in [`docs/research/prime-agent-refinement.ts`](docs/research/prime-agent-refinement.ts). This package is an original implementation on the DSH plugin surface.
455
131
 
456
132
  ## License
457
133
 
458
- MIT. Independent project — not affiliated with DeepSeek.
134
+ [MIT](LICENSE)