dsh-continual-evolve 0.2.0 → 0.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (66) hide show
  1. package/README.md +83 -371
  2. package/README.zh.md +84 -235
  3. package/lib/apply.js +8 -2
  4. package/lib/approval.d.ts +6 -0
  5. package/lib/approval.js +9 -1
  6. package/lib/auto.d.ts +55 -4
  7. package/lib/auto.js +61 -5
  8. package/lib/benchmark-command.d.ts +9 -0
  9. package/lib/benchmark-command.js +333 -0
  10. package/lib/benchmark.d.ts +70 -0
  11. package/lib/benchmark.js +107 -1
  12. package/lib/command.d.ts +3 -0
  13. package/lib/command.js +62 -441
  14. package/lib/evaluate.d.ts +7 -0
  15. package/lib/evaluate.js +22 -7
  16. package/lib/evolve-event.d.ts +38 -0
  17. package/lib/evolve-event.js +49 -0
  18. package/lib/failures.d.ts +39 -0
  19. package/lib/failures.js +170 -0
  20. package/lib/fate.d.ts +5 -2
  21. package/lib/fate.js +13 -8
  22. package/lib/goal-command.d.ts +7 -0
  23. package/lib/goal-command.js +37 -0
  24. package/lib/index.d.ts +51 -25
  25. package/lib/index.js +33 -1
  26. package/lib/inject.d.ts +24 -1
  27. package/lib/inject.js +93 -5
  28. package/lib/llm-text.d.ts +30 -0
  29. package/lib/llm-text.js +49 -0
  30. package/lib/mount-command.d.ts +10 -0
  31. package/lib/mount-command.js +48 -0
  32. package/lib/plan.js +5 -0
  33. package/lib/planner.d.ts +1 -1
  34. package/lib/planner.js +13 -39
  35. package/lib/promotion.d.ts +62 -0
  36. package/lib/promotion.js +102 -0
  37. package/lib/render.d.ts +1 -3
  38. package/lib/render.js +0 -4
  39. package/lib/review.d.ts +4 -1
  40. package/lib/review.js +10 -38
  41. package/lib/rollback.d.ts +1 -3
  42. package/lib/rollback.js +0 -8
  43. package/lib/score.d.ts +15 -0
  44. package/lib/score.js +74 -5
  45. package/lib/service.d.ts +2 -2
  46. package/lib/service.js +7 -3
  47. package/lib/skill-render.d.ts +23 -0
  48. package/lib/skill-render.js +68 -0
  49. package/lib/skill.d.ts +2 -5
  50. package/lib/skill.js +2 -29
  51. package/lib/skillquality.d.ts +1 -2
  52. package/lib/skillquality.js +2 -2
  53. package/lib/state.js +6 -1
  54. package/lib/store.d.ts +1 -3
  55. package/lib/store.js +0 -7
  56. package/lib/tool.js +22 -1
  57. package/lib/types.d.ts +8 -0
  58. package/lib/usage.d.ts +45 -0
  59. package/lib/usage.js +115 -0
  60. package/lib/validate.d.ts +12 -2
  61. package/lib/validate.js +26 -1
  62. package/lib/wrapup-command.d.ts +9 -0
  63. package/lib/wrapup-command.js +212 -0
  64. package/lib/wrapup.d.ts +29 -15
  65. package/lib/wrapup.js +69 -42
  66. package/package.json +10 -8
package/README.md CHANGED
@@ -7,414 +7,126 @@
7
7
  [![CI](https://github.com/ZK-Andy/dsh-continual-evolve/actions/workflows/ci.yml/badge.svg)](https://github.com/ZK-Andy/dsh-continual-evolve/actions/workflows/ci.yml)
8
8
  [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
9
9
  [![Node](https://img.shields.io/badge/node-%5E22.19%20%7C%7C%20%3E%3D24-339933)](package.json)
10
- [![Tests](https://img.shields.io/badge/tests-302%20passing-brightgreen)]()
11
- [![Status](https://img.shields.io/badge/status-all%20phases%20complete%20%C2%B7%20maintenance-ff69b4)]()
10
+ [![Tests](https://img.shields.io/badge/tests-527%20passing-brightgreen)]()
12
11
 
13
- Continual self-evolution for [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness): a versioned, auditable, rollback-safe layer of harness state — prompt notes, memories, skills, and subagent specs — refined from session trajectories.
12
+ Continual self-evolution for [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness): a versioned, auditable, rollback-safe harness state layer — prompt notes, memories, skills, subagent specs — refined from session trajectories.
14
13
 
15
- > **Status: all phases complete; in long-term maintenance.** Phases 1–3
16
- > shipped the full evolution loop: the pure-core engine, model tools and
17
- > the `/evolve` command, the automatic review gate (turn-interval +
18
- > compaction checkpoints, human approval for global edits), real
19
- > system-prompt injection (prompt notes + delegation specs, zero token
20
- > cost when empty), and the benchmark-driven validation loop (code-owned
21
- > scoring, non-regressive acceptance, rubric ACL). Since then the plugin
22
- > keeps growing with usage-driven enhancements — the memory layer (ranked
23
- > injection, trajectory citations, archive), per-installation rubric keys,
24
- > plugin-owned file logging, the session wrap-up (`/evolve wrapup`), and
25
- > the gate's automatic local-fate dimension (local entries get a promoted
26
- > or archived exit on the gate's own cadence — consulted first, never
27
- > written silently). See the Roadmap for the full shipped and candidate
28
- > lists.
29
-
30
- ## Background
31
-
32
- This project started as a research question: *can a harness improve itself,
33
- and what would a production-grade version look like?* Three lines of evidence
34
- shaped the answer:
35
-
36
- - **penguin-harness** demonstrated the concept (benchmark → evaluate →
37
- optimize → accept/rollback) but with **zero code-level enforcement** — every
38
- guarantee was a prompt contract. Its report (`docs/research/`) became the
39
- hardening checklist this project implements.
40
- - **prime-agent `/refine`** proved the engineering shape: versioned harness
41
- entries, atomic persistence, optimistic concurrency, inverse-op rollback.
42
- This package is an original implementation of that shape on the DSH plugin
43
- surface.
44
- - Academic work (Self-Harness, AHE, HarnessOpt-Bench) supplied the discipline:
45
- frozen evaluation runtime, code-owned aggregation, non-regressive
46
- acceptance.
47
-
48
- The result: **the model proposes, the code guarantees.** Every mechanical
49
- safety property (schema validation, snapshots, versioning, audit trail,
50
- acceptance decisions) is enforced in code — never by asking the model to
51
- behave.
14
+ **The model proposes, the code guarantees.** Every mechanical safety property — schema validation, atomic writes, snapshots, versioning, audit trail, acceptance decisions — is enforced in code, never by prompt discipline.
52
15
 
53
16
  ## Why
54
17
 
55
- Agents accumulate reusable experience in every session — repeated failures, durable facts, reusable procedures and then forget it at the next turn or session. This plugin makes that experience first-class persistent state:
56
-
57
- - **Versioned entries** keyed by kind (`prompt` / `memory` / `skill` / `subagent`), each with a recorded provenance and version
58
- - **Evidence trail**: every refinement appends an event carrying `trigger / changes / evidence / outcome`
59
- - **Deterministic rollback**: inverse edits are generated from applied results — no LLM re-guessing
60
- - **Code-enforced safety**, not prompt discipline: schema validation, atomic writes, corrupt-file degrade, optimistic concurrency, immutable base system prompt
61
- - **Local (session) and global (cross-session) scopes** with merge semantics
62
-
63
- ## Design provenance
64
-
65
- Inspired by three bodies of work (see [`docs/design.md`](docs/design.md)):
18
+ Agents accumulate reusable experience (repeated failures, durable facts, reusable procedures) and forget it next session. This plugin turns that experience into first-class state:
66
19
 
67
- - **prime-agent `/refine`** (MIT): the state model, atomic persistence, optimistic concurrency, per-edit validation, and inverse-op rollback this package implements annotated reference source in [`docs/research/prime-agent-refinement.ts`](docs/research/prime-agent-refinement.ts). The code here is an original implementation, written for the DSH plugin surface.
68
- - **penguin-harness** (Apache-2.0): the benchmark-driven evolution loop research report in [`docs/research/penguin-harness-self-evolution.md`](docs/research/penguin-harness-self-evolution.md); its prompt-only contracts are the anti-pattern this package hardens.
69
- - Academic: Self-Harness (arXiv 2606.09498), AHE (arXiv 2604.25850), HarnessOpt-Bench (arXiv 2608.06301).
20
+ - **Local scope** per session; **global scope** across sessions with merge semanticsplus mechanical promotion guards so only portable, substantial, non-duplicate knowledge reaches global
21
+ - **Deterministic rollback**: inverse edits generated from applied results no LLM re-guessing
22
+ - **Benchmark loop**: candidate refinements are evaluated against frozen cases by a separate scorer before acceptance (rubric encrypted at rest)
70
23
 
71
- ## Tech stack
24
+ ## How it works
72
25
 
73
- | Layer | Choice |
74
- |---|---|
75
- | Language | TypeScript (strict, ES2024, ESM) |
76
- | Runtime | Node `^22.19.0 \|\| >=24.0.0` (matches DSH) |
77
- | Plugin seam | `@deepseek-ai/cordis` (`name` / `apply` / `inject` entry) |
78
- | Package manager | pnpm (DSH ecosystem standard) |
79
- | Build | `tsc` → `lib/` (main `lib/index.js`, types `lib/index.d.ts`) |
80
- | Tests | Vitest |
81
- | Lint | oxlint (DSH official repo convention) |
82
- | License | MIT |
83
-
84
- ## Project layout
85
-
86
- ```
87
- dsh-continual-evolve/
88
- ├── package.json # exports / files / engines / scripts + dsh.bundle manifest
89
- ├── cordis.patch.yml # bundle patch (dsh plugin add activates on install)
90
- ├── tsconfig.json / .oxlintrc.json / .editorconfig / .gitignore
91
- ├── LICENSE / README.md / README.zh.md
92
- ├── docs/
93
- │ ├── design.md # full design doc (incl. hardening matrix)
94
- │ └── research/ # penguin-harness report + prime-agent reference source
95
- ├── src/
96
- │ ├── index.ts # cordis plugin entry (service mount + wiring)
97
- │ ├── types.ts # HarnessState / entry / edit / result types
98
- │ ├── state.ts # atomic persistence, corrupt degrade, merge, concurrency
99
- │ ├── validate.ts # code-enforced edit validation
100
- │ ├── apply.ts # per-edit apply pass with optimistic locking
101
- │ ├── rollback.ts # deterministic inverse-op rollback
102
- │ ├── plan.ts # proposal JSON parsing (truncation-aware)
103
- │ ├── tool.ts # evolve_* model-facing tools (5)
104
- │ ├── command.ts # /evolve command (incl. benchmark subcommands)
105
- │ ├── planner.ts # ctx.llm planner
106
- │ ├── render.ts # bounded prompt rendering
107
- │ ├── inject.ts # dynamic system-prompt section (prompt notes + delegation specs, ranked injection)
108
- │ ├── source.ts # trajectory citations (sessionId + event seqs of distilled entries)
109
- │ ├── auto.ts # auto-review gate (turn/compaction triggers + audit, global-aware view, local-fate phase)
110
- │ ├── fate.ts # gate local-fate dimension — auto promote/archive of local entries (consulted first, cooldown)
111
- │ ├── notify.ts # gate visibility — follow-up notice after an approved auto-refine
112
- │ ├── goal.ts # goal-driven evolution rounds (/evolve goal)
113
- │ ├── review.ts # gate LLM judgment (declines local duplicates of globally covered topics)
114
- │ ├── approval.ts # human approval for global edits
115
- │ ├── skill.ts # skill materialization ($DSH_HOME/skills/)
116
- │ ├── skillquality.ts # skill standard in the loop (skill-creator template reading + frontmatter code checks)
117
- │ ├── mount.ts # hot-mounted skill plugins (loader.create + boot restore)
118
- │ ├── benchmark.ts # benchmark store
119
- │ ├── rubric.ts # rubric ACL (AES-256-GCM envelopes, auto-generated local key)
120
- │ ├── logfile.ts # plugin-owned file logging (JSONL exporter + rotation)
121
- │ ├── score.ts # code-owned aggregation + acceptance rule
122
- │ ├── evaluate.ts # two-stage evaluation runner (executor evidence → independent reviewer) + failure-cell protocol
123
- │ ├── pool.ts # bounded-concurrency worker pool for evaluation runs
124
- │ ├── store.ts # store layout + snapshots + result history
125
- │ ├── service.ts # evolution engine (onApplied hook)
126
- │ └── wrapup.ts # session wrap-up lifecycle (promote / split-promote → global, guarded archive; shared proposal builders)
127
- └── test/ # 23 files, 290 tests
128
- ```
26
+ 1. **Sediment** the model creates entries via `evolve_add`, or the automatic review gate proposes them from the session trajectory (turn-interval + compaction checkpoints).
27
+ 2. **Guard** — code-enforced validation: edit schema, blast-radius/scope coherence, and the promotion policy (project-scoped markers, thin content, near-duplicate detection keep the global store clean).
28
+ 3. **Approve** global writes require explicit human approval; local-fate proposals are consulted before they land.
29
+ 4. **Apply & inject** atomic apply with snapshot + audit event. Prompt notes and delegation specs inject into the system prompt (capped, relevance-ranked, zero tokens when empty); memories/skills appear as a capped directory index.
30
+ 5. **Validate & roll back** benchmarks score candidates against frozen cases; rejected candidates roll back deterministically.
129
31
 
130
32
  ## Install
131
33
 
132
34
  ```bash
133
35
  # from npm (installs and activates — ships its own bundle patch)
134
- dsh plugin --profile web add dsh-continual-evolve
36
+ dsh plugin add dsh-continual-evolve
135
37
 
136
- # or from source (first GitHub installs require approving the allowBuilds build step)
137
- dsh plugin --profile web add github:ZK-Andy/dsh-continual-evolve
38
+ # or from source (first GitHub installs require approving the allowBuilds step)
39
+ dsh plugin add ZK-Andy/dsh-continual-evolve
138
40
  ```
139
41
 
140
- Swap `web` for your profile name (`headless`, or a custom profile).
42
+ Restart `dsh web` after installing or updating.
141
43
 
142
- ## In-session usage (after restart)
44
+ ## Usage
143
45
 
144
- ```
145
- /evolve help + current local store
146
- /evolve list [global] list entries
147
- /evolve history applied refinements (ids for rollback)
148
- /evolve rollback <id> deterministically revert a refinement
149
- /evolve plan [msg] LLM planner against the current store
150
- /evolve wrapup assess this session's local entries: promote reusable ones to the
151
- global store (approval required), archive session-specific ones
152
- /evolve archive <id> hide an entry from injection (data kept, restorable)
153
- /evolve unarchive <id> restore an archived entry
154
- /evolve log [tail N] [session <id>] show the recent plugin log (default 50 lines; optional per-session filter)
155
- /evolve export <path> backup the local store to JSON
156
- /evolve import <path> restore a store from an export file
157
- /evolve mount <skillId> hot-mount a skill entry as a live cordis plugin (tool: skill_<name>)
158
- /evolve mount list list hot-mounted plugins (restored on boot)
159
- /evolve unmount <id> remove a hot-mounted plugin
160
- /evolve goal show the evolution goal (round-driven auto-review)
161
- /evolve goal <objective> create/update the evolution goal — while active, the review gate runs EVERY round
162
- /evolve goal done complete the evolution goal
163
- ```
164
-
165
- Model-facing tools: `evolve_list`, `evolve_add`, `evolve_update`, `evolve_delete`, `evolve_rollback`.
166
-
167
- ## Memory layer
168
-
169
- Beyond the persisted store itself, four features keep injected memory
170
- "understanding you" as entries grow (gap analysis vs. Mem0 / Letta / Zep /
171
- LangMem; no external services — everything is pure functions):
172
-
173
- - **Ranked injection** — when a kind holds more than the 6-entry cap, the
174
- injected block no longer shows the fixed first six: entries are scored by
175
- relevance to the agent's most recent direct user messages (keyword/BM25
176
- level: title hits weigh 2×) and then by recency (`updated_at`, 30-day
177
- half-life), so the freshest *and most relevant* entries fill the cap. The
178
- empty-store zero-token behavior is unchanged.
179
- - **Trajectory citations** — every newly created entry records
180
- `metadata.sourceSession` + `metadata.sourceSeqs` pointing at the direct
181
- user messages it was distilled from (DSH sessions are event-sourced with
182
- contiguous seqs, so the citation expands back into the durable session
183
- log). Listings show `src=<sessionId>:<seqs>`; old entries are not migrated
184
- and never error.
185
- - **Archive** — `/evolve archive <id>` hides an entry from injection
186
- (`metadata.archivedAt`, data kept, rollback-compatible) and
187
- `/evolve unarchive <id>` restores it. Archived entries are marked
188
- `[archived]` in `evolve_list` and skipped by injection; the overflow count
189
- excludes them.
190
- - **Session wrap-up** — a session's local entries otherwise become orphans when
191
- it ends (later sessions never see them). `/evolve wrapup` gives them an exit.
192
- Each entry is audited mechanically — global-coverage is judged by **title
193
- similarity only** (a bare id collision with a different title is intentionally
194
- NOT coverage; the actual matching global titles are shown to the assessor so
195
- it judges against real content) — then classified as `promote` / `archive` /
196
- `keep`. Promotions move reusable entries into the global store **through the
197
- human approval gate**, keeping their trajectory citation and adding a
198
- `sourcedFromLocal=<session>:<id>` back-link; the local copy is stamped
199
- `promotedTo` and retired from injection so it is never offered again.
200
- **Split promotion** (A-form): a mixed entry (durable facts + session snapshot)
201
- can be archived while carrying a cleaned `promote` sub-object — only the
202
- durable part lands globally, the snapshot stays in the archive. A **symmetric
203
- archive guard** requires user confirmation before an archive that is NOT
204
- globally covered AND was distilled from real user messages hides that content
205
- from future sessions (over-archiving gets the same protection as
206
- over-writing); operational entries still archive silently. Everything stays
207
- snapshot/versioned/rollbackable.
208
- - **Gate local-fate (automatic wrap-up)** — the same wrap-up machinery now
209
- runs inside the auto-review gate on its own cadence (`fateIntervalTurns`),
210
- so local entries get their exit while the session is still running instead
211
- of waiting for a manual `/evolve wrapup`. On each due gate run the audited
212
- candidates are classified by the assessor and partitioned by the same
213
- deterministic guards; the user is consulted FIRST before anything governed
214
- lands (one dialog covering promotes, split promotions and review-required
215
- archives — the consultSkillEdits pattern, with a decline cooldown). Covered
216
- or operational entries still archive silently, and at compaction the gate
217
- never opens a dialog: only silent archives apply, governed actions are
218
- deferred with an audit record pointing at `/evolve wrapup`. Every fate
219
- decision lands in `reviews.jsonl` (`approved` / `declined` / `deferred` /
220
- `assessed` / `failed`) and applied actions are visible via a follow-up
221
- notice. Apply writes are byte-identical to the wrap-up command (shared
222
- proposal builders).
223
- - **Global-aware gate** — the auto-review gate and planner judge the merged
224
- global + local state with every entry's real scope labeled, so a topic
225
- already covered by a global entry is declined instead of being re-sedimented
226
- as a local duplicate.
46
+ Commands (in-session):
227
47
 
228
- ## Skill standard in the loop
229
-
230
- The planner and the auto-review gate are raw `ctx.llm` calls — they do not
231
- live in an agent session, so they cannot load skills through the `skill`
232
- tool. To keep self-evolved skills on the quality bar, the plugin references
233
- the **skill-creator** / **skill-audit** skills (user-level skills distilled
234
- by the author from the official deepseek-harness 11 skills; template facts
235
- verified against deepseek-harness `47f9438`) at runtime they stay the
236
- single source of truth on disk, nothing is copied:
237
-
238
- - Every planning call receives a `<skill_quality_standard>` block: the
239
- `skill-creator/references/template.md` facts when those skills are
240
- installed (`<dshHome>/skills/`), or a builtin distilled guide otherwise
241
- (~1KB, low-frequency calls). The planner must ground skill proposals in a
242
- REAL trigger scenario from the trajectory, must not duplicate the
243
- official 11 skills or existing entries, and self-checks every proposed
244
- skill against the 7 structural features.
245
- - The gate judges skill-related trajectories against the skill-audit
246
- dimensions (frontmatter routing, structural features, paragraph skeleton,
247
- duplication) and declines proposals that would not meet the standard.
248
- - The mechanical frontmatter rules of `validate-frontmatter.mjs` are
249
- code-enforced at apply time: skill bodies must not open with a second
250
- `---` block (it would shadow the generated frontmatter), and resource
251
- references may not escape the skill directory. After materialization the
252
- rendered SKILL.md is re-checked and dangling `references/`/`scripts/`
253
- links are logged as warnings.
254
- - **Two skill forms** — `executable` skills keep the python reference
255
- contract (hot-mountable as tools); `guidance` skills are SKILL.md
256
- documents with no reference, the form for recurring multi-step workflows
257
- (session start/end routines, handoff procedures). Code enforces the
258
- split: a guidance skill must NOT carry a reference or arguments contract.
259
- - **User-governed skill creation** — the gate never writes a skill
260
- silently: when the planner proposes skill edits, the user is asked
261
- (固化/不固化) before they land; a rejected candidate is not offered
262
- again within a cooldown window. The rest of a proposal proceeds
263
- regardless, so memory/prompt distillation is never blocked by a skill
264
- decision.
265
-
266
- ## Logging
267
-
268
- Plugin-owned file logging: every cordis log message (from this plugin or any
269
- other) is appended to `<dshHome>/evolve/plugin.log` as JSONL (0600, rotated to
270
- `plugin.log.1` past `logMaxBytes`). It works no matter how `dsh web` is
271
- launched — no extra component to install, no startup-script dependency.
272
- View the tail with `/evolve log [tail N]`, or read the file directly:
273
-
274
- ```bash
275
- tail -f ~/.dsh/evolve/plugin.log # live
276
- /evolve log 100 # last 100 lines in the chat
277
- ```
278
-
279
- For live output in a foreground terminal, the official
280
- `@deepseek-ai/cordis-plugin-logger-console` plugin can be added to the
281
- profile (optional; the file log remains the baseline that always exists).
282
-
283
- ## Benchmark-driven validation (Phase 3)
284
-
285
- ```
286
- /evolve benchmark new <title> [runs] create a benchmark (runs = repeats per case, default 1)
287
- /evolve benchmark add-case <bid> <title> <statement> <rubric>
288
- /evolve benchmark list list benchmarks
289
- /evolve benchmark reset <bid> clear the scoreboard (re-run reference)
290
- /evolve benchmark status <bid> scoreboard + decisions
291
- /evolve benchmark run <bid> evaluate current state → reference
292
- /evolve benchmark run <bid> candidate <refinementId> evaluate post-refinement state → decide
293
- ```
294
-
295
- The loop: freeze a reference score → evolve a candidate (`/evolve plan`) →
296
- run the same case × run matrix against the post-refinement state → the
297
- **code-owned** acceptance rule keeps the candidate only if the overall mean
298
- strictly improves with no case regressing (Self-Harness style).
299
-
300
- **Evaluator/scorer separation (two-stage, gap A1)** — each case × run unit
301
- is a PAIR of fresh subagents:
302
-
303
- 1. the **executor** performs the task with its tools and records **concrete
304
- evidence** of what it did and found — it NEVER sees the rubric, so the
305
- agent under test cannot optimize toward or self-grade against the grading
306
- criteria;
307
- 2. an **independent reviewer** grades that evidence strictly against the
308
- rubric (the only branch that receives the decrypted rubric), eliminating
309
- the "self-produced and self-scored" bias.
310
-
311
- Each cell records the executor's session id, so a score can be drilled back
312
- to the exact transcript that produced it (trace evidence pointer, gap A4).
313
-
314
- **Failure-cell protocol (gap A2)** — a unit that cannot produce a score
315
- (rubric decrypt error, executor/reviewer crash, protocol error) is recorded
316
- as a **failed** cell, NEVER a zero: aggregation excludes failed cells from
317
- every mean and counts them (`/evolve benchmark status` shows `(N failed)`),
318
- and the acceptance rule rejects a round with more failed cells than
319
- `maxFailedCells` (0 by default) instead of silently averaging a 0 into the
320
- mean.
321
-
322
- Aggregation and decisions live in `src/score.ts`. Rubric isolation is by
323
- construction (the planner never sees rubric files, and the executor branch
324
- never decrypts); a rejection is recorded in the scoreboard and the
325
- refinement is rolled back automatically (`autoRollbackOnReject`, on by
326
- default).
327
-
328
- ### Real recorded run (ACCEPT)
329
-
330
- A live `dsh web` session, one case, one candidate — the first genuine
331
- acceptance:
332
-
333
- | Step | Command | Outcome |
334
- |---|---|---|
335
- | reference | `/evolve benchmark run lint_convention` | **90** — the evaluator agent actually grepped the harness store and reported *"lint/ruff/eslint/mypy appear in zero entries"* |
336
- | candidate | `/evolve plan 记住:写代码前必须先运行适用的 lint 检查` | creates `memory:convention_lint_before_code` |
337
- | re-evaluate | `/evolve benchmark run lint_convention candidate <id>` | **100** — evaluator ran `evolve_list`, hit the memory, quoted it verbatim |
338
- | decision | — | `overall: 90 → 100` · `lint_knowledge: 90 → 100` · **DECISION: ACCEPTED** |
339
-
340
- The executor does not grade model common sense — it inspects the actual
341
- harness state under test (grep, `evolve_list`) and records what it found;
342
- the independent reviewer grades that record. A harness change measurably
343
- moves the score. Earlier runs in the same session
344
- produced honest `REJECTED` decisions (0 → 0 stub cases, and 100 → 100
345
- where the baseline was already perfect).
48
+ | Command | Effect |
49
+ |---|---|
50
+ | `/evolve` | help + current local store |
51
+ | `/evolve list · history · rollback <id>` | inspect and revert (add `global` for the cross-session store) |
52
+ | `/evolve plan [msg]` | run the LLM planner against the store |
53
+ | `/evolve wrapup` | assess this session's local entries: promote / archive / keep |
54
+ | `/evolve archive · unarchive · demote <id>` | hide from injection (data kept, restorable) `demote` targets global noise |
55
+ | `/evolve failures` | aggregated failure classes (gate + benchmark) |
56
+ | `/evolve log [tail N] [session <id>]` | plugin log |
57
+ | `/evolve export · import <path>` | backup / restore a store |
58
+ | `/evolve mount · unmount <skillId>` | hot-mount an executable skill as a live plugin |
59
+ | `/evolve goal [objective · done · block]` | round-driven auto-review goal |
60
+ | `/evolve benchmark …` | case lifecycle, runs, acceptance |
61
+
62
+ Model tools: `evolve_list / add / update / delete / rollback`.
63
+
64
+ Injection shape: prompt notes and delegation specs inject with content (≤6/kind × 180 chars, relevance-ranked). Memories and skills appear as a directory index (`[kind:id] title`, capped at 15 lines with a fold counter) — full text via `evolve_list`. Empty store = zero injected tokens.
346
65
 
347
66
  ## Configuration
348
67
 
349
68
  | Key | Default | Meaning |
350
69
  |---|---|---|
351
70
  | `baseDir` | resolved DSH home | root for the `evolve/` stores |
352
- | `sectionOrder` | 118 | system-prompt section order |
353
- | `autoReview` | `false` | enable the automatic review gate (costs a cheap model call per interval) |
354
- | `reviewIntervalTurns` | 6 | gate runs when this many turns passed since the last review |
355
- | `maxReviewInputChars` | 40000 | trajectory slice handed to the gate |
356
- | `reviewBudgetTokens` | 4096 | output budget for the gate call |
357
- | `notifyOnAutoReview` | `true` | after an approved gate run that applied edits, queue a visible follow-up notice in the session (persisted entries + rollback command) |
358
- | `requireGlobalApproval` | `true` | cross-session (global) edits ask the user for "批准" before applying |
359
- | `skillsDir` | `<dshHome>/skills` | root where skill entries materialize as SKILL.md bundles |
360
- | `rubricKey` | auto-generated local key file (`<dshHome>/evolve/rubric.key`, 0600) → dev fallback | passphrase for AES-256-GCM rubric encryption (benchmark rubrics never touch the disk in plaintext). When unset, the plugin generates a random per-installation key file on first use — every install gets its own key, no setup needed; `DSH_EVOLVE_RUBRIC_KEY` is the environment-variable override |
361
- | `logToFile` | `true` | write all cordis log messages to `<dshHome>/evolve/plugin.log` (JSONL, 0600) — plugin-owned logging works with any launch method, no extra component to install |
362
- | `logLevel` | `1` | file log level: 0=error, 1=info, 2=warn, 3=debug |
363
- | `logMaxBytes` | 5 MiB | rotate the log to `plugin.log.1` when it exceeds this size |
364
- | `autoRollbackOnReject` | `true` | after a benchmark decision rejects a candidate, roll the refinement back automatically (same engine path as `/evolve rollback` — deterministic, snapshotted, audited) |
365
- | `localFate` | `true` | gate local-fate dimension: the gate audits the session's local entries on its own cadence and proposes promote/archive — consulted first, never written silently (only meaningful with `autoReview`) |
366
- | `fateIntervalTurns` | follows `reviewIntervalTurns` | minimum turns between local-fate assessments on the turn-interval path (compaction is unconditional) |
367
-
368
- Example (profile `cordis.patch.yml`):
71
+ | `autoReview` | `false` | enable the automatic review gate |
72
+ | `reviewIntervalTurns` | `6` | gate cadence on the turn-interval path |
73
+ | `maxReviewInputChars` | `40000` | trajectory slice handed to the gate |
74
+ | `reviewBudgetTokens` | `4096` | output budget for the gate call |
75
+ | `notifyOnAutoReview` | `true` | visible follow-up notice after an applied gate run |
76
+ | `requireGlobalApproval` | `true` | global edits ask for explicit approval |
77
+ | `localFate` | `true` | gate audits local entries and proposes promote/archive (consulted, never silent) |
78
+ | `fateIntervalTurns` | follows `reviewIntervalTurns` | minimum turns between fate assessments |
79
+ | `goalBlockedWrapupTurns` | `3` | consecutive blocked-goal gate runs trigger one fate assessment (`0` disables) |
80
+ | `promotionBlockPatterns` | POSIX paths, session ids, `~/.dsh` | content matching these is project-scoped and never promoted to global |
81
+ | `promotionMinChars` | `100` | whole promotions below this length stay local |
82
+ | `injectionDirectoryLines` | `15` | entry-directory lines per build before folding into a counter |
83
+ | `sectionOrder` | `118` | system-prompt section order |
84
+ | `skillsDir` | `<dshHome>/skills` | where skill entries materialize as SKILL.md bundles |
85
+ | `rubricKey` | auto-generated key file | AES-256-GCM passphrase for benchmark rubrics (`DSH_EVOLVE_RUBRIC_KEY` overrides) |
86
+ | `logToFile` / `logLevel` / `logMaxBytes` | `true` / `1` / 5 MiB | plugin-owned JSONL file log with rotation |
87
+ | `autoRollbackOnReject` | `true` | deterministic rollback after a benchmark rejection |
88
+ | `reviewModel` | agent's own | optional cheaper model for the gate (`"provider/model"`) |
89
+
90
+ Example profile patch:
369
91
 
370
92
  ```yaml
371
- - insert:
372
- - id: continual-evolve
373
- name: 'dsh-continual-evolve'
374
- config:
375
- autoReview: true
376
- reviewIntervalTurns: 6
93
+ - id: continual-evolve
94
+ config:
95
+ autoReview: true
96
+ reviewIntervalTurns: 6
377
97
  ```
378
98
 
379
99
  ## Development
380
100
 
381
101
  ```bash
382
- pnpm install # install dev deps
383
- pnpm dev # tsc --watch
384
- pnpm build # tsc -> lib/
385
- pnpm test # vitest run
386
- pnpm lint # oxlint src test
102
+ pnpm install && pnpm build # deps + tsc -> lib/
103
+ pnpm test # vitest (527 tests)
104
+ pnpm test:coverage # v8 coverage, thresholds enforced in CI
105
+ pnpm lint # oxlint src test
387
106
  ```
388
107
 
389
- Hit a wall? See [`docs/FAQ.md`](docs/FAQ.md) — real failure/fix records (service planes, schema DSL, structured output, gate counting, verifying prompt injection).
390
-
391
- Where we still lag behind prime-agent `/refine` and penguin-harness — and what to build next: [`docs/gap-analysis.md`](docs/gap-analysis.md) (P0 shipped: evaluator/scorer separation, failure-cell protocol; next: P1 runtime provenance checks, usage statistics, auto-decay).
392
-
393
- ## Roadmap
108
+ Project layout:
394
109
 
395
- **Shipped**
110
+ ```
111
+ ├── src/ # engine, tools, commands, gate, fate, benchmark, usage…
112
+ ├── test/ # vitest suites (33 files)
113
+ ├── lib/ # build output (tsc)
114
+ ├── docs/
115
+ │ ├── design.md # full design doc (hardening matrix)
116
+ │ ├── FAQ.md # real failure/fix records
117
+ │ ├── gap-analysis.md # vs prime-agent /refine + penguin-harness
118
+ │ ├── experiment-bootstrap.md
119
+ │ ├── archive/ # closed point-in-time reports
120
+ │ └── research/ # penguin report + prime-agent annotated source
121
+ ├── examples/README.md # seed benchmark cases
122
+ └── .agents/ # AI collaboration layer (AGENTS.md, skills, ADR notes)
123
+ ```
396
124
 
397
- - **Phases 1–3 (done)**: pure-core engine (state model, validation, apply, rollback, proposal parsing) → `evolve_*` tools + `/evolve` command + `ctx.llm` planner → auto-refine review gate (turn-interval + compaction checkpoints, visible follow-up notices), global-scope human approval, executable skills, real system-prompt injection (prompt notes + delegation specs, inherited by subagents), benchmark-driven validation loop (code-owned scoreboard, non-regressive acceptance, rubric isolation by construction), hot-mounted skill plugins, goal-driven evolution rounds.
398
- - **2026-08 maintenance wave (done)**:
399
- - **memory layer** — ranked injection (relevance + recency scoring fills the per-kind cap), trajectory citations (`metadata.sourceSession` + `sourceSeqs`, shown as `src=session:seqs`), archive/unarchive (`/evolve archive <id>`, injection skips archived entries), global-aware gate (declines local duplicates of globally covered topics)
400
- - **per-installation rubric key** — auto-generated local key file (`<dshHome>/evolve/rubric.key`, 0600); no more publicly known dev key
401
- - **plugin-owned file logging** — every cordis log message lands in `<dshHome>/evolve/plugin.log` (JSONL, 0600, rotated), viewable via `/evolve log`; works with any launch method, no extra component to install
402
- - **trajectory-grounded planning** — `/evolve plan` (and every planner call, including the gate's refine step) now reads the session trajectory: the caller's recent direct user messages are extracted from the session log and fed to the planner as a `<session_trajectory>` block, so proposals are grounded in what the user actually said (explicit `trajectory` overrides; empty trajectory is omitted at zero cost)
403
- - **gate-proposed archiving** — stale entries are a first-class refine target: the planner can emit `action: "archive"` (kind + id only), which stamps `metadata.archivedAt` through the normal apply path — snapshot, version bump, audit event, and a deterministic rollback inverse that restores the pre-archive state. Archive hides from injection but never deletes; re-archiving an archived entry is rejected, and the base system prompt stays immutable
404
- - **automatic rollback on benchmark rejection** — the acceptance loop is closed: when the code-owned decision rejects a candidate, the refinement is reverted automatically through the same engine path as `/evolve rollback` (deterministic inverse edits, snapshotted and audited; configurable via `autoRollbackOnReject`, on by default). Failures report the manual fallback instead of throwing
405
- - **per-session log filtering** — `/evolve log [tail N] [session <id>]` keeps only the lines mentioning a given session id (exact token match, drawn from the rendered message and raw args); gate records now carry the session id in their log line
406
- - **skill standard in the loop** — the planner and gate now author and judge skill entries against the skill-creator/skill-audit standard (author-distilled from the official deepseek-harness 11 skills): every plan call injects the `template.md` facts (builtin distilled guide as fallback) as `<skill_quality_standard>`; apply code-enforces the frontmatter mechanics (no shadowing `---`, no escaping resource refs); materialized SKILL.md files are re-checked and dangling resource references are logged;
407
- - **guidance skills + user-governed creation** — a second skill form (SKILL.md documents without a python reference) lets recurring workflows be proposed as skills; the gate offers every auto-created skill to the user (固化/不固化) before it lands, with a rejection cooldown — skills grow under governance, never silently
408
- - **2026-08-17 wrap-up wave (done)**:
409
- - **`/evolve wrapup`** — a session's local entries get a real exit at session end: mechanical audit (local candidates + global-coverage detection; coverage judges **title similarity only** — a bare id collision with a different title is deliberately NOT coverage, and the real matching global titles are shown to the assessor) → LLM classification (`promote` / `archive` / `keep` + A-form split promotion: archive a mixed entry while promoting a cleaned durable sub-object) → deterministic guards re-checked at apply time (promote can never write a global duplicate; the symmetric archive guard requires user confirmation before an uncovered, user-sourced archive hides content; splits that duplicate a global topic drop to plain archive) → one human approval gate for every global create
410
- - **gate local-fate dimension** — the wrap-up machinery now runs inside the auto-review gate on its own cadence (`fateIntervalTurns`, compaction unconditional): local entries are audited, classified and partitioned while the session is still running; governed actions are consulted first (one dialog, decline cooldown), covered/operational entries archive silently, compaction applies only silent archives and defers governed actions with an audit record; every decision lands in `reviews.jsonl` and applied actions get a follow-up notice. Apply writes are shared with the wrap-up command (byte-identical proposals)
411
- - **2026-08-17 gap P0 (done)**:
412
- - **evaluator/scorer separation** — benchmark evaluation is now two-stage (gap A1): the executor performs the task and records concrete evidence without ever seeing the rubric; an independent reviewer grades that evidence against the rubric (the only branch that decrypts it). The assessed agent can no longer optimize toward or self-grade against the criteria.
413
- - **failure-cell protocol** — cells carry `status: ok|failed` (gap A2): failed units are excluded from every mean and counted, and the acceptance rule rejects rounds with failures beyond `maxFailedCells` (0 default) instead of averaging a zero into the mean. Scoreboard status/run surfaces failed counts and per-cell reasons.
414
- - **trace evidence pointer** — each cell records the executor's session id (gap A4), so a score drills back to the exact transcript that earned it
125
+ ## Docs & provenance
415
126
 
416
- The upcoming/candidates list is empty for now future work is driven by real usage.
127
+ - Design: [`docs/design.md`](docs/design.md) · Pitfalls: [`docs/FAQ.md`](docs/FAQ.md) · Gap analysis: [`docs/gap-analysis.md`](docs/gap-analysis.md) · D2 experiment: [`docs/experiment-bootstrap.md`](docs/experiment-bootstrap.md)
128
+ - Lineage: **penguin-harness** (concept; Apache-2.0) — report in [`docs/research/penguin-harness-self-evolution.md`](docs/research/penguin-harness-self-evolution.md); **prime-agent `/refine`** (engineering shape; MIT) — annotated reference source in [`docs/research/prime-agent-refinement.ts`](docs/research/prime-agent-refinement.ts). This package is an original implementation on the DSH plugin surface.
417
129
 
418
130
  ## License
419
131
 
420
- MIT. Independent project — not affiliated with DeepSeek.
132
+ [MIT](LICENSE)