dsh-continual-evolve 0.2.0 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +83 -371
- package/README.zh.md +84 -235
- package/lib/apply.js +8 -2
- package/lib/approval.d.ts +6 -0
- package/lib/approval.js +9 -1
- package/lib/auto.d.ts +55 -4
- package/lib/auto.js +61 -5
- package/lib/benchmark-command.d.ts +9 -0
- package/lib/benchmark-command.js +333 -0
- package/lib/benchmark.d.ts +70 -0
- package/lib/benchmark.js +107 -1
- package/lib/command.d.ts +3 -0
- package/lib/command.js +62 -441
- package/lib/evaluate.d.ts +7 -0
- package/lib/evaluate.js +22 -7
- package/lib/evolve-event.d.ts +38 -0
- package/lib/evolve-event.js +49 -0
- package/lib/failures.d.ts +39 -0
- package/lib/failures.js +170 -0
- package/lib/fate.d.ts +5 -2
- package/lib/fate.js +13 -8
- package/lib/goal-command.d.ts +7 -0
- package/lib/goal-command.js +37 -0
- package/lib/index.d.ts +51 -25
- package/lib/index.js +33 -1
- package/lib/inject.d.ts +24 -1
- package/lib/inject.js +93 -5
- package/lib/llm-text.d.ts +30 -0
- package/lib/llm-text.js +49 -0
- package/lib/mount-command.d.ts +10 -0
- package/lib/mount-command.js +48 -0
- package/lib/plan.js +5 -0
- package/lib/planner.d.ts +1 -1
- package/lib/planner.js +13 -39
- package/lib/promotion.d.ts +62 -0
- package/lib/promotion.js +102 -0
- package/lib/render.d.ts +1 -3
- package/lib/render.js +0 -4
- package/lib/review.d.ts +4 -1
- package/lib/review.js +10 -38
- package/lib/rollback.d.ts +1 -3
- package/lib/rollback.js +0 -8
- package/lib/score.d.ts +15 -0
- package/lib/score.js +74 -5
- package/lib/service.d.ts +2 -2
- package/lib/service.js +7 -3
- package/lib/skill-render.d.ts +23 -0
- package/lib/skill-render.js +68 -0
- package/lib/skill.d.ts +2 -5
- package/lib/skill.js +2 -29
- package/lib/skillquality.d.ts +1 -2
- package/lib/skillquality.js +2 -2
- package/lib/state.js +6 -1
- package/lib/store.d.ts +1 -3
- package/lib/store.js +0 -7
- package/lib/tool.js +22 -1
- package/lib/types.d.ts +8 -0
- package/lib/usage.d.ts +45 -0
- package/lib/usage.js +115 -0
- package/lib/validate.d.ts +12 -2
- package/lib/validate.js +26 -1
- package/lib/wrapup-command.d.ts +9 -0
- package/lib/wrapup-command.js +212 -0
- package/lib/wrapup.d.ts +29 -15
- package/lib/wrapup.js +69 -42
- package/package.json +10 -8
package/README.md
CHANGED
|
@@ -7,414 +7,126 @@
|
|
|
7
7
|
[](https://github.com/ZK-Andy/dsh-continual-evolve/actions/workflows/ci.yml)
|
|
8
8
|
[](LICENSE)
|
|
9
9
|
[](package.json)
|
|
10
|
-
[]()
|
|
10
|
+
[]()
|
|
12
11
|
|
|
13
|
-
Continual self-evolution for [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness): a versioned, auditable, rollback-safe
|
|
12
|
+
Continual self-evolution for [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness): a versioned, auditable, rollback-safe harness state layer — prompt notes, memories, skills, subagent specs — refined from session trajectories.
|
|
14
13
|
|
|
15
|
-
|
|
16
|
-
> shipped the full evolution loop: the pure-core engine, model tools and
|
|
17
|
-
> the `/evolve` command, the automatic review gate (turn-interval +
|
|
18
|
-
> compaction checkpoints, human approval for global edits), real
|
|
19
|
-
> system-prompt injection (prompt notes + delegation specs, zero token
|
|
20
|
-
> cost when empty), and the benchmark-driven validation loop (code-owned
|
|
21
|
-
> scoring, non-regressive acceptance, rubric ACL). Since then the plugin
|
|
22
|
-
> keeps growing with usage-driven enhancements — the memory layer (ranked
|
|
23
|
-
> injection, trajectory citations, archive), per-installation rubric keys,
|
|
24
|
-
> plugin-owned file logging, the session wrap-up (`/evolve wrapup`), and
|
|
25
|
-
> the gate's automatic local-fate dimension (local entries get a promoted
|
|
26
|
-
> or archived exit on the gate's own cadence — consulted first, never
|
|
27
|
-
> written silently). See the Roadmap for the full shipped and candidate
|
|
28
|
-
> lists.
|
|
29
|
-
|
|
30
|
-
## Background
|
|
31
|
-
|
|
32
|
-
This project started as a research question: *can a harness improve itself,
|
|
33
|
-
and what would a production-grade version look like?* Three lines of evidence
|
|
34
|
-
shaped the answer:
|
|
35
|
-
|
|
36
|
-
- **penguin-harness** demonstrated the concept (benchmark → evaluate →
|
|
37
|
-
optimize → accept/rollback) but with **zero code-level enforcement** — every
|
|
38
|
-
guarantee was a prompt contract. Its report (`docs/research/`) became the
|
|
39
|
-
hardening checklist this project implements.
|
|
40
|
-
- **prime-agent `/refine`** proved the engineering shape: versioned harness
|
|
41
|
-
entries, atomic persistence, optimistic concurrency, inverse-op rollback.
|
|
42
|
-
This package is an original implementation of that shape on the DSH plugin
|
|
43
|
-
surface.
|
|
44
|
-
- Academic work (Self-Harness, AHE, HarnessOpt-Bench) supplied the discipline:
|
|
45
|
-
frozen evaluation runtime, code-owned aggregation, non-regressive
|
|
46
|
-
acceptance.
|
|
47
|
-
|
|
48
|
-
The result: **the model proposes, the code guarantees.** Every mechanical
|
|
49
|
-
safety property (schema validation, snapshots, versioning, audit trail,
|
|
50
|
-
acceptance decisions) is enforced in code — never by asking the model to
|
|
51
|
-
behave.
|
|
14
|
+
**The model proposes, the code guarantees.** Every mechanical safety property — schema validation, atomic writes, snapshots, versioning, audit trail, acceptance decisions — is enforced in code, never by prompt discipline.
|
|
52
15
|
|
|
53
16
|
## Why
|
|
54
17
|
|
|
55
|
-
Agents accumulate reusable experience
|
|
56
|
-
|
|
57
|
-
- **Versioned entries** keyed by kind (`prompt` / `memory` / `skill` / `subagent`), each with a recorded provenance and version
|
|
58
|
-
- **Evidence trail**: every refinement appends an event carrying `trigger / changes / evidence / outcome`
|
|
59
|
-
- **Deterministic rollback**: inverse edits are generated from applied results — no LLM re-guessing
|
|
60
|
-
- **Code-enforced safety**, not prompt discipline: schema validation, atomic writes, corrupt-file degrade, optimistic concurrency, immutable base system prompt
|
|
61
|
-
- **Local (session) and global (cross-session) scopes** with merge semantics
|
|
62
|
-
|
|
63
|
-
## Design provenance
|
|
64
|
-
|
|
65
|
-
Inspired by three bodies of work (see [`docs/design.md`](docs/design.md)):
|
|
18
|
+
Agents accumulate reusable experience (repeated failures, durable facts, reusable procedures) and forget it next session. This plugin turns that experience into first-class state:
|
|
66
19
|
|
|
67
|
-
- **
|
|
68
|
-
- **
|
|
69
|
-
-
|
|
20
|
+
- **Local scope** per session; **global scope** across sessions with merge semantics — plus mechanical promotion guards so only portable, substantial, non-duplicate knowledge reaches global
|
|
21
|
+
- **Deterministic rollback**: inverse edits generated from applied results — no LLM re-guessing
|
|
22
|
+
- **Benchmark loop**: candidate refinements are evaluated against frozen cases by a separate scorer before acceptance (rubric encrypted at rest)
|
|
70
23
|
|
|
71
|
-
##
|
|
24
|
+
## How it works
|
|
72
25
|
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
| Package manager | pnpm (DSH ecosystem standard) |
|
|
79
|
-
| Build | `tsc` → `lib/` (main `lib/index.js`, types `lib/index.d.ts`) |
|
|
80
|
-
| Tests | Vitest |
|
|
81
|
-
| Lint | oxlint (DSH official repo convention) |
|
|
82
|
-
| License | MIT |
|
|
83
|
-
|
|
84
|
-
## Project layout
|
|
85
|
-
|
|
86
|
-
```
|
|
87
|
-
dsh-continual-evolve/
|
|
88
|
-
├── package.json # exports / files / engines / scripts + dsh.bundle manifest
|
|
89
|
-
├── cordis.patch.yml # bundle patch (dsh plugin add activates on install)
|
|
90
|
-
├── tsconfig.json / .oxlintrc.json / .editorconfig / .gitignore
|
|
91
|
-
├── LICENSE / README.md / README.zh.md
|
|
92
|
-
├── docs/
|
|
93
|
-
│ ├── design.md # full design doc (incl. hardening matrix)
|
|
94
|
-
│ └── research/ # penguin-harness report + prime-agent reference source
|
|
95
|
-
├── src/
|
|
96
|
-
│ ├── index.ts # cordis plugin entry (service mount + wiring)
|
|
97
|
-
│ ├── types.ts # HarnessState / entry / edit / result types
|
|
98
|
-
│ ├── state.ts # atomic persistence, corrupt degrade, merge, concurrency
|
|
99
|
-
│ ├── validate.ts # code-enforced edit validation
|
|
100
|
-
│ ├── apply.ts # per-edit apply pass with optimistic locking
|
|
101
|
-
│ ├── rollback.ts # deterministic inverse-op rollback
|
|
102
|
-
│ ├── plan.ts # proposal JSON parsing (truncation-aware)
|
|
103
|
-
│ ├── tool.ts # evolve_* model-facing tools (5)
|
|
104
|
-
│ ├── command.ts # /evolve command (incl. benchmark subcommands)
|
|
105
|
-
│ ├── planner.ts # ctx.llm planner
|
|
106
|
-
│ ├── render.ts # bounded prompt rendering
|
|
107
|
-
│ ├── inject.ts # dynamic system-prompt section (prompt notes + delegation specs, ranked injection)
|
|
108
|
-
│ ├── source.ts # trajectory citations (sessionId + event seqs of distilled entries)
|
|
109
|
-
│ ├── auto.ts # auto-review gate (turn/compaction triggers + audit, global-aware view, local-fate phase)
|
|
110
|
-
│ ├── fate.ts # gate local-fate dimension — auto promote/archive of local entries (consulted first, cooldown)
|
|
111
|
-
│ ├── notify.ts # gate visibility — follow-up notice after an approved auto-refine
|
|
112
|
-
│ ├── goal.ts # goal-driven evolution rounds (/evolve goal)
|
|
113
|
-
│ ├── review.ts # gate LLM judgment (declines local duplicates of globally covered topics)
|
|
114
|
-
│ ├── approval.ts # human approval for global edits
|
|
115
|
-
│ ├── skill.ts # skill materialization ($DSH_HOME/skills/)
|
|
116
|
-
│ ├── skillquality.ts # skill standard in the loop (skill-creator template reading + frontmatter code checks)
|
|
117
|
-
│ ├── mount.ts # hot-mounted skill plugins (loader.create + boot restore)
|
|
118
|
-
│ ├── benchmark.ts # benchmark store
|
|
119
|
-
│ ├── rubric.ts # rubric ACL (AES-256-GCM envelopes, auto-generated local key)
|
|
120
|
-
│ ├── logfile.ts # plugin-owned file logging (JSONL exporter + rotation)
|
|
121
|
-
│ ├── score.ts # code-owned aggregation + acceptance rule
|
|
122
|
-
│ ├── evaluate.ts # two-stage evaluation runner (executor evidence → independent reviewer) + failure-cell protocol
|
|
123
|
-
│ ├── pool.ts # bounded-concurrency worker pool for evaluation runs
|
|
124
|
-
│ ├── store.ts # store layout + snapshots + result history
|
|
125
|
-
│ ├── service.ts # evolution engine (onApplied hook)
|
|
126
|
-
│ └── wrapup.ts # session wrap-up lifecycle (promote / split-promote → global, guarded archive; shared proposal builders)
|
|
127
|
-
└── test/ # 23 files, 290 tests
|
|
128
|
-
```
|
|
26
|
+
1. **Sediment** — the model creates entries via `evolve_add`, or the automatic review gate proposes them from the session trajectory (turn-interval + compaction checkpoints).
|
|
27
|
+
2. **Guard** — code-enforced validation: edit schema, blast-radius/scope coherence, and the promotion policy (project-scoped markers, thin content, near-duplicate detection keep the global store clean).
|
|
28
|
+
3. **Approve** — global writes require explicit human approval; local-fate proposals are consulted before they land.
|
|
29
|
+
4. **Apply & inject** — atomic apply with snapshot + audit event. Prompt notes and delegation specs inject into the system prompt (capped, relevance-ranked, zero tokens when empty); memories/skills appear as a capped directory index.
|
|
30
|
+
5. **Validate & roll back** — benchmarks score candidates against frozen cases; rejected candidates roll back deterministically.
|
|
129
31
|
|
|
130
32
|
## Install
|
|
131
33
|
|
|
132
34
|
```bash
|
|
133
35
|
# from npm (installs and activates — ships its own bundle patch)
|
|
134
|
-
dsh plugin
|
|
36
|
+
dsh plugin add dsh-continual-evolve
|
|
135
37
|
|
|
136
|
-
# or from source (first GitHub installs require approving the allowBuilds
|
|
137
|
-
dsh plugin
|
|
38
|
+
# or from source (first GitHub installs require approving the allowBuilds step)
|
|
39
|
+
dsh plugin add ZK-Andy/dsh-continual-evolve
|
|
138
40
|
```
|
|
139
41
|
|
|
140
|
-
|
|
42
|
+
Restart `dsh web` after installing or updating.
|
|
141
43
|
|
|
142
|
-
##
|
|
44
|
+
## Usage
|
|
143
45
|
|
|
144
|
-
|
|
145
|
-
/evolve help + current local store
|
|
146
|
-
/evolve list [global] list entries
|
|
147
|
-
/evolve history applied refinements (ids for rollback)
|
|
148
|
-
/evolve rollback <id> deterministically revert a refinement
|
|
149
|
-
/evolve plan [msg] LLM planner against the current store
|
|
150
|
-
/evolve wrapup assess this session's local entries: promote reusable ones to the
|
|
151
|
-
global store (approval required), archive session-specific ones
|
|
152
|
-
/evolve archive <id> hide an entry from injection (data kept, restorable)
|
|
153
|
-
/evolve unarchive <id> restore an archived entry
|
|
154
|
-
/evolve log [tail N] [session <id>] show the recent plugin log (default 50 lines; optional per-session filter)
|
|
155
|
-
/evolve export <path> backup the local store to JSON
|
|
156
|
-
/evolve import <path> restore a store from an export file
|
|
157
|
-
/evolve mount <skillId> hot-mount a skill entry as a live cordis plugin (tool: skill_<name>)
|
|
158
|
-
/evolve mount list list hot-mounted plugins (restored on boot)
|
|
159
|
-
/evolve unmount <id> remove a hot-mounted plugin
|
|
160
|
-
/evolve goal show the evolution goal (round-driven auto-review)
|
|
161
|
-
/evolve goal <objective> create/update the evolution goal — while active, the review gate runs EVERY round
|
|
162
|
-
/evolve goal done complete the evolution goal
|
|
163
|
-
```
|
|
164
|
-
|
|
165
|
-
Model-facing tools: `evolve_list`, `evolve_add`, `evolve_update`, `evolve_delete`, `evolve_rollback`.
|
|
166
|
-
|
|
167
|
-
## Memory layer
|
|
168
|
-
|
|
169
|
-
Beyond the persisted store itself, four features keep injected memory
|
|
170
|
-
"understanding you" as entries grow (gap analysis vs. Mem0 / Letta / Zep /
|
|
171
|
-
LangMem; no external services — everything is pure functions):
|
|
172
|
-
|
|
173
|
-
- **Ranked injection** — when a kind holds more than the 6-entry cap, the
|
|
174
|
-
injected block no longer shows the fixed first six: entries are scored by
|
|
175
|
-
relevance to the agent's most recent direct user messages (keyword/BM25
|
|
176
|
-
level: title hits weigh 2×) and then by recency (`updated_at`, 30-day
|
|
177
|
-
half-life), so the freshest *and most relevant* entries fill the cap. The
|
|
178
|
-
empty-store zero-token behavior is unchanged.
|
|
179
|
-
- **Trajectory citations** — every newly created entry records
|
|
180
|
-
`metadata.sourceSession` + `metadata.sourceSeqs` pointing at the direct
|
|
181
|
-
user messages it was distilled from (DSH sessions are event-sourced with
|
|
182
|
-
contiguous seqs, so the citation expands back into the durable session
|
|
183
|
-
log). Listings show `src=<sessionId>:<seqs>`; old entries are not migrated
|
|
184
|
-
and never error.
|
|
185
|
-
- **Archive** — `/evolve archive <id>` hides an entry from injection
|
|
186
|
-
(`metadata.archivedAt`, data kept, rollback-compatible) and
|
|
187
|
-
`/evolve unarchive <id>` restores it. Archived entries are marked
|
|
188
|
-
`[archived]` in `evolve_list` and skipped by injection; the overflow count
|
|
189
|
-
excludes them.
|
|
190
|
-
- **Session wrap-up** — a session's local entries otherwise become orphans when
|
|
191
|
-
it ends (later sessions never see them). `/evolve wrapup` gives them an exit.
|
|
192
|
-
Each entry is audited mechanically — global-coverage is judged by **title
|
|
193
|
-
similarity only** (a bare id collision with a different title is intentionally
|
|
194
|
-
NOT coverage; the actual matching global titles are shown to the assessor so
|
|
195
|
-
it judges against real content) — then classified as `promote` / `archive` /
|
|
196
|
-
`keep`. Promotions move reusable entries into the global store **through the
|
|
197
|
-
human approval gate**, keeping their trajectory citation and adding a
|
|
198
|
-
`sourcedFromLocal=<session>:<id>` back-link; the local copy is stamped
|
|
199
|
-
`promotedTo` and retired from injection so it is never offered again.
|
|
200
|
-
**Split promotion** (A-form): a mixed entry (durable facts + session snapshot)
|
|
201
|
-
can be archived while carrying a cleaned `promote` sub-object — only the
|
|
202
|
-
durable part lands globally, the snapshot stays in the archive. A **symmetric
|
|
203
|
-
archive guard** requires user confirmation before an archive that is NOT
|
|
204
|
-
globally covered AND was distilled from real user messages hides that content
|
|
205
|
-
from future sessions (over-archiving gets the same protection as
|
|
206
|
-
over-writing); operational entries still archive silently. Everything stays
|
|
207
|
-
snapshot/versioned/rollbackable.
|
|
208
|
-
- **Gate local-fate (automatic wrap-up)** — the same wrap-up machinery now
|
|
209
|
-
runs inside the auto-review gate on its own cadence (`fateIntervalTurns`),
|
|
210
|
-
so local entries get their exit while the session is still running instead
|
|
211
|
-
of waiting for a manual `/evolve wrapup`. On each due gate run the audited
|
|
212
|
-
candidates are classified by the assessor and partitioned by the same
|
|
213
|
-
deterministic guards; the user is consulted FIRST before anything governed
|
|
214
|
-
lands (one dialog covering promotes, split promotions and review-required
|
|
215
|
-
archives — the consultSkillEdits pattern, with a decline cooldown). Covered
|
|
216
|
-
or operational entries still archive silently, and at compaction the gate
|
|
217
|
-
never opens a dialog: only silent archives apply, governed actions are
|
|
218
|
-
deferred with an audit record pointing at `/evolve wrapup`. Every fate
|
|
219
|
-
decision lands in `reviews.jsonl` (`approved` / `declined` / `deferred` /
|
|
220
|
-
`assessed` / `failed`) and applied actions are visible via a follow-up
|
|
221
|
-
notice. Apply writes are byte-identical to the wrap-up command (shared
|
|
222
|
-
proposal builders).
|
|
223
|
-
- **Global-aware gate** — the auto-review gate and planner judge the merged
|
|
224
|
-
global + local state with every entry's real scope labeled, so a topic
|
|
225
|
-
already covered by a global entry is declined instead of being re-sedimented
|
|
226
|
-
as a local duplicate.
|
|
46
|
+
Commands (in-session):
|
|
227
47
|
|
|
228
|
-
|
|
229
|
-
|
|
230
|
-
|
|
231
|
-
|
|
232
|
-
|
|
233
|
-
|
|
234
|
-
|
|
235
|
-
|
|
236
|
-
|
|
237
|
-
|
|
238
|
-
-
|
|
239
|
-
|
|
240
|
-
|
|
241
|
-
|
|
242
|
-
|
|
243
|
-
|
|
244
|
-
|
|
245
|
-
- The gate judges skill-related trajectories against the skill-audit
|
|
246
|
-
dimensions (frontmatter routing, structural features, paragraph skeleton,
|
|
247
|
-
duplication) and declines proposals that would not meet the standard.
|
|
248
|
-
- The mechanical frontmatter rules of `validate-frontmatter.mjs` are
|
|
249
|
-
code-enforced at apply time: skill bodies must not open with a second
|
|
250
|
-
`---` block (it would shadow the generated frontmatter), and resource
|
|
251
|
-
references may not escape the skill directory. After materialization the
|
|
252
|
-
rendered SKILL.md is re-checked and dangling `references/`/`scripts/`
|
|
253
|
-
links are logged as warnings.
|
|
254
|
-
- **Two skill forms** — `executable` skills keep the python reference
|
|
255
|
-
contract (hot-mountable as tools); `guidance` skills are SKILL.md
|
|
256
|
-
documents with no reference, the form for recurring multi-step workflows
|
|
257
|
-
(session start/end routines, handoff procedures). Code enforces the
|
|
258
|
-
split: a guidance skill must NOT carry a reference or arguments contract.
|
|
259
|
-
- **User-governed skill creation** — the gate never writes a skill
|
|
260
|
-
silently: when the planner proposes skill edits, the user is asked
|
|
261
|
-
(固化/不固化) before they land; a rejected candidate is not offered
|
|
262
|
-
again within a cooldown window. The rest of a proposal proceeds
|
|
263
|
-
regardless, so memory/prompt distillation is never blocked by a skill
|
|
264
|
-
decision.
|
|
265
|
-
|
|
266
|
-
## Logging
|
|
267
|
-
|
|
268
|
-
Plugin-owned file logging: every cordis log message (from this plugin or any
|
|
269
|
-
other) is appended to `<dshHome>/evolve/plugin.log` as JSONL (0600, rotated to
|
|
270
|
-
`plugin.log.1` past `logMaxBytes`). It works no matter how `dsh web` is
|
|
271
|
-
launched — no extra component to install, no startup-script dependency.
|
|
272
|
-
View the tail with `/evolve log [tail N]`, or read the file directly:
|
|
273
|
-
|
|
274
|
-
```bash
|
|
275
|
-
tail -f ~/.dsh/evolve/plugin.log # live
|
|
276
|
-
/evolve log 100 # last 100 lines in the chat
|
|
277
|
-
```
|
|
278
|
-
|
|
279
|
-
For live output in a foreground terminal, the official
|
|
280
|
-
`@deepseek-ai/cordis-plugin-logger-console` plugin can be added to the
|
|
281
|
-
profile (optional; the file log remains the baseline that always exists).
|
|
282
|
-
|
|
283
|
-
## Benchmark-driven validation (Phase 3)
|
|
284
|
-
|
|
285
|
-
```
|
|
286
|
-
/evolve benchmark new <title> [runs] create a benchmark (runs = repeats per case, default 1)
|
|
287
|
-
/evolve benchmark add-case <bid> <title> <statement> <rubric>
|
|
288
|
-
/evolve benchmark list list benchmarks
|
|
289
|
-
/evolve benchmark reset <bid> clear the scoreboard (re-run reference)
|
|
290
|
-
/evolve benchmark status <bid> scoreboard + decisions
|
|
291
|
-
/evolve benchmark run <bid> evaluate current state → reference
|
|
292
|
-
/evolve benchmark run <bid> candidate <refinementId> evaluate post-refinement state → decide
|
|
293
|
-
```
|
|
294
|
-
|
|
295
|
-
The loop: freeze a reference score → evolve a candidate (`/evolve plan`) →
|
|
296
|
-
run the same case × run matrix against the post-refinement state → the
|
|
297
|
-
**code-owned** acceptance rule keeps the candidate only if the overall mean
|
|
298
|
-
strictly improves with no case regressing (Self-Harness style).
|
|
299
|
-
|
|
300
|
-
**Evaluator/scorer separation (two-stage, gap A1)** — each case × run unit
|
|
301
|
-
is a PAIR of fresh subagents:
|
|
302
|
-
|
|
303
|
-
1. the **executor** performs the task with its tools and records **concrete
|
|
304
|
-
evidence** of what it did and found — it NEVER sees the rubric, so the
|
|
305
|
-
agent under test cannot optimize toward or self-grade against the grading
|
|
306
|
-
criteria;
|
|
307
|
-
2. an **independent reviewer** grades that evidence strictly against the
|
|
308
|
-
rubric (the only branch that receives the decrypted rubric), eliminating
|
|
309
|
-
the "self-produced and self-scored" bias.
|
|
310
|
-
|
|
311
|
-
Each cell records the executor's session id, so a score can be drilled back
|
|
312
|
-
to the exact transcript that produced it (trace evidence pointer, gap A4).
|
|
313
|
-
|
|
314
|
-
**Failure-cell protocol (gap A2)** — a unit that cannot produce a score
|
|
315
|
-
(rubric decrypt error, executor/reviewer crash, protocol error) is recorded
|
|
316
|
-
as a **failed** cell, NEVER a zero: aggregation excludes failed cells from
|
|
317
|
-
every mean and counts them (`/evolve benchmark status` shows `(N failed)`),
|
|
318
|
-
and the acceptance rule rejects a round with more failed cells than
|
|
319
|
-
`maxFailedCells` (0 by default) instead of silently averaging a 0 into the
|
|
320
|
-
mean.
|
|
321
|
-
|
|
322
|
-
Aggregation and decisions live in `src/score.ts`. Rubric isolation is by
|
|
323
|
-
construction (the planner never sees rubric files, and the executor branch
|
|
324
|
-
never decrypts); a rejection is recorded in the scoreboard and the
|
|
325
|
-
refinement is rolled back automatically (`autoRollbackOnReject`, on by
|
|
326
|
-
default).
|
|
327
|
-
|
|
328
|
-
### Real recorded run (ACCEPT)
|
|
329
|
-
|
|
330
|
-
A live `dsh web` session, one case, one candidate — the first genuine
|
|
331
|
-
acceptance:
|
|
332
|
-
|
|
333
|
-
| Step | Command | Outcome |
|
|
334
|
-
|---|---|---|
|
|
335
|
-
| reference | `/evolve benchmark run lint_convention` | **90** — the evaluator agent actually grepped the harness store and reported *"lint/ruff/eslint/mypy appear in zero entries"* |
|
|
336
|
-
| candidate | `/evolve plan 记住:写代码前必须先运行适用的 lint 检查` | creates `memory:convention_lint_before_code` |
|
|
337
|
-
| re-evaluate | `/evolve benchmark run lint_convention candidate <id>` | **100** — evaluator ran `evolve_list`, hit the memory, quoted it verbatim |
|
|
338
|
-
| decision | — | `overall: 90 → 100` · `lint_knowledge: 90 → 100` · **DECISION: ACCEPTED** |
|
|
339
|
-
|
|
340
|
-
The executor does not grade model common sense — it inspects the actual
|
|
341
|
-
harness state under test (grep, `evolve_list`) and records what it found;
|
|
342
|
-
the independent reviewer grades that record. A harness change measurably
|
|
343
|
-
moves the score. Earlier runs in the same session
|
|
344
|
-
produced honest `REJECTED` decisions (0 → 0 stub cases, and 100 → 100
|
|
345
|
-
where the baseline was already perfect).
|
|
48
|
+
| Command | Effect |
|
|
49
|
+
|---|---|
|
|
50
|
+
| `/evolve` | help + current local store |
|
|
51
|
+
| `/evolve list · history · rollback <id>` | inspect and revert (add `global` for the cross-session store) |
|
|
52
|
+
| `/evolve plan [msg]` | run the LLM planner against the store |
|
|
53
|
+
| `/evolve wrapup` | assess this session's local entries: promote / archive / keep |
|
|
54
|
+
| `/evolve archive · unarchive · demote <id>` | hide from injection (data kept, restorable) — `demote` targets global noise |
|
|
55
|
+
| `/evolve failures` | aggregated failure classes (gate + benchmark) |
|
|
56
|
+
| `/evolve log [tail N] [session <id>]` | plugin log |
|
|
57
|
+
| `/evolve export · import <path>` | backup / restore a store |
|
|
58
|
+
| `/evolve mount · unmount <skillId>` | hot-mount an executable skill as a live plugin |
|
|
59
|
+
| `/evolve goal [objective · done · block]` | round-driven auto-review goal |
|
|
60
|
+
| `/evolve benchmark …` | case lifecycle, runs, acceptance |
|
|
61
|
+
|
|
62
|
+
Model tools: `evolve_list / add / update / delete / rollback`.
|
|
63
|
+
|
|
64
|
+
Injection shape: prompt notes and delegation specs inject with content (≤6/kind × 180 chars, relevance-ranked). Memories and skills appear as a directory index (`[kind:id] title`, capped at 15 lines with a fold counter) — full text via `evolve_list`. Empty store = zero injected tokens.
|
|
346
65
|
|
|
347
66
|
## Configuration
|
|
348
67
|
|
|
349
68
|
| Key | Default | Meaning |
|
|
350
69
|
|---|---|---|
|
|
351
70
|
| `baseDir` | resolved DSH home | root for the `evolve/` stores |
|
|
352
|
-
| `
|
|
353
|
-
| `
|
|
354
|
-
| `
|
|
355
|
-
| `
|
|
356
|
-
| `
|
|
357
|
-
| `
|
|
358
|
-
| `
|
|
359
|
-
| `
|
|
360
|
-
| `
|
|
361
|
-
| `
|
|
362
|
-
| `
|
|
363
|
-
| `
|
|
364
|
-
| `
|
|
365
|
-
| `
|
|
366
|
-
| `
|
|
367
|
-
|
|
368
|
-
|
|
71
|
+
| `autoReview` | `false` | enable the automatic review gate |
|
|
72
|
+
| `reviewIntervalTurns` | `6` | gate cadence on the turn-interval path |
|
|
73
|
+
| `maxReviewInputChars` | `40000` | trajectory slice handed to the gate |
|
|
74
|
+
| `reviewBudgetTokens` | `4096` | output budget for the gate call |
|
|
75
|
+
| `notifyOnAutoReview` | `true` | visible follow-up notice after an applied gate run |
|
|
76
|
+
| `requireGlobalApproval` | `true` | global edits ask for explicit approval |
|
|
77
|
+
| `localFate` | `true` | gate audits local entries and proposes promote/archive (consulted, never silent) |
|
|
78
|
+
| `fateIntervalTurns` | follows `reviewIntervalTurns` | minimum turns between fate assessments |
|
|
79
|
+
| `goalBlockedWrapupTurns` | `3` | consecutive blocked-goal gate runs trigger one fate assessment (`0` disables) |
|
|
80
|
+
| `promotionBlockPatterns` | POSIX paths, session ids, `~/.dsh` | content matching these is project-scoped and never promoted to global |
|
|
81
|
+
| `promotionMinChars` | `100` | whole promotions below this length stay local |
|
|
82
|
+
| `injectionDirectoryLines` | `15` | entry-directory lines per build before folding into a counter |
|
|
83
|
+
| `sectionOrder` | `118` | system-prompt section order |
|
|
84
|
+
| `skillsDir` | `<dshHome>/skills` | where skill entries materialize as SKILL.md bundles |
|
|
85
|
+
| `rubricKey` | auto-generated key file | AES-256-GCM passphrase for benchmark rubrics (`DSH_EVOLVE_RUBRIC_KEY` overrides) |
|
|
86
|
+
| `logToFile` / `logLevel` / `logMaxBytes` | `true` / `1` / 5 MiB | plugin-owned JSONL file log with rotation |
|
|
87
|
+
| `autoRollbackOnReject` | `true` | deterministic rollback after a benchmark rejection |
|
|
88
|
+
| `reviewModel` | agent's own | optional cheaper model for the gate (`"provider/model"`) |
|
|
89
|
+
|
|
90
|
+
Example profile patch:
|
|
369
91
|
|
|
370
92
|
```yaml
|
|
371
|
-
-
|
|
372
|
-
|
|
373
|
-
|
|
374
|
-
|
|
375
|
-
autoReview: true
|
|
376
|
-
reviewIntervalTurns: 6
|
|
93
|
+
- id: continual-evolve
|
|
94
|
+
config:
|
|
95
|
+
autoReview: true
|
|
96
|
+
reviewIntervalTurns: 6
|
|
377
97
|
```
|
|
378
98
|
|
|
379
99
|
## Development
|
|
380
100
|
|
|
381
101
|
```bash
|
|
382
|
-
pnpm install
|
|
383
|
-
pnpm
|
|
384
|
-
pnpm
|
|
385
|
-
pnpm
|
|
386
|
-
pnpm lint # oxlint src test
|
|
102
|
+
pnpm install && pnpm build # deps + tsc -> lib/
|
|
103
|
+
pnpm test # vitest (527 tests)
|
|
104
|
+
pnpm test:coverage # v8 coverage, thresholds enforced in CI
|
|
105
|
+
pnpm lint # oxlint src test
|
|
387
106
|
```
|
|
388
107
|
|
|
389
|
-
|
|
390
|
-
|
|
391
|
-
Where we still lag behind prime-agent `/refine` and penguin-harness — and what to build next: [`docs/gap-analysis.md`](docs/gap-analysis.md) (P0 shipped: evaluator/scorer separation, failure-cell protocol; next: P1 runtime provenance checks, usage statistics, auto-decay).
|
|
392
|
-
|
|
393
|
-
## Roadmap
|
|
108
|
+
Project layout:
|
|
394
109
|
|
|
395
|
-
|
|
110
|
+
```
|
|
111
|
+
├── src/ # engine, tools, commands, gate, fate, benchmark, usage…
|
|
112
|
+
├── test/ # vitest suites (33 files)
|
|
113
|
+
├── lib/ # build output (tsc)
|
|
114
|
+
├── docs/
|
|
115
|
+
│ ├── design.md # full design doc (hardening matrix)
|
|
116
|
+
│ ├── FAQ.md # real failure/fix records
|
|
117
|
+
│ ├── gap-analysis.md # vs prime-agent /refine + penguin-harness
|
|
118
|
+
│ ├── experiment-bootstrap.md
|
|
119
|
+
│ ├── archive/ # closed point-in-time reports
|
|
120
|
+
│ └── research/ # penguin report + prime-agent annotated source
|
|
121
|
+
├── examples/README.md # seed benchmark cases
|
|
122
|
+
└── .agents/ # AI collaboration layer (AGENTS.md, skills, ADR notes)
|
|
123
|
+
```
|
|
396
124
|
|
|
397
|
-
|
|
398
|
-
- **2026-08 maintenance wave (done)**:
|
|
399
|
-
- **memory layer** — ranked injection (relevance + recency scoring fills the per-kind cap), trajectory citations (`metadata.sourceSession` + `sourceSeqs`, shown as `src=session:seqs`), archive/unarchive (`/evolve archive <id>`, injection skips archived entries), global-aware gate (declines local duplicates of globally covered topics)
|
|
400
|
-
- **per-installation rubric key** — auto-generated local key file (`<dshHome>/evolve/rubric.key`, 0600); no more publicly known dev key
|
|
401
|
-
- **plugin-owned file logging** — every cordis log message lands in `<dshHome>/evolve/plugin.log` (JSONL, 0600, rotated), viewable via `/evolve log`; works with any launch method, no extra component to install
|
|
402
|
-
- **trajectory-grounded planning** — `/evolve plan` (and every planner call, including the gate's refine step) now reads the session trajectory: the caller's recent direct user messages are extracted from the session log and fed to the planner as a `<session_trajectory>` block, so proposals are grounded in what the user actually said (explicit `trajectory` overrides; empty trajectory is omitted at zero cost)
|
|
403
|
-
- **gate-proposed archiving** — stale entries are a first-class refine target: the planner can emit `action: "archive"` (kind + id only), which stamps `metadata.archivedAt` through the normal apply path — snapshot, version bump, audit event, and a deterministic rollback inverse that restores the pre-archive state. Archive hides from injection but never deletes; re-archiving an archived entry is rejected, and the base system prompt stays immutable
|
|
404
|
-
- **automatic rollback on benchmark rejection** — the acceptance loop is closed: when the code-owned decision rejects a candidate, the refinement is reverted automatically through the same engine path as `/evolve rollback` (deterministic inverse edits, snapshotted and audited; configurable via `autoRollbackOnReject`, on by default). Failures report the manual fallback instead of throwing
|
|
405
|
-
- **per-session log filtering** — `/evolve log [tail N] [session <id>]` keeps only the lines mentioning a given session id (exact token match, drawn from the rendered message and raw args); gate records now carry the session id in their log line
|
|
406
|
-
- **skill standard in the loop** — the planner and gate now author and judge skill entries against the skill-creator/skill-audit standard (author-distilled from the official deepseek-harness 11 skills): every plan call injects the `template.md` facts (builtin distilled guide as fallback) as `<skill_quality_standard>`; apply code-enforces the frontmatter mechanics (no shadowing `---`, no escaping resource refs); materialized SKILL.md files are re-checked and dangling resource references are logged;
|
|
407
|
-
- **guidance skills + user-governed creation** — a second skill form (SKILL.md documents without a python reference) lets recurring workflows be proposed as skills; the gate offers every auto-created skill to the user (固化/不固化) before it lands, with a rejection cooldown — skills grow under governance, never silently
|
|
408
|
-
- **2026-08-17 wrap-up wave (done)**:
|
|
409
|
-
- **`/evolve wrapup`** — a session's local entries get a real exit at session end: mechanical audit (local candidates + global-coverage detection; coverage judges **title similarity only** — a bare id collision with a different title is deliberately NOT coverage, and the real matching global titles are shown to the assessor) → LLM classification (`promote` / `archive` / `keep` + A-form split promotion: archive a mixed entry while promoting a cleaned durable sub-object) → deterministic guards re-checked at apply time (promote can never write a global duplicate; the symmetric archive guard requires user confirmation before an uncovered, user-sourced archive hides content; splits that duplicate a global topic drop to plain archive) → one human approval gate for every global create
|
|
410
|
-
- **gate local-fate dimension** — the wrap-up machinery now runs inside the auto-review gate on its own cadence (`fateIntervalTurns`, compaction unconditional): local entries are audited, classified and partitioned while the session is still running; governed actions are consulted first (one dialog, decline cooldown), covered/operational entries archive silently, compaction applies only silent archives and defers governed actions with an audit record; every decision lands in `reviews.jsonl` and applied actions get a follow-up notice. Apply writes are shared with the wrap-up command (byte-identical proposals)
|
|
411
|
-
- **2026-08-17 gap P0 (done)**:
|
|
412
|
-
- **evaluator/scorer separation** — benchmark evaluation is now two-stage (gap A1): the executor performs the task and records concrete evidence without ever seeing the rubric; an independent reviewer grades that evidence against the rubric (the only branch that decrypts it). The assessed agent can no longer optimize toward or self-grade against the criteria.
|
|
413
|
-
- **failure-cell protocol** — cells carry `status: ok|failed` (gap A2): failed units are excluded from every mean and counted, and the acceptance rule rejects rounds with failures beyond `maxFailedCells` (0 default) instead of averaging a zero into the mean. Scoreboard status/run surfaces failed counts and per-cell reasons.
|
|
414
|
-
- **trace evidence pointer** — each cell records the executor's session id (gap A4), so a score drills back to the exact transcript that earned it
|
|
125
|
+
## Docs & provenance
|
|
415
126
|
|
|
416
|
-
|
|
127
|
+
- Design: [`docs/design.md`](docs/design.md) · Pitfalls: [`docs/FAQ.md`](docs/FAQ.md) · Gap analysis: [`docs/gap-analysis.md`](docs/gap-analysis.md) · D2 experiment: [`docs/experiment-bootstrap.md`](docs/experiment-bootstrap.md)
|
|
128
|
+
- Lineage: **penguin-harness** (concept; Apache-2.0) — report in [`docs/research/penguin-harness-self-evolution.md`](docs/research/penguin-harness-self-evolution.md); **prime-agent `/refine`** (engineering shape; MIT) — annotated reference source in [`docs/research/prime-agent-refinement.ts`](docs/research/prime-agent-refinement.ts). This package is an original implementation on the DSH plugin surface.
|
|
417
129
|
|
|
418
130
|
## License
|
|
419
131
|
|
|
420
|
-
MIT
|
|
132
|
+
[MIT](LICENSE)
|