dsh-harbor-evolution 0.1.0 → 0.2.0

package/README.md

@@ -4,31 +4,53 @@ Cordis bundle for evaluating immutable DeepSeek Harness Candidates with Harbor.
 
 ## Install
 
-Install `harbor-dsh-evolution` into the same Python environment as Harbor, then add this bundle to the DSH profile:
+Install `harbor-dsh-evolution` into the same Python environment as Harbor, then add this bundle to the DSH profile you actually run. Use `web` for `dsh web`; use `headless` only for the command-line Agent:
 
 ```bash
 uv venv .venv
 uv pip install --python .venv/bin/python harbor-dsh-evolution
 source .venv/bin/activate
-pnpm dlx @deepseek-ai/dsh@0.1.0-rc.6 plugin --profile headless add dsh-harbor-evolution
+pnpm dlx @deepseek-ai/dsh@0.1.0-rc.6 plugin --profile web add -w dsh-harbor-evolution@0.2.0
 ```
 
+Restart DSH after installation. Launch it from the Agent workspace and expose the two Python executables to the DSH process:
+
+```bash
+cd /absolute/path/to/your-agent-workspace
+HARBOR_BIN=/absolute/path/to/.venv/bin/harbor \
+HARBOR_DSH_BIN=/absolute/path/to/.venv/bin/harbor-dsh \
+pnpm dlx @deepseek-ai/dsh@0.1.0-rc.6 web
+```
+
+In the Web UI, search for the Cordis plugin id `harbor-evolution`. See the [complete local DSH Web quickstart](https://github.com/istarwyh/harbor-self-evolving/blob/main/docs/dsh-web-quickstart.md) for persistent configuration, a first evaluation, Candidate comparison, and troubleshooting.
+
 For development from this repository:
 
 ```bash
-./hse dsh-install headless
+./hse dsh-install web
 ```
 
-`-w` is required by pnpm when the DSH profile directory is its workspace root.
+`-w` is required because the DSH profile directory is its pnpm workspace root.
+
+The bundle inserts one `harbor-evolution` entry and registers this project's model- and user-invocable Skill through the official DSH Skill Registry:
 
-The bundle inserts one `harbor-evolution` entry and registers four model-facing tools:
+- `evolve-agent-with-harbor`
+
+The Skill guides the Agent through workspace inspection, requirements clarification, safe initialization, baseline evaluation, evidence-based diagnosis, one controlled Candidate change, regression comparison, and a Promotion Gate recommendation. Invoke it explicitly from Web or TUI with:
+
+```text
+/evolve-agent-with-harbor
+Inspect this workspace and help me clarify and initialize a stable Harbor self-evolution loop.
+```
+
+The Agent can also select it automatically when the request matches its catalog description. The Skill orchestrates four model-facing tools:
 
 - `harbor_candidate_snapshot`
 - `harbor_eval_run`
 - `harbor_eval_result`
 - `harbor_candidate_compare`
 
-The shortest evaluation call needs only `candidatePath` and `datasetPath`. Candidate identity defaults to `package.json`, the Job name is generated automatically, and `harbor_eval_run` returns the completed evaluation summary directly.
+The shortest direct evaluation call needs only `candidatePath` and `datasetPath`. Candidate identity defaults to `package.json`, the Job name is generated automatically, and `harbor_eval_run` returns the completed evaluation summary directly. Prefer the Skill for a new project because it will not run or compare Jobs until the material evaluation contract is resolved.
 
 The helper launches a fixed DSH version through `pnpm dlx`, so it does not conflict with a long-running `npx` DSH process. When installed from this monorepo, the bundle automatically discovers the sibling Python virtual environment created by `./hse dsh-install`. Otherwise it uses `HARBOR_BIN` / `HARBOR_DSH_BIN`, then falls back to `PATH`. Override the inserted entry in the profile's `cordis.patch.yml` when the binaries or Python source live elsewhere:
 
@@ -39,9 +61,11 @@ The helper launches a fixed DSH version through `pnpm dlx`, so it does not confl
     jobsDir: jobs
     harborBin: /workspace/venv/bin/harbor
     harborDshBin: /workspace/venv/bin/harbor-dsh
-    pythonPath: /workspace/harbor-self-evolving/packages/harbor-plugin/src
+    pythonPath: ""
 ```
 
+Keep `pythonPath` empty when using the published PyPI package. Set it only when developing the Python package from a source checkout.
+
 This replaces the complete config for the entry, following DSH patch semantics.
 
 The plugin never mutates the active DSH profile and never deploys a Candidate. Another system updates the Champion only after the external Promotion Gate passes.

package/index.js

@@ -5,9 +5,10 @@ import path from 'node:path'
 import { fileURLToPath } from 'node:url'
 
 import { compareCandidates, readEvaluation, runEvaluation, snapshot } from './lib/evolution.js'
+import { loadBundledSkill } from './lib/official-skill.js'
 
 export const name = 'harbor-evolution'
-export const inject = ['tools']
+export const inject = ['tools', 'skills']
 
 const packageDir = path.dirname(fileURLToPath(import.meta.url))
 const checkoutPythonPackage = path.resolve(packageDir, '../harbor-plugin')
@@ -55,6 +56,8 @@ export function apply(ctx, config) {
     ),
   }
 
+  ctx.skills.register(loadBundledSkill())
+
   ctx.tools.register(jsonTool({
     name: 'harbor_candidate_snapshot',
     description: 'Freeze a DeepSeek Harness Cordis composition as an immutable Candidate manifest. Candidate id and version default to package.json.',

package/lib/official-skill.js

@@ -0,0 +1,62 @@
+import { readFileSync } from 'node:fs'
+import path from 'node:path'
+import { fileURLToPath } from 'node:url'
+
+const moduleDir = path.dirname(fileURLToPath(import.meta.url))
+const skillDir = path.resolve(moduleDir, '../skills/evolve-agent-with-harbor')
+const skillPath = path.join(skillDir, 'SKILL.md')
+
+function parseScalar(value) {
+  const trimmed = value.trim()
+  if (trimmed.startsWith('"') && trimmed.endsWith('"')) return JSON.parse(trimmed)
+  if (trimmed.startsWith("'") && trimmed.endsWith("'")) return trimmed.slice(1, -1)
+  return trimmed
+}
+
+function parseBoolean(value, field) {
+  const normalized = String(value).toLowerCase()
+  if (['true', 'yes', 'on', '1'].includes(normalized)) return true
+  if (['false', 'no', 'off', '0'].includes(normalized)) return false
+  throw new TypeError(`bundled Skill frontmatter field "${field}" must be a boolean`)
+}
+
+function parseSkillMarkdown(raw) {
+  const match = raw.match(/^---\r?\n([\s\S]*?)\r?\n---(?:\r?\n|$)([\s\S]*)$/)
+  if (!match) throw new Error(`bundled Skill is missing YAML frontmatter: ${skillPath}`)
+
+  const metadata = {}
+  for (const line of match[1].split(/\r?\n/)) {
+    if (!line.trim()) continue
+    const separator = line.indexOf(':')
+    if (separator <= 0) throw new Error(`invalid bundled Skill frontmatter line: ${line}`)
+    metadata[line.slice(0, separator).trim()] = parseScalar(line.slice(separator + 1))
+  }
+
+  for (const field of ['name', 'description']) {
+    if (typeof metadata[field] !== 'string' || !metadata[field]) {
+      throw new Error(`bundled Skill frontmatter requires "${field}"`)
+    }
+  }
+
+  return {
+    name: metadata.name,
+    description: metadata.description,
+    ...(metadata.whenToUse ? { whenToUse: metadata.whenToUse } : {}),
+    invocation: {
+      modelInvocable: metadata['disable-model-invocation'] === undefined
+        ? true
+        : !parseBoolean(metadata['disable-model-invocation'], 'disable-model-invocation'),
+      userInvocable: metadata['user-invocable'] === undefined
+        ? true
+        : parseBoolean(metadata['user-invocable'], 'user-invocable'),
+    },
+    source: 'npm:dsh-harbor-evolution',
+    resourceBase: { kind: 'directory', path: skillDir },
+    content: match[2].trim(),
+    path: skillPath,
+  }
+}
+
+export function loadBundledSkill() {
+  return parseSkillMarkdown(readFileSync(skillPath, 'utf8'))
+}

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "dsh-harbor-evolution",
-  "version": "0.1.0",
-  "description": "DeepSeek Harness plugin for snapshotting Cordis Candidates and evaluating them with Harbor.",
+  "version": "0.2.0",
+  "description": "DeepSeek Harness plugin and bundled Skill for safely evolving Cordis Candidates with Harbor.",
   "type": "module",
   "main": "index.js",
   "exports": "./index.js",
@@ -12,6 +12,7 @@
     "index.js",
     "lib/",
     "bin/",
+    "skills/",
     "cordis.patch.yml",
     "README.md",
     "LICENSE"
@@ -26,12 +27,15 @@
     }
   },
   "peerDependencies": {
+    "@deepseek-ai/dsh-skill": "^0.1.0-rc.6",
     "@deepseek-ai/dsh-tools": "^0.1.0-rc.6"
   },
   "dependencies": {
     "@deepseek-ai/schemastery": "3.18.1"
   },
   "devDependencies": {
+    "@deepseek-ai/cordis": "4.0.1",
+    "@deepseek-ai/dsh-skill": "0.1.0-rc.6",
     "@deepseek-ai/dsh-tools": "0.1.0-rc.6",
     "@deepseek-ai/schemastery": "3.18.1"
   },

package/skills/evolve-agent-with-harbor/SKILL.md

@@ -0,0 +1,127 @@
+---
+name: evolve-agent-with-harbor
+description: Initialize, evaluate, compare, and safely improve a DeepSeek Harness Agent with Harbor. Use when the user asks to set up Harbor evaluation or Agent self-evolution, clarify an evaluation contract, create a Candidate, Dataset, or Promotion Policy, investigate a failed Job, or decide whether a Candidate should replace a baseline.
+---
+
+# Evolve Agent With Harbor
+
+Turn a vague improvement request into a reproducible evaluation contract, immutable Candidates, comparable Harbor Jobs, and an auditable promotion recommendation. Treat Harbor as the experiment boundary; deployment remains outside this workflow.
+
+## Choose the operating mode
+
+Infer the narrowest mode that satisfies the request:
+
+- **Clarify**: define what progress means and what may change.
+- **Initialize**: create the missing Candidate, Dataset, and Promotion Policy structure.
+- **Evaluate**: snapshot and run one Candidate.
+- **Compare**: compare an existing baseline Job with a new Candidate Job.
+- **Evolve**: run the complete baseline, diagnosis, controlled change, regression, and gate loop.
+- **Meta-evaluate**: optimize a Verifier or Judge against human ground truth rather than optimizing the business Agent.
+
+Do not expand an evaluate-only request into code mutation or deployment.
+
+## Clarify the evaluation contract
+
+Inspect the workspace and existing configuration before asking questions. Summarize known values, then resolve only material gaps. Prefer no more than three grouped questions in one turn.
+
+Establish these fields:
+
+1. Business behavior and the failure being improved.
+2. Candidate path and stable Agent product identity.
+3. Harbor Dataset path, test population, and environment constraints.
+4. Primary metric, minimum metrics, non-regression metrics, and tolerances.
+5. Baseline Candidate or baseline Job.
+6. Allowed mutation surface, forbidden files, and side-effect boundaries.
+7. Run budget, repeat policy for stochastic Agents, and stopping condition.
+8. Promotion owner and external CI/CD boundary.
+
+Candidate and Dataset paths are hard blockers for a run. Baseline and promotion criteria are hard blockers for a comparison. Do not invent ground truth, metrics, or deployment authority. Offer explicit draft defaults when helpful, but obtain acceptance before using them as the evaluation contract.
+
+## Initialize safely
+
+When required files are missing, read `references/initialization.md` before creating them. Initialize only inside the configured `projectRoot` and preserve existing files.
+
+- Keep the baseline immutable; create a new versioned Candidate directory for every optimization attempt.
+- Keep secrets out of Candidate files. Inject equal credentials and permissions at runtime.
+- Pin direct and transitive dependencies with a lockfile.
+- Make the Verifier emit a primary reward plus diagnostic metrics and failure evidence.
+- Version the Promotion Policy. A policy or Verifier change requires a fresh baseline.
+- Use test accounts, mocks, or sandboxes for business side effects.
+
+If file-editing capabilities are unavailable, produce the exact initialization plan and unresolved choices instead of pretending files were created.
+
+## Run the stable evolution loop
+
+### 1. Establish the baseline
+
+Call `harbor_candidate_snapshot` for the baseline Candidate. Then call `harbor_eval_run` with the accepted Candidate and Dataset paths. Record the Candidate id, version, digest, Job path, evaluation-context digest, metrics, exceptions, and failed trials.
+
+`harbor_eval_run` already snapshots again before execution. Treat a digest mismatch as a real Candidate change, not as noise.
+
+### 2. Diagnose before changing
+
+Use the summary returned by `harbor_eval_run`; call `harbor_eval_result` only when reopening an existing Job or when the stable summary is needed again. Read failed samples and trajectories when available.
+
+Classify each failure as one of:
+
+- Agent capability or policy failure.
+- Tool-call, search, citation, or output-contract failure.
+- Dataset, Verifier, or ground-truth defect.
+- Infrastructure, dependency, permission, timeout, or deployment failure.
+- Stochastic variance requiring repeats.
+
+Do not optimize the Agent to compensate for a broken evaluation environment. Do not leak holdout answers or ground truth into the Candidate.
+
+### 3. Make one controlled change
+
+State one hypothesis that connects evidence to the proposed change. Create a new immutable Candidate version and modify only the accepted mutation surface. Never edit the baseline Candidate in place.
+
+Snapshot the new Candidate and verify that its digest differs. If it does not differ, stop because no new Candidate exists.
+
+### 4. Re-run under the same context
+
+Call `harbor_eval_run` with the same Dataset and evaluation settings. For stochastic Agents, use the accepted repeat policy for both baseline and Candidate; never cherry-pick the best run.
+
+Compare only Jobs whose `evaluation-context` digests match. If the context changed, establish a new baseline instead of claiming improvement.
+
+### 5. Apply the deterministic gate
+
+Call `harbor_candidate_compare` with the baseline Job, Candidate Job, and accepted Promotion Policy. Respect its decision:
+
+- `PROMOTE`: recommend promotion and provide the evidence package.
+- `REJECT`: keep the current Champion and explain each failed criterion.
+
+The gate is a recommendation boundary. Never deploy, mutate the active DSH profile, merge code, or replace the Champion unless the user separately authorizes the external CI/CD action.
+
+## Preserve experimental invariants
+
+- One Job binds one Candidate digest; one Candidate may have many Jobs.
+- Baseline and Candidate must share the same evaluation-context digest.
+- Keep Candidate, evaluation context, policy, summaries, trajectories, and gate report as checkpoints.
+- Separate infrastructure failures from capability failures in every report.
+- Change one causal factor per iteration unless the user explicitly accepts a bundled experiment.
+- Use hidden or held-out evaluation data for promotion; do not let the Optimizer train directly on it.
+- Report uncertainty when sample size or stochastic variance prevents a stable conclusion.
+
+## Handle evaluator meta-evaluation
+
+When the object being improved is the evaluator, rotate the roles:
+
+- Candidate is a Verifier or Judge version.
+- Dataset contains cases with independently maintained human ground truth.
+- Metrics measure evaluator alignment, such as RCR, bias, variance, calibration, latency, and cost.
+- Promotion compares evaluator Candidates under a fixed GT set and policy.
+
+Never let the Candidate Judge provide its own final ground truth or promotion decision.
+
+## Report each cycle
+
+Return a compact audit record containing:
+
+- Accepted evaluation contract and any remaining assumptions.
+- Baseline and Candidate ids, versions, digests, and Job paths.
+- Evaluation-context and Promotion Policy identities.
+- Primary and diagnostic metric deltas.
+- Representative failure evidence and root-cause classification.
+- Gate decision with exact reasons.
+- External action required for promotion and the next controlled hypothesis if rejected.

package/skills/evolve-agent-with-harbor/references/initialization.md

@@ -0,0 +1,114 @@
+# Initialization Reference
+
+Load this reference only when the user needs a new Harbor self-evolution workspace or is missing one of the required contracts.
+
+## Readiness checklist
+
+Confirm these prerequisites before the first Job:
+
+- Docker is available to Harbor.
+- Node.js 22+, the selected DSH version, `harbor`, and `harbor-dsh` are available to the DSH process.
+- `projectRoot` is the intended workspace security boundary.
+- Candidate, Dataset, Job, and Promotion Policy paths stay within `projectRoot`.
+- Runtime credentials use evaluation accounts and are not stored in Candidate files.
+
+## Clarification worksheet
+
+Use known repository evidence first. Ask the user only for unresolved choices.
+
+| Contract field | Example | Why it matters |
+| --- | --- | --- |
+| Target behavior | Produce cited research answers | Defines task success |
+| Candidate identity | `deep-research-agent` | Keeps v1/v2 in one product line |
+| Candidate path | `candidates/deep-research/v1` | Defines what is snapshotted |
+| Dataset path | `datasets/deep-research-regression` | Fixes tasks and Verifier |
+| Primary metric | `reward` | Ranks Candidates |
+| Minimums | completion and citation >= 0.95 | Prevents unsafe tradeoffs |
+| Non-regression | tool success, latency | Protects existing capability |
+| Mutation surface | prompt and search plugin only | Controls causal attribution |
+| Repeat policy | 5 fixed-seed runs | Controls stochastic variance |
+| Promotion owner | CI gate plus human approval | Keeps deployment external |
+
+## Recommended layout
+
+```text
+agent-workspace/
+├── candidates/
+│   └── <agent-id>/
+│       ├── v1/
+│       │   ├── cordis.yml
+│       │   ├── package.json
+│       │   ├── package-lock.json
+│       │   └── business plugins...
+│       └── v2/
+├── datasets/
+│   └── <suite>/
+│       ├── task.toml
+│       ├── instruction.md
+│       ├── environment/Dockerfile
+│       └── tests/
+│           ├── test.sh
+│           └── verifier files...
+├── policies/
+│   └── <suite>.json
+└── jobs/
+```
+
+Generate `candidate-manifest.json` with `harbor_candidate_snapshot`; do not hand-author it. One immutable Candidate may be evaluated by many Jobs.
+
+## Candidate rules
+
+A Candidate is the complete DSH/Cordis composition required to reproduce behavior:
+
+- `package.json` supplies the default Candidate id and version.
+- `cordis.yml` composes the model, tools, skills, loop, storage, and business plugins.
+- A lockfile pins transitive runtime dependencies.
+- Local plugin files and prompts belong in the Candidate and therefore affect its digest.
+- Secrets, mutable session state, Jobs, and production deployment configuration do not belong in the Candidate.
+
+Copy an existing known-good Candidate when possible. If none exists, create the smallest valid DSH composition for the actual business Agent; do not invent a model provider or credential scheme.
+
+## Dataset and Verifier rules
+
+A Harbor Dataset contains one or more fixed Tasks. Each Task should define:
+
+- `task.toml`: Task identity, timeouts, environment, and Harbor contract.
+- `instruction.md`: the behavior requested from the Candidate.
+- `environment/Dockerfile`: a reproducible sandbox with required runtime dependencies.
+- `tests/test.sh`: the verifier entrypoint.
+- Verifier code that writes primary and diagnostic metrics to Harbor's reward output.
+
+For a research Agent, useful diagnostic metrics include task completion, tool-call success, valid-search rate, citation correctness, latency, and cost. Keep their exact definitions in version control. Do not convert tool errors, empty searches, or invalid citations into prose-only observations; expose them as metrics or structured failure evidence.
+
+## Promotion Policy starter
+
+Create this only after the user accepts the metric names and thresholds:
+
+```json
+{
+  "schema_version": 1,
+  "primary_metric": "reward",
+  "min_improvement": 0.05,
+  "minimums": {
+    "task_completion": 0.95
+  },
+  "non_regression": [
+    "tool_call_success",
+    "citation_correctness"
+  ],
+  "non_regression_tolerance": 0.0
+}
+```
+
+This is a structural example, not a universal default. Thresholds must reflect business risk and sample size. Version the policy; changing it invalidates comparisons made under the old promotion contract.
+
+## First-cycle handoff
+
+Before calling an evaluation tool, show the user:
+
+1. The resolved Candidate, Dataset, Job, and policy paths.
+2. The metric and promotion contract.
+3. The mutation and side-effect boundaries.
+4. Any assumptions still being used.
+
+Then snapshot and run the baseline. Do not initialize v2 until baseline evidence identifies a concrete hypothesis.