dsh-harbor-evolution 0.3.1 → 0.5.0

package/skills/evolve-agent-with-harbor/SKILL.md

@@ -1,127 +1,154 @@
 ---
 name: evolve-agent-with-harbor
-description: Initialize, evaluate, compare, and safely improve a DeepSeek Harness Agent with Harbor. Use when the user asks to set up Harbor evaluation or Agent self-evolution, clarify an evaluation contract, create a Candidate, Dataset, or Promotion Policy, investigate a failed Job, or decide whether a Candidate should replace a baseline.
+description: Architect, initialize, diagnose, evaluate, compare, and safely improve a DeepSeek Harness business Agent with Harbor Evaluation Stack, Dataset Manifest, Context v2, Architecture Doctor, and Promotion Gate. Use for Harbor setup, Agent self-evolution, vertical-search evaluation loops, evaluation architecture review, failed Job diagnosis, Candidate optimization, evaluator meta-evaluation, or promotion decisions.
 ---
 
 # Evolve Agent With Harbor
 
-Turn a vague improvement request into a reproducible evaluation contract, immutable Candidates, comparable Harbor Jobs, and an auditable promotion recommendation. Treat Harbor as the experiment boundary; deployment remains outside this workflow.
+Build a reproducible improvement loop around three boundaries:
 
-## Choose the operating mode
+- **Generator/Candidate**: the immutable DSH Agent composition being improved.
+- **Evaluator/Evaluation Stack**: Integration, Renderer, Evaluator, Rubric, Diagnoser, Optimizer, Runner, Reporter, and Judge identities.
+- **Optimizer**: proposes one evidence-linked Candidate change; it never controls the final Gate.
 
-Infer the narrowest mode that satisfies the request:
+Treat Harbor as the experiment boundary. Deployment, CI/CD, and Champion replacement remain external actions requiring separate authority.
 
-- **Clarify**: define what progress means and what may change.
-- **Initialize**: create the missing Candidate, Dataset, and Promotion Policy structure.
-- **Evaluate**: snapshot and run one Candidate.
-- **Compare**: compare an existing baseline Job with a new Candidate Job.
-- **Evolve**: run the complete baseline, diagnosis, controlled change, regression, and gate loop.
-- **Meta-evaluate**: optimize a Verifier or Judge against human ground truth rather than optimizing the business Agent.
+## Select the narrowest mode
 
-Do not expand an evaluate-only request into code mutation or deployment.
+- **Clarify**: define progress, identities, constraints, and promotion ownership.
+- **Architecture**: inspect role boundaries and run `harbor_evolution_doctor`.
+- **Initialize**: read `references/initialization.md`, obtain explicit values, then call `harbor_evolution_init`.
+- **Diagnostic**: investigate failures without making a promotion claim.
+- **Promotion**: run a `promotion-eligible` Job and apply the deterministic Gate.
+- **Evolve**: baseline → diagnose → one controlled change → regression Job → Gate.
+- **Meta-evaluate**: improve an Evaluator/Judge against independently maintained human GT.
 
-## Clarify the evaluation contract
+Do not turn an inspection or diagnostic request into Agent mutation or deployment.
 
-Inspect the workspace and existing configuration before asking questions. Summarize known values, then resolve only material gaps. Prefer no more than three grouped questions in one turn.
+## Clarify before initializing
 
-Establish these fields:
+Inspect the workspace first. Resolve only material gaps, preferably in no more than three grouped questions. Obtain:
 
-1. Business behavior and the failure being improved.
-2. Candidate path and stable Agent product identity.
-3. Harbor Dataset path, test population, and environment constraints.
-4. Primary metric, minimum metrics, non-regression metrics, and tolerances.
-5. Baseline Candidate or baseline Job.
-6. Allowed mutation surface, forbidden files, and side-effect boundaries.
-7. Run budget, repeat policy for stochastic Agents, and stopping condition.
-8. Promotion owner and external CI/CD boundary.
+1. Business behavior, failure pattern, and Candidate path/product identity.
+2. Dataset path/id/version, task population, holdout boundary, and side-effect sandbox.
+3. Evaluation Stack id/version and one entry for every required role.
+4. Judge provider/model/version/parameters without credentials.
+5. Evaluation Contract id/version, primary metric and direction, diagnostic metrics, groups, and hard requirements.
+6. Promotion Policy id/version, delta, minimums, maximums, non-regression metrics, and metric directions.
+7. Baseline Job/Candidate, repeat policy, run budget, stopping rule, allowed mutation surface, and forbidden files.
+8. Promotion owner and external CI/CD handoff.
 
-Candidate and Dataset paths are hard blockers for a run. Baseline and promotion criteria are hard blockers for a comparison. Do not invent ground truth, metrics, or deployment authority. Offer explicit draft defaults when helpful, but obtain acceptance before using them as the evaluation contract.
+Never invent GT, a Judge model, reward definitions, thresholds, credentials, or deployment authority. Offer draft values only when clearly labeled and accepted.
 
-## Initialize safely
+## Enforce the strict architecture
 
-When required files are missing, read `references/initialization.md` before creating them. Initialize only inside the configured `projectRoot` and preserve existing files.
+Require these before every Job:
 
-- Keep the baseline immutable; create a new versioned Candidate directory for every optimization attempt.
-- Keep secrets out of Candidate files. Inject equal credentials and permissions at runtime.
-- Pin direct and transitive dependencies with a lockfile.
-- Make the Verifier emit a primary reward plus diagnostic metrics and failure evidence.
-- Version the Promotion Policy. A policy or Verifier change requires a fresh baseline.
-- Use test accounts, mocks, or sandboxes for business side effects.
+- `candidate-manifest.json` verified against the Candidate files.
+- `dataset-manifest.json` with unique task ids, non-empty instructions, safe paths, and a matching source digest.
+- `.harbor/evaluation-stack.yml` with all eight roles, Judge identity, and Evaluation Contract.
+- Evaluation Context v2 preview.
 
-If file-editing capabilities are unavailable, produce the exact initialization plan and unresolved choices instead of pretending files were created.
+Before a formal Job, call in order:
 
-## Run the stable evolution loop
+1. `harbor_candidate_snapshot`
+2. `harbor_dataset_validate`
+3. `harbor_evolution_doctor`
+4. `harbor_context_preview`
 
-### 1. Establish the baseline
+Do not launch a `promotion-eligible` Job when Doctor reports an error, no comparable baseline exists, or `fresh_baseline_required` is true. A diagnostic Job may investigate architecture warnings, but still requires a valid Candidate, Dataset Manifest, Evaluation Stack, and Context v2.
 
-Call `harbor_candidate_snapshot` for the baseline Candidate. Then call `harbor_eval_run` with the accepted Candidate and Dataset paths. Record the Candidate id, version, digest, Job path, evaluation-context digest, metrics, exceptions, and failed trials.
+Keep Runner orchestration-only. Treat these as architecture errors:
 
-`harbor_eval_run` already snapshots again before execution. Treat a digest mismatch as a real Candidate change, not as noise.
+- Runner combines HTTP integration, rubric, and Judge logic.
+- Runner makes a promotion/Champion decision.
 
-### 2. Diagnose before changing
+## Initialize without overwriting
 
-Use the summary returned by `harbor_eval_run`; call `harbor_eval_result` only when reopening an existing Job or when the stable summary is needed again. Read failed samples and trajectories when available.
+Read `references/initialization.md` when required files are missing. After the user accepts all required identities and metric semantics, call `harbor_evolution_init`. It preserves existing files and creates explicit placeholders that still require business implementation.
 
-Classify each failure as one of:
+After initialization:
 
-- Agent capability or policy failure.
-- Tool-call, search, citation, or output-contract failure.
-- Dataset, Verifier, or ground-truth defect.
-- Infrastructure, dependency, permission, timeout, or deployment failure.
-- Stochastic variance requiring repeats.
+- Replace placeholders with real role implementations.
+- Pin Candidate dependencies and keep secrets runtime-injected.
+- Re-snapshot the Dataset after intentional Dataset changes.
+- Run Doctor again; initialization success is not evaluation readiness.
+
+## Determine comparability correctly
+
+Use the Context v2 `digest`, not timestamps or Job names.
+
+A fresh baseline is required when any of these change:
+
+- Dataset id, version, or source digest.
+- Integration, Renderer, Evaluator, or Rubric identity.
+- Judge provider, model, version, or parameters.
+- Runner marked `semantic: true`.
+- Harbor or integration runtime identity.
 
-Do not optimize the Agent to compensate for a broken evaluation environment. Do not leak holdout answers or ground truth into the Candidate.
+Diagnoser, Optimizer, Reporter, and non-semantic Runner changes remain comparable but change the full audit digest. A Candidate digest must differ from the baseline Candidate digest. Promotion Policy is reapplied as a separately versioned decision contract; changing it does not rewrite Evaluation Context.
 
-### 3. Make one controlled change
+## Run the evolution loop
 
-State one hypothesis that connects evidence to the proposed change. Create a new immutable Candidate version and modify only the accepted mutation surface. Never edit the baseline Candidate in place.
+### Establish a baseline
 
-Snapshot the new Candidate and verify that its digest differs. If it does not differ, stop because no new Candidate exists.
+Call `harbor_eval_run` with Candidate, Dataset, Stack, explicit `mode`, and a Policy for `promotion-eligible`. Preserve Candidate, Dataset, Stack, Context, Doctor, Contract, Trial assessments, Population report, Summary, and later Promotion report.
+
+Never cherry-pick stochastic runs. Apply the accepted repeat/seed policy symmetrically.
+
+### Diagnose before changing
+
+Use `harbor_eval_result` only to reopen a stable Job summary. Inspect Trial assessments and classify each failure as:
+
+- Candidate capability or policy.
+- Tool-call, invalid search, citation, or output-contract failure.
+- Dataset, Evaluator, Rubric, Judge, or GT defect.
+- Infrastructure, dependency, permission, timeout, or deployment failure.
+- Stochastic uncertainty.
 
-### 4. Re-run under the same context
+Do not optimize the Candidate around broken evaluation infrastructure. Never leak holdout answers or GT into Candidate prompts, skills, tools, or memory.
 
-Call `harbor_eval_run` with the same Dataset and evaluation settings. For stochastic Agents, use the accepted repeat policy for both baseline and Candidate; never cherry-pick the best run.
+### Propose one controlled change
 
-Compare only Jobs whose `evaluation-context` digests match. If the context changed, establish a new baseline instead of claiming improvement.
+Require every optimization hypothesis to include:
 
-### 5. Apply the deterministic gate
+- Evidence references to Job/Trial/findings.
+- Root-cause classification.
+- Expected metric effect.
+- Exact mutation surface and forbidden surface.
+- Rollback condition.
 
-Call `harbor_candidate_compare` with the baseline Job, Candidate Job, and accepted Promotion Policy. Respect its decision:
+Create a new immutable Candidate version; never edit the baseline in place. Stop when the new digest is unchanged.
 
-- `PROMOTE`: recommend promotion and provide the evidence package.
-- `REJECT`: keep the current Champion and explain each failed criterion.
+### Re-run and gate
 
-The gate is a recommendation boundary. Never deploy, mutate the active DSH profile, merge code, or replace the Champion unless the user separately authorizes the external CI/CD action.
+Call `harbor_context_preview`; establish a fresh baseline if needed. Run the Candidate under the same comparable Context. Then call `harbor_candidate_compare`.
 
-## Preserve experimental invariants
+- `PROMOTE`: recommend external promotion with the complete evidence package.
+- `REJECT`: keep the Champion and explain every structured reason code.
 
-- One Job binds one Candidate digest; one Candidate may have many Jobs.
-- Baseline and Candidate must share the same evaluation-context digest.
-- Keep Candidate, evaluation context, policy, summaries, trajectories, and gate report as checkpoints.
-- Separate infrastructure failures from capability failures in every report.
-- Change one causal factor per iteration unless the user explicitly accepts a bundled experiment.
-- Use hidden or held-out evaluation data for promotion; do not let the Optimizer train directly on it.
-- Report uncertainty when sample size or stochastic variance prevents a stable conclusion.
+Never bypass `INFRASTRUCTURE_EXCEPTION_PRESENT`, `ARTIFACT_SCHEMA_INVALID`, Dataset/Stack/Rubric/Judge mismatch, or non-regression failures.
 
 ## Handle evaluator meta-evaluation
 
-When the object being improved is the evaluator, rotate the roles:
+Rotate roles when improving the Evaluator:
 
-- Candidate is a Verifier or Judge version.
-- Dataset contains cases with independently maintained human ground truth.
-- Metrics measure evaluator alignment, such as RCR, bias, variance, calibration, latency, and cost.
-- Promotion compares evaluator Candidates under a fixed GT set and policy.
+- Candidate is the Evaluator/Rubric/Judge version.
+- Dataset contains independently maintained human GT.
+- Metrics include RCR, bias, variance, calibration, latency, and cost as appropriate.
+- The Candidate evaluator must not author its own GT or final promotion decision.
 
-Never let the Candidate Judge provide its own final ground truth or promotion decision.
+Manage evaluator Candidates and meta-evaluation Jobs with the same Manifest, Context v2, Doctor, evidence, and Gate rules.
 
 ## Report each cycle
 
-Return a compact audit record containing:
+Return:
 
-- Accepted evaluation contract and any remaining assumptions.
-- Baseline and Candidate ids, versions, digests, and Job paths.
-- Evaluation-context and Promotion Policy identities.
-- Primary and diagnostic metric deltas.
-- Representative failure evidence and root-cause classification.
-- Gate decision with exact reasons.
-- External action required for promotion and the next controlled hypothesis if rejected.
+- Accepted Evaluation Contract and unresolved assumptions.
+- Candidate, Dataset, Stack, Context, Judge, and Policy identities.
+- Comparable baseline or fresh-baseline decision.
+- Metric deltas, exception counts, Population groups, and artifact validation.
+- Representative Trial evidence and root-cause classes.
+- Controlled change hypothesis and mutation surface.
+- Gate decision with exact reason codes.
+- External CI/CD action still required.

package/skills/evolve-agent-with-harbor/references/initialization.md

@@ -1,114 +1,120 @@
-# Initialization Reference
+# Strict Project Initialization
 
-Load this reference only when the user needs a new Harbor self-evolution workspace or is missing one of the required contracts.
+Load this reference only when the project is missing the Evaluation Stack structure or the user asks to initialize it.
 
-## Readiness checklist
+## Readiness worksheet
 
-Confirm these prerequisites before the first Job:
+Resolve every value before calling `harbor_evolution_init`:
 
-- Docker is available to Harbor.
-- Node.js 22+, the selected DSH version, `harbor`, and `harbor-dsh` are available to the DSH process.
-- `projectRoot` is the intended workspace security boundary.
-- Candidate, Dataset, Job, and Promotion Policy paths stay within `projectRoot`.
-- Runtime credentials use evaluation accounts and are not stored in Candidate files.
+| Field | Required meaning |
+| --- | --- |
+| `datasetPath` | Existing Harbor Dataset inside `projectRoot` |
+| `stackId` / `stackVersion` | Stable identity of the complete evaluation architecture |
+| `datasetId` / `datasetVersion` | Stable identity of task population and GT boundary |
+| `contractId` / `contractVersion` | Stable metric semantics |
+| `primaryMetric` / `primaryDirection` | Exact reward key and `maximize` or `minimize` |
+| Judge provider/model/version | Reproducible Judge identity, never credentials |
+| Policy id/version | Stable Gate identity |
+| `minImprovement` | Accepted primary-metric delta |
 
-## Clarification worksheet
+Also establish diagnostic metrics, min/max thresholds, non-regression metrics, mutation surface, repeat policy, and promotion owner. The initializer creates a minimal Policy; update its explicit placeholders before a formal Gate.
 
-Use known repository evidence first. Ask the user only for unresolved choices.
-
-| Contract field | Example | Why it matters |
-| --- | --- | --- |
-| Target behavior | Produce cited research answers | Defines task success |
-| Candidate identity | `deep-research-agent` | Keeps v1/v2 in one product line |
-| Candidate path | `candidates/deep-research/v1` | Defines what is snapshotted |
-| Dataset path | `datasets/deep-research-regression` | Fixes tasks and Verifier |
-| Primary metric | `reward` | Ranks Candidates |
-| Minimums | completion and citation >= 0.95 | Prevents unsafe tradeoffs |
-| Non-regression | tool success, latency | Protects existing capability |
-| Mutation surface | prompt and search plugin only | Controls causal attribution |
-| Repeat policy | 5 fixed-seed runs | Controls stochastic variance |
-| Promotion owner | CI gate plus human approval | Keeps deployment external |
-
-## Recommended layout
+## Generated layout
 
 ```text
-agent-workspace/
+projectRoot/
+├── .harbor/
+│   ├── evolution.yml
+│   └── evaluation-stack.yml
 ├── candidates/
-│   └── <agent-id>/
-│       ├── v1/
-│       │   ├── cordis.yml
-│       │   ├── package.json
-│       │   ├── package-lock.json
-│       │   └── business plugins...
-│       └── v2/
 ├── datasets/
-│   └── <suite>/
-│       ├── task.toml
-│       ├── instruction.md
-│       ├── environment/Dockerfile
-│       └── tests/
-│           ├── test.sh
-│           └── verifier files...
-├── policies/
-│   └── <suite>.json
+├── integrations/default.py
+├── renderers/default.py
+├── evaluators/default.py
+├── rubrics/default.md
+├── diagnosers/default.py
+├── optimizers/default.py
+├── runners/harbor.py
+├── reporters/default.py
+├── policies/promotion.json
 └── jobs/
 ```
 
-Generate `candidate-manifest.json` with `harbor_candidate_snapshot`; do not hand-author it. One immutable Candidate may be evaluated by many Jobs.
-
-## Candidate rules
-
-A Candidate is the complete DSH/Cordis composition required to reproduce behavior:
-
-- `package.json` supplies the default Candidate id and version.
-- `cordis.yml` composes the model, tools, skills, loop, storage, and business plugins.
-- A lockfile pins transitive runtime dependencies.
-- Local plugin files and prompts belong in the Candidate and therefore affect its digest.
-- Secrets, mutable session state, Jobs, and production deployment configuration do not belong in the Candidate.
-
-Copy an existing known-good Candidate when possible. If none exists, create the smallest valid DSH composition for the actual business Agent; do not invent a model provider or credential scheme.
+The initializer never overwrites existing files. `created` and `preserved` in its result are part of the audit. Placeholder components provide identities, not a finished business evaluator.
+
+## Evaluation Stack shape
+
+```yaml
+schema_version: 1
+stack_id: vertical-search
+version: 1.0.0
+components:
+  integration: { id: search-api, version: 1.0.0, entry: integrations/default.py }
+  renderer: { id: search-renderer, version: 1.0.0, entry: renderers/default.py }
+  evaluator: { id: search-evaluator, version: 1.0.0, entry: evaluators/default.py }
+  rubric: { id: search-rubric, version: 1.0.0, entry: rubrics/default.md }
+  diagnoser: { id: search-diagnoser, version: 1.0.0, entry: diagnosers/default.py }
+  optimizer: { id: search-optimizer, version: 1.0.0, entry: optimizers/default.py }
+  runner: { id: harbor-runner, version: 1.0.0, entry: runners/harbor.py, semantic: false }
+  reporter: { id: search-reporter, version: 1.0.0, entry: reporters/default.py }
+judge:
+  provider: accepted-provider
+  model: accepted-model
+  version: pinned-version
+  parameters: { temperature: 0 }
+evaluation_contract:
+  contract_id: vertical-search
+  version: 1.0.0
+  primary_metric: reward
+  metrics:
+    - { id: reward, label: Overall reward, direction: maximize }
+    - { id: valid_search_rate, label: Valid search rate, direction: maximize }
+    - { id: citation_accuracy, label: Citation accuracy, direction: maximize }
+```
 
-## Dataset and Verifier rules
+Do not duplicate Evaluator logic inside each Task. Reference one Stack Evaluator from task metadata when task-specific routing is needed.
 
-A Harbor Dataset contains one or more fixed Tasks. Each Task should define:
+## Dataset rules
 
-- `task.toml`: Task identity, timeouts, environment, and Harbor contract.
-- `instruction.md`: the behavior requested from the Candidate.
-- `environment/Dockerfile`: a reproducible sandbox with required runtime dependencies.
-- `tests/test.sh`: the verifier entrypoint.
-- Verifier code that writes primary and diagnostic metrics to Harbor's reward output.
+Keep `dataset-manifest.json` at Dataset root. Generate it through initialization or `harbor-dsh dataset snapshot`; do not hand-edit its digest. Validation rejects:
 
-For a research Agent, useful diagnostic metrics include task completion, tool-call success, valid-search rate, citation correctness, latency, and cost. Keep their exact definitions in version control. Do not convert tool errors, empty searches, or invalid citations into prose-only observations; expose them as metrics or structured failure evidence.
+- Missing/duplicate Task ids or empty instructions.
+- Duplicate normalized queries.
+- Missing/out-of-root paths or symlinks.
+- Source/file counts that no longer match.
+- Secret-bearing metadata fields.
 
-## Promotion Policy starter
+Intentional Dataset changes require a new Dataset version, a new snapshot, and a fresh baseline.
 
-Create this only after the user accepts the metric names and thresholds:
+## Promotion Policy v2
 
 ```json
 {
-  "schema_version": 1,
+  "schema_version": 2,
+  "policy_id": "vertical-search",
+  "version": "1.0.0",
   "primary_metric": "reward",
+  "primary_direction": "maximize",
   "min_improvement": 0.05,
-  "minimums": {
-    "task_completion": 0.95
-  },
-  "non_regression": [
-    "tool_call_success",
-    "citation_correctness"
-  ],
-  "non_regression_tolerance": 0.0
+  "minimums": { "citation_accuracy": 0.9 },
+  "maximums": { "latency_seconds": 20 },
+  "non_regression": ["valid_search_rate"],
+  "metric_directions": { "valid_search_rate": "maximize" },
+  "non_regression_tolerance": 0.0,
+  "hard_requirements": ["exception_free", "artifact_schema_valid", "doctor_error_free"]
 }
 ```
 
-This is a structural example, not a universal default. Thresholds must reflect business risk and sample size. Version the policy; changing it invalidates comparisons made under the old promotion contract.
+Use business-accepted thresholds. Do not assume all metrics are `/10`, maximized, or universal across domains.
 
-## First-cycle handoff
+## Handoff before the first Job
 
-Before calling an evaluation tool, show the user:
+Show the user:
 
-1. The resolved Candidate, Dataset, Job, and policy paths.
-2. The metric and promotion contract.
-3. The mutation and side-effect boundaries.
-4. Any assumptions still being used.
+1. Resolved Candidate, Dataset, Stack, Policy, and Jobs paths.
+2. Role identities and which ones affect reward comparability.
+3. Metric directions, thresholds, groups, and hard requirements.
+4. Holdout, mutation, side-effect, repeat, and deployment boundaries.
+5. Doctor findings and Context preview.
 
-Then snapshot and run the baseline. Do not initialize v2 until baseline evidence identifies a concrete hypothesis.
+Start with a baseline. Do not create Candidate v2 until baseline evidence supports one controlled hypothesis.