dsh-harbor-evolution 0.7.2 → 0.8.0

package/skills/evolve-agent-with-harbor/references/evaluator-upgrade.md

@@ -14,6 +14,35 @@ Inspect `harbor_eval_result view=governance` and representative Trial evidence f
 
 Do not use the Candidate Agent to author its own GT. Do not infer GT from the current Evaluator output.
 
+## Ingest reviewed reports in the user's language
+
+Prefer evidence the user already has over a schema questionnaire. Accept:
+
+- a pasted report followed by a pasted score, review, or recommendation;
+- report and review file paths;
+- a directory containing several report/review pairs;
+- an exported table or document whose rows can be traced back to individual reports.
+
+Ask for missing report/review pairing only when it cannot be inferred safely. Query text, source material, citations, and execution traces are useful context but optional unless the active Rubric needs them.
+
+Preserve the raw report and raw review before normalization. Build a review draft with one row per report and Criterion:
+
+| Field | Rule |
+| --- | --- |
+| Report | Stable case label and source path or pasted-input reference |
+| Criterion | Map to the active Rubric; propose a new Criterion only when the review clearly introduces a distinct standard |
+| Score | Preserve an explicit score; otherwise infer only through the accepted Rubric scale |
+| Reason | Extract the reviewer's stated basis without strengthening it |
+| Recommendation | Preserve the reviewer's suggestion; if absent, leave it empty rather than inventing expert intent. It does not block a confirmed scoring label |
+| Evidence | Keep the shortest exact source excerpt that supports the normalized row |
+| Status | `explicit`, `inferred`, or `unresolved` |
+
+Show this draft in plain language. Do not mention protocol fields in the first confirmation. Ask a targeted question such as “这里更接近部分满足（0.5），还是完全不满足（0）？” instead of asking the user to edit JSON. Combine all unresolved rows into one short confirmation when possible.
+
+Only confirmed rows may become formal GT. Preserve the raw material alongside the normalized case so later reviewers can audit whether the Agent translated the text correctly. Record who or what produced the review, when available. If the reviewer source is unknown, state that provenance remains unresolved and keep the Dataset diagnostic-only.
+
+One reviewed report is useful for wiring and discussion but cannot establish general reliability. When several examples exist, keep representative normal cases and historical failures, and reserve an untouched group for final comparison. The Optimizer may learn from the tuning group; neither the Candidate evaluator nor the Optimizer may see holdout labels before the final measurement.
+
 ## Create a new immutable evaluator identity
 
 Show the current identity, source, proposed diff, and expected semantic impact. Create new files and increment component plus Evaluation Stack versions. Never edit a historical Evaluator, Rubric, Judge identity, or old Job artifact in place.
@@ -37,6 +66,8 @@ Treat changes to any of these as reward-semantic changes requiring a new Context
 
 ## Meta-evaluate before adopting
 
+The current capability is an independent artifact workflow, not a Harbor Job kind: `harbor_ground_truth_init` creates the provenance-bearing GT draft, repeated Evaluator runs produce fixed observations, and `harbor_evaluator_meta_evaluate` writes `meta-evaluation-report/v1`. It does not run inside a Historical Generation Job, and it does not emit a `job_kind=evaluator-meta-evaluation` Job. A future dedicated Job may package this existing workflow into a recoverable lifecycle.
+
 Rotate roles:
 
 - Candidate: the new Evaluator/Rubric/Judge version.
@@ -47,7 +78,9 @@ Rotate roles:
 
 Use the same GT set, repeat policy, and measurement procedure for old and new evaluator Candidates. Report coverage, invalid measurements, aggregate metrics, disagreement slices, latency, cost, and representative errors. Do not select only favorable runs.
 
-GT is not synonymous with human labeling. It may be human, programmatic, consensus-based, produced by a separately pinned model, or imported from an external standard. Require explicit provenance and independence from the Candidate evaluator in every case. Use `harbor_ground_truth_init` to create the versioned draft and `harbor_evaluator_meta_evaluate` to calculate ESF, SCE, and RCR from repeated observations.
+GT is not synonymous with human labeling. It may be human, programmatic, consensus-based, produced by a separately pinned model, or imported from an external standard. Require explicit provenance and independence from the Candidate evaluator in every case. Use `harbor_ground_truth_init` to create the versioned draft and `harbor_evaluator_meta_evaluate` to calculate ESF, SCE, and RCR from repeated observations. Keep a Historical Job's own `evaluator_meta_evaluation.status=not-run`; never attach a separate report implicitly or derive it from Session Population scores.
+
+The user supplies reports and review evidence; the Agent performs the protocol adaptation. Do not ask the user to hand-author `ground-truth/v1` or `evaluator-observations/v1`. After confirmation, generate the strict files, validate them, run repeated Evaluator observations, and retain a reversible mapping from every normalized field to its raw source.
 
 ## Re-baseline Agent progress
 

package/skills/evolve-agent-with-harbor/references/initialization.md

@@ -9,7 +9,7 @@ Keep onboarding anchored on four visible concepts:
 | User concept | Plain-language prompt | Accepted input | Compiled architecture |
 | --- | --- | --- | --- |
 | 评测集 (Dataset) | 测什么？ | One Query, file, instruction directory, or Dataset | Dataset tasks, manifest, population, diagnostic/regression mode |
-| 生成器 (Generator) | 谁来回答？ | curl or local/detected Agent | Candidate snapshot, Integration, Renderer, runtime identity |
+| 生成器 (Generator) | 谁来回答？ | curl, local/detected Agent, or “使用当前 Harbor Agent 模型” | Candidate snapshot, Integration, Renderer, runtime identity, and optional pinned model binding |
 | 评测器（评测标准） (Evaluator) | 怎样算好？ | evaluator curl/path, or natural-language criteria | Evaluator, Rubric, Judge identity, Evaluation Contract |
 | 优化器 (Optimizer) | 谁根据结果改进？ | current Agent, Codex, Claude Code, or local command | Optimizer identity, mutation surface, rollback workflow |
 
@@ -17,6 +17,8 @@ Inspect first and prefill everything reliable. Ask only about missing rows. Use
 
 Before any write, show the four rows plus inferred workspace, diagnostic/promotion scope, and deferred capabilities. The user may start initialization, modify the card, or open advanced configuration.
 
+When the accepted Generator is “使用当前 Harbor Agent 模型”, call `harbor_model_binding` after confirmation. Write the returned `candidate_model_binding` as `model-binding.json` before Candidate snapshot. The binding is immutable Candidate identity; runtime access still uses the per-Job Host Model Broker and its short-lived capability. Never write authentication material or a Host auth-file path into the Candidate directory.
+
 ## Internal compilation worksheet
 
 The Agent, not the user, compiles the accepted concept card into the strict `harbor_evolution_init` arguments:
@@ -24,6 +26,7 @@ The Agent, not the user, compiles the accepted concept card into the strict `har
 | Field | Required meaning |
 | --- | --- |
 | `datasetPath` | Existing Harbor Dataset inside `projectRoot` |
+| `workspaceSubdir` | Optional namespace for an independent Stack; use it when another Stack already owns the root `.harbor/` |
 | `stackId` / `stackVersion` | Stable identity of the complete evaluation architecture |
 | `datasetId` / `datasetVersion` | Stable identity of task population and GT boundary |
 | `contractId` / `contractVersion` | Stable metric semantics |
@@ -59,7 +62,7 @@ projectRoot/
 └── jobs/
 ```
 
-The initializer never overwrites existing files. `created` and `preserved` in its result are part of the audit. Placeholder components provide identities, not a finished business evaluator.
+The initializer never overwrites existing files. `created` and `preserved` in its result are part of the audit. If an existing Stack has a different id, initialization fails with `STACK_ALREADY_EXISTS_DIFFERENT_ID`; choose a separate `workspaceSubdir`. Placeholder components provide identities, not a finished business evaluator.
 
 ## Evaluation Stack shape
 
@@ -111,6 +114,10 @@ Keep `dataset-manifest.json` at Dataset root. Generate it through initialization
 - Missing/out-of-root paths or symlinks.
 - Source/file counts that no longer match.
 - Secret-bearing metadata fields.
+- A Task at the Dataset root, a deeply nested Task, or any manifest Task Harbor cannot resolve as an immediate child directory.
+- Missing Harbor 1.4 identity (`schema_version = "1.4"` and `[task].name = "org/name"`).
+
+When the user only needs a first wiring check, `harbor_quick_diagnostic_init` creates this standard structure plus a minimal Host-model Candidate. It is explicitly `promotion_eligible=false`: its verifier proves execution reached Harbor but does not apply the saved business Rubric draft.
 
 Intentional Dataset changes require a new Dataset version, a new snapshot, and a fresh baseline.