dsh-harbor-evolution 0.7.2 → 0.8.0

package/README.md

@@ -2,7 +2,7 @@
 
 Installable DeepSeek Harness Plugin + Skill for running stable Harbor evaluation and controlled Agent evolution loops, with a native DSH Web dashboard.
 
-The package gives DSH twelve strict Harbor tools, dedicated Tool cards, a nine-stage Evaluation Workbench, an installation Doctor, and the model- and user-invocable `evolve-agent-with-harbor` Skill. The Skill starts with four user-facing concepts—Dataset (what to test), Generator (who answers), Evaluator plus criteria (what good means), and Optimizer (who improves it)—then compiles accepted choices into the strict Evaluation Stack. It validates Dataset identity, checks Trial Lifecycle and Score Validity, governs independent Ground Truth meta-evaluation, diagnoses evidence provenance, limits each iteration to one controlled Candidate change, and invokes the Promotion Gate only as an explicit action.
+The package gives DSH sixteen strict Harbor tools, dedicated Tool cards, a nine-stage Evaluation Workbench, an installation Doctor, and the model- and user-invocable `evolve-agent-with-harbor` Skill. The Skill starts with four user-facing concepts—Dataset (what to test), Generator (who answers), Evaluator plus criteria (what good means), and Optimizer (who improves it)—then compiles accepted choices into the strict Evaluation Stack. When no Dataset is supplied, it can instead preview recent completed DSH Sessions and evaluate each immutable Session as one Historical Trial without rerunning a Candidate. A DSH Generator may explicitly pin the current default model as a non-secret Candidate identity while retaining the per-Job Host Broker credential boundary. The Plugin validates Dataset identity, checks Trial Lifecycle and Score Validity, governs independent Ground Truth meta-evaluation, diagnoses evidence provenance, limits each iteration to one controlled Candidate change, and invokes the Promotion Gate only as an explicit action.
 
 ## Install
 
@@ -14,8 +14,8 @@ npx --yes dsh-harbor-evolution@latest setup --project-root "$PWD"
 
 The setup command installs both required runtimes:
 
-- `harbor-dsh-evolution==0.7.2` in a managed Python environment.
-- `dsh-harbor-evolution@0.7.2` in the selected DSH profile.
+- `harbor-dsh-evolution==0.8.0` in a managed Python environment.
+- `dsh-harbor-evolution@0.8.0` in the selected DSH profile.
 
 It then stores the absolute Harbor executable paths and a fallback `projectRoot` in the profile's `harbor-evolution` block and verifies the integration. Agent Tool calls always use the calling session's absolute working directory as their project root; the configured value remains the Web Workbench and non-Agent fallback. Existing unrelated profile entries are preserved, and rerunning setup updates the same block.
 
@@ -37,12 +37,20 @@ Users may provide a single Query or Dataset path, a Generator curl or local Agen
 The Plugin registers:
 
 - `harbor_candidate_snapshot`
+- `harbor_model_binding`
 - `harbor_evolution_init`
 - `harbor_evolution_doctor`
+- `harbor_quick_diagnostic_init`
+- `harbor_session_diagnostic_preview`
+- `harbor_session_diagnostic_run`
 - `harbor_dataset_validate`
 - `harbor_context_preview`
 - `harbor_eval_run`
 - `harbor_eval_result`
+- `harbor_evaluator_inspect`
+- `harbor_evaluator_update`
+- `harbor_ground_truth_init`
+- `harbor_evaluator_meta_evaluate`
 - `harbor_candidate_compare`
 
 In the `web` profile, the same package also registers:
@@ -51,7 +59,7 @@ In the `web` profile, the same package also registers:
 - descriptor-authorized Evaluator/Rubric source editing for `script` and `llm-as-judge` implementations, with optimistic concurrency and mandatory new identities;
 - a `harbor-dsh-evaluator/v1` interface shared by deterministic scripts and LLM-as-Judge implementations;
 - compact result cards for all Harbor Tool calls;
-- a `Harbor Evolution` Settings section that checks the configured project, Evaluation Stack, Jobs directory, and CLI paths.
+- a `Harbor Evolution` Settings section that checks the configured project, Evaluation Stack, Jobs directory, and CLI paths, supports process-local `projectRoot` reload, and checks npm for a newer formal release without silently installing it.
 
 The Web UI is intentionally read-only. Starting an evaluation or deciding promotion remains an explicit Agent + Skill workflow, so a page refresh can never launch an expensive Job.
 
@@ -63,6 +71,10 @@ Before each Job, the Plugin snapshots the current DSH Agent selection—provider
 
 `harbor_eval_run`, `harbor_context_preview`, and `harbor_evolution_doctor` inherit that selection by default. Advanced callers can override `candidateProvider` and `candidateModel` only as a pair, plus an optional `candidateReasoningEffort`. `openai-codex` performs a GPT Auth sign-in check before Harbor starts. The resulting model binding is part of Context v2 comparison identity, so any provider/model/reasoning change requires a new baseline.
 
+`harbor_model_binding` returns the current default selection as a credential-free `model-binding.json` draft. Once included before Candidate snapshot, it enters the Candidate digest and becomes the required Job model identity. Conflicting Job or Plugin overrides fail before Harbor starts. Even for `openai-codex`, the Candidate receives only the short-lived Broker capability—never the Host OAuth file or an upstream API key.
+
+When Settings opens, the Host performs a bounded npm registry check and caches successful results. An available release is shown with its exact installer command and release link. The browser never installs, rewrites a DSH profile, or restarts DSH; registry failures are non-blocking.
+
 `harbor_eval_result` defaults to the stable Summary. Use `view=job`, `view=dataset`, `view=progress`, `view=trial` plus a returned `trialId`, or `view=governance` to inspect sanitized instructions, generated output, evidence, and evaluator source without coupling the Agent to artifact file paths.
 
 ## What setup writes

package/index.js

@@ -6,7 +6,10 @@ import { fileURLToPath } from 'node:url'
 
 import { loadBundledSkill } from './lib/official-skill.js'
 import { CandidateModelRuntime } from './lib/model-runtime.js'
+import { RUNTIME_POLICY } from './lib/runtime-identity.js'
+import { SessionDiagnosticService } from './lib/session-diagnostic.js'
 import { EvolutionService } from './lib/service.js'
+import { runHistoricalEvaluation } from './lib/evolution.js'
 import { installDashboardWeb } from './lib/web.js'
 
 export const name = 'harbor-evolution'
@@ -26,9 +29,10 @@ export const Config = Schema.object({
   jobsDir: Schema.string().default('jobs'),
   harborBin: Schema.string().default(''),
   harborDshBin: Schema.string().default(''),
-  dshVersion: Schema.string().default('0.1.0-rc.6'),
   agentImportPath: Schema.string().default('harbor_dsh_evolution.agent:DshCandidateAgent'),
   pluginImportPath: Schema.string().default('dsh-evolution'),
+  historicalAgentImportPath: Schema.string().default('harbor_dsh_evolution.session_agent:SessionObservationAgent'),
+  historicalPluginImportPath: Schema.string().default('dsh-historical-evaluation'),
   pythonPath: Schema.string().default(''),
   timeoutMs: Schema.number().default(1800000),
   candidateProvider: Schema.string().default(''),
@@ -38,6 +42,8 @@ export const Config = Schema.object({
   modelBrokerAdvertisedHost: Schema.string().default('host.docker.internal'),
   modelBrokerMaxRequests: Schema.number().min(1).default(1000),
   modelBrokerMaxRequestBytes: Schema.number().min(1024).default(33554432),
+  sessionMaxReads: Schema.number().min(1).default(100),
+  sessionReadConcurrency: Schema.number().min(1).default(4),
 })
 
 function jsonTool(definition, execute) {
@@ -61,9 +67,16 @@ function toolProjectRoot(exec) {
   return path.resolve(cwd)
 }
 
+export function synchronizeWorkbenchProjectRoot(service, exec) {
+  const projectRoot = toolProjectRoot(exec)
+  service.activateProjectRoot(projectRoot, 'agent-session')
+  return projectRoot
+}
+
 export function apply(ctx, config) {
   const resolved = {
     ...config,
+    runtimePolicy: RUNTIME_POLICY,
     projectRoot: path.resolve(config.projectRoot),
     harborBin: config.harborBin || process.env.HARBOR_BIN || checkoutExecutable('harbor'),
     harborDshBin: config.harborDshBin || process.env.HARBOR_DSH_BIN || checkoutExecutable('harbor-dsh'),
@@ -74,12 +87,18 @@ export function apply(ctx, config) {
     ),
   }
   const modelRuntime = new CandidateModelRuntime(ctx, resolved)
-  const metadata = { pluginVersion: packageJson.version }
+  const metadata = { pluginVersion: packageJson.version, projectRootSource: 'configured' }
   const service = new EvolutionService(resolved, metadata, modelRuntime)
-  const serviceForTool = exec => new EvolutionService({
-    ...resolved,
-    projectRoot: toolProjectRoot(exec),
-  }, metadata, modelRuntime)
+  const sessionDiagnostic = new SessionDiagnosticService({
+    ctx,
+    config: resolved,
+    modelRuntime,
+    runHistoricalEvaluation,
+  })
+  const serviceForTool = exec => {
+    const projectRoot = synchronizeWorkbenchProjectRoot(service, exec)
+    return new EvolutionService({ ...resolved, projectRoot }, metadata, modelRuntime)
+  }
 
   ctx.skills.register(loadBundledSkill())
   installDashboardWeb(ctx, service)
@@ -94,11 +113,18 @@ export function apply(ctx, config) {
     },
   }, (args, exec) => serviceForTool(exec).snapshot(args)))
 
+  ctx.tools.register(jsonTool({
+    name: 'harbor_model_binding',
+    description: 'Freeze the current DSH default provider, model, and reasoning identity into a non-secret model-binding.json draft. Runtime access still uses the short-lived Host Model Broker capability.',
+    parameters: {},
+  }, (_args, exec) => serviceForTool(exec).modelBinding()))
+
   ctx.tools.register(jsonTool({
     name: 'harbor_evolution_init',
     description: 'Compile an accepted Dataset, Generator, Evaluator/criteria, and Optimizer onboarding card into a strict, non-overwriting Evaluation Stack project. Detailed identity fields are internal tool inputs, not a user questionnaire.',
     parameters: {
       datasetPath: { type: 'string', required: true },
+      workspaceSubdir: { type: 'string', description: 'Optional namespace under the current project root. Defaults to the project root; use it to host multiple independent Harbor projects.' },
       stackId: { type: 'string', required: true },
       stackVersion: { type: 'string', required: true },
       datasetId: { type: 'string', required: true },
@@ -131,6 +157,44 @@ export function apply(ctx, config) {
     },
   }, (args, exec) => serviceForTool(exec).doctor(args)))
 
+  ctx.tools.register(jsonTool({
+    name: 'harbor_quick_diagnostic_init',
+    description: 'Create a non-overwriting Harbor 1.4 wiring diagnostic with one Query, a minimal Host-model Candidate, a runnable Task, and an explicit non-promotion Evaluator. The supplied Rubric is recorded as a draft but is not treated as executed.',
+    parameters: {
+      query: { type: 'string', required: true },
+      rubric: { type: 'string', required: true },
+      workspaceSubdir: { type: 'string', description: 'Defaults to harbor-diagnostic under the current Agent session directory.' },
+    },
+  }, (args, exec) => serviceForTool(exec).quickDiagnostic(args)))
+
+  ctx.tools.register(jsonTool({
+    name: 'harbor_session_diagnostic_preview',
+    description: 'Preview up to 10 recently active, completed top-level DSH Sessions in the exact current workspace as an observe-existing Historical Generation Job. Returns only safe metadata and an owner-bound 15-minute selection token; it never exposes raw Session ids or transcript/tool payloads.',
+    parameters: {
+      limit: { type: 'number', description: 'Number of eligible recent Sessions, from 1 to 10. Defaults to 10.' },
+      createdAfter: { type: 'string', description: 'Optional ISO-8601 lower bound on Session creation time. Use it to narrow an exact scan when the workspace exceeds the configured read budget.' },
+      includeFeedback: { type: 'boolean', description: 'Include only feedback counts in Preview and redacted feedback in the frozen Batch. Defaults to true.' },
+      evaluatorProvider: { type: 'string', description: 'Optional Judge provider; supply together with evaluatorModel. Defaults to the calling DSH Agent model and is frozen into the confirmation token.' },
+      evaluatorModel: { type: 'string' },
+      evaluatorReasoningEffort: { type: 'string', description: 'Optional Judge reasoning effort; requires explicit evaluatorProvider and evaluatorModel.' },
+    },
+  }, (args, exec) => {
+    synchronizeWorkbenchProjectRoot(service, exec)
+    return sessionDiagnostic.preview(args, exec)
+  }))
+
+  ctx.tools.register(jsonTool({
+    name: 'harbor_session_diagnostic_run',
+    description: 'Consume a confirmed Session selection token, revalidate every immutable source boundary, write a private redacted Historical Generation Batch, materialize one Harbor Trial per Session, and run the non-promotion Historical Job.',
+    parameters: {
+      selectionToken: { type: 'string', required: true },
+      jobName: { type: 'string' },
+    },
+  }, (args, exec) => {
+    synchronizeWorkbenchProjectRoot(service, exec)
+    return sessionDiagnostic.run(args, exec)
+  }))
+
   ctx.tools.register(jsonTool({
     name: 'harbor_dataset_validate',
     description: 'Validate dataset-manifest.json, task uniqueness, instructions, paths, sensitive metadata, and the immutable source digest.',
@@ -211,6 +275,7 @@ export function apply(ctx, config) {
     description: 'Create a non-overwriting Ground Truth draft for evaluator meta-evaluation. GT may be human, programmatic, consensus, model, or external, but must have explicit provenance and remain independent of the Candidate evaluator.',
     parameters: {
       outputPath: { type: 'string', description: 'Defaults to .harbor/ground-truth.json' },
+      evaluationRoot: { type: 'string', description: 'Optional evaluation workspace root used to register custom Ground Truth paths.' },
       groundTruthId: { type: 'string', required: true },
       version: { type: 'string', required: true },
       sourceKind: { type: 'string', required: true, description: 'human, programmatic, consensus, model, or external' },
@@ -227,6 +292,7 @@ export function apply(ctx, config) {
       groundTruthPath: { type: 'string', description: 'Defaults to .harbor/ground-truth.json' },
       observationsPath: { type: 'string', required: true },
       outputPath: { type: 'string', description: 'Defaults to .harbor/meta-evaluation-report.json' },
+      evaluationRoot: { type: 'string', description: 'Optional evaluation workspace root used to register custom report paths.' },
     },
   }, (args, exec) => serviceForTool(exec).evaluatorMetaEvaluate(args)))
 

package/lib/candidate.js

@@ -2,7 +2,10 @@ import { createHash } from 'node:crypto'
 import { mkdir, readFile, readdir, stat, writeFile } from 'node:fs/promises'
 import path from 'node:path'
 
+import { CANDIDATE_ACP_PACKAGE, DSH_RUNTIME_VERSION, RUNTIME_POLICY } from './runtime-identity.js'
+
 export const MANIFEST_NAME = 'candidate-manifest.json'
+export const MODEL_BINDING_NAME = 'model-binding.json'
 const DIGEST_PREFIX = Buffer.from('harbor-dsh-candidate-v1\0')
 const EXCLUDED_DIRS = new Set(['.git', 'node_modules', '__pycache__', '.harbor-runtime'])
 const EXCLUDED_FILES = new Set([MANIFEST_NAME, '.DS_Store'])
@@ -16,6 +19,7 @@ const CREDENTIAL_FILES = new Set([
   'id_rsa',
   'id_ed25519',
 ])
+const MODEL_BINDING_KEYS = new Set(['schema_version', 'source', 'provider', 'model', 'reasoning_effort'])
 
 async function walk(root, current = root) {
   const entries = await readdir(current, { withFileTypes: true })
@@ -33,6 +37,45 @@ async function walk(root, current = root) {
   return files.sort((a, b) => Buffer.compare(Buffer.from(a.relative), Buffer.from(b.relative)))
 }
 
+export async function loadModelBinding(candidateDir) {
+  const pathname = path.join(path.resolve(candidateDir), MODEL_BINDING_NAME)
+  let value
+  try {
+    value = JSON.parse(await readFile(pathname, 'utf8'))
+  } catch (error) {
+    if (error.code === 'ENOENT') return undefined
+    if (error instanceof SyntaxError) throw new Error(`${MODEL_BINDING_NAME} is not valid JSON`)
+    throw error
+  }
+  if (value === null || typeof value !== 'object' || Array.isArray(value)) {
+    throw new Error(`${MODEL_BINDING_NAME} must be an object`)
+  }
+  const unknown = Object.keys(value).filter(key => !MODEL_BINDING_KEYS.has(key)).sort()
+  if (unknown.length) {
+    throw new Error(`${MODEL_BINDING_NAME} contains unsupported or secret-bearing fields: ${unknown.join(', ')}`)
+  }
+  if (value.schema_version !== 1) throw new Error(`${MODEL_BINDING_NAME} requires schema_version=1`)
+  const source = typeof value.source === 'string' ? value.source.trim() : ''
+  const provider = typeof value.provider === 'string' ? value.provider.trim() : ''
+  const model = typeof value.model === 'string' ? value.model.trim() : ''
+  if (!source || !provider || !model) {
+    throw new Error(`${MODEL_BINDING_NAME} requires non-empty source, provider, and model`)
+  }
+  const reasoningEffort = value.reasoning_effort === undefined
+    ? undefined
+    : typeof value.reasoning_effort === 'string' ? value.reasoning_effort.trim() : ''
+  if (value.reasoning_effort !== undefined && !reasoningEffort) {
+    throw new Error(`${MODEL_BINDING_NAME} reasoning_effort must be a non-empty string when present`)
+  }
+  return {
+    schema_version: 1,
+    source,
+    provider,
+    model,
+    ...(reasoningEffort ? { reasoning_effort: reasoningEffort } : {}),
+  }
+}
+
 export async function computeCandidate(candidateDir) {
   const root = path.resolve(candidateDir)
   if (!(await stat(root)).isDirectory()) throw new Error(`Candidate path is not a directory: ${root}`)
@@ -83,6 +126,7 @@ async function validateCandidateContract(root) {
   if (credentialPaths.length > 0) {
     throw new Error(`Candidate contains credential-bearing files: ${credentialPaths.join(', ')}; inject credentials at runtime instead`)
   }
+  await loadModelBinding(root)
 }
 
 export async function snapshotCandidate(candidateDir, options = {}) {
@@ -96,11 +140,18 @@ export async function snapshotCandidate(candidateDir, options = {}) {
   }
   const candidateId = options.candidateId ?? packageJson.name
   const version = options.version ?? packageJson.version
-  const runtimeVersion = options.runtimeVersion ?? '0.1.0-rc.6'
-  if (!candidateId || !version || !runtimeVersion) {
-    throw new Error('Candidate id, version, and runtime version must not be empty; set package.json name/version or pass explicit values')
+  if (!candidateId || !version) {
+    throw new Error('Candidate id and version must not be empty; set package.json name/version or pass explicit values')
   }
   const computed = await computeCandidate(root)
+  const metadata = { ...(options.metadata ?? {}) }
+  const modelBinding = await loadModelBinding(root)
+  if (modelBinding) {
+    if (metadata.model_binding !== undefined && JSON.stringify(metadata.model_binding) !== JSON.stringify(modelBinding)) {
+      throw new Error('Candidate metadata model_binding must match model-binding.json')
+    }
+    metadata.model_binding = modelBinding
+  }
   const manifest = {
     schema_version: 1,
     candidate_id: String(candidateId),
@@ -109,11 +160,13 @@ export async function snapshotCandidate(candidateDir, options = {}) {
     created_at: new Date().toISOString(),
     runtime: {
       kind: 'deepseek-harness',
-      version: String(runtimeVersion),
+      policy: RUNTIME_POLICY,
+      version: DSH_RUNTIME_VERSION,
+      package: CANDIDATE_ACP_PACKAGE,
       transport: 'acp',
     },
     files: computed.files,
-    metadata: options.metadata ?? {},
+    metadata,
   }
   await mkdir(root, { recursive: true })
   await writeFile(path.join(root, MANIFEST_NAME), `${JSON.stringify(manifest, null, 2)}\n`)