dsh-harbor-evolution 0.8.2 → 0.9.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +28 -6
- package/index.js +120 -19
- package/lib/action-drafts.js +443 -0
- package/lib/bounded-process.js +190 -0
- package/lib/candidate-runtime.js +160 -0
- package/lib/candidate.js +7 -11
- package/lib/client.js +4110 -292
- package/lib/composer-context.js +56 -0
- package/lib/credential-redaction.js +155 -0
- package/lib/dashboard.js +417 -98
- package/lib/diagnostic-observation.js +175 -0
- package/lib/diagnostic-runner.js +206 -0
- package/lib/evaluator-saves.js +129 -0
- package/lib/evolution.js +126 -32
- package/lib/historical-run-lock.js +102 -0
- package/lib/historical-web.js +52 -16
- package/lib/interaction-objects.js +56 -0
- package/lib/model-runtime.js +48 -3
- package/lib/process.js +27 -1
- package/lib/runtime-identity.js +0 -1
- package/lib/service.js +1427 -28
- package/lib/session-diagnostic.js +0 -1
- package/lib/session-redaction.js +17 -31
- package/lib/session-selection.js +5 -3
- package/lib/trial-selection.js +46 -0
- package/lib/ui-context.js +518 -0
- package/lib/web.js +32 -6
- package/lib/workbench-health.js +27 -0
- package/package.json +4 -4
- package/skills/evolve-agent-with-harbor/SKILL.md +33 -4
package/README.md
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
Installable DeepSeek Harness Plugin + Skill for running stable Harbor evaluation and controlled Agent evolution loops, with a native DSH Web dashboard.
|
|
4
4
|
|
|
5
|
-
The package gives DSH
|
|
5
|
+
The package gives DSH nineteen strict Harbor tools, dedicated Tool cards, an object-first Evaluation Workbench, a same-session Copilot Dock, an installation Doctor, and the model- and user-invocable `evolve-agent-with-harbor` Skill. The Skill starts with four user-facing concepts—Dataset (what to test), Generator (who answers), Evaluator plus criteria (what good means), and Optimizer (who improves it)—then compiles accepted choices into the strict Evaluation Stack. When no Dataset is supplied, it can instead preview recent completed DSH Sessions and evaluate each immutable Session as one Historical Trial without rerunning a Candidate. A DSH Generator may explicitly pin the current default model as a non-secret Candidate identity while retaining the per-Job Host Broker credential boundary. The Plugin validates Dataset identity, checks Trial Lifecycle and Score Validity, governs independent Ground Truth meta-evaluation, diagnoses evidence provenance, limits each iteration to one controlled Candidate change, and invokes the Promotion Gate only as an explicit action.
|
|
6
6
|
|
|
7
7
|
## Install
|
|
8
8
|
|
|
@@ -14,8 +14,8 @@ npx --yes dsh-harbor-evolution@latest setup --project-root "$PWD"
|
|
|
14
14
|
|
|
15
15
|
The setup command installs both required runtimes:
|
|
16
16
|
|
|
17
|
-
- `harbor-dsh-evolution==0.
|
|
18
|
-
- `dsh-harbor-evolution@0.
|
|
17
|
+
- `harbor-dsh-evolution==0.9.2` in a managed Python environment.
|
|
18
|
+
- `dsh-harbor-evolution@0.9.2` in the selected DSH profile.
|
|
19
19
|
|
|
20
20
|
It then stores the absolute Harbor executable paths and a fallback `projectRoot` in the profile's `harbor-evolution` block and verifies the integration. Agent Tool calls always use the calling session's absolute working directory as their project root; the configured value remains the Web Workbench and non-Agent fallback. Existing unrelated profile entries are preserved, and rerunning setup updates the same block.
|
|
21
21
|
|
|
@@ -46,9 +46,12 @@ The Plugin registers:
|
|
|
46
46
|
- `harbor_session_diagnostic_preview`
|
|
47
47
|
- `harbor_session_diagnostic_run`
|
|
48
48
|
- `harbor_dataset_validate`
|
|
49
|
-
- `harbor_context_preview`
|
|
49
|
+
- `harbor_context_preview` (refreshes `candidate-manifest.json` under one-shot approval before returning the preview)
|
|
50
50
|
- `harbor_eval_run`
|
|
51
51
|
- `harbor_eval_result`
|
|
52
|
+
- `harbor_resolve_page_context`
|
|
53
|
+
- `harbor_get_evidence`
|
|
54
|
+
- `harbor_propose_action` (proposal only; never confirms or executes a mutation)
|
|
52
55
|
- `harbor_evaluator_inspect`
|
|
53
56
|
- `harbor_evaluator_update`
|
|
54
57
|
- `harbor_ground_truth_init`
|
|
@@ -57,14 +60,29 @@ The Plugin registers:
|
|
|
57
60
|
|
|
58
61
|
In the `web` profile, the same package also registers:
|
|
59
62
|
|
|
60
|
-
- a localized
|
|
63
|
+
- a localized object-first Workbench (Summary, Trials, Pipeline, Optimization, Compare/Gate, Evaluator/Rubric, Artifacts, Audit) that directly exposes fixed experiment identities, Agent-visible Dataset queries/instructions, safe business-artifact previews, Ground Truth meta-evaluation, paginated per-Trial evidence and recommendations, Population validity/coverage, controlled optimization hypotheses, and Baseline/Gate deltas; raw JSON remains in the audit drawer;
|
|
64
|
+
- a Composer Context Capsule that freezes a Job, Trial, Criterion, or Evidence selection only after an explicit `Ask AI` / `Ask about this` action or `@harbor` selection; ordinary sends never inherit the visible Harbor page, and the one-shot reference clears after its turn is sent;
|
|
65
|
+
- a same-session Harbor Copilot Dock that renders the real Chat Session's running state, Tool progress, final answer, failure, and Stop action without copying conversation history; a typed `harbor.navigate` result performs only an allowlisted, read-only Harbor location change when its object, page-Session, and generation preconditions still match, while Back restores the prior workspace, page, stage, Trial filters/sort/focus, Compare Baseline, and scroll position;
|
|
61
66
|
- a first-class `Evaluate recent Sessions` action that previews up to ten safe Session records, shows the frozen Evaluator/Judge and diagnostic boundaries, requires explicit confirmation, runs in the background, and opens the completed Job;
|
|
62
67
|
- descriptor-authorized Evaluator/Rubric source editing for `script` and `llm-as-judge` implementations, with optimistic concurrency and mandatory new identities;
|
|
63
68
|
- a `harbor-dsh-evaluator/v1` interface shared by deterministic scripts and LLM-as-Judge implementations;
|
|
64
69
|
- compact result cards for all Harbor Tool calls;
|
|
70
|
+
- explicit local-object selection and frozen Trial-set selectors (fixed IDs/revisions or a query snapshot, at most 1000 members), independently removable context chips, and source fragment Ask;
|
|
71
|
+
- AI proposal cards with deterministic Preflight, explicit review, idempotent confirmation and append-only local operation journals; Candidate/Gate/handoff output is a saved draft, not an applied resource change. Evaluator source proposals can be opened in the reviewed version editor. Selected Compare executes a read only;
|
|
65
72
|
- a `Harbor Evolution` Settings section that checks the configured project, Evaluation Stack, Jobs directory, and CLI paths, supports process-local `projectRoot` reload, and checks npm for a newer formal release without silently installing it.
|
|
66
73
|
|
|
67
|
-
|
|
74
|
+
### Start with an object, not a command
|
|
75
|
+
|
|
76
|
+
1. Open an evaluation result. Select a task, score, evidence item, or saved source fragment.
|
|
77
|
+
2. Choose **Ask AI** or a suggested question. Harbor prepares a visible reference in the existing Composer; review, add your question, and send.
|
|
78
|
+
3. Continue chatting in the same Session. The Copilot shows ordinary follow-ups and recent discussion history. An ordinary answer does not inherit the old answer's evidence-validity badge. Reattach the original object when you need fresh evidence or a modification.
|
|
79
|
+
4. For scoring rules, select saved lines and choose **Suggest a change**. **Review and edit** opens the matching file directly. The AI may populate an unchanged editor, but never replaces your manual edits. Review the diff and explicitly save to create new identities; this does not run an evaluation or Gate.
|
|
80
|
+
|
|
81
|
+
Unsaved source edits are isolated by Session, workspace, Job, and file and retained in this browser tab's `sessionStorage`. File/view switches and refresh can recover them; closing the tab may discard them. Storage failures are shown, with an in-memory fallback and a leave-page warning for unpersisted edits. Source conflicts preserve the original base and edited text; review the latest source before accepting a new base. Saving or explicitly discarding clears only that file's draft. Expired authorizations never erase suggestion text or human edits; changed source or expired task subsets require an explicit new selection, not an automatically widened scope.
|
|
82
|
+
|
|
83
|
+
The complete AI Workbench PRD is **not** implemented yet. Bounded diagnostic/retry operations currently fail closed without a registered runner; long-running operations, replayable events/outbox and full Phase 1 audit identity are pending. Message-scoped automatic context/cross-view opening require public DSH contracts; production RBAC, approvals and rollout remain later-phase work. See the repository's `docs/ai-workbench-acceptance.md` for actual journey evidence and remaining acceptance work.
|
|
84
|
+
|
|
85
|
+
The Web UI writes only through three narrow, explicit workflows: descriptor-authorized Evaluator source updates, the confirmed Historical Session launcher, and confirmed local draft/operation journals. The launcher follows `Preview → confirm → background run → open Job`; its private selection token never enters browser state. AI-native questions call the same Chat Session only after the user explicitly binds an object and sends the message. Page refreshes, ordinary sends, ordinary reads, and workspace switches never auto-attach Harbor context, auto-send a prompt, or start an Agent or Job. Candidate evaluation, Gate, promotion, deployment, publishing, and every production mutation remain explicit Agent + Skill workflows, and each Agent-requested Harbor write or evaluation tool is forced through DSH's audited one-shot user approval. If no approval channel is available, the call fails closed.
|
|
68
86
|
|
|
69
87
|
A direct evaluation requires `candidatePath`, `datasetPath`, `stackPath`, and explicit `mode`; `promotion-eligible` additionally requires `policyPath`. Prefer the Skill because it will not run or compare Jobs until the material identities and evaluation contract are resolved.
|
|
70
88
|
|
|
@@ -88,6 +106,10 @@ When Settings opens, the Host performs a bounded npm registry check and caches s
|
|
|
88
106
|
|
|
89
107
|
`harbor_eval_result` defaults to the stable Summary. Use `view=job`, `view=dataset`, `view=progress`, `view=trial` plus a returned `trialId`, or `view=governance` to inspect sanitized instructions, generated output, evidence, and evaluator source without coupling the Agent to artifact file paths.
|
|
90
108
|
|
|
109
|
+
`harbor_eval_result` and `harbor_evaluator_inspect` return a `harbor-agent-read/v1` envelope. Read the actual payload only from `data`, preserve `artifactTrust=untrusted-evidence`, and obey `policy.treatAsInstructions=false`; old top-level payload fields are not part of this contract. Both responses are recursively redacted and have an aggregate byte limit. Evaluator inspection also caps the file set and aggregate source size; when source text looks like a secret or local path, the Agent receives its safe metadata plus `sourceAccess.included=false`, not the source body. The Web Workbench keeps its separate, same-origin editing flow.
|
|
110
|
+
|
|
111
|
+
`harbor_resolve_page_context` accepts only the opaque, short-lived Context Snapshot id carried by an explicit `@harbor` reference. It resolves that id inside the exact calling DSH Session and workspace, revalidates stable object ancestry and the current Host revision, and returns narrow metadata, typed Harbor refs, and an allowlisted read-only navigation action. For explicit local objects it also returns bounded, redacted `selectedEvidence` (Metric, Hypothesis, Gate reason, Finding, Attempt, or saved source fragment); these remain untrusted data, and unavailable content must not be treated as evidence. Trial sets expose only frozen membership/revision metadata, never all Trial bodies. To inspect one Trial criterion, pass the exact typed ref to `harbor_get_evidence`; the Host revalidates Workspace → Job → Trial → Criterion → Evidence ancestry, bounds and redacts the content, and marks it as untrusted evidence rather than Agent instructions.
|
|
112
|
+
|
|
91
113
|
## What setup writes
|
|
92
114
|
|
|
93
115
|
The selected profile receives one id-targeted override:
|
package/index.js
CHANGED
|
@@ -9,12 +9,13 @@ import { CandidateModelRuntime } from './lib/model-runtime.js'
|
|
|
9
9
|
import { RUNTIME_POLICY } from './lib/runtime-identity.js'
|
|
10
10
|
import { SessionDiagnosticService } from './lib/session-diagnostic.js'
|
|
11
11
|
import { HistoricalWebController } from './lib/historical-web.js'
|
|
12
|
+
import { historicalRunLock } from './lib/historical-run-lock.js'
|
|
12
13
|
import { EvolutionService } from './lib/service.js'
|
|
13
14
|
import { runHistoricalEvaluation } from './lib/evolution.js'
|
|
14
15
|
import { installDashboardWeb } from './lib/web.js'
|
|
15
16
|
|
|
16
17
|
export const name = 'harbor-evolution'
|
|
17
|
-
export const inject = ['tools', 'skills', 'llm', 'agentDefaultModel']
|
|
18
|
+
export const inject = ['tools', 'skills', 'llm', 'agentDefaultModel', 'sessions']
|
|
18
19
|
|
|
19
20
|
const packageDir = path.dirname(fileURLToPath(import.meta.url))
|
|
20
21
|
const checkoutPythonPackage = path.resolve(packageDir, '../harbor-plugin')
|
|
@@ -42,6 +43,7 @@ export const Config = Schema.object({
|
|
|
42
43
|
modelBrokerBindHost: Schema.string().default('127.0.0.1'),
|
|
43
44
|
modelBrokerAdvertisedHost: Schema.string().default('host.docker.internal'),
|
|
44
45
|
modelBrokerMaxRequests: Schema.number().min(1).default(1000),
|
|
46
|
+
modelBrokerMaxResponseBytes: Schema.number().min(1).default(4194304),
|
|
45
47
|
modelBrokerMaxRequestBytes: Schema.number().min(1024).default(33554432),
|
|
46
48
|
sessionMaxReads: Schema.number().min(1).default(100),
|
|
47
49
|
sessionReadConcurrency: Schema.number().min(1).default(4),
|
|
@@ -60,6 +62,17 @@ function jsonTool(definition, execute) {
|
|
|
60
62
|
})
|
|
61
63
|
}
|
|
62
64
|
|
|
65
|
+
function objectTool(definition, execute) {
|
|
66
|
+
return defineTool({
|
|
67
|
+
...definition,
|
|
68
|
+
output: {
|
|
69
|
+
schema: { type: 'object', additionalProperties: true },
|
|
70
|
+
render: (_args, value) => [{ type: 'text', text: JSON.stringify(value, null, 2) }],
|
|
71
|
+
},
|
|
72
|
+
execute,
|
|
73
|
+
})
|
|
74
|
+
}
|
|
75
|
+
|
|
63
76
|
function toolProjectRoot(exec) {
|
|
64
77
|
const cwd = exec?.agent?.session?.header?.cwd
|
|
65
78
|
if (typeof cwd !== 'string' || !path.isAbsolute(cwd)) {
|
|
@@ -68,9 +81,17 @@ function toolProjectRoot(exec) {
|
|
|
68
81
|
return path.resolve(cwd)
|
|
69
82
|
}
|
|
70
83
|
|
|
84
|
+
function toolSessionId(exec) {
|
|
85
|
+
const sessionId = exec?.agent?.session?.header?.id
|
|
86
|
+
if (typeof sessionId !== 'string' || !sessionId) {
|
|
87
|
+
throw new Error('Harbor tools require an Agent session identity')
|
|
88
|
+
}
|
|
89
|
+
return sessionId
|
|
90
|
+
}
|
|
91
|
+
|
|
71
92
|
export function synchronizeWorkbenchProjectRoot(service, exec) {
|
|
72
93
|
const projectRoot = toolProjectRoot(exec)
|
|
73
|
-
service.activateProjectRoot(projectRoot, 'agent-session')
|
|
94
|
+
service.activateProjectRoot(projectRoot, 'agent-session', toolSessionId(exec))
|
|
74
95
|
return projectRoot
|
|
75
96
|
}
|
|
76
97
|
|
|
@@ -88,15 +109,53 @@ export function apply(ctx, config) {
|
|
|
88
109
|
),
|
|
89
110
|
}
|
|
90
111
|
const modelRuntime = new CandidateModelRuntime(ctx, resolved)
|
|
91
|
-
const metadata = {
|
|
112
|
+
const metadata = {
|
|
113
|
+
pluginVersion: packageJson.version,
|
|
114
|
+
projectRootSource: 'configured',
|
|
115
|
+
sessionProjectRoot: sessionId => {
|
|
116
|
+
const cwd = ctx.sessions?.get?.(sessionId)?.header?.cwd
|
|
117
|
+
return typeof cwd === 'string' && path.isAbsolute(cwd) ? path.resolve(cwd) : undefined
|
|
118
|
+
},
|
|
119
|
+
}
|
|
92
120
|
const service = new EvolutionService(resolved, metadata, modelRuntime)
|
|
121
|
+
// Cordis hot reload/unload must not orphan accepted background diagnostics.
|
|
122
|
+
// Hard process termination still leaves the durable claim for manual recovery.
|
|
123
|
+
if (typeof ctx.effect === 'function') ctx.effect(() => () => service.actionDrafts.dispose())
|
|
124
|
+
const approvalChannelAvailable = typeof ctx.on === 'function'
|
|
125
|
+
if (!approvalChannelAvailable) {
|
|
126
|
+
const error = new Error('HARBOR_APPROVAL_HOOK_UNAVAILABLE: Harbor requires the DSH tools/pre-execute approval seam before registering Agent tools.')
|
|
127
|
+
error.code = 'HARBOR_APPROVAL_HOOK_UNAVAILABLE'
|
|
128
|
+
throw error
|
|
129
|
+
}
|
|
130
|
+
const mutatingAgentTools = new Set()
|
|
131
|
+
const mutatingJsonTool = (definition, execute) => {
|
|
132
|
+
mutatingAgentTools.add(definition.name)
|
|
133
|
+
return jsonTool(definition, execute)
|
|
134
|
+
}
|
|
135
|
+
|
|
136
|
+
// Artifact text is untrusted. Any Agent-requested Harbor write or evaluation
|
|
137
|
+
// must cross DSH's audited, one-shot user approval seam. With approval disabled
|
|
138
|
+
// or unavailable the tool runtime fails closed; read/ask/navigate tools remain
|
|
139
|
+
// approval-free.
|
|
140
|
+
ctx.on('tools/pre-execute', async (exec, next) => {
|
|
141
|
+
const downstream = await next()
|
|
142
|
+
if (downstream?.kind !== 'allow' || !mutatingAgentTools.has(exec?.name)) return downstream
|
|
143
|
+
return {
|
|
144
|
+
kind: 'ask',
|
|
145
|
+
reason: `Harbor tool ${exec.name} can write artifacts or start evaluation work and requires explicit one-shot approval.`,
|
|
146
|
+
}
|
|
147
|
+
}, { prepend: true })
|
|
93
148
|
const sessionDiagnostic = new SessionDiagnosticService({
|
|
94
149
|
ctx,
|
|
95
150
|
config: resolved,
|
|
96
151
|
modelRuntime,
|
|
97
152
|
runHistoricalEvaluation,
|
|
98
153
|
})
|
|
99
|
-
const historicalWeb = new HistoricalWebController({
|
|
154
|
+
const historicalWeb = new HistoricalWebController({
|
|
155
|
+
service,
|
|
156
|
+
sessionDiagnostic,
|
|
157
|
+
runLock: historicalRunLock,
|
|
158
|
+
})
|
|
100
159
|
const serviceForTool = exec => {
|
|
101
160
|
const projectRoot = synchronizeWorkbenchProjectRoot(service, exec)
|
|
102
161
|
return new EvolutionService({ ...resolved, projectRoot }, metadata, modelRuntime)
|
|
@@ -105,7 +164,7 @@ export function apply(ctx, config) {
|
|
|
105
164
|
ctx.skills.register(loadBundledSkill())
|
|
106
165
|
installDashboardWeb(ctx, service, historicalWeb)
|
|
107
166
|
|
|
108
|
-
ctx.tools.register(
|
|
167
|
+
ctx.tools.register(mutatingJsonTool({
|
|
109
168
|
name: 'harbor_candidate_snapshot',
|
|
110
169
|
description: 'Freeze a DeepSeek Harness Cordis composition as an immutable Candidate manifest. Candidate id and version default to package.json.',
|
|
111
170
|
parameters: {
|
|
@@ -121,7 +180,7 @@ export function apply(ctx, config) {
|
|
|
121
180
|
parameters: {},
|
|
122
181
|
}, (_args, exec) => serviceForTool(exec).modelBinding()))
|
|
123
182
|
|
|
124
|
-
ctx.tools.register(
|
|
183
|
+
ctx.tools.register(mutatingJsonTool({
|
|
125
184
|
name: 'harbor_evolution_init',
|
|
126
185
|
description: 'Compile an accepted Dataset, Generator, Evaluator/criteria, and Optimizer onboarding card into a strict, non-overwriting Evaluation Stack project. Detailed identity fields are internal tool inputs, not a user questionnaire.',
|
|
127
186
|
parameters: {
|
|
@@ -159,7 +218,7 @@ export function apply(ctx, config) {
|
|
|
159
218
|
},
|
|
160
219
|
}, (args, exec) => serviceForTool(exec).doctor(args)))
|
|
161
220
|
|
|
162
|
-
ctx.tools.register(
|
|
221
|
+
ctx.tools.register(mutatingJsonTool({
|
|
163
222
|
name: 'harbor_quick_diagnostic_init',
|
|
164
223
|
description: 'Create a non-overwriting Harbor 1.4 wiring diagnostic with one Query, a minimal Host-model Candidate, a runnable Task, and an explicit non-promotion Evaluator. The supplied Rubric is recorded as a draft but is not treated as executed.',
|
|
165
224
|
parameters: {
|
|
@@ -185,7 +244,7 @@ export function apply(ctx, config) {
|
|
|
185
244
|
return sessionDiagnostic.preview(args, exec)
|
|
186
245
|
}))
|
|
187
246
|
|
|
188
|
-
ctx.tools.register(
|
|
247
|
+
ctx.tools.register(mutatingJsonTool({
|
|
189
248
|
name: 'harbor_session_diagnostic_run',
|
|
190
249
|
description: 'Consume a confirmed Session selection token, revalidate every immutable source boundary, write a private redacted Historical Generation Batch, materialize one Harbor Trial per Session, and run the non-promotion Historical Job.',
|
|
191
250
|
parameters: {
|
|
@@ -193,8 +252,12 @@ export function apply(ctx, config) {
|
|
|
193
252
|
jobName: { type: 'string' },
|
|
194
253
|
},
|
|
195
254
|
}, (args, exec) => {
|
|
196
|
-
synchronizeWorkbenchProjectRoot(service, exec)
|
|
197
|
-
return
|
|
255
|
+
const projectRoot = synchronizeWorkbenchProjectRoot(service, exec)
|
|
256
|
+
return historicalRunLock.runExclusive(
|
|
257
|
+
{ projectRoot, jobsDir: resolved.jobsDir },
|
|
258
|
+
() => sessionDiagnostic.run(args, exec),
|
|
259
|
+
{ channel: 'agent' },
|
|
260
|
+
)
|
|
198
261
|
}))
|
|
199
262
|
|
|
200
263
|
ctx.tools.register(jsonTool({
|
|
@@ -205,9 +268,9 @@ export function apply(ctx, config) {
|
|
|
205
268
|
},
|
|
206
269
|
}, (args, exec) => serviceForTool(exec).validateDataset(args)))
|
|
207
270
|
|
|
208
|
-
ctx.tools.register(
|
|
271
|
+
ctx.tools.register(mutatingJsonTool({
|
|
209
272
|
name: 'harbor_context_preview',
|
|
210
|
-
description: '
|
|
273
|
+
description: 'Refresh the Candidate manifest, then preview Evaluation Context v2 and find comparable baselines before launching a Job. The manifest write requires one-shot approval.',
|
|
211
274
|
parameters: {
|
|
212
275
|
candidatePath: { type: 'string', required: true },
|
|
213
276
|
candidateId: { type: 'string' },
|
|
@@ -221,7 +284,7 @@ export function apply(ctx, config) {
|
|
|
221
284
|
},
|
|
222
285
|
}, (args, exec) => serviceForTool(exec).previewContext(args)))
|
|
223
286
|
|
|
224
|
-
ctx.tools.register(
|
|
287
|
+
ctx.tools.register(mutatingJsonTool({
|
|
225
288
|
name: 'harbor_eval_run',
|
|
226
289
|
description: 'Run a strict diagnostic or promotion-eligible Harbor Job bound to Candidate, Dataset Manifest, Evaluation Stack, and Context v2 identities.',
|
|
227
290
|
parameters: {
|
|
@@ -241,7 +304,7 @@ export function apply(ctx, config) {
|
|
|
241
304
|
|
|
242
305
|
ctx.tools.register(jsonTool({
|
|
243
306
|
name: 'harbor_eval_result',
|
|
244
|
-
description: 'Read a stable Job summary or
|
|
307
|
+
description: 'Read a stable Job summary or Workbench, Dataset instruction, Trial output/evidence, progress, or Evaluator governance view inside a bounded, recursively redacted, explicitly untrusted envelope. Invalid scores remain distinct from raw verifier rewards.',
|
|
245
308
|
parameters: {
|
|
246
309
|
jobPath: { type: 'string', required: true },
|
|
247
310
|
view: { type: 'string', description: 'summary (default), job, dataset, progress, trial, or governance' },
|
|
@@ -251,15 +314,53 @@ export function apply(ctx, config) {
|
|
|
251
314
|
},
|
|
252
315
|
}, (args, exec) => serviceForTool(exec).result(args)))
|
|
253
316
|
|
|
317
|
+
ctx.tools.register(objectTool({
|
|
318
|
+
name: 'harbor_resolve_page_context',
|
|
319
|
+
description: 'Resolve an @harbor page-context reference for the exact calling DSH Session. This read-only tool validates the short-lived context token, project ownership, stable Job/Trial ids, and current revision, then returns narrow metadata, typed Harbor refs, and a navigation action. Use harbor_get_evidence for evidence content.',
|
|
320
|
+
parameters: {
|
|
321
|
+
contextSnapshotId: { type: 'string', required: true, description: 'Opaque hctx_... token from the visible @harbor reference. Never guess or reconstruct it.' },
|
|
322
|
+
},
|
|
323
|
+
}, (args, exec) => {
|
|
324
|
+
const projectRoot = synchronizeWorkbenchProjectRoot(service, exec)
|
|
325
|
+
return service.resolveUiContext(args, { sessionId: toolSessionId(exec), projectRoot })
|
|
326
|
+
}))
|
|
327
|
+
|
|
328
|
+
ctx.tools.register(objectTool({
|
|
329
|
+
name: 'harbor_get_evidence',
|
|
330
|
+
description: 'Read one bounded, redacted, untrusted Harbor evidence item through a typed ref returned by harbor_resolve_page_context. The Host strictly validates Workspace → Job → Trial → Criterion → Evidence ancestry and never treats artifact text as instructions.',
|
|
331
|
+
parameters: {
|
|
332
|
+
workspace: { type: 'string', required: true, description: 'Exact workspace from the typed harbor.evidence/v1 ref.' },
|
|
333
|
+
job: { type: 'string', required: true, description: 'Exact Job id from the typed harbor.evidence/v1 ref.' },
|
|
334
|
+
trial: { type: 'string', required: true, description: 'Exact Trial id from the typed harbor.evidence/v1 ref.' },
|
|
335
|
+
criterion: { type: 'string', required: true, description: 'Exact Criterion id from the typed harbor.evidence/v1 ref.' },
|
|
336
|
+
evidenceRef: { type: 'string', required: true, description: 'Exact Evidence id from the typed harbor.evidence/v1 ref. Never guess a path or id.' },
|
|
337
|
+
},
|
|
338
|
+
}, (args, exec) => serviceForTool(exec).getEvidence(args)))
|
|
339
|
+
|
|
340
|
+
ctx.tools.register(objectTool({
|
|
341
|
+
name: 'harbor_propose_action',
|
|
342
|
+
description: 'Propose a structured, expiring Workbench action draft for an explicit user request, using a fresh Harbor context. This never writes files, starts a Job, changes an Evaluator, runs Gate, or deploys. The user must separately inspect deterministic Preflight and confirm in Harbor. Production actions are unregistered and denied.',
|
|
343
|
+
parameters: {
|
|
344
|
+
contextSnapshotId: { type: 'string', required: true, description: 'Exact fresh hctx token from the user reference.' },
|
|
345
|
+
kind: { type: 'string', required: true, description: 'candidate-draft, evaluator-draft, compare, diagnostic-evaluation, retry-infrastructure, gate-request, or deployment-handoff. Offline execution may be blocked by missing registered runner capabilities.' },
|
|
346
|
+
summary: { type: 'string', required: true, description: 'One bounded proposed change. No credentials or local paths.' },
|
|
347
|
+
rationale: { type: 'string', description: 'Evidence-supported reason and uncertainty, not authorization.' },
|
|
348
|
+
replacement: { type: 'string', description: 'Only evaluator-draft: replacement text for the exact saved source fragment. No file paths. The Host supplies the before text and digest.' },
|
|
349
|
+
},
|
|
350
|
+
}, (args, exec) => {
|
|
351
|
+
const projectRoot = synchronizeWorkbenchProjectRoot(service, exec)
|
|
352
|
+
return service.proposeAction(args, { sessionId: toolSessionId(exec), projectRoot })
|
|
353
|
+
}))
|
|
354
|
+
|
|
254
355
|
ctx.tools.register(jsonTool({
|
|
255
356
|
name: 'harbor_evaluator_inspect',
|
|
256
|
-
description: 'Inspect the active harbor-dsh-evaluator/v1 descriptor, implementation kind, ternary Criteria, and
|
|
357
|
+
description: 'Inspect the active harbor-dsh-evaluator/v1 descriptor, implementation kind, ternary Criteria, and a bounded set of editable source files inside an explicitly untrusted envelope. Source text containing secret- or local-path-shaped values is omitted.',
|
|
257
358
|
parameters: {
|
|
258
359
|
stackPath: { type: 'string', description: 'Defaults to .harbor/evaluation-stack.yml' },
|
|
259
360
|
},
|
|
260
361
|
}, (args, exec) => serviceForTool(exec).evaluatorInspect(args)))
|
|
261
362
|
|
|
262
|
-
ctx.tools.register(
|
|
363
|
+
ctx.tools.register(mutatingJsonTool({
|
|
263
364
|
name: 'harbor_evaluator_update',
|
|
264
365
|
description: 'Update one descriptor-authorized Evaluator source file with optimistic concurrency. Requires new Evaluator and Stack identities and never runs evaluation or Gate automatically.',
|
|
265
366
|
parameters: {
|
|
@@ -272,7 +373,7 @@ export function apply(ctx, config) {
|
|
|
272
373
|
},
|
|
273
374
|
}, (args, exec) => serviceForTool(exec).evaluator(args)))
|
|
274
375
|
|
|
275
|
-
ctx.tools.register(
|
|
376
|
+
ctx.tools.register(mutatingJsonTool({
|
|
276
377
|
name: 'harbor_ground_truth_init',
|
|
277
378
|
description: 'Create a non-overwriting Ground Truth draft for evaluator meta-evaluation. GT may be human, programmatic, consensus, model, or external, but must have explicit provenance and remain independent of the Candidate evaluator.',
|
|
278
379
|
parameters: {
|
|
@@ -287,7 +388,7 @@ export function apply(ctx, config) {
|
|
|
287
388
|
},
|
|
288
389
|
}, (args, exec) => serviceForTool(exec).groundTruthInitialize(args)))
|
|
289
390
|
|
|
290
|
-
ctx.tools.register(
|
|
391
|
+
ctx.tools.register(mutatingJsonTool({
|
|
291
392
|
name: 'harbor_evaluator_meta_evaluate',
|
|
292
393
|
description: 'Compare repeated evaluator-observations/v1 with independent ground-truth/v1 and write an ESF, SCE, and RCR meta-evaluation report.',
|
|
293
394
|
parameters: {
|
|
@@ -298,7 +399,7 @@ export function apply(ctx, config) {
|
|
|
298
399
|
},
|
|
299
400
|
}, (args, exec) => serviceForTool(exec).evaluatorMetaEvaluate(args)))
|
|
300
401
|
|
|
301
|
-
ctx.tools.register(
|
|
402
|
+
ctx.tools.register(mutatingJsonTool({
|
|
302
403
|
name: 'harbor_candidate_compare',
|
|
303
404
|
description: 'Apply the deterministic Promotion Gate to a baseline Job and a Candidate Job.',
|
|
304
405
|
parameters: {
|