harness-everything 0.11.6 → 0.12.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +1 -1
- package/.claude-plugin/plugin.json +1 -1
- package/CHANGELOG.md +45 -0
- package/README.md +7 -3
- package/behavioral-evals/README.md +37 -0
- package/behavioral-evals/cases/self-evolve-retrieval-recurrence.yaml +48 -0
- package/behavioral-evals/paired-benchmark.js +211 -5
- package/ci/contract-integrity.test.js +68 -2
- package/ci/mechanism-2a-rule-of-3.test.js +71 -0
- package/ci/mechanism-2ah-workflow-lifecycle.test.js +276 -0
- package/ci/mechanism-2c-state-persist.test.js +45 -2
- package/ci/mechanism-2t-issue-20.test.js +0 -8
- package/ci/mechanism-30-opencode-plugin-loadability.test.js +39 -7
- package/ci/mechanism-32-paired-benchmark.test.js +139 -0
- package/ci/openai-plugin-package.test.js +17 -0
- package/ci/self-evolve-loop.test.js +37 -2
- package/ci/telemetry.test.js +31 -1
- package/docs/architecture.md +17 -0
- package/docs/audit.md +52 -5
- package/docs/release-evidence.md +3 -0
- package/docs/workflow-runtime.md +13 -0
- package/docs/workflows/environment-detection.md +47 -37
- package/docs/workflows/eval-harness.md +45 -38
- package/docs/workflows/fable-discipline.md +45 -33
- package/docs/workflows/git-commit.md +53 -45
- package/docs/workflows/grill-me.md +36 -38
- package/docs/workflows/grill-with-docs.md +36 -46
- package/docs/workflows/harness-everything.md +59 -57
- package/docs/workflows/improve-codebase-architecture.md +36 -44
- package/docs/workflows/install-cognitive-os.md +36 -29
- package/docs/workflows/repo-docs.md +29 -40
- package/docs/workflows/rewrite-commits.md +40 -36
- package/docs/workflows/security-review.md +34 -53
- package/docs/workflows/skill-creator.md +39 -56
- package/docs/workflows/skill-style.md +35 -40
- package/docs/workflows/using-git-worktrees.md +40 -44
- package/docs/workflows/verification-loop.md +43 -48
- package/docs/workflows/verify-before-claim.md +37 -40
- package/docs/workflows/zoom-out.md +39 -51
- package/grill-with-docs/ADR-FORMAT.md +22 -0
- package/grill-with-docs/SKILL.md +1 -1
- package/harness-everything/SKILL.md +1 -1
- package/harness-everything/scripts/kernel-router.js +38 -10
- package/hooks/scripts/lib/learning-opportunity.js +22 -0
- package/hooks/scripts/lib/telemetry.js +40 -0
- package/hooks/scripts/lib/workflow-isolation.js +30 -1
- package/hooks/scripts/lib/workflow-runtime.js +18 -1
- package/hooks/scripts/rule-of-3-tracker.js +17 -9
- package/hooks/scripts/state-persist.js +19 -7
- package/hooks/scripts/workflow-disposition.js +19 -8
- package/hooks/scripts/workflow-gate.js +21 -4
- package/multi-agent-workspace/SKILL.md +1 -1
- package/multi-agent-workspace/scripts/index_memory.js +28 -0
- package/opencode-plugin/plugin.json +1 -1
- package/package.json +1 -1
- package/plugins/harness-everything/.codex-plugin/plugin.json +1 -1
- package/plugins/harness-everything/contract-integrity/scripts/audit.js +536 -0
- package/plugins/harness-everything/hooks/scripts/lib/learning-opportunity.js +22 -0
- package/plugins/harness-everything/hooks/scripts/lib/telemetry.js +40 -0
- package/plugins/harness-everything/hooks/scripts/lib/workflow-isolation.js +30 -1
- package/plugins/harness-everything/hooks/scripts/lib/workflow-runtime.js +18 -1
- package/plugins/harness-everything/hooks/scripts/rule-of-3-tracker.js +17 -9
- package/plugins/harness-everything/hooks/scripts/state-persist.js +19 -7
- package/plugins/harness-everything/hooks/scripts/workflow-disposition.js +19 -8
- package/plugins/harness-everything/hooks/scripts/workflow-gate.js +21 -4
- package/plugins/harness-everything/plugin.json +1 -1
- package/plugins/harness-everything/skills/grill-with-docs/ADR-FORMAT.md +22 -0
- package/plugins/harness-everything/skills/grill-with-docs/SKILL.md +1 -1
- package/plugins/harness-everything/skills/harness-everything/SKILL.md +1 -1
- package/plugins/harness-everything/skills/harness-everything/scripts/kernel-router.js +38 -10
- package/plugins/harness-everything/skills/multi-agent-workspace/SKILL.md +1 -1
- package/plugins/harness-everything/skills/multi-agent-workspace/scripts/index_memory.js +28 -0
- package/plugins/harness-everything/skills/self-evolve/SKILL.md +1 -1
- package/plugins/harness-everything/skills/self-evolve/scripts/lesson-candidate.js +52 -0
- package/plugins/harness-everything/skills/to-spec/SKILL.md +1 -1
- package/plugins/harness-everything/skills/to-spec/references/process.md +26 -0
- package/plugins/harness-everything/skills/to-spec/templates/cli-reference.md +14 -0
- package/plugins/harness-everything/skills/to-spec/templates/dev-doc.md +14 -0
- package/plugins/harness-everything/skills/to-spec/templates/feature-spec.md +14 -0
- package/plugins/harness-everything/skills/to-spec/templates/schema-doc.md +14 -0
- package/plugins/harness-everything/skills/to-tickets/SKILL.md +1 -1
- package/plugins/harness-everything/skills/to-tickets/references/publishing-and-templates.md +20 -1
- package/plugins/harness-everything/skills/verification-loop/SKILL.md +3 -2
- package/plugins/harness-everything/skills/verification-loop/references/verification-phases.md +25 -1
- package/plugins/harness-everything/skills/verification-loop/scripts/contract-integrity-audit.js +31 -0
- package/plugins/harness-everything/skills/verification-loop/templates/verification-report.template.md +3 -0
- package/scripts/sync-openai-plugin.js +12 -0
- package/self-evolve/SKILL.md +1 -1
- package/self-evolve/scripts/lesson-candidate.js +52 -0
- package/telemetry/schemas/event.schema.json +141 -22
- package/telemetry/scripts/report.js +42 -0
- package/to-spec/SKILL.md +1 -1
- package/to-spec/references/process.md +26 -0
- package/to-spec/templates/cli-reference.md +14 -0
- package/to-spec/templates/dev-doc.md +14 -0
- package/to-spec/templates/feature-spec.md +14 -0
- package/to-spec/templates/schema-doc.md +14 -0
- package/to-tickets/SKILL.md +1 -1
- package/to-tickets/references/publishing-and-templates.md +20 -1
- package/verification-loop/SKILL.md +3 -2
- package/verification-loop/references/verification-phases.md +25 -1
- package/verification-loop/scripts/contract-integrity-audit.js +31 -0
- package/verification-loop/templates/verification-report.template.md +3 -0
- package/docs/issue-20-rollup.md +0 -47
- package/docs/reviews/pr-142-issue-131.md +0 -89
|
@@ -162,6 +162,23 @@ function settleMutationProbe(context, key, afterFingerprint, tool) {
|
|
|
162
162
|
});
|
|
163
163
|
}
|
|
164
164
|
|
|
165
|
+
function settleMutationProbeConservative(context, key, tool, evidence = 'unobservable-workspace') {
|
|
166
|
+
return withMutationProbeLock(context, () => {
|
|
167
|
+
const file = mutationProbeFile(context, key);
|
|
168
|
+
const probe = readJson(file);
|
|
169
|
+
if (!probe || probe.schemaVersion !== 1) return null;
|
|
170
|
+
refreshWorkflowForProbe(context);
|
|
171
|
+
fs.unlinkSync(file);
|
|
172
|
+
context.workflow.lastMutationAt = Date.now();
|
|
173
|
+
if (probe.reserveIteration && context.workflow.strategy === 'iterative-single') {
|
|
174
|
+
recordBudgetEvent(context, 'iteration', { evidence: String(tool || 'shell') + ':' + evidence });
|
|
175
|
+
} else {
|
|
176
|
+
saveWorkflow(context);
|
|
177
|
+
}
|
|
178
|
+
return { changed: true, probe, conservative: true };
|
|
179
|
+
});
|
|
180
|
+
}
|
|
181
|
+
|
|
165
182
|
function discardMutationProbe(context, key) {
|
|
166
183
|
return withMutationProbeLock(context, () => {
|
|
167
184
|
const file = mutationProbeFile(context, key);
|
|
@@ -396,4 +413,4 @@ function readHookInput(decide) {
|
|
|
396
413
|
process.stdin.on('error', finish);
|
|
397
414
|
}
|
|
398
415
|
|
|
399
|
-
module.exports = { OPEN_STATES, SAFE_ID, WORKFLOW_CONTROLLER_COMMANDS, isMajorWorkflow, loadWorkflow, saveWorkflow, budgetLimits, ensureWorkflowBudget, mutationProbeReservations, assertIterationCapacity, registerMutationProbe, peekMutationProbe, settleMutationProbe, discardMutationProbe, clearMutationProbes, recordBudgetEvent, resetWorkflowBudget, syncBudgetToRun, matchingRun, unresolvedStages, readHookInput };
|
|
416
|
+
module.exports = { OPEN_STATES, SAFE_ID, WORKFLOW_CONTROLLER_COMMANDS, isMajorWorkflow, loadWorkflow, saveWorkflow, budgetLimits, ensureWorkflowBudget, mutationProbeReservations, assertIterationCapacity, registerMutationProbe, peekMutationProbe, settleMutationProbe, settleMutationProbeConservative, discardMutationProbe, clearMutationProbes, recordBudgetEvent, resetWorkflowBudget, syncBudgetToRun, matchingRun, unresolvedStages, readHookInput };
|
|
@@ -17,6 +17,7 @@ process.stdin.on('end', () => {
|
|
|
17
17
|
const payload = JSON.parse(inputData);
|
|
18
18
|
const hookEventName = payload.hook_event_name || payload.hookEventName || '';
|
|
19
19
|
const isFailureEvent = hookEventName === 'PostToolUseFailure';
|
|
20
|
+
const isSuccessEvent = hookEventName === 'PostToolUse';
|
|
20
21
|
const toolResponse = payload.tool_response || {};
|
|
21
22
|
const stdout = toolResponse.stdout ?? payload.stdout ?? '';
|
|
22
23
|
const stderr = toolResponse.stderr ?? payload.stderr ?? '';
|
|
@@ -25,10 +26,10 @@ process.stdin.on('end', () => {
|
|
|
25
26
|
const exitCode = typeof rawExitCode === 'number' ? rawExitCode : undefined;
|
|
26
27
|
|
|
27
28
|
const explicitFailure = exitCode !== undefined && exitCode !== 0;
|
|
28
|
-
const explicitSuccess = exitCode === 0;
|
|
29
29
|
const stderrSignal = typeof stderr === 'string' && stderr.trim().length > 0;
|
|
30
30
|
const looksLikeError = /\b(error|fail|failed|failure|exception|fatal|panic|traceback|denied|refused|cannot|unable)\b/i.test(stderr);
|
|
31
|
-
const isFailure = isFailureEvent || explicitFailure ||
|
|
31
|
+
const isFailure = isFailureEvent || explicitFailure ||
|
|
32
|
+
(!isSuccessEvent && exitCode === undefined && stderrSignal && looksLikeError);
|
|
32
33
|
const errorText = isFailureEvent
|
|
33
34
|
? (failureText || (stderrSignal ? stderr : stdout) || '')
|
|
34
35
|
: ((stderrSignal ? stderr : stdout) || failureText || '');
|
|
@@ -47,19 +48,22 @@ process.stdin.on('end', () => {
|
|
|
47
48
|
const filePath = payload.tool_input?.file_path || payload.tool_input?.filePath || '';
|
|
48
49
|
const command = payload.tool_input?.command || '';
|
|
49
50
|
|
|
50
|
-
// Categorize failure type for context-aware thresholds
|
|
51
|
+
// Categorize failure type for context-aware thresholds.
|
|
52
|
+
// NOTE (#167): match anchored diagnostic phrases, not bare substrings.
|
|
53
|
+
// A bare `denied`/`timeout` substring matches file names such as
|
|
54
|
+
// `workflow-mutation-denied.js` and misclassifies the failure.
|
|
51
55
|
let category = 'unknown';
|
|
52
|
-
if (
|
|
56
|
+
if (/\bSyntaxError\b|Unexpected token|Syntax Error/i.test(errorText)) {
|
|
53
57
|
category = 'syntax';
|
|
54
|
-
} else if (
|
|
58
|
+
} else if (/\bENOENT\b|\bnot found\b|No such file|找不到|不存在/i.test(errorText)) {
|
|
55
59
|
category = 'environment';
|
|
56
|
-
} else if (
|
|
60
|
+
} else if (/\bETIMEDOUT\b|TimeoutError|\btimeout\b|\btimed out\b|\btiming out\b|超時|逾時/i.test(errorText)) {
|
|
57
61
|
category = 'timeout';
|
|
58
62
|
} else if (/test.*fail|assertion|AssertionError|測試.*失敗|斷言.*錯誤/i.test(errorText)) {
|
|
59
63
|
category = 'test';
|
|
60
|
-
} else if (
|
|
64
|
+
} else if (/\bpermission denied\b|\bEACCES\b|\bEPERM\b|權限.*(拒絕|不足|被拒)|拒絕.*權限/i.test(errorText)) {
|
|
61
65
|
category = 'permission';
|
|
62
|
-
} else if (
|
|
66
|
+
} else if (/\bdependency\b|\bmodule not found\b|Cannot find module|模組.*找不到/i.test(errorText)) {
|
|
63
67
|
category = 'dependency';
|
|
64
68
|
}
|
|
65
69
|
|
|
@@ -114,7 +118,11 @@ process.stdin.on('end', () => {
|
|
|
114
118
|
state.lastFailureAt = Date.now();
|
|
115
119
|
|
|
116
120
|
fs.writeFileSync(stateFile, JSON.stringify(state, null, 2), 'utf8');
|
|
117
|
-
} else if (!
|
|
121
|
+
} else if (!isFailure) {
|
|
122
|
+
// NOTE (#166, follows #153): Claude Code PostToolUse success payloads
|
|
123
|
+
// carry no numeric exit code. The lifecycle event is authoritative:
|
|
124
|
+
// PostToolUse is success unless a numeric non-zero status contradicts it;
|
|
125
|
+
// hosts without lifecycle event names keep the conservative stderr fallback.
|
|
118
126
|
// A successful action after an accepted zoom-out is an objective recovery
|
|
119
127
|
// boundary. Emit only IDs/hashes/category; never copy command/output text.
|
|
120
128
|
if ((state.zoomOutCycles || 0) > 0 && state.lastHash) {
|
|
@@ -2,8 +2,8 @@
|
|
|
2
2
|
const fs = require('fs');
|
|
3
3
|
const path = require('path');
|
|
4
4
|
const { getWorkspaceRoot, getSessionDir } = require('./lib/harness-state');
|
|
5
|
-
const { loadWorkflow, saveWorkflow, peekMutationProbe, settleMutationProbe } = require('./lib/workflow-runtime');
|
|
6
|
-
const { cwdOf, shellProbeKey, workspaceFingerprint } = require('./lib/workflow-isolation');
|
|
5
|
+
const { loadWorkflow, saveWorkflow, isMajorWorkflow, peekMutationProbe, settleMutationProbe, settleMutationProbeConservative } = require('./lib/workflow-runtime');
|
|
6
|
+
const { cwdOf, shellProbeKey, workspaceFingerprint, directMutationInWorkspace } = require('./lib/workflow-isolation');
|
|
7
7
|
const { observeTool } = require('./lib/telemetry');
|
|
8
8
|
|
|
9
9
|
// Commands that count as "verification ran" for the Stop gate
|
|
@@ -42,6 +42,10 @@ function observeWorkspaceMutation(payload, root) {
|
|
|
42
42
|
try {
|
|
43
43
|
after = workspaceFingerprint(cwd);
|
|
44
44
|
} catch (error) {
|
|
45
|
+
if (!isMajorWorkflow(context.workflow)) {
|
|
46
|
+
const conservative = settleMutationProbeConservative(context, probeKey, toolName, 'unobservable-workspace');
|
|
47
|
+
return Boolean(conservative && conservative.changed);
|
|
48
|
+
}
|
|
45
49
|
context.workflow.state = 'blocked';
|
|
46
50
|
context.workflow.blockReason = 'mutation-observation-failed';
|
|
47
51
|
saveWorkflow(context);
|
|
@@ -79,9 +83,10 @@ function processState(payload) {
|
|
|
79
83
|
const exitCode = typeof rawExitCode === 'number' ? rawExitCode : undefined;
|
|
80
84
|
const stderrSignal = typeof stderr === 'string' && stderr.trim().length > 0;
|
|
81
85
|
const hookEvent = payload.hook_event_name || payload.hookEventName || '';
|
|
86
|
+
const isSuccessEvent = hookEvent === 'PostToolUse';
|
|
82
87
|
const isFailed = hookEvent === 'PostToolUseFailure' ||
|
|
83
88
|
(exitCode !== undefined && exitCode !== 0) ||
|
|
84
|
-
(exitCode === undefined && stderrSignal);
|
|
89
|
+
(!isSuccessEvent && exitCode === undefined && stderrSignal);
|
|
85
90
|
const toolName = payload.tool_name || payload.tool || 'command';
|
|
86
91
|
const observedMutation = observeWorkspaceMutation(payload, root);
|
|
87
92
|
|
|
@@ -97,8 +102,13 @@ function processState(payload) {
|
|
|
97
102
|
state.tool = toolName;
|
|
98
103
|
state.exitCode = exitCode;
|
|
99
104
|
state.errorSummary = output.trim();
|
|
100
|
-
} else if (
|
|
101
|
-
// Clear failed state or mark as idle/resolved
|
|
105
|
+
} else if (!isFailed && state.status === 'failed') {
|
|
106
|
+
// Clear failed state or mark as idle/resolved.
|
|
107
|
+
// NOTE (#166, follows #153): Claude Code Bash PostToolUse payloads
|
|
108
|
+
// carry no numeric exit code, so `exitCode === 0` never holds there.
|
|
109
|
+
// A PostToolUse event with no numeric non-zero exit status counts as
|
|
110
|
+
// success even when stderr contains diagnostics. Hosts that omit the
|
|
111
|
+
// lifecycle event name retain the conservative stderr fallback.
|
|
102
112
|
state.lastResolved = { tool: state.tool, timestamp: state.timestamp };
|
|
103
113
|
state.status = 'idle';
|
|
104
114
|
state.timestamp = new Date().toISOString();
|
|
@@ -107,8 +117,10 @@ function processState(payload) {
|
|
|
107
117
|
}
|
|
108
118
|
|
|
109
119
|
// Milestones for the Stop gate: when did the last mutation happen, and
|
|
110
|
-
// has any verification-ish command succeeded since.
|
|
111
|
-
|
|
120
|
+
// has any verification-ish command succeeded since. Direct-tool edits
|
|
121
|
+
// outside the workspace advance neither milestone (#165).
|
|
122
|
+
const isDirectTool = toolName === 'Edit' || toolName === 'Write' || toolName === 'apply_patch';
|
|
123
|
+
if ((isDirectTool && directMutationInWorkspace(payload, cwdOf(payload, root), root)) || observedMutation) {
|
|
112
124
|
state.lastEditAt = Date.now();
|
|
113
125
|
}
|
|
114
126
|
if ((toolName === 'Bash' || toolName === 'PowerShell') && !isFailed) {
|
|
@@ -3,7 +3,7 @@
|
|
|
3
3
|
|
|
4
4
|
const fs = require('fs');
|
|
5
5
|
const path = require('path');
|
|
6
|
-
const { loadWorkflow, saveWorkflow, matchingRun, OPEN_STATES, ensureWorkflowBudget, recordBudgetEvent, resetWorkflowBudget, syncBudgetToRun, WORKFLOW_CONTROLLER_COMMANDS } = require('./lib/workflow-runtime');
|
|
6
|
+
const { loadWorkflow, saveWorkflow, matchingRun, OPEN_STATES, budgetLimits, ensureWorkflowBudget, recordBudgetEvent, resetWorkflowBudget, syncBudgetToRun, WORKFLOW_CONTROLLER_COMMANDS } = require('./lib/workflow-runtime');
|
|
7
7
|
const { getWorkspaceRoot, readCurrentSession } = require('./lib/harness-state');
|
|
8
8
|
const { atomicWriteJson, readJson } = require('./lib/fable-contracts');
|
|
9
9
|
|
|
@@ -32,6 +32,20 @@ function consumer() {
|
|
|
32
32
|
throw new Error('Fable consumer unavailable');
|
|
33
33
|
}
|
|
34
34
|
|
|
35
|
+
function activatePlan(workflow, plan) {
|
|
36
|
+
workflow.workflowPlan = plan;
|
|
37
|
+
workflow.strategy = plan.strategy;
|
|
38
|
+
workflow.strategySelection = plan.strategySelection;
|
|
39
|
+
workflow.tier = plan.tier;
|
|
40
|
+
workflow.requiredInvariants = plan.requiredInvariants || [];
|
|
41
|
+
workflow.suggestedSkills = plan.suggestedSkills || [];
|
|
42
|
+
workflow.verification = plan.verification;
|
|
43
|
+
workflow.mutationIsolation = plan.mutationIsolation || { required: false };
|
|
44
|
+
const budget = ensureWorkflowBudget(workflow);
|
|
45
|
+
budget.limits = budgetLimits(workflow);
|
|
46
|
+
budget.updatedAt = new Date().toISOString();
|
|
47
|
+
}
|
|
48
|
+
|
|
35
49
|
function main() {
|
|
36
50
|
const args = parseArgs(process.argv.slice(2));
|
|
37
51
|
if (!WORKFLOW_CONTROLLER_COMMANDS.has(args.command)) throw new Error(usage());
|
|
@@ -47,10 +61,12 @@ function main() {
|
|
|
47
61
|
if (isReplan) recordBudgetEvent(context, 'replan', { evidence: 'workflow-disposition:start' });
|
|
48
62
|
const plan = workflow.pendingPlan || workflow.workflowPlan;
|
|
49
63
|
if (!String(plan.strategy || '').startsWith('fable-')) {
|
|
64
|
+
activatePlan(workflow, plan);
|
|
50
65
|
workflow.revision++;
|
|
51
66
|
workflow.state = 'running';
|
|
67
|
+
workflow.escapes = [];
|
|
68
|
+
delete workflow.pendingPlan;
|
|
52
69
|
delete workflow.blockReason;
|
|
53
|
-
ensureWorkflowBudget(workflow);
|
|
54
70
|
saveWorkflow(context);
|
|
55
71
|
process.stdout.write(JSON.stringify({ state: workflow.state, workflowId: workflow.workflowId, revision: workflow.revision }) + '\n');
|
|
56
72
|
return;
|
|
@@ -66,18 +82,13 @@ function main() {
|
|
|
66
82
|
routerContract: { workflowPlan: plan }, stages, workspaceRoot: context.root,
|
|
67
83
|
sessionId, workflowId: workflow.workflowId,
|
|
68
84
|
});
|
|
69
|
-
workflow
|
|
70
|
-
workflow.strategy = plan.strategy;
|
|
71
|
-
workflow.tier = plan.tier;
|
|
72
|
-
workflow.verification = plan.verification;
|
|
73
|
-
workflow.mutationIsolation = plan.mutationIsolation || workflow.mutationIsolation;
|
|
85
|
+
activatePlan(workflow, plan);
|
|
74
86
|
workflow.runId = run.runId;
|
|
75
87
|
workflow.revision++;
|
|
76
88
|
workflow.state = 'running';
|
|
77
89
|
workflow.escapes = [];
|
|
78
90
|
delete workflow.pendingPlan;
|
|
79
91
|
delete workflow.blockReason;
|
|
80
|
-
ensureWorkflowBudget(workflow);
|
|
81
92
|
} else if (args.command === 'revision') {
|
|
82
93
|
recordBudgetEvent(context, 'revision', { evidence: args.evidence || 'workflow-disposition:revision' });
|
|
83
94
|
} else if (args.command === 'reset-budget') {
|
|
@@ -4,7 +4,7 @@
|
|
|
4
4
|
const path = require('path');
|
|
5
5
|
const fs = require('fs');
|
|
6
6
|
const { loadWorkflow, saveWorkflow, isMajorWorkflow, matchingRun, registerMutationProbe, recordBudgetEvent, readHookInput, WORKFLOW_CONTROLLER_COMMANDS } = require('./lib/workflow-runtime');
|
|
7
|
-
const { key, classifyShell, workspaceFingerprint, shellProbeKey, cwdOf, commandOf, mutationPaths, linkedWorktree, assertTargets, assertShellScope } = require('./lib/workflow-isolation');
|
|
7
|
+
const { key, classifyShell, workspaceFingerprint, shellProbeKey, cwdOf, commandOf, mutationPaths, directMutationInWorkspace, linkedWorktree, assertTargets, assertShellScope } = require('./lib/workflow-isolation');
|
|
8
8
|
|
|
9
9
|
const DIRECT = new Set(['Edit', 'Write', 'apply_patch']);
|
|
10
10
|
const SHELL = new Set(['Bash', 'PowerShell', 'exec_command']);
|
|
@@ -48,6 +48,10 @@ function decide(payload) {
|
|
|
48
48
|
throw new Error('selected Fable workflow has no correlated run. Write workflow-stages.json at the displayed session path, then use workflow-disposition.js start.');
|
|
49
49
|
}
|
|
50
50
|
if (DIRECT.has(tool)) {
|
|
51
|
+
// Bookkeeping outside the workspace (scratchpad, memory files, temp
|
|
52
|
+
// notes) is not a workspace mutation: it consumes no Tier-2 iteration
|
|
53
|
+
// and advances no mutation milestone (#165).
|
|
54
|
+
if (!directMutationInWorkspace(payload, cwd, root)) return;
|
|
51
55
|
if (workflow.strategy === 'iterative-single') {
|
|
52
56
|
recordBudgetEvent(context, 'iteration', { evidence: `${tool}:direct-mutation` });
|
|
53
57
|
}
|
|
@@ -58,9 +62,22 @@ function decide(payload) {
|
|
|
58
62
|
|
|
59
63
|
// Shell safety and shell mutation accounting are intentionally separate.
|
|
60
64
|
// `untrusted` means "not proven read-only", not "a mutation occurred".
|
|
61
|
-
// Capture an opaque pre-execution workspace fingerprint
|
|
62
|
-
//
|
|
63
|
-
|
|
65
|
+
// Capture an opaque pre-execution workspace fingerprint when possible. A
|
|
66
|
+
// non-major workflow may run outside Git, where mutation observation is an
|
|
67
|
+
// accounting aid rather than an isolation boundary; degrade conservatively
|
|
68
|
+
// instead of blocking the user's command (#162).
|
|
69
|
+
let fingerprint;
|
|
70
|
+
try {
|
|
71
|
+
fingerprint = workspaceFingerprint(cwd);
|
|
72
|
+
} catch (error) {
|
|
73
|
+
if (isMajorWorkflow(workflow)) throw error;
|
|
74
|
+
if (workflow.strategy === 'iterative-single') {
|
|
75
|
+
recordBudgetEvent(context, 'iteration', { evidence: 'shell:unobservable-workspace' });
|
|
76
|
+
}
|
|
77
|
+
workflow.lastMutationAt = Date.now();
|
|
78
|
+
saveWorkflow(context);
|
|
79
|
+
return;
|
|
80
|
+
}
|
|
64
81
|
const probeKey = shellProbeKey(payload, cwd);
|
|
65
82
|
registerMutationProbe(context, probeKey, fingerprint, workflow.strategy === 'iterative-single');
|
|
66
83
|
}
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"$schema": "https://agent-plugins.org/schemas/1.0.0/plugin.schema.json",
|
|
3
3
|
"name": "harness-everything",
|
|
4
|
-
"version": "0.
|
|
4
|
+
"version": "0.12.0",
|
|
5
5
|
"description": "Invariant-first engineering harness for routing, verification, and failure recovery while preserving agent autonomy.",
|
|
6
6
|
"author": {
|
|
7
7
|
"name": "Miya Daniel",
|
|
@@ -22,6 +22,28 @@ Only include these when they add genuine value. Most ADRs won't need them.
|
|
|
22
22
|
- **Considered Options** — only when the rejected alternatives are worth remembering
|
|
23
23
|
- **Consequences** — only when non-obvious downstream effects need to be called out
|
|
24
24
|
|
|
25
|
+
## Decision lifecycle and contract lineage
|
|
26
|
+
|
|
27
|
+
Once an ADR is **accepted**, keep its decision and rationale historically stable. Do not rewrite an accepted ADR to make a later design look as though it was always the decision.
|
|
28
|
+
|
|
29
|
+
Use compact metadata when the ADR participates in a behavior contract:
|
|
30
|
+
|
|
31
|
+
```md
|
|
32
|
+
ADR-ID: ADR-0001
|
|
33
|
+
Status: proposed | accepted | deprecated | superseded by ADR-0002
|
|
34
|
+
Affected requirements: REQ-001, REQ-004
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
When a later change materially changes the decision or its rationale:
|
|
38
|
+
|
|
39
|
+
1. create a new sequential ADR;
|
|
40
|
+
2. state the new decision and why the prior trade-off changed;
|
|
41
|
+
3. mark the earlier ADR `superseded by ADR-NNNN`;
|
|
42
|
+
4. update the living spec requirements affected by that decision;
|
|
43
|
+
5. let the structured contract trace carry detailed ticket/test/implementation lineage.
|
|
44
|
+
|
|
45
|
+
A ticket or implementation note never silently becomes the new architectural source of truth. If only implementation changes and the approved behavior/decision does not, keep the accepted ADR unchanged.
|
|
46
|
+
|
|
25
47
|
## Numbering
|
|
26
48
|
|
|
27
49
|
Scan `docs/adr/` for the highest existing number and increment by one.
|
|
@@ -33,12 +33,26 @@ function issueMemoryCapability(workflow) {
|
|
|
33
33
|
return token;
|
|
34
34
|
}
|
|
35
35
|
|
|
36
|
+
// Host-generated background-task notifications arrive as UserPromptSubmit
|
|
37
|
+
// hook payloads but are not user input (#168). Routing them like prompts
|
|
38
|
+
// replaces the workflow contract mid-turn (e.g. deferred -> a new Tier-2).
|
|
39
|
+
// A prompt that is entirely a notification envelope must never create or
|
|
40
|
+
// replace a workflow; the current contract is kept unchanged.
|
|
41
|
+
function isHostNotificationPrompt(prompt) {
|
|
42
|
+
const text = String(prompt || '').trim();
|
|
43
|
+
if (!text) return false;
|
|
44
|
+
return /^<task-notification[\s>][\s\S]*<\/task-notification>\s*$/i.test(text);
|
|
45
|
+
}
|
|
46
|
+
|
|
36
47
|
function persistWorkflow(plan, payload, prompt) {
|
|
37
48
|
if (!payload || !(payload.session_id || payload.sessionId)) return null;
|
|
38
49
|
const { runtime, hooksRoot } = loadRuntime();
|
|
39
50
|
const context = runtime.loadWorkflow(payload);
|
|
40
51
|
const { writeCurrentSession } = require(path.join(hooksRoot, 'lib/harness-state'));
|
|
41
52
|
writeCurrentSession(context.root, context.sessionId);
|
|
53
|
+
if (isHostNotificationPrompt(prompt)) {
|
|
54
|
+
return { ...context, hooksRoot, retained: true, hostNotification: true };
|
|
55
|
+
}
|
|
42
56
|
if (context.workflow && runtime.OPEN_STATES.has(context.workflow.state)) {
|
|
43
57
|
if (!context.workflow.workflowPlan) throw new Error('legacy unresolved workflow needs explicit migration; refusing to replace its obligations');
|
|
44
58
|
// Steering/status prompts cannot drop an unresolved execution contract.
|
|
@@ -92,17 +106,30 @@ function run(raw) {
|
|
|
92
106
|
try { payload = JSON.parse(raw); } catch (_) { console.error('[Workflow Router] Invalid hook payload; lifecycle persistence unavailable.'); }
|
|
93
107
|
}
|
|
94
108
|
const prompt = promptArg || (typeof payload?.prompt === 'string' ? payload.prompt : '');
|
|
95
|
-
const
|
|
96
|
-
|
|
109
|
+
const hostNotification = isHostNotificationPrompt(prompt);
|
|
110
|
+
// Host-generated notifications are lifecycle events, not user-authored task
|
|
111
|
+
// input. Never feed their text into the router: when there is an active
|
|
112
|
+
// contract we retain it below; when there is none, the event is a no-op.
|
|
113
|
+
const result = hostNotification ? null : core.route(promptArg, raw || null);
|
|
114
|
+
let plan = result?.contract?.workflowPlan || null;
|
|
97
115
|
let persisted = null;
|
|
98
116
|
let failure = null;
|
|
99
117
|
try {
|
|
100
118
|
persisted = persistWorkflow(plan, payload, prompt);
|
|
101
|
-
if (persisted?.retained) plan = persisted.workflow.workflowPlan;
|
|
119
|
+
if (persisted?.retained && persisted.workflow) plan = persisted.workflow.workflowPlan;
|
|
102
120
|
} catch (error) { failure = error; }
|
|
103
|
-
if (
|
|
104
|
-
|
|
105
|
-
|
|
121
|
+
if (hostNotification && !persisted?.workflow) {
|
|
122
|
+
console.log('\n=> Host notification — no active execution contract; ignored for routing.');
|
|
123
|
+
if (failure) {
|
|
124
|
+
console.error('[Workflow Router] ' + failure.message);
|
|
125
|
+
process.exitCode = 2;
|
|
126
|
+
}
|
|
127
|
+
return;
|
|
128
|
+
}
|
|
129
|
+
if (result?.sanitized) console.log(result.sanitized);
|
|
130
|
+
if (result?.stderr) process.stderr.write(result.stderr);
|
|
131
|
+
if (persisted?.hostNotification) console.log('\n=> Host notification — routing unchanged; keeping the active execution contract.');
|
|
132
|
+
else if (persisted?.retained) console.log('\n=> Retaining unresolved workflow from earlier prompts; the checkpoint below is the active execution contract.');
|
|
106
133
|
core.printWorkflowPlan(plan);
|
|
107
134
|
core.printRoutingCheckpoint(plan);
|
|
108
135
|
core.printKernelContract(plan);
|
|
@@ -116,7 +143,7 @@ function run(raw) {
|
|
|
116
143
|
console.log(' - Tier 3 / Fable source and artifact mutations require Git worktree isolation; unavailable isolation means BLOCKED.');
|
|
117
144
|
console.log(' - Escape is limited to workflow-uncovered-scope or host-capability-unavailable, with stage, scope and evidence; other obligations remain mandatory.');
|
|
118
145
|
}
|
|
119
|
-
if (persisted) {
|
|
146
|
+
if (persisted?.workflow) {
|
|
120
147
|
const controller = path.join(persisted.hooksRoot, 'workflow-disposition.js');
|
|
121
148
|
console.log(' - Workflow id: ' + persisted.workflow.workflowId);
|
|
122
149
|
console.log(' - Memory write disposition: ' + (persisted.workflow.workflowPlan.memory?.write || 'none'));
|
|
@@ -130,12 +157,13 @@ function run(raw) {
|
|
|
130
157
|
if (failure) {
|
|
131
158
|
console.error('[Workflow Router] ' + failure.message);
|
|
132
159
|
process.exitCode = 2;
|
|
133
|
-
} else if (result.status !== 0) process.exitCode = result.status || 1;
|
|
160
|
+
} else if (result && result.status !== 0) process.exitCode = result.status || 1;
|
|
134
161
|
}
|
|
135
162
|
|
|
163
|
+
if (typeof module !== 'undefined') module.exports = { isHostNotificationPrompt };
|
|
164
|
+
|
|
136
165
|
if (process.argv.length > 2 || process.stdin.isTTY) run('');
|
|
137
|
-
else {
|
|
138
|
-
let input = '';
|
|
166
|
+
else { let input = '';
|
|
139
167
|
let finished = false;
|
|
140
168
|
const finish = () => { if (finished) return; finished = true; clearTimeout(timeout); run(input); };
|
|
141
169
|
const timeout = setTimeout(finish, 500);
|
|
@@ -5,6 +5,18 @@ const fs = require('fs');
|
|
|
5
5
|
const path = require('path');
|
|
6
6
|
const { getWorkspaceRoot, getRuntimeRoot, resolveProjectDocs } = require('./project-docs-resolver');
|
|
7
7
|
|
|
8
|
+
let emitLessonTelemetry = () => ({ ok: false, unavailable: true });
|
|
9
|
+
for (const relative of [
|
|
10
|
+
'../../hooks/scripts/lib/telemetry.js',
|
|
11
|
+
'../../../hooks/scripts/lib/telemetry.js',
|
|
12
|
+
]) {
|
|
13
|
+
const candidate = path.resolve(__dirname, relative);
|
|
14
|
+
if (fs.existsSync(candidate)) {
|
|
15
|
+
({ emitLessonTelemetry } = require(candidate));
|
|
16
|
+
break;
|
|
17
|
+
}
|
|
18
|
+
}
|
|
19
|
+
|
|
8
20
|
function option(args, ...names) {
|
|
9
21
|
for (const name of names) {
|
|
10
22
|
const index = args.indexOf(name);
|
|
@@ -166,6 +178,22 @@ function retrieveMemoryRecords({ workspace, task = '', requirement = '', role =
|
|
|
166
178
|
});
|
|
167
179
|
}
|
|
168
180
|
|
|
181
|
+
for (const record of included) {
|
|
182
|
+
if (!record.origin?.lessonCandidateId) continue;
|
|
183
|
+
emitLessonTelemetry({
|
|
184
|
+
event: 'lesson_retrieved',
|
|
185
|
+
root,
|
|
186
|
+
sessionId: record.writer?.sessionId || null,
|
|
187
|
+
candidateId: record.origin.lessonCandidateId,
|
|
188
|
+
observedAt: new Date(nowMs).toISOString(),
|
|
189
|
+
status: 'success',
|
|
190
|
+
reasonCodes: [
|
|
191
|
+
'scoped-memory-retrieval',
|
|
192
|
+
...(record.retrieval?.reasonCodes || []),
|
|
193
|
+
],
|
|
194
|
+
});
|
|
195
|
+
}
|
|
196
|
+
|
|
169
197
|
return {
|
|
170
198
|
schemaVersion: 1,
|
|
171
199
|
workspace: root,
|
|
@@ -6,6 +6,18 @@ const path = require('path');
|
|
|
6
6
|
const { spawnSync } = require('child_process');
|
|
7
7
|
const { contentHash, scoreRuleQuality, screenMemoryForPersistence } = require('./persist-memory');
|
|
8
8
|
|
|
9
|
+
let emitLessonTelemetry = () => ({ ok: false, unavailable: true });
|
|
10
|
+
for (const relative of [
|
|
11
|
+
'../../hooks/scripts/lib/telemetry.js',
|
|
12
|
+
'../../../hooks/scripts/lib/telemetry.js',
|
|
13
|
+
]) {
|
|
14
|
+
const candidate = path.resolve(__dirname, relative);
|
|
15
|
+
if (fs.existsSync(candidate)) {
|
|
16
|
+
({ emitLessonTelemetry } = require(candidate));
|
|
17
|
+
break;
|
|
18
|
+
}
|
|
19
|
+
}
|
|
20
|
+
|
|
9
21
|
let getWorkspaceRoot;
|
|
10
22
|
let getWorkspaceStateDir;
|
|
11
23
|
try {
|
|
@@ -69,6 +81,24 @@ function pushState(candidate, state, reasonCode, evidenceRef = null) {
|
|
|
69
81
|
}].slice(-32);
|
|
70
82
|
}
|
|
71
83
|
|
|
84
|
+
function emitCandidateLifecycle(input, candidate, event, state, status = 'unknown') {
|
|
85
|
+
const history = [...(candidate.history || [])].reverse().find(item => item.state === state);
|
|
86
|
+
const reasonCodes = [
|
|
87
|
+
history && history.reasonCode,
|
|
88
|
+
candidate.trigger && candidate.trigger.type ? `trigger:${candidate.trigger.type}` : null,
|
|
89
|
+
...(candidate.evaluation && Array.isArray(candidate.evaluation.reasonCodes) ? candidate.evaluation.reasonCodes : []),
|
|
90
|
+
].filter(Boolean);
|
|
91
|
+
return emitLessonTelemetry({
|
|
92
|
+
event,
|
|
93
|
+
root: input.workspace,
|
|
94
|
+
sessionId: input.sessionId || (candidate.provenance && candidate.provenance.sessionId),
|
|
95
|
+
candidateId: candidate.candidateId,
|
|
96
|
+
observedAt: history && history.observedAt,
|
|
97
|
+
status,
|
|
98
|
+
reasonCodes,
|
|
99
|
+
});
|
|
100
|
+
}
|
|
101
|
+
|
|
72
102
|
function replayability(candidate) {
|
|
73
103
|
if (candidate.trigger?.type === 'verifier-fail-pass' &&
|
|
74
104
|
candidate.evidence?.previousStatus === 'fail' &&
|
|
@@ -134,6 +164,8 @@ function evaluateCandidate(input) {
|
|
|
134
164
|
};
|
|
135
165
|
pushState(candidate, 'rejected', 'candidate-screening-or-quality-failed');
|
|
136
166
|
atomicWriteJson(file, candidate);
|
|
167
|
+
emitCandidateLifecycle(input, candidate, 'lesson_screened', 'screened', screening.allowed ? 'success' : 'failure');
|
|
168
|
+
emitCandidateLifecycle(input, candidate, 'lesson_rejected', 'rejected', 'failure');
|
|
137
169
|
return { file, candidate };
|
|
138
170
|
}
|
|
139
171
|
|
|
@@ -150,6 +182,15 @@ function evaluateCandidate(input) {
|
|
|
150
182
|
pushState(candidate, 'evaluated', replay.reasonCode);
|
|
151
183
|
pushState(candidate, disposition, disposition === 'accepted' ? 'eligible-for-governed-promotion' : 'paired-or-later-evidence-required');
|
|
152
184
|
atomicWriteJson(file, candidate);
|
|
185
|
+
emitCandidateLifecycle(input, candidate, 'lesson_screened', 'screened', 'success');
|
|
186
|
+
emitCandidateLifecycle(input, candidate, 'lesson_evaluated', 'evaluated', 'success');
|
|
187
|
+
emitCandidateLifecycle(
|
|
188
|
+
input,
|
|
189
|
+
candidate,
|
|
190
|
+
disposition === 'accepted' ? 'lesson_accepted' : 'lesson_inconclusive',
|
|
191
|
+
disposition,
|
|
192
|
+
disposition === 'accepted' ? 'success' : 'unknown',
|
|
193
|
+
);
|
|
153
194
|
return { file, candidate };
|
|
154
195
|
}
|
|
155
196
|
|
|
@@ -212,6 +253,7 @@ function promoteCandidate(input) {
|
|
|
212
253
|
if (persisted) pushState(candidate, 'persisted', 'accepted-and-authorized-memory-write');
|
|
213
254
|
else pushState(candidate, 'accepted', 'memory-governance-held-as-review-candidate');
|
|
214
255
|
atomicWriteJson(file, candidate);
|
|
256
|
+
if (persisted) emitCandidateLifecycle(input, candidate, 'lesson_persisted', 'persisted', 'success');
|
|
215
257
|
return { ok: true, exitCode: 0, persisted, stdout: result.stdout, stderr: result.stderr, candidate };
|
|
216
258
|
}
|
|
217
259
|
|
|
@@ -234,6 +276,16 @@ function observeCandidate(input) {
|
|
|
234
276
|
};
|
|
235
277
|
pushState(candidate, next, `lesson-${next}`, evidenceRef);
|
|
236
278
|
atomicWriteJson(file, candidate);
|
|
279
|
+
// Retrieval telemetry is emitted by the actual scoped retrieval path so the
|
|
280
|
+
// funnel measures exposure, not an operator-authored lifecycle annotation.
|
|
281
|
+
if (next !== 'retrieved') {
|
|
282
|
+
const event = {
|
|
283
|
+
validated: 'lesson_validated',
|
|
284
|
+
regressed: 'lesson_regressed',
|
|
285
|
+
superseded: 'lesson_superseded',
|
|
286
|
+
}[next];
|
|
287
|
+
if (event) emitCandidateLifecycle(input, candidate, event, next, next === 'regressed' ? 'failure' : 'success');
|
|
288
|
+
}
|
|
237
289
|
return { file, candidate };
|
|
238
290
|
}
|
|
239
291
|
|
|
@@ -60,6 +60,32 @@ What "confirm" means depends on the shape picked:
|
|
|
60
60
|
|
|
61
61
|
Skip this step only if the conversation already pinned down the relevant surface explicitly (e.g. a prior `grill-with-docs` pass already fixed the schema).
|
|
62
62
|
|
|
63
|
+
## Stable contract identity
|
|
64
|
+
|
|
65
|
+
When the published artifact is an authoritative behavior contract, every normative requirement must have a stable identity instead of relying on prose position.
|
|
66
|
+
|
|
67
|
+
Use this compact shape in the selected template:
|
|
68
|
+
|
|
69
|
+
```md
|
|
70
|
+
## Contract Requirements
|
|
71
|
+
|
|
72
|
+
### REQ-001 — <short behavior name>
|
|
73
|
+
**Status:** CURRENT
|
|
74
|
+
**Revision:** 1
|
|
75
|
+
**Decision lineage:** ADR-0001 (or `None`)
|
|
76
|
+
<normative behavior and observable acceptance boundary>
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
Rules:
|
|
80
|
+
|
|
81
|
+
- Never recycle a `REQ-*` ID for different behavior.
|
|
82
|
+
- Behavior changes increment the requirement revision and update the living spec **before** implementation is accepted.
|
|
83
|
+
- Superseded behavior remains traceable; do not erase the old requirement history merely because a ticket changed.
|
|
84
|
+
- If a ticket is classified `changeImpact: architecture`, publish the new/superseding ADR first and link it from the affected requirement.
|
|
85
|
+
- If the document is informative rather than authoritative, say so explicitly and link the authoritative requirement instead of inventing a parallel source of truth.
|
|
86
|
+
|
|
87
|
+
The detailed ADR -> spec -> ticket -> test -> implementation lineage belongs in the structured contract trace; keep the human spec concise.
|
|
88
|
+
|
|
63
89
|
## Publishing details
|
|
64
90
|
|
|
65
91
|
Write the doc using the adapted template at the resolved storage path:
|
|
@@ -28,4 +28,18 @@ Every distinct exit code or error response: what triggers it, what the caller sh
|
|
|
28
28
|
|
|
29
29
|
Other commands/endpoints this one is commonly paired with, supersedes, or is superseded by.
|
|
30
30
|
|
|
31
|
+
## Contract Requirements
|
|
32
|
+
|
|
33
|
+
For every normative behavior this document owns, assign a stable requirement identity:
|
|
34
|
+
|
|
35
|
+
```md
|
|
36
|
+
### REQ-001 — <short behavior name>
|
|
37
|
+
**Status:** CURRENT
|
|
38
|
+
**Revision:** 1
|
|
39
|
+
**Decision lineage:** ADR-0001 (or None)
|
|
40
|
+
<observable command/API behavior or error contract>
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
Do not recycle requirement IDs. A changed caller-visible contract increments the revision before implementation completion.
|
|
44
|
+
|
|
31
45
|
</spec-template>
|
|
@@ -24,4 +24,18 @@ What this decision costs, even though it was the right call.
|
|
|
24
24
|
|
|
25
25
|
How this ships (flag, gradual, all-at-once) and how to undo it if it's wrong.
|
|
26
26
|
|
|
27
|
+
## Contract Requirements
|
|
28
|
+
|
|
29
|
+
If this document owns normative behavior, assign stable requirement identities:
|
|
30
|
+
|
|
31
|
+
```md
|
|
32
|
+
### REQ-001 — <short behavior name>
|
|
33
|
+
**Status:** CURRENT
|
|
34
|
+
**Revision:** 1
|
|
35
|
+
**Decision lineage:** ADR-0001 (or None)
|
|
36
|
+
<observable behavior / acceptance boundary>
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
If this document is informative only, say so and link the authoritative requirement instead. Do not create a second source of truth.
|
|
40
|
+
|
|
27
41
|
</spec-template>
|