harness-everything 0.11.6 → 0.12.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +1 -1
- package/.claude-plugin/plugin.json +1 -1
- package/CHANGELOG.md +45 -0
- package/README.md +7 -3
- package/behavioral-evals/README.md +37 -0
- package/behavioral-evals/cases/self-evolve-retrieval-recurrence.yaml +48 -0
- package/behavioral-evals/paired-benchmark.js +211 -5
- package/ci/contract-integrity.test.js +68 -2
- package/ci/mechanism-2a-rule-of-3.test.js +71 -0
- package/ci/mechanism-2ah-workflow-lifecycle.test.js +276 -0
- package/ci/mechanism-2c-state-persist.test.js +45 -2
- package/ci/mechanism-2t-issue-20.test.js +0 -8
- package/ci/mechanism-30-opencode-plugin-loadability.test.js +39 -7
- package/ci/mechanism-32-paired-benchmark.test.js +139 -0
- package/ci/openai-plugin-package.test.js +17 -0
- package/ci/self-evolve-loop.test.js +37 -2
- package/ci/telemetry.test.js +31 -1
- package/docs/architecture.md +17 -0
- package/docs/audit.md +52 -5
- package/docs/release-evidence.md +3 -0
- package/docs/workflow-runtime.md +13 -0
- package/docs/workflows/environment-detection.md +47 -37
- package/docs/workflows/eval-harness.md +45 -38
- package/docs/workflows/fable-discipline.md +45 -33
- package/docs/workflows/git-commit.md +53 -45
- package/docs/workflows/grill-me.md +36 -38
- package/docs/workflows/grill-with-docs.md +36 -46
- package/docs/workflows/harness-everything.md +59 -57
- package/docs/workflows/improve-codebase-architecture.md +36 -44
- package/docs/workflows/install-cognitive-os.md +36 -29
- package/docs/workflows/repo-docs.md +29 -40
- package/docs/workflows/rewrite-commits.md +40 -36
- package/docs/workflows/security-review.md +34 -53
- package/docs/workflows/skill-creator.md +39 -56
- package/docs/workflows/skill-style.md +35 -40
- package/docs/workflows/using-git-worktrees.md +40 -44
- package/docs/workflows/verification-loop.md +43 -48
- package/docs/workflows/verify-before-claim.md +37 -40
- package/docs/workflows/zoom-out.md +39 -51
- package/grill-with-docs/ADR-FORMAT.md +22 -0
- package/grill-with-docs/SKILL.md +1 -1
- package/harness-everything/SKILL.md +1 -1
- package/harness-everything/scripts/kernel-router.js +38 -10
- package/hooks/scripts/lib/learning-opportunity.js +22 -0
- package/hooks/scripts/lib/telemetry.js +40 -0
- package/hooks/scripts/lib/workflow-isolation.js +30 -1
- package/hooks/scripts/lib/workflow-runtime.js +18 -1
- package/hooks/scripts/rule-of-3-tracker.js +17 -9
- package/hooks/scripts/state-persist.js +19 -7
- package/hooks/scripts/workflow-disposition.js +19 -8
- package/hooks/scripts/workflow-gate.js +21 -4
- package/multi-agent-workspace/SKILL.md +1 -1
- package/multi-agent-workspace/scripts/index_memory.js +28 -0
- package/opencode-plugin/plugin.json +1 -1
- package/package.json +1 -1
- package/plugins/harness-everything/.codex-plugin/plugin.json +1 -1
- package/plugins/harness-everything/contract-integrity/scripts/audit.js +536 -0
- package/plugins/harness-everything/hooks/scripts/lib/learning-opportunity.js +22 -0
- package/plugins/harness-everything/hooks/scripts/lib/telemetry.js +40 -0
- package/plugins/harness-everything/hooks/scripts/lib/workflow-isolation.js +30 -1
- package/plugins/harness-everything/hooks/scripts/lib/workflow-runtime.js +18 -1
- package/plugins/harness-everything/hooks/scripts/rule-of-3-tracker.js +17 -9
- package/plugins/harness-everything/hooks/scripts/state-persist.js +19 -7
- package/plugins/harness-everything/hooks/scripts/workflow-disposition.js +19 -8
- package/plugins/harness-everything/hooks/scripts/workflow-gate.js +21 -4
- package/plugins/harness-everything/plugin.json +1 -1
- package/plugins/harness-everything/skills/grill-with-docs/ADR-FORMAT.md +22 -0
- package/plugins/harness-everything/skills/grill-with-docs/SKILL.md +1 -1
- package/plugins/harness-everything/skills/harness-everything/SKILL.md +1 -1
- package/plugins/harness-everything/skills/harness-everything/scripts/kernel-router.js +38 -10
- package/plugins/harness-everything/skills/multi-agent-workspace/SKILL.md +1 -1
- package/plugins/harness-everything/skills/multi-agent-workspace/scripts/index_memory.js +28 -0
- package/plugins/harness-everything/skills/self-evolve/SKILL.md +1 -1
- package/plugins/harness-everything/skills/self-evolve/scripts/lesson-candidate.js +52 -0
- package/plugins/harness-everything/skills/to-spec/SKILL.md +1 -1
- package/plugins/harness-everything/skills/to-spec/references/process.md +26 -0
- package/plugins/harness-everything/skills/to-spec/templates/cli-reference.md +14 -0
- package/plugins/harness-everything/skills/to-spec/templates/dev-doc.md +14 -0
- package/plugins/harness-everything/skills/to-spec/templates/feature-spec.md +14 -0
- package/plugins/harness-everything/skills/to-spec/templates/schema-doc.md +14 -0
- package/plugins/harness-everything/skills/to-tickets/SKILL.md +1 -1
- package/plugins/harness-everything/skills/to-tickets/references/publishing-and-templates.md +20 -1
- package/plugins/harness-everything/skills/verification-loop/SKILL.md +3 -2
- package/plugins/harness-everything/skills/verification-loop/references/verification-phases.md +25 -1
- package/plugins/harness-everything/skills/verification-loop/scripts/contract-integrity-audit.js +31 -0
- package/plugins/harness-everything/skills/verification-loop/templates/verification-report.template.md +3 -0
- package/scripts/sync-openai-plugin.js +12 -0
- package/self-evolve/SKILL.md +1 -1
- package/self-evolve/scripts/lesson-candidate.js +52 -0
- package/telemetry/schemas/event.schema.json +141 -22
- package/telemetry/scripts/report.js +42 -0
- package/to-spec/SKILL.md +1 -1
- package/to-spec/references/process.md +26 -0
- package/to-spec/templates/cli-reference.md +14 -0
- package/to-spec/templates/dev-doc.md +14 -0
- package/to-spec/templates/feature-spec.md +14 -0
- package/to-spec/templates/schema-doc.md +14 -0
- package/to-tickets/SKILL.md +1 -1
- package/to-tickets/references/publishing-and-templates.md +20 -1
- package/verification-loop/SKILL.md +3 -2
- package/verification-loop/references/verification-phases.md +25 -1
- package/verification-loop/scripts/contract-integrity-audit.js +31 -0
- package/verification-loop/templates/verification-report.template.md +3 -0
- package/docs/issue-20-rollup.md +0 -47
- package/docs/reviews/pr-142-issue-131.md +0 -89
|
@@ -56,4 +56,18 @@ A description of the things that are out of scope for this spec.
|
|
|
56
56
|
|
|
57
57
|
Any further notes about the feature.
|
|
58
58
|
|
|
59
|
+
## Contract Requirements
|
|
60
|
+
|
|
61
|
+
For every normative behavior this document owns, assign a stable requirement identity:
|
|
62
|
+
|
|
63
|
+
```md
|
|
64
|
+
### REQ-001 — <short behavior name>
|
|
65
|
+
**Status:** CURRENT
|
|
66
|
+
**Revision:** 1
|
|
67
|
+
**Decision lineage:** ADR-0001 (or None)
|
|
68
|
+
<observable behavior / acceptance boundary>
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
Do not recycle requirement IDs. When approved behavior changes, increment the revision and reconcile the living spec before implementation completion.
|
|
72
|
+
|
|
59
73
|
</spec-template>
|
|
@@ -28,4 +28,18 @@ What breaks for existing consumers, if anything, and what the mitigation is (dua
|
|
|
28
28
|
|
|
29
29
|
One realistic, complete example instance of the schema.
|
|
30
30
|
|
|
31
|
+
## Contract Requirements
|
|
32
|
+
|
|
33
|
+
For every normative schema behavior this document owns, assign a stable requirement identity:
|
|
34
|
+
|
|
35
|
+
```md
|
|
36
|
+
### REQ-001 — <short behavior name>
|
|
37
|
+
**Status:** CURRENT
|
|
38
|
+
**Revision:** 1
|
|
39
|
+
**Decision lineage:** ADR-0001 (or None)
|
|
40
|
+
<observable validation, compatibility, or migration contract>
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
Do not recycle requirement IDs. A changed schema/validation contract increments the revision before implementation completion.
|
|
44
|
+
|
|
31
45
|
</spec-template>
|
|
@@ -48,6 +48,19 @@ Give each ticket its **blocking edges** — the other tickets that must complete
|
|
|
48
48
|
|
|
49
49
|
**Wide refactors are the exception to vertical slicing.** A **wide refactor** is one mechanical change — rename a column, retype a shared symbol — whose **blast radius** fans across the whole codebase, so a single edit breaks thousands of call sites at once and no vertical slice can land green. Don't force it into a tracer bullet; sequence it as **expand–contract**. First expand: add the new form beside the old so nothing breaks. Then migrate the call sites over in batches sized by blast radius (per package, per directory), each batch its own ticket blocked by the expand, keeping CI green batch to batch because the old form still exists. Finally contract: delete the old form once no caller remains, in a ticket blocked by every migrate batch. When even the batches can't stay green alone, keep the sequence but let them share an integration branch that all block a final integrate-and-verify ticket — green is promised only there.
|
|
50
50
|
|
|
51
|
+
# Contract lineage and change impact
|
|
52
|
+
|
|
53
|
+
When a ticket comes from an authoritative spec, it must carry the exact source requirement identity and revision. A ticket may refine implementation scope, but it never silently rewrites the approved contract.
|
|
54
|
+
|
|
55
|
+
Every ticket must classify its impact:
|
|
56
|
+
|
|
57
|
+
- **none** — bookkeeping/non-behavioral work; explain why no contract changes.
|
|
58
|
+
- **implementation** — implementation changes while approved behavior and architectural rationale stay unchanged.
|
|
59
|
+
- **behavior** — caller/user-visible behavior changes. Update the living spec requirement and revision first, then obtain requirement-linked RED evidence before accepting implementation.
|
|
60
|
+
- **architecture** — a hard-to-reverse decision/rationale changes. Publish a new/superseding ADR, update the living spec requirement/revision, then obtain RED evidence.
|
|
61
|
+
|
|
62
|
+
If the impact classification changes during implementation, stop treating the original ticket as completion authority. Reconcile the source artifacts first, then update the ticket's source revision.
|
|
63
|
+
|
|
51
64
|
# Ticket Templates
|
|
52
65
|
|
|
53
66
|
<local-ticket-template>
|
|
@@ -58,9 +71,15 @@ Give each ticket its **blocking edges** — the other tickets that must complete
|
|
|
58
71
|
|
|
59
72
|
**Blocked by:** the numbers/titles of the tickets that gate this one, or "None — can start immediately".
|
|
60
73
|
|
|
74
|
+
**Source requirements:** `REQ-001@rev1` (add every requirement this ticket changes or proves).
|
|
75
|
+
|
|
76
|
+
**Change impact:** `none | implementation | behavior | architecture`
|
|
77
|
+
|
|
78
|
+
**Impact rationale:** one sentence explaining why this classification is correct.
|
|
79
|
+
|
|
61
80
|
**Status:** ready-for-agent
|
|
62
81
|
|
|
63
|
-
- [ ] Acceptance criterion 1
|
|
82
|
+
- [ ] [REQ-001] Acceptance criterion 1
|
|
64
83
|
- [ ] Acceptance criterion 2
|
|
65
84
|
|
|
66
85
|
</local-ticket-template>
|
|
@@ -4,7 +4,7 @@ description: "Run objective verification gates (build, types, lint, tests, secur
|
|
|
4
4
|
license: Apache-2.0
|
|
5
5
|
metadata:
|
|
6
6
|
author: Miya Daniel
|
|
7
|
-
version: 0.
|
|
7
|
+
version: 0.12.0
|
|
8
8
|
---
|
|
9
9
|
|
|
10
10
|
# Verification Loop
|
|
@@ -41,6 +41,7 @@ and rerun from the first affected gate.
|
|
|
41
41
|
4. Tests and coverage: `npm run test -- --coverage`.
|
|
42
42
|
5. Security scan: secrets and stray logs.
|
|
43
43
|
6. Diff review: `git diff --stat` and unintended-change checks.
|
|
44
|
-
7.
|
|
44
|
+
7. Contract reconciliation when the work cites `REQ-*` / ADR/spec/ticket lineage or a contract-integrity trace exists: run the strict `<this-skill-dir>/scripts/contract-integrity-audit.js` gate with current #58 evidence. `AUDIT`, `NOT_EVALUATED`, stale source binding, or drift is **not ready** for delivery.
|
|
45
|
+
8. Fill `<this-skill-dir>/templates/verification-report.template.md` with evidence.
|
|
45
46
|
|
|
46
47
|
Deep dive: `<this-skill-dir>/references/verification-phases.md`
|
package/plugins/harness-everything/skills/verification-loop/references/verification-phases.md
CHANGED
|
@@ -10,8 +10,14 @@ flowchart TD
|
|
|
10
10
|
Lint --> Test[4. Phase 4: Test Suite Execution]
|
|
11
11
|
Test --> Security[5. Phase 5: Security & Secret Scan]
|
|
12
12
|
Security --> Diff[6. Phase 6: Diff Review]
|
|
13
|
+
Diff --> Contract{7. Contract lineage applicable?}
|
|
14
|
+
Contract -- Yes --> Reconcile[Run contract-integrity strict reconciliation]
|
|
15
|
+
Contract -- No --> CheckScript
|
|
16
|
+
Reconcile --> ContractResult{PASS?}
|
|
17
|
+
ContractResult -- No --> Fix
|
|
18
|
+
ContractResult -- Yes --> CheckScript
|
|
13
19
|
|
|
14
|
-
Diff --> CheckScript{
|
|
20
|
+
Diff --> CheckScript{8. Is verify-gate.js / Template Available?}
|
|
15
21
|
|
|
16
22
|
CheckScript -- Yes --> RunScript[Run harness-everything/scripts/verify-gate.js / Fill verification-report.template.md]
|
|
17
23
|
CheckScript -- No --> DirectReport[Output Verification Report Inline in Response]
|
|
@@ -93,6 +99,24 @@ Review each changed file for:
|
|
|
93
99
|
- Missing error handling
|
|
94
100
|
- Potential edge cases
|
|
95
101
|
|
|
102
|
+
## Phase 7: Contract Reconciliation
|
|
103
|
+
|
|
104
|
+
This phase is applicable when any of the following is true:
|
|
105
|
+
|
|
106
|
+
- the task/spec/ticket cites stable `REQ-*` identities;
|
|
107
|
+
- an ADR/spec/ticket change is classified as `behavior` or `architecture`;
|
|
108
|
+
- the project has a contract-integrity trace for the affected scope.
|
|
109
|
+
|
|
110
|
+
Use the current trace plus current #58 TDD evidence:
|
|
111
|
+
|
|
112
|
+
```bash
|
|
113
|
+
node "<this-skill-dir>/scripts/contract-integrity-audit.js" <trace.json> --tdd-evidence <tdd-evidence.json> --output <report.json> --markdown <report.md>
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
Delivery requires strict **PASS**. Treat `FAIL`, `AUDIT`, `NOT_EVALUATED`, source conflicts/defects, stale spec/test/implementation lineage, surviving required probes, and audit-only output as **NOT READY**. Do not downgrade the gate merely because build/tests are otherwise green.
|
|
117
|
+
|
|
118
|
+
If this phase is not applicable, record `NOT_APPLICABLE` with a one-line reason. Never use `NOT_APPLICABLE` to hide known requirement lineage.
|
|
119
|
+
|
|
96
120
|
## Output Format
|
|
97
121
|
|
|
98
122
|
After running all phases, fill in `verification-loop/templates/verification-report.template.md` with the actual results and present it.
|
package/plugins/harness-everything/skills/verification-loop/scripts/contract-integrity-audit.js
ADDED
|
@@ -0,0 +1,31 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
'use strict';
|
|
3
|
+
|
|
4
|
+
const fs = require('fs');
|
|
5
|
+
const path = require('path');
|
|
6
|
+
|
|
7
|
+
function resolveAuditRuntime() {
|
|
8
|
+
const candidates = [
|
|
9
|
+
path.resolve(__dirname, '../../contract-integrity/scripts/audit.js'),
|
|
10
|
+
path.resolve(__dirname, '../../../contract-integrity/scripts/audit.js'),
|
|
11
|
+
];
|
|
12
|
+
for (const candidate of candidates) {
|
|
13
|
+
if (fs.existsSync(candidate)) return candidate;
|
|
14
|
+
}
|
|
15
|
+
throw new Error('contract-integrity audit runtime is not packaged with this verification-loop installation');
|
|
16
|
+
}
|
|
17
|
+
|
|
18
|
+
function main() {
|
|
19
|
+
try {
|
|
20
|
+
const target = resolveAuditRuntime();
|
|
21
|
+
const { runCli } = require(target);
|
|
22
|
+
if (typeof runCli !== 'function') throw new Error('contract-integrity audit runtime does not export runCli');
|
|
23
|
+
process.exitCode = runCli([process.execPath, target, ...process.argv.slice(2)]);
|
|
24
|
+
} catch (error) {
|
|
25
|
+
console.error('[Verification Contract Gate] ' + error.message);
|
|
26
|
+
process.exitCode = 2;
|
|
27
|
+
}
|
|
28
|
+
}
|
|
29
|
+
|
|
30
|
+
if (require.main === module) main();
|
|
31
|
+
module.exports = { resolveAuditRuntime };
|
|
@@ -7,6 +7,9 @@ Lint: [PASS/FAIL] (X warnings)
|
|
|
7
7
|
Tests: [PASS/FAIL] (X/Y passed, Z% coverage)
|
|
8
8
|
Security: [PASS/FAIL] (X issues)
|
|
9
9
|
Diff: [X files changed]
|
|
10
|
+
Contract: [PASS/FAIL/NOT_APPLICABLE] (REQ IDs / trace report / reason)
|
|
11
|
+
Strict PASS required when contract lineage is applicable.
|
|
12
|
+
AUDIT or NOT_EVALUATED is NOT READY.
|
|
10
13
|
|
|
11
14
|
Overall: [READY/NOT READY] for PR
|
|
12
15
|
|
|
@@ -68,6 +68,18 @@ syncTree(path.join(ROOT, 'scripts', 'lib', 'workspace.js'), packageWorkspace);
|
|
|
68
68
|
const packageAdvisoryText = path.join(pluginRoot, 'scripts', 'lib', 'advisory-text.js');
|
|
69
69
|
syncTree(path.join(ROOT, 'scripts', 'lib', 'advisory-text.js'), packageAdvisoryText);
|
|
70
70
|
|
|
71
|
+
// Contract-integrity is shared runtime, not a directly-routed 27th skill.
|
|
72
|
+
// Package it under the plugin root and rewrite its one canonical TDD import
|
|
73
|
+
// to the packaged skill tree.
|
|
74
|
+
const packageContractAudit = path.join(pluginRoot, 'contract-integrity', 'scripts', 'audit.js');
|
|
75
|
+
syncTree(path.join(ROOT, 'contract-integrity', 'scripts', 'audit.js'), packageContractAudit);
|
|
76
|
+
const contractAuditSource = fs.readFileSync(packageContractAudit, 'utf8');
|
|
77
|
+
fs.writeFileSync(
|
|
78
|
+
packageContractAudit,
|
|
79
|
+
contractAuditSource.replace("require('../../tdd/scripts/quality-gate')", "require('../../skills/tdd/scripts/quality-gate')"),
|
|
80
|
+
'utf8'
|
|
81
|
+
);
|
|
82
|
+
|
|
71
83
|
// The source hook state module is shared with Claude and uses the repository
|
|
72
84
|
// layout. Rewrite only that import in the packaged copy; all other relative
|
|
73
85
|
// imports remain inside the package tree.
|
package/self-evolve/SKILL.md
CHANGED
|
@@ -6,6 +6,18 @@ const path = require('path');
|
|
|
6
6
|
const { spawnSync } = require('child_process');
|
|
7
7
|
const { contentHash, scoreRuleQuality, screenMemoryForPersistence } = require('./persist-memory');
|
|
8
8
|
|
|
9
|
+
let emitLessonTelemetry = () => ({ ok: false, unavailable: true });
|
|
10
|
+
for (const relative of [
|
|
11
|
+
'../../hooks/scripts/lib/telemetry.js',
|
|
12
|
+
'../../../hooks/scripts/lib/telemetry.js',
|
|
13
|
+
]) {
|
|
14
|
+
const candidate = path.resolve(__dirname, relative);
|
|
15
|
+
if (fs.existsSync(candidate)) {
|
|
16
|
+
({ emitLessonTelemetry } = require(candidate));
|
|
17
|
+
break;
|
|
18
|
+
}
|
|
19
|
+
}
|
|
20
|
+
|
|
9
21
|
let getWorkspaceRoot;
|
|
10
22
|
let getWorkspaceStateDir;
|
|
11
23
|
try {
|
|
@@ -69,6 +81,24 @@ function pushState(candidate, state, reasonCode, evidenceRef = null) {
|
|
|
69
81
|
}].slice(-32);
|
|
70
82
|
}
|
|
71
83
|
|
|
84
|
+
function emitCandidateLifecycle(input, candidate, event, state, status = 'unknown') {
|
|
85
|
+
const history = [...(candidate.history || [])].reverse().find(item => item.state === state);
|
|
86
|
+
const reasonCodes = [
|
|
87
|
+
history && history.reasonCode,
|
|
88
|
+
candidate.trigger && candidate.trigger.type ? `trigger:${candidate.trigger.type}` : null,
|
|
89
|
+
...(candidate.evaluation && Array.isArray(candidate.evaluation.reasonCodes) ? candidate.evaluation.reasonCodes : []),
|
|
90
|
+
].filter(Boolean);
|
|
91
|
+
return emitLessonTelemetry({
|
|
92
|
+
event,
|
|
93
|
+
root: input.workspace,
|
|
94
|
+
sessionId: input.sessionId || (candidate.provenance && candidate.provenance.sessionId),
|
|
95
|
+
candidateId: candidate.candidateId,
|
|
96
|
+
observedAt: history && history.observedAt,
|
|
97
|
+
status,
|
|
98
|
+
reasonCodes,
|
|
99
|
+
});
|
|
100
|
+
}
|
|
101
|
+
|
|
72
102
|
function replayability(candidate) {
|
|
73
103
|
if (candidate.trigger?.type === 'verifier-fail-pass' &&
|
|
74
104
|
candidate.evidence?.previousStatus === 'fail' &&
|
|
@@ -134,6 +164,8 @@ function evaluateCandidate(input) {
|
|
|
134
164
|
};
|
|
135
165
|
pushState(candidate, 'rejected', 'candidate-screening-or-quality-failed');
|
|
136
166
|
atomicWriteJson(file, candidate);
|
|
167
|
+
emitCandidateLifecycle(input, candidate, 'lesson_screened', 'screened', screening.allowed ? 'success' : 'failure');
|
|
168
|
+
emitCandidateLifecycle(input, candidate, 'lesson_rejected', 'rejected', 'failure');
|
|
137
169
|
return { file, candidate };
|
|
138
170
|
}
|
|
139
171
|
|
|
@@ -150,6 +182,15 @@ function evaluateCandidate(input) {
|
|
|
150
182
|
pushState(candidate, 'evaluated', replay.reasonCode);
|
|
151
183
|
pushState(candidate, disposition, disposition === 'accepted' ? 'eligible-for-governed-promotion' : 'paired-or-later-evidence-required');
|
|
152
184
|
atomicWriteJson(file, candidate);
|
|
185
|
+
emitCandidateLifecycle(input, candidate, 'lesson_screened', 'screened', 'success');
|
|
186
|
+
emitCandidateLifecycle(input, candidate, 'lesson_evaluated', 'evaluated', 'success');
|
|
187
|
+
emitCandidateLifecycle(
|
|
188
|
+
input,
|
|
189
|
+
candidate,
|
|
190
|
+
disposition === 'accepted' ? 'lesson_accepted' : 'lesson_inconclusive',
|
|
191
|
+
disposition,
|
|
192
|
+
disposition === 'accepted' ? 'success' : 'unknown',
|
|
193
|
+
);
|
|
153
194
|
return { file, candidate };
|
|
154
195
|
}
|
|
155
196
|
|
|
@@ -212,6 +253,7 @@ function promoteCandidate(input) {
|
|
|
212
253
|
if (persisted) pushState(candidate, 'persisted', 'accepted-and-authorized-memory-write');
|
|
213
254
|
else pushState(candidate, 'accepted', 'memory-governance-held-as-review-candidate');
|
|
214
255
|
atomicWriteJson(file, candidate);
|
|
256
|
+
if (persisted) emitCandidateLifecycle(input, candidate, 'lesson_persisted', 'persisted', 'success');
|
|
215
257
|
return { ok: true, exitCode: 0, persisted, stdout: result.stdout, stderr: result.stderr, candidate };
|
|
216
258
|
}
|
|
217
259
|
|
|
@@ -234,6 +276,16 @@ function observeCandidate(input) {
|
|
|
234
276
|
};
|
|
235
277
|
pushState(candidate, next, `lesson-${next}`, evidenceRef);
|
|
236
278
|
atomicWriteJson(file, candidate);
|
|
279
|
+
// Retrieval telemetry is emitted by the actual scoped retrieval path so the
|
|
280
|
+
// funnel measures exposure, not an operator-authored lifecycle annotation.
|
|
281
|
+
if (next !== 'retrieved') {
|
|
282
|
+
const event = {
|
|
283
|
+
validated: 'lesson_validated',
|
|
284
|
+
regressed: 'lesson_regressed',
|
|
285
|
+
superseded: 'lesson_superseded',
|
|
286
|
+
}[next];
|
|
287
|
+
if (event) emitCandidateLifecycle(input, candidate, event, next, next === 'regressed' ? 'failure' : 'success');
|
|
288
|
+
}
|
|
237
289
|
return { file, candidate };
|
|
238
290
|
}
|
|
239
291
|
|
|
@@ -4,36 +4,155 @@
|
|
|
4
4
|
"title": "Harness Telemetry Event",
|
|
5
5
|
"type": "object",
|
|
6
6
|
"required": [
|
|
7
|
-
"schemaVersion",
|
|
8
|
-
"
|
|
9
|
-
"
|
|
7
|
+
"schemaVersion",
|
|
8
|
+
"eventId",
|
|
9
|
+
"event",
|
|
10
|
+
"host",
|
|
11
|
+
"observedAt",
|
|
12
|
+
"sessionId",
|
|
13
|
+
"turnId",
|
|
14
|
+
"agentId",
|
|
15
|
+
"invocationId",
|
|
16
|
+
"skillName",
|
|
17
|
+
"skillVersion",
|
|
18
|
+
"toolName",
|
|
19
|
+
"status",
|
|
20
|
+
"retryCount",
|
|
21
|
+
"timing",
|
|
22
|
+
"reasonCodes"
|
|
10
23
|
],
|
|
11
24
|
"properties": {
|
|
12
|
-
"schemaVersion": {
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
"
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
"
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
25
|
+
"schemaVersion": {
|
|
26
|
+
"const": 1
|
|
27
|
+
},
|
|
28
|
+
"eventId": {
|
|
29
|
+
"type": "string",
|
|
30
|
+
"minLength": 1
|
|
31
|
+
},
|
|
32
|
+
"event": {
|
|
33
|
+
"enum": [
|
|
34
|
+
"skill.invoke",
|
|
35
|
+
"skill.loaded",
|
|
36
|
+
"skill.complete",
|
|
37
|
+
"tool.observed",
|
|
38
|
+
"learning_opportunity",
|
|
39
|
+
"lesson_proposed",
|
|
40
|
+
"lesson_screened",
|
|
41
|
+
"lesson_evaluated",
|
|
42
|
+
"lesson_accepted",
|
|
43
|
+
"lesson_rejected",
|
|
44
|
+
"lesson_inconclusive",
|
|
45
|
+
"lesson_persisted",
|
|
46
|
+
"lesson_retrieved",
|
|
47
|
+
"lesson_validated",
|
|
48
|
+
"lesson_regressed",
|
|
49
|
+
"lesson_superseded"
|
|
50
|
+
]
|
|
51
|
+
},
|
|
52
|
+
"host": {
|
|
53
|
+
"enum": [
|
|
54
|
+
"claude",
|
|
55
|
+
"codex",
|
|
56
|
+
"opencode",
|
|
57
|
+
"unknown"
|
|
58
|
+
]
|
|
59
|
+
},
|
|
60
|
+
"observedAt": {
|
|
61
|
+
"type": "string",
|
|
62
|
+
"format": "date-time"
|
|
63
|
+
},
|
|
64
|
+
"sessionId": {
|
|
65
|
+
"type": [
|
|
66
|
+
"string",
|
|
67
|
+
"null"
|
|
68
|
+
]
|
|
69
|
+
},
|
|
70
|
+
"turnId": {
|
|
71
|
+
"type": [
|
|
72
|
+
"string",
|
|
73
|
+
"null"
|
|
74
|
+
]
|
|
75
|
+
},
|
|
76
|
+
"agentId": {
|
|
77
|
+
"type": [
|
|
78
|
+
"string",
|
|
79
|
+
"null"
|
|
80
|
+
]
|
|
81
|
+
},
|
|
82
|
+
"invocationId": {
|
|
83
|
+
"type": [
|
|
84
|
+
"string",
|
|
85
|
+
"null"
|
|
86
|
+
]
|
|
87
|
+
},
|
|
88
|
+
"skillName": {
|
|
89
|
+
"type": [
|
|
90
|
+
"string",
|
|
91
|
+
"null"
|
|
92
|
+
]
|
|
93
|
+
},
|
|
94
|
+
"skillVersion": {
|
|
95
|
+
"type": [
|
|
96
|
+
"string",
|
|
97
|
+
"null"
|
|
98
|
+
]
|
|
99
|
+
},
|
|
100
|
+
"toolName": {
|
|
101
|
+
"type": [
|
|
102
|
+
"string",
|
|
103
|
+
"null"
|
|
104
|
+
]
|
|
105
|
+
},
|
|
106
|
+
"status": {
|
|
107
|
+
"enum": [
|
|
108
|
+
"unknown",
|
|
109
|
+
"success",
|
|
110
|
+
"failure",
|
|
111
|
+
"aborted"
|
|
112
|
+
]
|
|
113
|
+
},
|
|
114
|
+
"retryCount": {
|
|
115
|
+
"type": "integer",
|
|
116
|
+
"minimum": 0
|
|
117
|
+
},
|
|
26
118
|
"timing": {
|
|
27
119
|
"type": "object",
|
|
28
|
-
"required": [
|
|
120
|
+
"required": [
|
|
121
|
+
"skillLoadDurationMs",
|
|
122
|
+
"activeWindowMs",
|
|
123
|
+
"attributedToolDurationMs"
|
|
124
|
+
],
|
|
29
125
|
"properties": {
|
|
30
|
-
"skillLoadDurationMs": {
|
|
31
|
-
|
|
32
|
-
|
|
126
|
+
"skillLoadDurationMs": {
|
|
127
|
+
"type": [
|
|
128
|
+
"number",
|
|
129
|
+
"null"
|
|
130
|
+
],
|
|
131
|
+
"minimum": 0
|
|
132
|
+
},
|
|
133
|
+
"activeWindowMs": {
|
|
134
|
+
"type": [
|
|
135
|
+
"number",
|
|
136
|
+
"null"
|
|
137
|
+
],
|
|
138
|
+
"minimum": 0
|
|
139
|
+
},
|
|
140
|
+
"attributedToolDurationMs": {
|
|
141
|
+
"type": [
|
|
142
|
+
"number",
|
|
143
|
+
"null"
|
|
144
|
+
],
|
|
145
|
+
"minimum": 0
|
|
146
|
+
}
|
|
33
147
|
},
|
|
34
148
|
"additionalProperties": false
|
|
35
149
|
},
|
|
36
|
-
"reasonCodes": {
|
|
150
|
+
"reasonCodes": {
|
|
151
|
+
"type": "array",
|
|
152
|
+
"items": {
|
|
153
|
+
"type": "string"
|
|
154
|
+
}
|
|
155
|
+
}
|
|
37
156
|
},
|
|
38
157
|
"additionalProperties": false
|
|
39
158
|
}
|
|
@@ -4,6 +4,7 @@
|
|
|
4
4
|
const fs = require('fs');
|
|
5
5
|
const path = require('path');
|
|
6
6
|
const {
|
|
7
|
+
LESSON_EVENT_TYPES,
|
|
7
8
|
readEvents,
|
|
8
9
|
telemetryFile,
|
|
9
10
|
} = require('../../hooks/scripts/lib/telemetry');
|
|
@@ -28,8 +29,20 @@ function buildReport(events) {
|
|
|
28
29
|
const valid = events.filter(event => event && !event.invalid);
|
|
29
30
|
const invalid = events.filter(event => event && event.invalid);
|
|
30
31
|
const groups = new Map();
|
|
32
|
+
const lessonCandidateIds = new Set();
|
|
33
|
+
const lessonEvents = {};
|
|
34
|
+
const lessonCandidatesByEvent = new Map();
|
|
31
35
|
|
|
32
36
|
for (const event of valid) {
|
|
37
|
+
if (LESSON_EVENT_TYPES.has(event.event)) {
|
|
38
|
+
lessonEvents[event.event] = (lessonEvents[event.event] || 0) + 1;
|
|
39
|
+
if (event.invocationId) {
|
|
40
|
+
lessonCandidateIds.add(event.invocationId);
|
|
41
|
+
if (!lessonCandidatesByEvent.has(event.event)) lessonCandidatesByEvent.set(event.event, new Set());
|
|
42
|
+
lessonCandidatesByEvent.get(event.event).add(event.invocationId);
|
|
43
|
+
}
|
|
44
|
+
continue;
|
|
45
|
+
}
|
|
33
46
|
const key = `${event.host}::${event.skillName || '(unknown)'}`;
|
|
34
47
|
if (!groups.has(key)) groups.set(key, {
|
|
35
48
|
host: event.host,
|
|
@@ -80,11 +93,34 @@ function buildReport(events) {
|
|
|
80
93
|
attributedToolDurationMs: metric(g.attributedToolDurations),
|
|
81
94
|
})).sort((a, b) => `${a.host}/${a.skillName}`.localeCompare(`${b.host}/${b.skillName}`));
|
|
82
95
|
|
|
96
|
+
const uniqueLessonCount = event => lessonCandidatesByEvent.get(event)?.size || 0;
|
|
97
|
+
const selfEvolve = {
|
|
98
|
+
candidateCount: lessonCandidateIds.size,
|
|
99
|
+
eventCount: Object.values(lessonEvents).reduce((sum, value) => sum + value, 0),
|
|
100
|
+
funnel: {
|
|
101
|
+
opportunities: uniqueLessonCount('learning_opportunity'),
|
|
102
|
+
proposed: uniqueLessonCount('lesson_proposed'),
|
|
103
|
+
screened: uniqueLessonCount('lesson_screened'),
|
|
104
|
+
evaluated: uniqueLessonCount('lesson_evaluated'),
|
|
105
|
+
accepted: uniqueLessonCount('lesson_accepted'),
|
|
106
|
+
rejected: uniqueLessonCount('lesson_rejected'),
|
|
107
|
+
inconclusive: uniqueLessonCount('lesson_inconclusive'),
|
|
108
|
+
persisted: uniqueLessonCount('lesson_persisted'),
|
|
109
|
+
retrieved: uniqueLessonCount('lesson_retrieved'),
|
|
110
|
+
validated: uniqueLessonCount('lesson_validated'),
|
|
111
|
+
regressed: uniqueLessonCount('lesson_regressed'),
|
|
112
|
+
superseded: uniqueLessonCount('lesson_superseded'),
|
|
113
|
+
},
|
|
114
|
+
events: lessonEvents,
|
|
115
|
+
semantics: 'Counts are privacy-safe lifecycle observations. Acceptance/persistence/retrieval counts do not by themselves prove behavioral improvement.',
|
|
116
|
+
};
|
|
117
|
+
|
|
83
118
|
return {
|
|
84
119
|
schemaVersion: 1,
|
|
85
120
|
eventCount: valid.length,
|
|
86
121
|
invalidEventCount: invalid.length,
|
|
87
122
|
skills,
|
|
123
|
+
selfEvolve,
|
|
88
124
|
semantics: {
|
|
89
125
|
skillLoadDurationMs: 'PreToolUse(Skill) to correlated PostToolUse(Skill); framework/load latency only.',
|
|
90
126
|
activeWindowMs: 'Successful skill load to host Stop/turn boundary; context exposure window, not CPU time.',
|
|
@@ -104,6 +140,12 @@ function markdown(report) {
|
|
|
104
140
|
const pair = value => value.count ? `${value.p50}/${value.p95}` : '-';
|
|
105
141
|
lines.push(`| ${row.host} | ${row.skillName || '(unknown)'} | ${row.invocationCount} | ${row.completeEvents} | ${row.status.failure} | ${pair(row.skillLoadDurationMs)} | ${pair(row.activeWindowMs)} | ${pair(row.attributedToolDurationMs)} |`);
|
|
106
142
|
}
|
|
143
|
+
lines.push('', '## Self-Evolve Lifecycle Funnel', '');
|
|
144
|
+
const f = report.selfEvolve.funnel;
|
|
145
|
+
lines.push('| Opportunity | Proposed | Accepted | Rejected | Inconclusive | Persisted | Retrieved | Validated | Regressed |');
|
|
146
|
+
lines.push('| ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |');
|
|
147
|
+
lines.push(`| ${f.opportunities} | ${f.proposed} | ${f.accepted} | ${f.rejected} | ${f.inconclusive} | ${f.persisted} | ${f.retrieved} | ${f.validated} | ${f.regressed} |`);
|
|
148
|
+
lines.push('', '> Lesson funnel counts are lifecycle evidence, not proof that a persisted lesson improved behavior.');
|
|
107
149
|
lines.push('', `Events: ${report.eventCount}; invalid lines: ${report.invalidEventCount}.`);
|
|
108
150
|
lines.push('', '> Active window is not CPU/runtime. Attributed tool time is observational and non-exclusive when skills overlap.');
|
|
109
151
|
return `${lines.join('\n')}\n`;
|
package/to-spec/SKILL.md
CHANGED
|
@@ -60,6 +60,32 @@ What "confirm" means depends on the shape picked:
|
|
|
60
60
|
|
|
61
61
|
Skip this step only if the conversation already pinned down the relevant surface explicitly (e.g. a prior `grill-with-docs` pass already fixed the schema).
|
|
62
62
|
|
|
63
|
+
## Stable contract identity
|
|
64
|
+
|
|
65
|
+
When the published artifact is an authoritative behavior contract, every normative requirement must have a stable identity instead of relying on prose position.
|
|
66
|
+
|
|
67
|
+
Use this compact shape in the selected template:
|
|
68
|
+
|
|
69
|
+
```md
|
|
70
|
+
## Contract Requirements
|
|
71
|
+
|
|
72
|
+
### REQ-001 — <short behavior name>
|
|
73
|
+
**Status:** CURRENT
|
|
74
|
+
**Revision:** 1
|
|
75
|
+
**Decision lineage:** ADR-0001 (or `None`)
|
|
76
|
+
<normative behavior and observable acceptance boundary>
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
Rules:
|
|
80
|
+
|
|
81
|
+
- Never recycle a `REQ-*` ID for different behavior.
|
|
82
|
+
- Behavior changes increment the requirement revision and update the living spec **before** implementation is accepted.
|
|
83
|
+
- Superseded behavior remains traceable; do not erase the old requirement history merely because a ticket changed.
|
|
84
|
+
- If a ticket is classified `changeImpact: architecture`, publish the new/superseding ADR first and link it from the affected requirement.
|
|
85
|
+
- If the document is informative rather than authoritative, say so explicitly and link the authoritative requirement instead of inventing a parallel source of truth.
|
|
86
|
+
|
|
87
|
+
The detailed ADR -> spec -> ticket -> test -> implementation lineage belongs in the structured contract trace; keep the human spec concise.
|
|
88
|
+
|
|
63
89
|
## Publishing details
|
|
64
90
|
|
|
65
91
|
Write the doc using the adapted template at the resolved storage path:
|
|
@@ -28,4 +28,18 @@ Every distinct exit code or error response: what triggers it, what the caller sh
|
|
|
28
28
|
|
|
29
29
|
Other commands/endpoints this one is commonly paired with, supersedes, or is superseded by.
|
|
30
30
|
|
|
31
|
+
## Contract Requirements
|
|
32
|
+
|
|
33
|
+
For every normative behavior this document owns, assign a stable requirement identity:
|
|
34
|
+
|
|
35
|
+
```md
|
|
36
|
+
### REQ-001 — <short behavior name>
|
|
37
|
+
**Status:** CURRENT
|
|
38
|
+
**Revision:** 1
|
|
39
|
+
**Decision lineage:** ADR-0001 (or None)
|
|
40
|
+
<observable command/API behavior or error contract>
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
Do not recycle requirement IDs. A changed caller-visible contract increments the revision before implementation completion.
|
|
44
|
+
|
|
31
45
|
</spec-template>
|
|
@@ -24,4 +24,18 @@ What this decision costs, even though it was the right call.
|
|
|
24
24
|
|
|
25
25
|
How this ships (flag, gradual, all-at-once) and how to undo it if it's wrong.
|
|
26
26
|
|
|
27
|
+
## Contract Requirements
|
|
28
|
+
|
|
29
|
+
If this document owns normative behavior, assign stable requirement identities:
|
|
30
|
+
|
|
31
|
+
```md
|
|
32
|
+
### REQ-001 — <short behavior name>
|
|
33
|
+
**Status:** CURRENT
|
|
34
|
+
**Revision:** 1
|
|
35
|
+
**Decision lineage:** ADR-0001 (or None)
|
|
36
|
+
<observable behavior / acceptance boundary>
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
If this document is informative only, say so and link the authoritative requirement instead. Do not create a second source of truth.
|
|
40
|
+
|
|
27
41
|
</spec-template>
|