@holmes-lab/holmes-kit 0.8.1 → 0.9.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +36 -0
- package/README.md +3 -2
- package/dist/.build-id +1 -1
- package/dist/holmes/cli/doctor.js +12 -0
- package/dist/holmes/cli/mcp-schema-cost.d.ts +18 -0
- package/dist/holmes/cli/mcp-schema-cost.js +28 -0
- package/dist/holmes/config/config.d.ts +8 -0
- package/dist/holmes/config/config.js +1 -1
- package/dist/holmes/governance/constitution.d.ts +26 -0
- package/dist/holmes/governance/constitution.js +33 -0
- package/dist/holmes/hooks/stop.d.ts +24 -0
- package/dist/holmes/hooks/stop.js +83 -3
- package/dist/holmes/mcp/handlers.d.ts +19 -0
- package/dist/holmes/mcp/handlers.js +86 -3
- package/dist/holmes/mcp/tool-schemas.js +1 -0
- package/dist/holmes/review/mutate.d.ts +17 -0
- package/dist/holmes/review/mutate.js +66 -0
- package/dist/holmes/review/test-outcomes.d.ts +35 -0
- package/dist/holmes/review/test-outcomes.js +108 -0
- package/dist/holmes/review/test-runner.d.ts +30 -0
- package/dist/holmes/review/test-runner.js +71 -5
- package/dist/holmes/spec/kills.d.ts +14 -0
- package/dist/holmes/spec/kills.js +28 -0
- package/dist/holmes/spec/spec-store.d.ts +9 -0
- package/dist/holmes/spec/spec-store.js +17 -0
- package/dist/holmes/spec/validator.js +18 -0
- package/package.json +1 -1
- package/playbooks/tdd-slice/PLAYBOOK.md +82 -0
package/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,42 @@ All notable changes to this project will be documented in this file.
|
|
|
5
5
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
|
6
6
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
7
|
<!-- @implements A-SPEC-209 -->
|
|
8
|
+
## [0.9.0] - 2026-09-04
|
|
9
|
+
|
|
10
|
+
Internalized TDD discipline — RED-first evidence enforced by a new constitution article (ART-8),
|
|
11
|
+
not a prompt — plus a doctor transparency check and a BUG-1 authoring fix. All additive and
|
|
12
|
+
backward-compatible; existing behaviour is unchanged at the default settings.
|
|
13
|
+
|
|
14
|
+
### Added
|
|
15
|
+
|
|
16
|
+
- **Inbuilt TDD, made enforceable** (REQ-534): the superpowers TDD discipline is now a holmes-installed
|
|
17
|
+
skill AND deterministically enforced, not merely advised.
|
|
18
|
+
- **ART-8 RED-first evidence** (new constitution article): a changed A-SPEC must show a recorded
|
|
19
|
+
`red-assertion → green` sequence in the ledger. A `red-error` (a test that could not run — a
|
|
20
|
+
load/import/collection failure) is **not** a valid RED, so "the covering test failed *correctly*"
|
|
21
|
+
is judged mechanically. `test_run` classifies each covered file as `red-assertion | red-error |
|
|
22
|
+
green` and appends per-A-SPEC outcomes; the Stop hook reads them. Ships at
|
|
23
|
+
`guardrail.redFirstEvidence: track` (observe-first, non-blocking) — promote to `strict` per repo,
|
|
24
|
+
`off` disables. Evidence-gated (inert until outcomes are recorded) and jest-only for now.
|
|
25
|
+
- **`holmes-tdd-slice` skill**: restates the Iron Law and Red-Green-Refactor in holmes terms and
|
|
26
|
+
tags each rule with the article that enforces it (`[ART-1]`/`[ART-4]`/`[ART-8]`/honor-system).
|
|
27
|
+
Installed alongside the other recovery skills.
|
|
28
|
+
- **Discriminating power**: a T-SPEC may declare `kills:` (named mutations), and
|
|
29
|
+
`test_run --mutate <tspec>` applies each against the A-SPEC's source, reporting which SURVIVED
|
|
30
|
+
(a coverage gap). Selective and opt-in — never a blanket gate.
|
|
31
|
+
- **doctor `mcp schema cost`** (REQ-535): reports holmes-kit's own advertised MCP schema cost
|
|
32
|
+
(computed live from the tool set, no hardcoded number) and WARNs when `HOLMES_MCP_PROFILE=full`
|
|
33
|
+
re-advertises the hook-enforced gate-duplicate tools; advises client-side deferred loading.
|
|
34
|
+
Advisory only (never FAIL), and it never claims to have detected a client's resident behaviour.
|
|
35
|
+
|
|
36
|
+
### Fixed
|
|
37
|
+
|
|
38
|
+
- **BUG-1 remainder** (REQ-536): `spec_slice_init` now safe-quotes the `slice` value with `yamlScalar`
|
|
39
|
+
— it was the one raw interpolation among the slice builders, so a `sliceName` containing a colon or
|
|
40
|
+
quote could break the generated A-SPEC's frontmatter YAML. And `spec_approve` on a missing id now
|
|
41
|
+
surfaces any unparseable spec file (with a YAML hint) instead of a bare "not found", closing the
|
|
42
|
+
silent-loss that hid a broken file behind a "not found".
|
|
43
|
+
|
|
8
44
|
## [0.8.1] - 2026-09-03
|
|
9
45
|
|
|
10
46
|
Codex hard-enforcement completed and verified on real codex-cli 0.152.1 (GOAL-codex-enforcement),
|
package/README.md
CHANGED
|
@@ -14,9 +14,10 @@
|
|
|
14
14
|
|
|
15
15
|
---
|
|
16
16
|
|
|
17
|
-
### 🛡️ Currently Supported Features (v0.
|
|
17
|
+
### 🛡️ Currently Supported Features (v0.9.x Production Features)
|
|
18
18
|
|
|
19
19
|
- 📋 **Requirements & Specification Governance**: Strict **"No Spec, No Code"** enforcement with 4-tier spec chain traceability (`REQ ➔ H-SPEC ➔ A-SPEC ➔ T-SPEC`) and `// @implements A-SPEC-XXX` code anchors (comma-lists and every anchor in a file participate in the gate).
|
|
20
|
+
- 🔴 **Inbuilt TDD — RED-first, enforced not asked** *(new in 0.9.0)*: the test-first discipline is a holmes-installed `holmes-tdd-slice` skill **and** a new constitution article **ART-8**. A changed A-SPEC must show a recorded `red-assertion → green` sequence in the ledger; a `red-error` (a test that could not run) is not a valid RED, so "the covering test failed *correctly*" is judged mechanically, not on trust. `test_run` classifies each covered file (`red-assertion`/`red-error`/`green`) and records per-A-SPEC outcomes the Stop hook reads. Ships at `redFirstEvidence: track` (observe-first, non-blocking; `strict`/`off` per repo), evidence-gated and jest-only for now. A T-SPEC may also declare `kills:` mutations and `test_run --mutate` reports which SURVIVED (a coverage gap). Where superpowers *asks* for RED-first and discriminating power, holmes-kit *proves* them.
|
|
20
21
|
- 🤖 **Autonomous Approval** *(new in 0.8.0)*: with the out-of-band `HOLMES_AUTONOMOUS_APPROVAL` switch on, an agent seals **low/mid-risk** specs itself (ledgered under an `autonomous:<client>` actor) — while `gate-behavior` changes, architecture/gate/taint files, and every upstream `REQ`/`H-SPEC`/`C-SPEC` still ask a human through the in-session TUI. The switch is env-only; a session cannot set it (blocked like `HOLMES_ROLE`). Off = byte-identical to before.
|
|
21
22
|
- 🪧 **Session Banner + Update Notice** *(new in 0.8.0)*: every session start emits an English intro (version + governance rule + npm URL) to both the human transcript and the agent context (SessionStart hook + MCP `instructions`); when a newer published version is cached, an install-mode-aware update command is appended. Registry check is detached, fail-silent, and opts out via `HOLMES_NO_UPDATE_CHECK`/`CI`.
|
|
22
23
|
- 🧱 **Deterministic Gate, Hardened** *(new in 0.8.0)*: shell writes are judged at the segment's **effective working directory** (`cd sub && cat > ../src/x.ts` is sealed, legitimate out-of-tree scratch writes are freed); the governing anchor is the **whole set**, not the first match. Every gate change ships with two consecutive clean adversarial rounds.
|
|
@@ -24,7 +25,7 @@
|
|
|
24
25
|
- 🎯 **Graded Impact Surface** *(new in 0.3.0)*: `rankedImpact` (personalized-PageRank over the spec/code graph) beat its pre-registered naive baseline on **both recall and precision across 3 corpora (×1.6–×17)** — the necessary condition for any better-than-a-person phrasing, measured before claimed.
|
|
25
26
|
- 🐞 **Causal Defect Localization & CPG** *(equalized in 0.5–0.7)*: AST Code Property Graph (CFG/DDG/CDG) & Dataflow Taint reachability across 7 languages (TS/JS, Python, Go, Rust, Java, C/C++, C#) — **42 language×layer cells graded on measured evidence** (11 corpora, 39,344 functions, zero invariant violations; C++ conditional on 67.9% parse coverage, disclosed in the matrix).
|
|
26
27
|
- 📏 **Measured, Not Claimed** *(new in 0.3.x)*: performance is judged against a pre-registered modeled-human band (R 0.67–0.78 / P ≈0.9±). Current official grade: **band entry on recall; division-of-labor precision 0.727 = 81% of the modeled human — reproduced by an independent context-free judge on a fresh blind window.** No superhuman claims until both metrics exceed the band.
|
|
27
|
-
- 🧪 **Self-Healing & Diagnostic Doctor**: Automated integrity checks and self-healing auto-fix remediation (`holmes-kit doctor --fix` & `spec_remediate`) — wiring-handshake checks run on Windows natively as of 0.3.2.
|
|
28
|
+
- 🧪 **Self-Healing & Diagnostic Doctor**: Automated integrity checks and self-healing auto-fix remediation (`holmes-kit doctor --fix` & `spec_remediate`) — wiring-handshake checks run on Windows natively as of 0.3.2. As of 0.9.0, doctor also reports holmes-kit's own advertised **MCP schema token cost** (computed live) and warns when `HOLMES_MCP_PROFILE=full` needlessly re-advertises the hook-enforced gate-duplicate tools.
|
|
28
29
|
- 🔔 **Approval UX** *(new in 0.3.1)*: in-session approval dialogs forewarn their 120s deadline and, on expiry, the refusal says exactly where the decision went (`npx holmes-kit approve` out-of-band queue) — no more silently dead dialogs.
|
|
29
30
|
- 🚦 **Push & Server-Side Re-Validation** *(hardened in 0.8.0)*: a local `pre-push` evidence gate (test-run ledger head == push HEAD, green, executed > 0) plus a **server-side CI workflow** that re-runs `npm ci → build → full suite → tarball install probe`, so a `--no-verify` push or a hook-less clone is still caught.
|
|
30
31
|
- 📊 **Automated RTM & Taint Heatmap**: Interactive standalone HTML/SVG report generation (`generateRtmHeatmap`) for spec coverage and security dataflow reachability.
|
package/dist/.build-id
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
|
|
1
|
+
2a5de17-mtm0zp6n
|
|
@@ -63,6 +63,8 @@ const mcp_version_1 = require("./mcp-version");
|
|
|
63
63
|
const codex_toml_1 = require("./codex-toml");
|
|
64
64
|
const agents_1 = require("./agents");
|
|
65
65
|
const mcp_launcher_1 = require("./mcp-launcher");
|
|
66
|
+
const tool_schemas_1 = require("../mcp/tool-schemas");
|
|
67
|
+
const mcp_schema_cost_1 = require("./mcp-schema-cost");
|
|
66
68
|
/**
|
|
67
69
|
* Checks if a script path belongs to the packageRoot, supporting symlinked global installs.
|
|
68
70
|
*/
|
|
@@ -443,6 +445,16 @@ async function runDoctor(packageRoot, target, opts, extraChecks) {
|
|
|
443
445
|
else {
|
|
444
446
|
add('ledger signing', 'WARN', '원장이 서명되지 않습니다(HOLMES_LEDGER_KEY 없음) — 체인이 평문 해시라 파일을 쓸 수 있는 자가 승인 기록도 만들 수 있고, ART-5 의 원장 대조는 장벽이 아니라 비용이 됩니다', 'HOLMES_LEDGER_KEY 를 대역외(에이전트를 기동하는 사람의 환경)에서 설정한 뒤 서버를 다시 시작하십시오. 값은 이 보고서에 절대 출력되지 않습니다.');
|
|
445
447
|
}
|
|
448
|
+
// @implements A-SPEC-535.1
|
|
449
|
+
// Advisory: holmes-kit's OWN advertised MCP schema cost, and the one holmes-controlled
|
|
450
|
+
// misconfiguration (HOLMES_MCP_PROFILE=full re-advertising hook-enforced gate-duplicates). Never
|
|
451
|
+
// FAIL — efficiency, not correctness. The number is derived from TOOL_SCHEMAS at runtime (no drift),
|
|
452
|
+
// and the resident-client cost is stated conditionally: a static checker cannot observe whether the
|
|
453
|
+
// client keeps schemas resident, so it never claims to have detected it.
|
|
454
|
+
{
|
|
455
|
+
const sc = (0, mcp_schema_cost_1.mcpSchemaCost)(tool_schemas_1.TOOL_SCHEMAS, tool_schemas_1.HOOK_ENFORCED_TOOLS, process.env.HOLMES_MCP_PROFILE);
|
|
456
|
+
add('mcp schema cost', sc.level, sc.detail, sc.fix);
|
|
457
|
+
}
|
|
446
458
|
// @implements A-SPEC-457
|
|
447
459
|
// Whether the ART-2 truncation backstop (A-SPEC-455) is even armed. That backstop compares the
|
|
448
460
|
// committed ledger to the working copy and enumerates targets with `git ls-files .ax/ledger`; an
|
|
@@ -0,0 +1,18 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* doctor advisory (REQ-535): report holmes-kit's OWN advertised MCP schema cost and flag the one
|
|
3
|
+
* holmes-controlled misconfiguration (HOLMES_MCP_PROFILE=full re-advertising hook-enforced
|
|
4
|
+
* gate-duplicates). Pure — no I/O. Never FAIL (efficiency, not correctness). Never claims to have
|
|
5
|
+
* detected that the client keeps schemas resident (a static checker cannot observe that); the
|
|
6
|
+
* resident-client cost is stated conditionally.
|
|
7
|
+
*/
|
|
8
|
+
export interface SchemaCostResult {
|
|
9
|
+
level: 'PASS' | 'WARN';
|
|
10
|
+
detail: string;
|
|
11
|
+
fix?: string;
|
|
12
|
+
advertised: number;
|
|
13
|
+
tokens: number;
|
|
14
|
+
}
|
|
15
|
+
export declare function mcpSchemaCost(schemas: Record<string, {
|
|
16
|
+
description?: string;
|
|
17
|
+
inputSchema?: unknown;
|
|
18
|
+
}>, hidden: Set<string>, profileEnv: string | undefined): SchemaCostResult;
|
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
"use strict";
|
|
2
|
+
Object.defineProperty(exports, "__esModule", { value: true });
|
|
3
|
+
exports.mcpSchemaCost = mcpSchemaCost;
|
|
4
|
+
function mcpSchemaCost(schemas, hidden, profileEnv) {
|
|
5
|
+
const jsonLen = (n) => JSON.stringify({ name: n, description: schemas[n]?.description, inputSchema: schemas[n]?.inputSchema }).length;
|
|
6
|
+
const names = Object.keys(schemas);
|
|
7
|
+
const advertisedNames = names.filter((n) => !hidden.has(n));
|
|
8
|
+
const hiddenNames = names.filter((n) => hidden.has(n));
|
|
9
|
+
const est = (ns) => Math.round(ns.reduce((s, n) => s + jsonLen(n), 0) / 4);
|
|
10
|
+
const tokens = est(advertisedNames);
|
|
11
|
+
const advertised = advertisedNames.length;
|
|
12
|
+
if (profileEnv === 'full' && hiddenNames.length > 0) {
|
|
13
|
+
return {
|
|
14
|
+
level: 'WARN',
|
|
15
|
+
detail: `HOLMES_MCP_PROFILE=full re-advertises ${hiddenNames.length} hook-enforced gate-duplicate tool(s) (${hiddenNames.join(', ')}) — about +${est(hiddenNames)} tokens/turn on a client that keeps MCP schemas resident. They are already enforced deterministically by the Stop/PreToolUse hooks and stay callable by name.`,
|
|
16
|
+
fix: 'Unset HOLMES_MCP_PROFILE unless you specifically need to invoke these tools by name.',
|
|
17
|
+
advertised, tokens,
|
|
18
|
+
};
|
|
19
|
+
}
|
|
20
|
+
return {
|
|
21
|
+
level: 'PASS',
|
|
22
|
+
detail: `holmes-kit advertises ${advertised} MCP tool(s) ≈ ${tokens} tokens in the default profile`
|
|
23
|
+
+ `${hiddenNames.length ? ` (${hiddenNames.length} gate-duplicate tool(s) already hidden)` : ''}. `
|
|
24
|
+
+ `If your MCP client keeps tool schemas resident, this is re-sent each turn — a fixed cost that shrinks as the session grows.`,
|
|
25
|
+
fix: 'If your MCP client supports on-demand/deferred tool loading, enabling it removes this overhead with no loss (tools load when searched).',
|
|
26
|
+
advertised, tokens,
|
|
27
|
+
};
|
|
28
|
+
}
|
|
@@ -12,6 +12,14 @@ export interface AxConfig {
|
|
|
12
12
|
* silently impose a new one.
|
|
13
13
|
*/
|
|
14
14
|
preEditEvidence: 'off' | 'warn' | 'block';
|
|
15
|
+
/**
|
|
16
|
+
* @implements A-SPEC-534.3
|
|
17
|
+
* RED-first evidence enforcement (ART-8). `off` skips the check; `track` records violations
|
|
18
|
+
* without blocking a turn (observe-first, the ship default); `strict` blocks like any other
|
|
19
|
+
* article. Default `track` — unlike `preEditEvidence`, `track` never blocks, so observing
|
|
20
|
+
* RED-first across the repo is safe before an owner promotes it to `strict`.
|
|
21
|
+
*/
|
|
22
|
+
redFirstEvidence: 'off' | 'track' | 'strict';
|
|
15
23
|
};
|
|
16
24
|
highRiskDomains: string[];
|
|
17
25
|
storage: {
|
|
@@ -47,7 +47,7 @@ exports.DEFAULT_CONFIG = {
|
|
|
47
47
|
// first refusal is `no-analysis`. Warn reports the missing evidence without stopping work, which
|
|
48
48
|
// is also the only path that accumulates the data a later block decision would need. Raised as an
|
|
49
49
|
// authority question rather than decided by measurement — the user chose this level.
|
|
50
|
-
guardrail: { enforcement: 'block', enforceHighRisk: true, overrideRequiresAdr: true, preEditEvidence: 'warn' },
|
|
50
|
+
guardrail: { enforcement: 'block', enforceHighRisk: true, overrideRequiresAdr: true, preEditEvidence: 'warn', redFirstEvidence: 'track' },
|
|
51
51
|
highRiskDomains: ['@auth', '@payment'],
|
|
52
52
|
storage: { specStore: 'local-markdown' },
|
|
53
53
|
};
|
|
@@ -1,4 +1,5 @@
|
|
|
1
1
|
import { Spec } from '../spec/spec-parser';
|
|
2
|
+
import type { TestOutcome } from '../review/test-runner';
|
|
2
3
|
/**
|
|
3
4
|
* L1 — Governance CONSTITUTION (target-architecture §7 L1).
|
|
4
5
|
*
|
|
@@ -56,6 +57,31 @@ export interface ConstitutionContext {
|
|
|
56
57
|
* record cannot hide a missing test suite because the syntactic bound is still enforced.
|
|
57
58
|
*/
|
|
58
59
|
executedByAspec?: Record<string, number>;
|
|
60
|
+
/**
|
|
61
|
+
* @implements A-SPEC-534.2
|
|
62
|
+
* RED-first evidence (ART-8). `changedAspecs` are the A-SPECs whose source changed in this work
|
|
63
|
+
* unit; `outcomesByAspec` are the recorded per-A-SPEC test outcomes (from the ledger). ART-8 fires
|
|
64
|
+
* only in `strict` mode here — `track`/`off` are the caller's report-only / ignore concerns (the
|
|
65
|
+
* config read and the track-mode recording are I/O, done by the Stop hook). Evidence-gated like
|
|
66
|
+
* ART-4: an A-SPEC with no recorded outcomes is not a violation.
|
|
67
|
+
*/
|
|
68
|
+
redFirstMode?: 'strict' | 'track' | 'off';
|
|
69
|
+
changedAspecs?: string[];
|
|
70
|
+
outcomesByAspec?: Record<string, Array<{
|
|
71
|
+
outcome: TestOutcome;
|
|
72
|
+
ts: string;
|
|
73
|
+
}>>;
|
|
59
74
|
}
|
|
75
|
+
/**
|
|
76
|
+
* @implements A-SPEC-534.2
|
|
77
|
+
* ART-8 RED-first check (pure). For each changed A-SPEC that HAS recorded outcomes, require a
|
|
78
|
+
* `red-assertion` at ts_red followed by a `green` at ts_green > ts_red. A `red-error` is NOT a valid
|
|
79
|
+
* RED (a file that could not run its cases proves nothing). An A-SPEC with no outcomes is skipped
|
|
80
|
+
* (evidence-gated).
|
|
81
|
+
*/
|
|
82
|
+
export declare function redFirstViolations(changedAspecs: string[], outcomesByAspec: Record<string, Array<{
|
|
83
|
+
outcome: TestOutcome;
|
|
84
|
+
ts: string;
|
|
85
|
+
}>>): ConstitutionViolation[];
|
|
60
86
|
export declare const ARTICLES: Record<string, string>;
|
|
61
87
|
export declare function verifyConstitution(ctx: ConstitutionContext): ConstitutionViolation[];
|
|
@@ -1,11 +1,36 @@
|
|
|
1
1
|
"use strict";
|
|
2
2
|
Object.defineProperty(exports, "__esModule", { value: true });
|
|
3
3
|
exports.ARTICLES = void 0;
|
|
4
|
+
exports.redFirstViolations = redFirstViolations;
|
|
4
5
|
exports.verifyConstitution = verifyConstitution;
|
|
5
6
|
const basis_1 = require("../mcp/basis");
|
|
6
7
|
const spec_types_1 = require("../spec/spec-types");
|
|
7
8
|
const rtm_check_1 = require("../rtm/rtm-check");
|
|
8
9
|
const validator_1 = require("../spec/validator");
|
|
10
|
+
/**
|
|
11
|
+
* @implements A-SPEC-534.2
|
|
12
|
+
* ART-8 RED-first check (pure). For each changed A-SPEC that HAS recorded outcomes, require a
|
|
13
|
+
* `red-assertion` at ts_red followed by a `green` at ts_green > ts_red. A `red-error` is NOT a valid
|
|
14
|
+
* RED (a file that could not run its cases proves nothing). An A-SPEC with no outcomes is skipped
|
|
15
|
+
* (evidence-gated).
|
|
16
|
+
*/
|
|
17
|
+
function redFirstViolations(changedAspecs, outcomesByAspec) {
|
|
18
|
+
const v = [];
|
|
19
|
+
for (const aspec of changedAspecs) {
|
|
20
|
+
const outcomes = outcomesByAspec[aspec];
|
|
21
|
+
if (!outcomes || outcomes.length === 0)
|
|
22
|
+
continue; // evidence-gated: no record → no violation
|
|
23
|
+
// The EARLIEST red-assertion anchors the ordering; a green must come after it. A red-error is
|
|
24
|
+
// deliberately excluded — it means the cases never ran, so it cannot stand in for a real RED.
|
|
25
|
+
const reds = outcomes.filter((o) => o.outcome === 'red-assertion').map((o) => o.ts).sort();
|
|
26
|
+
const tsRed = reds[0];
|
|
27
|
+
const greenAfterRed = tsRed !== undefined && outcomes.some((o) => o.outcome === 'green' && o.ts > tsRed);
|
|
28
|
+
if (!greenAfterRed) {
|
|
29
|
+
v.push({ article: 'ART-8', detail: `${aspec}: source changed but no recorded red-assertion→green sequence — a covering test that never failed first proves nothing; make it fail (assertion, not error) before the code (ART-8)` });
|
|
30
|
+
}
|
|
31
|
+
}
|
|
32
|
+
return v;
|
|
33
|
+
}
|
|
9
34
|
exports.ARTICLES = {
|
|
10
35
|
'ART-7': '열린 치명 발견은 미완이다 — severity critical 이 open 인 한 완료가 없다 (important/minor 는 review_status 보고에만 남는다: 기록을 피하게 만드는 차단은 원장을 죽인다)',
|
|
11
36
|
'ART-6': '판단은 현재 빌드에서 내려야 한다 — 설치된 코드와 갈라진 서버에서 봉인된 리뷰 결과는 이미 사라진 세계를 기술한다',
|
|
@@ -14,6 +39,7 @@ exports.ARTICLES = {
|
|
|
14
39
|
'ART-3': 'Spec validity — every governed spec satisfies its type rules, including honest 4-quadrant GWT coverage',
|
|
15
40
|
'ART-4': 'Coverage honesty — declared coverage must be backed by real anchored test cases, not prose',
|
|
16
41
|
'ART-5': 'Approval is out-of-band — a spec cannot self-approve; governance config cannot be self-written',
|
|
42
|
+
'ART-8': 'Test-first is observed — a changed A-SPEC must show a recorded red-assertion→green sequence before it is done (a red-error is not a valid RED)',
|
|
17
43
|
};
|
|
18
44
|
function verifyConstitution(ctx) {
|
|
19
45
|
const governed = (0, spec_types_1.filterGoverned)(ctx.specs);
|
|
@@ -67,6 +93,13 @@ function verifyConstitution(ctx) {
|
|
|
67
93
|
}
|
|
68
94
|
}
|
|
69
95
|
}
|
|
96
|
+
// @implements A-SPEC-534.2
|
|
97
|
+
// ART-8: RED-first evidence. Only `strict` mode emits a BLOCKING violation here; `track` (report
|
|
98
|
+
// only) and `off` are the caller's I/O concern (the config read + track-mode ledger recording live
|
|
99
|
+
// in the Stop hook, 534.3). Evidence-gated inside redFirstViolations.
|
|
100
|
+
if (ctx.redFirstMode === 'strict' && ctx.changedAspecs && ctx.outcomesByAspec) {
|
|
101
|
+
v.push(...redFirstViolations(ctx.changedAspecs, ctx.outcomesByAspec));
|
|
102
|
+
}
|
|
70
103
|
// @implements A-SPEC-160
|
|
71
104
|
// ART-6: a judgement sealed while the answering server ran code that no longer matched what was
|
|
72
105
|
// installed. Measured 2026-08-08: such a server reported `impactedSpecs: []` for a commit touching
|
|
@@ -1,5 +1,6 @@
|
|
|
1
1
|
import { PendingRequest } from '../governance/approval-queue';
|
|
2
2
|
import { Spec } from '../spec/spec-parser';
|
|
3
|
+
import type { TestOutcome } from '../review/test-runner';
|
|
3
4
|
/**
|
|
4
5
|
* @implements A-SPEC-100.2
|
|
5
6
|
* Stop-hook governance gate (Phase-2 #1: push, not pull).
|
|
@@ -81,7 +82,26 @@ export interface StopEvidence {
|
|
|
81
82
|
* catches it already existed inside `rechainLedger` and ran from the CLI only.
|
|
82
83
|
*/
|
|
83
84
|
rolledBackLedgers?: string[];
|
|
85
|
+
/**
|
|
86
|
+
* @implements A-SPEC-534.4
|
|
87
|
+
* ART-8 RED-first evidence. `changedAspecs` are the A-SPECs whose source is dirty this turn;
|
|
88
|
+
* `outcomesByAspec` are their recorded outcomes at the current baseline HEAD; `redFirstMode` is the
|
|
89
|
+
* config posture. `strict` blocks via the constitution; `track` records to `tracked` without
|
|
90
|
+
* blocking; `off`/absent does nothing.
|
|
91
|
+
*/
|
|
92
|
+
changedAspecs?: string[];
|
|
93
|
+
outcomesByAspec?: Record<string, Array<{
|
|
94
|
+
outcome: TestOutcome;
|
|
95
|
+
ts: string;
|
|
96
|
+
}>>;
|
|
97
|
+
redFirstMode?: 'strict' | 'track' | 'off';
|
|
84
98
|
}
|
|
99
|
+
/**
|
|
100
|
+
* @implements A-SPEC-534.4
|
|
101
|
+
* ART-8 evidence (I/O half): the A-SPECs whose DIRTY source files carry an @implements anchor. git is
|
|
102
|
+
* a refinement — no repository means `undefined` (no signal), never a false clean.
|
|
103
|
+
*/
|
|
104
|
+
export declare function changedAnchoredAspecs(root: string): string[] | undefined;
|
|
85
105
|
/**
|
|
86
106
|
* @implements A-SPEC-452
|
|
87
107
|
* ART-1 evidence: which changed source files claim nothing.
|
|
@@ -127,6 +147,10 @@ export declare function evaluateStop(specs: Spec[], evidence?: StopEvidence): {
|
|
|
127
147
|
article: string;
|
|
128
148
|
detail: string;
|
|
129
149
|
}[];
|
|
150
|
+
tracked?: {
|
|
151
|
+
article: string;
|
|
152
|
+
detail: string;
|
|
153
|
+
}[];
|
|
130
154
|
};
|
|
131
155
|
/**
|
|
132
156
|
* @implements A-SPEC-134
|
|
@@ -34,6 +34,7 @@ var __importStar = (this && this.__importStar) || (function () {
|
|
|
34
34
|
})();
|
|
35
35
|
Object.defineProperty(exports, "__esModule", { value: true });
|
|
36
36
|
exports.MAX_CONSECUTIVE_BLOCKS = void 0;
|
|
37
|
+
exports.changedAnchoredAspecs = changedAnchoredAspecs;
|
|
37
38
|
exports.unanchoredChangedSources = unanchoredChangedSources;
|
|
38
39
|
exports.unrecordedApprovals = unrecordedApprovals;
|
|
39
40
|
exports.rolledBackLedgers = rolledBackLedgers;
|
|
@@ -56,10 +57,50 @@ const test_scope_1 = require("../rtm/test-scope");
|
|
|
56
57
|
const constitution_1 = require("../governance/constitution");
|
|
57
58
|
const provenance_chain_1 = require("../governance/provenance-chain");
|
|
58
59
|
const test_evidence_1 = require("../review/test-evidence");
|
|
60
|
+
const test_outcomes_1 = require("../review/test-outcomes");
|
|
61
|
+
const config_1 = require("../config/config");
|
|
59
62
|
const pre_tool_use_1 = require("./pre-tool-use");
|
|
60
63
|
const governance_history_1 = require("../guardrail/governance-history");
|
|
61
64
|
const constitution_debt_1 = require("../governance/constitution-debt");
|
|
62
65
|
const root_1 = require("../project/root");
|
|
66
|
+
/**
|
|
67
|
+
* @implements A-SPEC-534.4
|
|
68
|
+
* ART-8 evidence (I/O half): the A-SPECs whose DIRTY source files carry an @implements anchor. git is
|
|
69
|
+
* a refinement — no repository means `undefined` (no signal), never a false clean.
|
|
70
|
+
*/
|
|
71
|
+
function changedAnchoredAspecs(root) {
|
|
72
|
+
const SOURCE = /\.(?:ts|tsx|mts|cts|js|jsx|mjs|cjs|py|go|rs|java|kt|cs|rb|php|swift)$/;
|
|
73
|
+
const VENDORED = /^(?:reference|node_modules|dist|build|vendor|third_party)\//;
|
|
74
|
+
let raw;
|
|
75
|
+
try {
|
|
76
|
+
raw = (0, node_child_process_1.execFileSync)('git', ['status', '--porcelain', '-uall'], {
|
|
77
|
+
cwd: root, encoding: 'utf8', stdio: ['ignore', 'pipe', 'ignore'], env: (0, root_1.cleanSubprocessEnv)(),
|
|
78
|
+
});
|
|
79
|
+
}
|
|
80
|
+
catch {
|
|
81
|
+
return undefined;
|
|
82
|
+
}
|
|
83
|
+
const ids = new Set();
|
|
84
|
+
for (const line of raw.split('\n')) {
|
|
85
|
+
if (line.trim() === '')
|
|
86
|
+
continue;
|
|
87
|
+
let rel = line.slice(3).trim().replace(/^"|"$/g, '');
|
|
88
|
+
if (rel.includes(' -> '))
|
|
89
|
+
rel = rel.split(' -> ')[1]; // renames name the destination
|
|
90
|
+
if (!SOURCE.test(rel) || VENDORED.test(rel))
|
|
91
|
+
continue;
|
|
92
|
+
let text;
|
|
93
|
+
try {
|
|
94
|
+
text = fs.readFileSync(path.join(root, rel), 'utf8');
|
|
95
|
+
}
|
|
96
|
+
catch {
|
|
97
|
+
continue;
|
|
98
|
+
} // deleted/unreadable
|
|
99
|
+
for (const m of text.matchAll(/@implements\s+(A-SPEC-\d+(?:\.\d+)?)/g))
|
|
100
|
+
ids.add(m[1]);
|
|
101
|
+
}
|
|
102
|
+
return [...ids].sort();
|
|
103
|
+
}
|
|
63
104
|
/**
|
|
64
105
|
* @implements A-SPEC-452
|
|
65
106
|
* ART-1 evidence: which changed source files claim nothing.
|
|
@@ -248,7 +289,21 @@ function evaluateStop(specs, evidence) {
|
|
|
248
289
|
// L1: the Stop gate IS the constitution's re-verification point — every turn boundary re-runs the
|
|
249
290
|
// inviolable articles (ART-2 RTM, ART-3 validity incl. 4-quadrant GWT, ART-4 coverage evidence).
|
|
250
291
|
// The articles live in ONE place (governance/constitution.ts); this gate merely executes them.
|
|
251
|
-
|
|
292
|
+
// @implements A-SPEC-534.4 — ART-8 evidence rides through to the constitution, which emits a
|
|
293
|
+
// BLOCKING ART-8 violation only in `strict` mode. `track`/`off` produce none here.
|
|
294
|
+
const violations = (0, constitution_1.verifyConstitution)({
|
|
295
|
+
specs, testCasesByAspec: evidence?.testCasesByAspec, executedByAspec: evidence?.executedByAspec, findings: evidence?.findings,
|
|
296
|
+
redFirstMode: evidence?.redFirstMode, changedAspecs: evidence?.changedAspecs, outcomesByAspec: evidence?.outcomesByAspec,
|
|
297
|
+
});
|
|
298
|
+
// @implements A-SPEC-534.4 — `track` records ART-8 findings without blocking the turn. Computed
|
|
299
|
+
// separately (the constitution stays silent on ART-8 outside strict) and returned in `tracked` for
|
|
300
|
+
// the CLI to record; it never enters `problems`/the block decision.
|
|
301
|
+
let tracked;
|
|
302
|
+
if (evidence?.redFirstMode === 'track' && evidence.changedAspecs && evidence.outcomesByAspec) {
|
|
303
|
+
const t = (0, constitution_1.redFirstViolations)(evidence.changedAspecs, evidence.outcomesByAspec);
|
|
304
|
+
if (t.length)
|
|
305
|
+
tracked = t;
|
|
306
|
+
}
|
|
252
307
|
const problems = violations.map((x) => `[${x.article}] ${x.detail}`);
|
|
253
308
|
// @implements A-SPEC-247 — structured list so the caller can ask acknowledgeStop which of these
|
|
254
309
|
// are waiting on an owner. Mirrors `problems` exactly, including the two synthesized below.
|
|
@@ -287,7 +342,7 @@ function evaluateStop(specs, evidence) {
|
|
|
287
342
|
structured.push({ article: 'ART-2', detail });
|
|
288
343
|
}
|
|
289
344
|
if (problems.length === 0)
|
|
290
|
-
return { block: false };
|
|
345
|
+
return { block: false, ...(tracked ? { tracked } : {}) };
|
|
291
346
|
const shown = problems.slice(0, 20);
|
|
292
347
|
const more = problems.length > shown.length ? `\n…and ${problems.length - shown.length} more` : '';
|
|
293
348
|
// @implements A-SPEC-134 — the distinct articles feed the constitution-debt state on a cap-yield.
|
|
@@ -301,6 +356,7 @@ function evaluateStop(specs, evidence) {
|
|
|
301
356
|
block: true,
|
|
302
357
|
articles,
|
|
303
358
|
violations: structured,
|
|
359
|
+
...(tracked ? { tracked } : {}),
|
|
304
360
|
reason: `[Holmes-Kit] constitution gate: ${problems.length} article violation(s) must be ` +
|
|
305
361
|
`fixed before finishing:\n${shown.join('\n')}${more}`,
|
|
306
362
|
};
|
|
@@ -656,6 +712,25 @@ if (require.main === module) {
|
|
|
656
712
|
catch {
|
|
657
713
|
executedByAspec = undefined;
|
|
658
714
|
}
|
|
715
|
+
// @implements A-SPEC-534.4 — ART-8 RED-first evidence (I/O half). Outcomes recorded at the
|
|
716
|
+
// current baseline HEAD (where the dirty work's red+green ran); changed anchored A-SPECs from
|
|
717
|
+
// the working tree; the posture from config. Fail-open: any error leaves ART-8 inert this turn.
|
|
718
|
+
let redFirstMode;
|
|
719
|
+
let changedAspecs;
|
|
720
|
+
let outcomesByAspec;
|
|
721
|
+
try {
|
|
722
|
+
redFirstMode = (0, config_1.loadConfig)(stopProjectRoot()).guardrail.redFirstEvidence;
|
|
723
|
+
if (redFirstMode !== 'off') {
|
|
724
|
+
const head = (0, node_child_process_1.execFileSync)('git', ['rev-parse', 'HEAD'], { cwd: stopProjectRoot(), stdio: ['ignore', 'pipe', 'ignore'], env: (0, root_1.cleanSubprocessEnv)() }).toString().trim();
|
|
725
|
+
changedAspecs = changedAnchoredAspecs(stopProjectRoot());
|
|
726
|
+
outcomesByAspec = (0, test_outcomes_1.groupOutcomesByAspec)((0, test_outcomes_1.readOutcomes)(stopProjectRoot()), head);
|
|
727
|
+
}
|
|
728
|
+
}
|
|
729
|
+
catch {
|
|
730
|
+
redFirstMode = undefined;
|
|
731
|
+
changedAspecs = undefined;
|
|
732
|
+
outcomesByAspec = undefined;
|
|
733
|
+
}
|
|
659
734
|
// Provenance-chain verification (fail-open: a verify error skips the check, never crashes).
|
|
660
735
|
let provenance;
|
|
661
736
|
// @implements A-SPEC-148
|
|
@@ -690,7 +765,12 @@ if (require.main === module) {
|
|
|
690
765
|
const unrecorded = unrecordedApprovals(stopProjectRoot());
|
|
691
766
|
// @implements A-SPEC-455
|
|
692
767
|
const rolledBack = rolledBackLedgers(stopProjectRoot());
|
|
693
|
-
let out = evaluateStop(specs, { testCasesByAspec, provenance, executedByAspec, findings, findingsUnreadable, unanchoredChangedSources: unanchored, unrecordedApprovals: unrecorded, rolledBackLedgers: rolledBack });
|
|
768
|
+
let out = evaluateStop(specs, { testCasesByAspec, provenance, executedByAspec, findings, findingsUnreadable, unanchoredChangedSources: unanchored, unrecordedApprovals: unrecorded, rolledBackLedgers: rolledBack, redFirstMode, changedAspecs, outcomesByAspec });
|
|
769
|
+
// @implements A-SPEC-534.4 — track mode records ART-8 findings without blocking: surface them so
|
|
770
|
+
// the operator observes RED-first gaps before an owner promotes the posture to strict.
|
|
771
|
+
if (out.tracked && out.tracked.length > 0) {
|
|
772
|
+
process.stderr.write(`[Holmes-Kit] ART-8 RED-first (track): ${out.tracked.map((t) => t.detail).join(' | ')}\n`);
|
|
773
|
+
}
|
|
694
774
|
// @implements A-SPEC-247 — before deciding to re-block, ask whether every unresolved debt is
|
|
695
775
|
// already queued for the owner. If so, tell the user ONCE and let the turn finish; a single
|
|
696
776
|
// non-waiting violation and we block exactly as before.
|
|
@@ -347,7 +347,25 @@ declare function makeRawHandlers(store: SpecStore, opts?: ElicitOpts): {
|
|
|
347
347
|
head?: string;
|
|
348
348
|
since?: string;
|
|
349
349
|
mark?: string;
|
|
350
|
+
mutate?: string;
|
|
350
351
|
}): Promise<{
|
|
352
|
+
mutate: {
|
|
353
|
+
tspec: string;
|
|
354
|
+
aspec: string | undefined;
|
|
355
|
+
coveringFiles: string[];
|
|
356
|
+
results: ({
|
|
357
|
+
mutation: import("../spec/kills").Mutation;
|
|
358
|
+
applied: boolean;
|
|
359
|
+
reason: string;
|
|
360
|
+
} | {
|
|
361
|
+
applied: boolean;
|
|
362
|
+
verdict?: "killed" | "survived";
|
|
363
|
+
mutation: import("../spec/kills").Mutation;
|
|
364
|
+
reason?: undefined;
|
|
365
|
+
})[];
|
|
366
|
+
survivors: import("../spec/kills").Mutation[];
|
|
367
|
+
};
|
|
368
|
+
} | {
|
|
351
369
|
baselineRecorded?: string | undefined;
|
|
352
370
|
scopeFallback?: "full" | undefined;
|
|
353
371
|
tier: import("../rtm/test-scope").RegressionTier;
|
|
@@ -359,6 +377,7 @@ declare function makeRawHandlers(store: SpecStore, opts?: ElicitOpts): {
|
|
|
359
377
|
tail: string;
|
|
360
378
|
unresolvedFiles: string[];
|
|
361
379
|
changeSource: ChangeSourceInfo;
|
|
380
|
+
mutate?: undefined;
|
|
362
381
|
}>;
|
|
363
382
|
issue_localize(a: {
|
|
364
383
|
root: string;
|
|
@@ -68,7 +68,10 @@ const rtm_check_1 = require("../rtm/rtm-check");
|
|
|
68
68
|
const test_scope_1 = require("../rtm/test-scope");
|
|
69
69
|
const gap_analyzer_1 = require("../rtm/gap-analyzer");
|
|
70
70
|
const test_runner_1 = require("../review/test-runner");
|
|
71
|
+
const kills_1 = require("../spec/kills");
|
|
72
|
+
const mutate_1 = require("../review/mutate");
|
|
71
73
|
const test_evidence_1 = require("../review/test-evidence");
|
|
74
|
+
const test_outcomes_1 = require("../review/test-outcomes");
|
|
72
75
|
const localize_1 = require("../rtm/localize");
|
|
73
76
|
const maintenance_analyze_1 = require("./maintenance-analyze");
|
|
74
77
|
const maintenance_evidence_1 = require("./maintenance-evidence");
|
|
@@ -90,6 +93,53 @@ const root_1 = require("../project/root");
|
|
|
90
93
|
// under that tree stopped at the minted marker. This is the harm §8 closed for `review_record`,
|
|
91
94
|
// left open on the scan path. When the store is bound, the cache is the bound project's; otherwise
|
|
92
95
|
// only an anchored answer may be written to, and an unanchored one falls back to a temp cache.
|
|
96
|
+
// @implements A-SPEC-534.8
|
|
97
|
+
// The governed source file a `kills` mutation targets: one that @implements the A-SPEC AND contains
|
|
98
|
+
// the `where` literal. Bounded walk, skips vendored/test/hidden dirs. null when none qualifies.
|
|
99
|
+
function sourceFileWithMutation(root, aspecId, where) {
|
|
100
|
+
if (!where)
|
|
101
|
+
return null;
|
|
102
|
+
const SOURCE = /\.(?:ts|tsx|mts|cts|js|jsx|mjs|cjs|py|go|rs|java|kt|cs|rb|php|swift)$/;
|
|
103
|
+
let found = null;
|
|
104
|
+
const walk = (d) => {
|
|
105
|
+
if (found)
|
|
106
|
+
return;
|
|
107
|
+
let entries;
|
|
108
|
+
try {
|
|
109
|
+
entries = fs.readdirSync(d, { withFileTypes: true });
|
|
110
|
+
}
|
|
111
|
+
catch {
|
|
112
|
+
return;
|
|
113
|
+
}
|
|
114
|
+
for (const e of entries) {
|
|
115
|
+
if (found)
|
|
116
|
+
return;
|
|
117
|
+
if (e.name === 'node_modules' || e.name === 'dist' || e.name === 'reference'
|
|
118
|
+
|| e.name === 'vendor' || e.name === 'third_party' || e.name.startsWith('.'))
|
|
119
|
+
continue;
|
|
120
|
+
const p = path.join(d, e.name);
|
|
121
|
+
if (e.isDirectory()) {
|
|
122
|
+
walk(p);
|
|
123
|
+
continue;
|
|
124
|
+
}
|
|
125
|
+
if (!SOURCE.test(e.name) || /\.test\./.test(e.name))
|
|
126
|
+
continue;
|
|
127
|
+
let text;
|
|
128
|
+
try {
|
|
129
|
+
text = fs.readFileSync(p, 'utf8');
|
|
130
|
+
}
|
|
131
|
+
catch {
|
|
132
|
+
continue;
|
|
133
|
+
}
|
|
134
|
+
if (text.includes(`@implements ${aspecId}`) && text.includes(where)) {
|
|
135
|
+
found = p;
|
|
136
|
+
return;
|
|
137
|
+
}
|
|
138
|
+
}
|
|
139
|
+
};
|
|
140
|
+
walk(root);
|
|
141
|
+
return found;
|
|
142
|
+
}
|
|
93
143
|
const cacheDirFor = (root) => {
|
|
94
144
|
const r = (0, root_2.resolveProjectRoot)(root);
|
|
95
145
|
if (r.marker !== 'given')
|
|
@@ -1259,8 +1309,15 @@ function makeRawHandlers(store, opts) {
|
|
|
1259
1309
|
// element instead leaves a window in which an external edit is silently destroyed and the
|
|
1260
1310
|
// STALE content gets sealed (measured: 17 of 40 concurrent edits lost, 35-55ms window).
|
|
1261
1311
|
const cur = await store.read(a.id);
|
|
1262
|
-
|
|
1263
|
-
|
|
1312
|
+
// @implements A-SPEC-536.1 — BUG-1: a spec whose YAML is broken is dropped by read()/list(),
|
|
1313
|
+
// so a bare "not found" hid that the file EXISTS but cannot be parsed. Surface the skipped
|
|
1314
|
+
// files when there are any; byte-identical to the legacy message when there are none. The
|
|
1315
|
+
// store's specsRoot is read through the same cast the reachability checks use (A-SPEC-169).
|
|
1316
|
+
if (!cur) {
|
|
1317
|
+
const dir = store.specsRoot;
|
|
1318
|
+
const unreadable = typeof dir === 'string' ? (0, spec_store_1.unreadableSpecFiles)(dir) : [];
|
|
1319
|
+
return { ok: false, reason: (0, spec_store_1.notFoundReason)(a.id, unreadable) };
|
|
1320
|
+
}
|
|
1264
1321
|
const spec = cur.spec;
|
|
1265
1322
|
// @implements A-SPEC-188 — duplicates make the id ambiguous, for the SPEC and for its
|
|
1266
1323
|
// PARENTS alike. Round-3 probed the parent half: with a stray duplicate of the parent
|
|
@@ -1829,6 +1886,26 @@ function makeRawHandlers(store, opts) {
|
|
|
1829
1886
|
// Closes the decision->execution loop: scope -> run -> durable per-A-SPEC EXECUTION evidence
|
|
1830
1887
|
// (what the constitution's ART-4 prefers over the syntactic count).
|
|
1831
1888
|
const { root, specs, scanned, changedFiles, changedSymbols, changeSource, scopeFallback, anchorImpactedSpecs, changedTestFiles, unresolvedFiles } = await deriveChangedContext(store, a.root, a, 'test_run');
|
|
1889
|
+
// @implements A-SPEC-534.8 — `mutate`: run a T-SPEC's declared `kills` mutations against the
|
|
1890
|
+
// A-SPEC's source and report which SURVIVED (a discriminating-power gap). Selective, opt-in via
|
|
1891
|
+
// the argument; absent → the ordinary run below is untouched.
|
|
1892
|
+
if (a.mutate) {
|
|
1893
|
+
const tspec = specs.find((s) => s.id === a.mutate && s.type === 'T-SPEC');
|
|
1894
|
+
const kills = tspec ? (0, kills_1.parseKills)(tspec.frontmatter) : [];
|
|
1895
|
+
const aspecId = tspec?.dependsOn[0];
|
|
1896
|
+
const anchors = (0, test_scope_1.scanTestAnchors)(root);
|
|
1897
|
+
const coveringFiles = aspecId
|
|
1898
|
+
? Object.entries(anchors).filter(([, ids]) => ids.includes(aspecId)).map(([f]) => f) : [];
|
|
1899
|
+
const results = kills.map((m) => {
|
|
1900
|
+
const src = aspecId ? sourceFileWithMutation(root, aspecId, m.where) : null;
|
|
1901
|
+
if (!src)
|
|
1902
|
+
return { mutation: m, applied: false, reason: 'no governed source anchors this A-SPEC and contains `where`' };
|
|
1903
|
+
const r = (0, mutate_1.runKillsOnFile)(src, m, coveringFiles, (files) => (0, test_runner_1.runJestOutcomes)(files, root));
|
|
1904
|
+
return { mutation: m, ...r };
|
|
1905
|
+
});
|
|
1906
|
+
const survivors = results.filter((r) => r.verdict === 'survived').map((r) => r.mutation);
|
|
1907
|
+
return { mutate: { tspec: a.mutate, aspec: aspecId, coveringFiles, results, survivors } };
|
|
1908
|
+
}
|
|
1832
1909
|
const g = new rtm_graph_1.RtmGraph();
|
|
1833
1910
|
let testScope;
|
|
1834
1911
|
try {
|
|
@@ -1869,6 +1946,12 @@ function makeRawHandlers(store, opts) {
|
|
|
1869
1946
|
if (verified) {
|
|
1870
1947
|
(0, test_evidence_1.writeTestEvidence)(root, { ts: new Date().toISOString(), head, tier: testScope.tier, passed: true, executedByAspec });
|
|
1871
1948
|
}
|
|
1949
|
+
// @implements A-SPEC-534.5 — RED-first outcome evidence for ART-8. Recorded UNCONDITIONALLY,
|
|
1950
|
+
// unlike the green-only baseline above: a red-assertion recorded before the code is exactly what
|
|
1951
|
+
// the red→green sequence needs. Append-only, one record per (A-SPEC, outcome) at this HEAD.
|
|
1952
|
+
if (result.outcomeByFile) {
|
|
1953
|
+
(0, test_outcomes_1.appendOutcomes)(root, (0, test_outcomes_1.buildOutcomeRecords)(result.outcomeByFile, anchors, head, new Date().toISOString()));
|
|
1954
|
+
}
|
|
1872
1955
|
// @implements A-SPEC-128
|
|
1873
1956
|
// The baseline is written under EXACTLY the condition that already gates evidence: a run that
|
|
1874
1957
|
// actually executed and passed. A red or skipped run must never become the reference point for
|
|
@@ -3060,7 +3143,7 @@ id: ${aspecId}
|
|
|
3060
3143
|
type: A-SPEC
|
|
3061
3144
|
title: ${(0, yaml_scalar_1.yamlScalar)(`Architecture Specification for ${a.title}`)}
|
|
3062
3145
|
status: draft
|
|
3063
|
-
slice: ${a.sliceName}
|
|
3146
|
+
slice: ${(0, yaml_scalar_1.yamlScalar)(a.sliceName)}
|
|
3064
3147
|
priority: P1
|
|
3065
3148
|
independent_test: true
|
|
3066
3149
|
depends_on:
|
|
@@ -203,6 +203,7 @@ exports.TOOL_SCHEMAS = {
|
|
|
203
203
|
head: str('Head rev (e.g. HEAD). Optional — omit to compare against a baseline.'),
|
|
204
204
|
since: str('Baseline label to compare against when no git range is given (default `last-green`, recorded by a passing test_run). Works with or without version control.'),
|
|
205
205
|
mark: str('Baseline label to record when the run passes (default `last-green`). Nothing is recorded for a red or skipped run.'),
|
|
206
|
+
mutate: str('T-SPEC id (e.g. T-SPEC-129.1). When set, runs that spec’s declared `kills` mutations against the A-SPEC’s source and reports which SURVIVED — a discriminating-power gap. Selective, opt-in; the ordinary run is skipped.'),
|
|
206
207
|
},
|
|
207
208
|
required: ['root'],
|
|
208
209
|
},
|
|
@@ -0,0 +1,17 @@
|
|
|
1
|
+
import { Mutation } from '../spec/kills';
|
|
2
|
+
import { TestOutcome } from './test-runner';
|
|
3
|
+
/**
|
|
4
|
+
* Did the mutation KILL the covering tests? A kill is a `red-assertion` in at least one covering
|
|
5
|
+
* file — the mutation broke behaviour and a test caught it. A green (or absent, or red-error) result
|
|
6
|
+
* is `survived`: the tests did not catch the change, which is a discriminating-power gap. Pure.
|
|
7
|
+
*/
|
|
8
|
+
export declare function mutationVerdict(outcomeByFile: Record<string, TestOutcome>, coveringFiles: string[]): 'killed' | 'survived';
|
|
9
|
+
/**
|
|
10
|
+
* Apply one mutation to a source file, run the covering tests through the injected runner, judge the
|
|
11
|
+
* verdict, and ALWAYS restore the original source. `applied:false` when `where` is absent (the file
|
|
12
|
+
* is never touched).
|
|
13
|
+
*/
|
|
14
|
+
export declare function runKillsOnFile(sourceFile: string, mutation: Mutation, coveringFiles: string[], run: (files: string[]) => Record<string, TestOutcome>): {
|
|
15
|
+
applied: boolean;
|
|
16
|
+
verdict?: 'killed' | 'survived';
|
|
17
|
+
};
|
|
@@ -0,0 +1,66 @@
|
|
|
1
|
+
"use strict";
|
|
2
|
+
var __createBinding = (this && this.__createBinding) || (Object.create ? (function(o, m, k, k2) {
|
|
3
|
+
if (k2 === undefined) k2 = k;
|
|
4
|
+
var desc = Object.getOwnPropertyDescriptor(m, k);
|
|
5
|
+
if (!desc || ("get" in desc ? !m.__esModule : desc.writable || desc.configurable)) {
|
|
6
|
+
desc = { enumerable: true, get: function() { return m[k]; } };
|
|
7
|
+
}
|
|
8
|
+
Object.defineProperty(o, k2, desc);
|
|
9
|
+
}) : (function(o, m, k, k2) {
|
|
10
|
+
if (k2 === undefined) k2 = k;
|
|
11
|
+
o[k2] = m[k];
|
|
12
|
+
}));
|
|
13
|
+
var __setModuleDefault = (this && this.__setModuleDefault) || (Object.create ? (function(o, v) {
|
|
14
|
+
Object.defineProperty(o, "default", { enumerable: true, value: v });
|
|
15
|
+
}) : function(o, v) {
|
|
16
|
+
o["default"] = v;
|
|
17
|
+
});
|
|
18
|
+
var __importStar = (this && this.__importStar) || (function () {
|
|
19
|
+
var ownKeys = function(o) {
|
|
20
|
+
ownKeys = Object.getOwnPropertyNames || function (o) {
|
|
21
|
+
var ar = [];
|
|
22
|
+
for (var k in o) if (Object.prototype.hasOwnProperty.call(o, k)) ar[ar.length] = k;
|
|
23
|
+
return ar;
|
|
24
|
+
};
|
|
25
|
+
return ownKeys(o);
|
|
26
|
+
};
|
|
27
|
+
return function (mod) {
|
|
28
|
+
if (mod && mod.__esModule) return mod;
|
|
29
|
+
var result = {};
|
|
30
|
+
if (mod != null) for (var k = ownKeys(mod), i = 0; i < k.length; i++) if (k[i] !== "default") __createBinding(result, mod, k[i]);
|
|
31
|
+
__setModuleDefault(result, mod);
|
|
32
|
+
return result;
|
|
33
|
+
};
|
|
34
|
+
})();
|
|
35
|
+
Object.defineProperty(exports, "__esModule", { value: true });
|
|
36
|
+
exports.mutationVerdict = mutationVerdict;
|
|
37
|
+
exports.runKillsOnFile = runKillsOnFile;
|
|
38
|
+
// @implements A-SPEC-534.8
|
|
39
|
+
const fs = __importStar(require("node:fs"));
|
|
40
|
+
const kills_1 = require("../spec/kills");
|
|
41
|
+
/**
|
|
42
|
+
* Did the mutation KILL the covering tests? A kill is a `red-assertion` in at least one covering
|
|
43
|
+
* file — the mutation broke behaviour and a test caught it. A green (or absent, or red-error) result
|
|
44
|
+
* is `survived`: the tests did not catch the change, which is a discriminating-power gap. Pure.
|
|
45
|
+
*/
|
|
46
|
+
function mutationVerdict(outcomeByFile, coveringFiles) {
|
|
47
|
+
return coveringFiles.some((f) => outcomeByFile[f] === 'red-assertion') ? 'killed' : 'survived';
|
|
48
|
+
}
|
|
49
|
+
/**
|
|
50
|
+
* Apply one mutation to a source file, run the covering tests through the injected runner, judge the
|
|
51
|
+
* verdict, and ALWAYS restore the original source. `applied:false` when `where` is absent (the file
|
|
52
|
+
* is never touched).
|
|
53
|
+
*/
|
|
54
|
+
function runKillsOnFile(sourceFile, mutation, coveringFiles, run) {
|
|
55
|
+
const original = fs.readFileSync(sourceFile, 'utf8');
|
|
56
|
+
const mutated = (0, kills_1.applyMutation)(original, mutation);
|
|
57
|
+
if (mutated === null)
|
|
58
|
+
return { applied: false }; // where absent: never touch the file
|
|
59
|
+
fs.writeFileSync(sourceFile, mutated);
|
|
60
|
+
try {
|
|
61
|
+
return { applied: true, verdict: mutationVerdict(run(coveringFiles), coveringFiles) };
|
|
62
|
+
}
|
|
63
|
+
finally {
|
|
64
|
+
fs.writeFileSync(sourceFile, original); // ALWAYS restore, even if run() threw
|
|
65
|
+
}
|
|
66
|
+
}
|
|
@@ -0,0 +1,35 @@
|
|
|
1
|
+
import { TestOutcome } from './test-runner';
|
|
2
|
+
export declare const OUTCOMES_FILE: string;
|
|
3
|
+
/**
|
|
4
|
+
* A durable, append-only record of a per-A-SPEC test outcome (REQ-534 RED-first evidence). Unlike
|
|
5
|
+
* `test-evidence.json` (overwritten each run), the SEQUENCE matters here — a red-assertion followed
|
|
6
|
+
* by a green is what ART-8 reads — so outcomes accumulate. `head` stamps the baseline commit the
|
|
7
|
+
* run sat on; the red (before code) and green (after code) of one slice share it while the tree is
|
|
8
|
+
* still dirty, which is exactly when the Stop gate reads them.
|
|
9
|
+
*/
|
|
10
|
+
export interface OutcomeRecord {
|
|
11
|
+
aspec: string;
|
|
12
|
+
outcome: TestOutcome;
|
|
13
|
+
ts: string;
|
|
14
|
+
head: string;
|
|
15
|
+
testFileDigest?: string;
|
|
16
|
+
}
|
|
17
|
+
/** Append outcome records as JSONL lines. Fail-open (recording must never break a run). */
|
|
18
|
+
export declare function appendOutcomes(root: string, records: OutcomeRecord[]): boolean;
|
|
19
|
+
/** Read every outcome record; missing file → [], broken lines skipped (same convention as the other ledgers). */
|
|
20
|
+
export declare function readOutcomes(root: string): OutcomeRecord[];
|
|
21
|
+
/**
|
|
22
|
+
* @implements A-SPEC-534.5
|
|
23
|
+
* Expand per-file outcomes into per-A-SPEC records via the anchor map, stamping each with the run's
|
|
24
|
+
* head and ts. A file with no anchor (or an empty anchor list) contributes nothing — an outcome that
|
|
25
|
+
* cannot be attributed to an A-SPEC is not evidence about one. Pure.
|
|
26
|
+
*/
|
|
27
|
+
export declare function buildOutcomeRecords(outcomeByFile: Record<string, TestOutcome>, anchors: Record<string, string[]>, head: string, ts: string): OutcomeRecord[];
|
|
28
|
+
/**
|
|
29
|
+
* Group outcomes by A-SPEC, keeping only records stamped with the given baseline `head` — a stale
|
|
30
|
+
* record from another commit cannot vouch for the current work (the isFresh discipline). Pure.
|
|
31
|
+
*/
|
|
32
|
+
export declare function groupOutcomesByAspec(records: OutcomeRecord[], head: string): Record<string, Array<{
|
|
33
|
+
outcome: TestOutcome;
|
|
34
|
+
ts: string;
|
|
35
|
+
}>>;
|
|
@@ -0,0 +1,108 @@
|
|
|
1
|
+
"use strict";
|
|
2
|
+
var __createBinding = (this && this.__createBinding) || (Object.create ? (function(o, m, k, k2) {
|
|
3
|
+
if (k2 === undefined) k2 = k;
|
|
4
|
+
var desc = Object.getOwnPropertyDescriptor(m, k);
|
|
5
|
+
if (!desc || ("get" in desc ? !m.__esModule : desc.writable || desc.configurable)) {
|
|
6
|
+
desc = { enumerable: true, get: function() { return m[k]; } };
|
|
7
|
+
}
|
|
8
|
+
Object.defineProperty(o, k2, desc);
|
|
9
|
+
}) : (function(o, m, k, k2) {
|
|
10
|
+
if (k2 === undefined) k2 = k;
|
|
11
|
+
o[k2] = m[k];
|
|
12
|
+
}));
|
|
13
|
+
var __setModuleDefault = (this && this.__setModuleDefault) || (Object.create ? (function(o, v) {
|
|
14
|
+
Object.defineProperty(o, "default", { enumerable: true, value: v });
|
|
15
|
+
}) : function(o, v) {
|
|
16
|
+
o["default"] = v;
|
|
17
|
+
});
|
|
18
|
+
var __importStar = (this && this.__importStar) || (function () {
|
|
19
|
+
var ownKeys = function(o) {
|
|
20
|
+
ownKeys = Object.getOwnPropertyNames || function (o) {
|
|
21
|
+
var ar = [];
|
|
22
|
+
for (var k in o) if (Object.prototype.hasOwnProperty.call(o, k)) ar[ar.length] = k;
|
|
23
|
+
return ar;
|
|
24
|
+
};
|
|
25
|
+
return ownKeys(o);
|
|
26
|
+
};
|
|
27
|
+
return function (mod) {
|
|
28
|
+
if (mod && mod.__esModule) return mod;
|
|
29
|
+
var result = {};
|
|
30
|
+
if (mod != null) for (var k = ownKeys(mod), i = 0; i < k.length; i++) if (k[i] !== "default") __createBinding(result, mod, k[i]);
|
|
31
|
+
__setModuleDefault(result, mod);
|
|
32
|
+
return result;
|
|
33
|
+
};
|
|
34
|
+
})();
|
|
35
|
+
Object.defineProperty(exports, "__esModule", { value: true });
|
|
36
|
+
exports.OUTCOMES_FILE = void 0;
|
|
37
|
+
exports.appendOutcomes = appendOutcomes;
|
|
38
|
+
exports.readOutcomes = readOutcomes;
|
|
39
|
+
exports.buildOutcomeRecords = buildOutcomeRecords;
|
|
40
|
+
exports.groupOutcomesByAspec = groupOutcomesByAspec;
|
|
41
|
+
// @implements A-SPEC-534.3
|
|
42
|
+
const fs = __importStar(require("node:fs"));
|
|
43
|
+
const path = __importStar(require("node:path"));
|
|
44
|
+
exports.OUTCOMES_FILE = path.join('.ax', 'ledger', 'test-outcomes.jsonl');
|
|
45
|
+
/** Append outcome records as JSONL lines. Fail-open (recording must never break a run). */
|
|
46
|
+
function appendOutcomes(root, records) {
|
|
47
|
+
try {
|
|
48
|
+
const file = path.join(root, exports.OUTCOMES_FILE);
|
|
49
|
+
fs.mkdirSync(path.dirname(file), { recursive: true });
|
|
50
|
+
fs.appendFileSync(file, records.map((r) => `${JSON.stringify(r)}\n`).join(''));
|
|
51
|
+
return true;
|
|
52
|
+
}
|
|
53
|
+
catch {
|
|
54
|
+
return false;
|
|
55
|
+
}
|
|
56
|
+
}
|
|
57
|
+
/** Read every outcome record; missing file → [], broken lines skipped (same convention as the other ledgers). */
|
|
58
|
+
function readOutcomes(root) {
|
|
59
|
+
let text;
|
|
60
|
+
try {
|
|
61
|
+
text = fs.readFileSync(path.join(root, exports.OUTCOMES_FILE), 'utf8');
|
|
62
|
+
}
|
|
63
|
+
catch {
|
|
64
|
+
return [];
|
|
65
|
+
}
|
|
66
|
+
const out = [];
|
|
67
|
+
for (const line of text.split('\n')) {
|
|
68
|
+
const s = line.trim();
|
|
69
|
+
if (!s)
|
|
70
|
+
continue;
|
|
71
|
+
try {
|
|
72
|
+
const r = JSON.parse(s);
|
|
73
|
+
if (r && typeof r === 'object' && typeof r.aspec === 'string' && typeof r.outcome === 'string'
|
|
74
|
+
&& typeof r.ts === 'string' && typeof r.head === 'string') {
|
|
75
|
+
out.push(r);
|
|
76
|
+
}
|
|
77
|
+
}
|
|
78
|
+
catch { /* skip a corrupt line rather than fail the whole read */ }
|
|
79
|
+
}
|
|
80
|
+
return out;
|
|
81
|
+
}
|
|
82
|
+
/**
|
|
83
|
+
* @implements A-SPEC-534.5
|
|
84
|
+
* Expand per-file outcomes into per-A-SPEC records via the anchor map, stamping each with the run's
|
|
85
|
+
* head and ts. A file with no anchor (or an empty anchor list) contributes nothing — an outcome that
|
|
86
|
+
* cannot be attributed to an A-SPEC is not evidence about one. Pure.
|
|
87
|
+
*/
|
|
88
|
+
function buildOutcomeRecords(outcomeByFile, anchors, head, ts) {
|
|
89
|
+
const out = [];
|
|
90
|
+
for (const [file, outcome] of Object.entries(outcomeByFile)) {
|
|
91
|
+
for (const aspec of anchors[file] ?? [])
|
|
92
|
+
out.push({ aspec, outcome, ts, head });
|
|
93
|
+
}
|
|
94
|
+
return out;
|
|
95
|
+
}
|
|
96
|
+
/**
|
|
97
|
+
* Group outcomes by A-SPEC, keeping only records stamped with the given baseline `head` — a stale
|
|
98
|
+
* record from another commit cannot vouch for the current work (the isFresh discipline). Pure.
|
|
99
|
+
*/
|
|
100
|
+
function groupOutcomesByAspec(records, head) {
|
|
101
|
+
const out = {};
|
|
102
|
+
for (const r of records) {
|
|
103
|
+
if (r.head !== head)
|
|
104
|
+
continue;
|
|
105
|
+
(out[r.aspec] ??= []).push({ outcome: r.outcome, ts: r.ts });
|
|
106
|
+
}
|
|
107
|
+
return out;
|
|
108
|
+
}
|
|
@@ -32,6 +32,12 @@ export interface TestRunResult {
|
|
|
32
32
|
unsupported?: string[];
|
|
33
33
|
/** Which adapters actually ran, for evidence provenance. */
|
|
34
34
|
ranWith?: string[];
|
|
35
|
+
/**
|
|
36
|
+
* @implements A-SPEC-534.5
|
|
37
|
+
* Per-file RED-first outcome (jest only). red-assertion | red-error | green, for the outcome ledger
|
|
38
|
+
* ART-8 reads. Other ecosystems do not classify outcomes yet, so they contribute nothing here.
|
|
39
|
+
*/
|
|
40
|
+
outcomeByFile?: Record<string, TestOutcome>;
|
|
35
41
|
}
|
|
36
42
|
export type Ecosystem = 'jest' | 'pytest' | 'go' | 'rust' | 'java' | 'dotnet';
|
|
37
43
|
/** Ecosystem of a test file, by extension/convention. `null` when no runner adapter exists for it. */
|
|
@@ -51,6 +57,24 @@ export declare function parseGoTestJson(out: string): Record<string, number>;
|
|
|
51
57
|
* fabrications cannot produce evidence. Returns {} for unparseable output (caller falls back).
|
|
52
58
|
*/
|
|
53
59
|
export declare function parseExecutedCounts(stdout: string, cwd: string): Record<string, number>;
|
|
60
|
+
/**
|
|
61
|
+
* Per-file test outcome for RED-first evidence (REQ-534). `red-assertion` is a REAL red — a case ran
|
|
62
|
+
* and its assertion failed; `red-error` is a file that could not run its cases (load/collection/import
|
|
63
|
+
* error), which ART-8 (534.2) must NOT accept as a valid RED; `green` ran and all assertions passed.
|
|
64
|
+
*/
|
|
65
|
+
export type TestOutcome = 'red-assertion' | 'red-error' | 'green';
|
|
66
|
+
/**
|
|
67
|
+
* @implements A-SPEC-534.1
|
|
68
|
+
* Classify each jest-executed file into red-assertion | red-error | green from `jest --json` stdout.
|
|
69
|
+
* Pure; unparseable/empty input → {} (same fail-soft convention as parseExecutedCounts).
|
|
70
|
+
*
|
|
71
|
+
* A failed assertion anywhere in a file wins (`red-assertion`) — that is a real red. A file that ran
|
|
72
|
+
* NO assertions but errored (testExecError, or status 'failed' with an empty assertion list — a
|
|
73
|
+
* load/collection/import failure) is `red-error`, which ART-8 must not accept as a valid RED. A file
|
|
74
|
+
* that executed ≥1 assertion with none failing is `green`. A file that only pended/skipped carries no
|
|
75
|
+
* evidence and is omitted entirely.
|
|
76
|
+
*/
|
|
77
|
+
export declare function classifyJestOutcomes(stdout: string, cwd: string): Record<string, TestOutcome>;
|
|
54
78
|
/**
|
|
55
79
|
* Parse pytest's built-in JUnit XML into per-file EXECUTED case counts — the pytest analogue of
|
|
56
80
|
* parseExecutedCounts. `--junit-xml` ships with pytest core, so this needs no plugin, and it is the
|
|
@@ -75,6 +99,12 @@ export declare function parseJUnitXmlCounts(xml: string, ext?: string): Record<s
|
|
|
75
99
|
* there), and a repo with no local jest could never yield meaningful suite evidence anyway.
|
|
76
100
|
*/
|
|
77
101
|
export declare function jestEntry(cwd: string): string | null;
|
|
102
|
+
/**
|
|
103
|
+
* @implements A-SPEC-534.8
|
|
104
|
+
* Run specific jest files and return only their RED-first outcomes — the runner `test_run --mutate`
|
|
105
|
+
* injects into runKillsOnFile. Scoped, jest only (the classifier is jest-only), fail-soft to {}.
|
|
106
|
+
*/
|
|
107
|
+
export declare function runJestOutcomes(files: string[], cwd: string): Record<string, TestOutcome>;
|
|
78
108
|
/** Run the pytest half of a plan, taking execution evidence from pytest's built-in JUnit XML. */
|
|
79
109
|
export declare function runPytest(files: string[], mode: TestRunPlan['mode'], cwd: string): {
|
|
80
110
|
passed: boolean;
|
|
@@ -37,8 +37,10 @@ exports.planTestRun = planTestRun;
|
|
|
37
37
|
exports.ecosystemOf = ecosystemOf;
|
|
38
38
|
exports.parseGoTestJson = parseGoTestJson;
|
|
39
39
|
exports.parseExecutedCounts = parseExecutedCounts;
|
|
40
|
+
exports.classifyJestOutcomes = classifyJestOutcomes;
|
|
40
41
|
exports.parseJUnitXmlCounts = parseJUnitXmlCounts;
|
|
41
42
|
exports.jestEntry = jestEntry;
|
|
43
|
+
exports.runJestOutcomes = runJestOutcomes;
|
|
42
44
|
exports.runPytest = runPytest;
|
|
43
45
|
exports.parseCargoTest = parseCargoTest;
|
|
44
46
|
exports.runCargo = runCargo;
|
|
@@ -153,9 +155,57 @@ function parseExecutedCounts(stdout, cwd) {
|
|
|
153
155
|
const abs = tr.name ?? tr.testFilePath ?? '';
|
|
154
156
|
if (!abs)
|
|
155
157
|
continue;
|
|
156
|
-
const rel = abs.startsWith(cwd) ? abs.slice(cwd.length).replace(/^[/\\]/, '') : abs;
|
|
157
158
|
const ran = (tr.assertionResults ?? []).filter((a) => a.status === 'passed' || a.status === 'failed').length;
|
|
158
|
-
out[
|
|
159
|
+
out[relTestPath(abs, cwd)] = ran;
|
|
160
|
+
}
|
|
161
|
+
return out;
|
|
162
|
+
}
|
|
163
|
+
/** Repo-relative, forward-slashed key for a jest test-file path — shared by the parsers below. */
|
|
164
|
+
function relTestPath(abs, cwd) {
|
|
165
|
+
const rel = abs.startsWith(cwd) ? abs.slice(cwd.length).replace(/^[/\\]/, '') : abs;
|
|
166
|
+
return rel.split('\\').join('/');
|
|
167
|
+
}
|
|
168
|
+
/**
|
|
169
|
+
* @implements A-SPEC-534.1
|
|
170
|
+
* Classify each jest-executed file into red-assertion | red-error | green from `jest --json` stdout.
|
|
171
|
+
* Pure; unparseable/empty input → {} (same fail-soft convention as parseExecutedCounts).
|
|
172
|
+
*
|
|
173
|
+
* A failed assertion anywhere in a file wins (`red-assertion`) — that is a real red. A file that ran
|
|
174
|
+
* NO assertions but errored (testExecError, or status 'failed' with an empty assertion list — a
|
|
175
|
+
* load/collection/import failure) is `red-error`, which ART-8 must not accept as a valid RED. A file
|
|
176
|
+
* that executed ≥1 assertion with none failing is `green`. A file that only pended/skipped carries no
|
|
177
|
+
* evidence and is omitted entirely.
|
|
178
|
+
*/
|
|
179
|
+
function classifyJestOutcomes(stdout, cwd) {
|
|
180
|
+
const start = stdout.indexOf('{');
|
|
181
|
+
if (start < 0)
|
|
182
|
+
return {};
|
|
183
|
+
let data;
|
|
184
|
+
try {
|
|
185
|
+
data = JSON.parse(stdout.slice(start));
|
|
186
|
+
}
|
|
187
|
+
catch {
|
|
188
|
+
return {};
|
|
189
|
+
}
|
|
190
|
+
const out = {};
|
|
191
|
+
for (const tr of data?.testResults ?? []) {
|
|
192
|
+
const abs = tr.name ?? tr.testFilePath ?? '';
|
|
193
|
+
if (!abs)
|
|
194
|
+
continue;
|
|
195
|
+
const rel = relTestPath(abs, cwd);
|
|
196
|
+
const assertions = tr.assertionResults ?? [];
|
|
197
|
+
if (assertions.some((a) => a.status === 'failed')) {
|
|
198
|
+
out[rel] = 'red-assertion';
|
|
199
|
+
continue;
|
|
200
|
+
}
|
|
201
|
+
const executed = assertions.filter((a) => a.status === 'passed' || a.status === 'failed').length;
|
|
202
|
+
const errored = !!tr.testExecError || tr.status === 'failed';
|
|
203
|
+
if (executed === 0) {
|
|
204
|
+
if (errored)
|
|
205
|
+
out[rel] = 'red-error'; // ran nothing AND failed → could not execute its cases
|
|
206
|
+
continue; // otherwise all pending/skipped → no evidence, omit
|
|
207
|
+
}
|
|
208
|
+
out[rel] = errored ? 'red-error' : 'green';
|
|
159
209
|
}
|
|
160
210
|
return out;
|
|
161
211
|
}
|
|
@@ -248,14 +298,25 @@ function runJest(files, mode, cwd) {
|
|
|
248
298
|
// @implements A-SPEC-515.1 — the green path says what happened instead of handing back the
|
|
249
299
|
// whole `--json` document. `tailOf` wraps the result too, so an unexpectedly long summary is
|
|
250
300
|
// still bounded: a cap that any one path can escape is not a cap.
|
|
251
|
-
return { passed: true, tail: tailOf(summarizeJestJson(out) ?? out), executed: parseExecutedCounts(out, cwd) };
|
|
301
|
+
return { passed: true, tail: tailOf(summarizeJestJson(out) ?? out), executed: parseExecutedCounts(out, cwd), outcomes: classifyJestOutcomes(out, cwd) };
|
|
252
302
|
}
|
|
253
303
|
catch (e) {
|
|
254
304
|
const err = e;
|
|
255
|
-
// A failing suite still emits --json on stdout, so execution evidence
|
|
256
|
-
|
|
305
|
+
// A failing suite still emits --json on stdout, so execution evidence — and the red/green outcome
|
|
306
|
+
// classification (A-SPEC-534.1) — survives a red run.
|
|
307
|
+
return { passed: false, tail: tailOf(`${err.stdout ?? ''}\n${err.stderr ?? err.message ?? ''}`), executed: parseExecutedCounts(err.stdout ?? '', cwd), outcomes: classifyJestOutcomes(err.stdout ?? '', cwd) };
|
|
257
308
|
}
|
|
258
309
|
}
|
|
310
|
+
/**
|
|
311
|
+
* @implements A-SPEC-534.8
|
|
312
|
+
* Run specific jest files and return only their RED-first outcomes — the runner `test_run --mutate`
|
|
313
|
+
* injects into runKillsOnFile. Scoped, jest only (the classifier is jest-only), fail-soft to {}.
|
|
314
|
+
*/
|
|
315
|
+
function runJestOutcomes(files, cwd) {
|
|
316
|
+
if (files.length === 0)
|
|
317
|
+
return {};
|
|
318
|
+
return runJest(files, 'scoped', cwd).outcomes;
|
|
319
|
+
}
|
|
259
320
|
/** Run the pytest half of a plan, taking execution evidence from pytest's built-in JUnit XML. */
|
|
260
321
|
// @implements A-SPEC-502.1 — exported for the wiring test (a fake venv python capturing argv).
|
|
261
322
|
function runPytest(files, mode, cwd) {
|
|
@@ -592,12 +653,16 @@ function runTestScope(scope, cwd) {
|
|
|
592
653
|
let passed = true;
|
|
593
654
|
const tails = [];
|
|
594
655
|
const executedByFile = {};
|
|
656
|
+
const outcomeByFile = {}; // @implements A-SPEC-534.5 — jest only
|
|
595
657
|
for (const eco of [...groups.keys()].sort()) {
|
|
596
658
|
const r = runners[eco](groups.get(eco), plan.mode, cwd);
|
|
597
659
|
ran.push(eco);
|
|
598
660
|
passed = passed && r.passed;
|
|
599
661
|
tails.push(`[${eco}] ${r.tail}`);
|
|
600
662
|
Object.assign(executedByFile, r.executed);
|
|
663
|
+
const adapterOutcomes = r.outcomes;
|
|
664
|
+
if (adapterOutcomes)
|
|
665
|
+
Object.assign(outcomeByFile, adapterOutcomes);
|
|
601
666
|
// @implements A-SPEC-137.1 — an adapter whose toolchain is absent reports its files as
|
|
602
667
|
// unsupported; merge them so a scope that could not run them never reads as fully covered.
|
|
603
668
|
const adapterUnsupported = r.unsupported;
|
|
@@ -613,6 +678,7 @@ function runTestScope(scope, cwd) {
|
|
|
613
678
|
return {
|
|
614
679
|
tier: scope.tier, mode: plan.mode, ranFiles: plan.testFiles, passed, skipped: false,
|
|
615
680
|
tail: tails.join('\n'), executedByFile, ranWith: ran,
|
|
681
|
+
...(Object.keys(outcomeByFile).length ? { outcomeByFile } : {}),
|
|
616
682
|
...(unsupported.length ? { unsupported } : {}),
|
|
617
683
|
};
|
|
618
684
|
}
|
|
@@ -0,0 +1,14 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* A discriminating-power mutation (REQ-534 C-layer): a named change to production source that a
|
|
3
|
+
* covering test SHOULD turn red. Minimal DSL — a literal `where` replaced by `mutate`. `test_run
|
|
4
|
+
* --mutate` applies each and confirms the covering cases go red-assertion; a mutation that kills
|
|
5
|
+
* nothing is a coverage gap. Deliberately small; widen the DSL only when a real case needs it.
|
|
6
|
+
*/
|
|
7
|
+
export interface Mutation {
|
|
8
|
+
where: string;
|
|
9
|
+
mutate: string;
|
|
10
|
+
}
|
|
11
|
+
/** Extract well-formed {where,mutate} entries from `kills`. Lenient: anything else yields []. Pure. */
|
|
12
|
+
export declare function parseKills(frontmatter: Record<string, unknown>): Mutation[];
|
|
13
|
+
/** Replace the FIRST occurrence of `where` with `mutate`. null when `where` is empty or absent. Pure. */
|
|
14
|
+
export declare function applyMutation(source: string, m: Mutation): string | null;
|
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
"use strict";
|
|
2
|
+
Object.defineProperty(exports, "__esModule", { value: true });
|
|
3
|
+
exports.parseKills = parseKills;
|
|
4
|
+
exports.applyMutation = applyMutation;
|
|
5
|
+
/** Extract well-formed {where,mutate} entries from `kills`. Lenient: anything else yields []. Pure. */
|
|
6
|
+
function parseKills(frontmatter) {
|
|
7
|
+
const raw = frontmatter.kills;
|
|
8
|
+
if (!Array.isArray(raw))
|
|
9
|
+
return [];
|
|
10
|
+
const out = [];
|
|
11
|
+
for (const item of raw) {
|
|
12
|
+
if (item && typeof item === 'object'
|
|
13
|
+
&& typeof item.where === 'string'
|
|
14
|
+
&& typeof item.mutate === 'string') {
|
|
15
|
+
out.push({ where: item.where, mutate: item.mutate });
|
|
16
|
+
}
|
|
17
|
+
}
|
|
18
|
+
return out;
|
|
19
|
+
}
|
|
20
|
+
/** Replace the FIRST occurrence of `where` with `mutate`. null when `where` is empty or absent. Pure. */
|
|
21
|
+
function applyMutation(source, m) {
|
|
22
|
+
if (!m.where)
|
|
23
|
+
return null;
|
|
24
|
+
const i = source.indexOf(m.where);
|
|
25
|
+
if (i < 0)
|
|
26
|
+
return null;
|
|
27
|
+
return source.slice(0, i) + m.mutate + source.slice(i + m.where.length);
|
|
28
|
+
}
|
|
@@ -59,6 +59,15 @@ export interface SpecStore {
|
|
|
59
59
|
* exist" while looking straight at the file. The verdict stays; the silence does not.
|
|
60
60
|
*/
|
|
61
61
|
export declare function unreadableSpecFiles(specsDir: string): string[];
|
|
62
|
+
/**
|
|
63
|
+
* @implements A-SPEC-536.1
|
|
64
|
+
* The "not found" reason, enriched when the store is silently dropping unparseable files. BUG-1's
|
|
65
|
+
* silent-loss bit at the tool surface: a spec whose YAML frontmatter is broken is dropped by list()/
|
|
66
|
+
* read() and every handler answered a bare "spec <id> not found", hiding that the file EXISTS but
|
|
67
|
+
* cannot be parsed. When `unreadable` is empty the string is byte-identical to the legacy message
|
|
68
|
+
* (no regression); otherwise it names the skipped files and the usual cause.
|
|
69
|
+
*/
|
|
70
|
+
export declare function notFoundReason(id: string, unreadable: readonly string[]): string;
|
|
62
71
|
export declare class LocalMarkdownRepository implements SpecStore {
|
|
63
72
|
private readonly root;
|
|
64
73
|
constructor(root: string);
|
|
@@ -35,6 +35,7 @@ var __importStar = (this && this.__importStar) || (function () {
|
|
|
35
35
|
Object.defineProperty(exports, "__esModule", { value: true });
|
|
36
36
|
exports.MemorySpecStore = exports.LocalMarkdownRepository = exports.TargetPathOccupiedError = exports.SpecVersionConflictError = void 0;
|
|
37
37
|
exports.unreadableSpecFiles = unreadableSpecFiles;
|
|
38
|
+
exports.notFoundReason = notFoundReason;
|
|
38
39
|
const node_crypto_1 = require("node:crypto");
|
|
39
40
|
const fs = __importStar(require("node:fs"));
|
|
40
41
|
const path = __importStar(require("node:path"));
|
|
@@ -118,6 +119,22 @@ function unreadableSpecFiles(specsDir) {
|
|
|
118
119
|
walk(specsDir);
|
|
119
120
|
return out.sort();
|
|
120
121
|
}
|
|
122
|
+
/**
|
|
123
|
+
* @implements A-SPEC-536.1
|
|
124
|
+
* The "not found" reason, enriched when the store is silently dropping unparseable files. BUG-1's
|
|
125
|
+
* silent-loss bit at the tool surface: a spec whose YAML frontmatter is broken is dropped by list()/
|
|
126
|
+
* read() and every handler answered a bare "spec <id> not found", hiding that the file EXISTS but
|
|
127
|
+
* cannot be parsed. When `unreadable` is empty the string is byte-identical to the legacy message
|
|
128
|
+
* (no regression); otherwise it names the skipped files and the usual cause.
|
|
129
|
+
*/
|
|
130
|
+
function notFoundReason(id, unreadable) {
|
|
131
|
+
const base = `spec ${id} not found`;
|
|
132
|
+
if (unreadable.length === 0)
|
|
133
|
+
return base;
|
|
134
|
+
return `${base} — but ${unreadable.length} spec file(s) failed to parse and were skipped `
|
|
135
|
+
+ `(a malformed one may be this spec): ${unreadable.join(', ')}. `
|
|
136
|
+
+ `Check the YAML frontmatter (an unquoted ':' in a value is the usual cause).`;
|
|
137
|
+
}
|
|
121
138
|
class LocalMarkdownRepository {
|
|
122
139
|
root;
|
|
123
140
|
constructor(root) {
|
|
@@ -257,6 +257,24 @@ function validateSpec(spec, resolve) {
|
|
|
257
257
|
if (!def.parents.includes(parent.type))
|
|
258
258
|
err('wrong-parent-type', `parent ${pid} is ${parent.type}, expected ${def.parents.join('|')}`);
|
|
259
259
|
}
|
|
260
|
+
// @implements A-SPEC-534.7 — optional `kills` (discriminating-power mutations). Absent → no check
|
|
261
|
+
// (every existing T-SPEC is unaffected); present → it must be an array of {string where, string
|
|
262
|
+
// mutate}. A malformed shape is always an error, independent of status.
|
|
263
|
+
if (spec.type === 'T-SPEC' && spec.frontmatter.kills != null) {
|
|
264
|
+
const raw = spec.frontmatter.kills;
|
|
265
|
+
if (!Array.isArray(raw)) {
|
|
266
|
+
err('kills-shape', 'kills must be an array of {where, mutate}');
|
|
267
|
+
}
|
|
268
|
+
else {
|
|
269
|
+
raw.forEach((item, i) => {
|
|
270
|
+
const ok = !!item && typeof item === 'object'
|
|
271
|
+
&& typeof item.where === 'string'
|
|
272
|
+
&& typeof item.mutate === 'string';
|
|
273
|
+
if (!ok)
|
|
274
|
+
err('kills-shape', `kills[${i}] must be {where: string, mutate: string}`);
|
|
275
|
+
});
|
|
276
|
+
}
|
|
277
|
+
}
|
|
260
278
|
// gate: T-SPEC 4-quadrant coverage on approve attempt
|
|
261
279
|
if (spec.type === 'T-SPEC' && spec.status === 'approved') {
|
|
262
280
|
const cov = (spec.frontmatter.coverage ?? {});
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"//": "@implements A-SPEC-209",
|
|
3
3
|
"name": "@holmes-lab/holmes-kit",
|
|
4
|
-
"version": "0.
|
|
4
|
+
"version": "0.9.0",
|
|
5
5
|
"description": "Holmes-Kit — deterministic Agentic Software Engineering (ASE) harness with causal traceability (spec chain + D-CPG + RTM + phase guardrail)",
|
|
6
6
|
"main": "dist/holmes/mcp/server.js",
|
|
7
7
|
"types": "dist/holmes/mcp/server.d.ts",
|
|
@@ -0,0 +1,82 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: holmes-tdd-slice
|
|
3
|
+
description: >-
|
|
4
|
+
Use when writing or changing code under a Holmes-Kit gate — the built-in TDD discipline. Reach for
|
|
5
|
+
it when the gate says "Approved specification required before code modification", or when the Stop
|
|
6
|
+
hook prints an "ART-8 RED-first" observation. Restates the Iron Law and Red-Green-Refactor in holmes
|
|
7
|
+
terms and tags each rule with the article that enforces it (ART-1 no-spec-no-code, ART-4 coverage
|
|
8
|
+
honesty, ART-8 RED-first evidence), so you know which rules the gate checks and which are
|
|
9
|
+
honor-system. Key trap: a RED must be a red-assertion, not a red-error.
|
|
10
|
+
---
|
|
11
|
+
|
|
12
|
+
# tdd-slice
|
|
13
|
+
|
|
14
|
+
테스트를 먼저 쓰고, **실패를 눈으로 본 뒤**, 통과시키는 최소 코드를 쓴다. 이것은 권고가 아니다 —
|
|
15
|
+
holmes-kit은 이 규율의 상당 부분을 **헌법 조항으로 집행**한다. 그래서 이 스킬의 각 규칙 옆에는
|
|
16
|
+
그것을 강제하는 조항이 붙어 있다: `[ART-N]`은 게이트가 막는 것, `[명예제]`는 네 판단에 맡기는 것.
|
|
17
|
+
합리화로 빠져나갈 수 있는 규칙과 없는 규칙을 구별하라 — 없는 것은 정말 없다.
|
|
18
|
+
|
|
19
|
+
## Iron Law
|
|
20
|
+
|
|
21
|
+
**승인된 T-SPEC이 코드보다 먼저 있어야 하고, 그 커버 테스트가 코드 전에 RED로 관측돼야 한다.**
|
|
22
|
+
|
|
23
|
+
- `[ART-1]` No Spec, No Code — 소스는 승인된 A-SPEC(+그것을 depends_on 하는 승인된 T-SPEC) 아래서만
|
|
24
|
+
바뀐다. 게이트가 코드 쓰기를 막는다. `promote-slice`가 그 승격을 다룬다.
|
|
25
|
+
- `[ART-8]` 테스트-우선은 가정이 아니라 **관측**된다 — 소스가 바뀐 A-SPEC은 원장에
|
|
26
|
+
`red-assertion`(코드 전) → `green`(코드 후) 시퀀스를 남겨야 한다. 코드 뒤에 써서 즉시 통과한
|
|
27
|
+
테스트는 실패를 관측한 적이 없으니 아무것도 증명하지 못한다. 기본은 `track`(관측), 오너가
|
|
28
|
+
실측 후 `strict`(차단)로 올린다(config `redFirstEvidence`).
|
|
29
|
+
|
|
30
|
+
## Red-Green-Refactor (holmes 판)
|
|
31
|
+
|
|
32
|
+
1. **RED** — 커버 케이스를 `test_run`으로 돌려 **실패를 본다**. 이 실패는 `red-assertion`(케이스가
|
|
33
|
+
실행되고 어서션이 실패)이어야 한다.
|
|
34
|
+
2. **GREEN** — 통과시키는 **최소** 코드를 쓴다. `test_run`이 `green`을 기록한다.
|
|
35
|
+
3. **REFACTOR** — green을 유지한 채 정리한다. 원장은 그대로다.
|
|
36
|
+
|
|
37
|
+
`test_run`이 매 실행마다 커버 파일별 outcome을 분류(`red-assertion`/`red-error`/`green`)해 원장에
|
|
38
|
+
남긴다 — 이것이 ART-8이 읽는 증거다.
|
|
39
|
+
|
|
40
|
+
## RED은 red-assertion 이어야 한다 (가장 흔한 함정)
|
|
41
|
+
|
|
42
|
+
`[ART-8]` **`red-error`는 유효 RED가 아니다.** 심볼 부재로 인한 컴파일 실패, 임포트 오류,
|
|
43
|
+
수집(collection) 실패는 케이스가 **실행조차 못 된** 상태다 — 그것은 "실패를 올바른 이유로
|
|
44
|
+
관측했다"가 아니다. 함수가 아직 없어 컴파일이 깨지면, **틀린 값을 반환하는 스텁**을 먼저 넣어
|
|
45
|
+
어서션이 붉어지게(=`red-assertion`) 만든 뒤 구현하라. superpowers가 "fails *correctly*"라 부른 것을
|
|
46
|
+
holmes는 기계적으로 판별한다: `red-error`로는 red→green 시퀀스가 성립하지 않는다.
|
|
47
|
+
|
|
48
|
+
## 좋은 테스트의 규칙
|
|
49
|
+
|
|
50
|
+
- `[ART-4]` 선언한 커버리지는 **실제 앵커된 테스트 케이스**로 뒷받침돼야 한다 — 산문이 아니다.
|
|
51
|
+
T-SPEC이 커버를 선언하면 그 A-SPEC을 `@implements`로 앵커한 테스트 파일에 실제 케이스가 있고
|
|
52
|
+
실행돼야 한다. 없으면 게이트가 막는다.
|
|
53
|
+
- `[ART-4]` 앵커는 **파일당 첫 줄**에 둔다 — 스캐너가 파일당 첫 `@implements`만 잡는다. 기존
|
|
54
|
+
파일의 비-첫줄에 새 A-SPEC 앵커를 넣으면 ART-4에 안 보인다. 새 테스트는 새 파일로.
|
|
55
|
+
- `[명예제]` mock이 아니라 **실제 동작**을 단언하라. mock의 호출 여부를 재는 테스트는 제품이 아니라
|
|
56
|
+
네 mock을 시험한다. holmes의 T-SPEC 4분면(normal/negative/corner/boundary)은 관측 가능한 동작을
|
|
57
|
+
요구한다.
|
|
58
|
+
- `[명예제]` 테스트를 쓰기 전에 **그 테스트를 실패시킬 프로덕션 변경을 한 문장으로 말하라**. 말할
|
|
59
|
+
수 없다면 그 테스트는 무엇도 지키지 못한다. (C층 `kills:`가 이를 스펙 필드로 승격한다.)
|
|
60
|
+
|
|
61
|
+
## 집행 요약
|
|
62
|
+
|
|
63
|
+
| 규칙 | 집행 |
|
|
64
|
+
|---|---|
|
|
65
|
+
| 스펙 없이 코드 없음 | `[ART-1]` PreToolUse 게이트 |
|
|
66
|
+
| 커버 테스트가 실재·실행 | `[ART-4]` Stop 헌법 |
|
|
67
|
+
| 코드 전 red-assertion→green | `[ART-8]` Stop 헌법(track/strict) |
|
|
68
|
+
| mock 아닌 실동작·판별력 | `[명예제]` (C층 `kills:`로 선택 검증) |
|
|
69
|
+
|
|
70
|
+
## 절차
|
|
71
|
+
|
|
72
|
+
1. `promote-slice`로 대상 A-SPEC과 그 T-SPEC을 승인한다(`[ART-1]` 게이트를 연다).
|
|
73
|
+
2. 커버 테스트를 **먼저** 쓴다. 심볼이 없어 컴파일이 깨지면 틀린-값 스텁을 넣는다.
|
|
74
|
+
3. `test_run` — **red-assertion**을 본다. `red-error`면 그건 아직 RED가 아니다; 스텁으로 고쳐라.
|
|
75
|
+
4. 통과시키는 최소 코드를 쓴다.
|
|
76
|
+
5. `test_run` — `green`. 원장에 red→green이 남는다(`[ART-8]` 증거).
|
|
77
|
+
6. green 유지하며 refactor.
|
|
78
|
+
|
|
79
|
+
## 검증
|
|
80
|
+
|
|
81
|
+
`src/holmes/playbooks/tdd-slice.test.ts` — 이 스킬이 집행 태그(ART-1/4/8)와 red-assertion/red-error
|
|
82
|
+
구별을 담고, 설치기가 이를 발견함을 고정한다. 상시 스위트 포함.
|