bmad-module-skill-forge 4.0.0 → 6.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +1 -1
- package/README.md +3 -3
- package/docs/_data/pinned.yaml +19 -13
- package/docs/agents.md +1 -1
- package/docs/architecture.md +5 -5
- package/docs/campaign.md +1 -1
- package/docs/concepts.md +4 -4
- package/docs/examples.md +32 -25
- package/docs/getting-started.md +3 -3
- package/docs/how-it-works.md +2 -2
- package/docs/index.md +1 -1
- package/docs/skill-model.md +41 -39
- package/docs/troubleshooting.md +9 -3
- package/docs/verifying-a-skill.md +42 -21
- package/docs/why-skf.md +1 -1
- package/docs/workflows.md +6 -6
- package/package.json +1 -1
- package/src/knowledge/tool-resolution.md +6 -6
- package/src/shared/references/headless-gate-convention.md +4 -1
- package/src/shared/references/output-contract-schema.md +3 -3
- package/src/shared/references/pipeline-contracts.md +2 -1
- package/src/shared/scripts/schemas/skf-test-result-envelope.v1.json +1 -1
- package/src/shared/scripts/schemas/skf-update-result-envelope.v1.json +11 -3
- package/src/shared/scripts/schemas/skill-brief.v1.json +1 -1
- package/src/shared/scripts/skf-build-change-manifest.py +1180 -189
- package/src/shared/scripts/skf-classify-changed-files.py +15 -10
- package/src/shared/scripts/skf-count-tokens.py +54 -10
- package/src/shared/scripts/skf-detect-workspaces.py +67 -0
- package/src/shared/scripts/skf-emit-result-envelope.py +64 -20
- package/src/shared/scripts/skf-extract-public-api.py +472 -120
- package/src/shared/scripts/skf-extraction-inventory.py +10 -6
- package/src/shared/scripts/skf-extraction-snapshot.py +6 -6
- package/src/shared/scripts/skf-parse-gaps.py +136 -26
- package/src/shared/scripts/skf-render-metadata-stats.py +1 -1
- package/src/shared/scripts/skf-resolve-customization.py +197 -0
- package/src/shared/scripts/skf-shard-body.py +239 -41
- package/src/shared/scripts/skf-validate-output.py +12 -3
- package/src/shared/scripts/skf-verify-provenance-completeness.py +574 -45
- package/src/shared/tool-requirements.yaml +8 -2
- package/src/skf-analyze-source/SKILL.md +3 -2
- package/src/skf-analyze-source/assets/skill-brief-schema.md +1 -1
- package/src/skf-analyze-source/references/generate-briefs.md +1 -1
- package/src/skf-analyze-source/references/headless-contract.md +1 -1
- package/src/skf-audit-skill/SKILL.md +4 -3
- package/src/skf-audit-skill/references/re-index.md +2 -2
- package/src/skf-audit-skill/references/report.md +1 -1
- package/src/skf-brief-skill/SKILL.md +3 -2
- package/src/skf-brief-skill/assets/skill-brief-schema.md +2 -2
- package/src/skf-brief-skill/references/invocation-contract.md +2 -2
- package/src/skf-brief-skill/references/scope-definition.md +4 -4
- package/src/skf-campaign/SKILL.md +3 -2
- package/src/skf-campaign/references/campaign-contracts.md +2 -0
- package/src/skf-campaign/references/step-05-skill-loop.md +8 -3
- package/src/skf-campaign/references/step-10-export.md +3 -2
- package/src/skf-create-skill/SKILL.md +4 -3
- package/src/skf-create-skill/assets/skill-sections.md +1 -1
- package/src/skf-create-skill/references/ccc-index-check.md +1 -1
- package/src/skf-create-skill/references/component-extraction.md +1 -1
- package/src/skf-create-skill/references/extract.md +3 -2
- package/src/skf-create-skill/references/extraction-patterns.md +3 -3
- package/src/skf-create-skill/references/generate-artifacts.md +1 -1
- package/src/skf-create-skill/references/report.md +8 -3
- package/src/skf-create-skill/references/sub/ccc-discover.md +1 -1
- package/src/skf-create-skill/references/validate.md +1 -1
- package/src/skf-create-stack-skill/SKILL.md +3 -2
- package/src/skf-create-stack-skill/references/generate-output.md +1 -1
- package/src/skf-create-stack-skill/references/invocation-contract.md +1 -1
- package/src/skf-create-stack-skill/references/parallel-extract.md +2 -2
- package/src/skf-create-stack-skill/references/report.md +1 -1
- package/src/skf-create-stack-skill/references/validate.md +1 -1
- package/src/skf-drop-skill/SKILL.md +3 -2
- package/src/skf-drop-skill/references/invocation-contract.md +1 -1
- package/src/skf-drop-skill/references/report.md +1 -1
- package/src/skf-export-skill/SKILL.md +3 -2
- package/src/skf-export-skill/assets/snippet-format.md +5 -5
- package/src/skf-export-skill/references/generate-snippet.md +22 -15
- package/src/skf-export-skill/references/invocation-contract.md +1 -1
- package/src/skf-export-skill/references/load-skill.md +1 -1
- package/src/skf-export-skill/references/summary.md +1 -1
- package/src/skf-export-skill/references/token-report.md +29 -9
- package/src/skf-forger/SKILL.md +3 -2
- package/src/skf-forger/references/pipeline-mode.md +2 -2
- package/src/skf-quick-skill/SKILL.md +4 -3
- package/src/skf-quick-skill/references/finalize.md +1 -1
- package/src/skf-quick-skill/references/halt-contract.md +2 -2
- package/src/skf-quick-skill/references/write-and-validate.md +1 -1
- package/src/skf-refine-architecture/SKILL.md +8 -9
- package/src/skf-refine-architecture/customize.toml +1 -1
- package/src/skf-refine-architecture/references/report.md +1 -1
- package/src/skf-rename-skill/SKILL.md +3 -2
- package/src/skf-rename-skill/references/invocation-contract.md +2 -2
- package/src/skf-setup/SKILL.md +3 -2
- package/src/skf-setup/references/ccc-index.md +1 -1
- package/src/skf-setup/references/invocation-contract.md +1 -1
- package/src/skf-test-skill/SKILL.md +3 -2
- package/src/skf-test-skill/assets/output-section-formats.md +6 -4
- package/src/skf-test-skill/references/coherence-check.md +11 -8
- package/src/skf-test-skill/references/coverage-check-branches.md +81 -0
- package/src/skf-test-skill/references/coverage-check-tiers.md +3 -3
- package/src/skf-test-skill/references/coverage-check.md +60 -97
- package/src/skf-test-skill/references/init.md +2 -1
- package/src/skf-test-skill/references/inventory-contract.md +27 -0
- package/src/skf-test-skill/references/invocation-contract.md +3 -3
- package/src/skf-test-skill/references/migration-section-rules.md +26 -14
- package/src/skf-test-skill/references/report.md +5 -5
- package/src/skf-test-skill/references/scoring-rules.md +8 -7
- package/src/skf-test-skill/references/signature-comparison-rules.md +72 -0
- package/src/skf-test-skill/references/source-access-protocol.md +14 -10
- package/src/skf-test-skill/references/step-hard-gate.md +1 -0
- package/src/skf-test-skill/scripts/build-result-context.py +20 -3
- package/src/skf-test-skill/scripts/gap-ledger.py +456 -176
- package/src/skf-test-skill/scripts/load-coverage-inputs.py +516 -79
- package/src/skf-test-skill/scripts/locate-export-segments.py +685 -76
- package/src/skf-test-skill/scripts/reconcile-coverage.py +192 -39
- package/src/skf-test-skill/scripts/score-signatures.py +1205 -65
- package/src/skf-test-skill/scripts/stage-helper-payload.py +64 -13
- package/src/skf-test-skill/scripts/validate-inventory.py +350 -35
- package/src/skf-test-skill/templates/test-report-template.md +1 -0
- package/src/skf-update-skill/SKILL.md +3 -2
- package/src/skf-update-skill/references/detect-changes.md +1 -1
- package/src/skf-update-skill/references/gap-driven.md +54 -33
- package/src/skf-update-skill/references/invocation-contract.md +1 -1
- package/src/skf-update-skill/references/merge.md +15 -10
- package/src/skf-update-skill/references/re-extract.md +3 -1
- package/src/skf-update-skill/references/report.md +6 -5
- package/src/skf-update-skill/references/write.md +42 -20
- package/src/skf-verify-stack/SKILL.md +4 -3
- package/tools/cli/lib/tool-check.js +158 -2
|
@@ -2,8 +2,11 @@
|
|
|
2
2
|
nextStepFile: 'coherence-check.md'
|
|
3
3
|
outputFile: '{report_file}'
|
|
4
4
|
scoringRulesFile: 'references/scoring-rules.md'
|
|
5
|
+
signatureRulesFile: 'references/signature-comparison-rules.md'
|
|
6
|
+
inventoryContractFile: 'references/inventory-contract.md'
|
|
5
7
|
sourceAccessProtocol: 'references/source-access-protocol.md'
|
|
6
8
|
coverageTiersFile: 'references/coverage-check-tiers.md'
|
|
9
|
+
coverageBranchesFile: 'references/coverage-check-branches.md'
|
|
7
10
|
validateInventoryScript: 'scripts/validate-inventory.py'
|
|
8
11
|
coverageInputsScript: 'scripts/load-coverage-inputs.py'
|
|
9
12
|
scoreSignaturesScript: 'scripts/score-signatures.py'
|
|
@@ -33,7 +36,7 @@ detectWorkspacesProbeOrder:
|
|
|
33
36
|
|
|
34
37
|
## STEP GOAL:
|
|
35
38
|
|
|
36
|
-
Compare
|
|
39
|
+
Compare what SKILL.md documents (exports, functions, classes, types, interfaces) with the source API surface: missing documentation, undocumented exports, signature mismatches, at the forge tier's depth.
|
|
37
40
|
|
|
38
41
|
**Halt envelope.** Every HALT in this step names its `halt_reason` and phase and carries exit code 1. It releases the run lock first, whatever the release prints: from `{project-root}`, run `uv run {runLockHelper} release --lock "{forge_version}/.test-skill.lock" --owner "{run_owner}"` (SKILL.md Workflow Rules). In headless mode it then writes `{"phase": "<phase>", "reason": "<the halt message>", "halt_reason": "<halt_reason>", "skill_name": "{skill_name}", "report_path": "{report_file}"}`, adding `"path"` when the halt names one, to `{run_dir}/halt.json` and runs:
|
|
39
42
|
|
|
@@ -43,11 +46,11 @@ uv run {emitEnvelopeHelper} emit-halt --workflow skf-test-skill --run-dir "{run_
|
|
|
43
46
|
|
|
44
47
|
Display the line it prints, as the run's last line, then stop. If the emitter exits non-zero or prints no line, display the halt message alone. An interactive HALT displays its message and emits nothing.
|
|
45
48
|
|
|
46
|
-
**Run files.** Every script below writes its result into the run folder `{run_dir}` that init.md §6b created,
|
|
49
|
+
**Run files.** Every script below writes its result into the run folder `{run_dir}` that init.md §6b created, where the next one reads it by path. A subagent's response is saved there exactly as it came back, fence and all (the subagent writes it to the path it is given, or the parent saves it, with the Write tool), and passed by path, so a quote or an apostrophe in it never meets a shell. Every count, name list and score a script computes is read from the file it wrote, never re-derived by hand or copied between commands. Each command runs from `{project-root}`; a `{...Script}` path resolves relative to the skill root.
|
|
47
50
|
|
|
48
51
|
### 0. Check for Docs-Only Mode
|
|
49
52
|
|
|
50
|
-
A skill is docs-only when every citation it writes is `[EXT:...]`,
|
|
53
|
+
A skill is docs-only when every citation it writes is `[EXT:...]`, none local (`[AST:...]`, `[SRC:...]`, or a stack's `[from skill: ...]`). The citation census decides it:
|
|
51
54
|
|
|
52
55
|
```bash
|
|
53
56
|
uv run {coverageInputsScript} census --skill-dir "{resolved_skill_package}" --output "{run_dir}/census.json"
|
|
@@ -55,68 +58,56 @@ uv run {coverageInputsScript} census --skill-dir "{resolved_skill_package}" --ou
|
|
|
55
58
|
|
|
56
59
|
Bind `{docs_only_mode}` ← `docsOnly`. Exit 1 printed its error on stderr: HALT with it (`halt_reason: "helper-failed"`, phase `coverage-check:census`). §0b sets `{docs_only_mode}` too when source access reaches State 5.
|
|
57
60
|
|
|
58
|
-
**A docs-only run
|
|
59
|
-
|
|
60
|
-
"**Docs-only skill detected.** Coverage check evaluates documentation completeness rather than source code coverage."
|
|
61
|
+
**A docs-only run**: load `{coverageBranchesFile}` and follow its **Docs-Only Run** section.
|
|
61
62
|
|
|
62
63
|
A source-based skill continues with §0b.
|
|
63
64
|
|
|
64
65
|
### 0b. Load Source Access Protocol
|
|
65
66
|
|
|
66
|
-
Load `{sourceAccessProtocol}` and follow both sections: **Source API Surface Definition** (the
|
|
67
|
+
Load `{sourceAccessProtocol}` and follow both sections: **Source API Surface Definition** (the denominator clause §2b picks) and **Source Access Resolution** (the 5-state waterfall that decides how the source is read and sets `analysis_confidence`; State 5 makes the run docs-only).
|
|
67
68
|
|
|
68
|
-
**Workspace layout.** The Source API Surface Definition's monorepo tests read `{source_is_monorepo}
|
|
69
|
+
**Workspace layout.** The Source API Surface Definition's monorepo tests read `{source_is_monorepo}`. At State 1 (local source), resolve `{detectWorkspacesHelper}` ← first existing path in `{detectWorkspacesProbeOrder}` and run (`{stageHelperPayloadScript}` pipes it the tree and root manifests from disk; leave `--brief` out with no brief):
|
|
69
70
|
|
|
70
71
|
```bash
|
|
71
|
-
uv run {stageHelperPayloadScript} detect-workspaces --source-root "{source_path}" | uv run {detectWorkspacesHelper}
|
|
72
|
+
uv run {stageHelperPayloadScript} detect-workspaces --source-root "{source_path}" --brief "{forge_data_folder}/{skill_name}/skill-brief.yaml" | uv run {detectWorkspacesHelper}
|
|
72
73
|
```
|
|
73
74
|
|
|
74
|
-
Bind `{source_is_monorepo}` ← `is_monorepo` and `{source_workspace_kind}` ← `manifest_kind`. At States 2 to 5, or when the helper does not resolve or the command exits non-zero, the layout is unknown: bind `{source_is_monorepo}` ← false. §5 records the layout.
|
|
75
|
+
A workspace whose members `scope.exclude` drops is no workspace. Bind `{source_is_monorepo}` ← `is_monorepo` and `{source_workspace_kind}` ← `manifest_kind`. At States 2 to 5, or when the helper does not resolve or the command exits non-zero, the layout is unknown: bind `{source_is_monorepo}` ← false. §5 records the layout.
|
|
75
76
|
|
|
76
|
-
**Clause flags.** Every `surface` command in §2 takes `--monorepo` when `{source_is_monorepo}` is true, and the flags the Source API Surface Definition leaves to you.
|
|
77
|
+
**Clause flags.** Every `surface` command, in §2 or in a branch file it loads, takes `--monorepo` when `{source_is_monorepo}` is true, and the flags the Source API Surface Definition leaves to you.
|
|
77
78
|
|
|
78
79
|
### 1. Extract Documented Exports from SKILL.md
|
|
79
80
|
|
|
80
|
-
|
|
81
|
+
First write the inventory's floor, the names the skill documents in a fixed form, which the inventory must keep:
|
|
81
82
|
|
|
82
|
-
```
|
|
83
|
-
{
|
|
84
|
-
"exports": [
|
|
85
|
-
{"name": "functionName", "kind": "function", "params": "...", "return_type": "...", "description": "..."},
|
|
86
|
-
{"name": "ClassName", "kind": "class", "methods": ["..."], "properties": ["..."], "description": "..."},
|
|
87
|
-
{"name": "TypeName", "kind": "type", "fields": ["..."], "description": "..."},
|
|
88
|
-
{"name": "useHook", "kind": "hook", "usage_signature": "...", "description": "..."}
|
|
89
|
-
],
|
|
90
|
-
"references": ["references/api-reference.md"],
|
|
91
|
-
"cross_check_mismatches": [
|
|
92
|
-
{"export": "functionName", "skill_md_line": 42, "reference_file": "references/api-reference.md", "reference_line": 18, "issue": "SKILL.md shows (date: Date) => string, the reference (date: Date, format?: string) => string"}
|
|
93
|
-
]
|
|
94
|
-
}
|
|
83
|
+
```bash
|
|
84
|
+
uv run {coverageInputsScript} floor --skill-dir "{resolved_skill_package}" --output "{run_dir}/inventory-floor.json"
|
|
95
85
|
```
|
|
96
86
|
|
|
97
|
-
|
|
87
|
+
Exit 1 printed its error on stderr: HALT with it (`halt_reason: "helper-failed"`, phase `coverage-check:inventory`).
|
|
98
88
|
|
|
99
|
-
|
|
89
|
+
Delegate the reading of the skill under test to a subagent, given `{run_dir}` and the path to SKILL.md and to its `references/` directory when one exists: it reads SKILL.md, and every `references/*.md` file too when SKILL.md's `## Full` headings are absent or stubs. It is given `{run_dir}/inventory-floor.json` too, and lists each name there with an origin in a file it reads, choosing its kind from the name's `kindLabel`, the skill's own label, when it has one. For each export two or more of the files it reads document, it compares the SKILL.md body with each reference file, and two reference files with each other only when the body does not settle them (it does not document the export, or agrees with both), by the rules of `{signatureRulesFile}`, whose absolute path (under `{skill-root}`) it is given: parameters, the return type, and descriptions that contradict (brief against detailed is fine). That file alone says what a form may leave out. It writes only this JSON, with no prose and no commentary, to `{run_dir}/inventory-response.txt` with the Write tool, never a shell, and returns that path, or returns the JSON itself when it cannot write files. The JSON's shape and the contract the validator checks are in `{inventoryContractFile}`, whose absolute path (under `{skill-root}`) the subagent is given.
|
|
90
|
+
|
|
91
|
+
When the subagent returned the JSON rather than the path, or the file is absent or empty afterwards, save its response, as it came back, to `{run_dir}/inventory-response.txt` with the Write tool. The documented inventory is the one §1a validates: do not load SKILL.md or the reference files into the parent context. Without subagents, the parent loads `{inventoryContractFile}` and builds the same JSON from the same files and `inventory-floor.json` and saves it to that file; §1a's checks still run on it.
|
|
100
92
|
|
|
101
93
|
#### 1a. Parent-Side Schema Validation + Spot-Check
|
|
102
94
|
|
|
103
|
-
Before anything consumes the inventory, `{validateInventoryScript}` checks it against the §1 contract (a wrapping fence stripped)
|
|
95
|
+
Before anything consumes the inventory, `{validateInventoryScript}` checks it against the §1 contract (a wrapping fence stripped), spot-checks sampled names against SKILL.md and the subagent's listed reference files, and refuses it when it leaves out a floor name those files write. The look-up is the script's, never a grep: an export named `$state` or `a.b` is neither a shell variable nor a regex.
|
|
104
96
|
|
|
105
97
|
```bash
|
|
106
|
-
uv run {validateInventoryScript} --input "{run_dir}/inventory-response.txt" --skill-package "{resolved_skill_package}" --output "{run_dir}/inventory.json"
|
|
98
|
+
uv run {validateInventoryScript} --input "{run_dir}/inventory-response.txt" --skill-package "{resolved_skill_package}" --output "{run_dir}/inventory.json" --floor "{run_dir}/inventory-floor.json"
|
|
107
99
|
```
|
|
108
100
|
|
|
109
101
|
- `valid` true (exit 0): every later command reads `{run_dir}/inventory.json`.
|
|
110
|
-
- `valid` false (exit 2)
|
|
111
|
-
- Exit 1 (no input, a file that cannot be read, or a package with no readable SKILL.md): correct the file and run it again.
|
|
102
|
+
- `valid` false (exit 2), or exit 1: load `{coverageBranchesFile}` and follow its **Inventory Failures** section. Invalid a second time: HALT (`halt_reason: "inventory-invalid"`, phase `coverage-check:inventory`): "coverage-check: the subagent inventory failed {schema validation | the ground-truth spot-check | the floor check} twice: {the violations joined}."
|
|
112
103
|
|
|
113
104
|
### 1b. Cross-Check Split-Body Consistency
|
|
114
105
|
|
|
115
|
-
Only when the validated inventory's `references` is non-empty.
|
|
106
|
+
Only when the validated inventory's `references` is non-empty. Each `cross_check_mismatches` entry is a High `split-body-mismatch` gap. §5b records it titled `Split-body mismatch: {export} ({reference_file})`, with its `export` and its Source at the side the source contradicts (`{reference_file}` when no source settles it): update-skill edits that file to match the other without reading the source. An entry for two references the body does not settle is titled `Split-body mismatch: {export} ({other_file}, {reference_file})`.
|
|
116
107
|
|
|
117
108
|
### 2. Analyze Source Code (Tier-Dependent)
|
|
118
109
|
|
|
119
|
-
Identify the public API surface from the package entry points (
|
|
110
|
+
Identify the public API surface from the package entry points (0b), as deep as the tier allows. Whichever branch runs, `{coverageInputsScript} surface` writes it to `{run_dir}/surface.json`, which §2b, §2c, §4 and step 5 read. Names read by eye reach it as per-file results, `{"file": "<path>", "exports_found": [<each name>], "signature_mismatches": []}` for each file, saved to `{run_dir}/per-file-<n>.json` with the Write tool and given to `surface` with one `--per-file` each.
|
|
120
111
|
|
|
121
112
|
**Exits.** Every `{coverageInputsScript}` and `{scoreSignaturesScript}` command in §2 and §2b exits 0, or exits 1 with its error on stderr: HALT with it (`halt_reason: "helper-failed"`, phase `coverage-check:surface`). `surface` with a `--per-file`, and `score`, also exit 2 when a subagent response breaks the schema: re-dispatch that subagent once with its `violations[]` appended (or correct a result the main thread wrote), save the new response over the old one and run the command again; a second failure HALTs the same way.
|
|
122
113
|
|
|
@@ -138,38 +129,36 @@ Build the surface from it, with the metadata, the brief when the skill has one a
|
|
|
138
129
|
uv run {coverageInputsScript} surface --extraction "{run_dir}/extract-full.json" --metadata "{resolved_skill_package}/metadata.json" [--provenance "{forge_provenance_map}"] [--brief "{forge_data_folder}/{skill_name}/skill-brief.yaml"] [--monorepo] [--pattern-reference-intent] [--curated-subset] --output "{run_dir}/surface.json"
|
|
139
130
|
```
|
|
140
131
|
|
|
141
|
-
The surface holds
|
|
132
|
+
The surface holds what `{sourceAccessProtocol}` **What the Surface Holds** says (its exclusions and the stale-scope guard). §5's render lists `excluded.outsideScope`, `excluded.nestedEntries` and the restored names. Decide what to do from the JSON, not from the exit code:
|
|
142
133
|
|
|
143
134
|
- `extraction.fallback.needed` is true, or the runner printed no JSON on an exit other than 2: load `{coverageTiersFile}` and run its **Fallback Per-File Scan** in place of the rest of this tier.
|
|
144
|
-
- `extraction.
|
|
145
|
-
-
|
|
135
|
+
- `extraction.readByEye` is non-empty (a file the run could not parse or decode, or a failed call left unread): keep the surface, read by eye only the files it names, save a per-file result for each, and run `surface` again with them, once (the second call starts no new pass).
|
|
136
|
+
- In the Coverage Analysis section, list each `readByEye` file as read by eye, each `extraction.unread` file as unread.
|
|
137
|
+
- `extractionGaps[]` names exports no recipe found: they are on the surface; read their signature by eye at their `file` and `line`.
|
|
146
138
|
|
|
147
139
|
**Compare the signatures.** A script lists the comparisons; subagents judge them:
|
|
148
140
|
|
|
149
141
|
```bash
|
|
150
|
-
uv run {scoreSignaturesScript} plan --inventory "{run_dir}/inventory.json" --surface "{run_dir}/surface.json" --output "{run_dir}/signature-plan.json"
|
|
142
|
+
uv run {scoreSignaturesScript} plan --inventory "{run_dir}/inventory.json" --surface "{run_dir}/surface.json" --skill-dir "{resolved_skill_package}" --output "{run_dir}/signature-plan.json"
|
|
151
143
|
```
|
|
152
144
|
|
|
153
|
-
For each `files[]` entry, delegate one subagent with its checks. For each check it reads the full declaration at `{file}:{line}` (all of it: `signatureLine` is one physical line, and a multi-line signature cut there
|
|
145
|
+
For each `files[]` entry, delegate one subagent with its checks and the absolute path of `{signatureRulesFile}` (under `{skill-root}`). For each check it reads the full declaration at `{file}:{line}` (all of it: `signatureLine` is one physical line, and a multi-line signature cut there reads as a mismatch), compares it with the check's `documented` signature by the rules of that file, and returns only this JSON, with no prose and no markdown fences:
|
|
154
146
|
|
|
155
147
|
```json
|
|
156
|
-
{"file": "src/utils.ts", "signature_mismatches": [{"name": "formatDate", "line": 42, "source_sig": "(date: Date, format?: string) => string", "documented_sig": "(date: Date) => string", "issue": "missing optional parameter 'format'"}]}
|
|
148
|
+
{"file": "src/utils.ts", "signature_mismatches": [{"name": "formatDate", "line": 42, "source_sig": "(date: Date, format?: string) => string", "documented_sig": "(date: Date) => string", "issue": "missing optional parameter 'format'", "difference": "omitted-optional"}]}
|
|
157
149
|
```
|
|
158
150
|
|
|
159
|
-
|
|
151
|
+
Tag a mismatch's `difference` as `{signatureRulesFile}` says: `score` (§2b) checks the facts. One tagged `annotation-contradicted` also gives `evidence_line`, the body line that returns the documented type.
|
|
160
152
|
|
|
161
|
-
|
|
153
|
+
A check with `candidates` reads each and reports a mismatch only when the documented signature matches none: one, at the closest candidate, its `file` and `line` on the mismatch object (the result's `file` stays its `files[]` entry) and its `difference` tagged against it.
|
|
162
154
|
|
|
163
|
-
|
|
164
|
-
- **State 2** (`{forge_provenance_map}`, the provenance map init.md §2 bound): the union of its named exports and the `metadata.json` `exports[]` names, with `--fold`, `--keep` and `--fold-prefix` as the Source API Surface Definition's canonicalization says:
|
|
155
|
+
A check with `alternatives` (the other forms the skill documents for the name) is compared with each of them too, as `{signatureRulesFile}` says, each mismatch naming its form as `form`.
|
|
165
156
|
|
|
166
|
-
|
|
167
|
-
uv run {coverageInputsScript} surface --provenance "{forge_provenance_map}" --metadata "{resolved_skill_package}/metadata.json" [--brief "{forge_data_folder}/{skill_name}/skill-brief.yaml"] [--fold] [--keep <variant>] [--fold-prefix <prefix>] [--monorepo] [--pattern-reference-intent] [--curated-subset] --output "{run_dir}/surface.json"
|
|
168
|
-
```
|
|
157
|
+
Save each response to `{run_dir}/signatures-<n>.txt` with the Write tool. Without subagents, load `{signatureRulesFile}`, compare in the main thread by its rules and save the same JSON.
|
|
169
158
|
|
|
170
|
-
|
|
171
|
-
|
|
172
|
-
|
|
159
|
+
**Deep Tier (ast-grep + gh + QMD):** the Forge tier checks, and no coverage check of its own: init.md §5b's drift guard checks the local tree against the pinned commit, at any tier, when both exist; the recipe runner traces each re-export to the file that defines it when it runs (the fallback per-file scan does not); and the Forge tier's signature check compares each documented signature with its declaration. A documentation error you see that no script records (an import path whose module does not export the name, a type member that differs from its declaration) goes in coverage-check §5b's hand-written records as an Info `observation` titled `Not checked by script: {name}`, with `export` `{name}`, its Source at the documenting line and the remediation "Correct the documentation to match the source; no script checks this."
|
|
160
|
+
|
|
161
|
+
**States 2 to 4 (no local source):** the surface is what the skill recorded, so there is no signature to score: load `{coverageBranchesFile}` and follow its **States 2 to 4** section.
|
|
173
162
|
|
|
174
163
|
### 2b. Resolve the Denominator and Guard Zero Exports
|
|
175
164
|
|
|
@@ -179,7 +168,7 @@ Load the metadata and provenance-map counts once, for §2c, §4b and the numerat
|
|
|
179
168
|
uv run {coverageInputsScript} metadata --metadata "{resolved_skill_package}/metadata.json" [--provenance "{forge_provenance_map}"] --output "{run_dir}/coverage-inputs.json"
|
|
180
169
|
```
|
|
181
170
|
|
|
182
|
-
**Pick the denominator.** `surface.json` `denominator` holds the Source API Surface Definition clause (§0b) `surface` picked: its §2c `branch` (`barrel`, `scalar` or `stack`), `surfaceSet` (the set the barrel branch counts and the signature score reads) and `ladderStep`. When `absentSet` names a set the surface lacks (the scope sets need the brief's globs and names that carry their files, `subpaths` and `root` an extraction), `surfaceSet` is `all` and the `Denominator:` line (§4) adds `({absentSet} set absent: all set used)`. When `rebuild` is `provenance-fold`, rerun State 2's command
|
|
171
|
+
**Pick the denominator.** `surface.json` `denominator` holds the Source API Surface Definition clause (§0b) `surface` picked: its §2c `branch` (`barrel`, `scalar` or `stack`), `surfaceSet` (the set the barrel branch counts, as the scalar one does when `setCounted` is true, and the signature score reads) and `ladderStep`. When `absentSet` names a set the surface lacks (the scope sets need the brief's globs and names that carry their files, `subpaths` and `root` an extraction), `surfaceSet` is `all` and the `Denominator:` line (§4) adds `({absentSet} set absent: all set used)`. When `rebuild` is `provenance-fold`, load `{coverageBranchesFile}` and rerun State 2's command from its **States 2 to 4** section without `--metadata` and with `--fold` and `--pattern-reference-intent`, running nothing else from that section, and read the `denominator` it writes.
|
|
183
172
|
|
|
184
173
|
- **A stack skill** (`branch` `stack`): `coverage-inputs.json` `stack.denominator`. Its own barrel is empty by design, and its `[from skill: …]` citations never make it docs-only. **If `stack.denominator` is 0**: HALT (`halt_reason: "indeterminate-surface"`, phase `coverage-check:denominator`) with `Error: stack composition surface empty: {skill_name} cites no contracts, libraries, or integration pairs, so Export Coverage is undefined. Verify the stack was compiled from at least one constituent skill.` Do not write the Coverage Analysis section; this is an indeterminate state, not a FAIL.
|
|
185
174
|
- **A docs-only skill**: the `docsOnly` branch, over the inventory.
|
|
@@ -190,7 +179,7 @@ uv run {coverageInputsScript} metadata --metadata "{resolved_skill_package}/meta
|
|
|
190
179
|
uv run {numeratorVerifyScript} --inputs "{run_dir}/coverage-inputs.json" --skill-dir "{resolved_skill_package}" --output "{run_dir}/numerator.json"
|
|
191
180
|
```
|
|
192
181
|
|
|
193
|
-
§2c's scalar branch
|
|
182
|
+
Unless §2c's scalar branch counts its set, it takes the `verified` count as the numerator when `inflated` is true; §4b records the gap. Exit 1 or 2: HALT with its message (`halt_reason: "helper-failed"`, phase `coverage-check:numerator`).
|
|
194
183
|
|
|
195
184
|
**If the chosen denominator is 0 AND `docs_only_mode == false` AND `metadata.json.skill_type != "stack"`:** HALT (`halt_reason: "indeterminate-surface"`, phase `coverage-check:denominator`) with:
|
|
196
185
|
|
|
@@ -205,17 +194,17 @@ Export Coverage is undefined, and scoring would yield a vacuous PASS. Fix one of
|
|
|
205
194
|
|
|
206
195
|
Do not write the Coverage Analysis section or attach a score: this is an indeterminate state, not a FAIL. **If `docs_only_mode == true` and the validated inventory documents no item:** HALT the same way with "docs-only skill declares zero items: no API surface to test".
|
|
207
196
|
|
|
208
|
-
**Score the signatures** (Forge, Forge+ and Deep tier at State 1, once §2 compared them). Give `{scoreSignaturesScript}` `denominator.surfaceSet` as `--surface-set`. Type Coverage then counts that set's types alone, so a type the clause leaves out (a stratified skill's deferred Tier B) lowers neither Export Coverage nor Type Coverage. Give it one `--results` per saved response (the fallback scan's `per-file-<n>.txt`, else the `signatures-<n>.txt`):
|
|
197
|
+
**Score the signatures** (Forge, Forge+ and Deep tier at State 1, once §2 compared them). Give `{scoreSignaturesScript}` `denominator.surfaceSet` as `--surface-set`. Type Coverage then counts that set's types alone, so a type the clause leaves out (a stratified skill's deferred Tier B) lowers neither Export Coverage nor Type Coverage. Give it one `--results` per saved response (the fallback scan's `per-file-<n>.txt`, else the `signatures-<n>.txt`), and `--source-root` whenever the source is local (State 1, Forge tier or above), under a drift override too, since the judges read that tree:
|
|
209
198
|
|
|
210
199
|
```bash
|
|
211
|
-
uv run {scoreSignaturesScript} score --inventory "{run_dir}/inventory.json" --surface "{run_dir}/surface.json" --surface-set <set> --results "{run_dir}/signatures-<n>.txt" --output "{run_dir}/signatures.json" --gaps-output "{run_dir}/signature-gaps.json"
|
|
200
|
+
uv run {scoreSignaturesScript} score --inventory "{run_dir}/inventory.json" --surface "{run_dir}/surface.json" --surface-set <set> --plan "{run_dir}/signature-plan.json" [--source-root "{source_path}"] --results "{run_dir}/signatures-<n>.txt" --output "{run_dir}/signatures.json" --gaps-output "{run_dir}/signature-gaps.json"
|
|
212
201
|
```
|
|
213
202
|
|
|
214
|
-
It
|
|
203
|
+
It grades each mismatch of a compared name from facts it checks, never a `difference` alone, into `signature-gaps.json`: a Low `signature-mismatch` gap for a `parameter-name` difference in a file the plan offered when it is TS/JS, Go, Rust, Java, C or C++, an Info `observation` titled `Partial signature: {name}` for an `omitted-optional` one on a `partial` form, an Info `observation` titled `Annotation contradicted: {name}` (no change to the skill: report the annotation upstream) for an `annotation-contradicted` one whose facts check (a documented return type, the same parameters on both sides, and an `evidence_line` between the declaration and the next one, whose line returns in the `--source-root` file), a Critical `signature-mismatch` gap otherwise (Python included). A signature gap is titled `Signature mismatch: {name}`, with `export` `{name}`, its Source at `{file}:{line}` and both signatures in its Issue (for a name documented in several forms, each gap's title ends `(form {n} of {m})` and its Issue names that form); §5b appends that file to the ledger. A Low or Info one counts as matching, and §5 renders it so. §5's render lists its `warnings` (a mismatch for a name never compared, a category with nothing to compare).
|
|
215
204
|
|
|
216
205
|
### 2c. Reconcile Documented vs Source Surface (Deterministic Intersection)
|
|
217
206
|
|
|
218
|
-
`{reconcileScript}` reconciles the §1 inventory with the source surface and writes `{run_dir}/coverage.json`, whose `exportCoverage` is the Export Coverage
|
|
207
|
+
`{reconcileScript}` reconciles the §1 inventory with the source surface and writes `{run_dir}/coverage.json`, whose `exportCoverage` is the Export Coverage step 5 hands to the scoring script. Run the branch §2b picked:
|
|
219
208
|
|
|
220
209
|
1. **Barrel**, with the name set §2b picked as `--surface-set`:
|
|
221
210
|
|
|
@@ -223,19 +212,9 @@ It writes each mismatch of a compared name to `signature-gaps.json` as a Critica
|
|
|
223
212
|
uv run {reconcileScript} --denominator-source barrel --inventory "{run_dir}/inventory.json" --surface "{run_dir}/surface.json" --surface-set <set> --output "{run_dir}/coverage.json"
|
|
224
213
|
```
|
|
225
214
|
|
|
226
|
-
2. **Scalar
|
|
227
|
-
|
|
228
|
-
```bash
|
|
229
|
-
uv run {reconcileScript} --denominator-source scalar --coverage-inputs "{run_dir}/coverage-inputs.json" --inventory "{run_dir}/inventory.json" --skill-dir "{resolved_skill_package}" --verified "{run_dir}/numerator.json" --output "{run_dir}/coverage.json"
|
|
230
|
-
```
|
|
215
|
+
2. **Scalar or stack**: load `{coverageBranchesFile}` and follow its **Scalar and Stack Branches** section, which runs the branch's command.
|
|
231
216
|
|
|
232
|
-
3. **
|
|
233
|
-
|
|
234
|
-
```bash
|
|
235
|
-
uv run {reconcileScript} --denominator-source stack --coverage-inputs "{run_dir}/coverage-inputs.json" --skill-dir "{resolved_skill_package}" --output "{run_dir}/coverage.json"
|
|
236
|
-
```
|
|
237
|
-
|
|
238
|
-
4. **Docs-only**: Export Coverage is the documentation completeness ratio, complete items over documented items, and `incomplete[]` lists each incomplete item with the fields it lacks:
|
|
217
|
+
3. **Docs-only**: Export Coverage is the documentation completeness ratio, complete over documented items, and `incomplete[]` lists each incomplete item with the fields it lacks:
|
|
239
218
|
|
|
240
219
|
```bash
|
|
241
220
|
uv run {reconcileScript} --denominator-source docsOnly --inventory "{run_dir}/inventory.json" --output "{run_dir}/coverage.json"
|
|
@@ -243,50 +222,32 @@ It writes each mismatch of a compared name to `signature-gaps.json` as a Critica
|
|
|
243
222
|
|
|
244
223
|
Exit 2 printed an `INVALID_INPUT` envelope: correct the input it names and run it again. Exit 1: HALT with its stderr (`halt_reason: "helper-failed"`, phase `coverage-check:reconcile`).
|
|
245
224
|
|
|
246
|
-
**A surplus is a signal, not a pass.** The scalar and stack branches count the numerator and the denominator from independent sets, so `documented` can exceed `denominator`: the script then reports `numeratorSurplus` above 0 and `coverageCapped: true`, holds `exportCoverage` at 100 and keeps the raw ratio in `coverageUncapped`, and §5's render states the surplus beside both counts. On a skill with no `scope.tier_a_include` glob that matches a file, a large surplus is the deflated-denominator signature: check `guards.deflation` before accepting the 100.
|
|
247
|
-
|
|
248
225
|
**Classify what the script found.** §5b records each of these gaps from the run files:
|
|
249
226
|
|
|
250
|
-
- **Missing names** (barrel
|
|
251
|
-
- **
|
|
252
|
-
- **Stale names** (barrel branch: `stale`): each is a documented name the enumerated source surface lacks. Where ast-grep read the source at the pinned commit (Forge, Forge+ or Deep tier, `analysis_confidence` `full`, `workspaceDrift` not `overridden` in the `{outputFile}` frontmatter) and init.md §2 bound `{forge_provenance_map}`, check the lines the skill cites for them: resolve `{verifyProvenanceCompletenessHelper}` ← first existing path in `{verifyProvenanceCompletenessProbeOrder}` and run once:
|
|
227
|
+
- **Missing names** (`missing`, barrel or scalar): Medium `missing-export` gaps (Medium `missing-type` for a name `signatures.json` lists in `missingTypes`), each at the file `surface.json` records for it, so update-skill can re-extract the export from it; when `guards.inflation.fires` on a stratified-scope or multi-entry `scope.include` set, one Medium `denominator-inflation` gap beside the guard's at the brief lists them instead. A missing export does not block: it lowers Export Coverage.
|
|
228
|
+
- **Stale names** (`stale` when `staleApplicable` is true): each is a documented name the enumerated source surface lacks. Where ast-grep read the source at the pinned commit (Forge, Forge+ or Deep tier, `analysis_confidence` `full`, `workspaceDrift` not `overridden` in the `{outputFile}` frontmatter) and init.md §2 bound `{forge_provenance_map}`, check the lines the skill cites for them: resolve `{verifyProvenanceCompletenessHelper}` ← first existing path in `{verifyProvenanceCompletenessProbeOrder}` and run once:
|
|
253
229
|
|
|
254
230
|
```bash
|
|
255
231
|
uv run {verifyProvenanceCompletenessHelper} classify-stale --names "{run_dir}/coverage.json" --provenance "{forge_provenance_map}" --source-root "{source_path}" --inventory "{run_dir}/inventory.json" -o "{run_dir}/stale.json"
|
|
256
232
|
```
|
|
257
233
|
|
|
258
|
-
A name it marks `fabricated: true` (every map entry of the name cites a missing file or one that does not define it) is a Critical `fabricated-signature` gap at its `source`. A name with a `defined_at` (the source still declares it there, an import never counting, a module so named counting unless the name's inventory kind rules it out), or whose `declared_in` lists several files of which the skill's `[AST:]`/`[SRC:]` citations on lines naming it cite exactly one, is a documented extra: an Info `observation` gap at that declaration, when `surface.json` has an `extraction` and its `excluded.outsideScope` does not list the name (a brief scoped its file out). Any other stale name (no `defined_at` and no single cited declaring file, one scoped out, no `extraction`, a dotted name, the conditions above not met, a helper that does not resolve or exits non-zero) is a Medium `stale-documentation` gap.
|
|
234
|
+
A name it marks `fabricated: true` (every map entry of the name cites a missing file or one that does not define it) is a Critical `fabricated-signature` gap at its `source`. A name with a `defined_at` (the source still declares it there, an import never counting, a module so named counting unless the name's inventory kind rules it out), or whose `declared_in` lists several files of which the skill's `[AST:]`/`[SRC:]` citations on lines naming it cite exactly one, is a documented extra: an Info `observation` gap at that declaration, when `surface.json` has an `extraction` and its `excluded.outsideScope` does not list the name (a brief scoped its file out). Any other stale name (no `defined_at` and no single cited declaring file, one scoped out, no `extraction`, a dotted name, the conditions above not met, a helper that does not resolve or exits non-zero) is a Medium `stale-documentation` gap; one the brief's globs scope out says which.
|
|
259
235
|
|
|
260
236
|
### 4. The Denominator Record
|
|
261
237
|
|
|
262
|
-
Record the denominator source in the Coverage Analysis section with the annotation `{sourceAccessProtocol}` gives `denominator.clause` (§4b gives a stack's and a reference app's): `Denominator: {barrel | empty barrel (…) | stratified (…) | multi-entry (…) | specific-modules (…) | pattern-reference (…) | stack composition (…) | pattern-surface (…) | docs-only completeness}`, and the `canonical.summary` line when `surface.json` has a fold. When `denominator.clause` is `stratified-scope` or `multi-entry`, append
|
|
263
|
-
|
|
264
|
-
```markdown
|
|
265
|
-
**Denominator Candidates** (stratified-scope audit trail):
|
|
266
|
-
- `stats.effective_denominator`: {statsEffectiveDenominator | absent} {← chosen on the scalar branch}
|
|
267
|
-
- `scope.tier_a_include` union: {tierAIncludeUnion | absent} {← chosen if the barrel branch's set}
|
|
268
|
-
- `scope.include` union: {scopeIncludeUnion | absent} {← chosen if the barrel branch's set}
|
|
269
|
-
- exports-map subpath union: {subpathUnion | absent} {← chosen if the barrel branch's set}
|
|
270
|
-
- root barrel: {rootBarrel | absent} {secondary candidate: root-barrel-vs-subpath-union audit}
|
|
271
|
-
```
|
|
238
|
+
Record the denominator source in the Coverage Analysis section with the annotation `{sourceAccessProtocol}` gives `denominator.clause` (§4b gives a stack's and a reference app's): `Denominator: {barrel | empty barrel (…) | stratified (…) | multi-entry (…) | specific-modules (…) | pattern-reference (…) | stack composition (…) | pattern-surface (…) | docs-only completeness}`, and the `canonical.summary` line when `surface.json` has a fold. When `denominator.clause` is `stratified-scope` or `multi-entry`, load `{coverageBranchesFile}` and append the block of its **Denominator Candidates** section after the `Denominator:` line, with every count from `surface.json` `candidates` (`absent` for a null one).
|
|
272
239
|
|
|
273
|
-
The guards raise coverage gaps too, which §5b records from `surface.json`: `guards.deflation.fires` is a Medium `metadata-drift` gap, `guards.inflation.fires` a Medium `denominator-inflation` gap and `guards.staleScope.fires` a Medium `brief-scope-stale` gap at the brief.
|
|
240
|
+
The guards raise coverage gaps too, which §5b records from `surface.json`: `guards.deflation.fires` is a Medium `metadata-drift` gap (on its `tier_a_include` basis, a stale stored count to recount), `guards.inflation.fires` a Medium `denominator-inflation` gap (with deflation, both name one fix: the brief's tier A list) and `guards.staleScope.fires` a Medium `brief-scope-stale` gap at the brief.
|
|
274
241
|
|
|
275
242
|
### 4b. Metadata Export-Count Coherence Cross-Check
|
|
276
243
|
|
|
277
|
-
`{
|
|
244
|
+
Load `{coverageBranchesFile}` and follow its **Metadata Count Cross-Check** section.
|
|
278
245
|
|
|
279
|
-
|
|
280
|
-
uv run {coherenceScript} --inputs "{run_dir}/coverage-inputs.json" > "{run_dir}/metadata-coherence.json"
|
|
281
|
-
```
|
|
282
|
-
|
|
283
|
-
A non-zero exit leaves the script's error, or nothing, in that file: delete it, so §5b skips its call and records no count finding, and note the error in the Coverage Analysis section. §5b records each `findings[]` entry: a Medium entry as a Medium `metadata-drift` gap and an Info entry as an Info `multi-denominator` gap. They describe data quality and change no denominator. Record the denominator of a stack (`Denominator: stack composition ({N} cited contracts)` or `({N} libraries + integration pairs)`, from `stack.basis`) and of a reference app (`Denominator: pattern-surface ({pattern_surfaces_documented})`).
|
|
284
|
-
|
|
285
|
-
**The numerator ground truth §2b ran.** `numerator.json` `inflated: true` (`verified < declared`) is a High `numerator-inflation` gap listing its `absent[]` names, which §5b records; on the scalar branch those names also replace §2c's missing-count gap. `inflated: false` or `skipped: true` is no finding.
|
|
246
|
+
**The numerator ground truth §2b ran.** `numerator.json` `inflated: true` (`verified < declared`) is a High `numerator-inflation` gap listing its `absent[]` names, which §5b records; on a scalar branch that counts no set those names also replace the missing-count gap in the Scalar and Stack Branches section of coverage-check-branches.md. `inflated: false` or `skipped: true` is no finding.
|
|
286
247
|
|
|
287
248
|
### 4c. Provenance Line Check
|
|
288
249
|
|
|
289
|
-
Check that the provenance map records each export at the line that defines it (the `def` or declaration line
|
|
250
|
+
Check that the provenance map records each export at the line that defines it (the `def` or declaration line, not a decorator or blank line above it), so update-skill can move a wrong line. Run this section only when `analysis_confidence` is `full`, `{forge_version}/provenance-map.json` exists, and the `{outputFile}` frontmatter's `workspaceDrift` is not `overridden` (init.md §5b: the drifted tree is not the pinned commit). Otherwise skip to section 5.
|
|
290
251
|
|
|
291
252
|
Resolve `{verifyProvenanceCompletenessHelper}` ← first existing path in `{verifyProvenanceCompletenessProbeOrder}`. If neither path exists, skip to section 5: the check is advisory. Otherwise run:
|
|
292
253
|
|
|
@@ -298,9 +259,9 @@ uv run {verifyProvenanceCompletenessHelper} verify \
|
|
|
298
259
|
-o "{run_dir}/provenance-verify.json"
|
|
299
260
|
```
|
|
300
261
|
|
|
301
|
-
Rely on the file, not the exit code (exit 1 means findings); on exit 2 the helper writes no file: skip to section 5. §5b records each `stale[]` item
|
|
262
|
+
Rely on the file, not the exit code (exit 1 means findings); on exit 2 the helper writes no file: skip to section 5. §5b records each `stale[]` item by its `reason`: `line-not-definition` is a Low `provenance-line` gap when its `definition_lines` holds a line, else an Info `provenance-unverified` gap (the rules may not cover its shape: the export is unverified, not gone); `file-missing` (the pinned source lacks the file) and `line-out-of-bounds` (the line is past its end, or below 1) are Low `provenance-line` gaps; `line-invalid` (not a line number) is an Info `provenance-unverified` gap. A bookkeeping variant (`X_def`, `X_exact`, `a11y_X`) the cited file never names is checked against `X`. Its Source is exactly the `file:line` the map records (the file alone for `line-invalid`), the citation update-skill's rule R5 reads. A name §2c's `fabricated-signature` gap covers takes no such gap.
|
|
302
263
|
|
|
303
|
-
This section reports none of the helper's
|
|
264
|
+
This section reports none of the helper's `missing`, `orphaned` or `citations` findings, and no gap it records blocks the gate.
|
|
304
265
|
|
|
305
266
|
### 5. Write the Coverage Analysis Section
|
|
306
267
|
|
|
@@ -340,7 +301,9 @@ uv run {gapLedgerScript} append --ledger "{ledgerFile}" --stage coverage-check -
|
|
|
340
301
|
uv run {gapLedgerScript} append --ledger "{ledgerFile}" --stage coverage-check --from provenance-line --input "{run_dir}/provenance-verify.json"
|
|
341
302
|
```
|
|
342
303
|
|
|
343
|
-
|
|
304
|
+
**A split-body mismatch's Source** is a line inside the skill package: when §1b found one, load `{coverageBranchesFile}` and follow its **Split-Body Sources** section.
|
|
305
|
+
|
|
306
|
+
Then write the records no script wrote, §1b's split-body mismatches, in the Ledger Record Format of `{outputFormatsFile}` (at the Deep tier, §2's `Not checked by script: {name}` observations too), each naming its export in `export` and in its title: the ledger tells two gaps apart by their title, Source and `export`, and update-skill takes the export's name from `export`. Write them as one JSON array on the lines between the two markers, exactly as they are: the quoted marker hands them to the script unchanged, quotes, apostrophes and `$` included. Run the command even when the array is empty (`[]`): the hard gate refuses to decide until every stage before it has recorded, with gaps or without.
|
|
344
307
|
|
|
345
308
|
```bash
|
|
346
309
|
uv run {gapLedgerScript} append --ledger "{ledgerFile}" --stage coverage-check <<'SKF_GAPS'
|
|
@@ -356,6 +319,6 @@ Rely on the JSON of each command:
|
|
|
356
319
|
|
|
357
320
|
### 6. Report Coverage Results
|
|
358
321
|
|
|
359
|
-
Report
|
|
322
|
+
Report to the user from §5's render: the {forge_tier} tier, the **Files Analyzed** count, the three category scores and the issue count; the Coverage Analysis section has the details.
|
|
360
323
|
|
|
361
324
|
Append `'coverage-check'` to `stepsCompleted` in the `{outputFile}` frontmatter, then load and execute {nextStepFile}.
|
|
@@ -313,13 +313,14 @@ thresholdSource: '{threshold_source}'
|
|
|
313
313
|
analysisConfidence: ''
|
|
314
314
|
toolingStatus: '{ok|frontmatter-validator-timeout}'
|
|
315
315
|
workspaceDrift: '{not-checked|ok|overridden}'
|
|
316
|
+
health_check_dispatched: false
|
|
316
317
|
testDate: '{run_id timestamp ISO-8601 UTC}'
|
|
317
318
|
stepsCompleted: ['init']
|
|
318
319
|
nextWorkflow: ''
|
|
319
320
|
---
|
|
320
321
|
```
|
|
321
322
|
|
|
322
|
-
`toolingStatus` is the `tooling_status` §3b set: score.md §3a reads it back for Cap 1. `threshold` and `thresholdSource` are the pass threshold §1b resolved and its source: score.md §1 reads them back, and score.md §7 rewrites `threshold` with the threshold the verdict was settled against. `analysisConfidence` stays empty until score.md §7 writes the source access coverage-check resolved: `full`, `provenance-map`, `metadata-only`, `remote-only` or `docs-only`.
|
|
323
|
+
Replace the template's frontmatter with this block (the two hold the same keys). `toolingStatus` is the `tooling_status` §3b set: score.md §3a reads it back for Cap 1. `threshold` and `thresholdSource` are the pass threshold §1b resolved and its source: score.md §1 reads them back, and score.md §7 rewrites `threshold` with the threshold the verdict was settled against. `analysisConfidence` stays empty until score.md §7, or step-hard-gate.md §3 on a run the hard gate blocks, writes the source access coverage-check resolved: `full`, `provenance-map`, `metadata-only`, `remote-only` or `docs-only`. report.md §4c halts on a report that holds none of them.
|
|
323
324
|
|
|
324
325
|
### 7. Report Initialization Status
|
|
325
326
|
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
# Documented Inventory Contract
|
|
2
|
+
|
|
3
|
+
The JSON coverage-check.md §1's reader writes to `{run_dir}/inventory-response.txt`: every export the skill documents, in this shape.
|
|
4
|
+
|
|
5
|
+
```json
|
|
6
|
+
{
|
|
7
|
+
"exports": [
|
|
8
|
+
{"name": "functionName", "kind": "function", "params": "...", "return_type": "...", "description": "..."},
|
|
9
|
+
{"name": "ClassName", "kind": "class", "methods": ["..."], "properties": ["..."], "description": "..."},
|
|
10
|
+
{"name": "TypeName", "kind": "type", "fields": ["..."], "description": "..."},
|
|
11
|
+
{"name": "useHook", "kind": "hook", "usage_signature": "...", "description": "...", "signature_form": "partial"},
|
|
12
|
+
{"name": "ComponentName", "kind": "constant", "usage_signature": "<ComponentName prop={value} />", "description": "..."},
|
|
13
|
+
{"name": "Card", "kind": "function", "params": "{ title }", "description": "..."},
|
|
14
|
+
{"name": "Card", "kind": "function", "params": "{ title: string; footer?: ReactNode }", "description": "..."},
|
|
15
|
+
{"name": "Note", "kind": "class", "params": "each with the base fields id, scope=Scope(); content, title=None", "description": "..."}
|
|
16
|
+
],
|
|
17
|
+
"references": ["references/api-reference.md"],
|
|
18
|
+
"cross_check_mismatches": [
|
|
19
|
+
{"export": "functionName", "skill_md_line": 42, "reference_file": "references/api-reference.md", "reference_line": 18, "issue": "SKILL.md shows (date: Date) => string, references/api-reference.md (date: Date, format?: string) => string"},
|
|
20
|
+
{"export": "runTasks", "other_file": "references/api-reference.md", "other_line": 40, "reference_file": "references/pipelines.md", "reference_line": 12, "issue": "references/api-reference.md shows (tasks: Task[], data: unknown) => void, references/pipelines.md (tasks: Task[]) => void"}
|
|
21
|
+
]
|
|
22
|
+
}
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
The contract `validate-inventory.py` checks (coverage-check.md §1a): each `kind` is one of `function`, `class`, `type`, `constant`, `hook`, `interface`, `method`, `struct`, `enum`, `trait`, `macro` or `adapter`; every entry carries the `description` the skill gives it (an empty string when it gives none); a function or method carries `params` (`""` or `[]` when it takes none) and `return_type` as the skill documents them, and leaves a key out when the skill documents nothing for it. A `method`'s `name` is the bare method name and its `owner` the class or object it is called on (`agents` for `cognee.agents.get`, where the skill documents `agents` as a class; the class for one reached through a property, `client.messages`); `owner` is not validated. A function the skill documents in a module is a `function` named bare, never a `method`, even when it is called through the module's name (`register_sql_connection` for `cognee.tools.register_sql_connection`, where the skill documents `tools` as a module). Each `methods` item starts with the method's name, optionally after `async` (`async next_uuid(dep=None)`). A `const` holding a component (a `forwardRef`, `memo` or other wrapper call, or an alias like `const Tabs = TabsPrimitive.Root`) is a `constant` with the declaration or usage form the skill writes as `usage_signature`; a data constant carries no `usage_signature`, `params` or `return_type`. A name documented in several places is listed once per place, each entry with that place's form (`Card` above), and keeps one `kind` in every place. Where the skill documents several names and states members they share once, in the same place (a table row, a paragraph) or over a table for all its rows (a sentence or caption just above it, or its column header), each name's entry carries that shared text in its `params` (in its `usage_signature`, for a usage form), never in a `methods` item, which still starts with the method's name: the shared text first, then `; `, then the name's own members as the skill writes them, without the name, never the name's own members alone (`Note` above, from a row that documents `Note` and `Task`, each with the base fields `id, scope=Scope()`, as `Note(content, title=None)` and `Task(title, done=False)`). An entry whose signature the skill labels partial (simplified, key parameters, a `...` standing in for omitted members, not rest or variadic syntax; a label in a table's column header, such as `Props (simplified)`, labels each of its rows) carries `signature_form: "partial"`, any other `"full"` or no key: copy the label, never judge the form.
|
|
26
|
+
|
|
27
|
+
A `cross_check_mismatches` entry gives its `export`, `reference_file`, `reference_line` and an `issue` that names each form by its file's path, and the pair's other side: the SKILL.md body's line as `skill_md_line`, or, for two references, the one that sorts first in codepoint order, by path and then line, as `other_file` and `other_line`, with the one that sorts last as `reference_file` and `reference_line`. Each file is its path in the package, such as `references/api-reference.md`.
|
|
@@ -15,7 +15,7 @@ The full argument set, gate map, outputs, exit codes and `SKF_TEST_RESULT_JSON`
|
|
|
15
15
|
|
|
16
16
|
## Exit Codes
|
|
17
17
|
|
|
18
|
-
Every terminal state settles one exit code, so a headless automator branches on the verdict, and on any HARD HALT, without grepping message text. A skill run cannot set the process exit status of the agent that runs it (`claude -p` exits 0 whatever the run decided), so automators read `exit_code` from the `SKF_TEST_RESULT_JSON` line or
|
|
18
|
+
Every terminal state settles one exit code, so a headless automator branches on the verdict, and on any HARD HALT, without grepping message text. A skill run cannot set the process exit status of the agent that runs it (`claude -p` exits 0 whatever the run decided), so automators read `exit_code` from the run's own `SKF_TEST_RESULT_JSON` line. A run that printed no envelope line has no result to read, whatever the process exit code: it was cut off, or could not write its result, and `skf-test-skill-result-latest.json` may then describe an earlier run, so it is never read in its place.
|
|
19
19
|
|
|
20
20
|
| Code | Meaning | Raised by |
|
|
21
21
|
| ---- | -------------------- | ------------------------------------------------------------------------------------------ |
|
|
@@ -48,7 +48,7 @@ The emitter prints a finished run's line on stdout and a halt's or a blocked run
|
|
|
48
48
|
| `forge-tier-missing` | init.md §4 |
|
|
49
49
|
| `workspace-drift` | init.md §5b |
|
|
50
50
|
| `another-run-active` | init.md §6b, and the lock renewal in report.md §4c |
|
|
51
|
-
| `inventory-invalid` | coverage-check.md §1a: the documented inventory failed its schema check
|
|
51
|
+
| `inventory-invalid` | coverage-check.md §1a: the documented inventory failed its schema check, spot-check or floor check twice |
|
|
52
52
|
| `indeterminate-surface` | coverage-check.md §2b: zero exports, an empty stack surface or a docs-only skill that documents nothing, so no score can be attached |
|
|
53
53
|
| `step-completeness-violation` | step 4c §2 (a stage recorded nothing in the gap ledger) and report.md §4c |
|
|
54
54
|
| `report-anchor-missing` | report.md §4c |
|
|
@@ -59,7 +59,7 @@ When the emitter itself is not installed (init.md §0), no line can be printed:
|
|
|
59
59
|
|
|
60
60
|
## Result Files
|
|
61
61
|
|
|
62
|
-
report.md §4c runs the emitter with `--result-dir "{forge_version}"`. It writes the per-run record `skf-test-skill-result-{YYYYMMDD-HHmmss}.json` (UTC; it picks the name, `-2`, `-3` appended when a run already took that second's name) and then its copy `skf-test-skill-result-latest.json`, the
|
|
62
|
+
report.md §4c runs the emitter with `--result-dir "{forge_version}"`. It writes the per-run record `skf-test-skill-result-{YYYYMMDD-HHmmss}.json` (UTC; it picks the name, `-2`, `-3` appended when a run already took that second's name) and then its copy `skf-test-skill-result-latest.json`, the record of the last run whose emitter wrote this copy, at a stable path (a copy, not a symlink), each atomically. The record holds the envelope's fields, plus `skill`, `outputs[]` (the published report), `summary` (`score`, `threshold`, `result` (`PASS`, `PASS_WITH_DRIFT`, `FAIL` or `INCONCLUSIVE`), `testMode`, `activeCategories[]`, `inconclusiveReasons[]` when present, `threshold_fallback`, `original_threshold` and `evidence_report_path` when the threshold fallback fired, `gapCounts` with the gap ledger's gaps by severity, and `hardGate`), `runId` and `healthCheckDispatched`, and the emitter stamps `timestamp`, `run_id`, `headless_decisions` and `warnings` into it. A run the hard gate blocked writes it too, as a FAIL record, so its `skf-test-skill-result-latest.json` holds that FAIL, not an earlier run's verdict. A HALT writes no result file, and neither does a run cut off before report.md §4c: the copy then still holds an earlier run's verdict.
|
|
63
63
|
|
|
64
64
|
## Emitting a Halt
|
|
65
65
|
|
|
@@ -16,7 +16,8 @@ Execute this check only when both conditions are met:
|
|
|
16
16
|
the flat copy an older skill may still keep); the detection contract below
|
|
17
17
|
reads it.
|
|
18
18
|
|
|
19
|
-
If either condition fails, skip
|
|
19
|
+
If either condition fails, skip the check with no note and proceed to the next
|
|
20
|
+
section: a missing evidence report is not Case 4.
|
|
20
21
|
|
|
21
22
|
The check runs regardless of naive/contextual mode. T2-future annotations are a
|
|
22
23
|
property of the source code and enrichment data, not the skill type.
|
|
@@ -54,23 +55,26 @@ narrative body.
|
|
|
54
55
|
**Detection contract.** Read the frontmatter deterministically:
|
|
55
56
|
|
|
56
57
|
```bash
|
|
57
|
-
# Extract t2_future_count from frontmatter. Requires a `---` delimiter pair
|
|
58
|
-
|
|
58
|
+
# Extract t2_future_count from frontmatter. Requires a `---` delimiter pair;
|
|
59
|
+
# a trailing CR is stripped first, so a CRLF report reads the same.
|
|
60
|
+
awk '{sub(/\r$/,"")} /^---$/{c++;next} c==1 && /^t2_future_count:/{print $2; exit}' \
|
|
59
61
|
{forge_evidence_report}
|
|
60
62
|
```
|
|
61
63
|
|
|
62
|
-
- **Frontmatter missing OR `t2_future_count` absent
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
invert Case-1 vs Case-2/3
|
|
67
|
-
|
|
68
|
-
|
|
64
|
+
- **Frontmatter missing, OR `t2_future_count` absent or not a non-negative
|
|
65
|
+
integer** (empty, `null`, quoted) → treat as Case 4 (see below). Do not
|
|
66
|
+
fall back to grepping prose (`grep "T2-future"`): prose drift (heading
|
|
67
|
+
renames, alt phrasings like "forward-looking annotations", capitalization
|
|
68
|
+
variance) silently breaks detection and can invert Case-1 vs Case-2/3
|
|
69
|
+
severity.
|
|
70
|
+
- **`t2_future_count` a non-negative integer** (digits only) → use its value
|
|
71
|
+
for the Case Rules below.
|
|
69
72
|
|
|
70
73
|
The pinned field is emitted by `skf-create-skill/references/compile.md`
|
|
71
74
|
§7 (frontmatter-pinned fields), which always writes `t2_future_count: N`
|
|
72
75
|
(including 0). Legacy skills whose `evidence-report.md` predates the pinned
|
|
73
|
-
field land in Case 4
|
|
76
|
+
field land in Case 4, and so does every stack skill: create-stack-skill's
|
|
77
|
+
evidence report has no frontmatter.
|
|
74
78
|
|
|
75
79
|
### Case 1 — T2-future > 0 AND Section 4b absent
|
|
76
80
|
|
|
@@ -106,11 +110,19 @@ Flag as **Medium** severity gap:
|
|
|
106
110
|
Reviewer may downgrade to Low with inline justification on a case-by-case
|
|
107
111
|
basis.
|
|
108
112
|
|
|
109
|
-
### Case 4
|
|
113
|
+
### Case 4: evidence report has no pinned count
|
|
110
114
|
|
|
111
|
-
|
|
115
|
+
The evidence report exists but pins no usable `t2_future_count`: the field is
|
|
116
|
+
absent or not a non-negative integer, or the report has no frontmatter at all
|
|
117
|
+
(a create-skill report older than the field, or any create-stack-skill
|
|
118
|
+
report). Skip the Case Rules and raise no gap. Record this note on the
|
|
119
|
+
Coherence Analysis section's **Migration Section:** line (`{outputFormatsFile}`
|
|
120
|
+
gives it in both formats):
|
|
112
121
|
|
|
113
|
-
> "Section 4b verification skipped
|
|
122
|
+
> "Section 4b verification skipped: evidence-report.md pins no
|
|
123
|
+
> `t2_future_count`. For a skill create-skill made, re-run create-skill to
|
|
124
|
+
> regenerate the report with the field (create-stack-skill does not write
|
|
125
|
+
> it)."
|
|
114
126
|
|
|
115
127
|
## Output
|
|
116
128
|
|