bmad-module-skill-forge 3.0.0 → 5.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +1 -1
- package/README.md +3 -3
- package/docs/_data/pinned.yaml +19 -13
- package/docs/architecture.md +2 -2
- package/docs/concepts.md +4 -4
- package/docs/examples.md +33 -26
- package/docs/getting-started.md +3 -3
- package/docs/how-it-works.md +2 -2
- package/docs/index.md +1 -1
- package/docs/skill-model.md +41 -39
- package/docs/troubleshooting.md +8 -0
- package/docs/verifying-a-skill.md +46 -21
- package/docs/why-skf.md +1 -1
- package/docs/workflows.md +5 -5
- package/package.json +1 -1
- package/src/knowledge/qmd-registry.md +1 -1
- package/src/knowledge/version-paths.md +1 -1
- package/src/shared/health-check.md +2 -50
- package/src/shared/scripts/schemas/skf-audit-result-envelope.v1.json +2 -2
- package/src/shared/scripts/schemas/skf-update-result-envelope.v1.json +12 -4
- package/src/shared/scripts/schemas/skill-brief.v1.json +1 -1
- package/src/shared/scripts/skf-atomic-write.py +212 -59
- package/src/shared/scripts/skf-build-change-manifest.py +1437 -178
- package/src/shared/scripts/skf-classify-changed-files.py +78 -0
- package/src/shared/scripts/skf-compare-file-hashes.py +266 -104
- package/src/shared/scripts/skf-count-tokens.py +54 -10
- package/src/shared/scripts/skf-derive-assembly-shape.py +18 -1
- package/src/shared/scripts/skf-detect-scripts-assets.py +196 -54
- package/src/shared/scripts/skf-detect-workspaces.py +67 -0
- package/src/shared/scripts/skf-emit-result-envelope.py +6 -5
- package/src/shared/scripts/skf-extract-public-api.py +1673 -189
- package/src/shared/scripts/skf-extraction-inventory.py +584 -46
- package/src/shared/scripts/skf-extraction-snapshot.py +158 -33
- package/src/shared/scripts/skf-forge-tier-rw.py +26 -3
- package/src/shared/scripts/skf-github-probe.py +47 -8
- package/src/shared/scripts/skf-hash-content.py +14 -8
- package/src/shared/scripts/skf-load-provenance.py +3 -1
- package/src/shared/scripts/skf-merge-doc-urls.py +142 -19
- package/src/shared/scripts/skf-parse-gaps.py +135 -26
- package/src/shared/scripts/skf-preflight.py +15 -1
- package/src/shared/scripts/skf-provenance-gap-dispatch.py +25 -1
- package/src/shared/scripts/skf-render-metadata-stats.py +1 -1
- package/src/shared/scripts/skf-resolve-authoritative-files.py +29 -3
- package/src/shared/scripts/skf-resolve-customization.py +197 -0
- package/src/shared/scripts/skf-shard-body.py +239 -41
- package/src/shared/scripts/skf-structural-diff.py +133 -18
- package/src/shared/scripts/skf-update-active-symlink.py +115 -62
- package/src/shared/scripts/skf-validate-brief-schema.py +19 -1
- package/src/shared/scripts/skf-validate-output.py +12 -3
- package/src/shared/scripts/skf-verify-provenance-completeness.py +1439 -79
- package/src/shared/scripts/skf-write-skill-brief.py +41 -8
- package/src/shared/tool-requirements.yaml +8 -2
- package/src/skf-analyze-source/SKILL.md +2 -2
- package/src/skf-analyze-source/assets/skill-brief-schema.md +1 -1
- package/src/skf-analyze-source/references/map-unit-exports.md +1 -1
- package/src/skf-audit-skill/SKILL.md +2 -2
- package/src/skf-audit-skill/assets/drift-report-template.md +1 -0
- package/src/skf-audit-skill/references/headless-contract.md +1 -1
- package/src/skf-audit-skill/references/health-check.md +2 -0
- package/src/skf-audit-skill/references/init.md +2 -1
- package/src/skf-audit-skill/references/re-index.md +14 -9
- package/src/skf-audit-skill/references/report.md +2 -2
- package/src/skf-audit-skill/references/semantic-diff.md +15 -15
- package/src/skf-audit-skill/references/structural-diff.md +10 -5
- package/src/skf-audit-skill/scripts/render-drift-tables.py +29 -2
- package/src/skf-brief-skill/SKILL.md +3 -3
- package/src/skf-brief-skill/assets/scope-templates.md +1 -1
- package/src/skf-brief-skill/assets/skill-brief-schema.md +4 -11
- package/src/skf-brief-skill/references/analyze-target.md +6 -6
- package/src/skf-brief-skill/references/confirm-brief.md +1 -1
- package/src/skf-brief-skill/references/scope-definition.md +4 -4
- package/src/skf-brief-skill/references/step-auto-brief.md +11 -29
- package/src/skf-brief-skill/references/version-resolution.md +5 -9
- package/src/skf-campaign/SKILL.md +2 -2
- package/src/skf-campaign/references/step-05-skill-loop.md +6 -1
- package/src/skf-create-skill/SKILL.md +2 -2
- package/src/skf-create-skill/assets/compile-assembly-rules.md +5 -3
- package/src/skf-create-skill/assets/skill-sections.md +7 -7
- package/src/skf-create-skill/references/authoritative-files-protocol.md +2 -23
- package/src/skf-create-skill/references/compile.md +31 -60
- package/src/skf-create-skill/references/component-extraction.md +3 -3
- package/src/skf-create-skill/references/enrich.md +2 -2
- package/src/skf-create-skill/references/entry-points-by-hand.md +24 -7
- package/src/skf-create-skill/references/extract.md +23 -19
- package/src/skf-create-skill/references/extraction-patterns-tracing.md +9 -9
- package/src/skf-create-skill/references/extraction-patterns.md +10 -11
- package/src/skf-create-skill/references/generate-artifacts.md +1 -1
- package/src/skf-create-skill/references/report.md +21 -5
- package/src/skf-create-skill/references/source-resolution-protocols.md +12 -17
- package/src/skf-create-skill/references/sub/fetch-docs.md +1 -1
- package/src/skf-create-stack-skill/SKILL.md +2 -2
- package/src/skf-create-stack-skill/references/validate.md +1 -1
- package/src/skf-drop-skill/SKILL.md +2 -2
- package/src/skf-drop-skill/references/execute.md +5 -1
- package/src/skf-drop-skill/references/health-check.md +2 -0
- package/src/skf-drop-skill/references/invocation-contract.md +1 -1
- package/src/skf-drop-skill/references/report.md +2 -2
- package/src/skf-drop-skill/references/select.md +7 -3
- package/src/skf-export-skill/SKILL.md +2 -2
- package/src/skf-export-skill/assets/snippet-format.md +5 -5
- package/src/skf-export-skill/references/generate-snippet.md +22 -15
- package/src/skf-export-skill/references/load-skill.md +1 -1
- package/src/skf-export-skill/references/token-report.md +29 -9
- package/src/skf-forger/SKILL.md +2 -2
- package/src/skf-quick-skill/SKILL.md +2 -2
- package/src/skf-refine-architecture/SKILL.md +6 -8
- package/src/skf-rename-skill/SKILL.md +2 -2
- package/src/skf-setup/SKILL.md +2 -2
- package/src/skf-test-skill/SKILL.md +2 -2
- package/src/skf-test-skill/assets/output-section-formats.md +6 -4
- package/src/skf-test-skill/references/coherence-check.md +5 -4
- package/src/skf-test-skill/references/coverage-check-branches.md +61 -0
- package/src/skf-test-skill/references/coverage-check-tiers.md +5 -5
- package/src/skf-test-skill/references/coverage-check.md +69 -111
- package/src/skf-test-skill/references/init.md +5 -6
- package/src/skf-test-skill/references/migration-section-rules.md +26 -14
- package/src/skf-test-skill/references/report.md +11 -9
- package/src/skf-test-skill/references/scoring-rules.md +8 -6
- package/src/skf-test-skill/references/source-access-protocol.md +15 -18
- package/src/skf-test-skill/scripts/build-result-context.py +114 -21
- package/src/skf-test-skill/scripts/gap-ledger.py +580 -80
- package/src/skf-test-skill/scripts/load-coverage-inputs.py +646 -107
- package/src/skf-test-skill/scripts/locate-export-segments.py +101 -20
- package/src/skf-test-skill/scripts/reconcile-coverage.py +450 -23
- package/src/skf-test-skill/scripts/score-signatures.py +356 -61
- package/src/skf-test-skill/scripts/stage-helper-payload.py +64 -13
- package/src/skf-test-skill/scripts/validate-inventory.py +108 -9
- package/src/skf-test-skill/scripts/verify-declared-numerator.py +7 -5
- package/src/skf-test-skill/templates/test-report-template.md +1 -0
- package/src/skf-update-skill/SKILL.md +2 -2
- package/src/skf-update-skill/references/detect-changes.md +32 -13
- package/src/skf-update-skill/references/gap-driven.md +37 -22
- package/src/skf-update-skill/references/init.md +6 -2
- package/src/skf-update-skill/references/invocation-contract.md +1 -1
- package/src/skf-update-skill/references/merge.md +17 -12
- package/src/skf-update-skill/references/re-extract.md +4 -2
- package/src/skf-update-skill/references/report.md +7 -4
- package/src/skf-update-skill/references/write.md +77 -36
- package/src/skf-update-skill/scripts/skf-new-file-diff.py +157 -42
- package/src/skf-verify-stack/SKILL.md +2 -2
- package/tools/cli/lib/tool-check.js +158 -2
|
@@ -4,6 +4,7 @@ outputFile: '{report_file}'
|
|
|
4
4
|
scoringRulesFile: 'references/scoring-rules.md'
|
|
5
5
|
sourceAccessProtocol: 'references/source-access-protocol.md'
|
|
6
6
|
coverageTiersFile: 'references/coverage-check-tiers.md'
|
|
7
|
+
coverageBranchesFile: 'references/coverage-check-branches.md'
|
|
7
8
|
validateInventoryScript: 'scripts/validate-inventory.py'
|
|
8
9
|
coverageInputsScript: 'scripts/load-coverage-inputs.py'
|
|
9
10
|
scoreSignaturesScript: 'scripts/score-signatures.py'
|
|
@@ -33,7 +34,7 @@ detectWorkspacesProbeOrder:
|
|
|
33
34
|
|
|
34
35
|
## STEP GOAL:
|
|
35
36
|
|
|
36
|
-
Compare
|
|
37
|
+
Compare what SKILL.md documents (exports, functions, classes, types, interfaces) with the source API surface: missing documentation, undocumented exports, signature mismatches, at the forge tier's depth.
|
|
37
38
|
|
|
38
39
|
**Halt envelope.** Every HALT in this step names its `halt_reason` and phase and carries exit code 1. It releases the run lock first, whatever the release prints: from `{project-root}`, run `uv run {runLockHelper} release --lock "{forge_version}/.test-skill.lock" --owner "{run_owner}"` (SKILL.md Workflow Rules). In headless mode it then writes `{"phase": "<phase>", "reason": "<the halt message>", "halt_reason": "<halt_reason>", "skill_name": "{skill_name}", "report_path": "{report_file}"}`, adding `"path"` when the halt names one, to `{run_dir}/halt.json` and runs:
|
|
39
40
|
|
|
@@ -43,11 +44,11 @@ uv run {emitEnvelopeHelper} emit-halt --workflow skf-test-skill --run-dir "{run_
|
|
|
43
44
|
|
|
44
45
|
Display the line it prints, as the run's last line, then stop. If the emitter exits non-zero or prints no line, display the halt message alone. An interactive HALT displays its message and emits nothing.
|
|
45
46
|
|
|
46
|
-
**Run files.** Every script below writes its result into the run folder `{run_dir}` that init.md §6b created,
|
|
47
|
+
**Run files.** Every script below writes its result into the run folder `{run_dir}` that init.md §6b created, where the next one reads it by path. A subagent's response is saved there exactly as it came back, fence and all (the subagent writes it to the path it is given, or the parent saves it, with the Write tool), and passed by path, so a quote or an apostrophe in it never meets a shell. Every count, name list and score a script computes is read from the file it wrote, never re-derived by hand or copied between commands. Each command runs from `{project-root}`; a `{...Script}` path resolves relative to the skill root.
|
|
47
48
|
|
|
48
49
|
### 0. Check for Docs-Only Mode
|
|
49
50
|
|
|
50
|
-
A skill is docs-only when every citation it writes is `[EXT:...]`,
|
|
51
|
+
A skill is docs-only when every citation it writes is `[EXT:...]`, none local (`[AST:...]`, `[SRC:...]`, or a stack's `[from skill: ...]`). The citation census decides it:
|
|
51
52
|
|
|
52
53
|
```bash
|
|
53
54
|
uv run {coverageInputsScript} census --skill-dir "{resolved_skill_package}" --output "{run_dir}/census.json"
|
|
@@ -55,7 +56,7 @@ uv run {coverageInputsScript} census --skill-dir "{resolved_skill_package}" --ou
|
|
|
55
56
|
|
|
56
57
|
Bind `{docs_only_mode}` ← `docsOnly`. Exit 1 printed its error on stderr: HALT with it (`halt_reason: "helper-failed"`, phase `coverage-check:census`). §0b sets `{docs_only_mode}` too when source access reaches State 5.
|
|
57
58
|
|
|
58
|
-
**A docs-only run** measures documentation completeness as its Export Coverage, with `analysis_confidence` `docs-only`. It runs only §1, §1a and §1b, the docs-only guard of §2b, §2c's `docsOnly` branch, §4's
|
|
59
|
+
**A docs-only run** measures documentation completeness as its Export Coverage, with `analysis_confidence` `docs-only`. It runs only §1, §1a and §1b, the docs-only guard of §2b, §2c's `docsOnly` branch, §4's `Denominator: docs-only completeness` annotation, §5, §5b and §6. Step 5 passes `docsOnly: true` to the scoring script, which skips Signature Accuracy and Type Coverage at any tier.
|
|
59
60
|
|
|
60
61
|
"**Docs-only skill detected.** Coverage check evaluates documentation completeness rather than source code coverage."
|
|
61
62
|
|
|
@@ -63,19 +64,21 @@ A source-based skill continues with §0b.
|
|
|
63
64
|
|
|
64
65
|
### 0b. Load Source Access Protocol
|
|
65
66
|
|
|
66
|
-
Load `{sourceAccessProtocol}` and follow both sections: **Source API Surface Definition** (the
|
|
67
|
+
Load `{sourceAccessProtocol}` and follow both sections: **Source API Surface Definition** (the denominator clause §2b picks) and **Source Access Resolution** (the 5-state waterfall that decides how the source is read and sets `analysis_confidence`; State 5 makes the run docs-only).
|
|
67
68
|
|
|
68
|
-
**Workspace layout.** The Source API Surface Definition's monorepo tests read `{source_is_monorepo}
|
|
69
|
+
**Workspace layout.** The Source API Surface Definition's monorepo tests read `{source_is_monorepo}`. At State 1 (local source), resolve `{detectWorkspacesHelper}` ← first existing path in `{detectWorkspacesProbeOrder}` and run (`{stageHelperPayloadScript}` pipes it the tree and root manifests from disk; leave `--brief` out with no brief):
|
|
69
70
|
|
|
70
71
|
```bash
|
|
71
|
-
uv run {stageHelperPayloadScript} detect-workspaces --source-root "{source_path}" | uv run {detectWorkspacesHelper}
|
|
72
|
+
uv run {stageHelperPayloadScript} detect-workspaces --source-root "{source_path}" --brief "{forge_data_folder}/{skill_name}/skill-brief.yaml" | uv run {detectWorkspacesHelper}
|
|
72
73
|
```
|
|
73
74
|
|
|
74
|
-
Bind `{source_is_monorepo}` ← `is_monorepo` and `{source_workspace_kind}` ← `manifest_kind`. At States 2 to 5, or when the helper does not resolve or the command exits non-zero, the layout is unknown: bind `{source_is_monorepo}` ← false. §5 records the layout.
|
|
75
|
+
A workspace whose members `scope.exclude` drops is no workspace. Bind `{source_is_monorepo}` ← `is_monorepo` and `{source_workspace_kind}` ← `manifest_kind`. At States 2 to 5, or when the helper does not resolve or the command exits non-zero, the layout is unknown: bind `{source_is_monorepo}` ← false. §5 records the layout.
|
|
76
|
+
|
|
77
|
+
**Clause flags.** Every `surface` command, in §2 or in a branch file it loads, takes `--monorepo` when `{source_is_monorepo}` is true, and the flags the Source API Surface Definition leaves to you.
|
|
75
78
|
|
|
76
79
|
### 1. Extract Documented Exports from SKILL.md
|
|
77
80
|
|
|
78
|
-
Delegate the reading of the skill under test to a subagent, given the path to SKILL.md and to its `references/` directory when one exists
|
|
81
|
+
Delegate the reading of the skill under test to a subagent, given `{run_dir}` and the path to SKILL.md and to its `references/` directory when one exists: it reads SKILL.md, and every `references/*.md` file too when SKILL.md's `## Full` headings are absent or stubs. For each export both the SKILL.md body and a reference file document, it compares the two: parameters (name, type, order, optionality), the return type, and descriptions that contradict (brief against detailed is fine, as is a form the skill labels partial that leaves out only optional members). It writes only this JSON, with no prose and no commentary, to `{run_dir}/inventory-response.txt` with the Write tool, never a shell, and returns that path, or returns the JSON itself when it cannot write files:
|
|
79
82
|
|
|
80
83
|
```json
|
|
81
84
|
{
|
|
@@ -83,7 +86,7 @@ Delegate the reading of the skill under test to a subagent, given the path to SK
|
|
|
83
86
|
{"name": "functionName", "kind": "function", "params": "...", "return_type": "...", "description": "..."},
|
|
84
87
|
{"name": "ClassName", "kind": "class", "methods": ["..."], "properties": ["..."], "description": "..."},
|
|
85
88
|
{"name": "TypeName", "kind": "type", "fields": ["..."], "description": "..."},
|
|
86
|
-
{"name": "useHook", "kind": "hook", "usage_signature": "...", "description": "..."}
|
|
89
|
+
{"name": "useHook", "kind": "hook", "usage_signature": "...", "description": "...", "signature_form": "partial"}
|
|
87
90
|
],
|
|
88
91
|
"references": ["references/api-reference.md"],
|
|
89
92
|
"cross_check_mismatches": [
|
|
@@ -92,33 +95,32 @@ Delegate the reading of the skill under test to a subagent, given the path to SK
|
|
|
92
95
|
}
|
|
93
96
|
```
|
|
94
97
|
|
|
95
|
-
Tell the subagent the contract the validator checks: each `kind` is one of `function`, `class`, `type`, `constant`, `hook`, `interface`, `method`, `struct`, `enum`, `trait`, `macro` or `adapter`; every entry carries the `description` the skill gives it (an empty string when it gives none); a function or method carries `params` (`""` or `[]` when it takes none) and `return_type` as the skill documents them, and leaves a key out when the skill documents nothing for it.
|
|
98
|
+
Tell the subagent the contract the validator checks: each `kind` is one of `function`, `class`, `type`, `constant`, `hook`, `interface`, `method`, `struct`, `enum`, `trait`, `macro` or `adapter`; every entry carries the `description` the skill gives it (an empty string when it gives none); a function or method carries `params` (`""` or `[]` when it takes none) and `return_type` as the skill documents them, and leaves a key out when the skill documents nothing for it. An entry whose signature the skill labels partial (simplified, key parameters, a `...` standing in for omitted members, not rest or variadic syntax) carries `signature_form: "partial"`, any other `"full"` or no key: copy the label, never judge the form.
|
|
96
99
|
|
|
97
|
-
|
|
100
|
+
When the subagent returned the JSON rather than the path, or the file is absent or empty afterwards, save its response, as it came back, to `{run_dir}/inventory-response.txt` with the Write tool. The documented inventory is the one §1a validates: do not load SKILL.md or the reference files into the parent context. Without subagents, the parent builds the same JSON from the same files and saves it to that file; §1a's checks still run on it.
|
|
98
101
|
|
|
99
102
|
#### 1a. Parent-Side Schema Validation + Spot-Check
|
|
100
103
|
|
|
101
|
-
Before anything consumes the inventory, `{validateInventoryScript}` checks it against the §1 contract (a wrapping fence stripped) and spot-checks sampled names against SKILL.md and the reference files
|
|
104
|
+
Before anything consumes the inventory, `{validateInventoryScript}` checks it against the §1 contract (a wrapping fence stripped) and spot-checks sampled names against SKILL.md and the subagent's listed reference files. The look-up is the script's, never a grep: an export named `$state` or `a.b` is neither a shell variable nor a regex.
|
|
102
105
|
|
|
103
106
|
```bash
|
|
104
107
|
uv run {validateInventoryScript} --input "{run_dir}/inventory-response.txt" --skill-package "{resolved_skill_package}" --output "{run_dir}/inventory.json"
|
|
105
108
|
```
|
|
106
109
|
|
|
107
110
|
- `valid` true (exit 0): every later command reads `{run_dir}/inventory.json`.
|
|
108
|
-
- `valid` false (exit 2)
|
|
109
|
-
- Exit 1 (no input, a file that cannot be read, or a package with no readable SKILL.md): correct the file and run it again.
|
|
111
|
+
- `valid` false (exit 2), or exit 1: load `{coverageBranchesFile}` and follow its **Inventory Failures** section. Invalid a second time: HALT (`halt_reason: "inventory-invalid"`, phase `coverage-check:inventory`): "coverage-check: the subagent inventory failed {schema validation | the ground-truth spot-check} twice: {the violations joined}."
|
|
110
112
|
|
|
111
113
|
### 1b. Cross-Check Split-Body Consistency
|
|
112
114
|
|
|
113
|
-
Only when the validated inventory's `references` is non-empty.
|
|
115
|
+
Only when the validated inventory's `references` is non-empty. Each `cross_check_mismatches` entry is a High `split-body-mismatch` gap. §5b records it titled `Split-body mismatch: {export} ({reference_file})`, with its `export` and its Source at the side the source contradicts (the reference file when no source settles it): update-skill edits that file to match the other without reading the source.
|
|
114
116
|
|
|
115
117
|
### 2. Analyze Source Code (Tier-Dependent)
|
|
116
118
|
|
|
117
|
-
Identify the public API surface from the package entry points (
|
|
119
|
+
Identify the public API surface from the package entry points (0b), as deep as the tier allows. Whichever branch runs, `{coverageInputsScript} surface` writes it to `{run_dir}/surface.json`, which §2b, §2c, §4 and step 5 read. Names read by eye reach it as per-file results, `{"file": "<path>", "exports_found": [<each name>], "signature_mismatches": []}` for each file, saved to `{run_dir}/per-file-<n>.json` with the Write tool and given to `surface` with one `--per-file` each.
|
|
118
120
|
|
|
119
121
|
**Exits.** Every `{coverageInputsScript}` and `{scoreSignaturesScript}` command in §2 and §2b exits 0, or exits 1 with its error on stderr: HALT with it (`halt_reason: "helper-failed"`, phase `coverage-check:surface`). `surface` with a `--per-file`, and `score`, also exit 2 when a subagent response breaks the schema: re-dispatch that subagent once with its `violations[]` appended (or correct a result the main thread wrote), save the new response over the old one and run the command again; a second failure HALTs the same way.
|
|
120
122
|
|
|
121
|
-
**Quick Tier (no AST tools):** load `{coverageTiersFile}` and follow its **Quick Tier** section. Signatures cannot be verified at this tier
|
|
123
|
+
**Quick Tier (no AST tools):** load `{coverageTiersFile}` and follow its **Quick Tier** section. Signatures cannot be verified at this tier (§5's render marks them unverified).
|
|
122
124
|
|
|
123
125
|
**Forge Tier (ast-grep available):**
|
|
124
126
|
|
|
@@ -128,44 +130,42 @@ At State 1, run the recipe runner once over the source. Resolve `{extractPublicA
|
|
|
128
130
|
uv run {extractPublicApiHelper} --mode full --source-root "{source_path}" --brief "{forge_data_folder}/{skill_name}/skill-brief.yaml" --tier {detected_tier} --head-cap 0 --output "{run_dir}/extract-full.json"
|
|
129
131
|
```
|
|
130
132
|
|
|
131
|
-
|
|
133
|
+
Exit 2 is an input error (a brief the runner cannot read, say) or a failure the runner did not foresee, named in one line on stderr with no JSON: HALT with that line (`halt_reason: "helper-failed"`, phase `coverage-check:surface`). No Fallback Per-File Scan stands in for it: every flag here is fixed, so the error is a defect to surface.
|
|
134
|
+
|
|
135
|
+
Build the surface from it, with the metadata, the brief when the skill has one and, when init.md §2 bound one, the provenance map (the baselines of the §2b candidates and guards; they add no name):
|
|
132
136
|
|
|
133
137
|
```bash
|
|
134
|
-
uv run {coverageInputsScript} surface --extraction "{run_dir}/extract-full.json" --metadata "{resolved_skill_package}/metadata.json" [--provenance "{forge_provenance_map}"] --output "{run_dir}/surface.json"
|
|
138
|
+
uv run {coverageInputsScript} surface --extraction "{run_dir}/extract-full.json" --metadata "{resolved_skill_package}/metadata.json" [--provenance "{forge_provenance_map}"] [--brief "{forge_data_folder}/{skill_name}/skill-brief.yaml"] [--monorepo] [--pattern-reference-intent] [--curated-subset] --output "{run_dir}/surface.json"
|
|
135
139
|
```
|
|
136
140
|
|
|
137
|
-
The surface holds the names the entry points export, less those defined in a file the brief scopes out (`excluded.outsideScope
|
|
141
|
+
The surface holds the names the entry points export, less those defined in a file the brief scopes out (`excluded.outsideScope`); the recipes' internal matches are never part of it. A Python `__init__.py` nested in the folder of another package that exports a name is no barrel: its names (`excluded.nestedEntries`) leave only the `all` and `root` sets and the umbrella ratio, and `plan` still compares their signatures. When a `scope.include` glob matches no file (`guards.staleScope.fires`), the root exports defined in a file no glob covers and the brief does not exclude are back in it (`guards.staleScope.restored`). A stale `scope.tier_a_include` glob fires the guard too (`guards.staleScope.unmatchedTierAInclude`) and counts no name. §5's render lists `excluded.outsideScope`, `excluded.nestedEntries` and the restored names. Decide what to do from the JSON, not from the exit code:
|
|
138
142
|
|
|
139
|
-
- `extraction.fallback.needed` is true, the runner
|
|
140
|
-
- `extraction.
|
|
141
|
-
-
|
|
143
|
+
- `extraction.fallback.needed` is true, or the runner printed no JSON on an exit other than 2: load `{coverageTiersFile}` and run its **Fallback Per-File Scan** in place of the rest of this tier.
|
|
144
|
+
- `extraction.readByEye` is non-empty (a file the run could not parse or decode, or a failed call left unread): keep the surface, read by eye only the files it names, save a per-file result for each, and run `surface` again with them, once (the second call starts no new pass).
|
|
145
|
+
- In the Coverage Analysis section, list each `readByEye` file as read by eye, each `extraction.unread` file as unread.
|
|
146
|
+
- `extractionGaps[]` names exports no recipe found: they are on the surface; read their signature by eye at their `file` and `line`.
|
|
142
147
|
|
|
143
148
|
**Compare the signatures.** A script lists the comparisons; subagents judge them:
|
|
144
149
|
|
|
145
150
|
```bash
|
|
146
|
-
uv run {scoreSignaturesScript} plan --inventory "{run_dir}/inventory.json" --surface "{run_dir}/surface.json" --output "{run_dir}/signature-plan.json"
|
|
151
|
+
uv run {scoreSignaturesScript} plan --inventory "{run_dir}/inventory.json" --surface "{run_dir}/surface.json" --skill-dir "{resolved_skill_package}" --output "{run_dir}/signature-plan.json"
|
|
147
152
|
```
|
|
148
153
|
|
|
149
|
-
For each `files[]` entry, delegate one subagent with its checks. For each check it reads the full declaration at `{file}:{line}` (all of it: `signatureLine` is one physical line, and a multi-line signature cut there
|
|
154
|
+
For each `files[]` entry, delegate one subagent with its checks. For each check it reads the full declaration at `{file}:{line}` (all of it: `signatureLine` is one physical line, and a multi-line signature cut there reads as a mismatch), compares it with the check's `documented` signature (params: name, type, order, optionality; and the return type), and returns only this JSON, with no prose and no markdown fences:
|
|
150
155
|
|
|
151
156
|
```json
|
|
152
|
-
{"file": "src/utils.ts", "signature_mismatches": [{"name": "formatDate", "line": 42, "source_sig": "(date: Date, format?: string) => string", "documented_sig": "(date: Date) => string", "issue": "missing optional parameter 'format'"}]}
|
|
157
|
+
{"file": "src/utils.ts", "signature_mismatches": [{"name": "formatDate", "line": 42, "source_sig": "(date: Date, format?: string) => string", "documented_sig": "(date: Date) => string", "issue": "missing optional parameter 'format'", "difference": "omitted-optional"}]}
|
|
153
158
|
```
|
|
154
159
|
|
|
155
|
-
|
|
160
|
+
A mismatch carries `difference` when the two differ only in one of these ways, else no key: `parameter-name` (only parameter names differ: types, order, optionality, defaults and return type equal; a destructured property is part of a type) or `omitted-optional` (the documented form leaves out only members the source marks optional). A parameter or field with a default or a `?`, or a TS/JS rest parameter, is optional; a Rust `Option<T>` field a struct literal still requires is not. Tag it in any language and form: `score` (§2b) checks the facts. The example above, tagged `omitted-optional`, stays Critical against a full form.
|
|
156
161
|
|
|
157
|
-
|
|
162
|
+
A check with `candidates` reads each and reports a mismatch only when the documented signature matches none: one, at the closest candidate, its `file` and `line` on the mismatch object (the result's `file` stays its `files[]` entry) and its `difference` tagged against it. A declaration opening an overload set (Python `@overload` stubs, TS bodiless signatures) is read through its implementation: a documented signature matching the implementation or any overload is no mismatch.
|
|
158
163
|
|
|
159
|
-
|
|
160
|
-
- **State 2** (`{forge_provenance_map}`, the provenance map init.md §2 bound): the union of its named exports and the `metadata.json` `exports[]` names, with `--fold`, `--keep` and `--fold-prefix` as the Source API Surface Definition's canonicalization says:
|
|
164
|
+
Save each response to `{run_dir}/signatures-<n>.txt` with the Write tool. Without subagents, compare in the main thread and save the same JSON.
|
|
161
165
|
|
|
162
|
-
|
|
163
|
-
uv run {coverageInputsScript} surface --provenance "{forge_provenance_map}" --metadata "{resolved_skill_package}/metadata.json" [--brief "{forge_data_folder}/{skill_name}/skill-brief.yaml"] [--fold] [--keep <variant>] [--fold-prefix <prefix>] --output "{run_dir}/surface.json"
|
|
164
|
-
```
|
|
166
|
+
**Deep Tier (ast-grep + gh + QMD):** the Forge tier checks, plus: check with the gh CLI that the source repository matches the documented version, check type definitions against their source declarations, and trace re-exports to their origin.
|
|
165
167
|
|
|
166
|
-
|
|
167
|
-
- **State 3**: the `metadata.json` `exports[]` names: `surface --metadata "{resolved_skill_package}/metadata.json" --output "{run_dir}/surface.json"`.
|
|
168
|
-
- **State 4**: the names read remotely by eye, as per-file results: `surface --per-file "{run_dir}/per-file-<n>.json" [--brief "{forge_data_folder}/{skill_name}/skill-brief.yaml"] --metadata "{resolved_skill_package}/metadata.json" --output "{run_dir}/surface.json"`.
|
|
168
|
+
**States 2 to 4 (no local source):** the surface is what the skill recorded, so there is no signature to score: load `{coverageBranchesFile}` and follow its **States 2 to 4** section.
|
|
169
169
|
|
|
170
170
|
### 2b. Resolve the Denominator and Guard Zero Exports
|
|
171
171
|
|
|
@@ -175,11 +175,9 @@ Load the metadata and provenance-map counts once, for §2c, §4b and the numerat
|
|
|
175
175
|
uv run {coverageInputsScript} metadata --metadata "{resolved_skill_package}/metadata.json" [--provenance "{forge_provenance_map}"] --output "{run_dir}/coverage-inputs.json"
|
|
176
176
|
```
|
|
177
177
|
|
|
178
|
-
**Pick the denominator.**
|
|
178
|
+
**Pick the denominator.** `surface.json` `denominator` holds the Source API Surface Definition clause (§0b) `surface` picked: its §2c `branch` (`barrel`, `scalar` or `stack`), `surfaceSet` (the set the barrel branch counts, as the scalar one does when `setCounted` is true, and the signature score reads) and `ladderStep`. When `absentSet` names a set the surface lacks (the scope sets need the brief's globs and names that carry their files, `subpaths` and `root` an extraction), `surfaceSet` is `all` and the `Denominator:` line (§4) adds `({absentSet} set absent: all set used)`. When `rebuild` is `provenance-fold`, load `{coverageBranchesFile}` and rerun State 2's command from its **States 2 to 4** section without `--metadata` and with `--fold` and `--pattern-reference-intent`, running nothing else from that section, and read the `denominator` it writes.
|
|
179
179
|
|
|
180
|
-
- **
|
|
181
|
-
- **Stratified-scope priority 1**: the scalar branch over `stats.effective_denominator` (`candidates.statsEffectiveDenominator`), unless `guards.deflation.fires`: then the barrel branch over the `scope.include` set (`all` when the brief has no `scope.include`).
|
|
182
|
-
- **A stack skill** (`metadata.json.skill_type == "stack"`): the stack branch, over `coverage-inputs.json` `stack.denominator`. Its own barrel is empty by design, and its `[from skill: …]` citations never make it docs-only. **If `stack.denominator` is 0**: HALT (`halt_reason: "indeterminate-surface"`, phase `coverage-check:denominator`) with `Error: stack composition surface empty: {skill_name} cites no contracts, libraries, or integration pairs, so Export Coverage is undefined. Verify the stack was compiled from at least one constituent skill.` Do not write the Coverage Analysis section; this is an indeterminate state, not a FAIL.
|
|
180
|
+
- **A stack skill** (`branch` `stack`): `coverage-inputs.json` `stack.denominator`. Its own barrel is empty by design, and its `[from skill: …]` citations never make it docs-only. **If `stack.denominator` is 0**: HALT (`halt_reason: "indeterminate-surface"`, phase `coverage-check:denominator`) with `Error: stack composition surface empty: {skill_name} cites no contracts, libraries, or integration pairs, so Export Coverage is undefined. Verify the stack was compiled from at least one constituent skill.` Do not write the Coverage Analysis section; this is an indeterminate state, not a FAIL.
|
|
183
181
|
- **A docs-only skill**: the `docsOnly` branch, over the inventory.
|
|
184
182
|
|
|
185
183
|
**Numerator ground truth.** A declared count equal to the denominator (`stats.exports_documented == stats.effective_denominator`, the inflation signature) may be padded to a tautological 100%. Run the check on every source-based skill; without the signature (a stack and a reference app carry none) it reports `skipped`:
|
|
@@ -188,7 +186,7 @@ uv run {coverageInputsScript} metadata --metadata "{resolved_skill_package}/meta
|
|
|
188
186
|
uv run {numeratorVerifyScript} --inputs "{run_dir}/coverage-inputs.json" --skill-dir "{resolved_skill_package}" --output "{run_dir}/numerator.json"
|
|
189
187
|
```
|
|
190
188
|
|
|
191
|
-
§2c's scalar branch
|
|
189
|
+
Unless §2c's scalar branch counts its set, it takes the `verified` count as the numerator when `inflated` is true; §4b records the gap. Exit 1 or 2: HALT with its message (`halt_reason: "helper-failed"`, phase `coverage-check:numerator`).
|
|
192
190
|
|
|
193
191
|
**If the chosen denominator is 0 AND `docs_only_mode == false` AND `metadata.json.skill_type != "stack"`:** HALT (`halt_reason: "indeterminate-surface"`, phase `coverage-check:denominator`) with:
|
|
194
192
|
|
|
@@ -198,22 +196,22 @@ Error: indeterminate API surface: 0 exports discovered in source for {skill_name
|
|
|
198
196
|
Export Coverage is undefined, and scoring would yield a vacuous PASS. Fix one of:
|
|
199
197
|
- Set `scope.include` in the brief to point at the package's entry point(s)
|
|
200
198
|
- Add `[EXT:]` citations if this is actually a docs-only skill
|
|
201
|
-
- Verify the skill's
|
|
199
|
+
- Verify the skill's source_root / source_ref resolve to the intended tree
|
|
202
200
|
```
|
|
203
201
|
|
|
204
202
|
Do not write the Coverage Analysis section or attach a score: this is an indeterminate state, not a FAIL. **If `docs_only_mode == true` and the validated inventory documents no item:** HALT the same way with "docs-only skill declares zero items: no API surface to test".
|
|
205
203
|
|
|
206
|
-
**Score the signatures** (Forge, Forge+ and Deep tier at State 1, once §2 compared them). Give `{scoreSignaturesScript}`
|
|
204
|
+
**Score the signatures** (Forge, Forge+ and Deep tier at State 1, once §2 compared them). Give `{scoreSignaturesScript}` `denominator.surfaceSet` as `--surface-set`. Type Coverage then counts that set's types alone, so a type the clause leaves out (a stratified skill's deferred Tier B) lowers neither Export Coverage nor Type Coverage. Give it one `--results` per saved response (the fallback scan's `per-file-<n>.txt`, else the `signatures-<n>.txt`):
|
|
207
205
|
|
|
208
206
|
```bash
|
|
209
|
-
uv run {scoreSignaturesScript} score --inventory "{run_dir}/inventory.json" --surface "{run_dir}/surface.json" --surface-set <set> --results "{run_dir}/signatures-<n>.txt" --output "{run_dir}/signatures.json" --gaps-output "{run_dir}/signature-gaps.json"
|
|
207
|
+
uv run {scoreSignaturesScript} score --inventory "{run_dir}/inventory.json" --surface "{run_dir}/surface.json" --surface-set <set> --plan "{run_dir}/signature-plan.json" --results "{run_dir}/signatures-<n>.txt" --output "{run_dir}/signatures.json" --gaps-output "{run_dir}/signature-gaps.json"
|
|
210
208
|
```
|
|
211
209
|
|
|
212
|
-
It
|
|
210
|
+
It grades each mismatch of a compared name from facts it checks, never a `difference` alone, into `signature-gaps.json`: a Low `signature-mismatch` gap for a `parameter-name` difference in a file the plan offered when it is TS/JS, Go, Rust, Java, C or C++, an Info `observation` titled `Partial signature: {name}` for an `omitted-optional` one on a `partial` form, a Critical `signature-mismatch` gap otherwise (Python included). A signature gap is titled `Signature mismatch: {name}`, with `export` `{name}`, its Source at `{file}:{line}` and both signatures in its Issue; §5b appends that file to the ledger. A Low or Info one counts as matching, and §5 renders it so. §5's render lists its `warnings` (a mismatch for a name never compared, a category with nothing to compare).
|
|
213
211
|
|
|
214
212
|
### 2c. Reconcile Documented vs Source Surface (Deterministic Intersection)
|
|
215
213
|
|
|
216
|
-
`{reconcileScript}` reconciles the §1 inventory with the source surface and writes `{run_dir}/coverage.json`, whose `exportCoverage` is the Export Coverage
|
|
214
|
+
`{reconcileScript}` reconciles the §1 inventory with the source surface and writes `{run_dir}/coverage.json`, whose `exportCoverage` is the Export Coverage step 5 hands to the scoring script. Run the branch §2b picked:
|
|
217
215
|
|
|
218
216
|
1. **Barrel**, with the name set §2b picked as `--surface-set`:
|
|
219
217
|
|
|
@@ -221,19 +219,9 @@ It writes each mismatch of a compared name to `signature-gaps.json` as a Critica
|
|
|
221
219
|
uv run {reconcileScript} --denominator-source barrel --inventory "{run_dir}/inventory.json" --surface "{run_dir}/surface.json" --surface-set <set> --output "{run_dir}/coverage.json"
|
|
222
220
|
```
|
|
223
221
|
|
|
224
|
-
2. **Scalar
|
|
225
|
-
|
|
226
|
-
```bash
|
|
227
|
-
uv run {reconcileScript} --denominator-source scalar --coverage-inputs "{run_dir}/coverage-inputs.json" --inventory "{run_dir}/inventory.json" --skill-dir "{resolved_skill_package}" --verified "{run_dir}/numerator.json" --output "{run_dir}/coverage.json"
|
|
228
|
-
```
|
|
229
|
-
|
|
230
|
-
3. **Stack** (`metadata.json.skill_type == "stack"`):
|
|
222
|
+
2. **Scalar or stack**: load `{coverageBranchesFile}` and follow its **Scalar and Stack Branches** section, which runs the branch's command.
|
|
231
223
|
|
|
232
|
-
|
|
233
|
-
uv run {reconcileScript} --denominator-source stack --coverage-inputs "{run_dir}/coverage-inputs.json" --skill-dir "{resolved_skill_package}" --output "{run_dir}/coverage.json"
|
|
234
|
-
```
|
|
235
|
-
|
|
236
|
-
4. **Docs-only**: Export Coverage is the documentation completeness ratio, complete items over documented items, and `incomplete[]` lists each incomplete item with the fields it lacks:
|
|
224
|
+
3. **Docs-only**: Export Coverage is the documentation completeness ratio, complete over documented items, and `incomplete[]` lists each incomplete item with the fields it lacks:
|
|
237
225
|
|
|
238
226
|
```bash
|
|
239
227
|
uv run {reconcileScript} --denominator-source docsOnly --inventory "{run_dir}/inventory.json" --output "{run_dir}/coverage.json"
|
|
@@ -241,40 +229,22 @@ It writes each mismatch of a compared name to `signature-gaps.json` as a Critica
|
|
|
241
229
|
|
|
242
230
|
Exit 2 printed an `INVALID_INPUT` envelope: correct the input it names and run it again. Exit 1: HALT with its stderr (`halt_reason: "helper-failed"`, phase `coverage-check:reconcile`).
|
|
243
231
|
|
|
244
|
-
**A surplus is a signal, not a pass.** The scalar and stack branches count the numerator and the denominator from independent sets, so `documented` can exceed `denominator`: the script then reports `numeratorSurplus` above 0 and `coverageCapped: true`, holds `exportCoverage` at 100 and keeps the raw ratio in `coverageUncapped`. State the surplus beside both counts in the Coverage Analysis section. On a skill whose brief carries no `scope.tier_a_include`, a large surplus is the deflated-denominator signature: check `guards.deflation` before accepting the 100.
|
|
245
|
-
|
|
246
232
|
**Classify what the script found.** §5b records each of these gaps from the run files:
|
|
247
233
|
|
|
248
|
-
- **Missing names** (barrel
|
|
249
|
-
- **
|
|
250
|
-
- **Stale names** (barrel branch: `stale`): each is a documented name the enumerated source surface lacks. Where ast-grep read the source at the pinned commit (Forge, Forge+ or Deep tier, `analysis_confidence` `full`, `workspaceDrift` not `overridden` in the `{outputFile}` frontmatter) and init.md §2 bound `{forge_provenance_map}`, check the lines the skill cites for them: resolve `{verifyProvenanceCompletenessHelper}` ← first existing path in `{verifyProvenanceCompletenessProbeOrder}` and run once:
|
|
234
|
+
- **Missing names** (`missing`, barrel or scalar): Medium `missing-export` gaps (Medium `missing-type` for a name `signatures.json` lists in `missingTypes`), each at the file `surface.json` records for it, so update-skill can re-extract the export from it; when `guards.inflation.fires` on a stratified-scope or multi-entry `scope.include` set, one Medium `denominator-inflation` gap beside the guard's at the brief lists them instead. A missing export does not block: it lowers Export Coverage.
|
|
235
|
+
- **Stale names** (`stale` when `staleApplicable` is true): each is a documented name the enumerated source surface lacks. Where ast-grep read the source at the pinned commit (Forge, Forge+ or Deep tier, `analysis_confidence` `full`, `workspaceDrift` not `overridden` in the `{outputFile}` frontmatter) and init.md §2 bound `{forge_provenance_map}`, check the lines the skill cites for them: resolve `{verifyProvenanceCompletenessHelper}` ← first existing path in `{verifyProvenanceCompletenessProbeOrder}` and run once:
|
|
251
236
|
|
|
252
237
|
```bash
|
|
253
|
-
uv run {verifyProvenanceCompletenessHelper} classify-stale --names "{run_dir}/coverage.json" --provenance "{forge_provenance_map}" --source-root "{source_path}" -o "{run_dir}/stale.json"
|
|
238
|
+
uv run {verifyProvenanceCompletenessHelper} classify-stale --names "{run_dir}/coverage.json" --provenance "{forge_provenance_map}" --source-root "{source_path}" --inventory "{run_dir}/inventory.json" -o "{run_dir}/stale.json"
|
|
254
239
|
```
|
|
255
240
|
|
|
256
|
-
A name it marks `fabricated: true` (every map entry of the name cites a missing file or one that does not define it) is a Critical `fabricated-signature` gap at its `source`. Any other stale name (no
|
|
241
|
+
A name it marks `fabricated: true` (every map entry of the name cites a missing file or one that does not define it) is a Critical `fabricated-signature` gap at its `source`. A name with a `defined_at` (the source still declares it there, an import never counting, a module so named counting unless the name's inventory kind rules it out), or whose `declared_in` lists several files of which the skill's `[AST:]`/`[SRC:]` citations on lines naming it cite exactly one, is a documented extra: an Info `observation` gap at that declaration, when `surface.json` has an `extraction` and its `excluded.outsideScope` does not list the name (a brief scoped its file out). Any other stale name (no `defined_at` and no single cited declaring file, one scoped out, no `extraction`, a dotted name, the conditions above not met, a helper that does not resolve or exits non-zero) is a Medium `stale-documentation` gap; one the brief's globs scope out says which.
|
|
257
242
|
|
|
258
|
-
### 4.
|
|
243
|
+
### 4. The Denominator Record
|
|
259
244
|
|
|
260
|
-
|
|
245
|
+
Record the denominator source in the Coverage Analysis section with the annotation `{sourceAccessProtocol}` gives `denominator.clause` (§4b gives a stack's and a reference app's): `Denominator: {barrel | empty barrel (…) | stratified (…) | multi-entry (…) | specific-modules (…) | pattern-reference (…) | stack composition (…) | pattern-surface (…) | docs-only completeness}`, and the `canonical.summary` line when `surface.json` has a fold. When `denominator.clause` is `stratified-scope` or `multi-entry`, load `{coverageBranchesFile}` and append the block of its **Denominator Candidates** section after the `Denominator:` line, with every count from `surface.json` `candidates` (`absent` for a null one).
|
|
261
246
|
|
|
262
|
-
|
|
263
|
-
- **Signature Accuracy:** `signatures.json` `signatureAccuracy` (Forge, Forge+ and Deep tier at State 1; "N/A" otherwise)
|
|
264
|
-
- **Type Coverage:** `signatures.json` `typeCoverage`, over the types of the set §2b scored; "N/A" where Signature Accuracy is
|
|
265
|
-
|
|
266
|
-
Record the denominator source in the Coverage Analysis section with the annotation the matching clause of `{sourceAccessProtocol}` specifies: `Denominator: {barrel | stratified (…) | multi-entry (…) | specific-modules (…) | pattern-reference (…) | stack composition (…) | pattern-surface (…) | docs-only completeness}`, and the `canonical.summary` line when `surface.json` has a fold. When stratified-scope or multi-entry resolution picked the denominator, append this block right after the `Denominator:` line, with every count from `surface.json` `candidates` (`absent` for a null one):
|
|
267
|
-
|
|
268
|
-
```markdown
|
|
269
|
-
**Denominator Candidates** (stratified-scope audit trail):
|
|
270
|
-
- `stats.effective_denominator`: {statsEffectiveDenominator | absent} {← chosen if priority (1) applied}
|
|
271
|
-
- `scope.tier_a_include` union: {tierAIncludeUnion | absent} {← chosen if priority (2) applied}
|
|
272
|
-
- `scope.include` union: {scopeIncludeUnion | absent} {← chosen if priority (3) applied}
|
|
273
|
-
- exports-map subpath union: {subpathUnion | absent} {← chosen if the multi-entry clause applied}
|
|
274
|
-
- root barrel: {rootBarrel | absent} {secondary candidate: root-barrel-vs-subpath-union audit}
|
|
275
|
-
```
|
|
276
|
-
|
|
277
|
-
The guards raise coverage gaps too, which §5b records from `surface.json`: `guards.deflation.fires` is a Medium `metadata-drift` gap and `guards.inflation.fires` a Medium `denominator-inflation` gap.
|
|
247
|
+
The guards raise coverage gaps too, which §5b records from `surface.json`: `guards.deflation.fires` is a Medium `metadata-drift` gap (on its `tier_a_include` basis, a stale stored count to recount), `guards.inflation.fires` a Medium `denominator-inflation` gap (with deflation, both name one fix: the brief's tier A list) and `guards.staleScope.fires` a Medium `brief-scope-stale` gap at the brief.
|
|
278
248
|
|
|
279
249
|
### 4b. Metadata Export-Count Coherence Cross-Check
|
|
280
250
|
|
|
@@ -286,11 +256,11 @@ uv run {coherenceScript} --inputs "{run_dir}/coverage-inputs.json" > "{run_dir}/
|
|
|
286
256
|
|
|
287
257
|
A non-zero exit leaves the script's error, or nothing, in that file: delete it, so §5b skips its call and records no count finding, and note the error in the Coverage Analysis section. §5b records each `findings[]` entry: a Medium entry as a Medium `metadata-drift` gap and an Info entry as an Info `multi-denominator` gap. They describe data quality and change no denominator. Record the denominator of a stack (`Denominator: stack composition ({N} cited contracts)` or `({N} libraries + integration pairs)`, from `stack.basis`) and of a reference app (`Denominator: pattern-surface ({pattern_surfaces_documented})`).
|
|
288
258
|
|
|
289
|
-
**The numerator ground truth §2b ran.** `numerator.json` `inflated: true` (`verified < declared`) is a High `numerator-inflation` gap listing its `absent[]` names, which §5b records; on
|
|
259
|
+
**The numerator ground truth §2b ran.** `numerator.json` `inflated: true` (`verified < declared`) is a High `numerator-inflation` gap listing its `absent[]` names, which §5b records; on a scalar branch that counts no set those names also replace the missing-count gap in the Scalar and Stack Branches section of coverage-check-branches.md. `inflated: false` or `skipped: true` is no finding.
|
|
290
260
|
|
|
291
261
|
### 4c. Provenance Line Check
|
|
292
262
|
|
|
293
|
-
Check that the provenance map records each export at the line that defines it (the `def` or declaration line
|
|
263
|
+
Check that the provenance map records each export at the line that defines it (the `def` or declaration line, not a decorator or blank line above it), so update-skill can move a wrong line. Run this section only when `analysis_confidence` is `full`, `{forge_version}/provenance-map.json` exists, and the `{outputFile}` frontmatter's `workspaceDrift` is not `overridden` (init.md §5b: the drifted tree is not the pinned commit). Otherwise skip to section 5.
|
|
294
264
|
|
|
295
265
|
Resolve `{verifyProvenanceCompletenessHelper}` ← first existing path in `{verifyProvenanceCompletenessProbeOrder}`. If neither path exists, skip to section 5: the check is advisory. Otherwise run:
|
|
296
266
|
|
|
@@ -308,7 +278,13 @@ This section reports none of the helper's other findings, and neither gap blocks
|
|
|
308
278
|
|
|
309
279
|
### 5. Write the Coverage Analysis Section
|
|
310
280
|
|
|
311
|
-
Write the **Coverage Analysis** section in place of the template's `## Coverage Analysis` heading and the placeholder comment under it, in `{outputFile}`.
|
|
281
|
+
Write the **Coverage Analysis** section in place of the template's `## Coverage Analysis` heading and the placeholder comment under it, in `{outputFile}`. `{reconcileScript}` renders its counts, per-export table and category scores from the run files (leave out `--surface` on a docs-only run and `--signatures` when §2b scored none):
|
|
282
|
+
|
|
283
|
+
```bash
|
|
284
|
+
uv run {reconcileScript} render --coverage "{run_dir}/coverage.json" [--surface "{run_dir}/surface.json"] [--signatures "{run_dir}/signatures.json"]
|
|
285
|
+
```
|
|
286
|
+
|
|
287
|
+
Exit 0: write its output unchanged where the section below places it. Exit 2 is a usage error: correct the command and run it again. Exit 1 printed its error on stderr: HALT with it (`halt_reason: "helper-failed"`, phase `coverage-check:render`).
|
|
312
288
|
|
|
313
289
|
```markdown
|
|
314
290
|
## Coverage Analysis
|
|
@@ -317,32 +293,12 @@ Write the **Coverage Analysis** section in place of the template's `## Coverage
|
|
|
317
293
|
**Source Access:** {analysis_confidence} (full | provenance-map | metadata-only | remote-only | docs-only)
|
|
318
294
|
**Source Path:** {source_path}
|
|
319
295
|
**Workspace Layout:** {source_workspace_kind | single package | unknown (no local source, or the §0b helper failed)}
|
|
320
|
-
**Files Analyzed:** {count}
|
|
321
296
|
**Denominator:** {the §4 annotation}
|
|
297
|
+
{the Denominator Candidates block, when §4 adds one}
|
|
322
298
|
|
|
323
|
-
|
|
324
|
-
|
|
325
|
-
| Export | Type | Documented | Signature | Source Location | Status |
|
|
326
|
-
|--------|------|-----------|-----------|-----------------|--------|
|
|
327
|
-
| {name} | {kind} | yes/no | yes/no/unverified | {file}:{line} | PASS/FAIL/WARN |
|
|
328
|
-
|
|
329
|
-
### Coverage Summary
|
|
299
|
+
{the render output}
|
|
330
300
|
|
|
331
|
-
|
|
332
|
-
- **Documented:** {`documented`} ({`exportCoverage`}%)
|
|
333
|
-
- **Missing Documentation:** {`missingCount`; docs-only: the incomplete items, each with the fields it lacks}
|
|
334
|
-
- **Signature Mismatches:** {the length of `signatures.json` `mismatches`}
|
|
335
|
-
- **Stale Documentation:** {`staleCount`}
|
|
336
|
-
- **Scoring Warnings:** each entry of `signatures.json` `warnings`; omit this row when there is none
|
|
337
|
-
- **Numerator Surplus:** {`numeratorSurplus`}, uncapped coverage {`coverageUncapped`}%; only when `coverageCapped: true` (then `Missing Documentation: 0` is a floored residual)
|
|
338
|
-
|
|
339
|
-
### Category Scores
|
|
340
|
-
|
|
341
|
-
| Category | Score |
|
|
342
|
-
|----------|-------|
|
|
343
|
-
| Export Coverage | {N}% |
|
|
344
|
-
| Signature Accuracy | {N}% or N/A |
|
|
345
|
-
| Type Coverage | {N}% or N/A |
|
|
301
|
+
{the notes §2 to §4b add to this section}
|
|
346
302
|
```
|
|
347
303
|
|
|
348
304
|
### 5b. Record the Coverage Gaps
|
|
@@ -358,6 +314,8 @@ uv run {gapLedgerScript} append --ledger "{ledgerFile}" --stage coverage-check -
|
|
|
358
314
|
uv run {gapLedgerScript} append --ledger "{ledgerFile}" --stage coverage-check --from provenance-line --input "{run_dir}/provenance-verify.json"
|
|
359
315
|
```
|
|
360
316
|
|
|
317
|
+
**A split-body mismatch's Source** is a line inside the skill package. At State 1 and the Forge tier or above, with `workspaceDrift` not `overridden`, set the two forms its `issue` gives against the export's declaration at the `file` and `line` `surface.json` gives, when that one declaration decides (no overload set, no second declaring file): when the reference agrees and the body does not, the Source is `SKILL.md:{skill_md_line}`, else `{reference_file}:{reference_line}`. Its remediation names the file to edit and the file and line whose form it takes, citing the declaration's `file:line`; when both forms agree with it, it says so and names what differs; when neither does, or nothing decides, it says no source settled it.
|
|
318
|
+
|
|
361
319
|
Then write the records no script wrote, §1b's split-body mismatches, in the Ledger Record Format of `{outputFormatsFile}`, each naming its export in `export` and in its title: the ledger tells two gaps apart by their title, Source and `export`, and update-skill takes the export's name from `export`. Write them as one JSON array on the lines between the two markers, exactly as they are: the quoted marker hands them to the script unchanged, quotes, apostrophes and `$` included. Run the command even when the array is empty (`[]`): the hard gate refuses to decide until every stage before it has recorded, with gaps or without.
|
|
362
320
|
|
|
363
321
|
```bash
|
|
@@ -374,6 +332,6 @@ Rely on the JSON of each command:
|
|
|
374
332
|
|
|
375
333
|
### 6. Report Coverage Results
|
|
376
334
|
|
|
377
|
-
Report the
|
|
335
|
+
Report to the user from §5's render: the {forge_tier} tier, the **Files Analyzed** count, the three category scores and the issue count; the Coverage Analysis section has the details.
|
|
378
336
|
|
|
379
337
|
Append `'coverage-check'` to `stepsCompleted` in the `{outputFile}` frontmatter, then load and execute {nextStepFile}.
|
|
@@ -243,18 +243,16 @@ Read `{sidecar_path}/preferences.yaml`. If `tier_override` is set and is a valid
|
|
|
243
243
|
Read `metadata.json` to extract:
|
|
244
244
|
- `name` — display name
|
|
245
245
|
- `skill_type` — single or stack (needed for mode detection)
|
|
246
|
-
- `source_path`
|
|
246
|
+
- `source_root`: the source tree the skill was extracted from; bind `{source_path}` ← `source_root` (null or absent for a docs-only skill and a stack skill)
|
|
247
247
|
- `source_commit` — pinned commit the skill was extracted against (may be null for docs-only skills, `"local"` for non-git sources, or a per-repo map for stack skills)
|
|
248
248
|
- `source_ref` — pinned ref (tag/branch/`HEAD`) used at extraction time
|
|
249
249
|
- `generation_date` — when skill was generated
|
|
250
250
|
- `confidence_tier` — tier used during creation
|
|
251
251
|
- `generated_by`: the SKF workflow that built the skill (coverage-check §2 reads it at Quick tier)
|
|
252
252
|
|
|
253
|
-
If source path override was provided as optional input, use that instead.
|
|
254
|
-
|
|
255
253
|
### 5b. Verify Workspace HEAD Matches Pinned Commit
|
|
256
254
|
|
|
257
|
-
Test-skill reads `source_path` during coverage and coherence analysis. If the local workspace has drifted from `metadata.source_commit`, gap and signature-mismatch findings silently reflect the drifted tree, not the skill's pinned source: false positives that downstream update-skill runs may then "repair" by corrupting correct documentation.
|
|
255
|
+
Test-skill reads `{source_path}` during coverage and coherence analysis. If the local workspace has drifted from `metadata.source_commit`, gap and signature-mismatch findings silently reflect the drifted tree, not the skill's pinned source: false positives that downstream update-skill runs may then "repair" by corrupting correct documentation.
|
|
258
256
|
|
|
259
257
|
The guard runs through `{checkWorkspaceDriftHelper}`, never through `git` commands run by hand. Resolve it ← first existing path in `{checkWorkspaceDriftProbeOrder}`. If neither path exists, HALT (`halt_reason: "helper-missing"`, phase `init:workspace-drift`) with: "Error: cannot locate skf-check-workspace-drift.py at `{project-root}/_bmad/skf/shared/scripts/skf-check-workspace-drift.py` or `{project-root}/src/shared/scripts/skf-check-workspace-drift.py`. Install the SKF module or run from a development checkout with src/ present." No test report is written.
|
|
260
258
|
|
|
@@ -264,7 +262,7 @@ Run it once per source tree, from `{project-root}`, passing `--allow-drift` only
|
|
|
264
262
|
uv run {checkWorkspaceDriftHelper} "{tree}" --pinned-commit "{pinned_commit}" [--source-ref "{source_ref}"] [--allow-drift] --workflow test-skill
|
|
265
263
|
```
|
|
266
264
|
|
|
267
|
-
- **One tree:** `{tree}` is `{source_path}`, `{pinned_commit}` is `metadata.source_commit` (an empty string when it is null), and `{source_ref}` is `metadata.source_ref` when it is set.
|
|
265
|
+
- **One tree:** `{tree}` is `{source_path}`, `{pinned_commit}` is `metadata.source_commit` (an empty string when it is null), and `{source_ref}` is `metadata.source_ref` when it is set. When metadata.json records no `source_root` (null, empty or absent), make no call and log `workspace_drift_check: skipped (metadata.json records no source_root)`.
|
|
268
266
|
- **A stack skill** (`metadata.source_commit` is a `{repo_path: commit}` map): one call per entry, with that repo path as `{tree}`, its commit as `{pinned_commit}`, and the same key's value of `metadata.source_ref` as `{source_ref}` when that is a map too. Check every repo: do not skip stack skills.
|
|
269
267
|
- Make no call for a tree that is not on disk (coverage then reads the provenance map, State 2 of the source access protocol), and log `workspace_drift_check: skipped (no local source at {tree})`. The helper itself skips a tree with no pinned commit (`""` or `"local"`) and one that is not a git working tree.
|
|
270
268
|
|
|
@@ -315,13 +313,14 @@ thresholdSource: '{threshold_source}'
|
|
|
315
313
|
analysisConfidence: ''
|
|
316
314
|
toolingStatus: '{ok|frontmatter-validator-timeout}'
|
|
317
315
|
workspaceDrift: '{not-checked|ok|overridden}'
|
|
316
|
+
health_check_dispatched: false
|
|
318
317
|
testDate: '{run_id timestamp ISO-8601 UTC}'
|
|
319
318
|
stepsCompleted: ['init']
|
|
320
319
|
nextWorkflow: ''
|
|
321
320
|
---
|
|
322
321
|
```
|
|
323
322
|
|
|
324
|
-
`toolingStatus` is the `tooling_status` §3b set: score.md §3a reads it back for Cap 1. `threshold` and `thresholdSource` are the pass threshold §1b resolved and its source: score.md §1 reads them back, and score.md §7 rewrites `threshold` with the threshold the verdict was settled against. `analysisConfidence` stays empty until score.md §7 writes the source access coverage-check resolved: `full`, `provenance-map`, `metadata-only`, `remote-only` or `docs-only`.
|
|
323
|
+
Replace the template's frontmatter with this block (the two hold the same keys). `toolingStatus` is the `tooling_status` §3b set: score.md §3a reads it back for Cap 1. `threshold` and `thresholdSource` are the pass threshold §1b resolved and its source: score.md §1 reads them back, and score.md §7 rewrites `threshold` with the threshold the verdict was settled against. `analysisConfidence` stays empty until score.md §7 writes the source access coverage-check resolved: `full`, `provenance-map`, `metadata-only`, `remote-only` or `docs-only`.
|
|
325
324
|
|
|
326
325
|
### 7. Report Initialization Status
|
|
327
326
|
|
|
@@ -16,7 +16,8 @@ Execute this check only when both conditions are met:
|
|
|
16
16
|
the flat copy an older skill may still keep); the detection contract below
|
|
17
17
|
reads it.
|
|
18
18
|
|
|
19
|
-
If either condition fails, skip
|
|
19
|
+
If either condition fails, skip the check with no note and proceed to the next
|
|
20
|
+
section: a missing evidence report is not Case 4.
|
|
20
21
|
|
|
21
22
|
The check runs regardless of naive/contextual mode. T2-future annotations are a
|
|
22
23
|
property of the source code and enrichment data, not the skill type.
|
|
@@ -54,23 +55,26 @@ narrative body.
|
|
|
54
55
|
**Detection contract.** Read the frontmatter deterministically:
|
|
55
56
|
|
|
56
57
|
```bash
|
|
57
|
-
# Extract t2_future_count from frontmatter. Requires a `---` delimiter pair
|
|
58
|
-
|
|
58
|
+
# Extract t2_future_count from frontmatter. Requires a `---` delimiter pair;
|
|
59
|
+
# a trailing CR is stripped first, so a CRLF report reads the same.
|
|
60
|
+
awk '{sub(/\r$/,"")} /^---$/{c++;next} c==1 && /^t2_future_count:/{print $2; exit}' \
|
|
59
61
|
{forge_evidence_report}
|
|
60
62
|
```
|
|
61
63
|
|
|
62
|
-
- **Frontmatter missing OR `t2_future_count` absent
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
invert Case-1 vs Case-2/3
|
|
67
|
-
|
|
68
|
-
|
|
64
|
+
- **Frontmatter missing, OR `t2_future_count` absent or not a non-negative
|
|
65
|
+
integer** (empty, `null`, quoted) → treat as Case 4 (see below). Do not
|
|
66
|
+
fall back to grepping prose (`grep "T2-future"`): prose drift (heading
|
|
67
|
+
renames, alt phrasings like "forward-looking annotations", capitalization
|
|
68
|
+
variance) silently breaks detection and can invert Case-1 vs Case-2/3
|
|
69
|
+
severity.
|
|
70
|
+
- **`t2_future_count` a non-negative integer** (digits only) → use its value
|
|
71
|
+
for the Case Rules below.
|
|
69
72
|
|
|
70
73
|
The pinned field is emitted by `skf-create-skill/references/compile.md`
|
|
71
74
|
§7 (frontmatter-pinned fields), which always writes `t2_future_count: N`
|
|
72
75
|
(including 0). Legacy skills whose `evidence-report.md` predates the pinned
|
|
73
|
-
field land in Case 4
|
|
76
|
+
field land in Case 4, and so does every stack skill: create-stack-skill's
|
|
77
|
+
evidence report has no frontmatter.
|
|
74
78
|
|
|
75
79
|
### Case 1 — T2-future > 0 AND Section 4b absent
|
|
76
80
|
|
|
@@ -106,11 +110,19 @@ Flag as **Medium** severity gap:
|
|
|
106
110
|
Reviewer may downgrade to Low with inline justification on a case-by-case
|
|
107
111
|
basis.
|
|
108
112
|
|
|
109
|
-
### Case 4
|
|
113
|
+
### Case 4: evidence report has no pinned count
|
|
110
114
|
|
|
111
|
-
|
|
115
|
+
The evidence report exists but pins no usable `t2_future_count`: the field is
|
|
116
|
+
absent or not a non-negative integer, or the report has no frontmatter at all
|
|
117
|
+
(a create-skill report older than the field, or any create-stack-skill
|
|
118
|
+
report). Skip the Case Rules and raise no gap. Record this note on the
|
|
119
|
+
Coherence Analysis section's **Migration Section:** line (`{outputFormatsFile}`
|
|
120
|
+
gives it in both formats):
|
|
112
121
|
|
|
113
|
-
> "Section 4b verification skipped
|
|
122
|
+
> "Section 4b verification skipped: evidence-report.md pins no
|
|
123
|
+
> `t2_future_count`. For a skill create-skill made, re-run create-skill to
|
|
124
|
+
> regenerate the report with the field (create-stack-skill does not write
|
|
125
|
+
> it)."
|
|
114
126
|
|
|
115
127
|
## Output
|
|
116
128
|
|