bmad-module-skill-forge 2.2.0 → 3.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +1 -1
- package/README.md +49 -49
- package/docs/404.md +10 -8
- package/docs/_data/pinned.yaml +22 -13
- package/docs/_internal/STABILITY.md +58 -35
- package/docs/agents.md +39 -53
- package/docs/architecture.md +65 -51
- package/docs/bmad-synergy.md +53 -51
- package/docs/campaign.md +127 -75
- package/docs/concepts.md +34 -28
- package/docs/examples.md +70 -68
- package/docs/forge-auto.md +34 -31
- package/docs/getting-started.md +118 -57
- package/docs/how-it-works.md +25 -21
- package/docs/index.md +15 -15
- package/docs/skill-model.md +70 -59
- package/docs/troubleshooting.md +135 -18
- package/docs/verifying-a-skill.md +83 -57
- package/docs/why-skf.md +24 -22
- package/docs/workflows.md +274 -110
- package/package.json +18 -10
- package/src/forger/preferences.yaml +8 -1
- package/src/knowledge/agentskills-spec.md +2 -0
- package/src/knowledge/ccc-bridge.md +40 -24
- package/src/knowledge/confidence-tiers.md +9 -5
- package/src/knowledge/overview.md +3 -3
- package/src/knowledge/progressive-capability.md +8 -8
- package/src/knowledge/provenance-tracking.md +9 -5
- package/src/knowledge/qmd-registry.md +2 -2
- package/src/knowledge/skf-knowledge-index.csv +2 -2
- package/src/knowledge/skill-lifecycle.md +18 -11
- package/src/knowledge/split-body-strategy.md +0 -3
- package/src/knowledge/tool-resolution.md +4 -2
- package/src/knowledge/version-paths.md +101 -15
- package/src/module-help.csv +3 -3
- package/src/module.yaml +10 -0
- package/src/shared/data/ast-grep-recipes.yaml +608 -0
- package/src/shared/data/ide-context-files.json +30 -0
- package/src/shared/health-check.md +32 -12
- package/src/shared/references/description-guard-protocol.md +12 -9
- package/src/shared/references/feasibility-report-schema.md +19 -13
- package/src/shared/references/headless-gate-convention.md +55 -3
- package/src/shared/references/output-contract-schema.md +19 -8
- package/src/shared/references/pipeline-contracts.md +94 -29
- package/src/shared/references/tessl-review.md +77 -0
- package/src/shared/scripts/schemas/skf-analyze-result-envelope.v1.json +194 -0
- package/src/shared/scripts/schemas/skf-audit-result-envelope.v1.json +192 -0
- package/src/shared/scripts/schemas/skf-brief-result-envelope.v1.json +32 -4
- package/src/shared/scripts/schemas/skf-campaign-result-envelope.v1.json +174 -0
- package/src/shared/scripts/schemas/skf-create-skill-result-envelope.v1.json +284 -0
- package/src/shared/scripts/schemas/skf-drop-skill-result-envelope.v1.json +164 -0
- package/src/shared/scripts/schemas/skf-export-result-envelope.v1.json +160 -0
- package/src/shared/scripts/schemas/skf-quick-skill-result-envelope.v1.json +261 -0
- package/src/shared/scripts/schemas/skf-refine-architecture-result-envelope.v1.json +192 -0
- package/src/shared/scripts/schemas/skf-rename-skill-result-envelope.v1.json +179 -0
- package/src/shared/scripts/schemas/skf-setup-result-envelope.v1.json +27 -11
- package/src/shared/scripts/schemas/skf-stack-result-envelope.v1.json +176 -0
- package/src/shared/scripts/schemas/skf-test-result-envelope.v1.json +173 -0
- package/src/shared/scripts/schemas/skf-update-result-envelope.v1.json +19 -5
- package/src/shared/scripts/schemas/skf-verify-stack-result-envelope.v1.json +180 -0
- package/src/shared/scripts/schemas/skill-brief.v1.json +34 -0
- package/src/shared/scripts/schemas/workspace-detection.v1.json +1 -1
- package/src/shared/scripts/skf-atomic-write.py +30 -14
- package/src/shared/scripts/skf-build-change-manifest.py +2203 -24
- package/src/shared/scripts/skf-ccc-git-hygiene.py +532 -0
- package/src/shared/scripts/skf-check-unit-records.py +184 -0
- package/src/shared/scripts/skf-check-workspace-drift.py +538 -9
- package/src/shared/scripts/skf-classify-changed-files.py +694 -0
- package/src/shared/scripts/skf-comention-pairs.py +1333 -123
- package/src/shared/scripts/skf-compare-file-hashes.py +1 -1
- package/src/shared/scripts/skf-count-imports.py +1867 -0
- package/src/shared/scripts/skf-count-tokens.py +1 -1
- package/src/shared/scripts/skf-derive-assembly-shape.py +3 -2
- package/src/shared/scripts/skf-description-guard.py +99 -11
- package/src/shared/scripts/skf-detect-docs.py +694 -2
- package/src/shared/scripts/skf-detect-language.py +486 -107
- package/src/shared/scripts/skf-detect-registry.py +887 -0
- package/src/shared/scripts/skf-detect-scripts-assets.py +1 -1
- package/src/shared/scripts/skf-detect-tools.py +209 -27
- package/src/shared/scripts/skf-detect-workspaces.py +506 -30
- package/src/shared/scripts/skf-disqualify-candidates.py +821 -182
- package/src/shared/scripts/skf-emit-brief-result-envelope.py +69 -191
- package/src/shared/scripts/skf-emit-result-envelope.py +1604 -106
- package/src/shared/scripts/skf-enumerate-stack-skills.py +986 -117
- package/src/shared/scripts/skf-extract-public-api.py +4084 -85
- package/src/shared/scripts/skf-extraction-inventory.py +746 -0
- package/src/shared/scripts/skf-extraction-snapshot.py +539 -0
- package/src/shared/scripts/skf-fetch-temporal.py +674 -0
- package/src/shared/scripts/skf-find-cycles.py +97 -29
- package/src/shared/scripts/skf-find-test-report.py +476 -0
- package/src/shared/scripts/skf-forge-tier-rw.py +444 -203
- package/src/shared/scripts/skf-github-fetch.py +365 -0
- package/src/shared/scripts/skf-github-probe.py +661 -0
- package/src/shared/scripts/skf-hash-content.py +166 -12
- package/src/shared/scripts/skf-language-corpora.py +1 -1
- package/src/shared/scripts/skf-load-provenance.py +159 -27
- package/src/shared/scripts/skf-manifest-ops.py +1 -1
- package/src/shared/scripts/skf-merge-ccc-exclusions.py +2307 -223
- package/src/shared/scripts/skf-merge-doc-urls.py +1 -1
- package/src/shared/scripts/skf-names-present.py +221 -0
- package/src/shared/scripts/skf-pair-intersect.py +119 -28
- package/src/shared/scripts/skf-parse-gaps.py +1384 -0
- package/src/shared/scripts/skf-preapply.py +1 -1
- package/src/shared/scripts/skf-preflight.py +161 -27
- package/src/shared/scripts/skf-promote-staged.py +430 -0
- package/src/shared/scripts/skf-provenance-gap-dispatch.py +39 -7
- package/src/shared/scripts/skf-qmd-classify-collections.py +229 -45
- package/src/shared/scripts/skf-quick-batch.py +774 -0
- package/src/shared/scripts/skf-rebuild-managed-sections.py +972 -88
- package/src/shared/scripts/skf-recommend-scope-type.py +323 -156
- package/src/shared/scripts/skf-render-metadata-stats.py +156 -12
- package/src/shared/scripts/skf-render-quick-metadata.py +132 -28
- package/src/shared/scripts/skf-render-stack-metadata.py +976 -0
- package/src/shared/scripts/skf-resolve-authoritative-files.py +128 -13
- package/src/shared/scripts/skf-resolve-package.py +523 -106
- package/src/shared/scripts/skf-rewrite-skill-name.py +433 -47
- package/src/shared/scripts/skf-run-lock.py +577 -0
- package/src/shared/scripts/skf-scan-manifests.py +998 -57
- package/src/shared/scripts/skf-scan-skill-md-structure.py +1341 -37
- package/src/shared/scripts/skf-severity-classify.py +712 -108
- package/src/shared/scripts/skf-shape-detect.py +298 -204
- package/src/shared/scripts/skf-shard-body.py +9 -8
- package/src/shared/scripts/skf-skill-inventory.py +2412 -89
- package/src/shared/scripts/skf-skills-module.py +472 -0
- package/src/shared/scripts/skf-source-tree.py +1796 -0
- package/src/shared/scripts/skf-structural-diff.py +670 -119
- package/src/shared/scripts/skf-tessl-review.py +706 -0
- package/src/shared/scripts/skf-update-active-symlink.py +5 -5
- package/src/shared/scripts/skf-update-run-state.py +624 -0
- package/src/shared/scripts/skf-validate-brief-inputs.py +92 -29
- package/src/shared/scripts/skf-validate-brief-schema.py +4 -1
- package/src/shared/scripts/skf-validate-feasibility-report.py +577 -66
- package/src/shared/scripts/skf-validate-frontmatter.py +103 -26
- package/src/shared/scripts/skf-validate-output.py +712 -64
- package/src/shared/scripts/skf-validate-pins.py +1 -1
- package/src/shared/scripts/skf-verify-no-trace.py +116 -10
- package/src/shared/scripts/skf-verify-provenance-completeness.py +2919 -54
- package/src/shared/scripts/skf-write-skill-brief.py +367 -19
- package/src/shared/tool-requirements.yaml +201 -0
- package/src/skf-analyze-source/SKILL.md +41 -36
- package/src/skf-analyze-source/assets/skill-brief-schema.md +7 -8
- package/src/skf-analyze-source/customize.toml +36 -24
- package/src/skf-analyze-source/references/auto-docs-only.md +58 -20
- package/src/skf-analyze-source/references/continue.md +2 -13
- package/src/skf-analyze-source/references/discover-additional-source.md +147 -0
- package/src/skf-analyze-source/references/generate-briefs.md +121 -55
- package/src/skf-analyze-source/references/headless-contract.md +42 -15
- package/src/skf-analyze-source/references/identify-units.md +96 -110
- package/src/skf-analyze-source/references/init.md +73 -71
- package/src/skf-analyze-source/references/map-and-detect.md +123 -105
- package/src/skf-analyze-source/references/map-unit-exports.md +68 -0
- package/src/skf-analyze-source/references/recommend.md +51 -70
- package/src/skf-analyze-source/references/scan-project.md +60 -84
- package/src/skf-analyze-source/references/scan-root.md +39 -0
- package/src/skf-analyze-source/references/step-auto-scope-coexistence.md +47 -0
- package/src/skf-analyze-source/references/step-auto-scope-corpora.md +53 -0
- package/src/skf-analyze-source/references/step-auto-scope-split.md +91 -0
- package/src/skf-analyze-source/references/step-auto-scope.md +208 -293
- package/src/skf-analyze-source/references/step-shape-detect.md +14 -12
- package/src/skf-analyze-source/references/unit-detection-heuristics.md +37 -42
- package/src/skf-audit-skill/SKILL.md +43 -51
- package/src/skf-audit-skill/assets/drift-report-template.md +27 -3
- package/src/skf-audit-skill/customize.toml +41 -20
- package/src/skf-audit-skill/references/constituent-freshness.md +91 -0
- package/src/skf-audit-skill/references/doc-drift.md +141 -0
- package/src/skf-audit-skill/references/headless-contract.md +29 -0
- package/src/skf-audit-skill/references/init.md +148 -210
- package/src/skf-audit-skill/references/re-index.md +96 -108
- package/src/skf-audit-skill/references/report.md +96 -33
- package/src/skf-audit-skill/references/semantic-diff.md +27 -18
- package/src/skf-audit-skill/references/severity-classify.md +79 -65
- package/src/skf-audit-skill/references/severity-rules.md +15 -6
- package/src/skf-audit-skill/references/structural-diff.md +75 -90
- package/src/skf-audit-skill/references/upstream-checkout.md +77 -0
- package/src/skf-audit-skill/scripts/render-drift-tables.py +685 -0
- package/src/skf-brief-skill/SKILL.md +36 -25
- package/src/skf-brief-skill/assets/description-voice-examples.md +1 -1
- package/src/skf-brief-skill/assets/scope-templates.md +17 -24
- package/src/skf-brief-skill/assets/skill-brief-schema.md +29 -15
- package/src/skf-brief-skill/customize.toml +48 -20
- package/src/skf-brief-skill/references/analyze-target.md +142 -125
- package/src/skf-brief-skill/references/confirm-brief.md +10 -8
- package/src/skf-brief-skill/references/draft-checkpoint.md +36 -17
- package/src/skf-brief-skill/references/gather-intent-ratify.md +68 -0
- package/src/skf-brief-skill/references/gather-intent.md +87 -182
- package/src/skf-brief-skill/references/headless-args.md +84 -16
- package/src/skf-brief-skill/references/health-check.md +1 -1
- package/src/skf-brief-skill/references/invocation-contract.md +18 -13
- package/src/skf-brief-skill/references/portfolio-similarity-check.md +1 -1
- package/src/skf-brief-skill/references/qmd-collection-registration.md +7 -10
- package/src/skf-brief-skill/references/scope-definition.md +123 -47
- package/src/skf-brief-skill/references/step-auto-brief.md +24 -80
- package/src/skf-brief-skill/references/step-auto-validate.md +20 -121
- package/src/skf-brief-skill/references/version-resolution.md +3 -2
- package/src/skf-brief-skill/references/write-brief.md +66 -57
- package/src/skf-campaign/SKILL.md +31 -34
- package/src/skf-campaign/assets/campaign-state-schema.json +25 -4
- package/src/skf-campaign/customize.toml +47 -16
- package/src/skf-campaign/references/campaign-contracts.md +39 -29
- package/src/skf-campaign/references/campaign-directive-spec.md +4 -2
- package/src/skf-campaign/references/health-check.md +7 -16
- package/src/skf-campaign/references/step-01-setup.md +46 -84
- package/src/skf-campaign/references/step-02-strategy.md +31 -30
- package/src/skf-campaign/references/step-03-pins.md +26 -18
- package/src/skf-campaign/references/step-04-provenance.md +24 -23
- package/src/skf-campaign/references/step-05-skill-loop.md +67 -48
- package/src/skf-campaign/references/step-06-batch.md +41 -34
- package/src/skf-campaign/references/step-07-capstone.md +40 -25
- package/src/skf-campaign/references/step-08-verify.md +28 -29
- package/src/skf-campaign/references/step-09-refine.md +29 -30
- package/src/skf-campaign/references/step-10-export.md +48 -36
- package/src/skf-campaign/references/step-11-maintenance.md +31 -34
- package/src/skf-campaign/references/step-resume.md +37 -68
- package/src/skf-campaign/scripts/campaign-deps.py +57 -18
- package/src/skf-campaign/scripts/campaign-parse-manifest.py +361 -54
- package/src/skf-campaign/scripts/campaign-provenance.py +96 -53
- package/src/skf-campaign/scripts/campaign-quality-gate.py +592 -0
- package/src/skf-campaign/scripts/campaign-render-batch.py +257 -49
- package/src/skf-campaign/scripts/campaign-render-kickoff.py +343 -50
- package/src/skf-campaign/scripts/campaign-report.py +204 -37
- package/src/skf-campaign/scripts/campaign-state.py +1128 -0
- package/src/skf-campaign/scripts/campaign-status.py +10 -74
- package/src/skf-campaign/scripts/campaign-validate-pins.py +30 -3
- package/src/skf-campaign/scripts/campaign-validate-state.py +34 -4
- package/src/skf-campaign/templates/campaign-brief-template.yaml +4 -5
- package/src/skf-campaign/templates/campaign-report-template.md +5 -1
- package/src/skf-campaign/templates/kickoff-template.md +0 -14
- package/src/skf-create-skill/SKILL.md +27 -20
- package/src/skf-create-skill/assets/compile-assembly-rules.md +6 -5
- package/src/skf-create-skill/assets/skill-sections.md +15 -5
- package/src/skf-create-skill/customize.toml +20 -8
- package/src/skf-create-skill/references/authoritative-files-protocol.md +41 -42
- package/src/skf-create-skill/references/batch-mode.md +81 -0
- package/src/skf-create-skill/references/ccc-index-check.md +27 -0
- package/src/skf-create-skill/references/compile.md +107 -29
- package/src/skf-create-skill/references/component-extraction.md +113 -91
- package/src/skf-create-skill/references/enrich.md +41 -23
- package/src/skf-create-skill/references/entry-points-by-hand.md +20 -0
- package/src/skf-create-skill/references/extract.md +200 -161
- package/src/skf-create-skill/references/extraction-patterns-by-hand.md +141 -0
- package/src/skf-create-skill/references/extraction-patterns-tracing.md +2 -2
- package/src/skf-create-skill/references/extraction-patterns.md +584 -196
- package/src/skf-create-skill/references/generate-artifacts.md +145 -93
- package/src/skf-create-skill/references/health-check.md +1 -1
- package/src/skf-create-skill/references/load-brief.md +53 -36
- package/src/skf-create-skill/references/relabel-rule.md +12 -0
- package/src/skf-create-skill/references/report.md +28 -59
- package/src/skf-create-skill/references/source-resolution-protocols.md +49 -155
- package/src/skf-create-skill/references/step-auto-shard.md +20 -93
- package/src/skf-create-skill/references/step-doc-rot.md +44 -60
- package/src/skf-create-skill/references/step-doc-sources.md +28 -19
- package/src/skf-create-skill/references/sub/ccc-discover.md +18 -7
- package/src/skf-create-skill/references/sub/fetch-docs.md +106 -59
- package/src/skf-create-skill/references/sub/fetch-temporal.md +53 -166
- package/src/skf-create-skill/references/tier-degradation-rules.md +18 -9
- package/src/skf-create-skill/references/validate.md +173 -99
- package/src/skf-create-skill/scripts/scan-doc-rot.py +232 -99
- package/src/skf-create-stack-skill/SKILL.md +29 -56
- package/src/skf-create-stack-skill/assets/metadata-contract.md +51 -0
- package/src/skf-create-stack-skill/assets/provenance-map-schema.md +15 -15
- package/src/skf-create-stack-skill/assets/stack-skill-template.md +23 -57
- package/src/skf-create-stack-skill/customize.toml +23 -12
- package/src/skf-create-stack-skill/references/compile-stack.md +57 -66
- package/src/skf-create-stack-skill/references/compose-mode-rules.md +24 -44
- package/src/skf-create-stack-skill/references/detect-integrations.md +92 -95
- package/src/skf-create-stack-skill/references/detect-manifests.md +46 -71
- package/src/skf-create-stack-skill/references/generate-output.md +138 -128
- package/src/skf-create-stack-skill/references/init.md +90 -32
- package/src/skf-create-stack-skill/references/integration-patterns.md +1 -17
- package/src/skf-create-stack-skill/references/invocation-contract.md +34 -0
- package/src/skf-create-stack-skill/references/manifest-patterns.md +32 -20
- package/src/skf-create-stack-skill/references/parallel-extract.md +112 -38
- package/src/skf-create-stack-skill/references/rank-and-confirm.md +57 -75
- package/src/skf-create-stack-skill/references/report.md +39 -33
- package/src/skf-create-stack-skill/references/validate.md +20 -39
- package/src/skf-drop-skill/SKILL.md +28 -51
- package/src/skf-drop-skill/customize.toml +44 -30
- package/src/skf-drop-skill/references/execute.md +79 -146
- package/src/skf-drop-skill/references/invocation-contract.md +64 -0
- package/src/skf-drop-skill/references/report.md +46 -30
- package/src/skf-drop-skill/references/select.md +118 -124
- package/src/skf-drop-skill/scripts/dir-sizes.py +54 -26
- package/src/skf-drop-skill/scripts/drop-roster.py +322 -0
- package/src/skf-export-skill/SKILL.md +40 -62
- package/src/skf-export-skill/assets/managed-section-format.md +14 -50
- package/src/skf-export-skill/assets/snippet-format.md +3 -1
- package/src/skf-export-skill/customize.toml +42 -13
- package/src/skf-export-skill/references/generate-snippet.md +54 -46
- package/src/skf-export-skill/references/health-check.md +2 -0
- package/src/skf-export-skill/references/invocation-contract.md +15 -0
- package/src/skf-export-skill/references/load-skill.md +88 -62
- package/src/skf-export-skill/references/manifest-rebuild.md +9 -10
- package/src/skf-export-skill/references/multi-skill-mode.md +4 -4
- package/src/skf-export-skill/references/orphan-context-detection.md +13 -5
- package/src/skf-export-skill/references/orphan-row-detection.md +31 -22
- package/src/skf-export-skill/references/package.md +15 -73
- package/src/skf-export-skill/references/preflight-snippet-root-probe.md +51 -70
- package/src/skf-export-skill/references/result-envelope.md +31 -16
- package/src/skf-export-skill/references/summary.md +41 -17
- package/src/skf-export-skill/references/token-report.md +4 -12
- package/src/skf-export-skill/references/update-context.md +118 -237
- package/src/skf-forger/SKILL.md +61 -37
- package/src/skf-forger/bmad-skill-manifest.yaml +4 -1
- package/src/skf-forger/customize.toml +43 -0
- package/src/skf-forger/references/pipeline-mode.md +73 -18
- package/src/skf-forger/scripts/forge-status.py +299 -0
- package/src/skf-forger/scripts/parse-pipeline.py +322 -54
- package/src/skf-forger/scripts/pipeline-gate.py +428 -0
- package/src/skf-forger/scripts/pipeline-journal.py +1052 -0
- package/src/skf-quick-skill/SKILL.md +54 -53
- package/src/skf-quick-skill/assets/skill-template.md +4 -2
- package/src/skf-quick-skill/customize.toml +51 -20
- package/src/skf-quick-skill/references/batch-mode.md +74 -21
- package/src/skf-quick-skill/references/compile.md +30 -31
- package/src/skf-quick-skill/references/ecosystem-check.md +7 -57
- package/src/skf-quick-skill/references/finalize.md +69 -18
- package/src/skf-quick-skill/references/halt-contract.md +41 -29
- package/src/skf-quick-skill/references/health-check.md +5 -3
- package/src/skf-quick-skill/references/quick-extract.md +114 -45
- package/src/skf-quick-skill/references/registry-resolution.md +9 -64
- package/src/skf-quick-skill/references/resolve-target.md +116 -66
- package/src/skf-quick-skill/references/write-and-validate.md +79 -40
- package/src/skf-refine-architecture/SKILL.md +43 -53
- package/src/skf-refine-architecture/customize.toml +49 -20
- package/src/skf-refine-architecture/references/compile.md +97 -54
- package/src/skf-refine-architecture/references/exit-codes.md +30 -0
- package/src/skf-refine-architecture/references/finding-storage.md +21 -0
- package/src/skf-refine-architecture/references/gap-analysis.md +101 -38
- package/src/skf-refine-architecture/references/improvements.md +14 -20
- package/src/skf-refine-architecture/references/init.md +98 -57
- package/src/skf-refine-architecture/references/issue-detection.md +32 -39
- package/src/skf-refine-architecture/references/refinement-rules.md +37 -48
- package/src/skf-refine-architecture/references/report.md +65 -54
- package/src/skf-refine-architecture/scripts/skf-check-preservation.py +1619 -0
- package/src/skf-rename-skill/SKILL.md +19 -54
- package/src/skf-rename-skill/customize.toml +32 -26
- package/src/skf-rename-skill/references/execute.md +230 -184
- package/src/skf-rename-skill/references/exit-codes.md +7 -7
- package/src/skf-rename-skill/references/health-check.md +1 -7
- package/src/skf-rename-skill/references/invocation-contract.md +25 -0
- package/src/skf-rename-skill/references/report.md +63 -52
- package/src/skf-rename-skill/references/select.md +197 -115
- package/src/skf-rename-skill/scripts/skf-validate-rename-name.py +161 -21
- package/src/skf-setup/SKILL.md +26 -26
- package/src/skf-setup/customize.toml +14 -10
- package/src/skf-setup/references/auto-index.md +50 -63
- package/src/skf-setup/references/ccc-index.md +29 -90
- package/src/skf-setup/references/detect-and-tier.md +58 -67
- package/src/skf-setup/references/health-check.md +2 -0
- package/src/skf-setup/references/invocation-contract.md +14 -0
- package/src/skf-setup/references/report.md +43 -166
- package/src/skf-setup/references/tier-rules.md +5 -5
- package/src/skf-setup/references/write-config.md +34 -57
- package/src/skf-test-skill/SKILL.md +17 -57
- package/src/skf-test-skill/assets/output-section-formats.md +33 -34
- package/src/skf-test-skill/customize.toml +27 -13
- package/src/skf-test-skill/references/coherence-check.md +134 -62
- package/src/skf-test-skill/references/coverage-check-tiers.md +35 -0
- package/src/skf-test-skill/references/coverage-check.md +217 -254
- package/src/skf-test-skill/references/detect-mode.md +5 -5
- package/src/skf-test-skill/references/external-validators.md +94 -82
- package/src/skf-test-skill/references/health-check.md +6 -7
- package/src/skf-test-skill/references/init.md +163 -88
- package/src/skf-test-skill/references/invocation-contract.md +66 -0
- package/src/skf-test-skill/references/migration-section-rules.md +5 -2
- package/src/skf-test-skill/references/report.md +162 -142
- package/src/skf-test-skill/references/score.md +80 -107
- package/src/skf-test-skill/references/scoring-rules.md +41 -155
- package/src/skf-test-skill/references/source-access-protocol.md +19 -92
- package/src/skf-test-skill/references/step-hard-gate.md +51 -33
- package/src/skf-test-skill/scripts/aggregate-coherence.py +218 -30
- package/src/skf-test-skill/scripts/build-result-context.py +287 -0
- package/src/skf-test-skill/scripts/check-metadata-coherence.py +37 -4
- package/src/skf-test-skill/scripts/combine-external-scores.py +23 -6
- package/src/skf-test-skill/scripts/compute-score.py +223 -45
- package/src/skf-test-skill/scripts/gap-ledger.py +1381 -0
- package/src/skf-test-skill/scripts/hard-gate.py +203 -0
- package/src/skf-test-skill/scripts/load-coverage-inputs.py +760 -0
- package/src/skf-test-skill/scripts/locate-export-segments.py +835 -0
- package/src/skf-test-skill/scripts/reconcile-coverage.py +273 -79
- package/src/skf-test-skill/scripts/score-signatures.py +459 -0
- package/src/skf-test-skill/scripts/stage-helper-payload.py +148 -0
- package/src/skf-test-skill/scripts/validate-inventory.py +311 -16
- package/src/skf-test-skill/scripts/verify-declared-numerator.py +127 -52
- package/src/skf-test-skill/templates/test-report-template.md +21 -15
- package/src/skf-update-skill/SKILL.md +38 -25
- package/src/skf-update-skill/customize.toml +27 -12
- package/src/skf-update-skill/references/detect-changes.md +233 -222
- package/src/skf-update-skill/references/gap-driven.md +336 -0
- package/src/skf-update-skill/references/health-check.md +31 -5
- package/src/skf-update-skill/references/init.md +250 -84
- package/src/skf-update-skill/references/invocation-contract.md +12 -0
- package/src/skf-update-skill/references/manual-section-rules.md +6 -23
- package/src/skf-update-skill/references/merge.md +180 -47
- package/src/skf-update-skill/references/re-extract.md +53 -262
- package/src/skf-update-skill/references/report.md +100 -35
- package/src/skf-update-skill/references/write.md +331 -149
- package/src/skf-update-skill/scripts/skf-new-file-diff.py +29 -7
- package/src/skf-verify-stack/SKILL.md +23 -42
- package/src/skf-verify-stack/assets/feasibility-report-template.md +11 -5
- package/src/skf-verify-stack/customize.toml +55 -22
- package/src/skf-verify-stack/references/coverage-patterns.md +17 -8
- package/src/skf-verify-stack/references/coverage.md +67 -64
- package/src/skf-verify-stack/references/exit-codes.md +21 -8
- package/src/skf-verify-stack/references/init.md +107 -62
- package/src/skf-verify-stack/references/integration-verification-rules.md +12 -13
- package/src/skf-verify-stack/references/integrations.md +106 -65
- package/src/skf-verify-stack/references/report.md +59 -42
- package/src/skf-verify-stack/references/requirements.md +32 -9
- package/src/skf-verify-stack/references/synthesize.md +53 -39
- package/src/skf-verify-stack/scripts/skf-coverage-tally.py +398 -52
- package/src/skf-verify-stack/scripts/skf-previous-report.py +330 -0
- package/src/skf-verify-stack/scripts/skf-report-delta.py +453 -49
- package/src/skf-verify-stack/scripts/skf-verdict-rollup.py +240 -27
- package/tools/cli/commands/install.js +11 -4
- package/tools/cli/commands/status.js +15 -3
- package/tools/cli/commands/update.js +11 -4
- package/tools/cli/lib/tool-check.js +320 -0
- package/tools/cli/lib/version-check.js +22 -9
- package/tools/cli/skf-cli.js +4 -1
- package/.gitattributes +0 -16
- package/.nvmrc +0 -1
- package/docs/_internal/RELEASING.md +0 -679
- package/release-audits/v1.0.0-launch-audit.md +0 -728
- package/src/skf-audit-skill/references/step-doc-drift.md +0 -177
- package/src/skf-brief-skill/references/headless-source-authority-detection.md +0 -26
- package/src/skf-campaign/manifest.yaml +0 -15
- package/src/skf-create-skill/assets/tessl-dismissal-rules.md +0 -106
- package/src/skf-create-skill/references/ecosystem-check.md +0 -74
- package/src/skf-drop-skill/references/headless-contract.md +0 -18
- package/src/skf-update-skill/references/merge-conflict-rules.md +0 -15
- package/src/skf-update-skill/references/remote-source-resolution.md +0 -98
- package/src/skf-update-skill/references/validate.md +0 -124
- package/tools/validate-docs-drift.js +0 -233
- package/tools/validate-docs-links.js +0 -665
|
@@ -1,11 +1,30 @@
|
|
|
1
1
|
---
|
|
2
2
|
nextStepFile: 'coherence-check.md'
|
|
3
|
-
outputFile: '{
|
|
4
|
-
scoringRulesFile: '
|
|
3
|
+
outputFile: '{report_file}'
|
|
4
|
+
scoringRulesFile: 'references/scoring-rules.md'
|
|
5
5
|
sourceAccessProtocol: 'references/source-access-protocol.md'
|
|
6
|
+
coverageTiersFile: 'references/coverage-check-tiers.md'
|
|
7
|
+
validateInventoryScript: 'scripts/validate-inventory.py'
|
|
8
|
+
coverageInputsScript: 'scripts/load-coverage-inputs.py'
|
|
9
|
+
scoreSignaturesScript: 'scripts/score-signatures.py'
|
|
6
10
|
reconcileScript: 'scripts/reconcile-coverage.py'
|
|
7
11
|
coherenceScript: 'scripts/check-metadata-coherence.py'
|
|
8
12
|
numeratorVerifyScript: 'scripts/verify-declared-numerator.py'
|
|
13
|
+
stageHelperPayloadScript: 'scripts/stage-helper-payload.py'
|
|
14
|
+
outputFormatsFile: 'assets/output-section-formats.md'
|
|
15
|
+
ledgerFile: '{forge_version}/test-findings-{run_id}.json'
|
|
16
|
+
gapLedgerScript: 'scripts/gap-ledger.py'
|
|
17
|
+
# Each `<name>ProbeOrder` resolves `{<name>Helper}` to its first existing path:
|
|
18
|
+
# the installed SKF module first, the src/ dev checkout second.
|
|
19
|
+
verifyProvenanceCompletenessProbeOrder:
|
|
20
|
+
- '{project-root}/_bmad/skf/shared/scripts/skf-verify-provenance-completeness.py'
|
|
21
|
+
- '{project-root}/src/shared/scripts/skf-verify-provenance-completeness.py'
|
|
22
|
+
extractPublicApiProbeOrder:
|
|
23
|
+
- '{project-root}/_bmad/skf/shared/scripts/skf-extract-public-api.py'
|
|
24
|
+
- '{project-root}/src/shared/scripts/skf-extract-public-api.py'
|
|
25
|
+
detectWorkspacesProbeOrder:
|
|
26
|
+
- '{project-root}/_bmad/skf/shared/scripts/skf-detect-workspaces.py'
|
|
27
|
+
- '{project-root}/src/shared/scripts/skf-detect-workspaces.py'
|
|
9
28
|
---
|
|
10
29
|
|
|
11
30
|
<!-- Config: communicate in {communication_language}. -->
|
|
@@ -16,362 +35,280 @@ numeratorVerifyScript: 'scripts/verify-declared-numerator.py'
|
|
|
16
35
|
|
|
17
36
|
Compare the exports, functions, classes, types, and interfaces documented in SKILL.md against the actual source code API surface. Identify missing documentation, undocumented exports, and signature mismatches. Analysis depth scales with forge tier.
|
|
18
37
|
|
|
38
|
+
**Halt envelope.** Every HALT in this step names its `halt_reason` and phase and carries exit code 1. It releases the run lock first, whatever the release prints: from `{project-root}`, run `uv run {runLockHelper} release --lock "{forge_version}/.test-skill.lock" --owner "{run_owner}"` (SKILL.md Workflow Rules). In headless mode it then writes `{"phase": "<phase>", "reason": "<the halt message>", "halt_reason": "<halt_reason>", "skill_name": "{skill_name}", "report_path": "{report_file}"}`, adding `"path"` when the halt names one, to `{run_dir}/halt.json` and runs:
|
|
39
|
+
|
|
40
|
+
```bash
|
|
41
|
+
uv run {emitEnvelopeHelper} emit-halt --workflow skf-test-skill --run-dir "{run_dir}" --target stderr < "{run_dir}/halt.json"
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
Display the line it prints, as the run's last line, then stop. If the emitter exits non-zero or prints no line, display the halt message alone. An interactive HALT displays its message and emits nothing.
|
|
45
|
+
|
|
46
|
+
**Run files.** Every script below writes its result into the run folder `{run_dir}` that init.md §6b created, and the next one reads it from there by path. A subagent's response is saved there with the Write tool exactly as it came back, fence and all, and passed by path, so a quote or an apostrophe in it never meets a shell. Every count, name list and score a script computes is read from the file it wrote, never re-derived by hand or copied from one command into another. Each command runs from `{project-root}`; a `{...Script}` path resolves relative to the skill root.
|
|
47
|
+
|
|
19
48
|
### 0. Check for Docs-Only Mode
|
|
20
49
|
|
|
21
|
-
|
|
50
|
+
A skill is docs-only when every citation it writes is `[EXT:...]`, with no local source citation (`[AST:...]`, `[SRC:...]`, or a stack's `[from skill: ...]`). The citation census decides it:
|
|
22
51
|
|
|
23
|
-
|
|
52
|
+
```bash
|
|
53
|
+
uv run {coverageInputsScript} census --skill-dir "{resolved_skill_package}" --output "{run_dir}/census.json"
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
Bind `{docs_only_mode}` ← `docsOnly`. Exit 1 printed its error on stderr: HALT with it (`halt_reason: "helper-failed"`, phase `coverage-check:census`). §0b sets `{docs_only_mode}` too when source access reaches State 5.
|
|
24
57
|
|
|
25
|
-
**
|
|
58
|
+
**A docs-only run** measures documentation completeness as its Export Coverage, with `analysis_confidence` `docs-only`. It runs only §1, §1a and §1b, the docs-only guard of §2b, §2c's `docsOnly` branch, §4's Export Coverage line and its `Denominator: docs-only completeness` annotation, §5, §5b and §6. Step 5 passes `docsOnly: true` to the scoring script, which skips Signature Accuracy and Type Coverage at any tier.
|
|
26
59
|
|
|
27
60
|
"**Docs-only skill detected.** Coverage check evaluates documentation completeness rather than source code coverage."
|
|
28
61
|
|
|
29
|
-
|
|
62
|
+
A source-based skill continues with §0b.
|
|
30
63
|
|
|
31
64
|
### 0b. Load Source Access Protocol
|
|
32
65
|
|
|
33
|
-
Load `{sourceAccessProtocol}` and follow both sections:
|
|
34
|
-
1. **Source API Surface Definition** — determines what counts as the public API for coverage denominator
|
|
35
|
-
2. **Source Access Resolution** — 5-state waterfall to determine how source files will be read and sets `analysis_confidence`
|
|
66
|
+
Load `{sourceAccessProtocol}` and follow both sections: **Source API Surface Definition** (the clause that decides the coverage denominator, picked in §2b) and **Source Access Resolution** (the 5-state waterfall that decides how source files are read and sets `analysis_confidence`; State 5 makes the run docs-only).
|
|
36
67
|
|
|
37
|
-
|
|
68
|
+
**Workspace layout.** The Source API Surface Definition's monorepo tests read `{source_is_monorepo}`, which `{detectWorkspacesHelper}` decides. At State 1 (local source), resolve `{detectWorkspacesHelper}` ← first existing path in `{detectWorkspacesProbeOrder}` and run (`{stageHelperPayloadScript}` reads the source tree and the root manifests from disk and pipes them to the helper):
|
|
69
|
+
|
|
70
|
+
```bash
|
|
71
|
+
uv run {stageHelperPayloadScript} detect-workspaces --source-root "{source_path}" | uv run {detectWorkspacesHelper}
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
Bind `{source_is_monorepo}` ← `is_monorepo` and `{source_workspace_kind}` ← `manifest_kind`. At States 2 to 5, or when the helper does not resolve or the command exits non-zero, the layout is unknown: bind `{source_is_monorepo}` ← false. §5 records the layout.
|
|
38
75
|
|
|
39
|
-
|
|
76
|
+
### 1. Extract Documented Exports from SKILL.md
|
|
40
77
|
|
|
41
|
-
Delegate reading of the skill under test to a subagent
|
|
42
|
-
1. Read SKILL.md
|
|
43
|
-
2. If a `references/` directory exists alongside SKILL.md and SKILL.md's `## Full` headings are absent or stubs, also read all `references/*.md` files
|
|
44
|
-
3. only return this compact JSON inventory — no prose, no extra commentary:
|
|
78
|
+
Delegate the reading of the skill under test to a subagent, given the path to SKILL.md and to its `references/` directory when one exists. The subagent reads SKILL.md, and every `references/*.md` file as well when SKILL.md's `## Full` headings are absent or stubs. For each export documented both in the SKILL.md body and in a reference file, it compares the two: parameters (name, type, order, optionality), the return type, and descriptions that contradict (brief against detailed is fine). It returns only this JSON, with no prose and no commentary:
|
|
45
79
|
|
|
46
80
|
```json
|
|
47
81
|
{
|
|
48
82
|
"exports": [
|
|
49
83
|
{"name": "functionName", "kind": "function", "params": "...", "return_type": "...", "description": "..."},
|
|
50
|
-
{"name": "ClassName", "kind": "class", "methods": ["..."], "properties": ["..."]},
|
|
51
|
-
{"name": "TypeName", "kind": "type", "fields": ["..."]},
|
|
52
|
-
{"name": "
|
|
53
|
-
{"name": "useHook", "kind": "hook", "usage_signature": "..."}
|
|
84
|
+
{"name": "ClassName", "kind": "class", "methods": ["..."], "properties": ["..."], "description": "..."},
|
|
85
|
+
{"name": "TypeName", "kind": "type", "fields": ["..."], "description": "..."},
|
|
86
|
+
{"name": "useHook", "kind": "hook", "usage_signature": "...", "description": "..."}
|
|
54
87
|
],
|
|
55
|
-
"
|
|
56
|
-
"references": ["references/api-reference.md", "references/type-definitions.md"],
|
|
88
|
+
"references": ["references/api-reference.md"],
|
|
57
89
|
"cross_check_mismatches": [
|
|
58
|
-
{
|
|
59
|
-
"export": "functionName",
|
|
60
|
-
"skill_md_line": 42,
|
|
61
|
-
"reference_file": "references/api-reference.md",
|
|
62
|
-
"reference_line": 18,
|
|
63
|
-
"issue": "description of the signature mismatch"
|
|
64
|
-
}
|
|
90
|
+
{"export": "functionName", "skill_md_line": 42, "reference_file": "references/api-reference.md", "reference_line": 18, "issue": "SKILL.md shows (date: Date) => string, the reference (date: Date, format?: string) => string"}
|
|
65
91
|
]
|
|
66
92
|
}
|
|
67
93
|
```
|
|
68
94
|
|
|
69
|
-
|
|
95
|
+
Tell the subagent the contract the validator checks: each `kind` is one of `function`, `class`, `type`, `constant`, `hook`, `interface`, `method`, `struct`, `enum`, `trait`, `macro` or `adapter`; every entry carries the `description` the skill gives it (an empty string when it gives none); a function or method carries `params` (`""` or `[]` when it takes none) and `return_type` as the skill documents them, and leaves a key out when the skill documents nothing for it.
|
|
70
96
|
|
|
71
|
-
|
|
97
|
+
Save the subagent's response, as it came back, to `{run_dir}/inventory-response.txt` with the Write tool. The documented inventory is the one §1a validates: do not load SKILL.md or the reference files into the parent context. Without subagents, the parent reads the same files itself, builds the same JSON and saves it to the same file; §1a's checks still run on it.
|
|
72
98
|
|
|
73
99
|
#### 1a. Parent-Side Schema Validation + Spot-Check
|
|
74
100
|
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
**Schema validation (required keys + types) — delegated to `scripts/validate-inventory.py`.** Fence-stripping, JSON parsing, and the required-keys / per-entry-type / `kind`-enum / mismatch-field contract are structural validation with one correct verdict per input, so they run in the script, not in-prompt. Pipe the subagent's **raw** response (fence and all) to it exactly as §2c pipes the reconcile input:
|
|
101
|
+
Before anything consumes the inventory, `{validateInventoryScript}` checks it against the §1 contract (a wrapping fence stripped) and spot-checks sampled names against SKILL.md and the reference files the subagent listed. The look-up is the script's, never a grep: an export named `$state` or `a.b` is neither a shell variable nor a regex.
|
|
78
102
|
|
|
79
103
|
```bash
|
|
80
|
-
|
|
104
|
+
uv run {validateInventoryScript} --input "{run_dir}/inventory-response.txt" --skill-package "{resolved_skill_package}" --output "{run_dir}/inventory.json"
|
|
81
105
|
```
|
|
82
106
|
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
-
|
|
86
|
-
- Required keys present with correct types: `exports` (list), `cross_check_mismatches` (list — may be empty). Note: the parent already knows the skill name from workflow context (`{resolved_skill_package}` from step 1) — the subagent is not required to echo it back, and doing so introduces a contract-drift surface without improving verification.
|
|
87
|
-
- Each `exports[]` entry is a dict with at minimum `name` (non-empty string) and `kind` (one of `function|class|type|constant|hook|interface|method|struct|enum|trait|macro|adapter`). The enum spans the constructs SKF actually documents across languages and skill types: JS/TS (`function`/`class`/`type`/`constant`/`hook`/`interface`/`method`), Rust public-API items (`struct`/`enum`/`trait`/`macro` — alongside the shared `type`/`constant`/`function`), and stack-composition scaffolds (`adapter`). Malformed entries are counted in `rejectedCount`.
|
|
88
|
-
- Each non-empty `cross_check_mismatches[]` entry carries `export`, `skill_md_line`, `reference_file`, `reference_line`, `issue`.
|
|
107
|
+
- `valid` true (exit 0): every later command reads `{run_dir}/inventory.json`.
|
|
108
|
+
- `valid` false (exit 2): re-dispatch the §1 subagent once, with the script's `violations[]` appended to its instructions (an absent name as a name the skill does not write), save the new response over the old one and run the command again. Invalid a second time: HALT (`halt_reason: "inventory-invalid"`, phase `coverage-check:inventory`): "coverage-check: the subagent inventory failed {schema validation | the ground-truth spot-check} twice: {the violations joined}."
|
|
109
|
+
- Exit 1 (no input, a file that cannot be read, or a package with no readable SKILL.md): correct the file and run it again.
|
|
89
110
|
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
**Spot-check (ground-truth verification, zero-hallucination guard):** operate on the validated `inventory` from the script.
|
|
93
|
-
|
|
94
|
-
1. If `inventory.exports` is empty (`exportsCount == 0`): skip the spot-check (no names to verify). Zero-exports policy is handled in the §2b zero-exports guard.
|
|
95
|
-
2. Otherwise, sample `min(3, exportsCount)` exports deterministically — by default take indices `[0, len//2, len-1]` (first, middle, last) from `inventory.exports` after a stable sort by `name`.
|
|
96
|
-
3. For each sampled export, grep for the name across SKILL.md **and every reference file the subagent listed in `inventory.references`** (the documented surface of a split-body skill spans both): `grep -n "{export.name}" {resolved_skill_package}/SKILL.md {resolved_skill_package}/{each references[] path}` in the parent context. The name must appear at least once somewhere in that file set. Greping SKILL.md alone would false-HALT a split-body skill whose sampled export is documented only in a `references/*.md` file (a legitimate placement per §1 step 2 and the split-body note below).
|
|
97
|
-
4. If a sampled name returns zero matches across SKILL.md **and** all listed reference files, HALT "coverage-check: subagent inventory failed ground-truth spot-check — `{name}` claimed as export but absent from SKILL.md and the listed reference files".
|
|
98
|
-
|
|
99
|
-
These checks catch two hallucination classes: schema-shape drift (subagent paraphrased or dropped the contract) and fabricated exports (subagent invented names not in the document). Both are disqualifying for a grader skill — do not downgrade to a warning.
|
|
111
|
+
### 1b. Cross-Check Split-Body Consistency
|
|
100
112
|
|
|
101
|
-
|
|
113
|
+
Only when the validated inventory's `references` is non-empty. The SKILL.md body is authoritative: each `cross_check_mismatches` entry is a High `split-body-mismatch` gap. §5b records it titled `Split-body mismatch: {export}`, with its `export` and its Source at the reference file's line (`{reference_file}:{reference_line}`), inside the skill package: the reference file is the one to update, and update-skill repairs it without reading the source.
|
|
102
114
|
|
|
103
|
-
###
|
|
115
|
+
### 2. Analyze Source Code (Tier-Dependent)
|
|
104
116
|
|
|
105
|
-
|
|
117
|
+
Identify the public API surface from the package entry points (see 0b), at the depth the tier allows. Whichever branch runs, `{coverageInputsScript} surface` writes it to `{run_dir}/surface.json`, which §2b, §2c, §4 and step 5 read. Names read by eye reach it as per-file results, `{"file": "<path>", "exports_found": [<each name>], "signature_mismatches": []}` for each file, saved to `{run_dir}/per-file-<n>.json` with the Write tool and given to `surface` with one `--per-file` each.
|
|
106
118
|
|
|
107
|
-
|
|
119
|
+
**Exits.** Every `{coverageInputsScript}` and `{scoreSignaturesScript}` command in §2 and §2b exits 0, or exits 1 with its error on stderr: HALT with it (`halt_reason: "helper-failed"`, phase `coverage-check:surface`). `surface` with a `--per-file`, and `score`, also exit 2 when a subagent response breaks the schema: re-dispatch that subagent once with its `violations[]` appended (or correct a result the main thread wrote), save the new response over the old one and run the command again; a second failure HALTs the same way.
|
|
108
120
|
|
|
109
|
-
|
|
110
|
-
- **Return types:** exact type match
|
|
111
|
-
- **Description:** no contradictions (brief vs detailed is acceptable; conflicting semantics is not)
|
|
121
|
+
**Quick Tier (no AST tools):** load `{coverageTiersFile}` and follow its **Quick Tier** section. Signatures cannot be verified at this tier: note them as "unverified" in the report.
|
|
112
122
|
|
|
113
|
-
**
|
|
123
|
+
**Forge Tier (ast-grep available):**
|
|
114
124
|
|
|
115
|
-
|
|
125
|
+
At State 1, run the recipe runner once over the source. Resolve `{extractPublicApiHelper}` ← first existing path in `{extractPublicApiProbeOrder}` and run it (leave `--brief` out when the skill has no brief; `--head-cap 0` keeps every match, so no export is cut and read as missing):
|
|
116
126
|
|
|
117
|
-
```
|
|
118
|
-
{
|
|
119
|
-
"cross_check_mismatches": [
|
|
120
|
-
{
|
|
121
|
-
"export": "formatDate",
|
|
122
|
-
"skill_md_line": 42,
|
|
123
|
-
"reference_file": "references/api-reference.md",
|
|
124
|
-
"reference_line": 18,
|
|
125
|
-
"issue": "SKILL.md shows (date: Date) => string, reference shows (date: Date, format?: string) => string"
|
|
126
|
-
}
|
|
127
|
-
],
|
|
128
|
-
"exports_cross_checked": 12,
|
|
129
|
-
"mismatches_found": 1
|
|
130
|
-
}
|
|
127
|
+
```bash
|
|
128
|
+
uv run {extractPublicApiHelper} --mode full --source-root "{source_path}" --brief "{forge_data_folder}/{skill_name}/skill-brief.yaml" --tier {detected_tier} --head-cap 0 --output "{run_dir}/extract-full.json"
|
|
131
129
|
```
|
|
132
130
|
|
|
133
|
-
|
|
131
|
+
Build the surface from it, with the metadata and, when init.md §2 bound one, the provenance map (the baselines of the §2b candidates and guards; they add no name):
|
|
134
132
|
|
|
135
|
-
|
|
133
|
+
```bash
|
|
134
|
+
uv run {coverageInputsScript} surface --extraction "{run_dir}/extract-full.json" --metadata "{resolved_skill_package}/metadata.json" [--provenance "{forge_provenance_map}"] --output "{run_dir}/surface.json"
|
|
135
|
+
```
|
|
136
136
|
|
|
137
|
-
|
|
137
|
+
The surface holds the names the entry points export, less those defined in a file the brief scopes out (`excluded.outsideScope`, listed in the report); the recipes' internal matches are never part of it. Decide what to do from the JSON, not from the exit code:
|
|
138
138
|
|
|
139
|
-
|
|
140
|
-
-
|
|
141
|
-
-
|
|
142
|
-
- Compare against documented inventory by name matching
|
|
143
|
-
- Cannot verify signatures — note as "unverified" in report
|
|
139
|
+
- `extraction.fallback.needed` is true, the runner exited 2, or it printed no JSON: load `{coverageTiersFile}` and run its **Fallback Per-File Scan** in place of the rest of this tier.
|
|
140
|
+
- `extraction.status` is `incomplete` (exit 1): keep the surface, read by eye only the files `extraction.readByEye` names, save a per-file result for each, and run `surface` again with them.
|
|
141
|
+
- `extractionGaps[]` names exports no recipe found: they are on the surface, and their signature is read by eye at their `file` and `line`.
|
|
144
142
|
|
|
145
|
-
**
|
|
143
|
+
**Compare the signatures.** A script lists the comparisons; subagents judge them:
|
|
146
144
|
|
|
147
|
-
|
|
145
|
+
```bash
|
|
146
|
+
uv run {scoreSignaturesScript} plan --inventory "{run_dir}/inventory.json" --surface "{run_dir}/surface.json" --output "{run_dir}/signature-plan.json"
|
|
147
|
+
```
|
|
148
148
|
|
|
149
|
-
For
|
|
150
|
-
1. Uses ast-grep to extract all exported symbols with their full signatures (the `source_sig`)
|
|
151
|
-
2. Matches each export against the `documented_signatures` map supplied by the parent, comparing params (name, type, order, optionality) and return type
|
|
152
|
-
3. Returns only the JSON object below — no prose, no commentary, no markdown fences:
|
|
149
|
+
For each `files[]` entry, delegate one subagent with its checks. For each check it reads the full declaration at `{file}:{line}` (all of it: `signatureLine` is one physical line, and a multi-line signature cut there would read as a mismatch), compares it with the check's `documented` signature (params: name, type, order, optionality; and the return type), and returns only this JSON, with no prose and no markdown fences:
|
|
153
150
|
|
|
154
151
|
```json
|
|
155
|
-
{
|
|
156
|
-
"file": "src/utils.ts",
|
|
157
|
-
"exports_found": ["formatDate", "parseConfig", "ConfigType"],
|
|
158
|
-
"exports_documented": ["formatDate", "parseConfig"],
|
|
159
|
-
"missing_docs": ["ConfigType"],
|
|
160
|
-
"signature_mismatches": [
|
|
161
|
-
{
|
|
162
|
-
"name": "formatDate",
|
|
163
|
-
"source_sig": "(date: Date, format?: string) => string",
|
|
164
|
-
"documented_sig": "(date: Date) => string",
|
|
165
|
-
"issue": "missing optional parameter 'format'"
|
|
166
|
-
}
|
|
167
|
-
]
|
|
168
|
-
}
|
|
152
|
+
{"file": "src/utils.ts", "signature_mismatches": [{"name": "formatDate", "line": 42, "source_sig": "(date: Date, format?: string) => string", "documented_sig": "(date: Date) => string", "issue": "missing optional parameter 'format'"}]}
|
|
169
153
|
```
|
|
170
154
|
|
|
171
|
-
|
|
155
|
+
Save each response to `{run_dir}/signatures-<n>.txt` with the Write tool. Without subagents, compare in the main thread and save the same JSON.
|
|
172
156
|
|
|
173
|
-
**Deep Tier (ast-grep + gh + QMD):**
|
|
174
|
-
- All Forge tier checks, plus:
|
|
175
|
-
- Use gh CLI to verify source repository matches documented version
|
|
176
|
-
- Cross-check type definitions against their source declarations
|
|
177
|
-
- Verify re-exported symbols trace to their original source
|
|
157
|
+
**Deep Tier (ast-grep + gh + QMD):** all Forge tier checks, plus: verify with the gh CLI that the source repository matches the documented version, cross-check type definitions against their source declarations, and trace re-exported symbols to their original source.
|
|
178
158
|
|
|
179
|
-
|
|
159
|
+
**States 2 to 4 (no local source):** the surface is what the skill recorded, so there is no signature to score:
|
|
160
|
+
- **State 2** (`{forge_provenance_map}`, the provenance map init.md §2 bound): the union of its named exports and the `metadata.json` `exports[]` names, with `--fold`, `--keep` and `--fold-prefix` as the Source API Surface Definition's canonicalization says:
|
|
180
161
|
|
|
181
|
-
|
|
162
|
+
```bash
|
|
163
|
+
uv run {coverageInputsScript} surface --provenance "{forge_provenance_map}" --metadata "{resolved_skill_package}/metadata.json" [--brief "{forge_data_folder}/{skill_name}/skill-brief.yaml"] [--fold] [--keep <variant>] [--fold-prefix <prefix>] --output "{run_dir}/surface.json"
|
|
164
|
+
```
|
|
182
165
|
|
|
183
|
-
|
|
166
|
+
Step 5's scoring script reads its `state2` counts, and the report records its fold summary, `canonical.summary`.
|
|
167
|
+
- **State 3**: the `metadata.json` `exports[]` names: `surface --metadata "{resolved_skill_package}/metadata.json" --output "{run_dir}/surface.json"`.
|
|
168
|
+
- **State 4**: the names read remotely by eye, as per-file results: `surface --per-file "{run_dir}/per-file-<n>.json" [--brief "{forge_data_folder}/{skill_name}/skill-brief.yaml"] --metadata "{resolved_skill_package}/metadata.json" --output "{run_dir}/surface.json"`.
|
|
184
169
|
|
|
185
|
-
|
|
186
|
-
2. Otherwise the composition surface from `metadata.json`: `len(libraries) + len(integration_pairs)`.
|
|
170
|
+
### 2b. Resolve the Denominator and Guard Zero Exports
|
|
187
171
|
|
|
188
|
-
|
|
172
|
+
Load the metadata and provenance-map counts once, for §2c, §4b and the numerator check below (pass `--provenance` when init.md §2 bound `{forge_provenance_map}`):
|
|
189
173
|
|
|
190
|
-
|
|
174
|
+
```bash
|
|
175
|
+
uv run {coverageInputsScript} metadata --metadata "{resolved_skill_package}/metadata.json" [--provenance "{forge_provenance_map}"] --output "{run_dir}/coverage-inputs.json"
|
|
176
|
+
```
|
|
177
|
+
|
|
178
|
+
**Pick the denominator.** Which clause of the Source API Surface Definition (§0b) matches this skill is a judgment; the counts each clause names are in `surface.json` `candidates` and `guards`. The clause gives §2c its branch:
|
|
191
179
|
|
|
192
|
-
**
|
|
180
|
+
- **Standard barrel**, **multi-entry**, **specific-modules**, **pattern-reference**, State 2 or a priority-2/3 **stratified-scope** denominator: the barrel branch, over the `surface.json` set the clause names (`all`, `subpaths`, `tier_a_include` or `scope.include`). The scope sets need the brief's globs and names that carry their files (State 3's metadata names carry none), and `subpaths` and `root` an extraction. When the set is absent, the denominator is the `all` set, and the `Denominator:` line (§4) adds `({set} set absent: all set used)`.
|
|
181
|
+
- **Stratified-scope priority 1**: the scalar branch over `stats.effective_denominator` (`candidates.statsEffectiveDenominator`), unless `guards.deflation.fires`: then the barrel branch over the `scope.include` set (`all` when the brief has no `scope.include`).
|
|
182
|
+
- **A stack skill** (`metadata.json.skill_type == "stack"`): the stack branch, over `coverage-inputs.json` `stack.denominator`. Its own barrel is empty by design, and its `[from skill: …]` citations never make it docs-only. **If `stack.denominator` is 0**: HALT (`halt_reason: "indeterminate-surface"`, phase `coverage-check:denominator`) with `Error: stack composition surface empty: {skill_name} cites no contracts, libraries, or integration pairs, so Export Coverage is undefined. Verify the stack was compiled from at least one constituent skill.` Do not write the Coverage Analysis section; this is an indeterminate state, not a FAIL.
|
|
183
|
+
- **A docs-only skill**: the `docsOnly` branch, over the inventory.
|
|
193
184
|
|
|
185
|
+
**Numerator ground truth.** A declared count equal to the denominator (`stats.exports_documented == stats.effective_denominator`, the inflation signature) may be padded to a tautological 100%. Run the check on every source-based skill; without the signature (a stack and a reference app carry none) it reports `skipped`:
|
|
186
|
+
|
|
187
|
+
```bash
|
|
188
|
+
uv run {numeratorVerifyScript} --inputs "{run_dir}/coverage-inputs.json" --skill-dir "{resolved_skill_package}" --output "{run_dir}/numerator.json"
|
|
194
189
|
```
|
|
195
|
-
Error: indeterminate API surface — 0 exports discovered in source for {skill_name}.
|
|
196
190
|
|
|
197
|
-
|
|
198
|
-
Export Coverage is undefined (division by zero) and downstream scoring
|
|
199
|
-
would yield a vacuous PASS.
|
|
191
|
+
§2c's scalar branch takes its `verified` count as the numerator when `inflated` is true, and §4b records the gap. Exit 1 or 2: HALT with its message (`halt_reason: "helper-failed"`, phase `coverage-check:numerator`).
|
|
200
192
|
|
|
201
|
-
|
|
193
|
+
**If the chosen denominator is 0 AND `docs_only_mode == false` AND `metadata.json.skill_type != "stack"`:** HALT (`halt_reason: "indeterminate-surface"`, phase `coverage-check:denominator`) with:
|
|
194
|
+
|
|
195
|
+
```
|
|
196
|
+
Error: indeterminate API surface: 0 exports discovered in source for {skill_name}.
|
|
197
|
+
|
|
198
|
+
Export Coverage is undefined, and scoring would yield a vacuous PASS. Fix one of:
|
|
202
199
|
- Set `scope.include` in the brief to point at the package's entry point(s)
|
|
203
200
|
- Add `[EXT:]` citations if this is actually a docs-only skill
|
|
204
201
|
- Verify the skill's source_path / source_ref resolve to the intended tree
|
|
205
202
|
```
|
|
206
203
|
|
|
207
|
-
Do not write the Coverage Analysis section
|
|
208
|
-
|
|
209
|
-
**If `docs_only_mode == true` and the documented inventory is empty:** HALT with the analogous docs-only message ("docs-only skill declares zero items — no API surface to test").
|
|
210
|
-
|
|
211
|
-
### 2c. Reconcile Documented vs Source Surface (Deterministic Intersection)
|
|
204
|
+
Do not write the Coverage Analysis section or attach a score: this is an indeterminate state, not a FAIL. **If `docs_only_mode == true` and the validated inventory documents no item:** HALT the same way with "docs-only skill declares zero items: no API surface to test".
|
|
212
205
|
|
|
213
|
-
|
|
206
|
+
**Score the signatures** (Forge, Forge+ and Deep tier at State 1, once §2 compared them). Give `{scoreSignaturesScript}` the set the denominator counts as `--surface-set`: the set §2c's barrel branch reads, or for the scalar branch `tier_a_include` when `surface.json` has it, else `scope.include`, else `all`. Type Coverage then counts that set's types alone, so a type the clause leaves out (a stratified skill's deferred Tier B) lowers neither Export Coverage nor Type Coverage. Give it one `--results` per saved response (the fallback scan's `per-file-<n>.txt`, else the `signatures-<n>.txt`):
|
|
214
207
|
|
|
215
|
-
|
|
216
|
-
|
|
217
|
-
|
|
208
|
+
```bash
|
|
209
|
+
uv run {scoreSignaturesScript} score --inventory "{run_dir}/inventory.json" --surface "{run_dir}/surface.json" --surface-set <set> --results "{run_dir}/signatures-<n>.txt" --output "{run_dir}/signatures.json" --gaps-output "{run_dir}/signature-gaps.json"
|
|
210
|
+
```
|
|
218
211
|
|
|
219
|
-
|
|
212
|
+
It writes each mismatch of a compared name to `signature-gaps.json` as a Critical `signature-mismatch` gap titled `Signature mismatch: {name}`, with `export` `{name}`, its Source at `{file}:{line}` and both signatures in its Issue; §5b appends that file to the ledger. Its `warnings` (a mismatch for a name never compared, a category with nothing to compare) go in the Coverage Analysis section.
|
|
220
213
|
|
|
221
|
-
|
|
214
|
+
### 2c. Reconcile Documented vs Source Surface (Deterministic Intersection)
|
|
222
215
|
|
|
223
|
-
|
|
216
|
+
`{reconcileScript}` reconciles the §1 inventory with the source surface and writes `{run_dir}/coverage.json`, whose `exportCoverage` is the Export Coverage score step 5 hands to the scoring script. Run the branch §2b picked:
|
|
224
217
|
|
|
225
|
-
|
|
226
|
-
echo '<JSON>' | uv run {reconcileScript} --stdin
|
|
227
|
-
```
|
|
218
|
+
1. **Barrel**, with the name set §2b picked as `--surface-set`:
|
|
228
219
|
|
|
229
|
-
|
|
220
|
+
```bash
|
|
221
|
+
uv run {reconcileScript} --denominator-source barrel --inventory "{run_dir}/inventory.json" --surface "{run_dir}/surface.json" --surface-set <set> --output "{run_dir}/coverage.json"
|
|
222
|
+
```
|
|
230
223
|
|
|
231
|
-
|
|
232
|
-
{
|
|
233
|
-
"denominatorSource": "barrel | scalar | stack",
|
|
234
|
-
"exports": [ /* §1 inventory exports[] — used for barrel + scalar; kind:"method" entries are excluded automatically */ ],
|
|
235
|
-
"barrelSet": [ /* barrel: resolved enumerated name set, when §4 supplies one */ ],
|
|
236
|
-
"perFileResults": [ /* barrel: the §2 per-file results, unioned into barrel_set when no barrelSet */ ],
|
|
237
|
-
"denominatorValue": 0,
|
|
238
|
-
"compositionNames": [ /* stack: composition-surface names to grep */ ],
|
|
239
|
-
"skillPackagePath": "{resolved_skill_package}"
|
|
240
|
-
}
|
|
241
|
-
```
|
|
224
|
+
2. **Scalar**, over the `coverage-inputs.json` scalar; `--verified` makes §2b's `verified` count the numerator when the count was inflated:
|
|
242
225
|
|
|
243
|
-
|
|
226
|
+
```bash
|
|
227
|
+
uv run {reconcileScript} --denominator-source scalar --coverage-inputs "{run_dir}/coverage-inputs.json" --inventory "{run_dir}/inventory.json" --skill-dir "{resolved_skill_package}" --verified "{run_dir}/numerator.json" --output "{run_dir}/coverage.json"
|
|
228
|
+
```
|
|
244
229
|
|
|
245
|
-
|
|
246
|
-
- `missing` / `missingCount` — source names not documented (barrel enumerates the names; scalar/stack give only the residual count). Never negative.
|
|
247
|
-
- `stale` / `staleCount` — documented names not in source (barrel only; empty with `staleApplicable: false` for scalar/stack)
|
|
248
|
-
- `denominator` — `|barrel_set|` (barrel) or the resolved scalar/stack denominator
|
|
249
|
-
- `exportCoverage` — `documented / denominator * 100`, already rounded; capped at 100 on the scalar/stack branches
|
|
250
|
-
- `numeratorSurplus` / `coverageUncapped` / `coverageCapped` — scalar/stack only; see the surplus note below
|
|
230
|
+
3. **Stack** (`metadata.json.skill_type == "stack"`):
|
|
251
231
|
|
|
252
|
-
|
|
232
|
+
```bash
|
|
233
|
+
uv run {reconcileScript} --denominator-source stack --coverage-inputs "{run_dir}/coverage-inputs.json" --skill-dir "{resolved_skill_package}" --output "{run_dir}/coverage.json"
|
|
234
|
+
```
|
|
253
235
|
|
|
254
|
-
|
|
236
|
+
4. **Docs-only**: Export Coverage is the documentation completeness ratio, complete items over documented items, and `incomplete[]` lists each incomplete item with the fields it lacks:
|
|
255
237
|
|
|
256
|
-
|
|
238
|
+
```bash
|
|
239
|
+
uv run {reconcileScript} --denominator-source docsOnly --inventory "{run_dir}/inventory.json" --output "{run_dir}/coverage.json"
|
|
240
|
+
```
|
|
257
241
|
|
|
258
|
-
|
|
242
|
+
Exit 2 printed an `INVALID_INPUT` envelope: correct the input it names and run it again. Exit 1: HALT with its stderr (`halt_reason: "helper-failed"`, phase `coverage-check:reconcile`).
|
|
259
243
|
|
|
260
|
-
**
|
|
244
|
+
**A surplus is a signal, not a pass.** The scalar and stack branches count the numerator and the denominator from independent sets, so `documented` can exceed `denominator`: the script then reports `numeratorSurplus` above 0 and `coverageCapped: true`, holds `exportCoverage` at 100 and keeps the raw ratio in `coverageUncapped`. State the surplus beside both counts in the Coverage Analysis section. On a skill whose brief carries no `scope.tier_a_include`, a large surplus is the deflated-denominator signature: check `guards.deflation` before accepting the 100.
|
|
261
245
|
|
|
262
|
-
|
|
263
|
-
|--------|------|-----------|-----------------|-----------|--------|
|
|
264
|
-
| {name} | function/class/type | yes/no | yes/no/unverified | src/file.ts:42 | PASS/FAIL/WARN |
|
|
246
|
+
**Classify what the script found.** §5b records each of these gaps from the run files:
|
|
265
247
|
|
|
266
|
-
**
|
|
267
|
-
-
|
|
268
|
-
-
|
|
269
|
-
- Missing documentation: `missingCount`
|
|
270
|
-
- Signature mismatches: {N}
|
|
271
|
-
- Undocumented in SKILL.md but not in source (stale docs): `staleCount`
|
|
248
|
+
- **Missing names** (barrel branch: `missing`): Medium `missing-export` gaps (Medium `missing-type` for a name `signatures.json` lists in `missingTypes`), each at the file `surface.json` records for it, so update-skill can re-extract the export from it. A missing export does not block: it lowers Export Coverage.
|
|
249
|
+
- **Missing count** (scalar or stack branch, `missingCount` above 0): one Medium `missing-export` gap for the count. When §2b's numerator ground-truth check found the count inflated, its `absent[]` names replace this gap, one each: update-skill documents a named export, not a count.
|
|
250
|
+
- **Stale names** (barrel branch: `stale`): each is a documented name the enumerated source surface lacks. Where ast-grep read the source at the pinned commit (Forge, Forge+ or Deep tier, `analysis_confidence` `full`, `workspaceDrift` not `overridden` in the `{outputFile}` frontmatter) and init.md §2 bound `{forge_provenance_map}`, check the lines the skill cites for them: resolve `{verifyProvenanceCompletenessHelper}` ← first existing path in `{verifyProvenanceCompletenessProbeOrder}` and run once:
|
|
272
251
|
|
|
273
|
-
|
|
252
|
+
```bash
|
|
253
|
+
uv run {verifyProvenanceCompletenessHelper} classify-stale --names "{run_dir}/coverage.json" --provenance "{forge_provenance_map}" --source-root "{source_path}" -o "{run_dir}/stale.json"
|
|
254
|
+
```
|
|
274
255
|
|
|
275
|
-
|
|
256
|
+
A name it marks `fabricated: true` (every map entry of the name cites a missing file or one that does not define it) is a Critical `fabricated-signature` gap at its `source`. Any other stale name (no entry, the conditions above not met, a helper that does not resolve or exits non-zero) is a Medium `stale-documentation` gap.
|
|
276
257
|
|
|
277
|
-
|
|
258
|
+
### 4. Category Scores and the Denominator Record
|
|
278
259
|
|
|
279
|
-
|
|
280
|
-
- **Signature Accuracy:** (matching_signatures / total_documented) * 100 (Forge/Deep only, "N/A" for Quick)
|
|
281
|
-
- **Type Coverage:** (documented_types / total_types) * 100 (Forge/Deep only, "N/A" for Quick)
|
|
260
|
+
Each category score is read from its script's file, the file step 5 hands to the scoring script:
|
|
282
261
|
|
|
283
|
-
|
|
262
|
+
- **Export Coverage:** `coverage.json` `exportCoverage`
|
|
263
|
+
- **Signature Accuracy:** `signatures.json` `signatureAccuracy` (Forge, Forge+ and Deep tier at State 1; "N/A" otherwise)
|
|
264
|
+
- **Type Coverage:** `signatures.json` `typeCoverage`, over the types of the set §2b scored; "N/A" where Signature Accuracy is
|
|
284
265
|
|
|
285
|
-
|
|
286
|
-
Stratified-scope resolution picks ONE of three denominator candidates
|
|
287
|
-
(`stats.effective_denominator`, `tier_a_include` union, `scope.include` union)
|
|
288
|
-
per the priority above. To make the choice auditable, append a
|
|
289
|
-
`Denominator Candidates` block immediately after the `Denominator:` line listing
|
|
290
|
-
all three values — the chosen one explicitly marked and the other two recorded
|
|
291
|
-
as-observed (or `absent` when the candidate was not present for this skill):
|
|
266
|
+
Record the denominator source in the Coverage Analysis section with the annotation the matching clause of `{sourceAccessProtocol}` specifies: `Denominator: {barrel | stratified (…) | multi-entry (…) | specific-modules (…) | pattern-reference (…) | stack composition (…) | pattern-surface (…) | docs-only completeness}`, and the `canonical.summary` line when `surface.json` has a fold. When stratified-scope or multi-entry resolution picked the denominator, append this block right after the `Denominator:` line, with every count from `surface.json` `candidates` (`absent` for a null one):
|
|
292
267
|
|
|
293
268
|
```markdown
|
|
294
269
|
**Denominator Candidates** (stratified-scope audit trail):
|
|
295
|
-
- `stats.effective_denominator`: {
|
|
296
|
-
- `scope.tier_a_include` union: {
|
|
297
|
-
- `scope.include` union: {
|
|
298
|
-
- exports-map subpath union: {
|
|
299
|
-
- root barrel: {
|
|
270
|
+
- `stats.effective_denominator`: {statsEffectiveDenominator | absent} {← chosen if priority (1) applied}
|
|
271
|
+
- `scope.tier_a_include` union: {tierAIncludeUnion | absent} {← chosen if priority (2) applied}
|
|
272
|
+
- `scope.include` union: {scopeIncludeUnion | absent} {← chosen if priority (3) applied}
|
|
273
|
+
- exports-map subpath union: {subpathUnion | absent} {← chosen if the multi-entry clause applied}
|
|
274
|
+
- root barrel: {rootBarrel | absent} {secondary candidate: root-barrel-vs-subpath-union audit}
|
|
300
275
|
```
|
|
301
276
|
|
|
302
|
-
|
|
303
|
-
against the other two without re-running the extraction. A future reviewer who
|
|
304
|
-
suspects denominator gaming has the evidence inline. The `multi-entry` clause
|
|
305
|
-
requires the root-barrel named-export count in the `root barrel` row so the
|
|
306
|
-
root-barrel-vs-subpath-union choice is auditable.
|
|
277
|
+
The guards raise coverage gaps too, which §5b records from `surface.json`: `guards.deflation.fires` is a Medium `metadata-drift` gap and `guards.inflation.fires` a Medium `denominator-inflation` gap.
|
|
307
278
|
|
|
308
279
|
### 4b. Metadata Export-Count Coherence Cross-Check
|
|
309
280
|
|
|
310
|
-
|
|
311
|
-
|
|
312
|
-
**Stack-skill branch (`metadata.json.skill_type == "stack"`):** Skip the intra-cluster and cross-cluster count comparisons, the `confidence_distribution` sum check, AND the numerator ground-truth check below — none apply to a stack. For a stack the three counts measure intentionally *different* surfaces, so comparing them yields only false drift: `exports_documented` / `exports[]` measure the stack's own barrel (empty by design → `0`), the provenance-map enumerates the cited *constituent* contracts, and `confidence_distribution` bins those constituents — so for a stack treat `confidence_distribution` as a per-constituent count (it sums to the constituent count, not to `exports_documented`) and do not assert it against `exports_documented`. The numerator-inflation arm below targets an *individual* skill whose `exports_documented` was padded to equal `effective_denominator`; a stack's numerator is instead computed by full-grep in §2c's stack-skill branch and stack `metadata.json.stats` carries no `effective_denominator`, so the arm is not run. Record the denominator using the §2b source — `Denominator: stack composition ({N} cited contracts)` when the provenance-map supplied it, or `Denominator: stack composition ({N} libraries + integration pairs)` when the composition surface did — then proceed.
|
|
313
|
-
|
|
314
|
-
**Reference-app branch (`metadata.json.scope_type == "reference-app"`):** Skip the intra-cluster and cross-cluster count comparisons, the `confidence_distribution` sum check, AND the numerator ground-truth check below — none apply to a reference app, whose counts measure intentionally *different* surfaces. A reference app documents wiring/construct **pattern surfaces**, not a library export barrel, so `metadata.json.exports[]` is empty by design (`0`), `stats.exports_public_api` / `stats.pattern_surfaces_documented` count the documented pattern surfaces, and `confidence_distribution` bins the per-citation provenance entries (it sums to the citation count, not to `pattern_surfaces_documented`). Comparing these yields only false drift: a spurious Cluster-A "barrel drift" (`exports_public_api` vs `exports[].length == 0`) and a Cluster-B "documented-surface drift" (`pattern_surfaces_documented` vs the larger `confidence_distribution` sum). The numerator-inflation arm also does not apply — a reference app carries no `effective_denominator` (see the `skf-create-skill` reference-app carve-out), so the `exports_documented == effective_denominator` signature never fires. Record the denominator as `Denominator: pattern-surface ({pattern_surfaces_documented})` and proceed. (`referenceApp` and a stack skill are distinct signals — a skill is one or the other, never both, so only one of these two branches applies.)
|
|
315
|
-
|
|
316
|
-
**Collect available counts (skip any that are absent) and bin them into two clusters:**
|
|
317
|
-
|
|
318
|
-
**Cluster A — public-barrel surface** (what `__init__.py` / `index.ts` / `lib.rs` re-exports):
|
|
319
|
-
|
|
320
|
-
1. `metadata.json.stats.exports_public_api` — the declared public API count
|
|
321
|
-
2. `metadata.json.exports[]` array length — the enumerated public export list
|
|
322
|
-
|
|
323
|
-
**Cluster B — documented surface** (what was extracted and documented, including methods and submodule members):
|
|
324
|
-
|
|
325
|
-
3. `metadata.json.stats.exports_documented` — the declared documented count
|
|
326
|
-
4. Provenance-map **named-export count** (if `{forge_data_folder}/{skill_name}/provenance-map.json` exists) — pass the raw `export_name` values as `provenanceExportNames`; the script counts top-level named exports, **excluding entries whose `export_name` contains `::`** (impl-block methods like `Type::method`, which roll up under an already-counted type and are not separate barrel exports). Comparing the raw entry count instead false-positives on any method-enumerating provenance map (common for Rust/TS type-heavy skills): e.g. a map with 88 named exports + 48 `Type::method` entries reports 136 against `exports_documented` ≈ 92 → a spurious ~32% Cluster-B drift, while the comparable named-export count (88) agrees within ~4%.
|
|
327
|
-
5. `confidence_distribution` (`t1`, `t1_low`, `t2`, `t3`, when present in `metadata.json.stats`) — pass the tiers as `confidenceDistribution`; the script sums them. Every extracted/documented export is binned into exactly one confidence tier, so the distribution must sum to the documented-surface total; a divergence (e.g., distribution sums to 91 while `exports_documented` is 85) is an internal-consistency defect even when the two clusters look fine
|
|
328
|
-
|
|
329
|
-
Cluster assignment is canonical: `skf-create-skill` step 5 derives `exports_public_api` from entry-point validation and writes the `exports[]` array from the same barrel surface (see `skf-create-skill/references/compile.md:105`), while `exports_documented` tracks the broader documented surface that the provenance-map also enumerates.
|
|
330
|
-
|
|
331
|
-
**Delegate the drift arithmetic to `uv run {coherenceScript}`.** The intra-cluster / cross-cluster `>10%` comparisons are deterministic count arithmetic with one correct answer per input, so the script — not the prompt — owns the binning, the divergence percentages, and every skip condition (a cluster with fewer than two present counts, and clusters that agree within the threshold, are skipped inside the script). Build the input from the counts collected above and run it (`{coherenceScript}` resolves relative to the skill root):
|
|
281
|
+
`{coherenceScript}` compares the `coverage-inputs.json` counts and owns every skip (a stack, a reference app, a cluster with fewer than two counts, drift within its threshold):
|
|
332
282
|
|
|
333
283
|
```bash
|
|
334
|
-
|
|
284
|
+
uv run {coherenceScript} --inputs "{run_dir}/coverage-inputs.json" > "{run_dir}/metadata-coherence.json"
|
|
335
285
|
```
|
|
336
286
|
|
|
337
|
-
|
|
287
|
+
A non-zero exit leaves the script's error, or nothing, in that file: delete it, so §5b skips its call and records no count finding, and note the error in the Coverage Analysis section. §5b records each `findings[]` entry: a Medium entry as a Medium `metadata-drift` gap and an Info entry as an Info `multi-denominator` gap. They describe data quality and change no denominator. Record the denominator of a stack (`Denominator: stack composition ({N} cited contracts)` or `({N} libraries + integration pairs)`, from `stack.basis`) and of a reference app (`Denominator: pattern-surface ({pattern_surfaces_documented})`).
|
|
338
288
|
|
|
339
|
-
|
|
340
|
-
{
|
|
341
|
-
"skillType": "{metadata.json.skill_type or null}",
|
|
342
|
-
"scopeType": "{metadata.json.scope_type or null}",
|
|
343
|
-
"clusterA": {"exports_public_api": 0, "exports_length": 0},
|
|
344
|
-
"clusterB": {"exports_documented": 0},
|
|
345
|
-
"provenanceExportNames": [ /* raw provenance-map export_name values — the script excludes `::` impl-block methods and counts the rest */ ],
|
|
346
|
-
"confidenceDistribution": {"t1": 0, "t1_low": 0, "t2": 0, "t3": 0}
|
|
347
|
-
}
|
|
348
|
-
```
|
|
349
|
-
|
|
350
|
-
The script returns `{"skipped": bool, "clusterACounts": {...}, "clusterBCounts": {...}, "findings": [...]}`. Read `findings[]` and append each entry directly — **do not re-derive the percentages by hand.** Each entry carries `severity`, `title`, `detail` (the enumerated counts + drift %), and `category: structural/metadata coherence`:
|
|
289
|
+
**The numerator ground truth §2b ran.** `numerator.json` `inflated: true` (`verified < declared`) is a High `numerator-inflation` gap listing its `absent[]` names, which §5b records; on the scalar branch those names also replace §2c's missing-count gap. `inflated: false` or `skipped: true` is no finding.
|
|
351
290
|
|
|
352
|
-
|
|
353
|
-
- An **Info** `multi-denominator reporting — barrel vs documented surface` is expected for skills whose documented surface intentionally exceeds the barrel (methods, submodule members, re-exported classes) — it is not drift. The note exists so the test report makes the dual-denominator design visible and auditable without demanding action.
|
|
291
|
+
### 4c. Provenance Line Check
|
|
354
292
|
|
|
355
|
-
(
|
|
293
|
+
Check that the provenance map records each export at the line that defines it (the `def` or declaration line itself, not a decorator or blank line above it), so update-skill can move a wrong line. Run this section only when `analysis_confidence` is `full`, `{forge_version}/provenance-map.json` exists, and the `{outputFile}` frontmatter's `workspaceDrift` is not `overridden` (init.md §5b: the drifted tree is not the pinned commit). Otherwise skip to section 5.
|
|
356
294
|
|
|
357
|
-
|
|
295
|
+
Resolve `{verifyProvenanceCompletenessHelper}` ← first existing path in `{verifyProvenanceCompletenessProbeOrder}`. If neither path exists, skip to section 5: the check is advisory. Otherwise run:
|
|
358
296
|
|
|
359
297
|
```bash
|
|
360
|
-
|
|
298
|
+
uv run {verifyProvenanceCompletenessHelper} verify \
|
|
299
|
+
--metadata {resolved_skill_package}/metadata.json \
|
|
300
|
+
--provenance {forge_version}/provenance-map.json \
|
|
301
|
+
--source-root {source_path} \
|
|
302
|
+
-o "{run_dir}/provenance-verify.json"
|
|
361
303
|
```
|
|
362
304
|
|
|
363
|
-
|
|
364
|
-
|
|
365
|
-
- `inflated: false` (`verified == declared`) → the skill is genuinely fully documented; no finding, coverage stands.
|
|
366
|
-
- `inflated: true` (`verified < declared`) → emit a **High**-severity gap `numerator inflation — {declared − verified} of {declared} declared exports absent from SKILL.md/references` listing the script's `absent[]` names, and use `verified` as the Export Coverage numerator (overriding `exports_documented`). A numerator padded to equal the denominator otherwise produces a tautological 100% that passes the gate.
|
|
305
|
+
Rely on the file, not the exit code (exit 1 means findings); on exit 2 the helper writes no file: skip to section 5. §5b records each `stale[]` item whose `reason` is `line-not-definition`: a Low `provenance-line` gap when its `definition_lines` holds a line, else an Info `provenance-unverified` gap (the rules may not cover its shape: the export is unverified, not gone). Its Source is exactly the `file:line` the map records, the citation update-skill's rule R5 reads.
|
|
367
306
|
|
|
368
|
-
|
|
307
|
+
This section reports none of the helper's other findings, and neither gap blocks the gate.
|
|
369
308
|
|
|
370
|
-
|
|
309
|
+
### 5. Write the Coverage Analysis Section
|
|
371
310
|
|
|
372
|
-
|
|
373
|
-
|
|
374
|
-
Append the **Coverage Analysis** section to `{outputFile}`:
|
|
311
|
+
Write the **Coverage Analysis** section in place of the template's `## Coverage Analysis` heading and the placeholder comment under it, in `{outputFile}`. The counts come from `{run_dir}/coverage.json` and `{run_dir}/signatures.json`:
|
|
375
312
|
|
|
376
313
|
```markdown
|
|
377
314
|
## Coverage Analysis
|
|
@@ -379,23 +316,25 @@ Append the **Coverage Analysis** section to `{outputFile}`:
|
|
|
379
316
|
**Tier:** {forge_tier}
|
|
380
317
|
**Source Access:** {analysis_confidence} (full | provenance-map | metadata-only | remote-only | docs-only)
|
|
381
318
|
**Source Path:** {source_path}
|
|
319
|
+
**Workspace Layout:** {source_workspace_kind | single package | unknown (no local source, or the §0b helper failed)}
|
|
382
320
|
**Files Analyzed:** {count}
|
|
383
|
-
**Denominator:** {
|
|
321
|
+
**Denominator:** {the §4 annotation}
|
|
384
322
|
|
|
385
323
|
### Export Coverage
|
|
386
324
|
|
|
387
325
|
| Export | Type | Documented | Signature | Source Location | Status |
|
|
388
326
|
|--------|------|-----------|-----------|-----------------|--------|
|
|
389
|
-
|
|
|
327
|
+
| {name} | {kind} | yes/no | yes/no/unverified | {file}:{line} | PASS/FAIL/WARN |
|
|
390
328
|
|
|
391
329
|
### Coverage Summary
|
|
392
330
|
|
|
393
|
-
- **Exports Found:** {
|
|
394
|
-
- **Documented:** {
|
|
395
|
-
- **Missing Documentation:** {
|
|
396
|
-
- **Signature Mismatches:** {
|
|
397
|
-
- **Stale Documentation:** {
|
|
398
|
-
- **
|
|
331
|
+
- **Exports Found:** {`denominator`; docs-only: the documented items}
|
|
332
|
+
- **Documented:** {`documented`} ({`exportCoverage`}%)
|
|
333
|
+
- **Missing Documentation:** {`missingCount`; docs-only: the incomplete items, each with the fields it lacks}
|
|
334
|
+
- **Signature Mismatches:** {the length of `signatures.json` `mismatches`}
|
|
335
|
+
- **Stale Documentation:** {`staleCount`}
|
|
336
|
+
- **Scoring Warnings:** each entry of `signatures.json` `warnings`; omit this row when there is none
|
|
337
|
+
- **Numerator Surplus:** {`numeratorSurplus`}, uncapped coverage {`coverageUncapped`}%; only when `coverageCapped: true` (then `Missing Documentation: 0` is a floored residual)
|
|
399
338
|
|
|
400
339
|
### Category Scores
|
|
401
340
|
|
|
@@ -404,13 +343,37 @@ Append the **Coverage Analysis** section to `{outputFile}`:
|
|
|
404
343
|
| Export Coverage | {N}% |
|
|
405
344
|
| Signature Accuracy | {N}% or N/A |
|
|
406
345
|
| Type Coverage | {N}% or N/A |
|
|
346
|
+
```
|
|
407
347
|
|
|
408
|
-
|
|
348
|
+
### 5b. Record the Coverage Gaps
|
|
349
|
+
|
|
350
|
+
Record every gap this step found in the gap ledger `{ledgerFile}`: the hard gate (step 4c) decides from it, and the report step renders the Gap Report from it. `{gapLedgerScript}` writes the record of each gap a script found from that script's file: §2b's signature mismatches, §2c's missing names, missing count and stale names, §4's guard gaps, §4b's count findings and numerator gap, and §4c's provenance line gaps, each with the severity and category of its row in the Gap Severity table of `{scoringRulesFile}` and a fixed title, Source, `export` and Remediation. Run each command whose `--input` file this run wrote, leaving out a bracketed flag whose file does not exist: a run file this run did not write, or `--provenance` when init.md §2 bound no `{forge_provenance_map}`:
|
|
351
|
+
|
|
352
|
+
```bash
|
|
353
|
+
uv run {gapLedgerScript} append --ledger "{ledgerFile}" --stage coverage-check --input "{run_dir}/signature-gaps.json"
|
|
354
|
+
uv run {gapLedgerScript} append --ledger "{ledgerFile}" --stage coverage-check --from coverage --input "{run_dir}/coverage.json" [--surface "{run_dir}/surface.json"] [--signatures "{run_dir}/signatures.json"] [--numerator "{run_dir}/numerator.json"] [--stale "{run_dir}/stale.json"] [--provenance "{forge_provenance_map}"] --skill-dir "{resolved_skill_package}" --metadata "{resolved_skill_package}/metadata.json"
|
|
355
|
+
uv run {gapLedgerScript} append --ledger "{ledgerFile}" --stage coverage-check --from guards --input "{run_dir}/surface.json" --metadata "{resolved_skill_package}/metadata.json"
|
|
356
|
+
uv run {gapLedgerScript} append --ledger "{ledgerFile}" --stage coverage-check --from numerator --input "{run_dir}/numerator.json" --metadata "{resolved_skill_package}/metadata.json"
|
|
357
|
+
uv run {gapLedgerScript} append --ledger "{ledgerFile}" --stage coverage-check --from metadata-coherence --input "{run_dir}/metadata-coherence.json" --metadata "{resolved_skill_package}/metadata.json"
|
|
358
|
+
uv run {gapLedgerScript} append --ledger "{ledgerFile}" --stage coverage-check --from provenance-line --input "{run_dir}/provenance-verify.json"
|
|
409
359
|
```
|
|
410
360
|
|
|
411
|
-
|
|
361
|
+
Then write the records no script wrote, §1b's split-body mismatches, in the Ledger Record Format of `{outputFormatsFile}`, each naming its export in `export` and in its title: the ledger tells two gaps apart by their title, Source and `export`, and update-skill takes the export's name from `export`. Write them as one JSON array on the lines between the two markers, exactly as they are: the quoted marker hands them to the script unchanged, quotes, apostrophes and `$` included. Run the command even when the array is empty (`[]`): the hard gate refuses to decide until every stage before it has recorded, with gaps or without.
|
|
362
|
+
|
|
363
|
+
```bash
|
|
364
|
+
uv run {gapLedgerScript} append --ledger "{ledgerFile}" --stage coverage-check <<'SKF_GAPS'
|
|
365
|
+
<the records, one JSON array>
|
|
366
|
+
SKF_GAPS
|
|
367
|
+
```
|
|
368
|
+
|
|
369
|
+
Rely on the JSON of each command:
|
|
412
370
|
|
|
413
|
-
|
|
371
|
+
- Exit 0: the records are in the ledger. `appended` names the id each new record received, and `duplicates` the ones a rerun of this step had already recorded.
|
|
372
|
+
- Exit 2 (`INVALID_RECORD` or `INVALID_INPUT`): nothing was written. Correct each record `errors[]` names (its `index` counts from 0), or the file or flag the `error` names, and run the command again.
|
|
373
|
+
- Exit 1: HALT with the script's `error` (`halt_reason: "helper-failed"`, phase `coverage-check:ledger`).
|
|
374
|
+
|
|
375
|
+
### 6. Report Coverage Results
|
|
414
376
|
|
|
415
|
-
|
|
377
|
+
Report the coverage result to the user: the {forge_tier}-tier analysis of {file_count} source files, the documented ratios for exports / signatures / types (signatures and types are N/A for Quick tier), and the issue count. Full details are in the Coverage Analysis section.
|
|
416
378
|
|
|
379
|
+
Append `'coverage-check'` to `stepsCompleted` in the `{outputFile}` frontmatter, then load and execute {nextStepFile}.
|