bmad-module-skill-forge 0.9.0 → 0.10.1-alpha.0

package/src/skf-test-skill/references/scoring-rules.md

@@ -0,0 +1,207 @@
+# Scoring Rules
+
+## Default Threshold
+
+**Pass threshold:** 80%
+
+## Category Weights
+
+| Category               | Weight | Description                                                                               |
+|------------------------|--------|-------------------------------------------------------------------------------------------|
+| Export Coverage        | 36%    | Percentage of source exports documented in SKILL.md                                       |
+| Signature Accuracy     | 22%    | Documented signatures match actual source signatures                                      |
+| Type Coverage          | 14%    | Types and interfaces referenced are complete                                              |
+| Coherence (contextual) | 18%    | Cross-references valid, integration patterns complete                                     |
+| Coherence (naive)      | 0%     | Not applicable — weight redistributed to other categories                                 |
+| External Validation    | 10%    | Average of skill-check quality score + tessl average score (redistributed if unavailable) |
+
+## Naive Mode Weight Redistribution
+
+The following weights replace the default table for naive mode. The 18% coherence weight from the default table has been proportionally redistributed into these values. Do not re-redistribute for coherence (already handled in this table). Quick-tier redistribution (zeroing Signature Accuracy and Type Coverage) still applies as an additional step.
+
+When running in naive mode (no coherence category):
+- Export Coverage: 45%
+- Signature Accuracy: 25%
+- Type Coverage: 20%
+- External Validation: 10%
+
+## External Validation Unavailable
+
+When neither skill-check nor tessl is available, redistribute the 10% external validation weight proportionally to the other active categories. When only one tool is available, use that tool's score as the external validation score.
+
+## tessl and Split-Body Interaction
+
+tessl evaluates SKILL.md body content only — it does not read `references/*.md` files. After split-body extraction, the tessl content score will drop significantly (e.g., 65% to 38%) because Tier 2 content is no longer inline. This is expected behavior and does not reflect actual content quality. When reporting scores for a split-body skill, note: "tessl content score reflects post-split inline content only. Use the pre-split tessl score as the content quality baseline."
+
+## Tier-Dependent Scoring
+
+### Quick Tier (no tools)
+- Export Coverage: file/structure existence check only
+- Signature Accuracy: skipped (no AST)
+- Type Coverage: skipped (no AST)
+- Score based on: structural completeness only
+- Weight redistribution: skipped categories' weights (Signature Accuracy 22% + Type Coverage 14%) redistributed proportionally to remaining active categories
+
+### Docs-Only Mode (all [EXT:...] citations, any tier)
+
+When `docs_only_mode: true` is set by step-03 (indicating a skill where all SKILL.md citations are `[EXT:...]` format with no local source code):
+
+- **Signature Accuracy:** Not scored (no source to compare against)
+- **Type Coverage:** Not scored (no source to compare against)
+- **Weight redistribution:** Same as Quick tier — Signature Accuracy (22%) and Type Coverage (14%) weights redistributed proportionally to remaining active categories
+- **Export Coverage basis:** Documentation completeness rather than source coverage. Score = (documented_items_with_complete_descriptions / total_documented_items) * 100. A "complete" item has: description, parameters (if function/method), and return type (if function/method).
+- **Coherence:** Standard rules for the detected mode (naive or contextual) apply unchanged
+
+This is functionally identical to Quick tier weight redistribution but with a different coverage denominator (self-consistency instead of source comparison).
+
+**S4 external-validator requirement for docs-only:** docs-only mode removes two categories (Signature Accuracy, Type Coverage) from scoring. If External Validation is ALSO unavailable, the evidence base collapses to Coverage alone (naive) or Coverage + Coherence (contextual) — which in the naive/Quick case trips the minimum-evidence floor (INCONCLUSIVE). To keep docs-only skills gradable when external validators are present but still deterministic when they are missing: **when `docsOnly: true` AND `externalValidation is null`, step-05 MUST cap `totalScore` at `threshold - 1` (forcing FAIL) before the INCONCLUSIVE floor is evaluated.** This prevents a docs-only skill from PASSing with only one or two redistributed categories carrying all the weight. Implement in step-05 §4 as a pre-compare cap, recorded in the report as `scoring_notes: docs-only without external validators — capped below threshold per S4`.
+
+### Stack Skills (Any Tier)
+
+When `metadata.json.skill_type == "stack"` (set `stackSkill: true` in the scoring input):
+
+- **Signature Accuracy:** N/A — a stack skill's "signature surface" is the external library API it composes (pydantic, SQLAlchemy, FastAPI, etc.), not a proprietary surface the skill authors. Grading signatures against a surface the skill does not own produces meaningless numbers.
+- **Type Coverage:** N/A — same rationale; the type surface belongs to the external libraries.
+- **Weight redistribution:** Same as Quick tier / docs-only / State 2 — Signature Accuracy (22%) and Type Coverage (14%) weights redistributed proportionally to remaining active categories (Export Coverage, Coherence, External Validation).
+- **Applies regardless of detected tier** (Quick, Forge, Forge+, Deep) and is independent of `docsOnly` and `state2`. A stack skill can also be docs-only or State 2; the skip reasons combine additively (e.g. `"stack skill (external type surface) + State 2 (provenance-map)"`).
+- **Detection:** step-05 reads `metadata.json.skill_type` from the skill package. If the value is `"stack"`, set `stackSkill: true` in the scoring input JSON.
+
+Equivalence-class note: stack skills with `docsOnly:false` / `state2:false` map to the same equivalence class as State 2 contextual rows (class B) or State 2 naive rows (class D) — the redistribution math is identical; only the `skipReasons` string changes.
+
+### State 2 Source Access (Any Tier, Provenance-Map Only)
+
+When source is not locally available and analysis resolves to State 2 (provenance-map baseline per source-access-protocol.md):
+
+- **Signature Accuracy:** N/A — provenance-map stores parameters as flat string arrays; verification is string comparison only, not semantic AST verification. Type aliases (`str` vs `String`, `list` vs `List[Any]`) cannot be resolved without live source.
+- **Type Coverage:** N/A — cannot verify type completeness without local source access for AST re-parsing.
+- **Weight redistribution:** Same as Quick tier — Signature Accuracy (22%) and Type Coverage (14%) weights redistributed proportionally to remaining active categories (Export Coverage, Coherence, External Validation).
+- **Applies regardless of detected tier** (including Forge, Forge+, Deep) whenever `analysis_confidence` is `provenance-map` and local source is unavailable.
+- **Export Coverage denominator:** Uses the union of provenance-map entry names and metadata.json `exports[]` names (per source-access-protocol.md State 2 rules).
+
+Note: When provenance-map entries are predominantly T1 (AST-verified at compilation time), the coverage and name-matching data is already at highest confidence. The N/A categories reflect the inability to re-verify at test time, not low-quality extraction data.
+
+**S9 State 2 undercount risk acknowledgement:** provenance-map is a cached extraction snapshot — if the source has evolved since extraction, public API adds/removes will NOT surface in Export Coverage (denominator is frozen to the provenance-map union). When `state2: true` AND step-03 records any provenance vs metadata divergence (e.g. union > either source by >5%), apply a flat **10% deduction** to `exportCoverage` before calling the scoring script, AND set `analysis_confidence: provenance-map` (already set) with a report note: `scoring_notes: State 2 undercount risk acknowledged — 10% deduction applied to Export Coverage`. Rationale: the skill cannot be reliably scored on a frozen denominator when the cache is known to disagree with its own metadata; prefer understating over overstating.
+
+### Forge Tier (ast-grep)
+- Export Coverage: AST-backed export comparison
+- Signature Accuracy: AST-verified signature matching
+- Type Coverage: AST-verified type completeness
+- Full scoring formula applied
+
+### Forge+ Tier (ast-grep + ccc)
+- Same scoring as Forge tier — ccc provides pre-ranking but does not change scoring weights
+- Improved extraction coverage (from ccc pre-discovery) may increase T1 count, but scoring formula is identical to Forge
+- Full scoring formula applied
+
+### Deep Tier (ast-grep + gh + QMD)
+- All Forge tier checks plus:
+- Cross-repository reference verification
+- QMD knowledge enrichment for coherence
+- Full scoring formula with maximum depth
+- **Migration & Deprecation Warnings section:** If T2-future annotations exist in the enrichment data, verify that Section 4b is present in SKILL.md Tier 1 and that each warning traces to a T2 provenance citation. If no T2-future annotations exist, Section 4b should normally be absent (not empty). Presence/absence mismatch is a Medium severity gap — with one Info-severity exception for historical-migration content (completed package renames, consolidated import paths, shipped API cutovers that remain load-bearing for training-data drift remediation). See `steps-c/step-04-coherence-check.md` §2b/§5b for the three-case rule.
+
+## Redistribution Combinations Matrix (M3 — terminology + determinism)
+
+The table below enumerates every `(mode × tier × docsOnly × state2)` cell and
+the redistribution outcome. Two cells map to the same **equivalence class**
+whenever their skip set and base-weight table are identical — the scoring
+script then emits the same `weights` / `activeCategories` / `skippedCategories`
+output regardless of which specific cell produced it. The rightmost column
+cites the representative fixture in
+`test/fixtures/compute-score-contract.json` used to pin the class's math.
+
+| # | mode       | tier     | docsOnly | state2 | Base table     | Skipped (sig/type)    | Skipped (ext if null) | Equiv. class | Representative fixture        |
+|---|------------|----------|----------|--------|----------------|-----------------------|-----------------------|--------------|-------------------------------|
+| 1 | contextual | Deep     | F        | F      | contextual     | —                     | if null               | **A**        | `suite_a_all_active`          |
+| 2 | contextual | Forge+   | F        | F      | contextual     | —                     | if null               | A            | `suite_k_forge_plus`          |
+| 3 | contextual | Forge    | F        | F      | contextual     | —                     | if null               | A            | `suite_p_contextual_forge`    |
+| 4 | contextual | Quick    | F        | F      | contextual     | Quick tier            | if null               | **B**        | `suite_c_quick_tier`          |
+| 5 | contextual | Deep     | T        | F      | contextual     | docs-only             | if null               | B            | `suite_r_contextual_deep_docsonly` |
+| 6 | contextual | Forge+   | T        | F      | contextual     | docs-only             | if null               | B            | (equiv. — see `suite_r_*`)    |
+| 7 | contextual | Forge    | T        | F      | contextual     | docs-only             | if null               | B            | (equiv.)                      |
+| 8 | contextual | Quick    | T        | F      | contextual     | Quick tier + docs-only| if null               | B            | `suite_f_docs_only`           |
+| 9 | contextual | Deep     | F        | T      | contextual     | State 2               | if null               | B            | `suite_g_state2`              |
+| 10| contextual | Forge+   | F        | T      | contextual     | State 2               | if null               | B            | (equiv.)                      |
+| 11| contextual | Forge    | F        | T      | contextual     | State 2               | if null               | B            | (equiv.)                      |
+| 12| contextual | Quick    | F        | T      | contextual     | Quick + State 2       | if null               | B            | (equiv. — Quick+state2)       |
+| 13| contextual | *        | T        | T      | contextual     | docs-only + State 2   | if null               | B            | (equiv.)                      |
+| 14| naive      | Deep     | F        | F      | naive          | —                     | if null               | **C**        | `suite_q_naive_deep`          |
+| 15| naive      | Forge+   | F        | F      | naive          | —                     | if null               | C            | (equiv.)                      |
+| 16| naive      | Forge    | F        | F      | naive          | —                     | if null               | C            | `suite_b_naive`               |
+| 17| naive      | Quick    | F        | F      | naive          | Quick tier            | if null               | **D**        | `suite_e_triple_skip`         |
+| 18| naive      | Deep     | T        | F      | naive          | docs-only             | if null               | D            | `suite_s_naive_deep_docsonly` |
+| 19| naive      | Forge+   | T        | F      | naive          | docs-only             | if null               | D            | (equiv.)                      |
+| 20| naive      | Forge    | T        | F      | naive          | docs-only             | if null               | D            | (equiv.)                      |
+| 21| naive      | Quick    | T        | F      | naive          | Quick + docs-only     | if null               | D            | `suite_o_input_echo`          |
+| 22| naive      | Deep     | F        | T      | naive          | State 2               | if null               | D            | `suite_t_naive_state2`        |
+| 23| naive      | Forge+   | F        | T      | naive          | State 2               | if null               | D            | (equiv.)                      |
+| 24| naive      | Forge    | F        | T      | naive          | State 2               | if null               | D            | (equiv.)                      |
+| 25| naive      | Quick    | F        | T      | naive          | Quick + State 2       | if null               | D            | (equiv.)                      |
+| 26| naive      | *        | T        | T      | naive          | docs-only + State 2   | if null               | D            | (equiv.)                      |
+
+**How to read the matrix.** The four equivalence classes A/B/C/D are
+exhaustive: the scoring script's weight-redistribution output depends ONLY on
+(base table, sig/type skip, ext skip), so every row mapping to class X emits
+the same final weights for identical input scores. "(equiv.)" rows are *not*
+missing coverage — they reduce algebraically to a listed representative. The
+M3 fixture set ships at least one representative per equivalence class and one
+representative per unique skip-reason combination so the `skipReasons` string
+is also pinned (e.g. "Quick tier", "docs-only mode", "State 2
+(provenance-map)", "Quick tier + docs-only mode").
+
+**Quick-tier INCONCLUSIVE interaction.** Rows 4, 8, 12, 17, 21, 25 all force
+the minimum-evidence floor to evaluate "Quick tier + Export Coverage alone"
+cases separately — see `Result Determination` below. The matrix above is the
+pre-floor redistribution only; INCONCLUSIVE verdict determination runs after.
+
+## Score Calculation
+
+```
+score = sum(category_weight * category_score) for each category
+category_score = (items_passing / items_total) * 100
+```
+
+## Coherence Score Aggregation (Contextual Mode)
+
+```
+reference_validity = (valid_references / total_references) * 100
+integration_completeness = (complete_patterns / total_patterns) * 100
+combined_coherence = (reference_validity * 0.6) + (integration_completeness * 0.4)
+```
+
+If no integration patterns exist, combined coherence equals reference validity.
+
+## Result Determination
+
+Three-state gate — **PASS / FAIL / INCONCLUSIVE**. `INCONCLUSIVE` is not PASS and not FAIL; it signals insufficient evidence to grade the skill. Downstream workflows MUST treat `INCONCLUSIVE` as a hard gate — do not export, do not auto-retry, surface to the human.
+
+- **Minimum-Evidence Floor (MANDATORY — applies before PASS/FAIL comparison):**
+  - `active_categories` = count of categories with a non-zero final weight *after* all redistribution (Quick tier, docs-only, State 2, external-validator-unavailable). Categories with a redistributed weight of 0 do not count as active, even if they received a score.
+  - **If `active_categories < 2`** → force `result: INCONCLUSIVE` with rationale `"insufficient evidence: only {N} active category"`. A single active category cannot cross-validate itself and a PASS would be a false signal.
+  - **If `tier == "Quick"` AND the sole active contributor is Export Coverage** → force `result: INCONCLUSIVE` with rationale `"Quick tier: Export Coverage alone is insufficient evidence — add a second active category by upgrading tier or enabling external validators"`. This catches the degenerate case where every signature/type/coherence/external category gets redistributed to 0 and Export Coverage is doing all the work.
+  - The floor is enforced by `scripts/compute-score.py`. The step-05 scoring step reads `result` from the script output and writes it into the test report frontmatter unchanged.
+
+- Otherwise:
+  - score >= threshold → PASS
+  - score < threshold → FAIL
+
+The floor is intentionally conservative: skf-test-skill grades other skills, so a false PASS has catastrophic downstream effects (polluted exports, misleading feasibility data). Falling back to INCONCLUSIVE is always preferred over a low-evidence PASS.
+
+## Gap Severity
+
+| Severity | Criteria                                                                                                       |
+|----------|----------------------------------------------------------------------------------------------------------------|
+| Critical | Missing exported function/class documentation                                                                  |
+| High     | Signature mismatch between source and SKILL.md                                                                 |
+| Medium   | Missing type or interface documentation                                                                        |
+| Medium   | Migration section present/absent mismatch with T2-future annotation data (Deep tier)                           |
+| Medium   | Metadata drift — intra-cluster export counts diverge (barrel: `stats.exports_public_api` vs `exports[].length`; or documented-surface: `stats.exports_documented` vs provenance-map entry count; >10% divergence) |
+| Medium   | Denominator inflation — stratified-scope `scope.include` union exceeds provenance-map entry count by >25% (brief missing `scope.tier_a_include`) |
+| Medium   | Script/asset directory exists but no Scripts & Assets section in SKILL.md                                      |
+| Medium   | Scripts & Assets section references file not found in scripts/ or assets/ directory                            |
+| Low      | Script/asset file present without provenance entry in provenance-map.json file_entries                         |
+| Low      | Missing optional metadata or examples                                                                          |
+| Low      | Description trigger optimization recommended (third-person voice, negative triggers, or keyword coverage gaps) |
+| Info     | Style suggestions, non-blocking observations                                                                   |
+| Info     | Discovery testing not performed — realistic prompt testing recommended before export                           |
+| Info     | Multi-denominator reporting — barrel vs documented-surface clusters diverge by design (>10% cross-cluster)     |

package/src/skf-test-skill/references/source-access-protocol.md

@@ -0,0 +1,109 @@
+# Source Access Protocol
+
+## Source API Surface Definition
+
+**Source API surface** = the package's top-level public exports. These are the symbols reachable from the primary entry point without importing internal modules:
+
+- **Python:** symbols exported in `__init__.py` (including re-exports) — exclude private (`_prefixed`) names
+- **TypeScript/JavaScript:** named exports from `index.ts` / `index.js` — exclude unexported locals
+- **Go:** exported identifiers (capitalized) from the package's public-facing files
+- **Rust:** items in `pub use` from `lib.rs` or `mod.rs`
+- **Empty-barrel packages (copy-paste / subpath-only distribution):** If the primary entry point is empty or re-exports nothing (e.g., `export {};` in `index.ts`, an empty `__init__.py`, `lib.rs` with no `pub use`), the package does not expose a barrel API. Do **not** compute coverage against the empty barrel — the denominator would be zero and the score meaningless. Instead, consult the skill brief's `scope.include` globs (`forge-data/{skill_name}/skill-brief.yaml`) to identify the authorized entry points, and build the public API surface from the **union of named exports across those files**. The skill brief's `scope.notes` field should document this distribution model explicitly; if present, treat it as confirmation that the empty barrel is by design rather than a bug. If no skill brief is available and the barrel is empty, set `analysis_confidence: docs-only` and report that the source API surface could not be determined.
+
+- **Stratified-scope monorepo packages (curated subsets of multi-package repos):** If the source is a monorepo (detect via `packages/` layout, `workspaces` field in root `package.json`, `lerna.json`, `rush.json`, `nx.json`, or Cargo `[workspace]`) AND the skill brief's `scope.include` lists a curated file/directory subset rather than the full workspace, the coverage denominator must reflect only the authored surface, not the monorepo's global export count. This is distinct from the empty-barrel case: each workspace package may have a non-empty barrel, but the skill intentionally documents only a tiered subset.
+
+  **Resolution order:**
+
+  1. **Prefer `metadata.json.stats.effective_denominator`** when present. `skf-create-skill` step-05 §4 writes this field for stratified-scope skills. When set, use it directly as the `exports_public_api` count for coverage scoring.
+  2. **Fall back to live re-derivation** when `effective_denominator` is absent (older skills, quick-tier output, or skills compiled before this rule existed). Read the brief's scope globs from `forge-data/{skill_name}/skill-brief.yaml`, resolve them against `source_path`, filter out files matching `scope.exclude`, and compute the source API surface as the **union of named exports across the matched files only**. The skill brief's `scope.notes` field should document the stratification strategy (e.g., "Tier A: fully documented; Tier B: deferred to references; Tier C: excluded") — when present, treat it as confirmation that the curated subset is by design, not a scope gap.
+
+     **Honor `scope.tier_a_include` when present.** When re-deriving, prefer the brief-level `scope.tier_a_include` narrow include list over the coarse `scope.include`. `tier_a_include` is an optional brief field that lists only the authoring surface the brief actually intends to document (tier A), letting the denominator match the brief's authoring-vs-installing intent even when `scope.include` uses coarse globs that also match internal infrastructure. When `tier_a_include` is present, resolve its globs (still filtered by `scope.exclude`), compute the union across those files, and use that count as the denominator. When absent, fall back to resolving `scope.include`.
+
+  3. **Denominator inflation check (absent `tier_a_include`).** If re-derivation used `scope.include` because no `tier_a_include` was provided, compare the resulting union count against the provenance-map entry count (when provenance-map exists). If the `scope.include` union is more than 25% larger than the provenance-map entry count, the coarse globs are almost certainly sweeping in internal infrastructure that the brief did not intend to document. Emit a **Medium**-severity gap titled `denominator inflation — coarse scope.include union exceeds authored surface` that points the user at the brief for rescoping via `scope.tier_a_include`. Report both counts (`scope.include union: {N}`, `provenance-map entries: {M}`, `{percent}% inflation`) and state that the coverage score it produced is driven by denominator inflation rather than documentation gaps. The check is skipped when provenance-map is unavailable (there is no baseline to compare against).
+
+  Leave `analysis_confidence` unchanged (still `full` or `provenance-map` per the waterfall) — stratified scope does not degrade confidence, only the denominator. Annotate the coverage report with: `Stratified scope — denominator: {effective_denominator | tier_a_include union | scope.include union} ({N} files matched, {M} exports union)`.
+
+  **When this clause does NOT apply:** `scope.type: "full-library"` skills, single-package repositories, or stratified briefs where the full monorepo is intentionally in scope. For those, use the standard barrel-based denominator — **unless** the single-package repo is a pattern-reference app (see next bullet).
+
+- **Pattern-reference apps (non-library source):** If the source is a single-package repo whose purpose is demonstrating an integration pattern rather than distributing a library API — typical markers are `scope.type: "full-library"` **without** a barrel file at any recognized entry-point path (`__init__.py`, `index.ts`/`index.js`, `lib.rs`, `mod.rs`) AND without a monorepo layout — the skill's value lives in wiring patterns, not exports. None of the preceding three clauses fits: there is no barrel to count from, no empty-barrel `scope.include` to consult, and no monorepo stratification to re-derive.
+
+  **Trigger (either fires):**
+
+  1. `scope.notes` in `forge-data/{skill_name}/skill-brief.yaml` flags pattern-reference intent (phrases such as "Reference app, not a library", "pattern-reference", "embedded-pattern skill", or "skill value is the … pattern"). The `scope.notes` field is authoritative when the author wrote it.
+  2. Source tree lacks a barrel file at every recognized entry-point path AND the repo is not a monorepo (no `packages/`, `workspaces`, `lerna.json`, `rush.json`, `nx.json`, or Cargo `[workspace]`). Detected at test time by filesystem inspection of `{source_path}`.
+
+  **Denominator:** canonicalized provenance-map entry count (same canonicalization as the "Provenance-map canonicalization" section below). `skf-create-skill`'s extraction pass has already curated the provenance-map to the authored pattern surface; treat it as the authoritative enumeration of the skill's documented reach.
+
+  **Recommendation — prefer `tier_a_include`:** authors should add `scope.tier_a_include` to the brief listing the files that constitute the authored pattern surface, the same way stratified-scope briefs do. When `tier_a_include` is present, use its re-derived union (filtered by `scope.exclude`) as the denominator exactly as in the stratified-scope clause. When absent, fall back to the canonicalized provenance-map count — do not fabricate a denominator from arbitrary source-tree sweeps.
+
+  **Confidence:** leave `analysis_confidence` unchanged (still `full` or `provenance-map` per the waterfall). Pattern-reference does not degrade confidence — the surface is smaller than a library barrel, not lower quality. Annotate the coverage report with: `Pattern-reference — denominator: {tier_a_include union | canonicalized provenance-map count} ({N} pattern surfaces)`.
+
+  **When this clause does NOT apply:** any repo with a non-empty barrel file, any monorepo (use the stratified-scope clause), or any single-package repo whose `scope.type` is explicitly `public-api` / `specific-modules` / `component-library` / `docs-only` (those scope types have their own denominator semantics). Also does NOT apply if `scope.type: "reference-app"` exists in the enum (pending upgrade in `skf-create-skill/steps-c/step-03-extract.md`) — in that case the brief speaks for itself and this clause's filesystem trigger is moot.
+
+Internal module symbols are **excluded** from the coverage denominator unless they are explicitly documented in SKILL.md (in which case they count as documented extras, not missing coverage).
+
+This matches the extraction-patterns.md convention used during skill creation: coverage measures how well SKILL.md documents what users actually import, not the entire internal codebase.
+
+### Provenance-map canonicalization
+
+When the test-side intersects documented SKILL.md exports against a stratified-scope provenance-map, raw provenance-map entry names may include **bookkeeping variants** of the same underlying export. These variants are artifacts of how the source library structures its registry (e.g., Storybook's component-plus-story decomposition, accessibility renderer shadowing, exact-match versus fuzzy-match renderer disambiguation). Counting them as separate exports inflates the denominator and produces false "missing documentation" findings for names that are structurally duplicates of an already-documented base export.
+
+Before intersecting documented names against the provenance-map entry list, **fold bookkeeping variants back to their base name** using the rules below. This matches the convention `skf-create-skill` records in `metadata.json.stats.effective_denominator_source` (e.g., `"provenance-map canonicalized count (ThemesGlobals_def folds with ThemesGlobals under _def convention)"`) — the base form is authoritative; the variant form is a sibling record, not an independent export.
+
+**Folding rules (apply in order, case-sensitive):**
+
+1. **Suffix `_def`** — registry definition twin. `ThemesGlobals_def` folds to `ThemesGlobals`. Common in Storybook-style component registries where the definition object and the rendered component share the same public name.
+2. **Suffix `_exact`** — exact-match variant. `ButtonSpec_exact` folds to `ButtonSpec`. Common in matcher/renderer registries where an `_exact` sibling signals a stricter resolution path.
+3. **Prefix `a11y_`** — accessibility renderer shadow. `a11y_Checkbox` folds to `Checkbox`. Common in accessibility-wrapper layers that re-export every component under a parallel prefixed namespace.
+4. **Other renderer-prefix disambiguation** — when the library uses a prefix-namespace convention (e.g., `mobile_`, `web_`, `ssr_`) to shadow the base export, fold the prefix form back to the base. **Only apply when the base form is also present in the provenance-map** — otherwise the prefix form is the real export and should be kept. Document the specific prefix used in the test report so the rule is auditable.
+
+**How to apply:**
+
+1. Read all entry names from the provenance-map.
+2. Build a canonical-name set by applying the folding rules above — each variant maps to its base. Retain the original variant → base mapping for reporting.
+3. Intersect the documented SKILL.md export names against the **canonical** set, not the raw entry list.
+4. When computing `Export Coverage`, use the **canonical count** as the denominator — not the raw provenance-map entry count. This aligns the denominator with `metadata.json.stats.effective_denominator` (when present), which `skf-create-skill` already writes as the canonicalized count.
+5. In the test report, note the fold summary: `Provenance-map canonicalization: {N} raw entries → {M} canonical bases ({N−M} bookkeeping variants folded: _def×{a}, _exact×{b}, a11y_×{c}, other×{d})`. This makes the reduction auditable by future testers and update runs.
+
+**When to skip canonicalization:**
+
+- If the library's public surface genuinely distinguishes the variants (e.g., `a11y_Checkbox` is a separately-documented, separately-installed component and not a shadow), do not fold — the variant is a real export. Check SKILL.md for explicit documentation of the variant before folding. When in doubt, err on the side of not folding and report both forms.
+- If `metadata.json.stats.effective_denominator` is present and the provenance-map raw count matches it (no drift), canonicalization is not needed — the denominator is already canonical. Fold only when raw count > `effective_denominator` and the drift corresponds to recognizable bookkeeping suffixes/prefixes.
+- If drift remains after folding (e.g., raw 222 → canonical 215 but `effective_denominator` says 216), record the residual 1-count drift as an unexplained-reconciliation note in the test report. Do not fabricate additional fold rules to close the gap.
+
+## Source Access Resolution
+
+Before analysis, determine source access level. Walk through these states in order — use the first that succeeds:
+
+**State 1 — Local source available:**
+Check if `{source_path}` (from metadata.json `source_root`) exists on disk. If yes → full analysis at detected tier (AST + signatures). Set `analysis_confidence: full`.
+
+**State 2 — Local absent, provenance-map exists:**
+Check `{forge_data_folder}/{skill_name}/provenance-map.json`. If present AND contains at least 1 entry, use it as the baseline export inventory — each entry contains structured fields: `export_name`, `export_type`, `params[]`, `return_type`, `source_file`, `source_line`, `confidence`, and `ast_node_type`. Cross-reference against SKILL.md documented exports for name-matching and param-by-param coverage. Signature verification compares SKILL.md's documented params/return types against provenance-map entries directly.
+
+**Cross-reference with metadata.json:** After loading provenance-map entries, compare the entry count against `metadata.json`'s `exports[]` array length and `stats.exports_public_api` count. If metadata reports more exports than provenance-map entries:
+- Compute `gap = metadata.exports.length - provenance_map.entries.length`
+- Report: "Provenance-map contains {pmap_count} entries but metadata.json lists {meta_count} exports ({gap} gap). Coverage denominator uses the union."
+- Build the coverage denominator from the **union** of provenance-map entry names and metadata.json `exports[]` names. Exports present in metadata but absent from provenance-map are counted as "missing documentation" in the coverage calculation.
+- If metadata.json is unavailable or has no `exports[]` array, use provenance-map count alone with a note: "Coverage denominator is provenance-map only — may undercount if extraction was incomplete." If remote reading tools are available (zread, deepwiki, gh API, or similar), supplement by reading the entry point file for live signature verification. Set `analysis_confidence: provenance-map`.
+
+**State 2 limitations:** Signature verification at State 2 is **string comparison only**, not semantic. Provenance-map stores parameters as flat string arrays (e.g., `["data: Union[BinaryIO, list, str]"]`), so `str` vs `String` or `list` vs `List[Any]` would be treated as mismatches even when semantically equivalent. For full type-aware verification (handling type aliases, generic equivalence), State 1 (local source) with AST re-parsing is required. When the SKILL.md was compiled from the same provenance-map (typical for create-then-test flows), most strings will match. However, enrichment (step-04) and doc-fetching (step-03c) during compilation may alter parameter descriptions, add type annotations, or normalize signatures, causing mismatches even in create-then-test flows. Expect some string-level mismatches and treat them as compilation artifacts, not source drift signals, until signature fidelity is enforced by step-05's Signature Fidelity Rule (see `signature_source` field in provenance-map entries).
+
+**State 3 — No provenance-map, metadata exports exist (quick-skill path):**
+If no provenance-map.json exists (typical for quick-skill output), fall back to `metadata.json`'s `exports[]` array for the export name list. Coverage check becomes a self-consistency comparison: are all names in `exports[]` documented in SKILL.md with description, parameters, and return type? Signatures cannot be verified. If remote reading tools are available, supplement by reading the entry point for live export comparison. Set `analysis_confidence: metadata-only`.
+
+**State 4 — No local source, no forge-data, remote tools available:**
+If neither provenance-map nor metadata exports provide a usable baseline, but remote reading tools (zread, deepwiki, gh API, or similar) are available and `source_repo` is set in metadata.json, read the entry point remotely to build the export inventory from scratch. Name-matching only — no AST. Set `analysis_confidence: remote-only`.
+
+**State 5 — No source access at all:**
+If none of the above succeed, fall through to docs-only mode (as defined in step-03-coverage-check.md Section 0: pre-analysis source type detection). Set `analysis_confidence: docs-only`. Warn: "**No source access available.** Coverage check evaluates documentation self-consistency only. Re-run with local clone or remote access for source-backed verification."
+
+Set `analysis_confidence` in context for use in Section 2 analysis depth, step-05 output, and step-05 scoring.
+
+**Confidence tier mapping:** `full` = T1, `provenance-map` = T1, `metadata-only` = T1-low, `remote-only` = T1-low, `docs-only` = T3. This aligns with the T1/T1-low/T2/T3 scale used across all SKF workflows.
+
+**Degradation notice rules:** When `analysis_confidence` is `provenance-map`, check the `confidence` field of provenance-map entries before emitting a degradation recommendation:
+
+- **All/most entries T1 (AST-verified):** The provenance-map data is already at highest confidence. Do NOT recommend re-running with a local clone — it would produce identical results. Use: "Resolved via: provenance-map (T1 AST-verified at compilation time). Local clone not required — provenance data is already at highest confidence."
+- **Mixed T1/T1-low entries:** Report the breakdown. Recommend local clone only for the T1-low entries: "Resolved via: provenance-map ({n} T1, {m} T1-low). Re-run with local clone to upgrade T1-low entries to full AST verification."
+- **All/most entries T1-low or lower:** Keep the standard recommendation: "Re-run with local clone for full AST-backed verification."

package/src/skf-test-skill/scripts/compute-score.py

@@ -0,0 +1,310 @@
+# /// script
+# requires-python = ">=3.10"
+# dependencies = []
+# ///
+"""Deterministic Completeness Score Calculator.
+
+Pure-function scoring script for the SKF test-skill workflow (step-05).
+Implements the weight tables, skip conditions, and proportional redistribution
+defined in scoring-rules.md.
+
+CLI: python3 compute-score.py '<JSON>'
+"""
+
+from __future__ import annotations
+
+import json
+import math
+import sys
+
+# --- Weight Tables (from scoring-rules.md) ---
+
+CONTEXTUAL_WEIGHTS = {
+    "exportCoverage": 36,
+    "signatureAccuracy": 22,
+    "typeCoverage": 14,
+    "coherence": 18,
+    "externalValidation": 10,
+}
+
+NAIVE_WEIGHTS = {
+    "exportCoverage": 45,
+    "signatureAccuracy": 25,
+    "typeCoverage": 20,
+    "coherence": 0,
+    "externalValidation": 10,
+}
+
+CATEGORIES = [
+    "exportCoverage",
+    "signatureAccuracy",
+    "typeCoverage",
+    "coherence",
+    "externalValidation",
+]
+
+DEFAULT_THRESHOLD = 80
+
+
+# --- Helpers ---
+
+
+def round2(value):
+    """Round to 2 decimal places using JavaScript-compatible rounding.
+
+    JavaScript Math.round rounds .5 up (away from zero for positives).
+    Python's built-in round uses banker's rounding (.5 to even).
+    We replicate JS behavior: floor(value * 100 + 0.5) / 100.
+    """
+    return math.floor(value * 100 + 0.5) / 100
+
+
+def make_error(message):
+    return {"error": message, "code": "INVALID_INPUT"}
+
+
+# --- Validation ---
+
+
+BOOL_FIELDS = ("docsOnly", "state2", "stackSkill")
+
+
+def validate_input(inp):
+    if inp is None or not isinstance(inp, dict):
+        return "Input must be a JSON object"
+
+    if not inp.get("mode") or inp["mode"] not in ("contextual", "naive"):
+        return 'Missing or invalid required field: mode (must be "contextual" or "naive")'
+
+    valid_tiers = ["Quick", "Forge", "Forge+", "Deep"]
+    if not inp.get("tier") or inp["tier"] not in valid_tiers:
+        return f"Missing or invalid required field: tier (must be one of: {', '.join(valid_tiers)})"
+
+    # H3: reject string booleans — bare true/false only.
+    # `bool` is a subclass of `int` in Python; accept only actual bools or the
+    # absence of the field. Strings like "true"/"false" and ints 0/1 are rejected
+    # to catch the common YAML/JSON hand-editing mistake of quoting the value.
+    for field in BOOL_FIELDS:
+        if field in inp and inp[field] is not None:
+            if not isinstance(inp[field], bool):
+                return (
+                    f"Field `{field}` must be a bare boolean (true/false). "
+                    f"Got {type(inp[field]).__name__}: {inp[field]!r}. "
+                    "String 'true'/'false' or 0/1 is not accepted — "
+                    "this typically indicates a YAML/JSON quoting mistake."
+                )
+
+    if "scores" not in inp or not isinstance(inp.get("scores"), dict):
+        return "Missing required field: scores"
+
+    if inp["scores"].get("exportCoverage") is None:
+        return "scores.exportCoverage is required and cannot be null"
+
+    threshold = inp.get("threshold")
+    if threshold is not None:
+        if not isinstance(threshold, (int, float)) or threshold < 0 or threshold > 100:
+            return "threshold must be a number between 0 and 100"
+
+    for cat in CATEGORIES:
+        score = inp["scores"].get(cat)
+        if score is not None:
+            if isinstance(score, bool) or not isinstance(score, (int, float)):
+                return f"scores.{cat} must be a number or null, got: {type(score).__name__}"
+            if score < 0 or score > 100:
+                return f"scores.{cat} must be between 0 and 100, got: {score}"
+
+    return None
+
+
+# --- Core Scoring Function ---
+
+
+def compute_score(inp):
+    # 1. Validate
+    validation_error = validate_input(inp)
+    if validation_error:
+        return make_error(validation_error)
+
+    mode = inp["mode"]
+    tier = inp["tier"]
+    docs_only = inp.get("docsOnly") is True
+    state2 = inp.get("state2") is True
+    stack_skill = inp.get("stackSkill") is True
+    threshold = inp.get("threshold") if inp.get("threshold") is not None else DEFAULT_THRESHOLD
+    scores = inp["scores"]
+
+    # 2. Select base weight table
+    base_weights = dict(NAIVE_WEIGHTS if mode == "naive" else CONTEXTUAL_WEIGHTS)
+
+    # 3. Determine skip set
+    skip_reasons = {}
+    skip_sig_type = tier == "Quick" or docs_only or state2 or stack_skill
+
+    if skip_sig_type:
+        reasons = []
+        if tier == "Quick":
+            reasons.append("Quick tier")
+        if docs_only:
+            reasons.append("docs-only mode")
+        if state2:
+            reasons.append("State 2 (provenance-map)")
+        if stack_skill:
+            reasons.append("stack skill (external type surface)")
+        reason = " + ".join(reasons)
+        skip_reasons["signatureAccuracy"] = reason
+        skip_reasons["typeCoverage"] = reason
+
+    if scores.get("externalValidation") is None:
+        skip_reasons["externalValidation"] = "No external validators available"
+
+    # Collect warnings
+    warnings = []
+    for cat in skip_reasons:
+        if scores.get(cat) is not None:
+            warnings.append(
+                f"{cat} score provided ({scores[cat]}) but category is skipped — score ignored"
+            )
+
+    # Validate active categories have scores
+    skipped_set = set(skip_reasons.keys())
+    for cat in CATEGORIES:
+        is_active = cat not in skipped_set and base_weights[cat] > 0
+        score_missing = scores.get(cat) is None
+        if is_active and score_missing:
+            return make_error(
+                f"Category {cat} is active but score is null. "
+                "Provide a numeric score or set the appropriate skip condition."
+            )
+
+    # 4. Redistribute weights
+    adjusted_weights = dict(base_weights)
+    for cat in skip_reasons:
+        adjusted_weights[cat] = 0
+
+    sum_active_weights = sum(adjusted_weights[cat] for cat in CATEGORIES)
+
+    final_weights = {}
+    for cat in CATEGORIES:
+        if adjusted_weights[cat] == 0:
+            final_weights[cat] = 0
+        else:
+            final_weights[cat] = round2((adjusted_weights[cat] / sum_active_weights) * 100)
+
+    # 5. Compute weighted scores
+    weighted_scores = {}
+    for cat in CATEGORIES:
+        if final_weights[cat] == 0:
+            weighted_scores[cat] = 0
+        else:
+            weighted_scores[cat] = round2((final_weights[cat] / 100) * scores[cat])
+
+    # 6. Compute total
+    total_score = round2(sum(weighted_scores[cat] for cat in CATEGORIES))
+
+    # Weight sum for verification
+    weight_sum = round2(sum(final_weights[cat] for cat in CATEGORIES))
+
+    # 7. Determine result — MINIMUM-EVIDENCE FLOOR first, then PASS/FAIL.
+    # skf-test-skill grades other skills; a false PASS is catastrophic.
+    # If the evidence base is too thin to cross-validate itself, force
+    # INCONCLUSIVE (a gate, not a pass/fail). See scoring-rules.md.
+    active_categories = [cat for cat in CATEGORIES if final_weights[cat] > 0]
+    skipped_categories = [
+        cat for cat in CATEGORIES if cat in skipped_set or base_weights[cat] == 0
+    ]
+
+    floor_reasons = []
+    if len(active_categories) < 2:
+        floor_reasons.append(
+            f"insufficient evidence: only {len(active_categories)} active category"
+        )
+    elif tier == "Quick" and active_categories == ["exportCoverage"]:
+        # Defensive second check: if somehow there are >= 2 active categories
+        # but they collapse to just exportCoverage (shouldn't happen given the
+        # first clause, kept for robustness), still force INCONCLUSIVE.
+        floor_reasons.append(
+            "Quick tier: Export Coverage alone is insufficient evidence "
+            "— add a second active category by upgrading tier or enabling "
+            "external validators"
+        )
+    elif tier == "Quick":
+        # Cover the case where Export Coverage is the only active category
+        # carrying signal in Quick tier even when technically another
+        # non-contributing category survived redistribution.
+        non_export_active_scores = [
+            scores.get(cat) for cat in active_categories if cat != "exportCoverage"
+        ]
+        # All other active categories have a zero score => Export Coverage is
+        # the sole real contributor.
+        if active_categories and "exportCoverage" in active_categories and all(
+            (s == 0) for s in non_export_active_scores
+        ) and non_export_active_scores:
+            floor_reasons.append(
+                "Quick tier: Export Coverage is the sole scoring contributor "
+                "(other active categories scored 0) — insufficient evidence"
+            )
+
+    if floor_reasons:
+        result = "INCONCLUSIVE"
+    else:
+        result = "PASS" if total_score >= threshold else "FAIL"
+
+    # Build scores echo with null preservation
+    scores_echo = {}
+    for cat in CATEGORIES:
+        scores_echo[cat] = scores.get(cat)
+
+    output = {
+        "input": {
+            "mode": mode,
+            "tier": tier,
+            "docsOnly": docs_only,
+            "state2": state2,
+            "stackSkill": stack_skill,
+            "threshold": threshold,
+            "scores": scores_echo,
+        },
+        "activeCategories": active_categories,
+        "skippedCategories": skipped_categories,
+        "skipReasons": skip_reasons,
+        "weights": final_weights,
+        "weightedScores": weighted_scores,
+        "totalScore": total_score,
+        "threshold": threshold,
+        "result": result,
+        "weightSum": weight_sum,
+    }
+
+    if warnings:
+        output["warnings"] = warnings
+
+    if floor_reasons:
+        output["inconclusiveReasons"] = floor_reasons
+
+    return output
+
+
+# --- CLI Entry Point ---
+
+if __name__ == "__main__":
+    if len(sys.argv) < 2:
+        print("Usage: python3 compute-score.py '<JSON>'", file=sys.stderr)
+        print(
+            'Example: python3 compute-score.py \'{"mode":"contextual","tier":"Deep",'
+            '"scores":{"exportCoverage":92,"signatureAccuracy":85,"typeCoverage":100,'
+            '"coherence":80,"externalValidation":78}}\'',
+            file=sys.stderr,
+        )
+        sys.exit(1)
+
+    try:
+        data = json.loads(sys.argv[1])
+    except json.JSONDecodeError:
+        print(
+            json.dumps(make_error(f"Invalid JSON: {sys.argv[1][:100]}"), indent=2)
+        )
+        sys.exit(1)
+
+    result = compute_score(data)
+    print(json.dumps(result, indent=2))
+    sys.exit(1 if "error" in result else 0)