bmad-module-skill-forge 6.0.0 → 7.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +1 -1
- package/docs/_data/pinned.yaml +1 -1
- package/docs/workflows.md +4 -4
- package/package.json +1 -1
- package/src/shared/references/output-contract-schema.md +1 -1
- package/src/shared/scripts/skf-quick-batch.py +16 -3
- package/src/shared/scripts/skf-verify-provenance-completeness.py +209 -33
- package/src/skf-analyze-source/customize.toml +3 -2
- package/src/skf-analyze-source/references/auto-docs-only.md +2 -2
- package/src/skf-analyze-source/references/generate-briefs.md +5 -3
- package/src/skf-analyze-source/references/headless-contract.md +2 -2
- package/src/skf-analyze-source/references/step-auto-scope.md +2 -2
- package/src/skf-campaign/references/campaign-contracts.md +1 -1
- package/src/skf-campaign/references/step-07-capstone.md +3 -3
- package/src/skf-campaign/references/step-08-verify.md +3 -3
- package/src/skf-campaign/references/step-09-refine.md +3 -3
- package/src/skf-campaign/scripts/campaign-state.py +22 -5
- package/src/skf-create-skill/SKILL.md +2 -2
- package/src/skf-create-skill/customize.toml +4 -4
- package/src/skf-create-skill/references/batch-mode.md +3 -1
- package/src/skf-create-skill/references/report.md +7 -2
- package/src/skf-create-skill/references/validate.md +1 -1
- package/src/skf-create-stack-skill/references/validate.md +12 -4
- package/src/skf-quick-skill/SKILL.md +1 -1
- package/src/skf-quick-skill/references/batch-mode.md +2 -0
- package/src/skf-quick-skill/references/write-and-validate.md +19 -9
- package/src/skf-test-skill/references/coverage-check.md +2 -2
- package/src/skf-test-skill/references/inventory-contract.md +1 -1
- package/src/skf-test-skill/references/scoring-rules.md +2 -2
- package/src/skf-test-skill/references/signature-comparison-rules.md +3 -1
- package/src/skf-test-skill/scripts/load-coverage-inputs.py +241 -11
- package/src/skf-test-skill/scripts/score-signatures.py +9 -13
- package/src/skf-test-skill/scripts/validate-inventory.py +91 -12
- package/src/skf-update-skill/customize.toml +1 -1
- package/src/skf-update-skill/references/report.md +10 -3
- package/src/skf-update-skill/references/write.md +1 -1
|
@@ -51,7 +51,7 @@ Every write to `_campaign-state.yaml` goes through `scripts/campaign-state.py`.
|
|
|
51
51
|
| `set-skill` | step-05, step-10, step-resume | a skill's `status`, `quality_score`, `skill_path`, `brief_path` or `export` outcome (`--export`); `active` stamps `started_at` once, `completed` stamps `completed_at`, `pending` clears both and the `export` outcome |
|
|
52
52
|
| `append-workarounds` | step-05 | entries of a skill's `workarounds_applied`, each once |
|
|
53
53
|
| `apply-batch` | step-06 | the Tier B batch from its line-to-skill map: `--start`, `--results-file` or `--no-results` |
|
|
54
|
-
| `set-campaign` | steps 07 to 09 | `architecture_doc_path`, and the capstone, verification or refinement summary read from the sub-skill's envelope line (`capstone.verified` with the verification) |
|
|
54
|
+
| `set-campaign` | steps 07 to 09 | `architecture_doc_path`, and the capstone, verification or refinement summary read from the sub-skill's envelope line (`capstone.verified` with the verification); null for a stage skipped on purpose (`--no-<field>`) or a sub-skill that printed no line (`--no-result <field>`, a `not_recorded` row with halt reason `no-result`) |
|
|
55
55
|
| `set-stage` | steps 05, 10, 11 | `current_stage` alone |
|
|
56
56
|
| `recover` | step-resume | `.bak` over the primary, when `.bak` validates (exit 9 otherwise) |
|
|
57
57
|
| `archive` | SKILL.md (overwrite) | nothing: it moves the state, its `.bak` and the brief into a new folder under `archive/` and returns it |
|
|
@@ -50,7 +50,7 @@ Invoke `skf-create-stack-skill` with `--headless` and these inputs, so it asks n
|
|
|
50
50
|
- `stack_name`: the `stack_name` that `uv run {manifestScript} --stack-name {stateFile}` prints, the campaign name as a skill name cut to leave room for the `-stack` SS appends; when it prints null, leave the input out and SS names the stack `{project_name}-stack`
|
|
51
51
|
- `architecture_doc_path`: `campaign.architecture_doc_path`, only when state sets it
|
|
52
52
|
|
|
53
|
-
Keep the `SKF_STACK_RESULT_JSON` line it prints for §4, which records it. On `status` `error`,
|
|
53
|
+
Keep the `SKF_STACK_RESULT_JSON` line it prints for §4, which records it: on stdout, or on stderr when SS halts. On `status` `error`, log its `halt_reason` (type `event`); a run with that line on neither stream is recorded in §4. Verification and the campaign report still run either way.
|
|
54
54
|
|
|
55
55
|
### §4: Record the Capstone and Complete the Stage
|
|
56
56
|
|
|
@@ -69,10 +69,10 @@ From a `success` envelope it sets `campaign.capstone`, the campaign-level summar
|
|
|
69
69
|
- `campaign.capstone.verified`: `null` for now; set by the verify stage (step-08) once the stack is checked
|
|
70
70
|
- `campaign.capstone.completed_at`: the time of the write, from the clock
|
|
71
71
|
|
|
72
|
-
An `error` envelope sets `campaign.capstone` to `null` (the helper names it in `not_recorded`). When §2 found no skill to compose,
|
|
72
|
+
An `error` envelope sets `campaign.capstone` to `null` (the helper names it in `not_recorded`). When §2 found no skill to compose, record no capstone instead: `uv run {stateScript} set-campaign --state-file {stateFile} --no-capstone --stage 6`. When SS printed no `SKF_STACK_RESULT_JSON` line on stdout or stderr, there is no result to read, whatever its exit code: record it with `uv run {stateScript} set-campaign --state-file {stateFile} --no-result capstone --stage 6`, log "no SKF_STACK_RESULT_JSON line (exit <code>)" with SS's exit code (type `event`), and go on.
|
|
73
73
|
|
|
74
74
|
The capstone is a derived artifact: it is **not** tracked as a skill entry in the `skills[]` array. The constituent skill list and any verbose detail are reported in the step output and are available to downstream steps (verify, refine).
|
|
75
75
|
|
|
76
76
|
## OUTPUT
|
|
77
77
|
|
|
78
|
-
Display capstone summary: stack skill name, path, quality score, and the list of constituent skills. Chain to `{nextStepFile}`.
|
|
78
|
+
Display capstone summary: stack skill name, path, quality score, and the list of constituent skills, or "skipped" when §2 found no skill to compose, or "failed (<halt_reason>)" from the helper's `not_recorded` row. That row's `halt_reason` fills it (`failed (no-result)` when SS printed no line); when that is null, show `failed (exit <exit_code>)`, or `failed (no exit code)` when the exit code is null too. Chain to `{nextStepFile}`.
|
|
@@ -46,7 +46,7 @@ Once resolved (steps 2 and 3), persist the path at once, so the refine stage and
|
|
|
46
46
|
|
|
47
47
|
Invoke `skf-verify-stack` with `--headless --architecture-doc <path>`, where `<path>` is the architecture doc §3 resolved.
|
|
48
48
|
|
|
49
|
-
VS discovers skills from its own configured `{skills_output_folder}`; the campaign does NOT pass individual skill paths. Keep the `SKF_VERIFY_STACK_RESULT_JSON` line it prints on stdout
|
|
49
|
+
VS discovers skills from its own configured `{skills_output_folder}`; the campaign does NOT pass individual skill paths. Keep the `SKF_VERIFY_STACK_RESULT_JSON` line it prints for §5: on stdout, or on stderr when VS halts. Its fields are defined once, by `shared/scripts/schemas/skf-verify-stack-result-envelope.v1.json` (in the SKF module: `{project-root}/_bmad/skf/` when installed, `{project-root}/src/` during development); §5 reads `report_path`, `overall_verdict`, `coverage_percentage` and `recommendation_count` from it.
|
|
50
50
|
|
|
51
51
|
### §5: Record the VS Outcome and Complete the Stage
|
|
52
52
|
|
|
@@ -67,8 +67,8 @@ SKF_RESULT
|
|
|
67
67
|
|
|
68
68
|
It also sets `campaign.capstone.verified` to `true` when `overall_verdict` is `FEASIBLE`, otherwise `false` (only if a `campaign.capstone` entry exists from step-07).
|
|
69
69
|
|
|
70
|
-
**On VS failure** (a
|
|
70
|
+
**On VS failure** (a line whose `status` is `error` or whose `exit_code` is non-zero): the helper sets `campaign.verification` to `null` and names it in `not_recorded`; log that line's `exit_code` and `halt_reason` (type `event`). Verification failure does NOT block the campaign: it produces diagnostic information for operator review. When §3 skipped VS, record no verification instead: `uv run {stateScript} set-campaign --state-file {stateFile} --no-verification --stage 7`. When VS printed no `SKF_VERIFY_STACK_RESULT_JSON` line on stdout or stderr, there is no result to read, whatever its exit code: record it with `uv run {stateScript} set-campaign --state-file {stateFile} --no-result verification --stage 7`, log "no SKF_VERIFY_STACK_RESULT_JSON line (exit <code>)" with VS's exit code (type `event`), and go on.
|
|
71
71
|
|
|
72
72
|
## OUTPUT
|
|
73
73
|
|
|
74
|
-
Display verification summary: overall verdict (or "skipped" if architecture doc was not found), report path (if produced), and coverage percentage. Chain to `{nextStepFile}`.
|
|
74
|
+
Display verification summary: overall verdict (or "skipped" if architecture doc was not found, or "failed (<halt_reason>)" from the helper's `not_recorded` row), report path (if produced), and coverage percentage. That row's `halt_reason` fills it (`failed (no-result)` when VS printed no line); when that is null, show `failed (exit <exit_code>)`, or `failed (no exit code)` when the exit code is null too. Chain to `{nextStepFile}`.
|
|
@@ -56,7 +56,7 @@ skf-refine-architecture --headless --architecture-doc <arch_path> --vs-report-pa
|
|
|
56
56
|
- `--vs-report-path`: `campaign.verification.report_path` from §3, or `none` when state holds no report path.
|
|
57
57
|
- `--scope-skills`: comma-separated names of completed campaign skills (from `skills[]` where `status == "completed"`). Optional but improves focus by limiting refinement scope to campaign-relevant skills.
|
|
58
58
|
|
|
59
|
-
Keep the `SKF_REFINE_ARCHITECTURE_RESULT_JSON` line it prints on stdout
|
|
59
|
+
Keep the `SKF_REFINE_ARCHITECTURE_RESULT_JSON` line it prints for §5: on stdout, or on stderr when RA halts. Its fields are defined once, by `shared/scripts/schemas/skf-refine-architecture-result-envelope.v1.json` (in the SKF module: `{project-root}/_bmad/skf/` when installed, `{project-root}/src/` during development); §5 reads `status`, `refined_path`, `gap_count`, `issue_count` and `improvement_count` from it.
|
|
60
60
|
|
|
61
61
|
### §5: Record the RA Outcome and Complete the Stage
|
|
62
62
|
|
|
@@ -75,8 +75,8 @@ SKF_RESULT
|
|
|
75
75
|
- `campaign.refinement.issue_count`: from the envelope
|
|
76
76
|
- `campaign.refinement.improvement_count`: from the envelope
|
|
77
77
|
|
|
78
|
-
**On RA failure** (a
|
|
78
|
+
**On RA failure** (a line whose `status` is `error` or whose `exit_code` is non-zero): the helper sets `campaign.refinement` to `null` and names it in `not_recorded`; log that line's `exit_code` and `halt_reason` (type `event`). Refinement failure does NOT block the campaign: the campaign continues to export with whatever state exists. When §3 skipped RA, record no refinement instead: `uv run {stateScript} set-campaign --state-file {stateFile} --no-refinement --stage 8`. When RA printed no `SKF_REFINE_ARCHITECTURE_RESULT_JSON` line on stdout or stderr, there is no result to read, whatever its exit code: record it with `uv run {stateScript} set-campaign --state-file {stateFile} --no-result refinement --stage 8`, log "no SKF_REFINE_ARCHITECTURE_RESULT_JSON line (exit <code>)" with RA's exit code (type `event`), and go on.
|
|
79
79
|
|
|
80
80
|
## OUTPUT
|
|
81
81
|
|
|
82
|
-
Display refinement summary: refined architecture path (or "skipped" if architecture doc was not found), gap count, issue count, and improvement count. Chain to `{nextStepFile}`.
|
|
82
|
+
Display refinement summary: refined architecture path (or "skipped" if architecture doc was not found, or "failed (<halt_reason>)" from the helper's `not_recorded` row), gap count, issue count, and improvement count. That row's `halt_reason` fills it (`failed (no-result)` when RA printed no line); when that is null, show `failed (exit <exit_code>)`, or `failed (no exit code)` when the exit code is null too. Chain to `{nextStepFile}`.
|
|
@@ -80,8 +80,13 @@ Operations:
|
|
|
80
80
|
summary read from the sub-skill's result envelope line
|
|
81
81
|
(a file or `-`; the line's SKF_*_RESULT_JSON: prefix is
|
|
82
82
|
optional). A success envelope is recorded; an error
|
|
83
|
-
envelope, or --no-capstone and the like
|
|
84
|
-
|
|
83
|
+
envelope, or --no-capstone and the like (a stage
|
|
84
|
+
skipped on purpose), records null. --no-result <field>
|
|
85
|
+
(repeatable) records a sub-skill that printed no result
|
|
86
|
+
line: null, with halt reason no-result in its
|
|
87
|
+
not_recorded row; it does not go with that field's
|
|
88
|
+
--<field> or --no-<field>. A recorded verification also
|
|
89
|
+
sets capstone.verified.
|
|
85
90
|
recover Copy a .bak that validates over the primary (step-resume).
|
|
86
91
|
archive Move the state, its .bak and the brief, each one that
|
|
87
92
|
exists, into archive/<campaign name>-<UTC stamp>/ beside
|
|
@@ -130,7 +135,8 @@ CLI:
|
|
|
130
135
|
--entry <text> [--entry <text> ...] [--stage <N>]
|
|
131
136
|
uv run campaign-state.py set-campaign --state-file <p> [--architecture-doc-path <p>] \\
|
|
132
137
|
[--capstone <p|-> | --no-capstone] [--verification <p|-> | --no-verification] \\
|
|
133
|
-
[--refinement <p|-> | --no-refinement]
|
|
138
|
+
[--refinement <p|-> | --no-refinement] \\
|
|
139
|
+
[--no-result capstone|verification|refinement ...] [--stage <N>]
|
|
134
140
|
uv run campaign-state.py recover --state-file <p>
|
|
135
141
|
uv run campaign-state.py archive --state-file <p> --brief-file <p>
|
|
136
142
|
uv run campaign-state.py log --log-file <p> --type <decision|auto|event> --text <entry>
|
|
@@ -161,8 +167,8 @@ Exit codes (each is the campaign's HALT code for the same class):
|
|
|
161
167
|
2 invalid input: a missing or unreadable input file, an unknown skill,
|
|
162
168
|
output from another script that holds an error, a gate the gate script
|
|
163
169
|
rejects, a state that init would overwrite, a brief template that is
|
|
164
|
-
no YAML mapping, export flags that do not go
|
|
165
|
-
folder that exists
|
|
170
|
+
no YAML mapping, export flags or set-campaign flags that do not go
|
|
171
|
+
together, or an archive folder that exists
|
|
166
172
|
3 invalid state: the state is missing, malformed or fails the schema, or
|
|
167
173
|
the change would make it fail (nothing written)
|
|
168
174
|
4 apply-plan: a plan the stages cannot follow (circular-deps)
|
|
@@ -787,6 +793,11 @@ SUMMARIES = (("capstone", _capstone), ("verification", _verification), ("refinem
|
|
|
787
793
|
|
|
788
794
|
|
|
789
795
|
def op_set_campaign(args: argparse.Namespace) -> dict:
|
|
796
|
+
no_result = set(args.no_result or ())
|
|
797
|
+
for field, _build in SUMMARIES:
|
|
798
|
+
if field in no_result and (getattr(args, field) is not None or getattr(args, f"no_{field}")):
|
|
799
|
+
raise StateError(EXIT_INPUT, "input-invalid",
|
|
800
|
+
f"--no-result {field} does not go with --{field} or --no-{field}")
|
|
790
801
|
envelopes = {}
|
|
791
802
|
for field, _build in SUMMARIES:
|
|
792
803
|
source = getattr(args, field)
|
|
@@ -812,6 +823,9 @@ def op_set_campaign(args: argparse.Namespace) -> dict:
|
|
|
812
823
|
elif getattr(args, f"no_{field}"):
|
|
813
824
|
value = None
|
|
814
825
|
not_recorded.append({"field": field, "status": None, "halt_reason": None, "exit_code": None})
|
|
826
|
+
elif field in no_result:
|
|
827
|
+
value = None
|
|
828
|
+
not_recorded.append({"field": field, "status": None, "halt_reason": "no-result", "exit_code": None})
|
|
815
829
|
else:
|
|
816
830
|
continue
|
|
817
831
|
campaign[field] = value
|
|
@@ -1054,6 +1068,9 @@ def build_parser() -> argparse.ArgumentParser:
|
|
|
1054
1068
|
group = p.add_mutually_exclusive_group()
|
|
1055
1069
|
group.add_argument(f"--{field}", help=f"the {field} sub-skill's result envelope line, a file or -")
|
|
1056
1070
|
group.add_argument(f"--no-{field}", action="store_true", help=f"record campaign.{field} as null")
|
|
1071
|
+
p.add_argument("--no-result", action="append", choices=[f for f, _ in SUMMARIES], metavar="FIELD",
|
|
1072
|
+
help="record the field as null with halt reason no-result: its sub-skill printed no "
|
|
1073
|
+
"result line (repeat for several; not with that field's --FIELD or --no-FIELD)")
|
|
1057
1074
|
|
|
1058
1075
|
command("recover", "copy a valid .bak over the primary", stage=None)
|
|
1059
1076
|
|
|
@@ -65,8 +65,8 @@ These rules apply to every step in this workflow:
|
|
|
65
65
|
|--------|--------|
|
|
66
66
|
| **Inputs** | brief_path (a skill-brief.yaml path or a skill name) [optional: without it, step 1 offers the briefs in `{forge_data_folder}`, and a headless run halts unless there is one], --batch [optional: the briefs or folders of briefs after it, `{forge_data_folder}` when none is given, each validated before the first compiles] |
|
|
67
67
|
| **Gates** | step 1: Brief Selection (no brief named and several in `{forge_data_folder}`; headless: HARD HALT `brief-missing`, exit 2; one brief: picked and recorded as gate `brief-selection`) | step 3: Tag Choice Gate (several tags match the brief's version; headless: the first in priority order) | step 3 §2a: Authoritative-File Gate [P]/[S]/[U] (an authoritative AI documentation file outside the brief's scope; headless: deferred-headless, one auto-decision per path) | step 3: Review Gate [C]/[R] (with the zero-export warning when a source brief without `doc_urls` extracted no export; [R] with it or the head-cap warning) | step 3c: Zero-Export Gate [C]/[R] (a source brief with no export whose every `doc_urls` fetch failed) | step 3d: Demo-Exclusion Gate [Y], then Registry Gate [Y] (candidate found) or [S] (none found), component-library scope only, each skipped once the brief holds `scope.demo_patterns` or `scope.registry_path`, which step 3d writes when a user confirms or gives the patterns or the registry path |
|
|
68
|
-
| **Outputs** | SKILL.md, context-snippet.md, metadata.json, provenance-map.json, evidence-report.md, extraction-rules.yaml, references/; per brief, the result contract (`create-skill-result-{YYYYMMDD-HHmmss}.json` and its `-latest.json` copy in `{forge_version}`) and one `SKF_CREATE_SKILL_RESULT_JSON` line on stdout, on stderr at a HARD HALT |
|
|
69
|
-
| **Headless** | All gates auto-resolve with default action when `{headless_mode}` is true; each auto-decision reaches the envelope's and the result contract's `headless_decisions`. A caller reads the run's own `SKF_CREATE_SKILL_RESULT_JSON` line. A run that printed no envelope line has no result to read, whatever the process exit code: it was cut off, or could not write its result, and `create-skill-result-latest.json` may then describe an earlier run, so it is never read in its place. |
|
|
68
|
+
| **Outputs** | SKILL.md, context-snippet.md, metadata.json, provenance-map.json, evidence-report.md, extraction-rules.yaml, references/; per brief, the result contract (`create-skill-result-{YYYYMMDD-HHmmss}.json` and its `-latest.json` copy in `{forge_version}`) and one `SKF_CREATE_SKILL_RESULT_JSON` line on stdout, on stderr at a HARD HALT; and under `--batch` the batch summary (`create-skill-batch-{YYYYMMDD-HHmmss}.json` and its `-latest.json` copy in `{skills_output_folder}/_batch/`) |
|
|
69
|
+
| **Headless** | All gates auto-resolve with default action when `{headless_mode}` is true; each auto-decision reaches the envelope's and the result contract's `headless_decisions`. A caller reads the run's own `SKF_CREATE_SKILL_RESULT_JSON` line. A run that printed no envelope line has no result to read, whatever the process exit code: it was cut off, or could not write its result, and `create-skill-result-latest.json` may then describe an earlier run, so it is never read in its place. Under `--batch`, a caller reads the batch's `batch_summary` event and the summary file its `summary_path` names, never `create-skill-batch-latest.json` in its place: that copy holds the last batch summary written, which may be another batch's. A printed `batch_summary` event stays the batch's result even if a halt line follows it. A batch that printed no `batch_summary` event has no batch result, whatever the process exit code: a halt line of its own (phase `batch-mode`) says why it stopped, and with no halt line it was cut off or could not print its halt line. A target it printed a `done` or `fail` event for ended as that event says, and a brief the plan listed as not compiled ended failed with its message; a target with a `start` event and no end event has no result, and any other target with no event did not run in this session. |
|
|
70
70
|
| **Exit codes** | 0 for a finished brief; at a HARD HALT 2 (the brief is missing, malformed or invalid), 3 (no forge config, source, helper or documentation), 4 (a write failed), 5 (a refused or broken package), 6 (the user chose to refine the scope). A `--batch` run exits with the highest code among its briefs, 0 when every brief finished |
|
|
71
71
|
|
|
72
72
|
## On Activation
|
|
@@ -50,10 +50,10 @@ persistent_facts = [
|
|
|
50
50
|
# run a skill such as test-skill or export-skill. It is called as:
|
|
51
51
|
# <on_complete> --result-path=<{forge_version}/create-skill-result-latest.json>
|
|
52
52
|
# Under --batch it runs once per brief, each time with that brief's result
|
|
53
|
-
# file; a brief that halts, or whose result file
|
|
54
|
-
# call. Useful for a Slack or email notifier,
|
|
55
|
-
# or a git-add wrapper. A failure is reported
|
|
56
|
-
# fails the brief.
|
|
53
|
+
# file; a brief that halts, or whose result file or its -latest copy was
|
|
54
|
+
# not written, gets no call. Useful for a Slack or email notifier,
|
|
55
|
+
# dashboard ingest, a CI script or a git-add wrapper. A failure is reported
|
|
56
|
+
# on its own line and never fails the brief.
|
|
57
57
|
#
|
|
58
58
|
# Empty string = no-op (default).
|
|
59
59
|
|
|
@@ -78,4 +78,6 @@ The helper counts the briefs, sets the batch status and exit code, writes the su
|
|
|
78
78
|
|
|
79
79
|
Then load, read in full and execute `{healthCheckStepFile}`: the health check runs once per batch, here. The batch ends with the event's `exit_code`: 0 when every brief finished, else the highest exit code among the briefs that failed.
|
|
80
80
|
|
|
81
|
-
|
|
81
|
+
**Reading a batch.** Under `--batch`, a caller reads the batch's `batch_summary` event and the summary file its `summary_path` names, never `create-skill-batch-latest.json` in its place: that copy holds the last batch summary written, which may be another batch's. The event comes before the health check, so a caller reads it from the session's output stream (`claude -p --output-format stream-json`) or transcript, not from the final message. A printed `batch_summary` event stays the batch's result even if a halt line follows it. A batch that printed no `batch_summary` event has no batch result, whatever the process exit code: a halt line of its own (phase `batch-mode`) says why it stopped, and with no halt line it was cut off or could not print its halt line. A target it printed a `done` or `fail` event for ended as that event says, and a brief the plan listed as not compiled ended failed with its message; a target with a `start` event and no end event has no result, and any other target with no event did not run in this session. When the event's `summary_path` is null, the per-batch file was not written (its `summary_write_failed` warning says why): the event and `{batch_dir}/batch.jsonl`, which the helper then keeps, are the batch's result, never `create-skill-batch-latest.json`. A `summary_write_failed` warning beside a set `summary_path` means only the `-latest` copy failed: the file it names is still the batch's result. After a cut-off batch, run `--batch` again with the same briefs: while the batch has no summary, it finds the batch folder again and goes on with the brief it was working on (Execution, above). If `summarize` already ran but its event never reached the caller, `start` begins a new batch from the first brief, and that batch's summary is the per-batch file in `{skills_output_folder}/_batch/` whose `input_briefs` match.
|
|
82
|
+
|
|
83
|
+
If `summarize` exits non-zero (a brief is still pending, or the batch file is gone or unreadable), the health check does not run: display "**The batch stopped:** {its `message`}" and **HARD HALT** (exit code 4, `write-failed`, phase `batch-mode`; emit with nothing to stage in, as §1 does, with the reason "The batch stopped: <its message>"). The summary files may be missing then, but each brief's own result contract is in its `{forge_version}`, and `create-skill-batch-latest.json` may be another batch's, so it is never read in its place.
|
|
@@ -126,13 +126,18 @@ The evidence report in `outputs` carries the `## Auto-Decisions` audit table: a
|
|
|
126
126
|
uv run {resolveCustomizationHelper} --skill {skill-root} --project-root {project-root} --key workflow.on_complete
|
|
127
127
|
```
|
|
128
128
|
|
|
129
|
-
When `{onCompleteCommand}` is non-empty and the emitter wrote this brief's result contract (its `emit` exited 0), run it as a shell command from `{project-root}`, for this brief:
|
|
129
|
+
When `{onCompleteCommand}` is non-empty and the emitter wrote this brief's result contract (its `emit` exited 0, the line's `result_path` is not null and its `warnings` hold no `result_file_write_failed` entry naming `create-skill-result-latest.json`), run it as a shell command from `{project-root}`, for this brief:
|
|
130
130
|
|
|
131
131
|
```bash
|
|
132
132
|
{onCompleteCommand} --result-path={forge_version}/create-skill-result-latest.json
|
|
133
133
|
```
|
|
134
134
|
|
|
135
|
-
The hook runs after the contract, so a git-add, registry registration or notifier sees a complete package. A hook failure never fails the brief, since the skill is written and its result contract is final: display "on_complete failed for `{name}`: {its exit code and first stderr line}" and go on. When
|
|
135
|
+
The hook runs after the contract, so a git-add, registry registration or notifier sees a complete package. A hook failure never fails the brief, since the skill is written and its result contract is final: display "on_complete failed for `{name}`: {its exit code and first stderr line}" and go on. When `{onCompleteCommand}` is non-empty but the hook does not run, `--result-path` would name a missing file or an earlier compile's: skip the hook and display the line for its case:
|
|
136
|
+
|
|
137
|
+
- No `emit` succeeded (no path in `{emitEnvelopeProbeOrder}` exists, or `emit` failed twice): "on_complete skipped for `{name}`: no result contract was written ({the reason})".
|
|
138
|
+
- `emit` exited 0, but the line's `result_path` is null (the per-run record was not written) or a `result_file_write_failed` warning names `create-skill-result-latest.json` (the copy alone failed): "on_complete skipped for `{name}`: `create-skill-result-latest.json` was not written".
|
|
139
|
+
|
|
140
|
+
When `{onCompleteCommand}` is empty (bundled default), skip the invocation entirely.
|
|
136
141
|
|
|
137
142
|
**Remove the run folder.** The brief finished, so outside `--batch` delete `{run_dir}` (`rm -rf "{run_dir}"`); under `--batch`, batch-mode.md §3 reads the staged `result-context.json` there and removes the folder itself. A halted brief keeps its folder, with the decisions it recorded and its staged payloads.
|
|
138
143
|
|
|
@@ -221,7 +221,7 @@ npx skill-check check <staging-skill-dir> --format json
|
|
|
221
221
|
|
|
222
222
|
Record any security warnings in evidence-report. Security findings are advisory — they do not block artifact generation. If the full validation re-run produces a different quality score than section 2, update the evidence-report with the newer score.
|
|
223
223
|
|
|
224
|
-
**If security scan fails due to missing SNYK_TOKEN
|
|
224
|
+
**If security scan fails due to missing SNYK_TOKEN** (skill-check exits non-zero and its stderr names `SNYK_TOKEN`):
|
|
225
225
|
|
|
226
226
|
Display: "Security scan requires a Snyk Enterprise API token ([docs](https://docs.snyk.io/snyk-api/authentication-for-api)). Set `SNYK_TOKEN=your-token` in environment or `.env`, then re-run [SF] Setup Forge. Without Enterprise, use `--no-security-scan` to skip. Security scanning is optional and does not block skill compilation."
|
|
227
227
|
|
|
@@ -120,15 +120,23 @@ Record format violations as **WARNING** findings.
|
|
|
120
120
|
|
|
121
121
|
### 9. Security Scan (if skill-check available)
|
|
122
122
|
|
|
123
|
-
Run: `npx skill-check check <skill-dir> --format json` (security scan enabled by default).
|
|
123
|
+
Run: `npx skill-check check <skill-dir> --format json > "{run_dir}/skill-check.json" 2> "{run_dir}/skill-check.stderr"; echo "skill-check exited $?"` (security scan enabled by default), so stdout, stderr and the exit code are read apart.
|
|
124
124
|
|
|
125
|
-
|
|
125
|
+
**If `{run_dir}/skill-check.json` does not start with a JSON object**, skill-check stopped before it validated (exit 2 on a config, option or plugin error): record no finding, only the §10 note "security scan skipped: skill-check failed ({the stderr line that names the cause, or its last line})". Otherwise stdout can go on after the JSON (a `Wrote <path>` line when the project config sets a report path), so read only the JSON object at its start.
|
|
126
126
|
|
|
127
|
-
**
|
|
127
|
+
**Read the JSON, not the exit code.** skill-check exits 1 when `summary.errorCount` is above 0, when `--strict` or `--fail-on-warning` is passed and `summary.warningCount` is above 0 (this call passes neither), or when the scan exits non-zero, and 2 when no scan runner starts, so a non-zero exit is not a security finding. skill-check 1.2.0 puts no security key in the JSON. The scan exited non-zero, which without `--ci` means it did not finish, when skill-check exits 1 although `summary.errorCount` is 0: stderr is then the scan's last lines (its tail), even when blank. On exit 0, stderr is never a scan tail (npm and Node warnings can be there); when exit 1 has another cause, stderr is a scan tail only when it holds the scanner's own lines, such as its token message or a finding code. A finding code is `[`, a capital letter, three digits and `]` (such as `[E001]`), the codes skill-check's own scan summary counts. Read the scan from stderr and the exit code, the first that applies:
|
|
128
|
+
|
|
129
|
+
- **Could not run:** skill-check exits 2, the call stopped before it exited, or the scan exited non-zero and its tail carries no finding code. Record no finding, only a note in the §10 validation results: "security scan skipped: SNYK_TOKEN not configured" when the tail says to set the `SNYK_TOKEN` environment variable (the scanner prints "To use Agent Scan, set the SNYK_TOKEN environment variable."), otherwise "security scan skipped: scanner could not run ({the stderr line that names the cause, or its last line})", or "security scan skipped: scanner exited non-zero with no output" when stderr is blank.
|
|
130
|
+
- **Findings:** `security[]` holds findings, or the tail carries finding codes.
|
|
131
|
+
- **Neither:** record the §10 note "security scan exited 0; skill-check 1.2.0 does not pass the scan's findings on". Without `--ci`, agent-scan also exits 0 when its analysis fails (a 401 from a bad token, a 429, a network error), so this does not show that the scan found nothing (#892).
|
|
132
|
+
|
|
133
|
+
Record security findings as advisory **WARNING** findings: they do not block the report.
|
|
134
|
+
|
|
135
|
+
**If unavailable:** Skip, with the note "security scan skipped: skill-check unavailable" in the §10 validation results.
|
|
128
136
|
|
|
129
137
|
### 10. Display Validation Results
|
|
130
138
|
|
|
131
|
-
Report the validation outcome. If all checks passed, state so and name what was verified: file presence (`{count}/{count}`), SKILL.md structure, metadata.json fields, the `{lib_count}` library + `{pair_count}` integration reference files, and complete confidence-tier coverage. If there were findings, report the `{warning_count}` finding(s)
|
|
139
|
+
Report the validation outcome. If all checks passed, state so and name what was verified: file presence (`{count}/{count}`), SKILL.md structure, metadata.json fields, the `{lib_count}` library + `{pair_count}` integration reference files, and complete confidence-tier coverage. If there were findings, report the `{warning_count}` finding(s), each with severity, description, and file path, plus files present/expected and warning/error counts; when errors include missing files, note this may indicate a write failure in step 07. In both cases, list each §9 security scan note.
|
|
132
140
|
|
|
133
141
|
### 11. Auto-Proceed to Next Step
|
|
134
142
|
|
|
@@ -53,7 +53,7 @@ These rules apply to every step in this workflow:
|
|
|
53
53
|
| **Overrides** | `--language-hint`, `--scope-hint`, `--description`, `--exports`, `--skip-snippet`, `--no-active-pointer`, `--batch <file>`, `--fail-fast`: see On Activation step 4 |
|
|
54
54
|
| **Gates** | step 1: target input, ambiguous package name [C/n/U/X] (if another registry also holds the name; headless keeps the first registry's pick), multi-language disambiguation [C/n/A] (pick any detected language); step 2: ecosystem match [P/I/A] (none until agentskills.io has a registry API); step 3: repo-shape [C/A] + zero-exports rescue [R/P/A]; step 4: review [C/E/S/Q]; step 5: overwrite [Y/N] |
|
|
55
55
|
| **Outputs** | SKILL.md, context-snippet.md, metadata.json, active pointer, result contract (timestamped + `-latest` copy), one `SKF_QUICK_SKILL_RESULT_JSON` line (stdout when the run finishes, stderr at a HARD HALT), and under `--batch` the batch summary. Snippet and active pointer can be skipped per overrides. |
|
|
56
|
-
| **Headless** | All gates auto-resolve with default action when `{headless_mode}` is true; each auto-decision is recorded in the envelope's `headless_decisions`. A caller reads the run's own `SKF_QUICK_SKILL_RESULT_JSON` line. A run that printed no envelope line has no result to read, whatever the process exit code: it was cut off, or could not write its result, and the result contract's `-latest` copy may then describe an earlier run, so it is never read in its place. |
|
|
56
|
+
| **Headless** | All gates auto-resolve with default action when `{headless_mode}` is true; each auto-decision is recorded in the envelope's `headless_decisions`. A caller reads the run's own `SKF_QUICK_SKILL_RESULT_JSON` line. A run that printed no envelope line has no result to read, whatever the process exit code: it was cut off, or could not write its result, and the result contract's `-latest` copy may then describe an earlier run, so it is never read in its place. Under `--batch`, a caller reads the batch's `batch_summary` event and the summary file its `summary_path` names, never the batch summary's `-latest` copy in its place: that copy holds the last batch summary written, which may be another batch's. A printed `batch_summary` event stays the batch's result even if a halt line follows it. A batch that printed no `batch_summary` event has no batch result, whatever the process exit code: a halt line of its own (phase `batch-mode`, or `on-activation`) says why it stopped, and with no halt line it was cut off or could not print its halt line. A target it printed a `done` or `fail` event for ended as that event says; a target with a `start` event and no end event has no result, and a target with no event did not run in this session. |
|
|
57
57
|
| **Exit codes** | See `references/halt-contract.md`: the exit-code map and the envelope every HARD HALT emits, except On Activation step 1's check for `python3` and `uv`, which prints none |
|
|
58
58
|
|
|
59
59
|
## On Activation
|
|
@@ -93,6 +93,8 @@ uv run {quickBatchHelper} summarize --run-dir "{batch_dir}" --output-dir "{batch
|
|
|
93
93
|
|
|
94
94
|
The helper counts the targets, sets the batch status and exit code, writes the summary files below, appends the final status to the batch file (and removes the batch run folder when no target failed), and prints the `batch_summary` event on stderr: display it verbatim. Then print step 7's `start` event and load, read in full and execute `{healthCheckStepFile}`: the health check runs once per batch, here. The batch ends with the event's `exit_code`.
|
|
95
95
|
|
|
96
|
+
**Reading a batch.** Under `--batch`, a caller reads the batch's `batch_summary` event and the summary file its `summary_path` names, never `quick-skill-batch-latest.json` in its place: that copy holds the last batch summary written, which may be another batch's. The event comes before the health check, so a caller reads it from the session's output stream (`claude -p --output-format stream-json`) or transcript, not from the final message. A printed `batch_summary` event stays the batch's result even if a halt line follows it. A batch that printed no `batch_summary` event has no batch result, whatever the process exit code: a halt line of its own (phase `batch-mode`, or `on-activation`) says why it stopped, and with no halt line it was cut off or could not print its halt line. A target it printed a `done` or `fail` event for ended as that event says; a target with a `start` event and no end event has no result, and a target with no event did not run in this session. When the event's `summary_path` is null, the per-batch file was not written (its `summary_write_failed` warning says why): the event and `{batch_dir}/batch.jsonl`, which the helper then keeps, are the batch's result, never `quick-skill-batch-latest.json`. A `summary_write_failed` warning beside a set `summary_path` means only the `-latest` copy failed: the file it names is still the batch's result. After a cut-off batch, run the batch again: each run takes a new run folder (SKILL.md On Activation step 1), so it starts over from the first target.
|
|
97
|
+
|
|
96
98
|
## Batch summary contract
|
|
97
99
|
|
|
98
100
|
`summarize` writes the batch summary at:
|
|
@@ -112,23 +112,27 @@ Check:
|
|
|
112
112
|
Run: `npx skill-check -h`
|
|
113
113
|
|
|
114
114
|
- If succeeds (returns usage information): Continue to automated validation (section 4)
|
|
115
|
-
- If fails (command not found or error): skip to the frontmatter validator in section 4, and log "security scan skipped: skill-check unavailable" in the validation results
|
|
115
|
+
- If fails (command not found or error): skip to the frontmatter validator in section 4, bind `{security_result}` ← `skipped`, and log "security scan skipped: skill-check unavailable" in the validation results
|
|
116
116
|
|
|
117
117
|
### 4. Validate SKILL.md via skill-check (if available)
|
|
118
118
|
|
|
119
|
-
**If `npx skill-check` is available**, run validation and its security scan in one call against the skill package written in section 2. `--fix` can also rewrite the frontmatter `description` the user approved in step 4, which would leave SKILL.md out of step with `metadata.json`, so the call runs inside the description guard. Resolve `{descriptionGuardHelper}` ← first existing path in `{descriptionGuardProbeOrder}`, then run the three commands in order:
|
|
119
|
+
**If `npx skill-check` is available**, run validation and its security scan in one call against the skill package written in section 2. `--fix` can also rewrite the frontmatter `description` the user approved in step 4, which would leave SKILL.md out of step with `metadata.json`, so the call runs inside the description guard. Resolve `{descriptionGuardHelper}` ← first existing path in `{descriptionGuardProbeOrder}`, then run the three commands in order. `verify-restore` runs whatever skill-check exits; note that exit code:
|
|
120
120
|
|
|
121
121
|
```bash
|
|
122
122
|
uv run {descriptionGuardHelper} capture "{skill_package}/SKILL.md" > "{run_dir}/description-guard.json"
|
|
123
|
-
npx skill-check check "{skill_package}" --fix --format json
|
|
123
|
+
npx skill-check check "{skill_package}" --fix --format json > "{run_dir}/skill-check.json" 2> "{run_dir}/skill-check.stderr"; echo "skill-check exited $?"
|
|
124
124
|
uv run {descriptionGuardHelper} verify-restore "{skill_package}/SKILL.md" --captured-file "{run_dir}/description-guard.json"
|
|
125
125
|
```
|
|
126
126
|
|
|
127
|
+
**If no "skill-check exited" line came back**, the shell tool stopped the block before skill-check finished (a cold `uvx mcp-scan` install can outlast its timeout): run the `verify-restore` command on its own before reading `{run_dir}/skill-check.json`, so a description `--fix` rewrote is restored.
|
|
128
|
+
|
|
127
129
|
**Guard outputs.** `capture` stages the approved description in `{run_dir}/description-guard.json` and `verify-restore` reads it there, so it never passes through a shell string. Bind `{guard_restored}` ← `restored` and `{guard_diff_kind}` ← `diff_kind` from `verify-restore`. When `{guard_restored}` is true, the approved description is back in SKILL.md, matching `metadata.json`: record the validation note "description restored after `skill-check --fix` ({guard_diff_kind})", which §6 lists with the auto-fixed issues. `verify-restore` refuses an empty or whitespace-only captured description, and a file it cannot read (exit 1, SKILL.md untouched): re-run it with `--captured-file "{skill_package}/metadata.json"`, whose `description` step 4 rendered, never with an empty value, and if that exits 1 too, record a high-severity issue: the guard could not check the description. When it exits 2 with JSON on stdout (the JSON carries `restore_error`), the restore could not be written, so record a high-severity issue: SKILL.md's description no longer matches `metadata.json`.
|
|
128
130
|
|
|
129
|
-
**If no `{descriptionGuardProbeOrder}` path exists**, never run `--fix` unguarded. Run `npx skill-check check "{skill_package}" --format json` instead, which validates and scans without fixing, and log the validation issue "description guard unavailable (`skf-description-guard.py` missing): skill-check ran without `--fix`".
|
|
131
|
+
**If no `{descriptionGuardProbeOrder}` path exists**, never run `--fix` unguarded. Run `npx skill-check check "{skill_package}" --format json` instead, which validates and scans without fixing, with the same capture (`> "{run_dir}/skill-check.json" 2> "{run_dir}/skill-check.stderr"`, its exit code noted), and log the validation issue "description guard unavailable (`skf-description-guard.py` missing): skill-check ran without `--fix`".
|
|
132
|
+
|
|
133
|
+
**If `{run_dir}/skill-check.json` does not start with a JSON object**, skill-check stopped before it validated (exit 2 on a config, option or plugin error): log "skill-check failed: {the stderr line that names the cause, or its last line}" and "security scan skipped: skill-check failed ({that line})", keep `{quality_score}` null, bind `{security_result}` ← `skipped`, and run the frontmatter validator below, as when skill-check is not available. Otherwise stdout can go on after the JSON (a `Wrote <path>` line when the project config sets a report path), so read only the JSON object at its start.
|
|
130
134
|
|
|
131
|
-
**Parse JSON
|
|
135
|
+
**Parse the JSON in `{run_dir}/skill-check.json`** to extract:
|
|
132
136
|
- `scores[].score` — overall score (0-100); match the entry by `relativePath`/`skillId`
|
|
133
137
|
- `diagnostics[]` — remaining issues after auto-fix
|
|
134
138
|
- `fixed[]` — issues automatically corrected
|
|
@@ -136,6 +140,12 @@ uv run {descriptionGuardHelper} verify-restore "{skill_package}/SKILL.md" --capt
|
|
|
136
140
|
|
|
137
141
|
Bind `{quality_score}` ← that score, and record the remaining diagnostics and security findings as validation issues. `{quality_score}` is null when skill-check did not run.
|
|
138
142
|
|
|
143
|
+
**Read the JSON, not the exit code.** skill-check prints its JSON before the security scan runs, then exits 1 when `summary.errorCount` is above 0, when `--strict` or `--fail-on-warning` is passed and `summary.warningCount` is above 0 (this call passes neither), or when the scan exits non-zero, and 2 when no scan runner starts, so `summary.errorCount` alone is the validation verdict. skill-check 1.2.0 puts no security key in the JSON. The scan exited non-zero, which without `--ci` means it did not finish, when skill-check exits 1 although `summary.errorCount` is 0: stderr is then the scan's last lines (its tail), even when blank. On exit 0, stderr is never a scan tail (npm and Node warnings can be there); when exit 1 has another cause, stderr is a scan tail only when it holds the scanner's own lines, such as its token message or a finding code. A finding code is `[`, a capital letter, three digits and `]` (such as `[E001]`), the codes skill-check's own scan summary counts. Bind `{security_result}` from `{run_dir}/skill-check.stderr` and the exit code, the first that applies:
|
|
144
|
+
|
|
145
|
+
- `skipped`: §3 found no skill-check (§3's log is the note), skill-check exits 2, the call stopped before it exited, or the scan exited non-zero and its tail carries no finding code. Log in the validation results "security scan skipped: SNYK_TOKEN not configured" when the tail says to set the `SNYK_TOKEN` environment variable (the scanner prints "To use Agent Scan, set the SNYK_TOKEN environment variable."), otherwise "security scan skipped: scanner could not run ({the stderr line that names the cause, or its last line})", or "security scan skipped: scanner exited non-zero with no output" when stderr is blank.
|
|
146
|
+
- `warn`: `security[]` holds findings, or the tail carries finding codes. Record each finding as a validation issue.
|
|
147
|
+
- `pass`: otherwise. The scan exited 0 and skill-check reported no finding. Without `--ci`, agent-scan also exits 0 when its analysis fails (a 401 from a bad token, a 429, a network error), so this does not show that the scan found nothing (#892).
|
|
148
|
+
|
|
139
149
|
**If skill-check is not available**, run the shared frontmatter validator. Resolve `{frontmatterValidator}` from `{frontmatterValidatorProbeOrder}`; first existing path wins. If no candidate exists, nothing can check the frontmatter: HARD HALT with **exit code 3 (resolution-failure)**, in interactive mode too: "**`{skill_package}` was written but not validated.** skill-check is unavailable and SKF's frontmatter validator (`skf-validate-frontmatter.py`) is missing from `{project-root}/_bmad/skf/shared/scripts/`, so re-install SKF, then re-run." Stage `{"phase": "write-and-validate", "halt_reason": "resolution-failure", "reason": "The skill was written but not validated: the frontmatter validator is missing.", "skill_package": "{skill_package}", "outputs": {<each deliverable §2 wrote, as "metadata", "skill_md" or "context_snippet" with its path>}, "details": {"cause": "frontmatter-validator-missing"}}` as `{run_dir}/halt.json` and run `uv run {emitEnvelopeHelper} emit-halt --workflow skf-quick-skill --run-dir "{run_dir}" --result-dir "{skill_package}" --target stderr < "{run_dir}/halt.json"`.
|
|
140
150
|
|
|
141
151
|
```bash
|
|
@@ -160,7 +170,7 @@ The validator emits JSON with `result` (PASS/FAIL), `validation.skill_md.body[]`
|
|
|
160
170
|
|
|
161
171
|
"**Validation complete:**
|
|
162
172
|
|
|
163
|
-
**SKILL.md:** {pass/issues found} (quality score: {
|
|
173
|
+
**SKILL.md:** {pass/issues found}{ (quality score: {quality_score}/100), only when `{quality_score}` is not null}
|
|
164
174
|
{list any issues}
|
|
165
175
|
{list any auto-fixed issues, and the §4 description-guard note when the guard restored the description}
|
|
166
176
|
|
|
@@ -170,8 +180,8 @@ The validator emits JSON with `result` (PASS/FAIL), `validation.skill_md.body[]`
|
|
|
170
180
|
**metadata.json:** {pass/issues found}
|
|
171
181
|
{list any issues}
|
|
172
182
|
|
|
173
|
-
**Security:** {
|
|
174
|
-
{list any security findings}
|
|
183
|
+
**Security:** {security_result}
|
|
184
|
+
{list any security findings, or the §4 skipped note; for `pass`: "scan exited 0; skill-check 1.2.0 does not pass the scan's findings on"}
|
|
175
185
|
|
|
176
186
|
**Overall:** {pass / N issues found}
|
|
177
187
|
|
|
@@ -180,7 +190,7 @@ These issues are advisory for community-tier skills: fix them in the written fil
|
|
|
180
190
|
|
|
181
191
|
**Proceeding to finalize...**"
|
|
182
192
|
|
|
183
|
-
Set `validation_result` with pass/fail status, quality score, and issues list, and `{validation_issues}` ← `{"skill_md": <n>, "context_snippet": <n>, "metadata": <n>, "security": <n>}`, each `<n>` the number of entries the validators returned for that heading: `skill_md` counts skill-check's `diagnostics[]` (without skill-check, the frontmatter validator's `issues[]`) plus `validation.skill_md.body[]`; `context_snippet` counts `validation.context_snippet.issues[]`; `metadata` counts `validation.metadata.issues[]`; `security` counts skill-check's `security[]
|
|
193
|
+
Set `validation_result` with pass/fail status, quality score, and issues list, and `{validation_issues}` ← `{"skill_md": <n>, "context_snippet": <n>, "metadata": <n>, "security": <n>}`, each `<n>` the number of entries the validators returned for that heading: `skill_md` counts skill-check's `diagnostics[]` (without skill-check, the frontmatter validator's `issues[]`) plus `validation.skill_md.body[]`; `context_snippet` counts `validation.context_snippet.issues[]`; `metadata` counts `validation.metadata.issues[]`; `security` counts skill-check's `security[]` (or, without one, the finding codes §4 read from the tail, each occurrence one finding, as skill-check's own scan summary counts them). A check that did not run counts 0, and `fixed[]` and the guard note count nothing. Step 6 §3's summary carries `{quality_score}` and `{validation_issues}`.
|
|
184
194
|
|
|
185
195
|
### 7. Auto-Proceed to Finalize
|
|
186
196
|
|
|
@@ -86,13 +86,13 @@ uv run {coverageInputsScript} floor --skill-dir "{resolved_skill_package}" --out
|
|
|
86
86
|
|
|
87
87
|
Exit 1 printed its error on stderr: HALT with it (`halt_reason: "helper-failed"`, phase `coverage-check:inventory`).
|
|
88
88
|
|
|
89
|
-
Delegate the reading of the skill under test to a subagent, given `{run_dir}` and the path to SKILL.md and to its `references/` directory when one exists: it reads SKILL.md, and every `references/*.md` file too when SKILL.md's `## Full` headings are absent or stubs. It is given `{run_dir}/inventory-floor.json` too, and lists each name there with an origin in a file it reads, choosing its kind from the name's `kindLabel`, the skill's own label, when it has one. For each export two or more of the files it reads document, it compares the SKILL.md body with each reference file, and two reference files with each other only when the body does not settle them (it does not document the export, or agrees with both), by the rules of `{signatureRulesFile}`, whose absolute path (under `{skill-root}`) it is given: parameters, the return type, and descriptions that contradict (brief against detailed is fine). That file alone says what a form may leave out. It writes only this JSON, with no prose and no commentary, to `{run_dir}/inventory-response.txt` with the Write tool, never a shell, and returns that path, or returns the JSON itself when it cannot write files. The JSON's shape and the contract the validator checks are in `{inventoryContractFile}`, whose absolute path (under `{skill-root}`) the subagent is given.
|
|
89
|
+
Delegate the reading of the skill under test to a subagent, given `{run_dir}` and the path to SKILL.md and to its `references/` directory when one exists: it reads SKILL.md, and every `references/*.md` file too when SKILL.md's `## Full` headings are absent or stubs. It is given `{run_dir}/inventory-floor.json` too, and lists each name there with an origin in a file it reads, choosing its kind from the name's `kindLabel`, the skill's own label, when it has one. An origin's `signature` is that place's entry, its non-blank `params` and `return_type` kept as given; a blank one is the reader's. For each export two or more of the files it reads document, it compares the SKILL.md body with each reference file, and two reference files with each other only when the body does not settle them (it does not document the export, or agrees with both), by the rules of `{signatureRulesFile}`, whose absolute path (under `{skill-root}`) it is given: parameters, the return type, and descriptions that contradict (brief against detailed is fine). That file alone says what a form may leave out. It writes only this JSON, with no prose and no commentary, to `{run_dir}/inventory-response.txt` with the Write tool, never a shell, and returns that path, or returns the JSON itself when it cannot write files. The JSON's shape and the contract the validator checks are in `{inventoryContractFile}`, whose absolute path (under `{skill-root}`) the subagent is given.
|
|
90
90
|
|
|
91
91
|
When the subagent returned the JSON rather than the path, or the file is absent or empty afterwards, save its response, as it came back, to `{run_dir}/inventory-response.txt` with the Write tool. The documented inventory is the one §1a validates: do not load SKILL.md or the reference files into the parent context. Without subagents, the parent loads `{inventoryContractFile}` and builds the same JSON from the same files and `inventory-floor.json` and saves it to that file; §1a's checks still run on it.
|
|
92
92
|
|
|
93
93
|
#### 1a. Parent-Side Schema Validation + Spot-Check
|
|
94
94
|
|
|
95
|
-
Before anything consumes the inventory, `{validateInventoryScript}` checks it against the §1 contract (a wrapping fence stripped), spot-checks sampled names against SKILL.md and the subagent's listed reference files, and refuses it when it leaves out a floor name those files write
|
|
95
|
+
Before anything consumes the inventory, `{validateInventoryScript}` checks it against the §1 contract (a wrapping fence stripped), spot-checks sampled names against SKILL.md and the subagent's listed reference files, and refuses it when it leaves out a floor name those files write, or an origin's `signature`. The look-up is the script's, never a grep: an export named `$state` or `a.b` is neither a shell variable nor a regex.
|
|
96
96
|
|
|
97
97
|
```bash
|
|
98
98
|
uv run {validateInventoryScript} --input "{run_dir}/inventory-response.txt" --skill-package "{resolved_skill_package}" --output "{run_dir}/inventory.json" --floor "{run_dir}/inventory-floor.json"
|
|
@@ -22,6 +22,6 @@ The JSON coverage-check.md §1's reader writes to `{run_dir}/inventory-response.
|
|
|
22
22
|
}
|
|
23
23
|
```
|
|
24
24
|
|
|
25
|
-
The contract `validate-inventory.py` checks (coverage-check.md §1a): each `kind` is one of `function`, `class`, `type`, `constant`, `hook`, `interface`, `method`, `struct`, `enum`, `trait`, `macro` or `adapter`; every entry carries the `description` the skill gives it (an empty string when it gives none); a function or method carries `params` (`""` or `[]` when it takes none) and `return_type` as the skill documents them, and leaves a key out when the skill documents nothing for it. A `method`'s `name` is the bare method name and its `owner` the class or object it is called on (`agents` for `cognee.agents.get`, where the skill documents `agents` as a class; the class for one reached through a property, `client.messages`); `owner` is not validated. A function the skill documents in a module is a `function` named bare, never a `method`, even when it is called through the module's name (`register_sql_connection` for `cognee.tools.register_sql_connection`, where the skill documents `tools` as a module). Each `methods` item starts with the method's name, optionally after `async` (`async next_uuid(dep=None)`). A `const` holding a component (a `forwardRef`, `memo` or other wrapper call, or an alias like `const Tabs = TabsPrimitive.Root`) is a `constant` with the declaration or usage form the skill writes as `usage_signature`; a data constant carries no `usage_signature`, `params` or `return_type`. A name documented in several places is listed once per place, each entry with that place's form (`Card` above), and keeps one `kind` in every place. Where the skill documents several names and states members they share once, in the same place (a table row, a paragraph) or over a table for all its rows (a sentence or caption just above it, or its column header), each name's entry carries that shared text in its `params` (in its `usage_signature`, for a usage form), never in a `methods` item, which still starts with the method's name: the shared text first, then `; `, then the name's own members as the skill writes them, without the name, never the name's own members alone (`Note` above, from a row that documents `Note` and `Task`, each with the base fields `id, scope=Scope()`, as `Note(content, title=None)` and `Task(title, done=False)`). An entry whose signature the skill labels partial (simplified, key parameters, a `...` standing in for omitted members, not rest or variadic syntax; a label in a table's column header, such as `Props (simplified)`, labels each of its rows) carries `signature_form: "partial"`, any other `"full"` or no key: copy the label, never judge the form.
|
|
25
|
+
The contract `validate-inventory.py` checks (coverage-check.md §1a): each `kind` is one of `function`, `class`, `type`, `constant`, `hook`, `interface`, `method`, `struct`, `enum`, `trait`, `macro` or `adapter`; every entry carries the `description` the skill gives it (an empty string when it gives none); a function or method carries `params` (`""` or `[]` when it takes none) and `return_type` as the skill documents them, and leaves a key out when the skill documents nothing for it. A name written with empty parentheses in running text or a description cell is a mention, not a form: its entry leaves `params` and `usage_signature` out (a cell reading "`instrument()` wraps functions"). A description cell is any table cell but an export table's first cell and a cell whose column writes the call or signature, by its role (Signature, Props, Params, Parameters, Arguments, Usage, Syntax, Example and the like); a heading or an export table's first cell that writes `name()` is a form, recorded as written. A `method`'s `name` is the bare method name and its `owner` the class or object it is called on (`agents` for `cognee.agents.get`, where the skill documents `agents` as a class; the class for one reached through a property, `client.messages`); `owner` is not validated. A function the skill documents in a module is a `function` named bare, never a `method`, even when it is called through the module's name (`register_sql_connection` for `cognee.tools.register_sql_connection`, where the skill documents `tools` as a module). Each `methods` item starts with the method's name, optionally after `async` (`async next_uuid(dep=None)`). A `const` holding a component (a `forwardRef`, `memo` or other wrapper call, or an alias like `const Tabs = TabsPrimitive.Root`) is a `constant` with the declaration or usage form the skill writes as `usage_signature`; a data constant carries no `usage_signature`, `params` or `return_type`. A name documented in several places is listed once per place, each entry with that place's form (`Card` above), and keeps one `kind` in every place. A place `inventory-floor.json` gives a `signature` (a cited function declaration, a Signature cell, or a heading or an export table's first cell that writes one call) has that form as its entry, whatever its kind, its non-blank `params` and `return_type` kept as given (a `params` may open with the shared text the next sentence gives, then `; `); a blank one is the reader's, and a form the skill writes only in prose is an entry of its own. Where the skill documents several names and states members they share once, in the same place (a table row, a paragraph) or over a table for all its rows (a sentence or caption just above it, or its column header), each name's entry carries that shared text in its `params` (in its `usage_signature`, for a usage form), never in a `methods` item, which still starts with the method's name: the shared text first, then `; `, then the name's own members as the skill writes them, without the name, never the name's own members alone (`Note` above, from a row that documents `Note` and `Task`, each with the base fields `id, scope=Scope()`, as `Note(content, title=None)` and `Task(title, done=False)`). An entry whose signature the skill labels partial (simplified, key parameters, a `...` standing in for omitted members, not rest or variadic syntax; a label in a table's column header, such as `Props (simplified)`, labels each of its rows) carries `signature_form: "partial"`, any other `"full"` or no key: copy the label, never judge the form.
|
|
26
26
|
|
|
27
27
|
A `cross_check_mismatches` entry gives its `export`, `reference_file`, `reference_line` and an `issue` that names each form by its file's path, and the pair's other side: the SKILL.md body's line as `skill_md_line`, or, for two references, the one that sorts first in codepoint order, by path and then line, as `other_file` and `other_line`, with the one that sorts last as `reference_file` and `reference_line`. Each file is its path in the package, such as `references/api-reference.md`.
|
|
@@ -19,7 +19,7 @@ This table is the single source of truth for classifying a gap. Every stage reco
|
|
|
19
19
|
| Severity | Category | Criteria |
|
|
20
20
|
|----------|-----------------------------|----------|
|
|
21
21
|
| Critical | `signature-mismatch` | Wrong signature: a documented signature that differs from the source in its parameters (name, type, order, optionality) or its return type, as signature-comparison-rules.md reads each form (a component by its props; an annotation either side leaves out is not compared), in any way but those the Low `signature-mismatch` row and the Info `observation` row grade, such as a renamed Python parameter (the script cannot see which parameters a caller must pass by position), an optional parameter left out of a form the skill does not label partial, or a mismatch with no `difference` (coverage-check §2 and §2b) |
|
|
22
|
-
| Critical | `fabricated-signature` | Fabricated signature: a documented export absent at the source line the skill cites for it. Decided only where ast-grep enumerated the exports at the pinned commit: the name is outside that surface, and the cited file is missing or defines no such name (coverage-check §2c) |
|
|
22
|
+
| Critical | `fabricated-signature` | Fabricated signature: a documented export absent at the source line the skill cites for it. Decided only where ast-grep enumerated the exports at the pinned commit: the name is outside that surface, and the cited file is missing or defines no such name, and no item of an export list (`export { L as NAME }`, `export type { ... }`) or of a Python `from X import` clause elsewhere renames to it a name that file declares; any other rename (`export { default as NAME }`, `const NAME = L`, `type NAME = L`, an import then an export, Python `NAME = L`) stays fabricated (coverage-check §2c) |
|
|
23
23
|
| Critical | `broken-reference` | Broken reference (contextual mode): a reference whose target does not exist, such as a file not at the stated path, a skill not in the skills output folder, a type no module declares, or a `scripts/` or `assets/` file the skill names (coherence-check §4) |
|
|
24
24
|
| High | `inaccurate-reference` | Inaccurate reference (contextual mode): the target exists but does not match, such as a type the cited module does not export, a signature that differs, or an integration pattern that does not match the implementation (coherence-check §4) |
|
|
25
25
|
| High | `reference-escape` | A reference whose real path leaves the skill, its source and, for a stack, the skills output folder (coherence-check §4) |
|
|
@@ -45,7 +45,7 @@ This table is the single source of truth for classifying a gap. Every stage reco
|
|
|
45
45
|
| Low | `metadata` | Missing optional metadata or examples |
|
|
46
46
|
| Low | `description` | Description trigger optimization recommended (third-person voice, negative triggers, or keyword coverage gaps), a Tessl Review description suggestion included |
|
|
47
47
|
| Low | `external-validator` | A skill-check diagnostic or a Tessl Review validation finding the run left unresolved (external-validators §5b) |
|
|
48
|
-
| Info | `observation` | Style suggestions, non-blocking observations, and a documented extra: a documented name outside the enumerated surface that is not a fabricated signature and that the source still declares at column 0 (a module so named counts, unless the inventory gives the name a `function`, `hook`, `class`, `interface`, `enum` or `type` kind, no provenance map entry of the name types it a `module` or `package` and the module is not a TS/JS file that is itself the export, with an `export default` or a `module.exports =`; among several declaring files, the one the skill cites on a line naming it), found only in a Python or TS/JS source where classify-stale ran (the Forge, Forge+ or Deep tier, `analysis_confidence` `full`, `workspaceDrift` not `overridden` and a bound provenance map) and `surface.json` has a complete `extraction`: `Documented extra: {name}`, at that line, which update-skill never routes (coverage-check §2c). Also a partial signature: a mismatch the signature judge tags `omitted-optional` (defined in signature-comparison-rules.md, Difference tags) on a form the skill labels partial (simplified, key parameters, a `...` that stands in for omitted members, not rest or variadic syntax), which the inventory marks `signature_form: "partial"`, as score-signatures.py checks: `Partial signature: {name}`, at the declaration, counted as matching in Signature Accuracy (coverage-check §2 and §2b). Also a return annotation the body contradicts: a mismatch the signature judge tags `annotation-contradicted` (defined in signature-comparison-rules.md, Difference tags) whose documented signature states a return type and writes the source's parameters, and whose `evidence_line`, the body line that returns the documented type, comes after the declaration and before the next one and, with the local source, starts with `return` (or is a one-line declaration's own expression body), as score-signatures.py checks: `Annotation contradicted: {name}`, at the declaration, counted as matching in Signature Accuracy, with no change to the skill (the annotation is reported upstream) (coverage-check §2 and §2b) Also a documentation error a Deep-tier run sees that no script checks (an import path whose module does not export the name, a type member that differs from its declaration): `Not checked by script: {name}`, at the documenting line, written by hand (coverage-check §2 and §5b) |
|
|
48
|
+
| Info | `observation` | Style suggestions, non-blocking observations, and a documented extra: a documented name outside the enumerated surface that is not a fabricated signature and that the source still declares at column 0 (a module so named counts, unless the inventory gives the name a `function`, `hook`, `class`, `interface`, `enum` or `type` kind, no provenance map entry of the name types it a `module` or `package` and the module is not a TS/JS file that is itself the export, with an `export default` or a `module.exports =`; among several declaring files, the one the skill cites on a line naming it; an `aliased` name, which an export-list item or a Python `from X import` item renames from a name its cited file declares, at that item's line, a multi-line list's item at its own line), found only in a Python or TS/JS source where classify-stale ran (the Forge, Forge+ or Deep tier, `analysis_confidence` `full`, `workspaceDrift` not `overridden` and a bound provenance map) and `surface.json` has a complete `extraction`: `Documented extra: {name}`, at that line, which update-skill never routes (coverage-check §2c). Also a partial signature: a mismatch the signature judge tags `omitted-optional` (defined in signature-comparison-rules.md, Difference tags) on a form the skill labels partial (simplified, key parameters, a `...` that stands in for omitted members, not rest or variadic syntax), which the inventory marks `signature_form: "partial"`, as score-signatures.py checks: `Partial signature: {name}`, at the declaration, counted as matching in Signature Accuracy (coverage-check §2 and §2b). Also a return annotation the body contradicts: a mismatch the signature judge tags `annotation-contradicted` (defined in signature-comparison-rules.md, Difference tags) whose documented signature states a return type and writes the source's parameters, and whose `evidence_line`, the body line that returns the documented type, comes after the declaration and before the next one and, with the local source, starts with `return` (or is a one-line declaration's own expression body), as score-signatures.py checks: `Annotation contradicted: {name}`, at the declaration, counted as matching in Signature Accuracy, with no change to the skill (the annotation is reported upstream) (coverage-check §2 and §2b) Also a documentation error a Deep-tier run sees that no script checks (an import path whose module does not export the name, a type member that differs from its declaration): `Not checked by script: {name}`, at the documenting line, written by hand (coverage-check §2 and §5b) |
|
|
49
49
|
| Info | `provenance-unverified` | Provenance line not verified: the line-check rules found no definition line, or the recorded line is not a line number (`line-invalid`) (coverage-check §4c) |
|
|
50
50
|
| Info | `migration-section` | Historical migration content in Section 4b with no T2-future annotation (Case 2 of `migration-section-rules.md`) |
|
|
51
51
|
| Info | `multi-denominator` | Multi-denominator reporting: barrel and documented-surface clusters diverge by design (>10% cross-cluster) |
|
|
@@ -6,9 +6,11 @@ Every signature judge compares by these rules: coverage-check.md §1's split-bod
|
|
|
6
6
|
|
|
7
7
|
A form claims only what it writes, and only what it claims is compared. Its parameter list claims each parameter it lists, in that order, and no other: a parameter it leaves out is one it says the export does not take (Difference tags say when that is `omitted-optional`). A type or a return type it does not write claims nothing (Annotations).
|
|
8
8
|
|
|
9
|
+
A name written with empty parentheses in running text or a description cell is a mention, not a form: it claims nothing and is not compared. A description cell is any table cell but an export table's first cell and a cell whose column writes the call or signature, by its role (Signature, Props, Params, Parameters, Arguments, Usage, Syntax, Example and the like); a heading or an export table's first cell that writes `name()` is a form, an empty call (below).
|
|
10
|
+
|
|
9
11
|
Three kinds of form do not claim a complete parameter list:
|
|
10
12
|
|
|
11
|
-
- A call or usage example passes values (literals, variables, `name=value`): `RecursiveSplitter()`, `embed(text, model=m)`, `<Canvas of={X} />`. It is compared as a call: each argument it passes must exist, by position or by name, and every required parameter must be passed; an optional parameter it leaves out is no difference. An empty call `X()` is one too, so it matches a source whose parameters are all optional. A component usage `<X a={x} b="y" />` reads as a list of bare names of the props it sets, and a call `f(a, b)` as its arguments in order, each the parameter at its position (or, written `name=value`, the one it names). A JSX spread (`{...props}`) or children can pass a required prop, and a call's `...args` a required parameter, so neither leaves one out. An entry's `usage_signature` written as a usage is read this way, and each usage in a list of usages is compared as a form of its own, under its entry's `form` index.
|
|
13
|
+
- A call or usage example passes values (literals, variables, `name=value`): `RecursiveSplitter()`, `embed(text, model=m)`, `<Canvas of={X} />`. It is compared as a call: each argument it passes must exist, by position or by name, and every required parameter must be passed; an optional parameter it leaves out is no difference. An empty call `X()` written as a call example (in a heading, an export table's first cell, a call or signature column or a fenced example, never a mention) is one too, so it matches a source whose parameters are all optional. A component usage `<X a={x} b="y" />` reads as a list of bare names of the props it sets, and a call `f(a, b)` as its arguments in order, each the parameter at its position (or, written `name=value`, the one it names). A JSX spread (`{...props}`) or children can pass a required prop, and a call's `...args` a required parameter, so neither leaves one out. An entry's `usage_signature` written as a usage is read this way, and each usage in a list of usages is compared as a form of its own, under its entry's `form` index.
|
|
12
14
|
- An ellipsis (`...` or `…`) in a list stands for members it does not show and claims nothing about them: `(...)` and `(tasks, data, dataset, ...)` leave out no parameter.
|
|
13
15
|
- A list of bare names, with no type, default or `?` (`connect(host, port)`, `({ themes, defaultTheme })`), names parameters without claiming the list is complete: an optional member it leaves out is no difference, a required one is a mismatch.
|
|
14
16
|
|