@softspark/ai-toolkit 4.30.3 → 4.31.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (39) hide show
  1. package/CHANGELOG.md +34 -0
  2. package/README.md +10 -10
  3. package/app/.claude-plugin/plugin.json +1 -1
  4. package/app/agents/code-reviewer.md +14 -0
  5. package/app/agents/frontend-specialist.md +27 -24
  6. package/app/agents/infrastructure-validator.md +2 -2
  7. package/app/claude-app/skills/ai-toolkit-rules/SKILL.md +44 -1
  8. package/app/personas/frontend-lead.md +20 -12
  9. package/app/rules/claude-toolkit-rules.md +4 -1
  10. package/app/rules/edit-discipline.md +37 -0
  11. package/app/skills/app-builder/SKILL.md +37 -6
  12. package/app/skills/claude-toolkit-rules/SKILL.md +37 -0
  13. package/app/skills/design-engineering/SKILL.md +87 -102
  14. package/app/skills/design-engineering/reference/animation-recipes.md +67 -0
  15. package/app/skills/design-engineering/reference/component-craft.md +28 -0
  16. package/app/skills/documentation-standards/SKILL.md +1 -1
  17. package/app/skills/edit-discipline/SKILL.md +50 -0
  18. package/app/skills/git-conventions/SKILL.md +18 -0
  19. package/app/skills/output-mode/SKILL.md +41 -0
  20. package/app/skills/quality-gates/SKILL.md +23 -0
  21. package/app/skills/review/SKILL.md +10 -0
  22. package/kb/procedures/{ecosystem-sync-sop.md → sop-ecosystem-sync.md} +2 -2
  23. package/kb/procedures/{post-release-testing-sop.md → sop-post-release-testing.md} +3 -3
  24. package/kb/procedures/sop-pre-commit.md +96 -0
  25. package/kb/procedures/{release-preparation-sop.md → sop-release.md} +6 -6
  26. package/kb/reference/architecture-overview.md +1 -1
  27. package/kb/reference/copilot-compatibility.md +1 -1
  28. package/kb/reference/licensing.md +2 -2
  29. package/kb/reference/quick-wins-implementation-summary.md +1 -1
  30. package/kb/reference/supported-tools-registry.md +2 -2
  31. package/kb/troubleshooting/plugin-double-load.md +1 -1
  32. package/llms-full.txt +956 -849
  33. package/llms.txt +6 -5
  34. package/manifest.json +1 -1
  35. package/package.json +2 -1
  36. package/scripts/generate_toolkit_rules_skills.py +189 -0
  37. package/scripts/validate.py +1 -1
  38. /package/kb/procedures/{maintenance-sop.md → sop-maintenance.md} +0 -0
  39. /package/kb/procedures/{release-verification-sop.md → sop-release-verification.md} +0 -0
package/llms-full.txt CHANGED
@@ -32,11 +32,12 @@
32
32
  - [Closed: Tool-Output Token Reduction](kb/history/completed/tool-output-token-reduction-closed-20260727.md)
33
33
  - [How-To Guides](kb/howto/README.md)
34
34
  - [Plan: Cloud Security Pack — Multi-Cloud Audit](kb/planning/cloud-security-pack-plan.md)
35
- - [SOP: Ecosystem Sync](kb/procedures/ecosystem-sync-sop.md)
36
- - [SOP: AI Toolkit Maintenance](kb/procedures/maintenance-sop.md)
37
- - [SOP: Post-Release Testing](kb/procedures/post-release-testing-sop.md)
38
- - [SOP: Release Preparation](kb/procedures/release-preparation-sop.md)
39
- - [SOP: Release Verification](kb/procedures/release-verification-sop.md)
35
+ - [SOP: Ecosystem Sync](kb/procedures/sop-ecosystem-sync.md)
36
+ - [SOP: AI Toolkit Maintenance](kb/procedures/sop-maintenance.md)
37
+ - [SOP: Post-Release Testing](kb/procedures/sop-post-release-testing.md)
38
+ - [SOP: Pre-Commit Gate](kb/procedures/sop-pre-commit.md)
39
+ - [SOP: Release Verification](kb/procedures/sop-release-verification.md)
40
+ - [SOP: Release Preparation](kb/procedures/sop-release.md)
40
41
  - [Agents Catalog](kb/reference/agents-catalog.md)
41
42
  - [Anti-Pattern Registry Format](kb/reference/anti-pattern-registry-format.md)
42
43
  - [AI Toolkit Architecture](kb/reference/architecture-overview.md)
@@ -95,6 +96,7 @@
95
96
  - **chaos**: Injects controlled faults for resilience testing on non-prod. Triggers: chaos, fault injection, latency injection, dependency kill, resilience test.
96
97
  - **ci**: Detect/generate/debug CI pipeline config (GitHub Actions, GitLab CI). Triggers: CI setup, build pipeline, GitHub Actions config, debug CI, GitLab CI.
97
98
  - **ci-cd-patterns**: CI/CD: GitHub Actions, GitLab CI, Jenkins, caching, blue-green, canary. Triggers: CI, CD, pipeline, GitHub Actions, workflow YAML, release, canary, rollout.
99
+ - **claude-toolkit-rules**: SoftSpark working agreement: never guess a home directory path, give at least three alternatives, and apply a devil's advocate critique to decisions. Triggers: toolkit, conventions, workflow, alternatives, review.
98
100
  - **clean-code**: Code quality: meaningful names, SRP, DRY, small functions, guard clauses, refactoring. Triggers: clean code, naming, code smell, SRP, DRY, long function, god class, dead code.
99
101
  - **command-creator**: Creates new Claude Code slash commands with frontmatter and validation. Triggers: new slash command, create command, command scaffold.
100
102
  - **commit**: Creates Conventional Commits with pre-commit validation. Triggers: commit, conventional commit, git commit, message.
@@ -115,12 +117,14 @@
115
117
  - **docs**: Generates/updates README, API docs, architecture notes. Triggers: docs, README, API docs, architecture note, documentation.
116
118
  - **documentation-standards**: KB conventions: YAML frontmatter, 10-category taxonomy (reference/howto/procedures/troubleshooting/best-practices/decisions/runbooks/planning/business/templates). Triggers: kb/, SOP, runbook, howto, frontmatter, knowledge base.
117
119
  - **ecommerce-patterns**: E-commerce: cart, checkout, payments (Stripe/Adyen), order state, inventory, promos, tax. Triggers: cart, checkout, SKU, payment, Stripe, Shopify, Medusa, Magento, coupon, refund.
120
+ - **edit-discipline**: Change files with the edit and write tools, never by rewriting them through bash (sed, awk, tee, heredoc, redirection), and show `git diff` before reporting a file-changing task as done. Triggers: edit, write, modify, refactor, patch, fix, diff, review changes.
118
121
  - **evaluate**: Evaluates RAG retrieval and LLM-as-judge metrics (faithfulness, relevancy, context precision). Triggers: measure RAG quality, knowledge gap, RAG eval, golden dataset.
119
122
  - **evolve**: Analyzes agent/skill failures, drafts prompt/permission fixes. Triggers: improve agent, refine skill, system prompt, optimize agent.
120
123
  - **explain**: Explains code/architecture with Mermaid diagrams and sequence flows. Triggers: what does X do, how does Y work, explain code, sequence diagram.
121
124
  - **explore**: Explores codebase structure, stack, and architecture. Triggers: explore codebase, project structure, stack overview, architecture map.
122
125
  - **fix**: Applies targeted fix to known bug/lint error, verifies with same command that surfaced it. Triggers: fix, apply fix, fix bug, fix lint, targeted fix.
123
126
  - **flutter-patterns**: Flutter/Dart: widgets, state mgmt (Riverpod/Bloc), navigation, platform channels. Triggers: Flutter, Dart, widget, Riverpod, Bloc, pubspec, hot reload.
127
+ - **git-conventions**: Conventional Commits only (feat, fix, docs, refactor, test, chore); no AI co-authorship trailer and no AI signature in a commit message. Triggers: commit, git, message, changelog, release, pull request.
124
128
  - **git-mastery**: Advanced Git: rebase, bisect, reflog, cherry-pick, worktrees, LFS. Triggers: rebase, bisect, cherry-pick, reflog, force push, merge conflict, worktree.
125
129
  - **golang-rules**: Go coding rules: style, patterns, security, testing. Triggers: .go, go.mod, go.sum, Gin, Echo, Gorilla, testing, gofmt.
126
130
  - **grill-me**: Stress-tests a plan via Socratic questioning down each decision branch. Triggers: stress-test, grill me, validate assumptions, challenge plan, socratic review.
@@ -147,6 +151,7 @@
147
151
  - **observability-patterns**: Observability: structured logs, metrics (RED/USE), tracing, SLO/SLI. Triggers: logging, metrics, Prometheus, Grafana, OpenTelemetry, trace, monitoring.
148
152
  - **onboard**: Sets up ai-toolkit in a project: symlinks, CLAUDE.md, intent interview. Triggers: onboard, setup project, install ai-toolkit, migrate project.
149
153
  - **orchestrate**: Coordinates multiple specialized agents in parallel. Triggers: orchestrate, multi-agent, parallel agents, coordinate agents.
154
+ - **output-mode**: Answer concisely: lead with the result, no preamble, tables over prose, no trailing restatement of a diff the reader can already see. Triggers: response style, verbosity, summary, explanation, report.
150
155
  - **panic**: Emergency kill switch — halts all agents via lockfile gate. Triggers: panic, stop everything, kill switch, halt agents, agents looping.
151
156
  - **performance-profiling**: Performance: golden signals, p50/p95/p99, flame graphs, load testing. Triggers: performance, slow, latency, p99, flame graph, bottleneck, memory leak.
152
157
  - **persona**: Switches engineering persona at runtime. Triggers: persona, switch role, backend-lead, frontend-lead, devops-eng, junior-dev.
@@ -160,6 +165,7 @@
160
165
  - **prompt-caching-patterns**: Anthropic API prompt caching: TTL, breakpoints, stacking, invalidation, hit rate. Triggers: prompt caching, cache_control, cache breakpoint, cache TTL, hit rate.
161
166
  - **python-rules**: Python coding rules: style, patterns, security, testing. Triggers: .py, .pyi, pyproject.toml, requirements.txt, Pipfile, FastAPI, Django, Flask, pytest, SQLAlchemy, ruff, mypy.
162
167
  - **qa-session**: Interactive QA: user reports bugs conversationally, agent files GitHub issues. Triggers: QA session, report bug, file issue, conversational QA, bug intake.
168
+ - **quality-gates**: Plan before work over an hour, and hold the gates: ruff clean, mypy --strict clean, pytest coverage above 70 percent, no secrets in code. Triggers: quality, lint, mypy, pytest, coverage, gate, definition of done.
163
169
  - **rag-patterns**: RAG: embeddings, chunking, hybrid search (BM25+vector), reranking, CRAG, multi-hop. Triggers: RAG, embedding, pgvector, Qdrant, Pinecone, Weaviate, reranker, semantic search.
164
170
  - **refactor**: Refactors code for quality and maintainability. Triggers: refactor, clean up, restructure, improve code, modernize.
165
171
  - **refactor-plan**: Creates detailed refactor plan with tiny commits via interview, files as GitHub RFC. Triggers: refactor plan, refactoring RFC, incremental refactor, safe steps.
@@ -6929,7 +6935,7 @@ Examples:
6929
6935
 
6930
6936
  ---
6931
6937
 
6932
- ## kb/procedures/ecosystem-sync-sop.md
6938
+ ## kb/procedures/sop-ecosystem-sync.md
6933
6939
 
6934
6940
  ---
6935
6941
  title: "SOP: Ecosystem Sync"
@@ -7186,12 +7192,12 @@ jobs:
7186
7192
 
7187
7193
  - [Supported Tools Registry](../reference/supported-tools-registry.md) — human-readable per-tool breakdown
7188
7194
  - [MCP Editor Compatibility](../reference/mcp-editor-compatibility.md) — MCP-specific adapter table
7189
- - [Maintenance SOP](maintenance-sop.md) — general toolkit upkeep
7190
- - [Release Preparation SOP](release-preparation-sop.md) — run the doctor before tagging
7195
+ - [Maintenance SOP](sop-maintenance.md) — general toolkit upkeep
7196
+ - [Release Preparation SOP](sop-release.md) — run the doctor before tagging
7191
7197
 
7192
7198
  ---
7193
7199
 
7194
- ## kb/procedures/maintenance-sop.md
7200
+ ## kb/procedures/sop-maintenance.md
7195
7201
 
7196
7202
  ---
7197
7203
  title: "SOP: AI Toolkit Maintenance"
@@ -7705,7 +7711,7 @@ What `uninstall` does:
7705
7711
 
7706
7712
  ---
7707
7713
 
7708
- ## kb/procedures/post-release-testing-sop.md
7714
+ ## kb/procedures/sop-post-release-testing.md
7709
7715
 
7710
7716
  ---
7711
7717
  title: "SOP: Post-Release Testing"
@@ -7725,7 +7731,7 @@ actually install, from npm, rather than the working tree.
7725
7731
 
7726
7732
  Sibling procedures exist for `jira-mcp` and `legal-pl-pack`; this is the
7727
7733
  ai-toolkit equivalent. It complements
7728
- [Release Verification](release-verification-sop.md), which checks the toolkit
7734
+ [Release Verification](sop-release-verification.md), which checks the toolkit
7729
7735
  from the maintainer's own installed copy. The difference that matters: this one
7730
7736
  never writes to the maintainer's `~/.claude` or `~/.softspark`.
7731
7737
 
@@ -8019,1201 +8025,1302 @@ print(f'removed {sb} ({n} files)')
8019
8025
 
8020
8026
  ## Related
8021
8027
 
8022
- - [Release Preparation](release-preparation-sop.md) — run before tagging
8023
- - [Release Verification](release-verification-sop.md) — the maintainer-install checks
8028
+ - [Release Preparation](sop-release.md) — run before tagging
8029
+ - [Release Verification](sop-release-verification.md) — the maintainer-install checks
8024
8030
  - [rtk-pack Retirement](../history/completed/rtk-pack-retirement-20260727.md) — what happened the one time this SOP was written and not run
8025
8031
 
8026
8032
  ---
8027
8033
 
8028
- ## kb/procedures/release-preparation-sop.md
8034
+ ## kb/procedures/sop-pre-commit.md
8029
8035
 
8030
8036
  ---
8031
- title: "SOP: Release Preparation"
8037
+ title: "SOP: Pre-Commit Gate"
8032
8038
  category: procedures
8033
8039
  service: ai-toolkit
8034
- tags: [sop, release, version, publish, changelog, semver, provenance, sarif, ecosystem, shellcheck]
8035
- version: "1.15.0"
8036
- created: "2026-04-10"
8040
+ tags: [pre-commit, quality-gate, shellcheck, bats, validation, secrets]
8041
+ version: "1.0.0"
8042
+ created: "2026-09-02"
8037
8043
  last_updated: "2026-09-02"
8038
- description: "Step-by-step checklist for preparing a new ai-toolkit release — ecosystem-sync drift check, version sync, changelog, artifact regeneration, validation, branch CI, and tagging. Run BEFORE every git tag. Includes mandatory Provenance, SARIF, checksum-pin, ShellCheck, licensing, exact-tag assertions, and a green Ubuntu/macOS branch-CI gate before any release tag is created."
8044
+ description: "The local gate to run before every commit to ai-toolkit: staged-file and secret scan, ShellCheck, Python syntax, toolkit validation, and the Bats suite. Runs the same checks CI runs, so main does not go red after a push."
8039
8045
  ---
8040
8046
 
8041
- # SOP: Release Preparation
8047
+ # SOP: Pre-Commit Gate
8042
8048
 
8043
- Complete checklist for preparing a new `@softspark/ai-toolkit` release.
8044
- Run this **before** tagging. After tagging and publishing, run the
8045
- [Release Verification SOP](release-verification-sop.md) to smoke-test.
8049
+ ## Why this exists
8046
8050
 
8047
- **Pipeline:**
8048
- ```
8049
- Ecosystem Sync SOP (drift check + generator updates)
8050
-
8051
- Release Preparation (this SOP)
8052
-
8053
- git tag → CI publish → Release Verification SOP
8054
- ```
8051
+ CI runs on `main` and on pull requests, but it runs *after* the push. Every
8052
+ check below already exists in `.github/workflows/ci.yml`; running them locally
8053
+ first is what keeps `main` green and keeps a release from becoming a debug
8054
+ session. The v4.5.1 postmortem in `sop-release.md` is the case
8055
+ where a hook lint failure published while reddening `main` CI.
8055
8056
 
8056
- **Time:** 10-20 minutes (includes ecosystem sync review)
8057
+ This gate is not a substitute for `sop-release.md`. That SOP gates
8058
+ a *tag*; this one gates a *commit*.
8057
8059
 
8058
- ---
8060
+ ## Prerequisites
8059
8061
 
8060
- ## Quick Checklist (TL;DR)
8062
+ `shellcheck`, `python3` and `bats` must be on `PATH`. The suite is the one
8063
+ check that silently does nothing when its runner is absent, so install it
8064
+ before relying on this gate:
8061
8065
 
8062
8066
  ```bash
8063
- # 0. Ecosystem sync (mandatory for minor/major releases; optional for patch)
8064
- # Full procedure: kb/procedures/ecosystem-sync-sop.md
8065
- python3 scripts/ecosystem_doctor.py --format text > /tmp/eco-report.txt
8066
- cat /tmp/eco-report.txt
8067
- # If drift detected: stop here, follow ecosystem-sync-sop.md Phase 2-4 to
8068
- # classify each drift (A-F), update generators as needed, refresh snapshot,
8069
- # THEN resume this SOP.
8070
- python3 scripts/ecosystem_doctor.py --update # after all drift resolved
8071
-
8072
- # 1. Decide version bump
8073
- # patch (1.4.2 → 1.4.3): bugfix, typo, doc fix
8074
- # minor (1.4.2 → 1.5.0): new feature, new skill, new flag, any ecosystem-class-B/F change
8075
- # major (1.4.2 → 2.0.0): breaking change, any ecosystem-class-D removed path
8076
-
8077
- # 2. Sync version across all files
8078
- python3 scripts/sync_version.py X.Y.Z # if script exists, else manual
8079
-
8080
- # 3. Write CHANGELOG.md entry
8081
- # 4. Regenerate artifacts
8082
- python3 scripts/generate_agents_md.py > AGENTS.md
8083
- python3 scripts/generate_llms_txt.py > llms.txt
8084
- python3 scripts/generate_llms_txt.py --full > llms-full.txt
8085
-
8086
- # 5. Validate + audit + SARIF + shellcheck + test + ecosystem check
8087
- python3 scripts/validate.py --strict && python3 scripts/audit_skills.py --ci && python3 scripts/audit_skills.py --sarif > /tmp/audit.sarif && shellcheck --severity=warning app/hooks/*.sh && npm test
8088
-
8089
- # 5a. Supply-chain standard (v2.8.0+) — non-negotiable
8090
- grep -q -- '--provenance' .github/workflows/publish.yml || { echo "MISSING --provenance"; exit 1; }
8091
- grep -q 'id-token: write' .github/workflows/publish.yml || { echo "MISSING id-token: write"; exit 1; }
8092
- python3 scripts/audit_skills.py --permissions # review Bash/Write/Edit footprint
8093
-
8094
- # 5b. Ecosystem gate — snapshot must be current before tag
8095
- python3 scripts/ecosystem_doctor.py --offline --check || { echo "STALE ecosystem snapshot — re-run doctor"; exit 1; }
8096
-
8097
- # 5c. Licensing gate — SPDX headers, LICENSE, NOTICE, manifest consistency
8098
- npx bats tests/test_licensing.bats || { echo "LICENSING GATE FAILED"; exit 1; }
8099
-
8100
- # 6. Commit, push branch, and wait for the exact commit's full CI
8101
- git add -A && git commit -m "chore: release vX.Y.Z"
8102
- RELEASE_SHA=$(git rev-parse HEAD)
8103
- git push origin main || { echo "FAIL: release commit push failed"; exit 1; }
8104
- RUN_ID=""
8105
- for ATTEMPT in 1 2 3 4 5; do
8106
- RUN_ID=$(gh run list --workflow ci.yml --event push --commit "$RELEASE_SHA" \
8107
- --limit 1 --json databaseId --jq '.[0].databaseId')
8108
- [ -n "$RUN_ID" ] && break
8109
- echo "Waiting for CI run registration ($ATTEMPT/5)" >&2
8110
- [ "$ATTEMPT" -eq 5 ] || sleep 60
8111
- done
8112
- test -n "$RUN_ID" || { echo "FAIL: CI run not found for release commit"; exit 1; }
8113
- gh run watch "$RUN_ID" --exit-status \
8114
- || { echo "FAIL: release commit CI failed"; exit 1; }
8115
- for OS in ubuntu-latest macos-latest; do
8116
- gh run view "$RUN_ID" --json jobs \
8117
- --jq ".jobs[] | select(.name == \"Bats test suite ($OS)\") | .conclusion" \
8118
- | grep -qx success || { echo "FAIL: $OS test job not green"; exit 1; }
8119
- done
8120
- test "$(git rev-parse HEAD)" = "$RELEASE_SHA" || { echo "FAIL: local HEAD changed during CI"; exit 1; }
8121
- git fetch origin main || { echo "FAIL: origin/main refresh failed"; exit 1; }
8122
- test "$(git rev-parse origin/main)" = "$RELEASE_SHA" || { echo "FAIL: origin/main changed during CI"; exit 1; }
8123
-
8124
- # 6a. Only green branch CI authorizes tag creation
8125
- git tag vX.Y.Z "$RELEASE_SHA"
8126
-
8127
- # 6b. Assert the tag before pushing it (v4.19.0 postmortem, Phase 7)
8128
- test "$(git rev-parse vX.Y.Z)" = "$RELEASE_SHA" || { echo "FAIL: tag not on tested release commit"; exit 1; }
8129
- git show --no-patch --format=%s vX.Y.Z | grep -qx "chore: release vX.Y.Z" || { echo "FAIL: tag not on release commit"; exit 1; }
8130
-
8131
- # 6c. Push the single tag by full ref. Never --tags.
8132
- git push origin refs/tags/vX.Y.Z
8067
+ brew install bats-core shellcheck # macOS
8068
+ sudo apt-get install -y bats shellcheck # Debian / Ubuntu
8133
8069
  ```
8134
8070
 
8135
- ---
8071
+ ## The gate
8136
8072
 
8137
- ## Phase 0: Ecosystem Sync (MANDATORY for minor/major)
8073
+ Run all five from the repository root. Each mirrors a CI job.
8138
8074
 
8139
- Before touching version numbers, confirm the toolkit is aligned with the current state of every editor / platform it integrates with. Skipping this phase ships a release whose generators may lag a month-old CLI refactor, a rename of `.cursorrules` to `.cursor/rules/`, or a new hook event we do not yet emit.
8075
+ ```bash
8076
+ # 1. Staged files: secrets, large blobs, commit-type suggestion
8077
+ python3 app/skills/commit/scripts/pre-commit-check.py
8140
8078
 
8141
- **When this phase is mandatory:**
8142
- - Minor release (X.Y.0) — always
8143
- - Major release (X.0.0) — always
8144
- - Patch release (X.Y.Z) — only if the patch touches a generator or install flow
8079
+ # 2. ShellCheck hooks -- the CI job that has published while red before
8080
+ shellcheck --severity=warning app/hooks/*.sh app/plugins/*/hooks/*.sh
8145
8081
 
8146
- **When to skip:** pure doc-only patches, SOP edits, internal refactors that do not touch `scripts/generate_*` or `app/skills/*/SKILL.md`.
8082
+ # 3. Python syntax across every script the toolkit ships
8083
+ python3 -m py_compile scripts/*.py app/skills/*/scripts/*.py
8147
8084
 
8148
- ### 0.1 Run the doctor
8085
+ # 4. Toolkit integrity: agents, skills, registry drift, content quality
8086
+ npm run validate
8149
8087
 
8150
- ```bash
8151
- python3 scripts/ecosystem_doctor.py --format text | tee /tmp/eco-report.txt
8088
+ # 5. The suite. Run it ONCE -- see the single-run discipline below
8089
+ npm test
8152
8090
  ```
8153
8091
 
8154
- Output classifies every registered tool as **Clean**, **Drift**, or **Errored**.
8092
+ A non-zero exit from any of them is a stop, not a warning to note and push past.
8155
8093
 
8156
- ### 0.2 Act on drift
8094
+ ## Single-run discipline
8157
8095
 
8158
- For each drifting tool, follow [ecosystem-sync-sop.md](ecosystem-sync-sop.md) Phase 2-4:
8096
+ `npm test` runs the Bats suite with `--jobs 4`. Running it repeatedly to see
8097
+ whether a failure is "flaky" hides real ordering bugs and wastes minutes. Run
8098
+ it once. If it fails, read the failure and fix the cause; if the same test
8099
+ passes on a re-run without a code change, that instability is itself the bug
8100
+ and belongs in an issue, not in a retry.
8159
8101
 
8160
- | Drift class | Release impact |
8161
- |-------------|----------------|
8162
- | A (cosmetic reword) | No version impact — refresh snapshot, continue |
8163
- | B (new feature — integrate) | **Minor** version bump at minimum; new generator or extended generator |
8164
- | C (new feature — not adopted) | No impact — note in registry |
8165
- | D (deprecation) | **Minor** or **major** depending on user impact; add migration warning |
8166
- | E (feature promoted to default) | **Minor**; simplify generator, keep fallback comment |
8167
- | F (feature newly globally available) | **Minor**; may require new generator or new config path |
8102
+ ## What this gate does not cover
8168
8103
 
8169
- If any B/D/E/F changes land in this preparation pass, mention them explicitly in the CHANGELOG entry (Phase 3) under a `Ecosystem` subsection.
8104
+ - **Cross-platform.** The suite here runs on your machine only. Both
8105
+ `ubuntu-latest` and `macos-latest` must be green before a tag; that gate
8106
+ lives in `sop-release.md`, Phase 7, and is bound to the exact
8107
+ release commit.
8108
+ - **Required files.** The `required-files` CI job checks that every file the
8109
+ module template mandates is present and that the version is in sync across
8110
+ all four manifests. It is cheap and runs on every push.
8111
+ - **Provenance and SARIF.** Release-time concerns; see
8112
+ `sop-release.md`.
8170
8113
 
8171
- ### 0.3 Refresh snapshot
8114
+ ## Commit message
8172
8115
 
8173
- Once every drift is resolved (either by code change or by re-classifying as acceptable):
8116
+ Conventional Commits, and no AI co-authorship trailer:
8174
8117
 
8175
- ```bash
8176
- python3 scripts/ecosystem_doctor.py --update
8118
+ ```
8119
+ feat(scope): summary in the imperative
8120
+ fix(scope): summary in the imperative
8121
+ docs|refactor|test|chore(scope): ...
8177
8122
  ```
8178
8123
 
8179
- This writes the new baseline to `benchmarks/ecosystem-doctor-snapshot.json`. Commit it as part of the release commit.
8124
+ `pre-commit-check.py` suggests a type from the staged paths. It is a
8125
+ suggestion; the scope and the summary are yours.
8180
8126
 
8181
- ### 0.4 Gate
8127
+ ## Related
8182
8128
 
8183
- ```bash
8184
- python3 scripts/ecosystem_doctor.py --offline --check
8185
- ```
8129
+ - `sop-release.md` -- the pre-tag gate, including cross-platform CI
8130
+ - `sop-post-release-testing.md` -- what to verify after a release ships
8131
+ - `sop-release-verification.md` -- end-to-end smoke test of an installed build
8186
8132
 
8187
- Must exit `0`. If it exits `1`, the snapshot is stale — rerun Phase 0.3 or review the remaining drift.
8133
+ ---
8188
8134
 
8135
+ ## kb/procedures/sop-release-verification.md
8136
+
8137
+ ---
8138
+ title: "SOP: Release Verification"
8139
+ category: procedures
8140
+ service: ai-toolkit
8141
+ tags: [sop, verification, release, smoke-test, install, update, qa, provenance, sarif, dsh]
8142
+ version: "1.8.0"
8143
+ created: "2026-04-08"
8144
+ last_updated: "2026-09-01"
8145
+ description: "End-to-end smoke test after installing or updating @softspark/ai-toolkit. Verifies CLI, native Codex and GitHub Copilot surfaces, explicit DSH lifecycle, Claude app export, doctor, validation, tests, eject, provenance, SARIF, and per-skill permissions."
8189
8146
  ---
8190
8147
 
8191
- ## Phase 1: Determine Version Bump
8148
+ # SOP: Release Verification
8192
8149
 
8193
- Follow [Semantic Versioning](https://semver.org/):
8150
+ End-to-end smoke test after installing or updating `@softspark/ai-toolkit`.
8151
+ Verifies all critical paths from the user's perspective.
8194
8152
 
8195
- | Change Type | Bump | Examples |
8196
- |-------------|------|---------|
8197
- | Bugfix, typo, doc-only | **patch** | Fix install flag, correct description |
8198
- | New feature, skill, agent, flag | **minor** | Add `/hipaa-validate`, add `--output json` |
8199
- | Breaking CLI change, removed skill, config format change | **major** | Rename `install` to `setup`, remove skill |
8153
+ **Use this SOP when:**
8154
+ - After `npm install -g @softspark/ai-toolkit@latest`
8155
+ - After `ai-toolkit update`
8156
+ - Before tagging a new version (`git tag`)
8157
+ - Before publishing to npm (`npm publish`)
8158
+ - As a smoke test in CI/CD
8200
8159
 
8201
- **Rule:** When in doubt, bump minor.
8160
+ **Prerequisites:**
8161
+ - Node.js >= 18, Python 3, `bats`, git
8162
+ - `@softspark/ai-toolkit` installed globally
8163
+
8164
+ **Time:** 10-15 minutes (full), 2 minutes (quick checklist)
8202
8165
 
8203
8166
  ---
8204
8167
 
8205
- ## Phase 2: Sync Version in All Files
8168
+ ## Quick Checklist (TL;DR)
8206
8169
 
8207
- The canonical version lives in `package.json`. These files **must** match:
8170
+ The 14 core commands below must pass. Releases that change DSH must also complete Phase 10.
8208
8171
 
8209
- ### Mandatory sync (every release)
8172
+ ```bash
8173
+ # Pre-commit (Phase 0)
8174
+ python3 scripts/generate_agents_md.py > AGENTS.md # 1. Regenerate AGENTS.md
8175
+ python3 scripts/generate_llms_txt.py > llms.txt # 2. Regenerate llms.txt
8176
+ python3 scripts/validate.py --strict # 3. Validation passed?
8177
+ npm test > /tmp/npm-test.log 2>&1 && grep -c '^ok ' /tmp/npm-test.log && ! grep -q '^not ok' /tmp/npm-test.log # 4. All tests passed? (single run, cached)
8210
8178
 
8211
- | File | Field | How to update |
8212
- |------|-------|---------------|
8213
- | `package.json` | `"version": "X.Y.Z"` | Edit directly |
8214
- | `manifest.json` | `"version": "X.Y.Z"` | Edit directly |
8215
- | `app/.claude-plugin/plugin.json` | `"version": "X.Y.Z"` | Edit directly |
8179
+ # Post-install verification (Phases 1-7)
8180
+ ai-toolkit --version # 5. Version OK?
8181
+ ai-toolkit status # 6. Status OK?
8182
+ ai-toolkit doctor # 7. Health check passed?
8183
+ ai-toolkit install --dry-run # 8. Global install OK?
8184
+ python3 scripts/audit_skills.py --ci # 9. Security audit clean?
8216
8185
 
8217
- ### Auto-synced (no manual action)
8186
+ # Supply-chain verification (Phase 8, v2.8.0+)
8187
+ python3 scripts/audit_skills.py --sarif | python3 -c "import json,sys; assert json.load(sys.stdin)['version']=='2.1.0'; print('SARIF OK')" # 10. SARIF 2.1.0 well-formed?
8188
+ python3 scripts/audit_skills.py --permissions | head -30 # 11. Broad-access skills reviewed?
8189
+ npm view @softspark/ai-toolkit@X.Y.Z --json | python3 -c "import json,sys; d=json.load(sys.stdin); assert d['dist']['attestations']['provenance']['predicateType']=='https://slsa.dev/provenance/v1'; print('PROVENANCE OK')" # 12. Provenance attested on npm?
8190
+ python3 scripts/claude_app.py verify # 13. Claude Chat/Cowork plugin contract valid?
8218
8191
 
8219
- | File | Mechanism |
8220
- |------|-----------|
8221
- | `package-lock.json` | Regenerated by `npm install --package-lock-only` |
8192
+ # Deep-coverage verification (Phase 9, v3.0.0+)
8193
+ META="generate_agents_md.py|generate_llms_txt.py|generate_language_rules_skills.py"
8194
+ diff <(grep -oE 'scripts/generate_[a-z_]+\.py' kb/reference/supported-tools-registry.md | sort -u) <(ls scripts/generate_*.py | grep -vE "$META" | sort -u) && echo "OK: registry matches" # 14. Registry <-> generators drift?
8195
+ ```
8222
8196
 
8223
- ### Conditional sync (only if the doc was modified in this release)
8197
+ ---
8224
8198
 
8225
- | File | Field | When to update |
8226
- |------|-------|---------------|
8227
- | `kb/procedures/maintenance-sop.md` | frontmatter `version:` | If SOP content changed |
8228
- | `kb/reference/skills-catalog.md` | frontmatter `version:` | If skills added/removed |
8229
- | `kb/reference/agents-catalog.md` | frontmatter `version:` | If agents added/removed |
8230
- | `kb/reference/hooks-catalog.md` | frontmatter `version:` | If hooks changed |
8231
- | `kb/reference/architecture-overview.md` | frontmatter `version:` | If architecture changed |
8232
- | `kb/reference/distribution-model.md` | frontmatter `version:` | If install model changed |
8233
- | `kb/reference/global-install-model.md` | frontmatter `version:` | If install model changed |
8199
+ ## Phase 0: Pre-Commit & Pre-Push (2 min)
8234
8200
 
8235
- > **Note:** KB `version:` fields track the **document version**, not the toolkit version.
8236
- > Only bump them when the document content actually changes in this release.
8201
+ Run these commands **before every commit and push to main**. CI validates
8202
+ counts but does NOT auto-regenerate you must do it locally.
8237
8203
 
8238
- ### Count sync (if skills/agents/hooks changed)
8204
+ ```bash
8205
+ # 1. Regenerate generated artifacts
8206
+ python3 scripts/generate_agents_md.py > AGENTS.md
8207
+ python3 scripts/generate_llms_txt.py > llms.txt
8208
+ python3 scripts/generate_llms_txt.py --full > llms-full.txt
8239
8209
 
8240
- | File | What to check |
8241
- |------|---------------|
8242
- | `package.json` | `"description"` — skill/agent count |
8243
- | `README.md` | Badge counts, "What You Get" table |
8244
- | `app/ARCHITECTURE.md` | Section headings with counts |
8210
+ # 2. Validate everything (catches stale counts, missing assets)
8211
+ python3 scripts/validate.py --strict
8245
8212
 
8246
- > **Tip:** `validate.py --strict` catches count drift AND version mismatches
8247
- > (package.json vs manifest.json vs plugin.json) automatically.
8248
- > If validation passes, counts and versions are correct.
8213
+ # 3. Security audit
8214
+ python3 scripts/audit_skills.py --ci
8249
8215
 
8250
- ### Verification command
8216
+ # 4. Run tests
8217
+ npm test
8251
8218
 
8252
- After syncing, verify all mandatory files match:
8219
+ # 5. Stage and commit
8220
+ # AGENTS.md, GEMINI.md, and .github/copilot-instructions.md are generated and
8221
+ # gitignored — do NOT commit them (regenerated at prepublishOnly). Only the
8222
+ # doc artifacts llms.txt / llms-full.txt are committed.
8223
+ git add llms.txt llms-full.txt
8224
+ git add -p # stage your other changes
8225
+ git commit -m "feat: your change description"
8226
+ ```
8227
+
8228
+ **Why local?** Branch protection on `main` requires PRs and status checks.
8229
+ CI cannot push directly to `main`, so generated artifacts must be committed
8230
+ by the developer as part of their PR.
8253
8231
 
8232
+ **One-liner (copy-paste):**
8254
8233
  ```bash
8255
- VERSION=$(python3 -c "import json; print(json.load(open('package.json'))['version'])")
8256
- echo "Target: $VERSION"
8257
- echo "manifest.json: $(python3 -c "import json; print(json.load(open('manifest.json'))['version'])")"
8258
- echo "plugin.json: $(python3 -c "import json; print(json.load(open('app/.claude-plugin/plugin.json'))['version'])")"
8259
- echo "package-lock.json: $(python3 -c "import json; print(json.load(open('package-lock.json'))['version'])")"
8234
+ python3 scripts/generate_agents_md.py > AGENTS.md && python3 scripts/generate_llms_txt.py > llms.txt && python3 scripts/generate_llms_txt.py --full > llms-full.txt && python3 scripts/validate.py --strict && python3 scripts/audit_skills.py --ci && npm test
8260
8235
  ```
8261
8236
 
8262
- All four must print the same version. If not, fix before proceeding.
8237
+ ---
8263
8238
 
8264
- ### Public surface review
8239
+ ## Phase 1: CLI & Version (1 min)
8265
8240
 
8266
8241
  ```bash
8267
- python3 scripts/surface_manifest.py --update
8268
- git diff app/surface.json
8269
- ```
8270
-
8271
- **Every line the diff removes is a breaking change.** Restore it, or take the
8272
- deprecation path in `BACKWARD_COMPATIBILITY.md` and add a `DECISIONS.md` entry
8273
- before the tag. Lines added are new surface being adopted into protection — that
8274
- needs no ceremony.
8275
-
8276
- Do not run `--update` to make a red build green. The check fails because something
8277
- users depend on disappeared; regenerating the manifest deletes the evidence, not
8278
- the problem.
8279
-
8280
- ### Skill body budget ratchet
8281
-
8282
- `validate.py` prints the largest skill body on every run:
8283
-
8284
- ```
8285
- Body budget: largest is <skill> at <N> bytes (warn 18000, error 20000)
8242
+ ai-toolkit --version
8243
+ ai-toolkit --help
8244
+ which ai-toolkit
8286
8245
  ```
8287
8246
 
8288
- Once that number sits at least 2000 bytes under `SKILL_BODY_BUDGET_WARN`, lower
8289
- `SKILL_BODY_BUDGET_WARN` by 2000 in `scripts/validate.py` and ship the tightened
8290
- threshold with the release. Target floor is 12000.
8291
-
8292
- Two rules, both learned the hard way:
8293
-
8294
- - **Never lower a threshold in the same change that something violates it.** Split
8295
- the offending skill into `SKILL.md` + `reference/` first, prove it with
8296
- `python3 scripts/check_split.py <skill> --before <pre-split SKILL.md>`, then
8297
- tighten in a follow-up.
8298
- - **Never raise a threshold to make a red build green.** A body over budget means
8299
- detail that belongs in `reference/` is loading on every trigger match, including
8300
- the accidental ones. Raising the number hides the cost, it does not remove it.
8301
-
8302
- If the largest body has not moved since the previous release, leave the threshold
8303
- alone and say so in the release notes. A ratchet that never advances is honest;
8304
- one that advances past reality is not.
8247
+ **Verify:**
8248
+ - [ ] `--version` returns correct semver (e.g., `1.4.0`)
8249
+ - [ ] `--help` displays full command list without errors
8250
+ - [ ] `which` points to global npm bin path
8305
8251
 
8306
8252
  ---
8307
8253
 
8308
- ## Phase 3: Write CHANGELOG Entry
8254
+ ## Phase 2: Global Install & Status (2 min)
8309
8255
 
8310
- Add entry at the top of `CHANGELOG.md` (after the header, before previous release):
8256
+ ```bash
8257
+ ai-toolkit install --dry-run
8258
+ ai-toolkit status
8259
+ ```
8311
8260
 
8312
- ```markdown
8313
- ## vX.Y.Z Short Title (YYYY-MM-DD)
8261
+ **Verify `--dry-run`:**
8262
+ - [ ] Agents: 44
8263
+ - [ ] Skills: 108
8264
+ - [ ] Hooks merged into settings.json
8265
+ - [ ] "Other AI Tools" lists documented global targets (with `--editors`): aider, antigravity, augment, cline, codex, copilot, cursor, gemini, opencode, roo, windsurf. Scope varies: Codex uses `$CODEX_HOME` (default `~/.codex`) plus `$HOME/.agents/skills`; Copilot uses `$COPILOT_HOME` (default `~/.copilot`); cursor has only `~/.cursor/hooks.json`; antigravity has the `~/.gemini/*/skills` pointer. Cursor and Antigravity rules remain project-only.
8314
8266
 
8315
- ### Added
8316
- - **Feature name** description
8267
+ **Verify `status`:**
8268
+ - [ ] Version matches expected
8269
+ - [ ] Profile: minimal/standard/strict
8270
+ - [ ] Modules: list of installed modules
8271
+ - [ ] Latest: up to date / update available
8317
8272
 
8318
- ### Changed
8319
- - **What changed** — old behavior → new behavior
8273
+ ---
8320
8274
 
8321
- ### Fixed
8322
- - **Bug description** — what was broken and how it's fixed
8275
+ ## Phase 3: Doctor Health Check (1 min)
8323
8276
 
8324
- ### Removed
8325
- - **What was removed** — migration path if any
8277
+ ```bash
8278
+ ai-toolkit doctor
8326
8279
  ```
8327
8280
 
8328
- **Rules:**
8329
- - Use **bold** for feature names
8330
- - Start descriptions with a verb (Added, Changed, Fixed, Removed)
8331
- - Reference skill names with backticks and slash: `/hipaa-validate`
8332
- - Include script names: `scripts/hipaa_scan.py`
8333
- - Include count changes: `Skill count: 91 → 92`
8334
- - Date format: `YYYY-MM-DD`
8335
- - Title: short, descriptive, no version number repetition
8336
-
8337
- ### Update README "What's New" section
8338
-
8339
- **MANDATORY on every release.** Update the `## What's New in vX.Y.Z` section in `README.md`:
8340
-
8341
- 1. Change the heading version: `## What's New in vX.Y.Z`
8342
- 2. Replace bullet points with 3-5 highlights from this release
8343
- 3. **Keep only the latest version block.** Delete the previous `## What's New in vA.B.C` section(s). README is the shop window, not the archive — users see the current release, full history lives in `CHANGELOG.md`.
8344
- 4. Keep the `See [CHANGELOG.md](CHANGELOG.md) for full history.` link directly below the bullet list.
8281
+ **Expected sections (all OK):**
8282
+ - Environment: node, bash, python3, bats
8283
+ - Global Install: .claude exists, agents/skills symlinks (0 broken), settings.json hooks
8284
+ - Hook Scripts: all present and executable
8285
+ - Hook Configuration: 14 events registered
8286
+ - Generated Artifacts: AGENTS.md, llms.txt, llms-full.txt
8287
+ - Planned Assets: plugin.json, benchmarks, plugin packs
8288
+ - Benchmark Freshness: < 30 days
8289
+ - Stale Rules: all healthy
8345
8290
 
8346
- > **Warning:** This section is the first thing users see after the badges.
8347
- > A stale version here (e.g., "What's New in v2.1.3" when shipping v2.3.0)
8348
- > signals an unmaintained project. Do NOT skip this step.
8291
+ **Verify:**
8292
+ - [ ] `Errors: 0 | Warnings: 0`
8293
+ - [ ] `HEALTH CHECK PASSED`
8349
8294
 
8350
- > **Single-version rule:** README.md must contain **exactly one** `## What's New in vX.Y.Z` heading at any time. If you find multiple stacked (e.g. v2.6.1 + v2.6.0 + v2.5.0), that is a SOP drift — collapse to the latest on the next release commit.
8295
+ If doctor detects problems: `ai-toolkit doctor --fix` auto-repairs
8296
+ (broken symlinks, non-executable hooks, missing scripts, missing llms-full.txt).
8351
8297
 
8352
8298
  ---
8353
8299
 
8354
- ## Phase 4: Regenerate Artifacts
8355
-
8356
- Use the npm scripts, not the generators directly:
8300
+ ## Phase 4: Local Install (2 min)
8357
8301
 
8358
8302
  ```bash
8359
- npm run generate:agents # AI_TOOLKIT_NO_CUSTOM_RULES=1 python3 scripts/generate_agents_md.py > AGENTS.md
8360
- npm run generate:llms # llms.txt + llms-full.txt
8303
+ mkdir -p /tmp/ai-toolkit-verify && cd /tmp/ai-toolkit-verify
8304
+ git init -q
8305
+ ai-toolkit install --local --editors all --dry-run
8306
+ cd - && rm -rf /tmp/ai-toolkit-verify
8361
8307
  ```
8362
8308
 
8363
- `generate:agents` sets `AI_TOOLKIT_NO_CUSTOM_RULES=1`. Running
8364
- `generate_agents_md.py` bare picks up whatever is registered in the maintainer's
8365
- own `~/.softspark/ai-toolkit/rules/`, which then ships inside `AGENTS.md`.
8366
-
8367
- Check if anything actually changed:
8309
+ **Verify "Project-local" section:**
8310
+ - [ ] Would create: CLAUDE.md
8311
+ - [ ] Would create: .claude/settings.local.json
8312
+ - [ ] Would inject: .claude/constitution.md
8313
+ - [ ] Editors: all 11 listed (copilot, cursor, windsurf, cline, roo, aider, augment, antigravity, codex, gemini, opencode)
8314
+ - [ ] Would generate configs for each editor (legacy + directory-based)
8315
+ - [ ] Would install: .git/hooks/pre-commit
8316
+ - [ ] Would inject language rules (auto-detected)
8368
8317
 
8318
+ **Also test auto-detect (no --editors flag):**
8369
8319
  ```bash
8370
- git diff --stat AGENTS.md llms.txt llms-full.txt
8320
+ ai-toolkit install --local --dry-run
8321
+ # → Editors: none (empty project has no existing configs)
8371
8322
  ```
8372
8323
 
8373
- If no diff, the artifacts are already current. If there is a diff, stage them.
8374
-
8375
8324
  ---
8376
8325
 
8377
- ## Phase 5: Validate, Audit, Test
8378
-
8379
- Run the full quality gate:
8326
+ ## Phase 5: Validation & Security Audit (3 min)
8380
8327
 
8381
8328
  ```bash
8382
8329
  python3 scripts/validate.py --strict
8383
8330
  python3 scripts/audit_skills.py --ci
8384
- python3 scripts/audit_skills.py --sarif > audit.sarif # MANDATORY — GHAS ingest
8385
- python3 scripts/audit_skills.py --permissions # review Bash/Write/Edit footprint
8331
+ ```
8386
8332
 
8387
- # ShellCheck on hooks (added in 1.11.0). Mirrors the ci.yml "ShellCheck hooks"
8388
- # job. NOT run by validate.py, npm test, OR publish.yml so a hook with a
8389
- # ShellCheck warning passes every other gate AND still publishes on tag while
8390
- # turning main CI red. Run it here, before tagging.
8391
- shellcheck --severity=warning app/hooks/*.sh && echo "OK: shellcheck clean"
8333
+ **Verify validate.py:**
8334
+ - [ ] Agents: 44, Skills: 108, Tests: exactly the current README badge count
8335
+ - [ ] Hook events: 14, Hook scripts: >= 30
8336
+ - [ ] Plugin packs >= 10, KB documents >= 20
8337
+ - [ ] `Errors: 0 | Warnings: 0` → `VALIDATION PASSED`
8392
8338
 
8393
- # Registry / generator drift (added in 1.10.0). Meta-generators excluded.
8394
- META="generate_agents_md.py|generate_llms_txt.py|generate_language_rules_skills.py"
8395
- diff \
8396
- <(grep -oE 'scripts/generate_[a-z_]+\.py' kb/reference/supported-tools-registry.md | sort -u) \
8397
- <(ls scripts/generate_*.py | grep -vE "$META" | sort -u) \
8398
- && echo "OK: registry matches filesystem" \
8399
- || { echo "DRIFT: update supported-tools-registry.md before tagging"; exit 1; }
8339
+ **Verify audit_skills.py:**
8340
+ - [ ] `HIGH: 0` (MUST be zero — CI fails otherwise)
8341
+ - [ ] `WARN: 0`
8342
+ - [ ] `INFO: N` (acceptable — broad-access skills: orchestrate, swarm, workflow)
8400
8343
 
8401
- # Stage first IF this release adds or deletes a kb/ file. The test
8402
- # "npm package KB files match the tracked release set" compares `git ls-files
8403
- # kb` (the index) against what `npm pack` sees (the working tree), so an
8404
- # unstaged addition reads as "extra" and an unstaged deletion as "missing".
8405
- # Phase 6 stages, and it runs after this one, so the ordering fails the test
8406
- # for any release that touches the KB. Staging early costs nothing.
8407
- git status --porcelain kb/ | grep -qE '^(\?\?| D|\?M)' && git add -A kb/
8344
+ ---
8408
8345
 
8409
- # Run npm test ONCE, cache output, parse from file. The suite is 1400+ bats
8410
- # cases — rerunning it per check wastes minutes. Do not pipe npm test into
8411
- # tail/grep multiple times in the same session.
8346
+ ## Phase 6: Tests (3-5 min)
8347
+
8348
+ ```bash
8349
+ # Run ONCE, capture to file, then parse. Full suite is 669+ bats cases —
8350
+ # re-running it per check (tail / grep ok / grep not ok piped separately)
8351
+ # wastes minutes every release. Always cache the output.
8412
8352
  npm test > /tmp/npm-test.log 2>&1
8353
+ exit=$?
8413
8354
  tail -3 /tmp/npm-test.log
8414
- echo "ok: $(grep -c '^ok ' /tmp/npm-test.log) | not ok: $(grep -c '^not ok' /tmp/npm-test.log)"
8355
+ echo "ok: $(grep -c '^ok ' /tmp/npm-test.log)"
8356
+ echo "not ok: $(grep -c '^not ok' /tmp/npm-test.log)"
8357
+ echo "exit: $exit"
8415
8358
  ```
8416
8359
 
8417
- **Expected results:**
8418
- - `validate.py`: `Errors: 0 | Warnings: 0 | VALIDATION PASSED`
8419
- - `audit_skills.py --ci`: `HIGH: 0 | WARN: 0` (INFO is acceptable)
8420
- - `audit_skills.py --sarif`: valid JSON, non-empty `runs[0].tool.driver.rules`
8421
- - `audit_skills.py --permissions`: review `Skills with Bash + Write + Edit` list — any newly-added skill with broad access MUST be justified in the CHANGELOG entry
8422
- - `shellcheck --severity=warning app/hooks/*.sh`: no output, exit 0. A common false positive is `SC2034` on `INPUT` or env vars (e.g. `AI_TOOLKIT_HOOK_FORMAT`) that a *sourced* helper (`_hook-io.sh`) consumes — ShellCheck cannot see cross-file use. Fix with a `# shellcheck disable=SC2034` directive or `export`, matching `guard-destructive.sh`. Never tag with a red ShellCheck.
8423
- - Registry drift: `OK: registry matches filesystem`. If `DRIFT:` appears, add the missing `scripts/generate_*.py` rows to `kb/reference/supported-tools-registry.md` before tagging.
8424
- - `npm test`: `1..N` with zero `not ok` (read from the cached `/tmp/npm-test.log`, do not rerun)
8425
-
8426
- > **Why this matters (v4.5.1 postmortem):** `publish.yml` runs only `validate.py` + `npm test`, so it published v4.5.0 even though the `main` CI `ShellCheck hooks` job was red on two `SC2034` warnings in a new hook. The publish workflow does **not** depend on the CI workflow. Until that is fixed, ShellCheck is a manual pre-tag gate — run it here every time.
8360
+ **Verify:**
8361
+ - [ ] `exit == 0`
8362
+ - [ ] `ok == expected test count` (e.g., 945 on v3.0.0)
8363
+ - [ ] `not ok == 0`
8364
+ - [ ] Bats runs tests in parallel (4 jobs)
8365
+ - [ ] Groups: agents, autodetect, cli, generators, guards, hooks, inject,
8366
+ install, kb, mcp, readme, profiles, uninstall, validate
8427
8367
 
8428
- **One-liner:**
8368
+ **Anti-pattern — do NOT do this:**
8429
8369
  ```bash
8430
- python3 scripts/validate.py --strict && python3 scripts/audit_skills.py --ci && python3 scripts/audit_skills.py --sarif > audit.sarif && shellcheck --severity=warning app/hooks/*.sh && diff <(grep -oE 'scripts/generate_[a-z_]+\.py' kb/reference/supported-tools-registry.md | sort -u) <(ls scripts/generate_*.py | grep -vE 'generate_agents_md\.py|generate_llms_txt\.py|generate_language_rules_skills\.py' | sort -u) && npm test
8370
+ # Runs the full suite THREE times. Adds 1-3 min and pressures CI capacity.
8371
+ npm test 2>&1 | tail -3
8372
+ npm test 2>&1 | grep -c '^ok '
8373
+ npm test 2>&1 | grep -c '^not ok'
8431
8374
  ```
8432
8375
 
8433
- **If tests fail:** Fix the issue, do NOT skip. Common failures:
8434
- - Stale counts → re-run `generate:all` or fix README/ARCHITECTURE
8435
- - Missing frontmatter add to new KB docs
8436
- - Broken symlink `ai-toolkit doctor --fix`
8376
+ **Key test areas:**
8377
+ - Guards: rm -rf, DROP TABLE, git push --force blocked
8378
+ - Install: idempotent, profiles, --only/--skip, orphan cleanup
8379
+ - Eject: real files (not symlinks), inlined rules
8380
+ - Uninstall: removes toolkit, preserves user content
8437
8381
 
8438
- ### Phase 5c: Licensing Gate (v4.20.0+)
8382
+ ---
8439
8383
 
8440
- The project is Apache-2.0. Attribution only works if the artefact actually
8441
- carries it, and every part of that is mechanically checkable.
8384
+ ## Phase 7: Eject (1 min)
8442
8385
 
8443
8386
  ```bash
8444
- # The whole gate, enforced in CI. Run it here so a failure is caught before tagging.
8445
- npx bats tests/test_licensing.bats
8387
+ mkdir -p /tmp/ai-toolkit-eject-test
8388
+ cd /tmp/ai-toolkit-eject-test
8389
+ ai-toolkit eject
8390
+ cd - && rm -rf /tmp/ai-toolkit-eject-test
8446
8391
  ```
8447
8392
 
8448
- The seven assertions, and why each exists:
8393
+ **Verify:**
8394
+ - [ ] Agents copied as real files (not symlinks)
8395
+ - [ ] Skills copied as real directories
8396
+ - [ ] Rules inlined into CLAUDE.md
8397
+ - [ ] constitution.md and ARCHITECTURE.md copied
8398
+ - [ ] `output-styles/` directory present (v2.7.1+)
8449
8399
 
8450
- | Check | Fails when |
8451
- |---|---|
8452
- | Every shipped source file carries an SPDX header | A new `.py`/`.sh`/`.js`/`.bats` file was added without one — the common case, and the reason this is a test rather than a habit |
8453
- | Headers name Apache-2.0 and nothing else | A file was copied in from an MIT/GPL source with its own header intact |
8454
- | **No** markdown file carries an SPDX header | Someone "helpfully" ran the header script over `app/skills/` — headers there sit above parsed frontmatter and bill every session for it |
8455
- | `LICENSE` is the complete Apache 2.0 text | The file was truncated or replaced with a summary |
8456
- | `NOTICE` carries attribution, the source URL, §4(d) and the MIT-era notice | The attribution mechanism was gutted |
8457
- | `LICENSE` **and** `NOTICE` ship in the npm package | `package.json` `files` lost an entry — a NOTICE that never reaches the consumer cannot satisfy §4(d) |
8458
- | Every manifest declaring a licence declares Apache-2.0 | `package.json`, `manifest.json`, `plugin.json` and `package-lock.json` drifted apart |
8400
+ ---
8459
8401
 
8460
- **Adding source files in this release?** The header goes *after* the shebang,
8461
- never before it. Short SPDX form:
8402
+ ## Phase 8: Supply-Chain Verification (2 min, v2.8.0+)
8462
8403
 
8463
- ```
8464
- # SPDX-License-Identifier: Apache-2.0
8465
- # Copyright 2024-2026 Lukasz Krzemien (biuro@softspark.eu)
8466
- # Source: https://github.com/softspark/ai-toolkit
8467
- ```
8404
+ These checks enforce the v2.8.0 security standard on a freshly-published release. Run AFTER the `publish.yml` workflow completes on the tag.
8468
8405
 
8469
- `//` for JavaScript. Full convention and the reasoning behind the markdown
8470
- exclusion: [Licensing](../reference/licensing.md).
8406
+ ### 8.1 Provenance attestation on npm
8471
8407
 
8472
- **Changing the licence itself?** Do not hand-type the licence text. Take it
8473
- verbatim from a published copy and cross-verify against a second independent
8474
- copy before writing `LICENSE` — a rendered or summarised licence is not the
8475
- licence. Prior releases stay under their original terms; a licence change
8476
- applies going forward and revokes nothing already granted.
8408
+ ```bash
8409
+ VERSION="X.Y.Z" # the tag just published
8410
+ npm view "@softspark/ai-toolkit@${VERSION}" --json \
8411
+ | python3 -c "import json, sys; d=json.load(sys.stdin); att=d['dist'].get('attestations', {}); assert att.get('provenance', {}).get('predicateType') == 'https://slsa.dev/provenance/v1', f'NO PROVENANCE for {d[\"version\"]}'; print(f'PROVENANCE OK: {att[\"url\"]}')"
8412
+ ```
8477
8413
 
8478
- ### Phase 5a: Supply-Chain Hardening Verification (v2.8.0+)
8414
+ **Verify:**
8415
+ - [ ] Exit 0 and prints `PROVENANCE OK: https://registry.npmjs.org/...`
8416
+ - [ ] `https://www.npmjs.com/package/@softspark/ai-toolkit/v/${VERSION}` shows the green "Provenance" badge
8479
8417
 
8480
- These checks enforce the security standard introduced in v2.8.0. Do NOT tag a release until all pass.
8418
+ **If provenance is missing:** the `publish.yml` ran without `id-token: write` or `--provenance`. Restore them and cut a patch release a silently unsigned publish is a regression against the v2.8.0 standard.
8481
8419
 
8482
- **1. Publish workflow emits provenance:**
8420
+ ### 8.2 Audit SARIF output (for GHAS ingest)
8483
8421
 
8484
8422
  ```bash
8485
- grep -E '\-\-provenance|id-token: write' .github/workflows/publish.yml
8423
+ python3 scripts/audit_skills.py --sarif > /tmp/audit.sarif
8424
+ python3 -c "import json; d=json.load(open('/tmp/audit.sarif')); assert d['version']=='2.1.0' and d['runs'][0]['tool']['driver']['name']=='ai-toolkit-audit-skills'; print(f'SARIF OK: {len(d[\"runs\"][0][\"results\"])} results across {len(d[\"runs\"][0][\"tool\"][\"driver\"][\"rules\"])} rules')"
8486
8425
  ```
8487
8426
 
8488
- - [ ] Both markers present (`--provenance` flag + `id-token: write` permission)
8489
- - [ ] Any PR that changes `publish.yml` REQUIRES an approved security review
8427
+ **Verify:**
8428
+ - [ ] Valid SARIF 2.1.0
8429
+ - [ ] In the publishing repo, the CI job uploads `audit.sarif` via `github/codeql-action/upload-sarif@v3` so findings appear in the Security tab
8490
8430
 
8491
- **2. URL-sourced rules and hooks are checksum-pinned:**
8431
+ ### 8.3 Per-skill permissions report
8492
8432
 
8493
8433
  ```bash
8494
- # On a machine that has consumed URL rules/hooks at least once
8495
- # (schema_version 1: entries live under the .rules / .hooks key):
8496
- jq '.rules | to_entries | map(select(.value.url != null and (.value.sha256 // "" | length) == 0))' ~/.softspark/ai-toolkit/rules/sources.json
8497
- jq '.hooks | to_entries | map(select(.value.url != null and (.value.sha256 // "" | length) == 0))' ~/.softspark/ai-toolkit/hooks/external/sources.json
8434
+ python3 scripts/audit_skills.py --permissions | head -40
8498
8435
  ```
8499
8436
 
8500
- - [ ] Both queries return empty arrays (every URL entry has a `sha256`)
8501
- - [ ] If not, run `ai-toolkit update` to backfill missing hashes before tagging
8437
+ **Verify:**
8438
+ - [ ] Bash skill count has NOT jumped unexpectedly since the previous release
8439
+ - [ ] Any newly added entry under `Skills with Bash + Write + Edit` matches a CHANGELOG bullet that justifies the broad scope
8440
+ - [ ] JSON form (`--permissions --json`) is available for automated drift dashboards
8502
8441
 
8503
- **3. Audit SARIF output is well-formed:**
8442
+ ### 8.4 URL-sourced rules/hooks are checksum-pinned
8443
+
8444
+ `sources.json` is an envelope of the form `{"schema_version": 1, "rules"|"hooks": {...}}`, so the jq filter must pick the nested map before piping into the pin assertion. The URL-count in the success message ignores local `path:`-only entries — only entries with a `url:` field are pinned and counted.
8504
8445
 
8505
8446
  ```bash
8506
- python3 scripts/audit_skills.py --sarif | python3 -c "import json, sys; d=json.load(sys.stdin); assert d['version']=='2.1.0' and d['runs'][0]['tool']['driver']['name']; print('SARIF OK')"
8447
+ jq '.rules // .hooks // {}' ~/.softspark/ai-toolkit/rules/sources.json 2>/dev/null \
8448
+ | python3 -c "import json, sys; d=json.load(sys.stdin) or {}; bad=[n for n,v in d.items() if v.get('url') and not v.get('sha256')]; assert not bad, f'UNPINNED: {bad}'; url_n=sum(1 for v in d.values() if v.get('url')); print(f'RULE PIN OK: {url_n} URL rules, all with sha256')"
8449
+ jq '.hooks // {}' ~/.softspark/ai-toolkit/hooks/external/sources.json 2>/dev/null \
8450
+ | python3 -c "import json, sys; d=json.load(sys.stdin) or {}; bad=[n for n,v in d.items() if v.get('url') and not v.get('sha256')]; assert not bad, f'UNPINNED: {bad}'; url_n=sum(1 for v in d.values() if v.get('url')); print(f'HOOK PIN OK: {url_n} URL hooks, all with sha256')"
8507
8451
  ```
8508
8452
 
8509
- - [ ] Prints `SARIF OK`
8510
- - [ ] If the script ever grows new rule classes, extend the SARIF `rules[]` coverage before releasing
8453
+ **Verify:**
8454
+ - [ ] Both commands print `... PIN OK`
8455
+ - [ ] If any entry is unpinned, the `register_url_source()` call missed passing `content=` — fix the call-site and retag
8511
8456
 
8512
- **4. Strict-pin mode passes on CI** (optional, recommended for stable branches):
8457
+ ### 8.5 Strict-pin smoke test (optional but recommended)
8513
8458
 
8514
8459
  ```bash
8515
8460
  AI_TOOLKIT_STRICT_PIN=1 ai-toolkit update --dry-run
8516
8461
  ```
8517
8462
 
8463
+ **Verify:**
8518
8464
  - [ ] Exit 0, no `CHECKSUM CHANGED` line
8519
- - [ ] Any unexpected upstream change blocks the release until explicitly approved
8465
+ - [ ] If a checksum change was intentional (e.g. upstream rule update), document it in the CHANGELOG entry before tagging
8520
8466
 
8521
8467
  ---
8522
8468
 
8523
- ## Phase 6: Commit
8469
+ ## Phase 9: Deep-Coverage Checks (v3.0.0+)
8524
8470
 
8525
- Stage all release files:
8471
+ These verify the native-surface generators shipped in v3.0.0 actually emit the right files for the right profiles, and that the tool registry stays in sync with shipped generators.
8472
+
8473
+ > **Safety warning — HOME-scoped writes:** Running `--profile full` with `augment` in the editor list writes to `$HOME/.augment/settings.json` (Augment stores hooks under HOME, not per-project). Use `--dry-run` for verification unless you intend to carry ai-toolkit hook entries on this machine. The generator is marker-safe (only rewrites its own `_source: ai-toolkit` entries) but is still a side-effect.
8474
+
8475
+ ### 9.1 `--profile full` emits every native surface
8526
8476
 
8527
8477
  ```bash
8528
- git add package.json manifest.json app/.claude-plugin/plugin.json
8529
- git add package-lock.json
8530
- git add CHANGELOG.md
8531
- git add llms.txt llms-full.txt
8532
- # NOTE: AGENTS.md, GEMINI.md, and .github/copilot-instructions.md are generated
8533
- # editor configs and are gitignored — do NOT commit them. `prepublishOnly` runs
8534
- # `npm run generate:all`, so the shipped package (which lists AGENTS.md in
8535
- # package.json `files`) gets a fresh copy at publish time.
8536
- git add -p # review and stage any other changes
8478
+ D=/tmp/aitk-profile-full-${RANDOM} && mkdir -p "$D" && cd "$D" && git init -q
8479
+ ai-toolkit install --local --editors cursor,windsurf,gemini,augment,codex,copilot \
8480
+ --profile full --dry-run 2>&1 \
8481
+ | grep -E "\\.cursor/(hooks\\.json|agents)|\\.devin/hooks\\.v1\\.json|\\.gemini/(settings\\.json|commands)|\\.augment/(agents|commands)|\\.codex/(agents|hooks)|\\.agents/skills|\\.github/(agents|skills|instructions|prompts|hooks)"
8537
8482
  ```
8538
8483
 
8539
- Commit:
8484
+ **Verify** — at least the following lines appear:
8485
+ - [ ] `.cursor/hooks.json` and `.cursor/agents/`
8486
+ - [ ] `.devin/hooks.v1.json`
8487
+ - [ ] `.gemini/settings.json` hooks AND `.gemini/commands/`
8488
+ - [ ] `.augment/agents/` + `.augment/commands/` + `$HOME/.augment/settings.json`
8489
+ - [ ] `.codex/hooks.json` + `.codex/hooks/` + `.codex/agents/`
8490
+ - [ ] `.agents/skills/` (Codex native discovery path)
8491
+ - [ ] `.github/agents/` + `.github/skills/` + `.github/instructions/` + `.github/prompts/` + `.github/hooks/`
8492
+
8493
+ ### 9.2 Codex skills are profile-independent
8540
8494
 
8541
8495
  ```bash
8542
- git commit -m "chore: release vX.Y.Z"
8496
+ D=/tmp/aitk-codex-skills-${RANDOM} && mkdir -p "$D" && cd "$D" && git init -q
8497
+ ai-toolkit install --local --editors codex --profile minimal --dry-run 2>&1 \
8498
+ | grep -q "Would generate: .agents/skills" && echo "OK: Codex skills emit at minimal"
8499
+ ai-toolkit install --local --editors codex --profile standard --codex-skills --dry-run 2>&1 \
8500
+ | grep -q "Would refresh: .agents/skills" && echo "OK: legacy flag explicitly refreshes the same catalog"
8543
8501
  ```
8544
8502
 
8545
- ---
8503
+ **Verify:**
8504
+ - [ ] Selecting Codex emits all skills under `.agents/skills/` at every profile
8505
+ - [ ] `--codex-skills` remains an explicit refresh alias; it is not required for installation
8506
+ - [ ] No profile emits `.codex/skills/`
8546
8507
 
8547
- ## Phase 7: Push Branch, Verify CI, Tag, and Push Tag
8508
+ ### 9.3 Breaking-change surfaces land on `--profile standard`
8509
+
8510
+ The standard profile must include:
8511
+ - Copilot scoped instructions, prompts, custom agents, portable skills, and hooks
8512
+ - Gemini hooks (`.gemini/settings.json`)
8548
8513
 
8549
8514
  ```bash
8550
- RELEASE_SHA=$(git rev-parse HEAD)
8551
- git push origin main || { echo "FAIL: release commit push failed"; exit 1; }
8515
+ D=/tmp/aitk-breaking-${RANDOM} && mkdir -p "$D" && cd "$D" && git init -q
8516
+ ai-toolkit install --local --editors copilot,gemini --profile standard --dry-run 2>&1 \
8517
+ | tee /tmp/aitk-breaking.log
8518
+ grep -q "\\.github/instructions/" /tmp/aitk-breaking.log && echo "OK: Copilot dir layout at standard"
8519
+ grep -q "\\.github/agents/.*\\.github/skills/" /tmp/aitk-breaking.log && echo "OK: Copilot agents and skills at standard"
8520
+ grep -q "\\.github/hooks/" /tmp/aitk-breaking.log && echo "OK: Copilot hooks at standard"
8521
+ grep -q "\\.gemini/settings\\.json hooks" /tmp/aitk-breaking.log && echo "OK: Gemini hooks at standard"
8522
+ ```
8552
8523
 
8553
- # Bind the gate to the exact release commit. GitHub run registration is
8554
- # asynchronous, so retry at most five times and log every attempt.
8555
- RUN_ID=""
8556
- for ATTEMPT in 1 2 3 4 5; do
8557
- RUN_ID=$(gh run list --workflow ci.yml --event push --commit "$RELEASE_SHA" \
8558
- --limit 1 --json databaseId --jq '.[0].databaseId')
8559
- [ -n "$RUN_ID" ] && break
8560
- echo "Waiting for CI run registration ($ATTEMPT/5)" >&2
8561
- [ "$ATTEMPT" -eq 5 ] || sleep 60
8562
- done
8563
- test -n "$RUN_ID" || { echo "FAIL: CI run not found for release commit"; exit 1; }
8564
- gh run watch "$RUN_ID" --exit-status \
8565
- || { echo "FAIL: release commit CI failed"; exit 1; }
8524
+ **Verify all four lines print `OK:`**. If any is missing, native Copilot or Gemini coverage has regressed.
8566
8525
 
8567
- # publish.yml is Ubuntu-only. Require both full CI matrix jobs before creating
8568
- # the tag so a green publish job cannot bypass a red macOS test.
8569
- for OS in ubuntu-latest macos-latest; do
8570
- gh run view "$RUN_ID" --json jobs \
8571
- --jq ".jobs[] | select(.name == \"Bats test suite ($OS)\") | .conclusion" \
8572
- | grep -qx success || { echo "FAIL: $OS test job not green"; exit 1; }
8573
- done
8526
+ ### 9.4 Install is idempotent
8574
8527
 
8575
- # Refuse to tag if either local or remote main moved while CI was running.
8576
- test "$(git rev-parse HEAD)" = "$RELEASE_SHA" \
8577
- || { echo "FAIL: local HEAD changed during CI"; exit 1; }
8578
- git fetch origin main || { echo "FAIL: origin/main refresh failed"; exit 1; }
8579
- test "$(git rev-parse origin/main)" = "$RELEASE_SHA" \
8580
- || { echo "FAIL: origin/main changed during CI"; exit 1; }
8528
+ ```bash
8529
+ D=/tmp/aitk-idem-${RANDOM} && mkdir -p "$D" && cd "$D" && git init -q
8530
+ # Sort file paths before hashing find traversal order follows inode order,
8531
+ # which can shift between runs even when content is byte-identical, producing
8532
+ # false FAIL signals.
8533
+ ai-toolkit install --local --editors cursor,gemini --profile full >/dev/null 2>&1
8534
+ SHA1=$(find .cursor .gemini -type f -print0 | LC_ALL=C sort -z | xargs -0 shasum | shasum | awk '{print $1}')
8535
+ ai-toolkit install --local --editors cursor,gemini --profile full >/dev/null 2>&1
8536
+ SHA2=$(find .cursor .gemini -type f -print0 | LC_ALL=C sort -z | xargs -0 shasum | shasum | awk '{print $1}')
8537
+ [ "$SHA1" = "$SHA2" ] && echo "OK: idempotent" || echo "FAIL: install is not idempotent"
8538
+ ```
8581
8539
 
8582
- git tag vX.Y.Z "$RELEASE_SHA"
8540
+ **Verify:** prints `OK: idempotent`. A second run must produce byte-identical files in every managed path.
8583
8541
 
8584
- # Assert the tag before pushing it. Both checks are one line each and both
8585
- # have caught a real broken release.
8586
- test "$(git rev-parse vX.Y.Z)" = "$RELEASE_SHA" \
8587
- || { echo "FAIL: tag is not on the tested release commit"; exit 1; }
8588
- git show --no-patch --format=%s vX.Y.Z | grep -qx "chore: release vX.Y.Z" \
8589
- || { echo "FAIL: tag is not on the chore: release commit"; exit 1; }
8542
+ ### 9.5 Live-install JSON outputs parse
8590
8543
 
8591
- # Push the single release tag by its full ref.
8592
- git push origin refs/tags/vX.Y.Z
8544
+ The bats suite validates JSON shape at generation time. This re-checks that what actually landed on disk after a live install parses without errors.
8545
+
8546
+ ```bash
8547
+ D=/tmp/aitk-json-${RANDOM} && mkdir -p "$D" && cd "$D" && git init -q
8548
+ ai-toolkit install --local --editors cursor,windsurf,gemini,augment,codex,copilot --profile full >/dev/null 2>&1
8549
+ for f in .cursor/hooks.json .devin/hooks.v1.json .gemini/settings.json .codex/hooks.json .github/hooks/ai-toolkit.json "$HOME/.augment/settings.json"; do
8550
+ [ -f "$f" ] && python3 -c "import json; json.load(open('$f'))" && echo "OK: $f"
8551
+ done
8593
8552
  ```
8594
8553
 
8595
- **Why branch CI comes before the tag (v4.30.2 postmortem).** The publish
8596
- workflow runs only on Ubuntu and can publish while the separate macOS matrix job
8597
- is red. `v4.30.2` exposed this with a macOS-only DSH timing failure. A release
8598
- commit must therefore pass the complete Ubuntu/macOS branch workflow before its
8599
- tag exists; a successful publish workflow is not a substitute for green CI.
8554
+ **Verify:** each emitted file prints `OK: <path>`. Any `json.decoder.JSONDecodeError` means the merge logic corrupted the output.
8600
8555
 
8601
- **Why the assertions (v4.19.0 postmortem).** v4.19.0 was tagged on a commit
8602
- that contained only a KB document and still carried `package.json` version
8603
- `4.18.0`; the actual release sat in the commit above it under a recycled
8604
- `fix:` message. `publish.yml` fired, tried to publish a version already on
8605
- npm, and failed. Nothing on npm, a tag pointing at the wrong tree, and the
8606
- only way out was rewriting a pushed commit. Both assertions above catch this
8607
- in under a second. Run them.
8556
+ ### 9.6 Registry / generator drift check
8608
8557
 
8609
- **Never `git push --tags`.** It pushes every local tag at once, and GitHub
8610
- suppresses tag-triggered workflow runs when many tags arrive in a single push
8611
- — the workflow silently does not fire and nothing publishes. Push the single
8612
- release tag by its full ref, as above. (Sibling evidence: this is exactly how
8613
- rag-mcp's `1.0.3` image build was skipped, when a `--tags` push carried 37
8614
- tags at once.)
8558
+ `kb/reference/supported-tools-registry.md` should enumerate every per-editor `scripts/generate_*.py` we ship. Meta-generators (`generate_agents_md.py`, `generate_llms_txt.py`) are excluded they produce docs/artifacts, not editor configs.
8615
8559
 
8616
- This triggers `.github/workflows/publish.yml` which:
8617
- 1. Runs `validate.py --strict`
8618
- 2. Runs `npm test`
8619
- 3. Publishes to npm as `@softspark/ai-toolkit@X.Y.Z` with `--provenance` (SLSA v1 build attestation)
8560
+ ```bash
8561
+ META="generate_agents_md.py|generate_llms_txt.py|generate_language_rules_skills.py"
8562
+ REG=$(grep -oE 'scripts/generate_[a-z_]+\.py' kb/reference/supported-tools-registry.md | sort -u)
8563
+ FS=$(ls scripts/generate_*.py | grep -vE "$META" | sort -u)
8564
+ diff <(echo "$REG") <(echo "$FS") && echo "OK: registry matches filesystem" || echo "DRIFT: update supported-tools-registry.md"
8565
+ ```
8620
8566
 
8621
- **Provenance is non-negotiable.** If `id-token: write` permission or the `--provenance` flag is missing from `publish.yml`, fix it BEFORE tagging an unsigned release is a regression against the v2.8.0 standard.
8567
+ **Verify:** prints `OK: registry matches filesystem`. If not, add the missing rows to the registry before tagging the next release.
8622
8568
 
8623
- **After CI completes:** Run the [Release Verification SOP](release-verification-sop.md)
8624
- to smoke-test the published package AND verify the provenance attestation landed on npm.
8569
+ ### 9.7 Claude Chat / Cowork plugin validates and exports
8570
+
8571
+ ```bash
8572
+ python3 scripts/claude_app.py verify
8573
+ D=/tmp/aitk-claude-app-${RANDOM}
8574
+ python3 scripts/claude_app.py export --output "$D.zip" --no-custom-rules
8575
+ python3 -c "import zipfile; z=zipfile.ZipFile('$D.zip'); assert '.claude-plugin/plugin.json' in z.namelist(); assert 'claude-app/skills/ai-toolkit-rules/SKILL.md' in z.namelist(); print('OK: Claude app archive')"
8576
+ ```
8577
+
8578
+ **Verify:** the official validator exits 0; the archive contains the manifest,
8579
+ app-native rules skill, bundled agents/skills, and plugin-relative Cowork hooks.
8625
8580
 
8626
8581
  ---
8627
8582
 
8628
- ## Rollback
8583
+ ## Phase 10: Explicit DSH Qualification (v4.30.0+)
8629
8584
 
8630
- If a bad release was published:
8585
+ Run this phase whenever the release changes the `dsh` target, package pins, preset lifecycle, or DSH compatibility documentation. Use a new task-specific `DSH_HOME`; never replace `HOME` or reuse a regular profile.
8586
+
8587
+ Prerequisites: DSH `0.1.1-rc.2`, pnpm `>=11.7.0,<12.0.0`, Codex logged in through ChatGPT, Claude Code logged in natively, and GitHub Copilot CLI logged in natively. Do not supply provider API keys.
8631
8588
 
8632
8589
  ```bash
8633
- # Unpublish from npm (within 72h)
8634
- npm unpublish @softspark/ai-toolkit@X.Y.Z
8590
+ DSH_SMOKE_ROOT="$(mktemp -d)"
8591
+ export DSH_HOME="$DSH_SMOKE_ROOT/dsh-home"
8635
8592
 
8636
- # Or deprecate (preferred — doesn't break existing installs)
8637
- npm deprecate @softspark/ai-toolkit@X.Y.Z "Known issue: <description>. Use vA.B.C instead."
8593
+ ai-toolkit dsh install --profile web
8594
+ ai-toolkit dsh doctor --profile web
8595
+ dsh --profile web --host 127.0.0.1 --port 0 --no-open
8596
+ ```
8638
8597
 
8639
- # Delete tag
8640
- git tag -d vX.Y.Z
8641
- git push origin --delete vX.Y.Z
8598
+ In a new `softspark-orchestrator` session, select the `codex` provider and run two standalone marker prompts:
8599
+
8600
+ 1. `subagent_claude_code` returns an exact child marker, then the Codex parent returns its exact completion marker.
8601
+ 2. `subagent_gemini_copilot` returns an exact child marker, then the Codex parent returns its exact completion marker.
8602
+
8603
+ Stop DSH, then remove only the managed profile artifacts:
8604
+
8605
+ ```bash
8606
+ ai-toolkit dsh uninstall --profile web --yes
8642
8607
  ```
8643
8608
 
8609
+ **Verify:** both tool results have `isError=false`, both turns end as `completed`, `doctor` reports no recovery requirement before uninstall, and an unrelated preset fixture remains unchanged. Preserve only redacted event sequence evidence; never attach credentials, auth files, or full private prompts.
8610
+
8644
8611
  ---
8645
8612
 
8646
- ## Checklist Summary
8613
+ ## Troubleshooting
8647
8614
 
8648
- | # | Step | Command / Action | Pass Criteria |
8649
- |---|------|-----------------|---------------|
8650
- | 0a | Ecosystem drift check | `ecosystem_doctor.py --format text` | All tools Clean, or drift classified and resolved |
8651
- | 0b | Ecosystem snapshot refresh | `ecosystem_doctor.py --update` | `benchmarks/ecosystem-doctor-snapshot.json` updated |
8652
- | 0c | Ecosystem gate | `ecosystem_doctor.py --offline --check` | Exit 0 |
8653
- | 1 | Version bump type | Decide patch/minor/major | — |
8654
- | 2 | `package.json` version | Edit `"version"` | Matches target |
8655
- | 3 | `manifest.json` version | Edit `"version"` | Matches target |
8656
- | 4 | `plugin.json` version | Edit `"version"` | Matches target |
8657
- | 5 | `package-lock.json` | `npm install --package-lock-only` | Matches target |
8658
- | 6 | Count sync | Check `package.json` description, README | `validate.py` passes |
8659
- | 7 | CHANGELOG.md | Add release entry (incl. `Ecosystem` subsection if any B/D/E/F drift) | Entry exists for vX.Y.Z |
8660
- | 8 | Regenerate artifacts | `generate_agents_md.py`, `generate_llms_txt.py` | No unexpected diff |
8661
- | 9 | Validate | `validate.py --strict` | 0 errors, 0 warnings |
8662
- | 10 | Security audit (CI mode) | `audit_skills.py --ci` | 0 HIGH |
8663
- | 11 | Security audit (SARIF) | `audit_skills.py --sarif` | Valid SARIF 2.1.0 JSON |
8664
- | 12 | Per-skill permissions | `audit_skills.py --permissions` | New broad-access skills justified in CHANGELOG |
8665
- | 13 | ShellCheck hooks | `shellcheck --severity=warning app/hooks/*.sh` | Exit 0, no output (mirrors ci.yml; publish.yml does NOT run it) |
8666
- | 14 | Provenance flag check | `grep -- '--provenance' .github/workflows/publish.yml` | Present |
8667
- | 15 | Checksum-pin backfill | `sources.json` entries all have `sha256` | No unpinned URL sources |
8668
- | 15a | Licensing gate | `npx bats tests/test_licensing.bats` | 7/7 — SPDX headers, LICENSE, NOTICE, npm `files`, manifest consistency |
8669
- | 16 | Tests | `git add -A kb/` if the KB changed, then `npm test` | All pass |
8670
- | 17 | Commit | `git commit` | Clean working tree |
8671
- | 18 | Push branch | `git push origin main` | Exact release commit is on `origin/main` |
8672
- | 18a | Full branch CI | `gh run watch "$RUN_ID" --exit-status` plus matrix job assertions | Ubuntu and macOS Bats jobs both conclude `success` |
8673
- | 19 | Tag | `git tag vX.Y.Z` | Tag exists only after green branch CI |
8674
- | 19a | Tag is on tested SHA | `test "$(git rev-parse vX.Y.Z)" = "$RELEASE_SHA"` | Exit 0 |
8675
- | 19b | Tag is on the release commit | `git show --no-patch --format=%s vX.Y.Z` | Reads `chore: release vX.Y.Z` |
8676
- | 20 | Push the single tag | `git push origin refs/tags/vX.Y.Z` | Publish CI triggered with `id-token: write`. Never `--tags`. |
8615
+ ### `ai-toolkit: command not found`
8677
8616
 
8678
- ---
8617
+ ```bash
8618
+ npm install -g @softspark/ai-toolkit
8619
+ # or check PATH:
8620
+ export PATH="$(npm config get prefix)/bin:$PATH"
8621
+ ```
8679
8622
 
8680
- ## kb/procedures/release-verification-sop.md
8623
+ ### Doctor: broken symlinks
8681
8624
 
8682
- ---
8683
- title: "SOP: Release Verification"
8684
- category: procedures
8685
- service: ai-toolkit
8686
- tags: [sop, verification, release, smoke-test, install, update, qa, provenance, sarif, dsh]
8687
- version: "1.8.0"
8688
- created: "2026-04-08"
8689
- last_updated: "2026-09-01"
8690
- description: "End-to-end smoke test after installing or updating @softspark/ai-toolkit. Verifies CLI, native Codex and GitHub Copilot surfaces, explicit DSH lifecycle, Claude app export, doctor, validation, tests, eject, provenance, SARIF, and per-skill permissions."
8691
- ---
8625
+ ```bash
8626
+ ai-toolkit doctor --fix # auto-repair
8627
+ ai-toolkit update # or full re-install
8628
+ ```
8692
8629
 
8693
- # SOP: Release Verification
8630
+ ### Tests fail: missing bats
8694
8631
 
8695
- End-to-end smoke test after installing or updating `@softspark/ai-toolkit`.
8696
- Verifies all critical paths from the user's perspective.
8632
+ ```bash
8633
+ brew install bats-core # macOS
8634
+ npm install -g bats # cross-platform
8635
+ ```
8697
8636
 
8698
- **Use this SOP when:**
8699
- - After `npm install -g @softspark/ai-toolkit@latest`
8700
- - After `ai-toolkit update`
8701
- - Before tagging a new version (`git tag`)
8702
- - Before publishing to npm (`npm publish`)
8703
- - As a smoke test in CI/CD
8637
+ ### validate.py: stale counts
8704
8638
 
8705
- **Prerequisites:**
8706
- - Node.js >= 18, Python 3, `bats`, git
8707
- - `@softspark/ai-toolkit` installed globally
8639
+ README badges don't match the current agents/skills/tests counts.
8640
+ Update README.md and re-run.
8708
8641
 
8709
- **Time:** 10-15 minutes (full), 2 minutes (quick checklist)
8642
+ ### Eject: missing skills
8643
+
8644
+ ```bash
8645
+ ai-toolkit update # re-link missing symlinks
8646
+ ai-toolkit eject /tmp/test # retry
8647
+ ```
8710
8648
 
8711
8649
  ---
8712
8650
 
8713
- ## Quick Checklist (TL;DR)
8651
+ ## Success Criteria
8714
8652
 
8715
- The 14 core commands below must pass. Releases that change DSH must also complete Phase 10.
8653
+ | Area | Criterion |
8654
+ |------|-----------|
8655
+ | CLI | `--version` correct, `--help` full list, `status` current |
8656
+ | Health | `doctor`: 0 errors, 0 warnings, PASSED |
8657
+ | Install | `--dry-run` correct counts, `--local` all configs |
8658
+ | Quality | `validate.py --strict`: PASSED |
8659
+ | Security (baseline) | `audit_skills.py --ci`: 0 HIGH |
8660
+ | Security (SARIF) | `audit_skills.py --sarif`: valid SARIF 2.1.0 with non-empty rules array |
8661
+ | Security (permissions) | `audit_skills.py --permissions`: broad-access skills unchanged or justified in CHANGELOG |
8662
+ | Supply chain | `dist.attestations.provenance.predicateType == https://slsa.dev/provenance/v1` on npm |
8663
+ | Supply chain | All `sources.json` URL entries carry a `sha256`; `AI_TOOLKIT_STRICT_PIN=1 ai-toolkit update --dry-run` passes |
8664
+ | Tests | `npm test`: N/N passed, 0 failures |
8665
+ | Eject | Standalone `.claude/` with real files AND `output-styles/` directory |
8666
+ | Guards | Destructive commands blocked |
8667
+ | Deep coverage | `--profile full` emits every documented native surface, including Codex agents/skills/hooks and Copilot agents/skills/hooks |
8668
+ | Profile contract | Codex skills emit at every profile without `--codex-skills`; Copilot standard emits agents, skills, instructions, prompts, and hooks |
8669
+ | Idempotence | Second `install` run produces byte-identical output in every managed path |
8670
+ | Live JSON | Every generated `.json` file on disk parses as valid JSON |
8671
+ | Registry | `supported-tools-registry.md` enumerates every `scripts/generate_*.py` we ship |
8716
8672
 
8717
- ```bash
8718
- # Pre-commit (Phase 0)
8719
- python3 scripts/generate_agents_md.py > AGENTS.md # 1. Regenerate AGENTS.md
8720
- python3 scripts/generate_llms_txt.py > llms.txt # 2. Regenerate llms.txt
8721
- python3 scripts/validate.py --strict # 3. Validation passed?
8722
- npm test > /tmp/npm-test.log 2>&1 && grep -c '^ok ' /tmp/npm-test.log && ! grep -q '^not ok' /tmp/npm-test.log # 4. All tests passed? (single run, cached)
8673
+ ---
8723
8674
 
8724
- # Post-install verification (Phases 1-7)
8725
- ai-toolkit --version # 5. Version OK?
8726
- ai-toolkit status # 6. Status OK?
8727
- ai-toolkit doctor # 7. Health check passed?
8728
- ai-toolkit install --dry-run # 8. Global install OK?
8729
- python3 scripts/audit_skills.py --ci # 9. Security audit clean?
8675
+ ## kb/procedures/sop-release.md
8730
8676
 
8731
- # Supply-chain verification (Phase 8, v2.8.0+)
8732
- python3 scripts/audit_skills.py --sarif | python3 -c "import json,sys; assert json.load(sys.stdin)['version']=='2.1.0'; print('SARIF OK')" # 10. SARIF 2.1.0 well-formed?
8733
- python3 scripts/audit_skills.py --permissions | head -30 # 11. Broad-access skills reviewed?
8734
- npm view @softspark/ai-toolkit@X.Y.Z --json | python3 -c "import json,sys; d=json.load(sys.stdin); assert d['dist']['attestations']['provenance']['predicateType']=='https://slsa.dev/provenance/v1'; print('PROVENANCE OK')" # 12. Provenance attested on npm?
8735
- python3 scripts/claude_app.py verify # 13. Claude Chat/Cowork plugin contract valid?
8677
+ ---
8678
+ title: "SOP: Release Preparation"
8679
+ category: procedures
8680
+ service: ai-toolkit
8681
+ tags: [sop, release, version, publish, changelog, semver, provenance, sarif, ecosystem, shellcheck]
8682
+ version: "1.15.0"
8683
+ created: "2026-04-10"
8684
+ last_updated: "2026-09-02"
8685
+ description: "Step-by-step checklist for preparing a new ai-toolkit release — ecosystem-sync drift check, version sync, changelog, artifact regeneration, validation, branch CI, and tagging. Run BEFORE every git tag. Includes mandatory Provenance, SARIF, checksum-pin, ShellCheck, licensing, exact-tag assertions, and a green Ubuntu/macOS branch-CI gate before any release tag is created."
8686
+ ---
8736
8687
 
8737
- # Deep-coverage verification (Phase 9, v3.0.0+)
8738
- META="generate_agents_md.py|generate_llms_txt.py|generate_language_rules_skills.py"
8739
- diff <(grep -oE 'scripts/generate_[a-z_]+\.py' kb/reference/supported-tools-registry.md | sort -u) <(ls scripts/generate_*.py | grep -vE "$META" | sort -u) && echo "OK: registry matches" # 14. Registry <-> generators drift?
8688
+ # SOP: Release Preparation
8689
+
8690
+ Complete checklist for preparing a new `@softspark/ai-toolkit` release.
8691
+ Run this **before** tagging. After tagging and publishing, run the
8692
+ [Release Verification SOP](sop-release-verification.md) to smoke-test.
8693
+
8694
+ **Pipeline:**
8695
+ ```
8696
+ Ecosystem Sync SOP (drift check + generator updates)
8697
+
8698
+ Release Preparation (this SOP)
8699
+
8700
+ git tag → CI publish → Release Verification SOP
8740
8701
  ```
8741
8702
 
8742
- ---
8703
+ **Time:** 10-20 minutes (includes ecosystem sync review)
8743
8704
 
8744
- ## Phase 0: Pre-Commit & Pre-Push (2 min)
8705
+ ---
8745
8706
 
8746
- Run these commands **before every commit and push to main**. CI validates
8747
- counts but does NOT auto-regenerate — you must do it locally.
8707
+ ## Quick Checklist (TL;DR)
8748
8708
 
8749
8709
  ```bash
8750
- # 1. Regenerate generated artifacts
8710
+ # 0. Ecosystem sync (mandatory for minor/major releases; optional for patch)
8711
+ # Full procedure: kb/procedures/sop-ecosystem-sync.md
8712
+ python3 scripts/ecosystem_doctor.py --format text > /tmp/eco-report.txt
8713
+ cat /tmp/eco-report.txt
8714
+ # If drift detected: stop here, follow sop-ecosystem-sync.md Phase 2-4 to
8715
+ # classify each drift (A-F), update generators as needed, refresh snapshot,
8716
+ # THEN resume this SOP.
8717
+ python3 scripts/ecosystem_doctor.py --update # after all drift resolved
8718
+
8719
+ # 1. Decide version bump
8720
+ # patch (1.4.2 → 1.4.3): bugfix, typo, doc fix
8721
+ # minor (1.4.2 → 1.5.0): new feature, new skill, new flag, any ecosystem-class-B/F change
8722
+ # major (1.4.2 → 2.0.0): breaking change, any ecosystem-class-D removed path
8723
+
8724
+ # 2. Sync version across all files
8725
+ python3 scripts/sync_version.py X.Y.Z # if script exists, else manual
8726
+
8727
+ # 3. Write CHANGELOG.md entry
8728
+ # 4. Regenerate artifacts
8751
8729
  python3 scripts/generate_agents_md.py > AGENTS.md
8752
8730
  python3 scripts/generate_llms_txt.py > llms.txt
8753
8731
  python3 scripts/generate_llms_txt.py --full > llms-full.txt
8754
8732
 
8755
- # 2. Validate everything (catches stale counts, missing assets)
8756
- python3 scripts/validate.py --strict
8733
+ # 5. Validate + audit + SARIF + shellcheck + test + ecosystem check
8734
+ python3 scripts/validate.py --strict && python3 scripts/audit_skills.py --ci && python3 scripts/audit_skills.py --sarif > /tmp/audit.sarif && shellcheck --severity=warning app/hooks/*.sh && npm test
8757
8735
 
8758
- # 3. Security audit
8759
- python3 scripts/audit_skills.py --ci
8736
+ # 5a. Supply-chain standard (v2.8.0+) — non-negotiable
8737
+ grep -q -- '--provenance' .github/workflows/publish.yml || { echo "MISSING --provenance"; exit 1; }
8738
+ grep -q 'id-token: write' .github/workflows/publish.yml || { echo "MISSING id-token: write"; exit 1; }
8739
+ python3 scripts/audit_skills.py --permissions # review Bash/Write/Edit footprint
8760
8740
 
8761
- # 4. Run tests
8762
- npm test
8741
+ # 5b. Ecosystem gate — snapshot must be current before tag
8742
+ python3 scripts/ecosystem_doctor.py --offline --check || { echo "STALE ecosystem snapshot — re-run doctor"; exit 1; }
8763
8743
 
8764
- # 5. Stage and commit
8765
- # AGENTS.md, GEMINI.md, and .github/copilot-instructions.md are generated and
8766
- # gitignored — do NOT commit them (regenerated at prepublishOnly). Only the
8767
- # doc artifacts llms.txt / llms-full.txt are committed.
8768
- git add llms.txt llms-full.txt
8769
- git add -p # stage your other changes
8770
- git commit -m "feat: your change description"
8771
- ```
8744
+ # 5c. Licensing gate — SPDX headers, LICENSE, NOTICE, manifest consistency
8745
+ npx bats tests/test_licensing.bats || { echo "LICENSING GATE FAILED"; exit 1; }
8772
8746
 
8773
- **Why local?** Branch protection on `main` requires PRs and status checks.
8774
- CI cannot push directly to `main`, so generated artifacts must be committed
8775
- by the developer as part of their PR.
8747
+ # 6. Commit, push branch, and wait for the exact commit's full CI
8748
+ git add -A && git commit -m "chore: release vX.Y.Z"
8749
+ RELEASE_SHA=$(git rev-parse HEAD)
8750
+ git push origin main || { echo "FAIL: release commit push failed"; exit 1; }
8751
+ RUN_ID=""
8752
+ for ATTEMPT in 1 2 3 4 5; do
8753
+ RUN_ID=$(gh run list --workflow ci.yml --event push --commit "$RELEASE_SHA" \
8754
+ --limit 1 --json databaseId --jq '.[0].databaseId')
8755
+ [ -n "$RUN_ID" ] && break
8756
+ echo "Waiting for CI run registration ($ATTEMPT/5)" >&2
8757
+ [ "$ATTEMPT" -eq 5 ] || sleep 60
8758
+ done
8759
+ test -n "$RUN_ID" || { echo "FAIL: CI run not found for release commit"; exit 1; }
8760
+ gh run watch "$RUN_ID" --exit-status \
8761
+ || { echo "FAIL: release commit CI failed"; exit 1; }
8762
+ for OS in ubuntu-latest macos-latest; do
8763
+ gh run view "$RUN_ID" --json jobs \
8764
+ --jq ".jobs[] | select(.name == \"Bats test suite ($OS)\") | .conclusion" \
8765
+ | grep -qx success || { echo "FAIL: $OS test job not green"; exit 1; }
8766
+ done
8767
+ test "$(git rev-parse HEAD)" = "$RELEASE_SHA" || { echo "FAIL: local HEAD changed during CI"; exit 1; }
8768
+ git fetch origin main || { echo "FAIL: origin/main refresh failed"; exit 1; }
8769
+ test "$(git rev-parse origin/main)" = "$RELEASE_SHA" || { echo "FAIL: origin/main changed during CI"; exit 1; }
8776
8770
 
8777
- **One-liner (copy-paste):**
8778
- ```bash
8779
- python3 scripts/generate_agents_md.py > AGENTS.md && python3 scripts/generate_llms_txt.py > llms.txt && python3 scripts/generate_llms_txt.py --full > llms-full.txt && python3 scripts/validate.py --strict && python3 scripts/audit_skills.py --ci && npm test
8771
+ # 6a. Only green branch CI authorizes tag creation
8772
+ git tag vX.Y.Z "$RELEASE_SHA"
8773
+
8774
+ # 6b. Assert the tag before pushing it (v4.19.0 postmortem, Phase 7)
8775
+ test "$(git rev-parse vX.Y.Z)" = "$RELEASE_SHA" || { echo "FAIL: tag not on tested release commit"; exit 1; }
8776
+ git show --no-patch --format=%s vX.Y.Z | grep -qx "chore: release vX.Y.Z" || { echo "FAIL: tag not on release commit"; exit 1; }
8777
+
8778
+ # 6c. Push the single tag by full ref. Never --tags.
8779
+ git push origin refs/tags/vX.Y.Z
8780
8780
  ```
8781
8781
 
8782
8782
  ---
8783
8783
 
8784
- ## Phase 1: CLI & Version (1 min)
8784
+ ## Phase 0: Ecosystem Sync (MANDATORY for minor/major)
8785
8785
 
8786
- ```bash
8787
- ai-toolkit --version
8788
- ai-toolkit --help
8789
- which ai-toolkit
8790
- ```
8786
+ Before touching version numbers, confirm the toolkit is aligned with the current state of every editor / platform it integrates with. Skipping this phase ships a release whose generators may lag a month-old CLI refactor, a rename of `.cursorrules` to `.cursor/rules/`, or a new hook event we do not yet emit.
8791
8787
 
8792
- **Verify:**
8793
- - [ ] `--version` returns correct semver (e.g., `1.4.0`)
8794
- - [ ] `--help` displays full command list without errors
8795
- - [ ] `which` points to global npm bin path
8788
+ **When this phase is mandatory:**
8789
+ - Minor release (X.Y.0) — always
8790
+ - Major release (X.0.0) always
8791
+ - Patch release (X.Y.Z) only if the patch touches a generator or install flow
8796
8792
 
8797
- ---
8793
+ **When to skip:** pure doc-only patches, SOP edits, internal refactors that do not touch `scripts/generate_*` or `app/skills/*/SKILL.md`.
8798
8794
 
8799
- ## Phase 2: Global Install & Status (2 min)
8795
+ ### 0.1 Run the doctor
8800
8796
 
8801
8797
  ```bash
8802
- ai-toolkit install --dry-run
8803
- ai-toolkit status
8798
+ python3 scripts/ecosystem_doctor.py --format text | tee /tmp/eco-report.txt
8804
8799
  ```
8805
8800
 
8806
- **Verify `--dry-run`:**
8807
- - [ ] Agents: 44
8808
- - [ ] Skills: 108
8809
- - [ ] Hooks merged into settings.json
8810
- - [ ] "Other AI Tools" lists documented global targets (with `--editors`): aider, antigravity, augment, cline, codex, copilot, cursor, gemini, opencode, roo, windsurf. Scope varies: Codex uses `$CODEX_HOME` (default `~/.codex`) plus `$HOME/.agents/skills`; Copilot uses `$COPILOT_HOME` (default `~/.copilot`); cursor has only `~/.cursor/hooks.json`; antigravity has the `~/.gemini/*/skills` pointer. Cursor and Antigravity rules remain project-only.
8801
+ Output classifies every registered tool as **Clean**, **Drift**, or **Errored**.
8811
8802
 
8812
- **Verify `status`:**
8813
- - [ ] Version matches expected
8814
- - [ ] Profile: minimal/standard/strict
8815
- - [ ] Modules: list of installed modules
8816
- - [ ] Latest: up to date / update available
8803
+ ### 0.2 Act on drift
8817
8804
 
8818
- ---
8805
+ For each drifting tool, follow [sop-ecosystem-sync.md](sop-ecosystem-sync.md) Phase 2-4:
8819
8806
 
8820
- ## Phase 3: Doctor Health Check (1 min)
8807
+ | Drift class | Release impact |
8808
+ |-------------|----------------|
8809
+ | A (cosmetic reword) | No version impact — refresh snapshot, continue |
8810
+ | B (new feature — integrate) | **Minor** version bump at minimum; new generator or extended generator |
8811
+ | C (new feature — not adopted) | No impact — note in registry |
8812
+ | D (deprecation) | **Minor** or **major** depending on user impact; add migration warning |
8813
+ | E (feature promoted to default) | **Minor**; simplify generator, keep fallback comment |
8814
+ | F (feature newly globally available) | **Minor**; may require new generator or new config path |
8815
+
8816
+ If any B/D/E/F changes land in this preparation pass, mention them explicitly in the CHANGELOG entry (Phase 3) under a `Ecosystem` subsection.
8817
+
8818
+ ### 0.3 Refresh snapshot
8819
+
8820
+ Once every drift is resolved (either by code change or by re-classifying as acceptable):
8821
8821
 
8822
8822
  ```bash
8823
- ai-toolkit doctor
8823
+ python3 scripts/ecosystem_doctor.py --update
8824
8824
  ```
8825
8825
 
8826
- **Expected sections (all OK):**
8827
- - Environment: node, bash, python3, bats
8828
- - Global Install: .claude exists, agents/skills symlinks (0 broken), settings.json hooks
8829
- - Hook Scripts: all present and executable
8830
- - Hook Configuration: 14 events registered
8831
- - Generated Artifacts: AGENTS.md, llms.txt, llms-full.txt
8832
- - Planned Assets: plugin.json, benchmarks, plugin packs
8833
- - Benchmark Freshness: < 30 days
8834
- - Stale Rules: all healthy
8826
+ This writes the new baseline to `benchmarks/ecosystem-doctor-snapshot.json`. Commit it as part of the release commit.
8835
8827
 
8836
- **Verify:**
8837
- - [ ] `Errors: 0 | Warnings: 0`
8838
- - [ ] `HEALTH CHECK PASSED`
8828
+ ### 0.4 Gate
8839
8829
 
8840
- If doctor detects problems: `ai-toolkit doctor --fix` auto-repairs
8841
- (broken symlinks, non-executable hooks, missing scripts, missing llms-full.txt).
8830
+ ```bash
8831
+ python3 scripts/ecosystem_doctor.py --offline --check
8832
+ ```
8833
+
8834
+ Must exit `0`. If it exits `1`, the snapshot is stale — rerun Phase 0.3 or review the remaining drift.
8842
8835
 
8843
8836
  ---
8844
8837
 
8845
- ## Phase 4: Local Install (2 min)
8838
+ ## Phase 1: Determine Version Bump
8846
8839
 
8847
- ```bash
8848
- mkdir -p /tmp/ai-toolkit-verify && cd /tmp/ai-toolkit-verify
8849
- git init -q
8850
- ai-toolkit install --local --editors all --dry-run
8851
- cd - && rm -rf /tmp/ai-toolkit-verify
8852
- ```
8840
+ Follow [Semantic Versioning](https://semver.org/):
8853
8841
 
8854
- **Verify "Project-local" section:**
8855
- - [ ] Would create: CLAUDE.md
8856
- - [ ] Would create: .claude/settings.local.json
8857
- - [ ] Would inject: .claude/constitution.md
8858
- - [ ] Editors: all 11 listed (copilot, cursor, windsurf, cline, roo, aider, augment, antigravity, codex, gemini, opencode)
8859
- - [ ] Would generate configs for each editor (legacy + directory-based)
8860
- - [ ] Would install: .git/hooks/pre-commit
8861
- - [ ] Would inject language rules (auto-detected)
8842
+ | Change Type | Bump | Examples |
8843
+ |-------------|------|---------|
8844
+ | Bugfix, typo, doc-only | **patch** | Fix install flag, correct description |
8845
+ | New feature, skill, agent, flag | **minor** | Add `/hipaa-validate`, add `--output json` |
8846
+ | Breaking CLI change, removed skill, config format change | **major** | Rename `install` to `setup`, remove skill |
8862
8847
 
8863
- **Also test auto-detect (no --editors flag):**
8864
- ```bash
8865
- ai-toolkit install --local --dry-run
8866
- # → Editors: none (empty project has no existing configs)
8867
- ```
8848
+ **Rule:** When in doubt, bump minor.
8868
8849
 
8869
8850
  ---
8870
8851
 
8871
- ## Phase 5: Validation & Security Audit (3 min)
8852
+ ## Phase 2: Sync Version in All Files
8872
8853
 
8873
- ```bash
8874
- python3 scripts/validate.py --strict
8875
- python3 scripts/audit_skills.py --ci
8876
- ```
8854
+ The canonical version lives in `package.json`. These files **must** match:
8877
8855
 
8878
- **Verify validate.py:**
8879
- - [ ] Agents: 44, Skills: 108, Tests: exactly the current README badge count
8880
- - [ ] Hook events: 14, Hook scripts: >= 30
8881
- - [ ] Plugin packs >= 10, KB documents >= 20
8882
- - [ ] `Errors: 0 | Warnings: 0` → `VALIDATION PASSED`
8856
+ ### Mandatory sync (every release)
8883
8857
 
8884
- **Verify audit_skills.py:**
8885
- - [ ] `HIGH: 0` (MUST be zero — CI fails otherwise)
8886
- - [ ] `WARN: 0`
8887
- - [ ] `INFO: N` (acceptable broad-access skills: orchestrate, swarm, workflow)
8858
+ | File | Field | How to update |
8859
+ |------|-------|---------------|
8860
+ | `package.json` | `"version": "X.Y.Z"` | Edit directly |
8861
+ | `manifest.json` | `"version": "X.Y.Z"` | Edit directly |
8862
+ | `app/.claude-plugin/plugin.json` | `"version": "X.Y.Z"` | Edit directly |
8888
8863
 
8889
- ---
8864
+ ### Auto-synced (no manual action)
8890
8865
 
8891
- ## Phase 6: Tests (3-5 min)
8866
+ | File | Mechanism |
8867
+ |------|-----------|
8868
+ | `package-lock.json` | Regenerated by `npm install --package-lock-only` |
8869
+
8870
+ ### Conditional sync (only if the doc was modified in this release)
8871
+
8872
+ | File | Field | When to update |
8873
+ |------|-------|---------------|
8874
+ | `kb/procedures/sop-maintenance.md` | frontmatter `version:` | If SOP content changed |
8875
+ | `kb/reference/skills-catalog.md` | frontmatter `version:` | If skills added/removed |
8876
+ | `kb/reference/agents-catalog.md` | frontmatter `version:` | If agents added/removed |
8877
+ | `kb/reference/hooks-catalog.md` | frontmatter `version:` | If hooks changed |
8878
+ | `kb/reference/architecture-overview.md` | frontmatter `version:` | If architecture changed |
8879
+ | `kb/reference/distribution-model.md` | frontmatter `version:` | If install model changed |
8880
+ | `kb/reference/global-install-model.md` | frontmatter `version:` | If install model changed |
8881
+
8882
+ > **Note:** KB `version:` fields track the **document version**, not the toolkit version.
8883
+ > Only bump them when the document content actually changes in this release.
8884
+
8885
+ ### Count sync (if skills/agents/hooks changed)
8886
+
8887
+ | File | What to check |
8888
+ |------|---------------|
8889
+ | `package.json` | `"description"` — skill/agent count |
8890
+ | `README.md` | Badge counts, "What You Get" table |
8891
+ | `app/ARCHITECTURE.md` | Section headings with counts |
8892
+
8893
+ > **Tip:** `validate.py --strict` catches count drift AND version mismatches
8894
+ > (package.json vs manifest.json vs plugin.json) automatically.
8895
+ > If validation passes, counts and versions are correct.
8896
+
8897
+ ### Verification command
8898
+
8899
+ After syncing, verify all mandatory files match:
8892
8900
 
8893
8901
  ```bash
8894
- # Run ONCE, capture to file, then parse. Full suite is 669+ bats cases —
8895
- # re-running it per check (tail / grep ok / grep not ok piped separately)
8896
- # wastes minutes every release. Always cache the output.
8897
- npm test > /tmp/npm-test.log 2>&1
8898
- exit=$?
8899
- tail -3 /tmp/npm-test.log
8900
- echo "ok: $(grep -c '^ok ' /tmp/npm-test.log)"
8901
- echo "not ok: $(grep -c '^not ok' /tmp/npm-test.log)"
8902
- echo "exit: $exit"
8902
+ VERSION=$(python3 -c "import json; print(json.load(open('package.json'))['version'])")
8903
+ echo "Target: $VERSION"
8904
+ echo "manifest.json: $(python3 -c "import json; print(json.load(open('manifest.json'))['version'])")"
8905
+ echo "plugin.json: $(python3 -c "import json; print(json.load(open('app/.claude-plugin/plugin.json'))['version'])")"
8906
+ echo "package-lock.json: $(python3 -c "import json; print(json.load(open('package-lock.json'))['version'])")"
8903
8907
  ```
8904
8908
 
8905
- **Verify:**
8906
- - [ ] `exit == 0`
8907
- - [ ] `ok == expected test count` (e.g., 945 on v3.0.0)
8908
- - [ ] `not ok == 0`
8909
- - [ ] Bats runs tests in parallel (4 jobs)
8910
- - [ ] Groups: agents, autodetect, cli, generators, guards, hooks, inject,
8911
- install, kb, mcp, readme, profiles, uninstall, validate
8909
+ All four must print the same version. If not, fix before proceeding.
8910
+
8911
+ ### Public surface review
8912
8912
 
8913
- **Anti-pattern — do NOT do this:**
8914
8913
  ```bash
8915
- # Runs the full suite THREE times. Adds 1-3 min and pressures CI capacity.
8916
- npm test 2>&1 | tail -3
8917
- npm test 2>&1 | grep -c '^ok '
8918
- npm test 2>&1 | grep -c '^not ok'
8914
+ python3 scripts/surface_manifest.py --update
8915
+ git diff app/surface.json
8919
8916
  ```
8920
8917
 
8921
- **Key test areas:**
8922
- - Guards: rm -rf, DROP TABLE, git push --force blocked
8923
- - Install: idempotent, profiles, --only/--skip, orphan cleanup
8924
- - Eject: real files (not symlinks), inlined rules
8925
- - Uninstall: removes toolkit, preserves user content
8918
+ **Every line the diff removes is a breaking change.** Restore it, or take the
8919
+ deprecation path in `BACKWARD_COMPATIBILITY.md` and add a `DECISIONS.md` entry
8920
+ before the tag. Lines added are new surface being adopted into protection — that
8921
+ needs no ceremony.
8926
8922
 
8927
- ---
8923
+ Do not run `--update` to make a red build green. The check fails because something
8924
+ users depend on disappeared; regenerating the manifest deletes the evidence, not
8925
+ the problem.
8928
8926
 
8929
- ## Phase 7: Eject (1 min)
8927
+ ### Skill body budget ratchet
8930
8928
 
8931
- ```bash
8932
- mkdir -p /tmp/ai-toolkit-eject-test
8933
- cd /tmp/ai-toolkit-eject-test
8934
- ai-toolkit eject
8935
- cd - && rm -rf /tmp/ai-toolkit-eject-test
8929
+ `validate.py` prints the largest skill body on every run:
8930
+
8931
+ ```
8932
+ Body budget: largest is <skill> at <N> bytes (warn 18000, error 20000)
8936
8933
  ```
8937
8934
 
8938
- **Verify:**
8939
- - [ ] Agents copied as real files (not symlinks)
8940
- - [ ] Skills copied as real directories
8941
- - [ ] Rules inlined into CLAUDE.md
8942
- - [ ] constitution.md and ARCHITECTURE.md copied
8943
- - [ ] `output-styles/` directory present (v2.7.1+)
8935
+ Once that number sits at least 2000 bytes under `SKILL_BODY_BUDGET_WARN`, lower
8936
+ `SKILL_BODY_BUDGET_WARN` by 2000 in `scripts/validate.py` and ship the tightened
8937
+ threshold with the release. Target floor is 12000.
8944
8938
 
8945
- ---
8939
+ Two rules, both learned the hard way:
8946
8940
 
8947
- ## Phase 8: Supply-Chain Verification (2 min, v2.8.0+)
8941
+ - **Never lower a threshold in the same change that something violates it.** Split
8942
+ the offending skill into `SKILL.md` + `reference/` first, prove it with
8943
+ `python3 scripts/check_split.py <skill> --before <pre-split SKILL.md>`, then
8944
+ tighten in a follow-up.
8945
+ - **Never raise a threshold to make a red build green.** A body over budget means
8946
+ detail that belongs in `reference/` is loading on every trigger match, including
8947
+ the accidental ones. Raising the number hides the cost, it does not remove it.
8948
8948
 
8949
- These checks enforce the v2.8.0 security standard on a freshly-published release. Run AFTER the `publish.yml` workflow completes on the tag.
8949
+ If the largest body has not moved since the previous release, leave the threshold
8950
+ alone and say so in the release notes. A ratchet that never advances is honest;
8951
+ one that advances past reality is not.
8950
8952
 
8951
- ### 8.1 Provenance attestation on npm
8953
+ ---
8952
8954
 
8953
- ```bash
8954
- VERSION="X.Y.Z" # the tag just published
8955
- npm view "@softspark/ai-toolkit@${VERSION}" --json \
8956
- | python3 -c "import json, sys; d=json.load(sys.stdin); att=d['dist'].get('attestations', {}); assert att.get('provenance', {}).get('predicateType') == 'https://slsa.dev/provenance/v1', f'NO PROVENANCE for {d[\"version\"]}'; print(f'PROVENANCE OK: {att[\"url\"]}')"
8957
- ```
8955
+ ## Phase 3: Write CHANGELOG Entry
8958
8956
 
8959
- **Verify:**
8960
- - [ ] Exit 0 and prints `PROVENANCE OK: https://registry.npmjs.org/...`
8961
- - [ ] `https://www.npmjs.com/package/@softspark/ai-toolkit/v/${VERSION}` shows the green "Provenance" badge
8957
+ Add entry at the top of `CHANGELOG.md` (after the header, before previous release):
8962
8958
 
8963
- **If provenance is missing:** the `publish.yml` ran without `id-token: write` or `--provenance`. Restore them and cut a patch release — a silently unsigned publish is a regression against the v2.8.0 standard.
8959
+ ```markdown
8960
+ ## vX.Y.Z — Short Title (YYYY-MM-DD)
8964
8961
 
8965
- ### 8.2 Audit SARIF output (for GHAS ingest)
8962
+ ### Added
8963
+ - **Feature name** — description
8966
8964
 
8967
- ```bash
8968
- python3 scripts/audit_skills.py --sarif > /tmp/audit.sarif
8969
- python3 -c "import json; d=json.load(open('/tmp/audit.sarif')); assert d['version']=='2.1.0' and d['runs'][0]['tool']['driver']['name']=='ai-toolkit-audit-skills'; print(f'SARIF OK: {len(d[\"runs\"][0][\"results\"])} results across {len(d[\"runs\"][0][\"tool\"][\"driver\"][\"rules\"])} rules')"
8965
+ ### Changed
8966
+ - **What changed** old behavior → new behavior
8967
+
8968
+ ### Fixed
8969
+ - **Bug description** — what was broken and how it's fixed
8970
+
8971
+ ### Removed
8972
+ - **What was removed** — migration path if any
8970
8973
  ```
8971
8974
 
8972
- **Verify:**
8973
- - [ ] Valid SARIF 2.1.0
8974
- - [ ] In the publishing repo, the CI job uploads `audit.sarif` via `github/codeql-action/upload-sarif@v3` so findings appear in the Security tab
8975
+ **Rules:**
8976
+ - Use **bold** for feature names
8977
+ - Start descriptions with a verb (Added, Changed, Fixed, Removed)
8978
+ - Reference skill names with backticks and slash: `/hipaa-validate`
8979
+ - Include script names: `scripts/hipaa_scan.py`
8980
+ - Include count changes: `Skill count: 91 → 92`
8981
+ - Date format: `YYYY-MM-DD`
8982
+ - Title: short, descriptive, no version number repetition
8975
8983
 
8976
- ### 8.3 Per-skill permissions report
8984
+ ### Update README "What's New" section
8977
8985
 
8978
- ```bash
8979
- python3 scripts/audit_skills.py --permissions | head -40
8980
- ```
8986
+ **MANDATORY on every release.** Update the `## What's New in vX.Y.Z` section in `README.md`:
8981
8987
 
8982
- **Verify:**
8983
- - [ ] Bash skill count has NOT jumped unexpectedly since the previous release
8984
- - [ ] Any newly added entry under `Skills with Bash + Write + Edit` matches a CHANGELOG bullet that justifies the broad scope
8985
- - [ ] JSON form (`--permissions --json`) is available for automated drift dashboards
8988
+ 1. Change the heading version: `## What's New in vX.Y.Z`
8989
+ 2. Replace bullet points with 3-5 highlights from this release
8990
+ 3. **Keep only the latest version block.** Delete the previous `## What's New in vA.B.C` section(s). README is the shop window, not the archive — users see the current release, full history lives in `CHANGELOG.md`.
8991
+ 4. Keep the `See [CHANGELOG.md](CHANGELOG.md) for full history.` link directly below the bullet list.
8986
8992
 
8987
- ### 8.4 URL-sourced rules/hooks are checksum-pinned
8993
+ > **Warning:** This section is the first thing users see after the badges.
8994
+ > A stale version here (e.g., "What's New in v2.1.3" when shipping v2.3.0)
8995
+ > signals an unmaintained project. Do NOT skip this step.
8988
8996
 
8989
- `sources.json` is an envelope of the form `{"schema_version": 1, "rules"|"hooks": {...}}`, so the jq filter must pick the nested map before piping into the pin assertion. The URL-count in the success message ignores local `path:`-only entries only entries with a `url:` field are pinned and counted.
8997
+ > **Single-version rule:** README.md must contain **exactly one** `## What's New in vX.Y.Z` heading at any time. If you find multiple stacked (e.g. v2.6.1 + v2.6.0 + v2.5.0), that is a SOP driftcollapse to the latest on the next release commit.
8998
+
8999
+ ---
9000
+
9001
+ ## Phase 4: Regenerate Artifacts
9002
+
9003
+ Use the npm scripts, not the generators directly:
8990
9004
 
8991
9005
  ```bash
8992
- jq '.rules // .hooks // {}' ~/.softspark/ai-toolkit/rules/sources.json 2>/dev/null \
8993
- | python3 -c "import json, sys; d=json.load(sys.stdin) or {}; bad=[n for n,v in d.items() if v.get('url') and not v.get('sha256')]; assert not bad, f'UNPINNED: {bad}'; url_n=sum(1 for v in d.values() if v.get('url')); print(f'RULE PIN OK: {url_n} URL rules, all with sha256')"
8994
- jq '.hooks // {}' ~/.softspark/ai-toolkit/hooks/external/sources.json 2>/dev/null \
8995
- | python3 -c "import json, sys; d=json.load(sys.stdin) or {}; bad=[n for n,v in d.items() if v.get('url') and not v.get('sha256')]; assert not bad, f'UNPINNED: {bad}'; url_n=sum(1 for v in d.values() if v.get('url')); print(f'HOOK PIN OK: {url_n} URL hooks, all with sha256')"
9006
+ npm run generate:agents # AI_TOOLKIT_NO_CUSTOM_RULES=1 python3 scripts/generate_agents_md.py > AGENTS.md
9007
+ npm run generate:llms # llms.txt + llms-full.txt
8996
9008
  ```
8997
9009
 
8998
- **Verify:**
8999
- - [ ] Both commands print `... PIN OK`
9000
- - [ ] If any entry is unpinned, the `register_url_source()` call missed passing `content=` — fix the call-site and retag
9010
+ `generate:agents` sets `AI_TOOLKIT_NO_CUSTOM_RULES=1`. Running
9011
+ `generate_agents_md.py` bare picks up whatever is registered in the maintainer's
9012
+ own `~/.softspark/ai-toolkit/rules/`, which then ships inside `AGENTS.md`.
9001
9013
 
9002
- ### 8.5 Strict-pin smoke test (optional but recommended)
9014
+ Check if anything actually changed:
9003
9015
 
9004
9016
  ```bash
9005
- AI_TOOLKIT_STRICT_PIN=1 ai-toolkit update --dry-run
9017
+ git diff --stat AGENTS.md llms.txt llms-full.txt
9006
9018
  ```
9007
9019
 
9008
- **Verify:**
9009
- - [ ] Exit 0, no `CHECKSUM CHANGED` line
9010
- - [ ] If a checksum change was intentional (e.g. upstream rule update), document it in the CHANGELOG entry before tagging
9020
+ If no diff, the artifacts are already current. If there is a diff, stage them.
9011
9021
 
9012
9022
  ---
9013
9023
 
9014
- ## Phase 9: Deep-Coverage Checks (v3.0.0+)
9024
+ ## Phase 5: Validate, Audit, Test
9015
9025
 
9016
- These verify the native-surface generators shipped in v3.0.0 actually emit the right files for the right profiles, and that the tool registry stays in sync with shipped generators.
9026
+ Run the full quality gate:
9017
9027
 
9018
- > **Safety warning — HOME-scoped writes:** Running `--profile full` with `augment` in the editor list writes to `$HOME/.augment/settings.json` (Augment stores hooks under HOME, not per-project). Use `--dry-run` for verification unless you intend to carry ai-toolkit hook entries on this machine. The generator is marker-safe (only rewrites its own `_source: ai-toolkit` entries) but is still a side-effect.
9028
+ ```bash
9029
+ python3 scripts/validate.py --strict
9030
+ python3 scripts/audit_skills.py --ci
9031
+ python3 scripts/audit_skills.py --sarif > audit.sarif # MANDATORY — GHAS ingest
9032
+ python3 scripts/audit_skills.py --permissions # review Bash/Write/Edit footprint
9019
9033
 
9020
- ### 9.1 `--profile full` emits every native surface
9034
+ # ShellCheck on hooks (added in 1.11.0). Mirrors the ci.yml "ShellCheck hooks"
9035
+ # job. NOT run by validate.py, npm test, OR publish.yml — so a hook with a
9036
+ # ShellCheck warning passes every other gate AND still publishes on tag while
9037
+ # turning main CI red. Run it here, before tagging.
9038
+ shellcheck --severity=warning app/hooks/*.sh && echo "OK: shellcheck clean"
9021
9039
 
9022
- ```bash
9023
- D=/tmp/aitk-profile-full-${RANDOM} && mkdir -p "$D" && cd "$D" && git init -q
9024
- ai-toolkit install --local --editors cursor,windsurf,gemini,augment,codex,copilot \
9025
- --profile full --dry-run 2>&1 \
9026
- | grep -E "\\.cursor/(hooks\\.json|agents)|\\.devin/hooks\\.v1\\.json|\\.gemini/(settings\\.json|commands)|\\.augment/(agents|commands)|\\.codex/(agents|hooks)|\\.agents/skills|\\.github/(agents|skills|instructions|prompts|hooks)"
9040
+ # Registry / generator drift (added in 1.10.0). Meta-generators excluded.
9041
+ META="generate_agents_md.py|generate_llms_txt.py|generate_language_rules_skills.py"
9042
+ diff \
9043
+ <(grep -oE 'scripts/generate_[a-z_]+\.py' kb/reference/supported-tools-registry.md | sort -u) \
9044
+ <(ls scripts/generate_*.py | grep -vE "$META" | sort -u) \
9045
+ && echo "OK: registry matches filesystem" \
9046
+ || { echo "DRIFT: update supported-tools-registry.md before tagging"; exit 1; }
9047
+
9048
+ # Stage first IF this release adds or deletes a kb/ file. The test
9049
+ # "npm package KB files match the tracked release set" compares `git ls-files
9050
+ # kb` (the index) against what `npm pack` sees (the working tree), so an
9051
+ # unstaged addition reads as "extra" and an unstaged deletion as "missing".
9052
+ # Phase 6 stages, and it runs after this one, so the ordering fails the test
9053
+ # for any release that touches the KB. Staging early costs nothing.
9054
+ git status --porcelain kb/ | grep -qE '^(\?\?| D|\?M)' && git add -A kb/
9055
+
9056
+ # Run npm test ONCE, cache output, parse from file. The suite is 1400+ bats
9057
+ # cases — rerunning it per check wastes minutes. Do not pipe npm test into
9058
+ # tail/grep multiple times in the same session.
9059
+ npm test > /tmp/npm-test.log 2>&1
9060
+ tail -3 /tmp/npm-test.log
9061
+ echo "ok: $(grep -c '^ok ' /tmp/npm-test.log) | not ok: $(grep -c '^not ok' /tmp/npm-test.log)"
9027
9062
  ```
9028
9063
 
9029
- **Verify** — at least the following lines appear:
9030
- - [ ] `.cursor/hooks.json` and `.cursor/agents/`
9031
- - [ ] `.devin/hooks.v1.json`
9032
- - [ ] `.gemini/settings.json` hooks AND `.gemini/commands/`
9033
- - [ ] `.augment/agents/` + `.augment/commands/` + `$HOME/.augment/settings.json`
9034
- - [ ] `.codex/hooks.json` + `.codex/hooks/` + `.codex/agents/`
9035
- - [ ] `.agents/skills/` (Codex native discovery path)
9036
- - [ ] `.github/agents/` + `.github/skills/` + `.github/instructions/` + `.github/prompts/` + `.github/hooks/`
9064
+ **Expected results:**
9065
+ - `validate.py`: `Errors: 0 | Warnings: 0 | VALIDATION PASSED`
9066
+ - `audit_skills.py --ci`: `HIGH: 0 | WARN: 0` (INFO is acceptable)
9067
+ - `audit_skills.py --sarif`: valid JSON, non-empty `runs[0].tool.driver.rules`
9068
+ - `audit_skills.py --permissions`: review `Skills with Bash + Write + Edit` list — any newly-added skill with broad access MUST be justified in the CHANGELOG entry
9069
+ - `shellcheck --severity=warning app/hooks/*.sh`: no output, exit 0. A common false positive is `SC2034` on `INPUT` or env vars (e.g. `AI_TOOLKIT_HOOK_FORMAT`) that a *sourced* helper (`_hook-io.sh`) consumes — ShellCheck cannot see cross-file use. Fix with a `# shellcheck disable=SC2034` directive or `export`, matching `guard-destructive.sh`. Never tag with a red ShellCheck.
9070
+ - Registry drift: `OK: registry matches filesystem`. If `DRIFT:` appears, add the missing `scripts/generate_*.py` rows to `kb/reference/supported-tools-registry.md` before tagging.
9071
+ - `npm test`: `1..N` with zero `not ok` (read from the cached `/tmp/npm-test.log`, do not rerun)
9037
9072
 
9038
- ### 9.2 Codex skills are profile-independent
9073
+ > **Why this matters (v4.5.1 postmortem):** `publish.yml` runs only `validate.py` + `npm test`, so it published v4.5.0 even though the `main` CI `ShellCheck hooks` job was red on two `SC2034` warnings in a new hook. The publish workflow does **not** depend on the CI workflow. Until that is fixed, ShellCheck is a manual pre-tag gate — run it here every time.
9039
9074
 
9075
+ **One-liner:**
9040
9076
  ```bash
9041
- D=/tmp/aitk-codex-skills-${RANDOM} && mkdir -p "$D" && cd "$D" && git init -q
9042
- ai-toolkit install --local --editors codex --profile minimal --dry-run 2>&1 \
9043
- | grep -q "Would generate: .agents/skills" && echo "OK: Codex skills emit at minimal"
9044
- ai-toolkit install --local --editors codex --profile standard --codex-skills --dry-run 2>&1 \
9045
- | grep -q "Would refresh: .agents/skills" && echo "OK: legacy flag explicitly refreshes the same catalog"
9077
+ python3 scripts/validate.py --strict && python3 scripts/audit_skills.py --ci && python3 scripts/audit_skills.py --sarif > audit.sarif && shellcheck --severity=warning app/hooks/*.sh && diff <(grep -oE 'scripts/generate_[a-z_]+\.py' kb/reference/supported-tools-registry.md | sort -u) <(ls scripts/generate_*.py | grep -vE 'generate_agents_md\.py|generate_llms_txt\.py|generate_language_rules_skills\.py' | sort -u) && npm test
9046
9078
  ```
9047
9079
 
9048
- **Verify:**
9049
- - [ ] Selecting Codex emits all skills under `.agents/skills/` at every profile
9050
- - [ ] `--codex-skills` remains an explicit refresh alias; it is not required for installation
9051
- - [ ] No profile emits `.codex/skills/`
9080
+ **If tests fail:** Fix the issue, do NOT skip. Common failures:
9081
+ - Stale counts re-run `generate:all` or fix README/ARCHITECTURE
9082
+ - Missing frontmatter add to new KB docs
9083
+ - Broken symlink `ai-toolkit doctor --fix`
9052
9084
 
9053
- ### 9.3 Breaking-change surfaces land on `--profile standard`
9085
+ ### Phase 5c: Licensing Gate (v4.20.0+)
9054
9086
 
9055
- The standard profile must include:
9056
- - Copilot scoped instructions, prompts, custom agents, portable skills, and hooks
9057
- - Gemini hooks (`.gemini/settings.json`)
9087
+ The project is Apache-2.0. Attribution only works if the artefact actually
9088
+ carries it, and every part of that is mechanically checkable.
9058
9089
 
9059
9090
  ```bash
9060
- D=/tmp/aitk-breaking-${RANDOM} && mkdir -p "$D" && cd "$D" && git init -q
9061
- ai-toolkit install --local --editors copilot,gemini --profile standard --dry-run 2>&1 \
9062
- | tee /tmp/aitk-breaking.log
9063
- grep -q "\\.github/instructions/" /tmp/aitk-breaking.log && echo "OK: Copilot dir layout at standard"
9064
- grep -q "\\.github/agents/.*\\.github/skills/" /tmp/aitk-breaking.log && echo "OK: Copilot agents and skills at standard"
9065
- grep -q "\\.github/hooks/" /tmp/aitk-breaking.log && echo "OK: Copilot hooks at standard"
9066
- grep -q "\\.gemini/settings\\.json hooks" /tmp/aitk-breaking.log && echo "OK: Gemini hooks at standard"
9091
+ # The whole gate, enforced in CI. Run it here so a failure is caught before tagging.
9092
+ npx bats tests/test_licensing.bats
9093
+ ```
9094
+
9095
+ The seven assertions, and why each exists:
9096
+
9097
+ | Check | Fails when |
9098
+ |---|---|
9099
+ | Every shipped source file carries an SPDX header | A new `.py`/`.sh`/`.js`/`.bats` file was added without one — the common case, and the reason this is a test rather than a habit |
9100
+ | Headers name Apache-2.0 and nothing else | A file was copied in from an MIT/GPL source with its own header intact |
9101
+ | **No** markdown file carries an SPDX header | Someone "helpfully" ran the header script over `app/skills/` — headers there sit above parsed frontmatter and bill every session for it |
9102
+ | `LICENSE` is the complete Apache 2.0 text | The file was truncated or replaced with a summary |
9103
+ | `NOTICE` carries attribution, the source URL, §4(d) and the MIT-era notice | The attribution mechanism was gutted |
9104
+ | `LICENSE` **and** `NOTICE` ship in the npm package | `package.json` `files` lost an entry — a NOTICE that never reaches the consumer cannot satisfy §4(d) |
9105
+ | Every manifest declaring a licence declares Apache-2.0 | `package.json`, `manifest.json`, `plugin.json` and `package-lock.json` drifted apart |
9106
+
9107
+ **Adding source files in this release?** The header goes *after* the shebang,
9108
+ never before it. Short SPDX form:
9109
+
9110
+ ```
9111
+ # SPDX-License-Identifier: Apache-2.0
9112
+ # Copyright 2024-2026 Lukasz Krzemien (biuro@softspark.eu)
9113
+ # Source: https://github.com/softspark/ai-toolkit
9067
9114
  ```
9068
9115
 
9069
- **Verify all four lines print `OK:`**. If any is missing, native Copilot or Gemini coverage has regressed.
9116
+ `//` for JavaScript. Full convention and the reasoning behind the markdown
9117
+ exclusion: [Licensing](../reference/licensing.md).
9070
9118
 
9071
- ### 9.4 Install is idempotent
9119
+ **Changing the licence itself?** Do not hand-type the licence text. Take it
9120
+ verbatim from a published copy and cross-verify against a second independent
9121
+ copy before writing `LICENSE` — a rendered or summarised licence is not the
9122
+ licence. Prior releases stay under their original terms; a licence change
9123
+ applies going forward and revokes nothing already granted.
9124
+
9125
+ ### Phase 5a: Supply-Chain Hardening Verification (v2.8.0+)
9126
+
9127
+ These checks enforce the security standard introduced in v2.8.0. Do NOT tag a release until all pass.
9128
+
9129
+ **1. Publish workflow emits provenance:**
9072
9130
 
9073
9131
  ```bash
9074
- D=/tmp/aitk-idem-${RANDOM} && mkdir -p "$D" && cd "$D" && git init -q
9075
- # Sort file paths before hashing — find traversal order follows inode order,
9076
- # which can shift between runs even when content is byte-identical, producing
9077
- # false FAIL signals.
9078
- ai-toolkit install --local --editors cursor,gemini --profile full >/dev/null 2>&1
9079
- SHA1=$(find .cursor .gemini -type f -print0 | LC_ALL=C sort -z | xargs -0 shasum | shasum | awk '{print $1}')
9080
- ai-toolkit install --local --editors cursor,gemini --profile full >/dev/null 2>&1
9081
- SHA2=$(find .cursor .gemini -type f -print0 | LC_ALL=C sort -z | xargs -0 shasum | shasum | awk '{print $1}')
9082
- [ "$SHA1" = "$SHA2" ] && echo "OK: idempotent" || echo "FAIL: install is not idempotent"
9132
+ grep -E '\-\-provenance|id-token: write' .github/workflows/publish.yml
9083
9133
  ```
9084
9134
 
9085
- **Verify:** prints `OK: idempotent`. A second run must produce byte-identical files in every managed path.
9086
-
9087
- ### 9.5 Live-install JSON outputs parse
9135
+ - [ ] Both markers present (`--provenance` flag + `id-token: write` permission)
9136
+ - [ ] Any PR that changes `publish.yml` REQUIRES an approved security review
9088
9137
 
9089
- The bats suite validates JSON shape at generation time. This re-checks that what actually landed on disk after a live install parses without errors.
9138
+ **2. URL-sourced rules and hooks are checksum-pinned:**
9090
9139
 
9091
9140
  ```bash
9092
- D=/tmp/aitk-json-${RANDOM} && mkdir -p "$D" && cd "$D" && git init -q
9093
- ai-toolkit install --local --editors cursor,windsurf,gemini,augment,codex,copilot --profile full >/dev/null 2>&1
9094
- for f in .cursor/hooks.json .devin/hooks.v1.json .gemini/settings.json .codex/hooks.json .github/hooks/ai-toolkit.json "$HOME/.augment/settings.json"; do
9095
- [ -f "$f" ] && python3 -c "import json; json.load(open('$f'))" && echo "OK: $f"
9096
- done
9141
+ # On a machine that has consumed URL rules/hooks at least once
9142
+ # (schema_version 1: entries live under the .rules / .hooks key):
9143
+ jq '.rules | to_entries | map(select(.value.url != null and (.value.sha256 // "" | length) == 0))' ~/.softspark/ai-toolkit/rules/sources.json
9144
+ jq '.hooks | to_entries | map(select(.value.url != null and (.value.sha256 // "" | length) == 0))' ~/.softspark/ai-toolkit/hooks/external/sources.json
9097
9145
  ```
9098
9146
 
9099
- **Verify:** each emitted file prints `OK: <path>`. Any `json.decoder.JSONDecodeError` means the merge logic corrupted the output.
9100
-
9101
- ### 9.6 Registry / generator drift check
9147
+ - [ ] Both queries return empty arrays (every URL entry has a `sha256`)
9148
+ - [ ] If not, run `ai-toolkit update` to backfill missing hashes before tagging
9102
9149
 
9103
- `kb/reference/supported-tools-registry.md` should enumerate every per-editor `scripts/generate_*.py` we ship. Meta-generators (`generate_agents_md.py`, `generate_llms_txt.py`) are excluded — they produce docs/artifacts, not editor configs.
9150
+ **3. Audit SARIF output is well-formed:**
9104
9151
 
9105
9152
  ```bash
9106
- META="generate_agents_md.py|generate_llms_txt.py|generate_language_rules_skills.py"
9107
- REG=$(grep -oE 'scripts/generate_[a-z_]+\.py' kb/reference/supported-tools-registry.md | sort -u)
9108
- FS=$(ls scripts/generate_*.py | grep -vE "$META" | sort -u)
9109
- diff <(echo "$REG") <(echo "$FS") && echo "OK: registry matches filesystem" || echo "DRIFT: update supported-tools-registry.md"
9153
+ python3 scripts/audit_skills.py --sarif | python3 -c "import json, sys; d=json.load(sys.stdin); assert d['version']=='2.1.0' and d['runs'][0]['tool']['driver']['name']; print('SARIF OK')"
9110
9154
  ```
9111
9155
 
9112
- **Verify:** prints `OK: registry matches filesystem`. If not, add the missing rows to the registry before tagging the next release.
9156
+ - [ ] Prints `SARIF OK`
9157
+ - [ ] If the script ever grows new rule classes, extend the SARIF `rules[]` coverage before releasing
9113
9158
 
9114
- ### 9.7 Claude Chat / Cowork plugin validates and exports
9159
+ **4. Strict-pin mode passes on CI** (optional, recommended for stable branches):
9115
9160
 
9116
9161
  ```bash
9117
- python3 scripts/claude_app.py verify
9118
- D=/tmp/aitk-claude-app-${RANDOM}
9119
- python3 scripts/claude_app.py export --output "$D.zip" --no-custom-rules
9120
- python3 -c "import zipfile; z=zipfile.ZipFile('$D.zip'); assert '.claude-plugin/plugin.json' in z.namelist(); assert 'claude-app/skills/ai-toolkit-rules/SKILL.md' in z.namelist(); print('OK: Claude app archive')"
9162
+ AI_TOOLKIT_STRICT_PIN=1 ai-toolkit update --dry-run
9121
9163
  ```
9122
9164
 
9123
- **Verify:** the official validator exits 0; the archive contains the manifest,
9124
- app-native rules skill, bundled agents/skills, and plugin-relative Cowork hooks.
9165
+ - [ ] Exit 0, no `CHECKSUM CHANGED` line
9166
+ - [ ] Any unexpected upstream change blocks the release until explicitly approved
9125
9167
 
9126
9168
  ---
9127
9169
 
9128
- ## Phase 10: Explicit DSH Qualification (v4.30.0+)
9129
-
9130
- Run this phase whenever the release changes the `dsh` target, package pins, preset lifecycle, or DSH compatibility documentation. Use a new task-specific `DSH_HOME`; never replace `HOME` or reuse a regular profile.
9170
+ ## Phase 6: Commit
9131
9171
 
9132
- Prerequisites: DSH `0.1.1-rc.2`, pnpm `>=11.7.0,<12.0.0`, Codex logged in through ChatGPT, Claude Code logged in natively, and GitHub Copilot CLI logged in natively. Do not supply provider API keys.
9172
+ Stage all release files:
9133
9173
 
9134
9174
  ```bash
9135
- DSH_SMOKE_ROOT="$(mktemp -d)"
9136
- export DSH_HOME="$DSH_SMOKE_ROOT/dsh-home"
9137
-
9138
- ai-toolkit dsh install --profile web
9139
- ai-toolkit dsh doctor --profile web
9140
- dsh --profile web --host 127.0.0.1 --port 0 --no-open
9175
+ git add package.json manifest.json app/.claude-plugin/plugin.json
9176
+ git add package-lock.json
9177
+ git add CHANGELOG.md
9178
+ git add llms.txt llms-full.txt
9179
+ # NOTE: AGENTS.md, GEMINI.md, and .github/copilot-instructions.md are generated
9180
+ # editor configs and are gitignored — do NOT commit them. `prepublishOnly` runs
9181
+ # `npm run generate:all`, so the shipped package (which lists AGENTS.md in
9182
+ # package.json `files`) gets a fresh copy at publish time.
9183
+ git add -p # review and stage any other changes
9141
9184
  ```
9142
9185
 
9143
- In a new `softspark-orchestrator` session, select the `codex` provider and run two standalone marker prompts:
9186
+ Commit:
9144
9187
 
9145
- 1. `subagent_claude_code` returns an exact child marker, then the Codex parent returns its exact completion marker.
9146
- 2. `subagent_gemini_copilot` returns an exact child marker, then the Codex parent returns its exact completion marker.
9188
+ ```bash
9189
+ git commit -m "chore: release vX.Y.Z"
9190
+ ```
9147
9191
 
9148
- Stop DSH, then remove only the managed profile artifacts:
9192
+ ---
9193
+
9194
+ ## Phase 7: Push Branch, Verify CI, Tag, and Push Tag
9149
9195
 
9150
9196
  ```bash
9151
- ai-toolkit dsh uninstall --profile web --yes
9152
- ```
9197
+ RELEASE_SHA=$(git rev-parse HEAD)
9198
+ git push origin main || { echo "FAIL: release commit push failed"; exit 1; }
9153
9199
 
9154
- **Verify:** both tool results have `isError=false`, both turns end as `completed`, `doctor` reports no recovery requirement before uninstall, and an unrelated preset fixture remains unchanged. Preserve only redacted event sequence evidence; never attach credentials, auth files, or full private prompts.
9200
+ # Bind the gate to the exact release commit. GitHub run registration is
9201
+ # asynchronous, so retry at most five times and log every attempt.
9202
+ RUN_ID=""
9203
+ for ATTEMPT in 1 2 3 4 5; do
9204
+ RUN_ID=$(gh run list --workflow ci.yml --event push --commit "$RELEASE_SHA" \
9205
+ --limit 1 --json databaseId --jq '.[0].databaseId')
9206
+ [ -n "$RUN_ID" ] && break
9207
+ echo "Waiting for CI run registration ($ATTEMPT/5)" >&2
9208
+ [ "$ATTEMPT" -eq 5 ] || sleep 60
9209
+ done
9210
+ test -n "$RUN_ID" || { echo "FAIL: CI run not found for release commit"; exit 1; }
9211
+ gh run watch "$RUN_ID" --exit-status \
9212
+ || { echo "FAIL: release commit CI failed"; exit 1; }
9155
9213
 
9156
- ---
9214
+ # publish.yml is Ubuntu-only. Require both full CI matrix jobs before creating
9215
+ # the tag so a green publish job cannot bypass a red macOS test.
9216
+ for OS in ubuntu-latest macos-latest; do
9217
+ gh run view "$RUN_ID" --json jobs \
9218
+ --jq ".jobs[] | select(.name == \"Bats test suite ($OS)\") | .conclusion" \
9219
+ | grep -qx success || { echo "FAIL: $OS test job not green"; exit 1; }
9220
+ done
9157
9221
 
9158
- ## Troubleshooting
9222
+ # Refuse to tag if either local or remote main moved while CI was running.
9223
+ test "$(git rev-parse HEAD)" = "$RELEASE_SHA" \
9224
+ || { echo "FAIL: local HEAD changed during CI"; exit 1; }
9225
+ git fetch origin main || { echo "FAIL: origin/main refresh failed"; exit 1; }
9226
+ test "$(git rev-parse origin/main)" = "$RELEASE_SHA" \
9227
+ || { echo "FAIL: origin/main changed during CI"; exit 1; }
9159
9228
 
9160
- ### `ai-toolkit: command not found`
9229
+ git tag vX.Y.Z "$RELEASE_SHA"
9161
9230
 
9162
- ```bash
9163
- npm install -g @softspark/ai-toolkit
9164
- # or check PATH:
9165
- export PATH="$(npm config get prefix)/bin:$PATH"
9231
+ # Assert the tag before pushing it. Both checks are one line each and both
9232
+ # have caught a real broken release.
9233
+ test "$(git rev-parse vX.Y.Z)" = "$RELEASE_SHA" \
9234
+ || { echo "FAIL: tag is not on the tested release commit"; exit 1; }
9235
+ git show --no-patch --format=%s vX.Y.Z | grep -qx "chore: release vX.Y.Z" \
9236
+ || { echo "FAIL: tag is not on the chore: release commit"; exit 1; }
9237
+
9238
+ # Push the single release tag by its full ref.
9239
+ git push origin refs/tags/vX.Y.Z
9166
9240
  ```
9167
9241
 
9168
- ### Doctor: broken symlinks
9242
+ **Why branch CI comes before the tag (v4.30.2 postmortem).** The publish
9243
+ workflow runs only on Ubuntu and can publish while the separate macOS matrix job
9244
+ is red. `v4.30.2` exposed this with a macOS-only DSH timing failure. A release
9245
+ commit must therefore pass the complete Ubuntu/macOS branch workflow before its
9246
+ tag exists; a successful publish workflow is not a substitute for green CI.
9169
9247
 
9170
- ```bash
9171
- ai-toolkit doctor --fix # auto-repair
9172
- ai-toolkit update # or full re-install
9173
- ```
9248
+ **Why the assertions (v4.19.0 postmortem).** v4.19.0 was tagged on a commit
9249
+ that contained only a KB document and still carried `package.json` version
9250
+ `4.18.0`; the actual release sat in the commit above it under a recycled
9251
+ `fix:` message. `publish.yml` fired, tried to publish a version already on
9252
+ npm, and failed. Nothing on npm, a tag pointing at the wrong tree, and the
9253
+ only way out was rewriting a pushed commit. Both assertions above catch this
9254
+ in under a second. Run them.
9174
9255
 
9175
- ### Tests fail: missing bats
9256
+ **Never `git push --tags`.** It pushes every local tag at once, and GitHub
9257
+ suppresses tag-triggered workflow runs when many tags arrive in a single push
9258
+ — the workflow silently does not fire and nothing publishes. Push the single
9259
+ release tag by its full ref, as above. (Sibling evidence: this is exactly how
9260
+ rag-mcp's `1.0.3` image build was skipped, when a `--tags` push carried 37
9261
+ tags at once.)
9176
9262
 
9177
- ```bash
9178
- brew install bats-core # macOS
9179
- npm install -g bats # cross-platform
9180
- ```
9263
+ This triggers `.github/workflows/publish.yml` which:
9264
+ 1. Runs `validate.py --strict`
9265
+ 2. Runs `npm test`
9266
+ 3. Publishes to npm as `@softspark/ai-toolkit@X.Y.Z` with `--provenance` (SLSA v1 build attestation)
9181
9267
 
9182
- ### validate.py: stale counts
9268
+ **Provenance is non-negotiable.** If `id-token: write` permission or the `--provenance` flag is missing from `publish.yml`, fix it BEFORE tagging — an unsigned release is a regression against the v2.8.0 standard.
9183
9269
 
9184
- README badges don't match the current agents/skills/tests counts.
9185
- Update README.md and re-run.
9270
+ **After CI completes:** Run the [Release Verification SOP](sop-release-verification.md)
9271
+ to smoke-test the published package AND verify the provenance attestation landed on npm.
9186
9272
 
9187
- ### Eject: missing skills
9273
+ ---
9274
+
9275
+ ## Rollback
9276
+
9277
+ If a bad release was published:
9188
9278
 
9189
9279
  ```bash
9190
- ai-toolkit update # re-link missing symlinks
9191
- ai-toolkit eject /tmp/test # retry
9280
+ # Unpublish from npm (within 72h)
9281
+ npm unpublish @softspark/ai-toolkit@X.Y.Z
9282
+
9283
+ # Or deprecate (preferred — doesn't break existing installs)
9284
+ npm deprecate @softspark/ai-toolkit@X.Y.Z "Known issue: <description>. Use vA.B.C instead."
9285
+
9286
+ # Delete tag
9287
+ git tag -d vX.Y.Z
9288
+ git push origin --delete vX.Y.Z
9192
9289
  ```
9193
9290
 
9194
9291
  ---
9195
9292
 
9196
- ## Success Criteria
9293
+ ## Checklist Summary
9197
9294
 
9198
- | Area | Criterion |
9199
- |------|-----------|
9200
- | CLI | `--version` correct, `--help` full list, `status` current |
9201
- | Health | `doctor`: 0 errors, 0 warnings, PASSED |
9202
- | Install | `--dry-run` correct counts, `--local` all configs |
9203
- | Quality | `validate.py --strict`: PASSED |
9204
- | Security (baseline) | `audit_skills.py --ci`: 0 HIGH |
9205
- | Security (SARIF) | `audit_skills.py --sarif`: valid SARIF 2.1.0 with non-empty rules array |
9206
- | Security (permissions) | `audit_skills.py --permissions`: broad-access skills unchanged or justified in CHANGELOG |
9207
- | Supply chain | `dist.attestations.provenance.predicateType == https://slsa.dev/provenance/v1` on npm |
9208
- | Supply chain | All `sources.json` URL entries carry a `sha256`; `AI_TOOLKIT_STRICT_PIN=1 ai-toolkit update --dry-run` passes |
9209
- | Tests | `npm test`: N/N passed, 0 failures |
9210
- | Eject | Standalone `.claude/` with real files AND `output-styles/` directory |
9211
- | Guards | Destructive commands blocked |
9212
- | Deep coverage | `--profile full` emits every documented native surface, including Codex agents/skills/hooks and Copilot agents/skills/hooks |
9213
- | Profile contract | Codex skills emit at every profile without `--codex-skills`; Copilot standard emits agents, skills, instructions, prompts, and hooks |
9214
- | Idempotence | Second `install` run produces byte-identical output in every managed path |
9215
- | Live JSON | Every generated `.json` file on disk parses as valid JSON |
9216
- | Registry | `supported-tools-registry.md` enumerates every `scripts/generate_*.py` we ship |
9295
+ | # | Step | Command / Action | Pass Criteria |
9296
+ |---|------|-----------------|---------------|
9297
+ | 0a | Ecosystem drift check | `ecosystem_doctor.py --format text` | All tools Clean, or drift classified and resolved |
9298
+ | 0b | Ecosystem snapshot refresh | `ecosystem_doctor.py --update` | `benchmarks/ecosystem-doctor-snapshot.json` updated |
9299
+ | 0c | Ecosystem gate | `ecosystem_doctor.py --offline --check` | Exit 0 |
9300
+ | 1 | Version bump type | Decide patch/minor/major | — |
9301
+ | 2 | `package.json` version | Edit `"version"` | Matches target |
9302
+ | 3 | `manifest.json` version | Edit `"version"` | Matches target |
9303
+ | 4 | `plugin.json` version | Edit `"version"` | Matches target |
9304
+ | 5 | `package-lock.json` | `npm install --package-lock-only` | Matches target |
9305
+ | 6 | Count sync | Check `package.json` description, README | `validate.py` passes |
9306
+ | 7 | CHANGELOG.md | Add release entry (incl. `Ecosystem` subsection if any B/D/E/F drift) | Entry exists for vX.Y.Z |
9307
+ | 8 | Regenerate artifacts | `generate_agents_md.py`, `generate_llms_txt.py` | No unexpected diff |
9308
+ | 9 | Validate | `validate.py --strict` | 0 errors, 0 warnings |
9309
+ | 10 | Security audit (CI mode) | `audit_skills.py --ci` | 0 HIGH |
9310
+ | 11 | Security audit (SARIF) | `audit_skills.py --sarif` | Valid SARIF 2.1.0 JSON |
9311
+ | 12 | Per-skill permissions | `audit_skills.py --permissions` | New broad-access skills justified in CHANGELOG |
9312
+ | 13 | ShellCheck hooks | `shellcheck --severity=warning app/hooks/*.sh` | Exit 0, no output (mirrors ci.yml; publish.yml does NOT run it) |
9313
+ | 14 | Provenance flag check | `grep -- '--provenance' .github/workflows/publish.yml` | Present |
9314
+ | 15 | Checksum-pin backfill | `sources.json` entries all have `sha256` | No unpinned URL sources |
9315
+ | 15a | Licensing gate | `npx bats tests/test_licensing.bats` | 7/7 — SPDX headers, LICENSE, NOTICE, npm `files`, manifest consistency |
9316
+ | 16 | Tests | `git add -A kb/` if the KB changed, then `npm test` | All pass |
9317
+ | 17 | Commit | `git commit` | Clean working tree |
9318
+ | 18 | Push branch | `git push origin main` | Exact release commit is on `origin/main` |
9319
+ | 18a | Full branch CI | `gh run watch "$RUN_ID" --exit-status` plus matrix job assertions | Ubuntu and macOS Bats jobs both conclude `success` |
9320
+ | 19 | Tag | `git tag vX.Y.Z` | Tag exists only after green branch CI |
9321
+ | 19a | Tag is on tested SHA | `test "$(git rev-parse vX.Y.Z)" = "$RELEASE_SHA"` | Exit 0 |
9322
+ | 19b | Tag is on the release commit | `git show --no-patch --format=%s vX.Y.Z` | Reads `chore: release vX.Y.Z` |
9323
+ | 20 | Push the single tag | `git push origin refs/tags/vX.Y.Z` | Publish CI triggered with `id-token: write`. Never `--tags`. |
9217
9324
 
9218
9325
  ---
9219
9326
 
@@ -9801,7 +9908,7 @@ Three tiers determine how to approach a task:
9801
9908
  |------|-------|-----------|-------|
9802
9909
  | Task | `disable-model-invocation: true` | User via `/skill` only | 32 |
9803
9910
  | Hybrid | (neither) | User via `/skill` + agent knowledge | 31 |
9804
- | Knowledge | `user-invocable: false` | Claude auto-loads | 46 |
9911
+ | Knowledge | `user-invocable: false` | Claude auto-loads | 51 |
9805
9912
 
9806
9913
  ## Multi-Agent Execution
9807
9914
 
@@ -11738,7 +11845,7 @@ running the generators directly:
11738
11845
 
11739
11846
  - `scripts/generate_copilot.py` / `generate_copilot_hooks.py` contract tests
11740
11847
  (`tests/test_copilot.bats`, `tests/test_copilot_hooks.bats`).
11741
- - Release layout check in `kb/procedures/release-verification-sop.md` asserts the
11848
+ - Release layout check in `kb/procedures/sop-release-verification.md` asserts the
11742
11849
  `.github/{agents,skills,instructions,prompts,hooks}` surfaces.
11743
11850
  - `validate.py --strict` + `audit_skills.py --ci` in CI.
11744
11851
 
@@ -14313,7 +14420,7 @@ MIT remnant would need each contributor's agreement.
14313
14420
 
14314
14421
  It is a test rather than a checklist line on purpose. This project has two
14315
14422
  same-day postmortems about SOPs that existed and were skipped; CI does not skip.
14316
- [Release Preparation](../procedures/release-preparation-sop.md) Phase 5c runs the
14423
+ [Release Preparation](../procedures/sop-release.md) Phase 5c runs the
14317
14424
  same gate before tagging so a failure surfaces before the tag, not after.
14318
14425
 
14319
14426
  ## If the licence ever changes again
@@ -14326,7 +14433,7 @@ source, cross-verify against a second independent copy, and only then write
14326
14433
  ## Related
14327
14434
 
14328
14435
  - [`LICENSE`](../../LICENSE), [`NOTICE`](../../NOTICE)
14329
- - [Release Preparation SOP](../procedures/release-preparation-sop.md) — Phase 5c
14436
+ - [Release Preparation SOP](../procedures/sop-release.md) — Phase 5c
14330
14437
  - [Distribution Model](distribution-model.md) — what ships and where
14331
14438
 
14332
14439
  ---
@@ -16006,7 +16113,7 @@ Updated baseline docs:
16006
16113
  - `kb/reference/skills-catalog.md`
16007
16114
  - `kb/reference/plugin-pack-conventions.md`
16008
16115
  - `kb/reference/claude-ecosystem-benchmark-snapshot.md`
16009
- - `kb/procedures/maintenance-sop.md`
16116
+ - `kb/procedures/sop-maintenance.md`
16010
16117
 
16011
16118
  ## Validation Evidence
16012
16119
 
@@ -16860,13 +16967,13 @@ See [DSH Compatibility](./dsh-compatibility.md) for commands, topology, subscrip
16860
16967
  2. Delete its section from this document.
16861
16968
  3. Delete its snapshot entry from `benchmarks/ecosystem-doctor-snapshot.json` (or let the next `--update` prune it — currently not pruned automatically).
16862
16969
  4. Decide whether to keep the generator (`scripts/generate_<tool>_*.py`) for backwards compatibility or delete it.
16863
- 5. Remove references from `README.md`, `manifest.json` `description` field, and `kb/procedures/maintenance-sop.md` `Supported editors` line.
16970
+ 5. Remove references from `README.md`, `manifest.json` `description` field, and `kb/procedures/sop-maintenance.md` `Supported editors` line.
16864
16971
 
16865
16972
  ---
16866
16973
 
16867
16974
  ## Related
16868
16975
 
16869
- - [Ecosystem Sync SOP](../procedures/ecosystem-sync-sop.md) — how to use the doctor
16976
+ - [Ecosystem Sync SOP](../procedures/sop-ecosystem-sync.md) — how to use the doctor
16870
16977
  - [MCP Editor Compatibility](./mcp-editor-compatibility.md) — MCP-specific subset
16871
16978
  - `scripts/ecosystem_tools.json` — source of truth
16872
16979
  - `scripts/ecosystem_doctor.py` — drift detector
@@ -17399,7 +17506,7 @@ Expect `Total plugin skills loaded: 0`, `Total plugin agents loaded: 0`, and the
17399
17506
  ## Related
17400
17507
 
17401
17508
  - `kb/reference/global-install-model.md`
17402
- - `kb/procedures/maintenance-sop.md`
17509
+ - `kb/procedures/sop-maintenance.md`
17403
17510
 
17404
17511
  ---
17405
17512