@softspark/ai-toolkit 4.30.2 → 4.31.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +57 -0
- package/README.md +27 -28
- package/app/.claude-plugin/plugin.json +1 -1
- package/app/agents/code-reviewer.md +14 -0
- package/app/agents/frontend-specialist.md +27 -24
- package/app/agents/infrastructure-validator.md +2 -2
- package/app/claude-app/skills/ai-toolkit-rules/SKILL.md +44 -1
- package/app/personas/frontend-lead.md +20 -12
- package/app/rules/claude-toolkit-rules.md +4 -1
- package/app/rules/edit-discipline.md +37 -0
- package/app/skills/app-builder/SKILL.md +37 -6
- package/app/skills/claude-toolkit-rules/SKILL.md +37 -0
- package/app/skills/design-engineering/SKILL.md +87 -102
- package/app/skills/design-engineering/reference/animation-recipes.md +67 -0
- package/app/skills/design-engineering/reference/component-craft.md +28 -0
- package/app/skills/documentation-standards/SKILL.md +1 -1
- package/app/skills/edit-discipline/SKILL.md +50 -0
- package/app/skills/git-conventions/SKILL.md +18 -0
- package/app/skills/output-mode/SKILL.md +41 -0
- package/app/skills/quality-gates/SKILL.md +23 -0
- package/app/skills/review/SKILL.md +10 -0
- package/kb/procedures/{ecosystem-sync-sop.md → sop-ecosystem-sync.md} +2 -2
- package/kb/procedures/{post-release-testing-sop.md → sop-post-release-testing.md} +3 -3
- package/kb/procedures/sop-pre-commit.md +96 -0
- package/kb/procedures/{release-preparation-sop.md → sop-release.md} +87 -26
- package/kb/reference/architecture-overview.md +1 -1
- package/kb/reference/copilot-compatibility.md +1 -1
- package/kb/reference/licensing.md +2 -2
- package/kb/reference/quick-wins-implementation-summary.md +1 -1
- package/kb/reference/supported-tools-registry.md +2 -2
- package/kb/troubleshooting/plugin-double-load.md +1 -1
- package/llms-full.txt +974 -806
- package/llms.txt +6 -5
- package/manifest.json +1 -1
- package/package.json +2 -1
- package/scripts/doctor.py +1 -1
- package/scripts/generate_toolkit_rules_skills.py +189 -0
- package/scripts/validate.py +1 -1
- /package/kb/procedures/{maintenance-sop.md → sop-maintenance.md} +0 -0
- /package/kb/procedures/{release-verification-sop.md → sop-release-verification.md} +0 -0
package/llms-full.txt
CHANGED
|
@@ -32,11 +32,12 @@
|
|
|
32
32
|
- [Closed: Tool-Output Token Reduction](kb/history/completed/tool-output-token-reduction-closed-20260727.md)
|
|
33
33
|
- [How-To Guides](kb/howto/README.md)
|
|
34
34
|
- [Plan: Cloud Security Pack — Multi-Cloud Audit](kb/planning/cloud-security-pack-plan.md)
|
|
35
|
-
- [SOP: Ecosystem Sync](kb/procedures/ecosystem-sync
|
|
36
|
-
- [SOP: AI Toolkit Maintenance](kb/procedures/maintenance
|
|
37
|
-
- [SOP: Post-Release Testing](kb/procedures/post-release-testing
|
|
38
|
-
- [SOP:
|
|
39
|
-
- [SOP: Release Verification](kb/procedures/release-verification
|
|
35
|
+
- [SOP: Ecosystem Sync](kb/procedures/sop-ecosystem-sync.md)
|
|
36
|
+
- [SOP: AI Toolkit Maintenance](kb/procedures/sop-maintenance.md)
|
|
37
|
+
- [SOP: Post-Release Testing](kb/procedures/sop-post-release-testing.md)
|
|
38
|
+
- [SOP: Pre-Commit Gate](kb/procedures/sop-pre-commit.md)
|
|
39
|
+
- [SOP: Release Verification](kb/procedures/sop-release-verification.md)
|
|
40
|
+
- [SOP: Release Preparation](kb/procedures/sop-release.md)
|
|
40
41
|
- [Agents Catalog](kb/reference/agents-catalog.md)
|
|
41
42
|
- [Anti-Pattern Registry Format](kb/reference/anti-pattern-registry-format.md)
|
|
42
43
|
- [AI Toolkit Architecture](kb/reference/architecture-overview.md)
|
|
@@ -95,6 +96,7 @@
|
|
|
95
96
|
- **chaos**: Injects controlled faults for resilience testing on non-prod. Triggers: chaos, fault injection, latency injection, dependency kill, resilience test.
|
|
96
97
|
- **ci**: Detect/generate/debug CI pipeline config (GitHub Actions, GitLab CI). Triggers: CI setup, build pipeline, GitHub Actions config, debug CI, GitLab CI.
|
|
97
98
|
- **ci-cd-patterns**: CI/CD: GitHub Actions, GitLab CI, Jenkins, caching, blue-green, canary. Triggers: CI, CD, pipeline, GitHub Actions, workflow YAML, release, canary, rollout.
|
|
99
|
+
- **claude-toolkit-rules**: SoftSpark working agreement: never guess a home directory path, give at least three alternatives, and apply a devil's advocate critique to decisions. Triggers: toolkit, conventions, workflow, alternatives, review.
|
|
98
100
|
- **clean-code**: Code quality: meaningful names, SRP, DRY, small functions, guard clauses, refactoring. Triggers: clean code, naming, code smell, SRP, DRY, long function, god class, dead code.
|
|
99
101
|
- **command-creator**: Creates new Claude Code slash commands with frontmatter and validation. Triggers: new slash command, create command, command scaffold.
|
|
100
102
|
- **commit**: Creates Conventional Commits with pre-commit validation. Triggers: commit, conventional commit, git commit, message.
|
|
@@ -115,12 +117,14 @@
|
|
|
115
117
|
- **docs**: Generates/updates README, API docs, architecture notes. Triggers: docs, README, API docs, architecture note, documentation.
|
|
116
118
|
- **documentation-standards**: KB conventions: YAML frontmatter, 10-category taxonomy (reference/howto/procedures/troubleshooting/best-practices/decisions/runbooks/planning/business/templates). Triggers: kb/, SOP, runbook, howto, frontmatter, knowledge base.
|
|
117
119
|
- **ecommerce-patterns**: E-commerce: cart, checkout, payments (Stripe/Adyen), order state, inventory, promos, tax. Triggers: cart, checkout, SKU, payment, Stripe, Shopify, Medusa, Magento, coupon, refund.
|
|
120
|
+
- **edit-discipline**: Change files with the edit and write tools, never by rewriting them through bash (sed, awk, tee, heredoc, redirection), and show `git diff` before reporting a file-changing task as done. Triggers: edit, write, modify, refactor, patch, fix, diff, review changes.
|
|
118
121
|
- **evaluate**: Evaluates RAG retrieval and LLM-as-judge metrics (faithfulness, relevancy, context precision). Triggers: measure RAG quality, knowledge gap, RAG eval, golden dataset.
|
|
119
122
|
- **evolve**: Analyzes agent/skill failures, drafts prompt/permission fixes. Triggers: improve agent, refine skill, system prompt, optimize agent.
|
|
120
123
|
- **explain**: Explains code/architecture with Mermaid diagrams and sequence flows. Triggers: what does X do, how does Y work, explain code, sequence diagram.
|
|
121
124
|
- **explore**: Explores codebase structure, stack, and architecture. Triggers: explore codebase, project structure, stack overview, architecture map.
|
|
122
125
|
- **fix**: Applies targeted fix to known bug/lint error, verifies with same command that surfaced it. Triggers: fix, apply fix, fix bug, fix lint, targeted fix.
|
|
123
126
|
- **flutter-patterns**: Flutter/Dart: widgets, state mgmt (Riverpod/Bloc), navigation, platform channels. Triggers: Flutter, Dart, widget, Riverpod, Bloc, pubspec, hot reload.
|
|
127
|
+
- **git-conventions**: Conventional Commits only (feat, fix, docs, refactor, test, chore); no AI co-authorship trailer and no AI signature in a commit message. Triggers: commit, git, message, changelog, release, pull request.
|
|
124
128
|
- **git-mastery**: Advanced Git: rebase, bisect, reflog, cherry-pick, worktrees, LFS. Triggers: rebase, bisect, cherry-pick, reflog, force push, merge conflict, worktree.
|
|
125
129
|
- **golang-rules**: Go coding rules: style, patterns, security, testing. Triggers: .go, go.mod, go.sum, Gin, Echo, Gorilla, testing, gofmt.
|
|
126
130
|
- **grill-me**: Stress-tests a plan via Socratic questioning down each decision branch. Triggers: stress-test, grill me, validate assumptions, challenge plan, socratic review.
|
|
@@ -147,6 +151,7 @@
|
|
|
147
151
|
- **observability-patterns**: Observability: structured logs, metrics (RED/USE), tracing, SLO/SLI. Triggers: logging, metrics, Prometheus, Grafana, OpenTelemetry, trace, monitoring.
|
|
148
152
|
- **onboard**: Sets up ai-toolkit in a project: symlinks, CLAUDE.md, intent interview. Triggers: onboard, setup project, install ai-toolkit, migrate project.
|
|
149
153
|
- **orchestrate**: Coordinates multiple specialized agents in parallel. Triggers: orchestrate, multi-agent, parallel agents, coordinate agents.
|
|
154
|
+
- **output-mode**: Answer concisely: lead with the result, no preamble, tables over prose, no trailing restatement of a diff the reader can already see. Triggers: response style, verbosity, summary, explanation, report.
|
|
150
155
|
- **panic**: Emergency kill switch — halts all agents via lockfile gate. Triggers: panic, stop everything, kill switch, halt agents, agents looping.
|
|
151
156
|
- **performance-profiling**: Performance: golden signals, p50/p95/p99, flame graphs, load testing. Triggers: performance, slow, latency, p99, flame graph, bottleneck, memory leak.
|
|
152
157
|
- **persona**: Switches engineering persona at runtime. Triggers: persona, switch role, backend-lead, frontend-lead, devops-eng, junior-dev.
|
|
@@ -160,6 +165,7 @@
|
|
|
160
165
|
- **prompt-caching-patterns**: Anthropic API prompt caching: TTL, breakpoints, stacking, invalidation, hit rate. Triggers: prompt caching, cache_control, cache breakpoint, cache TTL, hit rate.
|
|
161
166
|
- **python-rules**: Python coding rules: style, patterns, security, testing. Triggers: .py, .pyi, pyproject.toml, requirements.txt, Pipfile, FastAPI, Django, Flask, pytest, SQLAlchemy, ruff, mypy.
|
|
162
167
|
- **qa-session**: Interactive QA: user reports bugs conversationally, agent files GitHub issues. Triggers: QA session, report bug, file issue, conversational QA, bug intake.
|
|
168
|
+
- **quality-gates**: Plan before work over an hour, and hold the gates: ruff clean, mypy --strict clean, pytest coverage above 70 percent, no secrets in code. Triggers: quality, lint, mypy, pytest, coverage, gate, definition of done.
|
|
163
169
|
- **rag-patterns**: RAG: embeddings, chunking, hybrid search (BM25+vector), reranking, CRAG, multi-hop. Triggers: RAG, embedding, pgvector, Qdrant, Pinecone, Weaviate, reranker, semantic search.
|
|
164
170
|
- **refactor**: Refactors code for quality and maintainability. Triggers: refactor, clean up, restructure, improve code, modernize.
|
|
165
171
|
- **refactor-plan**: Creates detailed refactor plan with tiny commits via interview, files as GitHub RFC. Triggers: refactor plan, refactoring RFC, incremental refactor, safe steps.
|
|
@@ -6929,7 +6935,7 @@ Examples:
|
|
|
6929
6935
|
|
|
6930
6936
|
---
|
|
6931
6937
|
|
|
6932
|
-
## kb/procedures/ecosystem-sync
|
|
6938
|
+
## kb/procedures/sop-ecosystem-sync.md
|
|
6933
6939
|
|
|
6934
6940
|
---
|
|
6935
6941
|
title: "SOP: Ecosystem Sync"
|
|
@@ -7186,12 +7192,12 @@ jobs:
|
|
|
7186
7192
|
|
|
7187
7193
|
- [Supported Tools Registry](../reference/supported-tools-registry.md) — human-readable per-tool breakdown
|
|
7188
7194
|
- [MCP Editor Compatibility](../reference/mcp-editor-compatibility.md) — MCP-specific adapter table
|
|
7189
|
-
- [Maintenance SOP](maintenance
|
|
7190
|
-
- [Release Preparation SOP](release
|
|
7195
|
+
- [Maintenance SOP](sop-maintenance.md) — general toolkit upkeep
|
|
7196
|
+
- [Release Preparation SOP](sop-release.md) — run the doctor before tagging
|
|
7191
7197
|
|
|
7192
7198
|
---
|
|
7193
7199
|
|
|
7194
|
-
## kb/procedures/maintenance
|
|
7200
|
+
## kb/procedures/sop-maintenance.md
|
|
7195
7201
|
|
|
7196
7202
|
---
|
|
7197
7203
|
title: "SOP: AI Toolkit Maintenance"
|
|
@@ -7705,7 +7711,7 @@ What `uninstall` does:
|
|
|
7705
7711
|
|
|
7706
7712
|
---
|
|
7707
7713
|
|
|
7708
|
-
## kb/procedures/post-release-testing
|
|
7714
|
+
## kb/procedures/sop-post-release-testing.md
|
|
7709
7715
|
|
|
7710
7716
|
---
|
|
7711
7717
|
title: "SOP: Post-Release Testing"
|
|
@@ -7725,7 +7731,7 @@ actually install, from npm, rather than the working tree.
|
|
|
7725
7731
|
|
|
7726
7732
|
Sibling procedures exist for `jira-mcp` and `legal-pl-pack`; this is the
|
|
7727
7733
|
ai-toolkit equivalent. It complements
|
|
7728
|
-
[Release Verification](release-verification
|
|
7734
|
+
[Release Verification](sop-release-verification.md), which checks the toolkit
|
|
7729
7735
|
from the maintainer's own installed copy. The difference that matters: this one
|
|
7730
7736
|
never writes to the maintainer's `~/.claude` or `~/.softspark`.
|
|
7731
7737
|
|
|
@@ -8019,1140 +8025,1302 @@ print(f'removed {sb} ({n} files)')
|
|
|
8019
8025
|
|
|
8020
8026
|
## Related
|
|
8021
8027
|
|
|
8022
|
-
- [Release Preparation](release
|
|
8023
|
-
- [Release Verification](release-verification
|
|
8028
|
+
- [Release Preparation](sop-release.md) — run before tagging
|
|
8029
|
+
- [Release Verification](sop-release-verification.md) — the maintainer-install checks
|
|
8024
8030
|
- [rtk-pack Retirement](../history/completed/rtk-pack-retirement-20260727.md) — what happened the one time this SOP was written and not run
|
|
8025
8031
|
|
|
8026
8032
|
---
|
|
8027
8033
|
|
|
8028
|
-
## kb/procedures/
|
|
8034
|
+
## kb/procedures/sop-pre-commit.md
|
|
8029
8035
|
|
|
8030
8036
|
---
|
|
8031
|
-
title: "SOP:
|
|
8037
|
+
title: "SOP: Pre-Commit Gate"
|
|
8032
8038
|
category: procedures
|
|
8033
8039
|
service: ai-toolkit
|
|
8034
|
-
tags: [
|
|
8035
|
-
version: "1.
|
|
8036
|
-
created: "2026-
|
|
8037
|
-
last_updated: "2026-
|
|
8038
|
-
description: "
|
|
8040
|
+
tags: [pre-commit, quality-gate, shellcheck, bats, validation, secrets]
|
|
8041
|
+
version: "1.0.0"
|
|
8042
|
+
created: "2026-09-02"
|
|
8043
|
+
last_updated: "2026-09-02"
|
|
8044
|
+
description: "The local gate to run before every commit to ai-toolkit: staged-file and secret scan, ShellCheck, Python syntax, toolkit validation, and the Bats suite. Runs the same checks CI runs, so main does not go red after a push."
|
|
8039
8045
|
---
|
|
8040
8046
|
|
|
8041
|
-
# SOP:
|
|
8047
|
+
# SOP: Pre-Commit Gate
|
|
8042
8048
|
|
|
8043
|
-
|
|
8044
|
-
Run this **before** tagging. After tagging and publishing, run the
|
|
8045
|
-
[Release Verification SOP](release-verification-sop.md) to smoke-test.
|
|
8049
|
+
## Why this exists
|
|
8046
8050
|
|
|
8047
|
-
|
|
8048
|
-
|
|
8049
|
-
|
|
8050
|
-
|
|
8051
|
-
|
|
8052
|
-
↓
|
|
8053
|
-
git tag → CI publish → Release Verification SOP
|
|
8054
|
-
```
|
|
8051
|
+
CI runs on `main` and on pull requests, but it runs *after* the push. Every
|
|
8052
|
+
check below already exists in `.github/workflows/ci.yml`; running them locally
|
|
8053
|
+
first is what keeps `main` green and keeps a release from becoming a debug
|
|
8054
|
+
session. The v4.5.1 postmortem in `sop-release.md` is the case
|
|
8055
|
+
where a hook lint failure published while reddening `main` CI.
|
|
8055
8056
|
|
|
8056
|
-
|
|
8057
|
+
This gate is not a substitute for `sop-release.md`. That SOP gates
|
|
8058
|
+
a *tag*; this one gates a *commit*.
|
|
8057
8059
|
|
|
8058
|
-
|
|
8060
|
+
## Prerequisites
|
|
8059
8061
|
|
|
8060
|
-
|
|
8062
|
+
`shellcheck`, `python3` and `bats` must be on `PATH`. The suite is the one
|
|
8063
|
+
check that silently does nothing when its runner is absent, so install it
|
|
8064
|
+
before relying on this gate:
|
|
8061
8065
|
|
|
8062
8066
|
```bash
|
|
8063
|
-
|
|
8064
|
-
#
|
|
8065
|
-
python3 scripts/ecosystem_doctor.py --format text > /tmp/eco-report.txt
|
|
8066
|
-
cat /tmp/eco-report.txt
|
|
8067
|
-
# If drift detected: stop here, follow ecosystem-sync-sop.md Phase 2-4 to
|
|
8068
|
-
# classify each drift (A-F), update generators as needed, refresh snapshot,
|
|
8069
|
-
# THEN resume this SOP.
|
|
8070
|
-
python3 scripts/ecosystem_doctor.py --update # after all drift resolved
|
|
8071
|
-
|
|
8072
|
-
# 1. Decide version bump
|
|
8073
|
-
# patch (1.4.2 → 1.4.3): bugfix, typo, doc fix
|
|
8074
|
-
# minor (1.4.2 → 1.5.0): new feature, new skill, new flag, any ecosystem-class-B/F change
|
|
8075
|
-
# major (1.4.2 → 2.0.0): breaking change, any ecosystem-class-D removed path
|
|
8076
|
-
|
|
8077
|
-
# 2. Sync version across all files
|
|
8078
|
-
python3 scripts/sync_version.py X.Y.Z # if script exists, else manual
|
|
8079
|
-
|
|
8080
|
-
# 3. Write CHANGELOG.md entry
|
|
8081
|
-
# 4. Regenerate artifacts
|
|
8082
|
-
python3 scripts/generate_agents_md.py > AGENTS.md
|
|
8083
|
-
python3 scripts/generate_llms_txt.py > llms.txt
|
|
8084
|
-
python3 scripts/generate_llms_txt.py --full > llms-full.txt
|
|
8085
|
-
|
|
8086
|
-
# 5. Validate + audit + SARIF + shellcheck + test + ecosystem check
|
|
8087
|
-
python3 scripts/validate.py --strict && python3 scripts/audit_skills.py --ci && python3 scripts/audit_skills.py --sarif > /tmp/audit.sarif && shellcheck --severity=warning app/hooks/*.sh && npm test
|
|
8088
|
-
|
|
8089
|
-
# 5a. Supply-chain standard (v2.8.0+) — non-negotiable
|
|
8090
|
-
grep -q -- '--provenance' .github/workflows/publish.yml || { echo "MISSING --provenance"; exit 1; }
|
|
8091
|
-
grep -q 'id-token: write' .github/workflows/publish.yml || { echo "MISSING id-token: write"; exit 1; }
|
|
8092
|
-
python3 scripts/audit_skills.py --permissions # review Bash/Write/Edit footprint
|
|
8093
|
-
|
|
8094
|
-
# 5b. Ecosystem gate — snapshot must be current before tag
|
|
8095
|
-
python3 scripts/ecosystem_doctor.py --offline --check || { echo "STALE ecosystem snapshot — re-run doctor"; exit 1; }
|
|
8096
|
-
|
|
8097
|
-
# 5c. Licensing gate — SPDX headers, LICENSE, NOTICE, manifest consistency
|
|
8098
|
-
npx bats tests/test_licensing.bats || { echo "LICENSING GATE FAILED"; exit 1; }
|
|
8099
|
-
|
|
8100
|
-
# 6. Commit + tag + push
|
|
8101
|
-
git add -A && git commit -m "chore: release vX.Y.Z"
|
|
8102
|
-
git tag vX.Y.Z
|
|
8103
|
-
|
|
8104
|
-
# 6a. Assert the tag before pushing it (v4.19.0 postmortem, Phase 7)
|
|
8105
|
-
test "$(git rev-parse vX.Y.Z)" = "$(git rev-parse HEAD)" || { echo "FAIL: tag not on HEAD"; exit 1; }
|
|
8106
|
-
git show --no-patch --format=%s vX.Y.Z | grep -qx "chore: release vX.Y.Z" || { echo "FAIL: tag not on release commit"; exit 1; }
|
|
8107
|
-
|
|
8108
|
-
# 6b. Branch first, then the single tag by full ref. Never --tags.
|
|
8109
|
-
git push origin main
|
|
8110
|
-
git push origin refs/tags/vX.Y.Z
|
|
8067
|
+
brew install bats-core shellcheck # macOS
|
|
8068
|
+
sudo apt-get install -y bats shellcheck # Debian / Ubuntu
|
|
8111
8069
|
```
|
|
8112
8070
|
|
|
8113
|
-
|
|
8071
|
+
## The gate
|
|
8114
8072
|
|
|
8115
|
-
|
|
8073
|
+
Run all five from the repository root. Each mirrors a CI job.
|
|
8116
8074
|
|
|
8117
|
-
|
|
8075
|
+
```bash
|
|
8076
|
+
# 1. Staged files: secrets, large blobs, commit-type suggestion
|
|
8077
|
+
python3 app/skills/commit/scripts/pre-commit-check.py
|
|
8118
8078
|
|
|
8119
|
-
|
|
8120
|
-
|
|
8121
|
-
- Major release (X.0.0) — always
|
|
8122
|
-
- Patch release (X.Y.Z) — only if the patch touches a generator or install flow
|
|
8079
|
+
# 2. ShellCheck hooks -- the CI job that has published while red before
|
|
8080
|
+
shellcheck --severity=warning app/hooks/*.sh app/plugins/*/hooks/*.sh
|
|
8123
8081
|
|
|
8124
|
-
|
|
8082
|
+
# 3. Python syntax across every script the toolkit ships
|
|
8083
|
+
python3 -m py_compile scripts/*.py app/skills/*/scripts/*.py
|
|
8125
8084
|
|
|
8126
|
-
|
|
8085
|
+
# 4. Toolkit integrity: agents, skills, registry drift, content quality
|
|
8086
|
+
npm run validate
|
|
8127
8087
|
|
|
8128
|
-
|
|
8129
|
-
|
|
8088
|
+
# 5. The suite. Run it ONCE -- see the single-run discipline below
|
|
8089
|
+
npm test
|
|
8130
8090
|
```
|
|
8131
8091
|
|
|
8132
|
-
|
|
8092
|
+
A non-zero exit from any of them is a stop, not a warning to note and push past.
|
|
8133
8093
|
|
|
8134
|
-
|
|
8094
|
+
## Single-run discipline
|
|
8135
8095
|
|
|
8136
|
-
|
|
8096
|
+
`npm test` runs the Bats suite with `--jobs 4`. Running it repeatedly to see
|
|
8097
|
+
whether a failure is "flaky" hides real ordering bugs and wastes minutes. Run
|
|
8098
|
+
it once. If it fails, read the failure and fix the cause; if the same test
|
|
8099
|
+
passes on a re-run without a code change, that instability is itself the bug
|
|
8100
|
+
and belongs in an issue, not in a retry.
|
|
8137
8101
|
|
|
8138
|
-
|
|
8139
|
-
|-------------|----------------|
|
|
8140
|
-
| A (cosmetic reword) | No version impact — refresh snapshot, continue |
|
|
8141
|
-
| B (new feature — integrate) | **Minor** version bump at minimum; new generator or extended generator |
|
|
8142
|
-
| C (new feature — not adopted) | No impact — note in registry |
|
|
8143
|
-
| D (deprecation) | **Minor** or **major** depending on user impact; add migration warning |
|
|
8144
|
-
| E (feature promoted to default) | **Minor**; simplify generator, keep fallback comment |
|
|
8145
|
-
| F (feature newly globally available) | **Minor**; may require new generator or new config path |
|
|
8102
|
+
## What this gate does not cover
|
|
8146
8103
|
|
|
8147
|
-
|
|
8104
|
+
- **Cross-platform.** The suite here runs on your machine only. Both
|
|
8105
|
+
`ubuntu-latest` and `macos-latest` must be green before a tag; that gate
|
|
8106
|
+
lives in `sop-release.md`, Phase 7, and is bound to the exact
|
|
8107
|
+
release commit.
|
|
8108
|
+
- **Required files.** The `required-files` CI job checks that every file the
|
|
8109
|
+
module template mandates is present and that the version is in sync across
|
|
8110
|
+
all four manifests. It is cheap and runs on every push.
|
|
8111
|
+
- **Provenance and SARIF.** Release-time concerns; see
|
|
8112
|
+
`sop-release.md`.
|
|
8148
8113
|
|
|
8149
|
-
|
|
8114
|
+
## Commit message
|
|
8150
8115
|
|
|
8151
|
-
|
|
8116
|
+
Conventional Commits, and no AI co-authorship trailer:
|
|
8152
8117
|
|
|
8153
|
-
```
|
|
8154
|
-
|
|
8118
|
+
```
|
|
8119
|
+
feat(scope): summary in the imperative
|
|
8120
|
+
fix(scope): summary in the imperative
|
|
8121
|
+
docs|refactor|test|chore(scope): ...
|
|
8155
8122
|
```
|
|
8156
8123
|
|
|
8157
|
-
|
|
8124
|
+
`pre-commit-check.py` suggests a type from the staged paths. It is a
|
|
8125
|
+
suggestion; the scope and the summary are yours.
|
|
8158
8126
|
|
|
8159
|
-
|
|
8127
|
+
## Related
|
|
8160
8128
|
|
|
8161
|
-
|
|
8162
|
-
|
|
8163
|
-
|
|
8129
|
+
- `sop-release.md` -- the pre-tag gate, including cross-platform CI
|
|
8130
|
+
- `sop-post-release-testing.md` -- what to verify after a release ships
|
|
8131
|
+
- `sop-release-verification.md` -- end-to-end smoke test of an installed build
|
|
8164
8132
|
|
|
8165
|
-
|
|
8133
|
+
---
|
|
8166
8134
|
|
|
8135
|
+
## kb/procedures/sop-release-verification.md
|
|
8136
|
+
|
|
8137
|
+
---
|
|
8138
|
+
title: "SOP: Release Verification"
|
|
8139
|
+
category: procedures
|
|
8140
|
+
service: ai-toolkit
|
|
8141
|
+
tags: [sop, verification, release, smoke-test, install, update, qa, provenance, sarif, dsh]
|
|
8142
|
+
version: "1.8.0"
|
|
8143
|
+
created: "2026-04-08"
|
|
8144
|
+
last_updated: "2026-09-01"
|
|
8145
|
+
description: "End-to-end smoke test after installing or updating @softspark/ai-toolkit. Verifies CLI, native Codex and GitHub Copilot surfaces, explicit DSH lifecycle, Claude app export, doctor, validation, tests, eject, provenance, SARIF, and per-skill permissions."
|
|
8167
8146
|
---
|
|
8168
8147
|
|
|
8169
|
-
|
|
8148
|
+
# SOP: Release Verification
|
|
8170
8149
|
|
|
8171
|
-
|
|
8150
|
+
End-to-end smoke test after installing or updating `@softspark/ai-toolkit`.
|
|
8151
|
+
Verifies all critical paths from the user's perspective.
|
|
8172
8152
|
|
|
8173
|
-
|
|
8174
|
-
|
|
8175
|
-
|
|
8176
|
-
|
|
8177
|
-
|
|
8153
|
+
**Use this SOP when:**
|
|
8154
|
+
- After `npm install -g @softspark/ai-toolkit@latest`
|
|
8155
|
+
- After `ai-toolkit update`
|
|
8156
|
+
- Before tagging a new version (`git tag`)
|
|
8157
|
+
- Before publishing to npm (`npm publish`)
|
|
8158
|
+
- As a smoke test in CI/CD
|
|
8178
8159
|
|
|
8179
|
-
**
|
|
8160
|
+
**Prerequisites:**
|
|
8161
|
+
- Node.js >= 18, Python 3, `bats`, git
|
|
8162
|
+
- `@softspark/ai-toolkit` installed globally
|
|
8163
|
+
|
|
8164
|
+
**Time:** 10-15 minutes (full), 2 minutes (quick checklist)
|
|
8180
8165
|
|
|
8181
8166
|
---
|
|
8182
8167
|
|
|
8183
|
-
##
|
|
8168
|
+
## Quick Checklist (TL;DR)
|
|
8184
8169
|
|
|
8185
|
-
The
|
|
8170
|
+
The 14 core commands below must pass. Releases that change DSH must also complete Phase 10.
|
|
8186
8171
|
|
|
8187
|
-
|
|
8172
|
+
```bash
|
|
8173
|
+
# Pre-commit (Phase 0)
|
|
8174
|
+
python3 scripts/generate_agents_md.py > AGENTS.md # 1. Regenerate AGENTS.md
|
|
8175
|
+
python3 scripts/generate_llms_txt.py > llms.txt # 2. Regenerate llms.txt
|
|
8176
|
+
python3 scripts/validate.py --strict # 3. Validation passed?
|
|
8177
|
+
npm test > /tmp/npm-test.log 2>&1 && grep -c '^ok ' /tmp/npm-test.log && ! grep -q '^not ok' /tmp/npm-test.log # 4. All tests passed? (single run, cached)
|
|
8188
8178
|
|
|
8189
|
-
|
|
8190
|
-
|
|
8191
|
-
|
|
8192
|
-
|
|
8193
|
-
|
|
8179
|
+
# Post-install verification (Phases 1-7)
|
|
8180
|
+
ai-toolkit --version # 5. Version OK?
|
|
8181
|
+
ai-toolkit status # 6. Status OK?
|
|
8182
|
+
ai-toolkit doctor # 7. Health check passed?
|
|
8183
|
+
ai-toolkit install --dry-run # 8. Global install OK?
|
|
8184
|
+
python3 scripts/audit_skills.py --ci # 9. Security audit clean?
|
|
8194
8185
|
|
|
8195
|
-
|
|
8186
|
+
# Supply-chain verification (Phase 8, v2.8.0+)
|
|
8187
|
+
python3 scripts/audit_skills.py --sarif | python3 -c "import json,sys; assert json.load(sys.stdin)['version']=='2.1.0'; print('SARIF OK')" # 10. SARIF 2.1.0 well-formed?
|
|
8188
|
+
python3 scripts/audit_skills.py --permissions | head -30 # 11. Broad-access skills reviewed?
|
|
8189
|
+
npm view @softspark/ai-toolkit@X.Y.Z --json | python3 -c "import json,sys; d=json.load(sys.stdin); assert d['dist']['attestations']['provenance']['predicateType']=='https://slsa.dev/provenance/v1'; print('PROVENANCE OK')" # 12. Provenance attested on npm?
|
|
8190
|
+
python3 scripts/claude_app.py verify # 13. Claude Chat/Cowork plugin contract valid?
|
|
8196
8191
|
|
|
8197
|
-
|
|
8198
|
-
|
|
8199
|
-
|
|
8192
|
+
# Deep-coverage verification (Phase 9, v3.0.0+)
|
|
8193
|
+
META="generate_agents_md.py|generate_llms_txt.py|generate_language_rules_skills.py"
|
|
8194
|
+
diff <(grep -oE 'scripts/generate_[a-z_]+\.py' kb/reference/supported-tools-registry.md | sort -u) <(ls scripts/generate_*.py | grep -vE "$META" | sort -u) && echo "OK: registry matches" # 14. Registry <-> generators drift?
|
|
8195
|
+
```
|
|
8200
8196
|
|
|
8201
|
-
|
|
8197
|
+
---
|
|
8202
8198
|
|
|
8203
|
-
|
|
8204
|
-
|------|-------|---------------|
|
|
8205
|
-
| `kb/procedures/maintenance-sop.md` | frontmatter `version:` | If SOP content changed |
|
|
8206
|
-
| `kb/reference/skills-catalog.md` | frontmatter `version:` | If skills added/removed |
|
|
8207
|
-
| `kb/reference/agents-catalog.md` | frontmatter `version:` | If agents added/removed |
|
|
8208
|
-
| `kb/reference/hooks-catalog.md` | frontmatter `version:` | If hooks changed |
|
|
8209
|
-
| `kb/reference/architecture-overview.md` | frontmatter `version:` | If architecture changed |
|
|
8210
|
-
| `kb/reference/distribution-model.md` | frontmatter `version:` | If install model changed |
|
|
8211
|
-
| `kb/reference/global-install-model.md` | frontmatter `version:` | If install model changed |
|
|
8199
|
+
## Phase 0: Pre-Commit & Pre-Push (2 min)
|
|
8212
8200
|
|
|
8213
|
-
|
|
8214
|
-
|
|
8201
|
+
Run these commands **before every commit and push to main**. CI validates
|
|
8202
|
+
counts but does NOT auto-regenerate — you must do it locally.
|
|
8215
8203
|
|
|
8216
|
-
|
|
8204
|
+
```bash
|
|
8205
|
+
# 1. Regenerate generated artifacts
|
|
8206
|
+
python3 scripts/generate_agents_md.py > AGENTS.md
|
|
8207
|
+
python3 scripts/generate_llms_txt.py > llms.txt
|
|
8208
|
+
python3 scripts/generate_llms_txt.py --full > llms-full.txt
|
|
8217
8209
|
|
|
8218
|
-
|
|
8219
|
-
|
|
8220
|
-
| `package.json` | `"description"` — skill/agent count |
|
|
8221
|
-
| `README.md` | Badge counts, "What You Get" table |
|
|
8222
|
-
| `app/ARCHITECTURE.md` | Section headings with counts |
|
|
8210
|
+
# 2. Validate everything (catches stale counts, missing assets)
|
|
8211
|
+
python3 scripts/validate.py --strict
|
|
8223
8212
|
|
|
8224
|
-
|
|
8225
|
-
|
|
8226
|
-
> If validation passes, counts and versions are correct.
|
|
8213
|
+
# 3. Security audit
|
|
8214
|
+
python3 scripts/audit_skills.py --ci
|
|
8227
8215
|
|
|
8228
|
-
|
|
8216
|
+
# 4. Run tests
|
|
8217
|
+
npm test
|
|
8229
8218
|
|
|
8230
|
-
|
|
8219
|
+
# 5. Stage and commit
|
|
8220
|
+
# AGENTS.md, GEMINI.md, and .github/copilot-instructions.md are generated and
|
|
8221
|
+
# gitignored — do NOT commit them (regenerated at prepublishOnly). Only the
|
|
8222
|
+
# doc artifacts llms.txt / llms-full.txt are committed.
|
|
8223
|
+
git add llms.txt llms-full.txt
|
|
8224
|
+
git add -p # stage your other changes
|
|
8225
|
+
git commit -m "feat: your change description"
|
|
8226
|
+
```
|
|
8227
|
+
|
|
8228
|
+
**Why local?** Branch protection on `main` requires PRs and status checks.
|
|
8229
|
+
CI cannot push directly to `main`, so generated artifacts must be committed
|
|
8230
|
+
by the developer as part of their PR.
|
|
8231
8231
|
|
|
8232
|
+
**One-liner (copy-paste):**
|
|
8232
8233
|
```bash
|
|
8233
|
-
|
|
8234
|
-
echo "Target: $VERSION"
|
|
8235
|
-
echo "manifest.json: $(python3 -c "import json; print(json.load(open('manifest.json'))['version'])")"
|
|
8236
|
-
echo "plugin.json: $(python3 -c "import json; print(json.load(open('app/.claude-plugin/plugin.json'))['version'])")"
|
|
8237
|
-
echo "package-lock.json: $(python3 -c "import json; print(json.load(open('package-lock.json'))['version'])")"
|
|
8234
|
+
python3 scripts/generate_agents_md.py > AGENTS.md && python3 scripts/generate_llms_txt.py > llms.txt && python3 scripts/generate_llms_txt.py --full > llms-full.txt && python3 scripts/validate.py --strict && python3 scripts/audit_skills.py --ci && npm test
|
|
8238
8235
|
```
|
|
8239
8236
|
|
|
8240
|
-
|
|
8237
|
+
---
|
|
8241
8238
|
|
|
8242
|
-
|
|
8239
|
+
## Phase 1: CLI & Version (1 min)
|
|
8243
8240
|
|
|
8244
8241
|
```bash
|
|
8245
|
-
|
|
8246
|
-
|
|
8242
|
+
ai-toolkit --version
|
|
8243
|
+
ai-toolkit --help
|
|
8244
|
+
which ai-toolkit
|
|
8247
8245
|
```
|
|
8248
8246
|
|
|
8249
|
-
**
|
|
8250
|
-
|
|
8251
|
-
|
|
8252
|
-
|
|
8253
|
-
|
|
8254
|
-
Do not run `--update` to make a red build green. The check fails because something
|
|
8255
|
-
users depend on disappeared; regenerating the manifest deletes the evidence, not
|
|
8256
|
-
the problem.
|
|
8247
|
+
**Verify:**
|
|
8248
|
+
- [ ] `--version` returns correct semver (e.g., `1.4.0`)
|
|
8249
|
+
- [ ] `--help` displays full command list without errors
|
|
8250
|
+
- [ ] `which` points to global npm bin path
|
|
8257
8251
|
|
|
8258
|
-
|
|
8252
|
+
---
|
|
8259
8253
|
|
|
8260
|
-
|
|
8254
|
+
## Phase 2: Global Install & Status (2 min)
|
|
8261
8255
|
|
|
8256
|
+
```bash
|
|
8257
|
+
ai-toolkit install --dry-run
|
|
8258
|
+
ai-toolkit status
|
|
8262
8259
|
```
|
|
8263
|
-
Body budget: largest is <skill> at <N> bytes (warn 18000, error 20000)
|
|
8264
|
-
```
|
|
8265
|
-
|
|
8266
|
-
Once that number sits at least 2000 bytes under `SKILL_BODY_BUDGET_WARN`, lower
|
|
8267
|
-
`SKILL_BODY_BUDGET_WARN` by 2000 in `scripts/validate.py` and ship the tightened
|
|
8268
|
-
threshold with the release. Target floor is 12000.
|
|
8269
8260
|
|
|
8270
|
-
|
|
8271
|
-
|
|
8272
|
-
-
|
|
8273
|
-
|
|
8274
|
-
|
|
8275
|
-
tighten in a follow-up.
|
|
8276
|
-
- **Never raise a threshold to make a red build green.** A body over budget means
|
|
8277
|
-
detail that belongs in `reference/` is loading on every trigger match, including
|
|
8278
|
-
the accidental ones. Raising the number hides the cost, it does not remove it.
|
|
8261
|
+
**Verify `--dry-run`:**
|
|
8262
|
+
- [ ] Agents: 44
|
|
8263
|
+
- [ ] Skills: 108
|
|
8264
|
+
- [ ] Hooks merged into settings.json
|
|
8265
|
+
- [ ] "Other AI Tools" lists documented global targets (with `--editors`): aider, antigravity, augment, cline, codex, copilot, cursor, gemini, opencode, roo, windsurf. Scope varies: Codex uses `$CODEX_HOME` (default `~/.codex`) plus `$HOME/.agents/skills`; Copilot uses `$COPILOT_HOME` (default `~/.copilot`); cursor has only `~/.cursor/hooks.json`; antigravity has the `~/.gemini/*/skills` pointer. Cursor and Antigravity rules remain project-only.
|
|
8279
8266
|
|
|
8280
|
-
|
|
8281
|
-
|
|
8282
|
-
|
|
8267
|
+
**Verify `status`:**
|
|
8268
|
+
- [ ] Version matches expected
|
|
8269
|
+
- [ ] Profile: minimal/standard/strict
|
|
8270
|
+
- [ ] Modules: list of installed modules
|
|
8271
|
+
- [ ] Latest: up to date / update available
|
|
8283
8272
|
|
|
8284
8273
|
---
|
|
8285
8274
|
|
|
8286
|
-
## Phase 3:
|
|
8287
|
-
|
|
8288
|
-
Add entry at the top of `CHANGELOG.md` (after the header, before previous release):
|
|
8289
|
-
|
|
8290
|
-
```markdown
|
|
8291
|
-
## vX.Y.Z — Short Title (YYYY-MM-DD)
|
|
8292
|
-
|
|
8293
|
-
### Added
|
|
8294
|
-
- **Feature name** — description
|
|
8295
|
-
|
|
8296
|
-
### Changed
|
|
8297
|
-
- **What changed** — old behavior → new behavior
|
|
8298
|
-
|
|
8299
|
-
### Fixed
|
|
8300
|
-
- **Bug description** — what was broken and how it's fixed
|
|
8275
|
+
## Phase 3: Doctor Health Check (1 min)
|
|
8301
8276
|
|
|
8302
|
-
|
|
8303
|
-
-
|
|
8277
|
+
```bash
|
|
8278
|
+
ai-toolkit doctor
|
|
8304
8279
|
```
|
|
8305
8280
|
|
|
8306
|
-
**
|
|
8307
|
-
-
|
|
8308
|
-
-
|
|
8309
|
-
-
|
|
8310
|
-
-
|
|
8311
|
-
-
|
|
8312
|
-
-
|
|
8313
|
-
-
|
|
8314
|
-
|
|
8315
|
-
### Update README "What's New" section
|
|
8316
|
-
|
|
8317
|
-
**MANDATORY on every release.** Update the `## What's New in vX.Y.Z` section in `README.md`:
|
|
8318
|
-
|
|
8319
|
-
1. Change the heading version: `## What's New in vX.Y.Z`
|
|
8320
|
-
2. Replace bullet points with 3-5 highlights from this release
|
|
8321
|
-
3. **Keep only the latest version block.** Delete the previous `## What's New in vA.B.C` section(s). README is the shop window, not the archive — users see the current release, full history lives in `CHANGELOG.md`.
|
|
8322
|
-
4. Keep the `See [CHANGELOG.md](CHANGELOG.md) for full history.` link directly below the bullet list.
|
|
8281
|
+
**Expected sections (all OK):**
|
|
8282
|
+
- Environment: node, bash, python3, bats
|
|
8283
|
+
- Global Install: .claude exists, agents/skills symlinks (0 broken), settings.json hooks
|
|
8284
|
+
- Hook Scripts: all present and executable
|
|
8285
|
+
- Hook Configuration: 14 events registered
|
|
8286
|
+
- Generated Artifacts: AGENTS.md, llms.txt, llms-full.txt
|
|
8287
|
+
- Planned Assets: plugin.json, benchmarks, plugin packs
|
|
8288
|
+
- Benchmark Freshness: < 30 days
|
|
8289
|
+
- Stale Rules: all healthy
|
|
8323
8290
|
|
|
8324
|
-
|
|
8325
|
-
|
|
8326
|
-
|
|
8291
|
+
**Verify:**
|
|
8292
|
+
- [ ] `Errors: 0 | Warnings: 0`
|
|
8293
|
+
- [ ] `HEALTH CHECK PASSED`
|
|
8327
8294
|
|
|
8328
|
-
|
|
8295
|
+
If doctor detects problems: `ai-toolkit doctor --fix` auto-repairs
|
|
8296
|
+
(broken symlinks, non-executable hooks, missing scripts, missing llms-full.txt).
|
|
8329
8297
|
|
|
8330
8298
|
---
|
|
8331
8299
|
|
|
8332
|
-
## Phase 4:
|
|
8333
|
-
|
|
8334
|
-
Use the npm scripts, not the generators directly:
|
|
8300
|
+
## Phase 4: Local Install (2 min)
|
|
8335
8301
|
|
|
8336
8302
|
```bash
|
|
8337
|
-
|
|
8338
|
-
|
|
8303
|
+
mkdir -p /tmp/ai-toolkit-verify && cd /tmp/ai-toolkit-verify
|
|
8304
|
+
git init -q
|
|
8305
|
+
ai-toolkit install --local --editors all --dry-run
|
|
8306
|
+
cd - && rm -rf /tmp/ai-toolkit-verify
|
|
8339
8307
|
```
|
|
8340
8308
|
|
|
8341
|
-
|
|
8342
|
-
|
|
8343
|
-
|
|
8344
|
-
|
|
8345
|
-
|
|
8309
|
+
**Verify "Project-local" section:**
|
|
8310
|
+
- [ ] Would create: CLAUDE.md
|
|
8311
|
+
- [ ] Would create: .claude/settings.local.json
|
|
8312
|
+
- [ ] Would inject: .claude/constitution.md
|
|
8313
|
+
- [ ] Editors: all 11 listed (copilot, cursor, windsurf, cline, roo, aider, augment, antigravity, codex, gemini, opencode)
|
|
8314
|
+
- [ ] Would generate configs for each editor (legacy + directory-based)
|
|
8315
|
+
- [ ] Would install: .git/hooks/pre-commit
|
|
8316
|
+
- [ ] Would inject language rules (auto-detected)
|
|
8346
8317
|
|
|
8318
|
+
**Also test auto-detect (no --editors flag):**
|
|
8347
8319
|
```bash
|
|
8348
|
-
|
|
8320
|
+
ai-toolkit install --local --dry-run
|
|
8321
|
+
# → Editors: none (empty project has no existing configs)
|
|
8349
8322
|
```
|
|
8350
8323
|
|
|
8351
|
-
If no diff, the artifacts are already current. If there is a diff, stage them.
|
|
8352
|
-
|
|
8353
8324
|
---
|
|
8354
8325
|
|
|
8355
|
-
## Phase 5:
|
|
8356
|
-
|
|
8357
|
-
Run the full quality gate:
|
|
8326
|
+
## Phase 5: Validation & Security Audit (3 min)
|
|
8358
8327
|
|
|
8359
8328
|
```bash
|
|
8360
8329
|
python3 scripts/validate.py --strict
|
|
8361
8330
|
python3 scripts/audit_skills.py --ci
|
|
8362
|
-
|
|
8363
|
-
python3 scripts/audit_skills.py --permissions # review Bash/Write/Edit footprint
|
|
8331
|
+
```
|
|
8364
8332
|
|
|
8365
|
-
|
|
8366
|
-
|
|
8367
|
-
|
|
8368
|
-
|
|
8369
|
-
|
|
8333
|
+
**Verify validate.py:**
|
|
8334
|
+
- [ ] Agents: 44, Skills: 108, Tests: exactly the current README badge count
|
|
8335
|
+
- [ ] Hook events: 14, Hook scripts: >= 30
|
|
8336
|
+
- [ ] Plugin packs >= 10, KB documents >= 20
|
|
8337
|
+
- [ ] `Errors: 0 | Warnings: 0` → `VALIDATION PASSED`
|
|
8370
8338
|
|
|
8371
|
-
|
|
8372
|
-
|
|
8373
|
-
|
|
8374
|
-
|
|
8375
|
-
<(ls scripts/generate_*.py | grep -vE "$META" | sort -u) \
|
|
8376
|
-
&& echo "OK: registry matches filesystem" \
|
|
8377
|
-
|| { echo "DRIFT: update supported-tools-registry.md before tagging"; exit 1; }
|
|
8339
|
+
**Verify audit_skills.py:**
|
|
8340
|
+
- [ ] `HIGH: 0` (MUST be zero — CI fails otherwise)
|
|
8341
|
+
- [ ] `WARN: 0`
|
|
8342
|
+
- [ ] `INFO: N` (acceptable — broad-access skills: orchestrate, swarm, workflow)
|
|
8378
8343
|
|
|
8379
|
-
|
|
8380
|
-
# "npm package KB files match the tracked release set" compares `git ls-files
|
|
8381
|
-
# kb` (the index) against what `npm pack` sees (the working tree), so an
|
|
8382
|
-
# unstaged addition reads as "extra" and an unstaged deletion as "missing".
|
|
8383
|
-
# Phase 6 stages, and it runs after this one, so the ordering fails the test
|
|
8384
|
-
# for any release that touches the KB. Staging early costs nothing.
|
|
8385
|
-
git status --porcelain kb/ | grep -qE '^(\?\?| D|\?M)' && git add -A kb/
|
|
8344
|
+
---
|
|
8386
8345
|
|
|
8387
|
-
|
|
8388
|
-
|
|
8389
|
-
|
|
8346
|
+
## Phase 6: Tests (3-5 min)
|
|
8347
|
+
|
|
8348
|
+
```bash
|
|
8349
|
+
# Run ONCE, capture to file, then parse. Full suite is 669+ bats cases —
|
|
8350
|
+
# re-running it per check (tail / grep ok / grep not ok piped separately)
|
|
8351
|
+
# wastes minutes every release. Always cache the output.
|
|
8390
8352
|
npm test > /tmp/npm-test.log 2>&1
|
|
8353
|
+
exit=$?
|
|
8391
8354
|
tail -3 /tmp/npm-test.log
|
|
8392
|
-
echo "ok:
|
|
8355
|
+
echo "ok: $(grep -c '^ok ' /tmp/npm-test.log)"
|
|
8356
|
+
echo "not ok: $(grep -c '^not ok' /tmp/npm-test.log)"
|
|
8357
|
+
echo "exit: $exit"
|
|
8393
8358
|
```
|
|
8394
8359
|
|
|
8395
|
-
**
|
|
8396
|
-
-
|
|
8397
|
-
-
|
|
8398
|
-
-
|
|
8399
|
-
-
|
|
8400
|
-
-
|
|
8401
|
-
|
|
8402
|
-
- `npm test`: `1..N` with zero `not ok` (read from the cached `/tmp/npm-test.log`, do not rerun)
|
|
8403
|
-
|
|
8404
|
-
> **Why this matters (v4.5.1 postmortem):** `publish.yml` runs only `validate.py` + `npm test`, so it published v4.5.0 even though the `main` CI `ShellCheck hooks` job was red on two `SC2034` warnings in a new hook. The publish workflow does **not** depend on the CI workflow. Until that is fixed, ShellCheck is a manual pre-tag gate — run it here every time.
|
|
8360
|
+
**Verify:**
|
|
8361
|
+
- [ ] `exit == 0`
|
|
8362
|
+
- [ ] `ok == expected test count` (e.g., 945 on v3.0.0)
|
|
8363
|
+
- [ ] `not ok == 0`
|
|
8364
|
+
- [ ] Bats runs tests in parallel (4 jobs)
|
|
8365
|
+
- [ ] Groups: agents, autodetect, cli, generators, guards, hooks, inject,
|
|
8366
|
+
install, kb, mcp, readme, profiles, uninstall, validate
|
|
8405
8367
|
|
|
8406
|
-
**
|
|
8368
|
+
**Anti-pattern — do NOT do this:**
|
|
8407
8369
|
```bash
|
|
8408
|
-
|
|
8370
|
+
# Runs the full suite THREE times. Adds 1-3 min and pressures CI capacity.
|
|
8371
|
+
npm test 2>&1 | tail -3
|
|
8372
|
+
npm test 2>&1 | grep -c '^ok '
|
|
8373
|
+
npm test 2>&1 | grep -c '^not ok'
|
|
8409
8374
|
```
|
|
8410
8375
|
|
|
8411
|
-
**
|
|
8412
|
-
-
|
|
8413
|
-
-
|
|
8414
|
-
-
|
|
8376
|
+
**Key test areas:**
|
|
8377
|
+
- Guards: rm -rf, DROP TABLE, git push --force blocked
|
|
8378
|
+
- Install: idempotent, profiles, --only/--skip, orphan cleanup
|
|
8379
|
+
- Eject: real files (not symlinks), inlined rules
|
|
8380
|
+
- Uninstall: removes toolkit, preserves user content
|
|
8415
8381
|
|
|
8416
|
-
|
|
8382
|
+
---
|
|
8417
8383
|
|
|
8418
|
-
|
|
8419
|
-
carries it, and every part of that is mechanically checkable.
|
|
8384
|
+
## Phase 7: Eject (1 min)
|
|
8420
8385
|
|
|
8421
8386
|
```bash
|
|
8422
|
-
|
|
8423
|
-
|
|
8387
|
+
mkdir -p /tmp/ai-toolkit-eject-test
|
|
8388
|
+
cd /tmp/ai-toolkit-eject-test
|
|
8389
|
+
ai-toolkit eject
|
|
8390
|
+
cd - && rm -rf /tmp/ai-toolkit-eject-test
|
|
8424
8391
|
```
|
|
8425
8392
|
|
|
8426
|
-
|
|
8393
|
+
**Verify:**
|
|
8394
|
+
- [ ] Agents copied as real files (not symlinks)
|
|
8395
|
+
- [ ] Skills copied as real directories
|
|
8396
|
+
- [ ] Rules inlined into CLAUDE.md
|
|
8397
|
+
- [ ] constitution.md and ARCHITECTURE.md copied
|
|
8398
|
+
- [ ] `output-styles/` directory present (v2.7.1+)
|
|
8427
8399
|
|
|
8428
|
-
|
|
8429
|
-
|---|---|
|
|
8430
|
-
| Every shipped source file carries an SPDX header | A new `.py`/`.sh`/`.js`/`.bats` file was added without one — the common case, and the reason this is a test rather than a habit |
|
|
8431
|
-
| Headers name Apache-2.0 and nothing else | A file was copied in from an MIT/GPL source with its own header intact |
|
|
8432
|
-
| **No** markdown file carries an SPDX header | Someone "helpfully" ran the header script over `app/skills/` — headers there sit above parsed frontmatter and bill every session for it |
|
|
8433
|
-
| `LICENSE` is the complete Apache 2.0 text | The file was truncated or replaced with a summary |
|
|
8434
|
-
| `NOTICE` carries attribution, the source URL, §4(d) and the MIT-era notice | The attribution mechanism was gutted |
|
|
8435
|
-
| `LICENSE` **and** `NOTICE` ship in the npm package | `package.json` `files` lost an entry — a NOTICE that never reaches the consumer cannot satisfy §4(d) |
|
|
8436
|
-
| Every manifest declaring a licence declares Apache-2.0 | `package.json`, `manifest.json`, `plugin.json` and `package-lock.json` drifted apart |
|
|
8400
|
+
---
|
|
8437
8401
|
|
|
8438
|
-
|
|
8439
|
-
never before it. Short SPDX form:
|
|
8402
|
+
## Phase 8: Supply-Chain Verification (2 min, v2.8.0+)
|
|
8440
8403
|
|
|
8441
|
-
|
|
8442
|
-
# SPDX-License-Identifier: Apache-2.0
|
|
8443
|
-
# Copyright 2024-2026 Lukasz Krzemien (biuro@softspark.eu)
|
|
8444
|
-
# Source: https://github.com/softspark/ai-toolkit
|
|
8445
|
-
```
|
|
8404
|
+
These checks enforce the v2.8.0 security standard on a freshly-published release. Run AFTER the `publish.yml` workflow completes on the tag.
|
|
8446
8405
|
|
|
8447
|
-
|
|
8448
|
-
exclusion: [Licensing](../reference/licensing.md).
|
|
8406
|
+
### 8.1 Provenance attestation on npm
|
|
8449
8407
|
|
|
8450
|
-
|
|
8451
|
-
|
|
8452
|
-
|
|
8453
|
-
|
|
8454
|
-
|
|
8408
|
+
```bash
|
|
8409
|
+
VERSION="X.Y.Z" # the tag just published
|
|
8410
|
+
npm view "@softspark/ai-toolkit@${VERSION}" --json \
|
|
8411
|
+
| python3 -c "import json, sys; d=json.load(sys.stdin); att=d['dist'].get('attestations', {}); assert att.get('provenance', {}).get('predicateType') == 'https://slsa.dev/provenance/v1', f'NO PROVENANCE for {d[\"version\"]}'; print(f'PROVENANCE OK: {att[\"url\"]}')"
|
|
8412
|
+
```
|
|
8455
8413
|
|
|
8456
|
-
|
|
8414
|
+
**Verify:**
|
|
8415
|
+
- [ ] Exit 0 and prints `PROVENANCE OK: https://registry.npmjs.org/...`
|
|
8416
|
+
- [ ] `https://www.npmjs.com/package/@softspark/ai-toolkit/v/${VERSION}` shows the green "Provenance" badge
|
|
8457
8417
|
|
|
8458
|
-
|
|
8418
|
+
**If provenance is missing:** the `publish.yml` ran without `id-token: write` or `--provenance`. Restore them and cut a patch release — a silently unsigned publish is a regression against the v2.8.0 standard.
|
|
8459
8419
|
|
|
8460
|
-
|
|
8420
|
+
### 8.2 Audit SARIF output (for GHAS ingest)
|
|
8461
8421
|
|
|
8462
8422
|
```bash
|
|
8463
|
-
|
|
8423
|
+
python3 scripts/audit_skills.py --sarif > /tmp/audit.sarif
|
|
8424
|
+
python3 -c "import json; d=json.load(open('/tmp/audit.sarif')); assert d['version']=='2.1.0' and d['runs'][0]['tool']['driver']['name']=='ai-toolkit-audit-skills'; print(f'SARIF OK: {len(d[\"runs\"][0][\"results\"])} results across {len(d[\"runs\"][0][\"tool\"][\"driver\"][\"rules\"])} rules')"
|
|
8464
8425
|
```
|
|
8465
8426
|
|
|
8466
|
-
|
|
8467
|
-
- [ ]
|
|
8427
|
+
**Verify:**
|
|
8428
|
+
- [ ] Valid SARIF 2.1.0
|
|
8429
|
+
- [ ] In the publishing repo, the CI job uploads `audit.sarif` via `github/codeql-action/upload-sarif@v3` so findings appear in the Security tab
|
|
8468
8430
|
|
|
8469
|
-
|
|
8431
|
+
### 8.3 Per-skill permissions report
|
|
8470
8432
|
|
|
8471
8433
|
```bash
|
|
8472
|
-
|
|
8473
|
-
# (schema_version 1: entries live under the .rules / .hooks key):
|
|
8474
|
-
jq '.rules | to_entries | map(select(.value.url != null and (.value.sha256 // "" | length) == 0))' ~/.softspark/ai-toolkit/rules/sources.json
|
|
8475
|
-
jq '.hooks | to_entries | map(select(.value.url != null and (.value.sha256 // "" | length) == 0))' ~/.softspark/ai-toolkit/hooks/external/sources.json
|
|
8434
|
+
python3 scripts/audit_skills.py --permissions | head -40
|
|
8476
8435
|
```
|
|
8477
8436
|
|
|
8478
|
-
|
|
8479
|
-
- [ ]
|
|
8437
|
+
**Verify:**
|
|
8438
|
+
- [ ] Bash skill count has NOT jumped unexpectedly since the previous release
|
|
8439
|
+
- [ ] Any newly added entry under `Skills with Bash + Write + Edit` matches a CHANGELOG bullet that justifies the broad scope
|
|
8440
|
+
- [ ] JSON form (`--permissions --json`) is available for automated drift dashboards
|
|
8480
8441
|
|
|
8481
|
-
|
|
8442
|
+
### 8.4 URL-sourced rules/hooks are checksum-pinned
|
|
8443
|
+
|
|
8444
|
+
`sources.json` is an envelope of the form `{"schema_version": 1, "rules"|"hooks": {...}}`, so the jq filter must pick the nested map before piping into the pin assertion. The URL-count in the success message ignores local `path:`-only entries — only entries with a `url:` field are pinned and counted.
|
|
8482
8445
|
|
|
8483
8446
|
```bash
|
|
8484
|
-
|
|
8447
|
+
jq '.rules // .hooks // {}' ~/.softspark/ai-toolkit/rules/sources.json 2>/dev/null \
|
|
8448
|
+
| python3 -c "import json, sys; d=json.load(sys.stdin) or {}; bad=[n for n,v in d.items() if v.get('url') and not v.get('sha256')]; assert not bad, f'UNPINNED: {bad}'; url_n=sum(1 for v in d.values() if v.get('url')); print(f'RULE PIN OK: {url_n} URL rules, all with sha256')"
|
|
8449
|
+
jq '.hooks // {}' ~/.softspark/ai-toolkit/hooks/external/sources.json 2>/dev/null \
|
|
8450
|
+
| python3 -c "import json, sys; d=json.load(sys.stdin) or {}; bad=[n for n,v in d.items() if v.get('url') and not v.get('sha256')]; assert not bad, f'UNPINNED: {bad}'; url_n=sum(1 for v in d.values() if v.get('url')); print(f'HOOK PIN OK: {url_n} URL hooks, all with sha256')"
|
|
8485
8451
|
```
|
|
8486
8452
|
|
|
8487
|
-
|
|
8488
|
-
- [ ]
|
|
8453
|
+
**Verify:**
|
|
8454
|
+
- [ ] Both commands print `... PIN OK`
|
|
8455
|
+
- [ ] If any entry is unpinned, the `register_url_source()` call missed passing `content=` — fix the call-site and retag
|
|
8489
8456
|
|
|
8490
|
-
|
|
8457
|
+
### 8.5 Strict-pin smoke test (optional but recommended)
|
|
8491
8458
|
|
|
8492
8459
|
```bash
|
|
8493
8460
|
AI_TOOLKIT_STRICT_PIN=1 ai-toolkit update --dry-run
|
|
8494
8461
|
```
|
|
8495
8462
|
|
|
8463
|
+
**Verify:**
|
|
8496
8464
|
- [ ] Exit 0, no `CHECKSUM CHANGED` line
|
|
8497
|
-
- [ ]
|
|
8465
|
+
- [ ] If a checksum change was intentional (e.g. upstream rule update), document it in the CHANGELOG entry before tagging
|
|
8498
8466
|
|
|
8499
8467
|
---
|
|
8500
8468
|
|
|
8501
|
-
## Phase
|
|
8469
|
+
## Phase 9: Deep-Coverage Checks (v3.0.0+)
|
|
8502
8470
|
|
|
8503
|
-
|
|
8471
|
+
These verify the native-surface generators shipped in v3.0.0 actually emit the right files for the right profiles, and that the tool registry stays in sync with shipped generators.
|
|
8472
|
+
|
|
8473
|
+
> **Safety warning — HOME-scoped writes:** Running `--profile full` with `augment` in the editor list writes to `$HOME/.augment/settings.json` (Augment stores hooks under HOME, not per-project). Use `--dry-run` for verification unless you intend to carry ai-toolkit hook entries on this machine. The generator is marker-safe (only rewrites its own `_source: ai-toolkit` entries) but is still a side-effect.
|
|
8474
|
+
|
|
8475
|
+
### 9.1 `--profile full` emits every native surface
|
|
8504
8476
|
|
|
8505
8477
|
```bash
|
|
8506
|
-
|
|
8507
|
-
|
|
8508
|
-
|
|
8509
|
-
|
|
8510
|
-
# NOTE: AGENTS.md, GEMINI.md, and .github/copilot-instructions.md are generated
|
|
8511
|
-
# editor configs and are gitignored — do NOT commit them. `prepublishOnly` runs
|
|
8512
|
-
# `npm run generate:all`, so the shipped package (which lists AGENTS.md in
|
|
8513
|
-
# package.json `files`) gets a fresh copy at publish time.
|
|
8514
|
-
git add -p # review and stage any other changes
|
|
8478
|
+
D=/tmp/aitk-profile-full-${RANDOM} && mkdir -p "$D" && cd "$D" && git init -q
|
|
8479
|
+
ai-toolkit install --local --editors cursor,windsurf,gemini,augment,codex,copilot \
|
|
8480
|
+
--profile full --dry-run 2>&1 \
|
|
8481
|
+
| grep -E "\\.cursor/(hooks\\.json|agents)|\\.devin/hooks\\.v1\\.json|\\.gemini/(settings\\.json|commands)|\\.augment/(agents|commands)|\\.codex/(agents|hooks)|\\.agents/skills|\\.github/(agents|skills|instructions|prompts|hooks)"
|
|
8515
8482
|
```
|
|
8516
8483
|
|
|
8517
|
-
|
|
8484
|
+
**Verify** — at least the following lines appear:
|
|
8485
|
+
- [ ] `.cursor/hooks.json` and `.cursor/agents/`
|
|
8486
|
+
- [ ] `.devin/hooks.v1.json`
|
|
8487
|
+
- [ ] `.gemini/settings.json` hooks AND `.gemini/commands/`
|
|
8488
|
+
- [ ] `.augment/agents/` + `.augment/commands/` + `$HOME/.augment/settings.json`
|
|
8489
|
+
- [ ] `.codex/hooks.json` + `.codex/hooks/` + `.codex/agents/`
|
|
8490
|
+
- [ ] `.agents/skills/` (Codex native discovery path)
|
|
8491
|
+
- [ ] `.github/agents/` + `.github/skills/` + `.github/instructions/` + `.github/prompts/` + `.github/hooks/`
|
|
8492
|
+
|
|
8493
|
+
### 9.2 Codex skills are profile-independent
|
|
8518
8494
|
|
|
8519
8495
|
```bash
|
|
8520
|
-
|
|
8496
|
+
D=/tmp/aitk-codex-skills-${RANDOM} && mkdir -p "$D" && cd "$D" && git init -q
|
|
8497
|
+
ai-toolkit install --local --editors codex --profile minimal --dry-run 2>&1 \
|
|
8498
|
+
| grep -q "Would generate: .agents/skills" && echo "OK: Codex skills emit at minimal"
|
|
8499
|
+
ai-toolkit install --local --editors codex --profile standard --codex-skills --dry-run 2>&1 \
|
|
8500
|
+
| grep -q "Would refresh: .agents/skills" && echo "OK: legacy flag explicitly refreshes the same catalog"
|
|
8521
8501
|
```
|
|
8522
8502
|
|
|
8523
|
-
|
|
8503
|
+
**Verify:**
|
|
8504
|
+
- [ ] Selecting Codex emits all skills under `.agents/skills/` at every profile
|
|
8505
|
+
- [ ] `--codex-skills` remains an explicit refresh alias; it is not required for installation
|
|
8506
|
+
- [ ] No profile emits `.codex/skills/`
|
|
8507
|
+
|
|
8508
|
+
### 9.3 Breaking-change surfaces land on `--profile standard`
|
|
8524
8509
|
|
|
8525
|
-
|
|
8510
|
+
The standard profile must include:
|
|
8511
|
+
- Copilot scoped instructions, prompts, custom agents, portable skills, and hooks
|
|
8512
|
+
- Gemini hooks (`.gemini/settings.json`)
|
|
8526
8513
|
|
|
8527
8514
|
```bash
|
|
8528
|
-
git
|
|
8515
|
+
D=/tmp/aitk-breaking-${RANDOM} && mkdir -p "$D" && cd "$D" && git init -q
|
|
8516
|
+
ai-toolkit install --local --editors copilot,gemini --profile standard --dry-run 2>&1 \
|
|
8517
|
+
| tee /tmp/aitk-breaking.log
|
|
8518
|
+
grep -q "\\.github/instructions/" /tmp/aitk-breaking.log && echo "OK: Copilot dir layout at standard"
|
|
8519
|
+
grep -q "\\.github/agents/.*\\.github/skills/" /tmp/aitk-breaking.log && echo "OK: Copilot agents and skills at standard"
|
|
8520
|
+
grep -q "\\.github/hooks/" /tmp/aitk-breaking.log && echo "OK: Copilot hooks at standard"
|
|
8521
|
+
grep -q "\\.gemini/settings\\.json hooks" /tmp/aitk-breaking.log && echo "OK: Gemini hooks at standard"
|
|
8522
|
+
```
|
|
8529
8523
|
|
|
8530
|
-
|
|
8531
|
-
# have caught a real broken release.
|
|
8532
|
-
test "$(git rev-parse vX.Y.Z)" = "$(git rev-parse HEAD)" \
|
|
8533
|
-
|| { echo "FAIL: tag is not on HEAD"; exit 1; }
|
|
8534
|
-
git show --no-patch --format=%s vX.Y.Z | grep -qx "chore: release vX.Y.Z" \
|
|
8535
|
-
|| { echo "FAIL: tag is not on the chore: release commit"; exit 1; }
|
|
8524
|
+
**Verify all four lines print `OK:`**. If any is missing, native Copilot or Gemini coverage has regressed.
|
|
8536
8525
|
|
|
8537
|
-
|
|
8538
|
-
git push origin main
|
|
8539
|
-
git push origin refs/tags/vX.Y.Z
|
|
8540
|
-
```
|
|
8526
|
+
### 9.4 Install is idempotent
|
|
8541
8527
|
|
|
8542
|
-
|
|
8543
|
-
|
|
8544
|
-
|
|
8545
|
-
|
|
8546
|
-
|
|
8547
|
-
|
|
8548
|
-
|
|
8528
|
+
```bash
|
|
8529
|
+
D=/tmp/aitk-idem-${RANDOM} && mkdir -p "$D" && cd "$D" && git init -q
|
|
8530
|
+
# Sort file paths before hashing — find traversal order follows inode order,
|
|
8531
|
+
# which can shift between runs even when content is byte-identical, producing
|
|
8532
|
+
# false FAIL signals.
|
|
8533
|
+
ai-toolkit install --local --editors cursor,gemini --profile full >/dev/null 2>&1
|
|
8534
|
+
SHA1=$(find .cursor .gemini -type f -print0 | LC_ALL=C sort -z | xargs -0 shasum | shasum | awk '{print $1}')
|
|
8535
|
+
ai-toolkit install --local --editors cursor,gemini --profile full >/dev/null 2>&1
|
|
8536
|
+
SHA2=$(find .cursor .gemini -type f -print0 | LC_ALL=C sort -z | xargs -0 shasum | shasum | awk '{print $1}')
|
|
8537
|
+
[ "$SHA1" = "$SHA2" ] && echo "OK: idempotent" || echo "FAIL: install is not idempotent"
|
|
8538
|
+
```
|
|
8549
8539
|
|
|
8550
|
-
**
|
|
8551
|
-
suppresses tag-triggered workflow runs when many tags arrive in a single push
|
|
8552
|
-
— the workflow silently does not fire and nothing publishes. Push the single
|
|
8553
|
-
release tag by its full ref, as above. (Sibling evidence: this is exactly how
|
|
8554
|
-
rag-mcp's `1.0.3` image build was skipped, when a `--tags` push carried 37
|
|
8555
|
-
tags at once.)
|
|
8540
|
+
**Verify:** prints `OK: idempotent`. A second run must produce byte-identical files in every managed path.
|
|
8556
8541
|
|
|
8557
|
-
|
|
8558
|
-
1. Runs `validate.py --strict`
|
|
8559
|
-
2. Runs `npm test`
|
|
8560
|
-
3. Publishes to npm as `@softspark/ai-toolkit@X.Y.Z` with `--provenance` (SLSA v1 build attestation)
|
|
8542
|
+
### 9.5 Live-install JSON outputs parse
|
|
8561
8543
|
|
|
8562
|
-
|
|
8544
|
+
The bats suite validates JSON shape at generation time. This re-checks that what actually landed on disk after a live install parses without errors.
|
|
8563
8545
|
|
|
8564
|
-
|
|
8565
|
-
|
|
8546
|
+
```bash
|
|
8547
|
+
D=/tmp/aitk-json-${RANDOM} && mkdir -p "$D" && cd "$D" && git init -q
|
|
8548
|
+
ai-toolkit install --local --editors cursor,windsurf,gemini,augment,codex,copilot --profile full >/dev/null 2>&1
|
|
8549
|
+
for f in .cursor/hooks.json .devin/hooks.v1.json .gemini/settings.json .codex/hooks.json .github/hooks/ai-toolkit.json "$HOME/.augment/settings.json"; do
|
|
8550
|
+
[ -f "$f" ] && python3 -c "import json; json.load(open('$f'))" && echo "OK: $f"
|
|
8551
|
+
done
|
|
8552
|
+
```
|
|
8566
8553
|
|
|
8567
|
-
|
|
8554
|
+
**Verify:** each emitted file prints `OK: <path>`. Any `json.decoder.JSONDecodeError` means the merge logic corrupted the output.
|
|
8568
8555
|
|
|
8569
|
-
|
|
8556
|
+
### 9.6 Registry / generator drift check
|
|
8570
8557
|
|
|
8571
|
-
|
|
8558
|
+
`kb/reference/supported-tools-registry.md` should enumerate every per-editor `scripts/generate_*.py` we ship. Meta-generators (`generate_agents_md.py`, `generate_llms_txt.py`) are excluded — they produce docs/artifacts, not editor configs.
|
|
8572
8559
|
|
|
8573
8560
|
```bash
|
|
8574
|
-
|
|
8575
|
-
|
|
8561
|
+
META="generate_agents_md.py|generate_llms_txt.py|generate_language_rules_skills.py"
|
|
8562
|
+
REG=$(grep -oE 'scripts/generate_[a-z_]+\.py' kb/reference/supported-tools-registry.md | sort -u)
|
|
8563
|
+
FS=$(ls scripts/generate_*.py | grep -vE "$META" | sort -u)
|
|
8564
|
+
diff <(echo "$REG") <(echo "$FS") && echo "OK: registry matches filesystem" || echo "DRIFT: update supported-tools-registry.md"
|
|
8565
|
+
```
|
|
8576
8566
|
|
|
8577
|
-
|
|
8578
|
-
npm deprecate @softspark/ai-toolkit@X.Y.Z "Known issue: <description>. Use vA.B.C instead."
|
|
8567
|
+
**Verify:** prints `OK: registry matches filesystem`. If not, add the missing rows to the registry before tagging the next release.
|
|
8579
8568
|
|
|
8580
|
-
|
|
8581
|
-
|
|
8582
|
-
|
|
8569
|
+
### 9.7 Claude Chat / Cowork plugin validates and exports
|
|
8570
|
+
|
|
8571
|
+
```bash
|
|
8572
|
+
python3 scripts/claude_app.py verify
|
|
8573
|
+
D=/tmp/aitk-claude-app-${RANDOM}
|
|
8574
|
+
python3 scripts/claude_app.py export --output "$D.zip" --no-custom-rules
|
|
8575
|
+
python3 -c "import zipfile; z=zipfile.ZipFile('$D.zip'); assert '.claude-plugin/plugin.json' in z.namelist(); assert 'claude-app/skills/ai-toolkit-rules/SKILL.md' in z.namelist(); print('OK: Claude app archive')"
|
|
8583
8576
|
```
|
|
8584
8577
|
|
|
8578
|
+
**Verify:** the official validator exits 0; the archive contains the manifest,
|
|
8579
|
+
app-native rules skill, bundled agents/skills, and plugin-relative Cowork hooks.
|
|
8580
|
+
|
|
8585
8581
|
---
|
|
8586
8582
|
|
|
8587
|
-
##
|
|
8583
|
+
## Phase 10: Explicit DSH Qualification (v4.30.0+)
|
|
8588
8584
|
|
|
8589
|
-
|
|
8590
|
-
|---|------|-----------------|---------------|
|
|
8591
|
-
| 0a | Ecosystem drift check | `ecosystem_doctor.py --format text` | All tools Clean, or drift classified and resolved |
|
|
8592
|
-
| 0b | Ecosystem snapshot refresh | `ecosystem_doctor.py --update` | `benchmarks/ecosystem-doctor-snapshot.json` updated |
|
|
8593
|
-
| 0c | Ecosystem gate | `ecosystem_doctor.py --offline --check` | Exit 0 |
|
|
8594
|
-
| 1 | Version bump type | Decide patch/minor/major | — |
|
|
8595
|
-
| 2 | `package.json` version | Edit `"version"` | Matches target |
|
|
8596
|
-
| 3 | `manifest.json` version | Edit `"version"` | Matches target |
|
|
8597
|
-
| 4 | `plugin.json` version | Edit `"version"` | Matches target |
|
|
8598
|
-
| 5 | `package-lock.json` | `npm install --package-lock-only` | Matches target |
|
|
8599
|
-
| 6 | Count sync | Check `package.json` description, README | `validate.py` passes |
|
|
8600
|
-
| 7 | CHANGELOG.md | Add release entry (incl. `Ecosystem` subsection if any B/D/E/F drift) | Entry exists for vX.Y.Z |
|
|
8601
|
-
| 8 | Regenerate artifacts | `generate_agents_md.py`, `generate_llms_txt.py` | No unexpected diff |
|
|
8602
|
-
| 9 | Validate | `validate.py --strict` | 0 errors, 0 warnings |
|
|
8603
|
-
| 10 | Security audit (CI mode) | `audit_skills.py --ci` | 0 HIGH |
|
|
8604
|
-
| 11 | Security audit (SARIF) | `audit_skills.py --sarif` | Valid SARIF 2.1.0 JSON |
|
|
8605
|
-
| 12 | Per-skill permissions | `audit_skills.py --permissions` | New broad-access skills justified in CHANGELOG |
|
|
8606
|
-
| 13 | ShellCheck hooks | `shellcheck --severity=warning app/hooks/*.sh` | Exit 0, no output (mirrors ci.yml; publish.yml does NOT run it) |
|
|
8607
|
-
| 14 | Provenance flag check | `grep -- '--provenance' .github/workflows/publish.yml` | Present |
|
|
8608
|
-
| 15 | Checksum-pin backfill | `sources.json` entries all have `sha256` | No unpinned URL sources |
|
|
8609
|
-
| 15a | Licensing gate | `npx bats tests/test_licensing.bats` | 7/7 — SPDX headers, LICENSE, NOTICE, npm `files`, manifest consistency |
|
|
8610
|
-
| 16 | Tests | `git add -A kb/` if the KB changed, then `npm test` | All pass |
|
|
8611
|
-
| 17 | Commit | `git commit` | Clean working tree |
|
|
8612
|
-
| 18 | Tag | `git tag vX.Y.Z` | Tag exists |
|
|
8613
|
-
| 18a | Tag is on HEAD | `test "$(git rev-parse vX.Y.Z)" = "$(git rev-parse HEAD)"` | Exit 0 |
|
|
8614
|
-
| 18b | Tag is on the release commit | `git show --no-patch --format=%s vX.Y.Z` | Reads `chore: release vX.Y.Z` |
|
|
8615
|
-
| 19 | Push branch, then the single tag | `git push origin main && git push origin refs/tags/vX.Y.Z` | CI triggered with `id-token: write`. Never `--tags`. |
|
|
8585
|
+
Run this phase whenever the release changes the `dsh` target, package pins, preset lifecycle, or DSH compatibility documentation. Use a new task-specific `DSH_HOME`; never replace `HOME` or reuse a regular profile.
|
|
8616
8586
|
|
|
8617
|
-
|
|
8587
|
+
Prerequisites: DSH `0.1.1-rc.2`, pnpm `>=11.7.0,<12.0.0`, Codex logged in through ChatGPT, Claude Code logged in natively, and GitHub Copilot CLI logged in natively. Do not supply provider API keys.
|
|
8618
8588
|
|
|
8619
|
-
|
|
8589
|
+
```bash
|
|
8590
|
+
DSH_SMOKE_ROOT="$(mktemp -d)"
|
|
8591
|
+
export DSH_HOME="$DSH_SMOKE_ROOT/dsh-home"
|
|
8620
8592
|
|
|
8621
|
-
|
|
8622
|
-
|
|
8623
|
-
|
|
8624
|
-
|
|
8625
|
-
tags: [sop, verification, release, smoke-test, install, update, qa, provenance, sarif, dsh]
|
|
8626
|
-
version: "1.8.0"
|
|
8627
|
-
created: "2026-04-08"
|
|
8628
|
-
last_updated: "2026-09-01"
|
|
8629
|
-
description: "End-to-end smoke test after installing or updating @softspark/ai-toolkit. Verifies CLI, native Codex and GitHub Copilot surfaces, explicit DSH lifecycle, Claude app export, doctor, validation, tests, eject, provenance, SARIF, and per-skill permissions."
|
|
8630
|
-
---
|
|
8593
|
+
ai-toolkit dsh install --profile web
|
|
8594
|
+
ai-toolkit dsh doctor --profile web
|
|
8595
|
+
dsh --profile web --host 127.0.0.1 --port 0 --no-open
|
|
8596
|
+
```
|
|
8631
8597
|
|
|
8632
|
-
|
|
8598
|
+
In a new `softspark-orchestrator` session, select the `codex` provider and run two standalone marker prompts:
|
|
8633
8599
|
|
|
8634
|
-
|
|
8635
|
-
|
|
8600
|
+
1. `subagent_claude_code` returns an exact child marker, then the Codex parent returns its exact completion marker.
|
|
8601
|
+
2. `subagent_gemini_copilot` returns an exact child marker, then the Codex parent returns its exact completion marker.
|
|
8636
8602
|
|
|
8637
|
-
|
|
8638
|
-
- After `npm install -g @softspark/ai-toolkit@latest`
|
|
8639
|
-
- After `ai-toolkit update`
|
|
8640
|
-
- Before tagging a new version (`git tag`)
|
|
8641
|
-
- Before publishing to npm (`npm publish`)
|
|
8642
|
-
- As a smoke test in CI/CD
|
|
8603
|
+
Stop DSH, then remove only the managed profile artifacts:
|
|
8643
8604
|
|
|
8644
|
-
|
|
8645
|
-
-
|
|
8646
|
-
|
|
8605
|
+
```bash
|
|
8606
|
+
ai-toolkit dsh uninstall --profile web --yes
|
|
8607
|
+
```
|
|
8647
8608
|
|
|
8648
|
-
**
|
|
8609
|
+
**Verify:** both tool results have `isError=false`, both turns end as `completed`, `doctor` reports no recovery requirement before uninstall, and an unrelated preset fixture remains unchanged. Preserve only redacted event sequence evidence; never attach credentials, auth files, or full private prompts.
|
|
8649
8610
|
|
|
8650
8611
|
---
|
|
8651
8612
|
|
|
8652
|
-
##
|
|
8613
|
+
## Troubleshooting
|
|
8653
8614
|
|
|
8654
|
-
|
|
8615
|
+
### `ai-toolkit: command not found`
|
|
8655
8616
|
|
|
8656
8617
|
```bash
|
|
8657
|
-
|
|
8658
|
-
|
|
8659
|
-
|
|
8660
|
-
|
|
8661
|
-
npm test > /tmp/npm-test.log 2>&1 && grep -c '^ok ' /tmp/npm-test.log && ! grep -q '^not ok' /tmp/npm-test.log # 4. All tests passed? (single run, cached)
|
|
8662
|
-
|
|
8663
|
-
# Post-install verification (Phases 1-7)
|
|
8664
|
-
ai-toolkit --version # 5. Version OK?
|
|
8665
|
-
ai-toolkit status # 6. Status OK?
|
|
8666
|
-
ai-toolkit doctor # 7. Health check passed?
|
|
8667
|
-
ai-toolkit install --dry-run # 8. Global install OK?
|
|
8668
|
-
python3 scripts/audit_skills.py --ci # 9. Security audit clean?
|
|
8618
|
+
npm install -g @softspark/ai-toolkit
|
|
8619
|
+
# or check PATH:
|
|
8620
|
+
export PATH="$(npm config get prefix)/bin:$PATH"
|
|
8621
|
+
```
|
|
8669
8622
|
|
|
8670
|
-
|
|
8671
|
-
python3 scripts/audit_skills.py --sarif | python3 -c "import json,sys; assert json.load(sys.stdin)['version']=='2.1.0'; print('SARIF OK')" # 10. SARIF 2.1.0 well-formed?
|
|
8672
|
-
python3 scripts/audit_skills.py --permissions | head -30 # 11. Broad-access skills reviewed?
|
|
8673
|
-
npm view @softspark/ai-toolkit@X.Y.Z --json | python3 -c "import json,sys; d=json.load(sys.stdin); assert d['dist']['attestations']['provenance']['predicateType']=='https://slsa.dev/provenance/v1'; print('PROVENANCE OK')" # 12. Provenance attested on npm?
|
|
8674
|
-
python3 scripts/claude_app.py verify # 13. Claude Chat/Cowork plugin contract valid?
|
|
8623
|
+
### Doctor: broken symlinks
|
|
8675
8624
|
|
|
8676
|
-
|
|
8677
|
-
|
|
8678
|
-
|
|
8625
|
+
```bash
|
|
8626
|
+
ai-toolkit doctor --fix # auto-repair
|
|
8627
|
+
ai-toolkit update # or full re-install
|
|
8628
|
+
```
|
|
8629
|
+
|
|
8630
|
+
### Tests fail: missing bats
|
|
8631
|
+
|
|
8632
|
+
```bash
|
|
8633
|
+
brew install bats-core # macOS
|
|
8634
|
+
npm install -g bats # cross-platform
|
|
8635
|
+
```
|
|
8636
|
+
|
|
8637
|
+
### validate.py: stale counts
|
|
8638
|
+
|
|
8639
|
+
README badges don't match the current agents/skills/tests counts.
|
|
8640
|
+
Update README.md and re-run.
|
|
8641
|
+
|
|
8642
|
+
### Eject: missing skills
|
|
8643
|
+
|
|
8644
|
+
```bash
|
|
8645
|
+
ai-toolkit update # re-link missing symlinks
|
|
8646
|
+
ai-toolkit eject /tmp/test # retry
|
|
8679
8647
|
```
|
|
8680
8648
|
|
|
8681
8649
|
---
|
|
8682
8650
|
|
|
8683
|
-
##
|
|
8651
|
+
## Success Criteria
|
|
8684
8652
|
|
|
8685
|
-
|
|
8686
|
-
|
|
8653
|
+
| Area | Criterion |
|
|
8654
|
+
|------|-----------|
|
|
8655
|
+
| CLI | `--version` correct, `--help` full list, `status` current |
|
|
8656
|
+
| Health | `doctor`: 0 errors, 0 warnings, PASSED |
|
|
8657
|
+
| Install | `--dry-run` correct counts, `--local` all configs |
|
|
8658
|
+
| Quality | `validate.py --strict`: PASSED |
|
|
8659
|
+
| Security (baseline) | `audit_skills.py --ci`: 0 HIGH |
|
|
8660
|
+
| Security (SARIF) | `audit_skills.py --sarif`: valid SARIF 2.1.0 with non-empty rules array |
|
|
8661
|
+
| Security (permissions) | `audit_skills.py --permissions`: broad-access skills unchanged or justified in CHANGELOG |
|
|
8662
|
+
| Supply chain | `dist.attestations.provenance.predicateType == https://slsa.dev/provenance/v1` on npm |
|
|
8663
|
+
| Supply chain | All `sources.json` URL entries carry a `sha256`; `AI_TOOLKIT_STRICT_PIN=1 ai-toolkit update --dry-run` passes |
|
|
8664
|
+
| Tests | `npm test`: N/N passed, 0 failures |
|
|
8665
|
+
| Eject | Standalone `.claude/` with real files AND `output-styles/` directory |
|
|
8666
|
+
| Guards | Destructive commands blocked |
|
|
8667
|
+
| Deep coverage | `--profile full` emits every documented native surface, including Codex agents/skills/hooks and Copilot agents/skills/hooks |
|
|
8668
|
+
| Profile contract | Codex skills emit at every profile without `--codex-skills`; Copilot standard emits agents, skills, instructions, prompts, and hooks |
|
|
8669
|
+
| Idempotence | Second `install` run produces byte-identical output in every managed path |
|
|
8670
|
+
| Live JSON | Every generated `.json` file on disk parses as valid JSON |
|
|
8671
|
+
| Registry | `supported-tools-registry.md` enumerates every `scripts/generate_*.py` we ship |
|
|
8672
|
+
|
|
8673
|
+
---
|
|
8674
|
+
|
|
8675
|
+
## kb/procedures/sop-release.md
|
|
8676
|
+
|
|
8677
|
+
---
|
|
8678
|
+
title: "SOP: Release Preparation"
|
|
8679
|
+
category: procedures
|
|
8680
|
+
service: ai-toolkit
|
|
8681
|
+
tags: [sop, release, version, publish, changelog, semver, provenance, sarif, ecosystem, shellcheck]
|
|
8682
|
+
version: "1.15.0"
|
|
8683
|
+
created: "2026-04-10"
|
|
8684
|
+
last_updated: "2026-09-02"
|
|
8685
|
+
description: "Step-by-step checklist for preparing a new ai-toolkit release — ecosystem-sync drift check, version sync, changelog, artifact regeneration, validation, branch CI, and tagging. Run BEFORE every git tag. Includes mandatory Provenance, SARIF, checksum-pin, ShellCheck, licensing, exact-tag assertions, and a green Ubuntu/macOS branch-CI gate before any release tag is created."
|
|
8686
|
+
---
|
|
8687
|
+
|
|
8688
|
+
# SOP: Release Preparation
|
|
8689
|
+
|
|
8690
|
+
Complete checklist for preparing a new `@softspark/ai-toolkit` release.
|
|
8691
|
+
Run this **before** tagging. After tagging and publishing, run the
|
|
8692
|
+
[Release Verification SOP](sop-release-verification.md) to smoke-test.
|
|
8693
|
+
|
|
8694
|
+
**Pipeline:**
|
|
8695
|
+
```
|
|
8696
|
+
Ecosystem Sync SOP (drift check + generator updates)
|
|
8697
|
+
↓
|
|
8698
|
+
Release Preparation (this SOP)
|
|
8699
|
+
↓
|
|
8700
|
+
git tag → CI publish → Release Verification SOP
|
|
8701
|
+
```
|
|
8702
|
+
|
|
8703
|
+
**Time:** 10-20 minutes (includes ecosystem sync review)
|
|
8704
|
+
|
|
8705
|
+
---
|
|
8706
|
+
|
|
8707
|
+
## Quick Checklist (TL;DR)
|
|
8687
8708
|
|
|
8688
8709
|
```bash
|
|
8689
|
-
#
|
|
8710
|
+
# 0. Ecosystem sync (mandatory for minor/major releases; optional for patch)
|
|
8711
|
+
# Full procedure: kb/procedures/sop-ecosystem-sync.md
|
|
8712
|
+
python3 scripts/ecosystem_doctor.py --format text > /tmp/eco-report.txt
|
|
8713
|
+
cat /tmp/eco-report.txt
|
|
8714
|
+
# If drift detected: stop here, follow sop-ecosystem-sync.md Phase 2-4 to
|
|
8715
|
+
# classify each drift (A-F), update generators as needed, refresh snapshot,
|
|
8716
|
+
# THEN resume this SOP.
|
|
8717
|
+
python3 scripts/ecosystem_doctor.py --update # after all drift resolved
|
|
8718
|
+
|
|
8719
|
+
# 1. Decide version bump
|
|
8720
|
+
# patch (1.4.2 → 1.4.3): bugfix, typo, doc fix
|
|
8721
|
+
# minor (1.4.2 → 1.5.0): new feature, new skill, new flag, any ecosystem-class-B/F change
|
|
8722
|
+
# major (1.4.2 → 2.0.0): breaking change, any ecosystem-class-D removed path
|
|
8723
|
+
|
|
8724
|
+
# 2. Sync version across all files
|
|
8725
|
+
python3 scripts/sync_version.py X.Y.Z # if script exists, else manual
|
|
8726
|
+
|
|
8727
|
+
# 3. Write CHANGELOG.md entry
|
|
8728
|
+
# 4. Regenerate artifacts
|
|
8690
8729
|
python3 scripts/generate_agents_md.py > AGENTS.md
|
|
8691
8730
|
python3 scripts/generate_llms_txt.py > llms.txt
|
|
8692
8731
|
python3 scripts/generate_llms_txt.py --full > llms-full.txt
|
|
8693
8732
|
|
|
8694
|
-
#
|
|
8695
|
-
python3 scripts/validate.py --strict
|
|
8733
|
+
# 5. Validate + audit + SARIF + shellcheck + test + ecosystem check
|
|
8734
|
+
python3 scripts/validate.py --strict && python3 scripts/audit_skills.py --ci && python3 scripts/audit_skills.py --sarif > /tmp/audit.sarif && shellcheck --severity=warning app/hooks/*.sh && npm test
|
|
8696
8735
|
|
|
8697
|
-
#
|
|
8698
|
-
|
|
8736
|
+
# 5a. Supply-chain standard (v2.8.0+) — non-negotiable
|
|
8737
|
+
grep -q -- '--provenance' .github/workflows/publish.yml || { echo "MISSING --provenance"; exit 1; }
|
|
8738
|
+
grep -q 'id-token: write' .github/workflows/publish.yml || { echo "MISSING id-token: write"; exit 1; }
|
|
8739
|
+
python3 scripts/audit_skills.py --permissions # review Bash/Write/Edit footprint
|
|
8699
8740
|
|
|
8700
|
-
#
|
|
8701
|
-
|
|
8741
|
+
# 5b. Ecosystem gate — snapshot must be current before tag
|
|
8742
|
+
python3 scripts/ecosystem_doctor.py --offline --check || { echo "STALE ecosystem snapshot — re-run doctor"; exit 1; }
|
|
8702
8743
|
|
|
8703
|
-
#
|
|
8704
|
-
|
|
8705
|
-
|
|
8706
|
-
#
|
|
8707
|
-
git add
|
|
8708
|
-
git
|
|
8709
|
-
git
|
|
8744
|
+
# 5c. Licensing gate — SPDX headers, LICENSE, NOTICE, manifest consistency
|
|
8745
|
+
npx bats tests/test_licensing.bats || { echo "LICENSING GATE FAILED"; exit 1; }
|
|
8746
|
+
|
|
8747
|
+
# 6. Commit, push branch, and wait for the exact commit's full CI
|
|
8748
|
+
git add -A && git commit -m "chore: release vX.Y.Z"
|
|
8749
|
+
RELEASE_SHA=$(git rev-parse HEAD)
|
|
8750
|
+
git push origin main || { echo "FAIL: release commit push failed"; exit 1; }
|
|
8751
|
+
RUN_ID=""
|
|
8752
|
+
for ATTEMPT in 1 2 3 4 5; do
|
|
8753
|
+
RUN_ID=$(gh run list --workflow ci.yml --event push --commit "$RELEASE_SHA" \
|
|
8754
|
+
--limit 1 --json databaseId --jq '.[0].databaseId')
|
|
8755
|
+
[ -n "$RUN_ID" ] && break
|
|
8756
|
+
echo "Waiting for CI run registration ($ATTEMPT/5)" >&2
|
|
8757
|
+
[ "$ATTEMPT" -eq 5 ] || sleep 60
|
|
8758
|
+
done
|
|
8759
|
+
test -n "$RUN_ID" || { echo "FAIL: CI run not found for release commit"; exit 1; }
|
|
8760
|
+
gh run watch "$RUN_ID" --exit-status \
|
|
8761
|
+
|| { echo "FAIL: release commit CI failed"; exit 1; }
|
|
8762
|
+
for OS in ubuntu-latest macos-latest; do
|
|
8763
|
+
gh run view "$RUN_ID" --json jobs \
|
|
8764
|
+
--jq ".jobs[] | select(.name == \"Bats test suite ($OS)\") | .conclusion" \
|
|
8765
|
+
| grep -qx success || { echo "FAIL: $OS test job not green"; exit 1; }
|
|
8766
|
+
done
|
|
8767
|
+
test "$(git rev-parse HEAD)" = "$RELEASE_SHA" || { echo "FAIL: local HEAD changed during CI"; exit 1; }
|
|
8768
|
+
git fetch origin main || { echo "FAIL: origin/main refresh failed"; exit 1; }
|
|
8769
|
+
test "$(git rev-parse origin/main)" = "$RELEASE_SHA" || { echo "FAIL: origin/main changed during CI"; exit 1; }
|
|
8770
|
+
|
|
8771
|
+
# 6a. Only green branch CI authorizes tag creation
|
|
8772
|
+
git tag vX.Y.Z "$RELEASE_SHA"
|
|
8773
|
+
|
|
8774
|
+
# 6b. Assert the tag before pushing it (v4.19.0 postmortem, Phase 7)
|
|
8775
|
+
test "$(git rev-parse vX.Y.Z)" = "$RELEASE_SHA" || { echo "FAIL: tag not on tested release commit"; exit 1; }
|
|
8776
|
+
git show --no-patch --format=%s vX.Y.Z | grep -qx "chore: release vX.Y.Z" || { echo "FAIL: tag not on release commit"; exit 1; }
|
|
8777
|
+
|
|
8778
|
+
# 6c. Push the single tag by full ref. Never --tags.
|
|
8779
|
+
git push origin refs/tags/vX.Y.Z
|
|
8710
8780
|
```
|
|
8711
8781
|
|
|
8712
|
-
|
|
8713
|
-
|
|
8714
|
-
|
|
8782
|
+
---
|
|
8783
|
+
|
|
8784
|
+
## Phase 0: Ecosystem Sync (MANDATORY for minor/major)
|
|
8785
|
+
|
|
8786
|
+
Before touching version numbers, confirm the toolkit is aligned with the current state of every editor / platform it integrates with. Skipping this phase ships a release whose generators may lag a month-old CLI refactor, a rename of `.cursorrules` to `.cursor/rules/`, or a new hook event we do not yet emit.
|
|
8787
|
+
|
|
8788
|
+
**When this phase is mandatory:**
|
|
8789
|
+
- Minor release (X.Y.0) — always
|
|
8790
|
+
- Major release (X.0.0) — always
|
|
8791
|
+
- Patch release (X.Y.Z) — only if the patch touches a generator or install flow
|
|
8792
|
+
|
|
8793
|
+
**When to skip:** pure doc-only patches, SOP edits, internal refactors that do not touch `scripts/generate_*` or `app/skills/*/SKILL.md`.
|
|
8794
|
+
|
|
8795
|
+
### 0.1 Run the doctor
|
|
8715
8796
|
|
|
8716
|
-
**One-liner (copy-paste):**
|
|
8717
8797
|
```bash
|
|
8718
|
-
python3 scripts/
|
|
8798
|
+
python3 scripts/ecosystem_doctor.py --format text | tee /tmp/eco-report.txt
|
|
8719
8799
|
```
|
|
8720
8800
|
|
|
8721
|
-
|
|
8801
|
+
Output classifies every registered tool as **Clean**, **Drift**, or **Errored**.
|
|
8722
8802
|
|
|
8723
|
-
|
|
8803
|
+
### 0.2 Act on drift
|
|
8804
|
+
|
|
8805
|
+
For each drifting tool, follow [sop-ecosystem-sync.md](sop-ecosystem-sync.md) Phase 2-4:
|
|
8806
|
+
|
|
8807
|
+
| Drift class | Release impact |
|
|
8808
|
+
|-------------|----------------|
|
|
8809
|
+
| A (cosmetic reword) | No version impact — refresh snapshot, continue |
|
|
8810
|
+
| B (new feature — integrate) | **Minor** version bump at minimum; new generator or extended generator |
|
|
8811
|
+
| C (new feature — not adopted) | No impact — note in registry |
|
|
8812
|
+
| D (deprecation) | **Minor** or **major** depending on user impact; add migration warning |
|
|
8813
|
+
| E (feature promoted to default) | **Minor**; simplify generator, keep fallback comment |
|
|
8814
|
+
| F (feature newly globally available) | **Minor**; may require new generator or new config path |
|
|
8815
|
+
|
|
8816
|
+
If any B/D/E/F changes land in this preparation pass, mention them explicitly in the CHANGELOG entry (Phase 3) under a `Ecosystem` subsection.
|
|
8817
|
+
|
|
8818
|
+
### 0.3 Refresh snapshot
|
|
8819
|
+
|
|
8820
|
+
Once every drift is resolved (either by code change or by re-classifying as acceptable):
|
|
8724
8821
|
|
|
8725
8822
|
```bash
|
|
8726
|
-
|
|
8727
|
-
ai-toolkit --help
|
|
8728
|
-
which ai-toolkit
|
|
8823
|
+
python3 scripts/ecosystem_doctor.py --update
|
|
8729
8824
|
```
|
|
8730
8825
|
|
|
8731
|
-
|
|
8732
|
-
|
|
8733
|
-
|
|
8734
|
-
|
|
8826
|
+
This writes the new baseline to `benchmarks/ecosystem-doctor-snapshot.json`. Commit it as part of the release commit.
|
|
8827
|
+
|
|
8828
|
+
### 0.4 Gate
|
|
8829
|
+
|
|
8830
|
+
```bash
|
|
8831
|
+
python3 scripts/ecosystem_doctor.py --offline --check
|
|
8832
|
+
```
|
|
8833
|
+
|
|
8834
|
+
Must exit `0`. If it exits `1`, the snapshot is stale — rerun Phase 0.3 or review the remaining drift.
|
|
8735
8835
|
|
|
8736
8836
|
---
|
|
8737
8837
|
|
|
8738
|
-
## Phase
|
|
8838
|
+
## Phase 1: Determine Version Bump
|
|
8839
|
+
|
|
8840
|
+
Follow [Semantic Versioning](https://semver.org/):
|
|
8841
|
+
|
|
8842
|
+
| Change Type | Bump | Examples |
|
|
8843
|
+
|-------------|------|---------|
|
|
8844
|
+
| Bugfix, typo, doc-only | **patch** | Fix install flag, correct description |
|
|
8845
|
+
| New feature, skill, agent, flag | **minor** | Add `/hipaa-validate`, add `--output json` |
|
|
8846
|
+
| Breaking CLI change, removed skill, config format change | **major** | Rename `install` to `setup`, remove skill |
|
|
8847
|
+
|
|
8848
|
+
**Rule:** When in doubt, bump minor.
|
|
8849
|
+
|
|
8850
|
+
---
|
|
8851
|
+
|
|
8852
|
+
## Phase 2: Sync Version in All Files
|
|
8853
|
+
|
|
8854
|
+
The canonical version lives in `package.json`. These files **must** match:
|
|
8855
|
+
|
|
8856
|
+
### Mandatory sync (every release)
|
|
8857
|
+
|
|
8858
|
+
| File | Field | How to update |
|
|
8859
|
+
|------|-------|---------------|
|
|
8860
|
+
| `package.json` | `"version": "X.Y.Z"` | Edit directly |
|
|
8861
|
+
| `manifest.json` | `"version": "X.Y.Z"` | Edit directly |
|
|
8862
|
+
| `app/.claude-plugin/plugin.json` | `"version": "X.Y.Z"` | Edit directly |
|
|
8863
|
+
|
|
8864
|
+
### Auto-synced (no manual action)
|
|
8865
|
+
|
|
8866
|
+
| File | Mechanism |
|
|
8867
|
+
|------|-----------|
|
|
8868
|
+
| `package-lock.json` | Regenerated by `npm install --package-lock-only` |
|
|
8869
|
+
|
|
8870
|
+
### Conditional sync (only if the doc was modified in this release)
|
|
8871
|
+
|
|
8872
|
+
| File | Field | When to update |
|
|
8873
|
+
|------|-------|---------------|
|
|
8874
|
+
| `kb/procedures/sop-maintenance.md` | frontmatter `version:` | If SOP content changed |
|
|
8875
|
+
| `kb/reference/skills-catalog.md` | frontmatter `version:` | If skills added/removed |
|
|
8876
|
+
| `kb/reference/agents-catalog.md` | frontmatter `version:` | If agents added/removed |
|
|
8877
|
+
| `kb/reference/hooks-catalog.md` | frontmatter `version:` | If hooks changed |
|
|
8878
|
+
| `kb/reference/architecture-overview.md` | frontmatter `version:` | If architecture changed |
|
|
8879
|
+
| `kb/reference/distribution-model.md` | frontmatter `version:` | If install model changed |
|
|
8880
|
+
| `kb/reference/global-install-model.md` | frontmatter `version:` | If install model changed |
|
|
8881
|
+
|
|
8882
|
+
> **Note:** KB `version:` fields track the **document version**, not the toolkit version.
|
|
8883
|
+
> Only bump them when the document content actually changes in this release.
|
|
8884
|
+
|
|
8885
|
+
### Count sync (if skills/agents/hooks changed)
|
|
8886
|
+
|
|
8887
|
+
| File | What to check |
|
|
8888
|
+
|------|---------------|
|
|
8889
|
+
| `package.json` | `"description"` — skill/agent count |
|
|
8890
|
+
| `README.md` | Badge counts, "What You Get" table |
|
|
8891
|
+
| `app/ARCHITECTURE.md` | Section headings with counts |
|
|
8892
|
+
|
|
8893
|
+
> **Tip:** `validate.py --strict` catches count drift AND version mismatches
|
|
8894
|
+
> (package.json vs manifest.json vs plugin.json) automatically.
|
|
8895
|
+
> If validation passes, counts and versions are correct.
|
|
8896
|
+
|
|
8897
|
+
### Verification command
|
|
8898
|
+
|
|
8899
|
+
After syncing, verify all mandatory files match:
|
|
8739
8900
|
|
|
8740
8901
|
```bash
|
|
8741
|
-
|
|
8742
|
-
|
|
8902
|
+
VERSION=$(python3 -c "import json; print(json.load(open('package.json'))['version'])")
|
|
8903
|
+
echo "Target: $VERSION"
|
|
8904
|
+
echo "manifest.json: $(python3 -c "import json; print(json.load(open('manifest.json'))['version'])")"
|
|
8905
|
+
echo "plugin.json: $(python3 -c "import json; print(json.load(open('app/.claude-plugin/plugin.json'))['version'])")"
|
|
8906
|
+
echo "package-lock.json: $(python3 -c "import json; print(json.load(open('package-lock.json'))['version'])")"
|
|
8743
8907
|
```
|
|
8744
8908
|
|
|
8745
|
-
|
|
8746
|
-
- [ ] Agents: 44
|
|
8747
|
-
- [ ] Skills: 108
|
|
8748
|
-
- [ ] Hooks merged into settings.json
|
|
8749
|
-
- [ ] "Other AI Tools" lists documented global targets (with `--editors`): aider, antigravity, augment, cline, codex, copilot, cursor, gemini, opencode, roo, windsurf. Scope varies: Codex uses `$CODEX_HOME` (default `~/.codex`) plus `$HOME/.agents/skills`; Copilot uses `$COPILOT_HOME` (default `~/.copilot`); cursor has only `~/.cursor/hooks.json`; antigravity has the `~/.gemini/*/skills` pointer. Cursor and Antigravity rules remain project-only.
|
|
8909
|
+
All four must print the same version. If not, fix before proceeding.
|
|
8750
8910
|
|
|
8751
|
-
|
|
8752
|
-
|
|
8753
|
-
|
|
8754
|
-
|
|
8755
|
-
|
|
8911
|
+
### Public surface review
|
|
8912
|
+
|
|
8913
|
+
```bash
|
|
8914
|
+
python3 scripts/surface_manifest.py --update
|
|
8915
|
+
git diff app/surface.json
|
|
8916
|
+
```
|
|
8917
|
+
|
|
8918
|
+
**Every line the diff removes is a breaking change.** Restore it, or take the
|
|
8919
|
+
deprecation path in `BACKWARD_COMPATIBILITY.md` and add a `DECISIONS.md` entry
|
|
8920
|
+
before the tag. Lines added are new surface being adopted into protection — that
|
|
8921
|
+
needs no ceremony.
|
|
8922
|
+
|
|
8923
|
+
Do not run `--update` to make a red build green. The check fails because something
|
|
8924
|
+
users depend on disappeared; regenerating the manifest deletes the evidence, not
|
|
8925
|
+
the problem.
|
|
8926
|
+
|
|
8927
|
+
### Skill body budget ratchet
|
|
8928
|
+
|
|
8929
|
+
`validate.py` prints the largest skill body on every run:
|
|
8930
|
+
|
|
8931
|
+
```
|
|
8932
|
+
Body budget: largest is <skill> at <N> bytes (warn 18000, error 20000)
|
|
8933
|
+
```
|
|
8934
|
+
|
|
8935
|
+
Once that number sits at least 2000 bytes under `SKILL_BODY_BUDGET_WARN`, lower
|
|
8936
|
+
`SKILL_BODY_BUDGET_WARN` by 2000 in `scripts/validate.py` and ship the tightened
|
|
8937
|
+
threshold with the release. Target floor is 12000.
|
|
8756
8938
|
|
|
8757
|
-
|
|
8939
|
+
Two rules, both learned the hard way:
|
|
8758
8940
|
|
|
8759
|
-
|
|
8941
|
+
- **Never lower a threshold in the same change that something violates it.** Split
|
|
8942
|
+
the offending skill into `SKILL.md` + `reference/` first, prove it with
|
|
8943
|
+
`python3 scripts/check_split.py <skill> --before <pre-split SKILL.md>`, then
|
|
8944
|
+
tighten in a follow-up.
|
|
8945
|
+
- **Never raise a threshold to make a red build green.** A body over budget means
|
|
8946
|
+
detail that belongs in `reference/` is loading on every trigger match, including
|
|
8947
|
+
the accidental ones. Raising the number hides the cost, it does not remove it.
|
|
8760
8948
|
|
|
8761
|
-
|
|
8762
|
-
|
|
8763
|
-
|
|
8949
|
+
If the largest body has not moved since the previous release, leave the threshold
|
|
8950
|
+
alone and say so in the release notes. A ratchet that never advances is honest;
|
|
8951
|
+
one that advances past reality is not.
|
|
8764
8952
|
|
|
8765
|
-
|
|
8766
|
-
- Environment: node, bash, python3, bats
|
|
8767
|
-
- Global Install: .claude exists, agents/skills symlinks (0 broken), settings.json hooks
|
|
8768
|
-
- Hook Scripts: all present and executable
|
|
8769
|
-
- Hook Configuration: 14 events registered
|
|
8770
|
-
- Generated Artifacts: AGENTS.md, llms.txt, llms-full.txt
|
|
8771
|
-
- Planned Assets: plugin.json, benchmarks, plugin packs
|
|
8772
|
-
- Benchmark Freshness: < 30 days
|
|
8773
|
-
- Stale Rules: all healthy
|
|
8953
|
+
---
|
|
8774
8954
|
|
|
8775
|
-
|
|
8776
|
-
- [ ] `Errors: 0 | Warnings: 0`
|
|
8777
|
-
- [ ] `HEALTH CHECK PASSED`
|
|
8955
|
+
## Phase 3: Write CHANGELOG Entry
|
|
8778
8956
|
|
|
8779
|
-
|
|
8780
|
-
(broken symlinks, non-executable hooks, missing scripts, missing llms-full.txt).
|
|
8957
|
+
Add entry at the top of `CHANGELOG.md` (after the header, before previous release):
|
|
8781
8958
|
|
|
8782
|
-
|
|
8959
|
+
```markdown
|
|
8960
|
+
## vX.Y.Z — Short Title (YYYY-MM-DD)
|
|
8783
8961
|
|
|
8784
|
-
|
|
8962
|
+
### Added
|
|
8963
|
+
- **Feature name** — description
|
|
8785
8964
|
|
|
8786
|
-
|
|
8787
|
-
|
|
8788
|
-
git init -q
|
|
8789
|
-
ai-toolkit install --local --editors all --dry-run
|
|
8790
|
-
cd - && rm -rf /tmp/ai-toolkit-verify
|
|
8791
|
-
```
|
|
8965
|
+
### Changed
|
|
8966
|
+
- **What changed** — old behavior → new behavior
|
|
8792
8967
|
|
|
8793
|
-
|
|
8794
|
-
-
|
|
8795
|
-
- [ ] Would create: .claude/settings.local.json
|
|
8796
|
-
- [ ] Would inject: .claude/constitution.md
|
|
8797
|
-
- [ ] Editors: all 11 listed (copilot, cursor, windsurf, cline, roo, aider, augment, antigravity, codex, gemini, opencode)
|
|
8798
|
-
- [ ] Would generate configs for each editor (legacy + directory-based)
|
|
8799
|
-
- [ ] Would install: .git/hooks/pre-commit
|
|
8800
|
-
- [ ] Would inject language rules (auto-detected)
|
|
8968
|
+
### Fixed
|
|
8969
|
+
- **Bug description** — what was broken and how it's fixed
|
|
8801
8970
|
|
|
8802
|
-
|
|
8803
|
-
|
|
8804
|
-
ai-toolkit install --local --dry-run
|
|
8805
|
-
# → Editors: none (empty project has no existing configs)
|
|
8971
|
+
### Removed
|
|
8972
|
+
- **What was removed** — migration path if any
|
|
8806
8973
|
```
|
|
8807
8974
|
|
|
8808
|
-
|
|
8975
|
+
**Rules:**
|
|
8976
|
+
- Use **bold** for feature names
|
|
8977
|
+
- Start descriptions with a verb (Added, Changed, Fixed, Removed)
|
|
8978
|
+
- Reference skill names with backticks and slash: `/hipaa-validate`
|
|
8979
|
+
- Include script names: `scripts/hipaa_scan.py`
|
|
8980
|
+
- Include count changes: `Skill count: 91 → 92`
|
|
8981
|
+
- Date format: `YYYY-MM-DD`
|
|
8982
|
+
- Title: short, descriptive, no version number repetition
|
|
8809
8983
|
|
|
8810
|
-
|
|
8984
|
+
### Update README "What's New" section
|
|
8811
8985
|
|
|
8812
|
-
|
|
8813
|
-
python3 scripts/validate.py --strict
|
|
8814
|
-
python3 scripts/audit_skills.py --ci
|
|
8815
|
-
```
|
|
8986
|
+
**MANDATORY on every release.** Update the `## What's New in vX.Y.Z` section in `README.md`:
|
|
8816
8987
|
|
|
8817
|
-
|
|
8818
|
-
|
|
8819
|
-
|
|
8820
|
-
|
|
8821
|
-
- [ ] `Errors: 0 | Warnings: 0` → `VALIDATION PASSED`
|
|
8988
|
+
1. Change the heading version: `## What's New in vX.Y.Z`
|
|
8989
|
+
2. Replace bullet points with 3-5 highlights from this release
|
|
8990
|
+
3. **Keep only the latest version block.** Delete the previous `## What's New in vA.B.C` section(s). README is the shop window, not the archive — users see the current release, full history lives in `CHANGELOG.md`.
|
|
8991
|
+
4. Keep the `See [CHANGELOG.md](CHANGELOG.md) for full history.` link directly below the bullet list.
|
|
8822
8992
|
|
|
8823
|
-
**
|
|
8824
|
-
|
|
8825
|
-
|
|
8826
|
-
- [ ] `INFO: N` (acceptable — broad-access skills: orchestrate, swarm, workflow)
|
|
8993
|
+
> **Warning:** This section is the first thing users see after the badges.
|
|
8994
|
+
> A stale version here (e.g., "What's New in v2.1.3" when shipping v2.3.0)
|
|
8995
|
+
> signals an unmaintained project. Do NOT skip this step.
|
|
8827
8996
|
|
|
8828
|
-
|
|
8997
|
+
> **Single-version rule:** README.md must contain **exactly one** `## What's New in vX.Y.Z` heading at any time. If you find multiple stacked (e.g. v2.6.1 + v2.6.0 + v2.5.0), that is a SOP drift — collapse to the latest on the next release commit.
|
|
8829
8998
|
|
|
8830
|
-
|
|
8999
|
+
---
|
|
8831
9000
|
|
|
8832
|
-
|
|
8833
|
-
# Run ONCE, capture to file, then parse. Full suite is 669+ bats cases —
|
|
8834
|
-
# re-running it per check (tail / grep ok / grep not ok piped separately)
|
|
8835
|
-
# wastes minutes every release. Always cache the output.
|
|
8836
|
-
npm test > /tmp/npm-test.log 2>&1
|
|
8837
|
-
exit=$?
|
|
8838
|
-
tail -3 /tmp/npm-test.log
|
|
8839
|
-
echo "ok: $(grep -c '^ok ' /tmp/npm-test.log)"
|
|
8840
|
-
echo "not ok: $(grep -c '^not ok' /tmp/npm-test.log)"
|
|
8841
|
-
echo "exit: $exit"
|
|
8842
|
-
```
|
|
9001
|
+
## Phase 4: Regenerate Artifacts
|
|
8843
9002
|
|
|
8844
|
-
|
|
8845
|
-
- [ ] `exit == 0`
|
|
8846
|
-
- [ ] `ok == expected test count` (e.g., 945 on v3.0.0)
|
|
8847
|
-
- [ ] `not ok == 0`
|
|
8848
|
-
- [ ] Bats runs tests in parallel (4 jobs)
|
|
8849
|
-
- [ ] Groups: agents, autodetect, cli, generators, guards, hooks, inject,
|
|
8850
|
-
install, kb, mcp, readme, profiles, uninstall, validate
|
|
9003
|
+
Use the npm scripts, not the generators directly:
|
|
8851
9004
|
|
|
8852
|
-
**Anti-pattern — do NOT do this:**
|
|
8853
9005
|
```bash
|
|
8854
|
-
|
|
8855
|
-
npm
|
|
8856
|
-
npm test 2>&1 | grep -c '^ok '
|
|
8857
|
-
npm test 2>&1 | grep -c '^not ok'
|
|
9006
|
+
npm run generate:agents # AI_TOOLKIT_NO_CUSTOM_RULES=1 python3 scripts/generate_agents_md.py > AGENTS.md
|
|
9007
|
+
npm run generate:llms # llms.txt + llms-full.txt
|
|
8858
9008
|
```
|
|
8859
9009
|
|
|
8860
|
-
|
|
8861
|
-
|
|
8862
|
-
-
|
|
8863
|
-
- Eject: real files (not symlinks), inlined rules
|
|
8864
|
-
- Uninstall: removes toolkit, preserves user content
|
|
8865
|
-
|
|
8866
|
-
---
|
|
9010
|
+
`generate:agents` sets `AI_TOOLKIT_NO_CUSTOM_RULES=1`. Running
|
|
9011
|
+
`generate_agents_md.py` bare picks up whatever is registered in the maintainer's
|
|
9012
|
+
own `~/.softspark/ai-toolkit/rules/`, which then ships inside `AGENTS.md`.
|
|
8867
9013
|
|
|
8868
|
-
|
|
9014
|
+
Check if anything actually changed:
|
|
8869
9015
|
|
|
8870
9016
|
```bash
|
|
8871
|
-
|
|
8872
|
-
cd /tmp/ai-toolkit-eject-test
|
|
8873
|
-
ai-toolkit eject
|
|
8874
|
-
cd - && rm -rf /tmp/ai-toolkit-eject-test
|
|
9017
|
+
git diff --stat AGENTS.md llms.txt llms-full.txt
|
|
8875
9018
|
```
|
|
8876
9019
|
|
|
8877
|
-
|
|
8878
|
-
- [ ] Agents copied as real files (not symlinks)
|
|
8879
|
-
- [ ] Skills copied as real directories
|
|
8880
|
-
- [ ] Rules inlined into CLAUDE.md
|
|
8881
|
-
- [ ] constitution.md and ARCHITECTURE.md copied
|
|
8882
|
-
- [ ] `output-styles/` directory present (v2.7.1+)
|
|
9020
|
+
If no diff, the artifacts are already current. If there is a diff, stage them.
|
|
8883
9021
|
|
|
8884
9022
|
---
|
|
8885
9023
|
|
|
8886
|
-
## Phase
|
|
8887
|
-
|
|
8888
|
-
These checks enforce the v2.8.0 security standard on a freshly-published release. Run AFTER the `publish.yml` workflow completes on the tag.
|
|
9024
|
+
## Phase 5: Validate, Audit, Test
|
|
8889
9025
|
|
|
8890
|
-
|
|
9026
|
+
Run the full quality gate:
|
|
8891
9027
|
|
|
8892
9028
|
```bash
|
|
8893
|
-
|
|
8894
|
-
|
|
8895
|
-
|
|
8896
|
-
|
|
9029
|
+
python3 scripts/validate.py --strict
|
|
9030
|
+
python3 scripts/audit_skills.py --ci
|
|
9031
|
+
python3 scripts/audit_skills.py --sarif > audit.sarif # MANDATORY — GHAS ingest
|
|
9032
|
+
python3 scripts/audit_skills.py --permissions # review Bash/Write/Edit footprint
|
|
8897
9033
|
|
|
8898
|
-
|
|
8899
|
-
|
|
8900
|
-
|
|
9034
|
+
# ShellCheck on hooks (added in 1.11.0). Mirrors the ci.yml "ShellCheck hooks"
|
|
9035
|
+
# job. NOT run by validate.py, npm test, OR publish.yml — so a hook with a
|
|
9036
|
+
# ShellCheck warning passes every other gate AND still publishes on tag while
|
|
9037
|
+
# turning main CI red. Run it here, before tagging.
|
|
9038
|
+
shellcheck --severity=warning app/hooks/*.sh && echo "OK: shellcheck clean"
|
|
8901
9039
|
|
|
8902
|
-
|
|
9040
|
+
# Registry / generator drift (added in 1.10.0). Meta-generators excluded.
|
|
9041
|
+
META="generate_agents_md.py|generate_llms_txt.py|generate_language_rules_skills.py"
|
|
9042
|
+
diff \
|
|
9043
|
+
<(grep -oE 'scripts/generate_[a-z_]+\.py' kb/reference/supported-tools-registry.md | sort -u) \
|
|
9044
|
+
<(ls scripts/generate_*.py | grep -vE "$META" | sort -u) \
|
|
9045
|
+
&& echo "OK: registry matches filesystem" \
|
|
9046
|
+
|| { echo "DRIFT: update supported-tools-registry.md before tagging"; exit 1; }
|
|
8903
9047
|
|
|
8904
|
-
|
|
9048
|
+
# Stage first IF this release adds or deletes a kb/ file. The test
|
|
9049
|
+
# "npm package KB files match the tracked release set" compares `git ls-files
|
|
9050
|
+
# kb` (the index) against what `npm pack` sees (the working tree), so an
|
|
9051
|
+
# unstaged addition reads as "extra" and an unstaged deletion as "missing".
|
|
9052
|
+
# Phase 6 stages, and it runs after this one, so the ordering fails the test
|
|
9053
|
+
# for any release that touches the KB. Staging early costs nothing.
|
|
9054
|
+
git status --porcelain kb/ | grep -qE '^(\?\?| D|\?M)' && git add -A kb/
|
|
8905
9055
|
|
|
8906
|
-
|
|
8907
|
-
|
|
8908
|
-
|
|
9056
|
+
# Run npm test ONCE, cache output, parse from file. The suite is 1400+ bats
|
|
9057
|
+
# cases — rerunning it per check wastes minutes. Do not pipe npm test into
|
|
9058
|
+
# tail/grep multiple times in the same session.
|
|
9059
|
+
npm test > /tmp/npm-test.log 2>&1
|
|
9060
|
+
tail -3 /tmp/npm-test.log
|
|
9061
|
+
echo "ok: $(grep -c '^ok ' /tmp/npm-test.log) | not ok: $(grep -c '^not ok' /tmp/npm-test.log)"
|
|
8909
9062
|
```
|
|
8910
9063
|
|
|
8911
|
-
**
|
|
8912
|
-
-
|
|
8913
|
-
-
|
|
9064
|
+
**Expected results:**
|
|
9065
|
+
- `validate.py`: `Errors: 0 | Warnings: 0 | VALIDATION PASSED`
|
|
9066
|
+
- `audit_skills.py --ci`: `HIGH: 0 | WARN: 0` (INFO is acceptable)
|
|
9067
|
+
- `audit_skills.py --sarif`: valid JSON, non-empty `runs[0].tool.driver.rules`
|
|
9068
|
+
- `audit_skills.py --permissions`: review `Skills with Bash + Write + Edit` list — any newly-added skill with broad access MUST be justified in the CHANGELOG entry
|
|
9069
|
+
- `shellcheck --severity=warning app/hooks/*.sh`: no output, exit 0. A common false positive is `SC2034` on `INPUT` or env vars (e.g. `AI_TOOLKIT_HOOK_FORMAT`) that a *sourced* helper (`_hook-io.sh`) consumes — ShellCheck cannot see cross-file use. Fix with a `# shellcheck disable=SC2034` directive or `export`, matching `guard-destructive.sh`. Never tag with a red ShellCheck.
|
|
9070
|
+
- Registry drift: `OK: registry matches filesystem`. If `DRIFT:` appears, add the missing `scripts/generate_*.py` rows to `kb/reference/supported-tools-registry.md` before tagging.
|
|
9071
|
+
- `npm test`: `1..N` with zero `not ok` (read from the cached `/tmp/npm-test.log`, do not rerun)
|
|
8914
9072
|
|
|
8915
|
-
|
|
9073
|
+
> **Why this matters (v4.5.1 postmortem):** `publish.yml` runs only `validate.py` + `npm test`, so it published v4.5.0 even though the `main` CI `ShellCheck hooks` job was red on two `SC2034` warnings in a new hook. The publish workflow does **not** depend on the CI workflow. Until that is fixed, ShellCheck is a manual pre-tag gate — run it here every time.
|
|
8916
9074
|
|
|
9075
|
+
**One-liner:**
|
|
8917
9076
|
```bash
|
|
8918
|
-
python3 scripts/audit_skills.py --
|
|
9077
|
+
python3 scripts/validate.py --strict && python3 scripts/audit_skills.py --ci && python3 scripts/audit_skills.py --sarif > audit.sarif && shellcheck --severity=warning app/hooks/*.sh && diff <(grep -oE 'scripts/generate_[a-z_]+\.py' kb/reference/supported-tools-registry.md | sort -u) <(ls scripts/generate_*.py | grep -vE 'generate_agents_md\.py|generate_llms_txt\.py|generate_language_rules_skills\.py' | sort -u) && npm test
|
|
8919
9078
|
```
|
|
8920
9079
|
|
|
8921
|
-
**
|
|
8922
|
-
-
|
|
8923
|
-
-
|
|
8924
|
-
-
|
|
8925
|
-
|
|
8926
|
-
### 8.4 URL-sourced rules/hooks are checksum-pinned
|
|
8927
|
-
|
|
8928
|
-
`sources.json` is an envelope of the form `{"schema_version": 1, "rules"|"hooks": {...}}`, so the jq filter must pick the nested map before piping into the pin assertion. The URL-count in the success message ignores local `path:`-only entries — only entries with a `url:` field are pinned and counted.
|
|
8929
|
-
|
|
8930
|
-
```bash
|
|
8931
|
-
jq '.rules // .hooks // {}' ~/.softspark/ai-toolkit/rules/sources.json 2>/dev/null \
|
|
8932
|
-
| python3 -c "import json, sys; d=json.load(sys.stdin) or {}; bad=[n for n,v in d.items() if v.get('url') and not v.get('sha256')]; assert not bad, f'UNPINNED: {bad}'; url_n=sum(1 for v in d.values() if v.get('url')); print(f'RULE PIN OK: {url_n} URL rules, all with sha256')"
|
|
8933
|
-
jq '.hooks // {}' ~/.softspark/ai-toolkit/hooks/external/sources.json 2>/dev/null \
|
|
8934
|
-
| python3 -c "import json, sys; d=json.load(sys.stdin) or {}; bad=[n for n,v in d.items() if v.get('url') and not v.get('sha256')]; assert not bad, f'UNPINNED: {bad}'; url_n=sum(1 for v in d.values() if v.get('url')); print(f'HOOK PIN OK: {url_n} URL hooks, all with sha256')"
|
|
8935
|
-
```
|
|
9080
|
+
**If tests fail:** Fix the issue, do NOT skip. Common failures:
|
|
9081
|
+
- Stale counts → re-run `generate:all` or fix README/ARCHITECTURE
|
|
9082
|
+
- Missing frontmatter → add to new KB docs
|
|
9083
|
+
- Broken symlink → `ai-toolkit doctor --fix`
|
|
8936
9084
|
|
|
8937
|
-
|
|
8938
|
-
- [ ] Both commands print `... PIN OK`
|
|
8939
|
-
- [ ] If any entry is unpinned, the `register_url_source()` call missed passing `content=` — fix the call-site and retag
|
|
9085
|
+
### Phase 5c: Licensing Gate (v4.20.0+)
|
|
8940
9086
|
|
|
8941
|
-
|
|
9087
|
+
The project is Apache-2.0. Attribution only works if the artefact actually
|
|
9088
|
+
carries it, and every part of that is mechanically checkable.
|
|
8942
9089
|
|
|
8943
9090
|
```bash
|
|
8944
|
-
|
|
9091
|
+
# The whole gate, enforced in CI. Run it here so a failure is caught before tagging.
|
|
9092
|
+
npx bats tests/test_licensing.bats
|
|
8945
9093
|
```
|
|
8946
9094
|
|
|
8947
|
-
|
|
8948
|
-
- [ ] Exit 0, no `CHECKSUM CHANGED` line
|
|
8949
|
-
- [ ] If a checksum change was intentional (e.g. upstream rule update), document it in the CHANGELOG entry before tagging
|
|
9095
|
+
The seven assertions, and why each exists:
|
|
8950
9096
|
|
|
8951
|
-
|
|
9097
|
+
| Check | Fails when |
|
|
9098
|
+
|---|---|
|
|
9099
|
+
| Every shipped source file carries an SPDX header | A new `.py`/`.sh`/`.js`/`.bats` file was added without one — the common case, and the reason this is a test rather than a habit |
|
|
9100
|
+
| Headers name Apache-2.0 and nothing else | A file was copied in from an MIT/GPL source with its own header intact |
|
|
9101
|
+
| **No** markdown file carries an SPDX header | Someone "helpfully" ran the header script over `app/skills/` — headers there sit above parsed frontmatter and bill every session for it |
|
|
9102
|
+
| `LICENSE` is the complete Apache 2.0 text | The file was truncated or replaced with a summary |
|
|
9103
|
+
| `NOTICE` carries attribution, the source URL, §4(d) and the MIT-era notice | The attribution mechanism was gutted |
|
|
9104
|
+
| `LICENSE` **and** `NOTICE` ship in the npm package | `package.json` `files` lost an entry — a NOTICE that never reaches the consumer cannot satisfy §4(d) |
|
|
9105
|
+
| Every manifest declaring a licence declares Apache-2.0 | `package.json`, `manifest.json`, `plugin.json` and `package-lock.json` drifted apart |
|
|
8952
9106
|
|
|
8953
|
-
|
|
9107
|
+
**Adding source files in this release?** The header goes *after* the shebang,
|
|
9108
|
+
never before it. Short SPDX form:
|
|
8954
9109
|
|
|
8955
|
-
|
|
9110
|
+
```
|
|
9111
|
+
# SPDX-License-Identifier: Apache-2.0
|
|
9112
|
+
# Copyright 2024-2026 Lukasz Krzemien (biuro@softspark.eu)
|
|
9113
|
+
# Source: https://github.com/softspark/ai-toolkit
|
|
9114
|
+
```
|
|
8956
9115
|
|
|
8957
|
-
|
|
9116
|
+
`//` for JavaScript. Full convention and the reasoning behind the markdown
|
|
9117
|
+
exclusion: [Licensing](../reference/licensing.md).
|
|
8958
9118
|
|
|
8959
|
-
|
|
9119
|
+
**Changing the licence itself?** Do not hand-type the licence text. Take it
|
|
9120
|
+
verbatim from a published copy and cross-verify against a second independent
|
|
9121
|
+
copy before writing `LICENSE` — a rendered or summarised licence is not the
|
|
9122
|
+
licence. Prior releases stay under their original terms; a licence change
|
|
9123
|
+
applies going forward and revokes nothing already granted.
|
|
8960
9124
|
|
|
8961
|
-
|
|
8962
|
-
D=/tmp/aitk-profile-full-${RANDOM} && mkdir -p "$D" && cd "$D" && git init -q
|
|
8963
|
-
ai-toolkit install --local --editors cursor,windsurf,gemini,augment,codex,copilot \
|
|
8964
|
-
--profile full --dry-run 2>&1 \
|
|
8965
|
-
| grep -E "\\.cursor/(hooks\\.json|agents)|\\.devin/hooks\\.v1\\.json|\\.gemini/(settings\\.json|commands)|\\.augment/(agents|commands)|\\.codex/(agents|hooks)|\\.agents/skills|\\.github/(agents|skills|instructions|prompts|hooks)"
|
|
8966
|
-
```
|
|
9125
|
+
### Phase 5a: Supply-Chain Hardening Verification (v2.8.0+)
|
|
8967
9126
|
|
|
8968
|
-
|
|
8969
|
-
- [ ] `.cursor/hooks.json` and `.cursor/agents/`
|
|
8970
|
-
- [ ] `.devin/hooks.v1.json`
|
|
8971
|
-
- [ ] `.gemini/settings.json` hooks AND `.gemini/commands/`
|
|
8972
|
-
- [ ] `.augment/agents/` + `.augment/commands/` + `$HOME/.augment/settings.json`
|
|
8973
|
-
- [ ] `.codex/hooks.json` + `.codex/hooks/` + `.codex/agents/`
|
|
8974
|
-
- [ ] `.agents/skills/` (Codex native discovery path)
|
|
8975
|
-
- [ ] `.github/agents/` + `.github/skills/` + `.github/instructions/` + `.github/prompts/` + `.github/hooks/`
|
|
9127
|
+
These checks enforce the security standard introduced in v2.8.0. Do NOT tag a release until all pass.
|
|
8976
9128
|
|
|
8977
|
-
|
|
9129
|
+
**1. Publish workflow emits provenance:**
|
|
8978
9130
|
|
|
8979
9131
|
```bash
|
|
8980
|
-
|
|
8981
|
-
ai-toolkit install --local --editors codex --profile minimal --dry-run 2>&1 \
|
|
8982
|
-
| grep -q "Would generate: .agents/skills" && echo "OK: Codex skills emit at minimal"
|
|
8983
|
-
ai-toolkit install --local --editors codex --profile standard --codex-skills --dry-run 2>&1 \
|
|
8984
|
-
| grep -q "Would refresh: .agents/skills" && echo "OK: legacy flag explicitly refreshes the same catalog"
|
|
9132
|
+
grep -E '\-\-provenance|id-token: write' .github/workflows/publish.yml
|
|
8985
9133
|
```
|
|
8986
9134
|
|
|
8987
|
-
|
|
8988
|
-
- [ ]
|
|
8989
|
-
- [ ] `--codex-skills` remains an explicit refresh alias; it is not required for installation
|
|
8990
|
-
- [ ] No profile emits `.codex/skills/`
|
|
8991
|
-
|
|
8992
|
-
### 9.3 Breaking-change surfaces land on `--profile standard`
|
|
9135
|
+
- [ ] Both markers present (`--provenance` flag + `id-token: write` permission)
|
|
9136
|
+
- [ ] Any PR that changes `publish.yml` REQUIRES an approved security review
|
|
8993
9137
|
|
|
8994
|
-
|
|
8995
|
-
- Copilot scoped instructions, prompts, custom agents, portable skills, and hooks
|
|
8996
|
-
- Gemini hooks (`.gemini/settings.json`)
|
|
9138
|
+
**2. URL-sourced rules and hooks are checksum-pinned:**
|
|
8997
9139
|
|
|
8998
9140
|
```bash
|
|
8999
|
-
|
|
9000
|
-
|
|
9001
|
-
|
|
9002
|
-
|
|
9003
|
-
grep -q "\\.github/agents/.*\\.github/skills/" /tmp/aitk-breaking.log && echo "OK: Copilot agents and skills at standard"
|
|
9004
|
-
grep -q "\\.github/hooks/" /tmp/aitk-breaking.log && echo "OK: Copilot hooks at standard"
|
|
9005
|
-
grep -q "\\.gemini/settings\\.json hooks" /tmp/aitk-breaking.log && echo "OK: Gemini hooks at standard"
|
|
9141
|
+
# On a machine that has consumed URL rules/hooks at least once
|
|
9142
|
+
# (schema_version 1: entries live under the .rules / .hooks key):
|
|
9143
|
+
jq '.rules | to_entries | map(select(.value.url != null and (.value.sha256 // "" | length) == 0))' ~/.softspark/ai-toolkit/rules/sources.json
|
|
9144
|
+
jq '.hooks | to_entries | map(select(.value.url != null and (.value.sha256 // "" | length) == 0))' ~/.softspark/ai-toolkit/hooks/external/sources.json
|
|
9006
9145
|
```
|
|
9007
9146
|
|
|
9008
|
-
|
|
9147
|
+
- [ ] Both queries return empty arrays (every URL entry has a `sha256`)
|
|
9148
|
+
- [ ] If not, run `ai-toolkit update` to backfill missing hashes before tagging
|
|
9009
9149
|
|
|
9010
|
-
|
|
9150
|
+
**3. Audit SARIF output is well-formed:**
|
|
9011
9151
|
|
|
9012
9152
|
```bash
|
|
9013
|
-
|
|
9014
|
-
# Sort file paths before hashing — find traversal order follows inode order,
|
|
9015
|
-
# which can shift between runs even when content is byte-identical, producing
|
|
9016
|
-
# false FAIL signals.
|
|
9017
|
-
ai-toolkit install --local --editors cursor,gemini --profile full >/dev/null 2>&1
|
|
9018
|
-
SHA1=$(find .cursor .gemini -type f -print0 | LC_ALL=C sort -z | xargs -0 shasum | shasum | awk '{print $1}')
|
|
9019
|
-
ai-toolkit install --local --editors cursor,gemini --profile full >/dev/null 2>&1
|
|
9020
|
-
SHA2=$(find .cursor .gemini -type f -print0 | LC_ALL=C sort -z | xargs -0 shasum | shasum | awk '{print $1}')
|
|
9021
|
-
[ "$SHA1" = "$SHA2" ] && echo "OK: idempotent" || echo "FAIL: install is not idempotent"
|
|
9153
|
+
python3 scripts/audit_skills.py --sarif | python3 -c "import json, sys; d=json.load(sys.stdin); assert d['version']=='2.1.0' and d['runs'][0]['tool']['driver']['name']; print('SARIF OK')"
|
|
9022
9154
|
```
|
|
9023
9155
|
|
|
9024
|
-
|
|
9025
|
-
|
|
9026
|
-
### 9.5 Live-install JSON outputs parse
|
|
9156
|
+
- [ ] Prints `SARIF OK`
|
|
9157
|
+
- [ ] If the script ever grows new rule classes, extend the SARIF `rules[]` coverage before releasing
|
|
9027
9158
|
|
|
9028
|
-
|
|
9159
|
+
**4. Strict-pin mode passes on CI** (optional, recommended for stable branches):
|
|
9029
9160
|
|
|
9030
9161
|
```bash
|
|
9031
|
-
|
|
9032
|
-
ai-toolkit install --local --editors cursor,windsurf,gemini,augment,codex,copilot --profile full >/dev/null 2>&1
|
|
9033
|
-
for f in .cursor/hooks.json .devin/hooks.v1.json .gemini/settings.json .codex/hooks.json .github/hooks/ai-toolkit.json "$HOME/.augment/settings.json"; do
|
|
9034
|
-
[ -f "$f" ] && python3 -c "import json; json.load(open('$f'))" && echo "OK: $f"
|
|
9035
|
-
done
|
|
9162
|
+
AI_TOOLKIT_STRICT_PIN=1 ai-toolkit update --dry-run
|
|
9036
9163
|
```
|
|
9037
9164
|
|
|
9038
|
-
|
|
9165
|
+
- [ ] Exit 0, no `CHECKSUM CHANGED` line
|
|
9166
|
+
- [ ] Any unexpected upstream change blocks the release until explicitly approved
|
|
9039
9167
|
|
|
9040
|
-
|
|
9168
|
+
---
|
|
9041
9169
|
|
|
9042
|
-
|
|
9170
|
+
## Phase 6: Commit
|
|
9171
|
+
|
|
9172
|
+
Stage all release files:
|
|
9043
9173
|
|
|
9044
9174
|
```bash
|
|
9045
|
-
|
|
9046
|
-
|
|
9047
|
-
|
|
9048
|
-
|
|
9175
|
+
git add package.json manifest.json app/.claude-plugin/plugin.json
|
|
9176
|
+
git add package-lock.json
|
|
9177
|
+
git add CHANGELOG.md
|
|
9178
|
+
git add llms.txt llms-full.txt
|
|
9179
|
+
# NOTE: AGENTS.md, GEMINI.md, and .github/copilot-instructions.md are generated
|
|
9180
|
+
# editor configs and are gitignored — do NOT commit them. `prepublishOnly` runs
|
|
9181
|
+
# `npm run generate:all`, so the shipped package (which lists AGENTS.md in
|
|
9182
|
+
# package.json `files`) gets a fresh copy at publish time.
|
|
9183
|
+
git add -p # review and stage any other changes
|
|
9049
9184
|
```
|
|
9050
9185
|
|
|
9051
|
-
|
|
9052
|
-
|
|
9053
|
-
### 9.7 Claude Chat / Cowork plugin validates and exports
|
|
9186
|
+
Commit:
|
|
9054
9187
|
|
|
9055
9188
|
```bash
|
|
9056
|
-
|
|
9057
|
-
D=/tmp/aitk-claude-app-${RANDOM}
|
|
9058
|
-
python3 scripts/claude_app.py export --output "$D.zip" --no-custom-rules
|
|
9059
|
-
python3 -c "import zipfile; z=zipfile.ZipFile('$D.zip'); assert '.claude-plugin/plugin.json' in z.namelist(); assert 'claude-app/skills/ai-toolkit-rules/SKILL.md' in z.namelist(); print('OK: Claude app archive')"
|
|
9189
|
+
git commit -m "chore: release vX.Y.Z"
|
|
9060
9190
|
```
|
|
9061
9191
|
|
|
9062
|
-
**Verify:** the official validator exits 0; the archive contains the manifest,
|
|
9063
|
-
app-native rules skill, bundled agents/skills, and plugin-relative Cowork hooks.
|
|
9064
|
-
|
|
9065
9192
|
---
|
|
9066
9193
|
|
|
9067
|
-
## Phase
|
|
9068
|
-
|
|
9069
|
-
Run this phase whenever the release changes the `dsh` target, package pins, preset lifecycle, or DSH compatibility documentation. Use a new task-specific `DSH_HOME`; never replace `HOME` or reuse a regular profile.
|
|
9070
|
-
|
|
9071
|
-
Prerequisites: DSH `0.1.1-rc.2`, pnpm `>=11.7.0,<12.0.0`, Codex logged in through ChatGPT, Claude Code logged in natively, and GitHub Copilot CLI logged in natively. Do not supply provider API keys.
|
|
9194
|
+
## Phase 7: Push Branch, Verify CI, Tag, and Push Tag
|
|
9072
9195
|
|
|
9073
9196
|
```bash
|
|
9074
|
-
|
|
9075
|
-
|
|
9076
|
-
|
|
9077
|
-
|
|
9078
|
-
|
|
9079
|
-
|
|
9080
|
-
|
|
9197
|
+
RELEASE_SHA=$(git rev-parse HEAD)
|
|
9198
|
+
git push origin main || { echo "FAIL: release commit push failed"; exit 1; }
|
|
9199
|
+
|
|
9200
|
+
# Bind the gate to the exact release commit. GitHub run registration is
|
|
9201
|
+
# asynchronous, so retry at most five times and log every attempt.
|
|
9202
|
+
RUN_ID=""
|
|
9203
|
+
for ATTEMPT in 1 2 3 4 5; do
|
|
9204
|
+
RUN_ID=$(gh run list --workflow ci.yml --event push --commit "$RELEASE_SHA" \
|
|
9205
|
+
--limit 1 --json databaseId --jq '.[0].databaseId')
|
|
9206
|
+
[ -n "$RUN_ID" ] && break
|
|
9207
|
+
echo "Waiting for CI run registration ($ATTEMPT/5)" >&2
|
|
9208
|
+
[ "$ATTEMPT" -eq 5 ] || sleep 60
|
|
9209
|
+
done
|
|
9210
|
+
test -n "$RUN_ID" || { echo "FAIL: CI run not found for release commit"; exit 1; }
|
|
9211
|
+
gh run watch "$RUN_ID" --exit-status \
|
|
9212
|
+
|| { echo "FAIL: release commit CI failed"; exit 1; }
|
|
9213
|
+
|
|
9214
|
+
# publish.yml is Ubuntu-only. Require both full CI matrix jobs before creating
|
|
9215
|
+
# the tag so a green publish job cannot bypass a red macOS test.
|
|
9216
|
+
for OS in ubuntu-latest macos-latest; do
|
|
9217
|
+
gh run view "$RUN_ID" --json jobs \
|
|
9218
|
+
--jq ".jobs[] | select(.name == \"Bats test suite ($OS)\") | .conclusion" \
|
|
9219
|
+
| grep -qx success || { echo "FAIL: $OS test job not green"; exit 1; }
|
|
9220
|
+
done
|
|
9081
9221
|
|
|
9082
|
-
|
|
9222
|
+
# Refuse to tag if either local or remote main moved while CI was running.
|
|
9223
|
+
test "$(git rev-parse HEAD)" = "$RELEASE_SHA" \
|
|
9224
|
+
|| { echo "FAIL: local HEAD changed during CI"; exit 1; }
|
|
9225
|
+
git fetch origin main || { echo "FAIL: origin/main refresh failed"; exit 1; }
|
|
9226
|
+
test "$(git rev-parse origin/main)" = "$RELEASE_SHA" \
|
|
9227
|
+
|| { echo "FAIL: origin/main changed during CI"; exit 1; }
|
|
9083
9228
|
|
|
9084
|
-
|
|
9085
|
-
2. `subagent_gemini_copilot` returns an exact child marker, then the Codex parent returns its exact completion marker.
|
|
9229
|
+
git tag vX.Y.Z "$RELEASE_SHA"
|
|
9086
9230
|
|
|
9087
|
-
|
|
9231
|
+
# Assert the tag before pushing it. Both checks are one line each and both
|
|
9232
|
+
# have caught a real broken release.
|
|
9233
|
+
test "$(git rev-parse vX.Y.Z)" = "$RELEASE_SHA" \
|
|
9234
|
+
|| { echo "FAIL: tag is not on the tested release commit"; exit 1; }
|
|
9235
|
+
git show --no-patch --format=%s vX.Y.Z | grep -qx "chore: release vX.Y.Z" \
|
|
9236
|
+
|| { echo "FAIL: tag is not on the chore: release commit"; exit 1; }
|
|
9088
9237
|
|
|
9089
|
-
|
|
9090
|
-
|
|
9238
|
+
# Push the single release tag by its full ref.
|
|
9239
|
+
git push origin refs/tags/vX.Y.Z
|
|
9091
9240
|
```
|
|
9092
9241
|
|
|
9093
|
-
**
|
|
9094
|
-
|
|
9095
|
-
|
|
9242
|
+
**Why branch CI comes before the tag (v4.30.2 postmortem).** The publish
|
|
9243
|
+
workflow runs only on Ubuntu and can publish while the separate macOS matrix job
|
|
9244
|
+
is red. `v4.30.2` exposed this with a macOS-only DSH timing failure. A release
|
|
9245
|
+
commit must therefore pass the complete Ubuntu/macOS branch workflow before its
|
|
9246
|
+
tag exists; a successful publish workflow is not a substitute for green CI.
|
|
9096
9247
|
|
|
9097
|
-
|
|
9248
|
+
**Why the assertions (v4.19.0 postmortem).** v4.19.0 was tagged on a commit
|
|
9249
|
+
that contained only a KB document and still carried `package.json` version
|
|
9250
|
+
`4.18.0`; the actual release sat in the commit above it under a recycled
|
|
9251
|
+
`fix:` message. `publish.yml` fired, tried to publish a version already on
|
|
9252
|
+
npm, and failed. Nothing on npm, a tag pointing at the wrong tree, and the
|
|
9253
|
+
only way out was rewriting a pushed commit. Both assertions above catch this
|
|
9254
|
+
in under a second. Run them.
|
|
9098
9255
|
|
|
9099
|
-
|
|
9256
|
+
**Never `git push --tags`.** It pushes every local tag at once, and GitHub
|
|
9257
|
+
suppresses tag-triggered workflow runs when many tags arrive in a single push
|
|
9258
|
+
— the workflow silently does not fire and nothing publishes. Push the single
|
|
9259
|
+
release tag by its full ref, as above. (Sibling evidence: this is exactly how
|
|
9260
|
+
rag-mcp's `1.0.3` image build was skipped, when a `--tags` push carried 37
|
|
9261
|
+
tags at once.)
|
|
9100
9262
|
|
|
9101
|
-
|
|
9102
|
-
|
|
9103
|
-
|
|
9104
|
-
|
|
9105
|
-
```
|
|
9263
|
+
This triggers `.github/workflows/publish.yml` which:
|
|
9264
|
+
1. Runs `validate.py --strict`
|
|
9265
|
+
2. Runs `npm test`
|
|
9266
|
+
3. Publishes to npm as `@softspark/ai-toolkit@X.Y.Z` with `--provenance` (SLSA v1 build attestation)
|
|
9106
9267
|
|
|
9107
|
-
|
|
9268
|
+
**Provenance is non-negotiable.** If `id-token: write` permission or the `--provenance` flag is missing from `publish.yml`, fix it BEFORE tagging — an unsigned release is a regression against the v2.8.0 standard.
|
|
9108
9269
|
|
|
9109
|
-
|
|
9110
|
-
|
|
9111
|
-
ai-toolkit update # or full re-install
|
|
9112
|
-
```
|
|
9270
|
+
**After CI completes:** Run the [Release Verification SOP](sop-release-verification.md)
|
|
9271
|
+
to smoke-test the published package AND verify the provenance attestation landed on npm.
|
|
9113
9272
|
|
|
9114
|
-
|
|
9273
|
+
---
|
|
9115
9274
|
|
|
9116
|
-
|
|
9117
|
-
brew install bats-core # macOS
|
|
9118
|
-
npm install -g bats # cross-platform
|
|
9119
|
-
```
|
|
9275
|
+
## Rollback
|
|
9120
9276
|
|
|
9121
|
-
|
|
9277
|
+
If a bad release was published:
|
|
9122
9278
|
|
|
9123
|
-
|
|
9124
|
-
|
|
9279
|
+
```bash
|
|
9280
|
+
# Unpublish from npm (within 72h)
|
|
9281
|
+
npm unpublish @softspark/ai-toolkit@X.Y.Z
|
|
9125
9282
|
|
|
9126
|
-
|
|
9283
|
+
# Or deprecate (preferred — doesn't break existing installs)
|
|
9284
|
+
npm deprecate @softspark/ai-toolkit@X.Y.Z "Known issue: <description>. Use vA.B.C instead."
|
|
9127
9285
|
|
|
9128
|
-
|
|
9129
|
-
|
|
9130
|
-
|
|
9286
|
+
# Delete tag
|
|
9287
|
+
git tag -d vX.Y.Z
|
|
9288
|
+
git push origin --delete vX.Y.Z
|
|
9131
9289
|
```
|
|
9132
9290
|
|
|
9133
9291
|
---
|
|
9134
9292
|
|
|
9135
|
-
##
|
|
9293
|
+
## Checklist Summary
|
|
9136
9294
|
|
|
9137
|
-
|
|
|
9138
|
-
|
|
9139
|
-
|
|
|
9140
|
-
|
|
|
9141
|
-
|
|
|
9142
|
-
|
|
|
9143
|
-
|
|
|
9144
|
-
|
|
|
9145
|
-
|
|
|
9146
|
-
|
|
|
9147
|
-
|
|
|
9148
|
-
|
|
|
9149
|
-
|
|
|
9150
|
-
|
|
|
9151
|
-
|
|
|
9152
|
-
|
|
|
9153
|
-
|
|
|
9154
|
-
|
|
|
9155
|
-
|
|
|
9295
|
+
| # | Step | Command / Action | Pass Criteria |
|
|
9296
|
+
|---|------|-----------------|---------------|
|
|
9297
|
+
| 0a | Ecosystem drift check | `ecosystem_doctor.py --format text` | All tools Clean, or drift classified and resolved |
|
|
9298
|
+
| 0b | Ecosystem snapshot refresh | `ecosystem_doctor.py --update` | `benchmarks/ecosystem-doctor-snapshot.json` updated |
|
|
9299
|
+
| 0c | Ecosystem gate | `ecosystem_doctor.py --offline --check` | Exit 0 |
|
|
9300
|
+
| 1 | Version bump type | Decide patch/minor/major | — |
|
|
9301
|
+
| 2 | `package.json` version | Edit `"version"` | Matches target |
|
|
9302
|
+
| 3 | `manifest.json` version | Edit `"version"` | Matches target |
|
|
9303
|
+
| 4 | `plugin.json` version | Edit `"version"` | Matches target |
|
|
9304
|
+
| 5 | `package-lock.json` | `npm install --package-lock-only` | Matches target |
|
|
9305
|
+
| 6 | Count sync | Check `package.json` description, README | `validate.py` passes |
|
|
9306
|
+
| 7 | CHANGELOG.md | Add release entry (incl. `Ecosystem` subsection if any B/D/E/F drift) | Entry exists for vX.Y.Z |
|
|
9307
|
+
| 8 | Regenerate artifacts | `generate_agents_md.py`, `generate_llms_txt.py` | No unexpected diff |
|
|
9308
|
+
| 9 | Validate | `validate.py --strict` | 0 errors, 0 warnings |
|
|
9309
|
+
| 10 | Security audit (CI mode) | `audit_skills.py --ci` | 0 HIGH |
|
|
9310
|
+
| 11 | Security audit (SARIF) | `audit_skills.py --sarif` | Valid SARIF 2.1.0 JSON |
|
|
9311
|
+
| 12 | Per-skill permissions | `audit_skills.py --permissions` | New broad-access skills justified in CHANGELOG |
|
|
9312
|
+
| 13 | ShellCheck hooks | `shellcheck --severity=warning app/hooks/*.sh` | Exit 0, no output (mirrors ci.yml; publish.yml does NOT run it) |
|
|
9313
|
+
| 14 | Provenance flag check | `grep -- '--provenance' .github/workflows/publish.yml` | Present |
|
|
9314
|
+
| 15 | Checksum-pin backfill | `sources.json` entries all have `sha256` | No unpinned URL sources |
|
|
9315
|
+
| 15a | Licensing gate | `npx bats tests/test_licensing.bats` | 7/7 — SPDX headers, LICENSE, NOTICE, npm `files`, manifest consistency |
|
|
9316
|
+
| 16 | Tests | `git add -A kb/` if the KB changed, then `npm test` | All pass |
|
|
9317
|
+
| 17 | Commit | `git commit` | Clean working tree |
|
|
9318
|
+
| 18 | Push branch | `git push origin main` | Exact release commit is on `origin/main` |
|
|
9319
|
+
| 18a | Full branch CI | `gh run watch "$RUN_ID" --exit-status` plus matrix job assertions | Ubuntu and macOS Bats jobs both conclude `success` |
|
|
9320
|
+
| 19 | Tag | `git tag vX.Y.Z` | Tag exists only after green branch CI |
|
|
9321
|
+
| 19a | Tag is on tested SHA | `test "$(git rev-parse vX.Y.Z)" = "$RELEASE_SHA"` | Exit 0 |
|
|
9322
|
+
| 19b | Tag is on the release commit | `git show --no-patch --format=%s vX.Y.Z` | Reads `chore: release vX.Y.Z` |
|
|
9323
|
+
| 20 | Push the single tag | `git push origin refs/tags/vX.Y.Z` | Publish CI triggered with `id-token: write`. Never `--tags`. |
|
|
9156
9324
|
|
|
9157
9325
|
---
|
|
9158
9326
|
|
|
@@ -9740,7 +9908,7 @@ Three tiers determine how to approach a task:
|
|
|
9740
9908
|
|------|-------|-----------|-------|
|
|
9741
9909
|
| Task | `disable-model-invocation: true` | User via `/skill` only | 32 |
|
|
9742
9910
|
| Hybrid | (neither) | User via `/skill` + agent knowledge | 31 |
|
|
9743
|
-
| Knowledge | `user-invocable: false` | Claude auto-loads |
|
|
9911
|
+
| Knowledge | `user-invocable: false` | Claude auto-loads | 51 |
|
|
9744
9912
|
|
|
9745
9913
|
## Multi-Agent Execution
|
|
9746
9914
|
|
|
@@ -11677,7 +11845,7 @@ running the generators directly:
|
|
|
11677
11845
|
|
|
11678
11846
|
- `scripts/generate_copilot.py` / `generate_copilot_hooks.py` contract tests
|
|
11679
11847
|
(`tests/test_copilot.bats`, `tests/test_copilot_hooks.bats`).
|
|
11680
|
-
- Release layout check in `kb/procedures/release-verification
|
|
11848
|
+
- Release layout check in `kb/procedures/sop-release-verification.md` asserts the
|
|
11681
11849
|
`.github/{agents,skills,instructions,prompts,hooks}` surfaces.
|
|
11682
11850
|
- `validate.py --strict` + `audit_skills.py --ci` in CI.
|
|
11683
11851
|
|
|
@@ -14252,7 +14420,7 @@ MIT remnant would need each contributor's agreement.
|
|
|
14252
14420
|
|
|
14253
14421
|
It is a test rather than a checklist line on purpose. This project has two
|
|
14254
14422
|
same-day postmortems about SOPs that existed and were skipped; CI does not skip.
|
|
14255
|
-
[Release Preparation](../procedures/release
|
|
14423
|
+
[Release Preparation](../procedures/sop-release.md) Phase 5c runs the
|
|
14256
14424
|
same gate before tagging so a failure surfaces before the tag, not after.
|
|
14257
14425
|
|
|
14258
14426
|
## If the licence ever changes again
|
|
@@ -14265,7 +14433,7 @@ source, cross-verify against a second independent copy, and only then write
|
|
|
14265
14433
|
## Related
|
|
14266
14434
|
|
|
14267
14435
|
- [`LICENSE`](../../LICENSE), [`NOTICE`](../../NOTICE)
|
|
14268
|
-
- [Release Preparation SOP](../procedures/release
|
|
14436
|
+
- [Release Preparation SOP](../procedures/sop-release.md) — Phase 5c
|
|
14269
14437
|
- [Distribution Model](distribution-model.md) — what ships and where
|
|
14270
14438
|
|
|
14271
14439
|
---
|
|
@@ -15945,7 +16113,7 @@ Updated baseline docs:
|
|
|
15945
16113
|
- `kb/reference/skills-catalog.md`
|
|
15946
16114
|
- `kb/reference/plugin-pack-conventions.md`
|
|
15947
16115
|
- `kb/reference/claude-ecosystem-benchmark-snapshot.md`
|
|
15948
|
-
- `kb/procedures/maintenance
|
|
16116
|
+
- `kb/procedures/sop-maintenance.md`
|
|
15949
16117
|
|
|
15950
16118
|
## Validation Evidence
|
|
15951
16119
|
|
|
@@ -16799,13 +16967,13 @@ See [DSH Compatibility](./dsh-compatibility.md) for commands, topology, subscrip
|
|
|
16799
16967
|
2. Delete its section from this document.
|
|
16800
16968
|
3. Delete its snapshot entry from `benchmarks/ecosystem-doctor-snapshot.json` (or let the next `--update` prune it — currently not pruned automatically).
|
|
16801
16969
|
4. Decide whether to keep the generator (`scripts/generate_<tool>_*.py`) for backwards compatibility or delete it.
|
|
16802
|
-
5. Remove references from `README.md`, `manifest.json` `description` field, and `kb/procedures/maintenance
|
|
16970
|
+
5. Remove references from `README.md`, `manifest.json` `description` field, and `kb/procedures/sop-maintenance.md` `Supported editors` line.
|
|
16803
16971
|
|
|
16804
16972
|
---
|
|
16805
16973
|
|
|
16806
16974
|
## Related
|
|
16807
16975
|
|
|
16808
|
-
- [Ecosystem Sync SOP](../procedures/ecosystem-sync
|
|
16976
|
+
- [Ecosystem Sync SOP](../procedures/sop-ecosystem-sync.md) — how to use the doctor
|
|
16809
16977
|
- [MCP Editor Compatibility](./mcp-editor-compatibility.md) — MCP-specific subset
|
|
16810
16978
|
- `scripts/ecosystem_tools.json` — source of truth
|
|
16811
16979
|
- `scripts/ecosystem_doctor.py` — drift detector
|
|
@@ -17338,7 +17506,7 @@ Expect `Total plugin skills loaded: 0`, `Total plugin agents loaded: 0`, and the
|
|
|
17338
17506
|
## Related
|
|
17339
17507
|
|
|
17340
17508
|
- `kb/reference/global-install-model.md`
|
|
17341
|
-
- `kb/procedures/maintenance
|
|
17509
|
+
- `kb/procedures/sop-maintenance.md`
|
|
17342
17510
|
|
|
17343
17511
|
---
|
|
17344
17512
|
|