@softspark/ai-toolkit 4.30.3 → 4.32.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +123 -0
- package/README.md +32 -25
- package/app/.claude-plugin/plugin.json +1 -1
- package/app/agents/code-reviewer.md +14 -0
- package/app/agents/frontend-specialist.md +27 -24
- package/app/agents/infrastructure-validator.md +2 -2
- package/app/claude-app/hooks/hooks.json +4 -2
- package/app/claude-app/skills/ai-toolkit-rules/SKILL.md +74 -17
- package/app/hooks/quality-gate.sh +9 -2
- package/app/hooks.json +4 -2
- package/app/personas/frontend-lead.md +20 -12
- package/app/rules/claude-toolkit-rules.md +4 -1
- package/app/rules/common/git-team.md +33 -0
- package/app/rules/common/git-workflow.md +6 -20
- package/app/rules/common/performance.md +25 -1
- package/app/rules/common/testing.md +7 -1
- package/app/rules/edit-discipline.md +37 -0
- package/app/skills/app-builder/SKILL.md +37 -6
- package/app/skills/claude-toolkit-rules/SKILL.md +37 -0
- package/app/skills/design-engineering/SKILL.md +87 -102
- package/app/skills/design-engineering/reference/animation-recipes.md +67 -0
- package/app/skills/design-engineering/reference/component-craft.md +28 -0
- package/app/skills/documentation-standards/SKILL.md +1 -1
- package/app/skills/edit-discipline/SKILL.md +50 -0
- package/app/skills/git-conventions/SKILL.md +18 -0
- package/app/skills/output-mode/SKILL.md +41 -0
- package/app/skills/quality-gates/SKILL.md +23 -0
- package/app/skills/review/SKILL.md +10 -0
- package/benchmarks/ecosystem-doctor-snapshot.json +17 -15
- package/bin/ai-toolkit.js +2 -0
- package/kb/procedures/{ecosystem-sync-sop.md → sop-ecosystem-sync.md} +2 -2
- package/kb/procedures/{maintenance-sop.md → sop-maintenance.md} +6 -3
- package/kb/procedures/{post-release-testing-sop.md → sop-post-release-testing.md} +3 -3
- package/kb/procedures/sop-pre-commit.md +96 -0
- package/kb/procedures/{release-preparation-sop.md → sop-release.md} +6 -6
- package/kb/reference/architecture-overview.md +1 -1
- package/kb/reference/cli-reference.md +3 -2
- package/kb/reference/copilot-compatibility.md +1 -1
- package/kb/reference/global-install-model.md +16 -3
- package/kb/reference/hooks-catalog.md +5 -3
- package/kb/reference/language-rules.md +28 -10
- package/kb/reference/licensing.md +2 -2
- package/kb/reference/quick-wins-implementation-summary.md +1 -1
- package/kb/reference/supported-tools-registry.md +2 -2
- package/kb/reference/unique-features.md +2 -1
- package/kb/troubleshooting/plugin-double-load.md +1 -1
- package/llms-full.txt +1016 -871
- package/llms.txt +6 -5
- package/manifest.json +2 -2
- package/package.json +6 -2
- package/scripts/benchmark_ecosystem.py +0 -1
- package/scripts/check_split.py +11 -9
- package/scripts/claude_app.py +5 -7
- package/scripts/codex_skill_adapter.py +4 -12
- package/scripts/compile_slm.py +10 -26
- package/scripts/doctor.py +322 -0
- package/scripts/evaluate_skills.py +1 -1
- package/scripts/frontmatter.py +452 -29
- package/scripts/generate_augment_rules.py +4 -4
- package/scripts/generate_cursor_mdc.py +2 -3
- package/scripts/generate_language_rules_skills.py +8 -14
- package/scripts/generate_llms_txt.py +1 -15
- package/scripts/generate_opencode_agents.py +0 -1
- package/scripts/generate_opencode_skills.py +2 -20
- package/scripts/generate_toolkit_rules_skills.py +189 -0
- package/scripts/generate_windsurf_rules.py +0 -1
- package/scripts/generator_base.py +0 -1
- package/scripts/inject_hook_cli.py +15 -2
- package/scripts/inject_mcp_cli.py +1 -2
- package/scripts/install.py +32 -1
- package/scripts/install_git_hooks.py +0 -1
- package/scripts/install_steps/ai_tools.py +65 -25
- package/scripts/install_steps/markers.py +6 -6
- package/scripts/install_steps/skill_scope.py +188 -0
- package/scripts/instruction_core.py +5 -8
- package/scripts/merge-hooks.py +13 -3
- package/scripts/pack_codebase.py +1 -1
- package/scripts/surface_manifest.py +6 -7
- package/scripts/validate.py +181 -12
- /package/kb/procedures/{release-verification-sop.md → sop-release-verification.md} +0 -0
package/llms-full.txt
CHANGED
|
@@ -32,11 +32,12 @@
|
|
|
32
32
|
- [Closed: Tool-Output Token Reduction](kb/history/completed/tool-output-token-reduction-closed-20260727.md)
|
|
33
33
|
- [How-To Guides](kb/howto/README.md)
|
|
34
34
|
- [Plan: Cloud Security Pack — Multi-Cloud Audit](kb/planning/cloud-security-pack-plan.md)
|
|
35
|
-
- [SOP: Ecosystem Sync](kb/procedures/ecosystem-sync
|
|
36
|
-
- [SOP: AI Toolkit Maintenance](kb/procedures/maintenance
|
|
37
|
-
- [SOP: Post-Release Testing](kb/procedures/post-release-testing
|
|
38
|
-
- [SOP:
|
|
39
|
-
- [SOP: Release Verification](kb/procedures/release-verification
|
|
35
|
+
- [SOP: Ecosystem Sync](kb/procedures/sop-ecosystem-sync.md)
|
|
36
|
+
- [SOP: AI Toolkit Maintenance](kb/procedures/sop-maintenance.md)
|
|
37
|
+
- [SOP: Post-Release Testing](kb/procedures/sop-post-release-testing.md)
|
|
38
|
+
- [SOP: Pre-Commit Gate](kb/procedures/sop-pre-commit.md)
|
|
39
|
+
- [SOP: Release Verification](kb/procedures/sop-release-verification.md)
|
|
40
|
+
- [SOP: Release Preparation](kb/procedures/sop-release.md)
|
|
40
41
|
- [Agents Catalog](kb/reference/agents-catalog.md)
|
|
41
42
|
- [Anti-Pattern Registry Format](kb/reference/anti-pattern-registry-format.md)
|
|
42
43
|
- [AI Toolkit Architecture](kb/reference/architecture-overview.md)
|
|
@@ -95,6 +96,7 @@
|
|
|
95
96
|
- **chaos**: Injects controlled faults for resilience testing on non-prod. Triggers: chaos, fault injection, latency injection, dependency kill, resilience test.
|
|
96
97
|
- **ci**: Detect/generate/debug CI pipeline config (GitHub Actions, GitLab CI). Triggers: CI setup, build pipeline, GitHub Actions config, debug CI, GitLab CI.
|
|
97
98
|
- **ci-cd-patterns**: CI/CD: GitHub Actions, GitLab CI, Jenkins, caching, blue-green, canary. Triggers: CI, CD, pipeline, GitHub Actions, workflow YAML, release, canary, rollout.
|
|
99
|
+
- **claude-toolkit-rules**: SoftSpark working agreement: never guess a home directory path, give at least three alternatives, and apply a devil's advocate critique to decisions. Triggers: toolkit, conventions, workflow, alternatives, review.
|
|
98
100
|
- **clean-code**: Code quality: meaningful names, SRP, DRY, small functions, guard clauses, refactoring. Triggers: clean code, naming, code smell, SRP, DRY, long function, god class, dead code.
|
|
99
101
|
- **command-creator**: Creates new Claude Code slash commands with frontmatter and validation. Triggers: new slash command, create command, command scaffold.
|
|
100
102
|
- **commit**: Creates Conventional Commits with pre-commit validation. Triggers: commit, conventional commit, git commit, message.
|
|
@@ -115,12 +117,14 @@
|
|
|
115
117
|
- **docs**: Generates/updates README, API docs, architecture notes. Triggers: docs, README, API docs, architecture note, documentation.
|
|
116
118
|
- **documentation-standards**: KB conventions: YAML frontmatter, 10-category taxonomy (reference/howto/procedures/troubleshooting/best-practices/decisions/runbooks/planning/business/templates). Triggers: kb/, SOP, runbook, howto, frontmatter, knowledge base.
|
|
117
119
|
- **ecommerce-patterns**: E-commerce: cart, checkout, payments (Stripe/Adyen), order state, inventory, promos, tax. Triggers: cart, checkout, SKU, payment, Stripe, Shopify, Medusa, Magento, coupon, refund.
|
|
120
|
+
- **edit-discipline**: Change files with the edit and write tools, never by rewriting them through bash (sed, awk, tee, heredoc, redirection), and show `git diff` before reporting a file-changing task as done. Triggers: edit, write, modify, refactor, patch, fix, diff, review changes.
|
|
118
121
|
- **evaluate**: Evaluates RAG retrieval and LLM-as-judge metrics (faithfulness, relevancy, context precision). Triggers: measure RAG quality, knowledge gap, RAG eval, golden dataset.
|
|
119
122
|
- **evolve**: Analyzes agent/skill failures, drafts prompt/permission fixes. Triggers: improve agent, refine skill, system prompt, optimize agent.
|
|
120
123
|
- **explain**: Explains code/architecture with Mermaid diagrams and sequence flows. Triggers: what does X do, how does Y work, explain code, sequence diagram.
|
|
121
124
|
- **explore**: Explores codebase structure, stack, and architecture. Triggers: explore codebase, project structure, stack overview, architecture map.
|
|
122
125
|
- **fix**: Applies targeted fix to known bug/lint error, verifies with same command that surfaced it. Triggers: fix, apply fix, fix bug, fix lint, targeted fix.
|
|
123
126
|
- **flutter-patterns**: Flutter/Dart: widgets, state mgmt (Riverpod/Bloc), navigation, platform channels. Triggers: Flutter, Dart, widget, Riverpod, Bloc, pubspec, hot reload.
|
|
127
|
+
- **git-conventions**: Conventional Commits only (feat, fix, docs, refactor, test, chore); no AI co-authorship trailer and no AI signature in a commit message. Triggers: commit, git, message, changelog, release, pull request.
|
|
124
128
|
- **git-mastery**: Advanced Git: rebase, bisect, reflog, cherry-pick, worktrees, LFS. Triggers: rebase, bisect, cherry-pick, reflog, force push, merge conflict, worktree.
|
|
125
129
|
- **golang-rules**: Go coding rules: style, patterns, security, testing. Triggers: .go, go.mod, go.sum, Gin, Echo, Gorilla, testing, gofmt.
|
|
126
130
|
- **grill-me**: Stress-tests a plan via Socratic questioning down each decision branch. Triggers: stress-test, grill me, validate assumptions, challenge plan, socratic review.
|
|
@@ -147,6 +151,7 @@
|
|
|
147
151
|
- **observability-patterns**: Observability: structured logs, metrics (RED/USE), tracing, SLO/SLI. Triggers: logging, metrics, Prometheus, Grafana, OpenTelemetry, trace, monitoring.
|
|
148
152
|
- **onboard**: Sets up ai-toolkit in a project: symlinks, CLAUDE.md, intent interview. Triggers: onboard, setup project, install ai-toolkit, migrate project.
|
|
149
153
|
- **orchestrate**: Coordinates multiple specialized agents in parallel. Triggers: orchestrate, multi-agent, parallel agents, coordinate agents.
|
|
154
|
+
- **output-mode**: Answer concisely: lead with the result, no preamble, tables over prose, no trailing restatement of a diff the reader can already see. Triggers: response style, verbosity, summary, explanation, report.
|
|
150
155
|
- **panic**: Emergency kill switch — halts all agents via lockfile gate. Triggers: panic, stop everything, kill switch, halt agents, agents looping.
|
|
151
156
|
- **performance-profiling**: Performance: golden signals, p50/p95/p99, flame graphs, load testing. Triggers: performance, slow, latency, p99, flame graph, bottleneck, memory leak.
|
|
152
157
|
- **persona**: Switches engineering persona at runtime. Triggers: persona, switch role, backend-lead, frontend-lead, devops-eng, junior-dev.
|
|
@@ -160,6 +165,7 @@
|
|
|
160
165
|
- **prompt-caching-patterns**: Anthropic API prompt caching: TTL, breakpoints, stacking, invalidation, hit rate. Triggers: prompt caching, cache_control, cache breakpoint, cache TTL, hit rate.
|
|
161
166
|
- **python-rules**: Python coding rules: style, patterns, security, testing. Triggers: .py, .pyi, pyproject.toml, requirements.txt, Pipfile, FastAPI, Django, Flask, pytest, SQLAlchemy, ruff, mypy.
|
|
162
167
|
- **qa-session**: Interactive QA: user reports bugs conversationally, agent files GitHub issues. Triggers: QA session, report bug, file issue, conversational QA, bug intake.
|
|
168
|
+
- **quality-gates**: Plan before work over an hour, and hold the gates: ruff clean, mypy --strict clean, pytest coverage above 70 percent, no secrets in code. Triggers: quality, lint, mypy, pytest, coverage, gate, definition of done.
|
|
163
169
|
- **rag-patterns**: RAG: embeddings, chunking, hybrid search (BM25+vector), reranking, CRAG, multi-hop. Triggers: RAG, embedding, pgvector, Qdrant, Pinecone, Weaviate, reranker, semantic search.
|
|
164
170
|
- **refactor**: Refactors code for quality and maintainability. Triggers: refactor, clean up, restructure, improve code, modernize.
|
|
165
171
|
- **refactor-plan**: Creates detailed refactor plan with tiny commits via interview, files as GitHub RFC. Triggers: refactor plan, refactoring RFC, incremental refactor, safe steps.
|
|
@@ -6929,7 +6935,7 @@ Examples:
|
|
|
6929
6935
|
|
|
6930
6936
|
---
|
|
6931
6937
|
|
|
6932
|
-
## kb/procedures/ecosystem-sync
|
|
6938
|
+
## kb/procedures/sop-ecosystem-sync.md
|
|
6933
6939
|
|
|
6934
6940
|
---
|
|
6935
6941
|
title: "SOP: Ecosystem Sync"
|
|
@@ -7186,21 +7192,21 @@ jobs:
|
|
|
7186
7192
|
|
|
7187
7193
|
- [Supported Tools Registry](../reference/supported-tools-registry.md) — human-readable per-tool breakdown
|
|
7188
7194
|
- [MCP Editor Compatibility](../reference/mcp-editor-compatibility.md) — MCP-specific adapter table
|
|
7189
|
-
- [Maintenance SOP](maintenance
|
|
7190
|
-
- [Release Preparation SOP](release
|
|
7195
|
+
- [Maintenance SOP](sop-maintenance.md) — general toolkit upkeep
|
|
7196
|
+
- [Release Preparation SOP](sop-release.md) — run the doctor before tagging
|
|
7191
7197
|
|
|
7192
7198
|
---
|
|
7193
7199
|
|
|
7194
|
-
## kb/procedures/maintenance
|
|
7200
|
+
## kb/procedures/sop-maintenance.md
|
|
7195
7201
|
|
|
7196
7202
|
---
|
|
7197
7203
|
title: "SOP: AI Toolkit Maintenance"
|
|
7198
7204
|
category: procedures
|
|
7199
7205
|
service: ai-toolkit
|
|
7200
7206
|
tags: [sop, maintenance, agents, skills, install]
|
|
7201
|
-
version: "3.
|
|
7207
|
+
version: "3.5.0"
|
|
7202
7208
|
created: "2026-03-23"
|
|
7203
|
-
last_updated: "2026-
|
|
7209
|
+
last_updated: "2026-09-04"
|
|
7204
7210
|
description: "Standard operating procedures for installing, maintaining, and evolving the ai-toolkit."
|
|
7205
7211
|
---
|
|
7206
7212
|
|
|
@@ -7424,7 +7430,9 @@ that runtime should receive the change.
|
|
|
7424
7430
|
greets the user with a traceback, and one that reads stdin must answer an
|
|
7425
7431
|
empty stdin with an error rather than blocking forever.
|
|
7426
7432
|
3. Update `kb/reference/skills-catalog.md` and `app/ARCHITECTURE.md`
|
|
7427
|
-
4. Run `scripts/validate.py` — it checks the invocation, the body budget,
|
|
7433
|
+
4. Run `scripts/validate.py` — it checks the invocation, the body budget, the
|
|
7434
|
+
description budget (warn over 400 characters, error over 1024; unquoted
|
|
7435
|
+
descriptions containing `: ` or ` #` are rejected, use `>-`), and
|
|
7428
7436
|
`reference/` link resolution
|
|
7429
7437
|
5. Run `python3 scripts/surface_manifest.py` before the next release to adopt the
|
|
7430
7438
|
new skill into the protected surface
|
|
@@ -7500,6 +7508,7 @@ bats tests/test_install.bats tests/test_codex.bats
|
|
|
7500
7508
|
bats tests/test_claude_app.bats tests/test_hooks_per_editor.bats
|
|
7501
7509
|
python3 scripts/claude_app.py verify
|
|
7502
7510
|
python3 scripts/validate.py --strict
|
|
7511
|
+
npm run test:py # pytest over scripts/ logic incl. the frontmatter corpus test (dev venv, see CONTRIBUTING.md)
|
|
7503
7512
|
```
|
|
7504
7513
|
|
|
7505
7514
|
## Managing Plugins
|
|
@@ -7705,7 +7714,7 @@ What `uninstall` does:
|
|
|
7705
7714
|
|
|
7706
7715
|
---
|
|
7707
7716
|
|
|
7708
|
-
## kb/procedures/post-release-testing
|
|
7717
|
+
## kb/procedures/sop-post-release-testing.md
|
|
7709
7718
|
|
|
7710
7719
|
---
|
|
7711
7720
|
title: "SOP: Post-Release Testing"
|
|
@@ -7725,7 +7734,7 @@ actually install, from npm, rather than the working tree.
|
|
|
7725
7734
|
|
|
7726
7735
|
Sibling procedures exist for `jira-mcp` and `legal-pl-pack`; this is the
|
|
7727
7736
|
ai-toolkit equivalent. It complements
|
|
7728
|
-
[Release Verification](release-verification
|
|
7737
|
+
[Release Verification](sop-release-verification.md), which checks the toolkit
|
|
7729
7738
|
from the maintainer's own installed copy. The difference that matters: this one
|
|
7730
7739
|
never writes to the maintainer's `~/.claude` or `~/.softspark`.
|
|
7731
7740
|
|
|
@@ -8019,1201 +8028,1302 @@ print(f'removed {sb} ({n} files)')
|
|
|
8019
8028
|
|
|
8020
8029
|
## Related
|
|
8021
8030
|
|
|
8022
|
-
- [Release Preparation](release
|
|
8023
|
-
- [Release Verification](release-verification
|
|
8031
|
+
- [Release Preparation](sop-release.md) — run before tagging
|
|
8032
|
+
- [Release Verification](sop-release-verification.md) — the maintainer-install checks
|
|
8024
8033
|
- [rtk-pack Retirement](../history/completed/rtk-pack-retirement-20260727.md) — what happened the one time this SOP was written and not run
|
|
8025
8034
|
|
|
8026
8035
|
---
|
|
8027
8036
|
|
|
8028
|
-
## kb/procedures/
|
|
8037
|
+
## kb/procedures/sop-pre-commit.md
|
|
8029
8038
|
|
|
8030
8039
|
---
|
|
8031
|
-
title: "SOP:
|
|
8040
|
+
title: "SOP: Pre-Commit Gate"
|
|
8032
8041
|
category: procedures
|
|
8033
8042
|
service: ai-toolkit
|
|
8034
|
-
tags: [
|
|
8035
|
-
version: "1.
|
|
8036
|
-
created: "2026-
|
|
8043
|
+
tags: [pre-commit, quality-gate, shellcheck, bats, validation, secrets]
|
|
8044
|
+
version: "1.0.0"
|
|
8045
|
+
created: "2026-09-02"
|
|
8037
8046
|
last_updated: "2026-09-02"
|
|
8038
|
-
description: "
|
|
8047
|
+
description: "The local gate to run before every commit to ai-toolkit: staged-file and secret scan, ShellCheck, Python syntax, toolkit validation, and the Bats suite. Runs the same checks CI runs, so main does not go red after a push."
|
|
8039
8048
|
---
|
|
8040
8049
|
|
|
8041
|
-
# SOP:
|
|
8050
|
+
# SOP: Pre-Commit Gate
|
|
8042
8051
|
|
|
8043
|
-
|
|
8044
|
-
Run this **before** tagging. After tagging and publishing, run the
|
|
8045
|
-
[Release Verification SOP](release-verification-sop.md) to smoke-test.
|
|
8052
|
+
## Why this exists
|
|
8046
8053
|
|
|
8047
|
-
|
|
8048
|
-
|
|
8049
|
-
|
|
8050
|
-
|
|
8051
|
-
|
|
8052
|
-
↓
|
|
8053
|
-
git tag → CI publish → Release Verification SOP
|
|
8054
|
-
```
|
|
8054
|
+
CI runs on `main` and on pull requests, but it runs *after* the push. Every
|
|
8055
|
+
check below already exists in `.github/workflows/ci.yml`; running them locally
|
|
8056
|
+
first is what keeps `main` green and keeps a release from becoming a debug
|
|
8057
|
+
session. The v4.5.1 postmortem in `sop-release.md` is the case
|
|
8058
|
+
where a hook lint failure published while reddening `main` CI.
|
|
8055
8059
|
|
|
8056
|
-
|
|
8060
|
+
This gate is not a substitute for `sop-release.md`. That SOP gates
|
|
8061
|
+
a *tag*; this one gates a *commit*.
|
|
8057
8062
|
|
|
8058
|
-
|
|
8063
|
+
## Prerequisites
|
|
8059
8064
|
|
|
8060
|
-
|
|
8065
|
+
`shellcheck`, `python3` and `bats` must be on `PATH`. The suite is the one
|
|
8066
|
+
check that silently does nothing when its runner is absent, so install it
|
|
8067
|
+
before relying on this gate:
|
|
8061
8068
|
|
|
8062
8069
|
```bash
|
|
8063
|
-
|
|
8064
|
-
#
|
|
8065
|
-
python3 scripts/ecosystem_doctor.py --format text > /tmp/eco-report.txt
|
|
8066
|
-
cat /tmp/eco-report.txt
|
|
8067
|
-
# If drift detected: stop here, follow ecosystem-sync-sop.md Phase 2-4 to
|
|
8068
|
-
# classify each drift (A-F), update generators as needed, refresh snapshot,
|
|
8069
|
-
# THEN resume this SOP.
|
|
8070
|
-
python3 scripts/ecosystem_doctor.py --update # after all drift resolved
|
|
8071
|
-
|
|
8072
|
-
# 1. Decide version bump
|
|
8073
|
-
# patch (1.4.2 → 1.4.3): bugfix, typo, doc fix
|
|
8074
|
-
# minor (1.4.2 → 1.5.0): new feature, new skill, new flag, any ecosystem-class-B/F change
|
|
8075
|
-
# major (1.4.2 → 2.0.0): breaking change, any ecosystem-class-D removed path
|
|
8076
|
-
|
|
8077
|
-
# 2. Sync version across all files
|
|
8078
|
-
python3 scripts/sync_version.py X.Y.Z # if script exists, else manual
|
|
8079
|
-
|
|
8080
|
-
# 3. Write CHANGELOG.md entry
|
|
8081
|
-
# 4. Regenerate artifacts
|
|
8082
|
-
python3 scripts/generate_agents_md.py > AGENTS.md
|
|
8083
|
-
python3 scripts/generate_llms_txt.py > llms.txt
|
|
8084
|
-
python3 scripts/generate_llms_txt.py --full > llms-full.txt
|
|
8085
|
-
|
|
8086
|
-
# 5. Validate + audit + SARIF + shellcheck + test + ecosystem check
|
|
8087
|
-
python3 scripts/validate.py --strict && python3 scripts/audit_skills.py --ci && python3 scripts/audit_skills.py --sarif > /tmp/audit.sarif && shellcheck --severity=warning app/hooks/*.sh && npm test
|
|
8088
|
-
|
|
8089
|
-
# 5a. Supply-chain standard (v2.8.0+) — non-negotiable
|
|
8090
|
-
grep -q -- '--provenance' .github/workflows/publish.yml || { echo "MISSING --provenance"; exit 1; }
|
|
8091
|
-
grep -q 'id-token: write' .github/workflows/publish.yml || { echo "MISSING id-token: write"; exit 1; }
|
|
8092
|
-
python3 scripts/audit_skills.py --permissions # review Bash/Write/Edit footprint
|
|
8093
|
-
|
|
8094
|
-
# 5b. Ecosystem gate — snapshot must be current before tag
|
|
8095
|
-
python3 scripts/ecosystem_doctor.py --offline --check || { echo "STALE ecosystem snapshot — re-run doctor"; exit 1; }
|
|
8096
|
-
|
|
8097
|
-
# 5c. Licensing gate — SPDX headers, LICENSE, NOTICE, manifest consistency
|
|
8098
|
-
npx bats tests/test_licensing.bats || { echo "LICENSING GATE FAILED"; exit 1; }
|
|
8099
|
-
|
|
8100
|
-
# 6. Commit, push branch, and wait for the exact commit's full CI
|
|
8101
|
-
git add -A && git commit -m "chore: release vX.Y.Z"
|
|
8102
|
-
RELEASE_SHA=$(git rev-parse HEAD)
|
|
8103
|
-
git push origin main || { echo "FAIL: release commit push failed"; exit 1; }
|
|
8104
|
-
RUN_ID=""
|
|
8105
|
-
for ATTEMPT in 1 2 3 4 5; do
|
|
8106
|
-
RUN_ID=$(gh run list --workflow ci.yml --event push --commit "$RELEASE_SHA" \
|
|
8107
|
-
--limit 1 --json databaseId --jq '.[0].databaseId')
|
|
8108
|
-
[ -n "$RUN_ID" ] && break
|
|
8109
|
-
echo "Waiting for CI run registration ($ATTEMPT/5)" >&2
|
|
8110
|
-
[ "$ATTEMPT" -eq 5 ] || sleep 60
|
|
8111
|
-
done
|
|
8112
|
-
test -n "$RUN_ID" || { echo "FAIL: CI run not found for release commit"; exit 1; }
|
|
8113
|
-
gh run watch "$RUN_ID" --exit-status \
|
|
8114
|
-
|| { echo "FAIL: release commit CI failed"; exit 1; }
|
|
8115
|
-
for OS in ubuntu-latest macos-latest; do
|
|
8116
|
-
gh run view "$RUN_ID" --json jobs \
|
|
8117
|
-
--jq ".jobs[] | select(.name == \"Bats test suite ($OS)\") | .conclusion" \
|
|
8118
|
-
| grep -qx success || { echo "FAIL: $OS test job not green"; exit 1; }
|
|
8119
|
-
done
|
|
8120
|
-
test "$(git rev-parse HEAD)" = "$RELEASE_SHA" || { echo "FAIL: local HEAD changed during CI"; exit 1; }
|
|
8121
|
-
git fetch origin main || { echo "FAIL: origin/main refresh failed"; exit 1; }
|
|
8122
|
-
test "$(git rev-parse origin/main)" = "$RELEASE_SHA" || { echo "FAIL: origin/main changed during CI"; exit 1; }
|
|
8123
|
-
|
|
8124
|
-
# 6a. Only green branch CI authorizes tag creation
|
|
8125
|
-
git tag vX.Y.Z "$RELEASE_SHA"
|
|
8126
|
-
|
|
8127
|
-
# 6b. Assert the tag before pushing it (v4.19.0 postmortem, Phase 7)
|
|
8128
|
-
test "$(git rev-parse vX.Y.Z)" = "$RELEASE_SHA" || { echo "FAIL: tag not on tested release commit"; exit 1; }
|
|
8129
|
-
git show --no-patch --format=%s vX.Y.Z | grep -qx "chore: release vX.Y.Z" || { echo "FAIL: tag not on release commit"; exit 1; }
|
|
8130
|
-
|
|
8131
|
-
# 6c. Push the single tag by full ref. Never --tags.
|
|
8132
|
-
git push origin refs/tags/vX.Y.Z
|
|
8070
|
+
brew install bats-core shellcheck # macOS
|
|
8071
|
+
sudo apt-get install -y bats shellcheck # Debian / Ubuntu
|
|
8133
8072
|
```
|
|
8134
8073
|
|
|
8135
|
-
|
|
8074
|
+
## The gate
|
|
8136
8075
|
|
|
8137
|
-
|
|
8076
|
+
Run all five from the repository root. Each mirrors a CI job.
|
|
8138
8077
|
|
|
8139
|
-
|
|
8078
|
+
```bash
|
|
8079
|
+
# 1. Staged files: secrets, large blobs, commit-type suggestion
|
|
8080
|
+
python3 app/skills/commit/scripts/pre-commit-check.py
|
|
8140
8081
|
|
|
8141
|
-
|
|
8142
|
-
|
|
8143
|
-
- Major release (X.0.0) — always
|
|
8144
|
-
- Patch release (X.Y.Z) — only if the patch touches a generator or install flow
|
|
8082
|
+
# 2. ShellCheck hooks -- the CI job that has published while red before
|
|
8083
|
+
shellcheck --severity=warning app/hooks/*.sh app/plugins/*/hooks/*.sh
|
|
8145
8084
|
|
|
8146
|
-
|
|
8085
|
+
# 3. Python syntax across every script the toolkit ships
|
|
8086
|
+
python3 -m py_compile scripts/*.py app/skills/*/scripts/*.py
|
|
8147
8087
|
|
|
8148
|
-
|
|
8088
|
+
# 4. Toolkit integrity: agents, skills, registry drift, content quality
|
|
8089
|
+
npm run validate
|
|
8149
8090
|
|
|
8150
|
-
|
|
8151
|
-
|
|
8091
|
+
# 5. The suite. Run it ONCE -- see the single-run discipline below
|
|
8092
|
+
npm test
|
|
8152
8093
|
```
|
|
8153
8094
|
|
|
8154
|
-
|
|
8095
|
+
A non-zero exit from any of them is a stop, not a warning to note and push past.
|
|
8155
8096
|
|
|
8156
|
-
|
|
8097
|
+
## Single-run discipline
|
|
8157
8098
|
|
|
8158
|
-
|
|
8099
|
+
`npm test` runs the Bats suite with `--jobs 4`. Running it repeatedly to see
|
|
8100
|
+
whether a failure is "flaky" hides real ordering bugs and wastes minutes. Run
|
|
8101
|
+
it once. If it fails, read the failure and fix the cause; if the same test
|
|
8102
|
+
passes on a re-run without a code change, that instability is itself the bug
|
|
8103
|
+
and belongs in an issue, not in a retry.
|
|
8159
8104
|
|
|
8160
|
-
|
|
8161
|
-
|-------------|----------------|
|
|
8162
|
-
| A (cosmetic reword) | No version impact — refresh snapshot, continue |
|
|
8163
|
-
| B (new feature — integrate) | **Minor** version bump at minimum; new generator or extended generator |
|
|
8164
|
-
| C (new feature — not adopted) | No impact — note in registry |
|
|
8165
|
-
| D (deprecation) | **Minor** or **major** depending on user impact; add migration warning |
|
|
8166
|
-
| E (feature promoted to default) | **Minor**; simplify generator, keep fallback comment |
|
|
8167
|
-
| F (feature newly globally available) | **Minor**; may require new generator or new config path |
|
|
8105
|
+
## What this gate does not cover
|
|
8168
8106
|
|
|
8169
|
-
|
|
8107
|
+
- **Cross-platform.** The suite here runs on your machine only. Both
|
|
8108
|
+
`ubuntu-latest` and `macos-latest` must be green before a tag; that gate
|
|
8109
|
+
lives in `sop-release.md`, Phase 7, and is bound to the exact
|
|
8110
|
+
release commit.
|
|
8111
|
+
- **Required files.** The `required-files` CI job checks that every file the
|
|
8112
|
+
module template mandates is present and that the version is in sync across
|
|
8113
|
+
all four manifests. It is cheap and runs on every push.
|
|
8114
|
+
- **Provenance and SARIF.** Release-time concerns; see
|
|
8115
|
+
`sop-release.md`.
|
|
8170
8116
|
|
|
8171
|
-
|
|
8117
|
+
## Commit message
|
|
8172
8118
|
|
|
8173
|
-
|
|
8119
|
+
Conventional Commits, and no AI co-authorship trailer:
|
|
8174
8120
|
|
|
8175
|
-
```
|
|
8176
|
-
|
|
8121
|
+
```
|
|
8122
|
+
feat(scope): summary in the imperative
|
|
8123
|
+
fix(scope): summary in the imperative
|
|
8124
|
+
docs|refactor|test|chore(scope): ...
|
|
8177
8125
|
```
|
|
8178
8126
|
|
|
8179
|
-
|
|
8127
|
+
`pre-commit-check.py` suggests a type from the staged paths. It is a
|
|
8128
|
+
suggestion; the scope and the summary are yours.
|
|
8180
8129
|
|
|
8181
|
-
|
|
8130
|
+
## Related
|
|
8182
8131
|
|
|
8183
|
-
|
|
8184
|
-
|
|
8185
|
-
|
|
8132
|
+
- `sop-release.md` -- the pre-tag gate, including cross-platform CI
|
|
8133
|
+
- `sop-post-release-testing.md` -- what to verify after a release ships
|
|
8134
|
+
- `sop-release-verification.md` -- end-to-end smoke test of an installed build
|
|
8186
8135
|
|
|
8187
|
-
|
|
8136
|
+
---
|
|
8137
|
+
|
|
8138
|
+
## kb/procedures/sop-release-verification.md
|
|
8188
8139
|
|
|
8140
|
+
---
|
|
8141
|
+
title: "SOP: Release Verification"
|
|
8142
|
+
category: procedures
|
|
8143
|
+
service: ai-toolkit
|
|
8144
|
+
tags: [sop, verification, release, smoke-test, install, update, qa, provenance, sarif, dsh]
|
|
8145
|
+
version: "1.8.0"
|
|
8146
|
+
created: "2026-04-08"
|
|
8147
|
+
last_updated: "2026-09-01"
|
|
8148
|
+
description: "End-to-end smoke test after installing or updating @softspark/ai-toolkit. Verifies CLI, native Codex and GitHub Copilot surfaces, explicit DSH lifecycle, Claude app export, doctor, validation, tests, eject, provenance, SARIF, and per-skill permissions."
|
|
8189
8149
|
---
|
|
8190
8150
|
|
|
8191
|
-
|
|
8151
|
+
# SOP: Release Verification
|
|
8192
8152
|
|
|
8193
|
-
|
|
8153
|
+
End-to-end smoke test after installing or updating `@softspark/ai-toolkit`.
|
|
8154
|
+
Verifies all critical paths from the user's perspective.
|
|
8194
8155
|
|
|
8195
|
-
|
|
8196
|
-
|
|
8197
|
-
|
|
8198
|
-
|
|
8199
|
-
|
|
8156
|
+
**Use this SOP when:**
|
|
8157
|
+
- After `npm install -g @softspark/ai-toolkit@latest`
|
|
8158
|
+
- After `ai-toolkit update`
|
|
8159
|
+
- Before tagging a new version (`git tag`)
|
|
8160
|
+
- Before publishing to npm (`npm publish`)
|
|
8161
|
+
- As a smoke test in CI/CD
|
|
8200
8162
|
|
|
8201
|
-
**
|
|
8163
|
+
**Prerequisites:**
|
|
8164
|
+
- Node.js >= 18, Python 3, `bats`, git
|
|
8165
|
+
- `@softspark/ai-toolkit` installed globally
|
|
8166
|
+
|
|
8167
|
+
**Time:** 10-15 minutes (full), 2 minutes (quick checklist)
|
|
8202
8168
|
|
|
8203
8169
|
---
|
|
8204
8170
|
|
|
8205
|
-
##
|
|
8171
|
+
## Quick Checklist (TL;DR)
|
|
8206
8172
|
|
|
8207
|
-
The
|
|
8173
|
+
The 14 core commands below must pass. Releases that change DSH must also complete Phase 10.
|
|
8208
8174
|
|
|
8209
|
-
|
|
8175
|
+
```bash
|
|
8176
|
+
# Pre-commit (Phase 0)
|
|
8177
|
+
python3 scripts/generate_agents_md.py > AGENTS.md # 1. Regenerate AGENTS.md
|
|
8178
|
+
python3 scripts/generate_llms_txt.py > llms.txt # 2. Regenerate llms.txt
|
|
8179
|
+
python3 scripts/validate.py --strict # 3. Validation passed?
|
|
8180
|
+
npm test > /tmp/npm-test.log 2>&1 && grep -c '^ok ' /tmp/npm-test.log && ! grep -q '^not ok' /tmp/npm-test.log # 4. All tests passed? (single run, cached)
|
|
8210
8181
|
|
|
8211
|
-
|
|
8212
|
-
|
|
8213
|
-
|
|
8214
|
-
|
|
8215
|
-
|
|
8182
|
+
# Post-install verification (Phases 1-7)
|
|
8183
|
+
ai-toolkit --version # 5. Version OK?
|
|
8184
|
+
ai-toolkit status # 6. Status OK?
|
|
8185
|
+
ai-toolkit doctor # 7. Health check passed?
|
|
8186
|
+
ai-toolkit install --dry-run # 8. Global install OK?
|
|
8187
|
+
python3 scripts/audit_skills.py --ci # 9. Security audit clean?
|
|
8216
8188
|
|
|
8217
|
-
|
|
8189
|
+
# Supply-chain verification (Phase 8, v2.8.0+)
|
|
8190
|
+
python3 scripts/audit_skills.py --sarif | python3 -c "import json,sys; assert json.load(sys.stdin)['version']=='2.1.0'; print('SARIF OK')" # 10. SARIF 2.1.0 well-formed?
|
|
8191
|
+
python3 scripts/audit_skills.py --permissions | head -30 # 11. Broad-access skills reviewed?
|
|
8192
|
+
npm view @softspark/ai-toolkit@X.Y.Z --json | python3 -c "import json,sys; d=json.load(sys.stdin); assert d['dist']['attestations']['provenance']['predicateType']=='https://slsa.dev/provenance/v1'; print('PROVENANCE OK')" # 12. Provenance attested on npm?
|
|
8193
|
+
python3 scripts/claude_app.py verify # 13. Claude Chat/Cowork plugin contract valid?
|
|
8218
8194
|
|
|
8219
|
-
|
|
8220
|
-
|
|
8221
|
-
|
|
8195
|
+
# Deep-coverage verification (Phase 9, v3.0.0+)
|
|
8196
|
+
META="generate_agents_md.py|generate_llms_txt.py|generate_language_rules_skills.py"
|
|
8197
|
+
diff <(grep -oE 'scripts/generate_[a-z_]+\.py' kb/reference/supported-tools-registry.md | sort -u) <(ls scripts/generate_*.py | grep -vE "$META" | sort -u) && echo "OK: registry matches" # 14. Registry <-> generators drift?
|
|
8198
|
+
```
|
|
8222
8199
|
|
|
8223
|
-
|
|
8200
|
+
---
|
|
8224
8201
|
|
|
8225
|
-
|
|
8226
|
-
|------|-------|---------------|
|
|
8227
|
-
| `kb/procedures/maintenance-sop.md` | frontmatter `version:` | If SOP content changed |
|
|
8228
|
-
| `kb/reference/skills-catalog.md` | frontmatter `version:` | If skills added/removed |
|
|
8229
|
-
| `kb/reference/agents-catalog.md` | frontmatter `version:` | If agents added/removed |
|
|
8230
|
-
| `kb/reference/hooks-catalog.md` | frontmatter `version:` | If hooks changed |
|
|
8231
|
-
| `kb/reference/architecture-overview.md` | frontmatter `version:` | If architecture changed |
|
|
8232
|
-
| `kb/reference/distribution-model.md` | frontmatter `version:` | If install model changed |
|
|
8233
|
-
| `kb/reference/global-install-model.md` | frontmatter `version:` | If install model changed |
|
|
8202
|
+
## Phase 0: Pre-Commit & Pre-Push (2 min)
|
|
8234
8203
|
|
|
8235
|
-
|
|
8236
|
-
|
|
8204
|
+
Run these commands **before every commit and push to main**. CI validates
|
|
8205
|
+
counts but does NOT auto-regenerate — you must do it locally.
|
|
8237
8206
|
|
|
8238
|
-
|
|
8207
|
+
```bash
|
|
8208
|
+
# 1. Regenerate generated artifacts
|
|
8209
|
+
python3 scripts/generate_agents_md.py > AGENTS.md
|
|
8210
|
+
python3 scripts/generate_llms_txt.py > llms.txt
|
|
8211
|
+
python3 scripts/generate_llms_txt.py --full > llms-full.txt
|
|
8239
8212
|
|
|
8240
|
-
|
|
8241
|
-
|
|
8242
|
-
| `package.json` | `"description"` — skill/agent count |
|
|
8243
|
-
| `README.md` | Badge counts, "What You Get" table |
|
|
8244
|
-
| `app/ARCHITECTURE.md` | Section headings with counts |
|
|
8213
|
+
# 2. Validate everything (catches stale counts, missing assets)
|
|
8214
|
+
python3 scripts/validate.py --strict
|
|
8245
8215
|
|
|
8246
|
-
|
|
8247
|
-
|
|
8248
|
-
> If validation passes, counts and versions are correct.
|
|
8216
|
+
# 3. Security audit
|
|
8217
|
+
python3 scripts/audit_skills.py --ci
|
|
8249
8218
|
|
|
8250
|
-
|
|
8219
|
+
# 4. Run tests
|
|
8220
|
+
npm test
|
|
8251
8221
|
|
|
8252
|
-
|
|
8222
|
+
# 5. Stage and commit
|
|
8223
|
+
# AGENTS.md, GEMINI.md, and .github/copilot-instructions.md are generated and
|
|
8224
|
+
# gitignored — do NOT commit them (regenerated at prepublishOnly). Only the
|
|
8225
|
+
# doc artifacts llms.txt / llms-full.txt are committed.
|
|
8226
|
+
git add llms.txt llms-full.txt
|
|
8227
|
+
git add -p # stage your other changes
|
|
8228
|
+
git commit -m "feat: your change description"
|
|
8229
|
+
```
|
|
8230
|
+
|
|
8231
|
+
**Why local?** Branch protection on `main` requires PRs and status checks.
|
|
8232
|
+
CI cannot push directly to `main`, so generated artifacts must be committed
|
|
8233
|
+
by the developer as part of their PR.
|
|
8253
8234
|
|
|
8235
|
+
**One-liner (copy-paste):**
|
|
8254
8236
|
```bash
|
|
8255
|
-
|
|
8256
|
-
echo "Target: $VERSION"
|
|
8257
|
-
echo "manifest.json: $(python3 -c "import json; print(json.load(open('manifest.json'))['version'])")"
|
|
8258
|
-
echo "plugin.json: $(python3 -c "import json; print(json.load(open('app/.claude-plugin/plugin.json'))['version'])")"
|
|
8259
|
-
echo "package-lock.json: $(python3 -c "import json; print(json.load(open('package-lock.json'))['version'])")"
|
|
8237
|
+
python3 scripts/generate_agents_md.py > AGENTS.md && python3 scripts/generate_llms_txt.py > llms.txt && python3 scripts/generate_llms_txt.py --full > llms-full.txt && python3 scripts/validate.py --strict && python3 scripts/audit_skills.py --ci && npm test
|
|
8260
8238
|
```
|
|
8261
8239
|
|
|
8262
|
-
|
|
8240
|
+
---
|
|
8263
8241
|
|
|
8264
|
-
|
|
8242
|
+
## Phase 1: CLI & Version (1 min)
|
|
8265
8243
|
|
|
8266
8244
|
```bash
|
|
8267
|
-
|
|
8268
|
-
|
|
8245
|
+
ai-toolkit --version
|
|
8246
|
+
ai-toolkit --help
|
|
8247
|
+
which ai-toolkit
|
|
8269
8248
|
```
|
|
8270
8249
|
|
|
8271
|
-
**
|
|
8272
|
-
|
|
8273
|
-
|
|
8274
|
-
|
|
8275
|
-
|
|
8276
|
-
Do not run `--update` to make a red build green. The check fails because something
|
|
8277
|
-
users depend on disappeared; regenerating the manifest deletes the evidence, not
|
|
8278
|
-
the problem.
|
|
8250
|
+
**Verify:**
|
|
8251
|
+
- [ ] `--version` returns correct semver (e.g., `1.4.0`)
|
|
8252
|
+
- [ ] `--help` displays full command list without errors
|
|
8253
|
+
- [ ] `which` points to global npm bin path
|
|
8279
8254
|
|
|
8280
|
-
|
|
8255
|
+
---
|
|
8281
8256
|
|
|
8282
|
-
|
|
8257
|
+
## Phase 2: Global Install & Status (2 min)
|
|
8283
8258
|
|
|
8284
|
-
```
|
|
8285
|
-
|
|
8259
|
+
```bash
|
|
8260
|
+
ai-toolkit install --dry-run
|
|
8261
|
+
ai-toolkit status
|
|
8286
8262
|
```
|
|
8287
8263
|
|
|
8288
|
-
|
|
8289
|
-
|
|
8290
|
-
|
|
8291
|
-
|
|
8292
|
-
|
|
8293
|
-
|
|
8294
|
-
- **Never lower a threshold in the same change that something violates it.** Split
|
|
8295
|
-
the offending skill into `SKILL.md` + `reference/` first, prove it with
|
|
8296
|
-
`python3 scripts/check_split.py <skill> --before <pre-split SKILL.md>`, then
|
|
8297
|
-
tighten in a follow-up.
|
|
8298
|
-
- **Never raise a threshold to make a red build green.** A body over budget means
|
|
8299
|
-
detail that belongs in `reference/` is loading on every trigger match, including
|
|
8300
|
-
the accidental ones. Raising the number hides the cost, it does not remove it.
|
|
8264
|
+
**Verify `--dry-run`:**
|
|
8265
|
+
- [ ] Agents: 44
|
|
8266
|
+
- [ ] Skills: 108
|
|
8267
|
+
- [ ] Hooks merged into settings.json
|
|
8268
|
+
- [ ] "Other AI Tools" lists documented global targets (with `--editors`): aider, antigravity, augment, cline, codex, copilot, cursor, gemini, opencode, roo, windsurf. Scope varies: Codex uses `$CODEX_HOME` (default `~/.codex`) plus `$HOME/.agents/skills`; Copilot uses `$COPILOT_HOME` (default `~/.copilot`); cursor has only `~/.cursor/hooks.json`; antigravity has the `~/.gemini/*/skills` pointer. Cursor and Antigravity rules remain project-only.
|
|
8301
8269
|
|
|
8302
|
-
|
|
8303
|
-
|
|
8304
|
-
|
|
8270
|
+
**Verify `status`:**
|
|
8271
|
+
- [ ] Version matches expected
|
|
8272
|
+
- [ ] Profile: minimal/standard/strict
|
|
8273
|
+
- [ ] Modules: list of installed modules
|
|
8274
|
+
- [ ] Latest: up to date / update available
|
|
8305
8275
|
|
|
8306
8276
|
---
|
|
8307
8277
|
|
|
8308
|
-
## Phase 3:
|
|
8309
|
-
|
|
8310
|
-
Add entry at the top of `CHANGELOG.md` (after the header, before previous release):
|
|
8311
|
-
|
|
8312
|
-
```markdown
|
|
8313
|
-
## vX.Y.Z — Short Title (YYYY-MM-DD)
|
|
8314
|
-
|
|
8315
|
-
### Added
|
|
8316
|
-
- **Feature name** — description
|
|
8317
|
-
|
|
8318
|
-
### Changed
|
|
8319
|
-
- **What changed** — old behavior → new behavior
|
|
8320
|
-
|
|
8321
|
-
### Fixed
|
|
8322
|
-
- **Bug description** — what was broken and how it's fixed
|
|
8278
|
+
## Phase 3: Doctor Health Check (1 min)
|
|
8323
8279
|
|
|
8324
|
-
|
|
8325
|
-
-
|
|
8280
|
+
```bash
|
|
8281
|
+
ai-toolkit doctor
|
|
8326
8282
|
```
|
|
8327
8283
|
|
|
8328
|
-
**
|
|
8329
|
-
-
|
|
8330
|
-
-
|
|
8331
|
-
-
|
|
8332
|
-
-
|
|
8333
|
-
-
|
|
8334
|
-
-
|
|
8335
|
-
-
|
|
8336
|
-
|
|
8337
|
-
### Update README "What's New" section
|
|
8338
|
-
|
|
8339
|
-
**MANDATORY on every release.** Update the `## What's New in vX.Y.Z` section in `README.md`:
|
|
8340
|
-
|
|
8341
|
-
1. Change the heading version: `## What's New in vX.Y.Z`
|
|
8342
|
-
2. Replace bullet points with 3-5 highlights from this release
|
|
8343
|
-
3. **Keep only the latest version block.** Delete the previous `## What's New in vA.B.C` section(s). README is the shop window, not the archive — users see the current release, full history lives in `CHANGELOG.md`.
|
|
8344
|
-
4. Keep the `See [CHANGELOG.md](CHANGELOG.md) for full history.` link directly below the bullet list.
|
|
8284
|
+
**Expected sections (all OK):**
|
|
8285
|
+
- Environment: node, bash, python3, bats
|
|
8286
|
+
- Global Install: .claude exists, agents/skills symlinks (0 broken), settings.json hooks
|
|
8287
|
+
- Hook Scripts: all present and executable
|
|
8288
|
+
- Hook Configuration: 14 events registered
|
|
8289
|
+
- Generated Artifacts: AGENTS.md, llms.txt, llms-full.txt
|
|
8290
|
+
- Planned Assets: plugin.json, benchmarks, plugin packs
|
|
8291
|
+
- Benchmark Freshness: < 30 days
|
|
8292
|
+
- Stale Rules: all healthy
|
|
8345
8293
|
|
|
8346
|
-
|
|
8347
|
-
|
|
8348
|
-
|
|
8294
|
+
**Verify:**
|
|
8295
|
+
- [ ] `Errors: 0 | Warnings: 0`
|
|
8296
|
+
- [ ] `HEALTH CHECK PASSED`
|
|
8349
8297
|
|
|
8350
|
-
|
|
8298
|
+
If doctor detects problems: `ai-toolkit doctor --fix` auto-repairs
|
|
8299
|
+
(broken symlinks, non-executable hooks, missing scripts, missing llms-full.txt).
|
|
8351
8300
|
|
|
8352
8301
|
---
|
|
8353
8302
|
|
|
8354
|
-
## Phase 4:
|
|
8355
|
-
|
|
8356
|
-
Use the npm scripts, not the generators directly:
|
|
8303
|
+
## Phase 4: Local Install (2 min)
|
|
8357
8304
|
|
|
8358
8305
|
```bash
|
|
8359
|
-
|
|
8360
|
-
|
|
8306
|
+
mkdir -p /tmp/ai-toolkit-verify && cd /tmp/ai-toolkit-verify
|
|
8307
|
+
git init -q
|
|
8308
|
+
ai-toolkit install --local --editors all --dry-run
|
|
8309
|
+
cd - && rm -rf /tmp/ai-toolkit-verify
|
|
8361
8310
|
```
|
|
8362
8311
|
|
|
8363
|
-
|
|
8364
|
-
|
|
8365
|
-
|
|
8366
|
-
|
|
8367
|
-
|
|
8312
|
+
**Verify "Project-local" section:**
|
|
8313
|
+
- [ ] Would create: CLAUDE.md
|
|
8314
|
+
- [ ] Would create: .claude/settings.local.json
|
|
8315
|
+
- [ ] Would inject: .claude/constitution.md
|
|
8316
|
+
- [ ] Editors: all 11 listed (copilot, cursor, windsurf, cline, roo, aider, augment, antigravity, codex, gemini, opencode)
|
|
8317
|
+
- [ ] Would generate configs for each editor (legacy + directory-based)
|
|
8318
|
+
- [ ] Would install: .git/hooks/pre-commit
|
|
8319
|
+
- [ ] Would inject language rules (auto-detected)
|
|
8368
8320
|
|
|
8321
|
+
**Also test auto-detect (no --editors flag):**
|
|
8369
8322
|
```bash
|
|
8370
|
-
|
|
8323
|
+
ai-toolkit install --local --dry-run
|
|
8324
|
+
# → Editors: none (empty project has no existing configs)
|
|
8371
8325
|
```
|
|
8372
8326
|
|
|
8373
|
-
If no diff, the artifacts are already current. If there is a diff, stage them.
|
|
8374
|
-
|
|
8375
8327
|
---
|
|
8376
8328
|
|
|
8377
|
-
## Phase 5:
|
|
8378
|
-
|
|
8379
|
-
Run the full quality gate:
|
|
8329
|
+
## Phase 5: Validation & Security Audit (3 min)
|
|
8380
8330
|
|
|
8381
8331
|
```bash
|
|
8382
8332
|
python3 scripts/validate.py --strict
|
|
8383
8333
|
python3 scripts/audit_skills.py --ci
|
|
8384
|
-
|
|
8385
|
-
python3 scripts/audit_skills.py --permissions # review Bash/Write/Edit footprint
|
|
8334
|
+
```
|
|
8386
8335
|
|
|
8387
|
-
|
|
8388
|
-
|
|
8389
|
-
|
|
8390
|
-
|
|
8391
|
-
|
|
8336
|
+
**Verify validate.py:**
|
|
8337
|
+
- [ ] Agents: 44, Skills: 108, Tests: exactly the current README badge count
|
|
8338
|
+
- [ ] Hook events: 14, Hook scripts: >= 30
|
|
8339
|
+
- [ ] Plugin packs >= 10, KB documents >= 20
|
|
8340
|
+
- [ ] `Errors: 0 | Warnings: 0` → `VALIDATION PASSED`
|
|
8392
8341
|
|
|
8393
|
-
|
|
8394
|
-
|
|
8395
|
-
|
|
8396
|
-
|
|
8397
|
-
<(ls scripts/generate_*.py | grep -vE "$META" | sort -u) \
|
|
8398
|
-
&& echo "OK: registry matches filesystem" \
|
|
8399
|
-
|| { echo "DRIFT: update supported-tools-registry.md before tagging"; exit 1; }
|
|
8342
|
+
**Verify audit_skills.py:**
|
|
8343
|
+
- [ ] `HIGH: 0` (MUST be zero — CI fails otherwise)
|
|
8344
|
+
- [ ] `WARN: 0`
|
|
8345
|
+
- [ ] `INFO: N` (acceptable — broad-access skills: orchestrate, swarm, workflow)
|
|
8400
8346
|
|
|
8401
|
-
|
|
8402
|
-
# "npm package KB files match the tracked release set" compares `git ls-files
|
|
8403
|
-
# kb` (the index) against what `npm pack` sees (the working tree), so an
|
|
8404
|
-
# unstaged addition reads as "extra" and an unstaged deletion as "missing".
|
|
8405
|
-
# Phase 6 stages, and it runs after this one, so the ordering fails the test
|
|
8406
|
-
# for any release that touches the KB. Staging early costs nothing.
|
|
8407
|
-
git status --porcelain kb/ | grep -qE '^(\?\?| D|\?M)' && git add -A kb/
|
|
8347
|
+
---
|
|
8408
8348
|
|
|
8409
|
-
|
|
8410
|
-
|
|
8411
|
-
|
|
8349
|
+
## Phase 6: Tests (3-5 min)
|
|
8350
|
+
|
|
8351
|
+
```bash
|
|
8352
|
+
# Run ONCE, capture to file, then parse. Full suite is 669+ bats cases —
|
|
8353
|
+
# re-running it per check (tail / grep ok / grep not ok piped separately)
|
|
8354
|
+
# wastes minutes every release. Always cache the output.
|
|
8412
8355
|
npm test > /tmp/npm-test.log 2>&1
|
|
8356
|
+
exit=$?
|
|
8413
8357
|
tail -3 /tmp/npm-test.log
|
|
8414
|
-
echo "ok:
|
|
8358
|
+
echo "ok: $(grep -c '^ok ' /tmp/npm-test.log)"
|
|
8359
|
+
echo "not ok: $(grep -c '^not ok' /tmp/npm-test.log)"
|
|
8360
|
+
echo "exit: $exit"
|
|
8415
8361
|
```
|
|
8416
8362
|
|
|
8417
|
-
**
|
|
8418
|
-
-
|
|
8419
|
-
-
|
|
8420
|
-
-
|
|
8421
|
-
-
|
|
8422
|
-
-
|
|
8423
|
-
|
|
8424
|
-
- `npm test`: `1..N` with zero `not ok` (read from the cached `/tmp/npm-test.log`, do not rerun)
|
|
8425
|
-
|
|
8426
|
-
> **Why this matters (v4.5.1 postmortem):** `publish.yml` runs only `validate.py` + `npm test`, so it published v4.5.0 even though the `main` CI `ShellCheck hooks` job was red on two `SC2034` warnings in a new hook. The publish workflow does **not** depend on the CI workflow. Until that is fixed, ShellCheck is a manual pre-tag gate — run it here every time.
|
|
8363
|
+
**Verify:**
|
|
8364
|
+
- [ ] `exit == 0`
|
|
8365
|
+
- [ ] `ok == expected test count` (e.g., 945 on v3.0.0)
|
|
8366
|
+
- [ ] `not ok == 0`
|
|
8367
|
+
- [ ] Bats runs tests in parallel (4 jobs)
|
|
8368
|
+
- [ ] Groups: agents, autodetect, cli, generators, guards, hooks, inject,
|
|
8369
|
+
install, kb, mcp, readme, profiles, uninstall, validate
|
|
8427
8370
|
|
|
8428
|
-
**
|
|
8371
|
+
**Anti-pattern — do NOT do this:**
|
|
8429
8372
|
```bash
|
|
8430
|
-
|
|
8373
|
+
# Runs the full suite THREE times. Adds 1-3 min and pressures CI capacity.
|
|
8374
|
+
npm test 2>&1 | tail -3
|
|
8375
|
+
npm test 2>&1 | grep -c '^ok '
|
|
8376
|
+
npm test 2>&1 | grep -c '^not ok'
|
|
8431
8377
|
```
|
|
8432
8378
|
|
|
8433
|
-
**
|
|
8434
|
-
-
|
|
8435
|
-
-
|
|
8436
|
-
-
|
|
8379
|
+
**Key test areas:**
|
|
8380
|
+
- Guards: rm -rf, DROP TABLE, git push --force blocked
|
|
8381
|
+
- Install: idempotent, profiles, --only/--skip, orphan cleanup
|
|
8382
|
+
- Eject: real files (not symlinks), inlined rules
|
|
8383
|
+
- Uninstall: removes toolkit, preserves user content
|
|
8437
8384
|
|
|
8438
|
-
|
|
8385
|
+
---
|
|
8439
8386
|
|
|
8440
|
-
|
|
8441
|
-
carries it, and every part of that is mechanically checkable.
|
|
8387
|
+
## Phase 7: Eject (1 min)
|
|
8442
8388
|
|
|
8443
8389
|
```bash
|
|
8444
|
-
|
|
8445
|
-
|
|
8390
|
+
mkdir -p /tmp/ai-toolkit-eject-test
|
|
8391
|
+
cd /tmp/ai-toolkit-eject-test
|
|
8392
|
+
ai-toolkit eject
|
|
8393
|
+
cd - && rm -rf /tmp/ai-toolkit-eject-test
|
|
8446
8394
|
```
|
|
8447
8395
|
|
|
8448
|
-
|
|
8396
|
+
**Verify:**
|
|
8397
|
+
- [ ] Agents copied as real files (not symlinks)
|
|
8398
|
+
- [ ] Skills copied as real directories
|
|
8399
|
+
- [ ] Rules inlined into CLAUDE.md
|
|
8400
|
+
- [ ] constitution.md and ARCHITECTURE.md copied
|
|
8401
|
+
- [ ] `output-styles/` directory present (v2.7.1+)
|
|
8449
8402
|
|
|
8450
|
-
|
|
8451
|
-
|---|---|
|
|
8452
|
-
| Every shipped source file carries an SPDX header | A new `.py`/`.sh`/`.js`/`.bats` file was added without one — the common case, and the reason this is a test rather than a habit |
|
|
8453
|
-
| Headers name Apache-2.0 and nothing else | A file was copied in from an MIT/GPL source with its own header intact |
|
|
8454
|
-
| **No** markdown file carries an SPDX header | Someone "helpfully" ran the header script over `app/skills/` — headers there sit above parsed frontmatter and bill every session for it |
|
|
8455
|
-
| `LICENSE` is the complete Apache 2.0 text | The file was truncated or replaced with a summary |
|
|
8456
|
-
| `NOTICE` carries attribution, the source URL, §4(d) and the MIT-era notice | The attribution mechanism was gutted |
|
|
8457
|
-
| `LICENSE` **and** `NOTICE` ship in the npm package | `package.json` `files` lost an entry — a NOTICE that never reaches the consumer cannot satisfy §4(d) |
|
|
8458
|
-
| Every manifest declaring a licence declares Apache-2.0 | `package.json`, `manifest.json`, `plugin.json` and `package-lock.json` drifted apart |
|
|
8403
|
+
---
|
|
8459
8404
|
|
|
8460
|
-
|
|
8461
|
-
never before it. Short SPDX form:
|
|
8405
|
+
## Phase 8: Supply-Chain Verification (2 min, v2.8.0+)
|
|
8462
8406
|
|
|
8463
|
-
|
|
8464
|
-
# SPDX-License-Identifier: Apache-2.0
|
|
8465
|
-
# Copyright 2024-2026 Lukasz Krzemien (biuro@softspark.eu)
|
|
8466
|
-
# Source: https://github.com/softspark/ai-toolkit
|
|
8467
|
-
```
|
|
8407
|
+
These checks enforce the v2.8.0 security standard on a freshly-published release. Run AFTER the `publish.yml` workflow completes on the tag.
|
|
8468
8408
|
|
|
8469
|
-
|
|
8470
|
-
exclusion: [Licensing](../reference/licensing.md).
|
|
8409
|
+
### 8.1 Provenance attestation on npm
|
|
8471
8410
|
|
|
8472
|
-
|
|
8473
|
-
|
|
8474
|
-
|
|
8475
|
-
|
|
8476
|
-
|
|
8411
|
+
```bash
|
|
8412
|
+
VERSION="X.Y.Z" # the tag just published
|
|
8413
|
+
npm view "@softspark/ai-toolkit@${VERSION}" --json \
|
|
8414
|
+
| python3 -c "import json, sys; d=json.load(sys.stdin); att=d['dist'].get('attestations', {}); assert att.get('provenance', {}).get('predicateType') == 'https://slsa.dev/provenance/v1', f'NO PROVENANCE for {d[\"version\"]}'; print(f'PROVENANCE OK: {att[\"url\"]}')"
|
|
8415
|
+
```
|
|
8477
8416
|
|
|
8478
|
-
|
|
8417
|
+
**Verify:**
|
|
8418
|
+
- [ ] Exit 0 and prints `PROVENANCE OK: https://registry.npmjs.org/...`
|
|
8419
|
+
- [ ] `https://www.npmjs.com/package/@softspark/ai-toolkit/v/${VERSION}` shows the green "Provenance" badge
|
|
8479
8420
|
|
|
8480
|
-
|
|
8421
|
+
**If provenance is missing:** the `publish.yml` ran without `id-token: write` or `--provenance`. Restore them and cut a patch release — a silently unsigned publish is a regression against the v2.8.0 standard.
|
|
8481
8422
|
|
|
8482
|
-
|
|
8423
|
+
### 8.2 Audit SARIF output (for GHAS ingest)
|
|
8483
8424
|
|
|
8484
8425
|
```bash
|
|
8485
|
-
|
|
8426
|
+
python3 scripts/audit_skills.py --sarif > /tmp/audit.sarif
|
|
8427
|
+
python3 -c "import json; d=json.load(open('/tmp/audit.sarif')); assert d['version']=='2.1.0' and d['runs'][0]['tool']['driver']['name']=='ai-toolkit-audit-skills'; print(f'SARIF OK: {len(d[\"runs\"][0][\"results\"])} results across {len(d[\"runs\"][0][\"tool\"][\"driver\"][\"rules\"])} rules')"
|
|
8486
8428
|
```
|
|
8487
8429
|
|
|
8488
|
-
|
|
8489
|
-
- [ ]
|
|
8430
|
+
**Verify:**
|
|
8431
|
+
- [ ] Valid SARIF 2.1.0
|
|
8432
|
+
- [ ] In the publishing repo, the CI job uploads `audit.sarif` via `github/codeql-action/upload-sarif@v3` so findings appear in the Security tab
|
|
8490
8433
|
|
|
8491
|
-
|
|
8434
|
+
### 8.3 Per-skill permissions report
|
|
8492
8435
|
|
|
8493
8436
|
```bash
|
|
8494
|
-
|
|
8495
|
-
# (schema_version 1: entries live under the .rules / .hooks key):
|
|
8496
|
-
jq '.rules | to_entries | map(select(.value.url != null and (.value.sha256 // "" | length) == 0))' ~/.softspark/ai-toolkit/rules/sources.json
|
|
8497
|
-
jq '.hooks | to_entries | map(select(.value.url != null and (.value.sha256 // "" | length) == 0))' ~/.softspark/ai-toolkit/hooks/external/sources.json
|
|
8437
|
+
python3 scripts/audit_skills.py --permissions | head -40
|
|
8498
8438
|
```
|
|
8499
8439
|
|
|
8500
|
-
|
|
8501
|
-
- [ ]
|
|
8440
|
+
**Verify:**
|
|
8441
|
+
- [ ] Bash skill count has NOT jumped unexpectedly since the previous release
|
|
8442
|
+
- [ ] Any newly added entry under `Skills with Bash + Write + Edit` matches a CHANGELOG bullet that justifies the broad scope
|
|
8443
|
+
- [ ] JSON form (`--permissions --json`) is available for automated drift dashboards
|
|
8502
8444
|
|
|
8503
|
-
|
|
8445
|
+
### 8.4 URL-sourced rules/hooks are checksum-pinned
|
|
8446
|
+
|
|
8447
|
+
`sources.json` is an envelope of the form `{"schema_version": 1, "rules"|"hooks": {...}}`, so the jq filter must pick the nested map before piping into the pin assertion. The URL-count in the success message ignores local `path:`-only entries — only entries with a `url:` field are pinned and counted.
|
|
8504
8448
|
|
|
8505
8449
|
```bash
|
|
8506
|
-
|
|
8450
|
+
jq '.rules // .hooks // {}' ~/.softspark/ai-toolkit/rules/sources.json 2>/dev/null \
|
|
8451
|
+
| python3 -c "import json, sys; d=json.load(sys.stdin) or {}; bad=[n for n,v in d.items() if v.get('url') and not v.get('sha256')]; assert not bad, f'UNPINNED: {bad}'; url_n=sum(1 for v in d.values() if v.get('url')); print(f'RULE PIN OK: {url_n} URL rules, all with sha256')"
|
|
8452
|
+
jq '.hooks // {}' ~/.softspark/ai-toolkit/hooks/external/sources.json 2>/dev/null \
|
|
8453
|
+
| python3 -c "import json, sys; d=json.load(sys.stdin) or {}; bad=[n for n,v in d.items() if v.get('url') and not v.get('sha256')]; assert not bad, f'UNPINNED: {bad}'; url_n=sum(1 for v in d.values() if v.get('url')); print(f'HOOK PIN OK: {url_n} URL hooks, all with sha256')"
|
|
8507
8454
|
```
|
|
8508
8455
|
|
|
8509
|
-
|
|
8510
|
-
- [ ]
|
|
8456
|
+
**Verify:**
|
|
8457
|
+
- [ ] Both commands print `... PIN OK`
|
|
8458
|
+
- [ ] If any entry is unpinned, the `register_url_source()` call missed passing `content=` — fix the call-site and retag
|
|
8511
8459
|
|
|
8512
|
-
|
|
8460
|
+
### 8.5 Strict-pin smoke test (optional but recommended)
|
|
8513
8461
|
|
|
8514
8462
|
```bash
|
|
8515
8463
|
AI_TOOLKIT_STRICT_PIN=1 ai-toolkit update --dry-run
|
|
8516
8464
|
```
|
|
8517
8465
|
|
|
8466
|
+
**Verify:**
|
|
8518
8467
|
- [ ] Exit 0, no `CHECKSUM CHANGED` line
|
|
8519
|
-
- [ ]
|
|
8468
|
+
- [ ] If a checksum change was intentional (e.g. upstream rule update), document it in the CHANGELOG entry before tagging
|
|
8520
8469
|
|
|
8521
8470
|
---
|
|
8522
8471
|
|
|
8523
|
-
## Phase
|
|
8472
|
+
## Phase 9: Deep-Coverage Checks (v3.0.0+)
|
|
8524
8473
|
|
|
8525
|
-
|
|
8474
|
+
These verify the native-surface generators shipped in v3.0.0 actually emit the right files for the right profiles, and that the tool registry stays in sync with shipped generators.
|
|
8475
|
+
|
|
8476
|
+
> **Safety warning — HOME-scoped writes:** Running `--profile full` with `augment` in the editor list writes to `$HOME/.augment/settings.json` (Augment stores hooks under HOME, not per-project). Use `--dry-run` for verification unless you intend to carry ai-toolkit hook entries on this machine. The generator is marker-safe (only rewrites its own `_source: ai-toolkit` entries) but is still a side-effect.
|
|
8477
|
+
|
|
8478
|
+
### 9.1 `--profile full` emits every native surface
|
|
8526
8479
|
|
|
8527
8480
|
```bash
|
|
8528
|
-
|
|
8529
|
-
|
|
8530
|
-
|
|
8531
|
-
|
|
8532
|
-
# NOTE: AGENTS.md, GEMINI.md, and .github/copilot-instructions.md are generated
|
|
8533
|
-
# editor configs and are gitignored — do NOT commit them. `prepublishOnly` runs
|
|
8534
|
-
# `npm run generate:all`, so the shipped package (which lists AGENTS.md in
|
|
8535
|
-
# package.json `files`) gets a fresh copy at publish time.
|
|
8536
|
-
git add -p # review and stage any other changes
|
|
8481
|
+
D=/tmp/aitk-profile-full-${RANDOM} && mkdir -p "$D" && cd "$D" && git init -q
|
|
8482
|
+
ai-toolkit install --local --editors cursor,windsurf,gemini,augment,codex,copilot \
|
|
8483
|
+
--profile full --dry-run 2>&1 \
|
|
8484
|
+
| grep -E "\\.cursor/(hooks\\.json|agents)|\\.devin/hooks\\.v1\\.json|\\.gemini/(settings\\.json|commands)|\\.augment/(agents|commands)|\\.codex/(agents|hooks)|\\.agents/skills|\\.github/(agents|skills|instructions|prompts|hooks)"
|
|
8537
8485
|
```
|
|
8538
8486
|
|
|
8539
|
-
|
|
8487
|
+
**Verify** — at least the following lines appear:
|
|
8488
|
+
- [ ] `.cursor/hooks.json` and `.cursor/agents/`
|
|
8489
|
+
- [ ] `.devin/hooks.v1.json`
|
|
8490
|
+
- [ ] `.gemini/settings.json` hooks AND `.gemini/commands/`
|
|
8491
|
+
- [ ] `.augment/agents/` + `.augment/commands/` + `$HOME/.augment/settings.json`
|
|
8492
|
+
- [ ] `.codex/hooks.json` + `.codex/hooks/` + `.codex/agents/`
|
|
8493
|
+
- [ ] `.agents/skills/` (Codex native discovery path)
|
|
8494
|
+
- [ ] `.github/agents/` + `.github/skills/` + `.github/instructions/` + `.github/prompts/` + `.github/hooks/`
|
|
8495
|
+
|
|
8496
|
+
### 9.2 Codex skills are profile-independent
|
|
8540
8497
|
|
|
8541
8498
|
```bash
|
|
8542
|
-
|
|
8499
|
+
D=/tmp/aitk-codex-skills-${RANDOM} && mkdir -p "$D" && cd "$D" && git init -q
|
|
8500
|
+
ai-toolkit install --local --editors codex --profile minimal --dry-run 2>&1 \
|
|
8501
|
+
| grep -q "Would generate: .agents/skills" && echo "OK: Codex skills emit at minimal"
|
|
8502
|
+
ai-toolkit install --local --editors codex --profile standard --codex-skills --dry-run 2>&1 \
|
|
8503
|
+
| grep -q "Would refresh: .agents/skills" && echo "OK: legacy flag explicitly refreshes the same catalog"
|
|
8543
8504
|
```
|
|
8544
8505
|
|
|
8545
|
-
|
|
8506
|
+
**Verify:**
|
|
8507
|
+
- [ ] Selecting Codex emits all skills under `.agents/skills/` at every profile
|
|
8508
|
+
- [ ] `--codex-skills` remains an explicit refresh alias; it is not required for installation
|
|
8509
|
+
- [ ] No profile emits `.codex/skills/`
|
|
8546
8510
|
|
|
8547
|
-
|
|
8511
|
+
### 9.3 Breaking-change surfaces land on `--profile standard`
|
|
8512
|
+
|
|
8513
|
+
The standard profile must include:
|
|
8514
|
+
- Copilot scoped instructions, prompts, custom agents, portable skills, and hooks
|
|
8515
|
+
- Gemini hooks (`.gemini/settings.json`)
|
|
8548
8516
|
|
|
8549
8517
|
```bash
|
|
8550
|
-
|
|
8551
|
-
|
|
8518
|
+
D=/tmp/aitk-breaking-${RANDOM} && mkdir -p "$D" && cd "$D" && git init -q
|
|
8519
|
+
ai-toolkit install --local --editors copilot,gemini --profile standard --dry-run 2>&1 \
|
|
8520
|
+
| tee /tmp/aitk-breaking.log
|
|
8521
|
+
grep -q "\\.github/instructions/" /tmp/aitk-breaking.log && echo "OK: Copilot dir layout at standard"
|
|
8522
|
+
grep -q "\\.github/agents/.*\\.github/skills/" /tmp/aitk-breaking.log && echo "OK: Copilot agents and skills at standard"
|
|
8523
|
+
grep -q "\\.github/hooks/" /tmp/aitk-breaking.log && echo "OK: Copilot hooks at standard"
|
|
8524
|
+
grep -q "\\.gemini/settings\\.json hooks" /tmp/aitk-breaking.log && echo "OK: Gemini hooks at standard"
|
|
8525
|
+
```
|
|
8552
8526
|
|
|
8553
|
-
|
|
8554
|
-
# asynchronous, so retry at most five times and log every attempt.
|
|
8555
|
-
RUN_ID=""
|
|
8556
|
-
for ATTEMPT in 1 2 3 4 5; do
|
|
8557
|
-
RUN_ID=$(gh run list --workflow ci.yml --event push --commit "$RELEASE_SHA" \
|
|
8558
|
-
--limit 1 --json databaseId --jq '.[0].databaseId')
|
|
8559
|
-
[ -n "$RUN_ID" ] && break
|
|
8560
|
-
echo "Waiting for CI run registration ($ATTEMPT/5)" >&2
|
|
8561
|
-
[ "$ATTEMPT" -eq 5 ] || sleep 60
|
|
8562
|
-
done
|
|
8563
|
-
test -n "$RUN_ID" || { echo "FAIL: CI run not found for release commit"; exit 1; }
|
|
8564
|
-
gh run watch "$RUN_ID" --exit-status \
|
|
8565
|
-
|| { echo "FAIL: release commit CI failed"; exit 1; }
|
|
8527
|
+
**Verify all four lines print `OK:`**. If any is missing, native Copilot or Gemini coverage has regressed.
|
|
8566
8528
|
|
|
8567
|
-
|
|
8568
|
-
# the tag so a green publish job cannot bypass a red macOS test.
|
|
8569
|
-
for OS in ubuntu-latest macos-latest; do
|
|
8570
|
-
gh run view "$RUN_ID" --json jobs \
|
|
8571
|
-
--jq ".jobs[] | select(.name == \"Bats test suite ($OS)\") | .conclusion" \
|
|
8572
|
-
| grep -qx success || { echo "FAIL: $OS test job not green"; exit 1; }
|
|
8573
|
-
done
|
|
8529
|
+
### 9.4 Install is idempotent
|
|
8574
8530
|
|
|
8575
|
-
|
|
8576
|
-
|
|
8577
|
-
|
|
8578
|
-
|
|
8579
|
-
|
|
8580
|
-
|
|
8531
|
+
```bash
|
|
8532
|
+
D=/tmp/aitk-idem-${RANDOM} && mkdir -p "$D" && cd "$D" && git init -q
|
|
8533
|
+
# Sort file paths before hashing — find traversal order follows inode order,
|
|
8534
|
+
# which can shift between runs even when content is byte-identical, producing
|
|
8535
|
+
# false FAIL signals.
|
|
8536
|
+
ai-toolkit install --local --editors cursor,gemini --profile full >/dev/null 2>&1
|
|
8537
|
+
SHA1=$(find .cursor .gemini -type f -print0 | LC_ALL=C sort -z | xargs -0 shasum | shasum | awk '{print $1}')
|
|
8538
|
+
ai-toolkit install --local --editors cursor,gemini --profile full >/dev/null 2>&1
|
|
8539
|
+
SHA2=$(find .cursor .gemini -type f -print0 | LC_ALL=C sort -z | xargs -0 shasum | shasum | awk '{print $1}')
|
|
8540
|
+
[ "$SHA1" = "$SHA2" ] && echo "OK: idempotent" || echo "FAIL: install is not idempotent"
|
|
8541
|
+
```
|
|
8581
8542
|
|
|
8582
|
-
|
|
8543
|
+
**Verify:** prints `OK: idempotent`. A second run must produce byte-identical files in every managed path.
|
|
8583
8544
|
|
|
8584
|
-
|
|
8585
|
-
# have caught a real broken release.
|
|
8586
|
-
test "$(git rev-parse vX.Y.Z)" = "$RELEASE_SHA" \
|
|
8587
|
-
|| { echo "FAIL: tag is not on the tested release commit"; exit 1; }
|
|
8588
|
-
git show --no-patch --format=%s vX.Y.Z | grep -qx "chore: release vX.Y.Z" \
|
|
8589
|
-
|| { echo "FAIL: tag is not on the chore: release commit"; exit 1; }
|
|
8545
|
+
### 9.5 Live-install JSON outputs parse
|
|
8590
8546
|
|
|
8591
|
-
|
|
8592
|
-
|
|
8547
|
+
The bats suite validates JSON shape at generation time. This re-checks that what actually landed on disk after a live install parses without errors.
|
|
8548
|
+
|
|
8549
|
+
```bash
|
|
8550
|
+
D=/tmp/aitk-json-${RANDOM} && mkdir -p "$D" && cd "$D" && git init -q
|
|
8551
|
+
ai-toolkit install --local --editors cursor,windsurf,gemini,augment,codex,copilot --profile full >/dev/null 2>&1
|
|
8552
|
+
for f in .cursor/hooks.json .devin/hooks.v1.json .gemini/settings.json .codex/hooks.json .github/hooks/ai-toolkit.json "$HOME/.augment/settings.json"; do
|
|
8553
|
+
[ -f "$f" ] && python3 -c "import json; json.load(open('$f'))" && echo "OK: $f"
|
|
8554
|
+
done
|
|
8593
8555
|
```
|
|
8594
8556
|
|
|
8595
|
-
**
|
|
8596
|
-
workflow runs only on Ubuntu and can publish while the separate macOS matrix job
|
|
8597
|
-
is red. `v4.30.2` exposed this with a macOS-only DSH timing failure. A release
|
|
8598
|
-
commit must therefore pass the complete Ubuntu/macOS branch workflow before its
|
|
8599
|
-
tag exists; a successful publish workflow is not a substitute for green CI.
|
|
8557
|
+
**Verify:** each emitted file prints `OK: <path>`. Any `json.decoder.JSONDecodeError` means the merge logic corrupted the output.
|
|
8600
8558
|
|
|
8601
|
-
|
|
8602
|
-
that contained only a KB document and still carried `package.json` version
|
|
8603
|
-
`4.18.0`; the actual release sat in the commit above it under a recycled
|
|
8604
|
-
`fix:` message. `publish.yml` fired, tried to publish a version already on
|
|
8605
|
-
npm, and failed. Nothing on npm, a tag pointing at the wrong tree, and the
|
|
8606
|
-
only way out was rewriting a pushed commit. Both assertions above catch this
|
|
8607
|
-
in under a second. Run them.
|
|
8559
|
+
### 9.6 Registry / generator drift check
|
|
8608
8560
|
|
|
8609
|
-
|
|
8610
|
-
suppresses tag-triggered workflow runs when many tags arrive in a single push
|
|
8611
|
-
— the workflow silently does not fire and nothing publishes. Push the single
|
|
8612
|
-
release tag by its full ref, as above. (Sibling evidence: this is exactly how
|
|
8613
|
-
rag-mcp's `1.0.3` image build was skipped, when a `--tags` push carried 37
|
|
8614
|
-
tags at once.)
|
|
8561
|
+
`kb/reference/supported-tools-registry.md` should enumerate every per-editor `scripts/generate_*.py` we ship. Meta-generators (`generate_agents_md.py`, `generate_llms_txt.py`) are excluded — they produce docs/artifacts, not editor configs.
|
|
8615
8562
|
|
|
8616
|
-
|
|
8617
|
-
|
|
8618
|
-
|
|
8619
|
-
|
|
8563
|
+
```bash
|
|
8564
|
+
META="generate_agents_md.py|generate_llms_txt.py|generate_language_rules_skills.py"
|
|
8565
|
+
REG=$(grep -oE 'scripts/generate_[a-z_]+\.py' kb/reference/supported-tools-registry.md | sort -u)
|
|
8566
|
+
FS=$(ls scripts/generate_*.py | grep -vE "$META" | sort -u)
|
|
8567
|
+
diff <(echo "$REG") <(echo "$FS") && echo "OK: registry matches filesystem" || echo "DRIFT: update supported-tools-registry.md"
|
|
8568
|
+
```
|
|
8620
8569
|
|
|
8621
|
-
**
|
|
8570
|
+
**Verify:** prints `OK: registry matches filesystem`. If not, add the missing rows to the registry before tagging the next release.
|
|
8622
8571
|
|
|
8623
|
-
|
|
8624
|
-
|
|
8572
|
+
### 9.7 Claude Chat / Cowork plugin validates and exports
|
|
8573
|
+
|
|
8574
|
+
```bash
|
|
8575
|
+
python3 scripts/claude_app.py verify
|
|
8576
|
+
D=/tmp/aitk-claude-app-${RANDOM}
|
|
8577
|
+
python3 scripts/claude_app.py export --output "$D.zip" --no-custom-rules
|
|
8578
|
+
python3 -c "import zipfile; z=zipfile.ZipFile('$D.zip'); assert '.claude-plugin/plugin.json' in z.namelist(); assert 'claude-app/skills/ai-toolkit-rules/SKILL.md' in z.namelist(); print('OK: Claude app archive')"
|
|
8579
|
+
```
|
|
8580
|
+
|
|
8581
|
+
**Verify:** the official validator exits 0; the archive contains the manifest,
|
|
8582
|
+
app-native rules skill, bundled agents/skills, and plugin-relative Cowork hooks.
|
|
8625
8583
|
|
|
8626
8584
|
---
|
|
8627
8585
|
|
|
8628
|
-
##
|
|
8586
|
+
## Phase 10: Explicit DSH Qualification (v4.30.0+)
|
|
8629
8587
|
|
|
8630
|
-
|
|
8588
|
+
Run this phase whenever the release changes the `dsh` target, package pins, preset lifecycle, or DSH compatibility documentation. Use a new task-specific `DSH_HOME`; never replace `HOME` or reuse a regular profile.
|
|
8589
|
+
|
|
8590
|
+
Prerequisites: DSH `0.1.1-rc.2`, pnpm `>=11.7.0,<12.0.0`, Codex logged in through ChatGPT, Claude Code logged in natively, and GitHub Copilot CLI logged in natively. Do not supply provider API keys.
|
|
8631
8591
|
|
|
8632
8592
|
```bash
|
|
8633
|
-
|
|
8634
|
-
|
|
8593
|
+
DSH_SMOKE_ROOT="$(mktemp -d)"
|
|
8594
|
+
export DSH_HOME="$DSH_SMOKE_ROOT/dsh-home"
|
|
8635
8595
|
|
|
8636
|
-
|
|
8637
|
-
|
|
8596
|
+
ai-toolkit dsh install --profile web
|
|
8597
|
+
ai-toolkit dsh doctor --profile web
|
|
8598
|
+
dsh --profile web --host 127.0.0.1 --port 0 --no-open
|
|
8599
|
+
```
|
|
8638
8600
|
|
|
8639
|
-
|
|
8640
|
-
|
|
8641
|
-
|
|
8601
|
+
In a new `softspark-orchestrator` session, select the `codex` provider and run two standalone marker prompts:
|
|
8602
|
+
|
|
8603
|
+
1. `subagent_claude_code` returns an exact child marker, then the Codex parent returns its exact completion marker.
|
|
8604
|
+
2. `subagent_gemini_copilot` returns an exact child marker, then the Codex parent returns its exact completion marker.
|
|
8605
|
+
|
|
8606
|
+
Stop DSH, then remove only the managed profile artifacts:
|
|
8607
|
+
|
|
8608
|
+
```bash
|
|
8609
|
+
ai-toolkit dsh uninstall --profile web --yes
|
|
8642
8610
|
```
|
|
8643
8611
|
|
|
8612
|
+
**Verify:** both tool results have `isError=false`, both turns end as `completed`, `doctor` reports no recovery requirement before uninstall, and an unrelated preset fixture remains unchanged. Preserve only redacted event sequence evidence; never attach credentials, auth files, or full private prompts.
|
|
8613
|
+
|
|
8644
8614
|
---
|
|
8645
8615
|
|
|
8646
|
-
##
|
|
8616
|
+
## Troubleshooting
|
|
8647
8617
|
|
|
8648
|
-
|
|
8649
|
-
|
|
8650
|
-
|
|
8651
|
-
|
|
8652
|
-
|
|
8653
|
-
|
|
8654
|
-
|
|
8655
|
-
|
|
8656
|
-
|
|
8657
|
-
|
|
8658
|
-
|
|
8659
|
-
|
|
8660
|
-
|
|
8661
|
-
|
|
8662
|
-
|
|
8663
|
-
|
|
8664
|
-
|
|
8665
|
-
|
|
8666
|
-
|
|
8667
|
-
|
|
8668
|
-
|
|
8669
|
-
|
|
8670
|
-
|
|
8671
|
-
|
|
8672
|
-
|
|
8673
|
-
|
|
8674
|
-
|
|
8675
|
-
|
|
8676
|
-
|
|
8618
|
+
### `ai-toolkit: command not found`
|
|
8619
|
+
|
|
8620
|
+
```bash
|
|
8621
|
+
npm install -g @softspark/ai-toolkit
|
|
8622
|
+
# or check PATH:
|
|
8623
|
+
export PATH="$(npm config get prefix)/bin:$PATH"
|
|
8624
|
+
```
|
|
8625
|
+
|
|
8626
|
+
### Doctor: broken symlinks
|
|
8627
|
+
|
|
8628
|
+
```bash
|
|
8629
|
+
ai-toolkit doctor --fix # auto-repair
|
|
8630
|
+
ai-toolkit update # or full re-install
|
|
8631
|
+
```
|
|
8632
|
+
|
|
8633
|
+
### Tests fail: missing bats
|
|
8634
|
+
|
|
8635
|
+
```bash
|
|
8636
|
+
brew install bats-core # macOS
|
|
8637
|
+
npm install -g bats # cross-platform
|
|
8638
|
+
```
|
|
8639
|
+
|
|
8640
|
+
### validate.py: stale counts
|
|
8641
|
+
|
|
8642
|
+
README badges don't match the current agents/skills/tests counts.
|
|
8643
|
+
Update README.md and re-run.
|
|
8644
|
+
|
|
8645
|
+
### Eject: missing skills
|
|
8646
|
+
|
|
8647
|
+
```bash
|
|
8648
|
+
ai-toolkit update # re-link missing symlinks
|
|
8649
|
+
ai-toolkit eject /tmp/test # retry
|
|
8650
|
+
```
|
|
8677
8651
|
|
|
8678
8652
|
---
|
|
8679
8653
|
|
|
8680
|
-
##
|
|
8654
|
+
## Success Criteria
|
|
8655
|
+
|
|
8656
|
+
| Area | Criterion |
|
|
8657
|
+
|------|-----------|
|
|
8658
|
+
| CLI | `--version` correct, `--help` full list, `status` current |
|
|
8659
|
+
| Health | `doctor`: 0 errors, 0 warnings, PASSED |
|
|
8660
|
+
| Install | `--dry-run` correct counts, `--local` all configs |
|
|
8661
|
+
| Quality | `validate.py --strict`: PASSED |
|
|
8662
|
+
| Security (baseline) | `audit_skills.py --ci`: 0 HIGH |
|
|
8663
|
+
| Security (SARIF) | `audit_skills.py --sarif`: valid SARIF 2.1.0 with non-empty rules array |
|
|
8664
|
+
| Security (permissions) | `audit_skills.py --permissions`: broad-access skills unchanged or justified in CHANGELOG |
|
|
8665
|
+
| Supply chain | `dist.attestations.provenance.predicateType == https://slsa.dev/provenance/v1` on npm |
|
|
8666
|
+
| Supply chain | All `sources.json` URL entries carry a `sha256`; `AI_TOOLKIT_STRICT_PIN=1 ai-toolkit update --dry-run` passes |
|
|
8667
|
+
| Tests | `npm test`: N/N passed, 0 failures |
|
|
8668
|
+
| Eject | Standalone `.claude/` with real files AND `output-styles/` directory |
|
|
8669
|
+
| Guards | Destructive commands blocked |
|
|
8670
|
+
| Deep coverage | `--profile full` emits every documented native surface, including Codex agents/skills/hooks and Copilot agents/skills/hooks |
|
|
8671
|
+
| Profile contract | Codex skills emit at every profile without `--codex-skills`; Copilot standard emits agents, skills, instructions, prompts, and hooks |
|
|
8672
|
+
| Idempotence | Second `install` run produces byte-identical output in every managed path |
|
|
8673
|
+
| Live JSON | Every generated `.json` file on disk parses as valid JSON |
|
|
8674
|
+
| Registry | `supported-tools-registry.md` enumerates every `scripts/generate_*.py` we ship |
|
|
8681
8675
|
|
|
8682
8676
|
---
|
|
8683
|
-
|
|
8677
|
+
|
|
8678
|
+
## kb/procedures/sop-release.md
|
|
8679
|
+
|
|
8680
|
+
---
|
|
8681
|
+
title: "SOP: Release Preparation"
|
|
8684
8682
|
category: procedures
|
|
8685
8683
|
service: ai-toolkit
|
|
8686
|
-
tags: [sop,
|
|
8687
|
-
version: "1.
|
|
8688
|
-
created: "2026-04-
|
|
8689
|
-
last_updated: "2026-09-
|
|
8690
|
-
description: "
|
|
8684
|
+
tags: [sop, release, version, publish, changelog, semver, provenance, sarif, ecosystem, shellcheck]
|
|
8685
|
+
version: "1.15.0"
|
|
8686
|
+
created: "2026-04-10"
|
|
8687
|
+
last_updated: "2026-09-02"
|
|
8688
|
+
description: "Step-by-step checklist for preparing a new ai-toolkit release — ecosystem-sync drift check, version sync, changelog, artifact regeneration, validation, branch CI, and tagging. Run BEFORE every git tag. Includes mandatory Provenance, SARIF, checksum-pin, ShellCheck, licensing, exact-tag assertions, and a green Ubuntu/macOS branch-CI gate before any release tag is created."
|
|
8691
8689
|
---
|
|
8692
8690
|
|
|
8693
|
-
# SOP: Release
|
|
8694
|
-
|
|
8695
|
-
End-to-end smoke test after installing or updating `@softspark/ai-toolkit`.
|
|
8696
|
-
Verifies all critical paths from the user's perspective.
|
|
8691
|
+
# SOP: Release Preparation
|
|
8697
8692
|
|
|
8698
|
-
|
|
8699
|
-
|
|
8700
|
-
-
|
|
8701
|
-
- Before tagging a new version (`git tag`)
|
|
8702
|
-
- Before publishing to npm (`npm publish`)
|
|
8703
|
-
- As a smoke test in CI/CD
|
|
8693
|
+
Complete checklist for preparing a new `@softspark/ai-toolkit` release.
|
|
8694
|
+
Run this **before** tagging. After tagging and publishing, run the
|
|
8695
|
+
[Release Verification SOP](sop-release-verification.md) to smoke-test.
|
|
8704
8696
|
|
|
8705
|
-
**
|
|
8706
|
-
|
|
8707
|
-
|
|
8697
|
+
**Pipeline:**
|
|
8698
|
+
```
|
|
8699
|
+
Ecosystem Sync SOP (drift check + generator updates)
|
|
8700
|
+
↓
|
|
8701
|
+
Release Preparation (this SOP)
|
|
8702
|
+
↓
|
|
8703
|
+
git tag → CI publish → Release Verification SOP
|
|
8704
|
+
```
|
|
8708
8705
|
|
|
8709
|
-
**Time:** 10-
|
|
8706
|
+
**Time:** 10-20 minutes (includes ecosystem sync review)
|
|
8710
8707
|
|
|
8711
8708
|
---
|
|
8712
8709
|
|
|
8713
8710
|
## Quick Checklist (TL;DR)
|
|
8714
8711
|
|
|
8715
|
-
The 14 core commands below must pass. Releases that change DSH must also complete Phase 10.
|
|
8716
|
-
|
|
8717
8712
|
```bash
|
|
8718
|
-
#
|
|
8719
|
-
|
|
8720
|
-
python3 scripts/
|
|
8721
|
-
|
|
8722
|
-
|
|
8713
|
+
# 0. Ecosystem sync (mandatory for minor/major releases; optional for patch)
|
|
8714
|
+
# Full procedure: kb/procedures/sop-ecosystem-sync.md
|
|
8715
|
+
python3 scripts/ecosystem_doctor.py --format text > /tmp/eco-report.txt
|
|
8716
|
+
cat /tmp/eco-report.txt
|
|
8717
|
+
# If drift detected: stop here, follow sop-ecosystem-sync.md Phase 2-4 to
|
|
8718
|
+
# classify each drift (A-F), update generators as needed, refresh snapshot,
|
|
8719
|
+
# THEN resume this SOP.
|
|
8720
|
+
python3 scripts/ecosystem_doctor.py --update # after all drift resolved
|
|
8723
8721
|
|
|
8724
|
-
#
|
|
8725
|
-
|
|
8726
|
-
|
|
8727
|
-
|
|
8728
|
-
ai-toolkit install --dry-run # 8. Global install OK?
|
|
8729
|
-
python3 scripts/audit_skills.py --ci # 9. Security audit clean?
|
|
8722
|
+
# 1. Decide version bump
|
|
8723
|
+
# patch (1.4.2 → 1.4.3): bugfix, typo, doc fix
|
|
8724
|
+
# minor (1.4.2 → 1.5.0): new feature, new skill, new flag, any ecosystem-class-B/F change
|
|
8725
|
+
# major (1.4.2 → 2.0.0): breaking change, any ecosystem-class-D removed path
|
|
8730
8726
|
|
|
8731
|
-
#
|
|
8732
|
-
python3 scripts/
|
|
8733
|
-
python3 scripts/audit_skills.py --permissions | head -30 # 11. Broad-access skills reviewed?
|
|
8734
|
-
npm view @softspark/ai-toolkit@X.Y.Z --json | python3 -c "import json,sys; d=json.load(sys.stdin); assert d['dist']['attestations']['provenance']['predicateType']=='https://slsa.dev/provenance/v1'; print('PROVENANCE OK')" # 12. Provenance attested on npm?
|
|
8735
|
-
python3 scripts/claude_app.py verify # 13. Claude Chat/Cowork plugin contract valid?
|
|
8727
|
+
# 2. Sync version across all files
|
|
8728
|
+
python3 scripts/sync_version.py X.Y.Z # if script exists, else manual
|
|
8736
8729
|
|
|
8737
|
-
#
|
|
8738
|
-
|
|
8739
|
-
|
|
8730
|
+
# 3. Write CHANGELOG.md entry
|
|
8731
|
+
# 4. Regenerate artifacts
|
|
8732
|
+
python3 scripts/generate_agents_md.py > AGENTS.md
|
|
8733
|
+
python3 scripts/generate_llms_txt.py > llms.txt
|
|
8734
|
+
python3 scripts/generate_llms_txt.py --full > llms-full.txt
|
|
8735
|
+
|
|
8736
|
+
# 5. Validate + audit + SARIF + shellcheck + test + ecosystem check
|
|
8737
|
+
python3 scripts/validate.py --strict && python3 scripts/audit_skills.py --ci && python3 scripts/audit_skills.py --sarif > /tmp/audit.sarif && shellcheck --severity=warning app/hooks/*.sh && npm test
|
|
8738
|
+
|
|
8739
|
+
# 5a. Supply-chain standard (v2.8.0+) — non-negotiable
|
|
8740
|
+
grep -q -- '--provenance' .github/workflows/publish.yml || { echo "MISSING --provenance"; exit 1; }
|
|
8741
|
+
grep -q 'id-token: write' .github/workflows/publish.yml || { echo "MISSING id-token: write"; exit 1; }
|
|
8742
|
+
python3 scripts/audit_skills.py --permissions # review Bash/Write/Edit footprint
|
|
8743
|
+
|
|
8744
|
+
# 5b. Ecosystem gate — snapshot must be current before tag
|
|
8745
|
+
python3 scripts/ecosystem_doctor.py --offline --check || { echo "STALE ecosystem snapshot — re-run doctor"; exit 1; }
|
|
8746
|
+
|
|
8747
|
+
# 5c. Licensing gate — SPDX headers, LICENSE, NOTICE, manifest consistency
|
|
8748
|
+
npx bats tests/test_licensing.bats || { echo "LICENSING GATE FAILED"; exit 1; }
|
|
8749
|
+
|
|
8750
|
+
# 6. Commit, push branch, and wait for the exact commit's full CI
|
|
8751
|
+
git add -A && git commit -m "chore: release vX.Y.Z"
|
|
8752
|
+
RELEASE_SHA=$(git rev-parse HEAD)
|
|
8753
|
+
git push origin main || { echo "FAIL: release commit push failed"; exit 1; }
|
|
8754
|
+
RUN_ID=""
|
|
8755
|
+
for ATTEMPT in 1 2 3 4 5; do
|
|
8756
|
+
RUN_ID=$(gh run list --workflow ci.yml --event push --commit "$RELEASE_SHA" \
|
|
8757
|
+
--limit 1 --json databaseId --jq '.[0].databaseId')
|
|
8758
|
+
[ -n "$RUN_ID" ] && break
|
|
8759
|
+
echo "Waiting for CI run registration ($ATTEMPT/5)" >&2
|
|
8760
|
+
[ "$ATTEMPT" -eq 5 ] || sleep 60
|
|
8761
|
+
done
|
|
8762
|
+
test -n "$RUN_ID" || { echo "FAIL: CI run not found for release commit"; exit 1; }
|
|
8763
|
+
gh run watch "$RUN_ID" --exit-status \
|
|
8764
|
+
|| { echo "FAIL: release commit CI failed"; exit 1; }
|
|
8765
|
+
for OS in ubuntu-latest macos-latest; do
|
|
8766
|
+
gh run view "$RUN_ID" --json jobs \
|
|
8767
|
+
--jq ".jobs[] | select(.name == \"Bats test suite ($OS)\") | .conclusion" \
|
|
8768
|
+
| grep -qx success || { echo "FAIL: $OS test job not green"; exit 1; }
|
|
8769
|
+
done
|
|
8770
|
+
test "$(git rev-parse HEAD)" = "$RELEASE_SHA" || { echo "FAIL: local HEAD changed during CI"; exit 1; }
|
|
8771
|
+
git fetch origin main || { echo "FAIL: origin/main refresh failed"; exit 1; }
|
|
8772
|
+
test "$(git rev-parse origin/main)" = "$RELEASE_SHA" || { echo "FAIL: origin/main changed during CI"; exit 1; }
|
|
8773
|
+
|
|
8774
|
+
# 6a. Only green branch CI authorizes tag creation
|
|
8775
|
+
git tag vX.Y.Z "$RELEASE_SHA"
|
|
8776
|
+
|
|
8777
|
+
# 6b. Assert the tag before pushing it (v4.19.0 postmortem, Phase 7)
|
|
8778
|
+
test "$(git rev-parse vX.Y.Z)" = "$RELEASE_SHA" || { echo "FAIL: tag not on tested release commit"; exit 1; }
|
|
8779
|
+
git show --no-patch --format=%s vX.Y.Z | grep -qx "chore: release vX.Y.Z" || { echo "FAIL: tag not on release commit"; exit 1; }
|
|
8780
|
+
|
|
8781
|
+
# 6c. Push the single tag by full ref. Never --tags.
|
|
8782
|
+
git push origin refs/tags/vX.Y.Z
|
|
8740
8783
|
```
|
|
8741
8784
|
|
|
8742
8785
|
---
|
|
8743
8786
|
|
|
8744
|
-
## Phase 0:
|
|
8787
|
+
## Phase 0: Ecosystem Sync (MANDATORY for minor/major)
|
|
8745
8788
|
|
|
8746
|
-
|
|
8747
|
-
|
|
8789
|
+
Before touching version numbers, confirm the toolkit is aligned with the current state of every editor / platform it integrates with. Skipping this phase ships a release whose generators may lag a month-old CLI refactor, a rename of `.cursorrules` to `.cursor/rules/`, or a new hook event we do not yet emit.
|
|
8790
|
+
|
|
8791
|
+
**When this phase is mandatory:**
|
|
8792
|
+
- Minor release (X.Y.0) — always
|
|
8793
|
+
- Major release (X.0.0) — always
|
|
8794
|
+
- Patch release (X.Y.Z) — only if the patch touches a generator or install flow
|
|
8795
|
+
|
|
8796
|
+
**When to skip:** pure doc-only patches, SOP edits, internal refactors that do not touch `scripts/generate_*` or `app/skills/*/SKILL.md`.
|
|
8797
|
+
|
|
8798
|
+
### 0.1 Run the doctor
|
|
8748
8799
|
|
|
8749
8800
|
```bash
|
|
8750
|
-
|
|
8751
|
-
|
|
8752
|
-
python3 scripts/generate_llms_txt.py > llms.txt
|
|
8753
|
-
python3 scripts/generate_llms_txt.py --full > llms-full.txt
|
|
8801
|
+
python3 scripts/ecosystem_doctor.py --format text | tee /tmp/eco-report.txt
|
|
8802
|
+
```
|
|
8754
8803
|
|
|
8755
|
-
|
|
8756
|
-
python3 scripts/validate.py --strict
|
|
8804
|
+
Output classifies every registered tool as **Clean**, **Drift**, or **Errored**.
|
|
8757
8805
|
|
|
8758
|
-
|
|
8759
|
-
python3 scripts/audit_skills.py --ci
|
|
8806
|
+
### 0.2 Act on drift
|
|
8760
8807
|
|
|
8761
|
-
|
|
8762
|
-
npm test
|
|
8808
|
+
For each drifting tool, follow [sop-ecosystem-sync.md](sop-ecosystem-sync.md) Phase 2-4:
|
|
8763
8809
|
|
|
8764
|
-
|
|
8765
|
-
|
|
8766
|
-
|
|
8767
|
-
|
|
8768
|
-
|
|
8769
|
-
|
|
8770
|
-
|
|
8771
|
-
|
|
8810
|
+
| Drift class | Release impact |
|
|
8811
|
+
|-------------|----------------|
|
|
8812
|
+
| A (cosmetic reword) | No version impact — refresh snapshot, continue |
|
|
8813
|
+
| B (new feature — integrate) | **Minor** version bump at minimum; new generator or extended generator |
|
|
8814
|
+
| C (new feature — not adopted) | No impact — note in registry |
|
|
8815
|
+
| D (deprecation) | **Minor** or **major** depending on user impact; add migration warning |
|
|
8816
|
+
| E (feature promoted to default) | **Minor**; simplify generator, keep fallback comment |
|
|
8817
|
+
| F (feature newly globally available) | **Minor**; may require new generator or new config path |
|
|
8772
8818
|
|
|
8773
|
-
|
|
8774
|
-
|
|
8775
|
-
|
|
8819
|
+
If any B/D/E/F changes land in this preparation pass, mention them explicitly in the CHANGELOG entry (Phase 3) under a `Ecosystem` subsection.
|
|
8820
|
+
|
|
8821
|
+
### 0.3 Refresh snapshot
|
|
8822
|
+
|
|
8823
|
+
Once every drift is resolved (either by code change or by re-classifying as acceptable):
|
|
8776
8824
|
|
|
8777
|
-
**One-liner (copy-paste):**
|
|
8778
8825
|
```bash
|
|
8779
|
-
python3 scripts/
|
|
8826
|
+
python3 scripts/ecosystem_doctor.py --update
|
|
8780
8827
|
```
|
|
8781
8828
|
|
|
8782
|
-
|
|
8829
|
+
This writes the new baseline to `benchmarks/ecosystem-doctor-snapshot.json`. Commit it as part of the release commit.
|
|
8783
8830
|
|
|
8784
|
-
|
|
8831
|
+
### 0.4 Gate
|
|
8785
8832
|
|
|
8786
8833
|
```bash
|
|
8787
|
-
|
|
8788
|
-
ai-toolkit --help
|
|
8789
|
-
which ai-toolkit
|
|
8834
|
+
python3 scripts/ecosystem_doctor.py --offline --check
|
|
8790
8835
|
```
|
|
8791
8836
|
|
|
8792
|
-
|
|
8793
|
-
- [ ] `--version` returns correct semver (e.g., `1.4.0`)
|
|
8794
|
-
- [ ] `--help` displays full command list without errors
|
|
8795
|
-
- [ ] `which` points to global npm bin path
|
|
8837
|
+
Must exit `0`. If it exits `1`, the snapshot is stale — rerun Phase 0.3 or review the remaining drift.
|
|
8796
8838
|
|
|
8797
8839
|
---
|
|
8798
8840
|
|
|
8799
|
-
## Phase
|
|
8841
|
+
## Phase 1: Determine Version Bump
|
|
8800
8842
|
|
|
8801
|
-
|
|
8802
|
-
ai-toolkit install --dry-run
|
|
8803
|
-
ai-toolkit status
|
|
8804
|
-
```
|
|
8843
|
+
Follow [Semantic Versioning](https://semver.org/):
|
|
8805
8844
|
|
|
8806
|
-
|
|
8807
|
-
|
|
8808
|
-
-
|
|
8809
|
-
|
|
8810
|
-
|
|
8845
|
+
| Change Type | Bump | Examples |
|
|
8846
|
+
|-------------|------|---------|
|
|
8847
|
+
| Bugfix, typo, doc-only | **patch** | Fix install flag, correct description |
|
|
8848
|
+
| New feature, skill, agent, flag | **minor** | Add `/hipaa-validate`, add `--output json` |
|
|
8849
|
+
| Breaking CLI change, removed skill, config format change | **major** | Rename `install` to `setup`, remove skill |
|
|
8811
8850
|
|
|
8812
|
-
**
|
|
8813
|
-
- [ ] Version matches expected
|
|
8814
|
-
- [ ] Profile: minimal/standard/strict
|
|
8815
|
-
- [ ] Modules: list of installed modules
|
|
8816
|
-
- [ ] Latest: up to date / update available
|
|
8851
|
+
**Rule:** When in doubt, bump minor.
|
|
8817
8852
|
|
|
8818
8853
|
---
|
|
8819
8854
|
|
|
8820
|
-
## Phase
|
|
8855
|
+
## Phase 2: Sync Version in All Files
|
|
8821
8856
|
|
|
8822
|
-
|
|
8823
|
-
ai-toolkit doctor
|
|
8824
|
-
```
|
|
8857
|
+
The canonical version lives in `package.json`. These files **must** match:
|
|
8825
8858
|
|
|
8826
|
-
|
|
8827
|
-
- Environment: node, bash, python3, bats
|
|
8828
|
-
- Global Install: .claude exists, agents/skills symlinks (0 broken), settings.json hooks
|
|
8829
|
-
- Hook Scripts: all present and executable
|
|
8830
|
-
- Hook Configuration: 14 events registered
|
|
8831
|
-
- Generated Artifacts: AGENTS.md, llms.txt, llms-full.txt
|
|
8832
|
-
- Planned Assets: plugin.json, benchmarks, plugin packs
|
|
8833
|
-
- Benchmark Freshness: < 30 days
|
|
8834
|
-
- Stale Rules: all healthy
|
|
8859
|
+
### Mandatory sync (every release)
|
|
8835
8860
|
|
|
8836
|
-
|
|
8837
|
-
|
|
8838
|
-
|
|
8861
|
+
| File | Field | How to update |
|
|
8862
|
+
|------|-------|---------------|
|
|
8863
|
+
| `package.json` | `"version": "X.Y.Z"` | Edit directly |
|
|
8864
|
+
| `manifest.json` | `"version": "X.Y.Z"` | Edit directly |
|
|
8865
|
+
| `app/.claude-plugin/plugin.json` | `"version": "X.Y.Z"` | Edit directly |
|
|
8839
8866
|
|
|
8840
|
-
|
|
8841
|
-
(broken symlinks, non-executable hooks, missing scripts, missing llms-full.txt).
|
|
8867
|
+
### Auto-synced (no manual action)
|
|
8842
8868
|
|
|
8843
|
-
|
|
8869
|
+
| File | Mechanism |
|
|
8870
|
+
|------|-----------|
|
|
8871
|
+
| `package-lock.json` | Regenerated by `npm install --package-lock-only` |
|
|
8844
8872
|
|
|
8845
|
-
|
|
8873
|
+
### Conditional sync (only if the doc was modified in this release)
|
|
8874
|
+
|
|
8875
|
+
| File | Field | When to update |
|
|
8876
|
+
|------|-------|---------------|
|
|
8877
|
+
| `kb/procedures/sop-maintenance.md` | frontmatter `version:` | If SOP content changed |
|
|
8878
|
+
| `kb/reference/skills-catalog.md` | frontmatter `version:` | If skills added/removed |
|
|
8879
|
+
| `kb/reference/agents-catalog.md` | frontmatter `version:` | If agents added/removed |
|
|
8880
|
+
| `kb/reference/hooks-catalog.md` | frontmatter `version:` | If hooks changed |
|
|
8881
|
+
| `kb/reference/architecture-overview.md` | frontmatter `version:` | If architecture changed |
|
|
8882
|
+
| `kb/reference/distribution-model.md` | frontmatter `version:` | If install model changed |
|
|
8883
|
+
| `kb/reference/global-install-model.md` | frontmatter `version:` | If install model changed |
|
|
8884
|
+
|
|
8885
|
+
> **Note:** KB `version:` fields track the **document version**, not the toolkit version.
|
|
8886
|
+
> Only bump them when the document content actually changes in this release.
|
|
8887
|
+
|
|
8888
|
+
### Count sync (if skills/agents/hooks changed)
|
|
8889
|
+
|
|
8890
|
+
| File | What to check |
|
|
8891
|
+
|------|---------------|
|
|
8892
|
+
| `package.json` | `"description"` — skill/agent count |
|
|
8893
|
+
| `README.md` | Badge counts, "What You Get" table |
|
|
8894
|
+
| `app/ARCHITECTURE.md` | Section headings with counts |
|
|
8895
|
+
|
|
8896
|
+
> **Tip:** `validate.py --strict` catches count drift AND version mismatches
|
|
8897
|
+
> (package.json vs manifest.json vs plugin.json) automatically.
|
|
8898
|
+
> If validation passes, counts and versions are correct.
|
|
8899
|
+
|
|
8900
|
+
### Verification command
|
|
8901
|
+
|
|
8902
|
+
After syncing, verify all mandatory files match:
|
|
8846
8903
|
|
|
8847
8904
|
```bash
|
|
8848
|
-
|
|
8849
|
-
|
|
8850
|
-
|
|
8851
|
-
|
|
8905
|
+
VERSION=$(python3 -c "import json; print(json.load(open('package.json'))['version'])")
|
|
8906
|
+
echo "Target: $VERSION"
|
|
8907
|
+
echo "manifest.json: $(python3 -c "import json; print(json.load(open('manifest.json'))['version'])")"
|
|
8908
|
+
echo "plugin.json: $(python3 -c "import json; print(json.load(open('app/.claude-plugin/plugin.json'))['version'])")"
|
|
8909
|
+
echo "package-lock.json: $(python3 -c "import json; print(json.load(open('package-lock.json'))['version'])")"
|
|
8852
8910
|
```
|
|
8853
8911
|
|
|
8854
|
-
|
|
8855
|
-
|
|
8856
|
-
|
|
8857
|
-
- [ ] Would inject: .claude/constitution.md
|
|
8858
|
-
- [ ] Editors: all 11 listed (copilot, cursor, windsurf, cline, roo, aider, augment, antigravity, codex, gemini, opencode)
|
|
8859
|
-
- [ ] Would generate configs for each editor (legacy + directory-based)
|
|
8860
|
-
- [ ] Would install: .git/hooks/pre-commit
|
|
8861
|
-
- [ ] Would inject language rules (auto-detected)
|
|
8912
|
+
All four must print the same version. If not, fix before proceeding.
|
|
8913
|
+
|
|
8914
|
+
### Public surface review
|
|
8862
8915
|
|
|
8863
|
-
**Also test auto-detect (no --editors flag):**
|
|
8864
8916
|
```bash
|
|
8865
|
-
|
|
8866
|
-
|
|
8917
|
+
python3 scripts/surface_manifest.py --update
|
|
8918
|
+
git diff app/surface.json
|
|
8867
8919
|
```
|
|
8868
8920
|
|
|
8869
|
-
|
|
8921
|
+
**Every line the diff removes is a breaking change.** Restore it, or take the
|
|
8922
|
+
deprecation path in `BACKWARD_COMPATIBILITY.md` and add a `DECISIONS.md` entry
|
|
8923
|
+
before the tag. Lines added are new surface being adopted into protection — that
|
|
8924
|
+
needs no ceremony.
|
|
8870
8925
|
|
|
8871
|
-
|
|
8926
|
+
Do not run `--update` to make a red build green. The check fails because something
|
|
8927
|
+
users depend on disappeared; regenerating the manifest deletes the evidence, not
|
|
8928
|
+
the problem.
|
|
8872
8929
|
|
|
8873
|
-
|
|
8874
|
-
|
|
8875
|
-
|
|
8930
|
+
### Skill body budget ratchet
|
|
8931
|
+
|
|
8932
|
+
`validate.py` prints the largest skill body on every run:
|
|
8933
|
+
|
|
8934
|
+
```
|
|
8935
|
+
Body budget: largest is <skill> at <N> bytes (warn 18000, error 20000)
|
|
8876
8936
|
```
|
|
8877
8937
|
|
|
8878
|
-
|
|
8879
|
-
|
|
8880
|
-
|
|
8881
|
-
- [ ] Plugin packs >= 10, KB documents >= 20
|
|
8882
|
-
- [ ] `Errors: 0 | Warnings: 0` → `VALIDATION PASSED`
|
|
8938
|
+
Once that number sits at least 2000 bytes under `SKILL_BODY_BUDGET_WARN`, lower
|
|
8939
|
+
`SKILL_BODY_BUDGET_WARN` by 2000 in `scripts/validate.py` and ship the tightened
|
|
8940
|
+
threshold with the release. Target floor is 12000.
|
|
8883
8941
|
|
|
8884
|
-
|
|
8885
|
-
|
|
8886
|
-
-
|
|
8887
|
-
|
|
8942
|
+
Two rules, both learned the hard way:
|
|
8943
|
+
|
|
8944
|
+
- **Never lower a threshold in the same change that something violates it.** Split
|
|
8945
|
+
the offending skill into `SKILL.md` + `reference/` first, prove it with
|
|
8946
|
+
`python3 scripts/check_split.py <skill> --before <pre-split SKILL.md>`, then
|
|
8947
|
+
tighten in a follow-up.
|
|
8948
|
+
- **Never raise a threshold to make a red build green.** A body over budget means
|
|
8949
|
+
detail that belongs in `reference/` is loading on every trigger match, including
|
|
8950
|
+
the accidental ones. Raising the number hides the cost, it does not remove it.
|
|
8888
8951
|
|
|
8889
|
-
|
|
8952
|
+
If the largest body has not moved since the previous release, leave the threshold
|
|
8953
|
+
alone and say so in the release notes. A ratchet that never advances is honest;
|
|
8954
|
+
one that advances past reality is not.
|
|
8890
8955
|
|
|
8891
|
-
|
|
8956
|
+
---
|
|
8892
8957
|
|
|
8893
|
-
|
|
8894
|
-
# Run ONCE, capture to file, then parse. Full suite is 669+ bats cases —
|
|
8895
|
-
# re-running it per check (tail / grep ok / grep not ok piped separately)
|
|
8896
|
-
# wastes minutes every release. Always cache the output.
|
|
8897
|
-
npm test > /tmp/npm-test.log 2>&1
|
|
8898
|
-
exit=$?
|
|
8899
|
-
tail -3 /tmp/npm-test.log
|
|
8900
|
-
echo "ok: $(grep -c '^ok ' /tmp/npm-test.log)"
|
|
8901
|
-
echo "not ok: $(grep -c '^not ok' /tmp/npm-test.log)"
|
|
8902
|
-
echo "exit: $exit"
|
|
8903
|
-
```
|
|
8958
|
+
## Phase 3: Write CHANGELOG Entry
|
|
8904
8959
|
|
|
8905
|
-
|
|
8906
|
-
- [ ] `exit == 0`
|
|
8907
|
-
- [ ] `ok == expected test count` (e.g., 945 on v3.0.0)
|
|
8908
|
-
- [ ] `not ok == 0`
|
|
8909
|
-
- [ ] Bats runs tests in parallel (4 jobs)
|
|
8910
|
-
- [ ] Groups: agents, autodetect, cli, generators, guards, hooks, inject,
|
|
8911
|
-
install, kb, mcp, readme, profiles, uninstall, validate
|
|
8960
|
+
Add entry at the top of `CHANGELOG.md` (after the header, before previous release):
|
|
8912
8961
|
|
|
8913
|
-
|
|
8914
|
-
|
|
8915
|
-
# Runs the full suite THREE times. Adds 1-3 min and pressures CI capacity.
|
|
8916
|
-
npm test 2>&1 | tail -3
|
|
8917
|
-
npm test 2>&1 | grep -c '^ok '
|
|
8918
|
-
npm test 2>&1 | grep -c '^not ok'
|
|
8919
|
-
```
|
|
8962
|
+
```markdown
|
|
8963
|
+
## vX.Y.Z — Short Title (YYYY-MM-DD)
|
|
8920
8964
|
|
|
8921
|
-
|
|
8922
|
-
-
|
|
8923
|
-
- Install: idempotent, profiles, --only/--skip, orphan cleanup
|
|
8924
|
-
- Eject: real files (not symlinks), inlined rules
|
|
8925
|
-
- Uninstall: removes toolkit, preserves user content
|
|
8965
|
+
### Added
|
|
8966
|
+
- **Feature name** — description
|
|
8926
8967
|
|
|
8927
|
-
|
|
8968
|
+
### Changed
|
|
8969
|
+
- **What changed** — old behavior → new behavior
|
|
8928
8970
|
|
|
8929
|
-
|
|
8971
|
+
### Fixed
|
|
8972
|
+
- **Bug description** — what was broken and how it's fixed
|
|
8930
8973
|
|
|
8931
|
-
|
|
8932
|
-
|
|
8933
|
-
cd /tmp/ai-toolkit-eject-test
|
|
8934
|
-
ai-toolkit eject
|
|
8935
|
-
cd - && rm -rf /tmp/ai-toolkit-eject-test
|
|
8974
|
+
### Removed
|
|
8975
|
+
- **What was removed** — migration path if any
|
|
8936
8976
|
```
|
|
8937
8977
|
|
|
8938
|
-
**
|
|
8939
|
-
-
|
|
8940
|
-
-
|
|
8941
|
-
-
|
|
8942
|
-
-
|
|
8943
|
-
-
|
|
8978
|
+
**Rules:**
|
|
8979
|
+
- Use **bold** for feature names
|
|
8980
|
+
- Start descriptions with a verb (Added, Changed, Fixed, Removed)
|
|
8981
|
+
- Reference skill names with backticks and slash: `/hipaa-validate`
|
|
8982
|
+
- Include script names: `scripts/hipaa_scan.py`
|
|
8983
|
+
- Include count changes: `Skill count: 91 → 92`
|
|
8984
|
+
- Date format: `YYYY-MM-DD`
|
|
8985
|
+
- Title: short, descriptive, no version number repetition
|
|
8944
8986
|
|
|
8945
|
-
|
|
8987
|
+
### Update README "What's New" section
|
|
8946
8988
|
|
|
8947
|
-
|
|
8989
|
+
**MANDATORY on every release.** Update the `## What's New in vX.Y.Z` section in `README.md`:
|
|
8948
8990
|
|
|
8949
|
-
|
|
8991
|
+
1. Change the heading version: `## What's New in vX.Y.Z`
|
|
8992
|
+
2. Replace bullet points with 3-5 highlights from this release
|
|
8993
|
+
3. **Keep only the latest version block.** Delete the previous `## What's New in vA.B.C` section(s). README is the shop window, not the archive — users see the current release, full history lives in `CHANGELOG.md`.
|
|
8994
|
+
4. Keep the `See [CHANGELOG.md](CHANGELOG.md) for full history.` link directly below the bullet list.
|
|
8950
8995
|
|
|
8951
|
-
|
|
8996
|
+
> **Warning:** This section is the first thing users see after the badges.
|
|
8997
|
+
> A stale version here (e.g., "What's New in v2.1.3" when shipping v2.3.0)
|
|
8998
|
+
> signals an unmaintained project. Do NOT skip this step.
|
|
8952
8999
|
|
|
8953
|
-
|
|
8954
|
-
VERSION="X.Y.Z" # the tag just published
|
|
8955
|
-
npm view "@softspark/ai-toolkit@${VERSION}" --json \
|
|
8956
|
-
| python3 -c "import json, sys; d=json.load(sys.stdin); att=d['dist'].get('attestations', {}); assert att.get('provenance', {}).get('predicateType') == 'https://slsa.dev/provenance/v1', f'NO PROVENANCE for {d[\"version\"]}'; print(f'PROVENANCE OK: {att[\"url\"]}')"
|
|
8957
|
-
```
|
|
9000
|
+
> **Single-version rule:** README.md must contain **exactly one** `## What's New in vX.Y.Z` heading at any time. If you find multiple stacked (e.g. v2.6.1 + v2.6.0 + v2.5.0), that is a SOP drift — collapse to the latest on the next release commit.
|
|
8958
9001
|
|
|
8959
|
-
|
|
8960
|
-
- [ ] Exit 0 and prints `PROVENANCE OK: https://registry.npmjs.org/...`
|
|
8961
|
-
- [ ] `https://www.npmjs.com/package/@softspark/ai-toolkit/v/${VERSION}` shows the green "Provenance" badge
|
|
9002
|
+
---
|
|
8962
9003
|
|
|
8963
|
-
|
|
9004
|
+
## Phase 4: Regenerate Artifacts
|
|
8964
9005
|
|
|
8965
|
-
|
|
9006
|
+
Use the npm scripts, not the generators directly:
|
|
8966
9007
|
|
|
8967
9008
|
```bash
|
|
8968
|
-
python3 scripts/
|
|
8969
|
-
|
|
9009
|
+
npm run generate:agents # AI_TOOLKIT_NO_CUSTOM_RULES=1 python3 scripts/generate_agents_md.py > AGENTS.md
|
|
9010
|
+
npm run generate:llms # llms.txt + llms-full.txt
|
|
8970
9011
|
```
|
|
8971
9012
|
|
|
8972
|
-
|
|
8973
|
-
|
|
8974
|
-
|
|
9013
|
+
`generate:agents` sets `AI_TOOLKIT_NO_CUSTOM_RULES=1`. Running
|
|
9014
|
+
`generate_agents_md.py` bare picks up whatever is registered in the maintainer's
|
|
9015
|
+
own `~/.softspark/ai-toolkit/rules/`, which then ships inside `AGENTS.md`.
|
|
8975
9016
|
|
|
8976
|
-
|
|
9017
|
+
Check if anything actually changed:
|
|
8977
9018
|
|
|
8978
9019
|
```bash
|
|
8979
|
-
|
|
9020
|
+
git diff --stat AGENTS.md llms.txt llms-full.txt
|
|
8980
9021
|
```
|
|
8981
9022
|
|
|
8982
|
-
|
|
8983
|
-
- [ ] Bash skill count has NOT jumped unexpectedly since the previous release
|
|
8984
|
-
- [ ] Any newly added entry under `Skills with Bash + Write + Edit` matches a CHANGELOG bullet that justifies the broad scope
|
|
8985
|
-
- [ ] JSON form (`--permissions --json`) is available for automated drift dashboards
|
|
8986
|
-
|
|
8987
|
-
### 8.4 URL-sourced rules/hooks are checksum-pinned
|
|
8988
|
-
|
|
8989
|
-
`sources.json` is an envelope of the form `{"schema_version": 1, "rules"|"hooks": {...}}`, so the jq filter must pick the nested map before piping into the pin assertion. The URL-count in the success message ignores local `path:`-only entries — only entries with a `url:` field are pinned and counted.
|
|
9023
|
+
If no diff, the artifacts are already current. If there is a diff, stage them.
|
|
8990
9024
|
|
|
8991
|
-
|
|
8992
|
-
jq '.rules // .hooks // {}' ~/.softspark/ai-toolkit/rules/sources.json 2>/dev/null \
|
|
8993
|
-
| python3 -c "import json, sys; d=json.load(sys.stdin) or {}; bad=[n for n,v in d.items() if v.get('url') and not v.get('sha256')]; assert not bad, f'UNPINNED: {bad}'; url_n=sum(1 for v in d.values() if v.get('url')); print(f'RULE PIN OK: {url_n} URL rules, all with sha256')"
|
|
8994
|
-
jq '.hooks // {}' ~/.softspark/ai-toolkit/hooks/external/sources.json 2>/dev/null \
|
|
8995
|
-
| python3 -c "import json, sys; d=json.load(sys.stdin) or {}; bad=[n for n,v in d.items() if v.get('url') and not v.get('sha256')]; assert not bad, f'UNPINNED: {bad}'; url_n=sum(1 for v in d.values() if v.get('url')); print(f'HOOK PIN OK: {url_n} URL hooks, all with sha256')"
|
|
8996
|
-
```
|
|
9025
|
+
---
|
|
8997
9026
|
|
|
8998
|
-
|
|
8999
|
-
- [ ] Both commands print `... PIN OK`
|
|
9000
|
-
- [ ] If any entry is unpinned, the `register_url_source()` call missed passing `content=` — fix the call-site and retag
|
|
9027
|
+
## Phase 5: Validate, Audit, Test
|
|
9001
9028
|
|
|
9002
|
-
|
|
9029
|
+
Run the full quality gate:
|
|
9003
9030
|
|
|
9004
9031
|
```bash
|
|
9005
|
-
|
|
9006
|
-
|
|
9032
|
+
python3 scripts/validate.py --strict
|
|
9033
|
+
python3 scripts/audit_skills.py --ci
|
|
9034
|
+
python3 scripts/audit_skills.py --sarif > audit.sarif # MANDATORY — GHAS ingest
|
|
9035
|
+
python3 scripts/audit_skills.py --permissions # review Bash/Write/Edit footprint
|
|
9007
9036
|
|
|
9008
|
-
|
|
9009
|
-
|
|
9010
|
-
|
|
9037
|
+
# ShellCheck on hooks (added in 1.11.0). Mirrors the ci.yml "ShellCheck hooks"
|
|
9038
|
+
# job. NOT run by validate.py, npm test, OR publish.yml — so a hook with a
|
|
9039
|
+
# ShellCheck warning passes every other gate AND still publishes on tag while
|
|
9040
|
+
# turning main CI red. Run it here, before tagging.
|
|
9041
|
+
shellcheck --severity=warning app/hooks/*.sh && echo "OK: shellcheck clean"
|
|
9011
9042
|
|
|
9012
|
-
|
|
9043
|
+
# Registry / generator drift (added in 1.10.0). Meta-generators excluded.
|
|
9044
|
+
META="generate_agents_md.py|generate_llms_txt.py|generate_language_rules_skills.py"
|
|
9045
|
+
diff \
|
|
9046
|
+
<(grep -oE 'scripts/generate_[a-z_]+\.py' kb/reference/supported-tools-registry.md | sort -u) \
|
|
9047
|
+
<(ls scripts/generate_*.py | grep -vE "$META" | sort -u) \
|
|
9048
|
+
&& echo "OK: registry matches filesystem" \
|
|
9049
|
+
|| { echo "DRIFT: update supported-tools-registry.md before tagging"; exit 1; }
|
|
9013
9050
|
|
|
9014
|
-
|
|
9051
|
+
# Stage first IF this release adds or deletes a kb/ file. The test
|
|
9052
|
+
# "npm package KB files match the tracked release set" compares `git ls-files
|
|
9053
|
+
# kb` (the index) against what `npm pack` sees (the working tree), so an
|
|
9054
|
+
# unstaged addition reads as "extra" and an unstaged deletion as "missing".
|
|
9055
|
+
# Phase 6 stages, and it runs after this one, so the ordering fails the test
|
|
9056
|
+
# for any release that touches the KB. Staging early costs nothing.
|
|
9057
|
+
git status --porcelain kb/ | grep -qE '^(\?\?| D|\?M)' && git add -A kb/
|
|
9015
9058
|
|
|
9016
|
-
|
|
9059
|
+
# Run npm test ONCE, cache output, parse from file. The suite is 1400+ bats
|
|
9060
|
+
# cases — rerunning it per check wastes minutes. Do not pipe npm test into
|
|
9061
|
+
# tail/grep multiple times in the same session.
|
|
9062
|
+
npm test > /tmp/npm-test.log 2>&1
|
|
9063
|
+
tail -3 /tmp/npm-test.log
|
|
9064
|
+
echo "ok: $(grep -c '^ok ' /tmp/npm-test.log) | not ok: $(grep -c '^not ok' /tmp/npm-test.log)"
|
|
9065
|
+
```
|
|
9017
9066
|
|
|
9018
|
-
|
|
9067
|
+
**Expected results:**
|
|
9068
|
+
- `validate.py`: `Errors: 0 | Warnings: 0 | VALIDATION PASSED`
|
|
9069
|
+
- `audit_skills.py --ci`: `HIGH: 0 | WARN: 0` (INFO is acceptable)
|
|
9070
|
+
- `audit_skills.py --sarif`: valid JSON, non-empty `runs[0].tool.driver.rules`
|
|
9071
|
+
- `audit_skills.py --permissions`: review `Skills with Bash + Write + Edit` list — any newly-added skill with broad access MUST be justified in the CHANGELOG entry
|
|
9072
|
+
- `shellcheck --severity=warning app/hooks/*.sh`: no output, exit 0. A common false positive is `SC2034` on `INPUT` or env vars (e.g. `AI_TOOLKIT_HOOK_FORMAT`) that a *sourced* helper (`_hook-io.sh`) consumes — ShellCheck cannot see cross-file use. Fix with a `# shellcheck disable=SC2034` directive or `export`, matching `guard-destructive.sh`. Never tag with a red ShellCheck.
|
|
9073
|
+
- Registry drift: `OK: registry matches filesystem`. If `DRIFT:` appears, add the missing `scripts/generate_*.py` rows to `kb/reference/supported-tools-registry.md` before tagging.
|
|
9074
|
+
- `npm test`: `1..N` with zero `not ok` (read from the cached `/tmp/npm-test.log`, do not rerun)
|
|
9019
9075
|
|
|
9020
|
-
|
|
9076
|
+
> **Why this matters (v4.5.1 postmortem):** `publish.yml` runs only `validate.py` + `npm test`, so it published v4.5.0 even though the `main` CI `ShellCheck hooks` job was red on two `SC2034` warnings in a new hook. The publish workflow does **not** depend on the CI workflow. Until that is fixed, ShellCheck is a manual pre-tag gate — run it here every time.
|
|
9021
9077
|
|
|
9078
|
+
**One-liner:**
|
|
9022
9079
|
```bash
|
|
9023
|
-
|
|
9024
|
-
ai-toolkit install --local --editors cursor,windsurf,gemini,augment,codex,copilot \
|
|
9025
|
-
--profile full --dry-run 2>&1 \
|
|
9026
|
-
| grep -E "\\.cursor/(hooks\\.json|agents)|\\.devin/hooks\\.v1\\.json|\\.gemini/(settings\\.json|commands)|\\.augment/(agents|commands)|\\.codex/(agents|hooks)|\\.agents/skills|\\.github/(agents|skills|instructions|prompts|hooks)"
|
|
9080
|
+
python3 scripts/validate.py --strict && python3 scripts/audit_skills.py --ci && python3 scripts/audit_skills.py --sarif > audit.sarif && shellcheck --severity=warning app/hooks/*.sh && diff <(grep -oE 'scripts/generate_[a-z_]+\.py' kb/reference/supported-tools-registry.md | sort -u) <(ls scripts/generate_*.py | grep -vE 'generate_agents_md\.py|generate_llms_txt\.py|generate_language_rules_skills\.py' | sort -u) && npm test
|
|
9027
9081
|
```
|
|
9028
9082
|
|
|
9029
|
-
**
|
|
9030
|
-
-
|
|
9031
|
-
-
|
|
9032
|
-
-
|
|
9033
|
-
- [ ] `.augment/agents/` + `.augment/commands/` + `$HOME/.augment/settings.json`
|
|
9034
|
-
- [ ] `.codex/hooks.json` + `.codex/hooks/` + `.codex/agents/`
|
|
9035
|
-
- [ ] `.agents/skills/` (Codex native discovery path)
|
|
9036
|
-
- [ ] `.github/agents/` + `.github/skills/` + `.github/instructions/` + `.github/prompts/` + `.github/hooks/`
|
|
9083
|
+
**If tests fail:** Fix the issue, do NOT skip. Common failures:
|
|
9084
|
+
- Stale counts → re-run `generate:all` or fix README/ARCHITECTURE
|
|
9085
|
+
- Missing frontmatter → add to new KB docs
|
|
9086
|
+
- Broken symlink → `ai-toolkit doctor --fix`
|
|
9037
9087
|
|
|
9038
|
-
###
|
|
9088
|
+
### Phase 5c: Licensing Gate (v4.20.0+)
|
|
9089
|
+
|
|
9090
|
+
The project is Apache-2.0. Attribution only works if the artefact actually
|
|
9091
|
+
carries it, and every part of that is mechanically checkable.
|
|
9039
9092
|
|
|
9040
9093
|
```bash
|
|
9041
|
-
|
|
9042
|
-
|
|
9043
|
-
| grep -q "Would generate: .agents/skills" && echo "OK: Codex skills emit at minimal"
|
|
9044
|
-
ai-toolkit install --local --editors codex --profile standard --codex-skills --dry-run 2>&1 \
|
|
9045
|
-
| grep -q "Would refresh: .agents/skills" && echo "OK: legacy flag explicitly refreshes the same catalog"
|
|
9094
|
+
# The whole gate, enforced in CI. Run it here so a failure is caught before tagging.
|
|
9095
|
+
npx bats tests/test_licensing.bats
|
|
9046
9096
|
```
|
|
9047
9097
|
|
|
9048
|
-
|
|
9049
|
-
- [ ] Selecting Codex emits all skills under `.agents/skills/` at every profile
|
|
9050
|
-
- [ ] `--codex-skills` remains an explicit refresh alias; it is not required for installation
|
|
9051
|
-
- [ ] No profile emits `.codex/skills/`
|
|
9098
|
+
The seven assertions, and why each exists:
|
|
9052
9099
|
|
|
9053
|
-
|
|
9100
|
+
| Check | Fails when |
|
|
9101
|
+
|---|---|
|
|
9102
|
+
| Every shipped source file carries an SPDX header | A new `.py`/`.sh`/`.js`/`.bats` file was added without one — the common case, and the reason this is a test rather than a habit |
|
|
9103
|
+
| Headers name Apache-2.0 and nothing else | A file was copied in from an MIT/GPL source with its own header intact |
|
|
9104
|
+
| **No** markdown file carries an SPDX header | Someone "helpfully" ran the header script over `app/skills/` — headers there sit above parsed frontmatter and bill every session for it |
|
|
9105
|
+
| `LICENSE` is the complete Apache 2.0 text | The file was truncated or replaced with a summary |
|
|
9106
|
+
| `NOTICE` carries attribution, the source URL, §4(d) and the MIT-era notice | The attribution mechanism was gutted |
|
|
9107
|
+
| `LICENSE` **and** `NOTICE` ship in the npm package | `package.json` `files` lost an entry — a NOTICE that never reaches the consumer cannot satisfy §4(d) |
|
|
9108
|
+
| Every manifest declaring a licence declares Apache-2.0 | `package.json`, `manifest.json`, `plugin.json` and `package-lock.json` drifted apart |
|
|
9054
9109
|
|
|
9055
|
-
The
|
|
9056
|
-
|
|
9057
|
-
- Gemini hooks (`.gemini/settings.json`)
|
|
9110
|
+
**Adding source files in this release?** The header goes *after* the shebang,
|
|
9111
|
+
never before it. Short SPDX form:
|
|
9058
9112
|
|
|
9059
|
-
```bash
|
|
9060
|
-
D=/tmp/aitk-breaking-${RANDOM} && mkdir -p "$D" && cd "$D" && git init -q
|
|
9061
|
-
ai-toolkit install --local --editors copilot,gemini --profile standard --dry-run 2>&1 \
|
|
9062
|
-
| tee /tmp/aitk-breaking.log
|
|
9063
|
-
grep -q "\\.github/instructions/" /tmp/aitk-breaking.log && echo "OK: Copilot dir layout at standard"
|
|
9064
|
-
grep -q "\\.github/agents/.*\\.github/skills/" /tmp/aitk-breaking.log && echo "OK: Copilot agents and skills at standard"
|
|
9065
|
-
grep -q "\\.github/hooks/" /tmp/aitk-breaking.log && echo "OK: Copilot hooks at standard"
|
|
9066
|
-
grep -q "\\.gemini/settings\\.json hooks" /tmp/aitk-breaking.log && echo "OK: Gemini hooks at standard"
|
|
9067
9113
|
```
|
|
9114
|
+
# SPDX-License-Identifier: Apache-2.0
|
|
9115
|
+
# Copyright 2024-2026 Lukasz Krzemien (biuro@softspark.eu)
|
|
9116
|
+
# Source: https://github.com/softspark/ai-toolkit
|
|
9117
|
+
```
|
|
9118
|
+
|
|
9119
|
+
`//` for JavaScript. Full convention and the reasoning behind the markdown
|
|
9120
|
+
exclusion: [Licensing](../reference/licensing.md).
|
|
9121
|
+
|
|
9122
|
+
**Changing the licence itself?** Do not hand-type the licence text. Take it
|
|
9123
|
+
verbatim from a published copy and cross-verify against a second independent
|
|
9124
|
+
copy before writing `LICENSE` — a rendered or summarised licence is not the
|
|
9125
|
+
licence. Prior releases stay under their original terms; a licence change
|
|
9126
|
+
applies going forward and revokes nothing already granted.
|
|
9127
|
+
|
|
9128
|
+
### Phase 5a: Supply-Chain Hardening Verification (v2.8.0+)
|
|
9068
9129
|
|
|
9069
|
-
|
|
9130
|
+
These checks enforce the security standard introduced in v2.8.0. Do NOT tag a release until all pass.
|
|
9070
9131
|
|
|
9071
|
-
|
|
9132
|
+
**1. Publish workflow emits provenance:**
|
|
9072
9133
|
|
|
9073
9134
|
```bash
|
|
9074
|
-
|
|
9075
|
-
# Sort file paths before hashing — find traversal order follows inode order,
|
|
9076
|
-
# which can shift between runs even when content is byte-identical, producing
|
|
9077
|
-
# false FAIL signals.
|
|
9078
|
-
ai-toolkit install --local --editors cursor,gemini --profile full >/dev/null 2>&1
|
|
9079
|
-
SHA1=$(find .cursor .gemini -type f -print0 | LC_ALL=C sort -z | xargs -0 shasum | shasum | awk '{print $1}')
|
|
9080
|
-
ai-toolkit install --local --editors cursor,gemini --profile full >/dev/null 2>&1
|
|
9081
|
-
SHA2=$(find .cursor .gemini -type f -print0 | LC_ALL=C sort -z | xargs -0 shasum | shasum | awk '{print $1}')
|
|
9082
|
-
[ "$SHA1" = "$SHA2" ] && echo "OK: idempotent" || echo "FAIL: install is not idempotent"
|
|
9135
|
+
grep -E '\-\-provenance|id-token: write' .github/workflows/publish.yml
|
|
9083
9136
|
```
|
|
9084
9137
|
|
|
9085
|
-
|
|
9086
|
-
|
|
9087
|
-
### 9.5 Live-install JSON outputs parse
|
|
9138
|
+
- [ ] Both markers present (`--provenance` flag + `id-token: write` permission)
|
|
9139
|
+
- [ ] Any PR that changes `publish.yml` REQUIRES an approved security review
|
|
9088
9140
|
|
|
9089
|
-
|
|
9141
|
+
**2. URL-sourced rules and hooks are checksum-pinned:**
|
|
9090
9142
|
|
|
9091
9143
|
```bash
|
|
9092
|
-
|
|
9093
|
-
|
|
9094
|
-
|
|
9095
|
-
|
|
9096
|
-
done
|
|
9144
|
+
# On a machine that has consumed URL rules/hooks at least once
|
|
9145
|
+
# (schema_version 1: entries live under the .rules / .hooks key):
|
|
9146
|
+
jq '.rules | to_entries | map(select(.value.url != null and (.value.sha256 // "" | length) == 0))' ~/.softspark/ai-toolkit/rules/sources.json
|
|
9147
|
+
jq '.hooks | to_entries | map(select(.value.url != null and (.value.sha256 // "" | length) == 0))' ~/.softspark/ai-toolkit/hooks/external/sources.json
|
|
9097
9148
|
```
|
|
9098
9149
|
|
|
9099
|
-
|
|
9100
|
-
|
|
9101
|
-
### 9.6 Registry / generator drift check
|
|
9150
|
+
- [ ] Both queries return empty arrays (every URL entry has a `sha256`)
|
|
9151
|
+
- [ ] If not, run `ai-toolkit update` to backfill missing hashes before tagging
|
|
9102
9152
|
|
|
9103
|
-
|
|
9153
|
+
**3. Audit SARIF output is well-formed:**
|
|
9104
9154
|
|
|
9105
9155
|
```bash
|
|
9106
|
-
|
|
9107
|
-
REG=$(grep -oE 'scripts/generate_[a-z_]+\.py' kb/reference/supported-tools-registry.md | sort -u)
|
|
9108
|
-
FS=$(ls scripts/generate_*.py | grep -vE "$META" | sort -u)
|
|
9109
|
-
diff <(echo "$REG") <(echo "$FS") && echo "OK: registry matches filesystem" || echo "DRIFT: update supported-tools-registry.md"
|
|
9156
|
+
python3 scripts/audit_skills.py --sarif | python3 -c "import json, sys; d=json.load(sys.stdin); assert d['version']=='2.1.0' and d['runs'][0]['tool']['driver']['name']; print('SARIF OK')"
|
|
9110
9157
|
```
|
|
9111
9158
|
|
|
9112
|
-
|
|
9159
|
+
- [ ] Prints `SARIF OK`
|
|
9160
|
+
- [ ] If the script ever grows new rule classes, extend the SARIF `rules[]` coverage before releasing
|
|
9113
9161
|
|
|
9114
|
-
|
|
9162
|
+
**4. Strict-pin mode passes on CI** (optional, recommended for stable branches):
|
|
9115
9163
|
|
|
9116
9164
|
```bash
|
|
9117
|
-
|
|
9118
|
-
D=/tmp/aitk-claude-app-${RANDOM}
|
|
9119
|
-
python3 scripts/claude_app.py export --output "$D.zip" --no-custom-rules
|
|
9120
|
-
python3 -c "import zipfile; z=zipfile.ZipFile('$D.zip'); assert '.claude-plugin/plugin.json' in z.namelist(); assert 'claude-app/skills/ai-toolkit-rules/SKILL.md' in z.namelist(); print('OK: Claude app archive')"
|
|
9165
|
+
AI_TOOLKIT_STRICT_PIN=1 ai-toolkit update --dry-run
|
|
9121
9166
|
```
|
|
9122
9167
|
|
|
9123
|
-
|
|
9124
|
-
|
|
9168
|
+
- [ ] Exit 0, no `CHECKSUM CHANGED` line
|
|
9169
|
+
- [ ] Any unexpected upstream change blocks the release until explicitly approved
|
|
9125
9170
|
|
|
9126
9171
|
---
|
|
9127
9172
|
|
|
9128
|
-
## Phase
|
|
9129
|
-
|
|
9130
|
-
Run this phase whenever the release changes the `dsh` target, package pins, preset lifecycle, or DSH compatibility documentation. Use a new task-specific `DSH_HOME`; never replace `HOME` or reuse a regular profile.
|
|
9173
|
+
## Phase 6: Commit
|
|
9131
9174
|
|
|
9132
|
-
|
|
9175
|
+
Stage all release files:
|
|
9133
9176
|
|
|
9134
9177
|
```bash
|
|
9135
|
-
|
|
9136
|
-
|
|
9137
|
-
|
|
9138
|
-
|
|
9139
|
-
|
|
9140
|
-
|
|
9178
|
+
git add package.json manifest.json app/.claude-plugin/plugin.json
|
|
9179
|
+
git add package-lock.json
|
|
9180
|
+
git add CHANGELOG.md
|
|
9181
|
+
git add llms.txt llms-full.txt
|
|
9182
|
+
# NOTE: AGENTS.md, GEMINI.md, and .github/copilot-instructions.md are generated
|
|
9183
|
+
# editor configs and are gitignored — do NOT commit them. `prepublishOnly` runs
|
|
9184
|
+
# `npm run generate:all`, so the shipped package (which lists AGENTS.md in
|
|
9185
|
+
# package.json `files`) gets a fresh copy at publish time.
|
|
9186
|
+
git add -p # review and stage any other changes
|
|
9141
9187
|
```
|
|
9142
9188
|
|
|
9143
|
-
|
|
9189
|
+
Commit:
|
|
9144
9190
|
|
|
9145
|
-
|
|
9146
|
-
|
|
9191
|
+
```bash
|
|
9192
|
+
git commit -m "chore: release vX.Y.Z"
|
|
9193
|
+
```
|
|
9147
9194
|
|
|
9148
|
-
|
|
9195
|
+
---
|
|
9196
|
+
|
|
9197
|
+
## Phase 7: Push Branch, Verify CI, Tag, and Push Tag
|
|
9149
9198
|
|
|
9150
9199
|
```bash
|
|
9151
|
-
|
|
9152
|
-
|
|
9200
|
+
RELEASE_SHA=$(git rev-parse HEAD)
|
|
9201
|
+
git push origin main || { echo "FAIL: release commit push failed"; exit 1; }
|
|
9153
9202
|
|
|
9154
|
-
|
|
9203
|
+
# Bind the gate to the exact release commit. GitHub run registration is
|
|
9204
|
+
# asynchronous, so retry at most five times and log every attempt.
|
|
9205
|
+
RUN_ID=""
|
|
9206
|
+
for ATTEMPT in 1 2 3 4 5; do
|
|
9207
|
+
RUN_ID=$(gh run list --workflow ci.yml --event push --commit "$RELEASE_SHA" \
|
|
9208
|
+
--limit 1 --json databaseId --jq '.[0].databaseId')
|
|
9209
|
+
[ -n "$RUN_ID" ] && break
|
|
9210
|
+
echo "Waiting for CI run registration ($ATTEMPT/5)" >&2
|
|
9211
|
+
[ "$ATTEMPT" -eq 5 ] || sleep 60
|
|
9212
|
+
done
|
|
9213
|
+
test -n "$RUN_ID" || { echo "FAIL: CI run not found for release commit"; exit 1; }
|
|
9214
|
+
gh run watch "$RUN_ID" --exit-status \
|
|
9215
|
+
|| { echo "FAIL: release commit CI failed"; exit 1; }
|
|
9155
9216
|
|
|
9156
|
-
|
|
9217
|
+
# publish.yml is Ubuntu-only. Require both full CI matrix jobs before creating
|
|
9218
|
+
# the tag so a green publish job cannot bypass a red macOS test.
|
|
9219
|
+
for OS in ubuntu-latest macos-latest; do
|
|
9220
|
+
gh run view "$RUN_ID" --json jobs \
|
|
9221
|
+
--jq ".jobs[] | select(.name == \"Bats test suite ($OS)\") | .conclusion" \
|
|
9222
|
+
| grep -qx success || { echo "FAIL: $OS test job not green"; exit 1; }
|
|
9223
|
+
done
|
|
9157
9224
|
|
|
9158
|
-
|
|
9225
|
+
# Refuse to tag if either local or remote main moved while CI was running.
|
|
9226
|
+
test "$(git rev-parse HEAD)" = "$RELEASE_SHA" \
|
|
9227
|
+
|| { echo "FAIL: local HEAD changed during CI"; exit 1; }
|
|
9228
|
+
git fetch origin main || { echo "FAIL: origin/main refresh failed"; exit 1; }
|
|
9229
|
+
test "$(git rev-parse origin/main)" = "$RELEASE_SHA" \
|
|
9230
|
+
|| { echo "FAIL: origin/main changed during CI"; exit 1; }
|
|
9159
9231
|
|
|
9160
|
-
|
|
9232
|
+
git tag vX.Y.Z "$RELEASE_SHA"
|
|
9161
9233
|
|
|
9162
|
-
|
|
9163
|
-
|
|
9164
|
-
|
|
9165
|
-
|
|
9234
|
+
# Assert the tag before pushing it. Both checks are one line each and both
|
|
9235
|
+
# have caught a real broken release.
|
|
9236
|
+
test "$(git rev-parse vX.Y.Z)" = "$RELEASE_SHA" \
|
|
9237
|
+
|| { echo "FAIL: tag is not on the tested release commit"; exit 1; }
|
|
9238
|
+
git show --no-patch --format=%s vX.Y.Z | grep -qx "chore: release vX.Y.Z" \
|
|
9239
|
+
|| { echo "FAIL: tag is not on the chore: release commit"; exit 1; }
|
|
9240
|
+
|
|
9241
|
+
# Push the single release tag by its full ref.
|
|
9242
|
+
git push origin refs/tags/vX.Y.Z
|
|
9166
9243
|
```
|
|
9167
9244
|
|
|
9168
|
-
|
|
9245
|
+
**Why branch CI comes before the tag (v4.30.2 postmortem).** The publish
|
|
9246
|
+
workflow runs only on Ubuntu and can publish while the separate macOS matrix job
|
|
9247
|
+
is red. `v4.30.2` exposed this with a macOS-only DSH timing failure. A release
|
|
9248
|
+
commit must therefore pass the complete Ubuntu/macOS branch workflow before its
|
|
9249
|
+
tag exists; a successful publish workflow is not a substitute for green CI.
|
|
9169
9250
|
|
|
9170
|
-
|
|
9171
|
-
|
|
9172
|
-
|
|
9173
|
-
|
|
9251
|
+
**Why the assertions (v4.19.0 postmortem).** v4.19.0 was tagged on a commit
|
|
9252
|
+
that contained only a KB document and still carried `package.json` version
|
|
9253
|
+
`4.18.0`; the actual release sat in the commit above it under a recycled
|
|
9254
|
+
`fix:` message. `publish.yml` fired, tried to publish a version already on
|
|
9255
|
+
npm, and failed. Nothing on npm, a tag pointing at the wrong tree, and the
|
|
9256
|
+
only way out was rewriting a pushed commit. Both assertions above catch this
|
|
9257
|
+
in under a second. Run them.
|
|
9174
9258
|
|
|
9175
|
-
|
|
9259
|
+
**Never `git push --tags`.** It pushes every local tag at once, and GitHub
|
|
9260
|
+
suppresses tag-triggered workflow runs when many tags arrive in a single push
|
|
9261
|
+
— the workflow silently does not fire and nothing publishes. Push the single
|
|
9262
|
+
release tag by its full ref, as above. (Sibling evidence: this is exactly how
|
|
9263
|
+
rag-mcp's `1.0.3` image build was skipped, when a `--tags` push carried 37
|
|
9264
|
+
tags at once.)
|
|
9176
9265
|
|
|
9177
|
-
|
|
9178
|
-
|
|
9179
|
-
|
|
9180
|
-
|
|
9266
|
+
This triggers `.github/workflows/publish.yml` which:
|
|
9267
|
+
1. Runs `validate.py --strict`
|
|
9268
|
+
2. Runs `npm test`
|
|
9269
|
+
3. Publishes to npm as `@softspark/ai-toolkit@X.Y.Z` with `--provenance` (SLSA v1 build attestation)
|
|
9181
9270
|
|
|
9182
|
-
|
|
9271
|
+
**Provenance is non-negotiable.** If `id-token: write` permission or the `--provenance` flag is missing from `publish.yml`, fix it BEFORE tagging — an unsigned release is a regression against the v2.8.0 standard.
|
|
9183
9272
|
|
|
9184
|
-
|
|
9185
|
-
|
|
9273
|
+
**After CI completes:** Run the [Release Verification SOP](sop-release-verification.md)
|
|
9274
|
+
to smoke-test the published package AND verify the provenance attestation landed on npm.
|
|
9186
9275
|
|
|
9187
|
-
|
|
9276
|
+
---
|
|
9277
|
+
|
|
9278
|
+
## Rollback
|
|
9279
|
+
|
|
9280
|
+
If a bad release was published:
|
|
9188
9281
|
|
|
9189
9282
|
```bash
|
|
9190
|
-
|
|
9191
|
-
ai-toolkit
|
|
9283
|
+
# Unpublish from npm (within 72h)
|
|
9284
|
+
npm unpublish @softspark/ai-toolkit@X.Y.Z
|
|
9285
|
+
|
|
9286
|
+
# Or deprecate (preferred — doesn't break existing installs)
|
|
9287
|
+
npm deprecate @softspark/ai-toolkit@X.Y.Z "Known issue: <description>. Use vA.B.C instead."
|
|
9288
|
+
|
|
9289
|
+
# Delete tag
|
|
9290
|
+
git tag -d vX.Y.Z
|
|
9291
|
+
git push origin --delete vX.Y.Z
|
|
9192
9292
|
```
|
|
9193
9293
|
|
|
9194
9294
|
---
|
|
9195
9295
|
|
|
9196
|
-
##
|
|
9296
|
+
## Checklist Summary
|
|
9197
9297
|
|
|
9198
|
-
|
|
|
9199
|
-
|
|
9200
|
-
|
|
|
9201
|
-
|
|
|
9202
|
-
|
|
|
9203
|
-
|
|
|
9204
|
-
|
|
|
9205
|
-
|
|
|
9206
|
-
|
|
|
9207
|
-
|
|
|
9208
|
-
|
|
|
9209
|
-
|
|
|
9210
|
-
|
|
|
9211
|
-
|
|
|
9212
|
-
|
|
|
9213
|
-
|
|
|
9214
|
-
|
|
|
9215
|
-
|
|
|
9216
|
-
|
|
|
9298
|
+
| # | Step | Command / Action | Pass Criteria |
|
|
9299
|
+
|---|------|-----------------|---------------|
|
|
9300
|
+
| 0a | Ecosystem drift check | `ecosystem_doctor.py --format text` | All tools Clean, or drift classified and resolved |
|
|
9301
|
+
| 0b | Ecosystem snapshot refresh | `ecosystem_doctor.py --update` | `benchmarks/ecosystem-doctor-snapshot.json` updated |
|
|
9302
|
+
| 0c | Ecosystem gate | `ecosystem_doctor.py --offline --check` | Exit 0 |
|
|
9303
|
+
| 1 | Version bump type | Decide patch/minor/major | — |
|
|
9304
|
+
| 2 | `package.json` version | Edit `"version"` | Matches target |
|
|
9305
|
+
| 3 | `manifest.json` version | Edit `"version"` | Matches target |
|
|
9306
|
+
| 4 | `plugin.json` version | Edit `"version"` | Matches target |
|
|
9307
|
+
| 5 | `package-lock.json` | `npm install --package-lock-only` | Matches target |
|
|
9308
|
+
| 6 | Count sync | Check `package.json` description, README | `validate.py` passes |
|
|
9309
|
+
| 7 | CHANGELOG.md | Add release entry (incl. `Ecosystem` subsection if any B/D/E/F drift) | Entry exists for vX.Y.Z |
|
|
9310
|
+
| 8 | Regenerate artifacts | `generate_agents_md.py`, `generate_llms_txt.py` | No unexpected diff |
|
|
9311
|
+
| 9 | Validate | `validate.py --strict` | 0 errors, 0 warnings |
|
|
9312
|
+
| 10 | Security audit (CI mode) | `audit_skills.py --ci` | 0 HIGH |
|
|
9313
|
+
| 11 | Security audit (SARIF) | `audit_skills.py --sarif` | Valid SARIF 2.1.0 JSON |
|
|
9314
|
+
| 12 | Per-skill permissions | `audit_skills.py --permissions` | New broad-access skills justified in CHANGELOG |
|
|
9315
|
+
| 13 | ShellCheck hooks | `shellcheck --severity=warning app/hooks/*.sh` | Exit 0, no output (mirrors ci.yml; publish.yml does NOT run it) |
|
|
9316
|
+
| 14 | Provenance flag check | `grep -- '--provenance' .github/workflows/publish.yml` | Present |
|
|
9317
|
+
| 15 | Checksum-pin backfill | `sources.json` entries all have `sha256` | No unpinned URL sources |
|
|
9318
|
+
| 15a | Licensing gate | `npx bats tests/test_licensing.bats` | 7/7 — SPDX headers, LICENSE, NOTICE, npm `files`, manifest consistency |
|
|
9319
|
+
| 16 | Tests | `git add -A kb/` if the KB changed, then `npm test` | All pass |
|
|
9320
|
+
| 17 | Commit | `git commit` | Clean working tree |
|
|
9321
|
+
| 18 | Push branch | `git push origin main` | Exact release commit is on `origin/main` |
|
|
9322
|
+
| 18a | Full branch CI | `gh run watch "$RUN_ID" --exit-status` plus matrix job assertions | Ubuntu and macOS Bats jobs both conclude `success` |
|
|
9323
|
+
| 19 | Tag | `git tag vX.Y.Z` | Tag exists only after green branch CI |
|
|
9324
|
+
| 19a | Tag is on tested SHA | `test "$(git rev-parse vX.Y.Z)" = "$RELEASE_SHA"` | Exit 0 |
|
|
9325
|
+
| 19b | Tag is on the release commit | `git show --no-patch --format=%s vX.Y.Z` | Reads `chore: release vX.Y.Z` |
|
|
9326
|
+
| 20 | Push the single tag | `git push origin refs/tags/vX.Y.Z` | Publish CI triggered with `id-token: write`. Never `--tags`. |
|
|
9217
9327
|
|
|
9218
9328
|
---
|
|
9219
9329
|
|
|
@@ -9801,7 +9911,7 @@ Three tiers determine how to approach a task:
|
|
|
9801
9911
|
|------|-------|-----------|-------|
|
|
9802
9912
|
| Task | `disable-model-invocation: true` | User via `/skill` only | 32 |
|
|
9803
9913
|
| Hybrid | (neither) | User via `/skill` + agent knowledge | 31 |
|
|
9804
|
-
| Knowledge | `user-invocable: false` | Claude auto-loads |
|
|
9914
|
+
| Knowledge | `user-invocable: false` | Claude auto-loads | 51 |
|
|
9805
9915
|
|
|
9806
9916
|
## Multi-Agent Execution
|
|
9807
9917
|
|
|
@@ -10346,7 +10456,7 @@ category: reference
|
|
|
10346
10456
|
service: ai-toolkit
|
|
10347
10457
|
tags: [cli, commands, reference, install, update, plugin, mcp, telemetry]
|
|
10348
10458
|
created: "2026-04-13"
|
|
10349
|
-
last_updated: "2026-
|
|
10459
|
+
last_updated: "2026-09-04"
|
|
10350
10460
|
description: "Complete CLI reference for all ai-toolkit commands, options, and flags."
|
|
10351
10461
|
---
|
|
10352
10462
|
|
|
@@ -10361,6 +10471,7 @@ Usage: ai-toolkit <command> [options]
|
|
|
10361
10471
|
| Command | Description |
|
|
10362
10472
|
|---------|-------------|
|
|
10363
10473
|
| `install` | First-time global install into `~/.claude/` + Cursor, Windsurf, Gemini |
|
|
10474
|
+
| `install --language-skills detected\|all` | `detected` (default): turn off `<lang>-rules`/`<lang>-patterns` skills for languages no registered project uses, via `skillOverrides` in `~/.claude/settings.json` (entries are tracked in `state.json` and restored when a project brings the language back; a user's own override is never touched); `all`: keep every language skill on. The choice persists across `install`/`update` |
|
|
10364
10475
|
| `install --local` | Claude Code configs only; add `--editors all` or `--editors cursor,aider` for other tools |
|
|
10365
10476
|
| `update` | Re-apply toolkit after `npm install -g @softspark/ai-toolkit@latest` |
|
|
10366
10477
|
| `update --local` | Re-apply + auto-detect editors from existing project files |
|
|
@@ -10368,7 +10479,7 @@ Usage: ai-toolkit <command> [options]
|
|
|
10368
10479
|
| `status` | Show installed modules and version |
|
|
10369
10480
|
| `uninstall` | Remove toolkit from `~/.claude/` |
|
|
10370
10481
|
| `validate` | Verify toolkit integrity (`--strict` for CI-grade, warnings = errors) |
|
|
10371
|
-
| `doctor` | Diagnose install health, hooks, quick-win assets,
|
|
10482
|
+
| `doctor` | Diagnose install health, hooks, quick-win assets, artifact drift, context budget (est. resident tokens of the skill/agent listings and user rules, plus skills with zero recorded use; read-only, prints the `skillOverrides` key to paste), and permission rules (`permissions.allow` wildcards on interpreters, task runners, package installs, `gh api`, `curl`, `git fetch`, destructive commands; warns only, never edits) |
|
|
10372
10483
|
| `doctor --fix` | Auto-repair broken symlinks, missing hooks, stale artifacts |
|
|
10373
10484
|
| `eject [dir]` | Export standalone config (no symlinks, no toolkit dependency) |
|
|
10374
10485
|
| `claude-app export [--output FILE] [--no-custom-rules] [--verify]` | Build an uploadable Claude Chat/Desktop/Cowork plugin ZIP and global-instructions file |
|
|
@@ -11738,7 +11849,7 @@ running the generators directly:
|
|
|
11738
11849
|
|
|
11739
11850
|
- `scripts/generate_copilot.py` / `generate_copilot_hooks.py` contract tests
|
|
11740
11851
|
(`tests/test_copilot.bats`, `tests/test_copilot_hooks.bats`).
|
|
11741
|
-
- Release layout check in `kb/procedures/release-verification
|
|
11852
|
+
- Release layout check in `kb/procedures/sop-release-verification.md` asserts the
|
|
11742
11853
|
`.github/{agents,skills,instructions,prompts,hooks}` surfaces.
|
|
11743
11854
|
- `validate.py --strict` + `audit_skills.py --ci` in CI.
|
|
11744
11855
|
|
|
@@ -12673,9 +12784,9 @@ title: "Global Install Model"
|
|
|
12673
12784
|
category: reference
|
|
12674
12785
|
service: ai-toolkit
|
|
12675
12786
|
tags: [install, global, claude, codex, plugins, local-setup]
|
|
12676
|
-
version: "3.
|
|
12787
|
+
version: "3.5.0"
|
|
12677
12788
|
created: "2026-03-26"
|
|
12678
|
-
last_updated: "2026-
|
|
12789
|
+
last_updated: "2026-09-04"
|
|
12679
12790
|
description: "Reference description of Claude Code global install, Claude app plugin export, project-local editor setup, global Codex plugin layering, and command responsibilities in ai-toolkit."
|
|
12680
12791
|
---
|
|
12681
12792
|
|
|
@@ -12687,6 +12798,19 @@ description: "Reference description of Claude Code global install, Claude app pl
|
|
|
12687
12798
|
|
|
12688
12799
|
That means one machine-level install provides agents, skills, hooks, constitution, and rule files to every project without committing toolkit boilerplate into each repository.
|
|
12689
12800
|
|
|
12801
|
+
Language knowledge skills are the one part of the global install that is
|
|
12802
|
+
scoped by evidence (v4.32.0). Every `<lang>-rules` / `<lang>-patterns` skill is
|
|
12803
|
+
still symlinked, but `install` and `update` read the languages detected across
|
|
12804
|
+
the projects registered in `projects.json` and set `skillOverrides: off` in
|
|
12805
|
+
`~/.claude/settings.json` for the others, so their descriptions stop loading
|
|
12806
|
+
into every session. The entries the toolkit wrote are listed in `state.json`
|
|
12807
|
+
under `managed_skill_overrides`; only those are ever removed again (when a
|
|
12808
|
+
newly registered project brings the language back), a user's own override is
|
|
12809
|
+
left alone. With no registered project on disk nothing is disabled.
|
|
12810
|
+
`--language-skills all` restores everything and persists the choice
|
|
12811
|
+
(`language_skill_scope` in `state.json`); `ai-toolkit doctor` reports the
|
|
12812
|
+
resulting listing size and zero-use skills.
|
|
12813
|
+
|
|
12690
12814
|
Other editor targets are opt-in and only use documented file surfaces. Cursor
|
|
12691
12815
|
rules stay project-local because Cursor's global user rules are managed through
|
|
12692
12816
|
the settings UI, not a stable merge-safe file. Codex supports both project and
|
|
@@ -12729,7 +12853,7 @@ The `--profile` flag controls how much of each editor's native surface is activa
|
|
|
12729
12853
|
|---------|-----------|----------|
|
|
12730
12854
|
| `minimal` | Smallest editor surface. Copilot still receives its root instructions, native agents, and self-contained skills; Codex still receives instructions, agents, skills, and native safety hooks. | You want the smallest supported footprint. |
|
|
12731
12855
|
| `standard` (default) | Claude Code + editor rule files. Includes Gemini and Antigravity hooks plus native Copilot instructions, agents, portable skills, and hooks. | Day-to-day installs. Most users. |
|
|
12732
|
-
| `strict` | Everything in `standard` plus git-hook wiring for commit-time safety checks
|
|
12856
|
+
| `strict` | Everything in `standard` plus git-hook wiring for commit-time safety checks and the `git-team` common rule (feature branches, PR size, required approval, review SLA). | Tight team with zero tolerance for drift. A solo maintainer releasing straight to `main` wants `standard`, which keeps the solo-safe `git-workflow` core only. |
|
|
12733
12857
|
| `full` | Every native surface across every editor: hooks, sub-agents, custom commands, skill pointers for Cursor / Windsurf / Gemini / Augment / Antigravity. | You want maximum coverage and understand that each editor will carry generated files under its own layout. |
|
|
12734
12858
|
|
|
12735
12859
|
Codex installs materialize the full skill catalog under `.agents/skills/`
|
|
@@ -13118,9 +13242,9 @@ title: "Hooks Catalog"
|
|
|
13118
13242
|
category: reference
|
|
13119
13243
|
service: ai-toolkit
|
|
13120
13244
|
tags: [hooks, quality, safety, enforcement, settings.json]
|
|
13121
|
-
version: "1.
|
|
13245
|
+
version: "1.12.0"
|
|
13122
13246
|
created: "2026-03-27"
|
|
13123
|
-
last_updated: "2026-
|
|
13247
|
+
last_updated: "2026-09-04"
|
|
13124
13248
|
description: "Complete reference of all ai-toolkit hooks: events, scripts, installation, and runtime behavior."
|
|
13125
13249
|
---
|
|
13126
13250
|
|
|
@@ -13305,6 +13429,7 @@ Skipped when `TOOLKIT_HOOK_PROFILE=minimal`.
|
|
|
13305
13429
|
| Matcher | *(all)* |
|
|
13306
13430
|
| Script | `~/.softspark/ai-toolkit/hooks/quality-check.sh` |
|
|
13307
13431
|
| Fires | After every Claude response |
|
|
13432
|
+
| Mode | Background (`"async": true`). The linter (`npx tsc`, `ruff`) was the bulk of a 3.3 s median Stop chain; its output is advisory and reaches you when it finishes instead of holding the turn open |
|
|
13308
13433
|
|
|
13309
13434
|
**Action:** Runs language-appropriate linter:
|
|
13310
13435
|
- Python: `ruff check .`
|
|
@@ -13323,6 +13448,7 @@ Skipped when `TOOLKIT_HOOK_PROFILE=minimal`.
|
|
|
13323
13448
|
| Matcher | *(all)* |
|
|
13324
13449
|
| Script | `~/.softspark/ai-toolkit/hooks/save-session.sh` |
|
|
13325
13450
|
| Fires | After every Claude response |
|
|
13451
|
+
| Mode | Background (`"async": true`): writes a file and exits 0, nothing to wait for |
|
|
13326
13452
|
|
|
13327
13453
|
**Action:** Writes enriched session context to the per-repo session store (`~/.softspark/ai-toolkit/sessions/<repo-key>/session-context.md`) for cross-session persistence. Captures:
|
|
13328
13454
|
- Session ID and last assistant message (first 5 lines)
|
|
@@ -13340,7 +13466,7 @@ Skipped when `TOOLKIT_HOOK_PROFILE=minimal`.
|
|
|
13340
13466
|
| Script | `~/.softspark/ai-toolkit/hooks/quality-gate.sh` |
|
|
13341
13467
|
| Fires | Before Claude is allowed to finish a response |
|
|
13342
13468
|
|
|
13343
|
-
**Action:** Runs lint/typecheck. **Blocks stopping (exit 2)** if errors found, so Claude must continue and fix the issues. Missing local tooling is reported as skipped rather than blocking the session.
|
|
13469
|
+
**Action:** Runs lint/typecheck. **Blocks stopping (exit 2)** if errors found, so Claude must continue and fix the issues. Missing local tooling is reported as skipped rather than blocking the session. Ruff runs only when the project configured it (`ruff.toml`, `.ruff.toml`, or a `[tool.ruff]` table in `pyproject.toml`); a `pyproject.toml` that only carries build or pytest/mypy metadata does not trigger `ruff check .` under whatever configuration the machine resolves (v4.32.0).
|
|
13344
13470
|
|
|
13345
13471
|
Skipped when `TOOLKIT_HOOK_PROFILE=minimal`.
|
|
13346
13472
|
|
|
@@ -13938,9 +14064,9 @@ title: "Language Rules System"
|
|
|
13938
14064
|
category: reference
|
|
13939
14065
|
service: ai-toolkit
|
|
13940
14066
|
tags: [rules, languages, coding-style, testing, patterns, security]
|
|
13941
|
-
version: "2.
|
|
14067
|
+
version: "2.2.0"
|
|
13942
14068
|
created: "2026-04-07"
|
|
13943
|
-
last_updated: "2026-
|
|
14069
|
+
last_updated: "2026-09-04"
|
|
13944
14070
|
description: "Reference for the language-specific rules system: 13 per-language rule sets shipped as knowledge skills, plus common rules installed as Claude Code path-scoped project rules."
|
|
13945
14071
|
---
|
|
13946
14072
|
|
|
@@ -13990,7 +14116,7 @@ app/rules/
|
|
|
13990
14116
|
└── medplum/
|
|
13991
14117
|
```
|
|
13992
14118
|
|
|
13993
|
-
**Total: 13 per-language directories × 5 files + 1 common directory ×
|
|
14119
|
+
**Total: 13 per-language directories × 5 files + 1 common directory × 6 files (5 in every profile + `git-team` in `strict`) + 3 standalone files** (see README.md for canonical count). Per-language directories ship as `<lang>-rules` knowledge skills; the common directory is installed as Claude Code `.claude/rules/ai-toolkit-*.md` files.
|
|
13994
14120
|
|
|
13995
14121
|
## Supported Languages
|
|
13996
14122
|
|
|
@@ -14080,13 +14206,16 @@ Common rules are installed as path-scoped Claude Code rule files:
|
|
|
14080
14206
|
```
|
|
14081
14207
|
.claude/rules/
|
|
14082
14208
|
├── ai-toolkit-coding-style.md
|
|
14209
|
+
├── ai-toolkit-git-team.md # --profile strict only
|
|
14083
14210
|
├── ai-toolkit-git-workflow.md
|
|
14084
14211
|
├── ai-toolkit-performance.md
|
|
14085
14212
|
├── ai-toolkit-security.md
|
|
14086
14213
|
└── ai-toolkit-testing.md
|
|
14087
14214
|
```
|
|
14088
14215
|
|
|
14089
|
-
|
|
14216
|
+
A source rule may also carry `profiles:` (same block-list form). `git-team` declares `profiles: ["strict"]`: branching, pull-request, and review conventions for teams, kept out of `standard` so a solo maintainer who releases straight to `main` is not told to open PRs against themselves. Rerunning `install --local` with a different profile adds or removes the managed file. Only the Claude Code local install honours `profiles`; the Claude app export, `compile-slm`, and editor `lang-common` bundles still receive every common rule (see `DECISIONS.md`, 2026-09-04).
|
|
14217
|
+
|
|
14218
|
+
Each file's `paths` frontmatter is copied from the source rule in `app/rules/common/<category>.md`. A source rule without a `paths` block is always-on:
|
|
14090
14219
|
|
|
14091
14220
|
```yaml
|
|
14092
14221
|
---
|
|
@@ -14095,6 +14224,18 @@ paths:
|
|
|
14095
14224
|
---
|
|
14096
14225
|
```
|
|
14097
14226
|
|
|
14227
|
+
A source rule with a `paths` block is path-scoped, so Claude Code loads it only when a matching file is touched. As of v4.32.0 `testing` (`**/*.test.*`, `**/*.spec.*`, `**/test_*`, `**/*_test.*`, `**/tests/**`) and `performance` (source-file extensions plus `**/*.sql`) are scoped; `coding-style`, `git-workflow`, and `security` stay always-on because they carry prohibitions that must hold in every session. To change a scope, edit the source frontmatter; `validate.py` rejects inline lists and unquoted globs because the installer reads only the block form:
|
|
14228
|
+
|
|
14229
|
+
```yaml
|
|
14230
|
+
---
|
|
14231
|
+
language: common
|
|
14232
|
+
category: testing
|
|
14233
|
+
version: "1.1.0"
|
|
14234
|
+
paths:
|
|
14235
|
+
- "**/tests/**"
|
|
14236
|
+
---
|
|
14237
|
+
```
|
|
14238
|
+
|
|
14098
14239
|
The project `CLAUDE.md` receives only a compact index between a single named marker (the per-language markers from v1.x are no longer used):
|
|
14099
14240
|
|
|
14100
14241
|
```
|
|
@@ -14102,10 +14243,13 @@ The project `CLAUDE.md` receives only a compact index between a single named mar
|
|
|
14102
14243
|
# Language Rules
|
|
14103
14244
|
|
|
14104
14245
|
Common ai-toolkit rules live in `.claude/rules/ai-toolkit-*.md`
|
|
14105
|
-
with Claude Code `paths` frontmatter
|
|
14106
|
-
|
|
14246
|
+
with Claude Code `paths` frontmatter instead of expanding this
|
|
14247
|
+
CLAUDE.md. Always-on rules load in every session; path-scoped rules
|
|
14248
|
+
load only when a matching file is touched.
|
|
14107
14249
|
|
|
14108
|
-
|
|
14250
|
+
Always-on: `.claude/rules/ai-toolkit-coding-style.md`, ...
|
|
14251
|
+
|
|
14252
|
+
Path-scoped: `.claude/rules/ai-toolkit-performance.md`, ...
|
|
14109
14253
|
|
|
14110
14254
|
Language-specific rules live in `<lang>-rules` knowledge skills (e.g.
|
|
14111
14255
|
`python-rules`, `typescript-rules`) and load automatically when their
|
|
@@ -14142,7 +14286,7 @@ Language rules are tracked as modules in `manifest.json`:
|
|
|
14142
14286
|
|
|
14143
14287
|
| Module | Description |
|
|
14144
14288
|
|--------|-------------|
|
|
14145
|
-
| `rules-common` | Common coding rules (5
|
|
14289
|
+
| `rules-common` | Common coding rules (6 files: 5 in every profile, `git-team` in `strict` only), included in `standard` profile |
|
|
14146
14290
|
| `rules-typescript` | TypeScript-specific rules |
|
|
14147
14291
|
| `rules-python` | Python-specific rules |
|
|
14148
14292
|
| `rules-golang` | Go-specific rules |
|
|
@@ -14163,9 +14307,9 @@ Language rules are tracked as modules in `manifest.json`:
|
|
|
14163
14307
|
|---|---|---|---|
|
|
14164
14308
|
| Source | `app/rules/common/` | `app/rules/<lang>/` | `app/skills/<name>/SKILL.md` |
|
|
14165
14309
|
| Delivery to Claude | Path-scoped `.claude/rules/ai-toolkit-*.md` files (`--local`) + compact `CLAUDE.md` index | Generated as `<lang>-rules` knowledge skills, loaded contextually | Loaded contextually by description match |
|
|
14166
|
-
| Visibility |
|
|
14310
|
+
| Visibility | Always-on (`coding-style`, `git-workflow`, `security`) or loaded when a matching file is touched (`testing`, `performance`) | Loaded when triggers match (file extensions, framework names) | Loaded when triggers match |
|
|
14167
14311
|
| Scope | Language-agnostic standards (security, git, testing, perf, style) | Per-language coding-style, frameworks, patterns, security, testing | Domain skills (testing, debugging, RAG, etc.) |
|
|
14168
|
-
| Install | `ai-toolkit install --local` | Global install (skills directory is symlinked) | Global install |
|
|
14312
|
+
| Install | `ai-toolkit install --local` | Global install (skills directory is symlinked); skills for languages no registered project uses are turned off via `skillOverrides` (`--language-skills detected`, the default) unless `--language-skills all` was chosen | Global install |
|
|
14169
14313
|
| Other editors | Inlined into editor-specific rule files | Inlined into editor-specific rule files (still full content, not skills) | N/A |
|
|
14170
14314
|
|
|
14171
14315
|
Per-language content delivered as a knowledge skill is the same Markdown that other editors receive inlined. The split exists only for Claude, where the Agent Skills progressive-disclosure mechanism keeps the system prompt small.
|
|
@@ -14313,7 +14457,7 @@ MIT remnant would need each contributor's agreement.
|
|
|
14313
14457
|
|
|
14314
14458
|
It is a test rather than a checklist line on purpose. This project has two
|
|
14315
14459
|
same-day postmortems about SOPs that existed and were skipped; CI does not skip.
|
|
14316
|
-
[Release Preparation](../procedures/release
|
|
14460
|
+
[Release Preparation](../procedures/sop-release.md) Phase 5c runs the
|
|
14317
14461
|
same gate before tagging so a failure surfaces before the tag, not after.
|
|
14318
14462
|
|
|
14319
14463
|
## If the licence ever changes again
|
|
@@ -14326,7 +14470,7 @@ source, cross-verify against a second independent copy, and only then write
|
|
|
14326
14470
|
## Related
|
|
14327
14471
|
|
|
14328
14472
|
- [`LICENSE`](../../LICENSE), [`NOTICE`](../../NOTICE)
|
|
14329
|
-
- [Release Preparation SOP](../procedures/release
|
|
14473
|
+
- [Release Preparation SOP](../procedures/sop-release.md) — Phase 5c
|
|
14330
14474
|
- [Distribution Model](distribution-model.md) — what ships and where
|
|
14331
14475
|
|
|
14332
14476
|
---
|
|
@@ -16006,7 +16150,7 @@ Updated baseline docs:
|
|
|
16006
16150
|
- `kb/reference/skills-catalog.md`
|
|
16007
16151
|
- `kb/reference/plugin-pack-conventions.md`
|
|
16008
16152
|
- `kb/reference/claude-ecosystem-benchmark-snapshot.md`
|
|
16009
|
-
- `kb/procedures/maintenance
|
|
16153
|
+
- `kb/procedures/sop-maintenance.md`
|
|
16010
16154
|
|
|
16011
16155
|
## Validation Evidence
|
|
16012
16156
|
|
|
@@ -16860,13 +17004,13 @@ See [DSH Compatibility](./dsh-compatibility.md) for commands, topology, subscrip
|
|
|
16860
17004
|
2. Delete its section from this document.
|
|
16861
17005
|
3. Delete its snapshot entry from `benchmarks/ecosystem-doctor-snapshot.json` (or let the next `--update` prune it — currently not pruned automatically).
|
|
16862
17006
|
4. Decide whether to keep the generator (`scripts/generate_<tool>_*.py`) for backwards compatibility or delete it.
|
|
16863
|
-
5. Remove references from `README.md`, `manifest.json` `description` field, and `kb/procedures/maintenance
|
|
17007
|
+
5. Remove references from `README.md`, `manifest.json` `description` field, and `kb/procedures/sop-maintenance.md` `Supported editors` line.
|
|
16864
17008
|
|
|
16865
17009
|
---
|
|
16866
17010
|
|
|
16867
17011
|
## Related
|
|
16868
17012
|
|
|
16869
|
-
- [Ecosystem Sync SOP](../procedures/ecosystem-sync
|
|
17013
|
+
- [Ecosystem Sync SOP](../procedures/sop-ecosystem-sync.md) — how to use the doctor
|
|
16870
17014
|
- [MCP Editor Compatibility](./mcp-editor-compatibility.md) — MCP-specific subset
|
|
16871
17015
|
- `scripts/ecosystem_tools.json` — source of truth
|
|
16872
17016
|
- `scripts/ecosystem_doctor.py` — drift detector
|
|
@@ -16963,7 +17107,7 @@ category: reference
|
|
|
16963
17107
|
service: ai-toolkit
|
|
16964
17108
|
tags: [features, differentiators, constitution, hooks, security, tdd, memory]
|
|
16965
17109
|
created: "2026-04-13"
|
|
16966
|
-
last_updated: "2026-
|
|
17110
|
+
last_updated: "2026-09-04"
|
|
16967
17111
|
description: "Detailed description of ai-toolkit's unique features: constitution enforcement, hooks system, security scanning, effort budgeting, quality gates, and more."
|
|
16968
17112
|
---
|
|
16969
17113
|
|
|
@@ -17078,6 +17222,7 @@ added after the thing it checks had already shipped broken.
|
|
|
17078
17222
|
| **Public surface** (`surface_manifest.py` + `app/surface.json`) | a skill, agent, CLI command, frontmatter field, hook, KB category or pack disappearing | `BACKWARD_COMPATIBILITY.md` listed the surfaces; renaming one still left every check green |
|
|
17079
17223
|
| **Script invocation** (`validate.py`) | a skill running its own script by any path other than `${CLAUDE_SKILL_DIR}`, or through the wrong interpreter | nine skills shipped invocations that resolved only in the repo, never on an installed machine |
|
|
17080
17224
|
| **Skill body budget** (`validate.py`) | a `SKILL.md` body over 20,000 bytes; warns over 18,000 | the body loads on every trigger match, including accidental ones; three skills sat at 24–35 KB |
|
|
17225
|
+
| **Skill description budget** (`validate.py`) | a description over 1,024 characters (the Agent Skills cap; the runtime truncates past it), or an unquoted description containing `: ` or ` #` (strict YAML drops every field, `allowed-tools` included); warns over 400; applied to plugin-pack skills too | two installed skills sat at 791 and 883 characters and made up 8% of the model's skill listing on their own |
|
|
17081
17226
|
| **Split integrity** (`check_split.py`) | a body → `reference/` refactor that lost a fenced code line, a required section, the description, or a working link | run by hand during a split; caught a heading corrupted inside a fenced example on first use |
|
|
17082
17227
|
|
|
17083
17228
|
The surface check is deliberately one-directional: **removals fail, additions pass.**
|
|
@@ -17399,7 +17544,7 @@ Expect `Total plugin skills loaded: 0`, `Total plugin agents loaded: 0`, and the
|
|
|
17399
17544
|
## Related
|
|
17400
17545
|
|
|
17401
17546
|
- `kb/reference/global-install-model.md`
|
|
17402
|
-
- `kb/procedures/maintenance
|
|
17547
|
+
- `kb/procedures/sop-maintenance.md`
|
|
17403
17548
|
|
|
17404
17549
|
---
|
|
17405
17550
|
|