@softspark/ai-toolkit 4.30.3 → 4.31.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +34 -0
- package/README.md +10 -10
- package/app/.claude-plugin/plugin.json +1 -1
- package/app/agents/code-reviewer.md +14 -0
- package/app/agents/frontend-specialist.md +27 -24
- package/app/agents/infrastructure-validator.md +2 -2
- package/app/claude-app/skills/ai-toolkit-rules/SKILL.md +44 -1
- package/app/personas/frontend-lead.md +20 -12
- package/app/rules/claude-toolkit-rules.md +4 -1
- package/app/rules/edit-discipline.md +37 -0
- package/app/skills/app-builder/SKILL.md +37 -6
- package/app/skills/claude-toolkit-rules/SKILL.md +37 -0
- package/app/skills/design-engineering/SKILL.md +87 -102
- package/app/skills/design-engineering/reference/animation-recipes.md +67 -0
- package/app/skills/design-engineering/reference/component-craft.md +28 -0
- package/app/skills/documentation-standards/SKILL.md +1 -1
- package/app/skills/edit-discipline/SKILL.md +50 -0
- package/app/skills/git-conventions/SKILL.md +18 -0
- package/app/skills/output-mode/SKILL.md +41 -0
- package/app/skills/quality-gates/SKILL.md +23 -0
- package/app/skills/review/SKILL.md +10 -0
- package/kb/procedures/{ecosystem-sync-sop.md → sop-ecosystem-sync.md} +2 -2
- package/kb/procedures/{post-release-testing-sop.md → sop-post-release-testing.md} +3 -3
- package/kb/procedures/sop-pre-commit.md +96 -0
- package/kb/procedures/{release-preparation-sop.md → sop-release.md} +6 -6
- package/kb/reference/architecture-overview.md +1 -1
- package/kb/reference/copilot-compatibility.md +1 -1
- package/kb/reference/licensing.md +2 -2
- package/kb/reference/quick-wins-implementation-summary.md +1 -1
- package/kb/reference/supported-tools-registry.md +2 -2
- package/kb/troubleshooting/plugin-double-load.md +1 -1
- package/llms-full.txt +956 -849
- package/llms.txt +6 -5
- package/manifest.json +1 -1
- package/package.json +2 -1
- package/scripts/generate_toolkit_rules_skills.py +189 -0
- package/scripts/validate.py +1 -1
- /package/kb/procedures/{maintenance-sop.md → sop-maintenance.md} +0 -0
- /package/kb/procedures/{release-verification-sop.md → sop-release-verification.md} +0 -0
package/llms-full.txt
CHANGED
|
@@ -32,11 +32,12 @@
|
|
|
32
32
|
- [Closed: Tool-Output Token Reduction](kb/history/completed/tool-output-token-reduction-closed-20260727.md)
|
|
33
33
|
- [How-To Guides](kb/howto/README.md)
|
|
34
34
|
- [Plan: Cloud Security Pack — Multi-Cloud Audit](kb/planning/cloud-security-pack-plan.md)
|
|
35
|
-
- [SOP: Ecosystem Sync](kb/procedures/ecosystem-sync
|
|
36
|
-
- [SOP: AI Toolkit Maintenance](kb/procedures/maintenance
|
|
37
|
-
- [SOP: Post-Release Testing](kb/procedures/post-release-testing
|
|
38
|
-
- [SOP:
|
|
39
|
-
- [SOP: Release Verification](kb/procedures/release-verification
|
|
35
|
+
- [SOP: Ecosystem Sync](kb/procedures/sop-ecosystem-sync.md)
|
|
36
|
+
- [SOP: AI Toolkit Maintenance](kb/procedures/sop-maintenance.md)
|
|
37
|
+
- [SOP: Post-Release Testing](kb/procedures/sop-post-release-testing.md)
|
|
38
|
+
- [SOP: Pre-Commit Gate](kb/procedures/sop-pre-commit.md)
|
|
39
|
+
- [SOP: Release Verification](kb/procedures/sop-release-verification.md)
|
|
40
|
+
- [SOP: Release Preparation](kb/procedures/sop-release.md)
|
|
40
41
|
- [Agents Catalog](kb/reference/agents-catalog.md)
|
|
41
42
|
- [Anti-Pattern Registry Format](kb/reference/anti-pattern-registry-format.md)
|
|
42
43
|
- [AI Toolkit Architecture](kb/reference/architecture-overview.md)
|
|
@@ -95,6 +96,7 @@
|
|
|
95
96
|
- **chaos**: Injects controlled faults for resilience testing on non-prod. Triggers: chaos, fault injection, latency injection, dependency kill, resilience test.
|
|
96
97
|
- **ci**: Detect/generate/debug CI pipeline config (GitHub Actions, GitLab CI). Triggers: CI setup, build pipeline, GitHub Actions config, debug CI, GitLab CI.
|
|
97
98
|
- **ci-cd-patterns**: CI/CD: GitHub Actions, GitLab CI, Jenkins, caching, blue-green, canary. Triggers: CI, CD, pipeline, GitHub Actions, workflow YAML, release, canary, rollout.
|
|
99
|
+
- **claude-toolkit-rules**: SoftSpark working agreement: never guess a home directory path, give at least three alternatives, and apply a devil's advocate critique to decisions. Triggers: toolkit, conventions, workflow, alternatives, review.
|
|
98
100
|
- **clean-code**: Code quality: meaningful names, SRP, DRY, small functions, guard clauses, refactoring. Triggers: clean code, naming, code smell, SRP, DRY, long function, god class, dead code.
|
|
99
101
|
- **command-creator**: Creates new Claude Code slash commands with frontmatter and validation. Triggers: new slash command, create command, command scaffold.
|
|
100
102
|
- **commit**: Creates Conventional Commits with pre-commit validation. Triggers: commit, conventional commit, git commit, message.
|
|
@@ -115,12 +117,14 @@
|
|
|
115
117
|
- **docs**: Generates/updates README, API docs, architecture notes. Triggers: docs, README, API docs, architecture note, documentation.
|
|
116
118
|
- **documentation-standards**: KB conventions: YAML frontmatter, 10-category taxonomy (reference/howto/procedures/troubleshooting/best-practices/decisions/runbooks/planning/business/templates). Triggers: kb/, SOP, runbook, howto, frontmatter, knowledge base.
|
|
117
119
|
- **ecommerce-patterns**: E-commerce: cart, checkout, payments (Stripe/Adyen), order state, inventory, promos, tax. Triggers: cart, checkout, SKU, payment, Stripe, Shopify, Medusa, Magento, coupon, refund.
|
|
120
|
+
- **edit-discipline**: Change files with the edit and write tools, never by rewriting them through bash (sed, awk, tee, heredoc, redirection), and show `git diff` before reporting a file-changing task as done. Triggers: edit, write, modify, refactor, patch, fix, diff, review changes.
|
|
118
121
|
- **evaluate**: Evaluates RAG retrieval and LLM-as-judge metrics (faithfulness, relevancy, context precision). Triggers: measure RAG quality, knowledge gap, RAG eval, golden dataset.
|
|
119
122
|
- **evolve**: Analyzes agent/skill failures, drafts prompt/permission fixes. Triggers: improve agent, refine skill, system prompt, optimize agent.
|
|
120
123
|
- **explain**: Explains code/architecture with Mermaid diagrams and sequence flows. Triggers: what does X do, how does Y work, explain code, sequence diagram.
|
|
121
124
|
- **explore**: Explores codebase structure, stack, and architecture. Triggers: explore codebase, project structure, stack overview, architecture map.
|
|
122
125
|
- **fix**: Applies targeted fix to known bug/lint error, verifies with same command that surfaced it. Triggers: fix, apply fix, fix bug, fix lint, targeted fix.
|
|
123
126
|
- **flutter-patterns**: Flutter/Dart: widgets, state mgmt (Riverpod/Bloc), navigation, platform channels. Triggers: Flutter, Dart, widget, Riverpod, Bloc, pubspec, hot reload.
|
|
127
|
+
- **git-conventions**: Conventional Commits only (feat, fix, docs, refactor, test, chore); no AI co-authorship trailer and no AI signature in a commit message. Triggers: commit, git, message, changelog, release, pull request.
|
|
124
128
|
- **git-mastery**: Advanced Git: rebase, bisect, reflog, cherry-pick, worktrees, LFS. Triggers: rebase, bisect, cherry-pick, reflog, force push, merge conflict, worktree.
|
|
125
129
|
- **golang-rules**: Go coding rules: style, patterns, security, testing. Triggers: .go, go.mod, go.sum, Gin, Echo, Gorilla, testing, gofmt.
|
|
126
130
|
- **grill-me**: Stress-tests a plan via Socratic questioning down each decision branch. Triggers: stress-test, grill me, validate assumptions, challenge plan, socratic review.
|
|
@@ -147,6 +151,7 @@
|
|
|
147
151
|
- **observability-patterns**: Observability: structured logs, metrics (RED/USE), tracing, SLO/SLI. Triggers: logging, metrics, Prometheus, Grafana, OpenTelemetry, trace, monitoring.
|
|
148
152
|
- **onboard**: Sets up ai-toolkit in a project: symlinks, CLAUDE.md, intent interview. Triggers: onboard, setup project, install ai-toolkit, migrate project.
|
|
149
153
|
- **orchestrate**: Coordinates multiple specialized agents in parallel. Triggers: orchestrate, multi-agent, parallel agents, coordinate agents.
|
|
154
|
+
- **output-mode**: Answer concisely: lead with the result, no preamble, tables over prose, no trailing restatement of a diff the reader can already see. Triggers: response style, verbosity, summary, explanation, report.
|
|
150
155
|
- **panic**: Emergency kill switch — halts all agents via lockfile gate. Triggers: panic, stop everything, kill switch, halt agents, agents looping.
|
|
151
156
|
- **performance-profiling**: Performance: golden signals, p50/p95/p99, flame graphs, load testing. Triggers: performance, slow, latency, p99, flame graph, bottleneck, memory leak.
|
|
152
157
|
- **persona**: Switches engineering persona at runtime. Triggers: persona, switch role, backend-lead, frontend-lead, devops-eng, junior-dev.
|
|
@@ -160,6 +165,7 @@
|
|
|
160
165
|
- **prompt-caching-patterns**: Anthropic API prompt caching: TTL, breakpoints, stacking, invalidation, hit rate. Triggers: prompt caching, cache_control, cache breakpoint, cache TTL, hit rate.
|
|
161
166
|
- **python-rules**: Python coding rules: style, patterns, security, testing. Triggers: .py, .pyi, pyproject.toml, requirements.txt, Pipfile, FastAPI, Django, Flask, pytest, SQLAlchemy, ruff, mypy.
|
|
162
167
|
- **qa-session**: Interactive QA: user reports bugs conversationally, agent files GitHub issues. Triggers: QA session, report bug, file issue, conversational QA, bug intake.
|
|
168
|
+
- **quality-gates**: Plan before work over an hour, and hold the gates: ruff clean, mypy --strict clean, pytest coverage above 70 percent, no secrets in code. Triggers: quality, lint, mypy, pytest, coverage, gate, definition of done.
|
|
163
169
|
- **rag-patterns**: RAG: embeddings, chunking, hybrid search (BM25+vector), reranking, CRAG, multi-hop. Triggers: RAG, embedding, pgvector, Qdrant, Pinecone, Weaviate, reranker, semantic search.
|
|
164
170
|
- **refactor**: Refactors code for quality and maintainability. Triggers: refactor, clean up, restructure, improve code, modernize.
|
|
165
171
|
- **refactor-plan**: Creates detailed refactor plan with tiny commits via interview, files as GitHub RFC. Triggers: refactor plan, refactoring RFC, incremental refactor, safe steps.
|
|
@@ -6929,7 +6935,7 @@ Examples:
|
|
|
6929
6935
|
|
|
6930
6936
|
---
|
|
6931
6937
|
|
|
6932
|
-
## kb/procedures/ecosystem-sync
|
|
6938
|
+
## kb/procedures/sop-ecosystem-sync.md
|
|
6933
6939
|
|
|
6934
6940
|
---
|
|
6935
6941
|
title: "SOP: Ecosystem Sync"
|
|
@@ -7186,12 +7192,12 @@ jobs:
|
|
|
7186
7192
|
|
|
7187
7193
|
- [Supported Tools Registry](../reference/supported-tools-registry.md) — human-readable per-tool breakdown
|
|
7188
7194
|
- [MCP Editor Compatibility](../reference/mcp-editor-compatibility.md) — MCP-specific adapter table
|
|
7189
|
-
- [Maintenance SOP](maintenance
|
|
7190
|
-
- [Release Preparation SOP](release
|
|
7195
|
+
- [Maintenance SOP](sop-maintenance.md) — general toolkit upkeep
|
|
7196
|
+
- [Release Preparation SOP](sop-release.md) — run the doctor before tagging
|
|
7191
7197
|
|
|
7192
7198
|
---
|
|
7193
7199
|
|
|
7194
|
-
## kb/procedures/maintenance
|
|
7200
|
+
## kb/procedures/sop-maintenance.md
|
|
7195
7201
|
|
|
7196
7202
|
---
|
|
7197
7203
|
title: "SOP: AI Toolkit Maintenance"
|
|
@@ -7705,7 +7711,7 @@ What `uninstall` does:
|
|
|
7705
7711
|
|
|
7706
7712
|
---
|
|
7707
7713
|
|
|
7708
|
-
## kb/procedures/post-release-testing
|
|
7714
|
+
## kb/procedures/sop-post-release-testing.md
|
|
7709
7715
|
|
|
7710
7716
|
---
|
|
7711
7717
|
title: "SOP: Post-Release Testing"
|
|
@@ -7725,7 +7731,7 @@ actually install, from npm, rather than the working tree.
|
|
|
7725
7731
|
|
|
7726
7732
|
Sibling procedures exist for `jira-mcp` and `legal-pl-pack`; this is the
|
|
7727
7733
|
ai-toolkit equivalent. It complements
|
|
7728
|
-
[Release Verification](release-verification
|
|
7734
|
+
[Release Verification](sop-release-verification.md), which checks the toolkit
|
|
7729
7735
|
from the maintainer's own installed copy. The difference that matters: this one
|
|
7730
7736
|
never writes to the maintainer's `~/.claude` or `~/.softspark`.
|
|
7731
7737
|
|
|
@@ -8019,1201 +8025,1302 @@ print(f'removed {sb} ({n} files)')
|
|
|
8019
8025
|
|
|
8020
8026
|
## Related
|
|
8021
8027
|
|
|
8022
|
-
- [Release Preparation](release
|
|
8023
|
-
- [Release Verification](release-verification
|
|
8028
|
+
- [Release Preparation](sop-release.md) — run before tagging
|
|
8029
|
+
- [Release Verification](sop-release-verification.md) — the maintainer-install checks
|
|
8024
8030
|
- [rtk-pack Retirement](../history/completed/rtk-pack-retirement-20260727.md) — what happened the one time this SOP was written and not run
|
|
8025
8031
|
|
|
8026
8032
|
---
|
|
8027
8033
|
|
|
8028
|
-
## kb/procedures/
|
|
8034
|
+
## kb/procedures/sop-pre-commit.md
|
|
8029
8035
|
|
|
8030
8036
|
---
|
|
8031
|
-
title: "SOP:
|
|
8037
|
+
title: "SOP: Pre-Commit Gate"
|
|
8032
8038
|
category: procedures
|
|
8033
8039
|
service: ai-toolkit
|
|
8034
|
-
tags: [
|
|
8035
|
-
version: "1.
|
|
8036
|
-
created: "2026-
|
|
8040
|
+
tags: [pre-commit, quality-gate, shellcheck, bats, validation, secrets]
|
|
8041
|
+
version: "1.0.0"
|
|
8042
|
+
created: "2026-09-02"
|
|
8037
8043
|
last_updated: "2026-09-02"
|
|
8038
|
-
description: "
|
|
8044
|
+
description: "The local gate to run before every commit to ai-toolkit: staged-file and secret scan, ShellCheck, Python syntax, toolkit validation, and the Bats suite. Runs the same checks CI runs, so main does not go red after a push."
|
|
8039
8045
|
---
|
|
8040
8046
|
|
|
8041
|
-
# SOP:
|
|
8047
|
+
# SOP: Pre-Commit Gate
|
|
8042
8048
|
|
|
8043
|
-
|
|
8044
|
-
Run this **before** tagging. After tagging and publishing, run the
|
|
8045
|
-
[Release Verification SOP](release-verification-sop.md) to smoke-test.
|
|
8049
|
+
## Why this exists
|
|
8046
8050
|
|
|
8047
|
-
|
|
8048
|
-
|
|
8049
|
-
|
|
8050
|
-
|
|
8051
|
-
|
|
8052
|
-
↓
|
|
8053
|
-
git tag → CI publish → Release Verification SOP
|
|
8054
|
-
```
|
|
8051
|
+
CI runs on `main` and on pull requests, but it runs *after* the push. Every
|
|
8052
|
+
check below already exists in `.github/workflows/ci.yml`; running them locally
|
|
8053
|
+
first is what keeps `main` green and keeps a release from becoming a debug
|
|
8054
|
+
session. The v4.5.1 postmortem in `sop-release.md` is the case
|
|
8055
|
+
where a hook lint failure published while reddening `main` CI.
|
|
8055
8056
|
|
|
8056
|
-
|
|
8057
|
+
This gate is not a substitute for `sop-release.md`. That SOP gates
|
|
8058
|
+
a *tag*; this one gates a *commit*.
|
|
8057
8059
|
|
|
8058
|
-
|
|
8060
|
+
## Prerequisites
|
|
8059
8061
|
|
|
8060
|
-
|
|
8062
|
+
`shellcheck`, `python3` and `bats` must be on `PATH`. The suite is the one
|
|
8063
|
+
check that silently does nothing when its runner is absent, so install it
|
|
8064
|
+
before relying on this gate:
|
|
8061
8065
|
|
|
8062
8066
|
```bash
|
|
8063
|
-
|
|
8064
|
-
#
|
|
8065
|
-
python3 scripts/ecosystem_doctor.py --format text > /tmp/eco-report.txt
|
|
8066
|
-
cat /tmp/eco-report.txt
|
|
8067
|
-
# If drift detected: stop here, follow ecosystem-sync-sop.md Phase 2-4 to
|
|
8068
|
-
# classify each drift (A-F), update generators as needed, refresh snapshot,
|
|
8069
|
-
# THEN resume this SOP.
|
|
8070
|
-
python3 scripts/ecosystem_doctor.py --update # after all drift resolved
|
|
8071
|
-
|
|
8072
|
-
# 1. Decide version bump
|
|
8073
|
-
# patch (1.4.2 → 1.4.3): bugfix, typo, doc fix
|
|
8074
|
-
# minor (1.4.2 → 1.5.0): new feature, new skill, new flag, any ecosystem-class-B/F change
|
|
8075
|
-
# major (1.4.2 → 2.0.0): breaking change, any ecosystem-class-D removed path
|
|
8076
|
-
|
|
8077
|
-
# 2. Sync version across all files
|
|
8078
|
-
python3 scripts/sync_version.py X.Y.Z # if script exists, else manual
|
|
8079
|
-
|
|
8080
|
-
# 3. Write CHANGELOG.md entry
|
|
8081
|
-
# 4. Regenerate artifacts
|
|
8082
|
-
python3 scripts/generate_agents_md.py > AGENTS.md
|
|
8083
|
-
python3 scripts/generate_llms_txt.py > llms.txt
|
|
8084
|
-
python3 scripts/generate_llms_txt.py --full > llms-full.txt
|
|
8085
|
-
|
|
8086
|
-
# 5. Validate + audit + SARIF + shellcheck + test + ecosystem check
|
|
8087
|
-
python3 scripts/validate.py --strict && python3 scripts/audit_skills.py --ci && python3 scripts/audit_skills.py --sarif > /tmp/audit.sarif && shellcheck --severity=warning app/hooks/*.sh && npm test
|
|
8088
|
-
|
|
8089
|
-
# 5a. Supply-chain standard (v2.8.0+) — non-negotiable
|
|
8090
|
-
grep -q -- '--provenance' .github/workflows/publish.yml || { echo "MISSING --provenance"; exit 1; }
|
|
8091
|
-
grep -q 'id-token: write' .github/workflows/publish.yml || { echo "MISSING id-token: write"; exit 1; }
|
|
8092
|
-
python3 scripts/audit_skills.py --permissions # review Bash/Write/Edit footprint
|
|
8093
|
-
|
|
8094
|
-
# 5b. Ecosystem gate — snapshot must be current before tag
|
|
8095
|
-
python3 scripts/ecosystem_doctor.py --offline --check || { echo "STALE ecosystem snapshot — re-run doctor"; exit 1; }
|
|
8096
|
-
|
|
8097
|
-
# 5c. Licensing gate — SPDX headers, LICENSE, NOTICE, manifest consistency
|
|
8098
|
-
npx bats tests/test_licensing.bats || { echo "LICENSING GATE FAILED"; exit 1; }
|
|
8099
|
-
|
|
8100
|
-
# 6. Commit, push branch, and wait for the exact commit's full CI
|
|
8101
|
-
git add -A && git commit -m "chore: release vX.Y.Z"
|
|
8102
|
-
RELEASE_SHA=$(git rev-parse HEAD)
|
|
8103
|
-
git push origin main || { echo "FAIL: release commit push failed"; exit 1; }
|
|
8104
|
-
RUN_ID=""
|
|
8105
|
-
for ATTEMPT in 1 2 3 4 5; do
|
|
8106
|
-
RUN_ID=$(gh run list --workflow ci.yml --event push --commit "$RELEASE_SHA" \
|
|
8107
|
-
--limit 1 --json databaseId --jq '.[0].databaseId')
|
|
8108
|
-
[ -n "$RUN_ID" ] && break
|
|
8109
|
-
echo "Waiting for CI run registration ($ATTEMPT/5)" >&2
|
|
8110
|
-
[ "$ATTEMPT" -eq 5 ] || sleep 60
|
|
8111
|
-
done
|
|
8112
|
-
test -n "$RUN_ID" || { echo "FAIL: CI run not found for release commit"; exit 1; }
|
|
8113
|
-
gh run watch "$RUN_ID" --exit-status \
|
|
8114
|
-
|| { echo "FAIL: release commit CI failed"; exit 1; }
|
|
8115
|
-
for OS in ubuntu-latest macos-latest; do
|
|
8116
|
-
gh run view "$RUN_ID" --json jobs \
|
|
8117
|
-
--jq ".jobs[] | select(.name == \"Bats test suite ($OS)\") | .conclusion" \
|
|
8118
|
-
| grep -qx success || { echo "FAIL: $OS test job not green"; exit 1; }
|
|
8119
|
-
done
|
|
8120
|
-
test "$(git rev-parse HEAD)" = "$RELEASE_SHA" || { echo "FAIL: local HEAD changed during CI"; exit 1; }
|
|
8121
|
-
git fetch origin main || { echo "FAIL: origin/main refresh failed"; exit 1; }
|
|
8122
|
-
test "$(git rev-parse origin/main)" = "$RELEASE_SHA" || { echo "FAIL: origin/main changed during CI"; exit 1; }
|
|
8123
|
-
|
|
8124
|
-
# 6a. Only green branch CI authorizes tag creation
|
|
8125
|
-
git tag vX.Y.Z "$RELEASE_SHA"
|
|
8126
|
-
|
|
8127
|
-
# 6b. Assert the tag before pushing it (v4.19.0 postmortem, Phase 7)
|
|
8128
|
-
test "$(git rev-parse vX.Y.Z)" = "$RELEASE_SHA" || { echo "FAIL: tag not on tested release commit"; exit 1; }
|
|
8129
|
-
git show --no-patch --format=%s vX.Y.Z | grep -qx "chore: release vX.Y.Z" || { echo "FAIL: tag not on release commit"; exit 1; }
|
|
8130
|
-
|
|
8131
|
-
# 6c. Push the single tag by full ref. Never --tags.
|
|
8132
|
-
git push origin refs/tags/vX.Y.Z
|
|
8067
|
+
brew install bats-core shellcheck # macOS
|
|
8068
|
+
sudo apt-get install -y bats shellcheck # Debian / Ubuntu
|
|
8133
8069
|
```
|
|
8134
8070
|
|
|
8135
|
-
|
|
8071
|
+
## The gate
|
|
8136
8072
|
|
|
8137
|
-
|
|
8073
|
+
Run all five from the repository root. Each mirrors a CI job.
|
|
8138
8074
|
|
|
8139
|
-
|
|
8075
|
+
```bash
|
|
8076
|
+
# 1. Staged files: secrets, large blobs, commit-type suggestion
|
|
8077
|
+
python3 app/skills/commit/scripts/pre-commit-check.py
|
|
8140
8078
|
|
|
8141
|
-
|
|
8142
|
-
|
|
8143
|
-
- Major release (X.0.0) — always
|
|
8144
|
-
- Patch release (X.Y.Z) — only if the patch touches a generator or install flow
|
|
8079
|
+
# 2. ShellCheck hooks -- the CI job that has published while red before
|
|
8080
|
+
shellcheck --severity=warning app/hooks/*.sh app/plugins/*/hooks/*.sh
|
|
8145
8081
|
|
|
8146
|
-
|
|
8082
|
+
# 3. Python syntax across every script the toolkit ships
|
|
8083
|
+
python3 -m py_compile scripts/*.py app/skills/*/scripts/*.py
|
|
8147
8084
|
|
|
8148
|
-
|
|
8085
|
+
# 4. Toolkit integrity: agents, skills, registry drift, content quality
|
|
8086
|
+
npm run validate
|
|
8149
8087
|
|
|
8150
|
-
|
|
8151
|
-
|
|
8088
|
+
# 5. The suite. Run it ONCE -- see the single-run discipline below
|
|
8089
|
+
npm test
|
|
8152
8090
|
```
|
|
8153
8091
|
|
|
8154
|
-
|
|
8092
|
+
A non-zero exit from any of them is a stop, not a warning to note and push past.
|
|
8155
8093
|
|
|
8156
|
-
|
|
8094
|
+
## Single-run discipline
|
|
8157
8095
|
|
|
8158
|
-
|
|
8096
|
+
`npm test` runs the Bats suite with `--jobs 4`. Running it repeatedly to see
|
|
8097
|
+
whether a failure is "flaky" hides real ordering bugs and wastes minutes. Run
|
|
8098
|
+
it once. If it fails, read the failure and fix the cause; if the same test
|
|
8099
|
+
passes on a re-run without a code change, that instability is itself the bug
|
|
8100
|
+
and belongs in an issue, not in a retry.
|
|
8159
8101
|
|
|
8160
|
-
|
|
8161
|
-
|-------------|----------------|
|
|
8162
|
-
| A (cosmetic reword) | No version impact — refresh snapshot, continue |
|
|
8163
|
-
| B (new feature — integrate) | **Minor** version bump at minimum; new generator or extended generator |
|
|
8164
|
-
| C (new feature — not adopted) | No impact — note in registry |
|
|
8165
|
-
| D (deprecation) | **Minor** or **major** depending on user impact; add migration warning |
|
|
8166
|
-
| E (feature promoted to default) | **Minor**; simplify generator, keep fallback comment |
|
|
8167
|
-
| F (feature newly globally available) | **Minor**; may require new generator or new config path |
|
|
8102
|
+
## What this gate does not cover
|
|
8168
8103
|
|
|
8169
|
-
|
|
8104
|
+
- **Cross-platform.** The suite here runs on your machine only. Both
|
|
8105
|
+
`ubuntu-latest` and `macos-latest` must be green before a tag; that gate
|
|
8106
|
+
lives in `sop-release.md`, Phase 7, and is bound to the exact
|
|
8107
|
+
release commit.
|
|
8108
|
+
- **Required files.** The `required-files` CI job checks that every file the
|
|
8109
|
+
module template mandates is present and that the version is in sync across
|
|
8110
|
+
all four manifests. It is cheap and runs on every push.
|
|
8111
|
+
- **Provenance and SARIF.** Release-time concerns; see
|
|
8112
|
+
`sop-release.md`.
|
|
8170
8113
|
|
|
8171
|
-
|
|
8114
|
+
## Commit message
|
|
8172
8115
|
|
|
8173
|
-
|
|
8116
|
+
Conventional Commits, and no AI co-authorship trailer:
|
|
8174
8117
|
|
|
8175
|
-
```
|
|
8176
|
-
|
|
8118
|
+
```
|
|
8119
|
+
feat(scope): summary in the imperative
|
|
8120
|
+
fix(scope): summary in the imperative
|
|
8121
|
+
docs|refactor|test|chore(scope): ...
|
|
8177
8122
|
```
|
|
8178
8123
|
|
|
8179
|
-
|
|
8124
|
+
`pre-commit-check.py` suggests a type from the staged paths. It is a
|
|
8125
|
+
suggestion; the scope and the summary are yours.
|
|
8180
8126
|
|
|
8181
|
-
|
|
8127
|
+
## Related
|
|
8182
8128
|
|
|
8183
|
-
|
|
8184
|
-
|
|
8185
|
-
|
|
8129
|
+
- `sop-release.md` -- the pre-tag gate, including cross-platform CI
|
|
8130
|
+
- `sop-post-release-testing.md` -- what to verify after a release ships
|
|
8131
|
+
- `sop-release-verification.md` -- end-to-end smoke test of an installed build
|
|
8186
8132
|
|
|
8187
|
-
|
|
8133
|
+
---
|
|
8188
8134
|
|
|
8135
|
+
## kb/procedures/sop-release-verification.md
|
|
8136
|
+
|
|
8137
|
+
---
|
|
8138
|
+
title: "SOP: Release Verification"
|
|
8139
|
+
category: procedures
|
|
8140
|
+
service: ai-toolkit
|
|
8141
|
+
tags: [sop, verification, release, smoke-test, install, update, qa, provenance, sarif, dsh]
|
|
8142
|
+
version: "1.8.0"
|
|
8143
|
+
created: "2026-04-08"
|
|
8144
|
+
last_updated: "2026-09-01"
|
|
8145
|
+
description: "End-to-end smoke test after installing or updating @softspark/ai-toolkit. Verifies CLI, native Codex and GitHub Copilot surfaces, explicit DSH lifecycle, Claude app export, doctor, validation, tests, eject, provenance, SARIF, and per-skill permissions."
|
|
8189
8146
|
---
|
|
8190
8147
|
|
|
8191
|
-
|
|
8148
|
+
# SOP: Release Verification
|
|
8192
8149
|
|
|
8193
|
-
|
|
8150
|
+
End-to-end smoke test after installing or updating `@softspark/ai-toolkit`.
|
|
8151
|
+
Verifies all critical paths from the user's perspective.
|
|
8194
8152
|
|
|
8195
|
-
|
|
8196
|
-
|
|
8197
|
-
|
|
8198
|
-
|
|
8199
|
-
|
|
8153
|
+
**Use this SOP when:**
|
|
8154
|
+
- After `npm install -g @softspark/ai-toolkit@latest`
|
|
8155
|
+
- After `ai-toolkit update`
|
|
8156
|
+
- Before tagging a new version (`git tag`)
|
|
8157
|
+
- Before publishing to npm (`npm publish`)
|
|
8158
|
+
- As a smoke test in CI/CD
|
|
8200
8159
|
|
|
8201
|
-
**
|
|
8160
|
+
**Prerequisites:**
|
|
8161
|
+
- Node.js >= 18, Python 3, `bats`, git
|
|
8162
|
+
- `@softspark/ai-toolkit` installed globally
|
|
8163
|
+
|
|
8164
|
+
**Time:** 10-15 minutes (full), 2 minutes (quick checklist)
|
|
8202
8165
|
|
|
8203
8166
|
---
|
|
8204
8167
|
|
|
8205
|
-
##
|
|
8168
|
+
## Quick Checklist (TL;DR)
|
|
8206
8169
|
|
|
8207
|
-
The
|
|
8170
|
+
The 14 core commands below must pass. Releases that change DSH must also complete Phase 10.
|
|
8208
8171
|
|
|
8209
|
-
|
|
8172
|
+
```bash
|
|
8173
|
+
# Pre-commit (Phase 0)
|
|
8174
|
+
python3 scripts/generate_agents_md.py > AGENTS.md # 1. Regenerate AGENTS.md
|
|
8175
|
+
python3 scripts/generate_llms_txt.py > llms.txt # 2. Regenerate llms.txt
|
|
8176
|
+
python3 scripts/validate.py --strict # 3. Validation passed?
|
|
8177
|
+
npm test > /tmp/npm-test.log 2>&1 && grep -c '^ok ' /tmp/npm-test.log && ! grep -q '^not ok' /tmp/npm-test.log # 4. All tests passed? (single run, cached)
|
|
8210
8178
|
|
|
8211
|
-
|
|
8212
|
-
|
|
8213
|
-
|
|
8214
|
-
|
|
8215
|
-
|
|
8179
|
+
# Post-install verification (Phases 1-7)
|
|
8180
|
+
ai-toolkit --version # 5. Version OK?
|
|
8181
|
+
ai-toolkit status # 6. Status OK?
|
|
8182
|
+
ai-toolkit doctor # 7. Health check passed?
|
|
8183
|
+
ai-toolkit install --dry-run # 8. Global install OK?
|
|
8184
|
+
python3 scripts/audit_skills.py --ci # 9. Security audit clean?
|
|
8216
8185
|
|
|
8217
|
-
|
|
8186
|
+
# Supply-chain verification (Phase 8, v2.8.0+)
|
|
8187
|
+
python3 scripts/audit_skills.py --sarif | python3 -c "import json,sys; assert json.load(sys.stdin)['version']=='2.1.0'; print('SARIF OK')" # 10. SARIF 2.1.0 well-formed?
|
|
8188
|
+
python3 scripts/audit_skills.py --permissions | head -30 # 11. Broad-access skills reviewed?
|
|
8189
|
+
npm view @softspark/ai-toolkit@X.Y.Z --json | python3 -c "import json,sys; d=json.load(sys.stdin); assert d['dist']['attestations']['provenance']['predicateType']=='https://slsa.dev/provenance/v1'; print('PROVENANCE OK')" # 12. Provenance attested on npm?
|
|
8190
|
+
python3 scripts/claude_app.py verify # 13. Claude Chat/Cowork plugin contract valid?
|
|
8218
8191
|
|
|
8219
|
-
|
|
8220
|
-
|
|
8221
|
-
|
|
8192
|
+
# Deep-coverage verification (Phase 9, v3.0.0+)
|
|
8193
|
+
META="generate_agents_md.py|generate_llms_txt.py|generate_language_rules_skills.py"
|
|
8194
|
+
diff <(grep -oE 'scripts/generate_[a-z_]+\.py' kb/reference/supported-tools-registry.md | sort -u) <(ls scripts/generate_*.py | grep -vE "$META" | sort -u) && echo "OK: registry matches" # 14. Registry <-> generators drift?
|
|
8195
|
+
```
|
|
8222
8196
|
|
|
8223
|
-
|
|
8197
|
+
---
|
|
8224
8198
|
|
|
8225
|
-
|
|
8226
|
-
|------|-------|---------------|
|
|
8227
|
-
| `kb/procedures/maintenance-sop.md` | frontmatter `version:` | If SOP content changed |
|
|
8228
|
-
| `kb/reference/skills-catalog.md` | frontmatter `version:` | If skills added/removed |
|
|
8229
|
-
| `kb/reference/agents-catalog.md` | frontmatter `version:` | If agents added/removed |
|
|
8230
|
-
| `kb/reference/hooks-catalog.md` | frontmatter `version:` | If hooks changed |
|
|
8231
|
-
| `kb/reference/architecture-overview.md` | frontmatter `version:` | If architecture changed |
|
|
8232
|
-
| `kb/reference/distribution-model.md` | frontmatter `version:` | If install model changed |
|
|
8233
|
-
| `kb/reference/global-install-model.md` | frontmatter `version:` | If install model changed |
|
|
8199
|
+
## Phase 0: Pre-Commit & Pre-Push (2 min)
|
|
8234
8200
|
|
|
8235
|
-
|
|
8236
|
-
|
|
8201
|
+
Run these commands **before every commit and push to main**. CI validates
|
|
8202
|
+
counts but does NOT auto-regenerate — you must do it locally.
|
|
8237
8203
|
|
|
8238
|
-
|
|
8204
|
+
```bash
|
|
8205
|
+
# 1. Regenerate generated artifacts
|
|
8206
|
+
python3 scripts/generate_agents_md.py > AGENTS.md
|
|
8207
|
+
python3 scripts/generate_llms_txt.py > llms.txt
|
|
8208
|
+
python3 scripts/generate_llms_txt.py --full > llms-full.txt
|
|
8239
8209
|
|
|
8240
|
-
|
|
8241
|
-
|
|
8242
|
-
| `package.json` | `"description"` — skill/agent count |
|
|
8243
|
-
| `README.md` | Badge counts, "What You Get" table |
|
|
8244
|
-
| `app/ARCHITECTURE.md` | Section headings with counts |
|
|
8210
|
+
# 2. Validate everything (catches stale counts, missing assets)
|
|
8211
|
+
python3 scripts/validate.py --strict
|
|
8245
8212
|
|
|
8246
|
-
|
|
8247
|
-
|
|
8248
|
-
> If validation passes, counts and versions are correct.
|
|
8213
|
+
# 3. Security audit
|
|
8214
|
+
python3 scripts/audit_skills.py --ci
|
|
8249
8215
|
|
|
8250
|
-
|
|
8216
|
+
# 4. Run tests
|
|
8217
|
+
npm test
|
|
8251
8218
|
|
|
8252
|
-
|
|
8219
|
+
# 5. Stage and commit
|
|
8220
|
+
# AGENTS.md, GEMINI.md, and .github/copilot-instructions.md are generated and
|
|
8221
|
+
# gitignored — do NOT commit them (regenerated at prepublishOnly). Only the
|
|
8222
|
+
# doc artifacts llms.txt / llms-full.txt are committed.
|
|
8223
|
+
git add llms.txt llms-full.txt
|
|
8224
|
+
git add -p # stage your other changes
|
|
8225
|
+
git commit -m "feat: your change description"
|
|
8226
|
+
```
|
|
8227
|
+
|
|
8228
|
+
**Why local?** Branch protection on `main` requires PRs and status checks.
|
|
8229
|
+
CI cannot push directly to `main`, so generated artifacts must be committed
|
|
8230
|
+
by the developer as part of their PR.
|
|
8253
8231
|
|
|
8232
|
+
**One-liner (copy-paste):**
|
|
8254
8233
|
```bash
|
|
8255
|
-
|
|
8256
|
-
echo "Target: $VERSION"
|
|
8257
|
-
echo "manifest.json: $(python3 -c "import json; print(json.load(open('manifest.json'))['version'])")"
|
|
8258
|
-
echo "plugin.json: $(python3 -c "import json; print(json.load(open('app/.claude-plugin/plugin.json'))['version'])")"
|
|
8259
|
-
echo "package-lock.json: $(python3 -c "import json; print(json.load(open('package-lock.json'))['version'])")"
|
|
8234
|
+
python3 scripts/generate_agents_md.py > AGENTS.md && python3 scripts/generate_llms_txt.py > llms.txt && python3 scripts/generate_llms_txt.py --full > llms-full.txt && python3 scripts/validate.py --strict && python3 scripts/audit_skills.py --ci && npm test
|
|
8260
8235
|
```
|
|
8261
8236
|
|
|
8262
|
-
|
|
8237
|
+
---
|
|
8263
8238
|
|
|
8264
|
-
|
|
8239
|
+
## Phase 1: CLI & Version (1 min)
|
|
8265
8240
|
|
|
8266
8241
|
```bash
|
|
8267
|
-
|
|
8268
|
-
|
|
8269
|
-
|
|
8270
|
-
|
|
8271
|
-
**Every line the diff removes is a breaking change.** Restore it, or take the
|
|
8272
|
-
deprecation path in `BACKWARD_COMPATIBILITY.md` and add a `DECISIONS.md` entry
|
|
8273
|
-
before the tag. Lines added are new surface being adopted into protection — that
|
|
8274
|
-
needs no ceremony.
|
|
8275
|
-
|
|
8276
|
-
Do not run `--update` to make a red build green. The check fails because something
|
|
8277
|
-
users depend on disappeared; regenerating the manifest deletes the evidence, not
|
|
8278
|
-
the problem.
|
|
8279
|
-
|
|
8280
|
-
### Skill body budget ratchet
|
|
8281
|
-
|
|
8282
|
-
`validate.py` prints the largest skill body on every run:
|
|
8283
|
-
|
|
8284
|
-
```
|
|
8285
|
-
Body budget: largest is <skill> at <N> bytes (warn 18000, error 20000)
|
|
8242
|
+
ai-toolkit --version
|
|
8243
|
+
ai-toolkit --help
|
|
8244
|
+
which ai-toolkit
|
|
8286
8245
|
```
|
|
8287
8246
|
|
|
8288
|
-
|
|
8289
|
-
`
|
|
8290
|
-
|
|
8291
|
-
|
|
8292
|
-
Two rules, both learned the hard way:
|
|
8293
|
-
|
|
8294
|
-
- **Never lower a threshold in the same change that something violates it.** Split
|
|
8295
|
-
the offending skill into `SKILL.md` + `reference/` first, prove it with
|
|
8296
|
-
`python3 scripts/check_split.py <skill> --before <pre-split SKILL.md>`, then
|
|
8297
|
-
tighten in a follow-up.
|
|
8298
|
-
- **Never raise a threshold to make a red build green.** A body over budget means
|
|
8299
|
-
detail that belongs in `reference/` is loading on every trigger match, including
|
|
8300
|
-
the accidental ones. Raising the number hides the cost, it does not remove it.
|
|
8301
|
-
|
|
8302
|
-
If the largest body has not moved since the previous release, leave the threshold
|
|
8303
|
-
alone and say so in the release notes. A ratchet that never advances is honest;
|
|
8304
|
-
one that advances past reality is not.
|
|
8247
|
+
**Verify:**
|
|
8248
|
+
- [ ] `--version` returns correct semver (e.g., `1.4.0`)
|
|
8249
|
+
- [ ] `--help` displays full command list without errors
|
|
8250
|
+
- [ ] `which` points to global npm bin path
|
|
8305
8251
|
|
|
8306
8252
|
---
|
|
8307
8253
|
|
|
8308
|
-
## Phase
|
|
8254
|
+
## Phase 2: Global Install & Status (2 min)
|
|
8309
8255
|
|
|
8310
|
-
|
|
8256
|
+
```bash
|
|
8257
|
+
ai-toolkit install --dry-run
|
|
8258
|
+
ai-toolkit status
|
|
8259
|
+
```
|
|
8311
8260
|
|
|
8312
|
-
|
|
8313
|
-
|
|
8261
|
+
**Verify `--dry-run`:**
|
|
8262
|
+
- [ ] Agents: 44
|
|
8263
|
+
- [ ] Skills: 108
|
|
8264
|
+
- [ ] Hooks merged into settings.json
|
|
8265
|
+
- [ ] "Other AI Tools" lists documented global targets (with `--editors`): aider, antigravity, augment, cline, codex, copilot, cursor, gemini, opencode, roo, windsurf. Scope varies: Codex uses `$CODEX_HOME` (default `~/.codex`) plus `$HOME/.agents/skills`; Copilot uses `$COPILOT_HOME` (default `~/.copilot`); cursor has only `~/.cursor/hooks.json`; antigravity has the `~/.gemini/*/skills` pointer. Cursor and Antigravity rules remain project-only.
|
|
8314
8266
|
|
|
8315
|
-
|
|
8316
|
-
-
|
|
8267
|
+
**Verify `status`:**
|
|
8268
|
+
- [ ] Version matches expected
|
|
8269
|
+
- [ ] Profile: minimal/standard/strict
|
|
8270
|
+
- [ ] Modules: list of installed modules
|
|
8271
|
+
- [ ] Latest: up to date / update available
|
|
8317
8272
|
|
|
8318
|
-
|
|
8319
|
-
- **What changed** — old behavior → new behavior
|
|
8273
|
+
---
|
|
8320
8274
|
|
|
8321
|
-
|
|
8322
|
-
- **Bug description** — what was broken and how it's fixed
|
|
8275
|
+
## Phase 3: Doctor Health Check (1 min)
|
|
8323
8276
|
|
|
8324
|
-
|
|
8325
|
-
-
|
|
8277
|
+
```bash
|
|
8278
|
+
ai-toolkit doctor
|
|
8326
8279
|
```
|
|
8327
8280
|
|
|
8328
|
-
**
|
|
8329
|
-
-
|
|
8330
|
-
-
|
|
8331
|
-
-
|
|
8332
|
-
-
|
|
8333
|
-
-
|
|
8334
|
-
-
|
|
8335
|
-
-
|
|
8336
|
-
|
|
8337
|
-
### Update README "What's New" section
|
|
8338
|
-
|
|
8339
|
-
**MANDATORY on every release.** Update the `## What's New in vX.Y.Z` section in `README.md`:
|
|
8340
|
-
|
|
8341
|
-
1. Change the heading version: `## What's New in vX.Y.Z`
|
|
8342
|
-
2. Replace bullet points with 3-5 highlights from this release
|
|
8343
|
-
3. **Keep only the latest version block.** Delete the previous `## What's New in vA.B.C` section(s). README is the shop window, not the archive — users see the current release, full history lives in `CHANGELOG.md`.
|
|
8344
|
-
4. Keep the `See [CHANGELOG.md](CHANGELOG.md) for full history.` link directly below the bullet list.
|
|
8281
|
+
**Expected sections (all OK):**
|
|
8282
|
+
- Environment: node, bash, python3, bats
|
|
8283
|
+
- Global Install: .claude exists, agents/skills symlinks (0 broken), settings.json hooks
|
|
8284
|
+
- Hook Scripts: all present and executable
|
|
8285
|
+
- Hook Configuration: 14 events registered
|
|
8286
|
+
- Generated Artifacts: AGENTS.md, llms.txt, llms-full.txt
|
|
8287
|
+
- Planned Assets: plugin.json, benchmarks, plugin packs
|
|
8288
|
+
- Benchmark Freshness: < 30 days
|
|
8289
|
+
- Stale Rules: all healthy
|
|
8345
8290
|
|
|
8346
|
-
|
|
8347
|
-
|
|
8348
|
-
|
|
8291
|
+
**Verify:**
|
|
8292
|
+
- [ ] `Errors: 0 | Warnings: 0`
|
|
8293
|
+
- [ ] `HEALTH CHECK PASSED`
|
|
8349
8294
|
|
|
8350
|
-
|
|
8295
|
+
If doctor detects problems: `ai-toolkit doctor --fix` auto-repairs
|
|
8296
|
+
(broken symlinks, non-executable hooks, missing scripts, missing llms-full.txt).
|
|
8351
8297
|
|
|
8352
8298
|
---
|
|
8353
8299
|
|
|
8354
|
-
## Phase 4:
|
|
8355
|
-
|
|
8356
|
-
Use the npm scripts, not the generators directly:
|
|
8300
|
+
## Phase 4: Local Install (2 min)
|
|
8357
8301
|
|
|
8358
8302
|
```bash
|
|
8359
|
-
|
|
8360
|
-
|
|
8303
|
+
mkdir -p /tmp/ai-toolkit-verify && cd /tmp/ai-toolkit-verify
|
|
8304
|
+
git init -q
|
|
8305
|
+
ai-toolkit install --local --editors all --dry-run
|
|
8306
|
+
cd - && rm -rf /tmp/ai-toolkit-verify
|
|
8361
8307
|
```
|
|
8362
8308
|
|
|
8363
|
-
|
|
8364
|
-
|
|
8365
|
-
|
|
8366
|
-
|
|
8367
|
-
|
|
8309
|
+
**Verify "Project-local" section:**
|
|
8310
|
+
- [ ] Would create: CLAUDE.md
|
|
8311
|
+
- [ ] Would create: .claude/settings.local.json
|
|
8312
|
+
- [ ] Would inject: .claude/constitution.md
|
|
8313
|
+
- [ ] Editors: all 11 listed (copilot, cursor, windsurf, cline, roo, aider, augment, antigravity, codex, gemini, opencode)
|
|
8314
|
+
- [ ] Would generate configs for each editor (legacy + directory-based)
|
|
8315
|
+
- [ ] Would install: .git/hooks/pre-commit
|
|
8316
|
+
- [ ] Would inject language rules (auto-detected)
|
|
8368
8317
|
|
|
8318
|
+
**Also test auto-detect (no --editors flag):**
|
|
8369
8319
|
```bash
|
|
8370
|
-
|
|
8320
|
+
ai-toolkit install --local --dry-run
|
|
8321
|
+
# → Editors: none (empty project has no existing configs)
|
|
8371
8322
|
```
|
|
8372
8323
|
|
|
8373
|
-
If no diff, the artifacts are already current. If there is a diff, stage them.
|
|
8374
|
-
|
|
8375
8324
|
---
|
|
8376
8325
|
|
|
8377
|
-
## Phase 5:
|
|
8378
|
-
|
|
8379
|
-
Run the full quality gate:
|
|
8326
|
+
## Phase 5: Validation & Security Audit (3 min)
|
|
8380
8327
|
|
|
8381
8328
|
```bash
|
|
8382
8329
|
python3 scripts/validate.py --strict
|
|
8383
8330
|
python3 scripts/audit_skills.py --ci
|
|
8384
|
-
|
|
8385
|
-
python3 scripts/audit_skills.py --permissions # review Bash/Write/Edit footprint
|
|
8331
|
+
```
|
|
8386
8332
|
|
|
8387
|
-
|
|
8388
|
-
|
|
8389
|
-
|
|
8390
|
-
|
|
8391
|
-
|
|
8333
|
+
**Verify validate.py:**
|
|
8334
|
+
- [ ] Agents: 44, Skills: 108, Tests: exactly the current README badge count
|
|
8335
|
+
- [ ] Hook events: 14, Hook scripts: >= 30
|
|
8336
|
+
- [ ] Plugin packs >= 10, KB documents >= 20
|
|
8337
|
+
- [ ] `Errors: 0 | Warnings: 0` → `VALIDATION PASSED`
|
|
8392
8338
|
|
|
8393
|
-
|
|
8394
|
-
|
|
8395
|
-
|
|
8396
|
-
|
|
8397
|
-
<(ls scripts/generate_*.py | grep -vE "$META" | sort -u) \
|
|
8398
|
-
&& echo "OK: registry matches filesystem" \
|
|
8399
|
-
|| { echo "DRIFT: update supported-tools-registry.md before tagging"; exit 1; }
|
|
8339
|
+
**Verify audit_skills.py:**
|
|
8340
|
+
- [ ] `HIGH: 0` (MUST be zero — CI fails otherwise)
|
|
8341
|
+
- [ ] `WARN: 0`
|
|
8342
|
+
- [ ] `INFO: N` (acceptable — broad-access skills: orchestrate, swarm, workflow)
|
|
8400
8343
|
|
|
8401
|
-
|
|
8402
|
-
# "npm package KB files match the tracked release set" compares `git ls-files
|
|
8403
|
-
# kb` (the index) against what `npm pack` sees (the working tree), so an
|
|
8404
|
-
# unstaged addition reads as "extra" and an unstaged deletion as "missing".
|
|
8405
|
-
# Phase 6 stages, and it runs after this one, so the ordering fails the test
|
|
8406
|
-
# for any release that touches the KB. Staging early costs nothing.
|
|
8407
|
-
git status --porcelain kb/ | grep -qE '^(\?\?| D|\?M)' && git add -A kb/
|
|
8344
|
+
---
|
|
8408
8345
|
|
|
8409
|
-
|
|
8410
|
-
|
|
8411
|
-
|
|
8346
|
+
## Phase 6: Tests (3-5 min)
|
|
8347
|
+
|
|
8348
|
+
```bash
|
|
8349
|
+
# Run ONCE, capture to file, then parse. Full suite is 669+ bats cases —
|
|
8350
|
+
# re-running it per check (tail / grep ok / grep not ok piped separately)
|
|
8351
|
+
# wastes minutes every release. Always cache the output.
|
|
8412
8352
|
npm test > /tmp/npm-test.log 2>&1
|
|
8353
|
+
exit=$?
|
|
8413
8354
|
tail -3 /tmp/npm-test.log
|
|
8414
|
-
echo "ok:
|
|
8355
|
+
echo "ok: $(grep -c '^ok ' /tmp/npm-test.log)"
|
|
8356
|
+
echo "not ok: $(grep -c '^not ok' /tmp/npm-test.log)"
|
|
8357
|
+
echo "exit: $exit"
|
|
8415
8358
|
```
|
|
8416
8359
|
|
|
8417
|
-
**
|
|
8418
|
-
-
|
|
8419
|
-
-
|
|
8420
|
-
-
|
|
8421
|
-
-
|
|
8422
|
-
-
|
|
8423
|
-
|
|
8424
|
-
- `npm test`: `1..N` with zero `not ok` (read from the cached `/tmp/npm-test.log`, do not rerun)
|
|
8425
|
-
|
|
8426
|
-
> **Why this matters (v4.5.1 postmortem):** `publish.yml` runs only `validate.py` + `npm test`, so it published v4.5.0 even though the `main` CI `ShellCheck hooks` job was red on two `SC2034` warnings in a new hook. The publish workflow does **not** depend on the CI workflow. Until that is fixed, ShellCheck is a manual pre-tag gate — run it here every time.
|
|
8360
|
+
**Verify:**
|
|
8361
|
+
- [ ] `exit == 0`
|
|
8362
|
+
- [ ] `ok == expected test count` (e.g., 945 on v3.0.0)
|
|
8363
|
+
- [ ] `not ok == 0`
|
|
8364
|
+
- [ ] Bats runs tests in parallel (4 jobs)
|
|
8365
|
+
- [ ] Groups: agents, autodetect, cli, generators, guards, hooks, inject,
|
|
8366
|
+
install, kb, mcp, readme, profiles, uninstall, validate
|
|
8427
8367
|
|
|
8428
|
-
**
|
|
8368
|
+
**Anti-pattern — do NOT do this:**
|
|
8429
8369
|
```bash
|
|
8430
|
-
|
|
8370
|
+
# Runs the full suite THREE times. Adds 1-3 min and pressures CI capacity.
|
|
8371
|
+
npm test 2>&1 | tail -3
|
|
8372
|
+
npm test 2>&1 | grep -c '^ok '
|
|
8373
|
+
npm test 2>&1 | grep -c '^not ok'
|
|
8431
8374
|
```
|
|
8432
8375
|
|
|
8433
|
-
**
|
|
8434
|
-
-
|
|
8435
|
-
-
|
|
8436
|
-
-
|
|
8376
|
+
**Key test areas:**
|
|
8377
|
+
- Guards: rm -rf, DROP TABLE, git push --force blocked
|
|
8378
|
+
- Install: idempotent, profiles, --only/--skip, orphan cleanup
|
|
8379
|
+
- Eject: real files (not symlinks), inlined rules
|
|
8380
|
+
- Uninstall: removes toolkit, preserves user content
|
|
8437
8381
|
|
|
8438
|
-
|
|
8382
|
+
---
|
|
8439
8383
|
|
|
8440
|
-
|
|
8441
|
-
carries it, and every part of that is mechanically checkable.
|
|
8384
|
+
## Phase 7: Eject (1 min)
|
|
8442
8385
|
|
|
8443
8386
|
```bash
|
|
8444
|
-
|
|
8445
|
-
|
|
8387
|
+
mkdir -p /tmp/ai-toolkit-eject-test
|
|
8388
|
+
cd /tmp/ai-toolkit-eject-test
|
|
8389
|
+
ai-toolkit eject
|
|
8390
|
+
cd - && rm -rf /tmp/ai-toolkit-eject-test
|
|
8446
8391
|
```
|
|
8447
8392
|
|
|
8448
|
-
|
|
8393
|
+
**Verify:**
|
|
8394
|
+
- [ ] Agents copied as real files (not symlinks)
|
|
8395
|
+
- [ ] Skills copied as real directories
|
|
8396
|
+
- [ ] Rules inlined into CLAUDE.md
|
|
8397
|
+
- [ ] constitution.md and ARCHITECTURE.md copied
|
|
8398
|
+
- [ ] `output-styles/` directory present (v2.7.1+)
|
|
8449
8399
|
|
|
8450
|
-
|
|
8451
|
-
|---|---|
|
|
8452
|
-
| Every shipped source file carries an SPDX header | A new `.py`/`.sh`/`.js`/`.bats` file was added without one — the common case, and the reason this is a test rather than a habit |
|
|
8453
|
-
| Headers name Apache-2.0 and nothing else | A file was copied in from an MIT/GPL source with its own header intact |
|
|
8454
|
-
| **No** markdown file carries an SPDX header | Someone "helpfully" ran the header script over `app/skills/` — headers there sit above parsed frontmatter and bill every session for it |
|
|
8455
|
-
| `LICENSE` is the complete Apache 2.0 text | The file was truncated or replaced with a summary |
|
|
8456
|
-
| `NOTICE` carries attribution, the source URL, §4(d) and the MIT-era notice | The attribution mechanism was gutted |
|
|
8457
|
-
| `LICENSE` **and** `NOTICE` ship in the npm package | `package.json` `files` lost an entry — a NOTICE that never reaches the consumer cannot satisfy §4(d) |
|
|
8458
|
-
| Every manifest declaring a licence declares Apache-2.0 | `package.json`, `manifest.json`, `plugin.json` and `package-lock.json` drifted apart |
|
|
8400
|
+
---
|
|
8459
8401
|
|
|
8460
|
-
|
|
8461
|
-
never before it. Short SPDX form:
|
|
8402
|
+
## Phase 8: Supply-Chain Verification (2 min, v2.8.0+)
|
|
8462
8403
|
|
|
8463
|
-
|
|
8464
|
-
# SPDX-License-Identifier: Apache-2.0
|
|
8465
|
-
# Copyright 2024-2026 Lukasz Krzemien (biuro@softspark.eu)
|
|
8466
|
-
# Source: https://github.com/softspark/ai-toolkit
|
|
8467
|
-
```
|
|
8404
|
+
These checks enforce the v2.8.0 security standard on a freshly-published release. Run AFTER the `publish.yml` workflow completes on the tag.
|
|
8468
8405
|
|
|
8469
|
-
|
|
8470
|
-
exclusion: [Licensing](../reference/licensing.md).
|
|
8406
|
+
### 8.1 Provenance attestation on npm
|
|
8471
8407
|
|
|
8472
|
-
|
|
8473
|
-
|
|
8474
|
-
|
|
8475
|
-
|
|
8476
|
-
|
|
8408
|
+
```bash
|
|
8409
|
+
VERSION="X.Y.Z" # the tag just published
|
|
8410
|
+
npm view "@softspark/ai-toolkit@${VERSION}" --json \
|
|
8411
|
+
| python3 -c "import json, sys; d=json.load(sys.stdin); att=d['dist'].get('attestations', {}); assert att.get('provenance', {}).get('predicateType') == 'https://slsa.dev/provenance/v1', f'NO PROVENANCE for {d[\"version\"]}'; print(f'PROVENANCE OK: {att[\"url\"]}')"
|
|
8412
|
+
```
|
|
8477
8413
|
|
|
8478
|
-
|
|
8414
|
+
**Verify:**
|
|
8415
|
+
- [ ] Exit 0 and prints `PROVENANCE OK: https://registry.npmjs.org/...`
|
|
8416
|
+
- [ ] `https://www.npmjs.com/package/@softspark/ai-toolkit/v/${VERSION}` shows the green "Provenance" badge
|
|
8479
8417
|
|
|
8480
|
-
|
|
8418
|
+
**If provenance is missing:** the `publish.yml` ran without `id-token: write` or `--provenance`. Restore them and cut a patch release — a silently unsigned publish is a regression against the v2.8.0 standard.
|
|
8481
8419
|
|
|
8482
|
-
|
|
8420
|
+
### 8.2 Audit SARIF output (for GHAS ingest)
|
|
8483
8421
|
|
|
8484
8422
|
```bash
|
|
8485
|
-
|
|
8423
|
+
python3 scripts/audit_skills.py --sarif > /tmp/audit.sarif
|
|
8424
|
+
python3 -c "import json; d=json.load(open('/tmp/audit.sarif')); assert d['version']=='2.1.0' and d['runs'][0]['tool']['driver']['name']=='ai-toolkit-audit-skills'; print(f'SARIF OK: {len(d[\"runs\"][0][\"results\"])} results across {len(d[\"runs\"][0][\"tool\"][\"driver\"][\"rules\"])} rules')"
|
|
8486
8425
|
```
|
|
8487
8426
|
|
|
8488
|
-
|
|
8489
|
-
- [ ]
|
|
8427
|
+
**Verify:**
|
|
8428
|
+
- [ ] Valid SARIF 2.1.0
|
|
8429
|
+
- [ ] In the publishing repo, the CI job uploads `audit.sarif` via `github/codeql-action/upload-sarif@v3` so findings appear in the Security tab
|
|
8490
8430
|
|
|
8491
|
-
|
|
8431
|
+
### 8.3 Per-skill permissions report
|
|
8492
8432
|
|
|
8493
8433
|
```bash
|
|
8494
|
-
|
|
8495
|
-
# (schema_version 1: entries live under the .rules / .hooks key):
|
|
8496
|
-
jq '.rules | to_entries | map(select(.value.url != null and (.value.sha256 // "" | length) == 0))' ~/.softspark/ai-toolkit/rules/sources.json
|
|
8497
|
-
jq '.hooks | to_entries | map(select(.value.url != null and (.value.sha256 // "" | length) == 0))' ~/.softspark/ai-toolkit/hooks/external/sources.json
|
|
8434
|
+
python3 scripts/audit_skills.py --permissions | head -40
|
|
8498
8435
|
```
|
|
8499
8436
|
|
|
8500
|
-
|
|
8501
|
-
- [ ]
|
|
8437
|
+
**Verify:**
|
|
8438
|
+
- [ ] Bash skill count has NOT jumped unexpectedly since the previous release
|
|
8439
|
+
- [ ] Any newly added entry under `Skills with Bash + Write + Edit` matches a CHANGELOG bullet that justifies the broad scope
|
|
8440
|
+
- [ ] JSON form (`--permissions --json`) is available for automated drift dashboards
|
|
8502
8441
|
|
|
8503
|
-
|
|
8442
|
+
### 8.4 URL-sourced rules/hooks are checksum-pinned
|
|
8443
|
+
|
|
8444
|
+
`sources.json` is an envelope of the form `{"schema_version": 1, "rules"|"hooks": {...}}`, so the jq filter must pick the nested map before piping into the pin assertion. The URL-count in the success message ignores local `path:`-only entries — only entries with a `url:` field are pinned and counted.
|
|
8504
8445
|
|
|
8505
8446
|
```bash
|
|
8506
|
-
|
|
8447
|
+
jq '.rules // .hooks // {}' ~/.softspark/ai-toolkit/rules/sources.json 2>/dev/null \
|
|
8448
|
+
| python3 -c "import json, sys; d=json.load(sys.stdin) or {}; bad=[n for n,v in d.items() if v.get('url') and not v.get('sha256')]; assert not bad, f'UNPINNED: {bad}'; url_n=sum(1 for v in d.values() if v.get('url')); print(f'RULE PIN OK: {url_n} URL rules, all with sha256')"
|
|
8449
|
+
jq '.hooks // {}' ~/.softspark/ai-toolkit/hooks/external/sources.json 2>/dev/null \
|
|
8450
|
+
| python3 -c "import json, sys; d=json.load(sys.stdin) or {}; bad=[n for n,v in d.items() if v.get('url') and not v.get('sha256')]; assert not bad, f'UNPINNED: {bad}'; url_n=sum(1 for v in d.values() if v.get('url')); print(f'HOOK PIN OK: {url_n} URL hooks, all with sha256')"
|
|
8507
8451
|
```
|
|
8508
8452
|
|
|
8509
|
-
|
|
8510
|
-
- [ ]
|
|
8453
|
+
**Verify:**
|
|
8454
|
+
- [ ] Both commands print `... PIN OK`
|
|
8455
|
+
- [ ] If any entry is unpinned, the `register_url_source()` call missed passing `content=` — fix the call-site and retag
|
|
8511
8456
|
|
|
8512
|
-
|
|
8457
|
+
### 8.5 Strict-pin smoke test (optional but recommended)
|
|
8513
8458
|
|
|
8514
8459
|
```bash
|
|
8515
8460
|
AI_TOOLKIT_STRICT_PIN=1 ai-toolkit update --dry-run
|
|
8516
8461
|
```
|
|
8517
8462
|
|
|
8463
|
+
**Verify:**
|
|
8518
8464
|
- [ ] Exit 0, no `CHECKSUM CHANGED` line
|
|
8519
|
-
- [ ]
|
|
8465
|
+
- [ ] If a checksum change was intentional (e.g. upstream rule update), document it in the CHANGELOG entry before tagging
|
|
8520
8466
|
|
|
8521
8467
|
---
|
|
8522
8468
|
|
|
8523
|
-
## Phase
|
|
8469
|
+
## Phase 9: Deep-Coverage Checks (v3.0.0+)
|
|
8524
8470
|
|
|
8525
|
-
|
|
8471
|
+
These verify the native-surface generators shipped in v3.0.0 actually emit the right files for the right profiles, and that the tool registry stays in sync with shipped generators.
|
|
8472
|
+
|
|
8473
|
+
> **Safety warning — HOME-scoped writes:** Running `--profile full` with `augment` in the editor list writes to `$HOME/.augment/settings.json` (Augment stores hooks under HOME, not per-project). Use `--dry-run` for verification unless you intend to carry ai-toolkit hook entries on this machine. The generator is marker-safe (only rewrites its own `_source: ai-toolkit` entries) but is still a side-effect.
|
|
8474
|
+
|
|
8475
|
+
### 9.1 `--profile full` emits every native surface
|
|
8526
8476
|
|
|
8527
8477
|
```bash
|
|
8528
|
-
|
|
8529
|
-
|
|
8530
|
-
|
|
8531
|
-
|
|
8532
|
-
# NOTE: AGENTS.md, GEMINI.md, and .github/copilot-instructions.md are generated
|
|
8533
|
-
# editor configs and are gitignored — do NOT commit them. `prepublishOnly` runs
|
|
8534
|
-
# `npm run generate:all`, so the shipped package (which lists AGENTS.md in
|
|
8535
|
-
# package.json `files`) gets a fresh copy at publish time.
|
|
8536
|
-
git add -p # review and stage any other changes
|
|
8478
|
+
D=/tmp/aitk-profile-full-${RANDOM} && mkdir -p "$D" && cd "$D" && git init -q
|
|
8479
|
+
ai-toolkit install --local --editors cursor,windsurf,gemini,augment,codex,copilot \
|
|
8480
|
+
--profile full --dry-run 2>&1 \
|
|
8481
|
+
| grep -E "\\.cursor/(hooks\\.json|agents)|\\.devin/hooks\\.v1\\.json|\\.gemini/(settings\\.json|commands)|\\.augment/(agents|commands)|\\.codex/(agents|hooks)|\\.agents/skills|\\.github/(agents|skills|instructions|prompts|hooks)"
|
|
8537
8482
|
```
|
|
8538
8483
|
|
|
8539
|
-
|
|
8484
|
+
**Verify** — at least the following lines appear:
|
|
8485
|
+
- [ ] `.cursor/hooks.json` and `.cursor/agents/`
|
|
8486
|
+
- [ ] `.devin/hooks.v1.json`
|
|
8487
|
+
- [ ] `.gemini/settings.json` hooks AND `.gemini/commands/`
|
|
8488
|
+
- [ ] `.augment/agents/` + `.augment/commands/` + `$HOME/.augment/settings.json`
|
|
8489
|
+
- [ ] `.codex/hooks.json` + `.codex/hooks/` + `.codex/agents/`
|
|
8490
|
+
- [ ] `.agents/skills/` (Codex native discovery path)
|
|
8491
|
+
- [ ] `.github/agents/` + `.github/skills/` + `.github/instructions/` + `.github/prompts/` + `.github/hooks/`
|
|
8492
|
+
|
|
8493
|
+
### 9.2 Codex skills are profile-independent
|
|
8540
8494
|
|
|
8541
8495
|
```bash
|
|
8542
|
-
|
|
8496
|
+
D=/tmp/aitk-codex-skills-${RANDOM} && mkdir -p "$D" && cd "$D" && git init -q
|
|
8497
|
+
ai-toolkit install --local --editors codex --profile minimal --dry-run 2>&1 \
|
|
8498
|
+
| grep -q "Would generate: .agents/skills" && echo "OK: Codex skills emit at minimal"
|
|
8499
|
+
ai-toolkit install --local --editors codex --profile standard --codex-skills --dry-run 2>&1 \
|
|
8500
|
+
| grep -q "Would refresh: .agents/skills" && echo "OK: legacy flag explicitly refreshes the same catalog"
|
|
8543
8501
|
```
|
|
8544
8502
|
|
|
8545
|
-
|
|
8503
|
+
**Verify:**
|
|
8504
|
+
- [ ] Selecting Codex emits all skills under `.agents/skills/` at every profile
|
|
8505
|
+
- [ ] `--codex-skills` remains an explicit refresh alias; it is not required for installation
|
|
8506
|
+
- [ ] No profile emits `.codex/skills/`
|
|
8546
8507
|
|
|
8547
|
-
|
|
8508
|
+
### 9.3 Breaking-change surfaces land on `--profile standard`
|
|
8509
|
+
|
|
8510
|
+
The standard profile must include:
|
|
8511
|
+
- Copilot scoped instructions, prompts, custom agents, portable skills, and hooks
|
|
8512
|
+
- Gemini hooks (`.gemini/settings.json`)
|
|
8548
8513
|
|
|
8549
8514
|
```bash
|
|
8550
|
-
|
|
8551
|
-
|
|
8515
|
+
D=/tmp/aitk-breaking-${RANDOM} && mkdir -p "$D" && cd "$D" && git init -q
|
|
8516
|
+
ai-toolkit install --local --editors copilot,gemini --profile standard --dry-run 2>&1 \
|
|
8517
|
+
| tee /tmp/aitk-breaking.log
|
|
8518
|
+
grep -q "\\.github/instructions/" /tmp/aitk-breaking.log && echo "OK: Copilot dir layout at standard"
|
|
8519
|
+
grep -q "\\.github/agents/.*\\.github/skills/" /tmp/aitk-breaking.log && echo "OK: Copilot agents and skills at standard"
|
|
8520
|
+
grep -q "\\.github/hooks/" /tmp/aitk-breaking.log && echo "OK: Copilot hooks at standard"
|
|
8521
|
+
grep -q "\\.gemini/settings\\.json hooks" /tmp/aitk-breaking.log && echo "OK: Gemini hooks at standard"
|
|
8522
|
+
```
|
|
8552
8523
|
|
|
8553
|
-
|
|
8554
|
-
# asynchronous, so retry at most five times and log every attempt.
|
|
8555
|
-
RUN_ID=""
|
|
8556
|
-
for ATTEMPT in 1 2 3 4 5; do
|
|
8557
|
-
RUN_ID=$(gh run list --workflow ci.yml --event push --commit "$RELEASE_SHA" \
|
|
8558
|
-
--limit 1 --json databaseId --jq '.[0].databaseId')
|
|
8559
|
-
[ -n "$RUN_ID" ] && break
|
|
8560
|
-
echo "Waiting for CI run registration ($ATTEMPT/5)" >&2
|
|
8561
|
-
[ "$ATTEMPT" -eq 5 ] || sleep 60
|
|
8562
|
-
done
|
|
8563
|
-
test -n "$RUN_ID" || { echo "FAIL: CI run not found for release commit"; exit 1; }
|
|
8564
|
-
gh run watch "$RUN_ID" --exit-status \
|
|
8565
|
-
|| { echo "FAIL: release commit CI failed"; exit 1; }
|
|
8524
|
+
**Verify all four lines print `OK:`**. If any is missing, native Copilot or Gemini coverage has regressed.
|
|
8566
8525
|
|
|
8567
|
-
|
|
8568
|
-
# the tag so a green publish job cannot bypass a red macOS test.
|
|
8569
|
-
for OS in ubuntu-latest macos-latest; do
|
|
8570
|
-
gh run view "$RUN_ID" --json jobs \
|
|
8571
|
-
--jq ".jobs[] | select(.name == \"Bats test suite ($OS)\") | .conclusion" \
|
|
8572
|
-
| grep -qx success || { echo "FAIL: $OS test job not green"; exit 1; }
|
|
8573
|
-
done
|
|
8526
|
+
### 9.4 Install is idempotent
|
|
8574
8527
|
|
|
8575
|
-
|
|
8576
|
-
|
|
8577
|
-
|
|
8578
|
-
|
|
8579
|
-
|
|
8580
|
-
|
|
8528
|
+
```bash
|
|
8529
|
+
D=/tmp/aitk-idem-${RANDOM} && mkdir -p "$D" && cd "$D" && git init -q
|
|
8530
|
+
# Sort file paths before hashing — find traversal order follows inode order,
|
|
8531
|
+
# which can shift between runs even when content is byte-identical, producing
|
|
8532
|
+
# false FAIL signals.
|
|
8533
|
+
ai-toolkit install --local --editors cursor,gemini --profile full >/dev/null 2>&1
|
|
8534
|
+
SHA1=$(find .cursor .gemini -type f -print0 | LC_ALL=C sort -z | xargs -0 shasum | shasum | awk '{print $1}')
|
|
8535
|
+
ai-toolkit install --local --editors cursor,gemini --profile full >/dev/null 2>&1
|
|
8536
|
+
SHA2=$(find .cursor .gemini -type f -print0 | LC_ALL=C sort -z | xargs -0 shasum | shasum | awk '{print $1}')
|
|
8537
|
+
[ "$SHA1" = "$SHA2" ] && echo "OK: idempotent" || echo "FAIL: install is not idempotent"
|
|
8538
|
+
```
|
|
8581
8539
|
|
|
8582
|
-
|
|
8540
|
+
**Verify:** prints `OK: idempotent`. A second run must produce byte-identical files in every managed path.
|
|
8583
8541
|
|
|
8584
|
-
|
|
8585
|
-
# have caught a real broken release.
|
|
8586
|
-
test "$(git rev-parse vX.Y.Z)" = "$RELEASE_SHA" \
|
|
8587
|
-
|| { echo "FAIL: tag is not on the tested release commit"; exit 1; }
|
|
8588
|
-
git show --no-patch --format=%s vX.Y.Z | grep -qx "chore: release vX.Y.Z" \
|
|
8589
|
-
|| { echo "FAIL: tag is not on the chore: release commit"; exit 1; }
|
|
8542
|
+
### 9.5 Live-install JSON outputs parse
|
|
8590
8543
|
|
|
8591
|
-
|
|
8592
|
-
|
|
8544
|
+
The bats suite validates JSON shape at generation time. This re-checks that what actually landed on disk after a live install parses without errors.
|
|
8545
|
+
|
|
8546
|
+
```bash
|
|
8547
|
+
D=/tmp/aitk-json-${RANDOM} && mkdir -p "$D" && cd "$D" && git init -q
|
|
8548
|
+
ai-toolkit install --local --editors cursor,windsurf,gemini,augment,codex,copilot --profile full >/dev/null 2>&1
|
|
8549
|
+
for f in .cursor/hooks.json .devin/hooks.v1.json .gemini/settings.json .codex/hooks.json .github/hooks/ai-toolkit.json "$HOME/.augment/settings.json"; do
|
|
8550
|
+
[ -f "$f" ] && python3 -c "import json; json.load(open('$f'))" && echo "OK: $f"
|
|
8551
|
+
done
|
|
8593
8552
|
```
|
|
8594
8553
|
|
|
8595
|
-
**
|
|
8596
|
-
workflow runs only on Ubuntu and can publish while the separate macOS matrix job
|
|
8597
|
-
is red. `v4.30.2` exposed this with a macOS-only DSH timing failure. A release
|
|
8598
|
-
commit must therefore pass the complete Ubuntu/macOS branch workflow before its
|
|
8599
|
-
tag exists; a successful publish workflow is not a substitute for green CI.
|
|
8554
|
+
**Verify:** each emitted file prints `OK: <path>`. Any `json.decoder.JSONDecodeError` means the merge logic corrupted the output.
|
|
8600
8555
|
|
|
8601
|
-
|
|
8602
|
-
that contained only a KB document and still carried `package.json` version
|
|
8603
|
-
`4.18.0`; the actual release sat in the commit above it under a recycled
|
|
8604
|
-
`fix:` message. `publish.yml` fired, tried to publish a version already on
|
|
8605
|
-
npm, and failed. Nothing on npm, a tag pointing at the wrong tree, and the
|
|
8606
|
-
only way out was rewriting a pushed commit. Both assertions above catch this
|
|
8607
|
-
in under a second. Run them.
|
|
8556
|
+
### 9.6 Registry / generator drift check
|
|
8608
8557
|
|
|
8609
|
-
|
|
8610
|
-
suppresses tag-triggered workflow runs when many tags arrive in a single push
|
|
8611
|
-
— the workflow silently does not fire and nothing publishes. Push the single
|
|
8612
|
-
release tag by its full ref, as above. (Sibling evidence: this is exactly how
|
|
8613
|
-
rag-mcp's `1.0.3` image build was skipped, when a `--tags` push carried 37
|
|
8614
|
-
tags at once.)
|
|
8558
|
+
`kb/reference/supported-tools-registry.md` should enumerate every per-editor `scripts/generate_*.py` we ship. Meta-generators (`generate_agents_md.py`, `generate_llms_txt.py`) are excluded — they produce docs/artifacts, not editor configs.
|
|
8615
8559
|
|
|
8616
|
-
|
|
8617
|
-
|
|
8618
|
-
|
|
8619
|
-
|
|
8560
|
+
```bash
|
|
8561
|
+
META="generate_agents_md.py|generate_llms_txt.py|generate_language_rules_skills.py"
|
|
8562
|
+
REG=$(grep -oE 'scripts/generate_[a-z_]+\.py' kb/reference/supported-tools-registry.md | sort -u)
|
|
8563
|
+
FS=$(ls scripts/generate_*.py | grep -vE "$META" | sort -u)
|
|
8564
|
+
diff <(echo "$REG") <(echo "$FS") && echo "OK: registry matches filesystem" || echo "DRIFT: update supported-tools-registry.md"
|
|
8565
|
+
```
|
|
8620
8566
|
|
|
8621
|
-
**
|
|
8567
|
+
**Verify:** prints `OK: registry matches filesystem`. If not, add the missing rows to the registry before tagging the next release.
|
|
8622
8568
|
|
|
8623
|
-
|
|
8624
|
-
|
|
8569
|
+
### 9.7 Claude Chat / Cowork plugin validates and exports
|
|
8570
|
+
|
|
8571
|
+
```bash
|
|
8572
|
+
python3 scripts/claude_app.py verify
|
|
8573
|
+
D=/tmp/aitk-claude-app-${RANDOM}
|
|
8574
|
+
python3 scripts/claude_app.py export --output "$D.zip" --no-custom-rules
|
|
8575
|
+
python3 -c "import zipfile; z=zipfile.ZipFile('$D.zip'); assert '.claude-plugin/plugin.json' in z.namelist(); assert 'claude-app/skills/ai-toolkit-rules/SKILL.md' in z.namelist(); print('OK: Claude app archive')"
|
|
8576
|
+
```
|
|
8577
|
+
|
|
8578
|
+
**Verify:** the official validator exits 0; the archive contains the manifest,
|
|
8579
|
+
app-native rules skill, bundled agents/skills, and plugin-relative Cowork hooks.
|
|
8625
8580
|
|
|
8626
8581
|
---
|
|
8627
8582
|
|
|
8628
|
-
##
|
|
8583
|
+
## Phase 10: Explicit DSH Qualification (v4.30.0+)
|
|
8629
8584
|
|
|
8630
|
-
|
|
8585
|
+
Run this phase whenever the release changes the `dsh` target, package pins, preset lifecycle, or DSH compatibility documentation. Use a new task-specific `DSH_HOME`; never replace `HOME` or reuse a regular profile.
|
|
8586
|
+
|
|
8587
|
+
Prerequisites: DSH `0.1.1-rc.2`, pnpm `>=11.7.0,<12.0.0`, Codex logged in through ChatGPT, Claude Code logged in natively, and GitHub Copilot CLI logged in natively. Do not supply provider API keys.
|
|
8631
8588
|
|
|
8632
8589
|
```bash
|
|
8633
|
-
|
|
8634
|
-
|
|
8590
|
+
DSH_SMOKE_ROOT="$(mktemp -d)"
|
|
8591
|
+
export DSH_HOME="$DSH_SMOKE_ROOT/dsh-home"
|
|
8635
8592
|
|
|
8636
|
-
|
|
8637
|
-
|
|
8593
|
+
ai-toolkit dsh install --profile web
|
|
8594
|
+
ai-toolkit dsh doctor --profile web
|
|
8595
|
+
dsh --profile web --host 127.0.0.1 --port 0 --no-open
|
|
8596
|
+
```
|
|
8638
8597
|
|
|
8639
|
-
|
|
8640
|
-
|
|
8641
|
-
|
|
8598
|
+
In a new `softspark-orchestrator` session, select the `codex` provider and run two standalone marker prompts:
|
|
8599
|
+
|
|
8600
|
+
1. `subagent_claude_code` returns an exact child marker, then the Codex parent returns its exact completion marker.
|
|
8601
|
+
2. `subagent_gemini_copilot` returns an exact child marker, then the Codex parent returns its exact completion marker.
|
|
8602
|
+
|
|
8603
|
+
Stop DSH, then remove only the managed profile artifacts:
|
|
8604
|
+
|
|
8605
|
+
```bash
|
|
8606
|
+
ai-toolkit dsh uninstall --profile web --yes
|
|
8642
8607
|
```
|
|
8643
8608
|
|
|
8609
|
+
**Verify:** both tool results have `isError=false`, both turns end as `completed`, `doctor` reports no recovery requirement before uninstall, and an unrelated preset fixture remains unchanged. Preserve only redacted event sequence evidence; never attach credentials, auth files, or full private prompts.
|
|
8610
|
+
|
|
8644
8611
|
---
|
|
8645
8612
|
|
|
8646
|
-
##
|
|
8613
|
+
## Troubleshooting
|
|
8647
8614
|
|
|
8648
|
-
|
|
8649
|
-
|---|------|-----------------|---------------|
|
|
8650
|
-
| 0a | Ecosystem drift check | `ecosystem_doctor.py --format text` | All tools Clean, or drift classified and resolved |
|
|
8651
|
-
| 0b | Ecosystem snapshot refresh | `ecosystem_doctor.py --update` | `benchmarks/ecosystem-doctor-snapshot.json` updated |
|
|
8652
|
-
| 0c | Ecosystem gate | `ecosystem_doctor.py --offline --check` | Exit 0 |
|
|
8653
|
-
| 1 | Version bump type | Decide patch/minor/major | — |
|
|
8654
|
-
| 2 | `package.json` version | Edit `"version"` | Matches target |
|
|
8655
|
-
| 3 | `manifest.json` version | Edit `"version"` | Matches target |
|
|
8656
|
-
| 4 | `plugin.json` version | Edit `"version"` | Matches target |
|
|
8657
|
-
| 5 | `package-lock.json` | `npm install --package-lock-only` | Matches target |
|
|
8658
|
-
| 6 | Count sync | Check `package.json` description, README | `validate.py` passes |
|
|
8659
|
-
| 7 | CHANGELOG.md | Add release entry (incl. `Ecosystem` subsection if any B/D/E/F drift) | Entry exists for vX.Y.Z |
|
|
8660
|
-
| 8 | Regenerate artifacts | `generate_agents_md.py`, `generate_llms_txt.py` | No unexpected diff |
|
|
8661
|
-
| 9 | Validate | `validate.py --strict` | 0 errors, 0 warnings |
|
|
8662
|
-
| 10 | Security audit (CI mode) | `audit_skills.py --ci` | 0 HIGH |
|
|
8663
|
-
| 11 | Security audit (SARIF) | `audit_skills.py --sarif` | Valid SARIF 2.1.0 JSON |
|
|
8664
|
-
| 12 | Per-skill permissions | `audit_skills.py --permissions` | New broad-access skills justified in CHANGELOG |
|
|
8665
|
-
| 13 | ShellCheck hooks | `shellcheck --severity=warning app/hooks/*.sh` | Exit 0, no output (mirrors ci.yml; publish.yml does NOT run it) |
|
|
8666
|
-
| 14 | Provenance flag check | `grep -- '--provenance' .github/workflows/publish.yml` | Present |
|
|
8667
|
-
| 15 | Checksum-pin backfill | `sources.json` entries all have `sha256` | No unpinned URL sources |
|
|
8668
|
-
| 15a | Licensing gate | `npx bats tests/test_licensing.bats` | 7/7 — SPDX headers, LICENSE, NOTICE, npm `files`, manifest consistency |
|
|
8669
|
-
| 16 | Tests | `git add -A kb/` if the KB changed, then `npm test` | All pass |
|
|
8670
|
-
| 17 | Commit | `git commit` | Clean working tree |
|
|
8671
|
-
| 18 | Push branch | `git push origin main` | Exact release commit is on `origin/main` |
|
|
8672
|
-
| 18a | Full branch CI | `gh run watch "$RUN_ID" --exit-status` plus matrix job assertions | Ubuntu and macOS Bats jobs both conclude `success` |
|
|
8673
|
-
| 19 | Tag | `git tag vX.Y.Z` | Tag exists only after green branch CI |
|
|
8674
|
-
| 19a | Tag is on tested SHA | `test "$(git rev-parse vX.Y.Z)" = "$RELEASE_SHA"` | Exit 0 |
|
|
8675
|
-
| 19b | Tag is on the release commit | `git show --no-patch --format=%s vX.Y.Z` | Reads `chore: release vX.Y.Z` |
|
|
8676
|
-
| 20 | Push the single tag | `git push origin refs/tags/vX.Y.Z` | Publish CI triggered with `id-token: write`. Never `--tags`. |
|
|
8615
|
+
### `ai-toolkit: command not found`
|
|
8677
8616
|
|
|
8678
|
-
|
|
8617
|
+
```bash
|
|
8618
|
+
npm install -g @softspark/ai-toolkit
|
|
8619
|
+
# or check PATH:
|
|
8620
|
+
export PATH="$(npm config get prefix)/bin:$PATH"
|
|
8621
|
+
```
|
|
8679
8622
|
|
|
8680
|
-
|
|
8623
|
+
### Doctor: broken symlinks
|
|
8681
8624
|
|
|
8682
|
-
|
|
8683
|
-
|
|
8684
|
-
|
|
8685
|
-
|
|
8686
|
-
tags: [sop, verification, release, smoke-test, install, update, qa, provenance, sarif, dsh]
|
|
8687
|
-
version: "1.8.0"
|
|
8688
|
-
created: "2026-04-08"
|
|
8689
|
-
last_updated: "2026-09-01"
|
|
8690
|
-
description: "End-to-end smoke test after installing or updating @softspark/ai-toolkit. Verifies CLI, native Codex and GitHub Copilot surfaces, explicit DSH lifecycle, Claude app export, doctor, validation, tests, eject, provenance, SARIF, and per-skill permissions."
|
|
8691
|
-
---
|
|
8625
|
+
```bash
|
|
8626
|
+
ai-toolkit doctor --fix # auto-repair
|
|
8627
|
+
ai-toolkit update # or full re-install
|
|
8628
|
+
```
|
|
8692
8629
|
|
|
8693
|
-
|
|
8630
|
+
### Tests fail: missing bats
|
|
8694
8631
|
|
|
8695
|
-
|
|
8696
|
-
|
|
8632
|
+
```bash
|
|
8633
|
+
brew install bats-core # macOS
|
|
8634
|
+
npm install -g bats # cross-platform
|
|
8635
|
+
```
|
|
8697
8636
|
|
|
8698
|
-
|
|
8699
|
-
- After `npm install -g @softspark/ai-toolkit@latest`
|
|
8700
|
-
- After `ai-toolkit update`
|
|
8701
|
-
- Before tagging a new version (`git tag`)
|
|
8702
|
-
- Before publishing to npm (`npm publish`)
|
|
8703
|
-
- As a smoke test in CI/CD
|
|
8637
|
+
### validate.py: stale counts
|
|
8704
8638
|
|
|
8705
|
-
|
|
8706
|
-
|
|
8707
|
-
- `@softspark/ai-toolkit` installed globally
|
|
8639
|
+
README badges don't match the current agents/skills/tests counts.
|
|
8640
|
+
Update README.md and re-run.
|
|
8708
8641
|
|
|
8709
|
-
|
|
8642
|
+
### Eject: missing skills
|
|
8643
|
+
|
|
8644
|
+
```bash
|
|
8645
|
+
ai-toolkit update # re-link missing symlinks
|
|
8646
|
+
ai-toolkit eject /tmp/test # retry
|
|
8647
|
+
```
|
|
8710
8648
|
|
|
8711
8649
|
---
|
|
8712
8650
|
|
|
8713
|
-
##
|
|
8651
|
+
## Success Criteria
|
|
8714
8652
|
|
|
8715
|
-
|
|
8653
|
+
| Area | Criterion |
|
|
8654
|
+
|------|-----------|
|
|
8655
|
+
| CLI | `--version` correct, `--help` full list, `status` current |
|
|
8656
|
+
| Health | `doctor`: 0 errors, 0 warnings, PASSED |
|
|
8657
|
+
| Install | `--dry-run` correct counts, `--local` all configs |
|
|
8658
|
+
| Quality | `validate.py --strict`: PASSED |
|
|
8659
|
+
| Security (baseline) | `audit_skills.py --ci`: 0 HIGH |
|
|
8660
|
+
| Security (SARIF) | `audit_skills.py --sarif`: valid SARIF 2.1.0 with non-empty rules array |
|
|
8661
|
+
| Security (permissions) | `audit_skills.py --permissions`: broad-access skills unchanged or justified in CHANGELOG |
|
|
8662
|
+
| Supply chain | `dist.attestations.provenance.predicateType == https://slsa.dev/provenance/v1` on npm |
|
|
8663
|
+
| Supply chain | All `sources.json` URL entries carry a `sha256`; `AI_TOOLKIT_STRICT_PIN=1 ai-toolkit update --dry-run` passes |
|
|
8664
|
+
| Tests | `npm test`: N/N passed, 0 failures |
|
|
8665
|
+
| Eject | Standalone `.claude/` with real files AND `output-styles/` directory |
|
|
8666
|
+
| Guards | Destructive commands blocked |
|
|
8667
|
+
| Deep coverage | `--profile full` emits every documented native surface, including Codex agents/skills/hooks and Copilot agents/skills/hooks |
|
|
8668
|
+
| Profile contract | Codex skills emit at every profile without `--codex-skills`; Copilot standard emits agents, skills, instructions, prompts, and hooks |
|
|
8669
|
+
| Idempotence | Second `install` run produces byte-identical output in every managed path |
|
|
8670
|
+
| Live JSON | Every generated `.json` file on disk parses as valid JSON |
|
|
8671
|
+
| Registry | `supported-tools-registry.md` enumerates every `scripts/generate_*.py` we ship |
|
|
8716
8672
|
|
|
8717
|
-
|
|
8718
|
-
# Pre-commit (Phase 0)
|
|
8719
|
-
python3 scripts/generate_agents_md.py > AGENTS.md # 1. Regenerate AGENTS.md
|
|
8720
|
-
python3 scripts/generate_llms_txt.py > llms.txt # 2. Regenerate llms.txt
|
|
8721
|
-
python3 scripts/validate.py --strict # 3. Validation passed?
|
|
8722
|
-
npm test > /tmp/npm-test.log 2>&1 && grep -c '^ok ' /tmp/npm-test.log && ! grep -q '^not ok' /tmp/npm-test.log # 4. All tests passed? (single run, cached)
|
|
8673
|
+
---
|
|
8723
8674
|
|
|
8724
|
-
|
|
8725
|
-
ai-toolkit --version # 5. Version OK?
|
|
8726
|
-
ai-toolkit status # 6. Status OK?
|
|
8727
|
-
ai-toolkit doctor # 7. Health check passed?
|
|
8728
|
-
ai-toolkit install --dry-run # 8. Global install OK?
|
|
8729
|
-
python3 scripts/audit_skills.py --ci # 9. Security audit clean?
|
|
8675
|
+
## kb/procedures/sop-release.md
|
|
8730
8676
|
|
|
8731
|
-
|
|
8732
|
-
|
|
8733
|
-
|
|
8734
|
-
|
|
8735
|
-
|
|
8677
|
+
---
|
|
8678
|
+
title: "SOP: Release Preparation"
|
|
8679
|
+
category: procedures
|
|
8680
|
+
service: ai-toolkit
|
|
8681
|
+
tags: [sop, release, version, publish, changelog, semver, provenance, sarif, ecosystem, shellcheck]
|
|
8682
|
+
version: "1.15.0"
|
|
8683
|
+
created: "2026-04-10"
|
|
8684
|
+
last_updated: "2026-09-02"
|
|
8685
|
+
description: "Step-by-step checklist for preparing a new ai-toolkit release — ecosystem-sync drift check, version sync, changelog, artifact regeneration, validation, branch CI, and tagging. Run BEFORE every git tag. Includes mandatory Provenance, SARIF, checksum-pin, ShellCheck, licensing, exact-tag assertions, and a green Ubuntu/macOS branch-CI gate before any release tag is created."
|
|
8686
|
+
---
|
|
8736
8687
|
|
|
8737
|
-
#
|
|
8738
|
-
|
|
8739
|
-
|
|
8688
|
+
# SOP: Release Preparation
|
|
8689
|
+
|
|
8690
|
+
Complete checklist for preparing a new `@softspark/ai-toolkit` release.
|
|
8691
|
+
Run this **before** tagging. After tagging and publishing, run the
|
|
8692
|
+
[Release Verification SOP](sop-release-verification.md) to smoke-test.
|
|
8693
|
+
|
|
8694
|
+
**Pipeline:**
|
|
8695
|
+
```
|
|
8696
|
+
Ecosystem Sync SOP (drift check + generator updates)
|
|
8697
|
+
↓
|
|
8698
|
+
Release Preparation (this SOP)
|
|
8699
|
+
↓
|
|
8700
|
+
git tag → CI publish → Release Verification SOP
|
|
8740
8701
|
```
|
|
8741
8702
|
|
|
8742
|
-
|
|
8703
|
+
**Time:** 10-20 minutes (includes ecosystem sync review)
|
|
8743
8704
|
|
|
8744
|
-
|
|
8705
|
+
---
|
|
8745
8706
|
|
|
8746
|
-
|
|
8747
|
-
counts but does NOT auto-regenerate — you must do it locally.
|
|
8707
|
+
## Quick Checklist (TL;DR)
|
|
8748
8708
|
|
|
8749
8709
|
```bash
|
|
8750
|
-
#
|
|
8710
|
+
# 0. Ecosystem sync (mandatory for minor/major releases; optional for patch)
|
|
8711
|
+
# Full procedure: kb/procedures/sop-ecosystem-sync.md
|
|
8712
|
+
python3 scripts/ecosystem_doctor.py --format text > /tmp/eco-report.txt
|
|
8713
|
+
cat /tmp/eco-report.txt
|
|
8714
|
+
# If drift detected: stop here, follow sop-ecosystem-sync.md Phase 2-4 to
|
|
8715
|
+
# classify each drift (A-F), update generators as needed, refresh snapshot,
|
|
8716
|
+
# THEN resume this SOP.
|
|
8717
|
+
python3 scripts/ecosystem_doctor.py --update # after all drift resolved
|
|
8718
|
+
|
|
8719
|
+
# 1. Decide version bump
|
|
8720
|
+
# patch (1.4.2 → 1.4.3): bugfix, typo, doc fix
|
|
8721
|
+
# minor (1.4.2 → 1.5.0): new feature, new skill, new flag, any ecosystem-class-B/F change
|
|
8722
|
+
# major (1.4.2 → 2.0.0): breaking change, any ecosystem-class-D removed path
|
|
8723
|
+
|
|
8724
|
+
# 2. Sync version across all files
|
|
8725
|
+
python3 scripts/sync_version.py X.Y.Z # if script exists, else manual
|
|
8726
|
+
|
|
8727
|
+
# 3. Write CHANGELOG.md entry
|
|
8728
|
+
# 4. Regenerate artifacts
|
|
8751
8729
|
python3 scripts/generate_agents_md.py > AGENTS.md
|
|
8752
8730
|
python3 scripts/generate_llms_txt.py > llms.txt
|
|
8753
8731
|
python3 scripts/generate_llms_txt.py --full > llms-full.txt
|
|
8754
8732
|
|
|
8755
|
-
#
|
|
8756
|
-
python3 scripts/validate.py --strict
|
|
8733
|
+
# 5. Validate + audit + SARIF + shellcheck + test + ecosystem check
|
|
8734
|
+
python3 scripts/validate.py --strict && python3 scripts/audit_skills.py --ci && python3 scripts/audit_skills.py --sarif > /tmp/audit.sarif && shellcheck --severity=warning app/hooks/*.sh && npm test
|
|
8757
8735
|
|
|
8758
|
-
#
|
|
8759
|
-
|
|
8736
|
+
# 5a. Supply-chain standard (v2.8.0+) — non-negotiable
|
|
8737
|
+
grep -q -- '--provenance' .github/workflows/publish.yml || { echo "MISSING --provenance"; exit 1; }
|
|
8738
|
+
grep -q 'id-token: write' .github/workflows/publish.yml || { echo "MISSING id-token: write"; exit 1; }
|
|
8739
|
+
python3 scripts/audit_skills.py --permissions # review Bash/Write/Edit footprint
|
|
8760
8740
|
|
|
8761
|
-
#
|
|
8762
|
-
|
|
8741
|
+
# 5b. Ecosystem gate — snapshot must be current before tag
|
|
8742
|
+
python3 scripts/ecosystem_doctor.py --offline --check || { echo "STALE ecosystem snapshot — re-run doctor"; exit 1; }
|
|
8763
8743
|
|
|
8764
|
-
#
|
|
8765
|
-
|
|
8766
|
-
# gitignored — do NOT commit them (regenerated at prepublishOnly). Only the
|
|
8767
|
-
# doc artifacts llms.txt / llms-full.txt are committed.
|
|
8768
|
-
git add llms.txt llms-full.txt
|
|
8769
|
-
git add -p # stage your other changes
|
|
8770
|
-
git commit -m "feat: your change description"
|
|
8771
|
-
```
|
|
8744
|
+
# 5c. Licensing gate — SPDX headers, LICENSE, NOTICE, manifest consistency
|
|
8745
|
+
npx bats tests/test_licensing.bats || { echo "LICENSING GATE FAILED"; exit 1; }
|
|
8772
8746
|
|
|
8773
|
-
|
|
8774
|
-
|
|
8775
|
-
|
|
8747
|
+
# 6. Commit, push branch, and wait for the exact commit's full CI
|
|
8748
|
+
git add -A && git commit -m "chore: release vX.Y.Z"
|
|
8749
|
+
RELEASE_SHA=$(git rev-parse HEAD)
|
|
8750
|
+
git push origin main || { echo "FAIL: release commit push failed"; exit 1; }
|
|
8751
|
+
RUN_ID=""
|
|
8752
|
+
for ATTEMPT in 1 2 3 4 5; do
|
|
8753
|
+
RUN_ID=$(gh run list --workflow ci.yml --event push --commit "$RELEASE_SHA" \
|
|
8754
|
+
--limit 1 --json databaseId --jq '.[0].databaseId')
|
|
8755
|
+
[ -n "$RUN_ID" ] && break
|
|
8756
|
+
echo "Waiting for CI run registration ($ATTEMPT/5)" >&2
|
|
8757
|
+
[ "$ATTEMPT" -eq 5 ] || sleep 60
|
|
8758
|
+
done
|
|
8759
|
+
test -n "$RUN_ID" || { echo "FAIL: CI run not found for release commit"; exit 1; }
|
|
8760
|
+
gh run watch "$RUN_ID" --exit-status \
|
|
8761
|
+
|| { echo "FAIL: release commit CI failed"; exit 1; }
|
|
8762
|
+
for OS in ubuntu-latest macos-latest; do
|
|
8763
|
+
gh run view "$RUN_ID" --json jobs \
|
|
8764
|
+
--jq ".jobs[] | select(.name == \"Bats test suite ($OS)\") | .conclusion" \
|
|
8765
|
+
| grep -qx success || { echo "FAIL: $OS test job not green"; exit 1; }
|
|
8766
|
+
done
|
|
8767
|
+
test "$(git rev-parse HEAD)" = "$RELEASE_SHA" || { echo "FAIL: local HEAD changed during CI"; exit 1; }
|
|
8768
|
+
git fetch origin main || { echo "FAIL: origin/main refresh failed"; exit 1; }
|
|
8769
|
+
test "$(git rev-parse origin/main)" = "$RELEASE_SHA" || { echo "FAIL: origin/main changed during CI"; exit 1; }
|
|
8776
8770
|
|
|
8777
|
-
|
|
8778
|
-
|
|
8779
|
-
|
|
8771
|
+
# 6a. Only green branch CI authorizes tag creation
|
|
8772
|
+
git tag vX.Y.Z "$RELEASE_SHA"
|
|
8773
|
+
|
|
8774
|
+
# 6b. Assert the tag before pushing it (v4.19.0 postmortem, Phase 7)
|
|
8775
|
+
test "$(git rev-parse vX.Y.Z)" = "$RELEASE_SHA" || { echo "FAIL: tag not on tested release commit"; exit 1; }
|
|
8776
|
+
git show --no-patch --format=%s vX.Y.Z | grep -qx "chore: release vX.Y.Z" || { echo "FAIL: tag not on release commit"; exit 1; }
|
|
8777
|
+
|
|
8778
|
+
# 6c. Push the single tag by full ref. Never --tags.
|
|
8779
|
+
git push origin refs/tags/vX.Y.Z
|
|
8780
8780
|
```
|
|
8781
8781
|
|
|
8782
8782
|
---
|
|
8783
8783
|
|
|
8784
|
-
## Phase
|
|
8784
|
+
## Phase 0: Ecosystem Sync (MANDATORY for minor/major)
|
|
8785
8785
|
|
|
8786
|
-
|
|
8787
|
-
ai-toolkit --version
|
|
8788
|
-
ai-toolkit --help
|
|
8789
|
-
which ai-toolkit
|
|
8790
|
-
```
|
|
8786
|
+
Before touching version numbers, confirm the toolkit is aligned with the current state of every editor / platform it integrates with. Skipping this phase ships a release whose generators may lag a month-old CLI refactor, a rename of `.cursorrules` to `.cursor/rules/`, or a new hook event we do not yet emit.
|
|
8791
8787
|
|
|
8792
|
-
**
|
|
8793
|
-
-
|
|
8794
|
-
-
|
|
8795
|
-
-
|
|
8788
|
+
**When this phase is mandatory:**
|
|
8789
|
+
- Minor release (X.Y.0) — always
|
|
8790
|
+
- Major release (X.0.0) — always
|
|
8791
|
+
- Patch release (X.Y.Z) — only if the patch touches a generator or install flow
|
|
8796
8792
|
|
|
8797
|
-
|
|
8793
|
+
**When to skip:** pure doc-only patches, SOP edits, internal refactors that do not touch `scripts/generate_*` or `app/skills/*/SKILL.md`.
|
|
8798
8794
|
|
|
8799
|
-
|
|
8795
|
+
### 0.1 Run the doctor
|
|
8800
8796
|
|
|
8801
8797
|
```bash
|
|
8802
|
-
|
|
8803
|
-
ai-toolkit status
|
|
8798
|
+
python3 scripts/ecosystem_doctor.py --format text | tee /tmp/eco-report.txt
|
|
8804
8799
|
```
|
|
8805
8800
|
|
|
8806
|
-
**
|
|
8807
|
-
- [ ] Agents: 44
|
|
8808
|
-
- [ ] Skills: 108
|
|
8809
|
-
- [ ] Hooks merged into settings.json
|
|
8810
|
-
- [ ] "Other AI Tools" lists documented global targets (with `--editors`): aider, antigravity, augment, cline, codex, copilot, cursor, gemini, opencode, roo, windsurf. Scope varies: Codex uses `$CODEX_HOME` (default `~/.codex`) plus `$HOME/.agents/skills`; Copilot uses `$COPILOT_HOME` (default `~/.copilot`); cursor has only `~/.cursor/hooks.json`; antigravity has the `~/.gemini/*/skills` pointer. Cursor and Antigravity rules remain project-only.
|
|
8801
|
+
Output classifies every registered tool as **Clean**, **Drift**, or **Errored**.
|
|
8811
8802
|
|
|
8812
|
-
|
|
8813
|
-
- [ ] Version matches expected
|
|
8814
|
-
- [ ] Profile: minimal/standard/strict
|
|
8815
|
-
- [ ] Modules: list of installed modules
|
|
8816
|
-
- [ ] Latest: up to date / update available
|
|
8803
|
+
### 0.2 Act on drift
|
|
8817
8804
|
|
|
8818
|
-
|
|
8805
|
+
For each drifting tool, follow [sop-ecosystem-sync.md](sop-ecosystem-sync.md) Phase 2-4:
|
|
8819
8806
|
|
|
8820
|
-
|
|
8807
|
+
| Drift class | Release impact |
|
|
8808
|
+
|-------------|----------------|
|
|
8809
|
+
| A (cosmetic reword) | No version impact — refresh snapshot, continue |
|
|
8810
|
+
| B (new feature — integrate) | **Minor** version bump at minimum; new generator or extended generator |
|
|
8811
|
+
| C (new feature — not adopted) | No impact — note in registry |
|
|
8812
|
+
| D (deprecation) | **Minor** or **major** depending on user impact; add migration warning |
|
|
8813
|
+
| E (feature promoted to default) | **Minor**; simplify generator, keep fallback comment |
|
|
8814
|
+
| F (feature newly globally available) | **Minor**; may require new generator or new config path |
|
|
8815
|
+
|
|
8816
|
+
If any B/D/E/F changes land in this preparation pass, mention them explicitly in the CHANGELOG entry (Phase 3) under a `Ecosystem` subsection.
|
|
8817
|
+
|
|
8818
|
+
### 0.3 Refresh snapshot
|
|
8819
|
+
|
|
8820
|
+
Once every drift is resolved (either by code change or by re-classifying as acceptable):
|
|
8821
8821
|
|
|
8822
8822
|
```bash
|
|
8823
|
-
|
|
8823
|
+
python3 scripts/ecosystem_doctor.py --update
|
|
8824
8824
|
```
|
|
8825
8825
|
|
|
8826
|
-
|
|
8827
|
-
- Environment: node, bash, python3, bats
|
|
8828
|
-
- Global Install: .claude exists, agents/skills symlinks (0 broken), settings.json hooks
|
|
8829
|
-
- Hook Scripts: all present and executable
|
|
8830
|
-
- Hook Configuration: 14 events registered
|
|
8831
|
-
- Generated Artifacts: AGENTS.md, llms.txt, llms-full.txt
|
|
8832
|
-
- Planned Assets: plugin.json, benchmarks, plugin packs
|
|
8833
|
-
- Benchmark Freshness: < 30 days
|
|
8834
|
-
- Stale Rules: all healthy
|
|
8826
|
+
This writes the new baseline to `benchmarks/ecosystem-doctor-snapshot.json`. Commit it as part of the release commit.
|
|
8835
8827
|
|
|
8836
|
-
|
|
8837
|
-
- [ ] `Errors: 0 | Warnings: 0`
|
|
8838
|
-
- [ ] `HEALTH CHECK PASSED`
|
|
8828
|
+
### 0.4 Gate
|
|
8839
8829
|
|
|
8840
|
-
|
|
8841
|
-
|
|
8830
|
+
```bash
|
|
8831
|
+
python3 scripts/ecosystem_doctor.py --offline --check
|
|
8832
|
+
```
|
|
8833
|
+
|
|
8834
|
+
Must exit `0`. If it exits `1`, the snapshot is stale — rerun Phase 0.3 or review the remaining drift.
|
|
8842
8835
|
|
|
8843
8836
|
---
|
|
8844
8837
|
|
|
8845
|
-
## Phase
|
|
8838
|
+
## Phase 1: Determine Version Bump
|
|
8846
8839
|
|
|
8847
|
-
|
|
8848
|
-
mkdir -p /tmp/ai-toolkit-verify && cd /tmp/ai-toolkit-verify
|
|
8849
|
-
git init -q
|
|
8850
|
-
ai-toolkit install --local --editors all --dry-run
|
|
8851
|
-
cd - && rm -rf /tmp/ai-toolkit-verify
|
|
8852
|
-
```
|
|
8840
|
+
Follow [Semantic Versioning](https://semver.org/):
|
|
8853
8841
|
|
|
8854
|
-
|
|
8855
|
-
|
|
8856
|
-
-
|
|
8857
|
-
-
|
|
8858
|
-
|
|
8859
|
-
- [ ] Would generate configs for each editor (legacy + directory-based)
|
|
8860
|
-
- [ ] Would install: .git/hooks/pre-commit
|
|
8861
|
-
- [ ] Would inject language rules (auto-detected)
|
|
8842
|
+
| Change Type | Bump | Examples |
|
|
8843
|
+
|-------------|------|---------|
|
|
8844
|
+
| Bugfix, typo, doc-only | **patch** | Fix install flag, correct description |
|
|
8845
|
+
| New feature, skill, agent, flag | **minor** | Add `/hipaa-validate`, add `--output json` |
|
|
8846
|
+
| Breaking CLI change, removed skill, config format change | **major** | Rename `install` to `setup`, remove skill |
|
|
8862
8847
|
|
|
8863
|
-
**
|
|
8864
|
-
```bash
|
|
8865
|
-
ai-toolkit install --local --dry-run
|
|
8866
|
-
# → Editors: none (empty project has no existing configs)
|
|
8867
|
-
```
|
|
8848
|
+
**Rule:** When in doubt, bump minor.
|
|
8868
8849
|
|
|
8869
8850
|
---
|
|
8870
8851
|
|
|
8871
|
-
## Phase
|
|
8852
|
+
## Phase 2: Sync Version in All Files
|
|
8872
8853
|
|
|
8873
|
-
|
|
8874
|
-
python3 scripts/validate.py --strict
|
|
8875
|
-
python3 scripts/audit_skills.py --ci
|
|
8876
|
-
```
|
|
8854
|
+
The canonical version lives in `package.json`. These files **must** match:
|
|
8877
8855
|
|
|
8878
|
-
|
|
8879
|
-
- [ ] Agents: 44, Skills: 108, Tests: exactly the current README badge count
|
|
8880
|
-
- [ ] Hook events: 14, Hook scripts: >= 30
|
|
8881
|
-
- [ ] Plugin packs >= 10, KB documents >= 20
|
|
8882
|
-
- [ ] `Errors: 0 | Warnings: 0` → `VALIDATION PASSED`
|
|
8856
|
+
### Mandatory sync (every release)
|
|
8883
8857
|
|
|
8884
|
-
|
|
8885
|
-
|
|
8886
|
-
|
|
8887
|
-
|
|
8858
|
+
| File | Field | How to update |
|
|
8859
|
+
|------|-------|---------------|
|
|
8860
|
+
| `package.json` | `"version": "X.Y.Z"` | Edit directly |
|
|
8861
|
+
| `manifest.json` | `"version": "X.Y.Z"` | Edit directly |
|
|
8862
|
+
| `app/.claude-plugin/plugin.json` | `"version": "X.Y.Z"` | Edit directly |
|
|
8888
8863
|
|
|
8889
|
-
|
|
8864
|
+
### Auto-synced (no manual action)
|
|
8890
8865
|
|
|
8891
|
-
|
|
8866
|
+
| File | Mechanism |
|
|
8867
|
+
|------|-----------|
|
|
8868
|
+
| `package-lock.json` | Regenerated by `npm install --package-lock-only` |
|
|
8869
|
+
|
|
8870
|
+
### Conditional sync (only if the doc was modified in this release)
|
|
8871
|
+
|
|
8872
|
+
| File | Field | When to update |
|
|
8873
|
+
|------|-------|---------------|
|
|
8874
|
+
| `kb/procedures/sop-maintenance.md` | frontmatter `version:` | If SOP content changed |
|
|
8875
|
+
| `kb/reference/skills-catalog.md` | frontmatter `version:` | If skills added/removed |
|
|
8876
|
+
| `kb/reference/agents-catalog.md` | frontmatter `version:` | If agents added/removed |
|
|
8877
|
+
| `kb/reference/hooks-catalog.md` | frontmatter `version:` | If hooks changed |
|
|
8878
|
+
| `kb/reference/architecture-overview.md` | frontmatter `version:` | If architecture changed |
|
|
8879
|
+
| `kb/reference/distribution-model.md` | frontmatter `version:` | If install model changed |
|
|
8880
|
+
| `kb/reference/global-install-model.md` | frontmatter `version:` | If install model changed |
|
|
8881
|
+
|
|
8882
|
+
> **Note:** KB `version:` fields track the **document version**, not the toolkit version.
|
|
8883
|
+
> Only bump them when the document content actually changes in this release.
|
|
8884
|
+
|
|
8885
|
+
### Count sync (if skills/agents/hooks changed)
|
|
8886
|
+
|
|
8887
|
+
| File | What to check |
|
|
8888
|
+
|------|---------------|
|
|
8889
|
+
| `package.json` | `"description"` — skill/agent count |
|
|
8890
|
+
| `README.md` | Badge counts, "What You Get" table |
|
|
8891
|
+
| `app/ARCHITECTURE.md` | Section headings with counts |
|
|
8892
|
+
|
|
8893
|
+
> **Tip:** `validate.py --strict` catches count drift AND version mismatches
|
|
8894
|
+
> (package.json vs manifest.json vs plugin.json) automatically.
|
|
8895
|
+
> If validation passes, counts and versions are correct.
|
|
8896
|
+
|
|
8897
|
+
### Verification command
|
|
8898
|
+
|
|
8899
|
+
After syncing, verify all mandatory files match:
|
|
8892
8900
|
|
|
8893
8901
|
```bash
|
|
8894
|
-
|
|
8895
|
-
|
|
8896
|
-
|
|
8897
|
-
|
|
8898
|
-
|
|
8899
|
-
tail -3 /tmp/npm-test.log
|
|
8900
|
-
echo "ok: $(grep -c '^ok ' /tmp/npm-test.log)"
|
|
8901
|
-
echo "not ok: $(grep -c '^not ok' /tmp/npm-test.log)"
|
|
8902
|
-
echo "exit: $exit"
|
|
8902
|
+
VERSION=$(python3 -c "import json; print(json.load(open('package.json'))['version'])")
|
|
8903
|
+
echo "Target: $VERSION"
|
|
8904
|
+
echo "manifest.json: $(python3 -c "import json; print(json.load(open('manifest.json'))['version'])")"
|
|
8905
|
+
echo "plugin.json: $(python3 -c "import json; print(json.load(open('app/.claude-plugin/plugin.json'))['version'])")"
|
|
8906
|
+
echo "package-lock.json: $(python3 -c "import json; print(json.load(open('package-lock.json'))['version'])")"
|
|
8903
8907
|
```
|
|
8904
8908
|
|
|
8905
|
-
|
|
8906
|
-
|
|
8907
|
-
|
|
8908
|
-
- [ ] `not ok == 0`
|
|
8909
|
-
- [ ] Bats runs tests in parallel (4 jobs)
|
|
8910
|
-
- [ ] Groups: agents, autodetect, cli, generators, guards, hooks, inject,
|
|
8911
|
-
install, kb, mcp, readme, profiles, uninstall, validate
|
|
8909
|
+
All four must print the same version. If not, fix before proceeding.
|
|
8910
|
+
|
|
8911
|
+
### Public surface review
|
|
8912
8912
|
|
|
8913
|
-
**Anti-pattern — do NOT do this:**
|
|
8914
8913
|
```bash
|
|
8915
|
-
|
|
8916
|
-
|
|
8917
|
-
npm test 2>&1 | grep -c '^ok '
|
|
8918
|
-
npm test 2>&1 | grep -c '^not ok'
|
|
8914
|
+
python3 scripts/surface_manifest.py --update
|
|
8915
|
+
git diff app/surface.json
|
|
8919
8916
|
```
|
|
8920
8917
|
|
|
8921
|
-
**
|
|
8922
|
-
|
|
8923
|
-
|
|
8924
|
-
|
|
8925
|
-
- Uninstall: removes toolkit, preserves user content
|
|
8918
|
+
**Every line the diff removes is a breaking change.** Restore it, or take the
|
|
8919
|
+
deprecation path in `BACKWARD_COMPATIBILITY.md` and add a `DECISIONS.md` entry
|
|
8920
|
+
before the tag. Lines added are new surface being adopted into protection — that
|
|
8921
|
+
needs no ceremony.
|
|
8926
8922
|
|
|
8927
|
-
|
|
8923
|
+
Do not run `--update` to make a red build green. The check fails because something
|
|
8924
|
+
users depend on disappeared; regenerating the manifest deletes the evidence, not
|
|
8925
|
+
the problem.
|
|
8928
8926
|
|
|
8929
|
-
|
|
8927
|
+
### Skill body budget ratchet
|
|
8930
8928
|
|
|
8931
|
-
|
|
8932
|
-
|
|
8933
|
-
|
|
8934
|
-
|
|
8935
|
-
cd - && rm -rf /tmp/ai-toolkit-eject-test
|
|
8929
|
+
`validate.py` prints the largest skill body on every run:
|
|
8930
|
+
|
|
8931
|
+
```
|
|
8932
|
+
Body budget: largest is <skill> at <N> bytes (warn 18000, error 20000)
|
|
8936
8933
|
```
|
|
8937
8934
|
|
|
8938
|
-
|
|
8939
|
-
|
|
8940
|
-
|
|
8941
|
-
- [ ] Rules inlined into CLAUDE.md
|
|
8942
|
-
- [ ] constitution.md and ARCHITECTURE.md copied
|
|
8943
|
-
- [ ] `output-styles/` directory present (v2.7.1+)
|
|
8935
|
+
Once that number sits at least 2000 bytes under `SKILL_BODY_BUDGET_WARN`, lower
|
|
8936
|
+
`SKILL_BODY_BUDGET_WARN` by 2000 in `scripts/validate.py` and ship the tightened
|
|
8937
|
+
threshold with the release. Target floor is 12000.
|
|
8944
8938
|
|
|
8945
|
-
|
|
8939
|
+
Two rules, both learned the hard way:
|
|
8946
8940
|
|
|
8947
|
-
|
|
8941
|
+
- **Never lower a threshold in the same change that something violates it.** Split
|
|
8942
|
+
the offending skill into `SKILL.md` + `reference/` first, prove it with
|
|
8943
|
+
`python3 scripts/check_split.py <skill> --before <pre-split SKILL.md>`, then
|
|
8944
|
+
tighten in a follow-up.
|
|
8945
|
+
- **Never raise a threshold to make a red build green.** A body over budget means
|
|
8946
|
+
detail that belongs in `reference/` is loading on every trigger match, including
|
|
8947
|
+
the accidental ones. Raising the number hides the cost, it does not remove it.
|
|
8948
8948
|
|
|
8949
|
-
|
|
8949
|
+
If the largest body has not moved since the previous release, leave the threshold
|
|
8950
|
+
alone and say so in the release notes. A ratchet that never advances is honest;
|
|
8951
|
+
one that advances past reality is not.
|
|
8950
8952
|
|
|
8951
|
-
|
|
8953
|
+
---
|
|
8952
8954
|
|
|
8953
|
-
|
|
8954
|
-
VERSION="X.Y.Z" # the tag just published
|
|
8955
|
-
npm view "@softspark/ai-toolkit@${VERSION}" --json \
|
|
8956
|
-
| python3 -c "import json, sys; d=json.load(sys.stdin); att=d['dist'].get('attestations', {}); assert att.get('provenance', {}).get('predicateType') == 'https://slsa.dev/provenance/v1', f'NO PROVENANCE for {d[\"version\"]}'; print(f'PROVENANCE OK: {att[\"url\"]}')"
|
|
8957
|
-
```
|
|
8955
|
+
## Phase 3: Write CHANGELOG Entry
|
|
8958
8956
|
|
|
8959
|
-
|
|
8960
|
-
- [ ] Exit 0 and prints `PROVENANCE OK: https://registry.npmjs.org/...`
|
|
8961
|
-
- [ ] `https://www.npmjs.com/package/@softspark/ai-toolkit/v/${VERSION}` shows the green "Provenance" badge
|
|
8957
|
+
Add entry at the top of `CHANGELOG.md` (after the header, before previous release):
|
|
8962
8958
|
|
|
8963
|
-
|
|
8959
|
+
```markdown
|
|
8960
|
+
## vX.Y.Z — Short Title (YYYY-MM-DD)
|
|
8964
8961
|
|
|
8965
|
-
###
|
|
8962
|
+
### Added
|
|
8963
|
+
- **Feature name** — description
|
|
8966
8964
|
|
|
8967
|
-
|
|
8968
|
-
|
|
8969
|
-
|
|
8965
|
+
### Changed
|
|
8966
|
+
- **What changed** — old behavior → new behavior
|
|
8967
|
+
|
|
8968
|
+
### Fixed
|
|
8969
|
+
- **Bug description** — what was broken and how it's fixed
|
|
8970
|
+
|
|
8971
|
+
### Removed
|
|
8972
|
+
- **What was removed** — migration path if any
|
|
8970
8973
|
```
|
|
8971
8974
|
|
|
8972
|
-
**
|
|
8973
|
-
-
|
|
8974
|
-
-
|
|
8975
|
+
**Rules:**
|
|
8976
|
+
- Use **bold** for feature names
|
|
8977
|
+
- Start descriptions with a verb (Added, Changed, Fixed, Removed)
|
|
8978
|
+
- Reference skill names with backticks and slash: `/hipaa-validate`
|
|
8979
|
+
- Include script names: `scripts/hipaa_scan.py`
|
|
8980
|
+
- Include count changes: `Skill count: 91 → 92`
|
|
8981
|
+
- Date format: `YYYY-MM-DD`
|
|
8982
|
+
- Title: short, descriptive, no version number repetition
|
|
8975
8983
|
|
|
8976
|
-
###
|
|
8984
|
+
### Update README "What's New" section
|
|
8977
8985
|
|
|
8978
|
-
|
|
8979
|
-
python3 scripts/audit_skills.py --permissions | head -40
|
|
8980
|
-
```
|
|
8986
|
+
**MANDATORY on every release.** Update the `## What's New in vX.Y.Z` section in `README.md`:
|
|
8981
8987
|
|
|
8982
|
-
|
|
8983
|
-
|
|
8984
|
-
|
|
8985
|
-
|
|
8988
|
+
1. Change the heading version: `## What's New in vX.Y.Z`
|
|
8989
|
+
2. Replace bullet points with 3-5 highlights from this release
|
|
8990
|
+
3. **Keep only the latest version block.** Delete the previous `## What's New in vA.B.C` section(s). README is the shop window, not the archive — users see the current release, full history lives in `CHANGELOG.md`.
|
|
8991
|
+
4. Keep the `See [CHANGELOG.md](CHANGELOG.md) for full history.` link directly below the bullet list.
|
|
8986
8992
|
|
|
8987
|
-
|
|
8993
|
+
> **Warning:** This section is the first thing users see after the badges.
|
|
8994
|
+
> A stale version here (e.g., "What's New in v2.1.3" when shipping v2.3.0)
|
|
8995
|
+
> signals an unmaintained project. Do NOT skip this step.
|
|
8988
8996
|
|
|
8989
|
-
|
|
8997
|
+
> **Single-version rule:** README.md must contain **exactly one** `## What's New in vX.Y.Z` heading at any time. If you find multiple stacked (e.g. v2.6.1 + v2.6.0 + v2.5.0), that is a SOP drift — collapse to the latest on the next release commit.
|
|
8998
|
+
|
|
8999
|
+
---
|
|
9000
|
+
|
|
9001
|
+
## Phase 4: Regenerate Artifacts
|
|
9002
|
+
|
|
9003
|
+
Use the npm scripts, not the generators directly:
|
|
8990
9004
|
|
|
8991
9005
|
```bash
|
|
8992
|
-
|
|
8993
|
-
|
|
8994
|
-
jq '.hooks // {}' ~/.softspark/ai-toolkit/hooks/external/sources.json 2>/dev/null \
|
|
8995
|
-
| python3 -c "import json, sys; d=json.load(sys.stdin) or {}; bad=[n for n,v in d.items() if v.get('url') and not v.get('sha256')]; assert not bad, f'UNPINNED: {bad}'; url_n=sum(1 for v in d.values() if v.get('url')); print(f'HOOK PIN OK: {url_n} URL hooks, all with sha256')"
|
|
9006
|
+
npm run generate:agents # AI_TOOLKIT_NO_CUSTOM_RULES=1 python3 scripts/generate_agents_md.py > AGENTS.md
|
|
9007
|
+
npm run generate:llms # llms.txt + llms-full.txt
|
|
8996
9008
|
```
|
|
8997
9009
|
|
|
8998
|
-
|
|
8999
|
-
|
|
9000
|
-
-
|
|
9010
|
+
`generate:agents` sets `AI_TOOLKIT_NO_CUSTOM_RULES=1`. Running
|
|
9011
|
+
`generate_agents_md.py` bare picks up whatever is registered in the maintainer's
|
|
9012
|
+
own `~/.softspark/ai-toolkit/rules/`, which then ships inside `AGENTS.md`.
|
|
9001
9013
|
|
|
9002
|
-
|
|
9014
|
+
Check if anything actually changed:
|
|
9003
9015
|
|
|
9004
9016
|
```bash
|
|
9005
|
-
|
|
9017
|
+
git diff --stat AGENTS.md llms.txt llms-full.txt
|
|
9006
9018
|
```
|
|
9007
9019
|
|
|
9008
|
-
|
|
9009
|
-
- [ ] Exit 0, no `CHECKSUM CHANGED` line
|
|
9010
|
-
- [ ] If a checksum change was intentional (e.g. upstream rule update), document it in the CHANGELOG entry before tagging
|
|
9020
|
+
If no diff, the artifacts are already current. If there is a diff, stage them.
|
|
9011
9021
|
|
|
9012
9022
|
---
|
|
9013
9023
|
|
|
9014
|
-
## Phase
|
|
9024
|
+
## Phase 5: Validate, Audit, Test
|
|
9015
9025
|
|
|
9016
|
-
|
|
9026
|
+
Run the full quality gate:
|
|
9017
9027
|
|
|
9018
|
-
|
|
9028
|
+
```bash
|
|
9029
|
+
python3 scripts/validate.py --strict
|
|
9030
|
+
python3 scripts/audit_skills.py --ci
|
|
9031
|
+
python3 scripts/audit_skills.py --sarif > audit.sarif # MANDATORY — GHAS ingest
|
|
9032
|
+
python3 scripts/audit_skills.py --permissions # review Bash/Write/Edit footprint
|
|
9019
9033
|
|
|
9020
|
-
|
|
9034
|
+
# ShellCheck on hooks (added in 1.11.0). Mirrors the ci.yml "ShellCheck hooks"
|
|
9035
|
+
# job. NOT run by validate.py, npm test, OR publish.yml — so a hook with a
|
|
9036
|
+
# ShellCheck warning passes every other gate AND still publishes on tag while
|
|
9037
|
+
# turning main CI red. Run it here, before tagging.
|
|
9038
|
+
shellcheck --severity=warning app/hooks/*.sh && echo "OK: shellcheck clean"
|
|
9021
9039
|
|
|
9022
|
-
|
|
9023
|
-
|
|
9024
|
-
|
|
9025
|
-
|
|
9026
|
-
| grep -
|
|
9040
|
+
# Registry / generator drift (added in 1.10.0). Meta-generators excluded.
|
|
9041
|
+
META="generate_agents_md.py|generate_llms_txt.py|generate_language_rules_skills.py"
|
|
9042
|
+
diff \
|
|
9043
|
+
<(grep -oE 'scripts/generate_[a-z_]+\.py' kb/reference/supported-tools-registry.md | sort -u) \
|
|
9044
|
+
<(ls scripts/generate_*.py | grep -vE "$META" | sort -u) \
|
|
9045
|
+
&& echo "OK: registry matches filesystem" \
|
|
9046
|
+
|| { echo "DRIFT: update supported-tools-registry.md before tagging"; exit 1; }
|
|
9047
|
+
|
|
9048
|
+
# Stage first IF this release adds or deletes a kb/ file. The test
|
|
9049
|
+
# "npm package KB files match the tracked release set" compares `git ls-files
|
|
9050
|
+
# kb` (the index) against what `npm pack` sees (the working tree), so an
|
|
9051
|
+
# unstaged addition reads as "extra" and an unstaged deletion as "missing".
|
|
9052
|
+
# Phase 6 stages, and it runs after this one, so the ordering fails the test
|
|
9053
|
+
# for any release that touches the KB. Staging early costs nothing.
|
|
9054
|
+
git status --porcelain kb/ | grep -qE '^(\?\?| D|\?M)' && git add -A kb/
|
|
9055
|
+
|
|
9056
|
+
# Run npm test ONCE, cache output, parse from file. The suite is 1400+ bats
|
|
9057
|
+
# cases — rerunning it per check wastes minutes. Do not pipe npm test into
|
|
9058
|
+
# tail/grep multiple times in the same session.
|
|
9059
|
+
npm test > /tmp/npm-test.log 2>&1
|
|
9060
|
+
tail -3 /tmp/npm-test.log
|
|
9061
|
+
echo "ok: $(grep -c '^ok ' /tmp/npm-test.log) | not ok: $(grep -c '^not ok' /tmp/npm-test.log)"
|
|
9027
9062
|
```
|
|
9028
9063
|
|
|
9029
|
-
**
|
|
9030
|
-
-
|
|
9031
|
-
-
|
|
9032
|
-
-
|
|
9033
|
-
-
|
|
9034
|
-
-
|
|
9035
|
-
-
|
|
9036
|
-
-
|
|
9064
|
+
**Expected results:**
|
|
9065
|
+
- `validate.py`: `Errors: 0 | Warnings: 0 | VALIDATION PASSED`
|
|
9066
|
+
- `audit_skills.py --ci`: `HIGH: 0 | WARN: 0` (INFO is acceptable)
|
|
9067
|
+
- `audit_skills.py --sarif`: valid JSON, non-empty `runs[0].tool.driver.rules`
|
|
9068
|
+
- `audit_skills.py --permissions`: review `Skills with Bash + Write + Edit` list — any newly-added skill with broad access MUST be justified in the CHANGELOG entry
|
|
9069
|
+
- `shellcheck --severity=warning app/hooks/*.sh`: no output, exit 0. A common false positive is `SC2034` on `INPUT` or env vars (e.g. `AI_TOOLKIT_HOOK_FORMAT`) that a *sourced* helper (`_hook-io.sh`) consumes — ShellCheck cannot see cross-file use. Fix with a `# shellcheck disable=SC2034` directive or `export`, matching `guard-destructive.sh`. Never tag with a red ShellCheck.
|
|
9070
|
+
- Registry drift: `OK: registry matches filesystem`. If `DRIFT:` appears, add the missing `scripts/generate_*.py` rows to `kb/reference/supported-tools-registry.md` before tagging.
|
|
9071
|
+
- `npm test`: `1..N` with zero `not ok` (read from the cached `/tmp/npm-test.log`, do not rerun)
|
|
9037
9072
|
|
|
9038
|
-
|
|
9073
|
+
> **Why this matters (v4.5.1 postmortem):** `publish.yml` runs only `validate.py` + `npm test`, so it published v4.5.0 even though the `main` CI `ShellCheck hooks` job was red on two `SC2034` warnings in a new hook. The publish workflow does **not** depend on the CI workflow. Until that is fixed, ShellCheck is a manual pre-tag gate — run it here every time.
|
|
9039
9074
|
|
|
9075
|
+
**One-liner:**
|
|
9040
9076
|
```bash
|
|
9041
|
-
|
|
9042
|
-
ai-toolkit install --local --editors codex --profile minimal --dry-run 2>&1 \
|
|
9043
|
-
| grep -q "Would generate: .agents/skills" && echo "OK: Codex skills emit at minimal"
|
|
9044
|
-
ai-toolkit install --local --editors codex --profile standard --codex-skills --dry-run 2>&1 \
|
|
9045
|
-
| grep -q "Would refresh: .agents/skills" && echo "OK: legacy flag explicitly refreshes the same catalog"
|
|
9077
|
+
python3 scripts/validate.py --strict && python3 scripts/audit_skills.py --ci && python3 scripts/audit_skills.py --sarif > audit.sarif && shellcheck --severity=warning app/hooks/*.sh && diff <(grep -oE 'scripts/generate_[a-z_]+\.py' kb/reference/supported-tools-registry.md | sort -u) <(ls scripts/generate_*.py | grep -vE 'generate_agents_md\.py|generate_llms_txt\.py|generate_language_rules_skills\.py' | sort -u) && npm test
|
|
9046
9078
|
```
|
|
9047
9079
|
|
|
9048
|
-
**
|
|
9049
|
-
-
|
|
9050
|
-
-
|
|
9051
|
-
-
|
|
9080
|
+
**If tests fail:** Fix the issue, do NOT skip. Common failures:
|
|
9081
|
+
- Stale counts → re-run `generate:all` or fix README/ARCHITECTURE
|
|
9082
|
+
- Missing frontmatter → add to new KB docs
|
|
9083
|
+
- Broken symlink → `ai-toolkit doctor --fix`
|
|
9052
9084
|
|
|
9053
|
-
###
|
|
9085
|
+
### Phase 5c: Licensing Gate (v4.20.0+)
|
|
9054
9086
|
|
|
9055
|
-
The
|
|
9056
|
-
|
|
9057
|
-
- Gemini hooks (`.gemini/settings.json`)
|
|
9087
|
+
The project is Apache-2.0. Attribution only works if the artefact actually
|
|
9088
|
+
carries it, and every part of that is mechanically checkable.
|
|
9058
9089
|
|
|
9059
9090
|
```bash
|
|
9060
|
-
|
|
9061
|
-
|
|
9062
|
-
|
|
9063
|
-
|
|
9064
|
-
|
|
9065
|
-
|
|
9066
|
-
|
|
9091
|
+
# The whole gate, enforced in CI. Run it here so a failure is caught before tagging.
|
|
9092
|
+
npx bats tests/test_licensing.bats
|
|
9093
|
+
```
|
|
9094
|
+
|
|
9095
|
+
The seven assertions, and why each exists:
|
|
9096
|
+
|
|
9097
|
+
| Check | Fails when |
|
|
9098
|
+
|---|---|
|
|
9099
|
+
| Every shipped source file carries an SPDX header | A new `.py`/`.sh`/`.js`/`.bats` file was added without one — the common case, and the reason this is a test rather than a habit |
|
|
9100
|
+
| Headers name Apache-2.0 and nothing else | A file was copied in from an MIT/GPL source with its own header intact |
|
|
9101
|
+
| **No** markdown file carries an SPDX header | Someone "helpfully" ran the header script over `app/skills/` — headers there sit above parsed frontmatter and bill every session for it |
|
|
9102
|
+
| `LICENSE` is the complete Apache 2.0 text | The file was truncated or replaced with a summary |
|
|
9103
|
+
| `NOTICE` carries attribution, the source URL, §4(d) and the MIT-era notice | The attribution mechanism was gutted |
|
|
9104
|
+
| `LICENSE` **and** `NOTICE` ship in the npm package | `package.json` `files` lost an entry — a NOTICE that never reaches the consumer cannot satisfy §4(d) |
|
|
9105
|
+
| Every manifest declaring a licence declares Apache-2.0 | `package.json`, `manifest.json`, `plugin.json` and `package-lock.json` drifted apart |
|
|
9106
|
+
|
|
9107
|
+
**Adding source files in this release?** The header goes *after* the shebang,
|
|
9108
|
+
never before it. Short SPDX form:
|
|
9109
|
+
|
|
9110
|
+
```
|
|
9111
|
+
# SPDX-License-Identifier: Apache-2.0
|
|
9112
|
+
# Copyright 2024-2026 Lukasz Krzemien (biuro@softspark.eu)
|
|
9113
|
+
# Source: https://github.com/softspark/ai-toolkit
|
|
9067
9114
|
```
|
|
9068
9115
|
|
|
9069
|
-
|
|
9116
|
+
`//` for JavaScript. Full convention and the reasoning behind the markdown
|
|
9117
|
+
exclusion: [Licensing](../reference/licensing.md).
|
|
9070
9118
|
|
|
9071
|
-
|
|
9119
|
+
**Changing the licence itself?** Do not hand-type the licence text. Take it
|
|
9120
|
+
verbatim from a published copy and cross-verify against a second independent
|
|
9121
|
+
copy before writing `LICENSE` — a rendered or summarised licence is not the
|
|
9122
|
+
licence. Prior releases stay under their original terms; a licence change
|
|
9123
|
+
applies going forward and revokes nothing already granted.
|
|
9124
|
+
|
|
9125
|
+
### Phase 5a: Supply-Chain Hardening Verification (v2.8.0+)
|
|
9126
|
+
|
|
9127
|
+
These checks enforce the security standard introduced in v2.8.0. Do NOT tag a release until all pass.
|
|
9128
|
+
|
|
9129
|
+
**1. Publish workflow emits provenance:**
|
|
9072
9130
|
|
|
9073
9131
|
```bash
|
|
9074
|
-
|
|
9075
|
-
# Sort file paths before hashing — find traversal order follows inode order,
|
|
9076
|
-
# which can shift between runs even when content is byte-identical, producing
|
|
9077
|
-
# false FAIL signals.
|
|
9078
|
-
ai-toolkit install --local --editors cursor,gemini --profile full >/dev/null 2>&1
|
|
9079
|
-
SHA1=$(find .cursor .gemini -type f -print0 | LC_ALL=C sort -z | xargs -0 shasum | shasum | awk '{print $1}')
|
|
9080
|
-
ai-toolkit install --local --editors cursor,gemini --profile full >/dev/null 2>&1
|
|
9081
|
-
SHA2=$(find .cursor .gemini -type f -print0 | LC_ALL=C sort -z | xargs -0 shasum | shasum | awk '{print $1}')
|
|
9082
|
-
[ "$SHA1" = "$SHA2" ] && echo "OK: idempotent" || echo "FAIL: install is not idempotent"
|
|
9132
|
+
grep -E '\-\-provenance|id-token: write' .github/workflows/publish.yml
|
|
9083
9133
|
```
|
|
9084
9134
|
|
|
9085
|
-
|
|
9086
|
-
|
|
9087
|
-
### 9.5 Live-install JSON outputs parse
|
|
9135
|
+
- [ ] Both markers present (`--provenance` flag + `id-token: write` permission)
|
|
9136
|
+
- [ ] Any PR that changes `publish.yml` REQUIRES an approved security review
|
|
9088
9137
|
|
|
9089
|
-
|
|
9138
|
+
**2. URL-sourced rules and hooks are checksum-pinned:**
|
|
9090
9139
|
|
|
9091
9140
|
```bash
|
|
9092
|
-
|
|
9093
|
-
|
|
9094
|
-
|
|
9095
|
-
|
|
9096
|
-
done
|
|
9141
|
+
# On a machine that has consumed URL rules/hooks at least once
|
|
9142
|
+
# (schema_version 1: entries live under the .rules / .hooks key):
|
|
9143
|
+
jq '.rules | to_entries | map(select(.value.url != null and (.value.sha256 // "" | length) == 0))' ~/.softspark/ai-toolkit/rules/sources.json
|
|
9144
|
+
jq '.hooks | to_entries | map(select(.value.url != null and (.value.sha256 // "" | length) == 0))' ~/.softspark/ai-toolkit/hooks/external/sources.json
|
|
9097
9145
|
```
|
|
9098
9146
|
|
|
9099
|
-
|
|
9100
|
-
|
|
9101
|
-
### 9.6 Registry / generator drift check
|
|
9147
|
+
- [ ] Both queries return empty arrays (every URL entry has a `sha256`)
|
|
9148
|
+
- [ ] If not, run `ai-toolkit update` to backfill missing hashes before tagging
|
|
9102
9149
|
|
|
9103
|
-
|
|
9150
|
+
**3. Audit SARIF output is well-formed:**
|
|
9104
9151
|
|
|
9105
9152
|
```bash
|
|
9106
|
-
|
|
9107
|
-
REG=$(grep -oE 'scripts/generate_[a-z_]+\.py' kb/reference/supported-tools-registry.md | sort -u)
|
|
9108
|
-
FS=$(ls scripts/generate_*.py | grep -vE "$META" | sort -u)
|
|
9109
|
-
diff <(echo "$REG") <(echo "$FS") && echo "OK: registry matches filesystem" || echo "DRIFT: update supported-tools-registry.md"
|
|
9153
|
+
python3 scripts/audit_skills.py --sarif | python3 -c "import json, sys; d=json.load(sys.stdin); assert d['version']=='2.1.0' and d['runs'][0]['tool']['driver']['name']; print('SARIF OK')"
|
|
9110
9154
|
```
|
|
9111
9155
|
|
|
9112
|
-
|
|
9156
|
+
- [ ] Prints `SARIF OK`
|
|
9157
|
+
- [ ] If the script ever grows new rule classes, extend the SARIF `rules[]` coverage before releasing
|
|
9113
9158
|
|
|
9114
|
-
|
|
9159
|
+
**4. Strict-pin mode passes on CI** (optional, recommended for stable branches):
|
|
9115
9160
|
|
|
9116
9161
|
```bash
|
|
9117
|
-
|
|
9118
|
-
D=/tmp/aitk-claude-app-${RANDOM}
|
|
9119
|
-
python3 scripts/claude_app.py export --output "$D.zip" --no-custom-rules
|
|
9120
|
-
python3 -c "import zipfile; z=zipfile.ZipFile('$D.zip'); assert '.claude-plugin/plugin.json' in z.namelist(); assert 'claude-app/skills/ai-toolkit-rules/SKILL.md' in z.namelist(); print('OK: Claude app archive')"
|
|
9162
|
+
AI_TOOLKIT_STRICT_PIN=1 ai-toolkit update --dry-run
|
|
9121
9163
|
```
|
|
9122
9164
|
|
|
9123
|
-
|
|
9124
|
-
|
|
9165
|
+
- [ ] Exit 0, no `CHECKSUM CHANGED` line
|
|
9166
|
+
- [ ] Any unexpected upstream change blocks the release until explicitly approved
|
|
9125
9167
|
|
|
9126
9168
|
---
|
|
9127
9169
|
|
|
9128
|
-
## Phase
|
|
9129
|
-
|
|
9130
|
-
Run this phase whenever the release changes the `dsh` target, package pins, preset lifecycle, or DSH compatibility documentation. Use a new task-specific `DSH_HOME`; never replace `HOME` or reuse a regular profile.
|
|
9170
|
+
## Phase 6: Commit
|
|
9131
9171
|
|
|
9132
|
-
|
|
9172
|
+
Stage all release files:
|
|
9133
9173
|
|
|
9134
9174
|
```bash
|
|
9135
|
-
|
|
9136
|
-
|
|
9137
|
-
|
|
9138
|
-
|
|
9139
|
-
|
|
9140
|
-
|
|
9175
|
+
git add package.json manifest.json app/.claude-plugin/plugin.json
|
|
9176
|
+
git add package-lock.json
|
|
9177
|
+
git add CHANGELOG.md
|
|
9178
|
+
git add llms.txt llms-full.txt
|
|
9179
|
+
# NOTE: AGENTS.md, GEMINI.md, and .github/copilot-instructions.md are generated
|
|
9180
|
+
# editor configs and are gitignored — do NOT commit them. `prepublishOnly` runs
|
|
9181
|
+
# `npm run generate:all`, so the shipped package (which lists AGENTS.md in
|
|
9182
|
+
# package.json `files`) gets a fresh copy at publish time.
|
|
9183
|
+
git add -p # review and stage any other changes
|
|
9141
9184
|
```
|
|
9142
9185
|
|
|
9143
|
-
|
|
9186
|
+
Commit:
|
|
9144
9187
|
|
|
9145
|
-
|
|
9146
|
-
|
|
9188
|
+
```bash
|
|
9189
|
+
git commit -m "chore: release vX.Y.Z"
|
|
9190
|
+
```
|
|
9147
9191
|
|
|
9148
|
-
|
|
9192
|
+
---
|
|
9193
|
+
|
|
9194
|
+
## Phase 7: Push Branch, Verify CI, Tag, and Push Tag
|
|
9149
9195
|
|
|
9150
9196
|
```bash
|
|
9151
|
-
|
|
9152
|
-
|
|
9197
|
+
RELEASE_SHA=$(git rev-parse HEAD)
|
|
9198
|
+
git push origin main || { echo "FAIL: release commit push failed"; exit 1; }
|
|
9153
9199
|
|
|
9154
|
-
|
|
9200
|
+
# Bind the gate to the exact release commit. GitHub run registration is
|
|
9201
|
+
# asynchronous, so retry at most five times and log every attempt.
|
|
9202
|
+
RUN_ID=""
|
|
9203
|
+
for ATTEMPT in 1 2 3 4 5; do
|
|
9204
|
+
RUN_ID=$(gh run list --workflow ci.yml --event push --commit "$RELEASE_SHA" \
|
|
9205
|
+
--limit 1 --json databaseId --jq '.[0].databaseId')
|
|
9206
|
+
[ -n "$RUN_ID" ] && break
|
|
9207
|
+
echo "Waiting for CI run registration ($ATTEMPT/5)" >&2
|
|
9208
|
+
[ "$ATTEMPT" -eq 5 ] || sleep 60
|
|
9209
|
+
done
|
|
9210
|
+
test -n "$RUN_ID" || { echo "FAIL: CI run not found for release commit"; exit 1; }
|
|
9211
|
+
gh run watch "$RUN_ID" --exit-status \
|
|
9212
|
+
|| { echo "FAIL: release commit CI failed"; exit 1; }
|
|
9155
9213
|
|
|
9156
|
-
|
|
9214
|
+
# publish.yml is Ubuntu-only. Require both full CI matrix jobs before creating
|
|
9215
|
+
# the tag so a green publish job cannot bypass a red macOS test.
|
|
9216
|
+
for OS in ubuntu-latest macos-latest; do
|
|
9217
|
+
gh run view "$RUN_ID" --json jobs \
|
|
9218
|
+
--jq ".jobs[] | select(.name == \"Bats test suite ($OS)\") | .conclusion" \
|
|
9219
|
+
| grep -qx success || { echo "FAIL: $OS test job not green"; exit 1; }
|
|
9220
|
+
done
|
|
9157
9221
|
|
|
9158
|
-
|
|
9222
|
+
# Refuse to tag if either local or remote main moved while CI was running.
|
|
9223
|
+
test "$(git rev-parse HEAD)" = "$RELEASE_SHA" \
|
|
9224
|
+
|| { echo "FAIL: local HEAD changed during CI"; exit 1; }
|
|
9225
|
+
git fetch origin main || { echo "FAIL: origin/main refresh failed"; exit 1; }
|
|
9226
|
+
test "$(git rev-parse origin/main)" = "$RELEASE_SHA" \
|
|
9227
|
+
|| { echo "FAIL: origin/main changed during CI"; exit 1; }
|
|
9159
9228
|
|
|
9160
|
-
|
|
9229
|
+
git tag vX.Y.Z "$RELEASE_SHA"
|
|
9161
9230
|
|
|
9162
|
-
|
|
9163
|
-
|
|
9164
|
-
|
|
9165
|
-
|
|
9231
|
+
# Assert the tag before pushing it. Both checks are one line each and both
|
|
9232
|
+
# have caught a real broken release.
|
|
9233
|
+
test "$(git rev-parse vX.Y.Z)" = "$RELEASE_SHA" \
|
|
9234
|
+
|| { echo "FAIL: tag is not on the tested release commit"; exit 1; }
|
|
9235
|
+
git show --no-patch --format=%s vX.Y.Z | grep -qx "chore: release vX.Y.Z" \
|
|
9236
|
+
|| { echo "FAIL: tag is not on the chore: release commit"; exit 1; }
|
|
9237
|
+
|
|
9238
|
+
# Push the single release tag by its full ref.
|
|
9239
|
+
git push origin refs/tags/vX.Y.Z
|
|
9166
9240
|
```
|
|
9167
9241
|
|
|
9168
|
-
|
|
9242
|
+
**Why branch CI comes before the tag (v4.30.2 postmortem).** The publish
|
|
9243
|
+
workflow runs only on Ubuntu and can publish while the separate macOS matrix job
|
|
9244
|
+
is red. `v4.30.2` exposed this with a macOS-only DSH timing failure. A release
|
|
9245
|
+
commit must therefore pass the complete Ubuntu/macOS branch workflow before its
|
|
9246
|
+
tag exists; a successful publish workflow is not a substitute for green CI.
|
|
9169
9247
|
|
|
9170
|
-
|
|
9171
|
-
|
|
9172
|
-
|
|
9173
|
-
|
|
9248
|
+
**Why the assertions (v4.19.0 postmortem).** v4.19.0 was tagged on a commit
|
|
9249
|
+
that contained only a KB document and still carried `package.json` version
|
|
9250
|
+
`4.18.0`; the actual release sat in the commit above it under a recycled
|
|
9251
|
+
`fix:` message. `publish.yml` fired, tried to publish a version already on
|
|
9252
|
+
npm, and failed. Nothing on npm, a tag pointing at the wrong tree, and the
|
|
9253
|
+
only way out was rewriting a pushed commit. Both assertions above catch this
|
|
9254
|
+
in under a second. Run them.
|
|
9174
9255
|
|
|
9175
|
-
|
|
9256
|
+
**Never `git push --tags`.** It pushes every local tag at once, and GitHub
|
|
9257
|
+
suppresses tag-triggered workflow runs when many tags arrive in a single push
|
|
9258
|
+
— the workflow silently does not fire and nothing publishes. Push the single
|
|
9259
|
+
release tag by its full ref, as above. (Sibling evidence: this is exactly how
|
|
9260
|
+
rag-mcp's `1.0.3` image build was skipped, when a `--tags` push carried 37
|
|
9261
|
+
tags at once.)
|
|
9176
9262
|
|
|
9177
|
-
|
|
9178
|
-
|
|
9179
|
-
|
|
9180
|
-
|
|
9263
|
+
This triggers `.github/workflows/publish.yml` which:
|
|
9264
|
+
1. Runs `validate.py --strict`
|
|
9265
|
+
2. Runs `npm test`
|
|
9266
|
+
3. Publishes to npm as `@softspark/ai-toolkit@X.Y.Z` with `--provenance` (SLSA v1 build attestation)
|
|
9181
9267
|
|
|
9182
|
-
|
|
9268
|
+
**Provenance is non-negotiable.** If `id-token: write` permission or the `--provenance` flag is missing from `publish.yml`, fix it BEFORE tagging — an unsigned release is a regression against the v2.8.0 standard.
|
|
9183
9269
|
|
|
9184
|
-
|
|
9185
|
-
|
|
9270
|
+
**After CI completes:** Run the [Release Verification SOP](sop-release-verification.md)
|
|
9271
|
+
to smoke-test the published package AND verify the provenance attestation landed on npm.
|
|
9186
9272
|
|
|
9187
|
-
|
|
9273
|
+
---
|
|
9274
|
+
|
|
9275
|
+
## Rollback
|
|
9276
|
+
|
|
9277
|
+
If a bad release was published:
|
|
9188
9278
|
|
|
9189
9279
|
```bash
|
|
9190
|
-
|
|
9191
|
-
ai-toolkit
|
|
9280
|
+
# Unpublish from npm (within 72h)
|
|
9281
|
+
npm unpublish @softspark/ai-toolkit@X.Y.Z
|
|
9282
|
+
|
|
9283
|
+
# Or deprecate (preferred — doesn't break existing installs)
|
|
9284
|
+
npm deprecate @softspark/ai-toolkit@X.Y.Z "Known issue: <description>. Use vA.B.C instead."
|
|
9285
|
+
|
|
9286
|
+
# Delete tag
|
|
9287
|
+
git tag -d vX.Y.Z
|
|
9288
|
+
git push origin --delete vX.Y.Z
|
|
9192
9289
|
```
|
|
9193
9290
|
|
|
9194
9291
|
---
|
|
9195
9292
|
|
|
9196
|
-
##
|
|
9293
|
+
## Checklist Summary
|
|
9197
9294
|
|
|
9198
|
-
|
|
|
9199
|
-
|
|
9200
|
-
|
|
|
9201
|
-
|
|
|
9202
|
-
|
|
|
9203
|
-
|
|
|
9204
|
-
|
|
|
9205
|
-
|
|
|
9206
|
-
|
|
|
9207
|
-
|
|
|
9208
|
-
|
|
|
9209
|
-
|
|
|
9210
|
-
|
|
|
9211
|
-
|
|
|
9212
|
-
|
|
|
9213
|
-
|
|
|
9214
|
-
|
|
|
9215
|
-
|
|
|
9216
|
-
|
|
|
9295
|
+
| # | Step | Command / Action | Pass Criteria |
|
|
9296
|
+
|---|------|-----------------|---------------|
|
|
9297
|
+
| 0a | Ecosystem drift check | `ecosystem_doctor.py --format text` | All tools Clean, or drift classified and resolved |
|
|
9298
|
+
| 0b | Ecosystem snapshot refresh | `ecosystem_doctor.py --update` | `benchmarks/ecosystem-doctor-snapshot.json` updated |
|
|
9299
|
+
| 0c | Ecosystem gate | `ecosystem_doctor.py --offline --check` | Exit 0 |
|
|
9300
|
+
| 1 | Version bump type | Decide patch/minor/major | — |
|
|
9301
|
+
| 2 | `package.json` version | Edit `"version"` | Matches target |
|
|
9302
|
+
| 3 | `manifest.json` version | Edit `"version"` | Matches target |
|
|
9303
|
+
| 4 | `plugin.json` version | Edit `"version"` | Matches target |
|
|
9304
|
+
| 5 | `package-lock.json` | `npm install --package-lock-only` | Matches target |
|
|
9305
|
+
| 6 | Count sync | Check `package.json` description, README | `validate.py` passes |
|
|
9306
|
+
| 7 | CHANGELOG.md | Add release entry (incl. `Ecosystem` subsection if any B/D/E/F drift) | Entry exists for vX.Y.Z |
|
|
9307
|
+
| 8 | Regenerate artifacts | `generate_agents_md.py`, `generate_llms_txt.py` | No unexpected diff |
|
|
9308
|
+
| 9 | Validate | `validate.py --strict` | 0 errors, 0 warnings |
|
|
9309
|
+
| 10 | Security audit (CI mode) | `audit_skills.py --ci` | 0 HIGH |
|
|
9310
|
+
| 11 | Security audit (SARIF) | `audit_skills.py --sarif` | Valid SARIF 2.1.0 JSON |
|
|
9311
|
+
| 12 | Per-skill permissions | `audit_skills.py --permissions` | New broad-access skills justified in CHANGELOG |
|
|
9312
|
+
| 13 | ShellCheck hooks | `shellcheck --severity=warning app/hooks/*.sh` | Exit 0, no output (mirrors ci.yml; publish.yml does NOT run it) |
|
|
9313
|
+
| 14 | Provenance flag check | `grep -- '--provenance' .github/workflows/publish.yml` | Present |
|
|
9314
|
+
| 15 | Checksum-pin backfill | `sources.json` entries all have `sha256` | No unpinned URL sources |
|
|
9315
|
+
| 15a | Licensing gate | `npx bats tests/test_licensing.bats` | 7/7 — SPDX headers, LICENSE, NOTICE, npm `files`, manifest consistency |
|
|
9316
|
+
| 16 | Tests | `git add -A kb/` if the KB changed, then `npm test` | All pass |
|
|
9317
|
+
| 17 | Commit | `git commit` | Clean working tree |
|
|
9318
|
+
| 18 | Push branch | `git push origin main` | Exact release commit is on `origin/main` |
|
|
9319
|
+
| 18a | Full branch CI | `gh run watch "$RUN_ID" --exit-status` plus matrix job assertions | Ubuntu and macOS Bats jobs both conclude `success` |
|
|
9320
|
+
| 19 | Tag | `git tag vX.Y.Z` | Tag exists only after green branch CI |
|
|
9321
|
+
| 19a | Tag is on tested SHA | `test "$(git rev-parse vX.Y.Z)" = "$RELEASE_SHA"` | Exit 0 |
|
|
9322
|
+
| 19b | Tag is on the release commit | `git show --no-patch --format=%s vX.Y.Z` | Reads `chore: release vX.Y.Z` |
|
|
9323
|
+
| 20 | Push the single tag | `git push origin refs/tags/vX.Y.Z` | Publish CI triggered with `id-token: write`. Never `--tags`. |
|
|
9217
9324
|
|
|
9218
9325
|
---
|
|
9219
9326
|
|
|
@@ -9801,7 +9908,7 @@ Three tiers determine how to approach a task:
|
|
|
9801
9908
|
|------|-------|-----------|-------|
|
|
9802
9909
|
| Task | `disable-model-invocation: true` | User via `/skill` only | 32 |
|
|
9803
9910
|
| Hybrid | (neither) | User via `/skill` + agent knowledge | 31 |
|
|
9804
|
-
| Knowledge | `user-invocable: false` | Claude auto-loads |
|
|
9911
|
+
| Knowledge | `user-invocable: false` | Claude auto-loads | 51 |
|
|
9805
9912
|
|
|
9806
9913
|
## Multi-Agent Execution
|
|
9807
9914
|
|
|
@@ -11738,7 +11845,7 @@ running the generators directly:
|
|
|
11738
11845
|
|
|
11739
11846
|
- `scripts/generate_copilot.py` / `generate_copilot_hooks.py` contract tests
|
|
11740
11847
|
(`tests/test_copilot.bats`, `tests/test_copilot_hooks.bats`).
|
|
11741
|
-
- Release layout check in `kb/procedures/release-verification
|
|
11848
|
+
- Release layout check in `kb/procedures/sop-release-verification.md` asserts the
|
|
11742
11849
|
`.github/{agents,skills,instructions,prompts,hooks}` surfaces.
|
|
11743
11850
|
- `validate.py --strict` + `audit_skills.py --ci` in CI.
|
|
11744
11851
|
|
|
@@ -14313,7 +14420,7 @@ MIT remnant would need each contributor's agreement.
|
|
|
14313
14420
|
|
|
14314
14421
|
It is a test rather than a checklist line on purpose. This project has two
|
|
14315
14422
|
same-day postmortems about SOPs that existed and were skipped; CI does not skip.
|
|
14316
|
-
[Release Preparation](../procedures/release
|
|
14423
|
+
[Release Preparation](../procedures/sop-release.md) Phase 5c runs the
|
|
14317
14424
|
same gate before tagging so a failure surfaces before the tag, not after.
|
|
14318
14425
|
|
|
14319
14426
|
## If the licence ever changes again
|
|
@@ -14326,7 +14433,7 @@ source, cross-verify against a second independent copy, and only then write
|
|
|
14326
14433
|
## Related
|
|
14327
14434
|
|
|
14328
14435
|
- [`LICENSE`](../../LICENSE), [`NOTICE`](../../NOTICE)
|
|
14329
|
-
- [Release Preparation SOP](../procedures/release
|
|
14436
|
+
- [Release Preparation SOP](../procedures/sop-release.md) — Phase 5c
|
|
14330
14437
|
- [Distribution Model](distribution-model.md) — what ships and where
|
|
14331
14438
|
|
|
14332
14439
|
---
|
|
@@ -16006,7 +16113,7 @@ Updated baseline docs:
|
|
|
16006
16113
|
- `kb/reference/skills-catalog.md`
|
|
16007
16114
|
- `kb/reference/plugin-pack-conventions.md`
|
|
16008
16115
|
- `kb/reference/claude-ecosystem-benchmark-snapshot.md`
|
|
16009
|
-
- `kb/procedures/maintenance
|
|
16116
|
+
- `kb/procedures/sop-maintenance.md`
|
|
16010
16117
|
|
|
16011
16118
|
## Validation Evidence
|
|
16012
16119
|
|
|
@@ -16860,13 +16967,13 @@ See [DSH Compatibility](./dsh-compatibility.md) for commands, topology, subscrip
|
|
|
16860
16967
|
2. Delete its section from this document.
|
|
16861
16968
|
3. Delete its snapshot entry from `benchmarks/ecosystem-doctor-snapshot.json` (or let the next `--update` prune it — currently not pruned automatically).
|
|
16862
16969
|
4. Decide whether to keep the generator (`scripts/generate_<tool>_*.py`) for backwards compatibility or delete it.
|
|
16863
|
-
5. Remove references from `README.md`, `manifest.json` `description` field, and `kb/procedures/maintenance
|
|
16970
|
+
5. Remove references from `README.md`, `manifest.json` `description` field, and `kb/procedures/sop-maintenance.md` `Supported editors` line.
|
|
16864
16971
|
|
|
16865
16972
|
---
|
|
16866
16973
|
|
|
16867
16974
|
## Related
|
|
16868
16975
|
|
|
16869
|
-
- [Ecosystem Sync SOP](../procedures/ecosystem-sync
|
|
16976
|
+
- [Ecosystem Sync SOP](../procedures/sop-ecosystem-sync.md) — how to use the doctor
|
|
16870
16977
|
- [MCP Editor Compatibility](./mcp-editor-compatibility.md) — MCP-specific subset
|
|
16871
16978
|
- `scripts/ecosystem_tools.json` — source of truth
|
|
16872
16979
|
- `scripts/ecosystem_doctor.py` — drift detector
|
|
@@ -17399,7 +17506,7 @@ Expect `Total plugin skills loaded: 0`, `Total plugin agents loaded: 0`, and the
|
|
|
17399
17506
|
## Related
|
|
17400
17507
|
|
|
17401
17508
|
- `kb/reference/global-install-model.md`
|
|
17402
|
-
- `kb/procedures/maintenance
|
|
17509
|
+
- `kb/procedures/sop-maintenance.md`
|
|
17403
17510
|
|
|
17404
17511
|
---
|
|
17405
17512
|
|