@softspark/ai-toolkit 2.8.0 → 2.10.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +31 -31
- package/CHANGELOG.md +29 -1
- package/README.md +4 -5
- package/app/.claude-plugin/plugin.json +1 -1
- package/app/agents/system-governor.md +98 -8
- package/app/constitution.md +7 -1
- package/app/rules/common/coding-style.md +12 -2
- package/app/skills/api-patterns/SKILL.md +1 -1
- package/app/skills/app-builder/SKILL.md +1 -1
- package/app/skills/architecture-decision/SKILL.md +1 -1
- package/app/skills/ci-cd-patterns/SKILL.md +1 -1
- package/app/skills/clean-code/SKILL.md +6 -1
- package/app/skills/csharp-patterns/SKILL.md +1 -1
- package/app/skills/database-patterns/SKILL.md +1 -1
- package/app/skills/debugging-tactics/SKILL.md +1 -1
- package/app/skills/design-engineering/SKILL.md +1 -1
- package/app/skills/docker-devops/SKILL.md +1 -1
- package/app/skills/documentation-standards/SKILL.md +1 -1
- package/app/skills/ecommerce-patterns/SKILL.md +1 -1
- package/app/skills/flutter-patterns/SKILL.md +1 -1
- package/app/skills/git-mastery/SKILL.md +1 -1
- package/app/skills/hive-mind/SKILL.md +1 -1
- package/app/skills/java-patterns/SKILL.md +1 -1
- package/app/skills/kotlin-patterns/SKILL.md +1 -1
- package/app/skills/mcp-patterns/SKILL.md +1 -1
- package/app/skills/migration-patterns/SKILL.md +1 -1
- package/app/skills/observability-patterns/SKILL.md +1 -1
- package/app/skills/performance-profiling/SKILL.md +1 -1
- package/app/skills/plan-writing/SKILL.md +1 -1
- package/app/skills/rag-patterns/SKILL.md +1 -1
- package/app/skills/refactor-plan/SKILL.md +1 -0
- package/app/skills/research-mastery/SKILL.md +1 -1
- package/app/skills/ruby-patterns/SKILL.md +1 -1
- package/app/skills/rust-patterns/SKILL.md +1 -1
- package/app/skills/security-patterns/SKILL.md +1 -1
- package/app/skills/swift-patterns/SKILL.md +1 -1
- package/app/skills/testing-patterns/SKILL.md +1 -1
- package/app/skills/typescript-patterns/SKILL.md +1 -1
- package/app/skills/verification-before-completion/SKILL.md +7 -0
- package/kb/procedures/release-preparation-sop.md +72 -15
- package/kb/procedures/release-verification-sop.md +82 -7
- package/llms-full.txt +184 -52
- package/manifest.json +1 -1
- package/package.json +1 -1
- package/scripts/audit_skills.py +68 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: java-patterns
|
|
3
|
-
description: "
|
|
3
|
+
description: "Java development patterns: Spring Boot, CompletableFuture, records, sealed types, streams, JPA/Hibernate, Maven/Gradle, virtual threads (Loom). Triggers: Java, Spring, Spring Boot, JPA, Hibernate, Maven, Gradle, CompletableFuture, record type, sealed class, virtual thread. Load when writing or reviewing Java code."
|
|
4
4
|
effort: medium
|
|
5
5
|
user-invocable: false
|
|
6
6
|
allowed-tools: Read
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: kotlin-patterns
|
|
3
|
-
description: "
|
|
3
|
+
description: "Kotlin development patterns: coroutines, Flow, sealed classes, data classes, extension functions, null safety, Ktor, Jetpack Compose, KMP. Triggers: Kotlin, coroutine, Flow, suspend, Ktor, Android Kotlin, Jetpack Compose, sealed class, data class, KMP, kotlinx. Load when writing or reviewing Kotlin code."
|
|
4
4
|
effort: medium
|
|
5
5
|
user-invocable: false
|
|
6
6
|
allowed-tools: Read
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: mcp-patterns
|
|
3
|
-
description: "
|
|
3
|
+
description: "MCP (Model Context Protocol) server design: tool schemas, resource patterns, transport selection (stdio/SSE), client configuration, error handling, capability negotiation. Triggers: MCP, Model Context Protocol, MCP server, MCP tool, MCP resource, JSON-RPC, stdio transport, SSE transport, Claude Desktop config, Cursor MCP. Load when building or integrating MCP servers."
|
|
4
4
|
effort: medium
|
|
5
5
|
user-invocable: false
|
|
6
6
|
allowed-tools: Read
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: migration-patterns
|
|
3
|
-
description: "
|
|
3
|
+
description: "Zero-downtime database migration patterns: expand-contract, double-write, backfill, blue-green schema changes, feature flags, rollback safety, online DDL. Triggers: migration, schema change, zero-downtime, expand-contract, double-write, backfill, ALTER TABLE, column rename, safe deploy, online DDL. Load when planning non-trivial DB schema changes."
|
|
4
4
|
effort: medium
|
|
5
5
|
user-invocable: false
|
|
6
6
|
allowed-tools: Read
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: observability-patterns
|
|
3
|
-
description: "
|
|
3
|
+
description: "Observability: structured logging, metrics (RED/USE/four golden signals), distributed tracing (OpenTelemetry), correlation IDs, log aggregation, SLO/SLI. Triggers: logging, log level, metrics, Prometheus, Grafana, OpenTelemetry, trace, span, structured log, observability, monitoring, SLO, SLI, alerting. Load when adding or reviewing logs, metrics, or traces."
|
|
4
4
|
effort: medium
|
|
5
5
|
user-invocable: false
|
|
6
6
|
allowed-tools: Read
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: performance-profiling
|
|
3
|
-
description: "
|
|
3
|
+
description: "Performance measurement and optimization: four golden signals (latency/traffic/errors/saturation), p50/p95/p99, baseline-change-measure loop, flame graphs, load testing. Triggers: performance, slow, latency, p99, flame graph, profile, bottleneck, optimization, load test, benchmark, CPU profiling, memory leak. Load when diagnosing or optimizing slow code or services."
|
|
4
4
|
effort: medium
|
|
5
5
|
allowed-tools: Read, Grep
|
|
6
6
|
user-invocable: false
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: plan-writing
|
|
3
|
-
description: "
|
|
3
|
+
description: "Implementation plan and pre-mortem drafting: phase breakdown, success criteria, risks, rollback plan, acceptance tests, estimated effort. Triggers: implementation plan, pre-mortem, phased plan, project plan, task breakdown, success criteria, rollback strategy, risk register. Load when user asks to write a plan or pre-mortem document."
|
|
4
4
|
effort: medium
|
|
5
5
|
user-invocable: false
|
|
6
6
|
allowed-tools: Read
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: rag-patterns
|
|
3
|
-
description: "
|
|
3
|
+
description: "RAG architecture: embeddings, chunking strategies, hybrid search (BM25 + vector), reranking, CRAG/self-correcting, multi-hop reasoning, evaluation metrics. Triggers: RAG, embedding, vector search, pgvector, Qdrant, Pinecone, Weaviate, chunking, reranker, retrieval, hybrid search, semantic search, knowledge base, cosine similarity. Load when building or tuning RAG systems."
|
|
4
4
|
effort: medium
|
|
5
5
|
user-invocable: false
|
|
6
6
|
allowed-tools: Read
|
|
@@ -110,3 +110,4 @@ What is explicitly NOT part of this refactor.
|
|
|
110
110
|
- No file paths or code snippets in the issue (durability)
|
|
111
111
|
- File immediately via `gh issue create` — don't ask for review
|
|
112
112
|
- Interview thoroughly before planning
|
|
113
|
+
- **Dead code cleanup is mandatory per step, not deferred** (Constitution Art. VI.1): every refactor step must leave the repo with zero orphaned references. "We'll delete the old code in a later step" is only acceptable for transitional double-write / expand-contract phases where both paths are temporarily live — and the cleanup step must be explicitly listed in the plan, not implied.
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: research-mastery
|
|
3
|
-
description: "
|
|
3
|
+
description: "Hierarchical information retrieval following strict order: KB first (smart_query/crag_search), then MCP/Context7, then web search, then LLM knowledge as last resort. Triggers: research, find information, verify fact, synthesize sources, fact-check, cross-reference, multi-source, cite sources, investigate topic. Load when any task requires external or cross-source knowledge."
|
|
4
4
|
effort: medium
|
|
5
5
|
user-invocable: false
|
|
6
6
|
allowed-tools: Read
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: ruby-patterns
|
|
3
|
-
description: "
|
|
3
|
+
description: "Ruby and Rails development patterns: blocks, metaprogramming, ActiveRecord, Sidekiq, RSpec, Sorbet/RBS, Hanami, Roda, Rack middleware. Triggers: Ruby, Rails, ActiveRecord, Sidekiq, RSpec, gem, Gemfile, bundler, rake, Hanami, Sorbet. Load when writing or reviewing Ruby code."
|
|
4
4
|
effort: medium
|
|
5
5
|
user-invocable: false
|
|
6
6
|
allowed-tools: Read
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: rust-patterns
|
|
3
|
-
description: "
|
|
3
|
+
description: "Rust development patterns: ownership, borrowing, lifetimes, async (Tokio), error handling (Result/anyhow/thiserror), traits, macros, zero-cost abstractions, unsafe boundaries. Triggers: Rust, borrow checker, lifetime, Tokio, async Rust, cargo, trait, impl, Result, unsafe, lifetime annotation, clippy. Load when writing or reviewing Rust code."
|
|
4
4
|
effort: medium
|
|
5
5
|
user-invocable: false
|
|
6
6
|
allowed-tools: Read
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: security-patterns
|
|
3
|
-
description: "
|
|
3
|
+
description: "Application security: OWASP Top 10, authN/authZ, input validation, secrets management, TLS, CSRF/XSS/SQLi, session handling, JWT, rate limiting, CSP. Triggers: security, OWASP, auth, JWT, CSRF, XSS, SQL injection, secrets, encryption, TLS, authentication, authorization, CSP, CORS, password hashing. Load when touching auth code, handling user input, or doing security review."
|
|
4
4
|
effort: medium
|
|
5
5
|
user-invocable: false
|
|
6
6
|
allowed-tools: Read
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: swift-patterns
|
|
3
|
-
description: "
|
|
3
|
+
description: "Swift and iOS development patterns: SwiftUI, Combine, async/await, property wrappers, actors, Swift Package Manager, Core Data, UIKit interop, @MainActor. Triggers: Swift, SwiftUI, Combine, iOS, Xcode, actor, property wrapper, Core Data, SPM, UIKit, @MainActor, @State, @Binding. Load when writing or reviewing Swift code."
|
|
4
4
|
effort: medium
|
|
5
5
|
user-invocable: false
|
|
6
6
|
allowed-tools: Read
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: testing-patterns
|
|
3
|
-
description: "
|
|
3
|
+
description: "Testing strategy and craft: pyramid vs trophy, unit/integration/e2e split, fixtures, mocks vs fakes vs stubs, AAA pattern, flaky test diagnosis, coverage goals, property-based testing. Triggers: test, testing strategy, fixture, mock, stub, AAA, unit test, integration test, e2e, Playwright, Cypress, flaky, coverage, TDD, test pyramid. Load when writing, reviewing, or designing test suites."
|
|
4
4
|
effort: medium
|
|
5
5
|
user-invocable: false
|
|
6
6
|
allowed-tools: Read
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: typescript-patterns
|
|
3
|
-
description: "
|
|
3
|
+
description: "TypeScript type safety patterns: strict mode, generics, conditional types, template literals, discriminated unions, branded types, Zod, satisfies operator, const assertions. Triggers: TypeScript, TS, generics, conditional type, utility type, strict, Zod, satisfies, discriminated union, type safety, type narrowing, template literal type. Load when writing or reviewing TypeScript code."
|
|
4
4
|
effort: medium
|
|
5
5
|
user-invocable: false
|
|
6
6
|
allowed-tools: Read
|
|
@@ -54,6 +54,9 @@ Skip any step = lying, not verifying
|
|
|
54
54
|
| Regression test works | Red-green cycle verified | Test passes once |
|
|
55
55
|
| Agent completed | VCS diff shows changes | Agent reports "success" |
|
|
56
56
|
| Requirements met | Line-by-line checklist | Tests passing |
|
|
57
|
+
| No dead code (Art. VI.1) | Grep for every removed/renamed symbol: 0 references | "I cleaned up what I touched" |
|
|
58
|
+
| Behavior change covered (Art. VI.2) | Integration test for the API surface + unit test + docs updated | Unit test on the helper only |
|
|
59
|
+
| Diff is clean (Art. VI.4) | Re-read full diff: no orphaned imports, no stale docs, no skipped fixes | "I only changed what I needed" |
|
|
57
60
|
|
|
58
61
|
## Red Flags — STOP
|
|
59
62
|
|
|
@@ -102,6 +105,10 @@ CORRECT: Agent reports success → Check VCS diff → Verify changes → Report
|
|
|
102
105
|
WRONG: Trust agent report at face value
|
|
103
106
|
```
|
|
104
107
|
|
|
108
|
+
## Constitutional Anchors
|
|
109
|
+
|
|
110
|
+
This skill enforces **Constitution Art. VI.4 (Verify Before Claiming Done)**. The diff re-read is not optional: before any completion claim, confirm no orphaned references, no missing test coverage for changed paths, no stale docs. A task is not done while any of those exist.
|
|
111
|
+
|
|
105
112
|
## The Bottom Line
|
|
106
113
|
|
|
107
114
|
Run the command. Read the output. THEN claim the result.
|
|
@@ -2,11 +2,11 @@
|
|
|
2
2
|
title: "SOP: Release Preparation"
|
|
3
3
|
category: procedures
|
|
4
4
|
service: ai-toolkit
|
|
5
|
-
tags: [sop, release, version, publish, changelog, semver]
|
|
6
|
-
version: "1.
|
|
5
|
+
tags: [sop, release, version, publish, changelog, semver, provenance, sarif]
|
|
6
|
+
version: "1.7.0"
|
|
7
7
|
created: "2026-04-10"
|
|
8
|
-
last_updated: "2026-04-
|
|
9
|
-
description: "Step-by-step checklist for preparing a new ai-toolkit release — version sync, changelog, artifact regeneration, validation, and tagging. Run BEFORE every git tag."
|
|
8
|
+
last_updated: "2026-04-18"
|
|
9
|
+
description: "Step-by-step checklist for preparing a new ai-toolkit release — version sync, changelog, artifact regeneration, validation, and tagging. Run BEFORE every git tag. Includes mandatory Provenance, SARIF, and checksum-pin checks added in v2.8.0."
|
|
10
10
|
---
|
|
11
11
|
|
|
12
12
|
# SOP: Release Preparation
|
|
@@ -42,8 +42,13 @@ python3 scripts/generate_codex_rules.py .
|
|
|
42
42
|
python3 scripts/generate_llms_txt.py > llms.txt
|
|
43
43
|
python3 scripts/generate_llms_txt.py --full > llms-full.txt
|
|
44
44
|
|
|
45
|
-
# 5. Validate + audit + test
|
|
46
|
-
python3 scripts/validate.py --strict && python3 scripts/audit_skills.py --ci && npm test
|
|
45
|
+
# 5. Validate + audit + SARIF + test
|
|
46
|
+
python3 scripts/validate.py --strict && python3 scripts/audit_skills.py --ci && python3 scripts/audit_skills.py --sarif > /tmp/audit.sarif && npm test
|
|
47
|
+
|
|
48
|
+
# 5a. Supply-chain standard (v2.8.0+) — non-negotiable
|
|
49
|
+
grep -q -- '--provenance' .github/workflows/publish.yml || { echo "MISSING --provenance"; exit 1; }
|
|
50
|
+
grep -q 'id-token: write' .github/workflows/publish.yml || { echo "MISSING id-token: write"; exit 1; }
|
|
51
|
+
python3 scripts/audit_skills.py --permissions # review Bash/Write/Edit footprint
|
|
47
52
|
|
|
48
53
|
# 6. Commit + tag + push
|
|
49
54
|
git add -A && git commit -m "chore: release vX.Y.Z"
|
|
@@ -200,17 +205,21 @@ Run the full quality gate:
|
|
|
200
205
|
```bash
|
|
201
206
|
python3 scripts/validate.py --strict
|
|
202
207
|
python3 scripts/audit_skills.py --ci
|
|
208
|
+
python3 scripts/audit_skills.py --sarif > audit.sarif # MANDATORY — GHAS ingest
|
|
209
|
+
python3 scripts/audit_skills.py --permissions # review Bash/Write/Edit footprint
|
|
203
210
|
npm test
|
|
204
211
|
```
|
|
205
212
|
|
|
206
213
|
**Expected results:**
|
|
207
214
|
- `validate.py`: `Errors: 0 | Warnings: 0 | VALIDATION PASSED`
|
|
208
|
-
- `audit_skills.py`: `HIGH: 0 | WARN: 0` (INFO is acceptable)
|
|
215
|
+
- `audit_skills.py --ci`: `HIGH: 0 | WARN: 0` (INFO is acceptable)
|
|
216
|
+
- `audit_skills.py --sarif`: valid JSON, non-empty `runs[0].tool.driver.rules`
|
|
217
|
+
- `audit_skills.py --permissions`: review `Skills with Bash + Write + Edit` list — any newly-added skill with broad access MUST be justified in the CHANGELOG entry
|
|
209
218
|
- `npm test`: `1..N` with zero `not ok`
|
|
210
219
|
|
|
211
220
|
**One-liner:**
|
|
212
221
|
```bash
|
|
213
|
-
python3 scripts/validate.py --strict && python3 scripts/audit_skills.py --ci && npm test
|
|
222
|
+
python3 scripts/validate.py --strict && python3 scripts/audit_skills.py --ci && python3 scripts/audit_skills.py --sarif > audit.sarif && npm test
|
|
214
223
|
```
|
|
215
224
|
|
|
216
225
|
**If tests fail:** Fix the issue, do NOT skip. Common failures:
|
|
@@ -218,6 +227,48 @@ python3 scripts/validate.py --strict && python3 scripts/audit_skills.py --ci &&
|
|
|
218
227
|
- Missing frontmatter → add to new KB docs
|
|
219
228
|
- Broken symlink → `ai-toolkit doctor --fix`
|
|
220
229
|
|
|
230
|
+
### Phase 5a: Supply-Chain Hardening Verification (v2.8.0+)
|
|
231
|
+
|
|
232
|
+
These checks enforce the security standard introduced in v2.8.0. Do NOT tag a release until all pass.
|
|
233
|
+
|
|
234
|
+
**1. Publish workflow emits provenance:**
|
|
235
|
+
|
|
236
|
+
```bash
|
|
237
|
+
grep -E '\-\-provenance|id-token: write' .github/workflows/publish.yml
|
|
238
|
+
```
|
|
239
|
+
|
|
240
|
+
- [ ] Both markers present (`--provenance` flag + `id-token: write` permission)
|
|
241
|
+
- [ ] Any PR that changes `publish.yml` REQUIRES an approved security review
|
|
242
|
+
|
|
243
|
+
**2. URL-sourced rules and hooks are checksum-pinned:**
|
|
244
|
+
|
|
245
|
+
```bash
|
|
246
|
+
# On a machine that has consumed URL rules/hooks at least once:
|
|
247
|
+
jq 'to_entries | map(select(.value.url != null and (.value.sha256 // "" | length) == 0))' ~/.softspark/ai-toolkit/rules/sources.json
|
|
248
|
+
jq 'to_entries | map(select(.value.url != null and (.value.sha256 // "" | length) == 0))' ~/.softspark/ai-toolkit/hooks/external/sources.json
|
|
249
|
+
```
|
|
250
|
+
|
|
251
|
+
- [ ] Both queries return empty arrays (every URL entry has a `sha256`)
|
|
252
|
+
- [ ] If not, run `ai-toolkit update` to backfill missing hashes before tagging
|
|
253
|
+
|
|
254
|
+
**3. Audit SARIF output is well-formed:**
|
|
255
|
+
|
|
256
|
+
```bash
|
|
257
|
+
python3 scripts/audit_skills.py --sarif | python3 -c "import json, sys; d=json.load(sys.stdin); assert d['version']=='2.1.0' and d['runs'][0]['tool']['driver']['name']; print('SARIF OK')"
|
|
258
|
+
```
|
|
259
|
+
|
|
260
|
+
- [ ] Prints `SARIF OK`
|
|
261
|
+
- [ ] If the script ever grows new rule classes, extend the SARIF `rules[]` coverage before releasing
|
|
262
|
+
|
|
263
|
+
**4. Strict-pin mode passes on CI** (optional, recommended for stable branches):
|
|
264
|
+
|
|
265
|
+
```bash
|
|
266
|
+
AI_TOOLKIT_STRICT_PIN=1 ai-toolkit update --dry-run
|
|
267
|
+
```
|
|
268
|
+
|
|
269
|
+
- [ ] Exit 0, no `CHECKSUM CHANGED` line
|
|
270
|
+
- [ ] Any unexpected upstream change blocks the release until explicitly approved
|
|
271
|
+
|
|
221
272
|
---
|
|
222
273
|
|
|
223
274
|
## Phase 6: Commit
|
|
@@ -250,10 +301,12 @@ git push origin main --tags
|
|
|
250
301
|
This triggers `.github/workflows/publish.yml` which:
|
|
251
302
|
1. Runs `validate.py --strict`
|
|
252
303
|
2. Runs `npm test`
|
|
253
|
-
3. Publishes to npm as `@softspark/ai-toolkit@X.Y.Z`
|
|
304
|
+
3. Publishes to npm as `@softspark/ai-toolkit@X.Y.Z` with `--provenance` (SLSA v1 build attestation)
|
|
305
|
+
|
|
306
|
+
**Provenance is non-negotiable.** If `id-token: write` permission or the `--provenance` flag is missing from `publish.yml`, fix it BEFORE tagging — an unsigned release is a regression against the v2.8.0 standard.
|
|
254
307
|
|
|
255
308
|
**After CI completes:** Run the [Release Verification SOP](release-verification-sop.md)
|
|
256
|
-
to smoke-test the published package.
|
|
309
|
+
to smoke-test the published package AND verify the provenance attestation landed on npm.
|
|
257
310
|
|
|
258
311
|
---
|
|
259
312
|
|
|
@@ -288,8 +341,12 @@ git push origin --delete vX.Y.Z
|
|
|
288
341
|
| 7 | CHANGELOG.md | Add release entry | Entry exists for vX.Y.Z |
|
|
289
342
|
| 8 | Regenerate artifacts | `generate_agents_md.py`, `generate_codex_rules.py`, `generate_llms_txt.py` | No unexpected diff |
|
|
290
343
|
| 9 | Validate | `validate.py --strict` | 0 errors, 0 warnings |
|
|
291
|
-
| 10 | Security audit | `audit_skills.py --ci` | 0 HIGH |
|
|
292
|
-
| 11 |
|
|
293
|
-
| 12 |
|
|
294
|
-
| 13 |
|
|
295
|
-
| 14 |
|
|
344
|
+
| 10 | Security audit (CI mode) | `audit_skills.py --ci` | 0 HIGH |
|
|
345
|
+
| 11 | Security audit (SARIF) | `audit_skills.py --sarif` | Valid SARIF 2.1.0 JSON |
|
|
346
|
+
| 12 | Per-skill permissions | `audit_skills.py --permissions` | New broad-access skills justified in CHANGELOG |
|
|
347
|
+
| 13 | Provenance flag check | `grep -- '--provenance' .github/workflows/publish.yml` | Present |
|
|
348
|
+
| 14 | Checksum-pin backfill | `sources.json` entries all have `sha256` | No unpinned URL sources |
|
|
349
|
+
| 15 | Tests | `npm test` | All pass |
|
|
350
|
+
| 16 | Commit | `git commit` | Clean working tree |
|
|
351
|
+
| 17 | Tag | `git tag vX.Y.Z` | Tag exists |
|
|
352
|
+
| 18 | Push | `git push origin main --tags` | CI triggered with `id-token: write` |
|
|
@@ -2,11 +2,11 @@
|
|
|
2
2
|
title: "SOP: Release Verification"
|
|
3
3
|
category: procedures
|
|
4
4
|
service: ai-toolkit
|
|
5
|
-
tags: [sop, verification, release, smoke-test, install, update, qa]
|
|
6
|
-
version: "1.
|
|
5
|
+
tags: [sop, verification, release, smoke-test, install, update, qa, provenance, sarif]
|
|
6
|
+
version: "1.2.0"
|
|
7
7
|
created: "2026-04-08"
|
|
8
|
-
last_updated: "2026-04-
|
|
9
|
-
description: "End-to-end smoke test after installing or updating @softspark/ai-toolkit — verifies CLI, install, doctor, validation, tests, and
|
|
8
|
+
last_updated: "2026-04-18"
|
|
9
|
+
description: "End-to-end smoke test after installing or updating @softspark/ai-toolkit — verifies CLI, install, doctor, validation, tests, eject, npm provenance attestation, SARIF audit, and per-skill permissions. Reflects the v2.8.0 supply-chain standard."
|
|
10
10
|
---
|
|
11
11
|
|
|
12
12
|
# SOP: Release Verification
|
|
@@ -31,7 +31,7 @@ Verifies all critical paths from the user's perspective.
|
|
|
31
31
|
|
|
32
32
|
## Quick Checklist (TL;DR)
|
|
33
33
|
|
|
34
|
-
|
|
34
|
+
13 commands — if all pass, the release is ready:
|
|
35
35
|
|
|
36
36
|
```bash
|
|
37
37
|
# Pre-commit (Phase 0)
|
|
@@ -47,6 +47,11 @@ ai-toolkit status # 7. Status OK?
|
|
|
47
47
|
ai-toolkit doctor # 8. Health check passed?
|
|
48
48
|
ai-toolkit install --dry-run # 9. Global install OK?
|
|
49
49
|
python3 scripts/audit_skills.py --ci # 10. Security audit clean?
|
|
50
|
+
|
|
51
|
+
# Supply-chain verification (Phase 8, v2.8.0+)
|
|
52
|
+
python3 scripts/audit_skills.py --sarif | python3 -c "import json,sys; assert json.load(sys.stdin)['version']=='2.1.0'; print('SARIF OK')" # 11. SARIF 2.1.0 well-formed?
|
|
53
|
+
python3 scripts/audit_skills.py --permissions | head -30 # 12. Broad-access skills reviewed?
|
|
54
|
+
npm view @softspark/ai-toolkit@X.Y.Z --json | python3 -c "import json,sys; d=json.load(sys.stdin); assert d['dist']['attestations']['provenance']['predicateType']=='https://slsa.dev/provenance/v1'; print('PROVENANCE OK')" # 13. Provenance attested on npm?
|
|
50
55
|
```
|
|
51
56
|
|
|
52
57
|
---
|
|
@@ -230,6 +235,72 @@ cd - && rm -rf /tmp/ai-toolkit-eject-test
|
|
|
230
235
|
- [ ] Skills copied as real directories
|
|
231
236
|
- [ ] Rules inlined into CLAUDE.md
|
|
232
237
|
- [ ] constitution.md and ARCHITECTURE.md copied
|
|
238
|
+
- [ ] `output-styles/` directory present (v2.7.1+)
|
|
239
|
+
|
|
240
|
+
---
|
|
241
|
+
|
|
242
|
+
## Phase 8: Supply-Chain Verification (2 min, v2.8.0+)
|
|
243
|
+
|
|
244
|
+
These checks enforce the v2.8.0 security standard on a freshly-published release. Run AFTER the `publish.yml` workflow completes on the tag.
|
|
245
|
+
|
|
246
|
+
### 8.1 Provenance attestation on npm
|
|
247
|
+
|
|
248
|
+
```bash
|
|
249
|
+
VERSION="X.Y.Z" # the tag just published
|
|
250
|
+
npm view "@softspark/ai-toolkit@${VERSION}" --json \
|
|
251
|
+
| python3 -c "import json, sys; d=json.load(sys.stdin); att=d['dist'].get('attestations', {}); assert att.get('provenance', {}).get('predicateType') == 'https://slsa.dev/provenance/v1', f'NO PROVENANCE for {d[\"version\"]}'; print(f'PROVENANCE OK: {att[\"url\"]}')"
|
|
252
|
+
```
|
|
253
|
+
|
|
254
|
+
**Verify:**
|
|
255
|
+
- [ ] Exit 0 and prints `PROVENANCE OK: https://registry.npmjs.org/...`
|
|
256
|
+
- [ ] `https://www.npmjs.com/package/@softspark/ai-toolkit/v/${VERSION}` shows the green "Provenance" badge
|
|
257
|
+
|
|
258
|
+
**If provenance is missing:** the `publish.yml` ran without `id-token: write` or `--provenance`. Restore them and cut a patch release — a silently unsigned publish is a regression against the v2.8.0 standard.
|
|
259
|
+
|
|
260
|
+
### 8.2 Audit SARIF output (for GHAS ingest)
|
|
261
|
+
|
|
262
|
+
```bash
|
|
263
|
+
python3 scripts/audit_skills.py --sarif > /tmp/audit.sarif
|
|
264
|
+
python3 -c "import json; d=json.load(open('/tmp/audit.sarif')); assert d['version']=='2.1.0' and d['runs'][0]['tool']['driver']['name']=='ai-toolkit-audit-skills'; print(f'SARIF OK: {len(d[\"runs\"][0][\"results\"])} results across {len(d[\"runs\"][0][\"tool\"][\"driver\"][\"rules\"])} rules')"
|
|
265
|
+
```
|
|
266
|
+
|
|
267
|
+
**Verify:**
|
|
268
|
+
- [ ] Valid SARIF 2.1.0
|
|
269
|
+
- [ ] In the publishing repo, the CI job uploads `audit.sarif` via `github/codeql-action/upload-sarif@v3` so findings appear in the Security tab
|
|
270
|
+
|
|
271
|
+
### 8.3 Per-skill permissions report
|
|
272
|
+
|
|
273
|
+
```bash
|
|
274
|
+
python3 scripts/audit_skills.py --permissions | head -40
|
|
275
|
+
```
|
|
276
|
+
|
|
277
|
+
**Verify:**
|
|
278
|
+
- [ ] Bash skill count has NOT jumped unexpectedly since the previous release
|
|
279
|
+
- [ ] Any newly added entry under `Skills with Bash + Write + Edit` matches a CHANGELOG bullet that justifies the broad scope
|
|
280
|
+
- [ ] JSON form (`--permissions --json`) is available for automated drift dashboards
|
|
281
|
+
|
|
282
|
+
### 8.4 URL-sourced rules/hooks are checksum-pinned
|
|
283
|
+
|
|
284
|
+
```bash
|
|
285
|
+
jq '.rules // .hooks // {}' ~/.softspark/ai-toolkit/rules/sources.json 2>/dev/null \
|
|
286
|
+
| python3 -c "import json, sys; d=json.load(sys.stdin) or {}; bad=[n for n,v in d.items() if v.get('url') and not v.get('sha256')]; assert not bad, f'UNPINNED: {bad}'; print(f'RULE PIN OK: {len(d)} URL rules, all with sha256')"
|
|
287
|
+
jq '.hooks // {}' ~/.softspark/ai-toolkit/hooks/external/sources.json 2>/dev/null \
|
|
288
|
+
| python3 -c "import json, sys; d=json.load(sys.stdin) or {}; bad=[n for n,v in d.items() if v.get('url') and not v.get('sha256')]; assert not bad, f'UNPINNED: {bad}'; print(f'HOOK PIN OK: {len(d)} URL hooks, all with sha256')"
|
|
289
|
+
```
|
|
290
|
+
|
|
291
|
+
**Verify:**
|
|
292
|
+
- [ ] Both commands print `... PIN OK`
|
|
293
|
+
- [ ] If any entry is unpinned, the `register_url_source()` call missed passing `content=` — fix the call-site and retag
|
|
294
|
+
|
|
295
|
+
### 8.5 Strict-pin smoke test (optional but recommended)
|
|
296
|
+
|
|
297
|
+
```bash
|
|
298
|
+
AI_TOOLKIT_STRICT_PIN=1 ai-toolkit update --dry-run
|
|
299
|
+
```
|
|
300
|
+
|
|
301
|
+
**Verify:**
|
|
302
|
+
- [ ] Exit 0, no `CHECKSUM CHANGED` line
|
|
303
|
+
- [ ] If a checksum change was intentional (e.g. upstream rule update), document it in the CHANGELOG entry before tagging
|
|
233
304
|
|
|
234
305
|
---
|
|
235
306
|
|
|
@@ -279,7 +350,11 @@ ai-toolkit eject /tmp/test # retry
|
|
|
279
350
|
| Health | `doctor`: 0 errors, 0 warnings, PASSED |
|
|
280
351
|
| Install | `--dry-run` correct counts, `--local` all configs |
|
|
281
352
|
| Quality | `validate.py --strict`: PASSED |
|
|
282
|
-
| Security | `audit_skills.py --ci`: 0 HIGH |
|
|
353
|
+
| Security (baseline) | `audit_skills.py --ci`: 0 HIGH |
|
|
354
|
+
| Security (SARIF) | `audit_skills.py --sarif`: valid SARIF 2.1.0 with non-empty rules array |
|
|
355
|
+
| Security (permissions) | `audit_skills.py --permissions`: broad-access skills unchanged or justified in CHANGELOG |
|
|
356
|
+
| Supply chain | `dist.attestations.provenance.predicateType == https://slsa.dev/provenance/v1` on npm |
|
|
357
|
+
| Supply chain | All `sources.json` URL entries carry a `sha256`; `AI_TOOLKIT_STRICT_PIN=1 ai-toolkit update --dry-run` passes |
|
|
283
358
|
| Tests | `npm test`: N/N passed, 0 failures |
|
|
284
|
-
| Eject | Standalone
|
|
359
|
+
| Eject | Standalone `.claude/` with real files AND `output-styles/` directory |
|
|
285
360
|
| Guards | Destructive commands blocked |
|