@softspark/ai-toolkit 4.32.3 → 4.32.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +36 -0
- package/README.md +17 -23
- package/app/.claude-plugin/plugin.json +1 -1
- package/app/ARCHITECTURE.md +1 -1
- package/app/claude-app/skills/ai-toolkit-rules/SKILL.md +8 -1
- package/app/rules/common/security.md +2 -1
- package/app/rules/common/testing.md +6 -0
- package/app/skills/api-patterns/SKILL.md +37 -11
- package/app/skills/api-patterns/reference/error-contracts.md +88 -0
- package/app/skills/review/SKILL.md +5 -0
- package/app/skills/security-patterns/SKILL.md +2 -2
- package/app/skills/security-patterns/reference/input-validation.md +69 -1
- package/app/skills/testing-patterns/SKILL.md +19 -0
- package/benchmarks/ecosystem-doctor-snapshot.json +13 -13
- package/kb/procedures/sop-post-release-testing.md +159 -58
- package/kb/procedures/sop-release-verification.md +38 -13
- package/kb/procedures/sop-release.md +9 -2
- package/kb/reference/architecture-overview.md +3 -3
- package/kb/reference/language-rules.md +7 -1
- package/kb/reference/skills-catalog.md +2 -2
- package/llms-full.txt +219 -80
- package/manifest.json +3 -3
- package/package.json +2 -2
|
@@ -3,10 +3,10 @@ title: "SOP: Post-Release Testing"
|
|
|
3
3
|
category: procedures
|
|
4
4
|
service: ai-toolkit
|
|
5
5
|
tags: [sop, post-release, smoke-test, npm, sandbox, plugin-pack, provenance, isolation]
|
|
6
|
-
version: "1.2.
|
|
6
|
+
version: "1.2.1"
|
|
7
7
|
created: "2026-07-26"
|
|
8
|
-
last_updated: "2026-
|
|
9
|
-
description: "Smoke-test a published @softspark/ai-toolkit
|
|
8
|
+
last_updated: "2026-09-06"
|
|
9
|
+
description: "Smoke-test a published @softspark/ai-toolkit npm artifact in a disposable container or VM with its default HOME, no host settings or credential mounts, host-side configuration fingerprints, and retained evidence. Covers provenance, CLI, doctor, installed skill scripts, scanner wiring, and the plugin-pack lifecycle."
|
|
10
10
|
---
|
|
11
11
|
|
|
12
12
|
# SOP: Post-Release Testing
|
|
@@ -16,73 +16,147 @@ actually install, from npm, rather than the working tree.
|
|
|
16
16
|
|
|
17
17
|
Sibling procedures exist for `jira-mcp` and `legal-pl-pack`; this is the
|
|
18
18
|
ai-toolkit equivalent. It complements
|
|
19
|
-
[Release Verification](sop-release-verification.md),
|
|
20
|
-
|
|
21
|
-
|
|
19
|
+
[Release Verification](sop-release-verification.md), whose cross-editor checks
|
|
20
|
+
must use the same isolated published artifact. Neither procedure installs or
|
|
21
|
+
updates the maintainer's working copy.
|
|
22
22
|
|
|
23
23
|
**Time:** 10 minutes.
|
|
24
24
|
|
|
25
25
|
## Why isolation is the first step, not a detail
|
|
26
26
|
|
|
27
|
-
The toolkit
|
|
28
|
-
|
|
29
|
-
|
|
27
|
+
The toolkit writes settings beneath the current user's home directory. Run the
|
|
28
|
+
published artifact in a disposable Docker container or VM with its normal HOME.
|
|
29
|
+
Do not override host HOME, CODEX_HOME, or another editor's configuration root as
|
|
30
|
+
a substitute for isolation. Do not expose the host home, credentials, SSH agent,
|
|
31
|
+
Docker socket, or existing toolkit installation to the test environment.
|
|
30
32
|
|
|
31
|
-
|
|
32
|
-
Nothing is global.
|
|
33
|
+
## Phase 1: Create and verify an isolated test environment
|
|
33
34
|
|
|
34
|
-
|
|
35
|
+
The recipe below uses Docker. A disposable VM is equivalent only when host shared
|
|
36
|
+
folders and authentication forwarding are absent. The host needs Docker and
|
|
37
|
+
Python 3; the container prerequisites are installed separately below.
|
|
38
|
+
|
|
39
|
+
**Host terminal:** keep this terminal open for Phases 7 and 8. The fingerprint
|
|
40
|
+
reads only these toolkit-managed settings files and records hashes and presence,
|
|
41
|
+
never their contents. Add a path only after verifying that the tested installer
|
|
42
|
+
actually manages it.
|
|
35
43
|
|
|
36
44
|
```bash
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
45
|
+
set -o pipefail
|
|
46
|
+
VERSION="X.Y.Z"
|
|
47
|
+
EVIDENCE=$(mktemp -d "${TMPDIR:-/tmp}/ai-toolkit-release-${VERSION}.XXXXXX")
|
|
48
|
+
SMOKE_CONTAINER=$(python3 -c 'import uuid; print("ai-toolkit-smoke-" + uuid.uuid4().hex)')
|
|
49
|
+
|
|
50
|
+
fingerprint_host() {
|
|
51
|
+
python3 - <<'PY'
|
|
52
|
+
import hashlib
|
|
53
|
+
import json
|
|
54
|
+
import os
|
|
55
|
+
from pathlib import Path
|
|
56
|
+
|
|
57
|
+
home = Path.home()
|
|
58
|
+
managed = [
|
|
59
|
+
".claude/settings.json",
|
|
60
|
+
".claude.json",
|
|
61
|
+
".softspark/ai-toolkit/plugins.json",
|
|
62
|
+
".codex/config.toml",
|
|
63
|
+
".cursor/mcp.json",
|
|
64
|
+
".gemini/settings.json",
|
|
65
|
+
".config/opencode/opencode.json",
|
|
66
|
+
]
|
|
67
|
+
rows = {}
|
|
68
|
+
for relative in managed:
|
|
69
|
+
path = home / relative
|
|
70
|
+
row = {"exists": path.exists() or path.is_symlink()}
|
|
71
|
+
if path.is_symlink():
|
|
72
|
+
row["link_sha256"] = hashlib.sha256(os.readlink(path).encode()).hexdigest()
|
|
73
|
+
if path.is_file():
|
|
74
|
+
row["sha256"] = hashlib.sha256(path.read_bytes()).hexdigest()
|
|
75
|
+
rows[relative] = row
|
|
76
|
+
print(json.dumps({
|
|
77
|
+
"host_home_sha256": hashlib.sha256(str(home).encode()).hexdigest(),
|
|
78
|
+
"files": rows,
|
|
79
|
+
}, sort_keys=True, indent=2))
|
|
80
|
+
PY
|
|
81
|
+
}
|
|
82
|
+
|
|
83
|
+
fingerprint_host > "$EVIDENCE/host-before.json"
|
|
84
|
+
docker run -d --name "$SMOKE_CONTAINER" \
|
|
85
|
+
--label "org.softspark.release-smoke=$SMOKE_CONTAINER" \
|
|
86
|
+
--env "VERSION=$VERSION" \
|
|
87
|
+
node:22-bookworm sleep infinity
|
|
88
|
+
docker inspect "$SMOKE_CONTAINER" > "$EVIDENCE/container-before.json"
|
|
89
|
+
python3 - "$EVIDENCE/container-before.json" <<'PY'
|
|
90
|
+
import json
|
|
91
|
+
import sys
|
|
92
|
+
|
|
93
|
+
container = json.load(open(sys.argv[1], encoding="utf-8"))[0]
|
|
94
|
+
assert container["Mounts"] == [], "Smoke container must have no mounts"
|
|
95
|
+
host = container["HostConfig"]
|
|
96
|
+
assert not host["Privileged"], "Privileged containers are forbidden"
|
|
97
|
+
assert host["NetworkMode"] != "host", "Do not share the host network namespace"
|
|
98
|
+
assert host["PidMode"] != "host", "Do not share host processes"
|
|
99
|
+
PY
|
|
100
|
+
docker exec "$SMOKE_CONTAINER" bash -lc \
|
|
101
|
+
'test "$HOME" = "$(getent passwd "$(id -u)" | cut -d: -f6)"'
|
|
102
|
+
docker exec "$SMOKE_CONTAINER" bash -lc \
|
|
103
|
+
'apt-get update && apt-get install -y --no-install-recommends python3 python3-yaml git coreutils jq bats shellcheck ca-certificates curl util-linux' \
|
|
104
|
+
2>&1 | tee "$EVIDENCE/bootstrap.log"
|
|
42
105
|
```
|
|
43
106
|
|
|
44
|
-
|
|
107
|
+
Install any additional prerequisite declared by the pack under test only inside
|
|
108
|
+
the container. GNU coreutils supplies timeout; util-linux supplies the session
|
|
109
|
+
recorder. Use docker cp for evidence transfer, not host temporary-directory bind
|
|
110
|
+
mounts: a remote Docker daemon may not see the host's /private/tmp.
|
|
111
|
+
|
|
112
|
+
**Enter the container, then run Phases 2 through 6 there:**
|
|
45
113
|
|
|
46
114
|
```bash
|
|
47
|
-
|
|
48
|
-
import json, pathlib
|
|
49
|
-
p = pathlib.Path('$SB/../real-before.json')
|
|
50
|
-
import os
|
|
51
|
-
home = pathlib.Path(os.path.expanduser('~'))
|
|
52
|
-
" 2>/dev/null
|
|
53
|
-
# Simpler: note what exists today.
|
|
54
|
-
cat ~/.softspark/ai-toolkit/plugins.json 2>/dev/null
|
|
115
|
+
docker exec -it "$SMOKE_CONTAINER" bash
|
|
55
116
|
```
|
|
56
117
|
|
|
118
|
+
Inside that container shell:
|
|
119
|
+
|
|
120
|
+
```bash
|
|
121
|
+
SB=/tmp/ai-toolkit-smoke
|
|
122
|
+
AT="$SB/npm/bin/ai-toolkit"
|
|
123
|
+
mkdir -p "$SB/npm" "$SB/evidence"
|
|
124
|
+
export SB AT
|
|
125
|
+
exec script -q -e -a "$SB/evidence/session.log" -c 'bash --noprofile --norc'
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
Keep HOME at the container user's default. VERSION was passed when the container
|
|
129
|
+
was created; SB and the npm prefix are disposable container paths. No command in
|
|
130
|
+
Phases 2 through 6 runs in the host shell.
|
|
131
|
+
|
|
57
132
|
## Phase 2: Provenance
|
|
58
133
|
|
|
59
134
|
Do this before installing anything: an unsigned publish is a release-blocking
|
|
60
135
|
regression, and there is no point smoke-testing a build you would have to redo.
|
|
61
136
|
|
|
62
137
|
```bash
|
|
63
|
-
VERSION
|
|
64
|
-
|
|
65
|
-
| python3 -c "
|
|
138
|
+
npm view "@softspark/ai-toolkit@${VERSION}" --json > "$SB/evidence/npm-view.json"
|
|
139
|
+
python3 -c "
|
|
66
140
|
import json, sys
|
|
67
|
-
d = json.load(sys.
|
|
141
|
+
d = json.load(open(sys.argv[1], encoding='utf-8')); att = d['dist'].get('attestations', {})
|
|
68
142
|
pt = att.get('provenance', {}).get('predicateType')
|
|
69
143
|
assert pt == 'https://slsa.dev/provenance/v1', f'NO PROVENANCE: {pt}'
|
|
70
144
|
print('PROVENANCE OK:', att['url'])
|
|
71
|
-
"
|
|
145
|
+
" "$SB/evidence/npm-view.json"
|
|
72
146
|
```
|
|
73
147
|
|
|
74
148
|
## Phase 3: Install from npm
|
|
75
149
|
|
|
76
150
|
```bash
|
|
77
151
|
npm install -g --prefix "$SB/npm" "@softspark/ai-toolkit@${VERSION}"
|
|
78
|
-
"$AT" --version # must equal VERSION
|
|
152
|
+
"$AT" --version | tee "$SB/evidence/version.txt" # must equal VERSION
|
|
79
153
|
"$AT" --help >/dev/null && echo "help OK"
|
|
80
154
|
```
|
|
81
155
|
|
|
82
156
|
## Phase 4: Core surfaces
|
|
83
157
|
|
|
84
158
|
```bash
|
|
85
|
-
"$AT" install #
|
|
159
|
+
"$AT" install # global install inside the container's default HOME
|
|
86
160
|
"$AT" doctor # must end: Errors: 0 | Warnings: 0
|
|
87
161
|
"$AT" status
|
|
88
162
|
"$AT" plugin list # pack count must match app/plugins/
|
|
@@ -120,8 +194,13 @@ for D in "$HOME"/.claude/skills/*/; do
|
|
|
120
194
|
rel=${ref##*\$\{CLAUDE_SKILL_DIR\}/}
|
|
121
195
|
printf '%-22s %-10s %-26s ' "$s" "$interp" "$rel"
|
|
122
196
|
[ -f "$D/$rel" ] || { echo 'PATH DOES NOT RESOLVE'; continue; }
|
|
123
|
-
out=$(CLAUDE_SKILL_DIR="$D" timeout 20 "$interp" "$D/$rel" --help </dev/null 2>&1
|
|
124
|
-
|
|
197
|
+
if out=$(CLAUDE_SKILL_DIR="$D" timeout 20 "$interp" "$D/$rel" --help </dev/null 2>&1); then
|
|
198
|
+
rc=0
|
|
199
|
+
else
|
|
200
|
+
rc=$?
|
|
201
|
+
fi
|
|
202
|
+
printf '%s\n' "$out" > "$SB/evidence/skill-$s.log"
|
|
203
|
+
printf 'rc=%s %s\n' "$rc" "$(printf '%s\n' "$out" | head -1 | cut -c1-40)"
|
|
125
204
|
done
|
|
126
205
|
```
|
|
127
206
|
|
|
@@ -262,37 +341,59 @@ again, pointing its source-override variable at a dead URL:
|
|
|
262
341
|
- [ ] `plugin status` says the pack is inert and names the fix
|
|
263
342
|
- [ ] Re-installing without the broken source recovers
|
|
264
343
|
|
|
265
|
-
## Phase 7:
|
|
344
|
+
## Phase 7: Verify the host configuration
|
|
345
|
+
|
|
346
|
+
Exit the recorded container shell. This returns to the unchanged host terminal
|
|
347
|
+
from Phase 1. Run fingerprint_host there, not through docker exec and not in a
|
|
348
|
+
shell that changed HOME:
|
|
266
349
|
|
|
267
350
|
```bash
|
|
268
|
-
|
|
269
|
-
|
|
270
|
-
|
|
271
|
-
|
|
272
|
-
|
|
273
|
-
|
|
274
|
-
|
|
275
|
-
"
|
|
351
|
+
fingerprint_host > "$EVIDENCE/host-after.json"
|
|
352
|
+
cmp -s "$EVIDENCE/host-before.json" "$EVIDENCE/host-after.json" || {
|
|
353
|
+
diff -u "$EVIDENCE/host-before.json" "$EVIDENCE/host-after.json"
|
|
354
|
+
echo "Host configuration changed: investigate before accepting the release."
|
|
355
|
+
exit 1
|
|
356
|
+
}
|
|
357
|
+
docker inspect "$SMOKE_CONTAINER" > "$EVIDENCE/container-after.json"
|
|
276
358
|
```
|
|
277
359
|
|
|
278
|
-
|
|
360
|
+
The fingerprints must match. This proves the enumerated managed settings stayed
|
|
361
|
+
unchanged; the recorded container configuration separately proves there were no
|
|
362
|
+
host mounts or shared host namespaces. Do not claim to have hashed the whole
|
|
363
|
+
home directory, or print settings contents to demonstrate isolation.
|
|
279
364
|
|
|
280
|
-
## Phase 8:
|
|
365
|
+
## Phase 8: Preserve evidence and remove only the owned container
|
|
281
366
|
|
|
282
|
-
|
|
283
|
-
|
|
367
|
+
Run on the host, after leaving the container shell. Confirm ownership before any
|
|
368
|
+
cleanup, then copy the recorded session, npm provenance metadata, and CLI version.
|
|
369
|
+
Keep the host evidence directory for the release record.
|
|
284
370
|
|
|
285
371
|
```bash
|
|
286
|
-
|
|
287
|
-
|
|
288
|
-
|
|
289
|
-
|
|
290
|
-
|
|
291
|
-
|
|
292
|
-
|
|
293
|
-
"
|
|
372
|
+
test "$(docker inspect --format '{{index .Config.Labels "org.softspark.release-smoke"}}' "$SMOKE_CONTAINER")" = "$SMOKE_CONTAINER" || {
|
|
373
|
+
echo "Container ownership does not match; refusing cleanup."
|
|
374
|
+
exit 1
|
|
375
|
+
}
|
|
376
|
+
docker logs "$SMOKE_CONTAINER" > "$EVIDENCE/container.log" 2>&1
|
|
377
|
+
mkdir -p "$EVIDENCE/container"
|
|
378
|
+
docker cp "$SMOKE_CONTAINER:/tmp/ai-toolkit-smoke/evidence/." "$EVIDENCE/container/" || {
|
|
379
|
+
echo "Evidence transfer failed; keep the container and investigate."
|
|
380
|
+
exit 1
|
|
381
|
+
}
|
|
382
|
+
test -f "$EVIDENCE/container/session.log" || {
|
|
383
|
+
echo "Session evidence is missing; refusing cleanup."
|
|
384
|
+
exit 1
|
|
385
|
+
}
|
|
386
|
+
docker stop "$SMOKE_CONTAINER"
|
|
387
|
+
docker rm "$SMOKE_CONTAINER"
|
|
388
|
+
printf 'Evidence retained: %s\n' "$EVIDENCE"
|
|
294
389
|
```
|
|
295
390
|
|
|
391
|
+
There are no host bind mounts or named volumes to delete. Never bypass a
|
|
392
|
+
destructive-command guard with Python, shutil.rmtree, another interpreter, or a
|
|
393
|
+
different deletion tool. If a guard rejects cleanup, leave the owned container
|
|
394
|
+
and evidence in place and report the rejection through the normal approval
|
|
395
|
+
mechanism. Do not prune Docker resources or delete unrelated temporary files.
|
|
396
|
+
|
|
296
397
|
## Success criteria
|
|
297
398
|
|
|
298
399
|
| Area | Criterion |
|
|
@@ -306,10 +407,10 @@ print(f'removed {sb} ({n} files)')
|
|
|
306
407
|
| Pack update | Current version silent; stale version updates and re-records |
|
|
307
408
|
| Pack removal | Zero residue in `~/.softspark` and `settings.json`; re-install works |
|
|
308
409
|
| Degraded path | Fetch failure is inert, loud in status, and recoverable |
|
|
309
|
-
| Isolation |
|
|
410
|
+
| Isolation | Host-side managed-settings fingerprints match; container has no host mounts or shared host namespaces |
|
|
310
411
|
|
|
311
412
|
## Related
|
|
312
413
|
|
|
313
414
|
- [Release Preparation](sop-release.md) — run before tagging
|
|
314
|
-
- [Release Verification](sop-release-verification.md) — the
|
|
415
|
+
- [Release Verification](sop-release-verification.md) — cross-editor checks of the isolated npm artifact
|
|
315
416
|
- [rtk-pack Retirement](../history/completed/rtk-pack-retirement-20260727.md) — what happened the one time this SOP was written and not run
|
|
@@ -3,9 +3,9 @@ title: "SOP: Release Verification"
|
|
|
3
3
|
category: procedures
|
|
4
4
|
service: ai-toolkit
|
|
5
5
|
tags: [sop, verification, release, smoke-test, install, update, qa, provenance, sarif, dsh]
|
|
6
|
-
version: "1.8.
|
|
6
|
+
version: "1.8.1"
|
|
7
7
|
created: "2026-04-08"
|
|
8
|
-
last_updated: "2026-09-
|
|
8
|
+
last_updated: "2026-09-06"
|
|
9
9
|
description: "End-to-end smoke test after installing or updating @softspark/ai-toolkit. Verifies CLI, native Codex and GitHub Copilot surfaces, explicit DSH lifecycle, Claude app export, doctor, validation, tests, eject, provenance, SARIF, and per-skill permissions."
|
|
10
10
|
---
|
|
11
11
|
|
|
@@ -23,12 +23,30 @@ Verifies all critical paths from the user's perspective.
|
|
|
23
23
|
|
|
24
24
|
**Prerequisites:**
|
|
25
25
|
- Node.js >= 18, Python 3, `bats`, git
|
|
26
|
-
- `@softspark/ai-toolkit` installed
|
|
26
|
+
- The target version of `@softspark/ai-toolkit` installed in a disposable test environment
|
|
27
27
|
|
|
28
28
|
**Time:** 10-15 minutes (full), 2 minutes (quick checklist)
|
|
29
29
|
|
|
30
30
|
---
|
|
31
31
|
|
|
32
|
+
## Verification environment
|
|
33
|
+
|
|
34
|
+
Run installation, update, eject and editor smoke checks in a disposable
|
|
35
|
+
container or VM with its own OS user and default home directory. Do not
|
|
36
|
+
reassign `HOME` or `CODEX_HOME`, mount the operator's home/authentication
|
|
37
|
+
directories, or alter the global installation used by an active session.
|
|
38
|
+
Use the packed release candidate before publication and the exact npm version
|
|
39
|
+
after publication; the installed package must be the source of runtime checks.
|
|
40
|
+
Source validation and test commands still run from the matching release checkout.
|
|
41
|
+
|
|
42
|
+
A scratch project alone does not isolate home-scoped writes. In particular,
|
|
43
|
+
the live Augment checks in Phase 9 write user settings. Execute them inside
|
|
44
|
+
the disposable environment, retain logs outside it, and remove only resources
|
|
45
|
+
created for this verification run. Do not treat a dry-run as proof that emitted
|
|
46
|
+
files parse or that a second install is idempotent.
|
|
47
|
+
|
|
48
|
+
---
|
|
49
|
+
|
|
32
50
|
## Quick Checklist (TL;DR)
|
|
33
51
|
|
|
34
52
|
The 14 core commands below must pass. Releases that change DSH must also complete Phase 10.
|
|
@@ -123,9 +141,11 @@ ai-toolkit status
|
|
|
123
141
|
```
|
|
124
142
|
|
|
125
143
|
**Verify `--dry-run`:**
|
|
126
|
-
- [ ]
|
|
127
|
-
|
|
128
|
-
|
|
144
|
+
- [ ] Agent and skill totals match the target package's `app/agents/` and
|
|
145
|
+
`app/skills/*/SKILL.md` inventory; compare with that release's validator
|
|
146
|
+
output and README badges, not a number copied from an older release
|
|
147
|
+
- [ ] Dry-run describes the planned hook merge; the isolated installed copy
|
|
148
|
+
contains the expected hooks in settings.json
|
|
129
149
|
- [ ] "Other AI Tools" lists documented global targets (with `--editors`): aider, antigravity, augment, cline, codex, copilot, cursor, gemini, opencode, roo, windsurf. Scope varies: Codex uses `$CODEX_HOME` (default `~/.codex`) plus `$HOME/.agents/skills`; Copilot uses `$COPILOT_HOME` (default `~/.copilot`); cursor has only `~/.cursor/hooks.json`; antigravity has the `~/.gemini/*/skills` pointer. Cursor and Antigravity rules remain project-only.
|
|
130
150
|
|
|
131
151
|
**Verify `status`:**
|
|
@@ -195,9 +215,12 @@ python3 scripts/audit_skills.py --ci
|
|
|
195
215
|
```
|
|
196
216
|
|
|
197
217
|
**Verify validate.py:**
|
|
198
|
-
- [ ]
|
|
199
|
-
|
|
200
|
-
- [ ]
|
|
218
|
+
- [ ] Agent, skill and Bats test totals match the current release inventory
|
|
219
|
+
and README badges, as checked by the validator's metadata contracts
|
|
220
|
+
- [ ] Hook events/scripts match `app/hooks.json` and the shipped hook files
|
|
221
|
+
- [ ] Every shipped plugin pack and KB document passes its validator; compare
|
|
222
|
+
inventory with the release checkout rather than requiring an obsolete
|
|
223
|
+
fixed minimum number of packs or documents
|
|
201
224
|
- [ ] `Errors: 0 | Warnings: 0` → `VALIDATION PASSED`
|
|
202
225
|
|
|
203
226
|
**Verify audit_skills.py:**
|
|
@@ -210,7 +233,7 @@ python3 scripts/audit_skills.py --ci
|
|
|
210
233
|
## Phase 6: Tests (3-5 min)
|
|
211
234
|
|
|
212
235
|
```bash
|
|
213
|
-
# Run ONCE, capture to file, then parse.
|
|
236
|
+
# Run ONCE, capture to file, then parse. Use the current release's Bats count;
|
|
214
237
|
# re-running it per check (tail / grep ok / grep not ok piped separately)
|
|
215
238
|
# wastes minutes every release. Always cache the output.
|
|
216
239
|
npm test > /tmp/npm-test.log 2>&1
|
|
@@ -223,7 +246,7 @@ echo "exit: $exit"
|
|
|
223
246
|
|
|
224
247
|
**Verify:**
|
|
225
248
|
- [ ] `exit == 0`
|
|
226
|
-
- [ ] `ok == expected test count`
|
|
249
|
+
- [ ] `ok == expected test count` from the current release's metadata contracts
|
|
227
250
|
- [ ] `not ok == 0`
|
|
228
251
|
- [ ] Bats runs tests in parallel (4 jobs)
|
|
229
252
|
- [ ] Groups: agents, autodetect, cli, generators, guards, hooks, inject,
|
|
@@ -334,7 +357,7 @@ AI_TOOLKIT_STRICT_PIN=1 ai-toolkit update --dry-run
|
|
|
334
357
|
|
|
335
358
|
These verify the native-surface generators shipped in v3.0.0 actually emit the right files for the right profiles, and that the tool registry stays in sync with shipped generators.
|
|
336
359
|
|
|
337
|
-
>
|
|
360
|
+
> Run this phase inside the disposable verification environment. `--profile full` with `augment` writes to `$HOME/.augment/settings.json`, so a temporary project on the operator's machine is insufficient. Dry-run checks cover the planned paths; Phases 9.4 and 9.5 must also exercise actual writes in isolation.
|
|
338
361
|
|
|
339
362
|
### 9.1 `--profile full` emits every native surface
|
|
340
363
|
|
|
@@ -411,7 +434,9 @@ The bats suite validates JSON shape at generation time. This re-checks that what
|
|
|
411
434
|
D=/tmp/aitk-json-${RANDOM} && mkdir -p "$D" && cd "$D" && git init -q
|
|
412
435
|
ai-toolkit install --local --editors cursor,windsurf,gemini,augment,codex,copilot --profile full >/dev/null 2>&1
|
|
413
436
|
for f in .cursor/hooks.json .devin/hooks.v1.json .gemini/settings.json .codex/hooks.json .github/hooks/ai-toolkit.json "$HOME/.augment/settings.json"; do
|
|
414
|
-
[ -f "$f" ]
|
|
437
|
+
[ -f "$f" ] || { echo "MISSING: $f"; exit 1; }
|
|
438
|
+
python3 -c 'import json, sys; json.load(open(sys.argv[1]))' "$f" || exit 1
|
|
439
|
+
echo "OK: $f"
|
|
415
440
|
done
|
|
416
441
|
```
|
|
417
442
|
|
|
@@ -3,9 +3,9 @@ title: "SOP: Release Preparation"
|
|
|
3
3
|
category: procedures
|
|
4
4
|
service: ai-toolkit
|
|
5
5
|
tags: [sop, release, version, publish, changelog, semver, provenance, sarif, ecosystem, shellcheck]
|
|
6
|
-
version: "1.15.
|
|
6
|
+
version: "1.15.1"
|
|
7
7
|
created: "2026-04-10"
|
|
8
|
-
last_updated: "2026-09-
|
|
8
|
+
last_updated: "2026-09-06"
|
|
9
9
|
description: "Step-by-step checklist for preparing a new ai-toolkit release — ecosystem-sync drift check, version sync, changelog, artifact regeneration, validation, branch CI, and tagging. Run BEFORE every git tag. Includes mandatory Provenance, SARIF, checksum-pin, ShellCheck, licensing, exact-tag assertions, and a green Ubuntu/macOS branch-CI gate before any release tag is created."
|
|
10
10
|
---
|
|
11
11
|
|
|
@@ -15,6 +15,13 @@ Complete checklist for preparing a new `@softspark/ai-toolkit` release.
|
|
|
15
15
|
Run this **before** tagging. After tagging and publishing, run the
|
|
16
16
|
[Release Verification SOP](sop-release-verification.md) to smoke-test.
|
|
17
17
|
|
|
18
|
+
Installation smoke uses a disposable container or VM with its own default
|
|
19
|
+
home directory, as described in the verification SOP. Test the packed release
|
|
20
|
+
candidate before publishing and the exact npm version afterward. Keep the
|
|
21
|
+
operator's installed toolkit, editor settings and authentication directories
|
|
22
|
+
outside that environment. Compare component counts with the current release
|
|
23
|
+
inventory and validator output instead of historical constants in a checklist.
|
|
24
|
+
|
|
18
25
|
**Pipeline:**
|
|
19
26
|
```
|
|
20
27
|
Ecosystem Sync SOP (drift check + generator updates)
|
|
@@ -3,9 +3,9 @@ title: "AI Toolkit - Architecture Overview"
|
|
|
3
3
|
category: reference
|
|
4
4
|
service: ai-toolkit
|
|
5
5
|
tags: [architecture, overview, design, structure]
|
|
6
|
-
version: "1.10.
|
|
6
|
+
version: "1.10.1"
|
|
7
7
|
created: "2026-03-23"
|
|
8
|
-
last_updated: "2026-09-
|
|
8
|
+
last_updated: "2026-09-06"
|
|
9
9
|
description: "Architecture of ai-toolkit: install ownership, runtime adapters, the explicit DSH target, skill tiers, and project integration."
|
|
10
10
|
---
|
|
11
11
|
|
|
@@ -315,7 +315,7 @@ Agents (code-reviewer, debugger, devops-implementer, ...)
|
|
|
315
315
|
|
|
316
316
|
## Quality Hooks
|
|
317
317
|
|
|
318
|
-
|
|
318
|
+
28 entries across 14 lifecycle events. See [hooks-catalog.md](hooks-catalog.md) for full details.
|
|
319
319
|
|
|
320
320
|
| Hook | Trigger | Script | Action |
|
|
321
321
|
|------|---------|--------|--------|
|
|
@@ -5,7 +5,7 @@ service: ai-toolkit
|
|
|
5
5
|
tags: [rules, languages, coding-style, testing, patterns, security]
|
|
6
6
|
version: "2.2.0"
|
|
7
7
|
created: "2026-04-07"
|
|
8
|
-
last_updated: "2026-09-
|
|
8
|
+
last_updated: "2026-09-06"
|
|
9
9
|
description: "Reference for the language-specific rules system: 13 per-language rule sets shipped as knowledge skills, plus common rules installed as Claude Code path-scoped project rules."
|
|
10
10
|
---
|
|
11
11
|
|
|
@@ -78,6 +78,12 @@ app/rules/
|
|
|
78
78
|
|
|
79
79
|
## Rule Categories
|
|
80
80
|
|
|
81
|
+
The common security rules distinguish safe, actionable failure messages from
|
|
82
|
+
private diagnostics. Common testing rules cover API error contracts and prohibit
|
|
83
|
+
overlapping runners that reset a shared database. The `api-patterns` skill carries
|
|
84
|
+
the focused error-contract guidance; the `review` checklist checks the same
|
|
85
|
+
failure boundaries. These are content rules, not new hooks or runtime permissions.
|
|
86
|
+
|
|
81
87
|
| Category | Filename | Content |
|
|
82
88
|
|----------|----------|---------|
|
|
83
89
|
| `coding-style` | `coding-style.md` | Naming, formatting, idiomatic constructs, linter config |
|
|
@@ -5,7 +5,7 @@ service: ai-toolkit
|
|
|
5
5
|
tags: [skills, domain-knowledge, catalog, task-skills, hybrid-skills]
|
|
6
6
|
version: "1.5.0"
|
|
7
7
|
created: "2026-03-23"
|
|
8
|
-
last_updated: "2026-
|
|
8
|
+
last_updated: "2026-09-06"
|
|
9
9
|
description: "Complete skills catalog with task, hybrid, and knowledge skills. Includes Codex adaptation notes, effort levels, skill-scoped hooks, executable scripts, security auditor, and persona presets."
|
|
10
10
|
---
|
|
11
11
|
|
|
@@ -123,7 +123,7 @@ Hybrid skills combine slash-command invocation with domain knowledge that agents
|
|
|
123
123
|
| Skill | Directory | Domain |
|
|
124
124
|
|-------|-----------|--------|
|
|
125
125
|
| **app-builder** | `skills/app-builder/` | Full-stack application architecture |
|
|
126
|
-
| **api-patterns** | `skills/api-patterns/` |
|
|
126
|
+
| **api-patterns** | `skills/api-patterns/` | API design, versioning, actionable error contracts and safe retries; focused `reference/error-contracts.md` |
|
|
127
127
|
| **database-patterns** | `skills/database-patterns/` | Schema design, indexing, query optimization |
|
|
128
128
|
| **flutter-patterns** | `skills/flutter-patterns/` | Flutter/Dart architecture, state management |
|
|
129
129
|
| **ecommerce-patterns** | `skills/ecommerce-patterns/` | E-commerce: catalog, cart, checkout, payments |
|