@softspark/ai-toolkit 4.22.1 → 4.23.1

package/CHANGELOG.md

@@ -7,6 +7,71 @@ Versioning follows [Semantic Versioning](https://semver.org/).
 
 ---
 
+## v4.23.1 — a pack could ship an agent nobody installed (2026-08-18)
+
+### Fixed
+
+- **`plugin install` never linked an agent the pack shipped itself.**
+  `_install_claude_agents` resolved `includes.agents` only against the toolkit's
+  own `app/agents/<name>.md`, so a self-contained pack got `WARN agent not found`
+  and the agent silently stayed uninstalled. The asymmetry was visible inside the
+  same file: skills already resolved through `_resolve_skill_source`, which falls
+  back to the pack directory, and `_remove_claude_pack_links` already unlinked
+  agent symlinks pointing into a pack — a removal path for links the installer
+  could never create.
+
+  Agents now go through `_resolve_agent_source`, mirroring skills: a core agent in
+  `app/agents` still wins, otherwise `<pack>/agents/<name>.md` is linked. Packs
+  that merely reference a core agent behave exactly as before.
+
+- **`validate.py` reported self-contained packs as broken.** `validate_references`
+  checked `app/agents` and `app/skills` only, so a pack shipping its own assets
+  drew `References missing agent: X` and `References missing skill: Y` even though
+  the installer handled the skill correctly. It now also accepts
+  `<pack>/agents/<name>.md` and `<pack>/skills/<name>/SKILL.md`; a genuinely
+  missing reference still errors. `pack_dir` is an optional argument, so existing
+  callers are unaffected.
+
+  Found by the post-release smoke test of a downstream pack (`legal-pl-pack`),
+  not by a user report.
+
+### Added
+
+- **Three tests covering the class.** `tests/test_plugin.bats` builds a throwaway
+  toolkit whose only pack ships its own agent and skill, then asserts the agent is
+  linked into the pack (not `app/agents`), that removal drops the link, and that
+  reference validation accepts pack-shipped assets while still failing on a
+  genuinely missing one.
+
+## v4.23.0 — a compliance scanner that says nothing must say why (2026-08-06)
+
+### Fixed
+
+- **`hipaa-validate` scanned a manifest-less project silently clean.** Category
+  1, 3, 4 and 7 patterns are language-tagged and fire only for a detected
+  language, and detection read manifest files only. A directory of Python with
+  PHI in its loggers therefore reported `HIGH: 0` while every Python rule sat
+  unused — the worst possible output for a compliance tool, because zero reads
+  as compliant. Found by the post-release SOP on v4.22.1.
+
+  Detection now falls back to the extensions of the files being scanned when no
+  manifest is present. **Manifests still win**, so a project that already
+  declared itself keeps exactly the behaviour it had.
+
+### Changed
+
+- **The scan summary reports how the language was decided.** A new
+  `language_detection` field says `manifest`, `file extensions (no manifest
+  found)`, or `none`. When it is `none` the text report writes to stderr that
+  language-tagged patterns did not run and that a zero-finding result means
+  unscanned rather than compliant.
+
+  **Upgrade note:** a project scanned without a manifest may now report HIGH
+  findings where v4.22.x reported none. Those findings were always there. A CI
+  job gating on the exit code can start failing on real PHI exposure.
+
+---
+
 ## v4.22.1 — nine skills, not four, could not find their own scripts (2026-08-06)
 
 v4.22.0 claimed to fix four skills whose documented script path never resolved.

package/README.md

@@ -6,41 +6,18 @@
 [![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)
 [![Skills](https://img.shields.io/badge/skills-109-brightgreen)](app/skills/)
 [![Agents](https://img.shields.io/badge/agents-44-blue)](app/agents/)
-[![Tests](https://img.shields.io/badge/tests-1513%20passing-success)](tests/)
-
-## What's New in v4.22.1
-
-**v4.22.1** — nine skills, not four, could not find their own scripts. v4.22.0
-grepped for one spelling of the bug and missed five more, a `.py` file run
-through `bash`, and a script that crashed on `--help`. `validate.py` now fails
-the build on the entire class, and the post-release SOP runs every skill's
-documented command from the installed copy — which is how these were found.
-
-## What's New in v4.22.0
-
-v4.22.0 turns compatibility promises into tests, and makes two validator skills
-actually run the scanners they ship.
-
-- **The public surface is checked, not documented.** `app/surface.json` pins 275
-  entries — skills, agents, frontmatter fields, CLI commands, hook scripts and
-  events, KB categories, plugin packs. Removing any of them fails `npm test`.
-  Adding is free: a surface nobody has installed has no users to break.
-  `BACKWARD_COMPATIBILITY.md` says which surfaces are load-bearing and how to
-  deprecate one properly.
-- **`a11y-validate` and `seo-validate` shipped scanners that nothing invoked.**
-  1,196 lines of working Python sat on disk while the model was told to grep the
-  pattern tables by hand. Both now run their script for a deterministic baseline,
-  then take an explicit manual pass — with a measured coverage table saying which
-  criteria the script handles and which are yours.
-- **`/brainstorm`** — the first planning skill allowed to conclude "do not build
-  this". Every other one produces an artifact; a pipeline that cannot say no is a
-  yes-machine.
-- **Skill bodies have a budget.** Error above 20,000 bytes, warn above 18,000,
-  headroom printed every run. Three validator skills were split into
-  `reference/`, taking 47 KB off the hot path, with `scripts/check_split.py`
-  proving nothing was lost.
-- **Four skills invoked their scripts through a path that never resolved.**
-  `$(dirname "$0")` expands to the shell's directory, not the skill's.
+[![Tests](https://img.shields.io/badge/tests-1519%20passing-success)](tests/)
+
+## What's New in v4.23.1
+
+**v4.23.1** — a plugin pack that shipped its own agent never got it installed.
+`plugin install` resolved `includes.agents` against the toolkit's own
+`app/agents/`, printed `WARN agent not found`, and moved on — while the removal
+path already knew how to unlink a pack-owned agent, and skills had resolved from
+the pack directory all along. `validate.py` had the same blind spot for both
+agents and skills, so a self-contained pack was reported broken twice over.
+Agents now resolve from the pack first-class, core assets still win, and
+validation accepts either. Found by a downstream pack's post-release smoke test.
 
 See [CHANGELOG.md](CHANGELOG.md) for full history.
 

package/app/.claude-plugin/plugin.json

@@ -3,7 +3,7 @@
   "name": "ai-toolkit",
   "displayName": "AI Toolkit",
   "description": "Professional-grade engineering skills, agents, rules, and lifecycle guardrails for Claude Code, Claude Chat, and Cowork.",
-  "version": "4.22.1",
+  "version": "4.23.1",
   "author": {
     "name": "SoftSpark",
     "url": "https://github.com/softspark"

package/app/skills/hipaa-validate/SKILL.md

@@ -161,6 +161,9 @@ This distinction helps compliance officers prioritize immediate remediation (def
 
 ## Gotchas
 
+- **Check `language_detection` in the summary before trusting a zero.** Category 1, 3, 4 and 7 patterns are language-tagged and only fire for a detected language. Manifests decide first (`pyproject.toml`, `package.json`, `go.mod`, …); without one the scanner falls back to file extensions. If it reports `languages: ["any"]` with `language_detection: "none"`, the language rules never ran and `HIGH: 0` means *unscanned*, not *compliant* — the scanner prints that warning to stderr, so a run whose stderr is discarded loses it.
+- Scanning a monorepo package or a subdirectory can put you below the manifest. The extension fallback covers the common case, but a directory of `.sql`, `.yaml` or templates resolves to no language at all — scan from the level that holds the manifest.
+
 - Test fixtures and seed data often contain **synthetic** PHI that looks real (SSN-shaped IDs, formatted phone numbers, sample email addresses). Flag them but lower severity — production code handling the same patterns is the actual risk.
 - HIPAA §164.312(b) requires audit logging but does not specify a format. "Logs exist" is not evidence of compliance — the logs must capture WHO (authenticated user), WHAT (action), WHEN (timestamp), WHERE (resource), and they must be immutable (append-only or write-once storage).
 - Encryption-at-rest varies silently by storage layer. RDS auto-encrypts new volumes since 2017, but older DB snapshots may not be; S3 bucket policies can override instance-level encryption. Treat "encryption enabled" as a claim to verify with the cloud provider, not a state to trust.

package/app/skills/hipaa-validate/scripts/hipaa_scan.py

@@ -302,8 +302,24 @@ def context_gate(files: list[Path], keywords: list[str],
 # Language detection (Step 1)
 # ---------------------------------------------------------------------------
 
-def detect_languages(root: Path) -> set[str]:
-    """Detect project languages from manifest files."""
+def detect_languages(root: Path, files: list[Path] | None = None) -> tuple[set[str], str]:
+    """Detect project languages. Returns (languages, how_they_were_detected).
+
+    Manifest files decide when present. When none is found, fall back to the
+    extensions of the files actually being scanned.
+
+    The fallback exists because language-tagged patterns fire only when the
+    project language includes the tag, so a manifest-less directory of Python
+    scanned clean while every Python rule sat unused — and the report said
+    `HIGH: 0`, which reads as compliant rather than unscanned. Silence is the
+    worst possible output for a compliance tool. A monorepo package, a scan
+    pointed at a subdirectory, or a service whose manifest lives one level up
+    all hit this.
+
+    Manifests still win: a project that declares itself keeps exactly the
+    behaviour it had, so upgrading changes nothing for anyone who was already
+    being scanned properly.
+    """
     langs = set()
     for indicator, lang in LANGUAGE_INDICATORS.items():
         if "*" in indicator:
@@ -314,7 +330,20 @@ def detect_languages(root: Path) -> set[str]:
         else:
             if (root / indicator).exists():
                 langs.add(lang)
-    return langs or {"any"}
+    if langs:
+        return langs, "manifest"
+
+    if files:
+        ext_to_lang: dict[str, set[str]] = {}
+        for lang, exts in LANG_EXTENSIONS.items():
+            for ext in exts:
+                ext_to_lang.setdefault(ext, set()).add(lang)
+        for f in files:
+            langs |= ext_to_lang.get(f.suffix, set())
+        if langs:
+            return langs, "file extensions (no manifest found)"
+
+    return {"any"}, "none — language-tagged patterns were skipped"
 
 
 # ---------------------------------------------------------------------------
@@ -616,7 +645,8 @@ def format_text_report(findings: list[dict], mode: str,
 
 def format_json_report(findings: list[dict], mode: str,
                        phi_count: int, scanned_count: int,
-                       categories_run: list[int], languages: list[str]) -> str:
+                       categories_run: list[int], languages: list[str],
+                       language_source: str = "manifest") -> str:
     """Format findings as structured JSON for CI integration."""
     high_count = sum(1 for f in findings if f["severity"] == "HIGH")
     warn_count = sum(1 for f in findings if f["severity"] == "WARN")
@@ -628,6 +658,7 @@ def format_json_report(findings: list[dict], mode: str,
             "files_scanned": scanned_count,
             "categories_run": sorted(categories_run),
             "languages": sorted(languages),
+            "language_detection": language_source,
             "high": high_count,
             "warn": warn_count,
             "total": len(findings),
@@ -723,8 +754,9 @@ def main() -> None:
               file=sys.stderr)
         print("Consider narrowing the scan path for targeted results.", file=sys.stderr)
 
-    # Step 1: Detect languages
-    languages = detect_languages(root)
+    # Step 1: Detect languages. Categories 1-2 scan everything, so the extension
+    # fallback looks at all collected files, not just the PHI-adjacent subset.
+    languages, lang_source = detect_languages(root, all_files)
 
     # Step 2: Run categories
     findings: list[dict] = []
@@ -789,10 +821,18 @@ def main() -> None:
     if args.output == "json":
         print(format_json_report(findings, args.mode, phi_count,
                                  scanned_count, categories_run,
-                                 sorted(languages)))
+                                 sorted(languages), lang_source))
     else:
         print(format_text_report(findings, args.mode, phi_count,
                                  scanned_count, categories_run))
+        if languages == {"any"}:
+            # Never let an unscanned run read as a clean one.
+            print("\nWARNING: no project language identified "
+                  f"({lang_source}).", file=sys.stderr)
+            print("Language-tagged patterns (Python, JS/TS, Go, Java, Ruby, C#) "
+                  "did NOT run.", file=sys.stderr)
+            print("A zero-finding result here means unscanned, not compliant. "
+                  "Scan from the directory holding the manifest.", file=sys.stderr)
 
     # Exit code: non-zero if HIGH findings
     high_count = sum(1 for f in findings if f["severity"] == "HIGH")

package/kb/procedures/maintenance-sop.md

@@ -3,9 +3,9 @@ title: "SOP: AI Toolkit Maintenance"
 category: procedures
 service: ai-toolkit
 tags: [sop, maintenance, agents, skills, install]
-version: "3.3.0"
+version: "3.4.0"
 created: "2026-03-23"
-last_updated: "2026-07-14"
+last_updated: "2026-08-06"
 description: "Standard operating procedures for installing, maintaining, and evolving the ai-toolkit."
 ---
 
@@ -209,8 +209,30 @@ that runtime should receive the change.
    user-invocable: false            # knowledge skill
    ---
    ```
-2. Update `kb/reference/skills-catalog.md` and `app/ARCHITECTURE.md`
-3. Run `scripts/validate.py`
+2. If the skill ships an executable, put it in `app/skills/<skill-name>/scripts/`
+   and invoke it **only** as:
+   ```bash
+   python3 ${CLAUDE_SKILL_DIR}/scripts/<name>.py [args]
+   ```
+   `${CLAUDE_SKILL_DIR}` is the sole path that resolves once the skill is
+   installed. `scripts/<name>.py` resolves against the user's working directory,
+   `app/skills/<name>/scripts/…` against a repo they do not have, and
+   `$(dirname "$0")` against the shell's directory — all three look right in the
+   source and fail for every user. Nine skills shipped with one of them before
+   `validate.py` started failing the build on it.
+
+   Match the interpreter to the file (`python3` for `.py`, never `bash`, never
+   bare `python`). The frontmatter `scripts:` list stays relative — it declares
+   ownership rather than running anything.
+
+   Add `--help` handling. A script that treats `--help` as a positional argument
+   greets the user with a traceback, and one that reads stdin must answer an
+   empty stdin with an error rather than blocking forever.
+3. Update `kb/reference/skills-catalog.md` and `app/ARCHITECTURE.md`
+4. Run `scripts/validate.py` — it checks the invocation, the body budget, and
+   `reference/` link resolution
+5. Run `python3 scripts/surface_manifest.py` before the next release to adopt the
+   new skill into the protected surface
 
 ## Adding a New Hook
 

package/kb/reference/architecture-overview.md

@@ -3,7 +3,7 @@ title: "AI Toolkit - Architecture Overview"
 category: reference
 service: ai-toolkit
 tags: [architecture, overview, design, structure]
-version: "1.6.1"
+version: "1.7.0"
 created: "2026-03-23"
 last_updated: "2026-08-06"
 description: "Architecture of ai-toolkit: directory layout, Claude app export, global install model, editor-aware MCP install, Codex translation layer, skill tiers, and integration with projects."
@@ -50,7 +50,10 @@ ai-toolkit/
     inject_section_cli.py    # Marker-based content injection (canonical implementation)
     _common.py               # Shared helper for generators (frontmatter, agents/skills emission)
     merge-hooks.py           # JSON merge for hooks into settings.json (inject/strip modes)
-    validate.py              # Toolkit integrity check
+    validate.py              # Toolkit integrity check (+ skill body budget, script-invocation gate)
+    surface_manifest.py      # Public-surface snapshot vs app/surface.json; removals fail the build
+    check_split.py           # Split gate: proves a SKILL.md -> reference/ refactor lost nothing
+    sync_badges.py           # Derives README count badges from the tree (runs inside generate:all)
     evaluate_skills.py       # Skill quality report
     generate_agents_md.py    # Regenerates AGENTS.md
     generate_cursor_rules.py # Generates .cursorrules (sources _common.py)

package/kb/reference/skill-templates.md

@@ -3,10 +3,10 @@ title: "AI Toolkit - Skill Templates"
 category: reference
 service: ai-toolkit
 tags: [templates, scaffolding, create, skills]
-version: "1.0.0"
+version: "1.1.0"
 created: "2026-03-29"
-last_updated: "2026-04-01"
-description: "5 skill templates for scaffolding new skills: linter, reviewer, generator, workflow, knowledge."
+last_updated: "2026-08-06"
+description: "5 skill templates for scaffolding new skills: linter, reviewer, generator, workflow, knowledge. Includes the ${CLAUDE_SKILL_DIR} rule for shipping executable scripts."
 ---
 
 # Skill Templates
@@ -47,4 +47,37 @@ Templates are stored in `app/templates/skill/{type}/SKILL.md.template`.
 
 1. Edit the generated `app/skills/{name}/SKILL.md`
 2. Add `reference/` or `templates/` subdirectories if needed
-3. Run `ai-toolkit validate` to verify
+3. If the skill ships an executable, invoke it **only** through
+   `${CLAUDE_SKILL_DIR}` — see [Shipping a script](#shipping-a-script)
+4. Run `ai-toolkit validate` to verify
+
+## Shipping a script
+
+Put executables in `app/skills/{name}/scripts/` and document them as:
+
+```bash
+python3 ${CLAUDE_SKILL_DIR}/scripts/{name}.py [args]
+```
+
+That is the only path that resolves once a skill is installed and symlinked into
+`~/.claude/skills/`. Three other spellings look correct in the repo and fail for
+every user:
+
+| Spelling | Resolves against | Result after install |
+|----------|------------------|----------------------|
+| `scripts/foo.py` | the user's working directory | not found |
+| `app/skills/{name}/scripts/foo.py` | a repo the user does not have | not found |
+| `$(dirname "$0")/scripts/foo.py` | the shell's directory | not found |
+
+Nine skills shipped with one of these before `validate.py` began failing the
+build on it. The check also enforces that the interpreter matches the file — a
+`.py` run through `bash` is an error, and bare `python` a warning, because it is
+missing or Python 2 on many systems.
+
+The frontmatter `scripts:` list stays relative. It declares which files the skill
+owns; it does not run them.
+
+**Give the script `--help`.** A script that treats `--help` as a positional
+argument answers with a traceback, and a stdin filter with no input must return
+an error rather than blocking forever — the post-release SOP probes every shipped
+script exactly this way.

package/kb/reference/unique-features.md

@@ -4,7 +4,7 @@ category: reference
 service: ai-toolkit
 tags: [features, differentiators, constitution, hooks, security, tdd, memory]
 created: "2026-04-13"
-last_updated: "2026-07-10"
+last_updated: "2026-08-06"
 description: "Detailed description of ai-toolkit's unique features: constitution enforcement, hooks system, security scanning, effort budgeting, quality gates, and more."
 ---
 
@@ -109,6 +109,29 @@ The `Stop` hook runs after every response across 5 languages:
 | Dart | dart analyze | dart analyze |
 | Go | go vet | go vet |
 
+## 5b. Repo-Integrity Gates (`validate.py` + `npm test`)
+
+Four gates that fail the build rather than documenting a rule and hoping. Each was
+added after the thing it checks had already shipped broken.
+
+| Gate | What it fails on | Why prose was not enough |
+|------|------------------|--------------------------|
+| **Public surface** (`surface_manifest.py` + `app/surface.json`) | a skill, agent, CLI command, frontmatter field, hook, KB category or pack disappearing | `BACKWARD_COMPATIBILITY.md` listed the surfaces; renaming one still left every check green |
+| **Script invocation** (`validate.py`) | a skill running its own script by any path other than `${CLAUDE_SKILL_DIR}`, or through the wrong interpreter | nine skills shipped invocations that resolved only in the repo, never on an installed machine |
+| **Skill body budget** (`validate.py`) | a `SKILL.md` body over 20,000 bytes; warns over 18,000 | the body loads on every trigger match, including accidental ones; three skills sat at 24–35 KB |
+| **Split integrity** (`check_split.py`) | a body → `reference/` refactor that lost a fenced code line, a required section, the description, or a working link | run by hand during a split; caught a heading corrupted inside a fenced example on first use |
+
+The surface check is deliberately one-directional: **removals fail, additions pass.**
+A surface nobody has installed has no users to break, and letting additions through
+silently is what stops the gate from becoming a tax people learn to bypass. The
+manifest is never auto-regenerated — if `generate:all` rewrote it, deleting a skill
+would delete its entry in the same breath and the check would prove nothing.
+
+`sync_badges.py` closes the matching hole in the other direction: README count
+badges are derived from the tree inside `generate:all`, before `validate.py --strict`
+reads them, so adding a test can no longer redden the build until someone edits a
+number by hand.
+
 ## 6. Iron Law Enforcement
 
 Three skills enforce non-negotiable quality gates with anti-rationalization tables:

package/llms-full.txt

@@ -6527,9 +6527,9 @@ title: "SOP: AI Toolkit Maintenance"
 category: procedures
 service: ai-toolkit
 tags: [sop, maintenance, agents, skills, install]
-version: "3.3.0"
+version: "3.4.0"
 created: "2026-03-23"
-last_updated: "2026-07-14"
+last_updated: "2026-08-06"
 description: "Standard operating procedures for installing, maintaining, and evolving the ai-toolkit."
 ---
 
@@ -6733,8 +6733,30 @@ that runtime should receive the change.
    user-invocable: false            # knowledge skill
    ---
    ```
-2. Update `kb/reference/skills-catalog.md` and `app/ARCHITECTURE.md`
-3. Run `scripts/validate.py`
+2. If the skill ships an executable, put it in `app/skills/<skill-name>/scripts/`
+   and invoke it **only** as:
+   ```bash
+   python3 ${CLAUDE_SKILL_DIR}/scripts/<name>.py [args]
+   ```
+   `${CLAUDE_SKILL_DIR}` is the sole path that resolves once the skill is
+   installed. `scripts/<name>.py` resolves against the user's working directory,
+   `app/skills/<name>/scripts/…` against a repo they do not have, and
+   `$(dirname "$0")` against the shell's directory — all three look right in the
+   source and fail for every user. Nine skills shipped with one of them before
+   `validate.py` started failing the build on it.
+
+   Match the interpreter to the file (`python3` for `.py`, never `bash`, never
+   bare `python`). The frontmatter `scripts:` list stays relative — it declares
+   ownership rather than running anything.
+
+   Add `--help` handling. A script that treats `--help` as a positional argument
+   greets the user with a traceback, and one that reads stdin must answer an
+   empty stdin with an error rather than blocking forever.
+3. Update `kb/reference/skills-catalog.md` and `app/ARCHITECTURE.md`
+4. Run `scripts/validate.py` — it checks the invocation, the body budget, and
+   `reference/` link resolution
+5. Run `python3 scripts/surface_manifest.py` before the next release to adopt the
+   new skill into the protected surface
 
 ## Adding a New Hook
 
@@ -8783,7 +8805,7 @@ title: "AI Toolkit - Architecture Overview"
 category: reference
 service: ai-toolkit
 tags: [architecture, overview, design, structure]
-version: "1.6.1"
+version: "1.7.0"
 created: "2026-03-23"
 last_updated: "2026-08-06"
 description: "Architecture of ai-toolkit: directory layout, Claude app export, global install model, editor-aware MCP install, Codex translation layer, skill tiers, and integration with projects."
@@ -8830,7 +8852,10 @@ ai-toolkit/
     inject_section_cli.py    # Marker-based content injection (canonical implementation)
     _common.py               # Shared helper for generators (frontmatter, agents/skills emission)
     merge-hooks.py           # JSON merge for hooks into settings.json (inject/strip modes)
-    validate.py              # Toolkit integrity check
+    validate.py              # Toolkit integrity check (+ skill body budget, script-invocation gate)
+    surface_manifest.py      # Public-surface snapshot vs app/surface.json; removals fail the build
+    check_split.py           # Split gate: proves a SKILL.md -> reference/ refactor lost nothing
+    sync_badges.py           # Derives README count badges from the tree (runs inside generate:all)
     evaluate_skills.py       # Skill quality report
     generate_agents_md.py    # Regenerates AGENTS.md
     generate_cursor_rules.py # Generates .cursorrules (sources _common.py)
@@ -14588,10 +14613,10 @@ title: "AI Toolkit - Skill Templates"
 category: reference
 service: ai-toolkit
 tags: [templates, scaffolding, create, skills]
-version: "1.0.0"
+version: "1.1.0"
 created: "2026-03-29"
-last_updated: "2026-04-01"
-description: "5 skill templates for scaffolding new skills: linter, reviewer, generator, workflow, knowledge."
+last_updated: "2026-08-06"
+description: "5 skill templates for scaffolding new skills: linter, reviewer, generator, workflow, knowledge. Includes the ${CLAUDE_SKILL_DIR} rule for shipping executable scripts."
 ---
 
 # Skill Templates
@@ -14632,7 +14657,40 @@ Templates are stored in `app/templates/skill/{type}/SKILL.md.template`.
 
 1. Edit the generated `app/skills/{name}/SKILL.md`
 2. Add `reference/` or `templates/` subdirectories if needed
-3. Run `ai-toolkit validate` to verify
+3. If the skill ships an executable, invoke it **only** through
+   `${CLAUDE_SKILL_DIR}` — see [Shipping a script](#shipping-a-script)
+4. Run `ai-toolkit validate` to verify
+
+## Shipping a script
+
+Put executables in `app/skills/{name}/scripts/` and document them as:
+
+```bash
+python3 ${CLAUDE_SKILL_DIR}/scripts/{name}.py [args]
+```
+
+That is the only path that resolves once a skill is installed and symlinked into
+`~/.claude/skills/`. Three other spellings look correct in the repo and fail for
+every user:
+
+| Spelling | Resolves against | Result after install |
+|----------|------------------|----------------------|
+| `scripts/foo.py` | the user's working directory | not found |
+| `app/skills/{name}/scripts/foo.py` | a repo the user does not have | not found |
+| `$(dirname "$0")/scripts/foo.py` | the shell's directory | not found |
+
+Nine skills shipped with one of these before `validate.py` began failing the
+build on it. The check also enforces that the interpreter matches the file — a
+`.py` run through `bash` is an error, and bare `python` a warning, because it is
+missing or Python 2 on many systems.
+
+The frontmatter `scripts:` list stays relative. It declares which files the skill
+owns; it does not run them.
+
+**Give the script `--help`.** A script that treats `--help` as a positional
+argument answers with a traceback, and a stdin filter with no input must return
+an error rather than blocking forever — the post-release SOP probes every shipped
+script exactly this way.
 
 ---
 
@@ -15457,7 +15515,7 @@ category: reference
 service: ai-toolkit
 tags: [features, differentiators, constitution, hooks, security, tdd, memory]
 created: "2026-04-13"
-last_updated: "2026-07-10"
+last_updated: "2026-08-06"
 description: "Detailed description of ai-toolkit's unique features: constitution enforcement, hooks system, security scanning, effort budgeting, quality gates, and more."
 ---
 
@@ -15562,6 +15620,29 @@ The `Stop` hook runs after every response across 5 languages:
 | Dart | dart analyze | dart analyze |
 | Go | go vet | go vet |
 
+## 5b. Repo-Integrity Gates (`validate.py` + `npm test`)
+
+Four gates that fail the build rather than documenting a rule and hoping. Each was
+added after the thing it checks had already shipped broken.
+
+| Gate | What it fails on | Why prose was not enough |
+|------|------------------|--------------------------|
+| **Public surface** (`surface_manifest.py` + `app/surface.json`) | a skill, agent, CLI command, frontmatter field, hook, KB category or pack disappearing | `BACKWARD_COMPATIBILITY.md` listed the surfaces; renaming one still left every check green |
+| **Script invocation** (`validate.py`) | a skill running its own script by any path other than `${CLAUDE_SKILL_DIR}`, or through the wrong interpreter | nine skills shipped invocations that resolved only in the repo, never on an installed machine |
+| **Skill body budget** (`validate.py`) | a `SKILL.md` body over 20,000 bytes; warns over 18,000 | the body loads on every trigger match, including accidental ones; three skills sat at 24–35 KB |
+| **Split integrity** (`check_split.py`) | a body → `reference/` refactor that lost a fenced code line, a required section, the description, or a working link | run by hand during a split; caught a heading corrupted inside a fenced example on first use |
+
+The surface check is deliberately one-directional: **removals fail, additions pass.**
+A surface nobody has installed has no users to break, and letting additions through
+silently is what stops the gate from becoming a tax people learn to bypass. The
+manifest is never auto-regenerated — if `generate:all` rewrote it, deleting a skill
+would delete its entry in the same breath and the check would prove nothing.
+
+`sync_badges.py` closes the matching hole in the other direction: README count
+badges are derived from the tree inside `generate:all`, before `validate.py --strict`
+reads them, so adding a test can no longer redden the build until someone edits a
+number by hand.
+
 ## 6. Iron Law Enforcement
 
 Three skills enforce non-negotiable quality gates with anti-rationalization tables:

package/manifest.json

@@ -1,5 +1,5 @@
 {
-  "version": "4.22.1",
+  "version": "4.23.1",
   "components": {
     "agents": {
       "description": "44 specialized agents (orchestrator, backend, frontend, security, devops, etc.)",

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@softspark/ai-toolkit",
-  "version": "4.22.1",
+  "version": "4.23.1",
   "description": "AI coding toolkit: 109 skills, 44 agents, 12 developer-tool integrations, recoverable native tool-output filtering, Claude Chat/Cowork export, safety constitution, SARIF audit, and signed npm provenance.",
   "keywords": [
     "claude",

package/scripts/plugin.py

@@ -215,6 +215,23 @@ def _resolve_skill_source(pack_dir: Path, skill: str) -> Path | None:
     return None
 
 
+def _resolve_agent_source(pack_dir: Path, agent: str) -> Path | None:
+    """Locate the agent file a pack references.
+
+    Mirrors _resolve_skill_source: a core agent in app/agents wins, otherwise the
+    pack may ship its own under <pack>/agents/. Resolving only against app/agents
+    is what made every pack-shipped agent print "WARN agent not found" while
+    _remove_claude_pack_links already knew how to unlink one.
+    """
+    core = app_dir / "agents" / f"{agent}.md"
+    plugin = pack_dir / "agents" / f"{agent}.md"
+    if core.is_file():
+        return core
+    if plugin.is_file():
+        return plugin
+    return None
+
+
 def _resolve_rule_source(pack_dir: Path, rule_name: str) -> tuple[Path, bool] | None:
     candidates = [
         (pack_dir / f"{rule_name}.md", False),
@@ -577,13 +594,13 @@ def _install_claude_skills(pack: dict, pack_dir: Path, installed_items: list[str
             print(f"    WARN skill not found: {skill}")
 
 
-def _install_claude_agents(pack: dict, installed_items: list[str]) -> None:
+def _install_claude_agents(pack: dict, pack_dir: Path, installed_items: list[str]) -> None:
     for agent in pack.get("includes", {}).get("agents", []):
         agent_file = CLAUDE_DIR / "agents" / f"{agent}.md"
-        source_file = app_dir / "agents" / f"{agent}.md"
+        source_file = _resolve_agent_source(pack_dir, agent)
         if agent_file.exists() or agent_file.is_symlink():
             print(f"    OK agent: {agent}")
-        elif source_file.is_file():
+        elif source_file is not None:
             agent_file.parent.mkdir(parents=True, exist_ok=True)
             agent_file.symlink_to(source_file)
             print(f"    Linked agent: {agent}")
@@ -676,7 +693,7 @@ def install_pack_claude(name: str, pack: dict, pack_dir: Path) -> bool:
     rule_specs = _resolve_pack_rules(pack, pack_dir)
 
     _ensure_core_hook_scripts()
-    _install_claude_agents(pack, installed_items)
+    _install_claude_agents(pack, pack_dir, installed_items)
     _install_claude_skills(pack, pack_dir, installed_items)
     _copy_plugin_hook_scripts(name, hook_specs, installed_items)
     _copy_plugin_scripts(name, pack_dir, installed_items)

package/scripts/plugin_schema.py

@@ -139,20 +139,32 @@ def validate_references(
     data: dict,
     agents_dir: Path,
     skills_dir: Path,
+    pack_dir: Path | None = None,
 ) -> list[str]:
     """Validate that referenced agents and skills exist.
 
+    A pack may either reference a core asset (app/agents, app/skills) or ship its
+    own under <pack>/agents and <pack>/skills. Checking only the core directories
+    reported every self-contained pack as broken even though the installer links
+    its skills fine. pack_dir stays optional so existing callers keep working.
+
     Returns a list of error messages.
     """
     errors: list[str] = []
     includes = data.get("includes", {})
 
     for agent in includes.get("agents", []):
-        if not (agents_dir / f"{agent}.md").is_file():
+        candidates = [agents_dir / f"{agent}.md"]
+        if pack_dir is not None:
+            candidates.append(pack_dir / "agents" / f"{agent}.md")
+        if not any(c.is_file() for c in candidates):
             errors.append(f"References missing agent: {agent}")
 
     for skill in includes.get("skills", []):
-        if not (skills_dir / skill / "SKILL.md").is_file():
+        candidates = [skills_dir / skill / "SKILL.md"]
+        if pack_dir is not None:
+            candidates.append(pack_dir / "skills" / skill / "SKILL.md")
+        if not any(c.is_file() for c in candidates):
             errors.append(f"References missing skill: {skill}")
 
     return errors

package/scripts/validate.py

@@ -764,6 +764,7 @@ def _validate_pack_refs(tk_dir: Path, pack_path: Path, d: dict,
         d,
         agents_dir=tk_dir / "app" / "agents",
         skills_dir=tk_dir / "app" / "skills",
+        pack_dir=pack_path,
     )
     for err in ref_errors:
         vr.error(f"app/plugins/{pack_name}/plugin.json {err}")