@softspark/ai-toolkit 4.22.0 → 4.23.0

package/CHANGELOG.md

@@ -7,6 +7,72 @@ Versioning follows [Semantic Versioning](https://semver.org/).
 
 ---
 
+## v4.23.0 — a compliance scanner that says nothing must say why (2026-08-06)
+
+### Fixed
+
+- **`hipaa-validate` scanned a manifest-less project silently clean.** Category
+  1, 3, 4 and 7 patterns are language-tagged and fire only for a detected
+  language, and detection read manifest files only. A directory of Python with
+  PHI in its loggers therefore reported `HIGH: 0` while every Python rule sat
+  unused — the worst possible output for a compliance tool, because zero reads
+  as compliant. Found by the post-release SOP on v4.22.1.
+
+  Detection now falls back to the extensions of the files being scanned when no
+  manifest is present. **Manifests still win**, so a project that already
+  declared itself keeps exactly the behaviour it had.
+
+### Changed
+
+- **The scan summary reports how the language was decided.** A new
+  `language_detection` field says `manifest`, `file extensions (no manifest
+  found)`, or `none`. When it is `none` the text report writes to stderr that
+  language-tagged patterns did not run and that a zero-finding result means
+  unscanned rather than compliant.
+
+  **Upgrade note:** a project scanned without a manifest may now report HIGH
+  findings where v4.22.x reported none. Those findings were always there. A CI
+  job gating on the exit code can start failing on real PHI exposure.
+
+---
+
+## v4.22.1 — nine skills, not four, could not find their own scripts (2026-08-06)
+
+v4.22.0 claimed to fix four skills whose documented script path never resolved.
+The first real run of the post-release SOP found five more, plus a wrong
+interpreter and a script that crashed on `--help`. The v4.22.0 fix had grepped
+for one spelling of the bug and missed the others.
+
+### Fixed
+
+- **Five more skills ran their own script through a path that cannot resolve
+  after install**: `brand-voice` (repo-relative `app/skills/.../measure.py`),
+  `ci`, `deploy`, `migrate` and `rollback` (cwd-relative `scripts/*.py`). Four
+  further example lines in `debug`, `fix`, `pr` and `review` had the same defect
+  in a secondary snippet while the primary invocation was already correct.
+- **`rollback` ran a Python file through `bash`.** `bash scripts/rollback_info.py`
+  was wrong twice over.
+- **`explore` used `python` rather than `python3`**, which is missing or Python 2
+  on many systems.
+- **`explore`'s `visualize.py` crashed with a traceback on `--help`**, treating
+  the flag as a directory to scan. It now prints usage, and reports a JSON error
+  with exit 1 for a path that does not exist.
+
+### Added
+
+- **`validate.py` fails the build on the whole class.**
+  `_validate_skill_script_invocations` checks every documented command that runs
+  a skill-owned script: the path must go through `${CLAUDE_SKILL_DIR}`, and the
+  interpreter must match the file. `python` instead of `python3` is a warning.
+  Repo-level scripts such as `scripts/validate.py` are correctly ignored, and the
+  frontmatter `scripts:` list stays relative because it declares rather than runs.
+- **Post-release SOP phases 4b and 4c** (`post-release-testing-sop.md` v1.1.0):
+  run every skill's documented invocation from the installed copy, and prove
+  scanner-wrapping skills actually scan a fixture with known defects. Phase 4b
+  found every defect in this release on its first run.
+
+---
+
 ## v4.22.0 — the public surface is a test, not a promise (2026-08-06)
 
 ### Added

package/README.md

@@ -6,7 +6,24 @@
 [![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)
 [![Skills](https://img.shields.io/badge/skills-109-brightgreen)](app/skills/)
 [![Agents](https://img.shields.io/badge/agents-44-blue)](app/agents/)
-[![Tests](https://img.shields.io/badge/tests-1508%20passing-success)](tests/)
+[![Tests](https://img.shields.io/badge/tests-1516%20passing-success)](tests/)
+
+## What's New in v4.23.0
+
+**v4.23.0** — `hipaa-validate` scanned a project without a manifest silently
+clean: its language-tagged patterns never ran, and the report said `HIGH: 0`.
+Zero reads as compliant. Detection now falls back to file extensions, the summary
+says how the language was decided, and an undecidable run warns that a zero means
+unscanned. Manifest-declaring projects are unaffected. Found by the post-release
+SOP, not by a user.
+
+## What's New in v4.22.1
+
+**v4.22.1** — nine skills, not four, could not find their own scripts. v4.22.0
+grepped for one spelling of the bug and missed five more, a `.py` file run
+through `bash`, and a script that crashed on `--help`. `validate.py` now fails
+the build on the entire class, and the post-release SOP runs every skill's
+documented command from the installed copy — which is how these were found.
 
 ## What's New in v4.22.0
 

package/app/.claude-plugin/plugin.json

@@ -3,7 +3,7 @@
   "name": "ai-toolkit",
   "displayName": "AI Toolkit",
   "description": "Professional-grade engineering skills, agents, rules, and lifecycle guardrails for Claude Code, Claude Chat, and Cowork.",
-  "version": "4.22.0",
+  "version": "4.23.0",
   "author": {
     "name": "SoftSpark",
     "url": "https://github.com/softspark"

package/app/skills/brand-voice/SKILL.md

@@ -162,7 +162,7 @@ Modes change *prose*, not *data*. If you cut a fact to fit a length budget, you
 
 ### Measurement
 
-Run `python3 app/skills/brand-voice/scripts/measure.py --fixtures tests/fixtures/output-modes/` to compare baseline vs mode tokens on the fixture set. Report shows per-fixture deltas and an aggregate ratio.
+Run `python3 ${CLAUDE_SKILL_DIR}/scripts/measure.py --fixtures tests/fixtures/output-modes/` to compare baseline vs mode tokens on the fixture set. Report shows per-fixture deltas and an aggregate ratio.
 
 ## When NOT to Load
 

package/app/skills/ci/SKILL.md

@@ -27,7 +27,7 @@ Generate, update, or troubleshoot CI/CD pipeline configuration based on project
 
 Detect CI platform and analyze configuration:
 ```bash
-python3 scripts/ci-detect.py [directory]
+python3 ${CLAUDE_SKILL_DIR}/scripts/ci-detect.py [directory]
 ```
 
 Returns JSON with:

package/app/skills/debug/SKILL.md

@@ -28,7 +28,7 @@ Pipe error output through the error parser for structured diagnosis:
 your_command 2>&1 | python3 ${CLAUDE_SKILL_DIR}/scripts/error-parser.py
 
 # Or from a log file
-cat /var/log/app/error.log | python3 scripts/error-parser.py
+cat /var/log/app/error.log | python3 ${CLAUDE_SKILL_DIR}/scripts/error-parser.py
 ```
 
 The script outputs JSON with:

package/app/skills/deploy/SKILL.md

@@ -50,7 +50,7 @@ Deploy application to target environment.
 
 Run automated pre-deployment checks:
 ```bash
-python3 scripts/pre_deploy_check.py [environment]
+python3 ${CLAUDE_SKILL_DIR}/scripts/pre_deploy_check.py [environment]
 ```
 
 Returns JSON with pass/fail for each check:

package/app/skills/explore/SKILL.md

@@ -99,7 +99,7 @@ It does NOT write or modify any files.
 For an interactive HTML tree visualization of the codebase:
 
 ```bash
-python ${CLAUDE_SKILL_DIR}/scripts/visualize.py .
+python3 ${CLAUDE_SKILL_DIR}/scripts/visualize.py .
 ```
 
 This generates `codebase-map.html` with collapsible directories, file sizes, and type-colored indicators.

package/app/skills/explore/scripts/visualize.py

@@ -108,8 +108,22 @@ data.children.forEach(c=>render(c,document.getElementById('root')));
 </script></body></html>'''
     output.write_text(html)
 
+USAGE = """usage: visualize.py [PATH]
+
+Scan PATH (default: current directory) and write an interactive codebase map to
+codebase-map.html, then open it in the default browser.
+"""
+
 if __name__ == '__main__':
-    target = Path(sys.argv[1] if len(sys.argv) > 1 else '.').resolve()
+    if len(sys.argv) > 1 and sys.argv[1] in ('-h', '--help'):
+        print(USAGE)
+        raise SystemExit(0)
+
+    target = Path(sys.argv[1] if len(sys.argv) > 1 else '.')
+    if not target.is_dir():
+        print(f'{{"error": "Not a directory: {target}"}}')
+        raise SystemExit(1)
+    target = target.resolve()
     stats = {"files": 0, "dirs": 0, "extensions": Counter(), "ext_sizes": Counter()}
     data = scan(target, stats)
     out = Path('codebase-map.html')

package/app/skills/fix/SKILL.md

@@ -27,8 +27,8 @@ Before entering the fix loop, classify errors to prioritize auto-fixable ones:
 ```bash
 # Pipe lint or test output
 ruff check . 2>&1 | python3 ${CLAUDE_SKILL_DIR}/scripts/error-classifier.py
-mypy src/ 2>&1 | python3 scripts/error-classifier.py
-npx eslint . 2>&1 | python3 scripts/error-classifier.py
+mypy src/ 2>&1 | python3 ${CLAUDE_SKILL_DIR}/scripts/error-classifier.py
+npx eslint . 2>&1 | python3 ${CLAUDE_SKILL_DIR}/scripts/error-classifier.py
 ```
 
 The script outputs JSON with:

package/app/skills/hipaa-validate/SKILL.md

@@ -161,6 +161,9 @@ This distinction helps compliance officers prioritize immediate remediation (def
 
 ## Gotchas
 
+- **Check `language_detection` in the summary before trusting a zero.** Category 1, 3, 4 and 7 patterns are language-tagged and only fire for a detected language. Manifests decide first (`pyproject.toml`, `package.json`, `go.mod`, …); without one the scanner falls back to file extensions. If it reports `languages: ["any"]` with `language_detection: "none"`, the language rules never ran and `HIGH: 0` means *unscanned*, not *compliant* — the scanner prints that warning to stderr, so a run whose stderr is discarded loses it.
+- Scanning a monorepo package or a subdirectory can put you below the manifest. The extension fallback covers the common case, but a directory of `.sql`, `.yaml` or templates resolves to no language at all — scan from the level that holds the manifest.
+
 - Test fixtures and seed data often contain **synthetic** PHI that looks real (SSN-shaped IDs, formatted phone numbers, sample email addresses). Flag them but lower severity — production code handling the same patterns is the actual risk.
 - HIPAA §164.312(b) requires audit logging but does not specify a format. "Logs exist" is not evidence of compliance — the logs must capture WHO (authenticated user), WHAT (action), WHEN (timestamp), WHERE (resource), and they must be immutable (append-only or write-once storage).
 - Encryption-at-rest varies silently by storage layer. RDS auto-encrypts new volumes since 2017, but older DB snapshots may not be; S3 bucket policies can override instance-level encryption. Treat "encryption enabled" as a claim to verify with the cloud provider, not a state to trust.

package/app/skills/hipaa-validate/scripts/hipaa_scan.py

@@ -302,8 +302,24 @@ def context_gate(files: list[Path], keywords: list[str],
 # Language detection (Step 1)
 # ---------------------------------------------------------------------------
 
-def detect_languages(root: Path) -> set[str]:
-    """Detect project languages from manifest files."""
+def detect_languages(root: Path, files: list[Path] | None = None) -> tuple[set[str], str]:
+    """Detect project languages. Returns (languages, how_they_were_detected).
+
+    Manifest files decide when present. When none is found, fall back to the
+    extensions of the files actually being scanned.
+
+    The fallback exists because language-tagged patterns fire only when the
+    project language includes the tag, so a manifest-less directory of Python
+    scanned clean while every Python rule sat unused — and the report said
+    `HIGH: 0`, which reads as compliant rather than unscanned. Silence is the
+    worst possible output for a compliance tool. A monorepo package, a scan
+    pointed at a subdirectory, or a service whose manifest lives one level up
+    all hit this.
+
+    Manifests still win: a project that declares itself keeps exactly the
+    behaviour it had, so upgrading changes nothing for anyone who was already
+    being scanned properly.
+    """
     langs = set()
     for indicator, lang in LANGUAGE_INDICATORS.items():
         if "*" in indicator:
@@ -314,7 +330,20 @@ def detect_languages(root: Path) -> set[str]:
         else:
             if (root / indicator).exists():
                 langs.add(lang)
-    return langs or {"any"}
+    if langs:
+        return langs, "manifest"
+
+    if files:
+        ext_to_lang: dict[str, set[str]] = {}
+        for lang, exts in LANG_EXTENSIONS.items():
+            for ext in exts:
+                ext_to_lang.setdefault(ext, set()).add(lang)
+        for f in files:
+            langs |= ext_to_lang.get(f.suffix, set())
+        if langs:
+            return langs, "file extensions (no manifest found)"
+
+    return {"any"}, "none — language-tagged patterns were skipped"
 
 
 # ---------------------------------------------------------------------------
@@ -616,7 +645,8 @@ def format_text_report(findings: list[dict], mode: str,
 
 def format_json_report(findings: list[dict], mode: str,
                        phi_count: int, scanned_count: int,
-                       categories_run: list[int], languages: list[str]) -> str:
+                       categories_run: list[int], languages: list[str],
+                       language_source: str = "manifest") -> str:
     """Format findings as structured JSON for CI integration."""
     high_count = sum(1 for f in findings if f["severity"] == "HIGH")
     warn_count = sum(1 for f in findings if f["severity"] == "WARN")
@@ -628,6 +658,7 @@ def format_json_report(findings: list[dict], mode: str,
             "files_scanned": scanned_count,
             "categories_run": sorted(categories_run),
             "languages": sorted(languages),
+            "language_detection": language_source,
             "high": high_count,
             "warn": warn_count,
             "total": len(findings),
@@ -723,8 +754,9 @@ def main() -> None:
               file=sys.stderr)
         print("Consider narrowing the scan path for targeted results.", file=sys.stderr)
 
-    # Step 1: Detect languages
-    languages = detect_languages(root)
+    # Step 1: Detect languages. Categories 1-2 scan everything, so the extension
+    # fallback looks at all collected files, not just the PHI-adjacent subset.
+    languages, lang_source = detect_languages(root, all_files)
 
     # Step 2: Run categories
     findings: list[dict] = []
@@ -789,10 +821,18 @@ def main() -> None:
     if args.output == "json":
         print(format_json_report(findings, args.mode, phi_count,
                                  scanned_count, categories_run,
-                                 sorted(languages)))
+                                 sorted(languages), lang_source))
     else:
         print(format_text_report(findings, args.mode, phi_count,
                                  scanned_count, categories_run))
+        if languages == {"any"}:
+            # Never let an unscanned run read as a clean one.
+            print("\nWARNING: no project language identified "
+                  f"({lang_source}).", file=sys.stderr)
+            print("Language-tagged patterns (Python, JS/TS, Go, Java, Ruby, C#) "
+                  "did NOT run.", file=sys.stderr)
+            print("A zero-finding result here means unscanned, not compliant. "
+                  "Scan from the directory holding the manifest.", file=sys.stderr)
 
     # Exit code: non-zero if HIGH findings
     high_count = sum(1 for f in findings if f["severity"] == "HIGH")

package/app/skills/migrate/SKILL.md

@@ -31,7 +31,7 @@ Create, run, or manage database migrations with auto-detection of the migration
 
 Detect migration tool and report status:
 ```bash
-python3 scripts/migration-status.py [directory]
+python3 ${CLAUDE_SKILL_DIR}/scripts/migration-status.py [directory]
 ```
 
 Returns JSON with: `tool`, `config_file`, `migrations_dir`, `total_migrations`, `latest`, `commands{}` (status/create/upgrade/downgrade).

package/app/skills/pr/SKILL.md

@@ -30,7 +30,7 @@ Generate a structured PR summary from the commit history before writing the PR d
 ```bash
 python3 ${CLAUDE_SKILL_DIR}/scripts/pr-summary.py [base_branch]
 # Default base branch: main
-# Example: python3 scripts/pr-summary.py develop
+# Example: python3 ${CLAUDE_SKILL_DIR}/scripts/pr-summary.py develop
 ```
 
 The script outputs JSON with:

package/app/skills/review/SKILL.md

@@ -43,7 +43,7 @@ Before starting manual review, run the diff analyzer script to get a structured
 ```bash
 python3 ${CLAUDE_SKILL_DIR}/scripts/diff-analyzer.py [base_branch]
 # Default base branch: main
-# Example: python3 scripts/diff-analyzer.py develop
+# Example: python3 ${CLAUDE_SKILL_DIR}/scripts/diff-analyzer.py develop
 ```
 
 The script outputs JSON with:

package/app/skills/rollback/SKILL.md

@@ -59,7 +59,7 @@ git reset --soft <commit-sha>  # Keep changes staged
 
 Run the rollback info script to assess current state before rolling back:
 ```bash
-bash scripts/rollback_info.py
+python3 ${CLAUDE_SKILL_DIR}/scripts/rollback_info.py
 ```
 
 Returns JSON with:

package/kb/procedures/post-release-testing-sop.md

@@ -3,10 +3,10 @@ title: "SOP: Post-Release Testing"
 category: procedures
 service: ai-toolkit
 tags: [sop, post-release, smoke-test, npm, sandbox, plugin-pack, provenance, isolation]
-version: "1.0.0"
+version: "1.1.0"
 created: "2026-07-26"
-last_updated: "2026-07-26"
-description: "Smoke-test a published @softspark/ai-toolkit release from npm in an isolated HOME and npm prefix, without touching the maintainer's real install. Covers provenance, CLI, doctor, and the full plugin-pack lifecycle including the degraded-install path. Written for v4.18.0 and not run; v4.18.0 shipped a pack that broke every command it touched, and every step here would have caught it."
+last_updated: "2026-08-06"
+description: "Smoke-test a published @softspark/ai-toolkit release from npm in an isolated HOME and npm prefix, without touching the maintainer's real install. Covers provenance, CLI, doctor, per-skill script resolution, scanner wiring, and the full plugin-pack lifecycle including the degraded-install path. Written for v4.18.0 and not run; v4.18.0 shipped a pack that broke every command it touched. First actually run on v4.22.0, which added Phases 4b and 4c after that release fixed four skills whose documented script path had never resolved and two that shipped a scanner nothing invoked."
 ---
 
 # SOP: Post-Release Testing
@@ -92,6 +92,95 @@ npm install -g --prefix "$SB/npm" "@softspark/ai-toolkit@${VERSION}"
 `skills directory missing`.** That is the sandbox being empty, not a defect.
 Install first, then judge doctor.
 
+## Phase 4b: Skill scripts resolve and run from the installed copy
+
+23 skills ship an executable script under `scripts/`. A skill body invokes it
+through `${CLAUDE_SKILL_DIR}`, which only resolves once the skill is installed —
+so a wrong path is invisible in the working tree and invisible to `validate.py`,
+which checks that the file exists on disk, never that the documented command
+finds it.
+
+This is exactly how four skills shipped with `$(dirname "$0")`, which expands to
+the shell's directory rather than the skill's. Every one of them had been broken
+for as long as the line existed.
+
+Iterate over every skill that ships a script. Do not hand-pick the ones the
+release touched — the point of this phase is to catch the ones nobody remembered.
+
+```bash
+for D in "$HOME"/.claude/skills/*/; do
+  s=$(basename "$D")
+  [ -d "$D/scripts" ] || continue
+  # Match ANY interpreter and ANY extension. Narrowing this to `python3` and
+  # `.py` is how the first version of this phase reported "no invocation" for a
+  # skill that used `python`, and missed one that ran a .py file through bash.
+  ref=$(grep -ohE '(python3?|bash|sh|node) +\$\{CLAUDE_SKILL_DIR\}/scripts/[A-Za-z0-9_.-]+' "$D/SKILL.md" | head -1)
+  [ -n "$ref" ] || { printf '%-22s NO ${CLAUDE_SKILL_DIR} INVOCATION\n' "$s"; continue; }
+  interp=${ref%% *}
+  rel=${ref##*\$\{CLAUDE_SKILL_DIR\}/}
+  printf '%-22s %-10s %-26s ' "$s" "$interp" "$rel"
+  [ -f "$D/$rel" ] || { echo 'PATH DOES NOT RESOLVE'; continue; }
+  out=$(CLAUDE_SKILL_DIR="$D" timeout 20 "$interp" "$D/$rel" --help </dev/null 2>&1 | head -1)
+  printf 'rc=%s %s\n' "$?" "$(echo "$out" | cut -c1-40)"
+done
+```
+
+**Verify:**
+- [ ] Every skill with a `scripts/` directory has a documented invocation
+- [ ] Every documented path resolves to a file that exists
+- [ ] Every script exits without a traceback and without hanging
+- [ ] The interpreter matches the file: no `.py` through `bash`, no bare `python`
+
+A skill reported as `NO ${CLAUDE_SKILL_DIR} INVOCATION` is not automatically a
+defect — some ship assets rather than executables (`write-a-prd` ships `.html`,
+`.js` and `.cjs`). Read the skill before filing it. What *is* always a defect is
+an invocation that names a skill-owned script through any other path.
+
+`validate.py` now fails the build on that class (`_validate_skill_script_invocations`),
+so this phase is the second line rather than the first. Keep it: the validator
+reasons about the source, this runs the real thing.
+
+**`</dev/null` and `timeout` are not defensive padding.** Several of these
+scripts are stdin filters (`error-parser.py`, `error-classifier.py`) documented
+as `command 2>&1 | python3 …`. Probing one with `--help` and an open stdin
+blocks forever, and the run looks like a slow test rather than a hung one. A
+correct stdin filter answers an empty stdin with a JSON error and `rc=0`; a
+traceback there is a real finding.
+
+## Phase 4c: Skills that wrap a scanner actually scan
+
+For any skill whose body tells the model to run a scanner, presence of the script
+is not evidence the wiring works. Build a fixture with known defects and confirm
+the scanner reports them.
+
+```bash
+FX="$SB/fixture"; mkdir -p "$FX"
+cat > "$FX/index.html" <<'EOF'
+<!DOCTYPE html><html><head><title>t</title></head>
+<body><h1>A</h1><h3>skipped h2</h3><img src="x.png"><input type="text"><div onclick="go()">click</div></body></html>
+EOF
+printf '{"name":"fx","dependencies":{"react":"18"}}\n' > "$FX/package.json"
+
+for s in a11y-validate seo-validate hipaa-validate cve-scan; do
+  D="$HOME/.claude/skills/$s"
+  script=$(ls "$D"/scripts/*.py 2>/dev/null | head -1)
+  [ -n "$script" ] || continue
+  echo "=== $s ==="
+  CLAUDE_SKILL_DIR="$D" timeout 60 python3 "$script" "$FX" --output json </dev/null 2>&1 | head -3
+done
+```
+
+**Verify:**
+- [ ] The scanner returns findings, not an empty set — the fixture has real defects
+- [ ] Findings span more than one category, proving the whole check set ran
+- [ ] A scanner that exits non-zero on findings is doing its job, not failing
+
+`a11y-validate` and `seo-validate` shipped working scanners that **no step in
+either skill invoked** for their entire life before v4.22.0. The model was told to
+grep the pattern tables by hand instead. Nothing in the test suite noticed,
+because a script nobody calls still passes every check that asks whether it
+exists.
+
 ## Phase 5: Plugin-pack lifecycle
 
 Run this for any pack the release touched. For a pack that downloads a binary,

package/llms-full.txt

@@ -7019,10 +7019,10 @@ title: "SOP: Post-Release Testing"
 category: procedures
 service: ai-toolkit
 tags: [sop, post-release, smoke-test, npm, sandbox, plugin-pack, provenance, isolation]
-version: "1.0.0"
+version: "1.1.0"
 created: "2026-07-26"
-last_updated: "2026-07-26"
-description: "Smoke-test a published @softspark/ai-toolkit release from npm in an isolated HOME and npm prefix, without touching the maintainer's real install. Covers provenance, CLI, doctor, and the full plugin-pack lifecycle including the degraded-install path. Written for v4.18.0 and not run; v4.18.0 shipped a pack that broke every command it touched, and every step here would have caught it."
+last_updated: "2026-08-06"
+description: "Smoke-test a published @softspark/ai-toolkit release from npm in an isolated HOME and npm prefix, without touching the maintainer's real install. Covers provenance, CLI, doctor, per-skill script resolution, scanner wiring, and the full plugin-pack lifecycle including the degraded-install path. Written for v4.18.0 and not run; v4.18.0 shipped a pack that broke every command it touched. First actually run on v4.22.0, which added Phases 4b and 4c after that release fixed four skills whose documented script path had never resolved and two that shipped a scanner nothing invoked."
 ---
 
 # SOP: Post-Release Testing
@@ -7108,6 +7108,95 @@ npm install -g --prefix "$SB/npm" "@softspark/ai-toolkit@${VERSION}"
 `skills directory missing`.** That is the sandbox being empty, not a defect.
 Install first, then judge doctor.
 
+## Phase 4b: Skill scripts resolve and run from the installed copy
+
+23 skills ship an executable script under `scripts/`. A skill body invokes it
+through `${CLAUDE_SKILL_DIR}`, which only resolves once the skill is installed —
+so a wrong path is invisible in the working tree and invisible to `validate.py`,
+which checks that the file exists on disk, never that the documented command
+finds it.
+
+This is exactly how four skills shipped with `$(dirname "$0")`, which expands to
+the shell's directory rather than the skill's. Every one of them had been broken
+for as long as the line existed.
+
+Iterate over every skill that ships a script. Do not hand-pick the ones the
+release touched — the point of this phase is to catch the ones nobody remembered.
+
+```bash
+for D in "$HOME"/.claude/skills/*/; do
+  s=$(basename "$D")
+  [ -d "$D/scripts" ] || continue
+  # Match ANY interpreter and ANY extension. Narrowing this to `python3` and
+  # `.py` is how the first version of this phase reported "no invocation" for a
+  # skill that used `python`, and missed one that ran a .py file through bash.
+  ref=$(grep -ohE '(python3?|bash|sh|node) +\$\{CLAUDE_SKILL_DIR\}/scripts/[A-Za-z0-9_.-]+' "$D/SKILL.md" | head -1)
+  [ -n "$ref" ] || { printf '%-22s NO ${CLAUDE_SKILL_DIR} INVOCATION\n' "$s"; continue; }
+  interp=${ref%% *}
+  rel=${ref##*\$\{CLAUDE_SKILL_DIR\}/}
+  printf '%-22s %-10s %-26s ' "$s" "$interp" "$rel"
+  [ -f "$D/$rel" ] || { echo 'PATH DOES NOT RESOLVE'; continue; }
+  out=$(CLAUDE_SKILL_DIR="$D" timeout 20 "$interp" "$D/$rel" --help </dev/null 2>&1 | head -1)
+  printf 'rc=%s %s\n' "$?" "$(echo "$out" | cut -c1-40)"
+done
+```
+
+**Verify:**
+- [ ] Every skill with a `scripts/` directory has a documented invocation
+- [ ] Every documented path resolves to a file that exists
+- [ ] Every script exits without a traceback and without hanging
+- [ ] The interpreter matches the file: no `.py` through `bash`, no bare `python`
+
+A skill reported as `NO ${CLAUDE_SKILL_DIR} INVOCATION` is not automatically a
+defect — some ship assets rather than executables (`write-a-prd` ships `.html`,
+`.js` and `.cjs`). Read the skill before filing it. What *is* always a defect is
+an invocation that names a skill-owned script through any other path.
+
+`validate.py` now fails the build on that class (`_validate_skill_script_invocations`),
+so this phase is the second line rather than the first. Keep it: the validator
+reasons about the source, this runs the real thing.
+
+**`</dev/null` and `timeout` are not defensive padding.** Several of these
+scripts are stdin filters (`error-parser.py`, `error-classifier.py`) documented
+as `command 2>&1 | python3 …`. Probing one with `--help` and an open stdin
+blocks forever, and the run looks like a slow test rather than a hung one. A
+correct stdin filter answers an empty stdin with a JSON error and `rc=0`; a
+traceback there is a real finding.
+
+## Phase 4c: Skills that wrap a scanner actually scan
+
+For any skill whose body tells the model to run a scanner, presence of the script
+is not evidence the wiring works. Build a fixture with known defects and confirm
+the scanner reports them.
+
+```bash
+FX="$SB/fixture"; mkdir -p "$FX"
+cat > "$FX/index.html" <<'EOF'
+<!DOCTYPE html><html><head><title>t</title></head>
+<body><h1>A</h1><h3>skipped h2</h3><img src="x.png"><input type="text"><div onclick="go()">click</div></body></html>
+EOF
+printf '{"name":"fx","dependencies":{"react":"18"}}\n' > "$FX/package.json"
+
+for s in a11y-validate seo-validate hipaa-validate cve-scan; do
+  D="$HOME/.claude/skills/$s"
+  script=$(ls "$D"/scripts/*.py 2>/dev/null | head -1)
+  [ -n "$script" ] || continue
+  echo "=== $s ==="
+  CLAUDE_SKILL_DIR="$D" timeout 60 python3 "$script" "$FX" --output json </dev/null 2>&1 | head -3
+done
+```
+
+**Verify:**
+- [ ] The scanner returns findings, not an empty set — the fixture has real defects
+- [ ] Findings span more than one category, proving the whole check set ran
+- [ ] A scanner that exits non-zero on findings is doing its job, not failing
+
+`a11y-validate` and `seo-validate` shipped working scanners that **no step in
+either skill invoked** for their entire life before v4.22.0. The model was told to
+grep the pattern tables by hand instead. Nothing in the test suite noticed,
+because a script nobody calls still passes every check that asks whether it
+exists.
+
 ## Phase 5: Plugin-pack lifecycle
 
 Run this for any pack the release touched. For a pack that downloads a binary,

package/manifest.json

@@ -1,5 +1,5 @@
 {
-  "version": "4.22.0",
+  "version": "4.23.0",
   "components": {
     "agents": {
       "description": "44 specialized agents (orchestrator, backend, frontend, security, devops, etc.)",

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@softspark/ai-toolkit",
-  "version": "4.22.0",
+  "version": "4.23.0",
   "description": "AI coding toolkit: 109 skills, 44 agents, 12 developer-tool integrations, recoverable native tool-output filtering, Claude Chat/Cowork export, safety constitution, SARIF audit, and signed npm provenance.",
   "keywords": [
     "claude",

package/scripts/validate.py

@@ -379,6 +379,71 @@ def _validate_skill_frontmatter(tk_dir: Path, skill_path: Path,
         vr.warn(f"{name} - Description exceeds 1024 characters")
 
 
+def _validate_skill_script_invocations(skill_path: Path, vr: ValidationResult) -> None:
+    """Every documented run of a skill-owned script must resolve after install.
+
+    A skill's own scripts live beside it and are reachable only through
+    `${CLAUDE_SKILL_DIR}`. A relative path such as `python3 scripts/foo.py`
+    resolves against the user's working directory, and a repo-relative one such
+    as `app/skills/<name>/scripts/foo.py` against a tree the user does not have.
+    Both look correct in the source and fail for every installed user.
+
+    Nine skills shipped with one of those forms before this check existed.
+    `_validate_skill_references` never caught them because it asks whether the
+    file is on disk, not whether the documented command can find it.
+
+    Only executable invocations are checked. The frontmatter `scripts:` list
+    declares ownership and is correctly relative; prose that names a script in
+    backticks runs nothing.
+    """
+    name = skill_path.name
+    scripts_dir = skill_path / "scripts"
+    if not scripts_dir.is_dir():
+        return
+    owned = {p.name for p in scripts_dir.iterdir() if p.is_file()}
+    if not owned:
+        return
+
+    skill_file = skill_path / "SKILL.md"
+    text = skill_file.read_text(encoding="utf-8")
+
+    # Skip the frontmatter block: `scripts:` entries there are declarations.
+    lines = text.splitlines()
+    start = 0
+    if lines and lines[0].strip() == "---":
+        for idx in range(1, len(lines)):
+            if lines[idx].strip() == "---":
+                start = idx + 1
+                break
+
+    interpreters = {"python3": {".py"}, "python": {".py"}, "bash": {".sh"}, "sh": {".sh"}}
+    invocation = re.compile(r"\b(python3|python|bash|sh)\s+(\S+)")
+
+    for offset, line in enumerate(lines[start:], start=start + 1):
+        for interp, target in invocation.findall(line):
+            base = target.rsplit("/", 1)[-1].strip("\"'`")
+            if base not in owned:
+                continue  # a repo-level script such as scripts/validate.py
+            if "${CLAUDE_SKILL_DIR}" not in target:
+                vr.error(
+                    f"skills/{name}/SKILL.md:{offset}: runs its own '{base}' via "
+                    f"'{target}' - use ${{CLAUDE_SKILL_DIR}}/scripts/{base}, which "
+                    f"is the only path that resolves after install"
+                )
+            suffix = base[base.rfind("."):] if "." in base else ""
+            allowed = interpreters.get(interp, set())
+            if suffix and allowed and suffix not in allowed:
+                vr.error(
+                    f"skills/{name}/SKILL.md:{offset}: runs '{base}' with "
+                    f"'{interp}' - wrong interpreter for a {suffix} file"
+                )
+            elif interp == "python":
+                vr.warn(
+                    f"skills/{name}/SKILL.md:{offset}: uses 'python' for '{base}' "
+                    f"- prefer 'python3', 'python' is absent or Python 2 on many systems"
+                )
+
+
 def _validate_skill_references(tk_dir: Path, skill_path: Path,
                                fm_lines: list[str], vr: ValidationResult) -> None:
     """Validate agent refs, depends-on, context/agent co-occurrence, and reference links."""
@@ -412,6 +477,8 @@ def _validate_skill_references(tk_dir: Path, skill_path: Path,
     if _fm_has(fm_lines, "run-mode"):
         vr.warn(f"{name} - Uses deprecated 'run-mode' field (rename to 'context:')")
 
+    _validate_skill_script_invocations(skill_path, vr)
+
     ref_dir = skill_path / "reference"
     if ref_dir.is_dir():
         content = skill_file.read_text(encoding="utf-8")