@csark0812/skeleton 1.5.4 → 1.5.5

package/README.md

@@ -6,11 +6,9 @@
 
 Agent repos get messy fast. Skills get copied around, docs disagree, links go stale, and nobody remembers which file is actually canonical.
 
-Skeleton is an SSOT linter for that layer. You define the contract once; Skeleton checks it locally and in CI.
+Skeleton is an SSOT linter for that layer. Define the contract once; Skeleton checks it locally and in CI. If a canonical doc disappears, a registry drifts, a skill index stops matching disk, or a generated reference gets edited by hand, the audit fails before merge.
 
-If a canonical doc disappears, a registry drifts, a skill index stops matching disk, or a generated reference gets edited by hand, the audit fails before merge.
-
-Think ESLint, but for the docs and skills your agents rely on.
+Think ESLint — for the docs and skills your agents rely on.
 
 Skeleton is **not** a runtime agent harness. It doesn't execute tools, enforce permissions, or manage memory. It checks whether the repo around those systems still holds together.
 
@@ -18,9 +16,7 @@ Skeleton is **not** a runtime agent harness. It doesn't execute tools, enforce p
 
 Agents can read the repo. They can't reliably infer which of three conflicting docs wins, whether a synced skill should be edited here, or which validation command actually proves a change.
 
-That stuff needs to be explicit. More importantly, it needs to stay true after the next 50 PRs.
-
-Skeleton turns those repo conventions into checks:
+That needs to be explicit — and stay true after the next 50 PRs. Skeleton turns those conventions into checks:
 
 | Code repos                                              | Agent repos                                                                            |
 | ------------------------------------------------------- | -------------------------------------------------------------------------------------- |
@@ -28,47 +24,50 @@ Skeleton turns those repo conventions into checks:
 | `eslint --fix` on changed files                         | `skeleton validate changed` on changed docs and skills                                 |
 | Pre-commit + CI gate                                    | `--staged` pre-commit + `--base` CI gate                                               |
 
-Skill linters answer: _"Is this SKILL.md well-formed?"_
+Skill linters ask: _"Is this SKILL.md well-formed?"_
+
+Skeleton asks the repo-level question: _"Does this whole thing still agree with itself?"_
+
+## Why a clean SSOT helps agents
+
+### Question
 
-Skeleton answers the repo-level question: _"Does this whole thing still agree with itself?"_
+Does an intact Skeleton contract change agent behavior — grounding on the right doc, picking the right validation lane, and how much work it takes to get there?
 
-## What the research says
+### What we did
 
-Research on agent context is still early, but the direction is useful: more context isn't automatically better.
+We ran a paired live A/B harness with [`@post-print/agent-test`](https://www.npmjs.com/package/@post-print/agent-test): `skeleton-clean` vs `skeleton-messy`. Same prompts and scenario set; the only intentional difference was registry / conflict structure and context profile.
 
-- A [2026 study of repo-level context files](https://doi.org/10.48550/arxiv.2602.11988) tested 438 coding tasks. Human-written files improved resolution by 4% on average; generated files reduced it by 3%. Both increased inference cost by more than 20%. The recommendation was pretty direct: keep instructions minimal and include what the agent can't infer.
-- A [2025 METR randomized trial](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/) found experienced open-source developers took 19% longer with early-2025 AI tools while believing they were faster.
+Scenarios covered contested grounding (conflicting docs), docs-only validation routing, canonical grounding, owned-skill routing, and customize ownership. Protocol: **N=10** sequential paired compares on 2026-07-17; McNemar on paired pass/fail; median token deltas with a bootstrap CI on the mean.
 
-Those studies motivate Skeleton’s constraint (small useful context, clear ownership, verified SSOT). They do **not** measure Skeleton itself.
+Full method: [refs/llm-harness.md](refs/llm-harness.md). Suites: [agent-suites/README.md](agent-suites/README.md). Aggregated numbers: [SUMMARY.md](agent-suites/evidence/SUMMARY.md). Side-by-side excerpts: [evidence/transcripts/](agent-suites/evidence/transcripts/).
 
-### Skeleton-specific benchmark (what we measure)
+### What improved
 
-We run a paired live A/B harness (`skeleton-clean` vs `skeleton-messy`) with [`@post-print/agent-test`](https://www.npmjs.com/package/@post-print/agent-test): same prompts, differing registry / conflict structure and context profile. Tasks cover registry grounding, validation-lane choice, and customize ownership.
+![Pass rate by scenario — clean vs messy](agent-suites/evidence/charts/pass-rates.svg)
 
-- Method and significance gates: [refs/llm-harness.md](refs/llm-harness.md) (target **N=10** independent compares; McNemar on pass/fail; token/tool deltas)
-- Suite definitions: [agent-suites/README.md](agent-suites/README.md)
-- Numbers + transcript excerpts: [agent-suites/evidence/](agent-suites/evidence/)
+![Median extra tokens on messy](agent-suites/evidence/charts/token-delta.svg)
 
-### What the evidence shows (N=10)
+On tasks that depend on an intact SSOT, the clean fixture was both more accurate and cheaper:
 
-Ten sequential live compares (2026-07-17) are summarized in [SUMMARY.md](agent-suites/evidence/SUMMARY.md). Gates passed: grounding McNemar p < 0.05 (clean > messy) and positive grounding median token Δ (`gates.readmeFinalClaimsAllowed: true`).
+- **Contested grounding** — In every paired run, clean settled on the registry canonical; messy never did (McNemar p = 0.002). Clean hops the registry; messy thrashes across conflicting docs.
+- **Docs routing** — Clean consistently chose the correct audit lane; messy invented a non-existent `audit all` path (McNemar p = 0.002).
+- **Token cost** — Across grounding tasks, messy used a median **~312k** more tokens than clean (bootstrap 95% CI on the mean excludes 0). Pass rate alone understates the gap: messy can still luck into an answer while spending far more.
 
-| Signal | Clean | Messy | Notes |
-| ------ | ----- | ----- | ----- |
-| Contested grounding (`conflicting docs`) | **10/10** | **0/10** | McNemar 10/0, p = 0.002 — registry SoT |
-| Docs routing | **10/10** | **0/10** | McNemar 10/0, p = 0.002 — messy invents `audit all` |
-| Canonical grounding | 10/10 | 8/10 | Pass rate not significant; messy still burns tokens |
-| Skill routing + customize | 10/10 | 10/10 | Tied — caller `AGENTS.md` already teaches both |
-| Grounding tokens (messy − clean) | — | median **~312k** more | Bootstrap 95% CI on mean excludes 0 |
+Two scenarios did **not** show a clean accuracy win:
 
-Side-by-side excerpts (median sequential run): [evidence/transcripts/](agent-suites/evidence/transcripts/) — clean registry hop + correct webhook vs messy conflict thrash and forbidden `audit all`.
+- **Canonical grounding** — Pass rates were close; the difference was not significant. Cost still favored clean.
+- **Skill routing + customize** — Tied. The caller `AGENTS.md` already encodes both rules, so the fixtures did not separate on those prompts.
 
-### What this does not prove
+Charts regenerate from `SUMMARY.json` via `bun run agent:evidence:charts`.
 
-- Not a general coding-task / SWE-bench success claim
-- Live model and prompt variance; fixture tasks only
-- Skill/customize may not separate when `AGENTS.md` already encodes the correct rule
-- Canonical pass rate alone understates cost — messy often lucks into the answer while spending far more tokens
+### Limits
+
+Fixture A/B on Skeleton’s own contract (live model + prompt variance) — not a general coding-task or SWE-bench claim. Skill/customize may not separate when the entry doc already teaches the correct rule.
+
+### Industry context
+
+Not a Skeleton measurement — broader context research points the same direction: more context isn’t free. A [2026 study of repo-level context files](https://doi.org/10.48550/arxiv.2602.11988) (438 tasks) found human-written files helped ~4% on average, generated files hurt ~3%, and both raised inference cost >20%. A [2025 METR trial](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/) found experienced OSS developers took 19% longer with early-2025 AI tools while believing they were faster. Those papers motivate small instructions and explicit ownership; they do **not** measure this tool.
 
 ## Quick start
 
@@ -98,7 +97,7 @@ Flag details: [install](docs/developer/install.md).
 - **Prose policy** (optional plugins) — YAML pattern rules; idle with no plugins
 - **Shell / JSON syntax** — lightweight checks on changed `.sh` and `.json` files
 
-Skeleton doesn't replace your code gates. Keep TypeScript, Python, Nx, pytest, etc. in the repo that owns them.
+Skeleton doesn't replace your code gates. Keep TypeScript, Python, Nx, pytest, and the rest in the repo that owns them.
 
 ## The `.skeleton/` contract
 
@@ -123,7 +122,7 @@ Register it:
 skeleton register docs/developer/api.md
 ```
 
-Synced skills stay pristine. Project overrides live in `.skeleton/customize/<slug>.md` and inject when the skill is read. No editing copied `SKILL.md` files.
+Synced skills stay pristine. Project overrides live in `.skeleton/customize/<slug>.md` and inject when the skill is read — no editing copied `SKILL.md` files.
 
 ## Commands
 
@@ -154,8 +153,6 @@ CI: `skeleton validate changed --base origin/main` (global rules first, then cha
 
 ## Ecosystem
 
-The model is simple:
-
 | Layer             | Role                                                                                           |
 | ----------------- | ---------------------------------------------------------------------------------------------- |
 | **Skeleton**      | Defines and checks the SSOT contract                                                           |
@@ -187,6 +184,8 @@ bun run check
 
 `bun run check` = lint + test + typecheck + build + `audit:self`.
 
-`validate:changed` is docs/config only for path-scoped work — it skips code/config extensions (see table above). Owned skill-body edits need `audit skills`. All-skip, owned skill paths (alone or mixed with docs), and missing paths exit non-zero. For code: `bun test`, `bun run typecheck`, `bun run build`.
+`validate:changed` is docs/config only for path-scoped work — it skips code/config extensions (see table above). Owned skill-body edits need `audit skills`. All-skip, owned skill paths (alone or mixed with docs), and missing paths exit non-zero.
+
+For code: `bun test`, `bun run typecheck`, `bun run build`.
 
 Optional: `brew install pre-commit` (or `pipx install pre-commit`), then `pre-commit install` to wire `.pre-commit-config.yaml`.

package/dist/cli.js

@@ -17385,6 +17385,11 @@ function collectCoverageCandidateFiles(root, exclude) {
   }
   return [...files];
 }
+function excludeForeignSkillDocMetaPaths(docMetaPaths, skillIndex) {
+  if (!skillIndex)
+    return docMetaPaths;
+  return docMetaPaths.filter((rel) => !isForeignSkillPath(rel, skillIndex));
+}
 function collectDocMetaPaths(config, root, registryPaths, skillIndex) {
   const paths = [];
   for (const abs of expandPatterns(root, ["docs/*/README.md"], mergedExcludes(config))) {
@@ -17409,7 +17414,7 @@ function collectDocMetaPaths(config, root, registryPaths, skillIndex) {
       paths.push(normalizeRelPath(relative4(root, abs)));
     }
   }
-  return [...new Set(paths)];
+  return excludeForeignSkillDocMetaPaths([...new Set(paths)], skillIndex);
 }
 function validateScanRoots(config, root) {
   const missing = [];
@@ -17419,11 +17424,11 @@ function validateScanRoots(config, root) {
   }
   return missing;
 }
-function filterDocMetaPaths(docMetaPaths, paths) {
+function filterDocMetaPaths(docMetaPaths, paths, skillIndex) {
   if (paths.length === 0)
-    return docMetaPaths;
+    return excludeForeignSkillDocMetaPaths(docMetaPaths, skillIndex);
   const normalizedPaths = paths.map((path) => normalizeRelPath(path));
-  return docMetaPaths.filter((rel) => normalizedPaths.some((path) => rel === path || rel.startsWith(`${path}/`)));
+  return excludeForeignSkillDocMetaPaths(docMetaPaths.filter((rel) => normalizedPaths.some((path) => rel === path || rel.startsWith(`${path}/`))), skillIndex);
 }
 function filterToPaths(files, paths, root) {
   const normalizedPaths = paths.map((path) => normalizeRelPath(path));
@@ -17516,7 +17521,7 @@ function createContext(options = {}) {
     root,
     config,
     files,
-    docMetaPaths: options.paths && options.paths.length > 0 ? filterDocMetaPaths(allDocMetaPaths, options.paths) : allDocMetaPaths,
+    docMetaPaths: options.paths && options.paths.length > 0 ? filterDocMetaPaths(allDocMetaPaths, options.paths, skillIndex) : allDocMetaPaths,
     registryPaths: registry.paths,
     registryHasTableHeader: registry.hasTableHeader,
     retiredSkills: new Set(retiredSkills(config)),

package/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "@csark0812/skeleton",
-	"version": "1.5.4",
+	"version": "1.5.5",
 	"description": "SSOT audit CLI for agent harness repos",
 	"type": "module",
 	"bin": {
@@ -53,6 +53,7 @@
 		"agent:test:doctor": "agent-test --doctor",
 		"agent:test:validate": "agent-test --validate-only --validate-paths --suites-dir agent-suites",
 		"agent:evidence:aggregate": "bun scripts/agent-evidence/aggregate-compares.ts",
+		"agent:evidence:charts": "bun scripts/agent-evidence/render-charts.ts",
 		"agent:evidence:excerpt": "bun scripts/agent-evidence/excerpt-transcript.ts",
 		"agent:evidence:archive": "bash scripts/agent-evidence/archive-run.sh",
 		"prepack": "bun test && bun run typecheck && bun run build"