@csark0812/skeleton 1.5.3 → 1.5.4

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (2) hide show
  1. package/README.md +39 -12
  2. package/package.json +11 -1
package/README.md CHANGED
@@ -2,7 +2,7 @@
2
2
 
3
3
  **Source of truth for** Package overview.
4
4
 
5
- <!-- doc-meta: owner=eng | last-reviewed=2026-07-15 -->
5
+ <!-- doc-meta: owner=eng | last-reviewed=2026-07-17 -->
6
6
 
7
7
  Agent repos get messy fast. Skills get copied around, docs disagree, links go stale, and nobody remembers which file is actually canonical.
8
8
 
@@ -34,14 +34,41 @@ Skeleton answers the repo-level question: _"Does this whole thing still agree wi
34
34
 
35
35
  ## What the research says
36
36
 
37
- Research here is still early, but the direction is useful: more context isn't automatically better.
37
+ Research on agent context is still early, but the direction is useful: more context isn't automatically better.
38
38
 
39
39
  - A [2026 study of repo-level context files](https://doi.org/10.48550/arxiv.2602.11988) tested 438 coding tasks. Human-written files improved resolution by 4% on average; generated files reduced it by 3%. Both increased inference cost by more than 20%. The recommendation was pretty direct: keep instructions minimal and include what the agent can't infer.
40
40
  - A [2025 METR randomized trial](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/) found experienced open-source developers took 19% longer with early-2025 AI tools while believing they were faster.
41
41
 
42
- Neither study proves Skeleton improves task success. That needs a Skeleton-specific benchmark.
42
+ Those studies motivate Skeleton’s constraint (small useful context, clear ownership, verified SSOT). They do **not** measure Skeleton itself.
43
43
 
44
- What they do support is the constraint Skeleton is built around: keep the useful context small, make ownership clear, and verify the rest instead of trusting that it stayed correct.
44
+ ### Skeleton-specific benchmark (what we measure)
45
+
46
+ We run a paired live A/B harness (`skeleton-clean` vs `skeleton-messy`) with [`@post-print/agent-test`](https://www.npmjs.com/package/@post-print/agent-test): same prompts, differing registry / conflict structure and context profile. Tasks cover registry grounding, validation-lane choice, and customize ownership.
47
+
48
+ - Method and significance gates: [refs/llm-harness.md](refs/llm-harness.md) (target **N=10** independent compares; McNemar on pass/fail; token/tool deltas)
49
+ - Suite definitions: [agent-suites/README.md](agent-suites/README.md)
50
+ - Numbers + transcript excerpts: [agent-suites/evidence/](agent-suites/evidence/)
51
+
52
+ ### What the evidence shows (N=10)
53
+
54
+ Ten sequential live compares (2026-07-17) are summarized in [SUMMARY.md](agent-suites/evidence/SUMMARY.md). Gates passed: grounding McNemar p < 0.05 (clean > messy) and positive grounding median token Δ (`gates.readmeFinalClaimsAllowed: true`).
55
+
56
+ | Signal | Clean | Messy | Notes |
57
+ | ------ | ----- | ----- | ----- |
58
+ | Contested grounding (`conflicting docs`) | **10/10** | **0/10** | McNemar 10/0, p = 0.002 — registry SoT |
59
+ | Docs routing | **10/10** | **0/10** | McNemar 10/0, p = 0.002 — messy invents `audit all` |
60
+ | Canonical grounding | 10/10 | 8/10 | Pass rate not significant; messy still burns tokens |
61
+ | Skill routing + customize | 10/10 | 10/10 | Tied — caller `AGENTS.md` already teaches both |
62
+ | Grounding tokens (messy − clean) | — | median **~312k** more | Bootstrap 95% CI on mean excludes 0 |
63
+
64
+ Side-by-side excerpts (median sequential run): [evidence/transcripts/](agent-suites/evidence/transcripts/) — clean registry hop + correct webhook vs messy conflict thrash and forbidden `audit all`.
65
+
66
+ ### What this does not prove
67
+
68
+ - Not a general coding-task / SWE-bench success claim
69
+ - Live model and prompt variance; fixture tasks only
70
+ - Skill/customize may not separate when `AGENTS.md` already encodes the correct rule
71
+ - Canonical pass rate alone understates cost — messy often lucks into the answer while spending far more tokens
45
72
 
46
73
  ## Quick start
47
74
 
@@ -112,14 +139,14 @@ skeleton customize resolve <slug>
112
139
 
113
140
  **Validate changed** routes git diffs to the right audit:
114
141
 
115
- | Path | Action |
116
- | ----------------------------------------------------------------------------------- | ------------------------------------------------------------- |
117
- | Docs in scan perimeter | path-scoped audit |
118
- | Owned skill bodies (`SKILL.md` trees) | exit 1 → run `audit skills` |
119
- | Foreign / lockfile-synced skill bodies | skip → lint in the owning skills/toolbox repo |
120
- | `.sh`, `.bash`, `.zsh` | shellcheck or `bash -n` |
121
- | Other `.json` | JSONC-tolerant syntax check |
122
- | `.ts`, `.tsx`, `.js`, `.jsx`, `.mjs`, `.cjs`, `.py`, `package.json`, `project.json` | skip (exits 1 if all skip) |
142
+ | Path | Action |
143
+ | ----------------------------------------------------------------------------------- | --------------------------------------------- |
144
+ | Docs in scan perimeter | path-scoped audit |
145
+ | Owned skill bodies (`SKILL.md` trees) | exit 1 → run `audit skills` |
146
+ | Foreign / lockfile-synced skill bodies | skip → lint in the owning skills/toolbox repo |
147
+ | `.sh`, `.bash`, `.zsh` | shellcheck or `bash -n` |
148
+ | Other `.json` | JSONC-tolerant syntax check |
149
+ | `.ts`, `.tsx`, `.js`, `.jsx`, `.mjs`, `.cjs`, `.py`, `package.json`, `project.json` | skip (exits 1 if all skip) |
123
150
 
124
151
  Pre-commit: `skeleton validate changed --staged` (path-scoped, fast).
125
152
 
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@csark0812/skeleton",
3
- "version": "1.5.3",
3
+ "version": "1.5.4",
4
4
  "description": "SSOT audit CLI for agent harness repos",
5
5
  "type": "module",
6
6
  "bin": {
@@ -47,6 +47,14 @@
47
47
  "check": "bun run lint && bun test && bun run typecheck && bun run build && bun run audit:self",
48
48
  "start": "bun src/cli.ts --help",
49
49
  "dev": "bun src/cli.ts --help",
50
+ "agent:test:live": "agent-test --suites-dir agent-suites --live --fail-on=behavior",
51
+ "agent:test:live:compare": "agent-test --suites-dir agent-suites --live --compare-pairs skeleton-clean:skeleton-messy --fail-on=behavior --out-dir \"${TMPDIR:-/tmp}/skeleton-compare\"",
52
+ "agent:test:live:debug": "agent-test --suites-dir agent-suites --live --fail-on=behavior --debug",
53
+ "agent:test:doctor": "agent-test --doctor",
54
+ "agent:test:validate": "agent-test --validate-only --validate-paths --suites-dir agent-suites",
55
+ "agent:evidence:aggregate": "bun scripts/agent-evidence/aggregate-compares.ts",
56
+ "agent:evidence:excerpt": "bun scripts/agent-evidence/excerpt-transcript.ts",
57
+ "agent:evidence:archive": "bash scripts/agent-evidence/archive-run.sh",
50
58
  "prepack": "bun test && bun run typecheck && bun run build"
51
59
  },
52
60
  "dependencies": {
@@ -60,6 +68,8 @@
60
68
  },
61
69
  "devDependencies": {
62
70
  "@biomejs/biome": "^2.5.3",
71
+ "@cursor/sdk": "^1.0.23",
72
+ "@post-print/agent-test": "^0.2.7",
63
73
  "@types/bun": "^1.3.8",
64
74
  "@types/github-slugger": "^2.0.0",
65
75
  "typescript": "~7.0.2"