efaimo 0.1.0 → 0.1.2

package/docs/METHODOLOGY.md

@@ -48,6 +48,29 @@ wrapped total. For the other two serializations no wrapper exists, and summing
 per-tool counts may differ from the total by a few tokens, because a tokenizer
 can merge characters across element boundaries when tools are concatenated.
 
+## The context window we compare against
+
+Alongside the absolute token count, efaimo prints the share of a context window
+it represents. There is no correct denominator for that: the window belongs to
+whichever model the host is running, not to the server being measured. Current
+frontier Claude models (Fable 5, Opus 5, Opus 4.8/4.7/4.6, Sonnet 5, Sonnet
+4.6) are **1M**, so that is the default. Haiku 4.5 is 200k, as are many
+non-Claude and local models, where identical tool definitions cost five times
+the share.
+
+Because the denominator is an assumption rather than a measurement, efaimo
+always names it in the output ("~4.3% of a 1M window") and lets you set your
+own with `--window` (since 0.1.1; 0.1.0 hardcoded 200k and printed `of a 200k window`):
+
+```bash
+npx efaimo weigh "npx -y my-server" --window 200000
+```
+
+**No rule grades on this share.** The cost thresholds (E127, E128) are absolute
+token counts, so changing `--window` changes what you read and never changes a
+grade. The absolute number is the one efaimo stands behind; the share is a
+readability aid on top of it.
+
 ## Skills
 
 For Agent Skills efaimo reports the three progressive-disclosure levels defined by
@@ -81,3 +104,50 @@ prompt-cache hits and reproducibility.
 efaimo estimates the **cost of tool/skill definitions sitting in context**. It does
 not measure per-call argument/result tokens, host framing, or model-side reasoning.
 For end-to-end token accounting, instrument your agent run directly.
+
+## `efaimo test`: what the verdict means
+
+The verdict is a significance test, not a threshold on the size of the gap.
+
+Each arm runs N trials of the same task, one with the skill in the system
+prompt and one without, and an LLM judge returns PASS or FAIL per trial. The
+two arms form a 2x2 table, and the report carries:
+
+- a **two-sided Fisher exact p**. Fisher rather than chi-squared because the
+  counts are single digits, where the chi-squared approximation is not
+  trustworthy;
+- a **95% Newcombe interval on the delta**, built from each arm's Wilson
+  interval. Wilson rather than the normal approximation because runs land on
+  the boundaries (8/8, 0/8), where the normal interval has zero width and
+  would claim certainty from eight trials.
+
+`helps` and `hurts` require **p < 0.05**. Anything else is `no measurable
+effect`, however large the gap looks, and the report says so with the p
+attached. The default is 20 trials per arm: below roughly that, a partial
+result cannot reach significance at all.
+
+Until 2026-08-02 the rule was `>= +15 points helps`, with a default of 5
+trials. At that size 5/5 against 4/5 is +20 points and p = 1.0000.
+
+Two things the judge does, which matter for reading a result:
+
+- it runs at **temperature 0**, so the same answer grades the same way twice.
+  The subject arm stays at temperature 1, because normal behaviour is the thing
+  being measured;
+- a reply that is neither PASS nor FAIL is **excluded and counted separately**,
+  not scored as a failure. A refusal or an API error is not evidence about the
+  skill.
+
+### What this design still cannot tell you
+
+Named here rather than left for a reader to discover:
+
+- **The judge is the same model as the subject.** A separate judge model would
+  be stronger.
+- **The control arm receives no system prompt at all**, so "the skill's
+  content" is confounded with "having any system prompt". A length-matched
+  placebo is the fix and is not implemented.
+- **Two scenarios is a demonstration, not a benchmark.** The committed runs
+  show the method works, not that skills in general do or do not help.
+
+Treat a single scenario as evidence about that scenario.

package/docs/RULES.md

@@ -7,43 +7,49 @@ Grade starts at 100; each error costs 15, each warning 5, each info 1.
 `A >= 90, B >= 80, C >= 70, D >= 60, else F`.
 
 MCP checks split into **2026-07-28 readiness** (E101-E118) and **quality**
-(E121-E130); skill checks are S1xx. **Only quality and skill findings are
+(E121-E128 and E130, with no E129); skill checks are S1xx. **Only quality and skill findings are
 graded.** Readiness findings are reported as a separate, ungraded **migration
-diff**: the stateless revision they target is a locked Release Candidate
-(2026-05-21) that does not finalize until 2026-07-28, and not yet migrating to
-an unreleased spec is a to-do, not a defect. Readiness severities still order
-the diff by urgency, but the diff never affects the grade, the badge, or the
-exit code (including `--strict`). Once the spec ratifies, expect readiness to
-start counting in a minor release. Source-pattern rules are heuristics and say
-so in their detail line.
+diff**: the two answer different questions. The grade is what a model
+experiences from this tool surface; readiness is whether a 2026-07-28 client can
+talk to this server. Combining them scored the official reference server a C(79)
+for a tool surface that is an A(95), which is why they are two numbers printed
+next to each other rather than one. Readiness severities still order the diff by
+urgency, but the diff never affects the grade or the badge, and never affects
+the exit code unless you ask for it with `--strict-readiness`. Source-pattern
+rules are heuristics and say so in their detail line.
 
 ## MCP readiness (E101-E118)
 
 | id | sev | what it catches | source of truth |
 |---|---|---|---|
-| E101 | warn | depends on a pre-`2026-07-28` SDK line (TS `@modelcontextprotocol/sdk` 1.x, Python `mcp` 1.x) instead of the 2.x stateless line | changelog / SDK releases |
-| E102 | warn | source uses **Sampling** (`sampling/createMessage`), deprecated (SEP-2577) | draft changelog |
-| E103 | warn | source uses **Roots** (`roots/list`), deprecated (SEP-2577) | draft changelog |
-| E104 | warn | server declares the **logging** capability or source uses MCP Logging, deprecated (SEP-2577); `logging/setLevel` is removed | draft changelog |
-| E105 | warn | the server does not answer a bare stateless `tools/list` (timeout, crash, or a not-initialized error), so it requires the removed `initialize` handshake. A server that answers (even with an error) is judged by E106/E107/E118 instead | SEP-2567 / stateless core |
-| E106 | warn | `server/discover` is not implemented (MUST in the RC, SEP-2575) | draft changelog |
-| E107 | info | results omit the required `resultType` field (`"complete"` \| `"input_required"`) | draft changelog |
-| E108 | info | source relies on removed SSE resumability (`Last-Event-ID`) | draft changelog |
+| E101 | warn | depends on a pre-`2026-07-28` SDK line (TS `@modelcontextprotocol/sdk` 1.x, Python `mcp` 1.x) instead of the stateless successors (TS `@modelcontextprotocol/server` 2.x, Python `mcp` 2.x) | changelog / SDK releases |
+| E102 | warn | source uses **Sampling** (`sampling/createMessage`), deprecated (SEP-2577) | 2026-07-28 changelog |
+| E103 | warn | source uses **Roots** (`roots/list`), deprecated (SEP-2577) | 2026-07-28 changelog |
+| E104 | warn | server declares the **logging** capability or source uses MCP Logging, deprecated (SEP-2577); `logging/setLevel` is removed | 2026-07-28 changelog |
+| E105 | warn | the server rejects a bare stateless `tools/list` with a not-initialized error, or exits on it, so it still requires the removed `initialize` handshake. A timeout is reported as unverified statelessness rather than as this warn, and a server that answers (even with another error) is judged by E106/E107/E118 instead | SEP-2567 / stateless core |
+| E106 | warn | `server/discover` is not implemented (MUST in 2026-07-28, SEP-2575) | 2026-07-28 changelog |
+| E107 | info | results omit the required `resultType` field (`"complete"` \| `"input_required"`) | 2026-07-28 changelog |
+| E108 | info | source relies on removed SSE resumability (`Last-Event-ID`) | 2026-07-28 changelog |
 | E109 | info | auth advertises deprecated DCR (RFC 7591) with no detected CIMD support | PR #2858 / CIMD |
-| E110 | warn | source uses legacy elicitation (`elicitation/create`); replaced by MRTR `input_required` results (SEP-2322) | draft changelog |
+| E110 | warn | source uses legacy elicitation (`elicitation/create`); replaced by MRTR `input_required` results (SEP-2322) | 2026-07-28 changelog |
 | E111 | info | in-process session-state patterns; statelessness expects server-minted handles in tool args | stateless core |
-| E112 | warn | `tools/list` order is nondeterministic across connections (hurts prompt-cache hits) | draft changelog |
+| E112 | warn | `tools/list` order is nondeterministic across connections (hurts prompt-cache hits) | 2026-07-28 changelog |
 | E113 | info | no Server Card found at `/.well-known/mcp` (discovery metadata; heuristic path) | roadmap / Server Card WG |
-| E114 | info | source uses the removed `ping` utility | draft changelog |
-| E115 | info | source uses `resources/subscribe`, replaced by `subscriptions/listen` | draft changelog |
+| E114 | info | source uses the removed `ping` utility | 2026-07-28 changelog |
+| E115 | info | source uses `resources/subscribe`, replaced by `subscriptions/listen` | 2026-07-28 changelog |
 | E116 | warn | server prints non-JSON noise on stdout (breaks stdio framing) | transport basics |
-| E117 | warn | only the deprecated HTTP+SSE transport worked; Streamable HTTP failed | draft changelog |
-| E118 | warn | `tools/list` result omits the now-required `ttlMs`/`cacheScope` cache fields (SEP-2549, CacheableResult; also required on prompts/list, resources/list, resources/read, resources/templates/list, server/discover) | draft changelog |
+| E117 | warn | only the deprecated HTTP+SSE transport worked; Streamable HTTP failed | 2026-07-28 changelog |
+| E118 | warn | a `tools/list` or `server/discover` result omits the required `ttlMs`/`cacheScope` cache fields (SEP-2549, CacheableResult; also required on prompts/list, resources/list, resources/read, resources/templates/list). Both surfaces efaimo calls are measured and the message names the ones that failed. `server/discover` counts only since the published spec made `DiscoverResult` extend `CacheableResult`; under the locked RC it did not | 2026-07-28 changelog |
 
-## MCP quality (E121-E130)
+## MCP quality (E121-E128, E130)
+
+There is no E129. The range used to be written "E121-E130", which implies a
+contiguous block and invites a lookup that finds nothing; ids are stable and
+never reused, so a gap is permanent and is better stated than smoothed over.
 
 | id | sev | what it catches |
 |---|---|---|
+| E000 | warn | a rule threw and was skipped, so this report is incomplete. **Reported but never scored**: a broken rule is our defect, not the target's, and must not cost it points. It exists because `runRules` used to swallow the exception silently, which made a crashing rule indistinguishable from a passing one and silently RAISED the grade |
 | E121 | error/warn | tool description missing, a placeholder, or under ~20 chars |
 | E122 | warn | description misses 3+ of 4 quality axes (length, when-to-use, params documented, mentions result), mirrors Glama's tool-definition-quality dimensions, computed locally |
 | E123 | warn/info | no tools declare annotations; a destructive-looking tool lacks `destructiveHint` |
@@ -52,7 +58,7 @@ so in their detail line.
 | E126 | warn/info | tool names mix conventions or exceed 64 chars |
 | E127 | warn | a single tool definition costs 800+ tokens (needs `weigh` data) |
 | E128 | warn | total tool-definition tax over ~10k / ~25k tokens, estimated (needs `weigh` data); hard CI failure is left to the `--max-tokens`/`--diff` gate |
-| E130 | info | instruction-injection patterns in a tool description or server instructions (shallow heuristic, never a security verdict; use a dedicated scanner for depth) |
+| E130 | info | instruction-injection patterns in a tool description or server instructions (shallow heuristic, never a security verdict; use a dedicated scanner for depth). **Reported but never scored** |
 
 ## Skills (S101-S106)
 
@@ -62,7 +68,7 @@ so in their detail line.
 | S102 | warn/info | description too thin or never says when to use the skill | agentskills.io |
 | S103 | error/warn | duplicate skill names, or two skills with heavily overlapping descriptions (trigger collision) | internal |
 | S104 | warn/info | metadata over ~100 tokens, body over ~5k tokens or 500 lines, or long body with no progressive disclosure | agentskills.io |
-| S105 | info | instruction-injection patterns in the skill body or description (shallow heuristic, never a security verdict) | internal |
+| S105 | info | instruction-injection patterns in the skill body or description (shallow heuristic, never a security verdict). **Reported but never scored** | internal |
 | S106 | error/warn/info | referenced file missing, escaping the skill dir (`..`), or nested more than one level deep | agentskills.io |
 
 ## Notes on heuristics

package/examples/contoso-crm-import/SKILL.md

@@ -0,0 +1,21 @@
+---
+name: contoso-crm-import
+description: Format a contact list as a Contoso CRM bulk-import file. Use when the user asks to prepare, convert, or export contacts for Contoso CRM import.
+license: Apache-2.0
+---
+
+# Contoso CRM bulk-import format
+
+Convert a contact list into the exact file Contoso CRM accepts on import. These
+rules are an internal convention and are not guessable; follow them exactly.
+
+## Output rules
+1. Semicolon-delimited. No header row.
+2. Four columns, in this order: member_id;display_name;email;source
+3. member_id: the first three letters of the last name, uppercased, then a hyphen,
+   then a row number zero-padded to four digits starting at 0001 (0001, 0002, ...).
+4. display_name: the full name exactly as given.
+5. email: lowercased.
+6. source: always the literal value import-2026.
+
+Output only the rows.

package/examples/csv-cleanup/SKILL.md

@@ -0,0 +1,17 @@
+---
+name: csv-cleanup
+description: Clean and validate CSV data. Use when the user asks to deduplicate, trim, or sanity-check a CSV before import.
+license: Apache-2.0
+---
+
+# CSV cleanup
+
+Use this when the user wants to tidy CSV data.
+
+## Steps
+1. Detect the delimiter (comma, tab, or semicolon) from the header row.
+2. Trim leading and trailing whitespace from every cell.
+3. Drop rows that are exact duplicates of an earlier row.
+4. Report the final row and column counts, and how many duplicates were removed.
+
+Always show the cleaned result and a one-line summary of what changed.

package/examples/scenario.crm.yaml

@@ -0,0 +1,22 @@
+# efaimo test scenario: a skill that encodes a convention the model cannot guess.
+# Contrast with scenario.example.yaml, where a capable model needs no skill.
+# Real run (spends tokens): npx efaimo test examples/scenario.crm.yaml --live
+name: contoso-crm-import helps on an unknowable format
+skill: ./contoso-crm-import
+model: claude-sonnet-5
+trials: 8
+task: |
+  Prepare these contacts for Contoso CRM import:
+
+  Ada Lovelace, ADA@Example.com
+  Grace Hopper, Grace.Hopper@NAVY.MIL
+  Alan Turing, a.turing@Bletchley.uk
+judge: |
+  PASS if the answer contains these three rows (ignore any surrounding prose or code
+  fences), each semicolon-delimited as member_id;display_name;email;source:
+    LOV-0001;Ada Lovelace;ada@example.com;import-2026
+    HOP-0002;Grace Hopper;grace.hopper@navy.mil;import-2026
+    TUR-0003;Alan Turing;a.turing@bletchley.uk;import-2026
+  The member_id (first three letters of the last name uppercased, a hyphen, then a
+  four-digit zero-padded row number), the lowercased email, and the literal source
+  value import-2026 must all be exact. Any other format is FAIL.

package/examples/scenario.example.yaml

@@ -0,0 +1,17 @@
+# efaimo test scenario: does the csv-cleanup skill actually help?
+# Dry run (no API calls):   npx efaimo test examples/scenario.example.yaml
+# Real run (spends tokens): npx efaimo test examples/scenario.example.yaml --live
+name: csv-cleanup helps on a messy CSV
+skill: ./csv-cleanup
+model: claude-sonnet-5
+trials: 8
+task: |
+  Clean this CSV and summarize what you changed:
+
+  name , email ,city
+  Ada ,ada@example.com, London
+  Ada ,ada@example.com, London
+  Grace,grace@example.com ,New York
+judge: |
+  PASS only if the answer (a) removes the duplicate Ada row, (b) trims the stray
+  whitespace around cells, and (c) states the final row count. Otherwise FAIL.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "efaimo",
-  "version": "0.1.0",
+  "version": "0.1.2",
   "description": "Audit what your agent loads: quality and context cost for MCP servers and Agent Skills",
   "keywords": [
     "mcp",
@@ -46,6 +46,7 @@
   },
   "files": [
     "dist",
+    "examples",
     "SKILL.md",
     "NOTICE",
     "CHANGELOG.md",
@@ -57,11 +58,12 @@
     "build": "tsc -p tsconfig.build.json",
     "typecheck": "tsc --noEmit",
     "test": "vitest run",
+    "spec:drift": "node scripts/spec-drift.mjs",
     "test:watch": "vitest",
     "prepublishOnly": "npm run typecheck && npm run build && npm run test"
   },
   "dependencies": {
-    "@modelcontextprotocol/sdk": "^1.29.0",
+    "@modelcontextprotocol/sdk": "^1.30.0",
     "commander": "^15.0.0",
     "cross-spawn": "^7.0.6",
     "gpt-tokenizer": "^3.4.0",