efaimo 0.1.2 → 0.3.0

package/docs/RULES.md

@@ -6,9 +6,14 @@ Severities: **error** (will bite you), **warn** (should fix), **info** (worth kn
 Grade starts at 100; each error costs 15, each warning 5, each info 1.
 `A >= 90, B >= 80, C >= 70, D >= 60, else F`.
 
+Every JSON report carries a `rulesVersion`. The tool version does not identify a ruleset on its own: a patch
+release can change what a rule fires on, so a published grade is only
+reproducible next to the ruleset that produced it.
+
 MCP checks split into **2026-07-28 readiness** (E101-E118) and **quality**
-(E121-E128 and E130, with no E129); skill checks are S1xx. **Only quality and skill findings are
-graded.** Readiness findings are reported as a separate, ungraded **migration
+(E121-E128 and E130, with no E129); skill checks are S1xx; **findability**
+(E141-E145) is a third family reported by `efaimo find`.
+**Only quality and skill findings are graded.** Readiness findings are reported as a separate, ungraded **migration
 diff**: the two answer different questions. The grade is what a model
 experiences from this tool surface; readiness is whether a 2026-07-28 client can
 talk to this server. Combining them scored the official reference server a C(79)
@@ -54,7 +59,7 @@ never reused, so a gap is permanent and is better stated than smoothed over.
 | E122 | warn | description misses 3+ of 4 quality axes (length, when-to-use, params documented, mentions result), mirrors Glama's tool-definition-quality dimensions, computed locally |
 | E123 | warn/info | no tools declare annotations; a destructive-looking tool lacks `destructiveHint` |
 | E124 | warn | many undocumented parameters, or an oversized `enum` inflating every prompt |
-| E125 | warn | tool count high enough (30+, 60+) to degrade routing and inflate context |
+| E125 | warn | tool count high enough (30+, 60+) to degrade routing and inflate context. The lower boundary is not ours: Anthropic's tool search documentation states that "Claude's ability to pick the right tool degrades once you exceed 30-50 available tools". The 60 boundary is ours, and has no published source |
 | E126 | warn/info | tool names mix conventions or exceed 64 chars |
 | E127 | warn | a single tool definition costs 800+ tokens (needs `weigh` data) |
 | E128 | warn | total tool-definition tax over ~10k / ~25k tokens, estimated (needs `weigh` data); hard CI failure is left to the `--max-tokens`/`--diff` gate |
@@ -71,6 +76,38 @@ never reused, so a gap is permanent and is better stated than smoothed over.
 | S105 | info | instruction-injection patterns in the skill body or description (shallow heuristic, never a security verdict). **Reported but never scored** | internal |
 | S106 | error/warn/info | referenced file missing, escaping the skill dir (`..`), or nested more than one level deep | agentskills.io |
 
+## Findability (E141-E145)
+
+Reported by `efaimo find`, never by `check`, and **never graded**. They answer a
+question the other two families do not: when a host marks tools
+`defer_loading: true`, the definitions stay out of the context window until a
+search finds them, so a tool nothing surfaces costs nothing and does nothing.
+Anthropic recommends turning that on once definitions pass ~10k tokens.
+
+`find` prints two measured proportions, not a letter grade (ADR-030), and they
+are different kinds of claim:
+
+- **`distinct`** is a property of the catalog. A tool that owns no term the
+  other tools lack cannot be matched by any query that does not also match a
+  competitor. No model of how anyone searches is involved. This is the headline
+  and the number `--min-distinct` gates on.
+- **`probe`** is a simulation: BM25 over the four searchable fields, each tool
+  queried with the top terms of its own description. It reads 100% on the
+  official reference server and on `@playwright/mcp`, so it is a floor test
+  that catches an empty description or a literal duplicate, not a ranking. The
+  output says so every run.
+
+Method, parameters and limits: [METHODOLOGY.md](./METHODOLOGY.md).
+
+| id | sev | what it catches |
+|---|---|---|
+| E141 | warn | the tool owns no term that no other tool has, so no search returns it without also returning a competitor. The one rule here that is not a heuristic |
+| E142 | warn | the probe does not return the tool inside the result window (default 5), or its description offers no word to search for at all |
+| E143 | warn | two tools score identically for one of their own queries, so only the alphabetical tie-break separates them |
+| E144 | info | the tool name is built only from generic words (`run`, `execute_query`), so a broad pattern search reaches it only through its description |
+| E145 | warn | Anthropic's guidance recommends tool search for a catalog this shape (10 or more tools, or over 10k tokens of definitions) **and** some tools cannot be singled out: once the catalog is deferred, those tools cost nothing and do nothing |
+| E146 | info | every word the tool owns comes from its own name, so nothing in its description is unique. Names are searchable, so this is not E141; it means only someone who already knows the name can single it out |
+
 ## Notes on heuristics
 
 Rules that scan **source code** (E101-E104, E108, E110, E111, E114, E115) are

package/examples/scenario.crm.yaml

@@ -4,7 +4,15 @@
 name: contoso-crm-import helps on an unknowable format
 skill: ./contoso-crm-import
 model: claude-sonnet-5
+# 8, because that is what the two committed runs in the README used. The dry
+# run prints what a plan this size can detect: at 8 per arm nothing under about
+# 63 points can reach p < 0.05, so raise it for anything less clear-cut than
+# these two. The default when the key is absent is 20.
 trials: 8
+
+# The judge defaults to the model above, which means a model grades its own
+# answers. Point it somewhere else to remove that. (unreleased: needs 0.2.0)
+# judge_model: claude-opus-5
 task: |
   Prepare these contacts for Contoso CRM import:
 

package/examples/scenario.example.yaml

@@ -4,7 +4,15 @@
 name: csv-cleanup helps on a messy CSV
 skill: ./csv-cleanup
 model: claude-sonnet-5
+# 8, because that is what the two committed runs in the README used. The dry
+# run prints what a plan this size can detect: at 8 per arm nothing under about
+# 63 points can reach p < 0.05, so raise it for anything less clear-cut than
+# these two. The default when the key is absent is 20.
 trials: 8
+
+# The judge defaults to the model above, which means a model grades its own
+# answers. Point it somewhere else to remove that. (unreleased: needs 0.2.0)
+# judge_model: claude-opus-5
 task: |
   Clean this CSV and summarize what you changed:
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "efaimo",
-  "version": "0.1.2",
+  "version": "0.3.0",
   "description": "Audit what your agent loads: quality and context cost for MCP servers and Agent Skills",
   "keywords": [
     "mcp",
@@ -58,6 +58,8 @@
     "build": "tsc -p tsconfig.build.json",
     "typecheck": "tsc --noEmit",
     "test": "vitest run",
+    "preflight": "node scripts/preflight.mjs",
+    "preflight:fast": "node scripts/preflight.mjs --fast",
     "spec:drift": "node scripts/spec-drift.mjs",
     "test:watch": "vitest",
     "prepublishOnly": "npm run typecheck && npm run build && npm run test"