efaimo 0.1.2 → 0.2.0

package/dist/weigh/weigh.js

@@ -33,13 +33,19 @@ export async function weighServer(intro, opts = {}) {
         "token counts are estimates using the o200k_base tokenizer; hosts add fixed framing text on top (see docs/METHODOLOGY.md)",
     ];
     let anthropicExactTotal;
+    let anthropicExactModel;
     if (opts.anthropicApiKey && intro.tools.length > 0) {
-        anthropicExactTotal = await countClaudeToolTokens(intro.tools, {
+        const exact = await countClaudeToolTokens(intro.tools, {
             apiKey: opts.anthropicApiKey,
             model: opts.anthropicModel,
         });
-        if (anthropicExactTotal !== undefined) {
-            notes.push("anthropic-exact measured via /v1/messages/count_tokens (tools delta)");
+        if (exact !== undefined) {
+            anthropicExactTotal = exact.tokens;
+            anthropicExactModel = exact.model;
+            // Name the model. Claude model lines do not share a tokenizer, so this
+            // figure is exact for `exact.model` and an estimate for anything else,
+            // and a reader comparing two runs has to be able to see which is which.
+            notes.push(`anthropic-exact measured via /v1/messages/count_tokens (tools delta) against ${exact.model}`);
         }
         else {
             notes.push("anthropic-exact measurement failed; showing o200k estimates only");
@@ -56,6 +62,7 @@ export async function weighServer(intro, opts = {}) {
         framingTokens,
         instructionsTokens: count(intro.instructions ?? ""),
         anthropicExactTotal,
+        anthropicExactModel,
         notes,
     };
 }

package/docs/INTEGRATIONS.md

@@ -36,6 +36,18 @@ Ratchet MCP tool-definition cost with the budget gate (commit `base.json` once w
 - run: npx efaimo weigh "npx -y your-server" --diff base.json --allow-increase 10
 ```
 
+Keep every tool findable once a host defers your catalog. `--min-distinct` fails the run
+when a tool owns no word the other tools lack, which means no search can return
+it without also returning a competitor:
+
+```yaml
+- run: npx efaimo find "npx -y your-server" --min-distinct 100
+```
+
+Start it at the number you have today rather than at 100 and raise it as you
+fix tools; the command prints the line to copy. It refuses to run on a one-tool
+catalog, where the figure is 100% by construction, rather than passing.
+
 Post the grade on the PR with `--md` (pipe into any comment action):
 
 ```yaml
@@ -108,7 +120,9 @@ the current SDK.
 
 ## Programmatic use
 
-Every capability is also a typed function:
+Every capability is also a typed function. `analyzeFind`, `runFindRules` and the
+two `find` renderers are exported alongside the rest; the rule
+tables themselves stay private, like the other two families.
 
 ```ts
 import { weighServer, introspectServer, checkSkillSet } from "efaimo";

package/docs/METHODOLOGY.md

@@ -131,19 +131,28 @@ trials. At that size 5/5 against 4/5 is +20 points and p = 1.0000.
 
 Two things the judge does, which matter for reading a result:
 
-- it runs at **temperature 0**, so the same answer grades the same way twice.
-  The subject arm stays at temperature 1, because normal behaviour is the thing
-  being measured;
+- it is asked to sample deterministically **where the model still accepts a
+  sampling parameter**. Claude removed `temperature` from the 4.7 line onward,
+  so on Sonnet 5, Opus 5 and their siblings the judge runs at the model's own
+  default and its variance is part of the measurement. The report says so on
+  every run that this applies to, rather than claiming a determinism it does
+  not have. The subject arm is unaffected: it
+  wanted temperature 1, which is the default anyway;
 - a reply that is neither PASS nor FAIL is **excluded and counted separately**,
   not scored as a failure. A refusal or an API error is not evidence about the
-  skill.
+  skill. Exclusion is only unbiased while it falls on both arms alike, so the
+  report also carries a **two-sided Fisher exact p on the unparseable counts**,
+  and a skew significant at the same 0.05 makes the verdict `inconclusive`:
+  two pass rates computed over different populations cannot be subtracted.
 
 ### What this design still cannot tell you
 
 Named here rather than left for a reader to discover:
 
-- **The judge is the same model as the subject.** A separate judge model would
-  be stronger.
+- **The judge defaults to the same model as the subject**, so part of any
+  measured effect is a model preferring its own output. `judge_model:` in the
+  scenario, or `--judge-model`, points it at another model, and the report
+  prints both. Subject and judge may be different providers.
 - **The control arm receives no system prompt at all**, so "the skill's
   content" is confounded with "having any system prompt". A length-matched
   placebo is the fix and is not implemented.
@@ -151,3 +160,98 @@ Named here rather than left for a reader to discover:
   show the method works, not that skills in general do or do not help.
 
 Treat a single scenario as evidence about that scenario.
+
+## `efaimo find`: findability
+
+A host may mark tools `defer_loading: true`, which keeps their definitions out
+of the system-prompt prefix until a search returns them. Anthropic lists five
+conditions for turning that on, joined by "any of the following apply", and two
+of them can be checked from a tool list: **ten or more tools**, or **more than
+10k tokens of definitions**. `find` checks both and names the one that fired.
+An earlier version implemented only the token clause, which told a reader that
+a 24-tool, 3.5k-token catalog was probably loaded up front when Anthropic's
+first condition already applied to it.
+
+Under deferral, a tool nothing surfaces costs no context and provides no
+capability, so "can this be found" becomes a separate question from "what does
+this cost" and "how good is this tool".
+
+`find` answers it with two numbers that are **not the same kind of claim**, and
+prints both with their denominators. In JSON they are `data.distinct` and
+`data.probe`; the terminal uses the same two words.
+
+### `distinct`: a property of the catalog
+
+For every tool, which of its terms does no other tool in the catalog have?
+
+If the answer is none, then every word that tool contains appears somewhere
+else, so **no query exists that matches it without also matching a
+competitor**. That is not a prediction about a particular search. It follows
+from the index, and the only assumption in it is the tokenizer.
+
+The precise limit of the claim, stated because it is easy to overstate: a tool
+with no exclusive vocabulary can still be ranked *first* for a shared query,
+because BM25 also weighs term frequency and document length. What it can never
+do is come back alone. Owning a word is a much stronger position than winning a
+tie-break.
+
+Terms come from the four fields the tool search documentation names as
+searchable: **tool name, description, argument names, argument descriptions**.
+Tokenization splits camelCase, `snake_case` and `kebab-case` onto the same
+terms, lowercases, drops single characters, and does no stemming, so `page` and
+`pages` are different terms (`src/find/tokenize.ts`, identified in JSON as
+`data.method.tokenizer`, a string that changes when the splitting rules do).
+English function words are excluded from ownership as well as from queries: a
+tool can be the only one in a catalog that says "the", and counting that as
+vocabulary it owns would be true and useless.
+
+**`title` is deliberately not indexed**, though MCP tools may carry one.
+Anthropic's list of searchable fields does not include it, and a `title` is
+usually the name restated for humans, so reading it would put the tool's own
+name back into its own probe. Indexing a field the real search cannot read also
+makes the number optimistic: a tool would "own" a word no query can reach.
+Measured cost of getting this wrong: Notion's server scored 17/24 with `title`
+indexed and 16/24 without, and the 16 is the honest one.
+
+**At one tool the number cannot be computed.** Every term is trivially
+exclusive, so the figure is 100% for any tool whatsoever. The report says so and
+`--min-distinct` refuses to be evaluated rather than reporting a pass.
+
+### `probe`: a simulation, and a saturated one
+
+BM25 (k1=1.2, b=0.75) over those same fields, with each tool queried using the
+highest tf-idf terms **of its own description** and a result window of 5, which
+is the documented default number of results a search returns.
+
+Two honest limits:
+
+- The real search runs server-side. Its analyzer and parameters are not
+  published, so this is a model of a documented mechanism, not a reproduction.
+- **It saturates.** Measured on four public servers: 13/13, 24/24, 26/26 and
+  24/24, and every query variant tried (tf-idf, raw tf, leading terms) gave the
+  same 100%. The query comes from the tool's own words, which is friendlier
+  than anything a user would type. Treat it as a floor test that catches an
+  empty description or a literal duplicate, not as a ranking.
+  [The findability report](../research/findability/REPORT.md) has the numbers
+  and the commands that reproduce them.
+
+The query deliberately excludes the tool's **name** and its **title**. An
+earlier version included both and the metric could barely fail: a unique name
+token has maximal idf, so it entered every query and pulled its own tool to the
+top, and two tools with identical descriptions both scored a clean first place.
+The name stays in the index, so a name whose words echo the description still
+helps it rank.
+
+### Why there is no findability grade
+
+Because the two numbers answer different questions and only one of them is a
+measurement rather than a model, and because a letter would have inherited the
+defect the quality grade already has: it cannot see size. See ADR-030. The gate
+is `--min-distinct`, which the operator sets, on the number that can actually
+fail at two tools and up.
+
+The colour the terminal puts on `distinct` is not a grade either: red means the
+E141 condition holds (some tool owns nothing), green means it does not. An
+earlier version banded it at 95 and 80 percent, two cutoffs that appeared in no
+rule and no document, inside the one command whose design note says it does not
+grade.

package/docs/RULES.md

@@ -6,9 +6,14 @@ Severities: **error** (will bite you), **warn** (should fix), **info** (worth kn
 Grade starts at 100; each error costs 15, each warning 5, each info 1.
 `A >= 90, B >= 80, C >= 70, D >= 60, else F`.
 
+Every JSON report carries a `rulesVersion`. The tool version does not identify a ruleset on its own: a patch
+release can change what a rule fires on, so a published grade is only
+reproducible next to the ruleset that produced it.
+
 MCP checks split into **2026-07-28 readiness** (E101-E118) and **quality**
-(E121-E128 and E130, with no E129); skill checks are S1xx. **Only quality and skill findings are
-graded.** Readiness findings are reported as a separate, ungraded **migration
+(E121-E128 and E130, with no E129); skill checks are S1xx; **findability**
+(E141-E145) is a third family reported by `efaimo find`.
+**Only quality and skill findings are graded.** Readiness findings are reported as a separate, ungraded **migration
 diff**: the two answer different questions. The grade is what a model
 experiences from this tool surface; readiness is whether a 2026-07-28 client can
 talk to this server. Combining them scored the official reference server a C(79)
@@ -54,7 +59,7 @@ never reused, so a gap is permanent and is better stated than smoothed over.
 | E122 | warn | description misses 3+ of 4 quality axes (length, when-to-use, params documented, mentions result), mirrors Glama's tool-definition-quality dimensions, computed locally |
 | E123 | warn/info | no tools declare annotations; a destructive-looking tool lacks `destructiveHint` |
 | E124 | warn | many undocumented parameters, or an oversized `enum` inflating every prompt |
-| E125 | warn | tool count high enough (30+, 60+) to degrade routing and inflate context |
+| E125 | warn | tool count high enough (30+, 60+) to degrade routing and inflate context. The lower boundary is not ours: Anthropic's tool search documentation states that "Claude's ability to pick the right tool degrades once you exceed 30-50 available tools". The 60 boundary is ours, and has no published source |
 | E126 | warn/info | tool names mix conventions or exceed 64 chars |
 | E127 | warn | a single tool definition costs 800+ tokens (needs `weigh` data) |
 | E128 | warn | total tool-definition tax over ~10k / ~25k tokens, estimated (needs `weigh` data); hard CI failure is left to the `--max-tokens`/`--diff` gate |
@@ -71,6 +76,38 @@ never reused, so a gap is permanent and is better stated than smoothed over.
 | S105 | info | instruction-injection patterns in the skill body or description (shallow heuristic, never a security verdict). **Reported but never scored** | internal |
 | S106 | error/warn/info | referenced file missing, escaping the skill dir (`..`), or nested more than one level deep | agentskills.io |
 
+## Findability (E141-E145)
+
+Reported by `efaimo find`, never by `check`, and **never graded**. They answer a
+question the other two families do not: when a host marks tools
+`defer_loading: true`, the definitions stay out of the context window until a
+search finds them, so a tool nothing surfaces costs nothing and does nothing.
+Anthropic recommends turning that on once definitions pass ~10k tokens.
+
+`find` prints two measured proportions, not a letter grade (ADR-030), and they
+are different kinds of claim:
+
+- **`distinct`** is a property of the catalog. A tool that owns no term the
+  other tools lack cannot be matched by any query that does not also match a
+  competitor. No model of how anyone searches is involved. This is the headline
+  and the number `--min-distinct` gates on.
+- **`probe`** is a simulation: BM25 over the four searchable fields, each tool
+  queried with the top terms of its own description. It reads 100% on the
+  official reference server and on `@playwright/mcp`, so it is a floor test
+  that catches an empty description or a literal duplicate, not a ranking. The
+  output says so every run.
+
+Method, parameters and limits: [METHODOLOGY.md](./METHODOLOGY.md).
+
+| id | sev | what it catches |
+|---|---|---|
+| E141 | warn | the tool owns no term that no other tool has, so no search returns it without also returning a competitor. The one rule here that is not a heuristic |
+| E142 | warn | the probe does not return the tool inside the result window (default 5), or its description offers no word to search for at all |
+| E143 | warn | two tools score identically for one of their own queries, so only the alphabetical tie-break separates them |
+| E144 | info | the tool name is built only from generic words (`run`, `execute_query`), so a broad pattern search reaches it only through its description |
+| E145 | warn | Anthropic's guidance recommends tool search for a catalog this shape (10 or more tools, or over 10k tokens of definitions) **and** some tools cannot be singled out: once the catalog is deferred, those tools cost nothing and do nothing |
+| E146 | info | every word the tool owns comes from its own name, so nothing in its description is unique. Names are searchable, so this is not E141; it means only someone who already knows the name can single it out |
+
 ## Notes on heuristics
 
 Rules that scan **source code** (E101-E104, E108, E110, E111, E114, E115) are

package/examples/scenario.crm.yaml

@@ -4,7 +4,15 @@
 name: contoso-crm-import helps on an unknowable format
 skill: ./contoso-crm-import
 model: claude-sonnet-5
+# 8, because that is what the two committed runs in the README used. The dry
+# run prints what a plan this size can detect: at 8 per arm nothing under about
+# 63 points can reach p < 0.05, so raise it for anything less clear-cut than
+# these two. The default when the key is absent is 20.
 trials: 8
+
+# The judge defaults to the model above, which means a model grades its own
+# answers. Point it somewhere else to remove that. (unreleased: needs 0.2.0)
+# judge_model: claude-opus-5
 task: |
   Prepare these contacts for Contoso CRM import:
 

package/examples/scenario.example.yaml

@@ -4,7 +4,15 @@
 name: csv-cleanup helps on a messy CSV
 skill: ./csv-cleanup
 model: claude-sonnet-5
+# 8, because that is what the two committed runs in the README used. The dry
+# run prints what a plan this size can detect: at 8 per arm nothing under about
+# 63 points can reach p < 0.05, so raise it for anything less clear-cut than
+# these two. The default when the key is absent is 20.
 trials: 8
+
+# The judge defaults to the model above, which means a model grades its own
+# answers. Point it somewhere else to remove that. (unreleased: needs 0.2.0)
+# judge_model: claude-opus-5
 task: |
   Clean this CSV and summarize what you changed:
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "efaimo",
-  "version": "0.1.2",
+  "version": "0.2.0",
   "description": "Audit what your agent loads: quality and context cost for MCP servers and Agent Skills",
   "keywords": [
     "mcp",