efaimo 0.1.1 → 0.2.0

package/CHANGELOG.md

@@ -4,6 +4,126 @@ All notable changes to efaimo are documented here. Format follows
 [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); this project uses
 [Semantic Versioning](https://semver.org/).
 
+## [0.2.0] - 2026-09-02
+
+### Added
+
+- **`efaimo find`**: does anything actually surface these tools? A host can
+  mark tools `defer_loading: true`, which keeps their definitions out of the
+  context window until a search returns them, and Anthropic recommends that
+  once definitions pass ~10k tokens. Under deferral a tool nothing surfaces
+  costs no context and provides no capability. Two numbers, offline and
+  deterministic, no API key:
+  - `distinct`, the headline: how many tools own a term no other tool in the
+    catalog has. A tool with none cannot be matched by any query that does not
+    also match a competitor, which follows from the index rather than from a
+    model of how anyone searches. Measured on `@playwright/mcp` 0.0.78: 22 of
+    24, with `browser_close` and `browser_navigate` owning nothing.
+  - `probe`, secondary and labelled: a simulated BM25 search over the four
+    fields the tool search documentation names as searchable. It reads 100% on
+    both the official reference server and playwright, so the output says on
+    the page that it is a floor test and not a ranking.
+  - New ungraded rule family E141-E145, `--min-distinct` as the CI gate,
+    `--top` to change the simulated result window, `--json` / `--md`.
+    Reasoning in ADR-030; method and its limits in docs/METHODOLOGY.md.
+- `rulesVersion` in every JSON envelope, because the tool version does not
+  identify a ruleset: a patch release can change what a rule fires on, and a
+  published grade is only reproducible next to the rules that produced it. The
+  rule inventory is pinned by a test (ADR-032).
+- `--judge-model` and scenario key `judge_model` for `efaimo test`, so the
+  model under test no longer has to grade its own answers. Subject and judge
+  are routed independently and may be different providers.
+- `--no-timestamp` on `weigh`, `check` and `find`.
+
+### Fixed
+
+- **`efaimo test --live` could not run on its own default model.** The runner
+  sent `temperature` on every request, and Claude removed sampling parameters
+  from the 4.7 line onward, so `claude-sonnet-5` rejected the first call with a
+  400 that is correctly not retried. Sampling parameters now go only to models
+  that still accept them, the request body is a pure function with a test
+  against it, and where the judge cannot be pinned the report says so instead
+  of implying a determinism it does not have. ADR-031.
+- `efaimo test` reports a Fisher exact p on the **unparseable** counts as well
+  as on the passes, and calls a run `inconclusive` when the excluded trials are
+  skewed across the arms: two pass rates computed over different populations
+  cannot be subtracted.
+- `--anthropic` names the model it measured against, in the output and in the
+  JSON. Claude model lines do not share a tokenizer, so a count without its
+  model cannot be reproduced. This is the rule `--window` already followed for
+  the context-window denominator.
+- `weigh --out` no longer writes `generatedAt` into the baseline. A baseline
+  exists to be compared against, and a field that differs on every write is the
+  one field a comparison must ignore.
+
+### Changed
+
+- docs/RULES.md now cites Anthropic's tool search documentation for E125's
+  lower threshold, which previously carried no source at all (the MCP quality
+  table has never had a source column). The thresholds themselves are
+  unchanged, and the citation covers the 30 boundary only: Anthropic writes
+  that tool selection "degrades once you exceed 30-50 available tools", which
+  says nothing about the 60 boundary, and the table says so.
+
+## [0.1.2] - 2026-08-03
+
+The 0.1.2 code backlog: edge-case fixes surfaced by the five-agent review, none
+of which changes a published number or the documented command surface, plus the
+security and CI hardening already staged.
+
+### Fixed
+
+- `check --mcp <directory>` with no source files and no MCP SDK dependency now
+  fails instead of returning a clean A(100): an empty scan is a failure, never a
+  pass.
+- S101: a skill with no frontmatter reports its missing `name` and `description`
+  as errors, so a total absence no longer grades higher than a broken one.
+- E105: a server that exits on a bare stateless `tools/list` is flagged. The
+  crash outcome was unhandled, so a crashing server looked more migrated than
+  one that answers with a not-initialized error.
+- E124: the "N of M parameters lack descriptions" summary is no longer dropped
+  by the display cap on schemas with five or more oversized enums.
+- S106: the escape check is lexical (the resolved path), not a `..` substring,
+  so a file merely named `notes..md` is not flagged as escaping.
+- A Poetry `mcp = "^2.0"` constraint in `pyproject.toml` is read as the 2.x
+  line, not legacy; the version regex had captured the `=` assignment.
+- `weigh --diff` with a skill baseline fails with a clear message instead of a
+  raw TypeError; a server baseline is required.
+- `--max-tokens` and `--allow-increase` reject non-positive / negative values
+  instead of turning the gate into an always-fail with a nonsense message.
+- `efaimo test --live` against an OpenAI o-series reasoning model no longer
+  sends an unsupported `temperature` (a 400) and gives it token headroom.
+- The live test harness retries a transient 429 or 5xx with backoff instead of
+  discarding every already-paid trial on the first blip.
+- The test report prints a clean note instead of "95% interval NaN to NaN" when
+  an arm produces no scoreable trial.
+- A dead stdio target fails in about one `--timeout`, not two: the legacy
+  handshake and the stateless fallback now share the budget.
+- Internal: dropped a dead exported constant and the unused per-pattern
+  `severity` on the injection heuristics (every hit is info + ungraded by
+  design).
+
+### Security
+
+- The lockfile was re-resolved so the dev/CI tree carries
+  `@hono/node-server` 2.0.12 instead of the 1.19.x line that
+  GHSA-frvp-7c67-39w9 covers (moderate, path traversal in `serve-static`, in
+  the half of the MCP SDK this CLI never executes). A fresh `npx efaimo`
+  install already resolved the patched line; this aligns the committed tree
+  with what users actually get, and `pnpm audit --prod` is clean again.
+
+### Changed
+
+- `@modelcontextprotocol/sdk` floor moved within-range from ^1.29.0 to ^1.30.0
+  (what a fresh install resolves anyway). The full suite (136 tests) is green.
+- CI: the reference-server smoke is pinned to
+  `@modelcontextprotocol/server-everything@2026.7.4`. Per-commit CI should be
+  deterministic; news about the reference server's latest belongs to a
+  scheduled run, not to a push.
+- CI: `spec-drift.yml` and the example workflow now pin actions by commit SHA
+  like the other workflows, and the example declares least-privilege
+  `permissions`.
+
 ## [0.1.1] - 2026-08-02
 
 Everything below was on `main` and unpublished. That gap was itself the
@@ -32,7 +152,7 @@ that notices the next revision by itself.
   hostile server could put ESC[1A ESC[2K in a tool name and rewrite the grade
   line printed above its own finding, so the reader saw "grade A (100), 0
   errors" while the contradiction was erased. `--no-color` did not help; that
-  only suppresses colours efaimo adds.
+  only suppresses colors efaimo adds.
 - `.env` loading is restricted to `ANTHROPIC_API_KEY` and `OPENAI_API_KEY`.
   It previously set ANY key from a working-directory `.env`, so a cloned
   repository could set `NODE_TLS_REJECT_UNAUTHORIZED=0` and silently disable
@@ -40,8 +160,8 @@ that notices the next revision by itself.
   key.
 - Skill references that resolve outside the skill directory are no longer read
   by `weigh`. S106 already warned about them, but only after the read.
-- `SKILL.md` is read through the same 512KB-class cap as referenced files
-  (2MB), instead of an uncapped read.
+- `SKILL.md` is read through a capped read (2MB) on the same guarded path that
+  caps referenced files at 512KB, instead of an uncapped read.
 - The published composite Action no longer interpolates its inputs into a bash
   `run:` body, which was script injection in the consumer's runner.
 
@@ -101,7 +221,7 @@ that notices the next revision by itself.
   extend `CacheableResult`, so that surface is required now, is measured now,
   and the message names whichever surface failed.
 
-## [0.1.0]
+## [0.1.0] - 2026-07-18
 
 First release.
 
@@ -114,7 +234,7 @@ First release.
   broken or auth-gated server is skipped with a reason, not fatal.
   `--out`/`--diff`/`--max-tokens`/`--allow-increase` for CI budget gates,
   `--badge` for a shields endpoint.
-- `efaimo check --mcp`: a quality grade (E121-E130: descriptions, schemas,
+- `efaimo check --mcp`: a quality grade (E121-E128, E130: descriptions, schemas,
   annotations, cost) plus a separate, ungraded 2026-07-28 migration diff
   (E101-E118: what the stateless spec breaks and how to fix it, each item naming
   its SEP). Speaks both the legacy handshake and bare stateless requests, so
@@ -138,5 +258,8 @@ First release.
   Documented rule set (`docs/RULES.md`), token methodology (`docs/METHODOLOGY.md`),
   and integration guide (`docs/INTEGRATIONS.md`).
 
+[unreleased]: https://github.com/efaimo-ai/efaimo/compare/v0.2.0...HEAD
+[0.2.0]: https://github.com/efaimo-ai/efaimo/releases/tag/v0.2.0
+[0.1.2]: https://github.com/efaimo-ai/efaimo/releases/tag/v0.1.2
 [0.1.1]: https://github.com/efaimo-ai/efaimo/releases/tag/v0.1.1
 [0.1.0]: https://github.com/efaimo-ai/efaimo/releases/tag/v0.1.0

package/README.md

@@ -53,7 +53,7 @@ hygiene. Run it on a folder and every skill gets its own grade, not one aggregat
 **We graded 36 public Agent Skills**: every skill in `anthropics/skills`,
 `anthropics/claude-cookbooks`, and `obra/superpowers`, at pinned commits. 97%
 score an A, but even this curated set has real issues: Anthropic's own
-`claude-api` scores a C with an over-limit description and ~18k-token
+`claude-api` scores a C (73) with an over-limit description and ~18k-token
 instructions, and the median skill's instructions run ~1,632 tokens loaded on
 every trigger. Full report, the corpus manifest, and the two commands that
 reproduce it: [the Skills Quality Index](./research/skills-index/REPORT.md).
@@ -176,6 +176,56 @@ grade A (95)   quality: 0 errors  1 warning  0 info
 
 </details>
 
+### Can a search even single your tools out?
+
+A host can mark tools `defer_loading: true`, which keeps their definitions out of
+the context window until a search returns them; Anthropic recommends turning that
+on once tool definitions pass ~10k tokens. Under deferral, a tool nothing
+surfaces costs no context and provides no capability, and neither `weigh` nor
+`check` looks at that.
+
+```bash
+npx efaimo find "npx -y my-mcp-server"
+```
+
+Two numbers come back, and they are different kinds of claim.
+
+**`distinct`** is a property of your catalog: how many tools own a term no other
+tool has. A tool that owns none cannot be matched by any query that does not also
+match a competitor. That follows from the index, not from a model of how anyone
+searches, and it is the number `--min-distinct` gates on in CI.
+
+```text
+distinct  22/24 (91.7%)   tools that own a word no other tool has
+probe     24/24 (100%)    returned by a simulated search for their own description
+
+  words  tool                     vocabulary no other tool has
+      0  browser_close            none: every word also on browser_click, browser_drag, browser_drop
+      0  browser_navigate         none: every word also on browser_tabs, browser_navigate_back, browser_network_requests
+      1  browser_evaluate         evaluate
+      1  browser_hover            hover
+      2  browser_resize           window, resize
+```
+
+*(That is `@playwright/mcp` 0.0.78, a well-maintained server, verbatim. Two of
+its twenty-four tools have no word of their own, and two more own only the word
+in their own name, which nobody describing a task would type.)*
+
+Four public servers measured this way, with the commands that reproduce each
+row, are in [the findability report](./research/findability/REPORT.md). The
+weakest was Notion's official server at 16 of 24, where `API-get-user` and
+`API-get-users` are different tools and nothing in their vocabulary separates
+them.
+
+**`probe`** is a simulated BM25 search over the four fields tool search matches
+on, and it is labelled a simulation because it is one. It reads 100% on both the
+official reference server and playwright, so it is a floor test that catches an
+empty description or a literal duplicate, not a ranking. `find` says that on
+every run rather than letting a saturated number look like a good score.
+
+There is no findability grade, on purpose:
+[docs/METHODOLOGY.md](./docs/METHODOLOGY.md) has the method, its limits, and why.
+
 ## From an agent
 
 `efaimo mcp` runs efaimo as a small, read-only MCP server, so an agent can lint or
@@ -252,8 +302,16 @@ And a mitigation you cannot measure is a mitigation you are trusting on faith:
 `weigh` reports the definitions your server actually ships, which is the input
 every mitigation operates on.
 
-If deferred loading covers your whole setup, you may not need this tool. Run
-`weigh` once and find out; that answer is worth more than our opinion.
+**And there is a fourth reason we did not have when this section was written.** Deferring a tool does not remove the question, it replaces
+it: a deferred tool that no search returns costs nothing and does nothing, so
+the thing worth measuring becomes whether each tool can be picked out of the
+catalog at all. `efaimo find` measures that, and on four public servers it
+found two to eight tools apiece that own no word their neighbours lack.
+Deferral is what makes findability matter, not what makes it moot.
+
+If deferred loading covers your whole setup, run `weigh` and `find` once and
+see what it costs you and what it hides; those answers are worth more than our
+opinion.
 
 ## Install
 
@@ -296,6 +354,7 @@ More recipes (pre-commit, GitLab, editor audit, programmatic use):
 | **S101 to S106** | skills: frontmatter and trigger quality, trigger collisions, context budget, reference integrity, injection hygiene |
 | **E101 to E118** | MCP 2026-07-28 readiness: deprecated primitives, statelessness, `server/discover`, `resultType`, cache fields, transport |
 | **E121 to E128, E130** | MCP quality: description quality, annotations, schema hygiene, tool-count and token-cost budgets |
+| **E141 to E146** | MCP findability: exclusive vocabulary, whether a search returns the tool, indistinguishable pairs, names that carry no domain word |
 
 `E000` is not a finding about your server: it means one of efaimo's own rules
 threw and was skipped, so the report is incomplete. It is never scored.
@@ -303,15 +362,18 @@ threw and was skipped, so the report is incomplete. It is never scored.
 Quality (E12x-E13x) and skill (S) findings set the letter grade; readiness
 findings (E101-E118) are reported as an ungraded migration diff, which
 `--strict-readiness` can turn into a non-zero exit
-without touching the grade.
+without touching the grade. Findability findings (E14x) belong to `efaimo find`
+and never touch a grade either.
 Every finding carries a stable id you can suppress or link. See
 [docs/RULES.md](./docs/RULES.md).
 
 ## Roadmap
 
-- Harden `efaimo test`: a separately chosen judge model, confidence intervals on
-  the delta, multi-turn tool-use trials, and judge calibration, so the
-  experimental harness earns unqualified trust.
+- Harden `efaimo test`: a length-matched placebo arm (the control currently gets
+  no system prompt at all, which confounds the skill's content with having any
+  system prompt), multi-turn tool-use trials, and judge calibration, so the
+  experimental harness earns unqualified trust. A separately chosen judge model
+  and confidence intervals on the delta are done.
 - Readiness rules for the two 2026-07-28 changes that have none yet: the
   `Mcp-Method` / `Mcp-Name` request headers (SEP-2243), and Tasks moving out of
   core into the `io.modelcontextprotocol/tasks` extension (SEP-2663).

package/SKILL.md

@@ -1,9 +1,9 @@
 ---
 name: efaimo
-description: Audit MCP servers and Agent Skills for quality and context-window cost. Use when the user wants to check an MCP server's token cost, lint a skill, verify 2026-07-28 MCP spec readiness, or add an efaimo badge or CI gate.
+description: Audit MCP servers and Agent Skills for quality and context-window cost. Use when the user wants to check an MCP server's token cost, lint a skill, verify 2026-07-28 MCP spec readiness, find out whether a search can single out a server's tools under deferred tool loading, or add an efaimo badge or CI gate.
 license: Apache-2.0
 metadata:
-  version: "0.1.1"
+  version: "0.2.0"
   homepage: "https://efaimo.ai"
 ---
 
@@ -26,8 +26,8 @@ you would be willing to run yourself. A remote target (`https://...`) is not
 executed and is the safe option for something you do not trust. `--client`
 reads the MCP config files in the current directory as well as your user
 config, so running it inside a repository you did not write means starting the
-servers that repository configured. This warning lives in `SECURITY.md` and did
-not live here, which is the wrong way round: the policy is where humans look,
+servers that repository configured. Until 2026-08-02 this warning lived only in
+`SECURITY.md`, which was the wrong way round: the policy is where humans look,
 and this file is where agents take orders.
 
 - **Weigh context cost of an MCP server**
@@ -49,6 +49,11 @@ and this file is where agents take orders.
 - **Lint a skill**
   `npx efaimo check --skill ./skills/` validates frontmatter, trigger quality,
   context budget, file references, and injection patterns.
+- **Check whether a search can single a tool out**
+  `npx efaimo find "npx -y <server>"` reports how many tools own a term no
+  other tool in the catalog has. A tool that owns none cannot be matched by any
+  query that does not also match a competitor, which matters once a host defers
+  tool loading. `--min-distinct 100` is the CI gate.
 - **Gate a pull request on context cost**
   Save a baseline with `npx efaimo weigh "<server>" --out base.json`, then in CI
   run `npx efaimo weigh "<server>" --diff base.json --allow-increase 10` to fail
@@ -56,11 +61,13 @@ and this file is where agents take orders.
 
 ## Reading the output
 
-Findings carry a stable rule id (E1xx MCP readiness, E12x-E13x MCP quality, S1xx
-skills) and a severity. The letter grade covers quality and skill findings only;
-2026-07-28 readiness items appear separately as an ungraded migration diff. Token
-numbers are o200k estimates unless `--anthropic` is used; see the methodology doc
-for the full method.
+Findings carry a stable rule id (E1xx MCP readiness, E12x-E13x MCP quality,
+E14x findability, S1xx skills) and a severity. The letter grade
+covers quality and skill findings only; 2026-07-28 readiness items appear
+separately as an ungraded migration diff, and findability is a
+separate command with no grade at all. Token numbers are o200k estimates unless
+`--anthropic` is used, which names the model it measured against; see the
+methodology doc for the full method.
 
 ## Notes
 

package/dist/check/check.js

@@ -50,6 +50,14 @@ export async function checkMcpTarget(target, opts = {}) {
 }
 export function checkMcpRepoOnly(repoPath, label) {
     const repo = scanRepo(repoPath);
+    // An empty result is a failure, never a pass. With no source files read and no
+    // dependency manifest naming an MCP SDK, there was nothing to examine here, and
+    // a clean A(100) would be a grade for a directory we never looked inside. Fail
+    // loudly, the way checkSkillSet does for a missing SKILL.md.
+    if (repo.filesScanned === 0 && !repo.sdk) {
+        throw new Error(`no MCP server source found under "${repoPath}": no code files and no SDK dependency. ` +
+            `Point --mcp at the server's run command, or at a directory that contains its source.`);
+    }
     const emptyIntro = {
         targetLabel: label,
         transport: "stdio",