efaimo 0.1.2 → 0.3.0

package/CHANGELOG.md

@@ -4,6 +4,164 @@ All notable changes to efaimo are documented here. Format follows
 [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); this project uses
 [Semantic Versioning](https://semver.org/).
 
+## [0.3.0] - 2026-09-03
+
+### Added
+
+- **`check --out` and `check --diff`**: what moved between two
+  audits of the same subject. Same grammar as `weigh --out` / `weigh --diff`,
+  which is why it is a flag rather than a sixth subcommand. Nothing published
+  says whether skills and MCP servers are getting heavier or whether their
+  quality holds, because that needs the same thing measured twice and nothing
+  made the comparison cheap.
+
+  A delta is exactly the shape a broken measurement takes, so the controls a
+  careful person would run by hand are enforced instead of documented:
+
+  - **Rules drift is a hard stop.** Different `rulesVersion` between the two
+    runs means a moved grade cannot be attributed to the subject, because the
+    ruler moved too. `--allow-rules-drift` proceeds and marks
+    every grade line unattributable.
+  - **An empty pairing fails.** Nothing in common is two reports about
+    different subjects, not "no change".
+  - **The three token costs are never summed.** Metadata is carried
+    permanently, the body loads on trigger, referenced files load on demand;
+    one combined percentage would describe none of them.
+  - **A dominant mover is named beside its total**, because when one subject is
+    most of a change, the total is about that subject and not the population.
+
+  `--fail-on-regression` exits 1 when any subject present in
+  both runs scored lower.
+
+  Measured on the public skills corpus, 2026-07-17 pinned against 2026-09-03
+  HEAD, same ruleset both sides: metadata -0.8%, body +7.5%, referenced files
+  +61.4% with one skill at 96.5% of that movement; grades 0 improved and 4
+  worsened.
+
+- **`efaimo find` takes several servers**. Pass more than one
+  and their catalogs are merged into one and measured together, each tool
+  labelled with where it came from. A flag-shaped change rather than a new
+  subcommand, because the engine already took a flat tool array and only the
+  CLI surface was single-target.
+
+  The reason it is worth having: a server's author keeps their own tool names
+  apart as a matter of course, nobody coordinates across the several servers a
+  person actually installs, and the model sees one flat list with no
+  indication of origin. A single-server run is structurally unable to see the
+  collision that actually bites.
+
+  The origin is carried for the report and deliberately kept out of the index.
+  Prefixing names with their server would hand every tool a term no other tool
+  has, and a catalog of indistinguishable tools would score a perfect 100.
+  There is a test for exactly that, and it had to be rewritten once: its first
+  origin labels were "server-one" and "server-two", whose only surviving token
+  is "server" for both, so the sabotage it existed to catch passed.
+
+  Measured live on `server-everything@2026.7.4` plus `@playwright/mcp@0.0.78`:
+  37 tools merged, 35 distinct (94.6%). No cross-server collision between
+  those two, which is the right answer for a filesystem server and a browser
+  server, and the two tools owning nothing are playwright's own already
+  documented `browser_close` and `browser_navigate`.
+
+### Fixed
+
+- **S102 no longer passes filler**. The rule read
+  `description.length < 20` and then asked only whether one of four trigger
+  words appeared anywhere, so `Useful for various tasks.` scored a clean
+  A (100) in silence because "for" sits inside "for various tasks". It was
+  testing for the presence of a trigger WORD rather than for the presence of a
+  trigger. A skill whose whole body was one sentence also scored 100.
+
+  Two floors replace it, both grounded on the 38 public skills in
+  `research/skills-index/manifest.json` measured 2026-09-03, where the shortest
+  real description is 68 characters and the thinnest carries 4 distinct content
+  terms: under 40 characters, or fewer than 4 distinct terms surviving the
+  stopword list, is a warning. The second is the same measurement `find` makes
+  on tool catalogs, since a description with no distinguishing content cannot
+  be matched to a task whatever words it contains.
+
+  Impact on the corpus: none. The 38 skills produce zero S102 findings before
+  and after, and the grade distribution is unchanged. This was found by
+  sabotaging the grader while reviewing two new skills, not by a report.
+
+  Note that this changes grades in principle, so it is deliberately sequenced
+  after the 2026-09-03 delta measurement, whose reproduction is pinned to
+  `v0.2.0`.
+
+### Known
+
+- **`check --skill <dir>` does not find every skill under a tree**
+  _(present in 0.2.0 as well)_. It walks
+  `<root>/<name>/SKILL.md` and `<root>/skills/<name>/SKILL.md` but misses skills
+  inside a dot directory (`.claude/skills/<name>/`) and skills one level deeper
+  (`skills/custom_skills/<name>/`). On the corpus above it found 34 where
+  `scripts/skills-index.mjs` found 38, all four in the same repository. Two of
+  this tool's own walkers disagreeing about the same tree is a defect whichever
+  is right; a `--diff` built on the narrower walk silently compares an
+  incomplete population. Not fixed here because the fix is a design decision
+  about depth and dot directories, separate from this change.
+
+## [0.2.0] - 2026-09-02
+
+### Added
+
+- **`efaimo find`**: does anything actually surface these tools? A host can
+  mark tools `defer_loading: true`, which keeps their definitions out of the
+  context window until a search returns them, and Anthropic recommends that
+  once definitions pass ~10k tokens. Under deferral a tool nothing surfaces
+  costs no context and provides no capability. Two numbers, offline and
+  deterministic, no API key:
+  - `distinct`, the headline: how many tools own a term no other tool in the
+    catalog has. A tool with none cannot be matched by any query that does not
+    also match a competitor, which follows from the index rather than from a
+    model of how anyone searches. Measured on `@playwright/mcp` 0.0.78: 22 of
+    24, with `browser_close` and `browser_navigate` owning nothing.
+  - `probe`, secondary and labelled: a simulated BM25 search over the four
+    fields the tool search documentation names as searchable. It reads 100% on
+    both the official reference server and playwright, so the output says on
+    the page that it is a floor test and not a ranking.
+  - New ungraded rule family E141-E145, `--min-distinct` as the CI gate,
+    `--top` to change the simulated result window, `--json` / `--md`.
+    Reasoning in ADR-030; method and its limits in docs/METHODOLOGY.md.
+- `rulesVersion` in every JSON envelope, because the tool version does not
+  identify a ruleset: a patch release can change what a rule fires on, and a
+  published grade is only reproducible next to the rules that produced it. The
+  rule inventory is pinned by a test (ADR-032).
+- `--judge-model` and scenario key `judge_model` for `efaimo test`, so the
+  model under test no longer has to grade its own answers. Subject and judge
+  are routed independently and may be different providers.
+- `--no-timestamp` on `weigh`, `check` and `find`.
+
+### Fixed
+
+- **`efaimo test --live` could not run on its own default model.** The runner
+  sent `temperature` on every request, and Claude removed sampling parameters
+  from the 4.7 line onward, so `claude-sonnet-5` rejected the first call with a
+  400 that is correctly not retried. Sampling parameters now go only to models
+  that still accept them, the request body is a pure function with a test
+  against it, and where the judge cannot be pinned the report says so instead
+  of implying a determinism it does not have. ADR-031.
+- `efaimo test` reports a Fisher exact p on the **unparseable** counts as well
+  as on the passes, and calls a run `inconclusive` when the excluded trials are
+  skewed across the arms: two pass rates computed over different populations
+  cannot be subtracted.
+- `--anthropic` names the model it measured against, in the output and in the
+  JSON. Claude model lines do not share a tokenizer, so a count without its
+  model cannot be reproduced. This is the rule `--window` already followed for
+  the context-window denominator.
+- `weigh --out` no longer writes `generatedAt` into the baseline. A baseline
+  exists to be compared against, and a field that differs on every write is the
+  one field a comparison must ignore.
+
+### Changed
+
+- docs/RULES.md now cites Anthropic's tool search documentation for E125's
+  lower threshold, which previously carried no source at all (the MCP quality
+  table has never had a source column). The thresholds themselves are
+  unchanged, and the citation covers the 30 boundary only: Anthropic writes
+  that tool selection "degrades once you exceed 30-50 available tools", which
+  says nothing about the 60 boundary, and the table says so.
+
 ## [0.1.2] - 2026-08-03
 
 The 0.1.2 code backlog: edge-case fixes surfaced by the five-agent review, none
@@ -197,7 +355,8 @@ First release.
   Documented rule set (`docs/RULES.md`), token methodology (`docs/METHODOLOGY.md`),
   and integration guide (`docs/INTEGRATIONS.md`).
 
-[unreleased]: https://github.com/efaimo-ai/efaimo/compare/v0.1.2...HEAD
+[0.3.0]: https://github.com/efaimo-ai/efaimo/releases/tag/v0.3.0
+[0.2.0]: https://github.com/efaimo-ai/efaimo/releases/tag/v0.2.0
 [0.1.2]: https://github.com/efaimo-ai/efaimo/releases/tag/v0.1.2
 [0.1.1]: https://github.com/efaimo-ai/efaimo/releases/tag/v0.1.1
 [0.1.0]: https://github.com/efaimo-ai/efaimo/releases/tag/v0.1.0

package/README.md

@@ -58,6 +58,13 @@ instructions, and the median skill's instructions run ~1,632 tokens loaded on
 every trigger. Full report, the corpus manifest, and the two commands that
 reproduce it: [the Skills Quality Index](./research/skills-index/REPORT.md).
 
+That index is a photograph. [Measured twice](./research/skills-delta-2026-09-03/REPORT.md)
+re-runs the same corpus seven weeks later with the same ruleset: the cost that
+sits in context permanently moved -0.8%, the body +7.0%, and referenced files,
+which load out of sight and which no rule here counts, +61.4%. Grades moved one
+way only, 0 improved against 4 worse. The uncomfortable part points inward: the
+two columns this tool has thresholds for are the two that held.
+
 ### Does the skill actually help? (experimental)
 
 Linting tells you a skill is well-formed. It does not tell you the skill makes the
@@ -176,6 +183,56 @@ grade A (95)   quality: 0 errors  1 warning  0 info
 
 </details>
 
+### Can a search even single your tools out?
+
+A host can mark tools `defer_loading: true`, which keeps their definitions out of
+the context window until a search returns them; Anthropic recommends turning that
+on once tool definitions pass ~10k tokens. Under deferral, a tool nothing
+surfaces costs no context and provides no capability, and neither `weigh` nor
+`check` looks at that.
+
+```bash
+npx efaimo find "npx -y my-mcp-server"
+```
+
+Two numbers come back, and they are different kinds of claim.
+
+**`distinct`** is a property of your catalog: how many tools own a term no other
+tool has. A tool that owns none cannot be matched by any query that does not also
+match a competitor. That follows from the index, not from a model of how anyone
+searches, and it is the number `--min-distinct` gates on in CI.
+
+```text
+distinct  22/24 (91.7%)   tools that own a word no other tool has
+probe     24/24 (100%)    returned by a simulated search for their own description
+
+  words  tool                     vocabulary no other tool has
+      0  browser_close            none: every word also on browser_click, browser_drag, browser_drop
+      0  browser_navigate         none: every word also on browser_tabs, browser_navigate_back, browser_network_requests
+      1  browser_evaluate         evaluate
+      1  browser_hover            hover
+      2  browser_resize           window, resize
+```
+
+*(That is `@playwright/mcp` 0.0.78, a well-maintained server, verbatim. Two of
+its twenty-four tools have no word of their own, and two more own only the word
+in their own name, which nobody describing a task would type.)*
+
+Four public servers measured this way, with the commands that reproduce each
+row, are in [the findability report](./research/findability/REPORT.md). The
+weakest was Notion's official server at 16 of 24, where `API-get-user` and
+`API-get-users` are different tools and nothing in their vocabulary separates
+them.
+
+**`probe`** is a simulated BM25 search over the four fields tool search matches
+on, and it is labelled a simulation because it is one. It reads 100% on both the
+official reference server and playwright, so it is a floor test that catches an
+empty description or a literal duplicate, not a ranking. `find` says that on
+every run rather than letting a saturated number look like a good score.
+
+There is no findability grade, on purpose:
+[docs/METHODOLOGY.md](./docs/METHODOLOGY.md) has the method, its limits, and why.
+
 ## From an agent
 
 `efaimo mcp` runs efaimo as a small, read-only MCP server, so an agent can lint or
@@ -252,8 +309,16 @@ And a mitigation you cannot measure is a mitigation you are trusting on faith:
 `weigh` reports the definitions your server actually ships, which is the input
 every mitigation operates on.
 
-If deferred loading covers your whole setup, you may not need this tool. Run
-`weigh` once and find out; that answer is worth more than our opinion.
+**And there is a fourth reason we did not have when this section was written.** Deferring a tool does not remove the question, it replaces
+it: a deferred tool that no search returns costs nothing and does nothing, so
+the thing worth measuring becomes whether each tool can be picked out of the
+catalog at all. `efaimo find` measures that, and on four public servers it
+found two to eight tools apiece that own no word their neighbours lack.
+Deferral is what makes findability matter, not what makes it moot.
+
+If deferred loading covers your whole setup, run `weigh` and `find` once and
+see what it costs you and what it hides; those answers are worth more than our
+opinion.
 
 ## Install
 
@@ -296,6 +361,7 @@ More recipes (pre-commit, GitLab, editor audit, programmatic use):
 | **S101 to S106** | skills: frontmatter and trigger quality, trigger collisions, context budget, reference integrity, injection hygiene |
 | **E101 to E118** | MCP 2026-07-28 readiness: deprecated primitives, statelessness, `server/discover`, `resultType`, cache fields, transport |
 | **E121 to E128, E130** | MCP quality: description quality, annotations, schema hygiene, tool-count and token-cost budgets |
+| **E141 to E146** | MCP findability: exclusive vocabulary, whether a search returns the tool, indistinguishable pairs, names that carry no domain word |
 
 `E000` is not a finding about your server: it means one of efaimo's own rules
 threw and was skipped, so the report is incomplete. It is never scored.
@@ -303,15 +369,18 @@ threw and was skipped, so the report is incomplete. It is never scored.
 Quality (E12x-E13x) and skill (S) findings set the letter grade; readiness
 findings (E101-E118) are reported as an ungraded migration diff, which
 `--strict-readiness` can turn into a non-zero exit
-without touching the grade.
+without touching the grade. Findability findings (E14x) belong to `efaimo find`
+and never touch a grade either.
 Every finding carries a stable id you can suppress or link. See
 [docs/RULES.md](./docs/RULES.md).
 
 ## Roadmap
 
-- Harden `efaimo test`: a separately chosen judge model, confidence intervals on
-  the delta, multi-turn tool-use trials, and judge calibration, so the
-  experimental harness earns unqualified trust.
+- Harden `efaimo test`: a length-matched placebo arm (the control currently gets
+  no system prompt at all, which confounds the skill's content with having any
+  system prompt), multi-turn tool-use trials, and judge calibration, so the
+  experimental harness earns unqualified trust. A separately chosen judge model
+  and confidence intervals on the delta are done.
 - Readiness rules for the two 2026-07-28 changes that have none yet: the
   `Mcp-Method` / `Mcp-Name` request headers (SEP-2243), and Tasks moving out of
   core into the `io.modelcontextprotocol/tasks` extension (SEP-2663).

package/SKILL.md

@@ -1,9 +1,9 @@
 ---
 name: efaimo
-description: Audit MCP servers and Agent Skills for quality and context-window cost. Use when the user wants to check an MCP server's token cost, lint a skill, verify 2026-07-28 MCP spec readiness, or add an efaimo badge or CI gate.
+description: Audit MCP servers and Agent Skills for quality and context-window cost. Use when the user wants to check an MCP server's token cost, lint a skill, verify 2026-07-28 MCP spec readiness, find out whether a search can single out a server's tools under deferred tool loading, or add an efaimo badge or CI gate.
 license: Apache-2.0
 metadata:
-  version: "0.1.2"
+  version: "0.3.0"
   homepage: "https://efaimo.ai"
 ---
 
@@ -49,6 +49,11 @@ and this file is where agents take orders.
 - **Lint a skill**
   `npx efaimo check --skill ./skills/` validates frontmatter, trigger quality,
   context budget, file references, and injection patterns.
+- **Check whether a search can single a tool out**
+  `npx efaimo find "npx -y <server>"` reports how many tools own a term no
+  other tool in the catalog has. A tool that owns none cannot be matched by any
+  query that does not also match a competitor, which matters once a host defers
+  tool loading. `--min-distinct 100` is the CI gate.
 - **Gate a pull request on context cost**
   Save a baseline with `npx efaimo weigh "<server>" --out base.json`, then in CI
   run `npx efaimo weigh "<server>" --diff base.json --allow-increase 10` to fail
@@ -56,11 +61,13 @@ and this file is where agents take orders.
 
 ## Reading the output
 
-Findings carry a stable rule id (E1xx MCP readiness, E12x-E13x MCP quality, S1xx
-skills) and a severity. The letter grade covers quality and skill findings only;
-2026-07-28 readiness items appear separately as an ungraded migration diff. Token
-numbers are o200k estimates unless `--anthropic` is used; see the methodology doc
-for the full method.
+Findings carry a stable rule id (E1xx MCP readiness, E12x-E13x MCP quality,
+E14x findability, S1xx skills) and a severity. The letter grade
+covers quality and skill findings only; 2026-07-28 readiness items appear
+separately as an ungraded migration diff, and findability is a
+separate command with no grade at all. Token numbers are o200k estimates unless
+`--anthropic` is used, which names the model it measured against; see the
+methodology doc for the full method.
 
 ## Notes
 

package/dist/check/diff.d.ts

@@ -0,0 +1,124 @@
+/**
+ * What moved between two runs of the same audit.
+ *
+ * Why this exists. Everything else here measures a subject once. Nobody
+ * publishes whether MCP servers and Agent Skills are getting heavier, or
+ * whether their quality holds, because that needs the same thing measured
+ * twice and nothing made that comparison cheap enough to bother. Two pinned
+ * runs is the smallest artifact that can answer it.
+ *
+ * The reason it is a flag on `check` rather than a shell pipeline is that a delta
+ * is exactly the shape a broken measurement takes. Change the ruler between
+ * the two runs and every subject appears to move; pair the wrong items and
+ * everything looks new. Both mistakes produce a plausible number and neither
+ * announces itself, so the controls a careful person would run by hand are
+ * enforced here instead of documented:
+ *
+ * 1. RULES DRIFT IS A HARD STOP. If the two reports carry different
+ *    `rulesVersion` values, a grade that moved cannot be attributed to the
+ *    subject, because the ruler moved too. `--allow-rules-drift` proceeds and
+ *    every grade line is then marked unattributable, which is the honest
+ *    output rather than a silent one.
+ * 2. AN EMPTY PAIRING IS A FAILURE. If nothing pairs, that is not "no
+ *    change", it is two reports about different things, and reporting "0
+ *    changed" would be the same vacuous pass an empty grep gives.
+ * 3. THE THREE TOKEN COSTS ARE NEVER SUMMED. Metadata sits in context
+ *    permanently, the body loads on trigger, referenced files load on demand.
+ *    Adding them produces a single headline percentage that is arithmetically
+ *    true and tells the reader the wrong thing, because almost all of the
+ *    movement usually lives in the column that costs the least to carry.
+ * 4. A DOMINANT MOVER IS NAMED. When one item is most of a total's movement,
+ *    the total describes that item and not the population, so it is called
+ *    out beside the total rather than left for the reader to discover.
+ */
+export interface CheckDiffItem {
+    name: string;
+    score: number;
+    letter: string;
+    errors: number;
+    warns: number;
+    infos: number;
+    ruleIds: string[];
+    metadata?: number;
+    body?: number;
+    referenced?: number;
+}
+export interface CheckEnvelope {
+    tool?: string;
+    version?: string;
+    rulesVersion?: string;
+    kind?: string;
+    data?: unknown;
+}
+export interface GradeMove {
+    name: string;
+    before: {
+        score: number;
+        letter: string;
+    };
+    after: {
+        score: number;
+        letter: string;
+    };
+}
+export interface CostMove {
+    name: string;
+    before: number;
+    after: number;
+    delta: number;
+}
+export interface CostColumn {
+    column: "metadata" | "body" | "referenced";
+    when: string;
+    before: number;
+    after: number;
+    delta: number;
+    pct: number;
+    moved: CostMove[];
+    /** Set when a single item is more than half of the absolute movement. */
+    dominatedBy?: {
+        name: string;
+        delta: number;
+        shareOfMovement: number;
+    };
+}
+export interface CheckDiffResult {
+    kind: string;
+    versions: {
+        before: string;
+        after: string;
+    };
+    rulesVersions: {
+        before?: string;
+        after?: string;
+    };
+    rulesDrift: boolean;
+    paired: number;
+    added: string[];
+    removed: string[];
+    improved: GradeMove[];
+    worsened: GradeMove[];
+    held: number;
+    findingCounts: {
+        before: number;
+        after: number;
+    };
+    costs: CostColumn[];
+    notes: string[];
+}
+export declare class CheckDiffRefused extends Error {
+}
+/**
+ * Both shapes `check` emits, flattened to one list.
+ *
+ * A single target puts the report at `data` itself; a set puts one report per
+ * entry in `data.perSkill` and the token counts in a parallel `data.weigh`
+ * list. Pairing the two lists by name rather than by position matters: they
+ * are built by different passes and nothing guarantees the orders agree, and
+ * a positional join would silently attribute one skill's tokens to another.
+ */
+export declare function flattenCheck(envelope: CheckEnvelope): CheckDiffItem[];
+export interface CheckDiffOptions {
+    allowRulesDrift?: boolean;
+}
+export declare function diffCheck(before: CheckEnvelope, after: CheckEnvelope, opts?: CheckDiffOptions): CheckDiffResult;

package/dist/check/diff.js

@@ -0,0 +1,165 @@
+export class CheckDiffRefused extends Error {
+}
+const asRecord = (v) => v && typeof v === "object" && !Array.isArray(v) ? v : undefined;
+const num = (v) => (typeof v === "number" && Number.isFinite(v) ? v : 0);
+/**
+ * Both shapes `check` emits, flattened to one list.
+ *
+ * A single target puts the report at `data` itself; a set puts one report per
+ * entry in `data.perSkill` and the token counts in a parallel `data.weigh`
+ * list. Pairing the two lists by name rather than by position matters: they
+ * are built by different passes and nothing guarantees the orders agree, and
+ * a positional join would silently attribute one skill's tokens to another.
+ */
+export function flattenCheck(envelope) {
+    const data = asRecord(envelope.data);
+    if (!data)
+        return [];
+    const weighByName = new Map();
+    const weigh = asRecord(data.weigh);
+    if (Array.isArray(weigh?.perSkill)) {
+        for (const w of weigh.perSkill) {
+            const r = asRecord(w);
+            if (r && typeof r.name === "string")
+                weighByName.set(r.name, r);
+        }
+    }
+    const fromReport = (name, report) => {
+        const grade = asRecord(report.grade) ?? {};
+        const counts = asRecord(report.counts) ?? {};
+        const findings = Array.isArray(report.findings) ? report.findings : [];
+        const w = weighByName.get(name);
+        return {
+            name,
+            score: num(grade.score),
+            letter: typeof grade.letter === "string" ? grade.letter : "?",
+            errors: num(counts.error),
+            warns: num(counts.warn),
+            infos: num(counts.info),
+            ruleIds: findings.map((f) => f.ruleId).filter((r) => typeof r === "string"),
+            ...(w
+                ? {
+                    metadata: num(w.metadataTokens),
+                    body: num(w.bodyTokens),
+                    referenced: num(w.refFileTokens),
+                }
+                : {}),
+        };
+    };
+    if (Array.isArray(data.perSkill)) {
+        const out = [];
+        for (const entry of data.perSkill) {
+            const e = asRecord(entry);
+            const report = asRecord(e?.report);
+            const name = typeof e?.name === "string" ? e.name : undefined;
+            if (name && report)
+                out.push(fromReport(name, report));
+        }
+        return out;
+    }
+    const target = typeof data.target === "string" ? data.target : undefined;
+    if (target && asRecord(data.grade))
+        return [fromReport(target, data)];
+    return [];
+}
+export function diffCheck(before, after, opts = {}) {
+    for (const [label, env] of [
+        ["before", before],
+        ["after", after],
+    ]) {
+        if (env.tool !== "efaimo") {
+            throw new CheckDiffRefused(`the ${label} file is not an efaimo report (its "tool" field is ${JSON.stringify(env.tool)}); diff compares two runs of this tool, so pass JSON produced by --json`);
+        }
+    }
+    if (before.kind !== after.kind) {
+        throw new CheckDiffRefused(`these are different kinds of report (${before.kind} and ${after.kind}); comparing them would pair things that are not the same measurement`);
+    }
+    if (before.kind !== "check") {
+        throw new CheckDiffRefused(`diff currently compares "check" reports only, and these are "${before.kind}"; run \`efaimo check ... --json\` at both points`);
+    }
+    const rulesDrift = Boolean(before.rulesVersion && after.rulesVersion && before.rulesVersion !== after.rulesVersion);
+    if (rulesDrift && !opts.allowRulesDrift) {
+        throw new CheckDiffRefused(`the two runs used different rulesets (${before.rulesVersion} and ${after.rulesVersion}). A grade that moved cannot be attributed to the subject, because the ruler moved too. Re-run both points with one version of efaimo, or pass --allow-rules-drift to compare anyway and have every grade line marked unattributable.`);
+    }
+    const b = new Map(flattenCheck(before).map((i) => [i.name, i]));
+    const a = new Map(flattenCheck(after).map((i) => [i.name, i]));
+    const paired = [...b.keys()].filter((k) => a.has(k)).sort();
+    if (!paired.length) {
+        // An empty intersection and "nothing changed" are the same output, and
+        // only one of them is worth printing.
+        throw new CheckDiffRefused(`nothing pairs between the two reports (${b.size} items before, ${a.size} after, 0 in common). That is not "no change", it is two reports about different subjects.`);
+    }
+    const improved = [];
+    const worsened = [];
+    let held = 0;
+    for (const name of paired) {
+        const x = b.get(name);
+        const y = a.get(name);
+        const move = { name, before: { score: x.score, letter: x.letter }, after: { score: y.score, letter: y.letter } };
+        if (y.score > x.score)
+            improved.push(move);
+        else if (y.score < x.score)
+            worsened.push(move);
+        else
+            held++;
+    }
+    const COLUMNS = [
+        ["metadata", "always in context", (i) => i.metadata],
+        ["body", "loads on trigger", (i) => i.body],
+        ["referenced", "loads on demand", (i) => i.referenced],
+    ];
+    const costs = [];
+    for (const [column, when, pick] of COLUMNS) {
+        if (paired.some((n) => pick(b.get(n)) === undefined || pick(a.get(n)) === undefined))
+            continue;
+        const beforeTotal = paired.reduce((s, n) => s + (pick(b.get(n)) ?? 0), 0);
+        const afterTotal = paired.reduce((s, n) => s + (pick(a.get(n)) ?? 0), 0);
+        const moved = paired
+            .map((n) => ({ name: n, before: pick(b.get(n)) ?? 0, after: pick(a.get(n)) ?? 0 }))
+            .filter((m) => m.before !== m.after)
+            .map((m) => ({ ...m, delta: m.after - m.before }))
+            .sort((x, y) => y.delta - x.delta);
+        const totalAbs = moved.reduce((s, m) => s + Math.abs(m.delta), 0);
+        const biggest = [...moved].sort((x, y) => Math.abs(y.delta) - Math.abs(x.delta))[0];
+        const share = biggest && totalAbs ? Math.abs(biggest.delta) / totalAbs : 0;
+        costs.push({
+            column,
+            when,
+            before: beforeTotal,
+            after: afterTotal,
+            delta: afterTotal - beforeTotal,
+            pct: beforeTotal ? ((afterTotal - beforeTotal) / beforeTotal) * 100 : 0,
+            moved,
+            ...(share > 0.5 && biggest
+                ? { dominatedBy: { name: biggest.name, delta: biggest.delta, shareOfMovement: share * 100 } }
+                : {}),
+        });
+    }
+    const notes = [
+        "the three token columns are reported separately on purpose: metadata is carried permanently, the body loads on trigger, and referenced files load on demand, so a single summed percentage would describe none of them",
+    ];
+    if (rulesDrift) {
+        notes.push(`RULES DRIFT ACCEPTED (${before.rulesVersion} -> ${after.rulesVersion}): grade movement below cannot be attributed to the subjects, because the ruleset changed between the two runs`);
+    }
+    if (before.version !== after.version) {
+        notes.push(`the two runs used efaimo ${before.version} and ${after.version}; the rulesets matched, so grades are comparable, but token counts can move if the tokenizer or the walk changed`);
+    }
+    return {
+        kind: before.kind,
+        versions: { before: before.version ?? "?", after: after.version ?? "?" },
+        rulesVersions: { before: before.rulesVersion, after: after.rulesVersion },
+        rulesDrift,
+        paired: paired.length,
+        added: [...a.keys()].filter((k) => !b.has(k)).sort(),
+        removed: [...b.keys()].filter((k) => !a.has(k)).sort(),
+        improved,
+        worsened,
+        held,
+        findingCounts: {
+            before: [...b.values()].reduce((s, i) => s + i.errors + i.warns + i.infos, 0),
+            after: [...a.values()].reduce((s, i) => s + i.errors + i.warns + i.infos, 0),
+        },
+        costs,
+        notes,
+    };
+}