efaimo 0.2.0 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +152 -1
- package/README.md +11 -0
- package/SKILL.md +1 -1
- package/dist/check/check.js +15 -0
- package/dist/check/diff.d.ts +124 -0
- package/dist/check/diff.js +165 -0
- package/dist/cli.js +91 -16
- package/dist/core/types.d.ts +41 -0
- package/dist/find/find.d.ts +2 -0
- package/dist/find/find.js +2 -0
- package/dist/index.d.ts +1 -0
- package/dist/index.js +1 -0
- package/dist/reporters/pretty.d.ts +13 -0
- package/dist/reporters/pretty.js +55 -1
- package/dist/rules/skill/index.js +32 -1
- package/dist/skills/parse.d.ts +6 -0
- package/dist/skills/parse.js +47 -3
- package/dist/util/misc.d.ts +20 -2
- package/dist/util/misc.js +8 -2
- package/dist/version.d.ts +1 -1
- package/dist/version.js +1 -1
- package/docs/RULES.md +2 -1
- package/package.json +3 -1
package/CHANGELOG.md
CHANGED
|
@@ -4,6 +4,156 @@ All notable changes to efaimo are documented here. Format follows
|
|
|
4
4
|
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/); this project uses
|
|
5
5
|
[Semantic Versioning](https://semver.org/).
|
|
6
6
|
|
|
7
|
+
## [0.4.0] - 2026-09-03
|
|
8
|
+
|
|
9
|
+
### Fixed
|
|
10
|
+
|
|
11
|
+
- **`check --skill <dir>` finds skills it used to walk past**.
|
|
12
|
+
Two causes, both silent. The walk never entered a directory whose name starts
|
|
13
|
+
with a dot, so `.claude/skills/<name>/`, which is where a project keeps its
|
|
14
|
+
own skills, was invisible; and its depth bound of 3 missed layouts one level
|
|
15
|
+
deeper such as `skills/custom_skills/<name>/`. On the public corpus this
|
|
16
|
+
found 34 skills where `scripts/skills-index.mjs` found 38, and both numbers
|
|
17
|
+
were reported with confidence.
|
|
18
|
+
|
|
19
|
+
Dot directories are now entered for skill discovery and nowhere else, with
|
|
20
|
+
`.git` still excluded. The depth bound is 6 and is measured rather than
|
|
21
|
+
picked: the deepest real skill in that corpus sits 4 directories below a
|
|
22
|
+
directory of repositories and 3 below a repository root, so six leaves two
|
|
23
|
+
levels of headroom while still keeping an accidental `check --skill /` from
|
|
24
|
+
walking a disk.
|
|
25
|
+
|
|
26
|
+
**And when the bound does bite, it says so.** `SkillSet.truncatedAt` lists
|
|
27
|
+
the directories the walk refused to enter, because a bound that truncates in
|
|
28
|
+
silence is the same failure as a check that examines nothing. On the public
|
|
29
|
+
corpus it fires 16 times, all of them XML schema trees inside the `xlsx` and
|
|
30
|
+
`pptx` skills, which is the bound doing its job audibly.
|
|
31
|
+
|
|
32
|
+
- **One walker instead of two**. `scripts/skills-index.mjs`
|
|
33
|
+
carried its own copy with different rules, which is why the two disagreed.
|
|
34
|
+
It now calls the CLI's discovery, so they cannot drift apart again, and it
|
|
35
|
+
passes the truncation warning through. The published July index regenerates
|
|
36
|
+
byte-identically under the shared walk, which is how this was checked.
|
|
37
|
+
|
|
38
|
+
### Added
|
|
39
|
+
|
|
40
|
+
- **S107: a filename one capitalisation away from a skill**.
|
|
41
|
+
Set-level and warn-level, so it moves no grade, in the same way S103 treats
|
|
42
|
+
a collision that belongs to a pair rather than to either member.
|
|
43
|
+
|
|
44
|
+
The spec names `SKILL.md` exactly and discovery matches it exactly, so
|
|
45
|
+
`skill.md` is not a skill here on any platform. It is not nothing anywhere:
|
|
46
|
+
a case-insensitive filesystem, the default on macOS and Windows, may hand it
|
|
47
|
+
to a host that opens it by name, so the same repository can carry a working
|
|
48
|
+
skill on the author's laptop and no skill at all in Linux CI. Nothing in
|
|
49
|
+
this tool would previously have mentioned such a file, which is the worst
|
|
50
|
+
property a near miss can have. Five exist in the public corpus.
|
|
51
|
+
|
|
52
|
+
## [0.3.0] - 2026-09-03
|
|
53
|
+
|
|
54
|
+
### Added
|
|
55
|
+
|
|
56
|
+
- **`check --out` and `check --diff`**: what moved between two
|
|
57
|
+
audits of the same subject. Same grammar as `weigh --out` / `weigh --diff`,
|
|
58
|
+
which is why it is a flag rather than a sixth subcommand. Nothing published
|
|
59
|
+
says whether skills and MCP servers are getting heavier or whether their
|
|
60
|
+
quality holds, because that needs the same thing measured twice and nothing
|
|
61
|
+
made the comparison cheap.
|
|
62
|
+
|
|
63
|
+
A delta is exactly the shape a broken measurement takes, so the controls a
|
|
64
|
+
careful person would run by hand are enforced instead of documented:
|
|
65
|
+
|
|
66
|
+
- **Rules drift is a hard stop.** Different `rulesVersion` between the two
|
|
67
|
+
runs means a moved grade cannot be attributed to the subject, because the
|
|
68
|
+
ruler moved too. `--allow-rules-drift` proceeds and marks
|
|
69
|
+
every grade line unattributable.
|
|
70
|
+
- **An empty pairing fails.** Nothing in common is two reports about
|
|
71
|
+
different subjects, not "no change".
|
|
72
|
+
- **The three token costs are never summed.** Metadata is carried
|
|
73
|
+
permanently, the body loads on trigger, referenced files load on demand;
|
|
74
|
+
one combined percentage would describe none of them.
|
|
75
|
+
- **A dominant mover is named beside its total**, because when one subject is
|
|
76
|
+
most of a change, the total is about that subject and not the population.
|
|
77
|
+
|
|
78
|
+
`--fail-on-regression` exits 1 when any subject present in
|
|
79
|
+
both runs scored lower.
|
|
80
|
+
|
|
81
|
+
Measured on the public skills corpus, 2026-07-17 pinned against 2026-09-03
|
|
82
|
+
HEAD, same ruleset both sides, over the **32 skills `check --skill` pairs**:
|
|
83
|
+
metadata -0.8%, body +7.5%, referenced files +61.4% with one skill at 96.5%
|
|
84
|
+
of that movement; grades 0 improved and 4 worsened.
|
|
85
|
+
|
|
86
|
+
The population matters and is worth stating, because
|
|
87
|
+
`research/skills-delta-2026-09-03/REPORT.md` reports body **+7.0%** over
|
|
88
|
+
**36** paired skills for the same corpus on the same dates. Both are true.
|
|
89
|
+
`scripts/skills-index.mjs` finds four skills that `check --skill` does not,
|
|
90
|
+
for the walker reason in Known below, so the two are measuring slightly
|
|
91
|
+
different populations. Every other figure is identical between them; the
|
|
92
|
+
body percentage is the one the four missing skills move.
|
|
93
|
+
|
|
94
|
+
- **`efaimo find` takes several servers**. Pass more than one
|
|
95
|
+
and their catalogs are merged into one and measured together, each tool
|
|
96
|
+
labelled with where it came from. A flag-shaped change rather than a new
|
|
97
|
+
subcommand, because the engine already took a flat tool array and only the
|
|
98
|
+
CLI surface was single-target.
|
|
99
|
+
|
|
100
|
+
The reason it is worth having: a server's author keeps their own tool names
|
|
101
|
+
apart as a matter of course, nobody coordinates across the several servers a
|
|
102
|
+
person actually installs, and the model sees one flat list with no
|
|
103
|
+
indication of origin. A single-server run is structurally unable to see the
|
|
104
|
+
collision that actually bites.
|
|
105
|
+
|
|
106
|
+
The origin is carried for the report and deliberately kept out of the index.
|
|
107
|
+
Prefixing names with their server would hand every tool a term no other tool
|
|
108
|
+
has, and a catalog of indistinguishable tools would score a perfect 100.
|
|
109
|
+
There is a test for exactly that, and it had to be rewritten once: its first
|
|
110
|
+
origin labels were "server-one" and "server-two", whose only surviving token
|
|
111
|
+
is "server" for both, so the sabotage it existed to catch passed.
|
|
112
|
+
|
|
113
|
+
Measured live on `server-everything@2026.7.4` plus `@playwright/mcp@0.0.78`:
|
|
114
|
+
37 tools merged, 35 distinct (94.6%). No cross-server collision between
|
|
115
|
+
those two, which is the right answer for a filesystem server and a browser
|
|
116
|
+
server, and the two tools owning nothing are playwright's own already
|
|
117
|
+
documented `browser_close` and `browser_navigate`.
|
|
118
|
+
|
|
119
|
+
### Fixed
|
|
120
|
+
|
|
121
|
+
- **S102 no longer passes filler**. The rule read
|
|
122
|
+
`description.length < 20` and then asked only whether one of four trigger
|
|
123
|
+
words appeared anywhere, so `Useful for various tasks.` scored a clean
|
|
124
|
+
A (100) in silence because "for" sits inside "for various tasks". It was
|
|
125
|
+
testing for the presence of a trigger WORD rather than for the presence of a
|
|
126
|
+
trigger. A skill whose whole body was one sentence also scored 100.
|
|
127
|
+
|
|
128
|
+
Two floors replace it, both grounded on the 38 public skills in
|
|
129
|
+
`research/skills-index/manifest.json` measured 2026-09-03, where the shortest
|
|
130
|
+
real description is 68 characters and the thinnest carries 4 distinct content
|
|
131
|
+
terms: under 40 characters, or fewer than 4 distinct terms surviving the
|
|
132
|
+
stopword list, is a warning. The second is the same measurement `find` makes
|
|
133
|
+
on tool catalogs, since a description with no distinguishing content cannot
|
|
134
|
+
be matched to a task whatever words it contains.
|
|
135
|
+
|
|
136
|
+
Impact on the corpus: none. The 38 skills produce zero S102 findings before
|
|
137
|
+
and after, and the grade distribution is unchanged. This was found by
|
|
138
|
+
sabotaging the grader while reviewing two new skills, not by a report.
|
|
139
|
+
|
|
140
|
+
Note that this changes grades in principle, so it is deliberately sequenced
|
|
141
|
+
after the 2026-09-03 delta measurement, whose reproduction is pinned to
|
|
142
|
+
`v0.2.0`.
|
|
143
|
+
|
|
144
|
+
### Known
|
|
145
|
+
|
|
146
|
+
- **`check --skill <dir>` does not find every skill under a tree**
|
|
147
|
+
_(present in 0.2.0 as well)_. It walks
|
|
148
|
+
`<root>/<name>/SKILL.md` and `<root>/skills/<name>/SKILL.md` but misses skills
|
|
149
|
+
inside a dot directory (`.claude/skills/<name>/`) and skills one level deeper
|
|
150
|
+
(`skills/custom_skills/<name>/`). On the corpus above it found 34 where
|
|
151
|
+
`scripts/skills-index.mjs` found 38, all four in the same repository. Two of
|
|
152
|
+
this tool's own walkers disagreeing about the same tree is a defect whichever
|
|
153
|
+
is right; a `--diff` built on the narrower walk silently compares an
|
|
154
|
+
incomplete population. Not fixed here because the fix is a design decision
|
|
155
|
+
about depth and dot directories, separate from this change.
|
|
156
|
+
|
|
7
157
|
## [0.2.0] - 2026-09-02
|
|
8
158
|
|
|
9
159
|
### Added
|
|
@@ -258,7 +408,8 @@ First release.
|
|
|
258
408
|
Documented rule set (`docs/RULES.md`), token methodology (`docs/METHODOLOGY.md`),
|
|
259
409
|
and integration guide (`docs/INTEGRATIONS.md`).
|
|
260
410
|
|
|
261
|
-
[
|
|
411
|
+
[0.4.0]: https://github.com/efaimo-ai/efaimo/releases/tag/v0.4.0
|
|
412
|
+
[0.3.0]: https://github.com/efaimo-ai/efaimo/releases/tag/v0.3.0
|
|
262
413
|
[0.2.0]: https://github.com/efaimo-ai/efaimo/releases/tag/v0.2.0
|
|
263
414
|
[0.1.2]: https://github.com/efaimo-ai/efaimo/releases/tag/v0.1.2
|
|
264
415
|
[0.1.1]: https://github.com/efaimo-ai/efaimo/releases/tag/v0.1.1
|
package/README.md
CHANGED
|
@@ -58,6 +58,13 @@ instructions, and the median skill's instructions run ~1,632 tokens loaded on
|
|
|
58
58
|
every trigger. Full report, the corpus manifest, and the two commands that
|
|
59
59
|
reproduce it: [the Skills Quality Index](./research/skills-index/REPORT.md).
|
|
60
60
|
|
|
61
|
+
That index is a photograph. [Measured twice](./research/skills-delta-2026-09-03/REPORT.md)
|
|
62
|
+
re-runs the same corpus seven weeks later with the same ruleset: the cost that
|
|
63
|
+
sits in context permanently moved -0.8%, the body +7.0%, and referenced files,
|
|
64
|
+
which load out of sight and which no rule here counts, +61.4%. Grades moved one
|
|
65
|
+
way only, 0 improved against 4 worse. The uncomfortable part points inward: the
|
|
66
|
+
two columns this tool has thresholds for are the two that held.
|
|
67
|
+
|
|
61
68
|
### Does the skill actually help? (experimental)
|
|
62
69
|
|
|
63
70
|
Linting tells you a skill is well-formed. It does not tell you the skill makes the
|
|
@@ -399,3 +406,7 @@ Built by [efaimo ai](https://efaimo.ai), open tooling for the space between host
|
|
|
399
406
|
and tools. `efaimo` is the flagship CLI; capabilities grow as subcommands under one
|
|
400
407
|
name. Apache-2.0. See [CONTRIBUTING.md](./CONTRIBUTING.md) and
|
|
401
408
|
[docs/RULES.md](./docs/RULES.md) to add a rule.
|
|
409
|
+
|
|
410
|
+
EFAIMO AI is a registered trademark of the publisher (KR 40-2585820, class 42).
|
|
411
|
+
The lowercase `efaimo ai` used throughout is the same mark; the package name and
|
|
412
|
+
the commands stay lowercase.
|
package/SKILL.md
CHANGED
|
@@ -3,7 +3,7 @@ name: efaimo
|
|
|
3
3
|
description: Audit MCP servers and Agent Skills for quality and context-window cost. Use when the user wants to check an MCP server's token cost, lint a skill, verify 2026-07-28 MCP spec readiness, find out whether a search can single out a server's tools under deferred tool loading, or add an efaimo badge or CI gate.
|
|
4
4
|
license: Apache-2.0
|
|
5
5
|
metadata:
|
|
6
|
-
version: "0.
|
|
6
|
+
version: "0.4.0"
|
|
7
7
|
homepage: "https://efaimo.ai"
|
|
8
8
|
---
|
|
9
9
|
|
package/dist/check/check.js
CHANGED
|
@@ -85,5 +85,20 @@ export async function checkSkillSet(pathInput, label) {
|
|
|
85
85
|
return { name, dir: skill.dir, report: buildReport("skill", name, findings, []) };
|
|
86
86
|
});
|
|
87
87
|
const setFindings = set.skills.flatMap((skill) => runRules(setRules, { skill, set, weigh }));
|
|
88
|
+
// S107 is emitted here rather than by a SkillRule because its subject is a
|
|
89
|
+
// file that is NOT a skill: no rule that takes a parsed skill could ever see
|
|
90
|
+
// it. Set level and warn level on purpose, so it moves no grade, matching
|
|
91
|
+
// how S103 treats a collision that belongs to a pair rather than to either
|
|
92
|
+
// member.
|
|
93
|
+
for (const file of set.miscasedSkillFiles ?? []) {
|
|
94
|
+
setFindings.push({
|
|
95
|
+
ruleId: "S107",
|
|
96
|
+
severity: "warn",
|
|
97
|
+
title: "filename is one capitalisation away from a skill",
|
|
98
|
+
message: `"${path.relative(set.root, file)}" is not loaded here, because the spec names SKILL.md exactly. A case-insensitive filesystem (the macOS and Windows default) may still hand it to a host, so this can work on a laptop and be missing entirely in Linux CI.`,
|
|
99
|
+
target: path.relative(set.root, file),
|
|
100
|
+
fixHint: "rename it to SKILL.md, or delete it if it is not a skill",
|
|
101
|
+
});
|
|
102
|
+
}
|
|
88
103
|
return { label, root: set.root, perSkill, setFindings, weigh };
|
|
89
104
|
}
|
|
@@ -0,0 +1,124 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* What moved between two runs of the same audit.
|
|
3
|
+
*
|
|
4
|
+
* Why this exists. Everything else here measures a subject once. Nobody
|
|
5
|
+
* publishes whether MCP servers and Agent Skills are getting heavier, or
|
|
6
|
+
* whether their quality holds, because that needs the same thing measured
|
|
7
|
+
* twice and nothing made that comparison cheap enough to bother. Two pinned
|
|
8
|
+
* runs is the smallest artifact that can answer it.
|
|
9
|
+
*
|
|
10
|
+
* The reason it is a flag on `check` rather than a shell pipeline is that a delta
|
|
11
|
+
* is exactly the shape a broken measurement takes. Change the ruler between
|
|
12
|
+
* the two runs and every subject appears to move; pair the wrong items and
|
|
13
|
+
* everything looks new. Both mistakes produce a plausible number and neither
|
|
14
|
+
* announces itself, so the controls a careful person would run by hand are
|
|
15
|
+
* enforced here instead of documented:
|
|
16
|
+
*
|
|
17
|
+
* 1. RULES DRIFT IS A HARD STOP. If the two reports carry different
|
|
18
|
+
* `rulesVersion` values, a grade that moved cannot be attributed to the
|
|
19
|
+
* subject, because the ruler moved too. `--allow-rules-drift` proceeds and
|
|
20
|
+
* every grade line is then marked unattributable, which is the honest
|
|
21
|
+
* output rather than a silent one.
|
|
22
|
+
* 2. AN EMPTY PAIRING IS A FAILURE. If nothing pairs, that is not "no
|
|
23
|
+
* change", it is two reports about different things, and reporting "0
|
|
24
|
+
* changed" would be the same vacuous pass an empty grep gives.
|
|
25
|
+
* 3. THE THREE TOKEN COSTS ARE NEVER SUMMED. Metadata sits in context
|
|
26
|
+
* permanently, the body loads on trigger, referenced files load on demand.
|
|
27
|
+
* Adding them produces a single headline percentage that is arithmetically
|
|
28
|
+
* true and tells the reader the wrong thing, because almost all of the
|
|
29
|
+
* movement usually lives in the column that costs the least to carry.
|
|
30
|
+
* 4. A DOMINANT MOVER IS NAMED. When one item is most of a total's movement,
|
|
31
|
+
* the total describes that item and not the population, so it is called
|
|
32
|
+
* out beside the total rather than left for the reader to discover.
|
|
33
|
+
*/
|
|
34
|
+
export interface CheckDiffItem {
|
|
35
|
+
name: string;
|
|
36
|
+
score: number;
|
|
37
|
+
letter: string;
|
|
38
|
+
errors: number;
|
|
39
|
+
warns: number;
|
|
40
|
+
infos: number;
|
|
41
|
+
ruleIds: string[];
|
|
42
|
+
metadata?: number;
|
|
43
|
+
body?: number;
|
|
44
|
+
referenced?: number;
|
|
45
|
+
}
|
|
46
|
+
export interface CheckEnvelope {
|
|
47
|
+
tool?: string;
|
|
48
|
+
version?: string;
|
|
49
|
+
rulesVersion?: string;
|
|
50
|
+
kind?: string;
|
|
51
|
+
data?: unknown;
|
|
52
|
+
}
|
|
53
|
+
export interface GradeMove {
|
|
54
|
+
name: string;
|
|
55
|
+
before: {
|
|
56
|
+
score: number;
|
|
57
|
+
letter: string;
|
|
58
|
+
};
|
|
59
|
+
after: {
|
|
60
|
+
score: number;
|
|
61
|
+
letter: string;
|
|
62
|
+
};
|
|
63
|
+
}
|
|
64
|
+
export interface CostMove {
|
|
65
|
+
name: string;
|
|
66
|
+
before: number;
|
|
67
|
+
after: number;
|
|
68
|
+
delta: number;
|
|
69
|
+
}
|
|
70
|
+
export interface CostColumn {
|
|
71
|
+
column: "metadata" | "body" | "referenced";
|
|
72
|
+
when: string;
|
|
73
|
+
before: number;
|
|
74
|
+
after: number;
|
|
75
|
+
delta: number;
|
|
76
|
+
pct: number;
|
|
77
|
+
moved: CostMove[];
|
|
78
|
+
/** Set when a single item is more than half of the absolute movement. */
|
|
79
|
+
dominatedBy?: {
|
|
80
|
+
name: string;
|
|
81
|
+
delta: number;
|
|
82
|
+
shareOfMovement: number;
|
|
83
|
+
};
|
|
84
|
+
}
|
|
85
|
+
export interface CheckDiffResult {
|
|
86
|
+
kind: string;
|
|
87
|
+
versions: {
|
|
88
|
+
before: string;
|
|
89
|
+
after: string;
|
|
90
|
+
};
|
|
91
|
+
rulesVersions: {
|
|
92
|
+
before?: string;
|
|
93
|
+
after?: string;
|
|
94
|
+
};
|
|
95
|
+
rulesDrift: boolean;
|
|
96
|
+
paired: number;
|
|
97
|
+
added: string[];
|
|
98
|
+
removed: string[];
|
|
99
|
+
improved: GradeMove[];
|
|
100
|
+
worsened: GradeMove[];
|
|
101
|
+
held: number;
|
|
102
|
+
findingCounts: {
|
|
103
|
+
before: number;
|
|
104
|
+
after: number;
|
|
105
|
+
};
|
|
106
|
+
costs: CostColumn[];
|
|
107
|
+
notes: string[];
|
|
108
|
+
}
|
|
109
|
+
export declare class CheckDiffRefused extends Error {
|
|
110
|
+
}
|
|
111
|
+
/**
|
|
112
|
+
* Both shapes `check` emits, flattened to one list.
|
|
113
|
+
*
|
|
114
|
+
* A single target puts the report at `data` itself; a set puts one report per
|
|
115
|
+
* entry in `data.perSkill` and the token counts in a parallel `data.weigh`
|
|
116
|
+
* list. Pairing the two lists by name rather than by position matters: they
|
|
117
|
+
* are built by different passes and nothing guarantees the orders agree, and
|
|
118
|
+
* a positional join would silently attribute one skill's tokens to another.
|
|
119
|
+
*/
|
|
120
|
+
export declare function flattenCheck(envelope: CheckEnvelope): CheckDiffItem[];
|
|
121
|
+
export interface CheckDiffOptions {
|
|
122
|
+
allowRulesDrift?: boolean;
|
|
123
|
+
}
|
|
124
|
+
export declare function diffCheck(before: CheckEnvelope, after: CheckEnvelope, opts?: CheckDiffOptions): CheckDiffResult;
|
|
@@ -0,0 +1,165 @@
|
|
|
1
|
+
export class CheckDiffRefused extends Error {
|
|
2
|
+
}
|
|
3
|
+
const asRecord = (v) => v && typeof v === "object" && !Array.isArray(v) ? v : undefined;
|
|
4
|
+
const num = (v) => (typeof v === "number" && Number.isFinite(v) ? v : 0);
|
|
5
|
+
/**
|
|
6
|
+
* Both shapes `check` emits, flattened to one list.
|
|
7
|
+
*
|
|
8
|
+
* A single target puts the report at `data` itself; a set puts one report per
|
|
9
|
+
* entry in `data.perSkill` and the token counts in a parallel `data.weigh`
|
|
10
|
+
* list. Pairing the two lists by name rather than by position matters: they
|
|
11
|
+
* are built by different passes and nothing guarantees the orders agree, and
|
|
12
|
+
* a positional join would silently attribute one skill's tokens to another.
|
|
13
|
+
*/
|
|
14
|
+
export function flattenCheck(envelope) {
|
|
15
|
+
const data = asRecord(envelope.data);
|
|
16
|
+
if (!data)
|
|
17
|
+
return [];
|
|
18
|
+
const weighByName = new Map();
|
|
19
|
+
const weigh = asRecord(data.weigh);
|
|
20
|
+
if (Array.isArray(weigh?.perSkill)) {
|
|
21
|
+
for (const w of weigh.perSkill) {
|
|
22
|
+
const r = asRecord(w);
|
|
23
|
+
if (r && typeof r.name === "string")
|
|
24
|
+
weighByName.set(r.name, r);
|
|
25
|
+
}
|
|
26
|
+
}
|
|
27
|
+
const fromReport = (name, report) => {
|
|
28
|
+
const grade = asRecord(report.grade) ?? {};
|
|
29
|
+
const counts = asRecord(report.counts) ?? {};
|
|
30
|
+
const findings = Array.isArray(report.findings) ? report.findings : [];
|
|
31
|
+
const w = weighByName.get(name);
|
|
32
|
+
return {
|
|
33
|
+
name,
|
|
34
|
+
score: num(grade.score),
|
|
35
|
+
letter: typeof grade.letter === "string" ? grade.letter : "?",
|
|
36
|
+
errors: num(counts.error),
|
|
37
|
+
warns: num(counts.warn),
|
|
38
|
+
infos: num(counts.info),
|
|
39
|
+
ruleIds: findings.map((f) => f.ruleId).filter((r) => typeof r === "string"),
|
|
40
|
+
...(w
|
|
41
|
+
? {
|
|
42
|
+
metadata: num(w.metadataTokens),
|
|
43
|
+
body: num(w.bodyTokens),
|
|
44
|
+
referenced: num(w.refFileTokens),
|
|
45
|
+
}
|
|
46
|
+
: {}),
|
|
47
|
+
};
|
|
48
|
+
};
|
|
49
|
+
if (Array.isArray(data.perSkill)) {
|
|
50
|
+
const out = [];
|
|
51
|
+
for (const entry of data.perSkill) {
|
|
52
|
+
const e = asRecord(entry);
|
|
53
|
+
const report = asRecord(e?.report);
|
|
54
|
+
const name = typeof e?.name === "string" ? e.name : undefined;
|
|
55
|
+
if (name && report)
|
|
56
|
+
out.push(fromReport(name, report));
|
|
57
|
+
}
|
|
58
|
+
return out;
|
|
59
|
+
}
|
|
60
|
+
const target = typeof data.target === "string" ? data.target : undefined;
|
|
61
|
+
if (target && asRecord(data.grade))
|
|
62
|
+
return [fromReport(target, data)];
|
|
63
|
+
return [];
|
|
64
|
+
}
|
|
65
|
+
export function diffCheck(before, after, opts = {}) {
|
|
66
|
+
for (const [label, env] of [
|
|
67
|
+
["before", before],
|
|
68
|
+
["after", after],
|
|
69
|
+
]) {
|
|
70
|
+
if (env.tool !== "efaimo") {
|
|
71
|
+
throw new CheckDiffRefused(`the ${label} file is not an efaimo report (its "tool" field is ${JSON.stringify(env.tool)}); diff compares two runs of this tool, so pass JSON produced by --json`);
|
|
72
|
+
}
|
|
73
|
+
}
|
|
74
|
+
if (before.kind !== after.kind) {
|
|
75
|
+
throw new CheckDiffRefused(`these are different kinds of report (${before.kind} and ${after.kind}); comparing them would pair things that are not the same measurement`);
|
|
76
|
+
}
|
|
77
|
+
if (before.kind !== "check") {
|
|
78
|
+
throw new CheckDiffRefused(`diff currently compares "check" reports only, and these are "${before.kind}"; run \`efaimo check ... --json\` at both points`);
|
|
79
|
+
}
|
|
80
|
+
const rulesDrift = Boolean(before.rulesVersion && after.rulesVersion && before.rulesVersion !== after.rulesVersion);
|
|
81
|
+
if (rulesDrift && !opts.allowRulesDrift) {
|
|
82
|
+
throw new CheckDiffRefused(`the two runs used different rulesets (${before.rulesVersion} and ${after.rulesVersion}). A grade that moved cannot be attributed to the subject, because the ruler moved too. Re-run both points with one version of efaimo, or pass --allow-rules-drift to compare anyway and have every grade line marked unattributable.`);
|
|
83
|
+
}
|
|
84
|
+
const b = new Map(flattenCheck(before).map((i) => [i.name, i]));
|
|
85
|
+
const a = new Map(flattenCheck(after).map((i) => [i.name, i]));
|
|
86
|
+
const paired = [...b.keys()].filter((k) => a.has(k)).sort();
|
|
87
|
+
if (!paired.length) {
|
|
88
|
+
// An empty intersection and "nothing changed" are the same output, and
|
|
89
|
+
// only one of them is worth printing.
|
|
90
|
+
throw new CheckDiffRefused(`nothing pairs between the two reports (${b.size} items before, ${a.size} after, 0 in common). That is not "no change", it is two reports about different subjects.`);
|
|
91
|
+
}
|
|
92
|
+
const improved = [];
|
|
93
|
+
const worsened = [];
|
|
94
|
+
let held = 0;
|
|
95
|
+
for (const name of paired) {
|
|
96
|
+
const x = b.get(name);
|
|
97
|
+
const y = a.get(name);
|
|
98
|
+
const move = { name, before: { score: x.score, letter: x.letter }, after: { score: y.score, letter: y.letter } };
|
|
99
|
+
if (y.score > x.score)
|
|
100
|
+
improved.push(move);
|
|
101
|
+
else if (y.score < x.score)
|
|
102
|
+
worsened.push(move);
|
|
103
|
+
else
|
|
104
|
+
held++;
|
|
105
|
+
}
|
|
106
|
+
const COLUMNS = [
|
|
107
|
+
["metadata", "always in context", (i) => i.metadata],
|
|
108
|
+
["body", "loads on trigger", (i) => i.body],
|
|
109
|
+
["referenced", "loads on demand", (i) => i.referenced],
|
|
110
|
+
];
|
|
111
|
+
const costs = [];
|
|
112
|
+
for (const [column, when, pick] of COLUMNS) {
|
|
113
|
+
if (paired.some((n) => pick(b.get(n)) === undefined || pick(a.get(n)) === undefined))
|
|
114
|
+
continue;
|
|
115
|
+
const beforeTotal = paired.reduce((s, n) => s + (pick(b.get(n)) ?? 0), 0);
|
|
116
|
+
const afterTotal = paired.reduce((s, n) => s + (pick(a.get(n)) ?? 0), 0);
|
|
117
|
+
const moved = paired
|
|
118
|
+
.map((n) => ({ name: n, before: pick(b.get(n)) ?? 0, after: pick(a.get(n)) ?? 0 }))
|
|
119
|
+
.filter((m) => m.before !== m.after)
|
|
120
|
+
.map((m) => ({ ...m, delta: m.after - m.before }))
|
|
121
|
+
.sort((x, y) => y.delta - x.delta);
|
|
122
|
+
const totalAbs = moved.reduce((s, m) => s + Math.abs(m.delta), 0);
|
|
123
|
+
const biggest = [...moved].sort((x, y) => Math.abs(y.delta) - Math.abs(x.delta))[0];
|
|
124
|
+
const share = biggest && totalAbs ? Math.abs(biggest.delta) / totalAbs : 0;
|
|
125
|
+
costs.push({
|
|
126
|
+
column,
|
|
127
|
+
when,
|
|
128
|
+
before: beforeTotal,
|
|
129
|
+
after: afterTotal,
|
|
130
|
+
delta: afterTotal - beforeTotal,
|
|
131
|
+
pct: beforeTotal ? ((afterTotal - beforeTotal) / beforeTotal) * 100 : 0,
|
|
132
|
+
moved,
|
|
133
|
+
...(share > 0.5 && biggest
|
|
134
|
+
? { dominatedBy: { name: biggest.name, delta: biggest.delta, shareOfMovement: share * 100 } }
|
|
135
|
+
: {}),
|
|
136
|
+
});
|
|
137
|
+
}
|
|
138
|
+
const notes = [
|
|
139
|
+
"the three token columns are reported separately on purpose: metadata is carried permanently, the body loads on trigger, and referenced files load on demand, so a single summed percentage would describe none of them",
|
|
140
|
+
];
|
|
141
|
+
if (rulesDrift) {
|
|
142
|
+
notes.push(`RULES DRIFT ACCEPTED (${before.rulesVersion} -> ${after.rulesVersion}): grade movement below cannot be attributed to the subjects, because the ruleset changed between the two runs`);
|
|
143
|
+
}
|
|
144
|
+
if (before.version !== after.version) {
|
|
145
|
+
notes.push(`the two runs used efaimo ${before.version} and ${after.version}; the rulesets matched, so grades are comparable, but token counts can move if the tokenizer or the walk changed`);
|
|
146
|
+
}
|
|
147
|
+
return {
|
|
148
|
+
kind: before.kind,
|
|
149
|
+
versions: { before: before.version ?? "?", after: after.version ?? "?" },
|
|
150
|
+
rulesVersions: { before: before.rulesVersion, after: after.rulesVersion },
|
|
151
|
+
rulesDrift,
|
|
152
|
+
paired: paired.length,
|
|
153
|
+
added: [...a.keys()].filter((k) => !b.has(k)).sort(),
|
|
154
|
+
removed: [...b.keys()].filter((k) => !a.has(k)).sort(),
|
|
155
|
+
improved,
|
|
156
|
+
worsened,
|
|
157
|
+
held,
|
|
158
|
+
findingCounts: {
|
|
159
|
+
before: [...b.values()].reduce((s, i) => s + i.errors + i.warns + i.infos, 0),
|
|
160
|
+
after: [...a.values()].reduce((s, i) => s + i.errors + i.warns + i.infos, 0),
|
|
161
|
+
},
|
|
162
|
+
costs,
|
|
163
|
+
notes,
|
|
164
|
+
};
|
|
165
|
+
}
|
package/dist/cli.js
CHANGED
|
@@ -11,11 +11,12 @@ import { SPEC_VERSION } from "./clients/rawprobe.js";
|
|
|
11
11
|
import { weighServer, weighSkills } from "./weigh/weigh.js";
|
|
12
12
|
import { findSkills } from "./skills/parse.js";
|
|
13
13
|
import { diffServerWeigh } from "./weigh/diff.js";
|
|
14
|
+
import { diffCheck, CheckDiffRefused } from "./check/diff.js";
|
|
14
15
|
import { DEFAULT_CONTEXT_WINDOW, formatWindowShare, setContextWindow } from "./weigh/window.js";
|
|
15
16
|
import { checkMcpRepoOnly, checkMcpTarget, checkSkillSet } from "./check/check.js";
|
|
16
17
|
import { analyzeFind, DEFAULT_TOP_K } from "./find/find.js";
|
|
17
18
|
import { runFindRules } from "./core/engine.js";
|
|
18
|
-
import { renderCheckPretty, renderDiffPretty, renderFindFindingsPretty, renderFindPretty, renderScenarioPlan, renderServerWeighPretty, renderSkillSetPretty, renderSkillWeighPretty, renderTestReportPretty, setColor, } from "./reporters/pretty.js";
|
|
19
|
+
import { renderCheckPretty, renderDiffPretty, renderFindFindingsPretty, renderFindPretty, renderScenarioPlan, renderServerWeighPretty, renderSkillSetPretty, renderSkillWeighPretty, renderTestReportPretty, setColor, renderCheckDiffPretty, } from "./reporters/pretty.js";
|
|
19
20
|
import { parseScenario, runScenario } from "./testing/harness.js";
|
|
20
21
|
import { acceptsSamplingParams, anthropicRunner } from "./testing/anthropicRunner.js";
|
|
21
22
|
import { openaiRunner, providerForModel } from "./testing/openaiRunner.js";
|
|
@@ -303,6 +304,10 @@ program
|
|
|
303
304
|
.option("--md", "print Markdown")
|
|
304
305
|
.option("--no-timestamp", NO_TIMESTAMP_HELP)
|
|
305
306
|
.option("--badge [file]", "write a grade badge SVG + shields endpoint JSON")
|
|
307
|
+
.option("--out <file>", "write the JSON result to a file (usable as a --diff baseline)")
|
|
308
|
+
.option("--diff <baseline>", "compare against a baseline JSON from --out")
|
|
309
|
+
.option("--allow-rules-drift", "let --diff proceed when the two runs used different rulesets (grade movement is then unattributable)")
|
|
310
|
+
.option("--fail-on-regression", "with --diff, exit 1 if any subject present in both runs scored lower")
|
|
306
311
|
.option("--anthropic [model]", "use exact Claude token counts where relevant")
|
|
307
312
|
.option("--window <tokens>", "context window the share is reported against", String(DEFAULT_CONTEXT_WINDOW))
|
|
308
313
|
.option("--no-color", "disable colors")
|
|
@@ -348,7 +353,46 @@ program
|
|
|
348
353
|
});
|
|
349
354
|
report = res.report;
|
|
350
355
|
}
|
|
356
|
+
// A baseline is written unstamped for the same reason weigh's is: a file
|
|
357
|
+
// that differs on every write is the one field a comparison must ignore,
|
|
358
|
+
// and a committed baseline that diffs on nothing but the clock trains
|
|
359
|
+
// reviewers to skim the file they should be reading. The whole envelope is
|
|
360
|
+
// written rather than just its data, because the version and rulesVersion
|
|
361
|
+
// in it are what make the later comparison trustworthy.
|
|
362
|
+
const baselineAndDiff = (payload) => {
|
|
363
|
+
if (opts.out) {
|
|
364
|
+
fs.writeFileSync(opts.out, toJsonEnvelope("check", payload, { timestamp: false }));
|
|
365
|
+
console.error(pc.dim(`baseline written: ${opts.out}`));
|
|
366
|
+
}
|
|
367
|
+
if (!opts.diff)
|
|
368
|
+
return;
|
|
369
|
+
let baseline;
|
|
370
|
+
try {
|
|
371
|
+
baseline = JSON.parse(fs.readFileSync(opts.diff, "utf8"));
|
|
372
|
+
}
|
|
373
|
+
catch (e) {
|
|
374
|
+
fail(`could not read the baseline "${opts.diff}": ${e.message}`);
|
|
375
|
+
return;
|
|
376
|
+
}
|
|
377
|
+
const current = JSON.parse(toJsonEnvelope("check", payload, { timestamp: false }));
|
|
378
|
+
try {
|
|
379
|
+
const d = diffCheck(baseline, current, { allowRulesDrift: opts.allowRulesDrift });
|
|
380
|
+
console.log("");
|
|
381
|
+
console.log(renderCheckDiffPretty(d));
|
|
382
|
+
if (opts.failOnRegression && d.worsened.length)
|
|
383
|
+
process.exitCode = 1;
|
|
384
|
+
}
|
|
385
|
+
catch (e) {
|
|
386
|
+
// Every refusal here is a comparison that would have produced a
|
|
387
|
+
// plausible number from incomparable inputs, so it stops rather than
|
|
388
|
+
// printing one.
|
|
389
|
+
if (e instanceof CheckDiffRefused)
|
|
390
|
+
fail(e.message);
|
|
391
|
+
throw e;
|
|
392
|
+
}
|
|
393
|
+
};
|
|
351
394
|
if (skillSet) {
|
|
395
|
+
baselineAndDiff(skillSet);
|
|
352
396
|
if (opts.json)
|
|
353
397
|
console.log(toJsonEnvelope("check", skillSet, envelopeOpts(opts)));
|
|
354
398
|
else if (opts.md)
|
|
@@ -365,6 +409,7 @@ program
|
|
|
365
409
|
}
|
|
366
410
|
if (!report)
|
|
367
411
|
return;
|
|
412
|
+
baselineAndDiff(report);
|
|
368
413
|
if (opts.json)
|
|
369
414
|
console.log(toJsonEnvelope("check", report, envelopeOpts(opts)));
|
|
370
415
|
else if (opts.md)
|
|
@@ -424,7 +469,7 @@ program
|
|
|
424
469
|
program
|
|
425
470
|
.command("find")
|
|
426
471
|
.description("would a search surface these tools? (findability under deferred tool loading)")
|
|
427
|
-
.argument("[
|
|
472
|
+
.argument("[targets...]", "one or more MCP servers: stdio command | http(s) URL. Two or more are merged into one catalog and measured together")
|
|
428
473
|
.option("--stdio", "treat target as a stdio command string")
|
|
429
474
|
.option("--header <header>", 'HTTP header "Key: Value" (repeatable)', collectPairs(":"))
|
|
430
475
|
.option("--env <pair>", "KEY=VALUE for stdio servers (repeatable)", collectPairs("="))
|
|
@@ -440,10 +485,15 @@ program
|
|
|
440
485
|
"the catalog: a tool that owns no word the others lack cannot be matched by any query that does not\n" +
|
|
441
486
|
"also match a competitor. `probe` is a simulated BM25 search, offline and deterministic, and it\n" +
|
|
442
487
|
"saturates on well-formed catalogs, so it is a floor test rather than a ranking.\n" +
|
|
488
|
+
"Pass more than one server and their catalogs are MERGED and measured together, with each tool\n" +
|
|
489
|
+
"labelled by where it came from. That is the case worth checking: a server author keeps their own\n" +
|
|
490
|
+
"names apart, nobody coordinates across the several servers a person installs, and the model sees\n" +
|
|
491
|
+
"one flat list. A single-server run is structurally unable to see a collision between two of them.\n" +
|
|
443
492
|
"No API key, no network beyond connecting to the server. docs/METHODOLOGY.md has the method.")
|
|
444
|
-
.action(async (
|
|
493
|
+
.action(async (targetArgs, opts) => {
|
|
445
494
|
colorSetup(opts);
|
|
446
|
-
|
|
495
|
+
const args = targetArgs ?? [];
|
|
496
|
+
if (!args.length)
|
|
447
497
|
fail("nothing to search: pass an MCP server (stdio command or URL)");
|
|
448
498
|
const timeoutMs = seconds(opts.timeout) * 1000;
|
|
449
499
|
const topK = parseNumberOpt(opts.top, "--top", { min: 1 }) ?? DEFAULT_TOP_K;
|
|
@@ -456,20 +506,45 @@ program
|
|
|
456
506
|
const minDistinct = parseNumberOpt(opts.minDistinct, "--min-distinct", { min: 0 });
|
|
457
507
|
if (minDistinct !== undefined && minDistinct > 100)
|
|
458
508
|
fail(`--min-distinct is a percentage, got "${opts.minDistinct}"`);
|
|
459
|
-
const
|
|
460
|
-
|
|
461
|
-
|
|
462
|
-
|
|
509
|
+
const targets = args
|
|
510
|
+
.map((a) => resolveTarget(a, { forceStdio: opts.stdio, env: opts.env, headers: opts.header }))
|
|
511
|
+
.map((target) => {
|
|
512
|
+
if (target.kind === "skillset" || target.kind === "repo") {
|
|
513
|
+
fail(`"${target.label}" is a ${target.kind === "skillset" ? "skill path" : "source directory"}; find needs a live MCP server (stdio command or URL). ` +
|
|
514
|
+
`Skills have no searchable tool catalog; their trigger-overlap check is S103 in \`efaimo check --skill\`.`);
|
|
515
|
+
throw new Error("unreachable");
|
|
516
|
+
}
|
|
517
|
+
return target;
|
|
518
|
+
});
|
|
519
|
+
// Several servers are merged into ONE catalog rather than measured one at
|
|
520
|
+
// a time, because the failure worth finding only exists between them. A
|
|
521
|
+
// server's author keeps their own tool names apart as a matter of course;
|
|
522
|
+
// nobody coordinates across the five or ten servers a person actually
|
|
523
|
+
// installs, and the model sees all of them as one flat list with no
|
|
524
|
+
// indication of which came from where. Measuring them separately would
|
|
525
|
+
// report every catalog as tidy and miss exactly the collision that makes a
|
|
526
|
+
// model pick the wrong tool.
|
|
527
|
+
//
|
|
528
|
+
// The origin is attached for the report and NOT for the index. Prefixing
|
|
529
|
+
// names with their server would hand every tool a term no other tool has,
|
|
530
|
+
// and a catalog of indistinguishable tools would score a perfect 100.
|
|
531
|
+
let tools = [];
|
|
532
|
+
let definitionTokens = 0;
|
|
533
|
+
for (const target of targets) {
|
|
534
|
+
console.error(pc.dim(`connecting to ${target.label} ...`));
|
|
535
|
+
const intro = await introspectServer(target, { timeoutMs });
|
|
536
|
+
// Weighed only to answer one question: is this catalog big enough that a
|
|
537
|
+
// host would defer it. No API key is involved; the count is the local
|
|
538
|
+
// o200k estimate the rest of the tool uses.
|
|
539
|
+
const w = await weighServer(intro);
|
|
540
|
+
definitionTokens += w.totals.claudeStyle;
|
|
541
|
+
tools = tools.concat(targets.length > 1 ? intro.tools.map((tool) => ({ ...tool, origin: target.label })) : intro.tools);
|
|
463
542
|
}
|
|
464
|
-
|
|
465
|
-
const
|
|
466
|
-
// Weighed only to answer one question: is this catalog big enough that a
|
|
467
|
-
// host would defer it. No API key is involved; the count is the local
|
|
468
|
-
// o200k estimate the rest of the tool uses.
|
|
469
|
-
const weigh = await weighServer(intro);
|
|
470
|
-
const result = analyzeFind(target.label, intro.tools, {
|
|
543
|
+
const label = targets.length === 1 ? targets[0].label : `${targets.length} servers`;
|
|
544
|
+
const result = analyzeFind(label, tools, {
|
|
471
545
|
topK,
|
|
472
|
-
definitionTokens
|
|
546
|
+
definitionTokens,
|
|
547
|
+
sources: targets.map((x) => x.label),
|
|
473
548
|
});
|
|
474
549
|
const findings = runFindRules({ find: result });
|
|
475
550
|
if (opts.json) {
|
package/dist/core/types.d.ts
CHANGED
|
@@ -26,6 +26,17 @@ export interface ToolDef {
|
|
|
26
26
|
inputSchema?: unknown;
|
|
27
27
|
outputSchema?: unknown;
|
|
28
28
|
annotations?: Record<string, unknown>;
|
|
29
|
+
/**
|
|
30
|
+
* Which server this tool came from, when several are analysed together.
|
|
31
|
+
*
|
|
32
|
+
* DISPLAY ONLY. It is deliberately absent from `indexedTerms`, and putting
|
|
33
|
+
* it there would be the same bug the query builder already documents: a
|
|
34
|
+
* per-server token has maximal idf, so prefixing names with their origin
|
|
35
|
+
* would hand every tool a word no other tool has and report a catalog of
|
|
36
|
+
* indistinguishable tools as perfectly distinct. The measurement has to see
|
|
37
|
+
* exactly what a model sees, which is a flat list with no origin attached.
|
|
38
|
+
*/
|
|
39
|
+
origin?: string;
|
|
29
40
|
}
|
|
30
41
|
export interface ResourceDef {
|
|
31
42
|
uri: string;
|
|
@@ -188,6 +199,8 @@ export type WeighResult = ServerWeighResult | SkillSetWeighResult;
|
|
|
188
199
|
/** One tool's result in the findability report (`efaimo find`). */
|
|
189
200
|
export interface ToolFindEntry {
|
|
190
201
|
name: string;
|
|
202
|
+
/** The server this tool came from, when more than one was analysed. */
|
|
203
|
+
origin?: string;
|
|
191
204
|
/**
|
|
192
205
|
* Terms this tool has that no other tool in the catalog has.
|
|
193
206
|
*
|
|
@@ -247,6 +260,16 @@ export interface FindResult {
|
|
|
247
260
|
topK: number;
|
|
248
261
|
};
|
|
249
262
|
perTool: ToolFindEntry[];
|
|
263
|
+
/**
|
|
264
|
+
* The servers whose catalogs were merged, when more than one.
|
|
265
|
+
*
|
|
266
|
+
* Present only for a stack. A single server's author avoids collisions
|
|
267
|
+
* inside their own catalog as a matter of course, so a one-server run is
|
|
268
|
+
* structurally unable to see the failure that actually bites: two servers
|
|
269
|
+
* installed side by side, each internally tidy, exposing tools a model
|
|
270
|
+
* cannot tell apart.
|
|
271
|
+
*/
|
|
272
|
+
sources?: string[];
|
|
250
273
|
/**
|
|
251
274
|
* The headline: how many tools own at least one word no other tool has.
|
|
252
275
|
* A measured proportion, deliberately not a letter grade (ADR-030).
|
|
@@ -316,6 +339,24 @@ export interface SkillInfo {
|
|
|
316
339
|
export interface SkillSet {
|
|
317
340
|
root: string;
|
|
318
341
|
skills: SkillInfo[];
|
|
342
|
+
/**
|
|
343
|
+
* Directories the walk refused to enter because it hit its depth bound.
|
|
344
|
+
*
|
|
345
|
+
* Present only when the bound actually bit. A truncated search that reports
|
|
346
|
+
* a short list and no reason is the same failure as a check that examines
|
|
347
|
+
* nothing, so the caller is given the means to say what it did not look at.
|
|
348
|
+
*/
|
|
349
|
+
truncatedAt?: string[];
|
|
350
|
+
/**
|
|
351
|
+
* Files one capitalisation away from being a skill, such as `skill.md`.
|
|
352
|
+
*
|
|
353
|
+
* The spec names `SKILL.md` exactly and discovery matches it exactly, so
|
|
354
|
+
* these are not skills here on any platform. They may still be skills to a
|
|
355
|
+
* host reading a case-insensitive filesystem, which is the default on macOS
|
|
356
|
+
* and Windows, so the same repository can work on a laptop and have no
|
|
357
|
+
* skills at all in Linux CI. Reported by S107.
|
|
358
|
+
*/
|
|
359
|
+
miscasedSkillFiles?: string[];
|
|
319
360
|
}
|
|
320
361
|
export interface GradeInfo {
|
|
321
362
|
score: number;
|
package/dist/find/find.d.ts
CHANGED
|
@@ -112,6 +112,8 @@ export declare function indexedTerms(tool: ToolDef): string[];
|
|
|
112
112
|
*/
|
|
113
113
|
export declare function taskQuery(tool: ToolDef, index: Bm25Index, limit?: number): string[];
|
|
114
114
|
export interface FindOptions {
|
|
115
|
+
/** Labels of the servers merged into this catalog, when more than one. */
|
|
116
|
+
sources?: string[];
|
|
115
117
|
topK?: number;
|
|
116
118
|
queryTerms?: number;
|
|
117
119
|
/** Claude-style tool-definition tokens, used only to say whether deferral is likely. */
|
package/dist/find/find.js
CHANGED
|
@@ -312,6 +312,7 @@ export function analyzeFind(label, tools, opts = {}) {
|
|
|
312
312
|
: [];
|
|
313
313
|
return {
|
|
314
314
|
name: tool.name,
|
|
315
|
+
...(tool.origin ? { origin: tool.origin } : {}),
|
|
315
316
|
ownTerms: own.slice(0, 4),
|
|
316
317
|
ownTermCount: own.length,
|
|
317
318
|
ownOutsideNameCount: ownOutsideName.length,
|
|
@@ -380,6 +381,7 @@ export function analyzeFind(label, tools, opts = {}) {
|
|
|
380
381
|
return {
|
|
381
382
|
kind: "find",
|
|
382
383
|
label,
|
|
384
|
+
...(opts.sources && opts.sources.length > 1 ? { sources: opts.sources } : {}),
|
|
383
385
|
toolCount: tools.length,
|
|
384
386
|
method: { tokenizer: "efaimo-v1", bm25: { ...DEFAULT_BM25 }, queryTerms, topK },
|
|
385
387
|
perTool,
|
package/dist/index.d.ts
CHANGED
|
@@ -5,6 +5,7 @@ export { loadClientServers, SUPPORTED_CLIENTS } from "./targets/clientConfigs.js
|
|
|
5
5
|
export { introspectServer } from "./clients/introspect.js";
|
|
6
6
|
export { weighServer, weighSkills } from "./weigh/weigh.js";
|
|
7
7
|
export { diffServerWeigh, type WeighDiff } from "./weigh/diff.js";
|
|
8
|
+
export { diffCheck, flattenCheck, CheckDiffRefused, type CheckDiffResult, type CheckDiffItem } from "./check/diff.js";
|
|
8
9
|
export { countTokens } from "./weigh/tokens.js";
|
|
9
10
|
export { findSkills, parseSkillFile } from "./skills/parse.js";
|
|
10
11
|
export { analyzeFind, indexedTerms, taskQuery, DEFAULT_TOP_K, DEFAULT_QUERY_TERMS, DEFER_RECOMMENDED_TOKENS, DEFER_RECOMMENDED_TOOLS, type FindOptions } from "./find/find.js";
|
package/dist/index.js
CHANGED
|
@@ -11,6 +11,7 @@ export { loadClientServers, SUPPORTED_CLIENTS } from "./targets/clientConfigs.js
|
|
|
11
11
|
export { introspectServer } from "./clients/introspect.js";
|
|
12
12
|
export { weighServer, weighSkills } from "./weigh/weigh.js";
|
|
13
13
|
export { diffServerWeigh } from "./weigh/diff.js";
|
|
14
|
+
export { diffCheck, flattenCheck, CheckDiffRefused } from "./check/diff.js";
|
|
14
15
|
export { countTokens } from "./weigh/tokens.js";
|
|
15
16
|
// Skills
|
|
16
17
|
export { findSkills, parseSkillFile } from "./skills/parse.js";
|
|
@@ -2,6 +2,7 @@ import type { CheckReport, Finding, FindResult, ServerWeighResult, SkillSetWeigh
|
|
|
2
2
|
import type { CheckSkillResult } from "../check/check.js";
|
|
3
3
|
import type { Scenario, TestReport } from "../testing/harness.js";
|
|
4
4
|
import type { WeighDiff } from "../weigh/diff.js";
|
|
5
|
+
import type { CheckDiffResult } from "../check/diff.js";
|
|
5
6
|
export declare function setColor(on: boolean): void;
|
|
6
7
|
export declare function renderCheckPretty(report: CheckReport): string;
|
|
7
8
|
export declare function renderScenarioPlan(s: Scenario): string;
|
|
@@ -24,3 +25,15 @@ export declare function renderDiffPretty(d: WeighDiff, opts: {
|
|
|
24
25
|
maxTokens?: number;
|
|
25
26
|
allowIncreasePct?: number;
|
|
26
27
|
}): string;
|
|
28
|
+
/**
|
|
29
|
+
* What moved between two `check` runs.
|
|
30
|
+
*
|
|
31
|
+
* The three token columns stay in three rows on purpose. Summing them gives a
|
|
32
|
+
* single percentage that is arithmetically true and describes none of the
|
|
33
|
+
* three costs, because metadata is carried permanently while referenced files
|
|
34
|
+
* load only on demand, and in practice almost all movement lands in the column
|
|
35
|
+
* that costs the least to hold. A dominant mover is printed beside its total
|
|
36
|
+
* for the same reason: when one subject is most of a change, the total is
|
|
37
|
+
* about that subject rather than about the population.
|
|
38
|
+
*/
|
|
39
|
+
export declare function renderCheckDiffPretty(d: CheckDiffResult): string;
|
package/dist/reporters/pretty.js
CHANGED
|
@@ -290,6 +290,11 @@ export function renderFindPretty(f) {
|
|
|
290
290
|
});
|
|
291
291
|
const shown = byWorst.slice(0, 15);
|
|
292
292
|
const nameW = Math.min(34, Math.max(12, ...shown.map((t) => cell(t.name).length)));
|
|
293
|
+
if (f.sources && f.sources.length > 1) {
|
|
294
|
+
// The whole point of a stack run is which server a colliding tool came
|
|
295
|
+
// from, so it goes beside the name rather than into a footnote.
|
|
296
|
+
lines.push(paint(pc.dim, ` merged ${f.sources.length} catalogs: ${f.sources.join(", ")}`));
|
|
297
|
+
}
|
|
293
298
|
lines.push(paint(pc.dim, ` words ${"tool".padEnd(nameW)} vocabulary no other tool has`));
|
|
294
299
|
for (const t of shown) {
|
|
295
300
|
const count = t.ownTermCount === 0
|
|
@@ -313,7 +318,8 @@ export function renderFindPretty(f) {
|
|
|
313
318
|
: t.reachable
|
|
314
319
|
? ""
|
|
315
320
|
: paint(pc.red, ` [probe: rank ${t.rank}]`);
|
|
316
|
-
|
|
321
|
+
const from = t.origin ? paint(pc.dim, ` [${t.origin}]`) : "";
|
|
322
|
+
lines.push(` ${count} ${cell(t.name).padEnd(nameW)} ${tail}${probe}${from}`);
|
|
317
323
|
}
|
|
318
324
|
// Describe the hidden rows from the hidden rows, never from an assumption.
|
|
319
325
|
// This line used to read "(+N more, each owning at least one word)" without
|
|
@@ -397,3 +403,51 @@ export function renderDiffPretty(d, opts) {
|
|
|
397
403
|
}
|
|
398
404
|
return lines.join("\n");
|
|
399
405
|
}
|
|
406
|
+
/**
|
|
407
|
+
* What moved between two `check` runs.
|
|
408
|
+
*
|
|
409
|
+
* The three token columns stay in three rows on purpose. Summing them gives a
|
|
410
|
+
* single percentage that is arithmetically true and describes none of the
|
|
411
|
+
* three costs, because metadata is carried permanently while referenced files
|
|
412
|
+
* load only on demand, and in practice almost all movement lands in the column
|
|
413
|
+
* that costs the least to hold. A dominant mover is printed beside its total
|
|
414
|
+
* for the same reason: when one subject is most of a change, the total is
|
|
415
|
+
* about that subject rather than about the population.
|
|
416
|
+
*/
|
|
417
|
+
export function renderCheckDiffPretty(d) {
|
|
418
|
+
const lines = [];
|
|
419
|
+
lines.push(paint(pc.dim, `efaimo v${VERSION}`));
|
|
420
|
+
const rules = d.rulesVersions.before ?? "?";
|
|
421
|
+
lines.push(`diff check ${paint(pc.bold, `${d.paired} paired`)}${d.added.length ? `, ${d.added.length} added` : ""}${d.removed.length ? `, ${d.removed.length} removed` : ""}`);
|
|
422
|
+
lines.push(paint(pc.dim, d.rulesDrift
|
|
423
|
+
? `efaimo ${d.versions.before} -> ${d.versions.after}, rules ${d.rulesVersions.before} -> ${d.rulesVersions.after} DRIFTED`
|
|
424
|
+
: `efaimo ${d.versions.before} -> ${d.versions.after}, rules ${rules} on both sides`));
|
|
425
|
+
lines.push("");
|
|
426
|
+
const sign = (n) => (n > 0 ? `+${n.toLocaleString("en-US")}` : n.toLocaleString("en-US"));
|
|
427
|
+
const pct = (n) => `${n > 0 ? "+" : ""}${n.toFixed(1)}%`;
|
|
428
|
+
lines.push(paint(pc.bold, "grades"));
|
|
429
|
+
if (d.rulesDrift) {
|
|
430
|
+
lines.push(` ${paint(pc.yellow, "unattributable")}: the ruleset changed between the runs, so a moved grade may be the ruler`);
|
|
431
|
+
}
|
|
432
|
+
const g = (m) => ` ${m.name.padEnd(34)} ${m.before.letter} (${m.before.score}) -> ${m.after.letter} (${m.after.score})`;
|
|
433
|
+
for (const m of d.improved)
|
|
434
|
+
lines.push(paint(pc.green, g(m)));
|
|
435
|
+
for (const m of d.worsened)
|
|
436
|
+
lines.push(paint(pc.red, g(m)));
|
|
437
|
+
lines.push(paint(pc.dim, ` improved ${d.improved.length} worsened ${d.worsened.length} held ${d.held} findings ${d.findingCounts.before} -> ${d.findingCounts.after}`));
|
|
438
|
+
lines.push("");
|
|
439
|
+
if (d.costs.length) {
|
|
440
|
+
lines.push(paint(pc.bold, "token cost, by when it loads"));
|
|
441
|
+
for (const c of d.costs) {
|
|
442
|
+
const color = c.delta > 0 ? pc.yellow : c.delta < 0 ? pc.green : pc.dim;
|
|
443
|
+
lines.push(` ${c.column.padEnd(11)} ${paint(pc.dim, `(${c.when})`.padEnd(20))} ${String(c.before.toLocaleString("en-US")).padStart(9)} -> ${String(c.after.toLocaleString("en-US")).padStart(9)} ${paint(color, pct(c.pct))}`);
|
|
444
|
+
if (c.dominatedBy) {
|
|
445
|
+
lines.push(paint(pc.dim, ` ${c.dominatedBy.name} is ${c.dominatedBy.shareOfMovement.toFixed(1)}% of that movement (${sign(c.dominatedBy.delta)}); the total describes it, not the rest`));
|
|
446
|
+
}
|
|
447
|
+
}
|
|
448
|
+
lines.push("");
|
|
449
|
+
}
|
|
450
|
+
for (const n of d.notes)
|
|
451
|
+
lines.push(paint(pc.dim, ` ${n}`));
|
|
452
|
+
return lines.join("\n");
|
|
453
|
+
}
|
|
@@ -1,5 +1,6 @@
|
|
|
1
1
|
import path from "node:path";
|
|
2
2
|
import { scanTextForInjection } from "../injection.js";
|
|
3
|
+
import { tokenize, QUERY_STOPWORDS } from "../../find/tokenize.js";
|
|
3
4
|
import { formatTokens } from "../../util/misc.js";
|
|
4
5
|
const NAME_RE = /^[a-z0-9]+(?:-[a-z0-9]+)*$/;
|
|
5
6
|
const STANDARD_FIELDS = new Set(["name", "description", "license", "compatibility", "metadata", "allowed-tools"]);
|
|
@@ -77,6 +78,14 @@ const s101 = {
|
|
|
77
78
|
return findings;
|
|
78
79
|
},
|
|
79
80
|
};
|
|
81
|
+
// Grounded on the 38 public skills in research/skills-index/manifest.json,
|
|
82
|
+
// measured 2026-09-03. The shortest real description there is 68 characters
|
|
83
|
+
// (`template-skill`) and the fewest content terms any of them carries is 4,
|
|
84
|
+
// also `template-skill`. Both floors sit below every real description in that
|
|
85
|
+
// corpus and above the filler this rule exists to catch, and both are far
|
|
86
|
+
// under the spec's 1024-character ceiling.
|
|
87
|
+
const S102_MIN_CHARS = 40;
|
|
88
|
+
const S102_MIN_CONTENT_TERMS = 4;
|
|
80
89
|
const s102 = {
|
|
81
90
|
id: "S102",
|
|
82
91
|
title: "weak trigger description",
|
|
@@ -87,7 +96,19 @@ const s102 = {
|
|
|
87
96
|
return []; // S101 already errors
|
|
88
97
|
const findings = [];
|
|
89
98
|
const label = skillLabel(skill);
|
|
90
|
-
|
|
99
|
+
// The content this description carries once the words every description
|
|
100
|
+
// contains are removed. A length floor alone was not enough: until
|
|
101
|
+
// 2026-09-03 this rule read `desc.length < 20`, then asked only whether one
|
|
102
|
+
// of four trigger words appeared anywhere, so "Useful for various tasks."
|
|
103
|
+
// scored a clean 100 because "for" happens to sit inside "for various
|
|
104
|
+
// tasks". It tested for the presence of a trigger WORD rather than for the
|
|
105
|
+
// presence of a trigger, which is a keyword check wearing the costume of a
|
|
106
|
+
// semantic one. Counting what survives the stopword list is the same
|
|
107
|
+
// measurement `find` makes on tool catalogs: a description with almost no
|
|
108
|
+
// distinguishing content cannot be matched to a task by any host, whatever
|
|
109
|
+
// words it happens to include.
|
|
110
|
+
const contentTerms = new Set(tokenize(desc).filter((w) => !QUERY_STOPWORDS.has(w)));
|
|
111
|
+
if (desc.length < S102_MIN_CHARS) {
|
|
91
112
|
findings.push({
|
|
92
113
|
ruleId: "S102",
|
|
93
114
|
severity: "warn",
|
|
@@ -97,6 +118,16 @@ const s102 = {
|
|
|
97
118
|
fixHint: "state what the skill does AND when to use it, with concrete trigger words",
|
|
98
119
|
});
|
|
99
120
|
}
|
|
121
|
+
else if (contentTerms.size < S102_MIN_CONTENT_TERMS) {
|
|
122
|
+
findings.push({
|
|
123
|
+
ruleId: "S102",
|
|
124
|
+
severity: "warn",
|
|
125
|
+
title: "weak trigger description",
|
|
126
|
+
message: `description carries only ${contentTerms.size} distinct content ${contentTerms.size === 1 ? "term" : "terms"} once common words are removed; there is nothing here for a host to match a task against`,
|
|
127
|
+
target: label,
|
|
128
|
+
fixHint: "name the situations, artifacts and words a user would actually type",
|
|
129
|
+
});
|
|
130
|
+
}
|
|
100
131
|
else if (!/\buse\b|\bwhen\b|\bfor\b|\bhelps?\b/i.test(desc)) {
|
|
101
132
|
findings.push({
|
|
102
133
|
ruleId: "S102",
|
package/dist/skills/parse.d.ts
CHANGED
|
@@ -1,4 +1,10 @@
|
|
|
1
1
|
import type { SkillInfo, SkillSet } from "../core/types.js";
|
|
2
2
|
export declare function parseSkillFile(file: string): SkillInfo;
|
|
3
3
|
/** Resolve a path (SKILL.md file, skill dir, or a directory of skills) into a SkillSet. */
|
|
4
|
+
/**
|
|
5
|
+
* How deep skill discovery walks. See the note inside `findSkills`: measured
|
|
6
|
+
* against the public corpus, where the deepest real skill is four directories
|
|
7
|
+
* below a directory of repositories.
|
|
8
|
+
*/
|
|
9
|
+
export declare const SKILL_WALK_MAX_DEPTH = 6;
|
|
4
10
|
export declare function findSkills(input: string): SkillSet;
|
package/dist/skills/parse.js
CHANGED
|
@@ -129,6 +129,12 @@ export function parseSkillFile(file) {
|
|
|
129
129
|
return base;
|
|
130
130
|
}
|
|
131
131
|
/** Resolve a path (SKILL.md file, skill dir, or a directory of skills) into a SkillSet. */
|
|
132
|
+
/**
|
|
133
|
+
* How deep skill discovery walks. See the note inside `findSkills`: measured
|
|
134
|
+
* against the public corpus, where the deepest real skill is four directories
|
|
135
|
+
* below a directory of repositories.
|
|
136
|
+
*/
|
|
137
|
+
export const SKILL_WALK_MAX_DEPTH = 6;
|
|
132
138
|
export function findSkills(input) {
|
|
133
139
|
const abs = path.resolve(input);
|
|
134
140
|
const stat = fs.statSync(abs);
|
|
@@ -139,11 +145,49 @@ export function findSkills(input) {
|
|
|
139
145
|
if (fs.existsSync(direct)) {
|
|
140
146
|
return { root: abs, skills: [parseSkillFile(direct)] };
|
|
141
147
|
}
|
|
148
|
+
// Discovery is deliberate rather than incidental, because two walkers with
|
|
149
|
+
// different rules is what produced the 2026-09-03 defect: this found 34
|
|
150
|
+
// skills in a corpus where `scripts/skills-index.mjs` found 38, and both
|
|
151
|
+
// reported their own number with confidence.
|
|
152
|
+
//
|
|
153
|
+
// maxDepth 6 is measured, not picked. In the public corpus the deepest real
|
|
154
|
+
// skill sits 4 directories below a directory-of-repositories
|
|
155
|
+
// (`<corpus>/<repo>/skills/custom_skills/<name>/SKILL.md`) and 3 below a
|
|
156
|
+
// repository root. Six leaves two levels of headroom and keeps an accidental
|
|
157
|
+
// `check --skill /` from walking a disk. When the bound does bite it is
|
|
158
|
+
// reported rather than swallowed, in `truncatedAt`.
|
|
159
|
+
//
|
|
160
|
+
// Dot directories are entered here and nowhere else in this codebase:
|
|
161
|
+
// `.claude/skills/<name>/` is where a project keeps its own skills, so
|
|
162
|
+
// skipping them made this blind to the most common real layout. `.git` is
|
|
163
|
+
// still excluded by DEFAULT_SKIP_DIRS.
|
|
142
164
|
const skills = [];
|
|
143
|
-
|
|
144
|
-
|
|
165
|
+
const truncatedAt = [];
|
|
166
|
+
const miscased = [];
|
|
167
|
+
for (const f of walkFiles(abs, {
|
|
168
|
+
maxDepth: SKILL_WALK_MAX_DEPTH,
|
|
169
|
+
includeDotDirs: true,
|
|
170
|
+
onDepthLimit: (d) => truncatedAt.push(d),
|
|
171
|
+
})) {
|
|
172
|
+
const base = path.basename(f);
|
|
173
|
+
if (base === "SKILL.md")
|
|
145
174
|
skills.push(parseSkillFile(f));
|
|
175
|
+
// A file that is only a capitalisation away from being a skill. The spec
|
|
176
|
+
// names SKILL.md exactly, and this comparison is exact, so `skill.md` is
|
|
177
|
+
// invisible here on every platform. It is not invisible everywhere: a
|
|
178
|
+
// case-insensitive filesystem, which is the default on macOS and Windows,
|
|
179
|
+
// may hand it to a host that opens it by name, so the same repository has
|
|
180
|
+
// a working skill on the author's laptop and no skill at all in Linux CI.
|
|
181
|
+
// Nothing else in this tool would ever mention it, which is the worst
|
|
182
|
+
// property a near miss can have.
|
|
183
|
+
else if (base.toLowerCase() === "skill.md")
|
|
184
|
+
miscased.push(f);
|
|
146
185
|
}
|
|
147
186
|
skills.sort((a, b) => a.file.localeCompare(b.file));
|
|
148
|
-
return {
|
|
187
|
+
return {
|
|
188
|
+
root: abs,
|
|
189
|
+
skills,
|
|
190
|
+
...(truncatedAt.length ? { truncatedAt } : {}),
|
|
191
|
+
...(miscased.length ? { miscasedSkillFiles: miscased.sort() } : {}),
|
|
192
|
+
};
|
|
149
193
|
}
|
package/dist/util/misc.d.ts
CHANGED
|
@@ -3,10 +3,28 @@ export declare function parseCommandString(input: string): {
|
|
|
3
3
|
command: string;
|
|
4
4
|
args: string[];
|
|
5
5
|
};
|
|
6
|
-
export
|
|
6
|
+
export interface WalkOptions {
|
|
7
7
|
maxDepth?: number;
|
|
8
8
|
skipDirs?: Set<string>;
|
|
9
|
-
|
|
9
|
+
/**
|
|
10
|
+
* Descend into directories whose name starts with a dot.
|
|
11
|
+
*
|
|
12
|
+
* Off by default, because most of them are caches and metadata. On for skill
|
|
13
|
+
* discovery, because `.claude/skills/<name>/` is where a project keeps its
|
|
14
|
+
* own skills and a tool that cannot see the most common real layout is not
|
|
15
|
+
* auditing anything. `DEFAULT_SKIP_DIRS` still excludes `.git`.
|
|
16
|
+
*/
|
|
17
|
+
includeDotDirs?: boolean;
|
|
18
|
+
/**
|
|
19
|
+
* Called once for each directory not entered because `maxDepth` was reached.
|
|
20
|
+
*
|
|
21
|
+
* A bound that truncates in silence is the same failure as a check that
|
|
22
|
+
* examines nothing: the caller sees a short list and no reason for it. This
|
|
23
|
+
* lets the caller say so.
|
|
24
|
+
*/
|
|
25
|
+
onDepthLimit?: (dir: string) => void;
|
|
26
|
+
}
|
|
27
|
+
export declare function walkFiles(root: string, opts?: WalkOptions): Generator<string>;
|
|
10
28
|
export declare function readTextSafe(file: string, maxBytes?: number): string | undefined;
|
|
11
29
|
export declare function isBinaryPath(file: string): boolean;
|
|
12
30
|
export declare function extractVersion(s: string): number[] | undefined;
|
package/dist/util/misc.js
CHANGED
|
@@ -59,9 +59,15 @@ export function* walkFiles(root, opts = {}) {
|
|
|
59
59
|
for (const e of entries) {
|
|
60
60
|
const full = path.join(dir, e.name);
|
|
61
61
|
if (e.isDirectory()) {
|
|
62
|
-
if (
|
|
63
|
-
|
|
62
|
+
if (skip.has(e.name))
|
|
63
|
+
continue;
|
|
64
|
+
if (!opts.includeDotDirs && e.name.startsWith("."))
|
|
65
|
+
continue;
|
|
66
|
+
if (depth >= maxDepth) {
|
|
67
|
+
opts.onDepthLimit?.(full);
|
|
68
|
+
continue;
|
|
64
69
|
}
|
|
70
|
+
stack.push({ dir: full, depth: depth + 1 });
|
|
65
71
|
}
|
|
66
72
|
else if (e.isFile()) {
|
|
67
73
|
yield full;
|
package/dist/version.d.ts
CHANGED
|
@@ -1,2 +1,2 @@
|
|
|
1
1
|
/** Kept in sync with package.json by test/meta.test.ts. */
|
|
2
|
-
export declare const VERSION = "0.
|
|
2
|
+
export declare const VERSION = "0.4.0";
|
package/dist/version.js
CHANGED
|
@@ -1,2 +1,2 @@
|
|
|
1
1
|
/** Kept in sync with package.json by test/meta.test.ts. */
|
|
2
|
-
export const VERSION = "0.
|
|
2
|
+
export const VERSION = "0.4.0";
|
package/docs/RULES.md
CHANGED
|
@@ -65,7 +65,7 @@ never reused, so a gap is permanent and is better stated than smoothed over.
|
|
|
65
65
|
| E128 | warn | total tool-definition tax over ~10k / ~25k tokens, estimated (needs `weigh` data); hard CI failure is left to the `--max-tokens`/`--diff` gate |
|
|
66
66
|
| E130 | info | instruction-injection patterns in a tool description or server instructions (shallow heuristic, never a security verdict; use a dedicated scanner for depth). **Reported but never scored** |
|
|
67
67
|
|
|
68
|
-
## Skills (S101-
|
|
68
|
+
## Skills (S101-S107)
|
|
69
69
|
|
|
70
70
|
| id | sev | what it catches | spec |
|
|
71
71
|
|---|---|---|---|
|
|
@@ -75,6 +75,7 @@ never reused, so a gap is permanent and is better stated than smoothed over.
|
|
|
75
75
|
| S104 | warn/info | metadata over ~100 tokens, body over ~5k tokens or 500 lines, or long body with no progressive disclosure | agentskills.io |
|
|
76
76
|
| S105 | info | instruction-injection patterns in the skill body or description (shallow heuristic, never a security verdict). **Reported but never scored** | internal |
|
|
77
77
|
| S106 | error/warn/info | referenced file missing, escaping the skill dir (`..`), or nested more than one level deep | agentskills.io |
|
|
78
|
+
| S107 | warn | a file one capitalisation away from a skill, such as `skill.md`. Set-level and never scored: the spec names `SKILL.md` exactly, so it is not a skill here on any platform, but a case-insensitive filesystem (the macOS and Windows default) may hand it to a host, which makes a repository work on a laptop and have no skill at all in Linux CI | internal |
|
|
78
79
|
|
|
79
80
|
## Findability (E141-E145)
|
|
80
81
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "efaimo",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.4.0",
|
|
4
4
|
"description": "Audit what your agent loads: quality and context cost for MCP servers and Agent Skills",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"mcp",
|
|
@@ -58,6 +58,8 @@
|
|
|
58
58
|
"build": "tsc -p tsconfig.build.json",
|
|
59
59
|
"typecheck": "tsc --noEmit",
|
|
60
60
|
"test": "vitest run",
|
|
61
|
+
"preflight": "node scripts/preflight.mjs",
|
|
62
|
+
"preflight:fast": "node scripts/preflight.mjs --fast",
|
|
61
63
|
"spec:drift": "node scripts/spec-drift.mjs",
|
|
62
64
|
"test:watch": "vitest",
|
|
63
65
|
"prepublishOnly": "npm run typecheck && npm run build && npm run test"
|