@andrueandersoncs/lint-anything 0.1.0 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +58 -17
- package/dist/cli.js +53 -53
- package/dist/lib.js +699 -0
- package/dist/types/clef.d.ts +82 -0
- package/dist/types/item.d.ts +20 -0
- package/dist/types/lib.d.ts +22 -0
- package/dist/types/lint.d.ts +66 -0
- package/dist/types/policy.d.ts +56 -0
- package/dist/types/score.d.ts +195 -0
- package/dist/types/select.d.ts +36 -0
- package/package.json +24 -2
- package/skills/lint-anything/SKILL.md +40 -7
- package/skills/lint-anything/scripts/score.js +52 -0
package/README.md
CHANGED
|
@@ -56,9 +56,20 @@ Rule: every `TODO` or `FIXME` comment must reference a tracking issue, written a
|
|
|
56
56
|
|
|
57
57
|
A missing rules directory, one with no `.md` files, or an invalid policy stops the run.
|
|
58
58
|
|
|
59
|
-
Policies are meant to be written and tuned by a coding agent: give it the rule you want
|
|
59
|
+
Policies are meant to be written and tuned by a coding agent: give it the rule you want and let it iterate on the selector, the body, and the criteria together, measured against labeled code (below). Because the CLI adds nothing, any way of asking Clef that a request can express is available to a policy. `evals/corpus/selectors-excerpts/` is the earlier CLI's whole-file judging written as policies: its selector cuts files into 1,000-byte excerpts, and its body and criteria are that CLI's own words.
|
|
60
60
|
|
|
61
|
-
`skills/lint-anything/SKILL.md` is an agent skill covering running the CLI, acting on findings, writing and tuning policies, and the pitfalls selectors run into. It ships in the package (`node_modules/@andrueandersoncs/lint-anything/skills/lint-anything/`); copy that directory into a coding agent's skills directory.
|
|
61
|
+
`skills/lint-anything/SKILL.md` is an agent skill covering running the CLI, acting on findings, writing and tuning policies, and the pitfalls selectors run into. It ships in the package (`node_modules/@andrueandersoncs/lint-anything/skills/lint-anything/`) with its score script; copy that directory into a coding agent's skills directory.
|
|
62
|
+
|
|
63
|
+
### Tuning against labels
|
|
64
|
+
|
|
65
|
+
A policy's labels live in `.lint-anything/labels/<policy>.jsonl`, next to the rules and committed with them; the CLI never reads them. Each line labels one file (`path`, or `item` for items without a path) as `violates` (with the line ranges of its violations), `complies`, or `ambiguous`, and puts it in the `dev` or `test` split. The skill's script scores a policy against them from the project root, judging only items in labeled files:
|
|
66
|
+
|
|
67
|
+
```sh
|
|
68
|
+
bun <skill directory>/scripts/score.js <policy> # per split: reach, AUROC [95%], caught, surfaced, false alarms
|
|
69
|
+
bun <skill directory>/scripts/score.js <policy> candidate.md # and candidate − current, paired intervals
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
An item counts as a violation when its lines overlap a labeled violation, and a file scores its highest item score, so labels stay valid when the selector changes. It lists dev mistakes and unreached violations; test is only summarized, and a change is adopted when test improves outside noise. The skill asks for labels from two or three different models, with the user settling their disagreements. The evals score the corpus with the same code (`src/score.ts`).
|
|
62
73
|
|
|
63
74
|
### Selectors
|
|
64
75
|
|
|
@@ -163,16 +174,46 @@ An error that stops the run (exit `2`) is printed to stderr as one line, `error:
|
|
|
163
174
|
bun install
|
|
164
175
|
bunx tsc --noEmit
|
|
165
176
|
bun index.ts # run from source
|
|
166
|
-
bun run build #
|
|
167
|
-
bun pm pack # the tarball npm would get: dist/cli.js, skills
|
|
177
|
+
bun run build # dist/cli.js (the `lint-anything` bin), skills/lint-anything/scripts/score.js (the skill's score script), dist/lib.js and dist/types (the library)
|
|
178
|
+
bun pm pack # the tarball npm would get: dist/cli.js, dist/lib.js, dist/types, skills/ (with score.js), README, LICENSE
|
|
168
179
|
npm publish # runs prepublishOnly (type check, build) first; the package is public
|
|
169
180
|
```
|
|
170
181
|
|
|
171
|
-
The bundle includes every dependency, so the
|
|
182
|
+
The CLI bundle includes every dependency, so installing the package for the CLI installs nothing else. The version printed by `--version` is `package.json`'s, inlined at build time.
|
|
183
|
+
|
|
184
|
+
## Library
|
|
185
|
+
|
|
186
|
+
The package also exports the stages a run composes as an [Effect](https://effect.website) library (Bun only). `effect` 4.x is an optional peer dependency: install it to use the library.
|
|
187
|
+
|
|
188
|
+
| Export | What it does |
|
|
189
|
+
| --- | --- |
|
|
190
|
+
| `select(policy, cwd?, runs?)` | Runs a policy's selector; returns its items, sorted as they are judged |
|
|
191
|
+
| `judge(policy, items)` | Each item's violation probability, or `RequestTooLarge` (needs `Clef`) |
|
|
192
|
+
| `decide(policy, items, answers, threshold)` | Findings from those answers, `limit` included; pure |
|
|
193
|
+
| `lint(policies, threshold)` | All three for every policy, as the CLI runs them |
|
|
194
|
+
| `Clef.ask(source, state, questions)` | Yes/no questions under a shared state, packed, cached, retried; no policy needed |
|
|
195
|
+
| `Clef.layer(cacheDirectory?)` | The `Clef` service; reads the environment variables above |
|
|
196
|
+
| `loadPolicies(directory)`, `parsePolicy(name, source)` | Policies from Markdown (`loadPolicies` needs `FileSystem` and `Path`, e.g. `BunServices.layer` from `@effect/platform-bun`) |
|
|
197
|
+
| `loadLabels(policy)`, `scorePolicy(policy, labels, cwd?)` | A policy scored against `.lint-anything/labels/<policy>.jsonl`: per labeled file and item, with reach |
|
|
198
|
+
| `quality`, `auroc`, `pairedInterval`, `bootstrapInterval`, … | The metrics and 95% bootstrap intervals the evals and the score script report |
|
|
199
|
+
|
|
200
|
+
```ts
|
|
201
|
+
import { BunServices } from "@effect/platform-bun"
|
|
202
|
+
import { Effect, Layer } from "effect"
|
|
203
|
+
import { Clef, decide, defaultThreshold, judge, loadPolicies, select } from "@andrueandersoncs/lint-anything"
|
|
204
|
+
|
|
205
|
+
const findings = await Effect.runPromise(Effect.gen(function*() {
|
|
206
|
+
const [policy] = yield* loadPolicies(".lint-anything/rules")
|
|
207
|
+
const items = yield* select(policy!)
|
|
208
|
+
return decide(policy!, items, yield* judge(policy!, items), defaultThreshold)
|
|
209
|
+
}).pipe(Effect.provide(Layer.mergeAll(Clef.layer(".lint-anything/cache"), BunServices.layer))))
|
|
210
|
+
```
|
|
211
|
+
|
|
212
|
+
Selectors still run in their own process groups and are killed when the process exits. The library installs no signal handlers: a host that wants selectors killed on SIGHUP calls `killRunningSelectors()` from its own handler, as the CLI does.
|
|
172
213
|
|
|
173
214
|
## Evals
|
|
174
215
|
|
|
175
|
-
`evals/` scores policies (body, selector, criteria) and request shapes against a hand-labeled corpus, so changes can be measured instead of guessed.
|
|
216
|
+
`evals/` scores policies (body, selector, criteria) and request shapes against a hand-labeled corpus, so changes can be measured instead of guessed. Its item labeling, file scores, metrics, and intervals are `src/score.ts`, the same code the skill's score script reports with.
|
|
176
217
|
|
|
177
218
|
```sh
|
|
178
219
|
bun evals/validate.ts # check the corpus, print label counts per policy and origin
|
|
@@ -249,7 +290,7 @@ The same rules as the earlier excerpt CLI (`evals/RESEARCH.md`, H19), as excerpt
|
|
|
249
290
|
- Selectors built for each rule beat it: +0.016 to +0.019 AUROC, and at the same false-alarm rate they find 10–12 points more violations. At the CLI's cut-offs they catch 69% of test violations and surface 85%, with 13% false alarms (the excerpt CLI: 55%, 78%, 14%), for $14–16 per 1,000 files (excerpt CLI: $20–22).
|
|
250
291
|
- Per rule, a selector can still lose to the floor, and with 3–70 violations per rule and split the evidence is thin. Picking per rule on dev (custom selector only where its dev gain over the excerpt policy is outside noise) keeps every rule on the excerpt policy and gives up the pooled gain. On test, no-flag-parameters loses to its excerpt policy outside noise (0.977 → 0.915), although dev favored the selector (0.961 → 0.976): its selector only shows functions with a positional boolean, and Clef flags most of them. A coding agent tuning a policy should keep the excerpt policy as its baseline and hold out data to check.
|
|
251
292
|
- Control rules on test: 0.93 for Go errors, 1.0 for the other three.
|
|
252
|
-
- File size still matters a little. Real test files: AUROC 0.919 under 4 KB, 0.
|
|
293
|
+
- File size still matters a little. Real test files: AUROC 0.919 under 4 KB, 0.897 at 4–9 KB, 0.917 above 9 KB.
|
|
253
294
|
- Measured with earlier request shapes (`evals/RESEARCH.md`): hand-picked files are harder than random ones (semantic AUROC 0.804 curated, 0.882 sampled, 0.845 mined, H10–H11); Clef doesn't score famous or older code better (H14); LLM plants run slightly easier than real violations of the same policies, within noise (+0.033 [−0.020, +0.085], H12).
|
|
254
295
|
|
|
255
296
|
Rule wording, each set scored against gold labeled for its own text, through the corpus selectors (`wordings.ts --origin real`, real files):
|
|
@@ -258,10 +299,10 @@ Rule wording, each set scored against gold labeled for its own text, through the
|
|
|
258
299
|
| --- | --- | --- | --- | --- | --- |
|
|
259
300
|
| terse (one line per rule) | dev | 611 → 809 | 0.937 → 0.890 [−0.061, −0.033] | 13.0% → 26.6% | 68% → 77% |
|
|
260
301
|
| | test | 537 → 677 | 0.930 → 0.886 [−0.060, −0.030] | 13.3% → 26.8% | 69% → 74% |
|
|
261
|
-
| style-guide passages | dev | 611 → 755 | 0.937 → 0.909 [−0.040, −0.016] | 13.0% → 16.
|
|
262
|
-
| | test | 537 → 658 | 0.930 → 0.900 [−0.044, −0.
|
|
263
|
-
| AGENTS.md rules (16 policies) | dev | 591 → 834 | 0.939 → 0.895 [−0.060, −0.
|
|
264
|
-
| | test | 524 → 735 | 0.933 → 0.882 [−0.
|
|
302
|
+
| style-guide passages | dev | 611 → 755 | 0.937 → 0.909 [−0.040, −0.016] | 13.0% → 16.7% | 68% → 65% |
|
|
303
|
+
| | test | 537 → 658 | 0.930 → 0.900 [−0.044, −0.014] | 13.3% → 16.7% | 69% → 63% |
|
|
304
|
+
| AGENTS.md rules (16 policies) | dev | 591 → 834 | 0.939 → 0.895 [−0.060, −0.029] | 12.4% → 20.4% | 68% → 71% |
|
|
305
|
+
| | test | 524 → 735 | 0.933 → 0.882 [−0.069, −0.035] | 12.2% → 20.5% | 69% → 68% |
|
|
265
306
|
|
|
266
307
|
- A selector belongs to its rule. The corpus selectors were built for the polished rules, and a reworded rule means something different: its gold has 23–41% more violations, and the polished rules' selectors reach fewer of them (dev: 98% of gold ranges for the polished rules, 87–95% for the reworded ones). Through selectors built for other text, reworded rules lose 0.03–0.05 AUROC and raise false alarms. A rule needs a selector written for it.
|
|
267
308
|
|
|
@@ -289,8 +330,8 @@ Each feature has an intended purpose, an eval that measures it, and a result. Ch
|
|
|
289
330
|
| `limit` | Count rules judged reliably | `counts.ts` | Holds: 100% vs 92% (79% at the boundary) | yes |
|
|
290
331
|
| Per-policy cut-offs | Better verdicts than global cut-offs | `cutoffs.ts` | Fails: surfaced violations fall from 72% to 52% for +0.002 F1 | no |
|
|
291
332
|
| Policy file sent whole as the rule | Clef reads the rule with its frontmatter | `evals/.cache/research/ident/ablate.ts` (2026-10-22) | Replaced: with the excerpt framing, sending the body without the frontmatter (selector code) raised dev AUROC 0.910 → 0.919; the body alone is now the request's text | no |
|
|
292
|
-
| `ask` routing step | Drop items a rule doesn't govern, cutting false alarms | (commit `
|
|
293
|
-
| `check` subcommand, `--baseline` / `--changed-since`, `--sarif` | Example checks; adoption on existing code; code scanning | (commits `
|
|
333
|
+
| `ask` routing step | Drop items a rule doesn't govern, cutting false alarms | (commit `a4caa7c`) | Fails: dev false alarms rose, violations dropped | no |
|
|
334
|
+
| `check` subcommand, `--baseline` / `--changed-since`, `--sarif` | Example checks; adoption on existing code; code scanning | (commits `47705a7` to `cfab4c3`) | Each measured; see below | no: removed, because the CLI takes no subcommands and no flags but `--debug` |
|
|
294
335
|
|
|
295
336
|
#### Context from outside the file
|
|
296
337
|
|
|
@@ -387,10 +428,10 @@ Either default passes a clean checkout that has violations, and changed-since fa
|
|
|
387
428
|
|
|
388
429
|
These were built and measured with new flags and subcommands, then removed because the CLI takes no arguments. The code and evals are in the git history.
|
|
389
430
|
|
|
390
|
-
- `lint-anything check` (commit `
|
|
391
|
-
- `--baseline` and `--changed-since` (commit `
|
|
392
|
-
- `--sarif` (commit `
|
|
393
|
-
- `ask` routing step (commit `
|
|
431
|
+
- `lint-anything check` (commit `47705a7`): judged each policy's examples against the rule without them. It caught a rule about the wrong thing (examples agreeing fell from 90% to 48%) but not an inverted one (87%), and it failed two semantic policies in their own wording.
|
|
432
|
+
- `--baseline` and `--changed-since` (commit `9ddefd2`): on a generated repository, all five adoption scenarios behaved as expected; a baseline keyed by item text survived line shifts (one keyed by line would have called 7 of 11 accepted findings new).
|
|
433
|
+
- `--sarif` (commit `cfab4c3`): the log validated against the SARIF 2.1.0 schema and every finding mapped to one result.
|
|
434
|
+
- `ask` routing step (commit `a4caa7c`): routing the comments policy's items through "is this exempt?" raised dev false alarms from 8.2% to 11.5% and dropped 3 of 36 violating comments.
|
|
394
435
|
|
|
395
436
|
#### Counting with `limit`
|
|
396
437
|
|