@andrueandersoncs/lint-anything 0.1.1 → 0.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -56,9 +56,20 @@ Rule: every `TODO` or `FIXME` comment must reference a tracking issue, written a
56
56
 
57
57
  A missing rules directory, one with no `.md` files, or an invalid policy stops the run.
58
58
 
59
- Policies are meant to be written and tuned by a coding agent: give it the rule you want, files that break and follow it, and `evals/` as the way to measure, and let it iterate on the selector, the body, and the criteria together. Because the CLI adds nothing, any way of asking Clef that a request can express is available to a policy. `evals/corpus/selectors-excerpts/` is the earlier CLI's whole-file judging written as policies: its selector cuts files into 1,000-byte excerpts, and its body and criteria are that CLI's own words.
59
+ Policies are meant to be written and tuned by a coding agent: give it the rule you want and let it iterate on the selector, the body, and the criteria together, measured against labeled code (below). Because the CLI adds nothing, any way of asking Clef that a request can express is available to a policy. `evals/corpus/selectors-excerpts/` is the earlier CLI's whole-file judging written as policies: its selector cuts files into 1,000-byte excerpts, and its body and criteria are that CLI's own words.
60
60
 
61
- `skills/lint-anything/SKILL.md` is an agent skill covering running the CLI, acting on findings, writing and tuning policies, and the pitfalls selectors run into. It ships in the package (`node_modules/@andrueandersoncs/lint-anything/skills/lint-anything/`); copy that directory into a coding agent's skills directory.
61
+ `skills/lint-anything/SKILL.md` is an agent skill covering running the CLI, acting on findings, writing and tuning policies, and the pitfalls selectors run into. It ships in the package (`node_modules/@andrueandersoncs/lint-anything/skills/lint-anything/`) with its score script; copy that directory into a coding agent's skills directory.
62
+
63
+ ### Tuning against labels
64
+
65
+ A policy's labels live in `.lint-anything/labels/<policy>.jsonl`, next to the rules and committed with them; the CLI never reads them. Each line labels one file (`path`, or `item` for items without a path) as `violates` (with the line ranges of its violations), `complies`, or `ambiguous`, and puts it in the `dev` or `test` split. The skill's script scores a policy against them from the project root, judging only items in labeled files:
66
+
67
+ ```sh
68
+ bun <skill directory>/scripts/score.js <policy> # per split: reach, AUROC [95%], caught, surfaced, false alarms
69
+ bun <skill directory>/scripts/score.js <policy> candidate.md # and candidate − current, paired intervals
70
+ ```
71
+
72
+ An item counts as a violation when its lines overlap a labeled violation, and a file scores its highest item score, so labels stay valid when the selector changes. It lists dev mistakes and unreached violations; test is only summarized, and a change is adopted when test improves outside noise. The skill asks for labels from two or three different models, with the user settling their disagreements. The evals score the corpus with the same code (`src/score.ts`).
62
73
 
63
74
  ### Selectors
64
75
 
@@ -163,16 +174,46 @@ An error that stops the run (exit `2`) is printed to stderr as one line, `error:
163
174
  bun install
164
175
  bunx tsc --noEmit
165
176
  bun index.ts # run from source
166
- bun run build # bundle to dist/cli.js, the published `lint-anything` bin
167
- bun pm pack # the tarball npm would get: dist/cli.js, skills/, README, LICENSE
177
+ bun run build # dist/cli.js (the `lint-anything` bin), skills/lint-anything/scripts/score.js (the skill's score script), dist/lib.js and dist/types (the library)
178
+ bun pm pack # the tarball npm would get: dist/cli.js, dist/lib.js, dist/types, skills/ (with score.js), README, LICENSE
168
179
  npm publish # runs prepublishOnly (type check, build) first; the package is public
169
180
  ```
170
181
 
171
- The bundle includes every dependency, so the published package installs nothing else. The version printed by `--version` is `package.json`'s, inlined at build time.
182
+ The CLI bundle includes every dependency, so installing the package for the CLI installs nothing else. The version printed by `--version` is `package.json`'s, inlined at build time.
183
+
184
+ ## Library
185
+
186
+ The package also exports the stages a run composes as an [Effect](https://effect.website) library (Bun only). `effect` 4.1 or later (`^4.1.2`) is an optional peer dependency: install it to use the library.
187
+
188
+ | Export | What it does |
189
+ | --- | --- |
190
+ | `select(policy, cwd?, runs?)` | Runs a policy's selector; returns its items, sorted as they are judged |
191
+ | `judge(policy, items)` | Each item's violation probability, or `RequestTooLarge` (needs `Clef`) |
192
+ | `decide(policy, items, answers, threshold)` | Findings from those answers, `limit` included; pure |
193
+ | `lint(policies, threshold)` | All three for every policy, as the CLI runs them |
194
+ | `Clef.ask(source, state, questions)` | Yes/no questions under a shared state, packed, cached, retried; no policy needed |
195
+ | `Clef.layer(cacheDirectory?)` | The `Clef` service; reads the environment variables above |
196
+ | `loadPolicies(directory)`, `parsePolicy(name, source)` | Policies from Markdown (`loadPolicies` needs `FileSystem` and `Path`, e.g. `BunServices.layer` from `@effect/platform-bun`) |
197
+ | `loadLabels(policy)`, `scorePolicy(policy, labels, cwd?)` | A policy scored against `.lint-anything/labels/<policy>.jsonl`: per labeled file and item, with reach |
198
+ | `quality`, `auroc`, `pairedInterval`, `bootstrapInterval`, … | The metrics and 95% bootstrap intervals the evals and the score script report |
199
+
200
+ ```ts
201
+ import { BunServices } from "@effect/platform-bun"
202
+ import { Effect, Layer } from "effect"
203
+ import { Clef, decide, defaultThreshold, judge, loadPolicies, select } from "@andrueandersoncs/lint-anything"
204
+
205
+ const findings = await Effect.runPromise(Effect.gen(function*() {
206
+ const [policy] = yield* loadPolicies(".lint-anything/rules")
207
+ const items = yield* select(policy!)
208
+ return decide(policy!, items, yield* judge(policy!, items), defaultThreshold)
209
+ }).pipe(Effect.provide(Layer.mergeAll(Clef.layer(".lint-anything/cache"), BunServices.layer))))
210
+ ```
211
+
212
+ Selectors still run in their own process groups and are killed when the process exits. The library installs no signal handlers: a host that wants selectors killed on SIGHUP calls `killRunningSelectors()` from its own handler, as the CLI does.
172
213
 
173
214
  ## Evals
174
215
 
175
- `evals/` scores policies (body, selector, criteria) and request shapes against a hand-labeled corpus, so changes can be measured instead of guessed. It is also the loop a coding agent uses to write and tune a policy.
216
+ `evals/` scores policies (body, selector, criteria) and request shapes against a hand-labeled corpus, so changes can be measured instead of guessed. Its item labeling, file scores, metrics, and intervals are `src/score.ts`, the same code the skill's score script reports with.
176
217
 
177
218
  ```sh
178
219
  bun evals/validate.ts # check the corpus, print label counts per policy and origin
@@ -249,7 +290,7 @@ The same rules as the earlier excerpt CLI (`evals/RESEARCH.md`, H19), as excerpt
249
290
  - Selectors built for each rule beat it: +0.016 to +0.019 AUROC, and at the same false-alarm rate they find 10–12 points more violations. At the CLI's cut-offs they catch 69% of test violations and surface 85%, with 13% false alarms (the excerpt CLI: 55%, 78%, 14%), for $14–16 per 1,000 files (excerpt CLI: $20–22).
250
291
  - Per rule, a selector can still lose to the floor, and with 3–70 violations per rule and split the evidence is thin. Picking per rule on dev (custom selector only where its dev gain over the excerpt policy is outside noise) keeps every rule on the excerpt policy and gives up the pooled gain. On test, no-flag-parameters loses to its excerpt policy outside noise (0.977 → 0.915), although dev favored the selector (0.961 → 0.976): its selector only shows functions with a positional boolean, and Clef flags most of them. A coding agent tuning a policy should keep the excerpt policy as its baseline and hold out data to check.
251
292
  - Control rules on test: 0.93 for Go errors, 1.0 for the other three.
252
- - File size still matters a little. Real test files: AUROC 0.919 under 4 KB, 0.898 at 4–9 KB, 0.917 above 9 KB.
293
+ - File size still matters a little. Real test files: AUROC 0.919 under 4 KB, 0.897 at 4–9 KB, 0.917 above 9 KB.
253
294
  - Measured with earlier request shapes (`evals/RESEARCH.md`): hand-picked files are harder than random ones (semantic AUROC 0.804 curated, 0.882 sampled, 0.845 mined, H10–H11); Clef doesn't score famous or older code better (H14); LLM plants run slightly easier than real violations of the same policies, within noise (+0.033 [−0.020, +0.085], H12).
254
295
 
255
296
  Rule wording, each set scored against gold labeled for its own text, through the corpus selectors (`wordings.ts --origin real`, real files):
@@ -258,10 +299,10 @@ Rule wording, each set scored against gold labeled for its own text, through the
258
299
  | --- | --- | --- | --- | --- | --- |
259
300
  | terse (one line per rule) | dev | 611 → 809 | 0.937 → 0.890 [−0.061, −0.033] | 13.0% → 26.6% | 68% → 77% |
260
301
  | | test | 537 → 677 | 0.930 → 0.886 [−0.060, −0.030] | 13.3% → 26.8% | 69% → 74% |
261
- | style-guide passages | dev | 611 → 755 | 0.937 → 0.909 [−0.040, −0.016] | 13.0% → 16.6% | 68% → 65% |
262
- | | test | 537 → 658 | 0.930 → 0.900 [−0.044, −0.015] | 13.3% → 16.7% | 69% → 63% |
263
- | AGENTS.md rules (16 policies) | dev | 591 → 834 | 0.939 → 0.895 [−0.060, −0.030] | 12.4% → 20.4% | 68% → 71% |
264
- | | test | 524 → 735 | 0.933 → 0.882 [−0.068, −0.035] | 12.2% → 20.5% | 69% → 68% |
302
+ | style-guide passages | dev | 611 → 755 | 0.937 → 0.909 [−0.040, −0.016] | 13.0% → 16.7% | 68% → 65% |
303
+ | | test | 537 → 658 | 0.930 → 0.900 [−0.044, −0.014] | 13.3% → 16.7% | 69% → 63% |
304
+ | AGENTS.md rules (16 policies) | dev | 591 → 834 | 0.939 → 0.895 [−0.060, −0.029] | 12.4% → 20.4% | 68% → 71% |
305
+ | | test | 524 → 735 | 0.933 → 0.882 [−0.069, −0.035] | 12.2% → 20.5% | 69% → 68% |
265
306
 
266
307
  - A selector belongs to its rule. The corpus selectors were built for the polished rules, and a reworded rule means something different: its gold has 23–41% more violations, and the polished rules' selectors reach fewer of them (dev: 98% of gold ranges for the polished rules, 87–95% for the reworded ones). Through selectors built for other text, reworded rules lose 0.03–0.05 AUROC and raise false alarms. A rule needs a selector written for it.
267
308