@andrueandersoncs/lint-anything 0.4.0 → 0.5.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -82,7 +82,7 @@ Each question shows one comment. Decide whether it breaks the rule below.
82
82
  Rule: every `TODO` or `FIXME` comment must reference a tracking issue, written as `#123`, `ABC-123`, or a full issue URL.
83
83
  ````
84
84
 
85
- - Frontmatter holds `select` and `criteria` (`true` and `false`), all required, and `limit` (below). Any other key is an error.
85
+ - Frontmatter holds `select` and `criteria` (`true` and `false`), both required. Any other key is an error.
86
86
  - The body must not be empty.
87
87
  - The policy name is its path relative to the rules directory, without `.md` (for example `sub/todo-has-issue`).
88
88
 
@@ -125,28 +125,6 @@ An item counts as a violation when its lines overlap a labeled violation, and a
125
125
  - Structural units (functions, classes, handlers, tests) are easiest to select with a syntax-aware tool such as [ast-grep](https://ast-grep.github.io/). The eval corpus's policies (`evals/corpus/policies/`) and their scripts (`evals/steps/`) are worked examples: a selector lists the files it judges (`rg --files -g '*.ts'`) and pipes their paths to a script that cuts out the units.
126
126
  - Policies with the same `select` command share one run of it per lint run, so a selector several policies use can live in a script (for example `.lint-anything/lib/functions.sh`) that each calls.
127
127
 
128
- ### Counting items: `limit`
129
-
130
- A policy can count instead of reporting each item: with `limit`, its items are judged as usual, and then the policy reports one finding per file (or, with `per: run`, one for the whole run) that is a `violation` when more than `max` items are violations. Its reason says where the violating items are: their lines within the file, or (with `per: run`) their `path:lines`, else their ids. A group that isn't over the limit is `inconclusive` when its items that weren't judged (too large, or declined by the model) could put it over; the reason counts them by cause, as `N not judged (request too large)` or `N not judged (declined)`. With `per: file`, every item needs a `path`; an item without one stops the run (exit 2).
131
-
132
- ````md
133
- ---
134
- select: |
135
- rg --json -w 'TODO|FIXME' | jq -c '…'
136
- criteria:
137
- "true": "The comment has no tracking issue."
138
- "false": "The comment references a tracking issue."
139
- limit: { max: 3, per: file }
140
- ---
141
- Rule: every `TODO` or `FIXME` comment must reference a tracking issue.
142
- ````
143
-
144
- ```
145
- violation - src/many.ts todos 4 of 4 items violate the rule (at most 3 allowed): 2-2, 9-9, 14-14, 30-30
146
- ```
147
-
148
- Write the rule about one item, and put the count in `limit`: Clef judges items well and counts badly (see Evals). Items in `review` are mentioned in the reason but not counted.
149
-
150
128
  ## How it works
151
129
 
152
130
  Each policy's selector runs from the current directory, then the decision model judges the selected items. Up to 8 policies run at once, and at most 8 requests are in flight across the run. A request's text is the policy body; it carries one yes/no question per item, the item's text, with the policy's criteria. Each item gets its own finding. A policy whose selector emits nothing sends nothing. Findings are printed in policy order.
@@ -171,13 +149,12 @@ Clef's answer to one question shifts a little with the other questions in its re
171
149
 
172
150
  ## Output
173
151
 
174
- The CLI prints one line per non-passing finding — verdict, violation probability, location, policy, and reason — then a count per verdict. The location is the item's `path:start-end`, or its `id` when it has no path and lines. A decision model answers only with a probability, so an item's reason is the policy's `criteria.true`: what the policy says a violation is. A `limit` finding gives its file's path, and its count and the violating items' places as the reason:
152
+ The CLI prints one line per non-passing finding — verdict, violation probability, location, policy, and reason — then a count per verdict. The location is the item's `path:start-end`, or its `id` when it has no path and lines. A decision model answers only with a probability, so an item's reason is the policy's `criteria.true`: what the policy says a violation is.
175
153
 
176
154
  ```
177
- violation - src/a.ts todo-limit 2 of 3 items violate the rule (at most 1 allowed): 1-1, 3-3
178
155
  violation 0.97 src/a.ts:4-4 todo The comment has no tracking issue.
179
156
  violation 0.97 9acebc18 commits The commit message doesn't say why.
180
- 3 checks: 3 violation, 0 review, 0 inconclusive, 0 pass
157
+ 2 checks: 2 violation, 0 review, 0 inconclusive, 0 pass
181
158
  ```
182
159
 
183
160
  Only `violation` fails the run; `review` and `inconclusive` are reported but exit `0`.
@@ -222,7 +199,7 @@ The package also exports the stages a run composes as an [Effect](https://effect
222
199
  | --- | --- |
223
200
  | `select(policy, cwd?, runs?)` | Runs a policy's selector; returns its items, sorted as they are judged |
224
201
  | `judge(policy, items)` | Each item's violation probability, or why there is none: `RequestTooLarge` or `Declined` (needs `DecisionModel`) |
225
- | `decide(policy, items, answers, threshold)` | Findings from those answers, `limit` included; pure |
202
+ | `decide(policy, items, answers, threshold)` | One finding per item from those answers; pure |
226
203
  | `lint(policies, threshold)` | All three for every policy, as the CLI runs them |
227
204
  | `DecisionModel.ask(source, state, questions)` | Yes/no questions under a shared state, packed, cached, retried; no policy needed |
228
205
  | `DecisionModel.layer(cacheDirectory?)` | The `DecisionModel` service for the model the environment variables above configure (`configuredTarget`) |
@@ -265,7 +242,6 @@ bun evals/examples.ts # examples written into the policy bo
265
242
  bun evals/cutoffs.ts # per-policy cut-offs tuned on dev vs. global (stored scores)
266
243
  bun evals/cache.ts # the answer cache: requests on re-runs and edits (real requests)
267
244
  bun evals/packing.ts # repeatability of fresh runs (real requests) and batch independence
268
- bun evals/counts.ts # limit vs. asking the count rule of the whole file
269
245
  bun evals/mine.ts prometheus/prometheus # candidate real violations from PR review comments (GitHub API, cached)
270
246
  bun evals/sample.ts --per-repo 3 owner/repo … # random real files by language, test vs. source, and size (GitHub API, cached)
271
247
  bun evals/wordings.ts --split test # each rule-wording set vs. the polished policies, each scored against its own gold
@@ -364,7 +340,7 @@ Each feature has an intended purpose, an eval that measures it, and a result. Ch
364
340
  | Examples in the body (few-shot) | Examples improve judgments | `examples.ts` | Calibration only: Brier better on both splits, AUROC unchanged | yes (write them in the body) |
365
341
  | Sorted items | The same files get the same verdicts | `packing.ts` | Holds: two fresh runs went from 40 differing verdicts to 0 | yes |
366
342
  | Answer cache (always on) | Re-runs pay only for changes; stable verdicts | `cache.ts` | Holds: 0 requests unchanged, 4 of 38 after an edit, 0 vs 38 verdict changes in untouched files | yes |
367
- | `limit` | Count rules judged reliably | `counts.ts` | Holds: 100% vs 92% (79% at the boundary) | yes |
343
+ | `limit` | Count rules judged reliably | `counts.ts` | Holds: 100% vs 92% (79% at the boundary) | no: removed as unnecessary (below) |
368
344
  | Per-policy cut-offs | Better verdicts than global cut-offs | `cutoffs.ts` | Fails: surfaced violations fall from 72% to 52% for +0.002 F1 | no |
369
345
  | Policy file sent whole as the rule | Clef reads the rule with its frontmatter | `evals/.cache/research/ident/ablate.ts` (2026-10-22) | Replaced: with the excerpt framing, sending the body without the frontmatter (selector code) raised dev AUROC 0.910 → 0.919; the body alone is now the request's text | no |
370
346
  | `ask` routing step | Drop items a rule doesn't govern, cutting false alarms | (commit `a4caa7c`) | Fails: dev false alarms rose, violations dropped | no |
@@ -461,22 +437,12 @@ Each flag the CLI used to take is now fixed at one value. Whether that value is
461
437
 
462
438
  Either default passes a clean checkout that has violations, and changed-since fails outright outside a git repository (`git diff HEAD` errors, exit 2). Off reports everything the selectors find; a selector scopes what it checks.
463
439
 
464
- #### Removed from the CLI: `check`, baseline, changed-lines scope, SARIF
440
+ #### Removed from the CLI: `check`, baseline, changed-lines scope, SARIF, `limit`
465
441
 
466
- These were built and measured with new flags and subcommands, then removed because the CLI takes no arguments. The code and evals are in the git history.
442
+ These were built and measured, then removed: the flags and subcommands because the CLI takes no arguments, and `limit` because it was unnecessary. The code and evals are in the git history.
467
443
 
468
444
  - `lint-anything check` (commit `47705a7`): judged each policy's examples against the rule without them. It caught a rule about the wrong thing (examples agreeing fell from 90% to 48%) but not an inverted one (87%), and it failed two semantic policies in their own wording.
469
445
  - `--baseline` and `--changed-since` (commit `9ddefd2`): on a generated repository, all five adoption scenarios behaved as expected; a baseline keyed by item text survived line shifts (one keyed by line would have called 7 of 11 accepted findings new).
470
446
  - `--sarif` (commit `cfab4c3`): the log validated against the SARIF 2.1.0 schema and every finding mapped to one result.
471
447
  - `ask` routing step (commit `a4caa7c`): routing the comments policy's items through "is this exempt?" raised dev false alarms from 8.2% to 11.5% and dropped 3 of 36 violating comments.
472
-
473
- #### Counting with `limit`
474
-
475
- Purpose: rules about a count ("at most 3 issue-less TODOs per file") are judged reliably. `evals/counts.ts` generates 64 TypeScript, Python, and Go files with a known number of issue-less TODOs (0 to 7, 24 files at exactly 3 or 4) among TODOs that cite an issue, and judges "more than 3" two ways:
476
-
477
- | Approach | Accuracy | Files with exactly 3 or 4 | AUROC | $ / 1k files |
478
- | --- | --- | --- | --- | --- |
479
- | the count rule, its selector emitting the whole file | 92% (59/64) | 79% (19/24) | 0.983 | $0.095 |
480
- | each TODO judged against "must reference an issue", counted by `limit: { max: 3 }` | **100%** (64/64) | **100%** (24/24) | 1.000 | $0.163 |
481
-
482
- The purpose holds: every file's count was exact, while the whole-file judgment ranked files well but misplaced the boundary in a fifth of the files at it. It costs about 1.7× as much.
448
+ - `limit` (commit `d4a6620`, `evals/counts.ts`): judged each item, then reported one finding per file or per run, a violation when more than `max` items violated the rule. On 64 generated files with 0 to 7 issue-less TODOs, counting judged TODOs against "at most 3" was exact on every file, where asking the count rule of the whole file got 92% (79% of the 24 files at 3 or 4), at about 1.7× the cost.