@andrueandersoncs/lint-anything 0.3.0 → 0.5.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1,6 +1,6 @@
1
1
  # lint-anything
2
2
 
3
- Check anything against plain-English policies, judged by Cloudflare's [Clef](https://developers.cloudflare.com/workers-ai/models/clef/) decision model on Workers AI. A policy's selector decides what is checked: parts of files, facts derived from them, a repository's history, a third-party service.
3
+ Check anything against plain-English policies, judged by a decision model: Cloudflare's [Clef](https://developers.cloudflare.com/workers-ai/models/clef/) on Workers AI by default, or any model served behind a System One or OpenAI Decisions endpoint. A policy's selector decides what is checked: parts of files, facts derived from them, a repository's history, a third-party service.
4
4
 
5
5
  ## Run
6
6
 
@@ -15,25 +15,57 @@ bunx lint-anything # from the project root
15
15
 
16
16
  The command is `lint-anything`. It runs on Bun, not Node: without Bun on the `PATH` it fails with `env: bun: No such file or directory`.
17
17
 
18
- The CLI takes no arguments, and one flag, `--debug`: each run checks every policy. Policies are read from `.lint-anything/rules` under the current directory, and each policy's selector finds what it checks. Items are judged by the `clef` model and printed as text. Clef's answers are cached in `.lint-anything/cache/answers.sqlite` (add `.lint-anything/cache` to `.gitignore`). Effect CLI also provides its global flags (`--help`, `--version`, `--wizard`, `--completions`, `--log-level`); they are expected and stay.
18
+ The CLI takes no arguments, and one flag, `--debug`: each run checks every policy. Policies are read from `.lint-anything/rules` under the current directory, and each policy's selector finds what it checks. Items are judged by a decision model (`clef` unless configured otherwise, below) and printed as text. Answers are cached in `.lint-anything/cache/answers.sqlite` (add `.lint-anything/cache` to `.gitignore`). Effect CLI also provides its global flags (`--help`, `--version`, `--wizard`, `--completions`, `--log-level`); they are expected and stay.
19
19
 
20
20
  | Environment variable | Required | Meaning |
21
21
  | --- | --- | --- |
22
- | `CLOUDFLARE_ACCOUNT_ID` | yes | Workers AI account |
23
- | `CLOUDFLARE_AUTH_TOKEN` | yes | API token |
22
+ | `CLOUDFLARE_ACCOUNT_ID` | for Workers AI | Workers AI account |
23
+ | `CLOUDFLARE_AUTH_TOKEN` | for Workers AI | API token |
24
24
  | `CLOUDFLARE_API_BASE_URL` | no | Defaults to `https://api.cloudflare.com/client/v4` |
25
+ | `LINT_ANYTHING_ENDPOINT` | no | Another decision endpoint, as a full URL: the model is asked there instead of on Workers AI, and the `CLOUDFLARE_` variables aren't needed |
26
+ | `LINT_ANYTHING_MODEL` | with `LINT_ANYTHING_ENDPOINT` | The model to ask. Without an endpoint, a Cloudflare model on Workers AI: `clef` (the default) or, for example, `clef-flash` |
27
+ | `LINT_ANYTHING_API_KEY` | no | Bearer token for `LINT_ANYTHING_ENDPOINT`, if it needs one |
25
28
 
26
- Bun loads `.env` and `.env.local` from the current directory automatically.
29
+ Bun loads `.env` and `.env.local` from the current directory automatically. An empty variable counts as unset.
30
+
31
+ ### Choosing a decision model
32
+
33
+ A decision model answers yes/no questions with the probability of yes instead of writing text. Clef is one. lint-anything asks whichever model the environment names, and speaks the two decision APIs:
34
+
35
+ - **System One** (`POST …/v1/systemone`), the API Jev introduced: a `state`, then `noul` questions keyed by id, each with `instructions` and `criteria`. Clef on Workers AI speaks it, and so do most hosts and servers of decision models, such as [LiteLLM](https://docs.litellm.ai/docs/decisions)'s proxy (which routes to Jev, Clef, Perplexity, Databricks, Microsoft Foundry, vLLM, and others) and [LLM Gateway](https://llmgateway.io/blog/llm-decision-api).
36
+ - **OpenAI Decisions** (`POST …/v1/decisions`), used when the endpoint's path ends in `/decisions`: an `input`, then `predicate` questions. A predicate has no criteria, so each question's instructions are the item's text followed by `true: <criteria.true>` and `false: <criteria.false>` lines, as LiteLLM does when it sends criteria to OpenAI.
37
+
38
+ ```sh
39
+ # Clef-flash on Workers AI, with the CLOUDFLARE_ credentials above
40
+ LINT_ANYTHING_MODEL=clef-flash
41
+
42
+ # any System One server, such as a LiteLLM proxy
43
+ LINT_ANYTHING_ENDPOINT=http://localhost:4000/v1/systemone
44
+ LINT_ANYTHING_MODEL=typesafe/jev-latest
45
+ LINT_ANYTHING_API_KEY=sk-...
46
+
47
+ # OpenAI's Decisions API
48
+ LINT_ANYTHING_ENDPOINT=https://api.openai.com/v1/decisions
49
+ LINT_ANYTHING_MODEL=gpt-6-luna-decisions
50
+ LINT_ANYTHING_API_KEY=sk-...
51
+ ```
52
+
53
+ - Every model is asked the same way: the same request text, at most 64 questions and 64,000 bytes per request, the same 0.5 / 0.65 cut-offs. A model whose context window is under 64K tokens may truncate a full request. Serve a local model with a context of at least that size.
54
+ - Answers are cached per model, and per endpoint, so switching models never reuses another model's answers. Switching back finds the earlier model's answers still cached.
55
+ - A question the model refuses (an OpenAI Decisions `refusal`) is `inconclusive`, with the reason `<model> declined to answer`. It isn't cached, so it is asked again next run.
56
+ - `--debug` prices tokens only for Clef. For other models, the cost column shows `-`.
57
+ - Errors name the model: `error: typesafe/jev-latest responded with HTTP 401: …`. A configuration that can't work stops the run (exit 2) before anything is sent: an endpoint without `LINT_ANYTHING_MODEL`, an endpoint that isn't an http(s) URL, `LINT_ANYTHING_API_KEY` without an endpoint, or a model name like `typesafe/jev-latest` without an endpoint (Workers AI only serves Cloudflare models by name).
58
+ - Every accuracy number in this README was measured on Clef. Clef-flash scored much worse on the same corpus (Evals, "Defaults of the removed flags"), so check another model on your own labels (`score.js`, below) or on the corpus (`bun evals/run.ts` with the same variables set) before trusting its verdicts.
27
59
 
28
60
  ## Policies
29
61
 
30
- There are no built-in policies, and the CLI adds nothing to what a policy asks. A policy is a Markdown file anywhere under the rules directory (subdirectories included) that says everything Clef is asked:
62
+ There are no built-in policies, and the CLI adds nothing to what a policy asks. A policy is a Markdown file anywhere under the rules directory (subdirectories included) that says everything the decision model is asked:
31
63
 
32
- - the **body** (after the frontmatter) is the text of every request, sent exactly as written: the rule, and any framing or examples that help Clef judge it;
64
+ - the **body** (after the frontmatter) is the text of every request, sent exactly as written: the rule, and any framing or examples that help the model judge it;
33
65
  - the **`select`** command decides what is judged: which files, which parts of them, and what context goes with each part. Each item it emits is one yes/no question;
34
66
  - the **`criteria`** say what a yes and a no mean: `true` describes an item that violates the rule, `false` one that follows it.
35
67
 
36
- The CLI reads Clef's yes as "this item violates the rule", so a policy must have both criteria. Without them, Clef may read the question as whether the item *follows* the rule, which inverts every verdict. With only `Rule: every TODO must reference a tracking issue like #123.` as the body and no criteria, `// TODO(#12): tracked` scored 0.94 (`violation`) and `// TODO: fix this later` scored 0.02 (`pass`). With criteria added, they scored 0.02 and 0.94. A body that says what to decide ("Decide whether it breaks the rule below") helps too.
68
+ The CLI reads the model's yes as "this item violates the rule", so a policy must have both criteria. Without them, the model may read the question as whether the item *follows* the rule, which inverts every verdict. On Clef: With only `Rule: every TODO must reference a tracking issue like #123.` as the body and no criteria, `// TODO(#12): tracked` scored 0.94 (`violation`) and `// TODO: fix this later` scored 0.02 (`pass`). With criteria added, they scored 0.02 and 0.94. A body that says what to decide ("Decide whether it breaks the rule below") helps too.
37
69
 
38
70
  ````md
39
71
  ---
@@ -50,13 +82,13 @@ Each question shows one comment. Decide whether it breaks the rule below.
50
82
  Rule: every `TODO` or `FIXME` comment must reference a tracking issue, written as `#123`, `ABC-123`, or a full issue URL.
51
83
  ````
52
84
 
53
- - Frontmatter holds `select` and `criteria` (`true` and `false`), all required, and `limit` (below). Any other key is an error.
85
+ - Frontmatter holds `select` and `criteria` (`true` and `false`), both required. Any other key is an error.
54
86
  - The body must not be empty.
55
87
  - The policy name is its path relative to the rules directory, without `.md` (for example `sub/todo-has-issue`).
56
88
 
57
89
  A missing rules directory, one with no `.md` files, or an invalid policy stops the run.
58
90
 
59
- Policies are meant to be written and tuned by a coding agent: give it the rule you want and let it iterate on the selector, the body, and the criteria together, measured against labeled code (below). Because the CLI adds nothing, any way of asking Clef that a request can express is available to a policy. `evals/corpus/selectors-excerpts/` is the earlier CLI's whole-file judging written as policies: its selector cuts files into 1,000-byte excerpts, and its body and criteria are that CLI's own words.
91
+ Policies are meant to be written and tuned by a coding agent: give it the rule you want and let it iterate on the selector, the body, and the criteria together, measured against labeled code (below). Because the CLI adds nothing, any way of asking the model that a request can express is available to a policy. `evals/corpus/selectors-excerpts/` is the earlier CLI's whole-file judging written as policies: its selector cuts files into 1,000-byte excerpts, and its body and criteria are that CLI's own words.
60
92
 
61
93
  `skills/lint-anything/SKILL.md` is an agent skill covering running the CLI, acting on findings, writing and tuning policies, and the pitfalls selectors run into. It ships in the package (`node_modules/@andrueandersoncs/lint-anything/skills/lint-anything/`) with its score script; copy that directory into a coding agent's skills directory.
62
94
 
@@ -73,13 +105,13 @@ An item counts as a violation when its lines overlap a labeled violation, and a
73
105
 
74
106
  ### Selectors
75
107
 
76
- - The selector runs once per lint run with `sh -c`, from the current directory, with no arguments, an empty stdin, and the CLI's environment (`.env` included) without any variable whose name starts with `CLOUDFLARE_`, so a policy's shell never sees the API token. It finds what it checks itself: the example above searches the `.ts` and `.tsx` files under the current directory (`rg` skips what `.gitignore` ignores). A selector can select things that aren't files (`git log`, an API, a database), or facts derived from them.
108
+ - The selector runs once per lint run with `sh -c`, from the current directory, with no arguments, an empty stdin, and the CLI's environment (`.env` included) without any variable whose name starts with `CLOUDFLARE_` or `LINT_ANYTHING_`, so a policy's shell never sees the model's API token or key. It finds what it checks itself: the example above searches the `.ts` and `.tsx` files under the current directory (`rg` skips what `.gitignore` ignores). A selector can select things that aren't files (`git log`, an API, a database), or facts derived from them.
77
109
  - It must print JSONL, one item per line, with these fields and no others:
78
110
 
79
111
  | Field | Required | Meaning |
80
112
  | --- | --- | --- |
81
113
  | `id` | yes | Names the item in findings |
82
- | `text` | yes | Everything Clef reads for this item: the code or text to judge and any context, laid out as the selector likes |
114
+ | `text` | yes | Everything the model reads for this item: the code or text to judge and any context, laid out as the selector likes |
83
115
  | `path` | no | The item's file, for findings |
84
116
  | `lines` | no | `[start, end]`, 1-based and inclusive, for findings |
85
117
 
@@ -93,44 +125,23 @@ An item counts as a violation when its lines overlap a labeled violation, and a
93
125
  - Structural units (functions, classes, handlers, tests) are easiest to select with a syntax-aware tool such as [ast-grep](https://ast-grep.github.io/). The eval corpus's policies (`evals/corpus/policies/`) and their scripts (`evals/steps/`) are worked examples: a selector lists the files it judges (`rg --files -g '*.ts'`) and pipes their paths to a script that cuts out the units.
94
126
  - Policies with the same `select` command share one run of it per lint run, so a selector several policies use can live in a script (for example `.lint-anything/lib/functions.sh`) that each calls.
95
127
 
96
- ### Counting items: `limit`
97
-
98
- A policy can count instead of reporting each item: with `limit`, its items are judged as usual, and then the policy reports one finding per file (or, with `per: run`, one for the whole run) that is a `violation` when more than `max` items are violations. Its reason says where the violating items are: their lines within the file, or (with `per: run`) their `path:lines`, else their ids. A group that isn't over the limit is `inconclusive` when its items too large to judge could put it over. With `per: file`, every item needs a `path`; an item without one stops the run (exit 2).
99
-
100
- ````md
101
- ---
102
- select: |
103
- rg --json -w 'TODO|FIXME' | jq -c '…'
104
- criteria:
105
- "true": "The comment has no tracking issue."
106
- "false": "The comment references a tracking issue."
107
- limit: { max: 3, per: file }
108
- ---
109
- Rule: every `TODO` or `FIXME` comment must reference a tracking issue.
110
- ````
111
-
112
- ```
113
- violation - src/many.ts todos 4 of 4 items violate the rule (at most 3 allowed): 2-2, 9-9, 14-14, 30-30
114
- ```
115
-
116
- Write the rule about one item, and put the count in `limit`: Clef judges items well and counts badly (see Evals). Items in `review` are mentioned in the reason but not counted.
117
-
118
128
  ## How it works
119
129
 
120
- Each policy's selector runs from the current directory, then Clef judges the selected items. Up to 8 policies run at once, and at most 8 requests are in flight across the run. A request's text is the policy body; it carries one yes/no question per item, the item's text, with the policy's criteria. Each item gets its own finding. A policy whose selector emits nothing sends nothing. Findings are printed in policy order.
130
+ Each policy's selector runs from the current directory, then the decision model judges the selected items. Up to 8 policies run at once, and at most 8 requests are in flight across the run. A request's text is the policy body; it carries one yes/no question per item, the item's text, with the policy's criteria. Each item gets its own finding. A policy whose selector emits nothing sends nothing. Findings are printed in policy order.
121
131
 
122
- A request holds at most 64 questions (Clef's limit) and 64,000 bytes. Items that don't fit in one request are split across several.
132
+ A request holds at most 64 questions (Clef's limit) and 64,000 bytes, whichever model is asked. Items that don't fit in one request are split across several. Each request is POSTed to the model's endpoint with its API token or key as a bearer token: on Workers AI, `https://api.cloudflare.com/client/v4/accounts/<account>/ai/run/@cf/cloudflare/<model>`; otherwise `LINT_ANYTHING_ENDPOINT` as given.
123
133
 
124
134
  | Result | Verdict |
125
135
  | --- | --- |
126
136
  | `violates ≤ 0.5` | `pass` |
127
137
  | `violates ≥ 0.65` | `violation` |
128
138
  | otherwise | `review`: uncertain, including cases where careful reviewers would disagree |
129
- | request with only that item over the byte limit | `inconclusive` (not sent; Clef would truncate it) |
139
+ | request with only that item over the byte limit | `inconclusive` (not sent): `request is N bytes; Clef would truncate it`, or for another model `request is N bytes, over the 64000-byte request limit` |
140
+ | the model refused the question (OpenAI Decisions) | `inconclusive`: `<model> declined to answer` |
130
141
 
131
- An attempt that hasn't returned its whole response within 60 seconds counts as a transient failure. Transient HTTP failures (timeouts, network errors, 408, 429, 5xx) are retried up to 3 times with exponential backoff, waiting at least as long as a `Retry-After` header asks. A request that still fails stops the run (exit 2) with the HTTP status and Cloudflare's error, or the network error.
142
+ An attempt that hasn't returned its whole response within 60 seconds counts as a transient failure. Transient HTTP failures (timeouts, network errors, 408, 429, 5xx) are retried up to 3 times with exponential backoff, waiting at least as long as a `Retry-After` header asks. A request that still fails stops the run (exit 2) with the HTTP status and the endpoint's error, or the network error.
132
143
 
133
- Answers are cached on disk in one SQLite database, `.lint-anything/cache/answers.sqlite`, one row per question, keyed by the model, the request's text (the policy body), and the question (item text and criteria). Only questions whose inputs changed are sent: re-running when nothing changed sends nothing, and editing a policy re-asks only that policy. Each request's answers are written in one transaction as it returns, so a run stopped by a failed request keeps the answers it already received, and a crash can't leave a partial answer. Runs can share the cache at once (WAL mode; a writer waits up to 10 seconds for another's lock). A cache that can't be opened (for example, a file that isn't a SQLite database) stops the run (exit 2). Delete the directory to judge everything afresh.
144
+ Answers are cached on disk in one SQLite database, `.lint-anything/cache/answers.sqlite`, one row per question, keyed by the model (for a model at `LINT_ANYTHING_ENDPOINT`, its URL and name), the request's text (the policy body), and the question (item text and criteria). Only questions whose inputs changed are sent: re-running when nothing changed sends nothing, and editing a policy re-asks only that policy. Each request's answers are written in one transaction as it returns, so a run stopped by a failed request keeps the answers it already received, and a crash can't leave a partial answer. Runs can share the cache at once (WAL mode; a writer waits up to 10 seconds for another's lock). A cache that can't be opened (for example, a file that isn't a SQLite database) stops the run (exit 2). Delete the directory to judge everything afresh.
134
145
 
135
146
  A cached answer stands for 30 days; each run starts by deleting older answers, so their questions are asked again, answers follow updates to the model, and the cache doesn't grow without bound. A row can only hold a probability between 0 and 1. With 50,000 cached answers the database is 5.5 MB, and a run that takes every answer from it reads them in under half a second.
136
147
 
@@ -138,18 +149,17 @@ Clef's answer to one question shifts a little with the other questions in its re
138
149
 
139
150
  ## Output
140
151
 
141
- The CLI prints one line per non-passing finding — verdict, violation probability, location, policy, and reason — then a count per verdict. The location is the item's `path:start-end`, or its `id` when it has no path and lines. Clef answers only with a probability, so an item's reason is the policy's `criteria.true`: what the policy says a violation is. A `limit` finding gives its file's path, and its count and the violating items' places as the reason:
152
+ The CLI prints one line per non-passing finding — verdict, violation probability, location, policy, and reason — then a count per verdict. The location is the item's `path:start-end`, or its `id` when it has no path and lines. A decision model answers only with a probability, so an item's reason is the policy's `criteria.true`: what the policy says a violation is.
142
153
 
143
154
  ```
144
- violation - src/a.ts todo-limit 2 of 3 items violate the rule (at most 1 allowed): 1-1, 3-3
145
155
  violation 0.97 src/a.ts:4-4 todo The comment has no tracking issue.
146
156
  violation 0.97 9acebc18 commits The commit message doesn't say why.
147
- 3 checks: 3 violation, 0 review, 0 inconclusive, 0 pass
157
+ 2 checks: 2 violation, 0 review, 0 inconclusive, 0 pass
148
158
  ```
149
159
 
150
160
  Only `violation` fails the run; `review` and `inconclusive` are reported but exit `0`.
151
161
 
152
- With `--debug`, the CLI also prints, to stderr after the findings (or before the error, if the run stops), what each policy cost: requests sent, answers taken from the cache, input tokens Clef reported, and their cost at Clef's input-token price, then the total:
162
+ With `--debug`, the CLI also prints, to stderr after the findings (or before the error, if the run stops), what each policy cost: requests sent, answers taken from the cache, input tokens the model reported, and their cost at Clef's input-token price, then the total. For any other model the cost column is `-`, since lint-anything doesn't know its price; a line under the table counts requests whose response reported no token count.
153
163
 
154
164
  ```
155
165
  policy requests cached input tokens cost (USD)
@@ -162,7 +172,7 @@ total 1 3 379 $0.0001
162
172
  | --- | --- |
163
173
  | `0` | No violations |
164
174
  | `1` | At least one violation |
165
- | `2` | Invalid command line; policy, selector, configuration, or Clef API error |
175
+ | `2` | Invalid command line; policy, selector, configuration, or decision model API error |
166
176
  | `130` | Interrupted (SIGINT or SIGTERM): running selectors are killed and answers already received stay cached |
167
177
  | `129` | Killed by SIGHUP (terminal closed): running selectors are killed first |
168
178
 
@@ -188,11 +198,15 @@ The package also exports the stages a run composes as an [Effect](https://effect
188
198
  | Export | What it does |
189
199
  | --- | --- |
190
200
  | `select(policy, cwd?, runs?)` | Runs a policy's selector; returns its items, sorted as they are judged |
191
- | `judge(policy, items)` | Each item's violation probability, or `RequestTooLarge` (needs `Clef`) |
192
- | `decide(policy, items, answers, threshold)` | Findings from those answers, `limit` included; pure |
201
+ | `judge(policy, items)` | Each item's violation probability, or why there is none: `RequestTooLarge` or `Declined` (needs `DecisionModel`) |
202
+ | `decide(policy, items, answers, threshold)` | One finding per item from those answers; pure |
193
203
  | `lint(policies, threshold)` | All three for every policy, as the CLI runs them |
194
- | `Clef.ask(source, state, questions)` | Yes/no questions under a shared state, packed, cached, retried; no policy needed |
195
- | `Clef.layer(cacheDirectory?)` | The `Clef` service; reads the environment variables above |
204
+ | `DecisionModel.ask(source, state, questions)` | Yes/no questions under a shared state, packed, cached, retried; no policy needed |
205
+ | `DecisionModel.layer(cacheDirectory?)` | The `DecisionModel` service for the model the environment variables above configure (`configuredTarget`) |
206
+ | `DecisionModel.layerFor(target, cacheDirectory?)` | The same for a model given in code: `workersAiTarget({ accountId, token, model? })` or `endpointTarget({ url, model, token? })` |
207
+ | `planRequests(state, questions, model?)` | The exact request bodies a model would be sent (default: Clef's) |
208
+
209
+ `Clef` and `ClefError` remain as deprecated names for `DecisionModel` and `DecisionModelError`.
196
210
  | `loadPolicies(directory)`, `parsePolicy(name, source)` | Policies from Markdown (`loadPolicies` needs `FileSystem` and `Path`, e.g. `BunServices.layer` from `@effect/platform-bun`) |
197
211
  | `loadLabels(policy)`, `scorePolicy(policy, labels, cwd?)` | A policy scored against `.lint-anything/labels/<policy>.jsonl`: per labeled file and item, with reach |
198
212
  | `quality`, `auroc`, `pairedInterval`, `bootstrapInterval`, … | The metrics and 95% bootstrap intervals the evals and the score script report |
@@ -200,20 +214,20 @@ The package also exports the stages a run composes as an [Effect](https://effect
200
214
  ```ts
201
215
  import { BunServices } from "@effect/platform-bun"
202
216
  import { Effect, Layer } from "effect"
203
- import { Clef, decide, defaultThreshold, judge, loadPolicies, select } from "@andrueandersoncs/lint-anything"
217
+ import { DecisionModel, decide, defaultThreshold, judge, loadPolicies, select } from "@andrueandersoncs/lint-anything"
204
218
 
205
219
  const findings = await Effect.runPromise(Effect.gen(function*() {
206
220
  const [policy] = yield* loadPolicies(".lint-anything/rules")
207
221
  const items = yield* select(policy!)
208
222
  return decide(policy!, items, yield* judge(policy!, items), defaultThreshold)
209
- }).pipe(Effect.provide(Layer.mergeAll(Clef.layer(".lint-anything/cache"), BunServices.layer))))
223
+ }).pipe(Effect.provide(Layer.mergeAll(DecisionModel.layer(".lint-anything/cache"), BunServices.layer))))
210
224
  ```
211
225
 
212
226
  Selectors still run in their own process groups and are killed when the process exits. The library installs no signal handlers: a host that wants selectors killed on SIGHUP calls `killRunningSelectors()` from its own handler, as the CLI does.
213
227
 
214
228
  ## Evals
215
229
 
216
- `evals/` scores policies (body, selector, criteria) and request shapes against a hand-labeled corpus, so changes can be measured instead of guessed. Its item labeling, file scores, metrics, and intervals are `src/score.ts`, the same code the skill's score script reports with.
230
+ `evals/` scores policies (body, selector, criteria) and request shapes against a hand-labeled corpus, so changes can be measured instead of guessed. Its item labeling, file scores, metrics, and intervals are `src/score.ts`, the same code the skill's score script reports with. The evals ask the decision model the CLI would, configured by the same environment variables: Clef by default, so `LINT_ANYTHING_ENDPOINT=… LINT_ANYTHING_MODEL=… bun evals/run.ts` scores another model on the same cases. Its per-case scores go to their own `….model-<name>` results directory.
217
231
 
218
232
  ```sh
219
233
  bun evals/validate.ts # check the corpus, print label counts per policy and origin
@@ -223,12 +237,11 @@ bun evals/run.ts --split test --origin real --category semantic
223
237
  bun evals/run.ts production single # request shapes for the same items
224
238
  bun evals/run.ts --selectors evals/corpus/selectors-excerpts # another set of policies vs. the corpus policies
225
239
  bun evals/history.ts # context from outside the file: a dated git fixture
226
- bun evals/shared.ts # a shared selector runs once (stub Clef, no requests)
240
+ bun evals/shared.ts # a shared selector runs once (stub decision model, no requests)
227
241
  bun evals/examples.ts # examples written into the policy body, as few-shot
228
242
  bun evals/cutoffs.ts # per-policy cut-offs tuned on dev vs. global (stored scores)
229
243
  bun evals/cache.ts # the answer cache: requests on re-runs and edits (real requests)
230
244
  bun evals/packing.ts # repeatability of fresh runs (real requests) and batch independence
231
- bun evals/counts.ts # limit vs. asking the count rule of the whole file
232
245
  bun evals/mine.ts prometheus/prometheus # candidate real violations from PR review comments (GitHub API, cached)
233
246
  bun evals/sample.ts --per-repo 3 owner/repo … # random real files by language, test vs. source, and size (GitHub API, cached)
234
247
  bun evals/wordings.ts --split test # each rule-wording set vs. the polished policies, each scored against its own gold
@@ -270,7 +283,7 @@ bun evals/wordings.ts --split test # each rule-wording set vs. the polis
270
283
  - contrast-pair ordering, and cost per 1,000 files.
271
284
 
272
285
  Verdicts use `--thresholds floor,threshold`, which defaults to the CLI's. The table also shows the cut-offs that would maximize F1 on the scored cases; pick them on dev and pass them to test. The first variant is the baseline, and intervals are 95% paired bootstraps that resample files, because cases in one file share requests.
273
- - **Cache**: replies are cached in `evals/.cache` by model and exact request body, since Clef is deterministic. Reruns and re-scoring are free; only new requests are sent. Per-case scores go to `evals/results/<split>[.<origins>][.<category>][.wording-<set>]/<variant>.jsonl`.
286
+ - **Cache**: replies are cached in `evals/.cache` by model (Clef's under its name, as before other models) and exact request body, since a decision model is deterministic for a given body. Reruns and re-scoring are free and need no credentials; only new requests are sent. Per-case scores go to `evals/results/<split>[.<origins>][.<category>][.wording-<set>][.model-<name>]/<variant>.jsonl`, the model part only for a model other than Clef.
274
287
 
275
288
  Findings (2026-10-22, re-run 2026-10-23 with every answer from the cache; `run.ts --origin real --category semantic`, cut-offs 0.5 / 0.65):
276
289
 
@@ -327,7 +340,7 @@ Each feature has an intended purpose, an eval that measures it, and a result. Ch
327
340
  | Examples in the body (few-shot) | Examples improve judgments | `examples.ts` | Calibration only: Brier better on both splits, AUROC unchanged | yes (write them in the body) |
328
341
  | Sorted items | The same files get the same verdicts | `packing.ts` | Holds: two fresh runs went from 40 differing verdicts to 0 | yes |
329
342
  | Answer cache (always on) | Re-runs pay only for changes; stable verdicts | `cache.ts` | Holds: 0 requests unchanged, 4 of 38 after an edit, 0 vs 38 verdict changes in untouched files | yes |
330
- | `limit` | Count rules judged reliably | `counts.ts` | Holds: 100% vs 92% (79% at the boundary) | yes |
343
+ | `limit` | Count rules judged reliably | `counts.ts` | Holds: 100% vs 92% (79% at the boundary) | no: removed as unnecessary (below) |
331
344
  | Per-policy cut-offs | Better verdicts than global cut-offs | `cutoffs.ts` | Fails: surfaced violations fall from 72% to 52% for +0.002 F1 | no |
332
345
  | Policy file sent whole as the rule | Clef reads the rule with its frontmatter | `evals/.cache/research/ident/ablate.ts` (2026-10-22) | Replaced: with the excerpt framing, sending the body without the frontmatter (selector code) raised dev AUROC 0.910 → 0.919; the body alone is now the request's text | no |
333
346
  | `ask` routing step | Drop items a rule doesn't govern, cutting false alarms | (commit `a4caa7c`) | Fails: dev false alarms rose, violations dropped | no |
@@ -349,7 +362,7 @@ Purpose: context from outside the file makes rules about time or history judgeab
349
362
 
350
363
  #### Shared selector runs
351
364
 
352
- Purpose: policies that share a selector don't repeat the work. `evals/shared.ts` lints this repository's `src/*.ts` with six policies that share one selector, against a stub Clef that sends no requests, and counts executions through a log the selector appends to: it runs once instead of six times, and the findings (516, 86 per policy) are the same as without sharing. Sharing is scoped to one lint run.
365
+ Purpose: policies that share a selector don't repeat the work. `evals/shared.ts` lints this repository's `src/*.ts` with six policies that share one selector, against a stub decision model that sends no requests, and counts executions through a log the selector appends to: it runs once instead of six times, and the findings (516, 86 per policy) are the same as without sharing. Sharing is scoped to one lint run.
353
366
 
354
367
  #### Examples in the body
355
368
 
@@ -405,7 +418,7 @@ Each flag the CLI used to take is now fixed at one value. Whether that value is
405
418
  | Flag | Fixed at | Evidence | Best default? |
406
419
  | --- | --- | --- | --- |
407
420
  | `--threshold` | floor 0.5, threshold 0.65 | `run.ts`, corpus policies (2026-10-22): the dev F1 maximum is 0.575 / 0.725 (all cases). On all test cases it gives F1 0.591 vs. 0.580, but catches 63% of violations instead of 73% and surfaces 80% instead of 86% (false alarms 8% vs. 12%) | yes: F1 is flat across both, and 0.5 / 0.65 finds more violations |
408
- | `--model` | `clef` | `run.ts` with the earlier request shape (cached answers): clef-flash AUROC 0.879 → 0.764 [−0.150, −0.081] on dev, 0.869 → 0.771 [−0.146, −0.051] on test; false alarms 4% → 32% (dev) | yes; the CLI and evals now support only `clef`, and the `clef-flash` code is removed |
421
+ | `--model` | `clef` | `run.ts` with the earlier request shape (cached answers): clef-flash AUROC 0.879 → 0.764 [−0.150, −0.081] on dev, 0.869 → 0.771 [−0.146, −0.051] on test; false alarms 4% → 32% (dev) | yes: Clef stays the default. The flag stays removed; `LINT_ANYTHING_MODEL` and `LINT_ANYTHING_ENDPOINT` now pick another model (Run, "Choosing a decision model"), and the evals score whichever model they name |
409
422
  | `--no-cache` | cache on | `cache.ts` above | yes (on) |
410
423
  | `--changed-since` | off | Throwaway script on `adoption.ts`'s repository and stub (below) | yes (off) |
411
424
  | `--baseline` / `--write-baseline` | off | Same script | yes (off) |
@@ -424,22 +437,12 @@ Each flag the CLI used to take is now fixed at one value. Whether that value is
424
437
 
425
438
  Either default passes a clean checkout that has violations, and changed-since fails outright outside a git repository (`git diff HEAD` errors, exit 2). Off reports everything the selectors find; a selector scopes what it checks.
426
439
 
427
- #### Removed from the CLI: `check`, baseline, changed-lines scope, SARIF
440
+ #### Removed from the CLI: `check`, baseline, changed-lines scope, SARIF, `limit`
428
441
 
429
- These were built and measured with new flags and subcommands, then removed because the CLI takes no arguments. The code and evals are in the git history.
442
+ These were built and measured, then removed: the flags and subcommands because the CLI takes no arguments, and `limit` because it was unnecessary. The code and evals are in the git history.
430
443
 
431
444
  - `lint-anything check` (commit `47705a7`): judged each policy's examples against the rule without them. It caught a rule about the wrong thing (examples agreeing fell from 90% to 48%) but not an inverted one (87%), and it failed two semantic policies in their own wording.
432
445
  - `--baseline` and `--changed-since` (commit `9ddefd2`): on a generated repository, all five adoption scenarios behaved as expected; a baseline keyed by item text survived line shifts (one keyed by line would have called 7 of 11 accepted findings new).
433
446
  - `--sarif` (commit `cfab4c3`): the log validated against the SARIF 2.1.0 schema and every finding mapped to one result.
434
447
  - `ask` routing step (commit `a4caa7c`): routing the comments policy's items through "is this exempt?" raised dev false alarms from 8.2% to 11.5% and dropped 3 of 36 violating comments.
435
-
436
- #### Counting with `limit`
437
-
438
- Purpose: rules about a count ("at most 3 issue-less TODOs per file") are judged reliably. `evals/counts.ts` generates 64 TypeScript, Python, and Go files with a known number of issue-less TODOs (0 to 7, 24 files at exactly 3 or 4) among TODOs that cite an issue, and judges "more than 3" two ways:
439
-
440
- | Approach | Accuracy | Files with exactly 3 or 4 | AUROC | $ / 1k files |
441
- | --- | --- | --- | --- | --- |
442
- | the count rule, its selector emitting the whole file | 92% (59/64) | 79% (19/24) | 0.983 | $0.095 |
443
- | each TODO judged against "must reference an issue", counted by `limit: { max: 3 }` | **100%** (64/64) | **100%** (24/24) | 1.000 | $0.163 |
444
-
445
- The purpose holds: every file's count was exact, while the whole-file judgment ranked files well but misplaced the boundary in a fifth of the files at it. It costs about 1.7× as much.
448
+ - `limit` (commit `d4a6620`, `evals/counts.ts`): judged each item, then reported one finding per file or per run, a violation when more than `max` items violated the rule. On 64 generated files with 0 to 7 issue-less TODOs, counting judged TODOs against "at most 3" was exact on every file, where asking the count rule of the whole file got 92% (79% of the 24 files at 3 or 4), at about 1.7× the cost.