@andrueandersoncs/lint-anything 0.4.0 → 0.5.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1,6 +1,6 @@
1
1
  # lint-anything
2
2
 
3
- Check anything against plain-English policies, judged by a decision model: Cloudflare's [Clef](https://developers.cloudflare.com/workers-ai/models/clef/) on Workers AI by default, or any model served behind a System One or OpenAI Decisions endpoint. A policy's selector decides what is checked: parts of files, facts derived from them, a repository's history, a third-party service.
3
+ Check anything against plain-English policies, judged by the decision model of your choice: any model served behind a System One or OpenAI Decisions endpoint, or, with no endpoint set, a Cloudflare model on Workers AI ([Clef](https://developers.cloudflare.com/workers-ai/models/clef/) unless you name another). A policy's selector decides what is checked: parts of files, facts derived from them, a repository's history, a third-party service.
4
4
 
5
5
  ## Run
6
6
 
@@ -8,37 +8,43 @@ Requires [Bun](https://bun.sh) 1.2.23 or later, plus whatever your policies' sel
8
8
 
9
9
  ```sh
10
10
  bun add -d @andrueandersoncs/lint-anything # or: bun add -g @andrueandersoncs/lint-anything
11
+
12
+ # a decision model: any System One or OpenAI Decisions endpoint…
13
+ echo 'LINT_ANYTHING_ENDPOINT=http://localhost:4000/v1/systemone' >> .env
14
+ echo 'LINT_ANYTHING_MODEL=...' >> .env
15
+ echo 'LINT_ANYTHING_API_KEY=...' >> .env # if the endpoint needs one
16
+ # …or, with no endpoint, Workers AI
11
17
  echo 'CLOUDFLARE_ACCOUNT_ID=...' >> .env
12
18
  echo 'CLOUDFLARE_AUTH_TOKEN=...' >> .env
19
+
13
20
  bunx lint-anything # from the project root
14
21
  ```
15
22
 
16
23
  The command is `lint-anything`. It runs on Bun, not Node: without Bun on the `PATH` it fails with `env: bun: No such file or directory`.
17
24
 
18
- The CLI takes no arguments, and one flag, `--debug`: each run checks every policy. Policies are read from `.lint-anything/rules` under the current directory, and each policy's selector finds what it checks. Items are judged by a decision model (`clef` unless configured otherwise, below) and printed as text. Answers are cached in `.lint-anything/cache/answers.sqlite` (add `.lint-anything/cache` to `.gitignore`). Effect CLI also provides its global flags (`--help`, `--version`, `--wizard`, `--completions`, `--log-level`); they are expected and stay.
25
+ The CLI takes no arguments, and one flag, `--debug`: each run checks every policy. Policies are read from `.lint-anything/rules` under the current directory, and each policy's selector finds what it checks. Items are judged by the decision model the environment configures (below) and printed as text. Answers are cached in `.lint-anything/cache/answers.sqlite` (add `.lint-anything/cache` to `.gitignore`). Effect CLI also provides its global flags (`--help`, `--version`, `--wizard`, `--completions`, `--log-level`); they are expected and stay.
19
26
 
20
27
  | Environment variable | Required | Meaning |
21
28
  | --- | --- | --- |
22
- | `CLOUDFLARE_ACCOUNT_ID` | for Workers AI | Workers AI account |
23
- | `CLOUDFLARE_AUTH_TOKEN` | for Workers AI | API token |
24
- | `CLOUDFLARE_API_BASE_URL` | no | Defaults to `https://api.cloudflare.com/client/v4` |
25
- | `LINT_ANYTHING_ENDPOINT` | no | Another decision endpoint, as a full URL: the model is asked there instead of on Workers AI, and the `CLOUDFLARE_` variables aren't needed |
26
- | `LINT_ANYTHING_MODEL` | with `LINT_ANYTHING_ENDPOINT` | The model to ask. Without an endpoint, a Cloudflare model on Workers AI: `clef` (the default) or, for example, `clef-flash` |
29
+ | `LINT_ANYTHING_ENDPOINT` | no | The decision endpoint to ask, as a full URL: any System One or OpenAI Decisions endpoint. Without it, the model is asked on Workers AI |
30
+ | `LINT_ANYTHING_MODEL` | with `LINT_ANYTHING_ENDPOINT` | The model to ask. Without an endpoint, a Cloudflare model on Workers AI: `clef` unless set, or, for example, `clef-flash` |
27
31
  | `LINT_ANYTHING_API_KEY` | no | Bearer token for `LINT_ANYTHING_ENDPOINT`, if it needs one |
32
+ | `CLOUDFLARE_ACCOUNT_ID` | without an endpoint | Workers AI account |
33
+ | `CLOUDFLARE_AUTH_TOKEN` | without an endpoint | Workers AI API token |
34
+ | `CLOUDFLARE_API_BASE_URL` | no | Defaults to `https://api.cloudflare.com/client/v4` |
28
35
 
29
36
  Bun loads `.env` and `.env.local` from the current directory automatically. An empty variable counts as unset.
30
37
 
31
38
  ### Choosing a decision model
32
39
 
33
- A decision model answers yes/no questions with the probability of yes instead of writing text. Clef is one. lint-anything asks whichever model the environment names, and speaks the two decision APIs:
40
+ A decision model answers yes/no questions with the probability of yes instead of writing text. lint-anything isn't tied to one: it asks whichever model the environment names, and speaks the two decision APIs:
34
41
 
35
42
  - **System One** (`POST …/v1/systemone`), the API Jev introduced: a `state`, then `noul` questions keyed by id, each with `instructions` and `criteria`. Clef on Workers AI speaks it, and so do most hosts and servers of decision models, such as [LiteLLM](https://docs.litellm.ai/docs/decisions)'s proxy (which routes to Jev, Clef, Perplexity, Databricks, Microsoft Foundry, vLLM, and others) and [LLM Gateway](https://llmgateway.io/blog/llm-decision-api).
36
43
  - **OpenAI Decisions** (`POST …/v1/decisions`), used when the endpoint's path ends in `/decisions`: an `input`, then `predicate` questions. A predicate has no criteria, so each question's instructions are the item's text followed by `true: <criteria.true>` and `false: <criteria.false>` lines, as LiteLLM does when it sends criteria to OpenAI.
37
44
 
38
- ```sh
39
- # Clef-flash on Workers AI, with the CLOUDFLARE_ credentials above
40
- LINT_ANYTHING_MODEL=clef-flash
45
+ With no endpoint set, the model is a Cloudflare model on Workers AI, asked with the `CLOUDFLARE_` credentials: `clef` unless `LINT_ANYTHING_MODEL` names another.
41
46
 
47
+ ```sh
42
48
  # any System One server, such as a LiteLLM proxy
43
49
  LINT_ANYTHING_ENDPOINT=http://localhost:4000/v1/systemone
44
50
  LINT_ANYTHING_MODEL=typesafe/jev-latest
@@ -48,6 +54,9 @@ LINT_ANYTHING_API_KEY=sk-...
48
54
  LINT_ANYTHING_ENDPOINT=https://api.openai.com/v1/decisions
49
55
  LINT_ANYTHING_MODEL=gpt-6-luna-decisions
50
56
  LINT_ANYTHING_API_KEY=sk-...
57
+
58
+ # Clef-flash on Workers AI, with the CLOUDFLARE_ credentials
59
+ LINT_ANYTHING_MODEL=clef-flash
51
60
  ```
52
61
 
53
62
  - Every model is asked the same way: the same request text, at most 64 questions and 64,000 bytes per request, the same 0.5 / 0.65 cut-offs. A model whose context window is under 64K tokens may truncate a full request. Serve a local model with a context of at least that size.
@@ -55,7 +64,7 @@ LINT_ANYTHING_API_KEY=sk-...
55
64
  - A question the model refuses (an OpenAI Decisions `refusal`) is `inconclusive`, with the reason `<model> declined to answer`. It isn't cached, so it is asked again next run.
56
65
  - `--debug` prices tokens only for Clef. For other models, the cost column shows `-`.
57
66
  - Errors name the model: `error: typesafe/jev-latest responded with HTTP 401: …`. A configuration that can't work stops the run (exit 2) before anything is sent: an endpoint without `LINT_ANYTHING_MODEL`, an endpoint that isn't an http(s) URL, `LINT_ANYTHING_API_KEY` without an endpoint, or a model name like `typesafe/jev-latest` without an endpoint (Workers AI only serves Cloudflare models by name).
58
- - Every accuracy number in this README was measured on Clef. Clef-flash scored much worse on the same corpus (Evals, "Defaults of the removed flags"), so check another model on your own labels (`score.js`, below) or on the corpus (`bun evals/run.ts` with the same variables set) before trusting its verdicts.
67
+ - Every accuracy number in this README was measured on Clef, the model the evals were built with. Clef-flash scored much worse on the same corpus (Evals, "Defaults of the removed flags"), so check another model on your own labels (`score.js`, below) or on the corpus (`bun evals/run.ts` with the same variables set) before trusting its verdicts.
59
68
 
60
69
  ## Policies
61
70
 
@@ -82,7 +91,7 @@ Each question shows one comment. Decide whether it breaks the rule below.
82
91
  Rule: every `TODO` or `FIXME` comment must reference a tracking issue, written as `#123`, `ABC-123`, or a full issue URL.
83
92
  ````
84
93
 
85
- - Frontmatter holds `select` and `criteria` (`true` and `false`), all required, and `limit` (below). Any other key is an error.
94
+ - Frontmatter holds `select` and `criteria` (`true` and `false`), both required. Any other key is an error.
86
95
  - The body must not be empty.
87
96
  - The policy name is its path relative to the rules directory, without `.md` (for example `sub/todo-has-issue`).
88
97
 
@@ -119,39 +128,17 @@ An item counts as a violation when its lines overlap a labeled violation, and a
119
128
  - A selector still running after 10 minutes is killed, with every process it started, and stops the run (exit 2).
120
129
  - From the evals:
121
130
  - Items must reach every place a violation can be: a selector whose items miss the violating lines can't find them.
122
- - Context belongs in each item's text, and computed facts beat raw data: a TODO's age in days lets Clef judge "older than 90 days" at 99% accuracy; its commit date and today's date do not. Context costs input tokens.
131
+ - Context belongs in each item's text, and computed facts beat raw data: a TODO's age in days lets the model judge "older than 90 days" at 99% accuracy; its commit date and today's date do not. Context costs input tokens.
123
132
  - Packing a policy's items as questions of one request matches one request per item and costs 2.4× less.
124
133
  - An item's answer is cached by its text, so whatever the text holds decides when it is asked again. Line numbers in the text (as in the example above) re-ask every item below an edit in that file; text without them stays cached until the item itself changes. `path` and `lines` set the finding's location either way.
125
134
  - Structural units (functions, classes, handlers, tests) are easiest to select with a syntax-aware tool such as [ast-grep](https://ast-grep.github.io/). The eval corpus's policies (`evals/corpus/policies/`) and their scripts (`evals/steps/`) are worked examples: a selector lists the files it judges (`rg --files -g '*.ts'`) and pipes their paths to a script that cuts out the units.
126
135
  - Policies with the same `select` command share one run of it per lint run, so a selector several policies use can live in a script (for example `.lint-anything/lib/functions.sh`) that each calls.
127
136
 
128
- ### Counting items: `limit`
129
-
130
- A policy can count instead of reporting each item: with `limit`, its items are judged as usual, and then the policy reports one finding per file (or, with `per: run`, one for the whole run) that is a `violation` when more than `max` items are violations. Its reason says where the violating items are: their lines within the file, or (with `per: run`) their `path:lines`, else their ids. A group that isn't over the limit is `inconclusive` when its items that weren't judged (too large, or declined by the model) could put it over; the reason counts them by cause, as `N not judged (request too large)` or `N not judged (declined)`. With `per: file`, every item needs a `path`; an item without one stops the run (exit 2).
131
-
132
- ````md
133
- ---
134
- select: |
135
- rg --json -w 'TODO|FIXME' | jq -c '…'
136
- criteria:
137
- "true": "The comment has no tracking issue."
138
- "false": "The comment references a tracking issue."
139
- limit: { max: 3, per: file }
140
- ---
141
- Rule: every `TODO` or `FIXME` comment must reference a tracking issue.
142
- ````
143
-
144
- ```
145
- violation - src/many.ts todos 4 of 4 items violate the rule (at most 3 allowed): 2-2, 9-9, 14-14, 30-30
146
- ```
147
-
148
- Write the rule about one item, and put the count in `limit`: Clef judges items well and counts badly (see Evals). Items in `review` are mentioned in the reason but not counted.
149
-
150
137
  ## How it works
151
138
 
152
139
  Each policy's selector runs from the current directory, then the decision model judges the selected items. Up to 8 policies run at once, and at most 8 requests are in flight across the run. A request's text is the policy body; it carries one yes/no question per item, the item's text, with the policy's criteria. Each item gets its own finding. A policy whose selector emits nothing sends nothing. Findings are printed in policy order.
153
140
 
154
- A request holds at most 64 questions (Clef's limit) and 64,000 bytes, whichever model is asked. Items that don't fit in one request are split across several. Each request is POSTed to the model's endpoint with its API token or key as a bearer token: on Workers AI, `https://api.cloudflare.com/client/v4/accounts/<account>/ai/run/@cf/cloudflare/<model>`; otherwise `LINT_ANYTHING_ENDPOINT` as given.
141
+ A request holds at most 64 questions and 64,000 bytes, whichever model is asked (Clef's limits; every decision API that documents a limit allows at least as many questions). Items that don't fit in one request are split across several. Each request is POSTed to the model's endpoint with its API token or key as a bearer token: on Workers AI, `https://api.cloudflare.com/client/v4/accounts/<account>/ai/run/@cf/cloudflare/<model>`; otherwise `LINT_ANYTHING_ENDPOINT` as given.
155
142
 
156
143
  | Result | Verdict |
157
144
  | --- | --- |
@@ -167,17 +154,16 @@ Answers are cached on disk in one SQLite database, `.lint-anything/cache/answers
167
154
 
168
155
  A cached answer stands for 30 days; each run starts by deleting older answers, so their questions are asked again, answers follow updates to the model, and the cache doesn't grow without bound. A row can only hold a probability between 0 and 1. With 50,000 cached answers the database is 5.5 MB, and a run that takes every answer from it reads them in under half a second.
169
156
 
170
- Clef's answer to one question shifts a little with the other questions in its request (see Evals). A selector's items are sorted by path, line, and id before they are packed, so the same items always get the same answers; with the cache, an edit in one file also leaves the verdicts of every other file as they were.
157
+ A model's answer to one question can shift a little with the other questions in its request (Clef's does; see Evals). A selector's items are sorted by path, line, and id before they are packed, so the same items always get the same answers; with the cache, an edit in one file also leaves the verdicts of every other file as they were.
171
158
 
172
159
  ## Output
173
160
 
174
- The CLI prints one line per non-passing finding — verdict, violation probability, location, policy, and reason — then a count per verdict. The location is the item's `path:start-end`, or its `id` when it has no path and lines. A decision model answers only with a probability, so an item's reason is the policy's `criteria.true`: what the policy says a violation is. A `limit` finding gives its file's path, and its count and the violating items' places as the reason:
161
+ The CLI prints one line per non-passing finding — verdict, violation probability, location, policy, and reason — then a count per verdict. The location is the item's `path:start-end`, or its `id` when it has no path and lines. A decision model answers only with a probability, so an item's reason is the policy's `criteria.true`: what the policy says a violation is.
175
162
 
176
163
  ```
177
- violation - src/a.ts todo-limit 2 of 3 items violate the rule (at most 1 allowed): 1-1, 3-3
178
164
  violation 0.97 src/a.ts:4-4 todo The comment has no tracking issue.
179
165
  violation 0.97 9acebc18 commits The commit message doesn't say why.
180
- 3 checks: 3 violation, 0 review, 0 inconclusive, 0 pass
166
+ 2 checks: 2 violation, 0 review, 0 inconclusive, 0 pass
181
167
  ```
182
168
 
183
169
  Only `violation` fails the run; `review` and `inconclusive` are reported but exit `0`.
@@ -222,7 +208,7 @@ The package also exports the stages a run composes as an [Effect](https://effect
222
208
  | --- | --- |
223
209
  | `select(policy, cwd?, runs?)` | Runs a policy's selector; returns its items, sorted as they are judged |
224
210
  | `judge(policy, items)` | Each item's violation probability, or why there is none: `RequestTooLarge` or `Declined` (needs `DecisionModel`) |
225
- | `decide(policy, items, answers, threshold)` | Findings from those answers, `limit` included; pure |
211
+ | `decide(policy, items, answers, threshold)` | One finding per item from those answers; pure |
226
212
  | `lint(policies, threshold)` | All three for every policy, as the CLI runs them |
227
213
  | `DecisionModel.ask(source, state, questions)` | Yes/no questions under a shared state, packed, cached, retried; no policy needed |
228
214
  | `DecisionModel.layer(cacheDirectory?)` | The `DecisionModel` service for the model the environment variables above configure (`configuredTarget`) |
@@ -265,7 +251,6 @@ bun evals/examples.ts # examples written into the policy bo
265
251
  bun evals/cutoffs.ts # per-policy cut-offs tuned on dev vs. global (stored scores)
266
252
  bun evals/cache.ts # the answer cache: requests on re-runs and edits (real requests)
267
253
  bun evals/packing.ts # repeatability of fresh runs (real requests) and batch independence
268
- bun evals/counts.ts # limit vs. asking the count rule of the whole file
269
254
  bun evals/mine.ts prometheus/prometheus # candidate real violations from PR review comments (GitHub API, cached)
270
255
  bun evals/sample.ts --per-repo 3 owner/repo … # random real files by language, test vs. source, and size (GitHub API, cached)
271
256
  bun evals/wordings.ts --split test # each rule-wording set vs. the polished policies, each scored against its own gold
@@ -364,7 +349,7 @@ Each feature has an intended purpose, an eval that measures it, and a result. Ch
364
349
  | Examples in the body (few-shot) | Examples improve judgments | `examples.ts` | Calibration only: Brier better on both splits, AUROC unchanged | yes (write them in the body) |
365
350
  | Sorted items | The same files get the same verdicts | `packing.ts` | Holds: two fresh runs went from 40 differing verdicts to 0 | yes |
366
351
  | Answer cache (always on) | Re-runs pay only for changes; stable verdicts | `cache.ts` | Holds: 0 requests unchanged, 4 of 38 after an edit, 0 vs 38 verdict changes in untouched files | yes |
367
- | `limit` | Count rules judged reliably | `counts.ts` | Holds: 100% vs 92% (79% at the boundary) | yes |
352
+ | `limit` | Count rules judged reliably | `counts.ts` | Holds: 100% vs 92% (79% at the boundary) | no: removed as unnecessary (below) |
368
353
  | Per-policy cut-offs | Better verdicts than global cut-offs | `cutoffs.ts` | Fails: surfaced violations fall from 72% to 52% for +0.002 F1 | no |
369
354
  | Policy file sent whole as the rule | Clef reads the rule with its frontmatter | `evals/.cache/research/ident/ablate.ts` (2026-10-22) | Replaced: with the excerpt framing, sending the body without the frontmatter (selector code) raised dev AUROC 0.910 → 0.919; the body alone is now the request's text | no |
370
355
  | `ask` routing step | Drop items a rule doesn't govern, cutting false alarms | (commit `a4caa7c`) | Fails: dev false alarms rose, violations dropped | no |
@@ -382,7 +367,7 @@ Purpose: context from outside the file makes rules about time or history judgeab
382
367
  | … and its computed age in days | **1.000** [+0.077, +0.175] | **0.009** [−0.152, −0.099] | **99%** [+16, +31 pts] | **97%** | **0%** | 98% / 0% | $0.044 |
383
368
 
384
369
  - The purpose holds once the item states the fact the rule turns on. With the age in days, accuracy rises from 76% to 99% and false alarms from issue-less young TODOs vanish.
385
- - Clef doesn't do the date arithmetic: given today's date and each commit date, it still flags 88% of young issue-less TODOs. Compute facts in the selector; don't ask Clef to derive them.
370
+ - Clef doesn't do the date arithmetic: given today's date and each commit date, it still flags 88% of young issue-less TODOs. Compute facts in the selector; don't ask the model to derive them.
386
371
 
387
372
  #### Shared selector runs
388
373
 
@@ -461,22 +446,12 @@ Each flag the CLI used to take is now fixed at one value. Whether that value is
461
446
 
462
447
  Either default passes a clean checkout that has violations, and changed-since fails outright outside a git repository (`git diff HEAD` errors, exit 2). Off reports everything the selectors find; a selector scopes what it checks.
463
448
 
464
- #### Removed from the CLI: `check`, baseline, changed-lines scope, SARIF
449
+ #### Removed from the CLI: `check`, baseline, changed-lines scope, SARIF, `limit`
465
450
 
466
- These were built and measured with new flags and subcommands, then removed because the CLI takes no arguments. The code and evals are in the git history.
451
+ These were built and measured, then removed: the flags and subcommands because the CLI takes no arguments, and `limit` because it was unnecessary. The code and evals are in the git history.
467
452
 
468
453
  - `lint-anything check` (commit `47705a7`): judged each policy's examples against the rule without them. It caught a rule about the wrong thing (examples agreeing fell from 90% to 48%) but not an inverted one (87%), and it failed two semantic policies in their own wording.
469
454
  - `--baseline` and `--changed-since` (commit `9ddefd2`): on a generated repository, all five adoption scenarios behaved as expected; a baseline keyed by item text survived line shifts (one keyed by line would have called 7 of 11 accepted findings new).
470
455
  - `--sarif` (commit `cfab4c3`): the log validated against the SARIF 2.1.0 schema and every finding mapped to one result.
471
456
  - `ask` routing step (commit `a4caa7c`): routing the comments policy's items through "is this exempt?" raised dev false alarms from 8.2% to 11.5% and dropped 3 of 36 violating comments.
472
-
473
- #### Counting with `limit`
474
-
475
- Purpose: rules about a count ("at most 3 issue-less TODOs per file") are judged reliably. `evals/counts.ts` generates 64 TypeScript, Python, and Go files with a known number of issue-less TODOs (0 to 7, 24 files at exactly 3 or 4) among TODOs that cite an issue, and judges "more than 3" two ways:
476
-
477
- | Approach | Accuracy | Files with exactly 3 or 4 | AUROC | $ / 1k files |
478
- | --- | --- | --- | --- | --- |
479
- | the count rule, its selector emitting the whole file | 92% (59/64) | 79% (19/24) | 0.983 | $0.095 |
480
- | each TODO judged against "must reference an issue", counted by `limit: { max: 3 }` | **100%** (64/64) | **100%** (24/24) | 1.000 | $0.163 |
481
-
482
- The purpose holds: every file's count was exact, while the whole-file judgment ranked files well but misplaced the boundary in a fifth of the files at it. It costs about 1.7× as much.
457
+ - `limit` (commit `d4a6620`, `evals/counts.ts`): judged each item, then reported one finding per file or per run, a violation when more than `max` items violated the rule. On 64 generated files with 0 to 7 issue-less TODOs, counting judged TODOs against "at most 3" was exact on every file, where asking the count rule of the whole file got 92% (79% of the 24 files at 3 or 4), at about 1.7× the cost.