@andrueandersoncs/lint-anything 0.5.0 → 0.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +36 -20
- package/dist/cli.js +60 -60
- package/dist/lib.js +52 -6
- package/dist/types/decision-model.d.ts +24 -7
- package/dist/types/lib.d.ts +2 -2
- package/package.json +2 -2
- package/skills/lint-anything/SKILL.md +43 -14
- package/skills/lint-anything/scripts/score.js +43 -43
package/README.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# lint-anything
|
|
2
2
|
|
|
3
|
-
Check anything against plain-English policies, judged by
|
|
3
|
+
Check anything against plain-English policies, judged by the decision model of your choice: any model served behind a System One, OpenAI Decisions, or Gemini Decisions endpoint, or, with no endpoint set, a Cloudflare model on Workers AI ([Clef](https://developers.cloudflare.com/workers-ai/models/clef/) unless you name another). A policy's selector decides what is checked: parts of files, facts derived from them, a repository's history, a third-party service.
|
|
4
4
|
|
|
5
5
|
## Run
|
|
6
6
|
|
|
@@ -8,37 +8,44 @@ Requires [Bun](https://bun.sh) 1.2.23 or later, plus whatever your policies' sel
|
|
|
8
8
|
|
|
9
9
|
```sh
|
|
10
10
|
bun add -d @andrueandersoncs/lint-anything # or: bun add -g @andrueandersoncs/lint-anything
|
|
11
|
+
|
|
12
|
+
# a decision model: any System One, OpenAI Decisions, or Gemini Decisions endpoint…
|
|
13
|
+
echo 'LINT_ANYTHING_ENDPOINT=http://localhost:4000/v1/systemone' >> .env
|
|
14
|
+
echo 'LINT_ANYTHING_MODEL=...' >> .env
|
|
15
|
+
echo 'LINT_ANYTHING_API_KEY=...' >> .env # if the endpoint needs one
|
|
16
|
+
# …or, with no endpoint, Workers AI
|
|
11
17
|
echo 'CLOUDFLARE_ACCOUNT_ID=...' >> .env
|
|
12
18
|
echo 'CLOUDFLARE_AUTH_TOKEN=...' >> .env
|
|
19
|
+
|
|
13
20
|
bunx lint-anything # from the project root
|
|
14
21
|
```
|
|
15
22
|
|
|
16
23
|
The command is `lint-anything`. It runs on Bun, not Node: without Bun on the `PATH` it fails with `env: bun: No such file or directory`.
|
|
17
24
|
|
|
18
|
-
The CLI takes no arguments, and one flag, `--debug`: each run checks every policy. Policies are read from `.lint-anything/rules` under the current directory, and each policy's selector finds what it checks. Items are judged by
|
|
25
|
+
The CLI takes no arguments, and one flag, `--debug`: each run checks every policy. Policies are read from `.lint-anything/rules` under the current directory, and each policy's selector finds what it checks. Items are judged by the decision model the environment configures (below) and printed as text. Answers are cached in `.lint-anything/cache/answers.sqlite` (add `.lint-anything/cache` to `.gitignore`). Effect CLI also provides its global flags (`--help`, `--version`, `--wizard`, `--completions`, `--log-level`); they are expected and stay.
|
|
19
26
|
|
|
20
27
|
| Environment variable | Required | Meaning |
|
|
21
28
|
| --- | --- | --- |
|
|
22
|
-
| `
|
|
23
|
-
| `
|
|
29
|
+
| `LINT_ANYTHING_ENDPOINT` | no | The decision endpoint to ask, as a full URL: any System One, OpenAI Decisions, or Gemini Decisions endpoint. Without it, the model is asked on Workers AI |
|
|
30
|
+
| `LINT_ANYTHING_MODEL` | with `LINT_ANYTHING_ENDPOINT` | The model to ask. Without an endpoint, a Cloudflare model on Workers AI: `clef` unless set, or, for example, `clef-flash` |
|
|
31
|
+
| `LINT_ANYTHING_API_KEY` | no | API key or token for `LINT_ANYTHING_ENDPOINT`, if it needs one: sent as a bearer token, or to Google's Gemini API as `x-goog-api-key` |
|
|
32
|
+
| `CLOUDFLARE_ACCOUNT_ID` | without an endpoint | Workers AI account |
|
|
33
|
+
| `CLOUDFLARE_AUTH_TOKEN` | without an endpoint | Workers AI API token |
|
|
24
34
|
| `CLOUDFLARE_API_BASE_URL` | no | Defaults to `https://api.cloudflare.com/client/v4` |
|
|
25
|
-
| `LINT_ANYTHING_ENDPOINT` | no | Another decision endpoint, as a full URL: the model is asked there instead of on Workers AI, and the `CLOUDFLARE_` variables aren't needed |
|
|
26
|
-
| `LINT_ANYTHING_MODEL` | with `LINT_ANYTHING_ENDPOINT` | The model to ask. Without an endpoint, a Cloudflare model on Workers AI: `clef` (the default) or, for example, `clef-flash` |
|
|
27
|
-
| `LINT_ANYTHING_API_KEY` | no | Bearer token for `LINT_ANYTHING_ENDPOINT`, if it needs one |
|
|
28
35
|
|
|
29
36
|
Bun loads `.env` and `.env.local` from the current directory automatically. An empty variable counts as unset.
|
|
30
37
|
|
|
31
38
|
### Choosing a decision model
|
|
32
39
|
|
|
33
|
-
A decision model answers yes/no questions with the probability of yes instead of writing text.
|
|
40
|
+
A decision model answers yes/no questions with the probability of yes instead of writing text. lint-anything isn't tied to one: it asks whichever model the environment names, and speaks three decision APIs:
|
|
34
41
|
|
|
35
42
|
- **System One** (`POST …/v1/systemone`), the API Jev introduced: a `state`, then `noul` questions keyed by id, each with `instructions` and `criteria`. Clef on Workers AI speaks it, and so do most hosts and servers of decision models, such as [LiteLLM](https://docs.litellm.ai/docs/decisions)'s proxy (which routes to Jev, Clef, Perplexity, Databricks, Microsoft Foundry, vLLM, and others) and [LLM Gateway](https://llmgateway.io/blog/llm-decision-api).
|
|
36
43
|
- **OpenAI Decisions** (`POST …/v1/decisions`), used when the endpoint's path ends in `/decisions`: an `input`, then `predicate` questions. A predicate has no criteria, so each question's instructions are the item's text followed by `true: <criteria.true>` and `false: <criteria.false>` lines, as LiteLLM does when it sends criteria to OpenAI.
|
|
44
|
+
- **Gemini Decisions** (`POST https://generativelanguage.googleapis.com/v1beta/decisions`), used when the endpoint is on `generativelanguage.googleapis.com`: Google's [Decisions API](https://aistudio.google.com/docs/decisions-api) for Gemini Decision. An `input`, then `noul` questions keyed by id, with the criteria as `true_description` and `false_description`. The key goes in the `x-goog-api-key` header; create one in [Google AI Studio](https://aistudio.google.com/api-keys). On the free tier, Gemini Decision takes 50,000 input tokens a minute. A request over that is answered with 429 and a `Retry-After`, which the CLI waits out (up to 3 retries), so a first run over a large repository may stop with HTTP 429 until it fits in the quota or the project is on a paid tier.
|
|
37
45
|
|
|
38
|
-
|
|
39
|
-
# Clef-flash on Workers AI, with the CLOUDFLARE_ credentials above
|
|
40
|
-
LINT_ANYTHING_MODEL=clef-flash
|
|
46
|
+
With no endpoint set, the model is a Cloudflare model on Workers AI, asked with the `CLOUDFLARE_` credentials: `clef` unless `LINT_ANYTHING_MODEL` names another.
|
|
41
47
|
|
|
48
|
+
```sh
|
|
42
49
|
# any System One server, such as a LiteLLM proxy
|
|
43
50
|
LINT_ANYTHING_ENDPOINT=http://localhost:4000/v1/systemone
|
|
44
51
|
LINT_ANYTHING_MODEL=typesafe/jev-latest
|
|
@@ -48,14 +55,22 @@ LINT_ANYTHING_API_KEY=sk-...
|
|
|
48
55
|
LINT_ANYTHING_ENDPOINT=https://api.openai.com/v1/decisions
|
|
49
56
|
LINT_ANYTHING_MODEL=gpt-6-luna-decisions
|
|
50
57
|
LINT_ANYTHING_API_KEY=sk-...
|
|
58
|
+
|
|
59
|
+
# Google's Decisions API, with a Gemini API key
|
|
60
|
+
LINT_ANYTHING_ENDPOINT=https://generativelanguage.googleapis.com/v1beta/decisions
|
|
61
|
+
LINT_ANYTHING_MODEL=gemini-decision-1
|
|
62
|
+
LINT_ANYTHING_API_KEY=AIza...
|
|
63
|
+
|
|
64
|
+
# Clef-flash on Workers AI, with the CLOUDFLARE_ credentials
|
|
65
|
+
LINT_ANYTHING_MODEL=clef-flash
|
|
51
66
|
```
|
|
52
67
|
|
|
53
68
|
- Every model is asked the same way: the same request text, at most 64 questions and 64,000 bytes per request, the same 0.5 / 0.65 cut-offs. A model whose context window is under 64K tokens may truncate a full request. Serve a local model with a context of at least that size.
|
|
54
69
|
- Answers are cached per model, and per endpoint, so switching models never reuses another model's answers. Switching back finds the earlier model's answers still cached.
|
|
55
|
-
- A question the model refuses (an OpenAI Decisions `refusal`) is `inconclusive`, with the reason `<model> declined to answer`. It isn't cached, so it is asked again next run.
|
|
70
|
+
- A question the model refuses (an OpenAI Decisions `refusal`) is `inconclusive`, with the reason `<model> declined to answer`. It isn't cached, so it is asked again next run. Gemini's input safety filters block a whole request instead (HTTP 400 with an error code such as `safety`), so every question in that request is `inconclusive` the same way.
|
|
56
71
|
- `--debug` prices tokens only for Clef. For other models, the cost column shows `-`.
|
|
57
72
|
- Errors name the model: `error: typesafe/jev-latest responded with HTTP 401: …`. A configuration that can't work stops the run (exit 2) before anything is sent: an endpoint without `LINT_ANYTHING_MODEL`, an endpoint that isn't an http(s) URL, `LINT_ANYTHING_API_KEY` without an endpoint, or a model name like `typesafe/jev-latest` without an endpoint (Workers AI only serves Cloudflare models by name).
|
|
58
|
-
- Every accuracy number in this README was measured on Clef. Clef-flash scored much worse on the same corpus (Evals, "Defaults of the removed flags"), so check another model on your own labels (`score.js`, below) or on the corpus (`bun evals/run.ts` with the same variables set) before trusting its verdicts.
|
|
73
|
+
- Every accuracy number in this README was measured on Clef, the model the evals were built with. Clef-flash scored much worse on the same corpus (Evals, "Defaults of the removed flags"), so check another model on your own labels (`score.js`, below) or on the corpus (`bun evals/run.ts` with the same variables set) before trusting its verdicts. Gemini Decision 1 has been checked once, on the corpus's dev split (real files, semantic policies; [H27](evals/RESEARCH.md)): it ranks files as well as Clef (AUROC 0.937 on both) at 45% of the cost per file, but its scores run lower, so at the shipped cut-offs it catches fewer violations (59% vs. 68%) and raises fewer false alarms (7% vs. 13%).
|
|
59
74
|
|
|
60
75
|
## Policies
|
|
61
76
|
|
|
@@ -90,7 +105,7 @@ A missing rules directory, one with no `.md` files, or an invalid policy stops t
|
|
|
90
105
|
|
|
91
106
|
Policies are meant to be written and tuned by a coding agent: give it the rule you want and let it iterate on the selector, the body, and the criteria together, measured against labeled code (below). Because the CLI adds nothing, any way of asking the model that a request can express is available to a policy. `evals/corpus/selectors-excerpts/` is the earlier CLI's whole-file judging written as policies: its selector cuts files into 1,000-byte excerpts, and its body and criteria are that CLI's own words.
|
|
92
107
|
|
|
93
|
-
`skills/lint-anything/SKILL.md` is an agent skill covering running the CLI, acting on findings, writing and tuning policies, and the pitfalls selectors run into. It ships in the package (`node_modules/@andrueandersoncs/lint-anything/skills/lint-anything/`) with its score script; copy that directory into a coding agent's skills directory.
|
|
108
|
+
`skills/lint-anything/SKILL.md` is an agent skill covering running the CLI, acting on findings, sorting a project's AGENTS.md or contributor docs into what becomes a policy and what doesn't, writing and tuning policies, and the pitfalls selectors run into. It ships in the package (`node_modules/@andrueandersoncs/lint-anything/skills/lint-anything/`) with its score script; copy that directory into a coding agent's skills directory.
|
|
94
109
|
|
|
95
110
|
### Tuning against labels
|
|
96
111
|
|
|
@@ -101,7 +116,7 @@ bun <skill directory>/scripts/score.js <policy> # per split: re
|
|
|
101
116
|
bun <skill directory>/scripts/score.js <policy> candidate.md # and candidate − current, paired intervals
|
|
102
117
|
```
|
|
103
118
|
|
|
104
|
-
An item counts as a violation when its lines overlap a labeled violation, and a file scores its highest item score, so labels stay valid when the selector changes. It lists dev mistakes and unreached violations; test is only summarized
|
|
119
|
+
An item counts as a violation when its lines overlap a labeled violation, and a file scores its highest item score, so labels stay valid when the selector changes. It lists dev mistakes and unreached violations; test is only summarized. A change is picked on dev and decided once, on freshly labeled test files, which then move to dev: in simulated tuning runs on the corpus, rechecking one test split candidate after candidate adopted a candidate no better than the current policy in 23–34% of runs within ten candidates (`evals/RESEARCH.md`, H25). The skill asks for labels from two or three different models, with the user settling their disagreements. The evals score the corpus with the same code (`src/score.ts`).
|
|
105
120
|
|
|
106
121
|
### Selectors
|
|
107
122
|
|
|
@@ -119,7 +134,7 @@ An item counts as a violation when its lines overlap a labeled violation, and a
|
|
|
119
134
|
- A selector still running after 10 minutes is killed, with every process it started, and stops the run (exit 2).
|
|
120
135
|
- From the evals:
|
|
121
136
|
- Items must reach every place a violation can be: a selector whose items miss the violating lines can't find them.
|
|
122
|
-
- Context belongs in each item's text, and computed facts beat raw data: a TODO's age in days lets
|
|
137
|
+
- Context belongs in each item's text, and computed facts beat raw data: a TODO's age in days lets the model judge "older than 90 days" at 99% accuracy; its commit date and today's date do not. Context costs input tokens.
|
|
123
138
|
- Packing a policy's items as questions of one request matches one request per item and costs 2.4× less.
|
|
124
139
|
- An item's answer is cached by its text, so whatever the text holds decides when it is asked again. Line numbers in the text (as in the example above) re-ask every item below an edit in that file; text without them stays cached until the item itself changes. `path` and `lines` set the finding's location either way.
|
|
125
140
|
- Structural units (functions, classes, handlers, tests) are easiest to select with a syntax-aware tool such as [ast-grep](https://ast-grep.github.io/). The eval corpus's policies (`evals/corpus/policies/`) and their scripts (`evals/steps/`) are worked examples: a selector lists the files it judges (`rg --files -g '*.ts'`) and pipes their paths to a script that cuts out the units.
|
|
@@ -129,7 +144,7 @@ An item counts as a violation when its lines overlap a labeled violation, and a
|
|
|
129
144
|
|
|
130
145
|
Each policy's selector runs from the current directory, then the decision model judges the selected items. Up to 8 policies run at once, and at most 8 requests are in flight across the run. A request's text is the policy body; it carries one yes/no question per item, the item's text, with the policy's criteria. Each item gets its own finding. A policy whose selector emits nothing sends nothing. Findings are printed in policy order.
|
|
131
146
|
|
|
132
|
-
A request holds at most 64 questions
|
|
147
|
+
A request holds at most 64 questions and 64,000 bytes, whichever model is asked (Clef's limits; every decision API that documents a limit allows at least as many questions). Items that don't fit in one request are split across several. Each request is POSTed to the model's endpoint with its API token or key as a bearer token (to Google's Gemini API, as `x-goog-api-key`): on Workers AI, `https://api.cloudflare.com/client/v4/accounts/<account>/ai/run/@cf/cloudflare/<model>`; otherwise `LINT_ANYTHING_ENDPOINT` as given.
|
|
133
148
|
|
|
134
149
|
| Result | Verdict |
|
|
135
150
|
| --- | --- |
|
|
@@ -137,7 +152,7 @@ A request holds at most 64 questions (Clef's limit) and 64,000 bytes, whichever
|
|
|
137
152
|
| `violates ≥ 0.65` | `violation` |
|
|
138
153
|
| otherwise | `review`: uncertain, including cases where careful reviewers would disagree |
|
|
139
154
|
| request with only that item over the byte limit | `inconclusive` (not sent): `request is N bytes; Clef would truncate it`, or for another model `request is N bytes, over the 64000-byte request limit` |
|
|
140
|
-
| the model refused the question (OpenAI Decisions) | `inconclusive`: `<model> declined to answer` |
|
|
155
|
+
| the model refused the question (OpenAI Decisions), or its safety filters blocked the request (Gemini Decisions) | `inconclusive`: `<model> declined to answer` |
|
|
141
156
|
|
|
142
157
|
An attempt that hasn't returned its whole response within 60 seconds counts as a transient failure. Transient HTTP failures (timeouts, network errors, 408, 429, 5xx) are retried up to 3 times with exponential backoff, waiting at least as long as a `Retry-After` header asks. A request that still fails stops the run (exit 2) with the HTTP status and the endpoint's error, or the network error.
|
|
143
158
|
|
|
@@ -145,7 +160,7 @@ Answers are cached on disk in one SQLite database, `.lint-anything/cache/answers
|
|
|
145
160
|
|
|
146
161
|
A cached answer stands for 30 days; each run starts by deleting older answers, so their questions are asked again, answers follow updates to the model, and the cache doesn't grow without bound. A row can only hold a probability between 0 and 1. With 50,000 cached answers the database is 5.5 MB, and a run that takes every answer from it reads them in under half a second.
|
|
147
162
|
|
|
148
|
-
|
|
163
|
+
A model's answer to one question can shift a little with the other questions in its request (Clef's does; see Evals). A selector's items are sorted by path, line, and id before they are packed, so the same items always get the same answers; with the cache, an edit in one file also leaves the verdicts of every other file as they were.
|
|
149
164
|
|
|
150
165
|
## Output
|
|
151
166
|
|
|
@@ -240,6 +255,7 @@ bun evals/history.ts # context from outside the file: a da
|
|
|
240
255
|
bun evals/shared.ts # a shared selector runs once (stub decision model, no requests)
|
|
241
256
|
bun evals/examples.ts # examples written into the policy body, as few-shot
|
|
242
257
|
bun evals/cutoffs.ts # per-policy cut-offs tuned on dev vs. global (stored scores)
|
|
258
|
+
bun evals/spent.ts # tuning loops that reuse one test split vs. decide once (stored scores)
|
|
243
259
|
bun evals/cache.ts # the answer cache: requests on re-runs and edits (real requests)
|
|
244
260
|
bun evals/packing.ts # repeatability of fresh runs (real requests) and batch independence
|
|
245
261
|
bun evals/mine.ts prometheus/prometheus # candidate real violations from PR review comments (GitHub API, cached)
|
|
@@ -358,7 +374,7 @@ Purpose: context from outside the file makes rules about time or history judgeab
|
|
|
358
374
|
| … and its computed age in days | **1.000** [+0.077, +0.175] | **0.009** [−0.152, −0.099] | **99%** [+16, +31 pts] | **97%** | **0%** | 98% / 0% | $0.044 |
|
|
359
375
|
|
|
360
376
|
- The purpose holds once the item states the fact the rule turns on. With the age in days, accuracy rises from 76% to 99% and false alarms from issue-less young TODOs vanish.
|
|
361
|
-
- Clef doesn't do the date arithmetic: given today's date and each commit date, it still flags 88% of young issue-less TODOs. Compute facts in the selector; don't ask
|
|
377
|
+
- Clef doesn't do the date arithmetic: given today's date and each commit date, it still flags 88% of young issue-less TODOs. Compute facts in the selector; don't ask the model to derive them.
|
|
362
378
|
|
|
363
379
|
#### Shared selector runs
|
|
364
380
|
|