@andrueandersoncs/lint-anything 0.5.1 → 0.7.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1,6 +1,6 @@
1
1
  # lint-anything
2
2
 
3
- Check anything against plain-English policies, judged by the decision model of your choice: any model served behind a System One or OpenAI Decisions endpoint, or, with no endpoint set, a Cloudflare model on Workers AI ([Clef](https://developers.cloudflare.com/workers-ai/models/clef/) unless you name another). A policy's selector decides what is checked: parts of files, facts derived from them, a repository's history, a third-party service.
3
+ Check anything against plain-English policies, judged by the decision model of your choice: any model served behind a System One, OpenAI Decisions, or Gemini Decisions endpoint, or, with no endpoint set, a Cloudflare model on Workers AI ([Clef](https://developers.cloudflare.com/workers-ai/models/clef/) unless you name another). A policy's selector decides what is checked: parts of files, facts derived from them, a repository's history, a third-party service, and images, audio, and video, for models that take them.
4
4
 
5
5
  ## Run
6
6
 
@@ -9,7 +9,7 @@ Requires [Bun](https://bun.sh) 1.2.23 or later, plus whatever your policies' sel
9
9
  ```sh
10
10
  bun add -d @andrueandersoncs/lint-anything # or: bun add -g @andrueandersoncs/lint-anything
11
11
 
12
- # a decision model: any System One or OpenAI Decisions endpoint…
12
+ # a decision model: any System One, OpenAI Decisions, or Gemini Decisions endpoint…
13
13
  echo 'LINT_ANYTHING_ENDPOINT=http://localhost:4000/v1/systemone' >> .env
14
14
  echo 'LINT_ANYTHING_MODEL=...' >> .env
15
15
  echo 'LINT_ANYTHING_API_KEY=...' >> .env # if the endpoint needs one
@@ -26,9 +26,9 @@ The CLI takes no arguments, and one flag, `--debug`: each run checks every polic
26
26
 
27
27
  | Environment variable | Required | Meaning |
28
28
  | --- | --- | --- |
29
- | `LINT_ANYTHING_ENDPOINT` | no | The decision endpoint to ask, as a full URL: any System One or OpenAI Decisions endpoint. Without it, the model is asked on Workers AI |
29
+ | `LINT_ANYTHING_ENDPOINT` | no | The decision endpoint to ask, as a full URL: any System One, OpenAI Decisions, or Gemini Decisions endpoint. Without it, the model is asked on Workers AI |
30
30
  | `LINT_ANYTHING_MODEL` | with `LINT_ANYTHING_ENDPOINT` | The model to ask. Without an endpoint, a Cloudflare model on Workers AI: `clef` unless set, or, for example, `clef-flash` |
31
- | `LINT_ANYTHING_API_KEY` | no | Bearer token for `LINT_ANYTHING_ENDPOINT`, if it needs one |
31
+ | `LINT_ANYTHING_API_KEY` | no | API key or token for `LINT_ANYTHING_ENDPOINT`, if it needs one: sent as a bearer token, or to Google's Gemini API as `x-goog-api-key` |
32
32
  | `CLOUDFLARE_ACCOUNT_ID` | without an endpoint | Workers AI account |
33
33
  | `CLOUDFLARE_AUTH_TOKEN` | without an endpoint | Workers AI API token |
34
34
  | `CLOUDFLARE_API_BASE_URL` | no | Defaults to `https://api.cloudflare.com/client/v4` |
@@ -37,13 +37,26 @@ Bun loads `.env` and `.env.local` from the current directory automatically. An e
37
37
 
38
38
  ### Choosing a decision model
39
39
 
40
- A decision model answers yes/no questions with the probability of yes instead of writing text. lint-anything isn't tied to one: it asks whichever model the environment names, and speaks the two decision APIs:
40
+ A decision model answers yes/no questions with the probability of yes instead of writing text. lint-anything isn't tied to one: it asks whichever model the environment names, and speaks three decision APIs:
41
41
 
42
42
  - **System One** (`POST …/v1/systemone`), the API Jev introduced: a `state`, then `noul` questions keyed by id, each with `instructions` and `criteria`. Clef on Workers AI speaks it, and so do most hosts and servers of decision models, such as [LiteLLM](https://docs.litellm.ai/docs/decisions)'s proxy (which routes to Jev, Clef, Perplexity, Databricks, Microsoft Foundry, vLLM, and others) and [LLM Gateway](https://llmgateway.io/blog/llm-decision-api).
43
43
  - **OpenAI Decisions** (`POST …/v1/decisions`), used when the endpoint's path ends in `/decisions`: an `input`, then `predicate` questions. A predicate has no criteria, so each question's instructions are the item's text followed by `true: <criteria.true>` and `false: <criteria.false>` lines, as LiteLLM does when it sends criteria to OpenAI.
44
+ - **Gemini Decisions** (`POST https://generativelanguage.googleapis.com/v1beta/decisions`), used when the endpoint is on `generativelanguage.googleapis.com`: Google's [Decisions API](https://aistudio.google.com/docs/decisions-api) for Gemini Decision. An `input`, then `noul` questions keyed by id, with the criteria as `true_description` and `false_description`. The key goes in the `x-goog-api-key` header; create one in [Google AI Studio](https://aistudio.google.com/api-keys). On the free tier, Gemini Decision takes 50,000 input tokens a minute. A request over that is answered with 429 and a `Retry-After`, which the CLI waits out (up to 3 retries), so a first run over a large repository may stop with HTTP 429 until it fits in the quota or the project is on a paid tier.
44
45
 
45
46
  With no endpoint set, the model is a Cloudflare model on Workers AI, asked with the `CLOUDFLARE_` credentials: `clef` unless `LINT_ANYTHING_MODEL` names another.
46
47
 
48
+ Items can show the model images, audio, and video (`media`, under Selectors), as far as the model takes them:
49
+
50
+ | Model | Images | Audio | Video | Where a request carries them |
51
+ | --- | --- | --- | --- | --- |
52
+ | Gemini Decision 1 (Gemini Decisions) | yes | yes | yes | `input`: an `image`, `audio`, or `video` block per file, in base64, then the policy body as a `text` block |
53
+ | OpenAI Decisions | yes | no | no | `input`: one user message, an `input_image` data URL per image, then the policy body as `input_text` |
54
+ | `clef-omni` on Workers AI | yes | yes | yes | `images`, `audio`, and `videos`: data URLs, before the `state` |
55
+ | `clef`, `clef-flash` on Workers AI | yes | no | no | `images`, as above |
56
+ | any other System One model | its server decides | | | as for Clef; LiteLLM's proxy takes these lists and translates them for the model it routes to (Jev's images go through it, as `assets`) |
57
+
58
+ An item whose media a model doesn't take stops the run (exit 2) before anything is sent: `Clef takes text and images, not audio clips: prompts/welcome.wav`. Media are sent as given, so a model's other limits (Clef's 16 megapixels per image, clef-omni's 300 seconds of audio and 60 of video) are its endpoint's to enforce: a file over one stops the run with the endpoint's error. The evals are all text, so no accuracy number here covers media: check a media policy on labeled files (`score.js`) before trusting its verdicts.
59
+
47
60
  ```sh
48
61
  # any System One server, such as a LiteLLM proxy
49
62
  LINT_ANYTHING_ENDPOINT=http://localhost:4000/v1/systemone
@@ -55,16 +68,21 @@ LINT_ANYTHING_ENDPOINT=https://api.openai.com/v1/decisions
55
68
  LINT_ANYTHING_MODEL=gpt-6-luna-decisions
56
69
  LINT_ANYTHING_API_KEY=sk-...
57
70
 
71
+ # Google's Decisions API, with a Gemini API key
72
+ LINT_ANYTHING_ENDPOINT=https://generativelanguage.googleapis.com/v1beta/decisions
73
+ LINT_ANYTHING_MODEL=gemini-decision-1
74
+ LINT_ANYTHING_API_KEY=AIza...
75
+
58
76
  # Clef-flash on Workers AI, with the CLOUDFLARE_ credentials
59
77
  LINT_ANYTHING_MODEL=clef-flash
60
78
  ```
61
79
 
62
80
  - Every model is asked the same way: the same request text, at most 64 questions and 64,000 bytes per request, the same 0.5 / 0.65 cut-offs. A model whose context window is under 64K tokens may truncate a full request. Serve a local model with a context of at least that size.
63
81
  - Answers are cached per model, and per endpoint, so switching models never reuses another model's answers. Switching back finds the earlier model's answers still cached.
64
- - A question the model refuses (an OpenAI Decisions `refusal`) is `inconclusive`, with the reason `<model> declined to answer`. It isn't cached, so it is asked again next run.
82
+ - A question the model refuses (an OpenAI Decisions `refusal`) is `inconclusive`, with the reason `<model> declined to answer`. It isn't cached, so it is asked again next run. Gemini's input safety filters block a whole request instead (HTTP 400 with an error code such as `safety`), so every question in that request is `inconclusive` the same way.
65
83
  - `--debug` prices tokens only for Clef. For other models, the cost column shows `-`.
66
84
  - Errors name the model: `error: typesafe/jev-latest responded with HTTP 401: …`. A configuration that can't work stops the run (exit 2) before anything is sent: an endpoint without `LINT_ANYTHING_MODEL`, an endpoint that isn't an http(s) URL, `LINT_ANYTHING_API_KEY` without an endpoint, or a model name like `typesafe/jev-latest` without an endpoint (Workers AI only serves Cloudflare models by name).
67
- - Every accuracy number in this README was measured on Clef, the model the evals were built with. Clef-flash scored much worse on the same corpus (Evals, "Defaults of the removed flags"), so check another model on your own labels (`score.js`, below) or on the corpus (`bun evals/run.ts` with the same variables set) before trusting its verdicts.
85
+ - Every accuracy number in this README was measured on Clef, the model the evals were built with. Clef-flash scored much worse on the same corpus (Evals, "Defaults of the removed flags"), so check another model on your own labels (`score.js`, below) or on the corpus (`bun evals/run.ts` with the same variables set) before trusting its verdicts. Gemini Decision 1 has been checked once, on the corpus's dev split (real files, semantic policies; [H27](evals/RESEARCH.md)): it ranks files as well as Clef (AUROC 0.937 on both) at 45% of the cost per file, but its scores run lower, so at the shipped cut-offs it catches fewer violations (59% vs. 68%) and raises fewer false alarms (7% vs. 13%).
68
86
 
69
87
  ## Policies
70
88
 
@@ -99,7 +117,7 @@ A missing rules directory, one with no `.md` files, or an invalid policy stops t
99
117
 
100
118
  Policies are meant to be written and tuned by a coding agent: give it the rule you want and let it iterate on the selector, the body, and the criteria together, measured against labeled code (below). Because the CLI adds nothing, any way of asking the model that a request can express is available to a policy. `evals/corpus/selectors-excerpts/` is the earlier CLI's whole-file judging written as policies: its selector cuts files into 1,000-byte excerpts, and its body and criteria are that CLI's own words.
101
119
 
102
- `skills/lint-anything/SKILL.md` is an agent skill covering running the CLI, acting on findings, writing and tuning policies, and the pitfalls selectors run into. It ships in the package (`node_modules/@andrueandersoncs/lint-anything/skills/lint-anything/`) with its score script; copy that directory into a coding agent's skills directory.
120
+ `skills/lint-anything/SKILL.md` is an agent skill covering running the CLI, acting on findings, sorting a project's AGENTS.md or contributor docs into what becomes a policy and what doesn't, writing and tuning policies, and the pitfalls selectors run into. It ships in the package (`node_modules/@andrueandersoncs/lint-anything/skills/lint-anything/`) with its score script; copy that directory into a coding agent's skills directory.
103
121
 
104
122
  ### Tuning against labels
105
123
 
@@ -110,7 +128,7 @@ bun <skill directory>/scripts/score.js <policy> # per split: re
110
128
  bun <skill directory>/scripts/score.js <policy> candidate.md # and candidate − current, paired intervals
111
129
  ```
112
130
 
113
- An item counts as a violation when its lines overlap a labeled violation, and a file scores its highest item score, so labels stay valid when the selector changes. It lists dev mistakes and unreached violations; test is only summarized, and a change is adopted when test improves outside noise. The skill asks for labels from two or three different models, with the user settling their disagreements. The evals score the corpus with the same code (`src/score.ts`).
131
+ An item counts as a violation when its lines overlap a labeled violation, and a file scores its highest item score, so labels stay valid when the selector changes. It lists dev mistakes and unreached violations; test is only summarized. A change is picked on dev and decided once, on freshly labeled test files, which then move to dev: in simulated tuning runs on the corpus, rechecking one test split candidate after candidate adopted a candidate no better than the current policy in 23–34% of runs within ten candidates (`evals/RESEARCH.md`, H25). The skill asks for labels from two or three different models, with the user settling their disagreements. The evals score the corpus with the same code (`src/score.ts`).
114
132
 
115
133
  ### Selectors
116
134
 
@@ -123,14 +141,24 @@ An item counts as a violation when its lines overlap a labeled violation, and a
123
141
  | `text` | yes | Everything the model reads for this item: the code or text to judge and any context, laid out as the selector likes |
124
142
  | `path` | no | The item's file, for findings |
125
143
  | `lines` | no | `[start, end]`, 1-based and inclusive, for findings |
144
+ | `media` | no | Image, audio, and video files the model is shown with the item's text, by path from where the selector ran |
145
+
146
+ - `media` files are typed by extension: `.png`, `.jpg`, `.jpeg`, `.webp` images; `.wav`, `.mp3`, `.flac`, `.ogg` audio; `.mp4`, `.webm` video. An item shows at most 4 images, 4 audio clips, and 2 videos (Clef's and clef-omni's limits, kept for every model). The text still says what the item is (`"screenshot shots/signup.png"`): it is the question the model answers about the media.
147
+
148
+ ```yaml
149
+ select: |
150
+ for f in shots/*.png; do
151
+ jq -cn --arg f "$f" '{id: $f, path: $f, text: "screenshot \($f)", media: [$f]}'
152
+ done
153
+ ```
126
154
 
127
- - A non-zero exit or a line that isn't a valid item stops the run (exit 2). `rg` and `grep` exit 1 when nothing matches, so end a bare search with `|| true`, or pipe it on as above.
155
+ - A non-zero exit or a line that isn't a valid item (an unknown media type, too many files of a kind, or a media path that names no file included) stops the run (exit 2). `rg` and `grep` exit 1 when nothing matches, so end a bare search with `|| true`, or pipe it on as above.
128
156
  - A selector still running after 10 minutes is killed, with every process it started, and stops the run (exit 2).
129
157
  - From the evals:
130
158
  - Items must reach every place a violation can be: a selector whose items miss the violating lines can't find them.
131
159
  - Context belongs in each item's text, and computed facts beat raw data: a TODO's age in days lets the model judge "older than 90 days" at 99% accuracy; its commit date and today's date do not. Context costs input tokens.
132
160
  - Packing a policy's items as questions of one request matches one request per item and costs 2.4× less.
133
- - An item's answer is cached by its text, so whatever the text holds decides when it is asked again. Line numbers in the text (as in the example above) re-ask every item below an edit in that file; text without them stays cached until the item itself changes. `path` and `lines` set the finding's location either way.
161
+ - An item's answer is cached by its text and the contents of its media, so whatever they hold decides when it is asked again. Line numbers in the text (as in the example above) re-ask every item below an edit in that file; text without them stays cached until the item itself changes. `path` and `lines` set the finding's location either way.
134
162
  - Structural units (functions, classes, handlers, tests) are easiest to select with a syntax-aware tool such as [ast-grep](https://ast-grep.github.io/). The eval corpus's policies (`evals/corpus/policies/`) and their scripts (`evals/steps/`) are worked examples: a selector lists the files it judges (`rg --files -g '*.ts'`) and pipes their paths to a script that cuts out the units.
135
163
  - Policies with the same `select` command share one run of it per lint run, so a selector several policies use can live in a script (for example `.lint-anything/lib/functions.sh`) that each calls.
136
164
 
@@ -138,7 +166,7 @@ An item counts as a violation when its lines overlap a labeled violation, and a
138
166
 
139
167
  Each policy's selector runs from the current directory, then the decision model judges the selected items. Up to 8 policies run at once, and at most 8 requests are in flight across the run. A request's text is the policy body; it carries one yes/no question per item, the item's text, with the policy's criteria. Each item gets its own finding. A policy whose selector emits nothing sends nothing. Findings are printed in policy order.
140
168
 
141
- A request holds at most 64 questions and 64,000 bytes, whichever model is asked (Clef's limits; every decision API that documents a limit allows at least as many questions). Items that don't fit in one request are split across several. Each request is POSTed to the model's endpoint with its API token or key as a bearer token: on Workers AI, `https://api.cloudflare.com/client/v4/accounts/<account>/ai/run/@cf/cloudflare/<model>`; otherwise `LINT_ANYTHING_ENDPOINT` as given.
169
+ A request holds at most 64 questions and 64,000 bytes of text, whichever model is asked (Clef's limits; every decision API that documents a limit allows at least as many questions). Items that don't fit in one request are split across several. A request shows its media for every question in it (Clef and Gemini take media for the whole request, not per question), so items share a request only with items that show the same files, in the same order: one request per screenshot, say, however many questions it answers. An item's media are read when its request is sent, at most 4 MiB per image and 8 MiB in all (Clef's limits; 8 MiB in base64 fits Workers AI's 13 MiB request). Each request is POSTed to the model's endpoint with its API token or key as a bearer token (to Google's Gemini API, as `x-goog-api-key`): on Workers AI, `https://api.cloudflare.com/client/v4/accounts/<account>/ai/run/@cf/cloudflare/<model>`; otherwise `LINT_ANYTHING_ENDPOINT` as given.
142
170
 
143
171
  | Result | Verdict |
144
172
  | --- | --- |
@@ -146,11 +174,12 @@ A request holds at most 64 questions and 64,000 bytes, whichever model is asked
146
174
  | `violates ≥ 0.65` | `violation` |
147
175
  | otherwise | `review`: uncertain, including cases where careful reviewers would disagree |
148
176
  | request with only that item over the byte limit | `inconclusive` (not sent): `request is N bytes; Clef would truncate it`, or for another model `request is N bytes, over the 64000-byte request limit` |
149
- | the model refused the question (OpenAI Decisions) | `inconclusive`: `<model> declined to answer` |
177
+ | an image over 4 MiB, or media over 8 MiB together | `inconclusive` (not sent): `image <path> is N bytes, over the 4194304-byte image limit`, or `media are N bytes, over the 8388608-byte media limit` |
178
+ | the model refused the question (OpenAI Decisions), or its safety filters blocked the request (Gemini Decisions) | `inconclusive`: `<model> declined to answer` |
150
179
 
151
180
  An attempt that hasn't returned its whole response within 60 seconds counts as a transient failure. Transient HTTP failures (timeouts, network errors, 408, 429, 5xx) are retried up to 3 times with exponential backoff, waiting at least as long as a `Retry-After` header asks. A request that still fails stops the run (exit 2) with the HTTP status and the endpoint's error, or the network error.
152
181
 
153
- Answers are cached on disk in one SQLite database, `.lint-anything/cache/answers.sqlite`, one row per question, keyed by the model (for a model at `LINT_ANYTHING_ENDPOINT`, its URL and name), the request's text (the policy body), and the question (item text and criteria). Only questions whose inputs changed are sent: re-running when nothing changed sends nothing, and editing a policy re-asks only that policy. Each request's answers are written in one transaction as it returns, so a run stopped by a failed request keeps the answers it already received, and a crash can't leave a partial answer. Runs can share the cache at once (WAL mode; a writer waits up to 10 seconds for another's lock). A cache that can't be opened (for example, a file that isn't a SQLite database) stops the run (exit 2). Delete the directory to judge everything afresh.
182
+ Answers are cached on disk in one SQLite database, `.lint-anything/cache/answers.sqlite`, one row per question, keyed by the model (for a model at `LINT_ANYTHING_ENDPOINT`, its URL and name), the request's text (the policy body), the question (item text and criteria), and the type and SHA-256 of each media file it shows, so an edited image is asked again and a renamed one isn't. Each run reads every media file once to fingerprint it. Only questions whose inputs changed are sent: re-running when nothing changed sends nothing, and editing a policy re-asks only that policy. Each request's answers are written in one transaction as it returns, so a run stopped by a failed request keeps the answers it already received, and a crash can't leave a partial answer. Runs can share the cache at once (WAL mode; a writer waits up to 10 seconds for another's lock). A cache that can't be opened (for example, a file that isn't a SQLite database) stops the run (exit 2). Delete the directory to judge everything afresh.
154
183
 
155
184
  A cached answer stands for 30 days; each run starts by deleting older answers, so their questions are asked again, answers follow updates to the model, and the cache doesn't grow without bound. A row can only hold a probability between 0 and 1. With 50,000 cached answers the database is 5.5 MB, and a run that takes every answer from it reads them in under half a second.
156
185
 
@@ -249,6 +278,7 @@ bun evals/history.ts # context from outside the file: a da
249
278
  bun evals/shared.ts # a shared selector runs once (stub decision model, no requests)
250
279
  bun evals/examples.ts # examples written into the policy body, as few-shot
251
280
  bun evals/cutoffs.ts # per-policy cut-offs tuned on dev vs. global (stored scores)
281
+ bun evals/spent.ts # tuning loops that reuse one test split vs. decide once (stored scores)
252
282
  bun evals/cache.ts # the answer cache: requests on re-runs and edits (real requests)
253
283
  bun evals/packing.ts # repeatability of fresh runs (real requests) and batch independence
254
284
  bun evals/mine.ts prometheus/prometheus # candidate real violations from PR review comments (GitHub API, cached)