@andrueandersoncs/lint-anything 0.5.1 → 0.6.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1,6 +1,6 @@
1
1
  # lint-anything
2
2
 
3
- Check anything against plain-English policies, judged by the decision model of your choice: any model served behind a System One or OpenAI Decisions endpoint, or, with no endpoint set, a Cloudflare model on Workers AI ([Clef](https://developers.cloudflare.com/workers-ai/models/clef/) unless you name another). A policy's selector decides what is checked: parts of files, facts derived from them, a repository's history, a third-party service.
3
+ Check anything against plain-English policies, judged by the decision model of your choice: any model served behind a System One, OpenAI Decisions, or Gemini Decisions endpoint, or, with no endpoint set, a Cloudflare model on Workers AI ([Clef](https://developers.cloudflare.com/workers-ai/models/clef/) unless you name another). A policy's selector decides what is checked: parts of files, facts derived from them, a repository's history, a third-party service.
4
4
 
5
5
  ## Run
6
6
 
@@ -9,7 +9,7 @@ Requires [Bun](https://bun.sh) 1.2.23 or later, plus whatever your policies' sel
9
9
  ```sh
10
10
  bun add -d @andrueandersoncs/lint-anything # or: bun add -g @andrueandersoncs/lint-anything
11
11
 
12
- # a decision model: any System One or OpenAI Decisions endpoint…
12
+ # a decision model: any System One, OpenAI Decisions, or Gemini Decisions endpoint…
13
13
  echo 'LINT_ANYTHING_ENDPOINT=http://localhost:4000/v1/systemone' >> .env
14
14
  echo 'LINT_ANYTHING_MODEL=...' >> .env
15
15
  echo 'LINT_ANYTHING_API_KEY=...' >> .env # if the endpoint needs one
@@ -26,9 +26,9 @@ The CLI takes no arguments, and one flag, `--debug`: each run checks every polic
26
26
 
27
27
  | Environment variable | Required | Meaning |
28
28
  | --- | --- | --- |
29
- | `LINT_ANYTHING_ENDPOINT` | no | The decision endpoint to ask, as a full URL: any System One or OpenAI Decisions endpoint. Without it, the model is asked on Workers AI |
29
+ | `LINT_ANYTHING_ENDPOINT` | no | The decision endpoint to ask, as a full URL: any System One, OpenAI Decisions, or Gemini Decisions endpoint. Without it, the model is asked on Workers AI |
30
30
  | `LINT_ANYTHING_MODEL` | with `LINT_ANYTHING_ENDPOINT` | The model to ask. Without an endpoint, a Cloudflare model on Workers AI: `clef` unless set, or, for example, `clef-flash` |
31
- | `LINT_ANYTHING_API_KEY` | no | Bearer token for `LINT_ANYTHING_ENDPOINT`, if it needs one |
31
+ | `LINT_ANYTHING_API_KEY` | no | API key or token for `LINT_ANYTHING_ENDPOINT`, if it needs one: sent as a bearer token, or to Google's Gemini API as `x-goog-api-key` |
32
32
  | `CLOUDFLARE_ACCOUNT_ID` | without an endpoint | Workers AI account |
33
33
  | `CLOUDFLARE_AUTH_TOKEN` | without an endpoint | Workers AI API token |
34
34
  | `CLOUDFLARE_API_BASE_URL` | no | Defaults to `https://api.cloudflare.com/client/v4` |
@@ -37,10 +37,11 @@ Bun loads `.env` and `.env.local` from the current directory automatically. An e
37
37
 
38
38
  ### Choosing a decision model
39
39
 
40
- A decision model answers yes/no questions with the probability of yes instead of writing text. lint-anything isn't tied to one: it asks whichever model the environment names, and speaks the two decision APIs:
40
+ A decision model answers yes/no questions with the probability of yes instead of writing text. lint-anything isn't tied to one: it asks whichever model the environment names, and speaks three decision APIs:
41
41
 
42
42
  - **System One** (`POST …/v1/systemone`), the API Jev introduced: a `state`, then `noul` questions keyed by id, each with `instructions` and `criteria`. Clef on Workers AI speaks it, and so do most hosts and servers of decision models, such as [LiteLLM](https://docs.litellm.ai/docs/decisions)'s proxy (which routes to Jev, Clef, Perplexity, Databricks, Microsoft Foundry, vLLM, and others) and [LLM Gateway](https://llmgateway.io/blog/llm-decision-api).
43
43
  - **OpenAI Decisions** (`POST …/v1/decisions`), used when the endpoint's path ends in `/decisions`: an `input`, then `predicate` questions. A predicate has no criteria, so each question's instructions are the item's text followed by `true: <criteria.true>` and `false: <criteria.false>` lines, as LiteLLM does when it sends criteria to OpenAI.
44
+ - **Gemini Decisions** (`POST https://generativelanguage.googleapis.com/v1beta/decisions`), used when the endpoint is on `generativelanguage.googleapis.com`: Google's [Decisions API](https://aistudio.google.com/docs/decisions-api) for Gemini Decision. An `input`, then `noul` questions keyed by id, with the criteria as `true_description` and `false_description`. The key goes in the `x-goog-api-key` header; create one in [Google AI Studio](https://aistudio.google.com/api-keys). On the free tier, Gemini Decision takes 50,000 input tokens a minute. A request over that is answered with 429 and a `Retry-After`, which the CLI waits out (up to 3 retries), so a first run over a large repository may stop with HTTP 429 until it fits in the quota or the project is on a paid tier.
44
45
 
45
46
  With no endpoint set, the model is a Cloudflare model on Workers AI, asked with the `CLOUDFLARE_` credentials: `clef` unless `LINT_ANYTHING_MODEL` names another.
46
47
 
@@ -55,16 +56,21 @@ LINT_ANYTHING_ENDPOINT=https://api.openai.com/v1/decisions
55
56
  LINT_ANYTHING_MODEL=gpt-6-luna-decisions
56
57
  LINT_ANYTHING_API_KEY=sk-...
57
58
 
59
+ # Google's Decisions API, with a Gemini API key
60
+ LINT_ANYTHING_ENDPOINT=https://generativelanguage.googleapis.com/v1beta/decisions
61
+ LINT_ANYTHING_MODEL=gemini-decision-1
62
+ LINT_ANYTHING_API_KEY=AIza...
63
+
58
64
  # Clef-flash on Workers AI, with the CLOUDFLARE_ credentials
59
65
  LINT_ANYTHING_MODEL=clef-flash
60
66
  ```
61
67
 
62
68
  - Every model is asked the same way: the same request text, at most 64 questions and 64,000 bytes per request, the same 0.5 / 0.65 cut-offs. A model whose context window is under 64K tokens may truncate a full request. Serve a local model with a context of at least that size.
63
69
  - Answers are cached per model, and per endpoint, so switching models never reuses another model's answers. Switching back finds the earlier model's answers still cached.
64
- - A question the model refuses (an OpenAI Decisions `refusal`) is `inconclusive`, with the reason `<model> declined to answer`. It isn't cached, so it is asked again next run.
70
+ - A question the model refuses (an OpenAI Decisions `refusal`) is `inconclusive`, with the reason `<model> declined to answer`. It isn't cached, so it is asked again next run. Gemini's input safety filters block a whole request instead (HTTP 400 with an error code such as `safety`), so every question in that request is `inconclusive` the same way.
65
71
  - `--debug` prices tokens only for Clef. For other models, the cost column shows `-`.
66
72
  - Errors name the model: `error: typesafe/jev-latest responded with HTTP 401: …`. A configuration that can't work stops the run (exit 2) before anything is sent: an endpoint without `LINT_ANYTHING_MODEL`, an endpoint that isn't an http(s) URL, `LINT_ANYTHING_API_KEY` without an endpoint, or a model name like `typesafe/jev-latest` without an endpoint (Workers AI only serves Cloudflare models by name).
67
- - Every accuracy number in this README was measured on Clef, the model the evals were built with. Clef-flash scored much worse on the same corpus (Evals, "Defaults of the removed flags"), so check another model on your own labels (`score.js`, below) or on the corpus (`bun evals/run.ts` with the same variables set) before trusting its verdicts.
73
+ - Every accuracy number in this README was measured on Clef, the model the evals were built with. Clef-flash scored much worse on the same corpus (Evals, "Defaults of the removed flags"), so check another model on your own labels (`score.js`, below) or on the corpus (`bun evals/run.ts` with the same variables set) before trusting its verdicts. Gemini Decision 1 has been checked once, on the corpus's dev split (real files, semantic policies; [H27](evals/RESEARCH.md)): it ranks files as well as Clef (AUROC 0.937 on both) at 45% of the cost per file, but its scores run lower, so at the shipped cut-offs it catches fewer violations (59% vs. 68%) and raises fewer false alarms (7% vs. 13%).
68
74
 
69
75
  ## Policies
70
76
 
@@ -99,7 +105,7 @@ A missing rules directory, one with no `.md` files, or an invalid policy stops t
99
105
 
100
106
  Policies are meant to be written and tuned by a coding agent: give it the rule you want and let it iterate on the selector, the body, and the criteria together, measured against labeled code (below). Because the CLI adds nothing, any way of asking the model that a request can express is available to a policy. `evals/corpus/selectors-excerpts/` is the earlier CLI's whole-file judging written as policies: its selector cuts files into 1,000-byte excerpts, and its body and criteria are that CLI's own words.
101
107
 
102
- `skills/lint-anything/SKILL.md` is an agent skill covering running the CLI, acting on findings, writing and tuning policies, and the pitfalls selectors run into. It ships in the package (`node_modules/@andrueandersoncs/lint-anything/skills/lint-anything/`) with its score script; copy that directory into a coding agent's skills directory.
108
+ `skills/lint-anything/SKILL.md` is an agent skill covering running the CLI, acting on findings, sorting a project's AGENTS.md or contributor docs into what becomes a policy and what doesn't, writing and tuning policies, and the pitfalls selectors run into. It ships in the package (`node_modules/@andrueandersoncs/lint-anything/skills/lint-anything/`) with its score script; copy that directory into a coding agent's skills directory.
103
109
 
104
110
  ### Tuning against labels
105
111
 
@@ -110,7 +116,7 @@ bun <skill directory>/scripts/score.js <policy> # per split: re
110
116
  bun <skill directory>/scripts/score.js <policy> candidate.md # and candidate − current, paired intervals
111
117
  ```
112
118
 
113
- An item counts as a violation when its lines overlap a labeled violation, and a file scores its highest item score, so labels stay valid when the selector changes. It lists dev mistakes and unreached violations; test is only summarized, and a change is adopted when test improves outside noise. The skill asks for labels from two or three different models, with the user settling their disagreements. The evals score the corpus with the same code (`src/score.ts`).
119
+ An item counts as a violation when its lines overlap a labeled violation, and a file scores its highest item score, so labels stay valid when the selector changes. It lists dev mistakes and unreached violations; test is only summarized. A change is picked on dev and decided once, on freshly labeled test files, which then move to dev: in simulated tuning runs on the corpus, rechecking one test split candidate after candidate adopted a candidate no better than the current policy in 23–34% of runs within ten candidates (`evals/RESEARCH.md`, H25). The skill asks for labels from two or three different models, with the user settling their disagreements. The evals score the corpus with the same code (`src/score.ts`).
114
120
 
115
121
  ### Selectors
116
122
 
@@ -138,7 +144,7 @@ An item counts as a violation when its lines overlap a labeled violation, and a
138
144
 
139
145
  Each policy's selector runs from the current directory, then the decision model judges the selected items. Up to 8 policies run at once, and at most 8 requests are in flight across the run. A request's text is the policy body; it carries one yes/no question per item, the item's text, with the policy's criteria. Each item gets its own finding. A policy whose selector emits nothing sends nothing. Findings are printed in policy order.
140
146
 
141
- A request holds at most 64 questions and 64,000 bytes, whichever model is asked (Clef's limits; every decision API that documents a limit allows at least as many questions). Items that don't fit in one request are split across several. Each request is POSTed to the model's endpoint with its API token or key as a bearer token: on Workers AI, `https://api.cloudflare.com/client/v4/accounts/<account>/ai/run/@cf/cloudflare/<model>`; otherwise `LINT_ANYTHING_ENDPOINT` as given.
147
+ A request holds at most 64 questions and 64,000 bytes, whichever model is asked (Clef's limits; every decision API that documents a limit allows at least as many questions). Items that don't fit in one request are split across several. Each request is POSTed to the model's endpoint with its API token or key as a bearer token (to Google's Gemini API, as `x-goog-api-key`): on Workers AI, `https://api.cloudflare.com/client/v4/accounts/<account>/ai/run/@cf/cloudflare/<model>`; otherwise `LINT_ANYTHING_ENDPOINT` as given.
142
148
 
143
149
  | Result | Verdict |
144
150
  | --- | --- |
@@ -146,7 +152,7 @@ A request holds at most 64 questions and 64,000 bytes, whichever model is asked
146
152
  | `violates ≥ 0.65` | `violation` |
147
153
  | otherwise | `review`: uncertain, including cases where careful reviewers would disagree |
148
154
  | request with only that item over the byte limit | `inconclusive` (not sent): `request is N bytes; Clef would truncate it`, or for another model `request is N bytes, over the 64000-byte request limit` |
149
- | the model refused the question (OpenAI Decisions) | `inconclusive`: `<model> declined to answer` |
155
+ | the model refused the question (OpenAI Decisions), or its safety filters blocked the request (Gemini Decisions) | `inconclusive`: `<model> declined to answer` |
150
156
 
151
157
  An attempt that hasn't returned its whole response within 60 seconds counts as a transient failure. Transient HTTP failures (timeouts, network errors, 408, 429, 5xx) are retried up to 3 times with exponential backoff, waiting at least as long as a `Retry-After` header asks. A request that still fails stops the run (exit 2) with the HTTP status and the endpoint's error, or the network error.
152
158
 
@@ -249,6 +255,7 @@ bun evals/history.ts # context from outside the file: a da
249
255
  bun evals/shared.ts # a shared selector runs once (stub decision model, no requests)
250
256
  bun evals/examples.ts # examples written into the policy body, as few-shot
251
257
  bun evals/cutoffs.ts # per-policy cut-offs tuned on dev vs. global (stored scores)
258
+ bun evals/spent.ts # tuning loops that reuse one test split vs. decide once (stored scores)
252
259
  bun evals/cache.ts # the answer cache: requests on re-runs and edits (real requests)
253
260
  bun evals/packing.ts # repeatability of fresh runs (real requests) and batch independence
254
261
  bun evals/mine.ts prometheus/prometheus # candidate real violations from PR review comments (GitHub API, cached)