@andrueandersoncs/lint-anything 0.7.0 → 0.9.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -29,8 +29,11 @@ The CLI takes no arguments, and one flag, `--debug`: each run checks every polic
29
29
  | `LINT_ANYTHING_ENDPOINT` | no | The decision endpoint to ask, as a full URL: any System One, OpenAI Decisions, or Gemini Decisions endpoint. Without it, the model is asked on Workers AI |
30
30
  | `LINT_ANYTHING_MODEL` | with `LINT_ANYTHING_ENDPOINT` | The model to ask. Without an endpoint, a Cloudflare model on Workers AI: `clef` unless set, or, for example, `clef-flash` |
31
31
  | `LINT_ANYTHING_API_KEY` | no | API key or token for `LINT_ANYTHING_ENDPOINT`, if it needs one: sent as a bearer token, or to Google's Gemini API as `x-goog-api-key` |
32
- | `CLOUDFLARE_ACCOUNT_ID` | without an endpoint | Workers AI account |
33
- | `CLOUDFLARE_AUTH_TOKEN` | without an endpoint | Workers AI API token |
32
+ | `LINT_ANYTHING_SECONDARY_ENDPOINT` | no | Optional second decision endpoint for cross-family cascading: items whose primary probability lands in `[0.10, 0.90]` are also asked here and averaged with the primary score |
33
+ | `LINT_ANYTHING_SECONDARY_MODEL` | with `LINT_ANYTHING_SECONDARY_ENDPOINT` | Optional secondary model name. Set alone without `LINT_ANYTHING_SECONDARY_ENDPOINT` (for example `clef`) to cascade into a Cloudflare model on Workers AI using `CLOUDFLARE_*` |
34
+ | `LINT_ANYTHING_SECONDARY_API_KEY` | no | API key or token for `LINT_ANYTHING_SECONDARY_ENDPOINT`, if it needs one |
35
+ | `CLOUDFLARE_ACCOUNT_ID` | without an endpoint (or for a Workers AI secondary) | Workers AI account |
36
+ | `CLOUDFLARE_AUTH_TOKEN` | without an endpoint (or for a Workers AI secondary) | Workers AI API token |
34
37
  | `CLOUDFLARE_API_BASE_URL` | no | Defaults to `https://api.cloudflare.com/client/v4` |
35
38
 
36
39
  Bun loads `.env` and `.env.local` from the current directory automatically. An empty variable counts as unset.
@@ -75,8 +78,17 @@ LINT_ANYTHING_API_KEY=AIza...
75
78
 
76
79
  # Clef-flash on Workers AI, with the CLOUDFLARE_ credentials
77
80
  LINT_ANYTHING_MODEL=clef-flash
81
+
82
+ # Cross-family cascade: Gemini Decision 1 primary, escalating uncertain [0.10, 0.90] items to Clef on Workers AI
83
+ LINT_ANYTHING_ENDPOINT=https://generativelanguage.googleapis.com/v1beta/decisions
84
+ LINT_ANYTHING_MODEL=gemini-decision-1
85
+ LINT_ANYTHING_API_KEY=AIza...
86
+ LINT_ANYTHING_SECONDARY_MODEL=clef
87
+ # (plus CLOUDFLARE_ACCOUNT_ID and CLOUDFLARE_AUTH_TOKEN)
78
88
  ```
79
89
 
90
+ - **Cross-family cascading**: setting `LINT_ANYTHING_SECONDARY_MODEL` (and optionally `LINT_ANYTHING_SECONDARY_ENDPOINT` and `LINT_ANYTHING_SECONDARY_API_KEY`) asks the primary model first and escalates only items whose probability falls in `[0.10, 0.90]` to the secondary model, averaging both probabilities. Both models share the same SQLite cache under their own keys, so single-model and cascaded runs reuse cached answers. On the 216-file unseen holdout (`evals/RESEARCH.md`, H32/H34), `Gemini Decision 1 → Clef` escalated 31.6% of items, cut false alarms nearly in half (12.8% → 6.8%), raised TPR at 5% FPR to 63.3%, and improved File AUROC (0.920) and File Brier (0.080) for less than single-model Clef.
91
+
80
92
  - Every model is asked the same way: the same request text, at most 64 questions and 64,000 bytes per request, the same 0.5 / 0.65 cut-offs. A model whose context window is under 64K tokens may truncate a full request. Serve a local model with a context of at least that size.
81
93
  - Answers are cached per model, and per endpoint, so switching models never reuses another model's answers. Switching back finds the earlier model's answers still cached.
82
94
  - A question the model refuses (an OpenAI Decisions `refusal`) is `inconclusive`, with the reason `<model> declined to answer`. It isn't cached, so it is asked again next run. Gemini's input safety filters block a whole request instead (HTTP 400 with an error code such as `safety`), so every question in that request is `inconclusive` the same way.
@@ -117,7 +129,7 @@ A missing rules directory, one with no `.md` files, or an invalid policy stops t
117
129
 
118
130
  Policies are meant to be written and tuned by a coding agent: give it the rule you want and let it iterate on the selector, the body, and the criteria together, measured against labeled code (below). Because the CLI adds nothing, any way of asking the model that a request can express is available to a policy. `evals/corpus/selectors-excerpts/` is the earlier CLI's whole-file judging written as policies: its selector cuts files into 1,000-byte excerpts, and its body and criteria are that CLI's own words.
119
131
 
120
- `skills/lint-anything/SKILL.md` is an agent skill covering running the CLI, acting on findings, sorting a project's AGENTS.md or contributor docs into what becomes a policy and what doesn't, writing and tuning policies, and the pitfalls selectors run into. It ships in the package (`node_modules/@andrueandersoncs/lint-anything/skills/lint-anything/`) with its score script; copy that directory into a coding agent's skills directory.
132
+ `skills/lint-anything/SKILL.md` is an agent skill covering running the CLI, acting on findings, sorting a project's AGENTS.md or contributor docs into what becomes a policy and what doesn't, writing and tuning policies, and the pitfalls selectors run into. It ships in the package (`node_modules/@andrueandersoncs/lint-anything/skills/lint-anything/`) with its bundled `scripts/score.js` and `scripts/select.js` helpers; copy that directory into a coding agent's skills directory.
121
133
 
122
134
  ### Tuning against labels
123
135
 
@@ -159,7 +171,7 @@ An item counts as a violation when its lines overlap a labeled violation, and a
159
171
  - Context belongs in each item's text, and computed facts beat raw data: a TODO's age in days lets the model judge "older than 90 days" at 99% accuracy; its commit date and today's date do not. Context costs input tokens.
160
172
  - Packing a policy's items as questions of one request matches one request per item and costs 2.4× less.
161
173
  - An item's answer is cached by its text and the contents of its media, so whatever they hold decides when it is asked again. Line numbers in the text (as in the example above) re-ask every item below an edit in that file; text without them stays cached until the item itself changes. `path` and `lines` set the finding's location either way.
162
- - Structural units (functions, classes, handlers, tests) are easiest to select with a syntax-aware tool such as [ast-grep](https://ast-grep.github.io/). The eval corpus's policies (`evals/corpus/policies/`) and their scripts (`evals/steps/`) are worked examples: a selector lists the files it judges (`rg --files -g '*.ts'`) and pipes their paths to a script that cuts out the units.
174
+ - Structural units (functions, classes, handlers, tests, comments, exports, mutable state, magic literals) are easiest to select with the skill's bundled `scripts/select.js` helper (built from `evals/steps/`, run as `bun <skill>/scripts/select.js --help`) or directly with [ast-grep](https://ast-grep.github.io/). A selector lists the files it judges (`rg --files -g '*.{ts,tsx,js,py,go}'`) and pipes their paths through a file-cutting step (`functions`, `comments`, `errors`, `errors-go`, `literals`, `module-exports`, `module-state`, `module-outline`, `tests`, `duplicates`, `excerpts`), optional context steps (`file-role`, `enclosing`, `blame`), and `present` (pass `--no-lines` to omit line numbers from `text` for cache hits). Steps that parse ASTs (`functions`, `errors`, `errors-go`, `literals`, `tests`) resolve `ast-grep` from `PATH` or `./node_modules/.bin/ast-grep`.
163
175
  - Policies with the same `select` command share one run of it per lint run, so a selector several policies use can live in a script (for example `.lint-anything/lib/functions.sh`) that each calls.
164
176
 
165
177
  ## How it works
@@ -222,8 +234,8 @@ An error that stops the run (exit `2`) is printed to stderr as one line, `error:
222
234
  bun install
223
235
  bunx tsc --noEmit
224
236
  bun index.ts # run from source
225
- bun run build # dist/cli.js (the `lint-anything` bin), skills/lint-anything/scripts/score.js (the skill's score script), dist/lib.js and dist/types (the library)
226
- bun pm pack # the tarball npm would get: dist/cli.js, dist/lib.js, dist/types, skills/ (with score.js), README, LICENSE
237
+ bun run build # dist/cli.js (the `lint-anything` bin), skills/lint-anything/scripts/{score,select}.js, dist/lib.js and dist/types (the library)
238
+ bun pm pack # the tarball npm would get: dist/cli.js, dist/lib.js, dist/types, skills/ (with score.js and select.js), README, LICENSE
227
239
  npm publish # runs prepublishOnly (type check, build) first; the package is public
228
240
  ```
229
241
 
@@ -240,14 +252,15 @@ The package also exports the stages a run composes as an [Effect](https://effect
240
252
  | `decide(policy, items, answers, threshold)` | One finding per item from those answers; pure |
241
253
  | `lint(policies, threshold)` | All three for every policy, as the CLI runs them |
242
254
  | `DecisionModel.ask(source, state, questions)` | Yes/no questions under a shared state, packed, cached, retried; no policy needed |
243
- | `DecisionModel.layer(cacheDirectory?)` | The `DecisionModel` service for the model the environment variables above configure (`configuredTarget`) |
255
+ | `DecisionModel.layer(cacheDirectory?)` | The `DecisionModel` service for the model the environment variables above configure (`configuredTarget`, automatically cascading into `configuredSecondaryTarget` when set) |
244
256
  | `DecisionModel.layerFor(target, cacheDirectory?)` | The same for a model given in code: `workersAiTarget({ accountId, token, model? })` or `endpointTarget({ url, model, token? })` |
257
+ | `DecisionModel.cascadeLayer(primary, secondary, cacheDirectory?, options?)` | Cross-family cascade: asks `primary` first and escalates only `[low, high]` (default `[0.10, 0.90]`) probabilities to `secondary`, combining both answers |
245
258
  | `planRequests(state, questions, model?)` | The exact request bodies a model would be sent (default: Clef's) |
259
+ | `loadPolicies(directory)`, `parsePolicy(name, source)` | Policies from Markdown (`loadPolicies` needs `FileSystem` and `Path`, e.g. `BunServices.layer` from `@effect/platform-bun`) |
260
+ | `loadLabels(policy)`, `scorePolicy(policy, labels, cwd?)` | A policy scored against `.lint-anything/labels/<policy>.jsonl`: per labeled file and item, with reach (`labelFileItems`) |
261
+ | `quality`, `auroc`, `auprc`, `tprAtFpr`, `violationPrecision`, `pairedInterval`, `bootstrapInterval`, … | The metrics and 95% bootstrap intervals the evals and the score script report |
246
262
 
247
263
  `Clef` and `ClefError` remain as deprecated names for `DecisionModel` and `DecisionModelError`.
248
- | `loadPolicies(directory)`, `parsePolicy(name, source)` | Policies from Markdown (`loadPolicies` needs `FileSystem` and `Path`, e.g. `BunServices.layer` from `@effect/platform-bun`) |
249
- | `loadLabels(policy)`, `scorePolicy(policy, labels, cwd?)` | A policy scored against `.lint-anything/labels/<policy>.jsonl`: per labeled file and item, with reach |
250
- | `quality`, `auroc`, `pairedInterval`, `bootstrapInterval`, … | The metrics and 95% bootstrap intervals the evals and the score script report |
251
264
 
252
265
  ```ts
253
266
  import { BunServices } from "@effect/platform-bun"