@andrueandersoncs/lint-anything 0.5.0 → 0.5.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +25 -16
- package/dist/cli.js +1 -1
- package/package.json +2 -2
- package/skills/lint-anything/SKILL.md +7 -6
package/README.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# lint-anything
|
|
2
2
|
|
|
3
|
-
Check anything against plain-English policies, judged by
|
|
3
|
+
Check anything against plain-English policies, judged by the decision model of your choice: any model served behind a System One or OpenAI Decisions endpoint, or, with no endpoint set, a Cloudflare model on Workers AI ([Clef](https://developers.cloudflare.com/workers-ai/models/clef/) unless you name another). A policy's selector decides what is checked: parts of files, facts derived from them, a repository's history, a third-party service.
|
|
4
4
|
|
|
5
5
|
## Run
|
|
6
6
|
|
|
@@ -8,37 +8,43 @@ Requires [Bun](https://bun.sh) 1.2.23 or later, plus whatever your policies' sel
|
|
|
8
8
|
|
|
9
9
|
```sh
|
|
10
10
|
bun add -d @andrueandersoncs/lint-anything # or: bun add -g @andrueandersoncs/lint-anything
|
|
11
|
+
|
|
12
|
+
# a decision model: any System One or OpenAI Decisions endpoint…
|
|
13
|
+
echo 'LINT_ANYTHING_ENDPOINT=http://localhost:4000/v1/systemone' >> .env
|
|
14
|
+
echo 'LINT_ANYTHING_MODEL=...' >> .env
|
|
15
|
+
echo 'LINT_ANYTHING_API_KEY=...' >> .env # if the endpoint needs one
|
|
16
|
+
# …or, with no endpoint, Workers AI
|
|
11
17
|
echo 'CLOUDFLARE_ACCOUNT_ID=...' >> .env
|
|
12
18
|
echo 'CLOUDFLARE_AUTH_TOKEN=...' >> .env
|
|
19
|
+
|
|
13
20
|
bunx lint-anything # from the project root
|
|
14
21
|
```
|
|
15
22
|
|
|
16
23
|
The command is `lint-anything`. It runs on Bun, not Node: without Bun on the `PATH` it fails with `env: bun: No such file or directory`.
|
|
17
24
|
|
|
18
|
-
The CLI takes no arguments, and one flag, `--debug`: each run checks every policy. Policies are read from `.lint-anything/rules` under the current directory, and each policy's selector finds what it checks. Items are judged by
|
|
25
|
+
The CLI takes no arguments, and one flag, `--debug`: each run checks every policy. Policies are read from `.lint-anything/rules` under the current directory, and each policy's selector finds what it checks. Items are judged by the decision model the environment configures (below) and printed as text. Answers are cached in `.lint-anything/cache/answers.sqlite` (add `.lint-anything/cache` to `.gitignore`). Effect CLI also provides its global flags (`--help`, `--version`, `--wizard`, `--completions`, `--log-level`); they are expected and stay.
|
|
19
26
|
|
|
20
27
|
| Environment variable | Required | Meaning |
|
|
21
28
|
| --- | --- | --- |
|
|
22
|
-
| `
|
|
23
|
-
| `
|
|
24
|
-
| `CLOUDFLARE_API_BASE_URL` | no | Defaults to `https://api.cloudflare.com/client/v4` |
|
|
25
|
-
| `LINT_ANYTHING_ENDPOINT` | no | Another decision endpoint, as a full URL: the model is asked there instead of on Workers AI, and the `CLOUDFLARE_` variables aren't needed |
|
|
26
|
-
| `LINT_ANYTHING_MODEL` | with `LINT_ANYTHING_ENDPOINT` | The model to ask. Without an endpoint, a Cloudflare model on Workers AI: `clef` (the default) or, for example, `clef-flash` |
|
|
29
|
+
| `LINT_ANYTHING_ENDPOINT` | no | The decision endpoint to ask, as a full URL: any System One or OpenAI Decisions endpoint. Without it, the model is asked on Workers AI |
|
|
30
|
+
| `LINT_ANYTHING_MODEL` | with `LINT_ANYTHING_ENDPOINT` | The model to ask. Without an endpoint, a Cloudflare model on Workers AI: `clef` unless set, or, for example, `clef-flash` |
|
|
27
31
|
| `LINT_ANYTHING_API_KEY` | no | Bearer token for `LINT_ANYTHING_ENDPOINT`, if it needs one |
|
|
32
|
+
| `CLOUDFLARE_ACCOUNT_ID` | without an endpoint | Workers AI account |
|
|
33
|
+
| `CLOUDFLARE_AUTH_TOKEN` | without an endpoint | Workers AI API token |
|
|
34
|
+
| `CLOUDFLARE_API_BASE_URL` | no | Defaults to `https://api.cloudflare.com/client/v4` |
|
|
28
35
|
|
|
29
36
|
Bun loads `.env` and `.env.local` from the current directory automatically. An empty variable counts as unset.
|
|
30
37
|
|
|
31
38
|
### Choosing a decision model
|
|
32
39
|
|
|
33
|
-
A decision model answers yes/no questions with the probability of yes instead of writing text.
|
|
40
|
+
A decision model answers yes/no questions with the probability of yes instead of writing text. lint-anything isn't tied to one: it asks whichever model the environment names, and speaks the two decision APIs:
|
|
34
41
|
|
|
35
42
|
- **System One** (`POST …/v1/systemone`), the API Jev introduced: a `state`, then `noul` questions keyed by id, each with `instructions` and `criteria`. Clef on Workers AI speaks it, and so do most hosts and servers of decision models, such as [LiteLLM](https://docs.litellm.ai/docs/decisions)'s proxy (which routes to Jev, Clef, Perplexity, Databricks, Microsoft Foundry, vLLM, and others) and [LLM Gateway](https://llmgateway.io/blog/llm-decision-api).
|
|
36
43
|
- **OpenAI Decisions** (`POST …/v1/decisions`), used when the endpoint's path ends in `/decisions`: an `input`, then `predicate` questions. A predicate has no criteria, so each question's instructions are the item's text followed by `true: <criteria.true>` and `false: <criteria.false>` lines, as LiteLLM does when it sends criteria to OpenAI.
|
|
37
44
|
|
|
38
|
-
|
|
39
|
-
# Clef-flash on Workers AI, with the CLOUDFLARE_ credentials above
|
|
40
|
-
LINT_ANYTHING_MODEL=clef-flash
|
|
45
|
+
With no endpoint set, the model is a Cloudflare model on Workers AI, asked with the `CLOUDFLARE_` credentials: `clef` unless `LINT_ANYTHING_MODEL` names another.
|
|
41
46
|
|
|
47
|
+
```sh
|
|
42
48
|
# any System One server, such as a LiteLLM proxy
|
|
43
49
|
LINT_ANYTHING_ENDPOINT=http://localhost:4000/v1/systemone
|
|
44
50
|
LINT_ANYTHING_MODEL=typesafe/jev-latest
|
|
@@ -48,6 +54,9 @@ LINT_ANYTHING_API_KEY=sk-...
|
|
|
48
54
|
LINT_ANYTHING_ENDPOINT=https://api.openai.com/v1/decisions
|
|
49
55
|
LINT_ANYTHING_MODEL=gpt-6-luna-decisions
|
|
50
56
|
LINT_ANYTHING_API_KEY=sk-...
|
|
57
|
+
|
|
58
|
+
# Clef-flash on Workers AI, with the CLOUDFLARE_ credentials
|
|
59
|
+
LINT_ANYTHING_MODEL=clef-flash
|
|
51
60
|
```
|
|
52
61
|
|
|
53
62
|
- Every model is asked the same way: the same request text, at most 64 questions and 64,000 bytes per request, the same 0.5 / 0.65 cut-offs. A model whose context window is under 64K tokens may truncate a full request. Serve a local model with a context of at least that size.
|
|
@@ -55,7 +64,7 @@ LINT_ANYTHING_API_KEY=sk-...
|
|
|
55
64
|
- A question the model refuses (an OpenAI Decisions `refusal`) is `inconclusive`, with the reason `<model> declined to answer`. It isn't cached, so it is asked again next run.
|
|
56
65
|
- `--debug` prices tokens only for Clef. For other models, the cost column shows `-`.
|
|
57
66
|
- Errors name the model: `error: typesafe/jev-latest responded with HTTP 401: …`. A configuration that can't work stops the run (exit 2) before anything is sent: an endpoint without `LINT_ANYTHING_MODEL`, an endpoint that isn't an http(s) URL, `LINT_ANYTHING_API_KEY` without an endpoint, or a model name like `typesafe/jev-latest` without an endpoint (Workers AI only serves Cloudflare models by name).
|
|
58
|
-
- Every accuracy number in this README was measured on Clef. Clef-flash scored much worse on the same corpus (Evals, "Defaults of the removed flags"), so check another model on your own labels (`score.js`, below) or on the corpus (`bun evals/run.ts` with the same variables set) before trusting its verdicts.
|
|
67
|
+
- Every accuracy number in this README was measured on Clef, the model the evals were built with. Clef-flash scored much worse on the same corpus (Evals, "Defaults of the removed flags"), so check another model on your own labels (`score.js`, below) or on the corpus (`bun evals/run.ts` with the same variables set) before trusting its verdicts.
|
|
59
68
|
|
|
60
69
|
## Policies
|
|
61
70
|
|
|
@@ -119,7 +128,7 @@ An item counts as a violation when its lines overlap a labeled violation, and a
|
|
|
119
128
|
- A selector still running after 10 minutes is killed, with every process it started, and stops the run (exit 2).
|
|
120
129
|
- From the evals:
|
|
121
130
|
- Items must reach every place a violation can be: a selector whose items miss the violating lines can't find them.
|
|
122
|
-
- Context belongs in each item's text, and computed facts beat raw data: a TODO's age in days lets
|
|
131
|
+
- Context belongs in each item's text, and computed facts beat raw data: a TODO's age in days lets the model judge "older than 90 days" at 99% accuracy; its commit date and today's date do not. Context costs input tokens.
|
|
123
132
|
- Packing a policy's items as questions of one request matches one request per item and costs 2.4× less.
|
|
124
133
|
- An item's answer is cached by its text, so whatever the text holds decides when it is asked again. Line numbers in the text (as in the example above) re-ask every item below an edit in that file; text without them stays cached until the item itself changes. `path` and `lines` set the finding's location either way.
|
|
125
134
|
- Structural units (functions, classes, handlers, tests) are easiest to select with a syntax-aware tool such as [ast-grep](https://ast-grep.github.io/). The eval corpus's policies (`evals/corpus/policies/`) and their scripts (`evals/steps/`) are worked examples: a selector lists the files it judges (`rg --files -g '*.ts'`) and pipes their paths to a script that cuts out the units.
|
|
@@ -129,7 +138,7 @@ An item counts as a violation when its lines overlap a labeled violation, and a
|
|
|
129
138
|
|
|
130
139
|
Each policy's selector runs from the current directory, then the decision model judges the selected items. Up to 8 policies run at once, and at most 8 requests are in flight across the run. A request's text is the policy body; it carries one yes/no question per item, the item's text, with the policy's criteria. Each item gets its own finding. A policy whose selector emits nothing sends nothing. Findings are printed in policy order.
|
|
131
140
|
|
|
132
|
-
A request holds at most 64 questions
|
|
141
|
+
A request holds at most 64 questions and 64,000 bytes, whichever model is asked (Clef's limits; every decision API that documents a limit allows at least as many questions). Items that don't fit in one request are split across several. Each request is POSTed to the model's endpoint with its API token or key as a bearer token: on Workers AI, `https://api.cloudflare.com/client/v4/accounts/<account>/ai/run/@cf/cloudflare/<model>`; otherwise `LINT_ANYTHING_ENDPOINT` as given.
|
|
133
142
|
|
|
134
143
|
| Result | Verdict |
|
|
135
144
|
| --- | --- |
|
|
@@ -145,7 +154,7 @@ Answers are cached on disk in one SQLite database, `.lint-anything/cache/answers
|
|
|
145
154
|
|
|
146
155
|
A cached answer stands for 30 days; each run starts by deleting older answers, so their questions are asked again, answers follow updates to the model, and the cache doesn't grow without bound. A row can only hold a probability between 0 and 1. With 50,000 cached answers the database is 5.5 MB, and a run that takes every answer from it reads them in under half a second.
|
|
147
156
|
|
|
148
|
-
|
|
157
|
+
A model's answer to one question can shift a little with the other questions in its request (Clef's does; see Evals). A selector's items are sorted by path, line, and id before they are packed, so the same items always get the same answers; with the cache, an edit in one file also leaves the verdicts of every other file as they were.
|
|
149
158
|
|
|
150
159
|
## Output
|
|
151
160
|
|
|
@@ -358,7 +367,7 @@ Purpose: context from outside the file makes rules about time or history judgeab
|
|
|
358
367
|
| … and its computed age in days | **1.000** [+0.077, +0.175] | **0.009** [−0.152, −0.099] | **99%** [+16, +31 pts] | **97%** | **0%** | 98% / 0% | $0.044 |
|
|
359
368
|
|
|
360
369
|
- The purpose holds once the item states the fact the rule turns on. With the age in days, accuracy rises from 76% to 99% and false alarms from issue-less young TODOs vanish.
|
|
361
|
-
- Clef doesn't do the date arithmetic: given today's date and each commit date, it still flags 88% of young issue-less TODOs. Compute facts in the selector; don't ask
|
|
370
|
+
- Clef doesn't do the date arithmetic: given today's date and each commit date, it still flags 88% of young issue-less TODOs. Compute facts in the selector; don't ask the model to derive them.
|
|
362
371
|
|
|
363
372
|
#### Shared selector runs
|
|
364
373
|
|
package/dist/cli.js
CHANGED
|
@@ -270,6 +270,6 @@ false: ${t.criteria.false}`}),$H=Lt({answers:gn(Fo([Lt({type:hn("predicate"),nam
|
|
|
270
270
|
noul REAL NOT NULL CHECK (noul BETWEEN 0 AND 1),
|
|
271
271
|
answered INTEGER NOT NULL
|
|
272
272
|
) STRICT, WITHOUT ROWID`),n.run("CREATE INDEX IF NOT EXISTS answers_answered ON answers (answered)"),n.run("DELETE FROM answers WHERE answered <= ?",[t-OH]);let o=n.query("SELECT noul FROM answers WHERE key = ?"),s=n.query("INSERT OR REPLACE INTO answers (key, noul, answered) VALUES (?, ?, ?)"),a=(f,u)=>new Bun.CryptoHasher("sha256").update(`${r.cacheKey}\x00${f}\x00${JSON.stringify(C_(u))}`).digest(),c=n.transaction((f,u)=>{let l=Date.now();for(let{question:p,noul:m}of u)s.run(a(f,p),m,l)}),i=(f)=>new xc({message:`answer cache ${p_(e,d_)}: ${f instanceof Error?f.message:String(f)}`});return{get:(f,u)=>Fe({try:()=>u.map((l)=>o.get(a(f,l))?.noul),catch:i}),set:(f,u)=>Fe({try:()=>c(f,u),catch:i}),close:()=>n.close()}},Lu={requests:0,cached:0,inputTokens:0,unmetered:0},Qy=(e)=>{let t=Lu;for(let r of e)t={requests:t.requests+r.requests,cached:t.cached+r.cached,inputTokens:t.inputTokens+r.inputTokens,unmetered:t.unmetered+r.unmetered};return t},WH=(e,t)=>{if(sh(t))return y(`${e} did not respond within ${y_}, after ${E_} retries`);if(Jp(t)){let{reason:n}=t;if(n._tag!=="StatusCodeError"){let o="cause"in n&&n.cause instanceof Error?`: ${n.cause.message}`:"";return y(`${e} request failed: ${t.message}${o}`)}return n.response.text.pipe(bT("5 seconds"),M((o)=>o.trim().slice(0,500)),zR(()=>""),M((o)=>`${e} responded with HTTP ${n.response.status}${o===""?"":`: ${o}`}`))}let r=(t instanceof Error?t.message:String(t)).replace(/\s*\n\s*/g," ");return y(`${e} sent an unexpected response: ${r}`)};class so extends Ae()("lint-anything/DecisionModel"){static layer=(e)=>Qr(so,v(HH,(t)=>m_(t,e))).pipe(QC(f_));static layerFor=(e,t)=>Qr(so,m_(e,t)).pipe(QC(f_))}var m_=U(function*(e,t){let r=T_[e.format],n=(yield*n_).pipe(iH(Vs(e.token===void 0?(p)=>p:YB(e.token),KO)),sH,td(Vs(Xr((p)=>p.text),bT(y_))),uH({schedule:NR(500),times:E_})),o=t===void 0?void 0:yield*Jt(Fe({try:()=>qH(t,Date.now(),e),catch:(p)=>new xc({message:`cannot open the answer cache in ${t}: ${p instanceof Error?p.message:String(p)}`})}),(p)=>z(p.close)),s=yield*N$(vH),a=yield*K1(new Map),c=(p,m)=>G1(a,(g)=>new Map(g).set(p,Qy([g.get(p)??Lu,{...Lu,...m}]))),i=(p)=>new h_({bytes:p,message:e.name==="Clef"?`request is ${p} bytes; Clef would truncate it`:`request is ${p} bytes, over the ${Jy}-byte request limit`}),f=new g_({message:`${e.name} declined to answer`}),u=tn("DecisionModel.send")(function*(p,m){let g=yield*c(p,{requests:1}).pipe(Ft(ZB(e.url).pipe(rH(m.body,"application/json"),n.execute)),v((h)=>h.json),v(r.decode),s.withPermit,en((h)=>v(WH(e.name,h),(S)=>k(new xc({message:S})))));yield*c(p,g.inputTokens===void 0?{unmetered:1}:{inputTokens:g.inputTokens});let x=[];for(let{id:h,value:S}of m.questions){let b=g.answers.get(h);if(b===void 0)return yield*new xc({message:`${e.name} returned no answer for question ${h}`});x.push({value:S,answer:b})}return x}),l=tn("DecisionModel.ask")(function*(p,m,g){let x=[],h=[],S=o===void 0?[]:yield*o.get(m,g);if(g.forEach((w,O)=>{let E=S[O];if(E===void 0)h.push(O);else x[O]=me(E)}),h.length<g.length)yield*c(p,{cached:g.length-h.length});let{requests:b,oversized:T}=zH(m,h.map((w)=>g[w]),e);for(let{value:w,bytes:O}of T)x[h[w]]=le(i(O));return yield*mr(b,(w)=>u(p,w).pipe(Xr((O)=>{let E=[];for(let{value:C,answer:I}of O){let F=h[C];if(I==="declined")x[F]=le(f);else x[F]=me(I),E.push({question:g[F],noul:I})}return o===void 0?Q:o.set(m,E)})),{concurrency:"unbounded",discard:!0}),x});return so.of({target:e,ask:l,stats:V1(a)})});var k_=Lt({id:_s,text:_s,path:mn(_s),lines:mn(QU([Su,Su]).check(Eu(([e,t])=>1<=e&&e<=t||"lines must be [start, end] with 1 <= start <= end")))}),KH=mM(),A_=tn("parseItem")(function*(e){let t=yield*Fe({try:()=>JSON.parse(e),catch:()=>"not valid JSON"});if(typeof t!=="object"||t===null||Array.isArray(t))return yield*k("not an object");let r=Object.keys(t).filter((n)=>!(n in k_.fields));if(r.length>0)return yield*k(`unknown field ${r.join(", ")}; put what the model should read in text`);return yield*ec(k_)(t).pipe(Oe((n)=>KH(n.issue).issues.map(({path:o,message:s})=>`${o?.length?`${o.join(".")}: `:""}${s}`).join("; ")))});class rd extends et()("SelectorError",{message:re}){[rn]=2}var O_=10,__=(e)=>{try{process.kill(-e,"SIGKILL")}catch(t){let{code:r}=t;if(r!=="ESRCH"&&r!=="EPERM")throw t}},Yy=new Set,Xy=()=>{for(let e of Yy)__(e)};process.on("exit",Xy);var v_=async(e,t)=>{let r=Bun.spawn(["sh","-c",e],{cwd:t,env:Object.fromEntries(Object.entries(process.env).filter(([s])=>!s.startsWith("CLOUDFLARE_")&&!s.startsWith("LINT_ANYTHING_"))),stdin:"ignore",stdout:"pipe",stderr:"pipe",detached:!0});Yy.add(r.pid);let n=!1,o=setTimeout(()=>{n=!0,__(r.pid)},O_*60*1000);try{let[s,a,c]=await Promise.all([new Response(r.stdout).text(),new Response(r.stderr).text(),r.exited]);return{exitCode:c,timedOut:n,stdout:s,stderr:a}}finally{clearTimeout(o),Yy.delete(r.pid)}},P_=tn("runSelector")(function*(e,t,r){let n=yield*nl(()=>{if(r===void 0)return v_(e.select,t);let s=JSON.stringify([e.select,t??process.cwd()]),a=r.get(s);if(a===void 0)a=v_(e.select,t),r.set(s,a);return a});if(n.timedOut)return yield*new rd({message:`${e.name}: selector did not finish within ${O_} minutes and was killed`});if(n.exitCode!==0)return yield*new rd({message:`${e.name}: selector exited ${n.exitCode}: ${n.stderr.trim().slice(0,500)}`});let o=n.stdout.split(`
|
|
273
|
-
`);return yield*mr(o.flatMap((s,a)=>s.trim()===""?[]:[{line:s,number:a+1}]),({line:s,number:a})=>A_(s).pipe(Oe((c)=>new rd({message:`${e.name}: selector output line ${a} is not a valid item: ${c}`}))))});var VH=0.5,F_=0.65,GH=(e,t,r)=>e<=t?"pass":e>=r?"violation":"review",JH=(e,t)=>({instructions:t.text,criteria:e.criteria}),ZH=(e,t,r,n)=>ua(e,{onSuccess:(o)=>{let s=GH(o,VH,n);return{...t,verdict:s,violates:o,...s==="pass"?{}:{reason:r.criteria.true}}},onFailure:(o)=>({...t,verdict:"inconclusive",reason:o.message})}),QH=(e)=>({item:e.id,...e.path===void 0?{}:{path:e.path},...e.lines===void 0?{}:{lines:e.lines}}),YH=(e,t)=>(e.path??"").localeCompare(t.path??"")||(e.lines?.[0]??0)-(t.lines?.[0]??0)||e.id.localeCompare(t.id),XH=tn("select")(function*(e,t,r){return(yield*P_(e,t,r)).toSorted(YH)}),ej=tn("judge")(function*(e,t){return yield*(yield*so).ask(e.name,e.body,t.map((n)=>JH(e,n)))}),tj=(e,t,r,n)=>zP(t,r,(o,s)=>ZH(s,{policy:e.name,...QH(o)},e,n)),rj=8,R_=tn("lint")(function*(e,t){let r=new Map;return(yield*mr(e,(o)=>De(function*(){let s=yield*XH(o,void 0,r);return tj(o,s,yield*ej(o,s),t)}),{concurrency:rj})).flat()});class bn extends et()("PolicyError",{message:re}){[rn]=2}var $_=Lt({select:mn(_s),criteria:mn(Lt({true:mn(_s),false:mn(_s)}))}),nj=/^---\r?\n(.*?)\r?\n---(?:\r?\n|$)/s,oj=tn("parsePolicy")(function*(e,t){let r=nj.exec(t);if(r===null)return yield*new bn({message:`${e}: missing frontmatter`});let n=t.slice(r[0].length);if(n.trim()==="")return yield*new bn({message:`${e}: empty policy`});let o=yield*Fe(()=>Bun.YAML.parse(r[1]??"")).pipe(Oe((i)=>new bn({message:`${e}: invalid frontmatter: ${String(i.message??i)}`}))),s=Object.keys(typeof o==="object"&&o!==null?o:{}).filter((i)=>!(i in $_.fields));if(s.length>0)return yield*new bn({message:`${e}: unknown frontmatter ${s.join(", ")}; a policy has select and criteria, and its body is everything else the model reads`});let{select:a,criteria:c}=yield*ec($_)(o).pipe(Oe((i)=>new bn({message:`${e}: invalid frontmatter: ${i.message}`})));if(a===void 0)return yield*new bn({message:`${e}: frontmatter needs a select command that picks what the rule judges`});if(c?.true===void 0||c.false===void 0)return yield*new bn({message:`${e}: frontmatter needs criteria with "true" (what an item that violates the rule is) and "false" (what one that follows it is)`});return{name:e,body:n,select:a,criteria:{true:c.true,false:c.false}}}),M_=tn("loadPolicies")(function*(e){let t=yield*Ua,r=yield*Io,n=()=>new bn({message:`cannot read policies in ${e}`}),s=(yield*t.readDirectory(e,{recursive:!0}).pipe(Oe(n))).filter((a)=>a.endsWith(".md")).sort();if(s.length===0)return yield*new bn({message:`no policies in ${e}`});return yield*mr(s,(a)=>t.readFileString(r.join(e,a)).pipe(Oe(n),v((c)=>oj(a.slice(0,-3),c))))});var N_="0.5.
|
|
273
|
+
`);return yield*mr(o.flatMap((s,a)=>s.trim()===""?[]:[{line:s,number:a+1}]),({line:s,number:a})=>A_(s).pipe(Oe((c)=>new rd({message:`${e.name}: selector output line ${a} is not a valid item: ${c}`}))))});var VH=0.5,F_=0.65,GH=(e,t,r)=>e<=t?"pass":e>=r?"violation":"review",JH=(e,t)=>({instructions:t.text,criteria:e.criteria}),ZH=(e,t,r,n)=>ua(e,{onSuccess:(o)=>{let s=GH(o,VH,n);return{...t,verdict:s,violates:o,...s==="pass"?{}:{reason:r.criteria.true}}},onFailure:(o)=>({...t,verdict:"inconclusive",reason:o.message})}),QH=(e)=>({item:e.id,...e.path===void 0?{}:{path:e.path},...e.lines===void 0?{}:{lines:e.lines}}),YH=(e,t)=>(e.path??"").localeCompare(t.path??"")||(e.lines?.[0]??0)-(t.lines?.[0]??0)||e.id.localeCompare(t.id),XH=tn("select")(function*(e,t,r){return(yield*P_(e,t,r)).toSorted(YH)}),ej=tn("judge")(function*(e,t){return yield*(yield*so).ask(e.name,e.body,t.map((n)=>JH(e,n)))}),tj=(e,t,r,n)=>zP(t,r,(o,s)=>ZH(s,{policy:e.name,...QH(o)},e,n)),rj=8,R_=tn("lint")(function*(e,t){let r=new Map;return(yield*mr(e,(o)=>De(function*(){let s=yield*XH(o,void 0,r);return tj(o,s,yield*ej(o,s),t)}),{concurrency:rj})).flat()});class bn extends et()("PolicyError",{message:re}){[rn]=2}var $_=Lt({select:mn(_s),criteria:mn(Lt({true:mn(_s),false:mn(_s)}))}),nj=/^---\r?\n(.*?)\r?\n---(?:\r?\n|$)/s,oj=tn("parsePolicy")(function*(e,t){let r=nj.exec(t);if(r===null)return yield*new bn({message:`${e}: missing frontmatter`});let n=t.slice(r[0].length);if(n.trim()==="")return yield*new bn({message:`${e}: empty policy`});let o=yield*Fe(()=>Bun.YAML.parse(r[1]??"")).pipe(Oe((i)=>new bn({message:`${e}: invalid frontmatter: ${String(i.message??i)}`}))),s=Object.keys(typeof o==="object"&&o!==null?o:{}).filter((i)=>!(i in $_.fields));if(s.length>0)return yield*new bn({message:`${e}: unknown frontmatter ${s.join(", ")}; a policy has select and criteria, and its body is everything else the model reads`});let{select:a,criteria:c}=yield*ec($_)(o).pipe(Oe((i)=>new bn({message:`${e}: invalid frontmatter: ${i.message}`})));if(a===void 0)return yield*new bn({message:`${e}: frontmatter needs a select command that picks what the rule judges`});if(c?.true===void 0||c.false===void 0)return yield*new bn({message:`${e}: frontmatter needs criteria with "true" (what an item that violates the rule is) and "false" (what one that follows it is)`});return{name:e,body:n,select:a,criteria:{true:c.true,false:c.false}}}),M_=tn("loadPolicies")(function*(e){let t=yield*Ua,r=yield*Io,n=()=>new bn({message:`cannot read policies in ${e}`}),s=(yield*t.readDirectory(e,{recursive:!0}).pipe(Oe(n))).filter((a)=>a.endsWith(".md")).sort();if(s.length===0)return yield*new bn({message:`no policies in ${e}`});return yield*mr(s,(a)=>t.readFileString(r.join(e,a)).pipe(Oe(n),v((c)=>oj(a.slice(0,-3),c))))});var N_="0.5.1";process.once("SIGHUP",()=>{Xy(),process.kill(process.pid,"SIGHUP")});class U_ extends et()("ViolationsFound",{count:tc}){[rn]=1;[nn]=!1}class L_ extends et()("UsageError",{}){[rn]=2;[nn]=!1}class B_ extends et()("RunFailed",{}){[rn]=2;[nn]=!1}var aj=["PolicyError","SelectorError","DecisionModelError"],D_=".lint-anything/rules",cj=".lint-anything/cache",ij=["violation","review","inconclusive","pass"],uj=({path:e,item:t,lines:r})=>e!==void 0&&r!==void 0?`${e}:${r[0]}-${r[1]}`:t,fj=(e)=>{let t=e.filter((n)=>n.verdict!=="pass").map((n)=>[n.verdict.padEnd(12),(n.violates?.toFixed(2)??"-").padEnd(5),uj(n),n.policy,n.reason??""].join(" ").trimEnd()),r=ij.map((n)=>`${e.filter((o)=>o.verdict===n).length} ${n}`);return[...t,`${e.length} checks: ${r.join(", ")}`].join(`
|
|
274
274
|
`)},lj=(e,t,r)=>{let n=Qy(t.values()),o=[["policy","requests","cached","input tokens","cost (USD)"],...[...e.map((c)=>[c.name,t.get(c.name)??Lu]),["total",n]].map(([c,{requests:i,cached:f,inputTokens:u}])=>[c,String(i),String(f),String(u),r===void 0?"-":`$${(u*r).toFixed(4)}`])],s=o[0].map((c,i)=>Math.max(...o.map((f)=>f[i].length))),a=o.map((c)=>c.map((i,f)=>f===0?i.padEnd(s[f]):i.padStart(s[f])).join(" "));if(n.unmetered>0)a.push(`${n.unmetered} requests reported no token count; their cost is not included`);return a.join(`
|
|
275
275
|
`)},pj=UB("lint-anything",{debug:$u("debug").pipe(Mu(!1),Hs("Also print debugging details to stderr: each policy's requests, cached answers, input tokens, and cost"))},tn(function*({debug:e}){let t=yield*M_(D_),n=(yield*De(function*(){let o=yield*so,s=e?v(o.stats,(c)=>nc(lj(t,c,o.target.inputTokenPrice))):Q,a=yield*R_(t,F_).pipe(xr(()=>s));return yield*tt(fj(a)),yield*s,a}).pipe(Ph(so.layer(cj)))).filter((o)=>o.verdict==="violation").length;if(n>0)return yield*new U_({count:n})})).pipe(LB(`Check every policy in ${D_} with a decision model (Cloudflare Clef by default)`));pj.pipe(zB({version:N_}),Eo((e)=>_x(e)&&!(e._tag==="ShowHelp"&&e.errors.length===0),()=>k(new L_)),Eo((e)=>aj.includes(e._tag),(e)=>nc(`error: ${e.message}`).pipe(Ft(k(new B_)))),Ph(E1),GR);
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@andrueandersoncs/lint-anything",
|
|
3
|
-
"version": "0.5.
|
|
4
|
-
"description": "Check anything against plain-English policies, judged by
|
|
3
|
+
"version": "0.5.1",
|
|
4
|
+
"description": "Check anything against plain-English policies, judged by the decision model of your choice: any System One or OpenAI Decisions endpoint, or Workers AI",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"author": "Andrue Anderson",
|
|
7
7
|
"repository": {
|
|
@@ -1,11 +1,11 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: lint-anything
|
|
3
|
-
description: Run lint-anything, act on its findings, and write or tune its policies (plain-English rules in `.lint-anything/rules/*.md`, each with a shell selector, judged by
|
|
3
|
+
description: Run lint-anything, act on its findings, and write or tune its policies (plain-English rules in `.lint-anything/rules/*.md`, each with a shell selector, judged by the decision model the project configures: any System One or OpenAI Decisions endpoint, or Workers AI). Use when a project has a `.lint-anything/` directory, when asked to lint code against semantic or prose rules, to fix lint-anything findings, or to add, debug, or tune a rule.
|
|
4
4
|
---
|
|
5
5
|
|
|
6
6
|
# lint-anything
|
|
7
7
|
|
|
8
|
-
lint-anything checks a codebase against rules written in English. Each rule is a **policy**: a Markdown file whose shell **selector** cuts out the items to judge (functions, comments, commits, anything), and whose body and **criteria** tell a decision model what a violation is:
|
|
8
|
+
lint-anything checks a codebase against rules written in English. Each rule is a **policy**: a Markdown file whose shell **selector** cuts out the items to judge (functions, comments, commits, anything), and whose body and **criteria** tell a decision model what a violation is: whichever model the project configures. The model returns a violation probability for each item; the CLI turns it into a verdict.
|
|
9
9
|
|
|
10
10
|
It is a guide more than a gate: semantic rules are judged probabilistically. On the project's eval corpus, well-written policies catch about 69% of real violations outright, surface about 85% (counting `review`), and raise false alarms on about 13% of clean cases. Read findings critically; don't assume every `violation` is real or every `pass` is clean.
|
|
11
11
|
|
|
@@ -22,8 +22,9 @@ If `lint-anything` isn't found, install it with `bun add -d @andrueandersoncs/li
|
|
|
22
22
|
|
|
23
23
|
- **No arguments.** Every run checks every policy; you can't pass files, a rule name, a threshold, or an output format. To check less, change the selector. The only flag is `--debug`; `--help` and `--version` also exist.
|
|
24
24
|
- **The working directory matters.** Policies are read from `./.lint-anything/rules`, and selectors run in the current directory. Run from a subdirectory and you get `error: cannot read policies in .lint-anything/rules`: there is no upward search.
|
|
25
|
-
- **
|
|
26
|
-
- **
|
|
25
|
+
- **The decision model** is the project's choice, set in the environment or in `.env` / `.env.local` in the current directory. Don't change it to make findings go away. With `LINT_ANYTHING_ENDPOINT` (a full URL), the model named by `LINT_ANYTHING_MODEL` is asked there, and `LINT_ANYTHING_API_KEY` is its bearer token if it needs one. An endpoint whose path ends in `/decisions` speaks OpenAI's Decisions API; any other speaks System One (`/v1/systemone`: Jev, LiteLLM, LLM Gateway, and most decision-model servers).
|
|
26
|
+
- **Without an endpoint**, a Cloudflare model on Workers AI is asked: `clef`, or another that `LINT_ANYTHING_MODEL` names (`clef-flash`). `CLOUDFLARE_ACCOUNT_ID` and `CLOUDFLARE_AUTH_TOKEN` are then required on every run, even when every answer would come from the cache. `CLOUDFLARE_API_BASE_URL` is optional.
|
|
27
|
+
- **Accuracy numbers in this skill were measured on Clef.** A project on another model should score it on its labels before trusting it.
|
|
27
28
|
- **Add `.lint-anything/cache` to `.gitignore`.** The answer cache (`answers.sqlite` plus `-wal`/`-shm` files) is written there.
|
|
28
29
|
|
|
29
30
|
### Output
|
|
@@ -174,7 +175,7 @@ Selectors can select things other than files: `git log`, an API, a database. Use
|
|
|
174
175
|
What decides accuracy is what the item's text says (from the evals):
|
|
175
176
|
|
|
176
177
|
- **Items must reach every possible violation.** A violation that no item contains can't be found.
|
|
177
|
-
- **Compute facts; don't make the model derive them.** For "TODOs older than 90 days need an issue", giving the TODO's age in days scored 99% accuracy. Giving its commit date and today's date scored 84%, and the TODO alone 76%.
|
|
178
|
+
- **Compute facts; don't make the model derive them.** For "TODOs older than 90 days need an issue", giving the TODO's age in days scored 99% accuracy. Giving its commit date and today's date scored 84%, and the TODO alone 76%. Don't rely on the model for date math or counting (Clef got both wrong on the evals). Do it in the selector, with `git blame` or a script.
|
|
178
179
|
- **Add only the context the rule turns on**: the enclosing function for a magic number, the call sites for a flag parameter, whether a symbol is used for a public-surface rule. Context costs input tokens.
|
|
179
180
|
- **Line numbers in `text` reduce cache hits.** The cache is keyed on the text, so `"text": "src/a.ts:42: …"` gets asked again whenever a line above it is added or removed. Location is already in `path`/`lines`. Put the path in the text only if the rule needs it (test-file rules, say), and put line numbers in only if the rule needs them.
|
|
180
181
|
- **Selectors built for a rule beat generic ones.** If you reword a rule, check its selector still reaches what the new wording covers.
|
|
@@ -231,7 +232,7 @@ The lint-anything repository has worked examples: 21 policies in `evals/corpus/p
|
|
|
231
232
|
- Each answer is cached per question, keyed by model (and endpoint), policy body, item text, and criteria. Switching models never reuses another model's answers. Re-running with nothing changed sends no requests, and an edit re-asks only the items whose text changed. Editing a policy's body or criteria re-asks all of that policy's items.
|
|
232
233
|
- Answers expire after 30 days, so everything is asked again about once a month, and verdicts can shift slightly if the model has been updated.
|
|
233
234
|
- A machine without the cache (a fresh CI runner) pays for every item on every run, unless CI keeps `.lint-anything/cache` between runs.
|
|
234
|
-
-
|
|
235
|
+
- Cost depends on the model. `--debug` prices only Clef ($0.24 per million input tokens; output is free); on another model its cost column is `-`, and the project's provider bills it. On the evals, with Clef, rule-specific selectors cost about $14–16 per 1,000 files across 17 rules. Nothing caps spending: a selector that emits every line of a large repository sends all of them, so check the item count by hand first.
|
|
235
236
|
- Deleting `.lint-anything/cache` forces a fresh judgment of everything.
|
|
236
237
|
|
|
237
238
|
## Quick checklist for a new policy
|