@applesnort/crosscheck 0.2.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Joel Mangin
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/PROVENANCE.md

@@ -0,0 +1,57 @@
+# Provenance
+
+This file records where the content in this repository came from, so that anyone
+reusing it can see what it derives from.
+
+## The lens definitions
+
+`lenses/architect.md`, `lenses/check.md`, `lenses/security-check.md`, and
+`lenses/ux.md` were written from scratch on 2026-08-05.
+
+They were composed from the stated purpose of each lens and from the named public
+standards cited inside each file — OWASP Top 10 (2021) and ASVS for
+`security-check`, Nielsen's ten usability heuristics for `ux`, and explicit
+first-party criteria for `architect` and `check`. Citing a standard by identifier
+is not reuse of its text; no text from any standard is reproduced here.
+
+Earlier, personal versions of these four lenses existed in the author's local
+tooling. Their origin could not be established from available records, so they were
+deliberately **not** consulted while writing the versions in this repository, and
+no text from them carries forward. That is the reason these files exist as a fresh
+composition rather than as a cleanup of the originals.
+
+## The foreman and the library
+
+`foreman.md` describes the dispatch, verification, merge, and reporting method. It
+originates with the author, developed through practical use across 2026. It is not
+derived from a third-party source.
+
+`lib/`, `bin/`, `test/`, and `fixtures/` were written from scratch on 2026-08-05
+and have no third-party origin. They carry no dependencies, so nothing is
+vendored, and no code was adapted from another project.
+
+`lib/sarif.mjs` targets SARIF 2.1.0 as specified by OASIS. It implements the
+format against the published specification; no text or code from the
+specification is reproduced here. The two panel-specific additions —
+`properties.consensusScore` and the `crosscheck/*` notification descriptors — are
+this project's own, placed in the extension points the format provides for that
+purpose.
+
+The general idea of using multiple critic personas to review code is **not**
+original to this project and is not claimed as such. Comparable prior work
+includes Claude Code's built-in parallel code review, community multi-agent review
+panels, and the long-standing static-analysis practice of aggregating several
+tools and treating agreement between them as a confidence signal. What this
+repository offers is a specific, documented method — not a novel concept.
+
+## Deliberately not included
+
+- **An accessibility lens.** The author's personal version derives from the
+  GOV.UK accessibility personas (Ashleigh, Claudia, Ron), which are published by
+  the Government Digital Service under the Open Government Licence v3.0. Reuse is
+  permitted with attribution; rather than carry that obligation here, the lens is
+  omitted. Anyone adding one should use the GOV.UK personas directly and attribute
+  them under OGL v3.
+- **Domain-specific lenses.** Lenses encoding a particular product's domain, or
+  a particular named colleague's review preferences, are project-local by nature
+  and are not published.

package/README.md

@@ -0,0 +1,244 @@
+# crosscheck
+
+Run several independent review lenses over the same change in parallel, merge their
+findings into one deduped report, and emit it as **SARIF** — so LLM review findings
+land in the same places static-analysis findings already do.
+
+```bash
+# run a panel: crosscheck builds the prompts and merges the results,
+# your --exec command supplies the model
+npx @applesnort/crosscheck run lib/ --exec 'claude -p' --sarif panel.sarif
+
+# or merge output a panel already produced
+npx @applesnort/crosscheck report --in run.json
+```
+
+**crosscheck never talks to a model itself.** `--exec` names any command that takes
+one lens prompt on stdin and returns findings on stdout — `claude -p`, `llm -m ...`,
+or your own wrapper. crosscheck owns prompt construction, routing, fan-out, dedupe,
+and output; you own the model. `--dry-run` prints the roster and prompts without
+spawning anything.
+
+No dependencies, no install step, 153 tests.
+
+> **v0.x — the API is unstable.** The CLI commands and the `lib/` exports may
+> change shape before 1.0. Pin an exact version if you depend on it.
+>
+> Published as `@applesnort/crosscheck`; npm rejects the unscoped name as too
+> similar to the (abandoned) `cross-check`. Installed, the command is
+> `crosscheck`.
+
+## SARIF output
+
+Findings are emitted as [SARIF 2.1.0](https://docs.oasis-open.org/sarif/sarif/v2.1.0/sarif-v2.1.0.html),
+the OASIS interchange format static analyzers already speak. Every other
+multi-persona review panel emits prose for a human to read once. This one produces
+a document GitHub code scanning, editor problem panels, and security dashboards
+already know how to ingest — none of them needing to know a model wrote it.
+
+- One **rule per lens**, carrying the standards that lens cites, so consumers can
+  filter and configure by lens.
+- `partialFingerprints` for stable identity across runs, independent of severity,
+  fix text, and ordering.
+- Lens attribution and consensus score ride in `properties`, since SARIF has no
+  native concept for either.
+- A lens that died sets `executionSuccessful: false` and emits a
+  `toolExecutionNotification`. The gap is machine-readable, not a line of prose a
+  dashboard will drop.
+
+That last point is the design rule throughout: **every omission is disclosed.** A
+lens skipped for irrelevance, a lens whose agent died, and a finding refuted during
+verification are three different things and must read differently. A partial panel
+presented as a complete one is worse than no panel.
+
+## Why lenses instead of one review pass
+
+A single review pass optimizes for one kind of defect at a time. Ask for "problems"
+and you get whichever category the model reaches for first. Ask several narrow
+specialists — each told explicitly what it does *not* own — and the union covers
+more ground, because none of them is trading correctness findings against usability
+findings inside one context.
+
+The panel is a foreman, not a reviewer. It resolves the target, decides which lenses
+are relevant, dispatches them, and synthesizes what comes back. The lenses do the
+looking; the deterministic half — parse, dedupe, score, emit — is code with tests
+rather than prompt instructions. See [`foreman.md`](foreman.md).
+
+## Measuring whether your lenses are redundant
+
+Most implementations stop at "N agents agreed." That over-credits lenses whose
+remits overlap: two lenses looking at the same things agreeing tells you less than
+two that do not. So overlap is **measured** from a real run rather than assumed:
+
+```bash
+crosscheck overlap --in run.json --out overlap.json   # measure it
+crosscheck report  --in run.json --overlap overlap.json
+```
+
+This has produced discriminating results in both directions on real data. The same
+two lenses — written with deliberately opposed methods, a CWE taxonomy walk and
+sink-first flow tracing — measured **0.45** overlap against 66 OWASP Benchmark
+cases and **1.0** against a 20-case corpus where they returned byte-for-byte
+identical detections.
+
+At 1.0, agreement is scored as **one** effective confirmation rather than two,
+because that is what it is worth. That is a result you can act on: drop the
+redundant lens, or replace it with one that fails differently.
+
+Findings are ranked by those effective confirmations — 1 for a single lens, and for
+a set, 1 plus the summed independence of each distinct pair. **Whether that ranking
+predicts correctness is unproven**; six calibration rounds could not test it,
+because the lenses almost never erred. It is kept as a documented hypothesis rather
+than a validated feature — [the full record is here](fixtures/calibration/PREREGISTERED.md),
+including every failed prediction.
+
+### Matching is fuzzy, because real lens output is
+
+Two lenses never phrase a defect identically, and they anchor it on different lines.
+In one measured run, three lenses each found the same swallowed error and cited it
+at lines **54, 56, and 57**. An exact-match merge reports that as three findings and
+zero agreement.
+
+So findings cluster on line proximity (±3) **plus** issue similarity — a Jaccard
+index over content words, thresholded at `0.12`. Both defaults come from
+measurement: same-defect pairs scored `0.161`–`0.538`, while different defects
+sharing a line scored `0.038`–`0.050`. The threshold sits in that gap.
+`test/merge-realdata.test.mjs` pins both bands using verbatim lens output, so they
+cannot drift unnoticed.
+
+## Baselines
+
+On an existing codebase the first run returns everything already wrong, and the
+report gets closed unread. Record it, then report only what changed:
+
+```bash
+crosscheck baseline --in first-run.json --out .crosscheck-baseline.json
+crosscheck report   --in run.json --baseline .crosscheck-baseline.json
+```
+
+Suppressed counts are always reported, and baseline entries that stopped appearing
+are flagged — either they were fixed, or a lens quietly stopped running. A baseline
+that hides its own size is just a way to declare a codebase's problems normal.
+
+## Calibration
+
+A review panel is otherwise unfalsifiable: you cannot tell whether it works, whether
+a new lens helped, or whether a prompt edit made it worse.
+
+```bash
+crosscheck calibrate --in run.json --expected fixtures/calibration/expected.json
+```
+
+reports recall, precision, per-lens recall against only the defects that lens owns,
+severity agreement, and consensus precision beside single-lens precision. It exits
+non-zero when a planted defect was missed, so it works as a CI gate on the panel
+itself. `lib/corpus.mjs` scores externally authored corpora at case level, and
+`scripts/fetch-corpus.sh` pulls one without vendoring it.
+
+Six rounds have been run and recorded — two self-authored fixtures, an external
+corpus of 2,740 labeled cases, two model tiers, a purpose-built deception corpus,
+and a controlled prompt ablation. The results, the criteria fixed in advance of each
+round, every failed prediction, and one methodological error that voided a round are
+all in [`fixtures/calibration/PREREGISTERED.md`](fixtures/calibration/PREREGISTERED.md).
+
+The short version: the lenses were nearly always right, which is why the consensus
+claim above remains untested. The one false positive across all six rounds was
+produced by **both** lenses at once — both had to resolve a single opaque helper and
+both inferred its behaviour from its name. Independence of *method* does not give
+independence of *failure*.
+
+## What's here
+
+```
+foreman.md              the dispatch / verify / merge / report method
+lenses/
+  architect.md          structure, data shape, reversibility
+  check.md              correctness — boundaries, absent values, error paths
+  security-check.md     OWASP-framed application security
+  taint.md              sink-first data flow to a dangerous operation
+  ux.md                 usability under interruption and extreme states
+lib/
+  parse.mjs             lens text -> findings
+  merge.mjs             normalize, dedupe, consensus scoring
+  sarif.mjs             SARIF 2.1.0 writer
+  baseline.mjs          baseline record / filter / staleness
+  lenses.mjs            frontmatter, glob routing, roster validation
+  calibrate.mjs         score a run against planted defects
+  corpus.mjs            external corpora, case-level scoring
+lib/
+  prompt.mjs            lens prompt construction
+  run.mjs               roster planning and bounded fan-out
+bin/crosscheck.mjs      CLI: run | report | sarif | baseline | overlap | calibrate
+fixtures/calibration/   planted defects, ground truth, and the calibration record
+fixtures/deception/     20 modules that look safe and are not, or the reverse
+PROVENANCE.md           where all of this came from
+```
+
+Every lens shares one output contract, so the merge needs no per-lens parsing:
+
+```
+file:line — SEVERITY — issue — fix
+```
+
+`SEVERITY` is `BLOCK`, `FIX`, or `CONSIDER`; other vocabularies normalize onto it. A
+lens with nothing to say returns exactly `NO FINDINGS` — deliberately distinct from
+a lens that failed to run. Lines that do not match the contract are reported as
+unparsed rather than discarded, because a lens that starts narrating instead of
+reporting should not look like a clean one.
+
+Routing is declared in each lens's frontmatter (`when`, `owns`, `not-owns`), so a
+roster validates before any agent is dispatched. `not-owns` is required: a lens that
+never declines dilutes the signal everything else depends on.
+
+> **Note:** `fixtures/deception/src/` contains deliberately exploitable code and
+> helpers that behave differently from what their names promise. It exists to test
+> reviewers. Do not copy any of it into real software.
+
+## Using it
+
+The lenses are plain markdown prompts — nothing is tied to a particular agent
+framework. Any harness that can run N prompts concurrently and collect their text
+can drive this; feed the results in as `[{"lens": "check", "output": "..."}]`, with
+`null` for a lens that died.
+
+`crosscheck run` does this for you, bounded by `--concurrency`, and writes the raw
+lens text with `--out` so a run can be rescored later without paying the model
+again. Routing comes from each lens's `when` globs; `--only` and `--skip` override
+it, and every skip is reported with its reason.
+
+One property worth preserving if you build your own dispatcher: **dispatch out of
+band.** A parallel fan-out that renders inline floods the session you are working
+in and has to be killed to recover it.
+
+```bash
+npm test   # 153 tests, no dependencies
+```
+
+## Adding a lens
+
+A lens earns its place by finding what the others miss. Give it a remit narrow
+enough that it declines most changes, state what it does *not* own, and make it name
+the concrete trigger for every finding. Then measure it: add defects it should catch
+to the calibration fixture and check whether recall actually moved, and check the
+overlap figure to see whether it is telling you anything the existing lenses were
+not.
+
+If a lens encodes your product's domain, your storage conventions, or a particular
+reviewer's standards, keep it in your own project. The ones here are deliberately
+generic; the useful ones usually aren't.
+
+## Prior art
+
+Multiple critic personas reviewing code is not a new idea and is not claimed as one.
+Claude Code ships a parallel multi-agent code review; community multi-agent review
+panels exist; aggregating several analyzers and weighting their agreement is
+long-standing practice, formalized in SARIF.
+
+What is offered here is narrower: LLM review lenses as a **SARIF producer**, with
+lens redundancy measured rather than assumed, and a calibration harness that reports
+what it cannot establish as readily as what it can. See
+[`PROVENANCE.md`](PROVENANCE.md).
+
+## License
+
+MIT — see [`LICENSE`](LICENSE).