llm-panel 0.1.0__tar.gz

llm_panel-0.1.0/.gitignore

@@ -0,0 +1,9 @@
+__pycache__/
+*.pyc
+
+# codex-consult conversation state -- private transcripts, never commit
+.codex-consult/
+
+# shallow clones for the checkout arm -- gigabytes, rebuildable with `aacr-upstream checkout`
+recall/benchmarks/checkouts/
+dist/

llm_panel-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Jaret Arnold
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

llm_panel-0.1.0/PKG-INFO

@@ -0,0 +1,364 @@
+Metadata-Version: 2.5
+Name: llm-panel
+Version: 0.1.0
+Summary: Put one question to several LLMs independently and read every answer in full -- parallel judges, failure classes, cost, and an anonymized rebuttal round
+Project-URL: Homepage, https://github.com/musharna/llm-panel
+Project-URL: Issues, https://github.com/musharna/llm-panel/issues
+Project-URL: Changelog, https://github.com/musharna/llm-panel/blob/main/CHANGELOG.md
+Author: Jaret Arnold
+License-Expression: MIT
+License-File: LICENSE
+Keywords: code-review,judge,llm,multi-model,ollama,openrouter,panel
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: POSIX
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Quality Assurance
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+
+# llm-panel
+
+[![controls](https://github.com/musharna/llm-panel/actions/workflows/controls.yml/badge.svg)](https://github.com/musharna/llm-panel/actions/workflows/controls.yml)
+[![license: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
+[![python: 3.11+](https://img.shields.io/badge/python-3.11%2B-blue.svg)](pyproject.toml)
+
+Put the same question to several models independently, then read every answer in full.
+
+Judges run in parallel, never see each other's work, and answer from their own reading of
+your repo. An optional second round shows each of them the others' findings — anonymised —
+and asks them to defend or withdraw. The output is a single self-contained HTML page where
+that second round is grouped **by the finding being argued about**, so comparing what five
+models said about one line of code doesn't mean holding five documents in your head.
+
+It is not a voting machine. A panel _generates candidate defects_; it does not establish
+truth by counting agreements. Every finding still has to be checked against the code — and
+the tool's other half, [`recall/`](#what-it-actually-catches), exists to measure what the
+panel misses rather than assert what it catches.
+
+```
+llm-panel --diff "Which of these changes is most likely to be wrong?"
+panel-report --open          # render the newest run and open it
+panel-triage --bad           # which runs went wrong, across every run root
+```
+
+A real run, with the waiting compressed: three judges asked in parallel (two free-tier,
+one on a ChatGPT plan) landing as they finish, the scoreboard from `panel.md`, and
+`panel-report` rendering it ([recording](https://github.com/musharna/llm-panel/blob/main/docs/demo.cast)):
+
+![terminal: llm-panel asks three judges in parallel, reports each as it lands, then head -n 12 panel.md shows the scoreboard and panel-report writes the HTML page](https://raw.githubusercontent.com/musharna/llm-panel/main/docs/demo.gif)
+
+The rendered report — the scoreboard counts spend and names who answered; the
+citation-overlap tables show where the panel's attention landed (three judges reviewing a
+[cline](https://github.com/cline/cline) PR, converging on one line of `TerminalProcess.ts`):
+
+![panel report: scoreboard, bench, and citation-overlap tables](https://raw.githubusercontent.com/musharna/llm-panel/main/docs/report.png)
+
+**Contents:** [What's here](#whats-here) · [Install](#install) ·
+[Configure your roster](#configure-your-roster) · [Using it](#using-it) ·
+[What it actually catches](#what-it-actually-catches) ·
+[On real PRs](#on-real-prs-aacr-bench) · [Tests](#tests) ·
+[Known limitations](#known-limitations)
+
+## What's here
+
+```mermaid
+%%{init: {"theme": "neutral", "flowchart": {"wrappingWidth": 320}}}%%
+flowchart TB
+  Q["question  (+ --diff, or stdin)"] --> LP["llm-panel"]
+  LP --> A["judge A"]
+  LP -- "in parallel<br/>blind to each other<br/>reading your repo" --> B["judge B"]
+  LP --> C["judge C"]
+  A & B & C --> RUN["run directory<br/>one .md and one .prompt.md per judge,<br/>panel.md, run.json"]
+  A & B & C -.-> RB
+  RB["round 2, only with --rebut<br/>each judge sees the others' findings,<br/>anonymised as Reviewer A / B / C,<br/>and defends or withdraws"] -.-> RUN
+  style RB stroke-dasharray: 6 4
+  RUN --> REP["panel-report<br/>one self-contained HTML page,<br/>rebuttals grouped by finding"]
+  RUN --> TRI["panel-triage<br/>the runs that failed"]
+```
+
+| tool                   | what it does                                                                           |
+| ---------------------- | -------------------------------------------------------------------------------------- |
+| `llm-panel`            | asks the judges, in parallel, and writes the run to disk                               |
+| `panel-report`         | renders a run as one self-contained HTML page, grouped by claim                        |
+| `panel-triage`         | finds the runs that _failed_, which a listing shows as ordinary rows                   |
+| `recall/panel-recall`  | measures what the panel **misses**, against a corpus of planted defects                |
+| `recall/aacr-upstream` | runs the panel over AACR-Bench PRs and hands the findings to **upstream's** evaluator  |
+| `recall/aacr-score`    | invokes that evaluator, and refuses to report a number from a judge that isn't running |
+| `claimlib.py`          | the one measurement boundary: reviews → span-grounded observations                     |
+| `*-controls`           | the regression suites — 810 controls, every one tied to a defect that shipped          |
+
+## Install
+
+Pure Python 3.11+ standard library. No dependencies, no build step. Each tool is one
+readable file, so either install route runs identical code:
+
+```sh
+# as a package (entry points: llm-panel, panel-report, panel-triage)
+uv tool install llm-panel        # or: pipx install llm-panel
+
+# or as the files themselves
+git clone https://github.com/musharna/llm-panel ~/llm-panel
+ln -s ~/llm-panel/{llm-panel,panel-report,panel-triage} ~/.local/bin/
+```
+
+3.11 is a hard floor (the link renderer uses atomic groups, added to `re` in 3.11);
+`panel-report` says so at startup rather than failing part-way through a render.
+
+Judges reach models through command-line tools you install separately — none are bundled,
+and you need at most one to start:
+
+| tool       | who it is                                    | billing                                                                       |
+| ---------- | -------------------------------------------- | ----------------------------------------------------------------------------- |
+| `codex`    | OpenAI's CLI                                 | a ChatGPT plan, not metered API                                               |
+| `opencode` | multi-provider CLI most judges route through | your OpenRouter / HuggingFace keys                                            |
+| `claude`   | Anthropic's CLI                              | a claude.ai subscription (setting `ANTHROPIC_API_KEY` switches it to metered) |
+| `ollama`   | local models                                 | free, and no tool loop — see the caveat below                                 |
+
+If none are present the panel still runs, fails loudly, exits 4, and tells you what to
+install. A missing tool is one judge's problem, never the whole panel's.
+
+## Configure your roster
+
+**The built-in judge list is a default, not a fixture — it names the author's accounts.**
+Yours will be different. Point the roster at models you actually have:
+
+Copy [`roster.example.json`](roster.example.json) to
+`~/.config/llm-panel/roster.json` (`$XDG_CONFIG_HOME` honoured; `$LLM_PANEL_CONFIG` wins).
+It is strict JSON — no comments, no trailing commas — and a malformed config is **fatal**
+and names the offending key, because quietly falling back to the built-in roster would run
+a panel you didn't ask for, and bill you for it:
+
+```json
+{
+  "default": ["codex", "nemotron", "glm", "kimi", "or-deepseek", "or-grok"],
+  "judges": {
+    "my-gpt": {
+      "transport": "opencode",
+      "model": "openrouter/openai/gpt-5.6",
+      "family": "OpenAI"
+    },
+    "big-pickle": null
+  }
+}
+```
+
+`null` drops a shipped judge. `default` is the panel run when `--judges` is absent.
+`llm-panel --list` shows the roster offline and marks config-defined judges.
+`llm-panel --check` actually pings each one. `llm-panel --help-config` prints this schema.
+
+### Picking judges
+
+**"One per vendor" is _not_ the answer.** It is tempting to
+treat vendor labels as a proxy for independent opinions. The evidence says they aren't:
+[Kohli 2026](https://arxiv.org/html/2605.29800) measured cross-family judge correlation at
+φ̄=0.389 against same-family 0.437 — barely different — with the three _most_ correlated pairs
+being cross-family, and found that restricting to one judge per family made effective
+independence **worse** (n_eff 1.93 vs 2.18). Family is display metadata here, not policy.
+
+The six-judge set above did score 6/6 against this corpus where a two-vendor panel scored
+4/6, but **treat that as debugging evidence, not as a result**: the roster was repaired
+_because_ of what happened on those very fixtures, so the comparison is in-sample, and the
+six defects live in only two files (effective n≈2, 95% CI 61–100%).
+
+What to actually do: pick judges by what they find on _your_ code, and use `panel-recall` to
+measure it. The quantity worth maximising is each judge's **marginal rescue rate** — how
+often it catches something every other judge on the roster missed — not how many logos are
+represented.
+
+Two practical constraints: wall-clock is the **slowest** judge, not the sum, so one slow
+model sets the pace for every run; and `claude-*` are deliberately absent from the default,
+because when Claude wrote the code under review a Claude judge shares the author's blind
+spots. Add it explicitly when that isn't the case — it is strong.
+
+## Using it
+
+- `--diff` attaches the working-tree diff, so nobody has to describe the change —
+  including you, who would describe it favourably.
+- `--rebut` adds the anonymised second round. Worth it whenever a finding would trigger
+  real work: the first run of it killed three confident findings that were simply wrong.
+  To be precise about the report's grouping of that round: it keys on the rebuttal letter
+  each finding is given (A1, B2 …), so it collects the _discussion_ around one judge's
+  finding. It is **not** semantic clustering — two judges independently raising the same
+  underlying defect stay two findings, and without `--rebut` there is no grouping at all.
+- `--judges a,b,c` overrides the default panel. `codex~2` runs the same model a second
+  time as a **full, separate judge** — its own file, its own letter, its own row.
+  Collapsing repeats would hide exactly the disagreement that makes them worth running.
+- `--thread NAME` keeps a persistent conversation per judge. For design questions, not review.
+- Long questions go via stdin: `llm-panel - <<'ASK' … ASK`.
+
+Judges reading through `codex`/`opencode`/`claude` can **read your repo**. `ollama` judges
+answer from the prompt alone with no tool loop, so they cannot verify a claim against code.
+Treat their findings accordingly.
+
+The rebuttal round as rendered — every position each judge took on each finding, grouped
+by the finding under dispute, disagreements marked CONTESTED. This run: four free-tier
+judges asked to review llm-panel's own failure-classification code; one failed and is
+reported as `harness`, the other three upheld 7 findings, rejected 4, and missed 6:
+
+![rebuttal round: positions grouped by the finding being argued about](https://raw.githubusercontent.com/musharna/llm-panel/main/docs/rebuttal.png)
+
+## What it actually catches
+
+`recall/panel-recall` is the part most tools like this don't have: a corpus of defects
+planted in real code, each one **proven to misbehave by execution**, so "the panel missed
+it" is a measurement rather than an impression.
+
+> **At least one of four independent passes (codex ×2 + claude-opus ×2) matched 25 of 27
+> known targets in this controlled, single-file Python corpus.** That is a keyword-matched
+> lower bound on an easy corpus — not an estimate of real-world code-review capability.
+> 95% CI 76.6–97.9%, and that is before accounting for defects clustering within fixtures.
+
+**Read that next to a real-world number.** [CR-Bench](https://arxiv.org/html/2603.11078v1)
+(Nutanix, 2026) builds review tasks from _real_ bugs `git blame`d out of merged PRs in
+django, sympy, astropy and scikit-learn, and reports GPT-5.2 + Reflexion at **32.8% recall
+and 5.1% precision**. The gap between that and 25/27 is the corpus, not the panel: hand-
+planted single-mechanism defects in ~40-line files are far easier than real defects in
+mature codebases, and the two numbers are not even the same estimand — different agents,
+different context, different definitions of a hit.
+
+So this corpus is a **development instrument**, good for controlled A/Bs where ground truth
+must be known and iteration must be cheap (the abstention experiment below is exactly that).
+It is not evidence of absolute capability, and no number from it should be quoted as one.
+
+Three results worth knowing before you trust any of the output:
+
+- **Recall was limited by the roster, not by the models.** The two defects that panel
+  never found — a `.get(k, default)` that doesn't apply to an explicit `null`, and a
+  corrupt cache file silently becoming empty — are both found by a **six-vendor** panel
+  (OpenAI / NVIDIA / Zhipu / Moonshot / DeepSeek / xAI): 4/6 → **6/6** on those two
+  fixtures. The best two judges there, at 4/6 each, beat codex at 2/6 — and both were
+  broken or out of credit until the roster was repaired. If your panel is missing things,
+  check who is actually answering before concluding the models can't see it.
+- **Running the same model twice recovered nothing.** First passes 25/27, with repeats
+  25/27. The repeat-passes idea is well supported in the literature and did not reproduce
+  here. An earlier grader bug reported +1 and it was an artifact. Adding a _different
+  vendor_ did what adding a second pass of the same one could not.
+- **Letting judges say "nothing is wrong here" is a precision/recall trade, not a free
+  win either way.** One sentence of abstention licence is the whole difference.
+
+  |             | findings/fixture | false positives       |
+  | ----------- | ---------------- | --------------------- |
+  | licence on  | 0.42             | 0 / 6 judges          |
+  | licence off | 2.17             | 2, from 1 of 5 judges |
+
+  Findings-per-fixture is measured on fixtures that _do_ contain defects, where the extra
+  findings were verified **true** — so the licence suppresses real findings (one judge went
+  3.00 → 0.00 on files with genuine defects). False positives are measured on
+  `p01-exhaustive-codec`, the one fixture with **proven** absence rather than verified
+  scope — which is what makes a false-positive rate computable at all. There, the same
+  judge on the same code abstained with the licence and produced two demonstrably false
+  findings without it (it claimed int and str subclasses were rejected; `encode(MyInt(1))`
+  returns `'A'`).
+
+  So: the licence costs true findings and prevents false ones. Which you want depends on
+  whether chasing a false lead costs you more than missing a real defect. Caveat worth
+  stating: one proven fixture, eleven reviews.
+
+## On real PRs (AACR-Bench)
+
+The planted corpus above is a development instrument; the real-world numbers come from
+running the panel over [AACR-Bench](https://github.com/alibaba/aacr-bench) PRs and
+scoring the findings with **upstream's own evaluator** — a real LLM judge doing
+path → line → semantic matching, so the numbers are theirs, not a self-graded matcher's.
+On 18 PRs at full roster (extractor-3 re-measurements, 2026-08-28):
+
+| `--prompt-style`   | semantic recall | precision | findings read per validated hit |
+| ------------------ | --------------- | --------- | ------------------------------- |
+| `defect` (default) | 12.2%           | 16.5%     | 6.1                             |
+| `broad`            | 26.0%           | 13.2%     | 7.6                             |
+| `volume`           | 25.2%           | 7.9%      | 12.6                            |
+
+<picture>
+  <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/musharna/llm-panel/main/docs/bench-dark.png">
+  <img alt="recall against precision for the three prompt styles; error bars are the ±2 pp re-run noise floor" src="https://raw.githubusercontent.com/musharna/llm-panel/main/docs/bench-light.png" width="660">
+</picture>
+
+`broad` — asking for what a careful maintainer would actually raise — doubles the
+default's recall (McNemar on paired references, p = 0.0005). But the `volume` control
+shows what that class of gain is made of: it is the `defect` prompt plus one
+exhaustiveness clause, reaches the same recall (p = 1.0 vs broad), and pays for it with
+half of broad's precision. On a 35-PR replication the ordering holds on both transports
+while every arm's precision falls (broad ~9.7%, volume ~5.5–6.1%, ~16–18 findings read
+per hit). A declared cost cut over all of it settled the product default: **it stays
+`defect`**; the only candidate for a future default change is `broad`
+(`recall/benchmarks/cost-cut/README.md`).
+
+What keeps these numbers honest:
+
+- **The variance floor is measured.** Re-running the same judge on the same 35 PRs moves
+  up to ±3 human-reference matches of 150, with an evaluator replicate at exactly zero —
+  so effects under ~5–7 pp of recall are re-run noise at this n, which every subgroup
+  claim so far was (`recall/benchmarks/results-human-2arm-orgpt/perjudge35/`).
+- **Three earlier readings were withdrawn on re-measurement**: a DEFECT/IMPROVEMENT
+  split (my classifier was circular), "broad finds different hits" (pre-registered
+  replication on 35 fresh PRs, p = 0.40), and a transport/harness effect (its 13-PR
+  foothold did not survive a re-run; a same-transport re-run of another judge moved as
+  much). The audit trail is in the benchmark READMEs; nothing above rests on a withdrawn
+  claim.
+- **Diff-in-prompt review is the measured condition** — each panel runs in an empty
+  directory with the diff in the prompt. The paired repo-checkout arm moves recall
+  12.2% → 15.4% (p = 0.48) while _losing_ 7 of the diff arm's matches and gaining 11:
+  repo access changes what judges attend to more than it strictly adds
+  (`recall/benchmarks/results-checkout-3judge/README.md`).
+- **Location agreement overstates semantic agreement ~2x** (22.8% of references had a
+  finding at the right file and line; 12.2% had one a judge called the same concern) —
+  which is why scoring is delegated upstream instead of done by a local matcher.
+- **A degraded roster costs about half the recall** (6.5% vs 12.2% with one judge's
+  quota spent and a 300s timeout, same extractor). Check who actually answered before
+  reading any number.
+- **The panel does not discriminate accepted from rejected reviewer comments**
+  (12.2% vs 11.1%, Fisher p = 1.0).
+- **Unlocated findings are withheld from upstream, not handed over empty** — upstream's
+  filters treat a missing path or line as match-everything, and passing them through
+  inflated line matches from 20 to 50 on the first scoring run.
+
+Full run ledgers: `recall/benchmarks/results-human-2arm/README.md` and
+`results-human-2arm-orgpt/README.md`; data licensing in
+`recall/benchmarks/PROVENANCE.md`.
+
+## Tests
+
+```sh
+./claimlib-controls              #  83
+./llm-panel-controls             # 279
+./panel-report-controls          # 310
+./panel-triage-controls          #  15
+./recall/aacr-upstream-controls  #  96
+./recall/aacr-recut-controls     #  27
+cd recall && ./panel-recall selftest && python3 validate_corpus.py
+```
+
+CI runs all six suites on every push (Python 3.11 and 3.13).
+
+Every control corresponds to a defect that **shipped**, and each asserts the fixed
+behaviour _and_ — where the pre-fix input is representable — that the broken version would
+have failed on it. An assertion that passes on both the broken and the fixed code tells you
+nothing.
+
+## Known limitations
+
+- **The judge roster's shipped defaults will not work for you** until you configure it.
+- **Judges can read the working tree.** `--diff` sends untracked file contents to remote
+  APIs. Don't point it at a repo holding secrets you haven't gitignored.
+- **A panel is not a jury.** Independent models generate candidates; verification against
+  code, tests, and execution is still yours to do.
+- **Recall is measured on a 27-defect Python corpus.** That number does not transfer to
+  other languages or to defect classes the corpus doesn't contain.
+- **The headline recall numbers are prompt- and condition-specific.** They move with
+  `--prompt-style`, roster health, and diff-vs-checkout context — see
+  [On real PRs](#on-real-prs-aacr-bench) before quoting any of them.
+- **Every other fixture has verified _scope_, not proven absence.** Their known unplanted
+  defects are recorded in each `truth.json` and re-checked by execution in
+  `validate_corpus.py`, so a judge that finds one is not scored as wrong. Anything not yet
+  recorded still depresses the recall floor by making a true finding look like noise.
+
+## License
+
+MIT — see [LICENSE](LICENSE). The MIT grant covers the code in this repository; the
+benchmark data under `recall/benchmarks/` contains third-party material (PR diffs and
+review-comment text) that stays under its upstream terms — see
+[recall/benchmarks/PROVENANCE.md](recall/benchmarks/PROVENANCE.md).