evalrx 0.1.2__tar.gz

evalrx-0.1.2/LICENSE

@@ -0,0 +1,121 @@
+Creative Commons Legal Code
+
+CC0 1.0 Universal
+
+    CREATIVE COMMONS CORPORATION IS NOT A LAW FIRM AND DOES NOT PROVIDE
+    LEGAL SERVICES. DISTRIBUTION OF THIS DOCUMENT DOES NOT CREATE AN
+    ATTORNEY-CLIENT RELATIONSHIP. CREATIVE COMMONS PROVIDES THIS
+    INFORMATION ON AN "AS-IS" BASIS. CREATIVE COMMONS MAKES NO WARRANTIES
+    REGARDING THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS
+    PROVIDED HEREUNDER, AND DISCLAIMS LIABILITY FOR DAMAGES RESULTING FROM
+    THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS PROVIDED
+    HEREUNDER.
+
+Statement of Purpose
+
+The laws of most jurisdictions throughout the world automatically confer
+exclusive Copyright and Related Rights (defined below) upon the creator
+and subsequent owner(s) (each and all, an "owner") of an original work of
+authorship and/or a database (each, a "Work").
+
+Certain owners wish to permanently relinquish those rights to a Work for
+the purpose of contributing to a commons of creative, cultural and
+scientific works ("Commons") that the public can reliably and without fear
+of later claims of infringement build upon, modify, incorporate in other
+works, reuse and redistribute as freely as possible in any form whatsoever
+and for any purposes, including without limitation commercial purposes.
+These owners may contribute to the Commons to promote the ideal of a free
+culture and the further production of creative, cultural and scientific
+works, or to gain reputation or greater distribution for their Work in
+part through the use and efforts of others.
+
+For these and/or other purposes and motivations, and without any
+expectation of additional consideration or compensation, the person
+associating CC0 with a Work (the "Affirmer"), to the extent that he or she
+is an owner of Copyright and Related Rights in the Work, voluntarily
+elects to apply CC0 to the Work and publicly distribute the Work under its
+terms, with knowledge of his or her Copyright and Related Rights in the
+Work and the meaning and intended legal effect of CC0 on those rights.
+
+1. Copyright and Related Rights. A Work made available under CC0 may be
+protected by copyright and related or neighboring rights ("Copyright and
+Related Rights"). Copyright and Related Rights include, but are not
+limited to, the following:
+
+  i. the right to reproduce, adapt, distribute, perform, display,
+     communicate, and translate a Work;
+ ii. moral rights retained by the original author(s) and/or performer(s);
+iii. publicity and privacy rights pertaining to a person's image or
+     likeness depicted in a Work;
+ iv. rights protecting against unfair competition in regards to a Work,
+     subject to the limitations in paragraph 4(a), below;
+  v. rights protecting the extraction, dissemination, use and reuse of data
+     in a Work;
+ vi. database rights (such as those arising under Directive 96/9/EC of the
+     European Parliament and of the Council of 11 March 1996 on the legal
+     protection of databases, and under any national implementation
+     thereof, including any amended or successor version of such
+     directive); and
+vii. other similar, equivalent or corresponding rights throughout the
+     world based on applicable law or treaty, and any national
+     implementations thereof.
+
+2. Waiver. To the greatest extent permitted by, but not in contravention
+of, applicable law, Affirmer hereby overtly, fully, permanently,
+irrevocably and unconditionally waives, abandons, and surrenders all of
+Affirmer's Copyright and Related Rights and associated claims and causes
+of action, whether now known or unknown (including existing as well as
+future claims and causes of action), in the Work (i) in all territories
+worldwide, (ii) for the maximum duration provided by applicable law or
+treaty (including future time extensions), (iii) in any current or future
+medium and for any number of copies, and (iv) for any purpose whatsoever,
+including without limitation commercial, advertising or promotional
+purposes (the "Waiver"). Affirmer makes the Waiver for the benefit of each
+member of the public at large and to the detriment of Affirmer's heirs and
+successors, fully intending that such Waiver shall not be subject to
+revocation, rescission, cancellation, termination, or any other legal or
+equitable action to disrupt the quiet enjoyment of the Work by the public
+as contemplated by Affirmer's express Statement of Purpose.
+
+3. Public License Fallback. Should any part of the Waiver for any reason
+be judged legally invalid or ineffective under applicable law, then the
+Waiver shall be preserved to the maximum extent permitted taking into
+account Affirmer's express Statement of Purpose. In addition, to the
+extent the Waiver is so judged Affirmer hereby grants to each affected
+person a royalty-free, non transferable, non sublicensable, non exclusive,
+irrevocable and unconditional license to exercise Affirmer's Copyright and
+Related Rights in the Work (i) in all territories worldwide, (ii) for the
+maximum duration provided by applicable law or treaty (including future
+time extensions), (iii) in any current or future medium and for any number
+of copies, and (iv) for any purpose whatsoever, including without
+limitation commercial, advertising or promotional purposes (the
+"License"). The License shall be deemed effective as of the date CC0 was
+applied by Affirmer to the Work. Should any part of the License for any
+reason be judged legally invalid or ineffective under applicable law, such
+partial invalidity or ineffectiveness shall not invalidate the remainder
+of the License, and in such case Affirmer hereby affirms that he or she
+will not (i) exercise any of his or her remaining Copyright and Related
+Rights in the Work or (ii) assert any associated claims and causes of
+action with respect to the Work, in either case contrary to Affirmer's
+express Statement of Purpose.
+
+4. Limitations and Disclaimers.
+
+ a. No trademark or patent rights held by Affirmer are waived, abandoned,
+    surrendered, licensed or otherwise affected by this document.
+ b. Affirmer offers the Work as-is and makes no representations or
+    warranties of any kind concerning the Work, express, implied,
+    statutory or otherwise, including without limitation warranties of
+    title, merchantability, fitness for a particular purpose, non
+    infringement, or the absence of latent or other defects, accuracy, or
+    the present or absence of errors, whether or not discoverable, all to
+    the greatest extent permissible under applicable law.
+ c. Affirmer disclaims responsibility for clearing rights of other persons
+    that may apply to the Work or any use thereof, including without
+    limitation any person's Copyright and Related Rights in the Work.
+    Further, Affirmer disclaims responsibility for obtaining any necessary
+    consents, permissions or other rights required for any use of the
+    Work.
+ d. Affirmer understands and acknowledges that Creative Commons is not a
+    party to this document and has no duty or obligation with respect to
+    this CC0 or use of the Work.

evalrx-0.1.2/PKG-INFO

@@ -0,0 +1,532 @@
+Metadata-Version: 2.4
+Name: evalrx
+Version: 0.1.2
+Summary: Failure case analysis in the era of LLM.
+License: Creative Commons Legal Code
+        
+        CC0 1.0 Universal
+        
+            CREATIVE COMMONS CORPORATION IS NOT A LAW FIRM AND DOES NOT PROVIDE
+            LEGAL SERVICES. DISTRIBUTION OF THIS DOCUMENT DOES NOT CREATE AN
+            ATTORNEY-CLIENT RELATIONSHIP. CREATIVE COMMONS PROVIDES THIS
+            INFORMATION ON AN "AS-IS" BASIS. CREATIVE COMMONS MAKES NO WARRANTIES
+            REGARDING THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS
+            PROVIDED HEREUNDER, AND DISCLAIMS LIABILITY FOR DAMAGES RESULTING FROM
+            THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS PROVIDED
+            HEREUNDER.
+        
+        Statement of Purpose
+        
+        The laws of most jurisdictions throughout the world automatically confer
+        exclusive Copyright and Related Rights (defined below) upon the creator
+        and subsequent owner(s) (each and all, an "owner") of an original work of
+        authorship and/or a database (each, a "Work").
+        
+        Certain owners wish to permanently relinquish those rights to a Work for
+        the purpose of contributing to a commons of creative, cultural and
+        scientific works ("Commons") that the public can reliably and without fear
+        of later claims of infringement build upon, modify, incorporate in other
+        works, reuse and redistribute as freely as possible in any form whatsoever
+        and for any purposes, including without limitation commercial purposes.
+        These owners may contribute to the Commons to promote the ideal of a free
+        culture and the further production of creative, cultural and scientific
+        works, or to gain reputation or greater distribution for their Work in
+        part through the use and efforts of others.
+        
+        For these and/or other purposes and motivations, and without any
+        expectation of additional consideration or compensation, the person
+        associating CC0 with a Work (the "Affirmer"), to the extent that he or she
+        is an owner of Copyright and Related Rights in the Work, voluntarily
+        elects to apply CC0 to the Work and publicly distribute the Work under its
+        terms, with knowledge of his or her Copyright and Related Rights in the
+        Work and the meaning and intended legal effect of CC0 on those rights.
+        
+        1. Copyright and Related Rights. A Work made available under CC0 may be
+        protected by copyright and related or neighboring rights ("Copyright and
+        Related Rights"). Copyright and Related Rights include, but are not
+        limited to, the following:
+        
+          i. the right to reproduce, adapt, distribute, perform, display,
+             communicate, and translate a Work;
+         ii. moral rights retained by the original author(s) and/or performer(s);
+        iii. publicity and privacy rights pertaining to a person's image or
+             likeness depicted in a Work;
+         iv. rights protecting against unfair competition in regards to a Work,
+             subject to the limitations in paragraph 4(a), below;
+          v. rights protecting the extraction, dissemination, use and reuse of data
+             in a Work;
+         vi. database rights (such as those arising under Directive 96/9/EC of the
+             European Parliament and of the Council of 11 March 1996 on the legal
+             protection of databases, and under any national implementation
+             thereof, including any amended or successor version of such
+             directive); and
+        vii. other similar, equivalent or corresponding rights throughout the
+             world based on applicable law or treaty, and any national
+             implementations thereof.
+        
+        2. Waiver. To the greatest extent permitted by, but not in contravention
+        of, applicable law, Affirmer hereby overtly, fully, permanently,
+        irrevocably and unconditionally waives, abandons, and surrenders all of
+        Affirmer's Copyright and Related Rights and associated claims and causes
+        of action, whether now known or unknown (including existing as well as
+        future claims and causes of action), in the Work (i) in all territories
+        worldwide, (ii) for the maximum duration provided by applicable law or
+        treaty (including future time extensions), (iii) in any current or future
+        medium and for any number of copies, and (iv) for any purpose whatsoever,
+        including without limitation commercial, advertising or promotional
+        purposes (the "Waiver"). Affirmer makes the Waiver for the benefit of each
+        member of the public at large and to the detriment of Affirmer's heirs and
+        successors, fully intending that such Waiver shall not be subject to
+        revocation, rescission, cancellation, termination, or any other legal or
+        equitable action to disrupt the quiet enjoyment of the Work by the public
+        as contemplated by Affirmer's express Statement of Purpose.
+        
+        3. Public License Fallback. Should any part of the Waiver for any reason
+        be judged legally invalid or ineffective under applicable law, then the
+        Waiver shall be preserved to the maximum extent permitted taking into
+        account Affirmer's express Statement of Purpose. In addition, to the
+        extent the Waiver is so judged Affirmer hereby grants to each affected
+        person a royalty-free, non transferable, non sublicensable, non exclusive,
+        irrevocable and unconditional license to exercise Affirmer's Copyright and
+        Related Rights in the Work (i) in all territories worldwide, (ii) for the
+        maximum duration provided by applicable law or treaty (including future
+        time extensions), (iii) in any current or future medium and for any number
+        of copies, and (iv) for any purpose whatsoever, including without
+        limitation commercial, advertising or promotional purposes (the
+        "License"). The License shall be deemed effective as of the date CC0 was
+        applied by Affirmer to the Work. Should any part of the License for any
+        reason be judged legally invalid or ineffective under applicable law, such
+        partial invalidity or ineffectiveness shall not invalidate the remainder
+        of the License, and in such case Affirmer hereby affirms that he or she
+        will not (i) exercise any of his or her remaining Copyright and Related
+        Rights in the Work or (ii) assert any associated claims and causes of
+        action with respect to the Work, in either case contrary to Affirmer's
+        express Statement of Purpose.
+        
+        4. Limitations and Disclaimers.
+        
+         a. No trademark or patent rights held by Affirmer are waived, abandoned,
+            surrendered, licensed or otherwise affected by this document.
+         b. Affirmer offers the Work as-is and makes no representations or
+            warranties of any kind concerning the Work, express, implied,
+            statutory or otherwise, including without limitation warranties of
+            title, merchantability, fitness for a particular purpose, non
+            infringement, or the absence of latent or other defects, accuracy, or
+            the present or absence of errors, whether or not discoverable, all to
+            the greatest extent permissible under applicable law.
+         c. Affirmer disclaims responsibility for clearing rights of other persons
+            that may apply to the Work or any use thereof, including without
+            limitation any person's Copyright and Related Rights in the Work.
+            Further, Affirmer disclaims responsibility for obtaining any necessary
+            consents, permissions or other rights required for any use of the
+            Work.
+         d. Affirmer understands and acknowledges that Creative Commons is not a
+            party to this document and has no duty or obligation with respect to
+            this CC0 or use of the Work.
+        
+Project-URL: Homepage, https://github.com/evalvitals/evalrx
+Project-URL: Documentation, https://evalvitals.github.io/evalrx/
+Project-URL: Repository, https://github.com/evalvitals/evalrx
+Project-URL: Issues, https://github.com/evalvitals/evalrx/issues
+Project-URL: Changelog, https://github.com/evalvitals/evalrx/blob/main/CHANGELOG.md
+Keywords: llm,vlm,interpretability,failure-analysis,explainability
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: CC0 1.0 Universal (CC0 1.0) Public Domain Dedication
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: numpy>=1.24
+Provides-Extra: local
+Requires-Dist: torch>=2.0.0; extra == "local"
+Requires-Dist: transformers>=4.51.0; extra == "local"
+Requires-Dist: accelerate>=0.26.0; extra == "local"
+Provides-Extra: finetune
+Requires-Dist: peft>=0.10; extra == "finetune"
+Provides-Extra: api
+Requires-Dist: openai>=1.0; extra == "api"
+Provides-Extra: interp
+Requires-Dist: captum>=0.7; extra == "interp"
+Requires-Dist: inseq>=0.6; extra == "interp"
+Requires-Dist: nnsight>=0.3; extra == "interp"
+Provides-Extra: data
+Requires-Dist: datasets>=2.0; extra == "data"
+Requires-Dist: pillow>=9.0; extra == "data"
+Provides-Extra: contract
+Requires-Dist: pydantic>=2.7; extra == "contract"
+Provides-Extra: viz
+Requires-Dist: matplotlib>=3.7; extra == "viz"
+Provides-Extra: observability
+Requires-Dist: langfuse>=4.7; extra == "observability"
+Provides-Extra: ui
+Requires-Dist: fastapi>=0.115; extra == "ui"
+Requires-Dist: uvicorn>=0.30; extra == "ui"
+Requires-Dist: python-multipart>=0.0.9; extra == "ui"
+Provides-Extra: stats
+Requires-Dist: statsmodels>=0.14; extra == "stats"
+Requires-Dist: scikit-learn>=1.3; extra == "stats"
+Provides-Extra: cluster
+Requires-Dist: scikit-learn>=1.3; extra == "cluster"
+Requires-Dist: hdbscan>=0.8.33; extra == "cluster"
+Requires-Dist: langfuse>=4.7; extra == "cluster"
+Provides-Extra: gemini
+Requires-Dist: google-genai>=1.0; extra == "gemini"
+Provides-Extra: all
+Requires-Dist: torch>=2.0.0; extra == "all"
+Requires-Dist: transformers>=4.51.0; extra == "all"
+Requires-Dist: accelerate>=0.26.0; extra == "all"
+Requires-Dist: peft>=0.10; extra == "all"
+Requires-Dist: openai>=1.0; extra == "all"
+Requires-Dist: captum>=0.7; extra == "all"
+Requires-Dist: inseq>=0.6; extra == "all"
+Requires-Dist: nnsight>=0.3; extra == "all"
+Requires-Dist: matplotlib>=3.7; extra == "all"
+Requires-Dist: statsmodels>=0.14; extra == "all"
+Requires-Dist: scikit-learn>=1.3; extra == "all"
+Requires-Dist: hdbscan>=0.8.33; extra == "all"
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.0; extra == "dev"
+Requires-Dist: ruff>=0.4; extra == "dev"
+Requires-Dist: mypy>=1.8; extra == "dev"
+Requires-Dist: jsonschema>=4.0; extra == "dev"
+Dynamic: license-file
+
+<div align="center">
+
+# EvalRX
+
+### Your eval tells you *what* failed. EvalRX investigates *why*—and tests what fixes it.
+
+<img src="docs/assets/figures/model-health-teaser.svg" alt="A model-health signal enters an auto-research loop that probes, explains and intervenes; only a held-out verified pass updates the model to n+1, which recurs as the next research subject." width="640">
+
+[![PyPI version](https://img.shields.io/pypi/v/evalrx?color=3D8DFF)](https://pypi.org/project/evalrx/)
+[![Python versions](https://img.shields.io/pypi/pyversions/evalrx?color=3D8DFF)](https://pypi.org/project/evalrx/)
+[![CI](https://github.com/evalvitals/evalrx/actions/workflows/ci.yml/badge.svg)](https://github.com/evalvitals/evalrx/actions/workflows/ci.yml)
+[![Docs](https://img.shields.io/badge/docs-live-3D8DFF)](https://evalvitals.github.io/evalrx/overview/)
+[![Demo](https://img.shields.io/badge/demo-live-39A96B)](https://evalvitals.github.io/evalrx/demo/)
+[![License: CC0-1.0](https://img.shields.io/badge/license-CC0--1.0-39A96B)](LICENSE)
+
+[Get started](#quickstart-analyze-your-eval-logs) · [Documentation](https://evalvitals.github.io/evalrx/overview/) · [Live demo](https://evalvitals.github.io/evalrx/demo/) · [Examples](examples/README.md) · [PyPI](https://pypi.org/project/evalrx/)
+
+</div>
+
+Every eval stack ends at a score. EvalRX starts there and closes the loop:
+probe the model for failures, find the structure behind them, propose a
+mechanism, **test it on cases the analysis never saw**, then build a repair and
+prove it beats the unmodified baseline. When a repair fails, the loop escalates
+to a more invasive class of fix and tries again.
+
+The aim is a model that gets measurably better each time round — without a
+human guessing at the cause.
+
+```mermaid
+flowchart LR
+    A[Open-weight model] --> B[Probe]
+    B --> C[Explore]
+    C --> D[Diagnose]
+    D --> E{Held-out<br/>verification}
+    E -->|refuted| B
+    E -->|supported| F[Repair]
+    F --> G{Beats<br/>baseline?}
+    G -->|yes| H[Validated fix]
+    G -->|no| I[Escalate one tier]
+    I --> F
+```
+
+### The repair ladder
+
+"Fix it" is not one action. Repairs are ordered by how deeply they cut into the
+model — each rung buys causal reach and costs deployability:
+
+| | Intervention space | Status |
+|---|---|---|
+| **L1** | Prompt and instruction rewrites | ✅ |
+| **L2** | Scaffolds around an unchanged model — multi-call, tools, aggregation | ✅ |
+| **L3a** | Read internals — attention-guided cropping, contrastive decoding | ✅ |
+| **L3b** | Write internals — attention reweighting, activation steering | ✅ |
+| **L4** | **Parameter space — build a dataset, fine-tune, re-test** | ✅ LoRA on the LLM only; other recipe shapes recorded, not yet executed |
+
+Escalation is never automatic. The ceiling is yours to set (default L2); when
+every candidate at that ceiling fails paired validation, the loop *recommends*
+raising it rather than climbing on its own. At L4 the system always writes a
+complete fine-tune recipe; it *executes* the one shape v1 supports —
+LoRA on the language model, trained on a diagnosis-only pool you pass as
+`FixAgent(finetune_pool=...)`, and validated through the same paired McNemar
++ e-value machinery as every other tier — see
+[`fix_internals.py`](evalrx/eval_agent/stages/fix_internals.py) and
+[`fix_tiers.py`](evalrx/eval_agent/stages/fix_tiers.py).
+
+**L3b and L4 only exist for open weights.** You cannot modify a forward pass or
+fine-tune through somebody's API — which is why this is built on open models.
+
+### One typed shape per stage
+
+Every stage validates what it writes against a machine-readable contract
+(`evalrx/contract/`) and drops it in `<run>/contract/`. TypeScript for the
+whole pipeline is generated from the same Python — `python -m
+evalrx.contract.export --out docs/contract` — so a UI decodes a stage
+instead of re-deriving its shape from the event log.
+
+Modality lives in that contract as *slots*, never as a model-kind enum: LLM,
+VLM, ALM and AVLM are four subsets of `{text, image, audio, video}`, and
+analyzer routing follows the slots a **batch** fills rather than the ones a
+model declares. An omni model evaluated on an audio benchmark is diagnosed as
+an audio run.
+
+### Why the loop is trustworthy
+
+A self-improving system is only as good as its willingness to reject its own
+hypotheses. One that cannot will confidently ship repairs for problems it
+invented.
+
+We pointed EvalRX at three Qwen3-VL checkpoints and asked what predicts
+object hallucination. It found that attention focus share separates
+hallucinations from correct rejections at **AUC 0.82** — then, unprompted,
+argued that its *second* strongest signal was an artifact of how attention was
+extracted, not a real effect. It marked its own best number optimistic, because
+the threshold had been chosen on the rows it was scored on.
+
+**Most eval tools would have shipped you that second finding.**
+
+<img src="examples/m2_m3/deco_hallu_explore/reference_output/figures/attention_geometry_fail_vs_pass.png" alt="Attention geometry of hallucinations vs correct rejections across seven per-case signals, each panel labeled with its AUC and 95% confidence interval" width="100%">
+
+<sub>Generated by the run, not by hand. The contrast is drawn only over
+absent-object probes (n=126 FAIL, n=240 PASS) — the whole-sample version would
+have looked stronger and meant less.</sub>
+
+Held-out splits are taken *before* exploration, multiplicity is controlled with
+e-BH across the candidate family, and every fix is compared against the
+unchanged baseline. A run may end **inconclusive** — and frequently should.
+
+### Two runs you can read right now
+
+No install required — these are real runs, committed unmodified.
+
+| Run | What it shows |
+|---|---|
+| [**Attention & hallucination**](examples/m2_m3/deco_hallu_explore/reference_output/) | 606 real VLM cases across three checkpoints. Finds AUC 0.82, then attacks its own result. 1 of 4 candidate signals survives adjudication. |
+| [**The confound catch**](examples/m2_m3/synthetic_yield_explore/reference_output/) | Catalyst looks significant (ANOVA p = 0.080) until the run notices the groups differ by 21° in temperature. **0 of 4 signals confirmed** — the correct answer. |
+
+## Quickstart: Analyze Your Eval Logs
+
+Install EvalRX:
+
+```bash
+pip install evalrx
+```
+
+Then point it at a file or directory of JSON/JSONL results:
+
+```bash
+evalrx explore ./results \
+  --backend codex \
+  -q "What distinguishes failed cases from successful ones?" \
+  --serve-report
+```
+
+`codex` can be replaced with `claude_code`, `opencode`, `gemini_cli`,
+`kimi_cli`, or `antigravity`. The selected coding-agent CLI must be installed
+and authenticated separately.
+
+Open a finished run in the browser:
+
+```bash
+evalrx serve evalrx_explore_output
+```
+
+`serve` runs the report UI locally as a small server. For a single portable
+file — no server, suitable for sharing — export it instead:
+
+```bash
+evalrx report evalrx_explore_output --out report.html
+```
+
+Both read the same output directory and render the same UI; see the
+[CLI reference](https://evalvitals.github.io/evalrx/cli/) for the rest of the
+command set (Langfuse export, a runs panel over several experiments, …).
+
+EvalRX writes an auditable analysis bundle instead of returning only prose:
+
+```text
+evalrx_explore_output/
+├── exploratory_report.json   # observations, candidate signals, hypotheses
+├── records.json              # normalized records used by the analysis
+├── figures/                  # rendered charts
+├── tables/                   # analysis-ready tables
+└── analysis.py               # the generated code that was actually run
+```
+
+**A real bundled run:** on the synthetic-yield example, Explore identified
+temperature as the strongest observed correlate (`r = 0.90`, 95% CI 0.81–0.95),
+found pressure flat (`r = -0.14`) without converting that null into evidence of
+absence, and caught that the apparent catalyst effect tracks a 21-unit
+temperature imbalance between groups. Zero of four candidate signals cleared
+adjudication. [Read the committed bundle →](examples/m2_m3/synthetic_yield_explore/reference_output/)
+
+Already have your own analysis code? Use the analyzer toolkit directly, or
+feed the resulting cases into the full diagnosis loop. EvalRX does not
+require you to replace your existing eval or observability stack.
+
+## What Makes It Different
+
+| Typical eval workflow | EvalRX |
+|---|---|
+| Aggregate a metric | Investigate the cases behind the metric |
+| Browse failures manually | Search for recurring, structured failure modes |
+| Accept an LLM explanation | Turn explanations into falsifiable hypotheses |
+| Test on the same cases used for discovery | Separate exploration from held-out confirmation |
+| Report a promising prompt rewrite | Compare interventions with the unchanged baseline |
+| Choose either API-level or internal analysis | Negotiate black-box and white-box capabilities through one interface |
+
+Statistical gates use paired tests and e-values, including multiplicity control
+when several hypotheses or fixes are tried. A run may end **inconclusive**;
+EvalRX does not turn weak evidence into a success verdict.
+
+## Three Ways to Use EvalRX
+
+### 1. Explore — raw results to testable hypotheses
+
+`evalrx explore` recursively samples arbitrary JSON/JSONL shapes. The
+coding agent performs exploratory data analysis; the host records generated
+code, adjudicates host-checkable statistics, renders figures, and proposes
+1–3 falsifiable hypotheses.
+
+[Explore guide →](docs/m2_analysis.md)
+
+### 2. Investigate — failures to verified interventions
+
+`VLDiagnoseLoop` chains the full workflow:
+
+```text
+M1 targeted probes
+ → M2 exploratory and statistical analysis
+ → M3 diagnosis hypotheses
+ → M4 held-out hypothesis verification
+ → M5 surgery and tiered fixes
+```
+
+Interventions can range from prompt changes and scaffolds to read/write access
+to model internals. Each candidate is evaluated against the unmodified
+baseline; automatic escalation happens only when explicitly enabled.
+
+[Full-loop quickstart →](docs/quickstart.md#vldiagnoseloop--automated-failure-attribution-current) ·
+[Intervention guide →](docs/intervention.md)
+
+### 3. Analyze — one model, one question
+
+Every registered analyzer follows the same call shape:
+
+```python
+from evalrx import Capability, compose
+from evalrx.analyzers.attention.summary import AttentionAnalyzer
+
+model = compose(
+    "qwen2.5-7b-instruct",
+    "hf_local",
+    want={Capability.ATTENTION},
+)
+
+result = AttentionAnalyzer(layer=-1, top_k=5).run(
+    model, "The Eiffel Tower is in"
+)
+
+print(result.summary())
+```
+
+The analyzer zoo includes attention, uncertainty, hallucination, attribution,
+logit-lens, representation-geometry, and agent-trajectory analysis.
+
+[Browse the Analyzer Zoo →](docs/analyzers.md)
+
+## Installation
+
+The core install stays lightweight—no Torch required:
+
+```bash
+pip install evalrx
+```
+
+Add only the capabilities you need:
+
+```bash
+pip install "evalrx[api]"        # OpenAI-compatible API models
+pip install "evalrx[local]"      # local Hugging Face models + Torch
+pip install "evalrx[interp]"     # interpretability toolchains
+pip install "evalrx[viz]"        # plots
+pip install "evalrx[stats]"      # inferential statistics
+```
+
+For development:
+
+```bash
+git clone https://github.com/evalvitals/evalrx.git
+cd evalrx
+pip install -e ".[dev]"
+pytest -m "not gpu"
+```
+
+## Architecture in One Minute
+
+Model identity is separate from runtime, and analyzers declare the
+capabilities they need. The same model spec can run through a black-box API or
+a white-box local backend; only the available capability set changes.
+
+| Contract | Role |
+|---|---|
+| `ModelSpec` | Model identity: family, repository, architecture traits, modalities. |
+| `Backend` | Runtime: local internals, black-box API, or offline batch engine. |
+| `Model` | Runnable model with generation and optional internal capture. |
+| `Analyzer` | `Analyzer(**params).run(model, data) -> Result`. |
+| `Capability` | Matches analyzers to compatible model runtimes before execution. |
+| `FailureCase` | Prompts, labels, provenance, metadata, and agent trajectories. |
+| `Result` | Human-readable summary plus structured, serializable findings. |
+
+[Read the architecture guide →](docs/architecture.md)
+
+## Reproducible Examples
+
+Two examples ship with **committed output bundles** — readable without
+installing anything, marked 📦 below.
+
+| Example | What it demonstrates |
+|---|---|
+| 📦 [`synthetic_yield_explore`](examples/m2_m3/synthetic_yield_explore/reference_output/) | Standalone Explore on structured tabular outcomes — and a confound caught unprompted. |
+| 📦 [`deco_hallu_explore`](examples/m2_m3/deco_hallu_explore/reference_output/) | Explore → held-out hypothesis tests → tiered repair, on 606 real VLM cases. |
+| [`deco_hallu`](examples/m1_m5/deco_hallu/) | Decoupled multimodal hallucination diagnosis and intervention. |
+| [`qwen_attention`](examples/analyzer_demos/qwen_attention/) | White-box attention analysis on a local model. |
+
+[See all examples →](examples/README.md)
+
+## Documentation
+
+| | |
+|---|---|
+| [Quickstart](docs/quickstart.md) | Runnable examples and common entry points |
+| [Command-Line Interface](docs/cli.md) | Every `evalrx` subcommand — `explore`, `serve`, `report`, and the rest |
+| [Exploratory Analysis](docs/m2_analysis.md) | Standalone `evalrx explore` — descriptive analysis + hypothesis proposal |
+| [Intervention & Verification](docs/intervention.md) | Held-out hypothesis tests and the tiered repair ladder |
+| [Analyzer Zoo](docs/analyzers.md) | Reference tables of implemented analyzers and registered models |
+| [Architecture](docs/architecture.md) | Package structure and design contracts |
+| [Extending EvalRX](docs/extending.md) | How to add analyzers, specs, and backends |
+| [Roadmap](docs/roadmap.md) | Current implementation status and planned surfaces |
+
+The full site — same pages, searchable — is live at
+[evalvitals.github.io/evalrx](https://evalvitals.github.io/evalrx/overview/).
+
+## Project Status
+
+EvalRX is an early-stage research toolkit. Interfaces may evolve, and some
+full-loop examples require model weights, a GPU, or an external coding-agent
+CLI. Bug reports, reproducible failure cases, analyzer contributions, and
+evaluation integrations are welcome.
+
+If EvalRX helps you understand a model failure, consider starring the repo
+and sharing the smallest reproducible case—it makes the toolkit better for the
+next investigation.