autoresearcheval 0.1.0__tar.gz

autoresearcheval-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Titan Research Labs
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

autoresearcheval-0.1.0/PKG-INFO

@@ -0,0 +1,229 @@
+Metadata-Version: 2.4
+Name: autoresearcheval
+Version: 0.1.0
+Summary: Process-level failure diagnosis for autonomous research agents: trajectory -> structured analysis -> ARFT failure labels.
+Author: AutoResearchEval authors
+License: MIT
+Project-URL: Homepage, https://github.com/PrentisAI/AutoResearchEval
+Project-URL: Paper, https://arxiv.org/abs/2608.14905
+Project-URL: Source, https://github.com/PrentisAI/AutoResearchEval/tree/master/agent-as-a-judge
+Keywords: llm,agents,evaluation,failure-taxonomy,agent-as-a-judge,arft
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: httpx>=0.27
+Provides-Extra: stats
+Requires-Dist: pandas>=2.0; extra == "stats"
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == "dev"
+Requires-Dist: build; extra == "dev"
+Requires-Dist: twine; extra == "dev"
+Dynamic: license-file
+
+# autoresearcheval
+
+[![PyPI](https://img.shields.io/pypi/v/autoresearcheval)](https://pypi.org/project/autoresearcheval/)
+
+A two-stage pipeline for turning raw AI-agent research trajectories into a
+structured, evidence-grounded failure-taxonomy classification.
+
+```
+raw trajectory log  --[Stage 1]-->  analysis.md  --[Stage 2]-->  ARFT labels
+```
+
+```python
+from autoresearcheval import generate_analysis, label_arft, pattern_info
+
+analysis = generate_analysis("path/to/trajectory_dir")        # Stage 1
+result   = label_arft(analysis["analysis"], api_key="sk-...")  # Stage 2
+
+print(result["summary"], result["total_failures"])
+for code in result["failure_modes"]:
+    print(code, pattern_info(code)["name"])
+```
+
+A trajectory directory (or a single trajectory JSON, or a dict) and an API key are all
+that is required.
+
+**Stage 1** spawns one fresh Claude Code session per trajectory to write a deep,
+ONBOARDING-conformant `analysis.md` — a structured, six-stage critique (ideation,
+retrieval & synthesis, execution, analysis, writing, self-review) with a claim-by-claim
+verdict table and independent numerical sanity checks, not a summary.
+
+**Stage 2** classifies each `analysis.md` against **ARFT** (the AutoResearch Failure
+Taxonomy: `A.1`–`X.8`, 45 patterns spanning six lifecycle stages plus a cross-cutting
+layer, rolling up to four root-cause pillars — see [`ARFT.md`](https://github.com/PrentisAI/AutoResearchEval/blob/master/agent-as-a-judge/ARFT.md)) and rolls the
+results into a pattern × model matrix, a root-cause breakdown, co-occurrence stats, and
+cross-model agreement.
+
+## Why the judge is artifact-aware
+
+Many failures leave no trace in the report — a result the code never produced, a method
+the logs never ran — so catching them means checking the manuscript against the
+artifacts. Stage 1 therefore runs a fresh, zero-history session per trajectory, with
+shell access and no network, handed the full evidence package (task statement, execution
+log, delivered filesystem, the scorer's own source, every scoring call, read-only gold)
+and required to anchor every finding to a line, file, or value.
+
+Against three-expert annotation on 50 stratified trajectories this reaches **κ = 0.75**
+(pattern) and **0.83** (root cause), versus 0.53 / 0.62 for a single-call LLM-as-a-judge
+on the transcript alone. Almost all of the gain is recall — which is the point: artifact
+access is what makes transcript-invisible failures detectable.
+
+## Install
+
+```bash
+pip install autoresearcheval              # the library and the CLIs
+pip install 'autoresearcheval[stats]'     # adds pandas, needed for the corpus rollups
+```
+
+- Python 3.10+. The only hard dependency is `httpx`.
+- **Stage 2** needs an API key for any OpenAI-compatible chat-completions endpoint. Key
+  resolution: the `api_key` argument, then `ARFT_OPENROUTER_KEY`, then
+  `~/.openrouter_key`, then `OPENROUTER_API_KEY`. Point it elsewhere with `base_url=` or
+  the `AAJ_ENDPOINT` env var.
+- **Stage 1**, and Stage 2's alternate executor, spawn headless
+  [Claude Code](https://claude.com/product/claude-code) sessions and need the `claude`
+  CLI installed and authenticated.
+
+## The two calls
+
+| | What it does | Cost | Needs |
+|---|---|---|---|
+| `generate_analysis(trajectory, ...)` | reads the trajectory's artifacts and writes a six-stage critique with a claim-by-claim verdict table | minutes | the `claude` CLI |
+| `label_arft(analysis, ...)` | maps that critique onto the 45 ARFT codes | one completion | an API key |
+
+They are separate on purpose. Stage 1 is the expensive half, and it produces the
+artifact that makes the labels auditable; folding the two together would hide both.
+
+`generate_analysis` returns `{"task_id", "analysis", "qa", "path", "workspace",
+"duration_s", "returncode"}`. `qa["ok"]` is the depth gate described below — False means
+the analysis came back thinner than the framework's bar, not that the call failed.
+
+`label_arft` returns the classification plus `failure_modes` (every established code),
+`total_failures`, and `qa` (the schema-and-polarity gate). Each hit carries its `code`,
+`name`, `stage`, `pillar`, `root_cause`, `confidence`, `evidence` and `why`.
+
+### Telling the analyst about your harness
+
+Two facts change what counts as a finding: whether the harness's retrieval tools did
+real network I/O or were mocked, and whether gold values are reachable locally. By
+default the analyst is told to **work both out from the trajectory** and report what it
+concluded — so the defaults are correct for any harness and nothing needs editing.
+
+Override them with `retrieval_note=` / `gold_note=` (or the `RETRIEVAL_NOTE` /
+`GOLD_NOTE` constants for the batch CLI) **only if you can state the truth**. A declared
+fact beats an inferred one, but a wrong one is worse than neither: an analyst told that
+a real search tool is mocked will report fabricated retrieval that never happened, and
+one told that a shim is real will credit calibration against literature that was never
+fetched.
+
+## Batch CLIs
+
+The commands that produced the paper's corpus install alongside the library:
+`aaj-generate`, `aaj-classify`, `aaj-classify-cc`, `aaj-aggregate`, `aaj-status`,
+`aaj-verify`, `aaj-analysis-qa`, `aaj-label-qa`.
+
+## Quickstart — Stage 1: trajectory → analysis.md
+
+```bash
+aaj-generate --run-dir /path/to/your_model__your_suite \
+    --concurrency 4 --resume --model claude-opus-4-8
+```
+
+Expects `<run-dir>/traj/*.json`, one JSON object per trajectory with at least a
+`task_id` field and a log field `traj_tools.py` can recognize. `traj_tools.py` normalizes
+three log formats out of the box — Claude Code stream-JSON, Gemini CLI NDJSON, Codex CLI
+JSONL — so multi-megabyte logs need no truncation; see `traj_tools.detect_format`.
+Writes `<model>/<task_id>/analysis.md` under `./corpus` by default (override with the
+`AAJ_CORPUS_DIR` env var).
+
+### The depth exemplar
+
+Each session is handed two references: [`ONBOARDING.md`](https://github.com/PrentisAI/AutoResearchEval/blob/master/agent-as-a-judge/src/autoresearcheval/data/ONBOARDING.md) (the framework — workflow,
+iron rules, required skeleton) and [`analysis_long.md`](https://github.com/PrentisAI/AutoResearchEval/blob/master/agent-as-a-judge/src/autoresearcheval/data/analysis_long.md) (a worked example
+of the bar being met). The exemplar is a real analysis of a real trajectory, not a
+template: a microkinetics rollout whose headline finding is refuted by a sweep table the
+agent itself printed. It is what "every issue is a paragraph with a mechanism, a
+fair-credit reading and a numeric anchor, plus a `[stage | root cause]` trailer" looks
+like in practice, and it clears every gate of the quality checker:
+
+```bash
+python -c "from autoresearcheval import config; print(config.exemplar())"   # where it lives
+aaj-analysis-qa "$(python -c 'from autoresearcheval import config; print(config.exemplar())')" \
+    --reason "soft[current_density]"
+```
+
+Point `AAJ_EXEMPLAR` at a different file to calibrate against your own corpus instead.
+If neither exists, the prompt drops the exemplar line and falls back to ONBOARDING §3;
+depth then rests on the QA gate alone, so writing one reference analysis by hand for
+your own domain is worth the effort.
+
+Note what the exemplar depends on: several of its sharpest findings turn on knowing that
+this run's `WebSearch` was a shim while `WebFetch` was real. That is exactly the fact
+`RETRIEVAL_NOTE` carries into your own runs — get it wrong and the analyst will confidently
+make the opposite mistake.
+
+## Quickstart — Stage 2: analysis.md → ARFT classification
+
+```bash
+export ARFT_OPENROUTER_KEY=...        # or drop a key in ~/.openrouter_key
+scripts/run_all_arft_api.sh           # self-healing: resumes, retries QA failures
+```
+
+Reads `./corpus/<model>/<task>/analysis.md` (`$AAJ_CORPUS_DIR` — the same default
+Stage 1 writes to, so the two stages compose with no extra flags), writes
+per-analysis `<model>/<task_id>.json` plus the rolled-up stats to `./results`
+(`$AAJ_OUT_DIR`):
+
+| Output | What |
+|---|---|
+| `<model>/<task>.json` | Per-analysis classification, evidence-backed |
+| `agg.json` | Dense `[model, task, {code: score}]` grid |
+| `SUMMARY.md` | Pattern × model HIT/PARTIAL matrix, ranked |
+| `root_cause_stats.md` | Lifecycle stage × root-cause pillar breakdown |
+| `matrix_long.csv` | Tidy long-format table everything else derives from |
+| `cooccurrence.csv` / `agreement.csv` | Pattern co-occurrence; cross-model agreement |
+| `tables.tex` | Paper-ready LaTeX |
+| `UNCOVERED.md` | Findings that fit no existing pattern — taxonomy-gap review |
+
+Then check the result is trustworthy before you rely on it:
+
+```bash
+aaj-verify   # polarity regression + (optionally) a prior-run comparison
+```
+
+`aaj-verify` checks that the classifier isn't mistaking exculpatory language for a
+finding (a real failure mode — some diagnostic vocabulary shows up almost entirely in
+*clearing* statements in this kind of writeup) and, if you pass `--pass2` against a
+second independent run, reports per-pattern Cohen's κ so you know which codes are
+reliably distinguishable and which need their guide entry sharpened.
+
+## Taxonomy
+
+The 45-pattern label space, its four root-cause pillars, and what the 800-trajectory
+audit found are documented in [`ARFT.md`](https://github.com/PrentisAI/AutoResearchEval/blob/master/agent-as-a-judge/ARFT.md). The code list lives in
+`autoresearcheval/patterns.py` and the classifier's operational guide ships as package
+data (`config.arft_guide()`).
+
+## Reasoning budget
+
+The reasoning-token budget is the main quality lever on Stage 2 — don't turn it down.
+Disabling reasoning entirely measured **38% recall** against a hand-verified reference
+labelling, missing several genuinely-present patterns; `3000` reasoning tokens
+(the `reasoning_tokens` default) measured **81% recall**.
+
+Stage 1 is the heavier stage per item, being open-ended authoring rather than
+extraction. Use `--dry-run` / `--n` to size a pilot before committing to a full run.
+
+## License
+
+MIT — see `LICENSE`.

autoresearcheval-0.1.0/README.md

@@ -0,0 +1,199 @@
+# autoresearcheval
+
+[![PyPI](https://img.shields.io/pypi/v/autoresearcheval)](https://pypi.org/project/autoresearcheval/)
+
+A two-stage pipeline for turning raw AI-agent research trajectories into a
+structured, evidence-grounded failure-taxonomy classification.
+
+```
+raw trajectory log  --[Stage 1]-->  analysis.md  --[Stage 2]-->  ARFT labels
+```
+
+```python
+from autoresearcheval import generate_analysis, label_arft, pattern_info
+
+analysis = generate_analysis("path/to/trajectory_dir")        # Stage 1
+result   = label_arft(analysis["analysis"], api_key="sk-...")  # Stage 2
+
+print(result["summary"], result["total_failures"])
+for code in result["failure_modes"]:
+    print(code, pattern_info(code)["name"])
+```
+
+A trajectory directory (or a single trajectory JSON, or a dict) and an API key are all
+that is required.
+
+**Stage 1** spawns one fresh Claude Code session per trajectory to write a deep,
+ONBOARDING-conformant `analysis.md` — a structured, six-stage critique (ideation,
+retrieval & synthesis, execution, analysis, writing, self-review) with a claim-by-claim
+verdict table and independent numerical sanity checks, not a summary.
+
+**Stage 2** classifies each `analysis.md` against **ARFT** (the AutoResearch Failure
+Taxonomy: `A.1`–`X.8`, 45 patterns spanning six lifecycle stages plus a cross-cutting
+layer, rolling up to four root-cause pillars — see [`ARFT.md`](https://github.com/PrentisAI/AutoResearchEval/blob/master/agent-as-a-judge/ARFT.md)) and rolls the
+results into a pattern × model matrix, a root-cause breakdown, co-occurrence stats, and
+cross-model agreement.
+
+## Why the judge is artifact-aware
+
+Many failures leave no trace in the report — a result the code never produced, a method
+the logs never ran — so catching them means checking the manuscript against the
+artifacts. Stage 1 therefore runs a fresh, zero-history session per trajectory, with
+shell access and no network, handed the full evidence package (task statement, execution
+log, delivered filesystem, the scorer's own source, every scoring call, read-only gold)
+and required to anchor every finding to a line, file, or value.
+
+Against three-expert annotation on 50 stratified trajectories this reaches **κ = 0.75**
+(pattern) and **0.83** (root cause), versus 0.53 / 0.62 for a single-call LLM-as-a-judge
+on the transcript alone. Almost all of the gain is recall — which is the point: artifact
+access is what makes transcript-invisible failures detectable.
+
+## Install
+
+```bash
+pip install autoresearcheval              # the library and the CLIs
+pip install 'autoresearcheval[stats]'     # adds pandas, needed for the corpus rollups
+```
+
+- Python 3.10+. The only hard dependency is `httpx`.
+- **Stage 2** needs an API key for any OpenAI-compatible chat-completions endpoint. Key
+  resolution: the `api_key` argument, then `ARFT_OPENROUTER_KEY`, then
+  `~/.openrouter_key`, then `OPENROUTER_API_KEY`. Point it elsewhere with `base_url=` or
+  the `AAJ_ENDPOINT` env var.
+- **Stage 1**, and Stage 2's alternate executor, spawn headless
+  [Claude Code](https://claude.com/product/claude-code) sessions and need the `claude`
+  CLI installed and authenticated.
+
+## The two calls
+
+| | What it does | Cost | Needs |
+|---|---|---|---|
+| `generate_analysis(trajectory, ...)` | reads the trajectory's artifacts and writes a six-stage critique with a claim-by-claim verdict table | minutes | the `claude` CLI |
+| `label_arft(analysis, ...)` | maps that critique onto the 45 ARFT codes | one completion | an API key |
+
+They are separate on purpose. Stage 1 is the expensive half, and it produces the
+artifact that makes the labels auditable; folding the two together would hide both.
+
+`generate_analysis` returns `{"task_id", "analysis", "qa", "path", "workspace",
+"duration_s", "returncode"}`. `qa["ok"]` is the depth gate described below — False means
+the analysis came back thinner than the framework's bar, not that the call failed.
+
+`label_arft` returns the classification plus `failure_modes` (every established code),
+`total_failures`, and `qa` (the schema-and-polarity gate). Each hit carries its `code`,
+`name`, `stage`, `pillar`, `root_cause`, `confidence`, `evidence` and `why`.
+
+### Telling the analyst about your harness
+
+Two facts change what counts as a finding: whether the harness's retrieval tools did
+real network I/O or were mocked, and whether gold values are reachable locally. By
+default the analyst is told to **work both out from the trajectory** and report what it
+concluded — so the defaults are correct for any harness and nothing needs editing.
+
+Override them with `retrieval_note=` / `gold_note=` (or the `RETRIEVAL_NOTE` /
+`GOLD_NOTE` constants for the batch CLI) **only if you can state the truth**. A declared
+fact beats an inferred one, but a wrong one is worse than neither: an analyst told that
+a real search tool is mocked will report fabricated retrieval that never happened, and
+one told that a shim is real will credit calibration against literature that was never
+fetched.
+
+## Batch CLIs
+
+The commands that produced the paper's corpus install alongside the library:
+`aaj-generate`, `aaj-classify`, `aaj-classify-cc`, `aaj-aggregate`, `aaj-status`,
+`aaj-verify`, `aaj-analysis-qa`, `aaj-label-qa`.
+
+## Quickstart — Stage 1: trajectory → analysis.md
+
+```bash
+aaj-generate --run-dir /path/to/your_model__your_suite \
+    --concurrency 4 --resume --model claude-opus-4-8
+```
+
+Expects `<run-dir>/traj/*.json`, one JSON object per trajectory with at least a
+`task_id` field and a log field `traj_tools.py` can recognize. `traj_tools.py` normalizes
+three log formats out of the box — Claude Code stream-JSON, Gemini CLI NDJSON, Codex CLI
+JSONL — so multi-megabyte logs need no truncation; see `traj_tools.detect_format`.
+Writes `<model>/<task_id>/analysis.md` under `./corpus` by default (override with the
+`AAJ_CORPUS_DIR` env var).
+
+### The depth exemplar
+
+Each session is handed two references: [`ONBOARDING.md`](https://github.com/PrentisAI/AutoResearchEval/blob/master/agent-as-a-judge/src/autoresearcheval/data/ONBOARDING.md) (the framework — workflow,
+iron rules, required skeleton) and [`analysis_long.md`](https://github.com/PrentisAI/AutoResearchEval/blob/master/agent-as-a-judge/src/autoresearcheval/data/analysis_long.md) (a worked example
+of the bar being met). The exemplar is a real analysis of a real trajectory, not a
+template: a microkinetics rollout whose headline finding is refuted by a sweep table the
+agent itself printed. It is what "every issue is a paragraph with a mechanism, a
+fair-credit reading and a numeric anchor, plus a `[stage | root cause]` trailer" looks
+like in practice, and it clears every gate of the quality checker:
+
+```bash
+python -c "from autoresearcheval import config; print(config.exemplar())"   # where it lives
+aaj-analysis-qa "$(python -c 'from autoresearcheval import config; print(config.exemplar())')" \
+    --reason "soft[current_density]"
+```
+
+Point `AAJ_EXEMPLAR` at a different file to calibrate against your own corpus instead.
+If neither exists, the prompt drops the exemplar line and falls back to ONBOARDING §3;
+depth then rests on the QA gate alone, so writing one reference analysis by hand for
+your own domain is worth the effort.
+
+Note what the exemplar depends on: several of its sharpest findings turn on knowing that
+this run's `WebSearch` was a shim while `WebFetch` was real. That is exactly the fact
+`RETRIEVAL_NOTE` carries into your own runs — get it wrong and the analyst will confidently
+make the opposite mistake.
+
+## Quickstart — Stage 2: analysis.md → ARFT classification
+
+```bash
+export ARFT_OPENROUTER_KEY=...        # or drop a key in ~/.openrouter_key
+scripts/run_all_arft_api.sh           # self-healing: resumes, retries QA failures
+```
+
+Reads `./corpus/<model>/<task>/analysis.md` (`$AAJ_CORPUS_DIR` — the same default
+Stage 1 writes to, so the two stages compose with no extra flags), writes
+per-analysis `<model>/<task_id>.json` plus the rolled-up stats to `./results`
+(`$AAJ_OUT_DIR`):
+
+| Output | What |
+|---|---|
+| `<model>/<task>.json` | Per-analysis classification, evidence-backed |
+| `agg.json` | Dense `[model, task, {code: score}]` grid |
+| `SUMMARY.md` | Pattern × model HIT/PARTIAL matrix, ranked |
+| `root_cause_stats.md` | Lifecycle stage × root-cause pillar breakdown |
+| `matrix_long.csv` | Tidy long-format table everything else derives from |
+| `cooccurrence.csv` / `agreement.csv` | Pattern co-occurrence; cross-model agreement |
+| `tables.tex` | Paper-ready LaTeX |
+| `UNCOVERED.md` | Findings that fit no existing pattern — taxonomy-gap review |
+
+Then check the result is trustworthy before you rely on it:
+
+```bash
+aaj-verify   # polarity regression + (optionally) a prior-run comparison
+```
+
+`aaj-verify` checks that the classifier isn't mistaking exculpatory language for a
+finding (a real failure mode — some diagnostic vocabulary shows up almost entirely in
+*clearing* statements in this kind of writeup) and, if you pass `--pass2` against a
+second independent run, reports per-pattern Cohen's κ so you know which codes are
+reliably distinguishable and which need their guide entry sharpened.
+
+## Taxonomy
+
+The 45-pattern label space, its four root-cause pillars, and what the 800-trajectory
+audit found are documented in [`ARFT.md`](https://github.com/PrentisAI/AutoResearchEval/blob/master/agent-as-a-judge/ARFT.md). The code list lives in
+`autoresearcheval/patterns.py` and the classifier's operational guide ships as package
+data (`config.arft_guide()`).
+
+## Reasoning budget
+
+The reasoning-token budget is the main quality lever on Stage 2 — don't turn it down.
+Disabling reasoning entirely measured **38% recall** against a hand-verified reference
+labelling, missing several genuinely-present patterns; `3000` reasoning tokens
+(the `reasoning_tokens` default) measured **81% recall**.
+
+Stage 1 is the heavier stage per item, being open-ended authoring rather than
+extraction. Use `--dry-run` / `--n` to size a pilot before committing to a full run.
+
+## License
+
+MIT — see `LICENSE`.

autoresearcheval-0.1.0/pyproject.toml

@@ -0,0 +1,55 @@
+[build-system]
+requires = ["setuptools>=68", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "autoresearcheval"
+version = "0.1.0"
+description = "Process-level failure diagnosis for autonomous research agents: trajectory -> structured analysis -> ARFT failure labels."
+readme = "README.md"
+requires-python = ">=3.10"
+license = { text = "MIT" }
+authors = [{ name = "AutoResearchEval authors" }]
+keywords = ["llm", "agents", "evaluation", "failure-taxonomy", "agent-as-a-judge", "arft"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Science/Research",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+]
+dependencies = [
+    "httpx>=0.27",
+]
+
+[project.optional-dependencies]
+# Only the corpus-wide rollups need a dataframe library; a single label_arft() call
+# does not, so pandas stays out of the default install.
+stats = ["pandas>=2.0"]
+dev = ["pytest>=7", "build", "twine"]
+
+[project.urls]
+Homepage = "https://github.com/PrentisAI/AutoResearchEval"
+Paper = "https://arxiv.org/abs/2608.14905"
+Source = "https://github.com/PrentisAI/AutoResearchEval/tree/master/agent-as-a-judge"
+
+[project.scripts]
+aaj-generate = "autoresearcheval.generate:main"
+aaj-classify = "autoresearcheval.classify:main"
+aaj-classify-cc = "autoresearcheval.classify_cc:main"
+aaj-aggregate = "autoresearcheval.aggregate:main"
+aaj-status = "autoresearcheval.status:main"
+aaj-verify = "autoresearcheval.verify:main"
+aaj-analysis-qa = "autoresearcheval.analysis_qa:main"
+aaj-label-qa = "autoresearcheval.label_qa:main"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.setuptools.package-data]
+# The framework, the depth exemplar and the classifier guide are inputs the code reads
+# at runtime and hands to subprocesses by absolute path, so they must ship in the wheel.
+autoresearcheval = ["data/*.md"]

autoresearcheval-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

autoresearcheval-0.1.0/src/autoresearcheval/__init__.py

@@ -0,0 +1,36 @@
+"""AutoResearchEval — process-level failure diagnosis for autonomous research agents.
+
+Two stages, two calls:
+
+    from autoresearcheval import generate_analysis, label_arft
+
+    analysis = generate_analysis(trajectory, retrieval_note=..., gold_note=...)
+    result   = label_arft(analysis["analysis"], api_key="...")
+
+    for code in result["failure_modes"]:
+        print(code, pattern_info(code)["name"])
+
+Stage 1 reads a trajectory's artifacts — execution log, delivered files, the scorer's
+own source — and writes a structured six-stage critique. Stage 2 maps that critique onto
+**ARFT**, the AutoResearch Failure Taxonomy: 45 patterns across six lifecycle stages plus
+a cross-cutting layer, rolling up to four root-cause pillars.
+
+Judging against artifacts rather than the transcript alone is the point: on 50 stratified
+trajectories against three-expert annotation this reaches kappa 0.75 (pattern) and 0.83
+(root cause), versus 0.53 / 0.62 for a single-call judge shown only the transcript.
+Almost all of the difference is recall.
+
+The batch CLIs that produced the paper's corpus are installed alongside: ``aaj-generate``
+(Stage 1 over a run directory), ``aaj-classify`` (Stage 2 over a corpus), plus
+``aaj-aggregate``, ``aaj-status`` and ``aaj-verify``.
+"""
+from .api import all_patterns, generate_analysis, label_arft, pattern_info
+
+__version__ = "0.1.0"
+__all__ = [
+    "generate_analysis",
+    "label_arft",
+    "pattern_info",
+    "all_patterns",
+    "__version__",
+]