reasonbench 0.0.1__tar.gz

reasonbench-0.0.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) #Author
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

reasonbench-0.0.1/PKG-INFO

@@ -0,0 +1,242 @@
+Metadata-Version: 2.4
+Name: reasonbench
+Version: 0.0.1
+Summary: Benchmark for evaluating the stability of LLM reasoning strategies
+Author: AU-CLAN
+License: MIT
+Project-URL: Repository, https://github.com/au-clan/ReasonBench
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pandas
+Requires-Dist: diskcache
+Requires-Dist: omegaconf
+Requires-Dist: together
+Requires-Dist: openai
+Requires-Dist: sympy
+Requires-Dist: torch
+Requires-Dist: langchain
+Requires-Dist: langchain-community
+Requires-Dist: wikipedia
+Requires-Dist: joblib
+Requires-Dist: pyphen
+Requires-Dist: syllables
+Requires-Dist: pronouncing
+Requires-Dist: groq
+Requires-Dist: lazykey
+Requires-Dist: python-dotenv
+Requires-Dist: cachesaver
+Requires-Dist: huggingface-hub
+Dynamic: license-file
+
+# ReasonBENCH: Benchmarking the (In)Stability of LLM Reasoning
+
+**ReasonBENCH** is a benchmark suite and open-source library for controlled multi-run evaluation of LLM reasoning. It measures both the quality and stability of reasoning strategies by running repeated independent trials and reporting variance-aware metrics — including confidence intervals, run deviation, and global noise — rather than relying on single-run averages.
+
+> *Preliminary work. Under review by the International Conference on Machine Learning (ICML).*
+
+<!-- Leaderboard: http://reasonbench.github.io -->
+
+## Motivation
+
+LLM reasoning is typically evaluated using single runs, masking how much performance can vary across repeated executions. This practice obscures both reliability and cost, and can lead to misleading comparisons between methods and models. ReasonBENCH addresses this by repeating every model-strategy-task configuration with 10 independent trials and reporting distributional metrics alongside averages.
+
+Key findings from our evaluation:
+- **Run-to-run variability is substantial** — often large enough to change model/method rankings relative to single-run averages
+- **Quality and cost stability decouple** — the most accurate strategy is not necessarily the most stable, and vice versa
+- **Model scaling improves both quality and stability** — larger models within a family yield tighter distributions
+- **Prompt refinements improve quality but not stability** — clarifying prompts and parsers boosts accuracy without reducing run-to-run variance
+- **Reasoning effort scales cost, not quality** — increasing test-time reasoning effort primarily raises cost with limited and statistically insignificant quality gains
+
+## Reasoning Strategies
+
+We implement 10 representative reasoning strategies using a standardized interface:
+
+| Strategy | Type | Reference |
+|----------|------|-----------|
+| **IO** | Direct | — |
+| **CoT** | Direct | Wei et al., 2022 |
+| **CoT-SC** | Direct | Wang et al., 2023 |
+| **ReAct** | Adaptive | Yao et al., 2023b |
+| **Reflexion** | Adaptive | Shinn et al., 2023 |
+| **ToT-BFS** | Structured | Yao et al., 2023a |
+| **ToT-DFS** | Structured | Yao et al., 2023a |
+| **GoT** | Structured | Besta et al., 2024 |
+| **RAP** | Planning | Hao et al., 2023 |
+| **FoA** | Evolutionary | Klein et al., 2025 |
+
+## Benchmarks
+
+6 tasks spanning diverse reasoning domains:
+
+| Task | Domain | Metric | Size |
+|------|--------|--------|------|
+| **Game of 24** | Mathematical reasoning | Accuracy | 100 |
+| **SciBench** | Scientific reasoning | Accuracy (exact match) | 109 |
+| **HumanEval** | Code generation | pass@1 | 100 |
+| **HotPotQA** | Multi-hop QA | Exact match | 100 |
+| **Sonnet Writing** | Creative writing | Accuracy (rhyme + words) | 50 |
+| **HLE** | General reasoning (Humanity's Last Exam) | Accuracy | 50 |
+
+## Evaluated Models
+
+10 contemporary reasoning models from 6 providers:
+
+| Model | Provider |
+|-------|----------|
+| GPT-4.1 Nano, GPT-4.1 Mini | OpenAI |
+| GPT-5 Nano, GPT-5 Mini | OpenAI |
+| GPT-OSS 120B | Together AI |
+| DeepSeek R1 | Together AI |
+| Llama 4 Maverick | Together AI |
+| Qwen3-235B Thinking | Together AI |
+| Claude Haiku 4.5 | Anthropic |
+| Gemini 3 Flash | Google |
+
+## Setup
+
+```bash
+pip install -r requirements.txt
+```
+
+You also need [CacheSaver](https://github.com/au-clan/cachesaver) — a client-side inference optimization framework for efficient, affordable, and reproducible LLM inference:
+
+```bash
+pip install cachesaver
+```
+
+Set your API keys as environment variables:
+
+```bash
+export OPENAI_API_KEY="sk-..."
+# and/or other provider keys
+```
+
+## Quick Start
+
+The simplest way to run an experiment is via the shell script:
+
+```bash
+bash scripts/simple/simple.sh
+```
+
+Edit the variables at the top of `scripts/simple/simple.sh` to change the benchmark, method, model, and split.
+
+For direct invocation:
+
+```bash
+python scripts/simple/simple.py \
+    --benchmark game24 \
+    --method tot_bfs \
+    --split mini \
+    --provider openai \
+    --api_key OPENAI_API_KEY \
+    --model gpt-4.1-nano \
+    --temperature 1.0 \
+    --max_completion_tokens 10000 \
+    --top_p 1.0 \
+    --batch_size 1 \
+    --timeout 2.0 \
+    --correctness 1 \
+    --allow_batch_overflow 1 \
+    --ns_ratio 0.0 \
+    --value_cache
+```
+
+### Key arguments
+
+| Argument | Description |
+|----------|-------------|
+| `--benchmark` | Task name: `game24`, `humaneval`, `hotpotqa`, `scibench`, `hle`, `sonnetwriting` |
+| `--method` | Reasoning method: `io`, `cot`, `cot_sc`, `foa`, `tot_bfs`, `tot_dfs`, `got`, `react`, `rap` |
+| `--split` | Dataset split: `train`, `validation`, `test`, `mini` |
+| `--provider` | LLM provider: `openai`, `gemini`, `anthropic`, `groq`, `together` |
+| `--model` | Model identifier (e.g., `gpt-4.1-nano`, `claude-haiku-4-5`) |
+| `--ns_ratio` | Namespace ratio (0.0–1.0) for controlling parallel execution |
+
+## Evaluation Metrics
+
+For each model-strategy-task configuration, we report metrics along two dimensions:
+
+**Quality:**
+- **Average** — stratified bootstrap mean over runs; benchmarks treated as strata
+- **Run Deviation** — typical run-to-run deviation from the strategy mean per benchmark
+- **Noise (Global)** — variance of z-scored outcomes across all benchmarks
+- **Noise (Run)** — average within-benchmark z-score variance
+
+**Cost:**
+- Same four metrics computed over token usage and wall-clock time, expressed in USD
+
+## Configuration
+
+Method hyperparameters are defined per task in YAML files under `scripts/configs/`:
+
+```yaml
+# scripts/configs/game24.yaml
+tot_bfs:
+  num_selections: 3
+  num_steps: 4
+  num_evaluations: 3
+
+got:
+  num_selections: 5
+  num_steps: 4
+  num_generate: 10
+  num_evaluations: 3
+  num_best: 2
+```
+
+Decoding parameters (temperature, top_p, max tokens) are sourced from `scripts/configs/<task>.env`.
+
+## Architecture
+
+ReasonBENCH is organized around four core abstractions:
+
+- **Method** — specifies the reasoning strategy independently of the model or task. Integrates agents, the environment, and the model, and exposes a standard `solve()` interface.
+- **Environment** — formalizes task-specific dynamics: state transitions, action validation, terminal conditions, and evaluation.
+- **Agent** — defines the interface between methods, models, and states. Agents construct prompts, issue queries, and parse responses into actions.
+- **Model** — uniform interface for LLM providers, supporting async execution and integrated with CacheSaver for response caching and deduplication.
+
+```
+src/
+├── models/          # LLM provider adapters (OpenAI, Anthropic, Groq, Together, Gemini)
+├── methods/         # Reasoning strategy implementations
+├── tasks/           # Task definitions (state, environment, agents, prompts)
+│   ├── game24/
+│   ├── humaneval/
+│   ├── hotpotqa/
+│   └── ...
+├── __init__.py      # Factory registrations
+├── typedefs.py      # Core ABCs and type definitions
+└── utils.py         # Logging and utility functions
+
+scripts/
+├── simple/          # Single-run experiment scripts
+├── repeats/         # Batch/repeated experiment scripts
+├── cached/          # Cached inference scripts
+└── configs/         # YAML and .env configuration files
+
+datasets/            # Gzip-compressed task datasets
+tests/               # Pytest test suite
+```
+
+## Tests
+
+```bash
+pytest                                       # run all tests
+pytest tests/got/test_game24.py              # single file
+pytest tests/got/test_game24.py -k "test_x"  # single test
+```
+
+Tests use async fixtures and require valid API keys (Groq/OpenAI) for the mock LLM clients.
+
+## Citation
+
+```bibtex
+@inproceedings{reasonbench2025,
+  title={ReasonBENCH: Benchmarking the (In)Stability of LLM Reasoning},
+  author={Anonymous},
+  year={2025},
+  note={Under review at ICML}
+}
+```

reasonbench-0.0.1/README.md

@@ -0,0 +1,211 @@
+# ReasonBENCH: Benchmarking the (In)Stability of LLM Reasoning
+
+**ReasonBENCH** is a benchmark suite and open-source library for controlled multi-run evaluation of LLM reasoning. It measures both the quality and stability of reasoning strategies by running repeated independent trials and reporting variance-aware metrics — including confidence intervals, run deviation, and global noise — rather than relying on single-run averages.
+
+> *Preliminary work. Under review by the International Conference on Machine Learning (ICML).*
+
+<!-- Leaderboard: http://reasonbench.github.io -->
+
+## Motivation
+
+LLM reasoning is typically evaluated using single runs, masking how much performance can vary across repeated executions. This practice obscures both reliability and cost, and can lead to misleading comparisons between methods and models. ReasonBENCH addresses this by repeating every model-strategy-task configuration with 10 independent trials and reporting distributional metrics alongside averages.
+
+Key findings from our evaluation:
+- **Run-to-run variability is substantial** — often large enough to change model/method rankings relative to single-run averages
+- **Quality and cost stability decouple** — the most accurate strategy is not necessarily the most stable, and vice versa
+- **Model scaling improves both quality and stability** — larger models within a family yield tighter distributions
+- **Prompt refinements improve quality but not stability** — clarifying prompts and parsers boosts accuracy without reducing run-to-run variance
+- **Reasoning effort scales cost, not quality** — increasing test-time reasoning effort primarily raises cost with limited and statistically insignificant quality gains
+
+## Reasoning Strategies
+
+We implement 10 representative reasoning strategies using a standardized interface:
+
+| Strategy | Type | Reference |
+|----------|------|-----------|
+| **IO** | Direct | — |
+| **CoT** | Direct | Wei et al., 2022 |
+| **CoT-SC** | Direct | Wang et al., 2023 |
+| **ReAct** | Adaptive | Yao et al., 2023b |
+| **Reflexion** | Adaptive | Shinn et al., 2023 |
+| **ToT-BFS** | Structured | Yao et al., 2023a |
+| **ToT-DFS** | Structured | Yao et al., 2023a |
+| **GoT** | Structured | Besta et al., 2024 |
+| **RAP** | Planning | Hao et al., 2023 |
+| **FoA** | Evolutionary | Klein et al., 2025 |
+
+## Benchmarks
+
+6 tasks spanning diverse reasoning domains:
+
+| Task | Domain | Metric | Size |
+|------|--------|--------|------|
+| **Game of 24** | Mathematical reasoning | Accuracy | 100 |
+| **SciBench** | Scientific reasoning | Accuracy (exact match) | 109 |
+| **HumanEval** | Code generation | pass@1 | 100 |
+| **HotPotQA** | Multi-hop QA | Exact match | 100 |
+| **Sonnet Writing** | Creative writing | Accuracy (rhyme + words) | 50 |
+| **HLE** | General reasoning (Humanity's Last Exam) | Accuracy | 50 |
+
+## Evaluated Models
+
+10 contemporary reasoning models from 6 providers:
+
+| Model | Provider |
+|-------|----------|
+| GPT-4.1 Nano, GPT-4.1 Mini | OpenAI |
+| GPT-5 Nano, GPT-5 Mini | OpenAI |
+| GPT-OSS 120B | Together AI |
+| DeepSeek R1 | Together AI |
+| Llama 4 Maverick | Together AI |
+| Qwen3-235B Thinking | Together AI |
+| Claude Haiku 4.5 | Anthropic |
+| Gemini 3 Flash | Google |
+
+## Setup
+
+```bash
+pip install -r requirements.txt
+```
+
+You also need [CacheSaver](https://github.com/au-clan/cachesaver) — a client-side inference optimization framework for efficient, affordable, and reproducible LLM inference:
+
+```bash
+pip install cachesaver
+```
+
+Set your API keys as environment variables:
+
+```bash
+export OPENAI_API_KEY="sk-..."
+# and/or other provider keys
+```
+
+## Quick Start
+
+The simplest way to run an experiment is via the shell script:
+
+```bash
+bash scripts/simple/simple.sh
+```
+
+Edit the variables at the top of `scripts/simple/simple.sh` to change the benchmark, method, model, and split.
+
+For direct invocation:
+
+```bash
+python scripts/simple/simple.py \
+    --benchmark game24 \
+    --method tot_bfs \
+    --split mini \
+    --provider openai \
+    --api_key OPENAI_API_KEY \
+    --model gpt-4.1-nano \
+    --temperature 1.0 \
+    --max_completion_tokens 10000 \
+    --top_p 1.0 \
+    --batch_size 1 \
+    --timeout 2.0 \
+    --correctness 1 \
+    --allow_batch_overflow 1 \
+    --ns_ratio 0.0 \
+    --value_cache
+```
+
+### Key arguments
+
+| Argument | Description |
+|----------|-------------|
+| `--benchmark` | Task name: `game24`, `humaneval`, `hotpotqa`, `scibench`, `hle`, `sonnetwriting` |
+| `--method` | Reasoning method: `io`, `cot`, `cot_sc`, `foa`, `tot_bfs`, `tot_dfs`, `got`, `react`, `rap` |
+| `--split` | Dataset split: `train`, `validation`, `test`, `mini` |
+| `--provider` | LLM provider: `openai`, `gemini`, `anthropic`, `groq`, `together` |
+| `--model` | Model identifier (e.g., `gpt-4.1-nano`, `claude-haiku-4-5`) |
+| `--ns_ratio` | Namespace ratio (0.0–1.0) for controlling parallel execution |
+
+## Evaluation Metrics
+
+For each model-strategy-task configuration, we report metrics along two dimensions:
+
+**Quality:**
+- **Average** — stratified bootstrap mean over runs; benchmarks treated as strata
+- **Run Deviation** — typical run-to-run deviation from the strategy mean per benchmark
+- **Noise (Global)** — variance of z-scored outcomes across all benchmarks
+- **Noise (Run)** — average within-benchmark z-score variance
+
+**Cost:**
+- Same four metrics computed over token usage and wall-clock time, expressed in USD
+
+## Configuration
+
+Method hyperparameters are defined per task in YAML files under `scripts/configs/`:
+
+```yaml
+# scripts/configs/game24.yaml
+tot_bfs:
+  num_selections: 3
+  num_steps: 4
+  num_evaluations: 3
+
+got:
+  num_selections: 5
+  num_steps: 4
+  num_generate: 10
+  num_evaluations: 3
+  num_best: 2
+```
+
+Decoding parameters (temperature, top_p, max tokens) are sourced from `scripts/configs/<task>.env`.
+
+## Architecture
+
+ReasonBENCH is organized around four core abstractions:
+
+- **Method** — specifies the reasoning strategy independently of the model or task. Integrates agents, the environment, and the model, and exposes a standard `solve()` interface.
+- **Environment** — formalizes task-specific dynamics: state transitions, action validation, terminal conditions, and evaluation.
+- **Agent** — defines the interface between methods, models, and states. Agents construct prompts, issue queries, and parse responses into actions.
+- **Model** — uniform interface for LLM providers, supporting async execution and integrated with CacheSaver for response caching and deduplication.
+
+```
+src/
+├── models/          # LLM provider adapters (OpenAI, Anthropic, Groq, Together, Gemini)
+├── methods/         # Reasoning strategy implementations
+├── tasks/           # Task definitions (state, environment, agents, prompts)
+│   ├── game24/
+│   ├── humaneval/
+│   ├── hotpotqa/
+│   └── ...
+├── __init__.py      # Factory registrations
+├── typedefs.py      # Core ABCs and type definitions
+└── utils.py         # Logging and utility functions
+
+scripts/
+├── simple/          # Single-run experiment scripts
+├── repeats/         # Batch/repeated experiment scripts
+├── cached/          # Cached inference scripts
+└── configs/         # YAML and .env configuration files
+
+datasets/            # Gzip-compressed task datasets
+tests/               # Pytest test suite
+```
+
+## Tests
+
+```bash
+pytest                                       # run all tests
+pytest tests/got/test_game24.py              # single file
+pytest tests/got/test_game24.py -k "test_x"  # single test
+```
+
+Tests use async fixtures and require valid API keys (Groq/OpenAI) for the mock LLM clients.
+
+## Citation
+
+```bibtex
+@inproceedings{reasonbench2025,
+  title={ReasonBENCH: Benchmarking the (In)Stability of LLM Reasoning},
+  author={Anonymous},
+  year={2025},
+  note={Under review at ICML}
+}
+```

reasonbench-0.0.1/pyproject.toml

@@ -0,0 +1,50 @@
+[build-system]
+requires = ["setuptools>=68.0"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "reasonbench"
+version = "0.0.1"
+description = "Benchmark for evaluating the stability of LLM reasoning strategies"
+readme = "README.md"
+license = {text = "MIT"}
+requires-python = ">=3.10"
+authors = [
+    {name = "AU-CLAN"},
+]
+dependencies = [
+    "pandas",
+    "diskcache",
+    "omegaconf",
+    "together",
+    "openai",
+    "sympy",
+    "torch",
+    "langchain",
+    "langchain-community",
+    "wikipedia",
+    "joblib",
+    "pyphen",
+    "syllables",
+    "pronouncing",
+    "groq",
+    "lazykey",
+    "python-dotenv",
+    "cachesaver",
+    "huggingface-hub",
+]
+
+[project.urls]
+Repository = "https://github.com/au-clan/ReasonBench"
+
+[tool.setuptools.packages.find]
+include = ["reasonbench*"]
+
+[tool.setuptools.package-data]
+reasonbench = ["configs/*.yaml", "configs/*.env"]
+
+[tool.pytest.ini_options]
+pythonpath = ["."]
+asyncio_mode = "auto"
+asyncio_default_test_loop_scope = "class"
+asyncio_default_fixture_loop_scope = "function"

reasonbench-0.0.1/reasonbench/__init__.py

@@ -0,0 +1,126 @@
+from typing import TypedDict
+from .typedefs import DecodingParameters
+from .datasets import get_dataset_path
+
+class BenchmarkFactory:
+    registry = {}
+
+    @classmethod
+    def register(cls, benchmark_cls):
+        cls.registry[benchmark_cls.__name__.lower()] = benchmark_cls
+        return benchmark_cls
+
+    @classmethod
+    def get(cls, task: str, *args, **kwargs):
+        key = f"benchmark{task}".lower()
+        try:
+            path = get_dataset_path(task)
+            return cls.registry[key](path=path, *args, **kwargs)
+        except KeyError:
+            raise ValueError(f"No benchmark found for task={task}")
+        
+class EnvironmentFactory:
+    registry = {}
+
+    @classmethod
+    def register(cls, env_cls):
+        cls.registry[env_cls.__name__.lower()] = env_cls
+        return env_cls
+    
+    @classmethod
+    def get(cls, task: str, *args, **kwargs):
+        key = f"environment{task}".lower()
+        try:
+            return cls.registry[key](*args, **kwargs)
+        except KeyError:
+            raise ValueError(f"No environment found for task={task}")
+    
+class AgentFactory:
+    registry = {}
+
+    @classmethod
+    def register(cls, agent_cls):
+        cls.registry[agent_cls.__name__.lower()] = agent_cls
+        return agent_cls
+
+    @classmethod
+    def get(cls, agent_type: str, benchmark: str, *args, **kwargs):
+        key = f"agent{agent_type}{benchmark}".lower()
+        try:
+            return cls.registry[key]#(*args, **kwargs) : Not initialized
+        except KeyError:
+            raise ValueError(f"No agent found for type={agent_type}, benchmark={benchmark}")
+
+class AgentDictFactory:
+    registry = {}
+
+    @classmethod
+    def register(cls, agent_dict_cls):
+        cls.registry[agent_dict_cls.__name__.lower()] = agent_dict_cls
+        return agent_dict_cls
+
+    @classmethod
+    def get(cls, method: str, *args, **kwargs):
+        key = f"agentdict{method}".lower()
+        try:
+            return cls.registry[key](*args, **kwargs)
+        except KeyError:
+            raise ValueError(f"No agent dict found for method={method}")
+
+
+class MethodFactory:
+    registry = {}
+
+    @classmethod
+    def register(cls, method_cls):
+        cls.registry[method_cls.__name__.lower()] = method_cls
+        return method_cls
+
+    @classmethod
+    def get(cls, method: str, benchmark: str, params: DecodingParameters, *args, **kwargs):
+        key = f"method{method}".lower()
+
+        
+        if method == "io":
+            agents = {
+                "step": AgentFactory.get("io", benchmark),
+            }
+        elif method in ["cot", "cot_sc"]:
+            agents = {
+                "step": AgentFactory.get("cot", benchmark),
+            }
+        elif method == "foa":
+            agents = {
+                "step": AgentFactory.get("act", benchmark),
+                "evaluate": AgentFactory.get("evaluate", benchmark),
+            }
+        elif method in ["tot_bfs", "tot_dfs"]:
+            agents = {
+                "step": AgentFactory.get("bfs", benchmark),
+                "evaluate": AgentFactory.get("evaluate", benchmark),
+            }
+        elif method == "got":
+            agents = {
+                "step": AgentFactory.get("act", benchmark),
+                "aggregate": AgentFactory.get("aggregate", benchmark),
+                "evaluate": AgentFactory.get("evaluate", benchmark),
+            }
+        elif method == "rap":
+            agents = {
+                "step": AgentFactory.get("react", benchmark),
+                "evaluate": AgentFactory.get("selfevaluate", benchmark),
+            }
+        elif method == "react":
+            agents = {
+                "step": AgentFactory.get("react", benchmark),
+            }
+        else:
+            raise NotImplementedError(f"Method {method} is not implemented yet.")
+        
+        # For the moment, only supporting same params for all agents
+        agents.update({k+"_params": params for k in agents.keys()})
+
+        try:
+            return cls.registry[key](agents=agents, *args, **kwargs)
+        except KeyError:
+            raise ValueError(f"No method found for name={method}")