virex-bench 0.1.2__tar.gz

virex_bench-0.1.2/.agents/skills/simplify/SKILL.md

@@ -0,0 +1,75 @@
+---
+name: simplify
+description: Review changed code for reuse, quality, and efficiency, then fix any issues found.
+---
+
+# Simplify: Code Review and Cleanup
+
+Review all changed files for reuse, quality, and efficiency. Fix any issues found.
+
+## Project Context
+
+ViREx-Bench is a Vietnamese multi-step-reasoning benchmark built on **DSPy**, organized
+around four axes — task / model / strategy / decoding — plus shared modules. See `AGENTS.md`
+for the full layout. When reviewing and fixing, respect these boundaries:
+
+- **Do not edit** files under `data/` (datasets managed externally) or `uv.lock` by hand.
+- **Do not modify** generated/cached dirs: `.ruff_cache/`, `.cache/`, `_testing_results/`.
+- **Never introduce** committed secrets or `.env` contents.
+
+## Phase 1: Identify Changes
+
+Run `git diff` (or `git diff HEAD` if there are staged changes) to see what changed. If there are no git changes, review the most recently modified files that the user mentioned or that you edited earlier in this conversation.
+
+Ignore directories and files starting with `_` (e.g. `_testing_results/`, `_test_format.py`) — these are local/testing files, not part of the main codebase.
+
+## Phase 2: Launch Three Review Agents in Parallel
+
+Use the Agent tool to launch all three agents concurrently in a single message. Pass each agent the full diff so it has the complete context.
+
+### Agent 1: Code Reuse Review
+
+For each change:
+
+1. **Search for existing utilities and helpers** (`rg`) that could replace newly written code. Common locations: `virex_bench/utils.py`, `logger.py`, per-package `common.py` (e.g. `strategies/cr/common.py`, `strategies/tot/common.py`), and files adjacent to the changed ones.
+2. **Flag any new function that duplicates existing functionality.** Suggest the existing function to use instead — e.g. reimplementing `premises_to_text`, ad-hoc logger setup instead of `init_logger`, or a new registry lookup that duplicates `get_task` / `get_strategy` / `get_decoding`.
+3. **Flag any inline logic that could use an existing utility** — hand-rolled string manipulation, manual `os.path` handling, direct `os.environ` reads that should go through `virex_bench/envs.py`, and similar patterns are common candidates.
+
+### Agent 2: Code Quality Review
+
+Review the same changes for hacky patterns:
+
+1. **Redundant state**: state that duplicates existing state, cached values that could be derived
+2. **Parameter sprawl**: adding new parameters to a function instead of generalizing or restructuring existing ones; small helper methods that are referenced only once (inline them)
+3. **Copy-paste with slight variation**: near-duplicate code blocks that should be unified with a shared abstraction
+4. **Leaky abstractions**: exposing internal details that should be encapsulated, or breaking existing abstraction boundaries (e.g. bypassing the task/model/strategy/decoding registries, or reaching into a `dspy.Module`'s internals)
+5. **Stringly-typed code**: raw strings/dicts where a `pydantic.BaseModel`, enum, or string-union already exists; `dataclasses` where the codebase uses `pydantic.BaseModel`
+6. **Style mismatches**: truthy/falsy checks (`if x`) where `is None` / `is not None` is required for valid falsy values; `pathlib.Path` where the codebase uses `str` + `os.path`; short/abbreviated names (`ri`, `ip`, `el`) where verbose self-documenting names are the convention; `%`-style logging instead of f-strings; `print()` for diagnostics instead of the shared logger; `type[T]` misuse (passing an instance where a class is expected, e.g. `signature: type[dspy.Signature]`)
+7. **DSPy convention drift**: new strategies that aren't `dspy.Module` subclasses or don't follow the existing signature/module patterns in `strategies/`; signature docstrings or field descriptions written in a language other than **English** (task data stays native, but the instruction contract is English)
+8. **Unnecessary or over-verbose comments**: comments explaining WHAT the code does (well-named identifiers already do that), narrating the change, or referencing the task/caller — delete these; keep only non-obvious WHY (hidden constraints, subtle invariants, workarounds). For comments worth keeping, trim multi-line/rambling explanations down to a single concise line.
+
+### Agent 3: Efficiency Review
+
+Review the same changes for efficiency:
+
+1. **Unnecessary work**: redundant computations, repeated file reads, duplicate LM/API calls, N+1 patterns
+2. **Missed concurrency**: independent operations run sequentially when they could run in parallel (e.g. per-example LM calls, independent propose/evaluate steps in ToT/CR)
+3. **Hot-path bloat**: new blocking work added to the per-example evaluation loop or per-LM-call path
+4. **Unnecessary existence checks**: pre-checking file/resource existence before operating (TOCTOU anti-pattern) — operate directly and handle the error
+5. **Memory**: unbounded data structures, missing cleanup (e.g. accumulating full LM transcripts across a large eval run)
+6. **Overly broad operations**: reading entire files/datasets when only a portion is needed, loading all items when filtering for one
+
+## Phase 3: Fix Issues
+
+Wait for all three agents to complete. Aggregate their findings and fix each issue directly. If a finding is a false positive or not worth addressing, note it and move on — do not argue with the finding, just skip it.
+
+After fixing, confirm the changed files still pass the project's checks:
+
+```bash
+uv run --no-sync ruff check path/to/file.py
+uv run --no-sync ruff format path/to/file.py
+```
+
+Pyright runs in strict mode — do not introduce type errors, and only suppress with `# pyright: ignore` for genuine false positives, never to silence a legitimate error. If the change touches behavior, also run the relevant tests (`uv run --no-sync pytest tests/...`, which mirror the package layout).
+
+When done, briefly summarize what was fixed (or confirm the code was already clean).

virex_bench-0.1.2/.editorconfig

@@ -0,0 +1,31 @@
+# https://editorconfig.org/
+
+root = true
+
+[*]
+indent_style = space
+indent_size = 4
+
+end_of_line = lf
+charset = utf-8
+trim_trailing_whitespace = true
+insert_final_newline = true
+
+[*.md]
+trim_trailing_whitespace = false
+
+[*.py]
+max_line_length = 100
+
+[*.sh]
+indent_size = 2
+
+[*.{json,yaml,yml}]
+indent_size = 2
+
+[{Makefile,makefile}]
+indent_style = tab
+indent_size = 2
+
+[*.txt]
+insert_final_newline = false

virex_bench-0.1.2/.gitattributes

@@ -0,0 +1,3 @@
+* text=auto eol=lf
+*.ipynb linguist-generated
+pyrightconfig.json linguist-language=jsonc

virex_bench-0.1.2/.github/workflows/check_imports.py

@@ -0,0 +1,51 @@
+#!/usr/bin/env python3
+
+import argparse
+import importlib.util
+import pathlib
+import sys
+
+
+def check_imports(args: argparse.Namespace) -> None:
+    src_path = pathlib.Path(args.src_path)
+    num_files = 0
+    for pyfile in src_path.rglob("*.py"):
+        print("Trying to import", pyfile)
+        module_name = pyfile.with_suffix("").as_posix().replace("/", ".")
+        spec = importlib.util.spec_from_file_location(
+            name=module_name,
+            location=pyfile,
+        )
+        if spec is None:
+            print(f"Failed to load {pyfile}", file=sys.stderr)
+            raise SystemExit(1)
+
+        assert spec.loader is not None
+        module = importlib.util.module_from_spec(spec)
+        try:
+            spec.loader.exec_module(module)
+            num_files += 1
+        except Exception as err:
+            print(f"Error importing {module_name}: {err}", file=sys.stderr)
+            raise SystemExit(1) from err
+
+    print(f"All {num_files} imports checked successfully.")
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(
+        description="Check imports in Python files",
+        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
+    )
+    parser.add_argument(
+        "--src_path",
+        type=str,
+        help="Path to the source directory",
+        default="src",
+    )
+    args = parser.parse_args()
+    check_imports(args)
+
+
+if __name__ == "__main__":
+    main()

virex_bench-0.1.2/.github/workflows/lint-and-test.yml

@@ -0,0 +1,68 @@
+name: Lint and Test
+
+on:
+  push:
+    branches: [main, dev]
+  pull_request:
+    branches: [main, dev]
+  workflow_dispatch:
+
+jobs:
+  check-imports:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.12"]
+    steps:
+      - name: Clone the repository
+        uses: actions/checkout@v7
+      - name: Install the latest version of uv and set the python version
+        uses: astral-sh/setup-uv@d31148d669074a8d0a63714ba94f3201e7020bc3 # v8.3.0
+        with:
+          version: ">=0.9.21"
+          python-version: ${{ matrix.python-version }}
+          enable-cache: true
+      - name: Check imports
+        run: |
+          export PYTHONPATH="${{ github.workspace }}"
+          uv run --frozen python .github/workflows/check_imports.py --src_path virex_bench
+
+  lint-with-ruff:
+      runs-on: ubuntu-latest
+      steps:
+        - name: Clone the repository
+          uses: actions/checkout@v7
+
+        - name: Install the latest version of uv and set the python version
+          uses: astral-sh/setup-uv@d31148d669074a8d0a63714ba94f3201e7020bc3 # v8.3.0
+          with:
+            version: ">=0.9.21"
+            enable-cache: true
+
+        - name: Lint with ruff
+          run: |
+            uv run --frozen ruff check --output-format=github virex_bench tests scripts
+
+        - name: Check formatting with ruff
+          run: |
+            uv run --frozen ruff format --check --output-format=github virex_bench tests scripts
+
+  test-with-pytest:
+      runs-on: ubuntu-latest
+      strategy:
+        matrix:
+          python-version: ["3.12", "3.13", "3.14"]
+      steps:
+        - name: Clone the repository
+          uses: actions/checkout@v7
+
+        - name: Install the latest version of uv and set the python version
+          uses: astral-sh/setup-uv@d31148d669074a8d0a63714ba94f3201e7020bc3 # v8.3.0
+          with:
+            version: ">=0.9.21"
+            python-version: ${{ matrix.python-version }}
+            enable-cache: true
+
+        - name: Run tests with pytest
+          run: |
+            uv run --frozen pytest tests -v

virex_bench-0.1.2/.github/workflows/release.yml

@@ -0,0 +1,71 @@
+name: Release
+
+on:
+  release:
+    types: [published]
+
+permissions: {}
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Clone the repository
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+        with:
+          # hatch-vcs derives the version from git tags, so full history is required.
+          fetch-depth: 0
+
+      - name: Install the latest version of uv
+        uses: astral-sh/setup-uv@d31148d669074a8d0a63714ba94f3201e7020bc3 # v8.3.0
+        with:
+          version: ">=0.9.21"
+          enable-cache: true
+
+      - name: Build sdist and wheel
+        run: uv build
+
+      - name: Check distribution metadata
+        run: uvx twine check dist/*
+
+      - name: Upload distribution artifacts
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: dist
+          path: dist/
+
+  publish-testpypi:
+    needs: build
+    if: github.event.release.prerelease == true
+    runs-on: ubuntu-latest
+    environment: testpypi
+    permissions:
+      id-token: write
+    steps:
+      - name: Download distribution artifacts
+        uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
+        with:
+          name: dist
+          path: dist/
+
+      - name: Publish to TestPyPI
+        uses: pypa/gh-action-pypi-publish@cef221092ed1bacb1cc03d23a2d87d1d172e277b # v1.14.0
+        with:
+          repository-url: https://test.pypi.org/legacy/
+
+  publish-pypi:
+    needs: build
+    if: github.event.release.prerelease == false
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      id-token: write
+    steps:
+      - name: Download distribution artifacts
+        uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
+        with:
+          name: dist
+          path: dist/
+
+      - name: Publish to PyPI
+        uses: pypa/gh-action-pypi-publish@cef221092ed1bacb1cc03d23a2d87d1d172e277b # v1.14.0

virex_bench-0.1.2/.gitignore

@@ -0,0 +1,66 @@
+# python stuff
+__pycache__
+.ipynb_checkpoints
+
+# virtual environments
+.venv
+.venv.*
+.venv-*
+
+# env
+.env
+.env.*
+
+# unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+.tmp/
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# local stuff
+_*
+!__*__.py
+!/virex_bench/types/_*
+/checkpoints
+*.local
+# /.plans
+/.code-review
+/results
+/slides
+/notebooks
+/tmp/
+/report/
+/.bashrc
+/final_results/
+
+# datasets
+/data
+
+# log files
+*.log
+*.logs
+
+# cache
+.pytest_cache
+.ruff_cache
+
+# shell scripts
+s.sh
+
+# other
+*.bak
+*.backup
+/*.pdf
+/*.json
+!/pyrightconfig.json
+/refs

virex_bench-0.1.2/.ignore

@@ -0,0 +1,4 @@
+# logs file
+!*.log
+!/logs/
+!/log

virex_bench-0.1.2/.python-version

@@ -0,0 +1 @@
+3.12

virex_bench-0.1.2/AGENTS.md

@@ -0,0 +1,169 @@
+# AGENTS.md
+
+## Project
+
+**ViREx-Bench** (Vietnamese Reasoning Exploration Benchmark) is a lightweight framework
+for exploring **multi-step reasoning of LLMs for Vietnamese**.
+
+The project has two expected outputs:
+
+1. **A Vietnamese logical-reasoning dataset** — each row contains a set of *premises* and a
+   *question* (with a gold answer), framing logical-reasoning problems in Vietnamese.
+2. **An evaluation framework** that measures how much each inference-time reasoning strategy
+   improves over a direct-answer baseline on that dataset.
+
+The framework targets two orthogonal research axes (planned scope listed; see each
+registry for what is currently implemented):
+
+- **Prompting / inference-time scaling** (`strategies/`): Chain-of-Thought (CoT),
+  Tree-of-Thought (ToT) with pluggable search (beam, DFS, MCTS), Cumulative Reasoning
+  (CR), Self-Consistency, and Program-of-Thought with symbolic reasoning (Z3 solver).
+  *Implemented: direct baseline, CoT, CR, ToT (beam + DFS + MCTS search).*
+- **Decoding** (`decoding/`): the baseline (single candidate, or multiple candidates
+  aggregated by majority vote), Best-of-N with a verifier, and speculative decoding to
+  speed up inference. *Implemented: single-pass baseline, self-consistency (N-sample
+  majority vote + LLM aggregation).*
+
+**DSPy** is the main framework for implementing reasoning strategies. The CLI design is
+modeled on [`mteb`](https://github.com/embeddings-benchmark/mteb) — a CLI is sufficient
+(no web/leaderboard UI or serving layer).
+
+## Commands
+
+Always prefix Python/tool commands with `uv run --no-sync`, omit the `--no-sync` flag only when you actually needed to avoid syncing packages everytime (which take times).
+
+```bash
+# Lint
+uv run --no-sync ruff check
+
+# Format
+uv run --no-sync ruff format
+
+# Run the benchmark CLI (task × model × strategy × decoding)
+# Models are served behind an OpenAI-compatible endpoint (no local loading).
+uv run virex-bench run \
+    --model Qwen/Qwen3.5 \
+    --task vietnamese-logical-reasoning \
+    --strategy cot \
+    --backend openai \
+    --api-base http://localhost:8000/v1 \
+    --output-dir results
+
+# `vb` is a shorter alias for `virex-bench`
+uv run vb run --model Qwen/Qwen3.5 --strategy cot
+
+# Inspect available tasks / strategies
+uv run virex-bench tasks --list
+uv run virex-bench strategies --list
+```
+
+### File-scoped fast feedback
+
+```bash
+uv run --no-sync ruff check path/to/file.py
+uv run --no-sync ruff format path/to/file.py
+```
+
+## Project Structure
+
+The package is organized around four axes — **task** (dataset), **model** (backend),
+**strategy** (prompting), and **decoding** (aggregation) — wired together by the
+evaluation orchestrator. Layout is inspired by `mteb` (task × model registries + a thin
+CLI), with two extra axes for reasoning and decoding.
+
+```
+virex_bench/
+├── cli/            # argparse CLI: `virex-bench run | tasks | strategies`
+│   └── build_cli.py #  build_parser / main + subcommand handlers
+├── envs.py         # centralized, lazily-read environment variables (vLLM-style)
+├── logger.py       # init_logger(__name__) — shared logging setup
+├── utils.py        # shared helpers (premises_to_text — numbered premise block)
+├── types/          # pydantic data models + type aliases (mirrors mteb/types)
+│   ├── __init__.py #   re-exports all public types
+│   ├── _task.py    #   ReasoningExample, DatasetConfig, TaskMetadata
+│   ├── _result.py  #   TaskResult, EvaluationReport, JudgeOutcome
+│   └── _metric.py  #   ReasoningMetric callable alias
+├── tasks/          # reasoning datasets + registry
+│   ├── base.py     #   ReasoningTask base class
+│   ├── registry.py #   get_task / list_tasks
+│   └── vietnamese_logical_reasoning.py
+├── models/         # LM layer — OpenAI-compatible endpoints via dspy.LM
+│   ├── __init__.py #   get_model
+│   ├── base.py     #   BaseLM(dspy.LM) wrapper
+│   └── backends.py #   load_backend: build a BaseLM for an OpenAI-compatible endpoint
+├── strategies/     # PROMPTING / inference-time scaling (dspy.Module subclasses)
+│   ├── base.py     #   ReasoningStrategy base class (+ accepts_variant flag)
+│   ├── registry.py #   get_strategy / list_strategies + composite-name parsing (direct, cot, cr, tot[-beam|-dfs|-mcts])
+│   ├── direct.py   #   baseline: direct answer
+│   ├── cot.py      #   Chain-of-Thought
+│   ├── modules.py  #   reusable dspy modules: ChainOfThought, DualTask2ChainOfThought, ThinkingCaptureLM
+│   ├── cr/         #   Cumulative Reasoning strategy package (Zhang et al., 2023)
+│   │   ├── strategy.py # CRStrategy — propose/verify accumulation loop + solver
+│   │   └── common.py   # shared signatures + helpers (proposer/verifiers, CRConfig, verdict buckets)
+│   └── tot/        #   Tree-of-Thoughts strategy package (Yao et al., 2023)
+│       ├── strategy.py # ToTStrategy — propose/evaluate/aggregate over a thought tree
+│       ├── common.py  # shared signatures + helpers (proposer/evaluator, scoring, dedupe)
+│       └── search/     # pluggable search algorithms (beam, DFS, MCTS)
+│           ├── __init__.py # search registry: build_search / list_search
+│           ├── base.py     # SearchConfig, SearchResult, ThoughtSearch ABC
+│           ├── beam.py     # BeamSearch (ToT's BFS)
+│           ├── dfs.py      # DFSSearch (backtracking + value pruning)
+│           └── mcts.py     # MCTSSearch (Monte-Carlo Tree Search)
+├── decoding/       # DECODING / answer aggregation
+│   ├── base.py     #   DecodingStrategy base + SinglePass (single-candidate baseline)
+│   ├── registry.py #   get_decoding / list_decoding
+│   └── self_consistency.py # SelfConsistency: N-sample majority vote + LLM aggregation
+├── evaluation/     # orchestration + scoring + LLM-as-a-judge
+│   ├── evaluate.py #   task × model × strategy loop + report serialization (save_report)
+│   ├── metrics.py  #   accuracy metrics + METRIC_REGISTRY / get_metric / judge_example
+│   └── judge.py    #   LLMJudge base, LogicalReasoningJudge, JUDGE_REGISTRY / build_judge
+└── __main__.py     # `python -m virex_bench` entry point -> cli.main
+
+data/               # datasets (managed externally — do not edit by hand)
+notebooks/          # dataset-prep notebooks (e.g. eval-round → HF dataset conversion)
+scripts/            # dataset translation helpers + model-serving launchers (serving/)
+tests/              # pytest suite — mirrors the package layout (cli / evaluation / strategies [+ cr, tot])
+results/            # benchmark run outputs
+```
+
+## Development and Code Style
+
+### Development Guide
+
+- **Python 3.12** — use modern syntax (`str | None`, `list[int]`, etc.)
+- **Git**: Do not try to use `git checkout <file>` to discard or revert changes as you might cause loose uncommitted work.
+- **Formatter**: Ruff (`uv run --no-sync ruff format`) — double quotes, 4-space indent, LF endings, line length 99
+- **Linter**: Ruff (`uv run --no-sync ruff check`) — rules: B, C, E, F, I, W, RUF013, UP006
+- **Type checker**: Pyright in strict mode. Use `# pyright: ignore` to suppress false positive or workaround some limitation in the type checker. DO NOT silence legitimate type errors.
+- **Testing**: pytest (`uv run --no-sync pytest`)
+- **Imports**: sorted by Ruff (isort rules), grouped: stdlib → third-party → local
+- **DSPy**: Reasoning strategies are implemented as `dspy.Module` subclasses under `strategies/`. Follow the existing patterns there for signatures and modules. Write signature docstrings and field descriptions in **English** (the instruction language), even for Vietnamese tasks — the target models are multilingual, and English instructions keep the prompt contract consistent across tasks. The task data (premises/questions/answers) stays in its native language.
+- **Grepping**: Prefer using `rg` (ripgrep) over `grep` for speed and better defaults.
+
+### Code Style
+
+- **Pydantic**: Use `pydantic.BaseModel` for data models (examples, configs, results, reports) — not `dataclasses`. Serialize with `model_dump()` / `model_dump_json()`.
+- **Logging**: Use the shared logger — `from virex_bench.logger import init_logger` then `logger = init_logger(__name__)` at module top. Prefer f-strings in logging calls over `%`-style logger formatting. Do not call `print()` for diagnostics or configure `logging` directly (CLI user-facing output via `print` is fine).
+- **Env vars**: All environment variables go through `virex_bench/envs.py` — never read `os.environ` directly elsewhere. Add a typed declaration in the `TYPE_CHECKING` block and a lazy `lambda` entry in `environment_variables`, then access via `from virex_bench import envs` / `envs.VAR_NAME`. Access is lazy (read at use time, not import time), so set vars before first access.
+- **CLI**: `argparse`-based subcommands (`run`, `tasks`, `strategies`), modeled on `mteb`. Both `virex-bench` and the short alias `vb` map to `cli:main`. Keep command handlers thin — delegate to `evaluation/`, `tasks/`, `models/`.
+- **File path:** Use raw `str` for file paths and `os.path` for path manipulations. Avoid `pathlib.Path` to keep things simple and consistent across the codebase.
+- **Variable names:** Use verbose, self-documenting names. Avoid short abbreviations (e.g., `ri`, `ip`, `ua`, `ui`, `el`, `t`) — prefer `request_metadata`, `client_ip`, `client_user_agent`, `elapsed`, etc. Single-letter names are only acceptable as loop indices or in very narrow local scopes (e.g., list comprehensions).
+- **None check:** Use `if x is None` and `if x is not None` for checking if a variable is `None`. Avoid truthy/falsy checks like `if x` or `if not x` when the variable can have valid falsy values (e.g., empty string, zero, empty list).
+- **`type[T]` hints:** Use `type[T]` when a function/field expects a *class* (to instantiate or inspect later), not an instance — e.g. `signature: type[dspy.Signature]` and registries like `dict[str, type[ReasoningStrategy]]`. Pass the class itself (`DirectStrategy`), not an instance (`DirectStrategy()`).
+- **Comments:** Do not add decorative separator/banner comments — no `# --- text ---`, `# =====`, or full-line `# ---------` dividers between sections. If a section genuinely needs a title, use a plain `# Title` comment (no surrounding dashes/equal signs); otherwise rely on PEP 8's two blank lines between top-level definitions. Only add comments to explain non-obvious logic — never to delimit regions.
+- Do not create small helper methods that are referenced only once.
+
+## Code review
+
+When asked for reviewing changes, focus on:
+- Potential bugs introduced by the change
+- Check for behavior mismatches for refactoring changes
+- Code quality and readability (style, structure, naming, etc should match the existing codebase and be clear)
+- Ignore directories and files starting with `_` (e.g., `_testing_results/`, `_test_format.py`) as they are testing or local files and not part of the main codebase.
+
+## Boundaries
+
+- **Never commit** `.env` files or API keys/secrets
+- **Do not edit** files under `data/` directly — datasets are managed externally
+- **Do not modify** generated/cached directories: `.ruff_cache/`, `.cache/`, `_testing_results/`
+- **Do not touch** `uv.lock` by hand

virex_bench-0.1.2/CLAUDE.md

@@ -0,0 +1 @@
+AGENTS.md