gradetrail 0.1.0__tar.gz

gradetrail-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,33 @@
+name: ci
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.11", "3.12"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - run: pip install -e ".[dev]"
+      - run: ruff check .
+      - run: ruff format --check .
+      - run: pytest -q -m "not ray"
+
+  test-ray:
+    runs-on: ubuntu-latest
+    continue-on-error: true
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+      - run: pip install -e ".[dev,ray]"
+      - run: pytest -q -m ray

gradetrail-0.1.0/.gitignore

@@ -0,0 +1,9 @@
+__pycache__/
+*.egg-info/
+.pytest_cache/
+.ruff_cache/
+dist/
+.venv/
+*.sqlite
+results/
+.claude/

gradetrail-0.1.0/CLAUDE.md

@@ -0,0 +1,52 @@
+# reproeval
+
+A distributed LLM evaluation harness that treats evals like tests: declarative, cached, reproducible.
+
+## What this project is
+
+A Python library + CLI. A user defines an eval in a YAML spec (dataset, prompt template, model, scorer), runs it with `reproeval run spec.yaml`, and gets scored results plus a reproducibility manifest. Runs are cached, multi-provider, and can execute locally (asyncio) or distributed (Ray) behind the same interface.
+
+Target user: an ML researcher who wants eval results in 30 seconds of setup, not a platform to administer.
+
+## Architecture
+
+Modules and their single responsibility. Do not blur these boundaries.
+
+- `reproeval/spec.py` - Eval spec dataclasses + YAML loading/validation. The spec is the public contract. NEVER change the spec schema without asking me first.
+- `reproeval/providers/` - Provider abstraction. One base class, one module per provider (`anthropic.py`, `openai.py`, `openai_compatible.py`). ALL model API calls go through this layer. No direct HTTP/SDK calls anywhere else in the codebase.
+- `reproeval/scorers/` - Scoring. `deterministic.py` (exact match, regex, numeric tolerance) and `judge.py` (LLM-as-judge). Judge prompts are versioned artifacts referenced by the spec, never inlined ad hoc.
+- `reproeval/cache.py` - SQLite response cache. Key = sha256 of canonical JSON of (provider, model, resolved prompt, sampling params). Cache logic lives here and only here.
+- `reproeval/runner/` - Execution. `base.py` defines the Runner interface, `local.py` (asyncio + semaphore concurrency), `ray_runner.py` (Ray backend, added in week 3). Runners are swappable via config, callers never know which one they got.
+- `reproeval/manifest.py` - Run manifests: spec hash, model versions, timestamps, seed, git SHA, package version. Every run writes one.
+- `reproeval/results.py` - Results model + JSONL writing + summary stats (scores, cost, tokens, failure rate).
+- `reproeval/cli.py` - Thin CLI (typer). No business logic here, it only wires modules together.
+
+Design docs live in `docs/design/`. Read the relevant one before implementing a module.
+
+## Conventions
+
+- Python 3.11+. Type hints on every function signature. `from __future__ import annotations` at the top of each module.
+- Prefer functions and frozen dataclasses over classes with mutable state. Only use a class when there is real state or a real interface (providers, runners).
+- Errors: raise specific exceptions from `reproeval/errors.py`. Never bare `except:`. Never swallow exceptions silently.
+- Logging: `structlog`, structured key-value logs. Every provider call logs model, latency_ms, tokens, cached (bool), outcome. No print() outside the CLI.
+- Async: provider calls are async. Do not mix sync SDK calls into async paths, use the async clients.
+- Dependencies: keep them minimal. Ask before adding any new dependency. Current allowed set: pydantic, typer, structlog, anthropic, openai, pyyaml, aiosqlite, ray (week 3 only).
+- Tests: pytest + pytest-asyncio. Tests live in `tests/` mirroring the package layout. Mock at the provider boundary using recorded fixtures, never hit real APIs in tests.
+- Style: ruff for lint + format, config in pyproject.toml. Line length 100.
+
+## Workflow rules
+
+- Before writing code, state your plan in a few bullets and wait for my confirmation if the change touches more than one module.
+- Write or update tests first for any new behavior. I will review tests before you implement.
+- Run `ruff check . && pytest -q` before declaring anything done. If tests fail, fix them, do not weaken assertions to make them pass.
+- Small commits with imperative messages ("add sqlite cache keyed on request hash"). Never combine a refactor and a feature in one commit.
+- If you make a non-obvious design decision mid-task, append one line explaining it to `NOTES.md`.
+- Do not edit `docs/design/*` unless I explicitly ask. Those are my specs, not living docs.
+- Do not add a web UI, dashboard, or extra providers. Out of scope for v0.1 no matter how tempting.
+
+## Definition of done for any task
+
+1. `ruff check . && ruff format --check . && pytest -q` all passing, matching CI exactly.
+2. Public functions have docstrings (one line + args if non-obvious).
+3. NOTES.md updated if a decision was made.
+4. No TODOs left in code without an issue-style note in NOTES.md.

gradetrail-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Chitvan Patel
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

gradetrail-0.1.0/NOTES.md

@@ -0,0 +1,49 @@
+# Decision log
+
+- 2026-07-06: Regex patterns are validated structurally at load by substituting a probe value for jinja placeholders; field existence is checked separately in validate_against_dataset(). Keeps "fail fast on sample 0" without needing the dataset at spec-load time.
+- 2026-07-06: base_dir is carried on EvalSpec (set by load_spec) so relative dataset/judge paths resolve against the spec file, not the CWD. Excluded from identity hash.
+- 2026-07-06: SpecError messages surface only the first pydantic error, formatted as "field: fix". Full error list is overkill for a CLI tool.
+- 2026-07-06: ResponseCache.put() is an upsert (INSERT ... ON CONFLICT DO UPDATE) — last write for a given key wins, rather than raising on a duplicate key. The cache is a value store keyed by request identity, not an audit log; a re-run of the same resolved request (e.g. after a partial failure) should simply refresh the entry, not require callers to delete-then-insert.
+- 2026-07-06: cache.py opens SQLite with `PRAGMA journal_mode=WAL` and `PRAGMA busy_timeout=5000` so multiple connections to the same cache file (e.g. concurrent runner workers) don't fail with "database is locked" under normal contention.
+- 2026-07-06: created_at is generated in Python (`datetime.now(timezone.utc).isoformat()`) and stored as TEXT, not SQLite's `CURRENT_TIMESTAMP`, so the manifest gets microsecond-resolution, timezone-aware timestamps instead of second-resolution UTC strings.
+- 2026-07-06: `ResponseCache.connect()` verifies WAL mode actually took (`PRAGMA journal_mode` return value), not just fire-and-forget — some filesystems (network mounts, some containers) silently stay in rollback-journal mode, which would make `busy_timeout` do far less than expected. Raises `CacheError` if WAL can't be enabled, rather than continuing in a mode the concurrency guarantees weren't designed for.
+- 2026-07-06: `ResponseCache.put()` requires the response to be JSON-serializable as-is (no `default=str` coercion) — get() must return exactly what was put(), and silently stringifying e.g. a stray datetime would make that contract quietly false. Raises `CacheError` on a `TypeError` from `json.dumps` instead.
+- 2026-07-06: the single-connection concurrency test (`test_concurrent_get_and_put_do_not_corrupt_cache`) is really a smoke test — `aiosqlite` serializes all operations on one connection onto one background thread, so it can't exercise real lock contention. The two-connection test is the one that actually exercises WAL + busy_timeout under concurrent writers. True multi-process contention (multiple OS processes hitting one cache file) isn't covered yet; that gets exercised for real once the Ray runner lands in week 3.
+- 2026-07-06: `AnthropicProvider` constructs the real `anthropic.AsyncAnthropic` client with `max_retries=0`. The SDK has its own built-in retry policy; `Provider.complete()` in `base.py` is the only retry loop we want. Leaving the SDK's default retries on would compound with ours silently — e.g. `run.max_retries=5` here becomes up to 5x the SDK's own retries under the hood, and `run.timeout_s` no longer means what the spec says it means. Every future provider module must construct its client the same way.
+- 2026-07-06: `Provider.complete()` catches `TimeoutError` (from `asyncio.wait_for`) and `Exception` (dispatched to `classify()`), deliberately not `BaseException`. `asyncio.CancelledError` is a `BaseException` subclass (since Python 3.8), so it's never classified and never burns a retry — it propagates immediately out of `complete()`. This matters once the runner drives many concurrent `complete()` calls under a semaphore: cancelling one in flight (timeout, shutdown, a sibling task failing) must not get silently swallowed or retried. Confirmed empirically that an externally-cancelled task surfaces `CancelledError` through `wait_for`, not `TimeoutError` — the two are easy to conflate but the SDK/asyncio keep them distinct.
+- 2026-07-06: `ProviderResponse.model` is the model name as reported by the API response, not the requested `model.name` from the spec. These can differ (e.g. an alias resolving to a dated snapshot). This discrepancy is exactly what the run manifest (manifest.py, not yet built) should record — both the requested and served model belong in the reproducibility record.
+- 2026-07-06: `validate_against_dataset()` now also checks `ExactScorer.target_field` against sample 0 (docs/design/eval-spec.md rule 3 updated first). This was a real gap, not a hypothetical: only `prompt` and `scorer.pattern` were checked before, so a typo'd `target_field` would pass spec load and only surface as a `KeyError` deep in `score_exact()` at scoring time — exactly the class of bug fail-fast-at-load exists to prevent. Separate change from the scorers work below; keep it in its own commit once this repo has version control.
+- 2026-07-06: `score_judge()`'s `detail` field on a `judge_error` is the only debugging surface once a run summary reduces to error counts, so it's built deliberately: a `ProviderError` (judge model unreachable) includes `str(exc)` plus its chained `__cause__`; a parse failure (judge replied garbage) includes a truncated copy (~200 chars) of the raw reply, embedded directly in the `JudgeError` message by `_parse_reply()`. These are two different failure modes that both collapse into the same `state="judge_error"` — `detail` is what lets someone tell them apart after the fact.
+- 2026-07-06: `score_judge()` with `samples > 1` short-circuits on the first `judge_error` rather than running all `samples` and reporting partial agreement. This is a real methodology choice, not just an implementation shortcut: an alternative design would run every sample regardless of individual failures and report e.g. "2/3 judges agreed, 1 errored," which is defensible for noisy judge setups. Short-circuiting is simpler and cheaper (stops wasting API calls immediately) and is the right default for v0.1; revisit if judge flakiness in practice makes partial-agreement reporting worth the complexity.
+- 2026-07-06: `summarize()` raises `ResultsError` on any `SampleResult.state` outside `{"scored", "provider_error", "judge_error"}`, naming the sample_id and the bad state. `state` is a plain string on a dataclass (no pydantic validation at construction), and the counts are typed fields (`n_scored`/`n_provider_error`/`n_judge_error`), not a dict keyed by whatever string shows up — so a typo'd or future state would otherwise be silently dropped from every counter (`n_samples != n_scored + n_provider_error + n_judge_error`) and nobody would notice until a report's numbers looked wrong. Same loud-failure-over-silent-corruption principle as `ResponseCache.put()`'s `default=str` fix.
+- 2026-07-06: `ruff` is pinned to `~=0.15.20` (matching what CI installs) instead of a loose `>=0.4`, and the "Definition of done" in CLAUDE.md now runs `ruff format --check .` in addition to `ruff check .` — CI caught five already-`ruff check`-clean files that had never been run through `ruff format`, because the local done-criteria didn't check formatting at all.
+- 2026-07-06: `EvalSpec.load_samples()` now assigns the fallback sample id (the 1-based original file line number) at parse time, before shuffle/limit, whenever `dataset.id_field` is absent from a row. Previously this fallback would have had to be approximated by whatever consumes the loaded list (e.g. by position in the returned, possibly-shuffled list) — which breaks the moment `shuffle_seed` is set, since the same sample then gets a different id under a different seed, defeating the entire purpose of an id (cross-run result comparison). The design doc already promised "falls back to line number if absent," so this was a doc violation, not a judgment call. No signature change to `load_samples()`.
+- 2026-07-06: the default provider factory in `runner/local.py` raises plain `ReproevalError` (not `SpecError`) for an unimplemented `model.provider` (currently only `"anthropic"` exists). `SpecError` must always mean "your spec is invalid" — a user hitting this for `provider: openai` has a perfectly valid spec per the schema, the feature just doesn't exist yet in this version. Reusing `SpecError` here would send them hunting for a typo that isn't there. Error taxonomy is part of the user-facing API surface, not an implementation detail.
+- 2026-07-06: the run manifest's `served_models` is tracked on `LocalRunner` as a `set[str]` populated during `run()`, not stored per-sample on `SampleResult` — the alternative considered and passed on for v0.1. A `served_model` field on `SampleResult` would give per-sample granularity, which is what would actually catch a provider silently swapping model versions mid-run (the manifest's run-level set can't distinguish "every sample used model X" from "one sample used Y instead"). That's a plausible v0.2 item once results-diffing tooling exists to make use of it; logging the tradeoff now so it doesn't need re-deriving later.
+  **Superseded 2026-07-08**: promoted to `SampleResult.served_model: str | None` (populated on scored samples from `response.model`, including cache hits since the cache preserves it; `None` on error states). The Ray runner is what forced this — a `LocalRunner`-instance-attribute collector was always an in-process hack, and Ray proved it structurally rather than just aesthetically: mutable shared state genuinely doesn't survive a process boundary, so a Ray runner would have had no way to populate it via the old mechanism at all. `write_manifest`'s signature is unchanged; `cli.py` now derives `served_models` as `{r.served_model for r in results if r.served_model}`, computed identically for every backend. `LocalRunner.served_models` is gone; the shared per-sample pipeline (see the Ray runner entry below) never needed a collector parameter for this at all — one mechanism instead of two.
+- 2026-07-06: `cli.py` needs an (empty, docstring-only) `@app.callback()` on the Typer app even though there's only one command (`run`). Typer collapses a single-`@app.command()` app into "no subcommand name needed" mode (`len(registered_commands) == 1` and no callback and no sub-groups, per `typer/main.py:get_command`), so `reproeval run spec.yaml` would otherwise parse "run" as the value for `spec_path` and fail with a usage error. A callback (any callback, even a no-op) forces proper group/subcommand dispatch. Worth remembering the moment a second command is ever added — the callback stops being load-bearing then, but doesn't hurt to keep.
+- 2026-07-07: `reproeval/providers/openai.py` and `openai_compatible.py` added, so the "unimplemented provider" `ReproevalError` path noted on 2026-07-06 no longer exists — `model.provider` is a closed `Literal["anthropic", "openai", "openai_compatible"]`, and the default provider factory in `runner/local.py` now dispatches all three exhaustively (with `assert`s narrowing the final `openai_compatible` branch, not a fallback `raise`). The taxonomy reasoning in that earlier entry stays correct history, just no longer has a code path attached to it.
+- 2026-07-07: `OpenAICompatibleProvider` subclasses `OpenAIProvider` instead of duplicating `_complete`/`classify` — the two are identical except that `base_url` is required for the compatible path instead of absent. Accepted tradeoff: any future OpenAI-specific behavior change (e.g. the `max_tokens` rename below, a new response field, a model-family quirk) must be deliberately checked against compatible servers too, since it now changes both providers whether or not that's correct for both. Chosen over a literal copy because copies drift silently; this couples them visibly instead.
+- 2026-07-07: `OpenAIProvider._complete()` treats a missing `completion.usage` (nullable per the SDK's own type; some OpenAI-compatible servers, e.g. certain vLLM/proxy configurations, omit it on non-streaming calls) as `input_tokens=output_tokens=0` plus a structured `usage_missing` warning log (model), rather than letting the `AttributeError` surface. Per the runner's per-sample isolation, an unguarded `completion.usage.prompt_tokens` would turn a perfectly good, scoreable response into a mysterious `provider_error` on every sample from that server. Cost is already `None` for any (provider, model) pair not in `results.PRICING`, which covers every compatible-server model by construction, so zero tokens doesn't quietly misstate cost either.
+- 2026-07-07: `OpenAIProvider._complete()` sends `max_tokens`, not the newer `max_completion_tokens`. Deliberate for v0.1: `max_tokens` works on current mainstream OpenAI chat models and on OpenAI-compatible servers (vLLM et al.), most of which don't support the renamed param yet; OpenAI's own reasoning models reject `max_tokens` outright and will need it renamed behind a version/model check. Not building that conditional now — logging it so the gap is a known deferral, not a surprise when someone points this provider at a reasoning model.
+- 2026-07-08: Ray remote *functions* (not actors) reject `async def` outright — confirmed empirically against ray 2.56: `ValueError: 'async def' should not be used for remote tasks`. This isn't in most training data as a gotcha (Ray's actor docs cover async extensively; plain `@ray.remote` functions being sync-only is easy to miss). `ray_runner.py`'s `_run_batch` stays `async def` (so it can `await run_one_sample(...)` like every other caller of the shared pipeline); `_run_batch_sync` wraps it in `asyncio.run()` and is what actually gets passed to `ray.remote(...)`, applied as a plain function call inside `RayRunner.run()` rather than as a `@ray.remote` decorator, since `ray` itself is only imported lazily there.
+- 2026-07-08: `RayRunner.run()` collects each batch's `ray.get()` result in a loop, one ref at a time via `asyncio.to_thread`, rather than a single `ray.get(list_of_refs)` call. Necessary for per-batch fault isolation: `ray.get()` on a list raises atomically on the first failed ref, which would abort the whole driver loop instead of letting only that batch's samples convert to `provider_error`. All batches are still dispatched (and run) concurrently via `.remote()` before this loop starts — collecting in submission order only means a fast batch behind a slow one waits to be *collected*, not to *run*. Don't "optimize" this back into a batched `ray.get()` call; it would silently reintroduce the atomic-failure problem.
+- 2026-07-08: `cache.py`'s existing two-connection WAL test (2026-07-06 entry) is same-process — two `aiosqlite` connections from one Python process. Ray workers are the first genuine multi-process contention against the shared cache file, and that hasn't been stress-tested at scale (only a handful of samples across 2-3 workers in `tests/runner/test_ray_runner.py`). If real usage hits `database is locked` despite `busy_timeout=5000`, that's the known SQLite-under-heavy-multi-process-write pain point surfacing, not a design flaw to silently patch around — escalate with the actual error (contention pattern, worker count, batch size) rather than guessing between the two obvious levers (a longer timeout, or per-worker cache files merged by the driver after the run).
+  **Follow-up, same day**: it surfaced, in the real `-m ray` suite, as `database is locked` from `tests/runner/test_ray_runner.py::test_cache_shared_across_workers_and_visible_to_a_subsequent_run`, roughly 1 run in 4-7. The story in three beats, because the first two are as instructive as the fix:
+  1. **Hypothesis tested and falsified.** The obvious first suspect was pragma order in `ResponseCache.connect()`: `PRAGMA journal_mode=WAL` ran *before* `PRAGMA busy_timeout=5000`, so that first pragma executed with the connection's default timeout of 0. Reordering (busy_timeout first) is correct hygiene and was kept, but a 20-iteration loop after the reorder still failed 6/20 — the hypothesis didn't survive contact with a loop.
+  2. **Mechanism isolated outside Ray.** Rather than keep guessing inside the Ray/pytest/asyncio stack, the race was reproduced directly: `scripts/repro_sqlite_wal_race.py` spawns plain `multiprocessing.Process` workers (no asyncio, no aiosqlite, no Ray) that all race to open the same brand-new SQLite file and switch it to WAL. `busy_timeout=5000` set first, every time — and it still reproduces `sqlite3.OperationalError: database is locked`, about 1 trial in 10-15. Conclusion: `busy_timeout` governs ordinary read/write lock contention; it does not reliably cover the *one-time* mode-switch race, where multiple connections simultaneously attempt the first-ever switch of a fresh file from the default rollback journal to WAL (an operation that briefly needs an exclusive lock to rewrite the database header, outside the normal busy-handler path). This is a genuine SQLite/WAL edge case, not an aiosqlite or Ray bug.
+  3. **Fixed at the cache layer.** `ResponseCache.connect()` now retries its own connect-and-configure sequence up to 5 times on `sqlite3.OperationalError`, with a short jittered backoff (50-200ms — the race window is milliseconds), raising `CacheError` chained from the last underlying error after exhaustion. This belongs in the cache, not in whichever orchestrator happens to connect concurrently first: if the fix lived only in the Ray driver, the next concurrent entry point (two `reproeval run` processes started at once, a future distributed backend) would silently reintroduce the exact same race. `RayRunner.run()` separately pre-warms the cache (opens and closes it once before dispatching any batches) so workers almost always connect to an already-WAL file — an optimization on top of the correctness fix, not a substitute for it: it turns the retry path into a safety net that almost never fires, rather than a hot path.
+  Tests mock the retry path directly (`sqlite3.OperationalError` twice then success; permanent failure exhausting exactly 5 attempts) rather than depending on timing, since the race itself isn't deterministically reproducible in-suite. `scripts/repro_sqlite_wal_race.py` is the standalone reproduction case, kept as living documentation of the mechanism, not as a test the suite depends on.
+- 2026-07-08: the Ray backend's `served_models` support came for free once `SampleResult.served_model` existed (see the superseded entry above) — `RayRunner` needed zero special-casing for it, which is exactly the payoff of moving that data onto the results instead of a runner-instance attribute.
+- 2026-07-08: **first real 500-sample GSM8K benchmark run**: 500/500 scored, mean 0.9720 (486/500), $0.98, 176s at concurrency 8. Hand-audited all 14 zero-score samples; the breakdown matters more than the headline number:
+  - **3 harness bugs (samples 29, 390, 431)**: the model wrote a trailing sentence-period after the number ("Answer: 25."), and `scorer.pattern` ended `\s*$` with no tolerance for a literal `.` before the anchor. Not model errors — the model was right and our regex was too strict.
+  - **1 dataset-convention mismatch (sample 188)**: model answered "106.12" (dollars-and-cents), ground truth is "106" (GSM8K's answers are integers; this question's "correct" answer is itself a rounding/convention artifact of the dataset, not a math error). Left alone deliberately — see the limitation note below.
+  - **2 truncations (samples 120, 495)**: both hit exactly `max_tokens=512` and were cut off before any "Answer:" line appeared. Indistinguishable from a wrong answer using only `SampleResult` today — see the v0.2 idea below.
+  - **8 genuine model errors** (samples 13, 17, 307, 369, 372, 404, 426, 455): actual reasoning mistakes.
+  Implication: **measured accuracy (97.2%) understates true model accuracy** — 3/500 (0.6 points) of the "failures" were pure measurement error, and the 2 truncations are arguably indeterminate rather than wrong (another 0.4 points if excluded from the denominator). True accuracy is somewhere around 97.8-98.4%, depending on how you want to treat the truncations and the one convention-mismatch sample. Fixed the 3 harness bugs in this commit (see below); did not touch the convention mismatch or the truncations, since those are dataset/measurement-design questions, not bugs.
+  - **Known limitation, not fixed**: sample 188's "106.12" vs "106" is an integer-vs-decimal convention question (does the model's more precise answer count, or does GSM8K's ground truth being an integer make the convention itself part of the spec?). Deliberately left unpatched rather than loosened with fuzzy numeric matching, which would risk scoring genuinely wrong decimal answers as correct.
+  - **v0.2 idea, not implemented**: surface `stop_reason == "max_tokens"` (Anthropic's API reports this per response) as a field on `SampleResult`, so a truncated response is distinguishable from a wrong answer in the results/manifest without hand-auditing response text. Needs a `ProviderResponse`/`SampleResult` schema change; out of scope for this fix.
+- 2026-07-08: fixed the trailing-period scoring bug in `examples/gsm8k_500.yaml`'s pattern (`\.?` added before `\s*$`), not in `scripts/build_gsm8k_500.py`'s per-sample `answer_pattern`. Deliberate layer choice: trailing-period tolerance is a universal formatting concern that applies identically to every sample regardless of its value, unlike the comma-grouping tolerance, which is genuinely per-value (where the comma goes depends on the specific number's digit count). Verified `\.?` doesn't loosen matching beyond the period itself: `\s*$` still demands nothing-but-whitespace after it, so "Answer: 25.5" against ground truth "25" still fails (the "5" after the "." blocks the anchor) — confirmed both with a standalone regex check in `build_gsm8k_500.py`'s `_verify_pattern()` (runs on every invocation) and against the real `RegexScorer`. No JSONL regeneration needed since `answer_pattern` itself is unchanged; only the spec's literal template changed.
+  Re-scoring is free: `cache_key()` is built from `(provider, model, base_url, prompt, params)` only — the scorer/pattern never enters it — so all 500 of the real run's cached responses still hit under the new spec (confirmed directly against `.reproeval_cache.sqlite`: 500/500). `compute_identity()`, unlike the cache key, hashes the whole spec including the scorer, so the identity hash *does* change with this fix (confirmed: `4fff4291...` → `abe30645...`), meaning the next real run writes to a new `results/gsm8k-500-abe30645.../` directory rather than overwriting `results/gsm8k-500-4fff4291.../`.
+- 2026-07-08: **Ray effective concurrency is `n_workers × spec.run.concurrency`**, not `spec.run.concurrency` — each worker's `_run_batch` constructs its own `asyncio.Semaphore(spec.run.concurrency)` independently, so n_workers semaphores each admit up to `concurrency` in-flight requests simultaneously. Discovered empirically comparing the two real benchmark runs: local at concurrency 8 ran 8 in-flight and took 176.6s; Ray at 8 workers × concurrency 8 ran 64 in-flight and took 32.6s (5.4x), consistent with roughly 8x the in-flight requests against one shared API rate limit. This is intentional per-executor concurrency semantics (each worker owning its own semaphore is what lets `run_one_sample` stay unaware of which executor called it), now documented as such rather than left as an implicit multiplication someone has to notice from the source. **Open question for v0.2**: should the CLI warn (or normalize `--workers`/`concurrency` together) when `workers × concurrency` is large enough to risk tripping the provider's real rate limits? Not answered here — needs a real rate-limit-error signal to design against, not a guessed threshold.
+- 2026-07-08: **v0.2 idea, not implemented**: when the first N samples (e.g. 5-10) all fail with the identical fatal error (a missing/invalid API key is the common case), fail the run fast with a one-line summary instead of continuing through all 500 `sample_completed`/`provider_attempt` log lines to reach the same conclusion. Would need a way to detect "identical fatal detail, no successes yet" inside the runner without adding per-sample overhead to the common (working) case — not designed yet, just captured so it doesn't get re-discovered from scratch next time someone runs a broken spec against 500 samples.

gradetrail-0.1.0/PKG-INFO

@@ -0,0 +1,159 @@
+Metadata-Version: 2.4
+Name: gradetrail
+Version: 0.1.0
+Summary: A distributed LLM evaluation harness: declarative, cached, reproducible.
+Project-URL: Repository, https://github.com/Cpatel2000/reproeval
+Project-URL: Issues, https://github.com/Cpatel2000/reproeval/issues
+Author: Chitvan Patel
+License: MIT
+License-File: LICENSE
+Keywords: anthropic,benchmarking,evaluation,llm,openai,ray,testing
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.11
+Requires-Dist: aiosqlite>=0.19
+Requires-Dist: anthropic>=0.40
+Requires-Dist: jinja2>=3.1
+Requires-Dist: openai>=1.50
+Requires-Dist: pydantic>=2.5
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: structlog>=24.1
+Requires-Dist: typer>=0.12
+Provides-Extra: dev
+Requires-Dist: build>=1.2; extra == 'dev'
+Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Requires-Dist: ruff~=0.15.20; extra == 'dev'
+Requires-Dist: twine>=5.0; extra == 'dev'
+Provides-Extra: ray
+Requires-Dist: ray>=2.9; extra == 'ray'
+Description-Content-Type: text/markdown
+
+# reproeval
+
+[![CI](https://github.com/Cpatel2000/reproeval/actions/workflows/ci.yml/badge.svg)](https://github.com/Cpatel2000/reproeval/actions/workflows/ci.yml)
+
+A distributed LLM evaluation harness that treats evals like tests: declarative, cached, reproducible.
+
+![demo](docs/demo.gif)
+
+> PyPI publication pending.
+
+## Quickstart
+
+```bash
+pip install reproeval  # pending; for now: pip install git+https://github.com/Cpatel2000/reproeval
+export ANTHROPIC_API_KEY=sk-ant-...
+```
+
+Define an eval:
+
+```yaml
+# gsm8k_subset.yaml
+name: gsm8k-subset
+dataset:
+  path: data/gsm8k_subset.jsonl
+prompt: |
+  Solve the following math problem. End your response with the line
+  "Answer: <number>".
+
+  {{ question }}
+model:
+  provider: anthropic
+  name: claude-sonnet-4-6
+scorer:
+  type: regex
+  pattern: 'Answer:\s*{{ answer }}\s*$'
+```
+
+`data/gsm8k_subset.jsonl` is three JSONL rows, one sample per line:
+
+```json
+{"id": "1", "question": "Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?", "answer": "72"}
+{"id": "2", "question": "Weng earns $12 an hour for babysitting. Yesterday, she just did 50 minutes of babysitting. How much did she earn?", "answer": "10"}
+{"id": "3", "question": "James writes a 3-page letter to 2 different friends twice a week. How many pages does he write a year?", "answer": "624"}
+```
+
+Run it:
+
+```bash
+reproeval run gsm8k_subset.yaml
+```
+
+Output from an actual cold run of this exact spec:
+
+```
+Samples: 3 (scored=3, provider_error=0, judge_error=0)
+Mean score: 1.0000
+Tokens: 189 in / 213 out
+Cost: $0.0038
+Cache hits: 0/3
+Wall time: 2.50s
+```
+
+Run it again and it completes in about 40ms at $0.00: every response is cached, keyed on the request, not the scorer. The run writes `results/<name>-<identity-hash>/results.jsonl` (per-sample scores and responses) and `manifest.json` (spec hash, dataset hash, git SHA, timings, cost, so you can tell later exactly what produced a given number).
+
+## Why
+
+- **Cached**: responses are keyed on (provider, model, base_url, resolved prompt, params); re-runs are free, and a prompt edit invalidates only the affected samples.
+- **Reproducible**: every run writes a manifest (spec identity hash, dataset hash, judge file hash, requested vs served model, git SHA, reproeval version).
+- **Multi-provider**: Anthropic, OpenAI, and any OpenAI-compatible endpoint (vLLM, local inference servers), through one provider abstraction with a shared retry/backoff policy.
+- **Versioned judges**: LLM-as-judge prompts are separate, hashed files, not strings inlined in the spec.
+- **Distributed**: the same spec runs unchanged on a local asyncio backend or a single-machine Ray cluster; the backend is a CLI flag, not a spec field.
+
+## Prior art
+
+Established tools cover much of this space: [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) for academic benchmarks, [Inspect](https://inspect.ai-safety-institute.org.uk/) for safety evaluations, [promptfoo](https://github.com/promptfoo/promptfoo) for application testing. reproeval differs in three specific choices: the response cache is keyed independently of the scorer, so changing how you grade re-scores for free without re-calling the API; every run's identity is a hash over spec, dataset content, and judge file, so two result sets are comparable exactly when their hashes match; and the local and Ray backends share one spec format and one per-sample pipeline, so distribution is an execution detail rather than a rewrite.
+
+## Benchmark
+
+500 samples, GSM8K test split, `claude-sonnet-4-6`, temperature 0, `max_tokens: 512`. Roughly 46.6k input / 56k output tokens per cold run.
+
+| Run | Backend | In-flight requests | Wall time | Cost | Accuracy |
+|---|---|---|---|---|---|
+| Cold | local, concurrency 8 | 8 | 176.6s | $0.98 | 97.2%* |
+| Re-score after scorer fix | local, all cached | n/a | 0.65s | $0.00 | 97.8% |
+| Cold | ray, 8 workers | 64 (8 workers x concurrency 8) | 32.6s | $0.98 | 97.6% |
+| Warm | ray, 8 workers | n/a | 4.15s | $0.00 | 97.6% |
+
+\* 97.2% was measured before a scorer bug fix (see the failure audit below); 97.8% and 97.6% are post-fix.
+
+Three things worth being explicit about, since a benchmark table invites the wrong conclusions if read too quickly:
+
+1. **The ray-vs-local wall-time difference is a concurrency comparison, not framework magic.** Ray ran 64 requests in flight (8 workers, each applying `concurrency: 8` independently) while local ran 8, both against the same API rate limit. Ray's actual value here isn't raw speed, it's that the identical spec ran on a different execution backend with a one-flag change; nothing in the spec or scoring logic had to know or care.
+2. **The $0.00 re-score row is the reason the cache exists.** After fixing a scorer regex bug (below), re-measuring all 500 samples against the corrected pattern took 0.65s and zero API calls, because the cache is keyed on (provider, model, prompt, params), not on the scorer. Changing how you score never invalidates what you already paid to generate.
+3. **Temperature 0 does not guarantee identical outputs across runs.** The local run measured 97.8% and the Ray run measured 97.6% on what should be the same 500 completions; the difference is one sample, consistent with ordinary API-level nondeterminism at temperature 0, not a bug in either backend.
+
+### Failure audit
+
+97.2% understated the model's true accuracy by roughly 0.6 points due to measurement error, not model error. All 14 zero-score samples from the cold local run were inspected by hand:
+
+- 3 were scoring artifacts: the model wrote a trailing period after the answer ("Answer: 25."), which the regex didn't tolerate. Fixed.
+- 1 was a dataset-convention mismatch: the model gave a more precise decimal answer than GSM8K's integer ground truth. Left as-is rather than loosened into fuzzy matching.
+- 2 were truncations at the 512-token output ceiling, indistinguishable from a wrong answer without inspecting the raw response.
+- 8 were genuine model reasoning errors.
+
+Only the last 8 reflect the model actually being wrong.
+
+## Roadmap
+
+- [x] Local async runner with caching and cost tracking
+- [x] Ray execution backend
+- [ ] Surface `stop_reason` in per-sample results, so truncation at the token ceiling is distinguishable from a wrong answer (motivated by the failure audit above)
+- [ ] Fail fast when the first N samples all die with an identical fatal error, e.g. a missing API key, instead of logging 500 copies of it
+- [ ] Multi-turn and tool-use evals
+- [ ] HuggingFace dataset loader (dataset-specific conversion scripts exist today; no first-class loader in the spec yet)
+
+## Design
+
+See [docs/design/eval-spec.md](docs/design/eval-spec.md) for the spec schema and semantics. A writeup of debugging an intermittent SQLite WAL race under Ray is in [docs/blog/](docs/blog/).
+
+MIT license.

gradetrail-0.1.0/README.md

@@ -0,0 +1,120 @@
+# reproeval
+
+[![CI](https://github.com/Cpatel2000/reproeval/actions/workflows/ci.yml/badge.svg)](https://github.com/Cpatel2000/reproeval/actions/workflows/ci.yml)
+
+A distributed LLM evaluation harness that treats evals like tests: declarative, cached, reproducible.
+
+![demo](docs/demo.gif)
+
+> PyPI publication pending.
+
+## Quickstart
+
+```bash
+pip install reproeval  # pending; for now: pip install git+https://github.com/Cpatel2000/reproeval
+export ANTHROPIC_API_KEY=sk-ant-...
+```
+
+Define an eval:
+
+```yaml
+# gsm8k_subset.yaml
+name: gsm8k-subset
+dataset:
+  path: data/gsm8k_subset.jsonl
+prompt: |
+  Solve the following math problem. End your response with the line
+  "Answer: <number>".
+
+  {{ question }}
+model:
+  provider: anthropic
+  name: claude-sonnet-4-6
+scorer:
+  type: regex
+  pattern: 'Answer:\s*{{ answer }}\s*$'
+```
+
+`data/gsm8k_subset.jsonl` is three JSONL rows, one sample per line:
+
+```json
+{"id": "1", "question": "Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?", "answer": "72"}
+{"id": "2", "question": "Weng earns $12 an hour for babysitting. Yesterday, she just did 50 minutes of babysitting. How much did she earn?", "answer": "10"}
+{"id": "3", "question": "James writes a 3-page letter to 2 different friends twice a week. How many pages does he write a year?", "answer": "624"}
+```
+
+Run it:
+
+```bash
+reproeval run gsm8k_subset.yaml
+```
+
+Output from an actual cold run of this exact spec:
+
+```
+Samples: 3 (scored=3, provider_error=0, judge_error=0)
+Mean score: 1.0000
+Tokens: 189 in / 213 out
+Cost: $0.0038
+Cache hits: 0/3
+Wall time: 2.50s
+```
+
+Run it again and it completes in about 40ms at $0.00: every response is cached, keyed on the request, not the scorer. The run writes `results/<name>-<identity-hash>/results.jsonl` (per-sample scores and responses) and `manifest.json` (spec hash, dataset hash, git SHA, timings, cost, so you can tell later exactly what produced a given number).
+
+## Why
+
+- **Cached**: responses are keyed on (provider, model, base_url, resolved prompt, params); re-runs are free, and a prompt edit invalidates only the affected samples.
+- **Reproducible**: every run writes a manifest (spec identity hash, dataset hash, judge file hash, requested vs served model, git SHA, reproeval version).
+- **Multi-provider**: Anthropic, OpenAI, and any OpenAI-compatible endpoint (vLLM, local inference servers), through one provider abstraction with a shared retry/backoff policy.
+- **Versioned judges**: LLM-as-judge prompts are separate, hashed files, not strings inlined in the spec.
+- **Distributed**: the same spec runs unchanged on a local asyncio backend or a single-machine Ray cluster; the backend is a CLI flag, not a spec field.
+
+## Prior art
+
+Established tools cover much of this space: [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) for academic benchmarks, [Inspect](https://inspect.ai-safety-institute.org.uk/) for safety evaluations, [promptfoo](https://github.com/promptfoo/promptfoo) for application testing. reproeval differs in three specific choices: the response cache is keyed independently of the scorer, so changing how you grade re-scores for free without re-calling the API; every run's identity is a hash over spec, dataset content, and judge file, so two result sets are comparable exactly when their hashes match; and the local and Ray backends share one spec format and one per-sample pipeline, so distribution is an execution detail rather than a rewrite.
+
+## Benchmark
+
+500 samples, GSM8K test split, `claude-sonnet-4-6`, temperature 0, `max_tokens: 512`. Roughly 46.6k input / 56k output tokens per cold run.
+
+| Run | Backend | In-flight requests | Wall time | Cost | Accuracy |
+|---|---|---|---|---|---|
+| Cold | local, concurrency 8 | 8 | 176.6s | $0.98 | 97.2%* |
+| Re-score after scorer fix | local, all cached | n/a | 0.65s | $0.00 | 97.8% |
+| Cold | ray, 8 workers | 64 (8 workers x concurrency 8) | 32.6s | $0.98 | 97.6% |
+| Warm | ray, 8 workers | n/a | 4.15s | $0.00 | 97.6% |
+
+\* 97.2% was measured before a scorer bug fix (see the failure audit below); 97.8% and 97.6% are post-fix.
+
+Three things worth being explicit about, since a benchmark table invites the wrong conclusions if read too quickly:
+
+1. **The ray-vs-local wall-time difference is a concurrency comparison, not framework magic.** Ray ran 64 requests in flight (8 workers, each applying `concurrency: 8` independently) while local ran 8, both against the same API rate limit. Ray's actual value here isn't raw speed, it's that the identical spec ran on a different execution backend with a one-flag change; nothing in the spec or scoring logic had to know or care.
+2. **The $0.00 re-score row is the reason the cache exists.** After fixing a scorer regex bug (below), re-measuring all 500 samples against the corrected pattern took 0.65s and zero API calls, because the cache is keyed on (provider, model, prompt, params), not on the scorer. Changing how you score never invalidates what you already paid to generate.
+3. **Temperature 0 does not guarantee identical outputs across runs.** The local run measured 97.8% and the Ray run measured 97.6% on what should be the same 500 completions; the difference is one sample, consistent with ordinary API-level nondeterminism at temperature 0, not a bug in either backend.
+
+### Failure audit
+
+97.2% understated the model's true accuracy by roughly 0.6 points due to measurement error, not model error. All 14 zero-score samples from the cold local run were inspected by hand:
+
+- 3 were scoring artifacts: the model wrote a trailing period after the answer ("Answer: 25."), which the regex didn't tolerate. Fixed.
+- 1 was a dataset-convention mismatch: the model gave a more precise decimal answer than GSM8K's integer ground truth. Left as-is rather than loosened into fuzzy matching.
+- 2 were truncations at the 512-token output ceiling, indistinguishable from a wrong answer without inspecting the raw response.
+- 8 were genuine model reasoning errors.
+
+Only the last 8 reflect the model actually being wrong.
+
+## Roadmap
+
+- [x] Local async runner with caching and cost tracking
+- [x] Ray execution backend
+- [ ] Surface `stop_reason` in per-sample results, so truncation at the token ceiling is distinguishable from a wrong answer (motivated by the failure audit above)
+- [ ] Fail fast when the first N samples all die with an identical fatal error, e.g. a missing API key, instead of logging 500 copies of it
+- [ ] Multi-turn and tool-use evals
+- [ ] HuggingFace dataset loader (dataset-specific conversion scripts exist today; no first-class loader in the spec yet)
+
+## Design
+
+See [docs/design/eval-spec.md](docs/design/eval-spec.md) for the spec schema and semantics. A writeup of debugging an intermittent SQLite WAL race under Ray is in [docs/blog/](docs/blog/).
+
+MIT license.

gradetrail-0.1.0/docs/blog/sqlite-wal-race-under-ray.md

@@ -0,0 +1,103 @@
+**DRAFT. Voice pass pending.**
+
+# A 1-in-5 flake: debugging a SQLite WAL race under Ray
+
+## The symptom
+
+reproeval's Ray backend test suite boots a real, non-local-mode Ray cluster (`ray.init(local_mode=False, num_cpus=2)`) and runs the runner tests against it. One of those tests checks that multiple Ray workers writing to the same SQLite-backed response cache produce results a later run can read back correctly.
+
+It failed intermittently: not every run, roughly one in four to seven. The failure was always the same shape. A sample that should have scored came back as `provider_error`, with a detail message reading:
+
+```
+worker failed: RayTaskError(OperationalError)(OperationalError('database is locked'))
+```
+
+This is precisely the failure the cache's WAL (write-ahead log) journal mode and `busy_timeout` pragma exist to prevent. The cache had already been checked for concurrent-connection safety once before, including a same-process two-connection test. This was different: a real Ray cluster with genuinely separate worker processes, contending on one shared SQLite file for the first time.
+
+## First hypothesis: pragma order (wrong)
+
+`ResponseCache.connect()` set `journal_mode=WAL` before `busy_timeout`:
+
+```python
+async with conn.execute("PRAGMA journal_mode=WAL") as cursor:
+    (mode,) = await cursor.fetchone()
+...
+await conn.execute("PRAGMA busy_timeout=5000")
+```
+
+`busy_timeout` defaults to 0 on a fresh connection. If the WAL pragma runs before `busy_timeout` is set, that one statement executes with a zero-length timeout. The obvious fix was to reorder:
+
+```python
+await conn.execute("PRAGMA busy_timeout=5000")
+async with conn.execute("PRAGMA journal_mode=WAL") as cursor:
+    (mode,) = await cursor.fetchone()
+```
+
+This is correct hygiene and worth keeping regardless of whether it fixes anything. It did not fix anything. A 20-iteration loop of the real Ray-marked suite after the reorder still failed 6 times out of 20, statistically indistinguishable from before the change. The hypothesis did not survive contact with a loop.
+
+## Isolating the mechanism outside Ray
+
+Rather than keep debugging inside the combined Ray, pytest, asyncio, and aiosqlite stack, the next step was to strip all of that away and reproduce the underlying SQLite behavior directly, with nothing but the standard library:
+
+```python
+def _worker(db_path, worker_id, n_writes, results):
+    try:
+        conn = sqlite3.connect(db_path)
+        conn.execute("PRAGMA busy_timeout=5000")
+        (mode,) = conn.execute("PRAGMA journal_mode=WAL").fetchone()
+        ...
+    except sqlite3.OperationalError as exc:
+        results[worker_id] = f"OperationalError: {exc}"
+```
+
+Spawn three of these as separate `multiprocessing.Process` workers, all pointed at the same brand-new, empty SQLite file, all racing to run `PRAGMA journal_mode=WAL` at effectively the same instant. `busy_timeout=5000` is set on every connection before the WAL pragma runs: the already-"fixed" order.
+
+Run it 15 times. On one of them:
+
+```
+sqlite3.OperationalError: database is locked
+```
+
+on the `journal_mode=WAL` line. No Ray, no asyncio, no aiosqlite anywhere in the process. The mechanism was real, and it had nothing to do with Ray or with the pragma order this project controls.
+
+## The actual mechanism
+
+A fresh SQLite file starts in the default rollback-journal mode. Switching a connection to WAL mode is not an ordinary read or write. It briefly requires an exclusive lock to rewrite the database header, because the on-disk format itself changes. `busy_timeout` governs SQLite's normal busy-handler retry loop, the one that applies to ordinary lock contention once a database is already settled into a journal mode. It does not reliably cover this one-time, first-ever mode switch, which sits outside that path.
+
+In practice, when several connections open a brand-new file at nearly the same moment and all attempt the initial WAL switch, more than one can hit `SQLITE_BUSY` in a way `busy_timeout` does not smooth over. Once a file is already in WAL mode, later connections requesting WAL mode again are cheap no-ops and this race no longer applies. That is exactly why the failure only ever showed up against a fresh cache file, and why reproducing it needed genuinely separate processes, not threads or coroutines, racing the very first connection.
+
+## The fix, in two layers
+
+The correctness fix belongs in `ResponseCache.connect()` itself, not in whichever caller happens to connect first. If the fix lived only in the Ray runner, the next concurrent entry point (two `reproeval run` invocations started at once, a future distributed backend) would silently reintroduce the same race.
+
+```python
+async def connect(self) -> None:
+    last_exc = None
+    for attempt in range(1, _CONNECT_MAX_ATTEMPTS + 1):
+        try:
+            self._conn = await self._try_connect()
+            return
+        except sqlite3.OperationalError as exc:
+            last_exc = exc
+            if attempt == _CONNECT_MAX_ATTEMPTS:
+                break
+            await asyncio.sleep(random.uniform(_CONNECT_RETRY_MIN_S, _CONNECT_RETRY_MAX_S))
+    raise CacheError(
+        f"could not connect to {self._path} after {_CONNECT_MAX_ATTEMPTS} attempts "
+        f"(likely concurrent connections racing the initial WAL switch): {last_exc}"
+    ) from last_exc
+```
+
+Five attempts, a short jittered backoff (50-200ms, since the race window itself is on the order of milliseconds), and a `CacheError` chained from the last underlying exception if every attempt fails. A genuinely non-transient failure to enable WAL, an unsupported filesystem, for example, is not retried: only `sqlite3.OperationalError` triggers a retry, everything else raises immediately.
+
+On top of that, `RayRunner.run()` pre-warms the cache. It opens and closes a connection to the cache path once in the driver, before dispatching any worker, so the file is already in WAL mode by the time workers connect. This is an optimization on top of the fix, not a substitute for it: it turns the retry path into a safety net that almost never fires under real contention, rather than a hot path every worker has to negotiate at startup.
+
+## Verification
+
+The bounded retry is tested by mocking `aiosqlite.connect` to raise `sqlite3.OperationalError` a controlled number of times before succeeding, or always, rather than depending on timing, since the underlying race is not deterministically reproducible in a test suite. One test confirms recovery after two simulated failures, with a real WAL check and a real put/get round-trip afterward proving the pragmas actually ran on the successful attempt. Another confirms exactly `_CONNECT_MAX_ATTEMPTS` calls before a chained `CacheError`.
+
+The empirical confirmation was a loop, not a single run: `pytest -m ray` against the real cluster, 25 consecutive times, zero failures, down from roughly one failure in four to seven before the fix.
+
+## Takeaway
+
+The first, most plausible hypothesis, pragma order, was wrong, and a single passing test run would not have revealed that; only a loop did. The actual mechanism was a real SQLite and WAL edge case, isolated by deliberately removing every layer that was not necessary to reproduce it, Ray, asyncio, aiosqlite, down to a 30-line multiprocessing script. The fix belongs at the layer that owns the invariant, the cache, not the layer that happened to trigger it first, the Ray runner.