spancheck 0.1.0__tar.gz

spancheck-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Trevor J. Romack
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

spancheck-0.1.0/PKG-INFO

@@ -0,0 +1,180 @@
+Metadata-Version: 2.4
+Name: spancheck
+Version: 0.1.0
+Summary: Score a grounded-answer system on citation accuracy, abstention correctness, hallucination rate, cost and latency — with an audit log a compliance reviewer can read.
+Author-email: "Trevor J. Romack" <tjromack@gmail.com>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/tjromack/spancheck
+Project-URL: Source, https://github.com/tjromack/spancheck
+Keywords: rag,evaluation,llm,hallucination,citation,abstention,guardrails
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Operating System :: OS Independent
+Classifier: Intended Audience :: Developers
+Classifier: Topic :: Software Development :: Quality Assurance
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: dev
+Requires-Dist: pytest>=8; extra == "dev"
+Dynamic: license-file
+
+# spancheck
+
+> © 2026 Trevor J. Romack — **MIT-licensed** ([LICENSE](LICENSE)) · `pip install spancheck` · tjromack@gmail.com
+
+**Score a grounded-answer system on citation accuracy, abstention correctness, hallucination rate, cost and latency —
+and get an audit log a compliance reviewer can read.**
+
+`spancheck` is an installable, corpus-agnostic evaluation library for RAG and retrieval-then-answer pipelines. Point it
+at any system through a thin adapter, give it a test set, and it produces a scorecard on the four metrics that decide
+whether the system's answers can be trusted — plus a versioned audit log that reconstructs *why* each answer passed or
+failed.
+
+Library first, CLI second, GitHub Action third: the Python API is the product; the command line and the Action are thin
+wrappers over it.
+
+---
+
+## The four metrics
+
+| Metric | Question it answers | How it's scored |
+|---|---|---|
+| **Citation accuracy** | Does the cited span actually support the claim? | Deterministic span check *(the capability built first and tested deepest)* |
+| **Abstention correctness** | Does it decline when the answer isn't in the corpus, and answer when it is? | Deterministic |
+| **Hallucination rate** | What share of answers assert something the retrieved context doesn't support? | Deterministic groundedness + calibrated LLM-judge for the qualitative residue |
+| **Cost & latency** | Tokens / dollars / wall-clock per answer, against a budget | Deterministic, from a cached run — no network |
+
+## Status
+
+**Complete and dogfooded.** The measurement core (`Case` / `evaluate` / `Run` / `Scorecard` / `diff` / `gate`), the thin
+`adapter` contract, the deterministic graders, **citation-span verification** — the capability this build exists to
+add — the **cost/latency + versioned audit log** layer, the **calibrated LLM-judge**, and the **CLI + GitHub Action**
+are implemented, dependency-free (the judge reaches a model only through a caller-supplied callable), and tested
+(68 tests, green on a clean `pip install` and in CI). Citation verification splits into **provenance** (is the cited
+span really in the retrieved context, verbatim?) and **support** (does the span cover the claim?); both must hold, so a
+fabricated quote or a right-answer-wrong-citation cannot pass. A captured run re-scores **offline** (`Run.rescore`), and
+`run.audit_log()` emits a versioned, self-describing record with per-citation verdicts (schema in
+[`docs/AUDIT-LOG.md`](docs/AUDIT-LOG.md)). The LLM-judge is **opt-in** — the default grader set stays deterministic and
+offline — and is **calibrated before it is trusted**.
+
+## Results (measured)
+
+`spancheck` was run end-to-end against a real product pipeline as a black box (a grounded answer-a-document tool on
+`claude-sonnet-5`, over its own sample contract), and the judge was calibrated against a human-labelled gold set. Full
+write-up in [`docs/CASE-STUDY.md`](docs/CASE-STUDY.md); the audit log is committed at
+[`dogfood/suver_audit.json`](dogfood/suver_audit.json).
+
+| Metric | Result |
+|---|---|
+| Citation accuracy (real pipeline) | **1.00** — every cited span real and supporting |
+| Hallucination (groundedness) | **0 hallucinations** (pass-rate 1.00) |
+| Abstention correctness | 11/12 — spancheck **caught one false-abstention** on the real system |
+| Overall (12 cases) | **0.967** |
+| Judge vs human gold set — stub / `claude-sonnet-5` | **0.75 / 1.00** |
+| Tests | **68**, clean `pip install` + CI (3.11 & 3.12) |
+
+The one miss is the point: the eval found a real recall gap in a shipping system — a measured behaviour, not a spancheck
+bug. Reproduce with `python dogfood/run_dogfood.py` and `python -m spancheck.calibrate --real`.
+
+## Command line & CI
+
+The CLI is a thin wrapper over the library. It works out of the box against a shipped demo target — no key, no user
+code:
+
+```bash
+pip install spancheck          # or `pip install -e .` from a clone
+spancheck run examples/cases.jsonl --target spancheck.demo:system --out audit.json
+spancheck score audit.json                                    # recompute metrics from the cache, offline
+spancheck gate  audit.json citation_accuracy=1.0 groundedness=0.9   # exit 1 on failure (for CI)
+```
+
+Point `--target` at your own system with `module:callable` (any `input -> answer` function; the answer may be a string
+or a dict with `contexts`/`citations`/`usage`).
+
+The deterministic path needs no key. The **opt-in LLM-judge** (`--real` calibration, or `judge_support(provider=...)`)
+reads `ANTHROPIC_API_KEY` — set it in the environment, or drop it in a gitignored `.env` at the repo root
+(`ANTHROPIC_API_KEY=sk-ant-...`), which the CLI and `spancheck.calibrate` load automatically.
+
+As a **GitHub Action** (fails a PR on a regression):
+
+```yaml
+- uses: tjromack/spancheck@main
+  with:
+    cases: eval/cases.jsonl
+    target: myapp.rag:answer
+    thresholds: "citation_accuracy=1.0 groundedness=0.9"
+    baseline: baseline-audit.json   # optional: also fail on a pass-rate drop
+```
+
+## Lineage
+
+`spancheck` is the spinoff of [`llm-eval-guardrails-harness`](https://github.com/tjromack/llm-eval-guardrails-harness) —
+the version that proved the method against a single real target (a regulatory RAG copilot), with LLM-judge agreement
+1.00 against a human gold set. This package is the same method, extracted from an in-repo lab, generalised to any
+corpus, and packaged so a stranger can install it and point it at their own system. What carried over unchanged, what
+was rewritten and why, and what the harness got wrong that this fixes are recorded in [`LINEAGE.md`](LINEAGE.md).
+
+## Quickstart
+
+> The Phase-1 core below runs today (`pip install -e .`). Lines marked *(Phase 3)* / *(Phase 2)* are the intended shape
+> of what those phases add; see `Status` and `TODO.md`.
+
+```python
+from spancheck import Case, evaluate, adapter, citation_accuracy
+
+# 1. Wrap your system in a thin adapter: input -> {answer, contexts, citations, usage, latency_ms}
+#    (optional — evaluate() also accepts a plain input->answer callable and times it for you)
+my_system = adapter(lambda q: my_rag_pipeline(q))
+
+# 2. Describe your test set — answerable, unanswerable, and adversarial cases
+cases = [
+    Case(id="q1", input="What is the notice period?", category="answerable",
+         expected="30 days", meta={"answerable": True}),
+    Case(id="q2", input="What is the company's revenue?", category="unanswerable",
+         meta={"answerable": False}),  # should abstain
+]
+
+# 3. Score it — a run is captured once, then the metrics compute offline
+run = evaluate(cases, my_system)
+print(run.scorecard().by_grader())   # citation accuracy, abstention correctness, groundedness, PII-leak
+run.save("run.json")                 # persist a run for baselines / regression gating
+run.audit_log("audit.json")          # the versioned, compliance-readable record (per-citation verdicts + cost/latency)
+run.rescore([citation_accuracy(support_threshold=0.8)])  # re-score the SAME run offline — no system call
+```
+
+```bash
+# CLI (thin wrapper over the same API) — command surface exists; wired to the library in Phase 5
+spancheck run cases.jsonl --target my_module:my_system --out audit.json
+spancheck score audit.json        # recompute metrics from a cached run, no network
+```
+
+## Limits — what a passing score does *not* claim
+
+- **Deterministic support is a lexical proxy, not entailment.** By default, citation *support* is scored by how much of
+  the claim's wording the cited span covers — fast, offline, and unable to see a span that shares the claim's words but
+  **contradicts** it ("the notice period is *not* 30 days"). The **opt-in LLM-judge** upgrades this to true entailment
+  (`citation_accuracy(support_fn=judge_support(provider=...))`), and is **calibrated against a human gold set before it
+  is trusted** (`python -m spancheck.calibrate`). The deterministic core flags the proxy's boundary rather than hiding
+  it (a test pins the false-positive; the judge catches it).
+- **Provenance requires a verbatim quote.** A paraphrased citation fails provenance by design — a deliberate incentive
+  for a system to quote its sources exactly. `spancheck` does not (yet) match a citation by meaning.
+- **A citation isn't scoped to its named source.** A cited span found in *any* retrieved context passes; tying a
+  citation to the specific `source_id` it names is a planned refinement.
+- **Groundedness is a word-overlap proxy too**, and the metrics are only as good as the test set you bring. `spancheck`
+  measures a system against cases you author; it does not generate them.
+
+## Design pins
+
+- **Library-first, CLI-second** — the API is the product.
+- **No provider lock-in** — a thin adapter from day one; stub by default, any model behind a callable.
+- **Every metric computable without network given a cached run** — capture once, re-score offline.
+- **The audit-log schema is versioned** — a breaking change is a major version bump; it's a contract with a reviewer.
+- **Judge prompts live in version-controlled files** — a rubric change is a reviewable, version-stamped diff.
+- **No dependency on any other project's internals** — standalone; dogfooded as a black box, never coupled.
+
+## License
+
+**MIT** — see [LICENSE](LICENSE). Free to use, modify, and distribute; the library is a tool meant to be run against
+your own systems.

spancheck-0.1.0/README.md

@@ -0,0 +1,158 @@
+# spancheck
+
+> © 2026 Trevor J. Romack — **MIT-licensed** ([LICENSE](LICENSE)) · `pip install spancheck` · tjromack@gmail.com
+
+**Score a grounded-answer system on citation accuracy, abstention correctness, hallucination rate, cost and latency —
+and get an audit log a compliance reviewer can read.**
+
+`spancheck` is an installable, corpus-agnostic evaluation library for RAG and retrieval-then-answer pipelines. Point it
+at any system through a thin adapter, give it a test set, and it produces a scorecard on the four metrics that decide
+whether the system's answers can be trusted — plus a versioned audit log that reconstructs *why* each answer passed or
+failed.
+
+Library first, CLI second, GitHub Action third: the Python API is the product; the command line and the Action are thin
+wrappers over it.
+
+---
+
+## The four metrics
+
+| Metric | Question it answers | How it's scored |
+|---|---|---|
+| **Citation accuracy** | Does the cited span actually support the claim? | Deterministic span check *(the capability built first and tested deepest)* |
+| **Abstention correctness** | Does it decline when the answer isn't in the corpus, and answer when it is? | Deterministic |
+| **Hallucination rate** | What share of answers assert something the retrieved context doesn't support? | Deterministic groundedness + calibrated LLM-judge for the qualitative residue |
+| **Cost & latency** | Tokens / dollars / wall-clock per answer, against a budget | Deterministic, from a cached run — no network |
+
+## Status
+
+**Complete and dogfooded.** The measurement core (`Case` / `evaluate` / `Run` / `Scorecard` / `diff` / `gate`), the thin
+`adapter` contract, the deterministic graders, **citation-span verification** — the capability this build exists to
+add — the **cost/latency + versioned audit log** layer, the **calibrated LLM-judge**, and the **CLI + GitHub Action**
+are implemented, dependency-free (the judge reaches a model only through a caller-supplied callable), and tested
+(68 tests, green on a clean `pip install` and in CI). Citation verification splits into **provenance** (is the cited
+span really in the retrieved context, verbatim?) and **support** (does the span cover the claim?); both must hold, so a
+fabricated quote or a right-answer-wrong-citation cannot pass. A captured run re-scores **offline** (`Run.rescore`), and
+`run.audit_log()` emits a versioned, self-describing record with per-citation verdicts (schema in
+[`docs/AUDIT-LOG.md`](docs/AUDIT-LOG.md)). The LLM-judge is **opt-in** — the default grader set stays deterministic and
+offline — and is **calibrated before it is trusted**.
+
+## Results (measured)
+
+`spancheck` was run end-to-end against a real product pipeline as a black box (a grounded answer-a-document tool on
+`claude-sonnet-5`, over its own sample contract), and the judge was calibrated against a human-labelled gold set. Full
+write-up in [`docs/CASE-STUDY.md`](docs/CASE-STUDY.md); the audit log is committed at
+[`dogfood/suver_audit.json`](dogfood/suver_audit.json).
+
+| Metric | Result |
+|---|---|
+| Citation accuracy (real pipeline) | **1.00** — every cited span real and supporting |
+| Hallucination (groundedness) | **0 hallucinations** (pass-rate 1.00) |
+| Abstention correctness | 11/12 — spancheck **caught one false-abstention** on the real system |
+| Overall (12 cases) | **0.967** |
+| Judge vs human gold set — stub / `claude-sonnet-5` | **0.75 / 1.00** |
+| Tests | **68**, clean `pip install` + CI (3.11 & 3.12) |
+
+The one miss is the point: the eval found a real recall gap in a shipping system — a measured behaviour, not a spancheck
+bug. Reproduce with `python dogfood/run_dogfood.py` and `python -m spancheck.calibrate --real`.
+
+## Command line & CI
+
+The CLI is a thin wrapper over the library. It works out of the box against a shipped demo target — no key, no user
+code:
+
+```bash
+pip install spancheck          # or `pip install -e .` from a clone
+spancheck run examples/cases.jsonl --target spancheck.demo:system --out audit.json
+spancheck score audit.json                                    # recompute metrics from the cache, offline
+spancheck gate  audit.json citation_accuracy=1.0 groundedness=0.9   # exit 1 on failure (for CI)
+```
+
+Point `--target` at your own system with `module:callable` (any `input -> answer` function; the answer may be a string
+or a dict with `contexts`/`citations`/`usage`).
+
+The deterministic path needs no key. The **opt-in LLM-judge** (`--real` calibration, or `judge_support(provider=...)`)
+reads `ANTHROPIC_API_KEY` — set it in the environment, or drop it in a gitignored `.env` at the repo root
+(`ANTHROPIC_API_KEY=sk-ant-...`), which the CLI and `spancheck.calibrate` load automatically.
+
+As a **GitHub Action** (fails a PR on a regression):
+
+```yaml
+- uses: tjromack/spancheck@main
+  with:
+    cases: eval/cases.jsonl
+    target: myapp.rag:answer
+    thresholds: "citation_accuracy=1.0 groundedness=0.9"
+    baseline: baseline-audit.json   # optional: also fail on a pass-rate drop
+```
+
+## Lineage
+
+`spancheck` is the spinoff of [`llm-eval-guardrails-harness`](https://github.com/tjromack/llm-eval-guardrails-harness) —
+the version that proved the method against a single real target (a regulatory RAG copilot), with LLM-judge agreement
+1.00 against a human gold set. This package is the same method, extracted from an in-repo lab, generalised to any
+corpus, and packaged so a stranger can install it and point it at their own system. What carried over unchanged, what
+was rewritten and why, and what the harness got wrong that this fixes are recorded in [`LINEAGE.md`](LINEAGE.md).
+
+## Quickstart
+
+> The Phase-1 core below runs today (`pip install -e .`). Lines marked *(Phase 3)* / *(Phase 2)* are the intended shape
+> of what those phases add; see `Status` and `TODO.md`.
+
+```python
+from spancheck import Case, evaluate, adapter, citation_accuracy
+
+# 1. Wrap your system in a thin adapter: input -> {answer, contexts, citations, usage, latency_ms}
+#    (optional — evaluate() also accepts a plain input->answer callable and times it for you)
+my_system = adapter(lambda q: my_rag_pipeline(q))
+
+# 2. Describe your test set — answerable, unanswerable, and adversarial cases
+cases = [
+    Case(id="q1", input="What is the notice period?", category="answerable",
+         expected="30 days", meta={"answerable": True}),
+    Case(id="q2", input="What is the company's revenue?", category="unanswerable",
+         meta={"answerable": False}),  # should abstain
+]
+
+# 3. Score it — a run is captured once, then the metrics compute offline
+run = evaluate(cases, my_system)
+print(run.scorecard().by_grader())   # citation accuracy, abstention correctness, groundedness, PII-leak
+run.save("run.json")                 # persist a run for baselines / regression gating
+run.audit_log("audit.json")          # the versioned, compliance-readable record (per-citation verdicts + cost/latency)
+run.rescore([citation_accuracy(support_threshold=0.8)])  # re-score the SAME run offline — no system call
+```
+
+```bash
+# CLI (thin wrapper over the same API) — command surface exists; wired to the library in Phase 5
+spancheck run cases.jsonl --target my_module:my_system --out audit.json
+spancheck score audit.json        # recompute metrics from a cached run, no network
+```
+
+## Limits — what a passing score does *not* claim
+
+- **Deterministic support is a lexical proxy, not entailment.** By default, citation *support* is scored by how much of
+  the claim's wording the cited span covers — fast, offline, and unable to see a span that shares the claim's words but
+  **contradicts** it ("the notice period is *not* 30 days"). The **opt-in LLM-judge** upgrades this to true entailment
+  (`citation_accuracy(support_fn=judge_support(provider=...))`), and is **calibrated against a human gold set before it
+  is trusted** (`python -m spancheck.calibrate`). The deterministic core flags the proxy's boundary rather than hiding
+  it (a test pins the false-positive; the judge catches it).
+- **Provenance requires a verbatim quote.** A paraphrased citation fails provenance by design — a deliberate incentive
+  for a system to quote its sources exactly. `spancheck` does not (yet) match a citation by meaning.
+- **A citation isn't scoped to its named source.** A cited span found in *any* retrieved context passes; tying a
+  citation to the specific `source_id` it names is a planned refinement.
+- **Groundedness is a word-overlap proxy too**, and the metrics are only as good as the test set you bring. `spancheck`
+  measures a system against cases you author; it does not generate them.
+
+## Design pins
+
+- **Library-first, CLI-second** — the API is the product.
+- **No provider lock-in** — a thin adapter from day one; stub by default, any model behind a callable.
+- **Every metric computable without network given a cached run** — capture once, re-score offline.
+- **The audit-log schema is versioned** — a breaking change is a major version bump; it's a contract with a reviewer.
+- **Judge prompts live in version-controlled files** — a rubric change is a reviewable, version-stamped diff.
+- **No dependency on any other project's internals** — standalone; dogfooded as a black box, never coupled.
+
+## License
+
+**MIT** — see [LICENSE](LICENSE). Free to use, modify, and distribute; the library is a tool meant to be run against
+your own systems.

spancheck-0.1.0/pyproject.toml

@@ -0,0 +1,43 @@
+[build-system]
+requires = ["setuptools>=77"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "spancheck"
+version = "0.1.0"
+description = "Score a grounded-answer system on citation accuracy, abstention correctness, hallucination rate, cost and latency — with an audit log a compliance reviewer can read."
+readme = "README.md"
+requires-python = ">=3.11"
+license = "MIT"
+license-files = ["LICENSE"]
+authors = [{ name = "Trevor J. Romack", email = "tjromack@gmail.com" }]
+keywords = ["rag", "evaluation", "llm", "hallucination", "citation", "abstention", "guardrails"]
+classifiers = [
+  "Programming Language :: Python :: 3",
+  "Programming Language :: Python :: 3.11",
+  "Programming Language :: Python :: 3.12",
+  "Operating System :: OS Independent",
+  "Intended Audience :: Developers",
+  "Topic :: Software Development :: Quality Assurance",
+]
+# Standard-library first (design pin: a hard dependency is added only with a DECISIONS entry). None yet.
+dependencies = []
+
+[project.optional-dependencies]
+dev = ["pytest>=8"]
+
+[project.urls]
+Homepage = "https://github.com/tjromack/spancheck"
+Source = "https://github.com/tjromack/spancheck"
+
+[project.scripts]
+spancheck = "spancheck.cli:main"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.setuptools.package-data]
+spancheck = ["prompts/*.txt"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]

spancheck-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

spancheck-0.1.0/src/spancheck/__init__.py

@@ -0,0 +1,47 @@
+"""spancheck — score a grounded-answer system on citation accuracy, abstention correctness,
+hallucination rate, cost and latency, and emit an audit log a compliance reviewer can read.
+
+Public API (Phase 1 — the ported measurement core + the thin adapter contract):
+
+    from spancheck import Case, evaluate, adapter, Run, Scorecard, gate, diff
+
+Deterministic graders live in `spancheck.graders`. Citation-span verification (the capability this build exists to
+add) lands in Phase 2; the versioned audit log in Phase 3; the calibrated LLM-judge in Phase 4. See TODO.md.
+
+Nothing here hard-codes a model provider or depends on any other project's internals (CLAUDE.md design pins).
+"""
+from __future__ import annotations
+
+from .adapter import Output, adapter, normalize
+from .core import (
+    Case,
+    GradeResult,
+    CaseResult,
+    Run,
+    Scorecard,
+    evaluate,
+    run_eval,
+    diff,
+    gate,
+    default_graders,
+)
+from .span import citation_accuracy, verify_citation, CitationVerdict
+from .audit import build_audit_log, cost_latency, SCHEMA_VERSION
+from .judge import judge_entailment, judge_support, llm_judge, load_prompt
+from .calibrate import calibrate_judge
+from . import graders
+
+__version__ = "0.0.1"
+
+__all__ = [
+    # the declared public surface
+    "Case", "evaluate", "adapter", "Run", "Scorecard", "gate", "diff",
+    # citation-span verification (the capability spancheck is named for)
+    "citation_accuracy", "verify_citation", "CitationVerdict",
+    # cost/latency + the versioned audit log
+    "build_audit_log", "cost_latency", "SCHEMA_VERSION",
+    # the calibrated LLM-judge (opt-in; stub by default, no provider lock-in)
+    "judge_entailment", "judge_support", "llm_judge", "load_prompt", "calibrate_judge",
+    # supporting types + the grader library
+    "Output", "normalize", "GradeResult", "CaseResult", "run_eval", "default_graders", "graders",
+]

spancheck-0.1.0/src/spancheck/_env.py

@@ -0,0 +1,25 @@
+"""Tiny, dependency-free .env loader.
+
+Reads KEY=VALUE lines from a .env file into os.environ **without overwriting** anything already set (a real
+environment variable always wins). Called at CLI/calibration startup so a key placed in `.env` "just works" — no
+python-dotenv dependency, no vendor lock-in. `.env` is gitignored; a key never enters the repo.
+"""
+from __future__ import annotations
+
+import os
+
+
+def load_dotenv(path: str = ".env") -> None:
+    try:
+        with open(path, encoding="utf-8") as f:
+            for line in f:
+                line = line.strip()
+                if not line or line.startswith("#") or "=" not in line:
+                    continue
+                key, _, value = line.partition("=")
+                key = key.strip()
+                value = value.strip().strip('"').strip("'")
+                if key and key not in os.environ:
+                    os.environ[key] = value
+    except FileNotFoundError:
+        pass

spancheck-0.1.0/src/spancheck/_text.py

@@ -0,0 +1,29 @@
+"""Shared text helpers used by graders and the citation-span verifier.
+
+Kept in one place so the deterministic checks agree on what a "word" is and how text is normalised.
+Stdlib only.
+"""
+from __future__ import annotations
+
+import re
+
+WORD = re.compile(r"[a-z0-9]+")
+
+STOP = set("the a an and or of to in on for is are was were be been it its this that with as at by from "
+           "your you our we i they he she them his her their not no do does did have has had will would can "
+           "could should may might must shall into over under about than then so if but".split())
+
+
+def norm(s) -> str:
+    """Lowercase and collapse all whitespace to single spaces — the normalisation used for span provenance."""
+    return re.sub(r"\s+", " ", (s or "")).strip().lower()
+
+
+def words(s) -> set:
+    """All word tokens (lowercased) — used as the haystack when checking coverage."""
+    return set(WORD.findall((s or "").lower()))
+
+
+def content_words(s):
+    """Meaningful tokens: lowercased words that aren't stopwords and are longer than two characters."""
+    return [w for w in WORD.findall((s or "").lower()) if w not in STOP and len(w) > 2]

spancheck-0.1.0/src/spancheck/adapter.py

@@ -0,0 +1,90 @@
+"""The thin adapter contract — the seam between spancheck and any system under test.
+
+Design pin #2 (no provider lock-in) and #3 (every metric computable offline from a captured run) both live here:
+a system's answer is normalised **once** into a typed `Output` that carries everything the metrics need
+(the answer, the retrieved contexts, the citations, token/cost usage, and latency), so scoring never has to
+call the system again.
+
+`spancheck` never hard-codes a vendor. A caller supplies a plain `input -> answer` callable; `adapter()` wraps it,
+times it, and normalises whatever it returns. The return may be:
+
+  - a **str** — just the answer text;
+  - a **dict** — with any of `answer`/`text`, `contexts`/`context`, `citations`, `usage`, `latency_ms`, `abstained`;
+  - an **Output** — already normalised (idempotent).
+
+Rewritten from evallab's loose "str-or-dict" handling into a pinned dataclass so cost/latency and citations are
+first-class and always present (see LINEAGE.md, DECISIONS.md AB-DEC 006).
+"""
+from __future__ import annotations
+
+import time
+from dataclasses import dataclass, field, asdict
+
+
+@dataclass
+class Output:
+    """The normalised result of one system call — the single shape every grader reads.
+
+    `citations` is carried through as-is in Phase 1; its shape is firmed up by the citation-span verifier
+    (Phase 2). `abstained` is an explicit signal when the system provides one; when it is None the graders
+    infer abstention from the answer text.
+    """
+    answer: str = ""
+    contexts: list = field(default_factory=list)   # retrieved context passages the answer should be grounded in
+    citations: list = field(default_factory=list)  # citations backing the answer (span shape fixed in Phase 2)
+    usage: dict = field(default_factory=dict)       # e.g. {"input_tokens", "output_tokens", "cost_usd"}
+    latency_ms: float | None = None                 # wall-clock for the call; filled by adapter/evaluate if absent
+    abstained: bool | None = None                   # explicit abstention signal, if the system emits one
+    raw: object = None                              # the untouched original return, for the audit log
+
+    def to_dict(self) -> dict:
+        d = asdict(self)
+        # `raw` may not be JSON-serialisable; keep a string fallback for the audit log rather than crash.
+        try:
+            import json
+            json.dumps(d["raw"])
+        except (TypeError, ValueError):
+            d["raw"] = repr(self.raw)
+        return d
+
+
+def normalize(value) -> Output:
+    """Coerce a system's return (str | dict | Output) into an Output. Idempotent on Output."""
+    if isinstance(value, Output):
+        return value
+    if isinstance(value, dict):
+        answer = value.get("answer", value.get("text", ""))
+        ctx = value.get("contexts", value.get("context", []))
+        if not isinstance(ctx, list):
+            ctx = [ctx] if ctx else []
+        citations = value.get("citations", [])
+        if not isinstance(citations, list):
+            citations = [citations]
+        return Output(
+            answer="" if answer is None else str(answer),
+            contexts=[str(c) for c in ctx],
+            citations=citations,
+            usage=dict(value.get("usage", {}) or {}),
+            latency_ms=value.get("latency_ms"),
+            abstained=value.get("abstained"),
+            raw=value,
+        )
+    # a bare string (or anything else) is just the answer text
+    return Output(answer="" if value is None else str(value), raw=value)
+
+
+def adapter(fn, *, capture_time: bool = True):
+    """Wrap a caller's `input -> result` callable so it returns a normalised, timed `Output`.
+
+    Using `adapter()` is optional — `evaluate()` normalises and times its system either way — but it lets a caller
+    attach usage/citations at the source and hand `evaluate` a system that already speaks the contract.
+    """
+    def run(inp) -> Output:
+        start = time.perf_counter()
+        result = fn(inp)
+        elapsed_ms = (time.perf_counter() - start) * 1000.0
+        out = normalize(result)
+        if capture_time and out.latency_ms is None:
+            out.latency_ms = round(elapsed_ms, 3)
+        return out
+    return run