jevassert 0.1.0__tar.gz → 0.2.0__tar.gz

{jevassert-0.1.0 → jevassert-0.2.0}/AGENTS.md

@@ -13,11 +13,16 @@ Public repo: https://github.com/dtduc-git/jevassert · Apache-2.0.
 - `src/jevassert/client.py` — one HTTP client for `/v1/systemone`, retries on
   429/5xx, injectable transport. The API key comes from `TYPESAFE_API_KEY`
   only; never accept literal keys on the CLI, never log them.
+- `src/jevassert/backends.py` — `AdapterClient`: same interface, but records a
+  general-purpose LLM through the official `system-one-adapter`
+  (`--backend openai|anthropic`, optional extra `jevassert[adapter]`). Used by
+  the jev-packs benchmark; maps adapter errors onto `JevError` (auth fail-fast).
 - `src/jevassert/runner.py` — `record` (concurrent, per-case, `model=` override)
   and predictions JSONL read/write. One line per case, stable case order.
 - `src/jevassert/metrics.py` — items, accuracy, ECE (equal-mass bins), Brier
-  (Noul), coverage table, author-threshold coverage, cost/latency, gates.
-  Definitions are in the module docstring — keep them.
+  (Noul), coverage table, author-threshold coverage, cost/latency (prices
+  injectable for non-Jev backends), gates. Definitions are in the module
+  docstring — keep them.
 - `src/jevassert/compare.py` — paired comparison + exact McNemar (no scipy).
 - `src/jevassert/report.py` — markdown report (spec `evidence.md`) + JUnit XML.
 - `src/jevassert/cli.py` — `record | check | compare`, exit 0/1/2. `check`
@@ -67,4 +72,10 @@ Reconcile with SPEC was done 2026-09-19. Status:
 8. ~~Record evidence for the jev-packs registry packs~~ done 2026-09-19:
    9 packs verified (2,990 cases / 6,430 items), evidence regenerated from the
    committed recordings, jev-packs validator green.
-9. Publish: GitHub repo + push, PyPI trusted publisher, Action listing.
+9. ~~Publish: GitHub repo + push~~ done 2026-09-19; ~~PyPI trusted publisher +
+   `v0.1.0` release~~ done (PyPI `jevassert` 0.1.0, `uvx jevassert` verified).
+   Action Marketplace listing still open.
+10. **0.2.0 (unreleased)**: adapter backends (`--backend openai|anthropic`),
+    `check --input-price/--output-price`, reports the recorded models instead
+    of `pack.recorded_model`, action.yml price inputs + optional `[adapter]`
+    extra. Release before merging jev-packs result columns (CI pins 0.2.0).

{jevassert-0.1.0 → jevassert-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: jevassert
-Version: 0.1.0
+Version: 0.2.0
 Summary: Record/replay regression tests for Jev (TypeSafe System One) question packs — accuracy, calibration and cost gates in CI.
 Project-URL: Homepage, https://github.com/dtduc-git/jevassert
 Project-URL: Issues, https://github.com/dtduc-git/jevassert/issues
@@ -22,11 +22,15 @@ Classifier: Topic :: Software Development :: Testing
 Requires-Python: >=3.10
 Requires-Dist: httpx>=0.27
 Requires-Dist: pyyaml>=6.0
+Provides-Extra: adapter
+Requires-Dist: boto3>=1.34; extra == 'adapter'
+Requires-Dist: system-one-adapter[anthropic,openai]<0.3,>=0.2; extra == 'adapter'
 Description-Content-Type: text/markdown
 
 # jevassert
 
 [![CI](https://github.com/dtduc-git/jevassert/actions/workflows/ci.yml/badge.svg)](https://github.com/dtduc-git/jevassert/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/jevassert.svg)](https://pypi.org/project/jevassert/)
 [![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENSE)
 
 Regression tests for [Jev](https://typesafe.ai) question packs: assert
@@ -63,6 +67,42 @@ uvx jevassert record examples/demo-triage -o triage-next.jsonl
 uvx jevassert compare examples/demo-triage --a triage.jsonl --b triage-next.jsonl
 ```
 
+## Benchmarking other backends
+
+The same pack, runner and metrics can record a **general-purpose LLM** instead
+of Jev, through TypeSafe's official
+[system-one-adapter](https://github.com/typesafe-ai/system-one-adapter-python):
+same questions, same golden cases, no TypeSafe key involved. This is how the
+`results/` baselines in [jev-packs](https://github.com/dtduc-git/jev-packs) are
+produced — one ground truth, several backends.
+
+```sh
+pip install 'jevassert[adapter]'   # optional extra; brings the provider SDKs
+
+# any OpenAI-compatible endpoint (Ollama, vLLM, gateways)
+uvx jevassert record packs/sms-spam -o sms-qwen.jsonl \
+  --backend openai --model qwen2.5:7b --base-url http://localhost:11434/v1
+
+# Claude (needs ANTHROPIC_API_KEY)
+uvx jevassert record packs/sms-spam -o sms-claude.jsonl \
+  --backend anthropic --model claude-haiku-4-5
+```
+
+Adapter recordings use structured outputs in probabilities mode, normalize the
+distributions and allow two corrective retries on malformed output — the same
+settings for every LLM backend, so comparisons are like for like. Cost is
+priced at `check` time, not record time:
+
+```sh
+# self-hosted endpoint: free
+uvx jevassert check packs/sms-spam -p sms-qwen.jsonl --input-price 0 --output-price 0
+# hosted API at list price
+uvx jevassert check packs/sms-spam -p sms-claude.jsonl --input-price 1 --output-price 5
+```
+
+`check` reports the models actually present in the recording (not the pack's
+pinned version) and prices both input and output tokens.
+
 ## Packs
 
 A pack follows **spec v0** (canonical:
@@ -150,8 +190,8 @@ the Action enforces the gates on every pull request.
 
 | command | what it does |
 |---|---|
-| `jevassert record PACK -o FILE [--model M] [--resume] [--rpm N] [--dry-run] [--shuffle-options SEED] [--repeat N]` | call Jev for every case, write predictions JSONL |
-| `jevassert check PACK -p FILE [--failures] [--bootstrap N] [--target-precision P] [--partition dev\|test]` | compute metrics, evaluate gates, exit 0/1/2 |
+| `jevassert record PACK -o FILE [--model M] [--backend typesafe\|openai\|anthropic] [--base-url URL] [--resume] [--rpm N] [--dry-run] [--shuffle-options SEED] [--repeat N]` | call the backend for every case, write predictions JSONL |
+| `jevassert check PACK -p FILE [--failures] [--bootstrap N] [--target-precision P] [--input-price X] [--output-price Y] [--partition dev\|test]` | compute metrics, evaluate gates, exit 0/1/2 |
 | `jevassert compare PACK --a A --b B` | paired accuracy deltas + exact McNemar p-value |
 
 `record` extras: `--dry-run` estimates tokens/cost from the pack without sending
@@ -166,7 +206,9 @@ decision count across rounds.
 on test without fooling yourself.
 
 Environment: `TYPESAFE_API_KEY` (record only), `TYPESAFE_BASE_URL` (override
-the endpoint, e.g. a local Jev-compatible replica).
+the endpoint, e.g. a local Jev-compatible replica), `OPENAI_API_KEY` (only when
+the `openai` adapter backend points at a hosted endpoint), `ANTHROPIC_API_KEY`
+(`anthropic` adapter backend).
 
 ## Reading the numbers
 

{jevassert-0.1.0 → jevassert-0.2.0}/README.md

@@ -1,6 +1,7 @@
 # jevassert
 
 [![CI](https://github.com/dtduc-git/jevassert/actions/workflows/ci.yml/badge.svg)](https://github.com/dtduc-git/jevassert/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/jevassert.svg)](https://pypi.org/project/jevassert/)
 [![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENSE)
 
 Regression tests for [Jev](https://typesafe.ai) question packs: assert
@@ -37,6 +38,42 @@ uvx jevassert record examples/demo-triage -o triage-next.jsonl
 uvx jevassert compare examples/demo-triage --a triage.jsonl --b triage-next.jsonl
 ```
 
+## Benchmarking other backends
+
+The same pack, runner and metrics can record a **general-purpose LLM** instead
+of Jev, through TypeSafe's official
+[system-one-adapter](https://github.com/typesafe-ai/system-one-adapter-python):
+same questions, same golden cases, no TypeSafe key involved. This is how the
+`results/` baselines in [jev-packs](https://github.com/dtduc-git/jev-packs) are
+produced — one ground truth, several backends.
+
+```sh
+pip install 'jevassert[adapter]'   # optional extra; brings the provider SDKs
+
+# any OpenAI-compatible endpoint (Ollama, vLLM, gateways)
+uvx jevassert record packs/sms-spam -o sms-qwen.jsonl \
+  --backend openai --model qwen2.5:7b --base-url http://localhost:11434/v1
+
+# Claude (needs ANTHROPIC_API_KEY)
+uvx jevassert record packs/sms-spam -o sms-claude.jsonl \
+  --backend anthropic --model claude-haiku-4-5
+```
+
+Adapter recordings use structured outputs in probabilities mode, normalize the
+distributions and allow two corrective retries on malformed output — the same
+settings for every LLM backend, so comparisons are like for like. Cost is
+priced at `check` time, not record time:
+
+```sh
+# self-hosted endpoint: free
+uvx jevassert check packs/sms-spam -p sms-qwen.jsonl --input-price 0 --output-price 0
+# hosted API at list price
+uvx jevassert check packs/sms-spam -p sms-claude.jsonl --input-price 1 --output-price 5
+```
+
+`check` reports the models actually present in the recording (not the pack's
+pinned version) and prices both input and output tokens.
+
 ## Packs
 
 A pack follows **spec v0** (canonical:
@@ -124,8 +161,8 @@ the Action enforces the gates on every pull request.
 
 | command | what it does |
 |---|---|
-| `jevassert record PACK -o FILE [--model M] [--resume] [--rpm N] [--dry-run] [--shuffle-options SEED] [--repeat N]` | call Jev for every case, write predictions JSONL |
-| `jevassert check PACK -p FILE [--failures] [--bootstrap N] [--target-precision P] [--partition dev\|test]` | compute metrics, evaluate gates, exit 0/1/2 |
+| `jevassert record PACK -o FILE [--model M] [--backend typesafe\|openai\|anthropic] [--base-url URL] [--resume] [--rpm N] [--dry-run] [--shuffle-options SEED] [--repeat N]` | call the backend for every case, write predictions JSONL |
+| `jevassert check PACK -p FILE [--failures] [--bootstrap N] [--target-precision P] [--input-price X] [--output-price Y] [--partition dev\|test]` | compute metrics, evaluate gates, exit 0/1/2 |
 | `jevassert compare PACK --a A --b B` | paired accuracy deltas + exact McNemar p-value |
 
 `record` extras: `--dry-run` estimates tokens/cost from the pack without sending
@@ -140,7 +177,9 @@ decision count across rounds.
 on test without fooling yourself.
 
 Environment: `TYPESAFE_API_KEY` (record only), `TYPESAFE_BASE_URL` (override
-the endpoint, e.g. a local Jev-compatible replica).
+the endpoint, e.g. a local Jev-compatible replica), `OPENAI_API_KEY` (only when
+the `openai` adapter backend points at a hosted endpoint), `ANTHROPIC_API_KEY`
+(`anthropic` adapter backend).
 
 ## Reading the numbers
 

{jevassert-0.1.0 → jevassert-0.2.0}/action.yml

@@ -25,6 +25,14 @@ inputs:
     description: Where to write the markdown report (empty = skip)
     required: false
     default: ""
+  input-price:
+    description: USD per million input tokens for the cost line (default 0.042, Jev list price)
+    required: false
+    default: ""
+  output-price:
+    description: USD per million output tokens for the cost line (default 0 = free)
+    required: false
+    default: ""
 
 runs:
   using: composite
@@ -39,6 +47,8 @@ runs:
         JUNIT: ${{ inputs.junit }}
         REPORT: ${{ inputs.report }}
         VERSION: ${{ inputs.version }}
+        INPUT_PRICE: ${{ inputs.input-price }}
+        OUTPUT_PRICE: ${{ inputs.output-price }}
       run: |
         if [ "$VERSION" = "latest" ]; then
           package="jevassert"
@@ -49,4 +59,10 @@ runs:
         if [ -n "$REPORT" ]; then
           args+=(--report "$REPORT")
         fi
+        if [ -n "$INPUT_PRICE" ]; then
+          args+=(--input-price "$INPUT_PRICE")
+        fi
+        if [ -n "$OUTPUT_PRICE" ]; then
+          args+=(--output-price "$OUTPUT_PRICE")
+        fi
         uvx "$package" "${args[@]}"

{jevassert-0.1.0 → jevassert-0.2.0}/pyproject.toml

@@ -34,6 +34,12 @@ dependencies = [
   "pyyaml>=6.0",
 ]
 
+[project.optional-dependencies]
+adapter = [
+  "system-one-adapter[openai,anthropic]>=0.2,<0.3",
+  "boto3>=1.34",
+]
+
 [project.scripts]
 jevassert = "jevassert.cli:main"
 

{jevassert-0.1.0 → jevassert-0.2.0}/src/jevassert/__init__.py

@@ -1,3 +1,3 @@
 """jevassert — record/replay regression tests for Jev question packs."""
 
-__version__ = "0.1.0"
+__version__ = "0.2.0"

jevassert-0.2.0/src/jevassert/backends.py

@@ -0,0 +1,115 @@
+"""Recording backends: the TypeSafe API, or any LLM via ``system-one-adapter``.
+
+``record --backend openai|anthropic`` runs the same pack, the same runner and
+the same metrics against a general-purpose LLM through the official adapter
+(https://github.com/typesafe-ai/system-one-adapter-python). Used to record
+baseline evidence for the jev-packs benchmark, and to compare a candidate
+endpoint against Jev on identical ground truth.
+"""
+
+from __future__ import annotations
+
+import os
+import time
+from typing import Any
+
+from .client import JevError
+
+
+class AdapterClient:
+    """``JevClient``-shaped client backed by an LLM through system-one-adapter.
+
+    ``backend`` selects the adapter provider: ``openai`` covers any
+    OpenAI-compatible endpoint (Ollama, vLLM, gateways; set ``base_url``),
+    ``anthropic`` calls Claude directly. Recording never touches the TypeSafe
+    API, so no TypeSafe key is needed.
+    """
+
+    def __init__(self, backend: str, model: str, base_url: str | None = None) -> None:
+        try:
+            from system_one_adapter import RetryPolicy, SystemOneAdapterClient
+        except ImportError as exc:
+            raise JevError(
+                "adapter backends need system-one-adapter: pip install 'jevassert[adapter]'"
+            ) from exc
+        self.backend = backend
+        self.model_name = model
+        self.api_key = "unused"  # runner.record only checks presence
+        self._provider = _build_provider(backend, model, base_url)
+        self._client = SystemOneAdapterClient(
+            structured_outputs=True,
+            llm_answer_mode="probabilities",
+            normalize_probabilities=True,
+            n_retry_malformed_structure=2,
+            retry=RetryPolicy(max_retries=3),
+        )
+
+    def system_one(
+        self, state: Any, questions: dict[str, Any], model: str = ""
+    ) -> tuple[dict[str, Any], float]:
+        """Evaluate questions against the configured LLM; returns (response, latency_ms)."""
+        started = time.perf_counter()
+        try:
+            response = self._client.system_one(state, questions, model=self._provider)
+        except Exception as exc:  # adapter raises SDK error types
+            raise _as_jev_error(exc) from exc
+        return response.model_dump(), (time.perf_counter() - started) * 1000
+
+    def close(self) -> None:
+        self._client.close()
+
+
+def _build_provider(backend: str, model: str, base_url: str | None) -> Any:
+    if backend == "openai":
+        from system_one_adapter.providers.openai import OpenAIProvider
+
+        return OpenAIProvider(
+            model, base_url=base_url, api_key=os.environ.get("OPENAI_API_KEY") or "not-needed"
+        )
+    if backend == "anthropic":
+        if not os.environ.get("ANTHROPIC_API_KEY"):
+            raise JevError("ANTHROPIC_API_KEY is not set (required for --backend anthropic)")
+        from system_one_adapter.providers.anthropic import AnthropicProvider
+
+        return AnthropicProvider(model)
+    if backend == "bedrock":
+        return _build_bedrock_provider(model)
+    raise ValueError(f"unknown backend {backend!r} (expected openai, anthropic or bedrock)")
+
+
+def _build_bedrock_provider(model: str) -> Any:
+    """Claude through AWS Bedrock: the adapter's AnthropicProvider over AnthropicBedrock.
+
+    Credentials come from the standard AWS chain (``AWS_PROFILE``/``AWS_REGION``
+    or instance roles). Newer models need an inference profile ID, e.g.
+    ``global.anthropic.claude-haiku-4-5-20251001-v1:0``.
+    """
+    import anthropic
+    from system_one_adapter.providers.anthropic import AnthropicProvider
+
+    class BedrockProvider(AnthropicProvider):  # type: ignore[misc, valid-type]
+        def __init__(self, model_name: str) -> None:
+            super().__init__(model_name)
+            self._client = anthropic.AnthropicBedrock(max_retries=0)
+
+    return BedrockProvider(model)
+
+
+def _as_jev_error(exc: Exception) -> JevError:
+    """Map adapter/SDK failures onto ``JevError`` so fail-fast paths keep working."""
+    status = getattr(exc, "status", None)
+    if status is None:
+        text = str(exc).lower()
+        if any(
+            marker in text
+            for marker in (
+                "api key",
+                "authentication",
+                "unauthorized",
+                "credential",
+                "access denied",
+                "accessdenied",
+            )
+        ):
+            status = 401
+    return JevError(str(exc), status=status)

{jevassert-0.1.0 → jevassert-0.2.0}/src/jevassert/cli.py

@@ -11,6 +11,7 @@ from pathlib import Path
 from typing import Any
 
 from . import __version__
+from .backends import AdapterClient
 from .client import JevClient, JevError
 from .compare import CompareResult
 from .compare import compare as compare_recordings
@@ -61,7 +62,21 @@ def _build_parser() -> argparse.ArgumentParser:
         default=None,
         help="model version to record (default: pack.tested or jev-latest)",
     )
-    record_parser.add_argument("--base-url", default=None, help="override TYPESAFE_BASE_URL")
+    record_parser.add_argument(
+        "--backend",
+        choices=("typesafe", "openai", "anthropic", "bedrock"),
+        default="typesafe",
+        help="typesafe = TypeSafe API; openai = any OpenAI-compatible endpoint "
+        "(--base-url, e.g. Ollama); anthropic = Claude API; bedrock = Claude via "
+        "AWS Bedrock (AWS_PROFILE/AWS_REGION, use an inference profile id). The "
+        "last three go through system-one-adapter (pip install 'jevassert[adapter]') "
+        "and require --model",
+    )
+    record_parser.add_argument(
+        "--base-url",
+        default=None,
+        help="override TYPESAFE_BASE_URL (or the OpenAI-compatible endpoint for --backend openai)",
+    )
     record_parser.add_argument(
         "--resume",
         action="store_true",
@@ -113,6 +128,18 @@ def _build_parser() -> argparse.ArgumentParser:
         default=None,
         help="suggest the highest-coverage threshold that reaches this precision",
     )
+    check_parser.add_argument(
+        "--input-price",
+        type=float,
+        default=INPUT_USD_PER_MTOK,
+        help=f"USD per million input tokens (default {INPUT_USD_PER_MTOK}, the Jev list price)",
+    )
+    check_parser.add_argument(
+        "--output-price",
+        type=float,
+        default=0.0,
+        help="USD per million output tokens (default 0: Jev output tokens are free)",
+    )
     check_parser.add_argument(
         "--partition",
         choices=("all", "dev", "test"),
@@ -154,7 +181,13 @@ def _cmd_record(args: argparse.Namespace) -> int:
         print("jevassert: error: --repeat must be >= 1", file=sys.stderr)
         return 2
 
-    client = JevClient(base_url=args.base_url)
+    if args.backend == "typesafe":
+        client: JevClient | AdapterClient = JevClient(base_url=args.base_url)
+    else:
+        if not args.model:
+            print(f"jevassert: error: --backend {args.backend} requires --model", file=sys.stderr)
+            return 2
+        client = AdapterClient(args.backend, args.model, base_url=args.base_url)
     base_out = Path(args.out)
     total_errors = 0
     try:
@@ -254,27 +287,48 @@ def _cmd_check(args: argparse.Namespace) -> int:
             f"(seed {args.partition_seed}, ratio {args.partition_ratio})",
             file=sys.stderr,
         )
-    overall = compute(pack, predictions, bootstrap=args.bootstrap)
+    overall = compute(
+        pack,
+        predictions,
+        bootstrap=args.bootstrap,
+        input_usd_per_mtok=args.input_price,
+        output_usd_per_mtok=args.output_price,
+    )
     gates = evaluate_gates(pack, overall)
     suggestion = None
     if args.target_precision is not None:
         items, _, _ = build_items(pack, predictions)
         suggestion = suggest_threshold(items, args.target_precision)
+    models = _recorded_models(predictions)
 
     if args.junit:
         Path(args.junit).write_text(render_junit(pack, gates), encoding="utf-8")
     if args.report:
         Path(args.report).write_text(
-            render_markdown(pack, overall, gates, suggestion, args.target_precision),
+            render_markdown(pack, overall, gates, suggestion, args.target_precision, models),
             encoding="utf-8",
         )
 
     if args.json:
-        print(json.dumps(_payload(pack, overall, gates, suggestion), indent=2, ensure_ascii=False))
+        payload = _payload(pack, overall, gates, suggestion)
+        payload["recorded_models"] = models
+        payload["pricing"] = {
+            "input_usd_per_mtok": args.input_price,
+            "output_usd_per_mtok": args.output_price,
+        }
+        print(json.dumps(payload, indent=2, ensure_ascii=False))
     else:
         print(
             _human_summary(
-                pack, overall, gates, args.predictions, suggestion, args.target_precision
+                pack,
+                overall,
+                gates,
+                args.predictions,
+                suggestion,
+                args.target_precision,
+                models,
+                args.input_price,
+                args.output_price,
             )
         )
         if args.failures:
@@ -414,6 +468,12 @@ def _payload(
     }
 
 
+def _recorded_models(predictions: dict[str, dict[str, Any]]) -> list[str]:
+    return sorted(
+        {record["model"] for record in predictions.values() if isinstance(record.get("model"), str)}
+    )
+
+
 def _human_summary(
     pack: Pack,
     overall: OverallMetrics,
@@ -421,12 +481,14 @@ def _human_summary(
     predictions_path: str,
     suggestion: ThresholdSuggestion | None = None,
     target_precision: float | None = None,
+    models: list[str] | None = None,
+    input_price: float = INPUT_USD_PER_MTOK,
+    output_price: float = 0.0,
 ) -> str:
     lines: list[str] = []
     tested = f", tested {pack.tested}" if pack.tested else ", provisional"
-    lines.append(
-        f"jevassert — {pack.id} v{pack.version} (record model {pack.record_model}{tested})"
-    )
+    record_model = ", ".join(models) if models else pack.record_model
+    lines.append(f"jevassert — {pack.id} v{pack.version} (record model {record_model}{tested})")
     lines.append(
         f"recording: {predictions_path} — {overall.n_cases} cases, "
         f"{overall.case_errors} errors, {overall.missing_items} missing, {overall.n_items} items"
@@ -457,9 +519,12 @@ def _human_summary(
             "(see README: Reading the numbers)"
         )
     if overall.cost_per_case_usd is not None:
+        price = f"${input_price}/M input tokens"
+        if output_price:
+            price = f"${input_price}/M input + ${output_price}/M output tokens"
         lines.append(
             f"cost/case ${overall.cost_per_case_usd:.6f} "
-            f"(total ${overall.total_cost_usd:.4f} at ${0.042}/M input tokens)"
+            f"(total ${overall.total_cost_usd:.4f} at {price})"
         )
     if overall.p95_latency_ms is not None:
         lines.append(
@@ -490,7 +555,7 @@ def _human_summary(
     lines.append("")
     lines.append("gates:")
     if not gates:
-        lines.append("  (none declared in pack.yaml)")
+        lines.append("  (none declared in gates.yaml)")
     for gate in gates:
         mark = "PASS" if gate.ok else ("SKIP" if gate.ok is None else "FAIL")
         lines.append(f"  {mark} {gate.gate}: {gate.detail}")

{jevassert-0.1.0 → jevassert-0.2.0}/src/jevassert/metrics.py

@@ -207,6 +207,8 @@ def compute(
     predictions: dict[str, dict[str, Any]],
     bootstrap: int = 1000,
     seed: int = 0,
+    input_usd_per_mtok: float = INPUT_USD_PER_MTOK,
+    output_usd_per_mtok: float = 0.0,
 ) -> OverallMetrics:
     items, case_errors, missing_items = build_items(pack, predictions)
 
@@ -249,18 +251,27 @@ def compute(
     all_correct = [item.correct for item in items]
 
     total_input_tokens = 0
+    total_output_tokens = 0
     saw_usage = False
     latencies: list[float] = []
     for case in pack.cases:
         record = predictions.get(case.id) or {}
         usage = record.get("usage")
-        if isinstance(usage, dict) and isinstance(usage.get("input_tokens"), int | float):
-            total_input_tokens += int(usage["input_tokens"])
-            saw_usage = True
+        if isinstance(usage, dict):
+            if isinstance(usage.get("input_tokens"), int | float):
+                total_input_tokens += int(usage["input_tokens"])
+                saw_usage = True
+            if isinstance(usage.get("output_tokens"), int | float):
+                total_output_tokens += int(usage["output_tokens"])
         if isinstance(record.get("latency_ms"), int | float):
             latencies.append(float(record["latency_ms"]))
 
-    total_cost = (total_input_tokens * INPUT_USD_PER_MTOK / 1_000_000) if saw_usage else None
+    total_cost = (
+        (total_input_tokens * input_usd_per_mtok + total_output_tokens * output_usd_per_mtok)
+        / 1_000_000
+        if saw_usage
+        else None
+    )
     cost_per_case = (total_cost / len(pack.cases)) if total_cost is not None else None
 
     return OverallMetrics(

{jevassert-0.1.0 → jevassert-0.2.0}/src/jevassert/report.py

@@ -14,18 +14,20 @@ def render_markdown(
     gates: list[GateResult],
     suggestion: ThresholdSuggestion | None = None,
     target_precision: float | None = None,
+    recorded_models: list[str] | None = None,
 ) -> str:
     lines: list[str] = []
     lines.append(f"# jevassert report — {pack.id} v{pack.version}")
     lines.append("")
-    lines.append(
-        f"- model: `{pack.record_model}`"
-        + (
-            f" (recorded against `{pack.tested}`)"
-            if pack.tested
-            else " (provisional — no pinned version)"
-        )
-    )
+    models = list(recorded_models or ([pack.tested] if pack.tested else []))
+    model_label = ", ".join(f"`{m}`" for m in models) if models else f"`{pack.record_model}`"
+    if models and pack.tested and models == [pack.tested]:
+        suffix = f" (recorded against `{pack.tested}`)"
+    elif pack.tested:
+        suffix = f" (pack pinned to `{pack.tested}`)"
+    else:
+        suffix = " (provisional — no pinned version)"
+    lines.append(f"- model: {model_label}{suffix}")
     lines.append(
         f"- cases: {overall.n_cases} ({overall.case_errors} errors, "
         f"{overall.missing_items} missing answers)"