jevcheck 0.2.0__tar.gz

jevcheck-0.2.0/.github/workflows/ci.yml

@@ -0,0 +1,59 @@
+name: ci
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.11", "3.12"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Install
+        run: pip install -e ".[dev]"
+      - name: Test
+        run: pytest
+
+  packaging:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - name: Build wheel and sdist
+        run: |
+          pip install build
+          python -m build
+      - name: Installed-wheel CLI smoke
+        run: |
+          python -m venv "$RUNNER_TEMP/wheel"
+          "$RUNNER_TEMP/wheel/bin/pip" install dist/*.whl
+          "$RUNNER_TEMP/wheel/bin/jevcheck" eval fixtures/support-triage.json \
+            --candidate-model jev-1.14 \
+            --answers fixtures/replay-unchanged.json
+          "$RUNNER_TEMP/wheel/bin/jevcheck" compare fixtures/support-triage.json \
+            --from fixtures/replay-baseline.json \
+            --to jev-1.14 \
+            --answers fixtures/replay-unchanged.json
+          "$RUNNER_TEMP/wheel/bin/python" -c "import jevcheck,inspect; print(jevcheck.__version__)"
+      - name: sdist includes tests/fixtures and pytest collects
+        run: |
+          python -m venv "$RUNNER_TEMP/sdist"
+          "$RUNNER_TEMP/sdist/bin/pip" install dist/*.tar.gz pytest
+          mkdir -p "$RUNNER_TEMP/srcdist"
+          tar -xzf dist/*.tar.gz -C "$RUNNER_TEMP/srcdist"
+          srcdir=$(echo "$RUNNER_TEMP/srcdist"/jevcheck-*)
+          test -f "$srcdir/tests/helpers.py"
+          test -f "$srcdir/tests/conftest.py"
+          test -f "$srcdir/fixtures/support-triage.json"
+          test -f "$srcdir/docs/contract.md"
+          cd "$srcdir"
+          "$RUNNER_TEMP/sdist/bin/pytest"

jevcheck-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Nithilan Kumaran
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

jevcheck-0.2.0/MANIFEST.in

@@ -0,0 +1,10 @@
+graft tests
+graft fixtures
+graft docs
+include LICENSE
+include README.md
+include pyproject.toml
+recursive-include .github *.yml
+global-exclude *.py[cod]
+global-exclude __pycache__
+global-exclude .env

jevcheck-0.2.0/PKG-INFO

@@ -0,0 +1,162 @@
+Metadata-Version: 2.4
+Name: jevcheck
+Version: 0.2.0
+Summary: pytest for Jev — pin production behavioral contracts and eval candidate model upgrades
+Author: Nithilan Kumaran
+License: MIT
+Project-URL: Homepage, https://github.com/sathariels/jevcheck
+Project-URL: Documentation, https://github.com/sathariels/jevcheck
+Keywords: jev,typesafe,eval,contract,model-upgrade
+Classifier: Development Status :: 3 - Alpha
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Typing :: Typed
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pydantic>=2
+Requires-Dist: typesafe-sdk>=0.7.0
+Provides-Extra: dev
+Requires-Dist: pytest>=8; extra == "dev"
+Dynamic: license-file
+
+# jevcheck
+
+**pytest for Jev.** Pin what production is allowed to do, eval a candidate model, and fail the upgrade when answers flip or confidence drops.
+
+Probabilities and model versions move. A raw `0.94` is not a release decision. jevcheck records a **production contract** (baseline model + fixtures + expected answers) and evals a candidate against that fixture.
+
+**v0.1** `eval` is fixture-versus-candidate. **v0.2** adds two-model execution: `record` a baseline model's answers, then `compare` a candidate against that snapshot (or fetch both models live). See [`docs/adr-009-two-model-compare.md`](docs/adr-009-two-model-compare.md).
+
+The repo’s `jev-1.13` / `jev-1.14` strings are **unverified example pin labels** used by fixtures. A documented TypeSafe version pin (2026-09-19 model list) is `jev-1.13.0`. Floating aliases `jev-latest` and `jev-preview` are rejected unless you pass `--allow-unpinned`. With that opt-in, a response whose `model` is the concrete resolved ID (for example `jev-1.13.0`) is accepted; the eval report prints that resolved model. Concrete pins still require exact identity.
+
+```
+pin contract  →  ship on the pinned model  →  jevcheck eval     →  compatible or breaking
+                                          →  record → compare  →  compatible or breaking   (v0.2)
+```
+
+Reports: **unchanged** / **confidence regressions** / **answer flips**, with exact diffs (`billing→general`, `0.94→0.71`) and a nonzero exit on failure.
+
+## Install
+
+```bash
+pip install -e ".[dev]"
+export TYPESAFE_API_KEY=...   # live Jev only; never commit this
+```
+
+Auth is `TYPESAFE_API_KEY` only. Unit tests mock the network.
+
+## Pin → eval → upgrade
+
+1. Write a contract (see [`docs/contract.md`](docs/contract.md)) against the model you ship.
+2. Call Jev with that **explicit** model. `jev-latest` and `jev-preview` are rejected unless you opt in. Concrete pins require `response.model` to match exactly. An opted-in alias may resolve to a nonempty concrete (non-alias) response model, which is reported.
+3. Before upgrading, eval the candidate (example fixture label — not a verified live ID):
+
+```bash
+# Live call. Pin a catalog version such as jev-1.13.0 in production.
+# The example below matches this repo's replay fixtures only.
+jevcheck eval fixtures/support-triage.json --candidate-model jev-1.14
+```
+
+Replay recorded answers (CI / no key):
+
+```bash
+jevcheck eval fixtures/support-triage.json \
+  --candidate-model jev-1.14 \
+  --answers fixtures/replay-breaking.json
+```
+
+4. Compatible (exit 0) → change the pin. Breaking (exit 1) → read the diffs; do not upgrade.
+
+## Upgrade flow (v0.2): record baseline → compare candidate
+
+Two-model execution is a **v0.2** capability. `eval` above stays the v0.1 fixture path and is unchanged.
+
+1. Record answers from the model you ship. The file is the same replay JSON `eval --answers` already accepts:
+
+```bash
+# Live (needs TYPESAFE_API_KEY). Pin a catalog version such as jev-1.13.0 in production.
+jevcheck record fixtures/support-triage.json --out baseline-answers.json
+
+# CI / no key: copy a previously recorded snapshot through the same command.
+jevcheck record fixtures/support-triage.json \
+  --answers fixtures/replay-baseline.json \
+  --out baseline-answers.json
+```
+
+`record` writes only after every answer kind matches the contract question (choice / noul / score). A mismatch is exit 2. Opted-in aliases print the resolved response model in the summary (`jev-preview → jev-1.13.0`).
+
+2. Compare the candidate against that snapshot:
+
+```bash
+# Live candidate against recorded baseline
+jevcheck compare fixtures/support-triage.json \
+  --from baseline-answers.json \
+  --to jev-1.14
+
+# CI / no key: recorded baseline + candidate replay
+jevcheck compare fixtures/support-triage.json \
+  --from fixtures/replay-baseline.json \
+  --to jev-1.14 \
+  --answers fixtures/replay-breaking.json
+```
+
+3. Or fetch both models in one step (live; needs a key):
+
+```bash
+jevcheck compare fixtures/support-triage.json \
+  --from-model jev-1.13 \
+  --to jev-1.14
+```
+
+`--from` and `--from-model` are mutually exclusive. Identity rules are the v0.1 pins: concrete names must match `response.model` exactly; `jev-latest` / `jev-preview` still need `--allow-unpinned`. Exit codes stay 0 compatible / 1 breaking / 2 usage-or-identity / 3 ops.
+
+Verified System One fields: [`docs/jev-api.md`](docs/jev-api.md).
+
+## Example
+
+```python
+from jevcheck import JevClient, evaluate, load_contract
+
+contract = load_contract("fixtures/support-triage.json")
+client = JevClient(model="jev-1.14")  # example fixture label; pin a catalog version in production
+
+report = evaluate(
+    contract,
+    lambda case: client.system_one(
+        state=case.state,
+        questions=case.questions,
+        model="jev-1.14",
+    ),
+    candidate_model="jev-1.14",
+)
+print(report.summary())
+if report.breaking:
+    raise SystemExit(1)
+```
+
+A `Gate` helper exists for auto / ask-human / reject thresholds. It is optional and is not the product.
+
+## Develop
+
+```bash
+pip install -e ".[dev]"
+pytest
+python -m jevcheck eval fixtures/support-triage.json \
+  --candidate-model jev-1.14 \
+  --answers fixtures/replay-unchanged.json
+python -m jevcheck compare fixtures/support-triage.json \
+  --from fixtures/replay-baseline.json \
+  --to jev-1.14 \
+  --answers fixtures/replay-unchanged.json
+```
+
+## For agents / audits
+
+v0.2 two-model lock: [`docs/adr-009-two-model-compare.md`](docs/adr-009-two-model-compare.md).
+
+See [`docs/release-readiness-audit-v0.1.md`](docs/release-readiness-audit-v0.1.md). Recheck: [`docs/release-readiness-audit-v0.1-recheck.md`](docs/release-readiness-audit-v0.1-recheck.md).
+
+MIT.

jevcheck-0.2.0/README.md

@@ -0,0 +1,138 @@
+# jevcheck
+
+**pytest for Jev.** Pin what production is allowed to do, eval a candidate model, and fail the upgrade when answers flip or confidence drops.
+
+Probabilities and model versions move. A raw `0.94` is not a release decision. jevcheck records a **production contract** (baseline model + fixtures + expected answers) and evals a candidate against that fixture.
+
+**v0.1** `eval` is fixture-versus-candidate. **v0.2** adds two-model execution: `record` a baseline model's answers, then `compare` a candidate against that snapshot (or fetch both models live). See [`docs/adr-009-two-model-compare.md`](docs/adr-009-two-model-compare.md).
+
+The repo’s `jev-1.13` / `jev-1.14` strings are **unverified example pin labels** used by fixtures. A documented TypeSafe version pin (2026-09-19 model list) is `jev-1.13.0`. Floating aliases `jev-latest` and `jev-preview` are rejected unless you pass `--allow-unpinned`. With that opt-in, a response whose `model` is the concrete resolved ID (for example `jev-1.13.0`) is accepted; the eval report prints that resolved model. Concrete pins still require exact identity.
+
+```
+pin contract  →  ship on the pinned model  →  jevcheck eval     →  compatible or breaking
+                                          →  record → compare  →  compatible or breaking   (v0.2)
+```
+
+Reports: **unchanged** / **confidence regressions** / **answer flips**, with exact diffs (`billing→general`, `0.94→0.71`) and a nonzero exit on failure.
+
+## Install
+
+```bash
+pip install -e ".[dev]"
+export TYPESAFE_API_KEY=...   # live Jev only; never commit this
+```
+
+Auth is `TYPESAFE_API_KEY` only. Unit tests mock the network.
+
+## Pin → eval → upgrade
+
+1. Write a contract (see [`docs/contract.md`](docs/contract.md)) against the model you ship.
+2. Call Jev with that **explicit** model. `jev-latest` and `jev-preview` are rejected unless you opt in. Concrete pins require `response.model` to match exactly. An opted-in alias may resolve to a nonempty concrete (non-alias) response model, which is reported.
+3. Before upgrading, eval the candidate (example fixture label — not a verified live ID):
+
+```bash
+# Live call. Pin a catalog version such as jev-1.13.0 in production.
+# The example below matches this repo's replay fixtures only.
+jevcheck eval fixtures/support-triage.json --candidate-model jev-1.14
+```
+
+Replay recorded answers (CI / no key):
+
+```bash
+jevcheck eval fixtures/support-triage.json \
+  --candidate-model jev-1.14 \
+  --answers fixtures/replay-breaking.json
+```
+
+4. Compatible (exit 0) → change the pin. Breaking (exit 1) → read the diffs; do not upgrade.
+
+## Upgrade flow (v0.2): record baseline → compare candidate
+
+Two-model execution is a **v0.2** capability. `eval` above stays the v0.1 fixture path and is unchanged.
+
+1. Record answers from the model you ship. The file is the same replay JSON `eval --answers` already accepts:
+
+```bash
+# Live (needs TYPESAFE_API_KEY). Pin a catalog version such as jev-1.13.0 in production.
+jevcheck record fixtures/support-triage.json --out baseline-answers.json
+
+# CI / no key: copy a previously recorded snapshot through the same command.
+jevcheck record fixtures/support-triage.json \
+  --answers fixtures/replay-baseline.json \
+  --out baseline-answers.json
+```
+
+`record` writes only after every answer kind matches the contract question (choice / noul / score). A mismatch is exit 2. Opted-in aliases print the resolved response model in the summary (`jev-preview → jev-1.13.0`).
+
+2. Compare the candidate against that snapshot:
+
+```bash
+# Live candidate against recorded baseline
+jevcheck compare fixtures/support-triage.json \
+  --from baseline-answers.json \
+  --to jev-1.14
+
+# CI / no key: recorded baseline + candidate replay
+jevcheck compare fixtures/support-triage.json \
+  --from fixtures/replay-baseline.json \
+  --to jev-1.14 \
+  --answers fixtures/replay-breaking.json
+```
+
+3. Or fetch both models in one step (live; needs a key):
+
+```bash
+jevcheck compare fixtures/support-triage.json \
+  --from-model jev-1.13 \
+  --to jev-1.14
+```
+
+`--from` and `--from-model` are mutually exclusive. Identity rules are the v0.1 pins: concrete names must match `response.model` exactly; `jev-latest` / `jev-preview` still need `--allow-unpinned`. Exit codes stay 0 compatible / 1 breaking / 2 usage-or-identity / 3 ops.
+
+Verified System One fields: [`docs/jev-api.md`](docs/jev-api.md).
+
+## Example
+
+```python
+from jevcheck import JevClient, evaluate, load_contract
+
+contract = load_contract("fixtures/support-triage.json")
+client = JevClient(model="jev-1.14")  # example fixture label; pin a catalog version in production
+
+report = evaluate(
+    contract,
+    lambda case: client.system_one(
+        state=case.state,
+        questions=case.questions,
+        model="jev-1.14",
+    ),
+    candidate_model="jev-1.14",
+)
+print(report.summary())
+if report.breaking:
+    raise SystemExit(1)
+```
+
+A `Gate` helper exists for auto / ask-human / reject thresholds. It is optional and is not the product.
+
+## Develop
+
+```bash
+pip install -e ".[dev]"
+pytest
+python -m jevcheck eval fixtures/support-triage.json \
+  --candidate-model jev-1.14 \
+  --answers fixtures/replay-unchanged.json
+python -m jevcheck compare fixtures/support-triage.json \
+  --from fixtures/replay-baseline.json \
+  --to jev-1.14 \
+  --answers fixtures/replay-unchanged.json
+```
+
+## For agents / audits
+
+v0.2 two-model lock: [`docs/adr-009-two-model-compare.md`](docs/adr-009-two-model-compare.md).
+
+See [`docs/release-readiness-audit-v0.1.md`](docs/release-readiness-audit-v0.1.md). Recheck: [`docs/release-readiness-audit-v0.1-recheck.md`](docs/release-readiness-audit-v0.1-recheck.md).
+
+MIT.

jevcheck-0.2.0/docs/adr-009-two-model-compare.md

@@ -0,0 +1,64 @@
+# ADR 009 — Two-model / baseline comparison (v0.2 slice)
+
+Status: accepted for the v0.2 slice. Packaging version is `0.2.0`. The owner tags and publishes to PyPI after merge; this PR does not create a GitHub release. This document is the lock for `record` / `compare`.
+
+v0.1 `eval` remains fixture-versus-candidate. This slice adds a second, explicit path so upgrade safety can mean **baseline answers versus candidate answers**, not only authored `expect` versus candidate.
+
+## Commands
+
+| Command | Meaning |
+| --- | --- |
+| `record` | Given a v0.1 contract and a baseline model, fetch System One answers for every case and write a replay JSON. The file is the same shape as `eval --answers` (`dump_replay` / `load_replay`). |
+| `compare` | Evaluate a candidate against those recorded baseline answers, **or** fetch both models live. |
+| Baseline recording | The act of calling `record` (or an equivalent in-process `record_answers`). It does **not** rewrite the contract schema. Fixture `expect` stays the v0.1 authored contract. |
+
+Chosen CLI (one UX, documented here):
+
+```
+jevcheck record CONTRACT --out FILE [--baseline-model MODEL] [--answers REPLAY]
+jevcheck compare CONTRACT --from FILE --to MODEL [--answers CANDIDATE_REPLAY]
+jevcheck compare CONTRACT --from-model MODEL --to MODEL [--answers CANDIDATE_REPLAY]
+```
+
+- `--from FILE` is a recorded baseline replay. `--from-model MODEL` is a live baseline fetch. Exactly one is required.
+- `--to` is the candidate pin (same role as `eval --candidate-model`).
+- `--answers` on `record` / `compare` is a mocked replay so CI and tests need no key. On `compare` it is the **candidate** replay.
+- `--baseline-model` still overrides contract / JSONL metadata. With `--from-model`, that live pin is the baseline identity.
+- `eval CONTRACT --candidate-model … [--answers]` is unchanged.
+
+## How compare decides
+
+Compare does not invent a second scoring engine. It builds a derived contract: same cases, questions, defaults, and floors/tolerances; each `expect` field is filled from the baseline answer (`choice` / `noul`+`noul_true` / `score`, plus `baseline_confidence` from the [mapped scalar](jev-api.md#confidence-mapping-used-by-jevcheck)). Then it runs existing `evaluate()` on the candidate.
+
+`record` and `compare` reject a wrong-kind answer (choice↔ChoiceAnswer, noul↔NoulAnswer, score↔ScoreAnswer) as a usage error (exit 2) rather than writing a snapshot `compare` cannot load. Missing baseline cases or missing expected fields are usage errors, not silent flips. Candidate missing answers stay v0.1 **answer flips**.
+
+Compare does **not** invent a default `confidence_tolerance`. Drop detection uses the contract’s resolved defaults, same as v0.1.
+
+## Identity and pinning
+
+Reuse v0.1 `require_pinned` / `require_response_identity` for every requested model and every response `model`:
+
+- Concrete pins: exact string equality. `jev-1.13` is not `jev-1.13.0`.
+- Floating aliases (`jev-latest`, `jev-preview`, or any name containing `latest` / `preview`) stay rejected unless `--allow-unpinned`.
+- With that opt-in, a nonempty concrete (non-alias) response model is accepted and reported. Record and compare summaries print `alias → resolved` (for example `jev-preview → jev-1.13.0`). jevcheck never rewrites a name.
+
+`record` identity-checks the baseline. `compare --from` identity-checks each recorded baseline response against the contract / `--baseline-model` pin. `compare --from-model` identity-checks the live baseline. The candidate is identity-checked as in `eval`.
+
+## Exit codes (unchanged)
+
+| Code | Meaning |
+| --- | --- |
+| 0 | Compatible (`compare`) or successful write (`record`) |
+| 1 | Behavioral contract failure (flips / confidence regressions) |
+| 2 | Usage / input / identity / missing answers / unpinned without opt-in |
+| 3 | Operational (SDK HTTP / invalid API body) |
+
+Secret redaction on error text is unchanged.
+
+## Non-goals
+
+- Not a pytest plugin.
+- `Gate` is unchanged and is still not the product.
+- Contract JSON schema stays v0.1 (`version: "0.1"`). Record writes a replay file; it does not emit a new contract format.
+- No TypeSafe / System One fields beyond [docs/jev-api.md](jev-api.md).
+- No `0.2.0` git tag in this PR.

jevcheck-0.2.0/docs/contract.md

@@ -0,0 +1,125 @@
+# jevcheck contract format (v0.1)
+
+A **contract** is the production baseline: pinned model, fixtures, and expected answers. `jevcheck eval` compares a candidate model against that contract and reports unchanged cases, confidence regressions, and answer flips.
+
+This format is owner-locked for v0.1. Ask before changing it.
+
+## Files
+
+- **JSON** (recommended): one object with metadata + `cases`.
+- **JSONL**: one case object per line. Blank lines ignored. Optional first-line metadata object with `"_meta": true`.
+
+## Contract object
+
+```json
+{
+  "version": "0.1",
+  "name": "support-triage",
+  "baseline_model": "jev-1.13",
+  "allow_unpinned": false,
+  "defaults": {
+    "min_confidence": 0.8,
+    "confidence_tolerance": 0.1,
+    "noul_true_threshold": 0.5,
+    "noul_tolerance": 0.1,
+    "score_tolerance": 0.5
+  },
+  "cases": []
+}
+```
+
+| Field | Required | Meaning |
+| --- | --- | --- |
+| `version` | yes | Must be `"0.1"` |
+| `name` | no | Shown in the CLI summary |
+| `baseline_model` | yes for JSON | Pinned production model this contract was recorded against. Empty `""` is rejected. Floating aliases (`jev-latest`, `jev-preview`, or any name containing `latest`/`preview`) are rejected unless `allow_unpinned` / `--allow-unpinned`. jevcheck does not rewrite aliases: `jev-1.13` is not `jev-1.13.0`. |
+| `allow_unpinned` | no | If `true`, floating `jev-latest` / `jev-preview` names are allowed (default `false`) |
+| `defaults` | no | Applied when a case field omits the same key |
+| `cases` | yes for JSON | Fixture list |
+
+JSONL without a `_meta` line still needs `--baseline-model` on the CLI (or a per-eval override).
+
+## Case object
+
+```json
+{
+  "id": "ticket-001",
+  "state": "I was charged twice. Please fix this ASAP.",
+  "questions": {
+    "intent": {
+      "type": "choice",
+      "instructions": "What is this ticket about?",
+      "criteria": {"billing": null, "general": null}
+    }
+  },
+  "expect": {
+    "intent": {
+      "choice": "billing",
+      "min_confidence": 0.85,
+      "baseline_confidence": 0.94
+    }
+  }
+}
+```
+
+`state` and `questions` match the verified System One request: see `docs/jev-api.md`. Question objects use `type` of `noul`, `choice`, or `score` and only those official fields.
+
+Every key in `expect` must exist in `questions`. Extra unknown keys are rejected. An empty field expect (`{}`) is rejected. Kind-specific keys on the wrong question type (for example `score` on a choice question) are rejected. Shared confidence keys may appear on any field. This is an applicability check; the fixture key set is unchanged.
+
+## Field expectations
+
+All keys optional except that a field must state at least one *effective* constraint (a kind-specific answer key, `min_noul`, `min_confidence`, or `baseline_confidence` **together with** a resolved `confidence_tolerance` or `min_confidence` floor). A baseline-confidence-only expect with no applicable floor or tolerance after defaults is rejected (exit 2). Do not assume a default `confidence_tolerance`. Tolerance-only objects are rejected.
+
+| Key | Applies to | Meaning |
+| --- | --- | --- |
+| `choice` | choice | Expected winning label. Different label → **answer flip** |
+| `noul_true` | noul | Expected yes (`true`) or no (`false`) vs `noul_true_threshold`. Opposite polarity → **answer flip** |
+| `noul` | noul | Expected `noul` float. Used with `noul_tolerance` / `min_noul` |
+| `min_noul` | noul | Floor on the `noul` float (yes-probability) |
+| `noul_tolerance` | noul | Allowed absolute drift from `noul` (inclusive; exact 0.1 drop against 0.1 passes, with a 1e-9 epsilon so binary float subtraction of decimal tenths does not false-fail) |
+| `score` | score | Expected expected-score. Nearest integer level change → **answer flip** |
+| `score_tolerance` | score | **Level-flip suppression**, not an absolute max distance. If `\|actual - expected\|` is within this inclusive bound (same 1e-9 epsilon as confidence/noul; so 1.4 vs 1.6 at 0.2 is within), a nearest-level change is not a flip. When both values already share a nearest level (Python `round`, ties toward even: `1.5→2`, `2.5→2`), a larger float gap still counts as unchanged. Example: expected 1.8, actual 2.2, tolerance 0.1 → unchanged (both level 2). |
+| `min_confidence` | all | Floor on the [mapped scalar](jev-api.md#confidence-mapping-used-by-jevcheck) |
+| `baseline_confidence` | all | Previously recorded scalar (e.g. `0.94`) |
+| `confidence_tolerance` | all | Allowed drop from `baseline_confidence` (inclusive, same 1e-9 epsilon as noul drift) |
+
+Noul has no API `confidence`. The mapped scalar is `noul` itself.
+
+## Outcomes (per field, worst wins the case)
+
+1. **answer flip** — categorical change: choice label, noul yes/no, or score nearest level (unless `score_tolerance` suppresses that flip); or the answer is missing / wrong type.
+2. **confidence regression** — same answer, but the mapped scalar is below `min_confidence` / `min_noul`, or dropped more than `confidence_tolerance` from `baseline_confidence`, or noul drifted more than `noul_tolerance` from expected `noul`. Inclusive boundaries: a drop of exactly 0.1 against tolerance 0.1 is not a regression.
+3. **unchanged** — answer matches and scalars stay within the pinned floors/tolerances.
+
+Choice and score answers must include a nonempty probability map of finite values `>= 0` whose sum is `1 ± 1e-6`. Missing, null, empty, negative, NaN, or badly normalized maps are rejected (exit 2). The adapter does not invent an empty map. Score answers also require a nonempty `legend` (SDK-shaped responses always include it). Score `legend` / `probabilities` coming from typesafe-sdk 0.7.0 may use integer keys; jevcheck stringifies those keys.
+
+When `--candidate-model` is a **concrete pin**, every response `model` must equal that candidate exactly. A null or missing model is an error (never the string `"None"`). Opted-in floating aliases (`jev-latest`, `jev-preview`, or names containing those tokens) under `--allow-unpinned` accept a nonempty concrete response `model` that is not itself a floating alias; the eval report prints the resolved response model. Without `--allow-unpinned`, floating aliases stay rejected.
+
+Repo examples such as `jev-1.13` / `jev-1.14` are **unverified example pin labels** for fixtures, not a claim that those IDs exist on the live TypeSafe catalog. A documented version pin on the 2026-09-19 model list is `jev-1.13.0`.
+
+A higher candidate confidence is not a regression.
+
+## Replay file (`--answers`)
+
+For mocked evals and tests, a JSON object keyed by case `id`:
+
+```json
+{
+  "ticket-001": {
+    "model": "jev-1.14",
+    "usage": {"input_tokens": 10, "output_tokens": 4},
+    "answers": {
+      "intent": {
+        "type": "choice",
+        "choice": "general",
+        "confidence": 0.71,
+        "probabilities": {"billing": 0.29, "general": 0.71}
+      }
+    }
+  }
+}
+```
+
+Answer objects use only verified System One fields.
+
+`jevcheck record` (v0.2) writes this same file. `jevcheck compare --from` reads it as the baseline snapshot. See [`adr-009-two-model-compare.md`](adr-009-two-model-compare.md). `eval` remains fixture-versus-candidate.