@agentsfleet/orly 0.12.1 → 0.13.0

package/README.md

@@ -42,25 +42,30 @@ Works with Claude Code, Codex, OpenCode, and Amp.
 
 ---
 
-## 0.12: migration verification
+## 0.13: measured judgment comparison
 
-0.12.0 shipped migration verification and setup guidance. The `.orly/` layout
-and six existing `orly judge` questions originally shipped in 0.11.0.
-This release focuses on package verification, migration evidence and clearer setup guidance; it adds no new runtime judgment question.
+The 0.13.0 source candidate adds a deterministic examiner for unsupported completion claims.
+It freezes the six runtime questions and evaluates five proposed checks separately.
+Those checks cover obligations, evidence, production wiring, failure tests and finding resolution.
 
-The `agentsfleet` trial upgraded a dedicated worktree from 0.10.14 using the packed candidate.
-Owner files, hooks and verification commands survived; a second update wrote nothing.
-See the [release report](evals/release/release-report.md) for exact proof and remaining release checks.
+From a source checkout, run `bun evals/judgments/comparison/run.ts --check` to validate the frozen inputs.
+Expected output is JSON with six baseline questions, five candidates and `passed: true`.
+Run `bun evals/judgments/comparison/run.ts --offline` for whole-task exercises with hidden behavioral checks.
+A successful run reports `passed: true`; seeded false completions remain failed task outcomes.
 
-Jev, TypeSafe's bounded judgment model, supplies advice only when requested.
-Existing deterministic checks retain control of gate results.
-The comparison evaluator is parked for proposed 0.13 work; model accuracy and candidate benefits remain unmeasured.
-See the [judgment reference](docs/JUDGMENTS.md) for evidence selection, offline replay and explicit source upload.
+A separate explicit `jev-run.ts --live` command measures TypeSafe Jev answers against the frozen expectations.
+Its retained report lists every measured answer, grading match, disagreement and unavailable response.
+Replay reproduces every result with zero new requests. Autonomous agent improvement remains unmeasured.
+All five candidates remain evaluation-only; uncertain and failed replies stay in the report.
+An additional rehearsal uses the packed candidate and actual `agentsfleet` source in an isolated worktree.
+See the [evaluation design](docs/architecture/judgment-evaluation.md) for commands, evidence and limits.
+
+The `.orly/` layout, six runtime questions and 0.12.1 audit and check-diagnostic fixes remain unchanged.
+The [0.12 report](evals/release/release-report.md) retains the earlier package and consumer migration evidence.
+TypeSafe's Jev model supplies runtime advice through explicit `orly judge` requests; contributor measurement uses the separate live evaluator.
 
-0.12.1 avoids copying repository files for staged audits with no matching files
-and makes hosted-check monitoring show job states and links before reporting a failure.
 See [releases](https://github.com/agentsfleet/orly/releases) for published versions.
-The install commands below fetch the published package.
+The install commands below fetch the published package; 0.13.0 publication is a separate owner action.
 
 ---
 

package/docs/CHANGELOG.md

@@ -1,5 +1,17 @@
 # Changelog
 
+## 0.13.0 — 2026-10-07
+
+A source-checkout examiner checks unsupported completion claims and grades actual TypeSafe Jev answers against frozen expectations.
+
+- **Comparison** — freezes six runtime questions and evaluates five proposed checks with separately validated expected answers.
+- **Task evidence** — hidden behavioral checks reject seeded false completions; reports retain failed and missing attempts.
+- **Measurement** — retains typed Jev replies, uncertainty, failures and per-example grades; replay sends no new requests.
+- **OpenAPI guidance** — names the two surviving parity tests and makes build regeneration plus a clean diff an explicit verify step.
+- **Adoption** — keeps candidates evaluation-only because independent provenance and paired-task improvement remain unproved.
+
+The installed `.orly/` layout and 0.12.1 fixes remain unchanged.
+
 ## 0.12.1
 
 Staged audits avoid copying repository files when their filename scope is empty.

package/docs/JUDGMENTS.md

@@ -2,7 +2,7 @@
 type: reference
 audience: contributor
 verified: 2026-10-07
-product_version: 0.12.0
+product_version: 0.13.0
 executable: false
 ---
 
@@ -10,8 +10,9 @@ executable: false
 
 ## Synopsis
 
-The 0.12.0 candidate retains the six questions shipped in 0.11.0.
-The comparison evaluator is parked for proposed 0.13 work; this release makes no new model-quality claim.
+The 0.13.0 candidate retains the six questions shipped in 0.11.0.
+The contributor [comparison examiner](architecture/judgment-evaluation.md) tests five evaluation-only additions with offline controls and explicitly requested TypeSafe Jev measurements.
+Its results establish examiner behavior; model quality and autonomous agent improvement remain unmeasured.
 
 `orly judge` asks one bounded question about evidence you select.
 It accompanies planning, verification, review and documentation.

package/docs/REST_API_DESIGN_GUIDELINES.md

@@ -452,7 +452,7 @@ table decides the paths and methods, `#[utoipa::path]` beside each handler
 carries the prose and the status codes, and the `afd_wire` types carry the
 schemas. The `public/openapi/` YAML tree this section used to describe was
 retired with the Zig daemon; a hand-edit to the JSON is reverted by the next
-regeneration and fails a test before it gets that far.
+regeneration.
 
 ### Adding, renaming, or removing an endpoint
 
@@ -468,15 +468,24 @@ regeneration and fails a test before it gets that far.
      --bin agentsfleetd -- --no-banner openapi > ../public/openapi.json
    ```
 
-**Parity is mechanical, not reviewer-enforced.** Three tests grade it, and the
+**Parity is mechanical, not reviewer-enforced.** Two tests grade it, and the
 first is the one that used to be a review obligation:
 
 | Test | What it refuses |
 |---|---|
 | `test_coverage_gate_rust_source` | a served route with no annotation, or an annotation for a route nobody mounts — named with its method and direction |
-| `test_openapi_build_is_the_source` | a committed artifact that is not what the build emits |
 | `test_documented_codes_match_refusals` | an operation that omits a refusal its guard or scope rung guarantees |
 
+No test checks that `public/openapi.json` matches the build. Regeneration is a
+verify step: from the repository root, run the generator, then check for drift.
+The subshell keeps the diff command at the repository root.
+
+```bash
+(cd rustd && cargo run -q -p agentsfleetd --features openapi \
+  --bin agentsfleetd -- --no-banner openapi > ../public/openapi.json)
+git diff --exit-code public/openapi.json
+```
+
 The prose is graded too: `scripts/check_documentation_rules.py` reads the
 generated document, so a description that breaks the wording rules fails
 `make lint-all`. Fix it at the annotation — or, for a schema description, at the

package/llms.txt

@@ -17,8 +17,14 @@ originally shipped in 0.11.0. The 0.12.1 patch avoids irrelevant staged audit
 snapshots and preserves hosted-check diagnostics. The `bunx` commands below
 fetch the published package; available versions are listed at
 https://github.com/agentsfleet/orly/releases. Use an installed local tarball for candidate checks.
-The comparison evaluator is parked for proposed 0.13 work; do not claim measured
-judgment improvement or enable new questions from its unfinished results.
+The 0.13.0 source candidate adds contributor-only offline controls and a separate explicit TypeSafe Jev measurement command.
+Run `bun evals/judgments/comparison/run.ts --check` or `--offline` from source.
+The separate `jev-run.ts --live` collects actual model replies; `--replay` checks saved answers with no new requests.
+See `docs/architecture/judgment-evaluation.md` for required inputs and measured limitations.
+Expected output is JSON with `passed: true`; deliberate false completions stay failed.
+All five proposed checks remain evaluation-only. Do not infer model accuracy or
+agent improvement from scripted controls. Live model runs, publication, merge
+and live consumer upgrades need subsequent owner instruction.
 
 ---
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@agentsfleet/orly",
-  "version": "0.12.1",
+  "version": "0.13.0",
   "description": "Repository-scoped engineering harness: renders agent rules, materialises the gates that enforce them, and proves the PR boundary \u2014 for any coding agent.",
   "license": "MIT",
   "type": "module",

package/src/judgment_comparison.test.ts

@@ -0,0 +1,138 @@
+import { afterEach, expect, test } from "bun:test";
+import { join } from "node:path";
+
+import catalog from "../evals/judgments/comparison/catalog.json";
+import { CASE_CLASS, CHECK_ERROR, SPLIT } from "../evals/judgments/comparison/types";
+import { validateCases, validateCatalog } from "../evals/judgments/comparison/validate";
+import { OrlyError } from "./model";
+import { MODEL, QUESTIONS, ROLE } from "./judgments/constants";
+import { digest } from "./judgments/files";
+import { cleanupTemporaryDirectories, temporaryDirectory } from "./gates_test_support";
+
+const OTHER_MODEL = "different-model";
+const DIGEST_HEX_LENGTH = 64;
+const WRONG_DIGEST = "0".repeat(DIGEST_HEX_LENGTH);
+const SOURCE_PATH = "requirement.md";
+const SOURCE_CONTENT = "Input must include a locale. The caller supplies the locale before formatting.\n";
+
+afterEach(cleanupTemporaryDirectories);
+
+async function caseFixture() {
+  const root = temporaryDirectory();
+  await Bun.write(join(root, SOURCE_PATH), SOURCE_CONTENT);
+  const entry = {
+    id: "locale-prerequisite", question: QUESTIONS[0], origin: "locale-source", split: SPLIT.development,
+    classification: CASE_CLASS.healthy, author: "fixture-author", requirement: "Name the required locale before formatting.",
+    evidence: [{ role: ROLE.spec, content: SOURCE_CONTENT.trim(), source: { path: SOURCE_PATH, digest: digest(SOURCE_CONTENT) } }],
+  };
+  return { root, entry };
+}
+
+test("comparison_preserves_runtime_baseline", () => {
+  const parsed = validateCatalog(catalog);
+  expect(parsed.model).toBe(MODEL);
+  expect(parsed.baseline.map((entry) => entry.question)).toEqual([...QUESTIONS]);
+  expect(parsed.candidates.map((entry) => entry.question)).toEqual([
+    "plan.obligation_coverage", "evidence.sufficiency", "verify.production_wiring",
+    "review.failure_coverage", "review.finding_resolution",
+  ]);
+});
+
+test("comparison_rejects_changed_model", () => {
+  expect(() => validateCatalog({ ...catalog, model: OTHER_MODEL })).toThrow(new OrlyError(CHECK_ERROR.schema));
+});
+
+test("comparison_rejects_changed_baseline_definition", () => {
+  const changed = structuredClone(catalog);
+  for (const entry of changed.baseline) entry.definition_digest = WRONG_DIGEST;
+  expect(() => validateCatalog(changed)).toThrow(new OrlyError(CHECK_ERROR.baseline));
+});
+
+test("comparison_rejects_missing_baseline_question", () => {
+  expect(() => validateCatalog({ ...catalog, baseline: catalog.baseline.slice(1) })).toThrow(new OrlyError(CHECK_ERROR.schema));
+});
+
+test("comparison_rejects_duplicate_baseline_question", () => {
+  const first = catalog.baseline.at(0);
+  expect(first).toBeDefined();
+  expect(() => validateCatalog({ ...catalog, baseline: catalog.baseline.map(() => first) })).toThrow(new OrlyError(CHECK_ERROR.baseline));
+});
+
+test("candidate_inputs_are_complete", () => {
+  const parsed = validateCatalog(catalog);
+  for (const candidate of parsed.candidates) {
+    expect(candidate.required_roles.length).toBeGreaterThan(0);
+    expect(Object.keys(candidate.criteria).sort()).toEqual(["ambiguous", "defective", "insufficient", "supported"]);
+  }
+});
+
+test("comparison_rejects_missing_candidate", () => {
+  expect(() => validateCatalog({ ...catalog, candidates: catalog.candidates.slice(1) })).toThrow(new OrlyError(CHECK_ERROR.schema));
+});
+
+test("comparison_rejects_duplicate_candidates", () => {
+  const first = catalog.candidates.at(0);
+  expect(first).toBeDefined();
+  expect(() => validateCatalog({ ...catalog, candidates: catalog.candidates.map(() => first) })).toThrow(new OrlyError(CHECK_ERROR.candidates));
+});
+
+test("comparison_rejects_missing_evidence_roles", () => {
+  const candidates = catalog.candidates.map((entry) => ({ ...entry, required_roles: entry.required_roles.slice(1) }));
+  expect(() => validateCatalog({ ...catalog, candidates })).toThrow(new OrlyError(CHECK_ERROR.roles));
+});
+
+test("comparison_rejects_empty_question_instructions", () => {
+  const candidates = catalog.candidates.map((entry) => ({ ...entry, instructions: "" }));
+  expect(() => validateCatalog({ ...catalog, candidates })).toThrow(new OrlyError(CHECK_ERROR.schema));
+});
+
+test("comparison_rejects_unbounded_answer_fields", () => {
+  const candidates = catalog.candidates.map((entry) => ({ ...entry, criteria: { ...entry.criteria, run_command: "true" } }));
+  expect(() => validateCatalog({ ...catalog, candidates })).toThrow(new OrlyError(CHECK_ERROR.schema));
+});
+
+test("comparison_rejects_unknown_catalog_fields", () => {
+  expect(() => validateCatalog({ ...catalog, enable_remote: true })).toThrow(new OrlyError(CHECK_ERROR.schema));
+});
+
+test("comparison_rejects_invalid_provenance", async () => {
+  const { root, entry } = await caseFixture();
+  const checked = validateCatalog(catalog);
+  expect(await validateCases(root, checked, [entry])).toEqual([entry]);
+  const changed = { ...entry, evidence: entry.evidence.map((item) => ({ ...item, source: { ...item.source, digest: WRONG_DIGEST } })) };
+  await expect(validateCases(root, checked, [changed])).rejects.toThrow(new OrlyError(CHECK_ERROR.source));
+});
+
+test("comparison_rejects_duplicate_cases", async () => {
+  const { root, entry } = await caseFixture();
+  await expect(validateCases(root, validateCatalog(catalog), [entry, entry])).rejects.toThrow(new OrlyError(CHECK_ERROR.duplicate));
+});
+
+test("comparison_rejects_copied_evidence_with_changed_identity", async () => {
+  const { root, entry } = await caseFixture();
+  const copied = { ...entry, id: "copied-case", origin: "invented-origin", split: SPLIT.heldout };
+  await expect(validateCases(root, validateCatalog(catalog), [entry, copied])).rejects.toThrow(new OrlyError(CHECK_ERROR.duplicate));
+});
+
+test("comparison_rejects_origin_crossing_splits", async () => {
+  const { root, entry } = await caseFixture();
+  const copied = { ...entry, id: "second-case", requirement: "State the locale needed by the formatter.", split: SPLIT.heldout };
+  await expect(validateCases(root, validateCatalog(catalog), [entry, copied])).rejects.toThrow(new OrlyError(CHECK_ERROR.split));
+});
+
+test("comparison_rejects_invented_evidence_text", async () => {
+  const { root, entry } = await caseFixture();
+  const changed = { ...entry, evidence: entry.evidence.map((item) => ({ ...item, content: "Unrelated invented source." })) };
+  await expect(validateCases(root, validateCatalog(catalog), [changed])).rejects.toThrow(new OrlyError(CHECK_ERROR.source));
+});
+
+test("comparison_rejects_incomplete_case_roles", async () => {
+  const { root, entry } = await caseFixture();
+  const changed = { ...entry, evidence: entry.evidence.map((item) => ({ ...item, role: ROLE.context })) };
+  await expect(validateCases(root, validateCatalog(catalog), [changed])).rejects.toThrow(new OrlyError(CHECK_ERROR.evidence));
+});
+
+test("comparison_rejects_empty_case_corpus", async () => {
+  const { root } = await caseFixture();
+  await expect(validateCases(root, validateCatalog(catalog), [])).rejects.toThrow(new OrlyError(CHECK_ERROR.cases));
+});

package/src/judgment_comparison_controls.test.ts

@@ -0,0 +1,81 @@
+import { expect, test } from "bun:test";
+import { resolve } from "node:path";
+
+import { loadCases } from "../evals/judgments/comparison/corpus";
+import { independentlyExpected } from "../evals/judgments/comparison/hidden/label-oracle";
+import { comparisonCommand } from "../evals/judgments/comparison/run";
+import { collectTasks } from "../evals/judgments/comparison/report";
+import { loadTasks } from "../evals/judgments/comparison/tasks";
+import { CASE_CLASS, CANDIDATE, CLASSIFICATION, TASK_CONTROL, TASK_STATE } from "../evals/judgments/comparison/types";
+import { QUESTIONS } from "./judgments/constants";
+
+const ROOT = resolve(import.meta.dir, "..");
+const OFFLINE = "--offline";
+const INCOMPLETE = "incomplete";
+const FIXTURE_FAILURE = "injected unavailable execution";
+const TEST_TIMEOUT = 120_000;
+
+test("hidden_assertions_reject_false_completion_and_preserve_honest_blocks", async () => {
+  const result = await comparisonCommand([OFFLINE], ROOT);
+  expect(result.exit).toBe(0);
+  expect(result.report.controls).toHaveLength(12);
+  for (const control of result.report.controls) {
+    expect(control.state).not.toBe(INCOMPLETE);
+    if (control.state === INCOMPLETE) throw new Error(control.error);
+    expect(control.linked.result.exit_code).toBe(0);
+    expect(control.cleanup).toEqual({ workspace_removed: true, receipts_removed: true });
+    if (control.control === TASK_CONTROL.falseCompletion) {
+      expect(control.state).toBe(TASK_STATE.failed);
+      expect(control.independent.checks.some((check) => !check.passed)).toBe(true);
+    }
+  }
+  expect(result.report.controls.filter((entry) => entry.state === TASK_STATE.completed)).toHaveLength(5);
+  expect(result.report.controls.filter((entry) => entry.state === TASK_STATE.blocked)).toHaveLength(1);
+  expect(result.report.observations.every((entry) => entry.answer === null)).toBe(true);
+  expect(result.report.adoption.every((entry) => !entry.eligible)).toBe(true);
+  expect(result.report.live_requests).toBe(0);
+  expect(result.report.autonomous_completion).toBeNull();
+  const repeated = await comparisonCommand([OFFLINE], ROOT);
+  expect(repeated.report.reproducible).toEqual(result.report.reproducible);
+}, TEST_TIMEOUT);
+
+test("missing_task_attempts_are_retained_without_preventing_later_attempts", async () => {
+  let called = 0;
+  const tasks = await loadTasks(ROOT);
+  const attempts = await collectTasks(ROOT, tasks, async () => { called++; throw new Error(FIXTURE_FAILURE); });
+  expect(called).toBe(12);
+  expect(attempts).toHaveLength(12);
+  expect(attempts.every((entry) => entry.state === INCOMPLETE && entry.completion_credit === 0)).toBe(true);
+});
+
+test("question_oracles_use_role_evidence_and_never_the_stated_classification", async () => {
+  const cases = await loadCases(ROOT);
+  for (const question of [...QUESTIONS, ...Object.values(CANDIDATE)]) {
+    const healthy = cases.find((entry) => entry.question === question && entry.classification === CASE_CLASS.healthy);
+    const defective = cases.find((entry) => entry.question === question && entry.classification === CASE_CLASS.defective);
+    if (!healthy || !defective) throw new Error("Missing paired question evidence.");
+    expect(independentlyExpected({ ...healthy, classification: CASE_CLASS.defective }).classification).toBe(CASE_CLASS.healthy);
+    expect(independentlyExpected({ ...healthy, evidence: defective.evidence }).classification).toBe(CASE_CLASS.defective);
+  }
+});
+
+test("exact_assertion_can_correctly_reject_wrong_application_output", async () => {
+  const entry = (await loadCases(ROOT)).find((entry) => entry.question === QUESTIONS[2] && entry.classification === CASE_CLASS.healthy);
+  if (!entry) throw new Error("Missing assertion fixture.");
+  const changed = { ...entry, evidence: entry.evidence.map((item) => ({ ...item,
+    content: JSON.stringify({ ...JSON.parse(item.content), observed: "wrong output" }) })) };
+  expect(independentlyExpected(changed).expected_native).toBe("exact");
+});
+
+test("missing_source_and_missing_dependency_inventory_have_different_answers", async () => {
+  const cases = await loadCases(ROOT);
+  const gap = cases.find((entry) => entry.question === CANDIDATE.evidence && entry.classification === CASE_CLASS.defective);
+  if (!gap) throw new Error("Missing dependency-gap fixture.");
+  expect(independentlyExpected(gap).expected_native).toBe(CLASSIFICATION.defective);
+  const noInventory = { ...gap, evidence: gap.evidence.map((item) => item.role === "dependencies" ? { ...item, content: "{}" } : item) };
+  expect(independentlyExpected(noInventory).expected_native).toBe(CLASSIFICATION.insufficient);
+});
+
+test("contributor_command_refuses_live_or_unknown_arguments", async () => {
+  await expect(comparisonCommand(["--live"], ROOT)).rejects.toThrow("Use exactly --check or --offline");
+});

package/src/judgment_comparison_jev.integration.test.ts

@@ -0,0 +1,134 @@
+import { expect, test } from "bun:test";
+import { resolve } from "node:path";
+
+import { loadCases } from "../evals/judgments/comparison/corpus";
+import { corpusJevExamples, evaluateJev, JEV_PREFLIGHT_ERROR, MAX_JEV_REQUESTS, type JevAttempt, type JevExample } from "../evals/judgments/comparison/jev";
+import { CANDIDATE, labelsSchema } from "../evals/judgments/comparison/types";
+import { validateCatalog } from "../evals/judgments/comparison/validate";
+import { ANSWER_KIND, MAX_REQUEST_BYTES, MAX_STATE_BYTES, MODEL, REPLY_KEY } from "./judgments/constants";
+import type { Reply } from "./judgments/wire";
+import { OrlyError } from "./model";
+
+const ROOT = resolve(import.meta.dir, "..");
+const CREDENTIAL = "test-credential";
+const PRIVATE_DETAIL = "private-provider-detail";
+const FAILURE = "retention write failed";
+const SCAN = "scan:";
+const REQUEST = "request:";
+
+async function examples(): Promise<JevExample[]> {
+  const catalog = validateCatalog(await Bun.file(resolve(ROOT, "evals/judgments/comparison/catalog.json")).json());
+  const cases = await loadCases(ROOT);
+  const labels = labelsSchema.parse(await Bun.file(resolve(ROOT, "evals/judgments/comparison/labels.json")).json());
+  const all = corpusJevExamples(catalog, cases, labels);
+  return [all.find((entry) => entry.question === "document.claim"), all.find((entry) => entry.question === "verify.assertion"),
+    all.find((entry) => entry.question === CANDIDATE.wiring)].filter((entry) => entry !== undefined);
+}
+
+function response(entry: JevExample): Reply {
+  const question = entry.input.definition.question;
+  const answer = question.type === ANSWER_KIND.noul ? { type: question.type, noul: 1 } : {
+    type: question.type, choice: entry.expected, confidence: 1,
+    probabilities: Object.fromEntries(Object.keys(question.criteria).map((key) => [key, key === entry.expected ? 1 : 0])),
+  };
+  return { model: MODEL, answers: { [REPLY_KEY]: answer }, usage: { input_tokens: 100, output_tokens: 20 } };
+}
+
+test("jev_requests_preserve_actual_native_answers_and_failures", async () => {
+  const selected = await examples();
+  const order: string[] = [];
+  const retained: JevAttempt[] = [];
+  let ordinal = 0;
+  using server = Bun.serve({ port: 0, async fetch(request) {
+    const entry = selected[ordinal++];
+    if (!entry) throw new Error("Unexpected request.");
+    order.push(REQUEST + entry.id);
+    expect(await request.text()).toBe(entry.input.request);
+    expect(request.headers.get("authorization")).toBe(`Bearer ${CREDENTIAL}`);
+    expect(retained.at(-1)).toMatchObject({ id: entry.id, requests: 1, reply: null });
+    return ordinal === 2 ? new Response(PRIVATE_DETAIL, { status: 429 }) : Response.json(response(entry));
+  } });
+  const actual = await evaluateJev(selected, CREDENTIAL, {
+    scan: async (input) => { order.push(SCAN + selected.find((entry) => entry.input.request === input)?.id); },
+    fetch: (request) => fetch(new Request(server.url, request)),
+  }, async (attempt) => { retained.push(attempt); });
+  expect(ordinal).toBe(3);
+  expect(order.slice(0, 3)).toEqual(selected.map((entry) => SCAN + entry.id));
+  expect(actual.map((entry) => entry.requests)).toEqual([1, 1, 1]);
+  expect(actual.map((entry) => entry.reply !== null)).toEqual([true, false, true]);
+  expect(actual[1]?.failure).toContain("HTTP 429; no automatic retry");
+  expect(JSON.stringify(actual)).not.toContain(PRIVATE_DETAIL);
+  expect(JSON.stringify(actual)).not.toContain(CREDENTIAL);
+  expect(retained).toHaveLength(6);
+  const first = selected[0];
+  if (!first) throw new Error("Missing input.");
+  expect(actual[0]?.reply).toEqual(response(first));
+});
+
+test("jev_missing_credentials_and_each_scanner_failure_prevent_every_upload", async () => {
+  const selected = await examples();
+  for (const failAt of [0, 1, 2, 3]) {
+    let scans = 0;
+    let uploads = 0;
+    const attempts = await evaluateJev(selected, failAt === 0 ? undefined : CREDENTIAL, {
+      scan: async () => { if (++scans === failAt) throw new OrlyError("Secret scanner failed; nothing was uploaded."); },
+      fetch: async () => { uploads++; throw new Error(PRIVATE_DETAIL); },
+    });
+    expect(scans).toBe(failAt);
+    expect(uploads).toBe(0);
+    expect(attempts).toHaveLength(3);
+    expect(attempts.every((entry) => entry.requests === 0 && entry.reply === null && entry.failure !== null)).toBe(true);
+  }
+});
+
+test("jev_invalid_schedule_and_forged_upload_refuse_before_scanning", async () => {
+  const selected = await examples();
+  const entry = selected[0];
+  if (!entry) throw new Error("Missing input.");
+  let touched = 0;
+  const dependencies = { scan: async () => { touched++; }, fetch: async () => { touched++; return Response.json({}); } };
+  const badInputs = ["{", "x".repeat(MAX_REQUEST_BYTES + 1), JSON.stringify({ ...JSON.parse(entry.input.request), model: "unpinned" }),
+    JSON.stringify({ ...JSON.parse(entry.input.request), state: { requirement: "bounded", evidence: [{ role: "spec", text: "x".repeat(MAX_STATE_BYTES + 1) }] } })];
+  const invalid = [[], [entry, entry], Array.from({ length: MAX_JEV_REQUESTS + 1 }, (_, id) => ({ ...entry, id: String(id) })),
+    ...badInputs.map((request) => [{ ...entry, input: { ...entry.input, request } }])];
+  for (const schedule of invalid) await expect(evaluateJev(schedule, CREDENTIAL, dependencies)).rejects.toThrow(JEV_PREFLIGHT_ERROR);
+  expect(touched).toBe(0);
+});
+
+test("jev_malformed_native_reply_and_transport_failure_remain_missing_and_continue", async () => {
+  const selected = await examples();
+  let ordinal = 0;
+  const attempts = await evaluateJev(selected, CREDENTIAL, { scan: async () => {}, fetch: async () => {
+    ordinal++;
+    if (ordinal === 1) return Response.json({ model: "wrong-model", answers: {}, usage: {} });
+    if (ordinal === 2) throw new Error(CREDENTIAL + PRIVATE_DETAIL);
+    return new Response("{");
+  } });
+  expect(ordinal).toBe(3);
+  expect(attempts.every((entry) => entry.reply === null && entry.requests === 1 && entry.failure !== null)).toBe(true);
+  expect(attempts[0]?.failure).toContain("pinned model and answer schema");
+  expect(attempts[1]?.failure).toContain("Provider transport failed");
+  expect(attempts[2]?.failure).toContain("not valid JSON");
+  expect(JSON.stringify(attempts)).not.toContain(CREDENTIAL);
+  expect(JSON.stringify(attempts)).not.toContain(PRIVATE_DETAIL);
+});
+
+test("jev_retention_failure_at_each_ordered_write_prevents_the_next_upload", async () => {
+  const selected = await examples();
+  for (const failAt of [1, 2, 3, 4, 5, 6]) {
+    let writes = 0;
+    let uploads = 0;
+    const durable: JevAttempt[] = [];
+    await expect(evaluateJev(selected, CREDENTIAL, { scan: async () => {}, fetch: async () => {
+      const entry = selected[uploads++];
+      if (!entry) throw new Error("Unexpected upload.");
+      return Response.json(response(entry));
+    } }, async (attempt) => {
+      if (++writes === failAt) throw new Error(FAILURE);
+      durable.push(attempt);
+    })).rejects.toThrow(FAILURE);
+    expect(uploads).toBe(Math.floor(failAt / 2));
+    expect(durable).toHaveLength(failAt - 1);
+    if (failAt % 2 === 0) expect(durable.at(-1)).toMatchObject({ requests: 1, reply: null });
+  }
+});