psyclaw 0.27.23 → 0.28.1

package/vendor/ars/academic-pipeline/references/ai_research_failure_modes.md

@@ -0,0 +1,185 @@
+# AI Research Failure Mode Checklist
+
+**Status**: v3.2
+**Parent skill**: `academic-pipeline`
+**Used at**: Stage 2.5 INTEGRITY (blocking), Stage 4.5 FINAL INTEGRITY (blocking), Stage 6 PROCESS SUMMARY (reporting only)
+**Source**: Lu et al. (2026). Towards end-to-end automation of AI research. *Nature* 651, 914-919. doi:10.1038/s41586-026-10265-5 — Limitations section, Figure 2 (examples of failures in The AI Scientist's own accepted paper), Supplementary Information A.2.9 (debugging traces).
+
+---
+
+## Why this checklist exists
+
+Lu et al. built the first autonomous AI research system to pass blind peer review (ICLR 2025 workshop). Their Limitations section enumerates the specific failure modes they observed — and most of them apply equally to human-in-the-loop AI research workflows like ARS.
+
+These failures are dangerous because **they look like competent work**. A paper containing a hallucinated experimental result reads the same as a paper containing a real one. A shortcut-relying result reads the same as a genuine generalization. A methodology section describing experiments that were never actually run reads the same as a faithful account. The existing integrity verification catches citation hallucinations but is weak on the other failure modes.
+
+The checklist exists to make these failures legible: **at Stage 2.5 and Stage 4.5, the integrity reviewer must explicitly rule out each of the 7 modes, or flag which are suspected and block the pipeline until the user acknowledges.**
+
+This also extends the existing 5-type citation hallucination taxonomy (in `academic-paper-reviewer` references) into a broader 7-type AI research hallucination taxonomy. Citation hallucinations become mode 2 below.
+
+---
+
+## The 7 failure modes
+
+### Mode 1: Implementation bug passing AI self-review
+
+**What it is**: The analysis or experiment code has a bug (off-by-one, wrong variable, silent division-by-zero, type coercion, wrong flag) that produces numerically plausible but scientifically wrong results. The AI runs the code, looks at the output, sees nothing "obviously" wrong, and incorporates the result into the paper.
+
+**Lu 2026 example**: Supplementary A.2.9 traces show The AI Scientist repeatedly accepting experimental runs that had silent crashes or numerical instabilities because the top-level metric "looked reasonable". Figure 2 shows an ICLR reviewer catching one such issue in the accepted paper — the paper's main analysis depended on a setup that the code did not actually implement.
+
+**Detection questions at Stage 2.5**:
+- For every numerical result in the draft: does the user have a saved log, notebook, or script run that produced this number? If yes, was the exit code 0 and were there zero warnings? If no log is saved, flag.
+- Are any effect sizes suspiciously round (exactly 0.5, exactly 2x baseline, exactly zero variance across runs)? Suspiciously round numbers are a common signal of a constant leaking through a broken pipeline.
+- Do error bars / confidence intervals actually vary across conditions, or are they suspiciously identical?
+
+**Who catches it**: `methodology_reviewer_agent` at Stage 3 is downstream; the integrity gate at Stage 2.5 should ask the user directly.
+
+---
+
+### Mode 2: Hallucinated citation
+
+**What it is**: A reference that does not exist, is miscited (wrong year, wrong journal, wrong authors), or is attributed a finding it does not contain. This is the mode ARS already covers most thoroughly via the 5-type citation hallucination taxonomy in `academic-paper-reviewer/references/`. It is included here for completeness of the 7-mode taxonomy.
+
+**Lu 2026 example**: The AI Scientist pipeline includes a Semantic Scholar citation check to suppress this mode, acknowledging it as a primary failure class. PaperOrchestra (Song et al., 2026) extended this with a two-phase pipeline: web search discovery + sequential Semantic Scholar API verification (Levenshtein >= 0.70 title matching).
+
+**Detection (v3.3 update)**: Covered by the existing integrity verification, now strengthened with Semantic Scholar API batch verification (Phase A0 in `integrity_verification_agent`). See `deep-research/references/semantic_scholar_api_protocol.md` for the API protocol. The S2 API provides structured, machine-readable verification that catches fabricated DOIs (DOI_MISMATCH pattern) missed by manual WebSearch.
+
+**Who catches it**: `source_verification_agent` (Tier 0 S2 API + Tier 1 DOI + Tier 2 WebSearch) + `integrity_verification_agent` (Phase A0 + A1).
+
+---
+
+### Mode 3: Hallucinated experimental result
+
+**What it is**: A result that does not correspond to any actual experiment run. The AI writes "we observed a 12% improvement" when no run produced a 12% improvement — either it averaged differently than the paper claims, or it reported a number from a crashed run, or it invented the number to match the narrative.
+
+**Lu 2026 example**: Lu et al. specifically flag "hallucinated experimental results" as a Limitation, noting that automated reviewing struggles to detect them because the reviewer has no access to the underlying runs.
+
+**Detection questions at Stage 2.5**:
+- For every claim of the form "X% improvement" or "Y% reduction" or "outperforms baseline by Z": does the user have the raw numbers the paper's number was computed from?
+- Does the table in the draft match a saved CSV / tensor log / wandb run the user can point to?
+- Does the paper's "we ran N seeds" claim match the number of actual run directories the user has?
+
+**Who catches it**: integrity gate at 2.5. This is harder than citation checking because there's no external database to verify against — the verification is against the user's own experiment logs.
+
+---
+
+### Mode 4: Shortcut reliance
+
+**What it is**: The reported result is real but the model achieved it by exploiting a spurious feature rather than learning the intended generalization. A colour-biased MNIST model that gets 99% accuracy by reading the background colour, not the digit shape, is the canonical case.
+
+**Lu 2026 example**: Figure 2b shows this exact case — The AI Scientist proposed a method, tested it on colour-biased MNIST, got high accuracy, and wrote a paper claiming the method "solved" the task. A reviewer caught that the method was exploiting the colour shortcut, not learning the digit shape. The paper was revised.
+
+**Detection questions at Stage 2.5**:
+- Is there any controlled ablation that rules out the most obvious shortcut feature? If the paper tests on dataset D, has the author run on a D-variant where the shortcut feature is removed?
+- Does the paper's "ablation studies" section actually ablate the claimed mechanism, or does it ablate incidental hyperparameters?
+- Is the baseline strong enough that beating it requires the proposed mechanism, not just more compute?
+
+**Who catches it**: `devils_advocate_reviewer_agent` at Stage 3 is the natural home for this check, but it must be flagged at 2.5 so the user knows to prepare the ablation before Stage 3 arrives. Flag-only at 2.5, not block-only.
+
+---
+
+### Mode 5: Implementation bug reframed as novel insight
+
+**What it is**: The pipeline produces an unexpected result that is actually caused by a bug, but the narrative-writing stage reframes the unexpected behaviour as a novel finding. The paper claims "we discovered that X behaves unexpectedly under condition Y" when in reality X behaves the expected way and condition Y is being mis-implemented.
+
+**Lu 2026 example**: Lu et al. identify this as a compound failure mode — it requires Mode 1 (the bug) plus a writing-stage error in which the AI's narrative generator accepts the bug's output as real and builds a story around it. The paper reads *more* interesting than a bug-free version would have.
+
+**Detection questions at Stage 2.5**:
+- Does the draft contain any phrase like "surprisingly," "unexpectedly," "counterintuitively," or "contrary to our hypothesis"? For each such claim, can the user point to a literature reference that would have predicted the opposite? If no literature cites the opposite, the "surprise" may not be a surprise — it may be a bug.
+- Did the surprising result appear on the first run, or only after many debugging iterations? First-run surprises are high-risk for this mode.
+- Has the user attempted to reproduce the surprising result from scratch in a fresh environment?
+
+**Who catches it**: integrity gate at 2.5. This is the most ARS-specific mode — it is the interaction of a bug (Mode 1) with narrative seduction.
+
+---
+
+### Mode 6: Methodology fabrication
+
+**What it is**: The Methods section describes experiments, hyperparameters, datasets, or procedures that were not actually what the pipeline ran. The AI writes a plausible-sounding Methods section based on what a reasonable version of the experiment *would* look like, drifting from what actually happened.
+
+**Lu 2026 example**: Lu et al. note that the writing stage of The AI Scientist sometimes produced Methods text that was disconnected from the actual hyperparameter log. They added a cross-check against the experiment run config to mitigate this.
+
+**Detection questions at Stage 2.5**:
+- Does every number in the Methods section (learning rate, batch size, epochs, dataset size, train/val split) appear in the user's actual run config / log?
+- Does the Methods section describe any preprocessing step the user cannot point to in their code?
+- Does the Methods section use the past tense where the actual pipeline didn't run?
+
+**Who catches it**: integrity gate at 2.5. This requires the user to provide the actual run config as an integrity input, not just the paper text.
+
+---
+
+### Mode 7: Frame-lock at early pipeline stage
+
+**What it is**: A wrong commitment made in early stages (research question framing, methodology choice, hyperparameter direction) that subsequent stages cannot back out of because they are structurally downstream of the commitment. The paper ends up well-executed but is answering the wrong question or using a fundamentally unsuitable method.
+
+**Lu 2026 example**: Figure 3a traces The AI Scientist's agentic tree search and shows that most failed papers failed at Stage 2 (hyperparameter tuning) — the agent committed to a direction early and could not recover. This is the same frame-lock pattern ARS's anti-sycophancy protocol targets for dialogue, but here it applies to pipeline decisions.
+
+**Detection questions at Stage 2.5**:
+- If the user could go back to Stage 1 knowing what they know now, would they change the research question or methodology?
+- Does the paper's Discussion section contain any phrase like "in hindsight" or "we realized later"? These are frame-lock tells.
+- Is the paper's contribution better explained by the chosen framing, or despite it?
+
+**Who catches it**: integrity gate at 2.5. If flagged, user is offered the option to return to Stage 1 or Stage 2 rather than proceeding to Stage 3.
+
+---
+
+## How the checklist runs at each stage
+
+### At Stage 2.5 INTEGRITY (first integrity gate)
+
+Run all 7 modes. For each mode, produce one of three outcomes:
+
+- **CLEAR**: integrity reviewer has evidence that the mode does not apply. Record the evidence briefly.
+- **SUSPECTED**: one or more detection questions returned a concerning answer. Must be surfaced to the user.
+- **INSUFFICIENT EVIDENCE**: integrity reviewer cannot rule the mode in or out without user input (e.g., needs experiment logs the user hasn't provided).
+
+**Block condition**: pipeline blocks if **any** mode is SUSPECTED, or if Modes 1, 3, 5, or 6 are INSUFFICIENT EVIDENCE (these four require user-provided logs to rule out and should not be silently skipped). Modes 2, 4, 7 INSUFFICIENT EVIDENCE can proceed with a warning and will be re-checked at 4.5.
+
+**User acknowledgement options at block**:
+- Confirm the flag — return to Stage 2 WRITE (or earlier) to fix
+- Override with reasoning — user explicitly states why the flag is a false positive, reasoning is recorded in the process log for Stage 6
+- Revise the specific passage and re-run the check
+
+### At Stage 4.5 FINAL INTEGRITY
+
+Re-run all 7 modes. Additional rule: any mode that was SUSPECTED at 2.5 must be resolved by 4.5 (CLEAR or user-Overridden-with-reasoning). If the same mode is still SUSPECTED at 4.5, the pipeline re-blocks and refuses to proceed to Finalize until the issue is addressed — no amount of revision loops can skip this.
+
+### At Stage 6 PROCESS SUMMARY (AI Self-Reflection Report)
+
+Report only, no blocking. The Self-Reflection Report includes a "Failure Mode Audit Log" section listing, for each of the 7 modes:
+- Final status at 4.5 (CLEAR / OVERRIDDEN)
+- History: was it ever SUSPECTED during the pipeline? At which stage? How was it resolved?
+- If OVERRIDDEN: the user's reasoning
+
+This makes the failure-mode history part of the permanent process record, giving future readers (and the user themselves) visibility into what the AI-human collaboration had to defend against.
+
+---
+
+## Relationship to existing ARS checks
+
+| Existing check | Covers which modes |
+|---|---|
+| Citation hallucination taxonomy (5-type) | Mode 2 (fully) |
+| `source_verification_agent` | Mode 2 (cross-check) |
+| Existing Stage 2.5 integrity review | Mode 2, partial Mode 6 |
+| `devils_advocate_reviewer_agent` (Stage 3) | Mode 4, partial Mode 7 |
+| Anti-sycophancy protocol (v3.0) | Dialogue-level frame-lock, not pipeline-level Mode 7 |
+
+Gap coverage provided by this checklist: **Modes 1, 3, 5, 6, and the pipeline-level aspect of Mode 7**. These are the modes that were not previously systematically checked.
+
+---
+
+## Open questions (for v3.3)
+
+- **False positive rate**: Modes 1, 5, and 6 require the user to supply experiment logs. If the user is writing a purely theoretical paper or a qualitative study, many of these detection questions don't apply. The checklist needs a paper-type pre-filter that turns off inapplicable modes based on the paper type detected by `field_analyst_agent`. v3.2 ships with all modes always-on; v3.3 should add the pre-filter.
+- **Override auditing**: if a user overrides a flag, is the reasoning ever reviewed? In v3.2 it goes into the Stage 6 record only. A stronger version would flag overrides for peer review during Stage 3 so that a reviewer can push back on the user's reasoning.
+
+---
+
+## References
+
+- Lu, C. et al. (2026). Towards end-to-end automation of AI research. *Nature* 651, 914-919. [doi:10.1038/s41586-026-10265-5](https://doi.org/10.1038/s41586-026-10265-5) — Limitations section, Figure 2, Supplementary Information A.2.9.
+- ARS `academic-paper-reviewer/references/` — existing 5-type citation hallucination taxonomy (Mode 2).
+- ARS `academic-pipeline/references/claim_verification_protocol.md` — existing integrity verification that this checklist extends.
+- ARS `academic-pipeline/references/integrity_review_protocol.md` — existing integrity review protocol that Stage 2.5 follows.
+- ARS `ROADMAP_v3.2.md` — v3.2 integration plan, item 2.

package/vendor/ars/academic-pipeline/references/changelog.md

@@ -0,0 +1,14 @@
+# Changelog
+
+| Version | Date | Changes |
+|---------|------|---------|
+| 2.8 | 2026-04-21 | **Collaboration Depth Observer (v3.3.0)**: New `collaboration_depth_agent` (Agent Team grows from 3 to 4). Reads dialogue logs at every FULL/SLIM checkpoint and at pipeline completion; scores user-AI collaboration on 4 dimensions (Delegation Intensity / Cognitive Vigilance / Cognitive Reallocation / Zone Classification) per `shared/collaboration_depth_rubric.md` (rubric v1.0, canonical, CC-BY-NC 4.0). **Advisory only — never blocks.** MANDATORY checkpoints (2.5 / 4.5) do NOT invoke the observer (integrity gates preserved). Cross-model divergence flagged (not silently averaged) when `ARS_CROSS_MODEL` set. Rubric credits Wang, S., & Zhang, H. (2026). *IJETHE* 23:11. DOI 10.1186/s41239-026-00585-x. New lint: `scripts/check_collaboration_depth_rubric.py`. State tracker gains `dialogue_log_ref` per stage + `collaboration_depth_history[]` append-only. Reinforcement row added for checkpoint transitions. |
+| 2.7 | 2026-03-27 | **Style Profile in Material Passport**: Pipeline orchestrator now carries optional Style Profile (Schema 10 in `shared/handoff_schemas.md`) through all stages. Produced by academic-paper intake Step 10 when user provides past writing samples. Consumed by draft_writer (Stage 2) and report_compiler (Stage 1) as soft writing voice guide. Does not affect integrity verification or review stages. Coordinates with deep-research v2.4 and academic-paper v2.5 |
+| 2.6 | 2026-03-08 | **Handoff Data Schema**: Enhanced `shared/handoff_schemas.md` with 9 comprehensive schemas (RQ Brief, Bibliography, Synthesis, Paper Draft, Integrity Report, Review Report, Revision Roadmap, Response to Reviewers, Material Passport) with full field definitions, type constraints, and validation rules; orchestrator validates output against schemas before each transition. **Adaptive Checkpoint System**: Replaced static checkpoint template with 3-tier system (FULL/SLIM/MANDATORY) based on stage criticality and user engagement; FULL checkpoints include decision dashboard with metrics; SLIM auto-continues for experienced users; MANDATORY cannot be bypassed at integrity/review/finalization boundaries; awareness guard after 4+ auto-continues. **Mode Advisor**: New `references/mode_advisor.md` with unified cross-skill decision tree, common misconceptions table, user archetype recommendations, decision flowchart, and anti-patterns guide. **Team Collaboration Protocol**: New `references/team_collaboration_protocol.md` with 5 role definitions, per-transition handoff procedures, git branching/tagging strategy, conflict resolution matrix, and communication templates; state tracker extended with `assigned_to`, `approval_gate`, `team_notes` per stage and `schema_validation_log`. **Phase E Claim Verification**: New `references/claim_verification_protocol.md` with E1 claim extraction, E2 source tracing, E3 cross-referencing; verdict taxonomy (VERIFIED / MINOR_DISTORTION / MAJOR_DISTORTION / UNVERIFIABLE / UNVERIFIABLE_ACCESS); severity mapping (MAJOR_DISTORTION -> SERIOUS, UNVERIFIABLE -> SERIOUS, MINOR_DISTORTION -> MINOR, UNVERIFIABLE_ACCESS -> MEDIUM); integrated into integrity_verification_agent Mode 1 (30% spot-check) and Mode 2 (100%); pass/fail criteria updated to include Phase E verdicts. **Mid-Entry Material Passport Check**: Pipeline orchestrator now validates Material Passport on mid-entry; decision tree checks verification_status, freshness (< 24 hours), and content modification (version_label comparison); offers skip/spot-check/full re-verify options for Stage 2.5 when passport is valid; passport freshness validation rules added to `shared/handoff_schemas.md` |
+| 2.5 | 2026-03-08 | External Review Protocol: structured intake of real journal reviewer feedback (text/PDF/DOCX); 4-step workflow (parse -> strategic coaching -> revise + Response to Reviewers -> completeness check); differentiated behavior from internal simulated review (no default "accept all", risk assessment per comment, user confirmation of parsed items); explicit capability boundaries (AI verification ≠ reviewer satisfaction) |
+| 2.4 | 2026-03-08 | Stage 6 PROCESS SUMMARY: post-pipeline paper creation process record; asks user preferred language (zh/en/both); generates structured MD summarizing full human-AI collaboration history with user quotes, key decisions, iteration details, and lessons learned; mandatory final chapter: **Collaboration Quality Evaluation** (6 dimensions scored 1-100, bar chart visualization, What Worked Well / Missed Opportunities / Recommendations / Human vs AI Value-Add / Claude's Self-Reflection); compiles to PDF via LaTeX + tectonic; outputs `paper_creation_process_zh.pdf` + `paper_creation_process_en.pdf` |
+| 2.3 | 2026-03-08 | Stage 5 FINALIZE: mandatory formatting style prompt (APA 7.0 / Chicago / IEEE); PDF must compile from LaTeX via tectonic (no HTML-to-PDF); APA 7.0 uses `apa7` document class (`man` mode) with XeCJK for bilingual support; font stack: Times New Roman + Source Han Serif TC VF + Courier New |
+| 2.2 | 2026-03-05 | Checkpoint confirmation semantics (6 user commands with precise actions); mode switching rules (safe/dangerous/prohibited matrix); skill failure fallback matrix (per-stage degradation strategies); state ownership protocol (single source of truth with write access control); material version control (versioned artifacts with audit trail); cross-skill reference to `shared/handoff_schemas.md` |
+| 2.1 | 2026-03 | Added plagiarism detection protocol (Phase D); enhanced integrity_verification_agent with originality verification (D1 WebSearch, D2 self-plagiarism); updated both verification modes |
+| 2.0 | 2026-02 | Added Stage 2.5/4.5 integrity checks, two-stage review, mandatory checkpoints, Devil's Advocate, reproducibility guarantees, integrity_verification_agent |
+| 1.0 | 2026-02 | Initial version: 5+1 stage pipeline |

package/vendor/ars/academic-pipeline/references/claim_audit_calibration_protocol.md

@@ -0,0 +1,175 @@
+# Claim-Faithfulness Audit Calibration Protocol
+
+**Status**: v3.8
+**Parent agent**: `claim_ref_alignment_audit_agent`
+**Mode name**: `calibration`
+**Purpose**: Measure the audit agent's own false-negative rate (FNR) and false-positive rate (FPR) on alignment judgments (SUPPORTED / UNSUPPORTED / AMBIGUOUS / RETRIEVAL_FAILED) AND negative-constraint judgments (VIOLATED / NOT_VIOLATED) against a synthetic gold set, then gate CI on FNR < 0.15 AND FPR < 0.10.
+
+This protocol is modeled on `academic-paper-reviewer/references/calibration_mode_protocol.md`. The reviewer mode measures editorial decision accuracy on whole papers; this mode measures claim-to-source alignment accuracy on per-citation tuples. Both share the same FNR / FPR / threshold-gate vocabulary; the difference is unit of analysis.
+
+---
+
+## Why this mode exists
+
+The audit agent emits HIGH-WARN gate-refuse annotations on UNSUPPORTED rows and VIOLATED constraint matches. Those signals are only useful if the underlying judge is reliably distinguishing supported from unsupported claims. Without measurement, a judge that systematically over-flags AMBIGUOUS cases as UNSUPPORTED looks identical at the row level to a judge that nails the discrimination — but the former produces a flood of false HIGH-WARN refusals that erodes operator trust and eventually gets the agent ignored.
+
+Calibration mode closes the measurability gap. It does NOT try to make the judge perfect; it makes the judge's error profile legible BEFORE the agent ships to a new operator's pipeline.
+
+---
+
+## Inputs
+
+1. **Gold set**: 25 tuples (17 alignment incl. 5 #213 partial-support fixtures + 8 constraint per the canonical shipped fixture at `scripts/fixtures/claim_audit_calibration/gold_set.json`). Per-tuple shape and the four ingestion-gate rules (alignment-judgment coverage, ≥3 NOT_VIOLATED floor, required constraint fields, tuple_kind validity) are spelled out in **Gold tuple schema** below + spec §7.7 rules (a)-(d). The shipped fixture is the canonical example.
+
+2. **Judge function**: a callable matching the v3.8 judge interface (`claim_text`, `retrieved_excerpt`, `anchor_kind`, `anchor_value`, `active_constraints`, `judge_model` kwargs). The shipped test fixture pairs the gold set with a perfect-judge stub so T-C1 / T-C2 / T-C3 exercise the calibration tooling end-to-end without requiring a live LLM call. Production deployment plugs a real judge_fn in place of the stub.
+
+3. **Thresholds** (optional): `{"FNR": 0.15, "FPR": 0.10}` per spec §7.7 + §9 acceptance. Tightening these is a spec bump; loosening them is not permitted without a corresponding judge-quality-improvement justification recorded in the calibration report's `thresholds` block.
+
+---
+
+## Gold tuple schema
+
+Two discriminated `tuple_kind` shapes, validated at ingestion by `validate_gold_set` (raises `GoldSetValidationError` on the first rule violation).
+
+### Alignment tuple
+
+```jsonc
+{
+  "tuple_kind": "alignment",
+  "claim_text": "<the prose claim under test>",
+  "ref_text_excerpt": "<source-passage excerpt the judge will evaluate against; null for RETRIEVAL_FAILED tuples>",
+  "anchor": {"kind": "page" | "section" | "quote" | "paragraph" | "none", "value": "<...>"},
+  "expected_judgment": "SUPPORTED" | "UNSUPPORTED" | "AMBIGUOUS" | "RETRIEVAL_FAILED"
+}
+```
+
+Alignment tuples MUST NOT carry any of: `constraint_under_test_id`, `constraint_under_test_rule_text`, `manifest_fixture_path` (rule (b)). Mixing constraint fields onto an alignment tuple corrupts the per-class confusion matrix because the judge call shape differs between the two kinds.
+
+### Constraint tuple
+
+```jsonc
+{
+  "tuple_kind": "constraint",
+  "claim_text": "<the prose claim under test>",
+  "ref_text_excerpt": null,
+  "anchor": {"kind": "page" | "section" | ..., "value": "<...>"},
+  "expected_judgment": "VIOLATED" | "NOT_VIOLATED",
+  "constraint_under_test_id": "MNC-N" | "NC-CN-M",
+  "constraint_under_test_rule_text": "<inline rule text>",     // EITHER this
+  "manifest_fixture_path": "scripts/fixtures/.../mnc.json"     // OR this
+}
+```
+
+Both rule sources are accepted by `validate_gold_set` because manifest-bound constraints sometimes carry domain-specific rule text that authors want versioned alongside the manifest. **The v3.8.0 calibration runner supports only the inline `constraint_under_test_rule_text` form.** A manifest-only tuple validates clean per rule (c) but `run_calibration` raises `NotImplementedError` at run time rather than reach the judge with an empty rule (R2 codex P1 closure on the silent-skip risk that T-C3 is meant to prevent). The manifest-fixture resolver is a post-v3.8 deliverable; until it ships, author calibration fixtures with inline rule text.
+
+---
+
+## Process
+
+### Phase 0: Intake
+
+1. Call `validate_gold_set(tuples)`. Any of the four spec §7.7 rules (a)/(b)/(c)/(d) failing raises `GoldSetValidationError` with the offending tuple index and the rule name. Do NOT catch the exception — fix the gold set.
+
+### Phase 1: Per-tuple judge invocation
+
+For each tuple, the calibration runner calls `judge_fn(...)` with kwargs matching the v3.8 judge interface:
+- Alignment tuples (all four `expected_judgment` values including `RETRIEVAL_FAILED`) → `retrieved_excerpt = tuple.ref_text_excerpt` (may be `null` for `RETRIEVAL_FAILED` tuples), `active_constraints=[]`. The runner does NOT pre-filter `RETRIEVAL_FAILED` tuples — the judge stub is responsible for returning the matching label.
+- Constraint tuples → `retrieved_excerpt = tuple.ref_text_excerpt` (typically `null`), `active_constraints=[{"constraint_id": ..., "rule": ..., "scope": "MNC" | "NC"}]`. Scope is derived from `constraint_under_test_id`: `MNC-N` → `"MNC"`, `NC-CN-M` → `"NC"`. Malformed ids are rejected by `_derive_constraint_scope` with a diagnostic naming the expected shape.
+
+No ensembling at this layer. Spec §7.7 does not require N-run majority voting (reviewer mode does, but reviewer-paper unit-of-analysis is heavier and benefits more from variance reduction; per-tuple alignment is a simpler call). Re-running the same tuple is the operator's responsibility if a high-variance judge_model warrants it.
+
+### Phase 2: Confusion-matrix accumulation
+
+The runner accumulates two confusion matrices:
+
+- **Aggregate** (the rates that T-C1 gates against): a tuple contributes a FN when `expected_judgment ≠ actual judgment`. For alignment, every mismatched tuple counts as both FN (for the expected class) and FP (for the wrongly-picked class), so aggregate FNR and FPR are symmetric. For constraint tuples, VIOLATED is the positive class (gate-refuse signal); the matrix is the standard binary form.
+- **Per-class one-vs-rest** (the report block that T-C2 checks): for each of `SUPPORTED`, `UNSUPPORTED`, `AMBIGUOUS`, `violated_constraint`, the runner computes FNR / FPR with that class as the positive label. `n_positive` / `n_negative` denominators are surfaced so a "0.0 FNR on n_positive=0" entry is distinguishable from "0.0 FNR on n_positive=8".
+
+The fourth alignment class `RETRIEVAL_FAILED` is intentionally NOT surfaced in `per_class`: the pipeline sets it pre-judge at operational deployment, so judge-quality FNR/FPR against this label is uninformative for tooling validation. To keep the per-tuple call shape uniform, the runner still passes `RETRIEVAL_FAILED` tuples to `judge_fn` (no pre-filter) — the stub or real judge echoes the expected label, and the tuple contributes to aggregate FNR/FPR via the non-`per_class` aggregate path.
+
+### Phase 3: Threshold check
+
+If `report["FNR"] >= thresholds["FNR"]` OR `report["FPR"] >= thresholds["FPR"]`, the test (T-C1) fails. The report's `thresholds` block echoes the active gate values so CI failure messages can distinguish:
+
+- A regression (judge quality degraded — same gold set, higher FNR/FPR than the previous run).
+- A spec bump (tighter thresholds — same judge, but T-C1 now demands more).
+
+Both surface as the same CI failure; the report block makes the diagnosis explicit.
+
+### Phase 4: Report shape
+
+```jsonc
+{
+  "FNR": 0.0,
+  "FPR": 0.0,
+  "per_class": {
+    "SUPPORTED":           {"FNR": 0.0, "FPR": 0.0, "n_positive": 5, "n_negative": 12},
+    "UNSUPPORTED":         {"FNR": 0.0, "FPR": 0.0, "n_positive": 8, "n_negative": 9},
+    "AMBIGUOUS":           {"FNR": 0.0, "FPR": 0.0, "n_positive": 3, "n_negative": 14},
+    "violated_constraint": {"FNR": 0.0, "FPR": 0.0, "n_positive": 5, "n_negative": 3}
+  },
+  "partial_support": {"miss_rate": 0.0, "n_partial": 5},
+  "thresholds": {"FNR": 0.15, "FPR": 0.10},
+  "n_total": 25,
+  "n_alignment": 17,
+  "n_constraint": 8
+}
+```
+
+The shape is the canonical contract `scripts/test_claim_audit_calibration.py` (T-C1 / T-C2) pins against. Adding fields is non-breaking; removing or renaming is a spec bump.
+
+**Partial-support subset (`partial_support`, #213).** Gold tuples tagged `expected_prompt_verdict=PARTIAL` carry `expected_judgment=UNSUPPORTED` (the B1 normalization), so a judge that **stops decomposing and emits a bare `UNSUPPORTED`** matches the aggregate gate and hides the exact regression #213 closes. The subset metric counts a partial fixture as passed ONLY when the judge emits `UNSUPPORTED` AND a well-formed true-partial `sub_claim_breakdown` (≥2 items, ≥1 SUPPORTED AND ≥1 non-SUPPORTED — the INV-19 definition). `miss_rate = misses / n_partial`; a bare-`UNSUPPORTED` judge registers `miss_rate=1.0` here while the aggregate FNR stays green. The block is additive — a gold set with no partial fixtures yields `{"miss_rate": 0.0, "n_partial": 0}`.
+
+---
+
+## Failure cases this mode does NOT fix
+
+Calibration reports the judge's error profile on a **specific** gold set in a **specific** domain. It does not:
+
+- Predict performance on tuples outside the gold-set distribution (the canonical fixture is mid-domain synthetic; ML / clinical / qualitative judges should run domain-specific gold sets).
+- Detect rubric-discrimination-power problems on the LLM-as-judge side — that's the `#89 / gold fixtures` work tracked separately (per spec §2 out-of-scope).
+- Replace the post-calibration ramp-on plan recorded in `academic-pipeline/SKILL.md` mode flags. v3.8.0 ships the audit agent dispatch as opt-in default-OFF; T-C1 passing on the canonical gold set is necessary but not sufficient evidence for default-ON.
+
+If a deploying operator brings a gold set that is itself biased (all tuples from one venue, all post-2024, all heavily-redacted), calibration reports a biased profile. Emit a warning during intake if the gold set looks clustered — this protocol does not currently auto-detect clustering, but the operator's pre-deployment review should.
+
+**Partial-evidence regression IS caught (#213).** One failure the aggregate gate alone would NOT catch: a judge that regresses to emitting bare `UNSUPPORTED` on compound claims instead of decomposing them. Because partial fixtures carry `expected_judgment=UNSUPPORTED`, the aggregate FNR stays green on that regression. The `partial_support` subset metric exists precisely to surface it — `miss_rate > 0` means the judge stopped producing inspectable `sub_claim_breakdown` decompositions even though it got the headline label right. Treat a non-zero `partial_support.miss_rate` as a judge-quality regression on the §F.3.2 partial-evidence trap, distinct from an aggregate FNR/FPR threshold breach.
+
+---
+
+## Integration with existing modes
+
+| Mode | Interaction with calibration |
+|---|---|
+| Stage 4 → 5 audit dispatch | Calibration runs offline; results inform the `ARS_CLAIM_AUDIT` ramp-on decision. |
+| Cite-Time Provenance Finalizer | No direct coupling — calibration evaluates the judge in isolation; the finalizer's matrix is downstream. |
+| `/ars-mark-read` clearance | Calibration does not interact with mark-read state; HIGH-WARN-CLAIM-NOT-SUPPORTED stays non-clearable regardless of calibration outcome. |
+| Reviewer `calibration` mode | Sibling, not coupled. Both compute FNR/FPR but on different units (papers vs claim-tuples); both can run in the same session. |
+
+---
+
+## Resolved design decisions (2026-05-15, per spec §10 OQs)
+
+- **Activation**: opt-in. The audit agent ships with default thresholds and the canonical fixture; operators run `scripts/test_claim_audit_calibration` as a CI gate. Re-calibration with a domain-specific gold set is the operator's call, not auto-triggered.
+- **Threshold values**: FNR < 0.15 + FPR < 0.10. Tightened from the reviewer-mode 0.17 / 0.50 Lu 2026 reference points because the audit unit (per-claim) is simpler than the reviewer unit (whole paper) — the gate should track the stricter end of plausible LLM-as-judge accuracy.
+- **Ensembling**: not in v3.8.0. Per-tuple alignment calls are short and the judge's response shape is constrained; majority-vote ensembling adds cost without obvious accuracy gain for this unit of analysis. Re-evaluate if calibration evidence in v3.8.x shows high variance.
+- **Cross-model verification**: out of scope for the calibration runner — `ARS_CROSS_MODEL` interacts with the audit agent dispatch path, not the calibration script. An operator wanting cross-model calibration runs the runner twice with different `judge_model` settings.
+
+---
+
+## Running locally
+
+```bash
+PYTHONPATH=. python3 -m unittest scripts.test_claim_audit_calibration -v
+```
+
+The CI-equivalent invocation lives in `.github/workflows/spec-consistency.yml` (the v3.8 `#103` unittest step). Both paths exercise T-C1 (threshold gates), T-C2 (per-class FNR/FPR reporting), and T-C3 (gold-set shape integrity).
+
+---
+
+## References
+
+- Spec: `docs/design/2026-05-15-issue-103-claim-alignment-audit-spec.md` §7.7 (test contract) + §1 deliverable 7 (this doc) + §9 (acceptance criteria) + §13 step 10 (implementation order).
+- Issue: [academic-research-skills #103](https://github.com/Imbad0202/academic-research-skills/issues/103) acceptance criterion (FNR < 0.15 + FPR < 0.10 gates).
+- Parent agent: `academic-pipeline/agents/claim_ref_alignment_audit_agent.md` — dispatches calibration mode and consumes the report's thresholds block. The agent prompt and this doc form a two-way pair following the v3.6.5 protocol-doc convention (`literature_corpus_consumers.md` ↔ `bibliography_agent.md`).
+- Reviewer calibration baseline: `academic-paper-reviewer/references/calibration_mode_protocol.md` (same vocabulary, different unit of analysis).
+- Lu, C. et al. (2026). *Nature* 651, 914-919 — Table 1 reference rates for LLM-vs-human agreement on whole-paper decisions; this protocol uses tighter thresholds for the simpler per-claim unit.