dsh-aris-panel 0.1.0

package/skills/shared-references/compute-env-contract.md

@@ -0,0 +1,163 @@
+# Compute Environment Contract
+
+> One declarative spec for what an environment IS; per-provider knowledge for
+> how it gets built HERE; a content-hash ledger so "did the env change?" has a
+> mechanical answer; and a three-tier validation ladder whose top tier is a
+> fresh agent following the skill's own doc verbatim. Adapted from Anthropic's
+> Claude Science `compute-env-setup` skill (Apache-2.0); de-coupled from its
+> proprietary `host.*` runtime — everything here runs on plain bash + SSH +
+> subagents.
+
+Every ARIS compute skill (`/run-experiment`, `/experiment-queue`,
+`/serverless-modal`, `/vast-gpu`, `/qzcli`) needs the same three things for a
+job: a software stack (exact versions, often with load-bearing install order),
+possibly large weights placed where the tool looks, and a resource shape. What
+varies per provider is only HOW those materialize. Without a shared contract,
+each skill re-encodes provider quirks and every "environment is ready" claim is
+vibes. The classic failure this prevents: agent says "env ready", the overnight
+run dies at `import flash_attn`, 8 GPUs idle until morning.
+
+## 1. Provider shapes — recognize, don't choose
+
+You are rarely choosing a shape; you are recognizing which one this provider
+already is. The shape determines what "build", "register", and "resolve" mean.
+
+| Shape | ARIS examples | Build = | Env name resolves to |
+|---|---|---|---|
+| **Direct SSH host** (conda/venv) | personal GPU boxes, lab servers | YOU are the renderer: `conda create -n <name> python=<X>`, then run `pip_phases` in order | the conda env name itself (`conda run -n <name> …`) |
+| **Scheduler cluster** (Slurm/PBS; Qizhi-like platforms) | `/qzcli` targets | `module load` or a container image built OFF-cluster and pulled (compute nodes often have **no internet** — pre-stage everything) | scheduler directives + container path in shared scratch (mind purge windows) |
+| **Managed API** (serverless) | `/serverless-modal`, Vast.ai templates | the provider's image definition (Modal `Image`, Vast template) — render the same spec into it | the provider's opaque image ref, recorded in the ledger |
+
+## 2. The declarative spec (write WHAT once; render per provider)
+
+```yaml
+# env-spec: one dict per environment, portable across shapes
+base:        "cuda12.8 + python3.10"      # FROM-image / conda create versions
+system_pkgs: [git, tmux]                  # apt in a container; conda-forge subset on no-root hosts
+pip_phases:                               # ORDERED list of lists — each inner list = ONE pip call
+  - [torch==2.8.0]                        #   phase 1 first, so later packages
+  - [flash-attn --no-build-isolation]     #   can't drag torch to a wrong wheel
+  - [transformers, peft, accelerate]
+env:         {HF_ENDPOINT: "...", OMP_NUM_THREADS: "<tier.cpus>"}
+run_commands: []                          # escape-hatch shell (RUN / %post / plain SSH)
+weight_dirs: {chai: {path: /scratch/weights/chai, source: "tool's own loader", gated: false}}
+smoke:                                    # probes that run INSIDE the env on every shape
+  import_names: [torch, flash_attn]
+  gpu_tests:                              # each = {cmd, expect}; expect is the witness regex
+    - cmd: "python -c 'import torch;torch.manual_seed(0);x=torch.randn(8,8,device=\"cuda\");print(\"WITNESS\", (x@x).shape, torch.cuda.get_device_name())'"
+      expect: "^WITNESS torch.Size"
+  cli_checks:   [nvidia-smi]
+```
+
+- **`pip_phases` ordering IS the fix** for every "package A drags B to the
+  wrong version" problem: each phase is its own pip invocation, and pip leaves
+  an already-satisfied requirement alone unless asked to upgrade. Pin the
+  fought-over package in an EARLIER phase than the fighter.
+- A clean spec renders unchanged through every renderer. If you find yourself
+  adding a field only one backend understands, that field belongs in the
+  provider's ledger entry, not the spec.
+- **Weights**: small (<~500 MB) and read by every job → bake into the env at
+  build time. Large with a cache env var → persistent scratch + point the var
+  there. Populate with the **tool's own loader** (hand-curled layouts miss
+  marker files), then verify from the tool's perspective: run the real
+  entrypoint once against the staged dir and `du -sh` every subdir — 0 B means
+  a swallowed download error.
+
+## 3. The environment ledger (content-hash = mechanical staleness)
+
+Per provider, keep an append-friendly `.aris/compute/<provider>.md` (or the
+project's existing server-notes file). One block per env, keyed by a content
+hash of the spec, computed over an EXACT canonical form so two agents can
+never hash the same spec differently: parse the spec file, re-serialize as
+JSON with sorted keys and no whitespace, sha256, first 8 hex chars —
+
+```bash
+# spec stored as YAML (env-spec.yaml); requires PyYAML. If PyYAML is absent,
+# store the spec as JSON instead and drop the yaml import — same pipeline.
+python3 -c 'import sys,json,hashlib,yaml; \
+s=json.dumps(yaml.safe_load(open(sys.argv[1])),sort_keys=True,separators=(",",":")); \
+print(hashlib.sha256(s.encode()).hexdigest()[:8])' env-spec.yaml
+```
+
+Key order, comments, indentation, and trailing whitespace in the source file
+do NOT affect the hash — only the parsed content does:
+
+```
+### env: dllm@a3f9c2e1
+how: conda env "dllm" on <host>            # or: modal image ref / .sif path + partition
+tier: {cpus: 8, mem_gib: 64, gpus: 1}
+weights: HF_HOME=/scratch/hf (24 GB; purge-window 30d)
+validated: 2026-07-02 (witness + agent-follows-doc clean)
+gotcha: <any diagnosis-table row hit on THIS provider>
+```
+
+Spec changed → hash changes → **cache miss**: the ledger entry no longer
+matches and the env must be rebuilt (or a new block added). Spec unchanged →
+warm-reuse without rebuilding or re-validating tier 1–2. This turns "I think
+the env is the same as last week" into a string comparison. Note `.aris/` is
+gitignored by convention — the ledger is **project-local and uncommitted** by
+default (like `.aris/traces/`). If you want committed, git-blameable history,
+keep the ledger blocks in the project's tracked server-notes file instead;
+the block format is the contract, not the path.
+
+## 4. Validation — three tiers; the gap between them is where debugging lives
+
+1. **Import works** — `python -c "import <pkg>"` exits 0. Necessary, cheap,
+   catches almost nothing interesting.
+2. **Kernel-dispatch witness** — a tiny SEEDED forward pass that prints a
+   sentinel line (output shape + device name + non-emptiness). Catches "torch
+   sees the GPU but the kernel was compiled for an older SM", "the compiled
+   extension's `.so` isn't on the loader path", "inference writes to a
+   read-only cache". Keep the witness command in the spec's `smoke.gpu_tests`
+   with an `expect:` regex so the SAME probe runs on every backend. Cheap —
+   run on every build.
+3. **Agent-follows-doc** — the validation that actually matters and the one
+   that's easy to skip. Spawn a FRESH subagent that gets ONLY: the compute
+   skill's doc, the provider's ledger entry, and the documented invocation.
+   It must run the invocation **verbatim** — no improvisation, no fixing —
+   and report every point where the doc's claim and reality diverge. This is
+   where you find the doc says `--ligand` but the flag is
+   `--ligand_description`, or the weights path exists but lacks the completion
+   marker the tool checks. The author agent cannot self-certify its own doc
+   (it walks through on hidden knowledge the doc never wrote down — same
+   principle as `acceptance-gate.md`: the writer never acquits its own
+   artifact); the fresh agent's stuck-point IS the doc's lie. Expensive —
+   reserve for the two moments doc and env can drift: **after any env rebuild
+   or doc edit, and before declaring an env ready**.
+
+## 5. Diagnosis table (symptom → layer → fix)
+
+When a documented invocation fails, don't patch reflexively — ask which LAYER
+is wrong: spec, build, weights, resolution, or doc. Grep-able rows (container
+rows apply only to container shapes):
+
+| Symptom | Layer | Fix |
+|---|---|---|
+| `no kernel image is available for execution` | build/spec | torch compiled for older SM than this GPU — record `sm_range` in the ledger and route jobs; rebuild only if no compatible hardware |
+| `ModuleNotFoundError` for a package not in the spec | spec | a `--no-deps` install skipped a runtime dep — read the package's `pyproject.toml` and add an explicit phase |
+| Wrong torch/numpy version after install | spec | a later package's pin won — add a `force-reinstall --no-deps` snap-back phase after it |
+| `ImportError: libfoo.so: cannot open shared object` | build | compiled `.so` not on loader path — `find` it, add its dir to `LD_LIBRARY_PATH` |
+| Tool re-downloads despite populated weights | weights | `du -sh $CACHE_VAR` first: 0 B = swallowed error; non-zero = tool checks a marker file, stage that too |
+| `OSError: Read-only file system` under cache var | weights (container) | tool writes locks next to weights on an RO mount — symlink blobs into writable `/tmp` cache |
+| 80-way thread storm on a 4-CPU allocation | exec | `os.cpu_count()` returns the HOST's cores — export `OMP/MKL/OPENBLAS_NUM_THREADS=<tier.cpus>` on every backend |
+| First job slow, every later job equally slow | build | expensive precompute runs at job time in a non-persistent workdir — run it once at build time |
+| Job COMPLETED but output dir empty | exec | the wrapper writing the completion marker never ran — often `#!/bin/bash` on a runtime that only ships `/bin/sh` |
+
+Hit a row on a specific provider → append symptom + fix to that provider's
+ledger `gotcha:` line, so the next agent doesn't rediscover it.
+
+## How compute skills use this
+
+- **Before building**: read the provider's ledger. The env — or a near-match
+  to extend — may already exist; an unchanged hash means skip the rebuild.
+- **When building**: write the spec first (§2), render it for the shape (§1),
+  run tier-1/2 validation (§4), append the ledger block (§3).
+- **Before declaring ready** (and after any rebuild/doc edit): run the
+  agent-follows-doc pass (§4.3).
+- **On failure**: diagnosis table (§5) before patching; record provider-true
+  gotchas in the ledger.
+
+Attribution: the spec/ledger/three-tier-validation design is adapted from
+Anthropic's Claude Science `compute-env-setup` skill (Apache-2.0, re-hosted by
+HughYau/AcademicForge); this document ports it off the proprietary `host.*`
+runtime onto plain bash + SSH + ARIS subagents.

package/skills/shared-references/effort-contract.md

@@ -0,0 +1,183 @@
+# Effort Contract
+
+## Overview
+
+Every ARIS skill accepts an optional `effort` parameter that controls how much work the system does. This affects breadth, depth, iterations, and coverage — but **never** the quality of cross-model review.
+
+> Design stance: the unattended *procedure* (gather → reason → act → verify → repeat) is the engineered artifact, not any single prompt — after Karpathy's "write the loop, not the prompt" (LOOPS.md, *Field Notes on Agents That Run for Days*).
+
+```
+/any-skill "args" — effort: lite | balanced | max | beast
+```
+
+Default: `balanced` (current behavior, zero change for existing users).
+
+## Hard Invariants (NEVER changed by effort)
+
+| Setting | Value | Why |
+|---------|-------|-----|
+| Codex reasoning_effort | **≥ xhigh** (deep-audit skills run `ultra` — tier table in `reviewer-routing.md`) | Reviewer quality is non-negotiable. `effort` never moves the reviewer tier in either direction — and ARIS `— effort: max` is NOT Codex `model_reasoning_effort: max` (different axes: pipeline workload vs reviewer reasoning depth) |
+| DBLP/CrossRef citations | **on** | Citation integrity is non-negotiable |
+| Reviewer independence | **on** | Cross-model protocol is non-negotiable |
+| Experiment integrity | **on** | Fraud prevention is non-negotiable |
+| Sanity check | **on** | Safety is non-negotiable |
+| **Mandatory audit emission** | **always** | At `assurance: submission`, every mandatory audit emits a verdict (PASS/WARN/FAIL/NOT_APPLICABLE/BLOCKED/ERROR). Silent skip is forbidden. See `assurance-contract.md`. |
+| AUTO_PROCEED | **user decides** | Orthogonal to effort |
+| difficulty | **user decides** | Orthogonal to effort |
+| `assurance` | **derived from `effort`** (see Assurance Axis below) | Audit strictness is a separate axis from depth |
+
+## Four Levels
+
+### `lite` (~0.4x tokens)
+For budget-constrained users or quick explorations. Minimum viable depth.
+Implies `assurance: draft` (see below).
+
+### `balanced` (1x tokens) — DEFAULT
+Current ARIS behavior. What existing users get today. No breakage.
+Implies `assurance: draft`.
+
+### `max` (~2.5x tokens)
+Go deeper than defaults. More papers, more ideas, more rounds, more detail.
+Implies `assurance: submission` — mandatory audits are load-bearing.
+
+### `beast` (~5-8x tokens)
+No budget limit. Every knob to maximum. For top-venue submission sprints.
+Implies `assurance: submission` — mandatory audits are load-bearing and the
+final report is tagged `submission-ready` only when the verifier agrees.
+
+## Assurance Axis (separate concern from `effort`)
+
+Audit strictness lives on a second axis, `assurance`. Full contract:
+**`shared-references/assurance-contract.md`**.
+
+```
+— assurance: draft | submission
+```
+
+Default mapping (if `assurance` not given explicitly):
+
+| `effort` | implied `assurance` | Behavior |
+|----------|---------------------|----------|
+| `lite` | `draft` | Audits run only if content detector matches; silent skip allowed |
+| `balanced` | `draft` | Same as lite — current behavior, zero breakage |
+| `max` | `submission` | Every mandatory audit emits a verdict; verifier blocks Final Report on FAIL/BLOCKED/ERROR/STALE |
+| `beast` | `submission` | Same as max + final report tagged `submission-ready` |
+
+User can override independently:
+- `— effort: balanced, assurance: submission` → normal depth, strict audits
+- `— effort: beast, assurance: draft` → maximum depth, no audit gate (legal but discouraged for real submissions)
+
+**Why split the axes?** Historically `effort: beast` did not enforce audits — phases like `/proof-checker`, `/paper-claim-audit`, `/citation-audit` were gated by content detectors that allowed silent skip. A user reported `effort: beast` produced a "draft-quality" paper with all three submission gates skipped. The split makes audit strictness independently verifiable and stops conflating "do more work" with "be more rigorous."
+
+## Per-Skill Profiles
+
+### Discovery & Planning
+
+| Skill | Dimension | lite | balanced | max | beast |
+|-------|-----------|------|----------|-----|-------|
+| research-lit | papers found | 6-8 | 10-15 | 18-25 | 40-50 |
+| research-lit | query variants | 2 | 5 | 8 | 15+ |
+| research-lit | deep reads | 3 | 5-8 | 8 | 15+ |
+| idea-creator | ideas generated | 4-6 | 8-12 | 12-16 | 20-30 |
+| idea-creator | pilots | 1-2 | 2-3 | 3-4 | 5-6 |
+| novelty-check | claims checked | 2-3 | 3-4 | 4-6 | all |
+| novelty-check | closest works | top-3 | top-5 | top-8 | top-10+ |
+| research-refine | max rounds | 3 | 5 | 7 | 10+ |
+| research-refine | papers considered | 8 | 15 | 24 | 30+ |
+| experiment-plan | core experiments | 3 | 5 | 7 | 10+ |
+| experiment-plan | seeds | 1 | 3 | 5 | 5 |
+| experiment-plan | baseline families | 2 | 3 | 4 | 5+ |
+
+### Execution
+
+| Skill | Dimension | lite | balanced | max | beast |
+|-------|-----------|------|----------|-----|-------|
+| experiment-bridge | scope | sanity + main | main + basic ablation | + top ablation + robustness | full suite + cross-validation |
+| run-experiment | launches | smoke + main | smoke + multi-seed | + dry run + manifest | full config + multi-GPU parallel |
+| monitor-experiment | depth | latest log | log + JSON | + W&B + anomaly | real-time + auto-alert + trend |
+| analyze-results | findings | 3 | 5 | 8 | full-dimensional + stat tests |
+| ablation-planner | ablations | 2-3 | 4-5 | 6-8 | 10+ |
+
+### Review
+
+| Skill | Dimension | lite | balanced | max | beast |
+|-------|-----------|------|----------|-----|-------|
+| auto-review-loop | max rounds | 2 | 3-4 | 6 | 8+ (until converged) |
+| auto-review-loop | fixes per round | 1-2 | 3-4 | 4-6 | all actionable |
+| research-review | passes | 1 | 1 + follow-up | 1 + 2 follow-ups | 2 independent + cross-compare |
+| experiment-audit | depth | skip | basic 4 checks | full 6 checks | line-by-line + reproduce |
+
+### Writing & Rebuttal
+
+| Skill | Dimension | lite | balanced | max | beast |
+|-------|-----------|------|----------|-----|-------|
+| paper-plan | outline reviews | 0 | 1 | 2 | 3 |
+| paper-plan | citations/section | 2-3 | 4-5 | 5-8 | 8+ |
+| paper-figure | caption reviews | 1 | 1 | 2 | 3 |
+| paper-write | abstract variants | 1 | 1 | 2 | 3 |
+| paper-write | related work depth | shallow | standard | deep | exhaustive |
+| paper-compile | fix attempts | 2 | 3 | 4 | until zero warnings |
+| auto-paper-improvement | rounds | 1 | 2 | 3 | 5 |
+| paper-illustration | render iterations | 2 | 3 | 5 | 7 |
+| rebuttal | draft rounds | 1 | 2 | 3 | 5 |
+| rebuttal | stress tests | 0-1 | 1 | 2 | 3 |
+
+## How to Read Effort in a Skill
+
+Add this to the Constants section of each skill:
+
+```markdown
+## Constants
+
+- **EFFORT = `balanced`** — Work intensity. Options: `lite`, `balanced`, `max`, `beast`. Override: `— effort: max`
+```
+
+Then adjust numeric constants based on effort level. Example:
+
+```
+Parse $ARGUMENTS for `— effort:` directive.
+If not specified, default to `balanced`.
+
+Adjust constants:
+  if effort == "lite":     MAX_PAPERS = 8,  MAX_IDEAS = 6,  MAX_ROUNDS = 2
+  if effort == "balanced": MAX_PAPERS = 15, MAX_IDEAS = 12, MAX_ROUNDS = 4
+  if effort == "max":      MAX_PAPERS = 25, MAX_IDEAS = 16, MAX_ROUNDS = 6
+  if effort == "beast":    MAX_PAPERS = 50, MAX_IDEAS = 30, MAX_ROUNDS = 8
+```
+
+## Transparency
+
+Every skill should print its effort configuration at the start:
+
+```
+⚡ [effort: max] papers=25, ideas=16, rounds=6 | Codex: tier per reviewer-routing.md (floor xhigh)
+```
+
+## Precedence
+
+```
+explicit concrete knob (e.g., review_rounds: 2)
+  > explicit dimension override
+    > overall effort level
+      > skill default (balanced)
+```
+
+Example: `— effort: beast, review_rounds: 3` → everything beast except review capped at 3.
+
+For the `assurance` axis, precedence is independent:
+```
+explicit `assurance: ...` directive
+  > effort-implied default (lite/balanced → draft, max/beast → submission)
+    > skill default (draft)
+```
+
+Example: `— effort: balanced, assurance: submission` → normal depth knobs but submission-gate audit enforcement.
+
+## Token Cost Estimation
+
+| Level | LLM tokens | GPU/wall-clock | Best for |
+|-------|-----------|----------------|----------|
+| lite | ~0.4x | ~0.5x | Quick exploration, budget users |
+| balanced | 1x | 1x | Normal research workflow |
+| max | ~2.5x | ~2x | Serious submission prep |
+| beast | ~5-8x | ~3-4x | Top-venue final sprint |

package/skills/shared-references/evidence-precheck.md

@@ -0,0 +1,65 @@
+# Evidence Pre-check
+
+ARIS's claim audits (`/result-to-claim`, `/experiment-audit`, `/paper-claim-audit`)
+spend a cross-model (codex/gemini) call to judge whether a claim is supported. The
+cheapest, most common integrity failure is *hallucinated evidence*: a claim cites
+a number + a source file, and the file doesn't exist or the number isn't in it.
+You should not need a model call to catch that.
+
+## Two stages — and `verified` ≠ `correct`
+
+```
+stage 1  tools/evidence_check.py   deterministic · no model · fail-closed
+         catches HALLUCINATION — cited path missing, or cited value not in source.
+stage 2  the cross-model jury      codex/gemini
+         catches WRONG-BUT-REAL — the number IS in the file, but it doesn't
+         support the claim.
+```
+
+A `verified` from stage 1 means **only that the cited evidence exists** — never
+that the claim holds. Existence is execution-completeness (deterministic / safe
+same-model); *support* is a quality verdict that stays with the cross-model jury
+(`acceptance-gate.md`: the pre-check DRIVES a gate, it cannot ACQUIT a claim).
+This is the reconcile pattern — a model's self-report cross-checked against
+mechanical ground-truth (adapted from Hermes's curator reconcile-classifier),
+made into a cheap pre-gate that catches hallucination *before* the jury runs and
+spares the codex call on fabricated evidence.
+
+## Conservative by design
+
+The pre-check favors **false-negative over false-positive**: when in doubt it
+returns not-verified and lets the jury decide — it must never emit a false
+`verified`. A pure number is matched by **numeric-token equality** (so `73.2`
+matches `73.20` but `73` does NOT match `73.5`); a non-numeric value by
+normalized substring.
+
+## Where ARIS uses it
+
+- **`/result-to-claim`** Step 1.5: parse each claim's cited `(value, source)`,
+  run the batch pre-check, and **before the codex judgment** mark any claim whose
+  evidence is `path_missing` / `value_not_found` as **unsupported — evidence not
+  found**, and pass the per-claim pre-check status into the codex prompt so the
+  jury sees which claims have verified vs hallucinated evidence.
+- **To extend:** `/experiment-audit` (the "phantom results" check is exactly
+  this) and `/paper-claim-audit` (every reported number → its result file).
+
+## API / CLI
+
+```
+from evidence_check import check_claim, check_batch
+check_claim(value, source, root=".")   # -> {status: verified|path_missing|value_not_found, ...}
+check_batch([{value, source, id?}, ...], root)  # -> {results:[...], summary:{status: n}}
+```
+```
+python3 tools/evidence_check.py <root> --value 73.2 --source results/eval.json   # exit 0 verified
+python3 tools/evidence_check.py <root> --batch claims.json   # exit 1 if any claim hallucinated
+```
+
+## Cross-references
+- `acceptance-gate.md` — the pre-check is the deterministic DRIVE; the jury is the
+  ACQUIT. `verified` is existence (execution-completeness), not correctness.
+- `reviewer-independence.md` — the jury still reads the artifacts itself; the
+  pre-check only flags which claims have evidence to read, never pre-digests the
+  verdict.
+- `experiment-integrity.md` — fabricated/phantom results are exactly what stage 1
+  catches deterministically before stage 2.

package/skills/shared-references/experiment-integrity.md

@@ -0,0 +1,49 @@
+# Experiment Integrity Protocol
+
+## Core Principle
+
+**The model that writes experiment code must NOT be the model that judges experiment integrity.** This is the same principle as reviewer-independence, applied to experiments.
+
+## Prohibited Patterns
+
+### 1. Fake Ground Truth
+- ❌ Creating synthetic "reference" from model outputs and comparing against it
+- ❌ Using baseline model outputs as ground truth
+- ❌ Generating pseudo-GT that is structurally similar to predictions
+- ✅ Using dataset-provided ground truth
+- ✅ Using official evaluation scripts when available
+- ✅ Proxy evaluation is allowed IF explicitly labeled as `synthetic_proxy`
+
+### 2. Score Normalization Fraud
+- ❌ Dividing metrics by max/min of model's own output to get 0.99+
+- ❌ Rescaling scores to hide poor performance
+- ✅ Standard normalization (e.g., min-max across ALL methods including baselines)
+- ✅ Reporting raw and normalized scores side by side
+
+### 3. Phantom Results
+- ❌ Claiming results from files that don't exist
+- ❌ Referencing metrics from functions that are never called
+- ❌ Reporting TRACKER status as DONE when it's still TODO
+- ✅ Every claimed number must trace to an actual output file
+
+### 4. Insufficient Scope
+- ❌ Reporting 2-scene pilot as "comprehensive evaluation"
+- ❌ Using words like "robust", "extensive", "across settings" for tiny experiments
+- ✅ Honestly label scope: "pilot (N=2)", "preliminary", "limited evaluation"
+- ✅ State exact scope: N scenes, N seeds, N configurations
+
+## Evaluation Types (must be declared)
+
+| Type | Label | What it means | Claim ceiling |
+|------|-------|---------------|---------------|
+| Real GT | `real_gt` | Dataset-provided ground truth | Full performance claims |
+| Synthetic proxy | `synthetic_proxy` | Model-generated reference | "Proxy consistency" only |
+| Self-supervised | `self_supervised_proxy` | No GT by design | Relative improvement only |
+| Simulation | `simulation_only` | Simulated environment | "In simulation" qualifier |
+| Human eval | `human_eval` | Human judges | Subject to inter-rater stats |
+
+## Who Checks
+
+The **reviewer model** (different family from executor) performs integrity checks via `/experiment-audit`. The executor collects file paths; the reviewer reads code and results directly.
+
+**Never let the executor judge its own experiment integrity.**