psyclaw 0.27.22 → 0.28.0

package/vendor/ars/shared/agents/compliance_agent.md

@@ -0,0 +1,136 @@
+---
+name: compliance_agent
+description: "Runs PRISMA-trAIce + RAISE compliance checks at Stage 2.5 / 4.5 integrity gates and emits Schema 12 compliance_report"
+version: 1.0.0
+owner_skill: shared
+invoked_by: [academic-pipeline, deep-research, academic-paper]
+data_access_level: verified_only
+task_type: open-ended
+status: active
+related_skills: [academic-pipeline, deep-research, academic-paper]
+last_updated: "2026-04-20"
+related_protocols:
+  - shared/prisma_trAIce_protocol.md
+  - shared/raise_framework.md
+  - shared/compliance_checkpoint_protocol.md
+  - shared/compliance_report.schema.json
+  - academic-paper/references/anti_leakage_protocol.md
+  - academic-pipeline/references/ai_research_failure_modes.md
+---
+
+# compliance_agent
+
+Mode-aware agent that runs PRISMA-trAIce + RAISE compliance checks at Stage 2.5 / 4.5 Integrity Gates, producing Schema 12 `compliance_report` outputs.
+
+## Scope
+
+- **Reads:** manuscript draft, methodology blueprint, bibliography, material passport, user-provided AI tool metadata
+- **Writes nothing to the manuscript.** Output is a separate `compliance_report` handed to the orchestrator.
+- **Does not hallucinate missing items.** Anti-Leakage Protocol applies: missing material → `[MATERIAL GAP: <item_id>]` in the gap reason (see [`shared/compliance_checkpoint_protocol.md#canonical-gap-tag-vocabulary`](../compliance_checkpoint_protocol.md#canonical-gap-tag-vocabulary)).
+
+## Input contract
+
+```yaml
+compliance_mode: "systematic_review" | "primary_research" | "other_evidence_synthesis"
+stage: "2.5" | "4.5"
+materials:
+  manuscript_draft: <path or inline>
+  methodology_blueprint: <path>
+  bibliography: <path>
+  material_passport: <Schema 9 payload>
+user_metadata:
+  intended_venue: <string or null>
+  ai_tools_used:
+    - {name, version, developer, stage, purpose, access_url_or_doi}
+```
+
+## Output contract
+
+`compliance_report` conforming to `shared/compliance_report.schema.json` (Schema 12). Appended to `material_passport.compliance_history[]` by the orchestrator.
+
+## Dispatch logic
+
+```python
+if compliance_mode == "systematic_review":
+    run PRISMA-trAIce checks (Stage-specific item subset)
+    run RAISE full (principles + 8-role matrix)
+    block_decision respects tier semantics
+elif compliance_mode == "other_evidence_synthesis":
+    run PRISMA-trAIce checks in "adaptation" mode (items become Info)
+    run RAISE full
+    block_decision capped at warn
+elif compliance_mode == "primary_research":
+    run RAISE principles_only
+    set prisma_trAIce to null
+    block_decision capped at warn
+```
+
+## Stage behaviour
+
+| Stage | PRISMA-trAIce items checked | RAISE focus |
+|---|---|---|
+| 2.5 | M1, M2, M3, M4, M5, M6, M7, M8, M9, M10 | human_oversight, fit_for_purpose |
+| 4.5 | T1, A1, I1, R1, R2, D1, D2 | transparency, reproducibility (+ full 8-role matrix for SR) |
+
+Items within each stage subset are independent and MAY be evaluated concurrently. The agent SHOULD read `material_passport.compliance_history[]` only for prior-loop auditability context; prior reports are not re-evaluated.
+
+## Tier → decision mapping (SR mode)
+
+| Tier | FAIL → |
+|---|---|
+| Mandatory | `block_decision = block` |
+| Highly Recommended | `block_decision = warn` |
+| Recommended | `block_decision = pass` (logged as info) |
+| Optional | `block_decision = pass` (logged as info) |
+
+Multiple tier contributions aggregate using `max_severity` (see `compliance_checkpoint_protocol.md §Decision precedence`).
+
+### Mandatory-block surface message
+
+When a Mandatory-tier PRISMA-trAIce item triggers `block_decision = block`, the surfaced block message MUST include the following maturity note alongside the gap reason and override-ladder reference, so the user understands what authority is blocking them:
+
+> *Note: PRISMA-trAIce is currently a foundational proposal (Holst et al. 2025, *JMIR AI*, doi:[10.2196/80247](https://doi.org/10.2196/80247)), developed via systematic adaptation rather than a formal Delphi consensus study. Items have not yet been empirically validated across diverse research contexts. See `shared/prisma_trAIce_protocol.md` § Status disclaimer.*
+
+This note is informational only — it does not lower the block severity. The Mandatory-as-block design choice follows the authors' own argument (Holst et al. 2025) that non-transparent AI use is the higher-cost failure mode. Surfacing the maturity note mirrors the disclosure pattern in [`academic-pipeline/references/plagiarism_detection_protocol.md`](../../academic-pipeline/references/plagiarism_detection_protocol.md), which discloses heuristic-screening scope to the user.
+
+## Self-check protocol
+
+Before finalising the report, the agent runs four self-checks. Any flagged check requires re-examination:
+
+| ID | Question | On fail |
+|---|---|---|
+| CA-1 | Am I quoting PRISMA-trAIce items from memory or from `shared/prisma_trAIce_protocol.md`? | Re-read the protocol file; requote. |
+| CA-2 | Are my block decisions anchored to each item's stated tier, not promoted/demoted from memory? | Re-read tier assignments; correct any drift. |
+| CA-3 | **SR mode only.** Did all 17 items pass AND `evidence[]` is empty? (Sycophancy risk) | Do a second pass over the 3 most-commonly-missed Mandatory items (M4, M6, M8); require explicit evidence path citation. |
+| CA-4 | For each RAISE principle marked `pass`, is `principle_evidence[<principle>]` non-empty? | Downgrade the principle to `warn` with note `[WEAK EVIDENCE]`. |
+
+Self-check failures are not errors — they are the agent's guardrail. Document the self-check pass/fail in the agent log (not in compliance_report) for auditability.
+
+## Error behaviour
+
+| Error | Handling |
+|---|---|
+| Missing input material | Apply Anti-Leakage: mark `[MATERIAL GAP]`, item auto-FAILs, tier dictates block/warn. Never hallucinate. |
+| Mode/context mismatch (e.g. `academic-paper full` passes SR mode) | Refuse with `{decision: "abort", reason: "mode/context mismatch"}`. Orchestrator must re-evaluate and re-invoke. |
+| Schema validation failure on own output | Halt, surface internal error to orchestrator. Do NOT append invalid report to compliance_history. |
+| Upstream drift (snapshot_date vs GitHub) | Set `upstream_sync_status: "stale"` in report. Non-blocking. |
+
+<!-- harness-retirement 2026-06-10 (F-008): the "Never hallucinate." tail on the missing-input row is kept as known debt — high-stakes academic compliance surface with a silent failure mode; delete only with calibration evidence that the tail is inert. -->
+
+## Interaction with existing agents
+
+- Runs AFTER `integrity_verification_agent` (existing) at Stage 2.5 / 4.5 — compliance extends integrity, does not replace it.
+- Runs ALONGSIDE `failure_mode_checklist` (v3.2). Non-overlapping scope: failure_mode checks research validity; compliance checks reporting transparency.
+- Provides output consumed by `report_compiler_agent` (at Stage 6) for the AI Self-Reflection Report compliance summary.
+
+## Invocation protocol
+
+The orchestrator (or standalone skill) passes the input contract via the Agent tool with `model: sonnet` or higher (per user CLAUDE.md: never haiku). The agent returns the serialised compliance_report. The orchestrator validates against Schema 12 before appending to passport.
+
+## Related reading
+
+- `shared/prisma_trAIce_protocol.md` — item-level checks
+- `shared/raise_framework.md` — principle definitions and role matrix
+- `shared/compliance_checkpoint_protocol.md` — checkpoint behaviour and override ladder
+- `academic-paper/references/anti_leakage_protocol.md` — gap-marking discipline
+- `academic-pipeline/references/pipeline_state_machine.md` — FAIL-loop integration

package/vendor/ars/shared/artifact_reproducibility_pattern.md

@@ -0,0 +1,173 @@
+# Artifact Reproducibility Pattern (v3.3.5+)
+
+## Why this exists
+
+ARS produces research artifacts — draft papers, literature reviews, review reports. A year
+from now, you may want to reproduce one. Today's Material Passport (Schema 9) captures WHO
+produced the artifact and WHEN, but not WITH WHAT. Which LLM? Which ARS version? Which
+skill files? Which session materials?
+
+The `repro_lock` sub-block fills that gap.
+
+When the Anthropic automated-w2s-researcher paper (2026) claimed "2 researchers × 7 days
+vs 9 agents × 5 days," readers had no way to audit the conditions. This pattern ensures
+that ARS-produced artifacts carry their configuration on record — not as a replay mechanism,
+but as an honest documentation trail.
+
+## What this pattern IS NOT (read this first)
+
+This is non-negotiable. Every future user MUST read this section before assuming the lock
+guarantees anything it does not.
+
+1. **LLM outputs are not byte-reproducible even at temperature 0.** Model providers update
+   weights without changing the model id. `model.weight_stable: false` acknowledges this
+   explicitly. `model.id` is an identifier, not a weight hash.
+
+2. **Live external API queries are recorded as protocol versions, not snapshots.** The S2
+   API changes daily. A paper citing DOI 10.1000/x today may return different metadata
+   tomorrow. `s2_api_protocol_version` records which VERSION of our query logic ran, not
+   a snapshot of the response.
+
+3. **Mid-skill prompt mutations are not captured.** `prompts.hash_timing: "skill-load"` is
+   a declaration: hashes are taken when the skill loads, not per agent call. Late injection
+   of user-provided reference files, conditional branches, and dynamic prompt modifications
+   are NOT reflected in the hash. This is a known imprecision.
+
+4. **`stochasticity_declaration` is required, not optional.** Omitting it makes the lock
+   dishonest by implication. Every generated passport includes the declaration verbatim.
+
+If you want deterministic replay, you are reading the wrong pattern. This pattern gives
+you CONFIGURATION DOCUMENTATION sufficient to INVESTIGATE divergence — not to PREVENT it.
+
+## The block at a glance
+
+```yaml
+repro_lock:
+  schema_version: "1.0"                    # lock schema version
+  stochasticity_declaration: "LLM outputs are not byte-reproducible. This lockfile documents configuration, not a deterministic replay guarantee."
+  ars_version: "<suite-version-at-run-time>"   # e.g. "3.11.1" — the suite version when this run executed
+
+  model:
+    family: claude                          # provider family
+    id: "<model-id-at-run-time>"           # model identifier at run time, e.g. the inherited Claude Code session model (not weight hash)
+    weight_stable: false                    # always false until signed attestation exists
+
+  prompts:
+    hash_timing: skill-load                 # when hashes were taken
+    skill_md_hash: "sha256:..."            # hash of the skill SKILL.md file
+    agents_bundle_hash: "sha256:..."       # hash of agent prompt bundle
+
+  materials:
+    list_hash: "sha256:..."                # hash of [{filename, sha256}, ...] manifest
+    count: 8                               # number of session materials
+
+  external_protocols:
+    s2_api_protocol_version: "<s2-protocol-version-at-run-time>"   # the S2 query protocol version that ran
+    s2_snapshot_available: false           # whether a response cache exists
+
+  cross_model:
+    enabled: false                         # whether cross-model verification ran
+    secondary_model_id: null               # secondary model id if enabled
+```
+
+The block lives as an optional child of the Material Passport (Schema 9 in
+`shared/handoff_schemas.md`). A missing key fails the lint. A null value passes with a
+warning. A populated block with all required sub-fields passes silently.
+
+## Field-by-field rationale
+
+### schema_version
+
+`"1.0"` currently supported. Schema evolution policy: breaking changes bump major,
+additions bump minor. Old readers fail loudly on unknown versions — `check_repro_lock.py`
+will exit 1 if it encounters an unrecognized version, forcing the user to upgrade their
+tooling or audit the change manually.
+
+### ars_version
+
+Suite version from CHANGELOG. This couples the prompt hashes to the release: changing
+`ars_version` will change the bundle hash even if individual prompt files didn't change.
+That coupling is intentional — version boundaries mark coordinated changes.
+
+### model.weight_stable
+
+Currently always `false`. Reserved for a future world where a provider guarantees weight
+stability with a signed attestation. Not today. If you see `true`, treat it as suspicious
+until a signed attestation mechanism exists.
+
+### prompts.hash_timing
+
+Fixed at `"skill-load"` in v1.0. A future `"per-call"` variant would require runtime
+instrumentation not present today. The spec is left open for future work. For now, the
+hash captures the state of files at skill initialization, not at every agent invocation.
+
+### materials.list_hash
+
+Hash of a sorted manifest: `[{filename, sha256}, ...]` sorted by filename. If the user
+edits a session material between runs, the hash changes — which is the correct behavior.
+An identical hash across two runs means the input materials were identical; different
+hashes mean inputs diverged.
+
+### external_protocols.s2_snapshot_available
+
+Almost always `false` today. If a future version of the pipeline caches S2 responses per
+run and stores them alongside the passport, this flag flips to `true` and the snapshot path
+appears in the passport. Until that exists, S2 responses are live queries and are not
+reproducible.
+
+## How it composes with existing infrastructure
+
+- Material Passport (Schema 9) is the PARENT document. `repro_lock` is an OPTIONAL CHILD
+  field. A passport without `repro_lock` fails the v3.3.5+ lint; a passport with
+  `repro_lock: null` passes with a warning (explicit opt-out).
+- Existing passports with `repro_lock` absent fail the new lint. Existing passports with
+  `repro_lock: null` pass with a warning. Populated blocks pass silently.
+- The integrity gate (Stage 2.5, Stage 4.5 in the academic pipeline) does NOT read
+  `repro_lock`. The lock is for post-hoc reproducibility investigation, not runtime
+  validation. Adding it does not change pipeline behavior.
+- `check_repro_lock.py` can be run standalone on any passport file. It is not wired into
+  the CI lint suite by default — it validates on-demand against specific passport files.
+
+## Red flags when reading a populated repro_lock
+
+- `model.weight_stable: true` — this value is not implemented yet; if you see it, the
+  passport author is claiming something the infrastructure cannot verify. Treat as
+  suspicious until a signed attestation mechanism exists.
+- `prompts.hash_timing` anything other than `"skill-load"` — this is an unknown variant
+  that the current tooling does not understand. The lint will not flag it as invalid, but
+  the semantics are undefined.
+- Empty or stub hashes (`"sha256:"` prefix with no content, or a short placeholder string
+  like `"sha256:abc123"`) — the generator was broken or the author filled in a placeholder.
+  Hashes should be 64-character hex strings for SHA-256.
+- `stochasticity_declaration` modified from the verbatim required string — the author is
+  signalling they believe the lock is stronger than it is. Read the entire passport twice
+  before trusting any claims about reproducibility.
+
+## How to generate a lock (for agent authors)
+
+At skill-load time, compute `skill_md_hash` over the SKILL.md content and
+`agents_bundle_hash` over the concatenated agent prompts (canonical ordering). Build the
+materials manifest by listing session files with their SHA-256 digests, sort by filename,
+then hash the JSON serialization. Write the block to the passport before handing off to
+downstream stages. The `stochasticity_declaration` must be included verbatim — do not
+paraphrase or abbreviate it.
+
+## Honesty red line (restate)
+
+This pattern enables HONEST DOCUMENTATION. It does not enable REPLAY. Every field has a
+reason and a limitation. Users who treat the lock as a replay guarantee will be burned.
+Authors who publish passports without `stochasticity_declaration` are failing their readers.
+The locked configuration tells you what ran; it cannot tell you what would run again.
+
+The distinction matters: documentation lets a third party investigate what happened and
+ask whether divergence is expected or alarming. Replay would require weight-frozen models,
+deterministic schedulers, and cached external data — none of which ARS controls today.
+
+## Future evolution
+
+- v1.1: add `cache_snapshot_available: true` support once the pipeline caches S2 responses.
+  The snapshot path would appear as a sibling field in `external_protocols`.
+- v2.0: if weight-signed attestations exist, `model.weight_stable: true` becomes meaningful.
+  Until then the field is reserved.
+- Never: byte-level replay guarantees. LLMs don't work like that; no schema version will
+  change that fact.

package/vendor/ars/shared/benchmark_report.schema.json

@@ -0,0 +1,81 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "https://github.com/Imbad0202/academic-research-skills/shared/benchmark_report.schema.json",
+  "title": "ARS Benchmark Report",
+  "type": "object",
+  "required": [
+    "ars_version",
+    "task_definition",
+    "human_baseline",
+    "ars_run",
+    "metrics",
+    "caveats"
+  ],
+  "properties": {
+    "ars_version": {
+      "type": "string",
+      "pattern": "^\\d+\\.\\d+\\.\\d+$",
+      "description": "Exact ARS version the benchmark was run against"
+    },
+    "task_definition": {
+      "type": "object",
+      "required": ["description", "task_type", "outcome_gradable"],
+      "properties": {
+        "description": { "type": "string", "minLength": 1 },
+        "task_type": { "enum": ["outcome-gradable", "open-ended"] },
+        "outcome_gradable": { "type": "boolean" }
+      }
+    },
+    "human_baseline": {
+      "type": "object",
+      "required": ["sample_size", "author_independence", "hours_spent", "recruitment", "tools_allowed"],
+      "properties": {
+        "sample_size": { "type": "integer", "minimum": 1 },
+        "author_independence": {
+          "enum": ["author-conducted", "author-blinded", "third-party-conducted"],
+          "description": "author-conducted means the benchmark creators also did the human baseline (downward bias risk)"
+        },
+        "hours_spent": { "type": "number", "minimum": 0 },
+        "recruitment": { "type": "string", "minLength": 1 },
+        "tools_allowed": {
+          "type": "array",
+          "items": { "type": "string", "minLength": 1 },
+          "minItems": 1
+        }
+      }
+    },
+    "ars_run": {
+      "type": "object",
+      "required": ["hours_spent", "cost_usd", "skills_used", "data_access_level_declared"],
+      "properties": {
+        "hours_spent": { "type": "number", "minimum": 0 },
+        "cost_usd": { "type": "number", "minimum": 0 },
+        "skills_used": {
+          "type": "array",
+          "items": { "type": "string" },
+          "minItems": 1
+        },
+        "data_access_level_declared": {
+          "enum": ["raw", "redacted", "verified_only"]
+        }
+      }
+    },
+    "metrics": {
+      "type": "object",
+      "required": ["primary_metric", "primary_metric_value", "scoring_independence"],
+      "properties": {
+        "primary_metric": { "type": "string", "minLength": 1 },
+        "primary_metric_value": { "type": "number" },
+        "scoring_independence": {
+          "enum": ["authors-scored", "third-party-scored", "blind-scored", "self-scored"]
+        }
+      }
+    },
+    "caveats": {
+      "type": "array",
+      "items": { "type": "string", "minLength": 1 },
+      "minItems": 1,
+      "description": "Known limitations. Empty array not permitted — honest disclosure required."
+    }
+  }
+}

package/vendor/ars/shared/benchmark_report_pattern.md

@@ -0,0 +1,180 @@
+# Benchmark Report Pattern (v3.3.5+)
+
+**Status**: v3.3.5 — hub doc for benchmark disclosure schema  
+**Schema**: `shared/benchmark_report.schema.json`  
+**Validator**: `scripts/check_benchmark_report.py`  
+**Template**: `examples/benchmark_report_template.json`
+
+---
+
+## Why this exists
+
+The Anthropic automated-w2s-researcher (2026) paper headlined a performance comparison:
+"2 researchers × 7 days (PGR=0.23) vs 9 agents × 5 days (PGR=0.97)."
+
+Dramatic. Also: n=2, author-conducted, no independence, self-scored. Not a scientific
+comparison — a presentation artifact. The authors knew what the agent would produce before
+sitting down to do the human baseline; their "7 days" is implicitly benchmarked against
+their own foreknowledge. Unaware external researchers doing the same task cold would
+likely score higher. The gap may partly be a sandbagged human baseline.
+
+ARS inherits this risk the moment a user publishes an "ARS beats manual" claim. This
+pattern defines a mandatory schema for anyone publishing an ARS benchmark comparison.
+Reports that don't satisfy the schema aren't "ARS benchmarks" — they're anecdotes.
+
+---
+
+## The schema at a glance
+
+Six required top-level fields in `benchmark_report.schema.json`:
+
+- **`ars_version`** — exact semver the benchmark ran against; ties results to a specific
+  skill version so readers can check whether that version is still current.
+- **`task_definition`** — description, task_type (outcome-gradable or open-ended), and
+  outcome_gradable flag; distinguishes rubric-scored tasks from genuinely open-ended ones.
+- **`human_baseline`** — the five provenance fields that determine whether the human side
+  of the comparison is credible (see field-by-field rationale below).
+- **`ars_run`** — cost, time, skills used, and declared data access level; lets readers
+  check the "agents are cheap" or "agents are fast" claims against concrete numbers.
+- **`metrics`** — primary metric name, numeric value, and scoring independence; forces
+  explicit disclosure of who scored the outputs.
+- **`caveats`** — non-empty array of known limitations; at least one honest statement is
+  required by schema enforcement.
+
+---
+
+## Field-by-field rationale
+
+### `human_baseline`
+
+This block exists because the human side of a comparison is where benchmark credibility
+lives or dies. A weak human baseline inflates an agent's apparent advantage.
+
+- **`sample_size`** (integer, minimum 1): Schema rejects zero, forces an explicit number.
+  n=1 or n=2 get a validator warning rather than a hard failure — some tasks are
+  expert-bounded — but the number must be stated. Silence is worse than admitting n=1.
+
+- **`author_independence`** (enum: author-conducted | author-blinded | third-party-conducted):
+  `author-conducted` is the "2 authors did it themselves" trap: allowed, but the schema
+  flags the downward-bias risk. Authors know what a good ARS output looks like; their
+  human effort is calibrated against that prior. `author-blinded` (authors worked without
+  seeing ARS outputs first) is better. `third-party-conducted` eliminates experimenter
+  bias entirely.
+
+- **`hours_spent`** (number, minimum 0): Lets third parties sanity-check the time claim.
+  "7 days" reads differently against 20 papers versus 400.
+
+- **`recruitment`** (non-empty string): One sentence forces self-description of selection
+  bias. "Authors," "hired on Upwork," "convenience sample" are all honest. Blank is not
+  permitted.
+
+- **`tools_allowed`** (array of strings): "No tools" vs "ChatGPT allowed" is not the same
+  comparison as ARS. If the human baseline used no AI assistance while ARS ran full
+  pipeline, the comparison is testing infrastructure access, not cognitive quality.
+
+### `ars_run`
+
+- **`cost_usd`** (number, minimum 0): A run costing $500 in API tokens is not "cheap"
+  just because it took 3 hours. Hours and cost together are needed for a fair comparison.
+
+- **`data_access_level_declared`** (enum: raw | redacted | verified_only): If ARS ran
+  on `raw` data while the human baseline used redacted data, the benchmark is testing
+  data privilege, not skill. Cross-references ground-truth isolation from v3.3.2.
+
+- **`skills_used`** (array, minItems 1): A benchmark running `academic-pipeline`
+  full-mode is not comparable to one running only `deep-research`. Both are ARS;
+  they're different scope claims.
+
+### `metrics.scoring_independence`
+
+Four enum values: `authors-scored`, `third-party-scored`, `blind-scored`, `self-scored`.
+
+`self-scored` parses and exits 0, but the validator emits a warning to stderr. Self-scoring
+is the worst-case alignment failure: the same agents that produced the output are being
+asked whether the output is good. `blind-scored` (human raters without knowledge of which
+output came from ARS versus human baseline) is the minimum credible standard for any
+claim that will be shared publicly.
+
+### `caveats`
+
+Non-empty array, items non-empty strings, minItems 1. The schema physically prevents an
+empty caveats field. A benchmark report with no caveats either has no known limitations
+(implausible for any real-world evaluation) or the author didn't think about limitations
+(which disqualifies the report more than any specific limitation would).
+
+The caveats field is where sample size warnings, scorer bias, task-selection bias, and
+tool-access asymmetries should land when they don't rise to schema errors. It is the
+honest disclosure box.
+
+---
+
+## How to use
+
+1. Copy `examples/benchmark_report_template.json` to your benchmark directory.
+2. Fill every `FILL IN:` field with real values.
+3. Set `human_baseline.sample_size` to the actual number (schema rejects 0).
+4. Run: `python scripts/check_benchmark_report.py your-report.json`
+5. Fix all `ERROR:` lines (schema violations, exit 1).
+6. Read the `WARNING:` lines (stderr, exit 0). Either address them in your methodology or
+   document them explicitly in the `caveats` array.
+7. Publish the JSON file in your repository alongside the benchmark write-up. The
+   machine-readable format lets future scripts cross-check version-specific claims.
+
+---
+
+## What this pattern is NOT
+
+- **Not a quality gate for your methodology.** A report can be schema-valid and still be
+  methodologically weak. The schema forces disclosure; it does not evaluate task choice,
+  rubric validity, or baseline skill. Compliance is necessary but not sufficient.
+
+- **Not a way to launder self-scored or n=2 comparisons.** `self-scored` and
+  `sample_size: 2` produce a valid document. The warnings are loud. Schema compliance
+  means the claim is honest about its weaknesses — not that the claim is strong.
+
+- **Not a substitute for peer review.** The schema is the minimum bar for internal
+  credibility. Peer review catches confounded variables, cherry-picked tasks, post-hoc
+  metric selection — problems the schema cannot see.
+
+- **Not frozen.** A benchmark declaring `ars_version: 3.3.5` is always checked against
+  the v3.3.5 schema. Future versions may add fields without invalidating existing reports.
+
+---
+
+## Honesty red lines
+
+These are behaviors the schema is specifically designed to make visible. They are not
+prohibited — the schema accepts all of them — but they will appear in validator output
+or be apparent to any reader of the JSON.
+
+- **Omitting sample size.** The field is required. There is no valid ARS benchmark report
+  without a stated `human_baseline.sample_size`. n=1 is worse than n=10, but it is
+  infinitely better than unstated.
+
+- **Author-conducted with no caveat disclosure.** Using `author-conducted` is allowed.
+  Not noting it in `caveats` as a limitation is an editorial failure, not a schema
+  failure — but reviewers will notice.
+
+- **Self-scoring with no warning acknowledgment.** `self-scored` produces a warning on
+  every validator run. If you publish a report with `self-scored` and don't address it
+  in caveats, the omission is visible in the JSON to anyone who checks.
+
+- **Empty or single-word caveats.** The schema requires `minLength: 1` per item and
+  `minItems: 1` for the array. A caveat of `"none"` is technically valid but signals
+  that the author did not engage seriously with limitations.
+
+- **Claiming ARS advantage without disclosing cost_usd.** The field is required. A claim
+  that "ARS completed this in 4 hours vs 40 human hours" without a dollar figure omits
+  the dimension where the trade-off may be reversed.
+
+---
+
+## Future evolution
+
+Per-skill benchmark templates (a `deep-research`-specific template, an
+`academic-paper-reviewer`-specific template) are likely as v3.4 adds domain-specific
+evaluation protocols. The `cost_usd` field may split into `cost_usd_api` and
+`cost_usd_compute` once cloud execution costs become a factor. The
+`data_access_level_declared` enum may expand to match the v3.4 ground-truth tier
+vocabulary if that spec evolves. None of these changes will break existing v3.3.5
+reports, which remain valid against the version they declared in `ars_version`.