rememberstack 0.1.0__tar.gz

rememberstack-0.1.0/.agents/skills/eval-banana/SKILL.md

@@ -0,0 +1,226 @@
+---
+name: eval-banana
+description: Guide for using eval-banana, a lightweight aspect-based evaluation framework with YAML check definitions. Use when the user is working in a project with `eval_checks/` directories, wants to write or debug YAML eval definitions, needs to score LLM outputs or workflow behavior with pass/fail checks, is running `eval-banana` / `eb` CLI commands, or is setting up eval-banana in a new project. Covers deterministic checks, harness_judge checks, auto-discovery rules, context.json contract, config precedence, and report interpretation.
+---
+
+# eval-banana
+
+## Overview
+
+eval-banana is a lightweight evaluation framework. Check definitions live in YAML files under `eval_checks/` directories and are auto-discovered. Each check scores 0 or 1 (pass/fail) with equal weight. Two check types cover most needs. One YAML file per check — there is no suite wrapper.
+
+## The two check types
+
+| Type | Use when | Mechanism |
+|---|---|---|
+| `deterministic` | Asserting file content, structure, or values objectively | Python script via subprocess; exit 0 = pass, non-zero = fail |
+| `harness_judge` | Evaluating qualitative properties (coherence, tone, factuality) | Harness agent returns `{"score": 0\|1, "reason": "..."}` |
+
+**Default to `deterministic`** when the condition can be checked with code — it's the cheapest, most reliable, and requires no credentials.
+
+## Core workflow
+
+1. **Install** (once per machine): `uv tool install git+https://github.com/writeitai/eval-banana.git`
+2. **Initialize in project**: `eval-banana init` — creates `.eval-banana/config.toml`
+3. **Write checks**: add `*.yaml` files to any `eval_checks/` directory (they are auto-discovered from the project root)
+4. **Run**: `eval-banana run`
+5. **Read the report**: look under `.eval-banana/results/<run_id>/`
+
+## Common YAML fields
+
+Every check file starts with these fields regardless of type:
+
+```yaml
+schema_version: 1            # Always 1. Required. No default.
+id: my_check_id              # Unique across the project. Pattern: [a-zA-Z0-9_-]+
+type: deterministic          # One of: deterministic, harness_judge
+description: Human-readable  # Required. Non-empty.
+tags: [fast, critical]       # Optional list of free-form tags.
+```
+
+Plus type-specific fields below.
+
+## Writing a `deterministic` check
+
+Runs a Python script via subprocess. Exit code 0 = pass, non-zero = fail. Infrastructure problems (missing script file, OS execution failure) = error.
+
+```yaml
+schema_version: 1
+id: output_has_result_key
+type: deterministic
+description: output.json exists and contains a 'result' key.
+script: |
+  import json, sys
+  from pathlib import Path
+
+  ctx = json.loads(Path(sys.argv[1]).read_text())
+  project_root = Path(ctx["project_root"])
+  output = project_root / "output.json"
+  if not output.exists():
+      sys.exit(1)
+  data = json.loads(output.read_text())
+  if "result" not in data:
+      sys.exit(1)
+```
+
+Use `script: |` for inline Python, or `script_path: my_script.py` for an external script. The path is resolved **relative to the YAML file's directory**. Exactly one of `script` or `script_path` must be set.
+
+### The `context.json` contract (critical!)
+
+The script is invoked as `python <script> <context.json>`. Read `sys.argv[1]` to get the context path. It always has this exact shape:
+
+```json
+{
+  "check_id": "output_has_result_key",
+  "description": "output.json exists and contains a 'result' key.",
+  "project_root": "/abs/path/to/project",
+  "source_path": "/abs/path/to/project/eval_checks/my_check.yaml",
+  "output_dir": "/abs/path/.../.eval-banana/results/<run_id>/checks/output_has_result_key-<sha256-of-exact-id>"
+}
+```
+
+Key points:
+- `project_root` is absolute — use it to locate any file in the project.
+- The subprocess runs with `cwd = project_root`, so relative paths in the script also resolve from there.
+- `output_dir` is the durable directory for evidence produced by this check.
+  Its final component is the same safe stem used for result, prompt, stdout,
+  and stderr artifacts: a readable label of at most 40 characters plus the
+  full SHA-256 of the exact check ID. Never reconstruct it from `check_id`.
+
+### Deterministic failure mapping
+
+- `sys.exit(0)` or falling off the end → passed
+- `sys.exit(1)` or any non-zero exit → failed
+- `AssertionError` or any uncaught exception → failed (non-zero exit)
+- `FileNotFoundError` on the script itself → error
+
+## Writing a `harness_judge` check
+
+Invokes the configured harness agent with instructions. The agent can read files on its own. It must eventually emit JSON: `{"score": 0|1, "reason": "one sentence"}`.
+
+```yaml
+schema_version: 1
+id: readme_explains_install
+type: harness_judge
+description: README gives a new user enough info to install the package.
+instructions: |
+  Read README.md. Does it give a new user enough information to
+  install and run the package locally (environment setup, install
+  command, and how to invoke it)? Score 1 if yes, 0 if anything
+  critical is missing.
+```
+
+Guidelines for good instructions:
+- State the exact condition for score 1 and score 0.
+- Be binary — avoid "mostly", "partially", etc.
+- Reference concrete things to look for.
+- Tell the agent which files to read.
+- Keep it short. Long instructions confuse the judge.
+- Do not ask for scores outside {0, 1} — the parser rejects anything else as `error`.
+
+Optional fields:
+- `model: gpt-5.6-sol` — override the default harness model for this one check
+  (codex tiers: `gpt-5.6-sol`, `gpt-5.6-terra`, `gpt-5.6-luna`).
+
+## Auto-discovery rules
+
+- eval-banana walks from the project root (the directory containing `.eval-banana/`).
+- It finds every directory named exactly `eval_checks/`.
+- Inside each, it loads every `*.yaml` and `*.yml` file.
+- Check IDs must be unique across all discovered files. Duplicates are a **fatal load error**.
+- These directories are skipped by default: `.git`, `.hg`, `.svn`, `.venv`, `venv`, `node_modules`, `__pycache__`, `dist`, `build`.
+- Symlinked directories are not followed.
+
+Co-locate checks with the code they verify:
+
+```
+project/
+├── eval_checks/                     # Top-level checks
+│   └── overall_quality.yaml
+├── src/api/
+│   └── eval_checks/                 # API-specific checks
+│       └── response_schema.yaml
+└── frontend/
+    └── eval_checks/                 # Frontend E2E checks
+        └── login_flow.yaml
+```
+
+## Running checks
+
+```bash
+eval-banana run                        # Run everything
+eval-banana run --check-id my_check    # Run one check; relaxed validation of siblings
+eval-banana run --check-dir path/      # Only scan this directory
+eval-banana run --verbose              # Debug logging
+eval-banana run --pass-threshold 0.8   # Override pass ratio
+eval-banana list                       # Discover + print checks without running
+eval-banana validate                   # Validate YAML without executing anything
+eval-banana init [--force]  # Create project config
+```
+
+**`--check-id` is the debug escape hatch.** It uses relaxed validation — broken YAML in other files does NOT block a single targeted check. Use it when iterating on one check in a repo with incomplete checks elsewhere.
+
+## Reading results
+
+Each run writes to `.eval-banana/results/<run_id>/`:
+
+```
+<run_id>/
+├── report.json                # Machine-readable, full EvalReport
+├── report.md                  # Human-readable summary
+└── checks/
+    ├── <safe_check_id_stem>/            # Deterministic-check evidence directory
+    ├── <safe_check_id_stem>.json        # Per-check CheckResult
+    ├── <safe_check_id_stem>.prompt.txt  # Exact harness-judge input
+    ├── <safe_check_id_stem>.stdout.txt  # Captured stdout (only if non-empty)
+    └── <safe_check_id_stem>.stderr.txt  # Captured stderr (only if non-empty)
+```
+
+`<safe_check_id_stem>` is a bounded readable label plus the full SHA-256 of the
+exact check ID. All artifacts for one check share it, while IDs that differ only
+by case or normalization remain distinct. The evidence directory is created for
+deterministic checks; the prompt file is created for harness-judge checks.
+
+The console output shows:
+- Run ID
+- `points_earned/total_points` and percentage
+- PASS or FAIL verdict
+- Per-check list with reason (for failed) or error (for errored)
+
+**Pass criteria**: `run_passed = (points_earned / total_points) >= pass_threshold AND errored_checks == 0`. A single erroring check means the whole run fails, even if every other check passed.
+
+Exit code: 0 on `run_passed`, 1 otherwise. Usable directly in CI.
+
+## When to use which check type
+
+- **`deterministic`** — the condition is objective and testable with code:
+  - "file X exists and is non-empty"
+  - "the JSON has field Y"
+  - "no TODO comments in src/"
+  - "the CSV has N rows with these columns"
+
+- **`harness_judge`** — the condition is subjective or needs language understanding:
+  - "the error message is helpful to end users"
+  - "the generated summary captures the key points"
+  - "the tone is professional and friendly"
+  - "the docs explain the concept clearly"
+
+**If in doubt, prefer `deterministic`** — cheapest, most reliable, no credentials needed.
+
+## Common gotchas
+
+- **Forgetting `schema_version: 1`** — it has no default and omitting it is a validation error.
+- **Using `script:` AND `script_path:`** — exactly one must be set, never both.
+- **Putting a shell string in `command:`** — must be a list, e.g. `["pytest", "-q"]`, never `"pytest -q"`.
+- **Duplicate IDs across files** — fatal. Grep for the ID before adding a new check.
+- **Expecting judge checks to work without a harness** — configure `[harness] agent` or pass `--harness-agent`.
+- **Assuming `cwd` is where you invoked `eval-banana`** — deterministic scripts run with `cwd = project_root`.
+- **LLM judge returns prose instead of JSON** — the runner requires strict `{"score": 0|1, "reason": "..."}`. Tell the LLM to respond with JSON only.
+
+## References
+
+For deeper detail, read these as needed:
+
+- **`references/yaml-schema.md`** — Every field for every check type, validation rules, and edge cases. Read when writing a check with unusual requirements or debugging a validation error.
+- **`references/examples.md`** — Gallery of real-world check patterns (JSON validation, test runners, linters, LLM tone checks, UI flows). Read when looking for a template matching the current use case.
+- **`references/config.md`** — Full TOML config reference, precedence rules, harness settings, and environment variable list. Read when configuring eval-banana for a new project.

rememberstack-0.1.0/.agents/skills/eval-banana/references/config.md

@@ -0,0 +1,109 @@
+# Configuration reference
+
+Complete reference for eval-banana configuration: TOML layout, precedence rules, environment variables, and harness_judge setup.
+
+## Table of contents
+
+- File locations
+- Config sections
+- Precedence rules
+- Environment variables
+- Harness setup
+- Common config mistakes
+
+## File location
+
+Config lives at `.eval-banana/config.toml` in the project directory. It is found by walking upward from the current directory, so `eval-banana` works from any subdirectory.
+
+```bash
+eval-banana init          # Create project config
+eval-banana init --force  # Overwrite existing
+```
+
+## Config sections
+
+### `[core]` section
+
+| Key | Default | Description |
+|---|---|---|
+| `output_dir` | `.eval-banana/results` | Where run artifacts are written (relative to project root) |
+| `pass_threshold` | `1.0` | Minimum `points/total` ratio for the run to pass (0.0-1.0) |
+| `llm_max_input_chars` | `0` (disabled) | Max characters sent to `harness_judge` per target file; 0 = no limit |
+
+### `[harness]` section
+
+| Key | Default | Description |
+|---|---|---|
+| `agent` | unset | Built-in agent name such as `codex`, `claude`, or `gemini` |
+| `model` | unset | Override the template default model |
+| `reasoning_effort` | unset | Override the template reasoning effort |
+
+### `[discovery]` section
+
+| Key | Default | Description |
+|---|---|---|
+| `exclude_dirs` | `[".git", ".hg", ".svn", ".venv", "venv", "node_modules", "__pycache__", "dist", "build"]` | Directories to skip when walking for `eval_checks/` |
+
+Setting `exclude_dirs` replaces the built-in default list entirely, it does not append.
+
+## Precedence rules
+
+Config values are resolved in this order (highest priority first):
+
+1. **CLI arguments** (`--output-dir`, `--harness-model`, etc.)
+2. **`EVAL_BANANA_*` environment variables**
+3. **Project config** (`.eval-banana/config.toml`)
+4. **Built-in defaults**
+
+## Environment variables
+
+| Variable | Maps to |
+|---|---|
+| `EVAL_BANANA_OUTPUT_DIR` | `core.output_dir` |
+| `EVAL_BANANA_PASS_THRESHOLD` | `core.pass_threshold` |
+| `EVAL_BANANA_LLM_MAX_INPUT_CHARS` | `core.llm_max_input_chars` |
+| `EVAL_BANANA_HARNESS_AGENT` | `harness.agent` |
+| `EVAL_BANANA_HARNESS_MODEL` | `harness.model` |
+| `EVAL_BANANA_HARNESS_REASONING_EFFORT` | `harness.reasoning_effort` |
+
+## Harness setup
+
+Configure a harness agent when your project uses `harness_judge` checks.
+
+```toml
+[harness]
+agent = "codex"
+# codex GPT-5.6 tiers: gpt-5.6-sol (flagship, default), gpt-5.6-terra, gpt-5.6-luna
+model = "gpt-5.6-sol"
+reasoning_effort = "high"
+```
+
+If a project still contains a legacy `[llm]` section, eval-banana exits with a migration error instructing you to delete it and use `[harness]` / `[agents.*]`.
+
+## Generated config template
+
+Created by `eval-banana init` at `.eval-banana/config.toml`:
+
+```toml
+# Project-level eval-banana configuration.
+
+[core]
+output_dir = ".eval-banana/results"
+pass_threshold = 1.0
+llm_max_input_chars = 0
+
+[harness]
+agent = "codex"
+
+[discovery]
+exclude_dirs = [".git", ".hg", ".svn", ".venv", "venv", "node_modules", "__pycache__", "dist", "build"]
+```
+
+## Common config mistakes
+
+| Mistake | Symptom | Fix |
+|---|---|---|
+| `pass_threshold: 80` (integer) | All runs fail | Use `pass_threshold = 0.8` (float, 0.0-1.0) |
+| Relative `output_dir` resolved from wrong cwd | Results appear in unexpected locations | eval-banana always resolves from `project_root`, not `pwd` |
+| Leaving a stale `[llm]` section in config | eval-banana exits before loading config | Delete `[llm]` and configure `[harness]` / `[agents.*]` instead |
+| Replacing `exclude_dirs` with an incomplete list | `.git`, `.venv` get scanned | Lists replace, not merge — include all default entries |

rememberstack-0.1.0/.agents/skills/eval-banana/references/examples.md

@@ -0,0 +1,170 @@
+# Example check patterns
+
+Gallery of real-world eval-banana check patterns. Copy and adapt.
+
+## Table of contents
+
+- Deterministic: file existence and content
+- Deterministic: JSON schema validation
+- Deterministic: no forbidden tokens in source
+- Deterministic: counting records
+- Harness judge: README quality
+- Harness judge: tone / professionalism
+- Harness judge: factual consistency
+- Harness judge: multi-file comparison
+
+## Deterministic: file existence and content
+
+```yaml
+schema_version: 1
+id: changelog_exists_and_nonempty
+type: deterministic
+description: CHANGELOG.md exists and is at least 200 chars.
+script: |
+  import sys
+  from pathlib import Path
+
+  ctx_path = Path(sys.argv[1])
+  import json
+  ctx = json.loads(ctx_path.read_text())
+  changelog = Path(ctx["project_root"]) / "CHANGELOG.md"
+  if not changelog.exists():
+      print(f"missing: CHANGELOG.md", file=sys.stderr)
+      sys.exit(1)
+  content = changelog.read_text(encoding="utf-8")
+  if len(content) < 200:
+      print(f"only {len(content)} chars", file=sys.stderr)
+      sys.exit(1)
+```
+
+## Deterministic: JSON schema validation
+
+```yaml
+schema_version: 1
+id: output_matches_schema
+type: deterministic
+description: output.json has required top-level keys and correct types.
+script: |
+  import json, sys
+  from pathlib import Path
+
+  ctx = json.loads(Path(sys.argv[1]).read_text())
+  data = json.loads((Path(ctx["project_root"]) / "output.json").read_text())
+
+  required = {"id": str, "created_at": str, "items": list}
+  for key, expected_type in required.items():
+      if key not in data:
+          print(f"missing key: {key}", file=sys.stderr)
+          sys.exit(1)
+      if not isinstance(data[key], expected_type):
+          print(f"{key} should be {expected_type.__name__}", file=sys.stderr)
+          sys.exit(1)
+  if not data["items"]:
+      print("items is empty", file=sys.stderr)
+      sys.exit(1)
+```
+
+## Deterministic: no forbidden tokens in source
+
+```yaml
+schema_version: 1
+id: no_print_statements_in_src
+type: deterministic
+description: No print() calls in src/ (should use logging).
+script: |
+  import json, re, sys
+  from pathlib import Path
+
+  ctx = json.loads(Path(sys.argv[1]).read_text())
+  src_dir = Path(ctx["project_root"]) / "src"
+  pattern = re.compile(r"\bprint\(")
+  offenders = []
+  for py_file in src_dir.rglob("*.py"):
+      text = py_file.read_text(encoding="utf-8")
+      for line_no, line in enumerate(text.splitlines(), 1):
+          if line.strip().startswith("#"):
+              continue
+          if pattern.search(line):
+              offenders.append(f"{py_file}:{line_no}")
+  if offenders:
+      print("\n".join(offenders[:10]), file=sys.stderr)
+      sys.exit(1)
+```
+
+## Deterministic: counting records
+
+```yaml
+schema_version: 1
+id: users_csv_has_enough_rows
+type: deterministic
+description: users.csv has at least 100 rows (excluding header).
+script: |
+  import csv, json, sys
+  from pathlib import Path
+
+  ctx = json.loads(Path(sys.argv[1]).read_text())
+  path = Path(ctx["project_root"]) / "data" / "users.csv"
+  with path.open() as f:
+      reader = csv.reader(f)
+      next(reader, None)  # Skip header
+      row_count = sum(1 for _ in reader)
+  if row_count < 100:
+      print(f"only {row_count} rows", file=sys.stderr)
+      sys.exit(1)
+```
+
+## Harness judge: README quality
+
+```yaml
+schema_version: 1
+id: readme_has_quickstart
+type: harness_judge
+description: README contains a quickstart that a new user can follow in under 5 minutes.
+instructions: |
+  Read README.md. Look for a "Quick start" or "Getting started" section.
+  Score 1 ONLY if it contains: (1) an install command, (2) a minimum
+  config step if required, and (3) at least one example command to run.
+  Score 0 if any of these are missing or unclear.
+```
+
+## Harness judge: tone / professionalism
+
+```yaml
+schema_version: 1
+id: error_messages_are_helpful
+type: harness_judge
+description: Error messages in errors.log are helpful and professional.
+instructions: |
+  Read errors.log. Score 1 if the error messages: (a) explain what
+  went wrong in plain language, (b) suggest what to do next, and (c) do
+  NOT expose stack traces or internal paths to end users. Score 0 if
+  any message is cryptic, blames the user, or leaks internals.
+```
+
+## Harness judge: factual consistency
+
+```yaml
+schema_version: 1
+id: summary_matches_source
+type: harness_judge
+description: The generated summary accurately reflects the source data.
+instructions: |
+  Read summary.md and source_data.json. Compare the claims in the summary
+  against the facts in the source data. Score 1 if every numeric claim,
+  name, and date in the summary can be verified from the source data.
+  Score 0 if ANY claim is fabricated, hallucinated, or contradicts the source.
+```
+
+## Harness judge: multi-file comparison
+
+```yaml
+schema_version: 1
+id: docs_consistent_with_code
+type: harness_judge
+description: API docs describe the actual endpoints implemented in routes.py.
+instructions: |
+  Read docs/api.md and src/routes.py. For each endpoint documented in the
+  API docs, check if it exists in routes.py. Score 1 if every documented
+  endpoint is implemented AND every public endpoint in routes.py is
+  documented. Score 0 if there is any drift between the two files.
+```

rememberstack-0.1.0/.agents/skills/eval-banana/references/yaml-schema.md

@@ -0,0 +1,146 @@
+# YAML schema reference
+
+Complete field reference for eval-banana check definitions. Each check file defines a single check — there is no suite wrapper.
+
+## Table of contents
+
+- Common fields (all check types)
+- `deterministic` fields
+- `harness_judge` fields
+- Validation rules
+- Error messages and what they mean
+
+## Common fields
+
+| Field | Type | Required | Constraints | Notes |
+|---|---|---|---|---|
+| `schema_version` | int | **Yes** | Must equal `1` | No default. Omitting → validation error. |
+| `id` | string | **Yes** | Pattern `^[a-zA-Z0-9_-]+$`, non-empty after stripping | Must be unique across ALL discovered check files |
+| `type` | string | **Yes** | One of `deterministic`, `harness_judge` | Discriminator for the Pydantic union |
+| `description` | string | **Yes** | Non-empty after stripping | Human-readable, shown in reports |
+| `tags` | list[string] | No | — | Free-form metadata. Not yet used for filtering but allowed. |
+
+`extra="forbid"` is enabled — any unknown field fails validation.
+
+## `deterministic` check
+
+Type-specific fields:
+
+| Field | Type | Required | Notes |
+|---|---|---|---|
+| `script` | string | **One of** | Inline Python source code (use `script: \|` block scalar) |
+| `script_path` | string | **One of** | Path to external Python file, **relative to the YAML file's directory** |
+
+**Exactly one** of `script` or `script_path` must be set. Setting both is a validation error. Setting neither is a validation error.
+
+### Subprocess contract
+
+- Command: `python <script> <context_path>`
+- `cwd`: `project_root`
+- Environment: full parent env (no additional injection)
+
+### `context.json` shape
+
+Passed as `sys.argv[1]`. Always this exact shape:
+
+```json
+{
+  "check_id": "string",
+  "description": "string",
+  "project_root": "/abs/path",
+  "source_path": "/abs/path/to/check.yaml",
+  "output_dir": "/abs/path/to/checks/<safe_check_id_stem>"
+}
+```
+
+`output_dir` is an absolute, durable directory where the deterministic script
+may write evidence. `<safe_check_id_stem>` is a readable label of at most 40
+characters followed by the full SHA-256 of the exact check ID. Use the supplied
+path directly rather than deriving a directory from `check_id`; the same stem
+binds the check's evidence, result, prompt, stdout, and stderr artifacts while
+keeping case-only, normalized-label, and long IDs distinct.
+
+### Result mapping
+
+| Outcome | Status | Score |
+|---|---|---|
+| Exit 0 | `passed` | 1 |
+| Exit non-zero (includes `AssertionError`, uncaught exceptions, `sys.exit(1)`) | `failed` | 0 |
+| `FileNotFoundError` on script itself, `OSError` | `error` | 0 |
+
+`stdout` and `stderr` are captured on the `CheckResult` and written to
+`<run_output_dir>/checks/<safe_check_id_stem>.stdout.txt` and `.stderr.txt`
+(only if non-empty). The per-check result is
+`<safe_check_id_stem>.json`.
+
+## `harness_judge` check
+
+Type-specific fields:
+
+| Field | Type | Required | Notes |
+|---|---|---|---|
+| `instructions` | string | **Yes** | Non-empty. The evaluation prompt sent to the harness agent. |
+| `model` | string | No | Override `harness.model` for this check only |
+
+### Prompt shape
+
+The runner builds a prompt with:
+1. A fixed instruction asking for `{"score": 0|1, "reason": "..."}` JSON output
+2. The `description` as context
+3. The `instructions` as the evaluation criterion
+
+The harness agent can read project files on its own — tell it which files to check in the `instructions` field.
+
+The exact prompt is retained as
+`<run_output_dir>/checks/<safe_check_id_stem>.prompt.txt`. It shares its stem
+with the check result and any captured stdout or stderr.
+
+### Required LLM response format
+
+```json
+{"score": 0, "reason": "one sentence explanation"}
+```
+
+- `score` MUST be exactly `0` or `1`. Any other value → `error` result.
+- `reason` is optional but recommended. If present, must be a string.
+- Response must be valid JSON. Prose or malformed JSON → `error` result.
+
+### Result mapping
+
+| Outcome | Status | Score |
+|---|---|---|
+| Valid JSON, `score == 1` | `passed` | 1 |
+| Valid JSON, `score == 0` | `failed` | 0 |
+| Malformed JSON or score outside {0,1} | `error` | 0 |
+| Harness subprocess spawn/timeout error | `error` | 0 |
+
+## Validation rules summary
+
+The loader raises a `ValueError` naming the file path for any of these:
+
+- YAML parse error
+- Top-level YAML is not a dict
+- Any required field missing
+- `id` doesn't match `^[a-zA-Z0-9_-]+$`
+- `description` empty or whitespace-only
+- Unknown top-level field (blocked by `extra="forbid"`)
+- `type` not one of the allowed values
+- `script` AND `script_path` both set, or neither set (deterministic)
+- `instructions` empty (harness_judge)
+
+The runner raises `SystemExit` for:
+- Duplicate check IDs across files (shows both file paths)
+- No checks found after discovery + filtering
+- `--check-id` matches multiple files (also shows paths)
+
+## Common validation errors
+
+| Error text | Cause | Fix |
+|---|---|---|
+| `Field required [type=missing]` on `schema_version` | Forgot the field | Add `schema_version: 1` |
+| `Extra inputs are not permitted` | Unknown field | Remove or check spelling |
+| `script and script_path are mutually exclusive` | Both set | Remove one |
+| `deterministic check must have script or script_path` | Neither set | Add one |
+| `instructions must be non-empty` | Empty or missing on harness_judge | Add instructions |
+| `id does not match pattern` | Invalid chars (dots, spaces, etc.) | Use only `[a-zA-Z0-9_-]` |
+| `Duplicate check id 'X' found in: ...` | Same id in 2+ files | Rename one |