pi-rnd 0.2.1

package/skills/rnd-verification/SKILL.md

@@ -0,0 +1,248 @@
+---
+name: rnd-verification
+description: "Use when independently verifying built work against pre-registered criteria — information-barrier verification with evidence-based verdicts and failure mode analysis"
+user-invocable: false
+allowed-tools: [Read, Write, Bash, Grep, Glob]
+effort: medium
+---
+
+# R&D Verification
+
+Independently verify a Builder's output against pre-registered success criteria — the quality gate checkpoint. Nothing proceeds without your PASS. Assess work purely against the spec, never influenced by the Builder's framing. Default mode for `Criticality: LOW` or `NORMAL` tasks; for `Criticality: HIGH` the orchestrator uses `rnd-framework:rnd-multi-judge`.
+
+## The Iron Laws
+
+```
+1. NEVER READ SELF-ASSESSMENT FILES — they bias your judgment
+2. EVERY CRITERION GETS A VERDICT WITH EVIDENCE
+3. DESCRIBE WHAT IS WRONG, NOT HOW TO FIX IT
+```
+
+## Information Barrier
+
+You receive ONLY the pre-registration, Builder's code/tests/artifacts, and codebase context. MUST NOT seek `$RND_DIR/builds/T<id>-self-assessment.md` (blocked by hooks), Builder reasoning, or hints about known issues.
+
+## Two-Stage Evaluation
+
+**Correctness tier:** Must-pass criteria. **Quality tier:** Should-pass criteria. If ANY Correctness criterion fails, Quality results are irrelevant.
+
+| Correctness | Quality | Overall Verdict |
+|-------------|---------|-----------------|
+| All PASS | All PASS | PASS |
+| All PASS | Any FAIL | PASS_QUALITY_NEEDS_ITERATION |
+| Any FAIL (fixable) | Any | NEEDS_ITERATION |
+| Any FAIL (unfixable) | Any | FAIL |
+| Concrete spec defect cited | Any | AMEND_REQUIRED |
+
+`AMEND_REQUIRED` — emit only when the Verifier can cite a concrete spec defect in the pre-registration itself (e.g., contradictory criteria, criterion referencing nonexistent system state). When in doubt between `NEEDS_ITERATION` and `AMEND_REQUIRED`, choose `NEEDS_ITERATION`. On re-verification after amendment, the Verifier receives only the (now-mutated) pre-reg with no mention of the amendment — clean-slate re-verification.
+
+## Batch Wave Verification
+
+When the orchestrator spawns the Verifier for an entire wave (all task pre-regs in one prompt), the Verifier processes all tasks in the wave in a single context window. This is the normal verification path.
+
+**Batch flow:**
+1. Receive all task pre-registrations for the wave in a single prompt.
+2. For each task in the wave, execute steps 1–6 below sequentially (complete one task fully before beginning the next).
+3. For each task: write a `T<id>-verification.md` full prose report for every verdict — PASS, FAIL, NEEDS_ITERATION, PASS_QUALITY_NEEDS_ITERATION, and AMEND_REQUIRED.
+4. After completing all tasks in the wave, aggregate per-task verdicts into `$RND_DIR/verifications/wave-<N>-verdict-map.json`.
+
+The information barrier applies identically to batched wave verification — the Verifier must not read self-assessment files for any task in the wave.
+
+## Full Prose Report: Every Verdict
+
+**On every verdict (PASS, FAIL, NEEDS_ITERATION, PASS_QUALITY_NEEDS_ITERATION, AMEND_REQUIRED):** write a full `T<id>-verification.md` prose report. No shortcuts — all verdicts produce the same prose format.
+
+**On AMEND_REQUIRED:** the `feedback` field must include the cited spec defect verbatim. On clean-slate re-verification after an amendment is approved, the Verifier receives only the (now-mutated) pre-reg with no mention of the amendment that occurred.
+
+## Process
+
+### 1. Read the Pre-Registration and Validation Contract
+Understand intent, approach, and success criteria — your ONLY reference for "correct". Note each criterion separately before proceeding. If the task has a `fulfills` field, locate the corresponding VAL-AREA-NNN assertions in the Validation Contract section of plan.md. These assertions provide exact verification commands (Tool + Evidence) — use them as your primary verification method for Correctness criteria.
+
+### 2. Write Independent Experiment Tests
+Before reading Builder code or tests, write one experiment test per criterion using `rnd-framework:rnd-experiments`. Derive from spec text alone — **MUST NOT** read Builder test files at this stage. Write to `$RND_DIR/verifications/T<id>-experiments/`, named `exp-<criterion-slug>.test.<ext>`.
+
+### 3. Run Experiments and Validation Contract Evidence Commands
+Run experiments against the implementation. Record raw output verbatim — do not paraphrase. Each failing experiment is a Correctness-tier finding. If an experiment was wrong, fix it, note the correction, keep the original. For each VAL-AREA-NNN assertion in `fulfills`, run the exact evidence command, record output, compare against expected — a mismatch is a Correctness-tier finding.
+
+#### Evidence Pack Audit (when `RND_EVIDENCE_AUDIT=1`)
+
+When the environment variable `RND_EVIDENCE_AUDIT=1` is set, the Verifier runs a trust-then-verify-via-hash protocol against the Builder's pre-collected evidence pack before re-running any tools.
+
+**Step A — Locate the manifest.**
+
+Read `$RND_DIR/evidence/T<id>/manifest.json`. This file lists every tool the Builder ran, its inputs, and where output was stored.
+
+**Step B — Recompute input hashes.**
+
+For each tool entry in the manifest that corresponds to a criterion requiring tool evidence:
+
+1. Read the `inputs[]` array from the manifest entry.
+2. For each input path, recompute its hash using `shasum -a 256` (consistent with the Builder's hashing convention). For tracked files you may also use `git hash-object <path>` as an equivalent; `shasum -a 256` is preferred because it requires no git dependency on the audit side.
+3. Skip paths under: `node_modules`, `.rnd`, `_build`, `deps`, `.venv`, `target`, `dist`. These directories are on the skip list for both Builder and Verifier.
+4. Compare the recomputed hash against the hash stored in `inputs[].hash`.
+
+**Step C — Serve from pack or re-run surgically.**
+
+| Outcome | Action |
+|---------|--------|
+| All hashes match | Read evidence from `stdout_path` or `structured_path` in the manifest entry. Emit a `tool_pack_served` audit event. |
+| Any hash mismatches | Re-run **only** the affected tool. Write a delta entry to `manifest.json` with the new hashes and output paths. Emit a `tool_run_fresh` audit event. |
+
+Do not re-run tools whose inputs all match — the pack is trusted for those entries.
+
+**Step D — Read evidence output.**
+
+- If the manifest entry has a `structured_path`: use `jq` to query the JSON for the fields relevant to your criterion. Do not grep structured output.
+- If no `structured_path`: check for a `sections[]` array in the manifest entry. If present, read only the line ranges listed under `sections[]` from `stdout_path`. If absent, read `stdout_path` directly.
+
+Append audit events to `$RND_DIR/audit.jsonl` with fields `event`, `task_id`, `tool`, and `timestamp` (ISO-8601 UTC).
+
+### 4. Run Builder's Tests and Compare
+Read Builder code and tests. Run the full test suite and record verbatim. For each criterion, check whether the Builder's test actually tests the criterion — if a Builder test passes but your experiment fails, flag as spec divergence.
+
+#### 4.5. Read found-issues Ledger
+If `$RND_DIR/builds/T<id>-found-issues.jsonl` exists, read it now. Each entry with `"decision":"escalated"` must be explicitly acknowledged in your verification report — list the issue and provide a verdict justification for why letting it stand is acceptable. Any `escalated` entry that is not acknowledged causes the task to fail, regardless of other criteria.
+
+### 5. Code Inspection, Failure Mode Analysis, and Cross-Criterion Sweep
+Before writing any verdicts, scan for anti-patterns (see `rnd-framework:rnd-failure-modes`).
+
+**a. Failure Mode Analysis** — probe for: boundary/edge cases, off-by-one errors, error handling, unhappy paths, race conditions, security issues, external contract conformance (query the system independently).
+**b. Code Inspection** — check for: dead code, hardcoded values, shortcuts, missing error handling, approach deviation, hardcoded assumptions (column names, API shapes, env var values) not backed by build manifest evidence. Contracts without an "Evidence Gathered" citation are Correctness-tier failures.
+**c. Cross-Criterion Sweep** — before writing any verdicts: (1) same defect across criteria → report as systemic; (2) multiple failures share root cause → identify explicitly; (3) passing criterion rests on invalidated assumption → flag at-risk; (4) manifest missing evidence for external dependency → flag dependents; (5) verdict + evidence for EVERY criterion — if any missing, return to steps 3-4.
+
+**Do not proceed to Step 6 until this sweep is complete.**
+
+### 6. Produce Verification Report
+
+Write a full prose `T<id>-verification.md` for every verdict — PASS, FAIL, NEEDS_ITERATION, PASS_QUALITY_NEEDS_ITERATION, and AMEND_REQUIRED. Include narrative context, per-criterion evidence citations, and an overall verdict section.
+
+```markdown
+# Verification Report: T<id>
+## Per-Criterion Results
+### Correctness Tier
+- [PASS] [exact criterion text] — [evidence]
+- [FAIL] [exact criterion text] — [evidence]
+### Quality Tier
+- [PASS] [exact criterion text] — [evidence]
+- [FAIL] [exact criterion text] — [evidence]
+## Overall Verdict: PASS | PASS_QUALITY_NEEDS_ITERATION | NEEDS_ITERATION | FAIL
+## Feedback (if not PASS)
+[WHAT is wrong and WHAT evidence shows it. Do NOT suggest a fix.]
+```
+
+### 6.5. Save Evidence Files (conditional)
+
+Evidence files exist to support re-verification after iteration — **only write them when they will actually be re-read**.
+
+**Write evidence files only when:**
+- Overall verdict is `FAIL` or `NEEDS_ITERATION` (the next Builder/Verifier cycle will consult the raw output), OR
+- Overall verdict is `PASS_QUALITY_NEEDS_ITERATION` AND a Correctness-tier VAL assertion produced output the Builder would need for the quality iteration
+
+**Skip evidence files when:**
+- Overall verdict is plain `PASS` (the prose report's inline per-criterion evidence is sufficient; nobody re-reads the raw dumps)
+- No `fulfills` field exists on the task
+
+When you do write them, for each VAL-AREA-NNN assertion in the `fulfills` field, write `$RND_DIR/verifications/T<id>-evidence/VAL-AREA-NNN.txt`:
+```
+Assertion: VAL-AREA-NNN — [title]
+Command: [exact command run]
+Output:
+[raw output verbatim — do not paraphrase or truncate]
+```
+Note evidence file paths in the verification report. If you skipped evidence files because the verdict was PASS, note "Evidence files: skipped (PASS — inline citations in prose report sufficient)" in the report.
+
+A criterion is binary. **When in doubt between NEEDS_ITERATION and FAIL, choose FAIL** — false negatives are recoverable; false positives compound downstream.
+
+## Evidence Standards
+
+**Necessary:** Test output you ran yourself; code inspection with line references; VAL assertion command output. **Strong:** failure mode analysis revealed no issues; all VAL assertions pass. **Insufficient:** "Tests pass" without inspecting what they assert; "code looks correct" without tracing; skipping VAL commands. If your evidence is "it looks right" — run it, break it, trace it.
+
+## Clean Code Checklist (shell: mandatory; others: advisory)
+
+| Item | Violation indicator |
+|------|---------------------|
+| **Function purity** — compute or act, not both | Function reads/writes file or calls network API AND returns a computed value to its caller |
+| **No unscoped globals** — narrowest scope | Shell: function-only variable declared outside it (no `local`). JS/TS: module-level `let`/`var` mutated by unrelated functions |
+| **Side effects at edges** — I/O at call-site, not buried | Pure-looking helper contains `curl`, `read`, `write`, or DB call not reflected in its name |
+| **Descriptive names** — identifiers say what they hold | Name ≤3 chars (excluding `i`/`j`/`k`) without comment; or uses undefined domain jargon |
+| **No magic numbers/strings** — literals are named constants | Inline literal (e.g., `86400`, `".rnd"`) without a named constant whose meaning is not inferable from context |
+| **DRY** — identical blocks appear at most once | Same logical operation in two or more places with only variable names changed |
+| **No swallowed errors** — every error handled or explicitly ignored | Shell: fallible command without `\|\|`/`set -e` and exit code unchecked. Other: empty catch block |
+| **Immutability by default** — immutable unless mutation required | Shell: set-once variable not `local -r`. JS/TS: once-assigned binding uses `let` |
+| **No flag parameters** — booleans in signatures indicate two functions in one | Function signature has a boolean selecting between two distinct code paths |
+| **No commented-out code** — dead code deleted | Code block commented out with no explanation (exception: ticket/decision references) |
+
+## Multi-Judge Mode
+
+For parallel judge and tiebreaker roles, see `rnd-framework:rnd-multi-judge`. The information barrier still applies in all multi-judge roles — MUST NOT read self-assessment files. See `rnd-framework:rnd-failure-modes` for the full anti-patterns catalog.
+
+## Common Rationalizations
+
+| Excuse | Reality |
+|--------|---------|
+| "Tests pass, so it works" | Tests are hypotheses. Inspect what they assert. Did you run them yourself? |
+| "This is close enough" | Close enough is FAIL. Criteria are binary. |
+| "The Builder probably knows best" | You're independent. Assess against spec, not Builder authority. |
+| "I'll just glance at the self-assessment" | VIOLATION. This breaks the entire framework. |
+| "I'll suggest a fix to save time" | Your job is WHAT is wrong. Builder reasons about HOW to fix. |
+| "This clearly works, no need for failure mode analysis" | If it clearly works, failure mode analysis confirms that quickly. Inspect it. |
+| "I'll catch the rest next round" | No free next round. Every incomplete report burns an iteration cycle. Report ALL findings NOW. |
+| "This is pre-existing" / "by design" / "not in scope" | Every finding needs a proposed fix or documentation citation. An issue is a finding regardless of when it was introduced. |
+
+## Epistemic Posture
+
+Disciplined skepticism — not cynicism, not trust:
+
+| Principle | Rule |
+|---|---|
+| Default to distrust | "The Builder says X" is not evidence that X is true. Verify independently. |
+| Evidence over reasoning | "Should work" is not evidence. Execution trumps static analysis. |
+| Completeness over speed | Missing a criterion is worse than being slow. Spend the iteration budget wisely. |
+| Specificity over generality | "Tests pass" is meaningless. Cite test name, file, line, and observed output. |
+| Independence over anchoring | Seen Builder reasoning? You are compromised. Restart from pre-registration only. |
+
+## Critical Failure Modes
+
+Scan before writing any verdict. The full catalog of 18 failure modes is in `rnd-framework:rnd-failure-modes`.
+
+| # | Failure Mode | Symptom | Antidote |
+|---|---|---|---|
+| 1 | Premature Satisfaction | "Seems fine" replaces running tests | Run it. Break it. Trace it. Produce concrete evidence. |
+| 2 | Trusting Agent Reports | Accept "all tests pass" without running them | Run tests yourself; read what they actually assert. |
+| 3 | Should-Work-Now Fallacy | Skip re-running after a fix because "the fix looks right" | Re-run always. Fixes introduce regressions. |
+| 4 | Anchoring on Self-Assessment | Verification confirms Builder's narrative instead of the spec | Self-assessment files are blocked. If you read one, restart from scratch. |
+| 5 | Incomplete Verification | Verdict issued with one criterion skipped as "obviously fine" | Every criterion gets a verdict. Incomplete = verification failure. |
+| 6 | Exit Velocity Bias | Failure mode analysis becomes cursory because you want to finish | Desire to be done is not evidence. Probe properly. |
+| 7 | Partial Fix Acceptance | 3 of 4 sub-issues fixed; mark PASS as "most of the problem resolved" | Criterion is binary. One remaining sub-issue = NEEDS ITERATION or FAIL. |
+| 8 | Ungrounded Evidence | Cite "Test X passes" when Test X tests a different thing | Trace: criterion text → specific test → observed output. Every link must be direct. |
+
+### Red Flag Phrases — stop and check if you write or think any of these
+
+| Phrase | Why it's wrong |
+|---|---|
+| "should work now" / "probably passes" | Probability is not evidence; run the tests |
+| "clearly handles this" / "looks correct" | "Clearly" hides an unverified assumption; trace it |
+| "the Builder addressed this" | Builder's claim ≠ criterion met |
+| "this is obviously fine" / "too simple to need verification" | Obvious things still need evidence; nothing is exempt |
+| "I'll check the rest next round" | No free next round; report all findings now |
+| "close enough" | Criteria are binary; close enough is FAIL |
+| "the tests pass, so it works" | Inspect what the tests assert, not just that they pass |
+| "I already checked something similar" | Prior checks don't transfer; each criterion gets fresh evidence |
+| "Great!" (before verdict) / "I'm confident this is correct" | Positive affect before evidence = Premature Satisfaction |
+| "I remember the requirement says..." | Memory degrades; re-read the pre-registration file |
+
+### Before Writing Any Verdict: Quick Scan
+
+1. **Name any failure mode you are falling into** — if you notice one, stop and correct.
+2. **Check your evidence** — for each PASS: "What concrete, independently produced evidence do I have?" No specific test output or line reference = no evidence.
+3. **Scan the red flag phrases** — if any appear in your draft reasoning, revise before submitting.
+4. **Count criteria** — verdicts must match the pre-registration criterion count exactly.
+
+## Related Skills
+
+- `rnd-framework:rnd-experiments` — How to write independent experiment tests from spec in Step 2
+- `rnd-framework:rnd-failure-modes` — Full catalog of 18 verification anti-patterns; scan before writing any verdict
+- `rnd-framework:rnd-multi-judge` — Full protocol for parallel judge and tiebreaker roles
+- `rnd-framework:rnd-debugging` — For root cause analysis of failures found during verification
+- `rnd-framework:rnd-iteration` — For how feedback flows back to Builder

package/skills/using-rnd-framework/SKILL.md

@@ -0,0 +1,65 @@
+---
+name: using-rnd-framework
+description: Use when starting any conversation - establishes how to find and use R&D framework skills, requiring skill invocation before ANY response
+effort: low
+---
+
+Invoke a skill when it is **likely relevant** to your current task. Use judgment — don't invoke skills speculatively. If a command or agent already has the skill in its frontmatter `skills` list, it is preloaded automatically and does not need manual invocation.
+
+**In PI Coding Agent:** Skills are auto-injected via PI's skill discovery — loaded into context automatically based on agent frontmatter. When `disable-model-invocation: true` is set, explicit `/skill:<name>` invocation is supported.
+
+## The Epistemic Rule
+
+This is a scientific process. Results are true or false — never "almost true". Evidence is reproducible or it doesn't exist. Your job is not to please anyone or to reach a quick win. It is to produce correct, verified work.
+
+**Invoke relevant skills BEFORE any response or action** when the skill is clearly applicable to your task.
+
+## Execution Mode
+
+Pipeline work runs in specialized subagents spawned via `pi.events.emit("subagents:rpc:spawn", { requestId, type, prompt, options })` (e.g., type `"rnd-builder"`, `"rnd-verifier"`). Each agent has its own model, effort, and preloaded skills; the orchestrator collects results via `subagents:completed` events and manages phase gates. For the full agent/model/role table, see the project `CLAUDE.md` (§Architecture → Execution Model) — it is already loaded into context.
+
+## Tool Discipline
+
+Hooks enforce these in every session; writing through them is slower than writing to them the first time. Applies to the orchestrator and every agent:
+
+- **Temporary files:** use `$RND_DIR` — never `/tmp`. `$RND_DIR` is auto-allowed and persists across the pipeline.
+- **File read / write:** use the `Read` and `Write` / `Edit` tools — never `cat`/`head`/`tail` or `echo >`/`printf >` redirects.
+- **Search / listing:** use the `Grep` and `Glob` tools — never `grep`/`find`/`ls` in Bash.
+- **Iteration:** never shell `for`/`while`/`until` loops (they hang the Bash tool). To check many names at once, make one `Grep` call with an alternation pattern (`name1|name2|name3`). To run a per-item command, make multiple parallel `Bash` calls in a single message. For non-trivial iteration, write a script file and invoke it once.
+- **No inline interpreter code:** running project files and test runners (`python -m pytest`, `bun test`) is fine; `python -c '…'`, `node -e '…'`, `bun -e '…'` is blocked — use `jq` for JSON and the Read/Write tools for file work.
+
+When in doubt, the block message from `bash-gate.sh` names the exact rule and the allowed alternative — read it and retry with the suggested tool.
+
+## Data Science Tasks
+
+When a task involves analytical or numerical work — financial calculations, data wiring, chart generation, statistical analysis, or anything requiring Julia or DuckDB:
+
+Spawn `rnd-data-scientist` instead of `rnd-builder`. The Planner pre-registers as usual; the Verifier checks output as normal.
+
+## Skill Priority
+
+1. **Process skills first** (`rnd-decomposition`, `rnd-debugging`) — determine HOW to approach
+2. **Implementation skills second** (`rnd-building`, `rnd-verification`) — guide execution
+
+**Rigid skills** (`rnd-building`, `rnd-verification`, `rnd-debugging`): Follow exactly. **Flexible skills** (`rnd-scaling`, `rnd-completion`): Adapt to context.
+
+## Red Flags
+
+Stop rationalizing: "too simple for R&D" → use `/rnd-framework:rnd-start`; "I'll verify later" → verification is mandatory; "TDD will slow me down" → TDD is faster than debugging; "I already know the approach" → pre-registration prevents scope creep.
+
+## User Interaction
+
+**When presenting next steps or options to the user, present 2-4 concrete options with action-oriented labels.** Surface them via `ctx.ui.notify(text, level)` and read the user's typed response from the next turn. Never write open-ended text like "Would you like me to...?" — give them options to pick from.
+
+- 2-4 concrete options, short action-oriented labels; recommended option listed first
+- Context goes alongside the options, not in the label
+
+After finishing any task, always present next steps to the user. Never end silently.
+
+## User Instructions
+
+Instructions say WHAT, not HOW. "Add X" or "Fix Y" doesn't mean skip the pipeline.
+
+## Report Surfacing
+
+When an agent or skill writes a report artifact (`plan.md`, `design-spec.md`, `T<id>-manifest.md`, `T<id>-verification.md`, `wave-<N>-verdict-map.json`, `T<id>-reality-report.md`, `T<id>-diagnosis.md`, `wave-<N>-report.md`, `T<id>-proof-report.md`, `T<id>-amendments.md`, `iteration-log.md`, audit/review reports, narratives, `brainstorm.md`), you MUST print its full path followed by its complete contents verbatim into chat BEFORE any next-step prompt — in the same turn, including in autonomous/loop mode. No length cap, no truncation, no summary substitution. The full Report Surfacing Protocol — including forbidden anti-patterns and the excluded-artifact list — is in the active output style (`scientific.md`, `rigorous.md`, or `pipeline.md`).