planr 1.6.0 → 1.7.0

package/docs/ARCHITECTURE.md

@@ -49,6 +49,16 @@ Planr V1 is a single Rust binary with explicit module ownership. The crate stays
 - `src/rolefiles.rs`: static host workflow roles and Cursor skill payloads. It does not select or pin models; externally generated routing artifacts stay outside Planr ownership.
 - `src/util.rs`: small CLI-boundary utilities. Owns ids, timestamps, path helpers, output formatting, and safe file writes.
 
+## Eval Contract V1/V1.1 Ownership
+
+The frozen product contract lives in `docs/planr-spec/EVAL_CONTRACT_V1.md` until implementation promotes it into code. Its V1.1 efficiency-evidence amendment is additive and keeps the same owner split:
+
+- repository EvalSuite manifests and fixtures are authored inputs;
+- the suite loader owns validation, safe path resolution, normalization, canonical ordering, and digests;
+- SQLite owns immutable suite snapshots, runs, case results, samples, attempt lineage, metering-basis provenance, comparisons, invalidations, rescoring provenance, and eval evidence references;
+- the comparison engine owns verdicts, reason codes, matched effective-treatment compatibility, and derived failure-inclusive efficiency metrics;
+- existing map/log/review/approval state remains authoritative for work closure, so eval evidence may support a review but must never close or reopen items by itself.
+
 ## Boundary Rules
 
 - Command parsing belongs in `src/cli.rs`; process startup belongs in `src/main.rs`; command execution belongs under `src/app/`.

package/docs/CI.md

@@ -27,10 +27,11 @@ Run the full local gate:
 scripts/ci-local.sh
 ```
 
-`scripts/ci-local.sh` also runs the external consumer E2E project when `/Users/kregenrek/projects/planr-test` exists:
+`scripts/ci-local.sh` also runs the external consumer E2E project when the sibling `planr-test` checkout exists. Override its location with `PLANR_TEST_DIR`:
 
 ```bash
-cd /Users/kregenrek/projects/planr-test
+PLANR_TEST_DIR=/path/to/planr-test scripts/ci-local.sh
+cd /path/to/planr-test
 npm test
 npm run test:npm-planr
 ```
@@ -38,10 +39,10 @@ npm run test:npm-planr
 Run local security and leak checks:
 
 ```bash
-scripts/security-local.sh
+npm run security:check
 ```
 
-This uses BetterLeaks for secret history scanning and Trivy for filesystem vulnerability, secret, and misconfiguration scanning.
+This checks tracked files for machine-specific home paths, rejects forbidden staged paths, uses BetterLeaks for secret history scanning, and runs Trivy for filesystem vulnerability, secret, and misconfiguration scanning. If Lefthook is installed, run `lefthook install` once to activate the staged-path guard for commits and pushes.
 
 ## Supply Chain
 

package/docs/CLI_REFERENCE.md

@@ -59,6 +59,14 @@ planr review ingest <item-id> (--from feedback.json|--stdin)
 planr review artifact <review-item-id> [--out .planr/reviews/custom.review.md]
 planr review evidence <item-id> [--pr-url https://...]
 planr review close <review-item-id> --verdict complete|not-complete|unclear [--close-target]
+planr eval suite-check --input suite.json
+planr eval run --input run.json
+planr eval show suite|run|comparison|invalidation <id>
+planr eval compare <baseline-run-id> <candidate-run-id> [--policy-digest default]
+planr eval gate <comparison-id>
+planr eval invalidate run|comparison <target-id> --reason "..." [--reason-code code] [--replacement-hint "..."]
+planr eval rescore <run-id> [--id <new-run-id>]
+planr eval evidence-ref run|comparison <target-id> log|review|artifact <attachment-id> --item <item-id>
 planr close [item-id] --summary "..." [--next]
 planr done [item-id] --summary "..." [--files a --files b] [--cmd "..."] [--tests "..."] [--review] [--next] [--profile <id>]
 planr context add "text" [--item <item-id>] [--tag discovery]
@@ -129,6 +137,8 @@ With `--json`, responses follow one convention so agents never guess where data
 
 `review close` writes `.planr/reviews/<review-item-id>.review.md` and registers it as a review artifact. A `not-complete` or `unclear` verdict creates fix and follow-up review work; the follow-up review gates the same target item, so the chain keeps working with `--close-target`. With `--close-target` (complete verdicts only) the reviewed item is closed in the same command, provided it already has a completion log; the artifact is rendered after the target transition, so it snapshots the final target status. `--close-target` is also available through MCP `planr_review_close` and HTTP `POST /v1/reviews/{id}/close` (`"close_target": true`). `review close` responses include the same `remaining` progress snapshot as `done` and `close`. `--reviewer <id>` records the checker's identity on the review log, artifact, and event (defaults to the worker id), keeping maker and checker distinguishable in the audit trail. Closing an already-settled review fails with error code `already_closed` instead of silently duplicating evidence logs. The maker/checker split is derived, not declared: `review_mode` compares the closing reviewer identity against the target item's lease holder and reports `single_agent`, `independent`, or `unattributed` in the response, review log, artifact, and event — no ceremony note required. An `unattributed` close explains itself in the output: it means the target has no recorded lease (work was never picked or its lease was released), not that the reviewer's identity was missing.
 
+`eval` exposes the same stored evaluation services to CLI and MCP. Every eval JSON response is one document with the frozen envelope `ok`, `command`, `object`, `warnings`, `reasons`, and `error`; MCP tools return the same envelope inside the text content, including `isError` tool results. `suite-check` reports command `eval.suite.check`, stores or verifies an immutable suite snapshot, `run` starts a run from a JSON envelope and can record cases/samples plus a terminal status, `compare` persists a verdict from two stored runs, and `gate` exits non-zero for `regressed` or `insufficient_evidence` while still printing the verdict, blocker, effects, uncertainty, coverage/freshness gates, and deeper-read commands. Eval exit codes are fixed: `0` for accepted command outcomes, `1` for regression or promotion-gate failure, `2` for insufficient evidence, `3` for invalid or unsafe input, and `4` for infrastructure errors such as missing stored runs. Error objects include `code`, `message`, `reasons`, and `field`. `invalidate` records invalidation evidence for a run or comparison, and `rescore` starts a new run linked back to the source run. `evidence-ref` attaches a run or comparison to an existing log, review item, or artifact for audit/package portability; it is never closure authority and never changes map item status. MCP mirrors these operations as `planr_eval_suite_check`, `planr_eval_run`, `planr_eval_show`, `planr_eval_compare`, `planr_eval_gate`, `planr_eval_invalidate`, `planr_eval_rescore`, and `planr_eval_evidence_ref`. A repository-owned smoke suite lives at `examples/eval/planr-lifecycle-smoke.suite.json`; recalculate fixture digests before using it outside repo-local examples.
+
 `trace item` on a review item inlines the target item and its evidence logs under `target`, so a reviewer's first trace already contains what is being audited. The human (non-JSON) mode renders the packet: status, owner, links, logs. When the item has a declared route, a run profile, or a route observation, the trace gains a `routing` section — the declared route next to every run's actual client/profile and, when supplied, its independent requested → resolved → effective model/effort/context-fork stages, transition reason, policy/binding provenance, and per-dimension metering confidence. Unknown effective host values remain `null`/`unavailable`; requested values are never copied forward as proof. MCP `planr_trace_item` returns the identical JSON shape.
 
 `doctor` also reports the agent registry in all states without ever failing: absent (informational hint), degraded (parse error with line context), or loaded with profile/route counts, validation warnings, and per-artifact drift — a rendered role file whose content no longer matches what the current registry would render is flagged `drifted` with a `planr install <client> --force` hint, while files without the generated-from header are the user's (`manual`) and never flagged.

package/docs/IMPORT.md

@@ -7,10 +7,12 @@ planr import planr-package.json --preview
 planr import planr-package.json --confirm
 ```
 
-Planr packages are local-first JSON files created by `planr export`. They carry graph items, links, contexts, optional logs, optional plan file snapshots, and review artifacts.
+Planr packages are local-first JSON files created by `planr export`. They carry graph items, links, contexts, optional logs, optional eval evidence refs, optional plan file snapshots, and review artifacts.
 
 Imports are preview-first. Preview reports package metadata, create counts, and conflicting item ids before mutating the current project.
 
+Eval evidence exports only when logs are included. Packages carry immutable eval suite snapshots, runs, case results, raw samples, comparisons, invalidations, and evidence refs so imported refs resolve to local eval records and comparison verdicts can be reproduced from restored run evidence. Import preflights the whole eval graph before inserting package rows: duplicate immutable ids inside the package must be byte/field-identical, suite/run/comparison/invalidation/ref dependencies must resolve, case sample ids must match nested samples, and eval refs must have `closure_authority: false`. Existing immutable eval ids are accepted only when their stored content exactly matches the package, and imported eval timestamps/provenance are preserved. Review refs must point at a real `work_type=review` item linked to the claimed item. Restoring audit provenance never closes work or changes map ownership.
+
 Planr packages are local-first JSON. For encrypted sharing, review the JSON locally and encrypt the file with your team's standard tool, for example:
 
 ```bash

package/docs/MCP_CONTRACT.md

@@ -41,6 +41,7 @@ Required groups:
 - approval request, approve, deny, and list
 - artifact add, list, and show
 - event list and debug bundle preview
+- eval suite check, run, show, compare, gate, invalidate, rescore, and evidence refs
 - trace item, log add, and log read (including three-stage route observations)
 - provider-neutral agent registry reads and route overrides
 - review annotate, ingest, artifact, evidence, and close
@@ -60,6 +61,10 @@ Review feedback ingestion is advisory:
 
 HTTP mirrors the same rule: `GET /v1/reviews/:id/artifact` is read-only; `POST /v1/reviews/:id/artifact` writes an artifact explicitly.
 
+## Eval Evidence Contract
+
+`planr_eval_evidence_ref` mirrors `planr eval evidence-ref`: it attaches an eval run or comparison id to an existing log, review item, or artifact. The returned eval envelope records `closure_authority: false`; eval verdicts are audit evidence only and never close, approve, or otherwise mutate map item status.
+
 ## Install Contract
 
 `planr install <client> --dry-run` prints the complete client-owned MCP, role, skill, and hook-reconciliation paths for Codex, Claude Code, and Cursor without writing them. Non-dry install writes only repository-local files, with this ownership contract:

package/docs/TESTING.md

@@ -67,7 +67,7 @@ npm pack --dry-run
 Installer changes also require:
 
 ```bash
-/Users/kregenrek/.agents/skills/shellck/scripts/run_shellck.sh scripts/install.sh
+$HOME/.agents/skills/shellck/scripts/run_shellck.sh scripts/install.sh
 PREFIX="$(mktemp -d)" PLANR_DOWNLOAD=1 PLANR_RELEASE_BASE_URL="file://$PWD/dist" scripts/install.sh
 ```
 

package/docs/fixtures/mcp-contract.json

@@ -42,6 +42,14 @@
     "planr_artifact_show",
     "planr_event_list",
     "planr_debug_bundle",
+    "planr_eval_suite_check",
+    "planr_eval_run",
+    "planr_eval_show",
+    "planr_eval_compare",
+    "planr_eval_gate",
+    "planr_eval_invalidate",
+    "planr_eval_rescore",
+    "planr_eval_evidence_ref",
     "planr_log_add",
     "planr_review_annotate",
     "planr_review_ingest",
@@ -100,6 +108,10 @@
     "planr review artifact",
     "planr review evidence",
     "planr review close",
+    "planr eval suite-check",
+    "planr eval compare",
+    "planr eval gate",
+    "planr eval evidence-ref",
     "planr recover sweep",
     "planr agents list",
     "planr agents check",

package/docs/planr-spec/API_AND_DATA_MODEL.md

@@ -1,5 +1,7 @@
 # API And Data Model
 
+Eval Contract v1, including the additive v1.1 efficiency-evidence amendment, is frozen in [EVAL_CONTRACT_V1.md](EVAL_CONTRACT_V1.md). That document owns the canonical eval manifest fields, normalized snapshot, stored objects, attempt lineage, metering basis, effective treatment identity, derived efficiency metrics, status values, reason codes, evidence identities, response examples, exit codes, database rules, and safety constraints. This file keeps the higher-level product API summary.
+
 ## Storage Locations
 
 - SQLite: authoritative map graph state, picks, contexts, logs, runs, events, search indexes.