pi-evalset-lab 0.1.0

package/docs/org/project-docs-intake.questions.json

@@ -0,0 +1,60 @@
+{
+  "title": "Initialize organization and project documents",
+  "questions": [
+    {
+      "id": "org_purpose",
+      "type": "text",
+      "question": "What is the organization purpose?"
+    },
+    {
+      "id": "org_mission",
+      "type": "text",
+      "question": "What is the current organization mission?"
+    },
+    {
+      "id": "org_vision",
+      "type": "text",
+      "question": "What is the organization vision?"
+    },
+    {
+      "id": "org_strategic_objectives",
+      "type": "text",
+      "question": "List 3-5 organization strategic objectives."
+    },
+    {
+      "id": "org_core_values",
+      "type": "text",
+      "question": "List the core values that should drive ethics and culture."
+    },
+    {
+      "id": "project_purpose",
+      "type": "text",
+      "question": "What is this project purpose (distinct from organization purpose)?"
+    },
+    {
+      "id": "project_mission",
+      "type": "text",
+      "question": "What is the project mission for the next delivery cycle?"
+    },
+    {
+      "id": "project_vision",
+      "type": "text",
+      "question": "What is the project vision?"
+    },
+    {
+      "id": "project_strategic_objectives",
+      "type": "text",
+      "question": "List 3-5 project strategic objectives."
+    },
+    {
+      "id": "project_constraints",
+      "type": "text",
+      "question": "List hard constraints (time, quality, security, scope)."
+    },
+    {
+      "id": "project_success_criteria",
+      "type": "text",
+      "question": "How will success be measured?"
+    }
+  ]
+}

package/docs/project/foundation.md

@@ -0,0 +1,28 @@
+---
+summary: "Compact project model with explicit project-purpose framing."
+read_when:
+  - "Aligning project purpose, strategy, and delivery behavior."
+system4d:
+  container: "Project-level concepts and boundaries for this repository."
+  compass: "Translate project purpose into executable outcomes."
+  engine: "Project purpose -> mission -> vision -> strategic objectives."
+  fog: "Project scope and priorities can drift without explicit review."
+---
+
+# Project foundation model
+
+## Project purpose
+Build and maintain `pi-evalset-lab` as a practical extension package for fixed-task-set evaluations in pi.
+
+## Project mission (current cycle)
+Stabilize `/evalset` workflows, clarify execution behavior (especially `pi -e` and non-interactive mode), and keep reports reproducible.
+
+## Scope boundary
+- Organization purpose lives in [Organization operating model](../org/operating_model.md).
+- This project purpose is repository-specific and narrower.
+
+## Project operating principles
+- Keep changes small and reviewable.
+- Prefer explicit run metadata over implicit assumptions.
+- Document behavior when UX is surprising.
+- Validate structure after meaningful changes.

package/docs/project/incentives.md

@@ -0,0 +1,17 @@
+---
+summary: "Why this project matters and what outcomes are rewarded."
+read_when:
+  - "Prioritizing roadmap work or evaluating tradeoffs."
+system4d:
+  container: "Motivators and value model."
+  compass: "Favor changes that improve reliability and operator confidence."
+  engine: "Map work to measurable maintenance and delivery wins."
+  fog: "Incentives can drift if outcomes are not tracked."
+---
+
+# Incentives
+
+- Faster extension delivery.
+- Lower regression risk.
+- Clear docs for handoffs.
+- Repeatable release and maintenance workflow.

package/docs/project/resources.md

@@ -0,0 +1,26 @@
+---
+summary: "Key resources for building and maintaining the extension."
+read_when:
+  - "Looking up references, docs, or operational artifacts."
+system4d:
+  container: "Reference catalog for the project."
+  compass: "Centralize discovery paths for maintainers."
+  engine: "Link docs, scripts, and examples used in execution."
+  fog: "External links may become stale over time."
+---
+
+# Resources
+
+- [Extension entrypoint](../../extensions/evalset.ts)
+- [Prompt templates](../../prompts)
+- [Smoke dataset](../../examples/fixed-task-set.json)
+- [Larger dataset v2](../../examples/fixed-task-set-v2.json)
+- [Recommended dataset v3](../../examples/fixed-task-set-v3.json)
+- [Baseline system prompt](../../examples/system-baseline.txt)
+- [Candidate system prompt](../../examples/system-candidate.txt)
+- Local report artifacts path: `.evalset/reports/*.json` (and optional `*.html` exports)
+- [Project-local router extension](../../.pi/extensions/startup-intake-router.ts)
+- [Organization operating model](../org/operating_model.md)
+- [Interview questions](../org/project-docs-intake.questions.json)
+- [Security policy](../../policy/security-policy.json)
+- [Validation script](../../scripts/validate-structure.sh)

package/docs/project/skills.md

@@ -0,0 +1,17 @@
+---
+summary: "Skills needed to maintain and extend this package."
+read_when:
+  - "Onboarding maintainers or planning implementation work."
+system4d:
+  container: "Capability map for project contributors."
+  compass: "Build confidence in extension APIs, testing, and release hygiene."
+  engine: "Assess gaps and train through small delivery cycles."
+  fog: "Skill requirements change as scope expands."
+---
+
+# Skills
+
+- TypeScript extension development
+- Shell scripting and git hooks
+- Prompt template design
+- Security review and dependency hygiene

package/docs/project/strategic_goals.md

@@ -0,0 +1,18 @@
+---
+summary: "Long-horizon goals for extension package maturity."
+read_when:
+  - "Roadmapping quarterly or milestone-level outcomes."
+system4d:
+  container: "Strategic objective set."
+  compass: "Reliability, maintainability, and team adoption."
+  engine: "Sequence milestones with measurable checks."
+  fog: "Long-term assumptions may need periodic resets."
+---
+
+# Strategic goals
+
+1. Reach stable, documented `/evalset` behavior across interactive and non-interactive usage.
+2. Improve reproducibility signals (dataset hash, case hash, variant hash, run lineage) in all report paths.
+3. Keep extension docs concise and discoverable for new maintainers.
+4. Maintain secure, low-friction release and validation workflow.
+5. Expand confidence with focused tests for command parsing and report generation.

package/docs/project/tactical_goals.md

@@ -0,0 +1,39 @@
+---
+summary: "Near-term execution goals tied to current sprint cycles."
+read_when:
+  - "Planning immediate tasks and delivery scope."
+system4d:
+  container: "Tactical work queue framing."
+  compass: "Ship the next smallest valuable increment safely."
+  engine: "Break work into verifiable, low-risk tasks."
+  fog: "Unexpected integration constraints may reprioritize tasks."
+---
+
+# Tactical goals
+
+## Current cycle status
+
+### Completed
+- Added troubleshooting notes for `/evalset` invocation (interactive vs `pi -p` non-interactive).
+- Kept run/compare reports discoverable under `.evalset/reports/`.
+- Added stronger example datasets (`fixed-task-set-v2.json`, `fixed-task-set-v3.json`).
+- Synchronized docs and changelog with behavior updates.
+
+### Remaining
+1. Add smoke tests for argument parsing and report/scoring behavior.
+2. Add a repeatable JSON -> HTML report export helper.
+3. Complete npm publish after npmjs auth/registry setup.
+
+## Hard constraints
+- Time: keep work in small slices that can be reviewed quickly.
+- Quality: no regressions in existing `/evalset` commands.
+- Security: no insecure defaults in scripts or extension loading.
+- Scope: prioritize reproducibility and UX clarity over feature breadth.
+
+## Success criteria
+- [x] `npm run check` passes.
+- [x] Documented troubleshooting path for `pi -e` confusion exists.
+- [x] Reports include stable run metadata and are easy to locate.
+- [x] New maintainers can execute a compare workflow from README without intervention.
+- [ ] Automated tests cover parser/scoring basics.
+- [ ] Package is published on npm.

package/docs/project/vision.md

@@ -0,0 +1,21 @@
+---
+summary: "Product and technical vision for pi-evalset-lab."
+read_when:
+  - "Defining or revisiting project direction."
+system4d:
+  container: "Project north-star statement."
+  compass: "Build a reliable pi extension package with low maintenance overhead."
+  engine: "Translate goals into concrete implementation slices."
+  fog: "Real user workflows may reshape priorities."
+---
+
+# Vision
+
+Make `pi-evalset-lab` the lightweight default for comparing prompt/system variants with reproducible, inspectable reports.
+
+The project should feel predictable for maintainers:
+- clear command behavior in both interactive and `-p` modes,
+- clear report locations and run identity metadata,
+- clear docs for troubleshooting common setup confusion.
+
+Project vision is intentionally separate from organization vision in [Organization operating model](../org/operating_model.md).

package/examples/fixed-task-set-v2.json

@@ -0,0 +1,127 @@
+{
+  "name": "maintainer-clarity-v2",
+  "systemPrompt": "Answer in concise, plain English. Prefer direct wording over jargon.",
+  "cases": [
+    {
+      "id": "fixed-task-set-definition",
+      "input": "In plain English, what does a fixed task set mean in eval workflows?",
+      "expectContains": ["same", "tasks"]
+    },
+    {
+      "id": "fixed-task-set-benefit",
+      "input": "Why does using a fixed task set improve comparison quality across runs?",
+      "expectContains": ["compare", "runs"]
+    },
+    {
+      "id": "reproducibility-two-factors",
+      "input": "Name two things you should lock or record for reproducible eval runs.",
+      "expectContains": ["model", "dataset"]
+    },
+    {
+      "id": "pass-rate-calculation",
+      "input": "If 18 of 24 scored cases pass, what pass rate should be reported?",
+      "expectContains": ["75"]
+    },
+    {
+      "id": "scored-case-definition",
+      "input": "What makes a case \"scored\" in this evalset format?",
+      "expectContains": ["checks", "scored"]
+    },
+    {
+      "id": "variant-hash-purpose",
+      "input": "What does variantHash help you verify?",
+      "expectContains": ["variant", "hash"]
+    },
+    {
+      "id": "dataset-hash-purpose",
+      "input": "Why is datasetHash useful when comparing two reports?",
+      "expectContains": ["dataset", "same"]
+    },
+    {
+      "id": "delta-interpretation-speed-cost",
+      "input": "Candidate delta avg latency is -800ms and delta total cost is +0.0003. Summarize the tradeoff.",
+      "expectContains": ["faster", "cost"]
+    },
+    {
+      "id": "command-non-interactive-pattern",
+      "input": "Show the correct non-interactive pattern to run compare with a local extension file.",
+      "expectContains": ["pi -e", "-p", "/evalset compare"]
+    },
+    {
+      "id": "slash-command-shell",
+      "input": "Can you run /evalset directly in bash without pi? Answer yes or no and one reason.",
+      "expectContains": ["no", "pi"]
+    },
+    {
+      "id": "report-default-location",
+      "input": "If --out is omitted, where are evalset reports written by default?",
+      "expectContains": [".evalset/reports"]
+    },
+    {
+      "id": "model-prerequisite",
+      "input": "What should you do before /evalset run if no active model is selected?",
+      "expectContains": ["/model"]
+    },
+    {
+      "id": "max-cases-behavior",
+      "input": "What does --max-cases 5 do during run/compare?",
+      "expectContains": ["first", "5"]
+    },
+    {
+      "id": "temperature-range",
+      "input": "What is the accepted numeric range for --temperature in this extension?",
+      "expectContains": ["0", "2"]
+    },
+    {
+      "id": "system-merge-behavior",
+      "input": "If dataset.systemPrompt exists and --system-file is provided, are prompts merged or replaced?",
+      "expectContains": ["merge"]
+    },
+    {
+      "id": "mutually-exclusive-system-options",
+      "input": "Can --system-file and --system-text be used together?",
+      "expectContains": ["no"]
+    },
+    {
+      "id": "run-identity-fields",
+      "input": "Name any three run identity fields in a run report.",
+      "expectContains": ["runid", "datasethash", "caseshash"]
+    },
+    {
+      "id": "compare-identity-fields",
+      "input": "Name the two run-id fields that link baseline and candidate inside compare.run.",
+      "expectContains": ["baselinerunid", "candidaterunid"]
+    },
+    {
+      "id": "delta-passrate-zero",
+      "input": "If delta pass rate is 0, what does that imply?",
+      "expectContains": ["same", "pass rate"]
+    },
+    {
+      "id": "keyword-check-limitation",
+      "input": "Why can simple keyword checks be misleading for quality?",
+      "expectContains": ["wording", "false"]
+    },
+    {
+      "id": "improve-weak-evalset",
+      "input": "Give two concrete ways to improve a weak 3-case evalset.",
+      "expectContains": ["more", "cases", "criteria"]
+    },
+    {
+      "id": "no-overclaim-rollout",
+      "input": "Should this be pitched as a huge replacement right away? Answer yes or no and one short reason.",
+      "expectContains": ["no"],
+      "expectNotContains": ["huge replacement"]
+    },
+    {
+      "id": "stakeholder-brief",
+      "input": "Write a one-line stakeholder brief that includes pilot scope and measurement.",
+      "expectContains": ["pilot", "measure"]
+    },
+    {
+      "id": "tie-communication",
+      "input": "Baseline and candidate both scored 33.3% pass rate. How should that be communicated?",
+      "expectContains": ["same", "pass rate"]
+    }
+  ]
+}

package/examples/fixed-task-set-v3.json

@@ -0,0 +1,126 @@
+{
+  "name": "maintainer-clarity-v3",
+  "systemPrompt": "Answer in concise, plain English. Prefer direct wording over jargon.",
+  "cases": [
+    {
+      "id": "fixed-task-set-definition",
+      "input": "In plain English, what does a fixed task set mean in eval workflows?",
+      "expectContains": ["same", "tasks"]
+    },
+    {
+      "id": "fixed-task-set-benefit",
+      "input": "Why does a fixed task set improve comparison quality across runs?",
+      "expectContains": ["compar"]
+    },
+    {
+      "id": "reproducibility-two-factors",
+      "input": "Name two things you should lock or record for reproducible eval runs.",
+      "expectContains": ["model", "dataset"]
+    },
+    {
+      "id": "pass-rate-calculation",
+      "input": "If 18 of 24 scored cases pass, what pass rate should be reported?",
+      "expectContains": ["75"]
+    },
+    {
+      "id": "scored-case-definition",
+      "input": "What makes a case scored in this evalset format?",
+      "expectRegex": "([Ee]xpect|[Cc]heck|[Cc]riteria|[Rr]ule)"
+    },
+    {
+      "id": "variant-hash-purpose",
+      "input": "What does variantHash help you verify?",
+      "expectContains": ["variant", "hash"]
+    },
+    {
+      "id": "dataset-hash-purpose",
+      "input": "Why is datasetHash useful when comparing two reports?",
+      "expectContains": ["dataset", "hash"]
+    },
+    {
+      "id": "delta-interpretation-speed-cost",
+      "input": "Candidate delta avg latency is -800ms and delta total cost is +0.0003. Summarize the tradeoff.",
+      "expectContains": ["faster"]
+    },
+    {
+      "id": "command-non-interactive-pattern",
+      "input": "Show the correct non-interactive pattern to run compare from shell.",
+      "expectContains": ["pi", "-p", "evalset"]
+    },
+    {
+      "id": "slash-command-shell",
+      "input": "Can you run /evalset directly in bash without pi? Answer yes or no and one reason.",
+      "expectContains": ["no", "slash"]
+    },
+    {
+      "id": "report-default-location",
+      "input": "If --out is omitted, where are evalset reports written by default?",
+      "expectContains": ["evalset", "report"]
+    },
+    {
+      "id": "model-prerequisite",
+      "input": "What should you do before /evalset run if no active model is selected?",
+      "expectContains": ["/model"]
+    },
+    {
+      "id": "max-cases-behavior",
+      "input": "What does --max-cases 5 do during run/compare?",
+      "expectRegex": "([Ff]irst|[Ll]imit|5.*case|case.*5)"
+    },
+    {
+      "id": "temperature-range",
+      "input": "What is the accepted numeric range for --temperature in this extension?",
+      "expectContains": ["0", "2"]
+    },
+    {
+      "id": "system-merge-behavior",
+      "input": "If dataset.systemPrompt exists and --system-file is provided, are prompts merged or replaced?",
+      "expectContains": ["merge"]
+    },
+    {
+      "id": "mutually-exclusive-system-options",
+      "input": "Can --system-file and --system-text be used together?",
+      "expectRegex": "([Nn]o|[Ee]ither|[Oo]ne|not together|[Bb]oth)"
+    },
+    {
+      "id": "run-identity-fields",
+      "input": "Name any three run identity fields in a run report.",
+      "expectContains": ["run", "dataset", "case"]
+    },
+    {
+      "id": "compare-identity-fields",
+      "input": "Name the two run-id fields that link baseline and candidate inside compare.run.",
+      "expectContains": ["baseline", "candidate", "run"]
+    },
+    {
+      "id": "delta-passrate-zero",
+      "input": "If delta pass rate is 0, what does that imply?",
+      "expectRegex": "([Nn]o change|same pass rate|[Uu]nchanged)"
+    },
+    {
+      "id": "keyword-check-limitation",
+      "input": "Why can simple keyword checks be misleading for quality?",
+      "expectRegex": "([Kk]eyword).*(mislead|false|[Ee]rror|[Bb]rittle)|([Mm]islead|false|[Ee]rror|[Bb]rittle).*[Kk]eyword"
+    },
+    {
+      "id": "improve-weak-evalset",
+      "input": "Give two concrete ways to improve a weak 3-case evalset.",
+      "expectContains": ["more", "cases"]
+    },
+    {
+      "id": "no-overclaim-rollout",
+      "input": "Should this be pitched as a huge replacement right away? Answer yes or no and one short reason.",
+      "expectContains": ["no", "phase"]
+    },
+    {
+      "id": "stakeholder-brief",
+      "input": "Write a one-line stakeholder brief that includes pilot scope and measurement.",
+      "expectContains": ["pilot", "measure"]
+    },
+    {
+      "id": "tie-communication",
+      "input": "Baseline and candidate both scored 33.3% pass rate. How should that be communicated?",
+      "expectRegex": "([Ss]ame|[Nn]o difference|[Uu]nchanged)"
+    }
+  ]
+}

package/examples/fixed-task-set.json

@@ -0,0 +1,22 @@
+{
+  "name": "maintainer-clarity-smoke",
+  "systemPrompt": "Answer concisely and explicitly. No fluff.",
+  "cases": [
+    {
+      "id": "fixed-task-set-definition",
+      "input": "In one sentence, define what a fixed task set means for eval workflows.",
+      "expectContains": ["same tasks"]
+    },
+    {
+      "id": "extension-gaps",
+      "input": "Name two extension-level gaps that can affect reproducible eval workflows in pi.",
+      "expectContains": ["trace", "reproducibility"]
+    },
+    {
+      "id": "do-not-overclaim",
+      "input": "Should this be pitched as a huge replacement right away? Answer yes or no and one short reason.",
+      "expectContains": ["no"],
+      "expectNotContains": ["huge replacement"]
+    }
+  ]
+}

package/examples/system-baseline.txt

@@ -0,0 +1 @@
+You are a concise technical assistant. Keep answers short and avoid jargon.

package/examples/system-candidate.txt

@@ -0,0 +1,6 @@
+You are a concise technical assistant.
+
+Priorities:
+1) Explain terms in plain language first.
+2) Keep each answer to 2-4 sentences.
+3) Prefer concrete examples over abstractions.