pi-evalset-lab 0.1.0

package/CHANGELOG.md

@@ -0,0 +1,43 @@
+---
+summary: "Changelog for scaffold evolution."
+read_when:
+  - "Preparing a release or reviewing history."
+system4d:
+  container: "Release log for this extension package."
+  compass: "Track meaningful deltas per version."
+  engine: "Document changes at release boundaries."
+  fog: "Versioning policy may evolve with team preference."
+---
+
+# Changelog
+
+All notable changes to this project should be documented here.
+
+## [Unreleased]
+
+### Added
+
+- Added `/evalset` MVP command with subcommands:
+  - `init` to generate a starter fixed-task-set dataset
+  - `run` to evaluate one variant against a dataset
+  - `compare` to evaluate baseline vs candidate system prompts
+- Added example files in `examples/`:
+  - `fixed-task-set.json`
+  - `fixed-task-set-v2.json`
+  - `fixed-task-set-v3.json`
+  - `system-baseline.txt`
+  - `system-candidate.txt`
+- Added report output support to `.evalset/reports/*.json` with per-case and aggregate metrics.
+- Added run identity metadata to reports (`runId`, `datasetHash`, `casesHash`, `variantHash`).
+- Reduced session message payload size by storing only lightweight report metadata instead of full report bodies.
+
+### Changed
+
+- Clarified `/evalset` invocation docs: use `pi -p` (or `pi -e ... -p`) for non-interactive runs; `/evalset` is not a standalone shell binary.
+- Added the same non-interactive invocation note to `/evalset help` output.
+
+## [0.1.0] - 2026-02-08
+
+### Added
+
+- Initial production-ready scaffold generated from template v2.

package/CODE_OF_CONDUCT.md

@@ -0,0 +1,50 @@
+---
+summary: "Community behavior expectations and enforcement path."
+read_when:
+  - "Participating in issues, discussions, or pull requests."
+  - "Handling conduct incidents."
+system4d:
+  container: "Shared standards for respectful collaboration."
+  compass: "Safety, respect, and clarity over conflict escalation."
+  engine: "Observe -> report -> review -> enforce -> document."
+  fog: "Context around incidents can be incomplete at first report."
+---
+
+# Code of Conduct
+
+## Our commitment
+
+We want this repository to be a respectful, harassment-free space for everyone,
+regardless of background or identity.
+
+## Expected behavior
+
+- Be respectful and constructive.
+- Assume good intent, ask clarifying questions, and stay technical.
+- Accept feedback and correct mistakes quickly.
+- Keep discussions focused on project outcomes.
+
+## Unacceptable behavior
+
+- Harassment, threats, hate speech, or intimidation.
+- Personal attacks, repeated hostile behavior, or trolling.
+- Sharing private information without consent.
+- Sexualized language or unwanted attention.
+
+## Reporting incidents
+
+If you experience or witness unacceptable behavior:
+
+1. Contact maintainers privately using the channels listed in [SUPPORT.md](SUPPORT.md).
+2. Share relevant links/screenshots and timeline details.
+3. Do not post sensitive personal data publicly.
+
+## Enforcement
+
+Maintainers may remove or edit comments, close threads, or block contributors for
+behavior that violates this policy. Responses may include warning, temporary ban,
+or permanent ban depending on severity and repetition.
+
+## Attribution
+
+Adapted from the [Contributor Covenant](https://www.contributor-covenant.org/version/2/0/code_of_conduct.html).

package/CONTRIBUTING.md

@@ -0,0 +1,28 @@
+---
+summary: "Top-level contribution entrypoint linking to the detailed contributor guide."
+read_when:
+  - "Preparing to submit code or docs changes."
+  - "Looking for contribution quality gates."
+system4d:
+  container: "Contribution intake and quality policy."
+  compass: "Small, reviewable, verified changes."
+  engine: "Read guide -> implement -> validate -> open PR."
+  fog: "Project-specific constraints may evolve with release policy changes."
+---
+
+# Contributing
+
+Primary contributor guide: [docs/dev/CONTRIBUTING.md](docs/dev/CONTRIBUTING.md)
+
+## Minimum checklist
+
+1. Read applicable docs (`npm run docs:list`).
+2. Keep changes scoped.
+3. Run `npm run check`.
+4. Update docs/changelog when behavior changes.
+5. Open a PR with validation output.
+
+## Conduct + support
+
+- [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md)
+- [SUPPORT.md](SUPPORT.md)

package/NEXT_SESSION_PROMPT.md

@@ -0,0 +1,14 @@
+---
+summary: "Session handoff prompt for pi-evalset-lab."
+read_when:
+  - "Starting the next focused development session."
+system4d:
+  container: "Session handoff artifact."
+  compass: "Resume work quickly with explicit priorities."
+  engine: "Capture context, constraints, and next actions."
+  fog: "Staleness risk if not updated after major changes."
+---
+
+# Next session prompt for pi-evalset-lab
+
+Use this file to capture exact follow-up tasks for the next coding session.

package/README.md

@@ -0,0 +1,246 @@
+---
+summary: "Overview and quickstart for pi-evalset-lab."
+read_when:
+  - "Starting work in this repository."
+system4d:
+  container: "Repository scaffold for a pi extension package."
+  compass: "Ship small, safe, testable extension iterations."
+  engine: "Plan -> implement -> verify with docs and hooks in sync."
+  fog: "Unknown runtime integration edge cases until first live sync."
+---
+
+# pi-evalset-lab
+
+Production-ready starter scaffold for a pi extension package.
+
+## Quickstart
+
+1. Install dependencies (if you add any):
+
+   ```bash
+   npm install
+   ```
+
+2. Test with pi:
+
+   ```bash
+   pi -e ./extensions/evalset.ts
+   ```
+
+3. Install package into pi:
+
+   ```bash
+   pi install /absolute/path/to/pi-evalset-lab
+   ```
+
+## evalset command (MVP)
+
+This extension adds `/evalset` for fixed-task-set evaluation runs.
+
+### Commands
+
+```bash
+/evalset help
+/evalset init [dataset-path] [--force]
+/evalset run <dataset.json> [--system-file <path>] [--system-text <text>] [--variant <name>] [--max-cases <n>] [--temperature <n>] [--out <report.json>]
+/evalset compare <dataset.json> <baseline-system.txt> <candidate-system.txt> [--baseline-name <name>] [--candidate-name <name>] [--max-cases <n>] [--temperature <n>] [--out <report.json>]
+```
+
+### Running modes
+
+`/evalset` is a pi slash command, not a shell executable.
+
+Interactive mode:
+
+```bash
+pi -e ./extensions/evalset.ts
+# then inside pi:
+/evalset compare examples/fixed-task-set.json examples/system-baseline.txt examples/system-candidate.txt
+```
+
+Non-interactive mode (scripts/CI):
+
+```bash
+pi -e ./extensions/evalset.ts -p "/evalset compare examples/fixed-task-set.json examples/system-baseline.txt examples/system-candidate.txt"
+# or, if extension already installed/enabled:
+pi -p "/evalset compare examples/fixed-task-set.json examples/system-baseline.txt examples/system-candidate.txt"
+```
+
+### Example workflow (inside pi)
+
+```bash
+/evalset run examples/fixed-task-set.json --variant baseline
+/evalset compare examples/fixed-task-set.json examples/system-baseline.txt examples/system-candidate.txt
+```
+
+### Included datasets
+
+- `examples/fixed-task-set.json` — tiny smoke set (3 cases)
+- `examples/fixed-task-set-v2.json` — larger first pass set
+- `examples/fixed-task-set-v3.json` — less brittle checks (recommended)
+
+The command writes JSON reports to:
+- explicit `--out <path>` when provided
+- otherwise `.evalset/reports/*.json` under your current project directory
+
+Each report includes run identity metadata:
+- `runId`
+- `datasetHash`
+- `casesHash`
+- `variantHash` (run) or baseline/candidate variant hashes (compare)
+
+Session messages only keep lightweight report metadata (`reportPath`, ids, summary metrics), not full report bodies.
+
+## Optional core hooks (future, not required for this MVP)
+
+This extension works today without core changes. If we decide to harden further, optional core support could include:
+
+1. Stable agent-level lineage IDs (`runId`/`traceId`) across extension events.
+2. Explicit reproducibility capability metadata in `pi-ai` (e.g. seed support and determinism caveats per provider/model).
+3. Shared canonical provider payload hash helper in `pi-ai`.
+4. A headless agent-eval API for tool-heavy/full agent-loop benchmark runs.
+
+## Repository checks
+
+Run:
+
+```bash
+npm run check
+```
+
+This executes [scripts/validate-structure.sh](scripts/validate-structure.sh).
+
+## Release + security baseline
+
+This scaffold defaults to **release-please** for single-package release PR + tag flow (`vX.Y.Z`) and npm trusted publishing via OIDC.
+
+Included files:
+
+- [CI workflow](.github/workflows/ci.yml)
+- [release-please workflow](.github/workflows/release-please.yml)
+- [publish workflow](.github/workflows/publish.yml)
+- [Dependabot config](.github/dependabot.yml)
+- [CODEOWNERS](.github/CODEOWNERS)
+- [release-please config](.release-please-config.json)
+- [release-please manifest](.release-please-manifest.json)
+- [Security policy](SECURITY.md)
+
+Before first production release:
+
+1. Confirm/adjust owners in [.github/CODEOWNERS](.github/CODEOWNERS).
+2. Enable branch protection on `main`.
+3. Configure npm Trusted Publishing for this repo + [publish workflow](.github/workflows/publish.yml).
+4. Merge release PR from release-please, then publish from GitHub release.
+
+## Issue + PR intake baseline
+
+Included files:
+
+- [Bug report form](.github/ISSUE_TEMPLATE/bug-report.yml)
+- [Feature request form](.github/ISSUE_TEMPLATE/feature-request.yml)
+- [Docs request form](.github/ISSUE_TEMPLATE/docs.yml)
+- [Issue template config](.github/ISSUE_TEMPLATE/config.yml)
+- [PR template](.github/pull_request_template.md)
+- [Code of conduct](CODE_OF_CONDUCT.md)
+- [Support guide](SUPPORT.md)
+- [Top-level contributing guide](CONTRIBUTING.md)
+
+## Vouch trust gate baseline
+
+Included files:
+
+- [Vouched contributors list](.github/VOUCHED.td)
+- [PR trust gate workflow](.github/workflows/vouch-check-pr.yml)
+- [Issue-comment trust management workflow](.github/workflows/vouch-manage.yml)
+
+Default behavior:
+
+- PR workflow runs on `pull_request_target` (`opened`, `reopened`).
+- `require-vouch: true` and `auto-close: true` are enabled by default.
+- Maintainers can comment `vouch`, `denounce`, or `unvouch` on issues to update trust state.
+- Vouch actions are SHA pinned (`0e11a71bba23218a284d3ecca162e75a110fd7e3`) for reproducibility and supply-chain review.
+
+Bootstrap step:
+
+- Confirm/adjust entries in [.github/VOUCHED.td](.github/VOUCHED.td) before enforcing production policy.
+
+## Docs discovery
+
+Run:
+
+```bash
+npm run docs:list
+npm run docs:list:workspace
+npm run docs:list:json
+```
+
+Wrapper script: [scripts/docs-list.sh](scripts/docs-list.sh)
+
+Resolution order:
+1. `DOCS_LIST_SCRIPT`
+2. `./scripts/docs-list.mjs` (if vendored)
+3. `~/ai-society/core/agent-scripts/scripts/docs-list.mjs`
+
+## Copier lifecycle policy
+
+- Keep `.copier-answers.yml` committed.
+- Do not edit `.copier-answers.yml` manually.
+- Run from a clean destination repo (commit or stash pending changes first).
+- Use `copier update --trust` when `.copier-answers.yml` includes `_commit` and update is supported.
+- In non-interactive shells/CI, append `--defaults` to update/recopy.
+- Use `copier recopy --trust` when update is unavailable (for example local non-VCS source) or cannot reconcile cleanly.
+- After recopy, re-apply local deltas intentionally and run `npm run check`.
+
+## Hook behavior
+
+- Git uses `.githooks/pre-commit` (configured by [scripts/install-hooks.sh](scripts/install-hooks.sh)).
+- If `prek` is available, the hook runs `prek` using [prek.toml](prek.toml).
+- If `prek` is not available, the hook falls back to `scripts/validate-structure.sh`.
+
+Install options for `prek`:
+
+```bash
+npm add -D @j178/prek
+# or
+npm install -g @j178/prek
+```
+
+## Startup interview flow (project-local)
+
+- [`.pi/extensions/startup-intake-router.ts`](.pi/extensions/startup-intake-router.ts) watches the first non-command message in a session.
+- It converts your startup intent into a prefilled command:
+  - `/init-project-docs "<your intent>"`
+- [`.pi/prompts/init-project-docs.md`](.pi/prompts/init-project-docs.md) then drives the `interview` tool using [docs/org/project-docs-intake.questions.json](docs/org/project-docs-intake.questions.json).
+
+Utility commands:
+
+- `/startup-intake-router-status`
+- `/startup-intake-router-reset`
+
+## Live sync helper
+
+Use [scripts/sync-to-live.sh](scripts/sync-to-live.sh) to copy the package extension to
+`~/.pi/agent/extensions/`.
+
+Optional flags:
+
+- `--with-prompts`
+- `--with-policy`
+- `--all` (prompts + policy)
+
+After sync, run `/reload` in pi.
+
+## Docs map
+
+- [Organization operating model](docs/org/operating_model.md)
+- [Project foundation model](docs/project/foundation.md)
+- [Project vision](docs/project/vision.md)
+- [Project incentives](docs/project/incentives.md)
+- [Project resources](docs/project/resources.md)
+- [Project skills](docs/project/skills.md)
+- [Strategic goals](docs/project/strategic_goals.md)
+- [Tactical goals](docs/project/tactical_goals.md)
+- [Contributor guide](docs/dev/CONTRIBUTING.md)
+- [Extension SOP](docs/dev/EXTENSION_SOP.md)
+- [Next steps](docs/dev/next_steps.md)
+- [Status](docs/dev/status.md)

package/SECURITY.md

@@ -0,0 +1,34 @@
+---
+summary: "Security reporting process and release hardening baseline."
+read_when:
+  - "Reporting a vulnerability."
+  - "Reviewing release and workflow security controls."
+system4d:
+  container: "Security policy for maintainers and contributors."
+  compass: "Private reporting, least privilege, auditable releases."
+  engine: "Report privately -> triage -> patch -> verify -> disclose."
+  fog: "Dependency and ecosystem risk shifts over time."
+---
+
+# Security Policy
+
+## Supported versions
+
+Security fixes target the latest release and `main` branch.
+
+## Reporting a vulnerability
+
+Use **private reporting**.
+
+1. Preferred: GitHub Security tab -> **Report a vulnerability**.
+2. If private reporting is unavailable, open a minimal issue titled
+   `Security contact request` without exploit details and request a private channel.
+3. Include impact, affected versions, and reproduction steps.
+4. Avoid public disclosure until maintainers confirm a fix/release plan.
+
+## Release and supply-chain baseline
+
+- Release flow uses release-please PRs before tags/releases.
+- Publish flow uses npm Trusted Publishing (OIDC) and `npm publish --provenance`.
+- Workflow permissions default to read and elevate per job only.
+- Third-party actions must stay explicit; high-risk paths should be SHA pinned.

package/SUPPORT.md

@@ -0,0 +1,37 @@
+---
+summary: "How users and contributors request help or report problems."
+read_when:
+  - "Needing help, troubleshooting, or reporting non-security issues."
+  - "Deciding where to file bug/feature/docs requests."
+system4d:
+  container: "Support intake and routing guidance."
+  compass: "Route requests to the right channel with enough context."
+  engine: "Self-check -> search -> file focused issue -> iterate."
+  fog: "Reproduction context is often incomplete in first reports."
+---
+
+# Support
+
+## Before opening an issue
+
+1. Read [README.md](README.md) and relevant docs in `docs/`.
+2. Search existing issues/PRs for duplicates.
+3. Re-test on the latest release.
+
+## Open the right issue type
+
+Use the GitHub issue forms for:
+
+- Bug reports
+- Feature requests
+- Documentation improvements
+
+## Security reports
+
+For vulnerabilities, follow [SECURITY.md](SECURITY.md) and use private reporting.
+Do **not** post exploit details in public issues.
+
+## Maintainer response expectations
+
+This project may be maintained part-time. Triage and response times can vary,
+but actionable reports with reproduction details are prioritized.

package/docs/dev/CONTRIBUTING.md

@@ -0,0 +1,37 @@
+---
+summary: "Contribution workflow for this extension repository."
+read_when:
+  - "Before opening PRs or submitting local changes."
+system4d:
+  container: "Contributor process and quality gates."
+  compass: "Small, validated, documented changes."
+  engine: "Branch -> implement -> check -> document -> review."
+  fog: "Process details may adjust with team scale."
+---
+
+# Contributing
+
+## Workflow
+
+1. Create a focused branch.
+2. Run `npm run docs:list` and read matched docs before cross-cutting changes.
+3. Implement one scoped change.
+4. Run `npm run check`.
+5. Update docs/changelog where relevant.
+6. Open PR with concise rationale and validation output.
+
+## Standards
+
+- Keep diffs small and reviewable.
+- Preserve markdown frontmatter in generated docs.
+- Prefer explicit scripts over manual one-off commands.
+
+## Copier policy
+
+- Keep `.copier-answers.yml` in version control.
+- Do not edit `.copier-answers.yml` manually.
+- Run update/recopy from a clean destination repo (commit or stash pending changes first).
+- Use `copier update --trust` when `.copier-answers.yml` includes `_commit` and update is supported.
+- In non-interactive shells/CI, append `--defaults` to update/recopy.
+- Use `copier recopy --trust` when update is unavailable (for example local non-VCS source) or cannot reconcile cleanly.
+- After recopy, re-apply local deltas intentionally and run `npm run check`.

package/docs/dev/EXTENSION_SOP.md

@@ -0,0 +1,43 @@
+---
+summary: "Lifecycle SOP for extension delivery and maintenance."
+read_when:
+  - "Planning, implementing, verifying, releasing, or maintaining extension work."
+system4d:
+  container: "End-to-end extension operating procedure."
+  compass: "Consistent quality from idea to maintenance."
+  engine: "plan -> implement -> verify -> release -> maintain."
+  fog: "Unknowns resolved through incremental validation loops."
+---
+
+# Extension SOP
+
+## 1) Plan
+
+- Define scope and acceptance criteria.
+- Run `npm run docs:list` and read docs matching your task domain.
+- Capture work in `docs/dev/plans/`.
+- Confirm risks and dependencies.
+
+## 2) Implement
+
+- Build in small commits.
+- Keep command/tool behavior explicit.
+- Update docs as behavior changes.
+
+## 3) Verify
+
+- Run `npm run check`.
+- Execute relevant extension tests.
+- Validate prompt templates if changed.
+
+## 4) Release
+
+- Update `CHANGELOG.md`.
+- Tag/version according to team policy.
+- Sync extension to live pi when needed.
+
+## 5) Maintain
+
+- Monitor regressions and user feedback.
+- Re-run validation after dependency/script changes.
+- Keep `docs/dev/status.md` and `docs/dev/next_steps.md` current.

package/docs/dev/next_steps.md

@@ -0,0 +1,17 @@
+---
+summary: "Prioritized next actions for active development."
+read_when:
+  - "Starting a coding session or grooming tasks."
+system4d:
+  container: "Execution queue for maintainers."
+  compass: "Maintain momentum with clear, ordered tasks."
+  engine: "Do highest-leverage item first, then validate."
+  fog: "Task order may change after discovery."
+---
+
+# Next steps
+
+1. Complete npm publish on npmjs (login, registry, publish, verify install).
+2. Add at least one automated test for argument parsing and expectation scoring.
+3. Add a small repeatable report-share helper (JSON -> static HTML export command/script).
+4. Evaluate optional LLM-judge scoring mode (`expectJudgePrompt`) after parser/scoring tests exist.

package/docs/dev/plans/001-initial-plan.md

@@ -0,0 +1,24 @@
+---
+summary: "Initial implementation plan for first extension iteration."
+read_when:
+  - "Executing the first feature slice from scaffold state."
+system4d:
+  container: "Plan artifact for incremental delivery."
+  compass: "Move from scaffold to validated feature quickly."
+  engine: "Plan -> implement -> verify -> document."
+  fog: "Scope creep risk if tasks are not constrained."
+---
+
+# Plan 001: first feature slice
+
+## Objective
+
+Ship one useful command behavior end-to-end.
+
+## Steps
+
+1. Define expected command input/output.
+2. Implement logic in `extensions/`.
+3. Add tests in `tests/`.
+4. Run `npm run check`.
+5. Update `docs/dev/status.md` and `CHANGELOG.md`.

package/docs/dev/status.md

@@ -0,0 +1,21 @@
+---
+summary: "Current project status snapshot."
+read_when:
+  - "Checking project health or preparing handoff updates."
+system4d:
+  container: "State report for current branch/project."
+  compass: "Keep stakeholders aligned on progress and blockers."
+  engine: "Update status after meaningful implementation slices."
+  fog: "Status can stale quickly without disciplined updates."
+---
+
+# Status
+
+- Scaffold: complete
+- Extension behavior: `/evalset` MVP implemented (`init`, `run`, `compare`)
+- Example datasets: smoke + `fixed-task-set-v2.json` + `fixed-task-set-v3.json`
+- Invocation docs: clarified non-interactive usage via `pi -p` / `pi -e ... -p`
+- GitHub publish: `tryingET/pi-evalset-lab` created with release `v0.1.0`
+- npm publish: pending (`npm` auth + registry set to `https://registry.npmjs.org/`)
+- Validation hooks: installed
+- Tests: pending

package/docs/org/operating_model.md

@@ -0,0 +1,39 @@
+---
+summary: "Compact organization operating model and terminology."
+read_when:
+  - "Aligning organization-level purpose, mission, and strategy."
+system4d:
+  container: "Organization-level concepts shared across projects."
+  compass: "Keep strategy and culture aligned with organization purpose."
+  engine: "Purpose -> mission -> vision -> strategic objectives."
+  fog: "Terminology drift can create cross-project confusion."
+---
+
+# Organization operating model
+
+## Organization purpose
+Enable teams to build and operate AI-assisted software workflows that are reliable, auditable, and easy to improve.
+
+## Organization mission (current)
+Provide practical tooling, standards, and documentation that make safe AI engineering the default path.
+
+## Organization vision
+A trusted ecosystem where AI coding workflows are reproducible, understandable, and continuously improving.
+
+## Organization strategic objectives
+1. Standardize evaluation workflows across active projects.
+2. Improve run traceability and reproducibility metadata.
+3. Keep onboarding friction low through concise, current documentation.
+4. Maintain secure-by-default release and dependency hygiene.
+5. Reduce ambiguity between interactive and non-interactive execution modes.
+
+## Core values
+- Clarity first
+- Evidence over assumption
+- Small, verifiable increments
+- Safety for high-impact changes
+- Respectful collaboration
+
+## Boundary
+Organization purpose is cross-project.
+Project-specific purpose is defined in [Project foundation model](../project/foundation.md).