pi-evalset-lab 0.1.0 → 0.2.0

package/docs/dev/status.md

@@ -1,21 +0,0 @@
----
-summary: "Current project status snapshot."
-read_when:
-  - "Checking project health or preparing handoff updates."
-system4d:
-  container: "State report for current branch/project."
-  compass: "Keep stakeholders aligned on progress and blockers."
-  engine: "Update status after meaningful implementation slices."
-  fog: "Status can stale quickly without disciplined updates."
----
-
-# Status
-
-- Scaffold: complete
-- Extension behavior: `/evalset` MVP implemented (`init`, `run`, `compare`)
-- Example datasets: smoke + `fixed-task-set-v2.json` + `fixed-task-set-v3.json`
-- Invocation docs: clarified non-interactive usage via `pi -p` / `pi -e ... -p`
-- GitHub publish: `tryingET/pi-evalset-lab` created with release `v0.1.0`
-- npm publish: pending (`npm` auth + registry set to `https://registry.npmjs.org/`)
-- Validation hooks: installed
-- Tests: pending

package/docs/org/operating_model.md

@@ -1,39 +0,0 @@
----
-summary: "Compact organization operating model and terminology."
-read_when:
-  - "Aligning organization-level purpose, mission, and strategy."
-system4d:
-  container: "Organization-level concepts shared across projects."
-  compass: "Keep strategy and culture aligned with organization purpose."
-  engine: "Purpose -> mission -> vision -> strategic objectives."
-  fog: "Terminology drift can create cross-project confusion."
----
-
-# Organization operating model
-
-## Organization purpose
-Enable teams to build and operate AI-assisted software workflows that are reliable, auditable, and easy to improve.
-
-## Organization mission (current)
-Provide practical tooling, standards, and documentation that make safe AI engineering the default path.
-
-## Organization vision
-A trusted ecosystem where AI coding workflows are reproducible, understandable, and continuously improving.
-
-## Organization strategic objectives
-1. Standardize evaluation workflows across active projects.
-2. Improve run traceability and reproducibility metadata.
-3. Keep onboarding friction low through concise, current documentation.
-4. Maintain secure-by-default release and dependency hygiene.
-5. Reduce ambiguity between interactive and non-interactive execution modes.
-
-## Core values
-- Clarity first
-- Evidence over assumption
-- Small, verifiable increments
-- Safety for high-impact changes
-- Respectful collaboration
-
-## Boundary
-Organization purpose is cross-project.
-Project-specific purpose is defined in [Project foundation model](../project/foundation.md).

package/docs/org/project-docs-intake.questions.json

@@ -1,60 +0,0 @@
-{
-  "title": "Initialize organization and project documents",
-  "questions": [
-    {
-      "id": "org_purpose",
-      "type": "text",
-      "question": "What is the organization purpose?"
-    },
-    {
-      "id": "org_mission",
-      "type": "text",
-      "question": "What is the current organization mission?"
-    },
-    {
-      "id": "org_vision",
-      "type": "text",
-      "question": "What is the organization vision?"
-    },
-    {
-      "id": "org_strategic_objectives",
-      "type": "text",
-      "question": "List 3-5 organization strategic objectives."
-    },
-    {
-      "id": "org_core_values",
-      "type": "text",
-      "question": "List the core values that should drive ethics and culture."
-    },
-    {
-      "id": "project_purpose",
-      "type": "text",
-      "question": "What is this project purpose (distinct from organization purpose)?"
-    },
-    {
-      "id": "project_mission",
-      "type": "text",
-      "question": "What is the project mission for the next delivery cycle?"
-    },
-    {
-      "id": "project_vision",
-      "type": "text",
-      "question": "What is the project vision?"
-    },
-    {
-      "id": "project_strategic_objectives",
-      "type": "text",
-      "question": "List 3-5 project strategic objectives."
-    },
-    {
-      "id": "project_constraints",
-      "type": "text",
-      "question": "List hard constraints (time, quality, security, scope)."
-    },
-    {
-      "id": "project_success_criteria",
-      "type": "text",
-      "question": "How will success be measured?"
-    }
-  ]
-}

package/docs/project/foundation.md

@@ -1,28 +0,0 @@
----
-summary: "Compact project model with explicit project-purpose framing."
-read_when:
-  - "Aligning project purpose, strategy, and delivery behavior."
-system4d:
-  container: "Project-level concepts and boundaries for this repository."
-  compass: "Translate project purpose into executable outcomes."
-  engine: "Project purpose -> mission -> vision -> strategic objectives."
-  fog: "Project scope and priorities can drift without explicit review."
----
-
-# Project foundation model
-
-## Project purpose
-Build and maintain `pi-evalset-lab` as a practical extension package for fixed-task-set evaluations in pi.
-
-## Project mission (current cycle)
-Stabilize `/evalset` workflows, clarify execution behavior (especially `pi -e` and non-interactive mode), and keep reports reproducible.
-
-## Scope boundary
-- Organization purpose lives in [Organization operating model](../org/operating_model.md).
-- This project purpose is repository-specific and narrower.
-
-## Project operating principles
-- Keep changes small and reviewable.
-- Prefer explicit run metadata over implicit assumptions.
-- Document behavior when UX is surprising.
-- Validate structure after meaningful changes.

package/docs/project/incentives.md

@@ -1,17 +0,0 @@
----
-summary: "Why this project matters and what outcomes are rewarded."
-read_when:
-  - "Prioritizing roadmap work or evaluating tradeoffs."
-system4d:
-  container: "Motivators and value model."
-  compass: "Favor changes that improve reliability and operator confidence."
-  engine: "Map work to measurable maintenance and delivery wins."
-  fog: "Incentives can drift if outcomes are not tracked."
----
-
-# Incentives
-
-- Faster extension delivery.
-- Lower regression risk.
-- Clear docs for handoffs.
-- Repeatable release and maintenance workflow.

package/docs/project/resources.md

@@ -1,26 +0,0 @@
----
-summary: "Key resources for building and maintaining the extension."
-read_when:
-  - "Looking up references, docs, or operational artifacts."
-system4d:
-  container: "Reference catalog for the project."
-  compass: "Centralize discovery paths for maintainers."
-  engine: "Link docs, scripts, and examples used in execution."
-  fog: "External links may become stale over time."
----
-
-# Resources
-
-- [Extension entrypoint](../../extensions/evalset.ts)
-- [Prompt templates](../../prompts)
-- [Smoke dataset](../../examples/fixed-task-set.json)
-- [Larger dataset v2](../../examples/fixed-task-set-v2.json)
-- [Recommended dataset v3](../../examples/fixed-task-set-v3.json)
-- [Baseline system prompt](../../examples/system-baseline.txt)
-- [Candidate system prompt](../../examples/system-candidate.txt)
-- Local report artifacts path: `.evalset/reports/*.json` (and optional `*.html` exports)
-- [Project-local router extension](../../.pi/extensions/startup-intake-router.ts)
-- [Organization operating model](../org/operating_model.md)
-- [Interview questions](../org/project-docs-intake.questions.json)
-- [Security policy](../../policy/security-policy.json)
-- [Validation script](../../scripts/validate-structure.sh)

package/docs/project/skills.md

@@ -1,17 +0,0 @@
----
-summary: "Skills needed to maintain and extend this package."
-read_when:
-  - "Onboarding maintainers or planning implementation work."
-system4d:
-  container: "Capability map for project contributors."
-  compass: "Build confidence in extension APIs, testing, and release hygiene."
-  engine: "Assess gaps and train through small delivery cycles."
-  fog: "Skill requirements change as scope expands."
----
-
-# Skills
-
-- TypeScript extension development
-- Shell scripting and git hooks
-- Prompt template design
-- Security review and dependency hygiene

package/docs/project/strategic_goals.md

@@ -1,18 +0,0 @@
----
-summary: "Long-horizon goals for extension package maturity."
-read_when:
-  - "Roadmapping quarterly or milestone-level outcomes."
-system4d:
-  container: "Strategic objective set."
-  compass: "Reliability, maintainability, and team adoption."
-  engine: "Sequence milestones with measurable checks."
-  fog: "Long-term assumptions may need periodic resets."
----
-
-# Strategic goals
-
-1. Reach stable, documented `/evalset` behavior across interactive and non-interactive usage.
-2. Improve reproducibility signals (dataset hash, case hash, variant hash, run lineage) in all report paths.
-3. Keep extension docs concise and discoverable for new maintainers.
-4. Maintain secure, low-friction release and validation workflow.
-5. Expand confidence with focused tests for command parsing and report generation.

package/docs/project/tactical_goals.md

@@ -1,39 +0,0 @@
----
-summary: "Near-term execution goals tied to current sprint cycles."
-read_when:
-  - "Planning immediate tasks and delivery scope."
-system4d:
-  container: "Tactical work queue framing."
-  compass: "Ship the next smallest valuable increment safely."
-  engine: "Break work into verifiable, low-risk tasks."
-  fog: "Unexpected integration constraints may reprioritize tasks."
----
-
-# Tactical goals
-
-## Current cycle status
-
-### Completed
-- Added troubleshooting notes for `/evalset` invocation (interactive vs `pi -p` non-interactive).
-- Kept run/compare reports discoverable under `.evalset/reports/`.
-- Added stronger example datasets (`fixed-task-set-v2.json`, `fixed-task-set-v3.json`).
-- Synchronized docs and changelog with behavior updates.
-
-### Remaining
-1. Add smoke tests for argument parsing and report/scoring behavior.
-2. Add a repeatable JSON -> HTML report export helper.
-3. Complete npm publish after npmjs auth/registry setup.
-
-## Hard constraints
-- Time: keep work in small slices that can be reviewed quickly.
-- Quality: no regressions in existing `/evalset` commands.
-- Security: no insecure defaults in scripts or extension loading.
-- Scope: prioritize reproducibility and UX clarity over feature breadth.
-
-## Success criteria
-- [x] `npm run check` passes.
-- [x] Documented troubleshooting path for `pi -e` confusion exists.
-- [x] Reports include stable run metadata and are easy to locate.
-- [x] New maintainers can execute a compare workflow from README without intervention.
-- [ ] Automated tests cover parser/scoring basics.
-- [ ] Package is published on npm.

package/docs/project/vision.md

@@ -1,21 +0,0 @@
----
-summary: "Product and technical vision for pi-evalset-lab."
-read_when:
-  - "Defining or revisiting project direction."
-system4d:
-  container: "Project north-star statement."
-  compass: "Build a reliable pi extension package with low maintenance overhead."
-  engine: "Translate goals into concrete implementation slices."
-  fog: "Real user workflows may reshape priorities."
----
-
-# Vision
-
-Make `pi-evalset-lab` the lightweight default for comparing prompt/system variants with reproducible, inspectable reports.
-
-The project should feel predictable for maintainers:
-- clear command behavior in both interactive and `-p` modes,
-- clear report locations and run identity metadata,
-- clear docs for troubleshooting common setup confusion.
-
-Project vision is intentionally separate from organization vision in [Organization operating model](../org/operating_model.md).

package/policy/security-policy.json

@@ -1,10 +0,0 @@
-{
-  "default": {
-    "maxRiskScore": 70,
-    "requireIntegrity": true,
-    "requireSignatures": false,
-    "allowNpmDiffFallback": false,
-    "minReleaseAgeHours": 0
-  },
-  "packages": {}
-}

package/prek.toml

@@ -1,15 +0,0 @@
-minimum_prek_version = "0.3.0"
-
-[[repos]]
-repo = "local"
-hooks = [
-  {
-    id = "validate-structure",
-    name = "validate structure",
-    language = "system",
-    entry = "bash ./scripts/validate-structure.sh",
-    pass_filenames = false,
-    always_run = true,
-    stages = ["pre-commit"]
-  }
-]

package/scripts/docs-list.sh

@@ -1,50 +0,0 @@
-#!/usr/bin/env bash
-set -euo pipefail
-
-ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
-DEFAULT_SCRIPT="$HOME/ai-society/core/agent-scripts/scripts/docs-list.mjs"
-LOCAL_FALLBACK_SCRIPT="$ROOT_DIR/scripts/docs-list.mjs"
-
-usage() {
-  cat <<'USAGE'
-Usage: ./scripts/docs-list.sh [docs-list args]
-
-Resolves docs-list script in this order:
-1) DOCS_LIST_SCRIPT env var (absolute/relative path)
-2) Local fallback: ./scripts/docs-list.mjs
-3) Default global path: ~/ai-society/core/agent-scripts/scripts/docs-list.mjs
-
-Examples:
-  ./scripts/docs-list.sh
-  ./scripts/docs-list.sh --workspace --discover
-  ./scripts/docs-list.sh --json
-USAGE
-}
-
-if [[ "${1:-}" == "-h" || "${1:-}" == "--help" ]]; then
-  usage
-  exit 0
-fi
-
-if ! command -v node >/dev/null 2>&1; then
-  echo "Error: node is required to run docs-list." >&2
-  exit 1
-fi
-
-SCRIPT_PATH=""
-
-if [[ -n "${DOCS_LIST_SCRIPT:-}" ]]; then
-  SCRIPT_PATH="${DOCS_LIST_SCRIPT}"
-elif [[ -f "$LOCAL_FALLBACK_SCRIPT" ]]; then
-  SCRIPT_PATH="$LOCAL_FALLBACK_SCRIPT"
-else
-  SCRIPT_PATH="$DEFAULT_SCRIPT"
-fi
-
-if [[ ! -f "$SCRIPT_PATH" ]]; then
-  echo "Error: docs-list script not found: $SCRIPT_PATH" >&2
-  echo "Set DOCS_LIST_SCRIPT to your docs-list.mjs path, or install agent-scripts at: $DEFAULT_SCRIPT" >&2
-  exit 1
-fi
-
-node "$SCRIPT_PATH" "$@"

package/scripts/init-project-docs.sh

@@ -1,56 +0,0 @@
-#!/usr/bin/env bash
-set -euo pipefail
-
-ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
-QUESTIONS_FILE="$ROOT_DIR/docs/org/project-docs-intake.questions.json"
-ROUTER_FILE="$ROOT_DIR/.pi/extensions/startup-intake-router.ts"
-PROMPT_FILE="$ROOT_DIR/.pi/prompts/init-project-docs.md"
-
-INSTALL_INTERVIEW=false
-
-while [[ $# -gt 0 ]]; do
-  case "$1" in
-    --install-interview)
-      INSTALL_INTERVIEW=true
-      ;;
-    -h|--help)
-      echo "Usage: ./scripts/init-project-docs.sh [--install-interview]"
-      exit 0
-      ;;
-    *)
-      echo "Unknown arg: $1" >&2
-      echo "Usage: ./scripts/init-project-docs.sh [--install-interview]" >&2
-      exit 1
-      ;;
-  esac
-  shift
-done
-
-for required in "$QUESTIONS_FILE" "$ROUTER_FILE" "$PROMPT_FILE"; do
-  if [[ ! -f "$required" ]]; then
-    echo "Missing required file: $required" >&2
-    exit 1
-  fi
-done
-
-if [[ "$INSTALL_INTERVIEW" == "true" ]]; then
-  if command -v pi-interview >/dev/null 2>&1; then
-    pi-interview
-  else
-    echo "pi-interview command not found. Install the interactive-interview extension first, then rerun with --install-interview." >&2
-  fi
-fi
-
-echo "Interview-first document setup"
-echo ""
-echo "0) Optional: run npm run docs:list to review docs + read_when hints."
-echo "1) Ensure the interview extension is installed and loaded in pi."
-echo "2) Open this repository in pi."
-echo "3) Send a natural-language intent as the first non-command message in a session."
-echo "4) Router will prefill: /init-project-docs \"<intent>\""
-echo "5) Review/edit and run that command."
-echo ""
-echo "Fallback: run /init-project-docs manually."
-echo "Questions source: $QUESTIONS_FILE"
-echo ""
-echo "After updates, run: bash ./scripts/validate-structure.sh"

package/scripts/install-hooks.sh

@@ -1,13 +0,0 @@
-#!/usr/bin/env bash
-set -euo pipefail
-
-ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
-
-git -C "$ROOT_DIR" config core.hooksPath .githooks
-echo "Configured git hooks path: .githooks"
-
-if command -v prek >/dev/null 2>&1; then
-  echo "prek detected: pre-commit hook will run prek.toml"
-else
-  echo "prek not found: pre-commit hook will fallback to scripts/validate-structure.sh"
-fi

package/scripts/sync-to-live.sh

@@ -1,91 +0,0 @@
-#!/usr/bin/env bash
-set -euo pipefail
-
-WITH_POLICY=false
-WITH_PROMPTS=false
-
-usage() {
-  cat <<'USAGE'
-Usage: ./scripts/sync-to-live.sh [--with-policy] [--with-prompts] [--all]
-
-Copies all package extension entrypoints from ./extensions into ~/.pi/agent/extensions/.
-Optional flags also sync policy and prompt templates.
-USAGE
-}
-
-while [[ $# -gt 0 ]]; do
-  case "$1" in
-    --with-policy)
-      WITH_POLICY=true
-      ;;
-    --with-prompts)
-      WITH_PROMPTS=true
-      ;;
-    --all)
-      WITH_POLICY=true
-      WITH_PROMPTS=true
-      ;;
-    -h|--help)
-      usage
-      exit 0
-      ;;
-    *)
-      echo "Unknown arg: $1" >&2
-      usage
-      exit 1
-      ;;
-  esac
-  shift
-done
-
-ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
-SOURCE_DIR="$ROOT_DIR/extensions"
-TARGET_DIR="$HOME/.pi/agent/extensions"
-
-mkdir -p "$TARGET_DIR"
-
-shopt -s nullglob
-extension_files=("$SOURCE_DIR"/*.ts)
-if (( ${#extension_files[@]} == 0 )); then
-  echo "No extension files found in: $SOURCE_DIR" >&2
-  exit 1
-fi
-
-for source_file in "${extension_files[@]}"; do
-  target_file="$TARGET_DIR/$(basename "$source_file")"
-  cp "$source_file" "$target_file"
-  echo "Synced extension: $source_file -> $target_file"
-done
-shopt -u nullglob
-
-if [[ "$WITH_PROMPTS" == "true" ]]; then
-  PROMPT_SOURCE_DIR="$ROOT_DIR/prompts"
-  PROMPT_TARGET_DIR="$HOME/.pi/agent/prompts"
-  mkdir -p "$PROMPT_TARGET_DIR"
-
-  shopt -s nullglob
-  prompt_files=("$PROMPT_SOURCE_DIR"/*.md)
-  if (( ${#prompt_files[@]} == 0 )); then
-    echo "No prompt templates found in: $PROMPT_SOURCE_DIR"
-  else
-    for prompt_file in "${prompt_files[@]}"; do
-      cp "$prompt_file" "$PROMPT_TARGET_DIR/"
-      echo "Synced prompt: $prompt_file -> $PROMPT_TARGET_DIR/$(basename "$prompt_file")"
-    done
-  fi
-  shopt -u nullglob
-fi
-
-if [[ "$WITH_POLICY" == "true" ]]; then
-  POLICY_SOURCE="$ROOT_DIR/policy/security-policy.json"
-  POLICY_TARGET="$HOME/.pi/agent/security-policy.json"
-
-  if [[ -f "$POLICY_SOURCE" ]]; then
-    cp "$POLICY_SOURCE" "$POLICY_TARGET"
-    echo "Synced policy: $POLICY_SOURCE -> $POLICY_TARGET"
-  else
-    echo "Policy file not found: $POLICY_SOURCE (skipped)"
-  fi
-fi
-
-echo "Done. In pi, run /reload to pick up changes."