skills-eval 0.0.1

package/README.md

@@ -0,0 +1,92 @@
+# Skills Eval
+
+A tool for evaluating and improving Claude Code skills (agent prompts). Two main features:
+
+1. **Eval Pipeline** — Run skill outputs against test cases, grade them with assertions, compare iterations, and iterate on the skill prompt until quality improves.
+2. **CLI Playground** — Experiment with `claude -p` flag combinations (model, effort, tools, JSON schema, system prompt, MCP config, etc.) in a visual interface before using them in production.
+
+## Quick Start
+
+```bash
+bun install
+bun run dev
+```
+
+Open http://localhost:5173 for the eval pipeline, or http://localhost:5173/playground for the CLI playground.
+
+## Eval Pipeline
+
+The main product. Given a skill (a markdown prompt file) and context about the user's use case:
+
+1. **Generate test cases** — Create diverse prompts to exercise the skill
+2. **Run evals** — Execute the skill against test cases with and without the skill prompt
+3. **Grade results** — Write assertions and judge outputs
+4. **Compare iterations** — See how changes to the skill affect output quality
+5. **Improve** — Iterate on the skill prompt based on grading feedback
+
+### Usage
+
+```bash
+# Dev mode with example skill (single skill, HMR)
+bun run dev
+
+# With a custom skill
+bun run dev:cli -- --skill-path path/to/SKILL.md --context "description of the use case"
+
+# Production-like (build first)
+bun run build
+bun run dogfood:launch
+```
+
+### Multi-Skill Support
+
+You can evaluate multiple skills concurrently. Each skill gets its own server on an auto-selected port:
+
+```bash
+bun run build
+bun bin/skill-eval.ts --skill-path path/to/skill-a/SKILL.md --context "context a"
+bun bin/skill-eval.ts --skill-path path/to/skill-b/SKILL.md --context "context b"
+```
+
+Each launch auto-selects an available port (starting from 3000), opens a browser tab, and tracks its state in `.skill-eval/servers.json`. If a server is already running for a skill, subsequent launches reuse it instead of starting a new one.
+
+All skills within the same project share a single SQLite database (`.skill-eval/eval.db`), with data logically separated by skill ID.
+
+## CLI Playground
+
+An interactive tool for experimenting with `claude -p` (print mode) flags. Useful for understanding what settings produce the best results before wiring them into the eval pipeline.
+
+**Features:**
+- Controls panel with all relevant CLI flags (model, effort, output format, tools, JSON schema, system prompt, debug, MCP, setting sources, max turns)
+- Interdependency enforcement (e.g., verbose auto-enables stream-json, effort max requires opus)
+- Real execution via server-spawned `claude -p` with streaming output via WebSocket
+- Tabbed run history with per-run stats (duration, cost, tool counts, model)
+- Copy button generates a ready-to-paste CLI command (prompt piped via stdin)
+- Subfolder picker for `claude-cli/` configurations — each subfolder can have its own `.claude/settings.json`
+
+### CLI Config Subfolders
+
+```bash
+# Create a config for testing with specific settings
+mkdir -p claude-cli/my-experiment/.claude
+echo '{"permissions":{"allow":["Bash(*)"]}}' > claude-cli/my-experiment/.claude/settings.json
+```
+
+The playground auto-detects subfolders in `claude-cli/` and lets you pick which one to `cd` into before running.
+
+## Development
+
+```bash
+bun run dev          # Full dev (Fastify + Vite HMR)
+bun run build        # Build for production
+bun run typecheck    # Type check
+bun x vitest run     # Run tests
+bun run check        # Lint
+```
+
+## Stack
+
+- **Frontend**: React 19, Vite 8, Tailwind CSS v4, shadcn/ui (radix-nova)
+- **Backend**: Fastify 5, SQLite (drizzle-orm + libsql)
+- **Testing**: Vitest 4, Testing Library
+- **Runtime**: Bun, TypeScript 5.9