pi-para 0.5.2 → 0.6.0

package/README.md

@@ -1,287 +1,184 @@
 # pi-para
 
-A [pi](https://github.com/badlogic/pi-mono) extension that maintains a persistent, LLM-curated personal knowledge base structured by the [PARA method](https://fortelabs.com/blog/para/) (Projects, Areas, Resources, Archives). Inspired by [Karpathy's LLM wiki pattern](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f).
+A [pi](https://github.com/badlogic/pi-mono) extension that maintains a persistent, LLM-curated personal knowledge base structured by the [PARA method](https://fortelabs.com/blog/para/) — Projects, Areas, Resources, Archives.
 
-The LLM writes and maintains the wiki autonomously. You provide sources and ask questions. Search powered by [@picassio/qmd](https://github.com/picassio/qmd-1).
+pi-para runs as a normal Pi extension: no required daemon, no `systemd`, and no global QMD CLI. Search/indexing uses the embedded QMD SDK, and background capture/maintenance is coordinated by an in-process scheduler while Pi is open.
 
 ## Install
 
 ```bash
-pi install @picassio/pi-para
+pi install pi-para
+# or, for npm/npx setup flows:
+npx pi-para@latest setup
 ```
 
-Start pi — the extension loads automatically. Wiki directory is created on first session.
+POSIX one-liner:
 
-## Features
+```bash
+curl -fsSL https://raw.githubusercontent.com/picassio/pi-para/main/scripts/install.sh | bash
+```
 
-### Wiki Tools (8)
+Windows PowerShell:
 
-| Tool | Description |
-|------|-------------|
-| `wiki_ingest` | Ingest a URL, file, or text into the wiki |
-| `wiki_query` | Search the wiki with scope filtering, graph boost, freshness indicators |
-| `wiki_write` | Create or update wiki pages |
-| `wiki_read` | Read a specific wiki page + freshness indicator |
-| `wiki_move` | Move pages between PARA categories |
-| `wiki_lint` | Run health checks and auto-fix |
-| `wiki_summarize` | Summarize pages, categories, or entire wiki |
-| `wiki_migrate` | Batch-migrate all pages to current schema version |
-
-### Commands (11)
-
-| Command | Description |
-|---------|-------------|
-| `/wiki` | Status overview (scope, pages, log) |
-| `/wiki-search <query>` | Search the wiki |
-| `/wiki-ingest <url>` | Quick ingest |
-| `/wiki-capture [topic]` | Ask LLM to save knowledge via wiki_write |
-| `/wiki-scope` | Show or override project scope |
-| `/wiki-lint` | Run health checks |
-| `/wiki-summarize [target]` | Summarize wiki content |
-| `/wiki-settings` | Interactive settings (search, context, daemon, web UI, providers) |
-| `/wiki-daemon <cmd>` | Manage background capture daemon |
-| `/wiki-migrate` | Migrate all pages to current schema version |
-| `/wiki-project <name> <goal>` | Create or archive (`done`) a project page |
+```powershell
+irm https://raw.githubusercontent.com/picassio/pi-para/main/scripts/install.ps1 | iex
+```
 
-### Web Wiki UI
+Restart Pi after installing or changing extension registration.
 
-Browser-based wiki viewer, editor, and graph visualization.
+Then run:
 
+```bash
+pi-para doctor
 ```
-/wiki-settings → [WebWiki] → Enabled: true
+
+## What pi-para creates
+
+Default user files:
+
+```text
+~/.pi/para/config.jsonc       # canonical pi-para config
+~/.pi/para/secrets.json       # optional local secrets, chmod 0600
+~/.pi/wiki/                   # PARA markdown wiki
+~/.pi/wiki/.pi-para.sqlite    # scheduler queue/history/leases
+~/.pi/wiki/.qmd.sqlite        # QMD SDK search index
 ```
 
-Access at `http://<LAN-IP>:10973`. Features:
-- Page tree by PARA category (sidebar)
-- Markdown viewer with clickable `[[wikilinks]]`
-- Inline page editor
-- D3 force-directed graph view (page relationships)
-- Search
-- Move pages between categories
-- Activity log and session digests
-- Mobile-first responsive design, dark/light theme
+Legacy files such as `~/.pi/wiki/config.json`, `~/.pi/wiki/.daemon.sqlite`, and `~/.config/qmd/index.yml` are still read for migration/compatibility.
+
+## Core commands
 
-### GEPA Prompt Optimizer
+### Pi slash commands
 
-Automatically optimize all pi-para prompts, tool instructions, and skill guidelines using [DSPy GEPA](https://arxiv.org/abs/2507.19457) (Genetic-Pareto optimizer). Runs via `uv` with custom LLM providers — no litellm.
+| Command | Description |
+|---|---|
+| `/wiki` | Wiki status, current scope, page counts, recent log |
+| `/wiki-search <query>` | Search the wiki |
+| `/wiki-ingest <url-or-path-or-text>` | Ingest a source into `raw/` and synthesize pages |
+| `/wiki-capture [topic]` | Queue/capture current session knowledge |
+| `/wiki-scope [scope]` | Show or override current project scope |
+| `/wiki-lint [--report-only]` | Run wiki health checks |
+| `/wiki-summarize [target]` | Summarize a page/category/all |
+| `/wiki-settings` | Interactive settings for config/providers/scheduler |
+| `/wiki-scheduler [status\|queue\|history\|capture-history]` | Scheduler/capture observability |
+| `/wiki-daemon ...` | Compatibility alias for `/wiki-scheduler` |
+| `/wiki-migrate` | Batch-migrate pages to current schema |
+| `/wiki-project <name> <goal>` | Create/archive project pages |
+
+### CLI
 
 ```bash
-# List all 22 optimization targets
-pi-para-daemon gepa targets
+pi-para setup [--yes] [--dry-run] [--local PATH]
+pi-para doctor [--fix] [--json] [--test-capture-model]
+pi-para tasks [--status queued|running|done|failed]
+pi-para tasks show <id>
+pi-para tasks history [--task NAME] [--limit N]
+pi-para tasks retry [--task NAME]
+pi-para status [--json]
+pi-para capture-recent --hours 24
+pi-para providers
+pi-para providers set-secret <name> <value>
+pi-para providers remove-secret <name>
+pi-para gepa optimize --target capture-prompt
+```
 
-# Optimize a single target (uses config defaults: sonnet student + opus teacher)
-pi-para-daemon gepa optimize --target capture-prompt
+`pi-para-daemon` remains as a temporary compatibility binary, but the primary runtime is the in-process scheduler. Legacy daemon status is available as `pi-para legacy-status` for troubleshooting old installs.
 
-# Optimize all targets sequentially
-bash scripts/gepa/run-all.sh
+## Tools available to agents
 
-# Override models via CLI
-pi-para-daemon gepa optimize --target skill-para \
-  --student-model anthropic/claude-sonnet-4-20250514 \
-  --teacher-model anthropic/claude-opus-4-6 \
-  --judge-model anthropic/claude-sonnet-4-20250514
+| Tool | Description |
+|---|---|
+| `wiki_ingest` | Ingest URL/file/text into the wiki |
+| `wiki_query` | Search with scope/category/freshness context |
+| `wiki_edit` | Atomic exact `oldText → newText` page edits |
+| `wiki_write` | Create/append/replace pages, or legacy edit mode |
+| `wiki_read` | Read a page with freshness indicator |
+| `wiki_move` | Move a page between PARA categories |
+| `wiki_lint` | Run health checks and optional autofix |
+| `wiki_migrate` | Batch-migrate pages to current schema |
+| `wiki_summarize` | Summarize pages/categories/all |
 
-# Shorthand (--model = student, --reflection-model = teacher)
-pi-para-daemon gepa optimize --model anthropic/claude-sonnet-4-20250514 --reflection-model anthropic/claude-opus-4-6
+Prefer `wiki_edit` for surgical updates to existing pages.
 
-# See results
-pi-para-daemon gepa list
+## Configuration
 
-# Compare original vs optimized
-pi-para-daemon gepa compare --target capture-prompt
-```
+Use `/wiki-settings` or edit:
 
-**How it works:**
-
-1. TypeScript extracts all prompt/tool/skill texts from source files
-2. Calls `uv run scripts/gepa/optimize.py` with API keys from `~/.pi/agent/auth.json`
-3. Python builds a trainset from real wiki pages (93 pages → 30 train / 20 val)
-4. DSPy GEPA evolves each instruction using a lightweight proxy module (1 LLM call per eval, not 10-50)
-5. **Teacher** (Opus) proposes instruction mutations based on metric feedback
-6. **Student** (Sonnet) runs the proxy to generate output from the instruction
-7. **Judge** (Sonnet) scores on 6 dimensions: structure, PARA compliance, cross-references, security, completeness, actionability
-8. Optimized prompts saved to `~/.pi/wiki/gepa/optimized/`
-
-**Teacher/Student/Judge pattern** (DSPy GEPA):
-
-| Role | Default Model | What It Does | Calls/target |
-|------|---------------|--------------|--------|
-| **Student** | `claude-sonnet-4` | Runs proxy (generates wiki output) | ~460 |
-| **Teacher** | `claude-opus-4-6` | Proposes mutations via reflection | ~30 |
-| **Judge** | `claude-sonnet-4` | Scores output (LLM-as-judge metric) | ~460 |
-
-**22 optimization targets:**
-
-| Category | Targets |
-|----------|--------|
-| Prompt templates (12) | System, ingest, query, capture (×3), maintenance, processor, summarize, iterative, overview, lint |
-| Tool instructions (8) | wiki_ingest, wiki_query, wiki_write, wiki_read, wiki_move, wiki_lint, wiki_migrate, wiki_summarize |
-| Skills (2) | para (active PARA workflow), setup (installation guide) |
-
-**LLM providers** (custom `dspy.BaseLM` subclasses, no litellm):
-
-| Provider | Auth | Notes |
-|----------|------|-------|
-| Anthropic | OAuth token from `auth.json` | Claude Code billing headers + user-agent, auto token refresh |
-| MiniMax | API key | Via Anthropic-compatible API |
-| OpenRouter | API key | Via OpenAI-compatible API |
-
-**Configuration** (`~/.pi/wiki/config.json`):
-
-```json
-{
-  "gepa": {
-    "useOptimized": true,
-    "studentModel": "anthropic/claude-sonnet-4-20250514",
-    "teacherModel": "anthropic/claude-opus-4-6",
-    "judgeModel": null,
-    "auto": "light",
-    "threads": 2,
-    "seed": 42
-  }
-}
+```text
+~/.pi/para/config.jsonc
 ```
 
-- `useOptimized: true` — load optimized prompts at runtime (enabled by default)
-- `studentModel` — fast model for running proxy + judging (Sonnet recommended)
-- `teacherModel` — smart model for proposing mutations (Opus recommended)
-- `judgeModel` — model for scoring output (defaults to studentModel if null)
-- All settings overridable via CLI flags
+Credential refs use one of:
 
-When `useOptimized` is enabled, `getPrompt(name)` checks: user-generated → bundled optimized → original. Originals are always preserved.
+```text
+pi-auth:<provider>   # Pi AuthStorage, e.g. ~/.pi/agent/auth.json
+secret:<name>        # ~/.pi/para/secrets.json
+none                 # local/no-auth provider
+```
 
-**Budget presets:**
+Setup does not require or write API-key environment variables.
 
-| Preset | Rollouts | Time estimate | When to use |
-|--------|----------|---------------|-------------|
-| `light` | ~460 | 1-2 hours | Quick sanity check |
-| `medium` | ~1500 | 4-6 hours | Everyday optimization |
-| `heavy` | ~5000 | 12-24 hours | Final tuning |
+See [CONFIGURATION.md](./CONFIGURATION.md) for the full schema.
 
-**Prerequisites:** `uv` (Python package manager). DSPy + anthropic SDK installed automatically on first run.
+## Search and QMD
 
-### Background Capture Daemon
+QMD is used as an embedded SDK through `qmd-engine`; users do not need to install a global `qmd` CLI.
 
-Processes session files in the background after you quit pi:
+- BM25 keyword search works immediately.
+- Embedding/rerank can be configured in `/wiki-settings` or `~/.pi/para/config.jsonc`.
+- Legacy `~/.config/qmd/index.yml` is only a compatibility fallback.
 
-```bash
-# Via extension command
-/wiki-daemon start
-/wiki-daemon status
+## Background capture and maintenance
 
-# Via CLI
-pi-para-daemon start
-pi-para-daemon process-recent --hours 24
-```
+When Pi is running, pi-para starts an in-process scheduler. It handles:
 
-The daemon uses an Agent with session exploration tools (session_slice, session_search, session_stats) to iteratively explore sessions of any size and extract knowledge.
+- queued session capture,
+- startup catch-up from `.completed-sessions`,
+- index/search maintenance,
+- queue/history/lease tracking in `.pi-para.sqlite`.
 
-Auto-starts via systemd on Linux — run `./setup.sh` once for daemon service setup.
+No background OS service is required. If Pi is closed, queued work resumes on the next Pi session.
 
-## Architecture
+## Web Wiki UI
 
-```
-~/.pi/wiki/
-├── config.json        # all settings (context, search, daemon, web UI, gepa)
-├── schema.md          # PARA conventions
-├── index.md           # page catalog (auto-rebuilt)
-├── log.md             # activity log
-├── sessions.md        # session digest log
-├── .completed-sessions # daemon registry
-├── .daemon.sqlite     # daemon state
-├── .qmd.sqlite        # search index
-├── projects/          # active, goal-defined work
-├── areas/             # ongoing responsibilities
-├── resources/         # reference material
-├── archives/          # completed/deprecated
-├── raw/               # immutable source material
-└── gepa/              # GEPA optimizer state
-    ├── input/         # targets.json (generated per run)
-    ├── output/        # results.json (GEPA output)
-    ├── optimized/     # deployed optimized prompts (*.txt)
-    └── history/       # version history per target
+The optional web wiki can be enabled from `/wiki-settings`:
+
+```text
+/wiki-settings → [WebWiki] Enabled
 ```
 
-**Python scripts (GEPA optimizer):**
+It serves the wiki viewer/editor/graph on the configured host/port.
 
-```
-scripts/gepa/
-├── pyproject.toml     # uv project (dspy, anthropic, openai deps)
-├── optimize.py        # Main entry — runs DSPy GEPA
-├── lm_providers.py    # Custom BaseLM: AnthropicOAuthLM, MiniMaxLM, OpenRouterLM
-├── program.py         # PromptProxy DSPy Module (1 LLM call per eval)
-├── metric.py          # LLM-as-judge metric (6 dimensions)
-├── dataset.py         # Build train/val from real wiki pages
-└── wiki_reader.py     # Read wiki pages from disk
-```
+## GEPA Prompt Optimizer
 
-### How it works
-
-1. **During a session** — LLM uses wiki tools (query, write, ingest). Context injection provides wiki knowledge on every turn.
-2. **On /quit** — session registered in `.completed-sessions` (instant, <10ms)
-3. **Background** — daemon picks up session, Agent explores with tools, writes wiki pages
-4. **Next session** — new wiki pages appear in context injection
-
-### Freshness Verification
-
-Every `wiki_query` and `wiki_read` result includes a freshness indicator based on the page's last-updated date:
-
-| Age | Indicator | LLM Behavior |
-|-----|-----------|---------------|
-| < 7 days | ✅ FRESH | Trust normally |
-| 7-14 days | ✅ Recent | Trust normally |
-| 14-30 days | ⚠️ AGING | Verify claims about code/configs before trusting |
-| 30-90 days | ⚠️ STALE | Verify before trusting |
-| > 90 days | 🚨 VERY STALE | Likely outdated, verify everything |
-
-The LLM is instructed to:
-- **Verify** stale claims by checking actual code, configs, or files
-- **Self-heal** — fix incorrect wiki pages via `wiki_write(mode: 'edit')` when it discovers they're wrong
-- **Flag uncertainty** — tell the user when a claim can't be verified (e.g. external services)
-
-### Configuration
-
-All settings via `/wiki-settings` interactive menu, or edit `~/.pi/wiki/config.json`:
-
-```json
-{
-  "contextMaxTokens": 4000,
-  "searchLimit": 10,
-  "searchGraphBoost": true,
-  "lintAutoFix": true,
-  "lintStaleDays": 90,
-  "daemonModel": null,
-  "webWiki": { "enabled": true, "host": "0.0.0.0", "port": 10973 },
-  "gepa": { "useOptimized": false }
-}
-```
+pi-para can optimize prompts/tools/skills using DSPy GEPA via `uv` and custom DSPy `BaseLM` providers.
 
-Search providers in `~/.config/qmd/index.yml`:
-
-```yaml
-providers:
-  embed:
-    url: https://openrouter.ai/api/v1
-    key: sk-or-...
-    model: openai/text-embedding-3-small
-  chat:
-    url: https://api.minimaxi.com/anthropic
-    key: sk-cp-...
-    model: MiniMax-M2.7-highspeed
-    api: anthropic
-```
+Recommended roles:
 
-### Optional: Enhanced search
+| Role | Default |
+|---|---|
+| Student | `anthropic/claude-sonnet-4-20250514` |
+| Teacher/reflection | `anthropic/claude-opus-4-6` |
+| Judge | `anthropic/claude-sonnet-4-20250514` |
+
+Examples:
 
 ```bash
-npm install -g @picassio/qmd
+pi-para gepa targets
+pi-para gepa optimize --target capture-prompt
+pi-para gepa optimize --target skill-para \
+  --student-model anthropic/claude-sonnet-4-20250514 \
+  --teacher-model anthropic/claude-opus-4-6 \
+  --judge-model anthropic/claude-sonnet-4-20250514
+pi-para gepa list
+pi-para gepa compare --target capture-prompt
 ```
 
-Without qmd: BM25 keyword search. With qmd + providers: hybrid BM25 + vector + rerank.
+GEPA uses `~/.pi/agent/auth.json`/custom providers; it does not use LiteLLM.
 
-## Obsidian Compatibility
+## Obsidian compatibility
 
-The wiki is a directory of markdown files with YAML frontmatter and `[[wikilinks]]`. Open `~/.pi/wiki/` as an Obsidian vault for graph view, Dataview queries, and manual editing.
+The wiki is plain markdown with YAML frontmatter and `[[wikilinks]]`. You can open `~/.pi/wiki/` as an Obsidian vault.
 
 ## Development
 
@@ -289,16 +186,22 @@ The wiki is a directory of markdown files with YAML frontmatter and `[[wikilinks
 git clone https://github.com/picassio/pi-para.git
 cd pi-para
 npm install
-npm run check    # TypeScript check
-npm test         # Run tests (301 tests)
-npm run build    # Compile to dist/
+npm run check
+npm test
+npm run test:coverage
+npm run build
+npm run smoke:install
+
+# Local setup into Pi settings
+node ./dist/cli.js setup --local .
+```
 
-# Dev mode: symlink extension
-ln -sf $(pwd) ~/.pi/agent/extensions/pi-para
+More docs:
 
-# Run daemon locally
-npx tsx src/cli.ts start
-```
+- [SETUP.md](./SETUP.md)
+- [ARCHITECTURE.md](./ARCHITECTURE.md)
+- [STRUCTURE.md](./STRUCTURE.md)
+- [CONFIGURATION.md](./CONFIGURATION.md)
 
 ## License